2024-09-17T03:36:29.623 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T03:36:29.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T03:36:29.733 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469 branch: wip-pdonnell-testing-20240916.200549-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/strays} email: pdonnell@redhat.com first_in_suite: false job_id: '7908469' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240916.200549-debug ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: client: pid file: '' global: ms die on skipped message: false mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240916.200549-debug sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 476 sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf sleep_before_teardown: 0 subset: 320/512 suite: fs suite_branch: wip-pdonnell-testing-20240916.200549-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/X1ct/74uIAtWyMbN03U1IwGslFTOLGw+g5EBXz5zh8vg6DyF4lULj8OP83ADTk1SuvwwBB3niYyJM+EOYDjs= smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+B8jrcq2WyWnoS+RzNvCJaeBL4uSGLOQDqINksiCQ8QBvsZgw75cygt+G8c8zbb+56qjaDOdHgkJ5LTxvIGeo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_strays teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_02:09:51 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T03:36:29.734 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa; will attempt to use it 2024-09-17T03:36:29.734 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa/tasks 2024-09-17T03:36:29.735 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T03:36:29.736 INFO:teuthology.task.internal:Checking packages... 2024-09-17T03:36:29.758 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf' 2024-09-17T03:36:29.758 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T03:36:29.758 INFO:teuthology.packaging:ref: None 2024-09-17T03:36:29.758 INFO:teuthology.packaging:tag: None 2024-09-17T03:36:29.758 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240916.200549-debug 2024-09-17T03:36:29.758 INFO:teuthology.packaging:sha1: ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T03:36:29.758 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240916.200549-debug 2024-09-17T03:36:29.896 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4989.gce1b342b 2024-09-17T03:36:29.898 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T03:36:29.904 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T03:36:29.904 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T03:36:29.933 INFO:teuthology.task.internal:Saving configuration 2024-09-17T03:36:29.957 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T03:36:29.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T03:36:29.989 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 03:31:25.192827', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/X1ct/74uIAtWyMbN03U1IwGslFTOLGw+g5EBXz5zh8vg6DyF4lULj8OP83ADTk1SuvwwBB3niYyJM+EOYDjs='} 2024-09-17T03:36:30.009 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 03:31:25.191017', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+B8jrcq2WyWnoS+RzNvCJaeBL4uSGLOQDqINksiCQ8QBvsZgw75cygt+G8c8zbb+56qjaDOdHgkJ5LTxvIGeo='} 2024-09-17T03:36:30.009 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T03:36:30.015 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-17T03:36:30.015 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-17T03:36:30.016 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T03:36:30.064 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4207d33ac0>, signals=[15]) 2024-09-17T03:36:30.064 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T03:36:30.069 INFO:teuthology.task.internal:Opening connections... 2024-09-17T03:36:30.070 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:36:30.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:36:30.147 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:36:30.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:36:30.220 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T03:36:30.226 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-17T03:36:30.244 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-17T03:36:30.245 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-17T03:36:30.302 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-09-17T03:36:30.303 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T03:36:30.304 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T03:36:30.304 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T03:36:30.304 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-17T03:36:30.322 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-17T03:36:30.341 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-17T03:36:30.341 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-17T03:36:30.400 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-17T03:36:30.400 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T03:36:30.401 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T03:36:30.402 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-17T03:36:30.431 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T03:36:30.440 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T03:36:30.445 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T03:36:30.446 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-17T03:36:30.448 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-17T03:36:30.466 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T03:36:30.472 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T03:36:30.472 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-17T03:36:30.506 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-17T03:36:30.592 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T03:36:30.647 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T03:36:30.661 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-17T03:36:30.661 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-09-17T03:36:30.661 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-09-17T03:36:30.661 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T03:36:30.662 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T03:36:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:36:30.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-17T03:36:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:36:30.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-17T03:36:30.853 DEBUG:teuthology.task.kernel:sha1 for client.1 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:36:30.853 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-17T03:36:30.854 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-17T03:36:30.858 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:36:30.858 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-17T03:36:30.858 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-17T03:36:30.871 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-17T03:36:30.872 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:36:30.872 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-17T03:36:30.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum upgrade -y linux-firmware 2024-09-17T03:36:30.876 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-503.el9.x86_64 2024-09-17T03:36:30.876 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:36:30.876 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-17T03:36:30.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum upgrade -y linux-firmware 2024-09-17T03:36:33.216 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-17T03:36:34.145 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.3 MB 00:01 2024-09-17T03:36:34.958 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-17T03:36:37.410 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 9.4 MB/s | 20 MB 00:02 2024-09-17T03:36:42.875 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.5 MB 00:04 2024-09-17T03:36:43.229 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.5 MB 00:02 2024-09-17T03:36:45.028 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 19 kB 00:00 2024-09-17T03:36:45.234 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-17T03:36:46.430 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-17T03:36:46.574 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-17T03:36:51.885 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-17T03:36:51.987 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-09-17T03:36:53.870 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout: linux-firmware noarch 20240905-146.el9 baseos 388 M 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-17T03:36:53.879 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 1 Package 2024-09-17T03:36:53.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:36:53.880 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 388 M 2024-09-17T03:36:53.880 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T03:36:53.935 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout: linux-firmware noarch 20240905-146.el9 baseos 388 M 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:36:53.945 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-17T03:36:53.946 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-17T03:36:53.946 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 1 Package 2024-09-17T03:36:53.946 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:36:53.946 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 388 M 2024-09-17T03:36:53.946 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-17T03:36:54.083 INFO:teuthology.orchestra.run.smithi181.stdout:[MIRROR] linux-firmware-20240905-146.el9.noarch.rpm: Status code: 404 for http://us.mirrors.cicku.me/centos/9-stream/BaseOS/x86_64/os/Packages/linux-firmware-20240905-146.el9.noarch.rpm (IP: 172.65.90.22) 2024-09-17T03:36:54.154 INFO:teuthology.orchestra.run.smithi181.stdout:[MIRROR] linux-firmware-20240905-146.el9.noarch.rpm: Status code: 404 for https://us.mirrors.cicku.me/centos/9-stream/BaseOS/x86_64/os/Packages/linux-firmware-20240905-146.el9.noarch.rpm (IP: 172.65.90.21) 2024-09-17T03:38:21.059 INFO:teuthology.orchestra.run.smithi181.stdout:linux-firmware-20240905-146.el9.noarch.rpm 4.5 MB/s | 388 MB 01:27 2024-09-17T03:38:21.060 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T03:38:21.060 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.5 MB/s | 388 MB 01:27 2024-09-17T03:38:23.155 INFO:teuthology.orchestra.run.smithi129.stdout:linux-firmware-20240905-146.el9.noarch.rpm 4.4 MB/s | 388 MB 01:29 2024-09-17T03:38:23.155 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-17T03:38:23.155 INFO:teuthology.orchestra.run.smithi129.stdout:Total 4.4 MB/s | 388 MB 01:29 2024-09-17T03:38:24.189 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T03:38:24.350 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T03:38:24.350 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T03:38:26.279 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-17T03:38:26.442 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-17T03:38:26.442 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-17T03:38:27.629 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T03:38:27.629 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T03:38:29.696 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-17T03:38:29.697 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-17T03:38:37.027 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T03:38:37.753 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-17T03:38:37.842 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:38.325 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:38.325 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-17T03:38:38.972 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-17T03:38:39.087 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:39.088 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:38:39.088 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-17T03:38:39.088 INFO:teuthology.orchestra.run.smithi181.stdout: linux-firmware-20240905-146.el9.noarch 2024-09-17T03:38:39.088 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:38:39.088 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T03:38:39.427 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-17T03:38:39.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:38:39.568 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-17T03:38:39.568 DEBUG:teuthology.orchestra.run.smithi181:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-17T03:38:39.571 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi181.front.sepia.ceph.com to complete... 2024-09-17T03:38:39.604 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-17T03:38:39.694 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:40.184 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:40.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : linux-firmware-20240905-146.el9.noarch 1/2 2024-09-17T03:38:40.936 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 2/2 2024-09-17T03:38:40.937 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:38:40.937 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-17T03:38:40.937 INFO:teuthology.orchestra.run.smithi129.stdout: linux-firmware-20240905-146.el9.noarch 2024-09-17T03:38:40.937 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:38:40.937 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-17T03:38:41.262 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-17T03:38:41.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-17T03:38:41.402 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-17T03:38:41.402 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-17T03:38:41.405 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi129.front.sepia.ceph.com to complete... 2024-09-17T03:38:49.346 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17 03:38:49 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-17T03:38:49.347 INFO:teuthology.orchestra.run.smithi181.stderr:FINISHED --2024-09-17 03:38:49-- 2024-09-17T03:38:49.347 INFO:teuthology.orchestra.run.smithi181.stderr:Total wall clock time: 9.8s 2024-09-17T03:38:49.347 INFO:teuthology.orchestra.run.smithi181.stderr:Downloaded: 1 files, 343M in 9.5s (36.2 MB/s) 2024-09-17T03:38:49.350 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-17T03:38:49.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-17T03:38:51.407 INFO:teuthology.orchestra.run.smithi181.stdout:Verifying... ######################################## 2024-09-17T03:38:51.535 INFO:teuthology.orchestra.run.smithi181.stdout:Preparing... ######################################## 2024-09-17T03:38:51.594 INFO:teuthology.orchestra.run.smithi181.stdout:Updating / installing... 2024-09-17T03:38:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17 03:38:51 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-17T03:38:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:FINISHED --2024-09-17 03:38:51-- 2024-09-17T03:38:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:Total wall clock time: 10s 2024-09-17T03:38:51.917 INFO:teuthology.orchestra.run.smithi129.stderr:Downloaded: 1 files, 343M in 10s (34.0 MB/s) 2024-09-17T03:38:51.921 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-17T03:38:51.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-17T03:38:53.994 INFO:teuthology.orchestra.run.smithi129.stdout:Verifying... ######################################## 2024-09-17T03:38:54.053 INFO:teuthology.orchestra.run.smithi129.stdout:Preparing... ######################################## 2024-09-17T03:38:54.112 INFO:teuthology.orchestra.run.smithi129.stdout:Updating / installing... 2024-09-17T03:38:59.576 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-17T03:39:02.100 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-17T03:39:39.715 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-17T03:39:39.716 DEBUG:teuthology.orchestra.run.smithi181:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-17T03:39:39.802 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-17T03:39:39.803 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-17T03:39:39.804 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-17T03:39:39.805 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-17T03:39:39.806 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-17T03:39:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-17T03:39:39.808 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-17T03:39:39.809 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-17T03:39:39.810 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-17T03:39:39.811 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-17T03:39:39.812 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-17T03:39:39.813 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-17T03:39:39.814 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-17T03:39:39.815 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-17T03:39:39.816 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-17T03:39:39.817 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-17T03:39:39.818 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-17T03:39:39.819 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-17T03:39:39.820 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-17T03:39:39.821 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-17T03:39:39.822 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-17T03:39:39.823 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-17T03:39:39.824 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-17T03:39:39.825 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-17T03:39:39.826 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-17T03:39:39.827 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-17T03:39:39.828 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-17T03:39:39.829 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-17T03:39:39.830 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-17T03:39:39.831 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-17T03:39:39.832 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-17T03:39:39.833 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-17T03:39:39.834 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-17T03:39:39.835 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-17T03:39:39.836 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-17T03:39:39.837 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-17T03:39:39.838 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-17T03:39:39.839 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-17T03:39:39.840 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-17T03:39:39.841 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-17T03:39:39.842 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-17T03:39:39.843 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-17T03:39:39.844 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-17T03:39:39.845 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-17T03:39:39.846 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-17T03:39:39.847 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-17T03:39:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-17T03:39:39.849 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-17T03:39:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-17T03:39:39.851 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-17T03:39:39.852 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-17T03:39:39.853 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-17T03:39:39.854 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-17T03:39:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-17T03:39:39.856 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-17T03:39:39.857 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-17T03:39:39.858 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-17T03:39:39.859 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-17T03:39:39.860 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-17T03:39:39.861 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-17T03:39:39.862 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-17T03:39:39.863 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-17T03:39:39.864 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-17T03:39:39.865 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-17T03:39:39.866 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-17T03:39:39.867 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-17T03:39:39.868 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-17T03:39:39.869 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-17T03:39:39.870 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-17T03:39:39.871 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-17T03:39:39.872 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-17T03:39:39.873 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-17T03:39:39.874 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-17T03:39:39.875 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-17T03:39:39.876 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-17T03:39:39.877 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-17T03:39:39.878 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-17T03:39:39.879 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-17T03:39:39.880 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-17T03:39:39.881 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-17T03:39:39.882 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-17T03:39:39.883 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-17T03:39:39.884 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-17T03:39:39.885 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-17T03:39:39.886 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-17T03:39:39.887 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-17T03:39:39.888 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-17T03:39:39.889 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-17T03:39:39.890 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-17T03:39:39.891 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-17T03:39:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-17T03:39:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-17T03:39:39.894 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-17T03:39:39.895 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-17T03:39:39.896 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-17T03:39:39.897 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-17T03:39:39.898 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-17T03:39:39.899 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-17T03:39:39.900 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-17T03:39:39.901 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-17T03:39:39.902 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-17T03:39:39.903 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-17T03:39:39.904 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-17T03:39:39.905 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-17T03:39:39.906 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-17T03:39:39.907 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-17T03:39:39.908 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-17T03:39:39.909 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-17T03:39:39.910 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-17T03:39:39.911 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-17T03:39:39.912 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-17T03:39:39.913 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-17T03:39:39.914 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-17T03:39:39.915 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-17T03:39:39.916 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-17T03:39:39.917 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-17T03:39:39.918 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-17T03:39:39.919 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-17T03:39:39.920 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-17T03:39:39.921 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-17T03:39:39.922 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-17T03:39:39.923 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-17T03:39:39.924 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-17T03:39:39.925 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-17T03:39:39.926 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-17T03:39:39.927 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-17T03:39:39.928 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-17T03:39:39.929 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-17T03:39:39.930 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-17T03:39:39.931 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-17T03:39:39.932 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-17T03:39:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-17T03:39:39.934 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-17T03:39:39.935 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-17T03:39:39.936 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-17T03:39:39.937 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-17T03:39:39.938 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-17T03:39:39.939 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-17T03:39:39.940 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-17T03:39:39.941 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-17T03:39:39.942 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-17T03:39:39.943 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-17T03:39:39.944 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-17T03:39:39.945 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-17T03:39:39.946 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-17T03:39:39.947 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-17T03:39:39.948 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-17T03:39:39.949 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-17T03:39:39.950 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-17T03:39:39.951 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-17T03:39:39.952 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-17T03:39:39.953 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-17T03:39:39.954 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-17T03:39:39.955 INFO:teuthology.orchestra.run.smithi181.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-17T03:39:39.959 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.959 DEBUG:teuthology.orchestra.run.smithi181:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-17T03:39:39.983 INFO:teuthology.orchestra.run.smithi181.stdout:postinstall scriptlet (using /bin/sh): 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:for file in vmlinuz System.map config; do 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout: fi 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:done 2024-09-17T03:39:39.984 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:39.985 INFO:teuthology.orchestra.run.smithi181.stdout:fi 2024-09-17T03:39:39.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-17T03:39:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-17T03:39:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-17T03:39:40.062 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T03:39:40.063 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:39:40.064 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T03:39:40.065 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T03:39:40.065 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:40.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T03:39:40.593 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-17T03:39:41.941 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T03:39:41.976 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-17T03:39:41.986 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-17T03:39:42.001 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.GXigCwWYEk 2024-09-17T03:39:42.002 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GXigCwWYEk 2024-09-17T03:39:42.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.GXigCwWYEk 2024-09-17T03:39:42.213 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.GXigCwWYEk is 6KB 2024-09-17T03:39:42.263 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.GXigCwWYEk 2024-09-17T03:39:42.278 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T03:39:42.341 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T03:39:42.341 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T03:39:42.341 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T03:39:42.341 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-17T03:39:42.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-17T03:39:42.726 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-17T03:39:42.727 DEBUG:teuthology.orchestra.run.smithi129:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-17T03:39:42.813 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-17T03:39:42.814 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-17T03:39:42.815 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-17T03:39:42.816 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-17T03:39:42.817 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-17T03:39:42.818 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-17T03:39:42.819 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-17T03:39:42.820 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-17T03:39:42.821 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-17T03:39:42.822 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-17T03:39:42.823 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-17T03:39:42.824 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-17T03:39:42.825 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-17T03:39:42.826 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-17T03:39:42.827 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-17T03:39:42.828 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-17T03:39:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-17T03:39:42.830 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-17T03:39:42.831 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-17T03:39:42.832 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-17T03:39:42.833 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-17T03:39:42.834 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-17T03:39:42.835 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-17T03:39:42.836 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-17T03:39:42.837 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-17T03:39:42.840 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-17T03:39:42.841 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-17T03:39:42.842 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-17T03:39:42.843 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-17T03:39:42.844 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-17T03:39:42.845 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-17T03:39:42.846 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-17T03:39:42.847 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-17T03:39:42.848 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-17T03:39:42.849 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-17T03:39:42.850 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-17T03:39:42.851 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-17T03:39:42.852 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-17T03:39:42.853 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-17T03:39:42.854 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-17T03:39:42.855 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-17T03:39:42.856 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-17T03:39:42.857 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-17T03:39:42.858 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-17T03:39:42.859 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-17T03:39:42.860 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-17T03:39:42.861 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-17T03:39:42.862 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-17T03:39:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-17T03:39:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-17T03:39:42.865 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-17T03:39:42.866 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-17T03:39:42.867 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-17T03:39:42.868 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-17T03:39:42.869 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-17T03:39:42.870 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-17T03:39:42.871 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-17T03:39:42.872 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-17T03:39:42.873 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-17T03:39:42.874 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-17T03:39:42.875 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-17T03:39:42.876 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-17T03:39:42.877 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-17T03:39:42.878 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-17T03:39:42.879 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-17T03:39:42.880 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-17T03:39:42.881 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-17T03:39:42.882 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-17T03:39:42.883 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-17T03:39:42.884 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-17T03:39:42.885 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-17T03:39:42.886 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-17T03:39:42.887 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-17T03:39:42.888 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-17T03:39:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-17T03:39:42.890 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-17T03:39:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-17T03:39:42.892 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-17T03:39:42.893 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-17T03:39:42.894 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-17T03:39:42.895 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-17T03:39:42.896 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-17T03:39:42.897 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-17T03:39:42.898 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-17T03:39:42.899 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-17T03:39:42.900 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-17T03:39:42.901 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-17T03:39:42.902 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-17T03:39:42.903 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-17T03:39:42.904 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-17T03:39:42.905 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-17T03:39:42.906 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-17T03:39:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-17T03:39:42.908 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-17T03:39:42.909 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-17T03:39:42.910 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-17T03:39:42.911 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-17T03:39:42.912 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-17T03:39:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-17T03:39:42.914 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-17T03:39:42.915 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-17T03:39:42.916 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-17T03:39:42.917 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-17T03:39:42.918 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-17T03:39:42.919 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-17T03:39:42.920 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-17T03:39:42.921 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-17T03:39:42.922 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-17T03:39:42.923 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-17T03:39:42.924 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-17T03:39:42.925 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-17T03:39:42.926 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-17T03:39:42.927 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-17T03:39:42.928 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-17T03:39:42.929 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-17T03:39:42.930 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-17T03:39:42.931 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-17T03:39:42.932 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-17T03:39:42.933 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-17T03:39:42.934 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-17T03:39:42.935 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-17T03:39:42.936 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-17T03:39:42.937 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-17T03:39:42.938 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-17T03:39:42.939 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-17T03:39:42.940 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-17T03:39:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-17T03:39:42.942 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-17T03:39:42.943 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-17T03:39:42.944 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-17T03:39:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-17T03:39:42.946 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-17T03:39:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-17T03:39:42.948 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-17T03:39:42.949 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-17T03:39:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-17T03:39:42.951 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-17T03:39:42.952 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-17T03:39:42.953 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-17T03:39:42.954 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-17T03:39:42.955 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-17T03:39:42.956 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-17T03:39:42.957 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-17T03:39:42.958 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-17T03:39:42.959 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-17T03:39:42.960 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-17T03:39:42.961 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-17T03:39:42.962 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-17T03:39:42.963 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-17T03:39:42.964 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-17T03:39:42.965 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-17T03:39:42.966 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-17T03:39:42.966 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-17T03:39:42.966 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-17T03:39:42.966 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-17T03:39:42.966 INFO:teuthology.orchestra.run.smithi129.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-17T03:39:42.969 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.970 DEBUG:teuthology.orchestra.run.smithi129:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout:postinstall scriptlet (using /bin/sh): 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout:fi 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout:for file in vmlinuz System.map config; do 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-17T03:39:42.994 INFO:teuthology.orchestra.run.smithi129.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout: fi 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:done 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:fi 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:fi 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-17T03:39:42.995 INFO:teuthology.orchestra.run.smithi129.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-17T03:39:42.996 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:42.996 INFO:teuthology.orchestra.run.smithi129.stdout:fi 2024-09-17T03:39:42.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-09-17T03:39:43.071 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-09-17T03:39:43.071 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-09-17T03:39:43.071 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-09-17T03:39:43.071 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 82.el9 2024-09-17T03:39:43.071 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8274670 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T03:39:43.072 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:39:43.073 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T03:39:43.074 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T03:39:43.074 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:43.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T03:39:43.769 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-09-17T03:39:45.185 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T03:39:45.219 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-09-17T03:39:45.230 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-17T03:39:45.288 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.duyqGTkdp7 2024-09-17T03:39:45.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.duyqGTkdp7 2024-09-17T03:39:45.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.duyqGTkdp7 2024-09-17T03:39:45.499 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.duyqGTkdp7 is 6KB 2024-09-17T03:39:45.547 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.duyqGTkdp7 2024-09-17T03:39:45.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T03:39:45.625 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T03:39:45.625 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T03:39:45.625 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T03:39:45.625 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-17T03:39:45.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:39:45.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-09-17T03:40:12.540 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-09-17T03:40:12.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T03:40:12.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:40:15.788 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-17T03:40:15.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:40:15.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:40:50.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-17T03:40:59.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:40:59.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:41:12.553 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T03:41:17.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.129 2024-09-17T03:41:21.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T03:41:21.557 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:41:24.940 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-17T03:41:25.569 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T03:41:25.570 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-17T03:41:25.570 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-17T03:41:25.604 INFO:teuthology.orchestra.run.smithi181.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:41:25.604 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:41:25.604 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T03:41:25.604 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T03:41:26.607 DEBUG:teuthology.parallel:result is None 2024-09-17T03:41:29.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:41:29.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:41:31.543 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-17T03:41:32.191 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:41:32.191 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-17T03:41:32.191 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-17T03:41:32.226 INFO:teuthology.orchestra.run.smithi129.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:41:32.227 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-17T03:41:32.227 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T03:41:32.227 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T03:41:33.228 DEBUG:teuthology.parallel:result is None 2024-09-17T03:41:33.228 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T03:41:33.236 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T03:41:33.237 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T03:41:33.241 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T03:41:33.279 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T03:41:33.289 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T03:41:33.296 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T03:41:33.296 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T03:41:33.318 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T03:41:33.376 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T03:41:33.383 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T03:41:33.383 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T03:41:33.387 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T03:41:33.513 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T03:41:33.551 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T03:41:33.554 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T03:41:33.628 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T03:41:33.631 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T03:41:33.638 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T03:41:33.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T03:41:33.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T03:41:33.702 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T03:41:33.710 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T03:41:33.711 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T03:41:33.714 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T03:41:33.751 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T03:41:33.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T03:41:33.924 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T03:41:33.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T03:41:34.057 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:41:34.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T03:41:34.128 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T03:41:34.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T03:41:34.274 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T03:41:34.337 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T03:41:34.407 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:41:34.407 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T03:41:34.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-17T03:41:34.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-17T03:41:34.562 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T03:41:34.597 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T03:41:34.906 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T03:41:34.916 INFO:teuthology.task.internal:Starting timer... 2024-09-17T03:41:34.916 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T03:41:34.933 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T03:41:34.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-09-17T03:41:35.035 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-09-17T03:41:35.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-17T03:41:35.129 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-17T03:41:35.132 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T03:41:35.132 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-09-17T03:41:35.168 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-09-17T03:41:35.169 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-17T03:41:35.217 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-17T03:41:35.217 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi129.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-17T03:41:35.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T03:41:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:41:35.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T03:41:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:41:35.336 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T03:41:35.336 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-09-17T03:41:35.399 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-17T03:41:35.465 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T03:41:35.475 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T03:41:35.487 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T03:41:35.488 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi129.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T03:46:14.879 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi129.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-17T03:46:14.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:46:14.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:46:14.973 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-17T03:46:15.086 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-17T03:46:15.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T03:46:15.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T03:46:15.171 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-17T03:46:15.280 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-17T03:46:15.280 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T03:46:15.289 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T03:46:15.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T03:46:15.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T03:46:15.294 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T03:46:15.294 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T03:46:15.350 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T03:46:15.380 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T03:46:15.381 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T03:46:15.411 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T03:46:15.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-09-17T03:46:15.455 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-09-17T03:46:15.461 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-17T03:46:15.485 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-17T03:46:15.485 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T03:46:15.514 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T03:46:15.514 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T03:46:15.543 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T03:46:15.604 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.611 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.638 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.646 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T03:46:15.647 INFO:teuthology.run_tasks:Running task install... 2024-09-17T03:46:15.655 DEBUG:teuthology.task.install:project ceph 2024-09-17T03:46:15.655 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}} 2024-09-17T03:46:15.655 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'} 2024-09-17T03:46:15.655 INFO:teuthology.task.install:Using flavor: default 2024-09-17T03:46:15.661 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T03:46:15.661 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-09-17T03:46:15.661 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'tag': None, 'wait_for_package': False} 2024-09-17T03:46:15.662 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T03:46:15.664 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'tag': None, 'wait_for_package': False} 2024-09-17T03:46:15.664 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T03:46:15.809 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/ 2024-09-17T03:46:15.809 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4989.gce1b342b 2024-09-17T03:46:15.815 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/ 2024-09-17T03:46:15.815 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4989.gce1b342b 2024-09-17T03:46:15.925 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T03:46:15.925 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:46:15.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T03:46:15.930 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240916.200549-debug/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T03:46:15.930 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:46:15.930 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T03:46:15.981 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-17T03:46:15.982 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T03:46:15.987 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-17T03:46:15.987 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ce1b342bf8c6349a8c90cd607f83bff2443f21cf/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T03:46:16.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T03:46:16.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T03:46:16.157 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T03:46:16.162 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T03:46:16.212 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-09-17T03:46:16.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-09-17T03:46:16.217 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-17T03:46:16.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-17T03:46:16.491 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-17T03:46:16.492 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-09-17T03:46:16.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-17T03:46:16.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-17T03:46:17.209 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 292 kB/s | 84 kB 00:00 2024-09-17T03:46:17.220 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 282 kB/s | 84 kB 00:00 2024-09-17T03:46:17.518 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 91 kB/s | 24 kB 00:00 2024-09-17T03:46:17.528 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-17T03:46:17.810 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-17T03:46:17.812 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-17T03:46:19.162 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.3 MB 00:01 2024-09-17T03:46:19.221 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.3 MB 00:01 2024-09-17T03:46:21.417 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-17T03:46:26.193 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.5 MB 00:00 2024-09-17T03:46:28.704 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 20 MB 00:08 2024-09-17T03:46:28.742 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 19 kB 00:01 2024-09-17T03:46:29.963 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-17T03:46:34.297 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-17T03:46:35.831 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 104 kB/s | 19 kB 00:00 2024-09-17T03:46:36.236 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-17T03:46:37.401 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-17T03:46:38.623 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T03:46:38.624 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T03:46:38.636 INFO:teuthology.orchestra.run.smithi129.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-17T03:46:38.637 INFO:teuthology.orchestra.run.smithi129.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-17T03:46:38.638 INFO:teuthology.orchestra.run.smithi129.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-17T03:46:38.639 INFO:teuthology.orchestra.run.smithi129.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-17T03:46:38.640 INFO:teuthology.orchestra.run.smithi129.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-17T03:46:38.643 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-17T03:46:38.644 INFO:teuthology.orchestra.run.smithi129.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-17T03:46:38.711 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 6.6 k 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.5 M 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 873 k 2024-09-17T03:46:38.727 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 148 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 1.5 M 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 154 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 14 M 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 7.4 M 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 49 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 10 M 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 40 M 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 285 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 784 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 225 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 27 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 18 k 2024-09-17T03:46:38.728 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 32 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 766 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 127 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-17T03:46:38.729 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 162 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 321 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 305 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 99 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 86 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.1 M 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 170 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-17T03:46:38.730 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.4 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.2 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 20 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 34 k 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 2.3 M 2024-09-17T03:46:38.731 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 291 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.0 M 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 18 M 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 16 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 25 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T03:46:38.732 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 159 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 507 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.3 M 2024-09-17T03:46:38.733 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-17T03:46:38.734 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 45 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 143 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T03:46:38.735 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T03:46:38.736 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T03:46:38.737 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T03:46:38.738 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T03:46:38.739 INFO:teuthology.orchestra.run.smithi129.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout:Install 149 Packages 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 8 Packages 2024-09-17T03:46:38.740 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:46:38.741 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 228 M 2024-09-17T03:46:38.741 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-17T03:46:39.596 INFO:teuthology.orchestra.run.smithi129.stdout:(1/157): ceph-19.3.0-4989.gce1b342b.el9.x86_64. 51 kB/s | 6.6 kB 00:00 2024-09-17T03:46:39.771 INFO:teuthology.orchestra.run.smithi129.stdout:(2/157): ceph-fuse-19.3.0-4989.gce1b342b.el9.x8 4.9 MB/s | 873 kB 00:00 2024-09-17T03:46:39.838 INFO:teuthology.orchestra.run.smithi129.stdout:(3/157): ceph-immutable-object-cache-19.3.0-498 2.2 MB/s | 148 kB 00:00 2024-09-17T03:46:40.013 INFO:teuthology.orchestra.run.smithi129.stdout:(4/157): ceph-base-19.3.0-4989.gce1b342b.el9.x8 10 MB/s | 5.5 MB 00:00 2024-09-17T03:46:40.080 INFO:teuthology.orchestra.run.smithi129.stdout:(5/157): ceph-mds-19.3.0-4989.gce1b342b.el9.x86 9.4 MB/s | 2.3 MB 00:00 2024-09-17T03:46:40.139 INFO:teuthology.orchestra.run.smithi129.stdout:(6/157): ceph-mgr-19.3.0-4989.gce1b342b.el9.x86 12 MB/s | 1.5 MB 00:00 2024-09-17T03:46:40.497 INFO:teuthology.orchestra.run.smithi129.stdout:(7/157): ceph-mon-19.3.0-4989.gce1b342b.el9.x86 12 MB/s | 5.0 MB 00:00 2024-09-17T03:46:40.895 INFO:teuthology.orchestra.run.smithi129.stdout:(8/157): ceph-common-19.3.0-4989.gce1b342b.el9. 14 MB/s | 20 MB 00:01 2024-09-17T03:46:40.948 INFO:teuthology.orchestra.run.smithi129.stdout:(9/157): ceph-selinux-19.3.0-4989.gce1b342b.el9 479 kB/s | 25 kB 00:00 2024-09-17T03:46:41.424 INFO:teuthology.orchestra.run.smithi129.stdout:(10/157): ceph-osd-19.3.0-4989.gce1b342b.el9.x8 14 MB/s | 18 MB 00:01 2024-09-17T03:46:41.549 INFO:teuthology.orchestra.run.smithi129.stdout:(11/157): ceph-radosgw-19.3.0-4989.gce1b342b.el 9.9 MB/s | 10 MB 00:01 2024-09-17T03:46:41.574 INFO:teuthology.orchestra.run.smithi129.stdout:(12/157): cephfs-mirror-19.3.0-4989.gce1b342b.e 1.5 MB/s | 225 kB 00:00 2024-09-17T03:46:41.599 INFO:teuthology.orchestra.run.smithi129.stdout:(13/157): cephfs-shell-19.3.0-4989.gce1b342b.el 538 kB/s | 27 kB 00:00 2024-09-17T03:46:41.641 INFO:teuthology.orchestra.run.smithi129.stdout:(14/157): libcephfs-devel-19.3.0-4989.gce1b342b 477 kB/s | 32 kB 00:00 2024-09-17T03:46:41.692 INFO:teuthology.orchestra.run.smithi129.stdout:(15/157): libcephsqlite-19.3.0-4989.gce1b342b.e 3.1 MB/s | 159 kB 00:00 2024-09-17T03:46:41.734 INFO:teuthology.orchestra.run.smithi129.stdout:(16/157): libcephfs2-19.3.0-4989.gce1b342b.el9. 5.6 MB/s | 766 kB 00:00 2024-09-17T03:46:41.767 INFO:teuthology.orchestra.run.smithi129.stdout:(17/157): librados-devel-19.3.0-4989.gce1b342b. 1.7 MB/s | 127 kB 00:00 2024-09-17T03:46:41.809 INFO:teuthology.orchestra.run.smithi129.stdout:(18/157): libradosstriper1-19.3.0-4989.gce1b342 6.6 MB/s | 507 kB 00:00 2024-09-17T03:46:41.877 INFO:teuthology.orchestra.run.smithi129.stdout:(19/157): python3-ceph-argparse-19.3.0-4989.gce 675 kB/s | 45 kB 00:00 2024-09-17T03:46:41.953 INFO:teuthology.orchestra.run.smithi129.stdout:(20/157): python3-ceph-common-19.3.0-4989.gce1b 1.9 MB/s | 143 kB 00:00 2024-09-17T03:46:42.036 INFO:teuthology.orchestra.run.smithi129.stdout:(21/157): python3-cephfs-19.3.0-4989.gce1b342b. 1.9 MB/s | 162 kB 00:00 2024-09-17T03:46:42.119 INFO:teuthology.orchestra.run.smithi129.stdout:(22/157): librgw2-19.3.0-4989.gce1b342b.el9.x86 15 MB/s | 5.3 MB 00:00 2024-09-17T03:46:42.153 INFO:teuthology.orchestra.run.smithi129.stdout:(23/157): python3-rados-19.3.0-4989.gce1b342b.e 2.7 MB/s | 321 kB 00:00 2024-09-17T03:46:42.178 INFO:teuthology.orchestra.run.smithi129.stdout:(24/157): python3-rbd-19.3.0-4989.gce1b342b.el9 5.1 MB/s | 305 kB 00:00 2024-09-17T03:46:42.203 INFO:teuthology.orchestra.run.smithi129.stdout:(25/157): python3-rgw-19.3.0-4989.gce1b342b.el9 1.9 MB/s | 99 kB 00:00 2024-09-17T03:46:42.229 INFO:teuthology.orchestra.run.smithi129.stdout:(26/157): rbd-fuse-19.3.0-4989.gce1b342b.el9.x8 1.7 MB/s | 86 kB 00:00 2024-09-17T03:46:42.287 INFO:teuthology.orchestra.run.smithi129.stdout:(27/157): rbd-nbd-19.3.0-4989.gce1b342b.el9.x86 2.9 MB/s | 170 kB 00:00 2024-09-17T03:46:42.355 INFO:teuthology.orchestra.run.smithi129.stdout:(28/157): ceph-grafana-dashboards-19.3.0-4989.g 513 kB/s | 34 kB 00:00 2024-09-17T03:46:42.429 INFO:teuthology.orchestra.run.smithi129.stdout:(29/157): rbd-mirror-19.3.0-4989.gce1b342b.el9. 14 MB/s | 3.1 MB 00:00 2024-09-17T03:46:42.472 INFO:teuthology.orchestra.run.smithi129.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4989.gce1b342 1.3 MB/s | 154 kB 00:00 2024-09-17T03:46:43.014 INFO:teuthology.orchestra.run.smithi129.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 14 MB/s | 7.4 MB 00:00 2024-09-17T03:46:43.097 INFO:teuthology.orchestra.run.smithi129.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4989.gce 3.4 MB/s | 291 kB 00:00 2024-09-17T03:46:43.514 INFO:teuthology.orchestra.run.smithi129.stdout:(33/157): ceph-test-19.3.0-4989.gce1b342b.el9.x 16 MB/s | 40 MB 00:02 2024-09-17T03:46:43.540 INFO:teuthology.orchestra.run.smithi129.stdout:(34/157): ceph-mgr-rook-19.3.0-4989.gce1b342b.e 112 kB/s | 49 kB 00:00 2024-09-17T03:46:43.565 INFO:teuthology.orchestra.run.smithi129.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4989.gc 331 kB/s | 16 kB 00:00 2024-09-17T03:46:43.590 INFO:teuthology.orchestra.run.smithi129.stdout:(36/157): ceph-volume-19.3.0-4989.gce1b342b.el9 5.6 MB/s | 285 kB 00:00 2024-09-17T03:46:43.640 INFO:teuthology.orchestra.run.smithi129.stdout:(37/157): cephfs-top-19.3.0-4989.gce1b342b.el9. 368 kB/s | 18 kB 00:00 2024-09-17T03:46:43.683 INFO:teuthology.orchestra.run.smithi129.stdout:(38/157): cephadm-19.3.0-4989.gce1b342b.el9.noa 6.5 MB/s | 784 kB 00:00 2024-09-17T03:46:43.691 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T03:46:43.841 INFO:teuthology.orchestra.run.smithi129.stdout:(39/157): ceph-mgr-dashboard-19.3.0-4989.gce1b3 10 MB/s | 14 MB 00:01 2024-09-17T03:46:43.875 INFO:teuthology.orchestra.run.smithi129.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 342 kB/s | 80 kB 00:00 2024-09-17T03:46:44.025 INFO:teuthology.orchestra.run.smithi129.stdout:(41/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-17T03:46:44.075 INFO:teuthology.orchestra.run.smithi129.stdout:(42/157): libquadmath-11.5.0-2.el9.x86_64.rpm 818 kB/s | 191 kB 00:00 2024-09-17T03:46:44.126 INFO:teuthology.orchestra.run.smithi129.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 1.8 MB/s | 801 kB 00:00 2024-09-17T03:46:44.151 INFO:teuthology.orchestra.run.smithi129.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 106 kB 00:00 2024-09-17T03:46:44.185 INFO:teuthology.orchestra.run.smithi129.stdout:(45/157): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-17T03:46:44.210 INFO:teuthology.orchestra.run.smithi129.stdout:(46/157): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-17T03:46:44.244 INFO:teuthology.orchestra.run.smithi129.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-09-17T03:46:44.353 INFO:teuthology.orchestra.run.smithi129.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 3.7 MB/s | 1.2 MB 00:00 2024-09-17T03:46:45.061 INFO:teuthology.orchestra.run.smithi129.stdout:(49/157): boost-program-options-1.75.0-8.el9.x8 151 kB/s | 107 kB 00:00 2024-09-17T03:46:45.103 INFO:teuthology.orchestra.run.smithi129.stdout:(50/157): automake-1.16.2-8.el9.noarch.rpm 772 kB/s | 689 kB 00:00 2024-09-17T03:46:45.178 INFO:teuthology.orchestra.run.smithi129.stdout:(51/157): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 943 kB 00:00 2024-09-17T03:46:45.595 INFO:teuthology.orchestra.run.smithi129.stdout:(52/157): elfutils-libelf-devel-0.191-4.el9.x86 41 kB/s | 22 kB 00:00 2024-09-17T03:46:45.745 INFO:teuthology.orchestra.run.smithi129.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 58 kB/s | 33 kB 00:00 2024-09-17T03:46:45.871 INFO:teuthology.orchestra.run.smithi129.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 408 kB/s | 313 kB 00:00 2024-09-17T03:46:46.045 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T03:46:46.045 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T03:46:46.057 INFO:teuthology.orchestra.run.smithi181.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-17T03:46:46.058 INFO:teuthology.orchestra.run.smithi181.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-17T03:46:46.059 INFO:teuthology.orchestra.run.smithi181.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-17T03:46:46.060 INFO:teuthology.orchestra.run.smithi181.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-17T03:46:46.061 INFO:teuthology.orchestra.run.smithi181.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-17T03:46:46.064 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-17T03:46:46.065 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-17T03:46:46.130 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-17T03:46:46.155 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 6.6 k 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.5 M 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 873 k 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 148 k 2024-09-17T03:46:46.156 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 1.5 M 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 154 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 14 M 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 7.4 M 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 49 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 10 M 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 40 M 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 285 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 784 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 225 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 27 k 2024-09-17T03:46:46.157 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 18 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 32 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 766 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 127 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-17T03:46:46.158 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 162 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 321 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 305 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 99 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 86 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.1 M 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 170 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager x86_64 1:1.51.0-1.el9 baseos 2.3 M 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm x86_64 1:1.51.0-1.el9 baseos 1.8 M 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team x86_64 1:1.51.0-1.el9 baseos 41 k 2024-09-17T03:46:46.159 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui x86_64 1:1.51.0-1.el9 baseos 249 k 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.4 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 3.2 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 20 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 34 k 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 2.3 M 2024-09-17T03:46:46.160 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 291 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.0 M 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 18 M 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4989.gce1b342b.el9 ceph-noarch 16 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 25 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 159 k 2024-09-17T03:46:46.161 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 507 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 5.3 M 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-17T03:46:46.162 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-09-17T03:46:46.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 45 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-4989.gce1b342b.el9 ceph 143 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T03:46:46.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T03:46:46.165 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T03:46:46.166 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T03:46:46.167 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T03:46:46.168 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-17T03:46:46.169 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout:Install 149 Packages 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 8 Packages 2024-09-17T03:46:46.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:46:46.171 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 228 M 2024-09-17T03:46:46.171 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-17T03:46:46.246 INFO:teuthology.orchestra.run.smithi129.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 36 kB/s | 18 kB 00:00 2024-09-17T03:46:46.355 INFO:teuthology.orchestra.run.smithi129.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 23 kB/s | 11 kB 00:00 2024-09-17T03:46:46.664 INFO:teuthology.orchestra.run.smithi129.stdout:(57/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.8 MB/s | 3.0 MB 00:01 2024-09-17T03:46:46.709 INFO:teuthology.orchestra.run.smithi181.stdout:(1/157): ceph-19.3.0-4989.gce1b342b.el9.x86_64. 32 kB/s | 6.6 kB 00:00 2024-09-17T03:46:47.039 INFO:teuthology.orchestra.run.smithi129.stdout:(58/157): libnbd-1.20.2-2.el9.x86_64.rpm 215 kB/s | 171 kB 00:00 2024-09-17T03:46:47.072 INFO:teuthology.orchestra.run.smithi129.stdout:(59/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 223 kB/s | 160 kB 00:00 2024-09-17T03:46:47.175 INFO:teuthology.orchestra.run.smithi181.stdout:(2/157): ceph-fuse-19.3.0-4989.gce1b342b.el9.x8 1.8 MB/s | 873 kB 00:00 2024-09-17T03:46:47.242 INFO:teuthology.orchestra.run.smithi181.stdout:(3/157): ceph-immutable-object-cache-19.3.0-498 2.2 MB/s | 148 kB 00:00 2024-09-17T03:46:47.306 INFO:teuthology.orchestra.run.smithi129.stdout:(60/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 71 kB/s | 45 kB 00:00 2024-09-17T03:46:47.701 INFO:teuthology.orchestra.run.smithi181.stdout:(4/157): ceph-mds-19.3.0-4989.gce1b342b.el9.x86 5.0 MB/s | 2.3 MB 00:00 2024-09-17T03:46:47.860 INFO:teuthology.orchestra.run.smithi181.stdout:(5/157): ceph-base-19.3.0-4989.gce1b342b.el9.x8 4.1 MB/s | 5.5 MB 00:01 2024-09-17T03:46:47.915 INFO:teuthology.orchestra.run.smithi129.stdout:(61/157): librdkafka-1.6.1-102.el9.x86_64.rpm 757 kB/s | 662 kB 00:00 2024-09-17T03:46:47.939 INFO:teuthology.orchestra.run.smithi129.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 39 kB/s | 24 kB 00:00 2024-09-17T03:46:48.032 INFO:teuthology.orchestra.run.smithi129.stdout:(63/157): libtool-2.4.6-46.el9.x86_64.rpm 606 kB/s | 581 kB 00:00 2024-09-17T03:46:48.043 INFO:teuthology.orchestra.run.smithi181.stdout:(6/157): ceph-mgr-19.3.0-4989.gce1b342b.el9.x86 4.4 MB/s | 1.5 MB 00:00 2024-09-17T03:46:48.499 INFO:teuthology.orchestra.run.smithi129.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 81 kB/s | 47 kB 00:00 2024-09-17T03:46:48.594 INFO:teuthology.orchestra.run.smithi181.stdout:(7/157): ceph-mon-19.3.0-4989.gce1b342b.el9.x86 6.8 MB/s | 5.0 MB 00:00 2024-09-17T03:46:48.749 INFO:teuthology.orchestra.run.smithi129.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 263 kB/s | 188 kB 00:00 2024-09-17T03:46:48.791 INFO:teuthology.orchestra.run.smithi129.stdout:(66/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 344 kB/s | 292 kB 00:00 2024-09-17T03:46:49.074 INFO:teuthology.orchestra.run.smithi129.stdout:(67/157): openblas-0.3.26-2.el9.x86_64.rpm 67 kB/s | 38 kB 00:00 2024-09-17T03:46:49.377 INFO:teuthology.orchestra.run.smithi181.stdout:(8/157): ceph-common-19.3.0-4989.gce1b342b.el9. 6.9 MB/s | 20 MB 00:02 2024-09-17T03:46:49.427 INFO:teuthology.orchestra.run.smithi181.stdout:(9/157): ceph-selinux-19.3.0-4989.gce1b342b.el9 505 kB/s | 25 kB 00:00 2024-09-17T03:46:49.697 INFO:teuthology.orchestra.run.smithi181.stdout:(10/157): ceph-radosgw-19.3.0-4989.gce1b342b.el 9.4 MB/s | 10 MB 00:01 2024-09-17T03:46:49.866 INFO:teuthology.orchestra.run.smithi129.stdout:(68/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-09-17T03:46:49.919 INFO:teuthology.orchestra.run.smithi181.stdout:(11/157): ceph-osd-19.3.0-4989.gce1b342b.el9.x8 9.5 MB/s | 18 MB 00:01 2024-09-17T03:46:49.944 INFO:teuthology.orchestra.run.smithi181.stdout:(12/157): cephfs-mirror-19.3.0-4989.gce1b342b.e 910 kB/s | 225 kB 00:00 2024-09-17T03:46:49.950 INFO:teuthology.orchestra.run.smithi129.stdout:(69/157): openblas-openmp-0.3.26-2.el9.x86_64.r 4.1 MB/s | 4.9 MB 00:01 2024-09-17T03:46:49.970 INFO:teuthology.orchestra.run.smithi181.stdout:(13/157): cephfs-shell-19.3.0-4989.gce1b342b.el 542 kB/s | 27 kB 00:00 2024-09-17T03:46:49.995 INFO:teuthology.orchestra.run.smithi181.stdout:(14/157): libcephfs-devel-19.3.0-4989.gce1b342b 630 kB/s | 32 kB 00:00 2024-09-17T03:46:50.046 INFO:teuthology.orchestra.run.smithi181.stdout:(15/157): libcephsqlite-19.3.0-4989.gce1b342b.e 3.1 MB/s | 159 kB 00:00 2024-09-17T03:46:50.075 INFO:teuthology.orchestra.run.smithi129.stdout:(70/157): postgresql-13.16-1.el9.x86_64.rpm 1.6 MB/s | 1.6 MB 00:00 2024-09-17T03:46:50.079 INFO:teuthology.orchestra.run.smithi181.stdout:(16/157): libcephfs2-19.3.0-4989.gce1b342b.el9. 6.9 MB/s | 766 kB 00:00 2024-09-17T03:46:50.105 INFO:teuthology.orchestra.run.smithi181.stdout:(17/157): librados-devel-19.3.0-4989.gce1b342b. 2.1 MB/s | 127 kB 00:00 2024-09-17T03:46:50.138 INFO:teuthology.orchestra.run.smithi181.stdout:(18/157): libradosstriper1-19.3.0-4989.gce1b342 8.5 MB/s | 507 kB 00:00 2024-09-17T03:46:50.222 INFO:teuthology.orchestra.run.smithi181.stdout:(19/157): python3-ceph-argparse-19.3.0-4989.gce 541 kB/s | 45 kB 00:00 2024-09-17T03:46:50.264 INFO:teuthology.orchestra.run.smithi181.stdout:(20/157): python3-ceph-common-19.3.0-4989.gce1b 3.4 MB/s | 143 kB 00:00 2024-09-17T03:46:50.323 INFO:teuthology.orchestra.run.smithi181.stdout:(21/157): python3-cephfs-19.3.0-4989.gce1b342b. 2.7 MB/s | 162 kB 00:00 2024-09-17T03:46:50.389 INFO:teuthology.orchestra.run.smithi181.stdout:(22/157): python3-rados-19.3.0-4989.gce1b342b.e 4.7 MB/s | 321 kB 00:00 2024-09-17T03:46:50.456 INFO:teuthology.orchestra.run.smithi181.stdout:(23/157): python3-rbd-19.3.0-4989.gce1b342b.el9 4.5 MB/s | 305 kB 00:00 2024-09-17T03:46:50.532 INFO:teuthology.orchestra.run.smithi181.stdout:(24/157): librgw2-19.3.0-4989.gce1b342b.el9.x86 12 MB/s | 5.3 MB 00:00 2024-09-17T03:46:50.556 INFO:teuthology.orchestra.run.smithi181.stdout:(25/157): python3-rgw-19.3.0-4989.gce1b342b.el9 994 kB/s | 99 kB 00:00 2024-09-17T03:46:50.582 INFO:teuthology.orchestra.run.smithi181.stdout:(26/157): rbd-fuse-19.3.0-4989.gce1b342b.el9.x8 1.7 MB/s | 86 kB 00:00 2024-09-17T03:46:50.632 INFO:teuthology.orchestra.run.smithi181.stdout:(27/157): rbd-nbd-19.3.0-4989.gce1b342b.el9.x86 3.3 MB/s | 170 kB 00:00 2024-09-17T03:46:50.642 INFO:teuthology.orchestra.run.smithi129.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 199 kB/s | 137 kB 00:00 2024-09-17T03:46:50.708 INFO:teuthology.orchestra.run.smithi181.stdout:(28/157): ceph-grafana-dashboards-19.3.0-4989.g 457 kB/s | 34 kB 00:00 2024-09-17T03:46:50.776 INFO:teuthology.orchestra.run.smithi129.stdout:(72/157): postgresql-contrib-13.16-1.el9.x86_64 1.0 MB/s | 910 kB 00:00 2024-09-17T03:46:50.791 INFO:teuthology.orchestra.run.smithi181.stdout:(29/157): rbd-mirror-19.3.0-4989.gce1b342b.el9. 13 MB/s | 3.1 MB 00:00 2024-09-17T03:46:50.817 INFO:teuthology.orchestra.run.smithi181.stdout:(30/157): ceph-mgr-cephadm-19.3.0-4989.gce1b342 1.4 MB/s | 154 kB 00:00 2024-09-17T03:46:51.243 INFO:teuthology.orchestra.run.smithi129.stdout:(73/157): postgresql-server-13.16-1.el9.x86_64. 4.9 MB/s | 5.8 MB 00:01 2024-09-17T03:46:51.333 INFO:teuthology.orchestra.run.smithi181.stdout:(31/157): ceph-mgr-diskprediction-local-19.3.0- 14 MB/s | 7.4 MB 00:00 2024-09-17T03:46:51.400 INFO:teuthology.orchestra.run.smithi181.stdout:(32/157): ceph-mgr-modules-core-19.3.0-4989.gce 4.3 MB/s | 291 kB 00:00 2024-09-17T03:46:51.435 INFO:teuthology.orchestra.run.smithi129.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 133 kB/s | 87 kB 00:00 2024-09-17T03:46:51.451 INFO:teuthology.orchestra.run.smithi181.stdout:(33/157): ceph-mgr-rook-19.3.0-4989.gce1b342b.e 989 kB/s | 49 kB 00:00 2024-09-17T03:46:51.677 INFO:teuthology.orchestra.run.smithi129.stdout:(75/157): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-09-17T03:46:51.851 INFO:teuthology.orchestra.run.smithi181.stdout:(34/157): ceph-test-19.3.0-4989.gce1b342b.el9.x 16 MB/s | 40 MB 00:02 2024-09-17T03:46:51.868 INFO:teuthology.orchestra.run.smithi181.stdout:(35/157): ceph-prometheus-alerts-19.3.0-4989.gc 39 kB/s | 16 kB 00:00 2024-09-17T03:46:51.902 INFO:teuthology.orchestra.run.smithi181.stdout:(36/157): ceph-volume-19.3.0-4989.gce1b342b.el9 5.6 MB/s | 285 kB 00:00 2024-09-17T03:46:52.069 INFO:teuthology.orchestra.run.smithi181.stdout:(37/157): ceph-mgr-dashboard-19.3.0-4989.gce1b3 11 MB/s | 14 MB 00:01 2024-09-17T03:46:52.094 INFO:teuthology.orchestra.run.smithi181.stdout:(38/157): cephfs-top-19.3.0-4989.gce1b342b.el9. 95 kB/s | 18 kB 00:00 2024-09-17T03:46:52.128 INFO:teuthology.orchestra.run.smithi181.stdout:(39/157): cephadm-19.3.0-4989.gce1b342b.el9.noa 3.0 MB/s | 784 kB 00:00 2024-09-17T03:46:52.161 INFO:teuthology.orchestra.run.smithi129.stdout:(76/157): python3-colorama-0.4.6-3.el9.noarch.r 84 kB/s | 61 kB 00:00 2024-09-17T03:46:52.211 INFO:teuthology.orchestra.run.smithi181.stdout:(40/157): fuse-2.9.9-16.el9.x86_64.rpm 564 kB/s | 80 kB 00:00 2024-09-17T03:46:52.318 INFO:teuthology.orchestra.run.smithi181.stdout:(41/157): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-17T03:46:52.345 INFO:teuthology.orchestra.run.smithi181.stdout:(42/157): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-17T03:46:52.379 INFO:teuthology.orchestra.run.smithi181.stdout:(43/157): libgfortran-11.5.0-2.el9.x86_64.rpm 2.8 MB/s | 801 kB 00:00 2024-09-17T03:46:52.404 INFO:teuthology.orchestra.run.smithi181.stdout:(44/157): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-17T03:46:52.453 INFO:teuthology.orchestra.run.smithi129.stdout:(77/157): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-09-17T03:46:52.456 INFO:teuthology.orchestra.run.smithi181.stdout:(45/157): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-17T03:46:52.479 INFO:teuthology.orchestra.run.smithi181.stdout:(46/157): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 135 kB 00:00 2024-09-17T03:46:52.503 INFO:teuthology.orchestra.run.smithi129.stdout:(78/157): python3-devel-3.9.19-8.el9.x86_64.rpm 300 kB/s | 245 kB 00:00 2024-09-17T03:46:52.547 INFO:teuthology.orchestra.run.smithi181.stdout:(47/157): python3-urllib3-1.26.5-6.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-09-17T03:46:52.647 INFO:teuthology.orchestra.run.smithi181.stdout:(48/157): python3-cryptography-36.0.1-4.el9.x86 3.7 MB/s | 1.2 MB 00:00 2024-09-17T03:46:52.730 INFO:teuthology.orchestra.run.smithi181.stdout:(49/157): automake-1.16.2-8.el9.noarch.rpm 2.7 MB/s | 689 kB 00:00 2024-09-17T03:46:52.764 INFO:teuthology.orchestra.run.smithi181.stdout:(50/157): elfutils-libelf-devel-0.191-4.el9.x86 657 kB/s | 22 kB 00:00 2024-09-17T03:46:52.789 INFO:teuthology.orchestra.run.smithi181.stdout:(51/157): boost-program-options-1.75.0-8.el9.x8 754 kB/s | 107 kB 00:00 2024-09-17T03:46:52.848 INFO:teuthology.orchestra.run.smithi181.stdout:(52/157): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-09-17T03:46:52.865 INFO:teuthology.orchestra.run.smithi181.stdout:(53/157): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-09-17T03:46:52.898 INFO:teuthology.orchestra.run.smithi181.stdout:(54/157): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-09-17T03:46:52.915 INFO:teuthology.orchestra.run.smithi181.stdout:(55/157): flexiblas-openblas-openmp-3.0.4-8.el9 363 kB/s | 18 kB 00:00 2024-09-17T03:46:52.932 INFO:teuthology.orchestra.run.smithi181.stdout:(56/157): libaio-devel-0.3.111-13.el9.x86_64.rp 335 kB/s | 11 kB 00:00 2024-09-17T03:46:52.999 INFO:teuthology.orchestra.run.smithi181.stdout:(57/157): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-09-17T03:46:53.003 INFO:teuthology.orchestra.run.smithi129.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 48 kB/s | 26 kB 00:00 2024-09-17T03:46:53.025 INFO:teuthology.orchestra.run.smithi181.stdout:(58/157): libnbd-1.20.2-2.el9.x86_64.rpm 1.5 MB/s | 171 kB 00:00 2024-09-17T03:46:53.028 INFO:teuthology.orchestra.run.smithi129.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 36 kB/s | 19 kB 00:00 2024-09-17T03:46:53.041 INFO:teuthology.orchestra.run.smithi181.stdout:(59/157): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T03:46:53.062 INFO:teuthology.orchestra.run.smithi129.stdout:(81/157): python3-jinja2-2.11.3-6.el9.noarch.rp 276 kB/s | 248 kB 00:00 2024-09-17T03:46:53.066 INFO:teuthology.orchestra.run.smithi181.stdout:(60/157): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-17T03:46:53.109 INFO:teuthology.orchestra.run.smithi181.stdout:(61/157): libtool-2.4.6-46.el9.x86_64.rpm 8.5 MB/s | 581 kB 00:00 2024-09-17T03:46:53.125 INFO:teuthology.orchestra.run.smithi181.stdout:(62/157): libuuid-devel-2.37.4-20.el9.x86_64.rp 418 kB/s | 24 kB 00:00 2024-09-17T03:46:53.159 INFO:teuthology.orchestra.run.smithi181.stdout:(63/157): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-17T03:46:53.235 INFO:teuthology.orchestra.run.smithi181.stdout:(64/157): libzstd-devel-1.5.1-2.el9.x86_64.rpm 378 kB/s | 47 kB 00:00 2024-09-17T03:46:53.260 INFO:teuthology.orchestra.run.smithi181.stdout:(65/157): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-09-17T03:46:53.293 INFO:teuthology.orchestra.run.smithi181.stdout:(66/157): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-17T03:46:53.344 INFO:teuthology.orchestra.run.smithi181.stdout:(67/157): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-17T03:46:53.427 INFO:teuthology.orchestra.run.smithi181.stdout:(68/157): postgresql-13.16-1.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-09-17T03:46:53.503 INFO:teuthology.orchestra.run.smithi181.stdout:(69/157): openssl-devel-3.2.2-6.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-09-17T03:46:53.569 INFO:teuthology.orchestra.run.smithi181.stdout:(70/157): openblas-openmp-0.3.26-2.el9.x86_64.r 16 MB/s | 4.9 MB 00:00 2024-09-17T03:46:53.587 INFO:teuthology.orchestra.run.smithi129.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 62 kB/s | 35 kB 00:00 2024-09-17T03:46:53.594 INFO:teuthology.orchestra.run.smithi181.stdout:(71/157): postgresql-private-libs-13.16-1.el9.x 1.5 MB/s | 137 kB 00:00 2024-09-17T03:46:53.637 INFO:teuthology.orchestra.run.smithi181.stdout:(72/157): postgresql-contrib-13.16-1.el9.x86_64 4.3 MB/s | 910 kB 00:00 2024-09-17T03:46:53.704 INFO:teuthology.orchestra.run.smithi129.stdout:(83/157): python3-mako-1.1.4-6.el9.noarch.rpm 246 kB/s | 172 kB 00:00 2024-09-17T03:46:53.747 INFO:teuthology.orchestra.run.smithi181.stdout:(73/157): protobuf-3.14.0-13.el9.x86_64.rpm 6.6 MB/s | 1.0 MB 00:00 2024-09-17T03:46:53.779 INFO:teuthology.orchestra.run.smithi181.stdout:(74/157): python3-attrs-20.3.0-7.el9.noarch.rpm 617 kB/s | 87 kB 00:00 2024-09-17T03:46:53.854 INFO:teuthology.orchestra.run.smithi181.stdout:(75/157): python3-colorama-0.4.6-3.el9.noarch.r 820 kB/s | 61 kB 00:00 2024-09-17T03:46:53.946 INFO:teuthology.orchestra.run.smithi181.stdout:(76/157): postgresql-server-13.16-1.el9.x86_64. 15 MB/s | 5.8 MB 00:00 2024-09-17T03:46:53.971 INFO:teuthology.orchestra.run.smithi181.stdout:(77/157): python3-devel-3.9.19-8.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-09-17T03:46:54.005 INFO:teuthology.orchestra.run.smithi181.stdout:(78/157): python3-jinja2-2.11.3-6.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-09-17T03:46:54.030 INFO:teuthology.orchestra.run.smithi181.stdout:(79/157): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-09-17T03:46:54.056 INFO:teuthology.orchestra.run.smithi181.stdout:(80/157): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-09-17T03:46:54.073 INFO:teuthology.orchestra.run.smithi181.stdout:(81/157): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-17T03:46:54.098 INFO:teuthology.orchestra.run.smithi181.stdout:(82/157): python3-markupsafe-1.1.1-12.el9.x86_6 836 kB/s | 35 kB 00:00 2024-09-17T03:46:54.172 INFO:teuthology.orchestra.run.smithi129.stdout:(84/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 5.1 MB 00:01 2024-09-17T03:46:54.173 INFO:teuthology.orchestra.run.smithi181.stdout:(83/157): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-17T03:46:54.198 INFO:teuthology.orchestra.run.smithi181.stdout:(84/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-09-17T03:46:54.223 INFO:teuthology.orchestra.run.smithi181.stdout:(85/157): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-09-17T03:46:54.249 INFO:teuthology.orchestra.run.smithi181.stdout:(86/157): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-17T03:46:54.266 INFO:teuthology.orchestra.run.smithi181.stdout:(87/157): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-09-17T03:46:54.330 INFO:teuthology.orchestra.run.smithi181.stdout:(88/157): python3-protobuf-3.14.0-13.el9.noarch 3.2 MB/s | 269 kB 00:00 2024-09-17T03:46:54.358 INFO:teuthology.orchestra.run.smithi181.stdout:(89/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-09-17T03:46:54.383 INFO:teuthology.orchestra.run.smithi181.stdout:(90/157): python3-pyasn1-modules-0.4.8-6.el9.no 5.2 MB/s | 279 kB 00:00 2024-09-17T03:46:54.400 INFO:teuthology.orchestra.run.smithi181.stdout:(91/157): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-17T03:46:54.417 INFO:teuthology.orchestra.run.smithi181.stdout:(92/157): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-09-17T03:46:54.430 INFO:teuthology.orchestra.run.smithi129.stdout:(85/157): python3-numpy-f2py-1.20.1-5.el9.x86_6 298 kB/s | 250 kB 00:00 2024-09-17T03:46:54.509 INFO:teuthology.orchestra.run.smithi181.stdout:(93/157): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-17T03:46:54.526 INFO:teuthology.orchestra.run.smithi181.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-09-17T03:46:54.551 INFO:teuthology.orchestra.run.smithi181.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-09-17T03:46:54.577 INFO:teuthology.orchestra.run.smithi181.stdout:(96/157): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-09-17T03:46:54.580 INFO:teuthology.orchestra.run.smithi129.stdout:(86/157): python3-oauthlib-3.1.1-5.el9.noarch.r 254 kB/s | 222 kB 00:00 2024-09-17T03:46:54.593 INFO:teuthology.orchestra.run.smithi181.stdout:(97/157): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-09-17T03:46:54.611 INFO:teuthology.orchestra.run.smithi181.stdout:(98/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-09-17T03:46:54.652 INFO:teuthology.orchestra.run.smithi181.stdout:(99/157): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-09-17T03:46:54.678 INFO:teuthology.orchestra.run.smithi181.stdout:(100/157): socat-1.7.4.1-6.el9.x86_64.rpm 3.5 MB/s | 304 kB 00:00 2024-09-17T03:46:54.703 INFO:teuthology.orchestra.run.smithi181.stdout:(101/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-09-17T03:46:54.720 INFO:teuthology.orchestra.run.smithi181.stdout:(102/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-17T03:46:54.737 INFO:teuthology.orchestra.run.smithi181.stdout:(103/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-09-17T03:46:54.787 INFO:teuthology.orchestra.run.smithi181.stdout:(104/157): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-09-17T03:46:54.797 INFO:teuthology.orchestra.run.smithi129.stdout:(87/157): python3-packaging-20.9-5.el9.noarch.r 123 kB/s | 77 kB 00:00 2024-09-17T03:46:54.871 INFO:teuthology.orchestra.run.smithi181.stdout:(105/157): abseil-cpp-20211102.0-3.el9.x86_64.r 6.4 MB/s | 548 kB 00:00 2024-09-17T03:46:54.904 INFO:teuthology.orchestra.run.smithi181.stdout:(106/157): gperftools-libs-2.9.1-3.el9.x86_64.r 9.1 MB/s | 308 kB 00:00 2024-09-17T03:46:55.023 INFO:teuthology.orchestra.run.smithi129.stdout:(88/157): python3-prettytable-0.7.2-27.el9.noar 71 kB/s | 42 kB 00:00 2024-09-17T03:46:55.072 INFO:teuthology.orchestra.run.smithi181.stdout:(107/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-09-17T03:46:55.088 INFO:teuthology.orchestra.run.smithi181.stdout:(108/157): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-09-17T03:46:55.138 INFO:teuthology.orchestra.run.smithi181.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-09-17T03:46:55.155 INFO:teuthology.orchestra.run.smithi181.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 458 kB/s | 26 kB 00:00 2024-09-17T03:46:55.197 INFO:teuthology.orchestra.run.smithi181.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.1 MB/s | 67 kB 00:00 2024-09-17T03:46:55.223 INFO:teuthology.orchestra.run.smithi181.stdout:(112/157): liboath-2.6.7-2.el9.x86_64.rpm 593 kB/s | 49 kB 00:00 2024-09-17T03:46:55.248 INFO:teuthology.orchestra.run.smithi181.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 3.1 MB/s | 151 kB 00:00 2024-09-17T03:46:55.348 INFO:teuthology.orchestra.run.smithi181.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-09-17T03:46:55.364 INFO:teuthology.orchestra.run.smithi129.stdout:(89/157): python3-protobuf-3.14.0-13.el9.noarch 344 kB/s | 269 kB 00:00 2024-09-17T03:46:55.374 INFO:teuthology.orchestra.run.smithi181.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-17T03:46:55.407 INFO:teuthology.orchestra.run.smithi181.stdout:(116/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.5 MB/s | 839 kB 00:00 2024-09-17T03:46:55.432 INFO:teuthology.orchestra.run.smithi181.stdout:(117/157): python3-cachetools-4.2.4-1.el9.noarc 553 kB/s | 32 kB 00:00 2024-09-17T03:46:55.458 INFO:teuthology.orchestra.run.smithi181.stdout:(118/157): python3-certifi-2023.05.07-4.el9.noa 280 kB/s | 14 kB 00:00 2024-09-17T03:46:55.483 INFO:teuthology.orchestra.run.smithi181.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-09-17T03:46:55.525 INFO:teuthology.orchestra.run.smithi181.stdout:(120/157): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-09-17T03:46:55.531 INFO:teuthology.orchestra.run.smithi129.stdout:(90/157): python3-pyasn1-0.4.8-6.el9.noarch.rpm 216 kB/s | 159 kB 00:00 2024-09-17T03:46:55.559 INFO:teuthology.orchestra.run.smithi181.stdout:(121/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-09-17T03:46:55.584 INFO:teuthology.orchestra.run.smithi181.stdout:(122/157): python3-google-auth-2.34.0-1.el9.noa 3.7 MB/s | 220 kB 00:00 2024-09-17T03:46:55.617 INFO:teuthology.orchestra.run.smithi181.stdout:(123/157): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-09-17T03:46:55.642 INFO:teuthology.orchestra.run.smithi181.stdout:(124/157): python3-jaraco-8.2.1-3.el9.noarch.rp 430 kB/s | 11 kB 00:00 2024-09-17T03:46:55.668 INFO:teuthology.orchestra.run.smithi181.stdout:(125/157): python3-jaraco-classes-3.2.1-5.el9.n 707 kB/s | 18 kB 00:00 2024-09-17T03:46:55.693 INFO:teuthology.orchestra.run.smithi181.stdout:(126/157): python3-jaraco-collections-3.0.0-8.e 935 kB/s | 23 kB 00:00 2024-09-17T03:46:55.760 INFO:teuthology.orchestra.run.smithi181.stdout:(127/157): libarrow-9.0.0-11.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-09-17T03:46:55.777 INFO:teuthology.orchestra.run.smithi181.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 233 kB/s | 19 kB 00:00 2024-09-17T03:46:55.822 INFO:teuthology.orchestra.run.smithi181.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 320 kB/s | 20 kB 00:00 2024-09-17T03:46:55.844 INFO:teuthology.orchestra.run.smithi181.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 2.1 MB/s | 46 kB 00:00 2024-09-17T03:46:55.869 INFO:teuthology.orchestra.run.smithi181.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-17T03:46:55.896 INFO:teuthology.orchestra.run.smithi181.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.2 MB/s | 58 kB 00:00 2024-09-17T03:46:55.898 INFO:teuthology.orchestra.run.smithi129.stdout:(91/157): python3-pyasn1-modules-0.4.8-6.el9.no 323 kB/s | 279 kB 00:00 2024-09-17T03:46:55.929 INFO:teuthology.orchestra.run.smithi181.stdout:(133/157): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-17T03:46:55.955 INFO:teuthology.orchestra.run.smithi181.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 658 kB/s | 16 kB 00:00 2024-09-17T03:46:55.990 INFO:teuthology.orchestra.run.smithi129.stdout:(92/157): python3-pytz-2021.1-5.el9.noarch.rpm 82 kB/s | 51 kB 00:00 2024-09-17T03:46:55.997 INFO:teuthology.orchestra.run.smithi181.stdout:(135/157): python3-kubernetes-26.1.0-2.el9.noar 4.6 MB/s | 1.0 MB 00:00 2024-09-17T03:46:56.022 INFO:teuthology.orchestra.run.smithi181.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-09-17T03:46:56.048 INFO:teuthology.orchestra.run.smithi181.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-09-17T03:46:56.073 INFO:teuthology.orchestra.run.smithi181.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-09-17T03:46:56.098 INFO:teuthology.orchestra.run.smithi181.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T03:46:56.132 INFO:teuthology.orchestra.run.smithi129.stdout:(93/157): python3-requests-oauthlib-1.3.0-12.el 89 kB/s | 54 kB 00:00 2024-09-17T03:46:56.140 INFO:teuthology.orchestra.run.smithi181.stdout:(140/157): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-09-17T03:46:56.165 INFO:teuthology.orchestra.run.smithi181.stdout:(141/157): python3-tempora-5.0.0-2.el9.noarch.r 536 kB/s | 36 kB 00:00 2024-09-17T03:46:56.191 INFO:teuthology.orchestra.run.smithi181.stdout:(142/157): python3-routes-2.5.1-5.el9.noarch.rp 1.3 MB/s | 188 kB 00:00 2024-09-17T03:46:56.216 INFO:teuthology.orchestra.run.smithi181.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 988 kB/s | 74 kB 00:00 2024-09-17T03:46:56.233 INFO:teuthology.orchestra.run.smithi181.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-09-17T03:46:56.258 INFO:teuthology.orchestra.run.smithi181.stdout:(145/157): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-17T03:46:56.283 INFO:teuthology.orchestra.run.smithi181.stdout:(146/157): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-09-17T03:46:56.334 INFO:teuthology.orchestra.run.smithi181.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-17T03:46:56.359 INFO:teuthology.orchestra.run.smithi181.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-09-17T03:46:56.409 INFO:teuthology.orchestra.run.smithi181.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-17T03:46:56.601 INFO:teuthology.orchestra.run.smithi181.stdout:(150/157): librados2-19.3.0-4989.gce1b342b.el9. 13 MB/s | 3.4 MB 00:00 2024-09-17T03:46:56.616 INFO:teuthology.orchestra.run.smithi129.stdout:(94/157): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-09-17T03:46:56.759 INFO:teuthology.orchestra.run.smithi181.stdout:(151/157): librbd1-19.3.0-4989.gce1b342b.el9.x8 7.9 MB/s | 3.2 MB 00:00 2024-09-17T03:46:56.803 INFO:teuthology.orchestra.run.smithi129.stdout:(95/157): python3-wcwidth-0.2.5-8.el9.noarch.rp 66 kB/s | 44 kB 00:00 2024-09-17T03:46:56.877 INFO:teuthology.orchestra.run.smithi181.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 351 kB/s | 41 kB 00:00 2024-09-17T03:46:56.918 INFO:teuthology.orchestra.run.smithi181.stdout:(153/157): NetworkManager-1.51.0-1.el9.x86_64.r 4.6 MB/s | 2.3 MB 00:00 2024-09-17T03:46:56.969 INFO:teuthology.orchestra.run.smithi181.stdout:(154/157): NetworkManager-tui-1.51.0-1.el9.x86_ 2.6 MB/s | 249 kB 00:00 2024-09-17T03:46:57.111 INFO:teuthology.orchestra.run.smithi181.stdout:(155/157): NetworkManager-libnm-1.51.0-1.el9.x8 3.6 MB/s | 1.8 MB 00:00 2024-09-17T03:46:57.340 INFO:teuthology.orchestra.run.smithi181.stdout:(156/157): openssl-3.2.2-6.el9.x86_64.rpm 3.2 MB/s | 1.4 MB 00:00 2024-09-17T03:46:57.367 INFO:teuthology.orchestra.run.smithi129.stdout:(96/157): qatlib-service-24.02.0-1.el9.x86_64.r 64 kB/s | 36 kB 00:00 2024-09-17T03:46:57.411 INFO:teuthology.orchestra.run.smithi181.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 5.4 MB/s | 2.4 MB 00:00 2024-09-17T03:46:57.416 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-17T03:46:57.417 INFO:teuthology.orchestra.run.smithi181.stdout:Total 20 MB/s | 228 MB 00:11 2024-09-17T03:46:57.542 INFO:teuthology.orchestra.run.smithi129.stdout:(97/157): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.9 MB/s | 16 MB 00:01 2024-09-17T03:46:57.567 INFO:teuthology.orchestra.run.smithi129.stdout:(98/157): qatlib-24.02.0-1.el9.x86_64.rpm 232 kB/s | 221 kB 00:00 2024-09-17T03:46:58.068 INFO:teuthology.orchestra.run.smithi129.stdout:(99/157): qatzip-libs-1.2.0-1.el9.x86_64.rpm 68 kB/s | 47 kB 00:00 2024-09-17T03:46:58.193 INFO:teuthology.orchestra.run.smithi129.stdout:(100/157): uuid-1.6.2-55.el9.x86_64.rpm 92 kB/s | 58 kB 00:00 2024-09-17T03:46:58.335 INFO:teuthology.orchestra.run.smithi129.stdout:(101/157): socat-1.7.4.1-6.el9.x86_64.rpm 384 kB/s | 304 kB 00:00 2024-09-17T03:46:58.743 INFO:teuthology.orchestra.run.smithi129.stdout:(102/157): xfsprogs-devel-6.4.0-4.el9.x86_64.rp 142 kB/s | 96 kB 00:00 2024-09-17T03:46:58.835 INFO:teuthology.orchestra.run.smithi129.stdout:(103/157): lua-devel-5.4.4-4.el9.x86_64.rpm 246 kB/s | 22 kB 00:00 2024-09-17T03:46:58.886 INFO:teuthology.orchestra.run.smithi129.stdout:(104/157): xmlstarlet-1.6.1-20.el9.x86_64.rpm 92 kB/s | 64 kB 00:00 2024-09-17T03:46:58.911 INFO:teuthology.orchestra.run.smithi129.stdout:(105/157): zlib-devel-1.2.11-41.el9.x86_64.rpm 78 kB/s | 45 kB 00:00 2024-09-17T03:46:58.969 INFO:teuthology.orchestra.run.smithi129.stdout:(106/157): abseil-cpp-20211102.0-3.el9.x86_64.r 6.5 MB/s | 548 kB 00:00 2024-09-17T03:46:58.995 INFO:teuthology.orchestra.run.smithi129.stdout:(107/157): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-09-17T03:46:59.029 INFO:teuthology.orchestra.run.smithi129.stdout:(108/157): gperftools-libs-2.9.1-3.el9.x86_64.r 2.6 MB/s | 308 kB 00:00 2024-09-17T03:46:59.062 INFO:teuthology.orchestra.run.smithi129.stdout:(109/157): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-09-17T03:46:59.079 INFO:teuthology.orchestra.run.smithi129.stdout:(110/157): libarrow-doc-9.0.0-11.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-09-17T03:46:59.121 INFO:teuthology.orchestra.run.smithi129.stdout:(111/157): libunwind-1.6.2-1.el9.x86_64.rpm 2.2 MB/s | 67 kB 00:00 2024-09-17T03:46:59.146 INFO:teuthology.orchestra.run.smithi129.stdout:(112/157): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-09-17T03:46:59.172 INFO:teuthology.orchestra.run.smithi129.stdout:(113/157): luarocks-3.9.2-5.el9.noarch.rpm 3.1 MB/s | 151 kB 00:00 2024-09-17T03:46:59.230 INFO:teuthology.orchestra.run.smithi129.stdout:(114/157): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-09-17T03:46:59.264 INFO:teuthology.orchestra.run.smithi129.stdout:(115/157): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-09-17T03:46:59.311 INFO:teuthology.orchestra.run.smithi129.stdout:(116/157): python3-cachetools-4.2.4-1.el9.noarc 690 kB/s | 32 kB 00:00 2024-09-17T03:46:59.336 INFO:teuthology.orchestra.run.smithi129.stdout:(117/157): python3-certifi-2023.05.07-4.el9.noa 566 kB/s | 14 kB 00:00 2024-09-17T03:46:59.378 INFO:teuthology.orchestra.run.smithi129.stdout:(118/157): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-09-17T03:46:59.403 INFO:teuthology.orchestra.run.smithi129.stdout:(119/157): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-09-17T03:46:59.477 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-17T03:46:59.479 INFO:teuthology.orchestra.run.smithi129.stdout:(120/157): libarrow-9.0.0-11.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-09-17T03:46:59.504 INFO:teuthology.orchestra.run.smithi129.stdout:(121/157): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-09-17T03:46:59.529 INFO:teuthology.orchestra.run.smithi129.stdout:(122/157): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-09-17T03:46:59.555 INFO:teuthology.orchestra.run.smithi129.stdout:(123/157): python3-google-auth-2.34.0-1.el9.noa 2.9 MB/s | 220 kB 00:00 2024-09-17T03:46:59.580 INFO:teuthology.orchestra.run.smithi129.stdout:(124/157): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-09-17T03:46:59.605 INFO:teuthology.orchestra.run.smithi129.stdout:(125/157): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-09-17T03:46:59.630 INFO:teuthology.orchestra.run.smithi129.stdout:(126/157): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-09-17T03:46:59.656 INFO:teuthology.orchestra.run.smithi129.stdout:(127/157): python3-jaraco-collections-3.0.0-8.e 463 kB/s | 23 kB 00:00 2024-09-17T03:46:59.662 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-17T03:46:59.662 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-17T03:46:59.681 INFO:teuthology.orchestra.run.smithi129.stdout:(128/157): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-17T03:46:59.707 INFO:teuthology.orchestra.run.smithi129.stdout:(129/157): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-17T03:46:59.740 INFO:teuthology.orchestra.run.smithi129.stdout:(130/157): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-09-17T03:46:59.774 INFO:teuthology.orchestra.run.smithi129.stdout:(131/157): python3-more-itertools-8.12.0-2.el9. 2.3 MB/s | 79 kB 00:00 2024-09-17T03:46:59.800 INFO:teuthology.orchestra.run.smithi129.stdout:(132/157): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-17T03:46:59.841 INFO:teuthology.orchestra.run.smithi129.stdout:(133/157): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-09-17T03:46:59.866 INFO:teuthology.orchestra.run.smithi129.stdout:(134/157): python3-portend-3.1.0-2.el9.noarch.r 659 kB/s | 16 kB 00:00 2024-09-17T03:46:59.900 INFO:teuthology.orchestra.run.smithi129.stdout:(135/157): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-09-17T03:46:59.925 INFO:teuthology.orchestra.run.smithi129.stdout:(136/157): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-17T03:46:59.951 INFO:teuthology.orchestra.run.smithi129.stdout:(137/157): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-09-17T03:46:59.976 INFO:teuthology.orchestra.run.smithi129.stdout:(138/157): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-17T03:47:00.004 INFO:teuthology.orchestra.run.smithi129.stdout:(139/157): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T03:47:00.027 INFO:teuthology.orchestra.run.smithi129.stdout:(140/157): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-09-17T03:47:00.052 INFO:teuthology.orchestra.run.smithi129.stdout:(141/157): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-09-17T03:47:00.094 INFO:teuthology.orchestra.run.smithi129.stdout:(142/157): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-09-17T03:47:00.136 INFO:teuthology.orchestra.run.smithi129.stdout:(143/157): python3-typing-extensions-4.12.2-2.e 684 kB/s | 74 kB 00:00 2024-09-17T03:47:00.161 INFO:teuthology.orchestra.run.smithi129.stdout:(144/157): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-09-17T03:47:00.187 INFO:teuthology.orchestra.run.smithi129.stdout:(145/157): python3-zc-lockfile-2.0-10.el9.noarc 812 kB/s | 20 kB 00:00 2024-09-17T03:47:00.212 INFO:teuthology.orchestra.run.smithi129.stdout:(146/157): python3-webob-1.8.8-2.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-09-17T03:47:00.246 INFO:teuthology.orchestra.run.smithi129.stdout:(147/157): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-17T03:47:00.279 INFO:teuthology.orchestra.run.smithi129.stdout:(148/157): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-09-17T03:47:00.363 INFO:teuthology.orchestra.run.smithi129.stdout:(149/157): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-17T03:47:00.505 INFO:teuthology.orchestra.run.smithi129.stdout:(150/157): librados2-19.3.0-4989.gce1b342b.el9. 13 MB/s | 3.4 MB 00:00 2024-09-17T03:47:00.572 INFO:teuthology.orchestra.run.smithi129.stdout:(151/157): librbd1-19.3.0-4989.gce1b342b.el9.x8 11 MB/s | 3.2 MB 00:00 2024-09-17T03:47:00.730 INFO:teuthology.orchestra.run.smithi129.stdout:(152/157): NetworkManager-team-1.51.0-1.el9.x86 259 kB/s | 41 kB 00:00 2024-09-17T03:47:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:(153/157): NetworkManager-tui-1.51.0-1.el9.x86_ 1.6 MB/s | 249 kB 00:00 2024-09-17T03:47:01.240 INFO:teuthology.orchestra.run.smithi129.stdout:(154/157): openssl-3.2.2-6.el9.x86_64.rpm 3.8 MB/s | 1.4 MB 00:00 2024-09-17T03:47:01.281 INFO:teuthology.orchestra.run.smithi129.stdout:(155/157): NetworkManager-libnm-1.51.0-1.el9.x8 2.3 MB/s | 1.8 MB 00:00 2024-09-17T03:47:01.357 INFO:teuthology.orchestra.run.smithi129.stdout:(156/157): NetworkManager-1.51.0-1.el9.x86_64.r 2.3 MB/s | 2.3 MB 00:00 2024-09-17T03:47:01.989 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-17T03:47:01.989 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-17T03:47:02.007 INFO:teuthology.orchestra.run.smithi129.stdout:(157/157): openssl-libs-3.2.2-6.el9.x86_64.rpm 3.1 MB/s | 2.4 MB 00:00 2024-09-17T03:47:02.014 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-17T03:47:02.014 INFO:teuthology.orchestra.run.smithi129.stdout:Total 9.8 MB/s | 228 MB 00:23 2024-09-17T03:47:04.222 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-17T03:47:04.404 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-17T03:47:04.405 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-17T03:47:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-17T03:47:06.730 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-17T03:47:08.876 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-17T03:47:09.050 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-17T03:47:09.107 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-17T03:47:09.174 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-17T03:47:09.218 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-17T03:47:09.250 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-17T03:47:09.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-17T03:47:09.686 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-17T03:47:09.820 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:47:09.966 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:47:10.010 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:47:10.106 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:47:10.149 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T03:47:10.189 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-17T03:47:10.403 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-17T03:47:10.456 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-17T03:47:10.482 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:47:10.610 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:47:10.644 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T03:47:10.715 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T03:47:10.734 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T03:47:10.776 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T03:47:11.002 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T03:47:11.200 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T03:47:11.391 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-17T03:47:11.503 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:11.626 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:11.891 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:12.023 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-17T03:47:12.066 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-17T03:47:12.117 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-17T03:47:12.199 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-17T03:47:12.253 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-17T03:47:12.292 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-17T03:47:12.345 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-17T03:47:12.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-17T03:47:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-17T03:47:12.516 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-17T03:47:12.579 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-17T03:47:12.613 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-17T03:47:12.662 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-17T03:47:12.751 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-17T03:47:12.784 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 34/165 2024-09-17T03:47:12.828 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 35/165 2024-09-17T03:47:12.856 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 36/165 2024-09-17T03:47:12.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-17T03:47:13.000 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-17T03:47:13.100 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-17T03:47:13.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-17T03:47:13.329 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-17T03:47:13.505 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-17T03:47:13.687 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/165 2024-09-17T03:47:13.754 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/165 2024-09-17T03:47:13.820 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/165 2024-09-17T03:47:13.864 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/165 2024-09-17T03:47:13.897 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/165 2024-09-17T03:47:13.952 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/165 2024-09-17T03:47:13.978 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-17T03:47:14.088 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-17T03:47:14.123 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-17T03:47:14.264 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-17T03:47:14.348 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/165 2024-09-17T03:47:14.488 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:47:14.532 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 46/165 2024-09-17T03:47:14.595 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-17T03:47:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:47:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:47:14.770 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:47:14.784 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-17T03:47:14.804 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T03:47:14.845 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/165 2024-09-17T03:47:14.859 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-17T03:47:14.971 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-17T03:47:15.056 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 12/165 2024-09-17T03:47:15.075 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-17T03:47:15.120 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-13.16-1.el9.x86_64 13/165 2024-09-17T03:47:15.154 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:47:15.164 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-17T03:47:15.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-17T03:47:15.238 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-17T03:47:15.283 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:47:15.305 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-17T03:47:15.322 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T03:47:15.354 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-17T03:47:15.400 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-17T03:47:15.404 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 15/165 2024-09-17T03:47:15.439 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T03:47:15.450 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-17T03:47:15.482 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/165 2024-09-17T03:47:15.623 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-17T03:47:15.679 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-17T03:47:15.721 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-17T03:47:15.722 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T03:47:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-17T03:47:15.887 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-17T03:47:15.905 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/165 2024-09-17T03:47:15.930 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-17T03:47:15.959 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-17T03:47:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-17T03:47:16.072 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 18/165 2024-09-17T03:47:16.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-17T03:47:16.188 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:16.307 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:16.311 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-17T03:47:16.350 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-17T03:47:16.387 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-17T03:47:16.421 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-17T03:47:16.470 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-17T03:47:16.509 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-17T03:47:16.571 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-17T03:47:16.585 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.51.0-1.el9.x86_64 19/165 2024-09-17T03:47:16.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-17T03:47:16.639 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-17T03:47:16.691 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-17T03:47:16.731 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-17T03:47:16.737 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : re2-1:20211101-20.el9.x86_64 20/165 2024-09-17T03:47:16.785 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/165 2024-09-17T03:47:16.832 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-17T03:47:16.832 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/165 2024-09-17T03:47:16.882 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-17T03:47:16.913 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/165 2024-09-17T03:47:16.918 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-17T03:47:16.993 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/165 2024-09-17T03:47:17.032 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 25/165 2024-09-17T03:47:17.041 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-17T03:47:17.086 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/165 2024-09-17T03:47:17.124 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-17T03:47:17.125 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/165 2024-09-17T03:47:17.159 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/165 2024-09-17T03:47:17.254 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/165 2024-09-17T03:47:17.335 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 30/165 2024-09-17T03:47:17.371 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/165 2024-09-17T03:47:17.417 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 32/165 2024-09-17T03:47:17.500 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 33/165 2024-09-17T03:47:17.533 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 34/165 2024-09-17T03:47:17.577 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 35/165 2024-09-17T03:47:17.605 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 36/165 2024-09-17T03:47:17.685 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/165 2024-09-17T03:47:17.758 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/165 2024-09-17T03:47:17.873 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/165 2024-09-17T03:47:18.013 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-17T03:47:18.039 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/165 2024-09-17T03:47:18.094 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-17T03:47:18.103 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/165 2024-09-17T03:47:18.127 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-17T03:47:18.243 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-17T03:47:18.310 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-17T03:47:18.362 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-17T03:47:18.467 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-17T03:47:18.512 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-17T03:47:18.726 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/165 2024-09-17T03:47:18.774 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-17T03:47:18.801 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-17T03:47:18.828 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/165 2024-09-17T03:47:18.881 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/165 2024-09-17T03:47:18.902 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-17T03:47:18.925 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/165 2024-09-17T03:47:18.964 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 46/165 2024-09-17T03:47:19.024 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/165 2024-09-17T03:47:19.197 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 48/165 2024-09-17T03:47:19.258 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 49/165 2024-09-17T03:47:19.362 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/165 2024-09-17T03:47:19.435 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/165 2024-09-17T03:47:19.530 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/165 2024-09-17T03:47:19.564 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 53/165 2024-09-17T03:47:19.600 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 54/165 2024-09-17T03:47:19.669 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/165 2024-09-17T03:47:19.708 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/165 2024-09-17T03:47:19.714 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-17T03:47:19.752 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/165 2024-09-17T03:47:19.803 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/165 2024-09-17T03:47:19.864 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-17T03:47:19.971 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/165 2024-09-17T03:47:20.029 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/165 2024-09-17T03:47:20.071 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 61/165 2024-09-17T03:47:20.195 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/165 2024-09-17T03:47:20.245 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 63/165 2024-09-17T03:47:20.306 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/165 2024-09-17T03:47:20.351 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 65/165 2024-09-17T03:47:20.376 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/165 2024-09-17T03:47:20.494 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/165 2024-09-17T03:47:20.651 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/165 2024-09-17T03:47:20.693 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/165 2024-09-17T03:47:20.730 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/165 2024-09-17T03:47:20.772 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/165 2024-09-17T03:47:20.821 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/165 2024-09-17T03:47:20.860 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/165 2024-09-17T03:47:20.897 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/165 2024-09-17T03:47:20.932 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/165 2024-09-17T03:47:20.965 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/165 2024-09-17T03:47:21.013 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/165 2024-09-17T03:47:21.055 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/165 2024-09-17T03:47:21.153 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/165 2024-09-17T03:47:21.212 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/165 2024-09-17T03:47:21.250 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/165 2024-09-17T03:47:21.369 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/165 2024-09-17T03:47:21.429 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/165 2024-09-17T03:47:21.456 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-17T03:47:21.566 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-17T03:47:21.619 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-17T03:47:21.652 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-17T03:47:22.004 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-17T03:47:22.046 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T03:47:22.175 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T03:47:22.210 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 103/165 2024-09-17T03:47:22.259 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-17T03:47:22.335 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/165 2024-09-17T03:47:22.382 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/165 2024-09-17T03:47:22.416 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/165 2024-09-17T03:47:22.527 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/165 2024-09-17T03:47:22.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 88/165 2024-09-17T03:47:22.635 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 89/165 2024-09-17T03:47:22.747 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/165 2024-09-17T03:47:22.804 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-17T03:47:22.812 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/165 2024-09-17T03:47:22.875 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T03:47:22.949 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T03:47:22.968 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 107/165 2024-09-17T03:47:23.042 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.26-2.el9.x86_64 92/165 2024-09-17T03:47:23.074 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 93/165 2024-09-17T03:47:23.179 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/165 2024-09-17T03:47:23.981 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/165 2024-09-17T03:47:24.111 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/165 2024-09-17T03:47:25.472 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:25.506 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:25.655 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/165 2024-09-17T03:47:25.772 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/165 2024-09-17T03:47:25.816 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 99/165 2024-09-17T03:47:25.849 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 100/165 2024-09-17T03:47:26.172 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:26.212 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 101/165 2024-09-17T03:47:26.234 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T03:47:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T03:47:26.372 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 102/165 2024-09-17T03:47:26.383 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T03:47:26.384 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T03:47:26.384 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T03:47:26.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:26.408 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 103/165 2024-09-17T03:47:26.418 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T03:47:26.456 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 104/165 2024-09-17T03:47:26.999 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 105/165 2024-09-17T03:47:27.039 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T03:47:27.113 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 106/165 2024-09-17T03:47:27.149 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 107/165 2024-09-17T03:47:29.613 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:30.316 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 108/165 2024-09-17T03:47:30.360 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T03:47:30.470 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 109/165 2024-09-17T03:47:30.471 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T03:47:30.471 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T03:47:30.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:30.511 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-17T03:47:39.024 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-17T03:47:39.025 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-17T03:47:39.025 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:39.239 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T03:47:39.275 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T03:47:39.276 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:39.276 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T03:47:39.276 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:40.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T03:47:41.042 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T03:47:41.042 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:41.043 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T03:47:41.043 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T03:47:41.043 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T03:47:41.043 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:41.293 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T03:47:41.343 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T03:47:41.343 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:41.344 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T03:47:41.344 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T03:47:41.344 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T03:47:41.344 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:41.860 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T03:47:41.910 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T03:47:41.910 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:41.910 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T03:47:41.911 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T03:47:41.911 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T03:47:41.911 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:42.044 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-17T03:47:42.104 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-17T03:47:42.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-17T03:47:42.429 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-17T03:47:42.489 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-17T03:47:42.576 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-17T03:47:42.685 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-17T03:47:42.861 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-17T03:47:42.905 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-17T03:47:42.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-17T03:47:42.970 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-17T03:47:43.382 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 110/165 2024-09-17T03:47:43.382 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-09-17T03:47:43.382 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-09-17T03:47:43.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:43.553 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T03:47:43.583 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T03:47:43.586 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T03:47:43.620 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 127/165 2024-09-17T03:47:43.626 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 111/165 2024-09-17T03:47:43.627 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:43.627 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T03:47:43.627 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:43.710 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 128/165 2024-09-17T03:47:43.751 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T03:47:45.372 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 112/165 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T03:47:45.430 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:45.680 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 113/165 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T03:47:45.730 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:46.289 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 114/165 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T03:47:46.340 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:46.472 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : automake-1.16.2-8.el9.noarch 115/165 2024-09-17T03:47:46.533 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 116/165 2024-09-17T03:47:46.597 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 117/165 2024-09-17T03:47:46.820 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 118/165 2024-09-17T03:47:46.876 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 119/165 2024-09-17T03:47:46.962 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 120/165 2024-09-17T03:47:47.071 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 121/165 2024-09-17T03:47:47.243 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 122/165 2024-09-17T03:47:47.304 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 123/165 2024-09-17T03:47:47.342 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 124/165 2024-09-17T03:47:47.376 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-16.el9.x86_64 125/165 2024-09-17T03:47:47.433 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T03:47:47.506 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T03:47:47.939 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T03:47:47.976 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 126/165 2024-09-17T03:47:48.018 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 127/165 2024-09-17T03:47:48.099 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 128/165 2024-09-17T03:47:48.132 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T03:47:48.593 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T03:47:48.623 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T03:47:48.773 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T03:47:48.967 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 132/165 2024-09-17T03:47:49.000 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T03:47:49.051 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T03:47:49.051 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:49.052 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T03:47:49.052 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T03:47:49.052 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T03:47:49.052 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:49.109 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T03:47:49.135 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T03:47:49.274 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 135/165 2024-09-17T03:47:49.336 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 136/165 2024-09-17T03:47:50.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-17T03:47:50.428 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T03:47:50.479 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T03:47:50.479 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:50.479 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T03:47:50.479 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T03:47:50.480 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T03:47:50.480 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:50.537 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T03:47:50.586 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T03:47:50.586 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:50.586 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T03:47:50.586 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:50.638 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T03:47:50.688 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:51.039 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T03:47:51.089 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:51.775 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 129/165 2024-09-17T03:47:51.839 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T03:47:52.929 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 130/165 2024-09-17T03:47:52.980 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T03:47:53.132 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 131/165 2024-09-17T03:47:53.320 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 132/165 2024-09-17T03:47:53.358 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T03:47:53.409 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 133/165 2024-09-17T03:47:53.409 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:53.409 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T03:47:53.409 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T03:47:53.409 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T03:47:53.410 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:53.467 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T03:47:53.493 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 134/165 2024-09-17T03:47:53.602 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 135/165 2024-09-17T03:47:53.680 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 136/165 2024-09-17T03:47:54.735 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libtool-2.4.6-46.el9.x86_64 137/165 2024-09-17T03:47:54.778 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T03:47:54.830 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 138/165 2024-09-17T03:47:54.830 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:54.830 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T03:47:54.830 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T03:47:54.830 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T03:47:54.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:54.896 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T03:47:54.945 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 139/165 2024-09-17T03:47:54.945 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:54.945 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T03:47:54.945 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:55.005 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T03:47:55.055 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 140/165 2024-09-17T03:47:55.055 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:55.055 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-17T03:47:55.056 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T03:47:55.056 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-17T03:47:55.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:55.427 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 142/165 2024-09-17T03:47:55.430 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T03:47:55.468 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 143/165 2024-09-17T03:47:55.482 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 141/165 2024-09-17T03:47:55.482 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T03:47:55.482 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T03:47:55.482 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T03:47:55.483 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T03:47:55.483 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:47:55.520 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 144/165 2024-09-17T03:47:55.549 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-17T03:47:55.584 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-17T03:47:55.849 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 147/165 2024-09-17T03:47:55.898 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-17T03:47:55.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 149/165 2024-09-17T03:47:55.958 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-17T03:47:55.994 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-17T03:47:56.706 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:47:56.750 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:47:56.791 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:47:58.069 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 153/165 2024-09-17T03:47:58.174 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-17T03:47:58.230 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-17T03:47:58.365 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-17T03:47:58.401 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-17T03:47:58.427 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-17T03:47:58.463 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-17T03:47:58.463 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T03:47:58.571 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T03:47:58.571 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T03:47:58.688 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T03:47:58.712 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-17T03:47:58.728 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:47:58.728 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:47:58.981 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:47:58.981 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T03:47:59.074 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T03:47:59.148 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:47:59.383 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:47:59.384 INFO:teuthology.orchestra.run.smithi181.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-17T03:47:59.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:47:59.861 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 142/165 2024-09-17T03:47:59.903 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 143/165 2024-09-17T03:47:59.946 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 144/165 2024-09-17T03:47:59.984 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 145/165 2024-09-17T03:48:00.019 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 146/165 2024-09-17T03:48:00.262 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 147/165 2024-09-17T03:48:00.332 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 148/165 2024-09-17T03:48:00.362 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 149/165 2024-09-17T03:48:00.402 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-team-1:1.51.0-1.el9.x86_64 150/165 2024-09-17T03:48:00.446 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : NetworkManager-tui-1:1.51.0-1.el9.x86_64 151/165 2024-09-17T03:48:01.123 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:48:01.170 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:48:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 152/165 2024-09-17T03:48:02.494 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 153/165 2024-09-17T03:48:02.612 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 154/165 2024-09-17T03:48:02.666 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 155/165 2024-09-17T03:48:02.765 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flex-2.6.4-9.el9.x86_64 156/165 2024-09-17T03:48:02.814 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : bison-3.7.4-5.el9.x86_64 157/165 2024-09-17T03:48:02.849 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 158/165 2024-09-17T03:48:02.893 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 159/165 2024-09-17T03:48:02.893 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T03:48:03.001 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/165 2024-09-17T03:48:03.001 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T03:48:03.094 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 161/165 2024-09-17T03:48:03.117 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 162/165 2024-09-17T03:48:03.133 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:48:03.133 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:48:03.370 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 163/165 2024-09-17T03:48:03.370 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T03:48:03.479 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 164/165 2024-09-17T03:48:03.563 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:48:03.773 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:48:03.773 INFO:teuthology.orchestra.run.smithi129.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-17T03:48:03.773 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 1/165 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2/165 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 3/165 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 4/165 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 5/165 2024-09-17T03:48:05.568 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 6/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 7/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 11/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 12/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 13/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 15/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 16/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 17/165 2024-09-17T03:48:05.569 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 18/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 19/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 20/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 21/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 22/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 23/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 24/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 25/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 26/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 27/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 28/165 2024-09-17T03:48:05.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 29/165 2024-09-17T03:48:05.583 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 30/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 31/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 32/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 33/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 34/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 35/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 36/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 37/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 38/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 39/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-17T03:48:05.584 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-17T03:48:05.585 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-17T03:48:05.586 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-17T03:48:05.587 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-17T03:48:05.588 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-17T03:48:05.589 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-17T03:48:05.590 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-17T03:48:05.591 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-17T03:48:05.592 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 150/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 152/165 2024-09-17T03:48:05.593 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-17T03:48:05.594 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.771 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: automake-1.16.2-8.el9.noarch 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: bison-3.7.4-5.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.772 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.773 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: flex-2.6.4-9.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T03:48:07.774 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T03:48:07.775 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T03:48:07.776 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-13.16-1.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T03:48:07.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T03:48:07.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T03:48:07.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T03:48:07.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T03:48:07.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T03:48:07.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-17T03:48:07.783 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T03:48:07.784 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-17T03:48:07.784 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:07.784 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-17T03:48:08.422 DEBUG:teuthology.parallel:result is None 2024-09-17T03:48:10.163 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 1/165 2024-09-17T03:48:10.163 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2/165 2024-09-17T03:48:10.163 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 3/165 2024-09-17T03:48:10.163 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 4/165 2024-09-17T03:48:10.163 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4989.gce1b3 5/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 6/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 7/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 8/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 9/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 10/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 11/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 12/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 13/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 14/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_ 15/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 16/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 17/165 2024-09-17T03:48:10.164 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_6 18/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86 19/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 20/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el 21/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:19.3.0-4989.gce1b342b.el9. 22/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_6 23/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 24/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 25/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 26/165 2024-09-17T03:48:10.165 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 27/165 2024-09-17T03:48:10.175 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 28/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 29/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b. 30/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noa 31/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.n 32/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1 33/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el 34/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 35/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.e 36/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 37/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 38/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 39/165 2024-09-17T03:48:10.176 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 41/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 42/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/165 2024-09-17T03:48:10.177 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 58/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 59/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 60/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 61/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 62/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 63/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 64/165 2024-09-17T03:48:10.178 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 65/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-5.4.4-4.el9.x86_64 66/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 67/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 68/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 69/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-13.16-1.el9.x86_64 70/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 71/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 72/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 73/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 74/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 75/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 76/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 77/165 2024-09-17T03:48:10.179 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 78/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 79/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/165 2024-09-17T03:48:10.180 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 100/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 102/165 2024-09-17T03:48:10.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 104/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 108/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 110/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 111/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 114/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 115/165 2024-09-17T03:48:10.182 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 119/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 123/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 124/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 125/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 126/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 127/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 128/165 2024-09-17T03:48:10.183 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 129/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 130/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 131/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 132/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 133/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 134/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 135/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 136/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 137/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 138/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 139/165 2024-09-17T03:48:10.184 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 140/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 141/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 142/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 143/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 144/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 145/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 146/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 147/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : re2-1:20211101-20.el9.x86_64 148/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 149/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 150/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 151/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 152/165 2024-09-17T03:48:10.185 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 153/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.51.0-1.el9.x86_64 154/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 155/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.51.0-1.el9.x86_64 156/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 157/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.51.0-1.el9.x86_64 158/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 159/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.51.0-1.el9.x86_64 160/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 161/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 162/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 163/165 2024-09-17T03:48:10.186 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 164/165 2024-09-17T03:48:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 165/165 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-libnm-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-team-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: NetworkManager-tui-1:1.51.0-1.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-17T03:48:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: automake-1.16.2-8.el9.noarch 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: bison-3.7.4-5.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-volume-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-shell-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: cephfs-top-2:19.3.0-4989.gce1b342b.el9.noarch 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: flex-2.6.4-9.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T03:48:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T03:48:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T03:48:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-13.16-1.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T03:48:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-09-17T03:48:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T03:48:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T03:48:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T03:48:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T03:48:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:19.3.0-4989.gce1b342b.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-17T03:48:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T03:48:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-17T03:48:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:12.361 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-17T03:48:12.935 DEBUG:teuthology.parallel:result is None 2024-09-17T03:48:12.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T03:48:13.087 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T03:48:13.133 INFO:teuthology.orchestra.run.smithi129.stdout:19.3.0-4989.gce1b342b.el9 2024-09-17T03:48:13.133 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989.gce1b342b.el9 2024-09-17T03:48:13.134 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989.gce1b342b is installed. 2024-09-17T03:48:13.135 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ce1b342bf8c6349a8c90cd607f83bff2443f21cf 2024-09-17T03:48:13.272 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T03:48:13.322 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-4989.gce1b342b.el9 2024-09-17T03:48:13.322 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4989.gce1b342b.el9 2024-09-17T03:48:13.322 INFO:teuthology.task.install:The correct ceph version 19.3.0-4989.gce1b342b is installed. 2024-09-17T03:48:13.324 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T03:48:13.325 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:13.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T03:48:13.388 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:13.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T03:48:13.448 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T03:48:13.449 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:13.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T03:48:13.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T03:48:13.605 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:13.606 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T03:48:13.663 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T03:48:13.759 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T03:48:13.759 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:13.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T03:48:13.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T03:48:13.917 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:13.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T03:48:13.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T03:48:14.069 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T03:48:14.069 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:14.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T03:48:14.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T03:48:14.229 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:14.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T03:48:14.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T03:48:14.381 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T03:48:14.459 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T03:48:14.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /var/log/ceph 2024-09-17T03:48:14.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-17T03:48:14.518 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T03:48:14.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T03:48:14.522 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T03:48:14.577 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T03:48:14.577 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T03:48:14.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T03:48:14.642 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-17T03:48:14.642 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-17T03:48:14.643 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-17T03:48:14.643 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:14.643 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T03:48:14.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:14.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:14.902 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:14.902 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-17T03:48:14.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:15.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:15.165 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-17T03:48:15.165 INFO:tasks.ceph:config {'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T03:48:15.165 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469', 'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/strays}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7908469', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240916.200549-debug'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240916.200549-debug', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 476, 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'sleep_before_teardown': 0, 'subset': '320/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240916.200549-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_ce1b342bf8c6349a8c90cd607f83bff2443f21cf/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi129.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN/X1ct/74uIAtWyMbN03U1IwGslFTOLGw+g5EBXz5zh8vg6DyF4lULj8OP83ADTk1SuvwwBB3niYyJM+EOYDjs=', 'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF+B8jrcq2WyWnoS+RzNvCJaeBL4uSGLOQDqINksiCQ8QBvsZgw75cygt+G8c8zbb+56qjaDOdHgkJ5LTxvIGeo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf'}}, {'ceph': {'create_rbd_pool': False, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'client': {'pid file': ''}, 'global': {'ms die on skipped message': False}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': 'ce1b342bf8c6349a8c90cd607f83bff2443f21cf', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_strays']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_02:09:51', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T03:48:15.166 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T03:48:15.170 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-17T03:48:15.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T03:48:15.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T03:48:15.301 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:15.301 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T03:48:15.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T03:48:15.336 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-17 03:48:04.144302993 +0000 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-17 03:45:51.010572607 +0000 2024-09-17T03:48:15.412 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-17 03:45:51.010572607 +0000 2024-09-17T03:48:15.413 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-17 03:45:51.010572607 +0000 2024-09-17T03:48:15.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T03:48:15.504 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-17T03:48:15.504 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-17T03:48:15.505 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000344477 s, 1.5 MB/s 2024-09-17T03:48:15.507 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T03:48:15.543 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-17T03:48:15.618 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T03:48:15.618 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-17 03:48:04.145302971 +0000 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-17 03:45:51.484562403 +0000 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-17 03:45:51.484562403 +0000 2024-09-17T03:48:15.619 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-17 03:45:51.484562403 +0000 2024-09-17T03:48:15.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T03:48:15.711 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-17T03:48:15.711 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-17T03:48:15.711 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000307702 s, 1.7 MB/s 2024-09-17T03:48:15.714 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T03:48:15.750 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-17T03:48:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T03:48:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:15.825 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 670 Links: 1 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-17 03:48:04.145302971 +0000 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-17 03:45:51.947552437 +0000 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-17 03:45:51.947552437 +0000 2024-09-17T03:48:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-17 03:45:51.947552437 +0000 2024-09-17T03:48:15.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T03:48:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-17T03:48:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-17T03:48:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000259237 s, 2.0 MB/s 2024-09-17T03:48:15.921 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T03:48:15.958 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 6h/6d Inode: 691 Links: 1 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-17 03:48:04.146302950 +0000 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-17 03:45:52.418542298 +0000 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-17 03:45:52.418542298 +0000 2024-09-17T03:48:16.033 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-17 03:45:52.418542298 +0000 2024-09-17T03:48:16.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T03:48:16.125 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-17T03:48:16.125 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-17T03:48:16.125 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000289839 s, 1.8 MB/s 2024-09-17T03:48:16.128 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T03:48:16.164 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:16.164 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:16.164 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T03:48:16.203 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T03:48:16.204 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 640 Links: 1 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 03:47:59.737950684 +0000 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 03:45:51.041192141 +0000 2024-09-17T03:48:16.282 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 03:45:51.041192141 +0000 2024-09-17T03:48:16.283 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 03:45:51.041192141 +0000 2024-09-17T03:48:16.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T03:48:16.378 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T03:48:16.379 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T03:48:16.379 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000294895 s, 1.7 MB/s 2024-09-17T03:48:16.381 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T03:48:16.419 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 656 Links: 1 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 03:47:59.738950664 +0000 2024-09-17T03:48:16.495 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 03:45:51.473183226 +0000 2024-09-17T03:48:16.496 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 03:45:51.473183226 +0000 2024-09-17T03:48:16.496 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 03:45:51.473183226 +0000 2024-09-17T03:48:16.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T03:48:16.588 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T03:48:16.589 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T03:48:16.589 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000250728 s, 2.0 MB/s 2024-09-17T03:48:16.591 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T03:48:16.627 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-17T03:48:16.702 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T03:48:16.702 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:16.702 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 675 Links: 1 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 03:47:59.739950644 +0000 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 03:45:51.944173506 +0000 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 03:45:51.944173506 +0000 2024-09-17T03:48:16.703 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 03:45:51.944173506 +0000 2024-09-17T03:48:16.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T03:48:16.795 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T03:48:16.796 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T03:48:16.796 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000235664 s, 2.2 MB/s 2024-09-17T03:48:16.798 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T03:48:16.834 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 694 Links: 1 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T03:48:16.909 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-17 03:47:59.740950625 +0000 2024-09-17T03:48:16.910 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-17 03:45:52.387164363 +0000 2024-09-17T03:48:16.910 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-17 03:45:52.387164363 +0000 2024-09-17T03:48:16.910 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-17 03:45:52.387164363 +0000 2024-09-17T03:48:16.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T03:48:17.001 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-17T03:48:17.001 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-17T03:48:17.001 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000264785 s, 1.9 MB/s 2024-09-17T03:48:17.004 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T03:48:17.040 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:17.040 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T03:48:17.040 INFO:tasks.ceph:Generating config... 2024-09-17T03:48:17.041 INFO:tasks.ceph:[client] pid file = 2024-09-17T03:48:17.041 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-17T03:48:17.041 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T03:48:17.042 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T03:48:17.042 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T03:48:17.042 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T03:48:17.042 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T03:48:17.042 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T03:48:17.042 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T03:48:17.043 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T03:48:17.043 INFO:tasks.ceph:Setting up mon.a... 2024-09-17T03:48:17.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-17T03:48:17.197 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.keyring 2024-09-17T03:48:17.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-17T03:48:17.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T03:48:17.420 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.129'), ('mon.b', '172.21.15.181'), ('mon.c', '[v2:172.21.15.181:3301,v1:172.21.15.181:6790]')] 2024-09-17T03:48:17.421 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'pid file': ''}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}} 2024-09-17T03:48:17.422 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:17.422 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T03:48:17.456 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.129 --add b 172.21.15.181 --addv c '[v2:172.21.15.181:3301,v1:172.21.15.181:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: generated fsid c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:setting min_mon_release = quincy 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:epoch 0 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:fsid c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:48:17.637 INFO:teuthology.orchestra.run.smithi129.stdout:last_changed 2024-09-17T03:48:17.638749+0000 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:created 2024-09-17T03:48:17.638749+0000 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:min_mon_release 17 (quincy) 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:election_strategy: 1 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:0: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.a 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:1: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.b 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:2: [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] mon.c 2024-09-17T03:48:17.638 INFO:teuthology.orchestra.run.smithi129.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-17T03:48:17.646 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T03:48:17.680 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c5b53c7d-83e3-4793-838f-5ade77ae49c7... 2024-09-17T03:48:17.681 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:48:17.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:48:17.817 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.818 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:48:17.819 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:48:17.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:48:17.821 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:48:17.822 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:48:17.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:48:17.824 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:48:17.825 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:48:17.826 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:48:17.827 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:48:17.828 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.829 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.831 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.832 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:48:17.833 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:48:17.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:48:17.835 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:48:17.836 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:48:17.837 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:48:17.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:48:17.839 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:48:17.840 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:48:17.853 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-17T03:48:17.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-17T03:48:18.017 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T03:48:18.017 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:18.017 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-17T03:48:18.052 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:18.052 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-17T03:48:18.126 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:48:18.126 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:18.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T03:48:18.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T03:48:18.245 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:18.245 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:18.320 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:48:18.320 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:18.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-17T03:48:18.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-17T03:48:18.399 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:18.399 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:18.475 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T03:48:18.476 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T03:48:18.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T03:48:18.643 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-17T03:48:18.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T03:48:18.826 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-17T03:48:18.839 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T03:48:18.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T03:48:19.005 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-17T03:48:19.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-17T03:48:19.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T03:48:19.236 INFO:teuthology.orchestra.run.smithi129.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-17T03:48:19.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-17T03:48:19.302 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T03:48:19.467 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-17T03:48:19.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-17T03:48:19.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T03:48:19.704 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-17T03:48:19.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-17T03:48:19.771 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T03:48:19.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T03:48:19.915 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-17T03:48:19.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-17T03:48:20.089 INFO:teuthology.orchestra.run.smithi129.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-17T03:48:20.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T03:48:20.267 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-17T03:48:20.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-17T03:48:20.443 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-17T03:48:20.478 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T03:48:20.478 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi129.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-17T03:48:20.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-17T03:48:20.533 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:20.534 INFO:tasks.ceph:role: osd.0 2024-09-17T03:48:20.534 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:48:20.534 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T03:48:20.595 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:20.596 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:21.001 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-17T03:48:21.006 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-17T03:48:21.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-17T03:48:21.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-17T03:48:21.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-17T03:48:21.272 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:21.273 INFO:tasks.ceph:role: osd.1 2024-09-17T03:48:21.273 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:48:21.273 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T03:48:21.335 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:21.335 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:21.335 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:21.336 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:21.690 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-17T03:48:21.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-17T03:48:21.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-17T03:48:21.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-17T03:48:21.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-17T03:48:21.894 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:21.894 INFO:tasks.ceph:role: osd.2 2024-09-17T03:48:21.894 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:48:21.894 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:21.957 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:22.345 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-17T03:48:22.347 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-17T03:48:22.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-17T03:48:22.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-17T03:48:22.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-17T03:48:22.573 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:22.573 INFO:tasks.ceph:role: osd.3 2024-09-17T03:48:22.573 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:48:22.574 DEBUG:teuthology.orchestra.run.smithi129:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T03:48:22.636 INFO:teuthology.orchestra.run.smithi129.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:22.637 INFO:teuthology.orchestra.run.smithi129.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:23.051 INFO:teuthology.orchestra.run.smithi129.stdout:Discarding blocks...Done. 2024-09-17T03:48:23.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi129.front.sepia.ceph.com -o noatime 2024-09-17T03:48:23.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-17T03:48:23.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-17T03:48:23.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:23.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:23.405+0000 7fd3a43cc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-17T03:48:23.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:23.409+0000 7fd3a43cc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-17T03:48:23.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:23.409+0000 7fd3a43cc640 -1 bdev(0x5645faced900 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:23.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:23.410+0000 7fd3a43cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-17T03:48:26.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T03:48:26.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:26.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:26.713+0000 7f257d80b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-17T03:48:26.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:26.713+0000 7f257d80b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-17T03:48:26.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:26.714+0000 7f257d80b640 -1 bdev(0x563d72c79900 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:26.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:26.714+0000 7f257d80b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-17T03:48:29.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T03:48:30.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:30.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:30.162+0000 7f60c7614640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-17T03:48:30.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:30.162+0000 7f60c7614640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-17T03:48:30.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:30.162+0000 7f60c7614640 -1 bdev(0x5646929a7900 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:30.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:30.163+0000 7f60c7614640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-17T03:48:33.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T03:48:33.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:33.622+0000 7fc99a7dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-17T03:48:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:33.622+0000 7fc99a7dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-17T03:48:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:33.622+0000 7fc99a7dd640 -1 bdev(0x556981f89900 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:48:33.622+0000 7fc99a7dd640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-17T03:48:36.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T03:48:36.958 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-17T03:48:37.020 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:37.020 INFO:tasks.ceph:role: osd.4 2024-09-17T03:48:37.020 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:48:37.020 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T03:48:37.085 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:37.086 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:37.500 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T03:48:37.504 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T03:48:37.505 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-17T03:48:37.653 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-17T03:48:37.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-17T03:48:37.775 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:37.775 INFO:tasks.ceph:role: osd.5 2024-09-17T03:48:37.775 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:48:37.775 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T03:48:37.839 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:37.839 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:37.840 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:38.193 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T03:48:38.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T03:48:38.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-17T03:48:38.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-17T03:48:38.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-17T03:48:38.408 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:38.408 INFO:tasks.ceph:role: osd.6 2024-09-17T03:48:38.408 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:48:38.408 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T03:48:38.471 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:38.471 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:38.471 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:38.471 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:38.472 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:38.851 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T03:48:38.855 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T03:48:38.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-17T03:48:38.950 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-17T03:48:39.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-17T03:48:39.067 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-17T03:48:39.067 INFO:tasks.ceph:role: osd.7 2024-09-17T03:48:39.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:48:39.067 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T03:48:39.130 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-17T03:48:39.131 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T03:48:39.131 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T03:48:39.131 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T03:48:39.131 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T03:48:39.550 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-17T03:48:39.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-17T03:48:39.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-17T03:48:39.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-17T03:48:39.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:39.909 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:39.909+0000 7f0f82dfd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-17T03:48:39.914 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:39.914+0000 7f0f82dfd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-17T03:48:39.914 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:39.914+0000 7f0f82dfd640 -1 bdev(0x55776d3e7900 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:39.914 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:39.916+0000 7f0f82dfd640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-17T03:48:43.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T03:48:43.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:43.410 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:43.412+0000 7efc3c213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-17T03:48:43.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:43.412+0000 7efc3c213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-17T03:48:43.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:43.412+0000 7efc3c213640 -1 bdev(0x55b2fd52f900 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:43.411 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:43.412+0000 7efc3c213640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-17T03:48:45.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:45.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:48:46.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T03:48:46.509 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:46.665 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:46.666+0000 7f9a71444640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-17T03:48:46.666 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:46.666+0000 7f9a71444640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-17T03:48:46.666 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:46.666+0000 7f9a71444640 -1 bdev(0x55f199a57900 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:46.666 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:46.667+0000 7f9a71444640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-17T03:48:49.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T03:48:49.747 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:49.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:49.901+0000 7f6aee811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-17T03:48:49.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:49.902+0000 7f6aee811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-17T03:48:49.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:49.902+0000 7f6aee811640 -1 bdev(0x561e5b5a1900 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-17T03:48:49.902 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:48:49.902+0000 7f6aee811640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-17T03:48:52.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T03:48:52.954 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T03:48:52.954 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:52.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-17T03:48:53.016 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-17T03:48:53.114 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-17T03:48:53.210 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-17T03:48:53.306 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-17T03:48:53.402 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-17T03:48:53.497 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:53.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-17T03:48:53.591 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:53.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-17T03:48:53.648 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:53.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-17T03:48:53.744 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:53.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-17T03:48:53.838 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:53.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-17T03:48:53.934 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:53.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-17T03:48:54.030 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:54.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-17T03:48:54.125 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:54.126 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-17T03:48:54.221 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:54.221 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-17T03:48:54.258 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:48:54.258 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-17T03:48:54.334 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:54.335 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-17T03:48:54.369 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:48:54.370 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-17T03:48:54.444 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T03:48:54.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T03:48:54.449 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.y] 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA== 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ== 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:48:54.498 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.0] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.1] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.2] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.3] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr.x] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw== 2024-09-17T03:48:54.499 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.4] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.5] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.6] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[osd.7] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout:[client.2] 2024-09-17T03:48:54.500 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA== 2024-09-17T03:48:54.501 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2024-09-17T03:48:54.501 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg== 2024-09-17T03:48:54.501 INFO:teuthology.orchestra.run.smithi129.stdout:[client.3] 2024-09-17T03:48:54.501 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ== 2024-09-17T03:48:54.536 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.y] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg== 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-17T03:48:54.537 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.4] 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.5] 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ== 2024-09-17T03:48:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.6] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw== 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.7] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg== 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout:[client.2] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA== 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout:[client.1] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg== 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout:[client.3] 2024-09-17T03:48:54.539 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ== 2024-09-17T03:48:54.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T03:48:54.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T03:48:54.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:54.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:54.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:54.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:55.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:55.749 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T03:48:55.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T03:48:55.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:55.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:56.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:56.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-17T03:48:56.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.585 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T03:48:56.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:56.917 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.085 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T03:48:57.587 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T03:48:57.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-17T03:48:57.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T03:48:57.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-17T03:48:57.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-17T03:48:57.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T03:48:58.115 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-17T03:48:58.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-17T03:48:58.226 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-17T03:48:58.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-17T03:48:58.456 DEBUG:teuthology.orchestra.run.smithi129:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:58.460 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-17T03:48:58.494 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-17T03:48:58.495 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-17T03:48:58.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-17T03:48:58.537 INFO:tasks.ceph.mon.a:Started 2024-09-17T03:48:58.537 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-17T03:48:58.537 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-17T03:48:58.541 INFO:tasks.ceph.mon.b:Started 2024-09-17T03:48:58.541 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-17T03:48:58.541 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-17T03:48:58.545 INFO:tasks.ceph.mon.c:Started 2024-09-17T03:48:58.545 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-17T03:48:58.545 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-17T03:48:58.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-17T03:48:58.549 INFO:tasks.ceph.mgr.y:Started 2024-09-17T03:48:58.549 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-17T03:48:58.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-17T03:48:58.553 INFO:tasks.ceph.mgr.x:Started 2024-09-17T03:48:58.553 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-17T03:48:58.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-17T03:48:58.557 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-17T03:48:58.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-17T03:48:58.603 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-17T03:48:58.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-17T03:48:58.649 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-17T03:48:58.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-17T03:48:58.697 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-17T03:48:58.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-17T03:48:58.744 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-17T03:48:58.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-17T03:48:58.792 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-17T03:48:58.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-17T03:48:58.840 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-17T03:48:58.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-17T03:48:58.889 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-17T03:48:58.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-17T03:48:58.937 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-17T03:48:58.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-17T03:48:58.984 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-17T03:48:58.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-17T03:48:59.032 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-17T03:48:59.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-17T03:48:59.079 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-17T03:48:59.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-17T03:48:59.128 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-17T03:48:59.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-17T03:48:59.176 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-17T03:48:59.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-17T03:48:59.227 INFO:tasks.ceph:setting config [mgr] client_mount_timeout = 30 2024-09-17T03:48:59.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mgr client_mount_timeout 30 2024-09-17T03:48:59.277 INFO:tasks.ceph:setting config [mgr] debug_client = 20 2024-09-17T03:48:59.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mgr debug_client 20 2024-09-17T03:48:59.326 INFO:tasks.ceph:setting config [mgr] debug_mgr = 20 2024-09-17T03:48:59.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mgr debug_mgr 20 2024-09-17T03:48:59.372 INFO:tasks.ceph:setting config [mgr] debug_ms = 1 2024-09-17T03:48:59.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mgr debug_ms 1 2024-09-17T03:48:59.422 INFO:tasks.ceph:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-17T03:48:59.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mgr mon_warn_on_pool_no_app False 2024-09-17T03:48:59.471 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-17T03:48:59.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-17T03:48:59.527 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-17T03:48:59.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-17T03:48:59.575 DEBUG:tasks.ceph:set 22 configs 2024-09-17T03:48:59.575 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.137+0000 7f7f6ea00640 1 -- 172.21.15.129:0/3015844723 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v3)=0 v3) ==== 109+0+0 (secure 0 0 0) 0x7f7f7001b520 con 0x7f7f78158a90 2024-09-17T03:49:09.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.141+0000 7f0303400640 1 -- 172.21.15.129:0/2675068633 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v4)=0 v4) ==== 125+0+0 (secure 0 0 0) 0x7f02f4019040 con 0x7f02f0003be0 2024-09-17T03:49:09.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 -- 172.21.15.129:0/2675068633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02f0003be0 msgr2=0x7f030c0c7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 --2- 172.21.15.129:0/2675068633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02f0003be0 0x7f030c0c7740 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f030c0d43c0 tx=0x7f02f4012ee0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.147+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648013dc0 con 0x7f865c0d2660 2024-09-17T03:49:09.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.147+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f865c0ca340 con 0x7f865c0d2660 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.147+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8648046020 con 0x7f865c0d2660 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 -- 172.21.15.129:0/2675068633 shutdown_connections 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 --2- 172.21.15.129:0/2675068633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02f0004800 0x7f030c0b8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 --2- 172.21.15.129:0/2675068633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02f0003be0 0x7f030c0c7740 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 --2- 172.21.15.129:0/2675068633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02e8003b00 0x7f030c0b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.145+0000 7f0311800640 1 -- 172.21.15.129:0/2675068633 >> 172.21.15.129:0/2675068633 conn(0x7f030c09d8f0 msgr2=0x7f030c0b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.146+0000 7f0311800640 1 -- 172.21.15.129:0/2675068633 shutdown_connections 2024-09-17T03:49:09.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.146+0000 7f0311800640 1 -- 172.21.15.129:0/2675068633 wait complete. 2024-09-17T03:49:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 -- 172.21.15.129:0/3015844723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f78158a90 msgr2=0x7f7f78161580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 --2- 172.21.15.129:0/3015844723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f78158a90 0x7f7f78161580 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7f780b3c10 tx=0x7f7f7000fac0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 -- 172.21.15.129:0/3015844723 shutdown_connections 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 --2- 172.21.15.129:0/3015844723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f78158a90 0x7f7f78161580 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 --2- 172.21.15.129:0/3015844723 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f5c004d90 0x7f7f78161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 --2- 172.21.15.129:0/3015844723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f58005b70 0x7f7f7814bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.153+0000 7f7f7d600640 1 -- 172.21.15.129:0/3015844723 >> 172.21.15.129:0/3015844723 conn(0x7f7f7808bfc0 msgr2=0x7f7f780b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.151+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f86480067e0 con 0x7f865c0d2660 2024-09-17T03:49:09.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.154+0000 7f7f7d600640 1 -- 172.21.15.129:0/3015844723 shutdown_connections 2024-09-17T03:49:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.154+0000 7f7f7d600640 1 -- 172.21.15.129:0/3015844723 wait complete. 2024-09-17T03:49:09.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.154+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f865c0d0c40 con 0x7f865c0d2660 2024-09-17T03:49:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.164+0000 7eff5d600640 1 -- 172.21.15.129:0/2320726996 <== mon.2 v2:172.21.15.181:3301/0 9 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v5)=0 v5) ==== 121+0+0 (secure 0 0 0) 0x7eff54028ad0 con 0x7eff48004800 2024-09-17T03:49:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.162+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f864804d030 con 0x7f865c0d2660 2024-09-17T03:49:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.163+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f4d3c0050e0 con 0x7f4d50004d40 2024-09-17T03:49:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.162+0000 7fb024e00640 1 -- 172.21.15.129:0/2254852230 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7faff00050e0 con 0x7fb0201f2f10 2024-09-17T03:49:09.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.166+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fdee00050e0 con 0x7fdef0004060 2024-09-17T03:49:09.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.171+0000 7f1764c00640 1 -- 172.21.15.129:0/288722749 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v6)=0 v6) ==== 115+0+0 (secure 0 0 0) 0x7f17600037a0 con 0x7f174c0039e0 2024-09-17T03:49:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.176+0000 7f176f400640 1 -- 172.21.15.129:0/288722749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f174c0039e0 msgr2=0x7f17681f3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.176+0000 7f176f400640 1 --2- 172.21.15.129:0/288722749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f174c0039e0 0x7f17681f3dd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f176002cbe0 tx=0x7f1760001120 comp rx=0 tx=0).stop 2024-09-17T03:49:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.177+0000 7f176f400640 1 -- 172.21.15.129:0/288722749 shutdown_connections 2024-09-17T03:49:09.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.177+0000 7f176f400640 1 --2- 172.21.15.129:0/288722749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f174c005100 0x7f176809cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.177+0000 7f176f400640 1 --2- 172.21.15.129:0/288722749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f174c004370 0x7f17680c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.177+0000 7f176f400640 1 --2- 172.21.15.129:0/288722749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f174c0039e0 0x7f17681f3dd0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f176002cbe0 tx=0x7f1760001120 comp rx=0 tx=0).stop 2024-09-17T03:49:09.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.177+0000 7f176f400640 1 -- 172.21.15.129:0/288722749 >> 172.21.15.129:0/288722749 conn(0x7f17680b1cb0 msgr2=0x7f17680ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7f176f400640 1 -- 172.21.15.129:0/288722749 shutdown_connections 2024-09-17T03:49:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 -- 172.21.15.129:0/2320726996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff48004800 msgr2=0x7eff601fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 --2- 172.21.15.129:0/2320726996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff48004800 0x7eff601fc1f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7eff54000c00 tx=0x7eff540109b0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7f176f400640 1 -- 172.21.15.129:0/288722749 wait complete. 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 -- 172.21.15.129:0/2320726996 shutdown_connections 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 --2- 172.21.15.129:0/2320726996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff601fc730 0x7eff600ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 --2- 172.21.15.129:0/2320726996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff48004800 0x7eff601fc1f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 --2- 172.21.15.129:0/2320726996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff480039b0 0x7eff60152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.178+0000 7eff67e00640 1 -- 172.21.15.129:0/2320726996 >> 172.21.15.129:0/2320726996 conn(0x7eff6008bfc0 msgr2=0x7eff60161a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.179+0000 7eff67e00640 1 -- 172.21.15.129:0/2320726996 shutdown_connections 2024-09-17T03:49:09.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.179+0000 7eff67e00640 1 -- 172.21.15.129:0/2320726996 wait complete. 2024-09-17T03:49:09.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.186+0000 7f44dc800640 1 -- 172.21.15.129:0/978470310 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f44ac0050e0 con 0x7f44bc0039b0 2024-09-17T03:49:09.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.189+0000 7fdf02000640 1 -- 172.21.15.129:0/3169296349 <== mon.2 v2:172.21.15.181:3301/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v7)=0 v7) ==== 106+0+0 (secure 0 0 0) 0x7fdefc01c070 con 0x7fdef0004060 2024-09-17T03:49:09.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.199+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdef0004060 msgr2=0x7fdf0815d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.199+0000 7fdf10800640 1 --2- 172.21.15.129:0/3169296349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdef0004060 0x7fdf0815d590 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdefc004e60 tx=0x7fdefc013820 comp rx=0 tx=0).stop 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.209+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 shutdown_connections 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.209+0000 7fdf10800640 1 --2- 172.21.15.129:0/3169296349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdef0004e60 0x7fdf080b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.209+0000 7fdf10800640 1 --2- 172.21.15.129:0/3169296349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdef0004060 0x7fdf0815d590 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.209+0000 7fdf10800640 1 --2- 172.21.15.129:0/3169296349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdeec006620 0x7fdf081fc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.209+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 >> 172.21.15.129:0/3169296349 conn(0x7fdf080b20a0 msgr2=0x7fdf080ae630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.208+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f58740050e0 con 0x7f5880005c40 2024-09-17T03:49:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.212+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 shutdown_connections 2024-09-17T03:49:09.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.212+0000 7fdf10800640 1 -- 172.21.15.129:0/3169296349 wait complete. 2024-09-17T03:49:09.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.216+0000 7fb016000640 1 -- 172.21.15.129:0/2254852230 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v8)=0 v8) ==== 121+0+0 (secure 0 0 0) 0x7fb02015f960 con 0x7fb0201f2f10 2024-09-17T03:49:09.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.219+0000 7f4d65600640 1 -- 172.21.15.129:0/2919576531 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v9)=0 v9) ==== 111+0+0 (secure 0 0 0) 0x7f4d6000f3b0 con 0x7f4d50004d40 2024-09-17T03:49:09.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.224+0000 7fb00f400640 1 -- 172.21.15.129:0/2254852230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0201f2f10 msgr2=0x7fb02015f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.224+0000 7fb00f400640 1 --2- 172.21.15.129:0/2254852230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0201f2f10 0x7fb02015f010 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb01002c310 tx=0x7fb010001120 comp rx=0 tx=0).stop 2024-09-17T03:49:09.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.224+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f02180050e0 con 0x7f0228004800 2024-09-17T03:49:09.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d50004d40 msgr2=0x7f4d6815dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 --2- 172.21.15.129:0/2919576531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d50004d40 0x7f4d6815dd60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4d680cb3c0 tx=0x7f4d60012810 comp rx=0 tx=0).stop 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 shutdown_connections 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 --2- 172.21.15.129:0/2919576531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d50004d40 0x7f4d6815dd60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 --2- 172.21.15.129:0/2919576531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d50003fc0 0x7f4d6814b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 --2- 172.21.15.129:0/2919576531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d480036b0 0x7f4d6815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 >> 172.21.15.129:0/2919576531 conn(0x7f4d6809d680 msgr2=0x7f4d680b7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 shutdown_connections 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7f4d6ec00640 1 -- 172.21.15.129:0/2919576531 wait complete. 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7fb00f400640 1 -- 172.21.15.129:0/2254852230 shutdown_connections 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7fb00f400640 1 --2- 172.21.15.129:0/2254852230 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb02015f550 0x7fb020153880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7fb00f400640 1 --2- 172.21.15.129:0/2254852230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0201f2f10 0x7fb02015f010 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb01002c310 tx=0x7fb010001120 comp rx=0 tx=0).stop 2024-09-17T03:49:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7fb00f400640 1 --2- 172.21.15.129:0/2254852230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb004004800 0x7fb0200b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.227+0000 7fb00f400640 1 -- 172.21.15.129:0/2254852230 >> 172.21.15.129:0/2254852230 conn(0x7fb0200cdcb0 msgr2=0x7fb0200b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.228+0000 7fb00f400640 1 -- 172.21.15.129:0/2254852230 shutdown_connections 2024-09-17T03:49:09.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.233+0000 7fb00f400640 1 -- 172.21.15.129:0/2254852230 wait complete. 2024-09-17T03:49:09.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.244+0000 7f44ce000640 1 -- 172.21.15.129:0/978470310 <== mon.0 v2:172.21.15.129:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v10)=0 v10) ==== 120+0+0 (secure 0 0 0) 0x7f44d42023d0 con 0x7f44bc0039b0 2024-09-17T03:49:09.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.251+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7fd8680050e0 con 0x7fd87c0039b0 2024-09-17T03:49:09.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.255+0000 7f44c3400640 1 -- 172.21.15.129:0/978470310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44bc0039b0 msgr2=0x7f44d4201e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.255+0000 7f44c3400640 1 --2- 172.21.15.129:0/978470310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44bc0039b0 0x7f44d4201e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44d00146b0 tx=0x7f44d0014b70 comp rx=0 tx=0).stop 2024-09-17T03:49:09.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 -- 172.21.15.129:0/978470310 shutdown_connections 2024-09-17T03:49:09.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 --2- 172.21.15.129:0/978470310 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44bc005350 0x7f44d40c6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 --2- 172.21.15.129:0/978470310 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44bc004800 0x7f44d41533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 --2- 172.21.15.129:0/978470310 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44bc0039b0 0x7f44d4201e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 -- 172.21.15.129:0/978470310 >> 172.21.15.129:0/978470310 conn(0x7f44d406d5d0 msgr2=0x7f44d40734b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 -- 172.21.15.129:0/978470310 shutdown_connections 2024-09-17T03:49:09.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.257+0000 7f44c3400640 1 -- 172.21.15.129:0/978470310 wait complete. 2024-09-17T03:49:09.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.263+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fb9680050e0 con 0x7fb97c0039b0 2024-09-17T03:49:09.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.269+0000 7f589cc00640 1 -- 172.21.15.129:0/2996241628 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v11)=0 v11) ==== 120+0+0 (secure 0 0 0) 0x7f5898019040 con 0x7f5880005c40 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.278+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5880005c40 msgr2=0x7f58a0162220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.278+0000 7f58a7800640 1 --2- 172.21.15.129:0/2996241628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5880005c40 0x7f58a0162220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5880008eb0 tx=0x7f58980117b0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.278+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 shutdown_connections 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 --2- 172.21.15.129:0/2996241628 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5884004f40 0x7f58a009cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 --2- 172.21.15.129:0/2996241628 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5884004140 0x7f58a00b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 --2- 172.21.15.129:0/2996241628 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5880005c40 0x7f58a0162220 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 >> 172.21.15.129:0/2996241628 conn(0x7f58a00b20a0 msgr2=0x7f58a00cd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 shutdown_connections 2024-09-17T03:49:09.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.279+0000 7f58a7800640 1 -- 172.21.15.129:0/2996241628 wait complete. 2024-09-17T03:49:09.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.292+0000 7f023aa00640 1 -- 172.21.15.129:0/2392252776 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v12)=0 v12) ==== 106+0+0 (secure 0 0 0) 0x7f0230041660 con 0x7f0228004800 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0228004800 msgr2=0x7f02440c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 --2- 172.21.15.129:0/2392252776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0228004800 0x7f02440c5870 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f022800c380 tx=0x7f0230004730 comp rx=0 tx=0).stop 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 shutdown_connections 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 --2- 172.21.15.129:0/2392252776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0228005270 0x7f02440c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 --2- 172.21.15.129:0/2392252776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0228004800 0x7f02440c5870 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 --2- 172.21.15.129:0/2392252776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02280039b0 0x7f02441db500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.294+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 >> 172.21.15.129:0/2392252776 conn(0x7f024409d680 msgr2=0x7f02440b3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.302+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7fd630004170 con 0x7fd63c00eb60 2024-09-17T03:49:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.304+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 shutdown_connections 2024-09-17T03:49:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.304+0000 7f0243e00640 1 -- 172.21.15.129:0/2392252776 wait complete. 2024-09-17T03:49:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.310+0000 7fd890c00640 1 -- 172.21.15.129:0/1727513260 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v13)=0 v13) ==== 117+0+0 (secure 0 0 0) 0x7fd880018070 con 0x7fd87c0039b0 2024-09-17T03:49:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.313+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd87c0039b0 msgr2=0x7fd89415eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.313+0000 7fd89b600640 1 --2- 172.21.15.129:0/1727513260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd87c0039b0 0x7fd89415eb30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd89408d9b0 tx=0x7fd880013f10 comp rx=0 tx=0).stop 2024-09-17T03:49:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.317+0000 7fb991600640 1 -- 172.21.15.129:0/3002900562 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v14)=0 v14) ==== 119+0+0 (secure 0 0 0) 0x7fb9941f4750 con 0x7fb97c0039b0 2024-09-17T03:49:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 shutdown_connections 2024-09-17T03:49:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fd89b600640 1 --2- 172.21.15.129:0/1727513260 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd87c004800 0x7fd8940b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fd89b600640 1 --2- 172.21.15.129:0/1727513260 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd87c0039b0 0x7fd89415eb30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fd89b600640 1 --2- 172.21.15.129:0/1727513260 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8740094e0 0x7fd8941f5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 >> 172.21.15.129:0/1727513260 conn(0x7fd89408bfc0 msgr2=0x7fd8941617f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97c0039b0 msgr2=0x7fb9940c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.319+0000 7fb99c400640 1 --2- 172.21.15.129:0/3002900562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97c0039b0 0x7fb9940c3c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb980004c90 tx=0x7fb980001630 comp rx=0 tx=0).stop 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 shutdown_connections 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 --2- 172.21.15.129:0/3002900562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb97c004800 0x7fb9941f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 --2- 172.21.15.129:0/3002900562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb97c0039b0 0x7fb9940c3c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 --2- 172.21.15.129:0/3002900562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb974009310 0x7fb99409d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 >> 172.21.15.129:0/3002900562 conn(0x7fb9940b1cb0 msgr2=0x7fb9940b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 shutdown_connections 2024-09-17T03:49:09.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fd89b600640 1 -- 172.21.15.129:0/1727513260 wait complete. 2024-09-17T03:49:09.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 shutdown_connections 2024-09-17T03:49:09.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.320+0000 7fb99c400640 1 -- 172.21.15.129:0/3002900562 wait complete. 2024-09-17T03:49:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.321+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fdbb4005100 con 0x7fdbc0003b20 2024-09-17T03:49:09.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.325+0000 7fd652a00640 1 -- 172.21.15.129:0/3766849584 <== mon.2 v2:172.21.15.181:3301/0 9 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v15)=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7fd64c049750 con 0x7fd63c00eb60 2024-09-17T03:49:09.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.327+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd63c00eb60 msgr2=0x7fd65c156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.327+0000 7fd661c00640 1 --2- 172.21.15.129:0/3766849584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd63c00eb60 0x7fd65c156400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd64c02c7c0 tx=0x7fd64c004730 comp rx=0 tx=0).stop 2024-09-17T03:49:09.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.328+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fbc080050e0 con 0x7fbc1c0039b0 2024-09-17T03:49:09.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.340+0000 7fdbd6a00640 1 -- 172.21.15.129:0/4054576088 <== mon.2 v2:172.21.15.181:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fdbd000ebd0 con 0x7fdbc0003b20 2024-09-17T03:49:09.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fbc31600640 1 -- 172.21.15.129:0/1940754185 <== mon.1 v2:172.21.15.181:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7fbc200206a0 con 0x7fbc1c0039b0 2024-09-17T03:49:09.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.338+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 7 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f864802c780 con 0x7f865c0d2660 2024-09-17T03:49:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7f163cc00640 1 -- 172.21.15.129:0/2515353788 <== mon.1 v2:172.21.15.181:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f163403d7e0 con 0x7f16200066d0 2024-09-17T03:49:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f16140050e0 con 0x7f16200066d0 2024-09-17T03:49:09.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.338+0000 7f6f53e00640 1 -- 172.21.15.129:0/1544672453 <== mon.2 v2:172.21.15.181:3301/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f6f48025e20 con 0x7f6f5c1de7f0 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 shutdown_connections 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 --2- 172.21.15.129:0/3766849584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd65c158180 0x7fd65c09cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 --2- 172.21.15.129:0/3766849584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd65c0ad550 0x7fd65c09c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 --2- 172.21.15.129:0/3766849584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd63c00eb60 0x7fd65c156400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 >> 172.21.15.129:0/3766849584 conn(0x7fd65c06d7d0 msgr2=0x7fd65c15f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 shutdown_connections 2024-09-17T03:49:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.337+0000 7fd661c00640 1 -- 172.21.15.129:0/3766849584 wait complete. 2024-09-17T03:49:09.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.342+0000 7f8d76000640 1 -- 172.21.15.129:0/990091388 <== mon.0 v2:172.21.15.129:3300/0 9 ==== config(3 keys) ==== 85+0+0 (secure 0 0 0) 0x7f8d6c006ca0 con 0x7f8d7804d3d0 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.343+0000 7fdbd6a00640 1 -- 172.21.15.129:0/4054576088 <== mon.2 v2:172.21.15.181:3301/0 10 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7fdbd0015040 con 0x7fdbc0003b20 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc0003b20 msgr2=0x7fdbe014cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 --2- 172.21.15.129:0/4054576088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc0003b20 0x7fdbe014cca0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0010a10 tx=0x7fdbd0010ed0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 shutdown_connections 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 --2- 172.21.15.129:0/4054576088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc4004800 0x7fdbe01f40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 --2- 172.21.15.129:0/4054576088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc4003a00 0x7fdbe014d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 --2- 172.21.15.129:0/4054576088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc0003b20 0x7fdbe014cca0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.345+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 >> 172.21.15.129:0/4054576088 conn(0x7fdbe00b1cb0 msgr2=0x7fdbe00ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.346+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 shutdown_connections 2024-09-17T03:49:09.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.346+0000 7fdbe5a00640 1 -- 172.21.15.129:0/4054576088 wait complete. 2024-09-17T03:49:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.350+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 8 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f864803e450 con 0x7f865c0d2660 2024-09-17T03:49:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.350+0000 7f6f53e00640 1 -- 172.21.15.129:0/1544672453 <== mon.2 v2:172.21.15.181:3301/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f6f48025e20 con 0x7f6f5c1de7f0 2024-09-17T03:49:09.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.352+0000 7fbc31600640 1 -- 172.21.15.129:0/1940754185 <== mon.1 v2:172.21.15.181:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7fbc200206a0 con 0x7fbc1c0039b0 2024-09-17T03:49:09.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.352+0000 7fbc31600640 1 -- 172.21.15.129:0/1940754185 <== mon.1 v2:172.21.15.181:3300/0 11 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v17)=0 v17) ==== 114+0+0 (secure 0 0 0) 0x7fbc200208f0 con 0x7fbc1c0039b0 2024-09-17T03:49:09.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.353+0000 7f163cc00640 1 -- 172.21.15.129:0/2515353788 <== mon.1 v2:172.21.15.181:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f163403da00 con 0x7f16200066d0 2024-09-17T03:49:09.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.357+0000 7f8d76000640 1 -- 172.21.15.129:0/990091388 <== mon.0 v2:172.21.15.129:3300/0 10 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f8d6c0067e0 con 0x7f8d7804d3d0 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.358+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0039b0 msgr2=0x7fbc34157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.358+0000 7fbc3c000640 1 --2- 172.21.15.129:0/1940754185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0039b0 0x7fbc34157930 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc14005fe0 tx=0x7fbc20013960 comp rx=0 tx=0).stop 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.358+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 shutdown_connections 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.358+0000 7fbc3c000640 1 --2- 172.21.15.129:0/1940754185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c004d90 0x7fbc340ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.358+0000 7fbc3c000640 1 --2- 172.21.15.129:0/1940754185 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0039b0 0x7fbc34157930 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.359+0000 7fbc3c000640 1 --2- 172.21.15.129:0/1940754185 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc14003ad0 0x7fbc341f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.359+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 >> 172.21.15.129:0/1940754185 conn(0x7fbc3408bfc0 msgr2=0x7fbc340b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.359+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 shutdown_connections 2024-09-17T03:49:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.359+0000 7fbc3c000640 1 -- 172.21.15.129:0/1940754185 wait complete. 2024-09-17T03:49:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.369+0000 7f163cc00640 1 -- 172.21.15.129:0/2515353788 <== mon.1 v2:172.21.15.181:3300/0 11 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v18)=0 v18) ==== 119+0+0 (secure 0 0 0) 0x7f163403d410 con 0x7f16200066d0 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.371+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16200066d0 msgr2=0x7f16400c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.371+0000 7f1647a00640 1 --2- 172.21.15.129:0/2515353788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16200066d0 0x7f16400c8940 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1634006690 tx=0x7f16340012c0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.373+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 shutdown_connections 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.373+0000 7f1647a00640 1 --2- 172.21.15.129:0/2515353788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16401fcbc0 0x7f16400c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.373+0000 7f1647a00640 1 --2- 172.21.15.129:0/2515353788 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1628004800 0x7f16400ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.373+0000 7f1647a00640 1 --2- 172.21.15.129:0/2515353788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16200066d0 0x7f16400c8940 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.373+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 >> 172.21.15.129:0/2515353788 conn(0x7f164009d870 msgr2=0x7f16400d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.381+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 shutdown_connections 2024-09-17T03:49:09.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.381+0000 7f1647a00640 1 -- 172.21.15.129:0/2515353788 wait complete. 2024-09-17T03:49:09.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.381+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f86300050e0 con 0x7f865c0d2660 2024-09-17T03:49:09.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.382+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f8d54005100 con 0x7f8d7804d3d0 2024-09-17T03:49:09.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.385+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7f6f300050e0 con 0x7f6f5c1de7f0 2024-09-17T03:49:09.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.390+0000 7f8d76000640 1 -- 172.21.15.129:0/990091388 <== mon.0 v2:172.21.15.129:3300/0 11 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v19)=0 v19) ==== 122+0+0 (secure 0 0 0) 0x7f8d78181340 con 0x7f8d7804d3d0 2024-09-17T03:49:09.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.391+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7804d3d0 msgr2=0x7f8d781771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.391+0000 7f8d7f200640 1 --2- 172.21.15.129:0/990091388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7804d3d0 0x7f8d781771f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8d60010290 tx=0x7f8d6c001120 comp rx=0 tx=0).stop 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 shutdown_connections 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 --2- 172.21.15.129:0/990091388 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d781809f0 0x7f8d78180e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 --2- 172.21.15.129:0/990091388 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d7804d3d0 0x7f8d781771f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 --2- 172.21.15.129:0/990091388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d6000f6a0 0x7f8d780da4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 >> 172.21.15.129:0/990091388 conn(0x7f8d7800e200 msgr2=0x7f8d7800d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 shutdown_connections 2024-09-17T03:49:09.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.392+0000 7f8d7f200640 1 -- 172.21.15.129:0/990091388 wait complete. 2024-09-17T03:49:09.395 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.395 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.395 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.395+0000 7f8659600640 1 -- 172.21.15.129:0/3832340307 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v20)=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f864803e620 con 0x7f865c0d2660 2024-09-17T03:49:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f865c0d2660 msgr2=0x7f865c0a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 --2- 172.21.15.129:0/3832340307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f865c0d2660 0x7f865c0a1150 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8648032000 tx=0x7f8648001160 comp rx=0 tx=0).stop 2024-09-17T03:49:09.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 shutdown_connections 2024-09-17T03:49:09.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 --2- 172.21.15.129:0/3832340307 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f865c0a1690 0x7f865c0cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 --2- 172.21.15.129:0/3832340307 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f865c0d2660 0x7f865c0a1150 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 --2- 172.21.15.129:0/3832340307 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f865c1f9250 0x7f865c0d2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 >> 172.21.15.129:0/3832340307 conn(0x7f865c09d6b0 msgr2=0x7f865c0b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 shutdown_connections 2024-09-17T03:49:09.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.400+0000 7f8663000640 1 -- 172.21.15.129:0/3832340307 wait complete. 2024-09-17T03:49:09.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.403+0000 7f6f53e00640 1 -- 172.21.15.129:0/1544672453 <== mon.2 v2:172.21.15.181:3301/0 11 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v21)=0 v21) ==== 108+0+0 (secure 0 0 0) 0x7f6f5c1df870 con 0x7f6f5c1de7f0 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.404+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f5c1de7f0 msgr2=0x7f6f5c1d3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.404+0000 7f6f61a00640 1 --2- 172.21.15.129:0/1544672453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f5c1de7f0 0x7f6f5c1d3630 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f480119c0 tx=0x7f6f48011e80 comp rx=0 tx=0).stop 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 shutdown_connections 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 --2- 172.21.15.129:0/1544672453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f5c1def20 0x7f6f5c1df330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 --2- 172.21.15.129:0/1544672453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f5c1de7f0 0x7f6f5c1d3630 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 --2- 172.21.15.129:0/1544672453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f3c00f360 0x7f6f5c1e0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 >> 172.21.15.129:0/1544672453 conn(0x7f6f5c09d6a0 msgr2=0x7f6f5c0d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 shutdown_connections 2024-09-17T03:49:09.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.405+0000 7f6f61a00640 1 -- 172.21.15.129:0/1544672453 wait complete. 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.434+0000 7f10f1c00640 1 -- 172.21.15.129:0/3202918178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc00ebe0 msgr2=0x7f10cc00f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 --2- 172.21.15.129:0/3202918178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc00ebe0 0x7f10cc00f020 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00f6e0 tx=0x7f10dc02d8b0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10eb400640 1 -- 172.21.15.129:0/3202918178 reap_dead start 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 -- 172.21.15.129:0/3202918178 shutdown_connections 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 --2- 172.21.15.129:0/3202918178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10cc00ebe0 0x7f10cc00f020 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 -- 172.21.15.129:0/3202918178 >> 172.21.15.129:0/3202918178 conn(0x7f10ec06d550 msgr2=0x7f10ec071150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 -- 172.21.15.129:0/3202918178 shutdown_connections 2024-09-17T03:49:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.435+0000 7f10f1c00640 1 -- 172.21.15.129:0/3202918178 wait complete. 2024-09-17T03:49:09.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.437+0000 7f10f1c00640 1 Processor -- start 2024-09-17T03:49:09.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.437+0000 7f10f1c00640 1 -- start start 2024-09-17T03:49:09.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10cc00ebe0 0x7f10ec1f95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec1fa020 0x7f10ec14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec15f170 con 0x7f10ec14b6e0 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec06f880 con 0x7f10ec1fa020 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10f1c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10ec15f2f0 con 0x7f10cc00ebe0 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10ea000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10ea000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58900/0 (socket says 172.21.15.129:58900) 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10ea000640 1 -- 172.21.15.129:0/1256713492 learned_addr learned my addr 172.21.15.129:0/1256713492 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10eaa00640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10cc00ebe0 0x7f10ec1f95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.439+0000 7f10eb400640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec1fa020 0x7f10ec14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 -- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10cc00ebe0 msgr2=0x7f10ec1f95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10cc00ebe0 0x7f10ec1f95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 -- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec1fa020 msgr2=0x7f10ec14c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec1fa020 0x7f10ec14c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 -- 172.21.15.129:0/1256713492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec14e3d0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10ea000640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10d8003d50 tx=0x7f10d80016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d8012850 con 0x7f10ec14b6e0 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(4 keys) ==== 110+0+0 (secure 0 0 0) 0x7f10d8012ac0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ec14e6b0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d8011440 con 0x7f10ec14b6e0 2024-09-17T03:49:09.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.440+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ec0b6ba0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.441+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f10d8011970 con 0x7f10ec14b6e0 2024-09-17T03:49:09.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.441+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f10d8010700 con 0x7f10ec14b6e0 2024-09-17T03:49:09.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.442+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec1fa4f0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.444+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f10ec1fa4f0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.507+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f10ec14ebd0 con 0x7f10ec14b6e0 2024-09-17T03:49:09.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.510+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v22)=0 v22) ==== 123+0+0 (secure 0 0 0) 0x7f10d8021030 con 0x7f10ec14b6e0 2024-09-17T03:49:09.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.511+0000 7f10e2a00640 1 -- 172.21.15.129:0/1256713492 <== mon.0 v2:172.21.15.129:3300/0 8 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10d8016020 con 0x7f10ec14b6e0 2024-09-17T03:49:09.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 msgr2=0x7f10ec1f9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f10d8003d50 tx=0x7f10d80016b0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 shutdown_connections 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec1fa020 0x7f10ec14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec14b6e0 0x7f10ec1f9ae0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 --2- 172.21.15.129:0/1256713492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10cc00ebe0 0x7f10ec1f95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 >> 172.21.15.129:0/1256713492 conn(0x7f10ec06d550 msgr2=0x7f10ec153fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 shutdown_connections 2024-09-17T03:49:09.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.512+0000 7f10f1c00640 1 -- 172.21.15.129:0/1256713492 wait complete. 2024-09-17T03:49:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 -- 172.21.15.129:0/919152493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 msgr2=0x7fc6f000f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 --2- 172.21.15.129:0/919152493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc6f000f160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc005ca0 tx=0x7fc6fc02de00 comp rx=0 tx=0).stop 2024-09-17T03:49:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc715600640 1 -- 172.21.15.129:0/919152493 reap_dead start 2024-09-17T03:49:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 -- 172.21.15.129:0/919152493 shutdown_connections 2024-09-17T03:49:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 --2- 172.21.15.129:0/919152493 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc6f000f160 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 -- 172.21.15.129:0/919152493 >> 172.21.15.129:0/919152493 conn(0x7fc7100b1cb0 msgr2=0x7fc7100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 -- 172.21.15.129:0/919152493 shutdown_connections 2024-09-17T03:49:09.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.519+0000 7fc717600640 1 -- 172.21.15.129:0/919152493 wait complete. 2024-09-17T03:49:09.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.523+0000 7fc717600640 1 Processor -- start 2024-09-17T03:49:09.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.523+0000 7fc717600640 1 -- start start 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc71014e200 0x7fc710157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7100c3e50 0x7fc7100c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100cf190 con 0x7fc6f000ed50 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc7100cf010 con 0x7fc71014e200 2024-09-17T03:49:09.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc717600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc7100cf310 con 0x7fc7100c3e50 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58928/0 (socket says 172.21.15.129:58928) 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 -- 172.21.15.129:0/2704787407 learned_addr learned my addr 172.21.15.129:0/2704787407 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc70fe00640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc71014e200 0x7fc710157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc715600640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7100c3e50 0x7fc7100c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 -- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7100c3e50 msgr2=0x7fc7100c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7100c3e50 0x7fc7100c4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 -- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc71014e200 msgr2=0x7fc710157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc71014e200 0x7fc710157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.526+0000 7fc714c00640 1 -- 172.21.15.129:0/2704787407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7100c47a0 con 0x7fc6f000ed50 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc714c00640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc710072600 tx=0x7fc7040016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:09.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704012850 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc704003e50 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704012d80 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7100c5fc0 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.527+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7100c6580 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.528+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc7040117d0 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.528+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc704024350 con 0x7fc6f000ed50 2024-09-17T03:49:09.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.529+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7100ccca0 con 0x7fc6f000ed50 2024-09-17T03:49:09.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.531+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc7100c47a0 con 0x7fc6f000ed50 2024-09-17T03:49:09.539 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.540 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.596+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fc7100c6bd0 con 0x7fc6f000ed50 2024-09-17T03:49:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.599+0000 7fc70cc00640 1 -- 172.21.15.129:0/2704787407 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v23)=0 v23) ==== 108+0+0 (secure 0 0 0) 0x7fc704011ca0 con 0x7fc6f000ed50 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 msgr2=0x7fc71009b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc710072600 tx=0x7fc7040016b0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 shutdown_connections 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7100c3e50 0x7fc7100c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc71014e200 0x7fc710157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 --2- 172.21.15.129:0/2704787407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6f000ed50 0x7fc71009b960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 >> 172.21.15.129:0/2704787407 conn(0x7fc7100b1cb0 msgr2=0x7fc71015a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 shutdown_connections 2024-09-17T03:49:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.600+0000 7fc717600640 1 -- 172.21.15.129:0/2704787407 wait complete. 2024-09-17T03:49:09.622 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.623 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi129') 2024-09-17T03:49:09.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config dump 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 -- 172.21.15.129:0/3609298004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba00d1d40 msgr2=0x7f2ba00b5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 --2- 172.21.15.129:0/3609298004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba00d1d40 0x7f2ba00b5210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0074550 tx=0x7f2b9402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 -- 172.21.15.129:0/3609298004 shutdown_connections 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 --2- 172.21.15.129:0/3609298004 >> v2:172.21.15.181:3300/0 conn(0x7f2ba015c6f0 0x7f2ba00ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 --2- 172.21.15.129:0/3609298004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba00d1d40 0x7f2ba00b5210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 -- 172.21.15.129:0/3609298004 >> 172.21.15.129:0/3609298004 conn(0x7f2ba00b3f10 msgr2=0x7f2ba00b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 -- 172.21.15.129:0/3609298004 shutdown_connections 2024-09-17T03:49:09.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.844+0000 7f2ba7600640 1 -- 172.21.15.129:0/3609298004 wait complete. 2024-09-17T03:49:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.846+0000 7f2ba7600640 1 Processor -- start 2024-09-17T03:49:09.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.847+0000 7f2ba7600640 1 -- start start 2024-09-17T03:49:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ba00d1d40 0x7f2ba01e93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 0x7f2ba01e98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba00d11f0 con 0x7f2ba015c6f0 2024-09-17T03:49:09.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2ba00d1070 con 0x7f2ba00ccba0 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba7600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2ba00d1370 con 0x7f2ba00d1d40 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba5600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 0x7f2ba01e98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2b9fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ba00d1d40 0x7f2ba01e93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:41448/0 (socket says 172.21.15.129:41448) 2024-09-17T03:49:09.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.848+0000 7f2ba4c00640 1 -- 172.21.15.129:0/4125473011 learned_addr learned my addr 172.21.15.129:0/4125473011 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 -- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ba00d1d40 msgr2=0x7f2ba01e93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ba00d1d40 0x7f2ba01e93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 -- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 msgr2=0x7f2ba01553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 -- 172.21.15.129:0/4125473011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba00c8fa0 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba4c00640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba5600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 0x7f2ba01e98f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c730 tx=0x7f2b9800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98003b70 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b98003d10 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ba006ee20 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b98005330 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.849+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ba006f3e0 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.850+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2b980056e0 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.850+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2b98012aa0 con 0x7f2ba015c6f0 2024-09-17T03:49:09.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.850+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00d1870 con 0x7f2ba015c6f0 2024-09-17T03:49:09.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.853+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2b9800a640 con 0x7f2ba015c6f0 2024-09-17T03:49:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.915+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f2ba00b1a80 con 0x7f2ba015c6f0 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.916+0000 7f2b9cc00640 1 -- 172.21.15.129:0/4125473011 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v23) ==== 59+0+1472 (secure 0 0 0) 0x7f2b9800a820 con 0x7f2ba015c6f0 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:49:09.916 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:49:09.917 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:49:09.918 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 msgr2=0x7f2ba01e98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 0x7f2ba01e98f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c730 tx=0x7f2b9800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 shutdown_connections 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ba015c6f0 0x7f2ba01e98f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ba00d1d40 0x7f2ba01e93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 --2- 172.21.15.129:0/4125473011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ba00ccba0 0x7f2ba01553c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 >> 172.21.15.129:0/4125473011 conn(0x7f2ba00b3f10 msgr2=0x7f2ba00d22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 shutdown_connections 2024-09-17T03:49:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:09.917+0000 7f2ba7600640 1 -- 172.21.15.129:0/4125473011 wait complete. 2024-09-17T03:49:09.939 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T03:49:09.940 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph osd crush tunables default 2024-09-17T03:49:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.160+0000 7fbba7200640 1 -- 172.21.15.129:0/2887298945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 msgr2=0x7fbba80cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.160+0000 7fbba7200640 1 --2- 172.21.15.129:0/2887298945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba80cf3e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb94005fd0 tx=0x7fbb9402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.161+0000 7fbba7200640 1 -- 172.21.15.129:0/2887298945 shutdown_connections 2024-09-17T03:49:10.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.161+0000 7fbba7200640 1 --2- 172.21.15.129:0/2887298945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba80cf3e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.161+0000 7fbba7200640 1 -- 172.21.15.129:0/2887298945 >> 172.21.15.129:0/2887298945 conn(0x7fbba809d680 msgr2=0x7fbba80d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.161+0000 7fbba7200640 1 -- 172.21.15.129:0/2887298945 shutdown_connections 2024-09-17T03:49:10.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.161+0000 7fbba7200640 1 -- 172.21.15.129:0/2887298945 wait complete. 2024-09-17T03:49:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.163+0000 7fbba7200640 1 Processor -- start 2024-09-17T03:49:10.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.163+0000 7fbba7200640 1 -- start start 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbba80b42f0 0x7fbba814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbba80b96b0 0x7fbba81696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80b5040 con 0x7fbba80cefd0 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80b4ec0 con 0x7fbba80b96b0 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.164+0000 7fbba7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbba816a0a0 con 0x7fbba80b42f0 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:10.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbba80b96b0 0x7fbba81696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58968/0 (socket says 172.21.15.129:58968) 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 -- 172.21.15.129:0/1114999636 learned_addr learned my addr 172.21.15.129:0/1114999636 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba5e00640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbba80b42f0 0x7fbba814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 -- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbba80b42f0 msgr2=0x7fbba814f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbba80b42f0 0x7fbba814f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 -- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbba80b96b0 msgr2=0x7fbba81696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbba80b96b0 0x7fbba81696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 -- 172.21.15.129:0/1114999636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba814fb90 con 0x7fbba80cefd0 2024-09-17T03:49:10.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.165+0000 7fbba6800640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c730 tx=0x7fbba000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:10.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba0003b70 con 0x7fbba80cefd0 2024-09-17T03:49:10.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbba0003d10 con 0x7fbba80cefd0 2024-09-17T03:49:10.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba000aab0 con 0x7fbba80cefd0 2024-09-17T03:49:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba8160d20 con 0x7fbba80cefd0 2024-09-17T03:49:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba81612e0 con 0x7fbba80cefd0 2024-09-17T03:49:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbba000ac50 con 0x7fbba80cefd0 2024-09-17T03:49:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.166+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbba0012940 con 0x7fbba80cefd0 2024-09-17T03:49:10.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.167+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba80cf3e0 con 0x7fbba80cefd0 2024-09-17T03:49:10.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.169+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fbba00050d0 con 0x7fbba80cefd0 2024-09-17T03:49:10.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.236+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbba80d4eb0 con 0x7fbba80cefd0 2024-09-17T03:49:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.238+0000 7fbb9e000640 1 -- 172.21.15.129:0/1114999636 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fbba0012b40 con 0x7fbba80cefd0 2024-09-17T03:49:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:adjusted tunables profile to default 2024-09-17T03:49:10.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 msgr2=0x7fbba8167560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:10.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c730 tx=0x7fbba000cbf0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 shutdown_connections 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbba80cefd0 0x7fbba8167560 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbba80b96b0 0x7fbba81696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 --2- 172.21.15.129:0/1114999636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbba80b42f0 0x7fbba814f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 >> 172.21.15.129:0/1114999636 conn(0x7fbba809d680 msgr2=0x7fbba80b7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.239+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 shutdown_connections 2024-09-17T03:49:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:10.240+0000 7fbba7200640 1 -- 172.21.15.129:0/1114999636 wait complete. 2024-09-17T03:49:10.263 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T03:49:10.263 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-17T03:49:10.264 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:10.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-17T03:49:10.319 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:10.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-17T03:49:10.423 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:10.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-17T03:49:10.518 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:10.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-17T03:49:10.616 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:49:10.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-17T03:49:10.673 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:49:10.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-17T03:49:10.770 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:49:10.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-17T03:49:10.866 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:49:10.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-17T03:49:10.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 3f601c8a-535a-4179-b27b-7e244b416963 0 2024-09-17T03:49:11.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 -- 172.21.15.181:0/4121273666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 msgr2=0x7f5bb80d0fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 --2- 172.21.15.181:0/4121273666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb80d0fb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8005fd0 tx=0x7f5ba802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 -- 172.21.15.181:0/4121273666 shutdown_connections 2024-09-17T03:49:11.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 --2- 172.21.15.181:0/4121273666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb80d0fb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.219 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 -- 172.21.15.181:0/4121273666 >> 172.21.15.181:0/4121273666 conn(0x7f5bb809d8f0 msgr2=0x7f5bb80d45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.220 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 -- 172.21.15.181:0/4121273666 shutdown_connections 2024-09-17T03:49:11.220 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.219+0000 7f5bbde00640 1 -- 172.21.15.181:0/4121273666 wait complete. 2024-09-17T03:49:11.221 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.221+0000 7f5bbde00640 1 Processor -- start 2024-09-17T03:49:11.221 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.221+0000 7f5bbde00640 1 -- start start 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb81f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb80d14f0 0x7f5bb81fa210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb80d1f10 con 0x7f5bb80d14f0 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb80d1d90 con 0x7f5bb80b9b60 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bbde00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb816a5a0 con 0x7f5bb80b3a10 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bb6a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bb6a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51128/0 (socket says 172.21.15.181:51128) 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bafe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb81f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.225 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bb6a00640 1 -- 172.21.15.181:0/1970865842 learned_addr learned my addr 172.21.15.181:0/1970865842 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.223+0000 7f5bb7400640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb80d14f0 0x7f5bb81fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 -- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 msgr2=0x7f5bb81f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb81f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 -- 172.21.15.181:0/1970865842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb80d14f0 msgr2=0x7f5bb81fa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb80d14f0 0x7f5bb81fa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 -- 172.21.15.181:0/1970865842 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb815a610 con 0x7f5bb80b3a10 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bafe00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb81f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bb6a00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000c00 tx=0x7f5ba8003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba803e030 con 0x7f5bb80b3a10 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ba8003e60 con 0x7f5bb80b3a10 2024-09-17T03:49:11.226 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8035040 con 0x7f5bb80b3a10 2024-09-17T03:49:11.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb815a8f0 con 0x7f5bb80b3a10 2024-09-17T03:49:11.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.224+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb80b4000 con 0x7f5bb80b3a10 2024-09-17T03:49:11.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.225+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5ba8041030 con 0x7f5bb80b3a10 2024-09-17T03:49:11.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.225+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f5ba8043d30 con 0x7f5bb80b3a10 2024-09-17T03:49:11.227 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.225+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb80b91f0 con 0x7f5bb80b3a10 2024-09-17T03:49:11.229 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.230+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5bb80b91f0 con 0x7f5bb80b3a10 2024-09-17T03:49:11.302 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.302+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3f601c8a-535a-4179-b27b-7e244b416963", "id": 0} v 0) -- 0x7f5bb806fe90 con 0x7f5bb80b3a10 2024-09-17T03:49:11.307 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.307+0000 7f5baf400640 1 -- 172.21.15.181:0/1970865842 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3f601c8a-535a-4179-b27b-7e244b416963", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f5ba8043410 con 0x7f5bb80b3a10 2024-09-17T03:49:11.307 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-09-17T03:49:11.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 msgr2=0x7f5bb8159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8000c00 tx=0x7f5ba8003970 comp rx=0 tx=0).stop 2024-09-17T03:49:11.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 shutdown_connections 2024-09-17T03:49:11.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5bb80d14f0 0x7f5bb81fa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5bb80b9b60 0x7f5bb81f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 --2- 172.21.15.181:0/1970865842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5bb80b3a10 0x7f5bb8159ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 >> 172.21.15.181:0/1970865842 conn(0x7f5bb809d8f0 msgr2=0x7f5bb80b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 shutdown_connections 2024-09-17T03:49:11.309 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.308+0000 7f5bbde00640 1 -- 172.21.15.181:0/1970865842 wait complete. 2024-09-17T03:49:11.328 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 04e0f8c1-680e-4a2f-9cfd-2e682f9848ec 1 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 -- 172.21.15.181:0/4159341702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400b96b0 msgr2=0x7fc3400d0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 --2- 172.21.15.181:0/4159341702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400b96b0 0x7fc3400d0de0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc3340098e0 tx=0x7fc33402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 -- 172.21.15.181:0/4159341702 shutdown_connections 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 --2- 172.21.15.181:0/4159341702 >> v2:172.21.15.129:3300/0 conn(0x7fc3400d1320 0x7fc3400cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 --2- 172.21.15.181:0/4159341702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400b96b0 0x7fc3400d0de0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.543+0000 7fc347200640 1 -- 172.21.15.181:0/4159341702 >> 172.21.15.181:0/4159341702 conn(0x7fc34009d680 msgr2=0x7fc3400d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.544+0000 7fc347200640 1 -- 172.21.15.181:0/4159341702 shutdown_connections 2024-09-17T03:49:11.544 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.544+0000 7fc347200640 1 -- 172.21.15.181:0/4159341702 wait complete. 2024-09-17T03:49:11.546 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.546+0000 7fc347200640 1 Processor -- start 2024-09-17T03:49:11.546 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.546+0000 7fc347200640 1 -- start start 2024-09-17T03:49:11.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 0x7fc3401f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.548 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400d1320 0x7fc340161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3400b4a00 con 0x7fc3400b96b0 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc3400b50d0 con 0x7fc3400cf510 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc347200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3400b4b80 con 0x7fc3400d1320 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:52096/0 (socket says 172.21.15.181:52096) 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345400640 1 -- 172.21.15.181:0/1127145548 learned_addr learned my addr 172.21.15.181:0/1127145548 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 0x7fc3401f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc346800640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400d1320 0x7fc340161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 -- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400d1320 msgr2=0x7fc340161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400d1320 0x7fc340161a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.549 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 -- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 msgr2=0x7fc3400b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.550 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.550 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345e00640 1 -- 172.21.15.181:0/1127145548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3401f9220 con 0x7fc3400b96b0 2024-09-17T03:49:11.550 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.548+0000 7fc345400640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:11.550 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc345e00640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 0x7fc3401f59e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc334031000 tx=0x7fc3340316b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33403f070 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc334031950 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334031da0 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3401f9e30 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3401f97e0 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc334042070 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.549+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fc334004c70 con 0x7fc3400b96b0 2024-09-17T03:49:11.551 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.550+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3400b71f0 con 0x7fc3400b96b0 2024-09-17T03:49:11.552 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.553+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc3400b71f0 con 0x7fc3400b96b0 2024-09-17T03:49:11.624 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.624+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd new", "uuid": "04e0f8c1-680e-4a2f-9cfd-2e682f9848ec", "id": 1} v 0) -- 0x7fc3400d0270 con 0x7fc3400b96b0 2024-09-17T03:49:11.626 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.626+0000 7fc33a000640 1 -- 172.21.15.181:0/1127145548 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04e0f8c1-680e-4a2f-9cfd-2e682f9848ec", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fc3400d0270 con 0x7fc3400b96b0 2024-09-17T03:49:11.626 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-17T03:49:11.627 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 msgr2=0x7fc3401f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 0x7fc3401f59e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc334031000 tx=0x7fc3340316b0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 shutdown_connections 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3400d1320 0x7fc340161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3400cf510 0x7fc3400b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 --2- 172.21.15.181:0/1127145548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3400b96b0 0x7fc3401f59e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.627+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 >> 172.21.15.181:0/1127145548 conn(0x7fc34009d680 msgr2=0x7fc34006db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.628+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 shutdown_connections 2024-09-17T03:49:11.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.628+0000 7fc347200640 1 -- 172.21.15.181:0/1127145548 wait complete. 2024-09-17T03:49:11.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 8c23300f-de54-4c2c-8799-6f0f587ab816 2 2024-09-17T03:49:11.870 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 -- 172.21.15.181:0/2269821298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 msgr2=0x7f5f4c0cfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.870 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 --2- 172.21.15.181:0/2269821298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c0cfa70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5f40005fd0 tx=0x7f5f4002a540 comp rx=0 tx=0).stop 2024-09-17T03:49:11.870 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 -- 172.21.15.181:0/2269821298 shutdown_connections 2024-09-17T03:49:11.870 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 --2- 172.21.15.181:0/2269821298 >> v2:172.21.15.129:3300/0 conn(0x7f5f4c0d2fd0 0x7f5f4c0d2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.871 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 --2- 172.21.15.181:0/2269821298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c0cfa70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.871 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 -- 172.21.15.181:0/2269821298 >> 172.21.15.181:0/2269821298 conn(0x7f5f4c09d6b0 msgr2=0x7f5f4c0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.871 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 -- 172.21.15.181:0/2269821298 shutdown_connections 2024-09-17T03:49:11.871 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.870+0000 7f5f52e00640 1 -- 172.21.15.181:0/2269821298 wait complete. 2024-09-17T03:49:11.872 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.872+0000 7f5f52e00640 1 Processor -- start 2024-09-17T03:49:11.872 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.872+0000 7f5f52e00640 1 -- start start 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 0x7f5f4c1e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f4c0d4f90 0x7f5f4c1e9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c0b3e20 con 0x7f5f4c0d4f90 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c0b3ca0 con 0x7f5f4c0d2fd0 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5f4c0b3fa0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.875 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f51a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f51000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 0x7f5f4c1e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f51a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51168/0 (socket says 172.21.15.181:51168) 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f51a00640 1 -- 172.21.15.181:0/1908640494 learned_addr learned my addr 172.21.15.181:0/1908640494 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.874+0000 7f5f52400640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f4c0d4f90 0x7f5f4c1e9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 -- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 msgr2=0x7f5f4c1e7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 0x7f5f4c1e7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 -- 172.21.15.181:0/1908640494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f4c0d4f90 msgr2=0x7f5f4c1e9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f4c0d4f90 0x7f5f4c1e9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 -- 172.21.15.181:0/1908640494 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c1f63e0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51000640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 0x7f5f4c1e7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f51a00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f40000c00 tx=0x7f5f400038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:11.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4003e030 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f40003db0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f40035040 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c1f66c0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.875+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4c0b71a0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.878 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.876+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5f40041030 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.878 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.876+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f5f40043820 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.878 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.877+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4c0cfa70 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.878 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.879+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f5f40043410 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.953+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8c23300f-de54-4c2c-8799-6f0f587ab816", "id": 2} v 0) -- 0x7f5f4c0d19f0 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.957 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.957+0000 7f5f49600640 1 -- 172.21.15.181:0/1908640494 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c23300f-de54-4c2c-8799-6f0f587ab816", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f5f40004d90 con 0x7f5f4c0b9bf0 2024-09-17T03:49:11.957 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 msgr2=0x7f5f4c1f5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f40000c00 tx=0x7f5f400038a0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 shutdown_connections 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f4c0d4f90 0x7f5f4c1e9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f4c0d2fd0 0x7f5f4c1e7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.958 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 --2- 172.21.15.181:0/1908640494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f4c0b9bf0 0x7f5f4c1f5b40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:11.959 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 >> 172.21.15.181:0/1908640494 conn(0x7f5f4c09d6b0 msgr2=0x7f5f4c0b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:11.959 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 shutdown_connections 2024-09-17T03:49:11.959 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:11.958+0000 7f5f52e00640 1 -- 172.21.15.181:0/1908640494 wait complete. 2024-09-17T03:49:11.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9 3 2024-09-17T03:49:12.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 -- 172.21.15.181:0/1024289560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 msgr2=0x7f18740cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 --2- 172.21.15.181:0/1024289560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f18740cef70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f18680098e0 tx=0x7f186802a540 comp rx=0 tx=0).stop 2024-09-17T03:49:12.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 -- 172.21.15.181:0/1024289560 shutdown_connections 2024-09-17T03:49:12.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 --2- 172.21.15.181:0/1024289560 >> v2:172.21.15.181:3300/0 conn(0x7f18740d1ae0 0x7f18740b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.197 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 --2- 172.21.15.181:0/1024289560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f18740cef70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.198 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 -- 172.21.15.181:0/1024289560 >> 172.21.15.181:0/1024289560 conn(0x7f187409d660 msgr2=0x7f18740d43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.198 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 -- 172.21.15.181:0/1024289560 shutdown_connections 2024-09-17T03:49:12.198 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.197+0000 7f187b000640 1 -- 172.21.15.181:0/1024289560 wait complete. 2024-09-17T03:49:12.199 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.199+0000 7f187b000640 1 Processor -- start 2024-09-17T03:49:12.199 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.199+0000 7f187b000640 1 -- start start 2024-09-17T03:49:12.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18740cf4b0 0x7f18741f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18740d1ae0 0x7f18741f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18740b49b0 con 0x7f18740cf4b0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f18740b4830 con 0x7f18740d1ae0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187b000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f18740b4b30 con 0x7f18740b96d0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51190/0 (socket says 172.21.15.181:51190) 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879c00640 1 -- 172.21.15.181:0/3732311243 learned_addr learned my addr 172.21.15.181:0/3732311243 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f187a600640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18740d1ae0 0x7f18741f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879200640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18740cf4b0 0x7f18741f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879c00640 1 -- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18740d1ae0 msgr2=0x7f18741f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.201+0000 7f1879c00640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18740d1ae0 0x7f18741f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1879c00640 1 -- 172.21.15.181:0/3732311243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18740cf4b0 msgr2=0x7f18741f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1879c00640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18740cf4b0 0x7f18741f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1879c00640 1 -- 172.21.15.181:0/3732311243 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187415c6f0 con 0x7f18740b96d0 2024-09-17T03:49:12.203 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1879c00640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1868009a00 tx=0x7f18680052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868005cf0 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1868031b50 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18680047a0 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1874160b00 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.202+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f187415d6e0 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.203+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1868004940 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.203+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f1868033540 con 0x7f18740b96d0 2024-09-17T03:49:12.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.203+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18740cef70 con 0x7f18740b96d0 2024-09-17T03:49:12.205 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.206+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f186803f030 con 0x7f18740b96d0 2024-09-17T03:49:12.279 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.279+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9", "id": 3} v 0) -- 0x7f1874160de0 con 0x7f18740b96d0 2024-09-17T03:49:12.284 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.284+0000 7f1871600640 1 -- 172.21.15.181:0/3732311243 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f1868033790 con 0x7f18740b96d0 2024-09-17T03:49:12.284 INFO:teuthology.orchestra.run.smithi181.stdout:3 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.285+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 msgr2=0x7f187415c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.285+0000 7f187b000640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1868009a00 tx=0x7f18680052f0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 shutdown_connections 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18740d1ae0 0x7f18741f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18740cf4b0 0x7f18741f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 --2- 172.21.15.181:0/3732311243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18740b96d0 0x7f187415c1b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 >> 172.21.15.181:0/3732311243 conn(0x7f187409d660 msgr2=0x7f18740cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 shutdown_connections 2024-09-17T03:49:12.286 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.286+0000 7f187b000640 1 -- 172.21.15.181:0/3732311243 wait complete. 2024-09-17T03:49:12.308 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 435a59a0-2802-45e3-bd1b-23e4ccaa280b 4 2024-09-17T03:49:12.527 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 -- 172.21.15.181:0/1296846844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b39c0 msgr2=0x7fbd400b3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1296846844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b39c0 0x7fbd400b3dd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbd400d4dd0 tx=0x7fbd3802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 -- 172.21.15.181:0/1296846844 shutdown_connections 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1296846844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b39c0 0x7fbd400b3dd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1296846844 >> v2:172.21.15.181:3300/0 conn(0x7fbd400d0d30 0x7fbd400d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 -- 172.21.15.181:0/1296846844 >> 172.21.15.181:0/1296846844 conn(0x7fbd4009d670 msgr2=0x7fbd400d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.527+0000 7fbd45e00640 1 -- 172.21.15.181:0/1296846844 shutdown_connections 2024-09-17T03:49:12.528 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.528+0000 7fbd45e00640 1 -- 172.21.15.181:0/1296846844 wait complete. 2024-09-17T03:49:12.529 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.530+0000 7fbd45e00640 1 Processor -- start 2024-09-17T03:49:12.529 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.530+0000 7fbd45e00640 1 -- start start 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 0x7fbd400b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 0x7fbd400a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400b9c00 con 0x7fbd400b94d0 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400b9a80 con 0x7fbd400b39c0 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbd40162fd0 con 0x7fbd400d0d30 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd44a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd44a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:52140/0 (socket says 172.21.15.181:52140) 2024-09-17T03:49:12.532 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd45400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 0x7fbd400a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.531+0000 7fbd44a00640 1 -- 172.21.15.181:0/1774398468 learned_addr learned my addr 172.21.15.181:0/1774398468 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd3fe00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 0x7fbd400b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 -- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 msgr2=0x7fbd400c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 -- 172.21.15.181:0/1774398468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 msgr2=0x7fbd400b8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 0x7fbd400b8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 -- 172.21.15.181:0/1774398468 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd401f86d0 con 0x7fbd400d0d30 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd44a00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd3fe00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 0x7fbd400b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:12.533 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.532+0000 7fbd45400640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 0x7fbd400a1510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd400d04c0 tx=0x7fbd38031740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:12.534 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38031990 con 0x7fbd400d0d30 2024-09-17T03:49:12.534 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd38031c20 con 0x7fbd400d0d30 2024-09-17T03:49:12.534 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd380330d0 con 0x7fbd400d0d30 2024-09-17T03:49:12.534 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd400c8f90 con 0x7fbd400d0d30 2024-09-17T03:49:12.535 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd401f7930 con 0x7fbd400d0d30 2024-09-17T03:49:12.535 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbd38033270 con 0x7fbd400d0d30 2024-09-17T03:49:12.535 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.533+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fbd38004040 con 0x7fbd400d0d30 2024-09-17T03:49:12.535 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.534+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd400b3dd0 con 0x7fbd400d0d30 2024-09-17T03:49:12.536 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.536+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fbd3804a030 con 0x7fbd400d0d30 2024-09-17T03:49:12.607 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.607+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "435a59a0-2802-45e3-bd1b-23e4ccaa280b", "id": 4} v 0) -- 0x7fbd400d1a90 con 0x7fbd400d0d30 2024-09-17T03:49:12.612 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.612+0000 7fbd3cc00640 1 -- 172.21.15.181:0/1774398468 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "435a59a0-2802-45e3-bd1b-23e4ccaa280b", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fbd3803f030 con 0x7fbd400d0d30 2024-09-17T03:49:12.612 INFO:teuthology.orchestra.run.smithi181.stdout:4 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 msgr2=0x7fbd400a1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 0x7fbd400a1510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbd400d04c0 tx=0x7fbd38031740 comp rx=0 tx=0).stop 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 shutdown_connections 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd400d0d30 0x7fbd400a1510 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd400b94d0 0x7fbd400b8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 --2- 172.21.15.181:0/1774398468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd400b39c0 0x7fbd400c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.613 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 >> 172.21.15.181:0/1774398468 conn(0x7fbd4009d670 msgr2=0x7fbd400b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.614 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 shutdown_connections 2024-09-17T03:49:12.614 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.613+0000 7fbd45e00640 1 -- 172.21.15.181:0/1774398468 wait complete. 2024-09-17T03:49:12.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new cfc14b39-9ae7-49fa-9a0f-d3c95849cf87 5 2024-09-17T03:49:12.855 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 -- 172.21.15.181:0/2666286077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 msgr2=0x7f04ec0d0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 --2- 172.21.15.181:0/2666286077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec0d0de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f04dc0098e0 tx=0x7f04dc02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 -- 172.21.15.181:0/2666286077 shutdown_connections 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 --2- 172.21.15.181:0/2666286077 >> v2:172.21.15.129:3300/0 conn(0x7f04ec0d1320 0x7f04ec0cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 --2- 172.21.15.181:0/2666286077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec0d0de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 -- 172.21.15.181:0/2666286077 >> 172.21.15.181:0/2666286077 conn(0x7f04ec09d680 msgr2=0x7f04ec0d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 -- 172.21.15.181:0/2666286077 shutdown_connections 2024-09-17T03:49:12.856 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.855+0000 7f04f1600640 1 -- 172.21.15.181:0/2666286077 wait complete. 2024-09-17T03:49:12.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.858+0000 7f04f1600640 1 Processor -- start 2024-09-17T03:49:12.857 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.858+0000 7f04f1600640 1 -- start start 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04ec0cf510 0x7f04ec1650f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04ec0d1320 0x7f04ec0b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0b4a00 con 0x7f04ec0d1320 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0b50d0 con 0x7f04ec0cf510 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04f1600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec0b4b80 con 0x7f04ec0b96b0 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04eb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04ec0cf510 0x7f04ec1650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51208/0 (socket says 172.21.15.181:51208) 2024-09-17T03:49:12.860 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.859+0000 7f04ebe00640 1 -- 172.21.15.181:0/2118447344 learned_addr learned my addr 172.21.15.181:0/2118447344 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04f0c00640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04ec0d1320 0x7f04ec0b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 -- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04ec0cf510 msgr2=0x7f04ec1650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04ec0cf510 0x7f04ec1650f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 -- 172.21.15.181:0/2118447344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04ec0d1320 msgr2=0x7f04ec0b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04ec0d1320 0x7f04ec0b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 -- 172.21.15.181:0/2118447344 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec1f8c20 con 0x7f04ec0b96b0 2024-09-17T03:49:12.861 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04ebe00640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009a00 tx=0x7f04dc0316b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:12.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.860+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc03f070 con 0x7f04ec0b96b0 2024-09-17T03:49:12.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04dc031950 con 0x7f04ec0b96b0 2024-09-17T03:49:12.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc031da0 con 0x7f04ec0b96b0 2024-09-17T03:49:12.862 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec0b1410 con 0x7f04ec0b96b0 2024-09-17T03:49:12.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec0b19d0 con 0x7f04ec0b96b0 2024-09-17T03:49:12.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f04dc042070 con 0x7f04ec0b96b0 2024-09-17T03:49:12.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.861+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f04dc0047a0 con 0x7f04ec0b96b0 2024-09-17T03:49:12.863 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.862+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec0d0de0 con 0x7f04ec0b96b0 2024-09-17T03:49:12.864 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.864+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f04dc004a40 con 0x7f04ec0b96b0 2024-09-17T03:49:12.937 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.937+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cfc14b39-9ae7-49fa-9a0f-d3c95849cf87", "id": 5} v 0) -- 0x7f04ec1576c0 con 0x7f04ec0b96b0 2024-09-17T03:49:12.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.943+0000 7f04e3e00640 1 -- 172.21.15.181:0/2118447344 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfc14b39-9ae7-49fa-9a0f-d3c95849cf87", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f04dc004c20 con 0x7f04ec0b96b0 2024-09-17T03:49:12.943 INFO:teuthology.orchestra.run.smithi181.stdout:5 2024-09-17T03:49:12.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 msgr2=0x7f04ec157180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:12.944 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009a00 tx=0x7f04dc0316b0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 shutdown_connections 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04ec0d1320 0x7f04ec0b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04ec0cf510 0x7f04ec1650f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 --2- 172.21.15.181:0/2118447344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04ec0b96b0 0x7f04ec157180 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 >> 172.21.15.181:0/2118447344 conn(0x7f04ec09d680 msgr2=0x7f04ec0ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 shutdown_connections 2024-09-17T03:49:12.945 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:12.944+0000 7f04f1600640 1 -- 172.21.15.181:0/2118447344 wait complete. 2024-09-17T03:49:12.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 585d026e-6038-4ac2-a15b-440858a24e5b 6 2024-09-17T03:49:13.179 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1831375315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00d4ee0 msgr2=0x7f7eb00b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.179 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1831375315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00d4ee0 0x7f7eb00b3660 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009890 tx=0x7f7e9c02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1831375315 shutdown_connections 2024-09-17T03:49:13.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1831375315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00d4ee0 0x7f7eb00b3660 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1831375315 >> 172.21.15.181:0/1831375315 conn(0x7f7eb009d680 msgr2=0x7f7eb00d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:13.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1831375315 shutdown_connections 2024-09-17T03:49:13.180 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.179+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1831375315 wait complete. 2024-09-17T03:49:13.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.181+0000 7f7eb5e00640 1 Processor -- start 2024-09-17T03:49:13.182 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.181+0000 7f7eb5e00640 1 -- start start 2024-09-17T03:49:13.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 0x7f7eb00ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.184 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 0x7f7eb01f8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb00b4030 con 0x7f7eb00d0bb0 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb00b3eb0 con 0x7f7eb00d4ee0 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb00b41b0 con 0x7f7eb00b9540 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 0x7f7eb01f8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51220/0 (socket says 172.21.15.181:51220) 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eb4a00640 1 -- 172.21.15.181:0/1323978721 learned_addr learned my addr 172.21.15.181:0/1323978721 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.183+0000 7f7eafe00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 0x7f7eb00ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 -- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 msgr2=0x7f7eb01f8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 0x7f7eb01f8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 -- 172.21.15.181:0/1323978721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 msgr2=0x7f7eb00ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.185 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 0x7f7eb00ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 -- 172.21.15.181:0/1323978721 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb00cea00 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb5400640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 0x7f7eb01f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eafe00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 0x7f7eb00ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb4a00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c390 tx=0x7f7ea4001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4017070 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ea40035f0 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4003ac0 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb015a780 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.184+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb015ad40 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.185+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7ea401a070 con 0x7f7eb00b9540 2024-09-17T03:49:13.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.185+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f7ea4003c60 con 0x7f7eb00b9540 2024-09-17T03:49:13.187 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.185+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb00b3660 con 0x7f7eb00b9540 2024-09-17T03:49:13.187 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.188+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f7ea4021020 con 0x7f7eb00b9540 2024-09-17T03:49:13.259 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.259+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "585d026e-6038-4ac2-a15b-440858a24e5b", "id": 6} v 0) -- 0x7f7eb00d2650 con 0x7f7eb00b9540 2024-09-17T03:49:13.265 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.265+0000 7f7eacc00640 1 -- 172.21.15.181:0/1323978721 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "585d026e-6038-4ac2-a15b-440858a24e5b", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f7ea401c440 con 0x7f7eb00b9540 2024-09-17T03:49:13.265 INFO:teuthology.orchestra.run.smithi181.stdout:6 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 msgr2=0x7f7eb0162030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c390 tx=0x7f7ea4001e70 comp rx=0 tx=0).stop 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 shutdown_connections 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7eb00d4ee0 0x7f7eb01f8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7eb00d0bb0 0x7f7eb00ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 --2- 172.21.15.181:0/1323978721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7eb00b9540 0x7f7eb0162030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 >> 172.21.15.181:0/1323978721 conn(0x7f7eb009d680 msgr2=0x7f7eb0162e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 shutdown_connections 2024-09-17T03:49:13.267 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.266+0000 7f7eb5e00640 1 -- 172.21.15.181:0/1323978721 wait complete. 2024-09-17T03:49:13.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 607f6c33-e19c-4f70-8f2e-f8468f4fb228 7 2024-09-17T03:49:13.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 -- 172.21.15.181:0/1305273792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 msgr2=0x7fe9c80cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.501 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1305273792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c80cfd10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4005fd0 tx=0x7fe9b402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 -- 172.21.15.181:0/1305273792 shutdown_connections 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1305273792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c80cfd10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1305273792 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c80d4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 -- 172.21.15.181:0/1305273792 >> 172.21.15.181:0/1305273792 conn(0x7fe9c809d680 msgr2=0x7fe9c80d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 -- 172.21.15.181:0/1305273792 shutdown_connections 2024-09-17T03:49:13.502 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.501+0000 7fe9cd000640 1 -- 172.21.15.181:0/1305273792 wait complete. 2024-09-17T03:49:13.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.504+0000 7fe9cd000640 1 Processor -- start 2024-09-17T03:49:13.504 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.505+0000 7fe9cd000640 1 -- start start 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c81507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c80d0250 0x7fe9c81607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c80b3ca0 con 0x7fe9c80d0250 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c80b3e20 con 0x7fe9c80b3660 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9cd000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c80b3fa0 con 0x7fe9c80b9540 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9c6a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9c6a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.181:51240/0 (socket says 172.21.15.181:51240) 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9c6a00640 1 -- 172.21.15.181:0/1938477693 learned_addr learned my addr 172.21.15.181:0/1938477693 (peer_addr_for_me v2:172.21.15.181:0/0) 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.507+0000 7fe9c7400640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c81507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.508 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c7e00640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c80d0250 0x7fe9c81607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 -- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 msgr2=0x7fe9c81507a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c81507a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 -- 172.21.15.181:0/1938477693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c80d0250 msgr2=0x7fe9c81607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c80d0250 0x7fe9c81607c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.509 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 -- 172.21.15.181:0/1938477693 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c815f4c0 con 0x7fe9c80b9540 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9c6a00640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034040 tx=0x7fe9b40035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.508+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b403e030 con 0x7fe9c80b9540 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9b4003ae0 con 0x7fe9c80b9540 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c815f7a0 con 0x7fe9c80b9540 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4035040 con 0x7fe9c80b9540 2024-09-17T03:49:13.510 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c80cb8a0 con 0x7fe9c80b9540 2024-09-17T03:49:13.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe9b4041030 con 0x7fe9c80b9540 2024-09-17T03:49:13.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.509+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fe9b40435a0 con 0x7fe9c80b9540 2024-09-17T03:49:13.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.510+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c80b44c0 con 0x7fe9c80b9540 2024-09-17T03:49:13.512 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.512+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe9b4004b30 con 0x7fe9c80b9540 2024-09-17T03:49:13.595 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.595+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd new", "uuid": "607f6c33-e19c-4f70-8f2e-f8468f4fb228", "id": 7} v 0) -- 0x7fe9c80698c0 con 0x7fe9c80b9540 2024-09-17T03:49:13.599 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.600+0000 7fe9bf400640 1 -- 172.21.15.181:0/1938477693 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "607f6c33-e19c-4f70-8f2e-f8468f4fb228", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fe9b4004d10 con 0x7fe9c80b9540 2024-09-17T03:49:13.600 INFO:teuthology.orchestra.run.smithi181.stdout:7 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 msgr2=0x7fe9c8160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034040 tx=0x7fe9b40035f0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 shutdown_connections 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c80d0250 0x7fe9c81607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c80b9540 0x7fe9c8160280 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 --2- 172.21.15.181:0/1938477693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c80b3660 0x7fe9c81507a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 >> 172.21.15.181:0/1938477693 conn(0x7fe9c809d680 msgr2=0x7fe9c80b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:13.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 shutdown_connections 2024-09-17T03:49:13.602 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:13.601+0000 7fe9cd000640 1 -- 172.21.15.181:0/1938477693 wait complete. 2024-09-17T03:49:13.621 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-17T03:49:13.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-17T03:49:13.625 INFO:tasks.ceph.osd.0:Started 2024-09-17T03:49:13.626 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-17T03:49:13.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-17T03:49:13.629 INFO:tasks.ceph.osd.1:Started 2024-09-17T03:49:13.629 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-17T03:49:13.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-17T03:49:13.633 INFO:tasks.ceph.osd.2:Started 2024-09-17T03:49:13.633 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-17T03:49:13.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-17T03:49:13.638 INFO:tasks.ceph.osd.3:Started 2024-09-17T03:49:13.639 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-17T03:49:13.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-17T03:49:13.642 INFO:tasks.ceph.osd.4:Started 2024-09-17T03:49:13.643 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-17T03:49:13.643 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-17T03:49:13.646 INFO:tasks.ceph.osd.5:Started 2024-09-17T03:49:13.646 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-17T03:49:13.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-17T03:49:13.651 INFO:tasks.ceph.osd.6:Started 2024-09-17T03:49:13.651 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-17T03:49:13.651 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-17T03:49:13.655 INFO:tasks.ceph.osd.7:Started 2024-09-17T03:49:13.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:13.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 -- 172.21.15.129:0/1625752726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d0bb0 msgr2=0x7fa1f40d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/1625752726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d0bb0 0x7fa1f40d0fc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009be0 tx=0x7fa1ec02a920 comp rx=0 tx=0).stop 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 -- 172.21.15.129:0/1625752726 shutdown_connections 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/1625752726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d0bb0 0x7fa1f40d0fc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 -- 172.21.15.129:0/1625752726 >> 172.21.15.129:0/1625752726 conn(0x7fa1f409d680 msgr2=0x7fa1f40d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 -- 172.21.15.129:0/1625752726 shutdown_connections 2024-09-17T03:49:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.934+0000 7fa1f8a00640 1 -- 172.21.15.129:0/1625752726 wait complete. 2024-09-17T03:49:13.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.936+0000 7fa1f8a00640 1 Processor -- start 2024-09-17T03:49:13.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.936+0000 7fa1f8a00640 1 -- start start 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1f40b9540 0x7fa1f40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d4ee0 0x7fa1f40b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f415f100 con 0x7fa1f40b9540 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f415ef80 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f40b3ca0 con 0x7fa1f40d4ee0 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:57142/0 (socket says 172.21.15.129:57142) 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f2a00640 1 -- 172.21.15.129:0/2157370118 learned_addr learned my addr 172.21.15.129:0/2157370118 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f3400640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1f40b9540 0x7fa1f40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.938+0000 7fa1f3e00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d4ee0 0x7fa1f40b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 -- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d4ee0 msgr2=0x7fa1f40b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d4ee0 0x7fa1f40b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 -- 172.21.15.129:0/2157370118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1f40b9540 msgr2=0x7fa1f40caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1f40b9540 0x7fa1f40caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 -- 172.21.15.129:0/2157370118 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f41faca0 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000c400 tx=0x7fa1e000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0017070 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e00053c0 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0003850 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f40c9ce0 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.939+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f40ca2a0 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.940+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa1e001a070 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.940+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa1e0011c70 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.941+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1f41faa10 con 0x7fa1f40d0bb0 2024-09-17T03:49:13.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:13.943+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fa1e0021070 con 0x7fa1f40d0bb0 2024-09-17T03:49:14.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.006+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1f40d0fc0 con 0x7fa1f40d0bb0 2024-09-17T03:49:14.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.007+0000 7fa1eb400640 1 -- 172.21.15.129:0/2157370118 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa1e0011e70 con 0x7fa1f40d0bb0 2024-09-17T03:49:14.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:14.008 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":10,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:13.598108+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 msgr2=0x7fa1f40b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000c400 tx=0x7fa1e000c8c0 comp rx=0 tx=0).stop 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 shutdown_connections 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1f40d4ee0 0x7fa1f40b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1f40d0bb0 0x7fa1f40b51c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 --2- 172.21.15.129:0/2157370118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1f40b9540 0x7fa1f40caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.008+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 >> 172.21.15.129:0/2157370118 conn(0x7fa1f409d680 msgr2=0x7fa1f415e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.009+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 shutdown_connections 2024-09-17T03:49:14.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:14.009+0000 7fa1f8a00640 1 -- 172.21.15.129:0/2157370118 wait complete. 2024-09-17T03:49:14.028 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-17T03:49:14.029 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T03:49:14.835 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:15.035 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 -- 172.21.15.129:0/82210645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0800cc270 msgr2=0x7fb0800cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 --2- 172.21.15.129:0/82210645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0800cc270 0x7fb0800cf4f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb07800acf0 tx=0x7fb07802bcd0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 -- 172.21.15.129:0/82210645 shutdown_connections 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 --2- 172.21.15.129:0/82210645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0800cc270 0x7fb0800cf4f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 -- 172.21.15.129:0/82210645 >> 172.21.15.129:0/82210645 conn(0x7fb0800d0a30 msgr2=0x7fb0800b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 -- 172.21.15.129:0/82210645 shutdown_connections 2024-09-17T03:49:15.036 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.035+0000 7fb086400640 1 -- 172.21.15.129:0/82210645 wait complete. 2024-09-17T03:49:15.037 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.037+0000 7fb086400640 1 Processor -- start 2024-09-17T03:49:15.037 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.038+0000 7fb086400640 1 -- start start 2024-09-17T03:49:15.040 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:15.040 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 0x7fb08015b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:15.040 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb080154660 0x7fb08015b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0800caac0 con 0x7fb0800cc270 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0800ca940 con 0x7fb08015aec0 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb086400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb0800cac40 con 0x7fb080154660 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb07f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb07f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46696/0 (socket says 172.21.15.129:46696) 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.039+0000 7fb07f400640 1 -- 172.21.15.129:0/3065785083 learned_addr learned my addr 172.21.15.129:0/3065785083 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb076a00640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 0x7fb08015b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07fe00640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb080154660 0x7fb08015b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 -- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb080154660 msgr2=0x7fb08015b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb080154660 0x7fb08015b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 -- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 msgr2=0x7fb08015b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:15.041 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 0x7fb08015b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 -- 172.21.15.129:0/3065785083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0800b20c0 con 0x7fb0800cc270 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb076a00640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 0x7fb08015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb07f400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb07000c460 tx=0x7fb07000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0700052d0 con 0x7fb0800cc270 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb070005490 con 0x7fb0800cc270 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070012870 con 0x7fb0800cc270 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0800c8150 con 0x7fb0800cc270 2024-09-17T03:49:15.042 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.040+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0801f55e0 con 0x7fb0800cc270 2024-09-17T03:49:15.043 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.042+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0800cf4f0 con 0x7fb0800cc270 2024-09-17T03:49:15.043 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.042+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 2) ==== 50742+0+0 (secure 0 0 0) 0x7fb070012a10 con 0x7fb0800cc270 2024-09-17T03:49:15.043 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.042+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fb070051ee0 con 0x7fb0800cc270 2024-09-17T03:49:15.044 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.044+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fb070003d50 con 0x7fb0800cc270 2024-09-17T03:49:15.107 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.107+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb080073550 con 0x7fb0800cc270 2024-09-17T03:49:15.108 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.108+0000 7fb077e00640 1 -- 172.21.15.129:0/3065785083 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fb070010cf0 con 0x7fb0800cc270 2024-09-17T03:49:15.108 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-17T03:49:15.108 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":10,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:13.598108+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 msgr2=0x7fb0801f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb07000c460 tx=0x7fb07000c920 comp rx=0 tx=0).stop 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 shutdown_connections 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb080154660 0x7fb08015b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb08015aec0 0x7fb08015b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.109 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 --2- 172.21.15.129:0/3065785083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0800cc270 0x7fb0801f50a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:15.110 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 >> 172.21.15.129:0/3065785083 conn(0x7fb0800d0a30 msgr2=0x7fb08006cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:15.110 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 shutdown_connections 2024-09-17T03:49:15.110 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:15.109+0000 7fb086400640 1 -- 172.21.15.129:0/3065785083 wait complete. 2024-09-17T03:49:15.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-17T03:49:15.239 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-17T03:49:15.239+0000 7ff80ed02640 -1 Falling back to public interface 2024-09-17T03:49:15.279 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-09-17T03:49:15.279+0000 7fe206224640 -1 Falling back to public interface 2024-09-17T03:49:15.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:49:15.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:49:15.301 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-17T03:49:15.299+0000 7f8b5fc00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T03:49:15.301 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-17T03:49:15.300+0000 7f8b5fc00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T03:49:15.325 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-17T03:49:15.325+0000 7f4475416640 -1 Falling back to public interface 2024-09-17T03:49:15.327 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-17T03:49:15.327+0000 7f8d92a24640 -1 Falling back to public interface 2024-09-17T03:49:15.330 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-17T03:49:15.331+0000 7f7452260640 -1 Falling back to public interface 2024-09-17T03:49:15.331 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-17T03:49:15.331+0000 7ff0299b9640 -1 Falling back to public interface 2024-09-17T03:49:15.339 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-17T03:49:15.339+0000 7f5ef7cf1640 -1 Falling back to public interface 2024-09-17T03:49:15.365 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T03:49:15.365+0000 7f36f721d640 -1 Falling back to public interface 2024-09-17T03:49:19.229 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-09-17T03:49:19.229+0000 7fe206224640 -1 osd.6 0 log_to_monitors true 2024-09-17T03:49:19.254 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-17T03:49:19.254+0000 7f5ef7cf1640 -1 osd.0 0 log_to_monitors true 2024-09-17T03:49:19.371 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-17T03:49:19.371+0000 7ff80ed02640 -1 osd.4 0 log_to_monitors true 2024-09-17T03:49:19.479 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-17T03:49:19.479+0000 7f4475416640 -1 osd.1 0 log_to_monitors true 2024-09-17T03:49:19.516 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-17T03:49:19.516+0000 7f8d92a24640 -1 osd.5 0 log_to_monitors true 2024-09-17T03:49:19.552 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-17T03:49:19.552+0000 7ff0299b9640 -1 osd.2 0 log_to_monitors true 2024-09-17T03:49:19.562 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-17T03:49:19.562+0000 7f7452260640 -1 osd.3 0 log_to_monitors true 2024-09-17T03:49:19.798 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T03:49:19.798+0000 7f36f721d640 -1 osd.7 0 log_to_monitors true 2024-09-17T03:49:21.317 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-17T03:49:21.317+0000 7f5ef2c00640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T03:49:21.367 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-09-17T03:49:21.367+0000 7fe200a00640 -1 osd.6 0 waiting for initial osdmap 2024-09-17T03:49:21.477 INFO:tasks.ceph.osd.0.smithi129.stderr:2024-09-17T03:49:21.477+0000 7f5ee8c00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:21.510 INFO:tasks.ceph.osd.6.smithi181.stderr:2024-09-17T03:49:21.510+0000 7fe1f6a00640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:21.934 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:22.181 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 -- 172.21.15.129:0/3684758825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7e8004040 msgr2=0x7fe7e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:22.181 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 --2- 172.21.15.129:0/3684758825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7e8004040 0x7fe7e8004800 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00c2c0 tx=0x7fe7e8005830 comp rx=0 tx=0).stop 2024-09-17T03:49:22.182 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805400640 1 -- 172.21.15.129:0/3684758825 reap_dead start 2024-09-17T03:49:22.182 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 -- 172.21.15.129:0/3684758825 shutdown_connections 2024-09-17T03:49:22.182 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 -- 172.21.15.129:0/3684758825 >> 172.21.15.129:0/3684758825 conn(0x7fe80006ddc0 msgr2=0x7fe80006e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:22.182 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 -- 172.21.15.129:0/3684758825 shutdown_connections 2024-09-17T03:49:22.182 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.181+0000 7fe805e00640 1 -- 172.21.15.129:0/3684758825 wait complete. 2024-09-17T03:49:22.185 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.185+0000 7fe805e00640 1 Processor -- start 2024-09-17T03:49:22.185 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.185+0000 7fe805e00640 1 -- start start 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 0x7fe800153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 0x7fe8000b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe80015e940 con 0x7fe8000b51b0 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe80015e7c0 con 0x7fe800153950 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.186+0000 7fe805e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe80015eac0 con 0x7fe7e8004040 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.187+0000 7fe804a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.187+0000 7fe805400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 0x7fe8000b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:22.188 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.187+0000 7fe804a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:36172/0 (socket says 172.21.15.129:36172) 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.187+0000 7fe804a00640 1 -- 172.21.15.129:0/1902412394 learned_addr learned my addr 172.21.15.129:0/1902412394 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.187+0000 7fe7ffe00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 0x7fe800153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 -- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 msgr2=0x7fe800157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 -- 172.21.15.129:0/1902412394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 msgr2=0x7fe8000b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 0x7fe8000b55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 -- 172.21.15.129:0/1902412394 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000b5b00 con 0x7fe800153950 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe804a00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:22.189 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe805400640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 0x7fe8000b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.188+0000 7fe7ffe00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 0x7fe800153120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800bba0 tx=0x7fe7ec001d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.189+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec031040 con 0x7fe800153950 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.189+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe7ec003800 con 0x7fe800153950 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.189+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8001f9140 con 0x7fe800153950 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.189+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec006600 con 0x7fe800153950 2024-09-17T03:49:22.190 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.189+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8001f9700 con 0x7fe800153950 2024-09-17T03:49:22.191 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.190+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8000d2a10 con 0x7fe800153950 2024-09-17T03:49:22.191 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.192+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 4) ==== 101385+0+0 (secure 0 0 0) 0x7fe7ec034070 con 0x7fe800153950 2024-09-17T03:49:22.192 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.193+0000 7fe7fcc00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 0x7fe7e007b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:22.193 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.193+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2231+0+0 (secure 0 0 0) 0x7fe7ec0ba250 con 0x7fe800153950 2024-09-17T03:49:22.193 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.193+0000 7fe804a00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 0x7fe7e007b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:22.193 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.194+0000 7fe804a00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 0x7fe7e007b010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4012560 tx=0x7fe7f4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:22.198 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.198+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe7ec045060 con 0x7fe800153950 2024-09-17T03:49:22.482 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.482+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe8000b6c80 con 0x7fe800153950 2024-09-17T03:49:22.483 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.484+0000 7fe7fcc00640 1 -- 172.21.15.129:0/1902412394 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+8053 (secure 0 0 0) 0x7fe8000b5b00 con 0x7fe800153950 2024-09-17T03:49:22.484 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-17T03:49:22.484 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":14,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:22.407806+0000","last_up_change":"2024-09-17T03:49:22.407806+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:22.488 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 msgr2=0x7fe7e007b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:22.488 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 0x7fe7e007b010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7f4012560 tx=0x7fe7f4005f90 comp rx=0 tx=0).stop 2024-09-17T03:49:22.488 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 msgr2=0x7fe800153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:22.488 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 0x7fe800153120 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800bba0 tx=0x7fe7ec001d20 comp rx=0 tx=0).stop 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 shutdown_connections 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe7e0078b00 0x7fe7e007b010 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8000b51b0 0x7fe8000b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe800153950 0x7fe800153120 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 --2- 172.21.15.129:0/1902412394 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7e8004040 0x7fe800157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.488+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 >> 172.21.15.129:0/1902412394 conn(0x7fe80006ddc0 msgr2=0x7fe8000d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.489+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 shutdown_connections 2024-09-17T03:49:22.489 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:22.489+0000 7fe805e00640 1 -- 172.21.15.129:0/1902412394 wait complete. 2024-09-17T03:49:22.523 DEBUG:teuthology.misc:2 of 8 OSDs are up 2024-09-17T03:49:23.164 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-17T03:49:23.164+0000 7f744dc00640 -1 osd.3 0 waiting for initial osdmap 2024-09-17T03:49:23.164 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-17T03:49:23.164+0000 7ff025200640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T03:49:23.171 INFO:tasks.ceph.osd.2.smithi129.stderr:2024-09-17T03:49:23.171+0000 7ff01a800640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:23.172 INFO:tasks.ceph.osd.3.smithi129.stderr:2024-09-17T03:49:23.172+0000 7f7443200640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:23.196 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-17T03:49:23.196+0000 7f446fe00640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T03:49:23.202 INFO:tasks.ceph.osd.1.smithi129.stderr:2024-09-17T03:49:23.202+0000 7f4465e00640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:23.225 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T03:49:23.225+0000 7f36f1c00640 -1 osd.7 0 waiting for initial osdmap 2024-09-17T03:49:23.229 INFO:tasks.ceph.osd.7.smithi181.stderr:2024-09-17T03:49:23.229+0000 7f36e7c00640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:23.231 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-17T03:49:23.232+0000 7ff80a600640 -1 osd.4 0 waiting for initial osdmap 2024-09-17T03:49:23.243 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-17T03:49:23.244+0000 7f8d8e200640 -1 osd.5 0 waiting for initial osdmap 2024-09-17T03:49:23.245 INFO:tasks.ceph.osd.4.smithi181.stderr:2024-09-17T03:49:23.245+0000 7ff7ffc00640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:23.251 INFO:tasks.ceph.osd.5.smithi181.stderr:2024-09-17T03:49:23.251+0000 7f8d83800640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T03:49:29.331 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.540+0000 7fa1f2a00640 1 -- 172.21.15.129:0/83249927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d40051e0 msgr2=0x7fa1d400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.540+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/83249927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d40051e0 0x7fa1d400b9c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4005fd0 tx=0x7fa1d4005830 comp rx=0 tx=0).stop 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.540+0000 7fa1f2000640 1 -- 172.21.15.129:0/83249927 reap_dead start 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.540+0000 7fa1f2a00640 1 -- 172.21.15.129:0/83249927 shutdown_connections 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.540+0000 7fa1f2a00640 1 -- 172.21.15.129:0/83249927 >> 172.21.15.129:0/83249927 conn(0x7fa1ec06cde0 msgr2=0x7fa1ec06d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.541+0000 7fa1f2a00640 1 -- 172.21.15.129:0/83249927 shutdown_connections 2024-09-17T03:49:29.541 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.541+0000 7fa1f2a00640 1 -- 172.21.15.129:0/83249927 wait complete. 2024-09-17T03:49:29.543 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.543+0000 7fa1f2a00640 1 Processor -- start 2024-09-17T03:49:29.543 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.543+0000 7fa1f2a00640 1 -- start start 2024-09-17T03:49:29.545 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.544+0000 7fa1f2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d40051e0 0x7fa1ec1f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:29.545 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.544+0000 7fa1f2a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:29.545 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:29.545 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec15f7f0 con 0x7fa1ec0afb90 2024-09-17T03:49:29.545 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ec15f670 con 0x7fa1d40051e0 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ec06eb50 con 0x7fa1ec15e5d0 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f1600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d40051e0 0x7fa1ec1f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48100/0 (socket says 172.21.15.129:48100) 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 -- 172.21.15.129:0/3343675596 learned_addr learned my addr 172.21.15.129:0/3343675596 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 -- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1ec15e5d0 msgr2=0x7fa1ec0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 -- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d40051e0 msgr2=0x7fa1ec1f4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d40051e0 0x7fa1ec1f4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.546 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.545+0000 7fa1f2000640 1 -- 172.21.15.129:0/3343675596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ec1f4a70 con 0x7fa1ec0afb90 2024-09-17T03:49:29.547 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1f2000640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4036040 tx=0x7fa1e4033bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:29.547 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4042070 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1e4003970 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1ec15a870 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4038040 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.546+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1ec0ae430 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.547+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1ec155d50 con 0x7fa1ec0afb90 2024-09-17T03:49:29.548 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.548+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa1e4004030 con 0x7fa1ec0afb90 2024-09-17T03:49:29.549 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.550+0000 7fa1e9600640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 0x7fa1cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:29.549 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.550+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fa1e40bac00 con 0x7fa1ec0afb90 2024-09-17T03:49:29.550 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.550+0000 7fa1f1600640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 0x7fa1cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:29.550 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.551+0000 7fa1f1600640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 0x7fa1cc07b0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0001050 tx=0x7fa1e00011e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:29.555 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.555+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa1ec155d50 con 0x7fa1ec0afb90 2024-09-17T03:49:29.737 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.737+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1ec0b4390 con 0x7fa1ec0afb90 2024-09-17T03:49:29.738 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.738+0000 7fa1e9600640 1 -- 172.21.15.129:0/3343675596 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+12944 (secure 0 0 0) 0x7fa1e4082ca0 con 0x7fa1ec0afb90 2024-09-17T03:49:29.738 INFO:teuthology.misc.health.smithi129.stdout: 2024-09-17T03:49:29.738 INFO:teuthology.misc.health.smithi129.stdout:{"epoch":18,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:27.179911+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.740+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 msgr2=0x7fa1cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.740+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 0x7fa1cc07b0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0001050 tx=0x7fa1e00011e0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.740+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 msgr2=0x7fa1ec1f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.740+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4036040 tx=0x7fa1e4033bf0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.740+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 shutdown_connections 2024-09-17T03:49:29.741 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1cc078bc0 0x7fa1cc07b0d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1ec0afb90 0x7fa1ec1f8210 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1ec15e5d0 0x7fa1ec0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 --2- 172.21.15.129:0/3343675596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d40051e0 0x7fa1ec1f4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 >> 172.21.15.129:0/3343675596 conn(0x7fa1ec06cde0 msgr2=0x7fa1ec14bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 shutdown_connections 2024-09-17T03:49:29.742 INFO:teuthology.misc.health.smithi129.stderr:2024-09-17T03:49:29.741+0000 7fa1f2a00640 1 -- 172.21.15.129:0/3343675596 wait complete. 2024-09-17T03:49:29.760 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-17T03:49:29.760 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-17T03:49:29.760 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:49:29.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:49:29.764 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:49:29.764 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:49:29.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:49:29.768 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:49:29.768 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:49:29.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:49:29.773 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:49:29.773 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:49:29.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:49:29.777 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:49:29.777 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-17T03:49:29.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:29.960 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:49:29.960 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:49:29.978 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:49:29.979 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:49:29.979 INFO:tasks.ceph.mds.b.smithi181.stderr:2024-09-17T03:49:29.980+0000 7fdb5d000640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-17T03:49:30.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 -- 172.21.15.129:0/2168406318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf540 msgr2=0x7fc57406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 --2- 172.21.15.129:0/2168406318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf540 0x7fc57406ee70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc5740ce710 tx=0x7fc56002a4b0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 -- 172.21.15.129:0/2168406318 shutdown_connections 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 --2- 172.21.15.129:0/2168406318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf540 0x7fc57406ee70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 --2- 172.21.15.129:0/2168406318 >> v2:172.21.15.181:3300/0 conn(0x7fc57415ca50 0x7fc5740cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 -- 172.21.15.129:0/2168406318 >> 172.21.15.129:0/2168406318 conn(0x7fc57406d410 msgr2=0x7fc5740b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 -- 172.21.15.129:0/2168406318 shutdown_connections 2024-09-17T03:49:30.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.011+0000 7fc578a00640 1 -- 172.21.15.129:0/2168406318 wait complete. 2024-09-17T03:49:30.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.013+0000 7fc578a00640 1 Processor -- start 2024-09-17T03:49:30.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.013+0000 7fc578a00640 1 -- start start 2024-09-17T03:49:30.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5740cf540 0x7fc5740cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc57415ca50 0x7fc574143b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc57415de10 con 0x7fc5740cf950 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc57406f740 con 0x7fc5740cf540 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc578a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc57415df90 con 0x7fc57415ca50 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48136/0 (socket says 172.21.15.129:48136) 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 -- 172.21.15.129:0/634194264 learned_addr learned my addr 172.21.15.129:0/634194264 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc572800640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5740cf540 0x7fc5740cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc573200640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc57415ca50 0x7fc574143b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 -- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc57415ca50 msgr2=0x7fc574143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:30.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc57415ca50 0x7fc574143b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 -- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5740cf540 msgr2=0x7fc5740cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5740cf540 0x7fc5740cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.015+0000 7fc571e00640 1 -- 172.21.15.129:0/634194264 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc574157ea0 con 0x7fc5740cf950 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc571e00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc560000c00 tx=0x7fc5600039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56003e030 con 0x7fc5740cf950 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5600047a0 con 0x7fc5740cf950 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc574157340 con 0x7fc5740cf950 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560035040 con 0x7fc5740cf950 2024-09-17T03:49:30.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.016+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741f9d20 con 0x7fc5740cf950 2024-09-17T03:49:30.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.017+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57415ec00 con 0x7fc5740cf950 2024-09-17T03:49:30.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.018+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc560041030 con 0x7fc5740cf950 2024-09-17T03:49:30.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.019+0000 7fc56aa00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 0x7fc55407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:30.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.019+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fc5600bb290 con 0x7fc5740cf950 2024-09-17T03:49:30.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.020+0000 7fc572800640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 0x7fc55407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:30.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.020+0000 7fc572800640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 0x7fc55407b0a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc56400a2e0 tx=0x7fc56400a270 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:30.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.024+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc560083a10 con 0x7fc5740cf950 2024-09-17T03:49:30.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.243+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc57406fb50 con 0x7fc5740cf950 2024-09-17T03:49:30.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.244+0000 7fc56aa00640 1 -- 172.21.15.129:0/634194264 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1437 (secure 0 0 0) 0x7fc560083310 con 0x7fc5740cf950 2024-09-17T03:49:30.244 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:30.244 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"btime":"2024-09-17T03:49:29:968792+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-09-17T03:49:30.245 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 2 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 msgr2=0x7fc55407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 0x7fc55407b0a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc56400a2e0 tx=0x7fc56400a270 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 msgr2=0x7fc574074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc560000c00 tx=0x7fc5600039e0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 shutdown_connections 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc554078b90 0x7fc55407b0a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc57415ca50 0x7fc574143b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5740cf950 0x7fc574074490 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 --2- 172.21.15.129:0/634194264 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5740cf540 0x7fc5740cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:30.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 >> 172.21.15.129:0/634194264 conn(0x7fc57406d410 msgr2=0x7fc574145690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:30.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 shutdown_connections 2024-09-17T03:49:30.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:30.246+0000 7fc578a00640 1 -- 172.21.15.129:0/634194264 wait complete. 2024-09-17T03:49:32.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 -- 172.21.15.129:0/3008142742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b80053a0 msgr2=0x7f62b800ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 --2- 172.21.15.129:0/3008142742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b80053a0 0x7f62b800ba50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0016c0 tx=0x7f62b8005830 comp rx=0 tx=0).stop 2024-09-17T03:49:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 -- 172.21.15.129:0/3008142742 shutdown_connections 2024-09-17T03:49:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 --2- 172.21.15.129:0/3008142742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62b80053a0 0x7f62b800ba50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 -- 172.21.15.129:0/3008142742 >> 172.21.15.129:0/3008142742 conn(0x7f62d40b3090 msgr2=0x7f62d40b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 -- 172.21.15.129:0/3008142742 shutdown_connections 2024-09-17T03:49:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.501+0000 7f62d8a00640 1 -- 172.21.15.129:0/3008142742 wait complete. 2024-09-17T03:49:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.503+0000 7f62d8a00640 1 Processor -- start 2024-09-17T03:49:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.503+0000 7f62d8a00640 1 -- start start 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62b80053a0 0x7f62d41525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62d414bfa0 0x7f62d414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4155940 con 0x7f62d414c3b0 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f62d41557c0 con 0x7f62d414bfa0 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f62d4155ac0 con 0x7f62b80053a0 2024-09-17T03:49:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48164/0 (socket says 172.21.15.129:48164) 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 -- 172.21.15.129:0/7732786 learned_addr learned my addr 172.21.15.129:0/7732786 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d2800640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62b80053a0 0x7f62d41525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d1e00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62d414bfa0 0x7f62d414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 -- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62b80053a0 msgr2=0x7f62d41525e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62b80053a0 0x7f62d41525e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 -- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62d414bfa0 msgr2=0x7f62d414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62d414bfa0 0x7f62d414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.505+0000 7f62d3200640 1 -- 172.21.15.129:0/7732786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d41503e0 con 0x7f62d414c3b0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62d3200640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62cc02a930 tx=0x7f62cc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc037070 con 0x7f62d414c3b0 2024-09-17T03:49:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62cc003ca0 con 0x7f62d414c3b0 2024-09-17T03:49:32.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc032670 con 0x7f62d414c3b0 2024-09-17T03:49:32.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d4152fc0 con 0x7f62d414c3b0 2024-09-17T03:49:32.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.506+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d4150980 con 0x7f62d414c3b0 2024-09-17T03:49:32.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.507+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d4152d30 con 0x7f62d414c3b0 2024-09-17T03:49:32.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.508+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f62cc030020 con 0x7f62d414c3b0 2024-09-17T03:49:32.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.509+0000 7f62caa00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 0x7f62b407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.509+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f62cc0b9ad0 con 0x7f62d414c3b0 2024-09-17T03:49:32.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.510+0000 7f62d2800640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 0x7f62b407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.510+0000 7f62d2800640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 0x7f62b407b0a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62c4001db0 tx=0x7f62c4005040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:32.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.514+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f62cc042020 con 0x7f62d414c3b0 2024-09-17T03:49:32.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.733+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62d415f020 con 0x7f62d414c3b0 2024-09-17T03:49:32.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.734+0000 7f62caa00640 1 -- 172.21.15.129:0/7732786 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3912 (secure 0 0 0) 0x7f62cc082280 con 0x7f62d414c3b0 2024-09-17T03:49:32.734 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:32.734 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"btime":"2024-09-17T03:49:30:970166+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4374,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-09-17T03:49:32.734 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 3 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 msgr2=0x7f62b407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 0x7f62b407b0a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f62c4001db0 tx=0x7f62c4005040 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 msgr2=0x7f62d414f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f62cc02a930 tx=0x7f62cc0038a0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 shutdown_connections 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62b4078b90 0x7f62b407b0a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62d414c3b0 0x7f62d414f740 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62d414bfa0 0x7f62d414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.736+0000 7f62d8a00640 1 --2- 172.21.15.129:0/7732786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62b80053a0 0x7f62d41525e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.737+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 >> 172.21.15.129:0/7732786 conn(0x7f62d40b3090 msgr2=0x7f62d40ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:32.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.737+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 shutdown_connections 2024-09-17T03:49:32.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.737+0000 7f62d8a00640 1 -- 172.21.15.129:0/7732786 wait complete. 2024-09-17T03:49:32.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:49:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 -- 172.21.15.129:0/1131357055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 msgr2=0x7f92380af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 --2- 172.21.15.129:0/1131357055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380af3c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9238072dc0 tx=0x7f922802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 -- 172.21.15.129:0/1131357055 shutdown_connections 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 --2- 172.21.15.129:0/1131357055 >> v2:172.21.15.181:3300/0 conn(0x7f92380ccb50 0x7f92380ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 --2- 172.21.15.129:0/1131357055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380af3c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 -- 172.21.15.129:0/1131357055 >> 172.21.15.129:0/1131357055 conn(0x7f923806d7d0 msgr2=0x7f92380b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 -- 172.21.15.129:0/1131357055 shutdown_connections 2024-09-17T03:49:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.978+0000 7f923f800640 1 -- 172.21.15.129:0/1131357055 wait complete. 2024-09-17T03:49:32.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.980+0000 7f923f800640 1 Processor -- start 2024-09-17T03:49:32.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.981+0000 7f923f800640 1 -- start start 2024-09-17T03:49:32.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92380ccb50 0x7f923815f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923815d8a0 0x7f923809cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92381533a0 con 0x7f92380cc1e0 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f923815d3e0 con 0x7f923815d8a0 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9238153520 con 0x7f92380ccb50 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48188/0 (socket says 172.21.15.129:48188) 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.982+0000 7f923ce00640 1 -- 172.21.15.129:0/3279780909 learned_addr learned my addr 172.21.15.129:0/3279780909 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923d800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923815d8a0 0x7f923809cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f922fe00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92380ccb50 0x7f923815f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 -- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92380ccb50 msgr2=0x7f923815f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92380ccb50 0x7f923815f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 -- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923815d8a0 msgr2=0x7f923809cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923815d8a0 0x7f923809cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 -- 172.21.15.129:0/3279780909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923815e1a0 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923ce00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92280099c0 tx=0x7f92280039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922803e030 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92280047a0 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9228035040 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.983+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923815e480 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.984+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923815ea40 con 0x7f92380cc1e0 2024-09-17T03:49:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.985+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92380ccf60 con 0x7f92380cc1e0 2024-09-17T03:49:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.986+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9228041030 con 0x7f92380cc1e0 2024-09-17T03:49:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.987+0000 7f9235600640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 0x7f921407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.987+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7f92280bb150 con 0x7f92380cc1e0 2024-09-17T03:49:32.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.987+0000 7f922fe00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 0x7f921407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:32.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.988+0000 7f922fe00640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 0x7f921407b0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9238160180 tx=0x7f9220009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:32.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:32.991+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92280838d0 con 0x7f92380cc1e0 2024-09-17T03:49:33.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.176+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f923815ee20 con 0x7f92380cc1e0 2024-09-17T03:49:33.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.176+0000 7f9235600640 1 -- 172.21.15.129:0/3279780909 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) ==== 84+0+65 (secure 0 0 0) 0x7f92280831d0 con 0x7f92380cc1e0 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:49:33.177 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:49:33.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 msgr2=0x7f921407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:33.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 0x7f921407b0d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9238160180 tx=0x7f9220009040 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 msgr2=0x7f92380c9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f92280099c0 tx=0x7f92280039e0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 shutdown_connections 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9214078bc0 0x7f921407b0d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923815d8a0 0x7f923809cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92380ccb50 0x7f923815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 --2- 172.21.15.129:0/3279780909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92380cc1e0 0x7f92380c9570 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 >> 172.21.15.129:0/3279780909 conn(0x7f923806d7d0 msgr2=0x7f92380b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 shutdown_connections 2024-09-17T03:49:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.179+0000 7f923f800640 1 -- 172.21.15.129:0/3279780909 wait complete. 2024-09-17T03:49:33.202 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:49:33.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 -- 172.21.15.129:0/831744859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 msgr2=0x7fc4c00afe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 --2- 172.21.15.129:0/831744859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c00afe50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00b5e10 tx=0x7fc4b002a580 comp rx=0 tx=0).stop 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 -- 172.21.15.129:0/831744859 shutdown_connections 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 --2- 172.21.15.129:0/831744859 >> v2:172.21.15.181:3300/0 conn(0x7fc4c006e500 0x7fc4c015cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 --2- 172.21.15.129:0/831744859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c00afe50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 -- 172.21.15.129:0/831744859 >> 172.21.15.129:0/831744859 conn(0x7fc4c0073640 msgr2=0x7fc4c0073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 -- 172.21.15.129:0/831744859 shutdown_connections 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.426+0000 7fc4bf200640 1 -- 172.21.15.129:0/831744859 wait complete. 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.428+0000 7fc4bf200640 1 Processor -- start 2024-09-17T03:49:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.428+0000 7fc4bf200640 1 -- start start 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 0x7fc4c01f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 0x7fc4c01f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c015d4f0 con 0x7fc4c0075fa0 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c015d370 con 0x7fc4c00a9e30 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c015d670 con 0x7fc4c00cdde0 2024-09-17T03:49:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48210/0 (socket says 172.21.15.129:48210) 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bde00640 1 -- 172.21.15.129:0/3463256033 learned_addr learned my addr 172.21.15.129:0/3463256033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4be800640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 0x7fc4c01f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.430+0000 7fc4bd400640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 0x7fc4c01f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 -- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 msgr2=0x7fc4c01f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 0x7fc4c01f6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 -- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 msgr2=0x7fc4c01f7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 0x7fc4c01f7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 -- 172.21.15.129:0/3463256033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c01591f0 con 0x7fc4c0075fa0 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4be800640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 0x7fc4c01f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bd400640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 0x7fc4c01f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bde00640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0034040 tx=0x7fc4b0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b003e030 con 0x7fc4c0075fa0 2024-09-17T03:49:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4b0003e00 con 0x7fc4c0075fa0 2024-09-17T03:49:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b0035040 con 0x7fc4c0075fa0 2024-09-17T03:49:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c0158430 con 0x7fc4c0075fa0 2024-09-17T03:49:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.431+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c01589f0 con 0x7fc4c0075fa0 2024-09-17T03:49:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.432+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c015db90 con 0x7fc4c0075fa0 2024-09-17T03:49:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.434+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc4b0041030 con 0x7fc4c0075fa0 2024-09-17T03:49:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.434+0000 7fc4b6000640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 0x7fc4a007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.435+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4857+0+0 (secure 0 0 0) 0x7fc4b00bb260 con 0x7fc4c0075fa0 2024-09-17T03:49:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.435+0000 7fc4bd400640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 0x7fc4a007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.436+0000 7fc4bd400640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 0x7fc4a007b0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c006deb0 tx=0x7fc4ac009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.439+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc4b0083950 con 0x7fc4c0075fa0 2024-09-17T03:49:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:33.620+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc4c006dc00 con 0x7fc4c0075fa0 2024-09-17T03:49:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.354+0000 7fc4b6000640 1 -- 172.21.15.129:0/3463256033 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) ==== 138+0+0 (secure 0 0 0) 0x7fc4b0083250 con 0x7fc4c0075fa0 2024-09-17T03:49:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:49:34.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 msgr2=0x7fc4a007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 0x7fc4a007b0d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc4c006deb0 tx=0x7fc4ac009040 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 msgr2=0x7fc4c014a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0034040 tx=0x7fc4b0003910 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 shutdown_connections 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4a0078bc0 0x7fc4a007b0d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc4c00a9e30 0x7fc4c01f7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4c00cdde0 0x7fc4c01f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 --2- 172.21.15.129:0/3463256033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4c0075fa0 0x7fc4c014a3f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 >> 172.21.15.129:0/3463256033 conn(0x7fc4c0073640 msgr2=0x7fc4c006e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 shutdown_connections 2024-09-17T03:49:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.357+0000 7fc4bf200640 1 -- 172.21.15.129:0/3463256033 wait complete. 2024-09-17T03:49:34.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3814099874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac06d4f0 msgr2=0x7fe2ac152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3814099874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac06d4f0 0x7fe2ac152c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4005fd0 tx=0x7fe2a402a980 comp rx=0 tx=0).stop 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3814099874 shutdown_connections 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3814099874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac06d4f0 0x7fe2ac152c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3814099874 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac0ce800 0x7fe2ac15cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.613+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3814099874 >> 172.21.15.129:0/3814099874 conn(0x7fe2ac0cdb20 msgr2=0x7fe2ac0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.614+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3814099874 shutdown_connections 2024-09-17T03:49:34.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.614+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3814099874 wait complete. 2024-09-17T03:49:34.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.616+0000 7fe2b2c00640 1 Processor -- start 2024-09-17T03:49:34.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.616+0000 7fe2b2c00640 1 -- start start 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac072370 0x7fe2ac1f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2ac0cb150 0x7fe2ac0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac153650 con 0x7fe2ac0ce800 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ac1534d0 con 0x7fe2ac0cb150 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.617+0000 7fe2b2c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ac1537d0 con 0x7fe2ac072370 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39334/0 (socket says 172.21.15.129:39334) 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 -- 172.21.15.129:0/3005817759 learned_addr learned my addr 172.21.15.129:0/3005817759 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b1800640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac072370 0x7fe2ac1f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b2200640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2ac0cb150 0x7fe2ac0ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 -- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac072370 msgr2=0x7fe2ac1f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac072370 0x7fe2ac1f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 -- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2ac0cb150 msgr2=0x7fe2ac0ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2ac0cb150 0x7fe2ac0ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 -- 172.21.15.129:0/3005817759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ac0cefc0 con 0x7fe2ac0ce800 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.618+0000 7fe2b0e00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe29800c3f0 tx=0x7fe29800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:34.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.619+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298017070 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.619+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2980035f0 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.619+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ac0cf2a0 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.619+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298003a40 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.619+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ac0cf860 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.620+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2ac152c70 con 0x7fe2ac0ce800 2024-09-17T03:49:34.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.621+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe29801a070 con 0x7fe2ac0ce800 2024-09-17T03:49:34.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.623+0000 7fe2a9600640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 0x7fe28c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:34.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.623+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5227+0+0 (secure 0 0 0) 0x7fe298099df0 con 0x7fe2ac0ce800 2024-09-17T03:49:34.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.623+0000 7fe2b1800640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 0x7fe28c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:34.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.624+0000 7fe2b1800640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 0x7fe28c07b0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0c47b0 tx=0x7fe2a0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:34.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.628+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe298061d30 con 0x7fe2ac0ce800 2024-09-17T03:49:34.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:34.816+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fe2ac0ced50 con 0x7fe2ac0ce800 2024-09-17T03:49:35.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.358+0000 7fe2a9600640 1 -- 172.21.15.129:0/3005817759 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) ==== 170+0+0 (secure 0 0 0) 0x7fe298066760 con 0x7fe2ac0ce800 2024-09-17T03:49:35.358 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:49:35.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.361+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 msgr2=0x7fe28c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:35.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.361+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 0x7fe28c07b0d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0c47b0 tx=0x7fe2a0005f90 comp rx=0 tx=0).stop 2024-09-17T03:49:35.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.361+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 msgr2=0x7fe2ac15b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.361+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe29800c3f0 tx=0x7fe29800c8b0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 shutdown_connections 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe28c078bc0 0x7fe28c07b0d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2ac0cb150 0x7fe2ac0ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2ac0ce800 0x7fe2ac15b3e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 --2- 172.21.15.129:0/3005817759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2ac072370 0x7fe2ac1f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.362+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 >> 172.21.15.129:0/3005817759 conn(0x7fe2ac0cdb20 msgr2=0x7fe2ac160680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.363+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 shutdown_connections 2024-09-17T03:49:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.363+0000 7fe2b2c00640 1 -- 172.21.15.129:0/3005817759 wait complete. 2024-09-17T03:49:35.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:49:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 -- 172.21.15.129:0/2694125807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 msgr2=0x7f2bf80af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 --2- 172.21.15.129:0/2694125807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80af3c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2bf8072dc0 tx=0x7f2be802a560 comp rx=0 tx=0).stop 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 -- 172.21.15.129:0/2694125807 shutdown_connections 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 --2- 172.21.15.129:0/2694125807 >> v2:172.21.15.181:3300/0 conn(0x7f2bf80ccb50 0x7f2bf80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 --2- 172.21.15.129:0/2694125807 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80af3c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 -- 172.21.15.129:0/2694125807 >> 172.21.15.129:0/2694125807 conn(0x7f2bf806d7d0 msgr2=0x7f2bf80b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 -- 172.21.15.129:0/2694125807 shutdown_connections 2024-09-17T03:49:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.630+0000 7f2bfe400640 1 -- 172.21.15.129:0/2694125807 wait complete. 2024-09-17T03:49:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.633+0000 7f2bfe400640 1 Processor -- start 2024-09-17T03:49:35.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.633+0000 7f2bfe400640 1 -- start start 2024-09-17T03:49:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf80ccb50 0x7f2bf81529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 0x7f2bf815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf81533a0 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf815d3e0 con 0x7f2bf80ccb50 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bfe400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf8153520 con 0x7f2bf815d8a0 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bf7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bf7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39364/0 (socket says 172.21.15.129:39364) 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.634+0000 7f2bf7400640 1 -- 172.21.15.129:0/3980821354 learned_addr learned my addr 172.21.15.129:0/3980821354 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf6a00640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf80ccb50 0x7f2bf81529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7e00640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 0x7f2bf815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 -- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 msgr2=0x7f2bf815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 0x7f2bf815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 -- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf80ccb50 msgr2=0x7f2bf81529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf80ccb50 0x7f2bf81529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 -- 172.21.15.129:0/3980821354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf8073840 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7e00640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 0x7f2bf815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bf7400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2be8000c00 tx=0x7f2be8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.635+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be803e030 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.636+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2be8003e00 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.636+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be8035040 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.636+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf815ec30 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.636+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf8073ac0 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.637+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf80ccf60 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.638+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2be8041030 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.639+0000 7f2bef400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 0x7f2bd807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.639+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5609+0+0 (secure 0 0 0) 0x7f2be80bb380 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.639+0000 7f2bf6a00640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 0x7f2bd807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.640+0000 7f2bf6a00640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 0x7f2bd807b0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bf815bae0 tx=0x7f2be4009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:35.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.644+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2bf8073840 con 0x7f2bf80cc1e0 2024-09-17T03:49:35.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:35.863+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2bf815c840 con 0x7f2bf80cc1e0 2024-09-17T03:49:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.386+0000 7f2bef400640 1 -- 172.21.15.129:0/3980821354 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f2bf80cd4a0 con 0x7f2bf80cc1e0 2024-09-17T03:49:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 2 and data pool 3 2024-09-17T03:49:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.391+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 msgr2=0x7f2bd807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.391+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 0x7f2bd807b0d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bf815bae0 tx=0x7f2be4009060 comp rx=0 tx=0).stop 2024-09-17T03:49:36.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.391+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 msgr2=0x7f2bf80742c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.391+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2be8000c00 tx=0x7f2be8003910 comp rx=0 tx=0).stop 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 shutdown_connections 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd8078bc0 0x7f2bd807b0d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf815d8a0 0x7f2bf815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf80ccb50 0x7f2bf81529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 --2- 172.21.15.129:0/3980821354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf80cc1e0 0x7f2bf80742c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 >> 172.21.15.129:0/3980821354 conn(0x7f2bf806d7d0 msgr2=0x7f2bf80ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 shutdown_connections 2024-09-17T03:49:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.393+0000 7f2bfe400640 1 -- 172.21.15.129:0/3980821354 wait complete. 2024-09-17T03:49:36.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 -- 172.21.15.129:0/1296413063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00cefd0 msgr2=0x7f3de00cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 --2- 172.21.15.129:0/1296413063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00cefd0 0x7f3de00cf3e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3de008d2d0 tx=0x7f3dc802a530 comp rx=0 tx=0).stop 2024-09-17T03:49:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 -- 172.21.15.129:0/1296413063 shutdown_connections 2024-09-17T03:49:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 --2- 172.21.15.129:0/1296413063 >> v2:172.21.15.181:3300/0 conn(0x7f3de00b42f0 0x7f3de00b4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 --2- 172.21.15.129:0/1296413063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00cefd0 0x7f3de00cf3e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 -- 172.21.15.129:0/1296413063 >> 172.21.15.129:0/1296413063 conn(0x7f3de009d680 msgr2=0x7f3de00d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 -- 172.21.15.129:0/1296413063 shutdown_connections 2024-09-17T03:49:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.656+0000 7f3de5600640 1 -- 172.21.15.129:0/1296413063 wait complete. 2024-09-17T03:49:36.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.658+0000 7f3de5600640 1 Processor -- start 2024-09-17T03:49:36.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.658+0000 7f3de5600640 1 -- start start 2024-09-17T03:49:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:36.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3de00b96b0 0x7f3de015b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de00cefd0 0x7f3de015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3de00b9e50 con 0x7f3de00b42f0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3de00b9cd0 con 0x7f3de00cefd0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de5600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3de00b4ec0 con 0x7f3de00b96b0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39382/0 (socket says 172.21.15.129:39382) 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 -- 172.21.15.129:0/1309775390 learned_addr learned my addr 172.21.15.129:0/1309775390 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3de4c00640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de00cefd0 0x7f3de015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3dd7400640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3de00b96b0 0x7f3de015b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 -- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3de00b96b0 msgr2=0x7f3de015b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3de00b96b0 0x7f3de015b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 -- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de00cefd0 msgr2=0x7f3de015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de00cefd0 0x7f3de015b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.660+0000 7f3ddfe00640 1 -- 172.21.15.129:0/1309775390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3de01f53d0 con 0x7f3de00b42f0 2024-09-17T03:49:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3ddfe00640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000b410 tx=0x7f3dd000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0015020 con 0x7f3de00b42f0 2024-09-17T03:49:36.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3dd0011920 con 0x7f3de00b42f0 2024-09-17T03:49:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dd0011d70 con 0x7f3de00b42f0 2024-09-17T03:49:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3de01f5900 con 0x7f3de00b42f0 2024-09-17T03:49:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.661+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3de01f5ec0 con 0x7f3de00b42f0 2024-09-17T03:49:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.662+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de00b4700 con 0x7f3de00b42f0 2024-09-17T03:49:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.663+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3dd0018020 con 0x7f3de00b42f0 2024-09-17T03:49:36.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.664+0000 7f3ddcc00640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 0x7f3dbc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:36.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.664+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3dd009a970 con 0x7f3de00b42f0 2024-09-17T03:49:36.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.665+0000 7f3dd7400640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 0x7f3dbc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:36.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.666+0000 7f3dd7400640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 0x7f3dbc07b0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8033000 tx=0x7f3dc8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:36.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.669+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3dd0062750 con 0x7f3de00b42f0 2024-09-17T03:49:36.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.852+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3de00d0740 con 0x7f3de00b42f0 2024-09-17T03:49:36.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.854+0000 7f3ddcc00640 1 -- 172.21.15.129:0/1309775390 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16726 (secure 0 0 0) 0x7f3dd0067180 con 0x7f3de00b42f0 2024-09-17T03:49:36.854 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:36.854 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:35.864932+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:36.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 msgr2=0x7f3dbc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 0x7f3dbc07b0d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8033000 tx=0x7f3dc8004730 comp rx=0 tx=0).stop 2024-09-17T03:49:36.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 msgr2=0x7f3de015a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:36.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3dd000b410 tx=0x7f3dd000b8d0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 shutdown_connections 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3dbc078bc0 0x7f3dbc07b0d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de00cefd0 0x7f3de015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3de00b96b0 0x7f3de015b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 --2- 172.21.15.129:0/1309775390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3de00b42f0 0x7f3de015a6e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 >> 172.21.15.129:0/1309775390 conn(0x7f3de009d680 msgr2=0x7f3de00b33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 shutdown_connections 2024-09-17T03:49:36.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:36.856+0000 7f3de5600640 1 -- 172.21.15.129:0/1309775390 wait complete. 2024-09-17T03:49:36.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 -- 172.21.15.129:0/3766649068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 msgr2=0x7f437c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 --2- 172.21.15.129:0/3766649068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f437c004800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4380001660 tx=0x7f437c005830 comp rx=0 tx=0).stop 2024-09-17T03:49:37.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439a400640 1 -- 172.21.15.129:0/3766649068 reap_dead start 2024-09-17T03:49:37.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 -- 172.21.15.129:0/3766649068 shutdown_connections 2024-09-17T03:49:37.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 -- 172.21.15.129:0/3766649068 >> 172.21.15.129:0/3766649068 conn(0x7f43940b1cb0 msgr2=0x7f43940b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:37.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 -- 172.21.15.129:0/3766649068 shutdown_connections 2024-09-17T03:49:37.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.099+0000 7f439c400640 1 -- 172.21.15.129:0/3766649068 wait complete. 2024-09-17T03:49:37.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.101+0000 7f439c400640 1 Processor -- start 2024-09-17T03:49:37.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.101+0000 7f439c400640 1 -- start start 2024-09-17T03:49:37.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 0x7f4394159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43940c8c90 0x7f43940c90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4394073cc0 con 0x7f437c004000 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43940cbe20 con 0x7f439415a7f0 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439c400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4394073e40 con 0x7f43940c8c90 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39392/0 (socket says 172.21.15.129:39392) 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399a00640 1 -- 172.21.15.129:0/665941288 learned_addr learned my addr 172.21.15.129:0/665941288 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399000640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 0x7f4394159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f439a400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43940c8c90 0x7f43940c90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399a00640 1 -- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43940c8c90 msgr2=0x7f43940c90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.103+0000 7f4399a00640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43940c8c90 0x7f43940c90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4399a00640 1 -- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 msgr2=0x7f4394159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4399a00640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 0x7f4394159f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4399a00640 1 -- 172.21.15.129:0/665941288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43940c95e0 con 0x7f437c004000 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4399000640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 0x7f4394159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4399a00640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4388011cb0 tx=0x7f438800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:37.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388003e40 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4388004050 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4388004440 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43940c7a20 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.104+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43940c7fe0 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.105+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4394076820 con 0x7f437c004000 2024-09-17T03:49:37.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.106+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f43880045e0 con 0x7f437c004000 2024-09-17T03:49:37.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.107+0000 7f4391600640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 0x7f437407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.107+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f438809c590 con 0x7f437c004000 2024-09-17T03:49:37.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.108+0000 7f4399000640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 0x7f437407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.108+0000 7f4399000640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 0x7f437407b0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4380032000 tx=0x7f4380001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:37.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.112+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4394076820 con 0x7f437c004000 2024-09-17T03:49:37.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.296+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43940735b0 con 0x7f437c004000 2024-09-17T03:49:37.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.297+0000 7f4391600640 1 -- 172.21.15.129:0/665941288 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16726 (secure 0 0 0) 0x7f43940c95e0 con 0x7f437c004000 2024-09-17T03:49:37.297 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:37.298 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":21,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:35.864932+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 msgr2=0x7f437407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 0x7f437407b0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4380032000 tx=0x7f4380001190 comp rx=0 tx=0).stop 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 msgr2=0x7f43940c71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4388011cb0 tx=0x7f438800f790 comp rx=0 tx=0).stop 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 shutdown_connections 2024-09-17T03:49:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4374078bc0 0x7f437407b0d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43940c8c90 0x7f43940c90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f439415a7f0 0x7f4394159f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 --2- 172.21.15.129:0/665941288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c004000 0x7f43940c71d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 >> 172.21.15.129:0/665941288 conn(0x7f43940b1cb0 msgr2=0x7f43940acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 shutdown_connections 2024-09-17T03:49:37.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.300+0000 7f439c400640 1 -- 172.21.15.129:0/665941288 wait complete. 2024-09-17T03:49:37.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:49:37.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.545+0000 7f480de00640 1 -- 172.21.15.129:0/3352496890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080cf6f0 msgr2=0x7f48080cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.545+0000 7f480de00640 1 --2- 172.21.15.129:0/3352496890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080cf6f0 0x7f48080cc4c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f47f8005fd0 tx=0x7f47f802a670 comp rx=0 tx=0).stop 2024-09-17T03:49:37.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.546+0000 7f480de00640 1 -- 172.21.15.129:0/3352496890 shutdown_connections 2024-09-17T03:49:37.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.546+0000 7f480de00640 1 --2- 172.21.15.129:0/3352496890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080cf6f0 0x7f48080cc4c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.546+0000 7f480de00640 1 -- 172.21.15.129:0/3352496890 >> 172.21.15.129:0/3352496890 conn(0x7f48080b1cb0 msgr2=0x7f48080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:37.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.546+0000 7f480de00640 1 -- 172.21.15.129:0/3352496890 shutdown_connections 2024-09-17T03:49:37.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.546+0000 7f480de00640 1 -- 172.21.15.129:0/3352496890 wait complete. 2024-09-17T03:49:37.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.548+0000 7f480de00640 1 Processor -- start 2024-09-17T03:49:37.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.548+0000 7f480de00640 1 -- start start 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48080ce020 0x7f480809c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48080cf6f0 0x7f48080aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f48080cf340 con 0x7f48080ccb30 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f48080cf1c0 con 0x7f48080ce020 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.549+0000 7f480de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f48080701b0 con 0x7f48080cf6f0 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39420/0 (socket says 172.21.15.129:39420) 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 -- 172.21.15.129:0/1693974409 learned_addr learned my addr 172.21.15.129:0/1693974409 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4807400640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48080cf6f0 0x7f48080aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806000640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48080ce020 0x7f480809c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 -- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48080cf6f0 msgr2=0x7f48080aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48080cf6f0 0x7f48080aea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 -- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48080ce020 msgr2=0x7f480809c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48080ce020 0x7f480809c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:37.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 -- 172.21.15.129:0/1693974409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48081f92d0 con 0x7f48080ccb30 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.550+0000 7f4806a00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f8034040 tx=0x7f47f80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.551+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f803e030 con 0x7f48080ccb30 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.551+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47f80047a0 con 0x7f48080ccb30 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.551+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48081f95b0 con 0x7f48080ccb30 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.551+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8035040 con 0x7f48080ccb30 2024-09-17T03:49:37.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.551+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48081f9b70 con 0x7f48080ccb30 2024-09-17T03:49:37.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.552+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4808051080 con 0x7f48080ccb30 2024-09-17T03:49:37.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.553+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f47f8041030 con 0x7f48080ccb30 2024-09-17T03:49:37.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.554+0000 7f47ff400640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 0x7f47e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:37.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.554+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f47f80bb4e0 con 0x7f48080ccb30 2024-09-17T03:49:37.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.555+0000 7f4806000640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 0x7f47e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:37.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.555+0000 7f4806000640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 0x7f47e807b0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48081f3680 tx=0x7f47f4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:37.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.558+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4808051080 con 0x7f48080ccb30 2024-09-17T03:49:37.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:37.781+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f48080cc4c0 con 0x7f48080ccb30 2024-09-17T03:49:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.399+0000 7f47ff400640 1 -- 172.21.15.129:0/1693974409 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f47f8083360 con 0x7f48080ccb30 2024-09-17T03:49:38.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 msgr2=0x7f47e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:38.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 0x7f47e807b0a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f48081f3680 tx=0x7f47f4009040 comp rx=0 tx=0).stop 2024-09-17T03:49:38.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 msgr2=0x7f480814f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f8034040 tx=0x7f47f80039e0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 shutdown_connections 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47e8078b90 0x7f47e807b0a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48080cf6f0 0x7f48080aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48080ce020 0x7f480809c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 --2- 172.21.15.129:0/1693974409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48080ccb30 0x7f480814f520 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 >> 172.21.15.129:0/1693974409 conn(0x7f48080b1cb0 msgr2=0x7f48080742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 shutdown_connections 2024-09-17T03:49:38.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.402+0000 7f480de00640 1 -- 172.21.15.129:0/1693974409 wait complete. 2024-09-17T03:49:38.425 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:49:38.425 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:49:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 -- 172.21.15.129:0/1628238150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d64162d70 msgr2=0x7f8d64163180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 --2- 172.21.15.129:0/1628238150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d64162d70 0x7f8d64163180 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c005fd0 tx=0x7f8d5c02a8b0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 -- 172.21.15.129:0/1628238150 shutdown_connections 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 --2- 172.21.15.129:0/1628238150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d64162d70 0x7f8d64163180 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 --2- 172.21.15.129:0/1628238150 >> v2:172.21.15.181:3300/0 conn(0x7f8d640b5520 0x7f8d640cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 -- 172.21.15.129:0/1628238150 >> 172.21.15.129:0/1628238150 conn(0x7f8d6409d680 msgr2=0x7f8d640d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 -- 172.21.15.129:0/1628238150 shutdown_connections 2024-09-17T03:49:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.649+0000 7f8d69200640 1 -- 172.21.15.129:0/1628238150 wait complete. 2024-09-17T03:49:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.651+0000 7f8d69200640 1 Processor -- start 2024-09-17T03:49:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.652+0000 7f8d69200640 1 -- start start 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 0x7f8d640cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d640b71a0 0x7f8d640ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d64163a70 con 0x7f8d64162d70 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8d640d0d50 con 0x7f8d640b5520 2024-09-17T03:49:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d69200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8d640c91e0 con 0x7f8d640b71a0 2024-09-17T03:49:38.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:38.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39424/0 (socket says 172.21.15.129:39424) 2024-09-17T03:49:38.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.653+0000 7f8d63e00640 1 -- 172.21.15.129:0/3070438195 learned_addr learned my addr 172.21.15.129:0/3070438195 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63400640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 0x7f8d640cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d5aa00640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d640b71a0 0x7f8d640ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 -- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d640b71a0 msgr2=0x7f8d640ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d640b71a0 0x7f8d640ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 -- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 msgr2=0x7f8d640cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 0x7f8d640cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 -- 172.21.15.129:0/3070438195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d640cd910 con 0x7f8d64162d70 2024-09-17T03:49:38.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63400640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 0x7f8d640cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d63e00640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009cd0 tx=0x7f8d5c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c031990 con 0x7f8d64162d70 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d5c031c20 con 0x7f8d64162d70 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c0330d0 con 0x7f8d64162d70 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.654+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d641f8de0 con 0x7f8d64162d70 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.655+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d641f93a0 con 0x7f8d64162d70 2024-09-17T03:49:38.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.656+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d640cfd70 con 0x7f8d64162d70 2024-09-17T03:49:38.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.657+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8d5c033270 con 0x7f8d64162d70 2024-09-17T03:49:38.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.658+0000 7f8d5be00640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 0x7f8d4007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:38.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.658+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f8d5c0bc230 con 0x7f8d64162d70 2024-09-17T03:49:38.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.658+0000 7f8d63400640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 0x7f8d4007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:38.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.659+0000 7f8d63400640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 0x7f8d4007b0a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009c10 tx=0x7f8d540091e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:38.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.662+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8d5c03f030 con 0x7f8d64162d70 2024-09-17T03:49:38.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:38.879+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f8d6408e430 con 0x7f8d64162d70 2024-09-17T03:49:39.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.406+0000 7f8d5be00640 1 -- 172.21.15.129:0/3070438195 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f8d5c084660 con 0x7f8d64162d70 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.408+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 msgr2=0x7f8d4007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 0x7f8d4007b0a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d54009c10 tx=0x7f8d540091e0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 msgr2=0x7f8d640b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009cd0 tx=0x7f8d5c031700 comp rx=0 tx=0).stop 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 shutdown_connections 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d40078b90 0x7f8d4007b0a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d64162d70 0x7f8d640b8c00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d640b71a0 0x7f8d640ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 --2- 172.21.15.129:0/3070438195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d640b5520 0x7f8d640cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 >> 172.21.15.129:0/3070438195 conn(0x7f8d6409d680 msgr2=0x7f8d6406f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 shutdown_connections 2024-09-17T03:49:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.409+0000 7f8d69200640 1 -- 172.21.15.129:0/3070438195 wait complete. 2024-09-17T03:49:39.430 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:49:39.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 -- 172.21.15.129:0/266413782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0706c0 msgr2=0x7ff34c072ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 --2- 172.21.15.129:0/266413782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0706c0 0x7ff34c072ff0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff344005fd0 tx=0x7ff34402a820 comp rx=0 tx=0).stop 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 -- 172.21.15.129:0/266413782 shutdown_connections 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 --2- 172.21.15.129:0/266413782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0706c0 0x7ff34c072ff0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 --2- 172.21.15.129:0/266413782 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0cd3e0 0x7ff34c070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 -- 172.21.15.129:0/266413782 >> 172.21.15.129:0/266413782 conn(0x7ff34c06d7d0 msgr2=0x7ff34c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 -- 172.21.15.129:0/266413782 shutdown_connections 2024-09-17T03:49:39.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.652+0000 7ff353800640 1 -- 172.21.15.129:0/266413782 wait complete. 2024-09-17T03:49:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.655+0000 7ff353800640 1 Processor -- start 2024-09-17T03:49:39.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.655+0000 7ff353800640 1 -- start start 2024-09-17T03:49:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0706c0 0x7ff34c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 0x7ff34c1e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c073b20 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff34c0739a0 con 0x7ff34c0af780 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff353800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff34c073ca0 con 0x7ff34c0706c0 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39436/0 (socket says 172.21.15.129:39436) 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 -- 172.21.15.129:0/1637323471 learned_addr learned my addr 172.21.15.129:0/1637323471 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff343e00640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 0x7ff34c1e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff350e00640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0706c0 0x7ff34c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 -- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0706c0 msgr2=0x7ff34c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0706c0 0x7ff34c159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 -- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 msgr2=0x7ff34c1e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 0x7ff34c1e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 -- 172.21.15.129:0/1637323471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34c0c5370 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff343e00640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 0x7ff34c1e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff351800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff344035040 tx=0x7ff344032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.657+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344037040 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.658+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff344042070 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.658+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff344004260 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.658+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34c15d830 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.658+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff34c1f5160 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.659+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff34c1f4f60 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.660+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff344004400 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.661+0000 7ff349600640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 0x7ff32807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.661+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff3440bc690 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.661+0000 7ff350e00640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 0x7ff32807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:39.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.662+0000 7ff350e00640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 0x7ff32807b0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff33c000f60 tx=0x7ff33c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:39.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.664+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff34c0c5370 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.882+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff34c072ff0 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.883+0000 7ff349600640 1 -- 172.21.15.129:0/1637323471 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7ff3440046f0 con 0x7ff34c0cd3e0 2024-09-17T03:49:39.883 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:39.883 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:49:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 msgr2=0x7ff32807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 0x7ff32807b0a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff33c000f60 tx=0x7ff33c009040 comp rx=0 tx=0).stop 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 msgr2=0x7ff34c1e9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff344035040 tx=0x7ff344032d10 comp rx=0 tx=0).stop 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 shutdown_connections 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff328078b90 0x7ff32807b0a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff34c0cd3e0 0x7ff34c1e9240 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34c0af780 0x7ff34c1e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 --2- 172.21.15.129:0/1637323471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff34c0706c0 0x7ff34c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 >> 172.21.15.129:0/1637323471 conn(0x7ff34c06d7d0 msgr2=0x7ff34c0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 shutdown_connections 2024-09-17T03:49:39.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:39.886+0000 7ff353800640 1 -- 172.21.15.129:0/1637323471 wait complete. 2024-09-17T03:49:39.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:40.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 -- 172.21.15.129:0/728791429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cd240 msgr2=0x7f1b100cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 --2- 172.21.15.129:0/728791429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cd240 0x7f1b100cf010 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1afc005fd0 tx=0x7f1afc02b910 comp rx=0 tx=0).stop 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 -- 172.21.15.129:0/728791429 shutdown_connections 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 --2- 172.21.15.129:0/728791429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cd240 0x7f1b100cf010 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 -- 172.21.15.129:0/728791429 >> 172.21.15.129:0/728791429 conn(0x7f1b100b1cb0 msgr2=0x7f1b100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 -- 172.21.15.129:0/728791429 shutdown_connections 2024-09-17T03:49:40.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.124+0000 7f1b18000640 1 -- 172.21.15.129:0/728791429 wait complete. 2024-09-17T03:49:40.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.126+0000 7f1b18000640 1 Processor -- start 2024-09-17T03:49:40.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 -- start start 2024-09-17T03:49:40.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b100cfb90 0x7f1b1009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b1009af70 0x7f1b10156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1b100710a0 con 0x7f1b100cf660 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1b10070f20 con 0x7f1b1009af70 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b18000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1b100b36c0 con 0x7f1b100cfb90 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b15600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b15600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39458/0 (socket says 172.21.15.129:39458) 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.128+0000 7f1b15600640 1 -- 172.21.15.129:0/404680087 learned_addr learned my addr 172.21.15.129:0/404680087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b14c00640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b100cfb90 0x7f1b1009aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b16000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b1009af70 0x7f1b10156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 -- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b100cfb90 msgr2=0x7f1b1009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b100cfb90 0x7f1b1009aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 -- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b1009af70 msgr2=0x7f1b10156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b1009af70 0x7f1b10156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 -- 172.21.15.129:0/404680087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b101e8240 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b15600640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b04001eb0 tx=0x7f1b0400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04017070 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b040053c0 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04003830 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b101e8520 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.129+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b10074850 con 0x7f1b100cf660 2024-09-17T03:49:40.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.131+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b100b4cf0 con 0x7f1b100cf660 2024-09-17T03:49:40.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.132+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1b0401a070 con 0x7f1b100cf660 2024-09-17T03:49:40.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.133+0000 7f1b0d600640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 0x7f1af007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.133+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1b0409a0d0 con 0x7f1b100cf660 2024-09-17T03:49:40.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.133+0000 7f1b14c00640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 0x7f1af007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.134+0000 7f1b14c00640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 0x7f1af007b0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1afc035000 tx=0x7f1afc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:40.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.138+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1b04062500 con 0x7f1b100cf660 2024-09-17T03:49:40.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.322+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b100b2260 con 0x7f1b100cf660 2024-09-17T03:49:40.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.323+0000 7f1b0d600640 1 -- 172.21.15.129:0/404680087 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7f1b04061e00 con 0x7f1b100cf660 2024-09-17T03:49:40.322 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:40.322 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.325+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 msgr2=0x7f1af007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.325+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 0x7f1af007b0a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1afc035000 tx=0x7f1afc004730 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.325+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 msgr2=0x7f1b1009c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.325+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b04001eb0 tx=0x7f1b0400c890 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 shutdown_connections 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1af0078b90 0x7f1af007b0a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b1009af70 0x7f1b10156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b100cfb90 0x7f1b1009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 --2- 172.21.15.129:0/404680087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b100cf660 0x7f1b1009c290 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 >> 172.21.15.129:0/404680087 conn(0x7f1b100b1cb0 msgr2=0x7f1b1015a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:40.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 shutdown_connections 2024-09-17T03:49:40.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.326+0000 7f1b18000640 1 -- 172.21.15.129:0/404680087 wait complete. 2024-09-17T03:49:40.349 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:49:40.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:49:40.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 -- 172.21.15.129:0/2423657961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304073170 msgr2=0x7fe30415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 --2- 172.21.15.129:0/2423657961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304073170 0x7fe30415c780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4005fd0 tx=0x7fe2f402a540 comp rx=0 tx=0).stop 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 -- 172.21.15.129:0/2423657961 shutdown_connections 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 --2- 172.21.15.129:0/2423657961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 0x7fe304074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 --2- 172.21.15.129:0/2423657961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304073170 0x7fe30415c780 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 -- 172.21.15.129:0/2423657961 >> 172.21.15.129:0/2423657961 conn(0x7fe3040b1cb0 msgr2=0x7fe3040b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 -- 172.21.15.129:0/2423657961 shutdown_connections 2024-09-17T03:49:40.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.570+0000 7fe30a800640 1 -- 172.21.15.129:0/2423657961 wait complete. 2024-09-17T03:49:40.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.572+0000 7fe30a800640 1 Processor -- start 2024-09-17T03:49:40.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.572+0000 7fe30a800640 1 -- start start 2024-09-17T03:49:40.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 0x7fe30414c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe304073170 0x7fe3040cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe30406f5a0 con 0x7fe304070440 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe304074f80 con 0x7fe304073170 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe30a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe30406f720 con 0x7fe304070030 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39486/0 (socket says 172.21.15.129:39486) 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 -- 172.21.15.129:0/478298341 learned_addr learned my addr 172.21.15.129:0/478298341 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe303400640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 0x7fe30414c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe303e00640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe304073170 0x7fe3040cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 -- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 msgr2=0x7fe30414c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 0x7fe30414c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 -- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe304073170 msgr2=0x7fe3040cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe304073170 0x7fe3040cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.574+0000 7fe2faa00640 1 -- 172.21.15.129:0/478298341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3041f9040 con 0x7fe304070440 2024-09-17T03:49:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe2faa00640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e7c0 tx=0x7fe2ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:40.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00ccc0 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2ec00ce60 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3041f9320 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec005280 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.575+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30414cb50 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.576+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3040cc630 con 0x7fe304070440 2024-09-17T03:49:40.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.577+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe2ec005420 con 0x7fe304070440 2024-09-17T03:49:40.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.579+0000 7fe2fbe00640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 0x7fe2e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:40.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.579+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe2ec09b730 con 0x7fe304070440 2024-09-17T03:49:40.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.579+0000 7fe303400640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 0x7fe2e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:40.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.580+0000 7fe303400640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 0x7fe2e007b0a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4033000 tx=0x7fe2f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:40.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.584+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe2ec063ad0 con 0x7fe304070440 2024-09-17T03:49:40.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.762+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3041f8df0 con 0x7fe2e0078b90 2024-09-17T03:49:40.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.768+0000 7fe2fbe00640 1 -- 172.21.15.129:0/478298341 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251935 (secure 0 0 0) 0x7fe3041f8df0 con 0x7fe2e0078b90 2024-09-17T03:49:40.770 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:40.772 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:49:40.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 msgr2=0x7fe2e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 0x7fe2e007b0a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe2f4033000 tx=0x7fe2f4004730 comp rx=0 tx=0).stop 2024-09-17T03:49:40.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 msgr2=0x7fe3040cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00e7c0 tx=0x7fe2ec00ec80 comp rx=0 tx=0).stop 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 shutdown_connections 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2e0078b90 0x7fe2e007b0a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe304073170 0x7fe3040cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe304070440 0x7fe3040cbb20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 --2- 172.21.15.129:0/478298341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe304070030 0x7fe30414c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 >> 172.21.15.129:0/478298341 conn(0x7fe3040b1cb0 msgr2=0x7fe3040adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 shutdown_connections 2024-09-17T03:49:40.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:40.773+0000 7fe30a800640 1 -- 172.21.15.129:0/478298341 wait complete. 2024-09-17T03:49:40.794 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-17T03:49:40.293750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223376,"kb_used_data":2940,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751344,"statfs":{"total":773094113280,"available":772865376256,"internally_reserved":0,"allocated":3010560,"data_stored":1794120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936396"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589196+0000","last_change":"2024-09-17T03:49:36.406020+0000","last_active":"2024-09-17T03:49:37.589196+0000","last_peered":"2024-09-17T03:49:37.589196+0000","last_clean":"2024-09-17T03:49:37.589196+0000","last_became_active":"2024-09-17T03:49:36.405649+0000","last_became_peered":"2024-09-17T03:49:36.405649+0000","last_unstale":"2024-09-17T03:49:37.589196+0000","last_undegraded":"2024-09-17T03:49:37.589196+0000","last_fullsized":"2024-09-17T03:49:37.589196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377354+0000","last_change":"2024-09-17T03:49:36.400461+0000","last_active":"2024-09-17T03:49:37.377354+0000","last_peered":"2024-09-17T03:49:37.377354+0000","last_clean":"2024-09-17T03:49:37.377354+0000","last_became_active":"2024-09-17T03:49:36.398210+0000","last_became_peered":"2024-09-17T03:49:36.398210+0000","last_unstale":"2024-09-17T03:49:37.377354+0000","last_undegraded":"2024-09-17T03:49:37.377354+0000","last_fullsized":"2024-09-17T03:49:37.377354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589718+0000","last_change":"2024-09-17T03:49:36.397732+0000","last_active":"2024-09-17T03:49:37.589718+0000","last_peered":"2024-09-17T03:49:37.589718+0000","last_clean":"2024-09-17T03:49:37.589718+0000","last_became_active":"2024-09-17T03:49:36.397407+0000","last_became_peered":"2024-09-17T03:49:36.397407+0000","last_unstale":"2024-09-17T03:49:37.589718+0000","last_undegraded":"2024-09-17T03:49:37.589718+0000","last_fullsized":"2024-09-17T03:49:37.589718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379346+0000","last_change":"2024-09-17T03:49:36.410331+0000","last_active":"2024-09-17T03:49:37.379346+0000","last_peered":"2024-09-17T03:49:37.379346+0000","last_clean":"2024-09-17T03:49:37.379346+0000","last_became_active":"2024-09-17T03:49:36.409931+0000","last_became_peered":"2024-09-17T03:49:36.409931+0000","last_unstale":"2024-09-17T03:49:37.379346+0000","last_undegraded":"2024-09-17T03:49:37.379346+0000","last_fullsized":"2024-09-17T03:49:37.379346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589345+0000","last_change":"2024-09-17T03:49:36.415397+0000","last_active":"2024-09-17T03:49:37.589345+0000","last_peered":"2024-09-17T03:49:37.589345+0000","last_clean":"2024-09-17T03:49:37.589345+0000","last_became_active":"2024-09-17T03:49:36.415075+0000","last_became_peered":"2024-09-17T03:49:36.415075+0000","last_unstale":"2024-09-17T03:49:37.589345+0000","last_undegraded":"2024-09-17T03:49:37.589345+0000","last_fullsized":"2024-09-17T03:49:37.589345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:08.390183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379503+0000","last_change":"2024-09-17T03:49:36.405656+0000","last_active":"2024-09-17T03:49:37.379503+0000","last_peered":"2024-09-17T03:49:37.379503+0000","last_clean":"2024-09-17T03:49:37.379503+0000","last_became_active":"2024-09-17T03:49:36.404978+0000","last_became_peered":"2024-09-17T03:49:36.404978+0000","last_unstale":"2024-09-17T03:49:37.379503+0000","last_undegraded":"2024-09-17T03:49:37.379503+0000","last_fullsized":"2024-09-17T03:49:37.379503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588643+0000","last_change":"2024-09-17T03:49:36.402961+0000","last_active":"2024-09-17T03:49:37.588643+0000","last_peered":"2024-09-17T03:49:37.588643+0000","last_clean":"2024-09-17T03:49:37.588643+0000","last_became_active":"2024-09-17T03:49:36.402413+0000","last_became_peered":"2024-09-17T03:49:36.402413+0000","last_unstale":"2024-09-17T03:49:37.588643+0000","last_undegraded":"2024-09-17T03:49:37.588643+0000","last_fullsized":"2024-09-17T03:49:37.588643+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:09.378052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588967+0000","last_change":"2024-09-17T03:49:36.409564+0000","last_active":"2024-09-17T03:49:37.588967+0000","last_peered":"2024-09-17T03:49:37.588967+0000","last_clean":"2024-09-17T03:49:37.588967+0000","last_became_active":"2024-09-17T03:49:36.409150+0000","last_became_peered":"2024-09-17T03:49:36.409150+0000","last_unstale":"2024-09-17T03:49:37.588967+0000","last_undegraded":"2024-09-17T03:49:37.588967+0000","last_fullsized":"2024-09-17T03:49:37.588967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:09.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589233+0000","last_change":"2024-09-17T03:49:36.416599+0000","last_active":"2024-09-17T03:49:37.589233+0000","last_peered":"2024-09-17T03:49:37.589233+0000","last_clean":"2024-09-17T03:49:37.589233+0000","last_became_active":"2024-09-17T03:49:36.416100+0000","last_became_peered":"2024-09-17T03:49:36.416100+0000","last_unstale":"2024-09-17T03:49:37.589233+0000","last_undegraded":"2024-09-17T03:49:37.589233+0000","last_fullsized":"2024-09-17T03:49:37.589233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420661+0000","last_change":"2024-09-17T03:49:36.412006+0000","last_active":"2024-09-17T03:49:37.420661+0000","last_peered":"2024-09-17T03:49:37.420661+0000","last_clean":"2024-09-17T03:49:37.420661+0000","last_became_active":"2024-09-17T03:49:36.411697+0000","last_became_peered":"2024-09-17T03:49:36.411697+0000","last_unstale":"2024-09-17T03:49:37.420661+0000","last_undegraded":"2024-09-17T03:49:37.420661+0000","last_fullsized":"2024-09-17T03:49:37.420661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420825+0000","last_change":"2024-09-17T03:49:36.415050+0000","last_active":"2024-09-17T03:49:37.420825+0000","last_peered":"2024-09-17T03:49:37.420825+0000","last_clean":"2024-09-17T03:49:37.420825+0000","last_became_active":"2024-09-17T03:49:36.414808+0000","last_became_peered":"2024-09-17T03:49:36.414808+0000","last_unstale":"2024-09-17T03:49:37.420825+0000","last_undegraded":"2024-09-17T03:49:37.420825+0000","last_fullsized":"2024-09-17T03:49:37.420825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379278+0000","last_change":"2024-09-17T03:49:36.406397+0000","last_active":"2024-09-17T03:49:37.379278+0000","last_peered":"2024-09-17T03:49:37.379278+0000","last_clean":"2024-09-17T03:49:37.379278+0000","last_became_active":"2024-09-17T03:49:36.405791+0000","last_became_peered":"2024-09-17T03:49:36.405791+0000","last_unstale":"2024-09-17T03:49:37.379278+0000","last_undegraded":"2024-09-17T03:49:37.379278+0000","last_fullsized":"2024-09-17T03:49:37.379278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:05.635714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589437+0000","last_change":"2024-09-17T03:49:36.393913+0000","last_active":"2024-09-17T03:49:37.589437+0000","last_peered":"2024-09-17T03:49:37.589437+0000","last_clean":"2024-09-17T03:49:37.589437+0000","last_became_active":"2024-09-17T03:49:36.393589+0000","last_became_peered":"2024-09-17T03:49:36.393589+0000","last_unstale":"2024-09-17T03:49:37.589437+0000","last_undegraded":"2024-09-17T03:49:37.589437+0000","last_fullsized":"2024-09-17T03:49:37.589437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588596+0000","last_change":"2024-09-17T03:49:36.401985+0000","last_active":"2024-09-17T03:49:37.588596+0000","last_peered":"2024-09-17T03:49:37.588596+0000","last_clean":"2024-09-17T03:49:37.588596+0000","last_became_active":"2024-09-17T03:49:36.401682+0000","last_became_peered":"2024-09-17T03:49:36.401682+0000","last_unstale":"2024-09-17T03:49:37.588596+0000","last_undegraded":"2024-09-17T03:49:37.588596+0000","last_fullsized":"2024-09-17T03:49:37.588596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:46.086705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377130+0000","last_change":"2024-09-17T03:49:36.389920+0000","last_active":"2024-09-17T03:49:37.377130+0000","last_peered":"2024-09-17T03:49:37.377130+0000","last_clean":"2024-09-17T03:49:37.377130+0000","last_became_active":"2024-09-17T03:49:36.389645+0000","last_became_peered":"2024-09-17T03:49:36.389645+0000","last_unstale":"2024-09-17T03:49:37.377130+0000","last_undegraded":"2024-09-17T03:49:37.377130+0000","last_fullsized":"2024-09-17T03:49:37.377130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:20.042459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378538+0000","last_change":"2024-09-17T03:49:36.403578+0000","last_active":"2024-09-17T03:49:37.378538+0000","last_peered":"2024-09-17T03:49:37.378538+0000","last_clean":"2024-09-17T03:49:37.378538+0000","last_became_active":"2024-09-17T03:49:36.403353+0000","last_became_peered":"2024-09-17T03:49:36.403353+0000","last_unstale":"2024-09-17T03:49:37.378538+0000","last_undegraded":"2024-09-17T03:49:37.378538+0000","last_fullsized":"2024-09-17T03:49:37.378538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380751+0000","last_change":"2024-09-17T03:49:36.406004+0000","last_active":"2024-09-17T03:49:37.380751+0000","last_peered":"2024-09-17T03:49:37.380751+0000","last_clean":"2024-09-17T03:49:37.380751+0000","last_became_active":"2024-09-17T03:49:36.403931+0000","last_became_peered":"2024-09-17T03:49:36.403931+0000","last_unstale":"2024-09-17T03:49:37.380751+0000","last_undegraded":"2024-09-17T03:49:37.380751+0000","last_fullsized":"2024-09-17T03:49:37.380751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588086+0000","last_change":"2024-09-17T03:49:36.396161+0000","last_active":"2024-09-17T03:49:37.588086+0000","last_peered":"2024-09-17T03:49:37.588086+0000","last_clean":"2024-09-17T03:49:37.588086+0000","last_became_active":"2024-09-17T03:49:36.395717+0000","last_became_peered":"2024-09-17T03:49:36.395717+0000","last_unstale":"2024-09-17T03:49:37.588086+0000","last_undegraded":"2024-09-17T03:49:37.588086+0000","last_fullsized":"2024-09-17T03:49:37.588086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:42.475745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589346+0000","last_change":"2024-09-17T03:49:36.410754+0000","last_active":"2024-09-17T03:49:37.589346+0000","last_peered":"2024-09-17T03:49:37.589346+0000","last_clean":"2024-09-17T03:49:37.589346+0000","last_became_active":"2024-09-17T03:49:36.404715+0000","last_became_peered":"2024-09-17T03:49:36.404715+0000","last_unstale":"2024-09-17T03:49:37.589346+0000","last_undegraded":"2024-09-17T03:49:37.589346+0000","last_fullsized":"2024-09-17T03:49:37.589346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:51.120762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420579+0000","last_change":"2024-09-17T03:49:36.409394+0000","last_active":"2024-09-17T03:49:37.420579+0000","last_peered":"2024-09-17T03:49:37.420579+0000","last_clean":"2024-09-17T03:49:37.420579+0000","last_became_active":"2024-09-17T03:49:36.408771+0000","last_became_peered":"2024-09-17T03:49:36.408771+0000","last_unstale":"2024-09-17T03:49:37.420579+0000","last_undegraded":"2024-09-17T03:49:37.420579+0000","last_fullsized":"2024-09-17T03:49:37.420579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588435+0000","last_change":"2024-09-17T03:49:36.422246+0000","last_active":"2024-09-17T03:49:37.588435+0000","last_peered":"2024-09-17T03:49:37.588435+0000","last_clean":"2024-09-17T03:49:37.588435+0000","last_became_active":"2024-09-17T03:49:36.421985+0000","last_became_peered":"2024-09-17T03:49:36.421985+0000","last_unstale":"2024-09-17T03:49:37.588435+0000","last_undegraded":"2024-09-17T03:49:37.588435+0000","last_fullsized":"2024-09-17T03:49:37.588435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:13.173976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379785+0000","last_change":"2024-09-17T03:49:36.405326+0000","last_active":"2024-09-17T03:49:37.379785+0000","last_peered":"2024-09-17T03:49:37.379785+0000","last_clean":"2024-09-17T03:49:37.379785+0000","last_became_active":"2024-09-17T03:49:36.404541+0000","last_became_peered":"2024-09-17T03:49:36.404541+0000","last_unstale":"2024-09-17T03:49:37.379785+0000","last_undegraded":"2024-09-17T03:49:37.379785+0000","last_fullsized":"2024-09-17T03:49:37.379785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589367+0000","last_change":"2024-09-17T03:49:36.406574+0000","last_active":"2024-09-17T03:49:37.589367+0000","last_peered":"2024-09-17T03:49:37.589367+0000","last_clean":"2024-09-17T03:49:37.589367+0000","last_became_active":"2024-09-17T03:49:36.406162+0000","last_became_peered":"2024-09-17T03:49:36.406162+0000","last_unstale":"2024-09-17T03:49:37.589367+0000","last_undegraded":"2024-09-17T03:49:37.589367+0000","last_fullsized":"2024-09-17T03:49:37.589367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378837+0000","last_change":"2024-09-17T03:49:36.394666+0000","last_active":"2024-09-17T03:49:37.378837+0000","last_peered":"2024-09-17T03:49:37.378837+0000","last_clean":"2024-09-17T03:49:37.378837+0000","last_became_active":"2024-09-17T03:49:36.394259+0000","last_became_peered":"2024-09-17T03:49:36.394259+0000","last_unstale":"2024-09-17T03:49:37.378837+0000","last_undegraded":"2024-09-17T03:49:37.378837+0000","last_fullsized":"2024-09-17T03:49:37.378837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590715+0000","last_change":"2024-09-17T03:49:36.410498+0000","last_active":"2024-09-17T03:49:37.590715+0000","last_peered":"2024-09-17T03:49:37.590715+0000","last_clean":"2024-09-17T03:49:37.590715+0000","last_became_active":"2024-09-17T03:49:36.410228+0000","last_became_peered":"2024-09-17T03:49:36.410228+0000","last_unstale":"2024-09-17T03:49:37.590715+0000","last_undegraded":"2024-09-17T03:49:37.590715+0000","last_fullsized":"2024-09-17T03:49:37.590715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420810+0000","last_change":"2024-09-17T03:49:36.410333+0000","last_active":"2024-09-17T03:49:37.420810+0000","last_peered":"2024-09-17T03:49:37.420810+0000","last_clean":"2024-09-17T03:49:37.420810+0000","last_became_active":"2024-09-17T03:49:36.405435+0000","last_became_peered":"2024-09-17T03:49:36.405435+0000","last_unstale":"2024-09-17T03:49:37.420810+0000","last_undegraded":"2024-09-17T03:49:37.420810+0000","last_fullsized":"2024-09-17T03:49:37.420810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:21.315923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377561+0000","last_change":"2024-09-17T03:49:36.394974+0000","last_active":"2024-09-17T03:49:37.377561+0000","last_peered":"2024-09-17T03:49:37.377561+0000","last_clean":"2024-09-17T03:49:37.377561+0000","last_became_active":"2024-09-17T03:49:36.393921+0000","last_became_peered":"2024-09-17T03:49:36.393921+0000","last_unstale":"2024-09-17T03:49:37.377561+0000","last_undegraded":"2024-09-17T03:49:37.377561+0000","last_fullsized":"2024-09-17T03:49:37.377561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590561+0000","last_change":"2024-09-17T03:49:36.406505+0000","last_active":"2024-09-17T03:49:37.590561+0000","last_peered":"2024-09-17T03:49:37.590561+0000","last_clean":"2024-09-17T03:49:37.590561+0000","last_became_active":"2024-09-17T03:49:36.406154+0000","last_became_peered":"2024-09-17T03:49:36.406154+0000","last_unstale":"2024-09-17T03:49:37.590561+0000","last_undegraded":"2024-09-17T03:49:37.590561+0000","last_fullsized":"2024-09-17T03:49:37.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588653+0000","last_change":"2024-09-17T03:49:36.400848+0000","last_active":"2024-09-17T03:49:37.588653+0000","last_peered":"2024-09-17T03:49:37.588653+0000","last_clean":"2024-09-17T03:49:37.588653+0000","last_became_active":"2024-09-17T03:49:36.399273+0000","last_became_peered":"2024-09-17T03:49:36.399273+0000","last_unstale":"2024-09-17T03:49:37.588653+0000","last_undegraded":"2024-09-17T03:49:37.588653+0000","last_fullsized":"2024-09-17T03:49:37.588653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379849+0000","last_change":"2024-09-17T03:49:36.409112+0000","last_active":"2024-09-17T03:49:37.379849+0000","last_peered":"2024-09-17T03:49:37.379849+0000","last_clean":"2024-09-17T03:49:37.379849+0000","last_became_active":"2024-09-17T03:49:36.408890+0000","last_became_peered":"2024-09-17T03:49:36.408890+0000","last_unstale":"2024-09-17T03:49:37.379849+0000","last_undegraded":"2024-09-17T03:49:37.379849+0000","last_fullsized":"2024-09-17T03:49:37.379849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588078+0000","last_change":"2024-09-17T03:49:36.408545+0000","last_active":"2024-09-17T03:49:37.588078+0000","last_peered":"2024-09-17T03:49:37.588078+0000","last_clean":"2024-09-17T03:49:37.588078+0000","last_became_active":"2024-09-17T03:49:36.408285+0000","last_became_peered":"2024-09-17T03:49:36.408285+0000","last_unstale":"2024-09-17T03:49:37.588078+0000","last_undegraded":"2024-09-17T03:49:37.588078+0000","last_fullsized":"2024-09-17T03:49:37.588078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420257+0000","last_change":"2024-09-17T03:49:36.401317+0000","last_active":"2024-09-17T03:49:37.420257+0000","last_peered":"2024-09-17T03:49:37.420257+0000","last_clean":"2024-09-17T03:49:37.420257+0000","last_became_active":"2024-09-17T03:49:36.400953+0000","last_became_peered":"2024-09-17T03:49:36.400953+0000","last_unstale":"2024-09-17T03:49:37.420257+0000","last_undegraded":"2024-09-17T03:49:37.420257+0000","last_fullsized":"2024-09-17T03:49:37.420257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:45.195448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378778+0000","last_change":"2024-09-17T03:49:36.400419+0000","last_active":"2024-09-17T03:49:37.378778+0000","last_peered":"2024-09-17T03:49:37.378778+0000","last_clean":"2024-09-17T03:49:37.378778+0000","last_became_active":"2024-09-17T03:49:36.399021+0000","last_became_peered":"2024-09-17T03:49:36.399021+0000","last_unstale":"2024-09-17T03:49:37.378778+0000","last_undegraded":"2024-09-17T03:49:37.378778+0000","last_fullsized":"2024-09-17T03:49:37.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:50.779802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590324+0000","last_change":"2024-09-17T03:49:35.392535+0000","last_active":"2024-09-17T03:49:37.590324+0000","last_peered":"2024-09-17T03:49:37.590324+0000","last_clean":"2024-09-17T03:49:37.590324+0000","last_became_active":"2024-09-17T03:49:35.392219+0000","last_became_peered":"2024-09-17T03:49:35.392219+0000","last_unstale":"2024-09-17T03:49:37.590324+0000","last_undegraded":"2024-09-17T03:49:37.590324+0000","last_fullsized":"2024-09-17T03:49:37.590324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.459490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588861+0000","last_change":"2024-09-17T03:49:36.400903+0000","last_active":"2024-09-17T03:49:37.588861+0000","last_peered":"2024-09-17T03:49:37.588861+0000","last_clean":"2024-09-17T03:49:37.588861+0000","last_became_active":"2024-09-17T03:49:36.400109+0000","last_became_peered":"2024-09-17T03:49:36.400109+0000","last_unstale":"2024-09-17T03:49:37.588861+0000","last_undegraded":"2024-09-17T03:49:37.588861+0000","last_fullsized":"2024-09-17T03:49:37.588861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:34.979447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420155+0000","last_change":"2024-09-17T03:49:35.393512+0000","last_active":"2024-09-17T03:49:37.420155+0000","last_peered":"2024-09-17T03:49:37.420155+0000","last_clean":"2024-09-17T03:49:37.420155+0000","last_became_active":"2024-09-17T03:49:35.388644+0000","last_became_peered":"2024-09-17T03:49:35.388644+0000","last_unstale":"2024-09-17T03:49:37.420155+0000","last_undegraded":"2024-09-17T03:49:37.420155+0000","last_fullsized":"2024-09-17T03:49:37.420155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:02:36.651524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589477+0000","last_change":"2024-09-17T03:49:35.391294+0000","last_active":"2024-09-17T03:49:37.589477+0000","last_peered":"2024-09-17T03:49:37.589477+0000","last_clean":"2024-09-17T03:49:37.589477+0000","last_became_active":"2024-09-17T03:49:35.390970+0000","last_became_peered":"2024-09-17T03:49:35.390970+0000","last_unstale":"2024-09-17T03:49:37.589477+0000","last_undegraded":"2024-09-17T03:49:37.589477+0000","last_fullsized":"2024-09-17T03:49:37.589477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:45.058598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589296+0000","last_change":"2024-09-17T03:49:36.404871+0000","last_active":"2024-09-17T03:49:37.589296+0000","last_peered":"2024-09-17T03:49:37.589296+0000","last_clean":"2024-09-17T03:49:37.589296+0000","last_became_active":"2024-09-17T03:49:36.404452+0000","last_became_peered":"2024-09-17T03:49:36.404452+0000","last_unstale":"2024-09-17T03:49:37.589296+0000","last_undegraded":"2024-09-17T03:49:37.589296+0000","last_fullsized":"2024-09-17T03:49:37.589296+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378971+0000","last_change":"2024-09-17T03:49:35.388702+0000","last_active":"2024-09-17T03:49:37.378971+0000","last_peered":"2024-09-17T03:49:37.378971+0000","last_clean":"2024-09-17T03:49:37.378971+0000","last_became_active":"2024-09-17T03:49:35.388337+0000","last_became_peered":"2024-09-17T03:49:35.388337+0000","last_unstale":"2024-09-17T03:49:37.378971+0000","last_undegraded":"2024-09-17T03:49:37.378971+0000","last_fullsized":"2024-09-17T03:49:37.378971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377377+0000","last_change":"2024-09-17T03:49:36.394908+0000","last_active":"2024-09-17T03:49:37.377377+0000","last_peered":"2024-09-17T03:49:37.377377+0000","last_clean":"2024-09-17T03:49:37.377377+0000","last_became_active":"2024-09-17T03:49:36.394518+0000","last_became_peered":"2024-09-17T03:49:36.394518+0000","last_unstale":"2024-09-17T03:49:37.377377+0000","last_undegraded":"2024-09-17T03:49:37.377377+0000","last_fullsized":"2024-09-17T03:49:37.377377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420535+0000","last_change":"2024-09-17T03:49:35.389956+0000","last_active":"2024-09-17T03:49:37.420535+0000","last_peered":"2024-09-17T03:49:37.420535+0000","last_clean":"2024-09-17T03:49:37.420535+0000","last_became_active":"2024-09-17T03:49:35.389588+0000","last_became_peered":"2024-09-17T03:49:35.389588+0000","last_unstale":"2024-09-17T03:49:37.420535+0000","last_undegraded":"2024-09-17T03:49:37.420535+0000","last_fullsized":"2024-09-17T03:49:37.420535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.510096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380718+0000","last_change":"2024-09-17T03:49:36.405737+0000","last_active":"2024-09-17T03:49:37.380718+0000","last_peered":"2024-09-17T03:49:37.380718+0000","last_clean":"2024-09-17T03:49:37.380718+0000","last_became_active":"2024-09-17T03:49:36.401772+0000","last_became_peered":"2024-09-17T03:49:36.401772+0000","last_unstale":"2024-09-17T03:49:37.380718+0000","last_undegraded":"2024-09-17T03:49:37.380718+0000","last_fullsized":"2024-09-17T03:49:37.380718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380418+0000","last_change":"2024-09-17T03:49:35.395265+0000","last_active":"2024-09-17T03:49:37.380418+0000","last_peered":"2024-09-17T03:49:37.380418+0000","last_clean":"2024-09-17T03:49:37.380418+0000","last_became_active":"2024-09-17T03:49:35.394870+0000","last_became_peered":"2024-09-17T03:49:35.394870+0000","last_unstale":"2024-09-17T03:49:37.380418+0000","last_undegraded":"2024-09-17T03:49:37.380418+0000","last_fullsized":"2024-09-17T03:49:37.380418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590151+0000","last_change":"2024-09-17T03:49:36.407432+0000","last_active":"2024-09-17T03:49:37.590151+0000","last_peered":"2024-09-17T03:49:37.590151+0000","last_clean":"2024-09-17T03:49:37.590151+0000","last_became_active":"2024-09-17T03:49:36.407158+0000","last_became_peered":"2024-09-17T03:49:36.407158+0000","last_unstale":"2024-09-17T03:49:37.590151+0000","last_undegraded":"2024-09-17T03:49:37.590151+0000","last_fullsized":"2024-09-17T03:49:37.590151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377583+0000","last_change":"2024-09-17T03:49:35.390102+0000","last_active":"2024-09-17T03:49:37.377583+0000","last_peered":"2024-09-17T03:49:37.377583+0000","last_clean":"2024-09-17T03:49:37.377583+0000","last_became_active":"2024-09-17T03:49:35.389698+0000","last_became_peered":"2024-09-17T03:49:35.389698+0000","last_unstale":"2024-09-17T03:49:37.377583+0000","last_undegraded":"2024-09-17T03:49:37.377583+0000","last_fullsized":"2024-09-17T03:49:37.377583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379524+0000","last_change":"2024-09-17T03:49:36.406033+0000","last_active":"2024-09-17T03:49:37.379524+0000","last_peered":"2024-09-17T03:49:37.379524+0000","last_clean":"2024-09-17T03:49:37.379524+0000","last_became_active":"2024-09-17T03:49:36.405391+0000","last_became_peered":"2024-09-17T03:49:36.405391+0000","last_unstale":"2024-09-17T03:49:37.379524+0000","last_undegraded":"2024-09-17T03:49:37.379524+0000","last_fullsized":"2024-09-17T03:49:37.379524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:35.463684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420327+0000","last_change":"2024-09-17T03:49:35.392778+0000","last_active":"2024-09-17T03:49:37.420327+0000","last_peered":"2024-09-17T03:49:37.420327+0000","last_clean":"2024-09-17T03:49:37.420327+0000","last_became_active":"2024-09-17T03:49:35.391530+0000","last_became_peered":"2024-09-17T03:49:35.391530+0000","last_unstale":"2024-09-17T03:49:37.420327+0000","last_undegraded":"2024-09-17T03:49:37.420327+0000","last_fullsized":"2024-09-17T03:49:37.420327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:09.206550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588902+0000","last_change":"2024-09-17T03:49:36.404199+0000","last_active":"2024-09-17T03:49:37.588902+0000","last_peered":"2024-09-17T03:49:37.588902+0000","last_clean":"2024-09-17T03:49:37.588902+0000","last_became_active":"2024-09-17T03:49:36.398455+0000","last_became_peered":"2024-09-17T03:49:36.398455+0000","last_unstale":"2024-09-17T03:49:37.588902+0000","last_undegraded":"2024-09-17T03:49:37.588902+0000","last_fullsized":"2024-09-17T03:49:37.588902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588954+0000","last_change":"2024-09-17T03:49:35.387607+0000","last_active":"2024-09-17T03:49:37.588954+0000","last_peered":"2024-09-17T03:49:37.588954+0000","last_clean":"2024-09-17T03:49:37.588954+0000","last_became_active":"2024-09-17T03:49:35.387303+0000","last_became_peered":"2024-09-17T03:49:35.387303+0000","last_unstale":"2024-09-17T03:49:37.588954+0000","last_undegraded":"2024-09-17T03:49:37.588954+0000","last_fullsized":"2024-09-17T03:49:37.588954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:33.456173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588883+0000","last_change":"2024-09-17T03:49:36.413174+0000","last_active":"2024-09-17T03:49:37.588883+0000","last_peered":"2024-09-17T03:49:37.588883+0000","last_clean":"2024-09-17T03:49:37.588883+0000","last_became_active":"2024-09-17T03:49:36.412943+0000","last_became_peered":"2024-09-17T03:49:36.412943+0000","last_unstale":"2024-09-17T03:49:37.588883+0000","last_undegraded":"2024-09-17T03:49:37.588883+0000","last_fullsized":"2024-09-17T03:49:37.588883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379966+0000","last_change":"2024-09-17T03:49:35.388456+0000","last_active":"2024-09-17T03:49:37.379966+0000","last_peered":"2024-09-17T03:49:37.379966+0000","last_clean":"2024-09-17T03:49:37.379966+0000","last_became_active":"2024-09-17T03:49:35.388083+0000","last_became_peered":"2024-09-17T03:49:35.388083+0000","last_unstale":"2024-09-17T03:49:37.379966+0000","last_undegraded":"2024-09-17T03:49:37.379966+0000","last_fullsized":"2024-09-17T03:49:37.379966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588927+0000","last_change":"2024-09-17T03:49:36.414542+0000","last_active":"2024-09-17T03:49:37.588927+0000","last_peered":"2024-09-17T03:49:37.588927+0000","last_clean":"2024-09-17T03:49:37.588927+0000","last_became_active":"2024-09-17T03:49:36.414224+0000","last_became_peered":"2024-09-17T03:49:36.414224+0000","last_unstale":"2024-09-17T03:49:37.588927+0000","last_undegraded":"2024-09-17T03:49:37.588927+0000","last_fullsized":"2024-09-17T03:49:37.588927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420647+0000","last_change":"2024-09-17T03:49:35.388635+0000","last_active":"2024-09-17T03:49:37.420647+0000","last_peered":"2024-09-17T03:49:37.420647+0000","last_clean":"2024-09-17T03:49:37.420647+0000","last_became_active":"2024-09-17T03:49:35.388278+0000","last_became_peered":"2024-09-17T03:49:35.388278+0000","last_unstale":"2024-09-17T03:49:37.420647+0000","last_undegraded":"2024-09-17T03:49:37.420647+0000","last_fullsized":"2024-09-17T03:49:37.420647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:17.648757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420254+0000","last_change":"2024-09-17T03:49:36.401308+0000","last_active":"2024-09-17T03:49:37.420254+0000","last_peered":"2024-09-17T03:49:37.420254+0000","last_clean":"2024-09-17T03:49:37.420254+0000","last_became_active":"2024-09-17T03:49:36.400942+0000","last_became_peered":"2024-09-17T03:49:36.400942+0000","last_unstale":"2024-09-17T03:49:37.420254+0000","last_undegraded":"2024-09-17T03:49:37.420254+0000","last_fullsized":"2024-09-17T03:49:37.420254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588398+0000","last_change":"2024-09-17T03:49:35.372728+0000","last_active":"2024-09-17T03:49:37.588398+0000","last_peered":"2024-09-17T03:49:37.588398+0000","last_clean":"2024-09-17T03:49:37.588398+0000","last_became_active":"2024-09-17T03:49:35.372413+0000","last_became_peered":"2024-09-17T03:49:35.372413+0000","last_unstale":"2024-09-17T03:49:37.588398+0000","last_undegraded":"2024-09-17T03:49:37.588398+0000","last_fullsized":"2024-09-17T03:49:37.588398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588789+0000","last_change":"2024-09-17T03:49:36.413936+0000","last_active":"2024-09-17T03:49:37.588789+0000","last_peered":"2024-09-17T03:49:37.588789+0000","last_clean":"2024-09-17T03:49:37.588789+0000","last_became_active":"2024-09-17T03:49:36.402860+0000","last_became_peered":"2024-09-17T03:49:36.402860+0000","last_unstale":"2024-09-17T03:49:37.588789+0000","last_undegraded":"2024-09-17T03:49:37.588789+0000","last_fullsized":"2024-09-17T03:49:37.588789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380504+0000","last_change":"2024-09-17T03:49:35.385498+0000","last_active":"2024-09-17T03:49:37.380504+0000","last_peered":"2024-09-17T03:49:37.380504+0000","last_clean":"2024-09-17T03:49:37.380504+0000","last_became_active":"2024-09-17T03:49:35.385212+0000","last_became_peered":"2024-09-17T03:49:35.385212+0000","last_unstale":"2024-09-17T03:49:37.380504+0000","last_undegraded":"2024-09-17T03:49:37.380504+0000","last_fullsized":"2024-09-17T03:49:37.380504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:45.909870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589167+0000","last_change":"2024-09-17T03:49:36.413538+0000","last_active":"2024-09-17T03:49:37.589167+0000","last_peered":"2024-09-17T03:49:37.589167+0000","last_clean":"2024-09-17T03:49:37.589167+0000","last_became_active":"2024-09-17T03:49:36.413197+0000","last_became_peered":"2024-09-17T03:49:36.413197+0000","last_unstale":"2024-09-17T03:49:37.589167+0000","last_undegraded":"2024-09-17T03:49:37.589167+0000","last_fullsized":"2024-09-17T03:49:37.589167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588219+0000","last_change":"2024-09-17T03:49:35.384632+0000","last_active":"2024-09-17T03:49:37.588219+0000","last_peered":"2024-09-17T03:49:37.588219+0000","last_clean":"2024-09-17T03:49:37.588219+0000","last_became_active":"2024-09-17T03:49:35.384263+0000","last_became_peered":"2024-09-17T03:49:35.384263+0000","last_unstale":"2024-09-17T03:49:37.588219+0000","last_undegraded":"2024-09-17T03:49:37.588219+0000","last_fullsized":"2024-09-17T03:49:37.588219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589537+0000","last_change":"2024-09-17T03:49:36.415593+0000","last_active":"2024-09-17T03:49:37.589537+0000","last_peered":"2024-09-17T03:49:37.589537+0000","last_clean":"2024-09-17T03:49:37.589537+0000","last_became_active":"2024-09-17T03:49:36.411554+0000","last_became_peered":"2024-09-17T03:49:36.411554+0000","last_unstale":"2024-09-17T03:49:37.589537+0000","last_undegraded":"2024-09-17T03:49:37.589537+0000","last_fullsized":"2024-09-17T03:49:37.589537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:13.481011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377691+0000","last_change":"2024-09-17T03:49:35.387003+0000","last_active":"2024-09-17T03:49:37.377691+0000","last_peered":"2024-09-17T03:49:37.377691+0000","last_clean":"2024-09-17T03:49:37.377691+0000","last_became_active":"2024-09-17T03:49:35.386654+0000","last_became_peered":"2024-09-17T03:49:35.386654+0000","last_unstale":"2024-09-17T03:49:37.377691+0000","last_undegraded":"2024-09-17T03:49:37.377691+0000","last_fullsized":"2024-09-17T03:49:37.377691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:23.004272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379918+0000","last_change":"2024-09-17T03:49:36.395286+0000","last_active":"2024-09-17T03:49:37.379918+0000","last_peered":"2024-09-17T03:49:37.379918+0000","last_clean":"2024-09-17T03:49:37.379918+0000","last_became_active":"2024-09-17T03:49:36.394025+0000","last_became_peered":"2024-09-17T03:49:36.394025+0000","last_unstale":"2024-09-17T03:49:37.379918+0000","last_undegraded":"2024-09-17T03:49:37.379918+0000","last_fullsized":"2024-09-17T03:49:37.379918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588329+0000","last_change":"2024-09-17T03:49:35.389511+0000","last_active":"2024-09-17T03:49:37.588329+0000","last_peered":"2024-09-17T03:49:37.588329+0000","last_clean":"2024-09-17T03:49:37.588329+0000","last_became_active":"2024-09-17T03:49:35.389061+0000","last_became_peered":"2024-09-17T03:49:35.389061+0000","last_unstale":"2024-09-17T03:49:37.588329+0000","last_undegraded":"2024-09-17T03:49:37.588329+0000","last_fullsized":"2024-09-17T03:49:37.588329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589110+0000","last_change":"2024-09-17T03:49:36.418896+0000","last_active":"2024-09-17T03:49:37.589110+0000","last_peered":"2024-09-17T03:49:37.589110+0000","last_clean":"2024-09-17T03:49:37.589110+0000","last_became_active":"2024-09-17T03:49:36.418567+0000","last_became_peered":"2024-09-17T03:49:36.418567+0000","last_unstale":"2024-09-17T03:49:37.589110+0000","last_undegraded":"2024-09-17T03:49:37.589110+0000","last_fullsized":"2024-09-17T03:49:37.589110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588519+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:49:37.588519+0000","last_peered":"2024-09-17T03:49:37.588519+0000","last_clean":"2024-09-17T03:49:37.588519+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:49:37.588519+0000","last_undegraded":"2024-09-17T03:49:37.588519+0000","last_fullsized":"2024-09-17T03:49:37.588519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378615+0000","last_change":"2024-09-17T03:49:35.379012+0000","last_active":"2024-09-17T03:49:37.378615+0000","last_peered":"2024-09-17T03:49:37.378615+0000","last_clean":"2024-09-17T03:49:37.378615+0000","last_became_active":"2024-09-17T03:49:35.378349+0000","last_became_peered":"2024-09-17T03:49:35.378349+0000","last_unstale":"2024-09-17T03:49:37.378615+0000","last_undegraded":"2024-09-17T03:49:37.378615+0000","last_fullsized":"2024-09-17T03:49:37.378615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589522+0000","last_change":"2024-09-17T03:49:36.408105+0000","last_active":"2024-09-17T03:49:37.589522+0000","last_peered":"2024-09-17T03:49:37.589522+0000","last_clean":"2024-09-17T03:49:37.589522+0000","last_became_active":"2024-09-17T03:49:36.407814+0000","last_became_peered":"2024-09-17T03:49:36.407814+0000","last_unstale":"2024-09-17T03:49:37.589522+0000","last_undegraded":"2024-09-17T03:49:37.589522+0000","last_fullsized":"2024-09-17T03:49:37.589522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588438+0000","last_change":"2024-09-17T03:49:35.399235+0000","last_active":"2024-09-17T03:49:37.588438+0000","last_peered":"2024-09-17T03:49:37.588438+0000","last_clean":"2024-09-17T03:49:37.588438+0000","last_became_active":"2024-09-17T03:49:35.398860+0000","last_became_peered":"2024-09-17T03:49:35.398860+0000","last_unstale":"2024-09-17T03:49:37.588438+0000","last_undegraded":"2024-09-17T03:49:37.588438+0000","last_fullsized":"2024-09-17T03:49:37.588438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:34.455037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588130+0000","last_change":"2024-09-17T03:49:36.403778+0000","last_active":"2024-09-17T03:49:37.588130+0000","last_peered":"2024-09-17T03:49:37.588130+0000","last_clean":"2024-09-17T03:49:37.588130+0000","last_became_active":"2024-09-17T03:49:36.403408+0000","last_became_peered":"2024-09-17T03:49:36.403408+0000","last_unstale":"2024-09-17T03:49:37.588130+0000","last_undegraded":"2024-09-17T03:49:37.588130+0000","last_fullsized":"2024-09-17T03:49:37.588130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:19.846081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.587984+0000","last_change":"2024-09-17T03:49:35.384616+0000","last_active":"2024-09-17T03:49:37.587984+0000","last_peered":"2024-09-17T03:49:37.587984+0000","last_clean":"2024-09-17T03:49:37.587984+0000","last_became_active":"2024-09-17T03:49:35.384264+0000","last_became_peered":"2024-09-17T03:49:35.384264+0000","last_unstale":"2024-09-17T03:49:37.587984+0000","last_undegraded":"2024-09-17T03:49:37.587984+0000","last_fullsized":"2024-09-17T03:49:37.587984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377132+0000","last_change":"2024-09-17T03:49:36.381928+0000","last_active":"2024-09-17T03:49:37.377132+0000","last_peered":"2024-09-17T03:49:37.377132+0000","last_clean":"2024-09-17T03:49:37.377132+0000","last_became_active":"2024-09-17T03:49:36.381629+0000","last_became_peered":"2024-09-17T03:49:36.381629+0000","last_unstale":"2024-09-17T03:49:37.377132+0000","last_undegraded":"2024-09-17T03:49:37.377132+0000","last_fullsized":"2024-09-17T03:49:37.377132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379997+0000","last_change":"2024-09-17T03:49:35.396687+0000","last_active":"2024-09-17T03:49:37.379997+0000","last_peered":"2024-09-17T03:49:37.379997+0000","last_clean":"2024-09-17T03:49:37.379997+0000","last_became_active":"2024-09-17T03:49:35.390958+0000","last_became_peered":"2024-09-17T03:49:35.390958+0000","last_unstale":"2024-09-17T03:49:37.379997+0000","last_undegraded":"2024-09-17T03:49:37.379997+0000","last_fullsized":"2024-09-17T03:49:37.379997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378943+0000","last_change":"2024-09-17T03:49:36.406614+0000","last_active":"2024-09-17T03:49:37.378943+0000","last_peered":"2024-09-17T03:49:37.378943+0000","last_clean":"2024-09-17T03:49:37.378943+0000","last_became_active":"2024-09-17T03:49:36.403577+0000","last_became_peered":"2024-09-17T03:49:36.403577+0000","last_unstale":"2024-09-17T03:49:37.378943+0000","last_undegraded":"2024-09-17T03:49:37.378943+0000","last_fullsized":"2024-09-17T03:49:37.378943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:45.693650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377460+0000","last_change":"2024-09-17T03:49:35.385825+0000","last_active":"2024-09-17T03:49:37.377460+0000","last_peered":"2024-09-17T03:49:37.377460+0000","last_clean":"2024-09-17T03:49:37.377460+0000","last_became_active":"2024-09-17T03:49:35.385489+0000","last_became_peered":"2024-09-17T03:49:35.385489+0000","last_unstale":"2024-09-17T03:49:37.377460+0000","last_undegraded":"2024-09-17T03:49:37.377460+0000","last_fullsized":"2024-09-17T03:49:37.377460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380572+0000","last_change":"2024-09-17T03:49:36.413827+0000","last_active":"2024-09-17T03:49:37.380572+0000","last_peered":"2024-09-17T03:49:37.380572+0000","last_clean":"2024-09-17T03:49:37.380572+0000","last_became_active":"2024-09-17T03:49:36.413432+0000","last_became_peered":"2024-09-17T03:49:36.413432+0000","last_unstale":"2024-09-17T03:49:37.380572+0000","last_undegraded":"2024-09-17T03:49:37.380572+0000","last_fullsized":"2024-09-17T03:49:37.380572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589281+0000","last_change":"2024-09-17T03:49:35.396951+0000","last_active":"2024-09-17T03:49:37.589281+0000","last_peered":"2024-09-17T03:49:37.589281+0000","last_clean":"2024-09-17T03:49:37.589281+0000","last_became_active":"2024-09-17T03:49:35.396554+0000","last_became_peered":"2024-09-17T03:49:35.396554+0000","last_unstale":"2024-09-17T03:49:37.589281+0000","last_undegraded":"2024-09-17T03:49:37.589281+0000","last_fullsized":"2024-09-17T03:49:37.589281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:23.665085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377339+0000","last_change":"2024-09-17T03:49:36.387372+0000","last_active":"2024-09-17T03:49:37.377339+0000","last_peered":"2024-09-17T03:49:37.377339+0000","last_clean":"2024-09-17T03:49:37.377339+0000","last_became_active":"2024-09-17T03:49:36.384715+0000","last_became_peered":"2024-09-17T03:49:36.384715+0000","last_unstale":"2024-09-17T03:49:37.377339+0000","last_undegraded":"2024-09-17T03:49:37.377339+0000","last_fullsized":"2024-09-17T03:49:37.377339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.419437+0000","last_change":"2024-09-17T03:49:35.393853+0000","last_active":"2024-09-17T03:49:37.419437+0000","last_peered":"2024-09-17T03:49:37.419437+0000","last_clean":"2024-09-17T03:49:37.419437+0000","last_became_active":"2024-09-17T03:49:35.393551+0000","last_became_peered":"2024-09-17T03:49:35.393551+0000","last_unstale":"2024-09-17T03:49:37.419437+0000","last_undegraded":"2024-09-17T03:49:37.419437+0000","last_fullsized":"2024-09-17T03:49:37.419437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590925+0000","last_change":"2024-09-17T03:49:36.412628+0000","last_active":"2024-09-17T03:49:37.590925+0000","last_peered":"2024-09-17T03:49:37.590925+0000","last_clean":"2024-09-17T03:49:37.590925+0000","last_became_active":"2024-09-17T03:49:36.412320+0000","last_became_peered":"2024-09-17T03:49:36.412320+0000","last_unstale":"2024-09-17T03:49:37.590925+0000","last_undegraded":"2024-09-17T03:49:37.590925+0000","last_fullsized":"2024-09-17T03:49:37.590925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420603+0000","last_change":"2024-09-17T03:49:35.389093+0000","last_active":"2024-09-17T03:49:37.420603+0000","last_peered":"2024-09-17T03:49:37.420603+0000","last_clean":"2024-09-17T03:49:37.420603+0000","last_became_active":"2024-09-17T03:49:35.388625+0000","last_became_peered":"2024-09-17T03:49:35.388625+0000","last_unstale":"2024-09-17T03:49:37.420603+0000","last_undegraded":"2024-09-17T03:49:37.420603+0000","last_fullsized":"2024-09-17T03:49:37.420603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.987962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589830+0000","last_change":"2024-09-17T03:49:36.397328+0000","last_active":"2024-09-17T03:49:37.589830+0000","last_peered":"2024-09-17T03:49:37.589830+0000","last_clean":"2024-09-17T03:49:37.589830+0000","last_became_active":"2024-09-17T03:49:36.396910+0000","last_became_peered":"2024-09-17T03:49:36.396910+0000","last_unstale":"2024-09-17T03:49:37.589830+0000","last_undegraded":"2024-09-17T03:49:37.589830+0000","last_fullsized":"2024-09-17T03:49:37.589830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588441+0000","last_change":"2024-09-17T03:49:35.391714+0000","last_active":"2024-09-17T03:49:37.588441+0000","last_peered":"2024-09-17T03:49:37.588441+0000","last_clean":"2024-09-17T03:49:37.588441+0000","last_became_active":"2024-09-17T03:49:35.391374+0000","last_became_peered":"2024-09-17T03:49:35.391374+0000","last_unstale":"2024-09-17T03:49:37.588441+0000","last_undegraded":"2024-09-17T03:49:37.588441+0000","last_fullsized":"2024-09-17T03:49:37.588441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380255+0000","last_change":"2024-09-17T03:49:36.395381+0000","last_active":"2024-09-17T03:49:37.380255+0000","last_peered":"2024-09-17T03:49:37.380255+0000","last_clean":"2024-09-17T03:49:37.380255+0000","last_became_active":"2024-09-17T03:49:36.394430+0000","last_became_peered":"2024-09-17T03:49:36.394430+0000","last_unstale":"2024-09-17T03:49:37.380255+0000","last_undegraded":"2024-09-17T03:49:37.380255+0000","last_fullsized":"2024-09-17T03:49:37.380255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377304+0000","last_change":"2024-09-17T03:49:35.386828+0000","last_active":"2024-09-17T03:49:37.377304+0000","last_peered":"2024-09-17T03:49:37.377304+0000","last_clean":"2024-09-17T03:49:37.377304+0000","last_became_active":"2024-09-17T03:49:35.385425+0000","last_became_peered":"2024-09-17T03:49:35.385425+0000","last_unstale":"2024-09-17T03:49:37.377304+0000","last_undegraded":"2024-09-17T03:49:37.377304+0000","last_fullsized":"2024-09-17T03:49:37.377304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589280+0000","last_change":"2024-09-17T03:49:36.396082+0000","last_active":"2024-09-17T03:49:37.589280+0000","last_peered":"2024-09-17T03:49:37.589280+0000","last_clean":"2024-09-17T03:49:37.589280+0000","last_became_active":"2024-09-17T03:49:36.395733+0000","last_became_peered":"2024-09-17T03:49:36.395733+0000","last_unstale":"2024-09-17T03:49:37.589280+0000","last_undegraded":"2024-09-17T03:49:37.589280+0000","last_fullsized":"2024-09-17T03:49:37.589280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377167+0000","last_change":"2024-09-17T03:49:35.383855+0000","last_active":"2024-09-17T03:49:37.377167+0000","last_peered":"2024-09-17T03:49:37.377167+0000","last_clean":"2024-09-17T03:49:37.377167+0000","last_became_active":"2024-09-17T03:49:35.383513+0000","last_became_peered":"2024-09-17T03:49:35.383513+0000","last_unstale":"2024-09-17T03:49:37.377167+0000","last_undegraded":"2024-09-17T03:49:37.377167+0000","last_fullsized":"2024-09-17T03:49:37.377167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589610+0000","last_change":"2024-09-17T03:49:36.401383+0000","last_active":"2024-09-17T03:49:37.589610+0000","last_peered":"2024-09-17T03:49:37.589610+0000","last_clean":"2024-09-17T03:49:37.589610+0000","last_became_active":"2024-09-17T03:49:36.401112+0000","last_became_peered":"2024-09-17T03:49:36.401112+0000","last_unstale":"2024-09-17T03:49:37.589610+0000","last_undegraded":"2024-09-17T03:49:37.589610+0000","last_fullsized":"2024-09-17T03:49:37.589610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589087+0000","last_change":"2024-09-17T03:49:35.405334+0000","last_active":"2024-09-17T03:49:37.589087+0000","last_peered":"2024-09-17T03:49:37.589087+0000","last_clean":"2024-09-17T03:49:37.589087+0000","last_became_active":"2024-09-17T03:49:35.404955+0000","last_became_peered":"2024-09-17T03:49:35.404955+0000","last_unstale":"2024-09-17T03:49:37.589087+0000","last_undegraded":"2024-09-17T03:49:37.589087+0000","last_fullsized":"2024-09-17T03:49:37.589087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420485+0000","last_change":"2024-09-17T03:49:36.412896+0000","last_active":"2024-09-17T03:49:37.420485+0000","last_peered":"2024-09-17T03:49:37.420485+0000","last_clean":"2024-09-17T03:49:37.420485+0000","last_became_active":"2024-09-17T03:49:36.406254+0000","last_became_peered":"2024-09-17T03:49:36.406254+0000","last_unstale":"2024-09-17T03:49:37.420485+0000","last_undegraded":"2024-09-17T03:49:37.420485+0000","last_fullsized":"2024-09-17T03:49:37.420485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380554+0000","last_change":"2024-09-17T03:49:36.396792+0000","last_active":"2024-09-17T03:49:37.380554+0000","last_peered":"2024-09-17T03:49:37.380554+0000","last_clean":"2024-09-17T03:49:37.380554+0000","last_became_active":"2024-09-17T03:49:36.393651+0000","last_became_peered":"2024-09-17T03:49:36.393651+0000","last_unstale":"2024-09-17T03:49:37.380554+0000","last_undegraded":"2024-09-17T03:49:37.380554+0000","last_fullsized":"2024-09-17T03:49:37.380554+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588722+0000","last_change":"2024-09-17T03:49:35.398891+0000","last_active":"2024-09-17T03:49:37.588722+0000","last_peered":"2024-09-17T03:49:37.588722+0000","last_clean":"2024-09-17T03:49:37.588722+0000","last_became_active":"2024-09-17T03:49:35.398494+0000","last_became_peered":"2024-09-17T03:49:35.398494+0000","last_unstale":"2024-09-17T03:49:37.588722+0000","last_undegraded":"2024-09-17T03:49:37.588722+0000","last_fullsized":"2024-09-17T03:49:37.588722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:34.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379744+0000","last_change":"2024-09-17T03:49:36.394357+0000","last_active":"2024-09-17T03:49:37.379744+0000","last_peered":"2024-09-17T03:49:37.379744+0000","last_clean":"2024-09-17T03:49:37.379744+0000","last_became_active":"2024-09-17T03:49:36.393549+0000","last_became_peered":"2024-09-17T03:49:36.393549+0000","last_unstale":"2024-09-17T03:49:37.379744+0000","last_undegraded":"2024-09-17T03:49:37.379744+0000","last_fullsized":"2024-09-17T03:49:37.379744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589049+0000","last_change":"2024-09-17T03:49:35.398839+0000","last_active":"2024-09-17T03:49:37.589049+0000","last_peered":"2024-09-17T03:49:37.589049+0000","last_clean":"2024-09-17T03:49:37.589049+0000","last_became_active":"2024-09-17T03:49:35.398355+0000","last_became_peered":"2024-09-17T03:49:35.398355+0000","last_unstale":"2024-09-17T03:49:37.589049+0000","last_undegraded":"2024-09-17T03:49:37.589049+0000","last_fullsized":"2024-09-17T03:49:37.589049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:51.292462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589985+0000","last_change":"2024-09-17T03:49:36.409308+0000","last_active":"2024-09-17T03:49:37.589985+0000","last_peered":"2024-09-17T03:49:37.589985+0000","last_clean":"2024-09-17T03:49:37.589985+0000","last_became_active":"2024-09-17T03:49:36.408373+0000","last_became_peered":"2024-09-17T03:49:36.408373+0000","last_unstale":"2024-09-17T03:49:37.589985+0000","last_undegraded":"2024-09-17T03:49:37.589985+0000","last_fullsized":"2024-09-17T03:49:37.589985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590572+0000","last_change":"2024-09-17T03:49:35.381739+0000","last_active":"2024-09-17T03:49:37.590572+0000","last_peered":"2024-09-17T03:49:37.590572+0000","last_clean":"2024-09-17T03:49:37.590572+0000","last_became_active":"2024-09-17T03:49:35.381185+0000","last_became_peered":"2024-09-17T03:49:35.381185+0000","last_unstale":"2024-09-17T03:49:37.590572+0000","last_undegraded":"2024-09-17T03:49:37.590572+0000","last_fullsized":"2024-09-17T03:49:37.590572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590630+0000","last_change":"2024-09-17T03:49:36.401989+0000","last_active":"2024-09-17T03:49:37.590630+0000","last_peered":"2024-09-17T03:49:37.590630+0000","last_clean":"2024-09-17T03:49:37.590630+0000","last_became_active":"2024-09-17T03:49:36.399727+0000","last_became_peered":"2024-09-17T03:49:36.399727+0000","last_unstale":"2024-09-17T03:49:37.590630+0000","last_undegraded":"2024-09-17T03:49:37.590630+0000","last_fullsized":"2024-09-17T03:49:37.590630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590102+0000","last_change":"2024-09-17T03:49:35.386738+0000","last_active":"2024-09-17T03:49:37.590102+0000","last_peered":"2024-09-17T03:49:37.590102+0000","last_clean":"2024-09-17T03:49:37.590102+0000","last_became_active":"2024-09-17T03:49:35.386382+0000","last_became_peered":"2024-09-17T03:49:35.386382+0000","last_unstale":"2024-09-17T03:49:37.590102+0000","last_undegraded":"2024-09-17T03:49:37.590102+0000","last_fullsized":"2024-09-17T03:49:37.590102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:57.569543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":667808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":76984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":76858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":77018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":815104,"data_stored":661564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:49:40.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:49:40.981 INFO:tasks.ceph.mds.b.smithi181.stderr:2024-09-17T03:49:40.981+0000 7fdb55800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:49:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 -- 172.21.15.129:0/1185744998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0070440 msgr2=0x7f76a0074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 --2- 172.21.15.129:0/1185744998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0070440 0x7f76a0074b80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f76a0070e70 tx=0x7f769802a980 comp rx=0 tx=0).stop 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 -- 172.21.15.129:0/1185744998 shutdown_connections 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 --2- 172.21.15.129:0/1185744998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0070440 0x7f76a0074b80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 --2- 172.21.15.129:0/1185744998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070030 0x7f76a0074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 --2- 172.21.15.129:0/1185744998 >> v2:172.21.15.181:3300/0 conn(0x7f76a0073170 0x7f76a015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 -- 172.21.15.129:0/1185744998 >> 172.21.15.129:0/1185744998 conn(0x7f76a00b1cb0 msgr2=0x7f76a00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 -- 172.21.15.129:0/1185744998 shutdown_connections 2024-09-17T03:49:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.054+0000 7f76a7c00640 1 -- 172.21.15.129:0/1185744998 wait complete. 2024-09-17T03:49:41.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.057+0000 7f76a7c00640 1 Processor -- start 2024-09-17T03:49:41.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.057+0000 7f76a7c00640 1 -- start start 2024-09-17T03:49:41.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76a0070030 0x7f76a01f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070440 0x7f76a01f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76a006f690 con 0x7f76a0073170 2024-09-17T03:49:41.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f76a006f510 con 0x7f76a0070030 2024-09-17T03:49:41.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a7c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f76a006f810 con 0x7f76a0070440 2024-09-17T03:49:41.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39500/0 (socket says 172.21.15.129:39500) 2024-09-17T03:49:41.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.058+0000 7f76a5c00640 1 -- 172.21.15.129:0/3744628200 learned_addr learned my addr 172.21.15.129:0/3744628200 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5200640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76a0070030 0x7f76a01f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f769fe00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070440 0x7f76a01f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 -- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070440 msgr2=0x7f76a01f3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070440 0x7f76a01f3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 -- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76a0070030 msgr2=0x7f76a01f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76a0070030 0x7f76a01f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 -- 172.21.15.129:0/3744628200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a009d710 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.059+0000 7f76a5c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7698033040 tx=0x7f7698003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.060+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698034040 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.060+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7698004540 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.060+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698045480 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.060+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a009b650 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.060+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a009bc10 con 0x7f76a0073170 2024-09-17T03:49:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.061+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a015f3d0 con 0x7f76a0073170 2024-09-17T03:49:41.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.062+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7698037020 con 0x7f76a0073170 2024-09-17T03:49:41.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.063+0000 7f769cc00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 0x7f768007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.064+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f76980bc590 con 0x7f76a0073170 2024-09-17T03:49:41.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.064+0000 7f76a5200640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 0x7f768007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.065+0000 7f76a5200640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 0x7f768007b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7694009a40 tx=0x7f7694005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.068+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7698084940 con 0x7f76a0073170 2024-09-17T03:49:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.248+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f76a00ae3c0 con 0x7f7680078b90 2024-09-17T03:49:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.253+0000 7f769cc00640 1 -- 172.21.15.129:0/3744628200 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251935 (secure 0 0 0) 0x7f76a00ae3c0 con 0x7f7680078b90 2024-09-17T03:49:41.255 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:41.256 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:49:41.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.257+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 msgr2=0x7f768007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.257+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 0x7f768007b0a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7694009a40 tx=0x7f7694005f90 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.257+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 msgr2=0x7f76a009c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.257+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7698033040 tx=0x7f7698003f10 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 shutdown_connections 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7680078b90 0x7f768007b0a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76a0073170 0x7f76a009c660 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76a0070440 0x7f76a01f3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 --2- 172.21.15.129:0/3744628200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76a0070030 0x7f76a01f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 >> 172.21.15.129:0/3744628200 conn(0x7f76a00b1cb0 msgr2=0x7f76a00ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 shutdown_connections 2024-09-17T03:49:41.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.258+0000 7f76a7c00640 1 -- 172.21.15.129:0/3744628200 wait complete. 2024-09-17T03:49:41.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-17T03:49:40.293750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223376,"kb_used_data":2940,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751344,"statfs":{"total":773094113280,"available":772865376256,"internally_reserved":0,"allocated":3010560,"data_stored":1794120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936396"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589196+0000","last_change":"2024-09-17T03:49:36.406020+0000","last_active":"2024-09-17T03:49:37.589196+0000","last_peered":"2024-09-17T03:49:37.589196+0000","last_clean":"2024-09-17T03:49:37.589196+0000","last_became_active":"2024-09-17T03:49:36.405649+0000","last_became_peered":"2024-09-17T03:49:36.405649+0000","last_unstale":"2024-09-17T03:49:37.589196+0000","last_undegraded":"2024-09-17T03:49:37.589196+0000","last_fullsized":"2024-09-17T03:49:37.589196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377354+0000","last_change":"2024-09-17T03:49:36.400461+0000","last_active":"2024-09-17T03:49:37.377354+0000","last_peered":"2024-09-17T03:49:37.377354+0000","last_clean":"2024-09-17T03:49:37.377354+0000","last_became_active":"2024-09-17T03:49:36.398210+0000","last_became_peered":"2024-09-17T03:49:36.398210+0000","last_unstale":"2024-09-17T03:49:37.377354+0000","last_undegraded":"2024-09-17T03:49:37.377354+0000","last_fullsized":"2024-09-17T03:49:37.377354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589718+0000","last_change":"2024-09-17T03:49:36.397732+0000","last_active":"2024-09-17T03:49:37.589718+0000","last_peered":"2024-09-17T03:49:37.589718+0000","last_clean":"2024-09-17T03:49:37.589718+0000","last_became_active":"2024-09-17T03:49:36.397407+0000","last_became_peered":"2024-09-17T03:49:36.397407+0000","last_unstale":"2024-09-17T03:49:37.589718+0000","last_undegraded":"2024-09-17T03:49:37.589718+0000","last_fullsized":"2024-09-17T03:49:37.589718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379346+0000","last_change":"2024-09-17T03:49:36.410331+0000","last_active":"2024-09-17T03:49:37.379346+0000","last_peered":"2024-09-17T03:49:37.379346+0000","last_clean":"2024-09-17T03:49:37.379346+0000","last_became_active":"2024-09-17T03:49:36.409931+0000","last_became_peered":"2024-09-17T03:49:36.409931+0000","last_unstale":"2024-09-17T03:49:37.379346+0000","last_undegraded":"2024-09-17T03:49:37.379346+0000","last_fullsized":"2024-09-17T03:49:37.379346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589345+0000","last_change":"2024-09-17T03:49:36.415397+0000","last_active":"2024-09-17T03:49:37.589345+0000","last_peered":"2024-09-17T03:49:37.589345+0000","last_clean":"2024-09-17T03:49:37.589345+0000","last_became_active":"2024-09-17T03:49:36.415075+0000","last_became_peered":"2024-09-17T03:49:36.415075+0000","last_unstale":"2024-09-17T03:49:37.589345+0000","last_undegraded":"2024-09-17T03:49:37.589345+0000","last_fullsized":"2024-09-17T03:49:37.589345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:08.390183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379503+0000","last_change":"2024-09-17T03:49:36.405656+0000","last_active":"2024-09-17T03:49:37.379503+0000","last_peered":"2024-09-17T03:49:37.379503+0000","last_clean":"2024-09-17T03:49:37.379503+0000","last_became_active":"2024-09-17T03:49:36.404978+0000","last_became_peered":"2024-09-17T03:49:36.404978+0000","last_unstale":"2024-09-17T03:49:37.379503+0000","last_undegraded":"2024-09-17T03:49:37.379503+0000","last_fullsized":"2024-09-17T03:49:37.379503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588643+0000","last_change":"2024-09-17T03:49:36.402961+0000","last_active":"2024-09-17T03:49:37.588643+0000","last_peered":"2024-09-17T03:49:37.588643+0000","last_clean":"2024-09-17T03:49:37.588643+0000","last_became_active":"2024-09-17T03:49:36.402413+0000","last_became_peered":"2024-09-17T03:49:36.402413+0000","last_unstale":"2024-09-17T03:49:37.588643+0000","last_undegraded":"2024-09-17T03:49:37.588643+0000","last_fullsized":"2024-09-17T03:49:37.588643+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:09.378052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588967+0000","last_change":"2024-09-17T03:49:36.409564+0000","last_active":"2024-09-17T03:49:37.588967+0000","last_peered":"2024-09-17T03:49:37.588967+0000","last_clean":"2024-09-17T03:49:37.588967+0000","last_became_active":"2024-09-17T03:49:36.409150+0000","last_became_peered":"2024-09-17T03:49:36.409150+0000","last_unstale":"2024-09-17T03:49:37.588967+0000","last_undegraded":"2024-09-17T03:49:37.588967+0000","last_fullsized":"2024-09-17T03:49:37.588967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:09.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589233+0000","last_change":"2024-09-17T03:49:36.416599+0000","last_active":"2024-09-17T03:49:37.589233+0000","last_peered":"2024-09-17T03:49:37.589233+0000","last_clean":"2024-09-17T03:49:37.589233+0000","last_became_active":"2024-09-17T03:49:36.416100+0000","last_became_peered":"2024-09-17T03:49:36.416100+0000","last_unstale":"2024-09-17T03:49:37.589233+0000","last_undegraded":"2024-09-17T03:49:37.589233+0000","last_fullsized":"2024-09-17T03:49:37.589233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420661+0000","last_change":"2024-09-17T03:49:36.412006+0000","last_active":"2024-09-17T03:49:37.420661+0000","last_peered":"2024-09-17T03:49:37.420661+0000","last_clean":"2024-09-17T03:49:37.420661+0000","last_became_active":"2024-09-17T03:49:36.411697+0000","last_became_peered":"2024-09-17T03:49:36.411697+0000","last_unstale":"2024-09-17T03:49:37.420661+0000","last_undegraded":"2024-09-17T03:49:37.420661+0000","last_fullsized":"2024-09-17T03:49:37.420661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420825+0000","last_change":"2024-09-17T03:49:36.415050+0000","last_active":"2024-09-17T03:49:37.420825+0000","last_peered":"2024-09-17T03:49:37.420825+0000","last_clean":"2024-09-17T03:49:37.420825+0000","last_became_active":"2024-09-17T03:49:36.414808+0000","last_became_peered":"2024-09-17T03:49:36.414808+0000","last_unstale":"2024-09-17T03:49:37.420825+0000","last_undegraded":"2024-09-17T03:49:37.420825+0000","last_fullsized":"2024-09-17T03:49:37.420825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379278+0000","last_change":"2024-09-17T03:49:36.406397+0000","last_active":"2024-09-17T03:49:37.379278+0000","last_peered":"2024-09-17T03:49:37.379278+0000","last_clean":"2024-09-17T03:49:37.379278+0000","last_became_active":"2024-09-17T03:49:36.405791+0000","last_became_peered":"2024-09-17T03:49:36.405791+0000","last_unstale":"2024-09-17T03:49:37.379278+0000","last_undegraded":"2024-09-17T03:49:37.379278+0000","last_fullsized":"2024-09-17T03:49:37.379278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:05.635714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589437+0000","last_change":"2024-09-17T03:49:36.393913+0000","last_active":"2024-09-17T03:49:37.589437+0000","last_peered":"2024-09-17T03:49:37.589437+0000","last_clean":"2024-09-17T03:49:37.589437+0000","last_became_active":"2024-09-17T03:49:36.393589+0000","last_became_peered":"2024-09-17T03:49:36.393589+0000","last_unstale":"2024-09-17T03:49:37.589437+0000","last_undegraded":"2024-09-17T03:49:37.589437+0000","last_fullsized":"2024-09-17T03:49:37.589437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588596+0000","last_change":"2024-09-17T03:49:36.401985+0000","last_active":"2024-09-17T03:49:37.588596+0000","last_peered":"2024-09-17T03:49:37.588596+0000","last_clean":"2024-09-17T03:49:37.588596+0000","last_became_active":"2024-09-17T03:49:36.401682+0000","last_became_peered":"2024-09-17T03:49:36.401682+0000","last_unstale":"2024-09-17T03:49:37.588596+0000","last_undegraded":"2024-09-17T03:49:37.588596+0000","last_fullsized":"2024-09-17T03:49:37.588596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:46.086705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377130+0000","last_change":"2024-09-17T03:49:36.389920+0000","last_active":"2024-09-17T03:49:37.377130+0000","last_peered":"2024-09-17T03:49:37.377130+0000","last_clean":"2024-09-17T03:49:37.377130+0000","last_became_active":"2024-09-17T03:49:36.389645+0000","last_became_peered":"2024-09-17T03:49:36.389645+0000","last_unstale":"2024-09-17T03:49:37.377130+0000","last_undegraded":"2024-09-17T03:49:37.377130+0000","last_fullsized":"2024-09-17T03:49:37.377130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:20.042459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378538+0000","last_change":"2024-09-17T03:49:36.403578+0000","last_active":"2024-09-17T03:49:37.378538+0000","last_peered":"2024-09-17T03:49:37.378538+0000","last_clean":"2024-09-17T03:49:37.378538+0000","last_became_active":"2024-09-17T03:49:36.403353+0000","last_became_peered":"2024-09-17T03:49:36.403353+0000","last_unstale":"2024-09-17T03:49:37.378538+0000","last_undegraded":"2024-09-17T03:49:37.378538+0000","last_fullsized":"2024-09-17T03:49:37.378538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380751+0000","last_change":"2024-09-17T03:49:36.406004+0000","last_active":"2024-09-17T03:49:37.380751+0000","last_peered":"2024-09-17T03:49:37.380751+0000","last_clean":"2024-09-17T03:49:37.380751+0000","last_became_active":"2024-09-17T03:49:36.403931+0000","last_became_peered":"2024-09-17T03:49:36.403931+0000","last_unstale":"2024-09-17T03:49:37.380751+0000","last_undegraded":"2024-09-17T03:49:37.380751+0000","last_fullsized":"2024-09-17T03:49:37.380751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588086+0000","last_change":"2024-09-17T03:49:36.396161+0000","last_active":"2024-09-17T03:49:37.588086+0000","last_peered":"2024-09-17T03:49:37.588086+0000","last_clean":"2024-09-17T03:49:37.588086+0000","last_became_active":"2024-09-17T03:49:36.395717+0000","last_became_peered":"2024-09-17T03:49:36.395717+0000","last_unstale":"2024-09-17T03:49:37.588086+0000","last_undegraded":"2024-09-17T03:49:37.588086+0000","last_fullsized":"2024-09-17T03:49:37.588086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:42.475745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589346+0000","last_change":"2024-09-17T03:49:36.410754+0000","last_active":"2024-09-17T03:49:37.589346+0000","last_peered":"2024-09-17T03:49:37.589346+0000","last_clean":"2024-09-17T03:49:37.589346+0000","last_became_active":"2024-09-17T03:49:36.404715+0000","last_became_peered":"2024-09-17T03:49:36.404715+0000","last_unstale":"2024-09-17T03:49:37.589346+0000","last_undegraded":"2024-09-17T03:49:37.589346+0000","last_fullsized":"2024-09-17T03:49:37.589346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:51.120762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420579+0000","last_change":"2024-09-17T03:49:36.409394+0000","last_active":"2024-09-17T03:49:37.420579+0000","last_peered":"2024-09-17T03:49:37.420579+0000","last_clean":"2024-09-17T03:49:37.420579+0000","last_became_active":"2024-09-17T03:49:36.408771+0000","last_became_peered":"2024-09-17T03:49:36.408771+0000","last_unstale":"2024-09-17T03:49:37.420579+0000","last_undegraded":"2024-09-17T03:49:37.420579+0000","last_fullsized":"2024-09-17T03:49:37.420579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588435+0000","last_change":"2024-09-17T03:49:36.422246+0000","last_active":"2024-09-17T03:49:37.588435+0000","last_peered":"2024-09-17T03:49:37.588435+0000","last_clean":"2024-09-17T03:49:37.588435+0000","last_became_active":"2024-09-17T03:49:36.421985+0000","last_became_peered":"2024-09-17T03:49:36.421985+0000","last_unstale":"2024-09-17T03:49:37.588435+0000","last_undegraded":"2024-09-17T03:49:37.588435+0000","last_fullsized":"2024-09-17T03:49:37.588435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:13.173976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379785+0000","last_change":"2024-09-17T03:49:36.405326+0000","last_active":"2024-09-17T03:49:37.379785+0000","last_peered":"2024-09-17T03:49:37.379785+0000","last_clean":"2024-09-17T03:49:37.379785+0000","last_became_active":"2024-09-17T03:49:36.404541+0000","last_became_peered":"2024-09-17T03:49:36.404541+0000","last_unstale":"2024-09-17T03:49:37.379785+0000","last_undegraded":"2024-09-17T03:49:37.379785+0000","last_fullsized":"2024-09-17T03:49:37.379785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589367+0000","last_change":"2024-09-17T03:49:36.406574+0000","last_active":"2024-09-17T03:49:37.589367+0000","last_peered":"2024-09-17T03:49:37.589367+0000","last_clean":"2024-09-17T03:49:37.589367+0000","last_became_active":"2024-09-17T03:49:36.406162+0000","last_became_peered":"2024-09-17T03:49:36.406162+0000","last_unstale":"2024-09-17T03:49:37.589367+0000","last_undegraded":"2024-09-17T03:49:37.589367+0000","last_fullsized":"2024-09-17T03:49:37.589367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378837+0000","last_change":"2024-09-17T03:49:36.394666+0000","last_active":"2024-09-17T03:49:37.378837+0000","last_peered":"2024-09-17T03:49:37.378837+0000","last_clean":"2024-09-17T03:49:37.378837+0000","last_became_active":"2024-09-17T03:49:36.394259+0000","last_became_peered":"2024-09-17T03:49:36.394259+0000","last_unstale":"2024-09-17T03:49:37.378837+0000","last_undegraded":"2024-09-17T03:49:37.378837+0000","last_fullsized":"2024-09-17T03:49:37.378837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590715+0000","last_change":"2024-09-17T03:49:36.410498+0000","last_active":"2024-09-17T03:49:37.590715+0000","last_peered":"2024-09-17T03:49:37.590715+0000","last_clean":"2024-09-17T03:49:37.590715+0000","last_became_active":"2024-09-17T03:49:36.410228+0000","last_became_peered":"2024-09-17T03:49:36.410228+0000","last_unstale":"2024-09-17T03:49:37.590715+0000","last_undegraded":"2024-09-17T03:49:37.590715+0000","last_fullsized":"2024-09-17T03:49:37.590715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420810+0000","last_change":"2024-09-17T03:49:36.410333+0000","last_active":"2024-09-17T03:49:37.420810+0000","last_peered":"2024-09-17T03:49:37.420810+0000","last_clean":"2024-09-17T03:49:37.420810+0000","last_became_active":"2024-09-17T03:49:36.405435+0000","last_became_peered":"2024-09-17T03:49:36.405435+0000","last_unstale":"2024-09-17T03:49:37.420810+0000","last_undegraded":"2024-09-17T03:49:37.420810+0000","last_fullsized":"2024-09-17T03:49:37.420810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:21.315923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377561+0000","last_change":"2024-09-17T03:49:36.394974+0000","last_active":"2024-09-17T03:49:37.377561+0000","last_peered":"2024-09-17T03:49:37.377561+0000","last_clean":"2024-09-17T03:49:37.377561+0000","last_became_active":"2024-09-17T03:49:36.393921+0000","last_became_peered":"2024-09-17T03:49:36.393921+0000","last_unstale":"2024-09-17T03:49:37.377561+0000","last_undegraded":"2024-09-17T03:49:37.377561+0000","last_fullsized":"2024-09-17T03:49:37.377561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590561+0000","last_change":"2024-09-17T03:49:36.406505+0000","last_active":"2024-09-17T03:49:37.590561+0000","last_peered":"2024-09-17T03:49:37.590561+0000","last_clean":"2024-09-17T03:49:37.590561+0000","last_became_active":"2024-09-17T03:49:36.406154+0000","last_became_peered":"2024-09-17T03:49:36.406154+0000","last_unstale":"2024-09-17T03:49:37.590561+0000","last_undegraded":"2024-09-17T03:49:37.590561+0000","last_fullsized":"2024-09-17T03:49:37.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588653+0000","last_change":"2024-09-17T03:49:36.400848+0000","last_active":"2024-09-17T03:49:37.588653+0000","last_peered":"2024-09-17T03:49:37.588653+0000","last_clean":"2024-09-17T03:49:37.588653+0000","last_became_active":"2024-09-17T03:49:36.399273+0000","last_became_peered":"2024-09-17T03:49:36.399273+0000","last_unstale":"2024-09-17T03:49:37.588653+0000","last_undegraded":"2024-09-17T03:49:37.588653+0000","last_fullsized":"2024-09-17T03:49:37.588653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379849+0000","last_change":"2024-09-17T03:49:36.409112+0000","last_active":"2024-09-17T03:49:37.379849+0000","last_peered":"2024-09-17T03:49:37.379849+0000","last_clean":"2024-09-17T03:49:37.379849+0000","last_became_active":"2024-09-17T03:49:36.408890+0000","last_became_peered":"2024-09-17T03:49:36.408890+0000","last_unstale":"2024-09-17T03:49:37.379849+0000","last_undegraded":"2024-09-17T03:49:37.379849+0000","last_fullsized":"2024-09-17T03:49:37.379849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588078+0000","last_change":"2024-09-17T03:49:36.408545+0000","last_active":"2024-09-17T03:49:37.588078+0000","last_peered":"2024-09-17T03:49:37.588078+0000","last_clean":"2024-09-17T03:49:37.588078+0000","last_became_active":"2024-09-17T03:49:36.408285+0000","last_became_peered":"2024-09-17T03:49:36.408285+0000","last_unstale":"2024-09-17T03:49:37.588078+0000","last_undegraded":"2024-09-17T03:49:37.588078+0000","last_fullsized":"2024-09-17T03:49:37.588078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420257+0000","last_change":"2024-09-17T03:49:36.401317+0000","last_active":"2024-09-17T03:49:37.420257+0000","last_peered":"2024-09-17T03:49:37.420257+0000","last_clean":"2024-09-17T03:49:37.420257+0000","last_became_active":"2024-09-17T03:49:36.400953+0000","last_became_peered":"2024-09-17T03:49:36.400953+0000","last_unstale":"2024-09-17T03:49:37.420257+0000","last_undegraded":"2024-09-17T03:49:37.420257+0000","last_fullsized":"2024-09-17T03:49:37.420257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:45.195448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378778+0000","last_change":"2024-09-17T03:49:36.400419+0000","last_active":"2024-09-17T03:49:37.378778+0000","last_peered":"2024-09-17T03:49:37.378778+0000","last_clean":"2024-09-17T03:49:37.378778+0000","last_became_active":"2024-09-17T03:49:36.399021+0000","last_became_peered":"2024-09-17T03:49:36.399021+0000","last_unstale":"2024-09-17T03:49:37.378778+0000","last_undegraded":"2024-09-17T03:49:37.378778+0000","last_fullsized":"2024-09-17T03:49:37.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:50.779802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590324+0000","last_change":"2024-09-17T03:49:35.392535+0000","last_active":"2024-09-17T03:49:37.590324+0000","last_peered":"2024-09-17T03:49:37.590324+0000","last_clean":"2024-09-17T03:49:37.590324+0000","last_became_active":"2024-09-17T03:49:35.392219+0000","last_became_peered":"2024-09-17T03:49:35.392219+0000","last_unstale":"2024-09-17T03:49:37.590324+0000","last_undegraded":"2024-09-17T03:49:37.590324+0000","last_fullsized":"2024-09-17T03:49:37.590324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.459490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588861+0000","last_change":"2024-09-17T03:49:36.400903+0000","last_active":"2024-09-17T03:49:37.588861+0000","last_peered":"2024-09-17T03:49:37.588861+0000","last_clean":"2024-09-17T03:49:37.588861+0000","last_became_active":"2024-09-17T03:49:36.400109+0000","last_became_peered":"2024-09-17T03:49:36.400109+0000","last_unstale":"2024-09-17T03:49:37.588861+0000","last_undegraded":"2024-09-17T03:49:37.588861+0000","last_fullsized":"2024-09-17T03:49:37.588861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:34.979447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420155+0000","last_change":"2024-09-17T03:49:35.393512+0000","last_active":"2024-09-17T03:49:37.420155+0000","last_peered":"2024-09-17T03:49:37.420155+0000","last_clean":"2024-09-17T03:49:37.420155+0000","last_became_active":"2024-09-17T03:49:35.388644+0000","last_became_peered":"2024-09-17T03:49:35.388644+0000","last_unstale":"2024-09-17T03:49:37.420155+0000","last_undegraded":"2024-09-17T03:49:37.420155+0000","last_fullsized":"2024-09-17T03:49:37.420155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:02:36.651524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589477+0000","last_change":"2024-09-17T03:49:35.391294+0000","last_active":"2024-09-17T03:49:37.589477+0000","last_peered":"2024-09-17T03:49:37.589477+0000","last_clean":"2024-09-17T03:49:37.589477+0000","last_became_active":"2024-09-17T03:49:35.390970+0000","last_became_peered":"2024-09-17T03:49:35.390970+0000","last_unstale":"2024-09-17T03:49:37.589477+0000","last_undegraded":"2024-09-17T03:49:37.589477+0000","last_fullsized":"2024-09-17T03:49:37.589477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:45.058598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589296+0000","last_change":"2024-09-17T03:49:36.404871+0000","last_active":"2024-09-17T03:49:37.589296+0000","last_peered":"2024-09-17T03:49:37.589296+0000","last_clean":"2024-09-17T03:49:37.589296+0000","last_became_active":"2024-09-17T03:49:36.404452+0000","last_became_peered":"2024-09-17T03:49:36.404452+0000","last_unstale":"2024-09-17T03:49:37.589296+0000","last_undegraded":"2024-09-17T03:49:37.589296+0000","last_fullsized":"2024-09-17T03:49:37.589296+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378971+0000","last_change":"2024-09-17T03:49:35.388702+0000","last_active":"2024-09-17T03:49:37.378971+0000","last_peered":"2024-09-17T03:49:37.378971+0000","last_clean":"2024-09-17T03:49:37.378971+0000","last_became_active":"2024-09-17T03:49:35.388337+0000","last_became_peered":"2024-09-17T03:49:35.388337+0000","last_unstale":"2024-09-17T03:49:37.378971+0000","last_undegraded":"2024-09-17T03:49:37.378971+0000","last_fullsized":"2024-09-17T03:49:37.378971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377377+0000","last_change":"2024-09-17T03:49:36.394908+0000","last_active":"2024-09-17T03:49:37.377377+0000","last_peered":"2024-09-17T03:49:37.377377+0000","last_clean":"2024-09-17T03:49:37.377377+0000","last_became_active":"2024-09-17T03:49:36.394518+0000","last_became_peered":"2024-09-17T03:49:36.394518+0000","last_unstale":"2024-09-17T03:49:37.377377+0000","last_undegraded":"2024-09-17T03:49:37.377377+0000","last_fullsized":"2024-09-17T03:49:37.377377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420535+0000","last_change":"2024-09-17T03:49:35.389956+0000","last_active":"2024-09-17T03:49:37.420535+0000","last_peered":"2024-09-17T03:49:37.420535+0000","last_clean":"2024-09-17T03:49:37.420535+0000","last_became_active":"2024-09-17T03:49:35.389588+0000","last_became_peered":"2024-09-17T03:49:35.389588+0000","last_unstale":"2024-09-17T03:49:37.420535+0000","last_undegraded":"2024-09-17T03:49:37.420535+0000","last_fullsized":"2024-09-17T03:49:37.420535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.510096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380718+0000","last_change":"2024-09-17T03:49:36.405737+0000","last_active":"2024-09-17T03:49:37.380718+0000","last_peered":"2024-09-17T03:49:37.380718+0000","last_clean":"2024-09-17T03:49:37.380718+0000","last_became_active":"2024-09-17T03:49:36.401772+0000","last_became_peered":"2024-09-17T03:49:36.401772+0000","last_unstale":"2024-09-17T03:49:37.380718+0000","last_undegraded":"2024-09-17T03:49:37.380718+0000","last_fullsized":"2024-09-17T03:49:37.380718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380418+0000","last_change":"2024-09-17T03:49:35.395265+0000","last_active":"2024-09-17T03:49:37.380418+0000","last_peered":"2024-09-17T03:49:37.380418+0000","last_clean":"2024-09-17T03:49:37.380418+0000","last_became_active":"2024-09-17T03:49:35.394870+0000","last_became_peered":"2024-09-17T03:49:35.394870+0000","last_unstale":"2024-09-17T03:49:37.380418+0000","last_undegraded":"2024-09-17T03:49:37.380418+0000","last_fullsized":"2024-09-17T03:49:37.380418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590151+0000","last_change":"2024-09-17T03:49:36.407432+0000","last_active":"2024-09-17T03:49:37.590151+0000","last_peered":"2024-09-17T03:49:37.590151+0000","last_clean":"2024-09-17T03:49:37.590151+0000","last_became_active":"2024-09-17T03:49:36.407158+0000","last_became_peered":"2024-09-17T03:49:36.407158+0000","last_unstale":"2024-09-17T03:49:37.590151+0000","last_undegraded":"2024-09-17T03:49:37.590151+0000","last_fullsized":"2024-09-17T03:49:37.590151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377583+0000","last_change":"2024-09-17T03:49:35.390102+0000","last_active":"2024-09-17T03:49:37.377583+0000","last_peered":"2024-09-17T03:49:37.377583+0000","last_clean":"2024-09-17T03:49:37.377583+0000","last_became_active":"2024-09-17T03:49:35.389698+0000","last_became_peered":"2024-09-17T03:49:35.389698+0000","last_unstale":"2024-09-17T03:49:37.377583+0000","last_undegraded":"2024-09-17T03:49:37.377583+0000","last_fullsized":"2024-09-17T03:49:37.377583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379524+0000","last_change":"2024-09-17T03:49:36.406033+0000","last_active":"2024-09-17T03:49:37.379524+0000","last_peered":"2024-09-17T03:49:37.379524+0000","last_clean":"2024-09-17T03:49:37.379524+0000","last_became_active":"2024-09-17T03:49:36.405391+0000","last_became_peered":"2024-09-17T03:49:36.405391+0000","last_unstale":"2024-09-17T03:49:37.379524+0000","last_undegraded":"2024-09-17T03:49:37.379524+0000","last_fullsized":"2024-09-17T03:49:37.379524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:35.463684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420327+0000","last_change":"2024-09-17T03:49:35.392778+0000","last_active":"2024-09-17T03:49:37.420327+0000","last_peered":"2024-09-17T03:49:37.420327+0000","last_clean":"2024-09-17T03:49:37.420327+0000","last_became_active":"2024-09-17T03:49:35.391530+0000","last_became_peered":"2024-09-17T03:49:35.391530+0000","last_unstale":"2024-09-17T03:49:37.420327+0000","last_undegraded":"2024-09-17T03:49:37.420327+0000","last_fullsized":"2024-09-17T03:49:37.420327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:09.206550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588902+0000","last_change":"2024-09-17T03:49:36.404199+0000","last_active":"2024-09-17T03:49:37.588902+0000","last_peered":"2024-09-17T03:49:37.588902+0000","last_clean":"2024-09-17T03:49:37.588902+0000","last_became_active":"2024-09-17T03:49:36.398455+0000","last_became_peered":"2024-09-17T03:49:36.398455+0000","last_unstale":"2024-09-17T03:49:37.588902+0000","last_undegraded":"2024-09-17T03:49:37.588902+0000","last_fullsized":"2024-09-17T03:49:37.588902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588954+0000","last_change":"2024-09-17T03:49:35.387607+0000","last_active":"2024-09-17T03:49:37.588954+0000","last_peered":"2024-09-17T03:49:37.588954+0000","last_clean":"2024-09-17T03:49:37.588954+0000","last_became_active":"2024-09-17T03:49:35.387303+0000","last_became_peered":"2024-09-17T03:49:35.387303+0000","last_unstale":"2024-09-17T03:49:37.588954+0000","last_undegraded":"2024-09-17T03:49:37.588954+0000","last_fullsized":"2024-09-17T03:49:37.588954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:33.456173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588883+0000","last_change":"2024-09-17T03:49:36.413174+0000","last_active":"2024-09-17T03:49:37.588883+0000","last_peered":"2024-09-17T03:49:37.588883+0000","last_clean":"2024-09-17T03:49:37.588883+0000","last_became_active":"2024-09-17T03:49:36.412943+0000","last_became_peered":"2024-09-17T03:49:36.412943+0000","last_unstale":"2024-09-17T03:49:37.588883+0000","last_undegraded":"2024-09-17T03:49:37.588883+0000","last_fullsized":"2024-09-17T03:49:37.588883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379966+0000","last_change":"2024-09-17T03:49:35.388456+0000","last_active":"2024-09-17T03:49:37.379966+0000","last_peered":"2024-09-17T03:49:37.379966+0000","last_clean":"2024-09-17T03:49:37.379966+0000","last_became_active":"2024-09-17T03:49:35.388083+0000","last_became_peered":"2024-09-17T03:49:35.388083+0000","last_unstale":"2024-09-17T03:49:37.379966+0000","last_undegraded":"2024-09-17T03:49:37.379966+0000","last_fullsized":"2024-09-17T03:49:37.379966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588927+0000","last_change":"2024-09-17T03:49:36.414542+0000","last_active":"2024-09-17T03:49:37.588927+0000","last_peered":"2024-09-17T03:49:37.588927+0000","last_clean":"2024-09-17T03:49:37.588927+0000","last_became_active":"2024-09-17T03:49:36.414224+0000","last_became_peered":"2024-09-17T03:49:36.414224+0000","last_unstale":"2024-09-17T03:49:37.588927+0000","last_undegraded":"2024-09-17T03:49:37.588927+0000","last_fullsized":"2024-09-17T03:49:37.588927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420647+0000","last_change":"2024-09-17T03:49:35.388635+0000","last_active":"2024-09-17T03:49:37.420647+0000","last_peered":"2024-09-17T03:49:37.420647+0000","last_clean":"2024-09-17T03:49:37.420647+0000","last_became_active":"2024-09-17T03:49:35.388278+0000","last_became_peered":"2024-09-17T03:49:35.388278+0000","last_unstale":"2024-09-17T03:49:37.420647+0000","last_undegraded":"2024-09-17T03:49:37.420647+0000","last_fullsized":"2024-09-17T03:49:37.420647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:17.648757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420254+0000","last_change":"2024-09-17T03:49:36.401308+0000","last_active":"2024-09-17T03:49:37.420254+0000","last_peered":"2024-09-17T03:49:37.420254+0000","last_clean":"2024-09-17T03:49:37.420254+0000","last_became_active":"2024-09-17T03:49:36.400942+0000","last_became_peered":"2024-09-17T03:49:36.400942+0000","last_unstale":"2024-09-17T03:49:37.420254+0000","last_undegraded":"2024-09-17T03:49:37.420254+0000","last_fullsized":"2024-09-17T03:49:37.420254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588398+0000","last_change":"2024-09-17T03:49:35.372728+0000","last_active":"2024-09-17T03:49:37.588398+0000","last_peered":"2024-09-17T03:49:37.588398+0000","last_clean":"2024-09-17T03:49:37.588398+0000","last_became_active":"2024-09-17T03:49:35.372413+0000","last_became_peered":"2024-09-17T03:49:35.372413+0000","last_unstale":"2024-09-17T03:49:37.588398+0000","last_undegraded":"2024-09-17T03:49:37.588398+0000","last_fullsized":"2024-09-17T03:49:37.588398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588789+0000","last_change":"2024-09-17T03:49:36.413936+0000","last_active":"2024-09-17T03:49:37.588789+0000","last_peered":"2024-09-17T03:49:37.588789+0000","last_clean":"2024-09-17T03:49:37.588789+0000","last_became_active":"2024-09-17T03:49:36.402860+0000","last_became_peered":"2024-09-17T03:49:36.402860+0000","last_unstale":"2024-09-17T03:49:37.588789+0000","last_undegraded":"2024-09-17T03:49:37.588789+0000","last_fullsized":"2024-09-17T03:49:37.588789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380504+0000","last_change":"2024-09-17T03:49:35.385498+0000","last_active":"2024-09-17T03:49:37.380504+0000","last_peered":"2024-09-17T03:49:37.380504+0000","last_clean":"2024-09-17T03:49:37.380504+0000","last_became_active":"2024-09-17T03:49:35.385212+0000","last_became_peered":"2024-09-17T03:49:35.385212+0000","last_unstale":"2024-09-17T03:49:37.380504+0000","last_undegraded":"2024-09-17T03:49:37.380504+0000","last_fullsized":"2024-09-17T03:49:37.380504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:45.909870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589167+0000","last_change":"2024-09-17T03:49:36.413538+0000","last_active":"2024-09-17T03:49:37.589167+0000","last_peered":"2024-09-17T03:49:37.589167+0000","last_clean":"2024-09-17T03:49:37.589167+0000","last_became_active":"2024-09-17T03:49:36.413197+0000","last_became_peered":"2024-09-17T03:49:36.413197+0000","last_unstale":"2024-09-17T03:49:37.589167+0000","last_undegraded":"2024-09-17T03:49:37.589167+0000","last_fullsized":"2024-09-17T03:49:37.589167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588219+0000","last_change":"2024-09-17T03:49:35.384632+0000","last_active":"2024-09-17T03:49:37.588219+0000","last_peered":"2024-09-17T03:49:37.588219+0000","last_clean":"2024-09-17T03:49:37.588219+0000","last_became_active":"2024-09-17T03:49:35.384263+0000","last_became_peered":"2024-09-17T03:49:35.384263+0000","last_unstale":"2024-09-17T03:49:37.588219+0000","last_undegraded":"2024-09-17T03:49:37.588219+0000","last_fullsized":"2024-09-17T03:49:37.588219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589537+0000","last_change":"2024-09-17T03:49:36.415593+0000","last_active":"2024-09-17T03:49:37.589537+0000","last_peered":"2024-09-17T03:49:37.589537+0000","last_clean":"2024-09-17T03:49:37.589537+0000","last_became_active":"2024-09-17T03:49:36.411554+0000","last_became_peered":"2024-09-17T03:49:36.411554+0000","last_unstale":"2024-09-17T03:49:37.589537+0000","last_undegraded":"2024-09-17T03:49:37.589537+0000","last_fullsized":"2024-09-17T03:49:37.589537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:13.481011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377691+0000","last_change":"2024-09-17T03:49:35.387003+0000","last_active":"2024-09-17T03:49:37.377691+0000","last_peered":"2024-09-17T03:49:37.377691+0000","last_clean":"2024-09-17T03:49:37.377691+0000","last_became_active":"2024-09-17T03:49:35.386654+0000","last_became_peered":"2024-09-17T03:49:35.386654+0000","last_unstale":"2024-09-17T03:49:37.377691+0000","last_undegraded":"2024-09-17T03:49:37.377691+0000","last_fullsized":"2024-09-17T03:49:37.377691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:23.004272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379918+0000","last_change":"2024-09-17T03:49:36.395286+0000","last_active":"2024-09-17T03:49:37.379918+0000","last_peered":"2024-09-17T03:49:37.379918+0000","last_clean":"2024-09-17T03:49:37.379918+0000","last_became_active":"2024-09-17T03:49:36.394025+0000","last_became_peered":"2024-09-17T03:49:36.394025+0000","last_unstale":"2024-09-17T03:49:37.379918+0000","last_undegraded":"2024-09-17T03:49:37.379918+0000","last_fullsized":"2024-09-17T03:49:37.379918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588329+0000","last_change":"2024-09-17T03:49:35.389511+0000","last_active":"2024-09-17T03:49:37.588329+0000","last_peered":"2024-09-17T03:49:37.588329+0000","last_clean":"2024-09-17T03:49:37.588329+0000","last_became_active":"2024-09-17T03:49:35.389061+0000","last_became_peered":"2024-09-17T03:49:35.389061+0000","last_unstale":"2024-09-17T03:49:37.588329+0000","last_undegraded":"2024-09-17T03:49:37.588329+0000","last_fullsized":"2024-09-17T03:49:37.588329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589110+0000","last_change":"2024-09-17T03:49:36.418896+0000","last_active":"2024-09-17T03:49:37.589110+0000","last_peered":"2024-09-17T03:49:37.589110+0000","last_clean":"2024-09-17T03:49:37.589110+0000","last_became_active":"2024-09-17T03:49:36.418567+0000","last_became_peered":"2024-09-17T03:49:36.418567+0000","last_unstale":"2024-09-17T03:49:37.589110+0000","last_undegraded":"2024-09-17T03:49:37.589110+0000","last_fullsized":"2024-09-17T03:49:37.589110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588519+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:49:37.588519+0000","last_peered":"2024-09-17T03:49:37.588519+0000","last_clean":"2024-09-17T03:49:37.588519+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:49:37.588519+0000","last_undegraded":"2024-09-17T03:49:37.588519+0000","last_fullsized":"2024-09-17T03:49:37.588519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378615+0000","last_change":"2024-09-17T03:49:35.379012+0000","last_active":"2024-09-17T03:49:37.378615+0000","last_peered":"2024-09-17T03:49:37.378615+0000","last_clean":"2024-09-17T03:49:37.378615+0000","last_became_active":"2024-09-17T03:49:35.378349+0000","last_became_peered":"2024-09-17T03:49:35.378349+0000","last_unstale":"2024-09-17T03:49:37.378615+0000","last_undegraded":"2024-09-17T03:49:37.378615+0000","last_fullsized":"2024-09-17T03:49:37.378615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589522+0000","last_change":"2024-09-17T03:49:36.408105+0000","last_active":"2024-09-17T03:49:37.589522+0000","last_peered":"2024-09-17T03:49:37.589522+0000","last_clean":"2024-09-17T03:49:37.589522+0000","last_became_active":"2024-09-17T03:49:36.407814+0000","last_became_peered":"2024-09-17T03:49:36.407814+0000","last_unstale":"2024-09-17T03:49:37.589522+0000","last_undegraded":"2024-09-17T03:49:37.589522+0000","last_fullsized":"2024-09-17T03:49:37.589522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588438+0000","last_change":"2024-09-17T03:49:35.399235+0000","last_active":"2024-09-17T03:49:37.588438+0000","last_peered":"2024-09-17T03:49:37.588438+0000","last_clean":"2024-09-17T03:49:37.588438+0000","last_became_active":"2024-09-17T03:49:35.398860+0000","last_became_peered":"2024-09-17T03:49:35.398860+0000","last_unstale":"2024-09-17T03:49:37.588438+0000","last_undegraded":"2024-09-17T03:49:37.588438+0000","last_fullsized":"2024-09-17T03:49:37.588438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:34.455037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588130+0000","last_change":"2024-09-17T03:49:36.403778+0000","last_active":"2024-09-17T03:49:37.588130+0000","last_peered":"2024-09-17T03:49:37.588130+0000","last_clean":"2024-09-17T03:49:37.588130+0000","last_became_active":"2024-09-17T03:49:36.403408+0000","last_became_peered":"2024-09-17T03:49:36.403408+0000","last_unstale":"2024-09-17T03:49:37.588130+0000","last_undegraded":"2024-09-17T03:49:37.588130+0000","last_fullsized":"2024-09-17T03:49:37.588130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:19.846081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.587984+0000","last_change":"2024-09-17T03:49:35.384616+0000","last_active":"2024-09-17T03:49:37.587984+0000","last_peered":"2024-09-17T03:49:37.587984+0000","last_clean":"2024-09-17T03:49:37.587984+0000","last_became_active":"2024-09-17T03:49:35.384264+0000","last_became_peered":"2024-09-17T03:49:35.384264+0000","last_unstale":"2024-09-17T03:49:37.587984+0000","last_undegraded":"2024-09-17T03:49:37.587984+0000","last_fullsized":"2024-09-17T03:49:37.587984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377132+0000","last_change":"2024-09-17T03:49:36.381928+0000","last_active":"2024-09-17T03:49:37.377132+0000","last_peered":"2024-09-17T03:49:37.377132+0000","last_clean":"2024-09-17T03:49:37.377132+0000","last_became_active":"2024-09-17T03:49:36.381629+0000","last_became_peered":"2024-09-17T03:49:36.381629+0000","last_unstale":"2024-09-17T03:49:37.377132+0000","last_undegraded":"2024-09-17T03:49:37.377132+0000","last_fullsized":"2024-09-17T03:49:37.377132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379997+0000","last_change":"2024-09-17T03:49:35.396687+0000","last_active":"2024-09-17T03:49:37.379997+0000","last_peered":"2024-09-17T03:49:37.379997+0000","last_clean":"2024-09-17T03:49:37.379997+0000","last_became_active":"2024-09-17T03:49:35.390958+0000","last_became_peered":"2024-09-17T03:49:35.390958+0000","last_unstale":"2024-09-17T03:49:37.379997+0000","last_undegraded":"2024-09-17T03:49:37.379997+0000","last_fullsized":"2024-09-17T03:49:37.379997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378943+0000","last_change":"2024-09-17T03:49:36.406614+0000","last_active":"2024-09-17T03:49:37.378943+0000","last_peered":"2024-09-17T03:49:37.378943+0000","last_clean":"2024-09-17T03:49:37.378943+0000","last_became_active":"2024-09-17T03:49:36.403577+0000","last_became_peered":"2024-09-17T03:49:36.403577+0000","last_unstale":"2024-09-17T03:49:37.378943+0000","last_undegraded":"2024-09-17T03:49:37.378943+0000","last_fullsized":"2024-09-17T03:49:37.378943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:45.693650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377460+0000","last_change":"2024-09-17T03:49:35.385825+0000","last_active":"2024-09-17T03:49:37.377460+0000","last_peered":"2024-09-17T03:49:37.377460+0000","last_clean":"2024-09-17T03:49:37.377460+0000","last_became_active":"2024-09-17T03:49:35.385489+0000","last_became_peered":"2024-09-17T03:49:35.385489+0000","last_unstale":"2024-09-17T03:49:37.377460+0000","last_undegraded":"2024-09-17T03:49:37.377460+0000","last_fullsized":"2024-09-17T03:49:37.377460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380572+0000","last_change":"2024-09-17T03:49:36.413827+0000","last_active":"2024-09-17T03:49:37.380572+0000","last_peered":"2024-09-17T03:49:37.380572+0000","last_clean":"2024-09-17T03:49:37.380572+0000","last_became_active":"2024-09-17T03:49:36.413432+0000","last_became_peered":"2024-09-17T03:49:36.413432+0000","last_unstale":"2024-09-17T03:49:37.380572+0000","last_undegraded":"2024-09-17T03:49:37.380572+0000","last_fullsized":"2024-09-17T03:49:37.380572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589281+0000","last_change":"2024-09-17T03:49:35.396951+0000","last_active":"2024-09-17T03:49:37.589281+0000","last_peered":"2024-09-17T03:49:37.589281+0000","last_clean":"2024-09-17T03:49:37.589281+0000","last_became_active":"2024-09-17T03:49:35.396554+0000","last_became_peered":"2024-09-17T03:49:35.396554+0000","last_unstale":"2024-09-17T03:49:37.589281+0000","last_undegraded":"2024-09-17T03:49:37.589281+0000","last_fullsized":"2024-09-17T03:49:37.589281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:23.665085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377339+0000","last_change":"2024-09-17T03:49:36.387372+0000","last_active":"2024-09-17T03:49:37.377339+0000","last_peered":"2024-09-17T03:49:37.377339+0000","last_clean":"2024-09-17T03:49:37.377339+0000","last_became_active":"2024-09-17T03:49:36.384715+0000","last_became_peered":"2024-09-17T03:49:36.384715+0000","last_unstale":"2024-09-17T03:49:37.377339+0000","last_undegraded":"2024-09-17T03:49:37.377339+0000","last_fullsized":"2024-09-17T03:49:37.377339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.419437+0000","last_change":"2024-09-17T03:49:35.393853+0000","last_active":"2024-09-17T03:49:37.419437+0000","last_peered":"2024-09-17T03:49:37.419437+0000","last_clean":"2024-09-17T03:49:37.419437+0000","last_became_active":"2024-09-17T03:49:35.393551+0000","last_became_peered":"2024-09-17T03:49:35.393551+0000","last_unstale":"2024-09-17T03:49:37.419437+0000","last_undegraded":"2024-09-17T03:49:37.419437+0000","last_fullsized":"2024-09-17T03:49:37.419437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590925+0000","last_change":"2024-09-17T03:49:36.412628+0000","last_active":"2024-09-17T03:49:37.590925+0000","last_peered":"2024-09-17T03:49:37.590925+0000","last_clean":"2024-09-17T03:49:37.590925+0000","last_became_active":"2024-09-17T03:49:36.412320+0000","last_became_peered":"2024-09-17T03:49:36.412320+0000","last_unstale":"2024-09-17T03:49:37.590925+0000","last_undegraded":"2024-09-17T03:49:37.590925+0000","last_fullsized":"2024-09-17T03:49:37.590925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420603+0000","last_change":"2024-09-17T03:49:35.389093+0000","last_active":"2024-09-17T03:49:37.420603+0000","last_peered":"2024-09-17T03:49:37.420603+0000","last_clean":"2024-09-17T03:49:37.420603+0000","last_became_active":"2024-09-17T03:49:35.388625+0000","last_became_peered":"2024-09-17T03:49:35.388625+0000","last_unstale":"2024-09-17T03:49:37.420603+0000","last_undegraded":"2024-09-17T03:49:37.420603+0000","last_fullsized":"2024-09-17T03:49:37.420603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.987962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589830+0000","last_change":"2024-09-17T03:49:36.397328+0000","last_active":"2024-09-17T03:49:37.589830+0000","last_peered":"2024-09-17T03:49:37.589830+0000","last_clean":"2024-09-17T03:49:37.589830+0000","last_became_active":"2024-09-17T03:49:36.396910+0000","last_became_peered":"2024-09-17T03:49:36.396910+0000","last_unstale":"2024-09-17T03:49:37.589830+0000","last_undegraded":"2024-09-17T03:49:37.589830+0000","last_fullsized":"2024-09-17T03:49:37.589830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588441+0000","last_change":"2024-09-17T03:49:35.391714+0000","last_active":"2024-09-17T03:49:37.588441+0000","last_peered":"2024-09-17T03:49:37.588441+0000","last_clean":"2024-09-17T03:49:37.588441+0000","last_became_active":"2024-09-17T03:49:35.391374+0000","last_became_peered":"2024-09-17T03:49:35.391374+0000","last_unstale":"2024-09-17T03:49:37.588441+0000","last_undegraded":"2024-09-17T03:49:37.588441+0000","last_fullsized":"2024-09-17T03:49:37.588441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380255+0000","last_change":"2024-09-17T03:49:36.395381+0000","last_active":"2024-09-17T03:49:37.380255+0000","last_peered":"2024-09-17T03:49:37.380255+0000","last_clean":"2024-09-17T03:49:37.380255+0000","last_became_active":"2024-09-17T03:49:36.394430+0000","last_became_peered":"2024-09-17T03:49:36.394430+0000","last_unstale":"2024-09-17T03:49:37.380255+0000","last_undegraded":"2024-09-17T03:49:37.380255+0000","last_fullsized":"2024-09-17T03:49:37.380255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377304+0000","last_change":"2024-09-17T03:49:35.386828+0000","last_active":"2024-09-17T03:49:37.377304+0000","last_peered":"2024-09-17T03:49:37.377304+0000","last_clean":"2024-09-17T03:49:37.377304+0000","last_became_active":"2024-09-17T03:49:35.385425+0000","last_became_peered":"2024-09-17T03:49:35.385425+0000","last_unstale":"2024-09-17T03:49:37.377304+0000","last_undegraded":"2024-09-17T03:49:37.377304+0000","last_fullsized":"2024-09-17T03:49:37.377304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589280+0000","last_change":"2024-09-17T03:49:36.396082+0000","last_active":"2024-09-17T03:49:37.589280+0000","last_peered":"2024-09-17T03:49:37.589280+0000","last_clean":"2024-09-17T03:49:37.589280+0000","last_became_active":"2024-09-17T03:49:36.395733+0000","last_became_peered":"2024-09-17T03:49:36.395733+0000","last_unstale":"2024-09-17T03:49:37.589280+0000","last_undegraded":"2024-09-17T03:49:37.589280+0000","last_fullsized":"2024-09-17T03:49:37.589280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377167+0000","last_change":"2024-09-17T03:49:35.383855+0000","last_active":"2024-09-17T03:49:37.377167+0000","last_peered":"2024-09-17T03:49:37.377167+0000","last_clean":"2024-09-17T03:49:37.377167+0000","last_became_active":"2024-09-17T03:49:35.383513+0000","last_became_peered":"2024-09-17T03:49:35.383513+0000","last_unstale":"2024-09-17T03:49:37.377167+0000","last_undegraded":"2024-09-17T03:49:37.377167+0000","last_fullsized":"2024-09-17T03:49:37.377167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589610+0000","last_change":"2024-09-17T03:49:36.401383+0000","last_active":"2024-09-17T03:49:37.589610+0000","last_peered":"2024-09-17T03:49:37.589610+0000","last_clean":"2024-09-17T03:49:37.589610+0000","last_became_active":"2024-09-17T03:49:36.401112+0000","last_became_peered":"2024-09-17T03:49:36.401112+0000","last_unstale":"2024-09-17T03:49:37.589610+0000","last_undegraded":"2024-09-17T03:49:37.589610+0000","last_fullsized":"2024-09-17T03:49:37.589610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589087+0000","last_change":"2024-09-17T03:49:35.405334+0000","last_active":"2024-09-17T03:49:37.589087+0000","last_peered":"2024-09-17T03:49:37.589087+0000","last_clean":"2024-09-17T03:49:37.589087+0000","last_became_active":"2024-09-17T03:49:35.404955+0000","last_became_peered":"2024-09-17T03:49:35.404955+0000","last_unstale":"2024-09-17T03:49:37.589087+0000","last_undegraded":"2024-09-17T03:49:37.589087+0000","last_fullsized":"2024-09-17T03:49:37.589087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420485+0000","last_change":"2024-09-17T03:49:36.412896+0000","last_active":"2024-09-17T03:49:37.420485+0000","last_peered":"2024-09-17T03:49:37.420485+0000","last_clean":"2024-09-17T03:49:37.420485+0000","last_became_active":"2024-09-17T03:49:36.406254+0000","last_became_peered":"2024-09-17T03:49:36.406254+0000","last_unstale":"2024-09-17T03:49:37.420485+0000","last_undegraded":"2024-09-17T03:49:37.420485+0000","last_fullsized":"2024-09-17T03:49:37.420485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380554+0000","last_change":"2024-09-17T03:49:36.396792+0000","last_active":"2024-09-17T03:49:37.380554+0000","last_peered":"2024-09-17T03:49:37.380554+0000","last_clean":"2024-09-17T03:49:37.380554+0000","last_became_active":"2024-09-17T03:49:36.393651+0000","last_became_peered":"2024-09-17T03:49:36.393651+0000","last_unstale":"2024-09-17T03:49:37.380554+0000","last_undegraded":"2024-09-17T03:49:37.380554+0000","last_fullsized":"2024-09-17T03:49:37.380554+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588722+0000","last_change":"2024-09-17T03:49:35.398891+0000","last_active":"2024-09-17T03:49:37.588722+0000","last_peered":"2024-09-17T03:49:37.588722+0000","last_clean":"2024-09-17T03:49:37.588722+0000","last_became_active":"2024-09-17T03:49:35.398494+0000","last_became_peered":"2024-09-17T03:49:35.398494+0000","last_unstale":"2024-09-17T03:49:37.588722+0000","last_undegraded":"2024-09-17T03:49:37.588722+0000","last_fullsized":"2024-09-17T03:49:37.588722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:34.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379744+0000","last_change":"2024-09-17T03:49:36.394357+0000","last_active":"2024-09-17T03:49:37.379744+0000","last_peered":"2024-09-17T03:49:37.379744+0000","last_clean":"2024-09-17T03:49:37.379744+0000","last_became_active":"2024-09-17T03:49:36.393549+0000","last_became_peered":"2024-09-17T03:49:36.393549+0000","last_unstale":"2024-09-17T03:49:37.379744+0000","last_undegraded":"2024-09-17T03:49:37.379744+0000","last_fullsized":"2024-09-17T03:49:37.379744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589049+0000","last_change":"2024-09-17T03:49:35.398839+0000","last_active":"2024-09-17T03:49:37.589049+0000","last_peered":"2024-09-17T03:49:37.589049+0000","last_clean":"2024-09-17T03:49:37.589049+0000","last_became_active":"2024-09-17T03:49:35.398355+0000","last_became_peered":"2024-09-17T03:49:35.398355+0000","last_unstale":"2024-09-17T03:49:37.589049+0000","last_undegraded":"2024-09-17T03:49:37.589049+0000","last_fullsized":"2024-09-17T03:49:37.589049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:51.292462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589985+0000","last_change":"2024-09-17T03:49:36.409308+0000","last_active":"2024-09-17T03:49:37.589985+0000","last_peered":"2024-09-17T03:49:37.589985+0000","last_clean":"2024-09-17T03:49:37.589985+0000","last_became_active":"2024-09-17T03:49:36.408373+0000","last_became_peered":"2024-09-17T03:49:36.408373+0000","last_unstale":"2024-09-17T03:49:37.589985+0000","last_undegraded":"2024-09-17T03:49:37.589985+0000","last_fullsized":"2024-09-17T03:49:37.589985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590572+0000","last_change":"2024-09-17T03:49:35.381739+0000","last_active":"2024-09-17T03:49:37.590572+0000","last_peered":"2024-09-17T03:49:37.590572+0000","last_clean":"2024-09-17T03:49:37.590572+0000","last_became_active":"2024-09-17T03:49:35.381185+0000","last_became_peered":"2024-09-17T03:49:35.381185+0000","last_unstale":"2024-09-17T03:49:37.590572+0000","last_undegraded":"2024-09-17T03:49:37.590572+0000","last_fullsized":"2024-09-17T03:49:37.590572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590630+0000","last_change":"2024-09-17T03:49:36.401989+0000","last_active":"2024-09-17T03:49:37.590630+0000","last_peered":"2024-09-17T03:49:37.590630+0000","last_clean":"2024-09-17T03:49:37.590630+0000","last_became_active":"2024-09-17T03:49:36.399727+0000","last_became_peered":"2024-09-17T03:49:36.399727+0000","last_unstale":"2024-09-17T03:49:37.590630+0000","last_undegraded":"2024-09-17T03:49:37.590630+0000","last_fullsized":"2024-09-17T03:49:37.590630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590102+0000","last_change":"2024-09-17T03:49:35.386738+0000","last_active":"2024-09-17T03:49:37.590102+0000","last_peered":"2024-09-17T03:49:37.590102+0000","last_clean":"2024-09-17T03:49:37.590102+0000","last_became_active":"2024-09-17T03:49:35.386382+0000","last_became_peered":"2024-09-17T03:49:35.386382+0000","last_unstale":"2024-09-17T03:49:37.590102+0000","last_undegraded":"2024-09-17T03:49:37.590102+0000","last_fullsized":"2024-09-17T03:49:37.590102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:57.569543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":667808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":76984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509444,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509446,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":76858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":77018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542148,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":815104,"data_stored":661564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:49:41.281 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:49:41.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-17T03:49:41.282 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T03:49:41.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 -- 172.21.15.129:0/1658498977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0cff70 msgr2=0x7f01bc15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 --2- 172.21.15.129:0/1658498977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0cff70 0x7f01bc15c830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f01ac005fd0 tx=0x7f01ac02a540 comp rx=0 tx=0).stop 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 -- 172.21.15.129:0/1658498977 shutdown_connections 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 --2- 172.21.15.129:0/1658498977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc160c90 0x7f01bc0b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 --2- 172.21.15.129:0/1658498977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0cff70 0x7f01bc15c830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 -- 172.21.15.129:0/1658498977 >> 172.21.15.129:0/1658498977 conn(0x7f01bc0cee20 msgr2=0x7f01bc0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 -- 172.21.15.129:0/1658498977 shutdown_connections 2024-09-17T03:49:41.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.542+0000 7f01c3c00640 1 -- 172.21.15.129:0/1658498977 wait complete. 2024-09-17T03:49:41.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.544+0000 7f01c3c00640 1 Processor -- start 2024-09-17T03:49:41.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.545+0000 7f01c3c00640 1 -- start start 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 0x7f01bc0ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 0x7f01bc0b73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc073070 con 0x7f01bc0b4360 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f01bc072ef0 con 0x7f01bc160c90 2024-09-17T03:49:41.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c3c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f01bc0731f0 con 0x7f01bc0cff70 2024-09-17T03:49:41.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39522/0 (socket says 172.21.15.129:39522) 2024-09-17T03:49:41.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.546+0000 7f01c1200640 1 -- 172.21.15.129:0/3311716509 learned_addr learned my addr 172.21.15.129:0/3311716509 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 -- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 msgr2=0x7f01bc0ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01b3e00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 0x7f01bc0ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 0x7f01bc0b73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 0x7f01bc0ca550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 -- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 msgr2=0x7f01bc0b73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 0x7f01bc0b73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 -- 172.21.15.129:0/3311716509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01bc1f6bb0 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01b3e00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 0x7f01bc0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 0x7f01bc0b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c1200640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f01ac006010 tx=0x7f01ac0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac03e030 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01ac003d90 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01bc1f6e50 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac035040 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.547+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01bc0cc700 con 0x7f01bc0b4360 2024-09-17T03:49:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.548+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01bc0b3e20 con 0x7f01bc0b4360 2024-09-17T03:49:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.550+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f01ac041030 con 0x7f01bc0b4360 2024-09-17T03:49:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.550+0000 7f01b9600640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 0x7f019807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.551+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f01ac0bb5e0 con 0x7f01bc0b4360 2024-09-17T03:49:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.551+0000 7f01b3e00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 0x7f019807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.552+0000 7f01b3e00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 0x7f019807b0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f01a4005fd0 tx=0x7f01a4005d40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.555+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f01ac083980 con 0x7f01bc0b4360 2024-09-17T03:49:41.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.740+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f01bc152c40 con 0x7f01bc0b4360 2024-09-17T03:49:41.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.741+0000 7f01b9600640 1 -- 172.21.15.129:0/3311716509 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v31) ==== 86+0+160 (secure 0 0 0) 0x7f01ac083280 con 0x7f01bc0b4360 2024-09-17T03:49:41.740 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 msgr2=0x7f019807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 0x7f019807b0d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f01a4005fd0 tx=0x7f01a4005d40 comp rx=0 tx=0).stop 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 msgr2=0x7f01bc0c75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f01ac006010 tx=0x7f01ac0038a0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 shutdown_connections 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0198078bc0 0x7f019807b0d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01bc160c90 0x7f01bc0b73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01bc0cff70 0x7f01bc0ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 --2- 172.21.15.129:0/3311716509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01bc0b4360 0x7f01bc0c75e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 >> 172.21.15.129:0/3311716509 conn(0x7f01bc0cee20 msgr2=0x7f01bc0d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 shutdown_connections 2024-09-17T03:49:41.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.743+0000 7f01c3c00640 1 -- 172.21.15.129:0/3311716509 wait complete. 2024-09-17T03:49:41.765 INFO:teuthology.orchestra.run.smithi129.stdout:[{"version":31,"timestamp":"2024-09-17T03:49:23.245982+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"3951.474476"}]}] 2024-09-17T03:49:41.765 INFO:tasks.ceph_manager:config epoch is 31 2024-09-17T03:49:41.766 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T03:49:41.766 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-17T03:49:41.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 -- 172.21.15.129:0/4256593707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6780ccb50 msgr2=0x7fb6780ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 --2- 172.21.15.129:0/4256593707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6780ccb50 0x7fb6780ccf60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb670005fd0 tx=0x7fb67002aa90 comp rx=0 tx=0).stop 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 -- 172.21.15.129:0/4256593707 shutdown_connections 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 --2- 172.21.15.129:0/4256593707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6780ccb50 0x7fb6780ccf60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 --2- 172.21.15.129:0/4256593707 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780cc1e0 0x7fb6780af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.985+0000 7fb67dc00640 1 -- 172.21.15.129:0/4256593707 >> 172.21.15.129:0/4256593707 conn(0x7fb67806d7d0 msgr2=0x7fb6780b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.986+0000 7fb67dc00640 1 -- 172.21.15.129:0/4256593707 shutdown_connections 2024-09-17T03:49:41.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.986+0000 7fb67dc00640 1 -- 172.21.15.129:0/4256593707 wait complete. 2024-09-17T03:49:41.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.988+0000 7fb67dc00640 1 Processor -- start 2024-09-17T03:49:41.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.988+0000 7fb67dc00640 1 -- start start 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6780cc1e0 0x7fb67814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 0x7fb6781f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6781533a0 con 0x7fb67815d8a0 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb67815dcf0 con 0x7fb6780cc1e0 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.989+0000 7fb67dc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb678153520 con 0x7fb6780ccb50 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39548/0 (socket says 172.21.15.129:39548) 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 -- 172.21.15.129:0/877600497 learned_addr learned my addr 172.21.15.129:0/877600497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:41.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb676a00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6780cc1e0 0x7fb67814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb676000640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 0x7fb6781f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 -- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 msgr2=0x7fb6781f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 0x7fb6781f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 -- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6780cc1e0 msgr2=0x7fb67814e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6780cc1e0 0x7fb67814e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 -- 172.21.15.129:0/877600497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67814f440 con 0x7fb67815d8a0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb676000640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 0x7fb6781f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.990+0000 7fb677400640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb670009bf0 tx=0x7fb670003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.991+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670044070 con 0x7fb67815d8a0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.991+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb670004040 con 0x7fb67815d8a0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.991+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6781f6620 con 0x7fb67815d8a0 2024-09-17T03:49:41.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.991+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670004450 con 0x7fb67815d8a0 2024-09-17T03:49:41.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.991+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6781f93b0 con 0x7fb67815d8a0 2024-09-17T03:49:41.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.992+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6781f6390 con 0x7fb67815d8a0 2024-09-17T03:49:41.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.993+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb670047070 con 0x7fb67815d8a0 2024-09-17T03:49:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.994+0000 7fb66ea00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 0x7fb65807b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.994+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb6700bbb40 con 0x7fb67815d8a0 2024-09-17T03:49:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.994+0000 7fb676a00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 0x7fb65807b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.995+0000 7fb676a00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 0x7fb65807b120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb668009ac0 tx=0x7fb668005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:41.998+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb67814f440 con 0x7fb67815d8a0 2024-09-17T03:49:42.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.230+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb6781f6900 con 0x7fb67815d8a0 2024-09-17T03:49:42.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.233+0000 7fb66ea00640 1 -- 172.21.15.129:0/877600497 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+194720 (secure 0 0 0) 0x7fb6781f6900 con 0x7fb67815d8a0 2024-09-17T03:49:42.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 msgr2=0x7fb65807b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 0x7fb65807b120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb668009ac0 tx=0x7fb668005f90 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 msgr2=0x7fb67806ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb670009bf0 tx=0x7fb670003ba0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 shutdown_connections 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb658078c10 0x7fb65807b120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb67815d8a0 0x7fb67806ee90 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6780ccb50 0x7fb6781f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 --2- 172.21.15.129:0/877600497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6780cc1e0 0x7fb67814e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 >> 172.21.15.129:0/877600497 conn(0x7fb67806d7d0 msgr2=0x7fb6780b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:42.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 shutdown_connections 2024-09-17T03:49:42.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.237+0000 7fb67dc00640 1 -- 172.21.15.129:0/877600497 wait complete. 2024-09-17T03:49:42.259 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":5,"flags":0,"active_gid":4157,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3833218970},{"type":"v1","addr":"172.21.15.181:6801","nonce":3833218970}]},"active_addr":"172.21.15.181:6801/3833218970","active_change":"2024-09-17T03:49:14.280012+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4163,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2872372992}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":397829052}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":759453639}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3830936153}]}]} 2024-09-17T03:49:42.263 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-17T03:49:42.263 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-17T03:49:42.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:42.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 -- 172.21.15.129:0/3346804774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0b54d0 msgr2=0x7f525c0b58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 --2- 172.21.15.129:0/3346804774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0b54d0 0x7f525c0b58e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f525c071bd0 tx=0x7f525402aac0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 -- 172.21.15.129:0/3346804774 shutdown_connections 2024-09-17T03:49:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 --2- 172.21.15.129:0/3346804774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0b54d0 0x7f525c0b58e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 --2- 172.21.15.129:0/3346804774 >> v2:172.21.15.181:3300/0 conn(0x7f525c0af700 0x7f525c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 -- 172.21.15.129:0/3346804774 >> 172.21.15.129:0/3346804774 conn(0x7f525c06d870 msgr2=0x7f525c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 -- 172.21.15.129:0/3346804774 shutdown_connections 2024-09-17T03:49:42.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.487+0000 7f525be00640 1 -- 172.21.15.129:0/3346804774 wait complete. 2024-09-17T03:49:42.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.489+0000 7f525be00640 1 Processor -- start 2024-09-17T03:49:42.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.489+0000 7f525be00640 1 -- start start 2024-09-17T03:49:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f525c0af700 0x7f525c14dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525c0b54d0 0x7f525c0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f525c15d2b0 con 0x7f525c0735d0 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f525c15d130 con 0x7f525c0b54d0 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f525c15d430 con 0x7f525c0af700 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39562/0 (socket says 172.21.15.129:39562) 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525aa00640 1 -- 172.21.15.129:0/2776499036 learned_addr learned my addr 172.21.15.129:0/2776499036 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f5252000640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f525c0af700 0x7f525c14dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525b400640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525c0b54d0 0x7f525c0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525aa00640 1 -- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f525c0af700 msgr2=0x7f525c14dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.491+0000 7f525aa00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f525c0af700 0x7f525c14dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525aa00640 1 -- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525c0b54d0 msgr2=0x7f525c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525aa00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525c0b54d0 0x7f525c0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525aa00640 1 -- 172.21.15.129:0/2776499036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c0cdcc0 con 0x7f525c0735d0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525aa00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f524c00d740 tx=0x7f524c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c017070 con 0x7f525c0735d0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f524c009d20 con 0x7f525c0735d0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c0cdfa0 con 0x7f525c0735d0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c0106c0 con 0x7f525c0735d0 2024-09-17T03:49:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.492+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c0ce560 con 0x7f525c0735d0 2024-09-17T03:49:42.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.493+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c15d950 con 0x7f525c0735d0 2024-09-17T03:49:42.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.495+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f524c01a070 con 0x7f525c0735d0 2024-09-17T03:49:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.496+0000 7f5253400640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 0x7f523807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.496+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f524c09ae40 con 0x7f525c0735d0 2024-09-17T03:49:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.497+0000 7f5252000640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 0x7f523807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.497+0000 7f5252000640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 0x7f523807b0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f525c0ca4f0 tx=0x7f5244009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:42.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.501+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f524c062c20 con 0x7f525c0735d0 2024-09-17T03:49:42.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.684+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f525c074230 con 0x7f525c0735d0 2024-09-17T03:49:42.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.686+0000 7f5253400640 1 -- 172.21.15.129:0/2776499036 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7f524c067650 con 0x7f525c0735d0 2024-09-17T03:49:42.685 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:42.685 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 msgr2=0x7f523807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 0x7f523807b0a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f525c0ca4f0 tx=0x7f5244009040 comp rx=0 tx=0).stop 2024-09-17T03:49:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 msgr2=0x7f525c0cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f524c00d740 tx=0x7f524c00dc00 comp rx=0 tx=0).stop 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 shutdown_connections 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5238078b90 0x7f523807b0a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f525c0b54d0 0x7f525c0c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f525c0af700 0x7f525c14dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 --2- 172.21.15.129:0/2776499036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f525c0735d0 0x7f525c0cf270 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 >> 172.21.15.129:0/2776499036 conn(0x7f525c06d870 msgr2=0x7f525c0d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 shutdown_connections 2024-09-17T03:49:42.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.688+0000 7f525be00640 1 -- 172.21.15.129:0/2776499036 wait complete. 2024-09-17T03:49:42.708 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-17T03:49:42.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.929+0000 7ff656800640 1 -- 172.21.15.129:0/618325359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 msgr2=0x7ff65015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.929+0000 7ff656800640 1 --2- 172.21.15.129:0/618325359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff65015c780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff640005fd0 tx=0x7ff64002a540 comp rx=0 tx=0).stop 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 -- 172.21.15.129:0/618325359 shutdown_connections 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 --2- 172.21.15.129:0/618325359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 0x7ff650074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 --2- 172.21.15.129:0/618325359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff65015c780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 -- 172.21.15.129:0/618325359 >> 172.21.15.129:0/618325359 conn(0x7ff6500b1cb0 msgr2=0x7ff6500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 -- 172.21.15.129:0/618325359 shutdown_connections 2024-09-17T03:49:42.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.930+0000 7ff656800640 1 -- 172.21.15.129:0/618325359 wait complete. 2024-09-17T03:49:42.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.932+0000 7ff656800640 1 Processor -- start 2024-09-17T03:49:42.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.932+0000 7ff656800640 1 -- start start 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 0x7ff6500c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 0x7ff6500cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff65006f5a0 con 0x7ff650073170 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff650074f80 con 0x7ff650070440 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.933+0000 7ff656800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff65006f720 con 0x7ff650070030 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39576/0 (socket says 172.21.15.129:39576) 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 -- 172.21.15.129:0/1234637002 learned_addr learned my addr 172.21.15.129:0/1234637002 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64f400640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 0x7ff6500c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64ea00640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 0x7ff6500cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 -- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 msgr2=0x7ff6500c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 0x7ff6500c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 -- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 msgr2=0x7ff6500cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 0x7ff6500cbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 -- 172.21.15.129:0/1234637002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6500cac00 con 0x7ff650073170 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64ea00640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 0x7ff6500cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.934+0000 7ff64fe00640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff64800c730 tx=0x7ff64800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.935+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648017070 con 0x7ff650073170 2024-09-17T03:49:42.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.935+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff648003c60 con 0x7ff650073170 2024-09-17T03:49:42.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.935+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6500caee0 con 0x7ff650073170 2024-09-17T03:49:42.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.935+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6480052d0 con 0x7ff650073170 2024-09-17T03:49:42.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.935+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6500cb4a0 con 0x7ff650073170 2024-09-17T03:49:42.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.936+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65006fc40 con 0x7ff650073170 2024-09-17T03:49:42.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.937+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff64801a070 con 0x7ff650073170 2024-09-17T03:49:42.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.938+0000 7ff647400640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 0x7ff63007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:42.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.938+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff64809a5a0 con 0x7ff650073170 2024-09-17T03:49:42.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.938+0000 7ff64f400640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 0x7ff63007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:42.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.939+0000 7ff64f400640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 0x7ff63007b0a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:42.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:42.942+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff6480629d0 con 0x7ff650073170 2024-09-17T03:49:43.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.127+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6500cc760 con 0x7ff650073170 2024-09-17T03:49:43.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.128+0000 7ff647400640 1 -- 172.21.15.129:0/1234637002 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7ff6480622d0 con 0x7ff650073170 2024-09-17T03:49:43.127 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:43.127 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 msgr2=0x7ff63007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 0x7ff63007b0a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 msgr2=0x7ff6501588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff64800c730 tx=0x7ff64800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 shutdown_connections 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff630078b90 0x7ff63007b0a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff650073170 0x7ff6501588f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff650070440 0x7ff6500cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 --2- 172.21.15.129:0/1234637002 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff650070030 0x7ff6500c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.130+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 >> 172.21.15.129:0/1234637002 conn(0x7ff6500b1cb0 msgr2=0x7ff6500afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.131+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 shutdown_connections 2024-09-17T03:49:43.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.131+0000 7ff656800640 1 -- 172.21.15.129:0/1234637002 wait complete. 2024-09-17T03:49:43.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-17T03:49:43.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-17T03:49:43.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-17T03:49:43.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-17T03:49:43.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-17T03:49:43.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-17T03:49:43.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-17T03:49:43.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-17T03:49:43.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 -- 172.21.15.129:0/40808199 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76740ccde0 msgr2=0x7f76740cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 --2- 172.21.15.129:0/40808199 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76740ccde0 0x7f76740cd1f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7674162e50 tx=0x7f766002a410 comp rx=0 tx=0).stop 2024-09-17T03:49:43.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f34e5800640 1 -- 172.21.15.129:0/1361676937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8052360 msgr2=0x7f34d80e7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f34e5800640 1 --2- 172.21.15.129:0/1361676937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8052360 0x7f34d80e7280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34e006b4e0 tx=0x7f34e00a6680 comp rx=0 tx=0).stop 2024-09-17T03:49:43.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 -- 172.21.15.129:0/40808199 shutdown_connections 2024-09-17T03:49:43.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 --2- 172.21.15.129:0/40808199 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76740ccde0 0x7f76740cd1f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 --2- 172.21.15.129:0/40808199 >> v2:172.21.15.181:3300/0 conn(0x7f76741624d0 0x7f76741628e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 --2- 172.21.15.129:0/40808199 >> v2:172.21.15.129:3300/0 conn(0x7f767415f9a0 0x7f76740af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.550+0000 7f767a600640 1 -- 172.21.15.129:0/40808199 >> 172.21.15.129:0/40808199 conn(0x7f76740b20a0 msgr2=0x7f76740b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.551+0000 7f767a600640 1 -- 172.21.15.129:0/40808199 shutdown_connections 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.551+0000 7f767a600640 1 -- 172.21.15.129:0/40808199 wait complete. 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.551+0000 7f34e5800640 1 -- 172.21.15.129:0/1361676937 shutdown_connections 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.551+0000 7f34e5800640 1 --2- 172.21.15.129:0/1361676937 >> v2:172.21.15.181:3300/0 conn(0x7f34d8051630 0x7f34d8051a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.552+0000 7f34e5800640 1 --2- 172.21.15.129:0/1361676937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8052360 0x7f34d80e7280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.552+0000 7f34e5800640 1 -- 172.21.15.129:0/1361676937 >> 172.21.15.129:0/1361676937 conn(0x7f34d801d9a0 msgr2=0x7f34d8039660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.553+0000 7f34e5800640 1 -- 172.21.15.129:0/1361676937 shutdown_connections 2024-09-17T03:49:43.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.553+0000 7f34e5800640 1 -- 172.21.15.129:0/1361676937 wait complete. 2024-09-17T03:49:43.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.555+0000 7f767a600640 1 Processor -- start 2024-09-17T03:49:43.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.555+0000 7f767a600640 1 -- start start 2024-09-17T03:49:43.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.557+0000 7f34e5800640 1 Processor -- start 2024-09-17T03:49:43.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.557+0000 7f34e5800640 1 -- start start 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76740ccde0 0x7f7674156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f767415f9a0 0x7f7674073310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76741624d0 0x7f7674073850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f767415bf10 con 0x7f76741624d0 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f767415bd90 con 0x7f76740ccde0 2024-09-17T03:49:43.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.558+0000 7f767a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f767415c090 con 0x7f767415f9a0 2024-09-17T03:49:43.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.559+0000 7f7673400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76740ccde0 0x7f7674156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.559+0000 7f7673400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76740ccde0 0x7f7674156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59348/0 (socket says 172.21.15.129:59348) 2024-09-17T03:49:43.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.559+0000 7f7673400640 1 -- 172.21.15.129:0/2117965899 learned_addr learned my addr 172.21.15.129:0/2117965899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34d8005c00 0x7f34d804edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8051630 0x7f34d80d7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 0x7f34d804fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8052140 con 0x7f34d8005c00 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8051fc0 con 0x7f34d8052360 2024-09-17T03:49:43.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.560+0000 7f34e5800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f34d8052770 con 0x7f34d8051630 2024-09-17T03:49:43.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 0x7f34d804fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 0x7f34d804fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59368/0 (socket says 172.21.15.129:59368) 2024-09-17T03:49:43.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 -- 172.21.15.129:0/946213498 learned_addr learned my addr 172.21.15.129:0/946213498 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34dfe00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34d8005c00 0x7f34d804edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.559+0000 7f7673e00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76741624d0 0x7f7674073850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34df400640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8051630 0x7f34d80d7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 -- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8051630 msgr2=0x7f34d80d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.559+0000 7f7672a00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f767415f9a0 0x7f7674073310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7673e00640 1 -- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f767415f9a0 msgr2=0x7f7674073310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7673e00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f767415f9a0 0x7f7674073310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7673e00640 1 -- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76740ccde0 msgr2=0x7f7674156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7673e00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76740ccde0 0x7f7674156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7673e00640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7674156a60 con 0x7f76741624d0 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f7672a00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f767415f9a0 0x7f7674073310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34d8051630 0x7f34d80d7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 -- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34d8005c00 msgr2=0x7f34d804edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.561+0000 7f34e4e00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34d8005c00 0x7f34d804edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.562+0000 7f34e4e00640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d81799f0 con 0x7f34d8052360 2024-09-17T03:49:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.562+0000 7f0443c00640 1 -- 172.21.15.129:0/4117762142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 msgr2=0x7f043c0b8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.562+0000 7f0443c00640 1 --2- 172.21.15.129:0/4117762142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c0b8910 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f043c08d9c0 tx=0x7f042802a560 comp rx=0 tx=0).stop 2024-09-17T03:49:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f0443c00640 1 -- 172.21.15.129:0/4117762142 shutdown_connections 2024-09-17T03:49:43.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f0443c00640 1 --2- 172.21.15.129:0/4117762142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c0b8910 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f0443c00640 1 --2- 172.21.15.129:0/4117762142 >> v2:172.21.15.129:3300/0 conn(0x7f043c0b39a0 0x7f043c0b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f0443c00640 1 -- 172.21.15.129:0/4117762142 >> 172.21.15.129:0/4117762142 conn(0x7f043c08bfc0 msgr2=0x7f043c09d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f34e4e00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 0x7f34d804fc30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c3c0 tx=0x7f34d400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f7673e00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76741624d0 0x7f7674073850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7674071d60 tx=0x7f7660033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.563+0000 7f0443c00640 1 -- 172.21.15.129:0/4117762142 shutdown_connections 2024-09-17T03:49:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4017070 con 0x7f34d8052360 2024-09-17T03:49:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34d40053c0 con 0x7f34d8052360 2024-09-17T03:49:43.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4003850 con 0x7f34d8052360 2024-09-17T03:49:43.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660031b50 con 0x7f76741624d0 2024-09-17T03:49:43.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7660031de0 con 0x7f76741624d0 2024-09-17T03:49:43.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7660005280 con 0x7f76741624d0 2024-09-17T03:49:43.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7fdbc0600640 1 -- 172.21.15.129:0/3689663040 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80cffd0 msgr2=0x7fdbb80d4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d8179cd0 con 0x7f34d8052360 2024-09-17T03:49:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7fdbc0600640 1 --2- 172.21.15.129:0/3689663040 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80cffd0 0x7fdbb80d4940 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009be0 tx=0x7fdbb002a740 comp rx=0 tx=0).stop 2024-09-17T03:49:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.565+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d817a290 con 0x7f34d8052360 2024-09-17T03:49:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.565+0000 7fdbc0600640 1 -- 172.21.15.129:0/3689663040 shutdown_connections 2024-09-17T03:49:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.565+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7674074fb0 con 0x7f76741624d0 2024-09-17T03:49:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.565+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76740b44f0 con 0x7f76741624d0 2024-09-17T03:49:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.565+0000 7fdbc0600640 1 --2- 172.21.15.129:0/3689663040 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80cffd0 0x7fdbb80d4940 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 -- 172.21.15.129:0/3259098611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8700af630 msgr2=0x7fb8700ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 --2- 172.21.15.129:0/3259098611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8700af630 0x7fb8700ccb30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb85c005fd0 tx=0x7fb85c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f34d8050170 con 0x7f34d8052360 2024-09-17T03:49:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.564+0000 7f0443c00640 1 -- 172.21.15.129:0/4117762142 wait complete. 2024-09-17T03:49:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f6c5b200640 1 -- 172.21.15.129:0/2000061109 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0b3c90 msgr2=0x7f6c5c0d4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f6c5b200640 1 --2- 172.21.15.129:0/2000061109 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0b3c90 0x7f6c5c0d4590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009900 tx=0x7f6c4802a560 comp rx=0 tx=0).stop 2024-09-17T03:49:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 -- 172.21.15.129:0/3259098611 shutdown_connections 2024-09-17T03:49:43.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f6c5b200640 1 -- 172.21.15.129:0/2000061109 shutdown_connections 2024-09-17T03:49:43.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f6c5b200640 1 --2- 172.21.15.129:0/2000061109 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0b3c90 0x7f6c5c0d4590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 --2- 172.21.15.129:0/3259098611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8700af630 0x7fb8700ccb30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f6c5b200640 1 -- 172.21.15.129:0/2000061109 >> 172.21.15.129:0/2000061109 conn(0x7f6c5c08bfc0 msgr2=0x7f6c5c09d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 --2- 172.21.15.129:0/3259098611 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb870070f20 0x7fb8700cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7f0443c00640 1 Processor -- start 2024-09-17T03:49:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fb875600640 1 -- 172.21.15.129:0/3259098611 >> 172.21.15.129:0/3259098611 conn(0x7fb8700b1cb0 msgr2=0x7fb8700b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.566+0000 7fdbc0600640 1 -- 172.21.15.129:0/3689663040 >> 172.21.15.129:0/3689663040 conn(0x7fdbb808bfc0 msgr2=0x7fdbb809d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7fb875600640 1 -- 172.21.15.129:0/3259098611 shutdown_connections 2024-09-17T03:49:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7f6c5b200640 1 -- 172.21.15.129:0/2000061109 shutdown_connections 2024-09-17T03:49:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7fb875600640 1 -- 172.21.15.129:0/3259098611 wait complete. 2024-09-17T03:49:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7f0443c00640 1 -- start start 2024-09-17T03:49:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7f6c5b200640 1 -- 172.21.15.129:0/2000061109 wait complete. 2024-09-17T03:49:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7fdbc0600640 1 -- 172.21.15.129:0/3689663040 shutdown_connections 2024-09-17T03:49:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.568+0000 7ff6e9800640 1 -- 172.21.15.129:0/3888050976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 msgr2=0x7ff6c80101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.568+0000 7ff6e9800640 1 --2- 172.21.15.129:0/3888050976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6c80101e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc001c80 tx=0x7ff6c8005830 comp rx=0 tx=0).stop 2024-09-17T03:49:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.567+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7674074390 con 0x7f76741624d0 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.569+0000 7ff6e9800640 1 -- 172.21.15.129:0/3888050976 shutdown_connections 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.569+0000 7ff6e9800640 1 --2- 172.21.15.129:0/3888050976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6c80101e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.569+0000 7ff6e9800640 1 -- 172.21.15.129:0/3888050976 >> 172.21.15.129:0/3888050976 conn(0x7ff6e40b1cb0 msgr2=0x7ff6e40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7ff6e9800640 1 -- 172.21.15.129:0/3888050976 shutdown_connections 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7ff6e9800640 1 -- 172.21.15.129:0/3888050976 wait complete. 2024-09-17T03:49:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f34d401a070 con 0x7f34d8052360 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.568+0000 7fdbc0600640 1 -- 172.21.15.129:0/3689663040 wait complete. 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f34d3e00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34c0078bc0 0x7f34c007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f34d409a6d0 con 0x7f34d8052360 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f34d3e00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f34c0080e30 0x7f34c0083280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34d4009e50 con 0x7f34c0080e30 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f34d409aac0 con 0x7f34d8052360 2024-09-17T03:49:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f34dfe00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34c0078bc0 0x7f34c007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7fb875600640 1 Processor -- start 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7fb875600640 1 -- start start 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7660005420 con 0x7f76741624d0 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f766b400640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7654078be0 0x7f765407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f7660080420 con 0x7f76741624d0 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f7673400640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7654078be0 0x7f765407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f766b400640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f7654081b30 0x7f7654084020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7f6c5b200640 1 Processor -- start 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.570+0000 7f6c5b200640 1 -- start start 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 0x7f6c5c0c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0d4ad0 0x7f6c5c1623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c5b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c5c0d5000 0x7f6c5c15d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c5b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c0b1c60 con 0x7f6c5c0d5000 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c5b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5c0b1ae0 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c5b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c5c1606c0 con 0x7f6c5c0d4ad0 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c59e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 0x7f6c5c0c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c59400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0d4ad0 0x7f6c5c1623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c59e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 0x7f6c5c0c8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59388/0 (socket says 172.21.15.129:59388) 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f6c59e00640 1 -- 172.21.15.129:0/3696914978 learned_addr learned my addr 172.21.15.129:0/3696914978 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f043c0b39a0 0x7f043c0a1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c1e2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f043c0d43c0 0x7f043c0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f043c0b91e0 con 0x7f043c0d43c0 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f043c0b9060 con 0x7f043c0b8500 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0443c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f043c0b9360 con 0x7f043c0b39a0 2024-09-17T03:49:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f0441200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f043c0b39a0 0x7f043c0a1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.571+0000 7f043be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c1e2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f043be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c1e2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59378/0 (socket says 172.21.15.129:59378) 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f043be00640 1 -- 172.21.15.129:0/4211551084 learned_addr learned my addr 172.21.15.129:0/4211551084 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f0441c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f043c0d43c0 0x7f043c0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.572+0000 7f0441c00640 1 -- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f043c0b39a0 msgr2=0x7f043c0a1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f0441c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f043c0b39a0 0x7f043c0a1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f0441c00640 1 -- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 msgr2=0x7f043c1e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f0441c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f043c0b8500 0x7f043c1e2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.573+0000 7f0441c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c09ef60 con 0x7f043c0d43c0 2024-09-17T03:49:43.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 0x7fb870158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76540846f0 con 0x7f7654081b30 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f76600809a0 con 0x7f76741624d0 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f7672a00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f7654081b30 0x7f7654084020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 -- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0d4ad0 msgr2=0x7f6c5c1623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c5c0d4ad0 0x7f6c5c1623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 -- 172.21.15.129:0/3696914978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c5c0d5000 msgr2=0x7f6c5c15d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c5c0d5000 0x7f6c5c15d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5c0c86b0 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8700af630 0x7fb870159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8700cf660 0x7fb870159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8700cd070 con 0x7fb8700cf660 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8700cff10 con 0x7fb870070f20 2024-09-17T03:49:43.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb875600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8700cd1f0 con 0x7fb8700af630 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f7673400640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7654078be0 0x7f765407b0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f766c007e70 tx=0x7f766c0075e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c59e00640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 0x7f6c5c0c8170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00b410 tx=0x7f6c4c00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb86ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 0x7fb870158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7fb86f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8700cf660 0x7fb870159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7fb86e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8700af630 0x7fb870159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7fb86ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 0x7fb870158480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59394/0 (socket says 172.21.15.129:59394) 2024-09-17T03:49:43.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7fb86ea00640 1 -- 172.21.15.129:0/3738346005 learned_addr learned my addr 172.21.15.129:0/3738346005 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7ff6e9800640 1 Processor -- start 2024-09-17T03:49:43.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7fdbc0600640 1 Processor -- start 2024-09-17T03:49:43.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7f7672a00640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f7654081b30 0x7f7654084020 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c015020 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.574+0000 7f0441c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f043c0d43c0 0x7f043c0b1710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0428031000 tx=0x7f04280038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c4c011920 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c011d70 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5c161bf0 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7fdbc0600640 1 -- start start 2024-09-17T03:49:43.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7ff6e9800640 1 -- start start 2024-09-17T03:49:43.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.575+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c5c161e90 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 -- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8700af630 msgr2=0x7fb870159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8700af630 0x7fb870159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 -- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8700cf660 msgr2=0x7fb870159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8700cf660 0x7fb870159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8701e7930 con 0x7fb870070f20 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042803e030 con 0x7f043c0d43c0 2024-09-17T03:49:43.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb86ea00640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 0x7fb870158480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b9e0 tx=0x7fb86800bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0428003de0 con 0x7f043c0d43c0 2024-09-17T03:49:43.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428035040 con 0x7f043c0d43c0 2024-09-17T03:49:43.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f34df400640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f34c0080e30 0x7f34c0083280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76540846f0 con 0x7f7654081b30 2024-09-17T03:49:43.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868019070 con 0x7fb870070f20 2024-09-17T03:49:43.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb86800cc90 con 0x7fb870070f20 2024-09-17T03:49:43.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86800ce60 con 0x7fb870070f20 2024-09-17T03:49:43.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.576+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043c0a1350 con 0x7f043c0d43c0 2024-09-17T03:49:43.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7f34dfe00640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34c0078bc0 0x7f34c007b0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f34d8051320 tx=0x7f34e006cc30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80b3910 0x7fdbb815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 0x7fdbb815d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043c0a1910 con 0x7f043c0d43c0 2024-09-17T03:49:43.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbb80cffd0 0x7fdbb80b8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80b34c0 con 0x7fdbb80cffd0 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdbb80b3340 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fdbc0600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdbb80b5a80 con 0x7fdbb80b3910 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.578+0000 7f34df400640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f34c0080e30 0x7f34c0083280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8701589c0 con 0x7fb870070f20 2024-09-17T03:49:43.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8701faad0 con 0x7fb870070f20 2024-09-17T03:49:43.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f34d4009e50 con 0x7f34c0080e30 2024-09-17T03:49:43.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7ff6e9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6e40c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.577+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6c5c159d80 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.578+0000 7fdbbd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 0x7fdbb815d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.578+0000 7fdbbd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 0x7fdbb815d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59402/0 (socket says 172.21.15.129:59402) 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.578+0000 7fdbbd200640 1 -- 172.21.15.129:0/4001813881 learned_addr learned my addr 172.21.15.129:0/4001813881 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbd200640 1 -- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80b3910 msgr2=0x7fdbb815fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbdc00640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80b3910 0x7fdbb815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbd200640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80b3910 0x7fdbb815fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbd200640 1 -- 172.21.15.129:0/4001813881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbb80cffd0 msgr2=0x7fdbb80b8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbd200640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbb80cffd0 0x7fdbb80b8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7fdbbd200640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbb815dd50 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7fdbbdc00640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbb80b3910 0x7fdbb815fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f043c0b3db0 con 0x7f043c0d43c0 2024-09-17T03:49:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.578+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb8700cc1d0 con 0x7fb870070f20 2024-09-17T03:49:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.579+0000 7ff6e9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6e41f5010 0x7ff6e414d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e41f4450 0x7ff6e41f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e40cfd10 con 0x7ff6c800ddc0 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e40cfb90 con 0x7ff6e41f4450 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff6e40cfe90 con 0x7ff6e41f5010 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6e40c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6e40c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39646/0 (socket says 172.21.15.129:39646) 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e2a00640 1 -- 172.21.15.129:0/352997862 learned_addr learned my addr 172.21.15.129:0/352997862 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.580+0000 7ff6e3400640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e41f4450 0x7ff6e41f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e3400640 1 -- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6e41f5010 msgr2=0x7ff6e414d600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e3400640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6e41f5010 0x7ff6e414d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e3400640 1 -- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 msgr2=0x7ff6e40c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e3400640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6e40c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e3400640 1 -- 172.21.15.129:0/352997862 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e41f4da0 con 0x7ff6e41f4450 2024-09-17T03:49:43.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.581+0000 7ff6e2a00640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6c800ddc0 0x7ff6e40c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6e3400640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e41f4450 0x7ff6e41f4860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff6e409e2a0 tx=0x7ff6dc02ade0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7fdbbd200640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 0x7fdbb815d2d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdba400c530 tx=0x7fdba400c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0428041030 con 0x7f043c0d43c0 2024-09-17T03:49:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba4005480 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc006280 con 0x7ff6e41f4450 2024-09-17T03:49:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbb815e030 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.583+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbb8154ac0 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.583+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb86801c070 con 0x7fb870070f20 2024-09-17T03:49:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6dc006420 con 0x7ff6e41f4450 2024-09-17T03:49:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e415d680 con 0x7ff6e41f4450 2024-09-17T03:49:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc006870 con 0x7ff6e41f4450 2024-09-17T03:49:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.582+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e40c9880 con 0x7ff6e41f4450 2024-09-17T03:49:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.584+0000 7fb866a00640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb850078be0 0x7fb85007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.583+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c4c018020 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.584+0000 7f6c52000640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c3c078c90 0x7f6c3c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.584+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdbb8154ea0 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.584+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdba4005480 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.584+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba40035f0 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fb86809ac90 con 0x7fb870070f20 2024-09-17T03:49:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb86e000640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb850078be0 0x7fb85007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6c4c09a850 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f0438c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f041c078c90 0x7f041c07b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb866a00640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fb850080eb0 0x7fb850083300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb86803de50 con 0x7fb850080eb0 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fb86809b040 con 0x7fb870070f20 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f6c59400640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c3c078c90 0x7f6c3c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f6c52000640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f6c3c080f60 0x7f6c3c0833b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c4c018ec0 con 0x7f6c3c080f60 2024-09-17T03:49:43.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f04280bb3d0 con 0x7f043c0d43c0 2024-09-17T03:49:43.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f0438c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f041c080f60 0x7f041c0833b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0428034e90 con 0x7f041c080f60 2024-09-17T03:49:43.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f04280c2050 con 0x7f043c0d43c0 2024-09-17T03:49:43.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f6c4c09ac40 con 0x7f6c5c0b3c90 2024-09-17T03:49:43.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.585+0000 7fb86f400640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fb850080eb0 0x7fb850083300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.586+0000 7fb86e000640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb850078be0 0x7fb85007b0f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb85c033000 tx=0x7fb85c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.586+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff6dc033440 con 0x7ff6e41f4450 2024-09-17T03:49:43.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.586+0000 7f0441200640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f041c078c90 0x7f041c07b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.586+0000 7f043be00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f041c080f60 0x7f041c0833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.587+0000 7fb86f400640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fb850080eb0 0x7fb850083300 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.586+0000 7f6c5a800640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f6c3c080f60 0x7f6c3c0833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7f6c59400640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c3c078c90 0x7f6c3c07b1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c48033000 tx=0x7f6c48004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7ff6daa00640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6c4078d60 0x7ff6c407b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff6dc0ba1a0 con 0x7ff6e41f4450 2024-09-17T03:49:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7f0441200640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f041c078c90 0x7f041c07b1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0434005fd0 tx=0x7f0434009650 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7f6c5a800640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f6c3c080f60 0x7f6c3c0833b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7f043be00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f041c080f60 0x7f041c0833b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7ff6e9800640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7ff6bc001650 0x7ff6bc003b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6bc006c00 con 0x7ff6bc001650 2024-09-17T03:49:43.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb86803de50 con 0x7fb850080eb0 2024-09-17T03:49:43.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7ff6e2a00640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6c4078d60 0x7ff6c407b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7f3d71400640 1 -- 172.21.15.129:0/2187312378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d50004800 msgr2=0x7f3d50004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.588+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdba4003790 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7f3d71400640 1 --2- 172.21.15.129:0/2187312378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d50004800 0x7f3d50004c10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800c260 tx=0x7f3d50005830 comp rx=0 tx=0).stop 2024-09-17T03:49:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbb5600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdb98078dc0 0x7fdb9807b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7f3d71400640 1 -- 172.21.15.129:0/2187312378 shutdown_connections 2024-09-17T03:49:43.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7f3d71400640 1 --2- 172.21.15.129:0/2187312378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d50004800 0x7f3d50004c10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7f3d71400640 1 -- 172.21.15.129:0/2187312378 >> 172.21.15.129:0/2187312378 conn(0x7f3d6c08bfc0 msgr2=0x7f3d6c09d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7ff6e2000640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7ff6bc001650 0x7ff6bc003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fdba409af20 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbbdc00640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdb98078dc0 0x7fdb9807b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbb5600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fdb98081090 0x7fdb980834e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb98083bb0 con 0x7fdb98081090 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.589+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7fdba409b310 con 0x7fdbb80cfbc0 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7f3d71400640 1 -- 172.21.15.129:0/2187312378 shutdown_connections 2024-09-17T03:49:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c4c018ec0 con 0x7f6c3c080f60 2024-09-17T03:49:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0428034e90 con 0x7f041c080f60 2024-09-17T03:49:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7f3d71400640 1 -- 172.21.15.129:0/2187312378 wait complete. 2024-09-17T03:49:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7fdbbe600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fdb98081090 0x7fdb980834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.590+0000 7ff6e2a00640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6c4078d60 0x7ff6c407b270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001860 tx=0x7ff6d4001620 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.591+0000 7ff6e2000640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7ff6bc001650 0x7ff6bc003b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.591+0000 7fdbbdc00640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdb98078dc0 0x7fdb9807b2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbac001270 tx=0x7fdbac009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.592+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6bc006c00 con 0x7ff6bc001650 2024-09-17T03:49:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.593+0000 7f3d71400640 1 Processor -- start 2024-09-17T03:49:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.593+0000 7fdbbe600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fdb98081090 0x7fdb980834e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.593+0000 7f3d71400640 1 -- start start 2024-09-17T03:49:43.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.594+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdb98083bb0 con 0x7fdb98081090 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.595+0000 7f3d71400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d50004800 0x7f3d6c0cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.595+0000 7f3d71400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d6c09ea60 0x7f3d6c1f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.596+0000 7f3d71400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d6c0cc6a0 0x7f3d6c0c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.596+0000 7f3d71400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c0cf420 con 0x7f3d50004800 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.597+0000 7f3d69e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d6c09ea60 0x7f3d6c1f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.597+0000 7f3d6a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d50004800 0x7f3d6c0cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.597+0000 7f3d6a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d50004800 0x7f3d6c0cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39680/0 (socket says 172.21.15.129:39680) 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.597+0000 7f3d6a800640 1 -- 172.21.15.129:0/3561678435 learned_addr learned my addr 172.21.15.129:0/3561678435 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.597+0000 7f3d6b200640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d6c0cc6a0 0x7f3d6c0c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.598+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c0cf2a0 con 0x7f3d6c09ea60 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.598+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6c0cf5a0 con 0x7f3d6c0cc6a0 2024-09-17T03:49:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.599+0000 7f3d69e00640 1 -- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d6c0cc6a0 msgr2=0x7f3d6c0c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.599+0000 7f3d69e00640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d6c0cc6a0 0x7f3d6c0c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.599+0000 7f3d69e00640 1 -- 172.21.15.129:0/3561678435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d50004800 msgr2=0x7f3d6c0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.599+0000 7f3d69e00640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d50004800 0x7f3d6c0cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.599+0000 7f3d69e00640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c09f240 con 0x7f3d6c09ea60 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.601+0000 7f3d69e00640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d6c09ea60 0x7f3d6c1f0a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c500 tx=0x7f3d58004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.601+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58043070 con 0x7f3d6c09ea60 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.601+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d5803a070 con 0x7f3d6c09ea60 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.601+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58001cd0 con 0x7f3d6c09ea60 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.602+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c0a0940 con 0x7f3d6c09ea60 2024-09-17T03:49:43.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.603+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7674074d60 con 0x7f7654081b30 2024-09-17T03:49:43.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.602+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c0a0f00 con 0x7f3d6c09ea60 2024-09-17T03:49:43.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.603+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d6c0b4950 con 0x7f3d6c09ea60 2024-09-17T03:49:43.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.605+0000 7f766b400640 1 -- 172.21.15.129:0/2117965899 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7674074d60 con 0x7f7654081b30 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f7654081b30 msgr2=0x7f7654084020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f7654081b30 0x7f7654084020 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7654078be0 msgr2=0x7f765407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7654078be0 0x7f765407b0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f766c007e70 tx=0x7f766c0075e0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76741624d0 msgr2=0x7f7674073850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f767a600640 1 --2- 172.21.15.129:0/2117965899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76741624d0 0x7f7674073850 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7674071d60 tx=0x7f7660033ba0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f7673e00640 1 -- 172.21.15.129:0/2117965899 reap_dead start 2024-09-17T03:49:43.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.606+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d5803c070 con 0x7f3d6c09ea60 2024-09-17T03:49:43.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.607+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 shutdown_connections 2024-09-17T03:49:43.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.607+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 >> 172.21.15.129:0/2117965899 conn(0x7f76740b20a0 msgr2=0x7f767406f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.607+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 shutdown_connections 2024-09-17T03:49:43.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.607+0000 7f767a600640 1 -- 172.21.15.129:0/2117965899 wait complete. 2024-09-17T03:49:43.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.608+0000 7f3d62a00640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d4c078b90 0x7f3d4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.608+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3d580baa10 con 0x7f3d6c09ea60 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.609+0000 7f3d6a800640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d4c078b90 0x7f3d4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.609+0000 7f3d62a00640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f3d4c080e00 0x7f3d4c083250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.609+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d58006270 con 0x7f3d4c080e00 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.610+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f3d580bf2a0 con 0x7f3d6c09ea60 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.610+0000 7f3d6b200640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f3d4c080e00 0x7f3d4c083250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.610+0000 7f3d6a800640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d4c078b90 0x7f3d4c07b0a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006430 tx=0x7f3d64004ba0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.611+0000 7f3d6b200640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f3d4c080e00 0x7f3d4c083250 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.612+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d58006270 con 0x7f3d4c080e00 2024-09-17T03:49:43.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.615+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdbb815db00 con 0x7fdb98081090 2024-09-17T03:49:43.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.615+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34d804f300 con 0x7f34c0080e30 2024-09-17T03:49:43.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.616+0000 7f34d3e00640 1 -- 172.21.15.129:0/946213498 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f34d804f300 con 0x7f34c0080e30 2024-09-17T03:49:43.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.616+0000 7fdbb5600640 1 -- 172.21.15.129:0/4001813881 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdbb815db00 con 0x7fdb98081090 2024-09-17T03:49:43.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.616+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fdb98081090 msgr2=0x7fdb980834e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fdb98081090 0x7fdb980834e0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdb98078dc0 msgr2=0x7fdb9807b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdb98078dc0 0x7fdb9807b2d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbac001270 tx=0x7fdbac009040 comp rx=0 tx=0).stop 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 msgr2=0x7fdbb815d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 --2- 172.21.15.129:0/4001813881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbb80cfbc0 0x7fdbb815d2d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdba400c530 tx=0x7fdba400c9f0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbbe600640 1 -- 172.21.15.129:0/4001813881 reap_dead start 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 shutdown_connections 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.617+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 >> 172.21.15.129:0/4001813881 conn(0x7fdbb808bfc0 msgr2=0x7fdbb8160d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.618+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 shutdown_connections 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.618+0000 7fdbc0600640 1 -- 172.21.15.129:0/4001813881 wait complete. 2024-09-17T03:49:43.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.619+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f34c0080e30 msgr2=0x7f34c0083280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.619+0000 7f34e5800640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f34c0080e30 0x7f34c0083280 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.621+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb870076820 con 0x7fb850080eb0 2024-09-17T03:49:43.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.621+0000 7fb866a00640 1 -- 172.21.15.129:0/3738346005 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb870076820 con 0x7fb850080eb0 2024-09-17T03:49:43.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34c0078bc0 msgr2=0x7f34c007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34c0078bc0 0x7f34c007b0d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f34d8051320 tx=0x7f34e006cc30 comp rx=0 tx=0).stop 2024-09-17T03:49:43.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 msgr2=0x7f34d804fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 --2- 172.21.15.129:0/946213498 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34d8052360 0x7f34d804fc30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c3c0 tx=0x7f34d400c880 comp rx=0 tx=0).stop 2024-09-17T03:49:43.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fb850080eb0 msgr2=0x7fb850083300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7fb875600640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fb850080eb0 0x7fb850083300 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f043c09ecd0 con 0x7f041c080f60 2024-09-17T03:49:43.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e4e00640 1 -- 172.21.15.129:0/946213498 reap_dead start 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb850078be0 msgr2=0x7fb85007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb850078be0 0x7fb85007b0f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb85c033000 tx=0x7fb85c004730 comp rx=0 tx=0).stop 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 msgr2=0x7fb870158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 --2- 172.21.15.129:0/3738346005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb870070f20 0x7fb870158480 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b9e0 tx=0x7fb86800bea0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb86f400640 1 -- 172.21.15.129:0/3738346005 reap_dead start 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 shutdown_connections 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 >> 172.21.15.129:0/946213498 conn(0x7f34d801d9a0 msgr2=0x7f34d800a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.622+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 shutdown_connections 2024-09-17T03:49:43.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7f34e5800640 1 -- 172.21.15.129:0/946213498 wait complete. 2024-09-17T03:49:43.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 shutdown_connections 2024-09-17T03:49:43.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 >> 172.21.15.129:0/3738346005 conn(0x7fb8700b1cb0 msgr2=0x7fb8700b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.623+0000 7f0438c00640 1 -- 172.21.15.129:0/4211551084 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f043c09ecd0 con 0x7f041c080f60 2024-09-17T03:49:43.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f041c080f60 msgr2=0x7f041c0833b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f041c080f60 0x7f041c0833b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 shutdown_connections 2024-09-17T03:49:43.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7fb875600640 1 -- 172.21.15.129:0/3738346005 wait complete. 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f041c078c90 msgr2=0x7f041c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f041c078c90 0x7f041c07b1a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0434005fd0 tx=0x7f0434009650 comp rx=0 tx=0).stop 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f043c0d43c0 msgr2=0x7f043c0b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0443c00640 1 --2- 172.21.15.129:0/4211551084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f043c0d43c0 0x7f043c0b1710 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0428031000 tx=0x7f04280038b0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.624+0000 7f0441c00640 1 -- 172.21.15.129:0/4211551084 reap_dead start 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.625+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 shutdown_connections 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.625+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 >> 172.21.15.129:0/4211551084 conn(0x7f043c08bfc0 msgr2=0x7f043c1e6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.625+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 shutdown_connections 2024-09-17T03:49:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.626+0000 7f0443c00640 1 -- 172.21.15.129:0/4211551084 wait complete. 2024-09-17T03:49:43.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.626+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff6bc005ce0 con 0x7ff6bc001650 2024-09-17T03:49:43.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.628+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c5c0d4590 con 0x7f6c3c080f60 2024-09-17T03:49:43.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.629+0000 7ff6daa00640 1 -- 172.21.15.129:0/352997862 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff6bc005ce0 con 0x7ff6bc001650 2024-09-17T03:49:43.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.629+0000 7f6c52000640 1 -- 172.21.15.129:0/3696914978 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6c5c0d4590 con 0x7f6c3c080f60 2024-09-17T03:49:43.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.629+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7ff6bc001650 msgr2=0x7ff6bc003b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.629+0000 7ff6e9800640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7ff6bc001650 0x7ff6bc003b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.630+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f6c3c080f60 msgr2=0x7f6c3c0833b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.630+0000 7f6c5b200640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f6c3c080f60 0x7f6c3c0833b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6c4078d60 msgr2=0x7ff6c407b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6c4078d60 0x7ff6c407b270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4001860 tx=0x7ff6d4001620 comp rx=0 tx=0).stop 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e41f4450 msgr2=0x7ff6e41f4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 --2- 172.21.15.129:0/352997862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6e41f4450 0x7ff6e41f4860 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff6e409e2a0 tx=0x7ff6dc02ade0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e3400640 1 -- 172.21.15.129:0/352997862 reap_dead start 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c3c078c90 msgr2=0x7f6c3c07b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c3c078c90 0x7f6c3c07b1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c48033000 tx=0x7f6c48004730 comp rx=0 tx=0).stop 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 msgr2=0x7f6c5c0c8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 --2- 172.21.15.129:0/3696914978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c5c0b3c90 0x7f6c5c0c8170 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c00b410 tx=0x7f6c4c00b8d0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5a800640 1 -- 172.21.15.129:0/3696914978 reap_dead start 2024-09-17T03:49:43.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 shutdown_connections 2024-09-17T03:49:43.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 >> 172.21.15.129:0/352997862 conn(0x7ff6e40b1cb0 msgr2=0x7ff6e40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 shutdown_connections 2024-09-17T03:49:43.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.631+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 >> 172.21.15.129:0/3696914978 conn(0x7f6c5c08bfc0 msgr2=0x7f6c5c0b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.632+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 shutdown_connections 2024-09-17T03:49:43.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.632+0000 7ff6e9800640 1 -- 172.21.15.129:0/352997862 wait complete. 2024-09-17T03:49:43.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.633+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 shutdown_connections 2024-09-17T03:49:43.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.633+0000 7f6c5b200640 1 -- 172.21.15.129:0/3696914978 wait complete. 2024-09-17T03:49:43.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.638+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d6c09eff0 con 0x7f3d4c080e00 2024-09-17T03:49:43.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.639+0000 7f3d62a00640 1 -- 172.21.15.129:0/3561678435 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3d6c09eff0 con 0x7f3d4c080e00 2024-09-17T03:49:43.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f3d4c080e00 msgr2=0x7f3d4c083250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f3d4c080e00 0x7f3d4c083250 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d4c078b90 msgr2=0x7f3d4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d4c078b90 0x7f3d4c07b0a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3d64006430 tx=0x7f3d64004ba0 comp rx=0 tx=0).stop 2024-09-17T03:49:43.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d6c09ea60 msgr2=0x7f3d6c1f0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:43.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d71400640 1 --2- 172.21.15.129:0/3561678435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d6c09ea60 0x7f3d6c1f0a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3d5000c500 tx=0x7f3d58004c80 comp rx=0 tx=0).stop 2024-09-17T03:49:43.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.641+0000 7f3d6b200640 1 -- 172.21.15.129:0/3561678435 reap_dead start 2024-09-17T03:49:43.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.644+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 shutdown_connections 2024-09-17T03:49:43.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.644+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 >> 172.21.15.129:0/3561678435 conn(0x7f3d6c08bfc0 msgr2=0x7f3d6c0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:43.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.644+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 shutdown_connections 2024-09-17T03:49:43.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:43.644+0000 7f3d71400640 1 -- 172.21.15.129:0/3561678435 wait complete. 2024-09-17T03:49:43.644 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:43.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-17T03:49:43.660 INFO:teuthology.orchestra.run.smithi129.stdout:60129542150 2024-09-17T03:49:43.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-17T03:49:43.661 INFO:teuthology.orchestra.run.smithi129.stdout:64424509448 2024-09-17T03:49:43.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-17T03:49:43.662 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:43.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-17T03:49:43.664 INFO:teuthology.orchestra.run.smithi129.stdout:60129542151 2024-09-17T03:49:43.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-17T03:49:43.681 INFO:teuthology.orchestra.run.smithi129.stdout:64424509445 2024-09-17T03:49:43.681 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:43.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-17T03:49:43.682 INFO:teuthology.orchestra.run.smithi129.stdout:64424509445 2024-09-17T03:49:43.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-17T03:49:43.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-17T03:49:44.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 -- 172.21.15.129:0/2427042888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 msgr2=0x7f01f4074960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 --2- 172.21.15.129:0/2427042888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f4074960 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009830 tx=0x7f01e002a500 comp rx=0 tx=0).stop 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 -- 172.21.15.129:0/2427042888 shutdown_connections 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 --2- 172.21.15.129:0/2427042888 >> v2:172.21.15.181:3300/0 conn(0x7f01f40cfdc0 0x7f01f40d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 --2- 172.21.15.129:0/2427042888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f4074960 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 -- 172.21.15.129:0/2427042888 >> 172.21.15.129:0/2427042888 conn(0x7f01f406d4a0 msgr2=0x7f01f406ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 -- 172.21.15.129:0/2427042888 shutdown_connections 2024-09-17T03:49:44.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.026+0000 7f01fb800640 1 -- 172.21.15.129:0/2427042888 wait complete. 2024-09-17T03:49:44.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.029+0000 7fd0f5200640 1 -- 172.21.15.129:0/226328496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80537a0 msgr2=0x7fd0e8053bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.029+0000 7fd0f5200640 1 --2- 172.21.15.129:0/226328496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80537a0 0x7fd0e8053bb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8007a30 tx=0x7fd0e002a520 comp rx=0 tx=0).stop 2024-09-17T03:49:44.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.029+0000 7fd0f5200640 1 -- 172.21.15.129:0/226328496 shutdown_connections 2024-09-17T03:49:44.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7fd0f5200640 1 --2- 172.21.15.129:0/226328496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80537a0 0x7fd0e8053bb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7fd0f5200640 1 -- 172.21.15.129:0/226328496 >> 172.21.15.129:0/226328496 conn(0x7fd0e803a5e0 msgr2=0x7fd0e803a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7f01fb800640 1 Processor -- start 2024-09-17T03:49:44.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7f01fb800640 1 -- start start 2024-09-17T03:49:44.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7fd0f5200640 1 -- 172.21.15.129:0/226328496 shutdown_connections 2024-09-17T03:49:44.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.030+0000 7fd0f5200640 1 -- 172.21.15.129:0/226328496 wait complete. 2024-09-17T03:49:44.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.034+0000 7f01fb800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.034+0000 7fd0f5200640 1 Processor -- start 2024-09-17T03:49:44.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.034+0000 7fd0f5200640 1 -- start start 2024-09-17T03:49:44.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.034+0000 7f01f8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01f8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43266/0 (socket says 172.21.15.129:43266) 2024-09-17T03:49:44.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01f8e00640 1 -- 172.21.15.129:0/2269343474 learned_addr learned my addr 172.21.15.129:0/2269343474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01fb800640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01f40cfdc0 0x7f01f4138460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01fb800640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f41389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01fb800640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01f415bae0 con 0x7f01f4074ea0 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01fb800640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f01f415b960 con 0x7f01f40cfdc0 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.035+0000 7f01fb800640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f01f415bc60 con 0x7f01f4144cc0 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f3e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01f40cfdc0 0x7f01f4138460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f9800640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f41389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f8e00640 1 -- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 msgr2=0x7f01f41389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f8e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f41389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f8e00640 1 -- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01f40cfdc0 msgr2=0x7f01f4138460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f8e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01f40cfdc0 0x7f01f4138460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.036+0000 7f01f8e00640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f413f460 con 0x7f01f4074ea0 2024-09-17T03:49:44.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.037+0000 7fd0f5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 0x7fd0e80d8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.037+0000 7f01f8e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009950 tx=0x7f01e0031720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0f5200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0f5200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 0x7fd0e80def20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0f5200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8058340 con 0x7fd0e80537a0 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0f5200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e80581c0 con 0x7fd0e80e2f20 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 0x7fd0e80d8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0f5200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e80584c0 con 0x7fd0e80576a0 2024-09-17T03:49:44.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0003c40 con 0x7f01f4074ea0 2024-09-17T03:49:44.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7f01fb800640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f413f740 con 0x7f01f4074ea0 2024-09-17T03:49:44.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7f01fb800640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f413fd00 con 0x7f01f4074ea0 2024-09-17T03:49:44.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.039+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01e0005020 con 0x7f01f4074ea0 2024-09-17T03:49:44.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.039+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0003c40 con 0x7f01f4074ea0 2024-09-17T03:49:44.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0ed400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0ed400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:39958/0 (socket says 172.21.15.129:39958) 2024-09-17T03:49:44.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.038+0000 7fd0ed400640 1 -- 172.21.15.129:0/3793886474 learned_addr learned my addr 172.21.15.129:0/3793886474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.039+0000 7fd0ee800640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 0x7fd0e80def20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.042+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01c8005290 con 0x7f01f4074ea0 2024-09-17T03:49:44.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.043+0000 7fd0ee800640 1 -- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 msgr2=0x7fd0e817d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.043+0000 7fd0ee800640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.044+0000 7fd0ee800640 1 -- 172.21.15.129:0/3793886474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 msgr2=0x7fd0e80d8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.044+0000 7fd0ee800640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 0x7fd0e80d8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.043+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f01e00051f0 con 0x7f01f4074ea0 2024-09-17T03:49:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.044+0000 7fd0ee800640 1 -- 172.21.15.129:0/3793886474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e817e310 con 0x7fd0e80e2f20 2024-09-17T03:49:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.044+0000 7fd0ed400640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.045+0000 7fd0ee800640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 0x7fd0e80def20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc001b40 tx=0x7fd0dc00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.045+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc00ab40 con 0x7fd0e80e2f20 2024-09-17T03:49:44.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.046+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0dc00ace0 con 0x7fd0e80e2f20 2024-09-17T03:49:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.046+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e817da70 con 0x7fd0e80e2f20 2024-09-17T03:49:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.046+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e817e030 con 0x7fd0e80e2f20 2024-09-17T03:49:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.046+0000 7f01f0c00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 0x7f01d407b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.046+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f01e00bbbf0 con 0x7f01f4074ea0 2024-09-17T03:49:44.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.048+0000 7f01f3e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 0x7f01d407b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.047+0000 7fd0ede00640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 0x7fd0e80d8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:44.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.048+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0dc005330 con 0x7fd0e80e2f20 2024-09-17T03:49:44.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.048+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e803c300 con 0x7fd0e80e2f20 2024-09-17T03:49:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.049+0000 7f2691e00640 1 -- 172.21.15.129:0/1450185220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 msgr2=0x7f268c073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.049+0000 7f2691e00640 1 --2- 172.21.15.129:0/1450185220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c073160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2684005fd0 tx=0x7f268402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f2691e00640 1 -- 172.21.15.129:0/1450185220 shutdown_connections 2024-09-17T03:49:44.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f01f3e00640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 0x7f01d407b2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f01e8007d60 tx=0x7f01e8007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f2691e00640 1 --2- 172.21.15.129:0/1450185220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c073160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f2691e00640 1 -- 172.21.15.129:0/1450185220 >> 172.21.15.129:0/1450185220 conn(0x7f268c06d870 msgr2=0x7f268c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f2691e00640 1 -- 172.21.15.129:0/1450185220 shutdown_connections 2024-09-17T03:49:44.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.050+0000 7f2691e00640 1 -- 172.21.15.129:0/1450185220 wait complete. 2024-09-17T03:49:44.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.053+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f01e003f030 con 0x7f01f4074ea0 2024-09-17T03:49:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.054+0000 7f2691e00640 1 Processor -- start 2024-09-17T03:49:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.054+0000 7f3fd3c00640 1 -- 172.21.15.129:0/1436863916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0b4090 msgr2=0x7f3fcc0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.054+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/1436863916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc0b3550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc071020 tx=0x7f3fc402a8e0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f2691e00640 1 -- start start 2024-09-17T03:49:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f3fd3c00640 1 -- 172.21.15.129:0/1436863916 shutdown_connections 2024-09-17T03:49:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/1436863916 >> v2:172.21.15.181:3300/0 conn(0x7f3fcc0cfeb0 0x7f3fcc0d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/1436863916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc0b3550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f3fd3c00640 1 -- 172.21.15.129:0/1436863916 >> 172.21.15.129:0/1436863916 conn(0x7f3fcc0cdcc0 msgr2=0x7f3fcc0ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0dc0035f0 con 0x7fd0e80e2f20 2024-09-17T03:49:44.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.055+0000 7f3fd3c00640 1 -- 172.21.15.129:0/1436863916 shutdown_connections 2024-09-17T03:49:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7fd0e6000640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 0x7fd0d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7f2691e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 0x7f268c0b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fd0dc0a2690 con 0x7fd0e80e2f20 2024-09-17T03:49:44.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7f2691e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7f2691e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 0x7f268c15cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.058+0000 7fd0ede00640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 0x7fd0d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f268be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f268be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:45780/0 (socket says 172.21.15.129:45780) 2024-09-17T03:49:44.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.056+0000 7f3fd3c00640 1 -- 172.21.15.129:0/1436863916 wait complete. 2024-09-17T03:49:44.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2691e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f268c0cae20 con 0x7f268c0b22a0 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2690a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 0x7f268c0b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2690a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 0x7f268c0b1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:39970/0 (socket says 172.21.15.129:39970) 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2690a00640 1 -- 172.21.15.129:0/1813597030 learned_addr learned my addr 172.21.15.129:0/1813597030 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f268c070fa0 con 0x7f268c15e210 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.059+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f268c0cafa0 con 0x7f268c15a490 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.060+0000 7f2691400640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 0x7f268c15cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.060+0000 7f3fd3c00640 1 Processor -- start 2024-09-17T03:49:44.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f3fd3c00640 1 -- start start 2024-09-17T03:49:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7fd0ede00640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 0x7fd0d007b0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd0f006b8e0 tx=0x7fd0f0070040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 -- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 msgr2=0x7f268c0b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 0x7f268c0b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 -- 172.21.15.129:0/1813597030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 msgr2=0x7f268c15cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0e803c300 con 0x7fd0e80e2f20 2024-09-17T03:49:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 0x7f268c15cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c14d460 con 0x7f268c15e210 2024-09-17T03:49:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f2691400640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 0x7f268c15cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:44.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.061+0000 7f268be00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f268c0b6090 tx=0x7f267800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678015020 con 0x7f268c15e210 2024-09-17T03:49:44.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2678011920 con 0x7f268c15e210 2024-09-17T03:49:44.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c153360 con 0x7f268c15e210 2024-09-17T03:49:44.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2678011d70 con 0x7f268c15e210 2024-09-17T03:49:44.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.064+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c15bee0 con 0x7f268c15e210 2024-09-17T03:49:44.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.065+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2664005290 con 0x7f268c15e210 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fd1200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fd1200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:39986/0 (socket says 172.21.15.129:39986) 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fcbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fcbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:45790/0 (socket says 172.21.15.129:45790) 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fd3c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0cfeb0 0x7f3fcc1f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.066+0000 7f3fd1200640 1 -- 172.21.15.129:0/3529483738 learned_addr learned my addr 172.21.15.129:0/3529483738 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0b4a30 con 0x7f3fcc0cfeb0 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0b48b0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc0b4bb0 con 0x7f3fcc06fec0 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f6991000640 1 -- 172.21.15.129:0/3097831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0d4230 msgr2=0x7f698c0d4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f6991000640 1 --2- 172.21.15.129:0/3097831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0d4230 0x7f698c0d4640 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f698c08d4e0 tx=0x7f698402a4d0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.067+0000 7f3fd1c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0cfeb0 0x7f3fcc1f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f6991000640 1 -- 172.21.15.129:0/3097831142 shutdown_connections 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f6991000640 1 --2- 172.21.15.129:0/3097831142 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0d4230 0x7f698c0d4640 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f6991000640 1 --2- 172.21.15.129:0/3097831142 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0cfbc0 0x7f698c0cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f6991000640 1 --2- 172.21.15.129:0/3097831142 >> v2:172.21.15.181:3300/0 conn(0x7f698c0b3910 0x7f698c0b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f6991000640 1 -- 172.21.15.129:0/3097831142 >> 172.21.15.129:0/3097831142 conn(0x7f698c08bfc0 msgr2=0x7f698c09d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f3fcbe00640 1 -- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 msgr2=0x7f3fcc155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f3fcbe00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f3fcbe00640 1 -- 172.21.15.129:0/3529483738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0cfeb0 msgr2=0x7f3fcc1f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f3fcbe00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0cfeb0 0x7f3fcc1f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.068+0000 7f3fcbe00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc151580 con 0x7f3fcc0b4090 2024-09-17T03:49:44.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.069+0000 7f3fd1200640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:44.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.069+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2678018020 con 0x7f268c15e210 2024-09-17T03:49:44.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.070+0000 7f3fcbe00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0ced00 tx=0x7f3fb800c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.069+0000 7f6991000640 1 -- 172.21.15.129:0/3097831142 shutdown_connections 2024-09-17T03:49:44.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.069+0000 7f6991000640 1 -- 172.21.15.129:0/3097831142 wait complete. 2024-09-17T03:49:44.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.071+0000 7f2688c00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 0x7f266c07b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.071+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f267809a8e0 con 0x7f268c15e210 2024-09-17T03:49:44.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.071+0000 7f2690a00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 0x7f266c07b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.072+0000 7f2690a00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 0x7f266c07b170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2684006010 tx=0x7f268400b040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.073+0000 7f6991000640 1 Processor -- start 2024-09-17T03:49:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.073+0000 7f6991000640 1 -- start start 2024-09-17T03:49:44.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.074+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80052d0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.074+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fb8011770 con 0x7f3fcc0b4090 2024-09-17T03:49:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.074+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb8011ba0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.074+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc151820 con 0x7f3fcc0b4090 2024-09-17T03:49:44.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.074+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc151de0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.076+0000 7f6991000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 0x7f698c09e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.076+0000 7f6991000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.076+0000 7f6981400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6981400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43308/0 (socket says 172.21.15.129:43308) 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6981400640 1 -- 172.21.15.129:0/776651110 learned_addr learned my addr 172.21.15.129:0/776651110 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 0x7f698c1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b4d80 con 0x7f698c0cfbc0 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b5a80 con 0x7f698c0b3910 2024-09-17T03:49:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6989e00640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 0x7f698c09e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.077+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f698c0b5c00 con 0x7f698c0d4230 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f698a800640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 0x7f698c1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6989e00640 1 -- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 msgr2=0x7f698c1555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6989e00640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 0x7f698c1555a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6989e00640 1 -- 172.21.15.129:0/776651110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 msgr2=0x7f698c0cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6989e00640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6989e00640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c0a0a10 con 0x7f698c0b3910 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.078+0000 7f6981400640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.079+0000 7f698a800640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 0x7f698c1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:49:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.079+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4005290 con 0x7f3fcc0b4090 2024-09-17T03:49:44.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.079+0000 7f6989e00640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 0x7f698c09e200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f697c00d900 tx=0x7f697c00ddc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.080+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c017070 con 0x7f698c0b3910 2024-09-17T03:49:44.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.079+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3fb80035f0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.080+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f697c004e30 con 0x7f698c0b3910 2024-09-17T03:49:44.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.080+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c00bd70 con 0x7f698c0b3910 2024-09-17T03:49:44.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.080+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f267801f020 con 0x7f268c15e210 2024-09-17T03:49:44.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.081+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c0a0cf0 con 0x7f698c0b3910 2024-09-17T03:49:44.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.081+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c0a12b0 con 0x7f698c0b3910 2024-09-17T03:49:44.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.083+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c0d4f00 con 0x7f698c0b3910 2024-09-17T03:49:44.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.083+0000 7f3fc8c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 0x7f3fac07b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.084+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3fb809e2b0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.084+0000 7f3fd1200640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 0x7f3fac07b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.085+0000 7f3fd1200640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 0x7f3fac07b0f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40099a0 tx=0x7f3fc4004a40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.086+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3fb80667e0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.091+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f697c003970 con 0x7f698c0b3910 2024-09-17T03:49:44.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.093+0000 7f6982a00640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 0x7f696807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.094+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f697c09bc80 con 0x7f698c0b3910 2024-09-17T03:49:44.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.094+0000 7f6981400640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 0x7f696807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.095+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f697c09c090 con 0x7f698c0b3910 2024-09-17T03:49:44.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.095+0000 7f6981400640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 0x7f696807b0a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f698c1f9360 tx=0x7f697400a2d0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.115+0000 7fc10d600640 1 -- 172.21.15.129:0/3646230143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d2470 msgr2=0x7fc1080d2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.115+0000 7fc10d600640 1 --2- 172.21.15.129:0/3646230143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d2470 0x7fc1080d2880 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8005fd0 tx=0x7fc0f802a8d0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.116+0000 7fc10d600640 1 -- 172.21.15.129:0/3646230143 shutdown_connections 2024-09-17T03:49:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.116+0000 7fc10d600640 1 --2- 172.21.15.129:0/3646230143 >> v2:172.21.15.181:3300/0 conn(0x7fc10815a100 0x7fc10815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.116+0000 7fc10d600640 1 --2- 172.21.15.129:0/3646230143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d2470 0x7fc1080d2880 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.116+0000 7fc10d600640 1 -- 172.21.15.129:0/3646230143 >> 172.21.15.129:0/3646230143 conn(0x7fc1080b3f10 msgr2=0x7fc1080b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.117+0000 7fc10d600640 1 -- 172.21.15.129:0/3646230143 shutdown_connections 2024-09-17T03:49:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.117+0000 7fc10d600640 1 -- 172.21.15.129:0/3646230143 wait complete. 2024-09-17T03:49:44.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.120+0000 7fc10d600640 1 Processor -- start 2024-09-17T03:49:44.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.120+0000 7f1f25400640 1 -- 172.21.15.129:0/3784203503 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201624d0 msgr2=0x7f1f04004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.121+0000 7f1f25400640 1 --2- 172.21.15.129:0/3784203503 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201624d0 0x7f1f04004800 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009ed0 tx=0x7f1f04005830 comp rx=0 tx=0).stop 2024-09-17T03:49:44.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.121+0000 7f1f1f200640 1 -- 172.21.15.129:0/3784203503 reap_dead start 2024-09-17T03:49:44.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.121+0000 7fc10d600640 1 -- start start 2024-09-17T03:49:44.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.122+0000 7f1f25400640 1 -- 172.21.15.129:0/3784203503 shutdown_connections 2024-09-17T03:49:44.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.122+0000 7f1f25400640 1 -- 172.21.15.129:0/3784203503 >> 172.21.15.129:0/3784203503 conn(0x7f1f200b20a0 msgr2=0x7f1f200b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.122+0000 7f1f25400640 1 -- 172.21.15.129:0/3784203503 shutdown_connections 2024-09-17T03:49:44.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.122+0000 7f1f25400640 1 -- 172.21.15.129:0/3784203503 wait complete. 2024-09-17T03:49:44.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.124+0000 7fc10d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d02b0 0x7fc1081f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.124+0000 7fc10d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.125+0000 7fc10d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 0x7fc1080cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.125+0000 7fc106000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.125+0000 7fc106000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:45852/0 (socket says 172.21.15.129:45852) 2024-09-17T03:49:44.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.125+0000 7fc106000640 1 -- 172.21.15.129:0/4147850102 learned_addr learned my addr 172.21.15.129:0/4147850102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.125+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080b55a0 con 0x7fc10815a100 2024-09-17T03:49:44.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.126+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc1080b5420 con 0x7fc1080d2470 2024-09-17T03:49:44.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.126+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc1080b5720 con 0x7fc1080d02b0 2024-09-17T03:49:44.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.127+0000 7f1f25400640 1 Processor -- start 2024-09-17T03:49:44.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.127+0000 7fc107400640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 0x7fc1080cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.127+0000 7f1f25400640 1 -- start start 2024-09-17T03:49:44.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.128+0000 7fc106a00640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d02b0 0x7fc1081f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.128+0000 7fc106000640 1 -- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d02b0 msgr2=0x7fc1081f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.128+0000 7fc106000640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d02b0 0x7fc1081f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.128+0000 7fc106000640 1 -- 172.21.15.129:0/4147850102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 msgr2=0x7fc1080cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.129+0000 7fc106000640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 0x7fc1080cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.129+0000 7fc106000640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1080ce7a0 con 0x7fc1080d2470 2024-09-17T03:49:44.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.129+0000 7fc107400640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 0x7fc1080cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:44.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f346f800640 1 -- 172.21.15.129:0/4269851185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cd240 msgr2=0x7f34680b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f346f800640 1 --2- 172.21.15.129:0/4269851185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cd240 0x7f34680b2fb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f34680728e0 tx=0x7f346002a840 comp rx=0 tx=0).stop 2024-09-17T03:49:44.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7fc106000640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8006010 tx=0x7fc0f80305f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 0x7f1f2009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 0x7f1f2015c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f2015ce20 0x7f1f201f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1f20162c70 con 0x7f1f201624d0 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1f20162af0 con 0x7f1f2015ce20 2024-09-17T03:49:44.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.130+0000 7f1f25400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1f2015bd90 con 0x7f1f201598f0 2024-09-17T03:49:44.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f346f800640 1 -- 172.21.15.129:0/4269851185 shutdown_connections 2024-09-17T03:49:44.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f346f800640 1 --2- 172.21.15.129:0/4269851185 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cd240 0x7f34680b2fb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f346f800640 1 --2- 172.21.15.129:0/4269851185 >> v2:172.21.15.181:3300/0 conn(0x7f34680cf660 0x7f34680cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f346f800640 1 -- 172.21.15.129:0/4269851185 >> 172.21.15.129:0/4269851185 conn(0x7f34680b1cb0 msgr2=0x7f34680b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8030cb0 con 0x7fc1080d2470 2024-09-17T03:49:44.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f1f1de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 0x7f1f2015c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f1f1de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 0x7f1f2015c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:40034/0 (socket says 172.21.15.129:40034) 2024-09-17T03:49:44.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f1f1de00640 1 -- 172.21.15.129:0/2343003046 learned_addr learned my addr 172.21.15.129:0/2343003046 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.131+0000 7f1f1e800640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 0x7f1f2009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0f8043070 con 0x7fc1080d2470 2024-09-17T03:49:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f803b670 con 0x7fc1080d2470 2024-09-17T03:49:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1081fa190 con 0x7fc1080d2470 2024-09-17T03:49:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1081fa750 con 0x7fc1080d2470 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7f1f1e800640 1 -- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 msgr2=0x7f1f2015c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7f1f1e800640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 0x7f1f2015c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7f1f1e800640 1 -- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f2015ce20 msgr2=0x7f1f201f8e00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7f1f1e800640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f2015ce20 0x7f1f201f8e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.132+0000 7f1f1e800640 1 -- 172.21.15.129:0/2343003046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2009d480 con 0x7f1f201624d0 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.133+0000 7f1f1e800640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 0x7f1f2009cf40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d940 tx=0x7f1f1000de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.133+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f100062d0 con 0x7f1f201624d0 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.134+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f1000b8f0 con 0x7f1f201624d0 2024-09-17T03:49:44.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.134+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f200750b0 con 0x7f1f201624d0 2024-09-17T03:49:44.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.135+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1080d2880 con 0x7fc1080d2470 2024-09-17T03:49:44.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.135+0000 7f346f800640 1 -- 172.21.15.129:0/4269851185 shutdown_connections 2024-09-17T03:49:44.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.134+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000bd00 con 0x7f1f201624d0 2024-09-17T03:49:44.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.134+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f200742f0 con 0x7f1f201624d0 2024-09-17T03:49:44.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.135+0000 7f346f800640 1 -- 172.21.15.129:0/4269851185 wait complete. 2024-09-17T03:49:44.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.137+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc0f8062d10 con 0x7fc1080d2470 2024-09-17T03:49:44.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.139+0000 7fc0fea00640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 0x7fc0e407b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.139+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc0f80bba30 con 0x7fc1080d2470 2024-09-17T03:49:44.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.139+0000 7fc106a00640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 0x7fc0e407b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.138+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1f1001b470 con 0x7f1f201624d0 2024-09-17T03:49:44.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.140+0000 7f1f16a00640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.141+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f1f10099350 con 0x7f1f201624d0 2024-09-17T03:49:44.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.141+0000 7fc106a00640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 0x7fc0e407b170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0000f60 tx=0x7fc0f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.141+0000 7f1f1de00640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.141+0000 7f346f800640 1 Processor -- start 2024-09-17T03:49:44.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.141+0000 7f346f800640 1 -- start start 2024-09-17T03:49:44.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.142+0000 7f1f1de00640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f20073440 tx=0x7f1f18004a40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.142+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ef8005290 con 0x7f1f201624d0 2024-09-17T03:49:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.144+0000 7f346f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cf660 0x7f346809d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.144+0000 7f346f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34680cfb90 0x7f346809b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc0f8083e50 con 0x7fc1080d2470 2024-09-17T03:49:44.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.144+0000 7f346f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34680710a0 con 0x7f34680cf660 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3468070f20 con 0x7f34680cfb90 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f34680b34f0 con 0x7f346809dac0 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:40036/0 (socket says 172.21.15.129:40036) 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 -- 172.21.15.129:0/691332658 learned_addr learned my addr 172.21.15.129:0/691332658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f3467e00640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34680cfb90 0x7f346809b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 -- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34680cfb90 msgr2=0x7f346809b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34680cfb90 0x7f346809b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.145+0000 7f346d800640 1 -- 172.21.15.129:0/691332658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cf660 msgr2=0x7f346809d580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-17T03:49:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.146+0000 7f346ce00640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cf660 0x7f346809d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.146+0000 7f346d800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cf660 0x7f346809d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.146+0000 7f346d800640 1 -- 172.21.15.129:0/691332658 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34681f9910 con 0x7f346809dac0 2024-09-17T03:49:44.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.148+0000 7f346d800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3460009ca0 tx=0x7f3460032b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.150+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460037050 con 0x7f346809dac0 2024-09-17T03:49:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.150+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3460043070 con 0x7f346809dac0 2024-09-17T03:49:44.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.150+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460004610 con 0x7f346809dac0 2024-09-17T03:49:44.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.151+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34681f9bf0 con 0x7f346809dac0 2024-09-17T03:49:44.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.151+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34681fa1b0 con 0x7f346809dac0 2024-09-17T03:49:44.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.153+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34680b39a0 con 0x7f346809dac0 2024-09-17T03:49:44.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.154+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1f200cd730 con 0x7f1f201624d0 2024-09-17T03:49:44.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.157+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f346003e450 con 0x7f346809dac0 2024-09-17T03:49:44.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.158+0000 7f3464c00640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 0x7f344807b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.159+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f34600047b0 con 0x7f346809dac0 2024-09-17T03:49:44.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.159+0000 7f346ce00640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 0x7f344807b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.161+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3460084bd0 con 0x7f346809dac0 2024-09-17T03:49:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.161+0000 7f346ce00640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 0x7f344807b120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34540058c0 tx=0x7f3454005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.482+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f01c8005580 con 0x7f01f4074ea0 2024-09-17T03:49:44.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.483+0000 7f01f0c00640 1 -- 172.21.15.129:0/2269343474 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f01e0083a60 con 0x7f01f4074ea0 2024-09-17T03:49:44.483 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:44.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.484+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fd0e8053bb0 con 0x7fd0e80e2f20 2024-09-17T03:49:44.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.485+0000 7fd0e6000640 1 -- 172.21.15.129:0/3793886474 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd0e80540f0 con 0x7fd0e80e2f20 2024-09-17T03:49:44.487 INFO:teuthology.orchestra.run.smithi129.stdout:60129542150 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 msgr2=0x7fd0d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 0x7fd0d007b0d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd0f006b8e0 tx=0x7fd0f0070040 comp rx=0 tx=0).stop 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 msgr2=0x7fd0e80def20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 0x7fd0e80def20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc001b40 tx=0x7fd0dc00a430 comp rx=0 tx=0).stop 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 shutdown_connections 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078bc0 0x7fd0d007b0d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0e80e2f20 0x7fd0e80def20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0e80576a0 0x7fd0e817d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 --2- 172.21.15.129:0/3793886474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0e80537a0 0x7fd0e80d8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.489+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 >> 172.21.15.129:0/3793886474 conn(0x7fd0e803a5e0 msgr2=0x7fd0e8051bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.490+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 msgr2=0x7f01d407b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.490+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 0x7f01d407b2a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f01e8007d60 tx=0x7f01e8007cf0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.490+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 msgr2=0x7f01f4140500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.490+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f01e0009950 tx=0x7f01e0031720 comp rx=0 tx=0).stop 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 shutdown_connections 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f01d4078d90 0x7f01d407b2a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01f4144cc0 0x7f01f41389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01f40cfdc0 0x7f01f4138460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 shutdown_connections 2024-09-17T03:49:44.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 --2- 172.21.15.129:0/2269343474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01f4074ea0 0x7f01f4140500 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 >> 172.21.15.129:0/2269343474 conn(0x7f01f406d4a0 msgr2=0x7f01f415e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 shutdown_connections 2024-09-17T03:49:44.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.491+0000 7fd0f5200640 1 -- 172.21.15.129:0/3793886474 wait complete. 2024-09-17T03:49:44.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.492+0000 7f01e6000640 1 -- 172.21.15.129:0/2269343474 wait complete. 2024-09-17T03:49:44.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.500+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f3fa4005580 con 0x7f3fcc0b4090 2024-09-17T03:49:44.503 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:44.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.501+0000 7f3fc8c00640 1 -- 172.21.15.129:0/3529483738 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3fb806abc0 con 0x7f3fcc0b4090 2024-09-17T03:49:44.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.506+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 msgr2=0x7f3fac07b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.506+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 0x7f3fac07b0f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40099a0 tx=0x7f3fc4004a40 comp rx=0 tx=0).stop 2024-09-17T03:49:44.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.506+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 msgr2=0x7f3fcc1f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.506+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0ced00 tx=0x7f3fb800c7a0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.506+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 shutdown_connections 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.507+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3fac078be0 0x7f3fac07b0f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.507+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fcc0cfeb0 0x7f3fcc1f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.507+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fcc0b4090 0x7f3fcc1f6db0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.507+0000 7f3fd3c00640 1 --2- 172.21.15.129:0/3529483738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fcc06fec0 0x7f3fcc155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.507+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 >> 172.21.15.129:0/3529483738 conn(0x7f3fcc0cdcc0 msgr2=0x7f3fcc0743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.508+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 shutdown_connections 2024-09-17T03:49:44.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.508+0000 7f3fd3c00640 1 -- 172.21.15.129:0/3529483738 wait complete. 2024-09-17T03:49:44.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.513+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f2664005580 con 0x7f268c15e210 2024-09-17T03:49:44.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.514+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f698c0d02f0 con 0x7f698c0b3910 2024-09-17T03:49:44.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.514+0000 7f2688c00640 1 -- 172.21.15.129:0/1813597030 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f26780626b0 con 0x7f268c15e210 2024-09-17T03:49:44.514 INFO:teuthology.orchestra.run.smithi129.stdout:64424509448 2024-09-17T03:49:44.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.515+0000 7f6982a00640 1 -- 172.21.15.129:0/776651110 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f697c063a50 con 0x7f698c0b3910 2024-09-17T03:49:44.517 INFO:teuthology.orchestra.run.smithi129.stdout:60129542151 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 msgr2=0x7f266c07b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 0x7f266c07b170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2684006010 tx=0x7f268400b040 comp rx=0 tx=0).stop 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 msgr2=0x7f268c15c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f268c0b6090 tx=0x7f267800b8f0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 msgr2=0x7f696807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 0x7f696807b0a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f698c1f9360 tx=0x7f697400a2d0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 msgr2=0x7f698c09e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 0x7f698c09e200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f697c00d900 tx=0x7f697c00ddc0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 shutdown_connections 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f266c078c60 0x7f266c07b170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f268c0b22a0 0x7f268c15cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f268c15e210 0x7f268c15c9c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 --2- 172.21.15.129:0/1813597030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f268c15a490 0x7f268c0b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 >> 172.21.15.129:0/1813597030 conn(0x7f268c06d870 msgr2=0x7f268c0cf120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 shutdown_connections 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6968078b90 0x7f696807b0a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f698c0d4230 0x7f698c1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f698c0cfbc0 0x7f698c0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 --2- 172.21.15.129:0/776651110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f698c0b3910 0x7f698c09e200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.519+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 >> 172.21.15.129:0/776651110 conn(0x7f698c08bfc0 msgr2=0x7f698c0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.520+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 shutdown_connections 2024-09-17T03:49:44.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.520+0000 7f6991000640 1 -- 172.21.15.129:0/776651110 wait complete. 2024-09-17T03:49:44.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.520+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 shutdown_connections 2024-09-17T03:49:44.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.520+0000 7f2691e00640 1 -- 172.21.15.129:0/1813597030 wait complete. 2024-09-17T03:49:44.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.2 2024-09-17T03:49:44.532 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.6 2024-09-17T03:49:44.533 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-09-17T03:49:44.545 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2024-09-17T03:49:44.547 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.5 2024-09-17T03:49:44.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.564+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f1ef8005580 con 0x7f1f201624d0 2024-09-17T03:49:44.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.565+0000 7f1f16a00640 1 -- 172.21.15.129:0/2343003046 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1f200cd730 con 0x7f1f201624d0 2024-09-17T03:49:44.564 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.564 INFO:teuthology.orchestra.run.smithi129.stdout:64424509447 2024-09-17T03:49:44.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.566+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc1081f9f20 con 0x7fc1080d2470 2024-09-17T03:49:44.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7fc0fea00640 1 -- 172.21.15.129:0/4147850102 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc0f8083750 con 0x7fc1080d2470 2024-09-17T03:49:44.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 msgr2=0x7f1f0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.566 INFO:teuthology.orchestra.run.smithi129.stdout:64424509445 2024-09-17T03:49:44.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1f20073440 tx=0x7f1f18004a40 comp rx=0 tx=0).stop 2024-09-17T03:49:44.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 msgr2=0x7f1f2009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 0x7f1f2009cf40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000d940 tx=0x7f1f1000de00 comp rx=0 tx=0).stop 2024-09-17T03:49:44.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.567+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 shutdown_connections 2024-09-17T03:49:44.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f2015ce20 0x7f1f201f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f201598f0 0x7f1f2015c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 --2- 172.21.15.129:0/2343003046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f201624d0 0x7f1f2009cf40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 >> 172.21.15.129:0/2343003046 conn(0x7f1f200b20a0 msgr2=0x7f1f20161a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 shutdown_connections 2024-09-17T03:49:44.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.568+0000 7f1f25400640 1 -- 172.21.15.129:0/2343003046 wait complete. 2024-09-17T03:49:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 msgr2=0x7fc0e407b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 0x7fc0e407b170 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0000f60 tx=0x7fc0f0009040 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 msgr2=0x7fc1080ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8006010 tx=0x7fc0f80305f0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 shutdown_connections 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0e4078c60 0x7fc0e407b170 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc10815a100 0x7fc1080cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1080d2470 0x7fc1080ce260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7fc10d600640 1 --2- 172.21.15.129:0/4147850102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1080d02b0 0x7fc1081f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.570+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f34680ce240 con 0x7f346809dac0 2024-09-17T03:49:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.571+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 >> 172.21.15.129:0/4147850102 conn(0x7fc1080b3f10 msgr2=0x7fc1080b5e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.571+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 shutdown_connections 2024-09-17T03:49:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.571+0000 7fc10d600640 1 -- 172.21.15.129:0/4147850102 wait complete. 2024-09-17T03:49:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.571+0000 7f3464c00640 1 -- 172.21.15.129:0/691332658 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3460088fb0 con 0x7f346809dac0 2024-09-17T03:49:44.572 INFO:teuthology.orchestra.run.smithi129.stdout:64424509445 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 msgr2=0x7f344807b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 0x7f344807b120 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34540058c0 tx=0x7f3454005850 comp rx=0 tx=0).stop 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 msgr2=0x7f34680c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3460009ca0 tx=0x7f3460032b40 comp rx=0 tx=0).stop 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 shutdown_connections 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3448078c10 0x7f344807b120 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f346809dac0 0x7f34680c5ef0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34680cfb90 0x7f346809b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 --2- 172.21.15.129:0/691332658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34680cf660 0x7f346809d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 >> 172.21.15.129:0/691332658 conn(0x7f34680b1cb0 msgr2=0x7f34680b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 shutdown_connections 2024-09-17T03:49:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.574+0000 7f346f800640 1 -- 172.21.15.129:0/691332658 wait complete. 2024-09-17T03:49:44.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.4 2024-09-17T03:49:44.590 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.595 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.3 2024-09-17T03:49:44.595 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.596 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.7 2024-09-17T03:49:44.596 DEBUG:teuthology.parallel:result is None 2024-09-17T03:49:44.596 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:49:44.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 -- 172.21.15.129:0/2576099736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 msgr2=0x7f39f815ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 --2- 172.21.15.129:0/2576099736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f815ef10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f39f80b27e0 tx=0x7f39e002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 -- 172.21.15.129:0/2576099736 shutdown_connections 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 --2- 172.21.15.129:0/2576099736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f815ef10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 --2- 172.21.15.129:0/2576099736 >> v2:172.21.15.181:3300/0 conn(0x7f39f8070f20 0x7f39f815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 -- 172.21.15.129:0/2576099736 >> 172.21.15.129:0/2576099736 conn(0x7f39f806d7d0 msgr2=0x7f39f80b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 -- 172.21.15.129:0/2576099736 shutdown_connections 2024-09-17T03:49:44.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.854+0000 7f39fca00640 1 -- 172.21.15.129:0/2576099736 wait complete. 2024-09-17T03:49:44.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.857+0000 7f39fca00640 1 Processor -- start 2024-09-17T03:49:44.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.857+0000 7f39fca00640 1 -- start start 2024-09-17T03:49:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39f806ee90 0x7f39f81f64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 0x7f39f8157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39f806fb30 con 0x7f39f80cfae0 2024-09-17T03:49:44.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f39f806f9b0 con 0x7f39f806ee90 2024-09-17T03:49:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39fca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f39f80cc970 con 0x7f39f8070f20 2024-09-17T03:49:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39f6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39f6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43380/0 (socket says 172.21.15.129:43380) 2024-09-17T03:49:44.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.858+0000 7f39f6800640 1 -- 172.21.15.129:0/3255304970 learned_addr learned my addr 172.21.15.129:0/3255304970 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39efe00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 0x7f39f8157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f5e00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39f806ee90 0x7f39f81f64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 -- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 msgr2=0x7f39f8157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 0x7f39f8157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 -- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39f806ee90 msgr2=0x7f39f81f64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39f806ee90 0x7f39f81f64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 -- 172.21.15.129:0/3255304970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f81e43e0 con 0x7f39f80cfae0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39efe00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 0x7f39f8157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39f6800640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f39f000c700 tx=0x7f39f000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f0003b70 con 0x7f39f80cfae0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39f0003d10 con 0x7f39f80cfae0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f8157650 con 0x7f39f80cfae0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.859+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f000aab0 con 0x7f39f80cfae0 2024-09-17T03:49:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.860+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f81e4960 con 0x7f39f80cfae0 2024-09-17T03:49:44.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.861+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f81e9bf0 con 0x7f39f80cfae0 2024-09-17T03:49:44.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.862+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39f000ac50 con 0x7f39f80cfae0 2024-09-17T03:49:44.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.863+0000 7f39ee000640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 0x7f39d407b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:44.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.863+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f39f009abd0 con 0x7f39f80cfae0 2024-09-17T03:49:44.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.863+0000 7f39f5e00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 0x7f39d407b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:44.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.864+0000 7f39f5e00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 0x7f39d407b100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39e800ab30 tx=0x7f39e8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:44.868+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39f80b4780 con 0x7f39f80cfae0 2024-09-17T03:49:45.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.047+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39f80b3b50 con 0x7f39d4078bf0 2024-09-17T03:49:45.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.052+0000 7f39ee000640 1 -- 172.21.15.129:0/3255304970 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f39e805a7d0 con 0x7f39d4078bf0 2024-09-17T03:49:45.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:45.055 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:49:45.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 msgr2=0x7f39d407b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 0x7f39d407b100 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39e800ab30 tx=0x7f39e8005f90 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 msgr2=0x7f39f81e5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f39f000c700 tx=0x7f39f000cbc0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 shutdown_connections 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39d4078bf0 0x7f39d407b100 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39f80cfae0 0x7f39f81e5f30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39f8070f20 0x7f39f8157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 --2- 172.21.15.129:0/3255304970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39f806ee90 0x7f39f81f64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.056+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 >> 172.21.15.129:0/3255304970 conn(0x7f39f806d7d0 msgr2=0x7f39f81ea690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.057+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 shutdown_connections 2024-09-17T03:49:45.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.057+0000 7f39fca00640 1 -- 172.21.15.129:0/3255304970 wait complete. 2024-09-17T03:49:45.079 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-17T03:49:44.294980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223380,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751340,"statfs":{"total":773094113280,"available":772865372160,"internally_reserved":0,"allocated":3022848,"data_stored":1800420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589196+0000","last_change":"2024-09-17T03:49:36.406020+0000","last_active":"2024-09-17T03:49:37.589196+0000","last_peered":"2024-09-17T03:49:37.589196+0000","last_clean":"2024-09-17T03:49:37.589196+0000","last_became_active":"2024-09-17T03:49:36.405649+0000","last_became_peered":"2024-09-17T03:49:36.405649+0000","last_unstale":"2024-09-17T03:49:37.589196+0000","last_undegraded":"2024-09-17T03:49:37.589196+0000","last_fullsized":"2024-09-17T03:49:37.589196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377354+0000","last_change":"2024-09-17T03:49:36.400461+0000","last_active":"2024-09-17T03:49:37.377354+0000","last_peered":"2024-09-17T03:49:37.377354+0000","last_clean":"2024-09-17T03:49:37.377354+0000","last_became_active":"2024-09-17T03:49:36.398210+0000","last_became_peered":"2024-09-17T03:49:36.398210+0000","last_unstale":"2024-09-17T03:49:37.377354+0000","last_undegraded":"2024-09-17T03:49:37.377354+0000","last_fullsized":"2024-09-17T03:49:37.377354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589718+0000","last_change":"2024-09-17T03:49:36.397732+0000","last_active":"2024-09-17T03:49:37.589718+0000","last_peered":"2024-09-17T03:49:37.589718+0000","last_clean":"2024-09-17T03:49:37.589718+0000","last_became_active":"2024-09-17T03:49:36.397407+0000","last_became_peered":"2024-09-17T03:49:36.397407+0000","last_unstale":"2024-09-17T03:49:37.589718+0000","last_undegraded":"2024-09-17T03:49:37.589718+0000","last_fullsized":"2024-09-17T03:49:37.589718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379346+0000","last_change":"2024-09-17T03:49:36.410331+0000","last_active":"2024-09-17T03:49:37.379346+0000","last_peered":"2024-09-17T03:49:37.379346+0000","last_clean":"2024-09-17T03:49:37.379346+0000","last_became_active":"2024-09-17T03:49:36.409931+0000","last_became_peered":"2024-09-17T03:49:36.409931+0000","last_unstale":"2024-09-17T03:49:37.379346+0000","last_undegraded":"2024-09-17T03:49:37.379346+0000","last_fullsized":"2024-09-17T03:49:37.379346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589345+0000","last_change":"2024-09-17T03:49:36.415397+0000","last_active":"2024-09-17T03:49:37.589345+0000","last_peered":"2024-09-17T03:49:37.589345+0000","last_clean":"2024-09-17T03:49:37.589345+0000","last_became_active":"2024-09-17T03:49:36.415075+0000","last_became_peered":"2024-09-17T03:49:36.415075+0000","last_unstale":"2024-09-17T03:49:37.589345+0000","last_undegraded":"2024-09-17T03:49:37.589345+0000","last_fullsized":"2024-09-17T03:49:37.589345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:08.390183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379503+0000","last_change":"2024-09-17T03:49:36.405656+0000","last_active":"2024-09-17T03:49:37.379503+0000","last_peered":"2024-09-17T03:49:37.379503+0000","last_clean":"2024-09-17T03:49:37.379503+0000","last_became_active":"2024-09-17T03:49:36.404978+0000","last_became_peered":"2024-09-17T03:49:36.404978+0000","last_unstale":"2024-09-17T03:49:37.379503+0000","last_undegraded":"2024-09-17T03:49:37.379503+0000","last_fullsized":"2024-09-17T03:49:37.379503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588643+0000","last_change":"2024-09-17T03:49:36.402961+0000","last_active":"2024-09-17T03:49:37.588643+0000","last_peered":"2024-09-17T03:49:37.588643+0000","last_clean":"2024-09-17T03:49:37.588643+0000","last_became_active":"2024-09-17T03:49:36.402413+0000","last_became_peered":"2024-09-17T03:49:36.402413+0000","last_unstale":"2024-09-17T03:49:37.588643+0000","last_undegraded":"2024-09-17T03:49:37.588643+0000","last_fullsized":"2024-09-17T03:49:37.588643+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:09.378052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588967+0000","last_change":"2024-09-17T03:49:36.409564+0000","last_active":"2024-09-17T03:49:37.588967+0000","last_peered":"2024-09-17T03:49:37.588967+0000","last_clean":"2024-09-17T03:49:37.588967+0000","last_became_active":"2024-09-17T03:49:36.409150+0000","last_became_peered":"2024-09-17T03:49:36.409150+0000","last_unstale":"2024-09-17T03:49:37.588967+0000","last_undegraded":"2024-09-17T03:49:37.588967+0000","last_fullsized":"2024-09-17T03:49:37.588967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:09.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589233+0000","last_change":"2024-09-17T03:49:36.416599+0000","last_active":"2024-09-17T03:49:37.589233+0000","last_peered":"2024-09-17T03:49:37.589233+0000","last_clean":"2024-09-17T03:49:37.589233+0000","last_became_active":"2024-09-17T03:49:36.416100+0000","last_became_peered":"2024-09-17T03:49:36.416100+0000","last_unstale":"2024-09-17T03:49:37.589233+0000","last_undegraded":"2024-09-17T03:49:37.589233+0000","last_fullsized":"2024-09-17T03:49:37.589233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420661+0000","last_change":"2024-09-17T03:49:36.412006+0000","last_active":"2024-09-17T03:49:37.420661+0000","last_peered":"2024-09-17T03:49:37.420661+0000","last_clean":"2024-09-17T03:49:37.420661+0000","last_became_active":"2024-09-17T03:49:36.411697+0000","last_became_peered":"2024-09-17T03:49:36.411697+0000","last_unstale":"2024-09-17T03:49:37.420661+0000","last_undegraded":"2024-09-17T03:49:37.420661+0000","last_fullsized":"2024-09-17T03:49:37.420661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420825+0000","last_change":"2024-09-17T03:49:36.415050+0000","last_active":"2024-09-17T03:49:37.420825+0000","last_peered":"2024-09-17T03:49:37.420825+0000","last_clean":"2024-09-17T03:49:37.420825+0000","last_became_active":"2024-09-17T03:49:36.414808+0000","last_became_peered":"2024-09-17T03:49:36.414808+0000","last_unstale":"2024-09-17T03:49:37.420825+0000","last_undegraded":"2024-09-17T03:49:37.420825+0000","last_fullsized":"2024-09-17T03:49:37.420825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379278+0000","last_change":"2024-09-17T03:49:36.406397+0000","last_active":"2024-09-17T03:49:37.379278+0000","last_peered":"2024-09-17T03:49:37.379278+0000","last_clean":"2024-09-17T03:49:37.379278+0000","last_became_active":"2024-09-17T03:49:36.405791+0000","last_became_peered":"2024-09-17T03:49:36.405791+0000","last_unstale":"2024-09-17T03:49:37.379278+0000","last_undegraded":"2024-09-17T03:49:37.379278+0000","last_fullsized":"2024-09-17T03:49:37.379278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:05.635714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589437+0000","last_change":"2024-09-17T03:49:36.393913+0000","last_active":"2024-09-17T03:49:37.589437+0000","last_peered":"2024-09-17T03:49:37.589437+0000","last_clean":"2024-09-17T03:49:37.589437+0000","last_became_active":"2024-09-17T03:49:36.393589+0000","last_became_peered":"2024-09-17T03:49:36.393589+0000","last_unstale":"2024-09-17T03:49:37.589437+0000","last_undegraded":"2024-09-17T03:49:37.589437+0000","last_fullsized":"2024-09-17T03:49:37.589437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588596+0000","last_change":"2024-09-17T03:49:36.401985+0000","last_active":"2024-09-17T03:49:37.588596+0000","last_peered":"2024-09-17T03:49:37.588596+0000","last_clean":"2024-09-17T03:49:37.588596+0000","last_became_active":"2024-09-17T03:49:36.401682+0000","last_became_peered":"2024-09-17T03:49:36.401682+0000","last_unstale":"2024-09-17T03:49:37.588596+0000","last_undegraded":"2024-09-17T03:49:37.588596+0000","last_fullsized":"2024-09-17T03:49:37.588596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:46.086705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377130+0000","last_change":"2024-09-17T03:49:36.389920+0000","last_active":"2024-09-17T03:49:37.377130+0000","last_peered":"2024-09-17T03:49:37.377130+0000","last_clean":"2024-09-17T03:49:37.377130+0000","last_became_active":"2024-09-17T03:49:36.389645+0000","last_became_peered":"2024-09-17T03:49:36.389645+0000","last_unstale":"2024-09-17T03:49:37.377130+0000","last_undegraded":"2024-09-17T03:49:37.377130+0000","last_fullsized":"2024-09-17T03:49:37.377130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:20.042459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378538+0000","last_change":"2024-09-17T03:49:36.403578+0000","last_active":"2024-09-17T03:49:37.378538+0000","last_peered":"2024-09-17T03:49:37.378538+0000","last_clean":"2024-09-17T03:49:37.378538+0000","last_became_active":"2024-09-17T03:49:36.403353+0000","last_became_peered":"2024-09-17T03:49:36.403353+0000","last_unstale":"2024-09-17T03:49:37.378538+0000","last_undegraded":"2024-09-17T03:49:37.378538+0000","last_fullsized":"2024-09-17T03:49:37.378538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380751+0000","last_change":"2024-09-17T03:49:36.406004+0000","last_active":"2024-09-17T03:49:37.380751+0000","last_peered":"2024-09-17T03:49:37.380751+0000","last_clean":"2024-09-17T03:49:37.380751+0000","last_became_active":"2024-09-17T03:49:36.403931+0000","last_became_peered":"2024-09-17T03:49:36.403931+0000","last_unstale":"2024-09-17T03:49:37.380751+0000","last_undegraded":"2024-09-17T03:49:37.380751+0000","last_fullsized":"2024-09-17T03:49:37.380751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588086+0000","last_change":"2024-09-17T03:49:36.396161+0000","last_active":"2024-09-17T03:49:37.588086+0000","last_peered":"2024-09-17T03:49:37.588086+0000","last_clean":"2024-09-17T03:49:37.588086+0000","last_became_active":"2024-09-17T03:49:36.395717+0000","last_became_peered":"2024-09-17T03:49:36.395717+0000","last_unstale":"2024-09-17T03:49:37.588086+0000","last_undegraded":"2024-09-17T03:49:37.588086+0000","last_fullsized":"2024-09-17T03:49:37.588086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:42.475745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589346+0000","last_change":"2024-09-17T03:49:36.410754+0000","last_active":"2024-09-17T03:49:37.589346+0000","last_peered":"2024-09-17T03:49:37.589346+0000","last_clean":"2024-09-17T03:49:37.589346+0000","last_became_active":"2024-09-17T03:49:36.404715+0000","last_became_peered":"2024-09-17T03:49:36.404715+0000","last_unstale":"2024-09-17T03:49:37.589346+0000","last_undegraded":"2024-09-17T03:49:37.589346+0000","last_fullsized":"2024-09-17T03:49:37.589346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:51.120762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420579+0000","last_change":"2024-09-17T03:49:36.409394+0000","last_active":"2024-09-17T03:49:37.420579+0000","last_peered":"2024-09-17T03:49:37.420579+0000","last_clean":"2024-09-17T03:49:37.420579+0000","last_became_active":"2024-09-17T03:49:36.408771+0000","last_became_peered":"2024-09-17T03:49:36.408771+0000","last_unstale":"2024-09-17T03:49:37.420579+0000","last_undegraded":"2024-09-17T03:49:37.420579+0000","last_fullsized":"2024-09-17T03:49:37.420579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588435+0000","last_change":"2024-09-17T03:49:36.422246+0000","last_active":"2024-09-17T03:49:37.588435+0000","last_peered":"2024-09-17T03:49:37.588435+0000","last_clean":"2024-09-17T03:49:37.588435+0000","last_became_active":"2024-09-17T03:49:36.421985+0000","last_became_peered":"2024-09-17T03:49:36.421985+0000","last_unstale":"2024-09-17T03:49:37.588435+0000","last_undegraded":"2024-09-17T03:49:37.588435+0000","last_fullsized":"2024-09-17T03:49:37.588435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:13.173976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379785+0000","last_change":"2024-09-17T03:49:36.405326+0000","last_active":"2024-09-17T03:49:37.379785+0000","last_peered":"2024-09-17T03:49:37.379785+0000","last_clean":"2024-09-17T03:49:37.379785+0000","last_became_active":"2024-09-17T03:49:36.404541+0000","last_became_peered":"2024-09-17T03:49:36.404541+0000","last_unstale":"2024-09-17T03:49:37.379785+0000","last_undegraded":"2024-09-17T03:49:37.379785+0000","last_fullsized":"2024-09-17T03:49:37.379785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589367+0000","last_change":"2024-09-17T03:49:36.406574+0000","last_active":"2024-09-17T03:49:37.589367+0000","last_peered":"2024-09-17T03:49:37.589367+0000","last_clean":"2024-09-17T03:49:37.589367+0000","last_became_active":"2024-09-17T03:49:36.406162+0000","last_became_peered":"2024-09-17T03:49:36.406162+0000","last_unstale":"2024-09-17T03:49:37.589367+0000","last_undegraded":"2024-09-17T03:49:37.589367+0000","last_fullsized":"2024-09-17T03:49:37.589367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378837+0000","last_change":"2024-09-17T03:49:36.394666+0000","last_active":"2024-09-17T03:49:37.378837+0000","last_peered":"2024-09-17T03:49:37.378837+0000","last_clean":"2024-09-17T03:49:37.378837+0000","last_became_active":"2024-09-17T03:49:36.394259+0000","last_became_peered":"2024-09-17T03:49:36.394259+0000","last_unstale":"2024-09-17T03:49:37.378837+0000","last_undegraded":"2024-09-17T03:49:37.378837+0000","last_fullsized":"2024-09-17T03:49:37.378837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590715+0000","last_change":"2024-09-17T03:49:36.410498+0000","last_active":"2024-09-17T03:49:37.590715+0000","last_peered":"2024-09-17T03:49:37.590715+0000","last_clean":"2024-09-17T03:49:37.590715+0000","last_became_active":"2024-09-17T03:49:36.410228+0000","last_became_peered":"2024-09-17T03:49:36.410228+0000","last_unstale":"2024-09-17T03:49:37.590715+0000","last_undegraded":"2024-09-17T03:49:37.590715+0000","last_fullsized":"2024-09-17T03:49:37.590715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420810+0000","last_change":"2024-09-17T03:49:36.410333+0000","last_active":"2024-09-17T03:49:37.420810+0000","last_peered":"2024-09-17T03:49:37.420810+0000","last_clean":"2024-09-17T03:49:37.420810+0000","last_became_active":"2024-09-17T03:49:36.405435+0000","last_became_peered":"2024-09-17T03:49:36.405435+0000","last_unstale":"2024-09-17T03:49:37.420810+0000","last_undegraded":"2024-09-17T03:49:37.420810+0000","last_fullsized":"2024-09-17T03:49:37.420810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:21.315923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377561+0000","last_change":"2024-09-17T03:49:36.394974+0000","last_active":"2024-09-17T03:49:37.377561+0000","last_peered":"2024-09-17T03:49:37.377561+0000","last_clean":"2024-09-17T03:49:37.377561+0000","last_became_active":"2024-09-17T03:49:36.393921+0000","last_became_peered":"2024-09-17T03:49:36.393921+0000","last_unstale":"2024-09-17T03:49:37.377561+0000","last_undegraded":"2024-09-17T03:49:37.377561+0000","last_fullsized":"2024-09-17T03:49:37.377561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590561+0000","last_change":"2024-09-17T03:49:36.406505+0000","last_active":"2024-09-17T03:49:37.590561+0000","last_peered":"2024-09-17T03:49:37.590561+0000","last_clean":"2024-09-17T03:49:37.590561+0000","last_became_active":"2024-09-17T03:49:36.406154+0000","last_became_peered":"2024-09-17T03:49:36.406154+0000","last_unstale":"2024-09-17T03:49:37.590561+0000","last_undegraded":"2024-09-17T03:49:37.590561+0000","last_fullsized":"2024-09-17T03:49:37.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588653+0000","last_change":"2024-09-17T03:49:36.400848+0000","last_active":"2024-09-17T03:49:37.588653+0000","last_peered":"2024-09-17T03:49:37.588653+0000","last_clean":"2024-09-17T03:49:37.588653+0000","last_became_active":"2024-09-17T03:49:36.399273+0000","last_became_peered":"2024-09-17T03:49:36.399273+0000","last_unstale":"2024-09-17T03:49:37.588653+0000","last_undegraded":"2024-09-17T03:49:37.588653+0000","last_fullsized":"2024-09-17T03:49:37.588653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379849+0000","last_change":"2024-09-17T03:49:36.409112+0000","last_active":"2024-09-17T03:49:37.379849+0000","last_peered":"2024-09-17T03:49:37.379849+0000","last_clean":"2024-09-17T03:49:37.379849+0000","last_became_active":"2024-09-17T03:49:36.408890+0000","last_became_peered":"2024-09-17T03:49:36.408890+0000","last_unstale":"2024-09-17T03:49:37.379849+0000","last_undegraded":"2024-09-17T03:49:37.379849+0000","last_fullsized":"2024-09-17T03:49:37.379849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588078+0000","last_change":"2024-09-17T03:49:36.408545+0000","last_active":"2024-09-17T03:49:37.588078+0000","last_peered":"2024-09-17T03:49:37.588078+0000","last_clean":"2024-09-17T03:49:37.588078+0000","last_became_active":"2024-09-17T03:49:36.408285+0000","last_became_peered":"2024-09-17T03:49:36.408285+0000","last_unstale":"2024-09-17T03:49:37.588078+0000","last_undegraded":"2024-09-17T03:49:37.588078+0000","last_fullsized":"2024-09-17T03:49:37.588078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420257+0000","last_change":"2024-09-17T03:49:36.401317+0000","last_active":"2024-09-17T03:49:37.420257+0000","last_peered":"2024-09-17T03:49:37.420257+0000","last_clean":"2024-09-17T03:49:37.420257+0000","last_became_active":"2024-09-17T03:49:36.400953+0000","last_became_peered":"2024-09-17T03:49:36.400953+0000","last_unstale":"2024-09-17T03:49:37.420257+0000","last_undegraded":"2024-09-17T03:49:37.420257+0000","last_fullsized":"2024-09-17T03:49:37.420257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:45.195448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378778+0000","last_change":"2024-09-17T03:49:36.400419+0000","last_active":"2024-09-17T03:49:37.378778+0000","last_peered":"2024-09-17T03:49:37.378778+0000","last_clean":"2024-09-17T03:49:37.378778+0000","last_became_active":"2024-09-17T03:49:36.399021+0000","last_became_peered":"2024-09-17T03:49:36.399021+0000","last_unstale":"2024-09-17T03:49:37.378778+0000","last_undegraded":"2024-09-17T03:49:37.378778+0000","last_fullsized":"2024-09-17T03:49:37.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:50.779802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590324+0000","last_change":"2024-09-17T03:49:35.392535+0000","last_active":"2024-09-17T03:49:37.590324+0000","last_peered":"2024-09-17T03:49:37.590324+0000","last_clean":"2024-09-17T03:49:37.590324+0000","last_became_active":"2024-09-17T03:49:35.392219+0000","last_became_peered":"2024-09-17T03:49:35.392219+0000","last_unstale":"2024-09-17T03:49:37.590324+0000","last_undegraded":"2024-09-17T03:49:37.590324+0000","last_fullsized":"2024-09-17T03:49:37.590324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.459490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588861+0000","last_change":"2024-09-17T03:49:36.400903+0000","last_active":"2024-09-17T03:49:37.588861+0000","last_peered":"2024-09-17T03:49:37.588861+0000","last_clean":"2024-09-17T03:49:37.588861+0000","last_became_active":"2024-09-17T03:49:36.400109+0000","last_became_peered":"2024-09-17T03:49:36.400109+0000","last_unstale":"2024-09-17T03:49:37.588861+0000","last_undegraded":"2024-09-17T03:49:37.588861+0000","last_fullsized":"2024-09-17T03:49:37.588861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:34.979447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420155+0000","last_change":"2024-09-17T03:49:35.393512+0000","last_active":"2024-09-17T03:49:37.420155+0000","last_peered":"2024-09-17T03:49:37.420155+0000","last_clean":"2024-09-17T03:49:37.420155+0000","last_became_active":"2024-09-17T03:49:35.388644+0000","last_became_peered":"2024-09-17T03:49:35.388644+0000","last_unstale":"2024-09-17T03:49:37.420155+0000","last_undegraded":"2024-09-17T03:49:37.420155+0000","last_fullsized":"2024-09-17T03:49:37.420155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:02:36.651524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589477+0000","last_change":"2024-09-17T03:49:35.391294+0000","last_active":"2024-09-17T03:49:37.589477+0000","last_peered":"2024-09-17T03:49:37.589477+0000","last_clean":"2024-09-17T03:49:37.589477+0000","last_became_active":"2024-09-17T03:49:35.390970+0000","last_became_peered":"2024-09-17T03:49:35.390970+0000","last_unstale":"2024-09-17T03:49:37.589477+0000","last_undegraded":"2024-09-17T03:49:37.589477+0000","last_fullsized":"2024-09-17T03:49:37.589477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:45.058598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589296+0000","last_change":"2024-09-17T03:49:36.404871+0000","last_active":"2024-09-17T03:49:37.589296+0000","last_peered":"2024-09-17T03:49:37.589296+0000","last_clean":"2024-09-17T03:49:37.589296+0000","last_became_active":"2024-09-17T03:49:36.404452+0000","last_became_peered":"2024-09-17T03:49:36.404452+0000","last_unstale":"2024-09-17T03:49:37.589296+0000","last_undegraded":"2024-09-17T03:49:37.589296+0000","last_fullsized":"2024-09-17T03:49:37.589296+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378971+0000","last_change":"2024-09-17T03:49:35.388702+0000","last_active":"2024-09-17T03:49:37.378971+0000","last_peered":"2024-09-17T03:49:37.378971+0000","last_clean":"2024-09-17T03:49:37.378971+0000","last_became_active":"2024-09-17T03:49:35.388337+0000","last_became_peered":"2024-09-17T03:49:35.388337+0000","last_unstale":"2024-09-17T03:49:37.378971+0000","last_undegraded":"2024-09-17T03:49:37.378971+0000","last_fullsized":"2024-09-17T03:49:37.378971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377377+0000","last_change":"2024-09-17T03:49:36.394908+0000","last_active":"2024-09-17T03:49:37.377377+0000","last_peered":"2024-09-17T03:49:37.377377+0000","last_clean":"2024-09-17T03:49:37.377377+0000","last_became_active":"2024-09-17T03:49:36.394518+0000","last_became_peered":"2024-09-17T03:49:36.394518+0000","last_unstale":"2024-09-17T03:49:37.377377+0000","last_undegraded":"2024-09-17T03:49:37.377377+0000","last_fullsized":"2024-09-17T03:49:37.377377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420535+0000","last_change":"2024-09-17T03:49:35.389956+0000","last_active":"2024-09-17T03:49:37.420535+0000","last_peered":"2024-09-17T03:49:37.420535+0000","last_clean":"2024-09-17T03:49:37.420535+0000","last_became_active":"2024-09-17T03:49:35.389588+0000","last_became_peered":"2024-09-17T03:49:35.389588+0000","last_unstale":"2024-09-17T03:49:37.420535+0000","last_undegraded":"2024-09-17T03:49:37.420535+0000","last_fullsized":"2024-09-17T03:49:37.420535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.510096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380718+0000","last_change":"2024-09-17T03:49:36.405737+0000","last_active":"2024-09-17T03:49:37.380718+0000","last_peered":"2024-09-17T03:49:37.380718+0000","last_clean":"2024-09-17T03:49:37.380718+0000","last_became_active":"2024-09-17T03:49:36.401772+0000","last_became_peered":"2024-09-17T03:49:36.401772+0000","last_unstale":"2024-09-17T03:49:37.380718+0000","last_undegraded":"2024-09-17T03:49:37.380718+0000","last_fullsized":"2024-09-17T03:49:37.380718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380418+0000","last_change":"2024-09-17T03:49:35.395265+0000","last_active":"2024-09-17T03:49:37.380418+0000","last_peered":"2024-09-17T03:49:37.380418+0000","last_clean":"2024-09-17T03:49:37.380418+0000","last_became_active":"2024-09-17T03:49:35.394870+0000","last_became_peered":"2024-09-17T03:49:35.394870+0000","last_unstale":"2024-09-17T03:49:37.380418+0000","last_undegraded":"2024-09-17T03:49:37.380418+0000","last_fullsized":"2024-09-17T03:49:37.380418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590151+0000","last_change":"2024-09-17T03:49:36.407432+0000","last_active":"2024-09-17T03:49:37.590151+0000","last_peered":"2024-09-17T03:49:37.590151+0000","last_clean":"2024-09-17T03:49:37.590151+0000","last_became_active":"2024-09-17T03:49:36.407158+0000","last_became_peered":"2024-09-17T03:49:36.407158+0000","last_unstale":"2024-09-17T03:49:37.590151+0000","last_undegraded":"2024-09-17T03:49:37.590151+0000","last_fullsized":"2024-09-17T03:49:37.590151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377583+0000","last_change":"2024-09-17T03:49:35.390102+0000","last_active":"2024-09-17T03:49:37.377583+0000","last_peered":"2024-09-17T03:49:37.377583+0000","last_clean":"2024-09-17T03:49:37.377583+0000","last_became_active":"2024-09-17T03:49:35.389698+0000","last_became_peered":"2024-09-17T03:49:35.389698+0000","last_unstale":"2024-09-17T03:49:37.377583+0000","last_undegraded":"2024-09-17T03:49:37.377583+0000","last_fullsized":"2024-09-17T03:49:37.377583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379524+0000","last_change":"2024-09-17T03:49:36.406033+0000","last_active":"2024-09-17T03:49:37.379524+0000","last_peered":"2024-09-17T03:49:37.379524+0000","last_clean":"2024-09-17T03:49:37.379524+0000","last_became_active":"2024-09-17T03:49:36.405391+0000","last_became_peered":"2024-09-17T03:49:36.405391+0000","last_unstale":"2024-09-17T03:49:37.379524+0000","last_undegraded":"2024-09-17T03:49:37.379524+0000","last_fullsized":"2024-09-17T03:49:37.379524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:35.463684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420327+0000","last_change":"2024-09-17T03:49:35.392778+0000","last_active":"2024-09-17T03:49:37.420327+0000","last_peered":"2024-09-17T03:49:37.420327+0000","last_clean":"2024-09-17T03:49:37.420327+0000","last_became_active":"2024-09-17T03:49:35.391530+0000","last_became_peered":"2024-09-17T03:49:35.391530+0000","last_unstale":"2024-09-17T03:49:37.420327+0000","last_undegraded":"2024-09-17T03:49:37.420327+0000","last_fullsized":"2024-09-17T03:49:37.420327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:09.206550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588902+0000","last_change":"2024-09-17T03:49:36.404199+0000","last_active":"2024-09-17T03:49:37.588902+0000","last_peered":"2024-09-17T03:49:37.588902+0000","last_clean":"2024-09-17T03:49:37.588902+0000","last_became_active":"2024-09-17T03:49:36.398455+0000","last_became_peered":"2024-09-17T03:49:36.398455+0000","last_unstale":"2024-09-17T03:49:37.588902+0000","last_undegraded":"2024-09-17T03:49:37.588902+0000","last_fullsized":"2024-09-17T03:49:37.588902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588954+0000","last_change":"2024-09-17T03:49:35.387607+0000","last_active":"2024-09-17T03:49:37.588954+0000","last_peered":"2024-09-17T03:49:37.588954+0000","last_clean":"2024-09-17T03:49:37.588954+0000","last_became_active":"2024-09-17T03:49:35.387303+0000","last_became_peered":"2024-09-17T03:49:35.387303+0000","last_unstale":"2024-09-17T03:49:37.588954+0000","last_undegraded":"2024-09-17T03:49:37.588954+0000","last_fullsized":"2024-09-17T03:49:37.588954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:33.456173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588883+0000","last_change":"2024-09-17T03:49:36.413174+0000","last_active":"2024-09-17T03:49:37.588883+0000","last_peered":"2024-09-17T03:49:37.588883+0000","last_clean":"2024-09-17T03:49:37.588883+0000","last_became_active":"2024-09-17T03:49:36.412943+0000","last_became_peered":"2024-09-17T03:49:36.412943+0000","last_unstale":"2024-09-17T03:49:37.588883+0000","last_undegraded":"2024-09-17T03:49:37.588883+0000","last_fullsized":"2024-09-17T03:49:37.588883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379966+0000","last_change":"2024-09-17T03:49:35.388456+0000","last_active":"2024-09-17T03:49:37.379966+0000","last_peered":"2024-09-17T03:49:37.379966+0000","last_clean":"2024-09-17T03:49:37.379966+0000","last_became_active":"2024-09-17T03:49:35.388083+0000","last_became_peered":"2024-09-17T03:49:35.388083+0000","last_unstale":"2024-09-17T03:49:37.379966+0000","last_undegraded":"2024-09-17T03:49:37.379966+0000","last_fullsized":"2024-09-17T03:49:37.379966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588927+0000","last_change":"2024-09-17T03:49:36.414542+0000","last_active":"2024-09-17T03:49:37.588927+0000","last_peered":"2024-09-17T03:49:37.588927+0000","last_clean":"2024-09-17T03:49:37.588927+0000","last_became_active":"2024-09-17T03:49:36.414224+0000","last_became_peered":"2024-09-17T03:49:36.414224+0000","last_unstale":"2024-09-17T03:49:37.588927+0000","last_undegraded":"2024-09-17T03:49:37.588927+0000","last_fullsized":"2024-09-17T03:49:37.588927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420647+0000","last_change":"2024-09-17T03:49:35.388635+0000","last_active":"2024-09-17T03:49:37.420647+0000","last_peered":"2024-09-17T03:49:37.420647+0000","last_clean":"2024-09-17T03:49:37.420647+0000","last_became_active":"2024-09-17T03:49:35.388278+0000","last_became_peered":"2024-09-17T03:49:35.388278+0000","last_unstale":"2024-09-17T03:49:37.420647+0000","last_undegraded":"2024-09-17T03:49:37.420647+0000","last_fullsized":"2024-09-17T03:49:37.420647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:17.648757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420254+0000","last_change":"2024-09-17T03:49:36.401308+0000","last_active":"2024-09-17T03:49:37.420254+0000","last_peered":"2024-09-17T03:49:37.420254+0000","last_clean":"2024-09-17T03:49:37.420254+0000","last_became_active":"2024-09-17T03:49:36.400942+0000","last_became_peered":"2024-09-17T03:49:36.400942+0000","last_unstale":"2024-09-17T03:49:37.420254+0000","last_undegraded":"2024-09-17T03:49:37.420254+0000","last_fullsized":"2024-09-17T03:49:37.420254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588398+0000","last_change":"2024-09-17T03:49:35.372728+0000","last_active":"2024-09-17T03:49:37.588398+0000","last_peered":"2024-09-17T03:49:37.588398+0000","last_clean":"2024-09-17T03:49:37.588398+0000","last_became_active":"2024-09-17T03:49:35.372413+0000","last_became_peered":"2024-09-17T03:49:35.372413+0000","last_unstale":"2024-09-17T03:49:37.588398+0000","last_undegraded":"2024-09-17T03:49:37.588398+0000","last_fullsized":"2024-09-17T03:49:37.588398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588789+0000","last_change":"2024-09-17T03:49:36.413936+0000","last_active":"2024-09-17T03:49:37.588789+0000","last_peered":"2024-09-17T03:49:37.588789+0000","last_clean":"2024-09-17T03:49:37.588789+0000","last_became_active":"2024-09-17T03:49:36.402860+0000","last_became_peered":"2024-09-17T03:49:36.402860+0000","last_unstale":"2024-09-17T03:49:37.588789+0000","last_undegraded":"2024-09-17T03:49:37.588789+0000","last_fullsized":"2024-09-17T03:49:37.588789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380504+0000","last_change":"2024-09-17T03:49:35.385498+0000","last_active":"2024-09-17T03:49:37.380504+0000","last_peered":"2024-09-17T03:49:37.380504+0000","last_clean":"2024-09-17T03:49:37.380504+0000","last_became_active":"2024-09-17T03:49:35.385212+0000","last_became_peered":"2024-09-17T03:49:35.385212+0000","last_unstale":"2024-09-17T03:49:37.380504+0000","last_undegraded":"2024-09-17T03:49:37.380504+0000","last_fullsized":"2024-09-17T03:49:37.380504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:45.909870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589167+0000","last_change":"2024-09-17T03:49:36.413538+0000","last_active":"2024-09-17T03:49:37.589167+0000","last_peered":"2024-09-17T03:49:37.589167+0000","last_clean":"2024-09-17T03:49:37.589167+0000","last_became_active":"2024-09-17T03:49:36.413197+0000","last_became_peered":"2024-09-17T03:49:36.413197+0000","last_unstale":"2024-09-17T03:49:37.589167+0000","last_undegraded":"2024-09-17T03:49:37.589167+0000","last_fullsized":"2024-09-17T03:49:37.589167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588219+0000","last_change":"2024-09-17T03:49:35.384632+0000","last_active":"2024-09-17T03:49:37.588219+0000","last_peered":"2024-09-17T03:49:37.588219+0000","last_clean":"2024-09-17T03:49:37.588219+0000","last_became_active":"2024-09-17T03:49:35.384263+0000","last_became_peered":"2024-09-17T03:49:35.384263+0000","last_unstale":"2024-09-17T03:49:37.588219+0000","last_undegraded":"2024-09-17T03:49:37.588219+0000","last_fullsized":"2024-09-17T03:49:37.588219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589537+0000","last_change":"2024-09-17T03:49:36.415593+0000","last_active":"2024-09-17T03:49:37.589537+0000","last_peered":"2024-09-17T03:49:37.589537+0000","last_clean":"2024-09-17T03:49:37.589537+0000","last_became_active":"2024-09-17T03:49:36.411554+0000","last_became_peered":"2024-09-17T03:49:36.411554+0000","last_unstale":"2024-09-17T03:49:37.589537+0000","last_undegraded":"2024-09-17T03:49:37.589537+0000","last_fullsized":"2024-09-17T03:49:37.589537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:13.481011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377691+0000","last_change":"2024-09-17T03:49:35.387003+0000","last_active":"2024-09-17T03:49:37.377691+0000","last_peered":"2024-09-17T03:49:37.377691+0000","last_clean":"2024-09-17T03:49:37.377691+0000","last_became_active":"2024-09-17T03:49:35.386654+0000","last_became_peered":"2024-09-17T03:49:35.386654+0000","last_unstale":"2024-09-17T03:49:37.377691+0000","last_undegraded":"2024-09-17T03:49:37.377691+0000","last_fullsized":"2024-09-17T03:49:37.377691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:23.004272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379918+0000","last_change":"2024-09-17T03:49:36.395286+0000","last_active":"2024-09-17T03:49:37.379918+0000","last_peered":"2024-09-17T03:49:37.379918+0000","last_clean":"2024-09-17T03:49:37.379918+0000","last_became_active":"2024-09-17T03:49:36.394025+0000","last_became_peered":"2024-09-17T03:49:36.394025+0000","last_unstale":"2024-09-17T03:49:37.379918+0000","last_undegraded":"2024-09-17T03:49:37.379918+0000","last_fullsized":"2024-09-17T03:49:37.379918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588329+0000","last_change":"2024-09-17T03:49:35.389511+0000","last_active":"2024-09-17T03:49:37.588329+0000","last_peered":"2024-09-17T03:49:37.588329+0000","last_clean":"2024-09-17T03:49:37.588329+0000","last_became_active":"2024-09-17T03:49:35.389061+0000","last_became_peered":"2024-09-17T03:49:35.389061+0000","last_unstale":"2024-09-17T03:49:37.588329+0000","last_undegraded":"2024-09-17T03:49:37.588329+0000","last_fullsized":"2024-09-17T03:49:37.588329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589110+0000","last_change":"2024-09-17T03:49:36.418896+0000","last_active":"2024-09-17T03:49:37.589110+0000","last_peered":"2024-09-17T03:49:37.589110+0000","last_clean":"2024-09-17T03:49:37.589110+0000","last_became_active":"2024-09-17T03:49:36.418567+0000","last_became_peered":"2024-09-17T03:49:36.418567+0000","last_unstale":"2024-09-17T03:49:37.589110+0000","last_undegraded":"2024-09-17T03:49:37.589110+0000","last_fullsized":"2024-09-17T03:49:37.589110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588519+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:49:37.588519+0000","last_peered":"2024-09-17T03:49:37.588519+0000","last_clean":"2024-09-17T03:49:37.588519+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:49:37.588519+0000","last_undegraded":"2024-09-17T03:49:37.588519+0000","last_fullsized":"2024-09-17T03:49:37.588519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378615+0000","last_change":"2024-09-17T03:49:35.379012+0000","last_active":"2024-09-17T03:49:37.378615+0000","last_peered":"2024-09-17T03:49:37.378615+0000","last_clean":"2024-09-17T03:49:37.378615+0000","last_became_active":"2024-09-17T03:49:35.378349+0000","last_became_peered":"2024-09-17T03:49:35.378349+0000","last_unstale":"2024-09-17T03:49:37.378615+0000","last_undegraded":"2024-09-17T03:49:37.378615+0000","last_fullsized":"2024-09-17T03:49:37.378615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589522+0000","last_change":"2024-09-17T03:49:36.408105+0000","last_active":"2024-09-17T03:49:37.589522+0000","last_peered":"2024-09-17T03:49:37.589522+0000","last_clean":"2024-09-17T03:49:37.589522+0000","last_became_active":"2024-09-17T03:49:36.407814+0000","last_became_peered":"2024-09-17T03:49:36.407814+0000","last_unstale":"2024-09-17T03:49:37.589522+0000","last_undegraded":"2024-09-17T03:49:37.589522+0000","last_fullsized":"2024-09-17T03:49:37.589522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588438+0000","last_change":"2024-09-17T03:49:35.399235+0000","last_active":"2024-09-17T03:49:37.588438+0000","last_peered":"2024-09-17T03:49:37.588438+0000","last_clean":"2024-09-17T03:49:37.588438+0000","last_became_active":"2024-09-17T03:49:35.398860+0000","last_became_peered":"2024-09-17T03:49:35.398860+0000","last_unstale":"2024-09-17T03:49:37.588438+0000","last_undegraded":"2024-09-17T03:49:37.588438+0000","last_fullsized":"2024-09-17T03:49:37.588438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:34.455037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588130+0000","last_change":"2024-09-17T03:49:36.403778+0000","last_active":"2024-09-17T03:49:37.588130+0000","last_peered":"2024-09-17T03:49:37.588130+0000","last_clean":"2024-09-17T03:49:37.588130+0000","last_became_active":"2024-09-17T03:49:36.403408+0000","last_became_peered":"2024-09-17T03:49:36.403408+0000","last_unstale":"2024-09-17T03:49:37.588130+0000","last_undegraded":"2024-09-17T03:49:37.588130+0000","last_fullsized":"2024-09-17T03:49:37.588130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:19.846081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.587984+0000","last_change":"2024-09-17T03:49:35.384616+0000","last_active":"2024-09-17T03:49:37.587984+0000","last_peered":"2024-09-17T03:49:37.587984+0000","last_clean":"2024-09-17T03:49:37.587984+0000","last_became_active":"2024-09-17T03:49:35.384264+0000","last_became_peered":"2024-09-17T03:49:35.384264+0000","last_unstale":"2024-09-17T03:49:37.587984+0000","last_undegraded":"2024-09-17T03:49:37.587984+0000","last_fullsized":"2024-09-17T03:49:37.587984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377132+0000","last_change":"2024-09-17T03:49:36.381928+0000","last_active":"2024-09-17T03:49:37.377132+0000","last_peered":"2024-09-17T03:49:37.377132+0000","last_clean":"2024-09-17T03:49:37.377132+0000","last_became_active":"2024-09-17T03:49:36.381629+0000","last_became_peered":"2024-09-17T03:49:36.381629+0000","last_unstale":"2024-09-17T03:49:37.377132+0000","last_undegraded":"2024-09-17T03:49:37.377132+0000","last_fullsized":"2024-09-17T03:49:37.377132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379997+0000","last_change":"2024-09-17T03:49:35.396687+0000","last_active":"2024-09-17T03:49:37.379997+0000","last_peered":"2024-09-17T03:49:37.379997+0000","last_clean":"2024-09-17T03:49:37.379997+0000","last_became_active":"2024-09-17T03:49:35.390958+0000","last_became_peered":"2024-09-17T03:49:35.390958+0000","last_unstale":"2024-09-17T03:49:37.379997+0000","last_undegraded":"2024-09-17T03:49:37.379997+0000","last_fullsized":"2024-09-17T03:49:37.379997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378943+0000","last_change":"2024-09-17T03:49:36.406614+0000","last_active":"2024-09-17T03:49:37.378943+0000","last_peered":"2024-09-17T03:49:37.378943+0000","last_clean":"2024-09-17T03:49:37.378943+0000","last_became_active":"2024-09-17T03:49:36.403577+0000","last_became_peered":"2024-09-17T03:49:36.403577+0000","last_unstale":"2024-09-17T03:49:37.378943+0000","last_undegraded":"2024-09-17T03:49:37.378943+0000","last_fullsized":"2024-09-17T03:49:37.378943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:45.693650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377460+0000","last_change":"2024-09-17T03:49:35.385825+0000","last_active":"2024-09-17T03:49:37.377460+0000","last_peered":"2024-09-17T03:49:37.377460+0000","last_clean":"2024-09-17T03:49:37.377460+0000","last_became_active":"2024-09-17T03:49:35.385489+0000","last_became_peered":"2024-09-17T03:49:35.385489+0000","last_unstale":"2024-09-17T03:49:37.377460+0000","last_undegraded":"2024-09-17T03:49:37.377460+0000","last_fullsized":"2024-09-17T03:49:37.377460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380572+0000","last_change":"2024-09-17T03:49:36.413827+0000","last_active":"2024-09-17T03:49:37.380572+0000","last_peered":"2024-09-17T03:49:37.380572+0000","last_clean":"2024-09-17T03:49:37.380572+0000","last_became_active":"2024-09-17T03:49:36.413432+0000","last_became_peered":"2024-09-17T03:49:36.413432+0000","last_unstale":"2024-09-17T03:49:37.380572+0000","last_undegraded":"2024-09-17T03:49:37.380572+0000","last_fullsized":"2024-09-17T03:49:37.380572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589281+0000","last_change":"2024-09-17T03:49:35.396951+0000","last_active":"2024-09-17T03:49:37.589281+0000","last_peered":"2024-09-17T03:49:37.589281+0000","last_clean":"2024-09-17T03:49:37.589281+0000","last_became_active":"2024-09-17T03:49:35.396554+0000","last_became_peered":"2024-09-17T03:49:35.396554+0000","last_unstale":"2024-09-17T03:49:37.589281+0000","last_undegraded":"2024-09-17T03:49:37.589281+0000","last_fullsized":"2024-09-17T03:49:37.589281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:23.665085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377339+0000","last_change":"2024-09-17T03:49:36.387372+0000","last_active":"2024-09-17T03:49:37.377339+0000","last_peered":"2024-09-17T03:49:37.377339+0000","last_clean":"2024-09-17T03:49:37.377339+0000","last_became_active":"2024-09-17T03:49:36.384715+0000","last_became_peered":"2024-09-17T03:49:36.384715+0000","last_unstale":"2024-09-17T03:49:37.377339+0000","last_undegraded":"2024-09-17T03:49:37.377339+0000","last_fullsized":"2024-09-17T03:49:37.377339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.419437+0000","last_change":"2024-09-17T03:49:35.393853+0000","last_active":"2024-09-17T03:49:37.419437+0000","last_peered":"2024-09-17T03:49:37.419437+0000","last_clean":"2024-09-17T03:49:37.419437+0000","last_became_active":"2024-09-17T03:49:35.393551+0000","last_became_peered":"2024-09-17T03:49:35.393551+0000","last_unstale":"2024-09-17T03:49:37.419437+0000","last_undegraded":"2024-09-17T03:49:37.419437+0000","last_fullsized":"2024-09-17T03:49:37.419437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590925+0000","last_change":"2024-09-17T03:49:36.412628+0000","last_active":"2024-09-17T03:49:37.590925+0000","last_peered":"2024-09-17T03:49:37.590925+0000","last_clean":"2024-09-17T03:49:37.590925+0000","last_became_active":"2024-09-17T03:49:36.412320+0000","last_became_peered":"2024-09-17T03:49:36.412320+0000","last_unstale":"2024-09-17T03:49:37.590925+0000","last_undegraded":"2024-09-17T03:49:37.590925+0000","last_fullsized":"2024-09-17T03:49:37.590925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420603+0000","last_change":"2024-09-17T03:49:35.389093+0000","last_active":"2024-09-17T03:49:37.420603+0000","last_peered":"2024-09-17T03:49:37.420603+0000","last_clean":"2024-09-17T03:49:37.420603+0000","last_became_active":"2024-09-17T03:49:35.388625+0000","last_became_peered":"2024-09-17T03:49:35.388625+0000","last_unstale":"2024-09-17T03:49:37.420603+0000","last_undegraded":"2024-09-17T03:49:37.420603+0000","last_fullsized":"2024-09-17T03:49:37.420603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.987962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589830+0000","last_change":"2024-09-17T03:49:36.397328+0000","last_active":"2024-09-17T03:49:37.589830+0000","last_peered":"2024-09-17T03:49:37.589830+0000","last_clean":"2024-09-17T03:49:37.589830+0000","last_became_active":"2024-09-17T03:49:36.396910+0000","last_became_peered":"2024-09-17T03:49:36.396910+0000","last_unstale":"2024-09-17T03:49:37.589830+0000","last_undegraded":"2024-09-17T03:49:37.589830+0000","last_fullsized":"2024-09-17T03:49:37.589830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588441+0000","last_change":"2024-09-17T03:49:35.391714+0000","last_active":"2024-09-17T03:49:37.588441+0000","last_peered":"2024-09-17T03:49:37.588441+0000","last_clean":"2024-09-17T03:49:37.588441+0000","last_became_active":"2024-09-17T03:49:35.391374+0000","last_became_peered":"2024-09-17T03:49:35.391374+0000","last_unstale":"2024-09-17T03:49:37.588441+0000","last_undegraded":"2024-09-17T03:49:37.588441+0000","last_fullsized":"2024-09-17T03:49:37.588441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380255+0000","last_change":"2024-09-17T03:49:36.395381+0000","last_active":"2024-09-17T03:49:37.380255+0000","last_peered":"2024-09-17T03:49:37.380255+0000","last_clean":"2024-09-17T03:49:37.380255+0000","last_became_active":"2024-09-17T03:49:36.394430+0000","last_became_peered":"2024-09-17T03:49:36.394430+0000","last_unstale":"2024-09-17T03:49:37.380255+0000","last_undegraded":"2024-09-17T03:49:37.380255+0000","last_fullsized":"2024-09-17T03:49:37.380255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377304+0000","last_change":"2024-09-17T03:49:35.386828+0000","last_active":"2024-09-17T03:49:37.377304+0000","last_peered":"2024-09-17T03:49:37.377304+0000","last_clean":"2024-09-17T03:49:37.377304+0000","last_became_active":"2024-09-17T03:49:35.385425+0000","last_became_peered":"2024-09-17T03:49:35.385425+0000","last_unstale":"2024-09-17T03:49:37.377304+0000","last_undegraded":"2024-09-17T03:49:37.377304+0000","last_fullsized":"2024-09-17T03:49:37.377304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589280+0000","last_change":"2024-09-17T03:49:36.396082+0000","last_active":"2024-09-17T03:49:37.589280+0000","last_peered":"2024-09-17T03:49:37.589280+0000","last_clean":"2024-09-17T03:49:37.589280+0000","last_became_active":"2024-09-17T03:49:36.395733+0000","last_became_peered":"2024-09-17T03:49:36.395733+0000","last_unstale":"2024-09-17T03:49:37.589280+0000","last_undegraded":"2024-09-17T03:49:37.589280+0000","last_fullsized":"2024-09-17T03:49:37.589280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377167+0000","last_change":"2024-09-17T03:49:35.383855+0000","last_active":"2024-09-17T03:49:37.377167+0000","last_peered":"2024-09-17T03:49:37.377167+0000","last_clean":"2024-09-17T03:49:37.377167+0000","last_became_active":"2024-09-17T03:49:35.383513+0000","last_became_peered":"2024-09-17T03:49:35.383513+0000","last_unstale":"2024-09-17T03:49:37.377167+0000","last_undegraded":"2024-09-17T03:49:37.377167+0000","last_fullsized":"2024-09-17T03:49:37.377167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589610+0000","last_change":"2024-09-17T03:49:36.401383+0000","last_active":"2024-09-17T03:49:37.589610+0000","last_peered":"2024-09-17T03:49:37.589610+0000","last_clean":"2024-09-17T03:49:37.589610+0000","last_became_active":"2024-09-17T03:49:36.401112+0000","last_became_peered":"2024-09-17T03:49:36.401112+0000","last_unstale":"2024-09-17T03:49:37.589610+0000","last_undegraded":"2024-09-17T03:49:37.589610+0000","last_fullsized":"2024-09-17T03:49:37.589610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589087+0000","last_change":"2024-09-17T03:49:35.405334+0000","last_active":"2024-09-17T03:49:37.589087+0000","last_peered":"2024-09-17T03:49:37.589087+0000","last_clean":"2024-09-17T03:49:37.589087+0000","last_became_active":"2024-09-17T03:49:35.404955+0000","last_became_peered":"2024-09-17T03:49:35.404955+0000","last_unstale":"2024-09-17T03:49:37.589087+0000","last_undegraded":"2024-09-17T03:49:37.589087+0000","last_fullsized":"2024-09-17T03:49:37.589087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420485+0000","last_change":"2024-09-17T03:49:36.412896+0000","last_active":"2024-09-17T03:49:37.420485+0000","last_peered":"2024-09-17T03:49:37.420485+0000","last_clean":"2024-09-17T03:49:37.420485+0000","last_became_active":"2024-09-17T03:49:36.406254+0000","last_became_peered":"2024-09-17T03:49:36.406254+0000","last_unstale":"2024-09-17T03:49:37.420485+0000","last_undegraded":"2024-09-17T03:49:37.420485+0000","last_fullsized":"2024-09-17T03:49:37.420485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380554+0000","last_change":"2024-09-17T03:49:36.396792+0000","last_active":"2024-09-17T03:49:37.380554+0000","last_peered":"2024-09-17T03:49:37.380554+0000","last_clean":"2024-09-17T03:49:37.380554+0000","last_became_active":"2024-09-17T03:49:36.393651+0000","last_became_peered":"2024-09-17T03:49:36.393651+0000","last_unstale":"2024-09-17T03:49:37.380554+0000","last_undegraded":"2024-09-17T03:49:37.380554+0000","last_fullsized":"2024-09-17T03:49:37.380554+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588722+0000","last_change":"2024-09-17T03:49:35.398891+0000","last_active":"2024-09-17T03:49:37.588722+0000","last_peered":"2024-09-17T03:49:37.588722+0000","last_clean":"2024-09-17T03:49:37.588722+0000","last_became_active":"2024-09-17T03:49:35.398494+0000","last_became_peered":"2024-09-17T03:49:35.398494+0000","last_unstale":"2024-09-17T03:49:37.588722+0000","last_undegraded":"2024-09-17T03:49:37.588722+0000","last_fullsized":"2024-09-17T03:49:37.588722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:34.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379744+0000","last_change":"2024-09-17T03:49:36.394357+0000","last_active":"2024-09-17T03:49:37.379744+0000","last_peered":"2024-09-17T03:49:37.379744+0000","last_clean":"2024-09-17T03:49:37.379744+0000","last_became_active":"2024-09-17T03:49:36.393549+0000","last_became_peered":"2024-09-17T03:49:36.393549+0000","last_unstale":"2024-09-17T03:49:37.379744+0000","last_undegraded":"2024-09-17T03:49:37.379744+0000","last_fullsized":"2024-09-17T03:49:37.379744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589049+0000","last_change":"2024-09-17T03:49:35.398839+0000","last_active":"2024-09-17T03:49:37.589049+0000","last_peered":"2024-09-17T03:49:37.589049+0000","last_clean":"2024-09-17T03:49:37.589049+0000","last_became_active":"2024-09-17T03:49:35.398355+0000","last_became_peered":"2024-09-17T03:49:35.398355+0000","last_unstale":"2024-09-17T03:49:37.589049+0000","last_undegraded":"2024-09-17T03:49:37.589049+0000","last_fullsized":"2024-09-17T03:49:37.589049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:51.292462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589985+0000","last_change":"2024-09-17T03:49:36.409308+0000","last_active":"2024-09-17T03:49:37.589985+0000","last_peered":"2024-09-17T03:49:37.589985+0000","last_clean":"2024-09-17T03:49:37.589985+0000","last_became_active":"2024-09-17T03:49:36.408373+0000","last_became_peered":"2024-09-17T03:49:36.408373+0000","last_unstale":"2024-09-17T03:49:37.589985+0000","last_undegraded":"2024-09-17T03:49:37.589985+0000","last_fullsized":"2024-09-17T03:49:37.589985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590572+0000","last_change":"2024-09-17T03:49:35.381739+0000","last_active":"2024-09-17T03:49:37.590572+0000","last_peered":"2024-09-17T03:49:37.590572+0000","last_clean":"2024-09-17T03:49:37.590572+0000","last_became_active":"2024-09-17T03:49:35.381185+0000","last_became_peered":"2024-09-17T03:49:35.381185+0000","last_unstale":"2024-09-17T03:49:37.590572+0000","last_undegraded":"2024-09-17T03:49:37.590572+0000","last_fullsized":"2024-09-17T03:49:37.590572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590630+0000","last_change":"2024-09-17T03:49:36.401989+0000","last_active":"2024-09-17T03:49:37.590630+0000","last_peered":"2024-09-17T03:49:37.590630+0000","last_clean":"2024-09-17T03:49:37.590630+0000","last_became_active":"2024-09-17T03:49:36.399727+0000","last_became_peered":"2024-09-17T03:49:36.399727+0000","last_unstale":"2024-09-17T03:49:37.590630+0000","last_undegraded":"2024-09-17T03:49:37.590630+0000","last_fullsized":"2024-09-17T03:49:37.590630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590102+0000","last_change":"2024-09-17T03:49:35.386738+0000","last_active":"2024-09-17T03:49:37.590102+0000","last_peered":"2024-09-17T03:49:37.590102+0000","last_clean":"2024-09-17T03:49:37.590102+0000","last_became_active":"2024-09-17T03:49:35.386382+0000","last_became_peered":"2024-09-17T03:49:35.386382+0000","last_unstale":"2024-09-17T03:49:37.590102+0000","last_undegraded":"2024-09-17T03:49:37.590102+0000","last_fullsized":"2024-09-17T03:49:37.590102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:57.569543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":667808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":76984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":76858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":77018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":667864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:49:45.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.339+0000 7f607ba00640 1 -- 172.21.15.129:0/2979800282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 msgr2=0x7f6074074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.339+0000 7f607ba00640 1 --2- 172.21.15.129:0/2979800282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074074640 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6060005fd0 tx=0x7f606002a580 comp rx=0 tx=0).stop 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.339+0000 7f607ba00640 1 -- 172.21.15.129:0/2979800282 shutdown_connections 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.339+0000 7f607ba00640 1 --2- 172.21.15.129:0/2979800282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074074640 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.339+0000 7f607ba00640 1 -- 172.21.15.129:0/2979800282 >> 172.21.15.129:0/2979800282 conn(0x7f60740b1cb0 msgr2=0x7f60740b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.340+0000 7f607ba00640 1 -- 172.21.15.129:0/2979800282 shutdown_connections 2024-09-17T03:49:45.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.340+0000 7f607ba00640 1 -- 172.21.15.129:0/2979800282 wait complete. 2024-09-17T03:49:45.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.342+0000 7f607ba00640 1 Processor -- start 2024-09-17T03:49:45.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.342+0000 7f607ba00640 1 -- start start 2024-09-17T03:49:45.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 0x7f607415a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 0x7f607414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60740b3c40 con 0x7f6074070030 2024-09-17T03:49:45.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f60740b3ac0 con 0x7f6074070440 2024-09-17T03:49:45.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f607ba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f60740b3dc0 con 0x7f6074073170 2024-09-17T03:49:45.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.343+0000 7f6079000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43408/0 (socket says 172.21.15.129:43408) 2024-09-17T03:49:45.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 -- 172.21.15.129:0/3963830321 learned_addr learned my addr 172.21.15.129:0/3963830321 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079a00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 0x7f607414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6073e00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 0x7f607415a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 -- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 msgr2=0x7f607414b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 0x7f607414b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 -- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 msgr2=0x7f607415a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 0x7f607415a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 -- 172.21.15.129:0/3963830321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607409c450 con 0x7f6074070030 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079a00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 0x7f607414b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6073e00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 0x7f607415a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6079000640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f606800c3c0 tx=0x7f606800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068017070 con 0x7f6074070030 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60680035f0 con 0x7f6074070030 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607409c730 con 0x7f6074070030 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.344+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6068003a40 con 0x7f6074070030 2024-09-17T03:49:45.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.345+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607409ccf0 con 0x7f6074070030 2024-09-17T03:49:45.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.346+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6074076820 con 0x7f6074070030 2024-09-17T03:49:45.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.347+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f606801a070 con 0x7f6074070030 2024-09-17T03:49:45.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.348+0000 7f6070c00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 0x7f605407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.348+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6068099f90 con 0x7f6074070030 2024-09-17T03:49:45.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.348+0000 7f6073e00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 0x7f605407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.349+0000 7f6073e00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 0x7f605407b0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6060033000 tx=0x7f6060004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:45.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.352+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6068062450 con 0x7f6074070030 2024-09-17T03:49:45.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:49:45.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:49:45.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.538+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60740c3860 con 0x7f6054078b90 2024-09-17T03:49:45.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.543+0000 7f6070c00640 1 -- 172.21.15.129:0/3963830321 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f60740c3860 con 0x7f6054078b90 2024-09-17T03:49:45.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:45.546 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.547+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 msgr2=0x7f605407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.547+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 0x7f605407b0a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6060033000 tx=0x7f6060004730 comp rx=0 tx=0).stop 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.547+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 msgr2=0x7f6074159640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.547+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f606800c3c0 tx=0x7f606800c880 comp rx=0 tx=0).stop 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 shutdown_connections 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6054078b90 0x7f605407b0a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6074073170 0x7f607414b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6074070440 0x7f607415a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 --2- 172.21.15.129:0/3963830321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6074070030 0x7f6074159640 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 >> 172.21.15.129:0/3963830321 conn(0x7f60740b1cb0 msgr2=0x7f60740aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:45.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 shutdown_connections 2024-09-17T03:49:45.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.548+0000 7f607ba00640 1 -- 172.21.15.129:0/3963830321 wait complete. 2024-09-17T03:49:45.568 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-17T03:49:44.294980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223380,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751340,"statfs":{"total":773094113280,"available":772865372160,"internally_reserved":0,"allocated":3022848,"data_stored":1800420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589196+0000","last_change":"2024-09-17T03:49:36.406020+0000","last_active":"2024-09-17T03:49:37.589196+0000","last_peered":"2024-09-17T03:49:37.589196+0000","last_clean":"2024-09-17T03:49:37.589196+0000","last_became_active":"2024-09-17T03:49:36.405649+0000","last_became_peered":"2024-09-17T03:49:36.405649+0000","last_unstale":"2024-09-17T03:49:37.589196+0000","last_undegraded":"2024-09-17T03:49:37.589196+0000","last_fullsized":"2024-09-17T03:49:37.589196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377354+0000","last_change":"2024-09-17T03:49:36.400461+0000","last_active":"2024-09-17T03:49:37.377354+0000","last_peered":"2024-09-17T03:49:37.377354+0000","last_clean":"2024-09-17T03:49:37.377354+0000","last_became_active":"2024-09-17T03:49:36.398210+0000","last_became_peered":"2024-09-17T03:49:36.398210+0000","last_unstale":"2024-09-17T03:49:37.377354+0000","last_undegraded":"2024-09-17T03:49:37.377354+0000","last_fullsized":"2024-09-17T03:49:37.377354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589718+0000","last_change":"2024-09-17T03:49:36.397732+0000","last_active":"2024-09-17T03:49:37.589718+0000","last_peered":"2024-09-17T03:49:37.589718+0000","last_clean":"2024-09-17T03:49:37.589718+0000","last_became_active":"2024-09-17T03:49:36.397407+0000","last_became_peered":"2024-09-17T03:49:36.397407+0000","last_unstale":"2024-09-17T03:49:37.589718+0000","last_undegraded":"2024-09-17T03:49:37.589718+0000","last_fullsized":"2024-09-17T03:49:37.589718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379346+0000","last_change":"2024-09-17T03:49:36.410331+0000","last_active":"2024-09-17T03:49:37.379346+0000","last_peered":"2024-09-17T03:49:37.379346+0000","last_clean":"2024-09-17T03:49:37.379346+0000","last_became_active":"2024-09-17T03:49:36.409931+0000","last_became_peered":"2024-09-17T03:49:36.409931+0000","last_unstale":"2024-09-17T03:49:37.379346+0000","last_undegraded":"2024-09-17T03:49:37.379346+0000","last_fullsized":"2024-09-17T03:49:37.379346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589345+0000","last_change":"2024-09-17T03:49:36.415397+0000","last_active":"2024-09-17T03:49:37.589345+0000","last_peered":"2024-09-17T03:49:37.589345+0000","last_clean":"2024-09-17T03:49:37.589345+0000","last_became_active":"2024-09-17T03:49:36.415075+0000","last_became_peered":"2024-09-17T03:49:36.415075+0000","last_unstale":"2024-09-17T03:49:37.589345+0000","last_undegraded":"2024-09-17T03:49:37.589345+0000","last_fullsized":"2024-09-17T03:49:37.589345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:08.390183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379503+0000","last_change":"2024-09-17T03:49:36.405656+0000","last_active":"2024-09-17T03:49:37.379503+0000","last_peered":"2024-09-17T03:49:37.379503+0000","last_clean":"2024-09-17T03:49:37.379503+0000","last_became_active":"2024-09-17T03:49:36.404978+0000","last_became_peered":"2024-09-17T03:49:36.404978+0000","last_unstale":"2024-09-17T03:49:37.379503+0000","last_undegraded":"2024-09-17T03:49:37.379503+0000","last_fullsized":"2024-09-17T03:49:37.379503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588643+0000","last_change":"2024-09-17T03:49:36.402961+0000","last_active":"2024-09-17T03:49:37.588643+0000","last_peered":"2024-09-17T03:49:37.588643+0000","last_clean":"2024-09-17T03:49:37.588643+0000","last_became_active":"2024-09-17T03:49:36.402413+0000","last_became_peered":"2024-09-17T03:49:36.402413+0000","last_unstale":"2024-09-17T03:49:37.588643+0000","last_undegraded":"2024-09-17T03:49:37.588643+0000","last_fullsized":"2024-09-17T03:49:37.588643+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:09.378052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588967+0000","last_change":"2024-09-17T03:49:36.409564+0000","last_active":"2024-09-17T03:49:37.588967+0000","last_peered":"2024-09-17T03:49:37.588967+0000","last_clean":"2024-09-17T03:49:37.588967+0000","last_became_active":"2024-09-17T03:49:36.409150+0000","last_became_peered":"2024-09-17T03:49:36.409150+0000","last_unstale":"2024-09-17T03:49:37.588967+0000","last_undegraded":"2024-09-17T03:49:37.588967+0000","last_fullsized":"2024-09-17T03:49:37.588967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:09.331331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589233+0000","last_change":"2024-09-17T03:49:36.416599+0000","last_active":"2024-09-17T03:49:37.589233+0000","last_peered":"2024-09-17T03:49:37.589233+0000","last_clean":"2024-09-17T03:49:37.589233+0000","last_became_active":"2024-09-17T03:49:36.416100+0000","last_became_peered":"2024-09-17T03:49:36.416100+0000","last_unstale":"2024-09-17T03:49:37.589233+0000","last_undegraded":"2024-09-17T03:49:37.589233+0000","last_fullsized":"2024-09-17T03:49:37.589233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420661+0000","last_change":"2024-09-17T03:49:36.412006+0000","last_active":"2024-09-17T03:49:37.420661+0000","last_peered":"2024-09-17T03:49:37.420661+0000","last_clean":"2024-09-17T03:49:37.420661+0000","last_became_active":"2024-09-17T03:49:36.411697+0000","last_became_peered":"2024-09-17T03:49:36.411697+0000","last_unstale":"2024-09-17T03:49:37.420661+0000","last_undegraded":"2024-09-17T03:49:37.420661+0000","last_fullsized":"2024-09-17T03:49:37.420661+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420825+0000","last_change":"2024-09-17T03:49:36.415050+0000","last_active":"2024-09-17T03:49:37.420825+0000","last_peered":"2024-09-17T03:49:37.420825+0000","last_clean":"2024-09-17T03:49:37.420825+0000","last_became_active":"2024-09-17T03:49:36.414808+0000","last_became_peered":"2024-09-17T03:49:36.414808+0000","last_unstale":"2024-09-17T03:49:37.420825+0000","last_undegraded":"2024-09-17T03:49:37.420825+0000","last_fullsized":"2024-09-17T03:49:37.420825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379278+0000","last_change":"2024-09-17T03:49:36.406397+0000","last_active":"2024-09-17T03:49:37.379278+0000","last_peered":"2024-09-17T03:49:37.379278+0000","last_clean":"2024-09-17T03:49:37.379278+0000","last_became_active":"2024-09-17T03:49:36.405791+0000","last_became_peered":"2024-09-17T03:49:36.405791+0000","last_unstale":"2024-09-17T03:49:37.379278+0000","last_undegraded":"2024-09-17T03:49:37.379278+0000","last_fullsized":"2024-09-17T03:49:37.379278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:05.635714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589437+0000","last_change":"2024-09-17T03:49:36.393913+0000","last_active":"2024-09-17T03:49:37.589437+0000","last_peered":"2024-09-17T03:49:37.589437+0000","last_clean":"2024-09-17T03:49:37.589437+0000","last_became_active":"2024-09-17T03:49:36.393589+0000","last_became_peered":"2024-09-17T03:49:36.393589+0000","last_unstale":"2024-09-17T03:49:37.589437+0000","last_undegraded":"2024-09-17T03:49:37.589437+0000","last_fullsized":"2024-09-17T03:49:37.589437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:12.309621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588596+0000","last_change":"2024-09-17T03:49:36.401985+0000","last_active":"2024-09-17T03:49:37.588596+0000","last_peered":"2024-09-17T03:49:37.588596+0000","last_clean":"2024-09-17T03:49:37.588596+0000","last_became_active":"2024-09-17T03:49:36.401682+0000","last_became_peered":"2024-09-17T03:49:36.401682+0000","last_unstale":"2024-09-17T03:49:37.588596+0000","last_undegraded":"2024-09-17T03:49:37.588596+0000","last_fullsized":"2024-09-17T03:49:37.588596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:46.086705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377130+0000","last_change":"2024-09-17T03:49:36.389920+0000","last_active":"2024-09-17T03:49:37.377130+0000","last_peered":"2024-09-17T03:49:37.377130+0000","last_clean":"2024-09-17T03:49:37.377130+0000","last_became_active":"2024-09-17T03:49:36.389645+0000","last_became_peered":"2024-09-17T03:49:36.389645+0000","last_unstale":"2024-09-17T03:49:37.377130+0000","last_undegraded":"2024-09-17T03:49:37.377130+0000","last_fullsized":"2024-09-17T03:49:37.377130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:20.042459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378538+0000","last_change":"2024-09-17T03:49:36.403578+0000","last_active":"2024-09-17T03:49:37.378538+0000","last_peered":"2024-09-17T03:49:37.378538+0000","last_clean":"2024-09-17T03:49:37.378538+0000","last_became_active":"2024-09-17T03:49:36.403353+0000","last_became_peered":"2024-09-17T03:49:36.403353+0000","last_unstale":"2024-09-17T03:49:37.378538+0000","last_undegraded":"2024-09-17T03:49:37.378538+0000","last_fullsized":"2024-09-17T03:49:37.378538+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380751+0000","last_change":"2024-09-17T03:49:36.406004+0000","last_active":"2024-09-17T03:49:37.380751+0000","last_peered":"2024-09-17T03:49:37.380751+0000","last_clean":"2024-09-17T03:49:37.380751+0000","last_became_active":"2024-09-17T03:49:36.403931+0000","last_became_peered":"2024-09-17T03:49:36.403931+0000","last_unstale":"2024-09-17T03:49:37.380751+0000","last_undegraded":"2024-09-17T03:49:37.380751+0000","last_fullsized":"2024-09-17T03:49:37.380751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:39.435141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588086+0000","last_change":"2024-09-17T03:49:36.396161+0000","last_active":"2024-09-17T03:49:37.588086+0000","last_peered":"2024-09-17T03:49:37.588086+0000","last_clean":"2024-09-17T03:49:37.588086+0000","last_became_active":"2024-09-17T03:49:36.395717+0000","last_became_peered":"2024-09-17T03:49:36.395717+0000","last_unstale":"2024-09-17T03:49:37.588086+0000","last_undegraded":"2024-09-17T03:49:37.588086+0000","last_fullsized":"2024-09-17T03:49:37.588086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:42.475745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589346+0000","last_change":"2024-09-17T03:49:36.410754+0000","last_active":"2024-09-17T03:49:37.589346+0000","last_peered":"2024-09-17T03:49:37.589346+0000","last_clean":"2024-09-17T03:49:37.589346+0000","last_became_active":"2024-09-17T03:49:36.404715+0000","last_became_peered":"2024-09-17T03:49:36.404715+0000","last_unstale":"2024-09-17T03:49:37.589346+0000","last_undegraded":"2024-09-17T03:49:37.589346+0000","last_fullsized":"2024-09-17T03:49:37.589346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:51.120762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420579+0000","last_change":"2024-09-17T03:49:36.409394+0000","last_active":"2024-09-17T03:49:37.420579+0000","last_peered":"2024-09-17T03:49:37.420579+0000","last_clean":"2024-09-17T03:49:37.420579+0000","last_became_active":"2024-09-17T03:49:36.408771+0000","last_became_peered":"2024-09-17T03:49:36.408771+0000","last_unstale":"2024-09-17T03:49:37.420579+0000","last_undegraded":"2024-09-17T03:49:37.420579+0000","last_fullsized":"2024-09-17T03:49:37.420579+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:50.351452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588435+0000","last_change":"2024-09-17T03:49:36.422246+0000","last_active":"2024-09-17T03:49:37.588435+0000","last_peered":"2024-09-17T03:49:37.588435+0000","last_clean":"2024-09-17T03:49:37.588435+0000","last_became_active":"2024-09-17T03:49:36.421985+0000","last_became_peered":"2024-09-17T03:49:36.421985+0000","last_unstale":"2024-09-17T03:49:37.588435+0000","last_undegraded":"2024-09-17T03:49:37.588435+0000","last_fullsized":"2024-09-17T03:49:37.588435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:02:13.173976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379785+0000","last_change":"2024-09-17T03:49:36.405326+0000","last_active":"2024-09-17T03:49:37.379785+0000","last_peered":"2024-09-17T03:49:37.379785+0000","last_clean":"2024-09-17T03:49:37.379785+0000","last_became_active":"2024-09-17T03:49:36.404541+0000","last_became_peered":"2024-09-17T03:49:36.404541+0000","last_unstale":"2024-09-17T03:49:37.379785+0000","last_undegraded":"2024-09-17T03:49:37.379785+0000","last_fullsized":"2024-09-17T03:49:37.379785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589367+0000","last_change":"2024-09-17T03:49:36.406574+0000","last_active":"2024-09-17T03:49:37.589367+0000","last_peered":"2024-09-17T03:49:37.589367+0000","last_clean":"2024-09-17T03:49:37.589367+0000","last_became_active":"2024-09-17T03:49:36.406162+0000","last_became_peered":"2024-09-17T03:49:36.406162+0000","last_unstale":"2024-09-17T03:49:37.589367+0000","last_undegraded":"2024-09-17T03:49:37.589367+0000","last_fullsized":"2024-09-17T03:49:37.589367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378837+0000","last_change":"2024-09-17T03:49:36.394666+0000","last_active":"2024-09-17T03:49:37.378837+0000","last_peered":"2024-09-17T03:49:37.378837+0000","last_clean":"2024-09-17T03:49:37.378837+0000","last_became_active":"2024-09-17T03:49:36.394259+0000","last_became_peered":"2024-09-17T03:49:36.394259+0000","last_unstale":"2024-09-17T03:49:37.378837+0000","last_undegraded":"2024-09-17T03:49:37.378837+0000","last_fullsized":"2024-09-17T03:49:37.378837+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590715+0000","last_change":"2024-09-17T03:49:36.410498+0000","last_active":"2024-09-17T03:49:37.590715+0000","last_peered":"2024-09-17T03:49:37.590715+0000","last_clean":"2024-09-17T03:49:37.590715+0000","last_became_active":"2024-09-17T03:49:36.410228+0000","last_became_peered":"2024-09-17T03:49:36.410228+0000","last_unstale":"2024-09-17T03:49:37.590715+0000","last_undegraded":"2024-09-17T03:49:37.590715+0000","last_fullsized":"2024-09-17T03:49:37.590715+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420810+0000","last_change":"2024-09-17T03:49:36.410333+0000","last_active":"2024-09-17T03:49:37.420810+0000","last_peered":"2024-09-17T03:49:37.420810+0000","last_clean":"2024-09-17T03:49:37.420810+0000","last_became_active":"2024-09-17T03:49:36.405435+0000","last_became_peered":"2024-09-17T03:49:36.405435+0000","last_unstale":"2024-09-17T03:49:37.420810+0000","last_undegraded":"2024-09-17T03:49:37.420810+0000","last_fullsized":"2024-09-17T03:49:37.420810+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:21.315923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377561+0000","last_change":"2024-09-17T03:49:36.394974+0000","last_active":"2024-09-17T03:49:37.377561+0000","last_peered":"2024-09-17T03:49:37.377561+0000","last_clean":"2024-09-17T03:49:37.377561+0000","last_became_active":"2024-09-17T03:49:36.393921+0000","last_became_peered":"2024-09-17T03:49:36.393921+0000","last_unstale":"2024-09-17T03:49:37.377561+0000","last_undegraded":"2024-09-17T03:49:37.377561+0000","last_fullsized":"2024-09-17T03:49:37.377561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590561+0000","last_change":"2024-09-17T03:49:36.406505+0000","last_active":"2024-09-17T03:49:37.590561+0000","last_peered":"2024-09-17T03:49:37.590561+0000","last_clean":"2024-09-17T03:49:37.590561+0000","last_became_active":"2024-09-17T03:49:36.406154+0000","last_became_peered":"2024-09-17T03:49:36.406154+0000","last_unstale":"2024-09-17T03:49:37.590561+0000","last_undegraded":"2024-09-17T03:49:37.590561+0000","last_fullsized":"2024-09-17T03:49:37.590561+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:27.228941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588653+0000","last_change":"2024-09-17T03:49:36.400848+0000","last_active":"2024-09-17T03:49:37.588653+0000","last_peered":"2024-09-17T03:49:37.588653+0000","last_clean":"2024-09-17T03:49:37.588653+0000","last_became_active":"2024-09-17T03:49:36.399273+0000","last_became_peered":"2024-09-17T03:49:36.399273+0000","last_unstale":"2024-09-17T03:49:37.588653+0000","last_undegraded":"2024-09-17T03:49:37.588653+0000","last_fullsized":"2024-09-17T03:49:37.588653+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379849+0000","last_change":"2024-09-17T03:49:36.409112+0000","last_active":"2024-09-17T03:49:37.379849+0000","last_peered":"2024-09-17T03:49:37.379849+0000","last_clean":"2024-09-17T03:49:37.379849+0000","last_became_active":"2024-09-17T03:49:36.408890+0000","last_became_peered":"2024-09-17T03:49:36.408890+0000","last_unstale":"2024-09-17T03:49:37.379849+0000","last_undegraded":"2024-09-17T03:49:37.379849+0000","last_fullsized":"2024-09-17T03:49:37.379849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588078+0000","last_change":"2024-09-17T03:49:36.408545+0000","last_active":"2024-09-17T03:49:37.588078+0000","last_peered":"2024-09-17T03:49:37.588078+0000","last_clean":"2024-09-17T03:49:37.588078+0000","last_became_active":"2024-09-17T03:49:36.408285+0000","last_became_peered":"2024-09-17T03:49:36.408285+0000","last_unstale":"2024-09-17T03:49:37.588078+0000","last_undegraded":"2024-09-17T03:49:37.588078+0000","last_fullsized":"2024-09-17T03:49:37.588078+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420257+0000","last_change":"2024-09-17T03:49:36.401317+0000","last_active":"2024-09-17T03:49:37.420257+0000","last_peered":"2024-09-17T03:49:37.420257+0000","last_clean":"2024-09-17T03:49:37.420257+0000","last_became_active":"2024-09-17T03:49:36.400953+0000","last_became_peered":"2024-09-17T03:49:36.400953+0000","last_unstale":"2024-09-17T03:49:37.420257+0000","last_undegraded":"2024-09-17T03:49:37.420257+0000","last_fullsized":"2024-09-17T03:49:37.420257+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:43:45.195448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378778+0000","last_change":"2024-09-17T03:49:36.400419+0000","last_active":"2024-09-17T03:49:37.378778+0000","last_peered":"2024-09-17T03:49:37.378778+0000","last_clean":"2024-09-17T03:49:37.378778+0000","last_became_active":"2024-09-17T03:49:36.399021+0000","last_became_peered":"2024-09-17T03:49:36.399021+0000","last_unstale":"2024-09-17T03:49:37.378778+0000","last_undegraded":"2024-09-17T03:49:37.378778+0000","last_fullsized":"2024-09-17T03:49:37.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:50.779802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590324+0000","last_change":"2024-09-17T03:49:35.392535+0000","last_active":"2024-09-17T03:49:37.590324+0000","last_peered":"2024-09-17T03:49:37.590324+0000","last_clean":"2024-09-17T03:49:37.590324+0000","last_became_active":"2024-09-17T03:49:35.392219+0000","last_became_peered":"2024-09-17T03:49:35.392219+0000","last_unstale":"2024-09-17T03:49:37.590324+0000","last_undegraded":"2024-09-17T03:49:37.590324+0000","last_fullsized":"2024-09-17T03:49:37.590324+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.459490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588861+0000","last_change":"2024-09-17T03:49:36.400903+0000","last_active":"2024-09-17T03:49:37.588861+0000","last_peered":"2024-09-17T03:49:37.588861+0000","last_clean":"2024-09-17T03:49:37.588861+0000","last_became_active":"2024-09-17T03:49:36.400109+0000","last_became_peered":"2024-09-17T03:49:36.400109+0000","last_unstale":"2024-09-17T03:49:37.588861+0000","last_undegraded":"2024-09-17T03:49:37.588861+0000","last_fullsized":"2024-09-17T03:49:37.588861+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:34.979447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420155+0000","last_change":"2024-09-17T03:49:35.393512+0000","last_active":"2024-09-17T03:49:37.420155+0000","last_peered":"2024-09-17T03:49:37.420155+0000","last_clean":"2024-09-17T03:49:37.420155+0000","last_became_active":"2024-09-17T03:49:35.388644+0000","last_became_peered":"2024-09-17T03:49:35.388644+0000","last_unstale":"2024-09-17T03:49:37.420155+0000","last_undegraded":"2024-09-17T03:49:37.420155+0000","last_fullsized":"2024-09-17T03:49:37.420155+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:02:36.651524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589477+0000","last_change":"2024-09-17T03:49:35.391294+0000","last_active":"2024-09-17T03:49:37.589477+0000","last_peered":"2024-09-17T03:49:37.589477+0000","last_clean":"2024-09-17T03:49:37.589477+0000","last_became_active":"2024-09-17T03:49:35.390970+0000","last_became_peered":"2024-09-17T03:49:35.390970+0000","last_unstale":"2024-09-17T03:49:37.589477+0000","last_undegraded":"2024-09-17T03:49:37.589477+0000","last_fullsized":"2024-09-17T03:49:37.589477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:45.058598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589296+0000","last_change":"2024-09-17T03:49:36.404871+0000","last_active":"2024-09-17T03:49:37.589296+0000","last_peered":"2024-09-17T03:49:37.589296+0000","last_clean":"2024-09-17T03:49:37.589296+0000","last_became_active":"2024-09-17T03:49:36.404452+0000","last_became_peered":"2024-09-17T03:49:36.404452+0000","last_unstale":"2024-09-17T03:49:37.589296+0000","last_undegraded":"2024-09-17T03:49:37.589296+0000","last_fullsized":"2024-09-17T03:49:37.589296+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:04.311139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378971+0000","last_change":"2024-09-17T03:49:35.388702+0000","last_active":"2024-09-17T03:49:37.378971+0000","last_peered":"2024-09-17T03:49:37.378971+0000","last_clean":"2024-09-17T03:49:37.378971+0000","last_became_active":"2024-09-17T03:49:35.388337+0000","last_became_peered":"2024-09-17T03:49:35.388337+0000","last_unstale":"2024-09-17T03:49:37.378971+0000","last_undegraded":"2024-09-17T03:49:37.378971+0000","last_fullsized":"2024-09-17T03:49:37.378971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377377+0000","last_change":"2024-09-17T03:49:36.394908+0000","last_active":"2024-09-17T03:49:37.377377+0000","last_peered":"2024-09-17T03:49:37.377377+0000","last_clean":"2024-09-17T03:49:37.377377+0000","last_became_active":"2024-09-17T03:49:36.394518+0000","last_became_peered":"2024-09-17T03:49:36.394518+0000","last_unstale":"2024-09-17T03:49:37.377377+0000","last_undegraded":"2024-09-17T03:49:37.377377+0000","last_fullsized":"2024-09-17T03:49:37.377377+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420535+0000","last_change":"2024-09-17T03:49:35.389956+0000","last_active":"2024-09-17T03:49:37.420535+0000","last_peered":"2024-09-17T03:49:37.420535+0000","last_clean":"2024-09-17T03:49:37.420535+0000","last_became_active":"2024-09-17T03:49:35.389588+0000","last_became_peered":"2024-09-17T03:49:35.389588+0000","last_unstale":"2024-09-17T03:49:37.420535+0000","last_undegraded":"2024-09-17T03:49:37.420535+0000","last_fullsized":"2024-09-17T03:49:37.420535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:14.510096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380718+0000","last_change":"2024-09-17T03:49:36.405737+0000","last_active":"2024-09-17T03:49:37.380718+0000","last_peered":"2024-09-17T03:49:37.380718+0000","last_clean":"2024-09-17T03:49:37.380718+0000","last_became_active":"2024-09-17T03:49:36.401772+0000","last_became_peered":"2024-09-17T03:49:36.401772+0000","last_unstale":"2024-09-17T03:49:37.380718+0000","last_undegraded":"2024-09-17T03:49:37.380718+0000","last_fullsized":"2024-09-17T03:49:37.380718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:58.250852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380418+0000","last_change":"2024-09-17T03:49:35.395265+0000","last_active":"2024-09-17T03:49:37.380418+0000","last_peered":"2024-09-17T03:49:37.380418+0000","last_clean":"2024-09-17T03:49:37.380418+0000","last_became_active":"2024-09-17T03:49:35.394870+0000","last_became_peered":"2024-09-17T03:49:35.394870+0000","last_unstale":"2024-09-17T03:49:37.380418+0000","last_undegraded":"2024-09-17T03:49:37.380418+0000","last_fullsized":"2024-09-17T03:49:37.380418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590151+0000","last_change":"2024-09-17T03:49:36.407432+0000","last_active":"2024-09-17T03:49:37.590151+0000","last_peered":"2024-09-17T03:49:37.590151+0000","last_clean":"2024-09-17T03:49:37.590151+0000","last_became_active":"2024-09-17T03:49:36.407158+0000","last_became_peered":"2024-09-17T03:49:36.407158+0000","last_unstale":"2024-09-17T03:49:37.590151+0000","last_undegraded":"2024-09-17T03:49:37.590151+0000","last_fullsized":"2024-09-17T03:49:37.590151+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:15.562928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377583+0000","last_change":"2024-09-17T03:49:35.390102+0000","last_active":"2024-09-17T03:49:37.377583+0000","last_peered":"2024-09-17T03:49:37.377583+0000","last_clean":"2024-09-17T03:49:37.377583+0000","last_became_active":"2024-09-17T03:49:35.389698+0000","last_became_peered":"2024-09-17T03:49:35.389698+0000","last_unstale":"2024-09-17T03:49:37.377583+0000","last_undegraded":"2024-09-17T03:49:37.377583+0000","last_fullsized":"2024-09-17T03:49:37.377583+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379524+0000","last_change":"2024-09-17T03:49:36.406033+0000","last_active":"2024-09-17T03:49:37.379524+0000","last_peered":"2024-09-17T03:49:37.379524+0000","last_clean":"2024-09-17T03:49:37.379524+0000","last_became_active":"2024-09-17T03:49:36.405391+0000","last_became_peered":"2024-09-17T03:49:36.405391+0000","last_unstale":"2024-09-17T03:49:37.379524+0000","last_undegraded":"2024-09-17T03:49:37.379524+0000","last_fullsized":"2024-09-17T03:49:37.379524+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:35.463684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420327+0000","last_change":"2024-09-17T03:49:35.392778+0000","last_active":"2024-09-17T03:49:37.420327+0000","last_peered":"2024-09-17T03:49:37.420327+0000","last_clean":"2024-09-17T03:49:37.420327+0000","last_became_active":"2024-09-17T03:49:35.391530+0000","last_became_peered":"2024-09-17T03:49:35.391530+0000","last_unstale":"2024-09-17T03:49:37.420327+0000","last_undegraded":"2024-09-17T03:49:37.420327+0000","last_fullsized":"2024-09-17T03:49:37.420327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:09.206550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588902+0000","last_change":"2024-09-17T03:49:36.404199+0000","last_active":"2024-09-17T03:49:37.588902+0000","last_peered":"2024-09-17T03:49:37.588902+0000","last_clean":"2024-09-17T03:49:37.588902+0000","last_became_active":"2024-09-17T03:49:36.398455+0000","last_became_peered":"2024-09-17T03:49:36.398455+0000","last_unstale":"2024-09-17T03:49:37.588902+0000","last_undegraded":"2024-09-17T03:49:37.588902+0000","last_fullsized":"2024-09-17T03:49:37.588902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:49.127551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588954+0000","last_change":"2024-09-17T03:49:35.387607+0000","last_active":"2024-09-17T03:49:37.588954+0000","last_peered":"2024-09-17T03:49:37.588954+0000","last_clean":"2024-09-17T03:49:37.588954+0000","last_became_active":"2024-09-17T03:49:35.387303+0000","last_became_peered":"2024-09-17T03:49:35.387303+0000","last_unstale":"2024-09-17T03:49:37.588954+0000","last_undegraded":"2024-09-17T03:49:37.588954+0000","last_fullsized":"2024-09-17T03:49:37.588954+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:33.456173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588883+0000","last_change":"2024-09-17T03:49:36.413174+0000","last_active":"2024-09-17T03:49:37.588883+0000","last_peered":"2024-09-17T03:49:37.588883+0000","last_clean":"2024-09-17T03:49:37.588883+0000","last_became_active":"2024-09-17T03:49:36.412943+0000","last_became_peered":"2024-09-17T03:49:36.412943+0000","last_unstale":"2024-09-17T03:49:37.588883+0000","last_undegraded":"2024-09-17T03:49:37.588883+0000","last_fullsized":"2024-09-17T03:49:37.588883+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379966+0000","last_change":"2024-09-17T03:49:35.388456+0000","last_active":"2024-09-17T03:49:37.379966+0000","last_peered":"2024-09-17T03:49:37.379966+0000","last_clean":"2024-09-17T03:49:37.379966+0000","last_became_active":"2024-09-17T03:49:35.388083+0000","last_became_peered":"2024-09-17T03:49:35.388083+0000","last_unstale":"2024-09-17T03:49:37.379966+0000","last_undegraded":"2024-09-17T03:49:37.379966+0000","last_fullsized":"2024-09-17T03:49:37.379966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588927+0000","last_change":"2024-09-17T03:49:36.414542+0000","last_active":"2024-09-17T03:49:37.588927+0000","last_peered":"2024-09-17T03:49:37.588927+0000","last_clean":"2024-09-17T03:49:37.588927+0000","last_became_active":"2024-09-17T03:49:36.414224+0000","last_became_peered":"2024-09-17T03:49:36.414224+0000","last_unstale":"2024-09-17T03:49:37.588927+0000","last_undegraded":"2024-09-17T03:49:37.588927+0000","last_fullsized":"2024-09-17T03:49:37.588927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420647+0000","last_change":"2024-09-17T03:49:35.388635+0000","last_active":"2024-09-17T03:49:37.420647+0000","last_peered":"2024-09-17T03:49:37.420647+0000","last_clean":"2024-09-17T03:49:37.420647+0000","last_became_active":"2024-09-17T03:49:35.388278+0000","last_became_peered":"2024-09-17T03:49:35.388278+0000","last_unstale":"2024-09-17T03:49:37.420647+0000","last_undegraded":"2024-09-17T03:49:37.420647+0000","last_fullsized":"2024-09-17T03:49:37.420647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:17.648757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420254+0000","last_change":"2024-09-17T03:49:36.401308+0000","last_active":"2024-09-17T03:49:37.420254+0000","last_peered":"2024-09-17T03:49:37.420254+0000","last_clean":"2024-09-17T03:49:37.420254+0000","last_became_active":"2024-09-17T03:49:36.400942+0000","last_became_peered":"2024-09-17T03:49:36.400942+0000","last_unstale":"2024-09-17T03:49:37.420254+0000","last_undegraded":"2024-09-17T03:49:37.420254+0000","last_fullsized":"2024-09-17T03:49:37.420254+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:45.575607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":23,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588398+0000","last_change":"2024-09-17T03:49:35.372728+0000","last_active":"2024-09-17T03:49:37.588398+0000","last_peered":"2024-09-17T03:49:37.588398+0000","last_clean":"2024-09-17T03:49:37.588398+0000","last_became_active":"2024-09-17T03:49:35.372413+0000","last_became_peered":"2024-09-17T03:49:35.372413+0000","last_unstale":"2024-09-17T03:49:37.588398+0000","last_undegraded":"2024-09-17T03:49:37.588398+0000","last_fullsized":"2024-09-17T03:49:37.588398+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588789+0000","last_change":"2024-09-17T03:49:36.413936+0000","last_active":"2024-09-17T03:49:37.588789+0000","last_peered":"2024-09-17T03:49:37.588789+0000","last_clean":"2024-09-17T03:49:37.588789+0000","last_became_active":"2024-09-17T03:49:36.402860+0000","last_became_peered":"2024-09-17T03:49:36.402860+0000","last_unstale":"2024-09-17T03:49:37.588789+0000","last_undegraded":"2024-09-17T03:49:37.588789+0000","last_fullsized":"2024-09-17T03:49:37.588789+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380504+0000","last_change":"2024-09-17T03:49:35.385498+0000","last_active":"2024-09-17T03:49:37.380504+0000","last_peered":"2024-09-17T03:49:37.380504+0000","last_clean":"2024-09-17T03:49:37.380504+0000","last_became_active":"2024-09-17T03:49:35.385212+0000","last_became_peered":"2024-09-17T03:49:35.385212+0000","last_unstale":"2024-09-17T03:49:37.380504+0000","last_undegraded":"2024-09-17T03:49:37.380504+0000","last_fullsized":"2024-09-17T03:49:37.380504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:45.909870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589167+0000","last_change":"2024-09-17T03:49:36.413538+0000","last_active":"2024-09-17T03:49:37.589167+0000","last_peered":"2024-09-17T03:49:37.589167+0000","last_clean":"2024-09-17T03:49:37.589167+0000","last_became_active":"2024-09-17T03:49:36.413197+0000","last_became_peered":"2024-09-17T03:49:36.413197+0000","last_unstale":"2024-09-17T03:49:37.589167+0000","last_undegraded":"2024-09-17T03:49:37.589167+0000","last_fullsized":"2024-09-17T03:49:37.589167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588219+0000","last_change":"2024-09-17T03:49:35.384632+0000","last_active":"2024-09-17T03:49:37.588219+0000","last_peered":"2024-09-17T03:49:37.588219+0000","last_clean":"2024-09-17T03:49:37.588219+0000","last_became_active":"2024-09-17T03:49:35.384263+0000","last_became_peered":"2024-09-17T03:49:35.384263+0000","last_unstale":"2024-09-17T03:49:37.588219+0000","last_undegraded":"2024-09-17T03:49:37.588219+0000","last_fullsized":"2024-09-17T03:49:37.588219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589537+0000","last_change":"2024-09-17T03:49:36.415593+0000","last_active":"2024-09-17T03:49:37.589537+0000","last_peered":"2024-09-17T03:49:37.589537+0000","last_clean":"2024-09-17T03:49:37.589537+0000","last_became_active":"2024-09-17T03:49:36.411554+0000","last_became_peered":"2024-09-17T03:49:36.411554+0000","last_unstale":"2024-09-17T03:49:37.589537+0000","last_undegraded":"2024-09-17T03:49:37.589537+0000","last_fullsized":"2024-09-17T03:49:37.589537+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:13.481011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377691+0000","last_change":"2024-09-17T03:49:35.387003+0000","last_active":"2024-09-17T03:49:37.377691+0000","last_peered":"2024-09-17T03:49:37.377691+0000","last_clean":"2024-09-17T03:49:37.377691+0000","last_became_active":"2024-09-17T03:49:35.386654+0000","last_became_peered":"2024-09-17T03:49:35.386654+0000","last_unstale":"2024-09-17T03:49:37.377691+0000","last_undegraded":"2024-09-17T03:49:37.377691+0000","last_fullsized":"2024-09-17T03:49:37.377691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:23.004272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379918+0000","last_change":"2024-09-17T03:49:36.395286+0000","last_active":"2024-09-17T03:49:37.379918+0000","last_peered":"2024-09-17T03:49:37.379918+0000","last_clean":"2024-09-17T03:49:37.379918+0000","last_became_active":"2024-09-17T03:49:36.394025+0000","last_became_peered":"2024-09-17T03:49:36.394025+0000","last_unstale":"2024-09-17T03:49:37.379918+0000","last_undegraded":"2024-09-17T03:49:37.379918+0000","last_fullsized":"2024-09-17T03:49:37.379918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.848493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588329+0000","last_change":"2024-09-17T03:49:35.389511+0000","last_active":"2024-09-17T03:49:37.588329+0000","last_peered":"2024-09-17T03:49:37.588329+0000","last_clean":"2024-09-17T03:49:37.588329+0000","last_became_active":"2024-09-17T03:49:35.389061+0000","last_became_peered":"2024-09-17T03:49:35.389061+0000","last_unstale":"2024-09-17T03:49:37.588329+0000","last_undegraded":"2024-09-17T03:49:37.588329+0000","last_fullsized":"2024-09-17T03:49:37.588329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589110+0000","last_change":"2024-09-17T03:49:36.418896+0000","last_active":"2024-09-17T03:49:37.589110+0000","last_peered":"2024-09-17T03:49:37.589110+0000","last_clean":"2024-09-17T03:49:37.589110+0000","last_became_active":"2024-09-17T03:49:36.418567+0000","last_became_peered":"2024-09-17T03:49:36.418567+0000","last_unstale":"2024-09-17T03:49:37.589110+0000","last_undegraded":"2024-09-17T03:49:37.589110+0000","last_fullsized":"2024-09-17T03:49:37.589110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":111,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588519+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:49:37.588519+0000","last_peered":"2024-09-17T03:49:37.588519+0000","last_clean":"2024-09-17T03:49:37.588519+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:49:37.588519+0000","last_undegraded":"2024-09-17T03:49:37.588519+0000","last_fullsized":"2024-09-17T03:49:37.588519+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378615+0000","last_change":"2024-09-17T03:49:35.379012+0000","last_active":"2024-09-17T03:49:37.378615+0000","last_peered":"2024-09-17T03:49:37.378615+0000","last_clean":"2024-09-17T03:49:37.378615+0000","last_became_active":"2024-09-17T03:49:35.378349+0000","last_became_peered":"2024-09-17T03:49:35.378349+0000","last_unstale":"2024-09-17T03:49:37.378615+0000","last_undegraded":"2024-09-17T03:49:37.378615+0000","last_fullsized":"2024-09-17T03:49:37.378615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589522+0000","last_change":"2024-09-17T03:49:36.408105+0000","last_active":"2024-09-17T03:49:37.589522+0000","last_peered":"2024-09-17T03:49:37.589522+0000","last_clean":"2024-09-17T03:49:37.589522+0000","last_became_active":"2024-09-17T03:49:36.407814+0000","last_became_peered":"2024-09-17T03:49:36.407814+0000","last_unstale":"2024-09-17T03:49:37.589522+0000","last_undegraded":"2024-09-17T03:49:37.589522+0000","last_fullsized":"2024-09-17T03:49:37.589522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:20.252642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588438+0000","last_change":"2024-09-17T03:49:35.399235+0000","last_active":"2024-09-17T03:49:37.588438+0000","last_peered":"2024-09-17T03:49:37.588438+0000","last_clean":"2024-09-17T03:49:37.588438+0000","last_became_active":"2024-09-17T03:49:35.398860+0000","last_became_peered":"2024-09-17T03:49:35.398860+0000","last_unstale":"2024-09-17T03:49:37.588438+0000","last_undegraded":"2024-09-17T03:49:37.588438+0000","last_fullsized":"2024-09-17T03:49:37.588438+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:21:34.455037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588130+0000","last_change":"2024-09-17T03:49:36.403778+0000","last_active":"2024-09-17T03:49:37.588130+0000","last_peered":"2024-09-17T03:49:37.588130+0000","last_clean":"2024-09-17T03:49:37.588130+0000","last_became_active":"2024-09-17T03:49:36.403408+0000","last_became_peered":"2024-09-17T03:49:36.403408+0000","last_unstale":"2024-09-17T03:49:37.588130+0000","last_undegraded":"2024-09-17T03:49:37.588130+0000","last_fullsized":"2024-09-17T03:49:37.588130+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:23:19.846081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.587984+0000","last_change":"2024-09-17T03:49:35.384616+0000","last_active":"2024-09-17T03:49:37.587984+0000","last_peered":"2024-09-17T03:49:37.587984+0000","last_clean":"2024-09-17T03:49:37.587984+0000","last_became_active":"2024-09-17T03:49:35.384264+0000","last_became_peered":"2024-09-17T03:49:35.384264+0000","last_unstale":"2024-09-17T03:49:37.587984+0000","last_undegraded":"2024-09-17T03:49:37.587984+0000","last_fullsized":"2024-09-17T03:49:37.587984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377132+0000","last_change":"2024-09-17T03:49:36.381928+0000","last_active":"2024-09-17T03:49:37.377132+0000","last_peered":"2024-09-17T03:49:37.377132+0000","last_clean":"2024-09-17T03:49:37.377132+0000","last_became_active":"2024-09-17T03:49:36.381629+0000","last_became_peered":"2024-09-17T03:49:36.381629+0000","last_unstale":"2024-09-17T03:49:37.377132+0000","last_undegraded":"2024-09-17T03:49:37.377132+0000","last_fullsized":"2024-09-17T03:49:37.377132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379997+0000","last_change":"2024-09-17T03:49:35.396687+0000","last_active":"2024-09-17T03:49:37.379997+0000","last_peered":"2024-09-17T03:49:37.379997+0000","last_clean":"2024-09-17T03:49:37.379997+0000","last_became_active":"2024-09-17T03:49:35.390958+0000","last_became_peered":"2024-09-17T03:49:35.390958+0000","last_unstale":"2024-09-17T03:49:37.379997+0000","last_undegraded":"2024-09-17T03:49:37.379997+0000","last_fullsized":"2024-09-17T03:49:37.379997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.378943+0000","last_change":"2024-09-17T03:49:36.406614+0000","last_active":"2024-09-17T03:49:37.378943+0000","last_peered":"2024-09-17T03:49:37.378943+0000","last_clean":"2024-09-17T03:49:37.378943+0000","last_became_active":"2024-09-17T03:49:36.403577+0000","last_became_peered":"2024-09-17T03:49:36.403577+0000","last_unstale":"2024-09-17T03:49:37.378943+0000","last_undegraded":"2024-09-17T03:49:37.378943+0000","last_fullsized":"2024-09-17T03:49:37.378943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:45.693650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377460+0000","last_change":"2024-09-17T03:49:35.385825+0000","last_active":"2024-09-17T03:49:37.377460+0000","last_peered":"2024-09-17T03:49:37.377460+0000","last_clean":"2024-09-17T03:49:37.377460+0000","last_became_active":"2024-09-17T03:49:35.385489+0000","last_became_peered":"2024-09-17T03:49:35.385489+0000","last_unstale":"2024-09-17T03:49:37.377460+0000","last_undegraded":"2024-09-17T03:49:37.377460+0000","last_fullsized":"2024-09-17T03:49:37.377460+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:49.771155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380572+0000","last_change":"2024-09-17T03:49:36.413827+0000","last_active":"2024-09-17T03:49:37.380572+0000","last_peered":"2024-09-17T03:49:37.380572+0000","last_clean":"2024-09-17T03:49:37.380572+0000","last_became_active":"2024-09-17T03:49:36.413432+0000","last_became_peered":"2024-09-17T03:49:36.413432+0000","last_unstale":"2024-09-17T03:49:37.380572+0000","last_undegraded":"2024-09-17T03:49:37.380572+0000","last_fullsized":"2024-09-17T03:49:37.380572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:32.459167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589281+0000","last_change":"2024-09-17T03:49:35.396951+0000","last_active":"2024-09-17T03:49:37.589281+0000","last_peered":"2024-09-17T03:49:37.589281+0000","last_clean":"2024-09-17T03:49:37.589281+0000","last_became_active":"2024-09-17T03:49:35.396554+0000","last_became_peered":"2024-09-17T03:49:35.396554+0000","last_unstale":"2024-09-17T03:49:37.589281+0000","last_undegraded":"2024-09-17T03:49:37.589281+0000","last_fullsized":"2024-09-17T03:49:37.589281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:23.665085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377339+0000","last_change":"2024-09-17T03:49:36.387372+0000","last_active":"2024-09-17T03:49:37.377339+0000","last_peered":"2024-09-17T03:49:37.377339+0000","last_clean":"2024-09-17T03:49:37.377339+0000","last_became_active":"2024-09-17T03:49:36.384715+0000","last_became_peered":"2024-09-17T03:49:36.384715+0000","last_unstale":"2024-09-17T03:49:37.377339+0000","last_undegraded":"2024-09-17T03:49:37.377339+0000","last_fullsized":"2024-09-17T03:49:37.377339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:23.705043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.419437+0000","last_change":"2024-09-17T03:49:35.393853+0000","last_active":"2024-09-17T03:49:37.419437+0000","last_peered":"2024-09-17T03:49:37.419437+0000","last_clean":"2024-09-17T03:49:37.419437+0000","last_became_active":"2024-09-17T03:49:35.393551+0000","last_became_peered":"2024-09-17T03:49:35.393551+0000","last_unstale":"2024-09-17T03:49:37.419437+0000","last_undegraded":"2024-09-17T03:49:37.419437+0000","last_fullsized":"2024-09-17T03:49:37.419437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590925+0000","last_change":"2024-09-17T03:49:36.412628+0000","last_active":"2024-09-17T03:49:37.590925+0000","last_peered":"2024-09-17T03:49:37.590925+0000","last_clean":"2024-09-17T03:49:37.590925+0000","last_became_active":"2024-09-17T03:49:36.412320+0000","last_became_peered":"2024-09-17T03:49:36.412320+0000","last_unstale":"2024-09-17T03:49:37.590925+0000","last_undegraded":"2024-09-17T03:49:37.590925+0000","last_fullsized":"2024-09-17T03:49:37.590925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:34.456176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420603+0000","last_change":"2024-09-17T03:49:35.389093+0000","last_active":"2024-09-17T03:49:37.420603+0000","last_peered":"2024-09-17T03:49:37.420603+0000","last_clean":"2024-09-17T03:49:37.420603+0000","last_became_active":"2024-09-17T03:49:35.388625+0000","last_became_peered":"2024-09-17T03:49:35.388625+0000","last_unstale":"2024-09-17T03:49:37.420603+0000","last_undegraded":"2024-09-17T03:49:37.420603+0000","last_fullsized":"2024-09-17T03:49:37.420603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.987962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589830+0000","last_change":"2024-09-17T03:49:36.397328+0000","last_active":"2024-09-17T03:49:37.589830+0000","last_peered":"2024-09-17T03:49:37.589830+0000","last_clean":"2024-09-17T03:49:37.589830+0000","last_became_active":"2024-09-17T03:49:36.396910+0000","last_became_peered":"2024-09-17T03:49:36.396910+0000","last_unstale":"2024-09-17T03:49:37.589830+0000","last_undegraded":"2024-09-17T03:49:37.589830+0000","last_fullsized":"2024-09-17T03:49:37.589830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:46.456824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588441+0000","last_change":"2024-09-17T03:49:35.391714+0000","last_active":"2024-09-17T03:49:37.588441+0000","last_peered":"2024-09-17T03:49:37.588441+0000","last_clean":"2024-09-17T03:49:37.588441+0000","last_became_active":"2024-09-17T03:49:35.391374+0000","last_became_peered":"2024-09-17T03:49:35.391374+0000","last_unstale":"2024-09-17T03:49:37.588441+0000","last_undegraded":"2024-09-17T03:49:37.588441+0000","last_fullsized":"2024-09-17T03:49:37.588441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:04.627067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380255+0000","last_change":"2024-09-17T03:49:36.395381+0000","last_active":"2024-09-17T03:49:37.380255+0000","last_peered":"2024-09-17T03:49:37.380255+0000","last_clean":"2024-09-17T03:49:37.380255+0000","last_became_active":"2024-09-17T03:49:36.394430+0000","last_became_peered":"2024-09-17T03:49:36.394430+0000","last_unstale":"2024-09-17T03:49:37.380255+0000","last_undegraded":"2024-09-17T03:49:37.380255+0000","last_fullsized":"2024-09-17T03:49:37.380255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:25.487173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377304+0000","last_change":"2024-09-17T03:49:35.386828+0000","last_active":"2024-09-17T03:49:37.377304+0000","last_peered":"2024-09-17T03:49:37.377304+0000","last_clean":"2024-09-17T03:49:37.377304+0000","last_became_active":"2024-09-17T03:49:35.385425+0000","last_became_peered":"2024-09-17T03:49:35.385425+0000","last_unstale":"2024-09-17T03:49:37.377304+0000","last_undegraded":"2024-09-17T03:49:37.377304+0000","last_fullsized":"2024-09-17T03:49:37.377304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:44.685003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589280+0000","last_change":"2024-09-17T03:49:36.396082+0000","last_active":"2024-09-17T03:49:37.589280+0000","last_peered":"2024-09-17T03:49:37.589280+0000","last_clean":"2024-09-17T03:49:37.589280+0000","last_became_active":"2024-09-17T03:49:36.395733+0000","last_became_peered":"2024-09-17T03:49:36.395733+0000","last_unstale":"2024-09-17T03:49:37.589280+0000","last_undegraded":"2024-09-17T03:49:37.589280+0000","last_fullsized":"2024-09-17T03:49:37.589280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:03.159958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.377167+0000","last_change":"2024-09-17T03:49:35.383855+0000","last_active":"2024-09-17T03:49:37.377167+0000","last_peered":"2024-09-17T03:49:37.377167+0000","last_clean":"2024-09-17T03:49:37.377167+0000","last_became_active":"2024-09-17T03:49:35.383513+0000","last_became_peered":"2024-09-17T03:49:35.383513+0000","last_unstale":"2024-09-17T03:49:37.377167+0000","last_undegraded":"2024-09-17T03:49:37.377167+0000","last_fullsized":"2024-09-17T03:49:37.377167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:42.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589610+0000","last_change":"2024-09-17T03:49:36.401383+0000","last_active":"2024-09-17T03:49:37.589610+0000","last_peered":"2024-09-17T03:49:37.589610+0000","last_clean":"2024-09-17T03:49:37.589610+0000","last_became_active":"2024-09-17T03:49:36.401112+0000","last_became_peered":"2024-09-17T03:49:36.401112+0000","last_unstale":"2024-09-17T03:49:37.589610+0000","last_undegraded":"2024-09-17T03:49:37.589610+0000","last_fullsized":"2024-09-17T03:49:37.589610+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:32.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589087+0000","last_change":"2024-09-17T03:49:35.405334+0000","last_active":"2024-09-17T03:49:37.589087+0000","last_peered":"2024-09-17T03:49:37.589087+0000","last_clean":"2024-09-17T03:49:37.589087+0000","last_became_active":"2024-09-17T03:49:35.404955+0000","last_became_peered":"2024-09-17T03:49:35.404955+0000","last_unstale":"2024-09-17T03:49:37.589087+0000","last_undegraded":"2024-09-17T03:49:37.589087+0000","last_fullsized":"2024-09-17T03:49:37.589087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:24.478526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.420485+0000","last_change":"2024-09-17T03:49:36.412896+0000","last_active":"2024-09-17T03:49:37.420485+0000","last_peered":"2024-09-17T03:49:37.420485+0000","last_clean":"2024-09-17T03:49:37.420485+0000","last_became_active":"2024-09-17T03:49:36.406254+0000","last_became_peered":"2024-09-17T03:49:36.406254+0000","last_unstale":"2024-09-17T03:49:37.420485+0000","last_undegraded":"2024-09-17T03:49:37.420485+0000","last_fullsized":"2024-09-17T03:49:37.420485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:38.205718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.380554+0000","last_change":"2024-09-17T03:49:36.396792+0000","last_active":"2024-09-17T03:49:37.380554+0000","last_peered":"2024-09-17T03:49:37.380554+0000","last_clean":"2024-09-17T03:49:37.380554+0000","last_became_active":"2024-09-17T03:49:36.393651+0000","last_became_peered":"2024-09-17T03:49:36.393651+0000","last_unstale":"2024-09-17T03:49:37.380554+0000","last_undegraded":"2024-09-17T03:49:37.380554+0000","last_fullsized":"2024-09-17T03:49:37.380554+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:33.190170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.588722+0000","last_change":"2024-09-17T03:49:35.398891+0000","last_active":"2024-09-17T03:49:37.588722+0000","last_peered":"2024-09-17T03:49:37.588722+0000","last_clean":"2024-09-17T03:49:37.588722+0000","last_became_active":"2024-09-17T03:49:35.398494+0000","last_became_peered":"2024-09-17T03:49:35.398494+0000","last_unstale":"2024-09-17T03:49:37.588722+0000","last_undegraded":"2024-09-17T03:49:37.588722+0000","last_fullsized":"2024-09-17T03:49:37.588722+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:25:34.347073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.379744+0000","last_change":"2024-09-17T03:49:36.394357+0000","last_active":"2024-09-17T03:49:37.379744+0000","last_peered":"2024-09-17T03:49:37.379744+0000","last_clean":"2024-09-17T03:49:37.379744+0000","last_became_active":"2024-09-17T03:49:36.393549+0000","last_became_peered":"2024-09-17T03:49:36.393549+0000","last_unstale":"2024-09-17T03:49:37.379744+0000","last_undegraded":"2024-09-17T03:49:37.379744+0000","last_fullsized":"2024-09-17T03:49:37.379744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:03.411164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589049+0000","last_change":"2024-09-17T03:49:35.398839+0000","last_active":"2024-09-17T03:49:37.589049+0000","last_peered":"2024-09-17T03:49:37.589049+0000","last_clean":"2024-09-17T03:49:37.589049+0000","last_became_active":"2024-09-17T03:49:35.398355+0000","last_became_peered":"2024-09-17T03:49:35.398355+0000","last_unstale":"2024-09-17T03:49:37.589049+0000","last_undegraded":"2024-09-17T03:49:37.589049+0000","last_fullsized":"2024-09-17T03:49:37.589049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:51.292462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.589985+0000","last_change":"2024-09-17T03:49:36.409308+0000","last_active":"2024-09-17T03:49:37.589985+0000","last_peered":"2024-09-17T03:49:37.589985+0000","last_clean":"2024-09-17T03:49:37.589985+0000","last_became_active":"2024-09-17T03:49:36.408373+0000","last_became_peered":"2024-09-17T03:49:36.408373+0000","last_unstale":"2024-09-17T03:49:37.589985+0000","last_undegraded":"2024-09-17T03:49:37.589985+0000","last_fullsized":"2024-09-17T03:49:37.589985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:16:00.612951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590572+0000","last_change":"2024-09-17T03:49:35.381739+0000","last_active":"2024-09-17T03:49:37.590572+0000","last_peered":"2024-09-17T03:49:37.590572+0000","last_clean":"2024-09-17T03:49:37.590572+0000","last_became_active":"2024-09-17T03:49:35.381185+0000","last_became_peered":"2024-09-17T03:49:35.381185+0000","last_unstale":"2024-09-17T03:49:37.590572+0000","last_undegraded":"2024-09-17T03:49:37.590572+0000","last_fullsized":"2024-09-17T03:49:37.590572+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:10.483256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590630+0000","last_change":"2024-09-17T03:49:36.401989+0000","last_active":"2024-09-17T03:49:37.590630+0000","last_peered":"2024-09-17T03:49:37.590630+0000","last_clean":"2024-09-17T03:49:37.590630+0000","last_became_active":"2024-09-17T03:49:36.399727+0000","last_became_peered":"2024-09-17T03:49:36.399727+0000","last_unstale":"2024-09-17T03:49:37.590630+0000","last_undegraded":"2024-09-17T03:49:37.590630+0000","last_fullsized":"2024-09-17T03:49:37.590630+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:35.356097+0000","last_clean_scrub_stamp":"2024-09-17T03:49:35.356097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:19.246996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-09-17T03:49:37.590102+0000","last_change":"2024-09-17T03:49:35.386738+0000","last_active":"2024-09-17T03:49:37.590102+0000","last_peered":"2024-09-17T03:49:37.590102+0000","last_clean":"2024-09-17T03:49:37.590102+0000","last_became_active":"2024-09-17T03:49:35.386382+0000","last_became_peered":"2024-09-17T03:49:35.386382+0000","last_unstale":"2024-09-17T03:49:37.590102+0000","last_undegraded":"2024-09-17T03:49:37.590102+0000","last_fullsized":"2024-09-17T03:49:37.590102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:34.347450+0000","last_clean_scrub_stamp":"2024-09-17T03:49:34.347450+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:57.569543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":819200,"data_stored":667808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":225280,"data_stored":76948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":76984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":221184,"data_stored":76858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":229376,"data_stored":77018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":233472,"data_stored":78470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":667864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:49:45.572 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:49:45.572 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T03:49:45.572 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-17T03:49:45.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-17T03:49:45.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 -- 172.21.15.129:0/552470880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa415ca50 msgr2=0x7fcaa40cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/552470880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa415ca50 0x7fcaa40cc830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fca94005fd0 tx=0x7fca9402a540 comp rx=0 tx=0).stop 2024-09-17T03:49:45.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 -- 172.21.15.129:0/552470880 shutdown_connections 2024-09-17T03:49:45.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/552470880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf540 0x7fcaa406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/552470880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa415ca50 0x7fcaa40cc830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 -- 172.21.15.129:0/552470880 >> 172.21.15.129:0/552470880 conn(0x7fcaa406d410 msgr2=0x7fcaa40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:45.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 -- 172.21.15.129:0/552470880 shutdown_connections 2024-09-17T03:49:45.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.829+0000 7fcaa8e00640 1 -- 172.21.15.129:0/552470880 wait complete. 2024-09-17T03:49:45.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.831+0000 7fcaa8e00640 1 Processor -- start 2024-09-17T03:49:45.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.831+0000 7fcaa8e00640 1 -- start start 2024-09-17T03:49:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf950 0x7fcaa4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 0x7fcaa41f8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa415de10 con 0x7fcaa40cf540 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa415df90 con 0x7fcaa415ca50 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa415e110 con 0x7fcaa40cf950 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43432/0 (socket says 172.21.15.129:43432) 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 -- 172.21.15.129:0/3714546102 learned_addr learned my addr 172.21.15.129:0/3714546102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 0x7fcaa41f8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa2a00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf950 0x7fcaa4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 -- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf950 msgr2=0x7fcaa4152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf950 0x7fcaa4152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 -- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 msgr2=0x7fcaa41f8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 0x7fcaa41f8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3400640 1 -- 172.21.15.129:0/3714546102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa4154a20 con 0x7fcaa40cf540 2024-09-17T03:49:45.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.833+0000 7fcaa3e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 0x7fcaa41f8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fcaa3400640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fca94006010 tx=0x7fca940039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9403e030 con 0x7fcaa40cf540 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca940047a0 con 0x7fcaa40cf540 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa4154d00 con 0x7fcaa40cf540 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94035040 con 0x7fcaa40cf540 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.834+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa41552c0 con 0x7fcaa40cf540 2024-09-17T03:49:45.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.835+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa4155500 con 0x7fcaa40cf540 2024-09-17T03:49:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.836+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fca94041030 con 0x7fcaa40cf540 2024-09-17T03:49:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.837+0000 7fca9b400640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 0x7fca8407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.837+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fca940bb670 con 0x7fcaa40cf540 2024-09-17T03:49:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.838+0000 7fcaa2a00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 0x7fca8407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:45.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.838+0000 7fcaa2a00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 0x7fca8407b0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca90005fd0 tx=0x7fca90005da0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:45.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:45.841+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcaa4155500 con 0x7fcaa40cf540 2024-09-17T03:49:46.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.060+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fcaa415f380 con 0x7fcaa40cf540 2024-09-17T03:49:46.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.061+0000 7fca9b400640 1 -- 172.21.15.129:0/3714546102 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fcaa415f380 con 0x7fcaa40cf540 2024-09-17T03:49:46.060 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:46.060 INFO:teuthology.orchestra.run.smithi129.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T03:49:46.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 msgr2=0x7fca8407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 0x7fca8407b0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca90005fd0 tx=0x7fca90005da0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 msgr2=0x7fcaa40cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fca94006010 tx=0x7fca940039e0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 shutdown_connections 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078bc0 0x7fca8407b0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa415ca50 0x7fcaa41f8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa40cf950 0x7fcaa4152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 --2- 172.21.15.129:0/3714546102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa40cf540 0x7fcaa40cba20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 >> 172.21.15.129:0/3714546102 conn(0x7fcaa406d410 msgr2=0x7fcaa40b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 shutdown_connections 2024-09-17T03:49:46.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.063+0000 7fcaa8e00640 1 -- 172.21.15.129:0/3714546102 wait complete. 2024-09-17T03:49:46.084 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-17T03:49:46.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:46.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 -- 172.21.15.129:0/928669240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac0b9bd0 msgr2=0x7f6cac0b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 --2- 172.21.15.129:0/928669240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0b71a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009830 tx=0x7f6c9c02a410 comp rx=0 tx=0).stop 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 -- 172.21.15.129:0/928669240 shutdown_connections 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 --2- 172.21.15.129:0/928669240 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 0x7f6cac0b93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 --2- 172.21.15.129:0/928669240 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0b71a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 -- 172.21.15.129:0/928669240 >> 172.21.15.129:0/928669240 conn(0x7f6cac09d6c0 msgr2=0x7f6cac0d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 -- 172.21.15.129:0/928669240 shutdown_connections 2024-09-17T03:49:46.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.304+0000 7f6cabe00640 1 -- 172.21.15.129:0/928669240 wait complete. 2024-09-17T03:49:46.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.306+0000 7f6cabe00640 1 Processor -- start 2024-09-17T03:49:46.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.306+0000 7f6cabe00640 1 -- start start 2024-09-17T03:49:46.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 0x7f6cac1f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 0x7f6cac0cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac0b5360 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6cac0b51e0 con 0x7f6cac162440 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cabe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6cac0b54e0 con 0x7f6cac0b8fd0 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6caa000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6caa000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43436/0 (socket says 172.21.15.129:43436) 2024-09-17T03:49:46.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6caa000640 1 -- 172.21.15.129:0/3178162958 learned_addr learned my addr 172.21.15.129:0/3178162958 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6caaa00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 0x7f6cac1f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.308+0000 7f6cab400640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 0x7f6cac0cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 -- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 msgr2=0x7f6cac1f6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 0x7f6cac1f6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 -- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 msgr2=0x7f6cac0cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 0x7f6cac0cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 -- 172.21.15.129:0/3178162958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cac0d0b40 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6cab400640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 0x7f6cac0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:46.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6caa000640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c460 tx=0x7f6c9800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:46.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c980035f0 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c980037b0 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cac0d0de0 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003c00 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.309+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cac0cb3a0 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.310+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cac08cb00 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.311+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c9801b400 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.312+0000 7f6ca3400640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 0x7f6c8c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.312+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6c9809a900 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.313+0000 7f6caaa00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 0x7f6c8c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.313+0000 7f6caaa00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 0x7f6c8c07b0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:46.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.317+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6cac08cb00 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.537+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cac0b5a20 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.538+0000 7f6ca3400640 1 -- 172.21.15.129:0/3178162958 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7f6c9809a030 con 0x7f6cac0b9bd0 2024-09-17T03:49:46.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:46.540 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:49:46.542 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:49:46.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 msgr2=0x7f6c8c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 0x7f6c8c07b0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c034000 tx=0x7f6c9c004730 comp rx=0 tx=0).stop 2024-09-17T03:49:46.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 msgr2=0x7f6cac0c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800c460 tx=0x7f6c9800c920 comp rx=0 tx=0).stop 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 shutdown_connections 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c8c078bc0 0x7f6c8c07b0d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6cac162440 0x7f6cac0cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.540+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6cac0b9bd0 0x7f6cac0c7e80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.541+0000 7f6cabe00640 1 --2- 172.21.15.129:0/3178162958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6cac0b8fd0 0x7f6cac1f6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.541+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 >> 172.21.15.129:0/3178162958 conn(0x7f6cac09d6c0 msgr2=0x7f6cac06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.541+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 shutdown_connections 2024-09-17T03:49:46.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.541+0000 7f6cabe00640 1 -- 172.21.15.129:0/3178162958 wait complete. 2024-09-17T03:49:46.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 -- 172.21.15.129:0/1957368623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c0b4d30 msgr2=0x7fab8c0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 --2- 172.21.15.129:0/1957368623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c0b4d30 0x7fab8c0ccb10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fab8c072dc0 tx=0x7fab7402a560 comp rx=0 tx=0).stop 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 -- 172.21.15.129:0/1957368623 shutdown_connections 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 --2- 172.21.15.129:0/1957368623 >> v2:172.21.15.181:3300/0 conn(0x7fab8c06f2a0 0x7fab8c0cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 --2- 172.21.15.129:0/1957368623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c0b4d30 0x7fab8c0ccb10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 --2- 172.21.15.129:0/1957368623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c06ee90 0x7fab8c0b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.779+0000 7fab92800640 1 -- 172.21.15.129:0/1957368623 >> 172.21.15.129:0/1957368623 conn(0x7fab8c06d7d0 msgr2=0x7fab8c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:46.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.780+0000 7fab92800640 1 -- 172.21.15.129:0/1957368623 shutdown_connections 2024-09-17T03:49:46.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.780+0000 7fab92800640 1 -- 172.21.15.129:0/1957368623 wait complete. 2024-09-17T03:49:46.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.782+0000 7fab92800640 1 Processor -- start 2024-09-17T03:49:46.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.782+0000 7fab92800640 1 -- start start 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 0x7fab8c14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c0b4d30 0x7fab8c1f9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c15f7a0 con 0x7fab8c06ee90 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c15f620 con 0x7fab8c06f2a0 2024-09-17T03:49:46.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.783+0000 7fab92800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fab8c15f920 con 0x7fab8c0b4d30 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43450/0 (socket says 172.21.15.129:43450) 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 -- 172.21.15.129:0/3227091523 learned_addr learned my addr 172.21.15.129:0/3227091523 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8be00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c0b4d30 0x7fab8c1f9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab82a00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 0x7fab8c14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 -- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c0b4d30 msgr2=0x7fab8c1f9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c0b4d30 0x7fab8c1f9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 -- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 msgr2=0x7fab8c14c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 0x7fab8c14c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:46.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 -- 172.21.15.129:0/3227091523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8c09d190 con 0x7fab8c06ee90 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab82a00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 0x7fab8c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab8b400640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00eb30 tx=0x7fab7c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.784+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c004b20 con 0x7fab8c06ee90 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.785+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fab7c004ce0 con 0x7fab8c06ee90 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.785+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8c09d470 con 0x7fab8c06ee90 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.785+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7c005280 con 0x7fab8c06ee90 2024-09-17T03:49:46.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.785+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8c09da30 con 0x7fab8c06ee90 2024-09-17T03:49:46.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.786+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab8c15fe40 con 0x7fab8c06ee90 2024-09-17T03:49:46.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.787+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fab7c005420 con 0x7fab8c06ee90 2024-09-17T03:49:46.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.788+0000 7fab83e00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 0x7fab6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:46.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.788+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fab7c09b9f0 con 0x7fab8c06ee90 2024-09-17T03:49:46.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.788+0000 7fab82a00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 0x7fab6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:46.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.789+0000 7fab82a00640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 0x7fab6807b0a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fab74033000 tx=0x7fab74004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:46.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:46.792+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fab7c063ec0 con 0x7fab8c06ee90 2024-09-17T03:49:47.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.010+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab8c0750f0 con 0x7fab8c06ee90 2024-09-17T03:49:47.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.011+0000 7fab83e00640 1 -- 172.21.15.129:0/3227091523 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7fab7c0637c0 con 0x7fab8c06ee90 2024-09-17T03:49:47.010 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:47.010 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:49:47.010 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:49:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 msgr2=0x7fab6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 0x7fab6807b0a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fab74033000 tx=0x7fab74004730 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 msgr2=0x7fab8c09aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00eb30 tx=0x7fab7c00c2d0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 shutdown_connections 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fab68078b90 0x7fab6807b0a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fab8c0b4d30 0x7fab8c1f9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fab8c06f2a0 0x7fab8c14c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 --2- 172.21.15.129:0/3227091523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fab8c06ee90 0x7fab8c09aa30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 >> 172.21.15.129:0/3227091523 conn(0x7fab8c06d7d0 msgr2=0x7fab8c1614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 shutdown_connections 2024-09-17T03:49:47.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.013+0000 7fab92800640 1 -- 172.21.15.129:0/3227091523 wait complete. 2024-09-17T03:49:47.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 -- 172.21.15.129:0/2285790294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c006f3b0 msgr2=0x7f20c00cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 --2- 172.21.15.129:0/2285790294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c006f3b0 0x7f20c00cf080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f20b8005fd0 tx=0x7f20b802aa90 comp rx=0 tx=0).stop 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 -- 172.21.15.129:0/2285790294 shutdown_connections 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 --2- 172.21.15.129:0/2285790294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c006f3b0 0x7f20c00cf080 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 --2- 172.21.15.129:0/2285790294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c0151640 0x7f20c006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 -- 172.21.15.129:0/2285790294 >> 172.21.15.129:0/2285790294 conn(0x7f20c006d410 msgr2=0x7f20c00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.260+0000 7f20c5c00640 1 -- 172.21.15.129:0/2285790294 shutdown_connections 2024-09-17T03:49:47.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.261+0000 7f20c5c00640 1 -- 172.21.15.129:0/2285790294 wait complete. 2024-09-17T03:49:47.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.263+0000 7f20c5c00640 1 Processor -- start 2024-09-17T03:49:47.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.263+0000 7f20c5c00640 1 -- start start 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c006f3b0 0x7f20c01458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c0151640 0x7f20c0159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20c00cf740 con 0x7f20c015b8f0 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20c00cf5c0 con 0x7f20c0151640 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.264+0000 7f20c5c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f20c00cf8c0 con 0x7f20c006f3b0 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43462/0 (socket says 172.21.15.129:43462) 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 -- 172.21.15.129:0/3862711269 learned_addr learned my addr 172.21.15.129:0/3862711269 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c006f3b0 0x7f20c01458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20b7e00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c0151640 0x7f20c0159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 -- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c006f3b0 msgr2=0x7f20c01458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c006f3b0 0x7f20c01458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 -- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c0151640 msgr2=0x7f20c0159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c0151640 0x7f20c0159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 -- 172.21.15.129:0/3862711269 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c0150830 con 0x7f20c015b8f0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.265+0000 7f20c5200640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009bf0 tx=0x7f20b8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.266+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8044070 con 0x7f20c015b8f0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.266+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20b8004250 con 0x7f20c015b8f0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.266+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8004660 con 0x7f20c015b8f0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.266+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c0157f20 con 0x7f20c015b8f0 2024-09-17T03:49:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.266+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c01584d0 con 0x7f20c015b8f0 2024-09-17T03:49:47.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.267+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c00cf080 con 0x7f20c015b8f0 2024-09-17T03:49:47.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.268+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f20b8047070 con 0x7f20c015b8f0 2024-09-17T03:49:47.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.269+0000 7f20bcc00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 0x7f209c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.269+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f20b80bbd00 con 0x7f20c015b8f0 2024-09-17T03:49:47.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.270+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 0x7f209c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.270+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 0x7f209c07b0a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000f60 tx=0x7f20b0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:47.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.273+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f20b8032030 con 0x7f20c015b8f0 2024-09-17T03:49:47.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.458+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20c00b17b0 con 0x7f20c015b8f0 2024-09-17T03:49:47.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.459+0000 7f20bcc00640 1 -- 172.21.15.129:0/3862711269 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7f20b8083ae0 con 0x7f20c015b8f0 2024-09-17T03:49:47.458 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:47.458 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:49:47.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 msgr2=0x7f209c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 0x7f209c07b0a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000f60 tx=0x7f20b0009040 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 msgr2=0x7f20c00c6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009bf0 tx=0x7f20b8003ce0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 shutdown_connections 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078b90 0x7f209c07b0a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c015b8f0 0x7f20c00c6e50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c0151640 0x7f20c0159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 --2- 172.21.15.129:0/3862711269 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20c006f3b0 0x7f20c01458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 >> 172.21.15.129:0/3862711269 conn(0x7f20c006d410 msgr2=0x7f20c0154590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 shutdown_connections 2024-09-17T03:49:47.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.461+0000 7f20c5c00640 1 -- 172.21.15.129:0/3862711269 wait complete. 2024-09-17T03:49:47.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:49:47.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 -- 172.21.15.129:0/1923124753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa79c0041c0 msgr2=0x7fa79c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/1923124753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa79c0041c0 0x7fa79c004800 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0006f60 tx=0x7fa79c005830 comp rx=0 tx=0).stop 2024-09-17T03:49:47.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7b9e00640 1 -- 172.21.15.129:0/1923124753 reap_dead start 2024-09-17T03:49:47.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 -- 172.21.15.129:0/1923124753 shutdown_connections 2024-09-17T03:49:47.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 -- 172.21.15.129:0/1923124753 >> 172.21.15.129:0/1923124753 conn(0x7fa7b40b1cb0 msgr2=0x7fa7b40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:47.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 -- 172.21.15.129:0/1923124753 shutdown_connections 2024-09-17T03:49:47.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.703+0000 7fa7bbe00640 1 -- 172.21.15.129:0/1923124753 wait complete. 2024-09-17T03:49:47.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.705+0000 7fa7bbe00640 1 Processor -- start 2024-09-17T03:49:47.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.706+0000 7fa7bbe00640 1 -- start start 2024-09-17T03:49:47.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 0x7fa7b414bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7b40d06a0 0x7fa7b40d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b41655c0 con 0x7fa7b414b040 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4165440 con 0x7fa79c0041c0 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7bbe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b4165740 con 0x7fa7b40d06a0 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43472/0 (socket says 172.21.15.129:43472) 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.707+0000 7fa7b9e00640 1 -- 172.21.15.129:0/167883315 learned_addr learned my addr 172.21.15.129:0/167883315 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 -- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7b40d06a0 msgr2=0x7fa7b40d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7b40d06a0 0x7fa7b40d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 -- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 msgr2=0x7fa7b414bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9400640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 0x7fa7b414bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 0x7fa7b414bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 -- 172.21.15.129:0/167883315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b40d0ff0 con 0x7fa7b414b040 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9400640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 0x7fa7b414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b9e00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40cd410 tx=0x7fa7ac00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac004b20 con 0x7fa7b414b040 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7ac004cc0 con 0x7fa7b414b040 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b414b990 con 0x7fa7b414b040 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7ac003bd0 con 0x7fa7b414b040 2024-09-17T03:49:47.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.708+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b409d3e0 con 0x7fa7b414b040 2024-09-17T03:49:47.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.710+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b4074640 con 0x7fa7b414b040 2024-09-17T03:49:47.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.710+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa7ac004030 con 0x7fa7b414b040 2024-09-17T03:49:47.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.711+0000 7fa7b1600640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 0x7fa79407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:47.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.711+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fa7ac09b0b0 con 0x7fa7b414b040 2024-09-17T03:49:47.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.712+0000 7fa7b9400640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 0x7fa79407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:47.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.712+0000 7fa7b9400640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 0x7fa79407b0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8006fd0 tx=0x7fa7a8006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:47.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.716+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa7ac0634e0 con 0x7fa7b414b040 2024-09-17T03:49:47.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.932+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7b40c3860 con 0x7fa7b414b040 2024-09-17T03:49:47.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.933+0000 7fa7b1600640 1 -- 172.21.15.129:0/167883315 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7fa7ac062de0 con 0x7fa7b414b040 2024-09-17T03:49:47.933 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:49:47.933 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:49:47.933 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 msgr2=0x7fa79407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 0x7fa79407b0d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8006fd0 tx=0x7fa7a8006b10 comp rx=0 tx=0).stop 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 msgr2=0x7fa7b414b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa7b40cd410 tx=0x7fa7ac00f3c0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 shutdown_connections 2024-09-17T03:49:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa794078bc0 0x7fa79407b0d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7b414b040 0x7fa7b414b450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7b40d06a0 0x7fa7b40d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 --2- 172.21.15.129:0/167883315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa79c0041c0 0x7fa7b414bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 >> 172.21.15.129:0/167883315 conn(0x7fa7b40b1cb0 msgr2=0x7fa7b415b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 shutdown_connections 2024-09-17T03:49:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:47.936+0000 7fa7bbe00640 1 -- 172.21.15.129:0/167883315 wait complete. 2024-09-17T03:49:48.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:49:35.864880+0000', 'modified': '2024-09-17T03:49:38.881199+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4374': {'gid': 4374, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/3758496952', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 3758496952}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 3758496952}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4374, 'qdb_cluster': [4374]} 2024-09-17T03:49:48.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:49:48.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-17T03:49:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 -- 172.21.15.129:0/2197415735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c0b3b10 msgr2=0x7fe51c0afec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 --2- 172.21.15.129:0/2197415735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c0b3b10 0x7fe51c0afec0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe51c161030 tx=0x7fe50c02a540 comp rx=0 tx=0).stop 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 -- 172.21.15.129:0/2197415735 shutdown_connections 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 --2- 172.21.15.129:0/2197415735 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c153480 0x7fe51c15ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 --2- 172.21.15.129:0/2197415735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c0b3b10 0x7fe51c0afec0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 -- 172.21.15.129:0/2197415735 >> 172.21.15.129:0/2197415735 conn(0x7fe51c0d0830 msgr2=0x7fe51c0d0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 -- 172.21.15.129:0/2197415735 shutdown_connections 2024-09-17T03:49:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.585+0000 7fe521000640 1 -- 172.21.15.129:0/2197415735 wait complete. 2024-09-17T03:49:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.587+0000 7fe521000640 1 Processor -- start 2024-09-17T03:49:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.587+0000 7fe521000640 1 -- start start 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c0b3b10 0x7fe51c1f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 0x7fe51c1525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c15d210 con 0x7fe51c153480 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c15d090 con 0x7fe51c0b3b10 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe521000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c15d390 con 0x7fe51c160500 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43490/0 (socket says 172.21.15.129:43490) 2024-09-17T03:49:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 -- 172.21.15.129:0/1900646584 learned_addr learned my addr 172.21.15.129:0/1900646584 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51b400640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c0b3b10 0x7fe51c1f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51be00640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 0x7fe51c1525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 -- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 msgr2=0x7fe51c1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 0x7fe51c1525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 -- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c0b3b10 msgr2=0x7fe51c1f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c0b3b10 0x7fe51c1f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51aa00640 1 -- 172.21.15.129:0/1900646584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c1f7a10 con 0x7fe51c153480 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.589+0000 7fe51be00640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 0x7fe51c1525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c9a0 tx=0x7fe50800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe513400640 1 -- 172.21.15.129:0/1900646584 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508007bb0 con 0x7fe51c153480 2024-09-17T03:49:48.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe513400640 1 -- 172.21.15.129:0/1900646584 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe508007e40 con 0x7fe51c153480 2024-09-17T03:49:48.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe513400640 1 -- 172.21.15.129:0/1900646584 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50800f690 con 0x7fe51c153480 2024-09-17T03:49:48.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c156730 con 0x7fe51c153480 2024-09-17T03:49:48.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.590+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c154db0 con 0x7fe51c153480 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.592+0000 7fe513400640 1 -- 172.21.15.129:0/1900646584 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe50800f830 con 0x7fe51c153480 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.593+0000 7fe513400640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 0x7fe4fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.594+0000 7fe513400640 1 -- 172.21.15.129:0/1900646584 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe50809b500 con 0x7fe51c153480 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.594+0000 7fe51b400640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 0x7fe4fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.594+0000 7fe5296ac740 1 Processor -- start 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.594+0000 7fe5296ac740 1 -- start start 2024-09-17T03:49:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe51b400640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 0x7fe4fc07b0a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe50c033000 tx=0x7fe50c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 0x559550c4af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c1b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559550ba0430 con 0x55955084d3b0 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559550b905d0 con 0x559550876270 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.595+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x559550b7fa00 con 0x559550855590 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43506/0 (socket says 172.21.15.129:43506) 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 -- 172.21.15.129:0/1751197360 learned_addr learned my addr 172.21.15.129:0/1751197360 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51b400640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 0x559550c4af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 -- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 msgr2=0x559550c1b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51be00640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c1b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c1b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 -- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 msgr2=0x559550c4af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 0x559550c4af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 -- 172.21.15.129:0/1751197360 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559550b48520 con 0x55955084d3b0 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51b400640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 0x559550c4af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51be00640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c1b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe5080604a0 tx=0x7fe508062d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ede890e0c8778b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe507e00640 1 -- 172.21.15.129:0/1751197360 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50801d030 con 0x55955084d3b0 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe507e00640 1 -- 172.21.15.129:0/1751197360 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe508063230 con 0x55955084d3b0 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.596+0000 7fe507e00640 1 -- 172.21.15.129:0/1751197360 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508067d10 con 0x55955084d3b0 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 -- 172.21.15.129:0/1751197360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 msgr2=0x559550c4b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe5080604a0 tx=0x7fe508062d40 comp rx=0 tx=0).stop 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 -- 172.21.15.129:0/1751197360 shutdown_connections 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c1b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55955084d3b0 0x559550c4b4b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1751197360 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559550876270 0x559550c4af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 -- 172.21.15.129:0/1751197360 >> 172.21.15.129:0/1751197360 conn(0x55955086a3c0 msgr2=0x559550c20f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 -- 172.21.15.129:0/1751197360 shutdown_connections 2024-09-17T03:49:48.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.597+0000 7fe5296ac740 1 -- 172.21.15.129:0/1751197360 wait complete. 2024-09-17T03:49:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.598+0000 7fe5296ac740 1 -- start start 2024-09-17T03:49:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.599+0000 7fe5296ac740 1 Processor -- start 2024-09-17T03:49:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c43bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559550b26330 con 0x559550876270 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559550b83810 con 0x55955084d3b0 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5595508f0620 con 0x559550855590 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe5296ac740 10 client.0 _ll_register_callbacks cb 0x559550b2b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe51be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe51be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43522/0 (socket says 172.21.15.129:43522) 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe51be00640 1 -- 172.21.15.129:0/1924702976 learned_addr learned my addr 172.21.15.129:0/1924702976 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe51aa00640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.600+0000 7fe51b400640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c43bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 -- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 msgr2=0x559550c416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 -- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 msgr2=0x559550c43bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c43bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 -- 172.21.15.129:0/1924702976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595508e1850 con 0x559550876270 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe51be00640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe51400e4a0 tx=0x7fe51400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df51a2f75423d63 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe504c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514018060 con 0x559550876270 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe508063230 con 0x559550876270 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe514003d80 con 0x559550876270 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe51401e050 con 0x559550876270 2024-09-17T03:49:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe5296ac740 10 client.4644 fetch_fsmap learned FSMap version 8 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.601+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x5595509586b0 con 0x559550876270 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe508063230 con 0x559550876270 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe514021020 con 0x559550876270 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 10 client.4644 fetch_fsmap finished waiting for FSMap version 8 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 10 client.4644 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 20 client.4644 populate_metadata read hostname 'smithi129' 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550834e40 0x559550c78780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 4 client.4644 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 --> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559550b92520 con 0x559550834e40 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.602+0000 7fe51b400640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550834e40 0x559550c78780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.603+0000 7fe51b400640 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550834e40 0x559550c78780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b39d3af76258b9 server_cookie=d8c4c4a259836d50 in_seq=0 out_seq=0 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.603+0000 7fe504c00640 10 client.4644 ms_handle_connect on v2:172.21.15.181:6835/3758496952 2024-09-17T03:49:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe504c00640 1 -- 172.21.15.129:0/1924702976 <== mds.0 v2:172.21.15.181:6835/3758496952 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559550b92520 con 0x559550834e40 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe504c00640 10 client.4644 handle_command_reply: tid=0 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 client.4644 shutdown 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 msgr2=0x559550c44bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe51400e4a0 tx=0x7fe51400e960 comp rx=0 tx=0).stop 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 shutdown_connections 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550c44bd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550834e40 0x559550c78780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1924702976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c43bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 >> 172.21.15.129:0/1924702976 conn(0x55955086a3c0 msgr2=0x559550c16760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe504c00640 0 client.4644 ms_handle_reset on v2:172.21.15.181:6835/3758496952 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 shutdown_connections 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.604+0000 7fe5296ac740 1 -- 172.21.15.129:0/1924702976 wait complete. 2024-09-17T03:49:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.605+0000 7fe5296ac740 20 client.4644 trim_cache size 0 max 16384 2024-09-17T03:49:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.624+0000 7fe5296ac740 1 Processor -- start 2024-09-17T03:49:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.624+0000 7fe5296ac740 1 -- start start 2024-09-17T03:49:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c66b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 0x559550c67c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1fdc40 con 0x559550855590 2024-09-17T03:49:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c070610 con 0x55955084d3b0 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe51c140c70 con 0x559550876270 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43526/0 (socket says 172.21.15.129:43526) 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.625+0000 7fe51aa00640 1 -- 172.21.15.129:0/3718893637 learned_addr learned my addr 172.21.15.129:0/3718893637 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 -- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 msgr2=0x559550c67c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51be00640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 0x559550c67c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51b400640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c66b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 0x559550c67c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 -- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 msgr2=0x559550c66b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c66b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 -- 172.21.15.129:0/3718893637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559550c21c50 con 0x559550855590 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51b400640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c66b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:48.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51be00640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 0x559550c67c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:49:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe51aa00640 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe50801e030 tx=0x7fe508000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf206664d3c6196f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe507400640 1 -- 172.21.15.129:0/3718893637 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508067d10 con 0x559550855590 2024-09-17T03:49:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe507400640 1 -- 172.21.15.129:0/3718893637 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe50805f4a0 con 0x559550855590 2024-09-17T03:49:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.626+0000 7fe507400640 1 -- 172.21.15.129:0/3718893637 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50809b0b0 con 0x559550855590 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 -- 172.21.15.129:0/3718893637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 msgr2=0x559550c670c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe50801e030 tx=0x7fe508000bc0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 -- 172.21.15.129:0/3718893637 shutdown_connections 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550876270 0x559550c67c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550855590 0x559550c670c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 --2- 172.21.15.129:0/3718893637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c66b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 -- 172.21.15.129:0/3718893637 >> 172.21.15.129:0/3718893637 conn(0x55955086a3c0 msgr2=0x559550c449c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 -- 172.21.15.129:0/3718893637 shutdown_connections 2024-09-17T03:49:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.627+0000 7fe5296ac740 1 -- 172.21.15.129:0/3718893637 wait complete. 2024-09-17T03:49:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.628+0000 7fe5296ac740 1 -- start start 2024-09-17T03:49:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.628+0000 7fe5296ac740 1 Processor -- start 2024-09-17T03:49:48.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.629+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c88620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.629+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c66d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.629+0000 7fe5296ac740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550cdb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.629+0000 7fe5296ac740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559550c44160 con 0x559550876270 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.629+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1412c0 con 0x55955084d3b0 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe5296ac740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x559550c5d0b0 con 0x559550855590 2024-09-17T03:49:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe5296ac740 10 client.0 _ll_register_callbacks cb 0x7fe514001ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c88620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550cdb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c88620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:46044/0 (socket says 172.21.15.129:46044) 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c66d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51b400640 1 -- 172.21.15.129:0/1036773752 learned_addr learned my addr 172.21.15.129:0/1036773752 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 -- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 msgr2=0x559550c66d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c66d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 -- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 msgr2=0x559550c88620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c88620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 -- 172.21.15.129:0/1036773752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559550c1ecd0 con 0x559550876270 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe51be00640 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550cdb540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x559550c20e90 tx=0x7fe514003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0131dacadb9a1c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.630+0000 7fe504c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:49:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe514013850 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe5140139f0 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe50805f4a0 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe514033050 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 10 client.4650 fetch_fsmap learned FSMap version 8 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x559550c672b0 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe50805f4a0 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe514036020 con 0x559550876270 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 10 client.4650 fetch_fsmap finished waiting for FSMap version 8 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 10 client.4650 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 20 client.4650 populate_metadata read hostname 'smithi129' 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550ce5870 0x559550c7e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 4 client.4650 mds_command: new command op to 4374 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:49:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.631+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 --> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] -- command(tid 0: {"prefix": "status"}) -- 0x559550b8dfd0 con 0x559550ce5870 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.632+0000 7fe51b400640 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550ce5870 0x559550c7e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.632+0000 7fe51b400640 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550ce5870 0x559550c7e710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49827dc00612eb80 server_cookie=ba725d7deae94cdc in_seq=0 out_seq=0 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.632+0000 7fe504c00640 10 client.4650 ms_handle_connect on v2:172.21.15.181:6835/3758496952 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe504c00640 1 -- 172.21.15.129:0/1036773752 <== mds.0 v2:172.21.15.181:6835/3758496952 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x559550b8dfd0 con 0x559550ce5870 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe504c00640 10 client.4650 handle_command_reply: tid=0 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 client.4650 shutdown 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 msgr2=0x559550cdb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550cdb540 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x559550c20e90 tx=0x7fe514003e30 comp rx=0 tx=0).stop 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 shutdown_connections 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x559550876270 0x559550cdb540 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:6835/3758496952,v1:172.21.15.181:6837/3758496952] conn(0x559550ce5870 0x559550c7e710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x559550855590 0x559550c66d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 --2- 172.21.15.129:0/1036773752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55955084d3b0 0x559550c88620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe504c00640 0 client.4650 ms_handle_reset on v2:172.21.15.181:6835/3758496952 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 >> 172.21.15.129:0/1036773752 conn(0x55955086a3c0 msgr2=0x559550c5e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 shutdown_connections 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 1 -- 172.21.15.129:0/1036773752 wait complete. 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.633+0000 7fe5296ac740 20 client.4650 trim_cache size 0 max 16384 2024-09-17T03:49:48.637 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4374, 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 12.243354452, 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 8, 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 22, 2024-09-17T03:49:48.638 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 22, 2024-09-17T03:49:48.639 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 18.649293827000001 2024-09-17T03:49:48.639 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:49:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 msgr2=0x7fe4fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 0x7fe4fc07b0a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe50c033000 tx=0x7fe50c004730 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 msgr2=0x7fe51c152060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c9a0 tx=0x7fe50800ce60 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 shutdown_connections 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4fc078b90 0x7fe4fc07b0a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe51c160500 0x7fe51c1525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe51c153480 0x7fe51c152060 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 --2- 172.21.15.129:0/1900646584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe51c0b3b10 0x7fe51c1f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 >> 172.21.15.129:0/1900646584 conn(0x7fe51c0d0830 msgr2=0x7fe51c06d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 shutdown_connections 2024-09-17T03:49:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:48.634+0000 7fe521000640 1 -- 172.21.15.129:0/1900646584 wait complete. 2024-09-17T03:49:48.657 INFO:teuthology.run_tasks:Running task kclient... 2024-09-17T03:49:48.682 INFO:tasks.kclient:Mounting kernel clients... 2024-09-17T03:49:48.683 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-17T03:49:48.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.683 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:49:48.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.720 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-09-17T03:49:48.818 INFO:teuthology.orchestra.run.smithi129.stderr:Cannot find device "ceph-brx" 2024-09-17T03:49:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:49:48.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.822 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:49:48.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-09-17T03:49:48.954 INFO:teuthology.orchestra.run.smithi129.stderr:Cannot find device "ceph-brx" 2024-09-17T03:49:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:49:48.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.957 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T03:49:48.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:48.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link delete ceph-brx 2024-09-17T03:49:49.101 INFO:teuthology.orchestra.run.smithi181.stderr:Cannot find device "ceph-brx" 2024-09-17T03:49:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:49:49.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:49.104 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T03:49:49.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:49.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link delete ceph-brx 2024-09-17T03:49:49.239 INFO:teuthology.orchestra.run.smithi181.stderr:Cannot find device "ceph-brx" 2024-09-17T03:49:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:49:49.242 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-09-17T03:49:49.243 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T03:49:49.243 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:49:49.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 -- 172.21.15.129:0/1974886078 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 msgr2=0x7fe8ec0ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/1974886078 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec0ccb10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8009830 tx=0x7fe8d802a510 comp rx=0 tx=0).stop 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 -- 172.21.15.129:0/1974886078 shutdown_connections 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/1974886078 >> v2:172.21.15.181:3300/0 conn(0x7fe8ec0cd050 0x7fe8ec15f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/1974886078 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec0ccb10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 -- 172.21.15.129:0/1974886078 >> 172.21.15.129:0/1974886078 conn(0x7fe8ec06d7d0 msgr2=0x7fe8ec0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 -- 172.21.15.129:0/1974886078 shutdown_connections 2024-09-17T03:49:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.466+0000 7fe8f1e00640 1 -- 172.21.15.129:0/1974886078 wait complete. 2024-09-17T03:49:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.469+0000 7fe8f1e00640 1 Processor -- start 2024-09-17T03:49:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.469+0000 7fe8f1e00640 1 -- start start 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8ec0cd050 0x7fe8ec09af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec15f950 con 0x7fe8ec06ee90 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe8ec15c980 con 0x7fe8ec0cd050 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8f1e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe8ec15fad0 con 0x7fe8ec06f2a0 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.470+0000 7fe8eaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43538/0 (socket says 172.21.15.129:43538) 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 -- 172.21.15.129:0/2145034566 learned_addr learned my addr 172.21.15.129:0/2145034566 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eb400640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8ec0cd050 0x7fe8ec09af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8ea000640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:49.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 -- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 msgr2=0x7fe8ec09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:49.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec09aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 -- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8ec0cd050 msgr2=0x7fe8ec09af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:49.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8ec0cd050 0x7fe8ec09af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 -- 172.21.15.129:0/2145034566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8ec09da50 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8eaa00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c370 tx=0x7fe8dc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc006890 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8dc006b20 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.471+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc007020 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.472+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8ec09dd30 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.472+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8ec0b0330 con 0x7fe8ec06ee90 2024-09-17T03:49:49.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.473+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8ec0750f0 con 0x7fe8ec06ee90 2024-09-17T03:49:49.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.474+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe8dc019020 con 0x7fe8ec06ee90 2024-09-17T03:49:49.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.475+0000 7fe8e3400640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 0x7fe8cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:49.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.475+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fe8dc09a500 con 0x7fe8ec06ee90 2024-09-17T03:49:49.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.475+0000 7fe8ea000640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 0x7fe8cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:49.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.476+0000 7fe8ea000640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 0x7fe8cc07b0a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8034000 tx=0x7fe8d8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:49.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.479+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe8dc0628a0 con 0x7fe8ec06ee90 2024-09-17T03:49:49.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.698+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe8ec06fad0 con 0x7fe8ec06ee90 2024-09-17T03:49:49.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.698+0000 7fe8e3400640 1 -- 172.21.15.129:0/2145034566 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7fe8dc0621a0 con 0x7fe8ec06ee90 2024-09-17T03:49:49.697 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.700+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 msgr2=0x7fe8cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.700+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 0x7fe8cc07b0a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8034000 tx=0x7fe8d8004730 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.700+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 msgr2=0x7fe8ec0cb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.700+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc00c370 tx=0x7fe8dc00c830 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 shutdown_connections 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8cc078b90 0x7fe8cc07b0a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8ec0cd050 0x7fe8ec09af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8ec06f2a0 0x7fe8ec09aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 --2- 172.21.15.129:0/2145034566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8ec06ee90 0x7fe8ec0cb770 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:49.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 >> 172.21.15.129:0/2145034566 conn(0x7fe8ec06d7d0 msgr2=0x7fe8ec0702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 shutdown_connections 2024-09-17T03:49:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49.701+0000 7fe8f1e00640 1 -- 172.21.15.129:0/2145034566 wait complete. 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T03:49:49.724 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:49:49.725 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:49:49.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:49.725 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:49:49.760 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:49:49.760 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:49.761 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42702sec preferred_lft 42702sec 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:49:49.762 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:49.762 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T03:49:49.762 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add name ceph-brx type bridge 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr flush dev ceph-brx 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set ceph-brx up 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T03:49:49.763 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:49.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:50.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:50.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:50.189 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:49:50.246 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:49:50.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:50.250 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:49:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:49:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:49:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:49:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:49:50.306 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:49:50.307 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:50.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:50.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:50.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:50.918 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:49:50.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:50.955 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list-id 2024-09-17T03:49:51.032 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:51.032 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:51.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:49:51.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-17T03:49:51.033 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:51.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:51.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:51.212 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-17T03:49:51.213 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:51.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:51.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:51.632 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:51.633 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:51.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:49:51.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T03:49:51.633 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:51.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:51.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:51.800 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:51.800 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:49:51.835 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:49:51.835 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:51.835 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:49:51.911 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:49:51.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:51.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:49:52.112 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:49:52.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:49:52.113 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:52.761 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:49:52.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:49:52.761 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:52.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:52.762 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:49:52.796 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:49:52.797 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:49:52.798 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:49:52.798 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:49:52.798 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:49:52.798 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:49:52.798 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:49:52.801 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:49:52.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:52.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:49:52.858 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:52.959 INFO:teuthology.orchestra.run.smithi129.stdout:client4656 2024-09-17T03:49:52.960 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:52.960 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:49:52.995 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:53.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:53.134 INFO:teuthology.orchestra.run.smithi129.stdout:client4656 2024-09-17T03:49:53.134 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:53.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:53.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:53.234 INFO:teuthology.orchestra.run.smithi129.stdout:client4656 2024-09-17T03:49:53.235 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c5b53c7d-83e3-4793-838f-5ade77ae49c7.client4656/status') 2024-09-17T03:49:53.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:53.341 INFO:teuthology.orchestra.run.smithi129.stdout:instance: client.4656 (3)192.168.0.1:0/1731193521 2024-09-17T03:49:53.341 INFO:teuthology.orchestra.run.smithi129.stdout:blocklisted: false 2024-09-17T03:49:53.342 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records in 2024-09-17T03:49:53.342 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records out 2024-09-17T03:49:53.342 INFO:teuthology.orchestra.run.smithi129.stderr:69 bytes copied, 6.5757e-05 s, 1.0 MB/s 2024-09-17T03:49:53.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:53.352 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-09-17T03:49:53.352 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T03:49:53.352 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-17T03:49:53.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.574+0000 7f5e0ea00640 1 -- 172.21.15.129:0/1850261318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df0003a00 msgr2=0x7f5df0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.574+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/1850261318 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5df0003a00 0x7f5df0003e70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc00c290 tx=0x7f5df0005830 comp rx=0 tx=0).stop 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.575+0000 7f5e0e000640 1 -- 172.21.15.129:0/1850261318 reap_dead start 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.575+0000 7f5e0ea00640 1 -- 172.21.15.129:0/1850261318 shutdown_connections 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.575+0000 7f5e0ea00640 1 -- 172.21.15.129:0/1850261318 >> 172.21.15.129:0/1850261318 conn(0x7f5e0809d680 msgr2=0x7f5e080d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.575+0000 7f5e0ea00640 1 -- 172.21.15.129:0/1850261318 shutdown_connections 2024-09-17T03:49:53.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.575+0000 7f5e0ea00640 1 -- 172.21.15.129:0/1850261318 wait complete. 2024-09-17T03:49:53.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.577+0000 7f5e0ea00640 1 Processor -- start 2024-09-17T03:49:53.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.577+0000 7f5e0ea00640 1 -- start start 2024-09-17T03:49:53.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.578+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5df0003a00 0x7f5e08157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:53.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.578+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:53.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e0815cf70 0x7f5e0815d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:53.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e08163650 con 0x7f5e08155a50 2024-09-17T03:49:53.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081634d0 con 0x7f5e0815cf70 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0ea00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5e081637d0 con 0x7f5df0003a00 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43568/0 (socket says 172.21.15.129:43568) 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 -- 172.21.15.129:0/2474091163 learned_addr learned my addr 172.21.15.129:0/2474091163 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0d600640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5df0003a00 0x7f5e08157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0e000640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e0815cf70 0x7f5e0815d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 -- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5df0003a00 msgr2=0x7f5e08157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5df0003a00 0x7f5e08157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 -- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e0815cf70 msgr2=0x7f5e0815d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e0815cf70 0x7f5e0815d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.579+0000 7f5e0cc00640 1 -- 172.21.15.129:0/2474091163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0815d8c0 con 0x7f5e08155a50 2024-09-17T03:49:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e0cc00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5df4011900 tx=0x7f5df4011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:53.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df400fe70 con 0x7f5e08155a50 2024-09-17T03:49:53.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5df4004050 con 0x7f5e08155a50 2024-09-17T03:49:53.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e080cb530 con 0x7f5e08155a50 2024-09-17T03:49:53.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df40044a0 con 0x7f5e08155a50 2024-09-17T03:49:53.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.580+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e080cbaf0 con 0x7f5e08155a50 2024-09-17T03:49:53.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.581+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e08077030 con 0x7f5e08155a50 2024-09-17T03:49:53.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.582+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5df40035f0 con 0x7f5e08155a50 2024-09-17T03:49:53.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.583+0000 7f5e05600640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 0x7f5de807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:53.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.583+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5df409bd10 con 0x7f5e08155a50 2024-09-17T03:49:53.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.584+0000 7f5e0d600640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 0x7f5de807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:53.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.584+0000 7f5e0d600640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 0x7f5de807b0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc033000 tx=0x7f5e080b5c40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.588+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5e08077030 con 0x7f5e08155a50 2024-09-17T03:49:53.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.807+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5e080b75b0 con 0x7f5e08155a50 2024-09-17T03:49:53.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.807+0000 7f5e05600640 1 -- 172.21.15.129:0/2474091163 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f5e0815d8c0 con 0x7f5e08155a50 2024-09-17T03:49:53.807 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 msgr2=0x7f5de807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 0x7f5de807b0d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc033000 tx=0x7f5e080b5c40 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 msgr2=0x7f5e08154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5df4011900 tx=0x7f5df4011dc0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 shutdown_connections 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5de8078bc0 0x7f5de807b0d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e0815cf70 0x7f5e0815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e08155a50 0x7f5e08154760 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 --2- 172.21.15.129:0/2474091163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5df0003a00 0x7f5e08157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:53.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 >> 172.21.15.129:0/2474091163 conn(0x7f5e0809d680 msgr2=0x7f5e081618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:53.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 shutdown_connections 2024-09-17T03:49:53.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:53.810+0000 7f5e0ea00640 1 -- 172.21.15.129:0/2474091163 wait complete. 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:49:53.833 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-17T03:49:53.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:53.834 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:49:53.869 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:49:53.870 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42698sec preferred_lft 42698sec 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:49:53.871 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.872 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:49:53.872 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:49:53.872 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:49:53.872 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:53.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:53.872 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:49:53.963 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:49:53.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:53.967 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:49:54.003 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:49:54.003 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:49:54.003 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:49:54.003 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:49:54.003 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:49:54.004 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:54.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:54.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:54.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:54.200 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:49:54.235 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:49:54.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:54.236 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list-id 2024-09-17T03:49:54.313 INFO:teuthology.orchestra.run.smithi129.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-17T03:49:54.313 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:54.313 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:54.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:49:54.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-17T03:49:54.314 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:54.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:54.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:54.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:54.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-17T03:49:54.545 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:49:54.545 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:49:54.545 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether aa:60:25:f2:83:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::a860:25ff:fef2:837a/64 scope link 2024-09-17T03:49:54.546 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:54.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-17T03:49:54.603 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:49:54.603 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:49:54.603 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether aa:60:25:f2:83:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::a860:25ff:fef2:837a/64 scope link 2024-09-17T03:49:54.604 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:54.606 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-17T03:49:54.606 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-17T03:49:54.607 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:54.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:54.822 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:54.822 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:49:54.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.1 up 2024-09-17T03:49:54.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set dev brx.1 master ceph-brx 2024-09-17T03:49:54.822 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:49:54.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:54.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:54.988 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:54.988 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-17T03:49:55.024 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-17T03:49:55.024 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:55.024 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-17T03:49:55.098 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:49:55.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:55.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-17T03:49:55.293 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-17T03:49:55.293 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-17T03:49:55.293 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:55.313 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-17T03:49:55.313 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-17T03:49:55.314 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:55.314 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:55.314 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:55.344 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-09-17T03:49:55.344 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-09-17T03:49:55.345 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:49:55.348 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:55.348 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:55.348 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:49:55.348 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:49:55.348 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:49:55.350 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:49:55.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:55.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-17T03:49:55.407 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:55.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:55.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:55.499 INFO:teuthology.orchestra.run.smithi129.stdout:client4633 2024-09-17T03:49:55.499 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:49:55.499 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:49:55.534 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:55.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:55.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:55.664 INFO:teuthology.orchestra.run.smithi129.stdout:client4633 2024-09-17T03:49:55.664 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:55.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:55.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:55.753 INFO:teuthology.orchestra.run.smithi129.stdout:client4633 2024-09-17T03:49:55.753 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c5b53c7d-83e3-4793-838f-5ade77ae49c7.client4633/status') 2024-09-17T03:49:55.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:55.851 INFO:teuthology.orchestra.run.smithi129.stdout:instance: client.4633 (3)172.21.15.129:0/313590176 2024-09-17T03:49:55.851 INFO:teuthology.orchestra.run.smithi129.stdout:blocklisted: false 2024-09-17T03:49:55.851 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records in 2024-09-17T03:49:55.851 INFO:teuthology.orchestra.run.smithi129.stderr:0+1 records out 2024-09-17T03:49:55.851 INFO:teuthology.orchestra.run.smithi129.stderr:70 bytes copied, 6.4438e-05 s, 1.1 MB/s 2024-09-17T03:49:55.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:55.862 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-09-17T03:49:55.862 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T03:49:55.862 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-17T03:49:55.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 -- 172.21.15.129:0/1600572004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140cf6f0 msgr2=0x7faa140cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 --2- 172.21.15.129:0/1600572004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140cf6f0 0x7faa140cc4c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faa04009830 tx=0x7faa0402a5c0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 -- 172.21.15.129:0/1600572004 shutdown_connections 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 --2- 172.21.15.129:0/1600572004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140cf6f0 0x7faa140cc4c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 -- 172.21.15.129:0/1600572004 >> 172.21.15.129:0/1600572004 conn(0x7faa140b1cb0 msgr2=0x7faa140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.082+0000 7faa19e00640 1 -- 172.21.15.129:0/1600572004 shutdown_connections 2024-09-17T03:49:56.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.083+0000 7faa19e00640 1 -- 172.21.15.129:0/1600572004 wait complete. 2024-09-17T03:49:56.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.085+0000 7faa19e00640 1 Processor -- start 2024-09-17T03:49:56.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.085+0000 7faa19e00640 1 -- start start 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa140ce020 0x7faa1409c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa140cf6f0 0x7faa1409c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa140cf340 con 0x7faa140ccb30 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa140cf1c0 con 0x7faa140cf6f0 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa19e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faa140701b0 con 0x7faa140ce020 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa12a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa12a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40660/0 (socket says 172.21.15.129:40660) 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.086+0000 7faa12a00640 1 -- 172.21.15.129:0/2673913676 learned_addr learned my addr 172.21.15.129:0/2673913676 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa13400640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa140cf6f0 0x7faa1409c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12000640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa140ce020 0x7faa1409c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 -- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa140ce020 msgr2=0x7faa1409c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa140ce020 0x7faa1409c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 -- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa140cf6f0 msgr2=0x7faa1409c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa140cf6f0 0x7faa1409c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 -- 172.21.15.129:0/2673913676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1415f390 con 0x7faa140ccb30 2024-09-17T03:49:56.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa12a00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa0402abb0 tx=0x7faa04033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04031b50 con 0x7faa140ccb30 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.087+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa04031de0 con 0x7faa140ccb30 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.088+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa1409d320 con 0x7faa140ccb30 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.088+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04005280 con 0x7faa140ccb30 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.088+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa1415f630 con 0x7faa140ccb30 2024-09-17T03:49:56.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.089+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa140cc4c0 con 0x7faa140ccb30 2024-09-17T03:49:56.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.090+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faa04005420 con 0x7faa140ccb30 2024-09-17T03:49:56.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.091+0000 7faa0b400640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 0x7fa9f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:49:56.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.091+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7faa040bbf90 con 0x7faa140ccb30 2024-09-17T03:49:56.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.091+0000 7faa12000640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 0x7fa9f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:49:56.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.092+0000 7faa12000640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 0x7fa9f407b0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa140d0fa0 tx=0x7faa00009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:49:56.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.095+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faa0403f030 con 0x7faa140ccb30 2024-09-17T03:49:56.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.316+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faa140ad5f0 con 0x7faa140ccb30 2024-09-17T03:49:56.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.317+0000 7faa0b400640 1 -- 172.21.15.129:0/2673913676 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7faa04084330 con 0x7faa140ccb30 2024-09-17T03:49:56.317 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:49:56.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 msgr2=0x7fa9f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:56.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 0x7fa9f407b0a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa140d0fa0 tx=0x7faa00009040 comp rx=0 tx=0).stop 2024-09-17T03:49:56.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 msgr2=0x7faa14154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:49:56.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faa0402abb0 tx=0x7faa04033ba0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 shutdown_connections 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f4078b90 0x7fa9f407b0a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa140cf6f0 0x7faa1409c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa140ce020 0x7faa1409c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 --2- 172.21.15.129:0/2673913676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa140ccb30 0x7faa14154770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 >> 172.21.15.129:0/2673913676 conn(0x7faa140b1cb0 msgr2=0x7faa140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 shutdown_connections 2024-09-17T03:49:56.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:49:56.319+0000 7faa19e00640 1 -- 172.21.15.129:0/2673913676 wait complete. 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi181.front.sepia.ceph.com 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-17T03:49:56.344 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-17T03:49:56.345 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T03:49:56.345 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:49:56.345 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-17T03:49:56.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:56.345 DEBUG:teuthology.orchestra.run.smithi181:> ip addr 2024-09-17T03:49:56.384 INFO:teuthology.orchestra.run.smithi181.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 ::1/128 scope host 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:10 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f0 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:11 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f1 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:49:56.385 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9c brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9d brd ff:ff:ff:ff:ff:ff 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout: inet 172.21.15.181/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft 42688sec preferred_lft 42688sec 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::ec4:7aff:fe88:7b9d/64 scope link noprefixroute 2024-09-17T03:49:56.386 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:49:56.387 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link add name ceph-brx type bridge 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip addr flush dev ceph-brx 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set ceph-brx up 2024-09-17T03:49:56.387 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-17T03:49:56.388 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:49:56.511 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:56.814 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:56.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:56.825 DEBUG:teuthology.orchestra.run.smithi181:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:49:56.882 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-17T03:49:56.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:56.886 DEBUG:teuthology.orchestra.run.smithi181:> route 2024-09-17T03:49:56.936 INFO:teuthology.orchestra.run.smithi181.stdout:Kernel IP routing table 2024-09-17T03:49:56.937 INFO:teuthology.orchestra.run.smithi181.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:49:56.937 INFO:teuthology.orchestra.run.smithi181.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:49:56.937 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:49:56.937 INFO:teuthology.orchestra.run.smithi181.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:49:56.937 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:49:57.015 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:57.573 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:57.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:57.581 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T03:49:57.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:57.618 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list-id 2024-09-17T03:49:57.694 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:57.694 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:49:57.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T03:49:57.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-17T03:49:57.694 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:49:57.812 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:57.865 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:57.873 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-17T03:49:57.873 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-17T03:49:57.874 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-17T03:49:57.874 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:49:57.951 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:58.109 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:58.117 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:49:58.117 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:49:58.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set brx.0 up 2024-09-17T03:49:58.117 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set dev brx.0 master ceph-brx 2024-09-17T03:49:58.117 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:49:58.194 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:58.275 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:58.283 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:58.283 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-17T03:49:58.319 INFO:teuthology.orchestra.run.smithi181.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-17T03:49:58.319 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:49:58.320 DEBUG:teuthology.orchestra.run.smithi181:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-17T03:49:58.395 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:49:58.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:58.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-17T03:49:58.601 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-17T03:49:58.601 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-17T03:49:58.602 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:49:59.251 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-17T03:49:59.251 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-17T03:49:59.251 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:49:59.252 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:59.252 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:49:59.287 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:49:59.289 INFO:teuthology.orchestra.run.smithi181.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-17T03:49:59.289 INFO:teuthology.orchestra.run.smithi181.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:49:59.289 INFO:teuthology.orchestra.run.smithi181.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:49:59.289 INFO:teuthology.orchestra.run.smithi181.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:49:59.289 INFO:teuthology.orchestra.run.smithi181.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:49:59.292 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:49:59.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:49:59.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-17T03:49:59.349 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:59.429 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:59.433 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:59.441 INFO:teuthology.orchestra.run.smithi181.stdout:client4642 2024-09-17T03:49:59.441 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:49:59.441 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:49:59.480 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:59.596 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:59.599 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:59.607 INFO:teuthology.orchestra.run.smithi181.stdout:client4642 2024-09-17T03:49:59.607 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:49:59.685 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:49:59.688 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:59.696 INFO:teuthology.orchestra.run.smithi181.stdout:client4642 2024-09-17T03:49:59.696 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c5b53c7d-83e3-4793-838f-5ade77ae49c7.client4642/status') 2024-09-17T03:49:59.773 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:49:59.794 INFO:teuthology.orchestra.run.smithi181.stdout:instance: client.4642 (3)192.168.0.1:0/2105197640 2024-09-17T03:49:59.794 INFO:teuthology.orchestra.run.smithi181.stdout:blocklisted: false 2024-09-17T03:49:59.795 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records in 2024-09-17T03:49:59.795 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records out 2024-09-17T03:49:59.795 INFO:teuthology.orchestra.run.smithi181.stderr:69 bytes copied, 6.5779e-05 s, 1.0 MB/s 2024-09-17T03:49:59.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:49:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:49:59.806 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-09-17T03:49:59.806 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-17T03:49:59.806 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-17T03:49:59.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 -- 172.21.15.129:0/2203509538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 msgr2=0x7f5f7c0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 --2- 172.21.15.129:0/2203509538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c0b40b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5f68005fd0 tx=0x7f5f6802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 -- 172.21.15.129:0/2203509538 shutdown_connections 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 --2- 172.21.15.129:0/2203509538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c0b40b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 --2- 172.21.15.129:0/2203509538 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0cc1e0 0x7f5f7c0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 -- 172.21.15.129:0/2203509538 >> 172.21.15.129:0/2203509538 conn(0x7f5f7c06d7d0 msgr2=0x7f5f7c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.037+0000 7f5f83000640 1 -- 172.21.15.129:0/2203509538 shutdown_connections 2024-09-17T03:50:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.038+0000 7f5f83000640 1 -- 172.21.15.129:0/2203509538 wait complete. 2024-09-17T03:50:00.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.040+0000 7f5f83000640 1 Processor -- start 2024-09-17T03:50:00.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.040+0000 7f5f83000640 1 -- start start 2024-09-17T03:50:00.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 0x7f5f7c072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:00.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 0x7f5f7c151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c153520 con 0x7f5f7c15d8a0 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c1533a0 con 0x7f5f7c0cc1e0 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.041+0000 7f5f83000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5f7c1536a0 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 0x7f5f7c072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 0x7f5f7c151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f81000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 0x7f5f7c072f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:59608/0 (socket says 172.21.15.129:59608) 2024-09-17T03:50:00.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7be00640 1 -- 172.21.15.129:0/1855996232 learned_addr learned my addr 172.21.15.129:0/1855996232 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:00.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 -- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 msgr2=0x7f5f7c072f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 0x7f5f7c072f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 -- 172.21.15.129:0/1855996232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 msgr2=0x7f5f7c156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 -- 172.21.15.129:0/1855996232 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c1fa880 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f81000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.042+0000 7f5f7b400640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 0x7f5f7c151f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f68031000 tx=0x7f5f68005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.043+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f6803e030 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.043+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f68003a70 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.043+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c1f9d00 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.043+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68035040 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.043+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c1fa2c0 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.044+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f7c071740 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.045+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5f68041030 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.046+0000 7f5f73e00640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 0x7f5f5c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:00.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.046+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f5f680bb8e0 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.047+0000 7f5f7be00640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 0x7f5f5c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:00.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.047+0000 7f5f7be00640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 0x7f5f5c07b0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0049d0 tx=0x7f5f6c009500 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:00.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.051+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5f7c071740 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.271+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5f7c153b50 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.272+0000 7f5f73e00640 1 -- 172.21.15.129:0/1855996232 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f5f68083770 con 0x7f5f7c0ccb50 2024-09-17T03:50:00.272 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:50:00.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 msgr2=0x7f5f5c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 0x7f5f5c07b0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c0049d0 tx=0x7f5f6c009500 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 msgr2=0x7f5f7c151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 0x7f5f7c151f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5f68031000 tx=0x7f5f68005750 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 shutdown_connections 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078bc0 0x7f5f5c07b0d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f7c15d8a0 0x7f5f7c156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f7c0ccb50 0x7f5f7c151f00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 --2- 172.21.15.129:0/1855996232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f7c0cc1e0 0x7f5f7c072f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 >> 172.21.15.129:0/1855996232 conn(0x7f5f7c06d7d0 msgr2=0x7f5f7c15c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 shutdown_connections 2024-09-17T03:50:00.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:00.275+0000 7f5f83000640 1 -- 172.21.15.129:0/1855996232 wait complete. 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi181.front.sepia.ceph.com 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-17T03:50:00.302 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:50:00.303 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-17T03:50:00.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.303 DEBUG:teuthology.orchestra.run.smithi181:> ip addr 2024-09-17T03:50:00.339 INFO:teuthology.orchestra.run.smithi181.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:50:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:50:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:50:00.339 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 ::1/128 scope host 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:10 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f0 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:d9:39:11 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: altname enp7s0f1 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9c brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:50:00.340 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether 0c:c4:7a:88:7b:9d brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: inet 172.21.15.181/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft 42684sec preferred_lft 42684sec 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::ec4:7aff:fe88:7b9d/64 scope link noprefixroute 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether e2:a7:3d:98:f0:3d brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::7cd8:16ff:fe7a:119e/64 scope link 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:50:00.341 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether e2:a7:3d:98:f0:3d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T03:50:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::e0a7:3dff:fe98:f03d/64 scope link 2024-09-17T03:50:00.342 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:00.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.342 DEBUG:teuthology.orchestra.run.smithi181:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:50:00.435 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-17T03:50:00.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.438 DEBUG:teuthology.orchestra.run.smithi181:> route 2024-09-17T03:50:00.474 INFO:teuthology.orchestra.run.smithi181.stdout:Kernel IP routing table 2024-09-17T03:50:00.475 INFO:teuthology.orchestra.run.smithi181.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:50:00.475 INFO:teuthology.orchestra.run.smithi181.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:50:00.475 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:50:00.475 INFO:teuthology.orchestra.run.smithi181.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:50:00.475 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:50:00.592 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:00.662 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:00.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.671 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T03:50:00.705 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-17T03:50:00.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.706 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list-id 2024-09-17T03:50:00.780 INFO:teuthology.orchestra.run.smithi181.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-17T03:50:00.781 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:50:00.781 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:50:00.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-17T03:50:00.782 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-17T03:50:00.782 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:50:00.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:00.949 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:00.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:00.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 ::1/128 scope host 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether ca:6b:91:f2:8f:38 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.013 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::c86b:91ff:fef2:8f38/64 scope link 2024-09-17T03:50:01.014 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:01.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 ::1/128 scope host 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: link/ether ca:6b:91:f2:8f:38 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-17T03:50:01.071 INFO:teuthology.orchestra.run.smithi181.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-17T03:50:01.072 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.072 INFO:teuthology.orchestra.run.smithi181.stdout: inet6 fe80::c86b:91ff:fef2:8f38/64 scope link 2024-09-17T03:50:01.072 INFO:teuthology.orchestra.run.smithi181.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:01.073 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-17T03:50:01.074 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:50:01.152 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:01.280 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:01.288 DEBUG:teuthology.orchestra.run.smithi181:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:50:01.288 DEBUG:teuthology.orchestra.run.smithi181:> set -e 2024-09-17T03:50:01.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set brx.1 up 2024-09-17T03:50:01.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link set dev brx.1 master ceph-brx 2024-09-17T03:50:01.289 DEBUG:teuthology.orchestra.run.smithi181:> ') 2024-09-17T03:50:01.365 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:01.445 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:01.453 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:50:01.453 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-17T03:50:01.489 INFO:teuthology.orchestra.run.smithi181.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-17T03:50:01.489 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:50:01.489 DEBUG:teuthology.orchestra.run.smithi181:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-17T03:50:01.564 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:50:01.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:01.565 DEBUG:teuthology.orchestra.run.smithi181:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-17T03:50:01.761 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-17T03:50:01.761 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-17T03:50:01.761 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:50:01.779 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-17T03:50:01.779 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-17T03:50:01.779 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:50:01.779 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:50:01.780 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:invalid new device string format 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:50:01.809 INFO:teuthology.orchestra.run.smithi181.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:50:01.811 INFO:teuthology.orchestra.run.smithi181.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-09-17T03:50:01.811 INFO:teuthology.orchestra.run.smithi181.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:50:01.811 INFO:teuthology.orchestra.run.smithi181.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:50:01.811 INFO:teuthology.orchestra.run.smithi181.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:50:01.811 INFO:teuthology.orchestra.run.smithi181.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:50:01.814 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:50:01.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:01.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-17T03:50:01.871 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:50:01.950 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:50:01.952 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:01.960 INFO:teuthology.orchestra.run.smithi181.stdout:client4680 2024-09-17T03:50:01.960 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:01.960 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:01.995 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:50:02.112 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:50:02.115 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:02.123 INFO:teuthology.orchestra.run.smithi181.stdout:client4680 2024-09-17T03:50:02.123 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:50:02.201 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:50:02.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:02.212 INFO:teuthology.orchestra.run.smithi181.stdout:client4680 2024-09-17T03:50:02.212 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/c5b53c7d-83e3-4793-838f-5ade77ae49c7.client4680/status') 2024-09-17T03:50:02.289 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:02.311 INFO:teuthology.orchestra.run.smithi181.stdout:instance: client.4680 (3)172.21.15.181:0/3275678558 2024-09-17T03:50:02.311 INFO:teuthology.orchestra.run.smithi181.stdout:blocklisted: false 2024-09-17T03:50:02.311 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records in 2024-09-17T03:50:02.311 INFO:teuthology.orchestra.run.smithi181.stderr:0+1 records out 2024-09-17T03:50:02.311 INFO:teuthology.orchestra.run.smithi181.stderr:71 bytes copied, 5.2313e-05 s, 1.4 MB/s 2024-09-17T03:50:02.313 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-17T03:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:02.321 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-17T03:50:02.367 INFO:tasks.cephfs_test_runner:Starting test: test_dir_deletion (tasks.cephfs.test_strays.TestStrays) That when deleting a bunch of dentries and the containing 2024-09-17T03:50:02.367 INFO:tasks.cephfs_test_runner:test_dir_deletion (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:50:02.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:02.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 -- 172.21.15.129:0/2251019920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c15c5d0 msgr2=0x7f2f7c0cad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 --2- 172.21.15.129:0/2251019920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c15c5d0 0x7f2f7c0cad30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f2f64005fd0 tx=0x7f2f6402a4b0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 -- 172.21.15.129:0/2251019920 shutdown_connections 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 --2- 172.21.15.129:0/2251019920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c0b4b80 0x7f2f7c0cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 --2- 172.21.15.129:0/2251019920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c15c5d0 0x7f2f7c0cad30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 -- 172.21.15.129:0/2251019920 >> 172.21.15.129:0/2251019920 conn(0x7f2f7c06d870 msgr2=0x7f2f7c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 -- 172.21.15.129:0/2251019920 shutdown_connections 2024-09-17T03:50:02.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.596+0000 7f2f7be00640 1 -- 172.21.15.129:0/2251019920 wait complete. 2024-09-17T03:50:02.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.598+0000 7f2f7be00640 1 Processor -- start 2024-09-17T03:50:02.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.599+0000 7f2f7be00640 1 -- start start 2024-09-17T03:50:02.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f7c0b4b80 0x7f2f7c14ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c15c5d0 0x7f2f7c1e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d0560 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d03e0 con 0x7f2f7c0b4b80 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f7be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d06e0 con 0x7f2f7c15c5d0 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f72000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f72000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40672/0 (socket says 172.21.15.129:40672) 2024-09-17T03:50:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.600+0000 7f2f72000640 1 -- 172.21.15.129:0/1829170856 learned_addr learned my addr 172.21.15.129:0/1829170856 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f7aa00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f7c0b4b80 0x7f2f7c14ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f7b400640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c15c5d0 0x7f2f7c1e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 -- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c15c5d0 msgr2=0x7f2f7c1e5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c15c5d0 0x7f2f7c1e5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 -- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f7c0b4b80 msgr2=0x7f2f7c14ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f7c0b4b80 0x7f2f7c14ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 -- 172.21.15.129:0/1829170856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c0b23f0 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f72000640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f64000c00 tx=0x7f2f64004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6403e030 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f64005040 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f64035040 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.601+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c159d80 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.602+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c15a340 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.603+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c0d0bf0 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.604+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2f64041030 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.605+0000 7f2f73400640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 0x7f2f5807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:02.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.605+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f2f640bb300 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.605+0000 7f2f7aa00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 0x7f2f5807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:02.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.606+0000 7f2f7aa00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 0x7f2f5807b0d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:02.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.610+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2f64083840 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.791+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f7c06eff0 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.792+0000 7f2f73400640 1 -- 172.21.15.129:0/1829170856 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7f2f64083140 con 0x7f2f7c0ca5a0 2024-09-17T03:50:02.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:02.793 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 msgr2=0x7f2f5807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 0x7f2f5807b0d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 msgr2=0x7f2f7c15b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2f64000c00 tx=0x7f2f64004c40 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 shutdown_connections 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f58078bc0 0x7f2f5807b0d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f7c15c5d0 0x7f2f7c1e5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f7c0ca5a0 0x7f2f7c15b6a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 --2- 172.21.15.129:0/1829170856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f7c0b4b80 0x7f2f7c14ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 >> 172.21.15.129:0/1829170856 conn(0x7f2f7c06d870 msgr2=0x7f2f7c0c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:02.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 shutdown_connections 2024-09-17T03:50:02.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:02.795+0000 7f2f7be00640 1 -- 172.21.15.129:0/1829170856 wait complete. 2024-09-17T03:50:02.820 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-17T03:49:33.632984+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-17T03:49:34.828244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:50:02.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:50:03.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 -- 172.21.15.129:0/2708530991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 msgr2=0x7fcc780b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 --2- 172.21.15.129:0/2708530991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc780b40b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcc78071d60 tx=0x7fcc6402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 -- 172.21.15.129:0/2708530991 shutdown_connections 2024-09-17T03:50:03.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 --2- 172.21.15.129:0/2708530991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc780b40b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 --2- 172.21.15.129:0/2708530991 >> v2:172.21.15.181:3300/0 conn(0x7fcc780cc1e0 0x7fcc780af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 -- 172.21.15.129:0/2708530991 >> 172.21.15.129:0/2708530991 conn(0x7fcc7806d7d0 msgr2=0x7fcc780b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:03.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 -- 172.21.15.129:0/2708530991 shutdown_connections 2024-09-17T03:50:03.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.046+0000 7fcc80600640 1 -- 172.21.15.129:0/2708530991 wait complete. 2024-09-17T03:50:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.048+0000 7fcc80600640 1 Processor -- start 2024-09-17T03:50:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.048+0000 7fcc80600640 1 -- start start 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 0x7fcc7809c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 0x7fcc781f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc78153520 con 0x7fcc7815d8a0 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcc781533a0 con 0x7fcc780ccb50 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc80600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcc781536a0 con 0x7fcc780cc1e0 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40690/0 (socket says 172.21.15.129:40690) 2024-09-17T03:50:03.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 -- 172.21.15.129:0/134268787 learned_addr learned my addr 172.21.15.129:0/134268787 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:03.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7d200640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 0x7fcc781f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7dc00640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 0x7fcc7809c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 -- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 msgr2=0x7fcc7809c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 0x7fcc7809c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 -- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 msgr2=0x7fcc781f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 0x7fcc781f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.050+0000 7fcc7e600640 1 -- 172.21.15.129:0/134268787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc7809cd50 con 0x7fcc7815d8a0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc7d200640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 0x7fcc781f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc7dc00640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 0x7fcc7809c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc7e600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c6c0 tx=0x7fcc7000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc70017070 con 0x7fcc7815d8a0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc70003c60 con 0x7fcc7815d8a0 2024-09-17T03:50:03.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc700052d0 con 0x7fcc7815d8a0 2024-09-17T03:50:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc781f9170 con 0x7fcc7815d8a0 2024-09-17T03:50:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.051+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc781f9730 con 0x7fcc7815d8a0 2024-09-17T03:50:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.052+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc781f9930 con 0x7fcc7815d8a0 2024-09-17T03:50:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.053+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcc7001a070 con 0x7fcc7815d8a0 2024-09-17T03:50:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.054+0000 7fcc75600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 0x7fcc5807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.055+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcc7009a360 con 0x7fcc7815d8a0 2024-09-17T03:50:03.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.055+0000 7fcc7dc00640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 0x7fcc5807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.056+0000 7fcc7dc00640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 0x7fcc5807b0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ab0 tx=0x7fcc6c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.059+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcc781f9930 con 0x7fcc7815d8a0 2024-09-17T03:50:03.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.241+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcc78153b50 con 0x7fcc7815d8a0 2024-09-17T03:50:03.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.241+0000 7fcc75600640 1 -- 172.21.15.129:0/134268787 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fcc70062790 con 0x7fcc7815d8a0 2024-09-17T03:50:03.241 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:50:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 msgr2=0x7fcc5807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 0x7fcc5807b0a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009ab0 tx=0x7fcc6c005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:03.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 msgr2=0x7fcc781ef1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000c6c0 tx=0x7fcc7000cb80 comp rx=0 tx=0).stop 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 shutdown_connections 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc58078b90 0x7fcc5807b0a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc7815d8a0 0x7fcc781ef1d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc780ccb50 0x7fcc781f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 --2- 172.21.15.129:0/134268787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc780cc1e0 0x7fcc7809c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 >> 172.21.15.129:0/134268787 conn(0x7fcc7806d7d0 msgr2=0x7fcc780b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 shutdown_connections 2024-09-17T03:50:03.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.244+0000 7fcc80600640 1 -- 172.21.15.129:0/134268787 wait complete. 2024-09-17T03:50:03.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 -- 172.21.15.129:0/1505224595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff328003a50 msgr2=0x7ff328003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 --2- 172.21.15.129:0/1505224595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff328003a50 0x7ff328003ec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff334009b00 tx=0x7ff328005830 comp rx=0 tx=0).stop 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 -- 172.21.15.129:0/1505224595 shutdown_connections 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 --2- 172.21.15.129:0/1505224595 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff328003a50 0x7ff328003ec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 -- 172.21.15.129:0/1505224595 >> 172.21.15.129:0/1505224595 conn(0x7ff3440b1cb0 msgr2=0x7ff3440b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 -- 172.21.15.129:0/1505224595 shutdown_connections 2024-09-17T03:50:03.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.492+0000 7ff349200640 1 -- 172.21.15.129:0/1505224595 wait complete. 2024-09-17T03:50:03.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.494+0000 7ff349200640 1 Processor -- start 2024-09-17T03:50:03.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.494+0000 7ff349200640 1 -- start start 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff328003a50 0x7ff3441f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34409ca50 0x7ff34409ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3440cbd20 con 0x7ff344145700 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff34406ffa0 con 0x7ff34409ca50 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff349200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff344145bf0 con 0x7ff328003a50 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40712/0 (socket says 172.21.15.129:40712) 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 -- 172.21.15.129:0/4017704687 learned_addr learned my addr 172.21.15.129:0/4017704687 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341e00640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff328003a50 0x7ff3441f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff342800640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34409ca50 0x7ff34409ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 -- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff328003a50 msgr2=0x7ff3441f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff328003a50 0x7ff3441f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 -- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34409ca50 msgr2=0x7ff34409ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34409ca50 0x7ff34409ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.496+0000 7ff341400640 1 -- 172.21.15.129:0/4017704687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3440b3a80 con 0x7ff344145700 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff341400640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff330004d60 tx=0x7ff330001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3300107d0 con 0x7ff344145700 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff330016070 con 0x7ff344145700 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34415d140 con 0x7ff344145700 2024-09-17T03:50:03.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330010d00 con 0x7ff344145700 2024-09-17T03:50:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.497+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff34415d700 con 0x7ff344145700 2024-09-17T03:50:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.498+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff344070500 con 0x7ff344145700 2024-09-17T03:50:03.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.499+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff33001b460 con 0x7ff344145700 2024-09-17T03:50:03.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.500+0000 7ff33a000640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 0x7ff32407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.501+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff33009b510 con 0x7ff344145700 2024-09-17T03:50:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.501+0000 7ff341e00640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 0x7ff32407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.502+0000 7ff341e00640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 0x7ff32407b0a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff334035000 tx=0x7ff33402d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.505+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff330063940 con 0x7ff344145700 2024-09-17T03:50:03.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.687+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff3440cb320 con 0x7ff344145700 2024-09-17T03:50:03.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.687+0000 7ff33a000640 1 -- 172.21.15.129:0/4017704687 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7ff330063240 con 0x7ff344145700 2024-09-17T03:50:03.687 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 32 2024-09-17T03:50:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 msgr2=0x7ff32407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 0x7ff32407b0a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff334035000 tx=0x7ff33402d040 comp rx=0 tx=0).stop 2024-09-17T03:50:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 msgr2=0x7ff3441f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff330004d60 tx=0x7ff330001630 comp rx=0 tx=0).stop 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 shutdown_connections 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff324078b90 0x7ff32407b0a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff34409ca50 0x7ff34409ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff344145700 0x7ff3441f3180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 --2- 172.21.15.129:0/4017704687 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff328003a50 0x7ff3441f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 >> 172.21.15.129:0/4017704687 conn(0x7ff3440b1cb0 msgr2=0x7ff344073f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 shutdown_connections 2024-09-17T03:50:03.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.690+0000 7ff349200640 1 -- 172.21.15.129:0/4017704687 wait complete. 2024-09-17T03:50:03.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-17T03:50:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 -- 172.21.15.129:0/3967052327 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234415ca50 msgr2=0x7f23440cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 --2- 172.21.15.129:0/3967052327 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234415ca50 0x7f23440cc830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2338009860 tx=0x7f233802a510 comp rx=0 tx=0).stop 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 -- 172.21.15.129:0/3967052327 shutdown_connections 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 --2- 172.21.15.129:0/3967052327 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234415ca50 0x7f23440cc830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 -- 172.21.15.129:0/3967052327 >> 172.21.15.129:0/3967052327 conn(0x7f234406d410 msgr2=0x7f23440b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 -- 172.21.15.129:0/3967052327 shutdown_connections 2024-09-17T03:50:03.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.940+0000 7f234a600640 1 -- 172.21.15.129:0/3967052327 wait complete. 2024-09-17T03:50:03.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.942+0000 7f234a600640 1 Processor -- start 2024-09-17T03:50:03.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.943+0000 7f234a600640 1 -- start start 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23440cf540 0x7f234413cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234413d880 0x7f23440b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f234415e1a0 con 0x7f234415ca50 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f234415e020 con 0x7f23440cf540 2024-09-17T03:50:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f234a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f234415e320 con 0x7f234413d880 2024-09-17T03:50:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f2343e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f2343e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59272/0 (socket says 172.21.15.129:59272) 2024-09-17T03:50:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f2343e00640 1 -- 172.21.15.129:0/702239301 learned_addr learned my addr 172.21.15.129:0/702239301 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f2349c00640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234413d880 0x7f23440b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.944+0000 7f2349200640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23440cf540 0x7f234413cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 -- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234413d880 msgr2=0x7f23440b4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234413d880 0x7f23440b4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 -- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23440cf540 msgr2=0x7f234413cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23440cf540 0x7f234413cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 -- 172.21.15.129:0/702239301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234413d560 con 0x7f234415ca50 2024-09-17T03:50:03.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2343e00640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f233400c370 tx=0x7f233400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334011790 con 0x7f234415ca50 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2334017070 con 0x7f234415ca50 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334011ce0 con 0x7f234415ca50 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23440b4ed0 con 0x7f234415ca50 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.945+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2344145ec0 con 0x7f234415ca50 2024-09-17T03:50:03.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.947+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23440cad70 con 0x7f234415ca50 2024-09-17T03:50:03.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.948+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f233401c460 con 0x7f234415ca50 2024-09-17T03:50:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.949+0000 7f2340c00640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 0x7f232807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.949+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f233409a810 con 0x7f234415ca50 2024-09-17T03:50:03.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.949+0000 7f2349200640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 0x7f232807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:03.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.950+0000 7f2349200640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 0x7f232807b0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2338034000 tx=0x7f2338004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:03.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:03.953+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2334062720 con 0x7f234415ca50 2024-09-17T03:50:04.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.136+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f23440cc830 con 0x7f234415ca50 2024-09-17T03:50:04.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.137+0000 7f2340c00640 1 -- 172.21.15.129:0/702239301 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f2334067150 con 0x7f234415ca50 2024-09-17T03:50:04.137 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 msgr2=0x7f232807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 0x7f232807b0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2338034000 tx=0x7f2338004730 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 msgr2=0x7f23440b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f233400c370 tx=0x7f233400c830 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 shutdown_connections 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2328078bc0 0x7f232807b0d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234413d880 0x7f23440b4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234415ca50 0x7f23440b40e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 --2- 172.21.15.129:0/702239301 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23440cf540 0x7f234413cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.139+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 >> 172.21.15.129:0/702239301 conn(0x7f234406d410 msgr2=0x7f2344073650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.140+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 shutdown_connections 2024-09-17T03:50:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.140+0000 7f234a600640 1 -- 172.21.15.129:0/702239301 wait complete. 2024-09-17T03:50:04.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-09-17T03:50:04.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 -- 172.21.15.129:0/2073467218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 msgr2=0x7fc62806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 --2- 172.21.15.129:0/2073467218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc62806dcd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6281631f0 tx=0x7fc61c02a560 comp rx=0 tx=0).stop 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 -- 172.21.15.129:0/2073467218 shutdown_connections 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 --2- 172.21.15.129:0/2073467218 >> v2:172.21.15.181:3300/0 conn(0x7fc628074040 0x7fc62815f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 --2- 172.21.15.129:0/2073467218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc62806dcd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 -- 172.21.15.129:0/2073467218 >> 172.21.15.129:0/2073467218 conn(0x7fc6280cd6c0 msgr2=0x7fc6280cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 -- 172.21.15.129:0/2073467218 shutdown_connections 2024-09-17T03:50:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.385+0000 7fc62ea00640 1 -- 172.21.15.129:0/2073467218 wait complete. 2024-09-17T03:50:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.387+0000 7fc62ea00640 1 Processor -- start 2024-09-17T03:50:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.388+0000 7fc62ea00640 1 -- start start 2024-09-17T03:50:04.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:04.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 0x7fc6281e5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 0x7fc6281e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc62806faa0 con 0x7fc6280b5e10 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc62806f920 con 0x7fc6281530f0 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62ea00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc62806fc20 con 0x7fc6280ce3d0 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59302/0 (socket says 172.21.15.129:59302) 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.389+0000 7fc62d600640 1 -- 172.21.15.129:0/13036633 learned_addr learned my addr 172.21.15.129:0/13036633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62e000640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 0x7fc6281e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 -- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 msgr2=0x7fc6281e5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62cc00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 0x7fc6281e5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 0x7fc6281e5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 -- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 msgr2=0x7fc6281e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 0x7fc6281e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 -- 172.21.15.129:0/13036633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6281e58c0 con 0x7fc6280b5e10 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62e000640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 0x7fc6281e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62cc00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 0x7fc6281e5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62d600640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc61c034040 tx=0x7fc61c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c03e030 con 0x7fc6280b5e10 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc61c0047a0 con 0x7fc6280b5e10 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6281f9ac0 con 0x7fc6280b5e10 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc61c035040 con 0x7fc6280b5e10 2024-09-17T03:50:04.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.390+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc62815e710 con 0x7fc6280b5e10 2024-09-17T03:50:04.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.392+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6280cfb60 con 0x7fc6280b5e10 2024-09-17T03:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.393+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc61c041030 con 0x7fc6280b5e10 2024-09-17T03:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.394+0000 7fc625600640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 0x7fc60807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.394+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc61c0bb710 con 0x7fc6280b5e10 2024-09-17T03:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.394+0000 7fc62cc00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 0x7fc60807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.395+0000 7fc62cc00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 0x7fc60807b0a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc614000f60 tx=0x7fc614009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:04.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.398+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc6280cfb60 con 0x7fc6280b5e10 2024-09-17T03:50:04.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:04.654+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) -- 0x7fc62806a740 con 0x7fc6280b5e10 2024-09-17T03:50:05.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.515+0000 7fc625600640 1 -- 172.21.15.129:0/13036633 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v51) ==== 135+0+0 (secure 0 0 0) 0x7fc62806a740 con 0x7fc6280b5e10 2024-09-17T03:50:05.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 msgr2=0x7fc60807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 0x7fc60807b0a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc614000f60 tx=0x7fc614009040 comp rx=0 tx=0).stop 2024-09-17T03:50:05.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 msgr2=0x7fc628147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc61c034040 tx=0x7fc61c0039e0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 shutdown_connections 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc608078b90 0x7fc60807b0a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6281530f0 0x7fc6281e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6280ce3d0 0x7fc6281e5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 --2- 172.21.15.129:0/13036633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6280b5e10 0x7fc628147200 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 >> 172.21.15.129:0/13036633 conn(0x7fc6280cd6c0 msgr2=0x7fc628155390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 shutdown_connections 2024-09-17T03:50:05.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.518+0000 7fc62ea00640 1 -- 172.21.15.129:0/13036633 wait complete. 2024-09-17T03:50:05.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:50:05.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 -- 172.21.15.129:0/2699939424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec0cd080 msgr2=0x7f06ec06ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 --2- 172.21.15.129:0/2699939424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec0cd080 0x7f06ec06ee90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06e4005fd0 tx=0x7f06e402a780 comp rx=0 tx=0).stop 2024-09-17T03:50:05.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 -- 172.21.15.129:0/2699939424 shutdown_connections 2024-09-17T03:50:05.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 --2- 172.21.15.129:0/2699939424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec0cd080 0x7f06ec06ee90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 --2- 172.21.15.129:0/2699939424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 0x7f06ec0cfae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 -- 172.21.15.129:0/2699939424 >> 172.21.15.129:0/2699939424 conn(0x7f06ec06d7d0 msgr2=0x7f06ec0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:05.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 -- 172.21.15.129:0/2699939424 shutdown_connections 2024-09-17T03:50:05.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.765+0000 7f06f0a00640 1 -- 172.21.15.129:0/2699939424 wait complete. 2024-09-17T03:50:05.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.767+0000 7f06f0a00640 1 Processor -- start 2024-09-17T03:50:05.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.767+0000 7f06f0a00640 1 -- start start 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 0x7f06ec15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec15ef10 0x7f06ec15a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec070f20 con 0x7f06ec1f7270 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f06ec15f4b0 con 0x7f06ec15ef10 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06f0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f06ec0710a0 con 0x7f06ec15a9b0 2024-09-17T03:50:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:05.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59308/0 (socket says 172.21.15.129:59308) 2024-09-17T03:50:05.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 -- 172.21.15.129:0/3913873143 learned_addr learned my addr 172.21.15.129:0/3913873143 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:05.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06e9e00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 0x7f06ec15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:05.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06e9400640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec15ef10 0x7f06ec15a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 -- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 msgr2=0x7f06ec15b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 0x7f06ec15b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 -- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec15ef10 msgr2=0x7f06ec15a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.769+0000 7f06ea800640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec15ef10 0x7f06ec15a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06ea800640 1 -- 172.21.15.129:0/3913873143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06ec1f7bc0 con 0x7f06ec1f7270 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06ea800640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009cd0 tx=0x7f06e4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4040070 con 0x7f06ec1f7270 2024-09-17T03:50:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f06e4004560 con 0x7f06ec1f7270 2024-09-17T03:50:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e4003e10 con 0x7f06ec1f7270 2024-09-17T03:50:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06ec15cbe0 con 0x7f06ec1f7270 2024-09-17T03:50:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.770+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06ec09b8e0 con 0x7f06ec1f7270 2024-09-17T03:50:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.771+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06ec152520 con 0x7f06ec1f7270 2024-09-17T03:50:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.772+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f06e4043070 con 0x7f06ec1f7270 2024-09-17T03:50:05.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.774+0000 7f06e2000640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 0x7f06cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:05.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.774+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f06e40bbff0 con 0x7f06ec1f7270 2024-09-17T03:50:05.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.774+0000 7f06e9e00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 0x7f06cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:05.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.775+0000 7f06e9e00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 0x7f06cc07b0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0095f0 tx=0x7f06dc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:05.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.778+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f06ec1f7bc0 con 0x7f06ec1f7270 2024-09-17T03:50:05.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.961+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f06ec0af9c0 con 0x7f06ec1f7270 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.962+0000 7f06e2000640 1 -- 172.21.15.129:0/3913873143 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v31) ==== 59+0+2294 (secure 0 0 0) 0x7f06e4083d40 con 0x7f06ec1f7270 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:50:05.962 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:50:05.963 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:50:05.964 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:50:05.965 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:50:05.965 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 msgr2=0x7f06cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 0x7f06cc07b0a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0095f0 tx=0x7f06dc005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 msgr2=0x7f06ec1f7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009cd0 tx=0x7f06e4003c20 comp rx=0 tx=0).stop 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 shutdown_connections 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06cc078b90 0x7f06cc07b0a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06ec1f7270 0x7f06ec1f7680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06ec15ef10 0x7f06ec15a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 --2- 172.21.15.129:0/3913873143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06ec15a9b0 0x7f06ec15b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:05.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 >> 172.21.15.129:0/3913873143 conn(0x7f06ec06d7d0 msgr2=0x7f06ec0afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:05.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 shutdown_connections 2024-09-17T03:50:05.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:05.964+0000 7f06f0a00640 1 -- 172.21.15.129:0/3913873143 wait complete. 2024-09-17T03:50:05.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:50:06.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 -- 172.21.15.129:0/2364835567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 msgr2=0x7ff3e0074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:06.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 --2- 172.21.15.129:0/2364835567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e0074b80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff3e00710f0 tx=0x7ff3d802aac0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 -- 172.21.15.129:0/2364835567 shutdown_connections 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 --2- 172.21.15.129:0/2364835567 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e0074b80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 --2- 172.21.15.129:0/2364835567 >> v2:172.21.15.181:3300/0 conn(0x7ff3e0070030 0x7ff3e0074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 -- 172.21.15.129:0/2364835567 >> 172.21.15.129:0/2364835567 conn(0x7ff3e00b1cb0 msgr2=0x7ff3e00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 -- 172.21.15.129:0/2364835567 shutdown_connections 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.213+0000 7ff3e6400640 1 -- 172.21.15.129:0/2364835567 wait complete. 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.215+0000 7ff3e6400640 1 Processor -- start 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.215+0000 7ff3e6400640 1 -- start start 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 0x7ff3e015e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:06.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 0x7ff3e015ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e00b39a0 con 0x7ff3e0070440 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3e00b3820 con 0x7ff3e0070030 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3e6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff3e00b3b20 con 0x7ff3e0073170 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59324/0 (socket says 172.21.15.129:59324) 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 -- 172.21.15.129:0/445021439 learned_addr learned my addr 172.21.15.129:0/445021439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 -- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 msgr2=0x7ff3e015ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3dfe00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 0x7ff3e015ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3df400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 0x7ff3e015e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 0x7ff3e015ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 -- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 msgr2=0x7ff3e015e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 0x7ff3e015e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3d6a00640 1 -- 172.21.15.129:0/445021439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3e01e8410 con 0x7ff3e0070440 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3dfe00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 0x7ff3e015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.217+0000 7ff3df400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 0x7ff3e015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3d6a00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800d740 tx=0x7ff3c800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8017070 con 0x7ff3e0070440 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3c800de50 con 0x7ff3e0070440 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80099d0 con 0x7ff3e0070440 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3e01e86f0 con 0x7ff3e0070440 2024-09-17T03:50:06.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.218+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3e01e8cb0 con 0x7ff3e0070440 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.219+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3e01ea2e0 con 0x7ff3e0070440 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.220+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff3c8009b70 con 0x7ff3e0070440 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.221+0000 7ff3d7e00640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 0x7ff3bc07b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.222+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff3c809b180 con 0x7ff3e0070440 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.222+0000 7ff3df400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 0x7ff3bc07b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:06.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.222+0000 7ff3df400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 0x7ff3bc07b100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000f60 tx=0x7ff3d0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:06.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.226+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff3c8020020 con 0x7ff3e0070440 2024-09-17T03:50:06.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.410+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff3e00b4040 con 0x7ff3e0070440 2024-09-17T03:50:06.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.415+0000 7ff3d7e00640 1 -- 172.21.15.129:0/445021439 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v32)=0 v32) ==== 122+0+0 (secure 0 0 0) 0x7ff3c8063520 con 0x7ff3e0070440 2024-09-17T03:50:06.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 msgr2=0x7ff3bc07b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:06.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 0x7ff3bc07b100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0000f60 tx=0x7ff3d0009040 comp rx=0 tx=0).stop 2024-09-17T03:50:06.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 msgr2=0x7ff3e015a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:06.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800d740 tx=0x7ff3c800dc00 comp rx=0 tx=0).stop 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 shutdown_connections 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3bc078bf0 0x7ff3bc07b100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3e0073170 0x7ff3e015ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3e0070440 0x7ff3e015a5f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 --2- 172.21.15.129:0/445021439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3e0070030 0x7ff3e015e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 >> 172.21.15.129:0/445021439 conn(0x7ff3e00b1cb0 msgr2=0x7ff3e00adf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:06.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 shutdown_connections 2024-09-17T03:50:06.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:06.417+0000 7ff3e6400640 1 -- 172.21.15.129:0/445021439 wait complete. 2024-09-17T03:50:06.439 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:50:06.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:50:06.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:50:06.620 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:50:06.621 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:06.621 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:06.656 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:06.657 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:06.732 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:06.732 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:06.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 -- 172.21.15.129:0/3515253370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 msgr2=0x7f6c20005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 --2- 172.21.15.129:0/3515253370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c20005150 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005fd0 tx=0x7f6c20005830 comp rx=0 tx=0).stop 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c3be00640 1 -- 172.21.15.129:0/3515253370 reap_dead start 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 -- 172.21.15.129:0/3515253370 shutdown_connections 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 -- 172.21.15.129:0/3515253370 >> 172.21.15.129:0/3515253370 conn(0x7f6c3c06d7d0 msgr2=0x7f6c3c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 -- 172.21.15.129:0/3515253370 shutdown_connections 2024-09-17T03:50:07.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.068+0000 7f6c42800640 1 -- 172.21.15.129:0/3515253370 wait complete. 2024-09-17T03:50:07.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.071+0000 7f6c42800640 1 Processor -- start 2024-09-17T03:50:07.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.071+0000 7f6c42800640 1 -- start start 2024-09-17T03:50:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c3c1f6320 0x7f6c3c1f8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c3c1f9160 0x7f6c3c15b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0b4770 con 0x7f6c20004d40 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0b45f0 con 0x7f6c3c1f6320 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c42800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c0ccf60 con 0x7f6c3c1f9160 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c3b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c3b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59342/0 (socket says 172.21.15.129:59342) 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.072+0000 7f6c3b400640 1 -- 172.21.15.129:0/2558047672 learned_addr learned my addr 172.21.15.129:0/2558047672 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3aa00640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c3c1f6320 0x7f6c3c1f8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3be00640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c3c1f9160 0x7f6c3c15b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 -- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c3c1f9160 msgr2=0x7f6c3c15b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c3c1f9160 0x7f6c3c15b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 -- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c3c1f6320 msgr2=0x7f6c3c1f8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c3c1f6320 0x7f6c3c1f8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 -- 172.21.15.129:0/2558047672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c1f98e0 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c3b400640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c006d80 tx=0x7f6c2c0049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c015020 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c2c011960 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c011db0 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.073+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c1f5840 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.074+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c1f5e00 con 0x7f6c20004d40 2024-09-17T03:50:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.075+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c3c1f5630 con 0x7f6c20004d40 2024-09-17T03:50:07.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.076+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c2c018020 con 0x7f6c20004d40 2024-09-17T03:50:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.077+0000 7f6c33400640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 0x7f6c1c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.078+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f6c2c09b7c0 con 0x7f6c20004d40 2024-09-17T03:50:07.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.078+0000 7f6c3aa00640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 0x7f6c1c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.078+0000 7f6c3aa00640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 0x7f6c1c07b0a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0703a0 tx=0x7f6c28005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:07.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.082+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c2c01f020 con 0x7f6c20004d40 2024-09-17T03:50:07.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.276+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6c3c0b40b0 con 0x7f6c20004d40 2024-09-17T03:50:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.277+0000 7f6c33400640 1 -- 172.21.15.129:0/2558047672 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f6c3c0cd4a0 con 0x7f6c20004d40 2024-09-17T03:50:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 msgr2=0x7f6c1c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 0x7f6c1c07b0a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0703a0 tx=0x7f6c28005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 msgr2=0x7f6c3c148b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c006d80 tx=0x7f6c2c0049d0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 shutdown_connections 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c1c078b90 0x7f6c1c07b0a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c3c1f9160 0x7f6c3c15b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.279+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c3c1f6320 0x7f6c3c1f8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.280+0000 7f6c42800640 1 --2- 172.21.15.129:0/2558047672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c20004d40 0x7f6c3c148b50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.280+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 >> 172.21.15.129:0/2558047672 conn(0x7f6c3c06d7d0 msgr2=0x7f6c3c0cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.280+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 shutdown_connections 2024-09-17T03:50:07.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.280+0000 7f6c42800640 1 -- 172.21.15.129:0/2558047672 wait complete. 2024-09-17T03:50:07.304 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:50:07.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:07.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-17T03:50:07.365 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:50:07.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:07.365 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:50:07.423 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:07.423 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:07.463 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:07.464 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:07.541 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:07.542 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:07.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:50:07.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 -- 172.21.15.129:0/4169903951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 msgr2=0x7fef6400b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 --2- 172.21.15.129:0/4169903951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef6400b9e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fef78005fd0 tx=0x7fef64005830 comp rx=0 tx=0).stop 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 -- 172.21.15.129:0/4169903951 shutdown_connections 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 --2- 172.21.15.129:0/4169903951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef6400b9e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 -- 172.21.15.129:0/4169903951 >> 172.21.15.129:0/4169903951 conn(0x7fef8006d410 msgr2=0x7fef800b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 -- 172.21.15.129:0/4169903951 shutdown_connections 2024-09-17T03:50:07.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.842+0000 7fef85000640 1 -- 172.21.15.129:0/4169903951 wait complete. 2024-09-17T03:50:07.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.844+0000 7fef85000640 1 Processor -- start 2024-09-17T03:50:07.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.845+0000 7fef85000640 1 -- start start 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 0x7fef801f86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 0x7fef8013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef800ccbc0 con 0x7fef64005320 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef800cca40 con 0x7fef80144b30 2024-09-17T03:50:07.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef85000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fef800ccd40 con 0x7fef801f8c00 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59350/0 (socket says 172.21.15.129:59350) 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.846+0000 7fef7f400640 1 -- 172.21.15.129:0/324014658 learned_addr learned my addr 172.21.15.129:0/324014658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7ea00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 0x7fef801f86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7fe00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 0x7fef8013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 -- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 msgr2=0x7fef8013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 0x7fef8013e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 -- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 msgr2=0x7fef801f86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 0x7fef801f86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 -- 172.21.15.129:0/324014658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef801f9420 con 0x7fef64005320 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7ea00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 0x7fef801f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:07.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7fe00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 0x7fef8013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef7f400640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fef70001240 tx=0x7fef700039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70015070 con 0x7fef64005320 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef70011860 con 0x7fef64005320 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef70011cb0 con 0x7fef64005320 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef801f9140 con 0x7fef64005320 2024-09-17T03:50:07.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.847+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef80156800 con 0x7fef64005320 2024-09-17T03:50:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.848+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8006f0a0 con 0x7fef64005320 2024-09-17T03:50:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.850+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef70018070 con 0x7fef64005320 2024-09-17T03:50:07.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.850+0000 7fef77400640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 0x7fef6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.851+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fef7009b9d0 con 0x7fef64005320 2024-09-17T03:50:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.851+0000 7fef7ea00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 0x7fef6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:07.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.852+0000 7fef7ea00640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 0x7fef6007b0a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:07.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:07.855+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef70063d80 con 0x7fef64005320 2024-09-17T03:50:08.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.048+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fef8006fb50 con 0x7fef64005320 2024-09-17T03:50:08.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.048+0000 7fef77400640 1 -- 172.21.15.129:0/324014658 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7fef70063680 con 0x7fef64005320 2024-09-17T03:50:08.048 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 msgr2=0x7fef6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 0x7fef6007b0a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fef6c000c00 tx=0x7fef6c006b10 comp rx=0 tx=0).stop 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 msgr2=0x7fef80159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fef70001240 tx=0x7fef700039c0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 shutdown_connections 2024-09-17T03:50:08.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef60078b90 0x7fef6007b0a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef801f8c00 0x7fef8013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef80144b30 0x7fef801f86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.050+0000 7fef85000640 1 --2- 172.21.15.129:0/324014658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef64005320 0x7fef80159af0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.051+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 >> 172.21.15.129:0/324014658 conn(0x7fef8006d410 msgr2=0x7fef800b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.051+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 shutdown_connections 2024-09-17T03:50:08.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.051+0000 7fef85000640 1 -- 172.21.15.129:0/324014658 wait complete. 2024-09-17T03:50:08.072 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-17T03:50:08.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:08.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-09-17T03:50:08.135 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:50:08.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:08.135 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-17T03:50:08.182 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:08.182 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:08.218 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:08.218 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:08.292 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:08.293 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:08.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:50:08.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 -- 172.21.15.129:0/3682501141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 msgr2=0x7efdbc0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 --2- 172.21.15.129:0/3682501141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc0b40b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7efdbc071d60 tx=0x7efda402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 -- 172.21.15.129:0/3682501141 shutdown_connections 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 --2- 172.21.15.129:0/3682501141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc0b40b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 --2- 172.21.15.129:0/3682501141 >> v2:172.21.15.181:3300/0 conn(0x7efdbc0cc1e0 0x7efdbc0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 -- 172.21.15.129:0/3682501141 >> 172.21.15.129:0/3682501141 conn(0x7efdbc06d7d0 msgr2=0x7efdbc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 -- 172.21.15.129:0/3682501141 shutdown_connections 2024-09-17T03:50:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.632+0000 7efdc0c00640 1 -- 172.21.15.129:0/3682501141 wait complete. 2024-09-17T03:50:08.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.635+0000 7efdc0c00640 1 Processor -- start 2024-09-17T03:50:08.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.635+0000 7efdc0c00640 1 -- start start 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 0x7efdbc09d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc0ccb50 0x7efdbc1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc153520 con 0x7efdbc15d8a0 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc1533a0 con 0x7efdbc0ccb50 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.636+0000 7efdc0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efdbc1536a0 con 0x7efdbc0cc1e0 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59378/0 (socket says 172.21.15.129:59378) 2024-09-17T03:50:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 -- 172.21.15.129:0/2434076604 learned_addr learned my addr 172.21.15.129:0/2434076604 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdb3e00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc0ccb50 0x7efdbc1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdb9e00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 0x7efdbc09d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 -- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 msgr2=0x7efdbc09d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 0x7efdbc09d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 -- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc0ccb50 msgr2=0x7efdbc1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc0ccb50 0x7efdbc1525c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 -- 172.21.15.129:0/2434076604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc1f5a80 con 0x7efdbc15d8a0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdb9e00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 0x7efdbc09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.637+0000 7efdba800640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c6b0 tx=0x7efdb400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.638+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4003b70 con 0x7efdbc15d8a0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.638+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdb4003d30 con 0x7efdbc15d8a0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.638+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4005330 con 0x7efdbc15d8a0 2024-09-17T03:50:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.638+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc1f5fb0 con 0x7efdbc15d8a0 2024-09-17T03:50:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.638+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdbc1f6570 con 0x7efdbc15d8a0 2024-09-17T03:50:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.639+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdbc153b50 con 0x7efdbc15d8a0 2024-09-17T03:50:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.640+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efdb40054d0 con 0x7efdbc15d8a0 2024-09-17T03:50:08.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.641+0000 7efdb2000640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 0x7efd9807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:08.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.641+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7efdb409ab00 con 0x7efdbc15d8a0 2024-09-17T03:50:08.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.641+0000 7efdb9e00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 0x7efd9807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:08.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.642+0000 7efdb9e00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 0x7efd9807b0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efdac009b80 tx=0x7efdac005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:08.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.645+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efdb4062eb0 con 0x7efdbc15d8a0 2024-09-17T03:50:08.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.840+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efdbc1f5d40 con 0x7efdbc15d8a0 2024-09-17T03:50:08.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.841+0000 7efdb2000640 1 -- 172.21.15.129:0/2434076604 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7efdb40627b0 con 0x7efdbc15d8a0 2024-09-17T03:50:08.841 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:50:08.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 msgr2=0x7efd9807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 0x7efd9807b0a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efdac009b80 tx=0x7efdac005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 msgr2=0x7efdbc157d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efdb400c6b0 tx=0x7efdb400cb70 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 shutdown_connections 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd98078b90 0x7efd9807b0a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efdbc15d8a0 0x7efdbc157d60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efdbc0ccb50 0x7efdbc1525c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 --2- 172.21.15.129:0/2434076604 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efdbc0cc1e0 0x7efdbc09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 >> 172.21.15.129:0/2434076604 conn(0x7efdbc06d7d0 msgr2=0x7efdbc06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 shutdown_connections 2024-09-17T03:50:08.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:08.843+0000 7efdc0c00640 1 -- 172.21.15.129:0/2434076604 wait complete. 2024-09-17T03:50:08.866 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-09-17T03:50:08.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:08.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-09-17T03:50:08.926 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:50:08.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:08.926 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-17T03:50:08.962 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:08.962 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:08.998 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:08.998 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:09.074 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:09.074 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:09.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:50:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 -- 172.21.15.129:0/1239396849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180af700 msgr2=0x7f4b180caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 --2- 172.21.15.129:0/1239396849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180af700 0x7f4b180caa80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4b08009830 tx=0x7f4b0802a5b0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 -- 172.21.15.129:0/1239396849 shutdown_connections 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 --2- 172.21.15.129:0/1239396849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180af700 0x7f4b180caa80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 -- 172.21.15.129:0/1239396849 >> 172.21.15.129:0/1239396849 conn(0x7f4b1806d870 msgr2=0x7f4b1806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 -- 172.21.15.129:0/1239396849 shutdown_connections 2024-09-17T03:50:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.371+0000 7f4b17200640 1 -- 172.21.15.129:0/1239396849 wait complete. 2024-09-17T03:50:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.374+0000 7f4b17200640 1 Processor -- start 2024-09-17T03:50:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.374+0000 7f4b17200640 1 -- start start 2024-09-17T03:50:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180735d0 0x7f4b18156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b180af700 0x7f4b181565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1815d4b0 con 0x7f4b180b54d0 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1815d330 con 0x7f4b180735d0 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b17200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1815d630 con 0x7f4b180af700 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b16800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b16800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59388/0 (socket says 172.21.15.129:59388) 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.375+0000 7f4b16800640 1 -- 172.21.15.129:0/3952091743 learned_addr learned my addr 172.21.15.129:0/3952091743 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b15400640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b180af700 0x7f4b181565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b15e00640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180735d0 0x7f4b18156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 -- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b180af700 msgr2=0x7f4b181565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b180af700 0x7f4b181565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 -- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180735d0 msgr2=0x7f4b18156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180735d0 0x7f4b18156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 -- 172.21.15.129:0/3952091743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b181f5b90 con 0x7f4b180b54d0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b16800640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000cad0 tx=0x7f4b1000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10016020 con 0x7f4b180b54d0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b1000ac60 con 0x7f4b180b54d0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.376+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b181f5e70 con 0x7f4b180b54d0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.377+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b10012680 con 0x7f4b180b54d0 2024-09-17T03:50:09.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.377+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b181f6430 con 0x7f4b180b54d0 2024-09-17T03:50:09.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.378+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b181f5900 con 0x7f4b180b54d0 2024-09-17T03:50:09.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.379+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4b10019020 con 0x7f4b180b54d0 2024-09-17T03:50:09.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.380+0000 7f4b0e000640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 0x7f4af807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.380+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f4b1009b230 con 0x7f4b180b54d0 2024-09-17T03:50:09.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.380+0000 7f4b15e00640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 0x7f4af807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.381+0000 7f4b15e00640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 0x7f4af807b0a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b08034000 tx=0x7f4b08004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:09.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.385+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4b10020020 con 0x7f4b180b54d0 2024-09-17T03:50:09.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.578+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4b180caa80 con 0x7f4b180b54d0 2024-09-17T03:50:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.579+0000 7f4b0e000640 1 -- 172.21.15.129:0/3952091743 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f4b10063010 con 0x7f4b180b54d0 2024-09-17T03:50:09.579 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.581+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 msgr2=0x7f4af807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.581+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 0x7f4af807b0a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b08034000 tx=0x7f4b08004730 comp rx=0 tx=0).stop 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.581+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 msgr2=0x7f4b18154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.581+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000cad0 tx=0x7f4b1000cf90 comp rx=0 tx=0).stop 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 shutdown_connections 2024-09-17T03:50:09.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4af8078b90 0x7f4af807b0a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b180b54d0 0x7f4b18154060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b180af700 0x7f4b181565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 --2- 172.21.15.129:0/3952091743 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b180735d0 0x7f4b18156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 >> 172.21.15.129:0/3952091743 conn(0x7f4b1806d870 msgr2=0x7f4b1815e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 shutdown_connections 2024-09-17T03:50:09.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.582+0000 7f4b17200640 1 -- 172.21.15.129:0/3952091743 wait complete. 2024-09-17T03:50:09.606 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-09-17T03:50:09.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:09.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-09-17T03:50:09.667 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T03:50:09.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:09.667 DEBUG:teuthology.orchestra.run.smithi181:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-17T03:50:09.703 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:50:09.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:09.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 -- 172.21.15.129:0/964625632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 msgr2=0x7f864c167260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 --2- 172.21.15.129:0/964625632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c167260 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8644005fd0 tx=0x7f864402a820 comp rx=0 tx=0).stop 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 -- 172.21.15.129:0/964625632 shutdown_connections 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 --2- 172.21.15.129:0/964625632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c167260 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 -- 172.21.15.129:0/964625632 >> 172.21.15.129:0/964625632 conn(0x7f864c06d870 msgr2=0x7f864c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.922+0000 7f8652e00640 1 -- 172.21.15.129:0/964625632 shutdown_connections 2024-09-17T03:50:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.923+0000 7f8652e00640 1 -- 172.21.15.129:0/964625632 wait complete. 2024-09-17T03:50:09.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.925+0000 7f8652e00640 1 Processor -- start 2024-09-17T03:50:09.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.925+0000 7f8652e00640 1 -- start start 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 0x7f864c15d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f864c167920 con 0x7f864c0ca5a0 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f864c1677a0 con 0x7f864c0cad30 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8652e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f864c167aa0 con 0x7f864c15c5d0 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8651a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8651a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59390/0 (socket says 172.21.15.129:59390) 2024-09-17T03:50:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.926+0000 7f8651a00640 1 -- 172.21.15.129:0/4195618076 learned_addr learned my addr 172.21.15.129:0/4195618076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651000640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8652400640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 0x7f864c15d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 -- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 msgr2=0x7f864c15d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 0x7f864c15d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 -- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 msgr2=0x7f864c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c15cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 -- 172.21.15.129:0/4195618076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c15ab30 con 0x7f864c0ca5a0 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8652400640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 0x7f864c15d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:09.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8651a00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f864000c3c0 tx=0x7f864000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640011790 con 0x7f864c0ca5a0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8640017070 con 0x7f864c0ca5a0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.927+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640011ca0 con 0x7f864c0ca5a0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.928+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c1548c0 con 0x7f864c0ca5a0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.928+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c154b60 con 0x7f864c0ca5a0 2024-09-17T03:50:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.929+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c0b4470 con 0x7f864c0ca5a0 2024-09-17T03:50:09.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.930+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f864001c460 con 0x7f864c0ca5a0 2024-09-17T03:50:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.931+0000 7f8649600640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 0x7f862c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.931+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f864009a9e0 con 0x7f864c0ca5a0 2024-09-17T03:50:09.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.931+0000 7f8651000640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 0x7f862c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:09.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.932+0000 7f8651000640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 0x7f862c07b0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8638000f60 tx=0x7f8638009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:09.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:09.935+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8640062e30 con 0x7f864c0ca5a0 2024-09-17T03:50:10.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.151+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f864c159160 con 0x7f864c0ca5a0 2024-09-17T03:50:10.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.152+0000 7f8649600640 1 -- 172.21.15.129:0/4195618076 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7f8640062730 con 0x7f864c0ca5a0 2024-09-17T03:50:10.152 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:10.152 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:50:10.153 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.154+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 msgr2=0x7f862c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.154+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 0x7f862c07b0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8638000f60 tx=0x7f8638009040 comp rx=0 tx=0).stop 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.154+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 msgr2=0x7f864c1528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.154+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f864000c3c0 tx=0x7f864000c880 comp rx=0 tx=0).stop 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 shutdown_connections 2024-09-17T03:50:10.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f862c078bc0 0x7f862c07b0d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f864c15c5d0 0x7f864c15d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f864c0cad30 0x7f864c15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 --2- 172.21.15.129:0/4195618076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f864c0ca5a0 0x7f864c1528e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 >> 172.21.15.129:0/4195618076 conn(0x7f864c06d870 msgr2=0x7f864c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 shutdown_connections 2024-09-17T03:50:10.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.155+0000 7f8652e00640 1 -- 172.21.15.129:0/4195618076 wait complete. 2024-09-17T03:50:10.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 -- 172.21.15.129:0/3050137298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80706b0 msgr2=0x7f9ee80cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3050137298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80706b0 0x7f9ee80cba00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4005fd0 tx=0x7f9ed402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 -- 172.21.15.129:0/3050137298 shutdown_connections 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3050137298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee80cad30 0x7f9ee80735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3050137298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80706b0 0x7f9ee80cba00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 -- 172.21.15.129:0/3050137298 >> 172.21.15.129:0/3050137298 conn(0x7f9ee80b1cb0 msgr2=0x7f9ee80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 -- 172.21.15.129:0/3050137298 shutdown_connections 2024-09-17T03:50:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.397+0000 7f9ef0400640 1 -- 172.21.15.129:0/3050137298 wait complete. 2024-09-17T03:50:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.399+0000 7f9ef0400640 1 Processor -- start 2024-09-17T03:50:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.400+0000 7f9ef0400640 1 -- start start 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ee80cad30 0x7f9ee80c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee815dbb0 0x7f9ee815dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8073f00 con 0x7f9ee80ca800 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8073d80 con 0x7f9ee80cad30 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9ef0400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee8074080 con 0x7f9ee815dbb0 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9eeda00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9eeda00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59396/0 (socket says 172.21.15.129:59396) 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.401+0000 7f9eeda00640 1 -- 172.21.15.129:0/3414652529 learned_addr learned my addr 172.21.15.129:0/3414652529 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eee400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee815dbb0 0x7f9ee815dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eed000640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ee80cad30 0x7f9ee80c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 -- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee815dbb0 msgr2=0x7f9ee815dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee815dbb0 0x7f9ee815dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 -- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ee80cad30 msgr2=0x7f9ee80c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ee80cad30 0x7f9ee80c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 -- 172.21.15.129:0/3414652529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee80c71d0 con 0x7f9ee80ca800 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9eeda00640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c340 tx=0x7f9edc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.402+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0052d0 con 0x7f9ee80ca800 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.403+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9edc007020 con 0x7f9ee80ca800 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.403+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee80b1780 con 0x7f9ee80ca800 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.403+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc007470 con 0x7f9ee80ca800 2024-09-17T03:50:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.403+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee815d3b0 con 0x7f9ee80ca800 2024-09-17T03:50:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.404+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee80745e0 con 0x7f9ee80ca800 2024-09-17T03:50:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.405+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9edc0067d0 con 0x7f9ee80ca800 2024-09-17T03:50:10.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.406+0000 7f9ee5600640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 0x7f9ec807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.406+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f9edc09a1f0 con 0x7f9ee80ca800 2024-09-17T03:50:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.407+0000 7f9eed000640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 0x7f9ec807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.407+0000 7f9eed000640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 0x7f9ec807b0a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4033000 tx=0x7f9ed4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.410+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9edc061f40 con 0x7f9ee80ca800 2024-09-17T03:50:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.634+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ee80cba90 con 0x7f9ee80ca800 2024-09-17T03:50:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.635+0000 7f9ee5600640 1 -- 172.21.15.129:0/3414652529 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7f9edc066970 con 0x7f9ee80ca800 2024-09-17T03:50:10.635 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:10.635 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:50:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 msgr2=0x7f9ec807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 0x7f9ec807b0a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4033000 tx=0x7f9ed4004730 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 msgr2=0x7f9ee80b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c340 tx=0x7f9edc00c800 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 shutdown_connections 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ec8078b90 0x7f9ec807b0a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ee815dbb0 0x7f9ee815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ee80cad30 0x7f9ee80c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 --2- 172.21.15.129:0/3414652529 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ee80ca800 0x7f9ee80b0ef0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 >> 172.21.15.129:0/3414652529 conn(0x7f9ee80b1cb0 msgr2=0x7f9ee815b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.637+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 shutdown_connections 2024-09-17T03:50:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.638+0000 7f9ef0400640 1 -- 172.21.15.129:0/3414652529 wait complete. 2024-09-17T03:50:10.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 -- 172.21.15.129:0/157902618 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 msgr2=0x7ff94c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 --2- 172.21.15.129:0/157902618 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c0b3b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff9400098e0 tx=0x7ff94002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 -- 172.21.15.129:0/157902618 shutdown_connections 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 --2- 172.21.15.129:0/157902618 >> v2:172.21.15.181:3300/0 conn(0x7ff94c0b5a20 0x7ff94c0d2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 --2- 172.21.15.129:0/157902618 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c0b3b50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 -- 172.21.15.129:0/157902618 >> 172.21.15.129:0/157902618 conn(0x7ff94c09d6a0 msgr2=0x7ff94c0d43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 -- 172.21.15.129:0/157902618 shutdown_connections 2024-09-17T03:50:10.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.883+0000 7ff952c00640 1 -- 172.21.15.129:0/157902618 wait complete. 2024-09-17T03:50:10.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.885+0000 7ff952c00640 1 Processor -- start 2024-09-17T03:50:10.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.886+0000 7ff952c00640 1 -- start start 2024-09-17T03:50:10.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94c0b5a20 0x7ff94c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff94c0cfd20 con 0x7ff94c162470 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff94c0cfba0 con 0x7ff94c0b5a20 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff952c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff94c0cfea0 con 0x7ff94c163110 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff950e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff950e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59408/0 (socket says 172.21.15.129:59408) 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.887+0000 7ff950e00640 1 -- 172.21.15.129:0/2229342747 learned_addr learned my addr 172.21.15.129:0/2229342747 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff951800640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94c0b5a20 0x7ff94c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff952200640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 -- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 msgr2=0x7ff94c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c15e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 -- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94c0b5a20 msgr2=0x7ff94c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94c0b5a20 0x7ff94c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 -- 172.21.15.129:0/2229342747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94c15da80 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff950e00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e9f0 tx=0x7ff93800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.888+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93800cd80 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.889+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94c1f9670 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.889+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff938004020 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.889+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff938004470 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.889+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff94c15bac0 con 0x7ff94c162470 2024-09-17T03:50:10.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.890+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff94c08e450 con 0x7ff94c162470 2024-09-17T03:50:10.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.891+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff9380035f0 con 0x7ff94c162470 2024-09-17T03:50:10.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.892+0000 7ff949600640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 0x7ff92c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:10.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.892+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7ff93809b720 con 0x7ff94c162470 2024-09-17T03:50:10.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.892+0000 7ff951800640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 0x7ff92c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.893+0000 7ff951800640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 0x7ff92c07b0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff940034000 tx=0x7ff940004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:10.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:10.896+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff9380633f0 con 0x7ff94c162470 2024-09-17T03:50:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.080+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff94c1f9840 con 0x7ff94c162470 2024-09-17T03:50:11.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.081+0000 7ff949600640 1 -- 172.21.15.129:0/2229342747 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7ff938067e20 con 0x7ff94c162470 2024-09-17T03:50:11.081 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:11.081 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 msgr2=0x7ff92c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 0x7ff92c07b0d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff940034000 tx=0x7ff940004730 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 msgr2=0x7ff94c15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff93800e9f0 tx=0x7ff93800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 shutdown_connections 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff92c078bc0 0x7ff92c07b0d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff94c163110 0x7ff94c15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff94c162470 0x7ff94c15d540 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 --2- 172.21.15.129:0/2229342747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff94c0b5a20 0x7ff94c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 >> 172.21.15.129:0/2229342747 conn(0x7ff94c09d6a0 msgr2=0x7ff94c068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:11.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 shutdown_connections 2024-09-17T03:50:11.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.083+0000 7ff952c00640 1 -- 172.21.15.129:0/2229342747 wait complete. 2024-09-17T03:50:11.107 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:50:11.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 -- 172.21.15.129:0/3253885796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 msgr2=0x7fcf58004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 --2- 172.21.15.129:0/3253885796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf58004080 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fcf64001830 tx=0x7fcf58005830 comp rx=0 tx=0).stop 2024-09-17T03:50:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 -- 172.21.15.129:0/3253885796 shutdown_connections 2024-09-17T03:50:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 --2- 172.21.15.129:0/3253885796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf58004080 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 -- 172.21.15.129:0/3253885796 >> 172.21.15.129:0/3253885796 conn(0x7fcf7406d410 msgr2=0x7fcf740b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 -- 172.21.15.129:0/3253885796 shutdown_connections 2024-09-17T03:50:11.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.329+0000 7fcf73200640 1 -- 172.21.15.129:0/3253885796 wait complete. 2024-09-17T03:50:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.331+0000 7fcf73200640 1 Processor -- start 2024-09-17T03:50:11.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.331+0000 7fcf73200640 1 -- start start 2024-09-17T03:50:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf741f8cf0 0x7fcf741f9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcf74143e50 0x7fcf74144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcf740cf2f0 con 0x7fcf58003c30 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcf740cf170 con 0x7fcf741f8cf0 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf73200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcf740cf470 con 0x7fcf74143e50 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf71e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59426/0 (socket says 172.21.15.129:59426) 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf71e00640 1 -- 172.21.15.129:0/2280509210 learned_addr learned my addr 172.21.15.129:0/2280509210 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf72800640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcf74143e50 0x7fcf74144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.333+0000 7fcf69400640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf741f8cf0 0x7fcf741f9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 -- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcf74143e50 msgr2=0x7fcf74144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcf74143e50 0x7fcf74144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 -- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf741f8cf0 msgr2=0x7fcf741f9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf741f8cf0 0x7fcf741f9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 -- 172.21.15.129:0/2280509210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf741f9640 con 0x7fcf58003c30 2024-09-17T03:50:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf71e00640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcf6402a9f0 tx=0x7fcf640060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf64043070 con 0x7fcf58003c30 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcf64003af0 con 0x7fcf58003c30 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf740b3cd0 con 0x7fcf58003c30 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf640064c0 con 0x7fcf58003c30 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.334+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf740b4290 con 0x7fcf58003c30 2024-09-17T03:50:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.335+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf7415ca80 con 0x7fcf58003c30 2024-09-17T03:50:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.337+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcf64046070 con 0x7fcf58003c30 2024-09-17T03:50:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.337+0000 7fcf6aa00640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 0x7fcf5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.338+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fcf640baad0 con 0x7fcf58003c30 2024-09-17T03:50:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.338+0000 7fcf69400640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 0x7fcf5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.339+0000 7fcf69400640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 0x7fcf5007b0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf7413f170 tx=0x7fcf5c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.342+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcf741f9640 con 0x7fcf58003c30 2024-09-17T03:50:11.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.558+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf74141730 con 0x7fcf58003c30 2024-09-17T03:50:11.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.559+0000 7fcf6aa00640 1 -- 172.21.15.129:0/2280509210 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7fcf64082970 con 0x7fcf58003c30 2024-09-17T03:50:11.559 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:11.559 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:50:11.559 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 msgr2=0x7fcf5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 0x7fcf5007b0a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcf7413f170 tx=0x7fcf5c005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 msgr2=0x7fcf741f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcf6402a9f0 tx=0x7fcf640060e0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 shutdown_connections 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf50078b90 0x7fcf5007b0a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcf74143e50 0x7fcf74144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcf741f8cf0 0x7fcf741f9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 --2- 172.21.15.129:0/2280509210 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcf58003c30 0x7fcf741f5180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 >> 172.21.15.129:0/2280509210 conn(0x7fcf7406d410 msgr2=0x7fcf7415eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:11.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 shutdown_connections 2024-09-17T03:50:11.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.561+0000 7fcf73200640 1 -- 172.21.15.129:0/2280509210 wait complete. 2024-09-17T03:50:11.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:11.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 -- 172.21.15.129:0/1485312580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140cd240 msgr2=0x7f30140cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 --2- 172.21.15.129:0/1485312580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140cd240 0x7f30140cc4c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f301409ded0 tx=0x7f300002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 -- 172.21.15.129:0/1485312580 shutdown_connections 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 --2- 172.21.15.129:0/1485312580 >> v2:172.21.15.181:3300/0 conn(0x7f30140ce020 0x7f30140ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 --2- 172.21.15.129:0/1485312580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140cd240 0x7f30140cc4c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 -- 172.21.15.129:0/1485312580 >> 172.21.15.129:0/1485312580 conn(0x7f30140b1cb0 msgr2=0x7f30140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 -- 172.21.15.129:0/1485312580 shutdown_connections 2024-09-17T03:50:11.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.808+0000 7f301c600640 1 -- 172.21.15.129:0/1485312580 wait complete. 2024-09-17T03:50:11.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.811+0000 7f301c600640 1 Processor -- start 2024-09-17T03:50:11.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.811+0000 7f301c600640 1 -- start start 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30140cf6f0 0x7f301414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 0x7f3014160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30140701b0 con 0x7f30140ce020 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f30140cf3d0 con 0x7f3014160710 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.812+0000 7f301c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3014070330 con 0x7f30140cf6f0 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59438/0 (socket says 172.21.15.129:59438) 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 -- 172.21.15.129:0/2656772793 learned_addr learned my addr 172.21.15.129:0/2656772793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:11.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f301a600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 0x7f3014160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019200640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30140cf6f0 0x7f301414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 -- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30140cf6f0 msgr2=0x7f301414e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30140cf6f0 0x7f301414e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 -- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 msgr2=0x7f3014160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 0x7f3014160b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 -- 172.21.15.129:0/2656772793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301415af10 con 0x7f30140ce020 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f301a600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 0x7f3014160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.813+0000 7f3019c00640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f300800c370 tx=0x7f300800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.814+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008017070 con 0x7f30140ce020 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.814+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30080053c0 con 0x7f30140ce020 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.814+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301415b1f0 con 0x7f30140ce020 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.814+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008003850 con 0x7f30140ce020 2024-09-17T03:50:11.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.814+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30141601f0 con 0x7f30140ce020 2024-09-17T03:50:11.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.815+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30140ceae0 con 0x7f30140ce020 2024-09-17T03:50:11.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.816+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f300801a070 con 0x7f30140ce020 2024-09-17T03:50:11.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.817+0000 7f3011600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 0x7f2ff407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:11.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.817+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f300809a1d0 con 0x7f30140ce020 2024-09-17T03:50:11.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.818+0000 7f3019200640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 0x7f2ff407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:11.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.818+0000 7f3019200640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 0x7f2ff407b0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3000033000 tx=0x7f3000004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:11.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:11.821+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f30080624f0 con 0x7f30140ce020 2024-09-17T03:50:12.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.040+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30140b1400 con 0x7f30140ce020 2024-09-17T03:50:12.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.041+0000 7f3011600640 1 -- 172.21.15.129:0/2656772793 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5223 (secure 0 0 0) 0x7f3008061df0 con 0x7f30140ce020 2024-09-17T03:50:12.041 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:12.041 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":8,"btime":"2024-09-17T03:49:39:400808+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4319,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/941411172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":941411172},{"type":"v1","addr":"172.21.15.181:6836","nonce":941411172}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4362,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3650671999","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3650671999},{"type":"v1","addr":"172.21.15.129:6834","nonce":3650671999}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4365,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/792643212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":792643212},{"type":"v1","addr":"172.21.15.129:6835","nonce":792643212}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:49:35.864880+0000","modified":"2024-09-17T03:49:38.881199+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4374":{"gid":4374,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/3758496952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3758496952},{"type":"v1","addr":"172.21.15.181:6837","nonce":3758496952}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4374,"qdb_cluster":[4374]},"id":1}]} 2024-09-17T03:50:12.042 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 8 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 msgr2=0x7f2ff407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 0x7f2ff407b0a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3000033000 tx=0x7f3000004730 comp rx=0 tx=0).stop 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 msgr2=0x7f3014158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f300800c370 tx=0x7f300800c830 comp rx=0 tx=0).stop 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 shutdown_connections 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ff4078b90 0x7f2ff407b0a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3014160710 0x7f3014160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30140cf6f0 0x7f301414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 --2- 172.21.15.129:0/2656772793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30140ce020 0x7f3014158350 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.043+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 >> 172.21.15.129:0/2656772793 conn(0x7f30140b1cb0 msgr2=0x7f30140afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:12.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.044+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 shutdown_connections 2024-09-17T03:50:12.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.044+0000 7f301c600640 1 -- 172.21.15.129:0/2656772793 wait complete. 2024-09-17T03:50:12.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 -- 172.21.15.129:0/3458353490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 msgr2=0x7f3d380b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3458353490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d380b3b60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3d381623e0 tx=0x7f3d2802a560 comp rx=0 tx=0).stop 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 -- 172.21.15.129:0/3458353490 shutdown_connections 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3458353490 >> v2:172.21.15.181:3300/0 conn(0x7f3d380cf710 0x7f3d380cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3458353490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d380b3b60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 -- 172.21.15.129:0/3458353490 >> 172.21.15.129:0/3458353490 conn(0x7f3d3809d640 msgr2=0x7f3d380d4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 -- 172.21.15.129:0/3458353490 shutdown_connections 2024-09-17T03:50:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.286+0000 7f3d3e400640 1 -- 172.21.15.129:0/3458353490 wait complete. 2024-09-17T03:50:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.288+0000 7f3d3e400640 1 Processor -- start 2024-09-17T03:50:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.288+0000 7f3d3e400640 1 -- start start 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d380cf710 0x7f3d380c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d380d0060 0x7f3d380cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d380d1390 con 0x7f3d380b58b0 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d380d50a0 con 0x7f3d380d0060 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3e400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d380d1510 con 0x7f3d380cf710 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59464/0 (socket says 172.21.15.129:59464) 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 -- 172.21.15.129:0/3884694101 learned_addr learned my addr 172.21.15.129:0/3884694101 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d2fe00640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d380cf710 0x7f3d380c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3da00640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d380d0060 0x7f3d380cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 -- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d380cf710 msgr2=0x7f3d380c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d380cf710 0x7f3d380c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 -- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d380d0060 msgr2=0x7f3d380cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d380d0060 0x7f3d380cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.290+0000 7f3d3d000640 1 -- 172.21.15.129:0/3884694101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d380c7f80 con 0x7f3d380b58b0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d3d000640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3d280099d0 tx=0x7f3d280052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28031b50 con 0x7f3d380b58b0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d28031de0 con 0x7f3d380b58b0 2024-09-17T03:50:12.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d380c8260 con 0x7f3d380b58b0 2024-09-17T03:50:12.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28033330 con 0x7f3d380b58b0 2024-09-17T03:50:12.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.291+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d380c8820 con 0x7f3d380b58b0 2024-09-17T03:50:12.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.292+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d380d1960 con 0x7f3d380b58b0 2024-09-17T03:50:12.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.293+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d280334d0 con 0x7f3d380b58b0 2024-09-17T03:50:12.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.294+0000 7f3d35600640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 0x7f3d1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.294+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7f3d280bc290 con 0x7f3d380b58b0 2024-09-17T03:50:12.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.295+0000 7f3d2fe00640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 0x7f3d1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.296+0000 7f3d2fe00640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 0x7f3d1407b0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005d70 tx=0x7f3d20005d00 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:12.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.299+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d2803f030 con 0x7f3d380b58b0 2024-09-17T03:50:12.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.481+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d3808e3f0 con 0x7f3d380b58b0 2024-09-17T03:50:12.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.482+0000 7f3d35600640 1 -- 172.21.15.129:0/3884694101 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16713 (secure 0 0 0) 0x7f3d280846c0 con 0x7f3d380b58b0 2024-09-17T03:50:12.483 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:12.483 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":22,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:49:37.372776+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:49:33.632984+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-17T03:49:34.828244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 msgr2=0x7f3d1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 0x7f3d1407b0a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3d20005d70 tx=0x7f3d20005d00 comp rx=0 tx=0).stop 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 msgr2=0x7f3d381f6090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3d280099d0 tx=0x7f3d280052f0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 shutdown_connections 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d14078b90 0x7f3d1407b0a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d380d0060 0x7f3d380cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d380cf710 0x7f3d380c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 --2- 172.21.15.129:0/3884694101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d380b58b0 0x7f3d381f6090 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 >> 172.21.15.129:0/3884694101 conn(0x7f3d3809d640 msgr2=0x7f3d38068fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 shutdown_connections 2024-09-17T03:50:12.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.485+0000 7f3d3e400640 1 -- 172.21.15.129:0/3884694101 wait complete. 2024-09-17T03:50:12.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.728+0000 7fc913400640 1 -- 172.21.15.129:0/2102721087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8f4004800 msgr2=0x7fc8f4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.728+0000 7fc913400640 1 --2- 172.21.15.129:0/2102721087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8f4004800 0x7fc8f4004c10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009b50 tx=0x7fc8f4005830 comp rx=0 tx=0).stop 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.728+0000 7fc911400640 1 -- 172.21.15.129:0/2102721087 reap_dead start 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.729+0000 7fc913400640 1 -- 172.21.15.129:0/2102721087 shutdown_connections 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.729+0000 7fc913400640 1 -- 172.21.15.129:0/2102721087 >> 172.21.15.129:0/2102721087 conn(0x7fc90c06d7d0 msgr2=0x7fc90c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.729+0000 7fc913400640 1 -- 172.21.15.129:0/2102721087 shutdown_connections 2024-09-17T03:50:12.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.729+0000 7fc913400640 1 -- 172.21.15.129:0/2102721087 wait complete. 2024-09-17T03:50:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.731+0000 7fc913400640 1 Processor -- start 2024-09-17T03:50:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.731+0000 7fc913400640 1 -- start start 2024-09-17T03:50:12.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f4004800 0x7fc90c0cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc90c09b8a0 0x7fc90c144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c0ccf60 con 0x7fc90c1454d0 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc90c0af8e0 con 0x7fc8f4004800 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.732+0000 7fc913400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc90c0cd0e0 con 0x7fc90c09b8a0 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59492/0 (socket says 172.21.15.129:59492) 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 -- 172.21.15.129:0/4118261290 learned_addr learned my addr 172.21.15.129:0/4118261290 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc910a00640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f4004800 0x7fc90c0cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc90be00640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc90c09b8a0 0x7fc90c144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 -- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc90c09b8a0 msgr2=0x7fc90c144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc90c09b8a0 0x7fc90c144b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 -- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f4004800 msgr2=0x7fc90c0cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f4004800 0x7fc90c0cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 -- 172.21.15.129:0/4118261290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc90c0cb8f0 con 0x7fc90c1454d0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.733+0000 7fc911400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc904005fd0 tx=0x7fc904010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.734+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904003970 con 0x7fc90c1454d0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.734+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc904015070 con 0x7fc90c1454d0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.734+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904004030 con 0x7fc90c1454d0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.734+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc90c0ca5f0 con 0x7fc90c1454d0 2024-09-17T03:50:12.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.734+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc90c0cabb0 con 0x7fc90c1454d0 2024-09-17T03:50:12.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.735+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc90c0b40b0 con 0x7fc90c1454d0 2024-09-17T03:50:12.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.736+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9040041d0 con 0x7fc90c1454d0 2024-09-17T03:50:12.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.738+0000 7fc908c00640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 0x7fc8ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:12.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.738+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5709+0+0 (secure 0 0 0) 0x7fc90409b480 con 0x7fc90c1454d0 2024-09-17T03:50:12.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.738+0000 7fc910a00640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 0x7fc8ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:12.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.739+0000 7fc910a00640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 0x7fc8ec07b0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc900000fb0 tx=0x7fc900006910 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:12.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.742+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc904063820 con 0x7fc90c1454d0 2024-09-17T03:50:12.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.959+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc90c0b1b80 con 0x7fc90c1454d0 2024-09-17T03:50:12.959 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:50:12.960+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:50:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.962+0000 7fc908c00640 1 -- 172.21.15.129:0/4118261290 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fc904063120 con 0x7fc90c1454d0 2024-09-17T03:50:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 msgr2=0x7fc8ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 0x7fc8ec07b0d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc900000fb0 tx=0x7fc900006910 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 msgr2=0x7fc90c0adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc904005fd0 tx=0x7fc904010f40 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 shutdown_connections 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc8ec078bc0 0x7fc8ec07b0d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc90c1454d0 0x7fc90c0adfe0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc90c09b8a0 0x7fc90c144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 --2- 172.21.15.129:0/4118261290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8f4004800 0x7fc90c0cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:12.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 >> 172.21.15.129:0/4118261290 conn(0x7fc90c06d7d0 msgr2=0x7fc90c15b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 shutdown_connections 2024-09-17T03:50:12.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:12.965+0000 7fc913400640 1 -- 172.21.15.129:0/4118261290 wait complete. 2024-09-17T03:50:12.974 INFO:tasks.ceph.mds.b.smithi181.stderr: -8689> 2024-09-17T03:49:29.980+0000 7fdb5d000640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-17T03:50:12.977 INFO:tasks.ceph.mds.b.smithi181.stderr: -7275> 2024-09-17T03:49:40.981+0000 7fdb55800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:50:12.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:50:13.100 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:50:13.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 -- 172.21.15.129:0/782466845 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070440 msgr2=0x7ff708074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 --2- 172.21.15.129:0/782466845 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070440 0x7ff708074b80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff700005fd0 tx=0x7ff70002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 -- 172.21.15.129:0/782466845 shutdown_connections 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 --2- 172.21.15.129:0/782466845 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070440 0x7ff708074b80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 --2- 172.21.15.129:0/782466845 >> v2:172.21.15.181:3300/0 conn(0x7ff708070030 0x7ff708074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 -- 172.21.15.129:0/782466845 >> 172.21.15.129:0/782466845 conn(0x7ff7080b1cb0 msgr2=0x7ff7080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 -- 172.21.15.129:0/782466845 shutdown_connections 2024-09-17T03:50:13.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.214+0000 7ff70fc00640 1 -- 172.21.15.129:0/782466845 wait complete. 2024-09-17T03:50:13.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.216+0000 7ff70fc00640 1 Processor -- start 2024-09-17T03:50:13.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.216+0000 7ff70fc00640 1 -- start start 2024-09-17T03:50:13.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070030 0x7ff7081f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff708070440 0x7ff70809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7080b3a40 con 0x7ff708073170 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff7080b38c0 con 0x7ff708070440 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70fc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff7080b3bc0 con 0x7ff708070030 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59504/0 (socket says 172.21.15.129:59504) 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70dc00640 1 -- 172.21.15.129:0/4224321322 learned_addr learned my addr 172.21.15.129:0/4224321322 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff6ffe00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff708070440 0x7ff70809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.218+0000 7ff70d200640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070030 0x7ff7081f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:13.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 -- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070030 msgr2=0x7ff7081f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:13.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070030 0x7ff7081f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 -- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff708070440 msgr2=0x7ff70809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:13.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff708070440 0x7ff70809aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 -- 172.21.15.129:0/4224321322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70809d350 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70dc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff700006010 tx=0x7ff700031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff700003e00 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff700031a80 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7081f3630 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.219+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7000330d0 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.220+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7081f3bf0 con 0x7ff708073170 2024-09-17T03:50:13.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.221+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7080b40e0 con 0x7ff708073170 2024-09-17T03:50:13.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.222+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff700033270 con 0x7ff708073170 2024-09-17T03:50:13.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.223+0000 7ff705600640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 0x7ff6e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:13.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.223+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5795+0+0 (secure 0 0 0) 0x7ff7000bc170 con 0x7ff708073170 2024-09-17T03:50:13.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.223+0000 7ff70d200640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 0x7ff6e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:13.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.224+0000 7ff70d200640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 0x7ff6e407b0a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8007fd0 tx=0x7ff6f8001370 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:13.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.227+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff70003f030 con 0x7ff708073170 2024-09-17T03:50:13.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.443+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff70809d0e0 con 0x7ff708073170 2024-09-17T03:50:13.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.964+0000 7ff705600640 1 -- 172.21.15.129:0/4224321322 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7ff700083e60 con 0x7ff708073170 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 msgr2=0x7ff6e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 0x7ff6e407b0a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8007fd0 tx=0x7ff6f8001370 comp rx=0 tx=0).stop 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 msgr2=0x7ff70809af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff700006010 tx=0x7ff700031650 comp rx=0 tx=0).stop 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 shutdown_connections 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e4078b90 0x7ff6e407b0a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff708073170 0x7ff70809af70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff708070440 0x7ff70809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 --2- 172.21.15.129:0/4224321322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff708070030 0x7ff7081f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:13.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.967+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 >> 172.21.15.129:0/4224321322 conn(0x7ff7080b1cb0 msgr2=0x7ff7080b42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:13.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.968+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 shutdown_connections 2024-09-17T03:50:13.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:13.968+0000 7ff70fc00640 1 -- 172.21.15.129:0/4224321322 wait complete. 2024-09-17T03:50:13.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:50:14.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 -- 172.21.15.129:0/2884161981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd12c0053a0 msgr2=0x7fd12c00ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 --2- 172.21.15.129:0/2884161981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd12c0053a0 0x7fd12c00ba50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd140001dc0 tx=0x7fd12c005830 comp rx=0 tx=0).stop 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 -- 172.21.15.129:0/2884161981 shutdown_connections 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 --2- 172.21.15.129:0/2884161981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd12c0053a0 0x7fd12c00ba50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 -- 172.21.15.129:0/2884161981 >> 172.21.15.129:0/2884161981 conn(0x7fd1480d0130 msgr2=0x7fd1480d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 -- 172.21.15.129:0/2884161981 shutdown_connections 2024-09-17T03:50:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.220+0000 7fd14d400640 1 -- 172.21.15.129:0/2884161981 wait complete. 2024-09-17T03:50:14.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.222+0000 7fd14d400640 1 Processor -- start 2024-09-17T03:50:14.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.222+0000 7fd14d400640 1 -- start start 2024-09-17T03:50:14.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd12c0053a0 0x7fd1481552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd148155830 0x7fd1481f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1480cadc0 con 0x7fd1481f4550 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd148073b90 con 0x7fd12c0053a0 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd14d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd1480caf40 con 0x7fd148155830 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd147200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd147200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49450/0 (socket says 172.21.15.129:49450) 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd147200640 1 -- 172.21.15.129:0/4026683741 learned_addr learned my addr 172.21.15.129:0/4026683741 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd146800640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd12c0053a0 0x7fd1481552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.224+0000 7fd13de00640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd148155830 0x7fd1481f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 -- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd148155830 msgr2=0x7fd1481f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd148155830 0x7fd1481f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 -- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd12c0053a0 msgr2=0x7fd1481552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd12c0053a0 0x7fd1481552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 -- 172.21.15.129:0/4026683741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1480b4b60 con 0x7fd1481f4550 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd147200640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd14002acc0 tx=0x7fd140006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140003d80 con 0x7fd1481f4550 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd140005b20 con 0x7fd1481f4550 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd140005f70 con 0x7fd1481f4550 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1480b4e40 con 0x7fd1481f4550 2024-09-17T03:50:14.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.225+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1480b5400 con 0x7fd1481f4550 2024-09-17T03:50:14.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.226+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14809dad0 con 0x7fd1481f4550 2024-09-17T03:50:14.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.228+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd140005030 con 0x7fd1481f4550 2024-09-17T03:50:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.229+0000 7fd13f400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.229+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5795+0+0 (secure 0 0 0) 0x7fd1400baf80 con 0x7fd1481f4550 2024-09-17T03:50:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.229+0000 7fd146800640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:14.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.230+0000 7fd146800640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd138009c50 tx=0x7fd138005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:14.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.233+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd140043060 con 0x7fd1481f4550 2024-09-17T03:50:14.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.418+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd1480cb450 con 0x7fd1481f4550 2024-09-17T03:50:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.971+0000 7fd13f400640 1 -- 172.21.15.129:0/4026683741 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7fd140083380 con 0x7fd1481f4550 2024-09-17T03:50:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:50:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 msgr2=0x7fd12407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd138009c50 tx=0x7fd138005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 msgr2=0x7fd1480c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd14002acc0 tx=0x7fd140006ca0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 shutdown_connections 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1481f4550 0x7fd1480c88d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd148155830 0x7fd1481f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 --2- 172.21.15.129:0/4026683741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd12c0053a0 0x7fd1481552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 >> 172.21.15.129:0/4026683741 conn(0x7fd1480d0130 msgr2=0x7fd1480aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.976+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 shutdown_connections 2024-09-17T03:50:14.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:14.977+0000 7fd14d400640 1 -- 172.21.15.129:0/4026683741 wait complete. 2024-09-17T03:50:15.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:50:15.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.231+0000 7f76b5000640 1 -- 172.21.15.129:0/2519243939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b0070310 msgr2=0x7f76b015f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:15.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.231+0000 7f76b5000640 1 --2- 172.21.15.129:0/2519243939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b0070310 0x7f76b015f5f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f76b006e090 tx=0x7f769c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 -- 172.21.15.129:0/2519243939 shutdown_connections 2024-09-17T03:50:15.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 --2- 172.21.15.129:0/2519243939 >> v2:172.21.15.181:3300/0 conn(0x7f76b015fb30 0x7f76b00ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 --2- 172.21.15.129:0/2519243939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b0070310 0x7f76b015f5f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 -- 172.21.15.129:0/2519243939 >> 172.21.15.129:0/2519243939 conn(0x7f76b0073d90 msgr2=0x7f76b00741a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:15.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 -- 172.21.15.129:0/2519243939 shutdown_connections 2024-09-17T03:50:15.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.232+0000 7f76b5000640 1 -- 172.21.15.129:0/2519243939 wait complete. 2024-09-17T03:50:15.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.234+0000 7f76b5000640 1 Processor -- start 2024-09-17T03:50:15.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.234+0000 7f76b5000640 1 -- start start 2024-09-17T03:50:15.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.235+0000 7f76b5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b0070310 0x7f76b014f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:15.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.235+0000 7f76b5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:15.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76b5000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76b015fb30 0x7f76b015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76b5000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f76b00ae3a0 con 0x7f76b00af660 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76b5000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f76b00ae220 con 0x7f76b0070310 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76b5000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f76b00ae520 con 0x7f76b015fb30 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49470/0 (socket says 172.21.15.129:49470) 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 -- 172.21.15.129:0/3516515484 learned_addr learned my addr 172.21.15.129:0/3516515484 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ae800640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76b015fb30 0x7f76b015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ade00640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b0070310 0x7f76b014f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 -- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76b015fb30 msgr2=0x7f76b015b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76b015fb30 0x7f76b015b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 -- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b0070310 msgr2=0x7f76b014f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:15.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b0070310 0x7f76b014f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.236+0000 7f76ad400640 1 -- 172.21.15.129:0/3516515484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76b0157650 con 0x7f76b00af660 2024-09-17T03:50:15.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76ad400640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f769c02aaa0 tx=0x7f769c0052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:15.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c005cf0 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f769c031c40 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769c033330 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76b01f93b0 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.237+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76b01f9970 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.238+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76b01f6c50 con 0x7f76b00af660 2024-09-17T03:50:15.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.239+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f769c0334d0 con 0x7f76b00af660 2024-09-17T03:50:15.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.240+0000 7f76a6000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 0x7f769007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:15.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.241+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5473+0+0 (secure 0 0 0) 0x7f769c0bbff0 con 0x7f76b00af660 2024-09-17T03:50:15.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.241+0000 7f76ade00640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 0x7f769007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:15.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.242+0000 7f76ade00640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 0x7f769007b0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006fd0 tx=0x7f76a0008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:15.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.245+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f769c03f030 con 0x7f76b00af660 2024-09-17T03:50:15.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.431+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f76b0076500 con 0x7f76b00af660 2024-09-17T03:50:15.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:50:15.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:50:15.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.978+0000 7f76a6000640 1 -- 172.21.15.129:0/3516515484 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7f769c0844e0 con 0x7f76b00af660 2024-09-17T03:50:15.979 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:50:15.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.982+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 msgr2=0x7f769007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:15.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.982+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 0x7f769007b0d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f76a0006fd0 tx=0x7f76a0008040 comp rx=0 tx=0).stop 2024-09-17T03:50:15.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.982+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 msgr2=0x7f76b015af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:15.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.982+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f769c02aaa0 tx=0x7f769c0052f0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 shutdown_connections 2024-09-17T03:50:15.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7690078bc0 0x7f769007b0d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76b015fb30 0x7f76b015b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76b00af660 0x7f76b015af30 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 --2- 172.21.15.129:0/3516515484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76b0070310 0x7f76b014f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 >> 172.21.15.129:0/3516515484 conn(0x7f76b0073d90 msgr2=0x7f76b015e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 shutdown_connections 2024-09-17T03:50:15.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:15.983+0000 7f76b5000640 1 -- 172.21.15.129:0/3516515484 wait complete. 2024-09-17T03:50:16.015 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:50:16.015 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:50:16.015 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:50:16.015 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:50:16.016 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:50:16.016 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:50:16.016 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:50:16.016 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:50:16.016 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:50:16.016 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:50:16.017 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:50:16.017 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:50:16.017 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:50:16.017 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:50:16.017 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:50:16.017 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:50:16.036 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:50:16.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:50:16.037 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:50:16.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:50:16.041 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:50:16.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:50:16.045 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:50:16.045 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:50:16.080 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:50:16.080 DEBUG:teuthology.parallel:result is None 2024-09-17T03:50:16.082 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:50:16.082 DEBUG:teuthology.parallel:result is None 2024-09-17T03:50:16.088 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:50:16.088 DEBUG:teuthology.parallel:result is None 2024-09-17T03:50:16.091 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:50:16.091 DEBUG:teuthology.parallel:result is None 2024-09-17T03:50:16.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:50:16.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-09-17T03:50:16.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-09-17T03:50:16.247 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-17T03:50:16.280 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:50:16.280 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:50:16.286 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:50:16.286 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:50:16.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 -- 172.21.15.129:0/2875121853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80c9a00 msgr2=0x7f6ed80c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/2875121853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80c9a00 0x7f6ed80c9e10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0005fd0 tx=0x7f6ed002aa90 comp rx=0 tx=0).stop 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 -- 172.21.15.129:0/2875121853 shutdown_connections 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/2875121853 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80c9a00 0x7f6ed80c9e10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/2875121853 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cfdb0 0x7f6ed80d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 -- 172.21.15.129:0/2875121853 >> 172.21.15.129:0/2875121853 conn(0x7f6ed806d870 msgr2=0x7f6ed806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 -- 172.21.15.129:0/2875121853 shutdown_connections 2024-09-17T03:50:16.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.326+0000 7f6ed7e00640 1 -- 172.21.15.129:0/2875121853 wait complete. 2024-09-17T03:50:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.328+0000 7f6ed7e00640 1 Processor -- start 2024-09-17T03:50:16.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.328+0000 7f6ed7e00640 1 -- start start 2024-09-17T03:50:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed80c9a00 0x7f6ed81f6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cd000 0x7f6ed8137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed815f070 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed815eef0 con 0x7f6ed80c9a00 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed815f1f0 con 0x7f6ed80cd000 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49532/0 (socket says 172.21.15.129:49532) 2024-09-17T03:50:16.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7400640 1 -- 172.21.15.129:0/40334247 learned_addr learned my addr 172.21.15.129:0/40334247 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed6a00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed80c9a00 0x7f6ed81f6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ece000640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cd000 0x7f6ed8137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7400640 1 -- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cd000 msgr2=0x7f6ed8137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.330+0000 7f6ed7400640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cd000 0x7f6ed8137530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7400640 1 -- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed80c9a00 msgr2=0x7f6ed81f6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7400640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed80c9a00 0x7f6ed81f6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7400640 1 -- 172.21.15.129:0/40334247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed81f6a60 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7400640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0006010 tx=0x7f6ed0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0044070 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ed002e7f0 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00044b0 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed8137a70 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.331+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed80aed60 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.333+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed80ae7d0 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.334+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6ed0047070 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.335+0000 7f6ecf400640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 0x7f6eb407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.335+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7f6ed00bbb80 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.336+0000 7f6ed6a00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 0x7f6eb407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.336+0000 7f6ed6a00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 0x7f6eb407b0a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80016c0 tx=0x7f6ec8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.338+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ed004e080 con 0x7f6ed80cfdb0 2024-09-17T03:50:16.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:16.536+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f6ed815f710 con 0x7f6ed80cfdb0 2024-09-17T03:50:17.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.000+0000 7f6ecf400640 1 -- 172.21.15.129:0/40334247 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7f6ed81f6a60 con 0x7f6ed80cfdb0 2024-09-17T03:50:17.000 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 msgr2=0x7f6eb407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 0x7f6eb407b0a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80016c0 tx=0x7f6ec8009040 comp rx=0 tx=0).stop 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 msgr2=0x7f6ed81f9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6ed0006010 tx=0x7f6ed0003f10 comp rx=0 tx=0).stop 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 shutdown_connections 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb4078b90 0x7f6eb407b0a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed80cfdb0 0x7f6ed81f9090 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed80cd000 0x7f6ed8137530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 --2- 172.21.15.129:0/40334247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed80c9a00 0x7f6ed81f6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 >> 172.21.15.129:0/40334247 conn(0x7f6ed806d870 msgr2=0x7f6ed80ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 shutdown_connections 2024-09-17T03:50:17.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.003+0000 7f6ed7e00640 1 -- 172.21.15.129:0/40334247 wait complete. 2024-09-17T03:50:17.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:50:17.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 -- 172.21.15.129:0/2799858584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c341604f0 msgr2=0x7f8c34160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 --2- 172.21.15.129:0/2799858584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c341604f0 0x7f8c34160900 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8c34071a20 tx=0x7f8c2c02aa10 comp rx=0 tx=0).stop 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 -- 172.21.15.129:0/2799858584 shutdown_connections 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 --2- 172.21.15.129:0/2799858584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c341604f0 0x7f8c34160900 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 --2- 172.21.15.129:0/2799858584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 0x7f8c340705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 --2- 172.21.15.129:0/2799858584 >> v2:172.21.15.181:3300/0 conn(0x7f8c340cfde0 0x7f8c34070060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 -- 172.21.15.129:0/2799858584 >> 172.21.15.129:0/2799858584 conn(0x7f8c340b3090 msgr2=0x7f8c340b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 -- 172.21.15.129:0/2799858584 shutdown_connections 2024-09-17T03:50:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.247+0000 7f8c33200640 1 -- 172.21.15.129:0/2799858584 wait complete. 2024-09-17T03:50:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.250+0000 7f8c33200640 1 Processor -- start 2024-09-17T03:50:17.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.250+0000 7f8c33200640 1 -- start start 2024-09-17T03:50:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 0x7f8c341f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c341604f0 0x7f8c341f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3415d1b0 con 0x7f8c340cfde0 2024-09-17T03:50:17.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3415d030 con 0x7f8c341604f0 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c33200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3415d330 con 0x7f8c340d01f0 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.251+0000 7f8c31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49544/0 (socket says 172.21.15.129:49544) 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 -- 172.21.15.129:0/1460911744 learned_addr learned my addr 172.21.15.129:0/1460911744 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c32800640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c341604f0 0x7f8c341f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c2be00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 0x7f8c341f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 -- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 msgr2=0x7f8c341f6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 0x7f8c341f6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 -- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c341604f0 msgr2=0x7f8c341f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c341604f0 0x7f8c341f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 -- 172.21.15.129:0/1460911744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c34140210 con 0x7f8c340cfde0 2024-09-17T03:50:17.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c31e00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e7c0 tx=0x7f8c2400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.252+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2400ccc0 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.253+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c24010430 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.253+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c24010860 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.253+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c341406b0 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.253+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c34140c70 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.254+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3415d850 con 0x7f8c340cfde0 2024-09-17T03:50:17.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.255+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8c24010a00 con 0x7f8c340cfde0 2024-09-17T03:50:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.256+0000 7f8c2a000640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 0x7f8c1007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.256+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f8c2409b440 con 0x7f8c340cfde0 2024-09-17T03:50:17.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.256+0000 7f8c2be00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 0x7f8c1007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.257+0000 7f8c2be00640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 0x7f8c1007b0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c340728b0 tx=0x7f8c1c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:17.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.260+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8c240634e0 con 0x7f8c340cfde0 2024-09-17T03:50:17.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.515+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8c340ce8a0 con 0x7f8c340cfde0 2024-09-17T03:50:17.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.515+0000 7f8c2a000640 1 -- 172.21.15.129:0/1460911744 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) ==== 80+0+6645 (secure 0 0 0) 0x7f8c24067f10 con 0x7f8c340cfde0 2024-09-17T03:50:17.515 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:17.515 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:50:17.517 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.518 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:50:17.520 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:50:17.522 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.523 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.524 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.525 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.526 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:50:17.527 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-mds", 2024-09-17T03:50:17.528 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/OhmMt94MBAAAW6zjB3fPu4JPIldmST2IA==", 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-mds" 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-mgr", 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/Ohm9+p4MBAADu5LL019L+v794DoHq0QrQ==", 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-osd", 2024-09-17T03:50:17.529 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/OhmwBx5MBAAF6NbHrQyq8DzGOhVzaAA0w==", 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-osd" 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-rbd", 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/Ohm/Cd5MBAAKKw5dLMEYCETNYi4PN6GYQ==", 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/OhmYzB5MBAAE3AOyb4ukCy3bU5/G8MSKw==", 2024-09-17T03:50:17.530 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.bootstrap-rgw", 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQA0/Ohm9jh5MBAAy0DNJEY+MPSTcNg4i4lXAg==", 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:50:17.531 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:50:17.532 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.533 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:17.533 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:50:17.533 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 msgr2=0x7f8c1007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 0x7f8c1007b0d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8c340728b0 tx=0x7f8c1c005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 msgr2=0x7f8c34136960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8c2400e7c0 tx=0x7f8c2400ec80 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 shutdown_connections 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c10078bc0 0x7f8c1007b0d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c341604f0 0x7f8c341f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c340d01f0 0x7f8c341f6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 --2- 172.21.15.129:0/1460911744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c340cfde0 0x7f8c34136960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 >> 172.21.15.129:0/1460911744 conn(0x7f8c340b3090 msgr2=0x7f8c340b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:17.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 shutdown_connections 2024-09-17T03:50:17.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.518+0000 7f8c33200640 1 -- 172.21.15.129:0/1460911744 wait complete. 2024-09-17T03:50:17.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-17T03:50:17.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 -- 172.21.15.129:0/1036747413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 msgr2=0x7f9ad40cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 --2- 172.21.15.129:0/1036747413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad40cc830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40b2c80 tx=0x7f9acc02aa30 comp rx=0 tx=0).stop 2024-09-17T03:50:17.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 -- 172.21.15.129:0/1036747413 shutdown_connections 2024-09-17T03:50:17.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 --2- 172.21.15.129:0/1036747413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad40cc830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 --2- 172.21.15.129:0/1036747413 >> v2:172.21.15.181:3300/0 conn(0x7f9ad40cad30 0x7f9ad406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 --2- 172.21.15.129:0/1036747413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 0x7f9ad40cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 -- 172.21.15.129:0/1036747413 >> 172.21.15.129:0/1036747413 conn(0x7f9ad40b1cb0 msgr2=0x7f9ad40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:17.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 -- 172.21.15.129:0/1036747413 shutdown_connections 2024-09-17T03:50:17.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.762+0000 7f9adb600640 1 -- 172.21.15.129:0/1036747413 wait complete. 2024-09-17T03:50:17.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.764+0000 7f9adb600640 1 Processor -- start 2024-09-17T03:50:17.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.764+0000 7f9adb600640 1 -- start start 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 0x7f9ad40aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ad40cad30 0x7f9ad415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad40ccf80 con 0x7f9ad40704e0 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad40cec60 con 0x7f9ad40cad30 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9adb600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad40cd100 con 0x7f9ad40ca800 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49570/0 (socket says 172.21.15.129:49570) 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 -- 172.21.15.129:0/3127289544 learned_addr learned my addr 172.21.15.129:0/3127289544 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9acbe00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 0x7f9ad40aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad9600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ad40cad30 0x7f9ad415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 -- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 msgr2=0x7f9ad40aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 0x7f9ad40aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 -- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ad40cad30 msgr2=0x7f9ad415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.766+0000 7f9ad8c00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ad40cad30 0x7f9ad415c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9ad8c00640 1 -- 172.21.15.129:0/3127289544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad40c81a0 con 0x7f9ad40704e0 2024-09-17T03:50:17.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9ad8c00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e770 tx=0x7f9ac400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400ccc0 con 0x7f9ad40704e0 2024-09-17T03:50:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ac400ee80 con 0x7f9ad40704e0 2024-09-17T03:50:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad4157a70 con 0x7f9ad40704e0 2024-09-17T03:50:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4004290 con 0x7f9ad40704e0 2024-09-17T03:50:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.767+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad40adc90 con 0x7f9ad40704e0 2024-09-17T03:50:17.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.768+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad40cd690 con 0x7f9ad40704e0 2024-09-17T03:50:17.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.770+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9ac40035f0 con 0x7f9ad40704e0 2024-09-17T03:50:17.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.770+0000 7f9ad1600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 0x7f9ab007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.771+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f9ac409b1b0 con 0x7f9ad40704e0 2024-09-17T03:50:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.771+0000 7f9acbe00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 0x7f9ab007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.771+0000 7f9acbe00640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 0x7f9ab007b0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ad406fd10 tx=0x7f9abc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:17.775+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9ac4063380 con 0x7f9ad40704e0 2024-09-17T03:50:18.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.031+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f9ad40d0a70 con 0x7f9ad40704e0 2024-09-17T03:50:18.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.036+0000 7f9ad1600640 1 -- 172.21.15.129:0/3127289544 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v21) ==== 90+0+0 (secure 0 0 0) 0x7f9ac4067db0 con 0x7f9ad40704e0 2024-09-17T03:50:18.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 msgr2=0x7f9ab007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 0x7f9ab007b0d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9ad406fd10 tx=0x7f9abc005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:18.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 msgr2=0x7f9ad4157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e770 tx=0x7f9ac400ec30 comp rx=0 tx=0).stop 2024-09-17T03:50:18.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 shutdown_connections 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab0078bc0 0x7f9ab007b0d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9ad40cad30 0x7f9ad415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9ad40ca800 0x7f9ad40aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 --2- 172.21.15.129:0/3127289544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9ad40704e0 0x7f9ad4157530 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.038+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 >> 172.21.15.129:0/3127289544 conn(0x7f9ad40b1cb0 msgr2=0x7f9ad40b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.039+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 shutdown_connections 2024-09-17T03:50:18.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.039+0000 7f9adb600640 1 -- 172.21.15.129:0/3127289544 wait complete. 2024-09-17T03:50:18.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.285+0000 7ff3d0a00640 1 -- 172.21.15.129:0/742323994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0706c0 msgr2=0x7ff3cc072ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.285+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/742323994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0706c0 0x7ff3cc072ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4005fd0 tx=0x7ff3c402a820 comp rx=0 tx=0).stop 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 -- 172.21.15.129:0/742323994 shutdown_connections 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/742323994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0706c0 0x7ff3cc072ff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/742323994 >> v2:172.21.15.181:3300/0 conn(0x7ff3cc0cd3e0 0x7ff3cc070180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 -- 172.21.15.129:0/742323994 >> 172.21.15.129:0/742323994 conn(0x7ff3cc06d7d0 msgr2=0x7ff3cc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:18.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 -- 172.21.15.129:0/742323994 shutdown_connections 2024-09-17T03:50:18.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.286+0000 7ff3d0a00640 1 -- 172.21.15.129:0/742323994 wait complete. 2024-09-17T03:50:18.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.288+0000 7ff3d0a00640 1 Processor -- start 2024-09-17T03:50:18.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.288+0000 7ff3d0a00640 1 -- start start 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3cc0706c0 0x7ff3cc14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0af780 0x7ff3cc1f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc073bd0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3cc073a50 con 0x7ff3cc0706c0 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.289+0000 7ff3d0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff3cc15f930 con 0x7ff3cc0af780 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49576/0 (socket says 172.21.15.129:49576) 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3c1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0af780 0x7ff3cc1f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 -- 172.21.15.129:0/3992227972 learned_addr learned my addr 172.21.15.129:0/3992227972 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3c9e00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3cc0706c0 0x7ff3cc14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 -- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0af780 msgr2=0x7ff3cc1f8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0af780 0x7ff3cc1f8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 -- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3cc0706c0 msgr2=0x7ff3cc14b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3cc0706c0 0x7ff3cc14b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 -- 172.21.15.129:0/3992227972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3cc1f9a90 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.290+0000 7ff3ca800640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006010 tx=0x7ff3c4032be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.291+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4037040 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.291+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff3c4042070 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.291+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3cc14ddf0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.291+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c4004040 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.291+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3cc14e3b0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.292+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3cc1f9800 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.293+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff3c40041e0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.294+0000 7ff3c2a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 0x7ff3a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.294+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7ff3c40bb6a0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.295+0000 7ff3c9e00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 0x7ff3a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.296+0000 7ff3c9e00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 0x7ff3a807b0a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc007fd0 tx=0x7ff3bc001370 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:18.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.299+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff3c40836c0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.551+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7ff3cc072ff0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.554+0000 7ff3c2a00640 1 -- 172.21.15.129:0/3992227972 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v22) ==== 90+0+0 (secure 0 0 0) 0x7ff3c40880f0 con 0x7ff3cc0cd3e0 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 msgr2=0x7ff3a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 0x7ff3a807b0a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc007fd0 tx=0x7ff3bc001370 comp rx=0 tx=0).stop 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 msgr2=0x7ff3cc1f9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff3c4006010 tx=0x7ff3c4032be0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 shutdown_connections 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3a8078b90 0x7ff3a807b0a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3cc0cd3e0 0x7ff3cc1f9230 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3cc0af780 0x7ff3cc1f8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 --2- 172.21.15.129:0/3992227972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3cc0706c0 0x7ff3cc14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 >> 172.21.15.129:0/3992227972 conn(0x7ff3cc06d7d0 msgr2=0x7ff3cc0aefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 shutdown_connections 2024-09-17T03:50:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.558+0000 7ff3d0a00640 1 -- 172.21.15.129:0/3992227972 wait complete. 2024-09-17T03:50:18.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-17T03:50:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 -- 172.21.15.129:0/117852807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 msgr2=0x7fc504003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 --2- 172.21.15.129:0/117852807 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc504003ea0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc510009b50 tx=0x7fc504005830 comp rx=0 tx=0).stop 2024-09-17T03:50:18.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc521e00640 1 -- 172.21.15.129:0/117852807 reap_dead start 2024-09-17T03:50:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 -- 172.21.15.129:0/117852807 shutdown_connections 2024-09-17T03:50:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 -- 172.21.15.129:0/117852807 >> 172.21.15.129:0/117852807 conn(0x7fc51c06d7d0 msgr2=0x7fc51c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 -- 172.21.15.129:0/117852807 shutdown_connections 2024-09-17T03:50:18.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.801+0000 7fc523e00640 1 -- 172.21.15.129:0/117852807 wait complete. 2024-09-17T03:50:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.804+0000 7fc523e00640 1 Processor -- start 2024-09-17T03:50:18.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.804+0000 7fc523e00640 1 -- start start 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc51c0b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 0x7fc51c0ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c0af750 con 0x7fc51c150e20 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc51c0af5d0 con 0x7fc504003a30 2024-09-17T03:50:18.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.805+0000 7fc523e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc51c0af8d0 con 0x7fc51c0acf90 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49600/0 (socket says 172.21.15.129:49600) 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 -- 172.21.15.129:0/3112173572 learned_addr learned my addr 172.21.15.129:0/3112173572 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc521e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 0x7fc51c0ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc521400640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc51c0b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 -- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 msgr2=0x7fc51c0ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 0x7fc51c0ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 -- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 msgr2=0x7fc51c0b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc51c0b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 -- 172.21.15.129:0/3112173572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51c0ad8e0 con 0x7fc51c150e20 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc521e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 0x7fc51c0ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc521400640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc51c0b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc520a00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc50800d6a0 tx=0x7fc50800db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5080060d0 con 0x7fc51c150e20 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.806+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc508015020 con 0x7fc51c150e20 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.807+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc50800b830 con 0x7fc51c150e20 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.807+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc51c0b0dd0 con 0x7fc51c150e20 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.807+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc51c0b1390 con 0x7fc51c150e20 2024-09-17T03:50:18.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.808+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc51c0b40b0 con 0x7fc51c150e20 2024-09-17T03:50:18.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.809+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc50800b9d0 con 0x7fc51c150e20 2024-09-17T03:50:18.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.810+0000 7fc519600640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 0x7fc4fc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:18.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.810+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fc50809b1b0 con 0x7fc51c150e20 2024-09-17T03:50:18.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.810+0000 7fc521400640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 0x7fc4fc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:18.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.811+0000 7fc521400640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 0x7fc4fc07b0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc510035000 tx=0x7fc51002d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:18.815+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc50801f020 con 0x7fc51c150e20 2024-09-17T03:50:19.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.067+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7fc51c0b0b60 con 0x7fc51c150e20 2024-09-17T03:50:19.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.070+0000 7fc519600640 1 -- 172.21.15.129:0/3112173572 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v23) ==== 90+0+0 (secure 0 0 0) 0x7fc51c0ad8e0 con 0x7fc51c150e20 2024-09-17T03:50:19.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 msgr2=0x7fc4fc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 0x7fc4fc07b0d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc510035000 tx=0x7fc51002d040 comp rx=0 tx=0).stop 2024-09-17T03:50:19.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 msgr2=0x7fc51c14abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc50800d6a0 tx=0x7fc50800db60 comp rx=0 tx=0).stop 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 shutdown_connections 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4fc078bc0 0x7fc4fc07b0d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc51c0acf90 0x7fc51c0ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc51c150e20 0x7fc51c14abe0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 --2- 172.21.15.129:0/3112173572 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc504003a30 0x7fc51c0b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 >> 172.21.15.129:0/3112173572 conn(0x7fc51c06d7d0 msgr2=0x7fc51c15aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 shutdown_connections 2024-09-17T03:50:19.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.073+0000 7fc523e00640 1 -- 172.21.15.129:0/3112173572 wait complete. 2024-09-17T03:50:19.095 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.316+0000 7f89c1400640 1 -- 172.21.15.129:0/3709975470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a0005250 msgr2=0x7f89a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.316+0000 7f89c1400640 1 --2- 172.21.15.129:0/3709975470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a0005250 0x7f89a000b9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f89b4001dc0 tx=0x7f89a0005830 comp rx=0 tx=0).stop 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.316+0000 7f89c1400640 1 -- 172.21.15.129:0/3709975470 shutdown_connections 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.316+0000 7f89c1400640 1 --2- 172.21.15.129:0/3709975470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89a0005250 0x7f89a000b9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.316+0000 7f89c1400640 1 -- 172.21.15.129:0/3709975470 >> 172.21.15.129:0/3709975470 conn(0x7f89bc06d7d0 msgr2=0x7f89bc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:19.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.317+0000 7f89c1400640 1 -- 172.21.15.129:0/3709975470 shutdown_connections 2024-09-17T03:50:19.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.317+0000 7f89c1400640 1 -- 172.21.15.129:0/3709975470 wait complete. 2024-09-17T03:50:19.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.319+0000 7f89c1400640 1 Processor -- start 2024-09-17T03:50:19.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.319+0000 7f89c1400640 1 -- start start 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.320+0000 7f89c1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89a0005250 0x7f89bc148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.320+0000 7f89c1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.320+0000 7f89c1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89bc158080 0x7f89bc1f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89c1400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc15dc30 con 0x7f89bc14bc20 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89c1400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f89bc15dab0 con 0x7f89a0005250 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89c1400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f89bc15ddb0 con 0x7f89bc158080 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49620/0 (socket says 172.21.15.129:49620) 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 -- 172.21.15.129:0/3938075238 learned_addr learned my addr 172.21.15.129:0/3938075238 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:19.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89ba800640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89a0005250 0x7f89bc148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89bb200640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89bc158080 0x7f89bc1f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 -- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89bc158080 msgr2=0x7f89bc1f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89bc158080 0x7f89bc1f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 -- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89a0005250 msgr2=0x7f89bc148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89a0005250 0x7f89bc148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 -- 172.21.15.129:0/3938075238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89bc158490 con 0x7f89bc14bc20 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.321+0000 7f89b9e00640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f89a8011ab0 tx=0x7f89a8011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.322+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a801e070 con 0x7f89bc14bc20 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.322+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89a800fe70 con 0x7f89bc14bc20 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.322+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a80042b0 con 0x7f89bc14bc20 2024-09-17T03:50:19.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.322+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89bc155610 con 0x7f89bc14bc20 2024-09-17T03:50:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.322+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89bc156000 con 0x7f89bc14bc20 2024-09-17T03:50:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.323+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89bc155df0 con 0x7f89bc14bc20 2024-09-17T03:50:19.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.324+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f89a8021070 con 0x7f89bc14bc20 2024-09-17T03:50:19.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.326+0000 7f89b2a00640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 0x7f899c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.326+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f89a809b3a0 con 0x7f89bc14bc20 2024-09-17T03:50:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.326+0000 7f89ba800640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 0x7f899c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.327+0000 7f89ba800640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 0x7f899c07b0a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f89ac001010 tx=0x7f89ac0067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:19.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.330+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f89bc158490 con 0x7f89bc14bc20 2024-09-17T03:50:19.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.580+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f89bc0cd2e0 con 0x7f89bc14bc20 2024-09-17T03:50:19.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.583+0000 7f89b2a00640 1 -- 172.21.15.129:0/3938075238 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v24) ==== 90+0+0 (secure 0 0 0) 0x7f89a8063560 con 0x7f89bc14bc20 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 msgr2=0x7f899c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 0x7f899c07b0a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f89ac001010 tx=0x7f89ac0067a0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 msgr2=0x7f89bc157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f89a8011ab0 tx=0x7f89a8011f70 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 shutdown_connections 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f899c078b90 0x7f899c07b0a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89bc158080 0x7f89bc1f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89bc14bc20 0x7f89bc157b40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 --2- 172.21.15.129:0/3938075238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89a0005250 0x7f89bc148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 >> 172.21.15.129:0/3938075238 conn(0x7f89bc06d7d0 msgr2=0x7f89bc068e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 shutdown_connections 2024-09-17T03:50:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.586+0000 7f89c1400640 1 -- 172.21.15.129:0/3938075238 wait complete. 2024-09-17T03:50:19.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 -- 172.21.15.129:0/1868537207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cd240 msgr2=0x7f78840cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 --2- 172.21.15.129:0/1868537207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cd240 0x7f78840cf010 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f78840b58f0 tx=0x7f787002a560 comp rx=0 tx=0).stop 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 -- 172.21.15.129:0/1868537207 shutdown_connections 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 --2- 172.21.15.129:0/1868537207 >> v2:172.21.15.181:3300/0 conn(0x7f78840cfb90 0x7f78840b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 --2- 172.21.15.129:0/1868537207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cd240 0x7f78840cf010 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.829+0000 7f7889800640 1 -- 172.21.15.129:0/1868537207 >> 172.21.15.129:0/1868537207 conn(0x7f78840b1cb0 msgr2=0x7f78840b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.830+0000 7f7889800640 1 -- 172.21.15.129:0/1868537207 shutdown_connections 2024-09-17T03:50:19.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.830+0000 7f7889800640 1 -- 172.21.15.129:0/1868537207 wait complete. 2024-09-17T03:50:19.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.832+0000 7f7889800640 1 Processor -- start 2024-09-17T03:50:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.832+0000 7f7889800640 1 -- start start 2024-09-17T03:50:19.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840cf660 0x7f7884155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840cfb90 0x7f7884156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78840710a0 con 0x7f7884157590 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7884070f20 con 0x7f78840cfb90 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7889800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f78840b34f0 con 0x7f78840cf660 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.833+0000 7f7883400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49630/0 (socket says 172.21.15.129:49630) 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 -- 172.21.15.129:0/132771461 learned_addr learned my addr 172.21.15.129:0/132771461 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7882000640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840cfb90 0x7f7884156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7882a00640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840cf660 0x7f7884155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 -- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840cf660 msgr2=0x7f7884155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840cf660 0x7f7884155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 -- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840cfb90 msgr2=0x7f7884156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:19.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840cfb90 0x7f7884156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 -- 172.21.15.129:0/132771461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788415cd70 con 0x7f7884157590 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f7883400640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f787c00ca80 tx=0x7f787c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.834+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c007d90 con 0x7f7884157590 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.835+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f787c004c30 con 0x7f7884157590 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.835+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c005280 con 0x7f7884157590 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.835+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788415d050 con 0x7f7884157590 2024-09-17T03:50:19.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.835+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788415d610 con 0x7f7884157590 2024-09-17T03:50:19.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.836+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7884071740 con 0x7f7884157590 2024-09-17T03:50:19.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.837+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f787c005420 con 0x7f7884157590 2024-09-17T03:50:19.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.838+0000 7f787aa00640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:19.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.838+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f787c09b4c0 con 0x7f7884157590 2024-09-17T03:50:19.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.839+0000 7f7882a00640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:19.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.839+0000 7f7882a00640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78840c3830 tx=0x7f7874009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:19.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:19.843+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7884071740 con 0x7f7884157590 2024-09-17T03:50:20.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.095+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f78840b39a0 con 0x7f7884157590 2024-09-17T03:50:20.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.098+0000 7f787aa00640 1 -- 172.21.15.129:0/132771461 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v25) ==== 97+0+0 (secure 0 0 0) 0x7f787c014070 con 0x7f7884157590 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.100+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 msgr2=0x7f786407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.100+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78840c3830 tx=0x7f7874009040 comp rx=0 tx=0).stop 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.100+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 msgr2=0x7f7884156570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.100+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f787c00ca80 tx=0x7f787c00cf40 comp rx=0 tx=0).stop 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 shutdown_connections 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7884157590 0x7f7884156570 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840cfb90 0x7f7884156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 --2- 172.21.15.129:0/132771461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840cf660 0x7f7884155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 >> 172.21.15.129:0/132771461 conn(0x7f78840b1cb0 msgr2=0x7f78840b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:20.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 shutdown_connections 2024-09-17T03:50:20.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.101+0000 7f7889800640 1 -- 172.21.15.129:0/132771461 wait complete. 2024-09-17T03:50:20.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-17T03:50:20.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 -- 172.21.15.129:0/3083989848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b9bd0 msgr2=0x7fcbfc0b71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 --2- 172.21.15.129:0/3083989848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc0b71a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcbec005fd0 tx=0x7fcbec02a540 comp rx=0 tx=0).stop 2024-09-17T03:50:20.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 -- 172.21.15.129:0/3083989848 shutdown_connections 2024-09-17T03:50:20.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 --2- 172.21.15.129:0/3083989848 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcbfc162440 0x7fcbfc162850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 --2- 172.21.15.129:0/3083989848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc0b71a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 -- 172.21.15.129:0/3083989848 >> 172.21.15.129:0/3083989848 conn(0x7fcbfc09d6c0 msgr2=0x7fcbfc0d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:20.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 -- 172.21.15.129:0/3083989848 shutdown_connections 2024-09-17T03:50:20.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.348+0000 7fcbfb200640 1 -- 172.21.15.129:0/3083989848 wait complete. 2024-09-17T03:50:20.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.350+0000 7fcbfb200640 1 Processor -- start 2024-09-17T03:50:20.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.350+0000 7fcbfb200640 1 -- start start 2024-09-17T03:50:20.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc1606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcbfc162440 0x7fcbfc0cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0b5360 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0b51e0 con 0x7fcbfc0b9bd0 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbfb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0b54e0 con 0x7fcbfc162440 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49662/0 (socket says 172.21.15.129:49662) 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 -- 172.21.15.129:0/1501969330 learned_addr learned my addr 172.21.15.129:0/1501969330 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf1400640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc1606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 -- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcbfc162440 msgr2=0x7fcbfc0cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcbfc162440 0x7fcbfc0cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 -- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 msgr2=0x7fcbfc1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc1606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf9e00640 1 -- 172.21.15.129:0/1501969330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc0cb430 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.352+0000 7fcbf1400640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbf9e00640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006010 tx=0x7fcbec0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec03e030 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcbec003d90 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc0cb710 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec035040 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.353+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc0cbcd0 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.354+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc06a5e0 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.355+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcbec041030 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.356+0000 7fcbf2a00640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 0x7fcbd807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.357+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fcbec0bb270 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.357+0000 7fcbf1400640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 0x7fcbd807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.357+0000 7fcbf1400640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 0x7fcbd807b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4005860 tx=0x7fcbe4005dc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:20.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.361+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcbfc06a5e0 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.613+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fcbfc0b5a00 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.616+0000 7fcbf2a00640 1 -- 172.21.15.129:0/1501969330 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v26) ==== 90+0+0 (secure 0 0 0) 0x7fcbec083960 con 0x7fcbfc0b8fd0 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 msgr2=0x7fcbd807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 0x7fcbd807b0a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4005860 tx=0x7fcbe4005dc0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 msgr2=0x7fcbfc09e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcbec006010 tx=0x7fcbec0038a0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 shutdown_connections 2024-09-17T03:50:20.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcbd8078b90 0x7fcbd807b0a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcbfc162440 0x7fcbfc0cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcbfc0b9bd0 0x7fcbfc1606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 --2- 172.21.15.129:0/1501969330 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcbfc0b8fd0 0x7fcbfc09e030 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 >> 172.21.15.129:0/1501969330 conn(0x7fcbfc09d6c0 msgr2=0x7fcbfc0b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 shutdown_connections 2024-09-17T03:50:20.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.619+0000 7fcbfb200640 1 -- 172.21.15.129:0/1501969330 wait complete. 2024-09-17T03:50:20.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:50:20.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 -- 172.21.15.129:0/966975280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d015fbd0 msgr2=0x7f28d0152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 --2- 172.21.15.129:0/966975280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d015fbd0 0x7f28d0152360 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f28c0005fd0 tx=0x7f28c002aaa0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 -- 172.21.15.129:0/966975280 shutdown_connections 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 --2- 172.21.15.129:0/966975280 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d015fbd0 0x7f28d0152360 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 -- 172.21.15.129:0/966975280 >> 172.21.15.129:0/966975280 conn(0x7f28d006d410 msgr2=0x7f28d00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 -- 172.21.15.129:0/966975280 shutdown_connections 2024-09-17T03:50:20.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.866+0000 7f28cfe00640 1 -- 172.21.15.129:0/966975280 wait complete. 2024-09-17T03:50:20.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.868+0000 7f28cfe00640 1 Processor -- start 2024-09-17T03:50:20.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.868+0000 7f28cfe00640 1 -- start start 2024-09-17T03:50:20.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28d0072c30 0x7f28d00c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 0x7f28d00cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f28d00b4210 con 0x7f28d0148a70 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f28d00b4090 con 0x7f28d0072c30 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cfe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f28d00b4390 con 0x7f28d015b2a0 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49672/0 (socket says 172.21.15.129:49672) 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.870+0000 7f28cf400640 1 -- 172.21.15.129:0/1047589402 learned_addr learned my addr 172.21.15.129:0/1047589402 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cea00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28d0072c30 0x7f28d00c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28c6000640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 0x7f28d00cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 -- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 msgr2=0x7f28d00cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 0x7f28d00cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 -- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28d0072c30 msgr2=0x7f28d00c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28d0072c30 0x7f28d00c9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 -- 172.21.15.129:0/1047589402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d01f9530 con 0x7f28d0148a70 2024-09-17T03:50:20.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28c6000640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 0x7f28d00cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cf400640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c3c0 tx=0x7f28c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8017070 con 0x7f28d0148a70 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f28c80053c0 con 0x7f28d0148a70 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8003850 con 0x7f28d0148a70 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d0154260 con 0x7f28d0148a70 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.871+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d0154820 con 0x7f28d0148a70 2024-09-17T03:50:20.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.872+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d015fbd0 con 0x7f28d0148a70 2024-09-17T03:50:20.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.874+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f28c801a070 con 0x7f28d0148a70 2024-09-17T03:50:20.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.875+0000 7f28c7400640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 0x7f28ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:20.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.875+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f28c8099de0 con 0x7f28d0148a70 2024-09-17T03:50:20.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.876+0000 7f28cea00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 0x7f28ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:20.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.876+0000 7f28cea00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 0x7f28ac07b0a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c0035000 tx=0x7f28c0038040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:20.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:20.880+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f28d01528a0 con 0x7f28d0148a70 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.066+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f28d00750a0 con 0x7f28d0148a70 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.067+0000 7f28c7400640 1 -- 172.21.15.129:0/1047589402 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7f28c8062590 con 0x7f28d0148a70 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:21.067 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:50:21.068 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:50:21.068 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:21.068 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:50:21.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 msgr2=0x7f28ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:21.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 0x7f28ac07b0a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c0035000 tx=0x7f28c0038040 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 msgr2=0x7f28d0154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f28c800c3c0 tx=0x7f28c800c880 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 shutdown_connections 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ac078b90 0x7f28ac07b0a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f28d0148a70 0x7f28d0154d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f28d015b2a0 0x7f28d00cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 --2- 172.21.15.129:0/1047589402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28d0072c30 0x7f28d00c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 >> 172.21.15.129:0/1047589402 conn(0x7f28d006d410 msgr2=0x7f28d0068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 shutdown_connections 2024-09-17T03:50:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.069+0000 7f28cfe00640 1 -- 172.21.15.129:0/1047589402 wait complete. 2024-09-17T03:50:21.094 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:50:21.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:50:21.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 -- 172.21.15.129:0/3331366004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f610415c410 msgr2=0x7f61040cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:21.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 --2- 172.21.15.129:0/3331366004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f610415c410 0x7f61040cc5f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f60fc005fd0 tx=0x7f60fc02aaa0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 -- 172.21.15.129:0/3331366004 shutdown_connections 2024-09-17T03:50:21.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 --2- 172.21.15.129:0/3331366004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f610415c410 0x7f61040cc5f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 -- 172.21.15.129:0/3331366004 >> 172.21.15.129:0/3331366004 conn(0x7f610406ee70 msgr2=0x7f610406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:21.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 -- 172.21.15.129:0/3331366004 shutdown_connections 2024-09-17T03:50:21.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.319+0000 7f610a600640 1 -- 172.21.15.129:0/3331366004 wait complete. 2024-09-17T03:50:21.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.321+0000 7f610a600640 1 Processor -- start 2024-09-17T03:50:21.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.321+0000 7f610a600640 1 -- start start 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f610415c410 0x7f61041555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f610415a3d0 0x7f610415a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f61040cee40 con 0x7f6104073dd0 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f61040cecc0 con 0x7f610415a3d0 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f610a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f61040cefc0 con 0x7f610415c410 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49678/0 (socket says 172.21.15.129:49678) 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 -- 172.21.15.129:0/3296783368 learned_addr learned my addr 172.21.15.129:0/3296783368 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 -- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f610415c410 msgr2=0x7f61041555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:21.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f610415c410 0x7f61041555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 -- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f610415a3d0 msgr2=0x7f610415a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f610415a3d0 0x7f610415a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.323+0000 7f6109200640 1 -- 172.21.15.129:0/3296783368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610415ad20 con 0x7f6104073dd0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f6109200640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f60f800c3d0 tx=0x7f60f800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8017070 con 0x7f6104073dd0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f80053c0 con 0x7f6104073dd0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f610414d200 con 0x7f6104073dd0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8003850 con 0x7f6104073dd0 2024-09-17T03:50:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.324+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61041f8be0 con 0x7f6104073dd0 2024-09-17T03:50:21.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.325+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61040cf490 con 0x7f6104073dd0 2024-09-17T03:50:21.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.326+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f60f801a070 con 0x7f6104073dd0 2024-09-17T03:50:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.327+0000 7f6100c00640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 0x7f60e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.327+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f60f8099d40 con 0x7f6104073dd0 2024-09-17T03:50:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.328+0000 7f6103e00640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 0x7f60e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:21.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.329+0000 7f6103e00640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 0x7f60e407b0a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006fd0 tx=0x7f60f0006bc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.332+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f610415ad20 con 0x7f6104073dd0 2024-09-17T03:50:21.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:21.515+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f610415eab0 con 0x7f6104073dd0 2024-09-17T03:50:22.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.102+0000 7f6100c00640 1 -- 172.21.15.129:0/3296783368 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f60f8062560 con 0x7f6104073dd0 2024-09-17T03:50:22.102 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:50:22.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.105+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 msgr2=0x7f60e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:22.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.105+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 0x7f60e407b0a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f60f0006fd0 tx=0x7f60f0006bc0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.105+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 msgr2=0x7f61041f49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:22.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.105+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f60f800c3d0 tx=0x7f60f800c890 comp rx=0 tx=0).stop 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 shutdown_connections 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60e4078b90 0x7f60e407b0a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f610415a3d0 0x7f610415a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f610415c410 0x7f61041555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 --2- 172.21.15.129:0/3296783368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6104073dd0 0x7f61041f49a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.106+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 >> 172.21.15.129:0/3296783368 conn(0x7f610406ee70 msgr2=0x7f61040b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:22.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.107+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 shutdown_connections 2024-09-17T03:50:22.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.107+0000 7f610a600640 1 -- 172.21.15.129:0/3296783368 wait complete. 2024-09-17T03:50:22.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.364+0000 7fcb84200640 1 -- 172.21.15.129:0/1584646144 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0cc1e0 msgr2=0x7fcb7c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.364+0000 7fcb84200640 1 --2- 172.21.15.129:0/1584646144 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0cc1e0 0x7fcb7c0af3c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcb70005fd0 tx=0x7fcb7002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 -- 172.21.15.129:0/1584646144 shutdown_connections 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 --2- 172.21.15.129:0/1584646144 >> v2:172.21.15.181:3300/0 conn(0x7fcb7c0ccb50 0x7fcb7c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 --2- 172.21.15.129:0/1584646144 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0cc1e0 0x7fcb7c0af3c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 -- 172.21.15.129:0/1584646144 >> 172.21.15.129:0/1584646144 conn(0x7fcb7c06d7d0 msgr2=0x7fcb7c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 -- 172.21.15.129:0/1584646144 shutdown_connections 2024-09-17T03:50:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.365+0000 7fcb84200640 1 -- 172.21.15.129:0/1584646144 wait complete. 2024-09-17T03:50:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.367+0000 7fcb84200640 1 Processor -- start 2024-09-17T03:50:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.367+0000 7fcb84200640 1 -- start start 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.368+0000 7fcb84200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb7c0cc1e0 0x7fcb7c153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.368+0000 7fcb84200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0ccb50 0x7fcb7c149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb84200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb84200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c1533a0 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb84200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c15d400 con 0x7fcb7c0cc1e0 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb84200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7c153520 con 0x7fcb7c0ccb50 2024-09-17T03:50:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49686/0 (socket says 172.21.15.129:49686) 2024-09-17T03:50:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 -- 172.21.15.129:0/3507271811 learned_addr learned my addr 172.21.15.129:0/3507271811 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb81800640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb7c0cc1e0 0x7fcb7c153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb80e00640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0ccb50 0x7fcb7c149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 -- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0ccb50 msgr2=0x7fcb7c149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0ccb50 0x7fcb7c149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 -- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb7c0cc1e0 msgr2=0x7fcb7c153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb7c0cc1e0 0x7fcb7c153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.369+0000 7fcb82200640 1 -- 172.21.15.129:0/3507271811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c1541d0 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb82200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400ec20 tx=0x7fcb7400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74003b70 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c1f9280 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb74003d10 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74004280 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.370+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c1f9840 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.371+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c0af3c0 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.373+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcb74004420 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.373+0000 7fcb79600640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 0x7fcb5c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.374+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7fcb7409b710 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.374+0000 7fcb81800640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 0x7fcb5c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:22.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.375+0000 7fcb81800640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 0x7fcb5c07b0d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb70033000 tx=0x7fcb70004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:22.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.378+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcb74063650 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:22.562+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fcb7c153f60 con 0x7fcb7c15d8a0 2024-09-17T03:50:22.958 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-09-17T03:50:22.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-09-17T03:50:22.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-17T03:50:23.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.109+0000 7fcb79600640 1 -- 172.21.15.129:0/3507271811 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7fcb74068080 con 0x7fcb7c15d8a0 2024-09-17T03:50:23.110 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:50:23.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.112+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 msgr2=0x7fcb5c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:23.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.112+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 0x7fcb5c07b0d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb70033000 tx=0x7fcb70004730 comp rx=0 tx=0).stop 2024-09-17T03:50:23.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.112+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 msgr2=0x7fcb7c0adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:23.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.112+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400ec20 tx=0x7fcb7400c6a0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 shutdown_connections 2024-09-17T03:50:23.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb5c078bc0 0x7fcb5c07b0d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb7c15d8a0 0x7fcb7c0adfe0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb7c0ccb50 0x7fcb7c149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 --2- 172.21.15.129:0/3507271811 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb7c0cc1e0 0x7fcb7c153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 >> 172.21.15.129:0/3507271811 conn(0x7fcb7c06d7d0 msgr2=0x7fcb7c15c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.113+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 shutdown_connections 2024-09-17T03:50:23.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.114+0000 7fcb84200640 1 -- 172.21.15.129:0/3507271811 wait complete. 2024-09-17T03:50:23.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:50:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 -- 172.21.15.129:0/4137582530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0cc1e0 msgr2=0x7fca0c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 --2- 172.21.15.129:0/4137582530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0cc1e0 0x7fca0c0af3c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc005fd0 tx=0x7fc9fc02a540 comp rx=0 tx=0).stop 2024-09-17T03:50:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 -- 172.21.15.129:0/4137582530 shutdown_connections 2024-09-17T03:50:23.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 --2- 172.21.15.129:0/4137582530 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0ccb50 0x7fca0c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 --2- 172.21.15.129:0/4137582530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0cc1e0 0x7fca0c0af3c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 -- 172.21.15.129:0/4137582530 >> 172.21.15.129:0/4137582530 conn(0x7fca0c06d7d0 msgr2=0x7fca0c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:23.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 -- 172.21.15.129:0/4137582530 shutdown_connections 2024-09-17T03:50:23.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.381+0000 7fca12c00640 1 -- 172.21.15.129:0/4137582530 wait complete. 2024-09-17T03:50:23.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.383+0000 7fca12c00640 1 Processor -- start 2024-09-17T03:50:23.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.383+0000 7fca12c00640 1 -- start start 2024-09-17T03:50:23.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0cc1e0 0x7fca0c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:23.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:23.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca0c15d8a0 0x7fca0c15c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:23.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c153520 con 0x7fca0c0ccb50 2024-09-17T03:50:23.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c1533a0 con 0x7fca0c15d8a0 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca12c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fca0c1536a0 con 0x7fca0c0cc1e0 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49712/0 (socket says 172.21.15.129:49712) 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 -- 172.21.15.129:0/3607229074 learned_addr learned my addr 172.21.15.129:0/3607229074 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca0be00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0cc1e0 0x7fca0c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca10c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca0c15d8a0 0x7fca0c15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 -- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0cc1e0 msgr2=0x7fca0c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0cc1e0 0x7fca0c09aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 -- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca0c15d8a0 msgr2=0x7fca0c15c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca0c15d8a0 0x7fca0c15c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:23.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.385+0000 7fca03400640 1 -- 172.21.15.129:0/3607229074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c1f91d0 con 0x7fca0c0ccb50 2024-09-17T03:50:23.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca03400640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400b3e0 tx=0x7fc9f400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4015020 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9f4011940 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c1f94b0 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4011d90 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.386+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c1f9a70 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.387+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0c153bc0 con 0x7fca0c0ccb50 2024-09-17T03:50:23.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.388+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9f4018020 con 0x7fca0c0ccb50 2024-09-17T03:50:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.390+0000 7fca08c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 0x7fc9e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.390+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7fc9f409ab70 con 0x7fca0c0ccb50 2024-09-17T03:50:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.390+0000 7fca0be00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 0x7fc9e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:23.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.391+0000 7fca0be00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 0x7fc9e807b0a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc033000 tx=0x7fc9fc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:23.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.395+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9f401f020 con 0x7fca0c0ccb50 2024-09-17T03:50:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:23.615+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fca0c0cd970 con 0x7fca0c0ccb50 2024-09-17T03:50:24.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.111+0000 7fca08c00640 1 -- 172.21.15.129:0/3607229074 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7fc9f40629b0 con 0x7fca0c0ccb50 2024-09-17T03:50:24.112 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 4 and data pool 5 2024-09-17T03:50:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.116+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 msgr2=0x7fc9e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.116+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 0x7fc9e807b0a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc033000 tx=0x7fc9fc004730 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.116+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 msgr2=0x7fca0c15bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.116+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400b3e0 tx=0x7fc9f400b8a0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 shutdown_connections 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e8078b90 0x7fc9e807b0a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca0c15d8a0 0x7fca0c15c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca0c0ccb50 0x7fca0c15bb10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 --2- 172.21.15.129:0/3607229074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca0c0cc1e0 0x7fca0c09aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 >> 172.21.15.129:0/3607229074 conn(0x7fca0c06d7d0 msgr2=0x7fca0c06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:24.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 shutdown_connections 2024-09-17T03:50:24.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.118+0000 7fca12c00640 1 -- 172.21.15.129:0/3607229074 wait complete. 2024-09-17T03:50:24.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:24.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 -- 172.21.15.129:0/631430008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe00cf540 msgr2=0x7f9fe006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 --2- 172.21.15.129:0/631430008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe00cf540 0x7f9fe006ee70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc005fd0 tx=0x7f9fcc02a540 comp rx=0 tx=0).stop 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 -- 172.21.15.129:0/631430008 shutdown_connections 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 --2- 172.21.15.129:0/631430008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf950 0x7f9fe006f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 --2- 172.21.15.129:0/631430008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe00cf540 0x7f9fe006ee70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 -- 172.21.15.129:0/631430008 >> 172.21.15.129:0/631430008 conn(0x7f9fe006d410 msgr2=0x7f9fe00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 -- 172.21.15.129:0/631430008 shutdown_connections 2024-09-17T03:50:24.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.394+0000 7f9fe7000640 1 -- 172.21.15.129:0/631430008 wait complete. 2024-09-17T03:50:24.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.396+0000 7f9fe7000640 1 Processor -- start 2024-09-17T03:50:24.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.396+0000 7f9fe7000640 1 -- start start 2024-09-17T03:50:24.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf540 0x7f9fe01f9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 0x7f9fe0074220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe015ddd0 con 0x7f9fe00cf950 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe015dc50 con 0x7f9fe015ca50 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe7000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe015df50 con 0x7f9fe00cf540 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41106/0 (socket says 172.21.15.129:41106) 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5200640 1 -- 172.21.15.129:0/3947084633 learned_addr learned my addr 172.21.15.129:0/3947084633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5c00640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf540 0x7f9fe01f9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe6600640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 0x7f9fe0074220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5200640 1 -- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf540 msgr2=0x7f9fe01f9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.398+0000 7f9fe5200640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf540 0x7f9fe01f9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe5200640 1 -- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 msgr2=0x7f9fe0074220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe5200640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 0x7f9fe0074220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe5200640 1 -- 172.21.15.129:0/3947084633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe01f9d90 con 0x7f9fe00cf950 2024-09-17T03:50:24.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe6600640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 0x7f9fe0074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe5200640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc034040 tx=0x7f9fcc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc03e030 con 0x7f9fe00cf950 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fcc003d90 con 0x7f9fe00cf950 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe01f6800 con 0x7f9fe00cf950 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc035040 con 0x7f9fe00cf950 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.399+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01f5ed0 con 0x7f9fe00cf950 2024-09-17T03:50:24.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.400+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe0153ae0 con 0x7f9fe00cf950 2024-09-17T03:50:24.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.402+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9fcc041030 con 0x7f9fe00cf950 2024-09-17T03:50:24.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.402+0000 7f9fdd600640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 0x7f9fc007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.403+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f9fcc0bb830 con 0x7f9fe00cf950 2024-09-17T03:50:24.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.403+0000 7f9fe5c00640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 0x7f9fc007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.404+0000 7f9fe5c00640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 0x7f9fc007b0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40016c0 tx=0x7f9fd4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:24.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.408+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9fe01f9d90 con 0x7f9fe00cf950 2024-09-17T03:50:24.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.591+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fe015ebc0 con 0x7f9fe00cf950 2024-09-17T03:50:24.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.592+0000 7f9fdd600640 1 -- 172.21.15.129:0/3947084633 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16790 (secure 0 0 0) 0x7f9fcc083580 con 0x7f9fe00cf950 2024-09-17T03:50:24.592 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:24.593 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":32,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:23.616798+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:50:21.527684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-17T03:50:22.575211+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 msgr2=0x7f9fc007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 0x7f9fc007b0d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9fd40016c0 tx=0x7f9fd4009040 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 msgr2=0x7f9fe0144990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc034040 tx=0x7f9fcc0038a0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 shutdown_connections 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc0078bc0 0x7f9fc007b0d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe015ca50 0x7f9fe0074220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe00cf950 0x7f9fe0144990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 --2- 172.21.15.129:0/3947084633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe00cf540 0x7f9fe01f9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.594+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 >> 172.21.15.129:0/3947084633 conn(0x7f9fe006d410 msgr2=0x7f9fe0145890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:24.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.595+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 shutdown_connections 2024-09-17T03:50:24.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.595+0000 7f9fe7000640 1 -- 172.21.15.129:0/3947084633 wait complete. 2024-09-17T03:50:24.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:24.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 -- 172.21.15.129:0/3754347430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0cf550 msgr2=0x7fd4cc0d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/3754347430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc0d0c40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc08cff0 tx=0x7fd4b802a560 comp rx=0 tx=0).stop 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 -- 172.21.15.129:0/3754347430 shutdown_connections 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/3754347430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc0d0c40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/3754347430 >> v2:172.21.15.181:3300/0 conn(0x7fd4cc0b9c80 0x7fd4cc0cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 -- 172.21.15.129:0/3754347430 >> 172.21.15.129:0/3754347430 conn(0x7fd4cc09d680 msgr2=0x7fd4cc0d4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 -- 172.21.15.129:0/3754347430 shutdown_connections 2024-09-17T03:50:24.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.839+0000 7fd4d0c00640 1 -- 172.21.15.129:0/3754347430 wait complete. 2024-09-17T03:50:24.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.842+0000 7fd4d0c00640 1 Processor -- start 2024-09-17T03:50:24.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.842+0000 7fd4d0c00640 1 -- start start 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4cc0b9710 0x7fd4cc14db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc15eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc0d1300 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4cc0d1180 con 0x7fd4cc0cf550 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4d0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd4cc0d1480 con 0x7fd4cc0b9710 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4caa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4caa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41128/0 (socket says 172.21.15.129:41128) 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.843+0000 7fd4caa00640 1 -- 172.21.15.129:0/928577045 learned_addr learned my addr 172.21.15.129:0/928577045 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4cbe00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc15eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4cb400640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4cc0b9710 0x7fd4cc14db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 -- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4cc0b9710 msgr2=0x7fd4cc14db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4cc0b9710 0x7fd4cc14db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 -- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4cc0cf550 msgr2=0x7fd4cc15eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:24.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc15eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:24.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 -- 172.21.15.129:0/928577045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4cc15cbc0 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4caa00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8034040 tx=0x7fd4b8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:24.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.844+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b803e030 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.845+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4b8003d10 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.845+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b8035040 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.845+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4cc15f130 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.845+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4cc15f6f0 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.846+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4cc0cf010 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.847+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd4b8041030 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.848+0000 7fd4c3400640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 0x7fd4ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:24.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.848+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fd4b80bb550 con 0x7fd4cc0b9c80 2024-09-17T03:50:24.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.849+0000 7fd4cb400640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 0x7fd4ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:24.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.849+0000 7fd4cb400640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 0x7fd4ac07b0a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009b70 tx=0x7fd4bc005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:24.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:24.852+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4b80838f0 con 0x7fd4cc0b9c80 2024-09-17T03:50:25.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.037+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4cc08e380 con 0x7fd4cc0b9c80 2024-09-17T03:50:25.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.038+0000 7fd4c3400640 1 -- 172.21.15.129:0/928577045 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16790 (secure 0 0 0) 0x7fd4b80831f0 con 0x7fd4cc0b9c80 2024-09-17T03:50:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:25.039 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":32,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:23.616798+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:50:21.527684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-17T03:50:22.575211+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:25.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 msgr2=0x7fd4ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:25.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 0x7fd4ac07b0a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009b70 tx=0x7fd4bc005f30 comp rx=0 tx=0).stop 2024-09-17T03:50:25.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 msgr2=0x7fd4cc15e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:25.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8034040 tx=0x7fd4b8003820 comp rx=0 tx=0).stop 2024-09-17T03:50:25.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 shutdown_connections 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4ac078b90 0x7fd4ac07b0a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4cc0cf550 0x7fd4cc15eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4cc0b9c80 0x7fd4cc15e560 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 --2- 172.21.15.129:0/928577045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4cc0b9710 0x7fd4cc14db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 >> 172.21.15.129:0/928577045 conn(0x7fd4cc09d680 msgr2=0x7fd4cc06db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 shutdown_connections 2024-09-17T03:50:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.041+0000 7fd4d0c00640 1 -- 172.21.15.129:0/928577045 wait complete. 2024-09-17T03:50:25.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:50:25.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 -- 172.21.15.129:0/346159141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 msgr2=0x7f152415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:25.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 --2- 172.21.15.129:0/346159141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f152415c780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1518005fd0 tx=0x7f151802a540 comp rx=0 tx=0).stop 2024-09-17T03:50:25.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 -- 172.21.15.129:0/346159141 shutdown_connections 2024-09-17T03:50:25.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 --2- 172.21.15.129:0/346159141 >> v2:172.21.15.181:3300/0 conn(0x7f1524070440 0x7f15240b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 --2- 172.21.15.129:0/346159141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f152415c780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 -- 172.21.15.129:0/346159141 >> 172.21.15.129:0/346159141 conn(0x7f15240b1cb0 msgr2=0x7f15240b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:25.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 -- 172.21.15.129:0/346159141 shutdown_connections 2024-09-17T03:50:25.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.293+0000 7f152ba00640 1 -- 172.21.15.129:0/346159141 wait complete. 2024-09-17T03:50:25.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.295+0000 7f152ba00640 1 Processor -- start 2024-09-17T03:50:25.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.295+0000 7f152ba00640 1 -- start start 2024-09-17T03:50:25.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:25.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1524070440 0x7f15241595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f15241e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f15240b3b00 con 0x7f1524070030 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f15240b3980 con 0x7f1524070440 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f152ba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f15240b3c80 con 0x7f1524073170 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f1529000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f1529000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41138/0 (socket says 172.21.15.129:41138) 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f1529000640 1 -- 172.21.15.129:0/3195529383 learned_addr learned my addr 172.21.15.129:0/3195529383 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f1523e00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1524070440 0x7f15241595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.297+0000 7f1529a00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f15241e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 -- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 msgr2=0x7f15241e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f15241e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 -- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1524070440 msgr2=0x7f15241595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:25.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1524070440 0x7f15241595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 -- 172.21.15.129:0/3195529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152409dc30 con 0x7f1524070030 2024-09-17T03:50:25.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1529000640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1518006010 tx=0x7f15180038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151803e030 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1518003ca0 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15241f9430 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1518035040 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.298+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15241f99f0 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.299+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15240b4960 con 0x7f1524070030 2024-09-17T03:50:25.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.301+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1518041030 con 0x7f1524070030 2024-09-17T03:50:25.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.302+0000 7f1520c00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 0x7f150407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:25.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.302+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f15180bb2b0 con 0x7f1524070030 2024-09-17T03:50:25.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.302+0000 7f1523e00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 0x7f150407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:25.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.303+0000 7f1523e00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 0x7f150407b0a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f152409ab50 tx=0x7f1510009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:25.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.306+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1518083130 con 0x7f1524070030 2024-09-17T03:50:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:25.525+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f15240c3860 con 0x7f1524070030 2024-09-17T03:50:26.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.138+0000 7f1520c00640 1 -- 172.21.15.129:0/3195529383 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f1518087b60 con 0x7f1524070030 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.140+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 msgr2=0x7f150407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.140+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 0x7f150407b0a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f152409ab50 tx=0x7f1510009040 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.140+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 msgr2=0x7f152409d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.140+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1518006010 tx=0x7f15180038a0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 shutdown_connections 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1504078b90 0x7f150407b0a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1524073170 0x7f15241e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1524070440 0x7f15241595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 --2- 172.21.15.129:0/3195529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1524070030 0x7f152409d6f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 >> 172.21.15.129:0/3195529383 conn(0x7f15240b1cb0 msgr2=0x7f15240afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:26.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 shutdown_connections 2024-09-17T03:50:26.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.141+0000 7f152ba00640 1 -- 172.21.15.129:0/3195529383 wait complete. 2024-09-17T03:50:26.162 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:50:26.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:50:26.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 -- 172.21.15.129:0/2373461635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 msgr2=0x7fda1015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:26.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 --2- 172.21.15.129:0/2373461635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda1015c780 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fda00005fd0 tx=0x7fda0002a540 comp rx=0 tx=0).stop 2024-09-17T03:50:26.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 -- 172.21.15.129:0/2373461635 shutdown_connections 2024-09-17T03:50:26.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 --2- 172.21.15.129:0/2373461635 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 0x7fda10074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 --2- 172.21.15.129:0/2373461635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda1015c780 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 -- 172.21.15.129:0/2373461635 >> 172.21.15.129:0/2373461635 conn(0x7fda100b1cb0 msgr2=0x7fda100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 -- 172.21.15.129:0/2373461635 shutdown_connections 2024-09-17T03:50:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.385+0000 7fda17c00640 1 -- 172.21.15.129:0/2373461635 wait complete. 2024-09-17T03:50:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.387+0000 7fda17c00640 1 Processor -- start 2024-09-17T03:50:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.387+0000 7fda17c00640 1 -- start start 2024-09-17T03:50:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 0x7fda101586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 0x7fda10157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda1006f5a0 con 0x7fda10073170 2024-09-17T03:50:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fda10074f80 con 0x7fda10070440 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda17c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fda1006f720 con 0x7fda10070030 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41156/0 (socket says 172.21.15.129:41156) 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 -- 172.21.15.129:0/4076124590 learned_addr learned my addr 172.21.15.129:0/4076124590 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15200640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 0x7fda101586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda07e00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 0x7fda10157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 -- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 msgr2=0x7fda101586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 0x7fda101586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 -- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 msgr2=0x7fda10157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 0x7fda10157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 -- 172.21.15.129:0/4076124590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda10159220 con 0x7fda10073170 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda07e00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 0x7fda10157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda15c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c760 tx=0x7fda0800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08003b70 con 0x7fda10073170 2024-09-17T03:50:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.390+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda08003d10 con 0x7fda10073170 2024-09-17T03:50:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.391+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda10158f40 con 0x7fda10073170 2024-09-17T03:50:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.391+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda08005330 con 0x7fda10073170 2024-09-17T03:50:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.391+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda100c9320 con 0x7fda10073170 2024-09-17T03:50:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.392+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda10158cb0 con 0x7fda10073170 2024-09-17T03:50:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.393+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fda080054d0 con 0x7fda10073170 2024-09-17T03:50:26.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.394+0000 7fda0d600640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 0x7fd9ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:26.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.394+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fda0809aaf0 con 0x7fda10073170 2024-09-17T03:50:26.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.394+0000 7fda15200640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 0x7fd9ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:26.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.395+0000 7fda15200640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 0x7fd9ec07b0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda00033000 tx=0x7fda00004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:26.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.399+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fda08062e90 con 0x7fda10073170 2024-09-17T03:50:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:26.616+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fda100af030 con 0x7fda10073170 2024-09-17T03:50:27.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.145+0000 7fda0d600640 1 -- 172.21.15.129:0/4076124590 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v17) ==== 122+0+0 (secure 0 0 0) 0x7fda100af030 con 0x7fda10073170 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 msgr2=0x7fd9ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 0x7fd9ec07b0d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fda00033000 tx=0x7fda00004730 comp rx=0 tx=0).stop 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 msgr2=0x7fda101f8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fda0800c760 tx=0x7fda0800cb60 comp rx=0 tx=0).stop 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 shutdown_connections 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9ec078bc0 0x7fd9ec07b0d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda10073170 0x7fda101f8ab0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda10070440 0x7fda10157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 --2- 172.21.15.129:0/4076124590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda10070030 0x7fda101586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 >> 172.21.15.129:0/4076124590 conn(0x7fda100b1cb0 msgr2=0x7fda100ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 shutdown_connections 2024-09-17T03:50:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.148+0000 7fda17c00640 1 -- 172.21.15.129:0/4076124590 wait complete. 2024-09-17T03:50:27.169 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:50:27.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.388+0000 7fcc2c600640 1 -- 172.21.15.129:0/929709092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cf660 msgr2=0x7fcc240cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.388+0000 7fcc2c600640 1 --2- 172.21.15.129:0/929709092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cf660 0x7fcc240cf010 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcc2409e2c0 tx=0x7fcc1802a530 comp rx=0 tx=0).stop 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 -- 172.21.15.129:0/929709092 shutdown_connections 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 --2- 172.21.15.129:0/929709092 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 0x7fcc240ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 --2- 172.21.15.129:0/929709092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cf660 0x7fcc240cf010 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 -- 172.21.15.129:0/929709092 >> 172.21.15.129:0/929709092 conn(0x7fcc240b1cb0 msgr2=0x7fcc240b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:27.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 -- 172.21.15.129:0/929709092 shutdown_connections 2024-09-17T03:50:27.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.389+0000 7fcc2c600640 1 -- 172.21.15.129:0/929709092 wait complete. 2024-09-17T03:50:27.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.391+0000 7fcc2c600640 1 Processor -- start 2024-09-17T03:50:27.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.391+0000 7fcc2c600640 1 -- start start 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc240cf660 0x7fcc24158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 0x7fcc2409c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc24070f20 con 0x7fcc240cd070 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcc240af810 con 0x7fcc240cf660 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.392+0000 7fcc2c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcc240710a0 con 0x7fcc240cfb90 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41188/0 (socket says 172.21.15.129:41188) 2024-09-17T03:50:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 -- 172.21.15.129:0/3118701534 learned_addr learned my addr 172.21.15.129:0/3118701534 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:27.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29200640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc240cf660 0x7fcc24158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc2a600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 0x7fcc2409c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 -- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 msgr2=0x7fcc2409c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 0x7fcc2409c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 -- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc240cf660 msgr2=0x7fcc24158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc240cf660 0x7fcc24158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.393+0000 7fcc29c00640 1 -- 172.21.15.129:0/3118701534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2415dfe0 con 0x7fcc240cd070 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc29c00640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcc18034040 tx=0x7fcc180039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1803e030 con 0x7fcc240cd070 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc180047a0 con 0x7fcc240cd070 2024-09-17T03:50:27.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2415dd00 con 0x7fcc240cd070 2024-09-17T03:50:27.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18035040 con 0x7fcc240cd070 2024-09-17T03:50:27.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.394+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2415b560 con 0x7fcc240cd070 2024-09-17T03:50:27.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.395+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc240cf010 con 0x7fcc240cd070 2024-09-17T03:50:27.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.396+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcc18041030 con 0x7fcc240cd070 2024-09-17T03:50:27.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.397+0000 7fcc21600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 0x7fcc0407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.397+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fcc180bb5c0 con 0x7fcc240cd070 2024-09-17T03:50:27.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.398+0000 7fcc29200640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 0x7fcc0407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.399+0000 7fcc29200640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 0x7fcc0407b0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc240afcc0 tx=0x7fcc1000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:27.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.402+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcc18083310 con 0x7fcc240cd070 2024-09-17T03:50:27.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.619+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc24074e30 con 0x7fcc240cd070 2024-09-17T03:50:27.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.619+0000 7fcc21600640 1 -- 172.21.15.129:0/3118701534 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7fcc18087d40 con 0x7fcc240cd070 2024-09-17T03:50:27.618 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:27.619 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:27.619 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:27.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 msgr2=0x7fcc0407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 0x7fcc0407b0d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcc240afcc0 tx=0x7fcc1000b2e0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 msgr2=0x7fcc241f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcc18034040 tx=0x7fcc180039e0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 shutdown_connections 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc04078bc0 0x7fcc0407b0d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc240cfb90 0x7fcc2409c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc240cf660 0x7fcc24158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 --2- 172.21.15.129:0/3118701534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc240cd070 0x7fcc241f5490 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 >> 172.21.15.129:0/3118701534 conn(0x7fcc240b1cb0 msgr2=0x7fcc2415a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 shutdown_connections 2024-09-17T03:50:27.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.622+0000 7fcc2c600640 1 -- 172.21.15.129:0/3118701534 wait complete. 2024-09-17T03:50:27.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:27.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 -- 172.21.15.129:0/4057101202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 msgr2=0x7fc9a40cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 --2- 172.21.15.129:0/4057101202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a40cba00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc9a409e2e0 tx=0x7fc99402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 -- 172.21.15.129:0/4057101202 shutdown_connections 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 --2- 172.21.15.129:0/4057101202 >> v2:172.21.15.181:3300/0 conn(0x7fc9a40cad30 0x7fc9a406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 --2- 172.21.15.129:0/4057101202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a40cba00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 -- 172.21.15.129:0/4057101202 >> 172.21.15.129:0/4057101202 conn(0x7fc9a40b1cb0 msgr2=0x7fc9a40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 -- 172.21.15.129:0/4057101202 shutdown_connections 2024-09-17T03:50:27.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.866+0000 7fc9a9400640 1 -- 172.21.15.129:0/4057101202 wait complete. 2024-09-17T03:50:27.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.869+0000 7fc9a9400640 1 Processor -- start 2024-09-17T03:50:27.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.869+0000 7fc9a9400640 1 -- start start 2024-09-17T03:50:27.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9a40704e0 0x7fc9a415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a40cad30 0x7fc9a415cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a40ccf40 con 0x7fc9a40ca800 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a4074810 con 0x7fc9a40cad30 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a9400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a40cd0c0 con 0x7fc9a40704e0 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41204/0 (socket says 172.21.15.129:41204) 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.870+0000 7fc9a1e00640 1 -- 172.21.15.129:0/2761129490 learned_addr learned my addr 172.21.15.129:0/2761129490 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a2800640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9a40704e0 0x7fc9a415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a3200640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a40cad30 0x7fc9a415cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 -- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9a40704e0 msgr2=0x7fc9a415ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9a40704e0 0x7fc9a415ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 -- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a40cad30 msgr2=0x7fc9a415cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a40cad30 0x7fc9a415cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 -- 172.21.15.129:0/2761129490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a409aad0 con 0x7fc9a40ca800 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a1e00640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc99000d6d0 tx=0x7fc99000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990017070 con 0x7fc9a40ca800 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc990005330 con 0x7fc9a40ca800 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a415d4b0 con 0x7fc9a40ca800 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.871+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990009820 con 0x7fc9a40ca800 2024-09-17T03:50:27.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.872+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a415b2c0 con 0x7fc9a40ca800 2024-09-17T03:50:27.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.872+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a40cd5e0 con 0x7fc9a40ca800 2024-09-17T03:50:27.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.874+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9900099c0 con 0x7fc9a40ca800 2024-09-17T03:50:27.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.875+0000 7fc99aa00640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 0x7fc98407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:27.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.875+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc99009ae90 con 0x7fc9a40ca800 2024-09-17T03:50:27.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.875+0000 7fc9a2800640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 0x7fc98407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:27.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.876+0000 7fc9a2800640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 0x7fc98407b0a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc994033000 tx=0x7fc994004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:27.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:27.879+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9900632c0 con 0x7fc9a40ca800 2024-09-17T03:50:28.009 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T03:50:28.010+0000 7feaf8400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:50:28.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.062+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9a415d240 con 0x7fc9a40ca800 2024-09-17T03:50:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.064+0000 7fc99aa00640 1 -- 172.21.15.129:0/2761129490 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16777 (secure 0 0 0) 0x7fc990062bc0 con 0x7fc9a40ca800 2024-09-17T03:50:28.063 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:28.063 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":33,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:25.122284+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:50:21.527684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-17T03:50:22.575211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.066+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 msgr2=0x7fc98407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.066+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 0x7fc98407b0a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc994033000 tx=0x7fc994004730 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 msgr2=0x7fc9a415c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc99000d6d0 tx=0x7fc99000db90 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 shutdown_connections 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc984078b90 0x7fc98407b0a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a40cad30 0x7fc9a415cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a40ca800 0x7fc9a415c740 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 --2- 172.21.15.129:0/2761129490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9a40704e0 0x7fc9a415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 >> 172.21.15.129:0/2761129490 conn(0x7fc9a40b1cb0 msgr2=0x7fc9a4073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:28.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 shutdown_connections 2024-09-17T03:50:28.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.067+0000 7fc9a9400640 1 -- 172.21.15.129:0/2761129490 wait complete. 2024-09-17T03:50:28.087 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:50:28.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:50:28.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 -- 172.21.15.129:0/1537147250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 msgr2=0x7f5b4c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/1537147250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b4c004c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5b68072320 tx=0x7f5b4c005830 comp rx=0 tx=0).stop 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ce00640 1 -- 172.21.15.129:0/1537147250 reap_dead start 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 -- 172.21.15.129:0/1537147250 shutdown_connections 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 -- 172.21.15.129:0/1537147250 >> 172.21.15.129:0/1537147250 conn(0x7f5b680b1cb0 msgr2=0x7f5b680b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 -- 172.21.15.129:0/1537147250 shutdown_connections 2024-09-17T03:50:28.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.311+0000 7f5b6ee00640 1 -- 172.21.15.129:0/1537147250 wait complete. 2024-09-17T03:50:28.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.314+0000 7f5b6ee00640 1 Processor -- start 2024-09-17T03:50:28.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.314+0000 7f5b6ee00640 1 -- start start 2024-09-17T03:50:28.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b681e6a30 0x7f5b681e6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b6815df90 0x7f5b6815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b680b35d0 con 0x7f5b4c004800 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b6815ccc0 con 0x7f5b6815df90 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b6ee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5b680b3750 con 0x7f5b681e6a30 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.315+0000 7f5b67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41214/0 (socket says 172.21.15.129:41214) 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 -- 172.21.15.129:0/3310144337 learned_addr learned my addr 172.21.15.129:0/3310144337 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:28.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b5fe00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b681e6a30 0x7f5b681e6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b6ce00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b6815df90 0x7f5b6815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 -- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b681e6a30 msgr2=0x7f5b681e6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b681e6a30 0x7f5b681e6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 -- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b6815df90 msgr2=0x7f5b6815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b6815df90 0x7f5b6815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 -- 172.21.15.129:0/3310144337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b6815e8e0 con 0x7f5b4c004800 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b67e00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5b58011a90 tx=0x7f5b58011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.316+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5801e070 con 0x7f5b4c004800 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.317+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b5800fe70 con 0x7f5b4c004800 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.317+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b680d07a0 con 0x7f5b4c004800 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.317+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58019690 con 0x7f5b4c004800 2024-09-17T03:50:28.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.317+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b681e6350 con 0x7f5b4c004800 2024-09-17T03:50:28.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.318+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b680b3090 con 0x7f5b4c004800 2024-09-17T03:50:28.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.319+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5b58021070 con 0x7f5b4c004800 2024-09-17T03:50:28.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.320+0000 7f5b64c00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 0x7f5b4407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.320+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f5b5809b7a0 con 0x7f5b4c004800 2024-09-17T03:50:28.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.320+0000 7f5b5fe00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 0x7f5b4407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.321+0000 7f5b5fe00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 0x7f5b4407b0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5b50033000 tx=0x7f5b5002d000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:28.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.324+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5b6815e8e0 con 0x7f5b4c004800 2024-09-17T03:50:28.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.504+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5b680c3860 con 0x7f5b44078b90 2024-09-17T03:50:28.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.510+0000 7f5b64c00640 1 -- 172.21.15.129:0/3310144337 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251190 (secure 0 0 0) 0x7f5b50082550 con 0x7f5b44078b90 2024-09-17T03:50:28.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:28.514 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 msgr2=0x7f5b4407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 0x7f5b4407b0a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5b50033000 tx=0x7f5b5002d000 comp rx=0 tx=0).stop 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 msgr2=0x7f5b681e5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5b58011a90 tx=0x7f5b58011f50 comp rx=0 tx=0).stop 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 shutdown_connections 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b44078b90 0x7f5b4407b0a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b6815df90 0x7f5b6815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b681e6a30 0x7f5b681e6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 --2- 172.21.15.129:0/3310144337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b4c004800 0x7f5b681e5e10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 >> 172.21.15.129:0/3310144337 conn(0x7f5b680b1cb0 msgr2=0x7f5b680b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:28.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.514+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 shutdown_connections 2024-09-17T03:50:28.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.515+0000 7f5b6ee00640 1 -- 172.21.15.129:0/3310144337 wait complete. 2024-09-17T03:50:28.534 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-17T03:50:28.306834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224180,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750540,"statfs":{"total":773094113280,"available":772864552960,"internally_reserved":0,"allocated":3883008,"data_stored":2186530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001306"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124897+0000","last_change":"2024-09-17T03:50:23.124897+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124897+0000","last_undegraded":"2024-09-17T03:50:23.124897+0000","last_fullsized":"2024-09-17T03:50:23.124897+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149743+0000","last_change":"2024-09-17T03:50:24.146222+0000","last_active":"2024-09-17T03:50:24.149743+0000","last_peered":"2024-09-17T03:50:24.149743+0000","last_clean":"2024-09-17T03:50:24.149743+0000","last_became_active":"2024-09-17T03:50:24.145925+0000","last_became_peered":"2024-09-17T03:50:24.145925+0000","last_unstale":"2024-09-17T03:50:24.149743+0000","last_undegraded":"2024-09-17T03:50:24.149743+0000","last_fullsized":"2024-09-17T03:50:24.149743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150192+0000","last_change":"2024-09-17T03:50:24.146390+0000","last_active":"2024-09-17T03:50:24.150192+0000","last_peered":"2024-09-17T03:50:24.150192+0000","last_clean":"2024-09-17T03:50:24.150192+0000","last_became_active":"2024-09-17T03:50:24.146067+0000","last_became_peered":"2024-09-17T03:50:24.146067+0000","last_unstale":"2024-09-17T03:50:24.150192+0000","last_undegraded":"2024-09-17T03:50:24.150192+0000","last_fullsized":"2024-09-17T03:50:24.150192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161136+0000","last_change":"2024-09-17T03:50:24.156017+0000","last_active":"2024-09-17T03:50:24.161136+0000","last_peered":"2024-09-17T03:50:24.161136+0000","last_clean":"2024-09-17T03:50:24.161136+0000","last_became_active":"2024-09-17T03:50:24.155677+0000","last_became_peered":"2024-09-17T03:50:24.155677+0000","last_unstale":"2024-09-17T03:50:24.161136+0000","last_undegraded":"2024-09-17T03:50:24.161136+0000","last_fullsized":"2024-09-17T03:50:24.161136+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:13.449411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145398+0000","last_change":"2024-09-17T03:50:24.138985+0000","last_active":"2024-09-17T03:50:24.145398+0000","last_peered":"2024-09-17T03:50:24.145398+0000","last_clean":"2024-09-17T03:50:24.145398+0000","last_became_active":"2024-09-17T03:50:24.138569+0000","last_became_peered":"2024-09-17T03:50:24.138569+0000","last_unstale":"2024-09-17T03:50:24.145398+0000","last_undegraded":"2024-09-17T03:50:24.145398+0000","last_fullsized":"2024-09-17T03:50:24.145398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:51.159427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158315+0000","last_change":"2024-09-17T03:50:24.147892+0000","last_active":"2024-09-17T03:50:24.158315+0000","last_peered":"2024-09-17T03:50:24.158315+0000","last_clean":"2024-09-17T03:50:24.158315+0000","last_became_active":"2024-09-17T03:50:24.147616+0000","last_became_peered":"2024-09-17T03:50:24.147616+0000","last_unstale":"2024-09-17T03:50:24.158315+0000","last_undegraded":"2024-09-17T03:50:24.158315+0000","last_fullsized":"2024-09-17T03:50:24.158315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152457+0000","last_change":"2024-09-17T03:50:24.148414+0000","last_active":"2024-09-17T03:50:24.152457+0000","last_peered":"2024-09-17T03:50:24.152457+0000","last_clean":"2024-09-17T03:50:24.152457+0000","last_became_active":"2024-09-17T03:50:24.148020+0000","last_became_peered":"2024-09-17T03:50:24.148020+0000","last_unstale":"2024-09-17T03:50:24.152457+0000","last_undegraded":"2024-09-17T03:50:24.152457+0000","last_fullsized":"2024-09-17T03:50:24.152457+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150293+0000","last_change":"2024-09-17T03:50:24.147045+0000","last_active":"2024-09-17T03:50:24.150293+0000","last_peered":"2024-09-17T03:50:24.150293+0000","last_clean":"2024-09-17T03:50:24.150293+0000","last_became_active":"2024-09-17T03:50:24.146692+0000","last_became_peered":"2024-09-17T03:50:24.146692+0000","last_unstale":"2024-09-17T03:50:24.150293+0000","last_undegraded":"2024-09-17T03:50:24.150293+0000","last_fullsized":"2024-09-17T03:50:24.150293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155032+0000","last_change":"2024-09-17T03:50:24.149261+0000","last_active":"2024-09-17T03:50:24.155032+0000","last_peered":"2024-09-17T03:50:24.155032+0000","last_clean":"2024-09-17T03:50:24.155032+0000","last_became_active":"2024-09-17T03:50:24.148898+0000","last_became_peered":"2024-09-17T03:50:24.148898+0000","last_unstale":"2024-09-17T03:50:24.155032+0000","last_undegraded":"2024-09-17T03:50:24.155032+0000","last_fullsized":"2024-09-17T03:50:24.155032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:20.858522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160355+0000","last_change":"2024-09-17T03:50:24.152445+0000","last_active":"2024-09-17T03:50:24.160355+0000","last_peered":"2024-09-17T03:50:24.160355+0000","last_clean":"2024-09-17T03:50:24.160355+0000","last_became_active":"2024-09-17T03:50:24.152179+0000","last_became_peered":"2024-09-17T03:50:24.152179+0000","last_unstale":"2024-09-17T03:50:24.160355+0000","last_undegraded":"2024-09-17T03:50:24.160355+0000","last_fullsized":"2024-09-17T03:50:24.160355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:59.856041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157759+0000","last_change":"2024-09-17T03:50:24.155248+0000","last_active":"2024-09-17T03:50:24.157759+0000","last_peered":"2024-09-17T03:50:24.157759+0000","last_clean":"2024-09-17T03:50:24.157759+0000","last_became_active":"2024-09-17T03:50:24.154779+0000","last_became_peered":"2024-09-17T03:50:24.154779+0000","last_unstale":"2024-09-17T03:50:24.157759+0000","last_undegraded":"2024-09-17T03:50:24.157759+0000","last_fullsized":"2024-09-17T03:50:24.157759+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143858+0000","last_change":"2024-09-17T03:50:24.138736+0000","last_active":"2024-09-17T03:50:24.143858+0000","last_peered":"2024-09-17T03:50:24.143858+0000","last_clean":"2024-09-17T03:50:24.143858+0000","last_became_active":"2024-09-17T03:50:24.138424+0000","last_became_peered":"2024-09-17T03:50:24.138424+0000","last_unstale":"2024-09-17T03:50:24.143858+0000","last_undegraded":"2024-09-17T03:50:24.143858+0000","last_fullsized":"2024-09-17T03:50:24.143858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145922+0000","last_change":"2024-09-17T03:50:24.142434+0000","last_active":"2024-09-17T03:50:24.145922+0000","last_peered":"2024-09-17T03:50:24.145922+0000","last_clean":"2024-09-17T03:50:24.145922+0000","last_became_active":"2024-09-17T03:50:24.142083+0000","last_became_peered":"2024-09-17T03:50:24.142083+0000","last_unstale":"2024-09-17T03:50:24.145922+0000","last_undegraded":"2024-09-17T03:50:24.145922+0000","last_fullsized":"2024-09-17T03:50:24.145922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:49.596755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160739+0000","last_change":"2024-09-17T03:50:24.154707+0000","last_active":"2024-09-17T03:50:24.160739+0000","last_peered":"2024-09-17T03:50:24.160739+0000","last_clean":"2024-09-17T03:50:24.160739+0000","last_became_active":"2024-09-17T03:50:24.154328+0000","last_became_peered":"2024-09-17T03:50:24.154328+0000","last_unstale":"2024-09-17T03:50:24.160739+0000","last_undegraded":"2024-09-17T03:50:24.160739+0000","last_fullsized":"2024-09-17T03:50:24.160739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145415+0000","last_change":"2024-09-17T03:50:24.125931+0000","last_active":"2024-09-17T03:50:24.145415+0000","last_peered":"2024-09-17T03:50:24.145415+0000","last_clean":"2024-09-17T03:50:24.145415+0000","last_became_active":"2024-09-17T03:50:24.125300+0000","last_became_peered":"2024-09-17T03:50:24.125300+0000","last_unstale":"2024-09-17T03:50:24.145415+0000","last_undegraded":"2024-09-17T03:50:24.145415+0000","last_fullsized":"2024-09-17T03:50:24.145415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145644+0000","last_change":"2024-09-17T03:50:24.144536+0000","last_active":"2024-09-17T03:50:24.145644+0000","last_peered":"2024-09-17T03:50:24.145644+0000","last_clean":"2024-09-17T03:50:24.145644+0000","last_became_active":"2024-09-17T03:50:24.143960+0000","last_became_peered":"2024-09-17T03:50:24.143960+0000","last_unstale":"2024-09-17T03:50:24.145644+0000","last_undegraded":"2024-09-17T03:50:24.145644+0000","last_fullsized":"2024-09-17T03:50:24.145644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152188+0000","last_change":"2024-09-17T03:50:24.147697+0000","last_active":"2024-09-17T03:50:24.152188+0000","last_peered":"2024-09-17T03:50:24.152188+0000","last_clean":"2024-09-17T03:50:24.152188+0000","last_became_active":"2024-09-17T03:50:24.146930+0000","last_became_peered":"2024-09-17T03:50:24.146930+0000","last_unstale":"2024-09-17T03:50:24.152188+0000","last_undegraded":"2024-09-17T03:50:24.152188+0000","last_fullsized":"2024-09-17T03:50:24.152188+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162682+0000","last_change":"2024-09-17T03:50:24.159788+0000","last_active":"2024-09-17T03:50:24.162682+0000","last_peered":"2024-09-17T03:50:24.162682+0000","last_clean":"2024-09-17T03:50:24.162682+0000","last_became_active":"2024-09-17T03:50:24.159505+0000","last_became_peered":"2024-09-17T03:50:24.159505+0000","last_unstale":"2024-09-17T03:50:24.162682+0000","last_undegraded":"2024-09-17T03:50:24.162682+0000","last_fullsized":"2024-09-17T03:50:24.162682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161542+0000","last_change":"2024-09-17T03:50:24.154330+0000","last_active":"2024-09-17T03:50:24.161542+0000","last_peered":"2024-09-17T03:50:24.161542+0000","last_clean":"2024-09-17T03:50:24.161542+0000","last_became_active":"2024-09-17T03:50:24.153860+0000","last_became_peered":"2024-09-17T03:50:24.153860+0000","last_unstale":"2024-09-17T03:50:24.161542+0000","last_undegraded":"2024-09-17T03:50:24.161542+0000","last_fullsized":"2024-09-17T03:50:24.161542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.132028+0000","last_change":"2024-09-17T03:50:24.128012+0000","last_active":"2024-09-17T03:50:24.132028+0000","last_peered":"2024-09-17T03:50:24.132028+0000","last_clean":"2024-09-17T03:50:24.132028+0000","last_became_active":"2024-09-17T03:50:24.127633+0000","last_became_peered":"2024-09-17T03:50:24.127633+0000","last_unstale":"2024-09-17T03:50:24.132028+0000","last_undegraded":"2024-09-17T03:50:24.132028+0000","last_fullsized":"2024-09-17T03:50:24.132028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:14.977204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155095+0000","last_change":"2024-09-17T03:50:24.149865+0000","last_active":"2024-09-17T03:50:24.155095+0000","last_peered":"2024-09-17T03:50:24.155095+0000","last_clean":"2024-09-17T03:50:24.155095+0000","last_became_active":"2024-09-17T03:50:24.149569+0000","last_became_peered":"2024-09-17T03:50:24.149569+0000","last_unstale":"2024-09-17T03:50:24.155095+0000","last_undegraded":"2024-09-17T03:50:24.155095+0000","last_fullsized":"2024-09-17T03:50:24.155095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:02.922452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155400+0000","last_change":"2024-09-17T03:50:24.151774+0000","last_active":"2024-09-17T03:50:24.155400+0000","last_peered":"2024-09-17T03:50:24.155400+0000","last_clean":"2024-09-17T03:50:24.155400+0000","last_became_active":"2024-09-17T03:50:24.151379+0000","last_became_peered":"2024-09-17T03:50:24.151379+0000","last_unstale":"2024-09-17T03:50:24.155400+0000","last_undegraded":"2024-09-17T03:50:24.155400+0000","last_fullsized":"2024-09-17T03:50:24.155400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116168+0000","last_change":"2024-09-17T03:50:23.116168+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116168+0000","last_undegraded":"2024-09-17T03:50:23.116168+0000","last_fullsized":"2024-09-17T03:50:23.116168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.138232+0000","last_change":"2024-09-17T03:50:24.132642+0000","last_active":"2024-09-17T03:50:24.138232+0000","last_peered":"2024-09-17T03:50:24.138232+0000","last_clean":"2024-09-17T03:50:24.138232+0000","last_became_active":"2024-09-17T03:50:24.131051+0000","last_became_peered":"2024-09-17T03:50:24.131051+0000","last_unstale":"2024-09-17T03:50:24.138232+0000","last_undegraded":"2024-09-17T03:50:24.138232+0000","last_fullsized":"2024-09-17T03:50:24.138232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153734+0000","last_change":"2024-09-17T03:50:24.145119+0000","last_active":"2024-09-17T03:50:24.153734+0000","last_peered":"2024-09-17T03:50:24.153734+0000","last_clean":"2024-09-17T03:50:24.153734+0000","last_became_active":"2024-09-17T03:50:24.144619+0000","last_became_peered":"2024-09-17T03:50:24.144619+0000","last_unstale":"2024-09-17T03:50:24.153734+0000","last_undegraded":"2024-09-17T03:50:24.153734+0000","last_fullsized":"2024-09-17T03:50:24.153734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:44.451270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142707+0000","last_change":"2024-09-17T03:50:24.136492+0000","last_active":"2024-09-17T03:50:24.142707+0000","last_peered":"2024-09-17T03:50:24.142707+0000","last_clean":"2024-09-17T03:50:24.142707+0000","last_became_active":"2024-09-17T03:50:24.136206+0000","last_became_peered":"2024-09-17T03:50:24.136206+0000","last_unstale":"2024-09-17T03:50:24.142707+0000","last_undegraded":"2024-09-17T03:50:24.142707+0000","last_fullsized":"2024-09-17T03:50:24.142707+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161913+0000","last_change":"2024-09-17T03:50:24.155949+0000","last_active":"2024-09-17T03:50:24.161913+0000","last_peered":"2024-09-17T03:50:24.161913+0000","last_clean":"2024-09-17T03:50:24.161913+0000","last_became_active":"2024-09-17T03:50:24.155692+0000","last_became_peered":"2024-09-17T03:50:24.155692+0000","last_unstale":"2024-09-17T03:50:24.161913+0000","last_undegraded":"2024-09-17T03:50:24.161913+0000","last_fullsized":"2024-09-17T03:50:24.161913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:11.942003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142811+0000","last_change":"2024-09-17T03:50:24.136298+0000","last_active":"2024-09-17T03:50:24.142811+0000","last_peered":"2024-09-17T03:50:24.142811+0000","last_clean":"2024-09-17T03:50:24.142811+0000","last_became_active":"2024-09-17T03:50:24.135828+0000","last_became_peered":"2024-09-17T03:50:24.135828+0000","last_unstale":"2024-09-17T03:50:24.142811+0000","last_undegraded":"2024-09-17T03:50:24.142811+0000","last_fullsized":"2024-09-17T03:50:24.142811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155375+0000","last_change":"2024-09-17T03:50:24.152059+0000","last_active":"2024-09-17T03:50:24.155375+0000","last_peered":"2024-09-17T03:50:24.155375+0000","last_clean":"2024-09-17T03:50:24.155375+0000","last_became_active":"2024-09-17T03:50:24.151751+0000","last_became_peered":"2024-09-17T03:50:24.151751+0000","last_unstale":"2024-09-17T03:50:24.155375+0000","last_undegraded":"2024-09-17T03:50:24.155375+0000","last_fullsized":"2024-09-17T03:50:24.155375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152180+0000","last_change":"2024-09-17T03:50:24.147844+0000","last_active":"2024-09-17T03:50:24.152180+0000","last_peered":"2024-09-17T03:50:24.152180+0000","last_clean":"2024-09-17T03:50:24.152180+0000","last_became_active":"2024-09-17T03:50:24.144862+0000","last_became_peered":"2024-09-17T03:50:24.144862+0000","last_unstale":"2024-09-17T03:50:24.152180+0000","last_undegraded":"2024-09-17T03:50:24.152180+0000","last_fullsized":"2024-09-17T03:50:24.152180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.123020+0000","last_change":"2024-09-17T03:50:23.123020+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.123020+0000","last_undegraded":"2024-09-17T03:50:23.123020+0000","last_fullsized":"2024-09-17T03:50:23.123020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157205+0000","last_change":"2024-09-17T03:50:24.153513+0000","last_active":"2024-09-17T03:50:24.157205+0000","last_peered":"2024-09-17T03:50:24.157205+0000","last_clean":"2024-09-17T03:50:24.157205+0000","last_became_active":"2024-09-17T03:50:24.153118+0000","last_became_peered":"2024-09-17T03:50:24.153118+0000","last_unstale":"2024-09-17T03:50:24.157205+0000","last_undegraded":"2024-09-17T03:50:24.157205+0000","last_fullsized":"2024-09-17T03:50:24.157205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116164+0000","last_change":"2024-09-17T03:50:23.116164+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116164+0000","last_undegraded":"2024-09-17T03:50:23.116164+0000","last_fullsized":"2024-09-17T03:50:23.116164+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142829+0000","last_change":"2024-09-17T03:50:23.138230+0000","last_active":"2024-09-17T03:50:23.142829+0000","last_peered":"2024-09-17T03:50:23.142829+0000","last_clean":"2024-09-17T03:50:23.142829+0000","last_became_active":"2024-09-17T03:50:23.137347+0000","last_became_peered":"2024-09-17T03:50:23.137347+0000","last_unstale":"2024-09-17T03:50:23.142829+0000","last_undegraded":"2024-09-17T03:50:23.142829+0000","last_fullsized":"2024-09-17T03:50:23.142829+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124400+0000","last_change":"2024-09-17T03:50:23.124400+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124400+0000","last_undegraded":"2024-09-17T03:50:23.124400+0000","last_fullsized":"2024-09-17T03:50:23.124400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131789+0000","last_change":"2024-09-17T03:50:23.128500+0000","last_active":"2024-09-17T03:50:23.131789+0000","last_peered":"2024-09-17T03:50:23.131789+0000","last_clean":"2024-09-17T03:50:23.131789+0000","last_became_active":"2024-09-17T03:50:23.128184+0000","last_became_peered":"2024-09-17T03:50:23.128184+0000","last_unstale":"2024-09-17T03:50:23.131789+0000","last_undegraded":"2024-09-17T03:50:23.131789+0000","last_fullsized":"2024-09-17T03:50:23.131789+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155490+0000","last_change":"2024-09-17T03:50:24.152519+0000","last_active":"2024-09-17T03:50:24.155490+0000","last_peered":"2024-09-17T03:50:24.155490+0000","last_clean":"2024-09-17T03:50:24.155490+0000","last_became_active":"2024-09-17T03:50:24.150781+0000","last_became_peered":"2024-09-17T03:50:24.150781+0000","last_unstale":"2024-09-17T03:50:24.155490+0000","last_undegraded":"2024-09-17T03:50:24.155490+0000","last_fullsized":"2024-09-17T03:50:24.155490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158511+0000","last_change":"2024-09-17T03:50:23.132879+0000","last_active":"2024-09-17T03:50:24.158511+0000","last_peered":"2024-09-17T03:50:24.158511+0000","last_clean":"2024-09-17T03:50:24.158511+0000","last_became_active":"2024-09-17T03:50:23.131983+0000","last_became_peered":"2024-09-17T03:50:23.131983+0000","last_unstale":"2024-09-17T03:50:24.158511+0000","last_undegraded":"2024-09-17T03:50:24.158511+0000","last_fullsized":"2024-09-17T03:50:24.158511+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:39.860630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161940+0000","last_change":"2024-09-17T03:50:24.156534+0000","last_active":"2024-09-17T03:50:24.161940+0000","last_peered":"2024-09-17T03:50:24.161940+0000","last_clean":"2024-09-17T03:50:24.161940+0000","last_became_active":"2024-09-17T03:50:24.156249+0000","last_became_peered":"2024-09-17T03:50:24.156249+0000","last_unstale":"2024-09-17T03:50:24.161940+0000","last_undegraded":"2024-09-17T03:50:24.161940+0000","last_fullsized":"2024-09-17T03:50:24.161940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:41.217570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154140+0000","last_change":"2024-09-17T03:50:23.134919+0000","last_active":"2024-09-17T03:50:24.154140+0000","last_peered":"2024-09-17T03:50:24.154140+0000","last_clean":"2024-09-17T03:50:24.154140+0000","last_became_active":"2024-09-17T03:50:23.134136+0000","last_became_peered":"2024-09-17T03:50:23.134136+0000","last_unstale":"2024-09-17T03:50:24.154140+0000","last_undegraded":"2024-09-17T03:50:24.154140+0000","last_fullsized":"2024-09-17T03:50:24.154140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:06.997836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157723+0000","last_change":"2024-09-17T03:50:24.153900+0000","last_active":"2024-09-17T03:50:24.157723+0000","last_peered":"2024-09-17T03:50:24.157723+0000","last_clean":"2024-09-17T03:50:24.157723+0000","last_became_active":"2024-09-17T03:50:24.153608+0000","last_became_peered":"2024-09-17T03:50:24.153608+0000","last_unstale":"2024-09-17T03:50:24.157723+0000","last_undegraded":"2024-09-17T03:50:24.157723+0000","last_fullsized":"2024-09-17T03:50:24.157723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160395+0000","last_change":"2024-09-17T03:50:23.142748+0000","last_active":"2024-09-17T03:50:24.160395+0000","last_peered":"2024-09-17T03:50:24.160395+0000","last_clean":"2024-09-17T03:50:24.160395+0000","last_became_active":"2024-09-17T03:50:23.142402+0000","last_became_peered":"2024-09-17T03:50:23.142402+0000","last_unstale":"2024-09-17T03:50:24.160395+0000","last_undegraded":"2024-09-17T03:50:24.160395+0000","last_fullsized":"2024-09-17T03:50:24.160395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:26.214882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.148116+0000","last_change":"2024-09-17T03:50:24.143306+0000","last_active":"2024-09-17T03:50:24.148116+0000","last_peered":"2024-09-17T03:50:24.148116+0000","last_clean":"2024-09-17T03:50:24.148116+0000","last_became_active":"2024-09-17T03:50:24.143003+0000","last_became_peered":"2024-09-17T03:50:24.143003+0000","last_unstale":"2024-09-17T03:50:24.148116+0000","last_undegraded":"2024-09-17T03:50:24.148116+0000","last_fullsized":"2024-09-17T03:50:24.148116+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160380+0000","last_change":"2024-09-17T03:50:23.144925+0000","last_active":"2024-09-17T03:50:24.160380+0000","last_peered":"2024-09-17T03:50:24.160380+0000","last_clean":"2024-09-17T03:50:24.160380+0000","last_became_active":"2024-09-17T03:50:23.144566+0000","last_became_peered":"2024-09-17T03:50:23.144566+0000","last_unstale":"2024-09-17T03:50:24.160380+0000","last_undegraded":"2024-09-17T03:50:24.160380+0000","last_fullsized":"2024-09-17T03:50:24.160380+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:11.013666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155822+0000","last_change":"2024-09-17T03:50:24.151450+0000","last_active":"2024-09-17T03:50:24.155822+0000","last_peered":"2024-09-17T03:50:24.155822+0000","last_clean":"2024-09-17T03:50:24.155822+0000","last_became_active":"2024-09-17T03:50:24.151115+0000","last_became_peered":"2024-09-17T03:50:24.151115+0000","last_unstale":"2024-09-17T03:50:24.155822+0000","last_undegraded":"2024-09-17T03:50:24.155822+0000","last_fullsized":"2024-09-17T03:50:24.155822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.002921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124578+0000","last_change":"2024-09-17T03:50:23.133927+0000","last_active":"2024-09-17T03:50:24.124578+0000","last_peered":"2024-09-17T03:50:24.124578+0000","last_clean":"2024-09-17T03:50:24.124578+0000","last_became_active":"2024-09-17T03:50:23.133670+0000","last_became_peered":"2024-09-17T03:50:23.133670+0000","last_unstale":"2024-09-17T03:50:24.124578+0000","last_undegraded":"2024-09-17T03:50:24.124578+0000","last_fullsized":"2024-09-17T03:50:24.124578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157681+0000","last_change":"2024-09-17T03:50:24.151434+0000","last_active":"2024-09-17T03:50:24.157681+0000","last_peered":"2024-09-17T03:50:24.157681+0000","last_clean":"2024-09-17T03:50:24.157681+0000","last_became_active":"2024-09-17T03:50:24.149266+0000","last_became_peered":"2024-09-17T03:50:24.149266+0000","last_unstale":"2024-09-17T03:50:24.157681+0000","last_undegraded":"2024-09-17T03:50:24.157681+0000","last_fullsized":"2024-09-17T03:50:24.157681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:50:43.760650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146994+0000","last_change":"2024-09-17T03:50:23.143447+0000","last_active":"2024-09-17T03:50:24.146994+0000","last_peered":"2024-09-17T03:50:24.146994+0000","last_clean":"2024-09-17T03:50:24.146994+0000","last_became_active":"2024-09-17T03:50:23.143053+0000","last_became_peered":"2024-09-17T03:50:23.143053+0000","last_unstale":"2024-09-17T03:50:24.146994+0000","last_undegraded":"2024-09-17T03:50:24.146994+0000","last_fullsized":"2024-09-17T03:50:24.146994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:45.122730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143714+0000","last_change":"2024-09-17T03:50:24.138823+0000","last_active":"2024-09-17T03:50:24.143714+0000","last_peered":"2024-09-17T03:50:24.143714+0000","last_clean":"2024-09-17T03:50:24.143714+0000","last_became_active":"2024-09-17T03:50:24.138293+0000","last_became_peered":"2024-09-17T03:50:24.138293+0000","last_unstale":"2024-09-17T03:50:24.143714+0000","last_undegraded":"2024-09-17T03:50:24.143714+0000","last_fullsized":"2024-09-17T03:50:24.143714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126274+0000","last_change":"2024-09-17T03:50:23.139166+0000","last_active":"2024-09-17T03:50:24.126274+0000","last_peered":"2024-09-17T03:50:24.126274+0000","last_clean":"2024-09-17T03:50:24.126274+0000","last_became_active":"2024-09-17T03:50:23.138826+0000","last_became_peered":"2024-09-17T03:50:23.138826+0000","last_unstale":"2024-09-17T03:50:24.126274+0000","last_undegraded":"2024-09-17T03:50:24.126274+0000","last_fullsized":"2024-09-17T03:50:24.126274+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:07.180415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162653+0000","last_change":"2024-09-17T03:50:24.157448+0000","last_active":"2024-09-17T03:50:24.162653+0000","last_peered":"2024-09-17T03:50:24.162653+0000","last_clean":"2024-09-17T03:50:24.162653+0000","last_became_active":"2024-09-17T03:50:24.156599+0000","last_became_peered":"2024-09-17T03:50:24.156599+0000","last_unstale":"2024-09-17T03:50:24.162653+0000","last_undegraded":"2024-09-17T03:50:24.162653+0000","last_fullsized":"2024-09-17T03:50:24.162653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.123299+0000","last_change":"2024-09-17T03:50:23.122274+0000","last_active":"2024-09-17T03:50:23.123299+0000","last_peered":"2024-09-17T03:50:23.123299+0000","last_clean":"2024-09-17T03:50:23.123299+0000","last_became_active":"2024-09-17T03:50:23.121928+0000","last_became_peered":"2024-09-17T03:50:23.121928+0000","last_unstale":"2024-09-17T03:50:23.123299+0000","last_undegraded":"2024-09-17T03:50:23.123299+0000","last_fullsized":"2024-09-17T03:50:23.123299+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142845+0000","last_change":"2024-09-17T03:50:24.138004+0000","last_active":"2024-09-17T03:50:24.142845+0000","last_peered":"2024-09-17T03:50:24.142845+0000","last_clean":"2024-09-17T03:50:24.142845+0000","last_became_active":"2024-09-17T03:50:24.137679+0000","last_became_peered":"2024-09-17T03:50:24.137679+0000","last_unstale":"2024-09-17T03:50:24.142845+0000","last_undegraded":"2024-09-17T03:50:24.142845+0000","last_fullsized":"2024-09-17T03:50:24.142845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142928+0000","last_change":"2024-09-17T03:50:23.133114+0000","last_active":"2024-09-17T03:50:24.142928+0000","last_peered":"2024-09-17T03:50:24.142928+0000","last_clean":"2024-09-17T03:50:24.142928+0000","last_became_active":"2024-09-17T03:50:23.132725+0000","last_became_peered":"2024-09-17T03:50:23.132725+0000","last_unstale":"2024-09-17T03:50:24.142928+0000","last_undegraded":"2024-09-17T03:50:24.142928+0000","last_fullsized":"2024-09-17T03:50:24.142928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150161+0000","last_change":"2024-09-17T03:50:24.145715+0000","last_active":"2024-09-17T03:50:24.150161+0000","last_peered":"2024-09-17T03:50:24.150161+0000","last_clean":"2024-09-17T03:50:24.150161+0000","last_became_active":"2024-09-17T03:50:24.145392+0000","last_became_peered":"2024-09-17T03:50:24.145392+0000","last_unstale":"2024-09-17T03:50:24.150161+0000","last_undegraded":"2024-09-17T03:50:24.150161+0000","last_fullsized":"2024-09-17T03:50:24.150161+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.163518+0000","last_change":"2024-09-17T03:50:23.140412+0000","last_active":"2024-09-17T03:50:24.163518+0000","last_peered":"2024-09-17T03:50:24.163518+0000","last_clean":"2024-09-17T03:50:24.163518+0000","last_became_active":"2024-09-17T03:50:23.139391+0000","last_became_peered":"2024-09-17T03:50:23.139391+0000","last_unstale":"2024-09-17T03:50:24.163518+0000","last_undegraded":"2024-09-17T03:50:24.163518+0000","last_fullsized":"2024-09-17T03:50:24.163518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153912+0000","last_change":"2024-09-17T03:50:24.149315+0000","last_active":"2024-09-17T03:50:24.153912+0000","last_peered":"2024-09-17T03:50:24.153912+0000","last_clean":"2024-09-17T03:50:24.153912+0000","last_became_active":"2024-09-17T03:50:24.147345+0000","last_became_peered":"2024-09-17T03:50:24.147345+0000","last_unstale":"2024-09-17T03:50:24.153912+0000","last_undegraded":"2024-09-17T03:50:24.153912+0000","last_fullsized":"2024-09-17T03:50:24.153912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:42.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.156776+0000","last_change":"2024-09-17T03:50:23.143154+0000","last_active":"2024-09-17T03:50:24.156776+0000","last_peered":"2024-09-17T03:50:24.156776+0000","last_clean":"2024-09-17T03:50:24.156776+0000","last_became_active":"2024-09-17T03:50:23.142801+0000","last_became_peered":"2024-09-17T03:50:23.142801+0000","last_unstale":"2024-09-17T03:50:24.156776+0000","last_undegraded":"2024-09-17T03:50:24.156776+0000","last_fullsized":"2024-09-17T03:50:24.156776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:17.669331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149769+0000","last_change":"2024-09-17T03:50:24.147109+0000","last_active":"2024-09-17T03:50:24.149769+0000","last_peered":"2024-09-17T03:50:24.149769+0000","last_clean":"2024-09-17T03:50:24.149769+0000","last_became_active":"2024-09-17T03:50:24.146394+0000","last_became_peered":"2024-09-17T03:50:24.146394+0000","last_unstale":"2024-09-17T03:50:24.149769+0000","last_undegraded":"2024-09-17T03:50:24.149769+0000","last_fullsized":"2024-09-17T03:50:24.149769+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129931+0000","last_change":"2024-09-17T03:50:23.135165+0000","last_active":"2024-09-17T03:50:24.129931+0000","last_peered":"2024-09-17T03:50:24.129931+0000","last_clean":"2024-09-17T03:50:24.129931+0000","last_became_active":"2024-09-17T03:50:23.134575+0000","last_became_peered":"2024-09-17T03:50:23.134575+0000","last_unstale":"2024-09-17T03:50:24.129931+0000","last_undegraded":"2024-09-17T03:50:24.129931+0000","last_fullsized":"2024-09-17T03:50:24.129931+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:53.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157786+0000","last_change":"2024-09-17T03:50:24.155803+0000","last_active":"2024-09-17T03:50:24.157786+0000","last_peered":"2024-09-17T03:50:24.157786+0000","last_clean":"2024-09-17T03:50:24.157786+0000","last_became_active":"2024-09-17T03:50:24.154916+0000","last_became_peered":"2024-09-17T03:50:24.154916+0000","last_unstale":"2024-09-17T03:50:24.157786+0000","last_undegraded":"2024-09-17T03:50:24.157786+0000","last_fullsized":"2024-09-17T03:50:24.157786+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124783+0000","last_change":"2024-09-17T03:50:23.127502+0000","last_active":"2024-09-17T03:50:24.124783+0000","last_peered":"2024-09-17T03:50:24.124783+0000","last_clean":"2024-09-17T03:50:24.124783+0000","last_became_active":"2024-09-17T03:50:23.126531+0000","last_became_peered":"2024-09-17T03:50:23.126531+0000","last_unstale":"2024-09-17T03:50:24.124783+0000","last_undegraded":"2024-09-17T03:50:24.124783+0000","last_fullsized":"2024-09-17T03:50:24.124783+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:39.251639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122281+0000","last_change":"2024-09-17T03:50:23.122281+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122281+0000","last_undegraded":"2024-09-17T03:50:23.122281+0000","last_fullsized":"2024-09-17T03:50:23.122281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":21,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160054+0000","last_change":"2024-09-17T03:50:23.139904+0000","last_active":"2024-09-17T03:50:24.160054+0000","last_peered":"2024-09-17T03:50:24.160054+0000","last_clean":"2024-09-17T03:50:24.160054+0000","last_became_active":"2024-09-17T03:50:23.139601+0000","last_became_peered":"2024-09-17T03:50:23.139601+0000","last_unstale":"2024-09-17T03:50:24.160054+0000","last_undegraded":"2024-09-17T03:50:24.160054+0000","last_fullsized":"2024-09-17T03:50:24.160054+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:21.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155131+0000","last_change":"2024-09-17T03:50:24.150202+0000","last_active":"2024-09-17T03:50:24.155131+0000","last_peered":"2024-09-17T03:50:24.155131+0000","last_clean":"2024-09-17T03:50:24.155131+0000","last_became_active":"2024-09-17T03:50:24.149551+0000","last_became_peered":"2024-09-17T03:50:24.149551+0000","last_unstale":"2024-09-17T03:50:24.155131+0000","last_undegraded":"2024-09-17T03:50:24.155131+0000","last_fullsized":"2024-09-17T03:50:24.155131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.396580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161395+0000","last_change":"2024-09-17T03:50:23.140181+0000","last_active":"2024-09-17T03:50:24.161395+0000","last_peered":"2024-09-17T03:50:24.161395+0000","last_clean":"2024-09-17T03:50:24.161395+0000","last_became_active":"2024-09-17T03:50:23.139817+0000","last_became_peered":"2024-09-17T03:50:23.139817+0000","last_unstale":"2024-09-17T03:50:24.161395+0000","last_undegraded":"2024-09-17T03:50:24.161395+0000","last_fullsized":"2024-09-17T03:50:24.161395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:10.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146249+0000","last_change":"2024-09-17T03:50:24.135542+0000","last_active":"2024-09-17T03:50:24.146249+0000","last_peered":"2024-09-17T03:50:24.146249+0000","last_clean":"2024-09-17T03:50:24.146249+0000","last_became_active":"2024-09-17T03:50:24.135241+0000","last_became_peered":"2024-09-17T03:50:24.135241+0000","last_unstale":"2024-09-17T03:50:24.146249+0000","last_undegraded":"2024-09-17T03:50:24.146249+0000","last_fullsized":"2024-09-17T03:50:24.146249+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149965+0000","last_change":"2024-09-17T03:50:23.139663+0000","last_active":"2024-09-17T03:50:24.149965+0000","last_peered":"2024-09-17T03:50:24.149965+0000","last_clean":"2024-09-17T03:50:24.149965+0000","last_became_active":"2024-09-17T03:50:23.139294+0000","last_became_peered":"2024-09-17T03:50:23.139294+0000","last_unstale":"2024-09-17T03:50:24.149965+0000","last_undegraded":"2024-09-17T03:50:24.149965+0000","last_fullsized":"2024-09-17T03:50:24.149965+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.179574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160646+0000","last_change":"2024-09-17T03:50:24.155061+0000","last_active":"2024-09-17T03:50:24.160646+0000","last_peered":"2024-09-17T03:50:24.160646+0000","last_clean":"2024-09-17T03:50:24.160646+0000","last_became_active":"2024-09-17T03:50:24.154794+0000","last_became_peered":"2024-09-17T03:50:24.154794+0000","last_unstale":"2024-09-17T03:50:24.160646+0000","last_undegraded":"2024-09-17T03:50:24.160646+0000","last_fullsized":"2024-09-17T03:50:24.160646+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:04.104460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154914+0000","last_change":"2024-09-17T03:50:23.143762+0000","last_active":"2024-09-17T03:50:24.154914+0000","last_peered":"2024-09-17T03:50:24.154914+0000","last_clean":"2024-09-17T03:50:24.154914+0000","last_became_active":"2024-09-17T03:50:23.143435+0000","last_became_peered":"2024-09-17T03:50:23.143435+0000","last_unstale":"2024-09-17T03:50:24.154914+0000","last_undegraded":"2024-09-17T03:50:24.154914+0000","last_fullsized":"2024-09-17T03:50:24.154914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:52.817756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143007+0000","last_change":"2024-09-17T03:50:24.138195+0000","last_active":"2024-09-17T03:50:24.143007+0000","last_peered":"2024-09-17T03:50:24.143007+0000","last_clean":"2024-09-17T03:50:24.143007+0000","last_became_active":"2024-09-17T03:50:24.137835+0000","last_became_peered":"2024-09-17T03:50:24.137835+0000","last_unstale":"2024-09-17T03:50:24.143007+0000","last_undegraded":"2024-09-17T03:50:24.143007+0000","last_fullsized":"2024-09-17T03:50:24.143007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:06.995259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129821+0000","last_change":"2024-09-17T03:50:23.140276+0000","last_active":"2024-09-17T03:50:24.129821+0000","last_peered":"2024-09-17T03:50:24.129821+0000","last_clean":"2024-09-17T03:50:24.129821+0000","last_became_active":"2024-09-17T03:50:23.139757+0000","last_became_peered":"2024-09-17T03:50:23.139757+0000","last_unstale":"2024-09-17T03:50:24.129821+0000","last_undegraded":"2024-09-17T03:50:24.129821+0000","last_fullsized":"2024-09-17T03:50:24.129821+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:14:45.951042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157939+0000","last_change":"2024-09-17T03:50:24.152597+0000","last_active":"2024-09-17T03:50:24.157939+0000","last_peered":"2024-09-17T03:50:24.157939+0000","last_clean":"2024-09-17T03:50:24.157939+0000","last_became_active":"2024-09-17T03:50:24.152307+0000","last_became_peered":"2024-09-17T03:50:24.152307+0000","last_unstale":"2024-09-17T03:50:24.157939+0000","last_undegraded":"2024-09-17T03:50:24.157939+0000","last_fullsized":"2024-09-17T03:50:24.157939+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:20.938433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.132753+0000","last_change":"2024-09-17T03:50:23.130742+0000","last_active":"2024-09-17T03:50:23.132753+0000","last_peered":"2024-09-17T03:50:23.132753+0000","last_clean":"2024-09-17T03:50:23.132753+0000","last_became_active":"2024-09-17T03:50:23.130434+0000","last_became_peered":"2024-09-17T03:50:23.130434+0000","last_unstale":"2024-09-17T03:50:23.132753+0000","last_undegraded":"2024-09-17T03:50:23.132753+0000","last_fullsized":"2024-09-17T03:50:23.132753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.030426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145900+0000","last_change":"2024-09-17T03:50:24.141883+0000","last_active":"2024-09-17T03:50:24.145900+0000","last_peered":"2024-09-17T03:50:24.145900+0000","last_clean":"2024-09-17T03:50:24.145900+0000","last_became_active":"2024-09-17T03:50:24.141572+0000","last_became_peered":"2024-09-17T03:50:24.141572+0000","last_unstale":"2024-09-17T03:50:24.145900+0000","last_undegraded":"2024-09-17T03:50:24.145900+0000","last_fullsized":"2024-09-17T03:50:24.145900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:36.875814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.138532+0000","last_change":"2024-09-17T03:50:23.133695+0000","last_active":"2024-09-17T03:50:23.138532+0000","last_peered":"2024-09-17T03:50:23.138532+0000","last_clean":"2024-09-17T03:50:23.138532+0000","last_became_active":"2024-09-17T03:50:23.133125+0000","last_became_peered":"2024-09-17T03:50:23.133125+0000","last_unstale":"2024-09-17T03:50:23.138532+0000","last_undegraded":"2024-09-17T03:50:23.138532+0000","last_fullsized":"2024-09-17T03:50:23.138532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145857+0000","last_change":"2024-09-17T03:50:24.141825+0000","last_active":"2024-09-17T03:50:24.145857+0000","last_peered":"2024-09-17T03:50:24.145857+0000","last_clean":"2024-09-17T03:50:24.145857+0000","last_became_active":"2024-09-17T03:50:24.141522+0000","last_became_peered":"2024-09-17T03:50:24.141522+0000","last_unstale":"2024-09-17T03:50:24.145857+0000","last_undegraded":"2024-09-17T03:50:24.145857+0000","last_fullsized":"2024-09-17T03:50:24.145857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:22.204438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131053+0000","last_change":"2024-09-17T03:50:23.141992+0000","last_active":"2024-09-17T03:50:24.131053+0000","last_peered":"2024-09-17T03:50:24.131053+0000","last_clean":"2024-09-17T03:50:24.131053+0000","last_became_active":"2024-09-17T03:50:23.141642+0000","last_became_peered":"2024-09-17T03:50:23.141642+0000","last_unstale":"2024-09-17T03:50:24.131053+0000","last_undegraded":"2024-09-17T03:50:24.131053+0000","last_fullsized":"2024-09-17T03:50:24.131053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:12.799906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.144614+0000","last_change":"2024-09-17T03:50:24.142339+0000","last_active":"2024-09-17T03:50:24.144614+0000","last_peered":"2024-09-17T03:50:24.144614+0000","last_clean":"2024-09-17T03:50:24.144614+0000","last_became_active":"2024-09-17T03:50:24.142028+0000","last_became_peered":"2024-09-17T03:50:24.142028+0000","last_unstale":"2024-09-17T03:50:24.144614+0000","last_undegraded":"2024-09-17T03:50:24.144614+0000","last_fullsized":"2024-09-17T03:50:24.144614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.118295+0000","last_change":"2024-09-17T03:50:23.129908+0000","last_active":"2024-09-17T03:50:24.118295+0000","last_peered":"2024-09-17T03:50:24.118295+0000","last_clean":"2024-09-17T03:50:24.118295+0000","last_became_active":"2024-09-17T03:50:23.129543+0000","last_became_peered":"2024-09-17T03:50:23.129543+0000","last_unstale":"2024-09-17T03:50:24.118295+0000","last_undegraded":"2024-09-17T03:50:24.118295+0000","last_fullsized":"2024-09-17T03:50:24.118295+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129144+0000","last_change":"2024-09-17T03:50:24.124588+0000","last_active":"2024-09-17T03:50:24.129144+0000","last_peered":"2024-09-17T03:50:24.129144+0000","last_clean":"2024-09-17T03:50:24.129144+0000","last_became_active":"2024-09-17T03:50:24.124178+0000","last_became_peered":"2024-09-17T03:50:24.124178+0000","last_unstale":"2024-09-17T03:50:24.129144+0000","last_undegraded":"2024-09-17T03:50:24.129144+0000","last_fullsized":"2024-09-17T03:50:24.129144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:20.307754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150943+0000","last_change":"2024-09-17T03:50:23.137622+0000","last_active":"2024-09-17T03:50:24.150943+0000","last_peered":"2024-09-17T03:50:24.150943+0000","last_clean":"2024-09-17T03:50:24.150943+0000","last_became_active":"2024-09-17T03:50:23.137258+0000","last_became_peered":"2024-09-17T03:50:23.137258+0000","last_unstale":"2024-09-17T03:50:24.150943+0000","last_undegraded":"2024-09-17T03:50:24.150943+0000","last_fullsized":"2024-09-17T03:50:24.150943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:40.356807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146377+0000","last_change":"2024-09-17T03:50:24.140577+0000","last_active":"2024-09-17T03:50:24.146377+0000","last_peered":"2024-09-17T03:50:24.146377+0000","last_clean":"2024-09-17T03:50:24.146377+0000","last_became_active":"2024-09-17T03:50:24.140330+0000","last_became_peered":"2024-09-17T03:50:24.140330+0000","last_unstale":"2024-09-17T03:50:24.146377+0000","last_undegraded":"2024-09-17T03:50:24.146377+0000","last_fullsized":"2024-09-17T03:50:24.146377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131834+0000","last_change":"2024-09-17T03:50:23.130628+0000","last_active":"2024-09-17T03:50:23.131834+0000","last_peered":"2024-09-17T03:50:23.131834+0000","last_clean":"2024-09-17T03:50:23.131834+0000","last_became_active":"2024-09-17T03:50:23.130307+0000","last_became_peered":"2024-09-17T03:50:23.130307+0000","last_unstale":"2024-09-17T03:50:23.131834+0000","last_undegraded":"2024-09-17T03:50:23.131834+0000","last_fullsized":"2024-09-17T03:50:23.131834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:51.417524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116716+0000","last_change":"2024-09-17T03:50:23.116716+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116716+0000","last_undegraded":"2024-09-17T03:50:23.116716+0000","last_fullsized":"2024-09-17T03:50:23.116716+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.128468+0000","last_change":"2024-09-17T03:50:23.136531+0000","last_active":"2024-09-17T03:50:24.128468+0000","last_peered":"2024-09-17T03:50:24.128468+0000","last_clean":"2024-09-17T03:50:24.128468+0000","last_became_active":"2024-09-17T03:50:23.136207+0000","last_became_peered":"2024-09-17T03:50:23.136207+0000","last_unstale":"2024-09-17T03:50:24.128468+0000","last_undegraded":"2024-09-17T03:50:24.128468+0000","last_fullsized":"2024-09-17T03:50:24.128468+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:47.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122595+0000","last_change":"2024-09-17T03:50:23.122595+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122595+0000","last_undegraded":"2024-09-17T03:50:23.122595+0000","last_fullsized":"2024-09-17T03:50:23.122595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.128999+0000","last_change":"2024-09-17T03:50:23.124066+0000","last_active":"2024-09-17T03:50:23.128999+0000","last_peered":"2024-09-17T03:50:23.128999+0000","last_clean":"2024-09-17T03:50:23.128999+0000","last_became_active":"2024-09-17T03:50:23.123776+0000","last_became_peered":"2024-09-17T03:50:23.123776+0000","last_unstale":"2024-09-17T03:50:23.128999+0000","last_undegraded":"2024-09-17T03:50:23.128999+0000","last_fullsized":"2024-09-17T03:50:23.128999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116315+0000","last_change":"2024-09-17T03:50:23.116315+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116315+0000","last_undegraded":"2024-09-17T03:50:23.116315+0000","last_fullsized":"2024-09-17T03:50:23.116315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.130118+0000","last_change":"2024-09-17T03:50:23.136967+0000","last_active":"2024-09-17T03:50:24.130118+0000","last_peered":"2024-09-17T03:50:24.130118+0000","last_clean":"2024-09-17T03:50:24.130118+0000","last_became_active":"2024-09-17T03:50:23.136643+0000","last_became_peered":"2024-09-17T03:50:23.136643+0000","last_unstale":"2024-09-17T03:50:24.130118+0000","last_undegraded":"2024-09-17T03:50:24.130118+0000","last_fullsized":"2024-09-17T03:50:24.130118+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:58.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158254+0000","last_change":"2024-09-17T03:50:24.150858+0000","last_active":"2024-09-17T03:50:24.158254+0000","last_peered":"2024-09-17T03:50:24.158254+0000","last_clean":"2024-09-17T03:50:24.158254+0000","last_became_active":"2024-09-17T03:50:24.150509+0000","last_became_peered":"2024-09-17T03:50:24.150509+0000","last_unstale":"2024-09-17T03:50:24.158254+0000","last_undegraded":"2024-09-17T03:50:24.158254+0000","last_fullsized":"2024-09-17T03:50:24.158254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131336+0000","last_change":"2024-09-17T03:50:23.141022+0000","last_active":"2024-09-17T03:50:24.131336+0000","last_peered":"2024-09-17T03:50:24.131336+0000","last_clean":"2024-09-17T03:50:24.131336+0000","last_became_active":"2024-09-17T03:50:23.140264+0000","last_became_peered":"2024-09-17T03:50:23.140264+0000","last_unstale":"2024-09-17T03:50:24.131336+0000","last_undegraded":"2024-09-17T03:50:24.131336+0000","last_fullsized":"2024-09-17T03:50:24.131336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:55.135377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155579+0000","last_change":"2024-09-17T03:50:24.149913+0000","last_active":"2024-09-17T03:50:24.155579+0000","last_peered":"2024-09-17T03:50:24.155579+0000","last_clean":"2024-09-17T03:50:24.155579+0000","last_became_active":"2024-09-17T03:50:24.149591+0000","last_became_peered":"2024-09-17T03:50:24.149591+0000","last_unstale":"2024-09-17T03:50:24.155579+0000","last_undegraded":"2024-09-17T03:50:24.155579+0000","last_fullsized":"2024-09-17T03:50:24.155579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:25.250775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126284+0000","last_change":"2024-09-17T03:50:23.130536+0000","last_active":"2024-09-17T03:50:24.126284+0000","last_peered":"2024-09-17T03:50:24.126284+0000","last_clean":"2024-09-17T03:50:24.126284+0000","last_became_active":"2024-09-17T03:50:23.130176+0000","last_became_peered":"2024-09-17T03:50:23.130176+0000","last_unstale":"2024-09-17T03:50:24.126284+0000","last_undegraded":"2024-09-17T03:50:24.126284+0000","last_fullsized":"2024-09-17T03:50:24.126284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:26.703578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150296+0000","last_change":"2024-09-17T03:50:24.146715+0000","last_active":"2024-09-17T03:50:24.150296+0000","last_peered":"2024-09-17T03:50:24.150296+0000","last_clean":"2024-09-17T03:50:24.150296+0000","last_became_active":"2024-09-17T03:50:24.146329+0000","last_became_peered":"2024-09-17T03:50:24.146329+0000","last_unstale":"2024-09-17T03:50:24.150296+0000","last_undegraded":"2024-09-17T03:50:24.150296+0000","last_fullsized":"2024-09-17T03:50:24.150296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142795+0000","last_change":"2024-09-17T03:50:23.138899+0000","last_active":"2024-09-17T03:50:23.142795+0000","last_peered":"2024-09-17T03:50:23.142795+0000","last_clean":"2024-09-17T03:50:23.142795+0000","last_became_active":"2024-09-17T03:50:23.138546+0000","last_became_peered":"2024-09-17T03:50:23.138546+0000","last_unstale":"2024-09-17T03:50:23.142795+0000","last_undegraded":"2024-09-17T03:50:23.142795+0000","last_fullsized":"2024-09-17T03:50:23.142795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":128,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129940+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:50:24.129940+0000","last_peered":"2024-09-17T03:50:24.129940+0000","last_clean":"2024-09-17T03:50:24.129940+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:50:24.129940+0000","last_undegraded":"2024-09-17T03:50:24.129940+0000","last_fullsized":"2024-09-17T03:50:24.129940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":933888,"data_stored":717659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509454,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":121024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542159,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":711392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:50:28.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:50:28.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 -- 172.21.15.129:0/4216208916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428070030 msgr2=0x7f8428074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 --2- 172.21.15.129:0/4216208916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428070030 0x7f8428074640 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f84280b4b40 tx=0x7f841402a560 comp rx=0 tx=0).stop 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 -- 172.21.15.129:0/4216208916 shutdown_connections 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 --2- 172.21.15.129:0/4216208916 >> v2:172.21.15.181:3300/0 conn(0x7f8428070440 0x7f8428074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 --2- 172.21.15.129:0/4216208916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428070030 0x7f8428074640 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 -- 172.21.15.129:0/4216208916 >> 172.21.15.129:0/4216208916 conn(0x7f84280b1cb0 msgr2=0x7f84280b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 -- 172.21.15.129:0/4216208916 shutdown_connections 2024-09-17T03:50:28.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.795+0000 7f842f600640 1 -- 172.21.15.129:0/4216208916 wait complete. 2024-09-17T03:50:28.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.797+0000 7f842f600640 1 Processor -- start 2024-09-17T03:50:28.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.798+0000 7f842f600640 1 -- start start 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8428070030 0x7f8428153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8428070440 0x7f84281f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f84280b39f0 con 0x7f8428073170 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f84280b3870 con 0x7f8428070030 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842f600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f84280b3b70 con 0x7f8428070440 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41240/0 (socket says 172.21.15.129:41240) 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842d600640 1 -- 172.21.15.129:0/574964562 learned_addr learned my addr 172.21.15.129:0/574964562 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:28.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f842cc00640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8428070030 0x7f8428153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.799+0000 7f8427e00640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8428070440 0x7f84281f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 -- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8428070440 msgr2=0x7f84281f5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8428070440 0x7f84281f5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 -- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8428070030 msgr2=0x7f8428153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8428070030 0x7f8428153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 -- 172.21.15.129:0/574964562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84281538d0 con 0x7f8428073170 2024-09-17T03:50:28.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842d600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f842000e9c0 tx=0x7f842000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420004d20 con 0x7f8428073170 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f842000cb20 con 0x7f8428073170 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8420005020 con 0x7f8428073170 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.800+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84280c7e90 con 0x7f8428073170 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.801+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84280c8150 con 0x7f8428073170 2024-09-17T03:50:28.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.802+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84281f9be0 con 0x7f8428073170 2024-09-17T03:50:28.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.803+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f84200051c0 con 0x7f8428073170 2024-09-17T03:50:28.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.804+0000 7f8424c00640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:28.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.804+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f842009b660 con 0x7f8428073170 2024-09-17T03:50:28.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.804+0000 7f842cc00640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:28.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.805+0000 7f842cc00640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f841c006fd0 tx=0x7f841c009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:28.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.808+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8420016030 con 0x7f8428073170 2024-09-17T03:50:28.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.988+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f84280b40d0 con 0x7f8408078b90 2024-09-17T03:50:28.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.993+0000 7f8424c00640 1 -- 172.21.15.129:0/574964562 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251190 (secure 0 0 0) 0x7f84280b40d0 con 0x7f8408078b90 2024-09-17T03:50:28.995 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:28.996 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.997+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 msgr2=0x7f840807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.997+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f841c006fd0 tx=0x7f841c009060 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.997+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 msgr2=0x7f842809d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.997+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f842000e9c0 tx=0x7f842000ee80 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 shutdown_connections 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8428073170 0x7f842809d390 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8428070440 0x7f84281f5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 --2- 172.21.15.129:0/574964562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8428070030 0x7f8428153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:28.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 >> 172.21.15.129:0/574964562 conn(0x7f84280b1cb0 msgr2=0x7f842815ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:28.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 shutdown_connections 2024-09-17T03:50:28.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:28.998+0000 7f842f600640 1 -- 172.21.15.129:0/574964562 wait complete. 2024-09-17T03:50:29.021 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-17T03:50:28.306834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224180,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750540,"statfs":{"total":773094113280,"available":772864552960,"internally_reserved":0,"allocated":3883008,"data_stored":2186530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001306"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124897+0000","last_change":"2024-09-17T03:50:23.124897+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124897+0000","last_undegraded":"2024-09-17T03:50:23.124897+0000","last_fullsized":"2024-09-17T03:50:23.124897+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149743+0000","last_change":"2024-09-17T03:50:24.146222+0000","last_active":"2024-09-17T03:50:24.149743+0000","last_peered":"2024-09-17T03:50:24.149743+0000","last_clean":"2024-09-17T03:50:24.149743+0000","last_became_active":"2024-09-17T03:50:24.145925+0000","last_became_peered":"2024-09-17T03:50:24.145925+0000","last_unstale":"2024-09-17T03:50:24.149743+0000","last_undegraded":"2024-09-17T03:50:24.149743+0000","last_fullsized":"2024-09-17T03:50:24.149743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150192+0000","last_change":"2024-09-17T03:50:24.146390+0000","last_active":"2024-09-17T03:50:24.150192+0000","last_peered":"2024-09-17T03:50:24.150192+0000","last_clean":"2024-09-17T03:50:24.150192+0000","last_became_active":"2024-09-17T03:50:24.146067+0000","last_became_peered":"2024-09-17T03:50:24.146067+0000","last_unstale":"2024-09-17T03:50:24.150192+0000","last_undegraded":"2024-09-17T03:50:24.150192+0000","last_fullsized":"2024-09-17T03:50:24.150192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161136+0000","last_change":"2024-09-17T03:50:24.156017+0000","last_active":"2024-09-17T03:50:24.161136+0000","last_peered":"2024-09-17T03:50:24.161136+0000","last_clean":"2024-09-17T03:50:24.161136+0000","last_became_active":"2024-09-17T03:50:24.155677+0000","last_became_peered":"2024-09-17T03:50:24.155677+0000","last_unstale":"2024-09-17T03:50:24.161136+0000","last_undegraded":"2024-09-17T03:50:24.161136+0000","last_fullsized":"2024-09-17T03:50:24.161136+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:13.449411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145398+0000","last_change":"2024-09-17T03:50:24.138985+0000","last_active":"2024-09-17T03:50:24.145398+0000","last_peered":"2024-09-17T03:50:24.145398+0000","last_clean":"2024-09-17T03:50:24.145398+0000","last_became_active":"2024-09-17T03:50:24.138569+0000","last_became_peered":"2024-09-17T03:50:24.138569+0000","last_unstale":"2024-09-17T03:50:24.145398+0000","last_undegraded":"2024-09-17T03:50:24.145398+0000","last_fullsized":"2024-09-17T03:50:24.145398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:51.159427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158315+0000","last_change":"2024-09-17T03:50:24.147892+0000","last_active":"2024-09-17T03:50:24.158315+0000","last_peered":"2024-09-17T03:50:24.158315+0000","last_clean":"2024-09-17T03:50:24.158315+0000","last_became_active":"2024-09-17T03:50:24.147616+0000","last_became_peered":"2024-09-17T03:50:24.147616+0000","last_unstale":"2024-09-17T03:50:24.158315+0000","last_undegraded":"2024-09-17T03:50:24.158315+0000","last_fullsized":"2024-09-17T03:50:24.158315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152457+0000","last_change":"2024-09-17T03:50:24.148414+0000","last_active":"2024-09-17T03:50:24.152457+0000","last_peered":"2024-09-17T03:50:24.152457+0000","last_clean":"2024-09-17T03:50:24.152457+0000","last_became_active":"2024-09-17T03:50:24.148020+0000","last_became_peered":"2024-09-17T03:50:24.148020+0000","last_unstale":"2024-09-17T03:50:24.152457+0000","last_undegraded":"2024-09-17T03:50:24.152457+0000","last_fullsized":"2024-09-17T03:50:24.152457+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150293+0000","last_change":"2024-09-17T03:50:24.147045+0000","last_active":"2024-09-17T03:50:24.150293+0000","last_peered":"2024-09-17T03:50:24.150293+0000","last_clean":"2024-09-17T03:50:24.150293+0000","last_became_active":"2024-09-17T03:50:24.146692+0000","last_became_peered":"2024-09-17T03:50:24.146692+0000","last_unstale":"2024-09-17T03:50:24.150293+0000","last_undegraded":"2024-09-17T03:50:24.150293+0000","last_fullsized":"2024-09-17T03:50:24.150293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155032+0000","last_change":"2024-09-17T03:50:24.149261+0000","last_active":"2024-09-17T03:50:24.155032+0000","last_peered":"2024-09-17T03:50:24.155032+0000","last_clean":"2024-09-17T03:50:24.155032+0000","last_became_active":"2024-09-17T03:50:24.148898+0000","last_became_peered":"2024-09-17T03:50:24.148898+0000","last_unstale":"2024-09-17T03:50:24.155032+0000","last_undegraded":"2024-09-17T03:50:24.155032+0000","last_fullsized":"2024-09-17T03:50:24.155032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:20.858522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160355+0000","last_change":"2024-09-17T03:50:24.152445+0000","last_active":"2024-09-17T03:50:24.160355+0000","last_peered":"2024-09-17T03:50:24.160355+0000","last_clean":"2024-09-17T03:50:24.160355+0000","last_became_active":"2024-09-17T03:50:24.152179+0000","last_became_peered":"2024-09-17T03:50:24.152179+0000","last_unstale":"2024-09-17T03:50:24.160355+0000","last_undegraded":"2024-09-17T03:50:24.160355+0000","last_fullsized":"2024-09-17T03:50:24.160355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:59.856041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157759+0000","last_change":"2024-09-17T03:50:24.155248+0000","last_active":"2024-09-17T03:50:24.157759+0000","last_peered":"2024-09-17T03:50:24.157759+0000","last_clean":"2024-09-17T03:50:24.157759+0000","last_became_active":"2024-09-17T03:50:24.154779+0000","last_became_peered":"2024-09-17T03:50:24.154779+0000","last_unstale":"2024-09-17T03:50:24.157759+0000","last_undegraded":"2024-09-17T03:50:24.157759+0000","last_fullsized":"2024-09-17T03:50:24.157759+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143858+0000","last_change":"2024-09-17T03:50:24.138736+0000","last_active":"2024-09-17T03:50:24.143858+0000","last_peered":"2024-09-17T03:50:24.143858+0000","last_clean":"2024-09-17T03:50:24.143858+0000","last_became_active":"2024-09-17T03:50:24.138424+0000","last_became_peered":"2024-09-17T03:50:24.138424+0000","last_unstale":"2024-09-17T03:50:24.143858+0000","last_undegraded":"2024-09-17T03:50:24.143858+0000","last_fullsized":"2024-09-17T03:50:24.143858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145922+0000","last_change":"2024-09-17T03:50:24.142434+0000","last_active":"2024-09-17T03:50:24.145922+0000","last_peered":"2024-09-17T03:50:24.145922+0000","last_clean":"2024-09-17T03:50:24.145922+0000","last_became_active":"2024-09-17T03:50:24.142083+0000","last_became_peered":"2024-09-17T03:50:24.142083+0000","last_unstale":"2024-09-17T03:50:24.145922+0000","last_undegraded":"2024-09-17T03:50:24.145922+0000","last_fullsized":"2024-09-17T03:50:24.145922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:49.596755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160739+0000","last_change":"2024-09-17T03:50:24.154707+0000","last_active":"2024-09-17T03:50:24.160739+0000","last_peered":"2024-09-17T03:50:24.160739+0000","last_clean":"2024-09-17T03:50:24.160739+0000","last_became_active":"2024-09-17T03:50:24.154328+0000","last_became_peered":"2024-09-17T03:50:24.154328+0000","last_unstale":"2024-09-17T03:50:24.160739+0000","last_undegraded":"2024-09-17T03:50:24.160739+0000","last_fullsized":"2024-09-17T03:50:24.160739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145415+0000","last_change":"2024-09-17T03:50:24.125931+0000","last_active":"2024-09-17T03:50:24.145415+0000","last_peered":"2024-09-17T03:50:24.145415+0000","last_clean":"2024-09-17T03:50:24.145415+0000","last_became_active":"2024-09-17T03:50:24.125300+0000","last_became_peered":"2024-09-17T03:50:24.125300+0000","last_unstale":"2024-09-17T03:50:24.145415+0000","last_undegraded":"2024-09-17T03:50:24.145415+0000","last_fullsized":"2024-09-17T03:50:24.145415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145644+0000","last_change":"2024-09-17T03:50:24.144536+0000","last_active":"2024-09-17T03:50:24.145644+0000","last_peered":"2024-09-17T03:50:24.145644+0000","last_clean":"2024-09-17T03:50:24.145644+0000","last_became_active":"2024-09-17T03:50:24.143960+0000","last_became_peered":"2024-09-17T03:50:24.143960+0000","last_unstale":"2024-09-17T03:50:24.145644+0000","last_undegraded":"2024-09-17T03:50:24.145644+0000","last_fullsized":"2024-09-17T03:50:24.145644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152188+0000","last_change":"2024-09-17T03:50:24.147697+0000","last_active":"2024-09-17T03:50:24.152188+0000","last_peered":"2024-09-17T03:50:24.152188+0000","last_clean":"2024-09-17T03:50:24.152188+0000","last_became_active":"2024-09-17T03:50:24.146930+0000","last_became_peered":"2024-09-17T03:50:24.146930+0000","last_unstale":"2024-09-17T03:50:24.152188+0000","last_undegraded":"2024-09-17T03:50:24.152188+0000","last_fullsized":"2024-09-17T03:50:24.152188+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162682+0000","last_change":"2024-09-17T03:50:24.159788+0000","last_active":"2024-09-17T03:50:24.162682+0000","last_peered":"2024-09-17T03:50:24.162682+0000","last_clean":"2024-09-17T03:50:24.162682+0000","last_became_active":"2024-09-17T03:50:24.159505+0000","last_became_peered":"2024-09-17T03:50:24.159505+0000","last_unstale":"2024-09-17T03:50:24.162682+0000","last_undegraded":"2024-09-17T03:50:24.162682+0000","last_fullsized":"2024-09-17T03:50:24.162682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161542+0000","last_change":"2024-09-17T03:50:24.154330+0000","last_active":"2024-09-17T03:50:24.161542+0000","last_peered":"2024-09-17T03:50:24.161542+0000","last_clean":"2024-09-17T03:50:24.161542+0000","last_became_active":"2024-09-17T03:50:24.153860+0000","last_became_peered":"2024-09-17T03:50:24.153860+0000","last_unstale":"2024-09-17T03:50:24.161542+0000","last_undegraded":"2024-09-17T03:50:24.161542+0000","last_fullsized":"2024-09-17T03:50:24.161542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.132028+0000","last_change":"2024-09-17T03:50:24.128012+0000","last_active":"2024-09-17T03:50:24.132028+0000","last_peered":"2024-09-17T03:50:24.132028+0000","last_clean":"2024-09-17T03:50:24.132028+0000","last_became_active":"2024-09-17T03:50:24.127633+0000","last_became_peered":"2024-09-17T03:50:24.127633+0000","last_unstale":"2024-09-17T03:50:24.132028+0000","last_undegraded":"2024-09-17T03:50:24.132028+0000","last_fullsized":"2024-09-17T03:50:24.132028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:14.977204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155095+0000","last_change":"2024-09-17T03:50:24.149865+0000","last_active":"2024-09-17T03:50:24.155095+0000","last_peered":"2024-09-17T03:50:24.155095+0000","last_clean":"2024-09-17T03:50:24.155095+0000","last_became_active":"2024-09-17T03:50:24.149569+0000","last_became_peered":"2024-09-17T03:50:24.149569+0000","last_unstale":"2024-09-17T03:50:24.155095+0000","last_undegraded":"2024-09-17T03:50:24.155095+0000","last_fullsized":"2024-09-17T03:50:24.155095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:02.922452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155400+0000","last_change":"2024-09-17T03:50:24.151774+0000","last_active":"2024-09-17T03:50:24.155400+0000","last_peered":"2024-09-17T03:50:24.155400+0000","last_clean":"2024-09-17T03:50:24.155400+0000","last_became_active":"2024-09-17T03:50:24.151379+0000","last_became_peered":"2024-09-17T03:50:24.151379+0000","last_unstale":"2024-09-17T03:50:24.155400+0000","last_undegraded":"2024-09-17T03:50:24.155400+0000","last_fullsized":"2024-09-17T03:50:24.155400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116168+0000","last_change":"2024-09-17T03:50:23.116168+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116168+0000","last_undegraded":"2024-09-17T03:50:23.116168+0000","last_fullsized":"2024-09-17T03:50:23.116168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.138232+0000","last_change":"2024-09-17T03:50:24.132642+0000","last_active":"2024-09-17T03:50:24.138232+0000","last_peered":"2024-09-17T03:50:24.138232+0000","last_clean":"2024-09-17T03:50:24.138232+0000","last_became_active":"2024-09-17T03:50:24.131051+0000","last_became_peered":"2024-09-17T03:50:24.131051+0000","last_unstale":"2024-09-17T03:50:24.138232+0000","last_undegraded":"2024-09-17T03:50:24.138232+0000","last_fullsized":"2024-09-17T03:50:24.138232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153734+0000","last_change":"2024-09-17T03:50:24.145119+0000","last_active":"2024-09-17T03:50:24.153734+0000","last_peered":"2024-09-17T03:50:24.153734+0000","last_clean":"2024-09-17T03:50:24.153734+0000","last_became_active":"2024-09-17T03:50:24.144619+0000","last_became_peered":"2024-09-17T03:50:24.144619+0000","last_unstale":"2024-09-17T03:50:24.153734+0000","last_undegraded":"2024-09-17T03:50:24.153734+0000","last_fullsized":"2024-09-17T03:50:24.153734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:44.451270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142707+0000","last_change":"2024-09-17T03:50:24.136492+0000","last_active":"2024-09-17T03:50:24.142707+0000","last_peered":"2024-09-17T03:50:24.142707+0000","last_clean":"2024-09-17T03:50:24.142707+0000","last_became_active":"2024-09-17T03:50:24.136206+0000","last_became_peered":"2024-09-17T03:50:24.136206+0000","last_unstale":"2024-09-17T03:50:24.142707+0000","last_undegraded":"2024-09-17T03:50:24.142707+0000","last_fullsized":"2024-09-17T03:50:24.142707+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161913+0000","last_change":"2024-09-17T03:50:24.155949+0000","last_active":"2024-09-17T03:50:24.161913+0000","last_peered":"2024-09-17T03:50:24.161913+0000","last_clean":"2024-09-17T03:50:24.161913+0000","last_became_active":"2024-09-17T03:50:24.155692+0000","last_became_peered":"2024-09-17T03:50:24.155692+0000","last_unstale":"2024-09-17T03:50:24.161913+0000","last_undegraded":"2024-09-17T03:50:24.161913+0000","last_fullsized":"2024-09-17T03:50:24.161913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:11.942003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142811+0000","last_change":"2024-09-17T03:50:24.136298+0000","last_active":"2024-09-17T03:50:24.142811+0000","last_peered":"2024-09-17T03:50:24.142811+0000","last_clean":"2024-09-17T03:50:24.142811+0000","last_became_active":"2024-09-17T03:50:24.135828+0000","last_became_peered":"2024-09-17T03:50:24.135828+0000","last_unstale":"2024-09-17T03:50:24.142811+0000","last_undegraded":"2024-09-17T03:50:24.142811+0000","last_fullsized":"2024-09-17T03:50:24.142811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155375+0000","last_change":"2024-09-17T03:50:24.152059+0000","last_active":"2024-09-17T03:50:24.155375+0000","last_peered":"2024-09-17T03:50:24.155375+0000","last_clean":"2024-09-17T03:50:24.155375+0000","last_became_active":"2024-09-17T03:50:24.151751+0000","last_became_peered":"2024-09-17T03:50:24.151751+0000","last_unstale":"2024-09-17T03:50:24.155375+0000","last_undegraded":"2024-09-17T03:50:24.155375+0000","last_fullsized":"2024-09-17T03:50:24.155375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152180+0000","last_change":"2024-09-17T03:50:24.147844+0000","last_active":"2024-09-17T03:50:24.152180+0000","last_peered":"2024-09-17T03:50:24.152180+0000","last_clean":"2024-09-17T03:50:24.152180+0000","last_became_active":"2024-09-17T03:50:24.144862+0000","last_became_peered":"2024-09-17T03:50:24.144862+0000","last_unstale":"2024-09-17T03:50:24.152180+0000","last_undegraded":"2024-09-17T03:50:24.152180+0000","last_fullsized":"2024-09-17T03:50:24.152180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.123020+0000","last_change":"2024-09-17T03:50:23.123020+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.123020+0000","last_undegraded":"2024-09-17T03:50:23.123020+0000","last_fullsized":"2024-09-17T03:50:23.123020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157205+0000","last_change":"2024-09-17T03:50:24.153513+0000","last_active":"2024-09-17T03:50:24.157205+0000","last_peered":"2024-09-17T03:50:24.157205+0000","last_clean":"2024-09-17T03:50:24.157205+0000","last_became_active":"2024-09-17T03:50:24.153118+0000","last_became_peered":"2024-09-17T03:50:24.153118+0000","last_unstale":"2024-09-17T03:50:24.157205+0000","last_undegraded":"2024-09-17T03:50:24.157205+0000","last_fullsized":"2024-09-17T03:50:24.157205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116164+0000","last_change":"2024-09-17T03:50:23.116164+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116164+0000","last_undegraded":"2024-09-17T03:50:23.116164+0000","last_fullsized":"2024-09-17T03:50:23.116164+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142829+0000","last_change":"2024-09-17T03:50:23.138230+0000","last_active":"2024-09-17T03:50:23.142829+0000","last_peered":"2024-09-17T03:50:23.142829+0000","last_clean":"2024-09-17T03:50:23.142829+0000","last_became_active":"2024-09-17T03:50:23.137347+0000","last_became_peered":"2024-09-17T03:50:23.137347+0000","last_unstale":"2024-09-17T03:50:23.142829+0000","last_undegraded":"2024-09-17T03:50:23.142829+0000","last_fullsized":"2024-09-17T03:50:23.142829+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124400+0000","last_change":"2024-09-17T03:50:23.124400+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124400+0000","last_undegraded":"2024-09-17T03:50:23.124400+0000","last_fullsized":"2024-09-17T03:50:23.124400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131789+0000","last_change":"2024-09-17T03:50:23.128500+0000","last_active":"2024-09-17T03:50:23.131789+0000","last_peered":"2024-09-17T03:50:23.131789+0000","last_clean":"2024-09-17T03:50:23.131789+0000","last_became_active":"2024-09-17T03:50:23.128184+0000","last_became_peered":"2024-09-17T03:50:23.128184+0000","last_unstale":"2024-09-17T03:50:23.131789+0000","last_undegraded":"2024-09-17T03:50:23.131789+0000","last_fullsized":"2024-09-17T03:50:23.131789+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155490+0000","last_change":"2024-09-17T03:50:24.152519+0000","last_active":"2024-09-17T03:50:24.155490+0000","last_peered":"2024-09-17T03:50:24.155490+0000","last_clean":"2024-09-17T03:50:24.155490+0000","last_became_active":"2024-09-17T03:50:24.150781+0000","last_became_peered":"2024-09-17T03:50:24.150781+0000","last_unstale":"2024-09-17T03:50:24.155490+0000","last_undegraded":"2024-09-17T03:50:24.155490+0000","last_fullsized":"2024-09-17T03:50:24.155490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158511+0000","last_change":"2024-09-17T03:50:23.132879+0000","last_active":"2024-09-17T03:50:24.158511+0000","last_peered":"2024-09-17T03:50:24.158511+0000","last_clean":"2024-09-17T03:50:24.158511+0000","last_became_active":"2024-09-17T03:50:23.131983+0000","last_became_peered":"2024-09-17T03:50:23.131983+0000","last_unstale":"2024-09-17T03:50:24.158511+0000","last_undegraded":"2024-09-17T03:50:24.158511+0000","last_fullsized":"2024-09-17T03:50:24.158511+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:39.860630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161940+0000","last_change":"2024-09-17T03:50:24.156534+0000","last_active":"2024-09-17T03:50:24.161940+0000","last_peered":"2024-09-17T03:50:24.161940+0000","last_clean":"2024-09-17T03:50:24.161940+0000","last_became_active":"2024-09-17T03:50:24.156249+0000","last_became_peered":"2024-09-17T03:50:24.156249+0000","last_unstale":"2024-09-17T03:50:24.161940+0000","last_undegraded":"2024-09-17T03:50:24.161940+0000","last_fullsized":"2024-09-17T03:50:24.161940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:41.217570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154140+0000","last_change":"2024-09-17T03:50:23.134919+0000","last_active":"2024-09-17T03:50:24.154140+0000","last_peered":"2024-09-17T03:50:24.154140+0000","last_clean":"2024-09-17T03:50:24.154140+0000","last_became_active":"2024-09-17T03:50:23.134136+0000","last_became_peered":"2024-09-17T03:50:23.134136+0000","last_unstale":"2024-09-17T03:50:24.154140+0000","last_undegraded":"2024-09-17T03:50:24.154140+0000","last_fullsized":"2024-09-17T03:50:24.154140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:06.997836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157723+0000","last_change":"2024-09-17T03:50:24.153900+0000","last_active":"2024-09-17T03:50:24.157723+0000","last_peered":"2024-09-17T03:50:24.157723+0000","last_clean":"2024-09-17T03:50:24.157723+0000","last_became_active":"2024-09-17T03:50:24.153608+0000","last_became_peered":"2024-09-17T03:50:24.153608+0000","last_unstale":"2024-09-17T03:50:24.157723+0000","last_undegraded":"2024-09-17T03:50:24.157723+0000","last_fullsized":"2024-09-17T03:50:24.157723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160395+0000","last_change":"2024-09-17T03:50:23.142748+0000","last_active":"2024-09-17T03:50:24.160395+0000","last_peered":"2024-09-17T03:50:24.160395+0000","last_clean":"2024-09-17T03:50:24.160395+0000","last_became_active":"2024-09-17T03:50:23.142402+0000","last_became_peered":"2024-09-17T03:50:23.142402+0000","last_unstale":"2024-09-17T03:50:24.160395+0000","last_undegraded":"2024-09-17T03:50:24.160395+0000","last_fullsized":"2024-09-17T03:50:24.160395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:26.214882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.148116+0000","last_change":"2024-09-17T03:50:24.143306+0000","last_active":"2024-09-17T03:50:24.148116+0000","last_peered":"2024-09-17T03:50:24.148116+0000","last_clean":"2024-09-17T03:50:24.148116+0000","last_became_active":"2024-09-17T03:50:24.143003+0000","last_became_peered":"2024-09-17T03:50:24.143003+0000","last_unstale":"2024-09-17T03:50:24.148116+0000","last_undegraded":"2024-09-17T03:50:24.148116+0000","last_fullsized":"2024-09-17T03:50:24.148116+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160380+0000","last_change":"2024-09-17T03:50:23.144925+0000","last_active":"2024-09-17T03:50:24.160380+0000","last_peered":"2024-09-17T03:50:24.160380+0000","last_clean":"2024-09-17T03:50:24.160380+0000","last_became_active":"2024-09-17T03:50:23.144566+0000","last_became_peered":"2024-09-17T03:50:23.144566+0000","last_unstale":"2024-09-17T03:50:24.160380+0000","last_undegraded":"2024-09-17T03:50:24.160380+0000","last_fullsized":"2024-09-17T03:50:24.160380+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:11.013666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155822+0000","last_change":"2024-09-17T03:50:24.151450+0000","last_active":"2024-09-17T03:50:24.155822+0000","last_peered":"2024-09-17T03:50:24.155822+0000","last_clean":"2024-09-17T03:50:24.155822+0000","last_became_active":"2024-09-17T03:50:24.151115+0000","last_became_peered":"2024-09-17T03:50:24.151115+0000","last_unstale":"2024-09-17T03:50:24.155822+0000","last_undegraded":"2024-09-17T03:50:24.155822+0000","last_fullsized":"2024-09-17T03:50:24.155822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.002921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124578+0000","last_change":"2024-09-17T03:50:23.133927+0000","last_active":"2024-09-17T03:50:24.124578+0000","last_peered":"2024-09-17T03:50:24.124578+0000","last_clean":"2024-09-17T03:50:24.124578+0000","last_became_active":"2024-09-17T03:50:23.133670+0000","last_became_peered":"2024-09-17T03:50:23.133670+0000","last_unstale":"2024-09-17T03:50:24.124578+0000","last_undegraded":"2024-09-17T03:50:24.124578+0000","last_fullsized":"2024-09-17T03:50:24.124578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157681+0000","last_change":"2024-09-17T03:50:24.151434+0000","last_active":"2024-09-17T03:50:24.157681+0000","last_peered":"2024-09-17T03:50:24.157681+0000","last_clean":"2024-09-17T03:50:24.157681+0000","last_became_active":"2024-09-17T03:50:24.149266+0000","last_became_peered":"2024-09-17T03:50:24.149266+0000","last_unstale":"2024-09-17T03:50:24.157681+0000","last_undegraded":"2024-09-17T03:50:24.157681+0000","last_fullsized":"2024-09-17T03:50:24.157681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:50:43.760650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146994+0000","last_change":"2024-09-17T03:50:23.143447+0000","last_active":"2024-09-17T03:50:24.146994+0000","last_peered":"2024-09-17T03:50:24.146994+0000","last_clean":"2024-09-17T03:50:24.146994+0000","last_became_active":"2024-09-17T03:50:23.143053+0000","last_became_peered":"2024-09-17T03:50:23.143053+0000","last_unstale":"2024-09-17T03:50:24.146994+0000","last_undegraded":"2024-09-17T03:50:24.146994+0000","last_fullsized":"2024-09-17T03:50:24.146994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:45.122730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143714+0000","last_change":"2024-09-17T03:50:24.138823+0000","last_active":"2024-09-17T03:50:24.143714+0000","last_peered":"2024-09-17T03:50:24.143714+0000","last_clean":"2024-09-17T03:50:24.143714+0000","last_became_active":"2024-09-17T03:50:24.138293+0000","last_became_peered":"2024-09-17T03:50:24.138293+0000","last_unstale":"2024-09-17T03:50:24.143714+0000","last_undegraded":"2024-09-17T03:50:24.143714+0000","last_fullsized":"2024-09-17T03:50:24.143714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126274+0000","last_change":"2024-09-17T03:50:23.139166+0000","last_active":"2024-09-17T03:50:24.126274+0000","last_peered":"2024-09-17T03:50:24.126274+0000","last_clean":"2024-09-17T03:50:24.126274+0000","last_became_active":"2024-09-17T03:50:23.138826+0000","last_became_peered":"2024-09-17T03:50:23.138826+0000","last_unstale":"2024-09-17T03:50:24.126274+0000","last_undegraded":"2024-09-17T03:50:24.126274+0000","last_fullsized":"2024-09-17T03:50:24.126274+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:07.180415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162653+0000","last_change":"2024-09-17T03:50:24.157448+0000","last_active":"2024-09-17T03:50:24.162653+0000","last_peered":"2024-09-17T03:50:24.162653+0000","last_clean":"2024-09-17T03:50:24.162653+0000","last_became_active":"2024-09-17T03:50:24.156599+0000","last_became_peered":"2024-09-17T03:50:24.156599+0000","last_unstale":"2024-09-17T03:50:24.162653+0000","last_undegraded":"2024-09-17T03:50:24.162653+0000","last_fullsized":"2024-09-17T03:50:24.162653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.123299+0000","last_change":"2024-09-17T03:50:23.122274+0000","last_active":"2024-09-17T03:50:23.123299+0000","last_peered":"2024-09-17T03:50:23.123299+0000","last_clean":"2024-09-17T03:50:23.123299+0000","last_became_active":"2024-09-17T03:50:23.121928+0000","last_became_peered":"2024-09-17T03:50:23.121928+0000","last_unstale":"2024-09-17T03:50:23.123299+0000","last_undegraded":"2024-09-17T03:50:23.123299+0000","last_fullsized":"2024-09-17T03:50:23.123299+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142845+0000","last_change":"2024-09-17T03:50:24.138004+0000","last_active":"2024-09-17T03:50:24.142845+0000","last_peered":"2024-09-17T03:50:24.142845+0000","last_clean":"2024-09-17T03:50:24.142845+0000","last_became_active":"2024-09-17T03:50:24.137679+0000","last_became_peered":"2024-09-17T03:50:24.137679+0000","last_unstale":"2024-09-17T03:50:24.142845+0000","last_undegraded":"2024-09-17T03:50:24.142845+0000","last_fullsized":"2024-09-17T03:50:24.142845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142928+0000","last_change":"2024-09-17T03:50:23.133114+0000","last_active":"2024-09-17T03:50:24.142928+0000","last_peered":"2024-09-17T03:50:24.142928+0000","last_clean":"2024-09-17T03:50:24.142928+0000","last_became_active":"2024-09-17T03:50:23.132725+0000","last_became_peered":"2024-09-17T03:50:23.132725+0000","last_unstale":"2024-09-17T03:50:24.142928+0000","last_undegraded":"2024-09-17T03:50:24.142928+0000","last_fullsized":"2024-09-17T03:50:24.142928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150161+0000","last_change":"2024-09-17T03:50:24.145715+0000","last_active":"2024-09-17T03:50:24.150161+0000","last_peered":"2024-09-17T03:50:24.150161+0000","last_clean":"2024-09-17T03:50:24.150161+0000","last_became_active":"2024-09-17T03:50:24.145392+0000","last_became_peered":"2024-09-17T03:50:24.145392+0000","last_unstale":"2024-09-17T03:50:24.150161+0000","last_undegraded":"2024-09-17T03:50:24.150161+0000","last_fullsized":"2024-09-17T03:50:24.150161+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.163518+0000","last_change":"2024-09-17T03:50:23.140412+0000","last_active":"2024-09-17T03:50:24.163518+0000","last_peered":"2024-09-17T03:50:24.163518+0000","last_clean":"2024-09-17T03:50:24.163518+0000","last_became_active":"2024-09-17T03:50:23.139391+0000","last_became_peered":"2024-09-17T03:50:23.139391+0000","last_unstale":"2024-09-17T03:50:24.163518+0000","last_undegraded":"2024-09-17T03:50:24.163518+0000","last_fullsized":"2024-09-17T03:50:24.163518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153912+0000","last_change":"2024-09-17T03:50:24.149315+0000","last_active":"2024-09-17T03:50:24.153912+0000","last_peered":"2024-09-17T03:50:24.153912+0000","last_clean":"2024-09-17T03:50:24.153912+0000","last_became_active":"2024-09-17T03:50:24.147345+0000","last_became_peered":"2024-09-17T03:50:24.147345+0000","last_unstale":"2024-09-17T03:50:24.153912+0000","last_undegraded":"2024-09-17T03:50:24.153912+0000","last_fullsized":"2024-09-17T03:50:24.153912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:42.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.156776+0000","last_change":"2024-09-17T03:50:23.143154+0000","last_active":"2024-09-17T03:50:24.156776+0000","last_peered":"2024-09-17T03:50:24.156776+0000","last_clean":"2024-09-17T03:50:24.156776+0000","last_became_active":"2024-09-17T03:50:23.142801+0000","last_became_peered":"2024-09-17T03:50:23.142801+0000","last_unstale":"2024-09-17T03:50:24.156776+0000","last_undegraded":"2024-09-17T03:50:24.156776+0000","last_fullsized":"2024-09-17T03:50:24.156776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:17.669331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149769+0000","last_change":"2024-09-17T03:50:24.147109+0000","last_active":"2024-09-17T03:50:24.149769+0000","last_peered":"2024-09-17T03:50:24.149769+0000","last_clean":"2024-09-17T03:50:24.149769+0000","last_became_active":"2024-09-17T03:50:24.146394+0000","last_became_peered":"2024-09-17T03:50:24.146394+0000","last_unstale":"2024-09-17T03:50:24.149769+0000","last_undegraded":"2024-09-17T03:50:24.149769+0000","last_fullsized":"2024-09-17T03:50:24.149769+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129931+0000","last_change":"2024-09-17T03:50:23.135165+0000","last_active":"2024-09-17T03:50:24.129931+0000","last_peered":"2024-09-17T03:50:24.129931+0000","last_clean":"2024-09-17T03:50:24.129931+0000","last_became_active":"2024-09-17T03:50:23.134575+0000","last_became_peered":"2024-09-17T03:50:23.134575+0000","last_unstale":"2024-09-17T03:50:24.129931+0000","last_undegraded":"2024-09-17T03:50:24.129931+0000","last_fullsized":"2024-09-17T03:50:24.129931+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:53.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157786+0000","last_change":"2024-09-17T03:50:24.155803+0000","last_active":"2024-09-17T03:50:24.157786+0000","last_peered":"2024-09-17T03:50:24.157786+0000","last_clean":"2024-09-17T03:50:24.157786+0000","last_became_active":"2024-09-17T03:50:24.154916+0000","last_became_peered":"2024-09-17T03:50:24.154916+0000","last_unstale":"2024-09-17T03:50:24.157786+0000","last_undegraded":"2024-09-17T03:50:24.157786+0000","last_fullsized":"2024-09-17T03:50:24.157786+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124783+0000","last_change":"2024-09-17T03:50:23.127502+0000","last_active":"2024-09-17T03:50:24.124783+0000","last_peered":"2024-09-17T03:50:24.124783+0000","last_clean":"2024-09-17T03:50:24.124783+0000","last_became_active":"2024-09-17T03:50:23.126531+0000","last_became_peered":"2024-09-17T03:50:23.126531+0000","last_unstale":"2024-09-17T03:50:24.124783+0000","last_undegraded":"2024-09-17T03:50:24.124783+0000","last_fullsized":"2024-09-17T03:50:24.124783+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:39.251639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122281+0000","last_change":"2024-09-17T03:50:23.122281+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122281+0000","last_undegraded":"2024-09-17T03:50:23.122281+0000","last_fullsized":"2024-09-17T03:50:23.122281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":21,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160054+0000","last_change":"2024-09-17T03:50:23.139904+0000","last_active":"2024-09-17T03:50:24.160054+0000","last_peered":"2024-09-17T03:50:24.160054+0000","last_clean":"2024-09-17T03:50:24.160054+0000","last_became_active":"2024-09-17T03:50:23.139601+0000","last_became_peered":"2024-09-17T03:50:23.139601+0000","last_unstale":"2024-09-17T03:50:24.160054+0000","last_undegraded":"2024-09-17T03:50:24.160054+0000","last_fullsized":"2024-09-17T03:50:24.160054+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:21.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155131+0000","last_change":"2024-09-17T03:50:24.150202+0000","last_active":"2024-09-17T03:50:24.155131+0000","last_peered":"2024-09-17T03:50:24.155131+0000","last_clean":"2024-09-17T03:50:24.155131+0000","last_became_active":"2024-09-17T03:50:24.149551+0000","last_became_peered":"2024-09-17T03:50:24.149551+0000","last_unstale":"2024-09-17T03:50:24.155131+0000","last_undegraded":"2024-09-17T03:50:24.155131+0000","last_fullsized":"2024-09-17T03:50:24.155131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.396580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161395+0000","last_change":"2024-09-17T03:50:23.140181+0000","last_active":"2024-09-17T03:50:24.161395+0000","last_peered":"2024-09-17T03:50:24.161395+0000","last_clean":"2024-09-17T03:50:24.161395+0000","last_became_active":"2024-09-17T03:50:23.139817+0000","last_became_peered":"2024-09-17T03:50:23.139817+0000","last_unstale":"2024-09-17T03:50:24.161395+0000","last_undegraded":"2024-09-17T03:50:24.161395+0000","last_fullsized":"2024-09-17T03:50:24.161395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:10.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146249+0000","last_change":"2024-09-17T03:50:24.135542+0000","last_active":"2024-09-17T03:50:24.146249+0000","last_peered":"2024-09-17T03:50:24.146249+0000","last_clean":"2024-09-17T03:50:24.146249+0000","last_became_active":"2024-09-17T03:50:24.135241+0000","last_became_peered":"2024-09-17T03:50:24.135241+0000","last_unstale":"2024-09-17T03:50:24.146249+0000","last_undegraded":"2024-09-17T03:50:24.146249+0000","last_fullsized":"2024-09-17T03:50:24.146249+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149965+0000","last_change":"2024-09-17T03:50:23.139663+0000","last_active":"2024-09-17T03:50:24.149965+0000","last_peered":"2024-09-17T03:50:24.149965+0000","last_clean":"2024-09-17T03:50:24.149965+0000","last_became_active":"2024-09-17T03:50:23.139294+0000","last_became_peered":"2024-09-17T03:50:23.139294+0000","last_unstale":"2024-09-17T03:50:24.149965+0000","last_undegraded":"2024-09-17T03:50:24.149965+0000","last_fullsized":"2024-09-17T03:50:24.149965+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.179574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160646+0000","last_change":"2024-09-17T03:50:24.155061+0000","last_active":"2024-09-17T03:50:24.160646+0000","last_peered":"2024-09-17T03:50:24.160646+0000","last_clean":"2024-09-17T03:50:24.160646+0000","last_became_active":"2024-09-17T03:50:24.154794+0000","last_became_peered":"2024-09-17T03:50:24.154794+0000","last_unstale":"2024-09-17T03:50:24.160646+0000","last_undegraded":"2024-09-17T03:50:24.160646+0000","last_fullsized":"2024-09-17T03:50:24.160646+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:04.104460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154914+0000","last_change":"2024-09-17T03:50:23.143762+0000","last_active":"2024-09-17T03:50:24.154914+0000","last_peered":"2024-09-17T03:50:24.154914+0000","last_clean":"2024-09-17T03:50:24.154914+0000","last_became_active":"2024-09-17T03:50:23.143435+0000","last_became_peered":"2024-09-17T03:50:23.143435+0000","last_unstale":"2024-09-17T03:50:24.154914+0000","last_undegraded":"2024-09-17T03:50:24.154914+0000","last_fullsized":"2024-09-17T03:50:24.154914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:52.817756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143007+0000","last_change":"2024-09-17T03:50:24.138195+0000","last_active":"2024-09-17T03:50:24.143007+0000","last_peered":"2024-09-17T03:50:24.143007+0000","last_clean":"2024-09-17T03:50:24.143007+0000","last_became_active":"2024-09-17T03:50:24.137835+0000","last_became_peered":"2024-09-17T03:50:24.137835+0000","last_unstale":"2024-09-17T03:50:24.143007+0000","last_undegraded":"2024-09-17T03:50:24.143007+0000","last_fullsized":"2024-09-17T03:50:24.143007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:06.995259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129821+0000","last_change":"2024-09-17T03:50:23.140276+0000","last_active":"2024-09-17T03:50:24.129821+0000","last_peered":"2024-09-17T03:50:24.129821+0000","last_clean":"2024-09-17T03:50:24.129821+0000","last_became_active":"2024-09-17T03:50:23.139757+0000","last_became_peered":"2024-09-17T03:50:23.139757+0000","last_unstale":"2024-09-17T03:50:24.129821+0000","last_undegraded":"2024-09-17T03:50:24.129821+0000","last_fullsized":"2024-09-17T03:50:24.129821+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:14:45.951042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157939+0000","last_change":"2024-09-17T03:50:24.152597+0000","last_active":"2024-09-17T03:50:24.157939+0000","last_peered":"2024-09-17T03:50:24.157939+0000","last_clean":"2024-09-17T03:50:24.157939+0000","last_became_active":"2024-09-17T03:50:24.152307+0000","last_became_peered":"2024-09-17T03:50:24.152307+0000","last_unstale":"2024-09-17T03:50:24.157939+0000","last_undegraded":"2024-09-17T03:50:24.157939+0000","last_fullsized":"2024-09-17T03:50:24.157939+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:20.938433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.132753+0000","last_change":"2024-09-17T03:50:23.130742+0000","last_active":"2024-09-17T03:50:23.132753+0000","last_peered":"2024-09-17T03:50:23.132753+0000","last_clean":"2024-09-17T03:50:23.132753+0000","last_became_active":"2024-09-17T03:50:23.130434+0000","last_became_peered":"2024-09-17T03:50:23.130434+0000","last_unstale":"2024-09-17T03:50:23.132753+0000","last_undegraded":"2024-09-17T03:50:23.132753+0000","last_fullsized":"2024-09-17T03:50:23.132753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.030426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145900+0000","last_change":"2024-09-17T03:50:24.141883+0000","last_active":"2024-09-17T03:50:24.145900+0000","last_peered":"2024-09-17T03:50:24.145900+0000","last_clean":"2024-09-17T03:50:24.145900+0000","last_became_active":"2024-09-17T03:50:24.141572+0000","last_became_peered":"2024-09-17T03:50:24.141572+0000","last_unstale":"2024-09-17T03:50:24.145900+0000","last_undegraded":"2024-09-17T03:50:24.145900+0000","last_fullsized":"2024-09-17T03:50:24.145900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:36.875814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.138532+0000","last_change":"2024-09-17T03:50:23.133695+0000","last_active":"2024-09-17T03:50:23.138532+0000","last_peered":"2024-09-17T03:50:23.138532+0000","last_clean":"2024-09-17T03:50:23.138532+0000","last_became_active":"2024-09-17T03:50:23.133125+0000","last_became_peered":"2024-09-17T03:50:23.133125+0000","last_unstale":"2024-09-17T03:50:23.138532+0000","last_undegraded":"2024-09-17T03:50:23.138532+0000","last_fullsized":"2024-09-17T03:50:23.138532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145857+0000","last_change":"2024-09-17T03:50:24.141825+0000","last_active":"2024-09-17T03:50:24.145857+0000","last_peered":"2024-09-17T03:50:24.145857+0000","last_clean":"2024-09-17T03:50:24.145857+0000","last_became_active":"2024-09-17T03:50:24.141522+0000","last_became_peered":"2024-09-17T03:50:24.141522+0000","last_unstale":"2024-09-17T03:50:24.145857+0000","last_undegraded":"2024-09-17T03:50:24.145857+0000","last_fullsized":"2024-09-17T03:50:24.145857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:22.204438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131053+0000","last_change":"2024-09-17T03:50:23.141992+0000","last_active":"2024-09-17T03:50:24.131053+0000","last_peered":"2024-09-17T03:50:24.131053+0000","last_clean":"2024-09-17T03:50:24.131053+0000","last_became_active":"2024-09-17T03:50:23.141642+0000","last_became_peered":"2024-09-17T03:50:23.141642+0000","last_unstale":"2024-09-17T03:50:24.131053+0000","last_undegraded":"2024-09-17T03:50:24.131053+0000","last_fullsized":"2024-09-17T03:50:24.131053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:12.799906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.144614+0000","last_change":"2024-09-17T03:50:24.142339+0000","last_active":"2024-09-17T03:50:24.144614+0000","last_peered":"2024-09-17T03:50:24.144614+0000","last_clean":"2024-09-17T03:50:24.144614+0000","last_became_active":"2024-09-17T03:50:24.142028+0000","last_became_peered":"2024-09-17T03:50:24.142028+0000","last_unstale":"2024-09-17T03:50:24.144614+0000","last_undegraded":"2024-09-17T03:50:24.144614+0000","last_fullsized":"2024-09-17T03:50:24.144614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.118295+0000","last_change":"2024-09-17T03:50:23.129908+0000","last_active":"2024-09-17T03:50:24.118295+0000","last_peered":"2024-09-17T03:50:24.118295+0000","last_clean":"2024-09-17T03:50:24.118295+0000","last_became_active":"2024-09-17T03:50:23.129543+0000","last_became_peered":"2024-09-17T03:50:23.129543+0000","last_unstale":"2024-09-17T03:50:24.118295+0000","last_undegraded":"2024-09-17T03:50:24.118295+0000","last_fullsized":"2024-09-17T03:50:24.118295+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129144+0000","last_change":"2024-09-17T03:50:24.124588+0000","last_active":"2024-09-17T03:50:24.129144+0000","last_peered":"2024-09-17T03:50:24.129144+0000","last_clean":"2024-09-17T03:50:24.129144+0000","last_became_active":"2024-09-17T03:50:24.124178+0000","last_became_peered":"2024-09-17T03:50:24.124178+0000","last_unstale":"2024-09-17T03:50:24.129144+0000","last_undegraded":"2024-09-17T03:50:24.129144+0000","last_fullsized":"2024-09-17T03:50:24.129144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:20.307754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150943+0000","last_change":"2024-09-17T03:50:23.137622+0000","last_active":"2024-09-17T03:50:24.150943+0000","last_peered":"2024-09-17T03:50:24.150943+0000","last_clean":"2024-09-17T03:50:24.150943+0000","last_became_active":"2024-09-17T03:50:23.137258+0000","last_became_peered":"2024-09-17T03:50:23.137258+0000","last_unstale":"2024-09-17T03:50:24.150943+0000","last_undegraded":"2024-09-17T03:50:24.150943+0000","last_fullsized":"2024-09-17T03:50:24.150943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:40.356807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146377+0000","last_change":"2024-09-17T03:50:24.140577+0000","last_active":"2024-09-17T03:50:24.146377+0000","last_peered":"2024-09-17T03:50:24.146377+0000","last_clean":"2024-09-17T03:50:24.146377+0000","last_became_active":"2024-09-17T03:50:24.140330+0000","last_became_peered":"2024-09-17T03:50:24.140330+0000","last_unstale":"2024-09-17T03:50:24.146377+0000","last_undegraded":"2024-09-17T03:50:24.146377+0000","last_fullsized":"2024-09-17T03:50:24.146377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131834+0000","last_change":"2024-09-17T03:50:23.130628+0000","last_active":"2024-09-17T03:50:23.131834+0000","last_peered":"2024-09-17T03:50:23.131834+0000","last_clean":"2024-09-17T03:50:23.131834+0000","last_became_active":"2024-09-17T03:50:23.130307+0000","last_became_peered":"2024-09-17T03:50:23.130307+0000","last_unstale":"2024-09-17T03:50:23.131834+0000","last_undegraded":"2024-09-17T03:50:23.131834+0000","last_fullsized":"2024-09-17T03:50:23.131834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:51.417524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116716+0000","last_change":"2024-09-17T03:50:23.116716+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116716+0000","last_undegraded":"2024-09-17T03:50:23.116716+0000","last_fullsized":"2024-09-17T03:50:23.116716+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.128468+0000","last_change":"2024-09-17T03:50:23.136531+0000","last_active":"2024-09-17T03:50:24.128468+0000","last_peered":"2024-09-17T03:50:24.128468+0000","last_clean":"2024-09-17T03:50:24.128468+0000","last_became_active":"2024-09-17T03:50:23.136207+0000","last_became_peered":"2024-09-17T03:50:23.136207+0000","last_unstale":"2024-09-17T03:50:24.128468+0000","last_undegraded":"2024-09-17T03:50:24.128468+0000","last_fullsized":"2024-09-17T03:50:24.128468+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:47.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122595+0000","last_change":"2024-09-17T03:50:23.122595+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122595+0000","last_undegraded":"2024-09-17T03:50:23.122595+0000","last_fullsized":"2024-09-17T03:50:23.122595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.128999+0000","last_change":"2024-09-17T03:50:23.124066+0000","last_active":"2024-09-17T03:50:23.128999+0000","last_peered":"2024-09-17T03:50:23.128999+0000","last_clean":"2024-09-17T03:50:23.128999+0000","last_became_active":"2024-09-17T03:50:23.123776+0000","last_became_peered":"2024-09-17T03:50:23.123776+0000","last_unstale":"2024-09-17T03:50:23.128999+0000","last_undegraded":"2024-09-17T03:50:23.128999+0000","last_fullsized":"2024-09-17T03:50:23.128999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116315+0000","last_change":"2024-09-17T03:50:23.116315+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116315+0000","last_undegraded":"2024-09-17T03:50:23.116315+0000","last_fullsized":"2024-09-17T03:50:23.116315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.130118+0000","last_change":"2024-09-17T03:50:23.136967+0000","last_active":"2024-09-17T03:50:24.130118+0000","last_peered":"2024-09-17T03:50:24.130118+0000","last_clean":"2024-09-17T03:50:24.130118+0000","last_became_active":"2024-09-17T03:50:23.136643+0000","last_became_peered":"2024-09-17T03:50:23.136643+0000","last_unstale":"2024-09-17T03:50:24.130118+0000","last_undegraded":"2024-09-17T03:50:24.130118+0000","last_fullsized":"2024-09-17T03:50:24.130118+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:58.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158254+0000","last_change":"2024-09-17T03:50:24.150858+0000","last_active":"2024-09-17T03:50:24.158254+0000","last_peered":"2024-09-17T03:50:24.158254+0000","last_clean":"2024-09-17T03:50:24.158254+0000","last_became_active":"2024-09-17T03:50:24.150509+0000","last_became_peered":"2024-09-17T03:50:24.150509+0000","last_unstale":"2024-09-17T03:50:24.158254+0000","last_undegraded":"2024-09-17T03:50:24.158254+0000","last_fullsized":"2024-09-17T03:50:24.158254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131336+0000","last_change":"2024-09-17T03:50:23.141022+0000","last_active":"2024-09-17T03:50:24.131336+0000","last_peered":"2024-09-17T03:50:24.131336+0000","last_clean":"2024-09-17T03:50:24.131336+0000","last_became_active":"2024-09-17T03:50:23.140264+0000","last_became_peered":"2024-09-17T03:50:23.140264+0000","last_unstale":"2024-09-17T03:50:24.131336+0000","last_undegraded":"2024-09-17T03:50:24.131336+0000","last_fullsized":"2024-09-17T03:50:24.131336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:55.135377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155579+0000","last_change":"2024-09-17T03:50:24.149913+0000","last_active":"2024-09-17T03:50:24.155579+0000","last_peered":"2024-09-17T03:50:24.155579+0000","last_clean":"2024-09-17T03:50:24.155579+0000","last_became_active":"2024-09-17T03:50:24.149591+0000","last_became_peered":"2024-09-17T03:50:24.149591+0000","last_unstale":"2024-09-17T03:50:24.155579+0000","last_undegraded":"2024-09-17T03:50:24.155579+0000","last_fullsized":"2024-09-17T03:50:24.155579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:25.250775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126284+0000","last_change":"2024-09-17T03:50:23.130536+0000","last_active":"2024-09-17T03:50:24.126284+0000","last_peered":"2024-09-17T03:50:24.126284+0000","last_clean":"2024-09-17T03:50:24.126284+0000","last_became_active":"2024-09-17T03:50:23.130176+0000","last_became_peered":"2024-09-17T03:50:23.130176+0000","last_unstale":"2024-09-17T03:50:24.126284+0000","last_undegraded":"2024-09-17T03:50:24.126284+0000","last_fullsized":"2024-09-17T03:50:24.126284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:26.703578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150296+0000","last_change":"2024-09-17T03:50:24.146715+0000","last_active":"2024-09-17T03:50:24.150296+0000","last_peered":"2024-09-17T03:50:24.150296+0000","last_clean":"2024-09-17T03:50:24.150296+0000","last_became_active":"2024-09-17T03:50:24.146329+0000","last_became_peered":"2024-09-17T03:50:24.146329+0000","last_unstale":"2024-09-17T03:50:24.150296+0000","last_undegraded":"2024-09-17T03:50:24.150296+0000","last_fullsized":"2024-09-17T03:50:24.150296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142795+0000","last_change":"2024-09-17T03:50:23.138899+0000","last_active":"2024-09-17T03:50:23.142795+0000","last_peered":"2024-09-17T03:50:23.142795+0000","last_clean":"2024-09-17T03:50:23.142795+0000","last_became_active":"2024-09-17T03:50:23.138546+0000","last_became_peered":"2024-09-17T03:50:23.138546+0000","last_unstale":"2024-09-17T03:50:23.142795+0000","last_undegraded":"2024-09-17T03:50:23.142795+0000","last_fullsized":"2024-09-17T03:50:23.142795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":128,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129940+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:50:24.129940+0000","last_peered":"2024-09-17T03:50:24.129940+0000","last_clean":"2024-09-17T03:50:24.129940+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:50:24.129940+0000","last_undegraded":"2024-09-17T03:50:24.129940+0000","last_fullsized":"2024-09-17T03:50:24.129940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":933888,"data_stored":717659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509454,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":121024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542159,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":711392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:50:29.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:50:29.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 -- 172.21.15.129:0/639532493 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1080b40a0 msgr2=0x7ff108167450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 --2- 172.21.15.129:0/639532493 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1080b40a0 0x7ff108167450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff100009be0 tx=0x7ff10002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 -- 172.21.15.129:0/639532493 shutdown_connections 2024-09-17T03:50:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 --2- 172.21.15.129:0/639532493 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1080b40a0 0x7ff108167450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 -- 172.21.15.129:0/639532493 >> 172.21.15.129:0/639532493 conn(0x7ff1080b1cb0 msgr2=0x7ff1080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 -- 172.21.15.129:0/639532493 shutdown_connections 2024-09-17T03:50:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.283+0000 7ff10cc00640 1 -- 172.21.15.129:0/639532493 wait complete. 2024-09-17T03:50:29.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.285+0000 7ff10cc00640 1 Processor -- start 2024-09-17T03:50:29.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.285+0000 7ff10cc00640 1 -- start start 2024-09-17T03:50:29.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff108073170 0x7ff10814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1080b40a0 0x7ff1081533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff108167ba0 con 0x7ff108070030 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff108075090 con 0x7ff1080b40a0 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff10cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff108167d20 con 0x7ff108073170 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41248/0 (socket says 172.21.15.129:41248) 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 -- 172.21.15.129:0/2302502175 learned_addr learned my addr 172.21.15.129:0/2302502175 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff106800640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1080b40a0 0x7ff1081533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff0fd400640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff108073170 0x7ff10814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 -- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff108073170 msgr2=0x7ff10814e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff108073170 0x7ff10814e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 -- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1080b40a0 msgr2=0x7ff1081533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1080b40a0 0x7ff1081533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.287+0000 7ff105e00640 1 -- 172.21.15.129:0/2302502175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10814e800 con 0x7ff108070030 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff105e00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c4d0 tx=0x7ff0f800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8016020 con 0x7ff108070030 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff0f8012920 con 0x7ff108070030 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1080ca300 con 0x7ff108070030 2024-09-17T03:50:29.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8012d30 con 0x7ff108070030 2024-09-17T03:50:29.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.288+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1081f9f70 con 0x7ff108070030 2024-09-17T03:50:29.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.289+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff108074640 con 0x7ff108070030 2024-09-17T03:50:29.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.290+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff0f8019020 con 0x7ff108070030 2024-09-17T03:50:29.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.291+0000 7ff0fea00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 0x7ff0e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.291+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff0f809abd0 con 0x7ff108070030 2024-09-17T03:50:29.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.292+0000 7ff0fd400640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 0x7ff0e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.292+0000 7ff0fd400640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 0x7ff0e407b0a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff1080cb7a0 tx=0x7ff0f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:29.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.296+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff0f8020030 con 0x7ff108070030 2024-09-17T03:50:29.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.474+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1080c3860 con 0x7ff0e4078b90 2024-09-17T03:50:29.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.480+0000 7ff0fea00640 1 -- 172.21.15.129:0/2302502175 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251190 (secure 0 0 0) 0x7ff1080c3860 con 0x7ff0e4078b90 2024-09-17T03:50:29.483 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:29.485 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.483+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 msgr2=0x7ff0e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.483+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 0x7ff0e407b0a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff1080cb7a0 tx=0x7ff0f0009040 comp rx=0 tx=0).stop 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.483+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 msgr2=0x7ff1080cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.483+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff0f800c4d0 tx=0x7ff0f800c990 comp rx=0 tx=0).stop 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 shutdown_connections 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff0e4078b90 0x7ff0e407b0a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1080b40a0 0x7ff1081533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff108073170 0x7ff10814e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 --2- 172.21.15.129:0/2302502175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff108070030 0x7ff1080cba20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 >> 172.21.15.129:0/2302502175 conn(0x7ff1080b1cb0 msgr2=0x7ff108154d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:29.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 shutdown_connections 2024-09-17T03:50:29.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.484+0000 7ff10cc00640 1 -- 172.21.15.129:0/2302502175 wait complete. 2024-09-17T03:50:29.504 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":59,"stamp":"2024-09-17T03:50:28.306834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":140,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224180,"kb_used_data":3792,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750540,"statfs":{"total":773094113280,"available":772864552960,"internally_reserved":0,"allocated":3883008,"data_stored":2186530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001306"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124897+0000","last_change":"2024-09-17T03:50:23.124897+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124897+0000","last_undegraded":"2024-09-17T03:50:23.124897+0000","last_fullsized":"2024-09-17T03:50:23.124897+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149743+0000","last_change":"2024-09-17T03:50:24.146222+0000","last_active":"2024-09-17T03:50:24.149743+0000","last_peered":"2024-09-17T03:50:24.149743+0000","last_clean":"2024-09-17T03:50:24.149743+0000","last_became_active":"2024-09-17T03:50:24.145925+0000","last_became_peered":"2024-09-17T03:50:24.145925+0000","last_unstale":"2024-09-17T03:50:24.149743+0000","last_undegraded":"2024-09-17T03:50:24.149743+0000","last_fullsized":"2024-09-17T03:50:24.149743+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150192+0000","last_change":"2024-09-17T03:50:24.146390+0000","last_active":"2024-09-17T03:50:24.150192+0000","last_peered":"2024-09-17T03:50:24.150192+0000","last_clean":"2024-09-17T03:50:24.150192+0000","last_became_active":"2024-09-17T03:50:24.146067+0000","last_became_peered":"2024-09-17T03:50:24.146067+0000","last_unstale":"2024-09-17T03:50:24.150192+0000","last_undegraded":"2024-09-17T03:50:24.150192+0000","last_fullsized":"2024-09-17T03:50:24.150192+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161136+0000","last_change":"2024-09-17T03:50:24.156017+0000","last_active":"2024-09-17T03:50:24.161136+0000","last_peered":"2024-09-17T03:50:24.161136+0000","last_clean":"2024-09-17T03:50:24.161136+0000","last_became_active":"2024-09-17T03:50:24.155677+0000","last_became_peered":"2024-09-17T03:50:24.155677+0000","last_unstale":"2024-09-17T03:50:24.161136+0000","last_undegraded":"2024-09-17T03:50:24.161136+0000","last_fullsized":"2024-09-17T03:50:24.161136+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:13.449411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145398+0000","last_change":"2024-09-17T03:50:24.138985+0000","last_active":"2024-09-17T03:50:24.145398+0000","last_peered":"2024-09-17T03:50:24.145398+0000","last_clean":"2024-09-17T03:50:24.145398+0000","last_became_active":"2024-09-17T03:50:24.138569+0000","last_became_peered":"2024-09-17T03:50:24.138569+0000","last_unstale":"2024-09-17T03:50:24.145398+0000","last_undegraded":"2024-09-17T03:50:24.145398+0000","last_fullsized":"2024-09-17T03:50:24.145398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:51.159427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158315+0000","last_change":"2024-09-17T03:50:24.147892+0000","last_active":"2024-09-17T03:50:24.158315+0000","last_peered":"2024-09-17T03:50:24.158315+0000","last_clean":"2024-09-17T03:50:24.158315+0000","last_became_active":"2024-09-17T03:50:24.147616+0000","last_became_peered":"2024-09-17T03:50:24.147616+0000","last_unstale":"2024-09-17T03:50:24.158315+0000","last_undegraded":"2024-09-17T03:50:24.158315+0000","last_fullsized":"2024-09-17T03:50:24.158315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152457+0000","last_change":"2024-09-17T03:50:24.148414+0000","last_active":"2024-09-17T03:50:24.152457+0000","last_peered":"2024-09-17T03:50:24.152457+0000","last_clean":"2024-09-17T03:50:24.152457+0000","last_became_active":"2024-09-17T03:50:24.148020+0000","last_became_peered":"2024-09-17T03:50:24.148020+0000","last_unstale":"2024-09-17T03:50:24.152457+0000","last_undegraded":"2024-09-17T03:50:24.152457+0000","last_fullsized":"2024-09-17T03:50:24.152457+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150293+0000","last_change":"2024-09-17T03:50:24.147045+0000","last_active":"2024-09-17T03:50:24.150293+0000","last_peered":"2024-09-17T03:50:24.150293+0000","last_clean":"2024-09-17T03:50:24.150293+0000","last_became_active":"2024-09-17T03:50:24.146692+0000","last_became_peered":"2024-09-17T03:50:24.146692+0000","last_unstale":"2024-09-17T03:50:24.150293+0000","last_undegraded":"2024-09-17T03:50:24.150293+0000","last_fullsized":"2024-09-17T03:50:24.150293+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155032+0000","last_change":"2024-09-17T03:50:24.149261+0000","last_active":"2024-09-17T03:50:24.155032+0000","last_peered":"2024-09-17T03:50:24.155032+0000","last_clean":"2024-09-17T03:50:24.155032+0000","last_became_active":"2024-09-17T03:50:24.148898+0000","last_became_peered":"2024-09-17T03:50:24.148898+0000","last_unstale":"2024-09-17T03:50:24.155032+0000","last_undegraded":"2024-09-17T03:50:24.155032+0000","last_fullsized":"2024-09-17T03:50:24.155032+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:18:20.858522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160355+0000","last_change":"2024-09-17T03:50:24.152445+0000","last_active":"2024-09-17T03:50:24.160355+0000","last_peered":"2024-09-17T03:50:24.160355+0000","last_clean":"2024-09-17T03:50:24.160355+0000","last_became_active":"2024-09-17T03:50:24.152179+0000","last_became_peered":"2024-09-17T03:50:24.152179+0000","last_unstale":"2024-09-17T03:50:24.160355+0000","last_undegraded":"2024-09-17T03:50:24.160355+0000","last_fullsized":"2024-09-17T03:50:24.160355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:57:59.856041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157759+0000","last_change":"2024-09-17T03:50:24.155248+0000","last_active":"2024-09-17T03:50:24.157759+0000","last_peered":"2024-09-17T03:50:24.157759+0000","last_clean":"2024-09-17T03:50:24.157759+0000","last_became_active":"2024-09-17T03:50:24.154779+0000","last_became_peered":"2024-09-17T03:50:24.154779+0000","last_unstale":"2024-09-17T03:50:24.157759+0000","last_undegraded":"2024-09-17T03:50:24.157759+0000","last_fullsized":"2024-09-17T03:50:24.157759+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143858+0000","last_change":"2024-09-17T03:50:24.138736+0000","last_active":"2024-09-17T03:50:24.143858+0000","last_peered":"2024-09-17T03:50:24.143858+0000","last_clean":"2024-09-17T03:50:24.143858+0000","last_became_active":"2024-09-17T03:50:24.138424+0000","last_became_peered":"2024-09-17T03:50:24.138424+0000","last_unstale":"2024-09-17T03:50:24.143858+0000","last_undegraded":"2024-09-17T03:50:24.143858+0000","last_fullsized":"2024-09-17T03:50:24.143858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145922+0000","last_change":"2024-09-17T03:50:24.142434+0000","last_active":"2024-09-17T03:50:24.145922+0000","last_peered":"2024-09-17T03:50:24.145922+0000","last_clean":"2024-09-17T03:50:24.145922+0000","last_became_active":"2024-09-17T03:50:24.142083+0000","last_became_peered":"2024-09-17T03:50:24.142083+0000","last_unstale":"2024-09-17T03:50:24.145922+0000","last_undegraded":"2024-09-17T03:50:24.145922+0000","last_fullsized":"2024-09-17T03:50:24.145922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:49.596755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160739+0000","last_change":"2024-09-17T03:50:24.154707+0000","last_active":"2024-09-17T03:50:24.160739+0000","last_peered":"2024-09-17T03:50:24.160739+0000","last_clean":"2024-09-17T03:50:24.160739+0000","last_became_active":"2024-09-17T03:50:24.154328+0000","last_became_peered":"2024-09-17T03:50:24.154328+0000","last_unstale":"2024-09-17T03:50:24.160739+0000","last_undegraded":"2024-09-17T03:50:24.160739+0000","last_fullsized":"2024-09-17T03:50:24.160739+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145415+0000","last_change":"2024-09-17T03:50:24.125931+0000","last_active":"2024-09-17T03:50:24.145415+0000","last_peered":"2024-09-17T03:50:24.145415+0000","last_clean":"2024-09-17T03:50:24.145415+0000","last_became_active":"2024-09-17T03:50:24.125300+0000","last_became_peered":"2024-09-17T03:50:24.125300+0000","last_unstale":"2024-09-17T03:50:24.145415+0000","last_undegraded":"2024-09-17T03:50:24.145415+0000","last_fullsized":"2024-09-17T03:50:24.145415+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145644+0000","last_change":"2024-09-17T03:50:24.144536+0000","last_active":"2024-09-17T03:50:24.145644+0000","last_peered":"2024-09-17T03:50:24.145644+0000","last_clean":"2024-09-17T03:50:24.145644+0000","last_became_active":"2024-09-17T03:50:24.143960+0000","last_became_peered":"2024-09-17T03:50:24.143960+0000","last_unstale":"2024-09-17T03:50:24.145644+0000","last_undegraded":"2024-09-17T03:50:24.145644+0000","last_fullsized":"2024-09-17T03:50:24.145644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152188+0000","last_change":"2024-09-17T03:50:24.147697+0000","last_active":"2024-09-17T03:50:24.152188+0000","last_peered":"2024-09-17T03:50:24.152188+0000","last_clean":"2024-09-17T03:50:24.152188+0000","last_became_active":"2024-09-17T03:50:24.146930+0000","last_became_peered":"2024-09-17T03:50:24.146930+0000","last_unstale":"2024-09-17T03:50:24.152188+0000","last_undegraded":"2024-09-17T03:50:24.152188+0000","last_fullsized":"2024-09-17T03:50:24.152188+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162682+0000","last_change":"2024-09-17T03:50:24.159788+0000","last_active":"2024-09-17T03:50:24.162682+0000","last_peered":"2024-09-17T03:50:24.162682+0000","last_clean":"2024-09-17T03:50:24.162682+0000","last_became_active":"2024-09-17T03:50:24.159505+0000","last_became_peered":"2024-09-17T03:50:24.159505+0000","last_unstale":"2024-09-17T03:50:24.162682+0000","last_undegraded":"2024-09-17T03:50:24.162682+0000","last_fullsized":"2024-09-17T03:50:24.162682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161542+0000","last_change":"2024-09-17T03:50:24.154330+0000","last_active":"2024-09-17T03:50:24.161542+0000","last_peered":"2024-09-17T03:50:24.161542+0000","last_clean":"2024-09-17T03:50:24.161542+0000","last_became_active":"2024-09-17T03:50:24.153860+0000","last_became_peered":"2024-09-17T03:50:24.153860+0000","last_unstale":"2024-09-17T03:50:24.161542+0000","last_undegraded":"2024-09-17T03:50:24.161542+0000","last_fullsized":"2024-09-17T03:50:24.161542+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.132028+0000","last_change":"2024-09-17T03:50:24.128012+0000","last_active":"2024-09-17T03:50:24.132028+0000","last_peered":"2024-09-17T03:50:24.132028+0000","last_clean":"2024-09-17T03:50:24.132028+0000","last_became_active":"2024-09-17T03:50:24.127633+0000","last_became_peered":"2024-09-17T03:50:24.127633+0000","last_unstale":"2024-09-17T03:50:24.132028+0000","last_undegraded":"2024-09-17T03:50:24.132028+0000","last_fullsized":"2024-09-17T03:50:24.132028+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:14.977204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155095+0000","last_change":"2024-09-17T03:50:24.149865+0000","last_active":"2024-09-17T03:50:24.155095+0000","last_peered":"2024-09-17T03:50:24.155095+0000","last_clean":"2024-09-17T03:50:24.155095+0000","last_became_active":"2024-09-17T03:50:24.149569+0000","last_became_peered":"2024-09-17T03:50:24.149569+0000","last_unstale":"2024-09-17T03:50:24.155095+0000","last_undegraded":"2024-09-17T03:50:24.155095+0000","last_fullsized":"2024-09-17T03:50:24.155095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:02.922452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155400+0000","last_change":"2024-09-17T03:50:24.151774+0000","last_active":"2024-09-17T03:50:24.155400+0000","last_peered":"2024-09-17T03:50:24.155400+0000","last_clean":"2024-09-17T03:50:24.155400+0000","last_became_active":"2024-09-17T03:50:24.151379+0000","last_became_peered":"2024-09-17T03:50:24.151379+0000","last_unstale":"2024-09-17T03:50:24.155400+0000","last_undegraded":"2024-09-17T03:50:24.155400+0000","last_fullsized":"2024-09-17T03:50:24.155400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116168+0000","last_change":"2024-09-17T03:50:23.116168+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116168+0000","last_undegraded":"2024-09-17T03:50:23.116168+0000","last_fullsized":"2024-09-17T03:50:23.116168+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.138232+0000","last_change":"2024-09-17T03:50:24.132642+0000","last_active":"2024-09-17T03:50:24.138232+0000","last_peered":"2024-09-17T03:50:24.138232+0000","last_clean":"2024-09-17T03:50:24.138232+0000","last_became_active":"2024-09-17T03:50:24.131051+0000","last_became_peered":"2024-09-17T03:50:24.131051+0000","last_unstale":"2024-09-17T03:50:24.138232+0000","last_undegraded":"2024-09-17T03:50:24.138232+0000","last_fullsized":"2024-09-17T03:50:24.138232+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153734+0000","last_change":"2024-09-17T03:50:24.145119+0000","last_active":"2024-09-17T03:50:24.153734+0000","last_peered":"2024-09-17T03:50:24.153734+0000","last_clean":"2024-09-17T03:50:24.153734+0000","last_became_active":"2024-09-17T03:50:24.144619+0000","last_became_peered":"2024-09-17T03:50:24.144619+0000","last_unstale":"2024-09-17T03:50:24.153734+0000","last_undegraded":"2024-09-17T03:50:24.153734+0000","last_fullsized":"2024-09-17T03:50:24.153734+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:44.451270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142707+0000","last_change":"2024-09-17T03:50:24.136492+0000","last_active":"2024-09-17T03:50:24.142707+0000","last_peered":"2024-09-17T03:50:24.142707+0000","last_clean":"2024-09-17T03:50:24.142707+0000","last_became_active":"2024-09-17T03:50:24.136206+0000","last_became_peered":"2024-09-17T03:50:24.136206+0000","last_unstale":"2024-09-17T03:50:24.142707+0000","last_undegraded":"2024-09-17T03:50:24.142707+0000","last_fullsized":"2024-09-17T03:50:24.142707+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161913+0000","last_change":"2024-09-17T03:50:24.155949+0000","last_active":"2024-09-17T03:50:24.161913+0000","last_peered":"2024-09-17T03:50:24.161913+0000","last_clean":"2024-09-17T03:50:24.161913+0000","last_became_active":"2024-09-17T03:50:24.155692+0000","last_became_peered":"2024-09-17T03:50:24.155692+0000","last_unstale":"2024-09-17T03:50:24.161913+0000","last_undegraded":"2024-09-17T03:50:24.161913+0000","last_fullsized":"2024-09-17T03:50:24.161913+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:11.942003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142811+0000","last_change":"2024-09-17T03:50:24.136298+0000","last_active":"2024-09-17T03:50:24.142811+0000","last_peered":"2024-09-17T03:50:24.142811+0000","last_clean":"2024-09-17T03:50:24.142811+0000","last_became_active":"2024-09-17T03:50:24.135828+0000","last_became_peered":"2024-09-17T03:50:24.135828+0000","last_unstale":"2024-09-17T03:50:24.142811+0000","last_undegraded":"2024-09-17T03:50:24.142811+0000","last_fullsized":"2024-09-17T03:50:24.142811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155375+0000","last_change":"2024-09-17T03:50:24.152059+0000","last_active":"2024-09-17T03:50:24.155375+0000","last_peered":"2024-09-17T03:50:24.155375+0000","last_clean":"2024-09-17T03:50:24.155375+0000","last_became_active":"2024-09-17T03:50:24.151751+0000","last_became_peered":"2024-09-17T03:50:24.151751+0000","last_unstale":"2024-09-17T03:50:24.155375+0000","last_undegraded":"2024-09-17T03:50:24.155375+0000","last_fullsized":"2024-09-17T03:50:24.155375+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:10.943686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.152180+0000","last_change":"2024-09-17T03:50:24.147844+0000","last_active":"2024-09-17T03:50:24.152180+0000","last_peered":"2024-09-17T03:50:24.152180+0000","last_clean":"2024-09-17T03:50:24.152180+0000","last_became_active":"2024-09-17T03:50:24.144862+0000","last_became_peered":"2024-09-17T03:50:24.144862+0000","last_unstale":"2024-09-17T03:50:24.152180+0000","last_undegraded":"2024-09-17T03:50:24.152180+0000","last_fullsized":"2024-09-17T03:50:24.152180+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.123020+0000","last_change":"2024-09-17T03:50:23.123020+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.123020+0000","last_undegraded":"2024-09-17T03:50:23.123020+0000","last_fullsized":"2024-09-17T03:50:23.123020+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157205+0000","last_change":"2024-09-17T03:50:24.153513+0000","last_active":"2024-09-17T03:50:24.157205+0000","last_peered":"2024-09-17T03:50:24.157205+0000","last_clean":"2024-09-17T03:50:24.157205+0000","last_became_active":"2024-09-17T03:50:24.153118+0000","last_became_peered":"2024-09-17T03:50:24.153118+0000","last_unstale":"2024-09-17T03:50:24.157205+0000","last_undegraded":"2024-09-17T03:50:24.157205+0000","last_fullsized":"2024-09-17T03:50:24.157205+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116164+0000","last_change":"2024-09-17T03:50:23.116164+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116164+0000","last_undegraded":"2024-09-17T03:50:23.116164+0000","last_fullsized":"2024-09-17T03:50:23.116164+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142829+0000","last_change":"2024-09-17T03:50:23.138230+0000","last_active":"2024-09-17T03:50:23.142829+0000","last_peered":"2024-09-17T03:50:23.142829+0000","last_clean":"2024-09-17T03:50:23.142829+0000","last_became_active":"2024-09-17T03:50:23.137347+0000","last_became_peered":"2024-09-17T03:50:23.137347+0000","last_unstale":"2024-09-17T03:50:23.142829+0000","last_undegraded":"2024-09-17T03:50:23.142829+0000","last_fullsized":"2024-09-17T03:50:23.142829+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.124400+0000","last_change":"2024-09-17T03:50:23.124400+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.124400+0000","last_undegraded":"2024-09-17T03:50:23.124400+0000","last_fullsized":"2024-09-17T03:50:23.124400+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131789+0000","last_change":"2024-09-17T03:50:23.128500+0000","last_active":"2024-09-17T03:50:23.131789+0000","last_peered":"2024-09-17T03:50:23.131789+0000","last_clean":"2024-09-17T03:50:23.131789+0000","last_became_active":"2024-09-17T03:50:23.128184+0000","last_became_peered":"2024-09-17T03:50:23.128184+0000","last_unstale":"2024-09-17T03:50:23.131789+0000","last_undegraded":"2024-09-17T03:50:23.131789+0000","last_fullsized":"2024-09-17T03:50:23.131789+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155490+0000","last_change":"2024-09-17T03:50:24.152519+0000","last_active":"2024-09-17T03:50:24.155490+0000","last_peered":"2024-09-17T03:50:24.155490+0000","last_clean":"2024-09-17T03:50:24.155490+0000","last_became_active":"2024-09-17T03:50:24.150781+0000","last_became_peered":"2024-09-17T03:50:24.150781+0000","last_unstale":"2024-09-17T03:50:24.155490+0000","last_undegraded":"2024-09-17T03:50:24.155490+0000","last_fullsized":"2024-09-17T03:50:24.155490+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158511+0000","last_change":"2024-09-17T03:50:23.132879+0000","last_active":"2024-09-17T03:50:24.158511+0000","last_peered":"2024-09-17T03:50:24.158511+0000","last_clean":"2024-09-17T03:50:24.158511+0000","last_became_active":"2024-09-17T03:50:23.131983+0000","last_became_peered":"2024-09-17T03:50:23.131983+0000","last_unstale":"2024-09-17T03:50:24.158511+0000","last_undegraded":"2024-09-17T03:50:24.158511+0000","last_fullsized":"2024-09-17T03:50:24.158511+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:39.860630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161940+0000","last_change":"2024-09-17T03:50:24.156534+0000","last_active":"2024-09-17T03:50:24.161940+0000","last_peered":"2024-09-17T03:50:24.161940+0000","last_clean":"2024-09-17T03:50:24.161940+0000","last_became_active":"2024-09-17T03:50:24.156249+0000","last_became_peered":"2024-09-17T03:50:24.156249+0000","last_unstale":"2024-09-17T03:50:24.161940+0000","last_undegraded":"2024-09-17T03:50:24.161940+0000","last_fullsized":"2024-09-17T03:50:24.161940+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:41.217570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154140+0000","last_change":"2024-09-17T03:50:23.134919+0000","last_active":"2024-09-17T03:50:24.154140+0000","last_peered":"2024-09-17T03:50:24.154140+0000","last_clean":"2024-09-17T03:50:24.154140+0000","last_became_active":"2024-09-17T03:50:23.134136+0000","last_became_peered":"2024-09-17T03:50:23.134136+0000","last_unstale":"2024-09-17T03:50:24.154140+0000","last_undegraded":"2024-09-17T03:50:24.154140+0000","last_fullsized":"2024-09-17T03:50:24.154140+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:06.997836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157723+0000","last_change":"2024-09-17T03:50:24.153900+0000","last_active":"2024-09-17T03:50:24.157723+0000","last_peered":"2024-09-17T03:50:24.157723+0000","last_clean":"2024-09-17T03:50:24.157723+0000","last_became_active":"2024-09-17T03:50:24.153608+0000","last_became_peered":"2024-09-17T03:50:24.153608+0000","last_unstale":"2024-09-17T03:50:24.157723+0000","last_undegraded":"2024-09-17T03:50:24.157723+0000","last_fullsized":"2024-09-17T03:50:24.157723+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160395+0000","last_change":"2024-09-17T03:50:23.142748+0000","last_active":"2024-09-17T03:50:24.160395+0000","last_peered":"2024-09-17T03:50:24.160395+0000","last_clean":"2024-09-17T03:50:24.160395+0000","last_became_active":"2024-09-17T03:50:23.142402+0000","last_became_peered":"2024-09-17T03:50:23.142402+0000","last_unstale":"2024-09-17T03:50:24.160395+0000","last_undegraded":"2024-09-17T03:50:24.160395+0000","last_fullsized":"2024-09-17T03:50:24.160395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:26.214882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.148116+0000","last_change":"2024-09-17T03:50:24.143306+0000","last_active":"2024-09-17T03:50:24.148116+0000","last_peered":"2024-09-17T03:50:24.148116+0000","last_clean":"2024-09-17T03:50:24.148116+0000","last_became_active":"2024-09-17T03:50:24.143003+0000","last_became_peered":"2024-09-17T03:50:24.143003+0000","last_unstale":"2024-09-17T03:50:24.148116+0000","last_undegraded":"2024-09-17T03:50:24.148116+0000","last_fullsized":"2024-09-17T03:50:24.148116+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160380+0000","last_change":"2024-09-17T03:50:23.144925+0000","last_active":"2024-09-17T03:50:24.160380+0000","last_peered":"2024-09-17T03:50:24.160380+0000","last_clean":"2024-09-17T03:50:24.160380+0000","last_became_active":"2024-09-17T03:50:23.144566+0000","last_became_peered":"2024-09-17T03:50:23.144566+0000","last_unstale":"2024-09-17T03:50:24.160380+0000","last_undegraded":"2024-09-17T03:50:24.160380+0000","last_fullsized":"2024-09-17T03:50:24.160380+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:11.013666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155822+0000","last_change":"2024-09-17T03:50:24.151450+0000","last_active":"2024-09-17T03:50:24.155822+0000","last_peered":"2024-09-17T03:50:24.155822+0000","last_clean":"2024-09-17T03:50:24.155822+0000","last_became_active":"2024-09-17T03:50:24.151115+0000","last_became_peered":"2024-09-17T03:50:24.151115+0000","last_unstale":"2024-09-17T03:50:24.155822+0000","last_undegraded":"2024-09-17T03:50:24.155822+0000","last_fullsized":"2024-09-17T03:50:24.155822+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:54.002921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124578+0000","last_change":"2024-09-17T03:50:23.133927+0000","last_active":"2024-09-17T03:50:24.124578+0000","last_peered":"2024-09-17T03:50:24.124578+0000","last_clean":"2024-09-17T03:50:24.124578+0000","last_became_active":"2024-09-17T03:50:23.133670+0000","last_became_peered":"2024-09-17T03:50:23.133670+0000","last_unstale":"2024-09-17T03:50:24.124578+0000","last_undegraded":"2024-09-17T03:50:24.124578+0000","last_fullsized":"2024-09-17T03:50:24.124578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157681+0000","last_change":"2024-09-17T03:50:24.151434+0000","last_active":"2024-09-17T03:50:24.157681+0000","last_peered":"2024-09-17T03:50:24.157681+0000","last_clean":"2024-09-17T03:50:24.157681+0000","last_became_active":"2024-09-17T03:50:24.149266+0000","last_became_peered":"2024-09-17T03:50:24.149266+0000","last_unstale":"2024-09-17T03:50:24.157681+0000","last_undegraded":"2024-09-17T03:50:24.157681+0000","last_fullsized":"2024-09-17T03:50:24.157681+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:50:43.760650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146994+0000","last_change":"2024-09-17T03:50:23.143447+0000","last_active":"2024-09-17T03:50:24.146994+0000","last_peered":"2024-09-17T03:50:24.146994+0000","last_clean":"2024-09-17T03:50:24.146994+0000","last_became_active":"2024-09-17T03:50:23.143053+0000","last_became_peered":"2024-09-17T03:50:23.143053+0000","last_unstale":"2024-09-17T03:50:24.146994+0000","last_undegraded":"2024-09-17T03:50:24.146994+0000","last_fullsized":"2024-09-17T03:50:24.146994+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:45.122730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143714+0000","last_change":"2024-09-17T03:50:24.138823+0000","last_active":"2024-09-17T03:50:24.143714+0000","last_peered":"2024-09-17T03:50:24.143714+0000","last_clean":"2024-09-17T03:50:24.143714+0000","last_became_active":"2024-09-17T03:50:24.138293+0000","last_became_peered":"2024-09-17T03:50:24.138293+0000","last_unstale":"2024-09-17T03:50:24.143714+0000","last_undegraded":"2024-09-17T03:50:24.143714+0000","last_fullsized":"2024-09-17T03:50:24.143714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126274+0000","last_change":"2024-09-17T03:50:23.139166+0000","last_active":"2024-09-17T03:50:24.126274+0000","last_peered":"2024-09-17T03:50:24.126274+0000","last_clean":"2024-09-17T03:50:24.126274+0000","last_became_active":"2024-09-17T03:50:23.138826+0000","last_became_peered":"2024-09-17T03:50:23.138826+0000","last_unstale":"2024-09-17T03:50:24.126274+0000","last_undegraded":"2024-09-17T03:50:24.126274+0000","last_fullsized":"2024-09-17T03:50:24.126274+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:07.180415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.162653+0000","last_change":"2024-09-17T03:50:24.157448+0000","last_active":"2024-09-17T03:50:24.162653+0000","last_peered":"2024-09-17T03:50:24.162653+0000","last_clean":"2024-09-17T03:50:24.162653+0000","last_became_active":"2024-09-17T03:50:24.156599+0000","last_became_peered":"2024-09-17T03:50:24.156599+0000","last_unstale":"2024-09-17T03:50:24.162653+0000","last_undegraded":"2024-09-17T03:50:24.162653+0000","last_fullsized":"2024-09-17T03:50:24.162653+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.123299+0000","last_change":"2024-09-17T03:50:23.122274+0000","last_active":"2024-09-17T03:50:23.123299+0000","last_peered":"2024-09-17T03:50:23.123299+0000","last_clean":"2024-09-17T03:50:23.123299+0000","last_became_active":"2024-09-17T03:50:23.121928+0000","last_became_peered":"2024-09-17T03:50:23.121928+0000","last_unstale":"2024-09-17T03:50:23.123299+0000","last_undegraded":"2024-09-17T03:50:23.123299+0000","last_fullsized":"2024-09-17T03:50:23.123299+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142845+0000","last_change":"2024-09-17T03:50:24.138004+0000","last_active":"2024-09-17T03:50:24.142845+0000","last_peered":"2024-09-17T03:50:24.142845+0000","last_clean":"2024-09-17T03:50:24.142845+0000","last_became_active":"2024-09-17T03:50:24.137679+0000","last_became_peered":"2024-09-17T03:50:24.137679+0000","last_unstale":"2024-09-17T03:50:24.142845+0000","last_undegraded":"2024-09-17T03:50:24.142845+0000","last_fullsized":"2024-09-17T03:50:24.142845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.142928+0000","last_change":"2024-09-17T03:50:23.133114+0000","last_active":"2024-09-17T03:50:24.142928+0000","last_peered":"2024-09-17T03:50:24.142928+0000","last_clean":"2024-09-17T03:50:24.142928+0000","last_became_active":"2024-09-17T03:50:23.132725+0000","last_became_peered":"2024-09-17T03:50:23.132725+0000","last_unstale":"2024-09-17T03:50:24.142928+0000","last_undegraded":"2024-09-17T03:50:24.142928+0000","last_fullsized":"2024-09-17T03:50:24.142928+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150161+0000","last_change":"2024-09-17T03:50:24.145715+0000","last_active":"2024-09-17T03:50:24.150161+0000","last_peered":"2024-09-17T03:50:24.150161+0000","last_clean":"2024-09-17T03:50:24.150161+0000","last_became_active":"2024-09-17T03:50:24.145392+0000","last_became_peered":"2024-09-17T03:50:24.145392+0000","last_unstale":"2024-09-17T03:50:24.150161+0000","last_undegraded":"2024-09-17T03:50:24.150161+0000","last_fullsized":"2024-09-17T03:50:24.150161+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":19,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.163518+0000","last_change":"2024-09-17T03:50:23.140412+0000","last_active":"2024-09-17T03:50:24.163518+0000","last_peered":"2024-09-17T03:50:24.163518+0000","last_clean":"2024-09-17T03:50:24.163518+0000","last_became_active":"2024-09-17T03:50:23.139391+0000","last_became_peered":"2024-09-17T03:50:23.139391+0000","last_unstale":"2024-09-17T03:50:24.163518+0000","last_undegraded":"2024-09-17T03:50:24.163518+0000","last_fullsized":"2024-09-17T03:50:24.163518+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.153912+0000","last_change":"2024-09-17T03:50:24.149315+0000","last_active":"2024-09-17T03:50:24.153912+0000","last_peered":"2024-09-17T03:50:24.153912+0000","last_clean":"2024-09-17T03:50:24.153912+0000","last_became_active":"2024-09-17T03:50:24.147345+0000","last_became_peered":"2024-09-17T03:50:24.147345+0000","last_unstale":"2024-09-17T03:50:24.153912+0000","last_undegraded":"2024-09-17T03:50:24.153912+0000","last_fullsized":"2024-09-17T03:50:24.153912+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:42.420635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.156776+0000","last_change":"2024-09-17T03:50:23.143154+0000","last_active":"2024-09-17T03:50:24.156776+0000","last_peered":"2024-09-17T03:50:24.156776+0000","last_clean":"2024-09-17T03:50:24.156776+0000","last_became_active":"2024-09-17T03:50:23.142801+0000","last_became_peered":"2024-09-17T03:50:23.142801+0000","last_unstale":"2024-09-17T03:50:24.156776+0000","last_undegraded":"2024-09-17T03:50:24.156776+0000","last_fullsized":"2024-09-17T03:50:24.156776+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:17.669331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149769+0000","last_change":"2024-09-17T03:50:24.147109+0000","last_active":"2024-09-17T03:50:24.149769+0000","last_peered":"2024-09-17T03:50:24.149769+0000","last_clean":"2024-09-17T03:50:24.149769+0000","last_became_active":"2024-09-17T03:50:24.146394+0000","last_became_peered":"2024-09-17T03:50:24.146394+0000","last_unstale":"2024-09-17T03:50:24.149769+0000","last_undegraded":"2024-09-17T03:50:24.149769+0000","last_fullsized":"2024-09-17T03:50:24.149769+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129931+0000","last_change":"2024-09-17T03:50:23.135165+0000","last_active":"2024-09-17T03:50:24.129931+0000","last_peered":"2024-09-17T03:50:24.129931+0000","last_clean":"2024-09-17T03:50:24.129931+0000","last_became_active":"2024-09-17T03:50:23.134575+0000","last_became_peered":"2024-09-17T03:50:23.134575+0000","last_unstale":"2024-09-17T03:50:24.129931+0000","last_undegraded":"2024-09-17T03:50:24.129931+0000","last_fullsized":"2024-09-17T03:50:24.129931+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:53.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157786+0000","last_change":"2024-09-17T03:50:24.155803+0000","last_active":"2024-09-17T03:50:24.157786+0000","last_peered":"2024-09-17T03:50:24.157786+0000","last_clean":"2024-09-17T03:50:24.157786+0000","last_became_active":"2024-09-17T03:50:24.154916+0000","last_became_peered":"2024-09-17T03:50:24.154916+0000","last_unstale":"2024-09-17T03:50:24.157786+0000","last_undegraded":"2024-09-17T03:50:24.157786+0000","last_fullsized":"2024-09-17T03:50:24.157786+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.124783+0000","last_change":"2024-09-17T03:50:23.127502+0000","last_active":"2024-09-17T03:50:24.124783+0000","last_peered":"2024-09-17T03:50:24.124783+0000","last_clean":"2024-09-17T03:50:24.124783+0000","last_became_active":"2024-09-17T03:50:23.126531+0000","last_became_peered":"2024-09-17T03:50:23.126531+0000","last_unstale":"2024-09-17T03:50:24.124783+0000","last_undegraded":"2024-09-17T03:50:24.124783+0000","last_fullsized":"2024-09-17T03:50:24.124783+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:39.251639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122281+0000","last_change":"2024-09-17T03:50:23.122281+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122281+0000","last_undegraded":"2024-09-17T03:50:23.122281+0000","last_fullsized":"2024-09-17T03:50:23.122281+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":21,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160054+0000","last_change":"2024-09-17T03:50:23.139904+0000","last_active":"2024-09-17T03:50:24.160054+0000","last_peered":"2024-09-17T03:50:24.160054+0000","last_clean":"2024-09-17T03:50:24.160054+0000","last_became_active":"2024-09-17T03:50:23.139601+0000","last_became_peered":"2024-09-17T03:50:23.139601+0000","last_unstale":"2024-09-17T03:50:24.160054+0000","last_undegraded":"2024-09-17T03:50:24.160054+0000","last_fullsized":"2024-09-17T03:50:24.160054+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:21.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155131+0000","last_change":"2024-09-17T03:50:24.150202+0000","last_active":"2024-09-17T03:50:24.155131+0000","last_peered":"2024-09-17T03:50:24.155131+0000","last_clean":"2024-09-17T03:50:24.155131+0000","last_became_active":"2024-09-17T03:50:24.149551+0000","last_became_peered":"2024-09-17T03:50:24.149551+0000","last_unstale":"2024-09-17T03:50:24.155131+0000","last_undegraded":"2024-09-17T03:50:24.155131+0000","last_fullsized":"2024-09-17T03:50:24.155131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:16.396580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.161395+0000","last_change":"2024-09-17T03:50:23.140181+0000","last_active":"2024-09-17T03:50:24.161395+0000","last_peered":"2024-09-17T03:50:24.161395+0000","last_clean":"2024-09-17T03:50:24.161395+0000","last_became_active":"2024-09-17T03:50:23.139817+0000","last_became_peered":"2024-09-17T03:50:23.139817+0000","last_unstale":"2024-09-17T03:50:24.161395+0000","last_undegraded":"2024-09-17T03:50:24.161395+0000","last_fullsized":"2024-09-17T03:50:24.161395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:10.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146249+0000","last_change":"2024-09-17T03:50:24.135542+0000","last_active":"2024-09-17T03:50:24.146249+0000","last_peered":"2024-09-17T03:50:24.146249+0000","last_clean":"2024-09-17T03:50:24.146249+0000","last_became_active":"2024-09-17T03:50:24.135241+0000","last_became_peered":"2024-09-17T03:50:24.135241+0000","last_unstale":"2024-09-17T03:50:24.146249+0000","last_undegraded":"2024-09-17T03:50:24.146249+0000","last_fullsized":"2024-09-17T03:50:24.146249+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.149965+0000","last_change":"2024-09-17T03:50:23.139663+0000","last_active":"2024-09-17T03:50:24.149965+0000","last_peered":"2024-09-17T03:50:24.149965+0000","last_clean":"2024-09-17T03:50:24.149965+0000","last_became_active":"2024-09-17T03:50:23.139294+0000","last_became_peered":"2024-09-17T03:50:23.139294+0000","last_unstale":"2024-09-17T03:50:24.149965+0000","last_undegraded":"2024-09-17T03:50:24.149965+0000","last_fullsized":"2024-09-17T03:50:24.149965+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.179574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.160646+0000","last_change":"2024-09-17T03:50:24.155061+0000","last_active":"2024-09-17T03:50:24.160646+0000","last_peered":"2024-09-17T03:50:24.160646+0000","last_clean":"2024-09-17T03:50:24.160646+0000","last_became_active":"2024-09-17T03:50:24.154794+0000","last_became_peered":"2024-09-17T03:50:24.154794+0000","last_unstale":"2024-09-17T03:50:24.160646+0000","last_undegraded":"2024-09-17T03:50:24.160646+0000","last_fullsized":"2024-09-17T03:50:24.160646+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:04.104460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.154914+0000","last_change":"2024-09-17T03:50:23.143762+0000","last_active":"2024-09-17T03:50:24.154914+0000","last_peered":"2024-09-17T03:50:24.154914+0000","last_clean":"2024-09-17T03:50:24.154914+0000","last_became_active":"2024-09-17T03:50:23.143435+0000","last_became_peered":"2024-09-17T03:50:23.143435+0000","last_unstale":"2024-09-17T03:50:24.154914+0000","last_undegraded":"2024-09-17T03:50:24.154914+0000","last_fullsized":"2024-09-17T03:50:24.154914+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:52.817756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.143007+0000","last_change":"2024-09-17T03:50:24.138195+0000","last_active":"2024-09-17T03:50:24.143007+0000","last_peered":"2024-09-17T03:50:24.143007+0000","last_clean":"2024-09-17T03:50:24.143007+0000","last_became_active":"2024-09-17T03:50:24.137835+0000","last_became_peered":"2024-09-17T03:50:24.137835+0000","last_unstale":"2024-09-17T03:50:24.143007+0000","last_undegraded":"2024-09-17T03:50:24.143007+0000","last_fullsized":"2024-09-17T03:50:24.143007+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:06.995259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129821+0000","last_change":"2024-09-17T03:50:23.140276+0000","last_active":"2024-09-17T03:50:24.129821+0000","last_peered":"2024-09-17T03:50:24.129821+0000","last_clean":"2024-09-17T03:50:24.129821+0000","last_became_active":"2024-09-17T03:50:23.139757+0000","last_became_peered":"2024-09-17T03:50:23.139757+0000","last_unstale":"2024-09-17T03:50:24.129821+0000","last_undegraded":"2024-09-17T03:50:24.129821+0000","last_fullsized":"2024-09-17T03:50:24.129821+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:14:45.951042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.157939+0000","last_change":"2024-09-17T03:50:24.152597+0000","last_active":"2024-09-17T03:50:24.157939+0000","last_peered":"2024-09-17T03:50:24.157939+0000","last_clean":"2024-09-17T03:50:24.157939+0000","last_became_active":"2024-09-17T03:50:24.152307+0000","last_became_peered":"2024-09-17T03:50:24.152307+0000","last_unstale":"2024-09-17T03:50:24.157939+0000","last_undegraded":"2024-09-17T03:50:24.157939+0000","last_fullsized":"2024-09-17T03:50:24.157939+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:20.938433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.132753+0000","last_change":"2024-09-17T03:50:23.130742+0000","last_active":"2024-09-17T03:50:23.132753+0000","last_peered":"2024-09-17T03:50:23.132753+0000","last_clean":"2024-09-17T03:50:23.132753+0000","last_became_active":"2024-09-17T03:50:23.130434+0000","last_became_peered":"2024-09-17T03:50:23.130434+0000","last_unstale":"2024-09-17T03:50:23.132753+0000","last_undegraded":"2024-09-17T03:50:23.132753+0000","last_fullsized":"2024-09-17T03:50:23.132753+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.030426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145900+0000","last_change":"2024-09-17T03:50:24.141883+0000","last_active":"2024-09-17T03:50:24.145900+0000","last_peered":"2024-09-17T03:50:24.145900+0000","last_clean":"2024-09-17T03:50:24.145900+0000","last_became_active":"2024-09-17T03:50:24.141572+0000","last_became_peered":"2024-09-17T03:50:24.141572+0000","last_unstale":"2024-09-17T03:50:24.145900+0000","last_undegraded":"2024-09-17T03:50:24.145900+0000","last_fullsized":"2024-09-17T03:50:24.145900+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:47:36.875814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.138532+0000","last_change":"2024-09-17T03:50:23.133695+0000","last_active":"2024-09-17T03:50:23.138532+0000","last_peered":"2024-09-17T03:50:23.138532+0000","last_clean":"2024-09-17T03:50:23.138532+0000","last_became_active":"2024-09-17T03:50:23.133125+0000","last_became_peered":"2024-09-17T03:50:23.133125+0000","last_unstale":"2024-09-17T03:50:23.138532+0000","last_undegraded":"2024-09-17T03:50:23.138532+0000","last_fullsized":"2024-09-17T03:50:23.138532+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:18.945900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.145857+0000","last_change":"2024-09-17T03:50:24.141825+0000","last_active":"2024-09-17T03:50:24.145857+0000","last_peered":"2024-09-17T03:50:24.145857+0000","last_clean":"2024-09-17T03:50:24.145857+0000","last_became_active":"2024-09-17T03:50:24.141522+0000","last_became_peered":"2024-09-17T03:50:24.141522+0000","last_unstale":"2024-09-17T03:50:24.145857+0000","last_undegraded":"2024-09-17T03:50:24.145857+0000","last_fullsized":"2024-09-17T03:50:24.145857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:22.204438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131053+0000","last_change":"2024-09-17T03:50:23.141992+0000","last_active":"2024-09-17T03:50:24.131053+0000","last_peered":"2024-09-17T03:50:24.131053+0000","last_clean":"2024-09-17T03:50:24.131053+0000","last_became_active":"2024-09-17T03:50:23.141642+0000","last_became_peered":"2024-09-17T03:50:23.141642+0000","last_unstale":"2024-09-17T03:50:24.131053+0000","last_undegraded":"2024-09-17T03:50:24.131053+0000","last_fullsized":"2024-09-17T03:50:24.131053+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:12.799906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.144614+0000","last_change":"2024-09-17T03:50:24.142339+0000","last_active":"2024-09-17T03:50:24.144614+0000","last_peered":"2024-09-17T03:50:24.144614+0000","last_clean":"2024-09-17T03:50:24.144614+0000","last_became_active":"2024-09-17T03:50:24.142028+0000","last_became_peered":"2024-09-17T03:50:24.142028+0000","last_unstale":"2024-09-17T03:50:24.144614+0000","last_undegraded":"2024-09-17T03:50:24.144614+0000","last_fullsized":"2024-09-17T03:50:24.144614+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:13.044184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.118295+0000","last_change":"2024-09-17T03:50:23.129908+0000","last_active":"2024-09-17T03:50:24.118295+0000","last_peered":"2024-09-17T03:50:24.118295+0000","last_clean":"2024-09-17T03:50:24.118295+0000","last_became_active":"2024-09-17T03:50:23.129543+0000","last_became_peered":"2024-09-17T03:50:23.129543+0000","last_unstale":"2024-09-17T03:50:24.118295+0000","last_undegraded":"2024-09-17T03:50:24.118295+0000","last_fullsized":"2024-09-17T03:50:24.118295+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129144+0000","last_change":"2024-09-17T03:50:24.124588+0000","last_active":"2024-09-17T03:50:24.129144+0000","last_peered":"2024-09-17T03:50:24.129144+0000","last_clean":"2024-09-17T03:50:24.129144+0000","last_became_active":"2024-09-17T03:50:24.124178+0000","last_became_peered":"2024-09-17T03:50:24.124178+0000","last_unstale":"2024-09-17T03:50:24.129144+0000","last_undegraded":"2024-09-17T03:50:24.129144+0000","last_fullsized":"2024-09-17T03:50:24.129144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:20.307754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":15,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150943+0000","last_change":"2024-09-17T03:50:23.137622+0000","last_active":"2024-09-17T03:50:24.150943+0000","last_peered":"2024-09-17T03:50:24.150943+0000","last_clean":"2024-09-17T03:50:24.150943+0000","last_became_active":"2024-09-17T03:50:23.137258+0000","last_became_peered":"2024-09-17T03:50:23.137258+0000","last_unstale":"2024-09-17T03:50:24.150943+0000","last_undegraded":"2024-09-17T03:50:24.150943+0000","last_fullsized":"2024-09-17T03:50:24.150943+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:40.356807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.146377+0000","last_change":"2024-09-17T03:50:24.140577+0000","last_active":"2024-09-17T03:50:24.146377+0000","last_peered":"2024-09-17T03:50:24.146377+0000","last_clean":"2024-09-17T03:50:24.146377+0000","last_became_active":"2024-09-17T03:50:24.140330+0000","last_became_peered":"2024-09-17T03:50:24.140330+0000","last_unstale":"2024-09-17T03:50:24.146377+0000","last_undegraded":"2024-09-17T03:50:24.146377+0000","last_fullsized":"2024-09-17T03:50:24.146377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.131834+0000","last_change":"2024-09-17T03:50:23.130628+0000","last_active":"2024-09-17T03:50:23.131834+0000","last_peered":"2024-09-17T03:50:23.131834+0000","last_clean":"2024-09-17T03:50:23.131834+0000","last_became_active":"2024-09-17T03:50:23.130307+0000","last_became_peered":"2024-09-17T03:50:23.130307+0000","last_unstale":"2024-09-17T03:50:23.131834+0000","last_undegraded":"2024-09-17T03:50:23.131834+0000","last_fullsized":"2024-09-17T03:50:23.131834+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:51.417524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116716+0000","last_change":"2024-09-17T03:50:23.116716+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116716+0000","last_undegraded":"2024-09-17T03:50:23.116716+0000","last_fullsized":"2024-09-17T03:50:23.116716+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.128468+0000","last_change":"2024-09-17T03:50:23.136531+0000","last_active":"2024-09-17T03:50:24.128468+0000","last_peered":"2024-09-17T03:50:24.128468+0000","last_clean":"2024-09-17T03:50:24.128468+0000","last_became_active":"2024-09-17T03:50:23.136207+0000","last_became_peered":"2024-09-17T03:50:23.136207+0000","last_unstale":"2024-09-17T03:50:24.128468+0000","last_undegraded":"2024-09-17T03:50:24.128468+0000","last_fullsized":"2024-09-17T03:50:24.128468+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:47.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.122595+0000","last_change":"2024-09-17T03:50:23.122595+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.122595+0000","last_undegraded":"2024-09-17T03:50:23.122595+0000","last_fullsized":"2024-09-17T03:50:23.122595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.128999+0000","last_change":"2024-09-17T03:50:23.124066+0000","last_active":"2024-09-17T03:50:23.128999+0000","last_peered":"2024-09-17T03:50:23.128999+0000","last_clean":"2024-09-17T03:50:23.128999+0000","last_became_active":"2024-09-17T03:50:23.123776+0000","last_became_peered":"2024-09-17T03:50:23.123776+0000","last_unstale":"2024-09-17T03:50:23.128999+0000","last_undegraded":"2024-09-17T03:50:23.128999+0000","last_fullsized":"2024-09-17T03:50:23.128999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:28.460467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-09-17T03:50:23.116315+0000","last_change":"2024-09-17T03:50:23.116315+0000","last_active":"2024-09-17T03:50:23.104360+0000","last_peered":"2024-09-17T03:50:23.104360+0000","last_clean":"2024-09-17T03:50:23.104360+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:50:23.116315+0000","last_undegraded":"2024-09-17T03:50:23.116315+0000","last_fullsized":"2024-09-17T03:50:23.116315+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.130118+0000","last_change":"2024-09-17T03:50:23.136967+0000","last_active":"2024-09-17T03:50:24.130118+0000","last_peered":"2024-09-17T03:50:24.130118+0000","last_clean":"2024-09-17T03:50:24.130118+0000","last_became_active":"2024-09-17T03:50:23.136643+0000","last_became_peered":"2024-09-17T03:50:23.136643+0000","last_unstale":"2024-09-17T03:50:24.130118+0000","last_undegraded":"2024-09-17T03:50:24.130118+0000","last_fullsized":"2024-09-17T03:50:24.130118+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:58.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.158254+0000","last_change":"2024-09-17T03:50:24.150858+0000","last_active":"2024-09-17T03:50:24.158254+0000","last_peered":"2024-09-17T03:50:24.158254+0000","last_clean":"2024-09-17T03:50:24.158254+0000","last_became_active":"2024-09-17T03:50:24.150509+0000","last_became_peered":"2024-09-17T03:50:24.150509+0000","last_unstale":"2024-09-17T03:50:24.158254+0000","last_undegraded":"2024-09-17T03:50:24.158254+0000","last_fullsized":"2024-09-17T03:50:24.158254+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.131336+0000","last_change":"2024-09-17T03:50:23.141022+0000","last_active":"2024-09-17T03:50:24.131336+0000","last_peered":"2024-09-17T03:50:24.131336+0000","last_clean":"2024-09-17T03:50:24.131336+0000","last_became_active":"2024-09-17T03:50:23.140264+0000","last_became_peered":"2024-09-17T03:50:23.140264+0000","last_unstale":"2024-09-17T03:50:24.131336+0000","last_undegraded":"2024-09-17T03:50:24.131336+0000","last_fullsized":"2024-09-17T03:50:24.131336+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:55.135377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.155579+0000","last_change":"2024-09-17T03:50:24.149913+0000","last_active":"2024-09-17T03:50:24.155579+0000","last_peered":"2024-09-17T03:50:24.155579+0000","last_clean":"2024-09-17T03:50:24.155579+0000","last_became_active":"2024-09-17T03:50:24.149591+0000","last_became_peered":"2024-09-17T03:50:24.149591+0000","last_unstale":"2024-09-17T03:50:24.155579+0000","last_undegraded":"2024-09-17T03:50:24.155579+0000","last_fullsized":"2024-09-17T03:50:24.155579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:25.250775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.126284+0000","last_change":"2024-09-17T03:50:23.130536+0000","last_active":"2024-09-17T03:50:24.126284+0000","last_peered":"2024-09-17T03:50:24.126284+0000","last_clean":"2024-09-17T03:50:24.126284+0000","last_became_active":"2024-09-17T03:50:23.130176+0000","last_became_peered":"2024-09-17T03:50:23.130176+0000","last_unstale":"2024-09-17T03:50:24.126284+0000","last_undegraded":"2024-09-17T03:50:24.126284+0000","last_fullsized":"2024-09-17T03:50:24.126284+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:26.703578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.150296+0000","last_change":"2024-09-17T03:50:24.146715+0000","last_active":"2024-09-17T03:50:24.150296+0000","last_peered":"2024-09-17T03:50:24.150296+0000","last_clean":"2024-09-17T03:50:24.150296+0000","last_became_active":"2024-09-17T03:50:24.146329+0000","last_became_peered":"2024-09-17T03:50:24.146329+0000","last_unstale":"2024-09-17T03:50:24.150296+0000","last_undegraded":"2024-09-17T03:50:24.150296+0000","last_fullsized":"2024-09-17T03:50:24.150296+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:40.254708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-17T03:50:23.142795+0000","last_change":"2024-09-17T03:50:23.138899+0000","last_active":"2024-09-17T03:50:23.142795+0000","last_peered":"2024-09-17T03:50:23.142795+0000","last_clean":"2024-09-17T03:50:23.142795+0000","last_became_active":"2024-09-17T03:50:23.138546+0000","last_became_peered":"2024-09-17T03:50:23.138546+0000","last_unstale":"2024-09-17T03:50:23.142795+0000","last_undegraded":"2024-09-17T03:50:23.142795+0000","last_fullsized":"2024-09-17T03:50:23.142795+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:29.356203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":128,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-17T03:50:24.129940+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:50:24.129940+0000","last_peered":"2024-09-17T03:50:24.129940+0000","last_clean":"2024-09-17T03:50:24.129940+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:50:24.129940+0000","last_undegraded":"2024-09-17T03:50:24.129940+0000","last_fullsized":"2024-09-17T03:50:24.129940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31,"ondisk_log_size":31,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509454,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":933888,"data_stored":717659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509457,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509454,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":127291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542158,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":327680,"data_stored":121024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542159,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32088,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339752,"statfs":{"total":96636764160,"available":96603906048,"internally_reserved":0,"allocated":921600,"data_stored":711392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:50:29.508 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3f is not active+clean 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.124897+0000', 'last_change': '2024-09-17T03:50:23.124897+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.124897+0000', 'last_undegraded': '2024-09-17T03:50:23.124897+0000', 'last_fullsized': '2024-09-17T03:50:23.124897+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:PG 5.29 is not active+clean 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.116168+0000', 'last_change': '2024-09-17T03:50:23.116168+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.116168+0000', 'last_undegraded': '2024-09-17T03:50:23.116168+0000', 'last_fullsized': '2024-09-17T03:50:23.116168+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:PG 5.21 is not active+clean 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.123020+0000', 'last_change': '2024-09-17T03:50:23.123020+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.123020+0000', 'last_undegraded': '2024-09-17T03:50:23.123020+0000', 'last_fullsized': '2024-09-17T03:50:23.123020+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T03:50:29.509 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1e is not active+clean 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.116164+0000', 'last_change': '2024-09-17T03:50:23.116164+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.116164+0000', 'last_undegraded': '2024-09-17T03:50:23.116164+0000', 'last_fullsized': '2024-09-17T03:50:23.116164+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:PG 5.1f is not active+clean 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.124400+0000', 'last_change': '2024-09-17T03:50:23.124400+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.124400+0000', 'last_undegraded': '2024-09-17T03:50:23.124400+0000', 'last_fullsized': '2024-09-17T03:50:23.124400+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:PG 5.11 is not active+clean 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.122281+0000', 'last_change': '2024-09-17T03:50:23.122281+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.122281+0000', 'last_undegraded': '2024-09-17T03:50:23.122281+0000', 'last_fullsized': '2024-09-17T03:50:23.122281+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:PG 5.5 is not active+clean 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.116716+0000', 'last_change': '2024-09-17T03:50:23.116716+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.116716+0000', 'last_undegraded': '2024-09-17T03:50:23.116716+0000', 'last_fullsized': '2024-09-17T03:50:23.116716+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:PG 5.2 is not active+clean 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.122595+0000', 'last_change': '2024-09-17T03:50:23.122595+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.122595+0000', 'last_undegraded': '2024-09-17T03:50:23.122595+0000', 'last_fullsized': '2024-09-17T03:50:23.122595+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T03:50:29.510 INFO:tasks.ceph.ceph_manager.ceph:PG 5.3 is not active+clean 2024-09-17T03:50:29.511 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:50:23.116315+0000', 'last_change': '2024-09-17T03:50:23.116315+0000', 'last_active': '2024-09-17T03:50:23.104360+0000', 'last_peered': '2024-09-17T03:50:23.104360+0000', 'last_clean': '2024-09-17T03:50:23.104360+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:50:23.116315+0000', 'last_undegraded': '2024-09-17T03:50:23.116315+0000', 'last_fullsized': '2024-09-17T03:50:23.116315+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'last_clean_scrub_stamp': '2024-09-17T03:50:23.104360+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T03:50:29.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T03:50:29.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 -- 172.21.15.129:0/466359761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c06ee80 msgr2=0x7f643c09ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 --2- 172.21.15.129:0/466359761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c06ee80 0x7f643c09ac10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f643c0b3560 tx=0x7f642402a540 comp rx=0 tx=0).stop 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 -- 172.21.15.129:0/466359761 shutdown_connections 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 --2- 172.21.15.129:0/466359761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c06ee80 0x7f643c09ac10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 --2- 172.21.15.129:0/466359761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c0cf6c0 0x7f643c0b75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 -- 172.21.15.129:0/466359761 >> 172.21.15.129:0/466359761 conn(0x7f643c0ceba0 msgr2=0x7f643c0cefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:29.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 -- 172.21.15.129:0/466359761 shutdown_connections 2024-09-17T03:50:29.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.767+0000 7f6440a00640 1 -- 172.21.15.129:0/466359761 wait complete. 2024-09-17T03:50:29.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.769+0000 7f6440a00640 1 Processor -- start 2024-09-17T03:50:29.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.769+0000 7f6440a00640 1 -- start start 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f643c06ee80 0x7f643c0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c09b150 0x7f643c15c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f643c0d0ff0 con 0x7f643c0cf6c0 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f643c0d0e70 con 0x7f643c06ee80 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6440a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f643c168e10 con 0x7f643c09b150 2024-09-17T03:50:29.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f643be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f643be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41278/0 (socket says 172.21.15.129:41278) 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f643be00640 1 -- 172.21.15.129:0/1960975710 learned_addr learned my addr 172.21.15.129:0/1960975710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f643b400640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f643c06ee80 0x7f643c0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.771+0000 7f6432a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c09b150 0x7f643c15c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 -- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c09b150 msgr2=0x7f643c15c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c09b150 0x7f643c15c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 -- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f643c06ee80 msgr2=0x7f643c0b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f643c06ee80 0x7f643c0b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 -- 172.21.15.129:0/1960975710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c15cd60 con 0x7f643c0cf6c0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f643be00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f643400c730 tx=0x7f643400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434003b70 con 0x7f643c0cf6c0 2024-09-17T03:50:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6434003d10 con 0x7f643c0cf6c0 2024-09-17T03:50:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f643c1f8e30 con 0x7f643c0cf6c0 2024-09-17T03:50:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434005330 con 0x7f643c0cf6c0 2024-09-17T03:50:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.772+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643c1f93f0 con 0x7f643c0cf6c0 2024-09-17T03:50:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.773+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f643c15cf30 con 0x7f643c0cf6c0 2024-09-17T03:50:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.775+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f64340054d0 con 0x7f643c0cf6c0 2024-09-17T03:50:29.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.776+0000 7f6433e00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 0x7f641807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:29.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.776+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f643409aa40 con 0x7f643c0cf6c0 2024-09-17T03:50:29.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.776+0000 7f643b400640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 0x7f641807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:29.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.777+0000 7f643b400640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 0x7f641807b0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f643c073dc0 tx=0x7f642c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:29.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:29.780+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6434062790 con 0x7f643c0cf6c0 2024-09-17T03:50:30.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.032+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f643c0ce4e0 con 0x7f643c0cf6c0 2024-09-17T03:50:30.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.032+0000 7f6433e00640 1 -- 172.21.15.129:0/1960975710 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1688 (secure 0 0 0) 0x7f64340671c0 con 0x7f643c0cf6c0 2024-09-17T03:50:30.032 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:30.032 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":81,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":88},{"state_name":"creating+peering","count":9}],"num_pgs":97,"num_pools":3,"num_objects":20,"data_bytes":592196,"bytes_used":229560320,"bytes_avail":772864552960,"bytes_total":773094113280,"inactive_pgs_ratio":0.092783503234386444,"write_bytes_sec":2218,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":4845}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-17T03:50:26.306508+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 msgr2=0x7f641807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 0x7f641807b0a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f643c073dc0 tx=0x7f642c005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 msgr2=0x7f643c071e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f643400c730 tx=0x7f643400cb30 comp rx=0 tx=0).stop 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 shutdown_connections 2024-09-17T03:50:30.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6418078b90 0x7f641807b0a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f643c0cf6c0 0x7f643c071e20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f643c09b150 0x7f643c15c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 --2- 172.21.15.129:0/1960975710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f643c06ee80 0x7f643c0b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 >> 172.21.15.129:0/1960975710 conn(0x7f643c0ceba0 msgr2=0x7f643c168570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 shutdown_connections 2024-09-17T03:50:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.035+0000 7f6440a00640 1 -- 172.21.15.129:0/1960975710 wait complete. 2024-09-17T03:50:30.057 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T03:50:30.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:50:30.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 -- 172.21.15.129:0/1207907254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 msgr2=0x7f7e54004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 --2- 172.21.15.129:0/1207907254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e54004800 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7e70072320 tx=0x7f7e54005830 comp rx=0 tx=0).stop 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e6f400640 1 -- 172.21.15.129:0/1207907254 reap_dead start 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 -- 172.21.15.129:0/1207907254 shutdown_connections 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 -- 172.21.15.129:0/1207907254 >> 172.21.15.129:0/1207907254 conn(0x7f7e700b1cb0 msgr2=0x7f7e700b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 -- 172.21.15.129:0/1207907254 shutdown_connections 2024-09-17T03:50:30.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.278+0000 7f7e75600640 1 -- 172.21.15.129:0/1207907254 wait complete. 2024-09-17T03:50:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.280+0000 7f7e75600640 1 Processor -- start 2024-09-17T03:50:30.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.281+0000 7f7e75600640 1 -- start start 2024-09-17T03:50:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e7009be80 0x7f7e7009d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 0x7f7e701f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e700b37a0 con 0x7f7e540041c0 2024-09-17T03:50:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7015cc60 con 0x7f7e7009dbf0 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e75600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7e700b3920 con 0x7f7e7009be80 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e6ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e6ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41302/0 (socket says 172.21.15.129:41302) 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.282+0000 7f7e6ea00640 1 -- 172.21.15.129:0/2854386302 learned_addr learned my addr 172.21.15.129:0/2854386302 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 -- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e7009be80 msgr2=0x7f7e7009d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e7009be80 0x7f7e7009d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 -- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 msgr2=0x7f7e701f8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6f400640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 0x7f7e701f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:30.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 0x7f7e701f8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 -- 172.21.15.129:0/2854386302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7009e000 con 0x7f7e540041c0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6f400640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 0x7f7e701f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e6ea00640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e60011d60 tx=0x7f7e6000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60003d80 con 0x7f7e540041c0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e60011e50 con 0x7f7e540041c0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e700cbc30 con 0x7f7e540041c0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e60004d40 con 0x7f7e540041c0 2024-09-17T03:50:30.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.283+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e700cc1f0 con 0x7f7e540041c0 2024-09-17T03:50:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.284+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e700b3eb0 con 0x7f7e540041c0 2024-09-17T03:50:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.285+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7e60004050 con 0x7f7e540041c0 2024-09-17T03:50:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.286+0000 7f7e67400640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 0x7f7e4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.286+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7e6009bdc0 con 0x7f7e540041c0 2024-09-17T03:50:30.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.287+0000 7f7e66000640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 0x7f7e4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:30.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.288+0000 7f7e66000640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 0x7f7e4c07b0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7e58032000 tx=0x7f7e580012f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.291+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7e7009e000 con 0x7f7e540041c0 2024-09-17T03:50:30.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.475+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e700c7ec0 con 0x7f7e4c078b90 2024-09-17T03:50:30.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.480+0000 7f7e67400640 1 -- 172.21.15.129:0/2854386302 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251928 (secure 0 0 0) 0x7f7e5807d830 con 0x7f7e4c078b90 2024-09-17T03:50:30.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:30.483 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 msgr2=0x7f7e4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 0x7f7e4c07b0a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7e58032000 tx=0x7f7e580012f0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 msgr2=0x7f7e701f5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e60011d60 tx=0x7f7e6000f790 comp rx=0 tx=0).stop 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 shutdown_connections 2024-09-17T03:50:30.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e4c078b90 0x7f7e4c07b0a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e7009dbf0 0x7f7e701f8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e7009be80 0x7f7e7009d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 --2- 172.21.15.129:0/2854386302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e540041c0 0x7f7e701f5f40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.483+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 >> 172.21.15.129:0/2854386302 conn(0x7f7e700b1cb0 msgr2=0x7f7e700afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.484+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 shutdown_connections 2024-09-17T03:50:30.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:30.484+0000 7f7e75600640 1 -- 172.21.15.129:0/2854386302 wait complete. 2024-09-17T03:50:30.507 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-17T03:50:30.307207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224496,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750224,"statfs":{"total":773094113280,"available":772864229376,"internally_reserved":0,"allocated":4169728,"data_stored":2308484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.192970"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590433+0000","last_change":"2024-09-17T03:50:24.138380+0000","last_active":"2024-09-17T03:50:25.590433+0000","last_peered":"2024-09-17T03:50:25.590433+0000","last_clean":"2024-09-17T03:50:25.590433+0000","last_became_active":"2024-09-17T03:50:24.137737+0000","last_became_peered":"2024-09-17T03:50:24.137737+0000","last_unstale":"2024-09-17T03:50:25.590433+0000","last_undegraded":"2024-09-17T03:50:25.590433+0000","last_fullsized":"2024-09-17T03:50:25.590433+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:13.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133801+0000","last_change":"2024-09-17T03:50:24.146222+0000","last_active":"2024-09-17T03:50:25.133801+0000","last_peered":"2024-09-17T03:50:25.133801+0000","last_clean":"2024-09-17T03:50:25.133801+0000","last_became_active":"2024-09-17T03:50:24.145925+0000","last_became_peered":"2024-09-17T03:50:24.145925+0000","last_unstale":"2024-09-17T03:50:25.133801+0000","last_undegraded":"2024-09-17T03:50:25.133801+0000","last_fullsized":"2024-09-17T03:50:25.133801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:09.445391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590859+0000","last_change":"2024-09-17T03:50:24.146390+0000","last_active":"2024-09-17T03:50:25.590859+0000","last_peered":"2024-09-17T03:50:25.590859+0000","last_clean":"2024-09-17T03:50:25.590859+0000","last_became_active":"2024-09-17T03:50:24.146067+0000","last_became_peered":"2024-09-17T03:50:24.146067+0000","last_unstale":"2024-09-17T03:50:25.590859+0000","last_undegraded":"2024-09-17T03:50:25.590859+0000","last_fullsized":"2024-09-17T03:50:25.590859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590418+0000","last_change":"2024-09-17T03:50:24.156017+0000","last_active":"2024-09-17T03:50:25.590418+0000","last_peered":"2024-09-17T03:50:25.590418+0000","last_clean":"2024-09-17T03:50:25.590418+0000","last_became_active":"2024-09-17T03:50:24.155677+0000","last_became_peered":"2024-09-17T03:50:24.155677+0000","last_unstale":"2024-09-17T03:50:25.590418+0000","last_undegraded":"2024-09-17T03:50:25.590418+0000","last_fullsized":"2024-09-17T03:50:25.590418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591334+0000","last_change":"2024-09-17T03:50:24.138985+0000","last_active":"2024-09-17T03:50:25.591334+0000","last_peered":"2024-09-17T03:50:25.591334+0000","last_clean":"2024-09-17T03:50:25.591334+0000","last_became_active":"2024-09-17T03:50:24.138569+0000","last_became_peered":"2024-09-17T03:50:24.138569+0000","last_unstale":"2024-09-17T03:50:25.591334+0000","last_undegraded":"2024-09-17T03:50:25.591334+0000","last_fullsized":"2024-09-17T03:50:25.591334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:01.229274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133360+0000","last_change":"2024-09-17T03:50:24.147892+0000","last_active":"2024-09-17T03:50:25.133360+0000","last_peered":"2024-09-17T03:50:25.133360+0000","last_clean":"2024-09-17T03:50:25.133360+0000","last_became_active":"2024-09-17T03:50:24.147616+0000","last_became_peered":"2024-09-17T03:50:24.147616+0000","last_unstale":"2024-09-17T03:50:25.133360+0000","last_undegraded":"2024-09-17T03:50:25.133360+0000","last_fullsized":"2024-09-17T03:50:25.133360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136144+0000","last_change":"2024-09-17T03:50:24.148414+0000","last_active":"2024-09-17T03:50:25.136144+0000","last_peered":"2024-09-17T03:50:25.136144+0000","last_clean":"2024-09-17T03:50:25.136144+0000","last_became_active":"2024-09-17T03:50:24.148020+0000","last_became_peered":"2024-09-17T03:50:24.148020+0000","last_unstale":"2024-09-17T03:50:25.136144+0000","last_undegraded":"2024-09-17T03:50:25.136144+0000","last_fullsized":"2024-09-17T03:50:25.136144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134558+0000","last_change":"2024-09-17T03:50:24.147045+0000","last_active":"2024-09-17T03:50:25.134558+0000","last_peered":"2024-09-17T03:50:25.134558+0000","last_clean":"2024-09-17T03:50:25.134558+0000","last_became_active":"2024-09-17T03:50:24.146692+0000","last_became_peered":"2024-09-17T03:50:24.146692+0000","last_unstale":"2024-09-17T03:50:25.134558+0000","last_undegraded":"2024-09-17T03:50:25.134558+0000","last_fullsized":"2024-09-17T03:50:25.134558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136781+0000","last_change":"2024-09-17T03:50:24.149261+0000","last_active":"2024-09-17T03:50:25.136781+0000","last_peered":"2024-09-17T03:50:25.136781+0000","last_clean":"2024-09-17T03:50:25.136781+0000","last_became_active":"2024-09-17T03:50:24.148898+0000","last_became_peered":"2024-09-17T03:50:24.148898+0000","last_unstale":"2024-09-17T03:50:25.136781+0000","last_undegraded":"2024-09-17T03:50:25.136781+0000","last_fullsized":"2024-09-17T03:50:25.136781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591213+0000","last_change":"2024-09-17T03:50:24.152445+0000","last_active":"2024-09-17T03:50:25.591213+0000","last_peered":"2024-09-17T03:50:25.591213+0000","last_clean":"2024-09-17T03:50:25.591213+0000","last_became_active":"2024-09-17T03:50:24.152179+0000","last_became_peered":"2024-09-17T03:50:24.152179+0000","last_unstale":"2024-09-17T03:50:25.591213+0000","last_undegraded":"2024-09-17T03:50:25.591213+0000","last_fullsized":"2024-09-17T03:50:25.591213+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137221+0000","last_change":"2024-09-17T03:50:24.155248+0000","last_active":"2024-09-17T03:50:25.137221+0000","last_peered":"2024-09-17T03:50:25.137221+0000","last_clean":"2024-09-17T03:50:25.137221+0000","last_became_active":"2024-09-17T03:50:24.154779+0000","last_became_peered":"2024-09-17T03:50:24.154779+0000","last_unstale":"2024-09-17T03:50:25.137221+0000","last_undegraded":"2024-09-17T03:50:25.137221+0000","last_fullsized":"2024-09-17T03:50:25.137221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:56.127224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590002+0000","last_change":"2024-09-17T03:50:24.138736+0000","last_active":"2024-09-17T03:50:25.590002+0000","last_peered":"2024-09-17T03:50:25.590002+0000","last_clean":"2024-09-17T03:50:25.590002+0000","last_became_active":"2024-09-17T03:50:24.138424+0000","last_became_peered":"2024-09-17T03:50:24.138424+0000","last_unstale":"2024-09-17T03:50:25.590002+0000","last_undegraded":"2024-09-17T03:50:25.590002+0000","last_fullsized":"2024-09-17T03:50:25.590002+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591349+0000","last_change":"2024-09-17T03:50:24.142434+0000","last_active":"2024-09-17T03:50:25.591349+0000","last_peered":"2024-09-17T03:50:25.591349+0000","last_clean":"2024-09-17T03:50:25.591349+0000","last_became_active":"2024-09-17T03:50:24.142083+0000","last_became_peered":"2024-09-17T03:50:24.142083+0000","last_unstale":"2024-09-17T03:50:25.591349+0000","last_undegraded":"2024-09-17T03:50:25.591349+0000","last_fullsized":"2024-09-17T03:50:25.591349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:18.672400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133627+0000","last_change":"2024-09-17T03:50:24.154707+0000","last_active":"2024-09-17T03:50:25.133627+0000","last_peered":"2024-09-17T03:50:25.133627+0000","last_clean":"2024-09-17T03:50:25.133627+0000","last_became_active":"2024-09-17T03:50:24.154328+0000","last_became_peered":"2024-09-17T03:50:24.154328+0000","last_unstale":"2024-09-17T03:50:25.133627+0000","last_undegraded":"2024-09-17T03:50:25.133627+0000","last_fullsized":"2024-09-17T03:50:25.133627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:02:18.751353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590043+0000","last_change":"2024-09-17T03:50:24.125931+0000","last_active":"2024-09-17T03:50:25.590043+0000","last_peered":"2024-09-17T03:50:25.590043+0000","last_clean":"2024-09-17T03:50:25.590043+0000","last_became_active":"2024-09-17T03:50:24.125300+0000","last_became_peered":"2024-09-17T03:50:24.125300+0000","last_unstale":"2024-09-17T03:50:25.590043+0000","last_undegraded":"2024-09-17T03:50:25.590043+0000","last_fullsized":"2024-09-17T03:50:25.590043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135997+0000","last_change":"2024-09-17T03:50:24.144536+0000","last_active":"2024-09-17T03:50:25.135997+0000","last_peered":"2024-09-17T03:50:25.135997+0000","last_clean":"2024-09-17T03:50:25.135997+0000","last_became_active":"2024-09-17T03:50:24.143960+0000","last_became_peered":"2024-09-17T03:50:24.143960+0000","last_unstale":"2024-09-17T03:50:25.135997+0000","last_undegraded":"2024-09-17T03:50:25.135997+0000","last_fullsized":"2024-09-17T03:50:25.135997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137463+0000","last_change":"2024-09-17T03:50:24.147697+0000","last_active":"2024-09-17T03:50:25.137463+0000","last_peered":"2024-09-17T03:50:25.137463+0000","last_clean":"2024-09-17T03:50:25.137463+0000","last_became_active":"2024-09-17T03:50:24.146930+0000","last_became_peered":"2024-09-17T03:50:24.146930+0000","last_unstale":"2024-09-17T03:50:25.137463+0000","last_undegraded":"2024-09-17T03:50:25.137463+0000","last_fullsized":"2024-09-17T03:50:25.137463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:24.583890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133874+0000","last_change":"2024-09-17T03:50:24.159788+0000","last_active":"2024-09-17T03:50:25.133874+0000","last_peered":"2024-09-17T03:50:25.133874+0000","last_clean":"2024-09-17T03:50:25.133874+0000","last_became_active":"2024-09-17T03:50:24.159505+0000","last_became_peered":"2024-09-17T03:50:24.159505+0000","last_unstale":"2024-09-17T03:50:25.133874+0000","last_undegraded":"2024-09-17T03:50:25.133874+0000","last_fullsized":"2024-09-17T03:50:25.133874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:34.249002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133337+0000","last_change":"2024-09-17T03:50:24.154330+0000","last_active":"2024-09-17T03:50:25.133337+0000","last_peered":"2024-09-17T03:50:25.133337+0000","last_clean":"2024-09-17T03:50:25.133337+0000","last_became_active":"2024-09-17T03:50:24.153860+0000","last_became_peered":"2024-09-17T03:50:24.153860+0000","last_unstale":"2024-09-17T03:50:25.133337+0000","last_undegraded":"2024-09-17T03:50:25.133337+0000","last_fullsized":"2024-09-17T03:50:25.133337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590577+0000","last_change":"2024-09-17T03:50:24.128012+0000","last_active":"2024-09-17T03:50:25.590577+0000","last_peered":"2024-09-17T03:50:25.590577+0000","last_clean":"2024-09-17T03:50:25.590577+0000","last_became_active":"2024-09-17T03:50:24.127633+0000","last_became_peered":"2024-09-17T03:50:24.127633+0000","last_unstale":"2024-09-17T03:50:25.590577+0000","last_undegraded":"2024-09-17T03:50:25.590577+0000","last_fullsized":"2024-09-17T03:50:25.590577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:27.183403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134522+0000","last_change":"2024-09-17T03:50:24.149865+0000","last_active":"2024-09-17T03:50:25.134522+0000","last_peered":"2024-09-17T03:50:25.134522+0000","last_clean":"2024-09-17T03:50:25.134522+0000","last_became_active":"2024-09-17T03:50:24.149569+0000","last_became_peered":"2024-09-17T03:50:24.149569+0000","last_unstale":"2024-09-17T03:50:25.134522+0000","last_undegraded":"2024-09-17T03:50:25.134522+0000","last_fullsized":"2024-09-17T03:50:25.134522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134957+0000","last_change":"2024-09-17T03:50:24.151774+0000","last_active":"2024-09-17T03:50:25.134957+0000","last_peered":"2024-09-17T03:50:25.134957+0000","last_clean":"2024-09-17T03:50:25.134957+0000","last_became_active":"2024-09-17T03:50:24.151379+0000","last_became_peered":"2024-09-17T03:50:24.151379+0000","last_unstale":"2024-09-17T03:50:25.134957+0000","last_undegraded":"2024-09-17T03:50:25.134957+0000","last_fullsized":"2024-09-17T03:50:25.134957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589980+0000","last_change":"2024-09-17T03:50:24.136829+0000","last_active":"2024-09-17T03:50:25.589980+0000","last_peered":"2024-09-17T03:50:25.589980+0000","last_clean":"2024-09-17T03:50:25.589980+0000","last_became_active":"2024-09-17T03:50:24.136544+0000","last_became_peered":"2024-09-17T03:50:24.136544+0000","last_unstale":"2024-09-17T03:50:25.589980+0000","last_undegraded":"2024-09-17T03:50:25.589980+0000","last_fullsized":"2024-09-17T03:50:25.589980+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:44.698139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589782+0000","last_change":"2024-09-17T03:50:24.132642+0000","last_active":"2024-09-17T03:50:25.589782+0000","last_peered":"2024-09-17T03:50:25.589782+0000","last_clean":"2024-09-17T03:50:25.589782+0000","last_became_active":"2024-09-17T03:50:24.131051+0000","last_became_peered":"2024-09-17T03:50:24.131051+0000","last_unstale":"2024-09-17T03:50:25.589782+0000","last_undegraded":"2024-09-17T03:50:25.589782+0000","last_fullsized":"2024-09-17T03:50:25.589782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:34.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590575+0000","last_change":"2024-09-17T03:50:24.145119+0000","last_active":"2024-09-17T03:50:25.590575+0000","last_peered":"2024-09-17T03:50:25.590575+0000","last_clean":"2024-09-17T03:50:25.590575+0000","last_became_active":"2024-09-17T03:50:24.144619+0000","last_became_peered":"2024-09-17T03:50:24.144619+0000","last_unstale":"2024-09-17T03:50:25.590575+0000","last_undegraded":"2024-09-17T03:50:25.590575+0000","last_fullsized":"2024-09-17T03:50:25.590575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134040+0000","last_change":"2024-09-17T03:50:24.136492+0000","last_active":"2024-09-17T03:50:25.134040+0000","last_peered":"2024-09-17T03:50:25.134040+0000","last_clean":"2024-09-17T03:50:25.134040+0000","last_became_active":"2024-09-17T03:50:24.136206+0000","last_became_peered":"2024-09-17T03:50:24.136206+0000","last_unstale":"2024-09-17T03:50:25.134040+0000","last_undegraded":"2024-09-17T03:50:25.134040+0000","last_fullsized":"2024-09-17T03:50:25.134040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:25.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590744+0000","last_change":"2024-09-17T03:50:24.155949+0000","last_active":"2024-09-17T03:50:25.590744+0000","last_peered":"2024-09-17T03:50:25.590744+0000","last_clean":"2024-09-17T03:50:25.590744+0000","last_became_active":"2024-09-17T03:50:24.155692+0000","last_became_peered":"2024-09-17T03:50:24.155692+0000","last_unstale":"2024-09-17T03:50:25.590744+0000","last_undegraded":"2024-09-17T03:50:25.590744+0000","last_fullsized":"2024-09-17T03:50:25.590744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590305+0000","last_change":"2024-09-17T03:50:24.136298+0000","last_active":"2024-09-17T03:50:25.590305+0000","last_peered":"2024-09-17T03:50:25.590305+0000","last_clean":"2024-09-17T03:50:25.590305+0000","last_became_active":"2024-09-17T03:50:24.135828+0000","last_became_peered":"2024-09-17T03:50:24.135828+0000","last_unstale":"2024-09-17T03:50:25.590305+0000","last_undegraded":"2024-09-17T03:50:25.590305+0000","last_fullsized":"2024-09-17T03:50:25.590305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135035+0000","last_change":"2024-09-17T03:50:24.152059+0000","last_active":"2024-09-17T03:50:25.135035+0000","last_peered":"2024-09-17T03:50:25.135035+0000","last_clean":"2024-09-17T03:50:25.135035+0000","last_became_active":"2024-09-17T03:50:24.151751+0000","last_became_peered":"2024-09-17T03:50:24.151751+0000","last_unstale":"2024-09-17T03:50:25.135035+0000","last_undegraded":"2024-09-17T03:50:25.135035+0000","last_fullsized":"2024-09-17T03:50:25.135035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136461+0000","last_change":"2024-09-17T03:50:24.147844+0000","last_active":"2024-09-17T03:50:25.136461+0000","last_peered":"2024-09-17T03:50:25.136461+0000","last_clean":"2024-09-17T03:50:25.136461+0000","last_became_active":"2024-09-17T03:50:24.144862+0000","last_became_peered":"2024-09-17T03:50:24.144862+0000","last_unstale":"2024-09-17T03:50:25.136461+0000","last_undegraded":"2024-09-17T03:50:25.136461+0000","last_fullsized":"2024-09-17T03:50:25.136461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590186+0000","last_change":"2024-09-17T03:50:24.137755+0000","last_active":"2024-09-17T03:50:25.590186+0000","last_peered":"2024-09-17T03:50:25.590186+0000","last_clean":"2024-09-17T03:50:25.590186+0000","last_became_active":"2024-09-17T03:50:24.132876+0000","last_became_peered":"2024-09-17T03:50:24.132876+0000","last_unstale":"2024-09-17T03:50:25.590186+0000","last_undegraded":"2024-09-17T03:50:25.590186+0000","last_fullsized":"2024-09-17T03:50:25.590186+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:34.573301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134419+0000","last_change":"2024-09-17T03:50:24.153513+0000","last_active":"2024-09-17T03:50:25.134419+0000","last_peered":"2024-09-17T03:50:25.134419+0000","last_clean":"2024-09-17T03:50:25.134419+0000","last_became_active":"2024-09-17T03:50:24.153118+0000","last_became_peered":"2024-09-17T03:50:24.153118+0000","last_unstale":"2024-09-17T03:50:25.134419+0000","last_undegraded":"2024-09-17T03:50:25.134419+0000","last_fullsized":"2024-09-17T03:50:25.134419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590745+0000","last_change":"2024-09-17T03:50:24.139695+0000","last_active":"2024-09-17T03:50:25.590745+0000","last_peered":"2024-09-17T03:50:25.590745+0000","last_clean":"2024-09-17T03:50:25.590745+0000","last_became_active":"2024-09-17T03:50:24.136102+0000","last_became_peered":"2024-09-17T03:50:24.136102+0000","last_unstale":"2024-09-17T03:50:25.590745+0000","last_undegraded":"2024-09-17T03:50:25.590745+0000","last_fullsized":"2024-09-17T03:50:25.590745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:42:53.379705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590623+0000","last_change":"2024-09-17T03:50:23.138230+0000","last_active":"2024-09-17T03:50:25.590623+0000","last_peered":"2024-09-17T03:50:25.590623+0000","last_clean":"2024-09-17T03:50:25.590623+0000","last_became_active":"2024-09-17T03:50:23.137347+0000","last_became_peered":"2024-09-17T03:50:23.137347+0000","last_unstale":"2024-09-17T03:50:25.590623+0000","last_undegraded":"2024-09-17T03:50:25.590623+0000","last_fullsized":"2024-09-17T03:50:25.590623+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:13.017254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590598+0000","last_change":"2024-09-17T03:50:24.138595+0000","last_active":"2024-09-17T03:50:25.590598+0000","last_peered":"2024-09-17T03:50:25.590598+0000","last_clean":"2024-09-17T03:50:25.590598+0000","last_became_active":"2024-09-17T03:50:24.138208+0000","last_became_peered":"2024-09-17T03:50:24.138208+0000","last_unstale":"2024-09-17T03:50:25.590598+0000","last_undegraded":"2024-09-17T03:50:25.590598+0000","last_fullsized":"2024-09-17T03:50:25.590598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:28.964951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590824+0000","last_change":"2024-09-17T03:50:23.128500+0000","last_active":"2024-09-17T03:50:25.590824+0000","last_peered":"2024-09-17T03:50:25.590824+0000","last_clean":"2024-09-17T03:50:25.590824+0000","last_became_active":"2024-09-17T03:50:23.128184+0000","last_became_peered":"2024-09-17T03:50:23.128184+0000","last_unstale":"2024-09-17T03:50:25.590824+0000","last_undegraded":"2024-09-17T03:50:25.590824+0000","last_fullsized":"2024-09-17T03:50:25.590824+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:57.033701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135073+0000","last_change":"2024-09-17T03:50:24.152519+0000","last_active":"2024-09-17T03:50:25.135073+0000","last_peered":"2024-09-17T03:50:25.135073+0000","last_clean":"2024-09-17T03:50:25.135073+0000","last_became_active":"2024-09-17T03:50:24.150781+0000","last_became_peered":"2024-09-17T03:50:24.150781+0000","last_unstale":"2024-09-17T03:50:25.135073+0000","last_undegraded":"2024-09-17T03:50:25.135073+0000","last_fullsized":"2024-09-17T03:50:25.135073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133489+0000","last_change":"2024-09-17T03:50:23.132879+0000","last_active":"2024-09-17T03:50:25.133489+0000","last_peered":"2024-09-17T03:50:25.133489+0000","last_clean":"2024-09-17T03:50:25.133489+0000","last_became_active":"2024-09-17T03:50:23.131983+0000","last_became_peered":"2024-09-17T03:50:23.131983+0000","last_unstale":"2024-09-17T03:50:25.133489+0000","last_undegraded":"2024-09-17T03:50:25.133489+0000","last_fullsized":"2024-09-17T03:50:25.133489+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:39.860630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590970+0000","last_change":"2024-09-17T03:50:24.156534+0000","last_active":"2024-09-17T03:50:25.590970+0000","last_peered":"2024-09-17T03:50:25.590970+0000","last_clean":"2024-09-17T03:50:25.590970+0000","last_became_active":"2024-09-17T03:50:24.156249+0000","last_became_peered":"2024-09-17T03:50:24.156249+0000","last_unstale":"2024-09-17T03:50:25.590970+0000","last_undegraded":"2024-09-17T03:50:25.590970+0000","last_fullsized":"2024-09-17T03:50:25.590970+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590791+0000","last_change":"2024-09-17T03:50:23.134919+0000","last_active":"2024-09-17T03:50:25.590791+0000","last_peered":"2024-09-17T03:50:25.590791+0000","last_clean":"2024-09-17T03:50:25.590791+0000","last_became_active":"2024-09-17T03:50:23.134136+0000","last_became_peered":"2024-09-17T03:50:23.134136+0000","last_unstale":"2024-09-17T03:50:25.590791+0000","last_undegraded":"2024-09-17T03:50:25.590791+0000","last_fullsized":"2024-09-17T03:50:25.590791+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:06.997836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136713+0000","last_change":"2024-09-17T03:50:24.153900+0000","last_active":"2024-09-17T03:50:25.136713+0000","last_peered":"2024-09-17T03:50:25.136713+0000","last_clean":"2024-09-17T03:50:25.136713+0000","last_became_active":"2024-09-17T03:50:24.153608+0000","last_became_peered":"2024-09-17T03:50:24.153608+0000","last_unstale":"2024-09-17T03:50:25.136713+0000","last_undegraded":"2024-09-17T03:50:25.136713+0000","last_fullsized":"2024-09-17T03:50:25.136713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135221+0000","last_change":"2024-09-17T03:50:23.142748+0000","last_active":"2024-09-17T03:50:25.135221+0000","last_peered":"2024-09-17T03:50:25.135221+0000","last_clean":"2024-09-17T03:50:25.135221+0000","last_became_active":"2024-09-17T03:50:23.142402+0000","last_became_peered":"2024-09-17T03:50:23.142402+0000","last_unstale":"2024-09-17T03:50:25.135221+0000","last_undegraded":"2024-09-17T03:50:25.135221+0000","last_fullsized":"2024-09-17T03:50:25.135221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:26.214882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590470+0000","last_change":"2024-09-17T03:50:24.143306+0000","last_active":"2024-09-17T03:50:25.590470+0000","last_peered":"2024-09-17T03:50:25.590470+0000","last_clean":"2024-09-17T03:50:25.590470+0000","last_became_active":"2024-09-17T03:50:24.143003+0000","last_became_peered":"2024-09-17T03:50:24.143003+0000","last_unstale":"2024-09-17T03:50:25.590470+0000","last_undegraded":"2024-09-17T03:50:25.590470+0000","last_fullsized":"2024-09-17T03:50:25.590470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135059+0000","last_change":"2024-09-17T03:50:23.144925+0000","last_active":"2024-09-17T03:50:25.135059+0000","last_peered":"2024-09-17T03:50:25.135059+0000","last_clean":"2024-09-17T03:50:25.135059+0000","last_became_active":"2024-09-17T03:50:23.144566+0000","last_became_peered":"2024-09-17T03:50:23.144566+0000","last_unstale":"2024-09-17T03:50:25.135059+0000","last_undegraded":"2024-09-17T03:50:25.135059+0000","last_fullsized":"2024-09-17T03:50:25.135059+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:11.013666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134524+0000","last_change":"2024-09-17T03:50:24.151450+0000","last_active":"2024-09-17T03:50:25.134524+0000","last_peered":"2024-09-17T03:50:25.134524+0000","last_clean":"2024-09-17T03:50:25.134524+0000","last_became_active":"2024-09-17T03:50:24.151115+0000","last_became_peered":"2024-09-17T03:50:24.151115+0000","last_unstale":"2024-09-17T03:50:25.134524+0000","last_undegraded":"2024-09-17T03:50:25.134524+0000","last_fullsized":"2024-09-17T03:50:25.134524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133040+0000","last_change":"2024-09-17T03:50:23.133927+0000","last_active":"2024-09-17T03:50:25.133040+0000","last_peered":"2024-09-17T03:50:25.133040+0000","last_clean":"2024-09-17T03:50:25.133040+0000","last_became_active":"2024-09-17T03:50:23.133670+0000","last_became_peered":"2024-09-17T03:50:23.133670+0000","last_unstale":"2024-09-17T03:50:25.133040+0000","last_undegraded":"2024-09-17T03:50:25.133040+0000","last_fullsized":"2024-09-17T03:50:25.133040+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590144+0000","last_change":"2024-09-17T03:50:24.151434+0000","last_active":"2024-09-17T03:50:25.590144+0000","last_peered":"2024-09-17T03:50:25.590144+0000","last_clean":"2024-09-17T03:50:25.590144+0000","last_became_active":"2024-09-17T03:50:24.149266+0000","last_became_peered":"2024-09-17T03:50:24.149266+0000","last_unstale":"2024-09-17T03:50:25.590144+0000","last_undegraded":"2024-09-17T03:50:25.590144+0000","last_fullsized":"2024-09-17T03:50:25.590144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134650+0000","last_change":"2024-09-17T03:50:23.143447+0000","last_active":"2024-09-17T03:50:25.134650+0000","last_peered":"2024-09-17T03:50:25.134650+0000","last_clean":"2024-09-17T03:50:25.134650+0000","last_became_active":"2024-09-17T03:50:23.143053+0000","last_became_peered":"2024-09-17T03:50:23.143053+0000","last_unstale":"2024-09-17T03:50:25.134650+0000","last_undegraded":"2024-09-17T03:50:25.134650+0000","last_fullsized":"2024-09-17T03:50:25.134650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:45.122730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590536+0000","last_change":"2024-09-17T03:50:24.138823+0000","last_active":"2024-09-17T03:50:25.590536+0000","last_peered":"2024-09-17T03:50:25.590536+0000","last_clean":"2024-09-17T03:50:25.590536+0000","last_became_active":"2024-09-17T03:50:24.138293+0000","last_became_peered":"2024-09-17T03:50:24.138293+0000","last_unstale":"2024-09-17T03:50:25.590536+0000","last_undegraded":"2024-09-17T03:50:25.590536+0000","last_fullsized":"2024-09-17T03:50:25.590536+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133970+0000","last_change":"2024-09-17T03:50:23.139166+0000","last_active":"2024-09-17T03:50:25.133970+0000","last_peered":"2024-09-17T03:50:25.133970+0000","last_clean":"2024-09-17T03:50:25.133970+0000","last_became_active":"2024-09-17T03:50:23.138826+0000","last_became_peered":"2024-09-17T03:50:23.138826+0000","last_unstale":"2024-09-17T03:50:25.133970+0000","last_undegraded":"2024-09-17T03:50:25.133970+0000","last_fullsized":"2024-09-17T03:50:25.133970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:07.180415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133867+0000","last_change":"2024-09-17T03:50:24.157448+0000","last_active":"2024-09-17T03:50:25.133867+0000","last_peered":"2024-09-17T03:50:25.133867+0000","last_clean":"2024-09-17T03:50:25.133867+0000","last_became_active":"2024-09-17T03:50:24.156599+0000","last_became_peered":"2024-09-17T03:50:24.156599+0000","last_unstale":"2024-09-17T03:50:25.133867+0000","last_undegraded":"2024-09-17T03:50:25.133867+0000","last_fullsized":"2024-09-17T03:50:25.133867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:38.071027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590944+0000","last_change":"2024-09-17T03:50:23.122274+0000","last_active":"2024-09-17T03:50:25.590944+0000","last_peered":"2024-09-17T03:50:25.590944+0000","last_clean":"2024-09-17T03:50:25.590944+0000","last_became_active":"2024-09-17T03:50:23.121928+0000","last_became_peered":"2024-09-17T03:50:23.121928+0000","last_unstale":"2024-09-17T03:50:25.590944+0000","last_undegraded":"2024-09-17T03:50:25.590944+0000","last_fullsized":"2024-09-17T03:50:25.590944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:36.626649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133566+0000","last_change":"2024-09-17T03:50:24.138004+0000","last_active":"2024-09-17T03:50:25.133566+0000","last_peered":"2024-09-17T03:50:25.133566+0000","last_clean":"2024-09-17T03:50:25.133566+0000","last_became_active":"2024-09-17T03:50:24.137679+0000","last_became_peered":"2024-09-17T03:50:24.137679+0000","last_unstale":"2024-09-17T03:50:25.133566+0000","last_undegraded":"2024-09-17T03:50:25.133566+0000","last_fullsized":"2024-09-17T03:50:25.133566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:24.583890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590960+0000","last_change":"2024-09-17T03:50:23.133114+0000","last_active":"2024-09-17T03:50:25.590960+0000","last_peered":"2024-09-17T03:50:25.590960+0000","last_clean":"2024-09-17T03:50:25.590960+0000","last_became_active":"2024-09-17T03:50:23.132725+0000","last_became_peered":"2024-09-17T03:50:23.132725+0000","last_unstale":"2024-09-17T03:50:25.590960+0000","last_undegraded":"2024-09-17T03:50:25.590960+0000","last_fullsized":"2024-09-17T03:50:25.590960+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590361+0000","last_change":"2024-09-17T03:50:24.145715+0000","last_active":"2024-09-17T03:50:25.590361+0000","last_peered":"2024-09-17T03:50:25.590361+0000","last_clean":"2024-09-17T03:50:25.590361+0000","last_became_active":"2024-09-17T03:50:24.145392+0000","last_became_peered":"2024-09-17T03:50:24.145392+0000","last_unstale":"2024-09-17T03:50:25.590361+0000","last_undegraded":"2024-09-17T03:50:25.590361+0000","last_fullsized":"2024-09-17T03:50:25.590361+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":21,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133600+0000","last_change":"2024-09-17T03:50:23.140412+0000","last_active":"2024-09-17T03:50:25.133600+0000","last_peered":"2024-09-17T03:50:25.133600+0000","last_clean":"2024-09-17T03:50:25.133600+0000","last_became_active":"2024-09-17T03:50:23.139391+0000","last_became_peered":"2024-09-17T03:50:23.139391+0000","last_unstale":"2024-09-17T03:50:25.133600+0000","last_undegraded":"2024-09-17T03:50:25.133600+0000","last_fullsized":"2024-09-17T03:50:25.133600+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590621+0000","last_change":"2024-09-17T03:50:24.149315+0000","last_active":"2024-09-17T03:50:25.590621+0000","last_peered":"2024-09-17T03:50:25.590621+0000","last_clean":"2024-09-17T03:50:25.590621+0000","last_became_active":"2024-09-17T03:50:24.147345+0000","last_became_peered":"2024-09-17T03:50:24.147345+0000","last_unstale":"2024-09-17T03:50:25.590621+0000","last_undegraded":"2024-09-17T03:50:25.590621+0000","last_fullsized":"2024-09-17T03:50:25.590621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135266+0000","last_change":"2024-09-17T03:50:23.143154+0000","last_active":"2024-09-17T03:50:25.135266+0000","last_peered":"2024-09-17T03:50:25.135266+0000","last_clean":"2024-09-17T03:50:25.135266+0000","last_became_active":"2024-09-17T03:50:23.142801+0000","last_became_peered":"2024-09-17T03:50:23.142801+0000","last_unstale":"2024-09-17T03:50:25.135266+0000","last_undegraded":"2024-09-17T03:50:25.135266+0000","last_fullsized":"2024-09-17T03:50:25.135266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:17.669331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133412+0000","last_change":"2024-09-17T03:50:24.147109+0000","last_active":"2024-09-17T03:50:25.133412+0000","last_peered":"2024-09-17T03:50:25.133412+0000","last_clean":"2024-09-17T03:50:25.133412+0000","last_became_active":"2024-09-17T03:50:24.146394+0000","last_became_peered":"2024-09-17T03:50:24.146394+0000","last_unstale":"2024-09-17T03:50:25.133412+0000","last_undegraded":"2024-09-17T03:50:25.133412+0000","last_fullsized":"2024-09-17T03:50:25.133412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:34.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590683+0000","last_change":"2024-09-17T03:50:23.135165+0000","last_active":"2024-09-17T03:50:25.590683+0000","last_peered":"2024-09-17T03:50:25.590683+0000","last_clean":"2024-09-17T03:50:25.590683+0000","last_became_active":"2024-09-17T03:50:23.134575+0000","last_became_peered":"2024-09-17T03:50:23.134575+0000","last_unstale":"2024-09-17T03:50:25.590683+0000","last_undegraded":"2024-09-17T03:50:25.590683+0000","last_fullsized":"2024-09-17T03:50:25.590683+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:53.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135744+0000","last_change":"2024-09-17T03:50:24.155803+0000","last_active":"2024-09-17T03:50:25.135744+0000","last_peered":"2024-09-17T03:50:25.135744+0000","last_clean":"2024-09-17T03:50:25.135744+0000","last_became_active":"2024-09-17T03:50:24.154916+0000","last_became_peered":"2024-09-17T03:50:24.154916+0000","last_unstale":"2024-09-17T03:50:25.135744+0000","last_undegraded":"2024-09-17T03:50:25.135744+0000","last_fullsized":"2024-09-17T03:50:25.135744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133126+0000","last_change":"2024-09-17T03:50:23.127502+0000","last_active":"2024-09-17T03:50:25.133126+0000","last_peered":"2024-09-17T03:50:25.133126+0000","last_clean":"2024-09-17T03:50:25.133126+0000","last_became_active":"2024-09-17T03:50:23.126531+0000","last_became_peered":"2024-09-17T03:50:23.126531+0000","last_unstale":"2024-09-17T03:50:25.133126+0000","last_undegraded":"2024-09-17T03:50:25.133126+0000","last_fullsized":"2024-09-17T03:50:25.133126+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:39.251639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590222+0000","last_change":"2024-09-17T03:50:24.148579+0000","last_active":"2024-09-17T03:50:25.590222+0000","last_peered":"2024-09-17T03:50:25.590222+0000","last_clean":"2024-09-17T03:50:25.590222+0000","last_became_active":"2024-09-17T03:50:24.148272+0000","last_became_peered":"2024-09-17T03:50:24.148272+0000","last_unstale":"2024-09-17T03:50:25.590222+0000","last_undegraded":"2024-09-17T03:50:25.590222+0000","last_fullsized":"2024-09-17T03:50:25.590222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:35:08.398615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133060+0000","last_change":"2024-09-17T03:50:23.139904+0000","last_active":"2024-09-17T03:50:25.133060+0000","last_peered":"2024-09-17T03:50:25.133060+0000","last_clean":"2024-09-17T03:50:25.133060+0000","last_became_active":"2024-09-17T03:50:23.139601+0000","last_became_peered":"2024-09-17T03:50:23.139601+0000","last_unstale":"2024-09-17T03:50:25.133060+0000","last_undegraded":"2024-09-17T03:50:25.133060+0000","last_fullsized":"2024-09-17T03:50:25.133060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:21.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136782+0000","last_change":"2024-09-17T03:50:24.150202+0000","last_active":"2024-09-17T03:50:25.136782+0000","last_peered":"2024-09-17T03:50:25.136782+0000","last_clean":"2024-09-17T03:50:25.136782+0000","last_became_active":"2024-09-17T03:50:24.149551+0000","last_became_peered":"2024-09-17T03:50:24.149551+0000","last_unstale":"2024-09-17T03:50:25.136782+0000","last_undegraded":"2024-09-17T03:50:25.136782+0000","last_fullsized":"2024-09-17T03:50:25.136782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590829+0000","last_change":"2024-09-17T03:50:23.140181+0000","last_active":"2024-09-17T03:50:25.590829+0000","last_peered":"2024-09-17T03:50:25.590829+0000","last_clean":"2024-09-17T03:50:25.590829+0000","last_became_active":"2024-09-17T03:50:23.139817+0000","last_became_peered":"2024-09-17T03:50:23.139817+0000","last_unstale":"2024-09-17T03:50:25.590829+0000","last_undegraded":"2024-09-17T03:50:25.590829+0000","last_fullsized":"2024-09-17T03:50:25.590829+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:10.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590763+0000","last_change":"2024-09-17T03:50:24.135542+0000","last_active":"2024-09-17T03:50:25.590763+0000","last_peered":"2024-09-17T03:50:25.590763+0000","last_clean":"2024-09-17T03:50:25.590763+0000","last_became_active":"2024-09-17T03:50:24.135241+0000","last_became_peered":"2024-09-17T03:50:24.135241+0000","last_unstale":"2024-09-17T03:50:25.590763+0000","last_undegraded":"2024-09-17T03:50:25.590763+0000","last_fullsized":"2024-09-17T03:50:25.590763+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136769+0000","last_change":"2024-09-17T03:50:23.139663+0000","last_active":"2024-09-17T03:50:25.136769+0000","last_peered":"2024-09-17T03:50:25.136769+0000","last_clean":"2024-09-17T03:50:25.136769+0000","last_became_active":"2024-09-17T03:50:23.139294+0000","last_became_peered":"2024-09-17T03:50:23.139294+0000","last_unstale":"2024-09-17T03:50:25.136769+0000","last_undegraded":"2024-09-17T03:50:25.136769+0000","last_fullsized":"2024-09-17T03:50:25.136769+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.179574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590670+0000","last_change":"2024-09-17T03:50:24.155061+0000","last_active":"2024-09-17T03:50:25.590670+0000","last_peered":"2024-09-17T03:50:25.590670+0000","last_clean":"2024-09-17T03:50:25.590670+0000","last_became_active":"2024-09-17T03:50:24.154794+0000","last_became_peered":"2024-09-17T03:50:24.154794+0000","last_unstale":"2024-09-17T03:50:25.590670+0000","last_undegraded":"2024-09-17T03:50:25.590670+0000","last_fullsized":"2024-09-17T03:50:25.590670+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135372+0000","last_change":"2024-09-17T03:50:23.143762+0000","last_active":"2024-09-17T03:50:25.135372+0000","last_peered":"2024-09-17T03:50:25.135372+0000","last_clean":"2024-09-17T03:50:25.135372+0000","last_became_active":"2024-09-17T03:50:23.143435+0000","last_became_peered":"2024-09-17T03:50:23.143435+0000","last_unstale":"2024-09-17T03:50:25.135372+0000","last_undegraded":"2024-09-17T03:50:25.135372+0000","last_fullsized":"2024-09-17T03:50:25.135372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:52.817756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590480+0000","last_change":"2024-09-17T03:50:24.138195+0000","last_active":"2024-09-17T03:50:25.590480+0000","last_peered":"2024-09-17T03:50:25.590480+0000","last_clean":"2024-09-17T03:50:25.590480+0000","last_became_active":"2024-09-17T03:50:24.137835+0000","last_became_peered":"2024-09-17T03:50:24.137835+0000","last_unstale":"2024-09-17T03:50:25.590480+0000","last_undegraded":"2024-09-17T03:50:25.590480+0000","last_fullsized":"2024-09-17T03:50:25.590480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:45.999114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135343+0000","last_change":"2024-09-17T03:50:23.140276+0000","last_active":"2024-09-17T03:50:25.135343+0000","last_peered":"2024-09-17T03:50:25.135343+0000","last_clean":"2024-09-17T03:50:25.135343+0000","last_became_active":"2024-09-17T03:50:23.139757+0000","last_became_peered":"2024-09-17T03:50:23.139757+0000","last_unstale":"2024-09-17T03:50:25.135343+0000","last_undegraded":"2024-09-17T03:50:25.135343+0000","last_fullsized":"2024-09-17T03:50:25.135343+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:14:45.951042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591157+0000","last_change":"2024-09-17T03:50:24.152597+0000","last_active":"2024-09-17T03:50:25.591157+0000","last_peered":"2024-09-17T03:50:25.591157+0000","last_clean":"2024-09-17T03:50:25.591157+0000","last_became_active":"2024-09-17T03:50:24.152307+0000","last_became_peered":"2024-09-17T03:50:24.152307+0000","last_unstale":"2024-09-17T03:50:25.591157+0000","last_undegraded":"2024-09-17T03:50:25.591157+0000","last_fullsized":"2024-09-17T03:50:25.591157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:38.869025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589948+0000","last_change":"2024-09-17T03:50:23.130742+0000","last_active":"2024-09-17T03:50:25.589948+0000","last_peered":"2024-09-17T03:50:25.589948+0000","last_clean":"2024-09-17T03:50:25.589948+0000","last_became_active":"2024-09-17T03:50:23.130434+0000","last_became_peered":"2024-09-17T03:50:23.130434+0000","last_unstale":"2024-09-17T03:50:25.589948+0000","last_undegraded":"2024-09-17T03:50:25.589948+0000","last_fullsized":"2024-09-17T03:50:25.589948+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:13.017254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591534+0000","last_change":"2024-09-17T03:50:24.141883+0000","last_active":"2024-09-17T03:50:25.591534+0000","last_peered":"2024-09-17T03:50:25.591534+0000","last_clean":"2024-09-17T03:50:25.591534+0000","last_became_active":"2024-09-17T03:50:24.141572+0000","last_became_peered":"2024-09-17T03:50:24.141572+0000","last_unstale":"2024-09-17T03:50:25.591534+0000","last_undegraded":"2024-09-17T03:50:25.591534+0000","last_fullsized":"2024-09-17T03:50:25.591534+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:11.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591155+0000","last_change":"2024-09-17T03:50:23.133695+0000","last_active":"2024-09-17T03:50:25.591155+0000","last_peered":"2024-09-17T03:50:25.591155+0000","last_clean":"2024-09-17T03:50:25.591155+0000","last_became_active":"2024-09-17T03:50:23.133125+0000","last_became_peered":"2024-09-17T03:50:23.133125+0000","last_unstale":"2024-09-17T03:50:25.591155+0000","last_undegraded":"2024-09-17T03:50:25.591155+0000","last_fullsized":"2024-09-17T03:50:25.591155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:51.417524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589870+0000","last_change":"2024-09-17T03:50:24.141825+0000","last_active":"2024-09-17T03:50:25.589870+0000","last_peered":"2024-09-17T03:50:25.589870+0000","last_clean":"2024-09-17T03:50:25.589870+0000","last_became_active":"2024-09-17T03:50:24.141522+0000","last_became_peered":"2024-09-17T03:50:24.141522+0000","last_unstale":"2024-09-17T03:50:25.589870+0000","last_undegraded":"2024-09-17T03:50:25.589870+0000","last_fullsized":"2024-09-17T03:50:25.589870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:20.207429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134808+0000","last_change":"2024-09-17T03:50:23.141992+0000","last_active":"2024-09-17T03:50:25.134808+0000","last_peered":"2024-09-17T03:50:25.134808+0000","last_clean":"2024-09-17T03:50:25.134808+0000","last_became_active":"2024-09-17T03:50:23.141642+0000","last_became_peered":"2024-09-17T03:50:23.141642+0000","last_unstale":"2024-09-17T03:50:25.134808+0000","last_undegraded":"2024-09-17T03:50:25.134808+0000","last_fullsized":"2024-09-17T03:50:25.134808+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:12.799906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135846+0000","last_change":"2024-09-17T03:50:24.142339+0000","last_active":"2024-09-17T03:50:25.135846+0000","last_peered":"2024-09-17T03:50:25.135846+0000","last_clean":"2024-09-17T03:50:25.135846+0000","last_became_active":"2024-09-17T03:50:24.142028+0000","last_became_peered":"2024-09-17T03:50:24.142028+0000","last_unstale":"2024-09-17T03:50:25.135846+0000","last_undegraded":"2024-09-17T03:50:25.135846+0000","last_fullsized":"2024-09-17T03:50:25.135846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590076+0000","last_change":"2024-09-17T03:50:23.129908+0000","last_active":"2024-09-17T03:50:25.590076+0000","last_peered":"2024-09-17T03:50:25.590076+0000","last_clean":"2024-09-17T03:50:25.590076+0000","last_became_active":"2024-09-17T03:50:23.129543+0000","last_became_peered":"2024-09-17T03:50:23.129543+0000","last_unstale":"2024-09-17T03:50:25.590076+0000","last_undegraded":"2024-09-17T03:50:25.590076+0000","last_fullsized":"2024-09-17T03:50:25.590076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591135+0000","last_change":"2024-09-17T03:50:24.124588+0000","last_active":"2024-09-17T03:50:25.591135+0000","last_peered":"2024-09-17T03:50:25.591135+0000","last_clean":"2024-09-17T03:50:25.591135+0000","last_became_active":"2024-09-17T03:50:24.124178+0000","last_became_peered":"2024-09-17T03:50:24.124178+0000","last_unstale":"2024-09-17T03:50:25.591135+0000","last_undegraded":"2024-09-17T03:50:25.591135+0000","last_fullsized":"2024-09-17T03:50:25.591135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:41.718876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590917+0000","last_change":"2024-09-17T03:50:23.137622+0000","last_active":"2024-09-17T03:50:25.590917+0000","last_peered":"2024-09-17T03:50:25.590917+0000","last_clean":"2024-09-17T03:50:25.590917+0000","last_became_active":"2024-09-17T03:50:23.137258+0000","last_became_peered":"2024-09-17T03:50:23.137258+0000","last_unstale":"2024-09-17T03:50:25.590917+0000","last_undegraded":"2024-09-17T03:50:25.590917+0000","last_fullsized":"2024-09-17T03:50:25.590917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:40.356807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590857+0000","last_change":"2024-09-17T03:50:24.140577+0000","last_active":"2024-09-17T03:50:25.590857+0000","last_peered":"2024-09-17T03:50:25.590857+0000","last_clean":"2024-09-17T03:50:25.590857+0000","last_became_active":"2024-09-17T03:50:24.140330+0000","last_became_peered":"2024-09-17T03:50:24.140330+0000","last_unstale":"2024-09-17T03:50:25.590857+0000","last_undegraded":"2024-09-17T03:50:25.590857+0000","last_fullsized":"2024-09-17T03:50:25.590857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:00.638143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591009+0000","last_change":"2024-09-17T03:50:23.130628+0000","last_active":"2024-09-17T03:50:25.591009+0000","last_peered":"2024-09-17T03:50:25.591009+0000","last_clean":"2024-09-17T03:50:25.591009+0000","last_became_active":"2024-09-17T03:50:23.130307+0000","last_became_peered":"2024-09-17T03:50:23.130307+0000","last_unstale":"2024-09-17T03:50:25.591009+0000","last_undegraded":"2024-09-17T03:50:25.591009+0000","last_fullsized":"2024-09-17T03:50:25.591009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:26.309251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589837+0000","last_change":"2024-09-17T03:50:24.151860+0000","last_active":"2024-09-17T03:50:25.589837+0000","last_peered":"2024-09-17T03:50:25.589837+0000","last_clean":"2024-09-17T03:50:25.589837+0000","last_became_active":"2024-09-17T03:50:24.150602+0000","last_became_peered":"2024-09-17T03:50:24.150602+0000","last_unstale":"2024-09-17T03:50:25.589837+0000","last_undegraded":"2024-09-17T03:50:25.589837+0000","last_fullsized":"2024-09-17T03:50:25.589837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:13.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590934+0000","last_change":"2024-09-17T03:50:23.136531+0000","last_active":"2024-09-17T03:50:25.590934+0000","last_peered":"2024-09-17T03:50:25.590934+0000","last_clean":"2024-09-17T03:50:25.590934+0000","last_became_active":"2024-09-17T03:50:23.136207+0000","last_became_peered":"2024-09-17T03:50:23.136207+0000","last_unstale":"2024-09-17T03:50:25.590934+0000","last_undegraded":"2024-09-17T03:50:25.590934+0000","last_fullsized":"2024-09-17T03:50:25.590934+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:47.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590966+0000","last_change":"2024-09-17T03:50:24.140447+0000","last_active":"2024-09-17T03:50:25.590966+0000","last_peered":"2024-09-17T03:50:25.590966+0000","last_clean":"2024-09-17T03:50:25.590966+0000","last_became_active":"2024-09-17T03:50:24.140095+0000","last_became_peered":"2024-09-17T03:50:24.140095+0000","last_unstale":"2024-09-17T03:50:25.590966+0000","last_undegraded":"2024-09-17T03:50:25.590966+0000","last_fullsized":"2024-09-17T03:50:25.590966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:42:53.379705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590069+0000","last_change":"2024-09-17T03:50:23.124066+0000","last_active":"2024-09-17T03:50:25.590069+0000","last_peered":"2024-09-17T03:50:25.590069+0000","last_clean":"2024-09-17T03:50:25.590069+0000","last_became_active":"2024-09-17T03:50:23.123776+0000","last_became_peered":"2024-09-17T03:50:23.123776+0000","last_unstale":"2024-09-17T03:50:25.590069+0000","last_undegraded":"2024-09-17T03:50:25.590069+0000","last_fullsized":"2024-09-17T03:50:25.590069+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:39.566116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589894+0000","last_change":"2024-09-17T03:50:24.132547+0000","last_active":"2024-09-17T03:50:25.589894+0000","last_peered":"2024-09-17T03:50:25.589894+0000","last_clean":"2024-09-17T03:50:25.589894+0000","last_became_active":"2024-09-17T03:50:24.132193+0000","last_became_peered":"2024-09-17T03:50:24.132193+0000","last_unstale":"2024-09-17T03:50:25.589894+0000","last_undegraded":"2024-09-17T03:50:25.589894+0000","last_fullsized":"2024-09-17T03:50:25.589894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:28.964951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590804+0000","last_change":"2024-09-17T03:50:23.136967+0000","last_active":"2024-09-17T03:50:25.590804+0000","last_peered":"2024-09-17T03:50:25.590804+0000","last_clean":"2024-09-17T03:50:25.590804+0000","last_became_active":"2024-09-17T03:50:23.136643+0000","last_became_peered":"2024-09-17T03:50:23.136643+0000","last_unstale":"2024-09-17T03:50:25.590804+0000","last_undegraded":"2024-09-17T03:50:25.590804+0000","last_fullsized":"2024-09-17T03:50:25.590804+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:58.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133036+0000","last_change":"2024-09-17T03:50:24.150858+0000","last_active":"2024-09-17T03:50:25.133036+0000","last_peered":"2024-09-17T03:50:25.133036+0000","last_clean":"2024-09-17T03:50:25.133036+0000","last_became_active":"2024-09-17T03:50:24.150509+0000","last_became_peered":"2024-09-17T03:50:24.150509+0000","last_unstale":"2024-09-17T03:50:25.133036+0000","last_undegraded":"2024-09-17T03:50:25.133036+0000","last_fullsized":"2024-09-17T03:50:25.133036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134927+0000","last_change":"2024-09-17T03:50:23.141022+0000","last_active":"2024-09-17T03:50:25.134927+0000","last_peered":"2024-09-17T03:50:25.134927+0000","last_clean":"2024-09-17T03:50:25.134927+0000","last_became_active":"2024-09-17T03:50:23.140264+0000","last_became_peered":"2024-09-17T03:50:23.140264+0000","last_unstale":"2024-09-17T03:50:25.134927+0000","last_undegraded":"2024-09-17T03:50:25.134927+0000","last_fullsized":"2024-09-17T03:50:25.134927+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:55.135377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134615+0000","last_change":"2024-09-17T03:50:24.149913+0000","last_active":"2024-09-17T03:50:25.134615+0000","last_peered":"2024-09-17T03:50:25.134615+0000","last_clean":"2024-09-17T03:50:25.134615+0000","last_became_active":"2024-09-17T03:50:24.149591+0000","last_became_peered":"2024-09-17T03:50:24.149591+0000","last_unstale":"2024-09-17T03:50:25.134615+0000","last_undegraded":"2024-09-17T03:50:25.134615+0000","last_fullsized":"2024-09-17T03:50:25.134615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133185+0000","last_change":"2024-09-17T03:50:23.130536+0000","last_active":"2024-09-17T03:50:25.133185+0000","last_peered":"2024-09-17T03:50:25.133185+0000","last_clean":"2024-09-17T03:50:25.133185+0000","last_became_active":"2024-09-17T03:50:23.130176+0000","last_became_peered":"2024-09-17T03:50:23.130176+0000","last_unstale":"2024-09-17T03:50:25.133185+0000","last_undegraded":"2024-09-17T03:50:25.133185+0000","last_fullsized":"2024-09-17T03:50:25.133185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:26.703578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137102+0000","last_change":"2024-09-17T03:50:24.146715+0000","last_active":"2024-09-17T03:50:25.137102+0000","last_peered":"2024-09-17T03:50:25.137102+0000","last_clean":"2024-09-17T03:50:25.137102+0000","last_became_active":"2024-09-17T03:50:24.146329+0000","last_became_peered":"2024-09-17T03:50:24.146329+0000","last_unstale":"2024-09-17T03:50:25.137102+0000","last_undegraded":"2024-09-17T03:50:25.137102+0000","last_fullsized":"2024-09-17T03:50:25.137102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590037+0000","last_change":"2024-09-17T03:50:23.138899+0000","last_active":"2024-09-17T03:50:25.590037+0000","last_peered":"2024-09-17T03:50:25.590037+0000","last_clean":"2024-09-17T03:50:25.590037+0000","last_became_active":"2024-09-17T03:50:23.138546+0000","last_became_peered":"2024-09-17T03:50:23.138546+0000","last_unstale":"2024-09-17T03:50:25.590037+0000","last_undegraded":"2024-09-17T03:50:25.590037+0000","last_fullsized":"2024-09-17T03:50:25.590037+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.030426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":130,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136109+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:50:25.136109+0000","last_peered":"2024-09-17T03:50:25.136109+0000","last_clean":"2024-09-17T03:50:25.136109+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:50:25.136109+0000","last_undegraded":"2024-09-17T03:50:25.136109+0000","last_fullsized":"2024-09-17T03:50:25.136109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":730824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509458,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":372736,"data_stored":141038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":372736,"data_stored":140470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":380928,"data_stored":142000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":368640,"data_stored":140436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":380928,"data_stored":142000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542159,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":140982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":958464,"data_stored":730734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:50:33.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:50:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 -- 172.21.15.129:0/654139412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 msgr2=0x7f17500d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:33.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 --2- 172.21.15.129:0/654139412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500d04a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f175009dda0 tx=0x7f174802a9c0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 -- 172.21.15.129:0/654139412 shutdown_connections 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 --2- 172.21.15.129:0/654139412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500d04a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 --2- 172.21.15.129:0/654139412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500b3b60 0x7f17500cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 -- 172.21.15.129:0/654139412 >> 172.21.15.129:0/654139412 conn(0x7f175009d690 msgr2=0x7f17500d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 -- 172.21.15.129:0/654139412 shutdown_connections 2024-09-17T03:50:33.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.743+0000 7f1756200640 1 -- 172.21.15.129:0/654139412 wait complete. 2024-09-17T03:50:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.745+0000 7f1756200640 1 Processor -- start 2024-09-17T03:50:33.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.745+0000 7f1756200640 1 -- start start 2024-09-17T03:50:33.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17500b3b60 0x7f175014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500d2af0 0x7f17501f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1750161840 con 0x7f17500d0090 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17501616c0 con 0x7f17500b3b60 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1756200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17501619c0 con 0x7f17500d2af0 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1747e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1747e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41324/0 (socket says 172.21.15.129:41324) 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1747e00640 1 -- 172.21.15.129:0/2929772615 learned_addr learned my addr 172.21.15.129:0/2929772615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1755800640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500d2af0 0x7f17501f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.747+0000 7f1754e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17500b3b60 0x7f175014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 -- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500d2af0 msgr2=0x7f17501f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500d2af0 0x7f17501f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 -- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17500b3b60 msgr2=0x7f175014f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17500b3b60 0x7f175014f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 -- 172.21.15.129:0/2929772615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17500cd2f0 con 0x7f17500d0090 2024-09-17T03:50:33.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1747e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f173800c970 tx=0x7f173800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738007bb0 con 0x7f17500d0090 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1738007e40 con 0x7f17500d0090 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173800f690 con 0x7f17500d0090 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17500cd5d0 con 0x7f17500d0090 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.748+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17500cdb90 con 0x7f17500d0090 2024-09-17T03:50:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.750+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17500d04a0 con 0x7f17500d0090 2024-09-17T03:50:33.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.751+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f173800f830 con 0x7f17500d0090 2024-09-17T03:50:33.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.752+0000 7f174d600640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 0x7f172c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:33.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.752+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f173809b330 con 0x7f17500d0090 2024-09-17T03:50:33.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.752+0000 7f1754e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 0x7f172c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:33.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.753+0000 7f1754e00640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 0x7f172c07b0a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1740006fd0 tx=0x7f1740009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:33.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.756+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1738014030 con 0x7f17500d0090 2024-09-17T03:50:33.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.938+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f17500cd080 con 0x7f172c078b90 2024-09-17T03:50:33.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.943+0000 7f174d600640 1 -- 172.21.15.129:0/2929772615 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251928 (secure 0 0 0) 0x7f17500cd080 con 0x7f172c078b90 2024-09-17T03:50:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:33.947 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 msgr2=0x7f172c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 0x7f172c07b0a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1740006fd0 tx=0x7f1740009060 comp rx=0 tx=0).stop 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 msgr2=0x7f17500cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f173800c970 tx=0x7f173800ce30 comp rx=0 tx=0).stop 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 shutdown_connections 2024-09-17T03:50:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f172c078b90 0x7f172c07b0a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17500d2af0 0x7f17501f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17500d0090 0x7f17500cc830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 --2- 172.21.15.129:0/2929772615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17500b3b60 0x7f175014f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 >> 172.21.15.129:0/2929772615 conn(0x7f175009d690 msgr2=0x7f17500b18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 shutdown_connections 2024-09-17T03:50:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:33.947+0000 7f1756200640 1 -- 172.21.15.129:0/2929772615 wait complete. 2024-09-17T03:50:33.971 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-09-17T03:50:32.307891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":224496,"kb_used_data":4072,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754750224,"statfs":{"total":773094113280,"available":772864229376,"internally_reserved":0,"allocated":4169728,"data_stored":2308484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.192860"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590433+0000","last_change":"2024-09-17T03:50:24.138380+0000","last_active":"2024-09-17T03:50:25.590433+0000","last_peered":"2024-09-17T03:50:25.590433+0000","last_clean":"2024-09-17T03:50:25.590433+0000","last_became_active":"2024-09-17T03:50:24.137737+0000","last_became_peered":"2024-09-17T03:50:24.137737+0000","last_unstale":"2024-09-17T03:50:25.590433+0000","last_undegraded":"2024-09-17T03:50:25.590433+0000","last_fullsized":"2024-09-17T03:50:25.590433+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:13.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133801+0000","last_change":"2024-09-17T03:50:24.146222+0000","last_active":"2024-09-17T03:50:25.133801+0000","last_peered":"2024-09-17T03:50:25.133801+0000","last_clean":"2024-09-17T03:50:25.133801+0000","last_became_active":"2024-09-17T03:50:24.145925+0000","last_became_peered":"2024-09-17T03:50:24.145925+0000","last_unstale":"2024-09-17T03:50:25.133801+0000","last_undegraded":"2024-09-17T03:50:25.133801+0000","last_fullsized":"2024-09-17T03:50:25.133801+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:09.445391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590859+0000","last_change":"2024-09-17T03:50:24.146390+0000","last_active":"2024-09-17T03:50:25.590859+0000","last_peered":"2024-09-17T03:50:25.590859+0000","last_clean":"2024-09-17T03:50:25.590859+0000","last_became_active":"2024-09-17T03:50:24.146067+0000","last_became_peered":"2024-09-17T03:50:24.146067+0000","last_unstale":"2024-09-17T03:50:25.590859+0000","last_undegraded":"2024-09-17T03:50:25.590859+0000","last_fullsized":"2024-09-17T03:50:25.590859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590418+0000","last_change":"2024-09-17T03:50:24.156017+0000","last_active":"2024-09-17T03:50:25.590418+0000","last_peered":"2024-09-17T03:50:25.590418+0000","last_clean":"2024-09-17T03:50:25.590418+0000","last_became_active":"2024-09-17T03:50:24.155677+0000","last_became_peered":"2024-09-17T03:50:24.155677+0000","last_unstale":"2024-09-17T03:50:25.590418+0000","last_undegraded":"2024-09-17T03:50:25.590418+0000","last_fullsized":"2024-09-17T03:50:25.590418+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591334+0000","last_change":"2024-09-17T03:50:24.138985+0000","last_active":"2024-09-17T03:50:25.591334+0000","last_peered":"2024-09-17T03:50:25.591334+0000","last_clean":"2024-09-17T03:50:25.591334+0000","last_became_active":"2024-09-17T03:50:24.138569+0000","last_became_peered":"2024-09-17T03:50:24.138569+0000","last_unstale":"2024-09-17T03:50:25.591334+0000","last_undegraded":"2024-09-17T03:50:25.591334+0000","last_fullsized":"2024-09-17T03:50:25.591334+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:01.229274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133360+0000","last_change":"2024-09-17T03:50:24.147892+0000","last_active":"2024-09-17T03:50:25.133360+0000","last_peered":"2024-09-17T03:50:25.133360+0000","last_clean":"2024-09-17T03:50:25.133360+0000","last_became_active":"2024-09-17T03:50:24.147616+0000","last_became_peered":"2024-09-17T03:50:24.147616+0000","last_unstale":"2024-09-17T03:50:25.133360+0000","last_undegraded":"2024-09-17T03:50:25.133360+0000","last_fullsized":"2024-09-17T03:50:25.133360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136144+0000","last_change":"2024-09-17T03:50:24.148414+0000","last_active":"2024-09-17T03:50:25.136144+0000","last_peered":"2024-09-17T03:50:25.136144+0000","last_clean":"2024-09-17T03:50:25.136144+0000","last_became_active":"2024-09-17T03:50:24.148020+0000","last_became_peered":"2024-09-17T03:50:24.148020+0000","last_unstale":"2024-09-17T03:50:25.136144+0000","last_undegraded":"2024-09-17T03:50:25.136144+0000","last_fullsized":"2024-09-17T03:50:25.136144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134558+0000","last_change":"2024-09-17T03:50:24.147045+0000","last_active":"2024-09-17T03:50:25.134558+0000","last_peered":"2024-09-17T03:50:25.134558+0000","last_clean":"2024-09-17T03:50:25.134558+0000","last_became_active":"2024-09-17T03:50:24.146692+0000","last_became_peered":"2024-09-17T03:50:24.146692+0000","last_unstale":"2024-09-17T03:50:25.134558+0000","last_undegraded":"2024-09-17T03:50:25.134558+0000","last_fullsized":"2024-09-17T03:50:25.134558+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136781+0000","last_change":"2024-09-17T03:50:24.149261+0000","last_active":"2024-09-17T03:50:25.136781+0000","last_peered":"2024-09-17T03:50:25.136781+0000","last_clean":"2024-09-17T03:50:25.136781+0000","last_became_active":"2024-09-17T03:50:24.148898+0000","last_became_peered":"2024-09-17T03:50:24.148898+0000","last_unstale":"2024-09-17T03:50:25.136781+0000","last_undegraded":"2024-09-17T03:50:25.136781+0000","last_fullsized":"2024-09-17T03:50:25.136781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.371007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591213+0000","last_change":"2024-09-17T03:50:24.152445+0000","last_active":"2024-09-17T03:50:25.591213+0000","last_peered":"2024-09-17T03:50:25.591213+0000","last_clean":"2024-09-17T03:50:25.591213+0000","last_became_active":"2024-09-17T03:50:24.152179+0000","last_became_peered":"2024-09-17T03:50:24.152179+0000","last_unstale":"2024-09-17T03:50:25.591213+0000","last_undegraded":"2024-09-17T03:50:25.591213+0000","last_fullsized":"2024-09-17T03:50:25.591213+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137221+0000","last_change":"2024-09-17T03:50:24.155248+0000","last_active":"2024-09-17T03:50:25.137221+0000","last_peered":"2024-09-17T03:50:25.137221+0000","last_clean":"2024-09-17T03:50:25.137221+0000","last_became_active":"2024-09-17T03:50:24.154779+0000","last_became_peered":"2024-09-17T03:50:24.154779+0000","last_unstale":"2024-09-17T03:50:25.137221+0000","last_undegraded":"2024-09-17T03:50:25.137221+0000","last_fullsized":"2024-09-17T03:50:25.137221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:56.127224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590002+0000","last_change":"2024-09-17T03:50:24.138736+0000","last_active":"2024-09-17T03:50:25.590002+0000","last_peered":"2024-09-17T03:50:25.590002+0000","last_clean":"2024-09-17T03:50:25.590002+0000","last_became_active":"2024-09-17T03:50:24.138424+0000","last_became_peered":"2024-09-17T03:50:24.138424+0000","last_unstale":"2024-09-17T03:50:25.590002+0000","last_undegraded":"2024-09-17T03:50:25.590002+0000","last_fullsized":"2024-09-17T03:50:25.590002+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591349+0000","last_change":"2024-09-17T03:50:24.142434+0000","last_active":"2024-09-17T03:50:25.591349+0000","last_peered":"2024-09-17T03:50:25.591349+0000","last_clean":"2024-09-17T03:50:25.591349+0000","last_became_active":"2024-09-17T03:50:24.142083+0000","last_became_peered":"2024-09-17T03:50:24.142083+0000","last_unstale":"2024-09-17T03:50:25.591349+0000","last_undegraded":"2024-09-17T03:50:25.591349+0000","last_fullsized":"2024-09-17T03:50:25.591349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:18.672400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133627+0000","last_change":"2024-09-17T03:50:24.154707+0000","last_active":"2024-09-17T03:50:25.133627+0000","last_peered":"2024-09-17T03:50:25.133627+0000","last_clean":"2024-09-17T03:50:25.133627+0000","last_became_active":"2024-09-17T03:50:24.154328+0000","last_became_peered":"2024-09-17T03:50:24.154328+0000","last_unstale":"2024-09-17T03:50:25.133627+0000","last_undegraded":"2024-09-17T03:50:25.133627+0000","last_fullsized":"2024-09-17T03:50:25.133627+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:02:18.751353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590043+0000","last_change":"2024-09-17T03:50:24.125931+0000","last_active":"2024-09-17T03:50:25.590043+0000","last_peered":"2024-09-17T03:50:25.590043+0000","last_clean":"2024-09-17T03:50:25.590043+0000","last_became_active":"2024-09-17T03:50:24.125300+0000","last_became_peered":"2024-09-17T03:50:24.125300+0000","last_unstale":"2024-09-17T03:50:25.590043+0000","last_undegraded":"2024-09-17T03:50:25.590043+0000","last_fullsized":"2024-09-17T03:50:25.590043+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135997+0000","last_change":"2024-09-17T03:50:24.144536+0000","last_active":"2024-09-17T03:50:25.135997+0000","last_peered":"2024-09-17T03:50:25.135997+0000","last_clean":"2024-09-17T03:50:25.135997+0000","last_became_active":"2024-09-17T03:50:24.143960+0000","last_became_peered":"2024-09-17T03:50:24.143960+0000","last_unstale":"2024-09-17T03:50:25.135997+0000","last_undegraded":"2024-09-17T03:50:25.135997+0000","last_fullsized":"2024-09-17T03:50:25.135997+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137463+0000","last_change":"2024-09-17T03:50:24.147697+0000","last_active":"2024-09-17T03:50:25.137463+0000","last_peered":"2024-09-17T03:50:25.137463+0000","last_clean":"2024-09-17T03:50:25.137463+0000","last_became_active":"2024-09-17T03:50:24.146930+0000","last_became_peered":"2024-09-17T03:50:24.146930+0000","last_unstale":"2024-09-17T03:50:25.137463+0000","last_undegraded":"2024-09-17T03:50:25.137463+0000","last_fullsized":"2024-09-17T03:50:25.137463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:24.583890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133874+0000","last_change":"2024-09-17T03:50:24.159788+0000","last_active":"2024-09-17T03:50:25.133874+0000","last_peered":"2024-09-17T03:50:25.133874+0000","last_clean":"2024-09-17T03:50:25.133874+0000","last_became_active":"2024-09-17T03:50:24.159505+0000","last_became_peered":"2024-09-17T03:50:24.159505+0000","last_unstale":"2024-09-17T03:50:25.133874+0000","last_undegraded":"2024-09-17T03:50:25.133874+0000","last_fullsized":"2024-09-17T03:50:25.133874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:34.249002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133337+0000","last_change":"2024-09-17T03:50:24.154330+0000","last_active":"2024-09-17T03:50:25.133337+0000","last_peered":"2024-09-17T03:50:25.133337+0000","last_clean":"2024-09-17T03:50:25.133337+0000","last_became_active":"2024-09-17T03:50:24.153860+0000","last_became_peered":"2024-09-17T03:50:24.153860+0000","last_unstale":"2024-09-17T03:50:25.133337+0000","last_undegraded":"2024-09-17T03:50:25.133337+0000","last_fullsized":"2024-09-17T03:50:25.133337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590577+0000","last_change":"2024-09-17T03:50:24.128012+0000","last_active":"2024-09-17T03:50:25.590577+0000","last_peered":"2024-09-17T03:50:25.590577+0000","last_clean":"2024-09-17T03:50:25.590577+0000","last_became_active":"2024-09-17T03:50:24.127633+0000","last_became_peered":"2024-09-17T03:50:24.127633+0000","last_unstale":"2024-09-17T03:50:25.590577+0000","last_undegraded":"2024-09-17T03:50:25.590577+0000","last_fullsized":"2024-09-17T03:50:25.590577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:27.183403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134522+0000","last_change":"2024-09-17T03:50:24.149865+0000","last_active":"2024-09-17T03:50:25.134522+0000","last_peered":"2024-09-17T03:50:25.134522+0000","last_clean":"2024-09-17T03:50:25.134522+0000","last_became_active":"2024-09-17T03:50:24.149569+0000","last_became_peered":"2024-09-17T03:50:24.149569+0000","last_unstale":"2024-09-17T03:50:25.134522+0000","last_undegraded":"2024-09-17T03:50:25.134522+0000","last_fullsized":"2024-09-17T03:50:25.134522+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:40.863699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134957+0000","last_change":"2024-09-17T03:50:24.151774+0000","last_active":"2024-09-17T03:50:25.134957+0000","last_peered":"2024-09-17T03:50:25.134957+0000","last_clean":"2024-09-17T03:50:25.134957+0000","last_became_active":"2024-09-17T03:50:24.151379+0000","last_became_peered":"2024-09-17T03:50:24.151379+0000","last_unstale":"2024-09-17T03:50:25.134957+0000","last_undegraded":"2024-09-17T03:50:25.134957+0000","last_fullsized":"2024-09-17T03:50:25.134957+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:27.706648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589980+0000","last_change":"2024-09-17T03:50:24.136829+0000","last_active":"2024-09-17T03:50:25.589980+0000","last_peered":"2024-09-17T03:50:25.589980+0000","last_clean":"2024-09-17T03:50:25.589980+0000","last_became_active":"2024-09-17T03:50:24.136544+0000","last_became_peered":"2024-09-17T03:50:24.136544+0000","last_unstale":"2024-09-17T03:50:25.589980+0000","last_undegraded":"2024-09-17T03:50:25.589980+0000","last_fullsized":"2024-09-17T03:50:25.589980+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:44.698139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589782+0000","last_change":"2024-09-17T03:50:24.132642+0000","last_active":"2024-09-17T03:50:25.589782+0000","last_peered":"2024-09-17T03:50:25.589782+0000","last_clean":"2024-09-17T03:50:25.589782+0000","last_became_active":"2024-09-17T03:50:24.131051+0000","last_became_peered":"2024-09-17T03:50:24.131051+0000","last_unstale":"2024-09-17T03:50:25.589782+0000","last_undegraded":"2024-09-17T03:50:25.589782+0000","last_fullsized":"2024-09-17T03:50:25.589782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:34.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590575+0000","last_change":"2024-09-17T03:50:24.145119+0000","last_active":"2024-09-17T03:50:25.590575+0000","last_peered":"2024-09-17T03:50:25.590575+0000","last_clean":"2024-09-17T03:50:25.590575+0000","last_became_active":"2024-09-17T03:50:24.144619+0000","last_became_peered":"2024-09-17T03:50:24.144619+0000","last_unstale":"2024-09-17T03:50:25.590575+0000","last_undegraded":"2024-09-17T03:50:25.590575+0000","last_fullsized":"2024-09-17T03:50:25.590575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134040+0000","last_change":"2024-09-17T03:50:24.136492+0000","last_active":"2024-09-17T03:50:25.134040+0000","last_peered":"2024-09-17T03:50:25.134040+0000","last_clean":"2024-09-17T03:50:25.134040+0000","last_became_active":"2024-09-17T03:50:24.136206+0000","last_became_peered":"2024-09-17T03:50:24.136206+0000","last_unstale":"2024-09-17T03:50:25.134040+0000","last_undegraded":"2024-09-17T03:50:25.134040+0000","last_fullsized":"2024-09-17T03:50:25.134040+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:25.732787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590744+0000","last_change":"2024-09-17T03:50:24.155949+0000","last_active":"2024-09-17T03:50:25.590744+0000","last_peered":"2024-09-17T03:50:25.590744+0000","last_clean":"2024-09-17T03:50:25.590744+0000","last_became_active":"2024-09-17T03:50:24.155692+0000","last_became_peered":"2024-09-17T03:50:24.155692+0000","last_unstale":"2024-09-17T03:50:25.590744+0000","last_undegraded":"2024-09-17T03:50:25.590744+0000","last_fullsized":"2024-09-17T03:50:25.590744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:55.362303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590305+0000","last_change":"2024-09-17T03:50:24.136298+0000","last_active":"2024-09-17T03:50:25.590305+0000","last_peered":"2024-09-17T03:50:25.590305+0000","last_clean":"2024-09-17T03:50:25.590305+0000","last_became_active":"2024-09-17T03:50:24.135828+0000","last_became_peered":"2024-09-17T03:50:24.135828+0000","last_unstale":"2024-09-17T03:50:25.590305+0000","last_undegraded":"2024-09-17T03:50:25.590305+0000","last_fullsized":"2024-09-17T03:50:25.590305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135035+0000","last_change":"2024-09-17T03:50:24.152059+0000","last_active":"2024-09-17T03:50:25.135035+0000","last_peered":"2024-09-17T03:50:25.135035+0000","last_clean":"2024-09-17T03:50:25.135035+0000","last_became_active":"2024-09-17T03:50:24.151751+0000","last_became_peered":"2024-09-17T03:50:24.151751+0000","last_unstale":"2024-09-17T03:50:25.135035+0000","last_undegraded":"2024-09-17T03:50:25.135035+0000","last_fullsized":"2024-09-17T03:50:25.135035+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:32.063908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136461+0000","last_change":"2024-09-17T03:50:24.147844+0000","last_active":"2024-09-17T03:50:25.136461+0000","last_peered":"2024-09-17T03:50:25.136461+0000","last_clean":"2024-09-17T03:50:25.136461+0000","last_became_active":"2024-09-17T03:50:24.144862+0000","last_became_peered":"2024-09-17T03:50:24.144862+0000","last_unstale":"2024-09-17T03:50:25.136461+0000","last_undegraded":"2024-09-17T03:50:25.136461+0000","last_fullsized":"2024-09-17T03:50:25.136461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590186+0000","last_change":"2024-09-17T03:50:24.137755+0000","last_active":"2024-09-17T03:50:25.590186+0000","last_peered":"2024-09-17T03:50:25.590186+0000","last_clean":"2024-09-17T03:50:25.590186+0000","last_became_active":"2024-09-17T03:50:24.132876+0000","last_became_peered":"2024-09-17T03:50:24.132876+0000","last_unstale":"2024-09-17T03:50:25.590186+0000","last_undegraded":"2024-09-17T03:50:25.590186+0000","last_fullsized":"2024-09-17T03:50:25.590186+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:34.573301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134419+0000","last_change":"2024-09-17T03:50:24.153513+0000","last_active":"2024-09-17T03:50:25.134419+0000","last_peered":"2024-09-17T03:50:25.134419+0000","last_clean":"2024-09-17T03:50:25.134419+0000","last_became_active":"2024-09-17T03:50:24.153118+0000","last_became_peered":"2024-09-17T03:50:24.153118+0000","last_unstale":"2024-09-17T03:50:25.134419+0000","last_undegraded":"2024-09-17T03:50:25.134419+0000","last_fullsized":"2024-09-17T03:50:25.134419+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590745+0000","last_change":"2024-09-17T03:50:24.139695+0000","last_active":"2024-09-17T03:50:25.590745+0000","last_peered":"2024-09-17T03:50:25.590745+0000","last_clean":"2024-09-17T03:50:25.590745+0000","last_became_active":"2024-09-17T03:50:24.136102+0000","last_became_peered":"2024-09-17T03:50:24.136102+0000","last_unstale":"2024-09-17T03:50:25.590745+0000","last_undegraded":"2024-09-17T03:50:25.590745+0000","last_fullsized":"2024-09-17T03:50:25.590745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:42:53.379705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":19,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590623+0000","last_change":"2024-09-17T03:50:23.138230+0000","last_active":"2024-09-17T03:50:25.590623+0000","last_peered":"2024-09-17T03:50:25.590623+0000","last_clean":"2024-09-17T03:50:25.590623+0000","last_became_active":"2024-09-17T03:50:23.137347+0000","last_became_peered":"2024-09-17T03:50:23.137347+0000","last_unstale":"2024-09-17T03:50:25.590623+0000","last_undegraded":"2024-09-17T03:50:25.590623+0000","last_fullsized":"2024-09-17T03:50:25.590623+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:13.017254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590598+0000","last_change":"2024-09-17T03:50:24.138595+0000","last_active":"2024-09-17T03:50:25.590598+0000","last_peered":"2024-09-17T03:50:25.590598+0000","last_clean":"2024-09-17T03:50:25.590598+0000","last_became_active":"2024-09-17T03:50:24.138208+0000","last_became_peered":"2024-09-17T03:50:24.138208+0000","last_unstale":"2024-09-17T03:50:25.590598+0000","last_undegraded":"2024-09-17T03:50:25.590598+0000","last_fullsized":"2024-09-17T03:50:25.590598+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:28.964951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590824+0000","last_change":"2024-09-17T03:50:23.128500+0000","last_active":"2024-09-17T03:50:25.590824+0000","last_peered":"2024-09-17T03:50:25.590824+0000","last_clean":"2024-09-17T03:50:25.590824+0000","last_became_active":"2024-09-17T03:50:23.128184+0000","last_became_peered":"2024-09-17T03:50:23.128184+0000","last_unstale":"2024-09-17T03:50:25.590824+0000","last_undegraded":"2024-09-17T03:50:25.590824+0000","last_fullsized":"2024-09-17T03:50:25.590824+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:57.033701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135073+0000","last_change":"2024-09-17T03:50:24.152519+0000","last_active":"2024-09-17T03:50:25.135073+0000","last_peered":"2024-09-17T03:50:25.135073+0000","last_clean":"2024-09-17T03:50:25.135073+0000","last_became_active":"2024-09-17T03:50:24.150781+0000","last_became_peered":"2024-09-17T03:50:24.150781+0000","last_unstale":"2024-09-17T03:50:25.135073+0000","last_undegraded":"2024-09-17T03:50:25.135073+0000","last_fullsized":"2024-09-17T03:50:25.135073+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133489+0000","last_change":"2024-09-17T03:50:23.132879+0000","last_active":"2024-09-17T03:50:25.133489+0000","last_peered":"2024-09-17T03:50:25.133489+0000","last_clean":"2024-09-17T03:50:25.133489+0000","last_became_active":"2024-09-17T03:50:23.131983+0000","last_became_peered":"2024-09-17T03:50:23.131983+0000","last_unstale":"2024-09-17T03:50:25.133489+0000","last_undegraded":"2024-09-17T03:50:25.133489+0000","last_fullsized":"2024-09-17T03:50:25.133489+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:39.860630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590970+0000","last_change":"2024-09-17T03:50:24.156534+0000","last_active":"2024-09-17T03:50:25.590970+0000","last_peered":"2024-09-17T03:50:25.590970+0000","last_clean":"2024-09-17T03:50:25.590970+0000","last_became_active":"2024-09-17T03:50:24.156249+0000","last_became_peered":"2024-09-17T03:50:24.156249+0000","last_unstale":"2024-09-17T03:50:25.590970+0000","last_undegraded":"2024-09-17T03:50:25.590970+0000","last_fullsized":"2024-09-17T03:50:25.590970+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590791+0000","last_change":"2024-09-17T03:50:23.134919+0000","last_active":"2024-09-17T03:50:25.590791+0000","last_peered":"2024-09-17T03:50:25.590791+0000","last_clean":"2024-09-17T03:50:25.590791+0000","last_became_active":"2024-09-17T03:50:23.134136+0000","last_became_peered":"2024-09-17T03:50:23.134136+0000","last_unstale":"2024-09-17T03:50:25.590791+0000","last_undegraded":"2024-09-17T03:50:25.590791+0000","last_fullsized":"2024-09-17T03:50:25.590791+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:06.997836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136713+0000","last_change":"2024-09-17T03:50:24.153900+0000","last_active":"2024-09-17T03:50:25.136713+0000","last_peered":"2024-09-17T03:50:25.136713+0000","last_clean":"2024-09-17T03:50:25.136713+0000","last_became_active":"2024-09-17T03:50:24.153608+0000","last_became_peered":"2024-09-17T03:50:24.153608+0000","last_unstale":"2024-09-17T03:50:25.136713+0000","last_undegraded":"2024-09-17T03:50:25.136713+0000","last_fullsized":"2024-09-17T03:50:25.136713+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135221+0000","last_change":"2024-09-17T03:50:23.142748+0000","last_active":"2024-09-17T03:50:25.135221+0000","last_peered":"2024-09-17T03:50:25.135221+0000","last_clean":"2024-09-17T03:50:25.135221+0000","last_became_active":"2024-09-17T03:50:23.142402+0000","last_became_peered":"2024-09-17T03:50:23.142402+0000","last_unstale":"2024-09-17T03:50:25.135221+0000","last_undegraded":"2024-09-17T03:50:25.135221+0000","last_fullsized":"2024-09-17T03:50:25.135221+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:26.214882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590470+0000","last_change":"2024-09-17T03:50:24.143306+0000","last_active":"2024-09-17T03:50:25.590470+0000","last_peered":"2024-09-17T03:50:25.590470+0000","last_clean":"2024-09-17T03:50:25.590470+0000","last_became_active":"2024-09-17T03:50:24.143003+0000","last_became_peered":"2024-09-17T03:50:24.143003+0000","last_unstale":"2024-09-17T03:50:25.590470+0000","last_undegraded":"2024-09-17T03:50:25.590470+0000","last_fullsized":"2024-09-17T03:50:25.590470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:10.001434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135059+0000","last_change":"2024-09-17T03:50:23.144925+0000","last_active":"2024-09-17T03:50:25.135059+0000","last_peered":"2024-09-17T03:50:25.135059+0000","last_clean":"2024-09-17T03:50:25.135059+0000","last_became_active":"2024-09-17T03:50:23.144566+0000","last_became_peered":"2024-09-17T03:50:23.144566+0000","last_unstale":"2024-09-17T03:50:25.135059+0000","last_undegraded":"2024-09-17T03:50:25.135059+0000","last_fullsized":"2024-09-17T03:50:25.135059+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:11.013666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134524+0000","last_change":"2024-09-17T03:50:24.151450+0000","last_active":"2024-09-17T03:50:25.134524+0000","last_peered":"2024-09-17T03:50:25.134524+0000","last_clean":"2024-09-17T03:50:25.134524+0000","last_became_active":"2024-09-17T03:50:24.151115+0000","last_became_peered":"2024-09-17T03:50:24.151115+0000","last_unstale":"2024-09-17T03:50:25.134524+0000","last_undegraded":"2024-09-17T03:50:25.134524+0000","last_fullsized":"2024-09-17T03:50:25.134524+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:08.183484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133040+0000","last_change":"2024-09-17T03:50:23.133927+0000","last_active":"2024-09-17T03:50:25.133040+0000","last_peered":"2024-09-17T03:50:25.133040+0000","last_clean":"2024-09-17T03:50:25.133040+0000","last_became_active":"2024-09-17T03:50:23.133670+0000","last_became_peered":"2024-09-17T03:50:23.133670+0000","last_unstale":"2024-09-17T03:50:25.133040+0000","last_undegraded":"2024-09-17T03:50:25.133040+0000","last_fullsized":"2024-09-17T03:50:25.133040+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590144+0000","last_change":"2024-09-17T03:50:24.151434+0000","last_active":"2024-09-17T03:50:25.590144+0000","last_peered":"2024-09-17T03:50:25.590144+0000","last_clean":"2024-09-17T03:50:25.590144+0000","last_became_active":"2024-09-17T03:50:24.149266+0000","last_became_peered":"2024-09-17T03:50:24.149266+0000","last_unstale":"2024-09-17T03:50:25.590144+0000","last_undegraded":"2024-09-17T03:50:25.590144+0000","last_fullsized":"2024-09-17T03:50:25.590144+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:19.342464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134650+0000","last_change":"2024-09-17T03:50:23.143447+0000","last_active":"2024-09-17T03:50:25.134650+0000","last_peered":"2024-09-17T03:50:25.134650+0000","last_clean":"2024-09-17T03:50:25.134650+0000","last_became_active":"2024-09-17T03:50:23.143053+0000","last_became_peered":"2024-09-17T03:50:23.143053+0000","last_unstale":"2024-09-17T03:50:25.134650+0000","last_undegraded":"2024-09-17T03:50:25.134650+0000","last_fullsized":"2024-09-17T03:50:25.134650+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:45.122730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590536+0000","last_change":"2024-09-17T03:50:24.138823+0000","last_active":"2024-09-17T03:50:25.590536+0000","last_peered":"2024-09-17T03:50:25.590536+0000","last_clean":"2024-09-17T03:50:25.590536+0000","last_became_active":"2024-09-17T03:50:24.138293+0000","last_became_peered":"2024-09-17T03:50:24.138293+0000","last_unstale":"2024-09-17T03:50:25.590536+0000","last_undegraded":"2024-09-17T03:50:25.590536+0000","last_fullsized":"2024-09-17T03:50:25.590536+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:17.408589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133970+0000","last_change":"2024-09-17T03:50:23.139166+0000","last_active":"2024-09-17T03:50:25.133970+0000","last_peered":"2024-09-17T03:50:25.133970+0000","last_clean":"2024-09-17T03:50:25.133970+0000","last_became_active":"2024-09-17T03:50:23.138826+0000","last_became_peered":"2024-09-17T03:50:23.138826+0000","last_unstale":"2024-09-17T03:50:25.133970+0000","last_undegraded":"2024-09-17T03:50:25.133970+0000","last_fullsized":"2024-09-17T03:50:25.133970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:07.180415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133867+0000","last_change":"2024-09-17T03:50:24.157448+0000","last_active":"2024-09-17T03:50:25.133867+0000","last_peered":"2024-09-17T03:50:25.133867+0000","last_clean":"2024-09-17T03:50:25.133867+0000","last_became_active":"2024-09-17T03:50:24.156599+0000","last_became_peered":"2024-09-17T03:50:24.156599+0000","last_unstale":"2024-09-17T03:50:25.133867+0000","last_undegraded":"2024-09-17T03:50:25.133867+0000","last_fullsized":"2024-09-17T03:50:25.133867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:38.071027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590944+0000","last_change":"2024-09-17T03:50:23.122274+0000","last_active":"2024-09-17T03:50:25.590944+0000","last_peered":"2024-09-17T03:50:25.590944+0000","last_clean":"2024-09-17T03:50:25.590944+0000","last_became_active":"2024-09-17T03:50:23.121928+0000","last_became_peered":"2024-09-17T03:50:23.121928+0000","last_unstale":"2024-09-17T03:50:25.590944+0000","last_undegraded":"2024-09-17T03:50:25.590944+0000","last_fullsized":"2024-09-17T03:50:25.590944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:36.626649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133566+0000","last_change":"2024-09-17T03:50:24.138004+0000","last_active":"2024-09-17T03:50:25.133566+0000","last_peered":"2024-09-17T03:50:25.133566+0000","last_clean":"2024-09-17T03:50:25.133566+0000","last_became_active":"2024-09-17T03:50:24.137679+0000","last_became_peered":"2024-09-17T03:50:24.137679+0000","last_unstale":"2024-09-17T03:50:25.133566+0000","last_undegraded":"2024-09-17T03:50:25.133566+0000","last_fullsized":"2024-09-17T03:50:25.133566+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:24.583890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590960+0000","last_change":"2024-09-17T03:50:23.133114+0000","last_active":"2024-09-17T03:50:25.590960+0000","last_peered":"2024-09-17T03:50:25.590960+0000","last_clean":"2024-09-17T03:50:25.590960+0000","last_became_active":"2024-09-17T03:50:23.132725+0000","last_became_peered":"2024-09-17T03:50:23.132725+0000","last_unstale":"2024-09-17T03:50:25.590960+0000","last_undegraded":"2024-09-17T03:50:25.590960+0000","last_fullsized":"2024-09-17T03:50:25.590960+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:09.940617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590361+0000","last_change":"2024-09-17T03:50:24.145715+0000","last_active":"2024-09-17T03:50:25.590361+0000","last_peered":"2024-09-17T03:50:25.590361+0000","last_clean":"2024-09-17T03:50:25.590361+0000","last_became_active":"2024-09-17T03:50:24.145392+0000","last_became_peered":"2024-09-17T03:50:24.145392+0000","last_unstale":"2024-09-17T03:50:25.590361+0000","last_undegraded":"2024-09-17T03:50:25.590361+0000","last_fullsized":"2024-09-17T03:50:25.590361+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":21,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133600+0000","last_change":"2024-09-17T03:50:23.140412+0000","last_active":"2024-09-17T03:50:25.133600+0000","last_peered":"2024-09-17T03:50:25.133600+0000","last_clean":"2024-09-17T03:50:25.133600+0000","last_became_active":"2024-09-17T03:50:23.139391+0000","last_became_peered":"2024-09-17T03:50:23.139391+0000","last_unstale":"2024-09-17T03:50:25.133600+0000","last_undegraded":"2024-09-17T03:50:25.133600+0000","last_fullsized":"2024-09-17T03:50:25.133600+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590621+0000","last_change":"2024-09-17T03:50:24.149315+0000","last_active":"2024-09-17T03:50:25.590621+0000","last_peered":"2024-09-17T03:50:25.590621+0000","last_clean":"2024-09-17T03:50:25.590621+0000","last_became_active":"2024-09-17T03:50:24.147345+0000","last_became_peered":"2024-09-17T03:50:24.147345+0000","last_unstale":"2024-09-17T03:50:25.590621+0000","last_undegraded":"2024-09-17T03:50:25.590621+0000","last_fullsized":"2024-09-17T03:50:25.590621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:00.619400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135266+0000","last_change":"2024-09-17T03:50:23.143154+0000","last_active":"2024-09-17T03:50:25.135266+0000","last_peered":"2024-09-17T03:50:25.135266+0000","last_clean":"2024-09-17T03:50:25.135266+0000","last_became_active":"2024-09-17T03:50:23.142801+0000","last_became_peered":"2024-09-17T03:50:23.142801+0000","last_unstale":"2024-09-17T03:50:25.135266+0000","last_undegraded":"2024-09-17T03:50:25.135266+0000","last_fullsized":"2024-09-17T03:50:25.135266+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:17.669331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133412+0000","last_change":"2024-09-17T03:50:24.147109+0000","last_active":"2024-09-17T03:50:25.133412+0000","last_peered":"2024-09-17T03:50:25.133412+0000","last_clean":"2024-09-17T03:50:25.133412+0000","last_became_active":"2024-09-17T03:50:24.146394+0000","last_became_peered":"2024-09-17T03:50:24.146394+0000","last_unstale":"2024-09-17T03:50:25.133412+0000","last_undegraded":"2024-09-17T03:50:25.133412+0000","last_fullsized":"2024-09-17T03:50:25.133412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:34.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590683+0000","last_change":"2024-09-17T03:50:23.135165+0000","last_active":"2024-09-17T03:50:25.590683+0000","last_peered":"2024-09-17T03:50:25.590683+0000","last_clean":"2024-09-17T03:50:25.590683+0000","last_became_active":"2024-09-17T03:50:23.134575+0000","last_became_peered":"2024-09-17T03:50:23.134575+0000","last_unstale":"2024-09-17T03:50:25.590683+0000","last_undegraded":"2024-09-17T03:50:25.590683+0000","last_fullsized":"2024-09-17T03:50:25.590683+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:53.899794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135744+0000","last_change":"2024-09-17T03:50:24.155803+0000","last_active":"2024-09-17T03:50:25.135744+0000","last_peered":"2024-09-17T03:50:25.135744+0000","last_clean":"2024-09-17T03:50:25.135744+0000","last_became_active":"2024-09-17T03:50:24.154916+0000","last_became_peered":"2024-09-17T03:50:24.154916+0000","last_unstale":"2024-09-17T03:50:25.135744+0000","last_undegraded":"2024-09-17T03:50:25.135744+0000","last_fullsized":"2024-09-17T03:50:25.135744+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:27.193772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133126+0000","last_change":"2024-09-17T03:50:23.127502+0000","last_active":"2024-09-17T03:50:25.133126+0000","last_peered":"2024-09-17T03:50:25.133126+0000","last_clean":"2024-09-17T03:50:25.133126+0000","last_became_active":"2024-09-17T03:50:23.126531+0000","last_became_peered":"2024-09-17T03:50:23.126531+0000","last_unstale":"2024-09-17T03:50:25.133126+0000","last_undegraded":"2024-09-17T03:50:25.133126+0000","last_fullsized":"2024-09-17T03:50:25.133126+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:39.251639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590222+0000","last_change":"2024-09-17T03:50:24.148579+0000","last_active":"2024-09-17T03:50:25.590222+0000","last_peered":"2024-09-17T03:50:25.590222+0000","last_clean":"2024-09-17T03:50:25.590222+0000","last_became_active":"2024-09-17T03:50:24.148272+0000","last_became_peered":"2024-09-17T03:50:24.148272+0000","last_unstale":"2024-09-17T03:50:25.590222+0000","last_undegraded":"2024-09-17T03:50:25.590222+0000","last_fullsized":"2024-09-17T03:50:25.590222+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:35:08.398615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":23,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133060+0000","last_change":"2024-09-17T03:50:23.139904+0000","last_active":"2024-09-17T03:50:25.133060+0000","last_peered":"2024-09-17T03:50:25.133060+0000","last_clean":"2024-09-17T03:50:25.133060+0000","last_became_active":"2024-09-17T03:50:23.139601+0000","last_became_peered":"2024-09-17T03:50:23.139601+0000","last_unstale":"2024-09-17T03:50:25.133060+0000","last_undegraded":"2024-09-17T03:50:25.133060+0000","last_fullsized":"2024-09-17T03:50:25.133060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:21.747727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136782+0000","last_change":"2024-09-17T03:50:24.150202+0000","last_active":"2024-09-17T03:50:25.136782+0000","last_peered":"2024-09-17T03:50:25.136782+0000","last_clean":"2024-09-17T03:50:25.136782+0000","last_became_active":"2024-09-17T03:50:24.149551+0000","last_became_peered":"2024-09-17T03:50:24.149551+0000","last_unstale":"2024-09-17T03:50:25.136782+0000","last_undegraded":"2024-09-17T03:50:25.136782+0000","last_fullsized":"2024-09-17T03:50:25.136782+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.577196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590829+0000","last_change":"2024-09-17T03:50:23.140181+0000","last_active":"2024-09-17T03:50:25.590829+0000","last_peered":"2024-09-17T03:50:25.590829+0000","last_clean":"2024-09-17T03:50:25.590829+0000","last_became_active":"2024-09-17T03:50:23.139817+0000","last_became_peered":"2024-09-17T03:50:23.139817+0000","last_unstale":"2024-09-17T03:50:25.590829+0000","last_undegraded":"2024-09-17T03:50:25.590829+0000","last_fullsized":"2024-09-17T03:50:25.590829+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:10.046559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590763+0000","last_change":"2024-09-17T03:50:24.135542+0000","last_active":"2024-09-17T03:50:25.590763+0000","last_peered":"2024-09-17T03:50:25.590763+0000","last_clean":"2024-09-17T03:50:25.590763+0000","last_became_active":"2024-09-17T03:50:24.135241+0000","last_became_peered":"2024-09-17T03:50:24.135241+0000","last_unstale":"2024-09-17T03:50:25.590763+0000","last_undegraded":"2024-09-17T03:50:25.590763+0000","last_fullsized":"2024-09-17T03:50:25.590763+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:34.592422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136769+0000","last_change":"2024-09-17T03:50:23.139663+0000","last_active":"2024-09-17T03:50:25.136769+0000","last_peered":"2024-09-17T03:50:25.136769+0000","last_clean":"2024-09-17T03:50:25.136769+0000","last_became_active":"2024-09-17T03:50:23.139294+0000","last_became_peered":"2024-09-17T03:50:23.139294+0000","last_unstale":"2024-09-17T03:50:25.136769+0000","last_undegraded":"2024-09-17T03:50:25.136769+0000","last_fullsized":"2024-09-17T03:50:25.136769+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:53.179574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590670+0000","last_change":"2024-09-17T03:50:24.155061+0000","last_active":"2024-09-17T03:50:25.590670+0000","last_peered":"2024-09-17T03:50:25.590670+0000","last_clean":"2024-09-17T03:50:25.590670+0000","last_became_active":"2024-09-17T03:50:24.154794+0000","last_became_peered":"2024-09-17T03:50:24.154794+0000","last_unstale":"2024-09-17T03:50:25.590670+0000","last_undegraded":"2024-09-17T03:50:25.590670+0000","last_fullsized":"2024-09-17T03:50:25.590670+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135372+0000","last_change":"2024-09-17T03:50:23.143762+0000","last_active":"2024-09-17T03:50:25.135372+0000","last_peered":"2024-09-17T03:50:25.135372+0000","last_clean":"2024-09-17T03:50:25.135372+0000","last_became_active":"2024-09-17T03:50:23.143435+0000","last_became_peered":"2024-09-17T03:50:23.143435+0000","last_unstale":"2024-09-17T03:50:25.135372+0000","last_undegraded":"2024-09-17T03:50:25.135372+0000","last_fullsized":"2024-09-17T03:50:25.135372+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:52.817756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590480+0000","last_change":"2024-09-17T03:50:24.138195+0000","last_active":"2024-09-17T03:50:25.590480+0000","last_peered":"2024-09-17T03:50:25.590480+0000","last_clean":"2024-09-17T03:50:25.590480+0000","last_became_active":"2024-09-17T03:50:24.137835+0000","last_became_peered":"2024-09-17T03:50:24.137835+0000","last_unstale":"2024-09-17T03:50:25.590480+0000","last_undegraded":"2024-09-17T03:50:25.590480+0000","last_fullsized":"2024-09-17T03:50:25.590480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:45.999114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135343+0000","last_change":"2024-09-17T03:50:23.140276+0000","last_active":"2024-09-17T03:50:25.135343+0000","last_peered":"2024-09-17T03:50:25.135343+0000","last_clean":"2024-09-17T03:50:25.135343+0000","last_became_active":"2024-09-17T03:50:23.139757+0000","last_became_peered":"2024-09-17T03:50:23.139757+0000","last_unstale":"2024-09-17T03:50:25.135343+0000","last_undegraded":"2024-09-17T03:50:25.135343+0000","last_fullsized":"2024-09-17T03:50:25.135343+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:14:45.951042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591157+0000","last_change":"2024-09-17T03:50:24.152597+0000","last_active":"2024-09-17T03:50:25.591157+0000","last_peered":"2024-09-17T03:50:25.591157+0000","last_clean":"2024-09-17T03:50:25.591157+0000","last_became_active":"2024-09-17T03:50:24.152307+0000","last_became_peered":"2024-09-17T03:50:24.152307+0000","last_unstale":"2024-09-17T03:50:25.591157+0000","last_undegraded":"2024-09-17T03:50:25.591157+0000","last_fullsized":"2024-09-17T03:50:25.591157+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:38.869025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589948+0000","last_change":"2024-09-17T03:50:23.130742+0000","last_active":"2024-09-17T03:50:25.589948+0000","last_peered":"2024-09-17T03:50:25.589948+0000","last_clean":"2024-09-17T03:50:25.589948+0000","last_became_active":"2024-09-17T03:50:23.130434+0000","last_became_peered":"2024-09-17T03:50:23.130434+0000","last_unstale":"2024-09-17T03:50:25.589948+0000","last_undegraded":"2024-09-17T03:50:25.589948+0000","last_fullsized":"2024-09-17T03:50:25.589948+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:13.017254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591534+0000","last_change":"2024-09-17T03:50:24.141883+0000","last_active":"2024-09-17T03:50:25.591534+0000","last_peered":"2024-09-17T03:50:25.591534+0000","last_clean":"2024-09-17T03:50:25.591534+0000","last_became_active":"2024-09-17T03:50:24.141572+0000","last_became_peered":"2024-09-17T03:50:24.141572+0000","last_unstale":"2024-09-17T03:50:25.591534+0000","last_undegraded":"2024-09-17T03:50:25.591534+0000","last_fullsized":"2024-09-17T03:50:25.591534+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:11.049628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591155+0000","last_change":"2024-09-17T03:50:23.133695+0000","last_active":"2024-09-17T03:50:25.591155+0000","last_peered":"2024-09-17T03:50:25.591155+0000","last_clean":"2024-09-17T03:50:25.591155+0000","last_became_active":"2024-09-17T03:50:23.133125+0000","last_became_peered":"2024-09-17T03:50:23.133125+0000","last_unstale":"2024-09-17T03:50:25.591155+0000","last_undegraded":"2024-09-17T03:50:25.591155+0000","last_fullsized":"2024-09-17T03:50:25.591155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:51.417524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589870+0000","last_change":"2024-09-17T03:50:24.141825+0000","last_active":"2024-09-17T03:50:25.589870+0000","last_peered":"2024-09-17T03:50:25.589870+0000","last_clean":"2024-09-17T03:50:25.589870+0000","last_became_active":"2024-09-17T03:50:24.141522+0000","last_became_peered":"2024-09-17T03:50:24.141522+0000","last_unstale":"2024-09-17T03:50:25.589870+0000","last_undegraded":"2024-09-17T03:50:25.589870+0000","last_fullsized":"2024-09-17T03:50:25.589870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:20.207429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134808+0000","last_change":"2024-09-17T03:50:23.141992+0000","last_active":"2024-09-17T03:50:25.134808+0000","last_peered":"2024-09-17T03:50:25.134808+0000","last_clean":"2024-09-17T03:50:25.134808+0000","last_became_active":"2024-09-17T03:50:23.141642+0000","last_became_peered":"2024-09-17T03:50:23.141642+0000","last_unstale":"2024-09-17T03:50:25.134808+0000","last_undegraded":"2024-09-17T03:50:25.134808+0000","last_fullsized":"2024-09-17T03:50:25.134808+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:12.799906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.135846+0000","last_change":"2024-09-17T03:50:24.142339+0000","last_active":"2024-09-17T03:50:25.135846+0000","last_peered":"2024-09-17T03:50:25.135846+0000","last_clean":"2024-09-17T03:50:25.135846+0000","last_became_active":"2024-09-17T03:50:24.142028+0000","last_became_peered":"2024-09-17T03:50:24.142028+0000","last_unstale":"2024-09-17T03:50:25.135846+0000","last_undegraded":"2024-09-17T03:50:25.135846+0000","last_fullsized":"2024-09-17T03:50:25.135846+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:38.934187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590076+0000","last_change":"2024-09-17T03:50:23.129908+0000","last_active":"2024-09-17T03:50:25.590076+0000","last_peered":"2024-09-17T03:50:25.590076+0000","last_clean":"2024-09-17T03:50:25.590076+0000","last_became_active":"2024-09-17T03:50:23.129543+0000","last_became_peered":"2024-09-17T03:50:23.129543+0000","last_unstale":"2024-09-17T03:50:25.590076+0000","last_undegraded":"2024-09-17T03:50:25.590076+0000","last_fullsized":"2024-09-17T03:50:25.590076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:12.041115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591135+0000","last_change":"2024-09-17T03:50:24.124588+0000","last_active":"2024-09-17T03:50:25.591135+0000","last_peered":"2024-09-17T03:50:25.591135+0000","last_clean":"2024-09-17T03:50:25.591135+0000","last_became_active":"2024-09-17T03:50:24.124178+0000","last_became_peered":"2024-09-17T03:50:24.124178+0000","last_unstale":"2024-09-17T03:50:25.591135+0000","last_undegraded":"2024-09-17T03:50:25.591135+0000","last_fullsized":"2024-09-17T03:50:25.591135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:41.718876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590917+0000","last_change":"2024-09-17T03:50:23.137622+0000","last_active":"2024-09-17T03:50:25.590917+0000","last_peered":"2024-09-17T03:50:25.590917+0000","last_clean":"2024-09-17T03:50:25.590917+0000","last_became_active":"2024-09-17T03:50:23.137258+0000","last_became_peered":"2024-09-17T03:50:23.137258+0000","last_unstale":"2024-09-17T03:50:25.590917+0000","last_undegraded":"2024-09-17T03:50:25.590917+0000","last_fullsized":"2024-09-17T03:50:25.590917+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:40.356807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590857+0000","last_change":"2024-09-17T03:50:24.140577+0000","last_active":"2024-09-17T03:50:25.590857+0000","last_peered":"2024-09-17T03:50:25.590857+0000","last_clean":"2024-09-17T03:50:25.590857+0000","last_became_active":"2024-09-17T03:50:24.140330+0000","last_became_peered":"2024-09-17T03:50:24.140330+0000","last_unstale":"2024-09-17T03:50:25.590857+0000","last_undegraded":"2024-09-17T03:50:25.590857+0000","last_fullsized":"2024-09-17T03:50:25.590857+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:00.638143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.591009+0000","last_change":"2024-09-17T03:50:23.130628+0000","last_active":"2024-09-17T03:50:25.591009+0000","last_peered":"2024-09-17T03:50:25.591009+0000","last_clean":"2024-09-17T03:50:25.591009+0000","last_became_active":"2024-09-17T03:50:23.130307+0000","last_became_peered":"2024-09-17T03:50:23.130307+0000","last_unstale":"2024-09-17T03:50:25.591009+0000","last_undegraded":"2024-09-17T03:50:25.591009+0000","last_fullsized":"2024-09-17T03:50:25.591009+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:26.309251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589837+0000","last_change":"2024-09-17T03:50:24.151860+0000","last_active":"2024-09-17T03:50:25.589837+0000","last_peered":"2024-09-17T03:50:25.589837+0000","last_clean":"2024-09-17T03:50:25.589837+0000","last_became_active":"2024-09-17T03:50:24.150602+0000","last_became_peered":"2024-09-17T03:50:24.150602+0000","last_unstale":"2024-09-17T03:50:25.589837+0000","last_undegraded":"2024-09-17T03:50:25.589837+0000","last_fullsized":"2024-09-17T03:50:25.589837+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:13.191061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590934+0000","last_change":"2024-09-17T03:50:23.136531+0000","last_active":"2024-09-17T03:50:25.590934+0000","last_peered":"2024-09-17T03:50:25.590934+0000","last_clean":"2024-09-17T03:50:25.590934+0000","last_became_active":"2024-09-17T03:50:23.136207+0000","last_became_peered":"2024-09-17T03:50:23.136207+0000","last_unstale":"2024-09-17T03:50:25.590934+0000","last_undegraded":"2024-09-17T03:50:25.590934+0000","last_fullsized":"2024-09-17T03:50:25.590934+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:47.338444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590966+0000","last_change":"2024-09-17T03:50:24.140447+0000","last_active":"2024-09-17T03:50:25.590966+0000","last_peered":"2024-09-17T03:50:25.590966+0000","last_clean":"2024-09-17T03:50:25.590966+0000","last_became_active":"2024-09-17T03:50:24.140095+0000","last_became_peered":"2024-09-17T03:50:24.140095+0000","last_unstale":"2024-09-17T03:50:25.590966+0000","last_undegraded":"2024-09-17T03:50:25.590966+0000","last_fullsized":"2024-09-17T03:50:25.590966+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:42:53.379705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590069+0000","last_change":"2024-09-17T03:50:23.124066+0000","last_active":"2024-09-17T03:50:25.590069+0000","last_peered":"2024-09-17T03:50:25.590069+0000","last_clean":"2024-09-17T03:50:25.590069+0000","last_became_active":"2024-09-17T03:50:23.123776+0000","last_became_peered":"2024-09-17T03:50:23.123776+0000","last_unstale":"2024-09-17T03:50:25.590069+0000","last_undegraded":"2024-09-17T03:50:25.590069+0000","last_fullsized":"2024-09-17T03:50:25.590069+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:39.566116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.589894+0000","last_change":"2024-09-17T03:50:24.132547+0000","last_active":"2024-09-17T03:50:25.589894+0000","last_peered":"2024-09-17T03:50:25.589894+0000","last_clean":"2024-09-17T03:50:25.589894+0000","last_became_active":"2024-09-17T03:50:24.132193+0000","last_became_peered":"2024-09-17T03:50:24.132193+0000","last_unstale":"2024-09-17T03:50:25.589894+0000","last_undegraded":"2024-09-17T03:50:25.589894+0000","last_fullsized":"2024-09-17T03:50:25.589894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:28.964951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590804+0000","last_change":"2024-09-17T03:50:23.136967+0000","last_active":"2024-09-17T03:50:25.590804+0000","last_peered":"2024-09-17T03:50:25.590804+0000","last_clean":"2024-09-17T03:50:25.590804+0000","last_became_active":"2024-09-17T03:50:23.136643+0000","last_became_peered":"2024-09-17T03:50:23.136643+0000","last_unstale":"2024-09-17T03:50:25.590804+0000","last_undegraded":"2024-09-17T03:50:25.590804+0000","last_fullsized":"2024-09-17T03:50:25.590804+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:29:58.269661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133036+0000","last_change":"2024-09-17T03:50:24.150858+0000","last_active":"2024-09-17T03:50:25.133036+0000","last_peered":"2024-09-17T03:50:25.133036+0000","last_clean":"2024-09-17T03:50:25.133036+0000","last_became_active":"2024-09-17T03:50:24.150509+0000","last_became_peered":"2024-09-17T03:50:24.150509+0000","last_unstale":"2024-09-17T03:50:25.133036+0000","last_undegraded":"2024-09-17T03:50:25.133036+0000","last_fullsized":"2024-09-17T03:50:25.133036+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:49.530281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134927+0000","last_change":"2024-09-17T03:50:23.141022+0000","last_active":"2024-09-17T03:50:25.134927+0000","last_peered":"2024-09-17T03:50:25.134927+0000","last_clean":"2024-09-17T03:50:25.134927+0000","last_became_active":"2024-09-17T03:50:23.140264+0000","last_became_peered":"2024-09-17T03:50:23.140264+0000","last_unstale":"2024-09-17T03:50:25.134927+0000","last_undegraded":"2024-09-17T03:50:25.134927+0000","last_fullsized":"2024-09-17T03:50:25.134927+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:55.135377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.134615+0000","last_change":"2024-09-17T03:50:24.149913+0000","last_active":"2024-09-17T03:50:25.134615+0000","last_peered":"2024-09-17T03:50:25.134615+0000","last_clean":"2024-09-17T03:50:25.134615+0000","last_became_active":"2024-09-17T03:50:24.149591+0000","last_became_peered":"2024-09-17T03:50:24.149591+0000","last_unstale":"2024-09-17T03:50:25.134615+0000","last_undegraded":"2024-09-17T03:50:25.134615+0000","last_fullsized":"2024-09-17T03:50:25.134615+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:22.750796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.133185+0000","last_change":"2024-09-17T03:50:23.130536+0000","last_active":"2024-09-17T03:50:25.133185+0000","last_peered":"2024-09-17T03:50:25.133185+0000","last_clean":"2024-09-17T03:50:25.133185+0000","last_became_active":"2024-09-17T03:50:23.130176+0000","last_became_peered":"2024-09-17T03:50:23.130176+0000","last_unstale":"2024-09-17T03:50:25.133185+0000","last_undegraded":"2024-09-17T03:50:25.133185+0000","last_fullsized":"2024-09-17T03:50:25.133185+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:26.703578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.137102+0000","last_change":"2024-09-17T03:50:24.146715+0000","last_active":"2024-09-17T03:50:25.137102+0000","last_peered":"2024-09-17T03:50:25.137102+0000","last_clean":"2024-09-17T03:50:25.137102+0000","last_became_active":"2024-09-17T03:50:24.146329+0000","last_became_peered":"2024-09-17T03:50:24.146329+0000","last_unstale":"2024-09-17T03:50:25.137102+0000","last_undegraded":"2024-09-17T03:50:25.137102+0000","last_fullsized":"2024-09-17T03:50:25.137102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:23.104360+0000","last_clean_scrub_stamp":"2024-09-17T03:50:23.104360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:15.559038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.590037+0000","last_change":"2024-09-17T03:50:23.138899+0000","last_active":"2024-09-17T03:50:25.590037+0000","last_peered":"2024-09-17T03:50:25.590037+0000","last_clean":"2024-09-17T03:50:25.590037+0000","last_became_active":"2024-09-17T03:50:23.138546+0000","last_became_peered":"2024-09-17T03:50:23.138546+0000","last_unstale":"2024-09-17T03:50:25.590037+0000","last_undegraded":"2024-09-17T03:50:25.590037+0000","last_fullsized":"2024-09-17T03:50:25.590037+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:50:22.101290+0000","last_clean_scrub_stamp":"2024-09-17T03:50:22.101290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:49.030426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":130,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-17T03:50:25.136109+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:50:25.136109+0000","last_peered":"2024-09-17T03:50:25.136109+0000","last_clean":"2024-09-17T03:50:25.136109+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:50:25.136109+0000","last_undegraded":"2024-09-17T03:50:25.136109+0000","last_fullsized":"2024-09-17T03:50:25.136109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":962560,"data_stored":730824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509458,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":372736,"data_stored":141038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":372736,"data_stored":140470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":380928,"data_stored":142000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":368640,"data_stored":140436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":380928,"data_stored":142000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542159,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":372736,"data_stored":140982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542160,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":958464,"data_stored":730734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:50:33.976 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:50:33.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 -- 172.21.15.129:0/1234069244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488073170 msgr2=0x7f148815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 --2- 172.21.15.129:0/1234069244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488073170 0x7f148815c780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1478009830 tx=0x7f147802a520 comp rx=0 tx=0).stop 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 -- 172.21.15.129:0/1234069244 shutdown_connections 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 --2- 172.21.15.129:0/1234069244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070440 0x7f14880b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 --2- 172.21.15.129:0/1234069244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488073170 0x7f148815c780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.243+0000 7f148ee00640 1 -- 172.21.15.129:0/1234069244 >> 172.21.15.129:0/1234069244 conn(0x7f14880b1cb0 msgr2=0x7f14880b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.244+0000 7f148ee00640 1 -- 172.21.15.129:0/1234069244 shutdown_connections 2024-09-17T03:50:34.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.244+0000 7f148ee00640 1 -- 172.21.15.129:0/1234069244 wait complete. 2024-09-17T03:50:34.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.246+0000 7f148ee00640 1 Processor -- start 2024-09-17T03:50:34.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.246+0000 7f148ee00640 1 -- start start 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070030 0x7f148815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488070440 0x7f14880cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14880b3b00 con 0x7f1488073170 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14880b3980 con 0x7f1488070440 2024-09-17T03:50:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.247+0000 7f148ee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14880b3c80 con 0x7f1488070030 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42910/0 (socket says 172.21.15.129:42910) 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 -- 172.21.15.129:0/359806289 learned_addr learned my addr 172.21.15.129:0/359806289 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f147f400640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488070440 0x7f14880cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f1487e00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070030 0x7f148815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 -- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070030 msgr2=0x7f148815a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070030 0x7f148815a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 -- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488070440 msgr2=0x7f14880cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488070440 0x7f14880cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 -- 172.21.15.129:0/359806289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14881ec010 con 0x7f1488073170 2024-09-17T03:50:34.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.248+0000 7f148ce00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f148000c3a0 tx=0x7f148000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.249+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480003970 con 0x7f1488073170 2024-09-17T03:50:34.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.249+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1480003c00 con 0x7f1488073170 2024-09-17T03:50:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.249+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f148809bc00 con 0x7f1488073170 2024-09-17T03:50:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.249+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480004020 con 0x7f1488073170 2024-09-17T03:50:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.249+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14881ec2f0 con 0x7f1488073170 2024-09-17T03:50:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.250+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14880b4130 con 0x7f1488073170 2024-09-17T03:50:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.251+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14800041c0 con 0x7f1488073170 2024-09-17T03:50:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.252+0000 7f1484c00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 0x7f146407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.252+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f148009b570 con 0x7f1488073170 2024-09-17T03:50:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.253+0000 7f1487e00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 0x7f146407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.253+0000 7f1487e00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 0x7f146407b0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1478034000 tx=0x7f1478004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:34.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.256+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1480063350 con 0x7f1488073170 2024-09-17T03:50:34.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.505+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f14880c3860 con 0x7f1488073170 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.509+0000 7f1484c00640 1 -- 172.21.15.129:0/359806289 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) ==== 190+0+147 (secure 0 0 0) 0x7f1480067d80 con 0x7f1488073170 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:50:34.508 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:50:34.509 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 msgr2=0x7f146407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 0x7f146407b0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1478034000 tx=0x7f1478004730 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 msgr2=0x7f14880cfb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f148000c3a0 tx=0x7f148000c860 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 shutdown_connections 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1464078bc0 0x7f146407b0d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1488073170 0x7f14880cfb70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1488070440 0x7f14880cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 --2- 172.21.15.129:0/359806289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1488070030 0x7f148815a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.512+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 >> 172.21.15.129:0/359806289 conn(0x7f14880b1cb0 msgr2=0x7f14880afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:34.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.513+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 shutdown_connections 2024-09-17T03:50:34.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.513+0000 7f148ee00640 1 -- 172.21.15.129:0/359806289 wait complete. 2024-09-17T03:50:34.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:34.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 -- 172.21.15.129:0/1375945048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c0ce800 msgr2=0x7fe93c15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 --2- 172.21.15.129:0/1375945048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c0ce800 0x7fe93c15cef0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe92c005fd0 tx=0x7fe92c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 -- 172.21.15.129:0/1375945048 shutdown_connections 2024-09-17T03:50:34.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 --2- 172.21.15.129:0/1375945048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c072370 0x7fe93c06cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 --2- 172.21.15.129:0/1375945048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c0ce800 0x7fe93c15cef0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 -- 172.21.15.129:0/1375945048 >> 172.21.15.129:0/1375945048 conn(0x7fe93c0cdb20 msgr2=0x7fe93c0cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:34.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 -- 172.21.15.129:0/1375945048 shutdown_connections 2024-09-17T03:50:34.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.754+0000 7fe93b200640 1 -- 172.21.15.129:0/1375945048 wait complete. 2024-09-17T03:50:34.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.756+0000 7fe93b200640 1 Processor -- start 2024-09-17T03:50:34.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.756+0000 7fe93b200640 1 -- start start 2024-09-17T03:50:34.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe93c072370 0x7fe93c15a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c0ce800 0x7fe93c1ea290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c153710 con 0x7fe93c15d430 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe93c153590 con 0x7fe93c072370 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe93c153890 con 0x7fe93c0ce800 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42936/0 (socket says 172.21.15.129:42936) 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe93a800640 1 -- 172.21.15.129:0/2110161733 learned_addr learned my addr 172.21.15.129:0/2110161733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:34.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe939e00640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe93c072370 0x7fe93c15a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.758+0000 7fe931400640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c0ce800 0x7fe93c1ea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 -- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c0ce800 msgr2=0x7fe93c1ea290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c0ce800 0x7fe93c1ea290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 -- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe93c072370 msgr2=0x7fe93c15a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe93c072370 0x7fe93c15a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 -- 172.21.15.129:0/2110161733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93c1f8f90 con 0x7fe93c15d430 2024-09-17T03:50:34.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93a800640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe93400c760 tx=0x7fe93400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:34.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe934003b70 con 0x7fe93c15d430 2024-09-17T03:50:34.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe934003d10 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe93c1f9270 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe93400aab0 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.759+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe93c1f9830 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.760+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe93c06cde0 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.761+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe93400ac50 con 0x7fe93c15d430 2024-09-17T03:50:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.762+0000 7fe932a00640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 0x7fe91807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:34.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.762+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe93409a8d0 con 0x7fe93c15d430 2024-09-17T03:50:34.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.763+0000 7fe939e00640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 0x7fe91807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:34.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.764+0000 7fe939e00640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 0x7fe91807b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe92c033000 tx=0x7fe92c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:34.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.767+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe9340626c0 con 0x7fe93c15d430 2024-09-17T03:50:34.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.986+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe93c15eea0 con 0x7fe93c15d430 2024-09-17T03:50:34.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.986+0000 7fe932a00640 1 -- 172.21.15.129:0/2110161733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7fe9340670f0 con 0x7fe93c15d430 2024-09-17T03:50:34.986 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:34.986 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:34.986 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 msgr2=0x7fe91807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 0x7fe91807b0a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe92c033000 tx=0x7fe92c004730 comp rx=0 tx=0).stop 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 msgr2=0x7fe93c1ea7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe93400c760 tx=0x7fe93400cc20 comp rx=0 tx=0).stop 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 shutdown_connections 2024-09-17T03:50:34.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe918078b90 0x7fe91807b0a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe93c15d430 0x7fe93c1ea7d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe93c0ce800 0x7fe93c1ea290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 --2- 172.21.15.129:0/2110161733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe93c072370 0x7fe93c15a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 >> 172.21.15.129:0/2110161733 conn(0x7fe93c0cdb20 msgr2=0x7fe93c15a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 shutdown_connections 2024-09-17T03:50:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:34.989+0000 7fe93b200640 1 -- 172.21.15.129:0/2110161733 wait complete. 2024-09-17T03:50:35.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:50:23.616751+0000', 'modified': '2024-09-17T03:50:26.618188+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4845}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4845': {'gid': 4845, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/1447431306', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 1447431306}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1447431306}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4845, 'qdb_cluster': [4845]} 2024-09-17T03:50:35.412 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:50:35.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T03:50:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 -- 172.21.15.129:0/1990929332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4073170 msgr2=0x7f36e415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 --2- 172.21.15.129:0/1990929332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4073170 0x7f36e415c780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009830 tx=0x7f36d802a410 comp rx=0 tx=0).stop 2024-09-17T03:50:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 -- 172.21.15.129:0/1990929332 shutdown_connections 2024-09-17T03:50:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 --2- 172.21.15.129:0/1990929332 >> v2:172.21.15.181:3300/0 conn(0x7f36e4070440 0x7f36e40b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 --2- 172.21.15.129:0/1990929332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4073170 0x7f36e415c780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 -- 172.21.15.129:0/1990929332 >> 172.21.15.129:0/1990929332 conn(0x7f36e40b1cb0 msgr2=0x7f36e40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 -- 172.21.15.129:0/1990929332 shutdown_connections 2024-09-17T03:50:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.636+0000 7f36eb600640 1 -- 172.21.15.129:0/1990929332 wait complete. 2024-09-17T03:50:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.638+0000 7f36eb600640 1 Processor -- start 2024-09-17T03:50:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.638+0000 7f36eb600640 1 -- start start 2024-09-17T03:50:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4070440 0x7f36e40d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e4073170 0x7f36e41f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36e40b3b00 con 0x7f36e4070030 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36e40b3980 con 0x7f36e4073170 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36eb600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36e40b3c80 con 0x7f36e4070440 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36e8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36e8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42950/0 (socket says 172.21.15.129:42950) 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36e8c00640 1 -- 172.21.15.129:0/2662909800 learned_addr learned my addr 172.21.15.129:0/2662909800 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36e9600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e4073170 0x7f36e41f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.640+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4070440 0x7f36e40d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 -- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4070440 msgr2=0x7f36e40d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4070440 0x7f36e40d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 -- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e4073170 msgr2=0x7f36e41f6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e4073170 0x7f36e41f6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 -- 172.21.15.129:0/2662909800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e40cf350 con 0x7f36e4070030 2024-09-17T03:50:35.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e8c00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009950 tx=0x7f36d8003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e0c00640 1 -- 172.21.15.129:0/2662909800 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d803f030 con 0x7f36e4070030 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e40cf630 con 0x7f36e4070030 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e0c00640 1 -- 172.21.15.129:0/2662909800 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36d8004510 con 0x7f36e4070030 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36e0c00640 1 -- 172.21.15.129:0/2662909800 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8037040 con 0x7f36e4070030 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.641+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e40cfbf0 con 0x7f36e4070030 2024-09-17T03:50:35.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.644+0000 7f36e0c00640 1 -- 172.21.15.129:0/2662909800 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f36d8042070 con 0x7f36e4070030 2024-09-17T03:50:35.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.645+0000 7f36e0c00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 0x7f36c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.645+0000 7f36e0c00640 1 -- 172.21.15.129:0/2662909800 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f36d80baa40 con 0x7f36e4070030 2024-09-17T03:50:35.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.645+0000 7f36f2600740 1 Processor -- start 2024-09-17T03:50:35.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.645+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 0x7f36c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.645+0000 7f36f2600740 1 -- start start 2024-09-17T03:50:35.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.646+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 0x7f36c407b0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36e40c3d00 tx=0x7f36d0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288eeb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288afa590 0x558288ebbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558288e45430 con 0x558288af23b0 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558288e355d0 con 0x558288afa590 2024-09-17T03:50:35.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558288e24a00 con 0x558288b1b270 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42960/0 (socket says 172.21.15.129:42960) 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 -- 172.21.15.129:0/2536302020 learned_addr learned my addr 172.21.15.129:0/2536302020 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e9600640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288afa590 0x558288ebbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e8c00640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288eeb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 -- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 msgr2=0x558288eeb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288eeb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 -- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288afa590 msgr2=0x558288ebbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288afa590 0x558288ebbf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e3e00640 1 -- 172.21.15.129:0/2536302020 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558288ded520 con 0x558288af23b0 2024-09-17T03:50:35.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.647+0000 7f36e8c00640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288eeb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.648+0000 7f36e3e00640 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f36e4071340 tx=0x7f36d0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2691110fe1aa87d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.648+0000 7f36d5200640 1 -- 172.21.15.129:0/2536302020 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0025070 con 0x558288af23b0 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.648+0000 7f36d5200640 1 -- 172.21.15.129:0/2536302020 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36d00159b0 con 0x558288af23b0 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.648+0000 7f36d5200640 1 -- 172.21.15.129:0/2536302020 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0015d60 con 0x558288af23b0 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 -- 172.21.15.129:0/2536302020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 msgr2=0x558288eebeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f36e4071340 tx=0x7f36d0011000 comp rx=0 tx=0).stop 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 -- 172.21.15.129:0/2536302020 shutdown_connections 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288afa590 0x558288ebbf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288eebeb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 --2- 172.21.15.129:0/2536302020 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288eeb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 -- 172.21.15.129:0/2536302020 >> 172.21.15.129:0/2536302020 conn(0x558288b0f3c0 msgr2=0x558288ec1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 -- 172.21.15.129:0/2536302020 shutdown_connections 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.649+0000 7f36f2600740 1 -- 172.21.15.129:0/2536302020 wait complete. 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.650+0000 7f36f2600740 1 -- start start 2024-09-17T03:50:35.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.650+0000 7f36f2600740 1 Processor -- start 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288ee2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f00d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558288dcb330 con 0x558288b1b270 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558288e28810 con 0x558288af23b0 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558288b95620 con 0x558288afa590 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36f2600740 10 client.0 _ll_register_callbacks cb 0x558288dd0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42970/0 (socket says 172.21.15.129:42970) 2024-09-17T03:50:35.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 -- 172.21.15.129:0/3951375622 learned_addr learned my addr 172.21.15.129:0/3951375622 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e8c00640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288ee2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e3e00640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f00d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 -- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 msgr2=0x558288f00d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f00d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 -- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 msgr2=0x558288ee2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288ee2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.652+0000 7f36e9600640 1 -- 172.21.15.129:0/3951375622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558288b86850 con 0x558288b1b270 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36e9600640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0129b0 tx=0x7f36dc012e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3954629fc0dd205b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc003970 con 0x558288b1b270 2024-09-17T03:50:35.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f36dc01db10 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36d00159b0 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f36dc01dd10 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 10 client.5016 fetch_fsmap learned FSMap version 17 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x558288bfd6b0 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36d00159b0 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7f36dc003b10 con 0x558288b1b270 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 10 client.5016 fetch_fsmap finished waiting for FSMap version 17 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 10 client.5016 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 20 client.5016 populate_metadata read hostname 'smithi129' 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288ad9e40 0x558288f16020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 4 client.5016 mds_command: new command op to 4845 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.653+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 --> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558288e37520 con 0x558288ad9e40 2024-09-17T03:50:35.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.654+0000 7f36e8c00640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288ad9e40 0x558288f16020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.654+0000 7f36c9600640 10 client.5016 ms_handle_connect on v2:172.21.15.129:6833/1447431306 2024-09-17T03:50:35.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.654+0000 7f36e8c00640 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288ad9e40 0x558288f16020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b396c44690079548 server_cookie=d43e2fbeac6bb2c3 in_seq=0 out_seq=0 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36c9600640 1 -- 172.21.15.129:0/3951375622 <== mds.0 v2:172.21.15.129:6833/1447431306 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558288e37520 con 0x558288ad9e40 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36c9600640 10 client.5016 handle_command_reply: tid=0 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 client.5016 shutdown 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 msgr2=0x558288f0ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0129b0 tx=0x7f36dc012e70 comp rx=0 tx=0).stop 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 shutdown_connections 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288ad9e40 0x558288f16020 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288b1b270 0x558288f0ae30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f00d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 --2- 172.21.15.129:0/3951375622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288ee2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36c9600640 0 client.5016 ms_handle_reset on v2:172.21.15.129:6833/1447431306 2024-09-17T03:50:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.655+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 >> 172.21.15.129:0/3951375622 conn(0x558288b0f3c0 msgr2=0x558288ebcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.656+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 shutdown_connections 2024-09-17T03:50:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.656+0000 7f36f2600740 1 -- 172.21.15.129:0/3951375622 wait complete. 2024-09-17T03:50:35.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.656+0000 7f36f2600740 20 client.5016 trim_cache size 0 max 16384 2024-09-17T03:50:35.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.674+0000 7f36f2600740 1 Processor -- start 2024-09-17T03:50:35.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.675+0000 7f36f2600740 1 -- start start 2024-09-17T03:50:35.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288f07e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288f29db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41fef00 con 0x558288afa590 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36e41f38e0 con 0x558288af23b0 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36e41f3f30 con 0x558288b1b270 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42978/0 (socket says 172.21.15.129:42978) 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 -- 172.21.15.129:0/1424321290 learned_addr learned my addr 172.21.15.129:0/1424321290 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 -- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 msgr2=0x558288f29db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288f29db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 -- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 msgr2=0x558288f07e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288f07e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.676+0000 7f36e3e00640 1 -- 172.21.15.129:0/1424321290 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558288ec2650 con 0x558288afa590 2024-09-17T03:50:35.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.677+0000 7f36e3e00640 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f36d0020000 tx=0x7f36d0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2932b1d6d9e59fc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.677+0000 7f36cbe00640 1 -- 172.21.15.129:0/1424321290 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d002f070 con 0x558288afa590 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.677+0000 7f36cbe00640 1 -- 172.21.15.129:0/1424321290 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36d0004040 con 0x558288afa590 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.677+0000 7f36cbe00640 1 -- 172.21.15.129:0/1424321290 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0025040 con 0x558288afa590 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 -- 172.21.15.129:0/1424321290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 msgr2=0x558288f29870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f36d0020000 tx=0x7f36d0000ed0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 -- 172.21.15.129:0/1424321290 shutdown_connections 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288b1b270 0x558288f29db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288afa590 0x558288f29870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 --2- 172.21.15.129:0/1424321290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288af23b0 0x558288f07e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 -- 172.21.15.129:0/1424321290 >> 172.21.15.129:0/1424321290 conn(0x558288b0f3c0 msgr2=0x558288ebcc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 -- 172.21.15.129:0/1424321290 shutdown_connections 2024-09-17T03:50:35.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.678+0000 7f36f2600740 1 -- 172.21.15.129:0/1424321290 wait complete. 2024-09-17T03:50:35.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.679+0000 7f36f2600740 1 -- start start 2024-09-17T03:50:35.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.679+0000 7f36f2600740 1 Processor -- start 2024-09-17T03:50:35.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.680+0000 7f36f2600740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.680+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f73800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.680+0000 7f36f2600740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 0x558288f8e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.680+0000 7f36f2600740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558288ee62e0 con 0x558288af23b0 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.680+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558288f0b620 con 0x558288b1b270 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36f2600740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558288f19480 con 0x558288afa590 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36f2600740 10 client.0 _ll_register_callbacks cb 0x7f36dc012590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42984/0 (socket says 172.21.15.129:42984) 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 -- 172.21.15.129:0/863064677 learned_addr learned my addr 172.21.15.129:0/863064677 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e3e00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f73800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e9600640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 0x558288f8e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 -- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 msgr2=0x558288f73800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f73800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 -- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 msgr2=0x558288f8e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 0x558288f8e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 -- 172.21.15.129:0/863064677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558288ee1db0 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e9600640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 0x558288f8e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36e8c00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009780 tx=0x7f36d80822c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140c860d3437fe9f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36c9600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d808bdf0 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.681+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f36d8081950 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36d0004040 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f36d80871d0 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 10 client.5022 fetch_fsmap learned FSMap version 17 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x558288f8af70 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36d0004040 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 17) ==== 3502+0+0 (secure 0 0 0) 0x7f36d803a030 con 0x558288af23b0 2024-09-17T03:50:35.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 10 client.5022 fetch_fsmap finished waiting for FSMap version 17 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 10 client.5022 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 20 client.5022 populate_metadata read hostname 'smithi129' 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288f18960 0x558288ee3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 4 client.5022 mds_command: new command op to 4845 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 --> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] -- command(tid 0: {"prefix": "status"}) -- 0x558288ec0fb0 con 0x558288f18960 2024-09-17T03:50:35.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.682+0000 7f36e3e00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288f18960 0x558288ee3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36e3e00640 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288f18960 0x558288ee3610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0a8afbea2cd8f0 server_cookie=c5ed8a08c44f9b78 in_seq=0 out_seq=0 2024-09-17T03:50:35.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36c9600640 10 client.5022 ms_handle_connect on v2:172.21.15.129:6833/1447431306 2024-09-17T03:50:35.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36c9600640 1 -- 172.21.15.129:0/863064677 <== mds.0 v2:172.21.15.129:6833/1447431306 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x558288ec0fb0 con 0x558288f18960 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36c9600640 10 client.5022 handle_command_reply: tid=0 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 client.5022 shutdown 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 msgr2=0x558288ee1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009780 tx=0x7f36d80822c0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 shutdown_connections 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:6833/1447431306,v1:172.21.15.129:6835/1447431306] conn(0x558288f18960 0x558288ee3610 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558288b1b270 0x558288f8e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558288afa590 0x558288f73800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36c9600640 0 client.5022 ms_handle_reset on v2:172.21.15.129:6833/1447431306 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 --2- 172.21.15.129:0/863064677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558288af23b0 0x558288ee1700 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.683+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 >> 172.21.15.129:0/863064677 conn(0x558288b0f3c0 msgr2=0x558288ebf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 shutdown_connections 2024-09-17T03:50:35.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36f2600740 1 -- 172.21.15.129:0/863064677 wait complete. 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36f2600740 20 client.5022 trim_cache size 0 max 16384 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4845, 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:50:35.687 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.570039325, 2024-09-17T03:50:35.688 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 17, 2024-09-17T03:50:35.688 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 33, 2024-09-17T03:50:35.688 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 33, 2024-09-17T03:50:35.688 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 19.398408367999998 2024-09-17T03:50:35.688 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 msgr2=0x7f36c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 0x7f36c407b0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f36e40c3d00 tx=0x7f36d0009040 comp rx=0 tx=0).stop 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 msgr2=0x7f36e414b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009950 tx=0x7f36d8003b60 comp rx=0 tx=0).stop 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 shutdown_connections 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36c4078bc0 0x7f36c407b0d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e4073170 0x7f36e41f6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e4070440 0x7f36e40d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 --2- 172.21.15.129:0/2662909800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e4070030 0x7f36e414b2d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.684+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 >> 172.21.15.129:0/2662909800 conn(0x7f36e40b1cb0 msgr2=0x7f36e415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.685+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 shutdown_connections 2024-09-17T03:50:35.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.685+0000 7f36eb600640 1 -- 172.21.15.129:0/2662909800 wait complete. 2024-09-17T03:50:35.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:50:35.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 -- 172.21.15.129:0/2358010343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75280cf540 msgr2=0x7f752806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 --2- 172.21.15.129:0/2358010343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75280cf540 0x7f752806ee70 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7510005fd0 tx=0x7f751002a540 comp rx=0 tx=0).stop 2024-09-17T03:50:35.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 -- 172.21.15.129:0/2358010343 shutdown_connections 2024-09-17T03:50:35.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 --2- 172.21.15.129:0/2358010343 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf950 0x7f752806f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 --2- 172.21.15.129:0/2358010343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75280cf540 0x7f752806ee70 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 -- 172.21.15.129:0/2358010343 >> 172.21.15.129:0/2358010343 conn(0x7f752806d410 msgr2=0x7f75280b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:35.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 -- 172.21.15.129:0/2358010343 shutdown_connections 2024-09-17T03:50:35.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.927+0000 7f7527e00640 1 -- 172.21.15.129:0/2358010343 wait complete. 2024-09-17T03:50:35.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.929+0000 7f7527e00640 1 Processor -- start 2024-09-17T03:50:35.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.929+0000 7f7527e00640 1 -- start start 2024-09-17T03:50:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf540 0x7f7528154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75280cf950 0x7f7528143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f752815dde0 con 0x7f752815ca50 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f752815dc60 con 0x7f75280cf950 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f752815df60 con 0x7f75280cf540 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f751e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75280cf950 0x7f7528143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7526a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf540 0x7f7528154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43000/0 (socket says 172.21.15.129:43000) 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 -- 172.21.15.129:0/3639812657 learned_addr learned my addr 172.21.15.129:0/3639812657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 -- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf540 msgr2=0x7f7528154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf540 0x7f7528154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.931+0000 7f7527400640 1 -- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75280cf950 msgr2=0x7f7528143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f7527400640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75280cf950 0x7f7528143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:35.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f7527400640 1 -- 172.21.15.129:0/3639812657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75280afc50 con 0x7f752815ca50 2024-09-17T03:50:35.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f7527400640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f752000e960 tx=0x7f752000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520004d20 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f752000cac0 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7520005020 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75280aff30 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.932+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75280b04f0 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.933+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75281584d0 con 0x7f752815ca50 2024-09-17T03:50:35.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.935+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f75200051c0 con 0x7f752815ca50 2024-09-17T03:50:35.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.935+0000 7f751fe00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 0x7f750407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:35.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.936+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f752009b630 con 0x7f752815ca50 2024-09-17T03:50:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.936+0000 7f7526a00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 0x7f750407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.937+0000 7f7526a00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 0x7f750407b0d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75180016c0 tx=0x7f7518009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:35.940+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7520016030 con 0x7f752815ca50 2024-09-17T03:50:36.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.159+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f752815f590 con 0x7f752815ca50 2024-09-17T03:50:36.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.159+0000 7f751fe00640 1 -- 172.21.15.129:0/3639812657 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+73 (secure 0 0 0) 0x7f7520063410 con 0x7f752815ca50 2024-09-17T03:50:36.158 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.161+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 msgr2=0x7f750407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.161+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 0x7f750407b0d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75180016c0 tx=0x7f7518009040 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.161+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 msgr2=0x7f7528157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.161+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f752000e960 tx=0x7f752000ee20 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 shutdown_connections 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7504078bc0 0x7f750407b0d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f752815ca50 0x7f7528157a20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75280cf950 0x7f7528143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 --2- 172.21.15.129:0/3639812657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75280cf540 0x7f7528154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 >> 172.21.15.129:0/3639812657 conn(0x7f752806d410 msgr2=0x7f7528145820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 shutdown_connections 2024-09-17T03:50:36.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36.162+0000 7f7527e00640 1 -- 172.21.15.129:0/3639812657 wait complete. 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:50:36.185 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:50:36.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.186 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:50:36.222 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:50:36.222 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:50:36.222 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:50:36.222 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.222 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:36.223 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42656sec preferred_lft 42656sec 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.224 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:50:36.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.225 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:50:36.316 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:50:36.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.319 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:50:36.355 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:50:36.355 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:50:36.355 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:50:36.355 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:50:36.355 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:50:36.356 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:50:36.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:50:36.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:36.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.551 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:50:36.586 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:50:36.586 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:50:36.587 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:50:36.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:50:36.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:50:36.684 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:50:36.719 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:50:36.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:50:36.719 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:50:36.793 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:50:36.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:36.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:50:36.989 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:50:36.989 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:50:36.989 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:50:37.009 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:50:37.010 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:50:37.010 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:50:37.010 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:50:37.010 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:50:37.038 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:50:37.039 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:50:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:50:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:50:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:50:37.040 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:50:37.042 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:50:37.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:37.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:50:37.099 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:50:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:37 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:50:37.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:37.196 INFO:teuthology.orchestra.run.smithi129.stdout:client5031 2024-09-17T03:50:37.196 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:37.196 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:37.231 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:50:37.231 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> subdir = "delete_me" 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> size = 1024 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> file_count = 1000 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(0, file_count): 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> filename = "{0}_{1}.bin".format(i, size) 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> f.write(size * '"'"'x'"'"') 2024-09-17T03:50:37.232 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:50:38.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:38.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:38.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 -- 172.21.15.129:0/1868391187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 msgr2=0x7effb406ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:38.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 --2- 172.21.15.129:0/1868391187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb406ffa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7effb40c71a0 tx=0x7eff9c02a560 comp rx=0 tx=0).stop 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 -- 172.21.15.129:0/1868391187 shutdown_connections 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 --2- 172.21.15.129:0/1868391187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb406ffa0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 --2- 172.21.15.129:0/1868391187 >> v2:172.21.15.181:3300/0 conn(0x7effb40ca800 0x7effb40cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 -- 172.21.15.129:0/1868391187 >> 172.21.15.129:0/1868391187 conn(0x7effb40b1cb0 msgr2=0x7effb40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 -- 172.21.15.129:0/1868391187 shutdown_connections 2024-09-17T03:50:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.764+0000 7effb8a00640 1 -- 172.21.15.129:0/1868391187 wait complete. 2024-09-17T03:50:38.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.766+0000 7effb8a00640 1 Processor -- start 2024-09-17T03:50:38.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.766+0000 7effb8a00640 1 -- start start 2024-09-17T03:50:38.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb40704e0 0x7effb41f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:38.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 0x7effb414fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effb40ccf40 con 0x7effb40cad30 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7effb4074810 con 0x7effb40704e0 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7effb40cd0c0 con 0x7effb40ca800 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43016/0 (socket says 172.21.15.129:43016) 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 -- 172.21.15.129:0/656837178 learned_addr learned my addr 172.21.15.129:0/656837178 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:38.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb1e00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb40704e0 0x7effb41f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effa9400640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 0x7effb414fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 -- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 msgr2=0x7effb414fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 0x7effb414fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 -- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb40704e0 msgr2=0x7effb41f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb40704e0 0x7effb41f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effb2800640 1 -- 172.21.15.129:0/656837178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb41f6780 con 0x7effb40cad30 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.768+0000 7effa9400640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 0x7effb414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:38.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effb2800640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7effac00b970 tx=0x7effac00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:38.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac019070 con 0x7effb40cad30 2024-09-17T03:50:38.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effac00cc30 con 0x7effb40cad30 2024-09-17T03:50:38.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac00ce60 con 0x7effb40cad30 2024-09-17T03:50:38.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb40b0a10 con 0x7effb40cad30 2024-09-17T03:50:38.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.769+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb40b0fd0 con 0x7effb40cad30 2024-09-17T03:50:38.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.770+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb40cd5e0 con 0x7effb40cad30 2024-09-17T03:50:38.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.771+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7effac01c070 con 0x7effb40cad30 2024-09-17T03:50:38.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.772+0000 7effaaa00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 0x7eff9007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:38.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.772+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7effac09a9e0 con 0x7effb40cad30 2024-09-17T03:50:38.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.773+0000 7effb1e00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 0x7eff9007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:38.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.773+0000 7effb1e00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 0x7eff9007b0a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7effa4005fd0 tx=0x7effa4005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:38.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.776+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7effb41f6780 con 0x7effb40cad30 2024-09-17T03:50:39.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.999+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effb40ae890 con 0x7effb40cad30 2024-09-17T03:50:39.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:38.999+0000 7effaaa00640 1 -- 172.21.15.129:0/656837178 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7effac062d80 con 0x7effb40cad30 2024-09-17T03:50:39.001 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:39.001 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:39.001 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 msgr2=0x7eff9007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 0x7eff9007b0a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7effa4005fd0 tx=0x7effa4005850 comp rx=0 tx=0).stop 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 msgr2=0x7effb41526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7effac00b970 tx=0x7effac00be30 comp rx=0 tx=0).stop 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 shutdown_connections 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff90078b90 0x7eff9007b0a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effb40cad30 0x7effb41526c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effb40ca800 0x7effb414fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 --2- 172.21.15.129:0/656837178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effb40704e0 0x7effb41f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 >> 172.21.15.129:0/656837178 conn(0x7effb40b1cb0 msgr2=0x7effb40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 shutdown_connections 2024-09-17T03:50:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.002+0000 7effb8a00640 1 -- 172.21.15.129:0/656837178 wait complete. 2024-09-17T03:50:39.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-17T03:50:39.609 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:50:39.612 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:50:39.612 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:50:39.612 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:50:39.612 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T03:50:39.612 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:50:39.613 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/delete_me").st_ino) 2024-09-17T03:50:39.613 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:50:39.711 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627776 2024-09-17T03:50:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:50:39.722 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-09-17T03:50:39.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.903+0000 7f109071bf40 1 Processor -- start 2024-09-17T03:50:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.903+0000 7f109071bf40 1 -- start start 2024-09-17T03:50:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5610ae3431d0 con 0x5610ae464510 2024-09-17T03:50:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5610ae34ff90 con 0x5610ae46a420 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f109071bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5610ae2e4c60 con 0x5610ae3dfb40 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43046/0 (socket says 172.21.15.129:43046) 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 -- 172.21.15.129:0/1205142442 learned_addr learned my addr 172.21.15.129:0/1205142442 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1089800640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1088e00640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 -- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 msgr2=0x5610ae4675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 -- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 msgr2=0x5610ae3e01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1083e00640 1 -- 172.21.15.129:0/1205142442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610ae3d7a40 con 0x5610ae464510 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.905+0000 7f1088e00640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.906+0000 7f1083e00640 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1074026020 tx=0x7f1074002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7157a00e9a796df server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.906+0000 7f1082a00640 1 -- 172.21.15.129:0/1205142442 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074066070 con 0x5610ae464510 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.906+0000 7f1082a00640 1 -- 172.21.15.129:0/1205142442 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f107404ee20 con 0x5610ae464510 2024-09-17T03:50:39.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.906+0000 7f1082a00640 1 -- 172.21.15.129:0/1205142442 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074056670 con 0x5610ae464510 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 -- 172.21.15.129:0/1205142442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 msgr2=0x5610ae3d5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1074026020 tx=0x7f1074002fb0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 -- 172.21.15.129:0/1205142442 shutdown_connections 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3d5720 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 --2- 172.21.15.129:0/1205142442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 -- 172.21.15.129:0/1205142442 >> 172.21.15.129:0/1205142442 conn(0x5610ae2b5ac0 msgr2=0x5610ae3e12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 -- 172.21.15.129:0/1205142442 shutdown_connections 2024-09-17T03:50:39.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.907+0000 7f109071bf40 1 -- 172.21.15.129:0/1205142442 wait complete. 2024-09-17T03:50:39.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.908+0000 7f109071bf40 1 Processor -- start 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.908+0000 7f109071bf40 1 -- start start 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4631c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5610ae467b20 con 0x5610ae464510 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5610ae3d7370 con 0x5610ae46a420 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f109071bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5610ae469d70 con 0x5610ae3dfb40 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f1083e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f1083e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3e0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43062/0 (socket says 172.21.15.129:43062) 2024-09-17T03:50:39.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.909+0000 7f1083e00640 1 -- 172.21.15.129:0/3580088243 learned_addr learned my addr 172.21.15.129:0/3580088243 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 -- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 msgr2=0x5610ae4631c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5610ae3dfb40 0x5610ae4631c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 -- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 msgr2=0x5610ae3e11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5610ae46a420 0x5610ae3e11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 -- 172.21.15.129:0/3580088243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610ae3dc9f0 con 0x5610ae464510 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1083e00640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3e0c70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1074004c30 tx=0x7f1074058b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1081600640 1 -- 172.21.15.129:0/3580088243 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074066040 con 0x5610ae464510 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1081600640 1 -- 172.21.15.129:0/3580088243 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1074058d20 con 0x5610ae464510 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5610ae466af0 con 0x5610ae464510 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.910+0000 7f1081600640 1 -- 172.21.15.129:0/3580088243 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740582d0 con 0x5610ae464510 2024-09-17T03:50:39.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.911+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610ae4670b0 con 0x5610ae464510 2024-09-17T03:50:39.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.912+0000 7f1081600640 1 -- 172.21.15.129:0/3580088243 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1074056470 con 0x5610ae464510 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.913+0000 7f1081600640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10680775a0 0x7f1068079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.913+0000 7f1081600640 1 -- 172.21.15.129:0/3580088243 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f10740deeb0 con 0x5610ae464510 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.914+0000 7f109071bf40 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x5610ae5071e0 0x5610ae5075f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.914+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x5610ae3da310 con 0x5610ae5071e0 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.914+0000 7f1088e00640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10680775a0 0x7f1068079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.914+0000 7f1089800640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x5610ae5071e0 0x5610ae5075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.914+0000 7f1089800640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x5610ae5071e0 0x5610ae5075f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f1088e00640 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10680775a0 0x7f1068079ab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f60 tx=0x7f107c0034f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:39.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f1089800640 1 -- 172.21.15.129:0/3580088243 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(1 10000000000.00000000 [stat out=16b] v0'0 uv10 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x7f1084001460 con 0x5610ae5071e0 2024-09-17T03:50:39.917 INFO:teuthology.orchestra.run.smithi129.stdout:cephfs_metadata/10000000000.00000000 mtime 2024-09-17T03:50:39.000000+0000, size 0 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x5610ae5071e0 msgr2=0x5610ae5075f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x5610ae5071e0 0x5610ae5075f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10680775a0 msgr2=0x7f1068079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10680775a0 0x7f1068079ab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f107c000f60 tx=0x7f107c0034f0 comp rx=0 tx=0).stop 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 msgr2=0x5610ae3e0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 --2- 172.21.15.129:0/3580088243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5610ae464510 0x5610ae3e0c70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1074004c30 tx=0x7f1074058b30 comp rx=0 tx=0).stop 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f1089800640 1 -- 172.21.15.129:0/3580088243 reap_dead start 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 shutdown_connections 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 >> 172.21.15.129:0/3580088243 conn(0x5610ae2b5ac0 msgr2=0x5610ae394800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 shutdown_connections 2024-09-17T03:50:39.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:39.915+0000 7f109071bf40 1 -- 172.21.15.129:0/3580088243 wait complete. 2024-09-17T03:50:39.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:39.927 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-09-17T03:50:42.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 -- 172.21.15.129:0/2007721101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 msgr2=0x7fd40815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 --2- 172.21.15.129:0/2007721101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd40815c780 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd4080b4b40 tx=0x7fd3f802a4d0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 -- 172.21.15.129:0/2007721101 shutdown_connections 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 --2- 172.21.15.129:0/2007721101 >> v2:172.21.15.181:3300/0 conn(0x7fd408070440 0x7fd4080b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 --2- 172.21.15.129:0/2007721101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd40815c780 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 -- 172.21.15.129:0/2007721101 >> 172.21.15.129:0/2007721101 conn(0x7fd4080b1cb0 msgr2=0x7fd4080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 -- 172.21.15.129:0/2007721101 shutdown_connections 2024-09-17T03:50:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.270+0000 7fd40e800640 1 -- 172.21.15.129:0/2007721101 wait complete. 2024-09-17T03:50:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.273+0000 7fd40e800640 1 Processor -- start 2024-09-17T03:50:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.273+0000 7fd40e800640 1 -- start start 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 0x7fd4081f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd408070440 0x7fd4080cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4080b3a40 con 0x7fd408073170 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4080b38c0 con 0x7fd408070440 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd40e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd4080b3bc0 con 0x7fd408070030 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.274+0000 7fd407e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43080/0 (socket says 172.21.15.129:43080) 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 -- 172.21.15.129:0/4053721580 learned_addr learned my addr 172.21.15.129:0/4053721580 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407400640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 0x7fd4081f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:42.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd3fea00640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd408070440 0x7fd4080cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 -- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 msgr2=0x7fd4081f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 0x7fd4081f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 -- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd408070440 msgr2=0x7fd4080cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd408070440 0x7fd4080cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 -- 172.21.15.129:0/4053721580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4081f7ac0 con 0x7fd408073170 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407400640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 0x7fd4081f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd407e00640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd40000cab0 tx=0x7fd40000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.275+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd400007d90 con 0x7fd408073170 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.276+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd40000f560 con 0x7fd408073170 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.276+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40000f9b0 con 0x7fd408073170 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.276+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4081f7da0 con 0x7fd408073170 2024-09-17T03:50:42.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.276+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4081f3ce0 con 0x7fd408073170 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.277+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4081f7830 con 0x7fd408073170 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.278+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd40000fb50 con 0x7fd408073170 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.279+0000 7fd3ffe00640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 0x7fd3e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.279+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd40009b810 con 0x7fd408073170 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.280+0000 7fd407400640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 0x7fd3e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.280+0000 7fd407400640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 0x7fd3e407b0d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8033000 tx=0x7fd3f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:42.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.283+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd400014070 con 0x7fd408073170 2024-09-17T03:50:42.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.622+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4080b4100 con 0x7fd408073170 2024-09-17T03:50:42.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.623+0000 7fd3ffe00640 1 -- 172.21.15.129:0/4053721580 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7fd400063bb0 con 0x7fd408073170 2024-09-17T03:50:42.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:42.623 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:42.624 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:42.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.627+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 msgr2=0x7fd3e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:42.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.627+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 0x7fd3e407b0d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd3f8033000 tx=0x7fd3f8004730 comp rx=0 tx=0).stop 2024-09-17T03:50:42.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.627+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 msgr2=0x7fd4081f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:42.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.627+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd40000cab0 tx=0x7fd40000cf70 comp rx=0 tx=0).stop 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 shutdown_connections 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3e4078bc0 0x7fd3e407b0d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd408073170 0x7fd4081f6ee0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd408070440 0x7fd4080cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 --2- 172.21.15.129:0/4053721580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd408070030 0x7fd4081f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 >> 172.21.15.129:0/4053721580 conn(0x7fd4080b1cb0 msgr2=0x7fd4080afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 shutdown_connections 2024-09-17T03:50:42.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:42.630+0000 7fd40e800640 1 -- 172.21.15.129:0/4053721580 wait complete. 2024-09-17T03:50:42.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-17T03:50:43.202 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:50:43.202 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:50:43.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 -- 172.21.15.129:0/1124207409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc0cf200 msgr2=0x7f0dcc15ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 --2- 172.21.15.129:0/1124207409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc0cf200 0x7f0dcc15ba00 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc070e20 tx=0x7f0db802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 -- 172.21.15.129:0/1124207409 shutdown_connections 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 --2- 172.21.15.129:0/1124207409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc0cf200 0x7f0dcc15ba00 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 --2- 172.21.15.129:0/1124207409 >> v2:172.21.15.181:3300/0 conn(0x7f0dcc0d0500 0x7f0dcc0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 -- 172.21.15.129:0/1124207409 >> 172.21.15.129:0/1124207409 conn(0x7f0dcc06d870 msgr2=0x7f0dcc06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:43.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 -- 172.21.15.129:0/1124207409 shutdown_connections 2024-09-17T03:50:43.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.427+0000 7f0dd2200640 1 -- 172.21.15.129:0/1124207409 wait complete. 2024-09-17T03:50:43.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.429+0000 7f0dd2200640 1 Processor -- start 2024-09-17T03:50:43.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.429+0000 7f0dd2200640 1 -- start start 2024-09-17T03:50:43.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.430+0000 7f0dd2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0dcc0cf200 0x7f0dcc15af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:43.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dcc0d0500 0x7f0dcc1f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:43.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:43.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc15c4d0 con 0x7f0dcc15bf40 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc15c350 con 0x7f0dcc0d0500 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc15f700 con 0x7f0dcc0cf200 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0dcc0cf200 0x7f0dcc15af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dcbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dcc0d0500 0x7f0dcc1f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43094/0 (socket says 172.21.15.129:43094) 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 -- 172.21.15.129:0/2629964238 learned_addr learned my addr 172.21.15.129:0/2629964238 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 -- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0dcc0cf200 msgr2=0x7f0dcc15af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0dcc0cf200 0x7f0dcc15af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 -- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dcc0d0500 msgr2=0x7f0dcc1f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dcc0d0500 0x7f0dcc1f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.431+0000 7f0dd1800640 1 -- 172.21.15.129:0/2629964238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc138190 con 0x7f0dcc15bf40 2024-09-17T03:50:43.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dd1800640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c730 tx=0x7f0dc400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:43.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4003b70 con 0x7f0dcc15bf40 2024-09-17T03:50:43.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dc4003d10 con 0x7f0dcc15bf40 2024-09-17T03:50:43.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4005330 con 0x7f0dcc15bf40 2024-09-17T03:50:43.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc0b0eb0 con 0x7f0dcc15bf40 2024-09-17T03:50:43.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.432+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc0b1470 con 0x7f0dcc15bf40 2024-09-17T03:50:43.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.433+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc15fbb0 con 0x7f0dcc15bf40 2024-09-17T03:50:43.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.434+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0dc40054d0 con 0x7f0dcc15bf40 2024-09-17T03:50:43.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.436+0000 7f0dc8c00640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 0x7f0dac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:43.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.436+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0dc409a9c0 con 0x7f0dcc15bf40 2024-09-17T03:50:43.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.436+0000 7f0dd0e00640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 0x7f0dac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:43.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.437+0000 7f0dd0e00640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 0x7f0dac07b0a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009b30 tx=0x7f0dc0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:43.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.441+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0dc4062710 con 0x7f0dcc15bf40 2024-09-17T03:50:43.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.668+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dcc06f060 con 0x7f0dcc15bf40 2024-09-17T03:50:43.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.668+0000 7f0dc8c00640 1 -- 172.21.15.129:0/2629964238 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f0dc4067140 con 0x7f0dcc15bf40 2024-09-17T03:50:43.668 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:43.669 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.670+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 msgr2=0x7f0dac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.670+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 0x7f0dac07b0a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0dc0009b30 tx=0x7f0dc0005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.670+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 msgr2=0x7f0dcc137c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.670+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f0dc400c730 tx=0x7f0dc400cb30 comp rx=0 tx=0).stop 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 shutdown_connections 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dac078b90 0x7f0dac07b0a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0dcc15bf40 0x7f0dcc137c50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0dcc0d0500 0x7f0dcc1f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 --2- 172.21.15.129:0/2629964238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0dcc0cf200 0x7f0dcc15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 >> 172.21.15.129:0/2629964238 conn(0x7f0dcc06d870 msgr2=0x7f0dcc0cb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 shutdown_connections 2024-09-17T03:50:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:43.671+0000 7f0dd2200640 1 -- 172.21.15.129:0/2629964238 wait complete. 2024-09-17T03:50:43.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:50:43.937 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1001}} 2024-09-17T03:50:43.937 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1001 } } 2024-09-17T03:50:43.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:44.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 -- 172.21.15.129:0/161970623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 msgr2=0x7f08940d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 --2- 172.21.15.129:0/161970623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940d0fc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f088c005fd0 tx=0x7f088c02a840 comp rx=0 tx=0).stop 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 -- 172.21.15.129:0/161970623 shutdown_connections 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 --2- 172.21.15.129:0/161970623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940d0fc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 --2- 172.21.15.129:0/161970623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b9540 0x7f08940d4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 -- 172.21.15.129:0/161970623 >> 172.21.15.129:0/161970623 conn(0x7f089409d680 msgr2=0x7f08940d4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 -- 172.21.15.129:0/161970623 shutdown_connections 2024-09-17T03:50:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.164+0000 7f0893200640 1 -- 172.21.15.129:0/161970623 wait complete. 2024-09-17T03:50:44.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.167+0000 7f0893200640 1 Processor -- start 2024-09-17T03:50:44.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.168+0000 7f0893200640 1 -- start start 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b3660 0x7f08940cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08940b3e20 con 0x7f08940b9540 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f08940b3ca0 con 0x7f08940d0bb0 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0893200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f08940b3fa0 con 0x7f08940b3660 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0891400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:44.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0891400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47346/0 (socket says 172.21.15.129:47346) 2024-09-17T03:50:44.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0891400640 1 -- 172.21.15.129:0/1170093466 learned_addr learned my addr 172.21.15.129:0/1170093466 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:44.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0891e00640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b3660 0x7f08940cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:44.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.170+0000 7f0892800640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 -- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b3660 msgr2=0x7f08940cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b3660 0x7f08940cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 -- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 msgr2=0x7f08940cb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940cb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 -- 172.21.15.129:0/1170093466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08940ccd30 con 0x7f08940b9540 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0891400640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f088000c500 tx=0x7f088000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880011790 con 0x7f08940b9540 2024-09-17T03:50:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0880017070 con 0x7f08940b9540 2024-09-17T03:50:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880011cc0 con 0x7f08940b9540 2024-09-17T03:50:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08941f7220 con 0x7f08940b9540 2024-09-17T03:50:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.171+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08940ce1d0 con 0x7f08940b9540 2024-09-17T03:50:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.173+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08940b4c30 con 0x7f08940b9540 2024-09-17T03:50:44.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.175+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f088001c460 con 0x7f08940b9540 2024-09-17T03:50:44.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.177+0000 7f088a000640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 0x7f087407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:44.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.177+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f088009aad0 con 0x7f08940b9540 2024-09-17T03:50:44.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.177+0000 7f0891e00640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 0x7f087407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:44.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.178+0000 7f0891e00640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 0x7f087407b0a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0884009c00 tx=0x7f0884005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:44.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.185+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f08940ccd30 con 0x7f08940b9540 2024-09-17T03:50:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.569+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08941fa7d0 con 0x7f08940b9540 2024-09-17T03:50:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.570+0000 7f088a000640 1 -- 172.21.15.129:0/1170093466 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f08940d1500 con 0x7f08940b9540 2024-09-17T03:50:44.570 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:44.570 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 msgr2=0x7f087407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 0x7f087407b0a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0884009c00 tx=0x7f0884005f30 comp rx=0 tx=0).stop 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 msgr2=0x7f08941f6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f088000c500 tx=0x7f088000c9c0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 shutdown_connections 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0874078b90 0x7f087407b0a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08940d0bb0 0x7f08940cb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08940b9540 0x7f08941f6ce0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 --2- 172.21.15.129:0/1170093466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08940b3660 0x7f08940cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 >> 172.21.15.129:0/1170093466 conn(0x7f089409d680 msgr2=0x7f08940b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 shutdown_connections 2024-09-17T03:50:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:44.573+0000 7f0893200640 1 -- 172.21.15.129:0/1170093466 wait complete. 2024-09-17T03:50:44.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:50:44.834 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":966}} 2024-09-17T03:50:44.835 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 966 } } 2024-09-17T03:50:44.835 DEBUG:tasks.ceph_test_case:wait_until_equal: 966 != 1001, waiting (timeout=600)... 2024-09-17T03:50:45.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:50:45.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:50:49.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.065+0000 7fc385200640 1 -- 172.21.15.129:0/4122471107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cf660 msgr2=0x7fc3800cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.065+0000 7fc385200640 1 --2- 172.21.15.129:0/4122471107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cf660 0x7fc3800cf010 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc38009e2c0 tx=0x7fc37002a530 comp rx=0 tx=0).stop 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 -- 172.21.15.129:0/4122471107 shutdown_connections 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 --2- 172.21.15.129:0/4122471107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cfb90 0x7fc3800ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 --2- 172.21.15.129:0/4122471107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cf660 0x7fc3800cf010 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 -- 172.21.15.129:0/4122471107 >> 172.21.15.129:0/4122471107 conn(0x7fc3800b1cb0 msgr2=0x7fc3800b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 -- 172.21.15.129:0/4122471107 shutdown_connections 2024-09-17T03:50:50.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.066+0000 7fc385200640 1 -- 172.21.15.129:0/4122471107 wait complete. 2024-09-17T03:50:50.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.068+0000 7fc385200640 1 Processor -- start 2024-09-17T03:50:50.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.068+0000 7fc385200640 1 -- start start 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cd070 0x7fc3800c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 0x7fc3800c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc380070f20 con 0x7fc3800cfb90 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc3800af810 con 0x7fc3800cf660 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.069+0000 7fc385200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3800710a0 con 0x7fc3800cd070 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47370/0 (socket says 172.21.15.129:47370) 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 -- 172.21.15.129:0/1669957024 learned_addr learned my addr 172.21.15.129:0/1669957024 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37de00640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cd070 0x7fc3800c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37d400640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 0x7fc3800c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 -- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cd070 msgr2=0x7fc3800c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cd070 0x7fc3800c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 -- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 msgr2=0x7fc3800c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 0x7fc3800c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 -- 172.21.15.129:0/1669957024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc380149dc0 con 0x7fc3800cfb90 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37d400640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 0x7fc3800c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.070+0000 7fc37e800640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c6f0 tx=0x7fc37800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.071+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378017070 con 0x7fc3800cfb90 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.071+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc378003c90 con 0x7fc3800cfb90 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.071+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc378003e60 con 0x7fc3800cfb90 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.071+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3800ca600 con 0x7fc3800cfb90 2024-09-17T03:50:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.071+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3800ae870 con 0x7fc3800cfb90 2024-09-17T03:50:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.072+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3800ca370 con 0x7fc3800cfb90 2024-09-17T03:50:50.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.073+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc37801a070 con 0x7fc3800cfb90 2024-09-17T03:50:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.074+0000 7fc376000640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 0x7fc36007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.074+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc37809a260 con 0x7fc3800cfb90 2024-09-17T03:50:50.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.074+0000 7fc37de00640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 0x7fc36007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.075+0000 7fc37de00640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 0x7fc36007b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc370033000 tx=0x7fc370004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:50.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.078+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc380149dc0 con 0x7fc3800cfb90 2024-09-17T03:50:50.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.296+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3800cf010 con 0x7fc3800cfb90 2024-09-17T03:50:50.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.296+0000 7fc376000640 1 -- 172.21.15.129:0/1669957024 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7fc3800cd480 con 0x7fc3800cfb90 2024-09-17T03:50:50.297 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:50.297 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:50.297 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 msgr2=0x7fc36007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 0x7fc36007b0d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc370033000 tx=0x7fc370004730 comp rx=0 tx=0).stop 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 msgr2=0x7fc3801f9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fc37800c6f0 tx=0x7fc37800cbb0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 shutdown_connections 2024-09-17T03:50:50.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078bc0 0x7fc36007b0d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3800cfb90 0x7fc3801f9120 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3800cf660 0x7fc3800c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 --2- 172.21.15.129:0/1669957024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800cd070 0x7fc3800c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 >> 172.21.15.129:0/1669957024 conn(0x7fc3800b1cb0 msgr2=0x7fc3800afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 shutdown_connections 2024-09-17T03:50:50.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.299+0000 7fc385200640 1 -- 172.21.15.129:0/1669957024 wait complete. 2024-09-17T03:50:50.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:50:50.563 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":1001}} 2024-09-17T03:50:50.563 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1001 } } 2024-09-17T03:50:50.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 -- 172.21.15.129:0/150090242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 msgr2=0x7f2e0415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 --2- 172.21.15.129:0/150090242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e0415c780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2df8005fd0 tx=0x7f2df802aaa0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 -- 172.21.15.129:0/150090242 shutdown_connections 2024-09-17T03:50:50.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 --2- 172.21.15.129:0/150090242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070440 0x7f2e040b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 --2- 172.21.15.129:0/150090242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e0415c780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 -- 172.21.15.129:0/150090242 >> 172.21.15.129:0/150090242 conn(0x7f2e040b1cb0 msgr2=0x7f2e040b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 -- 172.21.15.129:0/150090242 shutdown_connections 2024-09-17T03:50:50.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.784+0000 7f2e0c200640 1 -- 172.21.15.129:0/150090242 wait complete. 2024-09-17T03:50:50.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.786+0000 7f2e0c200640 1 Processor -- start 2024-09-17T03:50:50.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.786+0000 7f2e0c200640 1 -- start start 2024-09-17T03:50:50.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.787+0000 7f2e0c200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 0x7f2e04154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.787+0000 7f2e0c200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e0c200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e040cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e0c200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e040b3b00 con 0x7f2e04070440 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e0c200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2e040b3980 con 0x7f2e04073170 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e0c200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2e040b3c80 con 0x7f2e04070030 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47380/0 (socket says 172.21.15.129:47380) 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 -- 172.21.15.129:0/233404184 learned_addr learned my addr 172.21.15.129:0/233404184 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e09800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 0x7f2e04154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e0a200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e040cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 -- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 msgr2=0x7f2e04154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 0x7f2e04154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 -- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 msgr2=0x7f2e040cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e040cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:50.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e08e00640 1 -- 172.21.15.129:0/233404184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e041550b0 con 0x7f2e04070440 2024-09-17T03:50:50.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.788+0000 7f2e09800640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 0x7f2e04154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:50:50.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e08e00640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2df000c370 tx=0x7f2df000c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0017070 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2df00053c0 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e040ca5e0 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df0003850 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.789+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0409af20 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.790+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e040ca070 con 0x7f2e04070440 2024-09-17T03:50:50.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.791+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2df001a070 con 0x7f2e04070440 2024-09-17T03:50:50.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.792+0000 7f2e01600640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 0x7f2de407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:50.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.793+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f2df0099fe0 con 0x7f2e04070440 2024-09-17T03:50:50.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.793+0000 7f2e09800640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 0x7f2de407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:50.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.794+0000 7f2e09800640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 0x7f2de407b0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2df8035000 tx=0x7f2df8038040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:50.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:50.797+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2e041550b0 con 0x7f2e04070440 2024-09-17T03:50:51.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.014+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e040b41a0 con 0x7f2e04070440 2024-09-17T03:50:51.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.015+0000 7f2e01600640 1 -- 172.21.15.129:0/233404184 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f2df0061d50 con 0x7f2e04070440 2024-09-17T03:50:51.015 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:51.015 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:51.016 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:51.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 msgr2=0x7f2de407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 0x7f2de407b0d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2df8035000 tx=0x7f2df8038040 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 msgr2=0x7f2e041515d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2df000c370 tx=0x7f2df000c770 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 shutdown_connections 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2de4078bc0 0x7f2de407b0d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e04073170 0x7f2e040cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e04070440 0x7f2e041515d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 --2- 172.21.15.129:0/233404184 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e04070030 0x7f2e04154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 >> 172.21.15.129:0/233404184 conn(0x7f2e040b1cb0 msgr2=0x7f2e040afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 shutdown_connections 2024-09-17T03:50:51.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.017+0000 7f2e0c200640 1 -- 172.21.15.129:0/233404184 wait complete. 2024-09-17T03:50:51.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-09-17T03:50:51.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":1001}} 2024-09-17T03:50:51.277 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1001 } } 2024-09-17T03:50:51.278 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat 10000000000.00000000 2024-09-17T03:50:51.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.418+0000 7f6ad8f20f40 1 Processor -- start 2024-09-17T03:50:51.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.418+0000 7f6ad8f20f40 1 -- start start 2024-09-17T03:50:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d1450 0x55f26475d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f264755510 0x55f26475df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f26475c340 con 0x55f2646d0b40 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f2645d5c60 con 0x55f2646d1450 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.419+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55f2646d1860 con 0x55f264755510 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47398/0 (socket says 172.21.15.129:47398) 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 -- 172.21.15.129:0/1828152120 learned_addr learned my addr 172.21.15.129:0/1828152120 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6acbe00640 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d1450 0x55f26475d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad2200640 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f264755510 0x55f26475df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 -- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f264755510 msgr2=0x55f26475df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f264755510 0x55f26475df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 -- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d1450 msgr2=0x55f26475d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d1450 0x55f26475d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 -- 172.21.15.129:0/1828152120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2645adf10 con 0x55f2646d0b40 2024-09-17T03:50:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6ad1800640 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4046600 tx=0x7f6ac4002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927d539ffa076e0e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.420+0000 7f6acb400640 1 -- 172.21.15.129:0/1828152120 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac404d960 con 0x55f2646d0b40 2024-09-17T03:50:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6acb400640 1 -- 172.21.15.129:0/1828152120 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ac4056670 con 0x55f2646d0b40 2024-09-17T03:50:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6acb400640 1 -- 172.21.15.129:0/1828152120 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4056ac0 con 0x55f2646d0b40 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/1828152120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 msgr2=0x55f26473cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4046600 tx=0x7f6ac4002fb0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/1828152120 shutdown_connections 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f264755510 0x55f26475df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d1450 0x55f26475d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/1828152120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f2646d0b40 0x55f26473cd60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/1828152120 >> 172.21.15.129:0/1828152120 conn(0x55f2645a6ac0 msgr2=0x55f26475f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/1828152120 shutdown_connections 2024-09-17T03:50:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.421+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/1828152120 wait complete. 2024-09-17T03:50:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.422+0000 7f6ad8f20f40 1 Processor -- start 2024-09-17T03:50:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.422+0000 7f6ad8f20f40 1 -- start start 2024-09-17T03:50:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d0b40 0x55f26473bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f2646d1450 0x55f2646c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 0x55f2646c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f2646cfc00 con 0x55f264755510 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f26473fef0 con 0x55f2646d0b40 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad8f20f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55f264741190 con 0x55f2646d1450 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 0x55f2646c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 0x55f2646c7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47404/0 (socket says 172.21.15.129:47404) 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad2200640 1 -- 172.21.15.129:0/783607756 learned_addr learned my addr 172.21.15.129:0/783607756 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6acbe00640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f2646d1450 0x55f2646c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.424+0000 7f6ad1800640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d0b40 0x55f26473bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 -- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f2646d1450 msgr2=0x55f2646c6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f2646d1450 0x55f2646c6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 -- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d0b40 msgr2=0x55f26473bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d0b40 0x55f26473bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 -- 172.21.15.129:0/783607756 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2646861c0 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad1800640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f2646d0b40 0x55f26473bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad2200640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 0x55f2646c7230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00b580 tx=0x7f6acc00df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6aca000640 1 -- 172.21.15.129:0/783607756 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00f900 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6aca000640 1 -- 172.21.15.129:0/783607756 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6acc00fb90 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6aca000640 1 -- 172.21.15.129:0/783607756 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc017420 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f2647ea210 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.425+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2647ea4d0 con 0x55f264755510 2024-09-17T03:50:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.427+0000 7f6aca000640 1 -- 172.21.15.129:0/783607756 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6acc0175c0 con 0x55f264755510 2024-09-17T03:50:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.428+0000 7f6aca000640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ab0077570 0x7f6ab0079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.428+0000 7f6aca000640 1 -- 172.21.15.129:0/783607756 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f6acc09a330 con 0x55f264755510 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6ad1800640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ab0077570 0x7f6ab0079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x55f2647f8750 0x55f26475a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x55f26475a940 con 0x55f2647f8750 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6acbe00640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x55f2647f8750 0x55f26475a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6ad1800640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ab0077570 0x7f6ab0079a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4046600 tx=0x7f6ac4004940 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.429+0000 7f6acbe00640 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x55f2647f8750 0x55f26475a400 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6acbe00640 1 -- 172.21.15.129:0/783607756 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(1 10000000000.00000000 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x7f6abc00b7a0 con 0x55f2647f8750 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr: error stat-ing cephfs_metadata/10000000000.00000000: (2) No such file or directory 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x55f2647f8750 msgr2=0x55f26475a400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x55f2647f8750 0x55f26475a400 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ab0077570 msgr2=0x7f6ab0079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ab0077570 0x7f6ab0079a80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4046600 tx=0x7f6ac4004940 comp rx=0 tx=0).stop 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 msgr2=0x55f2646c7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 --2- 172.21.15.129:0/783607756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f264755510 0x55f2646c7230 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00b580 tx=0x7f6acc00df90 comp rx=0 tx=0).stop 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad2200640 1 -- 172.21.15.129:0/783607756 reap_dead start 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 shutdown_connections 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 >> 172.21.15.129:0/783607756 conn(0x55f2645a6ac0 msgr2=0x55f26473c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 shutdown_connections 2024-09-17T03:50:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.430+0000 7f6ad8f20f40 1 -- 172.21.15.129:0/783607756 wait complete. 2024-09-17T03:50:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:50:51.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.660+0000 7f1857800640 1 -- 172.21.15.129:0/1595953248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500ca800 msgr2=0x7f18500cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.660+0000 7f1857800640 1 --2- 172.21.15.129:0/1595953248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500ca800 0x7f18500cba00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1850070dc0 tx=0x7f184402a560 comp rx=0 tx=0).stop 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.660+0000 7f1857800640 1 -- 172.21.15.129:0/1595953248 shutdown_connections 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.660+0000 7f1857800640 1 --2- 172.21.15.129:0/1595953248 >> v2:172.21.15.181:3300/0 conn(0x7f18500706b0 0x7f18500735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.660+0000 7f1857800640 1 --2- 172.21.15.129:0/1595953248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500ca800 0x7f18500cba00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.661+0000 7f1857800640 1 -- 172.21.15.129:0/1595953248 >> 172.21.15.129:0/1595953248 conn(0x7f18500b1cb0 msgr2=0x7f18500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.661+0000 7f1857800640 1 -- 172.21.15.129:0/1595953248 shutdown_connections 2024-09-17T03:50:51.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.661+0000 7f1857800640 1 -- 172.21.15.129:0/1595953248 wait complete. 2024-09-17T03:50:51.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.663+0000 7f1857800640 1 Processor -- start 2024-09-17T03:50:51.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.663+0000 7f1857800640 1 -- start start 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18500ca800 0x7f18501f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18500c4960 0x7f18500d00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1850073ec0 con 0x7f18500cad30 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1850073d40 con 0x7f18500ca800 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.664+0000 7f1857800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1850074040 con 0x7f18500c4960 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47430/0 (socket says 172.21.15.129:47430) 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 -- 172.21.15.129:0/1352094008 learned_addr learned my addr 172.21.15.129:0/1352094008 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:51.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f1854e00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18500ca800 0x7f18501f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f1855800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18500c4960 0x7f18500d00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 -- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18500c4960 msgr2=0x7f18500d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18500c4960 0x7f18500d00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 -- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18500ca800 msgr2=0x7f18501f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18500ca800 0x7f18501f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 -- 172.21.15.129:0/1352094008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185009ab60 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.665+0000 7f184fe00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f183c00b420 tx=0x7f183c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.666+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c011830 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.666+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f183c016070 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.666+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183c011d60 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.666+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185009ae40 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.666+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185009b400 con 0x7f18500cad30 2024-09-17T03:50:51.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.667+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18500c4de0 con 0x7f18500cad30 2024-09-17T03:50:51.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.668+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f183c01b460 con 0x7f18500cad30 2024-09-17T03:50:51.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.669+0000 7f184cc00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 0x7f183007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:51.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.669+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f183c09a7f0 con 0x7f18500cad30 2024-09-17T03:50:51.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.670+0000 7f1854e00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 0x7f183007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.670+0000 7f1854e00640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 0x7f183007b0a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:51.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.673+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f183c020020 con 0x7f18500cad30 2024-09-17T03:50:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.926+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f18500745d0 con 0x7f18500cad30 2024-09-17T03:50:51.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.926+0000 7f184cc00640 1 -- 172.21.15.129:0/1352094008 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1658 (secure 0 0 0) 0x7f183c062540 con 0x7f18500cad30 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772825718784, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 268394496, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 268394496, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00034716923255473375, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T03:50:51.927 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772825718784, 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 268394496, 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 268394496, 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00034716923255473375 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:51.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6179849353648024e-06, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367073296384 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:51.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 4, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 4383248, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 24, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 8624, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 8830976, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.2028755008941516e-05, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367073296384 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:50:51.930 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 5, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 0, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 0, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 0, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 0, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 0, 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367073296384 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:50:51.931 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:50:51.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 msgr2=0x7f183007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 0x7f183007b0a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1844033000 tx=0x7f1844004730 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 msgr2=0x7f18500c4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f183c00b420 tx=0x7f183c00b8e0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 shutdown_connections 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1830078b90 0x7f183007b0a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18500c4960 0x7f18500d00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18500cad30 0x7f18500c4420 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 --2- 172.21.15.129:0/1352094008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18500ca800 0x7f18501f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 >> 172.21.15.129:0/1352094008 conn(0x7f18500b1cb0 msgr2=0x7f185015b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 shutdown_connections 2024-09-17T03:50:51.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:51.929+0000 7f1857800640 1 -- 172.21.15.129:0/1352094008 wait complete. 2024-09-17T03:50:51.950 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-17T03:50:51.950 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:50:51.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:50:52.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:50:52.122 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:50:52.123 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:50:52.123 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:52.123 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:52.158 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:52.158 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:52.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 -- 172.21.15.129:0/3011268344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5358003a30 msgr2=0x7f5358003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 --2- 172.21.15.129:0/3011268344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5358003a30 0x7f5358003ea0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f536400c2e0 tx=0x7f5358005830 comp rx=0 tx=0).stop 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5373400640 1 -- 172.21.15.129:0/3011268344 reap_dead start 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 -- 172.21.15.129:0/3011268344 shutdown_connections 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 -- 172.21.15.129:0/3011268344 >> 172.21.15.129:0/3011268344 conn(0x7f53740b1cb0 msgr2=0x7f53740b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 -- 172.21.15.129:0/3011268344 shutdown_connections 2024-09-17T03:50:52.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.499+0000 7f5379800640 1 -- 172.21.15.129:0/3011268344 wait complete. 2024-09-17T03:50:52.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.501+0000 7f5379800640 1 Processor -- start 2024-09-17T03:50:52.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.502+0000 7f5379800640 1 -- start start 2024-09-17T03:50:52.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 0x7f537415e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:52.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53741571a0 0x7f537406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f53740cbb80 con 0x7f537406f3d0 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f53740cba00 con 0x7f53741571a0 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5379800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f53740cbd00 con 0x7f5358003a30 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5373400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5373400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47446/0 (socket says 172.21.15.129:47446) 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.503+0000 7f5373400640 1 -- 172.21.15.129:0/4187044050 learned_addr learned my addr 172.21.15.129:0/4187044050 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5372000640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53741571a0 0x7f537406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5372a00640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 0x7f537415e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 -- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 msgr2=0x7f537415e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 0x7f537415e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 -- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53741571a0 msgr2=0x7f537406ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53741571a0 0x7f537406ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 -- 172.21.15.129:0/4187044050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f537406f7e0 con 0x7f537406f3d0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5372a00640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 0x7f537415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5373400640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f53740cc780 tx=0x7f536c001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c011840 con 0x7f537406f3d0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f536c018070 con 0x7f537406f3d0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c011ca0 con 0x7f537406f3d0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.504+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5374157f00 con 0x7f537406f3d0 2024-09-17T03:50:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.505+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f537415bf80 con 0x7f537406f3d0 2024-09-17T03:50:52.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.506+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f537406ffa0 con 0x7f537406f3d0 2024-09-17T03:50:52.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.507+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f536c01d460 con 0x7f537406f3d0 2024-09-17T03:50:52.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.508+0000 7f536aa00640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 0x7f535407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:52.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.508+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f536c09b570 con 0x7f537406f3d0 2024-09-17T03:50:52.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.509+0000 7f5372a00640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 0x7f535407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:52.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.510+0000 7f5372a00640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 0x7f535407b0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5364033000 tx=0x7f53640019f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:52.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.512+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f537406f7e0 con 0x7f537406f3d0 2024-09-17T03:50:52.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.706+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f537415ab60 con 0x7f537406f3d0 2024-09-17T03:50:52.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.707+0000 7f536aa00640 1 -- 172.21.15.129:0/4187044050 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f536c0639b0 con 0x7f537406f3d0 2024-09-17T03:50:52.709 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 msgr2=0x7f535407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 0x7f535407b0d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5364033000 tx=0x7f53640019f0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 msgr2=0x7f53741f6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f53740cc780 tx=0x7f536c001c10 comp rx=0 tx=0).stop 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 shutdown_connections 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078bc0 0x7f535407b0d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537406f3d0 0x7f53741f6300 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53741571a0 0x7f537406ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 --2- 172.21.15.129:0/4187044050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5358003a30 0x7f537415e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:52.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.709+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 >> 172.21.15.129:0/4187044050 conn(0x7f53740b1cb0 msgr2=0x7f53741568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:52.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.710+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 shutdown_connections 2024-09-17T03:50:52.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:52.710+0000 7f5379800640 1 -- 172.21.15.129:0/4187044050 wait complete. 2024-09-17T03:50:52.732 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:50:52.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:52.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:50:52.820 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:50:52.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:50:52.820 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:50:52.871 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:50:52.872 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:52.872 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:52.908 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:50:52.909 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:50:52.909 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:50:52.909 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:52.947 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:50:52.947 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:50:52.947 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:50:52.947 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:50:52.984 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:50:52.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 -- 172.21.15.129:0/975449895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380cd3e0 msgr2=0x7f4738070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 --2- 172.21.15.129:0/975449895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380cd3e0 0x7f4738070180 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f47380cffa0 tx=0x7f472002a540 comp rx=0 tx=0).stop 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 -- 172.21.15.129:0/975449895 shutdown_connections 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 --2- 172.21.15.129:0/975449895 >> v2:172.21.15.181:3300/0 conn(0x7f47380706c0 0x7f4738072ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 --2- 172.21.15.129:0/975449895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380cd3e0 0x7f4738070180 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 --2- 172.21.15.129:0/975449895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380af780 0x7f47380b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 -- 172.21.15.129:0/975449895 >> 172.21.15.129:0/975449895 conn(0x7f473806d7d0 msgr2=0x7f47380b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:53.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 -- 172.21.15.129:0/975449895 shutdown_connections 2024-09-17T03:50:53.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.206+0000 7f473f800640 1 -- 172.21.15.129:0/975449895 wait complete. 2024-09-17T03:50:53.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.209+0000 7f473f800640 1 Processor -- start 2024-09-17T03:50:53.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.209+0000 7f473f800640 1 -- start start 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47380af780 0x7f47380adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 0x7f47380ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4738073930 con 0x7f47380706c0 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47380737b0 con 0x7f47380af780 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.210+0000 7f473f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4738073ab0 con 0x7f47380cd3e0 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47468/0 (socket says 172.21.15.129:47468) 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 -- 172.21.15.129:0/1932072095 learned_addr learned my addr 172.21.15.129:0/1932072095 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f472fe00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47380af780 0x7f47380adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473d800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 0x7f47380ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 -- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 msgr2=0x7f47380ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 0x7f47380ae460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 -- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47380af780 msgr2=0x7f47380adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47380af780 0x7f47380adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 -- 172.21.15.129:0/1932072095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f473813f370 con 0x7f47380706c0 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473d800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 0x7f47380ae460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:53.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.211+0000 7f473ce00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f472800e7c0 tx=0x7f472800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:53.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.212+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472800ccc0 con 0x7f47380706c0 2024-09-17T03:50:53.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.212+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f472800ce80 con 0x7f47380706c0 2024-09-17T03:50:53.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.212+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47380c9580 con 0x7f47380706c0 2024-09-17T03:50:53.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.212+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728005280 con 0x7f47380706c0 2024-09-17T03:50:53.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.212+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47380c9860 con 0x7f47380706c0 2024-09-17T03:50:53.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.213+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47380c9370 con 0x7f47380706c0 2024-09-17T03:50:53.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.214+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4728005420 con 0x7f47380706c0 2024-09-17T03:50:53.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.215+0000 7f4735600640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 0x7f471407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.215+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f472809b530 con 0x7f47380706c0 2024-09-17T03:50:53.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.215+0000 7f472fe00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 0x7f471407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.216+0000 7f472fe00640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 0x7f471407b0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4720033000 tx=0x7f4720004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:53.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.219+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4728063a00 con 0x7f47380706c0 2024-09-17T03:50:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.440+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47380b39c0 con 0x7f47380706c0 2024-09-17T03:50:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.441+0000 7f4735600640 1 -- 172.21.15.129:0/1932072095 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f4728063300 con 0x7f47380706c0 2024-09-17T03:50:53.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:53.441 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 msgr2=0x7f471407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 0x7f471407b0a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4720033000 tx=0x7f4720004730 comp rx=0 tx=0).stop 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 msgr2=0x7f473809d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f472800e7c0 tx=0x7f472800ec80 comp rx=0 tx=0).stop 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 shutdown_connections 2024-09-17T03:50:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4714078b90 0x7f471407b0a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47380cd3e0 0x7f47380ae460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47380af780 0x7f47380adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 --2- 172.21.15.129:0/1932072095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47380706c0 0x7f473809d3d0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 >> 172.21.15.129:0/1932072095 conn(0x7f473806d7d0 msgr2=0x7f47380b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 shutdown_connections 2024-09-17T03:50:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.444+0000 7f473f800640 1 -- 172.21.15.129:0/1932072095 wait complete. 2024-09-17T03:50:53.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.684+0000 7f4447a00640 1 -- 172.21.15.129:0/1156577273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 msgr2=0x7f44400ccbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.684+0000 7f4447a00640 1 --2- 172.21.15.129:0/1156577273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400ccbc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f4430005fd0 tx=0x7f443002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.684+0000 7f4447a00640 1 -- 172.21.15.129:0/1156577273 shutdown_connections 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.684+0000 7f4447a00640 1 --2- 172.21.15.129:0/1156577273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400ccbc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.684+0000 7f4447a00640 1 -- 172.21.15.129:0/1156577273 >> 172.21.15.129:0/1156577273 conn(0x7f444006d4f0 msgr2=0x7f4440071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.685+0000 7f4447a00640 1 -- 172.21.15.129:0/1156577273 shutdown_connections 2024-09-17T03:50:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.685+0000 7f4447a00640 1 -- 172.21.15.129:0/1156577273 wait complete. 2024-09-17T03:50:53.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.687+0000 7f4447a00640 1 Processor -- start 2024-09-17T03:50:53.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.687+0000 7f4447a00640 1 -- start start 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 0x7f4440151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44400cf320 0x7f44400b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f44400d15b0 con 0x7f44401600b0 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f44400d1430 con 0x7f44400b7cc0 2024-09-17T03:50:53.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.688+0000 7f4447a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f44400d1730 con 0x7f44400cf320 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47492/0 (socket says 172.21.15.129:47492) 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 -- 172.21.15.129:0/2642674694 learned_addr learned my addr 172.21.15.129:0/2642674694 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f443fe00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44400cf320 0x7f44400b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445000640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 0x7f4440151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 -- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44400cf320 msgr2=0x7f44400b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44400cf320 0x7f44400b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 -- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 msgr2=0x7f4440151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 0x7f4440151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 -- 172.21.15.129:0/2642674694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4440151640 con 0x7f44401600b0 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445000640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 0x7f4440151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.689+0000 7f4445a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f443800b9e0 tx=0x7f443800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:53.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.690+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4438003b70 con 0x7f44401600b0 2024-09-17T03:50:53.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.690+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4438003e00 con 0x7f44401600b0 2024-09-17T03:50:53.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.690+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444006f080 con 0x7f44401600b0 2024-09-17T03:50:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.690+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443800cb80 con 0x7f44401600b0 2024-09-17T03:50:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.690+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444006f640 con 0x7f44401600b0 2024-09-17T03:50:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.691+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44400ccbc0 con 0x7f44401600b0 2024-09-17T03:50:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.692+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4438004000 con 0x7f44401600b0 2024-09-17T03:50:53.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.693+0000 7f443cc00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 0x7f442007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:53.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.693+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f443809abc0 con 0x7f44401600b0 2024-09-17T03:50:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.694+0000 7f4445000640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 0x7f442007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:53.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.695+0000 7f4445000640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 0x7f442007b0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4430033000 tx=0x7f4430004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:53.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.701+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4438063000 con 0x7f44401600b0 2024-09-17T03:50:53.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.922+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f444015f9e0 con 0x7f44401600b0 2024-09-17T03:50:53.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.922+0000 7f443cc00640 1 -- 172.21.15.129:0/2642674694 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f4438062900 con 0x7f44401600b0 2024-09-17T03:50:53.923 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:53.923 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:53.923 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:53.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 msgr2=0x7f442007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 0x7f442007b0a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4430033000 tx=0x7f4430004730 comp rx=0 tx=0).stop 2024-09-17T03:50:53.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 msgr2=0x7f44400c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f443800b9e0 tx=0x7f443800bea0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 shutdown_connections 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4420078b90 0x7f442007b0a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44401600b0 0x7f44400c9c00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44400cf320 0x7f44400b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 --2- 172.21.15.129:0/2642674694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44400b7cc0 0x7f4440151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 >> 172.21.15.129:0/2642674694 conn(0x7f444006d4f0 msgr2=0x7f44400b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 shutdown_connections 2024-09-17T03:50:53.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:53.925+0000 7f4447a00640 1 -- 172.21.15.129:0/2642674694 wait complete. 2024-09-17T03:50:53.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 -- 172.21.15.129:0/4209169405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06ee90 msgr2=0x7f290c0b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 --2- 172.21.15.129:0/4209169405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06ee90 0x7f290c0b47f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2900005fd0 tx=0x7f290002aa90 comp rx=0 tx=0).stop 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 -- 172.21.15.129:0/4209169405 shutdown_connections 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 --2- 172.21.15.129:0/4209169405 >> v2:172.21.15.181:3300/0 conn(0x7f290c0b4d30 0x7f290c0ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 --2- 172.21.15.129:0/4209169405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06ee90 0x7f290c0b47f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.165+0000 7f2914400640 1 -- 172.21.15.129:0/4209169405 >> 172.21.15.129:0/4209169405 conn(0x7f290c06d7d0 msgr2=0x7f290c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.166+0000 7f2914400640 1 -- 172.21.15.129:0/4209169405 shutdown_connections 2024-09-17T03:50:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.166+0000 7f2914400640 1 -- 172.21.15.129:0/4209169405 wait complete. 2024-09-17T03:50:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.168+0000 7f2914400640 1 Processor -- start 2024-09-17T03:50:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.168+0000 7f2914400640 1 -- start start 2024-09-17T03:50:54.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06f2a0 0x7f290c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 0x7f290c09aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f290c15feb0 con 0x7f290c0b4d30 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f290c15fd30 con 0x7f290c06ee90 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.169+0000 7f2914400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f290c0cd050 con 0x7f290c06f2a0 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2912400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 0x7f290c09aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:39004/0 (socket says 172.21.15.129:39004) 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 -- 172.21.15.129:0/320542103 learned_addr learned my addr 172.21.15.129:0/320542103 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911000640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06f2a0 0x7f290c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 -- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06f2a0 msgr2=0x7f290c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06f2a0 0x7f290c0c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 -- 172.21.15.129:0/320542103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 msgr2=0x7f290c09aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 0x7f290c09aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2911a00640 1 -- 172.21.15.129:0/320542103 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c0aef50 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.170+0000 7f2912400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 0x7f290c09aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2911a00640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2900000f80 tx=0x7f2900003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900044070 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2900004250 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c0af230 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2900004660 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.171+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c09d570 con 0x7f290c06ee90 2024-09-17T03:50:54.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.172+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c0b47f0 con 0x7f290c06ee90 2024-09-17T03:50:54.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.174+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2900047070 con 0x7f290c06ee90 2024-09-17T03:50:54.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.175+0000 7f2909600640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 0x7f28ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.175+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f29000bc100 con 0x7f290c06ee90 2024-09-17T03:50:54.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.175+0000 7f2911000640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 0x7f28ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.176+0000 7f2911000640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 0x7f28ec07b0a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f290c1615c0 tx=0x7f28f8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.179+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2900032030 con 0x7f290c06ee90 2024-09-17T03:50:54.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.363+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f290c0750f0 con 0x7f290c06ee90 2024-09-17T03:50:54.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.366+0000 7f2909600640 1 -- 172.21.15.129:0/320542103 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16777 (secure 0 0 0) 0x7f2900083ff0 con 0x7f290c06ee90 2024-09-17T03:50:54.365 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:54.366 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":33,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:25.122284+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:50:21.527684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-17T03:50:22.575211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.367+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 msgr2=0x7f28ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.367+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 0x7f28ec07b0a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f290c1615c0 tx=0x7f28f8005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 msgr2=0x7f290c152ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2900000f80 tx=0x7f2900003ce0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 shutdown_connections 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f28ec078b90 0x7f28ec07b0a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f290c0b4d30 0x7f290c09aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f290c06f2a0 0x7f290c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 --2- 172.21.15.129:0/320542103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f290c06ee90 0x7f290c152ac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 >> 172.21.15.129:0/320542103 conn(0x7f290c06d7d0 msgr2=0x7f290c161640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 shutdown_connections 2024-09-17T03:50:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.368+0000 7f2914400640 1 -- 172.21.15.129:0/320542103 wait complete. 2024-09-17T03:50:54.390 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:50:54.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 -- 172.21.15.129:0/1393667289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540ccb30 msgr2=0x7fe5540ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 --2- 172.21.15.129:0/1393667289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540ccb30 0x7fe5540ccf40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe554071d60 tx=0x7fe54002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 -- 172.21.15.129:0/1393667289 shutdown_connections 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 --2- 172.21.15.129:0/1393667289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540ccb30 0x7fe5540ccf40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 --2- 172.21.15.129:0/1393667289 >> v2:172.21.15.181:3300/0 conn(0x7fe5540cf6f0 0x7fe5540cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.612+0000 7fe558a00640 1 -- 172.21.15.129:0/1393667289 >> 172.21.15.129:0/1393667289 conn(0x7fe5540b1cb0 msgr2=0x7fe5540b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.613+0000 7fe558a00640 1 -- 172.21.15.129:0/1393667289 shutdown_connections 2024-09-17T03:50:54.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.613+0000 7fe558a00640 1 -- 172.21.15.129:0/1393667289 wait complete. 2024-09-17T03:50:54.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.615+0000 7fe558a00640 1 Processor -- start 2024-09-17T03:50:54.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.615+0000 7fe558a00640 1 -- start start 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5540ccb30 0x7fe554161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5540ce020 0x7fe554153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe5540cecc0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe5540ceb40 con 0x7fe5540ce020 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.616+0000 7fe558a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe5540cf1c0 con 0x7fe5540ccb30 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59782/0 (socket says 172.21.15.129:59782) 2024-09-17T03:50:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 -- 172.21.15.129:0/3587817094 learned_addr learned my addr 172.21.15.129:0/3587817094 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe551400640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5540ce020 0x7fe554153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe551e00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5540ccb30 0x7fe554161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 -- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5540ccb30 msgr2=0x7fe554161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5540ccb30 0x7fe554161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 -- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5540ce020 msgr2=0x7fe554153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5540ce020 0x7fe554153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 -- 172.21.15.129:0/3587817094 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5541620f0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.617+0000 7fe552800640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00c6b0 tx=0x7fe54c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.618+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c003b70 con 0x7fe5540cf6f0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.618+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe54c003d30 con 0x7fe5540cf6f0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.618+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c005330 con 0x7fe5540cf6f0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.618+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5541f5b50 con 0x7fe5540cf6f0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.618+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5541f6110 con 0x7fe5540cf6f0 2024-09-17T03:50:54.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.619+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5541f58c0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.620+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe54c0054d0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.621+0000 7fe54a000640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 0x7fe53407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:54.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.622+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe54c09a940 con 0x7fe5540cf6f0 2024-09-17T03:50:54.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.622+0000 7fe551e00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 0x7fe53407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:54.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.623+0000 7fe551e00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 0x7fe53407b0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe554071290 tx=0x7fe54400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:54.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.626+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe5540cd480 con 0x7fe5540cf6f0 2024-09-17T03:50:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.845+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5540b57c0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.846+0000 7fe54a000640 1 -- 172.21.15.129:0/3587817094 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7fe5540b57c0 con 0x7fe5540cf6f0 2024-09-17T03:50:54.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:54.846 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:54.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 msgr2=0x7fe53407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 0x7fe53407b0a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe554071290 tx=0x7fe54400b2e0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 msgr2=0x7fe554074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00c6b0 tx=0x7fe54c00cb70 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 shutdown_connections 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe534078b90 0x7fe53407b0a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5540cf6f0 0x7fe554074620 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5540ce020 0x7fe554153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 --2- 172.21.15.129:0/3587817094 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5540ccb30 0x7fe554161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 >> 172.21.15.129:0/3587817094 conn(0x7fe5540b1cb0 msgr2=0x7fe5540736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 shutdown_connections 2024-09-17T03:50:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:54.848+0000 7fe558a00640 1 -- 172.21.15.129:0/3587817094 wait complete. 2024-09-17T03:50:54.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:50:55.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 -- 172.21.15.129:0/1261665008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf950 msgr2=0x7f36c406f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 --2- 172.21.15.129:0/1261665008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf950 0x7f36c406f3b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f36c4070dc0 tx=0x7f36bc02a980 comp rx=0 tx=0).stop 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 -- 172.21.15.129:0/1261665008 shutdown_connections 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 --2- 172.21.15.129:0/1261665008 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf950 0x7f36c406f3b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 --2- 172.21.15.129:0/1261665008 >> v2:172.21.15.181:3300/0 conn(0x7f36c40cf540 0x7f36c406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 --2- 172.21.15.129:0/1261665008 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c415ca50 0x7f36c40cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 -- 172.21.15.129:0/1261665008 >> 172.21.15.129:0/1261665008 conn(0x7f36c406d410 msgr2=0x7f36c40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:55.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 -- 172.21.15.129:0/1261665008 shutdown_connections 2024-09-17T03:50:55.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.091+0000 7f36ca400640 1 -- 172.21.15.129:0/1261665008 wait complete. 2024-09-17T03:50:55.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.093+0000 7f36ca400640 1 Processor -- start 2024-09-17T03:50:55.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.093+0000 7f36ca400640 1 -- start start 2024-09-17T03:50:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c40cf540 0x7f36c4153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c40cf950 0x7f36c40b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36c415dba0 con 0x7f36c415ca50 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36c415da20 con 0x7f36c40cf950 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36ca400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36c415dd20 con 0x7f36c40cf540 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36c9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59792/0 (socket says 172.21.15.129:59792) 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36c9a00640 1 -- 172.21.15.129:0/2826058315 learned_addr learned my addr 172.21.15.129:0/2826058315 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36bbe00640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c40cf950 0x7f36c40b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.095+0000 7f36c9000640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c40cf540 0x7f36c4153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 -- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c40cf540 msgr2=0x7f36c4153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c40cf540 0x7f36c4153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 -- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c40cf950 msgr2=0x7f36c40b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c40cf950 0x7f36c40b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 -- 172.21.15.129:0/2826058315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c4135be0 con 0x7f36c415ca50 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c9a00640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009d60 tx=0x7f36bc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc034040 con 0x7f36c415ca50 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36bc004540 con 0x7f36c415ca50 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c4135ec0 con 0x7f36c415ca50 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc045480 con 0x7f36c415ca50 2024-09-17T03:50:55.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.096+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c4136480 con 0x7f36c415ca50 2024-09-17T03:50:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.097+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c415e1d0 con 0x7f36c415ca50 2024-09-17T03:50:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.099+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f36bc037020 con 0x7f36c415ca50 2024-09-17T03:50:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.100+0000 7f36c1600640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 0x7f36a007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.100+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f36bc0bc570 con 0x7f36c415ca50 2024-09-17T03:50:55.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.100+0000 7f36c9000640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 0x7f36a007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.101+0000 7f36c9000640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 0x7f36a007b0d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f36b400bb50 tx=0x7f36b400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:55.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.104+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f36bc0842d0 con 0x7f36c415ca50 2024-09-17T03:50:55.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.324+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36c4135970 con 0x7f36c415ca50 2024-09-17T03:50:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.325+0000 7f36c1600640 1 -- 172.21.15.129:0/2826058315 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5232 (secure 0 0 0) 0x7f36bc088d00 con 0x7f36c415ca50 2024-09-17T03:50:55.325 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:55.325 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":17,"btime":"2024-09-17T03:50:27:139869+0000","default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:50:23.616751+0000","modified":"2024-09-17T03:50:26.618188+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4845},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4845":{"gid":4845,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1447431306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1447431306},{"type":"v1","addr":"172.21.15.129:6835","nonce":1447431306}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4845,"qdb_cluster":[4845]},"id":2}]} 2024-09-17T03:50:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 17 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 msgr2=0x7f36a007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 0x7f36a007b0d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f36b400bb50 tx=0x7f36b400b2e0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 msgr2=0x7f36c40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009d60 tx=0x7f36bc003f10 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 shutdown_connections 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a0078bc0 0x7f36a007b0d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c415ca50 0x7f36c40b42d0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c40cf950 0x7f36c40b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 --2- 172.21.15.129:0/2826058315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c40cf540 0x7f36c4153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.327+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 >> 172.21.15.129:0/2826058315 conn(0x7f36c406d410 msgr2=0x7f36c4073870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.328+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 shutdown_connections 2024-09-17T03:50:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.328+0000 7f36ca400640 1 -- 172.21.15.129:0/2826058315 wait complete. 2024-09-17T03:50:55.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:50:55.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 -- 172.21.15.129:0/2307149878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12fc0062d0 msgr2=0x7f12fc00cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 --2- 172.21.15.129:0/2307149878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12fc0062d0 0x7f12fc00cad0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f130c00c610 tx=0x7f12fc006940 comp rx=0 tx=0).stop 2024-09-17T03:50:55.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 -- 172.21.15.129:0/2307149878 shutdown_connections 2024-09-17T03:50:55.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 --2- 172.21.15.129:0/2307149878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12fc0062d0 0x7f12fc00cad0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 -- 172.21.15.129:0/2307149878 >> 172.21.15.129:0/2307149878 conn(0x7f131406d410 msgr2=0x7f13140b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:55.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 -- 172.21.15.129:0/2307149878 shutdown_connections 2024-09-17T03:50:55.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.572+0000 7f131a800640 1 -- 172.21.15.129:0/2307149878 wait complete. 2024-09-17T03:50:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.574+0000 7f131a800640 1 Processor -- start 2024-09-17T03:50:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.574+0000 7f131a800640 1 -- start start 2024-09-17T03:50:55.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12fc0062d0 0x7f13140c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1314147920 0x7f1314159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f13140732e0 con 0x7f1314147510 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1314073160 con 0x7f12fc0062d0 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f131a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1314073460 con 0x7f1314147920 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1318a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1318a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59802/0 (socket says 172.21.15.129:59802) 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1318a00640 1 -- 172.21.15.129:0/1399312462 learned_addr learned my addr 172.21.15.129:0/1399312462 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1319400640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12fc0062d0 0x7f13140c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1319e00640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1314147920 0x7f1314159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1318a00640 1 -- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1314147920 msgr2=0x7f1314159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.576+0000 7f1318a00640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1314147920 0x7f1314159a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1318a00640 1 -- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12fc0062d0 msgr2=0x7f13140c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1318a00640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12fc0062d0 0x7f13140c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1318a00640 1 -- 172.21.15.129:0/1399312462 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13141f8940 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1318a00640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1300004cf0 tx=0x7f1300001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300015020 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1300011960 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13141f8c20 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1300011db0 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.577+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13141f91e0 con 0x7f1314147510 2024-09-17T03:50:55.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.578+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1314073640 con 0x7f1314147510 2024-09-17T03:50:55.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.580+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1300018020 con 0x7f1314147510 2024-09-17T03:50:55.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.580+0000 7f1311600640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 0x7f12f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:55.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.581+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f130009b860 con 0x7f1314147510 2024-09-17T03:50:55.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.581+0000 7f1319400640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 0x7f12f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:55.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.582+0000 7f1319400640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 0x7f12f407b0a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1308009b10 tx=0x7f1308005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:55.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.585+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f130001f020 con 0x7f1314147510 2024-09-17T03:50:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.769+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13140c5360 con 0x7f1314147510 2024-09-17T03:50:55.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.770+0000 7f1311600640 1 -- 172.21.15.129:0/1399312462 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16777 (secure 0 0 0) 0x7f1300063c00 con 0x7f1314147510 2024-09-17T03:50:55.771 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:50:55.771 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":33,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:25.122284+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:50:21.527684+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-17T03:50:22.575211+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 msgr2=0x7f12f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 0x7f12f407b0a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1308009b10 tx=0x7f1308005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 msgr2=0x7f13141594c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f1300004cf0 tx=0x7f1300001730 comp rx=0 tx=0).stop 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 shutdown_connections 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12f4078b90 0x7f12f407b0a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1314147920 0x7f1314159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1314147510 0x7f13141594c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 --2- 172.21.15.129:0/1399312462 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12fc0062d0 0x7f13140c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 >> 172.21.15.129:0/1399312462 conn(0x7f131406d410 msgr2=0x7f13140b3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 shutdown_connections 2024-09-17T03:50:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:55.773+0000 7f131a800640 1 -- 172.21.15.129:0/1399312462 wait complete. 2024-09-17T03:50:55.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.018+0000 7fc398a00640 1 -- 172.21.15.129:0/842196802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940704e0 msgr2=0x7fc3940cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.018+0000 7fc398a00640 1 --2- 172.21.15.129:0/842196802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940704e0 0x7fc3940cc830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc3940c71a0 tx=0x7fc38c02aa30 comp rx=0 tx=0).stop 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 -- 172.21.15.129:0/842196802 shutdown_connections 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 --2- 172.21.15.129:0/842196802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940704e0 0x7fc3940cc830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 --2- 172.21.15.129:0/842196802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940cad30 0x7fc39406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 --2- 172.21.15.129:0/842196802 >> v2:172.21.15.181:3300/0 conn(0x7fc3940ca800 0x7fc3940cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 -- 172.21.15.129:0/842196802 >> 172.21.15.129:0/842196802 conn(0x7fc3940b1cb0 msgr2=0x7fc3940b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:56.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 -- 172.21.15.129:0/842196802 shutdown_connections 2024-09-17T03:50:56.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.019+0000 7fc398a00640 1 -- 172.21.15.129:0/842196802 wait complete. 2024-09-17T03:50:56.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.021+0000 7fc398a00640 1 Processor -- start 2024-09-17T03:50:56.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.021+0000 7fc398a00640 1 -- start start 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940704e0 0x7fc39414f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 0x7fc39415b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3940ccf80 con 0x7fc3940ca800 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc3940cec60 con 0x7fc3940cad30 2024-09-17T03:50:56.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.022+0000 7fc398a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3940cd100 con 0x7fc3940704e0 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59816/0 (socket says 172.21.15.129:59816) 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 -- 172.21.15.129:0/2647994087 learned_addr learned my addr 172.21.15.129:0/2647994087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391e00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940704e0 0x7fc39414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc392800640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 0x7fc39415b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 -- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940704e0 msgr2=0x7fc39414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940704e0 0x7fc39414f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 -- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 msgr2=0x7fc39415b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 0x7fc39415b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 -- 172.21.15.129:0/2647994087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3941585b0 con 0x7fc3940ca800 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc392800640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 0x7fc39415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:50:56.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.023+0000 7fc391400640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e7c0 tx=0x7fc38000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:56.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.024+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38000cc60 con 0x7fc3940ca800 2024-09-17T03:50:56.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.024+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc38000ce20 con 0x7fc3940ca800 2024-09-17T03:50:56.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.024+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3941f7260 con 0x7fc3940ca800 2024-09-17T03:50:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.024+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc380005280 con 0x7fc3940ca800 2024-09-17T03:50:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.024+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc394159da0 con 0x7fc3940ca800 2024-09-17T03:50:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.025+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc394157e80 con 0x7fc3940ca800 2024-09-17T03:50:56.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.026+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc380005420 con 0x7fc3940ca800 2024-09-17T03:50:56.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.027+0000 7fc38a000640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 0x7fc37407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.028+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc38009b570 con 0x7fc3940ca800 2024-09-17T03:50:56.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.028+0000 7fc391e00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 0x7fc37407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.029+0000 7fc391e00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 0x7fc37407b0a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3940cb250 tx=0x7fc384005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:56.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.032+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc380063a30 con 0x7fc3940ca800 2024-09-17T03:50:56.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.258+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc3940cd690 con 0x7fc3940ca800 2024-09-17T03:50:56.259 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:50:56.259+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:50:56.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.261+0000 7fc38a000640 1 -- 172.21.15.129:0/2647994087 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v18) ==== 161+0+0 (secure 0 0 0) 0x7fc380063330 con 0x7fc3940ca800 2024-09-17T03:50:56.261 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:50:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.265+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 msgr2=0x7fc37407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.265+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 0x7fc37407b0a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc3940cb250 tx=0x7fc384005f90 comp rx=0 tx=0).stop 2024-09-17T03:50:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.265+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 msgr2=0x7fc394152e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.265+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fc38000e7c0 tx=0x7fc38000ec80 comp rx=0 tx=0).stop 2024-09-17T03:50:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 shutdown_connections 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc374078b90 0x7fc37407b0a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3940cad30 0x7fc39415b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3940ca800 0x7fc394152e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 --2- 172.21.15.129:0/2647994087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3940704e0 0x7fc39414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 >> 172.21.15.129:0/2647994087 conn(0x7fc3940b1cb0 msgr2=0x7fc3940d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 shutdown_connections 2024-09-17T03:50:56.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.266+0000 7fc398a00640 1 -- 172.21.15.129:0/2647994087 wait complete. 2024-09-17T03:50:56.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:50:56.407 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:50:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 -- 172.21.15.129:0/2020483360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380cc1e0 msgr2=0x7f0a380af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 --2- 172.21.15.129:0/2020483360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380cc1e0 0x7f0a380af3c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a28005fd0 tx=0x7f0a2802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 -- 172.21.15.129:0/2020483360 shutdown_connections 2024-09-17T03:50:56.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 --2- 172.21.15.129:0/2020483360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380cc1e0 0x7f0a380af3c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 -- 172.21.15.129:0/2020483360 >> 172.21.15.129:0/2020483360 conn(0x7f0a3806d7d0 msgr2=0x7f0a380b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:56.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 -- 172.21.15.129:0/2020483360 shutdown_connections 2024-09-17T03:50:56.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.521+0000 7f0a3e000640 1 -- 172.21.15.129:0/2020483360 wait complete. 2024-09-17T03:50:56.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.523+0000 7f0a3e000640 1 Processor -- start 2024-09-17T03:50:56.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.523+0000 7f0a3e000640 1 -- start start 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a380cc1e0 0x7f0a3809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a3815d8a0 0x7f0a3814d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a38153520 con 0x7f0a380ccb50 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0a381533a0 con 0x7f0a3815d8a0 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a3e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0a381536a0 con 0x7f0a380cc1e0 2024-09-17T03:50:56.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59860/0 (socket says 172.21.15.129:59860) 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 -- 172.21.15.129:0/3674973017 learned_addr learned my addr 172.21.15.129:0/3674973017 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a37400640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a380cc1e0 0x7f0a3809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a37e00640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a3815d8a0 0x7f0a3814d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 -- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a380cc1e0 msgr2=0x7f0a3809d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a380cc1e0 0x7f0a3809d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 -- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a3815d8a0 msgr2=0x7f0a3814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a3815d8a0 0x7f0a3814d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.525+0000 7f0a2ea00640 1 -- 172.21.15.129:0/3674973017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a381f95a0 con 0x7f0a380ccb50 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a2ea00640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c3a0 tx=0x7f0a2000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20017070 con 0x7f0a380ccb50 2024-09-17T03:50:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a200053c0 con 0x7f0a380ccb50 2024-09-17T03:50:56.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20003850 con 0x7f0a380ccb50 2024-09-17T03:50:56.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a381f9880 con 0x7f0a380ccb50 2024-09-17T03:50:56.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.526+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a381f9e40 con 0x7f0a380ccb50 2024-09-17T03:50:56.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.527+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a38153b50 con 0x7f0a380ccb50 2024-09-17T03:50:56.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.529+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0a2001a070 con 0x7f0a380ccb50 2024-09-17T03:50:56.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.529+0000 7f0a2fe00640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 0x7f0a1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:56.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.530+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f0a2009a180 con 0x7f0a380ccb50 2024-09-17T03:50:56.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.530+0000 7f0a37400640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 0x7f0a1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:56.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.531+0000 7f0a37400640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 0x7f0a1407b0a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a28033000 tx=0x7f0a28004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:56.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.534+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a20062550 con 0x7f0a380ccb50 2024-09-17T03:50:56.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:56.754+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f0a380b0280 con 0x7f0a380ccb50 2024-09-17T03:50:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.269+0000 7f0a2fe00640 1 -- 172.21.15.129:0/3674973017 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f0a20061e50 con 0x7f0a380ccb50 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 msgr2=0x7f0a1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 0x7f0a1407b0a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0a28033000 tx=0x7f0a28004730 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 msgr2=0x7f0a3814d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000c3a0 tx=0x7f0a2000c860 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 shutdown_connections 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a14078b90 0x7f0a1407b0a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a3815d8a0 0x7f0a3814d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a380ccb50 0x7f0a3814d340 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 --2- 172.21.15.129:0/3674973017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a380cc1e0 0x7f0a3809d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.271+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 >> 172.21.15.129:0/3674973017 conn(0x7f0a3806d7d0 msgr2=0x7f0a3815aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.272+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 shutdown_connections 2024-09-17T03:50:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.272+0000 7f0a3e000640 1 -- 172.21.15.129:0/3674973017 wait complete. 2024-09-17T03:50:57.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:50:57.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 -- 172.21.15.129:0/3227884719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 msgr2=0x7f688415d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 --2- 172.21.15.129:0/3227884719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688415d4b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f688415c9c0 tx=0x7f687802a560 comp rx=0 tx=0).stop 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 -- 172.21.15.129:0/3227884719 shutdown_connections 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 --2- 172.21.15.129:0/3227884719 >> v2:172.21.15.181:3300/0 conn(0x7f6884073590 0x7f68840cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 --2- 172.21.15.129:0/3227884719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688415d4b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 -- 172.21.15.129:0/3227884719 >> 172.21.15.129:0/3227884719 conn(0x7f688406ee70 msgr2=0x7f688406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 -- 172.21.15.129:0/3227884719 shutdown_connections 2024-09-17T03:50:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.524+0000 7f6889e00640 1 -- 172.21.15.129:0/3227884719 wait complete. 2024-09-17T03:50:57.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.526+0000 7f6889e00640 1 Processor -- start 2024-09-17T03:50:57.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.527+0000 7f6889e00640 1 -- start start 2024-09-17T03:50:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 0x7f68841f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 0x7f688413b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68840cf2d0 con 0x7f6884074d80 2024-09-17T03:50:57.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f68840cf150 con 0x7f68840b5310 2024-09-17T03:50:57.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6889e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f68840cf450 con 0x7f6884073590 2024-09-17T03:50:57.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6883e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:57.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6883e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59882/0 (socket says 172.21.15.129:59882) 2024-09-17T03:50:57.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.528+0000 7f6883e00640 1 -- 172.21.15.129:0/2235864256 learned_addr learned my addr 172.21.15.129:0/2235864256 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6888a00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 0x7f68841f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6889400640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 0x7f688413b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 -- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 msgr2=0x7f68841f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 0x7f68841f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 -- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 msgr2=0x7f688413b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 0x7f688413b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:57.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 -- 172.21.15.129:0/2235864256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68840aa580 con 0x7f6884074d80 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6888a00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 0x7f68841f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6889400640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 0x7f688413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6883e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f687000d6a0 tx=0x7f687000db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870017070 con 0x7f6884074d80 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6870009b10 con 0x7f6884074d80 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68840c50a0 con 0x7f6884074d80 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.529+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870010460 con 0x7f6884074d80 2024-09-17T03:50:57.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.530+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6884144e10 con 0x7f6884074d80 2024-09-17T03:50:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.531+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68840cf920 con 0x7f6884074d80 2024-09-17T03:50:57.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.532+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f687001a070 con 0x7f6884074d80 2024-09-17T03:50:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.533+0000 7f6880c00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 0x7f686407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.533+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f687009ae10 con 0x7f6884074d80 2024-09-17T03:50:57.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.533+0000 7f6888a00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 0x7f686407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:57.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.534+0000 7f6888a00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 0x7f686407b0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6878033000 tx=0x7f6878004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:57.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.537+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f687009e030 con 0x7f6884074d80 2024-09-17T03:50:57.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:57.719+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f68841f6130 con 0x7f6884074d80 2024-09-17T03:50:58.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.434+0000 7f6880c00640 1 -- 172.21.15.129:0/2235864256 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f6870063270 con 0x7f6884074d80 2024-09-17T03:50:58.434 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 msgr2=0x7f686407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 0x7f686407b0d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6878033000 tx=0x7f6878004730 comp rx=0 tx=0).stop 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 msgr2=0x7f688413ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f687000d6a0 tx=0x7f687000db60 comp rx=0 tx=0).stop 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 shutdown_connections 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6864078bc0 0x7f686407b0d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68840b5310 0x7f688413b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6884074d80 0x7f688413ab40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 --2- 172.21.15.129:0/2235864256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6884073590 0x7f68841f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.438+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 >> 172.21.15.129:0/2235864256 conn(0x7f688406ee70 msgr2=0x7f68840cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.439+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 shutdown_connections 2024-09-17T03:50:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.439+0000 7f6889e00640 1 -- 172.21.15.129:0/2235864256 wait complete. 2024-09-17T03:50:58.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:50:58.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 -- 172.21.15.129:0/3043396801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3404004800 msgr2=0x7f3404004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:58.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 --2- 172.21.15.129:0/3043396801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3404004800 0x7f3404004c10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f341c0c3a50 tx=0x7f3404005830 comp rx=0 tx=0).stop 2024-09-17T03:50:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 -- 172.21.15.129:0/3043396801 shutdown_connections 2024-09-17T03:50:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 --2- 172.21.15.129:0/3043396801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3404004800 0x7f3404004c10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 -- 172.21.15.129:0/3043396801 >> 172.21.15.129:0/3043396801 conn(0x7f341c0b1c40 msgr2=0x7f341c0b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 -- 172.21.15.129:0/3043396801 shutdown_connections 2024-09-17T03:50:58.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.691+0000 7f3423a00640 1 -- 172.21.15.129:0/3043396801 wait complete. 2024-09-17T03:50:58.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.693+0000 7f3423a00640 1 Processor -- start 2024-09-17T03:50:58.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.694+0000 7f3423a00640 1 -- start start 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3404004800 0x7f341c14eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f341c09cce0 0x7f341c09d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0b30c0 con 0x7f341c0c9360 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0b2f40 con 0x7f3404004800 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3423a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f341c0b3240 con 0x7f341c09cce0 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3421a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3421a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59904/0 (socket says 172.21.15.129:59904) 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3421a00640 1 -- 172.21.15.129:0/257961022 learned_addr learned my addr 172.21.15.129:0/257961022 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f341be00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f341c09cce0 0x7f341c09d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:58.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.695+0000 7f3421000640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3404004800 0x7f341c14eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 -- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f341c09cce0 msgr2=0x7f341c09d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f341c09cce0 0x7f341c09d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 -- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3404004800 msgr2=0x7f341c14eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3404004800 0x7f341c14eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 -- 172.21.15.129:0/257961022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0c9cb0 con 0x7f341c0c9360 2024-09-17T03:50:58.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3421a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ca7b0 tx=0x7f3414001a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414011870 con 0x7f341c0c9360 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3414017070 con 0x7f341c0c9360 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c1f6340 con 0x7f341c0c9360 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414011da0 con 0x7f341c0c9360 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.696+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c1f6900 con 0x7f341c0c9360 2024-09-17T03:50:58.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.698+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c072ff0 con 0x7f341c0c9360 2024-09-17T03:50:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.699+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f341401c460 con 0x7f341c0c9360 2024-09-17T03:50:58.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.700+0000 7f3418c00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 0x7f33fc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.700+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5357+0+0 (secure 0 0 0) 0x7f341409b320 con 0x7f341c0c9360 2024-09-17T03:50:58.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.700+0000 7f3421000640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 0x7f33fc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:58.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.701+0000 7f3421000640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 0x7f33fc07b0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3410006b50 tx=0x7f3410001080 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:58.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.704+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f341c0c9cb0 con 0x7f341c0c9360 2024-09-17T03:50:58.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:58.889+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f341c0008d0 con 0x7f341c0c9360 2024-09-17T03:50:59.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.441+0000 7f3418c00640 1 -- 172.21.15.129:0/257961022 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7f3414068e20 con 0x7f341c0c9360 2024-09-17T03:50:59.441 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:50:59.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.444+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 msgr2=0x7f33fc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.444+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 0x7f33fc07b0d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3410006b50 tx=0x7f3410001080 comp rx=0 tx=0).stop 2024-09-17T03:50:59.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.444+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 msgr2=0x7f341c0c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.444+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ca7b0 tx=0x7f3414001a90 comp rx=0 tx=0).stop 2024-09-17T03:50:59.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 shutdown_connections 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33fc078bc0 0x7f33fc07b0d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f341c0c9360 0x7f341c0c9770 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f341c09cce0 0x7f341c09d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 --2- 172.21.15.129:0/257961022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3404004800 0x7f341c14eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 >> 172.21.15.129:0/257961022 conn(0x7f341c0b1c40 msgr2=0x7f341c073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 shutdown_connections 2024-09-17T03:50:59.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.445+0000 7f3423a00640 1 -- 172.21.15.129:0/257961022 wait complete. 2024-09-17T03:50:59.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:50:59.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 -- 172.21.15.129:0/1795256783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c0cc1e0 msgr2=0x7ff80c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 --2- 172.21.15.129:0/1795256783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c0cc1e0 0x7ff80c0af3c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff800009830 tx=0x7ff80002a510 comp rx=0 tx=0).stop 2024-09-17T03:50:59.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 -- 172.21.15.129:0/1795256783 shutdown_connections 2024-09-17T03:50:59.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 --2- 172.21.15.129:0/1795256783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c15d8a0 0x7ff80c0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 --2- 172.21.15.129:0/1795256783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c0cc1e0 0x7ff80c0af3c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 -- 172.21.15.129:0/1795256783 >> 172.21.15.129:0/1795256783 conn(0x7ff80c06d7d0 msgr2=0x7ff80c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:59.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 -- 172.21.15.129:0/1795256783 shutdown_connections 2024-09-17T03:50:59.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.698+0000 7ff813800640 1 -- 172.21.15.129:0/1795256783 wait complete. 2024-09-17T03:50:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.700+0000 7ff813800640 1 Processor -- start 2024-09-17T03:50:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.700+0000 7ff813800640 1 -- start start 2024-09-17T03:50:59.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c0cc1e0 0x7ff80c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:59.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c15d8a0 0x7ff80c1574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c153520 con 0x7ff80c0ccb50 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff80c1533a0 con 0x7ff80c15d8a0 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff813800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff80c1536a0 con 0x7ff80c0cc1e0 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:59.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59910/0 (socket says 172.21.15.129:59910) 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 -- 172.21.15.129:0/2498612775 learned_addr learned my addr 172.21.15.129:0/2498612775 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff811800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c15d8a0 0x7ff80c1574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff810e00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c0cc1e0 0x7ff80c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 -- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c0cc1e0 msgr2=0x7ff80c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c0cc1e0 0x7ff80c1438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 -- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c15d8a0 msgr2=0x7ff80c1574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c15d8a0 0x7ff80c1574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.702+0000 7ff80be00640 1 -- 172.21.15.129:0/2498612775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80c1f47f0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff80be00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9f0 tx=0x7ff7f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800cd80 con 0x7ff80c0ccb50 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff7f8004020 con 0x7ff80c0ccb50 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80c1f3d30 con 0x7ff80c0ccb50 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8004470 con 0x7ff80c0ccb50 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.703+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80c1f4510 con 0x7ff80c0ccb50 2024-09-17T03:50:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.704+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff80c0d0a20 con 0x7ff80c0ccb50 2024-09-17T03:50:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.705+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff7f80035f0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.706+0000 7ff808c00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 0x7ff7ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:50:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.706+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7ff7f809b2b0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.707+0000 7ff810e00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 0x7ff7ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:50:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.708+0000 7ff810e00640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 0x7ff7ec07b0a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff800034000 tx=0x7ff800004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:50:59.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.711+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff7f80633a0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.893+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7ff80c1f3fc0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.897+0000 7ff808c00640 1 -- 172.21.15.129:0/2498612775 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v33) ==== 71+0+0 (secure 0 0 0) 0x7ff7f8067dd0 con 0x7ff80c0ccb50 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 msgr2=0x7ff7ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 0x7ff7ec07b0a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff800034000 tx=0x7ff800004730 comp rx=0 tx=0).stop 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 msgr2=0x7ff80c1f37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800e9f0 tx=0x7ff7f800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 shutdown_connections 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff7ec078b90 0x7ff7ec07b0a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff80c15d8a0 0x7ff80c1574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff80c0ccb50 0x7ff80c1f37f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 --2- 172.21.15.129:0/2498612775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff80c0cc1e0 0x7ff80c1438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:50:59.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.899+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 >> 172.21.15.129:0/2498612775 conn(0x7ff80c06d7d0 msgr2=0x7ff80c0b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:50:59.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.900+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 shutdown_connections 2024-09-17T03:50:59.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:50:59.900+0000 7ff813800640 1 -- 172.21.15.129:0/2498612775 wait complete. 2024-09-17T03:50:59.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion' 2024-09-17T03:51:00.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 -- 172.21.15.129:0/4278721601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0b5be0 msgr2=0x7f22cc0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 --2- 172.21.15.129:0/4278721601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0b5be0 0x7f22cc0caa80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f22c4005fd0 tx=0x7f22c402aa90 comp rx=0 tx=0).stop 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 -- 172.21.15.129:0/4278721601 shutdown_connections 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 --2- 172.21.15.129:0/4278721601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0b5be0 0x7f22cc0caa80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 -- 172.21.15.129:0/4278721601 >> 172.21.15.129:0/4278721601 conn(0x7f22cc06d870 msgr2=0x7f22cc06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 -- 172.21.15.129:0/4278721601 shutdown_connections 2024-09-17T03:51:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.141+0000 7f22d0a00640 1 -- 172.21.15.129:0/4278721601 wait complete. 2024-09-17T03:51:00.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.143+0000 7f22d0a00640 1 Processor -- start 2024-09-17T03:51:00.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.144+0000 7f22d0a00640 1 -- start start 2024-09-17T03:51:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 0x7f22cc0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 0x7f22cc0b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc15d510 con 0x7f22cc0735d0 2024-09-17T03:51:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f22cc15d390 con 0x7f22cc1436a0 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22d0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f22cc15d690 con 0x7f22cc0af700 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22ca800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22ca800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59926/0 (socket says 172.21.15.129:59926) 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.145+0000 7f22ca800640 1 -- 172.21.15.129:0/3472945244 learned_addr learned my addr 172.21.15.129:0/3472945244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22c1e00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 0x7f22cc0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22cb200640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 0x7f22cc0b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 -- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 msgr2=0x7f22cc0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 0x7f22cc0b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 -- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 msgr2=0x7f22cc0b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 0x7f22cc0b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 -- 172.21.15.129:0/3472945244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22cc1f87c0 con 0x7f22cc0735d0 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22cb200640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 0x7f22cc0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22c1e00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 0x7f22cc0b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22ca800640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00c3f0 tx=0x7f22bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc017070 con 0x7f22cc0735d0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22bc0035f0 con 0x7f22cc0735d0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22bc003a40 con 0x7f22cc0735d0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.146+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22cc0b2a90 con 0x7f22cc0735d0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.147+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22cc1444d0 con 0x7f22cc0735d0 2024-09-17T03:51:00.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.148+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22cc0caa80 con 0x7f22cc0735d0 2024-09-17T03:51:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.149+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f22bc01a070 con 0x7f22cc0735d0 2024-09-17T03:51:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.150+0000 7f22c3400640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 0x7f22a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.150+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f22bc099ea0 con 0x7f22cc0735d0 2024-09-17T03:51:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.151+0000 7f22c1e00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 0x7f22a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.151+0000 7f22c1e00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 0x7f22a807b0a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22cc147d90 tx=0x7f22b4008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.154+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f22bc062540 con 0x7f22cc0735d0 2024-09-17T03:51:00.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.414+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]} v 0) -- 0x7f22cc150890 con 0x7f22cc0735d0 2024-09-17T03:51:00.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.443+0000 7f22c3400640 1 -- 172.21.15.129:0/3472945244 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dir_deletion"]}]=0 v98) ==== 132+0+0 (secure 0 0 0) 0x7f22bc061e40 con 0x7f22cc0735d0 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 msgr2=0x7f22a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 0x7f22a807b0a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f22cc147d90 tx=0x7f22b4008040 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 msgr2=0x7f22cc1f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f22bc00c3f0 tx=0x7f22bc00c8b0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 shutdown_connections 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a8078b90 0x7f22a807b0a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22cc1436a0 0x7f22cc0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22cc0af700 0x7f22cc0b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 --2- 172.21.15.129:0/3472945244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22cc0735d0 0x7f22cc1f9250 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.445+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 >> 172.21.15.129:0/3472945244 conn(0x7f22cc06d870 msgr2=0x7f22cc0ceeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.446+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 shutdown_connections 2024-09-17T03:51:00.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.446+0000 7f22d0a00640 1 -- 172.21.15.129:0/3472945244 wait complete. 2024-09-17T03:51:00.469 INFO:tasks.cephfs_test_runner:That when deleting a bunch of dentries and the containing ... ok 2024-09-17T03:51:00.470 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). 2024-09-17T03:51:00.470 INFO:tasks.cephfs_test_runner:test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:51:00.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.690+0000 7fdf74200640 1 -- 172.21.15.129:0/3576801159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c15d8a0 msgr2=0x7fdf6c0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.690+0000 7fdf74200640 1 --2- 172.21.15.129:0/3576801159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c0b40b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fdf58005fd0 tx=0x7fdf5802a530 comp rx=0 tx=0).stop 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 -- 172.21.15.129:0/3576801159 shutdown_connections 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 --2- 172.21.15.129:0/3576801159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c0b40b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 --2- 172.21.15.129:0/3576801159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 0x7fdf6c0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 -- 172.21.15.129:0/3576801159 >> 172.21.15.129:0/3576801159 conn(0x7fdf6c06d7d0 msgr2=0x7fdf6c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 -- 172.21.15.129:0/3576801159 shutdown_connections 2024-09-17T03:51:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.691+0000 7fdf74200640 1 -- 172.21.15.129:0/3576801159 wait complete. 2024-09-17T03:51:00.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.693+0000 7fdf74200640 1 Processor -- start 2024-09-17T03:51:00.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.693+0000 7fdf74200640 1 -- start start 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 0x7fdf6c156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c09cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c153520 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6c1533a0 con 0x7fdf6c15d8a0 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.694+0000 7fdf74200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6c1536a0 con 0x7fdf6c0cc1e0 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59932/0 (socket says 172.21.15.129:59932) 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 -- 172.21.15.129:0/4046988809 learned_addr learned my addr 172.21.15.129:0/4046988809 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf71800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 0x7fdf6c156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf72200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c09cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 -- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 msgr2=0x7fdf6c156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 0x7fdf6c156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 -- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf6c15d8a0 msgr2=0x7fdf6c09cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c09cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.695+0000 7fdf70e00640 1 -- 172.21.15.129:0/4046988809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf6c152720 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf70e00640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdf58034040 tx=0x7fdf580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5803e030 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf58003d90 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf6c1f46a0 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58035040 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.696+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf6c1f40e0 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.697+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf6c1f3b70 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.698+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdf58041030 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.700+0000 7fdf69600640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 0x7fdf4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.700+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fdf580bb590 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.700+0000 7fdf71800640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 0x7fdf4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:00.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.701+0000 7fdf71800640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 0x7fdf4c07b0a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf60004630 tx=0x7fdf6000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:00.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.705+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdf58083670 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.888+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdf6c153b50 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.889+0000 7fdf69600640 1 -- 172.21.15.129:0/4046988809 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+13082 (secure 0 0 0) 0x7fdf580880a0 con 0x7fdf6c0ccb50 2024-09-17T03:51:00.889 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:00.889 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":36,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:50:59.435615+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/1447431306":"2024-09-18T03:50:56.260569+0000","172.21.15.129:6833/1447431306":"2024-09-18T03:50:56.260569+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 msgr2=0x7fdf4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 0x7fdf4c07b0a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdf60004630 tx=0x7fdf6000b2e0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 msgr2=0x7fdf6c151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fdf58034040 tx=0x7fdf580038a0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 shutdown_connections 2024-09-17T03:51:00.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf4c078b90 0x7fdf4c07b0a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf6c15d8a0 0x7fdf6c09cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf6c0ccb50 0x7fdf6c151d70 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 --2- 172.21.15.129:0/4046988809 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf6c0cc1e0 0x7fdf6c156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 >> 172.21.15.129:0/4046988809 conn(0x7fdf6c06d7d0 msgr2=0x7fdf6c0b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 shutdown_connections 2024-09-17T03:51:00.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:00.892+0000 7fdf74200640 1 -- 172.21.15.129:0/4046988809 wait complete. 2024-09-17T03:51:00.918 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:51:00.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:51:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 -- 172.21.15.129:0/3646016492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c15ca50 msgr2=0x7fdd1c0cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 --2- 172.21.15.129:0/3646016492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c0cc830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0b5220 tx=0x7fdd0c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 -- 172.21.15.129:0/3646016492 shutdown_connections 2024-09-17T03:51:01.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 --2- 172.21.15.129:0/3646016492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 0x7fdd1c06ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 --2- 172.21.15.129:0/3646016492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c0cc830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 -- 172.21.15.129:0/3646016492 >> 172.21.15.129:0/3646016492 conn(0x7fdd1c06d410 msgr2=0x7fdd1c0b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:01.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 -- 172.21.15.129:0/3646016492 shutdown_connections 2024-09-17T03:51:01.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.142+0000 7fdd22400640 1 -- 172.21.15.129:0/3646016492 wait complete. 2024-09-17T03:51:01.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.144+0000 7fdd22400640 1 Processor -- start 2024-09-17T03:51:01.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.144+0000 7fdd22400640 1 -- start start 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 0x7fdd1c1e9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c15de10 con 0x7fdd1c0cf950 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1c15df90 con 0x7fdd1c15ca50 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd22400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1c15e110 con 0x7fdd1c0cf540 2024-09-17T03:51:01.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd13e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd13e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59950/0 (socket says 172.21.15.129:59950) 2024-09-17T03:51:01.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd13e00640 1 -- 172.21.15.129:0/4017897733 learned_addr learned my addr 172.21.15.129:0/4017897733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:01.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd21000640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 0x7fdd1c1e9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.146+0000 7fdd21a00640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 -- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 msgr2=0x7fdd1c1e9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 0x7fdd1c1e9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 -- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 msgr2=0x7fdd1c159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 -- 172.21.15.129:0/4017897733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1c1f3cb0 con 0x7fdd1c0cf950 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd21a00640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd13e00640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400c970 tx=0x7fdd0400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd04007bb0 con 0x7fdd1c0cf950 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd04007e40 con 0x7fdd1c0cf950 2024-09-17T03:51:01.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1c1f3f90 con 0x7fdd1c0cf950 2024-09-17T03:51:01.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0400f6b0 con 0x7fdd1c0cf950 2024-09-17T03:51:01.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.147+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1c1f4550 con 0x7fdd1c0cf950 2024-09-17T03:51:01.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.148+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1c06ee70 con 0x7fdd1c0cf950 2024-09-17T03:51:01.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.150+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdd0400f850 con 0x7fdd1c0cf950 2024-09-17T03:51:01.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.150+0000 7fdd19600640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 0x7fdcf807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.151+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7fdd0409b1d0 con 0x7fdd1c0cf950 2024-09-17T03:51:01.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.151+0000 7fdd21000640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 0x7fdcf807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.152+0000 7fdd21000640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 0x7fdcf807b0a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c033000 tx=0x7fdd0c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:01.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.155+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdd04014030 con 0x7fdd1c0cf950 2024-09-17T03:51:01.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.337+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdd1c06fb50 con 0x7fdd1c0cf950 2024-09-17T03:51:01.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.337+0000 7fdd19600640 1 -- 172.21.15.129:0/4017897733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7fdd040639c0 con 0x7fdd1c0cf950 2024-09-17T03:51:01.337 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:51:01.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 msgr2=0x7fdcf807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 0x7fdcf807b0a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c033000 tx=0x7fdd0c004730 comp rx=0 tx=0).stop 2024-09-17T03:51:01.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 msgr2=0x7fdd1c1e7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdd0400c970 tx=0x7fdd0400ce30 comp rx=0 tx=0).stop 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 shutdown_connections 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdcf8078b90 0x7fdcf807b0a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd1c15ca50 0x7fdd1c159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd1c0cf950 0x7fdd1c1e7190 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 --2- 172.21.15.129:0/4017897733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd1c0cf540 0x7fdd1c1e9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 >> 172.21.15.129:0/4017897733 conn(0x7fdd1c06d410 msgr2=0x7fdd1c073a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 shutdown_connections 2024-09-17T03:51:01.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.340+0000 7fdd22400640 1 -- 172.21.15.129:0/4017897733 wait complete. 2024-09-17T03:51:01.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 -- 172.21.15.129:0/3179250942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50073170 msgr2=0x7f9c5015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 --2- 172.21.15.129:0/3179250942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50073170 0x7f9c5015c780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f9c500b4b40 tx=0x7f9c4002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 -- 172.21.15.129:0/3179250942 shutdown_connections 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 --2- 172.21.15.129:0/3179250942 >> v2:172.21.15.181:3300/0 conn(0x7f9c50070440 0x7f9c50074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 --2- 172.21.15.129:0/3179250942 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 0x7f9c50074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 --2- 172.21.15.129:0/3179250942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50073170 0x7f9c5015c780 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 -- 172.21.15.129:0/3179250942 >> 172.21.15.129:0/3179250942 conn(0x7f9c500b1cb0 msgr2=0x7f9c500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:01.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 -- 172.21.15.129:0/3179250942 shutdown_connections 2024-09-17T03:51:01.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.583+0000 7f9c57400640 1 -- 172.21.15.129:0/3179250942 wait complete. 2024-09-17T03:51:01.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.585+0000 7f9c57400640 1 Processor -- start 2024-09-17T03:51:01.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.585+0000 7f9c57400640 1 -- start start 2024-09-17T03:51:01.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 0x7f9c5009b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c50073170 0x7f9c500c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5006f690 con 0x7f9c50070440 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5006f510 con 0x7f9c50073170 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c57400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9c5006f810 con 0x7f9c50070030 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59972/0 (socket says 172.21.15.129:59972) 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 -- 172.21.15.129:0/3326316939 learned_addr learned my addr 172.21.15.129:0/3326316939 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c54a00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 0x7f9c5009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c55400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c50073170 0x7f9c500c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 -- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 msgr2=0x7f9c5009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 0x7f9c5009b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 -- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c50073170 msgr2=0x7f9c500c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c50073170 0x7f9c500c7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:01.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.587+0000 7f9c47e00640 1 -- 172.21.15.129:0/3326316939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5009be90 con 0x7f9c50070440 2024-09-17T03:51:01.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c47e00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800c970 tx=0x7f9c3800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:01.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c38007bb0 con 0x7f9c50070440 2024-09-17T03:51:01.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c38007e40 con 0x7f9c50070440 2024-09-17T03:51:01.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c500caac0 con 0x7f9c50070440 2024-09-17T03:51:01.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3800f6b0 con 0x7f9c50070440 2024-09-17T03:51:01.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.588+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c500cb080 con 0x7f9c50070440 2024-09-17T03:51:01.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.589+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c500ca820 con 0x7f9c50070440 2024-09-17T03:51:01.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.590+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9c3800f850 con 0x7f9c50070440 2024-09-17T03:51:01.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.591+0000 7f9c4d600640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 0x7f9c2c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:01.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.591+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f9c3809b190 con 0x7f9c50070440 2024-09-17T03:51:01.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.592+0000 7f9c54a00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 0x7f9c2c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:01.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.592+0000 7f9c54a00640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 0x7f9c2c07b0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c40033000 tx=0x7f9c40004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:01.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.596+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9c500ca820 con 0x7f9c50070440 2024-09-17T03:51:01.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:01.849+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) -- 0x7f9c5015c780 con 0x7f9c50070440 2024-09-17T03:51:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.451+0000 7f9c4d600640 1 -- 172.21.15.129:0/3326316939 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v100) ==== 136+0+0 (secure 0 0 0) 0x7f9c38014030 con 0x7f9c50070440 2024-09-17T03:51:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.453+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 msgr2=0x7f9c2c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.453+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 0x7f9c2c07b0d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9c40033000 tx=0x7f9c40004730 comp rx=0 tx=0).stop 2024-09-17T03:51:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.453+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 msgr2=0x7f9c500cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.453+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800c970 tx=0x7f9c3800ce30 comp rx=0 tx=0).stop 2024-09-17T03:51:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 shutdown_connections 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c2c078bc0 0x7f9c2c07b0d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c50073170 0x7f9c500c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c50070440 0x7f9c500cbb60 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 --2- 172.21.15.129:0/3326316939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c50070030 0x7f9c5009b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 >> 172.21.15.129:0/3326316939 conn(0x7f9c500b1cb0 msgr2=0x7f9c5015ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 shutdown_connections 2024-09-17T03:51:02.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.454+0000 7f9c57400640 1 -- 172.21.15.129:0/3326316939 wait complete. 2024-09-17T03:51:02.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 -- 172.21.15.129:0/1412538736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce8004800 msgr2=0x7efce8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 --2- 172.21.15.129:0/1412538736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce8004800 0x7efce8004c10 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7efcec001800 tx=0x7efce8005830 comp rx=0 tx=0).stop 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd09400640 1 -- 172.21.15.129:0/1412538736 reap_dead start 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 -- 172.21.15.129:0/1412538736 shutdown_connections 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 -- 172.21.15.129:0/1412538736 >> 172.21.15.129:0/1412538736 conn(0x7efd040b1cb0 msgr2=0x7efd040b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:02.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 -- 172.21.15.129:0/1412538736 shutdown_connections 2024-09-17T03:51:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.700+0000 7efd0b400640 1 -- 172.21.15.129:0/1412538736 wait complete. 2024-09-17T03:51:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.702+0000 7efd0b400640 1 Processor -- start 2024-09-17T03:51:02.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.703+0000 7efd0b400640 1 -- start start 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce8004800 0x7efd041f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd040cb9b0 0x7efd04164580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd041655c0 con 0x7efd040cc300 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efd04165440 con 0x7efd040cb9b0 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd0b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efd04165740 con 0x7efce8004800 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd09400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd09400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59988/0 (socket says 172.21.15.129:59988) 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd09400640 1 -- 172.21.15.129:0/831133643 learned_addr learned my addr 172.21.15.129:0/831133643 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.704+0000 7efd08a00640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce8004800 0x7efd041f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efcfbe00640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd040cb9b0 0x7efd04164580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:02.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 -- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce8004800 msgr2=0x7efd041f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce8004800 0x7efd041f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 -- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd040cb9b0 msgr2=0x7efd04164580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd040cb9b0 0x7efd04164580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 -- 172.21.15.129:0/831133643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040ca910 con 0x7efd040cc300 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd09400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7efcfc005fd0 tx=0x7efcfc010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc003970 con 0x7efd040cc300 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.705+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcfc015070 con 0x7efd040cc300 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.706+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc004030 con 0x7efd040cc300 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.706+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd040cabf0 con 0x7efd040cc300 2024-09-17T03:51:02.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.706+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd040cb1b0 con 0x7efd040cc300 2024-09-17T03:51:02.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.707+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd04074640 con 0x7efd040cc300 2024-09-17T03:51:02.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.708+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efcfc0041d0 con 0x7efd040cc300 2024-09-17T03:51:02.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.709+0000 7efd01600640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 0x7efce007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:02.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.709+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7efcfc09b390 con 0x7efd040cc300 2024-09-17T03:51:02.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.709+0000 7efd08a00640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 0x7efce007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:02.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.710+0000 7efd08a00640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 0x7efce007b0a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcf4006c50 tx=0x7efcf40011c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:02.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.713+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efcfc063500 con 0x7efd040cc300 2024-09-17T03:51:02.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.897+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7efd040c3860 con 0x7efd040cc300 2024-09-17T03:51:02.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.897+0000 7efd01600640 1 -- 172.21.15.129:0/831133643 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v33) ==== 59+0+2294 (secure 0 0 0) 0x7efcfc067f30 con 0x7efd040cc300 2024-09-17T03:51:02.897 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:51:02.897 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:51:02.898 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:51:02.899 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:51:02.900 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.899+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 msgr2=0x7efce007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 0x7efce007b0a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcf4006c50 tx=0x7efcf40011c0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 msgr2=0x7efd04164ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7efcfc005fd0 tx=0x7efcfc010f40 comp rx=0 tx=0).stop 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 shutdown_connections 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efce0078b90 0x7efce007b0a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd040cc300 0x7efd04164ac0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd040cb9b0 0x7efd04164580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 --2- 172.21.15.129:0/831133643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce8004800 0x7efd041f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:02.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 >> 172.21.15.129:0/831133643 conn(0x7efd040b1cb0 msgr2=0x7efd040b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:02.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 shutdown_connections 2024-09-17T03:51:02.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:02.900+0000 7efd0b400640 1 -- 172.21.15.129:0/831133643 wait complete. 2024-09-17T03:51:02.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:51:03.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 -- 172.21.15.129:0/3573656032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 msgr2=0x7f22c0071250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 --2- 172.21.15.129:0/3573656032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c0071250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f22b0009830 tx=0x7f22b002a640 comp rx=0 tx=0).stop 2024-09-17T03:51:03.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 -- 172.21.15.129:0/3573656032 shutdown_connections 2024-09-17T03:51:03.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 --2- 172.21.15.129:0/3573656032 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c0071250 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 -- 172.21.15.129:0/3573656032 >> 172.21.15.129:0/3573656032 conn(0x7f22c00cdcb0 msgr2=0x7f22c00ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:03.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 -- 172.21.15.129:0/3573656032 shutdown_connections 2024-09-17T03:51:03.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.147+0000 7f22c6000640 1 -- 172.21.15.129:0/3573656032 wait complete. 2024-09-17T03:51:03.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.149+0000 7f22c6000640 1 Processor -- start 2024-09-17T03:51:03.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.149+0000 7f22c6000640 1 -- start start 2024-09-17T03:51:03.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22c00ce7d0 0x7f22c01f5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c01f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00d0540 con 0x7f22c015bc30 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f22c00d03c0 con 0x7f22c01600a0 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22c6000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f22c00d06c0 con 0x7f22c00ce7d0 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60002/0 (socket says 172.21.15.129:60002) 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 -- 172.21.15.129:0/299582873 learned_addr learned my addr 172.21.15.129:0/299582873 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bea00640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c01f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bf400640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22c00ce7d0 0x7f22c01f5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 -- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22c00ce7d0 msgr2=0x7f22c01f5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22c00ce7d0 0x7f22c01f5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 -- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 msgr2=0x7f22c01f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c01f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.151+0000 7f22bfe00640 1 -- 172.21.15.129:0/299582873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c015c580 con 0x7f22c015bc30 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22bfe00640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22b800c730 tx=0x7f22b800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8003b70 con 0x7f22c015bc30 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f22b8003d30 con 0x7f22c015bc30 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c00cb880 con 0x7f22c015bc30 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b80052d0 con 0x7f22c015bc30 2024-09-17T03:51:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.152+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c00cc060 con 0x7f22c015bc30 2024-09-17T03:51:03.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.153+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c0071250 con 0x7f22c015bc30 2024-09-17T03:51:03.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.154+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f22b8005680 con 0x7f22c015bc30 2024-09-17T03:51:03.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.155+0000 7f22b7400640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 0x7f22a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.155+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f22b809a940 con 0x7f22c015bc30 2024-09-17T03:51:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.156+0000 7f22bf400640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 0x7f22a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.156+0000 7f22bf400640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 0x7f22a007b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22b0034000 tx=0x7f22b0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:03.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.159+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f22c015c580 con 0x7f22c015bc30 2024-09-17T03:51:03.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.341+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f22c00b50d0 con 0x7f22c015bc30 2024-09-17T03:51:03.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.346+0000 7f22b7400640 1 -- 172.21.15.129:0/299582873 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) ==== 122+0+0 (secure 0 0 0) 0x7f22b8062a30 con 0x7f22c015bc30 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.348+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 msgr2=0x7f22a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.348+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 0x7f22a007b0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22b0034000 tx=0x7f22b0004730 comp rx=0 tx=0).stop 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.348+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 msgr2=0x7f22c015c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.348+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f22b800c730 tx=0x7f22b800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 shutdown_connections 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f22a0078b90 0x7f22a007b0a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22c015bc30 0x7f22c015c040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22c01600a0 0x7f22c01f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 --2- 172.21.15.129:0/299582873 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22c00ce7d0 0x7f22c01f5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 >> 172.21.15.129:0/299582873 conn(0x7f22c00cdcb0 msgr2=0x7f22c00ae640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:03.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 shutdown_connections 2024-09-17T03:51:03.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.349+0000 7f22c6000640 1 -- 172.21.15.129:0/299582873 wait complete. 2024-09-17T03:51:03.371 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:51:03.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:03.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:03.537 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:51:03.537 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:03.537 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:03.573 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:51:03.573 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:03.573 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:03.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:03.611 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:03.612 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:03.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:51:03.650 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:03.650 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:03.686 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:03.686 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:51:03.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:03.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 -- 172.21.15.129:0/747926563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5508004800 msgr2=0x7f5508004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 --2- 172.21.15.129:0/747926563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5508004800 0x7f5508004c10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c2e0 tx=0x7f5508005830 comp rx=0 tx=0).stop 2024-09-17T03:51:03.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 -- 172.21.15.129:0/747926563 shutdown_connections 2024-09-17T03:51:03.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 --2- 172.21.15.129:0/747926563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5508004800 0x7f5508004c10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 -- 172.21.15.129:0/747926563 >> 172.21.15.129:0/747926563 conn(0x7f552406d410 msgr2=0x7f55240b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:03.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 -- 172.21.15.129:0/747926563 shutdown_connections 2024-09-17T03:51:03.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.910+0000 7f5523200640 1 -- 172.21.15.129:0/747926563 wait complete. 2024-09-17T03:51:03.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.912+0000 7f5523200640 1 Processor -- start 2024-09-17T03:51:03.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.913+0000 7f5523200640 1 -- start start 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5508004800 0x7f5524140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55240c4e20 0x7f55240c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55240cf2f0 con 0x7f55240aee90 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55240cf170 con 0x7f5508004800 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5523200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f55240cf470 con 0x7f55240c4e20 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5522800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5522800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60006/0 (socket says 172.21.15.129:60006) 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5522800640 1 -- 172.21.15.129:0/340133565 learned_addr learned my addr 172.21.15.129:0/340133565 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.914+0000 7f5521e00640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5508004800 0x7f5524140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5519400640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55240c4e20 0x7f55240c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 -- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55240c4e20 msgr2=0x7f55240c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55240c4e20 0x7f55240c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 -- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5508004800 msgr2=0x7f5524140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:03.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5508004800 0x7f5524140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:03.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 -- 172.21.15.129:0/340133565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55240af7e0 con 0x7f55240aee90 2024-09-17T03:51:03.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5522800640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f551c005fd0 tx=0x7f551c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c003d80 con 0x7f55240aee90 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f551c004b20 con 0x7f55240aee90 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f551c004030 con 0x7f55240aee90 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55241f9730 con 0x7f55240aee90 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.915+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55241f8c50 con 0x7f55240aee90 2024-09-17T03:51:03.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.916+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552415ca80 con 0x7f55240aee90 2024-09-17T03:51:03.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.918+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f551c0041d0 con 0x7f55240aee90 2024-09-17T03:51:03.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.918+0000 7f551aa00640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 0x7f550007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:03.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.919+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f551c09b9d0 con 0x7f55240aee90 2024-09-17T03:51:03.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.919+0000 7f5521e00640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 0x7f550007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:03.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.920+0000 7f5521e00640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 0x7f550007b0a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f55240521d0 tx=0x7f5514006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:03.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:03.923+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f55240af7e0 con 0x7f55240aee90 2024-09-17T03:51:04.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.143+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f552415db90 con 0x7f55240aee90 2024-09-17T03:51:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.144+0000 7f551aa00640 1 -- 172.21.15.129:0/340133565 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+3920 (secure 0 0 0) 0x7f551c064110 con 0x7f55240aee90 2024-09-17T03:51:04.144 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:04.144 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":19,"btime":"2024-09-17T03:50:57:263422+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4604,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/679520734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":679520734},{"type":"v1","addr":"172.21.15.181:6836","nonce":679520734}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4816,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2313251273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2313251273},{"type":"v1","addr":"172.21.15.181:6837","nonce":2313251273}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3208812162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3208812162},{"type":"v1","addr":"172.21.15.129:6834","nonce":3208812162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5142,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3466176125","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3466176125},{"type":"v1","addr":"172.21.15.129:6835","nonce":3466176125}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-09-17T03:51:04.144 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 19 2024-09-17T03:51:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 msgr2=0x7f550007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 0x7f550007b0a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f55240521d0 tx=0x7f5514006b70 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 msgr2=0x7f55240af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f551c005fd0 tx=0x7f551c00f790 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 shutdown_connections 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5500078b90 0x7f550007b0a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55240aee90 0x7f55240af2a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55240c4e20 0x7f55240c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 --2- 172.21.15.129:0/340133565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5508004800 0x7f5524140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.146+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 >> 172.21.15.129:0/340133565 conn(0x7f552406d410 msgr2=0x7f55240b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.147+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 shutdown_connections 2024-09-17T03:51:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.147+0000 7f5523200640 1 -- 172.21.15.129:0/340133565 wait complete. 2024-09-17T03:51:04.170 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:51:04.170 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:51:04.170 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:51:04.170 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:04.171 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:51:04.171 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:51:04.171 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:51:04.171 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:04.171 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:51:04.171 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:51:04.171 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:51:04.172 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:04.172 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:51:04.172 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:51:04.172 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:51:04.172 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:04.191 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:51:04.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:51:04.194 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:51:04.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:51:04.195 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:51:04.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:51:04.196 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:51:04.196 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:51:04.237 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:51:04.238 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:04.240 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:51:04.240 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:04.241 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:51:04.241 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:04.242 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:51:04.242 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:04.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:51:04.433 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:51:04.433 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:51:04.438 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:51:04.438 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:51:04.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 -- 172.21.15.129:0/2448782194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 msgr2=0x7fcd70074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:04.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 --2- 172.21.15.129:0/2448782194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd70074640 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009830 tx=0x7fcd5c02a480 comp rx=0 tx=0).stop 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 -- 172.21.15.129:0/2448782194 shutdown_connections 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 --2- 172.21.15.129:0/2448782194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd70074640 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 --2- 172.21.15.129:0/2448782194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70073170 0x7fcd7015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 -- 172.21.15.129:0/2448782194 >> 172.21.15.129:0/2448782194 conn(0x7fcd700b1cb0 msgr2=0x7fcd700b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 -- 172.21.15.129:0/2448782194 shutdown_connections 2024-09-17T03:51:04.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.479+0000 7fcd76400640 1 -- 172.21.15.129:0/2448782194 wait complete. 2024-09-17T03:51:04.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.481+0000 7fcd76400640 1 Processor -- start 2024-09-17T03:51:04.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.481+0000 7fcd76400640 1 -- start start 2024-09-17T03:51:04.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:04.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 0x7fcd7009b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:04.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd70073170 0x7fcd7009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcd7006f5a0 con 0x7fcd70070030 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcd70074f80 con 0x7fcd70073170 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd76400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcd7006f720 con 0x7fcd70070440 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60062/0 (socket says 172.21.15.129:60062) 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd6f400640 1 -- 172.21.15.129:0/2992718423 learned_addr learned my addr 172.21.15.129:0/2992718423 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd6ea00640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 0x7fcd7009b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.483+0000 7fcd6fe00640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd70073170 0x7fcd7009aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 -- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 msgr2=0x7fcd7009b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 0x7fcd7009b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 -- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd70073170 msgr2=0x7fcd7009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd70073170 0x7fcd7009aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:04.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 -- 172.21.15.129:0/2992718423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd701f8b60 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6ea00640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 0x7fcd7009b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd6f400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e7c0 tx=0x7fcd6000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6000ccc0 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd6000ce80 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd701f6a40 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60005280 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.484+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd701f7000 con 0x7fcd70070030 2024-09-17T03:51:04.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.485+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd700c7ee0 con 0x7fcd70070030 2024-09-17T03:51:04.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.487+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcd60005420 con 0x7fcd70070030 2024-09-17T03:51:04.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.488+0000 7fcd67400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 0x7fcd5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:04.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.488+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4943+0+0 (secure 0 0 0) 0x7fcd6009b4d0 con 0x7fcd70070030 2024-09-17T03:51:04.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.488+0000 7fcd6ea00640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 0x7fcd5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:04.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.489+0000 7fcd6ea00640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 0x7fcd5007b0a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c034000 tx=0x7fcd5c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:04.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.496+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcd60063b70 con 0x7fcd70070030 2024-09-17T03:51:04.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:04.692+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fcd7006fc40 con 0x7fcd70070030 2024-09-17T03:51:05.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.488+0000 7fcd67400640 1 -- 172.21.15.129:0/2992718423 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) ==== 97+0+0 (secure 0 0 0) 0x7fcd60063470 con 0x7fcd70070030 2024-09-17T03:51:05.489 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:51:05.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 msgr2=0x7fcd5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:05.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 0x7fcd5007b0a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c034000 tx=0x7fcd5c004730 comp rx=0 tx=0).stop 2024-09-17T03:51:05.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 msgr2=0x7fcd7014e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:05.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e7c0 tx=0x7fcd6000ec80 comp rx=0 tx=0).stop 2024-09-17T03:51:05.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 shutdown_connections 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd50078b90 0x7fcd5007b0a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd70073170 0x7fcd7009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd70070440 0x7fcd7009b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 --2- 172.21.15.129:0/2992718423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd70070030 0x7fcd7014e6a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 >> 172.21.15.129:0/2992718423 conn(0x7fcd700b1cb0 msgr2=0x7fcd700cc1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 shutdown_connections 2024-09-17T03:51:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.492+0000 7fcd76400640 1 -- 172.21.15.129:0/2992718423 wait complete. 2024-09-17T03:51:05.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:51:05.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 -- 172.21.15.129:0/1603388136 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070030 msgr2=0x7f4468074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 --2- 172.21.15.129:0/1603388136 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070030 0x7f4468074640 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4454009830 tx=0x7f445402a480 comp rx=0 tx=0).stop 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 -- 172.21.15.129:0/1603388136 shutdown_connections 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 --2- 172.21.15.129:0/1603388136 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070030 0x7f4468074640 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 --2- 172.21.15.129:0/1603388136 >> v2:172.21.15.181:3300/0 conn(0x7f4468073170 0x7f446815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 -- 172.21.15.129:0/1603388136 >> 172.21.15.129:0/1603388136 conn(0x7f44680b1cb0 msgr2=0x7f44680b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 -- 172.21.15.129:0/1603388136 shutdown_connections 2024-09-17T03:51:05.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.742+0000 7f446d400640 1 -- 172.21.15.129:0/1603388136 wait complete. 2024-09-17T03:51:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.744+0000 7f446d400640 1 Processor -- start 2024-09-17T03:51:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.745+0000 7f446d400640 1 -- start start 2024-09-17T03:51:05.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:05.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070440 0x7f44680cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:05.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 0x7f44680cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:05.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f446806f5a0 con 0x7f4468070030 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4468074f80 con 0x7f4468073170 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f446d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f446806f720 con 0x7f4468070440 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f4466800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f4466800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60076/0 (socket says 172.21.15.129:60076) 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f4466800640 1 -- 172.21.15.129:0/2755446088 learned_addr learned my addr 172.21.15.129:0/2755446088 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f4465e00640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070440 0x7f44680cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.746+0000 7f4467200640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 0x7f44680cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 -- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070440 msgr2=0x7f44680cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070440 0x7f44680cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 -- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 msgr2=0x7f44680cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 0x7f44680cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:05.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 -- 172.21.15.129:0/2755446088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446809db50 con 0x7f4468070030 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4467200640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 0x7f44680cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f4466800640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9f0 tx=0x7f445800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f445800cd80 con 0x7f4468070030 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4458004020 con 0x7f4468070030 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4458004470 con 0x7f4468070030 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44680d0ac0 con 0x7f4468070030 2024-09-17T03:51:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.747+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44680cc110 con 0x7f4468070030 2024-09-17T03:51:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.748+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44680d08b0 con 0x7f4468070030 2024-09-17T03:51:05.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.750+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f44580035f0 con 0x7f4468070030 2024-09-17T03:51:05.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.751+0000 7f445ea00640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 0x7f444807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:05.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.751+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f445809b2c0 con 0x7f4468070030 2024-09-17T03:51:05.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.752+0000 7f4465e00640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 0x7f444807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:05.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.753+0000 7f4465e00640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 0x7f444807b0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4454034000 tx=0x7f4454004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:05.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:05.756+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4458063a40 con 0x7f4468070030 2024-09-17T03:51:06.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.006+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f446806fc40 con 0x7f4468070030 2024-09-17T03:51:06.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.007+0000 7f445ea00640 1 -- 172.21.15.129:0/2755446088 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) ==== 80+0+5304 (secure 0 0 0) 0x7f445809b0c0 con 0x7f4468070030 2024-09-17T03:51:06.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:06.007 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:51:06.007 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.008 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.009 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:51:06.010 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:51:06.011 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.011 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.011 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:06.011 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:06.011 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:51:06.012 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.013 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.013 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:06.013 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:06.013 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.013 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:51:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:51:06.014 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.015 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.016 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.017 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.018 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.019 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:51:06.020 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.022 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:51:06.022 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:51:06.022 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.022 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.023 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:06.024 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:06.025 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.025 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:06.026 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:51:06.027 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:06.029 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.030 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:51:06.031 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:51:06.031 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.031 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:06.031 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:51:06.031 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:51:06.032 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.033 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:51:06.034 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:51:06.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.009+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 msgr2=0x7f444807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.009+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 0x7f444807b0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4454034000 tx=0x7f4454004730 comp rx=0 tx=0).stop 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.009+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 msgr2=0x7f4468151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.009+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9f0 tx=0x7f445800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 shutdown_connections 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4448078bc0 0x7f444807b0d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4468073170 0x7f44680cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4468070440 0x7f44680cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 --2- 172.21.15.129:0/2755446088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4468070030 0x7f4468151410 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 >> 172.21.15.129:0/2755446088 conn(0x7f44680b1cb0 msgr2=0x7f44680ad9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:06.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 shutdown_connections 2024-09-17T03:51:06.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.010+0000 7f446d400640 1 -- 172.21.15.129:0/2755446088 wait complete. 2024-09-17T03:51:06.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.261+0000 7f0ef0c00640 1 -- 172.21.15.129:0/327054369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 msgr2=0x7f0eec0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.261+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/327054369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec0ccf60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4005fd0 tx=0x7f0ee402aa90 comp rx=0 tx=0).stop 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 -- 172.21.15.129:0/327054369 shutdown_connections 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/327054369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec0ccf60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/327054369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0af3c0 0x7f0eec0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 -- 172.21.15.129:0/327054369 >> 172.21.15.129:0/327054369 conn(0x7f0eec06d7d0 msgr2=0x7f0eec0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:06.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 -- 172.21.15.129:0/327054369 shutdown_connections 2024-09-17T03:51:06.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.262+0000 7f0ef0c00640 1 -- 172.21.15.129:0/327054369 wait complete. 2024-09-17T03:51:06.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.264+0000 7f0ef0c00640 1 Processor -- start 2024-09-17T03:51:06.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.264+0000 7f0ef0c00640 1 -- start start 2024-09-17T03:51:06.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eec0af3c0 0x7f0eec14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0cc1e0 0x7f0eec09b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec153520 con 0x7f0eec0ccb50 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec1533a0 con 0x7f0eec0af3c0 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.265+0000 7f0ef0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0eec1536a0 con 0x7f0eec0cc1e0 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60102/0 (socket says 172.21.15.129:60102) 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 -- 172.21.15.129:0/87336233 learned_addr learned my addr 172.21.15.129:0/87336233 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0ee9e00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eec0af3c0 0x7f0eec14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0ee9400640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0cc1e0 0x7f0eec09b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 -- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0cc1e0 msgr2=0x7f0eec09b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0cc1e0 0x7f0eec09b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 -- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eec0af3c0 msgr2=0x7f0eec14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eec0af3c0 0x7f0eec14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.266+0000 7f0eea800640 1 -- 172.21.15.129:0/87336233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec15e8b0 con 0x7f0eec0ccb50 2024-09-17T03:51:06.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0eea800640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009d90 tx=0x7f0ee4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:06.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4044070 con 0x7f0eec0ccb50 2024-09-17T03:51:06.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ee4004040 con 0x7f0eec0ccb50 2024-09-17T03:51:06.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004430 con 0x7f0eec0ccb50 2024-09-17T03:51:06.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec14d140 con 0x7f0eec0ccb50 2024-09-17T03:51:06.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.267+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec15d810 con 0x7f0eec0ccb50 2024-09-17T03:51:06.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.268+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0eec09ba90 con 0x7f0eec0ccb50 2024-09-17T03:51:06.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.269+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0ee4047070 con 0x7f0eec0ccb50 2024-09-17T03:51:06.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.271+0000 7f0ee2000640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 0x7f0ecc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.271+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f0ee40bb830 con 0x7f0eec0ccb50 2024-09-17T03:51:06.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.271+0000 7f0ee9e00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 0x7f0ecc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.272+0000 7f0ee9e00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 0x7f0ecc07b0a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0edc001720 tx=0x7f0edc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:06.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.275+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0ee4032030 con 0x7f0eec0ccb50 2024-09-17T03:51:06.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.461+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0eec153bc0 con 0x7f0eec0ccb50 2024-09-17T03:51:06.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.461+0000 7f0ee2000640 1 -- 172.21.15.129:0/87336233 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v41) ==== 84+0+65 (secure 0 0 0) 0x7f0ee4083960 con 0x7f0eec0ccb50 2024-09-17T03:51:06.462 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:06.462 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:51:06.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:06.463 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:51:06.463 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:51:06.463 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:06.463 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 msgr2=0x7f0ecc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 0x7f0ecc07b0a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0edc001720 tx=0x7f0edc009040 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 msgr2=0x7f0eec159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009d90 tx=0x7f0ee4003ce0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 shutdown_connections 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ecc078b90 0x7f0ecc07b0a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0eec0ccb50 0x7f0eec159230 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0eec0cc1e0 0x7f0eec09b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 --2- 172.21.15.129:0/87336233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0eec0af3c0 0x7f0eec14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 >> 172.21.15.129:0/87336233 conn(0x7f0eec06d7d0 msgr2=0x7f0eec0ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:06.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 shutdown_connections 2024-09-17T03:51:06.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.464+0000 7f0ef0c00640 1 -- 172.21.15.129:0/87336233 wait complete. 2024-09-17T03:51:06.487 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:51:06.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:51:06.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 -- 172.21.15.129:0/2591502411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 msgr2=0x7f2cb80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 --2- 172.21.15.129:0/2591502411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80cca70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009830 tx=0x7f2cac02a470 comp rx=0 tx=0).stop 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 -- 172.21.15.129:0/2591502411 shutdown_connections 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 --2- 172.21.15.129:0/2591502411 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb806fe50 0x7f2cb80caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 --2- 172.21.15.129:0/2591502411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80cca70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 -- 172.21.15.129:0/2591502411 >> 172.21.15.129:0/2591502411 conn(0x7f2cb80b1c40 msgr2=0x7f2cb80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 -- 172.21.15.129:0/2591502411 shutdown_connections 2024-09-17T03:51:06.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.712+0000 7f2cc0000640 1 -- 172.21.15.129:0/2591502411 wait complete. 2024-09-17T03:51:06.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.714+0000 7f2cc0000640 1 Processor -- start 2024-09-17T03:51:06.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.715+0000 7f2cc0000640 1 -- start start 2024-09-17T03:51:06.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb806fe50 0x7f2cb815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80ccfb0 0x7f2cb809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb80cb5f0 con 0x7f2cb80cc590 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb80cb470 con 0x7f2cb806fe50 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cc0000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb80cb770 con 0x7f2cb80ccfb0 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cbcc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cbcc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60124/0 (socket says 172.21.15.129:60124) 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.716+0000 7f2cbcc00640 1 -- 172.21.15.129:0/4222048321 learned_addr learned my addr 172.21.15.129:0/4222048321 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbe000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80ccfb0 0x7f2cb809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbd600640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb806fe50 0x7f2cb815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 -- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80ccfb0 msgr2=0x7f2cb809c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80ccfb0 0x7f2cb809c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 -- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb806fe50 msgr2=0x7f2cb815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb806fe50 0x7f2cb815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 -- 172.21.15.129:0/4222048321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb815da10 con 0x7f2cb80cc590 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cbcc00640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c760 tx=0x7f2ca400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4016020 con 0x7f2cb80cc590 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ca400a950 con 0x7f2cb80cc590 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb815dcf0 con 0x7f2cb80cc590 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.717+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca400ada0 con 0x7f2cb80cc590 2024-09-17T03:51:06.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.718+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb809cce0 con 0x7f2cb80cc590 2024-09-17T03:51:06.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.719+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb80c7110 con 0x7f2cb80cc590 2024-09-17T03:51:06.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.720+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2ca4019020 con 0x7f2cb80cc590 2024-09-17T03:51:06.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.721+0000 7f2cb5600640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 0x7f2c9807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:06.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.721+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4857+0+0 (secure 0 0 0) 0x7f2ca409a8d0 con 0x7f2cb80cc590 2024-09-17T03:51:06.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.721+0000 7f2cbd600640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 0x7f2c9807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:06.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.722+0000 7f2cbd600640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 0x7f2c9807b0d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2cac034000 tx=0x7f2cac004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:06.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.725+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2cb80c7110 con 0x7f2cb80cc590 2024-09-17T03:51:06.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:06.910+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2cb80cca70 con 0x7f2cb80cc590 2024-09-17T03:51:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.497+0000 7f2cb5600640 1 -- 172.21.15.129:0/4222048321 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v42) ==== 138+0+0 (secure 0 0 0) 0x7f2ca4020020 con 0x7f2cb80cc590 2024-09-17T03:51:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 msgr2=0x7f2c9807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 0x7f2c9807b0d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2cac034000 tx=0x7f2cac004730 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 msgr2=0x7f2cb80c3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400c760 tx=0x7f2ca400cc20 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 shutdown_connections 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c98078bc0 0x7f2c9807b0d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80ccfb0 0x7f2cb809c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cc590 0x7f2cb80c3cb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 --2- 172.21.15.129:0/4222048321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb806fe50 0x7f2cb815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.501+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 >> 172.21.15.129:0/4222048321 conn(0x7f2cb80b1c40 msgr2=0x7f2cb80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:07.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.502+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 shutdown_connections 2024-09-17T03:51:07.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.502+0000 7f2cc0000640 1 -- 172.21.15.129:0/4222048321 wait complete. 2024-09-17T03:51:07.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:51:07.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 -- 172.21.15.129:0/400905549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 msgr2=0x7fbacc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:07.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 --2- 172.21.15.129:0/400905549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbacc004c10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009b50 tx=0x7fbacc005830 comp rx=0 tx=0).stop 2024-09-17T03:51:07.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 -- 172.21.15.129:0/400905549 shutdown_connections 2024-09-17T03:51:07.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 --2- 172.21.15.129:0/400905549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbacc004c10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 -- 172.21.15.129:0/400905549 >> 172.21.15.129:0/400905549 conn(0x7fbae80b1cb0 msgr2=0x7fbae80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:07.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 -- 172.21.15.129:0/400905549 shutdown_connections 2024-09-17T03:51:07.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.758+0000 7fbaef800640 1 -- 172.21.15.129:0/400905549 wait complete. 2024-09-17T03:51:07.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.761+0000 7fbaef800640 1 Processor -- start 2024-09-17T03:51:07.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.761+0000 7fbaef800640 1 -- start start 2024-09-17T03:51:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae80b0af0 0x7fbae809b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbae815c590 0x7fbae809b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80701b0 con 0x7fbacc004800 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbae80cf3d0 con 0x7fbae80b0af0 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaef800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbae8070330 con 0x7fbae815c590 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaece00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaece00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60136/0 (socket says 172.21.15.129:60136) 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.762+0000 7fbaece00640 1 -- 172.21.15.129:0/2122336075 learned_addr learned my addr 172.21.15.129:0/2122336075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbadfe00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae80b0af0 0x7fbae809b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaed800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbae815c590 0x7fbae809b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 -- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbae815c590 msgr2=0x7fbae809b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbae815c590 0x7fbae809b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 -- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae80b0af0 msgr2=0x7fbae809b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae80b0af0 0x7fbae809b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:07.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 -- 172.21.15.129:0/2122336075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae81f5080 con 0x7fbacc004800 2024-09-17T03:51:07.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaece00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006680 tx=0x7fbad80066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:07.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8015070 con 0x7fbacc004800 2024-09-17T03:51:07.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbad8011900 con 0x7fbacc004800 2024-09-17T03:51:07.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8011d50 con 0x7fbacc004800 2024-09-17T03:51:07.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae81f5360 con 0x7fbacc004800 2024-09-17T03:51:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.763+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae81f5920 con 0x7fbacc004800 2024-09-17T03:51:07.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.765+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae80cc740 con 0x7fbacc004800 2024-09-17T03:51:07.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.766+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbad8018070 con 0x7fbacc004800 2024-09-17T03:51:07.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.767+0000 7fbae5600640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 0x7fbac407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:07.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.768+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5227+0+0 (secure 0 0 0) 0x7fbad809b610 con 0x7fbacc004800 2024-09-17T03:51:07.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.768+0000 7fbadfe00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 0x7fbac407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:07.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.769+0000 7fbadfe00640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 0x7fbac407b0a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbad0035000 tx=0x7fbad00046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:07.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.773+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbad8063b90 con 0x7fbacc004800 2024-09-17T03:51:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:07.960+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbae80b14a0 con 0x7fbacc004800 2024-09-17T03:51:08.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.507+0000 7fbae5600640 1 -- 172.21.15.129:0/2122336075 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v43) ==== 170+0+0 (secure 0 0 0) 0x7fbad8063490 con 0x7fbacc004800 2024-09-17T03:51:08.507 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:51:08.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 msgr2=0x7fbac407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 0x7fbac407b0a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbad0035000 tx=0x7fbad00046c0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 msgr2=0x7fbae814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fbad8006680 tx=0x7fbad80066b0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 shutdown_connections 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbac4078b90 0x7fbac407b0a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbae815c590 0x7fbae809b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbae80b0af0 0x7fbae809b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 --2- 172.21.15.129:0/2122336075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbacc004800 0x7fbae814f620 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 >> 172.21.15.129:0/2122336075 conn(0x7fbae80b1cb0 msgr2=0x7fbae815cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:08.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 shutdown_connections 2024-09-17T03:51:08.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.510+0000 7fbaef800640 1 -- 172.21.15.129:0/2122336075 wait complete. 2024-09-17T03:51:08.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 -- 172.21.15.129:0/419455239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3900004800 msgr2=0x7f3900004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 --2- 172.21.15.129:0/419455239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3900004800 0x7f3900004c10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f3904009ed0 tx=0x7f3900005830 comp rx=0 tx=0).stop 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 -- 172.21.15.129:0/419455239 shutdown_connections 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 --2- 172.21.15.129:0/419455239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3900004800 0x7f3900004c10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 -- 172.21.15.129:0/419455239 >> 172.21.15.129:0/419455239 conn(0x7f391c06ee50 msgr2=0x7f391c06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 -- 172.21.15.129:0/419455239 shutdown_connections 2024-09-17T03:51:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.775+0000 7f391b200640 1 -- 172.21.15.129:0/419455239 wait complete. 2024-09-17T03:51:08.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.778+0000 7f391b200640 1 Processor -- start 2024-09-17T03:51:08.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.778+0000 7f391b200640 1 -- start start 2024-09-17T03:51:08.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3900004800 0x7f391c1f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 0x7f391c14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0d0980 con 0x7f391c1f6960 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f391c0d0800 con 0x7f3900004800 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f391c0d0b00 con 0x7f391c15e0f0 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60156/0 (socket says 172.21.15.129:60156) 2024-09-17T03:51:08.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.779+0000 7f391a800640 1 -- 172.21.15.129:0/452135016 learned_addr learned my addr 172.21.15.129:0/452135016 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3919e00640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3900004800 0x7f391c1f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3911400640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 0x7f391c14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 -- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 msgr2=0x7f391c14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 0x7f391c14c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 -- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3900004800 msgr2=0x7f391c1f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3900004800 0x7f391c1f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 -- 172.21.15.129:0/452135016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c1f72b0 con 0x7f391c1f6960 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3911400640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 0x7f391c14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391a800640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f391c0752f0 tx=0x7f3914001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:08.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914011630 con 0x7f391c1f6960 2024-09-17T03:51:08.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3914018070 con 0x7f391c1f6960 2024-09-17T03:51:08.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c15e8d0 con 0x7f391c1f6960 2024-09-17T03:51:08.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.780+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914011a70 con 0x7f391c1f6960 2024-09-17T03:51:08.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.781+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c15ee90 con 0x7f391c1f6960 2024-09-17T03:51:08.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.782+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f391c15e640 con 0x7f391c1f6960 2024-09-17T03:51:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.783+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3914011c10 con 0x7f391c1f6960 2024-09-17T03:51:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.784+0000 7f3912a00640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 0x7f38f807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.784+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5609+0+0 (secure 0 0 0) 0x7f391409b260 con 0x7f391c1f6960 2024-09-17T03:51:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.784+0000 7f3919e00640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 0x7f38f807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:08.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.785+0000 7f3919e00640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 0x7f38f807b0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f391c0521d0 tx=0x7f390c0067d0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:08.789+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39140637b0 con 0x7f391c1f6960 2024-09-17T03:51:09.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.007+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f391c072d80 con 0x7f391c1f6960 2024-09-17T03:51:09.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.509+0000 7f3912a00640 1 -- 172.21.15.129:0/452135016 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) ==== 172+0+0 (secure 0 0 0) 0x7f39140630b0 con 0x7f391c1f6960 2024-09-17T03:51:09.510 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 6 and data pool 7 2024-09-17T03:51:09.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 msgr2=0x7f38f807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 0x7f38f807b0d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f391c0521d0 tx=0x7f390c0067d0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 msgr2=0x7f391c1f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f391c0752f0 tx=0x7f3914001630 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 shutdown_connections 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f8078bc0 0x7f38f807b0d0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f391c0521d0 tx=0x7f390c0067d0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391c1f6960 0x7f391c1f6d70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.515+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f391c15e0f0 0x7f391c14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.516+0000 7f391b200640 1 --2- 172.21.15.129:0/452135016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3900004800 0x7f391c1f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.516+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 >> 172.21.15.129:0/452135016 conn(0x7f391c06ee50 msgr2=0x7f391c068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:09.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.517+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 shutdown_connections 2024-09-17T03:51:09.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.519+0000 7f391b200640 1 -- 172.21.15.129:0/452135016 wait complete. 2024-09-17T03:51:09.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 -- 172.21.15.129:0/2560883798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200d2110 msgr2=0x7fa7200d2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 --2- 172.21.15.129:0/2560883798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200d2110 0x7fa7200d2520 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa7200d33a0 tx=0x7fa71002a530 comp rx=0 tx=0).stop 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 -- 172.21.15.129:0/2560883798 shutdown_connections 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 --2- 172.21.15.129:0/2560883798 >> v2:172.21.15.181:3300/0 conn(0x7fa7200b7460 0x7fa7200b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 --2- 172.21.15.129:0/2560883798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200d2110 0x7fa7200d2520 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 -- 172.21.15.129:0/2560883798 >> 172.21.15.129:0/2560883798 conn(0x7fa72009d6a0 msgr2=0x7fa7200d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:09.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 -- 172.21.15.129:0/2560883798 shutdown_connections 2024-09-17T03:51:09.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.788+0000 7fa727200640 1 -- 172.21.15.129:0/2560883798 wait complete. 2024-09-17T03:51:09.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.790+0000 7fa727200640 1 Processor -- start 2024-09-17T03:51:09.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.790+0000 7fa727200640 1 -- start start 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.791+0000 7fa727200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 0x7fa7200ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.791+0000 7fa727200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa727200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 0x7fa720153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa727200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa7200b8140 con 0x7fa7200b7460 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa727200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa7200b7fc0 con 0x7fa7200b5470 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa727200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa7200b82c0 con 0x7fa7200d2110 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60172/0 (socket says 172.21.15.129:60172) 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 -- 172.21.15.129:0/4252386660 learned_addr learned my addr 172.21.15.129:0/4252386660 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725e00640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 0x7fa7200ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:09.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa726800640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 0x7fa720153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 -- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 msgr2=0x7fa720153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 0x7fa720153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 -- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 msgr2=0x7fa7200ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 0x7fa7200ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725400640 1 -- 172.21.15.129:0/4252386660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7201f5d50 con 0x7fa7200b7460 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa726800640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 0x7fa720153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.792+0000 7fa725e00640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 0x7fa7200ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa725400640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa710009910 tx=0x7fa710005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71003f070 con 0x7fa7200b7460 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa710004340 con 0x7fa7200b7460 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710031bb0 con 0x7fa7200b7460 2024-09-17T03:51:09.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7201f6030 con 0x7fa7200b7460 2024-09-17T03:51:09.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.793+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7200cc5d0 con 0x7fa7200b7460 2024-09-17T03:51:09.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.794+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7201f5ac0 con 0x7fa7200b7460 2024-09-17T03:51:09.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.795+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa710042070 con 0x7fa7200b7460 2024-09-17T03:51:09.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.796+0000 7fa71a000640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 0x7fa70407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:09.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.796+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fa7100bbe40 con 0x7fa7200b7460 2024-09-17T03:51:09.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.797+0000 7fa725e00640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 0x7fa70407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:09.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.797+0000 7fa725e00640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 0x7fa70407b0a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa7140016a0 tx=0x7fa714009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:09.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.801+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa7100841e0 con 0x7fa7200b7460 2024-09-17T03:51:09.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.983+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7200b7870 con 0x7fa7200b7460 2024-09-17T03:51:09.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.984+0000 7fa71a000640 1 -- 172.21.15.129:0/4252386660 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16734 (secure 0 0 0) 0x7fa710031770 con 0x7fa7200b7460 2024-09-17T03:51:09.985 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:09.985 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":44,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:09.008786+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:06.922986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:07.972487+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:09.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 msgr2=0x7fa70407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 0x7fa70407b0a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa7140016a0 tx=0x7fa714009040 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 msgr2=0x7fa7201536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa710009910 tx=0x7fa710005270 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 shutdown_connections 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa704078b90 0x7fa70407b0a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7200d2110 0x7fa720153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7200b7460 0x7fa7201536e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 --2- 172.21.15.129:0/4252386660 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7200b5470 0x7fa7200ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 >> 172.21.15.129:0/4252386660 conn(0x7fa72009d6a0 msgr2=0x7fa7200b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 shutdown_connections 2024-09-17T03:51:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:09.987+0000 7fa727200640 1 -- 172.21.15.129:0/4252386660 wait complete. 2024-09-17T03:51:10.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 -- 172.21.15.129:0/3917290903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0ccb50 msgr2=0x7fe87c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 --2- 172.21.15.129:0/3917290903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0ccb50 0x7fe87c0ccf60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe874005fd0 tx=0x7fe87402a8b0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 -- 172.21.15.129:0/3917290903 shutdown_connections 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 --2- 172.21.15.129:0/3917290903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0ccb50 0x7fe87c0ccf60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 --2- 172.21.15.129:0/3917290903 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0af3c0 0x7fe87c0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.231+0000 7fe884000640 1 -- 172.21.15.129:0/3917290903 >> 172.21.15.129:0/3917290903 conn(0x7fe87c06d7d0 msgr2=0x7fe87c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.232+0000 7fe884000640 1 -- 172.21.15.129:0/3917290903 shutdown_connections 2024-09-17T03:51:10.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.232+0000 7fe884000640 1 -- 172.21.15.129:0/3917290903 wait complete. 2024-09-17T03:51:10.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.234+0000 7fe884000640 1 Processor -- start 2024-09-17T03:51:10.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.234+0000 7fe884000640 1 -- start start 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 0x7fe87c15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0ccb50 0x7fe87c15fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c153520 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c1533a0 con 0x7fe87c0af3c0 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.235+0000 7fe884000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c1536a0 con 0x7fe87c0ccb50 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60194/0 (socket says 172.21.15.129:60194) 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 -- 172.21.15.129:0/2919194181 learned_addr learned my addr 172.21.15.129:0/2919194181 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe882000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0ccb50 0x7fe87c15fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe881600640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 0x7fe87c15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 -- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0ccb50 msgr2=0x7fe87c15fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0ccb50 0x7fe87c15fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 -- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 msgr2=0x7fe87c15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 0x7fe87c15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 -- 172.21.15.129:0/2919194181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c074df0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe881600640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 0x7fe87c15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.236+0000 7fe880c00640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e790 tx=0x7fe86800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.237+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86800eea0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.237+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe86800cdb0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.237+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868010690 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.237+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c074830 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.237+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c15f360 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.238+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c153bc0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.239+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe868010830 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.240+0000 7fe879600640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 0x7fe85c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.240+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fe86809b6e0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.241+0000 7fe881600640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 0x7fe85c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.241+0000 7fe881600640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 0x7fe85c07b0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe8700013f0 tx=0x7fe870009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:10.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.244+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe8680634d0 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.429+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe87c06f950 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.430+0000 7fe879600640 1 -- 172.21.15.129:0/2919194181 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16734 (secure 0 0 0) 0x7fe868067f00 con 0x7fe87c0cc1e0 2024-09-17T03:51:10.431 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:10.431 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":44,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:09.008786+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:06.922986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:07.972487+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 msgr2=0x7fe85c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 0x7fe85c07b0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe8700013f0 tx=0x7fe870009040 comp rx=0 tx=0).stop 2024-09-17T03:51:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 msgr2=0x7fe87c1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e790 tx=0x7fe86800ec50 comp rx=0 tx=0).stop 2024-09-17T03:51:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 shutdown_connections 2024-09-17T03:51:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe85c078b90 0x7fe85c07b0a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe87c0ccb50 0x7fe87c15fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe87c0cc1e0 0x7fe87c1527a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 --2- 172.21.15.129:0/2919194181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe87c0af3c0 0x7fe87c15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 >> 172.21.15.129:0/2919194181 conn(0x7fe87c06d7d0 msgr2=0x7fe87c06ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:10.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 shutdown_connections 2024-09-17T03:51:10.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.433+0000 7fe884000640 1 -- 172.21.15.129:0/2919194181 wait complete. 2024-09-17T03:51:10.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.687+0000 7f35cac00640 1 -- 172.21.15.129:0/125157020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004d40 msgr2=0x7f35a8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.687+0000 7f35cac00640 1 --2- 172.21.15.129:0/125157020 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35a8004d40 0x7f35a8005150 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f35bc005fd0 tx=0x7f35a8005830 comp rx=0 tx=0).stop 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.687+0000 7f35c8c00640 1 -- 172.21.15.129:0/125157020 reap_dead start 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.687+0000 7f35cac00640 1 -- 172.21.15.129:0/125157020 shutdown_connections 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.687+0000 7f35cac00640 1 -- 172.21.15.129:0/125157020 >> 172.21.15.129:0/125157020 conn(0x7f35c40b1cb0 msgr2=0x7f35c40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.688+0000 7f35cac00640 1 -- 172.21.15.129:0/125157020 shutdown_connections 2024-09-17T03:51:10.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.688+0000 7f35cac00640 1 -- 172.21.15.129:0/125157020 wait complete. 2024-09-17T03:51:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.690+0000 7f35cac00640 1 Processor -- start 2024-09-17T03:51:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.690+0000 7f35cac00640 1 -- start start 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35a8004d40 0x7f35c40ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35c414bab0 0x7f35c41f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35c40b3470 con 0x7f35c414b160 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f35c40b32f0 con 0x7f35a8004d40 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.691+0000 7f35cac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f35c40b35f0 con 0x7f35c414bab0 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60212/0 (socket says 172.21.15.129:60212) 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 -- 172.21.15.129:0/1286774412 learned_addr learned my addr 172.21.15.129:0/1286774412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3e00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35a8004d40 0x7f35c40ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c8c00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35c414bab0 0x7f35c41f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 -- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35c414bab0 msgr2=0x7f35c41f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35c414bab0 0x7f35c41f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 -- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35a8004d40 msgr2=0x7f35c40ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35a8004d40 0x7f35c40ce5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 -- 172.21.15.129:0/1286774412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c40cffb0 con 0x7f35c414b160 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.692+0000 7f35c3400640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f35b0011780 tx=0x7f35b0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.693+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b000fd50 con 0x7f35c414b160 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.693+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f35b0005020 con 0x7f35c414b160 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.693+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c40cef10 con 0x7f35c414b160 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.693+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0005470 con 0x7f35c414b160 2024-09-17T03:51:10.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.693+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c40cf4d0 con 0x7f35c414b160 2024-09-17T03:51:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.694+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c40cec80 con 0x7f35c414b160 2024-09-17T03:51:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.695+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f35b00047a0 con 0x7f35c414b160 2024-09-17T03:51:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.696+0000 7f35bbe00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 0x7f35a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.696+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f35b009ba10 con 0x7f35c414b160 2024-09-17T03:51:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.696+0000 7f35c3e00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 0x7f35a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.697+0000 7f35c3e00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 0x7f35a407b0a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f35b4000fb0 tx=0x7f35b4001550 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.701+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f35b0063e40 con 0x7f35c414b160 2024-09-17T03:51:10.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:10.922+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f35c4074b60 con 0x7f35c414b160 2024-09-17T03:51:11.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.535+0000 7f35bbe00640 1 -- 172.21.15.129:0/1286774412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f35b0063740 con 0x7f35c414b160 2024-09-17T03:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 msgr2=0x7f35a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 0x7f35a407b0a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f35b4000fb0 tx=0x7f35b4001550 comp rx=0 tx=0).stop 2024-09-17T03:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 msgr2=0x7f35c414b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:11.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f35b0011780 tx=0x7f35b0011c40 comp rx=0 tx=0).stop 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 shutdown_connections 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35a4078b90 0x7f35a407b0a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35c414bab0 0x7f35c41f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35c414b160 0x7f35c414b570 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 --2- 172.21.15.129:0/1286774412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35a8004d40 0x7f35c40ce5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 >> 172.21.15.129:0/1286774412 conn(0x7f35c40b1cb0 msgr2=0x7f35c415eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 shutdown_connections 2024-09-17T03:51:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.538+0000 7f35cac00640 1 -- 172.21.15.129:0/1286774412 wait complete. 2024-09-17T03:51:11.567 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:51:11.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:51:11.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 -- 172.21.15.129:0/2700918559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa40cf540 msgr2=0x7f9aa406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 --2- 172.21.15.129:0/2700918559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa40cf540 0x7f9aa406ee70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c005fd0 tx=0x7f9a8c02a540 comp rx=0 tx=0).stop 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 -- 172.21.15.129:0/2700918559 shutdown_connections 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 --2- 172.21.15.129:0/2700918559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa40cf540 0x7f9aa406ee70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 --2- 172.21.15.129:0/2700918559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa415ca50 0x7f9aa40cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 -- 172.21.15.129:0/2700918559 >> 172.21.15.129:0/2700918559 conn(0x7f9aa406d410 msgr2=0x7f9aa40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 -- 172.21.15.129:0/2700918559 shutdown_connections 2024-09-17T03:51:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.790+0000 7f9aa3200640 1 -- 172.21.15.129:0/2700918559 wait complete. 2024-09-17T03:51:11.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.792+0000 7f9aa3200640 1 Processor -- start 2024-09-17T03:51:11.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.793+0000 7f9aa3200640 1 -- start start 2024-09-17T03:51:11.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa40cf540 0x7f9aa4141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:11.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9aa40cf950 0x7f9aa413eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:11.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa415de10 con 0x7f9aa415ca50 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa415df90 con 0x7f9aa40cf950 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa415e110 con 0x7f9aa40cf540 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60236/0 (socket says 172.21.15.129:60236) 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 -- 172.21.15.129:0/3664686450 learned_addr learned my addr 172.21.15.129:0/3664686450 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa1e00640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa40cf540 0x7f9aa4141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa1400640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9aa40cf950 0x7f9aa413eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 -- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa40cf540 msgr2=0x7f9aa4141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa40cf540 0x7f9aa4141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 -- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9aa40cf950 msgr2=0x7f9aa413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9aa40cf950 0x7f9aa413eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:11.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 -- 172.21.15.129:0/3664686450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa41f3920 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa2800640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c730 tx=0x7f9a9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c003b90 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a9c003d50 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c005330 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.795+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa41f3b90 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.796+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa41f3e30 con 0x7f9aa415ca50 2024-09-17T03:51:11.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.797+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aa4156540 con 0x7f9aa415ca50 2024-09-17T03:51:11.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.798+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9a9c0056e0 con 0x7f9aa415ca50 2024-09-17T03:51:11.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.799+0000 7f9a9a000640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 0x7f9a8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:11.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.799+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f9a9c09ac80 con 0x7f9aa415ca50 2024-09-17T03:51:11.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.799+0000 7f9aa1e00640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 0x7f9a8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:11.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.800+0000 7f9aa1e00640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 0x7f9a8007b0a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a94004630 tx=0x7f9a9400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:11.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:11.803+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9a9c0629d0 con 0x7f9aa415ca50 2024-09-17T03:51:12.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.018+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9aa406ee70 con 0x7f9aa415ca50 2024-09-17T03:51:12.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.541+0000 7f9a9a000640 1 -- 172.21.15.129:0/3664686450 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) ==== 122+0+0 (secure 0 0 0) 0x7f9a9c067400 con 0x7f9aa415ca50 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 msgr2=0x7f9a8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 0x7f9a8007b0a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a94004630 tx=0x7f9a9400b2e0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 msgr2=0x7f9aa4142b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00c730 tx=0x7f9a9c00cbf0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 shutdown_connections 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a80078b90 0x7f9a8007b0a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aa415ca50 0x7f9aa4142b90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9aa40cf950 0x7f9aa413eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 --2- 172.21.15.129:0/3664686450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aa40cf540 0x7f9aa4141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.544+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 >> 172.21.15.129:0/3664686450 conn(0x7f9aa406d410 msgr2=0x7f9aa4140a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.545+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 shutdown_connections 2024-09-17T03:51:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.545+0000 7f9aa3200640 1 -- 172.21.15.129:0/3664686450 wait complete. 2024-09-17T03:51:12.568 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:51:12.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:12.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 -- 172.21.15.129:0/2345453548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 msgr2=0x7f69a80cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:12.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 --2- 172.21.15.129:0/2345453548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80cc4c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6998009830 tx=0x7f699802a510 comp rx=0 tx=0).stop 2024-09-17T03:51:12.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 -- 172.21.15.129:0/2345453548 shutdown_connections 2024-09-17T03:51:12.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 --2- 172.21.15.129:0/2345453548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80cc4c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 -- 172.21.15.129:0/2345453548 >> 172.21.15.129:0/2345453548 conn(0x7f69a80b1cb0 msgr2=0x7f69a80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:12.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 -- 172.21.15.129:0/2345453548 shutdown_connections 2024-09-17T03:51:12.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.790+0000 7f69ad800640 1 -- 172.21.15.129:0/2345453548 wait complete. 2024-09-17T03:51:12.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.793+0000 7f69ad800640 1 Processor -- start 2024-09-17T03:51:12.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.793+0000 7f69ad800640 1 -- start start 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a80ccb30 0x7f69a809d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 0x7f69a8160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69a80cf340 con 0x7f69a80cf6f0 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69a80cf1c0 con 0x7f69a80ccb30 2024-09-17T03:51:12.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.794+0000 7f69ad800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f69a80701b0 con 0x7f69a80ce020 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60252/0 (socket says 172.21.15.129:60252) 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 -- 172.21.15.129:0/1209479680 learned_addr learned my addr 172.21.15.129:0/1209479680 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a6a00640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a80ccb30 0x7f69a809d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a6000640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 0x7f69a8160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 -- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 msgr2=0x7f69a8160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 0x7f69a8160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 -- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a80ccb30 msgr2=0x7f69a809d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a80ccb30 0x7f69a809d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 -- 172.21.15.129:0/1209479680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a809d8f0 con 0x7f69a80cf6f0 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a6000640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 0x7f69a8160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.795+0000 7f69a7400640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f69a000b990 tx=0x7f69a000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:12.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.796+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a0019070 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.796+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69a000cc90 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.796+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a00146a0 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.796+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a809bf40 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.796+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a809c500 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.797+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a809bcb0 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.798+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f69a001c070 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.799+0000 7f699ea00640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 0x7f698807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.799+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f69a009aa90 con 0x7f69a80cf6f0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.800+0000 7f69a6a00640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 0x7f698807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:12.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.800+0000 7f69a6a00640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 0x7f698807b0a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6998034000 tx=0x7f6998004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:12.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:12.803+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f69a809d8f0 con 0x7f69a80cf6f0 2024-09-17T03:51:13.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.021+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69a8074600 con 0x7f69a80cf6f0 2024-09-17T03:51:13.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.021+0000 7f699ea00640 1 -- 172.21.15.129:0/1209479680 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7f69a8074600 con 0x7f69a80cf6f0 2024-09-17T03:51:13.020 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:13.021 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:13.021 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:13.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 msgr2=0x7f698807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 0x7f698807b0a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6998034000 tx=0x7f6998004730 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 msgr2=0x7f69a80b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f69a000b990 tx=0x7f69a000be50 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 shutdown_connections 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6988078b90 0x7f698807b0a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69a80cf6f0 0x7f69a80b42a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69a80ce020 0x7f69a8160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 --2- 172.21.15.129:0/1209479680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69a80ccb30 0x7f69a809d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 >> 172.21.15.129:0/1209479680 conn(0x7f69a80b1cb0 msgr2=0x7f69a80d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 shutdown_connections 2024-09-17T03:51:13.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.024+0000 7f69ad800640 1 -- 172.21.15.129:0/1209479680 wait complete. 2024-09-17T03:51:13.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:13.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 -- 172.21.15.129:0/3300629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 msgr2=0x7f806015f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 --2- 172.21.15.129:0/3300629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f806015f490 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8058005fd0 tx=0x7f805802aa90 comp rx=0 tx=0).stop 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 -- 172.21.15.129:0/3300629766 shutdown_connections 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 --2- 172.21.15.129:0/3300629766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f806015f490 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 --2- 172.21.15.129:0/3300629766 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 0x7f806015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 -- 172.21.15.129:0/3300629766 >> 172.21.15.129:0/3300629766 conn(0x7f806006d410 msgr2=0x7f80600b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:13.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 -- 172.21.15.129:0/3300629766 shutdown_connections 2024-09-17T03:51:13.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.267+0000 7f8064a00640 1 -- 172.21.15.129:0/3300629766 wait complete. 2024-09-17T03:51:13.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.270+0000 7f8064a00640 1 Processor -- start 2024-09-17T03:51:13.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.270+0000 7f8064a00640 1 -- start start 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 0x7f80601f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f80601464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f806006f830 con 0x7f806014a240 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f806006f6b0 con 0x7f8060073040 2024-09-17T03:51:13.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.271+0000 7f8064a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f806006f9b0 con 0x7f8060072c30 2024-09-17T03:51:13.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60256/0 (socket says 172.21.15.129:60256) 2024-09-17T03:51:13.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 -- 172.21.15.129:0/2492554173 learned_addr learned my addr 172.21.15.129:0/2492554173 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:13.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805f400640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 0x7f80601f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f8056a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f80601464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 -- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 msgr2=0x7f80601f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 0x7f80601f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 -- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 msgr2=0x7f80601464d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f80601464d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.272+0000 7f805fe00640 1 -- 172.21.15.129:0/2492554173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8060156270 con 0x7f806014a240 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f805fe00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8058009bf0 tx=0x7f8058003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8058044070 con 0x7f806014a240 2024-09-17T03:51:13.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8058003e00 con 0x7f806014a240 2024-09-17T03:51:13.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80580044b0 con 0x7f806014a240 2024-09-17T03:51:13.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8060156550 con 0x7f806014a240 2024-09-17T03:51:13.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.273+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8060156b10 con 0x7f806014a240 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.274+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806014a790 con 0x7f806014a240 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.275+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8058047070 con 0x7f806014a240 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.276+0000 7f8057e00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 0x7f803c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.276+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f80580bbb60 con 0x7f806014a240 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.277+0000 7f805f400640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 0x7f803c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.277+0000 7f805f400640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 0x7f803c07b0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8050001720 tx=0x7f8050009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:13.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.280+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8058032030 con 0x7f806014a240 2024-09-17T03:51:13.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.465+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f806015fbd0 con 0x7f806014a240 2024-09-17T03:51:13.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.466+0000 7f8057e00640 1 -- 172.21.15.129:0/2492554173 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16721 (secure 0 0 0) 0x7f80580098b0 con 0x7f806014a240 2024-09-17T03:51:13.466 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:13.466 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:10.521773+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:06.922986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:07.972487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 msgr2=0x7f803c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 0x7f803c07b0d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8050001720 tx=0x7f8050009040 comp rx=0 tx=0).stop 2024-09-17T03:51:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 msgr2=0x7f80600cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8058009bf0 tx=0x7f8058003c10 comp rx=0 tx=0).stop 2024-09-17T03:51:13.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 shutdown_connections 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f803c078bc0 0x7f803c07b0d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806014a240 0x7f80600cf7e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8060073040 0x7f80601464d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 --2- 172.21.15.129:0/2492554173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8060072c30 0x7f80601f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 >> 172.21.15.129:0/2492554173 conn(0x7f806006d410 msgr2=0x7f806015e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 shutdown_connections 2024-09-17T03:51:13.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.469+0000 7f8064a00640 1 -- 172.21.15.129:0/2492554173 wait complete. 2024-09-17T03:51:13.491 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:51:13.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 -- 172.21.15.129:0/4188571584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d100ca800 msgr2=0x7f3d100cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 --2- 172.21.15.129:0/4188571584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d100ca800 0x7f3d100cba00 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d10070dc0 tx=0x7f3d0002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 -- 172.21.15.129:0/4188571584 shutdown_connections 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 --2- 172.21.15.129:0/4188571584 >> v2:172.21.15.181:3300/0 conn(0x7f3d100706b0 0x7f3d100735b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 --2- 172.21.15.129:0/4188571584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d100ca800 0x7f3d100cba00 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 -- 172.21.15.129:0/4188571584 >> 172.21.15.129:0/4188571584 conn(0x7f3d100b1cb0 msgr2=0x7f3d100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 -- 172.21.15.129:0/4188571584 shutdown_connections 2024-09-17T03:51:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.907+0000 7f3d15600640 1 -- 172.21.15.129:0/4188571584 wait complete. 2024-09-17T03:51:13.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.910+0000 7f3d15600640 1 Processor -- start 2024-09-17T03:51:13.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.910+0000 7f3d15600640 1 -- start start 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d100ca800 0x7f3d101597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d100cad30 0x7f3d1015d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d10073ec0 con 0x7f3d1009aa30 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d10073d40 con 0x7f3d100cad30 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d15600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d10074040 con 0x7f3d100ca800 2024-09-17T03:51:13.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.911+0000 7f3d0f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39374/0 (socket says 172.21.15.129:39374) 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 -- 172.21.15.129:0/2144186662 learned_addr learned my addr 172.21.15.129:0/2144186662 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0ea00640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d100ca800 0x7f3d101597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d07e00640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d100cad30 0x7f3d1015d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 -- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d100ca800 msgr2=0x7f3d101597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d100ca800 0x7f3d101597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 -- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d100cad30 msgr2=0x7f3d1015d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d100cad30 0x7f3d1015d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 -- 172.21.15.129:0/2144186662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d101f5730 con 0x7f3d1009aa30 2024-09-17T03:51:13.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.912+0000 7f3d0f400640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800e960 tx=0x7f3d0800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.913+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08004d20 con 0x7f3d1009aa30 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.913+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d0800cb20 con 0x7f3d1009aa30 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.913+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08005020 con 0x7f3d1009aa30 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.913+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d101f5a10 con 0x7f3d1009aa30 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.913+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d101f5fd0 con 0x7f3d1009aa30 2024-09-17T03:51:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.914+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d100b25f0 con 0x7f3d1009aa30 2024-09-17T03:51:13.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.915+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d080051c0 con 0x7f3d1009aa30 2024-09-17T03:51:13.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.916+0000 7f3d06a00640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 0x7f3cec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:13.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.916+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3d0809b610 con 0x7f3d1009aa30 2024-09-17T03:51:13.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.917+0000 7f3d0ea00640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 0x7f3cec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:13.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.918+0000 7f3d0ea00640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 0x7f3cec07b0d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3d00033000 tx=0x7f3d00004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:13.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:13.921+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d100b25f0 con 0x7f3d1009aa30 2024-09-17T03:51:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.102+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d101e7cf0 con 0x7f3cec078bc0 2024-09-17T03:51:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.107+0000 7f3d06a00640 1 -- 172.21.15.129:0/2144186662 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+244682 (secure 0 0 0) 0x7f3d101e7cf0 con 0x7f3cec078bc0 2024-09-17T03:51:14.109 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:14.110 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:14.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 msgr2=0x7f3cec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 0x7f3cec07b0d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3d00033000 tx=0x7f3d00004730 comp rx=0 tx=0).stop 2024-09-17T03:51:14.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 msgr2=0x7f3d1015d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800e960 tx=0x7f3d0800ee20 comp rx=0 tx=0).stop 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 shutdown_connections 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3cec078bc0 0x7f3cec07b0d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d1009aa30 0x7f3d1015d990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d100cad30 0x7f3d1015d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 --2- 172.21.15.129:0/2144186662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d100ca800 0x7f3d101597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 >> 172.21.15.129:0/2144186662 conn(0x7f3d100b1cb0 msgr2=0x7f3d1015b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:14.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 shutdown_connections 2024-09-17T03:51:14.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.111+0000 7f3d15600640 1 -- 172.21.15.129:0/2144186662 wait complete. 2024-09-17T03:51:14.136 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-17T03:51:12.318473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":142,"acting":142,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":254528,"kb_used_data":5008,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720192,"statfs":{"total":773094113280,"available":772833476608,"internally_reserved":0,"allocated":5128192,"data_stored":2734586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.818537"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518100+0000","last_change":"2024-09-17T03:51:08.518100+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518100+0000","last_undegraded":"2024-09-17T03:51:08.518100+0000","last_fullsized":"2024-09-17T03:51:08.518100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521176+0000","last_change":"2024-09-17T03:51:08.521176+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521176+0000","last_undegraded":"2024-09-17T03:51:08.521176+0000","last_fullsized":"2024-09-17T03:51:08.521176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.526816+0000","last_change":"2024-09-17T03:51:08.526816+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.526816+0000","last_undegraded":"2024-09-17T03:51:08.526816+0000","last_fullsized":"2024-09-17T03:51:08.526816+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521899+0000","last_change":"2024-09-17T03:51:08.521899+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521899+0000","last_undegraded":"2024-09-17T03:51:08.521899+0000","last_fullsized":"2024-09-17T03:51:08.521899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521676+0000","last_change":"2024-09-17T03:51:08.521676+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521676+0000","last_undegraded":"2024-09-17T03:51:08.521676+0000","last_fullsized":"2024-09-17T03:51:08.521676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518344+0000","last_change":"2024-09-17T03:51:08.518344+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518344+0000","last_undegraded":"2024-09-17T03:51:08.518344+0000","last_fullsized":"2024-09-17T03:51:08.518344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523028+0000","last_change":"2024-09-17T03:51:08.523028+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523028+0000","last_undegraded":"2024-09-17T03:51:08.523028+0000","last_fullsized":"2024-09-17T03:51:08.523028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521925+0000","last_change":"2024-09-17T03:51:08.521925+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521925+0000","last_undegraded":"2024-09-17T03:51:08.521925+0000","last_fullsized":"2024-09-17T03:51:08.521925+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520415+0000","last_change":"2024-09-17T03:51:08.520415+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520415+0000","last_undegraded":"2024-09-17T03:51:08.520415+0000","last_fullsized":"2024-09-17T03:51:08.520415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.525606+0000","last_change":"2024-09-17T03:51:08.525606+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.525606+0000","last_undegraded":"2024-09-17T03:51:08.525606+0000","last_fullsized":"2024-09-17T03:51:08.525606+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.524467+0000","last_change":"2024-09-17T03:51:08.524467+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.524467+0000","last_undegraded":"2024-09-17T03:51:08.524467+0000","last_fullsized":"2024-09-17T03:51:08.524467+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.524273+0000","last_change":"2024-09-17T03:51:08.524273+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.524273+0000","last_undegraded":"2024-09-17T03:51:08.524273+0000","last_fullsized":"2024-09-17T03:51:08.524273+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520494+0000","last_change":"2024-09-17T03:51:08.520494+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520494+0000","last_undegraded":"2024-09-17T03:51:08.520494+0000","last_fullsized":"2024-09-17T03:51:08.520494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.527164+0000","last_change":"2024-09-17T03:51:08.527164+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.527164+0000","last_undegraded":"2024-09-17T03:51:08.527164+0000","last_fullsized":"2024-09-17T03:51:08.527164+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.526237+0000","last_change":"2024-09-17T03:51:08.526237+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.526237+0000","last_undegraded":"2024-09-17T03:51:08.526237+0000","last_fullsized":"2024-09-17T03:51:08.526237+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.522524+0000","last_change":"2024-09-17T03:51:08.522524+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.522524+0000","last_undegraded":"2024-09-17T03:51:08.522524+0000","last_fullsized":"2024-09-17T03:51:08.522524+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523833+0000","last_change":"2024-09-17T03:51:08.523833+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523833+0000","last_undegraded":"2024-09-17T03:51:08.523833+0000","last_fullsized":"2024-09-17T03:51:08.523833+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.519903+0000","last_change":"2024-09-17T03:51:08.519903+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.519903+0000","last_undegraded":"2024-09-17T03:51:08.519903+0000","last_fullsized":"2024-09-17T03:51:08.519903+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521311+0000","last_change":"2024-09-17T03:51:08.521311+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521311+0000","last_undegraded":"2024-09-17T03:51:08.521311+0000","last_fullsized":"2024-09-17T03:51:08.521311+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514124+0000","last_change":"2024-09-17T03:51:08.514124+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514124+0000","last_undegraded":"2024-09-17T03:51:08.514124+0000","last_fullsized":"2024-09-17T03:51:08.514124+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539585+0000","last_change":"2024-09-17T03:51:08.536130+0000","last_active":"2024-09-17T03:51:08.539585+0000","last_peered":"2024-09-17T03:51:08.539585+0000","last_clean":"2024-09-17T03:51:08.539585+0000","last_became_active":"2024-09-17T03:51:08.535845+0000","last_became_peered":"2024-09-17T03:51:08.535845+0000","last_unstale":"2024-09-17T03:51:08.539585+0000","last_undegraded":"2024-09-17T03:51:08.539585+0000","last_fullsized":"2024-09-17T03:51:08.539585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.579356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.538800+0000","last_change":"2024-09-17T03:51:08.537833+0000","last_active":"2024-09-17T03:51:08.538800+0000","last_peered":"2024-09-17T03:51:08.538800+0000","last_clean":"2024-09-17T03:51:08.538800+0000","last_became_active":"2024-09-17T03:51:08.536906+0000","last_became_peered":"2024-09-17T03:51:08.536906+0000","last_unstale":"2024-09-17T03:51:08.538800+0000","last_undegraded":"2024-09-17T03:51:08.538800+0000","last_fullsized":"2024-09-17T03:51:08.538800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:59.377315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521613+0000","last_change":"2024-09-17T03:51:08.521613+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521613+0000","last_undegraded":"2024-09-17T03:51:08.521613+0000","last_fullsized":"2024-09-17T03:51:08.521613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535582+0000","last_change":"2024-09-17T03:51:08.532550+0000","last_active":"2024-09-17T03:51:08.535582+0000","last_peered":"2024-09-17T03:51:08.535582+0000","last_clean":"2024-09-17T03:51:08.535582+0000","last_became_active":"2024-09-17T03:51:08.532173+0000","last_became_peered":"2024-09-17T03:51:08.532173+0000","last_unstale":"2024-09-17T03:51:08.535582+0000","last_undegraded":"2024-09-17T03:51:08.535582+0000","last_fullsized":"2024-09-17T03:51:08.535582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:22.458883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514188+0000","last_change":"2024-09-17T03:51:08.514188+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514188+0000","last_undegraded":"2024-09-17T03:51:08.514188+0000","last_fullsized":"2024-09-17T03:51:08.514188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541632+0000","last_change":"2024-09-17T03:51:08.538223+0000","last_active":"2024-09-17T03:51:08.541632+0000","last_peered":"2024-09-17T03:51:08.541632+0000","last_clean":"2024-09-17T03:51:08.541632+0000","last_became_active":"2024-09-17T03:51:08.536819+0000","last_became_peered":"2024-09-17T03:51:08.536819+0000","last_unstale":"2024-09-17T03:51:08.541632+0000","last_undegraded":"2024-09-17T03:51:08.541632+0000","last_fullsized":"2024-09-17T03:51:08.541632+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523255+0000","last_change":"2024-09-17T03:51:08.523255+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523255+0000","last_undegraded":"2024-09-17T03:51:08.523255+0000","last_fullsized":"2024-09-17T03:51:08.523255+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537328+0000","last_change":"2024-09-17T03:51:08.533400+0000","last_active":"2024-09-17T03:51:08.537328+0000","last_peered":"2024-09-17T03:51:08.537328+0000","last_clean":"2024-09-17T03:51:08.537328+0000","last_became_active":"2024-09-17T03:51:08.532958+0000","last_became_peered":"2024-09-17T03:51:08.532958+0000","last_unstale":"2024-09-17T03:51:08.537328+0000","last_undegraded":"2024-09-17T03:51:08.537328+0000","last_fullsized":"2024-09-17T03:51:08.537328+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:41.046821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537431+0000","last_change":"2024-09-17T03:51:08.535613+0000","last_active":"2024-09-17T03:51:08.537431+0000","last_peered":"2024-09-17T03:51:08.537431+0000","last_clean":"2024-09-17T03:51:08.537431+0000","last_became_active":"2024-09-17T03:51:08.535214+0000","last_became_peered":"2024-09-17T03:51:08.535214+0000","last_unstale":"2024-09-17T03:51:08.537431+0000","last_undegraded":"2024-09-17T03:51:08.537431+0000","last_fullsized":"2024-09-17T03:51:08.537431+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:16:17.931275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536383+0000","last_change":"2024-09-17T03:51:08.532902+0000","last_active":"2024-09-17T03:51:08.536383+0000","last_peered":"2024-09-17T03:51:08.536383+0000","last_clean":"2024-09-17T03:51:08.536383+0000","last_became_active":"2024-09-17T03:51:08.532401+0000","last_became_peered":"2024-09-17T03:51:08.532401+0000","last_unstale":"2024-09-17T03:51:08.536383+0000","last_undegraded":"2024-09-17T03:51:08.536383+0000","last_fullsized":"2024-09-17T03:51:08.536383+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-09-17T03:51:07.506608+0000","last_change":"2024-09-17T03:51:07.506608+0000","last_active":"2024-09-17T03:51:07.492216+0000","last_peered":"2024-09-17T03:51:07.492216+0000","last_clean":"2024-09-17T03:51:07.492216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:07.506608+0000","last_undegraded":"2024-09-17T03:51:07.506608+0000","last_fullsized":"2024-09-17T03:51:07.506608+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523946+0000","last_change":"2024-09-17T03:51:08.523946+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523946+0000","last_undegraded":"2024-09-17T03:51:08.523946+0000","last_fullsized":"2024-09-17T03:51:08.523946+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537305+0000","last_change":"2024-09-17T03:51:08.535319+0000","last_active":"2024-09-17T03:51:08.537305+0000","last_peered":"2024-09-17T03:51:08.537305+0000","last_clean":"2024-09-17T03:51:08.537305+0000","last_became_active":"2024-09-17T03:51:08.534878+0000","last_became_peered":"2024-09-17T03:51:08.534878+0000","last_unstale":"2024-09-17T03:51:08.537305+0000","last_undegraded":"2024-09-17T03:51:08.537305+0000","last_fullsized":"2024-09-17T03:51:08.537305+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:45.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517437+0000","last_change":"2024-09-17T03:51:08.517437+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517437+0000","last_undegraded":"2024-09-17T03:51:08.517437+0000","last_fullsized":"2024-09-17T03:51:08.517437+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-09-17T03:51:07.505496+0000","last_change":"2024-09-17T03:51:07.505496+0000","last_active":"2024-09-17T03:51:07.492216+0000","last_peered":"2024-09-17T03:51:07.492216+0000","last_clean":"2024-09-17T03:51:07.492216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:07.505496+0000","last_undegraded":"2024-09-17T03:51:07.505496+0000","last_fullsized":"2024-09-17T03:51:07.505496+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518389+0000","last_change":"2024-09-17T03:51:08.518389+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518389+0000","last_undegraded":"2024-09-17T03:51:08.518389+0000","last_fullsized":"2024-09-17T03:51:08.518389+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542851+0000","last_change":"2024-09-17T03:51:08.541002+0000","last_active":"2024-09-17T03:51:08.542851+0000","last_peered":"2024-09-17T03:51:08.542851+0000","last_clean":"2024-09-17T03:51:08.542851+0000","last_became_active":"2024-09-17T03:51:08.540686+0000","last_became_peered":"2024-09-17T03:51:08.540686+0000","last_unstale":"2024-09-17T03:51:08.542851+0000","last_undegraded":"2024-09-17T03:51:08.542851+0000","last_fullsized":"2024-09-17T03:51:08.542851+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:20.604244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518861+0000","last_change":"2024-09-17T03:51:08.518861+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518861+0000","last_undegraded":"2024-09-17T03:51:08.518861+0000","last_fullsized":"2024-09-17T03:51:08.518861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542072+0000","last_change":"2024-09-17T03:51:08.538266+0000","last_active":"2024-09-17T03:51:08.542072+0000","last_peered":"2024-09-17T03:51:08.542072+0000","last_clean":"2024-09-17T03:51:08.542072+0000","last_became_active":"2024-09-17T03:51:08.537932+0000","last_became_peered":"2024-09-17T03:51:08.537932+0000","last_unstale":"2024-09-17T03:51:08.542072+0000","last_undegraded":"2024-09-17T03:51:08.542072+0000","last_fullsized":"2024-09-17T03:51:08.542072+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:24.642565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.533653+0000","last_change":"2024-09-17T03:51:08.528682+0000","last_active":"2024-09-17T03:51:08.533653+0000","last_peered":"2024-09-17T03:51:08.533653+0000","last_clean":"2024-09-17T03:51:08.533653+0000","last_became_active":"2024-09-17T03:51:08.527975+0000","last_became_peered":"2024-09-17T03:51:08.527975+0000","last_unstale":"2024-09-17T03:51:08.533653+0000","last_undegraded":"2024-09-17T03:51:08.533653+0000","last_fullsized":"2024-09-17T03:51:08.533653+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541590+0000","last_change":"2024-09-17T03:51:08.538721+0000","last_active":"2024-09-17T03:51:08.541590+0000","last_peered":"2024-09-17T03:51:08.541590+0000","last_clean":"2024-09-17T03:51:08.541590+0000","last_became_active":"2024-09-17T03:51:08.537978+0000","last_became_peered":"2024-09-17T03:51:08.537978+0000","last_unstale":"2024-09-17T03:51:08.541590+0000","last_undegraded":"2024-09-17T03:51:08.541590+0000","last_fullsized":"2024-09-17T03:51:08.541590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520408+0000","last_change":"2024-09-17T03:51:08.520408+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520408+0000","last_undegraded":"2024-09-17T03:51:08.520408+0000","last_fullsized":"2024-09-17T03:51:08.520408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539559+0000","last_change":"2024-09-17T03:51:08.535323+0000","last_active":"2024-09-17T03:51:08.539559+0000","last_peered":"2024-09-17T03:51:08.539559+0000","last_clean":"2024-09-17T03:51:08.539559+0000","last_became_active":"2024-09-17T03:51:08.534999+0000","last_became_peered":"2024-09-17T03:51:08.534999+0000","last_unstale":"2024-09-17T03:51:08.539559+0000","last_undegraded":"2024-09-17T03:51:08.539559+0000","last_fullsized":"2024-09-17T03:51:08.539559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:04.103238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-09-17T03:51:07.506020+0000","last_change":"2024-09-17T03:51:07.506020+0000","last_active":"2024-09-17T03:51:07.492216+0000","last_peered":"2024-09-17T03:51:07.492216+0000","last_clean":"2024-09-17T03:51:07.492216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:07.506020+0000","last_undegraded":"2024-09-17T03:51:07.506020+0000","last_fullsized":"2024-09-17T03:51:07.506020+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.525078+0000","last_change":"2024-09-17T03:51:08.525078+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.525078+0000","last_undegraded":"2024-09-17T03:51:08.525078+0000","last_fullsized":"2024-09-17T03:51:08.525078+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539208+0000","last_change":"2024-09-17T03:51:08.535330+0000","last_active":"2024-09-17T03:51:08.539208+0000","last_peered":"2024-09-17T03:51:08.539208+0000","last_clean":"2024-09-17T03:51:08.539208+0000","last_became_active":"2024-09-17T03:51:08.535035+0000","last_became_peered":"2024-09-17T03:51:08.535035+0000","last_unstale":"2024-09-17T03:51:08.539208+0000","last_undegraded":"2024-09-17T03:51:08.539208+0000","last_fullsized":"2024-09-17T03:51:08.539208+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:55.331543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.533659+0000","last_change":"2024-09-17T03:51:08.529671+0000","last_active":"2024-09-17T03:51:08.533659+0000","last_peered":"2024-09-17T03:51:08.533659+0000","last_clean":"2024-09-17T03:51:08.533659+0000","last_became_active":"2024-09-17T03:51:08.529313+0000","last_became_peered":"2024-09-17T03:51:08.529313+0000","last_unstale":"2024-09-17T03:51:08.533659+0000","last_undegraded":"2024-09-17T03:51:08.533659+0000","last_fullsized":"2024-09-17T03:51:08.533659+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:45.025999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535766+0000","last_change":"2024-09-17T03:51:08.531273+0000","last_active":"2024-09-17T03:51:08.535766+0000","last_peered":"2024-09-17T03:51:08.535766+0000","last_clean":"2024-09-17T03:51:08.535766+0000","last_became_active":"2024-09-17T03:51:08.530933+0000","last_became_peered":"2024-09-17T03:51:08.530933+0000","last_unstale":"2024-09-17T03:51:08.535766+0000","last_undegraded":"2024-09-17T03:51:08.535766+0000","last_fullsized":"2024-09-17T03:51:08.535766+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539491+0000","last_change":"2024-09-17T03:51:08.534140+0000","last_active":"2024-09-17T03:51:08.539491+0000","last_peered":"2024-09-17T03:51:08.539491+0000","last_clean":"2024-09-17T03:51:08.539491+0000","last_became_active":"2024-09-17T03:51:08.533878+0000","last_became_peered":"2024-09-17T03:51:08.533878+0000","last_unstale":"2024-09-17T03:51:08.539491+0000","last_undegraded":"2024-09-17T03:51:08.539491+0000","last_fullsized":"2024-09-17T03:51:08.539491+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:18.671953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521195+0000","last_change":"2024-09-17T03:51:08.521195+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521195+0000","last_undegraded":"2024-09-17T03:51:08.521195+0000","last_fullsized":"2024-09-17T03:51:08.521195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537539+0000","last_change":"2024-09-17T03:51:08.533378+0000","last_active":"2024-09-17T03:51:08.537539+0000","last_peered":"2024-09-17T03:51:08.537539+0000","last_clean":"2024-09-17T03:51:08.537539+0000","last_became_active":"2024-09-17T03:51:08.532682+0000","last_became_peered":"2024-09-17T03:51:08.532682+0000","last_unstale":"2024-09-17T03:51:08.537539+0000","last_undegraded":"2024-09-17T03:51:08.537539+0000","last_fullsized":"2024-09-17T03:51:08.537539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.184648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514113+0000","last_change":"2024-09-17T03:51:08.514113+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514113+0000","last_undegraded":"2024-09-17T03:51:08.514113+0000","last_fullsized":"2024-09-17T03:51:08.514113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535852+0000","last_change":"2024-09-17T03:51:08.531986+0000","last_active":"2024-09-17T03:51:08.535852+0000","last_peered":"2024-09-17T03:51:08.535852+0000","last_clean":"2024-09-17T03:51:08.535852+0000","last_became_active":"2024-09-17T03:51:08.531674+0000","last_became_peered":"2024-09-17T03:51:08.531674+0000","last_unstale":"2024-09-17T03:51:08.535852+0000","last_undegraded":"2024-09-17T03:51:08.535852+0000","last_fullsized":"2024-09-17T03:51:08.535852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520758+0000","last_change":"2024-09-17T03:51:08.520758+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520758+0000","last_undegraded":"2024-09-17T03:51:08.520758+0000","last_fullsized":"2024-09-17T03:51:08.520758+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-09-17T03:51:07.505393+0000","last_change":"2024-09-17T03:51:07.505393+0000","last_active":"2024-09-17T03:51:07.492216+0000","last_peered":"2024-09-17T03:51:07.492216+0000","last_clean":"2024-09-17T03:51:07.492216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:07.505393+0000","last_undegraded":"2024-09-17T03:51:07.505393+0000","last_fullsized":"2024-09-17T03:51:07.505393+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541521+0000","last_change":"2024-09-17T03:51:08.535527+0000","last_active":"2024-09-17T03:51:08.541521+0000","last_peered":"2024-09-17T03:51:08.541521+0000","last_clean":"2024-09-17T03:51:08.541521+0000","last_became_active":"2024-09-17T03:51:08.535182+0000","last_became_peered":"2024-09-17T03:51:08.535182+0000","last_unstale":"2024-09-17T03:51:08.541521+0000","last_undegraded":"2024-09-17T03:51:08.541521+0000","last_fullsized":"2024-09-17T03:51:08.541521+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537417+0000","last_change":"2024-09-17T03:51:08.531555+0000","last_active":"2024-09-17T03:51:08.537417+0000","last_peered":"2024-09-17T03:51:08.537417+0000","last_clean":"2024-09-17T03:51:08.537417+0000","last_became_active":"2024-09-17T03:51:08.531230+0000","last_became_peered":"2024-09-17T03:51:08.531230+0000","last_unstale":"2024-09-17T03:51:08.537417+0000","last_undegraded":"2024-09-17T03:51:08.537417+0000","last_fullsized":"2024-09-17T03:51:08.537417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:08.023937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.543358+0000","last_change":"2024-09-17T03:51:08.542303+0000","last_active":"2024-09-17T03:51:08.543358+0000","last_peered":"2024-09-17T03:51:08.543358+0000","last_clean":"2024-09-17T03:51:08.543358+0000","last_became_active":"2024-09-17T03:51:08.541884+0000","last_became_peered":"2024-09-17T03:51:08.541884+0000","last_unstale":"2024-09-17T03:51:08.543358+0000","last_undegraded":"2024-09-17T03:51:08.543358+0000","last_fullsized":"2024-09-17T03:51:08.543358+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.838696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536023+0000","last_change":"2024-09-17T03:51:08.533982+0000","last_active":"2024-09-17T03:51:08.536023+0000","last_peered":"2024-09-17T03:51:08.536023+0000","last_clean":"2024-09-17T03:51:08.536023+0000","last_became_active":"2024-09-17T03:51:08.529828+0000","last_became_peered":"2024-09-17T03:51:08.529828+0000","last_unstale":"2024-09-17T03:51:08.536023+0000","last_undegraded":"2024-09-17T03:51:08.536023+0000","last_fullsized":"2024-09-17T03:51:08.536023+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:18.636858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520810+0000","last_change":"2024-09-17T03:51:08.520810+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520810+0000","last_undegraded":"2024-09-17T03:51:08.520810+0000","last_fullsized":"2024-09-17T03:51:08.520810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542733+0000","last_change":"2024-09-17T03:51:08.540515+0000","last_active":"2024-09-17T03:51:08.542733+0000","last_peered":"2024-09-17T03:51:08.542733+0000","last_clean":"2024-09-17T03:51:08.542733+0000","last_became_active":"2024-09-17T03:51:08.540197+0000","last_became_peered":"2024-09-17T03:51:08.540197+0000","last_unstale":"2024-09-17T03:51:08.542733+0000","last_undegraded":"2024-09-17T03:51:08.542733+0000","last_fullsized":"2024-09-17T03:51:08.542733+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520049+0000","last_change":"2024-09-17T03:51:08.520049+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520049+0000","last_undegraded":"2024-09-17T03:51:08.520049+0000","last_fullsized":"2024-09-17T03:51:08.520049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542840+0000","last_change":"2024-09-17T03:51:08.538899+0000","last_active":"2024-09-17T03:51:08.542840+0000","last_peered":"2024-09-17T03:51:08.542840+0000","last_clean":"2024-09-17T03:51:08.542840+0000","last_became_active":"2024-09-17T03:51:08.538338+0000","last_became_peered":"2024-09-17T03:51:08.538338+0000","last_unstale":"2024-09-17T03:51:08.542840+0000","last_undegraded":"2024-09-17T03:51:08.542840+0000","last_fullsized":"2024-09-17T03:51:08.542840+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:03.099874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520013+0000","last_change":"2024-09-17T03:51:08.520013+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520013+0000","last_undegraded":"2024-09-17T03:51:08.520013+0000","last_fullsized":"2024-09-17T03:51:08.520013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536701+0000","last_change":"2024-09-17T03:51:08.535001+0000","last_active":"2024-09-17T03:51:08.536701+0000","last_peered":"2024-09-17T03:51:08.536701+0000","last_clean":"2024-09-17T03:51:08.536701+0000","last_became_active":"2024-09-17T03:51:08.531914+0000","last_became_peered":"2024-09-17T03:51:08.531914+0000","last_unstale":"2024-09-17T03:51:08.536701+0000","last_undegraded":"2024-09-17T03:51:08.536701+0000","last_fullsized":"2024-09-17T03:51:08.536701+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:10.120644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520905+0000","last_change":"2024-09-17T03:51:08.520905+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520905+0000","last_undegraded":"2024-09-17T03:51:08.520905+0000","last_fullsized":"2024-09-17T03:51:08.520905+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":42,"state":"creating+peering","last_fresh":"2024-09-17T03:51:07.505558+0000","last_change":"2024-09-17T03:51:07.505558+0000","last_active":"2024-09-17T03:51:07.492216+0000","last_peered":"2024-09-17T03:51:07.492216+0000","last_clean":"2024-09-17T03:51:07.492216+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:07.505558+0000","last_undegraded":"2024-09-17T03:51:07.505558+0000","last_fullsized":"2024-09-17T03:51:07.505558+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:51:08.499499+0000","last_change":"2024-09-17T03:51:08.499499+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.499499+0000","last_undegraded":"2024-09-17T03:51:08.499499+0000","last_fullsized":"2024-09-17T03:51:08.499499+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535595+0000","last_change":"2024-09-17T03:51:08.529407+0000","last_active":"2024-09-17T03:51:08.535595+0000","last_peered":"2024-09-17T03:51:08.535595+0000","last_clean":"2024-09-17T03:51:08.535595+0000","last_became_active":"2024-09-17T03:51:08.529046+0000","last_became_peered":"2024-09-17T03:51:08.529046+0000","last_unstale":"2024-09-17T03:51:08.535595+0000","last_undegraded":"2024-09-17T03:51:08.535595+0000","last_fullsized":"2024-09-17T03:51:08.535595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":148,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.524741+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:08.524741+0000","last_peered":"2024-09-17T03:51:08.524741+0000","last_clean":"2024-09-17T03:51:08.524741+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:08.524741+0000","last_undegraded":"2024-09-17T03:51:08.524741+0000","last_fullsized":"2024-09-17T03:51:08.524741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":76,"acting":76,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1089536,"data_stored":786166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509466,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542167,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":189531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542168,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1077248,"data_stored":779899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:14.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 -- 172.21.15.129:0/3670451302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414073170 msgr2=0x7f141415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 --2- 172.21.15.129:0/3670451302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414073170 0x7f141415c780 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1404005fd0 tx=0x7f140402a820 comp rx=0 tx=0).stop 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 -- 172.21.15.129:0/3670451302 shutdown_connections 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 --2- 172.21.15.129:0/3670451302 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 0x7f1414074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 --2- 172.21.15.129:0/3670451302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414073170 0x7f141415c780 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 -- 172.21.15.129:0/3670451302 >> 172.21.15.129:0/3670451302 conn(0x7f14140b1cb0 msgr2=0x7f14140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.399+0000 7f141b000640 1 -- 172.21.15.129:0/3670451302 shutdown_connections 2024-09-17T03:51:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.400+0000 7f141b000640 1 -- 172.21.15.129:0/3670451302 wait complete. 2024-09-17T03:51:14.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.402+0000 7f141b000640 1 Processor -- start 2024-09-17T03:51:14.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.402+0000 7f141b000640 1 -- start start 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 0x7f141414ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1414073170 0x7f14140cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f141406f5a0 con 0x7f1414070440 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1414074fe0 con 0x7f1414073170 2024-09-17T03:51:14.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f141b000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f141406f720 con 0x7f1414070030 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.403+0000 7f1413400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39390/0 (socket says 172.21.15.129:39390) 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 -- 172.21.15.129:0/2823531636 learned_addr learned my addr 172.21.15.129:0/2823531636 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413e00640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 0x7f141414ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1419000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1414073170 0x7f14140cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 -- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 msgr2=0x7f141414ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 0x7f141414ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 -- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1414073170 msgr2=0x7f14140cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1414073170 0x7f14140cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 -- 172.21.15.129:0/2823531636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14140c8930 con 0x7f1414070440 2024-09-17T03:51:14.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f1413400640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f140000d650 tx=0x7f140000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.404+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400017070 con 0x7f1414070440 2024-09-17T03:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.405+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1400009b10 con 0x7f1414070440 2024-09-17T03:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.405+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14141f8f40 con 0x7f1414070440 2024-09-17T03:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.405+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400010460 con 0x7f1414070440 2024-09-17T03:51:14.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.405+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141f9500 con 0x7f1414070440 2024-09-17T03:51:14.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.406+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141406fc40 con 0x7f1414070440 2024-09-17T03:51:14.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.407+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f140001a070 con 0x7f1414070440 2024-09-17T03:51:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.408+0000 7f140be00640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 0x7f13f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.408+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f140009adb0 con 0x7f1414070440 2024-09-17T03:51:14.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.408+0000 7f1413e00640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 0x7f13f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.409+0000 7f1413e00640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 0x7f13f407b0a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1404034000 tx=0x7f1404004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:14.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.412+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f140009f070 con 0x7f1414070440 2024-09-17T03:51:14.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.595+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14140c3860 con 0x7f13f4078b90 2024-09-17T03:51:14.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.600+0000 7f140be00640 1 -- 172.21.15.129:0/2823531636 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249961 (secure 0 0 0) 0x7f14140c3860 con 0x7f13f4078b90 2024-09-17T03:51:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:14.603 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:14.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 msgr2=0x7f13f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 0x7f13f407b0a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1404034000 tx=0x7f1404004ab0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 msgr2=0x7f14140c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f140000d650 tx=0x7f140000db10 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 shutdown_connections 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f13f4078b90 0x7f13f407b0a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1414073170 0x7f14140cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1414070440 0x7f14140c7e90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 --2- 172.21.15.129:0/2823531636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1414070030 0x7f141414ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 >> 172.21.15.129:0/2823531636 conn(0x7f14140b1cb0 msgr2=0x7f14140ae0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 shutdown_connections 2024-09-17T03:51:14.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.604+0000 7f141b000640 1 -- 172.21.15.129:0/2823531636 wait complete. 2024-09-17T03:51:14.625 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-17T03:51:14.318893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254692,"kb_used_data":5168,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720028,"statfs":{"total":773094113280,"available":772833308672,"internally_reserved":0,"allocated":5292032,"data_stored":2803372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518100+0000","last_change":"2024-09-17T03:51:08.518100+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518100+0000","last_undegraded":"2024-09-17T03:51:08.518100+0000","last_fullsized":"2024-09-17T03:51:08.518100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521176+0000","last_change":"2024-09-17T03:51:08.521176+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521176+0000","last_undegraded":"2024-09-17T03:51:08.521176+0000","last_fullsized":"2024-09-17T03:51:08.521176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532794+0000","last_change":"2024-09-17T03:51:09.544780+0000","last_active":"2024-09-17T03:51:10.532794+0000","last_peered":"2024-09-17T03:51:10.532794+0000","last_clean":"2024-09-17T03:51:10.532794+0000","last_became_active":"2024-09-17T03:51:09.544462+0000","last_became_peered":"2024-09-17T03:51:09.544462+0000","last_unstale":"2024-09-17T03:51:10.532794+0000","last_undegraded":"2024-09-17T03:51:10.532794+0000","last_fullsized":"2024-09-17T03:51:10.532794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:20.077167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787357+0000","last_change":"2024-09-17T03:51:09.553310+0000","last_active":"2024-09-17T03:51:10.787357+0000","last_peered":"2024-09-17T03:51:10.787357+0000","last_clean":"2024-09-17T03:51:10.787357+0000","last_became_active":"2024-09-17T03:51:09.552339+0000","last_became_peered":"2024-09-17T03:51:09.552339+0000","last_unstale":"2024-09-17T03:51:10.787357+0000","last_undegraded":"2024-09-17T03:51:10.787357+0000","last_fullsized":"2024-09-17T03:51:10.787357+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:59.902480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521899+0000","last_change":"2024-09-17T03:51:08.521899+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521899+0000","last_undegraded":"2024-09-17T03:51:08.521899+0000","last_fullsized":"2024-09-17T03:51:08.521899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521676+0000","last_change":"2024-09-17T03:51:08.521676+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521676+0000","last_undegraded":"2024-09-17T03:51:08.521676+0000","last_fullsized":"2024-09-17T03:51:08.521676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518344+0000","last_change":"2024-09-17T03:51:08.518344+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518344+0000","last_undegraded":"2024-09-17T03:51:08.518344+0000","last_fullsized":"2024-09-17T03:51:08.518344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785916+0000","last_change":"2024-09-17T03:51:09.553571+0000","last_active":"2024-09-17T03:51:10.785916+0000","last_peered":"2024-09-17T03:51:10.785916+0000","last_clean":"2024-09-17T03:51:10.785916+0000","last_became_active":"2024-09-17T03:51:09.553226+0000","last_became_peered":"2024-09-17T03:51:09.553226+0000","last_unstale":"2024-09-17T03:51:10.785916+0000","last_undegraded":"2024-09-17T03:51:10.785916+0000","last_fullsized":"2024-09-17T03:51:10.785916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:13.896356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787450+0000","last_change":"2024-09-17T03:51:09.558849+0000","last_active":"2024-09-17T03:51:10.787450+0000","last_peered":"2024-09-17T03:51:10.787450+0000","last_clean":"2024-09-17T03:51:10.787450+0000","last_became_active":"2024-09-17T03:51:09.558596+0000","last_became_peered":"2024-09-17T03:51:09.558596+0000","last_unstale":"2024-09-17T03:51:10.787450+0000","last_undegraded":"2024-09-17T03:51:10.787450+0000","last_fullsized":"2024-09-17T03:51:10.787450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:19.302750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787683+0000","last_change":"2024-09-17T03:51:09.552932+0000","last_active":"2024-09-17T03:51:10.787683+0000","last_peered":"2024-09-17T03:51:10.787683+0000","last_clean":"2024-09-17T03:51:10.787683+0000","last_became_active":"2024-09-17T03:51:09.552660+0000","last_became_peered":"2024-09-17T03:51:09.552660+0000","last_unstale":"2024-09-17T03:51:10.787683+0000","last_undegraded":"2024-09-17T03:51:10.787683+0000","last_fullsized":"2024-09-17T03:51:10.787683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:45.040579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787563+0000","last_change":"2024-09-17T03:51:09.556091+0000","last_active":"2024-09-17T03:51:10.787563+0000","last_peered":"2024-09-17T03:51:10.787563+0000","last_clean":"2024-09-17T03:51:10.787563+0000","last_became_active":"2024-09-17T03:51:09.555788+0000","last_became_peered":"2024-09-17T03:51:09.555788+0000","last_unstale":"2024-09-17T03:51:10.787563+0000","last_undegraded":"2024-09-17T03:51:10.787563+0000","last_fullsized":"2024-09-17T03:51:10.787563+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:35.856529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523028+0000","last_change":"2024-09-17T03:51:08.523028+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523028+0000","last_undegraded":"2024-09-17T03:51:08.523028+0000","last_fullsized":"2024-09-17T03:51:08.523028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786586+0000","last_change":"2024-09-17T03:51:09.549923+0000","last_active":"2024-09-17T03:51:10.786586+0000","last_peered":"2024-09-17T03:51:10.786586+0000","last_clean":"2024-09-17T03:51:10.786586+0000","last_became_active":"2024-09-17T03:51:09.549427+0000","last_became_peered":"2024-09-17T03:51:09.549427+0000","last_unstale":"2024-09-17T03:51:10.786586+0000","last_undegraded":"2024-09-17T03:51:10.786586+0000","last_fullsized":"2024-09-17T03:51:10.786586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:12.588911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786097+0000","last_change":"2024-09-17T03:51:09.557049+0000","last_active":"2024-09-17T03:51:10.786097+0000","last_peered":"2024-09-17T03:51:10.786097+0000","last_clean":"2024-09-17T03:51:10.786097+0000","last_became_active":"2024-09-17T03:51:09.556732+0000","last_became_peered":"2024-09-17T03:51:09.556732+0000","last_unstale":"2024-09-17T03:51:10.786097+0000","last_undegraded":"2024-09-17T03:51:10.786097+0000","last_fullsized":"2024-09-17T03:51:10.786097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:51.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520415+0000","last_change":"2024-09-17T03:51:08.520415+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520415+0000","last_undegraded":"2024-09-17T03:51:08.520415+0000","last_fullsized":"2024-09-17T03:51:08.520415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786655+0000","last_change":"2024-09-17T03:51:09.552281+0000","last_active":"2024-09-17T03:51:10.786655+0000","last_peered":"2024-09-17T03:51:10.786655+0000","last_clean":"2024-09-17T03:51:10.786655+0000","last_became_active":"2024-09-17T03:51:09.551898+0000","last_became_peered":"2024-09-17T03:51:09.551898+0000","last_unstale":"2024-09-17T03:51:10.786655+0000","last_undegraded":"2024-09-17T03:51:10.786655+0000","last_fullsized":"2024-09-17T03:51:10.786655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:19.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533660+0000","last_change":"2024-09-17T03:51:09.533608+0000","last_active":"2024-09-17T03:51:10.533660+0000","last_peered":"2024-09-17T03:51:10.533660+0000","last_clean":"2024-09-17T03:51:10.533660+0000","last_became_active":"2024-09-17T03:51:09.533263+0000","last_became_peered":"2024-09-17T03:51:09.533263+0000","last_unstale":"2024-09-17T03:51:10.533660+0000","last_undegraded":"2024-09-17T03:51:10.533660+0000","last_fullsized":"2024-09-17T03:51:10.533660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:18.712043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532553+0000","last_change":"2024-09-17T03:51:09.549893+0000","last_active":"2024-09-17T03:51:10.532553+0000","last_peered":"2024-09-17T03:51:10.532553+0000","last_clean":"2024-09-17T03:51:10.532553+0000","last_became_active":"2024-09-17T03:51:09.548155+0000","last_became_peered":"2024-09-17T03:51:09.548155+0000","last_unstale":"2024-09-17T03:51:10.532553+0000","last_undegraded":"2024-09-17T03:51:10.532553+0000","last_fullsized":"2024-09-17T03:51:10.532553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:06.020247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787285+0000","last_change":"2024-09-17T03:51:09.545828+0000","last_active":"2024-09-17T03:51:10.787285+0000","last_peered":"2024-09-17T03:51:10.787285+0000","last_clean":"2024-09-17T03:51:10.787285+0000","last_became_active":"2024-09-17T03:51:09.545207+0000","last_became_peered":"2024-09-17T03:51:09.545207+0000","last_unstale":"2024-09-17T03:51:10.787285+0000","last_undegraded":"2024-09-17T03:51:10.787285+0000","last_fullsized":"2024-09-17T03:51:10.787285+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:11.346092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786753+0000","last_change":"2024-09-17T03:51:09.550468+0000","last_active":"2024-09-17T03:51:10.786753+0000","last_peered":"2024-09-17T03:51:10.786753+0000","last_clean":"2024-09-17T03:51:10.786753+0000","last_became_active":"2024-09-17T03:51:09.550210+0000","last_became_peered":"2024-09-17T03:51:09.550210+0000","last_unstale":"2024-09-17T03:51:10.786753+0000","last_undegraded":"2024-09-17T03:51:10.786753+0000","last_fullsized":"2024-09-17T03:51:10.786753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:55.396574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520494+0000","last_change":"2024-09-17T03:51:08.520494+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520494+0000","last_undegraded":"2024-09-17T03:51:08.520494+0000","last_fullsized":"2024-09-17T03:51:08.520494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786228+0000","last_change":"2024-09-17T03:51:09.551749+0000","last_active":"2024-09-17T03:51:10.786228+0000","last_peered":"2024-09-17T03:51:10.786228+0000","last_clean":"2024-09-17T03:51:10.786228+0000","last_became_active":"2024-09-17T03:51:09.551425+0000","last_became_peered":"2024-09-17T03:51:09.551425+0000","last_unstale":"2024-09-17T03:51:10.786228+0000","last_undegraded":"2024-09-17T03:51:10.786228+0000","last_fullsized":"2024-09-17T03:51:10.786228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:31.877445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786776+0000","last_change":"2024-09-17T03:51:09.537760+0000","last_active":"2024-09-17T03:51:10.786776+0000","last_peered":"2024-09-17T03:51:10.786776+0000","last_clean":"2024-09-17T03:51:10.786776+0000","last_became_active":"2024-09-17T03:51:09.537430+0000","last_became_peered":"2024-09-17T03:51:09.537430+0000","last_unstale":"2024-09-17T03:51:10.786776+0000","last_undegraded":"2024-09-17T03:51:10.786776+0000","last_fullsized":"2024-09-17T03:51:10.786776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:40.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786706+0000","last_change":"2024-09-17T03:51:09.535683+0000","last_active":"2024-09-17T03:51:10.786706+0000","last_peered":"2024-09-17T03:51:10.786706+0000","last_clean":"2024-09-17T03:51:10.786706+0000","last_became_active":"2024-09-17T03:51:09.535368+0000","last_became_peered":"2024-09-17T03:51:09.535368+0000","last_unstale":"2024-09-17T03:51:10.786706+0000","last_undegraded":"2024-09-17T03:51:10.786706+0000","last_fullsized":"2024-09-17T03:51:10.786706+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:28.792675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787126+0000","last_change":"2024-09-17T03:51:09.531819+0000","last_active":"2024-09-17T03:51:10.787126+0000","last_peered":"2024-09-17T03:51:10.787126+0000","last_clean":"2024-09-17T03:51:10.787126+0000","last_became_active":"2024-09-17T03:51:09.531504+0000","last_became_peered":"2024-09-17T03:51:09.531504+0000","last_unstale":"2024-09-17T03:51:10.787126+0000","last_undegraded":"2024-09-17T03:51:10.787126+0000","last_fullsized":"2024-09-17T03:51:10.787126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:37.424982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786610+0000","last_change":"2024-09-17T03:51:09.561216+0000","last_active":"2024-09-17T03:51:10.786610+0000","last_peered":"2024-09-17T03:51:10.786610+0000","last_clean":"2024-09-17T03:51:10.786610+0000","last_became_active":"2024-09-17T03:51:09.560930+0000","last_became_peered":"2024-09-17T03:51:09.560930+0000","last_unstale":"2024-09-17T03:51:10.786610+0000","last_undegraded":"2024-09-17T03:51:10.786610+0000","last_fullsized":"2024-09-17T03:51:10.786610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:24.329326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533087+0000","last_change":"2024-09-17T03:51:09.532459+0000","last_active":"2024-09-17T03:51:10.533087+0000","last_peered":"2024-09-17T03:51:10.533087+0000","last_clean":"2024-09-17T03:51:10.533087+0000","last_became_active":"2024-09-17T03:51:09.532056+0000","last_became_peered":"2024-09-17T03:51:09.532056+0000","last_unstale":"2024-09-17T03:51:10.533087+0000","last_undegraded":"2024-09-17T03:51:10.533087+0000","last_fullsized":"2024-09-17T03:51:10.533087+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:07.247043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.522524+0000","last_change":"2024-09-17T03:51:08.522524+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.522524+0000","last_undegraded":"2024-09-17T03:51:08.522524+0000","last_fullsized":"2024-09-17T03:51:08.522524+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787120+0000","last_change":"2024-09-17T03:51:09.531111+0000","last_active":"2024-09-17T03:51:10.787120+0000","last_peered":"2024-09-17T03:51:10.787120+0000","last_clean":"2024-09-17T03:51:10.787120+0000","last_became_active":"2024-09-17T03:51:09.530769+0000","last_became_peered":"2024-09-17T03:51:09.530769+0000","last_unstale":"2024-09-17T03:51:10.787120+0000","last_undegraded":"2024-09-17T03:51:10.787120+0000","last_fullsized":"2024-09-17T03:51:10.787120+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:05.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.540408+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.540084+0000","last_became_peered":"2024-09-17T03:51:09.540084+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:09.874665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.519903+0000","last_change":"2024-09-17T03:51:08.519903+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.519903+0000","last_undegraded":"2024-09-17T03:51:08.519903+0000","last_fullsized":"2024-09-17T03:51:08.519903+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786259+0000","last_change":"2024-09-17T03:51:09.532104+0000","last_active":"2024-09-17T03:51:10.786259+0000","last_peered":"2024-09-17T03:51:10.786259+0000","last_clean":"2024-09-17T03:51:10.786259+0000","last_became_active":"2024-09-17T03:51:09.531801+0000","last_became_peered":"2024-09-17T03:51:09.531801+0000","last_unstale":"2024-09-17T03:51:10.786259+0000","last_undegraded":"2024-09-17T03:51:10.786259+0000","last_fullsized":"2024-09-17T03:51:10.786259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:40.757442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514124+0000","last_change":"2024-09-17T03:51:08.514124+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514124+0000","last_undegraded":"2024-09-17T03:51:08.514124+0000","last_fullsized":"2024-09-17T03:51:08.514124+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539585+0000","last_change":"2024-09-17T03:51:08.536130+0000","last_active":"2024-09-17T03:51:08.539585+0000","last_peered":"2024-09-17T03:51:08.539585+0000","last_clean":"2024-09-17T03:51:08.539585+0000","last_became_active":"2024-09-17T03:51:08.535845+0000","last_became_peered":"2024-09-17T03:51:08.535845+0000","last_unstale":"2024-09-17T03:51:08.539585+0000","last_undegraded":"2024-09-17T03:51:08.539585+0000","last_fullsized":"2024-09-17T03:51:08.539585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.579356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787189+0000","last_change":"2024-09-17T03:51:09.530989+0000","last_active":"2024-09-17T03:51:10.787189+0000","last_peered":"2024-09-17T03:51:10.787189+0000","last_clean":"2024-09-17T03:51:10.787189+0000","last_became_active":"2024-09-17T03:51:09.529800+0000","last_became_peered":"2024-09-17T03:51:09.529800+0000","last_unstale":"2024-09-17T03:51:10.787189+0000","last_undegraded":"2024-09-17T03:51:10.787189+0000","last_fullsized":"2024-09-17T03:51:10.787189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:22.141601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533204+0000","last_change":"2024-09-17T03:51:08.537833+0000","last_active":"2024-09-17T03:51:10.533204+0000","last_peered":"2024-09-17T03:51:10.533204+0000","last_clean":"2024-09-17T03:51:10.533204+0000","last_became_active":"2024-09-17T03:51:08.536906+0000","last_became_peered":"2024-09-17T03:51:08.536906+0000","last_unstale":"2024-09-17T03:51:10.533204+0000","last_undegraded":"2024-09-17T03:51:10.533204+0000","last_fullsized":"2024-09-17T03:51:10.533204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521613+0000","last_change":"2024-09-17T03:51:08.521613+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521613+0000","last_undegraded":"2024-09-17T03:51:08.521613+0000","last_fullsized":"2024-09-17T03:51:08.521613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535582+0000","last_change":"2024-09-17T03:51:08.532550+0000","last_active":"2024-09-17T03:51:08.535582+0000","last_peered":"2024-09-17T03:51:08.535582+0000","last_clean":"2024-09-17T03:51:08.535582+0000","last_became_active":"2024-09-17T03:51:08.532173+0000","last_became_peered":"2024-09-17T03:51:08.532173+0000","last_unstale":"2024-09-17T03:51:08.535582+0000","last_undegraded":"2024-09-17T03:51:08.535582+0000","last_fullsized":"2024-09-17T03:51:08.535582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:22.458883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514188+0000","last_change":"2024-09-17T03:51:08.514188+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514188+0000","last_undegraded":"2024-09-17T03:51:08.514188+0000","last_fullsized":"2024-09-17T03:51:08.514188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541632+0000","last_change":"2024-09-17T03:51:08.538223+0000","last_active":"2024-09-17T03:51:08.541632+0000","last_peered":"2024-09-17T03:51:08.541632+0000","last_clean":"2024-09-17T03:51:08.541632+0000","last_became_active":"2024-09-17T03:51:08.536819+0000","last_became_peered":"2024-09-17T03:51:08.536819+0000","last_unstale":"2024-09-17T03:51:08.541632+0000","last_undegraded":"2024-09-17T03:51:08.541632+0000","last_fullsized":"2024-09-17T03:51:08.541632+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.533568+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.533027+0000","last_became_peered":"2024-09-17T03:51:09.533027+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:39.579158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537328+0000","last_change":"2024-09-17T03:51:08.533400+0000","last_active":"2024-09-17T03:51:08.537328+0000","last_peered":"2024-09-17T03:51:08.537328+0000","last_clean":"2024-09-17T03:51:08.537328+0000","last_became_active":"2024-09-17T03:51:08.532958+0000","last_became_peered":"2024-09-17T03:51:08.532958+0000","last_unstale":"2024-09-17T03:51:08.537328+0000","last_undegraded":"2024-09-17T03:51:08.537328+0000","last_fullsized":"2024-09-17T03:51:08.537328+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:41.046821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786620+0000","last_change":"2024-09-17T03:51:09.543641+0000","last_active":"2024-09-17T03:51:10.786620+0000","last_peered":"2024-09-17T03:51:10.786620+0000","last_clean":"2024-09-17T03:51:10.786620+0000","last_became_active":"2024-09-17T03:51:09.543299+0000","last_became_peered":"2024-09-17T03:51:09.543299+0000","last_unstale":"2024-09-17T03:51:10.786620+0000","last_undegraded":"2024-09-17T03:51:10.786620+0000","last_fullsized":"2024-09-17T03:51:10.786620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:34.478969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534447+0000","last_change":"2024-09-17T03:51:08.535613+0000","last_active":"2024-09-17T03:51:10.534447+0000","last_peered":"2024-09-17T03:51:10.534447+0000","last_clean":"2024-09-17T03:51:10.534447+0000","last_became_active":"2024-09-17T03:51:08.535214+0000","last_became_peered":"2024-09-17T03:51:08.535214+0000","last_unstale":"2024-09-17T03:51:10.534447+0000","last_undegraded":"2024-09-17T03:51:10.534447+0000","last_fullsized":"2024-09-17T03:51:10.534447+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.931699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785982+0000","last_change":"2024-09-17T03:51:09.552322+0000","last_active":"2024-09-17T03:51:10.785982+0000","last_peered":"2024-09-17T03:51:10.785982+0000","last_clean":"2024-09-17T03:51:10.785982+0000","last_became_active":"2024-09-17T03:51:09.551914+0000","last_became_peered":"2024-09-17T03:51:09.551914+0000","last_unstale":"2024-09-17T03:51:10.785982+0000","last_undegraded":"2024-09-17T03:51:10.785982+0000","last_fullsized":"2024-09-17T03:51:10.785982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:38.111592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536383+0000","last_change":"2024-09-17T03:51:08.532902+0000","last_active":"2024-09-17T03:51:08.536383+0000","last_peered":"2024-09-17T03:51:08.536383+0000","last_clean":"2024-09-17T03:51:08.536383+0000","last_became_active":"2024-09-17T03:51:08.532401+0000","last_became_peered":"2024-09-17T03:51:08.532401+0000","last_unstale":"2024-09-17T03:51:08.536383+0000","last_undegraded":"2024-09-17T03:51:08.536383+0000","last_fullsized":"2024-09-17T03:51:08.536383+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786515+0000","last_change":"2024-09-17T03:51:09.541584+0000","last_active":"2024-09-17T03:51:10.786515+0000","last_peered":"2024-09-17T03:51:10.786515+0000","last_clean":"2024-09-17T03:51:10.786515+0000","last_became_active":"2024-09-17T03:51:09.541219+0000","last_became_peered":"2024-09-17T03:51:09.541219+0000","last_unstale":"2024-09-17T03:51:10.786515+0000","last_undegraded":"2024-09-17T03:51:10.786515+0000","last_fullsized":"2024-09-17T03:51:10.786515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:57.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785991+0000","last_change":"2024-09-17T03:51:08.522934+0000","last_active":"2024-09-17T03:51:10.785991+0000","last_peered":"2024-09-17T03:51:10.785991+0000","last_clean":"2024-09-17T03:51:10.785991+0000","last_became_active":"2024-09-17T03:51:08.522647+0000","last_became_peered":"2024-09-17T03:51:08.522647+0000","last_unstale":"2024-09-17T03:51:10.785991+0000","last_undegraded":"2024-09-17T03:51:10.785991+0000","last_fullsized":"2024-09-17T03:51:10.785991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:27.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532822+0000","last_change":"2024-09-17T03:51:09.550191+0000","last_active":"2024-09-17T03:51:10.532822+0000","last_peered":"2024-09-17T03:51:10.532822+0000","last_clean":"2024-09-17T03:51:10.532822+0000","last_became_active":"2024-09-17T03:51:09.548578+0000","last_became_peered":"2024-09-17T03:51:09.548578+0000","last_unstale":"2024-09-17T03:51:10.532822+0000","last_undegraded":"2024-09-17T03:51:10.532822+0000","last_fullsized":"2024-09-17T03:51:10.532822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:01.627905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533705+0000","last_change":"2024-09-17T03:51:08.535319+0000","last_active":"2024-09-17T03:51:10.533705+0000","last_peered":"2024-09-17T03:51:10.533705+0000","last_clean":"2024-09-17T03:51:10.533705+0000","last_became_active":"2024-09-17T03:51:08.534878+0000","last_became_peered":"2024-09-17T03:51:08.534878+0000","last_unstale":"2024-09-17T03:51:10.533705+0000","last_undegraded":"2024-09-17T03:51:10.533705+0000","last_fullsized":"2024-09-17T03:51:10.533705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517437+0000","last_change":"2024-09-17T03:51:08.517437+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517437+0000","last_undegraded":"2024-09-17T03:51:08.517437+0000","last_fullsized":"2024-09-17T03:51:08.517437+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787744+0000","last_change":"2024-09-17T03:51:08.538705+0000","last_active":"2024-09-17T03:51:10.787744+0000","last_peered":"2024-09-17T03:51:10.787744+0000","last_clean":"2024-09-17T03:51:10.787744+0000","last_became_active":"2024-09-17T03:51:08.535730+0000","last_became_peered":"2024-09-17T03:51:08.535730+0000","last_unstale":"2024-09-17T03:51:10.787744+0000","last_undegraded":"2024-09-17T03:51:10.787744+0000","last_fullsized":"2024-09-17T03:51:10.787744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:47.222685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518389+0000","last_change":"2024-09-17T03:51:08.518389+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518389+0000","last_undegraded":"2024-09-17T03:51:08.518389+0000","last_fullsized":"2024-09-17T03:51:08.518389+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542851+0000","last_change":"2024-09-17T03:51:08.541002+0000","last_active":"2024-09-17T03:51:08.542851+0000","last_peered":"2024-09-17T03:51:08.542851+0000","last_clean":"2024-09-17T03:51:08.542851+0000","last_became_active":"2024-09-17T03:51:08.540686+0000","last_became_peered":"2024-09-17T03:51:08.540686+0000","last_unstale":"2024-09-17T03:51:08.542851+0000","last_undegraded":"2024-09-17T03:51:08.542851+0000","last_fullsized":"2024-09-17T03:51:08.542851+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:20.604244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518861+0000","last_change":"2024-09-17T03:51:08.518861+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518861+0000","last_undegraded":"2024-09-17T03:51:08.518861+0000","last_fullsized":"2024-09-17T03:51:08.518861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":21,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786793+0000","last_change":"2024-09-17T03:51:08.538266+0000","last_active":"2024-09-17T03:51:10.786793+0000","last_peered":"2024-09-17T03:51:10.786793+0000","last_clean":"2024-09-17T03:51:10.786793+0000","last_became_active":"2024-09-17T03:51:08.537932+0000","last_became_peered":"2024-09-17T03:51:08.537932+0000","last_unstale":"2024-09-17T03:51:10.786793+0000","last_undegraded":"2024-09-17T03:51:10.786793+0000","last_fullsized":"2024-09-17T03:51:10.786793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:07.138652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786834+0000","last_change":"2024-09-17T03:51:09.552398+0000","last_active":"2024-09-17T03:51:10.786834+0000","last_peered":"2024-09-17T03:51:10.786834+0000","last_clean":"2024-09-17T03:51:10.786834+0000","last_became_active":"2024-09-17T03:51:09.551972+0000","last_became_peered":"2024-09-17T03:51:09.551972+0000","last_unstale":"2024-09-17T03:51:10.786834+0000","last_undegraded":"2024-09-17T03:51:10.786834+0000","last_fullsized":"2024-09-17T03:51:10.786834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:21.780774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.533653+0000","last_change":"2024-09-17T03:51:08.528682+0000","last_active":"2024-09-17T03:51:08.533653+0000","last_peered":"2024-09-17T03:51:08.533653+0000","last_clean":"2024-09-17T03:51:08.533653+0000","last_became_active":"2024-09-17T03:51:08.527975+0000","last_became_peered":"2024-09-17T03:51:08.527975+0000","last_unstale":"2024-09-17T03:51:08.533653+0000","last_undegraded":"2024-09-17T03:51:08.533653+0000","last_fullsized":"2024-09-17T03:51:08.533653+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786848+0000","last_change":"2024-09-17T03:51:09.549934+0000","last_active":"2024-09-17T03:51:10.786848+0000","last_peered":"2024-09-17T03:51:10.786848+0000","last_clean":"2024-09-17T03:51:10.786848+0000","last_became_active":"2024-09-17T03:51:09.547467+0000","last_became_peered":"2024-09-17T03:51:09.547467+0000","last_unstale":"2024-09-17T03:51:10.786848+0000","last_undegraded":"2024-09-17T03:51:10.786848+0000","last_fullsized":"2024-09-17T03:51:10.786848+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:40.182958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541590+0000","last_change":"2024-09-17T03:51:08.538721+0000","last_active":"2024-09-17T03:51:08.541590+0000","last_peered":"2024-09-17T03:51:08.541590+0000","last_clean":"2024-09-17T03:51:08.541590+0000","last_became_active":"2024-09-17T03:51:08.537978+0000","last_became_peered":"2024-09-17T03:51:08.537978+0000","last_unstale":"2024-09-17T03:51:08.541590+0000","last_undegraded":"2024-09-17T03:51:08.541590+0000","last_fullsized":"2024-09-17T03:51:08.541590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520408+0000","last_change":"2024-09-17T03:51:08.520408+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520408+0000","last_undegraded":"2024-09-17T03:51:08.520408+0000","last_fullsized":"2024-09-17T03:51:08.520408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539559+0000","last_change":"2024-09-17T03:51:08.535323+0000","last_active":"2024-09-17T03:51:08.539559+0000","last_peered":"2024-09-17T03:51:08.539559+0000","last_clean":"2024-09-17T03:51:08.539559+0000","last_became_active":"2024-09-17T03:51:08.534999+0000","last_became_peered":"2024-09-17T03:51:08.534999+0000","last_unstale":"2024-09-17T03:51:08.539559+0000","last_undegraded":"2024-09-17T03:51:08.539559+0000","last_fullsized":"2024-09-17T03:51:08.539559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:04.103238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":23,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786053+0000","last_change":"2024-09-17T03:51:08.530197+0000","last_active":"2024-09-17T03:51:10.786053+0000","last_peered":"2024-09-17T03:51:10.786053+0000","last_clean":"2024-09-17T03:51:10.786053+0000","last_became_active":"2024-09-17T03:51:08.529723+0000","last_became_peered":"2024-09-17T03:51:08.529723+0000","last_unstale":"2024-09-17T03:51:10.786053+0000","last_undegraded":"2024-09-17T03:51:10.786053+0000","last_fullsized":"2024-09-17T03:51:10.786053+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:24.054211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786825+0000","last_change":"2024-09-17T03:51:09.537261+0000","last_active":"2024-09-17T03:51:10.786825+0000","last_peered":"2024-09-17T03:51:10.786825+0000","last_clean":"2024-09-17T03:51:10.786825+0000","last_became_active":"2024-09-17T03:51:09.536865+0000","last_became_peered":"2024-09-17T03:51:09.536865+0000","last_unstale":"2024-09-17T03:51:10.786825+0000","last_undegraded":"2024-09-17T03:51:10.786825+0000","last_fullsized":"2024-09-17T03:51:10.786825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:02.803728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786694+0000","last_change":"2024-09-17T03:51:08.535330+0000","last_active":"2024-09-17T03:51:10.786694+0000","last_peered":"2024-09-17T03:51:10.786694+0000","last_clean":"2024-09-17T03:51:10.786694+0000","last_became_active":"2024-09-17T03:51:08.535035+0000","last_became_peered":"2024-09-17T03:51:08.535035+0000","last_unstale":"2024-09-17T03:51:10.786694+0000","last_undegraded":"2024-09-17T03:51:10.786694+0000","last_fullsized":"2024-09-17T03:51:10.786694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:25.251556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787536+0000","last_change":"2024-09-17T03:51:09.559584+0000","last_active":"2024-09-17T03:51:10.787536+0000","last_peered":"2024-09-17T03:51:10.787536+0000","last_clean":"2024-09-17T03:51:10.787536+0000","last_became_active":"2024-09-17T03:51:09.559215+0000","last_became_peered":"2024-09-17T03:51:09.559215+0000","last_unstale":"2024-09-17T03:51:10.787536+0000","last_undegraded":"2024-09-17T03:51:10.787536+0000","last_fullsized":"2024-09-17T03:51:10.787536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:48.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533214+0000","last_change":"2024-09-17T03:51:08.529671+0000","last_active":"2024-09-17T03:51:10.533214+0000","last_peered":"2024-09-17T03:51:10.533214+0000","last_clean":"2024-09-17T03:51:10.533214+0000","last_became_active":"2024-09-17T03:51:08.529313+0000","last_became_peered":"2024-09-17T03:51:08.529313+0000","last_unstale":"2024-09-17T03:51:10.533214+0000","last_undegraded":"2024-09-17T03:51:10.533214+0000","last_fullsized":"2024-09-17T03:51:10.533214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787006+0000","last_change":"2024-09-17T03:51:09.531351+0000","last_active":"2024-09-17T03:51:10.787006+0000","last_peered":"2024-09-17T03:51:10.787006+0000","last_clean":"2024-09-17T03:51:10.787006+0000","last_became_active":"2024-09-17T03:51:09.530270+0000","last_became_peered":"2024-09-17T03:51:09.530270+0000","last_unstale":"2024-09-17T03:51:10.787006+0000","last_undegraded":"2024-09-17T03:51:10.787006+0000","last_fullsized":"2024-09-17T03:51:10.787006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:38:05.287191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535766+0000","last_change":"2024-09-17T03:51:08.531273+0000","last_active":"2024-09-17T03:51:08.535766+0000","last_peered":"2024-09-17T03:51:08.535766+0000","last_clean":"2024-09-17T03:51:08.535766+0000","last_became_active":"2024-09-17T03:51:08.530933+0000","last_became_peered":"2024-09-17T03:51:08.530933+0000","last_unstale":"2024-09-17T03:51:08.535766+0000","last_undegraded":"2024-09-17T03:51:08.535766+0000","last_fullsized":"2024-09-17T03:51:08.535766+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787095+0000","last_change":"2024-09-17T03:51:09.545503+0000","last_active":"2024-09-17T03:51:10.787095+0000","last_peered":"2024-09-17T03:51:10.787095+0000","last_clean":"2024-09-17T03:51:10.787095+0000","last_became_active":"2024-09-17T03:51:09.544721+0000","last_became_peered":"2024-09-17T03:51:09.544721+0000","last_unstale":"2024-09-17T03:51:10.787095+0000","last_undegraded":"2024-09-17T03:51:10.787095+0000","last_fullsized":"2024-09-17T03:51:10.787095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:10.003938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539491+0000","last_change":"2024-09-17T03:51:08.534140+0000","last_active":"2024-09-17T03:51:08.539491+0000","last_peered":"2024-09-17T03:51:08.539491+0000","last_clean":"2024-09-17T03:51:08.539491+0000","last_became_active":"2024-09-17T03:51:08.533878+0000","last_became_peered":"2024-09-17T03:51:08.533878+0000","last_unstale":"2024-09-17T03:51:08.539491+0000","last_undegraded":"2024-09-17T03:51:08.539491+0000","last_fullsized":"2024-09-17T03:51:08.539491+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:18.671953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521195+0000","last_change":"2024-09-17T03:51:08.521195+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521195+0000","last_undegraded":"2024-09-17T03:51:08.521195+0000","last_fullsized":"2024-09-17T03:51:08.521195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537539+0000","last_change":"2024-09-17T03:51:08.533378+0000","last_active":"2024-09-17T03:51:08.537539+0000","last_peered":"2024-09-17T03:51:08.537539+0000","last_clean":"2024-09-17T03:51:08.537539+0000","last_became_active":"2024-09-17T03:51:08.532682+0000","last_became_peered":"2024-09-17T03:51:08.532682+0000","last_unstale":"2024-09-17T03:51:08.537539+0000","last_undegraded":"2024-09-17T03:51:08.537539+0000","last_fullsized":"2024-09-17T03:51:08.537539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.184648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514113+0000","last_change":"2024-09-17T03:51:08.514113+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514113+0000","last_undegraded":"2024-09-17T03:51:08.514113+0000","last_fullsized":"2024-09-17T03:51:08.514113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535852+0000","last_change":"2024-09-17T03:51:08.531986+0000","last_active":"2024-09-17T03:51:08.535852+0000","last_peered":"2024-09-17T03:51:08.535852+0000","last_clean":"2024-09-17T03:51:08.535852+0000","last_became_active":"2024-09-17T03:51:08.531674+0000","last_became_peered":"2024-09-17T03:51:08.531674+0000","last_unstale":"2024-09-17T03:51:08.535852+0000","last_undegraded":"2024-09-17T03:51:08.535852+0000","last_fullsized":"2024-09-17T03:51:08.535852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785978+0000","last_change":"2024-09-17T03:51:09.538946+0000","last_active":"2024-09-17T03:51:10.785978+0000","last_peered":"2024-09-17T03:51:10.785978+0000","last_clean":"2024-09-17T03:51:10.785978+0000","last_became_active":"2024-09-17T03:51:09.538617+0000","last_became_peered":"2024-09-17T03:51:09.538617+0000","last_unstale":"2024-09-17T03:51:10.785978+0000","last_undegraded":"2024-09-17T03:51:10.785978+0000","last_fullsized":"2024-09-17T03:51:10.785978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:46.014539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786247+0000","last_change":"2024-09-17T03:51:08.526124+0000","last_active":"2024-09-17T03:51:10.786247+0000","last_peered":"2024-09-17T03:51:10.786247+0000","last_clean":"2024-09-17T03:51:10.786247+0000","last_became_active":"2024-09-17T03:51:08.525837+0000","last_became_peered":"2024-09-17T03:51:08.525837+0000","last_unstale":"2024-09-17T03:51:10.786247+0000","last_undegraded":"2024-09-17T03:51:10.786247+0000","last_fullsized":"2024-09-17T03:51:10.786247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:09.038592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787015+0000","last_change":"2024-09-17T03:51:09.556563+0000","last_active":"2024-09-17T03:51:10.787015+0000","last_peered":"2024-09-17T03:51:10.787015+0000","last_clean":"2024-09-17T03:51:10.787015+0000","last_became_active":"2024-09-17T03:51:09.555716+0000","last_became_peered":"2024-09-17T03:51:09.555716+0000","last_unstale":"2024-09-17T03:51:10.787015+0000","last_undegraded":"2024-09-17T03:51:10.787015+0000","last_fullsized":"2024-09-17T03:51:10.787015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:30.216865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541521+0000","last_change":"2024-09-17T03:51:08.535527+0000","last_active":"2024-09-17T03:51:08.541521+0000","last_peered":"2024-09-17T03:51:08.541521+0000","last_clean":"2024-09-17T03:51:08.541521+0000","last_became_active":"2024-09-17T03:51:08.535182+0000","last_became_peered":"2024-09-17T03:51:08.535182+0000","last_unstale":"2024-09-17T03:51:08.541521+0000","last_undegraded":"2024-09-17T03:51:08.541521+0000","last_fullsized":"2024-09-17T03:51:08.541521+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787369+0000","last_change":"2024-09-17T03:51:09.552798+0000","last_active":"2024-09-17T03:51:10.787369+0000","last_peered":"2024-09-17T03:51:10.787369+0000","last_clean":"2024-09-17T03:51:10.787369+0000","last_became_active":"2024-09-17T03:51:09.552477+0000","last_became_peered":"2024-09-17T03:51:09.552477+0000","last_unstale":"2024-09-17T03:51:10.787369+0000","last_undegraded":"2024-09-17T03:51:10.787369+0000","last_fullsized":"2024-09-17T03:51:10.787369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:35.740581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537417+0000","last_change":"2024-09-17T03:51:08.531555+0000","last_active":"2024-09-17T03:51:08.537417+0000","last_peered":"2024-09-17T03:51:08.537417+0000","last_clean":"2024-09-17T03:51:08.537417+0000","last_became_active":"2024-09-17T03:51:08.531230+0000","last_became_peered":"2024-09-17T03:51:08.531230+0000","last_unstale":"2024-09-17T03:51:08.537417+0000","last_undegraded":"2024-09-17T03:51:08.537417+0000","last_fullsized":"2024-09-17T03:51:08.537417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:08.023937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787381+0000","last_change":"2024-09-17T03:51:09.558633+0000","last_active":"2024-09-17T03:51:10.787381+0000","last_peered":"2024-09-17T03:51:10.787381+0000","last_clean":"2024-09-17T03:51:10.787381+0000","last_became_active":"2024-09-17T03:51:09.558301+0000","last_became_peered":"2024-09-17T03:51:09.558301+0000","last_unstale":"2024-09-17T03:51:10.787381+0000","last_undegraded":"2024-09-17T03:51:10.787381+0000","last_fullsized":"2024-09-17T03:51:10.787381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:53.303687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.543358+0000","last_change":"2024-09-17T03:51:08.542303+0000","last_active":"2024-09-17T03:51:08.543358+0000","last_peered":"2024-09-17T03:51:08.543358+0000","last_clean":"2024-09-17T03:51:08.543358+0000","last_became_active":"2024-09-17T03:51:08.541884+0000","last_became_peered":"2024-09-17T03:51:08.541884+0000","last_unstale":"2024-09-17T03:51:08.543358+0000","last_undegraded":"2024-09-17T03:51:08.543358+0000","last_fullsized":"2024-09-17T03:51:08.543358+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.838696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536023+0000","last_change":"2024-09-17T03:51:08.533982+0000","last_active":"2024-09-17T03:51:08.536023+0000","last_peered":"2024-09-17T03:51:08.536023+0000","last_clean":"2024-09-17T03:51:08.536023+0000","last_became_active":"2024-09-17T03:51:08.529828+0000","last_became_peered":"2024-09-17T03:51:08.529828+0000","last_unstale":"2024-09-17T03:51:08.536023+0000","last_undegraded":"2024-09-17T03:51:08.536023+0000","last_fullsized":"2024-09-17T03:51:08.536023+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:18.636858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520810+0000","last_change":"2024-09-17T03:51:08.520810+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520810+0000","last_undegraded":"2024-09-17T03:51:08.520810+0000","last_fullsized":"2024-09-17T03:51:08.520810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542733+0000","last_change":"2024-09-17T03:51:08.540515+0000","last_active":"2024-09-17T03:51:08.542733+0000","last_peered":"2024-09-17T03:51:08.542733+0000","last_clean":"2024-09-17T03:51:08.542733+0000","last_became_active":"2024-09-17T03:51:08.540197+0000","last_became_peered":"2024-09-17T03:51:08.540197+0000","last_unstale":"2024-09-17T03:51:08.542733+0000","last_undegraded":"2024-09-17T03:51:08.542733+0000","last_fullsized":"2024-09-17T03:51:08.542733+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520049+0000","last_change":"2024-09-17T03:51:08.520049+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520049+0000","last_undegraded":"2024-09-17T03:51:08.520049+0000","last_fullsized":"2024-09-17T03:51:08.520049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534533+0000","last_change":"2024-09-17T03:51:08.538899+0000","last_active":"2024-09-17T03:51:10.534533+0000","last_peered":"2024-09-17T03:51:10.534533+0000","last_clean":"2024-09-17T03:51:10.534533+0000","last_became_active":"2024-09-17T03:51:08.538338+0000","last_became_peered":"2024-09-17T03:51:08.538338+0000","last_unstale":"2024-09-17T03:51:10.534533+0000","last_undegraded":"2024-09-17T03:51:10.534533+0000","last_fullsized":"2024-09-17T03:51:10.534533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:56.151879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520013+0000","last_change":"2024-09-17T03:51:08.520013+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520013+0000","last_undegraded":"2024-09-17T03:51:08.520013+0000","last_fullsized":"2024-09-17T03:51:08.520013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536701+0000","last_change":"2024-09-17T03:51:08.535001+0000","last_active":"2024-09-17T03:51:08.536701+0000","last_peered":"2024-09-17T03:51:08.536701+0000","last_clean":"2024-09-17T03:51:08.536701+0000","last_became_active":"2024-09-17T03:51:08.531914+0000","last_became_peered":"2024-09-17T03:51:08.531914+0000","last_unstale":"2024-09-17T03:51:08.536701+0000","last_undegraded":"2024-09-17T03:51:08.536701+0000","last_fullsized":"2024-09-17T03:51:08.536701+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:10.120644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532927+0000","last_change":"2024-09-17T03:51:09.540447+0000","last_active":"2024-09-17T03:51:10.532927+0000","last_peered":"2024-09-17T03:51:10.532927+0000","last_clean":"2024-09-17T03:51:10.532927+0000","last_became_active":"2024-09-17T03:51:09.540161+0000","last_became_peered":"2024-09-17T03:51:09.540161+0000","last_unstale":"2024-09-17T03:51:10.532927+0000","last_undegraded":"2024-09-17T03:51:10.532927+0000","last_fullsized":"2024-09-17T03:51:10.532927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:04.256313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786105+0000","last_change":"2024-09-17T03:51:08.525657+0000","last_active":"2024-09-17T03:51:10.786105+0000","last_peered":"2024-09-17T03:51:10.786105+0000","last_clean":"2024-09-17T03:51:10.786105+0000","last_became_active":"2024-09-17T03:51:08.525271+0000","last_became_peered":"2024-09-17T03:51:08.525271+0000","last_unstale":"2024-09-17T03:51:10.786105+0000","last_undegraded":"2024-09-17T03:51:10.786105+0000","last_fullsized":"2024-09-17T03:51:10.786105+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:55.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787764+0000","last_change":"2024-09-17T03:51:09.544994+0000","last_active":"2024-09-17T03:51:10.787764+0000","last_peered":"2024-09-17T03:51:10.787764+0000","last_clean":"2024-09-17T03:51:10.787764+0000","last_became_active":"2024-09-17T03:51:09.544748+0000","last_became_peered":"2024-09-17T03:51:09.544748+0000","last_unstale":"2024-09-17T03:51:10.787764+0000","last_undegraded":"2024-09-17T03:51:10.787764+0000","last_fullsized":"2024-09-17T03:51:10.787764+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:14.819738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535595+0000","last_change":"2024-09-17T03:51:08.529407+0000","last_active":"2024-09-17T03:51:08.535595+0000","last_peered":"2024-09-17T03:51:08.535595+0000","last_clean":"2024-09-17T03:51:08.535595+0000","last_became_active":"2024-09-17T03:51:08.529046+0000","last_became_peered":"2024-09-17T03:51:08.529046+0000","last_unstale":"2024-09-17T03:51:08.535595+0000","last_undegraded":"2024-09-17T03:51:08.535595+0000","last_fullsized":"2024-09-17T03:51:08.535595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":148,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.524741+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:08.524741+0000","last_peered":"2024-09-17T03:51:08.524741+0000","last_clean":"2024-09-17T03:51:08.524741+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:08.524741+0000","last_undegraded":"2024-09-17T03:51:08.524741+0000","last_fullsized":"2024-09-17T03:51:08.524741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1089536,"data_stored":786166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509466,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":210485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":536576,"data_stored":210507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":209545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":799275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:14.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 -- 172.21.15.129:0/412675945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a0003a30 msgr2=0x7f72a0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 --2- 172.21.15.129:0/412675945 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72a0003a30 0x7f72a0003ea0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f72ac001710 tx=0x7f72a0005830 comp rx=0 tx=0).stop 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bb400640 1 -- 172.21.15.129:0/412675945 reap_dead start 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 -- 172.21.15.129:0/412675945 shutdown_connections 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 -- 172.21.15.129:0/412675945 >> 172.21.15.129:0/412675945 conn(0x7f72bc09d620 msgr2=0x7f72bc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 -- 172.21.15.129:0/412675945 shutdown_connections 2024-09-17T03:51:14.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.891+0000 7f72bbe00640 1 -- 172.21.15.129:0/412675945 wait complete. 2024-09-17T03:51:14.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.894+0000 7f72bbe00640 1 Processor -- start 2024-09-17T03:51:14.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.894+0000 7f72bbe00640 1 -- start start 2024-09-17T03:51:14.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 0x7f72bc1f85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 0x7f72bc0d2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc0b7ce0 con 0x7f72bc14ff40 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f72bc0b7b60 con 0x7f72bc0d2810 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72bbe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f72bc0b7e60 con 0x7f72a0003a30 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72ba000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72ba000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39402/0 (socket says 172.21.15.129:39402) 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.895+0000 7f72ba000640 1 -- 172.21.15.129:0/1204728933 learned_addr learned my addr 172.21.15.129:0/1204728933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72bb400640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 0x7f72bc0d2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72baa00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 0x7f72bc1f85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 -- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 msgr2=0x7f72bc1f85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 0x7f72bc1f85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 -- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 msgr2=0x7f72bc0d2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:14.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 0x7f72bc0d2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 -- 172.21.15.129:0/1204728933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72bc0d3160 con 0x7f72bc14ff40 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72baa00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 0x7f72bc1f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72bb400640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 0x7f72bc0d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72ba000640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006680 tx=0x7f72a80066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011700 con 0x7f72bc14ff40 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72a8016020 con 0x7f72bc14ff40 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8011b60 con 0x7f72bc14ff40 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72bc0b3b90 con 0x7f72bc14ff40 2024-09-17T03:51:14.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.896+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72bc0b3e70 con 0x7f72bc14ff40 2024-09-17T03:51:14.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.897+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72bc0b4c10 con 0x7f72bc14ff40 2024-09-17T03:51:14.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.899+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f72a8011d00 con 0x7f72bc14ff40 2024-09-17T03:51:14.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.900+0000 7f72b3400640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 0x7f729c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:14.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.900+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f72a809b960 con 0x7f72bc14ff40 2024-09-17T03:51:14.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.900+0000 7f72baa00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 0x7f729c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:14.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.901+0000 7f72baa00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 0x7f729c07b0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0016e0 tx=0x7f72ac0011e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:14.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:14.904+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f72a8020020 con 0x7f72bc14ff40 2024-09-17T03:51:15.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.083+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72bc0a20b0 con 0x7f729c078bc0 2024-09-17T03:51:15.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.088+0000 7f72b3400640 1 -- 172.21.15.129:0/1204728933 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249961 (secure 0 0 0) 0x7f72bc0a20b0 con 0x7f729c078bc0 2024-09-17T03:51:15.090 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:15.091 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 msgr2=0x7f729c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 0x7f729c07b0d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0016e0 tx=0x7f72ac0011e0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 msgr2=0x7f72bc1f9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f72a8006680 tx=0x7f72a80066b0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 shutdown_connections 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f729c078bc0 0x7f729c07b0d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72bc0d2810 0x7f72bc0d2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72bc14ff40 0x7f72bc1f9700 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 --2- 172.21.15.129:0/1204728933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72a0003a30 0x7f72bc1f85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 >> 172.21.15.129:0/1204728933 conn(0x7f72bc09d620 msgr2=0x7f72bc068e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:15.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 shutdown_connections 2024-09-17T03:51:15.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.092+0000 7f72bbe00640 1 -- 172.21.15.129:0/1204728933 wait complete. 2024-09-17T03:51:15.112 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-17T03:51:14.318893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254692,"kb_used_data":5168,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720028,"statfs":{"total":773094113280,"available":772833308672,"internally_reserved":0,"allocated":5292032,"data_stored":2803372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518100+0000","last_change":"2024-09-17T03:51:08.518100+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518100+0000","last_undegraded":"2024-09-17T03:51:08.518100+0000","last_fullsized":"2024-09-17T03:51:08.518100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521176+0000","last_change":"2024-09-17T03:51:08.521176+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521176+0000","last_undegraded":"2024-09-17T03:51:08.521176+0000","last_fullsized":"2024-09-17T03:51:08.521176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532794+0000","last_change":"2024-09-17T03:51:09.544780+0000","last_active":"2024-09-17T03:51:10.532794+0000","last_peered":"2024-09-17T03:51:10.532794+0000","last_clean":"2024-09-17T03:51:10.532794+0000","last_became_active":"2024-09-17T03:51:09.544462+0000","last_became_peered":"2024-09-17T03:51:09.544462+0000","last_unstale":"2024-09-17T03:51:10.532794+0000","last_undegraded":"2024-09-17T03:51:10.532794+0000","last_fullsized":"2024-09-17T03:51:10.532794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:20.077167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787357+0000","last_change":"2024-09-17T03:51:09.553310+0000","last_active":"2024-09-17T03:51:10.787357+0000","last_peered":"2024-09-17T03:51:10.787357+0000","last_clean":"2024-09-17T03:51:10.787357+0000","last_became_active":"2024-09-17T03:51:09.552339+0000","last_became_peered":"2024-09-17T03:51:09.552339+0000","last_unstale":"2024-09-17T03:51:10.787357+0000","last_undegraded":"2024-09-17T03:51:10.787357+0000","last_fullsized":"2024-09-17T03:51:10.787357+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:59.902480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521899+0000","last_change":"2024-09-17T03:51:08.521899+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521899+0000","last_undegraded":"2024-09-17T03:51:08.521899+0000","last_fullsized":"2024-09-17T03:51:08.521899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521676+0000","last_change":"2024-09-17T03:51:08.521676+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521676+0000","last_undegraded":"2024-09-17T03:51:08.521676+0000","last_fullsized":"2024-09-17T03:51:08.521676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518344+0000","last_change":"2024-09-17T03:51:08.518344+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518344+0000","last_undegraded":"2024-09-17T03:51:08.518344+0000","last_fullsized":"2024-09-17T03:51:08.518344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785916+0000","last_change":"2024-09-17T03:51:09.553571+0000","last_active":"2024-09-17T03:51:10.785916+0000","last_peered":"2024-09-17T03:51:10.785916+0000","last_clean":"2024-09-17T03:51:10.785916+0000","last_became_active":"2024-09-17T03:51:09.553226+0000","last_became_peered":"2024-09-17T03:51:09.553226+0000","last_unstale":"2024-09-17T03:51:10.785916+0000","last_undegraded":"2024-09-17T03:51:10.785916+0000","last_fullsized":"2024-09-17T03:51:10.785916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:13.896356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787450+0000","last_change":"2024-09-17T03:51:09.558849+0000","last_active":"2024-09-17T03:51:10.787450+0000","last_peered":"2024-09-17T03:51:10.787450+0000","last_clean":"2024-09-17T03:51:10.787450+0000","last_became_active":"2024-09-17T03:51:09.558596+0000","last_became_peered":"2024-09-17T03:51:09.558596+0000","last_unstale":"2024-09-17T03:51:10.787450+0000","last_undegraded":"2024-09-17T03:51:10.787450+0000","last_fullsized":"2024-09-17T03:51:10.787450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:19.302750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787683+0000","last_change":"2024-09-17T03:51:09.552932+0000","last_active":"2024-09-17T03:51:10.787683+0000","last_peered":"2024-09-17T03:51:10.787683+0000","last_clean":"2024-09-17T03:51:10.787683+0000","last_became_active":"2024-09-17T03:51:09.552660+0000","last_became_peered":"2024-09-17T03:51:09.552660+0000","last_unstale":"2024-09-17T03:51:10.787683+0000","last_undegraded":"2024-09-17T03:51:10.787683+0000","last_fullsized":"2024-09-17T03:51:10.787683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:45.040579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787563+0000","last_change":"2024-09-17T03:51:09.556091+0000","last_active":"2024-09-17T03:51:10.787563+0000","last_peered":"2024-09-17T03:51:10.787563+0000","last_clean":"2024-09-17T03:51:10.787563+0000","last_became_active":"2024-09-17T03:51:09.555788+0000","last_became_peered":"2024-09-17T03:51:09.555788+0000","last_unstale":"2024-09-17T03:51:10.787563+0000","last_undegraded":"2024-09-17T03:51:10.787563+0000","last_fullsized":"2024-09-17T03:51:10.787563+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:35.856529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523028+0000","last_change":"2024-09-17T03:51:08.523028+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523028+0000","last_undegraded":"2024-09-17T03:51:08.523028+0000","last_fullsized":"2024-09-17T03:51:08.523028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786586+0000","last_change":"2024-09-17T03:51:09.549923+0000","last_active":"2024-09-17T03:51:10.786586+0000","last_peered":"2024-09-17T03:51:10.786586+0000","last_clean":"2024-09-17T03:51:10.786586+0000","last_became_active":"2024-09-17T03:51:09.549427+0000","last_became_peered":"2024-09-17T03:51:09.549427+0000","last_unstale":"2024-09-17T03:51:10.786586+0000","last_undegraded":"2024-09-17T03:51:10.786586+0000","last_fullsized":"2024-09-17T03:51:10.786586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:12.588911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786097+0000","last_change":"2024-09-17T03:51:09.557049+0000","last_active":"2024-09-17T03:51:10.786097+0000","last_peered":"2024-09-17T03:51:10.786097+0000","last_clean":"2024-09-17T03:51:10.786097+0000","last_became_active":"2024-09-17T03:51:09.556732+0000","last_became_peered":"2024-09-17T03:51:09.556732+0000","last_unstale":"2024-09-17T03:51:10.786097+0000","last_undegraded":"2024-09-17T03:51:10.786097+0000","last_fullsized":"2024-09-17T03:51:10.786097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:51.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520415+0000","last_change":"2024-09-17T03:51:08.520415+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520415+0000","last_undegraded":"2024-09-17T03:51:08.520415+0000","last_fullsized":"2024-09-17T03:51:08.520415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786655+0000","last_change":"2024-09-17T03:51:09.552281+0000","last_active":"2024-09-17T03:51:10.786655+0000","last_peered":"2024-09-17T03:51:10.786655+0000","last_clean":"2024-09-17T03:51:10.786655+0000","last_became_active":"2024-09-17T03:51:09.551898+0000","last_became_peered":"2024-09-17T03:51:09.551898+0000","last_unstale":"2024-09-17T03:51:10.786655+0000","last_undegraded":"2024-09-17T03:51:10.786655+0000","last_fullsized":"2024-09-17T03:51:10.786655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:19.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533660+0000","last_change":"2024-09-17T03:51:09.533608+0000","last_active":"2024-09-17T03:51:10.533660+0000","last_peered":"2024-09-17T03:51:10.533660+0000","last_clean":"2024-09-17T03:51:10.533660+0000","last_became_active":"2024-09-17T03:51:09.533263+0000","last_became_peered":"2024-09-17T03:51:09.533263+0000","last_unstale":"2024-09-17T03:51:10.533660+0000","last_undegraded":"2024-09-17T03:51:10.533660+0000","last_fullsized":"2024-09-17T03:51:10.533660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:18.712043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532553+0000","last_change":"2024-09-17T03:51:09.549893+0000","last_active":"2024-09-17T03:51:10.532553+0000","last_peered":"2024-09-17T03:51:10.532553+0000","last_clean":"2024-09-17T03:51:10.532553+0000","last_became_active":"2024-09-17T03:51:09.548155+0000","last_became_peered":"2024-09-17T03:51:09.548155+0000","last_unstale":"2024-09-17T03:51:10.532553+0000","last_undegraded":"2024-09-17T03:51:10.532553+0000","last_fullsized":"2024-09-17T03:51:10.532553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:06.020247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787285+0000","last_change":"2024-09-17T03:51:09.545828+0000","last_active":"2024-09-17T03:51:10.787285+0000","last_peered":"2024-09-17T03:51:10.787285+0000","last_clean":"2024-09-17T03:51:10.787285+0000","last_became_active":"2024-09-17T03:51:09.545207+0000","last_became_peered":"2024-09-17T03:51:09.545207+0000","last_unstale":"2024-09-17T03:51:10.787285+0000","last_undegraded":"2024-09-17T03:51:10.787285+0000","last_fullsized":"2024-09-17T03:51:10.787285+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:11.346092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786753+0000","last_change":"2024-09-17T03:51:09.550468+0000","last_active":"2024-09-17T03:51:10.786753+0000","last_peered":"2024-09-17T03:51:10.786753+0000","last_clean":"2024-09-17T03:51:10.786753+0000","last_became_active":"2024-09-17T03:51:09.550210+0000","last_became_peered":"2024-09-17T03:51:09.550210+0000","last_unstale":"2024-09-17T03:51:10.786753+0000","last_undegraded":"2024-09-17T03:51:10.786753+0000","last_fullsized":"2024-09-17T03:51:10.786753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:55.396574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520494+0000","last_change":"2024-09-17T03:51:08.520494+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520494+0000","last_undegraded":"2024-09-17T03:51:08.520494+0000","last_fullsized":"2024-09-17T03:51:08.520494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786228+0000","last_change":"2024-09-17T03:51:09.551749+0000","last_active":"2024-09-17T03:51:10.786228+0000","last_peered":"2024-09-17T03:51:10.786228+0000","last_clean":"2024-09-17T03:51:10.786228+0000","last_became_active":"2024-09-17T03:51:09.551425+0000","last_became_peered":"2024-09-17T03:51:09.551425+0000","last_unstale":"2024-09-17T03:51:10.786228+0000","last_undegraded":"2024-09-17T03:51:10.786228+0000","last_fullsized":"2024-09-17T03:51:10.786228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:31.877445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786776+0000","last_change":"2024-09-17T03:51:09.537760+0000","last_active":"2024-09-17T03:51:10.786776+0000","last_peered":"2024-09-17T03:51:10.786776+0000","last_clean":"2024-09-17T03:51:10.786776+0000","last_became_active":"2024-09-17T03:51:09.537430+0000","last_became_peered":"2024-09-17T03:51:09.537430+0000","last_unstale":"2024-09-17T03:51:10.786776+0000","last_undegraded":"2024-09-17T03:51:10.786776+0000","last_fullsized":"2024-09-17T03:51:10.786776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:40.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786706+0000","last_change":"2024-09-17T03:51:09.535683+0000","last_active":"2024-09-17T03:51:10.786706+0000","last_peered":"2024-09-17T03:51:10.786706+0000","last_clean":"2024-09-17T03:51:10.786706+0000","last_became_active":"2024-09-17T03:51:09.535368+0000","last_became_peered":"2024-09-17T03:51:09.535368+0000","last_unstale":"2024-09-17T03:51:10.786706+0000","last_undegraded":"2024-09-17T03:51:10.786706+0000","last_fullsized":"2024-09-17T03:51:10.786706+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:28.792675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787126+0000","last_change":"2024-09-17T03:51:09.531819+0000","last_active":"2024-09-17T03:51:10.787126+0000","last_peered":"2024-09-17T03:51:10.787126+0000","last_clean":"2024-09-17T03:51:10.787126+0000","last_became_active":"2024-09-17T03:51:09.531504+0000","last_became_peered":"2024-09-17T03:51:09.531504+0000","last_unstale":"2024-09-17T03:51:10.787126+0000","last_undegraded":"2024-09-17T03:51:10.787126+0000","last_fullsized":"2024-09-17T03:51:10.787126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:37.424982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786610+0000","last_change":"2024-09-17T03:51:09.561216+0000","last_active":"2024-09-17T03:51:10.786610+0000","last_peered":"2024-09-17T03:51:10.786610+0000","last_clean":"2024-09-17T03:51:10.786610+0000","last_became_active":"2024-09-17T03:51:09.560930+0000","last_became_peered":"2024-09-17T03:51:09.560930+0000","last_unstale":"2024-09-17T03:51:10.786610+0000","last_undegraded":"2024-09-17T03:51:10.786610+0000","last_fullsized":"2024-09-17T03:51:10.786610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:24.329326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533087+0000","last_change":"2024-09-17T03:51:09.532459+0000","last_active":"2024-09-17T03:51:10.533087+0000","last_peered":"2024-09-17T03:51:10.533087+0000","last_clean":"2024-09-17T03:51:10.533087+0000","last_became_active":"2024-09-17T03:51:09.532056+0000","last_became_peered":"2024-09-17T03:51:09.532056+0000","last_unstale":"2024-09-17T03:51:10.533087+0000","last_undegraded":"2024-09-17T03:51:10.533087+0000","last_fullsized":"2024-09-17T03:51:10.533087+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:07.247043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.522524+0000","last_change":"2024-09-17T03:51:08.522524+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.522524+0000","last_undegraded":"2024-09-17T03:51:08.522524+0000","last_fullsized":"2024-09-17T03:51:08.522524+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787120+0000","last_change":"2024-09-17T03:51:09.531111+0000","last_active":"2024-09-17T03:51:10.787120+0000","last_peered":"2024-09-17T03:51:10.787120+0000","last_clean":"2024-09-17T03:51:10.787120+0000","last_became_active":"2024-09-17T03:51:09.530769+0000","last_became_peered":"2024-09-17T03:51:09.530769+0000","last_unstale":"2024-09-17T03:51:10.787120+0000","last_undegraded":"2024-09-17T03:51:10.787120+0000","last_fullsized":"2024-09-17T03:51:10.787120+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:05.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.540408+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.540084+0000","last_became_peered":"2024-09-17T03:51:09.540084+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:09.874665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.519903+0000","last_change":"2024-09-17T03:51:08.519903+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.519903+0000","last_undegraded":"2024-09-17T03:51:08.519903+0000","last_fullsized":"2024-09-17T03:51:08.519903+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786259+0000","last_change":"2024-09-17T03:51:09.532104+0000","last_active":"2024-09-17T03:51:10.786259+0000","last_peered":"2024-09-17T03:51:10.786259+0000","last_clean":"2024-09-17T03:51:10.786259+0000","last_became_active":"2024-09-17T03:51:09.531801+0000","last_became_peered":"2024-09-17T03:51:09.531801+0000","last_unstale":"2024-09-17T03:51:10.786259+0000","last_undegraded":"2024-09-17T03:51:10.786259+0000","last_fullsized":"2024-09-17T03:51:10.786259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:40.757442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514124+0000","last_change":"2024-09-17T03:51:08.514124+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514124+0000","last_undegraded":"2024-09-17T03:51:08.514124+0000","last_fullsized":"2024-09-17T03:51:08.514124+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539585+0000","last_change":"2024-09-17T03:51:08.536130+0000","last_active":"2024-09-17T03:51:08.539585+0000","last_peered":"2024-09-17T03:51:08.539585+0000","last_clean":"2024-09-17T03:51:08.539585+0000","last_became_active":"2024-09-17T03:51:08.535845+0000","last_became_peered":"2024-09-17T03:51:08.535845+0000","last_unstale":"2024-09-17T03:51:08.539585+0000","last_undegraded":"2024-09-17T03:51:08.539585+0000","last_fullsized":"2024-09-17T03:51:08.539585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.579356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787189+0000","last_change":"2024-09-17T03:51:09.530989+0000","last_active":"2024-09-17T03:51:10.787189+0000","last_peered":"2024-09-17T03:51:10.787189+0000","last_clean":"2024-09-17T03:51:10.787189+0000","last_became_active":"2024-09-17T03:51:09.529800+0000","last_became_peered":"2024-09-17T03:51:09.529800+0000","last_unstale":"2024-09-17T03:51:10.787189+0000","last_undegraded":"2024-09-17T03:51:10.787189+0000","last_fullsized":"2024-09-17T03:51:10.787189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:22.141601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533204+0000","last_change":"2024-09-17T03:51:08.537833+0000","last_active":"2024-09-17T03:51:10.533204+0000","last_peered":"2024-09-17T03:51:10.533204+0000","last_clean":"2024-09-17T03:51:10.533204+0000","last_became_active":"2024-09-17T03:51:08.536906+0000","last_became_peered":"2024-09-17T03:51:08.536906+0000","last_unstale":"2024-09-17T03:51:10.533204+0000","last_undegraded":"2024-09-17T03:51:10.533204+0000","last_fullsized":"2024-09-17T03:51:10.533204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521613+0000","last_change":"2024-09-17T03:51:08.521613+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521613+0000","last_undegraded":"2024-09-17T03:51:08.521613+0000","last_fullsized":"2024-09-17T03:51:08.521613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535582+0000","last_change":"2024-09-17T03:51:08.532550+0000","last_active":"2024-09-17T03:51:08.535582+0000","last_peered":"2024-09-17T03:51:08.535582+0000","last_clean":"2024-09-17T03:51:08.535582+0000","last_became_active":"2024-09-17T03:51:08.532173+0000","last_became_peered":"2024-09-17T03:51:08.532173+0000","last_unstale":"2024-09-17T03:51:08.535582+0000","last_undegraded":"2024-09-17T03:51:08.535582+0000","last_fullsized":"2024-09-17T03:51:08.535582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:22.458883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514188+0000","last_change":"2024-09-17T03:51:08.514188+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514188+0000","last_undegraded":"2024-09-17T03:51:08.514188+0000","last_fullsized":"2024-09-17T03:51:08.514188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541632+0000","last_change":"2024-09-17T03:51:08.538223+0000","last_active":"2024-09-17T03:51:08.541632+0000","last_peered":"2024-09-17T03:51:08.541632+0000","last_clean":"2024-09-17T03:51:08.541632+0000","last_became_active":"2024-09-17T03:51:08.536819+0000","last_became_peered":"2024-09-17T03:51:08.536819+0000","last_unstale":"2024-09-17T03:51:08.541632+0000","last_undegraded":"2024-09-17T03:51:08.541632+0000","last_fullsized":"2024-09-17T03:51:08.541632+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.533568+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.533027+0000","last_became_peered":"2024-09-17T03:51:09.533027+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:39.579158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537328+0000","last_change":"2024-09-17T03:51:08.533400+0000","last_active":"2024-09-17T03:51:08.537328+0000","last_peered":"2024-09-17T03:51:08.537328+0000","last_clean":"2024-09-17T03:51:08.537328+0000","last_became_active":"2024-09-17T03:51:08.532958+0000","last_became_peered":"2024-09-17T03:51:08.532958+0000","last_unstale":"2024-09-17T03:51:08.537328+0000","last_undegraded":"2024-09-17T03:51:08.537328+0000","last_fullsized":"2024-09-17T03:51:08.537328+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:41.046821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786620+0000","last_change":"2024-09-17T03:51:09.543641+0000","last_active":"2024-09-17T03:51:10.786620+0000","last_peered":"2024-09-17T03:51:10.786620+0000","last_clean":"2024-09-17T03:51:10.786620+0000","last_became_active":"2024-09-17T03:51:09.543299+0000","last_became_peered":"2024-09-17T03:51:09.543299+0000","last_unstale":"2024-09-17T03:51:10.786620+0000","last_undegraded":"2024-09-17T03:51:10.786620+0000","last_fullsized":"2024-09-17T03:51:10.786620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:34.478969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534447+0000","last_change":"2024-09-17T03:51:08.535613+0000","last_active":"2024-09-17T03:51:10.534447+0000","last_peered":"2024-09-17T03:51:10.534447+0000","last_clean":"2024-09-17T03:51:10.534447+0000","last_became_active":"2024-09-17T03:51:08.535214+0000","last_became_peered":"2024-09-17T03:51:08.535214+0000","last_unstale":"2024-09-17T03:51:10.534447+0000","last_undegraded":"2024-09-17T03:51:10.534447+0000","last_fullsized":"2024-09-17T03:51:10.534447+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.931699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785982+0000","last_change":"2024-09-17T03:51:09.552322+0000","last_active":"2024-09-17T03:51:10.785982+0000","last_peered":"2024-09-17T03:51:10.785982+0000","last_clean":"2024-09-17T03:51:10.785982+0000","last_became_active":"2024-09-17T03:51:09.551914+0000","last_became_peered":"2024-09-17T03:51:09.551914+0000","last_unstale":"2024-09-17T03:51:10.785982+0000","last_undegraded":"2024-09-17T03:51:10.785982+0000","last_fullsized":"2024-09-17T03:51:10.785982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:38.111592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536383+0000","last_change":"2024-09-17T03:51:08.532902+0000","last_active":"2024-09-17T03:51:08.536383+0000","last_peered":"2024-09-17T03:51:08.536383+0000","last_clean":"2024-09-17T03:51:08.536383+0000","last_became_active":"2024-09-17T03:51:08.532401+0000","last_became_peered":"2024-09-17T03:51:08.532401+0000","last_unstale":"2024-09-17T03:51:08.536383+0000","last_undegraded":"2024-09-17T03:51:08.536383+0000","last_fullsized":"2024-09-17T03:51:08.536383+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786515+0000","last_change":"2024-09-17T03:51:09.541584+0000","last_active":"2024-09-17T03:51:10.786515+0000","last_peered":"2024-09-17T03:51:10.786515+0000","last_clean":"2024-09-17T03:51:10.786515+0000","last_became_active":"2024-09-17T03:51:09.541219+0000","last_became_peered":"2024-09-17T03:51:09.541219+0000","last_unstale":"2024-09-17T03:51:10.786515+0000","last_undegraded":"2024-09-17T03:51:10.786515+0000","last_fullsized":"2024-09-17T03:51:10.786515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:57.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785991+0000","last_change":"2024-09-17T03:51:08.522934+0000","last_active":"2024-09-17T03:51:10.785991+0000","last_peered":"2024-09-17T03:51:10.785991+0000","last_clean":"2024-09-17T03:51:10.785991+0000","last_became_active":"2024-09-17T03:51:08.522647+0000","last_became_peered":"2024-09-17T03:51:08.522647+0000","last_unstale":"2024-09-17T03:51:10.785991+0000","last_undegraded":"2024-09-17T03:51:10.785991+0000","last_fullsized":"2024-09-17T03:51:10.785991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:27.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532822+0000","last_change":"2024-09-17T03:51:09.550191+0000","last_active":"2024-09-17T03:51:10.532822+0000","last_peered":"2024-09-17T03:51:10.532822+0000","last_clean":"2024-09-17T03:51:10.532822+0000","last_became_active":"2024-09-17T03:51:09.548578+0000","last_became_peered":"2024-09-17T03:51:09.548578+0000","last_unstale":"2024-09-17T03:51:10.532822+0000","last_undegraded":"2024-09-17T03:51:10.532822+0000","last_fullsized":"2024-09-17T03:51:10.532822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:01.627905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533705+0000","last_change":"2024-09-17T03:51:08.535319+0000","last_active":"2024-09-17T03:51:10.533705+0000","last_peered":"2024-09-17T03:51:10.533705+0000","last_clean":"2024-09-17T03:51:10.533705+0000","last_became_active":"2024-09-17T03:51:08.534878+0000","last_became_peered":"2024-09-17T03:51:08.534878+0000","last_unstale":"2024-09-17T03:51:10.533705+0000","last_undegraded":"2024-09-17T03:51:10.533705+0000","last_fullsized":"2024-09-17T03:51:10.533705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517437+0000","last_change":"2024-09-17T03:51:08.517437+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517437+0000","last_undegraded":"2024-09-17T03:51:08.517437+0000","last_fullsized":"2024-09-17T03:51:08.517437+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787744+0000","last_change":"2024-09-17T03:51:08.538705+0000","last_active":"2024-09-17T03:51:10.787744+0000","last_peered":"2024-09-17T03:51:10.787744+0000","last_clean":"2024-09-17T03:51:10.787744+0000","last_became_active":"2024-09-17T03:51:08.535730+0000","last_became_peered":"2024-09-17T03:51:08.535730+0000","last_unstale":"2024-09-17T03:51:10.787744+0000","last_undegraded":"2024-09-17T03:51:10.787744+0000","last_fullsized":"2024-09-17T03:51:10.787744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:47.222685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518389+0000","last_change":"2024-09-17T03:51:08.518389+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518389+0000","last_undegraded":"2024-09-17T03:51:08.518389+0000","last_fullsized":"2024-09-17T03:51:08.518389+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542851+0000","last_change":"2024-09-17T03:51:08.541002+0000","last_active":"2024-09-17T03:51:08.542851+0000","last_peered":"2024-09-17T03:51:08.542851+0000","last_clean":"2024-09-17T03:51:08.542851+0000","last_became_active":"2024-09-17T03:51:08.540686+0000","last_became_peered":"2024-09-17T03:51:08.540686+0000","last_unstale":"2024-09-17T03:51:08.542851+0000","last_undegraded":"2024-09-17T03:51:08.542851+0000","last_fullsized":"2024-09-17T03:51:08.542851+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:20.604244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518861+0000","last_change":"2024-09-17T03:51:08.518861+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518861+0000","last_undegraded":"2024-09-17T03:51:08.518861+0000","last_fullsized":"2024-09-17T03:51:08.518861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":21,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786793+0000","last_change":"2024-09-17T03:51:08.538266+0000","last_active":"2024-09-17T03:51:10.786793+0000","last_peered":"2024-09-17T03:51:10.786793+0000","last_clean":"2024-09-17T03:51:10.786793+0000","last_became_active":"2024-09-17T03:51:08.537932+0000","last_became_peered":"2024-09-17T03:51:08.537932+0000","last_unstale":"2024-09-17T03:51:10.786793+0000","last_undegraded":"2024-09-17T03:51:10.786793+0000","last_fullsized":"2024-09-17T03:51:10.786793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:07.138652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786834+0000","last_change":"2024-09-17T03:51:09.552398+0000","last_active":"2024-09-17T03:51:10.786834+0000","last_peered":"2024-09-17T03:51:10.786834+0000","last_clean":"2024-09-17T03:51:10.786834+0000","last_became_active":"2024-09-17T03:51:09.551972+0000","last_became_peered":"2024-09-17T03:51:09.551972+0000","last_unstale":"2024-09-17T03:51:10.786834+0000","last_undegraded":"2024-09-17T03:51:10.786834+0000","last_fullsized":"2024-09-17T03:51:10.786834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:21.780774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.533653+0000","last_change":"2024-09-17T03:51:08.528682+0000","last_active":"2024-09-17T03:51:08.533653+0000","last_peered":"2024-09-17T03:51:08.533653+0000","last_clean":"2024-09-17T03:51:08.533653+0000","last_became_active":"2024-09-17T03:51:08.527975+0000","last_became_peered":"2024-09-17T03:51:08.527975+0000","last_unstale":"2024-09-17T03:51:08.533653+0000","last_undegraded":"2024-09-17T03:51:08.533653+0000","last_fullsized":"2024-09-17T03:51:08.533653+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786848+0000","last_change":"2024-09-17T03:51:09.549934+0000","last_active":"2024-09-17T03:51:10.786848+0000","last_peered":"2024-09-17T03:51:10.786848+0000","last_clean":"2024-09-17T03:51:10.786848+0000","last_became_active":"2024-09-17T03:51:09.547467+0000","last_became_peered":"2024-09-17T03:51:09.547467+0000","last_unstale":"2024-09-17T03:51:10.786848+0000","last_undegraded":"2024-09-17T03:51:10.786848+0000","last_fullsized":"2024-09-17T03:51:10.786848+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:40.182958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541590+0000","last_change":"2024-09-17T03:51:08.538721+0000","last_active":"2024-09-17T03:51:08.541590+0000","last_peered":"2024-09-17T03:51:08.541590+0000","last_clean":"2024-09-17T03:51:08.541590+0000","last_became_active":"2024-09-17T03:51:08.537978+0000","last_became_peered":"2024-09-17T03:51:08.537978+0000","last_unstale":"2024-09-17T03:51:08.541590+0000","last_undegraded":"2024-09-17T03:51:08.541590+0000","last_fullsized":"2024-09-17T03:51:08.541590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520408+0000","last_change":"2024-09-17T03:51:08.520408+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520408+0000","last_undegraded":"2024-09-17T03:51:08.520408+0000","last_fullsized":"2024-09-17T03:51:08.520408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539559+0000","last_change":"2024-09-17T03:51:08.535323+0000","last_active":"2024-09-17T03:51:08.539559+0000","last_peered":"2024-09-17T03:51:08.539559+0000","last_clean":"2024-09-17T03:51:08.539559+0000","last_became_active":"2024-09-17T03:51:08.534999+0000","last_became_peered":"2024-09-17T03:51:08.534999+0000","last_unstale":"2024-09-17T03:51:08.539559+0000","last_undegraded":"2024-09-17T03:51:08.539559+0000","last_fullsized":"2024-09-17T03:51:08.539559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:04.103238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":23,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786053+0000","last_change":"2024-09-17T03:51:08.530197+0000","last_active":"2024-09-17T03:51:10.786053+0000","last_peered":"2024-09-17T03:51:10.786053+0000","last_clean":"2024-09-17T03:51:10.786053+0000","last_became_active":"2024-09-17T03:51:08.529723+0000","last_became_peered":"2024-09-17T03:51:08.529723+0000","last_unstale":"2024-09-17T03:51:10.786053+0000","last_undegraded":"2024-09-17T03:51:10.786053+0000","last_fullsized":"2024-09-17T03:51:10.786053+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:24.054211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786825+0000","last_change":"2024-09-17T03:51:09.537261+0000","last_active":"2024-09-17T03:51:10.786825+0000","last_peered":"2024-09-17T03:51:10.786825+0000","last_clean":"2024-09-17T03:51:10.786825+0000","last_became_active":"2024-09-17T03:51:09.536865+0000","last_became_peered":"2024-09-17T03:51:09.536865+0000","last_unstale":"2024-09-17T03:51:10.786825+0000","last_undegraded":"2024-09-17T03:51:10.786825+0000","last_fullsized":"2024-09-17T03:51:10.786825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:02.803728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786694+0000","last_change":"2024-09-17T03:51:08.535330+0000","last_active":"2024-09-17T03:51:10.786694+0000","last_peered":"2024-09-17T03:51:10.786694+0000","last_clean":"2024-09-17T03:51:10.786694+0000","last_became_active":"2024-09-17T03:51:08.535035+0000","last_became_peered":"2024-09-17T03:51:08.535035+0000","last_unstale":"2024-09-17T03:51:10.786694+0000","last_undegraded":"2024-09-17T03:51:10.786694+0000","last_fullsized":"2024-09-17T03:51:10.786694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:25.251556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787536+0000","last_change":"2024-09-17T03:51:09.559584+0000","last_active":"2024-09-17T03:51:10.787536+0000","last_peered":"2024-09-17T03:51:10.787536+0000","last_clean":"2024-09-17T03:51:10.787536+0000","last_became_active":"2024-09-17T03:51:09.559215+0000","last_became_peered":"2024-09-17T03:51:09.559215+0000","last_unstale":"2024-09-17T03:51:10.787536+0000","last_undegraded":"2024-09-17T03:51:10.787536+0000","last_fullsized":"2024-09-17T03:51:10.787536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:48.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533214+0000","last_change":"2024-09-17T03:51:08.529671+0000","last_active":"2024-09-17T03:51:10.533214+0000","last_peered":"2024-09-17T03:51:10.533214+0000","last_clean":"2024-09-17T03:51:10.533214+0000","last_became_active":"2024-09-17T03:51:08.529313+0000","last_became_peered":"2024-09-17T03:51:08.529313+0000","last_unstale":"2024-09-17T03:51:10.533214+0000","last_undegraded":"2024-09-17T03:51:10.533214+0000","last_fullsized":"2024-09-17T03:51:10.533214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787006+0000","last_change":"2024-09-17T03:51:09.531351+0000","last_active":"2024-09-17T03:51:10.787006+0000","last_peered":"2024-09-17T03:51:10.787006+0000","last_clean":"2024-09-17T03:51:10.787006+0000","last_became_active":"2024-09-17T03:51:09.530270+0000","last_became_peered":"2024-09-17T03:51:09.530270+0000","last_unstale":"2024-09-17T03:51:10.787006+0000","last_undegraded":"2024-09-17T03:51:10.787006+0000","last_fullsized":"2024-09-17T03:51:10.787006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:38:05.287191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535766+0000","last_change":"2024-09-17T03:51:08.531273+0000","last_active":"2024-09-17T03:51:08.535766+0000","last_peered":"2024-09-17T03:51:08.535766+0000","last_clean":"2024-09-17T03:51:08.535766+0000","last_became_active":"2024-09-17T03:51:08.530933+0000","last_became_peered":"2024-09-17T03:51:08.530933+0000","last_unstale":"2024-09-17T03:51:08.535766+0000","last_undegraded":"2024-09-17T03:51:08.535766+0000","last_fullsized":"2024-09-17T03:51:08.535766+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787095+0000","last_change":"2024-09-17T03:51:09.545503+0000","last_active":"2024-09-17T03:51:10.787095+0000","last_peered":"2024-09-17T03:51:10.787095+0000","last_clean":"2024-09-17T03:51:10.787095+0000","last_became_active":"2024-09-17T03:51:09.544721+0000","last_became_peered":"2024-09-17T03:51:09.544721+0000","last_unstale":"2024-09-17T03:51:10.787095+0000","last_undegraded":"2024-09-17T03:51:10.787095+0000","last_fullsized":"2024-09-17T03:51:10.787095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:10.003938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539491+0000","last_change":"2024-09-17T03:51:08.534140+0000","last_active":"2024-09-17T03:51:08.539491+0000","last_peered":"2024-09-17T03:51:08.539491+0000","last_clean":"2024-09-17T03:51:08.539491+0000","last_became_active":"2024-09-17T03:51:08.533878+0000","last_became_peered":"2024-09-17T03:51:08.533878+0000","last_unstale":"2024-09-17T03:51:08.539491+0000","last_undegraded":"2024-09-17T03:51:08.539491+0000","last_fullsized":"2024-09-17T03:51:08.539491+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:18.671953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521195+0000","last_change":"2024-09-17T03:51:08.521195+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521195+0000","last_undegraded":"2024-09-17T03:51:08.521195+0000","last_fullsized":"2024-09-17T03:51:08.521195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537539+0000","last_change":"2024-09-17T03:51:08.533378+0000","last_active":"2024-09-17T03:51:08.537539+0000","last_peered":"2024-09-17T03:51:08.537539+0000","last_clean":"2024-09-17T03:51:08.537539+0000","last_became_active":"2024-09-17T03:51:08.532682+0000","last_became_peered":"2024-09-17T03:51:08.532682+0000","last_unstale":"2024-09-17T03:51:08.537539+0000","last_undegraded":"2024-09-17T03:51:08.537539+0000","last_fullsized":"2024-09-17T03:51:08.537539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.184648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514113+0000","last_change":"2024-09-17T03:51:08.514113+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514113+0000","last_undegraded":"2024-09-17T03:51:08.514113+0000","last_fullsized":"2024-09-17T03:51:08.514113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535852+0000","last_change":"2024-09-17T03:51:08.531986+0000","last_active":"2024-09-17T03:51:08.535852+0000","last_peered":"2024-09-17T03:51:08.535852+0000","last_clean":"2024-09-17T03:51:08.535852+0000","last_became_active":"2024-09-17T03:51:08.531674+0000","last_became_peered":"2024-09-17T03:51:08.531674+0000","last_unstale":"2024-09-17T03:51:08.535852+0000","last_undegraded":"2024-09-17T03:51:08.535852+0000","last_fullsized":"2024-09-17T03:51:08.535852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785978+0000","last_change":"2024-09-17T03:51:09.538946+0000","last_active":"2024-09-17T03:51:10.785978+0000","last_peered":"2024-09-17T03:51:10.785978+0000","last_clean":"2024-09-17T03:51:10.785978+0000","last_became_active":"2024-09-17T03:51:09.538617+0000","last_became_peered":"2024-09-17T03:51:09.538617+0000","last_unstale":"2024-09-17T03:51:10.785978+0000","last_undegraded":"2024-09-17T03:51:10.785978+0000","last_fullsized":"2024-09-17T03:51:10.785978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:46.014539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786247+0000","last_change":"2024-09-17T03:51:08.526124+0000","last_active":"2024-09-17T03:51:10.786247+0000","last_peered":"2024-09-17T03:51:10.786247+0000","last_clean":"2024-09-17T03:51:10.786247+0000","last_became_active":"2024-09-17T03:51:08.525837+0000","last_became_peered":"2024-09-17T03:51:08.525837+0000","last_unstale":"2024-09-17T03:51:10.786247+0000","last_undegraded":"2024-09-17T03:51:10.786247+0000","last_fullsized":"2024-09-17T03:51:10.786247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:09.038592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787015+0000","last_change":"2024-09-17T03:51:09.556563+0000","last_active":"2024-09-17T03:51:10.787015+0000","last_peered":"2024-09-17T03:51:10.787015+0000","last_clean":"2024-09-17T03:51:10.787015+0000","last_became_active":"2024-09-17T03:51:09.555716+0000","last_became_peered":"2024-09-17T03:51:09.555716+0000","last_unstale":"2024-09-17T03:51:10.787015+0000","last_undegraded":"2024-09-17T03:51:10.787015+0000","last_fullsized":"2024-09-17T03:51:10.787015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:30.216865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541521+0000","last_change":"2024-09-17T03:51:08.535527+0000","last_active":"2024-09-17T03:51:08.541521+0000","last_peered":"2024-09-17T03:51:08.541521+0000","last_clean":"2024-09-17T03:51:08.541521+0000","last_became_active":"2024-09-17T03:51:08.535182+0000","last_became_peered":"2024-09-17T03:51:08.535182+0000","last_unstale":"2024-09-17T03:51:08.541521+0000","last_undegraded":"2024-09-17T03:51:08.541521+0000","last_fullsized":"2024-09-17T03:51:08.541521+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787369+0000","last_change":"2024-09-17T03:51:09.552798+0000","last_active":"2024-09-17T03:51:10.787369+0000","last_peered":"2024-09-17T03:51:10.787369+0000","last_clean":"2024-09-17T03:51:10.787369+0000","last_became_active":"2024-09-17T03:51:09.552477+0000","last_became_peered":"2024-09-17T03:51:09.552477+0000","last_unstale":"2024-09-17T03:51:10.787369+0000","last_undegraded":"2024-09-17T03:51:10.787369+0000","last_fullsized":"2024-09-17T03:51:10.787369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:35.740581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537417+0000","last_change":"2024-09-17T03:51:08.531555+0000","last_active":"2024-09-17T03:51:08.537417+0000","last_peered":"2024-09-17T03:51:08.537417+0000","last_clean":"2024-09-17T03:51:08.537417+0000","last_became_active":"2024-09-17T03:51:08.531230+0000","last_became_peered":"2024-09-17T03:51:08.531230+0000","last_unstale":"2024-09-17T03:51:08.537417+0000","last_undegraded":"2024-09-17T03:51:08.537417+0000","last_fullsized":"2024-09-17T03:51:08.537417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:08.023937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787381+0000","last_change":"2024-09-17T03:51:09.558633+0000","last_active":"2024-09-17T03:51:10.787381+0000","last_peered":"2024-09-17T03:51:10.787381+0000","last_clean":"2024-09-17T03:51:10.787381+0000","last_became_active":"2024-09-17T03:51:09.558301+0000","last_became_peered":"2024-09-17T03:51:09.558301+0000","last_unstale":"2024-09-17T03:51:10.787381+0000","last_undegraded":"2024-09-17T03:51:10.787381+0000","last_fullsized":"2024-09-17T03:51:10.787381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:53.303687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.543358+0000","last_change":"2024-09-17T03:51:08.542303+0000","last_active":"2024-09-17T03:51:08.543358+0000","last_peered":"2024-09-17T03:51:08.543358+0000","last_clean":"2024-09-17T03:51:08.543358+0000","last_became_active":"2024-09-17T03:51:08.541884+0000","last_became_peered":"2024-09-17T03:51:08.541884+0000","last_unstale":"2024-09-17T03:51:08.543358+0000","last_undegraded":"2024-09-17T03:51:08.543358+0000","last_fullsized":"2024-09-17T03:51:08.543358+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.838696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536023+0000","last_change":"2024-09-17T03:51:08.533982+0000","last_active":"2024-09-17T03:51:08.536023+0000","last_peered":"2024-09-17T03:51:08.536023+0000","last_clean":"2024-09-17T03:51:08.536023+0000","last_became_active":"2024-09-17T03:51:08.529828+0000","last_became_peered":"2024-09-17T03:51:08.529828+0000","last_unstale":"2024-09-17T03:51:08.536023+0000","last_undegraded":"2024-09-17T03:51:08.536023+0000","last_fullsized":"2024-09-17T03:51:08.536023+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:18.636858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520810+0000","last_change":"2024-09-17T03:51:08.520810+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520810+0000","last_undegraded":"2024-09-17T03:51:08.520810+0000","last_fullsized":"2024-09-17T03:51:08.520810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542733+0000","last_change":"2024-09-17T03:51:08.540515+0000","last_active":"2024-09-17T03:51:08.542733+0000","last_peered":"2024-09-17T03:51:08.542733+0000","last_clean":"2024-09-17T03:51:08.542733+0000","last_became_active":"2024-09-17T03:51:08.540197+0000","last_became_peered":"2024-09-17T03:51:08.540197+0000","last_unstale":"2024-09-17T03:51:08.542733+0000","last_undegraded":"2024-09-17T03:51:08.542733+0000","last_fullsized":"2024-09-17T03:51:08.542733+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520049+0000","last_change":"2024-09-17T03:51:08.520049+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520049+0000","last_undegraded":"2024-09-17T03:51:08.520049+0000","last_fullsized":"2024-09-17T03:51:08.520049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534533+0000","last_change":"2024-09-17T03:51:08.538899+0000","last_active":"2024-09-17T03:51:10.534533+0000","last_peered":"2024-09-17T03:51:10.534533+0000","last_clean":"2024-09-17T03:51:10.534533+0000","last_became_active":"2024-09-17T03:51:08.538338+0000","last_became_peered":"2024-09-17T03:51:08.538338+0000","last_unstale":"2024-09-17T03:51:10.534533+0000","last_undegraded":"2024-09-17T03:51:10.534533+0000","last_fullsized":"2024-09-17T03:51:10.534533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:56.151879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520013+0000","last_change":"2024-09-17T03:51:08.520013+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520013+0000","last_undegraded":"2024-09-17T03:51:08.520013+0000","last_fullsized":"2024-09-17T03:51:08.520013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536701+0000","last_change":"2024-09-17T03:51:08.535001+0000","last_active":"2024-09-17T03:51:08.536701+0000","last_peered":"2024-09-17T03:51:08.536701+0000","last_clean":"2024-09-17T03:51:08.536701+0000","last_became_active":"2024-09-17T03:51:08.531914+0000","last_became_peered":"2024-09-17T03:51:08.531914+0000","last_unstale":"2024-09-17T03:51:08.536701+0000","last_undegraded":"2024-09-17T03:51:08.536701+0000","last_fullsized":"2024-09-17T03:51:08.536701+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:10.120644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532927+0000","last_change":"2024-09-17T03:51:09.540447+0000","last_active":"2024-09-17T03:51:10.532927+0000","last_peered":"2024-09-17T03:51:10.532927+0000","last_clean":"2024-09-17T03:51:10.532927+0000","last_became_active":"2024-09-17T03:51:09.540161+0000","last_became_peered":"2024-09-17T03:51:09.540161+0000","last_unstale":"2024-09-17T03:51:10.532927+0000","last_undegraded":"2024-09-17T03:51:10.532927+0000","last_fullsized":"2024-09-17T03:51:10.532927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:04.256313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786105+0000","last_change":"2024-09-17T03:51:08.525657+0000","last_active":"2024-09-17T03:51:10.786105+0000","last_peered":"2024-09-17T03:51:10.786105+0000","last_clean":"2024-09-17T03:51:10.786105+0000","last_became_active":"2024-09-17T03:51:08.525271+0000","last_became_peered":"2024-09-17T03:51:08.525271+0000","last_unstale":"2024-09-17T03:51:10.786105+0000","last_undegraded":"2024-09-17T03:51:10.786105+0000","last_fullsized":"2024-09-17T03:51:10.786105+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:55.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787764+0000","last_change":"2024-09-17T03:51:09.544994+0000","last_active":"2024-09-17T03:51:10.787764+0000","last_peered":"2024-09-17T03:51:10.787764+0000","last_clean":"2024-09-17T03:51:10.787764+0000","last_became_active":"2024-09-17T03:51:09.544748+0000","last_became_peered":"2024-09-17T03:51:09.544748+0000","last_unstale":"2024-09-17T03:51:10.787764+0000","last_undegraded":"2024-09-17T03:51:10.787764+0000","last_fullsized":"2024-09-17T03:51:10.787764+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:14.819738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535595+0000","last_change":"2024-09-17T03:51:08.529407+0000","last_active":"2024-09-17T03:51:08.535595+0000","last_peered":"2024-09-17T03:51:08.535595+0000","last_clean":"2024-09-17T03:51:08.535595+0000","last_became_active":"2024-09-17T03:51:08.529046+0000","last_became_peered":"2024-09-17T03:51:08.529046+0000","last_unstale":"2024-09-17T03:51:08.535595+0000","last_undegraded":"2024-09-17T03:51:08.535595+0000","last_fullsized":"2024-09-17T03:51:08.535595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":148,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.524741+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:08.524741+0000","last_peered":"2024-09-17T03:51:08.524741+0000","last_clean":"2024-09-17T03:51:08.524741+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:08.524741+0000","last_undegraded":"2024-09-17T03:51:08.524741+0000","last_fullsized":"2024-09-17T03:51:08.524741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1089536,"data_stored":786166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509466,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":210485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":536576,"data_stored":210507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":209545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":799275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:15.171 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3f is not active+clean 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.518100+0000', 'last_change': '2024-09-17T03:51:08.518100+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.518100+0000', 'last_undegraded': '2024-09-17T03:51:08.518100+0000', 'last_fullsized': '2024-09-17T03:51:08.518100+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3e is not active+clean 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.521176+0000', 'last_change': '2024-09-17T03:51:08.521176+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.521176+0000', 'last_undegraded': '2024-09-17T03:51:08.521176+0000', 'last_fullsized': '2024-09-17T03:51:08.521176+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3b is not active+clean 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.521899+0000', 'last_change': '2024-09-17T03:51:08.521899+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.521899+0000', 'last_undegraded': '2024-09-17T03:51:08.521899+0000', 'last_fullsized': '2024-09-17T03:51:08.521899+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3a is not active+clean 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.521676+0000', 'last_change': '2024-09-17T03:51:08.521676+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.521676+0000', 'last_undegraded': '2024-09-17T03:51:08.521676+0000', 'last_fullsized': '2024-09-17T03:51:08.521676+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:PG 7.39 is not active+clean 2024-09-17T03:51:15.172 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.518344+0000', 'last_change': '2024-09-17T03:51:08.518344+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.518344+0000', 'last_undegraded': '2024-09-17T03:51:08.518344+0000', 'last_fullsized': '2024-09-17T03:51:08.518344+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:PG 7.34 is not active+clean 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.523028+0000', 'last_change': '2024-09-17T03:51:08.523028+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.523028+0000', 'last_undegraded': '2024-09-17T03:51:08.523028+0000', 'last_fullsized': '2024-09-17T03:51:08.523028+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:PG 7.31 is not active+clean 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520415+0000', 'last_change': '2024-09-17T03:51:08.520415+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520415+0000', 'last_undegraded': '2024-09-17T03:51:08.520415+0000', 'last_fullsized': '2024-09-17T03:51:08.520415+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2b is not active+clean 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520494+0000', 'last_change': '2024-09-17T03:51:08.520494+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520494+0000', 'last_undegraded': '2024-09-17T03:51:08.520494+0000', 'last_fullsized': '2024-09-17T03:51:08.520494+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:PG 7.24 is not active+clean 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.522524+0000', 'last_change': '2024-09-17T03:51:08.522524+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.522524+0000', 'last_undegraded': '2024-09-17T03:51:08.522524+0000', 'last_fullsized': '2024-09-17T03:51:08.522524+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:PG 7.21 is not active+clean 2024-09-17T03:51:15.173 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.519903+0000', 'last_change': '2024-09-17T03:51:08.519903+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.519903+0000', 'last_undegraded': '2024-09-17T03:51:08.519903+0000', 'last_fullsized': '2024-09-17T03:51:08.519903+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1e is not active+clean 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.514124+0000', 'last_change': '2024-09-17T03:51:08.514124+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.514124+0000', 'last_undegraded': '2024-09-17T03:51:08.514124+0000', 'last_fullsized': '2024-09-17T03:51:08.514124+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1c is not active+clean 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.521613+0000', 'last_change': '2024-09-17T03:51:08.521613+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.521613+0000', 'last_undegraded': '2024-09-17T03:51:08.521613+0000', 'last_fullsized': '2024-09-17T03:51:08.521613+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:PG 7.1d is not active+clean 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.514188+0000', 'last_change': '2024-09-17T03:51:08.514188+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.514188+0000', 'last_undegraded': '2024-09-17T03:51:08.514188+0000', 'last_fullsized': '2024-09-17T03:51:08.514188+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:PG 7.17 is not active+clean 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.517437+0000', 'last_change': '2024-09-17T03:51:08.517437+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.517437+0000', 'last_undegraded': '2024-09-17T03:51:08.517437+0000', 'last_fullsized': '2024-09-17T03:51:08.517437+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:PG 7.14 is not active+clean 2024-09-17T03:51:15.174 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.518389+0000', 'last_change': '2024-09-17T03:51:08.518389+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.518389+0000', 'last_undegraded': '2024-09-17T03:51:08.518389+0000', 'last_fullsized': '2024-09-17T03:51:08.518389+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.15 is not active+clean 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.518861+0000', 'last_change': '2024-09-17T03:51:08.518861+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.518861+0000', 'last_undegraded': '2024-09-17T03:51:08.518861+0000', 'last_fullsized': '2024-09-17T03:51:08.518861+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.10 is not active+clean 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520408+0000', 'last_change': '2024-09-17T03:51:08.520408+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520408+0000', 'last_undegraded': '2024-09-17T03:51:08.520408+0000', 'last_fullsized': '2024-09-17T03:51:08.520408+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.11 is not active+clean 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.517359+0000', 'last_change': '2024-09-17T03:51:08.517359+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.517359+0000', 'last_undegraded': '2024-09-17T03:51:08.517359+0000', 'last_fullsized': '2024-09-17T03:51:08.517359+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.a is not active+clean 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.521195+0000', 'last_change': '2024-09-17T03:51:08.521195+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.521195+0000', 'last_undegraded': '2024-09-17T03:51:08.521195+0000', 'last_fullsized': '2024-09-17T03:51:08.521195+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.b is not active+clean 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.514113+0000', 'last_change': '2024-09-17T03:51:08.514113+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.514113+0000', 'last_undegraded': '2024-09-17T03:51:08.514113+0000', 'last_fullsized': '2024-09-17T03:51:08.514113+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:51:15.175 INFO:tasks.ceph.ceph_manager.ceph:PG 7.5 is not active+clean 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.517359+0000', 'last_change': '2024-09-17T03:51:08.517359+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.517359+0000', 'last_undegraded': '2024-09-17T03:51:08.517359+0000', 'last_fullsized': '2024-09-17T03:51:08.517359+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:PG 7.2 is not active+clean 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520810+0000', 'last_change': '2024-09-17T03:51:08.520810+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520810+0000', 'last_undegraded': '2024-09-17T03:51:08.520810+0000', 'last_fullsized': '2024-09-17T03:51:08.520810+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:PG 7.3 is not active+clean 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520049+0000', 'last_change': '2024-09-17T03:51:08.520049+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520049+0000', 'last_undegraded': '2024-09-17T03:51:08.520049+0000', 'last_fullsized': '2024-09-17T03:51:08.520049+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:PG 7.0 is not active+clean 2024-09-17T03:51:15.176 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '7.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 43, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:51:08.520013+0000', 'last_change': '2024-09-17T03:51:08.520013+0000', 'last_active': '2024-09-17T03:51:08.499499+0000', 'last_peered': '2024-09-17T03:51:08.499499+0000', 'last_clean': '2024-09-17T03:51:08.499499+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:51:08.520013+0000', 'last_undegraded': '2024-09-17T03:51:08.520013+0000', 'last_fullsized': '2024-09-17T03:51:08.520013+0000', 'mapping_epoch': 43, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 43, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'last_clean_scrub_stamp': '2024-09-17T03:51:08.499499+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T03:51:15.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T03:51:15.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.399+0000 7fd558a00640 1 -- 172.21.15.129:0/730568515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 msgr2=0x7fd538003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.399+0000 7fd558a00640 1 --2- 172.21.15.129:0/730568515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd538003ea0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd544001e00 tx=0x7fd538005830 comp rx=0 tx=0).stop 2024-09-17T03:51:15.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.400+0000 7fd553200640 1 -- 172.21.15.129:0/730568515 reap_dead start 2024-09-17T03:51:15.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.400+0000 7fd558a00640 1 -- 172.21.15.129:0/730568515 shutdown_connections 2024-09-17T03:51:15.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.400+0000 7fd558a00640 1 -- 172.21.15.129:0/730568515 >> 172.21.15.129:0/730568515 conn(0x7fd55409d680 msgr2=0x7fd5540d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:15.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.400+0000 7fd558a00640 1 -- 172.21.15.129:0/730568515 shutdown_connections 2024-09-17T03:51:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.400+0000 7fd558a00640 1 -- 172.21.15.129:0/730568515 wait complete. 2024-09-17T03:51:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.402+0000 7fd558a00640 1 Processor -- start 2024-09-17T03:51:15.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.402+0000 7fd558a00640 1 -- start start 2024-09-17T03:51:15.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.403+0000 7fd558a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.403+0000 7fd558a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5541530d0 0x7fd5540cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd558a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5541534e0 0x7fd5540ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd558a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd5540d1880 con 0x7fd538003a30 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd558a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd5540d1700 con 0x7fd5541530d0 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd558a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd5540d1a00 con 0x7fd5541534e0 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39430/0 (socket says 172.21.15.129:39430) 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 -- 172.21.15.129:0/1886691513 learned_addr learned my addr 172.21.15.129:0/1886691513 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd551e00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5541530d0 0x7fd5540cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd553200640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5541534e0 0x7fd5540ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 -- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5541534e0 msgr2=0x7fd5540ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5541534e0 0x7fd5540ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 -- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5541530d0 msgr2=0x7fd5540cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5541530d0 0x7fd5540cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 -- 172.21.15.129:0/1886691513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5541f5bf0 con 0x7fd538003a30 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.404+0000 7fd552800640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd544000f80 tx=0x7fd544003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.405+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5440062d0 con 0x7fd538003a30 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.405+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd544006560 con 0x7fd538003a30 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.405+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544033480 con 0x7fd538003a30 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.405+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5541f5ed0 con 0x7fd538003a30 2024-09-17T03:51:15.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.405+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5541f6490 con 0x7fd538003a30 2024-09-17T03:51:15.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.406+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5541f66d0 con 0x7fd538003a30 2024-09-17T03:51:15.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.407+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd544033620 con 0x7fd538003a30 2024-09-17T03:51:15.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.408+0000 7fd54aa00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 0x7fd53407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.408+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fd5440bb430 con 0x7fd538003a30 2024-09-17T03:51:15.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.409+0000 7fd551e00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 0x7fd53407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.409+0000 7fd551e00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 0x7fd53407b0a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd55415b920 tx=0x7fd5400067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:15.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.413+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd5541f66d0 con 0x7fd538003a30 2024-09-17T03:51:15.476 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:51:15.477+0000 7fe7a9a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:51:15.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.665+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fd5540cfd70 con 0x7fd538003a30 2024-09-17T03:51:15.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.665+0000 7fd54aa00640 1 -- 172.21.15.129:0/1886691513 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1559 (secure 0 0 0) 0x7fd5440839c0 con 0x7fd538003a30 2024-09-17T03:51:15.668 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:15.669 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":126,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":73},{"state_name":"creating+peering","count":24}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":592002,"bytes_used":260804608,"bytes_avail":772833308672,"bytes_total":773094113280,"inactive_pgs_ratio":0.24742268025875092,"write_bytes_sec":682,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"c","status":"up:active","gid":5232}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-17T03:50:56.314241+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 msgr2=0x7fd53407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 0x7fd53407b0a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd55415b920 tx=0x7fd5400067a0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 msgr2=0x7fd5540c85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd544000f80 tx=0x7fd544003a40 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 shutdown_connections 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd534078b90 0x7fd53407b0a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5541534e0 0x7fd5540ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5541530d0 0x7fd5540cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 --2- 172.21.15.129:0/1886691513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd538003a30 0x7fd5540c85f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 >> 172.21.15.129:0/1886691513 conn(0x7fd55409d680 msgr2=0x7fd5540b8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:15.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 shutdown_connections 2024-09-17T03:51:15.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.668+0000 7fd558a00640 1 -- 172.21.15.129:0/1886691513 wait complete. 2024-09-17T03:51:15.690 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T03:51:15.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:15.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:51:15.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 -- 172.21.15.129:0/1313035118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 msgr2=0x7fdf580af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 --2- 172.21.15.129:0/1313035118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf580af3c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009830 tx=0x7fdf4c02a510 comp rx=0 tx=0).stop 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 -- 172.21.15.129:0/1313035118 shutdown_connections 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 --2- 172.21.15.129:0/1313035118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 --2- 172.21.15.129:0/1313035118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf580af3c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 -- 172.21.15.129:0/1313035118 >> 172.21.15.129:0/1313035118 conn(0x7fdf5806d7d0 msgr2=0x7fdf580b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 -- 172.21.15.129:0/1313035118 shutdown_connections 2024-09-17T03:51:15.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.915+0000 7fdf60800640 1 -- 172.21.15.129:0/1313035118 wait complete. 2024-09-17T03:51:15.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.919+0000 7fdf60800640 1 Processor -- start 2024-09-17T03:51:15.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.919+0000 7fdf60800640 1 -- start start 2024-09-17T03:51:15.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf58150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf581533a0 con 0x7fdf5815d8a0 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf58153520 con 0x7fdf580cc1e0 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.920+0000 7fdf60800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdf581536a0 con 0x7fdf580ccb50 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39442/0 (socket says 172.21.15.129:39442) 2024-09-17T03:51:15.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 -- 172.21.15.129:0/1742272738 learned_addr learned my addr 172.21.15.129:0/1742272738 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5de00640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf58150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5d400640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 -- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 msgr2=0x7fdf580ad4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ad4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 -- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 msgr2=0x7fdf58150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf58150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5e800640 1 -- 172.21.15.129:0/1742272738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf580ca2f0 con 0x7fdf5815d8a0 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.921+0000 7fdf5d400640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf5e800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ea40 tx=0x7fdf5400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5400ce40 con 0x7fdf5815d8a0 2024-09-17T03:51:15.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf54004c10 con 0x7fdf5815d8a0 2024-09-17T03:51:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf580ca5d0 con 0x7fdf5815d8a0 2024-09-17T03:51:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54005020 con 0x7fdf5815d8a0 2024-09-17T03:51:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.922+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf580cab90 con 0x7fdf5815d8a0 2024-09-17T03:51:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.923+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf580ca060 con 0x7fdf5815d8a0 2024-09-17T03:51:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.924+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdf540051c0 con 0x7fdf5815d8a0 2024-09-17T03:51:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.925+0000 7fdf52000640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 0x7fdf3c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.925+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fdf5409bbd0 con 0x7fdf5815d8a0 2024-09-17T03:51:15.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.926+0000 7fdf5de00640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 0x7fdf3c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:15.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.926+0000 7fdf5de00640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 0x7fdf3c07b0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c034000 tx=0x7fdf4c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:15.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:15.930+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdf54064010 con 0x7fdf5815d8a0 2024-09-17T03:51:16.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.111+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf580cef90 con 0x7fdf3c078bc0 2024-09-17T03:51:16.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.116+0000 7fdf52000640 1 -- 172.21.15.129:0/1742272738 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249961 (secure 0 0 0) 0x7fdf4c004300 con 0x7fdf3c078bc0 2024-09-17T03:51:16.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:16.119 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.120+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 msgr2=0x7fdf3c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.120+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 0x7fdf3c07b0d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c034000 tx=0x7fdf4c004730 comp rx=0 tx=0).stop 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.120+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 msgr2=0x7fdf580ada00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.120+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400ea40 tx=0x7fdf5400ef00 comp rx=0 tx=0).stop 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 shutdown_connections 2024-09-17T03:51:16.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf3c078bc0 0x7fdf3c07b0d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf5815d8a0 0x7fdf580ada00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf580ccb50 0x7fdf580ad4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 --2- 172.21.15.129:0/1742272738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf580cc1e0 0x7fdf58150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 >> 172.21.15.129:0/1742272738 conn(0x7fdf5806d7d0 msgr2=0x7fdf580b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 shutdown_connections 2024-09-17T03:51:16.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:16.121+0000 7fdf60800640 1 -- 172.21.15.129:0/1742272738 wait complete. 2024-09-17T03:51:16.141 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-17T03:51:14.318893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592002,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":123,"num_write_kb":1632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254692,"kb_used_data":5168,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720028,"statfs":{"total":773094113280,"available":772833308672,"internally_reserved":0,"allocated":5292032,"data_stored":2803372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001657"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518100+0000","last_change":"2024-09-17T03:51:08.518100+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518100+0000","last_undegraded":"2024-09-17T03:51:08.518100+0000","last_fullsized":"2024-09-17T03:51:08.518100+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521176+0000","last_change":"2024-09-17T03:51:08.521176+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521176+0000","last_undegraded":"2024-09-17T03:51:08.521176+0000","last_fullsized":"2024-09-17T03:51:08.521176+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532794+0000","last_change":"2024-09-17T03:51:09.544780+0000","last_active":"2024-09-17T03:51:10.532794+0000","last_peered":"2024-09-17T03:51:10.532794+0000","last_clean":"2024-09-17T03:51:10.532794+0000","last_became_active":"2024-09-17T03:51:09.544462+0000","last_became_peered":"2024-09-17T03:51:09.544462+0000","last_unstale":"2024-09-17T03:51:10.532794+0000","last_undegraded":"2024-09-17T03:51:10.532794+0000","last_fullsized":"2024-09-17T03:51:10.532794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:20.077167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787357+0000","last_change":"2024-09-17T03:51:09.553310+0000","last_active":"2024-09-17T03:51:10.787357+0000","last_peered":"2024-09-17T03:51:10.787357+0000","last_clean":"2024-09-17T03:51:10.787357+0000","last_became_active":"2024-09-17T03:51:09.552339+0000","last_became_peered":"2024-09-17T03:51:09.552339+0000","last_unstale":"2024-09-17T03:51:10.787357+0000","last_undegraded":"2024-09-17T03:51:10.787357+0000","last_fullsized":"2024-09-17T03:51:10.787357+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:59.902480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521899+0000","last_change":"2024-09-17T03:51:08.521899+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521899+0000","last_undegraded":"2024-09-17T03:51:08.521899+0000","last_fullsized":"2024-09-17T03:51:08.521899+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521676+0000","last_change":"2024-09-17T03:51:08.521676+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521676+0000","last_undegraded":"2024-09-17T03:51:08.521676+0000","last_fullsized":"2024-09-17T03:51:08.521676+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518344+0000","last_change":"2024-09-17T03:51:08.518344+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518344+0000","last_undegraded":"2024-09-17T03:51:08.518344+0000","last_fullsized":"2024-09-17T03:51:08.518344+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785916+0000","last_change":"2024-09-17T03:51:09.553571+0000","last_active":"2024-09-17T03:51:10.785916+0000","last_peered":"2024-09-17T03:51:10.785916+0000","last_clean":"2024-09-17T03:51:10.785916+0000","last_became_active":"2024-09-17T03:51:09.553226+0000","last_became_peered":"2024-09-17T03:51:09.553226+0000","last_unstale":"2024-09-17T03:51:10.785916+0000","last_undegraded":"2024-09-17T03:51:10.785916+0000","last_fullsized":"2024-09-17T03:51:10.785916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:13.896356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787450+0000","last_change":"2024-09-17T03:51:09.558849+0000","last_active":"2024-09-17T03:51:10.787450+0000","last_peered":"2024-09-17T03:51:10.787450+0000","last_clean":"2024-09-17T03:51:10.787450+0000","last_became_active":"2024-09-17T03:51:09.558596+0000","last_became_peered":"2024-09-17T03:51:09.558596+0000","last_unstale":"2024-09-17T03:51:10.787450+0000","last_undegraded":"2024-09-17T03:51:10.787450+0000","last_fullsized":"2024-09-17T03:51:10.787450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:19.302750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787683+0000","last_change":"2024-09-17T03:51:09.552932+0000","last_active":"2024-09-17T03:51:10.787683+0000","last_peered":"2024-09-17T03:51:10.787683+0000","last_clean":"2024-09-17T03:51:10.787683+0000","last_became_active":"2024-09-17T03:51:09.552660+0000","last_became_peered":"2024-09-17T03:51:09.552660+0000","last_unstale":"2024-09-17T03:51:10.787683+0000","last_undegraded":"2024-09-17T03:51:10.787683+0000","last_fullsized":"2024-09-17T03:51:10.787683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:45.040579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787563+0000","last_change":"2024-09-17T03:51:09.556091+0000","last_active":"2024-09-17T03:51:10.787563+0000","last_peered":"2024-09-17T03:51:10.787563+0000","last_clean":"2024-09-17T03:51:10.787563+0000","last_became_active":"2024-09-17T03:51:09.555788+0000","last_became_peered":"2024-09-17T03:51:09.555788+0000","last_unstale":"2024-09-17T03:51:10.787563+0000","last_undegraded":"2024-09-17T03:51:10.787563+0000","last_fullsized":"2024-09-17T03:51:10.787563+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:35.856529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.523028+0000","last_change":"2024-09-17T03:51:08.523028+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.523028+0000","last_undegraded":"2024-09-17T03:51:08.523028+0000","last_fullsized":"2024-09-17T03:51:08.523028+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786586+0000","last_change":"2024-09-17T03:51:09.549923+0000","last_active":"2024-09-17T03:51:10.786586+0000","last_peered":"2024-09-17T03:51:10.786586+0000","last_clean":"2024-09-17T03:51:10.786586+0000","last_became_active":"2024-09-17T03:51:09.549427+0000","last_became_peered":"2024-09-17T03:51:09.549427+0000","last_unstale":"2024-09-17T03:51:10.786586+0000","last_undegraded":"2024-09-17T03:51:10.786586+0000","last_fullsized":"2024-09-17T03:51:10.786586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:12.588911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786097+0000","last_change":"2024-09-17T03:51:09.557049+0000","last_active":"2024-09-17T03:51:10.786097+0000","last_peered":"2024-09-17T03:51:10.786097+0000","last_clean":"2024-09-17T03:51:10.786097+0000","last_became_active":"2024-09-17T03:51:09.556732+0000","last_became_peered":"2024-09-17T03:51:09.556732+0000","last_unstale":"2024-09-17T03:51:10.786097+0000","last_undegraded":"2024-09-17T03:51:10.786097+0000","last_fullsized":"2024-09-17T03:51:10.786097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:51.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520415+0000","last_change":"2024-09-17T03:51:08.520415+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520415+0000","last_undegraded":"2024-09-17T03:51:08.520415+0000","last_fullsized":"2024-09-17T03:51:08.520415+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786655+0000","last_change":"2024-09-17T03:51:09.552281+0000","last_active":"2024-09-17T03:51:10.786655+0000","last_peered":"2024-09-17T03:51:10.786655+0000","last_clean":"2024-09-17T03:51:10.786655+0000","last_became_active":"2024-09-17T03:51:09.551898+0000","last_became_peered":"2024-09-17T03:51:09.551898+0000","last_unstale":"2024-09-17T03:51:10.786655+0000","last_undegraded":"2024-09-17T03:51:10.786655+0000","last_fullsized":"2024-09-17T03:51:10.786655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:19.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533660+0000","last_change":"2024-09-17T03:51:09.533608+0000","last_active":"2024-09-17T03:51:10.533660+0000","last_peered":"2024-09-17T03:51:10.533660+0000","last_clean":"2024-09-17T03:51:10.533660+0000","last_became_active":"2024-09-17T03:51:09.533263+0000","last_became_peered":"2024-09-17T03:51:09.533263+0000","last_unstale":"2024-09-17T03:51:10.533660+0000","last_undegraded":"2024-09-17T03:51:10.533660+0000","last_fullsized":"2024-09-17T03:51:10.533660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:18.712043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532553+0000","last_change":"2024-09-17T03:51:09.549893+0000","last_active":"2024-09-17T03:51:10.532553+0000","last_peered":"2024-09-17T03:51:10.532553+0000","last_clean":"2024-09-17T03:51:10.532553+0000","last_became_active":"2024-09-17T03:51:09.548155+0000","last_became_peered":"2024-09-17T03:51:09.548155+0000","last_unstale":"2024-09-17T03:51:10.532553+0000","last_undegraded":"2024-09-17T03:51:10.532553+0000","last_fullsized":"2024-09-17T03:51:10.532553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:06.020247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787285+0000","last_change":"2024-09-17T03:51:09.545828+0000","last_active":"2024-09-17T03:51:10.787285+0000","last_peered":"2024-09-17T03:51:10.787285+0000","last_clean":"2024-09-17T03:51:10.787285+0000","last_became_active":"2024-09-17T03:51:09.545207+0000","last_became_peered":"2024-09-17T03:51:09.545207+0000","last_unstale":"2024-09-17T03:51:10.787285+0000","last_undegraded":"2024-09-17T03:51:10.787285+0000","last_fullsized":"2024-09-17T03:51:10.787285+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:11.346092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786753+0000","last_change":"2024-09-17T03:51:09.550468+0000","last_active":"2024-09-17T03:51:10.786753+0000","last_peered":"2024-09-17T03:51:10.786753+0000","last_clean":"2024-09-17T03:51:10.786753+0000","last_became_active":"2024-09-17T03:51:09.550210+0000","last_became_peered":"2024-09-17T03:51:09.550210+0000","last_unstale":"2024-09-17T03:51:10.786753+0000","last_undegraded":"2024-09-17T03:51:10.786753+0000","last_fullsized":"2024-09-17T03:51:10.786753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:55.396574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520494+0000","last_change":"2024-09-17T03:51:08.520494+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520494+0000","last_undegraded":"2024-09-17T03:51:08.520494+0000","last_fullsized":"2024-09-17T03:51:08.520494+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786228+0000","last_change":"2024-09-17T03:51:09.551749+0000","last_active":"2024-09-17T03:51:10.786228+0000","last_peered":"2024-09-17T03:51:10.786228+0000","last_clean":"2024-09-17T03:51:10.786228+0000","last_became_active":"2024-09-17T03:51:09.551425+0000","last_became_peered":"2024-09-17T03:51:09.551425+0000","last_unstale":"2024-09-17T03:51:10.786228+0000","last_undegraded":"2024-09-17T03:51:10.786228+0000","last_fullsized":"2024-09-17T03:51:10.786228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:31.877445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786776+0000","last_change":"2024-09-17T03:51:09.537760+0000","last_active":"2024-09-17T03:51:10.786776+0000","last_peered":"2024-09-17T03:51:10.786776+0000","last_clean":"2024-09-17T03:51:10.786776+0000","last_became_active":"2024-09-17T03:51:09.537430+0000","last_became_peered":"2024-09-17T03:51:09.537430+0000","last_unstale":"2024-09-17T03:51:10.786776+0000","last_undegraded":"2024-09-17T03:51:10.786776+0000","last_fullsized":"2024-09-17T03:51:10.786776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:40.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786706+0000","last_change":"2024-09-17T03:51:09.535683+0000","last_active":"2024-09-17T03:51:10.786706+0000","last_peered":"2024-09-17T03:51:10.786706+0000","last_clean":"2024-09-17T03:51:10.786706+0000","last_became_active":"2024-09-17T03:51:09.535368+0000","last_became_peered":"2024-09-17T03:51:09.535368+0000","last_unstale":"2024-09-17T03:51:10.786706+0000","last_undegraded":"2024-09-17T03:51:10.786706+0000","last_fullsized":"2024-09-17T03:51:10.786706+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:28.792675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787126+0000","last_change":"2024-09-17T03:51:09.531819+0000","last_active":"2024-09-17T03:51:10.787126+0000","last_peered":"2024-09-17T03:51:10.787126+0000","last_clean":"2024-09-17T03:51:10.787126+0000","last_became_active":"2024-09-17T03:51:09.531504+0000","last_became_peered":"2024-09-17T03:51:09.531504+0000","last_unstale":"2024-09-17T03:51:10.787126+0000","last_undegraded":"2024-09-17T03:51:10.787126+0000","last_fullsized":"2024-09-17T03:51:10.787126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:37.424982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786610+0000","last_change":"2024-09-17T03:51:09.561216+0000","last_active":"2024-09-17T03:51:10.786610+0000","last_peered":"2024-09-17T03:51:10.786610+0000","last_clean":"2024-09-17T03:51:10.786610+0000","last_became_active":"2024-09-17T03:51:09.560930+0000","last_became_peered":"2024-09-17T03:51:09.560930+0000","last_unstale":"2024-09-17T03:51:10.786610+0000","last_undegraded":"2024-09-17T03:51:10.786610+0000","last_fullsized":"2024-09-17T03:51:10.786610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:24.329326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533087+0000","last_change":"2024-09-17T03:51:09.532459+0000","last_active":"2024-09-17T03:51:10.533087+0000","last_peered":"2024-09-17T03:51:10.533087+0000","last_clean":"2024-09-17T03:51:10.533087+0000","last_became_active":"2024-09-17T03:51:09.532056+0000","last_became_peered":"2024-09-17T03:51:09.532056+0000","last_unstale":"2024-09-17T03:51:10.533087+0000","last_undegraded":"2024-09-17T03:51:10.533087+0000","last_fullsized":"2024-09-17T03:51:10.533087+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:07.247043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.522524+0000","last_change":"2024-09-17T03:51:08.522524+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.522524+0000","last_undegraded":"2024-09-17T03:51:08.522524+0000","last_fullsized":"2024-09-17T03:51:08.522524+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787120+0000","last_change":"2024-09-17T03:51:09.531111+0000","last_active":"2024-09-17T03:51:10.787120+0000","last_peered":"2024-09-17T03:51:10.787120+0000","last_clean":"2024-09-17T03:51:10.787120+0000","last_became_active":"2024-09-17T03:51:09.530769+0000","last_became_peered":"2024-09-17T03:51:09.530769+0000","last_unstale":"2024-09-17T03:51:10.787120+0000","last_undegraded":"2024-09-17T03:51:10.787120+0000","last_fullsized":"2024-09-17T03:51:10.787120+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:05.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.540408+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.540084+0000","last_became_peered":"2024-09-17T03:51:09.540084+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:09.874665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.519903+0000","last_change":"2024-09-17T03:51:08.519903+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.519903+0000","last_undegraded":"2024-09-17T03:51:08.519903+0000","last_fullsized":"2024-09-17T03:51:08.519903+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786259+0000","last_change":"2024-09-17T03:51:09.532104+0000","last_active":"2024-09-17T03:51:10.786259+0000","last_peered":"2024-09-17T03:51:10.786259+0000","last_clean":"2024-09-17T03:51:10.786259+0000","last_became_active":"2024-09-17T03:51:09.531801+0000","last_became_peered":"2024-09-17T03:51:09.531801+0000","last_unstale":"2024-09-17T03:51:10.786259+0000","last_undegraded":"2024-09-17T03:51:10.786259+0000","last_fullsized":"2024-09-17T03:51:10.786259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:40.757442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514124+0000","last_change":"2024-09-17T03:51:08.514124+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514124+0000","last_undegraded":"2024-09-17T03:51:08.514124+0000","last_fullsized":"2024-09-17T03:51:08.514124+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539585+0000","last_change":"2024-09-17T03:51:08.536130+0000","last_active":"2024-09-17T03:51:08.539585+0000","last_peered":"2024-09-17T03:51:08.539585+0000","last_clean":"2024-09-17T03:51:08.539585+0000","last_became_active":"2024-09-17T03:51:08.535845+0000","last_became_peered":"2024-09-17T03:51:08.535845+0000","last_unstale":"2024-09-17T03:51:08.539585+0000","last_undegraded":"2024-09-17T03:51:08.539585+0000","last_fullsized":"2024-09-17T03:51:08.539585+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.579356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787189+0000","last_change":"2024-09-17T03:51:09.530989+0000","last_active":"2024-09-17T03:51:10.787189+0000","last_peered":"2024-09-17T03:51:10.787189+0000","last_clean":"2024-09-17T03:51:10.787189+0000","last_became_active":"2024-09-17T03:51:09.529800+0000","last_became_peered":"2024-09-17T03:51:09.529800+0000","last_unstale":"2024-09-17T03:51:10.787189+0000","last_undegraded":"2024-09-17T03:51:10.787189+0000","last_fullsized":"2024-09-17T03:51:10.787189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:22.141601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533204+0000","last_change":"2024-09-17T03:51:08.537833+0000","last_active":"2024-09-17T03:51:10.533204+0000","last_peered":"2024-09-17T03:51:10.533204+0000","last_clean":"2024-09-17T03:51:10.533204+0000","last_became_active":"2024-09-17T03:51:08.536906+0000","last_became_peered":"2024-09-17T03:51:08.536906+0000","last_unstale":"2024-09-17T03:51:10.533204+0000","last_undegraded":"2024-09-17T03:51:10.533204+0000","last_fullsized":"2024-09-17T03:51:10.533204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521613+0000","last_change":"2024-09-17T03:51:08.521613+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521613+0000","last_undegraded":"2024-09-17T03:51:08.521613+0000","last_fullsized":"2024-09-17T03:51:08.521613+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535582+0000","last_change":"2024-09-17T03:51:08.532550+0000","last_active":"2024-09-17T03:51:08.535582+0000","last_peered":"2024-09-17T03:51:08.535582+0000","last_clean":"2024-09-17T03:51:08.535582+0000","last_became_active":"2024-09-17T03:51:08.532173+0000","last_became_peered":"2024-09-17T03:51:08.532173+0000","last_unstale":"2024-09-17T03:51:08.535582+0000","last_undegraded":"2024-09-17T03:51:08.535582+0000","last_fullsized":"2024-09-17T03:51:08.535582+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:22.458883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514188+0000","last_change":"2024-09-17T03:51:08.514188+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514188+0000","last_undegraded":"2024-09-17T03:51:08.514188+0000","last_fullsized":"2024-09-17T03:51:08.514188+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541632+0000","last_change":"2024-09-17T03:51:08.538223+0000","last_active":"2024-09-17T03:51:08.541632+0000","last_peered":"2024-09-17T03:51:08.541632+0000","last_clean":"2024-09-17T03:51:08.541632+0000","last_became_active":"2024-09-17T03:51:08.536819+0000","last_became_peered":"2024-09-17T03:51:08.536819+0000","last_unstale":"2024-09-17T03:51:08.541632+0000","last_undegraded":"2024-09-17T03:51:08.541632+0000","last_fullsized":"2024-09-17T03:51:08.541632+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.533568+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.533027+0000","last_became_peered":"2024-09-17T03:51:09.533027+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:39.579158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537328+0000","last_change":"2024-09-17T03:51:08.533400+0000","last_active":"2024-09-17T03:51:08.537328+0000","last_peered":"2024-09-17T03:51:08.537328+0000","last_clean":"2024-09-17T03:51:08.537328+0000","last_became_active":"2024-09-17T03:51:08.532958+0000","last_became_peered":"2024-09-17T03:51:08.532958+0000","last_unstale":"2024-09-17T03:51:08.537328+0000","last_undegraded":"2024-09-17T03:51:08.537328+0000","last_fullsized":"2024-09-17T03:51:08.537328+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:41.046821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786620+0000","last_change":"2024-09-17T03:51:09.543641+0000","last_active":"2024-09-17T03:51:10.786620+0000","last_peered":"2024-09-17T03:51:10.786620+0000","last_clean":"2024-09-17T03:51:10.786620+0000","last_became_active":"2024-09-17T03:51:09.543299+0000","last_became_peered":"2024-09-17T03:51:09.543299+0000","last_unstale":"2024-09-17T03:51:10.786620+0000","last_undegraded":"2024-09-17T03:51:10.786620+0000","last_fullsized":"2024-09-17T03:51:10.786620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:34.478969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534447+0000","last_change":"2024-09-17T03:51:08.535613+0000","last_active":"2024-09-17T03:51:10.534447+0000","last_peered":"2024-09-17T03:51:10.534447+0000","last_clean":"2024-09-17T03:51:10.534447+0000","last_became_active":"2024-09-17T03:51:08.535214+0000","last_became_peered":"2024-09-17T03:51:08.535214+0000","last_unstale":"2024-09-17T03:51:10.534447+0000","last_undegraded":"2024-09-17T03:51:10.534447+0000","last_fullsized":"2024-09-17T03:51:10.534447+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.931699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785982+0000","last_change":"2024-09-17T03:51:09.552322+0000","last_active":"2024-09-17T03:51:10.785982+0000","last_peered":"2024-09-17T03:51:10.785982+0000","last_clean":"2024-09-17T03:51:10.785982+0000","last_became_active":"2024-09-17T03:51:09.551914+0000","last_became_peered":"2024-09-17T03:51:09.551914+0000","last_unstale":"2024-09-17T03:51:10.785982+0000","last_undegraded":"2024-09-17T03:51:10.785982+0000","last_fullsized":"2024-09-17T03:51:10.785982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:38.111592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536383+0000","last_change":"2024-09-17T03:51:08.532902+0000","last_active":"2024-09-17T03:51:08.536383+0000","last_peered":"2024-09-17T03:51:08.536383+0000","last_clean":"2024-09-17T03:51:08.536383+0000","last_became_active":"2024-09-17T03:51:08.532401+0000","last_became_peered":"2024-09-17T03:51:08.532401+0000","last_unstale":"2024-09-17T03:51:08.536383+0000","last_undegraded":"2024-09-17T03:51:08.536383+0000","last_fullsized":"2024-09-17T03:51:08.536383+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786515+0000","last_change":"2024-09-17T03:51:09.541584+0000","last_active":"2024-09-17T03:51:10.786515+0000","last_peered":"2024-09-17T03:51:10.786515+0000","last_clean":"2024-09-17T03:51:10.786515+0000","last_became_active":"2024-09-17T03:51:09.541219+0000","last_became_peered":"2024-09-17T03:51:09.541219+0000","last_unstale":"2024-09-17T03:51:10.786515+0000","last_undegraded":"2024-09-17T03:51:10.786515+0000","last_fullsized":"2024-09-17T03:51:10.786515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:57.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785991+0000","last_change":"2024-09-17T03:51:08.522934+0000","last_active":"2024-09-17T03:51:10.785991+0000","last_peered":"2024-09-17T03:51:10.785991+0000","last_clean":"2024-09-17T03:51:10.785991+0000","last_became_active":"2024-09-17T03:51:08.522647+0000","last_became_peered":"2024-09-17T03:51:08.522647+0000","last_unstale":"2024-09-17T03:51:10.785991+0000","last_undegraded":"2024-09-17T03:51:10.785991+0000","last_fullsized":"2024-09-17T03:51:10.785991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:27.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532822+0000","last_change":"2024-09-17T03:51:09.550191+0000","last_active":"2024-09-17T03:51:10.532822+0000","last_peered":"2024-09-17T03:51:10.532822+0000","last_clean":"2024-09-17T03:51:10.532822+0000","last_became_active":"2024-09-17T03:51:09.548578+0000","last_became_peered":"2024-09-17T03:51:09.548578+0000","last_unstale":"2024-09-17T03:51:10.532822+0000","last_undegraded":"2024-09-17T03:51:10.532822+0000","last_fullsized":"2024-09-17T03:51:10.532822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:01.627905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533705+0000","last_change":"2024-09-17T03:51:08.535319+0000","last_active":"2024-09-17T03:51:10.533705+0000","last_peered":"2024-09-17T03:51:10.533705+0000","last_clean":"2024-09-17T03:51:10.533705+0000","last_became_active":"2024-09-17T03:51:08.534878+0000","last_became_peered":"2024-09-17T03:51:08.534878+0000","last_unstale":"2024-09-17T03:51:10.533705+0000","last_undegraded":"2024-09-17T03:51:10.533705+0000","last_fullsized":"2024-09-17T03:51:10.533705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517437+0000","last_change":"2024-09-17T03:51:08.517437+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517437+0000","last_undegraded":"2024-09-17T03:51:08.517437+0000","last_fullsized":"2024-09-17T03:51:08.517437+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787744+0000","last_change":"2024-09-17T03:51:08.538705+0000","last_active":"2024-09-17T03:51:10.787744+0000","last_peered":"2024-09-17T03:51:10.787744+0000","last_clean":"2024-09-17T03:51:10.787744+0000","last_became_active":"2024-09-17T03:51:08.535730+0000","last_became_peered":"2024-09-17T03:51:08.535730+0000","last_unstale":"2024-09-17T03:51:10.787744+0000","last_undegraded":"2024-09-17T03:51:10.787744+0000","last_fullsized":"2024-09-17T03:51:10.787744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:47.222685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518389+0000","last_change":"2024-09-17T03:51:08.518389+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518389+0000","last_undegraded":"2024-09-17T03:51:08.518389+0000","last_fullsized":"2024-09-17T03:51:08.518389+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542851+0000","last_change":"2024-09-17T03:51:08.541002+0000","last_active":"2024-09-17T03:51:08.542851+0000","last_peered":"2024-09-17T03:51:08.542851+0000","last_clean":"2024-09-17T03:51:08.542851+0000","last_became_active":"2024-09-17T03:51:08.540686+0000","last_became_peered":"2024-09-17T03:51:08.540686+0000","last_unstale":"2024-09-17T03:51:08.542851+0000","last_undegraded":"2024-09-17T03:51:08.542851+0000","last_fullsized":"2024-09-17T03:51:08.542851+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:20.604244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.518861+0000","last_change":"2024-09-17T03:51:08.518861+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.518861+0000","last_undegraded":"2024-09-17T03:51:08.518861+0000","last_fullsized":"2024-09-17T03:51:08.518861+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":21,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786793+0000","last_change":"2024-09-17T03:51:08.538266+0000","last_active":"2024-09-17T03:51:10.786793+0000","last_peered":"2024-09-17T03:51:10.786793+0000","last_clean":"2024-09-17T03:51:10.786793+0000","last_became_active":"2024-09-17T03:51:08.537932+0000","last_became_peered":"2024-09-17T03:51:08.537932+0000","last_unstale":"2024-09-17T03:51:10.786793+0000","last_undegraded":"2024-09-17T03:51:10.786793+0000","last_fullsized":"2024-09-17T03:51:10.786793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:07.138652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786834+0000","last_change":"2024-09-17T03:51:09.552398+0000","last_active":"2024-09-17T03:51:10.786834+0000","last_peered":"2024-09-17T03:51:10.786834+0000","last_clean":"2024-09-17T03:51:10.786834+0000","last_became_active":"2024-09-17T03:51:09.551972+0000","last_became_peered":"2024-09-17T03:51:09.551972+0000","last_unstale":"2024-09-17T03:51:10.786834+0000","last_undegraded":"2024-09-17T03:51:10.786834+0000","last_fullsized":"2024-09-17T03:51:10.786834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:21.780774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.533653+0000","last_change":"2024-09-17T03:51:08.528682+0000","last_active":"2024-09-17T03:51:08.533653+0000","last_peered":"2024-09-17T03:51:08.533653+0000","last_clean":"2024-09-17T03:51:08.533653+0000","last_became_active":"2024-09-17T03:51:08.527975+0000","last_became_peered":"2024-09-17T03:51:08.527975+0000","last_unstale":"2024-09-17T03:51:08.533653+0000","last_undegraded":"2024-09-17T03:51:08.533653+0000","last_fullsized":"2024-09-17T03:51:08.533653+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786848+0000","last_change":"2024-09-17T03:51:09.549934+0000","last_active":"2024-09-17T03:51:10.786848+0000","last_peered":"2024-09-17T03:51:10.786848+0000","last_clean":"2024-09-17T03:51:10.786848+0000","last_became_active":"2024-09-17T03:51:09.547467+0000","last_became_peered":"2024-09-17T03:51:09.547467+0000","last_unstale":"2024-09-17T03:51:10.786848+0000","last_undegraded":"2024-09-17T03:51:10.786848+0000","last_fullsized":"2024-09-17T03:51:10.786848+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:40.182958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541590+0000","last_change":"2024-09-17T03:51:08.538721+0000","last_active":"2024-09-17T03:51:08.541590+0000","last_peered":"2024-09-17T03:51:08.541590+0000","last_clean":"2024-09-17T03:51:08.541590+0000","last_became_active":"2024-09-17T03:51:08.537978+0000","last_became_peered":"2024-09-17T03:51:08.537978+0000","last_unstale":"2024-09-17T03:51:08.541590+0000","last_undegraded":"2024-09-17T03:51:08.541590+0000","last_fullsized":"2024-09-17T03:51:08.541590+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520408+0000","last_change":"2024-09-17T03:51:08.520408+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520408+0000","last_undegraded":"2024-09-17T03:51:08.520408+0000","last_fullsized":"2024-09-17T03:51:08.520408+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539559+0000","last_change":"2024-09-17T03:51:08.535323+0000","last_active":"2024-09-17T03:51:08.539559+0000","last_peered":"2024-09-17T03:51:08.539559+0000","last_clean":"2024-09-17T03:51:08.539559+0000","last_became_active":"2024-09-17T03:51:08.534999+0000","last_became_peered":"2024-09-17T03:51:08.534999+0000","last_unstale":"2024-09-17T03:51:08.539559+0000","last_undegraded":"2024-09-17T03:51:08.539559+0000","last_fullsized":"2024-09-17T03:51:08.539559+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:04.103238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":23,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786053+0000","last_change":"2024-09-17T03:51:08.530197+0000","last_active":"2024-09-17T03:51:10.786053+0000","last_peered":"2024-09-17T03:51:10.786053+0000","last_clean":"2024-09-17T03:51:10.786053+0000","last_became_active":"2024-09-17T03:51:08.529723+0000","last_became_peered":"2024-09-17T03:51:08.529723+0000","last_unstale":"2024-09-17T03:51:10.786053+0000","last_undegraded":"2024-09-17T03:51:10.786053+0000","last_fullsized":"2024-09-17T03:51:10.786053+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:24.054211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786825+0000","last_change":"2024-09-17T03:51:09.537261+0000","last_active":"2024-09-17T03:51:10.786825+0000","last_peered":"2024-09-17T03:51:10.786825+0000","last_clean":"2024-09-17T03:51:10.786825+0000","last_became_active":"2024-09-17T03:51:09.536865+0000","last_became_peered":"2024-09-17T03:51:09.536865+0000","last_unstale":"2024-09-17T03:51:10.786825+0000","last_undegraded":"2024-09-17T03:51:10.786825+0000","last_fullsized":"2024-09-17T03:51:10.786825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:02.803728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786694+0000","last_change":"2024-09-17T03:51:08.535330+0000","last_active":"2024-09-17T03:51:10.786694+0000","last_peered":"2024-09-17T03:51:10.786694+0000","last_clean":"2024-09-17T03:51:10.786694+0000","last_became_active":"2024-09-17T03:51:08.535035+0000","last_became_peered":"2024-09-17T03:51:08.535035+0000","last_unstale":"2024-09-17T03:51:10.786694+0000","last_undegraded":"2024-09-17T03:51:10.786694+0000","last_fullsized":"2024-09-17T03:51:10.786694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:25.251556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787536+0000","last_change":"2024-09-17T03:51:09.559584+0000","last_active":"2024-09-17T03:51:10.787536+0000","last_peered":"2024-09-17T03:51:10.787536+0000","last_clean":"2024-09-17T03:51:10.787536+0000","last_became_active":"2024-09-17T03:51:09.559215+0000","last_became_peered":"2024-09-17T03:51:09.559215+0000","last_unstale":"2024-09-17T03:51:10.787536+0000","last_undegraded":"2024-09-17T03:51:10.787536+0000","last_fullsized":"2024-09-17T03:51:10.787536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:48.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533214+0000","last_change":"2024-09-17T03:51:08.529671+0000","last_active":"2024-09-17T03:51:10.533214+0000","last_peered":"2024-09-17T03:51:10.533214+0000","last_clean":"2024-09-17T03:51:10.533214+0000","last_became_active":"2024-09-17T03:51:08.529313+0000","last_became_peered":"2024-09-17T03:51:08.529313+0000","last_unstale":"2024-09-17T03:51:10.533214+0000","last_undegraded":"2024-09-17T03:51:10.533214+0000","last_fullsized":"2024-09-17T03:51:10.533214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787006+0000","last_change":"2024-09-17T03:51:09.531351+0000","last_active":"2024-09-17T03:51:10.787006+0000","last_peered":"2024-09-17T03:51:10.787006+0000","last_clean":"2024-09-17T03:51:10.787006+0000","last_became_active":"2024-09-17T03:51:09.530270+0000","last_became_peered":"2024-09-17T03:51:09.530270+0000","last_unstale":"2024-09-17T03:51:10.787006+0000","last_undegraded":"2024-09-17T03:51:10.787006+0000","last_fullsized":"2024-09-17T03:51:10.787006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:38:05.287191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535766+0000","last_change":"2024-09-17T03:51:08.531273+0000","last_active":"2024-09-17T03:51:08.535766+0000","last_peered":"2024-09-17T03:51:08.535766+0000","last_clean":"2024-09-17T03:51:08.535766+0000","last_became_active":"2024-09-17T03:51:08.530933+0000","last_became_peered":"2024-09-17T03:51:08.530933+0000","last_unstale":"2024-09-17T03:51:08.535766+0000","last_undegraded":"2024-09-17T03:51:08.535766+0000","last_fullsized":"2024-09-17T03:51:08.535766+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787095+0000","last_change":"2024-09-17T03:51:09.545503+0000","last_active":"2024-09-17T03:51:10.787095+0000","last_peered":"2024-09-17T03:51:10.787095+0000","last_clean":"2024-09-17T03:51:10.787095+0000","last_became_active":"2024-09-17T03:51:09.544721+0000","last_became_peered":"2024-09-17T03:51:09.544721+0000","last_unstale":"2024-09-17T03:51:10.787095+0000","last_undegraded":"2024-09-17T03:51:10.787095+0000","last_fullsized":"2024-09-17T03:51:10.787095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:10.003938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.539491+0000","last_change":"2024-09-17T03:51:08.534140+0000","last_active":"2024-09-17T03:51:08.539491+0000","last_peered":"2024-09-17T03:51:08.539491+0000","last_clean":"2024-09-17T03:51:08.539491+0000","last_became_active":"2024-09-17T03:51:08.533878+0000","last_became_peered":"2024-09-17T03:51:08.533878+0000","last_unstale":"2024-09-17T03:51:08.539491+0000","last_undegraded":"2024-09-17T03:51:08.539491+0000","last_fullsized":"2024-09-17T03:51:08.539491+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:18.671953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.521195+0000","last_change":"2024-09-17T03:51:08.521195+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.521195+0000","last_undegraded":"2024-09-17T03:51:08.521195+0000","last_fullsized":"2024-09-17T03:51:08.521195+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537539+0000","last_change":"2024-09-17T03:51:08.533378+0000","last_active":"2024-09-17T03:51:08.537539+0000","last_peered":"2024-09-17T03:51:08.537539+0000","last_clean":"2024-09-17T03:51:08.537539+0000","last_became_active":"2024-09-17T03:51:08.532682+0000","last_became_peered":"2024-09-17T03:51:08.532682+0000","last_unstale":"2024-09-17T03:51:08.537539+0000","last_undegraded":"2024-09-17T03:51:08.537539+0000","last_fullsized":"2024-09-17T03:51:08.537539+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.184648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.514113+0000","last_change":"2024-09-17T03:51:08.514113+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.514113+0000","last_undegraded":"2024-09-17T03:51:08.514113+0000","last_fullsized":"2024-09-17T03:51:08.514113+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535852+0000","last_change":"2024-09-17T03:51:08.531986+0000","last_active":"2024-09-17T03:51:08.535852+0000","last_peered":"2024-09-17T03:51:08.535852+0000","last_clean":"2024-09-17T03:51:08.535852+0000","last_became_active":"2024-09-17T03:51:08.531674+0000","last_became_peered":"2024-09-17T03:51:08.531674+0000","last_unstale":"2024-09-17T03:51:08.535852+0000","last_undegraded":"2024-09-17T03:51:08.535852+0000","last_fullsized":"2024-09-17T03:51:08.535852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785978+0000","last_change":"2024-09-17T03:51:09.538946+0000","last_active":"2024-09-17T03:51:10.785978+0000","last_peered":"2024-09-17T03:51:10.785978+0000","last_clean":"2024-09-17T03:51:10.785978+0000","last_became_active":"2024-09-17T03:51:09.538617+0000","last_became_peered":"2024-09-17T03:51:09.538617+0000","last_unstale":"2024-09-17T03:51:10.785978+0000","last_undegraded":"2024-09-17T03:51:10.785978+0000","last_fullsized":"2024-09-17T03:51:10.785978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:46.014539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786247+0000","last_change":"2024-09-17T03:51:08.526124+0000","last_active":"2024-09-17T03:51:10.786247+0000","last_peered":"2024-09-17T03:51:10.786247+0000","last_clean":"2024-09-17T03:51:10.786247+0000","last_became_active":"2024-09-17T03:51:08.525837+0000","last_became_peered":"2024-09-17T03:51:08.525837+0000","last_unstale":"2024-09-17T03:51:10.786247+0000","last_undegraded":"2024-09-17T03:51:10.786247+0000","last_fullsized":"2024-09-17T03:51:10.786247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:09.038592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787015+0000","last_change":"2024-09-17T03:51:09.556563+0000","last_active":"2024-09-17T03:51:10.787015+0000","last_peered":"2024-09-17T03:51:10.787015+0000","last_clean":"2024-09-17T03:51:10.787015+0000","last_became_active":"2024-09-17T03:51:09.555716+0000","last_became_peered":"2024-09-17T03:51:09.555716+0000","last_unstale":"2024-09-17T03:51:10.787015+0000","last_undegraded":"2024-09-17T03:51:10.787015+0000","last_fullsized":"2024-09-17T03:51:10.787015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:30.216865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.541521+0000","last_change":"2024-09-17T03:51:08.535527+0000","last_active":"2024-09-17T03:51:08.541521+0000","last_peered":"2024-09-17T03:51:08.541521+0000","last_clean":"2024-09-17T03:51:08.541521+0000","last_became_active":"2024-09-17T03:51:08.535182+0000","last_became_peered":"2024-09-17T03:51:08.535182+0000","last_unstale":"2024-09-17T03:51:08.541521+0000","last_undegraded":"2024-09-17T03:51:08.541521+0000","last_fullsized":"2024-09-17T03:51:08.541521+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787369+0000","last_change":"2024-09-17T03:51:09.552798+0000","last_active":"2024-09-17T03:51:10.787369+0000","last_peered":"2024-09-17T03:51:10.787369+0000","last_clean":"2024-09-17T03:51:10.787369+0000","last_became_active":"2024-09-17T03:51:09.552477+0000","last_became_peered":"2024-09-17T03:51:09.552477+0000","last_unstale":"2024-09-17T03:51:10.787369+0000","last_undegraded":"2024-09-17T03:51:10.787369+0000","last_fullsized":"2024-09-17T03:51:10.787369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:35.740581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.537417+0000","last_change":"2024-09-17T03:51:08.531555+0000","last_active":"2024-09-17T03:51:08.537417+0000","last_peered":"2024-09-17T03:51:08.537417+0000","last_clean":"2024-09-17T03:51:08.537417+0000","last_became_active":"2024-09-17T03:51:08.531230+0000","last_became_peered":"2024-09-17T03:51:08.531230+0000","last_unstale":"2024-09-17T03:51:08.537417+0000","last_undegraded":"2024-09-17T03:51:08.537417+0000","last_fullsized":"2024-09-17T03:51:08.537417+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:08.023937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787381+0000","last_change":"2024-09-17T03:51:09.558633+0000","last_active":"2024-09-17T03:51:10.787381+0000","last_peered":"2024-09-17T03:51:10.787381+0000","last_clean":"2024-09-17T03:51:10.787381+0000","last_became_active":"2024-09-17T03:51:09.558301+0000","last_became_peered":"2024-09-17T03:51:09.558301+0000","last_unstale":"2024-09-17T03:51:10.787381+0000","last_undegraded":"2024-09-17T03:51:10.787381+0000","last_fullsized":"2024-09-17T03:51:10.787381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:53.303687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.543358+0000","last_change":"2024-09-17T03:51:08.542303+0000","last_active":"2024-09-17T03:51:08.543358+0000","last_peered":"2024-09-17T03:51:08.543358+0000","last_clean":"2024-09-17T03:51:08.543358+0000","last_became_active":"2024-09-17T03:51:08.541884+0000","last_became_peered":"2024-09-17T03:51:08.541884+0000","last_unstale":"2024-09-17T03:51:08.543358+0000","last_undegraded":"2024-09-17T03:51:08.543358+0000","last_fullsized":"2024-09-17T03:51:08.543358+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:23.838696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.517359+0000","last_change":"2024-09-17T03:51:08.517359+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.517359+0000","last_undegraded":"2024-09-17T03:51:08.517359+0000","last_fullsized":"2024-09-17T03:51:08.517359+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536023+0000","last_change":"2024-09-17T03:51:08.533982+0000","last_active":"2024-09-17T03:51:08.536023+0000","last_peered":"2024-09-17T03:51:08.536023+0000","last_clean":"2024-09-17T03:51:08.536023+0000","last_became_active":"2024-09-17T03:51:08.529828+0000","last_became_peered":"2024-09-17T03:51:08.529828+0000","last_unstale":"2024-09-17T03:51:08.536023+0000","last_undegraded":"2024-09-17T03:51:08.536023+0000","last_fullsized":"2024-09-17T03:51:08.536023+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:18.636858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520810+0000","last_change":"2024-09-17T03:51:08.520810+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520810+0000","last_undegraded":"2024-09-17T03:51:08.520810+0000","last_fullsized":"2024-09-17T03:51:08.520810+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.542733+0000","last_change":"2024-09-17T03:51:08.540515+0000","last_active":"2024-09-17T03:51:08.542733+0000","last_peered":"2024-09-17T03:51:08.542733+0000","last_clean":"2024-09-17T03:51:08.542733+0000","last_became_active":"2024-09-17T03:51:08.540197+0000","last_became_peered":"2024-09-17T03:51:08.540197+0000","last_unstale":"2024-09-17T03:51:08.542733+0000","last_undegraded":"2024-09-17T03:51:08.542733+0000","last_fullsized":"2024-09-17T03:51:08.542733+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:54.825765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520049+0000","last_change":"2024-09-17T03:51:08.520049+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520049+0000","last_undegraded":"2024-09-17T03:51:08.520049+0000","last_fullsized":"2024-09-17T03:51:08.520049+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534533+0000","last_change":"2024-09-17T03:51:08.538899+0000","last_active":"2024-09-17T03:51:10.534533+0000","last_peered":"2024-09-17T03:51:10.534533+0000","last_clean":"2024-09-17T03:51:10.534533+0000","last_became_active":"2024-09-17T03:51:08.538338+0000","last_became_peered":"2024-09-17T03:51:08.538338+0000","last_unstale":"2024-09-17T03:51:10.534533+0000","last_undegraded":"2024-09-17T03:51:10.534533+0000","last_fullsized":"2024-09-17T03:51:10.534533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:56.151879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":1,"reported_epoch":43,"state":"creating+peering","last_fresh":"2024-09-17T03:51:08.520013+0000","last_change":"2024-09-17T03:51:08.520013+0000","last_active":"2024-09-17T03:51:08.499499+0000","last_peered":"2024-09-17T03:51:08.499499+0000","last_clean":"2024-09-17T03:51:08.499499+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:51:08.520013+0000","last_undegraded":"2024-09-17T03:51:08.520013+0000","last_fullsized":"2024-09-17T03:51:08.520013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.536701+0000","last_change":"2024-09-17T03:51:08.535001+0000","last_active":"2024-09-17T03:51:08.536701+0000","last_peered":"2024-09-17T03:51:08.536701+0000","last_clean":"2024-09-17T03:51:08.536701+0000","last_became_active":"2024-09-17T03:51:08.531914+0000","last_became_peered":"2024-09-17T03:51:08.531914+0000","last_unstale":"2024-09-17T03:51:08.536701+0000","last_undegraded":"2024-09-17T03:51:08.536701+0000","last_fullsized":"2024-09-17T03:51:08.536701+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:10.120644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532927+0000","last_change":"2024-09-17T03:51:09.540447+0000","last_active":"2024-09-17T03:51:10.532927+0000","last_peered":"2024-09-17T03:51:10.532927+0000","last_clean":"2024-09-17T03:51:10.532927+0000","last_became_active":"2024-09-17T03:51:09.540161+0000","last_became_peered":"2024-09-17T03:51:09.540161+0000","last_unstale":"2024-09-17T03:51:10.532927+0000","last_undegraded":"2024-09-17T03:51:10.532927+0000","last_fullsized":"2024-09-17T03:51:10.532927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:04.256313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786105+0000","last_change":"2024-09-17T03:51:08.525657+0000","last_active":"2024-09-17T03:51:10.786105+0000","last_peered":"2024-09-17T03:51:10.786105+0000","last_clean":"2024-09-17T03:51:10.786105+0000","last_became_active":"2024-09-17T03:51:08.525271+0000","last_became_peered":"2024-09-17T03:51:08.525271+0000","last_unstale":"2024-09-17T03:51:10.786105+0000","last_undegraded":"2024-09-17T03:51:10.786105+0000","last_fullsized":"2024-09-17T03:51:10.786105+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:55.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787764+0000","last_change":"2024-09-17T03:51:09.544994+0000","last_active":"2024-09-17T03:51:10.787764+0000","last_peered":"2024-09-17T03:51:10.787764+0000","last_clean":"2024-09-17T03:51:10.787764+0000","last_became_active":"2024-09-17T03:51:09.544748+0000","last_became_peered":"2024-09-17T03:51:09.544748+0000","last_unstale":"2024-09-17T03:51:10.787764+0000","last_undegraded":"2024-09-17T03:51:10.787764+0000","last_fullsized":"2024-09-17T03:51:10.787764+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:14.819738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.535595+0000","last_change":"2024-09-17T03:51:08.529407+0000","last_active":"2024-09-17T03:51:08.535595+0000","last_peered":"2024-09-17T03:51:08.535595+0000","last_clean":"2024-09-17T03:51:08.535595+0000","last_became_active":"2024-09-17T03:51:08.529046+0000","last_became_peered":"2024-09-17T03:51:08.529046+0000","last_unstale":"2024-09-17T03:51:08.535595+0000","last_undegraded":"2024-09-17T03:51:08.535595+0000","last_fullsized":"2024-09-17T03:51:08.535595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":148,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-17T03:51:08.524741+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:08.524741+0000","last_peered":"2024-09-17T03:51:08.524741+0000","last_clean":"2024-09-17T03:51:08.524741+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:08.524741+0000","last_undegraded":"2024-09-17T03:51:08.524741+0000","last_fullsized":"2024-09-17T03:51:08.524741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":1634,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509463,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1089536,"data_stored":786166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509466,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":210485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509463,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":536576,"data_stored":210507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509465,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":495616,"data_stored":195798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542168,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":209545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542169,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":799275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:19.146 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:19.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 -- 172.21.15.129:0/1825267614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d4004d40 msgr2=0x7f14d4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 --2- 172.21.15.129:0/1825267614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14d4004d40 0x7f14d4005190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f14e800c260 tx=0x7f14d4005830 comp rx=0 tx=0).stop 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f4a00640 1 -- 172.21.15.129:0/1825267614 reap_dead start 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 -- 172.21.15.129:0/1825267614 shutdown_connections 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 -- 172.21.15.129:0/1825267614 >> 172.21.15.129:0/1825267614 conn(0x7f14f00b1cb0 msgr2=0x7f14f00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 -- 172.21.15.129:0/1825267614 shutdown_connections 2024-09-17T03:51:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.382+0000 7f14f6a00640 1 -- 172.21.15.129:0/1825267614 wait complete. 2024-09-17T03:51:19.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.385+0000 7f14f6a00640 1 Processor -- start 2024-09-17T03:51:19.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.385+0000 7f14f6a00640 1 -- start start 2024-09-17T03:51:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f00cb1e0 0x7f14f01526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14f01f95c0 0x7f14f00c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14f00b37a0 con 0x7f14f009db30 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14f0075040 con 0x7f14f00cb1e0 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14f6a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14f00b3920 con 0x7f14f01f95c0 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39460/0 (socket says 172.21.15.129:39460) 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.386+0000 7f14ef400640 1 -- 172.21.15.129:0/3262695501 learned_addr learned my addr 172.21.15.129:0/3262695501 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14efe00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f00cb1e0 0x7f14f01526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14f4a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14f01f95c0 0x7f14f00c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 -- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14f01f95c0 msgr2=0x7f14f00c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14f01f95c0 0x7f14f00c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 -- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f00cb1e0 msgr2=0x7f14f01526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f00cb1e0 0x7f14f01526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 -- 172.21.15.129:0/3262695501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14f01f99d0 con 0x7f14f009db30 2024-09-17T03:51:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14ef400640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0118b0 tx=0x7f14dc011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc00f9a0 con 0x7f14f009db30 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14dc00fb40 con 0x7f14f009db30 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.387+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc005020 con 0x7f14f009db30 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.388+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14f00cb750 con 0x7f14f009db30 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.388+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14f00cbfd0 con 0x7f14f009db30 2024-09-17T03:51:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.389+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14f00b3e80 con 0x7f14f009db30 2024-09-17T03:51:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.390+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14dc0051c0 con 0x7f14f009db30 2024-09-17T03:51:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.391+0000 7f14e7e00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 0x7f14d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.391+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f14dc09b870 con 0x7f14f009db30 2024-09-17T03:51:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.392+0000 7f14efe00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 0x7f14d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.392+0000 7f14efe00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 0x7f14d007b0a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14f00c9ba0 tx=0x7f14e0006e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:19.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.395+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14f01f99d0 con 0x7f14f009db30 2024-09-17T03:51:19.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.578+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14f00c3860 con 0x7f14d0078b90 2024-09-17T03:51:19.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.583+0000 7f14e7e00640 1 -- 172.21.15.129:0/3262695501 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251924 (secure 0 0 0) 0x7f14f00c3860 con 0x7f14d0078b90 2024-09-17T03:51:19.584 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:19.585 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.586+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 msgr2=0x7f14d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.586+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 0x7f14d007b0a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f14f00c9ba0 tx=0x7f14e0006e90 comp rx=0 tx=0).stop 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.586+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 msgr2=0x7f14f01f9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.586+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0118b0 tx=0x7f14dc011d70 comp rx=0 tx=0).stop 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 shutdown_connections 2024-09-17T03:51:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14d0078b90 0x7f14d007b0a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14f01f95c0 0x7f14f00c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f009db30 0x7f14f01f9080 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 --2- 172.21.15.129:0/3262695501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f00cb1e0 0x7f14f01526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 >> 172.21.15.129:0/3262695501 conn(0x7f14f00b1cb0 msgr2=0x7f14f00ce260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 shutdown_connections 2024-09-17T03:51:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.587+0000 7f14f6a00640 1 -- 172.21.15.129:0/3262695501 wait complete. 2024-09-17T03:51:19.607 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-17T03:51:18.320042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254832,"kb_used_data":5288,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719888,"statfs":{"total":773094113280,"available":772833165312,"internally_reserved":0,"allocated":5414912,"data_stored":2856540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.807529"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722726+0000","last_change":"2024-09-17T03:51:09.550741+0000","last_active":"2024-09-17T03:51:10.722726+0000","last_peered":"2024-09-17T03:51:10.722726+0000","last_clean":"2024-09-17T03:51:10.722726+0000","last_became_active":"2024-09-17T03:51:09.550412+0000","last_became_peered":"2024-09-17T03:51:09.550412+0000","last_unstale":"2024-09-17T03:51:10.722726+0000","last_undegraded":"2024-09-17T03:51:10.722726+0000","last_fullsized":"2024-09-17T03:51:10.722726+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:35:09.661727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532805+0000","last_change":"2024-09-17T03:51:09.548800+0000","last_active":"2024-09-17T03:51:10.532805+0000","last_peered":"2024-09-17T03:51:10.532805+0000","last_clean":"2024-09-17T03:51:10.532805+0000","last_became_active":"2024-09-17T03:51:09.548414+0000","last_became_peered":"2024-09-17T03:51:09.548414+0000","last_unstale":"2024-09-17T03:51:10.532805+0000","last_undegraded":"2024-09-17T03:51:10.532805+0000","last_fullsized":"2024-09-17T03:51:10.532805+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:51.066867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532794+0000","last_change":"2024-09-17T03:51:09.544780+0000","last_active":"2024-09-17T03:51:10.532794+0000","last_peered":"2024-09-17T03:51:10.532794+0000","last_clean":"2024-09-17T03:51:10.532794+0000","last_became_active":"2024-09-17T03:51:09.544462+0000","last_became_peered":"2024-09-17T03:51:09.544462+0000","last_unstale":"2024-09-17T03:51:10.532794+0000","last_undegraded":"2024-09-17T03:51:10.532794+0000","last_fullsized":"2024-09-17T03:51:10.532794+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:20.077167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787357+0000","last_change":"2024-09-17T03:51:09.553310+0000","last_active":"2024-09-17T03:51:10.787357+0000","last_peered":"2024-09-17T03:51:10.787357+0000","last_clean":"2024-09-17T03:51:10.787357+0000","last_became_active":"2024-09-17T03:51:09.552339+0000","last_became_peered":"2024-09-17T03:51:09.552339+0000","last_unstale":"2024-09-17T03:51:10.787357+0000","last_undegraded":"2024-09-17T03:51:10.787357+0000","last_fullsized":"2024-09-17T03:51:10.787357+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:59.902480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532471+0000","last_change":"2024-09-17T03:51:09.543169+0000","last_active":"2024-09-17T03:51:10.532471+0000","last_peered":"2024-09-17T03:51:10.532471+0000","last_clean":"2024-09-17T03:51:10.532471+0000","last_became_active":"2024-09-17T03:51:09.542844+0000","last_became_peered":"2024-09-17T03:51:09.542844+0000","last_unstale":"2024-09-17T03:51:10.532471+0000","last_undegraded":"2024-09-17T03:51:10.532471+0000","last_fullsized":"2024-09-17T03:51:10.532471+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:39.142664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531153+0000","last_change":"2024-09-17T03:51:09.554685+0000","last_active":"2024-09-17T03:51:10.531153+0000","last_peered":"2024-09-17T03:51:10.531153+0000","last_clean":"2024-09-17T03:51:10.531153+0000","last_became_active":"2024-09-17T03:51:09.554063+0000","last_became_peered":"2024-09-17T03:51:09.554063+0000","last_unstale":"2024-09-17T03:51:10.531153+0000","last_undegraded":"2024-09-17T03:51:10.531153+0000","last_fullsized":"2024-09-17T03:51:10.531153+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:16.754480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722587+0000","last_change":"2024-09-17T03:51:09.544449+0000","last_active":"2024-09-17T03:51:10.722587+0000","last_peered":"2024-09-17T03:51:10.722587+0000","last_clean":"2024-09-17T03:51:10.722587+0000","last_became_active":"2024-09-17T03:51:09.543937+0000","last_became_peered":"2024-09-17T03:51:09.543937+0000","last_unstale":"2024-09-17T03:51:10.722587+0000","last_undegraded":"2024-09-17T03:51:10.722587+0000","last_fullsized":"2024-09-17T03:51:10.722587+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:52.783547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785916+0000","last_change":"2024-09-17T03:51:09.553571+0000","last_active":"2024-09-17T03:51:10.785916+0000","last_peered":"2024-09-17T03:51:10.785916+0000","last_clean":"2024-09-17T03:51:10.785916+0000","last_became_active":"2024-09-17T03:51:09.553226+0000","last_became_peered":"2024-09-17T03:51:09.553226+0000","last_unstale":"2024-09-17T03:51:10.785916+0000","last_undegraded":"2024-09-17T03:51:10.785916+0000","last_fullsized":"2024-09-17T03:51:10.785916+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:13.896356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787450+0000","last_change":"2024-09-17T03:51:09.558849+0000","last_active":"2024-09-17T03:51:10.787450+0000","last_peered":"2024-09-17T03:51:10.787450+0000","last_clean":"2024-09-17T03:51:10.787450+0000","last_became_active":"2024-09-17T03:51:09.558596+0000","last_became_peered":"2024-09-17T03:51:09.558596+0000","last_unstale":"2024-09-17T03:51:10.787450+0000","last_undegraded":"2024-09-17T03:51:10.787450+0000","last_fullsized":"2024-09-17T03:51:10.787450+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:19.302750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787683+0000","last_change":"2024-09-17T03:51:09.552932+0000","last_active":"2024-09-17T03:51:10.787683+0000","last_peered":"2024-09-17T03:51:10.787683+0000","last_clean":"2024-09-17T03:51:10.787683+0000","last_became_active":"2024-09-17T03:51:09.552660+0000","last_became_peered":"2024-09-17T03:51:09.552660+0000","last_unstale":"2024-09-17T03:51:10.787683+0000","last_undegraded":"2024-09-17T03:51:10.787683+0000","last_fullsized":"2024-09-17T03:51:10.787683+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:45.040579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787563+0000","last_change":"2024-09-17T03:51:09.556091+0000","last_active":"2024-09-17T03:51:10.787563+0000","last_peered":"2024-09-17T03:51:10.787563+0000","last_clean":"2024-09-17T03:51:10.787563+0000","last_became_active":"2024-09-17T03:51:09.555788+0000","last_became_peered":"2024-09-17T03:51:09.555788+0000","last_unstale":"2024-09-17T03:51:10.787563+0000","last_undegraded":"2024-09-17T03:51:10.787563+0000","last_fullsized":"2024-09-17T03:51:10.787563+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:35.856529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533265+0000","last_change":"2024-09-17T03:51:09.539267+0000","last_active":"2024-09-17T03:51:10.533265+0000","last_peered":"2024-09-17T03:51:10.533265+0000","last_clean":"2024-09-17T03:51:10.533265+0000","last_became_active":"2024-09-17T03:51:09.538365+0000","last_became_peered":"2024-09-17T03:51:09.538365+0000","last_unstale":"2024-09-17T03:51:10.533265+0000","last_undegraded":"2024-09-17T03:51:10.533265+0000","last_fullsized":"2024-09-17T03:51:10.533265+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:02.862989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786586+0000","last_change":"2024-09-17T03:51:09.549923+0000","last_active":"2024-09-17T03:51:10.786586+0000","last_peered":"2024-09-17T03:51:10.786586+0000","last_clean":"2024-09-17T03:51:10.786586+0000","last_became_active":"2024-09-17T03:51:09.549427+0000","last_became_peered":"2024-09-17T03:51:09.549427+0000","last_unstale":"2024-09-17T03:51:10.786586+0000","last_undegraded":"2024-09-17T03:51:10.786586+0000","last_fullsized":"2024-09-17T03:51:10.786586+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:12.588911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786097+0000","last_change":"2024-09-17T03:51:09.557049+0000","last_active":"2024-09-17T03:51:10.786097+0000","last_peered":"2024-09-17T03:51:10.786097+0000","last_clean":"2024-09-17T03:51:10.786097+0000","last_became_active":"2024-09-17T03:51:09.556732+0000","last_became_peered":"2024-09-17T03:51:09.556732+0000","last_unstale":"2024-09-17T03:51:10.786097+0000","last_undegraded":"2024-09-17T03:51:10.786097+0000","last_fullsized":"2024-09-17T03:51:10.786097+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:51.428569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.530967+0000","last_change":"2024-09-17T03:51:09.526468+0000","last_active":"2024-09-17T03:51:10.530967+0000","last_peered":"2024-09-17T03:51:10.530967+0000","last_clean":"2024-09-17T03:51:10.530967+0000","last_became_active":"2024-09-17T03:51:09.525975+0000","last_became_peered":"2024-09-17T03:51:09.525975+0000","last_unstale":"2024-09-17T03:51:10.530967+0000","last_undegraded":"2024-09-17T03:51:10.530967+0000","last_fullsized":"2024-09-17T03:51:10.530967+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:40.802036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786655+0000","last_change":"2024-09-17T03:51:09.552281+0000","last_active":"2024-09-17T03:51:10.786655+0000","last_peered":"2024-09-17T03:51:10.786655+0000","last_clean":"2024-09-17T03:51:10.786655+0000","last_became_active":"2024-09-17T03:51:09.551898+0000","last_became_peered":"2024-09-17T03:51:09.551898+0000","last_unstale":"2024-09-17T03:51:10.786655+0000","last_undegraded":"2024-09-17T03:51:10.786655+0000","last_fullsized":"2024-09-17T03:51:10.786655+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:19.691900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533660+0000","last_change":"2024-09-17T03:51:09.533608+0000","last_active":"2024-09-17T03:51:10.533660+0000","last_peered":"2024-09-17T03:51:10.533660+0000","last_clean":"2024-09-17T03:51:10.533660+0000","last_became_active":"2024-09-17T03:51:09.533263+0000","last_became_peered":"2024-09-17T03:51:09.533263+0000","last_unstale":"2024-09-17T03:51:10.533660+0000","last_undegraded":"2024-09-17T03:51:10.533660+0000","last_fullsized":"2024-09-17T03:51:10.533660+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:18.712043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532553+0000","last_change":"2024-09-17T03:51:09.549893+0000","last_active":"2024-09-17T03:51:10.532553+0000","last_peered":"2024-09-17T03:51:10.532553+0000","last_clean":"2024-09-17T03:51:10.532553+0000","last_became_active":"2024-09-17T03:51:09.548155+0000","last_became_peered":"2024-09-17T03:51:09.548155+0000","last_unstale":"2024-09-17T03:51:10.532553+0000","last_undegraded":"2024-09-17T03:51:10.532553+0000","last_fullsized":"2024-09-17T03:51:10.532553+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:06.020247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787285+0000","last_change":"2024-09-17T03:51:09.545828+0000","last_active":"2024-09-17T03:51:10.787285+0000","last_peered":"2024-09-17T03:51:10.787285+0000","last_clean":"2024-09-17T03:51:10.787285+0000","last_became_active":"2024-09-17T03:51:09.545207+0000","last_became_peered":"2024-09-17T03:51:09.545207+0000","last_unstale":"2024-09-17T03:51:10.787285+0000","last_undegraded":"2024-09-17T03:51:10.787285+0000","last_fullsized":"2024-09-17T03:51:10.787285+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:11.346092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786753+0000","last_change":"2024-09-17T03:51:09.550468+0000","last_active":"2024-09-17T03:51:10.786753+0000","last_peered":"2024-09-17T03:51:10.786753+0000","last_clean":"2024-09-17T03:51:10.786753+0000","last_became_active":"2024-09-17T03:51:09.550210+0000","last_became_peered":"2024-09-17T03:51:09.550210+0000","last_unstale":"2024-09-17T03:51:10.786753+0000","last_undegraded":"2024-09-17T03:51:10.786753+0000","last_fullsized":"2024-09-17T03:51:10.786753+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:55.396574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531290+0000","last_change":"2024-09-17T03:51:09.533682+0000","last_active":"2024-09-17T03:51:10.531290+0000","last_peered":"2024-09-17T03:51:10.531290+0000","last_clean":"2024-09-17T03:51:10.531290+0000","last_became_active":"2024-09-17T03:51:09.533352+0000","last_became_peered":"2024-09-17T03:51:09.533352+0000","last_unstale":"2024-09-17T03:51:10.531290+0000","last_undegraded":"2024-09-17T03:51:10.531290+0000","last_fullsized":"2024-09-17T03:51:10.531290+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:42.524860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786228+0000","last_change":"2024-09-17T03:51:09.551749+0000","last_active":"2024-09-17T03:51:10.786228+0000","last_peered":"2024-09-17T03:51:10.786228+0000","last_clean":"2024-09-17T03:51:10.786228+0000","last_became_active":"2024-09-17T03:51:09.551425+0000","last_became_peered":"2024-09-17T03:51:09.551425+0000","last_unstale":"2024-09-17T03:51:10.786228+0000","last_undegraded":"2024-09-17T03:51:10.786228+0000","last_fullsized":"2024-09-17T03:51:10.786228+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:31.877445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786776+0000","last_change":"2024-09-17T03:51:09.537760+0000","last_active":"2024-09-17T03:51:10.786776+0000","last_peered":"2024-09-17T03:51:10.786776+0000","last_clean":"2024-09-17T03:51:10.786776+0000","last_became_active":"2024-09-17T03:51:09.537430+0000","last_became_peered":"2024-09-17T03:51:09.537430+0000","last_unstale":"2024-09-17T03:51:10.786776+0000","last_undegraded":"2024-09-17T03:51:10.786776+0000","last_fullsized":"2024-09-17T03:51:10.786776+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:40.609175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786706+0000","last_change":"2024-09-17T03:51:09.535683+0000","last_active":"2024-09-17T03:51:10.786706+0000","last_peered":"2024-09-17T03:51:10.786706+0000","last_clean":"2024-09-17T03:51:10.786706+0000","last_became_active":"2024-09-17T03:51:09.535368+0000","last_became_peered":"2024-09-17T03:51:09.535368+0000","last_unstale":"2024-09-17T03:51:10.786706+0000","last_undegraded":"2024-09-17T03:51:10.786706+0000","last_fullsized":"2024-09-17T03:51:10.786706+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:28.792675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787126+0000","last_change":"2024-09-17T03:51:09.531819+0000","last_active":"2024-09-17T03:51:10.787126+0000","last_peered":"2024-09-17T03:51:10.787126+0000","last_clean":"2024-09-17T03:51:10.787126+0000","last_became_active":"2024-09-17T03:51:09.531504+0000","last_became_peered":"2024-09-17T03:51:09.531504+0000","last_unstale":"2024-09-17T03:51:10.787126+0000","last_undegraded":"2024-09-17T03:51:10.787126+0000","last_fullsized":"2024-09-17T03:51:10.787126+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:37.424982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786610+0000","last_change":"2024-09-17T03:51:09.561216+0000","last_active":"2024-09-17T03:51:10.786610+0000","last_peered":"2024-09-17T03:51:10.786610+0000","last_clean":"2024-09-17T03:51:10.786610+0000","last_became_active":"2024-09-17T03:51:09.560930+0000","last_became_peered":"2024-09-17T03:51:09.560930+0000","last_unstale":"2024-09-17T03:51:10.786610+0000","last_undegraded":"2024-09-17T03:51:10.786610+0000","last_fullsized":"2024-09-17T03:51:10.786610+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:24.329326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533087+0000","last_change":"2024-09-17T03:51:09.532459+0000","last_active":"2024-09-17T03:51:10.533087+0000","last_peered":"2024-09-17T03:51:10.533087+0000","last_clean":"2024-09-17T03:51:10.533087+0000","last_became_active":"2024-09-17T03:51:09.532056+0000","last_became_peered":"2024-09-17T03:51:09.532056+0000","last_unstale":"2024-09-17T03:51:10.533087+0000","last_undegraded":"2024-09-17T03:51:10.533087+0000","last_fullsized":"2024-09-17T03:51:10.533087+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:07.247043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533312+0000","last_change":"2024-09-17T03:51:09.541182+0000","last_active":"2024-09-17T03:51:10.533312+0000","last_peered":"2024-09-17T03:51:10.533312+0000","last_clean":"2024-09-17T03:51:10.533312+0000","last_became_active":"2024-09-17T03:51:09.540816+0000","last_became_peered":"2024-09-17T03:51:09.540816+0000","last_unstale":"2024-09-17T03:51:10.533312+0000","last_undegraded":"2024-09-17T03:51:10.533312+0000","last_fullsized":"2024-09-17T03:51:10.533312+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:26.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787120+0000","last_change":"2024-09-17T03:51:09.531111+0000","last_active":"2024-09-17T03:51:10.787120+0000","last_peered":"2024-09-17T03:51:10.787120+0000","last_clean":"2024-09-17T03:51:10.787120+0000","last_became_active":"2024-09-17T03:51:09.530769+0000","last_became_peered":"2024-09-17T03:51:09.530769+0000","last_unstale":"2024-09-17T03:51:10.787120+0000","last_undegraded":"2024-09-17T03:51:10.787120+0000","last_fullsized":"2024-09-17T03:51:10.787120+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:05.070396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.540408+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.540084+0000","last_became_peered":"2024-09-17T03:51:09.540084+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:09.874665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531013+0000","last_change":"2024-09-17T03:51:09.527146+0000","last_active":"2024-09-17T03:51:10.531013+0000","last_peered":"2024-09-17T03:51:10.531013+0000","last_clean":"2024-09-17T03:51:10.531013+0000","last_became_active":"2024-09-17T03:51:09.526875+0000","last_became_peered":"2024-09-17T03:51:09.526875+0000","last_unstale":"2024-09-17T03:51:10.531013+0000","last_undegraded":"2024-09-17T03:51:10.531013+0000","last_fullsized":"2024-09-17T03:51:10.531013+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:19.358351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786259+0000","last_change":"2024-09-17T03:51:09.532104+0000","last_active":"2024-09-17T03:51:10.786259+0000","last_peered":"2024-09-17T03:51:10.786259+0000","last_clean":"2024-09-17T03:51:10.786259+0000","last_became_active":"2024-09-17T03:51:09.531801+0000","last_became_peered":"2024-09-17T03:51:09.531801+0000","last_unstale":"2024-09-17T03:51:10.786259+0000","last_undegraded":"2024-09-17T03:51:10.786259+0000","last_fullsized":"2024-09-17T03:51:10.786259+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:40.757442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704630+0000","last_change":"2024-09-17T03:51:09.547774+0000","last_active":"2024-09-17T03:51:10.704630+0000","last_peered":"2024-09-17T03:51:10.704630+0000","last_clean":"2024-09-17T03:51:10.704630+0000","last_became_active":"2024-09-17T03:51:09.547518+0000","last_became_peered":"2024-09-17T03:51:09.547518+0000","last_unstale":"2024-09-17T03:51:10.704630+0000","last_undegraded":"2024-09-17T03:51:10.704630+0000","last_fullsized":"2024-09-17T03:51:10.704630+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:25.520763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"44'4","reported_seq":19,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531884+0000","last_change":"2024-09-17T03:51:08.536130+0000","last_active":"2024-09-17T03:51:10.531884+0000","last_peered":"2024-09-17T03:51:10.531884+0000","last_clean":"2024-09-17T03:51:10.531884+0000","last_became_active":"2024-09-17T03:51:08.535845+0000","last_became_peered":"2024-09-17T03:51:08.535845+0000","last_unstale":"2024-09-17T03:51:10.531884+0000","last_undegraded":"2024-09-17T03:51:10.531884+0000","last_fullsized":"2024-09-17T03:51:10.531884+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:38.135381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787189+0000","last_change":"2024-09-17T03:51:09.530989+0000","last_active":"2024-09-17T03:51:10.787189+0000","last_peered":"2024-09-17T03:51:10.787189+0000","last_clean":"2024-09-17T03:51:10.787189+0000","last_became_active":"2024-09-17T03:51:09.529800+0000","last_became_peered":"2024-09-17T03:51:09.529800+0000","last_unstale":"2024-09-17T03:51:10.787189+0000","last_undegraded":"2024-09-17T03:51:10.787189+0000","last_fullsized":"2024-09-17T03:51:10.787189+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:22.141601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533204+0000","last_change":"2024-09-17T03:51:08.537833+0000","last_active":"2024-09-17T03:51:10.533204+0000","last_peered":"2024-09-17T03:51:10.533204+0000","last_clean":"2024-09-17T03:51:10.533204+0000","last_became_active":"2024-09-17T03:51:08.536906+0000","last_became_peered":"2024-09-17T03:51:08.536906+0000","last_unstale":"2024-09-17T03:51:10.533204+0000","last_undegraded":"2024-09-17T03:51:10.533204+0000","last_fullsized":"2024-09-17T03:51:10.533204+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.312040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533549+0000","last_change":"2024-09-17T03:51:09.519759+0000","last_active":"2024-09-17T03:51:10.533549+0000","last_peered":"2024-09-17T03:51:10.533549+0000","last_clean":"2024-09-17T03:51:10.533549+0000","last_became_active":"2024-09-17T03:51:09.519352+0000","last_became_peered":"2024-09-17T03:51:09.519352+0000","last_unstale":"2024-09-17T03:51:10.533549+0000","last_undegraded":"2024-09-17T03:51:10.533549+0000","last_fullsized":"2024-09-17T03:51:10.533549+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:22.986673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704595+0000","last_change":"2024-09-17T03:51:08.532550+0000","last_active":"2024-09-17T03:51:10.704595+0000","last_peered":"2024-09-17T03:51:10.704595+0000","last_clean":"2024-09-17T03:51:10.704595+0000","last_became_active":"2024-09-17T03:51:08.532173+0000","last_became_peered":"2024-09-17T03:51:08.532173+0000","last_unstale":"2024-09-17T03:51:10.704595+0000","last_undegraded":"2024-09-17T03:51:10.704595+0000","last_fullsized":"2024-09-17T03:51:10.704595+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:07.928711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704647+0000","last_change":"2024-09-17T03:51:09.549884+0000","last_active":"2024-09-17T03:51:10.704647+0000","last_peered":"2024-09-17T03:51:10.704647+0000","last_clean":"2024-09-17T03:51:10.704647+0000","last_became_active":"2024-09-17T03:51:09.549527+0000","last_became_peered":"2024-09-17T03:51:09.549527+0000","last_unstale":"2024-09-17T03:51:10.704647+0000","last_undegraded":"2024-09-17T03:51:10.704647+0000","last_fullsized":"2024-09-17T03:51:10.704647+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:53.191931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722478+0000","last_change":"2024-09-17T03:51:08.538223+0000","last_active":"2024-09-17T03:51:10.722478+0000","last_peered":"2024-09-17T03:51:10.722478+0000","last_clean":"2024-09-17T03:51:10.722478+0000","last_became_active":"2024-09-17T03:51:08.536819+0000","last_became_peered":"2024-09-17T03:51:08.536819+0000","last_unstale":"2024-09-17T03:51:10.722478+0000","last_undegraded":"2024-09-17T03:51:10.722478+0000","last_fullsized":"2024-09-17T03:51:10.722478+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:53.664159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534249+0000","last_change":"2024-09-17T03:51:09.533568+0000","last_active":"2024-09-17T03:51:10.534249+0000","last_peered":"2024-09-17T03:51:10.534249+0000","last_clean":"2024-09-17T03:51:10.534249+0000","last_became_active":"2024-09-17T03:51:09.533027+0000","last_became_peered":"2024-09-17T03:51:09.533027+0000","last_unstale":"2024-09-17T03:51:10.534249+0000","last_undegraded":"2024-09-17T03:51:10.534249+0000","last_fullsized":"2024-09-17T03:51:10.534249+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:39.579158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531542+0000","last_change":"2024-09-17T03:51:08.533400+0000","last_active":"2024-09-17T03:51:10.531542+0000","last_peered":"2024-09-17T03:51:10.531542+0000","last_clean":"2024-09-17T03:51:10.531542+0000","last_became_active":"2024-09-17T03:51:08.532958+0000","last_became_peered":"2024-09-17T03:51:08.532958+0000","last_unstale":"2024-09-17T03:51:10.531542+0000","last_undegraded":"2024-09-17T03:51:10.531542+0000","last_fullsized":"2024-09-17T03:51:10.531542+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:50.059584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786620+0000","last_change":"2024-09-17T03:51:09.543641+0000","last_active":"2024-09-17T03:51:10.786620+0000","last_peered":"2024-09-17T03:51:10.786620+0000","last_clean":"2024-09-17T03:51:10.786620+0000","last_became_active":"2024-09-17T03:51:09.543299+0000","last_became_peered":"2024-09-17T03:51:09.543299+0000","last_unstale":"2024-09-17T03:51:10.786620+0000","last_undegraded":"2024-09-17T03:51:10.786620+0000","last_fullsized":"2024-09-17T03:51:10.786620+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:34.478969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534447+0000","last_change":"2024-09-17T03:51:08.535613+0000","last_active":"2024-09-17T03:51:10.534447+0000","last_peered":"2024-09-17T03:51:10.534447+0000","last_clean":"2024-09-17T03:51:10.534447+0000","last_became_active":"2024-09-17T03:51:08.535214+0000","last_became_peered":"2024-09-17T03:51:08.535214+0000","last_unstale":"2024-09-17T03:51:10.534447+0000","last_undegraded":"2024-09-17T03:51:10.534447+0000","last_fullsized":"2024-09-17T03:51:10.534447+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.931699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785982+0000","last_change":"2024-09-17T03:51:09.552322+0000","last_active":"2024-09-17T03:51:10.785982+0000","last_peered":"2024-09-17T03:51:10.785982+0000","last_clean":"2024-09-17T03:51:10.785982+0000","last_became_active":"2024-09-17T03:51:09.551914+0000","last_became_peered":"2024-09-17T03:51:09.551914+0000","last_unstale":"2024-09-17T03:51:10.785982+0000","last_undegraded":"2024-09-17T03:51:10.785982+0000","last_fullsized":"2024-09-17T03:51:10.785982+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:38.111592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531983+0000","last_change":"2024-09-17T03:51:08.532902+0000","last_active":"2024-09-17T03:51:10.531983+0000","last_peered":"2024-09-17T03:51:10.531983+0000","last_clean":"2024-09-17T03:51:10.531983+0000","last_became_active":"2024-09-17T03:51:08.532401+0000","last_became_peered":"2024-09-17T03:51:08.532401+0000","last_unstale":"2024-09-17T03:51:10.531983+0000","last_undegraded":"2024-09-17T03:51:10.531983+0000","last_fullsized":"2024-09-17T03:51:10.531983+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:29.562126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786515+0000","last_change":"2024-09-17T03:51:09.541584+0000","last_active":"2024-09-17T03:51:10.786515+0000","last_peered":"2024-09-17T03:51:10.786515+0000","last_clean":"2024-09-17T03:51:10.786515+0000","last_became_active":"2024-09-17T03:51:09.541219+0000","last_became_peered":"2024-09-17T03:51:09.541219+0000","last_unstale":"2024-09-17T03:51:10.786515+0000","last_undegraded":"2024-09-17T03:51:10.786515+0000","last_fullsized":"2024-09-17T03:51:10.786515+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:57.414841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785991+0000","last_change":"2024-09-17T03:51:08.522934+0000","last_active":"2024-09-17T03:51:10.785991+0000","last_peered":"2024-09-17T03:51:10.785991+0000","last_clean":"2024-09-17T03:51:10.785991+0000","last_became_active":"2024-09-17T03:51:08.522647+0000","last_became_peered":"2024-09-17T03:51:08.522647+0000","last_unstale":"2024-09-17T03:51:10.785991+0000","last_undegraded":"2024-09-17T03:51:10.785991+0000","last_fullsized":"2024-09-17T03:51:10.785991+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:27.118951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532822+0000","last_change":"2024-09-17T03:51:09.550191+0000","last_active":"2024-09-17T03:51:10.532822+0000","last_peered":"2024-09-17T03:51:10.532822+0000","last_clean":"2024-09-17T03:51:10.532822+0000","last_became_active":"2024-09-17T03:51:09.548578+0000","last_became_peered":"2024-09-17T03:51:09.548578+0000","last_unstale":"2024-09-17T03:51:10.532822+0000","last_undegraded":"2024-09-17T03:51:10.532822+0000","last_fullsized":"2024-09-17T03:51:10.532822+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:01.627905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533705+0000","last_change":"2024-09-17T03:51:08.535319+0000","last_active":"2024-09-17T03:51:10.533705+0000","last_peered":"2024-09-17T03:51:10.533705+0000","last_clean":"2024-09-17T03:51:10.533705+0000","last_became_active":"2024-09-17T03:51:08.534878+0000","last_became_peered":"2024-09-17T03:51:08.534878+0000","last_unstale":"2024-09-17T03:51:10.533705+0000","last_undegraded":"2024-09-17T03:51:10.533705+0000","last_fullsized":"2024-09-17T03:51:10.533705+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:42.378567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722895+0000","last_change":"2024-09-17T03:51:09.542160+0000","last_active":"2024-09-17T03:51:10.722895+0000","last_peered":"2024-09-17T03:51:10.722895+0000","last_clean":"2024-09-17T03:51:10.722895+0000","last_became_active":"2024-09-17T03:51:09.541850+0000","last_became_peered":"2024-09-17T03:51:09.541850+0000","last_unstale":"2024-09-17T03:51:10.722895+0000","last_undegraded":"2024-09-17T03:51:10.722895+0000","last_fullsized":"2024-09-17T03:51:10.722895+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:07.233043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787744+0000","last_change":"2024-09-17T03:51:08.538705+0000","last_active":"2024-09-17T03:51:10.787744+0000","last_peered":"2024-09-17T03:51:10.787744+0000","last_clean":"2024-09-17T03:51:10.787744+0000","last_became_active":"2024-09-17T03:51:08.535730+0000","last_became_peered":"2024-09-17T03:51:08.535730+0000","last_unstale":"2024-09-17T03:51:10.787744+0000","last_undegraded":"2024-09-17T03:51:10.787744+0000","last_fullsized":"2024-09-17T03:51:10.787744+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:47.222685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531566+0000","last_change":"2024-09-17T03:51:09.545363+0000","last_active":"2024-09-17T03:51:10.531566+0000","last_peered":"2024-09-17T03:51:10.531566+0000","last_clean":"2024-09-17T03:51:10.531566+0000","last_became_active":"2024-09-17T03:51:09.545035+0000","last_became_peered":"2024-09-17T03:51:09.545035+0000","last_unstale":"2024-09-17T03:51:10.531566+0000","last_undegraded":"2024-09-17T03:51:10.531566+0000","last_fullsized":"2024-09-17T03:51:10.531566+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:56.841798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"44'3","reported_seq":18,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722938+0000","last_change":"2024-09-17T03:51:08.541002+0000","last_active":"2024-09-17T03:51:10.722938+0000","last_peered":"2024-09-17T03:51:10.722938+0000","last_clean":"2024-09-17T03:51:10.722938+0000","last_became_active":"2024-09-17T03:51:08.540686+0000","last_became_peered":"2024-09-17T03:51:08.540686+0000","last_unstale":"2024-09-17T03:51:10.722938+0000","last_undegraded":"2024-09-17T03:51:10.722938+0000","last_fullsized":"2024-09-17T03:51:10.722938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:57.694277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722633+0000","last_change":"2024-09-17T03:51:09.555925+0000","last_active":"2024-09-17T03:51:10.722633+0000","last_peered":"2024-09-17T03:51:10.722633+0000","last_clean":"2024-09-17T03:51:10.722633+0000","last_became_active":"2024-09-17T03:51:09.555612+0000","last_became_peered":"2024-09-17T03:51:09.555612+0000","last_unstale":"2024-09-17T03:51:10.722633+0000","last_undegraded":"2024-09-17T03:51:10.722633+0000","last_fullsized":"2024-09-17T03:51:10.722633+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:47.553258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"44'6","reported_seq":21,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786793+0000","last_change":"2024-09-17T03:51:08.538266+0000","last_active":"2024-09-17T03:51:10.786793+0000","last_peered":"2024-09-17T03:51:10.786793+0000","last_clean":"2024-09-17T03:51:10.786793+0000","last_became_active":"2024-09-17T03:51:08.537932+0000","last_became_peered":"2024-09-17T03:51:08.537932+0000","last_unstale":"2024-09-17T03:51:10.786793+0000","last_undegraded":"2024-09-17T03:51:10.786793+0000","last_fullsized":"2024-09-17T03:51:10.786793+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:07.138652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786834+0000","last_change":"2024-09-17T03:51:09.552398+0000","last_active":"2024-09-17T03:51:10.786834+0000","last_peered":"2024-09-17T03:51:10.786834+0000","last_clean":"2024-09-17T03:51:10.786834+0000","last_became_active":"2024-09-17T03:51:09.551972+0000","last_became_peered":"2024-09-17T03:51:09.551972+0000","last_unstale":"2024-09-17T03:51:10.786834+0000","last_undegraded":"2024-09-17T03:51:10.786834+0000","last_fullsized":"2024-09-17T03:51:10.786834+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:21.780774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531670+0000","last_change":"2024-09-17T03:51:08.528682+0000","last_active":"2024-09-17T03:51:10.531670+0000","last_peered":"2024-09-17T03:51:10.531670+0000","last_clean":"2024-09-17T03:51:10.531670+0000","last_became_active":"2024-09-17T03:51:08.527975+0000","last_became_peered":"2024-09-17T03:51:08.527975+0000","last_unstale":"2024-09-17T03:51:10.531670+0000","last_undegraded":"2024-09-17T03:51:10.531670+0000","last_fullsized":"2024-09-17T03:51:10.531670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:01.855706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786848+0000","last_change":"2024-09-17T03:51:09.549934+0000","last_active":"2024-09-17T03:51:10.786848+0000","last_peered":"2024-09-17T03:51:10.786848+0000","last_clean":"2024-09-17T03:51:10.786848+0000","last_became_active":"2024-09-17T03:51:09.547467+0000","last_became_peered":"2024-09-17T03:51:09.547467+0000","last_unstale":"2024-09-17T03:51:10.786848+0000","last_undegraded":"2024-09-17T03:51:10.786848+0000","last_fullsized":"2024-09-17T03:51:10.786848+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:40.182958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722670+0000","last_change":"2024-09-17T03:51:08.538721+0000","last_active":"2024-09-17T03:51:10.722670+0000","last_peered":"2024-09-17T03:51:10.722670+0000","last_clean":"2024-09-17T03:51:10.722670+0000","last_became_active":"2024-09-17T03:51:08.537978+0000","last_became_peered":"2024-09-17T03:51:08.537978+0000","last_unstale":"2024-09-17T03:51:10.722670+0000","last_undegraded":"2024-09-17T03:51:10.722670+0000","last_fullsized":"2024-09-17T03:51:10.722670+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:24.513480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.530908+0000","last_change":"2024-09-17T03:51:09.543247+0000","last_active":"2024-09-17T03:51:10.530908+0000","last_peered":"2024-09-17T03:51:10.530908+0000","last_clean":"2024-09-17T03:51:10.530908+0000","last_became_active":"2024-09-17T03:51:09.542929+0000","last_became_peered":"2024-09-17T03:51:09.542929+0000","last_unstale":"2024-09-17T03:51:10.530908+0000","last_undegraded":"2024-09-17T03:51:10.530908+0000","last_fullsized":"2024-09-17T03:51:10.530908+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:24.650512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531170+0000","last_change":"2024-09-17T03:51:08.535323+0000","last_active":"2024-09-17T03:51:10.531170+0000","last_peered":"2024-09-17T03:51:10.531170+0000","last_clean":"2024-09-17T03:51:10.531170+0000","last_became_active":"2024-09-17T03:51:08.534999+0000","last_became_peered":"2024-09-17T03:51:08.534999+0000","last_unstale":"2024-09-17T03:51:10.531170+0000","last_undegraded":"2024-09-17T03:51:10.531170+0000","last_fullsized":"2024-09-17T03:51:10.531170+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:00.620622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722647+0000","last_change":"2024-09-17T03:51:09.548531+0000","last_active":"2024-09-17T03:51:10.722647+0000","last_peered":"2024-09-17T03:51:10.722647+0000","last_clean":"2024-09-17T03:51:10.722647+0000","last_became_active":"2024-09-17T03:51:09.548212+0000","last_became_peered":"2024-09-17T03:51:09.548212+0000","last_unstale":"2024-09-17T03:51:10.722647+0000","last_undegraded":"2024-09-17T03:51:10.722647+0000","last_fullsized":"2024-09-17T03:51:10.722647+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:18:48.982638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"44'8","reported_seq":23,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786053+0000","last_change":"2024-09-17T03:51:08.530197+0000","last_active":"2024-09-17T03:51:10.786053+0000","last_peered":"2024-09-17T03:51:10.786053+0000","last_clean":"2024-09-17T03:51:10.786053+0000","last_became_active":"2024-09-17T03:51:08.529723+0000","last_became_peered":"2024-09-17T03:51:08.529723+0000","last_unstale":"2024-09-17T03:51:10.786053+0000","last_undegraded":"2024-09-17T03:51:10.786053+0000","last_fullsized":"2024-09-17T03:51:10.786053+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:24.054211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786825+0000","last_change":"2024-09-17T03:51:09.537261+0000","last_active":"2024-09-17T03:51:10.786825+0000","last_peered":"2024-09-17T03:51:10.786825+0000","last_clean":"2024-09-17T03:51:10.786825+0000","last_became_active":"2024-09-17T03:51:09.536865+0000","last_became_peered":"2024-09-17T03:51:09.536865+0000","last_unstale":"2024-09-17T03:51:10.786825+0000","last_undegraded":"2024-09-17T03:51:10.786825+0000","last_fullsized":"2024-09-17T03:51:10.786825+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:02.803728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786694+0000","last_change":"2024-09-17T03:51:08.535330+0000","last_active":"2024-09-17T03:51:10.786694+0000","last_peered":"2024-09-17T03:51:10.786694+0000","last_clean":"2024-09-17T03:51:10.786694+0000","last_became_active":"2024-09-17T03:51:08.535035+0000","last_became_peered":"2024-09-17T03:51:08.535035+0000","last_unstale":"2024-09-17T03:51:10.786694+0000","last_undegraded":"2024-09-17T03:51:10.786694+0000","last_fullsized":"2024-09-17T03:51:10.786694+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:25.251556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787536+0000","last_change":"2024-09-17T03:51:09.559584+0000","last_active":"2024-09-17T03:51:10.787536+0000","last_peered":"2024-09-17T03:51:10.787536+0000","last_clean":"2024-09-17T03:51:10.787536+0000","last_became_active":"2024-09-17T03:51:09.559215+0000","last_became_peered":"2024-09-17T03:51:09.559215+0000","last_unstale":"2024-09-17T03:51:10.787536+0000","last_undegraded":"2024-09-17T03:51:10.787536+0000","last_fullsized":"2024-09-17T03:51:10.787536+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:48.857021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.533214+0000","last_change":"2024-09-17T03:51:08.529671+0000","last_active":"2024-09-17T03:51:10.533214+0000","last_peered":"2024-09-17T03:51:10.533214+0000","last_clean":"2024-09-17T03:51:10.533214+0000","last_became_active":"2024-09-17T03:51:08.529313+0000","last_became_peered":"2024-09-17T03:51:08.529313+0000","last_unstale":"2024-09-17T03:51:10.533214+0000","last_undegraded":"2024-09-17T03:51:10.533214+0000","last_fullsized":"2024-09-17T03:51:10.533214+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:57:51.643162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787006+0000","last_change":"2024-09-17T03:51:09.531351+0000","last_active":"2024-09-17T03:51:10.787006+0000","last_peered":"2024-09-17T03:51:10.787006+0000","last_clean":"2024-09-17T03:51:10.787006+0000","last_became_active":"2024-09-17T03:51:09.530270+0000","last_became_peered":"2024-09-17T03:51:09.530270+0000","last_unstale":"2024-09-17T03:51:10.787006+0000","last_undegraded":"2024-09-17T03:51:10.787006+0000","last_fullsized":"2024-09-17T03:51:10.787006+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:38:05.287191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"44'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532687+0000","last_change":"2024-09-17T03:51:08.531273+0000","last_active":"2024-09-17T03:51:10.532687+0000","last_peered":"2024-09-17T03:51:10.532687+0000","last_clean":"2024-09-17T03:51:10.532687+0000","last_became_active":"2024-09-17T03:51:08.530933+0000","last_became_peered":"2024-09-17T03:51:08.530933+0000","last_unstale":"2024-09-17T03:51:10.532687+0000","last_undegraded":"2024-09-17T03:51:10.532687+0000","last_fullsized":"2024-09-17T03:51:10.532687+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:24:32.058802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787095+0000","last_change":"2024-09-17T03:51:09.545503+0000","last_active":"2024-09-17T03:51:10.787095+0000","last_peered":"2024-09-17T03:51:10.787095+0000","last_clean":"2024-09-17T03:51:10.787095+0000","last_became_active":"2024-09-17T03:51:09.544721+0000","last_became_peered":"2024-09-17T03:51:09.544721+0000","last_unstale":"2024-09-17T03:51:10.787095+0000","last_undegraded":"2024-09-17T03:51:10.787095+0000","last_fullsized":"2024-09-17T03:51:10.787095+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:10.003938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531735+0000","last_change":"2024-09-17T03:51:08.534140+0000","last_active":"2024-09-17T03:51:10.531735+0000","last_peered":"2024-09-17T03:51:10.531735+0000","last_clean":"2024-09-17T03:51:10.531735+0000","last_became_active":"2024-09-17T03:51:08.533878+0000","last_became_peered":"2024-09-17T03:51:08.533878+0000","last_unstale":"2024-09-17T03:51:10.531735+0000","last_undegraded":"2024-09-17T03:51:10.531735+0000","last_fullsized":"2024-09-17T03:51:10.531735+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:26.509368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531142+0000","last_change":"2024-09-17T03:51:09.548786+0000","last_active":"2024-09-17T03:51:10.531142+0000","last_peered":"2024-09-17T03:51:10.531142+0000","last_clean":"2024-09-17T03:51:10.531142+0000","last_became_active":"2024-09-17T03:51:09.548475+0000","last_became_peered":"2024-09-17T03:51:09.548475+0000","last_unstale":"2024-09-17T03:51:10.531142+0000","last_undegraded":"2024-09-17T03:51:10.531142+0000","last_fullsized":"2024-09-17T03:51:10.531142+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:03.300865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531749+0000","last_change":"2024-09-17T03:51:08.533378+0000","last_active":"2024-09-17T03:51:10.531749+0000","last_peered":"2024-09-17T03:51:10.531749+0000","last_clean":"2024-09-17T03:51:10.531749+0000","last_became_active":"2024-09-17T03:51:08.532682+0000","last_became_peered":"2024-09-17T03:51:08.532682+0000","last_unstale":"2024-09-17T03:51:10.531749+0000","last_undegraded":"2024-09-17T03:51:10.531749+0000","last_fullsized":"2024-09-17T03:51:10.531749+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:25.905425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704333+0000","last_change":"2024-09-17T03:51:09.527784+0000","last_active":"2024-09-17T03:51:10.704333+0000","last_peered":"2024-09-17T03:51:10.704333+0000","last_clean":"2024-09-17T03:51:10.704333+0000","last_became_active":"2024-09-17T03:51:09.527488+0000","last_became_peered":"2024-09-17T03:51:09.527488+0000","last_unstale":"2024-09-17T03:51:10.704333+0000","last_undegraded":"2024-09-17T03:51:10.704333+0000","last_fullsized":"2024-09-17T03:51:10.704333+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:42.057452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532284+0000","last_change":"2024-09-17T03:51:08.531986+0000","last_active":"2024-09-17T03:51:10.532284+0000","last_peered":"2024-09-17T03:51:10.532284+0000","last_clean":"2024-09-17T03:51:10.532284+0000","last_became_active":"2024-09-17T03:51:08.531674+0000","last_became_peered":"2024-09-17T03:51:08.531674+0000","last_unstale":"2024-09-17T03:51:10.532284+0000","last_undegraded":"2024-09-17T03:51:10.532284+0000","last_fullsized":"2024-09-17T03:51:10.532284+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:52.184648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.785978+0000","last_change":"2024-09-17T03:51:09.538946+0000","last_active":"2024-09-17T03:51:10.785978+0000","last_peered":"2024-09-17T03:51:10.785978+0000","last_clean":"2024-09-17T03:51:10.785978+0000","last_became_active":"2024-09-17T03:51:09.538617+0000","last_became_peered":"2024-09-17T03:51:09.538617+0000","last_unstale":"2024-09-17T03:51:10.785978+0000","last_undegraded":"2024-09-17T03:51:10.785978+0000","last_fullsized":"2024-09-17T03:51:10.785978+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:46.014539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786247+0000","last_change":"2024-09-17T03:51:08.526124+0000","last_active":"2024-09-17T03:51:10.786247+0000","last_peered":"2024-09-17T03:51:10.786247+0000","last_clean":"2024-09-17T03:51:10.786247+0000","last_became_active":"2024-09-17T03:51:08.525837+0000","last_became_peered":"2024-09-17T03:51:08.525837+0000","last_unstale":"2024-09-17T03:51:10.786247+0000","last_undegraded":"2024-09-17T03:51:10.786247+0000","last_fullsized":"2024-09-17T03:51:10.786247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:09.038592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787015+0000","last_change":"2024-09-17T03:51:09.556563+0000","last_active":"2024-09-17T03:51:10.787015+0000","last_peered":"2024-09-17T03:51:10.787015+0000","last_clean":"2024-09-17T03:51:10.787015+0000","last_became_active":"2024-09-17T03:51:09.555716+0000","last_became_peered":"2024-09-17T03:51:09.555716+0000","last_unstale":"2024-09-17T03:51:10.787015+0000","last_undegraded":"2024-09-17T03:51:10.787015+0000","last_fullsized":"2024-09-17T03:51:10.787015+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:30.216865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722462+0000","last_change":"2024-09-17T03:51:08.535527+0000","last_active":"2024-09-17T03:51:10.722462+0000","last_peered":"2024-09-17T03:51:10.722462+0000","last_clean":"2024-09-17T03:51:10.722462+0000","last_became_active":"2024-09-17T03:51:08.535182+0000","last_became_peered":"2024-09-17T03:51:08.535182+0000","last_unstale":"2024-09-17T03:51:10.722462+0000","last_undegraded":"2024-09-17T03:51:10.722462+0000","last_fullsized":"2024-09-17T03:51:10.722462+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:41.050169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787369+0000","last_change":"2024-09-17T03:51:09.552798+0000","last_active":"2024-09-17T03:51:10.787369+0000","last_peered":"2024-09-17T03:51:10.787369+0000","last_clean":"2024-09-17T03:51:10.787369+0000","last_became_active":"2024-09-17T03:51:09.552477+0000","last_became_peered":"2024-09-17T03:51:09.552477+0000","last_unstale":"2024-09-17T03:51:10.787369+0000","last_undegraded":"2024-09-17T03:51:10.787369+0000","last_fullsized":"2024-09-17T03:51:10.787369+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:35.740581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"45'1","reported_seq":16,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531475+0000","last_change":"2024-09-17T03:51:08.531555+0000","last_active":"2024-09-17T03:51:10.531475+0000","last_peered":"2024-09-17T03:51:10.531475+0000","last_clean":"2024-09-17T03:51:10.531475+0000","last_became_active":"2024-09-17T03:51:08.531230+0000","last_became_peered":"2024-09-17T03:51:08.531230+0000","last_unstale":"2024-09-17T03:51:10.531475+0000","last_undegraded":"2024-09-17T03:51:10.531475+0000","last_fullsized":"2024-09-17T03:51:10.531475+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:49.383641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787381+0000","last_change":"2024-09-17T03:51:09.558633+0000","last_active":"2024-09-17T03:51:10.787381+0000","last_peered":"2024-09-17T03:51:10.787381+0000","last_clean":"2024-09-17T03:51:10.787381+0000","last_became_active":"2024-09-17T03:51:09.558301+0000","last_became_peered":"2024-09-17T03:51:09.558301+0000","last_unstale":"2024-09-17T03:51:10.787381+0000","last_undegraded":"2024-09-17T03:51:10.787381+0000","last_fullsized":"2024-09-17T03:51:10.787381+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:53.303687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.723056+0000","last_change":"2024-09-17T03:51:08.542303+0000","last_active":"2024-09-17T03:51:10.723056+0000","last_peered":"2024-09-17T03:51:10.723056+0000","last_clean":"2024-09-17T03:51:10.723056+0000","last_became_active":"2024-09-17T03:51:08.541884+0000","last_became_peered":"2024-09-17T03:51:08.541884+0000","last_unstale":"2024-09-17T03:51:10.723056+0000","last_undegraded":"2024-09-17T03:51:10.723056+0000","last_fullsized":"2024-09-17T03:51:10.723056+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:05.012964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722997+0000","last_change":"2024-09-17T03:51:09.546329+0000","last_active":"2024-09-17T03:51:10.722997+0000","last_peered":"2024-09-17T03:51:10.722997+0000","last_clean":"2024-09-17T03:51:10.722997+0000","last_became_active":"2024-09-17T03:51:09.545955+0000","last_became_peered":"2024-09-17T03:51:09.545955+0000","last_unstale":"2024-09-17T03:51:10.722997+0000","last_undegraded":"2024-09-17T03:51:10.722997+0000","last_fullsized":"2024-09-17T03:51:10.722997+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:59.129570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704276+0000","last_change":"2024-09-17T03:51:08.533982+0000","last_active":"2024-09-17T03:51:10.704276+0000","last_peered":"2024-09-17T03:51:10.704276+0000","last_clean":"2024-09-17T03:51:10.704276+0000","last_became_active":"2024-09-17T03:51:08.529828+0000","last_became_peered":"2024-09-17T03:51:08.529828+0000","last_unstale":"2024-09-17T03:51:10.704276+0000","last_undegraded":"2024-09-17T03:51:10.704276+0000","last_fullsized":"2024-09-17T03:51:10.704276+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:47.222628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531393+0000","last_change":"2024-09-17T03:51:09.542790+0000","last_active":"2024-09-17T03:51:10.531393+0000","last_peered":"2024-09-17T03:51:10.531393+0000","last_clean":"2024-09-17T03:51:10.531393+0000","last_became_active":"2024-09-17T03:51:09.541286+0000","last_became_peered":"2024-09-17T03:51:09.541286+0000","last_unstale":"2024-09-17T03:51:10.531393+0000","last_undegraded":"2024-09-17T03:51:10.531393+0000","last_fullsized":"2024-09-17T03:51:10.531393+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:55.401998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722938+0000","last_change":"2024-09-17T03:51:08.540515+0000","last_active":"2024-09-17T03:51:10.722938+0000","last_peered":"2024-09-17T03:51:10.722938+0000","last_clean":"2024-09-17T03:51:10.722938+0000","last_became_active":"2024-09-17T03:51:08.540197+0000","last_became_peered":"2024-09-17T03:51:08.540197+0000","last_unstale":"2024-09-17T03:51:10.722938+0000","last_undegraded":"2024-09-17T03:51:10.722938+0000","last_fullsized":"2024-09-17T03:51:10.722938+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:41.046821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.531523+0000","last_change":"2024-09-17T03:51:09.548118+0000","last_active":"2024-09-17T03:51:10.531523+0000","last_peered":"2024-09-17T03:51:10.531523+0000","last_clean":"2024-09-17T03:51:10.531523+0000","last_became_active":"2024-09-17T03:51:09.544581+0000","last_became_peered":"2024-09-17T03:51:09.544581+0000","last_unstale":"2024-09-17T03:51:10.531523+0000","last_undegraded":"2024-09-17T03:51:10.531523+0000","last_fullsized":"2024-09-17T03:51:10.531523+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:41.421187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.534533+0000","last_change":"2024-09-17T03:51:08.538899+0000","last_active":"2024-09-17T03:51:10.534533+0000","last_peered":"2024-09-17T03:51:10.534533+0000","last_clean":"2024-09-17T03:51:10.534533+0000","last_became_active":"2024-09-17T03:51:08.538338+0000","last_became_peered":"2024-09-17T03:51:08.538338+0000","last_unstale":"2024-09-17T03:51:10.534533+0000","last_undegraded":"2024-09-17T03:51:10.534533+0000","last_fullsized":"2024-09-17T03:51:10.534533+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:56.151879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532043+0000","last_change":"2024-09-17T03:51:09.527635+0000","last_active":"2024-09-17T03:51:10.532043+0000","last_peered":"2024-09-17T03:51:10.532043+0000","last_clean":"2024-09-17T03:51:10.532043+0000","last_became_active":"2024-09-17T03:51:09.525856+0000","last_became_peered":"2024-09-17T03:51:09.525856+0000","last_unstale":"2024-09-17T03:51:10.532043+0000","last_undegraded":"2024-09-17T03:51:10.532043+0000","last_fullsized":"2024-09-17T03:51:10.532043+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:50.390923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.704242+0000","last_change":"2024-09-17T03:51:08.535001+0000","last_active":"2024-09-17T03:51:10.704242+0000","last_peered":"2024-09-17T03:51:10.704242+0000","last_clean":"2024-09-17T03:51:10.704242+0000","last_became_active":"2024-09-17T03:51:08.531914+0000","last_became_peered":"2024-09-17T03:51:08.531914+0000","last_unstale":"2024-09-17T03:51:10.704242+0000","last_undegraded":"2024-09-17T03:51:10.704242+0000","last_fullsized":"2024-09-17T03:51:10.704242+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:54:35.813335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532927+0000","last_change":"2024-09-17T03:51:09.540447+0000","last_active":"2024-09-17T03:51:10.532927+0000","last_peered":"2024-09-17T03:51:10.532927+0000","last_clean":"2024-09-17T03:51:10.532927+0000","last_became_active":"2024-09-17T03:51:09.540161+0000","last_became_peered":"2024-09-17T03:51:09.540161+0000","last_unstale":"2024-09-17T03:51:10.532927+0000","last_undegraded":"2024-09-17T03:51:10.532927+0000","last_fullsized":"2024-09-17T03:51:10.532927+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:04.256313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":15,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.786105+0000","last_change":"2024-09-17T03:51:08.525657+0000","last_active":"2024-09-17T03:51:10.786105+0000","last_peered":"2024-09-17T03:51:10.786105+0000","last_clean":"2024-09-17T03:51:10.786105+0000","last_became_active":"2024-09-17T03:51:08.525271+0000","last_became_peered":"2024-09-17T03:51:08.525271+0000","last_unstale":"2024-09-17T03:51:10.786105+0000","last_undegraded":"2024-09-17T03:51:10.786105+0000","last_fullsized":"2024-09-17T03:51:10.786105+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:55.028248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":13,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.787764+0000","last_change":"2024-09-17T03:51:09.544994+0000","last_active":"2024-09-17T03:51:10.787764+0000","last_peered":"2024-09-17T03:51:10.787764+0000","last_clean":"2024-09-17T03:51:10.787764+0000","last_became_active":"2024-09-17T03:51:09.544748+0000","last_became_peered":"2024-09-17T03:51:09.544748+0000","last_unstale":"2024-09-17T03:51:10.787764+0000","last_undegraded":"2024-09-17T03:51:10.787764+0000","last_fullsized":"2024-09-17T03:51:10.787764+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":43,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:08.499499+0000","last_clean_scrub_stamp":"2024-09-17T03:51:08.499499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:14.819738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"44'2","reported_seq":17,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.532598+0000","last_change":"2024-09-17T03:51:08.529407+0000","last_active":"2024-09-17T03:51:10.532598+0000","last_peered":"2024-09-17T03:51:10.532598+0000","last_clean":"2024-09-17T03:51:10.532598+0000","last_became_active":"2024-09-17T03:51:08.529046+0000","last_became_peered":"2024-09-17T03:51:08.529046+0000","last_unstale":"2024-09-17T03:51:10.532598+0000","last_undegraded":"2024-09-17T03:51:10.532598+0000","last_fullsized":"2024-09-17T03:51:10.532598+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:07.492216+0000","last_clean_scrub_stamp":"2024-09-17T03:51:07.492216+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:08.023937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":152,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-17T03:51:10.722514+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:10.722514+0000","last_peered":"2024-09-17T03:51:10.722514+0000","last_clean":"2024-09-17T03:51:10.722514+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:10.722514+0000","last_undegraded":"2024-09-17T03:51:10.722514+0000","last_fullsized":"2024-09-17T03:51:10.722514+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":799331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509467,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":524288,"data_stored":208943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509466,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":210485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509464,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":532480,"data_stored":209511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509466,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":536576,"data_stored":210507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509466,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":524288,"data_stored":208943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542169,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31704,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":528384,"data_stored":209545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542170,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1118208,"data_stored":799275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:19.611 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:51:19.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.877+0000 7f7596400640 1 -- 172.21.15.129:0/415027051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75740051a0 msgr2=0x7f757400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.877+0000 7f7596400640 1 --2- 172.21.15.129:0/415027051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75740051a0 0x7f757400b9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f759015a5b0 tx=0x7f7574005830 comp rx=0 tx=0).stop 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.877+0000 7f7595a00640 1 -- 172.21.15.129:0/415027051 reap_dead start 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.877+0000 7f7596400640 1 -- 172.21.15.129:0/415027051 shutdown_connections 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.877+0000 7f7596400640 1 -- 172.21.15.129:0/415027051 >> 172.21.15.129:0/415027051 conn(0x7f759006d410 msgr2=0x7f75900b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.878+0000 7f7596400640 1 -- 172.21.15.129:0/415027051 shutdown_connections 2024-09-17T03:51:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.878+0000 7f7596400640 1 -- 172.21.15.129:0/415027051 wait complete. 2024-09-17T03:51:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.880+0000 7f7596400640 1 Processor -- start 2024-09-17T03:51:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.880+0000 7f7596400640 1 -- start start 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75740051a0 0x7f759013d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7590154e60 0x7f7590143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f759015d700 con 0x7f7590143e80 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f75900cfc10 con 0x7f75740051a0 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.881+0000 7f7596400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f759015d880 con 0x7f7590154e60 2024-09-17T03:51:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39480/0 (socket says 172.21.15.129:39480) 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 -- 172.21.15.129:0/3473950653 learned_addr learned my addr 172.21.15.129:0/3473950653 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595000640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75740051a0 0x7f759013d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f758fe00640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7590154e60 0x7f7590143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 -- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7590154e60 msgr2=0x7f7590143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7590154e60 0x7f7590143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 -- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75740051a0 msgr2=0x7f759013d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75740051a0 0x7f759013d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 -- 172.21.15.129:0/3473950653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759013dd50 con 0x7f7590143e80 2024-09-17T03:51:19.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.882+0000 7f7595a00640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7588036040 tx=0x7f7588033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.883+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588038040 con 0x7f7590143e80 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.883+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7588005020 con 0x7f7590143e80 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.883+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588005430 con 0x7f7590143e80 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.883+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7590153dc0 con 0x7f7590143e80 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.883+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7590154380 con 0x7f7590143e80 2024-09-17T03:51:19.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.884+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759006ee70 con 0x7f7590143e80 2024-09-17T03:51:19.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.885+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7588044030 con 0x7f7590143e80 2024-09-17T03:51:19.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.887+0000 7f758cc00640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 0x7f756c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:19.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.887+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f75880bad10 con 0x7f7590143e80 2024-09-17T03:51:19.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.887+0000 7f7595000640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 0x7f756c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:19.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.888+0000 7f7595000640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 0x7f756c07b0a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7580002230 tx=0x7f7580006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:19.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:19.891+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75880c0020 con 0x7f7590143e80 2024-09-17T03:51:20.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.144+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f759006fb50 con 0x7f7590143e80 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.147+0000 7f758cc00640 1 -- 172.21.15.129:0/3473950653 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v28) ==== 190+0+147 (secure 0 0 0) 0x7f758804b020 con 0x7f7590143e80 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:51:20.146 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:51:20.147 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:51:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 msgr2=0x7f756c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 0x7f756c07b0a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7580002230 tx=0x7f7580006040 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 msgr2=0x7f7590155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7588036040 tx=0x7f7588033670 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 shutdown_connections 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f756c078b90 0x7f756c07b0a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7590143e80 0x7f7590155ed0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7590154e60 0x7f7590143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 --2- 172.21.15.129:0/3473950653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75740051a0 0x7f759013d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.149+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 >> 172.21.15.129:0/3473950653 conn(0x7f759006d410 msgr2=0x7f75900afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.150+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 shutdown_connections 2024-09-17T03:51:20.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.150+0000 7f7596400640 1 -- 172.21.15.129:0/3473950653 wait complete. 2024-09-17T03:51:20.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.394+0000 7f3dc7e00640 1 -- 172.21.15.129:0/4172415991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dac0051a0 msgr2=0x7f3dac00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.394+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/4172415991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dac0051a0 0x7f3dac00b9c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3dc815a5b0 tx=0x7f3dac005830 comp rx=0 tx=0).stop 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.394+0000 7f3dc7400640 1 -- 172.21.15.129:0/4172415991 reap_dead start 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.395+0000 7f3dc7e00640 1 -- 172.21.15.129:0/4172415991 shutdown_connections 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.395+0000 7f3dc7e00640 1 -- 172.21.15.129:0/4172415991 >> 172.21.15.129:0/4172415991 conn(0x7f3dc806d410 msgr2=0x7f3dc80b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.395+0000 7f3dc7e00640 1 -- 172.21.15.129:0/4172415991 shutdown_connections 2024-09-17T03:51:20.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.395+0000 7f3dc7e00640 1 -- 172.21.15.129:0/4172415991 wait complete. 2024-09-17T03:51:20.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.397+0000 7f3dc7e00640 1 Processor -- start 2024-09-17T03:51:20.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.397+0000 7f3dc7e00640 1 -- start start 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.398+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3dac0051a0 0x7f3dc813cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.398+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.398+0000 7f3dc7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dc81e93d0 0x7f3dc8154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dc7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc815d700 con 0x7f3dc8143ae0 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dc7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc80cfc10 con 0x7f3dc81e93d0 2024-09-17T03:51:20.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dc7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc815d880 con 0x7f3dac0051a0 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39510/0 (socket says 172.21.15.129:39510) 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 -- 172.21.15.129:0/1980037054 learned_addr learned my addr 172.21.15.129:0/1980037054 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dc6a00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3dac0051a0 0x7f3dc813cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dc7400640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dc81e93d0 0x7f3dc8154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 -- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3dac0051a0 msgr2=0x7f3dc813cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3dac0051a0 0x7f3dc813cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 -- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dc81e93d0 msgr2=0x7f3dc8154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dc81e93d0 0x7f3dc8154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.399+0000 7f3dbe000640 1 -- 172.21.15.129:0/1980037054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc81e97e0 con 0x7f3dc8143ae0 2024-09-17T03:51:20.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dbe000640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3db0003ef0 tx=0x7f3db0001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0015070 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3db0011900 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc80b3320 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0011d50 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.400+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc80b38e0 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.401+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc806ee70 con 0x7f3dc8143ae0 2024-09-17T03:51:20.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.402+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3db0018070 con 0x7f3dc8143ae0 2024-09-17T03:51:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.403+0000 7f3dbfe00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 0x7f3da407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.403+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f3db009ba00 con 0x7f3dc8143ae0 2024-09-17T03:51:20.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.404+0000 7f3dc6a00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 0x7f3da407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:20.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.404+0000 7f3dc6a00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 0x7f3da407b0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3db8001eb0 tx=0x7f3db8000fe0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:20.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.408+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3db0006e60 con 0x7f3dc8143ae0 2024-09-17T03:51:20.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.633+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3dc806fb50 con 0x7f3dc8143ae0 2024-09-17T03:51:20.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.634+0000 7f3dbfe00640 1 -- 172.21.15.129:0/1980037054 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7f3db00637e0 con 0x7f3dc8143ae0 2024-09-17T03:51:20.633 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:20.633 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:20.633 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 msgr2=0x7f3da407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 0x7f3da407b0d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3db8001eb0 tx=0x7f3db8000fe0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 msgr2=0x7f3dc81e8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f3db0003ef0 tx=0x7f3db0001800 comp rx=0 tx=0).stop 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 shutdown_connections 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3da4078bc0 0x7f3da407b0d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3dc81e93d0 0x7f3dc8154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3dc8143ae0 0x7f3dc81e8e90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 --2- 172.21.15.129:0/1980037054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3dac0051a0 0x7f3dc813cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:20.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 >> 172.21.15.129:0/1980037054 conn(0x7f3dc806d410 msgr2=0x7f3dc8144310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:20.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 shutdown_connections 2024-09-17T03:51:20.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:20.636+0000 7f3dc7e00640 1 -- 172.21.15.129:0/1980037054 wait complete. 2024-09-17T03:51:21.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:51:09.008746+0000', 'modified': '2024-09-17T03:51:12.020277+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5232}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5232': {'gid': 5232, 'name': 'c', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/1529444990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 1529444990}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1529444990}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5232, 'qdb_cluster': [5232]} 2024-09-17T03:51:21.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:51:21.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 -- 172.21.15.129:0/2486727760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 msgr2=0x7f71580ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 --2- 172.21.15.129:0/2486727760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580ccf60 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7150005fd0 tx=0x7f715002aa00 comp rx=0 tx=0).stop 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 -- 172.21.15.129:0/2486727760 shutdown_connections 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 --2- 172.21.15.129:0/2486727760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580ccf60 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 --2- 172.21.15.129:0/2486727760 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71580cc1e0 0x7f71580af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 -- 172.21.15.129:0/2486727760 >> 172.21.15.129:0/2486727760 conn(0x7f715806d7d0 msgr2=0x7f71580b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 -- 172.21.15.129:0/2486727760 shutdown_connections 2024-09-17T03:51:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.286+0000 7f715f000640 1 -- 172.21.15.129:0/2486727760 wait complete. 2024-09-17T03:51:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.289+0000 7f715f000640 1 Processor -- start 2024-09-17T03:51:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.289+0000 7f715f000640 1 -- start start 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71580cc1e0 0x7f7158152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 0x7f71580ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71581533a0 con 0x7f71580ccb50 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f715815d400 con 0x7f71580cc1e0 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.290+0000 7f715f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7158153520 con 0x7f715815d8a0 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39532/0 (socket says 172.21.15.129:39532) 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 -- 172.21.15.129:0/4234952497 learned_addr learned my addr 172.21.15.129:0/4234952497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157e00640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71580cc1e0 0x7f7158152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f715d000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 0x7f71580ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 -- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 msgr2=0x7f71580ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 0x7f71580ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 -- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71580cc1e0 msgr2=0x7f7158152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71580cc1e0 0x7f7158152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 -- 172.21.15.129:0/4234952497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71581e4f00 con 0x7f71580ccb50 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f715d000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 0x7f71580ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.291+0000 7f7157400640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f714400b3f0 tx=0x7f714400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.292+0000 7f714fe00640 1 -- 172.21.15.129:0/4234952497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144015020 con 0x7f71580ccb50 2024-09-17T03:51:21.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.292+0000 7f714fe00640 1 -- 172.21.15.129:0/4234952497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7144011920 con 0x7f71580ccb50 2024-09-17T03:51:21.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.292+0000 7f714fe00640 1 -- 172.21.15.129:0/4234952497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144011d70 con 0x7f71580ccb50 2024-09-17T03:51:21.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.292+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71581fda00 con 0x7f71580ccb50 2024-09-17T03:51:21.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.292+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71581fdfc0 con 0x7f71580ccb50 2024-09-17T03:51:21.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.294+0000 7f714fe00640 1 -- 172.21.15.129:0/4234952497 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7144018020 con 0x7f71580ccb50 2024-09-17T03:51:21.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.295+0000 7f714fe00640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 0x7f713807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.295+0000 7f714fe00640 1 -- 172.21.15.129:0/4234952497 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f714409a9e0 con 0x7f71580ccb50 2024-09-17T03:51:21.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.295+0000 7f7157e00640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 0x7f713807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.296+0000 7f7166083740 1 Processor -- start 2024-09-17T03:51:21.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.296+0000 7f7166083740 1 -- start start 2024-09-17T03:51:21.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.296+0000 7f7157e00640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 0x7f713807b0a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7148005fd0 tx=0x7f7148005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b92571970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b92541f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563b924cb430 con 0x563b921783b0 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563b924bb5d0 con 0x563b92180590 2024-09-17T03:51:21.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563b924aaa00 con 0x563b921a1270 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39546/0 (socket says 172.21.15.129:39546) 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7157400640 1 -- 172.21.15.129:0/3668334028 learned_addr learned my addr 172.21.15.129:0/3668334028 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f7157e00640 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b92571970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.297+0000 7f715d000640 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b92541f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 -- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 msgr2=0x563b92571970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b92571970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 -- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 msgr2=0x563b92541f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b92541f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 -- 172.21.15.129:0/3668334028 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b92473520 con 0x563b921783b0 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f7157400640 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f714405e7c0 tx=0x7f7144062610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221f113e42ce5c38 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f714cc00640 1 -- 172.21.15.129:0/3668334028 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71440671f0 con 0x563b921783b0 2024-09-17T03:51:21.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f714cc00640 1 -- 172.21.15.129:0/3668334028 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f714406c0a0 con 0x563b921783b0 2024-09-17T03:51:21.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.298+0000 7f714cc00640 1 -- 172.21.15.129:0/3668334028 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144061bb0 con 0x563b921783b0 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 -- 172.21.15.129:0/3668334028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 msgr2=0x563b92571eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f714405e7c0 tx=0x7f7144062610 comp rx=0 tx=0).stop 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 -- 172.21.15.129:0/3668334028 shutdown_connections 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b92541f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92571eb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 --2- 172.21.15.129:0/3668334028 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b92571970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 -- 172.21.15.129:0/3668334028 >> 172.21.15.129:0/3668334028 conn(0x563b921953c0 msgr2=0x563b92547970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 -- 172.21.15.129:0/3668334028 shutdown_connections 2024-09-17T03:51:21.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.299+0000 7f7166083740 1 -- 172.21.15.129:0/3668334028 wait complete. 2024-09-17T03:51:21.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.300+0000 7f7166083740 1 -- start start 2024-09-17T03:51:21.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.300+0000 7f7166083740 1 Processor -- start 2024-09-17T03:51:21.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925863c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 0x563b9256f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563b92451330 con 0x563b921a1270 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563b924ae810 con 0x563b921783b0 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563b9221b620 con 0x563b92180590 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7166083740 10 client.0 _ll_register_callbacks cb 0x563b92456420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f715d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f715d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39552/0 (socket says 172.21.15.129:39552) 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f715d000640 1 -- 172.21.15.129:0/1887913175 learned_addr learned my addr 172.21.15.129:0/1887913175 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7157400640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 0x563b9256f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.302+0000 7f7157e00640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925863c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 -- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 msgr2=0x563b9256f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 0x563b9256f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 -- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 msgr2=0x563b925863c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925863c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 -- 172.21.15.129:0/1887913175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b9220c850 con 0x563b921a1270 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7157400640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 0x563b9256f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7157e00640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:21.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f715d000640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f715002aff0 tx=0x7f7150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1210ebefe1a0578f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7141600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150048060 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f714406c0a0 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.303+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f7150003d80 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f715004e050 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 10 client.5382 fetch_fsmap learned FSMap version 26 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x563b922836b0 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f714406c0a0 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 3502+0+0 (secure 0 0 0) 0x7f7150051020 con 0x563b921a1270 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 10 client.5382 fetch_fsmap finished waiting for FSMap version 26 2024-09-17T03:51:21.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 10 client.5382 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 20 client.5382 populate_metadata read hostname 'smithi129' 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b9215fe40 0x563b9259f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 4 client.5382 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 --> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b924bd520 con 0x563b9215fe40 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.304+0000 7f7157e00640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b9215fe40 0x563b9259f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.305+0000 7f7157e00640 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b9215fe40 0x563b9259f100 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da4dcaf16959aeb9 server_cookie=d67496f426f62ca5 in_seq=0 out_seq=0 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.305+0000 7f7141600640 10 client.5382 ms_handle_connect on v2:172.21.15.129:6833/1529444990 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7141600640 1 -- 172.21.15.129:0/1887913175 <== mds.0 v2:172.21.15.129:6833/1529444990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563b924bd520 con 0x563b9215fe40 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7141600640 10 client.5382 handle_command_reply: tid=0 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 client.5382 shutdown 2024-09-17T03:51:21.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 msgr2=0x563b9256dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f715002aff0 tx=0x7f7150004a40 comp rx=0 tx=0).stop 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 shutdown_connections 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b9215fe40 0x563b9259f100 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921a1270 0x563b9256dd50 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b92180590 0x563b9256f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 --2- 172.21.15.129:0/1887913175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925863c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7141600640 0 client.5382 ms_handle_reset on v2:172.21.15.129:6833/1529444990 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 >> 172.21.15.129:0/1887913175 conn(0x563b921953c0 msgr2=0x563b92582e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 shutdown_connections 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.306+0000 7f7166083740 1 -- 172.21.15.129:0/1887913175 wait complete. 2024-09-17T03:51:21.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.307+0000 7f7166083740 20 client.5382 trim_cache size 0 max 16384 2024-09-17T03:51:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.326+0000 7f7166083740 1 Processor -- start 2024-09-17T03:51:21.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.326+0000 7f7166083740 1 -- start start 2024-09-17T03:51:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9259b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7158205820 con 0x563b92180590 2024-09-17T03:51:21.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7158073680 con 0x563b921783b0 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7158073cd0 con 0x563b921a1270 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.327+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39560/0 (socket says 172.21.15.129:39560) 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 -- 172.21.15.129:0/630605651 learned_addr learned my addr 172.21.15.129:0/630605651 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 -- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 msgr2=0x563b9259b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9259b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 -- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 msgr2=0x563b925a59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 -- 172.21.15.129:0/630605651 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b92546a00 con 0x563b92180590 2024-09-17T03:51:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7157400640 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f714405e7c0 tx=0x7f7144004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afe7bebf6f9be37b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7143e00640 1 -- 172.21.15.129:0/630605651 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714406c0a0 con 0x563b92180590 2024-09-17T03:51:21.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7143e00640 1 -- 172.21.15.129:0/630605651 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71440671f0 con 0x563b92180590 2024-09-17T03:51:21.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.328+0000 7f7143e00640 1 -- 172.21.15.129:0/630605651 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714407ce10 con 0x563b92180590 2024-09-17T03:51:21.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 -- 172.21.15.129:0/630605651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 msgr2=0x563b9256d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f714405e7c0 tx=0x7f7144004680 comp rx=0 tx=0).stop 2024-09-17T03:51:21.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 -- 172.21.15.129:0/630605651 shutdown_connections 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9259b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b92180590 0x563b9256d270 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 --2- 172.21.15.129:0/630605651 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b921783b0 0x563b925a59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 -- 172.21.15.129:0/630605651 >> 172.21.15.129:0/630605651 conn(0x563b921953c0 msgr2=0x563b9256b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 -- 172.21.15.129:0/630605651 shutdown_connections 2024-09-17T03:51:21.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.329+0000 7f7166083740 1 -- 172.21.15.129:0/630605651 wait complete. 2024-09-17T03:51:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.330+0000 7f7166083740 1 -- start start 2024-09-17T03:51:21.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.330+0000 7f7166083740 1 Processor -- start 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b925a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9260c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563b9256c580 con 0x563b921783b0 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563b9256e6a0 con 0x563b92180590 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563b9256b890 con 0x563b921a1270 2024-09-17T03:51:21.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7166083740 10 client.0 _ll_register_callbacks cb 0x7f7150031d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7157e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7157e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39574/0 (socket says 172.21.15.129:39574) 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7157e00640 1 -- 172.21.15.129:0/3415901379 learned_addr learned my addr 172.21.15.129:0/3415901379 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f715d000640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9260c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.332+0000 7f7157400640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b925a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 -- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 msgr2=0x563b9260c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9260c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 -- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 msgr2=0x563b925a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b925a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 -- 172.21.15.129:0/3415901379 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b925ae5a0 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7157e00640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x563b92544230 tx=0x7f71480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87056fa54fb60abc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7141600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7148011850 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f71480119f0 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71440671f0 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.333+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7148039050 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 10 client.5388 fetch_fsmap learned FSMap version 26 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) -- 0x563b925a2380 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71440671f0 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 26) ==== 3502+0+0 (secure 0 0 0) 0x7f714803c020 con 0x563b921783b0 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 10 client.5388 fetch_fsmap finished waiting for FSMap version 26 2024-09-17T03:51:21.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 10 client.5388 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 20 client.5388 populate_metadata read hostname 'smithi129' 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b925a0ea0 0x563b9260fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 4 client.5388 mds_command: new command op to 5232 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 --> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] -- command(tid 0: {"prefix": "status"}) -- 0x563b924b8fd0 con 0x563b925a0ea0 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.334+0000 7f7157400640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b925a0ea0 0x563b9260fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7157400640 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b925a0ea0 0x563b9260fa00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698a0fd0280488d server_cookie=5d23c17bb859c7fb in_seq=0 out_seq=0 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7141600640 10 client.5388 ms_handle_connect on v2:172.21.15.129:6833/1529444990 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7141600640 1 -- 172.21.15.129:0/3415901379 <== mds.0 v2:172.21.15.129:6833/1529444990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x563b924b8fd0 con 0x563b925a0ea0 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7141600640 10 client.5388 handle_command_reply: tid=0 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 client.5388 shutdown 2024-09-17T03:51:21.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 msgr2=0x563b92568110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x563b92544230 tx=0x7f71480046c0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 shutdown_connections 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:6833/1529444990,v1:172.21.15.129:6835/1529444990] conn(0x563b925a0ea0 0x563b9260fa00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563b921a1270 0x563b9260c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563b92180590 0x563b925a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 --2- 172.21.15.129:0/3415901379 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563b921783b0 0x563b92568110 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7141600640 0 client.5388 ms_handle_reset on v2:172.21.15.129:6833/1529444990 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 >> 172.21.15.129:0/3415901379 conn(0x563b921953c0 msgr2=0x563b9259b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 shutdown_connections 2024-09-17T03:51:21.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.335+0000 7f7166083740 1 -- 172.21.15.129:0/3415901379 wait complete. 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f7166083740 20 client.5388 trim_cache size 0 max 16384 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 5232, 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:51:21.341 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.823948055000001, 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 26, 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 45, 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 45, 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.895758818000001 2024-09-17T03:51:21.342 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 msgr2=0x7f713807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 0x7f713807b0a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7148005fd0 tx=0x7f7148005850 comp rx=0 tx=0).stop 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 msgr2=0x7f71580adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f714400b3f0 tx=0x7f714400b8b0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 shutdown_connections 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7138078b90 0x7f713807b0a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f715815d8a0 0x7f71580ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71580ccb50 0x7f71580adfe0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 --2- 172.21.15.129:0/4234952497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71580cc1e0 0x7f7158152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 >> 172.21.15.129:0/4234952497 conn(0x7f715806d7d0 msgr2=0x7f715806ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 shutdown_connections 2024-09-17T03:51:21.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.336+0000 7f715f000640 1 -- 172.21.15.129:0/4234952497 wait complete. 2024-09-17T03:51:21.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:51:21.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 -- 172.21.15.129:0/622186569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244073170 msgr2=0x7f024415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 --2- 172.21.15.129:0/622186569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244073170 0x7f024415c780 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0238009830 tx=0x7f023802a530 comp rx=0 tx=0).stop 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 -- 172.21.15.129:0/622186569 shutdown_connections 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 --2- 172.21.15.129:0/622186569 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244073170 0x7f024415c780 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 -- 172.21.15.129:0/622186569 >> 172.21.15.129:0/622186569 conn(0x7f02440b1cb0 msgr2=0x7f02440b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 -- 172.21.15.129:0/622186569 shutdown_connections 2024-09-17T03:51:21.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.578+0000 7f024be00640 1 -- 172.21.15.129:0/622186569 wait complete. 2024-09-17T03:51:21.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.580+0000 7f024be00640 1 Processor -- start 2024-09-17T03:51:21.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.581+0000 7f024be00640 1 -- start start 2024-09-17T03:51:21.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 0x7f0244150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 0x7f024414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02440b3cd0 con 0x7f0244070030 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f02440b3b50 con 0x7f0244070440 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f024be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f02440b3e50 con 0x7f0244073170 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f0249400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f0249400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39596/0 (socket says 172.21.15.129:39596) 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f0249400640 1 -- 172.21.15.129:0/861216843 learned_addr learned my addr 172.21.15.129:0/861216843 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249e00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 0x7f024414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.582+0000 7f0248a00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 0x7f0244150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 -- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 msgr2=0x7f024414dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 0x7f024414dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 -- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 msgr2=0x7f0244150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 0x7f0244150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 -- 172.21.15.129:0/861216843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f024409ab60 con 0x7f0244070030 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249e00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 0x7f024414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0248a00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 0x7f0244150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0249400640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0238000c00 tx=0x7f0238033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0238031d60 con 0x7f0244070030 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0238033070 con 0x7f0244070030 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02380334c0 con 0x7f0244070030 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f024409ae40 con 0x7f0244070030 2024-09-17T03:51:21.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.583+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f024409d500 con 0x7f0244070030 2024-09-17T03:51:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.584+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02440b4300 con 0x7f0244070030 2024-09-17T03:51:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.585+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0238033660 con 0x7f0244070030 2024-09-17T03:51:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.586+0000 7f0241600640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 0x7f022407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:21.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.586+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f02380bc280 con 0x7f0244070030 2024-09-17T03:51:21.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.587+0000 7f0248a00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 0x7f022407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:21.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.588+0000 7f0248a00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 0x7f022407b0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0244154fa0 tx=0x7f0230009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:21.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.591+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f023803f030 con 0x7f0244070030 2024-09-17T03:51:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.807+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f02440c3860 con 0x7f0244070030 2024-09-17T03:51:21.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.808+0000 7f0241600640 1 -- 172.21.15.129:0/861216843 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f0238084060 con 0x7f0244070030 2024-09-17T03:51:21.808 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 msgr2=0x7f022407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 0x7f022407b0d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0244154fa0 tx=0x7f0230009040 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 msgr2=0x7f02440d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0238000c00 tx=0x7f0238033d60 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 shutdown_connections 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0224078bc0 0x7f022407b0d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0244073170 0x7f024414dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0244070440 0x7f0244150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 --2- 172.21.15.129:0/861216843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0244070030 0x7f02440d03d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 >> 172.21.15.129:0/861216843 conn(0x7f02440b1cb0 msgr2=0x7f024415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 shutdown_connections 2024-09-17T03:51:21.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:21.810+0000 7f024be00640 1 -- 172.21.15.129:0/861216843 wait complete. 2024-09-17T03:51:21.835 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:51:21.835 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:51:21.835 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:51:21.836 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:21.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:21.837 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:51:21.873 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42610sec preferred_lft 42610sec 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.875 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:51:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:51:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:21.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:21.877 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:51:21.967 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:51:21.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:21.971 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:51:22.006 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:51:22.007 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:51:22.007 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:51:22.007 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:51:22.007 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:51:22.007 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:51:22.007 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:51:22.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:51:22.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:51:22.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:51:22.008 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:51:22.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:51:22.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:51:22.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:22.202 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:51:22.238 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:51:22.238 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:51:22.238 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:22.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:22.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:51:22.334 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:51:22.334 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:22.370 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:22.370 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:51:22.370 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:22.445 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:51:22.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:22.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:22.641 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:22.641 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:22.641 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:22.661 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:22.661 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:22.662 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:22.662 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:51:22.662 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:51:22.695 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:51:22.696 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:51:22.696 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:51:22.696 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:51:22.697 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:51:22.697 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:51:22.699 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:51:22.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:22.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:22.756 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:51:22.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:22 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:51:22.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:51:22.851 INFO:teuthology.orchestra.run.smithi129.stdout:client4985 2024-09-17T03:51:22.852 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:22.852 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:22.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-09-17T03:51:23.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 -- 172.21.15.129:0/3530268412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc15c580 msgr2=0x7fdefc0d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3530268412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc15c580 0x7fdefc0d0800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdefc161100 tx=0x7fdeec02a4d0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 -- 172.21.15.129:0/3530268412 shutdown_connections 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3530268412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc0735f0 0x7fdefc160040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3530268412 >> v2:172.21.15.181:3300/0 conn(0x7fdefc072ca0 0x7fdefc0730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3530268412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc15c580 0x7fdefc0d0800 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 -- 172.21.15.129:0/3530268412 >> 172.21.15.129:0/3530268412 conn(0x7fdefc06ee50 msgr2=0x7fdefc06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 -- 172.21.15.129:0/3530268412 shutdown_connections 2024-09-17T03:51:23.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.152+0000 7fdf01a00640 1 -- 172.21.15.129:0/3530268412 wait complete. 2024-09-17T03:51:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.154+0000 7fdf01a00640 1 Processor -- start 2024-09-17T03:51:23.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.154+0000 7fdf01a00640 1 -- start start 2024-09-17T03:51:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc072ca0 0x7fdefc1515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:23.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 0x7fdefc1f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc160910 con 0x7fdefc0735f0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc160790 con 0x7fdefc15c580 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdefc160a90 con 0x7fdefc072ca0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdefb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdefb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39610/0 (socket says 172.21.15.129:39610) 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdefb400640 1 -- 172.21.15.129:0/3950007833 learned_addr learned my addr 172.21.15.129:0/3950007833 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdefbe00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc072ca0 0x7fdefc1515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.156+0000 7fdf01000640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 0x7fdefc1f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 -- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc072ca0 msgr2=0x7fdefc1515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc072ca0 0x7fdefc1515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 -- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 msgr2=0x7fdefc1f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 0x7fdefc1f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 -- 172.21.15.129:0/3950007833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc155120 con 0x7fdefc0735f0 2024-09-17T03:51:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdf01000640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 0x7fdefc1f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdefb400640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e710 tx=0x7fdee800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee800cab0 con 0x7fdefc0735f0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdee800cd40 con 0x7fdefc0735f0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee8010690 con 0x7fdefc0735f0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc15ecf0 con 0x7fdefc0735f0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.157+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc15f2b0 con 0x7fdefc0735f0 2024-09-17T03:51:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.158+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdefc068d00 con 0x7fdefc0735f0 2024-09-17T03:51:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.159+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdee8010830 con 0x7fdefc0735f0 2024-09-17T03:51:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.160+0000 7fdef3e00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 0x7fdedc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.160+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fdee809b690 con 0x7fdefc0735f0 2024-09-17T03:51:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.161+0000 7fdefbe00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 0x7fdedc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.162+0000 7fdefbe00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 0x7fdedc07b0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdeec033000 tx=0x7fdeec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.165+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdefc068d00 con 0x7fdefc0735f0 2024-09-17T03:51:23.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.350+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7fdefc160040 con 0x7fdefc0735f0 2024-09-17T03:51:23.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.355+0000 7fdef3e00640 1 -- 172.21.15.129:0/3950007833 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v35)=0 v35) ==== 124+0+0 (secure 0 0 0) 0x7fdee80633e0 con 0x7fdefc0735f0 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 msgr2=0x7fdedc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 0x7fdedc07b0d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdeec033000 tx=0x7fdeec004730 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 msgr2=0x7fdefc1f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdee800e710 tx=0x7fdee800ebd0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 shutdown_connections 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdedc078bc0 0x7fdedc07b0d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdefc15c580 0x7fdefc1f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdefc0735f0 0x7fdefc1f6c10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 --2- 172.21.15.129:0/3950007833 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdefc072ca0 0x7fdefc1515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:23.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 >> 172.21.15.129:0/3950007833 conn(0x7fdefc06ee50 msgr2=0x7fdefc158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:23.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 shutdown_connections 2024-09-17T03:51:23.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:23.357+0000 7fdf01a00640 1 -- 172.21.15.129:0/3950007833 wait complete. 2024-09-17T03:51:33.380 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:33.380 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:33.420 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:51:33.420 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> n = 51 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T03:51:33.421 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:51:33.422 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T03:51:33.423 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:51:33.629 INFO:teuthology.orchestra.run.smithi129.stderr:Traceback (most recent call last): 2024-09-17T03:51:33.629 INFO:teuthology.orchestra.run.smithi129.stderr: File "", line 20, in 2024-09-17T03:51:33.629 INFO:teuthology.orchestra.run.smithi129.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_50' 2024-09-17T03:51:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:33 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-09-17T03:51:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:51:33.643 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:51:33.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:33.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:33.871 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:51:33.871 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:51:33.872 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:33.872 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:33.907 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:33.907 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:33.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:51:34.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 -- 172.21.15.129:0/71735920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b4004800 msgr2=0x7f82b4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:34.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 --2- 172.21.15.129:0/71735920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82b4004800 0x7f82b4004c10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f82d0072320 tx=0x7f82b4005830 comp rx=0 tx=0).stop 2024-09-17T03:51:34.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82ce800640 1 -- 172.21.15.129:0/71735920 reap_dead start 2024-09-17T03:51:34.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 -- 172.21.15.129:0/71735920 shutdown_connections 2024-09-17T03:51:34.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 -- 172.21.15.129:0/71735920 >> 172.21.15.129:0/71735920 conn(0x7f82d00b1cb0 msgr2=0x7f82d00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:34.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 -- 172.21.15.129:0/71735920 shutdown_connections 2024-09-17T03:51:34.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.245+0000 7f82d4c00640 1 -- 172.21.15.129:0/71735920 wait complete. 2024-09-17T03:51:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.247+0000 7f82d4c00640 1 Processor -- start 2024-09-17T03:51:34.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.247+0000 7f82d4c00640 1 -- start start 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82d009da70 0x7f82d009de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82d00d0450 0x7f82d00d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82d00b37a0 con 0x7f82b4004800 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f82d015ccc0 con 0x7f82d00d0450 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82d4c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f82d00b3920 con 0x7f82d009da70 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:34.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47118/0 (socket says 172.21.15.129:47118) 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 -- 172.21.15.129:0/2933640089 learned_addr learned my addr 172.21.15.129:0/2933640089 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82c7e00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82d009da70 0x7f82d009de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82ce800640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82d00d0450 0x7f82d00d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 -- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82d009da70 msgr2=0x7f82d009de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82d009da70 0x7f82d009de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 -- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82d00d0450 msgr2=0x7f82d00d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82d00d0450 0x7f82d00d0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.249+0000 7f82cde00640 1 -- 172.21.15.129:0/2933640089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d01491a0 con 0x7f82b4004800 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82cde00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f82c0011b20 tx=0x7f82c0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c001e070 con 0x7f82b4004800 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82c000fe50 con 0x7f82b4004800 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d00d0da0 con 0x7f82b4004800 2024-09-17T03:51:34.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0019650 con 0x7f82b4004800 2024-09-17T03:51:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.250+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d01f6fd0 con 0x7f82b4004800 2024-09-17T03:51:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.251+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d00ad120 con 0x7f82b4004800 2024-09-17T03:51:34.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.252+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f82c0021070 con 0x7f82b4004800 2024-09-17T03:51:34.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.253+0000 7f82c6000640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 0x7f82ac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:34.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.253+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f82c009b800 con 0x7f82b4004800 2024-09-17T03:51:34.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.254+0000 7f82c7e00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 0x7f82ac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:34.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.254+0000 7f82c7e00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 0x7f82ac07b0d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82b8032000 tx=0x7f82b8001360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:34.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.258+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f82d00ad120 con 0x7f82b4004800 2024-09-17T03:51:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.454+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f82d00b3eb0 con 0x7f82b4004800 2024-09-17T03:51:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.454+0000 7f82c6000640 1 -- 172.21.15.129:0/2933640089 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v45) ==== 80+0+0 (secure 0 0 0) 0x7f82d00d0da0 con 0x7f82b4004800 2024-09-17T03:51:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 msgr2=0x7f82ac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 0x7f82ac07b0d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82b8032000 tx=0x7f82b8001360 comp rx=0 tx=0).stop 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 msgr2=0x7f82d0148c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f82c0011b20 tx=0x7f82c0009f60 comp rx=0 tx=0).stop 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 shutdown_connections 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82ac078bc0 0x7f82ac07b0d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82d00d0450 0x7f82d00d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82d009da70 0x7f82d009de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 --2- 172.21.15.129:0/2933640089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82b4004800 0x7f82d0148c60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 >> 172.21.15.129:0/2933640089 conn(0x7f82d00b1cb0 msgr2=0x7f82d015f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 shutdown_connections 2024-09-17T03:51:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:34.458+0000 7f82d4c00640 1 -- 172.21.15.129:0/2933640089 wait complete. 2024-09-17T03:51:34.480 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:51:34.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:34.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:34.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:51:34.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:34.623 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:51:34.672 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:51:34.673 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:34.673 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:34.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:34.710 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:51:34.710 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:34.710 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:34.748 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:51:34.748 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:51:34.748 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:34.748 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:34.786 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:34.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 -- 172.21.15.129:0/3206688353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0cf3d0 msgr2=0x7f1f0c15ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 --2- 172.21.15.129:0/3206688353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c15ba00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c070e50 tx=0x7f1ef402a540 comp rx=0 tx=0).stop 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 -- 172.21.15.129:0/3206688353 shutdown_connections 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 --2- 172.21.15.129:0/3206688353 >> v2:172.21.15.181:3300/0 conn(0x7f1f0c15bf40 0x7f1f0c073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 --2- 172.21.15.129:0/3206688353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c15ba00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 -- 172.21.15.129:0/3206688353 >> 172.21.15.129:0/3206688353 conn(0x7f1f0c06d870 msgr2=0x7f1f0c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 -- 172.21.15.129:0/3206688353 shutdown_connections 2024-09-17T03:51:35.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.010+0000 7f1f0be00640 1 -- 172.21.15.129:0/3206688353 wait complete. 2024-09-17T03:51:35.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.013+0000 7f1f0be00640 1 Processor -- start 2024-09-17T03:51:35.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.013+0000 7f1f0be00640 1 -- start start 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c1f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f0c15bf40 0x7f1f0c0ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c0744e0 con 0x7f1f0c0d0500 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c074360 con 0x7f1f0c0cf3d0 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.014+0000 7f1f0be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1f0c074660 con 0x7f1f0c15bf40 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47136/0 (socket says 172.21.15.129:47136) 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 -- 172.21.15.129:0/1891523963 learned_addr learned my addr 172.21.15.129:0/1891523963 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f0b400640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f0c15bf40 0x7f1f0c0ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f0aa00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c1f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 -- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f0c15bf40 msgr2=0x7f1f0c0ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f0c15bf40 0x7f1f0c0ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 -- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f0c0cf3d0 msgr2=0x7f1f0c1f57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c1f57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 -- 172.21.15.129:0/1891523963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0c1f8d30 con 0x7f1f0c0d0500 2024-09-17T03:51:35.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.015+0000 7f1f02000640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009780 tx=0x7f1ef4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.016+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4031d60 con 0x7f1f0c0d0500 2024-09-17T03:51:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.016+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ef4033e10 con 0x7f1f0c0d0500 2024-09-17T03:51:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.016+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4005260 con 0x7f1f0c0d0500 2024-09-17T03:51:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.016+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0c1f9010 con 0x7f1f0c0d0500 2024-09-17T03:51:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.016+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0c1f95d0 con 0x7f1f0c0d0500 2024-09-17T03:51:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.017+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f0c15ba00 con 0x7f1f0c0d0500 2024-09-17T03:51:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.018+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1ef4005400 con 0x7f1f0c0d0500 2024-09-17T03:51:35.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.019+0000 7f1f03e00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 0x7f1ee807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.019+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f1ef40bbea0 con 0x7f1f0c0d0500 2024-09-17T03:51:35.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.020+0000 7f1f0aa00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 0x7f1ee807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.020+0000 7f1f0aa00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 0x7f1ee807b0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c075370 tx=0x7f1efc007eb0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.023+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1ef403f030 con 0x7f1f0c0d0500 2024-09-17T03:51:35.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.242+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f0c068d90 con 0x7f1f0c0d0500 2024-09-17T03:51:35.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.243+0000 7f1f03e00640 1 -- 172.21.15.129:0/1891523963 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7f1ef4083c80 con 0x7f1f0c0d0500 2024-09-17T03:51:35.243 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:35.243 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:35.243 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:35.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 msgr2=0x7f1ee807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 0x7f1ee807b0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c075370 tx=0x7f1efc007eb0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 msgr2=0x7f1f0c0cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009780 tx=0x7f1ef4033c20 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 shutdown_connections 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ee8078b90 0x7f1ee807b0a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f0c15bf40 0x7f1f0c0ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f0c0d0500 0x7f1f0c0cc9d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 --2- 172.21.15.129:0/1891523963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f0c0cf3d0 0x7f1f0c1f57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 >> 172.21.15.129:0/1891523963 conn(0x7f1f0c06d870 msgr2=0x7f1f0c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 shutdown_connections 2024-09-17T03:51:35.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.245+0000 7f1f0be00640 1 -- 172.21.15.129:0/1891523963 wait complete. 2024-09-17T03:51:35.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 -- 172.21.15.129:0/1390521121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9710 msgr2=0x7fadec0d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 --2- 172.21.15.129:0/1390521121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9710 0x7fadec0d0c40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fadd4009860 tx=0x7fadd402a4a0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 -- 172.21.15.129:0/1390521121 shutdown_connections 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 --2- 172.21.15.129:0/1390521121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9710 0x7fadec0d0c40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 -- 172.21.15.129:0/1390521121 >> 172.21.15.129:0/1390521121 conn(0x7fadec09d680 msgr2=0x7fadec0d4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 -- 172.21.15.129:0/1390521121 shutdown_connections 2024-09-17T03:51:35.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.490+0000 7fadf2000640 1 -- 172.21.15.129:0/1390521121 wait complete. 2024-09-17T03:51:35.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.492+0000 7fadf2000640 1 Processor -- start 2024-09-17T03:51:35.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.493+0000 7fadf2000640 1 -- start start 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadec0b9710 0x7fadec0a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9c80 0x7fadec1f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fadec0b4b80 con 0x7fadec0d1180 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fadec0b4a00 con 0x7fadec0b9710 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fadec0b4d00 con 0x7fadec0b9c80 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47138/0 (socket says 172.21.15.129:47138) 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.494+0000 7fadf1600640 1 -- 172.21.15.129:0/158332418 learned_addr learned my addr 172.21.15.129:0/158332418 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf0c00640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadec0b9710 0x7fadec0a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fade3e00640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9c80 0x7fadec1f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 -- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9c80 msgr2=0x7fadec1f8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9c80 0x7fadec1f8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 -- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadec0b9710 msgr2=0x7fadec0a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadec0b9710 0x7fadec0a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 -- 172.21.15.129:0/158332418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadec1f9c40 con 0x7fadec0d1180 2024-09-17T03:51:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf1600640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fade400c730 tx=0x7fade400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade4003b70 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fade4003d10 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadec15ab60 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.495+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade400aab0 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.496+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadec15b120 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.497+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadec0d0c40 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.498+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fade400ac50 con 0x7fadec0d1180 2024-09-17T03:51:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.499+0000 7fade9600640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 0x7fadc807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.499+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fade409aa40 con 0x7fadec0d1180 2024-09-17T03:51:35.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.499+0000 7fadf0c00640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 0x7fadc807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.500+0000 7fadf0c00640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 0x7fadc807b0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faddc000fb0 tx=0x7faddc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.503+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fade4062820 con 0x7fadec0d1180 2024-09-17T03:51:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.722+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadec15eac0 con 0x7fadec0d1180 2024-09-17T03:51:35.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.723+0000 7fade9600640 1 -- 172.21.15.129:0/158332418 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7fadec15eac0 con 0x7fadec0d1180 2024-09-17T03:51:35.723 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:35.724 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:35.724 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.725+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 msgr2=0x7fadc807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.725+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 0x7fadc807b0d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faddc000fb0 tx=0x7faddc009040 comp rx=0 tx=0).stop 2024-09-17T03:51:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 msgr2=0x7fadec1f9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fade400c730 tx=0x7fade400cbf0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 shutdown_connections 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadc8078bc0 0x7fadc807b0d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fadec0d1180 0x7fadec1f9330 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fadec0b9c80 0x7fadec1f8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 --2- 172.21.15.129:0/158332418 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fadec0b9710 0x7fadec0a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 >> 172.21.15.129:0/158332418 conn(0x7fadec09d680 msgr2=0x7fadec0d2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 shutdown_connections 2024-09-17T03:51:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.726+0000 7fadf2000640 1 -- 172.21.15.129:0/158332418 wait complete. 2024-09-17T03:51:35.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:35.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 -- 172.21.15.129:0/2085497637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094072c30 msgr2=0x7f909415ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 --2- 172.21.15.129:0/2085497637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094072c30 0x7f909415ef50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f907c009830 tx=0x7f907c02a4a0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 -- 172.21.15.129:0/2085497637 shutdown_connections 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 --2- 172.21.15.129:0/2085497637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f909415f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 --2- 172.21.15.129:0/2085497637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094072c30 0x7f909415ef50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.969+0000 7f9093e00640 1 -- 172.21.15.129:0/2085497637 >> 172.21.15.129:0/2085497637 conn(0x7f909406d410 msgr2=0x7f90940b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.970+0000 7f9093e00640 1 -- 172.21.15.129:0/2085497637 shutdown_connections 2024-09-17T03:51:35.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.970+0000 7f9093e00640 1 -- 172.21.15.129:0/2085497637 wait complete. 2024-09-17T03:51:35.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.972+0000 7f9093e00640 1 Processor -- start 2024-09-17T03:51:35.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.972+0000 7f9093e00640 1 -- start start 2024-09-17T03:51:35.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f90941549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 0x7f90940cf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90940b3ff0 con 0x7f9094072c30 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f90940b3e70 con 0x7f9094154f20 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9093e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f90940b4170 con 0x7f9094073040 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.973+0000 7f9092a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47142/0 (socket says 172.21.15.129:47142) 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 -- 172.21.15.129:0/1508796984 learned_addr learned my addr 172.21.15.129:0/1508796984 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9093400640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 0x7f90940cf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f908a000640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f90941549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 -- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 msgr2=0x7f90941549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f90941549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 -- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 msgr2=0x7f90940cf9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 0x7f90940cf9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 -- 172.21.15.129:0/1508796984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90940cde70 con 0x7f9094072c30 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9093400640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 0x7f90940cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f908a000640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f90941549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f9092a00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f908400c530 tx=0x7f908400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084016020 con 0x7f9094072c30 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.974+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9084012940 con 0x7f9094072c30 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.975+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084012d90 con 0x7f9094072c30 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.975+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90940ce150 con 0x7f9094072c30 2024-09-17T03:51:35.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.975+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90940ce710 con 0x7f9094072c30 2024-09-17T03:51:35.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.976+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90940b46d0 con 0x7f9094072c30 2024-09-17T03:51:35.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.977+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9084019020 con 0x7f9094072c30 2024-09-17T03:51:35.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.978+0000 7f908b400640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 0x7f907007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:35.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.979+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f908409aea0 con 0x7f9094072c30 2024-09-17T03:51:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.979+0000 7f908a000640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 0x7f907007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.980+0000 7f908a000640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 0x7f907007b0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f907c034000 tx=0x7f907c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:35.983+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9084020020 con 0x7f9094072c30 2024-09-17T03:51:36.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.166+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90940750a0 con 0x7f9094072c30 2024-09-17T03:51:36.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.167+0000 7f908b400640 1 -- 172.21.15.129:0/1508796984 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16721 (secure 0 0 0) 0x7f9084063240 con 0x7f9094072c30 2024-09-17T03:51:36.168 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:36.168 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:10.521773+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:06.922986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:07.972487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 msgr2=0x7f907007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 0x7f907007b0a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f907c034000 tx=0x7f907c004730 comp rx=0 tx=0).stop 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 msgr2=0x7f9094153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f908400c530 tx=0x7f908400c9f0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 shutdown_connections 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9070078b90 0x7f907007b0a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9094154f20 0x7f90940cf9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9094073040 0x7f90941549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 --2- 172.21.15.129:0/1508796984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9094072c30 0x7f9094153fb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 >> 172.21.15.129:0/1508796984 conn(0x7f909406d410 msgr2=0x7f9094159550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:36.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 shutdown_connections 2024-09-17T03:51:36.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.170+0000 7f9093e00640 1 -- 172.21.15.129:0/1508796984 wait complete. 2024-09-17T03:51:36.196 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:51:36.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.418+0000 7f634ac00640 1 -- 172.21.15.129:0/476355393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 msgr2=0x7f634415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.418+0000 7f634ac00640 1 --2- 172.21.15.129:0/476355393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f634415c780 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6334005fd0 tx=0x7f633402a4d0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.418+0000 7f634ac00640 1 -- 172.21.15.129:0/476355393 shutdown_connections 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.418+0000 7f634ac00640 1 --2- 172.21.15.129:0/476355393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070030 0x7f6344074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.418+0000 7f634ac00640 1 --2- 172.21.15.129:0/476355393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f634415c780 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.419+0000 7f634ac00640 1 -- 172.21.15.129:0/476355393 >> 172.21.15.129:0/476355393 conn(0x7f63440b1cb0 msgr2=0x7f63440b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.419+0000 7f634ac00640 1 -- 172.21.15.129:0/476355393 shutdown_connections 2024-09-17T03:51:36.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.419+0000 7f634ac00640 1 -- 172.21.15.129:0/476355393 wait complete. 2024-09-17T03:51:36.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.421+0000 7f634ac00640 1 Processor -- start 2024-09-17T03:51:36.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.421+0000 7f634ac00640 1 -- start start 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6344070030 0x7f63441f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070440 0x7f63440ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f634406f5a0 con 0x7f6344073170 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6344074f80 con 0x7f6344070030 2024-09-17T03:51:36.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.422+0000 7f634ac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f634406f720 con 0x7f6344070440 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47172/0 (socket says 172.21.15.129:47172) 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 -- 172.21.15.129:0/4070443872 learned_addr learned my addr 172.21.15.129:0/4070443872 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6343e00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6344070030 0x7f63441f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6343400640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070440 0x7f63440ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 -- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070440 msgr2=0x7f63440ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070440 0x7f63440ce020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 -- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6344070030 msgr2=0x7f63441f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6344070030 0x7f63441f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 -- 172.21.15.129:0/4070443872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63440cbd40 con 0x7f6344073170 2024-09-17T03:51:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.423+0000 7f6348c00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f633c00ba10 tx=0x7f633c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.424+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c019070 con 0x7f6344073170 2024-09-17T03:51:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.424+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f633c00cc90 con 0x7f6344073170 2024-09-17T03:51:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.424+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c0146a0 con 0x7f6344073170 2024-09-17T03:51:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.424+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63440cc000 con 0x7f6344073170 2024-09-17T03:51:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.424+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63440cc2e0 con 0x7f6344073170 2024-09-17T03:51:36.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.425+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6344074640 con 0x7f6344073170 2024-09-17T03:51:36.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.426+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f633c01c070 con 0x7f6344073170 2024-09-17T03:51:36.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.427+0000 7f633be00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 0x7f632407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.427+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f633c09aa40 con 0x7f6344073170 2024-09-17T03:51:36.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.428+0000 7f6343e00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 0x7f632407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.428+0000 7f6343e00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 0x7f632407b0a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6334033000 tx=0x7f6334004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:36.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.432+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f633c062820 con 0x7f6344073170 2024-09-17T03:51:36.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.653+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63440c3860 con 0x7f6344073170 2024-09-17T03:51:36.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.654+0000 7f633be00640 1 -- 172.21.15.129:0/4070443872 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7f633c067250 con 0x7f6344073170 2024-09-17T03:51:36.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:36.654 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:36.654 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:36.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 msgr2=0x7f632407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 0x7f632407b0a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6334033000 tx=0x7f6334004730 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 msgr2=0x7f63441e9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f633c00ba10 tx=0x7f633c00bed0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 shutdown_connections 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6324078b90 0x7f632407b0a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6344073170 0x7f63441e9570 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6344070440 0x7f63440ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 --2- 172.21.15.129:0/4070443872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6344070030 0x7f63441f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 >> 172.21.15.129:0/4070443872 conn(0x7f63440b1cb0 msgr2=0x7f634409db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 shutdown_connections 2024-09-17T03:51:36.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.656+0000 7f634ac00640 1 -- 172.21.15.129:0/4070443872 wait complete. 2024-09-17T03:51:36.681 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 -- 172.21.15.129:0/2242933979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700ca800 msgr2=0x7f94700cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 --2- 172.21.15.129:0/2242933979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700ca800 0x7f94700cba00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9460005fd0 tx=0x7f946002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 -- 172.21.15.129:0/2242933979 shutdown_connections 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 --2- 172.21.15.129:0/2242933979 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 0x7f947006ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 --2- 172.21.15.129:0/2242933979 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700ca800 0x7f94700cba00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 -- 172.21.15.129:0/2242933979 >> 172.21.15.129:0/2242933979 conn(0x7f94700b1cb0 msgr2=0x7f94700b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 -- 172.21.15.129:0/2242933979 shutdown_connections 2024-09-17T03:51:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.901+0000 7f9476400640 1 -- 172.21.15.129:0/2242933979 wait complete. 2024-09-17T03:51:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.904+0000 7f9476400640 1 Processor -- start 2024-09-17T03:51:36.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.904+0000 7f9476400640 1 -- start start 2024-09-17T03:51:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.905+0000 7f9476400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.905+0000 7f9476400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 0x7f94700c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.905+0000 7f9476400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 0x7f947015e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f9476400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94700ccf40 con 0x7f94700704e0 2024-09-17T03:51:36.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f9476400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94700747d0 con 0x7f94700ca800 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f9476400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f94700cd0c0 con 0x7f94700cad30 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47192/0 (socket says 172.21.15.129:47192) 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 -- 172.21.15.129:0/614198328 learned_addr learned my addr 172.21.15.129:0/614198328 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946fe00640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 0x7f947015e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946ea00640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 0x7f94700c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 -- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 msgr2=0x7f947015e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 0x7f947015e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 -- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 msgr2=0x7f94700c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 0x7f94700c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946f400640 1 -- 172.21.15.129:0/614198328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94700c7fc0 con 0x7f94700704e0 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.906+0000 7f946ea00640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 0x7f94700c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f946f400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9460034040 tx=0x7f94600038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946003e030 con 0x7f94700704e0 2024-09-17T03:51:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9460003ca0 con 0x7f94700704e0 2024-09-17T03:51:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94700c3e40 con 0x7f94700704e0 2024-09-17T03:51:36.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9460035040 con 0x7f94700704e0 2024-09-17T03:51:36.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.907+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94700c59c0 con 0x7f94700704e0 2024-09-17T03:51:36.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.908+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94700c3c30 con 0x7f94700704e0 2024-09-17T03:51:36.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.909+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9460041030 con 0x7f94700704e0 2024-09-17T03:51:36.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.910+0000 7f9467400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 0x7f945007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:36.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.910+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f94600bb3a0 con 0x7f94700704e0 2024-09-17T03:51:36.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.911+0000 7f946ea00640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 0x7f945007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:36.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.912+0000 7f946ea00640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 0x7f945007b0a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f945c005fd0 tx=0x7f945c005ea0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:36.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:36.915+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9460083180 con 0x7f94700704e0 2024-09-17T03:51:37.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.136+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94700cd650 con 0x7f94700704e0 2024-09-17T03:51:37.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.137+0000 7f9467400640 1 -- 172.21.15.129:0/614198328 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+5235 (secure 0 0 0) 0x7f946008a0d0 con 0x7f94700704e0 2024-09-17T03:51:37.137 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:37.137 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":26,"btime":"2024-09-17T03:51:12:537303+0000","default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:09.008746+0000","modified":"2024-09-17T03:51:12.020277+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5232},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5232":{"gid":5232,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1529444990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1529444990},{"type":"v1","addr":"172.21.15.129:6835","nonce":1529444990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5232,"qdb_cluster":[5232]},"id":3}]} 2024-09-17T03:51:37.137 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 26 2024-09-17T03:51:37.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 msgr2=0x7f945007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 0x7f945007b0a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f945c005fd0 tx=0x7f945c005ea0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 msgr2=0x7f94700ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9460034040 tx=0x7f94600038a0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 shutdown_connections 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9450078b90 0x7f945007b0a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94700cad30 0x7f947015e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94700ca800 0x7f94700c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 --2- 172.21.15.129:0/614198328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94700704e0 0x7f94700ca180 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 >> 172.21.15.129:0/614198328 conn(0x7f94700b1cb0 msgr2=0x7f94700cc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 shutdown_connections 2024-09-17T03:51:37.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.139+0000 7f9476400640 1 -- 172.21.15.129:0/614198328 wait complete. 2024-09-17T03:51:37.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:37.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 -- 172.21.15.129:0/336625489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 msgr2=0x7fa910167450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 --2- 172.21.15.129:0/336625489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa910167450 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa908005fd0 tx=0x7fa90802a8d0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 -- 172.21.15.129:0/336625489 shutdown_connections 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 --2- 172.21.15.129:0/336625489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa910167450 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 -- 172.21.15.129:0/336625489 >> 172.21.15.129:0/336625489 conn(0x7fa9100b1cb0 msgr2=0x7fa9100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 -- 172.21.15.129:0/336625489 shutdown_connections 2024-09-17T03:51:37.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.384+0000 7fa917600640 1 -- 172.21.15.129:0/336625489 wait complete. 2024-09-17T03:51:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.386+0000 7fa917600640 1 Processor -- start 2024-09-17T03:51:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.387+0000 7fa917600640 1 -- start start 2024-09-17T03:51:37.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa910070030 0x7fa9101f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa91015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa910167ba0 con 0x7fa910073170 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa910075080 con 0x7fa9100b40a0 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa917600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa910167d20 con 0x7fa910070030 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa90fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa90fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47206/0 (socket says 172.21.15.129:47206) 2024-09-17T03:51:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa90fe00640 1 -- 172.21.15.129:0/3213561325 learned_addr learned my addr 172.21.15.129:0/3213561325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa914c00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa910070030 0x7fa9101f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.388+0000 7fa915600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa91015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 -- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa910070030 msgr2=0x7fa9101f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa910070030 0x7fa9101f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 -- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 msgr2=0x7fa91015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa91015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 -- 172.21.15.129:0/3213561325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9101f4a80 con 0x7fa910073170 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90fe00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c390 tx=0x7fa8fc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc017070 con 0x7fa910073170 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8fc0035f0 con 0x7fa910073170 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc003a20 con 0x7fa910073170 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.389+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91009bf30 con 0x7fa910073170 2024-09-17T03:51:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.390+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9100cfa60 con 0x7fa910073170 2024-09-17T03:51:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.391+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9100b5710 con 0x7fa910073170 2024-09-17T03:51:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.392+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa8fc01a070 con 0x7fa910073170 2024-09-17T03:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.393+0000 7fa90cc00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 0x7fa8f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.393+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7fa8fc099e60 con 0x7fa910073170 2024-09-17T03:51:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.394+0000 7fa914c00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 0x7fa8f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.394+0000 7fa914c00640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 0x7fa8f007b0a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa904007fd0 tx=0x7fa904007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:37.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.397+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa9100b5710 con 0x7fa910073170 2024-09-17T03:51:37.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.582+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa910074640 con 0x7fa910073170 2024-09-17T03:51:37.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.583+0000 7fa90cc00640 1 -- 172.21.15.129:0/3213561325 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+16721 (secure 0 0 0) 0x7fa8fc061d70 con 0x7fa910073170 2024-09-17T03:51:37.583 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:37.583 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:10.521773+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:06.922986+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:07.972487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 msgr2=0x7fa8f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 0x7fa8f007b0a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa904007fd0 tx=0x7fa904007e90 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 msgr2=0x7fa91009b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00c390 tx=0x7fa8fc00c850 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 shutdown_connections 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8f0078b90 0x7fa8f007b0a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9100b40a0 0x7fa91015f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa910073170 0x7fa91009b200 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 --2- 172.21.15.129:0/3213561325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa910070030 0x7fa9101f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.585+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 >> 172.21.15.129:0/3213561325 conn(0x7fa9100b1cb0 msgr2=0x7fa9100aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.586+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 shutdown_connections 2024-09-17T03:51:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.586+0000 7fa917600640 1 -- 172.21.15.129:0/3213561325 wait complete. 2024-09-17T03:51:37.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:51:37.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 -- 172.21.15.129:0/1641002424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0220004020 msgr2=0x7f0220004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 --2- 172.21.15.129:0/1641002424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0220004020 0x7f0220004800 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0224009b50 tx=0x7f0220005830 comp rx=0 tx=0).stop 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f023b400640 1 -- 172.21.15.129:0/1641002424 reap_dead start 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 -- 172.21.15.129:0/1641002424 shutdown_connections 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 -- 172.21.15.129:0/1641002424 >> 172.21.15.129:0/1641002424 conn(0x7f023c06d7d0 msgr2=0x7f023c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 -- 172.21.15.129:0/1641002424 shutdown_connections 2024-09-17T03:51:37.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.829+0000 7f0241e00640 1 -- 172.21.15.129:0/1641002424 wait complete. 2024-09-17T03:51:37.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.832+0000 7f0241e00640 1 Processor -- start 2024-09-17T03:51:37.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.832+0000 7f0241e00640 1 -- start start 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 0x7f023c149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f023c14c8f0 0x7f023c0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f023c15fb20 con 0x7f023c0aea40 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f023c15c9d0 con 0x7f023c14c8f0 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f0241e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f023c15fca0 con 0x7f0220004020 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f023b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.833+0000 7f023b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47236/0 (socket says 172.21.15.129:47236) 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 -- 172.21.15.129:0/4290010871 learned_addr learned my addr 172.21.15.129:0/4290010871 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f0232000640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f023c14c8f0 0x7f023c0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023aa00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 0x7f023c149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 -- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 msgr2=0x7f023c149710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 0x7f023c149710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 -- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f023c14c8f0 msgr2=0x7f023c0ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f023c14c8f0 0x7f023c0ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 -- 172.21.15.129:0/4290010871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c0aee50 con 0x7f023c0aea40 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023aa00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 0x7f023c149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f023b400640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f023c0cda40 tx=0x7f02340019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.834+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234011870 con 0x7f023c0aea40 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.835+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0234017070 con 0x7f023c0aea40 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.835+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0234011d60 con 0x7f023c0aea40 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.835+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c1fa0e0 con 0x7f023c0aea40 2024-09-17T03:51:37.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.835+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c1f9600 con 0x7f023c0aea40 2024-09-17T03:51:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.836+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c09ae60 con 0x7f023c0aea40 2024-09-17T03:51:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.837+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f023401c460 con 0x7f023c0aea40 2024-09-17T03:51:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.838+0000 7f0233e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 0x7f021807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.838+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5709+0+0 (secure 0 0 0) 0x7f023409b660 con 0x7f023c0aea40 2024-09-17T03:51:37.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.838+0000 7f023aa00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 0x7f021807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.839+0000 7f023aa00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 0x7f021807b0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f022c001880 tx=0x7f022c001d20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:37.842+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f023c0aee50 con 0x7f023c0aea40 2024-09-17T03:51:38.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.063+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f023c0ccbf0 con 0x7f023c0aea40 2024-09-17T03:51:38.063 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:51:38.063+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:51:38.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.066+0000 7f0233e00640 1 -- 172.21.15.129:0/4290010871 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7f0234063a00 con 0x7f023c0aea40 2024-09-17T03:51:38.066 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:51:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 msgr2=0x7f021807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 0x7f021807b0a0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f022c001880 tx=0x7f022c001d20 comp rx=0 tx=0).stop 2024-09-17T03:51:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 msgr2=0x7f023c09c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f023c0cda40 tx=0x7f02340019b0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 shutdown_connections 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0218078b90 0x7f021807b0a0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f023c0aea40 0x7f023c09c6e0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f023c14c8f0 0x7f023c0ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 --2- 172.21.15.129:0/4290010871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0220004020 0x7f023c149710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.069+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 >> 172.21.15.129:0/4290010871 conn(0x7f023c06d7d0 msgr2=0x7f023c161800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.070+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 shutdown_connections 2024-09-17T03:51:38.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.070+0000 7f0241e00640 1 -- 172.21.15.129:0/4290010871 wait complete. 2024-09-17T03:51:38.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:51:38.213 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 -- 172.21.15.129:0/1834003711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 msgr2=0x7f39d80cc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 --2- 172.21.15.129:0/1834003711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80cc2d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39d0005fd0 tx=0x7f39d002aa90 comp rx=0 tx=0).stop 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 -- 172.21.15.129:0/1834003711 shutdown_connections 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 --2- 172.21.15.129:0/1834003711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80cc2d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 --2- 172.21.15.129:0/1834003711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b5b70 0x7f39d815cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 -- 172.21.15.129:0/1834003711 >> 172.21.15.129:0/1834003711 conn(0x7f39d806d870 msgr2=0x7f39d806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 -- 172.21.15.129:0/1834003711 shutdown_connections 2024-09-17T03:51:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.326+0000 7f39d7e00640 1 -- 172.21.15.129:0/1834003711 wait complete. 2024-09-17T03:51:38.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.329+0000 7f39d7e00640 1 Processor -- start 2024-09-17T03:51:38.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.329+0000 7f39d7e00640 1 -- start start 2024-09-17T03:51:38.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39d8153780 0x7f39d81f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b0c50 0x7f39d8150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39d80ccd10 con 0x7f39d815d2e0 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f39d80ccb90 con 0x7f39d8153780 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.330+0000 7f39d7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f39d80cce90 con 0x7f39d80b0c50 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47264/0 (socket says 172.21.15.129:47264) 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 -- 172.21.15.129:0/3830607732 learned_addr learned my addr 172.21.15.129:0/3830607732 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39d7400640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b0c50 0x7f39d8150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39d6a00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39d8153780 0x7f39d81f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:38.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 -- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b0c50 msgr2=0x7f39d8150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b0c50 0x7f39d8150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 -- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39d8153780 msgr2=0x7f39d81f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39d8153780 0x7f39d81f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 -- 172.21.15.129:0/3830607732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d81f8c20 con 0x7f39d815d2e0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.331+0000 7f39ce000640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f39c000d770 tx=0x7f39c000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.332+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0017070 con 0x7f39d815d2e0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.332+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39c0009d20 con 0x7f39d815d2e0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.332+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d81f8f00 con 0x7f39d815d2e0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.332+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00106c0 con 0x7f39d815d2e0 2024-09-17T03:51:38.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.332+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d81f94c0 con 0x7f39d815d2e0 2024-09-17T03:51:38.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.333+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d80cd420 con 0x7f39d815d2e0 2024-09-17T03:51:38.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.334+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39c001a070 con 0x7f39d815d2e0 2024-09-17T03:51:38.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.335+0000 7f39cf400640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 0x7f39b407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:38.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.335+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f39c009aeb0 con 0x7f39d815d2e0 2024-09-17T03:51:38.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.336+0000 7f39d6a00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 0x7f39b407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:38.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.336+0000 7f39d6a00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 0x7f39b407b0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f60 tx=0x7f39c8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:38.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.339+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39c0062ba0 con 0x7f39d815d2e0 2024-09-17T03:51:38.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:38.560+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f39d81f89b0 con 0x7f39d815d2e0 2024-09-17T03:51:39.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.076+0000 7f39cf400640 1 -- 172.21.15.129:0/3830607732 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f39c00675d0 con 0x7f39d815d2e0 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 msgr2=0x7f39b407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 0x7f39b407b0d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f60 tx=0x7f39c8009040 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 msgr2=0x7f39d80b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f39c000d770 tx=0x7f39c000dc30 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 shutdown_connections 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39b4078bc0 0x7f39b407b0d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39d80b0c50 0x7f39d8150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39d815d2e0 0x7f39d80b4750 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 --2- 172.21.15.129:0/3830607732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39d8153780 0x7f39d81f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.079+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 >> 172.21.15.129:0/3830607732 conn(0x7f39d806d870 msgr2=0x7f39d81e6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:39.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.080+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 shutdown_connections 2024-09-17T03:51:39.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.080+0000 7f39d7e00640 1 -- 172.21.15.129:0/3830607732 wait complete. 2024-09-17T03:51:39.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:51:39.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 -- 172.21.15.129:0/2564573539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5784073590 msgr2=0x7f578415d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 --2- 172.21.15.129:0/2564573539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5784073590 0x7f578415d440 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f5770005fd0 tx=0x7f577002a540 comp rx=0 tx=0).stop 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 -- 172.21.15.129:0/2564573539 shutdown_connections 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 --2- 172.21.15.129:0/2564573539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5784073590 0x7f578415d440 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 -- 172.21.15.129:0/2564573539 >> 172.21.15.129:0/2564573539 conn(0x7f578406ee70 msgr2=0x7f578406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 -- 172.21.15.129:0/2564573539 shutdown_connections 2024-09-17T03:51:39.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.328+0000 7f578a000640 1 -- 172.21.15.129:0/2564573539 wait complete. 2024-09-17T03:51:39.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.331+0000 7f578a000640 1 Processor -- start 2024-09-17T03:51:39.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.331+0000 7f578a000640 1 -- start start 2024-09-17T03:51:39.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 0x7f57840c9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 0x7f57841f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57840cf490 con 0x7f57840b5310 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f57840cf310 con 0x7f5784073590 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f578a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f57840cf610 con 0x7f5784074d30 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f5789600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f5789600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47278/0 (socket says 172.21.15.129:47278) 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.332+0000 7f5789600640 1 -- 172.21.15.129:0/4165504875 learned_addr learned my addr 172.21.15.129:0/4165504875 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5783e00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 0x7f57841f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5788c00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 0x7f57840c9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 -- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 msgr2=0x7f57841f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 0x7f57841f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 -- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 msgr2=0x7f57840c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 0x7f57840c9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 -- 172.21.15.129:0/4165504875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57841e6ea0 con 0x7f57840b5310 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5783e00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 0x7f57841f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5788c00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 0x7f57840c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5789600640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f577c00b9e0 tx=0x7f577c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c003b70 con 0x7f57840b5310 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f577c003e00 con 0x7f57840b5310 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.333+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57841e7180 con 0x7f57840b5310 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.334+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c00cb80 con 0x7f57840b5310 2024-09-17T03:51:39.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.334+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57841e6870 con 0x7f57840b5310 2024-09-17T03:51:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.335+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57840cfb00 con 0x7f57840b5310 2024-09-17T03:51:39.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.336+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f577c004000 con 0x7f57840b5310 2024-09-17T03:51:39.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.337+0000 7f5780c00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 0x7f576407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:39.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.338+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5795+0+0 (secure 0 0 0) 0x7f577c09ab20 con 0x7f57840b5310 2024-09-17T03:51:39.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.338+0000 7f5788c00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 0x7f576407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:39.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.339+0000 7f5788c00640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 0x7f576407b0a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f57780016a0 tx=0x7f5778009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:39.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.342+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f577c063020 con 0x7f57840b5310 2024-09-17T03:51:39.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:39.525+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f57840b5970 con 0x7f57840b5310 2024-09-17T03:51:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.078+0000 7f5780c00640 1 -- 172.21.15.129:0/4165504875 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v47) ==== 181+0+0 (secure 0 0 0) 0x7f577c062920 con 0x7f57840b5310 2024-09-17T03:51:40.079 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:51:40.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 msgr2=0x7f576407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 0x7f576407b0a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f57780016a0 tx=0x7f5778009040 comp rx=0 tx=0).stop 2024-09-17T03:51:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 msgr2=0x7f5784143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f577c00b9e0 tx=0x7f577c00bea0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 shutdown_connections 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5764078b90 0x7f576407b0a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57840b5310 0x7f5784143f40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5784074d30 0x7f57841f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 --2- 172.21.15.129:0/4165504875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5784073590 0x7f57840c9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.082+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 >> 172.21.15.129:0/4165504875 conn(0x7f578406ee70 msgr2=0x7f578415ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.083+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 shutdown_connections 2024-09-17T03:51:40.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.083+0000 7f578a000640 1 -- 172.21.15.129:0/4165504875 wait complete. 2024-09-17T03:51:40.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:51:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 -- 172.21.15.129:0/3633825119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 msgr2=0x7f0af00cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 --2- 172.21.15.129:0/3633825119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af00cd3f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0af006fc70 tx=0x7f0ae002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 -- 172.21.15.129:0/3633825119 shutdown_connections 2024-09-17T03:51:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 --2- 172.21.15.129:0/3633825119 >> v2:172.21.15.181:3300/0 conn(0x7f0af00b5310 0x7f0af00b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 --2- 172.21.15.129:0/3633825119 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af00cd3f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 -- 172.21.15.129:0/3633825119 >> 172.21.15.129:0/3633825119 conn(0x7f0af006ee70 msgr2=0x7f0af006f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:40.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 -- 172.21.15.129:0/3633825119 shutdown_connections 2024-09-17T03:51:40.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.333+0000 7f0af5400640 1 -- 172.21.15.129:0/3633825119 wait complete. 2024-09-17T03:51:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.335+0000 7f0af5400640 1 Processor -- start 2024-09-17T03:51:40.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.335+0000 7f0af5400640 1 -- start start 2024-09-17T03:51:40.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af00b5310 0x7f0af01e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:40.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:40.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0af015d470 0x7f0af01f8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00cf130 con 0x7f0af015a120 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0af00cefb0 con 0x7f0af00b5310 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af5400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0af00cf2b0 con 0x7f0af015d470 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47302/0 (socket says 172.21.15.129:47302) 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 -- 172.21.15.129:0/1616442733 learned_addr learned my addr 172.21.15.129:0/1616442733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0aefe00640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af00b5310 0x7f0af01e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0af4a00640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0af015d470 0x7f0af01f8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:40.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 -- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0af015d470 msgr2=0x7f0af01f8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0af015d470 0x7f0af01f8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 -- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af00b5310 msgr2=0x7f0af01e4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af00b5310 0x7f0af01e4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.337+0000 7f0ae7400640 1 -- 172.21.15.129:0/1616442733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0af01e6640 con 0x7f0af015a120 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0ae7400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009fd0 tx=0x7f0ad800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8015020 con 0x7f0af015a120 2024-09-17T03:51:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ad8011920 con 0x7f0af015a120 2024-09-17T03:51:40.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad8011d70 con 0x7f0af015a120 2024-09-17T03:51:40.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0af01e55a0 con 0x7f0af015a120 2024-09-17T03:51:40.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.338+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0af01e5b60 con 0x7f0af015a120 2024-09-17T03:51:40.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.339+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0af015e320 con 0x7f0af015a120 2024-09-17T03:51:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.340+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0ad8018020 con 0x7f0af015a120 2024-09-17T03:51:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.341+0000 7f0aecc00640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 0x7f0acc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.341+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5357+0+0 (secure 0 0 0) 0x7f0ad809aad0 con 0x7f0af015a120 2024-09-17T03:51:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.342+0000 7f0aefe00640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 0x7f0acc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:40.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.342+0000 7f0aefe00640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 0x7f0acc07b0a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0033000 tx=0x7f0ae0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:40.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.346+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0af015e320 con 0x7f0af015a120 2024-09-17T03:51:40.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:40.530+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0af00cf880 con 0x7f0af015a120 2024-09-17T03:51:41.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.085+0000 7f0aecc00640 1 -- 172.21.15.129:0/1616442733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v48) ==== 169+0+0 (secure 0 0 0) 0x7f0ad801f020 con 0x7f0af015a120 2024-09-17T03:51:41.086 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:51:41.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.089+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 msgr2=0x7f0acc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.089+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 0x7f0acc07b0a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0033000 tx=0x7f0ae0004730 comp rx=0 tx=0).stop 2024-09-17T03:51:41.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.089+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 msgr2=0x7f0af01e9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.089+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f0ad8009fd0 tx=0x7f0ad800b870 comp rx=0 tx=0).stop 2024-09-17T03:51:41.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 shutdown_connections 2024-09-17T03:51:41.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0acc078b90 0x7f0acc07b0a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0af015d470 0x7f0af01f8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0af015a120 0x7f0af01e9df0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 --2- 172.21.15.129:0/1616442733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af00b5310 0x7f0af01e4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 >> 172.21.15.129:0/1616442733 conn(0x7f0af006ee70 msgr2=0x7f0af015ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:41.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.090+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 shutdown_connections 2024-09-17T03:51:41.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.091+0000 7f0af5400640 1 -- 172.21.15.129:0/1616442733 wait complete. 2024-09-17T03:51:41.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 -- 172.21.15.129:0/4263661312 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 msgr2=0x7f9a980b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/4263661312 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a980b3b50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9a88009850 tx=0x7f9a8802a5a0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 -- 172.21.15.129:0/4263661312 shutdown_connections 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/4263661312 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a980b3b50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 -- 172.21.15.129:0/4263661312 >> 172.21.15.129:0/4263661312 conn(0x7f9a9809d6a0 msgr2=0x7f9a980d43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 -- 172.21.15.129:0/4263661312 shutdown_connections 2024-09-17T03:51:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.343+0000 7f9a9cc00640 1 -- 172.21.15.129:0/4263661312 wait complete. 2024-09-17T03:51:41.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.345+0000 7f9a9cc00640 1 Processor -- start 2024-09-17T03:51:41.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.345+0000 7f9a9cc00640 1 -- start start 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a980b5a20 0x7f9a9815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a981f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a980cffa0 con 0x7f9a98162470 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9a980cfe20 con 0x7f9a980b5a20 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.346+0000 7f9a9cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980d0120 con 0x7f9a98163110 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47308/0 (socket says 172.21.15.129:47308) 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 -- 172.21.15.129:0/777038621 learned_addr learned my addr 172.21.15.129:0/777038621 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a97400640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a980b5a20 0x7f9a9815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a97e00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a981f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 -- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 msgr2=0x7f9a981f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a981f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 -- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a980b5a20 msgr2=0x7f9a9815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a980b5a20 0x7f9a9815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 -- 172.21.15.129:0/777038621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9815ec80 con 0x7f9a98162470 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.347+0000 7f9a8fe00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c430 tx=0x7f9a8000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.348+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80017070 con 0x7f9a98162470 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.348+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a800053e0 con 0x7f9a98162470 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.348+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80003850 con 0x7f9a98162470 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.348+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9815ef60 con 0x7f9a98162470 2024-09-17T03:51:41.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.348+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a981f59a0 con 0x7f9a98162470 2024-09-17T03:51:41.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.349+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a9815e9f0 con 0x7f9a98162470 2024-09-17T03:51:41.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.350+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9a8001a070 con 0x7f9a98162470 2024-09-17T03:51:41.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.351+0000 7f9a8f400640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 0x7f9a7407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.351+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f9a80099e50 con 0x7f9a98162470 2024-09-17T03:51:41.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.352+0000 7f9a97400640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 0x7f9a7407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.352+0000 7f9a97400640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 0x7f9a7407b0a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a88034000 tx=0x7f9a88004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:41.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.356+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9a80061ea0 con 0x7f9a98162470 2024-09-17T03:51:41.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.540+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f9a9808e450 con 0x7f9a98162470 2024-09-17T03:51:41.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.544+0000 7f9a8f400640 1 -- 172.21.15.129:0/777038621 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v36) ==== 71+0+0 (secure 0 0 0) 0x7f9a800668d0 con 0x7f9a98162470 2024-09-17T03:51:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 msgr2=0x7f9a7407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 0x7f9a7407b0a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9a88034000 tx=0x7f9a88004730 comp rx=0 tx=0).stop 2024-09-17T03:51:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 msgr2=0x7f9a98160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000c430 tx=0x7f9a8000c8f0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 shutdown_connections 2024-09-17T03:51:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a74078b90 0x7f9a7407b0a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a98163110 0x7f9a981f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a98162470 0x7f9a98160a20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 --2- 172.21.15.129:0/777038621 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a980b5a20 0x7f9a9815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 >> 172.21.15.129:0/777038621 conn(0x7f9a9809d6a0 msgr2=0x7f9a9806db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:41.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 shutdown_connections 2024-09-17T03:51:41.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.547+0000 7f9a9cc00640 1 -- 172.21.15.129:0/777038621 wait complete. 2024-09-17T03:51:41.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit' 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 -- 172.21.15.129:0/878461622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf950 msgr2=0x7ff49406f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 --2- 172.21.15.129:0/878461622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf950 0x7ff49406f3b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ff48c005fd0 tx=0x7ff48c02aa10 comp rx=0 tx=0).stop 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 -- 172.21.15.129:0/878461622 shutdown_connections 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 --2- 172.21.15.129:0/878461622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf950 0x7ff49406f3b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 --2- 172.21.15.129:0/878461622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4940cf540 0x7ff49406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 -- 172.21.15.129:0/878461622 >> 172.21.15.129:0/878461622 conn(0x7ff49406d410 msgr2=0x7ff4940b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 -- 172.21.15.129:0/878461622 shutdown_connections 2024-09-17T03:51:41.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.790+0000 7ff493e00640 1 -- 172.21.15.129:0/878461622 wait complete. 2024-09-17T03:51:41.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.793+0000 7ff493e00640 1 Processor -- start 2024-09-17T03:51:41.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.793+0000 7ff493e00640 1 -- start start 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4940cf950 0x7ff4940b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff49415ca50 0x7ff4941560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff49415de20 con 0x7ff4940cf540 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff49415dca0 con 0x7ff4940cf950 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff493e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff49415dfa0 con 0x7ff49415ca50 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff492a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff492a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47326/0 (socket says 172.21.15.129:47326) 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.794+0000 7ff492a00640 1 -- 172.21.15.129:0/2896540943 learned_addr learned my addr 172.21.15.129:0/2896540943 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492000640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4940cf950 0x7ff4940b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff493400640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff49415ca50 0x7ff4941560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 -- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff49415ca50 msgr2=0x7ff4941560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff49415ca50 0x7ff4941560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 -- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4940cf950 msgr2=0x7ff4940b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4940cf950 0x7ff4940b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 -- 172.21.15.129:0/2896540943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff494144d20 con 0x7ff4940cf540 2024-09-17T03:51:41.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff492a00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff48400c370 tx=0x7ff48400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4840052d0 con 0x7ff4940cf540 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff484007020 con 0x7ff4940cf540 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.795+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4941434e0 con 0x7ff4940cf540 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.796+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484007470 con 0x7ff4940cf540 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.796+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff494154800 con 0x7ff4940cf540 2024-09-17T03:51:41.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.797+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff49415ec10 con 0x7ff4940cf540 2024-09-17T03:51:41.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.798+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff4840067d0 con 0x7ff4940cf540 2024-09-17T03:51:41.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.799+0000 7ff48aa00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 0x7ff47407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:41.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.799+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7ff484099ee0 con 0x7ff4940cf540 2024-09-17T03:51:41.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.800+0000 7ff492000640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 0x7ff47407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:41.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.800+0000 7ff492000640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 0x7ff47407b0a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff494156f20 tx=0x7ff480005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:41.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:41.803+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff484061fc0 con 0x7ff4940cf540 2024-09-17T03:51:42.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.056+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]} v 0) -- 0x7ff4940b2360 con 0x7ff4940cf540 2024-09-17T03:51:42.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.087+0000 7ff48aa00640 1 -- 172.21.15.129:0/2896540943 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit"]}]=0 v133) ==== 133+0+0 (secure 0 0 0) 0x7ff4840669f0 con 0x7ff4940cf540 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 msgr2=0x7ff47407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 0x7ff47407b0a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff494156f20 tx=0x7ff480005f90 comp rx=0 tx=0).stop 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 msgr2=0x7ff494153ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7ff48400c370 tx=0x7ff48400c830 comp rx=0 tx=0).stop 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 shutdown_connections 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff474078b90 0x7ff47407b0a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff49415ca50 0x7ff4941560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4940cf950 0x7ff4940b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 --2- 172.21.15.129:0/2896540943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4940cf540 0x7ff494153ae0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.089+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 >> 172.21.15.129:0/2896540943 conn(0x7ff49406d410 msgr2=0x7ff494142bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:42.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.090+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 shutdown_connections 2024-09-17T03:51:42.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.090+0000 7ff493e00640 1 -- 172.21.15.129:0/2896540943 wait complete. 2024-09-17T03:51:42.116 INFO:tasks.cephfs_test_runner:That the directory fragment size cannot exceed mds_bal_fragment_size_max (using a limit of 50 in all configurations). ... ok 2024-09-17T03:51:42.116 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) That fragmentation (forced) will allow more entries to be created. 2024-09-17T03:51:42.116 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_fragmented (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:51:42.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 -- 172.21.15.129:0/817108148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3ac004800 msgr2=0x7fd3ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/817108148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3ac004800 0x7fd3ac004c10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8072320 tx=0x7fd3ac005830 comp rx=0 tx=0).stop 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3c7400640 1 -- 172.21.15.129:0/817108148 reap_dead start 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 -- 172.21.15.129:0/817108148 shutdown_connections 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 -- 172.21.15.129:0/817108148 >> 172.21.15.129:0/817108148 conn(0x7fd3c80b1cb0 msgr2=0x7fd3c80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:42.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 -- 172.21.15.129:0/817108148 shutdown_connections 2024-09-17T03:51:42.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.346+0000 7fd3cdc00640 1 -- 172.21.15.129:0/817108148 wait complete. 2024-09-17T03:51:42.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.349+0000 7fd3cdc00640 1 Processor -- start 2024-09-17T03:51:42.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.349+0000 7fd3cdc00640 1 -- start start 2024-09-17T03:51:42.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 0x7fd3c80c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd3c80d0a50 0x7fd3c81f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c806faa0 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd3c806f920 con 0x7fd3ac004800 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.350+0000 7fd3cdc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd3c806fc20 con 0x7fd3c80d0a50 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47344/0 (socket says 172.21.15.129:47344) 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 -- 172.21.15.129:0/2418213384 learned_addr learned my addr 172.21.15.129:0/2418213384 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c6a00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 0x7fd3c80c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3be000640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd3c80d0a50 0x7fd3c81f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 -- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd3c80d0a50 msgr2=0x7fd3c81f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd3c80d0a50 0x7fd3c81f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 -- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 msgr2=0x7fd3c80c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 0x7fd3c80c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 -- 172.21.15.129:0/2418213384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c81f6880 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c6a00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 0x7fd3c80c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3c7400640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8073600 tx=0x7fd3c0006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000fa80 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd3c000fc20 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.351+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3c80c82e0 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.352+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0011430 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.352+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3c80c88a0 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.353+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3c80c8050 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.354+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd3c00115d0 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.355+0000 7fd3bf400640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 0x7fd3a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.355+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fd3c009b400 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.356+0000 7fd3c6a00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 0x7fd3a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.356+0000 7fd3c6a00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 0x7fd3a407b0a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009c50 tx=0x7fd3b8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:42.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.360+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd3c0063b40 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.546+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3c815c780 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.547+0000 7fd3bf400640 1 -- 172.21.15.129:0/2418213384 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+13082 (secure 0 0 0) 0x7fd3c0063440 con 0x7fd3c80cf3d0 2024-09-17T03:51:42.547 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:42.547 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":48,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:41.080372+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/1529444990":"2024-09-18T03:51:38.064733+0000","172.21.15.129:6833/1529444990":"2024-09-18T03:51:38.064733+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:42.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 msgr2=0x7fd3a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 0x7fd3a407b0a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009c50 tx=0x7fd3b8005f90 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 msgr2=0x7fd3c80cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8073600 tx=0x7fd3c0006720 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 shutdown_connections 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3a4078b90 0x7fd3a407b0a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3c80cf3d0 0x7fd3c80cf7e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd3c80d0a50 0x7fd3c81f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 --2- 172.21.15.129:0/2418213384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3ac004800 0x7fd3c80c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 >> 172.21.15.129:0/2418213384 conn(0x7fd3c80b1cb0 msgr2=0x7fd3c80b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:42.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 shutdown_connections 2024-09-17T03:51:42.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.550+0000 7fd3cdc00640 1 -- 172.21.15.129:0/2418213384 wait complete. 2024-09-17T03:51:42.573 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:51:42.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 -- 172.21.15.129:0/3726885839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980af3c0 msgr2=0x7f9e980b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 --2- 172.21.15.129:0/3726885839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980af3c0 0x7f9e980b40b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9e98072dc0 tx=0x7f9e8002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 -- 172.21.15.129:0/3726885839 shutdown_connections 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 --2- 172.21.15.129:0/3726885839 >> v2:172.21.15.181:3300/0 conn(0x7f9e980ccb50 0x7f9e980ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 --2- 172.21.15.129:0/3726885839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980af3c0 0x7f9e980b40b0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 -- 172.21.15.129:0/3726885839 >> 172.21.15.129:0/3726885839 conn(0x7f9e9806d7d0 msgr2=0x7f9e980b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 -- 172.21.15.129:0/3726885839 shutdown_connections 2024-09-17T03:51:42.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.797+0000 7f9e9d200640 1 -- 172.21.15.129:0/3726885839 wait complete. 2024-09-17T03:51:42.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.800+0000 7f9e9d200640 1 Processor -- start 2024-09-17T03:51:42.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.800+0000 7f9e9d200640 1 -- start start 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e980af3c0 0x7f9e9815f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 0x7f9e98152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e981533a0 con 0x7f9e980ccb50 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e980af8e0 con 0x7f9e980af3c0 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e9d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e98153520 con 0x7f9e980cc1e0 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.801+0000 7f9e96800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47356/0 (socket says 172.21.15.129:47356) 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 -- 172.21.15.129:0/713880643 learned_addr learned my addr 172.21.15.129:0/713880643 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e95e00640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e980af3c0 0x7f9e9815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e8d400640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 0x7f9e98152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 -- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 msgr2=0x7f9e98152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 0x7f9e98152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 -- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e980af3c0 msgr2=0x7f9e9815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e980af3c0 0x7f9e9815f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 -- 172.21.15.129:0/713880643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e980735a0 con 0x7f9e980ccb50 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e8d400640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 0x7f9e98152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.802+0000 7f9e96800640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000ea80 tx=0x7f9e9000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.803+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90004b20 con 0x7f9e980ccb50 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.803+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e90004ce0 con 0x7f9e980ccb50 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.803+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90005280 con 0x7f9e980ccb50 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.803+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e98073880 con 0x7f9e980ccb50 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.803+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e98073e40 con 0x7f9e980ccb50 2024-09-17T03:51:42.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.804+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e98153a40 con 0x7f9e980ccb50 2024-09-17T03:51:42.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.805+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e90005420 con 0x7f9e980ccb50 2024-09-17T03:51:42.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.806+0000 7f9e8ea00640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 0x7f9e7407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:42.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.806+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f9e9009b750 con 0x7f9e980ccb50 2024-09-17T03:51:42.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.807+0000 7f9e95e00640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 0x7f9e7407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:42.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.808+0000 7f9e95e00640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 0x7f9e7407b0a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e88001720 tx=0x7f9e88009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:42.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.811+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e90063df0 con 0x7f9e980ccb50 2024-09-17T03:51:42.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.992+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e98073330 con 0x7f9e980ccb50 2024-09-17T03:51:42.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.993+0000 7f9e8ea00640 1 -- 172.21.15.129:0/713880643 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f9e900636f0 con 0x7f9e980ccb50 2024-09-17T03:51:42.993 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:51:42.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 msgr2=0x7f9e7407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 0x7f9e7407b0a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e88001720 tx=0x7f9e88009040 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 msgr2=0x7f9e980c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000ea80 tx=0x7f9e9000ef40 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 shutdown_connections 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e74078b90 0x7f9e7407b0a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e980ccb50 0x7f9e980c9610 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e980cc1e0 0x7f9e98152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 --2- 172.21.15.129:0/713880643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e980af3c0 0x7f9e9815f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 >> 172.21.15.129:0/713880643 conn(0x7f9e9806d7d0 msgr2=0x7f9e980ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 shutdown_connections 2024-09-17T03:51:42.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:42.995+0000 7f9e9d200640 1 -- 172.21.15.129:0/713880643 wait complete. 2024-09-17T03:51:43.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-09-17T03:51:43.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 -- 172.21.15.129:0/2730495712 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0cada0 msgr2=0x7f07bc15c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:43.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 --2- 172.21.15.129:0/2730495712 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0cada0 0x7f07bc15c4d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f07a4005fd0 tx=0x7f07a402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:51:43.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 -- 172.21.15.129:0/2730495712 shutdown_connections 2024-09-17T03:51:43.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 --2- 172.21.15.129:0/2730495712 >> v2:172.21.15.181:3300/0 conn(0x7f07bc0b4a90 0x7f07bc0cfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:43.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 --2- 172.21.15.129:0/2730495712 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0cada0 0x7f07bc15c4d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:43.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 -- 172.21.15.129:0/2730495712 >> 172.21.15.129:0/2730495712 conn(0x7f07bc06d870 msgr2=0x7f07bc06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:43.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 -- 172.21.15.129:0/2730495712 shutdown_connections 2024-09-17T03:51:43.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.241+0000 7f07c2600640 1 -- 172.21.15.129:0/2730495712 wait complete. 2024-09-17T03:51:43.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.243+0000 7f07c2600640 1 Processor -- start 2024-09-17T03:51:43.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.243+0000 7f07c2600640 1 -- start start 2024-09-17T03:51:43.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0b4a90 0x7f07bc159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07bc0cada0 0x7f07bc0cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc0d05a0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc0d0420 con 0x7f07bc0cada0 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c2600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f07bc0d0720 con 0x7f07bc0b4a90 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07b3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07b3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47360/0 (socket says 172.21.15.129:47360) 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07b3e00640 1 -- 172.21.15.129:0/4189956824 learned_addr learned my addr 172.21.15.129:0/4189956824 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c1200640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0b4a90 0x7f07bc159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.245+0000 7f07c1c00640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07bc0cada0 0x7f07bc0cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 -- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0b4a90 msgr2=0x7f07bc159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:43.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0b4a90 0x7f07bc159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 -- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07bc0cada0 msgr2=0x7f07bc0cca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07bc0cada0 0x7f07bc0cca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 -- 172.21.15.129:0/4189956824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07bc159bf0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b3e00640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07a4034040 tx=0x7f07a4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a403e030 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07bc159ed0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07a40039f0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4035040 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.246+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07bc15a490 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.247+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07bc15a6d0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.249+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f07a4041030 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.249+0000 7f07b9600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 0x7f079807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:43.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.250+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f07a40bae40 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.250+0000 7f07c1200640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 0x7f079807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:43.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.251+0000 7f07c1200640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 0x7f079807b0a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0017e0 tx=0x7f07ac009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:43.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.254+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f07bc15a6d0 con 0x7f07bc0ca5a0 2024-09-17T03:51:43.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:43.505+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) -- 0x7f07bc159980 con 0x7f07bc0ca5a0 2024-09-17T03:51:44.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.100+0000 7f07b9600640 1 -- 172.21.15.129:0/4189956824 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v135) ==== 147+0+0 (secure 0 0 0) 0x7f07a4082e10 con 0x7f07bc0ca5a0 2024-09-17T03:51:44.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 msgr2=0x7f079807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 0x7f079807b0a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0017e0 tx=0x7f07ac009040 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 msgr2=0x7f07bc1430c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f07a4034040 tx=0x7f07a4005e80 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 shutdown_connections 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0798078b90 0x7f079807b0a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f07bc0cada0 0x7f07bc0cca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07bc0ca5a0 0x7f07bc1430c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 --2- 172.21.15.129:0/4189956824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07bc0b4a90 0x7f07bc159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.103+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 >> 172.21.15.129:0/4189956824 conn(0x7f07bc06d870 msgr2=0x7f07bc15dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.104+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 shutdown_connections 2024-09-17T03:51:44.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.104+0000 7f07c2600640 1 -- 172.21.15.129:0/4189956824 wait complete. 2024-09-17T03:51:44.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 -- 172.21.15.129:0/2935838661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070440 msgr2=0x7f7ce8074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 --2- 172.21.15.129:0/2935838661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070440 0x7f7ce8074b80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8160370 tx=0x7f7ce002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 -- 172.21.15.129:0/2935838661 shutdown_connections 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 --2- 172.21.15.129:0/2935838661 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070440 0x7f7ce8074b80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 --2- 172.21.15.129:0/2935838661 >> v2:172.21.15.181:3300/0 conn(0x7f7ce8070030 0x7f7ce8074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 --2- 172.21.15.129:0/2935838661 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8073170 0x7f7ce815c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 -- 172.21.15.129:0/2935838661 >> 172.21.15.129:0/2935838661 conn(0x7f7ce80b1cb0 msgr2=0x7f7ce80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 -- 172.21.15.129:0/2935838661 shutdown_connections 2024-09-17T03:51:44.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.352+0000 7f7cee800640 1 -- 172.21.15.129:0/2935838661 wait complete. 2024-09-17T03:51:44.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.355+0000 7f7cee800640 1 Processor -- start 2024-09-17T03:51:44.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.355+0000 7f7cee800640 1 -- start start 2024-09-17T03:51:44.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8070440 0x7f7ce809d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce8073170 0x7f7ce80cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce806f690 con 0x7f7ce8070030 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce806f510 con 0x7f7ce8073170 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7cee800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7ce806f810 con 0x7f7ce8070440 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7ce7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7ce7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49586/0 (socket says 172.21.15.129:49586) 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.356+0000 7f7ce7400640 1 -- 172.21.15.129:0/2342311886 learned_addr learned my addr 172.21.15.129:0/2342311886 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7e00640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce8073170 0x7f7ce80cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7cdea00640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8070440 0x7f7ce809d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 -- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8070440 msgr2=0x7f7ce809d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8070440 0x7f7ce809d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 -- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce8073170 msgr2=0x7f7ce80cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce8073170 0x7f7ce80cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 -- 172.21.15.129:0/2342311886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce81e57f0 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7ce7400640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800ea20 tx=0x7f7cd800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd800cde0 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cd8011560 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce809ac10 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.357+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8011990 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.358+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce809b1d0 con 0x7f7ce8070030 2024-09-17T03:51:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.358+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce80c3860 con 0x7f7ce8070030 2024-09-17T03:51:44.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.360+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7cd8011b30 con 0x7f7ce8070030 2024-09-17T03:51:44.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.361+0000 7f7cdfe00640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 0x7f7cc407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.361+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f7cd809b6f0 con 0x7f7ce8070030 2024-09-17T03:51:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.361+0000 7f7cdea00640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 0x7f7cc407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.362+0000 7f7cdea00640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 0x7f7cc407b0a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005fd0 tx=0x7f7cd0005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:44.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.365+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7cd8016030 con 0x7f7ce8070030 2024-09-17T03:51:44.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.550+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f7ce80ae2c0 con 0x7f7ce8070030 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.551+0000 7f7cdfe00640 1 -- 172.21.15.129:0/2342311886 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v36) ==== 59+0+2294 (secure 0 0 0) 0x7f7cd8011e20 con 0x7f7ce8070030 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:51:44.551 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:51:44.552 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:51:44.553 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 msgr2=0x7f7cc407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 0x7f7cc407b0a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0005fd0 tx=0x7f7cd0005850 comp rx=0 tx=0).stop 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 msgr2=0x7f7ce81e4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800ea20 tx=0x7f7cd800eee0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 shutdown_connections 2024-09-17T03:51:44.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7cc4078b90 0x7f7cc407b0a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ce8073170 0x7f7ce80cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ce8070440 0x7f7ce809d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 --2- 172.21.15.129:0/2342311886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ce8070030 0x7f7ce81e4f80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 >> 172.21.15.129:0/2342311886 conn(0x7f7ce80b1cb0 msgr2=0x7f7ce80b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 shutdown_connections 2024-09-17T03:51:44.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.553+0000 7f7cee800640 1 -- 172.21.15.129:0/2342311886 wait complete. 2024-09-17T03:51:44.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:51:44.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 -- 172.21.15.129:0/4271179598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c806f580 msgr2=0x7fe4c815d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 --2- 172.21.15.129:0/4271179598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c806f580 0x7fe4c815d700 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0005fd0 tx=0x7fe4c002aa10 comp rx=0 tx=0).stop 2024-09-17T03:51:44.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 -- 172.21.15.129:0/4271179598 shutdown_connections 2024-09-17T03:51:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 --2- 172.21.15.129:0/4271179598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c806f580 0x7fe4c815d700 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 --2- 172.21.15.129:0/4271179598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 0x7fe4c80cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.803+0000 7fe4ce000640 1 -- 172.21.15.129:0/4271179598 >> 172.21.15.129:0/4271179598 conn(0x7fe4c806d410 msgr2=0x7fe4c80b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.804+0000 7fe4ce000640 1 -- 172.21.15.129:0/4271179598 shutdown_connections 2024-09-17T03:51:44.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.804+0000 7fe4ce000640 1 -- 172.21.15.129:0/4271179598 wait complete. 2024-09-17T03:51:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.806+0000 7fe4ce000640 1 Processor -- start 2024-09-17T03:51:44.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.806+0000 7fe4ce000640 1 -- start start 2024-09-17T03:51:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 0x7fe4c813c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 0x7fe4c8142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c815dfd0 con 0x7fe4c81429d0 2024-09-17T03:51:44.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c815de50 con 0x7fe4c80cf540 2024-09-17T03:51:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.807+0000 7fe4ce000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c815e150 con 0x7fe4c815ca50 2024-09-17T03:51:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4ccc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 0x7fe4c813c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4ccc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 0x7fe4c813c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:33442/0 (socket says 172.21.15.129:33442) 2024-09-17T03:51:44.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4ccc00640 1 -- 172.21.15.129:0/70494450 learned_addr learned my addr 172.21.15.129:0/70494450 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49592/0 (socket says 172.21.15.129:49592) 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4bfe00640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 0x7fe4c8142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 -- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 msgr2=0x7fe4c8142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 0x7fe4c8142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 -- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 msgr2=0x7fe4c813c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 0x7fe4c813c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 -- 172.21.15.129:0/70494450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c813cb70 con 0x7fe4c81429d0 2024-09-17T03:51:44.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.808+0000 7fe4cd600640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0032000 tx=0x7fe4c0003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:44.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.809+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0033050 con 0x7fe4c81429d0 2024-09-17T03:51:44.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.809+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c81f8f60 con 0x7fe4c81429d0 2024-09-17T03:51:44.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.809+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4c0004250 con 0x7fe4c81429d0 2024-09-17T03:51:44.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.809+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c00046a0 con 0x7fe4c81429d0 2024-09-17T03:51:44.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.809+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c81f9520 con 0x7fe4c81429d0 2024-09-17T03:51:44.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.810+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c815d700 con 0x7fe4c81429d0 2024-09-17T03:51:44.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.811+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe4c0036070 con 0x7fe4c81429d0 2024-09-17T03:51:44.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.812+0000 7fe4c5600640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 0x7fe4a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.812+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7fe4c00bb9c0 con 0x7fe4c81429d0 2024-09-17T03:51:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.813+0000 7fe4ccc00640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 0x7fe4a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:44.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.813+0000 7fe4ccc00640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 0x7fe4a407b0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009bf0 tx=0x7fe4b8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:44.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:44.817+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe4c813cb70 con 0x7fe4c81429d0 2024-09-17T03:51:45.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.003+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe4c8142f90 con 0x7fe4c81429d0 2024-09-17T03:51:45.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.007+0000 7fe4c5600640 1 -- 172.21.15.129:0/70494450 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v37)=0 v37) ==== 122+0+0 (secure 0 0 0) 0x7fe4c0083a20 con 0x7fe4c81429d0 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.009+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 msgr2=0x7fe4a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.009+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 0x7fe4a407b0d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8009bf0 tx=0x7fe4b8005f90 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.009+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 msgr2=0x7fe4c815b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.009+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0032000 tx=0x7fe4c0003ea0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 shutdown_connections 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe4a4078bc0 0x7fe4a407b0d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4c81429d0 0x7fe4c815b470 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4c815ca50 0x7fe4c8142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 --2- 172.21.15.129:0/70494450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4c80cf540 0x7fe4c813c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 >> 172.21.15.129:0/70494450 conn(0x7fe4c806d410 msgr2=0x7fe4c815ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 shutdown_connections 2024-09-17T03:51:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.010+0000 7fe4ce000640 1 -- 172.21.15.129:0/70494450 wait complete. 2024-09-17T03:51:45.031 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:51:45.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:45.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:51:45.210 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:51:45.210 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:45.210 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:45.246 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:51:45.246 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:45.246 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:45.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:45.283 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:45.283 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:45.321 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:51:45.321 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:51:45.321 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:45.358 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:51:45.359 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:51:45.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:45.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 -- 172.21.15.129:0/1782535991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 msgr2=0x7f29f40cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 --2- 172.21.15.129:0/1782535991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f40cc4c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f29e4005fd0 tx=0x7f29e402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 -- 172.21.15.129:0/1782535991 shutdown_connections 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 --2- 172.21.15.129:0/1782535991 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 0x7f29f40ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 --2- 172.21.15.129:0/1782535991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f40cc4c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 -- 172.21.15.129:0/1782535991 >> 172.21.15.129:0/1782535991 conn(0x7f29f40b1cb0 msgr2=0x7f29f40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 -- 172.21.15.129:0/1782535991 shutdown_connections 2024-09-17T03:51:45.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.581+0000 7f29fa000640 1 -- 172.21.15.129:0/1782535991 wait complete. 2024-09-17T03:51:45.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.583+0000 7f29fa000640 1 Processor -- start 2024-09-17T03:51:45.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.584+0000 7f29fa000640 1 -- start start 2024-09-17T03:51:45.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 0x7f29f414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 0x7f29f41f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29f40cf1c0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29f40af940 con 0x7f29f40ce020 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29fa000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f29f40cf340 con 0x7f29f40ccb30 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49612/0 (socket says 172.21.15.129:49612) 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.585+0000 7f29f3e00640 1 -- 172.21.15.129:0/3829956938 learned_addr learned my addr 172.21.15.129:0/3829956938 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3400640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 0x7f29f414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29eaa00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 0x7f29f41f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 -- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 msgr2=0x7f29f414cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 0x7f29f414cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 -- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 msgr2=0x7f29f41f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 0x7f29f41f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 -- 172.21.15.129:0/3829956938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f41f6dc0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29eaa00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 0x7f29f41f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29f3e00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00c730 tx=0x7f29ec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec017070 con 0x7f29f40cf6f0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29ec003c60 con 0x7f29f40cf6f0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f41fa1d0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.586+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0052d0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.587+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f41604a0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.587+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f40ccf40 con 0x7f29f40cf6f0 2024-09-17T03:51:45.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.589+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f29ec01a070 con 0x7f29f40cf6f0 2024-09-17T03:51:45.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.590+0000 7f29ebe00640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 0x7f29d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:45.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.590+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f29ec09a190 con 0x7f29f40cf6f0 2024-09-17T03:51:45.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.590+0000 7f29f3400640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 0x7f29d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:45.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.591+0000 7f29f3400640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 0x7f29d007b0a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f29e4033000 tx=0x7f29e4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:45.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.594+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f29f41f6dc0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.814+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29f40cc7e0 con 0x7f29f40cf6f0 2024-09-17T03:51:45.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.814+0000 7f29ebe00640 1 -- 172.21.15.129:0/3829956938 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+3923 (secure 0 0 0) 0x7f29f40cd480 con 0x7f29f40cf6f0 2024-09-17T03:51:45.814 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:45.815 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":28,"btime":"2024-09-17T03:51:39:067708+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4871,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2749789025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2749789025},{"type":"v1","addr":"172.21.15.181:6836","nonce":2749789025}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5176,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2721158002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2721158002},{"type":"v1","addr":"172.21.15.181:6837","nonce":2721158002}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5229,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080854228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080854228},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080854228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3152373926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3152373926},{"type":"v1","addr":"172.21.15.129:6835","nonce":3152373926}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[]} 2024-09-17T03:51:45.815 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 28 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 msgr2=0x7f29d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 0x7f29d007b0a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f29e4033000 tx=0x7f29e4004730 comp rx=0 tx=0).stop 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 msgr2=0x7f29f41f31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00c730 tx=0x7f29ec00cbf0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 shutdown_connections 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29d0078b90 0x7f29d007b0a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29f40cf6f0 0x7f29f41f31f0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29f40ce020 0x7f29f41f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 --2- 172.21.15.129:0/3829956938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29f40ccb30 0x7f29f414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 >> 172.21.15.129:0/3829956938 conn(0x7f29f40b1cb0 msgr2=0x7f29f40732f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 shutdown_connections 2024-09-17T03:51:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:45.817+0000 7f29fa000640 1 -- 172.21.15.129:0/3829956938 wait complete. 2024-09-17T03:51:45.841 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:51:45.841 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:51:45.842 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:51:45.842 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:45.842 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:51:45.842 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:51:45.842 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:51:45.842 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:45.843 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:51:45.843 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:51:45.843 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:51:45.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:45.843 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:51:45.844 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:51:45.844 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:51:45.844 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:51:45.866 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:51:45.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:51:45.868 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:51:45.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:51:45.869 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:51:45.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:51:45.870 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:51:45.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:51:45.911 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:51:45.911 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:45.912 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:51:45.912 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:45.914 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:51:45.914 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:45.915 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:51:45.915 DEBUG:teuthology.parallel:result is None 2024-09-17T03:51:45.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:51:45.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:51:45.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:51:46.108 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:51:46.108 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:51:46.113 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:51:46.116 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:51:46.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.148+0000 7f8665c00640 1 -- 172.21.15.129:0/3414305491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142270 msgr2=0x7f8660073e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:46.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.148+0000 7f8665c00640 1 --2- 172.21.15.129:0/3414305491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142270 0x7f8660073e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f864c009860 tx=0x7f864c02a5a0 comp rx=0 tx=0).stop 2024-09-17T03:51:46.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.148+0000 7f8665c00640 1 -- 172.21.15.129:0/3414305491 shutdown_connections 2024-09-17T03:51:46.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.148+0000 7f8665c00640 1 --2- 172.21.15.129:0/3414305491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142270 0x7f8660073e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:46.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.148+0000 7f8665c00640 1 -- 172.21.15.129:0/3414305491 >> 172.21.15.129:0/3414305491 conn(0x7f86600d15f0 msgr2=0x7f8660073740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:46.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.149+0000 7f8665c00640 1 -- 172.21.15.129:0/3414305491 shutdown_connections 2024-09-17T03:51:46.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.149+0000 7f8665c00640 1 -- 172.21.15.129:0/3414305491 wait complete. 2024-09-17T03:51:46.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.151+0000 7f8665c00640 1 Processor -- start 2024-09-17T03:51:46.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.151+0000 7f8665c00640 1 -- start start 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142680 0x7f866015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8660144270 0x7f866013adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86600d0780 con 0x7f8660142270 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8660074390 con 0x7f8660144270 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.152+0000 7f8665c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f866015c820 con 0x7f8660142680 2024-09-17T03:51:46.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49640/0 (socket says 172.21.15.129:49640) 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 -- 172.21.15.129:0/2765245077 learned_addr learned my addr 172.21.15.129:0/2765245077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865e000640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142680 0x7f866015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865f400640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8660144270 0x7f866013adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 -- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142680 msgr2=0x7f866015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142680 0x7f866015b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 -- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8660144270 msgr2=0x7f866013adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8660144270 0x7f866013adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 -- 172.21.15.129:0/2765245077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660159fb0 con 0x7f8660142270 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.153+0000 7f865ea00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f865000c400 tx=0x7f865000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:46.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.154+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650011790 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.154+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8650017070 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.154+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8650011cc0 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.154+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f866015a290 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.154+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f866015a850 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.155+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8660073e50 con 0x7f8660142270 2024-09-17T03:51:46.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.156+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f865001c460 con 0x7f8660142270 2024-09-17T03:51:46.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.157+0000 7f8656a00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 0x7f864007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:46.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.157+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4943+0+0 (secure 0 0 0) 0x7f865009a750 con 0x7f8660142270 2024-09-17T03:51:46.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.158+0000 7f865e000640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 0x7f864007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:46.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.158+0000 7f865e000640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 0x7f864007b0a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f864c034000 tx=0x7f864c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:46.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.161+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f86500627a0 con 0x7f8660142270 2024-09-17T03:51:46.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:46.356+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f866015b590 con 0x7f8660142270 2024-09-17T03:51:47.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.127+0000 7f8656a00640 1 -- 172.21.15.129:0/2765245077 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) ==== 97+0+0 (secure 0 0 0) 0x7f86500671d0 con 0x7f8660142270 2024-09-17T03:51:47.127 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:51:47.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 msgr2=0x7f864007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 0x7f864007b0a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f864c034000 tx=0x7f864c004730 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 msgr2=0x7f86601597f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f865000c400 tx=0x7f865000c8c0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 shutdown_connections 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8640078b90 0x7f864007b0a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8660144270 0x7f866013adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8660142680 0x7f866015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 --2- 172.21.15.129:0/2765245077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8660142270 0x7f86601597f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.130+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 >> 172.21.15.129:0/2765245077 conn(0x7f86600d15f0 msgr2=0x7f8660168c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.131+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 shutdown_connections 2024-09-17T03:51:47.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.131+0000 7f8665c00640 1 -- 172.21.15.129:0/2765245077 wait complete. 2024-09-17T03:51:47.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.383+0000 7f97ce000640 1 -- 172.21.15.129:0/3606959419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8070030 msgr2=0x7f97c8074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.383+0000 7f97ce000640 1 --2- 172.21.15.129:0/3606959419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8070030 0x7f97c8074640 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005fd0 tx=0x7f97b402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.383+0000 7f97ce000640 1 -- 172.21.15.129:0/3606959419 shutdown_connections 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.383+0000 7f97ce000640 1 --2- 172.21.15.129:0/3606959419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 0x7f97c8074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.384+0000 7f97ce000640 1 --2- 172.21.15.129:0/3606959419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8070030 0x7f97c8074640 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.384+0000 7f97ce000640 1 -- 172.21.15.129:0/3606959419 >> 172.21.15.129:0/3606959419 conn(0x7f97c80b1cb0 msgr2=0x7f97c80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.384+0000 7f97ce000640 1 -- 172.21.15.129:0/3606959419 shutdown_connections 2024-09-17T03:51:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.384+0000 7f97ce000640 1 -- 172.21.15.129:0/3606959419 wait complete. 2024-09-17T03:51:47.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.386+0000 7f97ce000640 1 Processor -- start 2024-09-17T03:51:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.386+0000 7f97ce000640 1 -- start start 2024-09-17T03:51:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c8070030 0x7f97c81f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 0x7f97c80d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80b3a40 con 0x7f97c8073170 2024-09-17T03:51:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97c80b38c0 con 0x7f97c8070030 2024-09-17T03:51:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.387+0000 7f97ce000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f97c80b3bc0 con 0x7f97c8070440 2024-09-17T03:51:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49660/0 (socket says 172.21.15.129:49660) 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 -- 172.21.15.129:0/3264456998 learned_addr learned my addr 172.21.15.129:0/3264456998 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7400640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c8070030 0x7f97c81f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c6a00640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 0x7f97c80d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 -- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 msgr2=0x7f97c80d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 0x7f97c80d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 -- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c8070030 msgr2=0x7f97c81f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c8070030 0x7f97c81f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.388+0000 7f97c7e00640 1 -- 172.21.15.129:0/3264456998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c81f6640 con 0x7f97c8073170 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.389+0000 7f97c7e00640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f97c000ea40 tx=0x7f97c000ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.390+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c000ce40 con 0x7f97c8073170 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.390+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97c0004c10 con 0x7f97c8073170 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.390+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c81f6920 con 0x7f97c8073170 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.390+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0005020 con 0x7f97c8073170 2024-09-17T03:51:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.390+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c81e6950 con 0x7f97c8073170 2024-09-17T03:51:47.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.391+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97c81f63b0 con 0x7f97c8073170 2024-09-17T03:51:47.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.393+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f97c00051c0 con 0x7f97c8073170 2024-09-17T03:51:47.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.394+0000 7f97bf400640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 0x7f97a807b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.395+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f97c009bb30 con 0x7f97c8073170 2024-09-17T03:51:47.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.395+0000 7f97c7400640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 0x7f97a807b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.396+0000 7f97c7400640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 0x7f97a807b120 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97b8001720 tx=0x7f97b8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:47.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.398+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f97c0027080 con 0x7f97c8073170 2024-09-17T03:51:47.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.650+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f97c809bd10 con 0x7f97c8073170 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.651+0000 7f97bf400640 1 -- 172.21.15.129:0/3264456998 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v28) ==== 80+0+5304 (secure 0 0 0) 0x7f97c809bd10 con 0x7f97c8073170 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:51:47.651 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:51:47.652 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.653 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.655 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.656 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.657 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.658 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.660 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.661 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:51:47.662 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:51:47.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.664 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:47.665 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:51:47.666 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.653+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 msgr2=0x7f97a807b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.653+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 0x7f97a807b120 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97b8001720 tx=0x7f97b8009040 comp rx=0 tx=0).stop 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.653+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 msgr2=0x7f97c809d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.653+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f97c000ea40 tx=0x7f97c000ef00 comp rx=0 tx=0).stop 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 shutdown_connections 2024-09-17T03:51:47.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97a8078c10 0x7f97a807b120 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c8073170 0x7f97c809d270 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97c8070440 0x7f97c80d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 --2- 172.21.15.129:0/3264456998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c8070030 0x7f97c81f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 >> 172.21.15.129:0/3264456998 conn(0x7f97c80b1cb0 msgr2=0x7f97c815e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 shutdown_connections 2024-09-17T03:51:47.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.654+0000 7f97ce000640 1 -- 172.21.15.129:0/3264456998 wait complete. 2024-09-17T03:51:47.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:51:47.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.897+0000 7efcb6000640 1 -- 172.21.15.129:0/3568064425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cd070 msgr2=0x7efcb00b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.897+0000 7efcb6000640 1 --2- 172.21.15.129:0/3568064425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cd070 0x7efcb00b2fb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7efcb00d1220 tx=0x7efca802a980 comp rx=0 tx=0).stop 2024-09-17T03:51:47.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 -- 172.21.15.129:0/3568064425 shutdown_connections 2024-09-17T03:51:47.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 --2- 172.21.15.129:0/3568064425 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cd070 0x7efcb00b2fb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 --2- 172.21.15.129:0/3568064425 >> v2:172.21.15.181:3300/0 conn(0x7efcb00cfb90 0x7efcb00ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 --2- 172.21.15.129:0/3568064425 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efcb00cf660 0x7efcb00cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 -- 172.21.15.129:0/3568064425 >> 172.21.15.129:0/3568064425 conn(0x7efcb00b1cb0 msgr2=0x7efcb00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 -- 172.21.15.129:0/3568064425 shutdown_connections 2024-09-17T03:51:47.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.898+0000 7efcb6000640 1 -- 172.21.15.129:0/3568064425 wait complete. 2024-09-17T03:51:47.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.900+0000 7efcb6000640 1 Processor -- start 2024-09-17T03:51:47.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.900+0000 7efcb6000640 1 -- start start 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.901+0000 7efcb6000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb00cd070 0x7efcb01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.901+0000 7efcb6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcb6000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efcb00cfb90 0x7efcb00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcb6000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efcb0070f20 con 0x7efcb00cf660 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcb6000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efcb00af810 con 0x7efcb00cd070 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcb6000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efcb00710a0 con 0x7efcb00cfb90 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49678/0 (socket says 172.21.15.129:49678) 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 -- 172.21.15.129:0/196740130 learned_addr learned my addr 172.21.15.129:0/196740130 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 -- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efcb00cfb90 msgr2=0x7efcb00ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efcb00cfb90 0x7efcb00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 -- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb00cd070 msgr2=0x7efcb01586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb00cd070 0x7efcb01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 -- 172.21.15.129:0/196740130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb015b6c0 con 0x7efcb00cf660 2024-09-17T03:51:47.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.902+0000 7efcaea00640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00e7c0 tx=0x7efc9c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.903+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00ccc0 con 0x7efcb00cf660 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.903+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc9c010430 con 0x7efcb00cf660 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.903+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb015b9a0 con 0x7efcb00cf660 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.903+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c010860 con 0x7efcb00cf660 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.903+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb015bf60 con 0x7efcb00cf660 2024-09-17T03:51:47.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.904+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb00cf010 con 0x7efcb00cf660 2024-09-17T03:51:47.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.905+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efc9c010a00 con 0x7efcb00cf660 2024-09-17T03:51:47.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.906+0000 7efca7400640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 0x7efc9007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:47.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.906+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7efc9c09b490 con 0x7efcb00cf660 2024-09-17T03:51:47.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.907+0000 7efcaf400640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 0x7efc9007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:47.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.907+0000 7efcaf400640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 0x7efc9007b0a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efca000afd0 tx=0x7efca000c040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:47.910+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efc9c063c20 con 0x7efcb00cf660 2024-09-17T03:51:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.094+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7efcb00717a0 con 0x7efcb00cf660 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.095+0000 7efca7400640 1 -- 172.21.15.129:0/196740130 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7efc9c063520 con 0x7efcb00cf660 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:51:48.095 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:51:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 msgr2=0x7efc9007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 0x7efc9007b0a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efca000afd0 tx=0x7efca000c040 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 msgr2=0x7efcb00adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00e7c0 tx=0x7efc9c00ec80 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 shutdown_connections 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc90078b90 0x7efc9007b0a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efcb00cfb90 0x7efcb00ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efcb00cf660 0x7efcb00adfe0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 --2- 172.21.15.129:0/196740130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efcb00cd070 0x7efcb01586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 >> 172.21.15.129:0/196740130 conn(0x7efcb00b1cb0 msgr2=0x7efcb00b01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 shutdown_connections 2024-09-17T03:51:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.097+0000 7efcb6000640 1 -- 172.21.15.129:0/196740130 wait complete. 2024-09-17T03:51:48.121 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:51:48.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:51:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.348+0000 7f3617400640 1 -- 172.21.15.129:0/3050899921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cb4d0 msgr2=0x7f36100acf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.348+0000 7f3617400640 1 --2- 172.21.15.129:0/3050899921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cb4d0 0x7f36100acf20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f3610069ea0 tx=0x7f360802a9a0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 -- 172.21.15.129:0/3050899921 shutdown_connections 2024-09-17T03:51:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 --2- 172.21.15.129:0/3050899921 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cb4d0 0x7f36100acf20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 --2- 172.21.15.129:0/3050899921 >> v2:172.21.15.181:3300/0 conn(0x7f36100cc560 0x7f3610073820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 -- 172.21.15.129:0/3050899921 >> 172.21.15.129:0/3050899921 conn(0x7f36100d0130 msgr2=0x7f36100d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 -- 172.21.15.129:0/3050899921 shutdown_connections 2024-09-17T03:51:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.349+0000 7f3617400640 1 -- 172.21.15.129:0/3050899921 wait complete. 2024-09-17T03:51:48.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.351+0000 7f3617400640 1 Processor -- start 2024-09-17T03:51:48.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.351+0000 7f3617400640 1 -- start start 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f361015fb80 0x7f361009ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36101f84d0 0x7f36101f88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36100ad960 con 0x7f36100cc560 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36100ad7e0 con 0x7f361015fb80 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.352+0000 7f3617400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36100adae0 con 0x7f36101f84d0 2024-09-17T03:51:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49692/0 (socket says 172.21.15.129:49692) 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 -- 172.21.15.129:0/1031539213 learned_addr learned my addr 172.21.15.129:0/1031539213 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3607e00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f361015fb80 0x7f361009ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3615400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36101f84d0 0x7f36101f88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 -- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36101f84d0 msgr2=0x7f36101f88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36101f84d0 0x7f36101f88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 -- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f361015fb80 msgr2=0x7f361009ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f361015fb80 0x7f361009ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 -- 172.21.15.129:0/1031539213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36101f8e20 con 0x7f36100cc560 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.353+0000 7f3614a00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f360000c9a0 tx=0x7f360000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.354+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600007bb0 con 0x7f36100cc560 2024-09-17T03:51:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.354+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3600007e40 con 0x7f36100cc560 2024-09-17T03:51:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.354+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361009d640 con 0x7f36100cc560 2024-09-17T03:51:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.354+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360000f690 con 0x7f36100cc560 2024-09-17T03:51:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.354+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36100b44e0 con 0x7f36100cc560 2024-09-17T03:51:48.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.355+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36100ae000 con 0x7f36100cc560 2024-09-17T03:51:48.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.356+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f360000f830 con 0x7f36100cc560 2024-09-17T03:51:48.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.357+0000 7f360d600640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 0x7f35ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:48.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.357+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 4857+0+0 (secure 0 0 0) 0x7f360009b1e0 con 0x7f36100cc560 2024-09-17T03:51:48.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.358+0000 7f3607e00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 0x7f35ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:48.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.358+0000 7f3607e00640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 0x7f35ec07b0d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36100c3cd0 tx=0x7f35f8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:48.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.361+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f36101f8e20 con 0x7f36100cc560 2024-09-17T03:51:48.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:48.545+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f361006f6c0 con 0x7f36100cc560 2024-09-17T03:51:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.133+0000 7f360d600640 1 -- 172.21.15.129:0/1031539213 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f3600099e20 con 0x7f36100cc560 2024-09-17T03:51:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 msgr2=0x7f35ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 0x7f35ec07b0d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36100c3cd0 tx=0x7f35f8009040 comp rx=0 tx=0).stop 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 msgr2=0x7f36101f7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f360000c9a0 tx=0x7f360000ce60 comp rx=0 tx=0).stop 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 shutdown_connections 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f35ec078bc0 0x7f35ec07b0d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36101f84d0 0x7f36101f88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f361015fb80 0x7f361009ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 --2- 172.21.15.129:0/1031539213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36100cc560 0x7f36101f7bd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 >> 172.21.15.129:0/1031539213 conn(0x7f36100d0130 msgr2=0x7f36100aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:49.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 shutdown_connections 2024-09-17T03:51:49.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.136+0000 7f3617400640 1 -- 172.21.15.129:0/1031539213 wait complete. 2024-09-17T03:51:49.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:51:49.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 -- 172.21.15.129:0/3164344983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140cad30 msgr2=0x7f3a1406ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:49.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 --2- 172.21.15.129:0/3164344983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140cad30 0x7f3a1406ffa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3a140c71a0 tx=0x7f39fc02a560 comp rx=0 tx=0).stop 2024-09-17T03:51:49.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 -- 172.21.15.129:0/3164344983 shutdown_connections 2024-09-17T03:51:49.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 --2- 172.21.15.129:0/3164344983 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140704e0 0x7f3a140cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 --2- 172.21.15.129:0/3164344983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140cad30 0x7f3a1406ffa0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 --2- 172.21.15.129:0/3164344983 >> v2:172.21.15.181:3300/0 conn(0x7f3a140ca800 0x7f3a140cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 -- 172.21.15.129:0/3164344983 >> 172.21.15.129:0/3164344983 conn(0x7f3a140b1cb0 msgr2=0x7f3a140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 -- 172.21.15.129:0/3164344983 shutdown_connections 2024-09-17T03:51:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.393+0000 7f3a1b400640 1 -- 172.21.15.129:0/3164344983 wait complete. 2024-09-17T03:51:49.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.395+0000 7f3a1b400640 1 Processor -- start 2024-09-17T03:51:49.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.395+0000 7f3a1b400640 1 -- start start 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140ca800 0x7f3a141f8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a140cad30 0x7f3a140ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a140cd140 con 0x7f3a140704e0 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a140ccfc0 con 0x7f3a140cad30 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a1b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3a140cd2c0 con 0x7f3a140ca800 2024-09-17T03:51:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a18a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a18a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49702/0 (socket says 172.21.15.129:49702) 2024-09-17T03:51:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a18a00640 1 -- 172.21.15.129:0/796951297 learned_addr learned my addr 172.21.15.129:0/796951297 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a19400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a140cad30 0x7f3a140ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.397+0000 7f3a0be00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140ca800 0x7f3a141f8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 -- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140ca800 msgr2=0x7f3a141f8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140ca800 0x7f3a141f8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 -- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a140cad30 msgr2=0x7f3a140ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a140cad30 0x7f3a140ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 -- 172.21.15.129:0/796951297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a141f96d0 con 0x7f3a140704e0 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a18a00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400d6d0 tx=0x7f3a0400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04017070 con 0x7f3a140704e0 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a04004310 con 0x7f3a140704e0 2024-09-17T03:51:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a141f99b0 con 0x7f3a140704e0 2024-09-17T03:51:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04009a80 con 0x7f3a140704e0 2024-09-17T03:51:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.398+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a140c5f20 con 0x7f3a140704e0 2024-09-17T03:51:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.400+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a140cba00 con 0x7f3a140704e0 2024-09-17T03:51:49.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.401+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a040035f0 con 0x7f3a140704e0 2024-09-17T03:51:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.402+0000 7f3a11600640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 0x7f39f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.402+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5227+0+0 (secure 0 0 0) 0x7f3a0409ad40 con 0x7f3a140704e0 2024-09-17T03:51:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.402+0000 7f3a0be00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 0x7f39f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.403+0000 7f3a0be00640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 0x7f39f007b0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39fc033000 tx=0x7f39fc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:49.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.407+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a04062d00 con 0x7f3a140704e0 2024-09-17T03:51:49.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:49.593+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3a140c6ee0 con 0x7f3a140704e0 2024-09-17T03:51:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.136+0000 7f3a11600640 1 -- 172.21.15.129:0/796951297 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7f3a04067730 con 0x7f3a140704e0 2024-09-17T03:51:50.137 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 msgr2=0x7f39f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 0x7f39f007b0a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39fc033000 tx=0x7f39fc004730 comp rx=0 tx=0).stop 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 msgr2=0x7f3a140c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400d6d0 tx=0x7f3a0400db90 comp rx=0 tx=0).stop 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 shutdown_connections 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39f0078b90 0x7f39f007b0a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.139+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a140cad30 0x7f3a140ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.140+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a140ca800 0x7f3a141f8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.140+0000 7f3a1b400640 1 --2- 172.21.15.129:0/796951297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a140704e0 0x7f3a140c8260 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.140+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 >> 172.21.15.129:0/796951297 conn(0x7f3a140b1cb0 msgr2=0x7f3a1415b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:50.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.140+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 shutdown_connections 2024-09-17T03:51:50.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.140+0000 7f3a1b400640 1 -- 172.21.15.129:0/796951297 wait complete. 2024-09-17T03:51:50.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:51:50.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 -- 172.21.15.129:0/1961315749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0af3c0 msgr2=0x7f2ccc0b40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:50.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 --2- 172.21.15.129:0/1961315749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0af3c0 0x7f2ccc0b40b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009860 tx=0x7f2cb802a5a0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 -- 172.21.15.129:0/1961315749 shutdown_connections 2024-09-17T03:51:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 --2- 172.21.15.129:0/1961315749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0af3c0 0x7f2ccc0b40b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 -- 172.21.15.129:0/1961315749 >> 172.21.15.129:0/1961315749 conn(0x7f2ccc06d7d0 msgr2=0x7f2ccc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 -- 172.21.15.129:0/1961315749 shutdown_connections 2024-09-17T03:51:50.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.410+0000 7f2cd4600640 1 -- 172.21.15.129:0/1961315749 wait complete. 2024-09-17T03:51:50.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.412+0000 7f2cd4600640 1 Processor -- start 2024-09-17T03:51:50.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.413+0000 7f2cd4600640 1 -- start start 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0cc1e0 0x7f2ccc1f59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 0x7f2ccc15f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc153520 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc1533a0 con 0x7f2ccc0ccb50 2024-09-17T03:51:50.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd4600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2ccc1536a0 con 0x7f2ccc0cc1e0 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49708/0 (socket says 172.21.15.129:49708) 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.414+0000 7f2cd1c00640 1 -- 172.21.15.129:0/391962907 learned_addr learned my addr 172.21.15.129:0/391962907 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd2600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 0x7f2ccc15f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1200640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0cc1e0 0x7f2ccc1f59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 -- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0cc1e0 msgr2=0x7f2ccc1f59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:50.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0cc1e0 0x7f2ccc1f59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 -- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 msgr2=0x7f2ccc15f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 0x7f2ccc15f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 -- 172.21.15.129:0/391962907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc1f6500 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd2600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 0x7f2ccc15f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd1c00640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000c430 tx=0x7f2cc000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0011850 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cc0017070 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.415+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc1f67e0 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.416+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0011da0 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.416+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc156270 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.417+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc153b20 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.418+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2cc001c460 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.420+0000 7f2cc9600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 0x7f2cac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.420+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5609+0+0 (secure 0 0 0) 0x7f2cc009aad0 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.420+0000 7f2cd1200640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 0x7f2cac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:50.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.421+0000 7f2cd1200640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 0x7f2cac07b0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8034000 tx=0x7f2cb8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:50.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.425+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2cc0062f60 con 0x7f2ccc0af3c0 2024-09-17T03:51:50.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:50.644+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2ccc0b1e50 con 0x7f2ccc0af3c0 2024-09-17T03:51:51.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.152+0000 7f2cc9600640 1 -- 172.21.15.129:0/391962907 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) ==== 172+0+0 (secure 0 0 0) 0x7f2cc0062860 con 0x7f2ccc0af3c0 2024-09-17T03:51:51.152 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 8 and data pool 9 2024-09-17T03:51:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.156+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 msgr2=0x7f2cac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.156+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 0x7f2cac07b0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8034000 tx=0x7f2cb8004730 comp rx=0 tx=0).stop 2024-09-17T03:51:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.156+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 msgr2=0x7f2ccc156790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.156+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000c430 tx=0x7f2cc000c8f0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 shutdown_connections 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cac078bc0 0x7f2cac07b0d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ccc0ccb50 0x7f2ccc15f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ccc0cc1e0 0x7f2ccc1f59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 --2- 172.21.15.129:0/391962907 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ccc0af3c0 0x7f2ccc156790 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.157+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 >> 172.21.15.129:0/391962907 conn(0x7f2ccc06d7d0 msgr2=0x7f2ccc0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.158+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 shutdown_connections 2024-09-17T03:51:51.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.159+0000 7f2cd4600640 1 -- 172.21.15.129:0/391962907 wait complete. 2024-09-17T03:51:51.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 -- 172.21.15.129:0/700422695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080706b0 msgr2=0x7fd7080cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 --2- 172.21.15.129:0/700422695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080706b0 0x7fd7080cba00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009830 tx=0x7fd6f402a5c0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 -- 172.21.15.129:0/700422695 shutdown_connections 2024-09-17T03:51:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 --2- 172.21.15.129:0/700422695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080706b0 0x7fd7080cba00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 -- 172.21.15.129:0/700422695 >> 172.21.15.129:0/700422695 conn(0x7fd7080b1cb0 msgr2=0x7fd7080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 -- 172.21.15.129:0/700422695 shutdown_connections 2024-09-17T03:51:51.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.429+0000 7fd70ec00640 1 -- 172.21.15.129:0/700422695 wait complete. 2024-09-17T03:51:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.431+0000 7fd70ec00640 1 Processor -- start 2024-09-17T03:51:51.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.431+0000 7fd70ec00640 1 -- start start 2024-09-17T03:51:51.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080ca800 0x7fd7081f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 0x7fd70809c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd708074130 con 0x7fd7080cad30 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd708073fb0 con 0x7fd7080ca800 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd7080742b0 con 0x7fd70809b7f0 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd707400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd707400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49712/0 (socket says 172.21.15.129:49712) 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd707400640 1 -- 172.21.15.129:0/3424284135 learned_addr learned my addr 172.21.15.129:0/3424284135 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd70cc00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 0x7fd70809c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.433+0000 7fd707e00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080ca800 0x7fd7081f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 -- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 msgr2=0x7fd70809c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 0x7fd70809c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 -- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080ca800 msgr2=0x7fd7081f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080ca800 0x7fd7081f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 -- 172.21.15.129:0/3424284135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd70815b140 con 0x7fd7080cad30 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd70cc00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 0x7fd70809c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd707400640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009950 tx=0x7fd6f4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4031d60 con 0x7fd7080cad30 2024-09-17T03:51:51.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6f4033e10 con 0x7fd7080cad30 2024-09-17T03:51:51.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f4005260 con 0x7fd7080cad30 2024-09-17T03:51:51.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd70809b5a0 con 0x7fd7080cad30 2024-09-17T03:51:51.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.434+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7080c6780 con 0x7fd7080cad30 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.436+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7080747d0 con 0x7fd7080cad30 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.437+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd6f4005400 con 0x7fd7080cad30 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.438+0000 7fd6ffe00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 0x7fd6e807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.438+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fd6f40bbfe0 con 0x7fd7080cad30 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.438+0000 7fd707e00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 0x7fd6e807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.439+0000 7fd707e00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 0x7fd6e807b0d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7080cb250 tx=0x7fd6f8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:51.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.442+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd6f403f030 con 0x7fd7080cad30 2024-09-17T03:51:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.628+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7080d0dc0 con 0x7fd7080cad30 2024-09-17T03:51:51.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.629+0000 7fd6ffe00640 1 -- 172.21.15.129:0/3424284135 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16786 (secure 0 0 0) 0x7fd6f4084300 con 0x7fd7080cad30 2024-09-17T03:51:51.629 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:51.630 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":55,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:50.646732+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:48.557870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:49.606077+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 msgr2=0x7fd6e807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 0x7fd6e807b0d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd7080cb250 tx=0x7fd6f8009040 comp rx=0 tx=0).stop 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 msgr2=0x7fd70809bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009950 tx=0x7fd6f4033c20 comp rx=0 tx=0).stop 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 shutdown_connections 2024-09-17T03:51:51.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd6e8078bc0 0x7fd6e807b0d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd70809b7f0 0x7fd70809c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7080cad30 0x7fd70809bd00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 --2- 172.21.15.129:0/3424284135 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7080ca800 0x7fd7081f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 >> 172.21.15.129:0/3424284135 conn(0x7fd7080b1cb0 msgr2=0x7fd7080d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 shutdown_connections 2024-09-17T03:51:51.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.632+0000 7fd70ec00640 1 -- 172.21.15.129:0/3424284135 wait complete. 2024-09-17T03:51:51.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 -- 172.21.15.129:0/3842911186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8070440 msgr2=0x7fcaa8074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 --2- 172.21.15.129:0/3842911186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8070440 0x7fcaa8074b80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fcaa80710f0 tx=0x7fcaa002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 -- 172.21.15.129:0/3842911186 shutdown_connections 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 --2- 172.21.15.129:0/3842911186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8070440 0x7fcaa8074b80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 --2- 172.21.15.129:0/3842911186 >> v2:172.21.15.181:3300/0 conn(0x7fcaa8070030 0x7fcaa8074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 -- 172.21.15.129:0/3842911186 >> 172.21.15.129:0/3842911186 conn(0x7fcaa80b1cb0 msgr2=0x7fcaa80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 -- 172.21.15.129:0/3842911186 shutdown_connections 2024-09-17T03:51:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.876+0000 7fcaaf200640 1 -- 172.21.15.129:0/3842911186 wait complete. 2024-09-17T03:51:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.878+0000 7fcaaf200640 1 Processor -- start 2024-09-17T03:51:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.879+0000 7fcaaf200640 1 -- start start 2024-09-17T03:51:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 0x7fcaa814db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 0x7fcaa80c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa80b39f0 con 0x7fcaa8073170 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa80b3870 con 0x7fcaa8070440 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaaf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa80b3b70 con 0x7fcaa8070030 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaad200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaad200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49724/0 (socket says 172.21.15.129:49724) 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaad200640 1 -- 172.21.15.129:0/1663746016 learned_addr learned my addr 172.21.15.129:0/1663746016 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.880+0000 7fcaa7e00640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 0x7fcaa814db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fca9f400640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 0x7fcaa80c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 -- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 msgr2=0x7fcaa814db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 0x7fcaa814db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 -- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 msgr2=0x7fcaa80c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:51.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 0x7fcaa80c9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 -- 172.21.15.129:0/1663746016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa80cf2c0 con 0x7fcaa8073170 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaa7e00640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 0x7fcaa814db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fca9f400640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 0x7fcaa80c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaad200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcaa002aea0 tx=0x7fcaa0031760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0031990 con 0x7fcaa8073170 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcaa0031c20 con 0x7fcaa8073170 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa00330d0 con 0x7fcaa8073170 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.881+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa80cf5a0 con 0x7fcaa8073170 2024-09-17T03:51:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.882+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa80cfb60 con 0x7fcaa8073170 2024-09-17T03:51:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.882+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa80b4090 con 0x7fcaa8073170 2024-09-17T03:51:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.884+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcaa0033270 con 0x7fcaa8073170 2024-09-17T03:51:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.885+0000 7fcaa4c00640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 0x7fca8407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.886+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5709+0+0 (secure 0 0 0) 0x7fcaa00bc720 con 0x7fcaa8073170 2024-09-17T03:51:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.886+0000 7fcaa7e00640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 0x7fca8407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.887+0000 7fcaa7e00640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 0x7fca8407b0a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca98000f60 tx=0x7fca98009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:51.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:51.889+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcaa0052080 con 0x7fcaa8073170 2024-09-17T03:51:52.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.073+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcaa809d920 con 0x7fcaa8073170 2024-09-17T03:51:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.074+0000 7fcaa4c00640 1 -- 172.21.15.129:0/1663746016 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16786 (secure 0 0 0) 0x7fcaa003f030 con 0x7fcaa8073170 2024-09-17T03:51:52.074 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:52.075 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":55,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:50.646732+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:48.557870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:49.606077+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.076+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 msgr2=0x7fca8407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.076+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 0x7fca8407b0a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fca98000f60 tx=0x7fca98009040 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.076+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 msgr2=0x7fcaa80d0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.076+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fcaa002aea0 tx=0x7fcaa0031760 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 shutdown_connections 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca84078b90 0x7fca8407b0a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcaa8073170 0x7fcaa80d0080 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcaa8070440 0x7fcaa80c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 --2- 172.21.15.129:0/1663746016 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcaa8070030 0x7fcaa814db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 >> 172.21.15.129:0/1663746016 conn(0x7fcaa80b1cb0 msgr2=0x7fcaa815b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:52.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 shutdown_connections 2024-09-17T03:51:52.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.077+0000 7fcaaf200640 1 -- 172.21.15.129:0/1663746016 wait complete. 2024-09-17T03:51:52.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:51:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.328+0000 7fdd70e00640 1 -- 172.21.15.129:0/88494745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c15ea70 msgr2=0x7fdd6c162ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.328+0000 7fdd70e00640 1 --2- 172.21.15.129:0/88494745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c15ea70 0x7fdd6c162ed0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0cd500 tx=0x7fdd5c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 -- 172.21.15.129:0/88494745 shutdown_connections 2024-09-17T03:51:52.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 --2- 172.21.15.129:0/88494745 >> v2:172.21.15.181:3300/0 conn(0x7fdd6c15f910 0x7fdd6c0c9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 --2- 172.21.15.129:0/88494745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c15ea70 0x7fdd6c162ed0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 -- 172.21.15.129:0/88494745 >> 172.21.15.129:0/88494745 conn(0x7fdd6c0cd530 msgr2=0x7fdd6c0cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 -- 172.21.15.129:0/88494745 shutdown_connections 2024-09-17T03:51:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.329+0000 7fdd70e00640 1 -- 172.21.15.129:0/88494745 wait complete. 2024-09-17T03:51:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.331+0000 7fdd70e00640 1 Processor -- start 2024-09-17T03:51:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.331+0000 7fdd70e00640 1 -- start start 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.332+0000 7fdd70e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.332+0000 7fdd70e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd6c15ea70 0x7fdd6c161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd70e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 0x7fdd6c1615b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd70e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6c15fea0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd70e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6c15fd20 con 0x7fdd6c15f910 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd70e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdd6c0cff80 con 0x7fdd6c15ea70 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49736/0 (socket says 172.21.15.129:49736) 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 -- 172.21.15.129:0/1663570489 learned_addr learned my addr 172.21.15.129:0/1663570489 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69400640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd6c15ea70 0x7fdd6c161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd6a800640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 0x7fdd6c1615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 -- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd6c15ea70 msgr2=0x7fdd6c161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd6c15ea70 0x7fdd6c161070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 -- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 msgr2=0x7fdd6c1615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 0x7fdd6c1615b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd69e00640 1 -- 172.21.15.129:0/1663570489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6c1579d0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.333+0000 7fdd6a800640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 0x7fdd6c1615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd69e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009780 tx=0x7fdd5c033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c031d60 con 0x7fdd6c0ca320 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd5c005020 con 0x7fdd6c0ca320 2024-09-17T03:51:52.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c005470 con 0x7fdd6c0ca320 2024-09-17T03:51:52.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6c1443c0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.334+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd6c145f20 con 0x7fdd6c0ca320 2024-09-17T03:51:52.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.335+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd6c0d04d0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.336+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdd5c0047a0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.337+0000 7fdd62000640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 0x7fdd4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:52.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.337+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fdd5c0bbfa0 con 0x7fdd6c0ca320 2024-09-17T03:51:52.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.338+0000 7fdd69400640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 0x7fdd4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:52.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.339+0000 7fdd69400640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 0x7fdd4c07b0a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c15d080 tx=0x7fdd5800a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:52.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.342+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdd5c03f030 con 0x7fdd6c0ca320 2024-09-17T03:51:52.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:52.565+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdd6c0cc3a0 con 0x7fdd6c0ca320 2024-09-17T03:51:53.130 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:51:53.130+0000 7fb53b600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:51:53.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.176+0000 7fdd62000640 1 -- 172.21.15.129:0/1663570489 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) ==== 118+0+0 (secure 0 0 0) 0x7fdd5c083d80 con 0x7fdd6c0ca320 2024-09-17T03:51:53.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 msgr2=0x7fdd4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:53.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 0x7fdd4c07b0a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c15d080 tx=0x7fdd5800a250 comp rx=0 tx=0).stop 2024-09-17T03:51:53.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 msgr2=0x7fdd6c143790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:53.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c009780 tx=0x7fdd5c033d60 comp rx=0 tx=0).stop 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 shutdown_connections 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd4c078b90 0x7fdd4c07b0a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd6c15f910 0x7fdd6c1615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd6c15ea70 0x7fdd6c161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 --2- 172.21.15.129:0/1663570489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd6c0ca320 0x7fdd6c143790 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 >> 172.21.15.129:0/1663570489 conn(0x7fdd6c0cd530 msgr2=0x7fdd6c06d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:53.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 shutdown_connections 2024-09-17T03:51:53.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.179+0000 7fdd70e00640 1 -- 172.21.15.129:0/1663570489 wait complete. 2024-09-17T03:51:53.205 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:51:53.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:51:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 -- 172.21.15.129:0/3756697122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79100051c0 msgr2=0x7f791000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 --2- 172.21.15.129:0/3756697122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79100051c0 0x7f791000b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7924001e40 tx=0x7f7910005830 comp rx=0 tx=0).stop 2024-09-17T03:51:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 -- 172.21.15.129:0/3756697122 shutdown_connections 2024-09-17T03:51:53.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 --2- 172.21.15.129:0/3756697122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79100051c0 0x7f791000b9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 -- 172.21.15.129:0/3756697122 >> 172.21.15.129:0/3756697122 conn(0x7f792c06d7d0 msgr2=0x7f792c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:53.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 -- 172.21.15.129:0/3756697122 shutdown_connections 2024-09-17T03:51:53.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.432+0000 7f7931600640 1 -- 172.21.15.129:0/3756697122 wait complete. 2024-09-17T03:51:53.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.434+0000 7f7931600640 1 Processor -- start 2024-09-17T03:51:53.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.434+0000 7f7931600640 1 -- start start 2024-09-17T03:51:53.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79100051c0 0x7f792c157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:53.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:53.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f792c09d460 0x7f792c14d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:53.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f792c15dc30 con 0x7f792c1575c0 2024-09-17T03:51:53.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f792c15dab0 con 0x7f79100051c0 2024-09-17T03:51:53.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7931600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f792c15ddb0 con 0x7f792c09d460 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49754/0 (socket says 172.21.15.129:49754) 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 -- 172.21.15.129:0/361376175 learned_addr learned my addr 172.21.15.129:0/361376175 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f792b400640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f792c09d460 0x7f792c14d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f792aa00640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79100051c0 0x7f792c157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 -- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f792c09d460 msgr2=0x7f792c14d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f792c09d460 0x7f792c14d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 -- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79100051c0 msgr2=0x7f792c157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79100051c0 0x7f792c157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.436+0000 7f7922000640 1 -- 172.21.15.129:0/361376175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f792c1f4040 con 0x7f792c1575c0 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7922000640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7914011ae0 tx=0x7f7914009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791401e070 con 0x7f792c1575c0 2024-09-17T03:51:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f791400fc10 con 0x7f792c1575c0 2024-09-17T03:51:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7914019410 con 0x7f792c1575c0 2024-09-17T03:51:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f792c156820 con 0x7f792c1575c0 2024-09-17T03:51:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.437+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f792c156ae0 con 0x7f792c1575c0 2024-09-17T03:51:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.438+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f792c0cd2e0 con 0x7f792c1575c0 2024-09-17T03:51:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.439+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7914021070 con 0x7f792c1575c0 2024-09-17T03:51:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.441+0000 7f7923400640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 0x7f790807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.441+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f791409b4c0 con 0x7f792c1575c0 2024-09-17T03:51:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.441+0000 7f792aa00640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 0x7f790807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:53.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.442+0000 7f792aa00640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 0x7f790807b0a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f791c001050 tx=0x7f791c001110 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.446+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f79140633b0 con 0x7f792c1575c0 2024-09-17T03:51:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:53.664+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f792c15c490 con 0x7f792c1575c0 2024-09-17T03:51:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.178+0000 7f7923400640 1 -- 172.21.15.129:0/361376175 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) ==== 122+0+0 (secure 0 0 0) 0x7f7914067de0 con 0x7f792c1575c0 2024-09-17T03:51:54.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 msgr2=0x7f790807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 0x7f790807b0a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f791c001050 tx=0x7f791c001110 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 msgr2=0x7f792c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7914011ae0 tx=0x7f7914009e60 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 shutdown_connections 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7908078b90 0x7f790807b0a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f792c09d460 0x7f792c14d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f792c1575c0 0x7f792c09cf20 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 --2- 172.21.15.129:0/361376175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79100051c0 0x7f792c157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 >> 172.21.15.129:0/361376175 conn(0x7f792c06d7d0 msgr2=0x7f792c06ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 shutdown_connections 2024-09-17T03:51:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.181+0000 7f7931600640 1 -- 172.21.15.129:0/361376175 wait complete. 2024-09-17T03:51:54.209 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:51:54.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:54.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 -- 172.21.15.129:0/3091856677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80b54d0 msgr2=0x7f8dd80b58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 --2- 172.21.15.129:0/3091856677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80b54d0 0x7f8dd80b58e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8153c20 tx=0x7f8dd002aa30 comp rx=0 tx=0).stop 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 -- 172.21.15.129:0/3091856677 shutdown_connections 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 --2- 172.21.15.129:0/3091856677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80b54d0 0x7f8dd80b58e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 --2- 172.21.15.129:0/3091856677 >> v2:172.21.15.181:3300/0 conn(0x7f8dd80735d0 0x7f8dd80caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 -- 172.21.15.129:0/3091856677 >> 172.21.15.129:0/3091856677 conn(0x7f8dd806d870 msgr2=0x7f8dd806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 -- 172.21.15.129:0/3091856677 shutdown_connections 2024-09-17T03:51:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.432+0000 7f8dd7200640 1 -- 172.21.15.129:0/3091856677 wait complete. 2024-09-17T03:51:54.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.434+0000 7f8dd7200640 1 Processor -- start 2024-09-17T03:51:54.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.435+0000 7f8dd7200640 1 -- start start 2024-09-17T03:51:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 0x7f8dd8141280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dd80b54d0 0x7f8dd81560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd815d240 con 0x7f8dd80af700 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8dd815d0c0 con 0x7f8dd80b54d0 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.436+0000 7f8dd7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8dd815d3c0 con 0x7f8dd80735d0 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49746/0 (socket says 172.21.15.129:49746) 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 -- 172.21.15.129:0/1203894801 learned_addr learned my addr 172.21.15.129:0/1203894801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dd6800640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dd80b54d0 0x7f8dd81560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dd5e00640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 0x7f8dd8141280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 -- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 msgr2=0x7f8dd8141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 0x7f8dd8141280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 -- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dd80b54d0 msgr2=0x7f8dd81560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dd80b54d0 0x7f8dd81560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dcd400640 1 -- 172.21.15.129:0/1203894801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dd81f5e50 con 0x7f8dd80af700 2024-09-17T03:51:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.437+0000 7f8dd5e00640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 0x7f8dd8141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dcd400640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d700 tx=0x7f8dc000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0017070 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dc0005310 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dd81f6130 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dc0009820 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.438+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dd80cb520 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.439+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd80cfb30 con 0x7f8dd80af700 2024-09-17T03:51:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.441+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8dc00099c0 con 0x7f8dd80af700 2024-09-17T03:51:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.442+0000 7f8dcea00640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 0x7f8db407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.442+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f8dc009ae30 con 0x7f8dd80af700 2024-09-17T03:51:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.443+0000 7f8dd5e00640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 0x7f8db407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.443+0000 7f8dd5e00640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 0x7f8db407b0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006fd0 tx=0x7f8dc8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:54.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.447+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8dd80cfb30 con 0x7f8dd80af700 2024-09-17T03:51:54.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.667+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dd815d900 con 0x7f8dd80af700 2024-09-17T03:51:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.668+0000 7f8dcea00640 1 -- 172.21.15.129:0/1203894801 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f8dc0062c10 con 0x7f8dd80af700 2024-09-17T03:51:54.668 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:54.668 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:51:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 msgr2=0x7f8db407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 0x7f8db407b0a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8dc8006fd0 tx=0x7f8dc8008040 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 msgr2=0x7f8dd8155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8dc000d700 tx=0x7f8dc000dbc0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 shutdown_connections 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8db4078b90 0x7f8db407b0a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dd80b54d0 0x7f8dd81560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8dd80af700 0x7f8dd8155b90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 --2- 172.21.15.129:0/1203894801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8dd80735d0 0x7f8dd8141280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.670+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 >> 172.21.15.129:0/1203894801 conn(0x7f8dd806d870 msgr2=0x7f8dd815db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.671+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 shutdown_connections 2024-09-17T03:51:54.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.671+0000 7f8dd7200640 1 -- 172.21.15.129:0/1203894801 wait complete. 2024-09-17T03:51:54.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:51:54.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 -- 172.21.15.129:0/3014855434 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43415fbd0 msgr2=0x7fd434152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 --2- 172.21.15.129:0/3014855434 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43415fbd0 0x7fd434152360 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd428009830 tx=0x7fd42802a530 comp rx=0 tx=0).stop 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 -- 172.21.15.129:0/3014855434 shutdown_connections 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 --2- 172.21.15.129:0/3014855434 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43415fbd0 0x7fd434152360 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 -- 172.21.15.129:0/3014855434 >> 172.21.15.129:0/3014855434 conn(0x7fd43406d410 msgr2=0x7fd4340b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 -- 172.21.15.129:0/3014855434 shutdown_connections 2024-09-17T03:51:54.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.918+0000 7fd439e00640 1 -- 172.21.15.129:0/3014855434 wait complete. 2024-09-17T03:51:54.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.920+0000 7fd439e00640 1 Processor -- start 2024-09-17T03:51:54.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.920+0000 7fd439e00640 1 -- start start 2024-09-17T03:51:54.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 0x7fd43413d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434154a90 0x7fd434154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4340b4200 con 0x7fd434156020 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4340b4080 con 0x7fd434154a90 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd4340b4380 con 0x7fd434072c30 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd433e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd433e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49750/0 (socket says 172.21.15.129:49750) 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd433e00640 1 -- 172.21.15.129:0/2148648063 learned_addr learned my addr 172.21.15.129:0/2148648063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd438a00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 0x7fd43413d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.922+0000 7fd439400640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434154a90 0x7fd434154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 -- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 msgr2=0x7fd43413d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 0x7fd43413d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 -- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434154a90 msgr2=0x7fd434154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434154a90 0x7fd434154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 -- 172.21.15.129:0/2148648063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4341553e0 con 0x7fd434156020 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd438a00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 0x7fd43413d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd433e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd42000c3c0 tx=0x7fd42000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420017070 con 0x7fd434156020 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4200035f0 con 0x7fd434156020 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420003a20 con 0x7fd434156020 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4341493a0 con 0x7fd434156020 2024-09-17T03:51:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.923+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd434156b30 con 0x7fd434156020 2024-09-17T03:51:54.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.924+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43415fbd0 con 0x7fd434156020 2024-09-17T03:51:54.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.926+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd42001a070 con 0x7fd434156020 2024-09-17T03:51:54.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.927+0000 7fd430c00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 0x7fd41407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:54.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.927+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fd420099f90 con 0x7fd434156020 2024-09-17T03:51:54.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.928+0000 7fd438a00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 0x7fd41407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:54.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.928+0000 7fd438a00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 0x7fd41407b0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd428034000 tx=0x7fd428004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:54.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:54.932+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4341553e0 con 0x7fd434156020 2024-09-17T03:51:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.117+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd434068fb0 con 0x7fd434156020 2024-09-17T03:51:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.119+0000 7fd430c00640 1 -- 172.21.15.129:0/2148648063 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16773 (secure 0 0 0) 0x7fd420061d70 con 0x7fd434156020 2024-09-17T03:51:55.119 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:55.119 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":56,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:52.164400+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:48.557870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:49.606077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:51:55.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 msgr2=0x7fd41407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 0x7fd41407b0a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd428034000 tx=0x7fd428004730 comp rx=0 tx=0).stop 2024-09-17T03:51:55.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 msgr2=0x7fd43413a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fd42000c3c0 tx=0x7fd42000c880 comp rx=0 tx=0).stop 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 shutdown_connections 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd414078b90 0x7fd41407b0a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd434154a90 0x7fd434154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd434156020 0x7fd43413a000 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 --2- 172.21.15.129:0/2148648063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd434072c30 0x7fd43413d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 >> 172.21.15.129:0/2148648063 conn(0x7fd43406d410 msgr2=0x7fd4340d0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 shutdown_connections 2024-09-17T03:51:55.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.121+0000 7fd439e00640 1 -- 172.21.15.129:0/2148648063 wait complete. 2024-09-17T03:51:55.145 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:51:55.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:55.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 -- 172.21.15.129:0/89534933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 msgr2=0x7fec4c0b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 --2- 172.21.15.129:0/89534933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c0b39c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec3c009860 tx=0x7fec3c02a530 comp rx=0 tx=0).stop 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 -- 172.21.15.129:0/89534933 shutdown_connections 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 --2- 172.21.15.129:0/89534933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c0b39c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 -- 172.21.15.129:0/89534933 >> 172.21.15.129:0/89534933 conn(0x7fec4c06d7d0 msgr2=0x7fec4c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 -- 172.21.15.129:0/89534933 shutdown_connections 2024-09-17T03:51:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.367+0000 7fec51400640 1 -- 172.21.15.129:0/89534933 wait complete. 2024-09-17T03:51:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.370+0000 7fec51400640 1 Processor -- start 2024-09-17T03:51:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.370+0000 7fec51400640 1 -- start start 2024-09-17T03:51:55.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec4c0cd3e0 0x7fec4c1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c166bf0 con 0x7fec4c072ff0 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fec4c166d70 con 0x7fec4c0cd3e0 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec51400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fec4c166ef0 con 0x7fec4c0af780 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec4a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec4a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49756/0 (socket says 172.21.15.129:49756) 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.371+0000 7fec4a800640 1 -- 172.21.15.129:0/780704332 learned_addr learned my addr 172.21.15.129:0/780704332 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec49e00640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4b200640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec4c0cd3e0 0x7fec4c1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 -- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 msgr2=0x7fec4c153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 -- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec4c0cd3e0 msgr2=0x7fec4c1538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec4c0cd3e0 0x7fec4c1538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 -- 172.21.15.129:0/780704332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec4c159d40 con 0x7fec4c072ff0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.372+0000 7fec4a800640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fec3c02ab20 tx=0x7fec3c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.373+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c005c90 con 0x7fec4c072ff0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.373+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec4c15a020 con 0x7fec4c072ff0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.373+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec3c005020 con 0x7fec4c072ff0 2024-09-17T03:51:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.373+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c005410 con 0x7fec4c072ff0 2024-09-17T03:51:55.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.373+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec4c15a5e0 con 0x7fec4c072ff0 2024-09-17T03:51:55.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.374+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec4c0b39c0 con 0x7fec4c072ff0 2024-09-17T03:51:55.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.375+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fec3c0317c0 con 0x7fec4c072ff0 2024-09-17T03:51:55.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.376+0000 7fec42a00640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 0x7fec2c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.376+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fec3c0bbe20 con 0x7fec4c072ff0 2024-09-17T03:51:55.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.377+0000 7fec49e00640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 0x7fec2c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.377+0000 7fec49e00640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 0x7fec2c07b0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec38001240 tx=0x7fec38009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:55.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.381+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec3c03f030 con 0x7fec4c072ff0 2024-09-17T03:51:55.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.563+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec4c159240 con 0x7fec2c078bc0 2024-09-17T03:51:55.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.568+0000 7fec42a00640 1 -- 172.21.15.129:0/780704332 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7fec4c159240 con 0x7fec2c078bc0 2024-09-17T03:51:55.571 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:55.572 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.572+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 msgr2=0x7fec2c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.572+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 0x7fec2c07b0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec38001240 tx=0x7fec38009040 comp rx=0 tx=0).stop 2024-09-17T03:51:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 msgr2=0x7fec4c151b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fec3c02ab20 tx=0x7fec3c003c10 comp rx=0 tx=0).stop 2024-09-17T03:51:55.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 shutdown_connections 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec2c078bc0 0x7fec2c07b0d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec4c0cd3e0 0x7fec4c1538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec4c0af780 0x7fec4c153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 --2- 172.21.15.129:0/780704332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec4c072ff0 0x7fec4c151b50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 >> 172.21.15.129:0/780704332 conn(0x7fec4c06d7d0 msgr2=0x7fec4c15c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 shutdown_connections 2024-09-17T03:51:55.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.573+0000 7fec51400640 1 -- 172.21.15.129:0/780704332 wait complete. 2024-09-17T03:51:55.594 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-17T03:51:54.329668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255956,"kb_used_data":6396,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718764,"statfs":{"total":773094113280,"available":772832014336,"internally_reserved":0,"allocated":6549504,"data_stored":3355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.193634"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174486+0000","last_change":"2024-09-17T03:51:51.199588+0000","last_active":"2024-09-17T03:51:52.174486+0000","last_peered":"2024-09-17T03:51:52.174486+0000","last_clean":"2024-09-17T03:51:52.174486+0000","last_became_active":"2024-09-17T03:51:51.199297+0000","last_became_peered":"2024-09-17T03:51:51.199297+0000","last_unstale":"2024-09-17T03:51:52.174486+0000","last_undegraded":"2024-09-17T03:51:52.174486+0000","last_fullsized":"2024-09-17T03:51:52.174486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:07.155723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173622+0000","last_change":"2024-09-17T03:51:51.187428+0000","last_active":"2024-09-17T03:51:52.173622+0000","last_peered":"2024-09-17T03:51:52.173622+0000","last_clean":"2024-09-17T03:51:52.173622+0000","last_became_active":"2024-09-17T03:51:51.186526+0000","last_became_peered":"2024-09-17T03:51:51.186526+0000","last_unstale":"2024-09-17T03:51:52.173622+0000","last_undegraded":"2024-09-17T03:51:52.173622+0000","last_fullsized":"2024-09-17T03:51:52.173622+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:07.429263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730917+0000","last_change":"2024-09-17T03:51:51.181527+0000","last_active":"2024-09-17T03:51:53.730917+0000","last_peered":"2024-09-17T03:51:53.730917+0000","last_clean":"2024-09-17T03:51:53.730917+0000","last_became_active":"2024-09-17T03:51:51.181226+0000","last_became_peered":"2024-09-17T03:51:51.181226+0000","last_unstale":"2024-09-17T03:51:53.730917+0000","last_undegraded":"2024-09-17T03:51:53.730917+0000","last_fullsized":"2024-09-17T03:51:53.730917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:23.640091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174401+0000","last_change":"2024-09-17T03:51:51.190529+0000","last_active":"2024-09-17T03:51:52.174401+0000","last_peered":"2024-09-17T03:51:52.174401+0000","last_clean":"2024-09-17T03:51:52.174401+0000","last_became_active":"2024-09-17T03:51:51.190206+0000","last_became_peered":"2024-09-17T03:51:51.190206+0000","last_unstale":"2024-09-17T03:51:52.174401+0000","last_undegraded":"2024-09-17T03:51:52.174401+0000","last_fullsized":"2024-09-17T03:51:52.174401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:11.743252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174132+0000","last_change":"2024-09-17T03:51:51.184036+0000","last_active":"2024-09-17T03:51:52.174132+0000","last_peered":"2024-09-17T03:51:52.174132+0000","last_clean":"2024-09-17T03:51:52.174132+0000","last_became_active":"2024-09-17T03:51:51.183692+0000","last_became_peered":"2024-09-17T03:51:51.183692+0000","last_unstale":"2024-09-17T03:51:52.174132+0000","last_undegraded":"2024-09-17T03:51:52.174132+0000","last_fullsized":"2024-09-17T03:51:52.174132+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183649+0000","last_change":"2024-09-17T03:51:51.197374+0000","last_active":"2024-09-17T03:51:52.183649+0000","last_peered":"2024-09-17T03:51:52.183649+0000","last_clean":"2024-09-17T03:51:52.183649+0000","last_became_active":"2024-09-17T03:51:51.197083+0000","last_became_peered":"2024-09-17T03:51:51.197083+0000","last_unstale":"2024-09-17T03:51:52.183649+0000","last_undegraded":"2024-09-17T03:51:52.183649+0000","last_fullsized":"2024-09-17T03:51:52.183649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:06.099902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205298+0000","last_change":"2024-09-17T03:51:51.183634+0000","last_active":"2024-09-17T03:51:52.205298+0000","last_peered":"2024-09-17T03:51:52.205298+0000","last_clean":"2024-09-17T03:51:52.205298+0000","last_became_active":"2024-09-17T03:51:51.182965+0000","last_became_peered":"2024-09-17T03:51:51.182965+0000","last_unstale":"2024-09-17T03:51:52.205298+0000","last_undegraded":"2024-09-17T03:51:52.205298+0000","last_fullsized":"2024-09-17T03:51:52.205298+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:35.830782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173124+0000","last_change":"2024-09-17T03:51:51.184044+0000","last_active":"2024-09-17T03:51:52.173124+0000","last_peered":"2024-09-17T03:51:52.173124+0000","last_clean":"2024-09-17T03:51:52.173124+0000","last_became_active":"2024-09-17T03:51:51.181280+0000","last_became_peered":"2024-09-17T03:51:51.181280+0000","last_unstale":"2024-09-17T03:51:52.173124+0000","last_undegraded":"2024-09-17T03:51:52.173124+0000","last_fullsized":"2024-09-17T03:51:52.173124+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:49.860271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205754+0000","last_change":"2024-09-17T03:51:51.178569+0000","last_active":"2024-09-17T03:51:52.205754+0000","last_peered":"2024-09-17T03:51:52.205754+0000","last_clean":"2024-09-17T03:51:52.205754+0000","last_became_active":"2024-09-17T03:51:51.178236+0000","last_became_peered":"2024-09-17T03:51:51.178236+0000","last_unstale":"2024-09-17T03:51:52.205754+0000","last_undegraded":"2024-09-17T03:51:52.205754+0000","last_fullsized":"2024-09-17T03:51:52.205754+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:42.860411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610517+0000","last_change":"2024-09-17T03:51:51.166633+0000","last_active":"2024-09-17T03:51:52.610517+0000","last_peered":"2024-09-17T03:51:52.610517+0000","last_clean":"2024-09-17T03:51:52.610517+0000","last_became_active":"2024-09-17T03:51:51.166390+0000","last_became_peered":"2024-09-17T03:51:51.166390+0000","last_unstale":"2024-09-17T03:51:52.610517+0000","last_undegraded":"2024-09-17T03:51:52.610517+0000","last_fullsized":"2024-09-17T03:51:52.610517+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:52.752910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610551+0000","last_change":"2024-09-17T03:51:51.176819+0000","last_active":"2024-09-17T03:51:52.610551+0000","last_peered":"2024-09-17T03:51:52.610551+0000","last_clean":"2024-09-17T03:51:52.610551+0000","last_became_active":"2024-09-17T03:51:51.175065+0000","last_became_peered":"2024-09-17T03:51:51.175065+0000","last_unstale":"2024-09-17T03:51:52.610551+0000","last_undegraded":"2024-09-17T03:51:52.610551+0000","last_fullsized":"2024-09-17T03:51:52.610551+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:37.036957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174710+0000","last_change":"2024-09-17T03:51:51.180175+0000","last_active":"2024-09-17T03:51:52.174710+0000","last_peered":"2024-09-17T03:51:52.174710+0000","last_clean":"2024-09-17T03:51:52.174710+0000","last_became_active":"2024-09-17T03:51:51.179659+0000","last_became_peered":"2024-09-17T03:51:51.179659+0000","last_unstale":"2024-09-17T03:51:52.174710+0000","last_undegraded":"2024-09-17T03:51:52.174710+0000","last_fullsized":"2024-09-17T03:51:52.174710+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:41.002793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.163002+0000","last_change":"2024-09-17T03:51:51.180210+0000","last_active":"2024-09-17T03:51:53.163002+0000","last_peered":"2024-09-17T03:51:53.163002+0000","last_clean":"2024-09-17T03:51:53.163002+0000","last_became_active":"2024-09-17T03:51:51.179663+0000","last_became_peered":"2024-09-17T03:51:51.179663+0000","last_unstale":"2024-09-17T03:51:53.163002+0000","last_undegraded":"2024-09-17T03:51:53.163002+0000","last_fullsized":"2024-09-17T03:51:53.163002+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:58.012527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205358+0000","last_change":"2024-09-17T03:51:51.186363+0000","last_active":"2024-09-17T03:51:52.205358+0000","last_peered":"2024-09-17T03:51:52.205358+0000","last_clean":"2024-09-17T03:51:52.205358+0000","last_became_active":"2024-09-17T03:51:51.185999+0000","last_became_peered":"2024-09-17T03:51:51.185999+0000","last_unstale":"2024-09-17T03:51:52.205358+0000","last_undegraded":"2024-09-17T03:51:52.205358+0000","last_fullsized":"2024-09-17T03:51:52.205358+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:20.736248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205458+0000","last_change":"2024-09-17T03:51:51.183952+0000","last_active":"2024-09-17T03:51:52.205458+0000","last_peered":"2024-09-17T03:51:52.205458+0000","last_clean":"2024-09-17T03:51:52.205458+0000","last_became_active":"2024-09-17T03:51:51.183349+0000","last_became_peered":"2024-09-17T03:51:51.183349+0000","last_unstale":"2024-09-17T03:51:52.205458+0000","last_undegraded":"2024-09-17T03:51:52.205458+0000","last_fullsized":"2024-09-17T03:51:52.205458+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:06.013944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184056+0000","last_change":"2024-09-17T03:51:51.168140+0000","last_active":"2024-09-17T03:51:52.184056+0000","last_peered":"2024-09-17T03:51:52.184056+0000","last_clean":"2024-09-17T03:51:52.184056+0000","last_became_active":"2024-09-17T03:51:51.167864+0000","last_became_peered":"2024-09-17T03:51:51.167864+0000","last_unstale":"2024-09-17T03:51:52.184056+0000","last_undegraded":"2024-09-17T03:51:52.184056+0000","last_fullsized":"2024-09-17T03:51:52.184056+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:59.424732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174702+0000","last_change":"2024-09-17T03:51:51.188186+0000","last_active":"2024-09-17T03:51:52.174702+0000","last_peered":"2024-09-17T03:51:52.174702+0000","last_clean":"2024-09-17T03:51:52.174702+0000","last_became_active":"2024-09-17T03:51:51.187915+0000","last_became_peered":"2024-09-17T03:51:51.187915+0000","last_unstale":"2024-09-17T03:51:52.174702+0000","last_undegraded":"2024-09-17T03:51:52.174702+0000","last_fullsized":"2024-09-17T03:51:52.174702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:12:14.938776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731395+0000","last_change":"2024-09-17T03:51:51.187987+0000","last_active":"2024-09-17T03:51:53.731395+0000","last_peered":"2024-09-17T03:51:53.731395+0000","last_clean":"2024-09-17T03:51:53.731395+0000","last_became_active":"2024-09-17T03:51:51.187662+0000","last_became_peered":"2024-09-17T03:51:51.187662+0000","last_unstale":"2024-09-17T03:51:53.731395+0000","last_undegraded":"2024-09-17T03:51:53.731395+0000","last_fullsized":"2024-09-17T03:51:53.731395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:11.849047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.175411+0000","last_change":"2024-09-17T03:51:51.200103+0000","last_active":"2024-09-17T03:51:52.175411+0000","last_peered":"2024-09-17T03:51:52.175411+0000","last_clean":"2024-09-17T03:51:52.175411+0000","last_became_active":"2024-09-17T03:51:51.199081+0000","last_became_peered":"2024-09-17T03:51:51.199081+0000","last_unstale":"2024-09-17T03:51:52.175411+0000","last_undegraded":"2024-09-17T03:51:52.175411+0000","last_fullsized":"2024-09-17T03:51:52.175411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:23.689063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174988+0000","last_change":"2024-09-17T03:51:51.187087+0000","last_active":"2024-09-17T03:51:52.174988+0000","last_peered":"2024-09-17T03:51:52.174988+0000","last_clean":"2024-09-17T03:51:52.174988+0000","last_became_active":"2024-09-17T03:51:51.183188+0000","last_became_peered":"2024-09-17T03:51:51.183188+0000","last_unstale":"2024-09-17T03:51:52.174988+0000","last_undegraded":"2024-09-17T03:51:52.174988+0000","last_fullsized":"2024-09-17T03:51:52.174988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:45:27.613702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730730+0000","last_change":"2024-09-17T03:51:51.189420+0000","last_active":"2024-09-17T03:51:53.730730+0000","last_peered":"2024-09-17T03:51:53.730730+0000","last_clean":"2024-09-17T03:51:53.730730+0000","last_became_active":"2024-09-17T03:51:51.187800+0000","last_became_peered":"2024-09-17T03:51:51.187800+0000","last_unstale":"2024-09-17T03:51:53.730730+0000","last_undegraded":"2024-09-17T03:51:53.730730+0000","last_fullsized":"2024-09-17T03:51:53.730730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:54.040014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174059+0000","last_change":"2024-09-17T03:51:51.193515+0000","last_active":"2024-09-17T03:51:52.174059+0000","last_peered":"2024-09-17T03:51:52.174059+0000","last_clean":"2024-09-17T03:51:52.174059+0000","last_became_active":"2024-09-17T03:51:51.193190+0000","last_became_peered":"2024-09-17T03:51:51.193190+0000","last_unstale":"2024-09-17T03:51:52.174059+0000","last_undegraded":"2024-09-17T03:51:52.174059+0000","last_fullsized":"2024-09-17T03:51:52.174059+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:02.818568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730847+0000","last_change":"2024-09-17T03:51:51.194861+0000","last_active":"2024-09-17T03:51:53.730847+0000","last_peered":"2024-09-17T03:51:53.730847+0000","last_clean":"2024-09-17T03:51:53.730847+0000","last_became_active":"2024-09-17T03:51:51.194543+0000","last_became_peered":"2024-09-17T03:51:51.194543+0000","last_unstale":"2024-09-17T03:51:53.730847+0000","last_undegraded":"2024-09-17T03:51:53.730847+0000","last_fullsized":"2024-09-17T03:51:53.730847+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:17.130737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162862+0000","last_change":"2024-09-17T03:51:51.174878+0000","last_active":"2024-09-17T03:51:53.162862+0000","last_peered":"2024-09-17T03:51:53.162862+0000","last_clean":"2024-09-17T03:51:53.162862+0000","last_became_active":"2024-09-17T03:51:51.172940+0000","last_became_peered":"2024-09-17T03:51:51.172940+0000","last_unstale":"2024-09-17T03:51:53.162862+0000","last_undegraded":"2024-09-17T03:51:53.162862+0000","last_fullsized":"2024-09-17T03:51:53.162862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:00.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174370+0000","last_change":"2024-09-17T03:51:51.187429+0000","last_active":"2024-09-17T03:51:52.174370+0000","last_peered":"2024-09-17T03:51:52.174370+0000","last_clean":"2024-09-17T03:51:52.174370+0000","last_became_active":"2024-09-17T03:51:51.187148+0000","last_became_peered":"2024-09-17T03:51:51.187148+0000","last_unstale":"2024-09-17T03:51:52.174370+0000","last_undegraded":"2024-09-17T03:51:52.174370+0000","last_fullsized":"2024-09-17T03:51:52.174370+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:23.692411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174492+0000","last_change":"2024-09-17T03:51:51.179360+0000","last_active":"2024-09-17T03:51:52.174492+0000","last_peered":"2024-09-17T03:51:52.174492+0000","last_clean":"2024-09-17T03:51:52.174492+0000","last_became_active":"2024-09-17T03:51:51.179027+0000","last_became_peered":"2024-09-17T03:51:51.179027+0000","last_unstale":"2024-09-17T03:51:52.174492+0000","last_undegraded":"2024-09-17T03:51:52.174492+0000","last_fullsized":"2024-09-17T03:51:52.174492+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:54.000018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183870+0000","last_change":"2024-09-17T03:51:51.177850+0000","last_active":"2024-09-17T03:51:52.183870+0000","last_peered":"2024-09-17T03:51:52.183870+0000","last_clean":"2024-09-17T03:51:52.183870+0000","last_became_active":"2024-09-17T03:51:51.175315+0000","last_became_peered":"2024-09-17T03:51:51.175315+0000","last_unstale":"2024-09-17T03:51:52.183870+0000","last_undegraded":"2024-09-17T03:51:52.183870+0000","last_fullsized":"2024-09-17T03:51:52.183870+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:08.948416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173591+0000","last_change":"2024-09-17T03:51:51.190479+0000","last_active":"2024-09-17T03:51:52.173591+0000","last_peered":"2024-09-17T03:51:52.173591+0000","last_clean":"2024-09-17T03:51:52.173591+0000","last_became_active":"2024-09-17T03:51:51.190198+0000","last_became_peered":"2024-09-17T03:51:51.190198+0000","last_unstale":"2024-09-17T03:51:52.173591+0000","last_undegraded":"2024-09-17T03:51:52.173591+0000","last_fullsized":"2024-09-17T03:51:52.173591+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:03.246486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174265+0000","last_change":"2024-09-17T03:51:51.185752+0000","last_active":"2024-09-17T03:51:52.174265+0000","last_peered":"2024-09-17T03:51:52.174265+0000","last_clean":"2024-09-17T03:51:52.174265+0000","last_became_active":"2024-09-17T03:51:51.185434+0000","last_became_peered":"2024-09-17T03:51:51.185434+0000","last_unstale":"2024-09-17T03:51:52.174265+0000","last_undegraded":"2024-09-17T03:51:52.174265+0000","last_fullsized":"2024-09-17T03:51:52.174265+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:13.195505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730746+0000","last_change":"2024-09-17T03:51:51.189838+0000","last_active":"2024-09-17T03:51:53.730746+0000","last_peered":"2024-09-17T03:51:53.730746+0000","last_clean":"2024-09-17T03:51:53.730746+0000","last_became_active":"2024-09-17T03:51:51.188891+0000","last_became_peered":"2024-09-17T03:51:51.188891+0000","last_unstale":"2024-09-17T03:51:53.730746+0000","last_undegraded":"2024-09-17T03:51:53.730746+0000","last_fullsized":"2024-09-17T03:51:53.730746+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:40.902497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173346+0000","last_change":"2024-09-17T03:51:51.184999+0000","last_active":"2024-09-17T03:51:52.173346+0000","last_peered":"2024-09-17T03:51:52.173346+0000","last_clean":"2024-09-17T03:51:52.173346+0000","last_became_active":"2024-09-17T03:51:51.184748+0000","last_became_peered":"2024-09-17T03:51:51.184748+0000","last_unstale":"2024-09-17T03:51:52.173346+0000","last_undegraded":"2024-09-17T03:51:52.173346+0000","last_fullsized":"2024-09-17T03:51:52.173346+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:37.468008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183516+0000","last_change":"2024-09-17T03:51:51.189753+0000","last_active":"2024-09-17T03:51:52.183516+0000","last_peered":"2024-09-17T03:51:52.183516+0000","last_clean":"2024-09-17T03:51:52.183516+0000","last_became_active":"2024-09-17T03:51:51.189422+0000","last_became_peered":"2024-09-17T03:51:51.189422+0000","last_unstale":"2024-09-17T03:51:52.183516+0000","last_undegraded":"2024-09-17T03:51:52.183516+0000","last_fullsized":"2024-09-17T03:51:52.183516+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:27.174556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184092+0000","last_change":"2024-09-17T03:51:51.180539+0000","last_active":"2024-09-17T03:51:52.184092+0000","last_peered":"2024-09-17T03:51:52.184092+0000","last_clean":"2024-09-17T03:51:52.184092+0000","last_became_active":"2024-09-17T03:51:51.179021+0000","last_became_peered":"2024-09-17T03:51:51.179021+0000","last_unstale":"2024-09-17T03:51:52.184092+0000","last_undegraded":"2024-09-17T03:51:52.184092+0000","last_fullsized":"2024-09-17T03:51:52.184092+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:45.147765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183961+0000","last_change":"2024-09-17T03:51:50.168035+0000","last_active":"2024-09-17T03:51:52.183961+0000","last_peered":"2024-09-17T03:51:52.183961+0000","last_clean":"2024-09-17T03:51:52.183961+0000","last_became_active":"2024-09-17T03:51:50.167704+0000","last_became_peered":"2024-09-17T03:51:50.167704+0000","last_unstale":"2024-09-17T03:51:52.183961+0000","last_undegraded":"2024-09-17T03:51:52.183961+0000","last_fullsized":"2024-09-17T03:51:52.183961+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:53.499840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610525+0000","last_change":"2024-09-17T03:51:51.176980+0000","last_active":"2024-09-17T03:51:52.610525+0000","last_peered":"2024-09-17T03:51:52.610525+0000","last_clean":"2024-09-17T03:51:52.610525+0000","last_became_active":"2024-09-17T03:51:51.176667+0000","last_became_peered":"2024-09-17T03:51:51.176667+0000","last_unstale":"2024-09-17T03:51:52.610525+0000","last_undegraded":"2024-09-17T03:51:52.610525+0000","last_fullsized":"2024-09-17T03:51:52.610525+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:24.159820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173682+0000","last_change":"2024-09-17T03:51:50.162857+0000","last_active":"2024-09-17T03:51:52.173682+0000","last_peered":"2024-09-17T03:51:52.173682+0000","last_clean":"2024-09-17T03:51:52.173682+0000","last_became_active":"2024-09-17T03:51:50.162573+0000","last_became_peered":"2024-09-17T03:51:50.162573+0000","last_unstale":"2024-09-17T03:51:52.173682+0000","last_undegraded":"2024-09-17T03:51:52.173682+0000","last_fullsized":"2024-09-17T03:51:52.173682+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:22.633136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184039+0000","last_change":"2024-09-17T03:51:51.170143+0000","last_active":"2024-09-17T03:51:52.184039+0000","last_peered":"2024-09-17T03:51:52.184039+0000","last_clean":"2024-09-17T03:51:52.184039+0000","last_became_active":"2024-09-17T03:51:51.169595+0000","last_became_peered":"2024-09-17T03:51:51.169595+0000","last_unstale":"2024-09-17T03:51:52.184039+0000","last_undegraded":"2024-09-17T03:51:52.184039+0000","last_fullsized":"2024-09-17T03:51:52.184039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:20.419660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184162+0000","last_change":"2024-09-17T03:51:50.165145+0000","last_active":"2024-09-17T03:51:52.184162+0000","last_peered":"2024-09-17T03:51:52.184162+0000","last_clean":"2024-09-17T03:51:52.184162+0000","last_became_active":"2024-09-17T03:51:50.164900+0000","last_became_peered":"2024-09-17T03:51:50.164900+0000","last_unstale":"2024-09-17T03:51:52.184162+0000","last_undegraded":"2024-09-17T03:51:52.184162+0000","last_fullsized":"2024-09-17T03:51:52.184162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:06.938674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183864+0000","last_change":"2024-09-17T03:51:51.177537+0000","last_active":"2024-09-17T03:51:52.183864+0000","last_peered":"2024-09-17T03:51:52.183864+0000","last_clean":"2024-09-17T03:51:52.183864+0000","last_became_active":"2024-09-17T03:51:51.174936+0000","last_became_peered":"2024-09-17T03:51:51.174936+0000","last_unstale":"2024-09-17T03:51:52.183864+0000","last_undegraded":"2024-09-17T03:51:52.183864+0000","last_fullsized":"2024-09-17T03:51:52.183864+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:04.586292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174779+0000","last_change":"2024-09-17T03:51:50.167732+0000","last_active":"2024-09-17T03:51:52.174779+0000","last_peered":"2024-09-17T03:51:52.174779+0000","last_clean":"2024-09-17T03:51:52.174779+0000","last_became_active":"2024-09-17T03:51:50.166704+0000","last_became_peered":"2024-09-17T03:51:50.166704+0000","last_unstale":"2024-09-17T03:51:52.174779+0000","last_undegraded":"2024-09-17T03:51:52.174779+0000","last_fullsized":"2024-09-17T03:51:52.174779+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:22.049192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174306+0000","last_change":"2024-09-17T03:51:51.195178+0000","last_active":"2024-09-17T03:51:52.174306+0000","last_peered":"2024-09-17T03:51:52.174306+0000","last_clean":"2024-09-17T03:51:52.174306+0000","last_became_active":"2024-09-17T03:51:51.193567+0000","last_became_peered":"2024-09-17T03:51:51.193567+0000","last_unstale":"2024-09-17T03:51:52.174306+0000","last_undegraded":"2024-09-17T03:51:52.174306+0000","last_fullsized":"2024-09-17T03:51:52.174306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:25.299820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205310+0000","last_change":"2024-09-17T03:51:50.153047+0000","last_active":"2024-09-17T03:51:52.205310+0000","last_peered":"2024-09-17T03:51:52.205310+0000","last_clean":"2024-09-17T03:51:52.205310+0000","last_became_active":"2024-09-17T03:51:50.152719+0000","last_became_peered":"2024-09-17T03:51:50.152719+0000","last_unstale":"2024-09-17T03:51:52.205310+0000","last_undegraded":"2024-09-17T03:51:52.205310+0000","last_fullsized":"2024-09-17T03:51:52.205310+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:21.430041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174037+0000","last_change":"2024-09-17T03:51:51.185697+0000","last_active":"2024-09-17T03:51:52.174037+0000","last_peered":"2024-09-17T03:51:52.174037+0000","last_clean":"2024-09-17T03:51:52.174037+0000","last_became_active":"2024-09-17T03:51:51.185429+0000","last_became_peered":"2024-09-17T03:51:51.185429+0000","last_unstale":"2024-09-17T03:51:52.174037+0000","last_undegraded":"2024-09-17T03:51:52.174037+0000","last_fullsized":"2024-09-17T03:51:52.174037+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:46.688579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183576+0000","last_change":"2024-09-17T03:51:50.166105+0000","last_active":"2024-09-17T03:51:52.183576+0000","last_peered":"2024-09-17T03:51:52.183576+0000","last_clean":"2024-09-17T03:51:52.183576+0000","last_became_active":"2024-09-17T03:51:50.165818+0000","last_became_peered":"2024-09-17T03:51:50.165818+0000","last_unstale":"2024-09-17T03:51:52.183576+0000","last_undegraded":"2024-09-17T03:51:52.183576+0000","last_fullsized":"2024-09-17T03:51:52.183576+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.880243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162846+0000","last_change":"2024-09-17T03:51:51.177152+0000","last_active":"2024-09-17T03:51:53.162846+0000","last_peered":"2024-09-17T03:51:53.162846+0000","last_clean":"2024-09-17T03:51:53.162846+0000","last_became_active":"2024-09-17T03:51:51.176806+0000","last_became_peered":"2024-09-17T03:51:51.176806+0000","last_unstale":"2024-09-17T03:51:53.162846+0000","last_undegraded":"2024-09-17T03:51:53.162846+0000","last_fullsized":"2024-09-17T03:51:53.162846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:09.307104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173657+0000","last_change":"2024-09-17T03:51:50.169610+0000","last_active":"2024-09-17T03:51:52.173657+0000","last_peered":"2024-09-17T03:51:52.173657+0000","last_clean":"2024-09-17T03:51:52.173657+0000","last_became_active":"2024-09-17T03:51:50.169318+0000","last_became_peered":"2024-09-17T03:51:50.169318+0000","last_unstale":"2024-09-17T03:51:52.173657+0000","last_undegraded":"2024-09-17T03:51:52.173657+0000","last_fullsized":"2024-09-17T03:51:52.173657+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.928869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173528+0000","last_change":"2024-09-17T03:51:51.192692+0000","last_active":"2024-09-17T03:51:52.173528+0000","last_peered":"2024-09-17T03:51:52.173528+0000","last_clean":"2024-09-17T03:51:52.173528+0000","last_became_active":"2024-09-17T03:51:51.192404+0000","last_became_peered":"2024-09-17T03:51:51.192404+0000","last_unstale":"2024-09-17T03:51:52.173528+0000","last_undegraded":"2024-09-17T03:51:52.173528+0000","last_fullsized":"2024-09-17T03:51:52.173528+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:25.262002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205190+0000","last_change":"2024-09-17T03:51:50.160392+0000","last_active":"2024-09-17T03:51:52.205190+0000","last_peered":"2024-09-17T03:51:52.205190+0000","last_clean":"2024-09-17T03:51:52.205190+0000","last_became_active":"2024-09-17T03:51:50.159973+0000","last_became_peered":"2024-09-17T03:51:50.159973+0000","last_unstale":"2024-09-17T03:51:52.205190+0000","last_undegraded":"2024-09-17T03:51:52.205190+0000","last_fullsized":"2024-09-17T03:51:52.205190+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:42.221660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205766+0000","last_change":"2024-09-17T03:51:51.184296+0000","last_active":"2024-09-17T03:51:52.205766+0000","last_peered":"2024-09-17T03:51:52.205766+0000","last_clean":"2024-09-17T03:51:52.205766+0000","last_became_active":"2024-09-17T03:51:51.183943+0000","last_became_peered":"2024-09-17T03:51:51.183943+0000","last_unstale":"2024-09-17T03:51:52.205766+0000","last_undegraded":"2024-09-17T03:51:52.205766+0000","last_fullsized":"2024-09-17T03:51:52.205766+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:55.346872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731380+0000","last_change":"2024-09-17T03:51:50.167430+0000","last_active":"2024-09-17T03:51:53.731380+0000","last_peered":"2024-09-17T03:51:53.731380+0000","last_clean":"2024-09-17T03:51:53.731380+0000","last_became_active":"2024-09-17T03:51:50.167110+0000","last_became_peered":"2024-09-17T03:51:50.167110+0000","last_unstale":"2024-09-17T03:51:53.731380+0000","last_undegraded":"2024-09-17T03:51:53.731380+0000","last_fullsized":"2024-09-17T03:51:53.731380+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:58.416043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731518+0000","last_change":"2024-09-17T03:51:51.192660+0000","last_active":"2024-09-17T03:51:53.731518+0000","last_peered":"2024-09-17T03:51:53.731518+0000","last_clean":"2024-09-17T03:51:53.731518+0000","last_became_active":"2024-09-17T03:51:51.192294+0000","last_became_peered":"2024-09-17T03:51:51.192294+0000","last_unstale":"2024-09-17T03:51:53.731518+0000","last_undegraded":"2024-09-17T03:51:53.731518+0000","last_fullsized":"2024-09-17T03:51:53.731518+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:06.260210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173787+0000","last_change":"2024-09-17T03:51:50.164231+0000","last_active":"2024-09-17T03:51:52.173787+0000","last_peered":"2024-09-17T03:51:52.173787+0000","last_clean":"2024-09-17T03:51:52.173787+0000","last_became_active":"2024-09-17T03:51:50.163955+0000","last_became_peered":"2024-09-17T03:51:50.163955+0000","last_unstale":"2024-09-17T03:51:52.173787+0000","last_undegraded":"2024-09-17T03:51:52.173787+0000","last_fullsized":"2024-09-17T03:51:52.173787+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:33:48.844463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610329+0000","last_change":"2024-09-17T03:51:51.183261+0000","last_active":"2024-09-17T03:51:52.610329+0000","last_peered":"2024-09-17T03:51:52.610329+0000","last_clean":"2024-09-17T03:51:52.610329+0000","last_became_active":"2024-09-17T03:51:51.182895+0000","last_became_peered":"2024-09-17T03:51:51.182895+0000","last_unstale":"2024-09-17T03:51:52.610329+0000","last_undegraded":"2024-09-17T03:51:52.610329+0000","last_fullsized":"2024-09-17T03:51:52.610329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:00.993310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731130+0000","last_change":"2024-09-17T03:51:50.162510+0000","last_active":"2024-09-17T03:51:53.731130+0000","last_peered":"2024-09-17T03:51:53.731130+0000","last_clean":"2024-09-17T03:51:53.731130+0000","last_became_active":"2024-09-17T03:51:50.162216+0000","last_became_peered":"2024-09-17T03:51:50.162216+0000","last_unstale":"2024-09-17T03:51:53.731130+0000","last_undegraded":"2024-09-17T03:51:53.731130+0000","last_fullsized":"2024-09-17T03:51:53.731130+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:22:36.660509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731050+0000","last_change":"2024-09-17T03:51:51.185826+0000","last_active":"2024-09-17T03:51:53.731050+0000","last_peered":"2024-09-17T03:51:53.731050+0000","last_clean":"2024-09-17T03:51:53.731050+0000","last_became_active":"2024-09-17T03:51:51.182569+0000","last_became_peered":"2024-09-17T03:51:51.182569+0000","last_unstale":"2024-09-17T03:51:53.731050+0000","last_undegraded":"2024-09-17T03:51:53.731050+0000","last_fullsized":"2024-09-17T03:51:53.731050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:33:49.851418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205622+0000","last_change":"2024-09-17T03:51:50.167752+0000","last_active":"2024-09-17T03:51:52.205622+0000","last_peered":"2024-09-17T03:51:52.205622+0000","last_clean":"2024-09-17T03:51:52.205622+0000","last_became_active":"2024-09-17T03:51:50.166168+0000","last_became_peered":"2024-09-17T03:51:50.166168+0000","last_unstale":"2024-09-17T03:51:52.205622+0000","last_undegraded":"2024-09-17T03:51:52.205622+0000","last_fullsized":"2024-09-17T03:51:52.205622+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:59.986355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173826+0000","last_change":"2024-09-17T03:51:51.186944+0000","last_active":"2024-09-17T03:51:52.173826+0000","last_peered":"2024-09-17T03:51:52.173826+0000","last_clean":"2024-09-17T03:51:52.173826+0000","last_became_active":"2024-09-17T03:51:51.184771+0000","last_became_peered":"2024-09-17T03:51:51.184771+0000","last_unstale":"2024-09-17T03:51:52.173826+0000","last_undegraded":"2024-09-17T03:51:52.173826+0000","last_fullsized":"2024-09-17T03:51:52.173826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:43.126626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173443+0000","last_change":"2024-09-17T03:51:50.163723+0000","last_active":"2024-09-17T03:51:52.173443+0000","last_peered":"2024-09-17T03:51:52.173443+0000","last_clean":"2024-09-17T03:51:52.173443+0000","last_became_active":"2024-09-17T03:51:50.163324+0000","last_became_peered":"2024-09-17T03:51:50.163324+0000","last_unstale":"2024-09-17T03:51:52.173443+0000","last_undegraded":"2024-09-17T03:51:52.173443+0000","last_fullsized":"2024-09-17T03:51:52.173443+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:16.123782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730879+0000","last_change":"2024-09-17T03:51:51.189553+0000","last_active":"2024-09-17T03:51:53.730879+0000","last_peered":"2024-09-17T03:51:53.730879+0000","last_clean":"2024-09-17T03:51:53.730879+0000","last_became_active":"2024-09-17T03:51:51.187557+0000","last_became_peered":"2024-09-17T03:51:51.187557+0000","last_unstale":"2024-09-17T03:51:53.730879+0000","last_undegraded":"2024-09-17T03:51:53.730879+0000","last_fullsized":"2024-09-17T03:51:53.730879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:26.699117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162654+0000","last_change":"2024-09-17T03:51:50.164735+0000","last_active":"2024-09-17T03:51:53.162654+0000","last_peered":"2024-09-17T03:51:53.162654+0000","last_clean":"2024-09-17T03:51:53.162654+0000","last_became_active":"2024-09-17T03:51:50.164378+0000","last_became_peered":"2024-09-17T03:51:50.164378+0000","last_unstale":"2024-09-17T03:51:53.162654+0000","last_undegraded":"2024-09-17T03:51:53.162654+0000","last_fullsized":"2024-09-17T03:51:53.162654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:35.436131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205301+0000","last_change":"2024-09-17T03:51:51.186075+0000","last_active":"2024-09-17T03:51:52.205301+0000","last_peered":"2024-09-17T03:51:52.205301+0000","last_clean":"2024-09-17T03:51:52.205301+0000","last_became_active":"2024-09-17T03:51:51.185742+0000","last_became_peered":"2024-09-17T03:51:51.185742+0000","last_unstale":"2024-09-17T03:51:52.205301+0000","last_undegraded":"2024-09-17T03:51:52.205301+0000","last_fullsized":"2024-09-17T03:51:52.205301+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.551761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173663+0000","last_change":"2024-09-17T03:51:50.166686+0000","last_active":"2024-09-17T03:51:52.173663+0000","last_peered":"2024-09-17T03:51:52.173663+0000","last_clean":"2024-09-17T03:51:52.173663+0000","last_became_active":"2024-09-17T03:51:50.165924+0000","last_became_peered":"2024-09-17T03:51:50.165924+0000","last_unstale":"2024-09-17T03:51:52.173663+0000","last_undegraded":"2024-09-17T03:51:52.173663+0000","last_fullsized":"2024-09-17T03:51:52.173663+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:23.152865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730823+0000","last_change":"2024-09-17T03:51:51.199074+0000","last_active":"2024-09-17T03:51:53.730823+0000","last_peered":"2024-09-17T03:51:53.730823+0000","last_clean":"2024-09-17T03:51:53.730823+0000","last_became_active":"2024-09-17T03:51:51.198758+0000","last_became_peered":"2024-09-17T03:51:51.198758+0000","last_unstale":"2024-09-17T03:51:53.730823+0000","last_undegraded":"2024-09-17T03:51:53.730823+0000","last_fullsized":"2024-09-17T03:51:53.730823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:04.175415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730361+0000","last_change":"2024-09-17T03:51:50.173075+0000","last_active":"2024-09-17T03:51:53.730361+0000","last_peered":"2024-09-17T03:51:53.730361+0000","last_clean":"2024-09-17T03:51:53.730361+0000","last_became_active":"2024-09-17T03:51:50.172690+0000","last_became_peered":"2024-09-17T03:51:50.172690+0000","last_unstale":"2024-09-17T03:51:53.730361+0000","last_undegraded":"2024-09-17T03:51:53.730361+0000","last_fullsized":"2024-09-17T03:51:53.730361+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:41.566073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731446+0000","last_change":"2024-09-17T03:51:51.181582+0000","last_active":"2024-09-17T03:51:53.731446+0000","last_peered":"2024-09-17T03:51:53.731446+0000","last_clean":"2024-09-17T03:51:53.731446+0000","last_became_active":"2024-09-17T03:51:51.181239+0000","last_became_peered":"2024-09-17T03:51:51.181239+0000","last_unstale":"2024-09-17T03:51:53.731446+0000","last_undegraded":"2024-09-17T03:51:53.731446+0000","last_fullsized":"2024-09-17T03:51:53.731446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:31.955299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174773+0000","last_change":"2024-09-17T03:51:50.167328+0000","last_active":"2024-09-17T03:51:52.174773+0000","last_peered":"2024-09-17T03:51:52.174773+0000","last_clean":"2024-09-17T03:51:52.174773+0000","last_became_active":"2024-09-17T03:51:50.167022+0000","last_became_peered":"2024-09-17T03:51:50.167022+0000","last_unstale":"2024-09-17T03:51:52.174773+0000","last_undegraded":"2024-09-17T03:51:52.174773+0000","last_fullsized":"2024-09-17T03:51:52.174773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:03.168460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174474+0000","last_change":"2024-09-17T03:51:51.166682+0000","last_active":"2024-09-17T03:51:52.174474+0000","last_peered":"2024-09-17T03:51:52.174474+0000","last_clean":"2024-09-17T03:51:52.174474+0000","last_became_active":"2024-09-17T03:51:51.166356+0000","last_became_peered":"2024-09-17T03:51:51.166356+0000","last_unstale":"2024-09-17T03:51:52.174474+0000","last_undegraded":"2024-09-17T03:51:52.174474+0000","last_fullsized":"2024-09-17T03:51:52.174474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184201+0000","last_change":"2024-09-17T03:51:50.156732+0000","last_active":"2024-09-17T03:51:52.184201+0000","last_peered":"2024-09-17T03:51:52.184201+0000","last_clean":"2024-09-17T03:51:52.184201+0000","last_became_active":"2024-09-17T03:51:50.156393+0000","last_became_peered":"2024-09-17T03:51:50.156393+0000","last_unstale":"2024-09-17T03:51:52.184201+0000","last_undegraded":"2024-09-17T03:51:52.184201+0000","last_fullsized":"2024-09-17T03:51:52.184201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:00.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173838+0000","last_change":"2024-09-17T03:51:51.182538+0000","last_active":"2024-09-17T03:51:52.173838+0000","last_peered":"2024-09-17T03:51:52.173838+0000","last_clean":"2024-09-17T03:51:52.173838+0000","last_became_active":"2024-09-17T03:51:51.182255+0000","last_became_peered":"2024-09-17T03:51:51.182255+0000","last_unstale":"2024-09-17T03:51:52.173838+0000","last_undegraded":"2024-09-17T03:51:52.173838+0000","last_fullsized":"2024-09-17T03:51:52.173838+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:31.579943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174264+0000","last_change":"2024-09-17T03:51:50.169032+0000","last_active":"2024-09-17T03:51:52.174264+0000","last_peered":"2024-09-17T03:51:52.174264+0000","last_clean":"2024-09-17T03:51:52.174264+0000","last_became_active":"2024-09-17T03:51:50.168178+0000","last_became_peered":"2024-09-17T03:51:50.168178+0000","last_unstale":"2024-09-17T03:51:52.174264+0000","last_undegraded":"2024-09-17T03:51:52.174264+0000","last_fullsized":"2024-09-17T03:51:52.174264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:37.469802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731094+0000","last_change":"2024-09-17T03:51:51.188487+0000","last_active":"2024-09-17T03:51:53.731094+0000","last_peered":"2024-09-17T03:51:53.731094+0000","last_clean":"2024-09-17T03:51:53.731094+0000","last_became_active":"2024-09-17T03:51:51.188169+0000","last_became_peered":"2024-09-17T03:51:51.188169+0000","last_unstale":"2024-09-17T03:51:53.731094+0000","last_undegraded":"2024-09-17T03:51:53.731094+0000","last_fullsized":"2024-09-17T03:51:53.731094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:18.318333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184064+0000","last_change":"2024-09-17T03:51:50.161576+0000","last_active":"2024-09-17T03:51:52.184064+0000","last_peered":"2024-09-17T03:51:52.184064+0000","last_clean":"2024-09-17T03:51:52.184064+0000","last_became_active":"2024-09-17T03:51:50.161249+0000","last_became_peered":"2024-09-17T03:51:50.161249+0000","last_unstale":"2024-09-17T03:51:52.184064+0000","last_undegraded":"2024-09-17T03:51:52.184064+0000","last_fullsized":"2024-09-17T03:51:52.184064+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:47:39.842117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174145+0000","last_change":"2024-09-17T03:51:51.186537+0000","last_active":"2024-09-17T03:51:52.174145+0000","last_peered":"2024-09-17T03:51:52.174145+0000","last_clean":"2024-09-17T03:51:52.174145+0000","last_became_active":"2024-09-17T03:51:51.186087+0000","last_became_peered":"2024-09-17T03:51:51.186087+0000","last_unstale":"2024-09-17T03:51:52.174145+0000","last_undegraded":"2024-09-17T03:51:52.174145+0000","last_fullsized":"2024-09-17T03:51:52.174145+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:54.654394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174342+0000","last_change":"2024-09-17T03:51:50.158489+0000","last_active":"2024-09-17T03:51:52.174342+0000","last_peered":"2024-09-17T03:51:52.174342+0000","last_clean":"2024-09-17T03:51:52.174342+0000","last_became_active":"2024-09-17T03:51:50.157069+0000","last_became_peered":"2024-09-17T03:51:50.157069+0000","last_unstale":"2024-09-17T03:51:52.174342+0000","last_undegraded":"2024-09-17T03:51:52.174342+0000","last_fullsized":"2024-09-17T03:51:52.174342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:25.692162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183880+0000","last_change":"2024-09-17T03:51:51.174656+0000","last_active":"2024-09-17T03:51:52.183880+0000","last_peered":"2024-09-17T03:51:52.183880+0000","last_clean":"2024-09-17T03:51:52.183880+0000","last_became_active":"2024-09-17T03:51:51.174341+0000","last_became_peered":"2024-09-17T03:51:51.174341+0000","last_unstale":"2024-09-17T03:51:52.183880+0000","last_undegraded":"2024-09-17T03:51:52.183880+0000","last_fullsized":"2024-09-17T03:51:52.183880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:21.924725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610096+0000","last_change":"2024-09-17T03:51:50.161426+0000","last_active":"2024-09-17T03:51:52.610096+0000","last_peered":"2024-09-17T03:51:52.610096+0000","last_clean":"2024-09-17T03:51:52.610096+0000","last_became_active":"2024-09-17T03:51:50.159750+0000","last_became_peered":"2024-09-17T03:51:50.159750+0000","last_unstale":"2024-09-17T03:51:52.610096+0000","last_undegraded":"2024-09-17T03:51:52.610096+0000","last_fullsized":"2024-09-17T03:51:52.610096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:28.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183722+0000","last_change":"2024-09-17T03:51:51.179189+0000","last_active":"2024-09-17T03:51:52.183722+0000","last_peered":"2024-09-17T03:51:52.183722+0000","last_clean":"2024-09-17T03:51:52.183722+0000","last_became_active":"2024-09-17T03:51:51.178873+0000","last_became_peered":"2024-09-17T03:51:51.178873+0000","last_unstale":"2024-09-17T03:51:52.183722+0000","last_undegraded":"2024-09-17T03:51:52.183722+0000","last_fullsized":"2024-09-17T03:51:52.183722+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:44.713006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173638+0000","last_change":"2024-09-17T03:51:50.171471+0000","last_active":"2024-09-17T03:51:52.173638+0000","last_peered":"2024-09-17T03:51:52.173638+0000","last_clean":"2024-09-17T03:51:52.173638+0000","last_became_active":"2024-09-17T03:51:50.169065+0000","last_became_peered":"2024-09-17T03:51:50.169065+0000","last_unstale":"2024-09-17T03:51:52.173638+0000","last_undegraded":"2024-09-17T03:51:52.173638+0000","last_fullsized":"2024-09-17T03:51:52.173638+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:57.382485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174698+0000","last_change":"2024-09-17T03:51:51.182912+0000","last_active":"2024-09-17T03:51:52.174698+0000","last_peered":"2024-09-17T03:51:52.174698+0000","last_clean":"2024-09-17T03:51:52.174698+0000","last_became_active":"2024-09-17T03:51:51.182620+0000","last_became_peered":"2024-09-17T03:51:51.182620+0000","last_unstale":"2024-09-17T03:51:52.174698+0000","last_undegraded":"2024-09-17T03:51:52.174698+0000","last_fullsized":"2024-09-17T03:51:52.174698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:28.435503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610332+0000","last_change":"2024-09-17T03:51:50.160376+0000","last_active":"2024-09-17T03:51:52.610332+0000","last_peered":"2024-09-17T03:51:52.610332+0000","last_clean":"2024-09-17T03:51:52.610332+0000","last_became_active":"2024-09-17T03:51:50.159950+0000","last_became_peered":"2024-09-17T03:51:50.159950+0000","last_unstale":"2024-09-17T03:51:52.610332+0000","last_undegraded":"2024-09-17T03:51:52.610332+0000","last_fullsized":"2024-09-17T03:51:52.610332+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:29.610642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174651+0000","last_change":"2024-09-17T03:51:51.178739+0000","last_active":"2024-09-17T03:51:52.174651+0000","last_peered":"2024-09-17T03:51:52.174651+0000","last_clean":"2024-09-17T03:51:52.174651+0000","last_became_active":"2024-09-17T03:51:51.177453+0000","last_became_peered":"2024-09-17T03:51:51.177453+0000","last_unstale":"2024-09-17T03:51:52.174651+0000","last_undegraded":"2024-09-17T03:51:52.174651+0000","last_fullsized":"2024-09-17T03:51:52.174651+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:15.833137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162007+0000","last_change":"2024-09-17T03:51:50.166779+0000","last_active":"2024-09-17T03:51:53.162007+0000","last_peered":"2024-09-17T03:51:53.162007+0000","last_clean":"2024-09-17T03:51:53.162007+0000","last_became_active":"2024-09-17T03:51:50.166433+0000","last_became_peered":"2024-09-17T03:51:50.166433+0000","last_unstale":"2024-09-17T03:51:53.162007+0000","last_undegraded":"2024-09-17T03:51:53.162007+0000","last_fullsized":"2024-09-17T03:51:53.162007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:43.023544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610361+0000","last_change":"2024-09-17T03:51:51.180525+0000","last_active":"2024-09-17T03:51:52.610361+0000","last_peered":"2024-09-17T03:51:52.610361+0000","last_clean":"2024-09-17T03:51:52.610361+0000","last_became_active":"2024-09-17T03:51:51.174354+0000","last_became_peered":"2024-09-17T03:51:51.174354+0000","last_unstale":"2024-09-17T03:51:52.610361+0000","last_undegraded":"2024-09-17T03:51:52.610361+0000","last_fullsized":"2024-09-17T03:51:52.610361+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:58.389440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.163000+0000","last_change":"2024-09-17T03:51:50.168996+0000","last_active":"2024-09-17T03:51:53.163000+0000","last_peered":"2024-09-17T03:51:53.163000+0000","last_clean":"2024-09-17T03:51:53.163000+0000","last_became_active":"2024-09-17T03:51:50.167892+0000","last_became_peered":"2024-09-17T03:51:50.167892+0000","last_unstale":"2024-09-17T03:51:53.163000+0000","last_undegraded":"2024-09-17T03:51:53.163000+0000","last_fullsized":"2024-09-17T03:51:53.163000+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:12.026481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184103+0000","last_change":"2024-09-17T03:51:51.177189+0000","last_active":"2024-09-17T03:51:52.184103+0000","last_peered":"2024-09-17T03:51:52.184103+0000","last_clean":"2024-09-17T03:51:52.184103+0000","last_became_active":"2024-09-17T03:51:51.174651+0000","last_became_peered":"2024-09-17T03:51:51.174651+0000","last_unstale":"2024-09-17T03:51:52.184103+0000","last_undegraded":"2024-09-17T03:51:52.184103+0000","last_fullsized":"2024-09-17T03:51:52.184103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:31:54.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174857+0000","last_change":"2024-09-17T03:51:50.164737+0000","last_active":"2024-09-17T03:51:52.174857+0000","last_peered":"2024-09-17T03:51:52.174857+0000","last_clean":"2024-09-17T03:51:52.174857+0000","last_became_active":"2024-09-17T03:51:50.164417+0000","last_became_peered":"2024-09-17T03:51:50.164417+0000","last_unstale":"2024-09-17T03:51:52.174857+0000","last_undegraded":"2024-09-17T03:51:52.174857+0000","last_fullsized":"2024-09-17T03:51:52.174857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:36.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174550+0000","last_change":"2024-09-17T03:51:51.170043+0000","last_active":"2024-09-17T03:51:52.174550+0000","last_peered":"2024-09-17T03:51:52.174550+0000","last_clean":"2024-09-17T03:51:52.174550+0000","last_became_active":"2024-09-17T03:51:51.169768+0000","last_became_peered":"2024-09-17T03:51:51.169768+0000","last_unstale":"2024-09-17T03:51:52.174550+0000","last_undegraded":"2024-09-17T03:51:52.174550+0000","last_fullsized":"2024-09-17T03:51:52.174550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:46.510444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610005+0000","last_change":"2024-09-17T03:51:50.164044+0000","last_active":"2024-09-17T03:51:52.610005+0000","last_peered":"2024-09-17T03:51:52.610005+0000","last_clean":"2024-09-17T03:51:52.610005+0000","last_became_active":"2024-09-17T03:51:50.163701+0000","last_became_peered":"2024-09-17T03:51:50.163701+0000","last_unstale":"2024-09-17T03:51:52.610005+0000","last_undegraded":"2024-09-17T03:51:52.610005+0000","last_fullsized":"2024-09-17T03:51:52.610005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:33.411551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731065+0000","last_change":"2024-09-17T03:51:51.196389+0000","last_active":"2024-09-17T03:51:53.731065+0000","last_peered":"2024-09-17T03:51:53.731065+0000","last_clean":"2024-09-17T03:51:53.731065+0000","last_became_active":"2024-09-17T03:51:51.196010+0000","last_became_peered":"2024-09-17T03:51:51.196010+0000","last_unstale":"2024-09-17T03:51:53.731065+0000","last_undegraded":"2024-09-17T03:51:53.731065+0000","last_fullsized":"2024-09-17T03:51:53.731065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:09.419164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205200+0000","last_change":"2024-09-17T03:51:50.156739+0000","last_active":"2024-09-17T03:51:52.205200+0000","last_peered":"2024-09-17T03:51:52.205200+0000","last_clean":"2024-09-17T03:51:52.205200+0000","last_became_active":"2024-09-17T03:51:50.155742+0000","last_became_peered":"2024-09-17T03:51:50.155742+0000","last_unstale":"2024-09-17T03:51:52.205200+0000","last_undegraded":"2024-09-17T03:51:52.205200+0000","last_fullsized":"2024-09-17T03:51:52.205200+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:05.278517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610117+0000","last_change":"2024-09-17T03:51:51.165566+0000","last_active":"2024-09-17T03:51:52.610117+0000","last_peered":"2024-09-17T03:51:52.610117+0000","last_clean":"2024-09-17T03:51:52.610117+0000","last_became_active":"2024-09-17T03:51:51.165141+0000","last_became_peered":"2024-09-17T03:51:51.165141+0000","last_unstale":"2024-09-17T03:51:52.610117+0000","last_undegraded":"2024-09-17T03:51:52.610117+0000","last_fullsized":"2024-09-17T03:51:52.610117+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:22.436996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610015+0000","last_change":"2024-09-17T03:51:50.168219+0000","last_active":"2024-09-17T03:51:52.610015+0000","last_peered":"2024-09-17T03:51:52.610015+0000","last_clean":"2024-09-17T03:51:52.610015+0000","last_became_active":"2024-09-17T03:51:50.167872+0000","last_became_peered":"2024-09-17T03:51:50.167872+0000","last_unstale":"2024-09-17T03:51:52.610015+0000","last_undegraded":"2024-09-17T03:51:52.610015+0000","last_fullsized":"2024-09-17T03:51:52.610015+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:35:50.289731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173794+0000","last_change":"2024-09-17T03:51:51.189223+0000","last_active":"2024-09-17T03:51:52.173794+0000","last_peered":"2024-09-17T03:51:52.173794+0000","last_clean":"2024-09-17T03:51:52.173794+0000","last_became_active":"2024-09-17T03:51:51.188908+0000","last_became_peered":"2024-09-17T03:51:51.188908+0000","last_unstale":"2024-09-17T03:51:52.173794+0000","last_undegraded":"2024-09-17T03:51:52.173794+0000","last_fullsized":"2024-09-17T03:51:52.173794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:11.721027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173198+0000","last_change":"2024-09-17T03:51:50.172844+0000","last_active":"2024-09-17T03:51:52.173198+0000","last_peered":"2024-09-17T03:51:52.173198+0000","last_clean":"2024-09-17T03:51:52.173198+0000","last_became_active":"2024-09-17T03:51:50.172507+0000","last_became_peered":"2024-09-17T03:51:50.172507+0000","last_unstale":"2024-09-17T03:51:52.173198+0000","last_undegraded":"2024-09-17T03:51:52.173198+0000","last_fullsized":"2024-09-17T03:51:52.173198+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:53.033059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173133+0000","last_change":"2024-09-17T03:51:51.183736+0000","last_active":"2024-09-17T03:51:52.173133+0000","last_peered":"2024-09-17T03:51:52.173133+0000","last_clean":"2024-09-17T03:51:52.173133+0000","last_became_active":"2024-09-17T03:51:51.179462+0000","last_became_peered":"2024-09-17T03:51:51.179462+0000","last_unstale":"2024-09-17T03:51:52.173133+0000","last_undegraded":"2024-09-17T03:51:52.173133+0000","last_fullsized":"2024-09-17T03:51:52.173133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:57.563505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162728+0000","last_change":"2024-09-17T03:51:50.171420+0000","last_active":"2024-09-17T03:51:53.162728+0000","last_peered":"2024-09-17T03:51:53.162728+0000","last_clean":"2024-09-17T03:51:53.162728+0000","last_became_active":"2024-09-17T03:51:50.171094+0000","last_became_peered":"2024-09-17T03:51:50.171094+0000","last_unstale":"2024-09-17T03:51:53.162728+0000","last_undegraded":"2024-09-17T03:51:53.162728+0000","last_fullsized":"2024-09-17T03:51:53.162728+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:00.559563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":172,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730375+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:53.730375+0000","last_peered":"2024-09-17T03:51:53.730375+0000","last_clean":"2024-09-17T03:51:53.730375+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:53.730375+0000","last_undegraded":"2024-09-17T03:51:53.730375+0000","last_fullsized":"2024-09-17T03:51:53.730375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509472,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1265664,"data_stored":862593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":675840,"data_stored":273677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509472,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509474,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":667648,"data_stored":272155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542176,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":651264,"data_stored":265765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1273856,"data_stored":864627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:55.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:51:55.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 -- 172.21.15.129:0/2570163669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92500ce020 msgr2=0x7f92500ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 --2- 172.21.15.129:0/2570163669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92500ce020 0x7f92500ce430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f92500cf170 tx=0x7f924802aac0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 -- 172.21.15.129:0/2570163669 shutdown_connections 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 --2- 172.21.15.129:0/2570163669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92500ce020 0x7f92500ce430 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 --2- 172.21.15.129:0/2570163669 >> v2:172.21.15.181:3300/0 conn(0x7f92500cd240 0x7f92500cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 -- 172.21.15.129:0/2570163669 >> 172.21.15.129:0/2570163669 conn(0x7f92500b1cb0 msgr2=0x7f92500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 -- 172.21.15.129:0/2570163669 shutdown_connections 2024-09-17T03:51:55.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.855+0000 7f9258600640 1 -- 172.21.15.129:0/2570163669 wait complete. 2024-09-17T03:51:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.857+0000 7f9258600640 1 Processor -- start 2024-09-17T03:51:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.858+0000 7f9258600640 1 -- start start 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92500ce020 0x7f92501ea370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92500cf6f0 0x7f92501f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92500701b0 con 0x7f92501616c0 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92500cf3d0 con 0x7f92500cf6f0 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9258600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9250070330 con 0x7f92500ce020 2024-09-17T03:51:55.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9256600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9256600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49778/0 (socket says 172.21.15.129:49778) 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.859+0000 7f9256600640 1 -- 172.21.15.129:0/2022710052 learned_addr learned my addr 172.21.15.129:0/2022710052 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9255200640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92500cf6f0 0x7f92501f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9255c00640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92500ce020 0x7f92501ea370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 -- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92500ce020 msgr2=0x7f92501ea370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92500ce020 0x7f92501ea370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 -- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92500cf6f0 msgr2=0x7f92501f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92500cf6f0 0x7f92501f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 -- 172.21.15.129:0/2022710052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92501ea8b0 con 0x7f92501616c0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9256600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d60 tx=0x7f9248003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248044070 con 0x7f92501616c0 2024-09-17T03:51:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9248003e50 con 0x7f92501616c0 2024-09-17T03:51:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92501f5f00 con 0x7f92501616c0 2024-09-17T03:51:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.860+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480044b0 con 0x7f92501616c0 2024-09-17T03:51:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.861+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92501e9b70 con 0x7f92501616c0 2024-09-17T03:51:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.862+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92500ceb30 con 0x7f92501616c0 2024-09-17T03:51:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.863+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9248004650 con 0x7f92501616c0 2024-09-17T03:51:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.864+0000 7f924d600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 0x7f923007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.864+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f92480bb2a0 con 0x7f92501616c0 2024-09-17T03:51:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.864+0000 7f9255c00640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 0x7f923007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.865+0000 7f9255c00640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 0x7f923007b0a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9250071290 tx=0x7f9244009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:55.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:55.868+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9248032030 con 0x7f92501616c0 2024-09-17T03:51:56.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.048+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92501f5c90 con 0x7f9230078b90 2024-09-17T03:51:56.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.053+0000 7f924d600640 1 -- 172.21.15.129:0/2022710052 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251943 (secure 0 0 0) 0x7f92501f5c90 con 0x7f9230078b90 2024-09-17T03:51:56.056 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:56.057 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 msgr2=0x7f923007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 0x7f923007b0a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9250071290 tx=0x7f9244009040 comp rx=0 tx=0).stop 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 msgr2=0x7f9250161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9248009d60 tx=0x7f9248003ce0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 shutdown_connections 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9230078b90 0x7f923007b0a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92501616c0 0x7f9250161ad0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92500cf6f0 0x7f92501f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 --2- 172.21.15.129:0/2022710052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92500ce020 0x7f92501ea370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 >> 172.21.15.129:0/2022710052 conn(0x7f92500b1cb0 msgr2=0x7f925009dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:56.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 shutdown_connections 2024-09-17T03:51:56.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.057+0000 7f9258600640 1 -- 172.21.15.129:0/2022710052 wait complete. 2024-09-17T03:51:56.081 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-17T03:51:54.329668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255956,"kb_used_data":6396,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754718764,"statfs":{"total":773094113280,"available":772832014336,"internally_reserved":0,"allocated":6549504,"data_stored":3355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.193634"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174486+0000","last_change":"2024-09-17T03:51:51.199588+0000","last_active":"2024-09-17T03:51:52.174486+0000","last_peered":"2024-09-17T03:51:52.174486+0000","last_clean":"2024-09-17T03:51:52.174486+0000","last_became_active":"2024-09-17T03:51:51.199297+0000","last_became_peered":"2024-09-17T03:51:51.199297+0000","last_unstale":"2024-09-17T03:51:52.174486+0000","last_undegraded":"2024-09-17T03:51:52.174486+0000","last_fullsized":"2024-09-17T03:51:52.174486+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:07.155723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173622+0000","last_change":"2024-09-17T03:51:51.187428+0000","last_active":"2024-09-17T03:51:52.173622+0000","last_peered":"2024-09-17T03:51:52.173622+0000","last_clean":"2024-09-17T03:51:52.173622+0000","last_became_active":"2024-09-17T03:51:51.186526+0000","last_became_peered":"2024-09-17T03:51:51.186526+0000","last_unstale":"2024-09-17T03:51:52.173622+0000","last_undegraded":"2024-09-17T03:51:52.173622+0000","last_fullsized":"2024-09-17T03:51:52.173622+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:07.429263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730917+0000","last_change":"2024-09-17T03:51:51.181527+0000","last_active":"2024-09-17T03:51:53.730917+0000","last_peered":"2024-09-17T03:51:53.730917+0000","last_clean":"2024-09-17T03:51:53.730917+0000","last_became_active":"2024-09-17T03:51:51.181226+0000","last_became_peered":"2024-09-17T03:51:51.181226+0000","last_unstale":"2024-09-17T03:51:53.730917+0000","last_undegraded":"2024-09-17T03:51:53.730917+0000","last_fullsized":"2024-09-17T03:51:53.730917+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:23.640091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174401+0000","last_change":"2024-09-17T03:51:51.190529+0000","last_active":"2024-09-17T03:51:52.174401+0000","last_peered":"2024-09-17T03:51:52.174401+0000","last_clean":"2024-09-17T03:51:52.174401+0000","last_became_active":"2024-09-17T03:51:51.190206+0000","last_became_peered":"2024-09-17T03:51:51.190206+0000","last_unstale":"2024-09-17T03:51:52.174401+0000","last_undegraded":"2024-09-17T03:51:52.174401+0000","last_fullsized":"2024-09-17T03:51:52.174401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:11.743252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174132+0000","last_change":"2024-09-17T03:51:51.184036+0000","last_active":"2024-09-17T03:51:52.174132+0000","last_peered":"2024-09-17T03:51:52.174132+0000","last_clean":"2024-09-17T03:51:52.174132+0000","last_became_active":"2024-09-17T03:51:51.183692+0000","last_became_peered":"2024-09-17T03:51:51.183692+0000","last_unstale":"2024-09-17T03:51:52.174132+0000","last_undegraded":"2024-09-17T03:51:52.174132+0000","last_fullsized":"2024-09-17T03:51:52.174132+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:06.480938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183649+0000","last_change":"2024-09-17T03:51:51.197374+0000","last_active":"2024-09-17T03:51:52.183649+0000","last_peered":"2024-09-17T03:51:52.183649+0000","last_clean":"2024-09-17T03:51:52.183649+0000","last_became_active":"2024-09-17T03:51:51.197083+0000","last_became_peered":"2024-09-17T03:51:51.197083+0000","last_unstale":"2024-09-17T03:51:52.183649+0000","last_undegraded":"2024-09-17T03:51:52.183649+0000","last_fullsized":"2024-09-17T03:51:52.183649+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:06.099902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205298+0000","last_change":"2024-09-17T03:51:51.183634+0000","last_active":"2024-09-17T03:51:52.205298+0000","last_peered":"2024-09-17T03:51:52.205298+0000","last_clean":"2024-09-17T03:51:52.205298+0000","last_became_active":"2024-09-17T03:51:51.182965+0000","last_became_peered":"2024-09-17T03:51:51.182965+0000","last_unstale":"2024-09-17T03:51:52.205298+0000","last_undegraded":"2024-09-17T03:51:52.205298+0000","last_fullsized":"2024-09-17T03:51:52.205298+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:35.830782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173124+0000","last_change":"2024-09-17T03:51:51.184044+0000","last_active":"2024-09-17T03:51:52.173124+0000","last_peered":"2024-09-17T03:51:52.173124+0000","last_clean":"2024-09-17T03:51:52.173124+0000","last_became_active":"2024-09-17T03:51:51.181280+0000","last_became_peered":"2024-09-17T03:51:51.181280+0000","last_unstale":"2024-09-17T03:51:52.173124+0000","last_undegraded":"2024-09-17T03:51:52.173124+0000","last_fullsized":"2024-09-17T03:51:52.173124+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:49.860271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205754+0000","last_change":"2024-09-17T03:51:51.178569+0000","last_active":"2024-09-17T03:51:52.205754+0000","last_peered":"2024-09-17T03:51:52.205754+0000","last_clean":"2024-09-17T03:51:52.205754+0000","last_became_active":"2024-09-17T03:51:51.178236+0000","last_became_peered":"2024-09-17T03:51:51.178236+0000","last_unstale":"2024-09-17T03:51:52.205754+0000","last_undegraded":"2024-09-17T03:51:52.205754+0000","last_fullsized":"2024-09-17T03:51:52.205754+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:53:42.860411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610517+0000","last_change":"2024-09-17T03:51:51.166633+0000","last_active":"2024-09-17T03:51:52.610517+0000","last_peered":"2024-09-17T03:51:52.610517+0000","last_clean":"2024-09-17T03:51:52.610517+0000","last_became_active":"2024-09-17T03:51:51.166390+0000","last_became_peered":"2024-09-17T03:51:51.166390+0000","last_unstale":"2024-09-17T03:51:52.610517+0000","last_undegraded":"2024-09-17T03:51:52.610517+0000","last_fullsized":"2024-09-17T03:51:52.610517+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:52.752910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610551+0000","last_change":"2024-09-17T03:51:51.176819+0000","last_active":"2024-09-17T03:51:52.610551+0000","last_peered":"2024-09-17T03:51:52.610551+0000","last_clean":"2024-09-17T03:51:52.610551+0000","last_became_active":"2024-09-17T03:51:51.175065+0000","last_became_peered":"2024-09-17T03:51:51.175065+0000","last_unstale":"2024-09-17T03:51:52.610551+0000","last_undegraded":"2024-09-17T03:51:52.610551+0000","last_fullsized":"2024-09-17T03:51:52.610551+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:37.036957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174710+0000","last_change":"2024-09-17T03:51:51.180175+0000","last_active":"2024-09-17T03:51:52.174710+0000","last_peered":"2024-09-17T03:51:52.174710+0000","last_clean":"2024-09-17T03:51:52.174710+0000","last_became_active":"2024-09-17T03:51:51.179659+0000","last_became_peered":"2024-09-17T03:51:51.179659+0000","last_unstale":"2024-09-17T03:51:52.174710+0000","last_undegraded":"2024-09-17T03:51:52.174710+0000","last_fullsized":"2024-09-17T03:51:52.174710+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:41.002793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.163002+0000","last_change":"2024-09-17T03:51:51.180210+0000","last_active":"2024-09-17T03:51:53.163002+0000","last_peered":"2024-09-17T03:51:53.163002+0000","last_clean":"2024-09-17T03:51:53.163002+0000","last_became_active":"2024-09-17T03:51:51.179663+0000","last_became_peered":"2024-09-17T03:51:51.179663+0000","last_unstale":"2024-09-17T03:51:53.163002+0000","last_undegraded":"2024-09-17T03:51:53.163002+0000","last_fullsized":"2024-09-17T03:51:53.163002+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:58.012527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205358+0000","last_change":"2024-09-17T03:51:51.186363+0000","last_active":"2024-09-17T03:51:52.205358+0000","last_peered":"2024-09-17T03:51:52.205358+0000","last_clean":"2024-09-17T03:51:52.205358+0000","last_became_active":"2024-09-17T03:51:51.185999+0000","last_became_peered":"2024-09-17T03:51:51.185999+0000","last_unstale":"2024-09-17T03:51:52.205358+0000","last_undegraded":"2024-09-17T03:51:52.205358+0000","last_fullsized":"2024-09-17T03:51:52.205358+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:20.736248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205458+0000","last_change":"2024-09-17T03:51:51.183952+0000","last_active":"2024-09-17T03:51:52.205458+0000","last_peered":"2024-09-17T03:51:52.205458+0000","last_clean":"2024-09-17T03:51:52.205458+0000","last_became_active":"2024-09-17T03:51:51.183349+0000","last_became_peered":"2024-09-17T03:51:51.183349+0000","last_unstale":"2024-09-17T03:51:52.205458+0000","last_undegraded":"2024-09-17T03:51:52.205458+0000","last_fullsized":"2024-09-17T03:51:52.205458+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:06.013944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184056+0000","last_change":"2024-09-17T03:51:51.168140+0000","last_active":"2024-09-17T03:51:52.184056+0000","last_peered":"2024-09-17T03:51:52.184056+0000","last_clean":"2024-09-17T03:51:52.184056+0000","last_became_active":"2024-09-17T03:51:51.167864+0000","last_became_peered":"2024-09-17T03:51:51.167864+0000","last_unstale":"2024-09-17T03:51:52.184056+0000","last_undegraded":"2024-09-17T03:51:52.184056+0000","last_fullsized":"2024-09-17T03:51:52.184056+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:59.424732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174702+0000","last_change":"2024-09-17T03:51:51.188186+0000","last_active":"2024-09-17T03:51:52.174702+0000","last_peered":"2024-09-17T03:51:52.174702+0000","last_clean":"2024-09-17T03:51:52.174702+0000","last_became_active":"2024-09-17T03:51:51.187915+0000","last_became_peered":"2024-09-17T03:51:51.187915+0000","last_unstale":"2024-09-17T03:51:52.174702+0000","last_undegraded":"2024-09-17T03:51:52.174702+0000","last_fullsized":"2024-09-17T03:51:52.174702+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:12:14.938776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731395+0000","last_change":"2024-09-17T03:51:51.187987+0000","last_active":"2024-09-17T03:51:53.731395+0000","last_peered":"2024-09-17T03:51:53.731395+0000","last_clean":"2024-09-17T03:51:53.731395+0000","last_became_active":"2024-09-17T03:51:51.187662+0000","last_became_peered":"2024-09-17T03:51:51.187662+0000","last_unstale":"2024-09-17T03:51:53.731395+0000","last_undegraded":"2024-09-17T03:51:53.731395+0000","last_fullsized":"2024-09-17T03:51:53.731395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:11.849047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.175411+0000","last_change":"2024-09-17T03:51:51.200103+0000","last_active":"2024-09-17T03:51:52.175411+0000","last_peered":"2024-09-17T03:51:52.175411+0000","last_clean":"2024-09-17T03:51:52.175411+0000","last_became_active":"2024-09-17T03:51:51.199081+0000","last_became_peered":"2024-09-17T03:51:51.199081+0000","last_unstale":"2024-09-17T03:51:52.175411+0000","last_undegraded":"2024-09-17T03:51:52.175411+0000","last_fullsized":"2024-09-17T03:51:52.175411+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:23.689063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174988+0000","last_change":"2024-09-17T03:51:51.187087+0000","last_active":"2024-09-17T03:51:52.174988+0000","last_peered":"2024-09-17T03:51:52.174988+0000","last_clean":"2024-09-17T03:51:52.174988+0000","last_became_active":"2024-09-17T03:51:51.183188+0000","last_became_peered":"2024-09-17T03:51:51.183188+0000","last_unstale":"2024-09-17T03:51:52.174988+0000","last_undegraded":"2024-09-17T03:51:52.174988+0000","last_fullsized":"2024-09-17T03:51:52.174988+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:45:27.613702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730730+0000","last_change":"2024-09-17T03:51:51.189420+0000","last_active":"2024-09-17T03:51:53.730730+0000","last_peered":"2024-09-17T03:51:53.730730+0000","last_clean":"2024-09-17T03:51:53.730730+0000","last_became_active":"2024-09-17T03:51:51.187800+0000","last_became_peered":"2024-09-17T03:51:51.187800+0000","last_unstale":"2024-09-17T03:51:53.730730+0000","last_undegraded":"2024-09-17T03:51:53.730730+0000","last_fullsized":"2024-09-17T03:51:53.730730+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:54.040014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174059+0000","last_change":"2024-09-17T03:51:51.193515+0000","last_active":"2024-09-17T03:51:52.174059+0000","last_peered":"2024-09-17T03:51:52.174059+0000","last_clean":"2024-09-17T03:51:52.174059+0000","last_became_active":"2024-09-17T03:51:51.193190+0000","last_became_peered":"2024-09-17T03:51:51.193190+0000","last_unstale":"2024-09-17T03:51:52.174059+0000","last_undegraded":"2024-09-17T03:51:52.174059+0000","last_fullsized":"2024-09-17T03:51:52.174059+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:02.818568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730847+0000","last_change":"2024-09-17T03:51:51.194861+0000","last_active":"2024-09-17T03:51:53.730847+0000","last_peered":"2024-09-17T03:51:53.730847+0000","last_clean":"2024-09-17T03:51:53.730847+0000","last_became_active":"2024-09-17T03:51:51.194543+0000","last_became_peered":"2024-09-17T03:51:51.194543+0000","last_unstale":"2024-09-17T03:51:53.730847+0000","last_undegraded":"2024-09-17T03:51:53.730847+0000","last_fullsized":"2024-09-17T03:51:53.730847+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:17.130737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162862+0000","last_change":"2024-09-17T03:51:51.174878+0000","last_active":"2024-09-17T03:51:53.162862+0000","last_peered":"2024-09-17T03:51:53.162862+0000","last_clean":"2024-09-17T03:51:53.162862+0000","last_became_active":"2024-09-17T03:51:51.172940+0000","last_became_peered":"2024-09-17T03:51:51.172940+0000","last_unstale":"2024-09-17T03:51:53.162862+0000","last_undegraded":"2024-09-17T03:51:53.162862+0000","last_fullsized":"2024-09-17T03:51:53.162862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:00.808514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174370+0000","last_change":"2024-09-17T03:51:51.187429+0000","last_active":"2024-09-17T03:51:52.174370+0000","last_peered":"2024-09-17T03:51:52.174370+0000","last_clean":"2024-09-17T03:51:52.174370+0000","last_became_active":"2024-09-17T03:51:51.187148+0000","last_became_peered":"2024-09-17T03:51:51.187148+0000","last_unstale":"2024-09-17T03:51:52.174370+0000","last_undegraded":"2024-09-17T03:51:52.174370+0000","last_fullsized":"2024-09-17T03:51:52.174370+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:23.692411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174492+0000","last_change":"2024-09-17T03:51:51.179360+0000","last_active":"2024-09-17T03:51:52.174492+0000","last_peered":"2024-09-17T03:51:52.174492+0000","last_clean":"2024-09-17T03:51:52.174492+0000","last_became_active":"2024-09-17T03:51:51.179027+0000","last_became_peered":"2024-09-17T03:51:51.179027+0000","last_unstale":"2024-09-17T03:51:52.174492+0000","last_undegraded":"2024-09-17T03:51:52.174492+0000","last_fullsized":"2024-09-17T03:51:52.174492+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:31:54.000018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183870+0000","last_change":"2024-09-17T03:51:51.177850+0000","last_active":"2024-09-17T03:51:52.183870+0000","last_peered":"2024-09-17T03:51:52.183870+0000","last_clean":"2024-09-17T03:51:52.183870+0000","last_became_active":"2024-09-17T03:51:51.175315+0000","last_became_peered":"2024-09-17T03:51:51.175315+0000","last_unstale":"2024-09-17T03:51:52.183870+0000","last_undegraded":"2024-09-17T03:51:52.183870+0000","last_fullsized":"2024-09-17T03:51:52.183870+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:08.948416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173591+0000","last_change":"2024-09-17T03:51:51.190479+0000","last_active":"2024-09-17T03:51:52.173591+0000","last_peered":"2024-09-17T03:51:52.173591+0000","last_clean":"2024-09-17T03:51:52.173591+0000","last_became_active":"2024-09-17T03:51:51.190198+0000","last_became_peered":"2024-09-17T03:51:51.190198+0000","last_unstale":"2024-09-17T03:51:52.173591+0000","last_undegraded":"2024-09-17T03:51:52.173591+0000","last_fullsized":"2024-09-17T03:51:52.173591+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:03.246486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174265+0000","last_change":"2024-09-17T03:51:51.185752+0000","last_active":"2024-09-17T03:51:52.174265+0000","last_peered":"2024-09-17T03:51:52.174265+0000","last_clean":"2024-09-17T03:51:52.174265+0000","last_became_active":"2024-09-17T03:51:51.185434+0000","last_became_peered":"2024-09-17T03:51:51.185434+0000","last_unstale":"2024-09-17T03:51:52.174265+0000","last_undegraded":"2024-09-17T03:51:52.174265+0000","last_fullsized":"2024-09-17T03:51:52.174265+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:13.195505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730746+0000","last_change":"2024-09-17T03:51:51.189838+0000","last_active":"2024-09-17T03:51:53.730746+0000","last_peered":"2024-09-17T03:51:53.730746+0000","last_clean":"2024-09-17T03:51:53.730746+0000","last_became_active":"2024-09-17T03:51:51.188891+0000","last_became_peered":"2024-09-17T03:51:51.188891+0000","last_unstale":"2024-09-17T03:51:53.730746+0000","last_undegraded":"2024-09-17T03:51:53.730746+0000","last_fullsized":"2024-09-17T03:51:53.730746+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:40.902497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173346+0000","last_change":"2024-09-17T03:51:51.184999+0000","last_active":"2024-09-17T03:51:52.173346+0000","last_peered":"2024-09-17T03:51:52.173346+0000","last_clean":"2024-09-17T03:51:52.173346+0000","last_became_active":"2024-09-17T03:51:51.184748+0000","last_became_peered":"2024-09-17T03:51:51.184748+0000","last_unstale":"2024-09-17T03:51:52.173346+0000","last_undegraded":"2024-09-17T03:51:52.173346+0000","last_fullsized":"2024-09-17T03:51:52.173346+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:37.468008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183516+0000","last_change":"2024-09-17T03:51:51.189753+0000","last_active":"2024-09-17T03:51:52.183516+0000","last_peered":"2024-09-17T03:51:52.183516+0000","last_clean":"2024-09-17T03:51:52.183516+0000","last_became_active":"2024-09-17T03:51:51.189422+0000","last_became_peered":"2024-09-17T03:51:51.189422+0000","last_unstale":"2024-09-17T03:51:52.183516+0000","last_undegraded":"2024-09-17T03:51:52.183516+0000","last_fullsized":"2024-09-17T03:51:52.183516+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:27.174556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184092+0000","last_change":"2024-09-17T03:51:51.180539+0000","last_active":"2024-09-17T03:51:52.184092+0000","last_peered":"2024-09-17T03:51:52.184092+0000","last_clean":"2024-09-17T03:51:52.184092+0000","last_became_active":"2024-09-17T03:51:51.179021+0000","last_became_peered":"2024-09-17T03:51:51.179021+0000","last_unstale":"2024-09-17T03:51:52.184092+0000","last_undegraded":"2024-09-17T03:51:52.184092+0000","last_fullsized":"2024-09-17T03:51:52.184092+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:45.147765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"55'4","reported_seq":19,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183961+0000","last_change":"2024-09-17T03:51:50.168035+0000","last_active":"2024-09-17T03:51:52.183961+0000","last_peered":"2024-09-17T03:51:52.183961+0000","last_clean":"2024-09-17T03:51:52.183961+0000","last_became_active":"2024-09-17T03:51:50.167704+0000","last_became_peered":"2024-09-17T03:51:50.167704+0000","last_unstale":"2024-09-17T03:51:52.183961+0000","last_undegraded":"2024-09-17T03:51:52.183961+0000","last_fullsized":"2024-09-17T03:51:52.183961+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:53.499840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610525+0000","last_change":"2024-09-17T03:51:51.176980+0000","last_active":"2024-09-17T03:51:52.610525+0000","last_peered":"2024-09-17T03:51:52.610525+0000","last_clean":"2024-09-17T03:51:52.610525+0000","last_became_active":"2024-09-17T03:51:51.176667+0000","last_became_peered":"2024-09-17T03:51:51.176667+0000","last_unstale":"2024-09-17T03:51:52.610525+0000","last_undegraded":"2024-09-17T03:51:52.610525+0000","last_fullsized":"2024-09-17T03:51:52.610525+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:24.159820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173682+0000","last_change":"2024-09-17T03:51:50.162857+0000","last_active":"2024-09-17T03:51:52.173682+0000","last_peered":"2024-09-17T03:51:52.173682+0000","last_clean":"2024-09-17T03:51:52.173682+0000","last_became_active":"2024-09-17T03:51:50.162573+0000","last_became_peered":"2024-09-17T03:51:50.162573+0000","last_unstale":"2024-09-17T03:51:52.173682+0000","last_undegraded":"2024-09-17T03:51:52.173682+0000","last_fullsized":"2024-09-17T03:51:52.173682+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:22.633136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184039+0000","last_change":"2024-09-17T03:51:51.170143+0000","last_active":"2024-09-17T03:51:52.184039+0000","last_peered":"2024-09-17T03:51:52.184039+0000","last_clean":"2024-09-17T03:51:52.184039+0000","last_became_active":"2024-09-17T03:51:51.169595+0000","last_became_peered":"2024-09-17T03:51:51.169595+0000","last_unstale":"2024-09-17T03:51:52.184039+0000","last_undegraded":"2024-09-17T03:51:52.184039+0000","last_fullsized":"2024-09-17T03:51:52.184039+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:20.419660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184162+0000","last_change":"2024-09-17T03:51:50.165145+0000","last_active":"2024-09-17T03:51:52.184162+0000","last_peered":"2024-09-17T03:51:52.184162+0000","last_clean":"2024-09-17T03:51:52.184162+0000","last_became_active":"2024-09-17T03:51:50.164900+0000","last_became_peered":"2024-09-17T03:51:50.164900+0000","last_unstale":"2024-09-17T03:51:52.184162+0000","last_undegraded":"2024-09-17T03:51:52.184162+0000","last_fullsized":"2024-09-17T03:51:52.184162+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:06.938674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183864+0000","last_change":"2024-09-17T03:51:51.177537+0000","last_active":"2024-09-17T03:51:52.183864+0000","last_peered":"2024-09-17T03:51:52.183864+0000","last_clean":"2024-09-17T03:51:52.183864+0000","last_became_active":"2024-09-17T03:51:51.174936+0000","last_became_peered":"2024-09-17T03:51:51.174936+0000","last_unstale":"2024-09-17T03:51:52.183864+0000","last_undegraded":"2024-09-17T03:51:52.183864+0000","last_fullsized":"2024-09-17T03:51:52.183864+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:04.586292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174779+0000","last_change":"2024-09-17T03:51:50.167732+0000","last_active":"2024-09-17T03:51:52.174779+0000","last_peered":"2024-09-17T03:51:52.174779+0000","last_clean":"2024-09-17T03:51:52.174779+0000","last_became_active":"2024-09-17T03:51:50.166704+0000","last_became_peered":"2024-09-17T03:51:50.166704+0000","last_unstale":"2024-09-17T03:51:52.174779+0000","last_undegraded":"2024-09-17T03:51:52.174779+0000","last_fullsized":"2024-09-17T03:51:52.174779+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:22.049192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174306+0000","last_change":"2024-09-17T03:51:51.195178+0000","last_active":"2024-09-17T03:51:52.174306+0000","last_peered":"2024-09-17T03:51:52.174306+0000","last_clean":"2024-09-17T03:51:52.174306+0000","last_became_active":"2024-09-17T03:51:51.193567+0000","last_became_peered":"2024-09-17T03:51:51.193567+0000","last_unstale":"2024-09-17T03:51:52.174306+0000","last_undegraded":"2024-09-17T03:51:52.174306+0000","last_fullsized":"2024-09-17T03:51:52.174306+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:25.299820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205310+0000","last_change":"2024-09-17T03:51:50.153047+0000","last_active":"2024-09-17T03:51:52.205310+0000","last_peered":"2024-09-17T03:51:52.205310+0000","last_clean":"2024-09-17T03:51:52.205310+0000","last_became_active":"2024-09-17T03:51:50.152719+0000","last_became_peered":"2024-09-17T03:51:50.152719+0000","last_unstale":"2024-09-17T03:51:52.205310+0000","last_undegraded":"2024-09-17T03:51:52.205310+0000","last_fullsized":"2024-09-17T03:51:52.205310+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:21.430041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174037+0000","last_change":"2024-09-17T03:51:51.185697+0000","last_active":"2024-09-17T03:51:52.174037+0000","last_peered":"2024-09-17T03:51:52.174037+0000","last_clean":"2024-09-17T03:51:52.174037+0000","last_became_active":"2024-09-17T03:51:51.185429+0000","last_became_peered":"2024-09-17T03:51:51.185429+0000","last_unstale":"2024-09-17T03:51:52.174037+0000","last_undegraded":"2024-09-17T03:51:52.174037+0000","last_fullsized":"2024-09-17T03:51:52.174037+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:46.688579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183576+0000","last_change":"2024-09-17T03:51:50.166105+0000","last_active":"2024-09-17T03:51:52.183576+0000","last_peered":"2024-09-17T03:51:52.183576+0000","last_clean":"2024-09-17T03:51:52.183576+0000","last_became_active":"2024-09-17T03:51:50.165818+0000","last_became_peered":"2024-09-17T03:51:50.165818+0000","last_unstale":"2024-09-17T03:51:52.183576+0000","last_undegraded":"2024-09-17T03:51:52.183576+0000","last_fullsized":"2024-09-17T03:51:52.183576+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:08.880243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162846+0000","last_change":"2024-09-17T03:51:51.177152+0000","last_active":"2024-09-17T03:51:53.162846+0000","last_peered":"2024-09-17T03:51:53.162846+0000","last_clean":"2024-09-17T03:51:53.162846+0000","last_became_active":"2024-09-17T03:51:51.176806+0000","last_became_peered":"2024-09-17T03:51:51.176806+0000","last_unstale":"2024-09-17T03:51:53.162846+0000","last_undegraded":"2024-09-17T03:51:53.162846+0000","last_fullsized":"2024-09-17T03:51:53.162846+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:09.307104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173657+0000","last_change":"2024-09-17T03:51:50.169610+0000","last_active":"2024-09-17T03:51:52.173657+0000","last_peered":"2024-09-17T03:51:52.173657+0000","last_clean":"2024-09-17T03:51:52.173657+0000","last_became_active":"2024-09-17T03:51:50.169318+0000","last_became_peered":"2024-09-17T03:51:50.169318+0000","last_unstale":"2024-09-17T03:51:52.173657+0000","last_undegraded":"2024-09-17T03:51:52.173657+0000","last_fullsized":"2024-09-17T03:51:52.173657+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:43.928869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173528+0000","last_change":"2024-09-17T03:51:51.192692+0000","last_active":"2024-09-17T03:51:52.173528+0000","last_peered":"2024-09-17T03:51:52.173528+0000","last_clean":"2024-09-17T03:51:52.173528+0000","last_became_active":"2024-09-17T03:51:51.192404+0000","last_became_peered":"2024-09-17T03:51:51.192404+0000","last_unstale":"2024-09-17T03:51:52.173528+0000","last_undegraded":"2024-09-17T03:51:52.173528+0000","last_fullsized":"2024-09-17T03:51:52.173528+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:25.262002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205190+0000","last_change":"2024-09-17T03:51:50.160392+0000","last_active":"2024-09-17T03:51:52.205190+0000","last_peered":"2024-09-17T03:51:52.205190+0000","last_clean":"2024-09-17T03:51:52.205190+0000","last_became_active":"2024-09-17T03:51:50.159973+0000","last_became_peered":"2024-09-17T03:51:50.159973+0000","last_unstale":"2024-09-17T03:51:52.205190+0000","last_undegraded":"2024-09-17T03:51:52.205190+0000","last_fullsized":"2024-09-17T03:51:52.205190+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:42.221660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205766+0000","last_change":"2024-09-17T03:51:51.184296+0000","last_active":"2024-09-17T03:51:52.205766+0000","last_peered":"2024-09-17T03:51:52.205766+0000","last_clean":"2024-09-17T03:51:52.205766+0000","last_became_active":"2024-09-17T03:51:51.183943+0000","last_became_peered":"2024-09-17T03:51:51.183943+0000","last_unstale":"2024-09-17T03:51:52.205766+0000","last_undegraded":"2024-09-17T03:51:52.205766+0000","last_fullsized":"2024-09-17T03:51:52.205766+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:55.346872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731380+0000","last_change":"2024-09-17T03:51:50.167430+0000","last_active":"2024-09-17T03:51:53.731380+0000","last_peered":"2024-09-17T03:51:53.731380+0000","last_clean":"2024-09-17T03:51:53.731380+0000","last_became_active":"2024-09-17T03:51:50.167110+0000","last_became_peered":"2024-09-17T03:51:50.167110+0000","last_unstale":"2024-09-17T03:51:53.731380+0000","last_undegraded":"2024-09-17T03:51:53.731380+0000","last_fullsized":"2024-09-17T03:51:53.731380+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:58.416043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731518+0000","last_change":"2024-09-17T03:51:51.192660+0000","last_active":"2024-09-17T03:51:53.731518+0000","last_peered":"2024-09-17T03:51:53.731518+0000","last_clean":"2024-09-17T03:51:53.731518+0000","last_became_active":"2024-09-17T03:51:51.192294+0000","last_became_peered":"2024-09-17T03:51:51.192294+0000","last_unstale":"2024-09-17T03:51:53.731518+0000","last_undegraded":"2024-09-17T03:51:53.731518+0000","last_fullsized":"2024-09-17T03:51:53.731518+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:06.260210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173787+0000","last_change":"2024-09-17T03:51:50.164231+0000","last_active":"2024-09-17T03:51:52.173787+0000","last_peered":"2024-09-17T03:51:52.173787+0000","last_clean":"2024-09-17T03:51:52.173787+0000","last_became_active":"2024-09-17T03:51:50.163955+0000","last_became_peered":"2024-09-17T03:51:50.163955+0000","last_unstale":"2024-09-17T03:51:52.173787+0000","last_undegraded":"2024-09-17T03:51:52.173787+0000","last_fullsized":"2024-09-17T03:51:52.173787+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:33:48.844463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610329+0000","last_change":"2024-09-17T03:51:51.183261+0000","last_active":"2024-09-17T03:51:52.610329+0000","last_peered":"2024-09-17T03:51:52.610329+0000","last_clean":"2024-09-17T03:51:52.610329+0000","last_became_active":"2024-09-17T03:51:51.182895+0000","last_became_peered":"2024-09-17T03:51:51.182895+0000","last_unstale":"2024-09-17T03:51:52.610329+0000","last_undegraded":"2024-09-17T03:51:52.610329+0000","last_fullsized":"2024-09-17T03:51:52.610329+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:00.993310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"55'3","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731130+0000","last_change":"2024-09-17T03:51:50.162510+0000","last_active":"2024-09-17T03:51:53.731130+0000","last_peered":"2024-09-17T03:51:53.731130+0000","last_clean":"2024-09-17T03:51:53.731130+0000","last_became_active":"2024-09-17T03:51:50.162216+0000","last_became_peered":"2024-09-17T03:51:50.162216+0000","last_unstale":"2024-09-17T03:51:53.731130+0000","last_undegraded":"2024-09-17T03:51:53.731130+0000","last_fullsized":"2024-09-17T03:51:53.731130+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:22:36.660509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731050+0000","last_change":"2024-09-17T03:51:51.185826+0000","last_active":"2024-09-17T03:51:53.731050+0000","last_peered":"2024-09-17T03:51:53.731050+0000","last_clean":"2024-09-17T03:51:53.731050+0000","last_became_active":"2024-09-17T03:51:51.182569+0000","last_became_peered":"2024-09-17T03:51:51.182569+0000","last_unstale":"2024-09-17T03:51:53.731050+0000","last_undegraded":"2024-09-17T03:51:53.731050+0000","last_fullsized":"2024-09-17T03:51:53.731050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:33:49.851418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"55'6","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205622+0000","last_change":"2024-09-17T03:51:50.167752+0000","last_active":"2024-09-17T03:51:52.205622+0000","last_peered":"2024-09-17T03:51:52.205622+0000","last_clean":"2024-09-17T03:51:52.205622+0000","last_became_active":"2024-09-17T03:51:50.166168+0000","last_became_peered":"2024-09-17T03:51:50.166168+0000","last_unstale":"2024-09-17T03:51:52.205622+0000","last_undegraded":"2024-09-17T03:51:52.205622+0000","last_fullsized":"2024-09-17T03:51:52.205622+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:59.986355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173826+0000","last_change":"2024-09-17T03:51:51.186944+0000","last_active":"2024-09-17T03:51:52.173826+0000","last_peered":"2024-09-17T03:51:52.173826+0000","last_clean":"2024-09-17T03:51:52.173826+0000","last_became_active":"2024-09-17T03:51:51.184771+0000","last_became_peered":"2024-09-17T03:51:51.184771+0000","last_unstale":"2024-09-17T03:51:52.173826+0000","last_undegraded":"2024-09-17T03:51:52.173826+0000","last_fullsized":"2024-09-17T03:51:52.173826+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:43.126626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173443+0000","last_change":"2024-09-17T03:51:50.163723+0000","last_active":"2024-09-17T03:51:52.173443+0000","last_peered":"2024-09-17T03:51:52.173443+0000","last_clean":"2024-09-17T03:51:52.173443+0000","last_became_active":"2024-09-17T03:51:50.163324+0000","last_became_peered":"2024-09-17T03:51:50.163324+0000","last_unstale":"2024-09-17T03:51:52.173443+0000","last_undegraded":"2024-09-17T03:51:52.173443+0000","last_fullsized":"2024-09-17T03:51:52.173443+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:16.123782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730879+0000","last_change":"2024-09-17T03:51:51.189553+0000","last_active":"2024-09-17T03:51:53.730879+0000","last_peered":"2024-09-17T03:51:53.730879+0000","last_clean":"2024-09-17T03:51:53.730879+0000","last_became_active":"2024-09-17T03:51:51.187557+0000","last_became_peered":"2024-09-17T03:51:51.187557+0000","last_unstale":"2024-09-17T03:51:53.730879+0000","last_undegraded":"2024-09-17T03:51:53.730879+0000","last_fullsized":"2024-09-17T03:51:53.730879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:26.699117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162654+0000","last_change":"2024-09-17T03:51:50.164735+0000","last_active":"2024-09-17T03:51:53.162654+0000","last_peered":"2024-09-17T03:51:53.162654+0000","last_clean":"2024-09-17T03:51:53.162654+0000","last_became_active":"2024-09-17T03:51:50.164378+0000","last_became_peered":"2024-09-17T03:51:50.164378+0000","last_unstale":"2024-09-17T03:51:53.162654+0000","last_undegraded":"2024-09-17T03:51:53.162654+0000","last_fullsized":"2024-09-17T03:51:53.162654+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:35.436131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205301+0000","last_change":"2024-09-17T03:51:51.186075+0000","last_active":"2024-09-17T03:51:52.205301+0000","last_peered":"2024-09-17T03:51:52.205301+0000","last_clean":"2024-09-17T03:51:52.205301+0000","last_became_active":"2024-09-17T03:51:51.185742+0000","last_became_peered":"2024-09-17T03:51:51.185742+0000","last_unstale":"2024-09-17T03:51:52.205301+0000","last_undegraded":"2024-09-17T03:51:52.205301+0000","last_fullsized":"2024-09-17T03:51:52.205301+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:34.551761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173663+0000","last_change":"2024-09-17T03:51:50.166686+0000","last_active":"2024-09-17T03:51:52.173663+0000","last_peered":"2024-09-17T03:51:52.173663+0000","last_clean":"2024-09-17T03:51:52.173663+0000","last_became_active":"2024-09-17T03:51:50.165924+0000","last_became_peered":"2024-09-17T03:51:50.165924+0000","last_unstale":"2024-09-17T03:51:52.173663+0000","last_undegraded":"2024-09-17T03:51:52.173663+0000","last_fullsized":"2024-09-17T03:51:52.173663+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:23.152865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730823+0000","last_change":"2024-09-17T03:51:51.199074+0000","last_active":"2024-09-17T03:51:53.730823+0000","last_peered":"2024-09-17T03:51:53.730823+0000","last_clean":"2024-09-17T03:51:53.730823+0000","last_became_active":"2024-09-17T03:51:51.198758+0000","last_became_peered":"2024-09-17T03:51:51.198758+0000","last_unstale":"2024-09-17T03:51:53.730823+0000","last_undegraded":"2024-09-17T03:51:53.730823+0000","last_fullsized":"2024-09-17T03:51:53.730823+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:04.175415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"55'8","reported_seq":23,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730361+0000","last_change":"2024-09-17T03:51:50.173075+0000","last_active":"2024-09-17T03:51:53.730361+0000","last_peered":"2024-09-17T03:51:53.730361+0000","last_clean":"2024-09-17T03:51:53.730361+0000","last_became_active":"2024-09-17T03:51:50.172690+0000","last_became_peered":"2024-09-17T03:51:50.172690+0000","last_unstale":"2024-09-17T03:51:53.730361+0000","last_undegraded":"2024-09-17T03:51:53.730361+0000","last_fullsized":"2024-09-17T03:51:53.730361+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:41.566073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731446+0000","last_change":"2024-09-17T03:51:51.181582+0000","last_active":"2024-09-17T03:51:53.731446+0000","last_peered":"2024-09-17T03:51:53.731446+0000","last_clean":"2024-09-17T03:51:53.731446+0000","last_became_active":"2024-09-17T03:51:51.181239+0000","last_became_peered":"2024-09-17T03:51:51.181239+0000","last_unstale":"2024-09-17T03:51:53.731446+0000","last_undegraded":"2024-09-17T03:51:53.731446+0000","last_fullsized":"2024-09-17T03:51:53.731446+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:31.955299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174773+0000","last_change":"2024-09-17T03:51:50.167328+0000","last_active":"2024-09-17T03:51:52.174773+0000","last_peered":"2024-09-17T03:51:52.174773+0000","last_clean":"2024-09-17T03:51:52.174773+0000","last_became_active":"2024-09-17T03:51:50.167022+0000","last_became_peered":"2024-09-17T03:51:50.167022+0000","last_unstale":"2024-09-17T03:51:52.174773+0000","last_undegraded":"2024-09-17T03:51:52.174773+0000","last_fullsized":"2024-09-17T03:51:52.174773+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:03.168460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174474+0000","last_change":"2024-09-17T03:51:51.166682+0000","last_active":"2024-09-17T03:51:52.174474+0000","last_peered":"2024-09-17T03:51:52.174474+0000","last_clean":"2024-09-17T03:51:52.174474+0000","last_became_active":"2024-09-17T03:51:51.166356+0000","last_became_peered":"2024-09-17T03:51:51.166356+0000","last_unstale":"2024-09-17T03:51:52.174474+0000","last_undegraded":"2024-09-17T03:51:52.174474+0000","last_fullsized":"2024-09-17T03:51:52.174474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:21.118305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184201+0000","last_change":"2024-09-17T03:51:50.156732+0000","last_active":"2024-09-17T03:51:52.184201+0000","last_peered":"2024-09-17T03:51:52.184201+0000","last_clean":"2024-09-17T03:51:52.184201+0000","last_became_active":"2024-09-17T03:51:50.156393+0000","last_became_peered":"2024-09-17T03:51:50.156393+0000","last_unstale":"2024-09-17T03:51:52.184201+0000","last_undegraded":"2024-09-17T03:51:52.184201+0000","last_fullsized":"2024-09-17T03:51:52.184201+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:00.850370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173838+0000","last_change":"2024-09-17T03:51:51.182538+0000","last_active":"2024-09-17T03:51:52.173838+0000","last_peered":"2024-09-17T03:51:52.173838+0000","last_clean":"2024-09-17T03:51:52.173838+0000","last_became_active":"2024-09-17T03:51:51.182255+0000","last_became_peered":"2024-09-17T03:51:51.182255+0000","last_unstale":"2024-09-17T03:51:52.173838+0000","last_undegraded":"2024-09-17T03:51:52.173838+0000","last_fullsized":"2024-09-17T03:51:52.173838+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:31.579943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"55'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174264+0000","last_change":"2024-09-17T03:51:50.169032+0000","last_active":"2024-09-17T03:51:52.174264+0000","last_peered":"2024-09-17T03:51:52.174264+0000","last_clean":"2024-09-17T03:51:52.174264+0000","last_became_active":"2024-09-17T03:51:50.168178+0000","last_became_peered":"2024-09-17T03:51:50.168178+0000","last_unstale":"2024-09-17T03:51:52.174264+0000","last_undegraded":"2024-09-17T03:51:52.174264+0000","last_fullsized":"2024-09-17T03:51:52.174264+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:37.469802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731094+0000","last_change":"2024-09-17T03:51:51.188487+0000","last_active":"2024-09-17T03:51:53.731094+0000","last_peered":"2024-09-17T03:51:53.731094+0000","last_clean":"2024-09-17T03:51:53.731094+0000","last_became_active":"2024-09-17T03:51:51.188169+0000","last_became_peered":"2024-09-17T03:51:51.188169+0000","last_unstale":"2024-09-17T03:51:53.731094+0000","last_undegraded":"2024-09-17T03:51:53.731094+0000","last_fullsized":"2024-09-17T03:51:53.731094+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:18.318333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184064+0000","last_change":"2024-09-17T03:51:50.161576+0000","last_active":"2024-09-17T03:51:52.184064+0000","last_peered":"2024-09-17T03:51:52.184064+0000","last_clean":"2024-09-17T03:51:52.184064+0000","last_became_active":"2024-09-17T03:51:50.161249+0000","last_became_peered":"2024-09-17T03:51:50.161249+0000","last_unstale":"2024-09-17T03:51:52.184064+0000","last_undegraded":"2024-09-17T03:51:52.184064+0000","last_fullsized":"2024-09-17T03:51:52.184064+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:47:39.842117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174145+0000","last_change":"2024-09-17T03:51:51.186537+0000","last_active":"2024-09-17T03:51:52.174145+0000","last_peered":"2024-09-17T03:51:52.174145+0000","last_clean":"2024-09-17T03:51:52.174145+0000","last_became_active":"2024-09-17T03:51:51.186087+0000","last_became_peered":"2024-09-17T03:51:51.186087+0000","last_unstale":"2024-09-17T03:51:52.174145+0000","last_undegraded":"2024-09-17T03:51:52.174145+0000","last_fullsized":"2024-09-17T03:51:52.174145+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:54:54.654394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174342+0000","last_change":"2024-09-17T03:51:50.158489+0000","last_active":"2024-09-17T03:51:52.174342+0000","last_peered":"2024-09-17T03:51:52.174342+0000","last_clean":"2024-09-17T03:51:52.174342+0000","last_became_active":"2024-09-17T03:51:50.157069+0000","last_became_peered":"2024-09-17T03:51:50.157069+0000","last_unstale":"2024-09-17T03:51:52.174342+0000","last_undegraded":"2024-09-17T03:51:52.174342+0000","last_fullsized":"2024-09-17T03:51:52.174342+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:25.692162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183880+0000","last_change":"2024-09-17T03:51:51.174656+0000","last_active":"2024-09-17T03:51:52.183880+0000","last_peered":"2024-09-17T03:51:52.183880+0000","last_clean":"2024-09-17T03:51:52.183880+0000","last_became_active":"2024-09-17T03:51:51.174341+0000","last_became_peered":"2024-09-17T03:51:51.174341+0000","last_unstale":"2024-09-17T03:51:52.183880+0000","last_undegraded":"2024-09-17T03:51:52.183880+0000","last_fullsized":"2024-09-17T03:51:52.183880+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:21.924725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610096+0000","last_change":"2024-09-17T03:51:50.161426+0000","last_active":"2024-09-17T03:51:52.610096+0000","last_peered":"2024-09-17T03:51:52.610096+0000","last_clean":"2024-09-17T03:51:52.610096+0000","last_became_active":"2024-09-17T03:51:50.159750+0000","last_became_peered":"2024-09-17T03:51:50.159750+0000","last_unstale":"2024-09-17T03:51:52.610096+0000","last_undegraded":"2024-09-17T03:51:52.610096+0000","last_fullsized":"2024-09-17T03:51:52.610096+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:02:28.181263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.183722+0000","last_change":"2024-09-17T03:51:51.179189+0000","last_active":"2024-09-17T03:51:52.183722+0000","last_peered":"2024-09-17T03:51:52.183722+0000","last_clean":"2024-09-17T03:51:52.183722+0000","last_became_active":"2024-09-17T03:51:51.178873+0000","last_became_peered":"2024-09-17T03:51:51.178873+0000","last_unstale":"2024-09-17T03:51:52.183722+0000","last_undegraded":"2024-09-17T03:51:52.183722+0000","last_fullsized":"2024-09-17T03:51:52.183722+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:44.713006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173638+0000","last_change":"2024-09-17T03:51:50.171471+0000","last_active":"2024-09-17T03:51:52.173638+0000","last_peered":"2024-09-17T03:51:52.173638+0000","last_clean":"2024-09-17T03:51:52.173638+0000","last_became_active":"2024-09-17T03:51:50.169065+0000","last_became_peered":"2024-09-17T03:51:50.169065+0000","last_unstale":"2024-09-17T03:51:52.173638+0000","last_undegraded":"2024-09-17T03:51:52.173638+0000","last_fullsized":"2024-09-17T03:51:52.173638+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:57.382485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174698+0000","last_change":"2024-09-17T03:51:51.182912+0000","last_active":"2024-09-17T03:51:52.174698+0000","last_peered":"2024-09-17T03:51:52.174698+0000","last_clean":"2024-09-17T03:51:52.174698+0000","last_became_active":"2024-09-17T03:51:51.182620+0000","last_became_peered":"2024-09-17T03:51:51.182620+0000","last_unstale":"2024-09-17T03:51:52.174698+0000","last_undegraded":"2024-09-17T03:51:52.174698+0000","last_fullsized":"2024-09-17T03:51:52.174698+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:28.435503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"55'2","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610332+0000","last_change":"2024-09-17T03:51:50.160376+0000","last_active":"2024-09-17T03:51:52.610332+0000","last_peered":"2024-09-17T03:51:52.610332+0000","last_clean":"2024-09-17T03:51:52.610332+0000","last_became_active":"2024-09-17T03:51:50.159950+0000","last_became_peered":"2024-09-17T03:51:50.159950+0000","last_unstale":"2024-09-17T03:51:52.610332+0000","last_undegraded":"2024-09-17T03:51:52.610332+0000","last_fullsized":"2024-09-17T03:51:52.610332+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:29.610642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174651+0000","last_change":"2024-09-17T03:51:51.178739+0000","last_active":"2024-09-17T03:51:52.174651+0000","last_peered":"2024-09-17T03:51:52.174651+0000","last_clean":"2024-09-17T03:51:52.174651+0000","last_became_active":"2024-09-17T03:51:51.177453+0000","last_became_peered":"2024-09-17T03:51:51.177453+0000","last_unstale":"2024-09-17T03:51:52.174651+0000","last_undegraded":"2024-09-17T03:51:52.174651+0000","last_fullsized":"2024-09-17T03:51:52.174651+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:15.833137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"56'1","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162007+0000","last_change":"2024-09-17T03:51:50.166779+0000","last_active":"2024-09-17T03:51:53.162007+0000","last_peered":"2024-09-17T03:51:53.162007+0000","last_clean":"2024-09-17T03:51:53.162007+0000","last_became_active":"2024-09-17T03:51:50.166433+0000","last_became_peered":"2024-09-17T03:51:50.166433+0000","last_unstale":"2024-09-17T03:51:53.162007+0000","last_undegraded":"2024-09-17T03:51:53.162007+0000","last_fullsized":"2024-09-17T03:51:53.162007+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:29:43.023544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610361+0000","last_change":"2024-09-17T03:51:51.180525+0000","last_active":"2024-09-17T03:51:52.610361+0000","last_peered":"2024-09-17T03:51:52.610361+0000","last_clean":"2024-09-17T03:51:52.610361+0000","last_became_active":"2024-09-17T03:51:51.174354+0000","last_became_peered":"2024-09-17T03:51:51.174354+0000","last_unstale":"2024-09-17T03:51:52.610361+0000","last_undegraded":"2024-09-17T03:51:52.610361+0000","last_fullsized":"2024-09-17T03:51:52.610361+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:15:58.389440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.163000+0000","last_change":"2024-09-17T03:51:50.168996+0000","last_active":"2024-09-17T03:51:53.163000+0000","last_peered":"2024-09-17T03:51:53.163000+0000","last_clean":"2024-09-17T03:51:53.163000+0000","last_became_active":"2024-09-17T03:51:50.167892+0000","last_became_peered":"2024-09-17T03:51:50.167892+0000","last_unstale":"2024-09-17T03:51:53.163000+0000","last_undegraded":"2024-09-17T03:51:53.163000+0000","last_fullsized":"2024-09-17T03:51:53.163000+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:12.026481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.184103+0000","last_change":"2024-09-17T03:51:51.177189+0000","last_active":"2024-09-17T03:51:52.184103+0000","last_peered":"2024-09-17T03:51:52.184103+0000","last_clean":"2024-09-17T03:51:52.184103+0000","last_became_active":"2024-09-17T03:51:51.174651+0000","last_became_peered":"2024-09-17T03:51:51.174651+0000","last_unstale":"2024-09-17T03:51:52.184103+0000","last_undegraded":"2024-09-17T03:51:52.184103+0000","last_fullsized":"2024-09-17T03:51:52.184103+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:31:54.841471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174857+0000","last_change":"2024-09-17T03:51:50.164737+0000","last_active":"2024-09-17T03:51:52.174857+0000","last_peered":"2024-09-17T03:51:52.174857+0000","last_clean":"2024-09-17T03:51:52.174857+0000","last_became_active":"2024-09-17T03:51:50.164417+0000","last_became_peered":"2024-09-17T03:51:50.164417+0000","last_unstale":"2024-09-17T03:51:52.174857+0000","last_undegraded":"2024-09-17T03:51:52.174857+0000","last_fullsized":"2024-09-17T03:51:52.174857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:36.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.174550+0000","last_change":"2024-09-17T03:51:51.170043+0000","last_active":"2024-09-17T03:51:52.174550+0000","last_peered":"2024-09-17T03:51:52.174550+0000","last_clean":"2024-09-17T03:51:52.174550+0000","last_became_active":"2024-09-17T03:51:51.169768+0000","last_became_peered":"2024-09-17T03:51:51.169768+0000","last_unstale":"2024-09-17T03:51:52.174550+0000","last_undegraded":"2024-09-17T03:51:52.174550+0000","last_fullsized":"2024-09-17T03:51:52.174550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:46.510444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610005+0000","last_change":"2024-09-17T03:51:50.164044+0000","last_active":"2024-09-17T03:51:52.610005+0000","last_peered":"2024-09-17T03:51:52.610005+0000","last_clean":"2024-09-17T03:51:52.610005+0000","last_became_active":"2024-09-17T03:51:50.163701+0000","last_became_peered":"2024-09-17T03:51:50.163701+0000","last_unstale":"2024-09-17T03:51:52.610005+0000","last_undegraded":"2024-09-17T03:51:52.610005+0000","last_fullsized":"2024-09-17T03:51:52.610005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:33.411551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.731065+0000","last_change":"2024-09-17T03:51:51.196389+0000","last_active":"2024-09-17T03:51:53.731065+0000","last_peered":"2024-09-17T03:51:53.731065+0000","last_clean":"2024-09-17T03:51:53.731065+0000","last_became_active":"2024-09-17T03:51:51.196010+0000","last_became_peered":"2024-09-17T03:51:51.196010+0000","last_unstale":"2024-09-17T03:51:53.731065+0000","last_undegraded":"2024-09-17T03:51:53.731065+0000","last_fullsized":"2024-09-17T03:51:53.731065+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:09.419164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.205200+0000","last_change":"2024-09-17T03:51:50.156739+0000","last_active":"2024-09-17T03:51:52.205200+0000","last_peered":"2024-09-17T03:51:52.205200+0000","last_clean":"2024-09-17T03:51:52.205200+0000","last_became_active":"2024-09-17T03:51:50.155742+0000","last_became_peered":"2024-09-17T03:51:50.155742+0000","last_unstale":"2024-09-17T03:51:52.205200+0000","last_undegraded":"2024-09-17T03:51:52.205200+0000","last_fullsized":"2024-09-17T03:51:52.205200+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:05.278517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610117+0000","last_change":"2024-09-17T03:51:51.165566+0000","last_active":"2024-09-17T03:51:52.610117+0000","last_peered":"2024-09-17T03:51:52.610117+0000","last_clean":"2024-09-17T03:51:52.610117+0000","last_became_active":"2024-09-17T03:51:51.165141+0000","last_became_peered":"2024-09-17T03:51:51.165141+0000","last_unstale":"2024-09-17T03:51:52.610117+0000","last_undegraded":"2024-09-17T03:51:52.610117+0000","last_fullsized":"2024-09-17T03:51:52.610117+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:22.436996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.610015+0000","last_change":"2024-09-17T03:51:50.168219+0000","last_active":"2024-09-17T03:51:52.610015+0000","last_peered":"2024-09-17T03:51:52.610015+0000","last_clean":"2024-09-17T03:51:52.610015+0000","last_became_active":"2024-09-17T03:51:50.167872+0000","last_became_peered":"2024-09-17T03:51:50.167872+0000","last_unstale":"2024-09-17T03:51:52.610015+0000","last_undegraded":"2024-09-17T03:51:52.610015+0000","last_fullsized":"2024-09-17T03:51:52.610015+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:35:50.289731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173794+0000","last_change":"2024-09-17T03:51:51.189223+0000","last_active":"2024-09-17T03:51:52.173794+0000","last_peered":"2024-09-17T03:51:52.173794+0000","last_clean":"2024-09-17T03:51:52.173794+0000","last_became_active":"2024-09-17T03:51:51.188908+0000","last_became_peered":"2024-09-17T03:51:51.188908+0000","last_unstale":"2024-09-17T03:51:52.173794+0000","last_undegraded":"2024-09-17T03:51:52.173794+0000","last_fullsized":"2024-09-17T03:51:52.173794+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:11.721027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173198+0000","last_change":"2024-09-17T03:51:50.172844+0000","last_active":"2024-09-17T03:51:52.173198+0000","last_peered":"2024-09-17T03:51:52.173198+0000","last_clean":"2024-09-17T03:51:52.173198+0000","last_became_active":"2024-09-17T03:51:50.172507+0000","last_became_peered":"2024-09-17T03:51:50.172507+0000","last_unstale":"2024-09-17T03:51:52.173198+0000","last_undegraded":"2024-09-17T03:51:52.173198+0000","last_fullsized":"2024-09-17T03:51:52.173198+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:39:53.033059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":13,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:52.173133+0000","last_change":"2024-09-17T03:51:51.183736+0000","last_active":"2024-09-17T03:51:52.173133+0000","last_peered":"2024-09-17T03:51:52.173133+0000","last_clean":"2024-09-17T03:51:52.173133+0000","last_became_active":"2024-09-17T03:51:51.179462+0000","last_became_peered":"2024-09-17T03:51:51.179462+0000","last_unstale":"2024-09-17T03:51:52.173133+0000","last_undegraded":"2024-09-17T03:51:52.173133+0000","last_fullsized":"2024-09-17T03:51:52.173133+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:50.134458+0000","last_clean_scrub_stamp":"2024-09-17T03:51:50.134458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:57.563505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.162728+0000","last_change":"2024-09-17T03:51:50.171420+0000","last_active":"2024-09-17T03:51:53.162728+0000","last_peered":"2024-09-17T03:51:53.162728+0000","last_clean":"2024-09-17T03:51:53.162728+0000","last_became_active":"2024-09-17T03:51:50.171094+0000","last_became_peered":"2024-09-17T03:51:50.171094+0000","last_unstale":"2024-09-17T03:51:53.162728+0000","last_undegraded":"2024-09-17T03:51:53.162728+0000","last_fullsized":"2024-09-17T03:51:53.162728+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:51:49.127503+0000","last_clean_scrub_stamp":"2024-09-17T03:51:49.127503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:00.559563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":172,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-09-17T03:51:53.730375+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:51:53.730375+0000","last_peered":"2024-09-17T03:51:53.730375+0000","last_clean":"2024-09-17T03:51:53.730375+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:51:53.730375+0000","last_undegraded":"2024-09-17T03:51:53.730375+0000","last_fullsized":"2024-09-17T03:51:53.730375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509472,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32424,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339416,"statfs":{"total":96636764160,"available":96603561984,"internally_reserved":0,"allocated":1265664,"data_stored":862593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509475,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509474,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":675840,"data_stored":273677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509472,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509474,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31852,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339988,"statfs":{"total":96636764160,"available":96604147712,"internally_reserved":0,"allocated":671744,"data_stored":272121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509474,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31848,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339992,"statfs":{"total":96636764160,"available":96604151808,"internally_reserved":0,"allocated":667648,"data_stored":272155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542176,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31832,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340008,"statfs":{"total":96636764160,"available":96604168192,"internally_reserved":0,"allocated":651264,"data_stored":265765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542177,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1273856,"data_stored":864627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:51:56.085 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:51:56.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.349+0000 7f48c6800640 1 -- 172.21.15.129:0/3775604827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070030 msgr2=0x7f48c0074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.349+0000 7f48c6800640 1 --2- 172.21.15.129:0/3775604827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070030 0x7f48c0074640 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f48ac005fd0 tx=0x7f48ac02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 -- 172.21.15.129:0/3775604827 shutdown_connections 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 --2- 172.21.15.129:0/3775604827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070440 0x7f48c0074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 --2- 172.21.15.129:0/3775604827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070030 0x7f48c0074640 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 -- 172.21.15.129:0/3775604827 >> 172.21.15.129:0/3775604827 conn(0x7f48c00b1cb0 msgr2=0x7f48c00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:56.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 -- 172.21.15.129:0/3775604827 shutdown_connections 2024-09-17T03:51:56.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.350+0000 7f48c6800640 1 -- 172.21.15.129:0/3775604827 wait complete. 2024-09-17T03:51:56.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.352+0000 7f48c6800640 1 Processor -- start 2024-09-17T03:51:56.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.352+0000 7f48c6800640 1 -- start start 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070030 0x7f48c0153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f48c00b3a40 con 0x7f48c0073170 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f48c00b38c0 con 0x7f48c0070440 2024-09-17T03:51:56.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.353+0000 7f48c6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f48c00b3bc0 con 0x7f48c0070030 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bf400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070030 0x7f48c0153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:46644/0 (socket says 172.21.15.129:46644) 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49784/0 (socket says 172.21.15.129:49784) 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 -- 172.21.15.129:0/2642273473 learned_addr learned my addr 172.21.15.129:0/2642273473 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 -- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070030 msgr2=0x7f48c0153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070030 0x7f48c0153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 -- 172.21.15.129:0/2642273473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 msgr2=0x7f48c009cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bea00640 1 -- 172.21.15.129:0/2642273473 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c01f7a70 con 0x7f48c0070440 2024-09-17T03:51:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.354+0000 7f48bfe00640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:51:56.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48bea00640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac005770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:56.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac03e030 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f48ac003a90 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c00c3d90 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48ac035040 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.355+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c009d140 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.356+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c00b40e0 con 0x7f48c0070440 2024-09-17T03:51:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.358+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f48ac041030 con 0x7f48c0070440 2024-09-17T03:51:56.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.358+0000 7f48b7400640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 0x7f48a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.359+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f48ac0bb120 con 0x7f48c0070440 2024-09-17T03:51:56.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.359+0000 7f48bf400640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 0x7f48a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.360+0000 7f48bf400640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 0x7f48a007b0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f48b0007fd0 tx=0x7f48b0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.365+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f48ac082e70 con 0x7f48c0070440 2024-09-17T03:51:56.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.614+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f48c00c3860 con 0x7f48c0070440 2024-09-17T03:51:56.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.622+0000 7f48b7400640 1 -- 172.21.15.129:0/2642273473 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v29) ==== 190+0+147 (secure 0 0 0) 0x7f48ac0878a0 con 0x7f48c0070440 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:51:56.622 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:51:56.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 msgr2=0x7f48a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 0x7f48a007b0a0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f48b0007fd0 tx=0x7f48b0009040 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 msgr2=0x7f48c00c4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f48ac000c00 tx=0x7f48ac005770 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 shutdown_connections 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f48a0078b90 0x7f48a007b0a0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f48c0073170 0x7f48c009cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48c0070440 0x7f48c00c4870 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 --2- 172.21.15.129:0/2642273473 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f48c0070030 0x7f48c0153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 >> 172.21.15.129:0/2642273473 conn(0x7f48c00b1cb0 msgr2=0x7f48c009daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:56.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 shutdown_connections 2024-09-17T03:51:56.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.624+0000 7f48c6800640 1 -- 172.21.15.129:0/2642273473 wait complete. 2024-09-17T03:51:56.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:51:56.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 -- 172.21.15.129:0/4273981198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780af700 msgr2=0x7f0a780caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 --2- 172.21.15.129:0/4273981198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780af700 0x7f0a780caa80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0a78074480 tx=0x7f0a6802a530 comp rx=0 tx=0).stop 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 -- 172.21.15.129:0/4273981198 shutdown_connections 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 --2- 172.21.15.129:0/4273981198 >> v2:172.21.15.181:3300/0 conn(0x7f0a780b54d0 0x7f0a780b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 --2- 172.21.15.129:0/4273981198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780af700 0x7f0a780caa80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 -- 172.21.15.129:0/4273981198 >> 172.21.15.129:0/4273981198 conn(0x7f0a7806d870 msgr2=0x7f0a7806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 -- 172.21.15.129:0/4273981198 shutdown_connections 2024-09-17T03:51:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.866+0000 7f0a7da00640 1 -- 172.21.15.129:0/4273981198 wait complete. 2024-09-17T03:51:56.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.869+0000 7f0a7da00640 1 Processor -- start 2024-09-17T03:51:56.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.869+0000 7f0a7da00640 1 -- start start 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a780af700 0x7f0a78142aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a780b54d0 0x7f0a78153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7815d250 con 0x7f0a780735d0 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7815d0d0 con 0x7f0a780b54d0 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a7da00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0a7815d3d0 con 0x7f0a780af700 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49798/0 (socket says 172.21.15.129:49798) 2024-09-17T03:51:56.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.870+0000 7f0a77e00640 1 -- 172.21.15.129:0/2320611527 learned_addr learned my addr 172.21.15.129:0/2320611527 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a7d000640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a780b54d0 0x7f0a78153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a6f400640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a780af700 0x7f0a78142aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 -- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a780af700 msgr2=0x7f0a78142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a780af700 0x7f0a78142aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 -- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a780b54d0 msgr2=0x7f0a78153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a780b54d0 0x7f0a78153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 -- 172.21.15.129:0/2320611527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a781f3f90 con 0x7f0a780735d0 2024-09-17T03:51:56.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a77e00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098f0 tx=0x7f0a68033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005c90 con 0x7f0a780735d0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.871+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a68031e50 con 0x7f0a780735d0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.872+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68005260 con 0x7f0a780735d0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.872+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781f4470 con 0x7f0a780735d0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.872+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a780afeb0 con 0x7f0a780735d0 2024-09-17T03:51:56.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.873+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7815d930 con 0x7f0a780735d0 2024-09-17T03:51:56.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.874+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0a68005400 con 0x7f0a780735d0 2024-09-17T03:51:56.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.875+0000 7f0a74c00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 0x7f0a5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:56.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.875+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f0a680bbea0 con 0x7f0a780735d0 2024-09-17T03:51:56.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.875+0000 7f0a6f400640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 0x7f0a5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:56.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.876+0000 7f0a6f400640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 0x7f0a5407b0a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a60007fd0 tx=0x7f0a60009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:56.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:56.879+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a781f3f90 con 0x7f0a780735d0 2024-09-17T03:51:57.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.096+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a781566f0 con 0x7f0a780735d0 2024-09-17T03:51:57.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.097+0000 7f0a74c00640 1 -- 172.21.15.129:0/2320611527 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f0a6803f030 con 0x7f0a780735d0 2024-09-17T03:51:57.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:51:57.097 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:51:57.097 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:51:57.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 msgr2=0x7f0a5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 0x7f0a5407b0a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a60007fd0 tx=0x7f0a60009040 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 msgr2=0x7f0a780b0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0a680098f0 tx=0x7f0a68033c20 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 shutdown_connections 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a54078b90 0x7f0a5407b0a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a780b54d0 0x7f0a78153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a780af700 0x7f0a78142aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 --2- 172.21.15.129:0/2320611527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a780735d0 0x7f0a780b0990 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 >> 172.21.15.129:0/2320611527 conn(0x7f0a7806d870 msgr2=0x7f0a780b4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.099+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 shutdown_connections 2024-09-17T03:51:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.100+0000 7f0a7da00640 1 -- 172.21.15.129:0/2320611527 wait complete. 2024-09-17T03:51:57.525 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:51:50.646668+0000', 'modified': '2024-09-17T03:51:53.666499+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5550}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5550': {'gid': 5550, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/2835182709', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2835182709}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 2835182709}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5550, 'qdb_cluster': [5550]} 2024-09-17T03:51:57.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:51:57.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:51:57.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 -- 172.21.15.129:0/345522498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaac003a50 msgr2=0x7fbaac003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 --2- 172.21.15.129:0/345522498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaac003a50 0x7fbaac003ec0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001690 tx=0x7fbaac005830 comp rx=0 tx=0).stop 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 -- 172.21.15.129:0/345522498 shutdown_connections 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 --2- 172.21.15.129:0/345522498 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaac003a50 0x7fbaac003ec0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 -- 172.21.15.129:0/345522498 >> 172.21.15.129:0/345522498 conn(0x7fbac40b1cb0 msgr2=0x7fbac40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 -- 172.21.15.129:0/345522498 shutdown_connections 2024-09-17T03:51:57.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.749+0000 7fbacbc00640 1 -- 172.21.15.129:0/345522498 wait complete. 2024-09-17T03:51:57.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 Processor -- start 2024-09-17T03:51:57.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 -- start start 2024-09-17T03:51:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaac003a50 0x7fbac41e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41f5cf0 0x7fbac41f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbac40cf340 con 0x7fbac409aa30 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbac40cf1c0 con 0x7fbac41f5cf0 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbacbc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbac40701b0 con 0x7fbaac003a50 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49818/0 (socket says 172.21.15.129:49818) 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.753+0000 7fbac3e00640 1 -- 172.21.15.129:0/2274991736 learned_addr learned my addr 172.21.15.129:0/2274991736 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac9c00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41f5cf0 0x7fbac41f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac9200640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaac003a50 0x7fbac41e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 -- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaac003a50 msgr2=0x7fbac41e85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaac003a50 0x7fbac41e85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 -- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41f5cf0 msgr2=0x7fbac41f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41f5cf0 0x7fbac41f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 -- 172.21.15.129:0/2274991736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac41f5550 con 0x7fbac409aa30 2024-09-17T03:51:57.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac3e00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006630 tx=0x7fbab00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac0c00640 1 -- 172.21.15.129:0/2274991736 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0015070 con 0x7fbac409aa30 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac0c00640 1 -- 172.21.15.129:0/2274991736 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab0011900 con 0x7fbac409aa30 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.754+0000 7fbac0c00640 1 -- 172.21.15.129:0/2274991736 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab0011d50 con 0x7fbac409aa30 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.755+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbac41ea510 con 0x7fbac409aa30 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.755+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbac415d340 con 0x7fbac409aa30 2024-09-17T03:51:57.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.757+0000 7fbac0c00640 1 -- 172.21.15.129:0/2274991736 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbab0018070 con 0x7fbac409aa30 2024-09-17T03:51:57.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.758+0000 7fbac0c00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 0x7fbaa407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.758+0000 7fbac0c00640 1 -- 172.21.15.129:0/2274991736 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fbab009bb60 con 0x7fbac409aa30 2024-09-17T03:51:57.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.759+0000 7fbad2e4b740 1 Processor -- start 2024-09-17T03:51:57.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.759+0000 7fbad2e4b740 1 -- start start 2024-09-17T03:51:57.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.759+0000 7fbac9200640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 0x7fbaa407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbac9200640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 0x7fbaa407b0a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbab8032000 tx=0x7fbab8001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890ddd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909ec590 0x564890dadf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564890d37430 con 0x5648909e43b0 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564890d275d0 con 0x564890a0d270 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564890d16a00 con 0x5648909ec590 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49830/0 (socket says 172.21.15.129:49830) 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.760+0000 7fbac3e00640 1 -- 172.21.15.129:0/3350568456 learned_addr learned my addr 172.21.15.129:0/3350568456 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac9200640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890ddd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac9c00640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909ec590 0x564890dadf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 -- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909ec590 msgr2=0x564890dadf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909ec590 0x564890dadf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 -- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 msgr2=0x564890ddd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890ddd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 -- 172.21.15.129:0/3350568456 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564890cdf520 con 0x5648909e43b0 2024-09-17T03:51:57.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac9200640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890ddd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:57.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbab0063850 tx=0x7fbab0006ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb24ecd57de189f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbab5600640 1 -- 172.21.15.129:0/3350568456 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab001f030 con 0x5648909e43b0 2024-09-17T03:51:57.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbab5600640 1 -- 172.21.15.129:0/3350568456 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab00682f0 con 0x5648909e43b0 2024-09-17T03:51:57.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.761+0000 7fbab5600640 1 -- 172.21.15.129:0/3350568456 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab006cf70 con 0x5648909e43b0 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3350568456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 msgr2=0x564890dddeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fbab0063850 tx=0x7fbab0006ee0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3350568456 shutdown_connections 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909ec590 0x564890dadf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890dddeb0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3350568456 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890ddd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3350568456 >> 172.21.15.129:0/3350568456 conn(0x564890a013c0 msgr2=0x564890db3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3350568456 shutdown_connections 2024-09-17T03:51:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.762+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3350568456 wait complete. 2024-09-17T03:51:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.763+0000 7fbad2e4b740 1 -- start start 2024-09-17T03:51:57.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.764+0000 7fbad2e4b740 1 Processor -- start 2024-09-17T03:51:57.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dae430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564890cbd330 con 0x5648909e43b0 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564890d1a810 con 0x5648909ec590 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564890a87620 con 0x564890a0d270 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbad2e4b740 10 client.0 _ll_register_callbacks cb 0x564890cc2420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbac9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbac9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49846/0 (socket says 172.21.15.129:49846) 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.765+0000 7fbac9200640 1 -- 172.21.15.129:0/3623991265 learned_addr learned my addr 172.21.15.129:0/3623991265 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9c00640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dae430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 -- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 msgr2=0x564890dd7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 -- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 msgr2=0x564890dae430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dae430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 -- 172.21.15.129:0/3623991265 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564890a78850 con 0x5648909e43b0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbac9200640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001690 tx=0x7fbab8006b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a765a193676e56 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbaaa000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.766+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab804f060 con 0x5648909e43b0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbab00682f0 con 0x5648909e43b0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fbab80039f0 con 0x5648909e43b0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbab8038040 con 0x5648909e43b0 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 10 client.5664 fetch_fsmap learned FSMap version 35 2024-09-17T03:51:57.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x564890aef6b0 con 0x5648909e43b0 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab00682f0 con 0x5648909e43b0 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fbab8049070 con 0x5648909e43b0 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 10 client.5664 fetch_fsmap finished waiting for FSMap version 35 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 10 client.5664 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 20 client.5664 populate_metadata read hostname 'smithi129' 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x5648909cbe40 0x564890e15650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 4 client.5664 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 --> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564890d29520 con 0x5648909cbe40 2024-09-17T03:51:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.767+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x5648909cbe40 0x564890e15650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.768+0000 7fbac3e00640 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x5648909cbe40 0x564890e15650 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d75be71247d2b03e server_cookie=8132fc336fcd4fcf in_seq=0 out_seq=0 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.768+0000 7fbaaa000640 10 client.5664 ms_handle_connect on v2:172.21.15.129:6832/2835182709 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbaaa000640 1 -- 172.21.15.129:0/3623991265 <== mds.0 v2:172.21.15.129:6832/2835182709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x564890d29520 con 0x5648909cbe40 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbaaa000640 10 client.5664 handle_command_reply: tid=0 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 client.5664 shutdown 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 msgr2=0x564890e08570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbab8001690 tx=0x7fbab8006b70 comp rx=0 tx=0).stop 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 shutdown_connections 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x5648909cbe40 0x564890e15650 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dae430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbaaa000640 0 client.5664 ms_handle_reset on v2:172.21.15.129:6832/2835182709 2024-09-17T03:51:57.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/3623991265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890e08570 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 >> 172.21.15.129:0/3623991265 conn(0x564890a013c0 msgr2=0x564890dfb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 shutdown_connections 2024-09-17T03:51:57.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 1 -- 172.21.15.129:0/3623991265 wait complete. 2024-09-17T03:51:57.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.769+0000 7fbad2e4b740 20 client.5664 trim_cache size 0 max 16384 2024-09-17T03:51:57.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.788+0000 7fbad2e4b740 1 Processor -- start 2024-09-17T03:51:57.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.789+0000 7fbad2e4b740 1 -- start start 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dda650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbac4201240 con 0x5648909e43b0 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbac40b49d0 con 0x5648909ec590 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbac415c380 con 0x564890a0d270 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49860/0 (socket says 172.21.15.129:49860) 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 -- 172.21.15.129:0/2385108079 learned_addr learned my addr 172.21.15.129:0/2385108079 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:57.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9c00640 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac3e00640 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dda650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 -- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 msgr2=0x564890dd3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 -- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 msgr2=0x564890dda650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.790+0000 7fbac9200640 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dda650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.791+0000 7fbac9200640 1 -- 172.21.15.129:0/2385108079 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564890db50a0 con 0x5648909e43b0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.791+0000 7fbac9200640 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbab803c460 tx=0x7fbab803c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d8c39d26c28d87 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.791+0000 7fbab4c00640 1 -- 172.21.15.129:0/2385108079 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab803c550 con 0x5648909e43b0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.791+0000 7fbab4c00640 1 -- 172.21.15.129:0/2385108079 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab803c7e0 con 0x5648909e43b0 2024-09-17T03:51:57.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.791+0000 7fbab4c00640 1 -- 172.21.15.129:0/2385108079 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab803cbf0 con 0x5648909e43b0 2024-09-17T03:51:57.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 -- 172.21.15.129:0/2385108079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 msgr2=0x564890db3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbab803c460 tx=0x7fbab803c490 comp rx=0 tx=0).stop 2024-09-17T03:51:57.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 -- 172.21.15.129:0/2385108079 shutdown_connections 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564890a0d270 0x564890dd3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5648909ec590 0x564890dda650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/2385108079 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909e43b0 0x564890db3940 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 -- 172.21.15.129:0/2385108079 >> 172.21.15.129:0/2385108079 conn(0x564890a013c0 msgr2=0x564890dd7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 -- 172.21.15.129:0/2385108079 shutdown_connections 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.792+0000 7fbad2e4b740 1 -- 172.21.15.129:0/2385108079 wait complete. 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.793+0000 7fbad2e4b740 1 -- start start 2024-09-17T03:51:57.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.793+0000 7fbad2e4b740 1 Processor -- start 2024-09-17T03:51:57.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.794+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909e43b0 0x564890dd65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890e04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564890dc49a0 con 0x5648909ec590 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564890dd7a70 con 0x564890a0d270 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564890e08e80 con 0x5648909e43b0 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbad2e4b740 10 client.0 _ll_register_callbacks cb 0x7fbab8001560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49872/0 (socket says 172.21.15.129:49872) 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 -- 172.21.15.129:0/506711227 learned_addr learned my addr 172.21.15.129:0/506711227 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac9200640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909e43b0 0x564890dd65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac9c00640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890e04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 -- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909e43b0 msgr2=0x564890dd65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909e43b0 0x564890dd65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 -- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 msgr2=0x564890e04d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890e04d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 -- 172.21.15.129:0/506711227 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564890db4650 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbac3e00640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbab0068260 tx=0x7fbab0099ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1787b7504009888f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.795+0000 7fbaaa000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbab001f040 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fbab006cf20 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbab803c7e0 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbab00682f0 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 10 client.5670 fetch_fsmap learned FSMap version 35 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x564890df2e20 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbab803c7e0 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 35) ==== 3502+0+0 (secure 0 0 0) 0x7fbab004e070 con 0x5648909ec590 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 10 client.5670 fetch_fsmap finished waiting for FSMap version 35 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 10 client.5670 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 20 client.5670 populate_metadata read hostname 'smithi129' 2024-09-17T03:51:57.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x564890e79a50 0x564890e07490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:57.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 4 client.5670 mds_command: new command op to 5550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:51:57.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 --> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] -- command(tid 0: {"prefix": "status"}) -- 0x564890d24fd0 con 0x564890e79a50 2024-09-17T03:51:57.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.796+0000 7fbac9200640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x564890e79a50 0x564890e07490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:57.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbaaa000640 10 client.5670 ms_handle_connect on v2:172.21.15.129:6832/2835182709 2024-09-17T03:51:57.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbac9200640 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x564890e79a50 0x564890e07490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d622dec2be159982 server_cookie=450f88f8e9960ec4 in_seq=0 out_seq=0 2024-09-17T03:51:57.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbaaa000640 1 -- 172.21.15.129:0/506711227 <== mds.0 v2:172.21.15.129:6832/2835182709 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x564890d24fd0 con 0x564890e79a50 2024-09-17T03:51:57.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbaaa000640 10 client.5670 handle_command_reply: tid=0 2024-09-17T03:51:57.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbad2e4b740 1 client.5670 shutdown 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 msgr2=0x564890e047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.797+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbab0068260 tx=0x7fbab0099ec0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 shutdown_connections 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:6832/2835182709,v1:172.21.15.129:6834/2835182709] conn(0x564890e79a50 0x564890e07490 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564890a0d270 0x564890e04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5648909ec590 0x564890e047e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbaaa000640 0 client.5670 ms_handle_reset on v2:172.21.15.129:6832/2835182709 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 --2- 172.21.15.129:0/506711227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5648909e43b0 0x564890dd65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 >> 172.21.15.129:0/506711227 conn(0x564890a013c0 msgr2=0x564890dda650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 shutdown_connections 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 1 -- 172.21.15.129:0/506711227 wait complete. 2024-09-17T03:51:57.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbad2e4b740 20 client.5670 trim_cache size 0 max 16384 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 5550, 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.6443945099999997, 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 35, 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 56, 2024-09-17T03:51:57.802 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 56, 2024-09-17T03:51:57.803 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.678293939 2024-09-17T03:51:57.803 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:51:57.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 msgr2=0x7fbaa407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 0x7fbaa407b0a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbab8032000 tx=0x7fbab8001190 comp rx=0 tx=0).stop 2024-09-17T03:51:57.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 msgr2=0x7fbac41f8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.798+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006630 tx=0x7fbab00039c0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 shutdown_connections 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbaa4078b90 0x7fbaa407b0a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbac41f5cf0 0x7fbac41f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbac409aa30 0x7fbac41f8f50 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 --2- 172.21.15.129:0/2274991736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaac003a50 0x7fbac41e85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 >> 172.21.15.129:0/2274991736 conn(0x7fbac40b1cb0 msgr2=0x7fbac4073a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 shutdown_connections 2024-09-17T03:51:57.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:57.799+0000 7fbacbc00640 1 -- 172.21.15.129:0/2274991736 wait complete. 2024-09-17T03:51:57.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.042+0000 7f10b0e00640 1 -- 172.21.15.129:0/2275748477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0cd3e0 msgr2=0x7f10ac070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.042+0000 7f10b0e00640 1 --2- 172.21.15.129:0/2275748477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0cd3e0 0x7f10ac070180 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1094009830 tx=0x7f109402a4b0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 -- 172.21.15.129:0/2275748477 shutdown_connections 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 --2- 172.21.15.129:0/2275748477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0cd3e0 0x7f10ac070180 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 --2- 172.21.15.129:0/2275748477 >> v2:172.21.15.181:3300/0 conn(0x7f10ac0af780 0x7f10ac0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 -- 172.21.15.129:0/2275748477 >> 172.21.15.129:0/2275748477 conn(0x7f10ac06d7d0 msgr2=0x7f10ac0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:58.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 -- 172.21.15.129:0/2275748477 shutdown_connections 2024-09-17T03:51:58.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.043+0000 7f10b0e00640 1 -- 172.21.15.129:0/2275748477 wait complete. 2024-09-17T03:51:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.045+0000 7f10b0e00640 1 Processor -- start 2024-09-17T03:51:58.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.045+0000 7f10b0e00640 1 -- start start 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.046+0000 7f10b0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.046+0000 7f10b0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0af780 0x7f10ac1f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.046+0000 7f10b0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 0x7f10ac148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10b0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac073b80 con 0x7f10ac0706c0 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10b0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac073a00 con 0x7f10ac0cd3e0 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10b0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10ac1f4b80 con 0x7f10ac0af780 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:58.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49878/0 (socket says 172.21.15.129:49878) 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 -- 172.21.15.129:0/3478851904 learned_addr learned my addr 172.21.15.129:0/3478851904 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a1400640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0af780 0x7f10ac1f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10aa800640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 0x7f10ac148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 -- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0af780 msgr2=0x7f10ac1f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0af780 0x7f10ac1f77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 -- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 msgr2=0x7f10ac148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 0x7f10ac148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 -- 172.21.15.129:0/3478851904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac1e7350 con 0x7f10ac0706c0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10aa800640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 0x7f10ac148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.047+0000 7f10a9e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f109c00e790 tx=0x7f109c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.048+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c00eea0 con 0x7f10ac0706c0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.048+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f109c00cdb0 con 0x7f10ac0706c0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.048+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c010690 con 0x7f10ac0706c0 2024-09-17T03:51:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.048+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac1e4f20 con 0x7f10ac0706c0 2024-09-17T03:51:58.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.048+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1e54e0 con 0x7f10ac0706c0 2024-09-17T03:51:58.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.049+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac0b39c0 con 0x7f10ac0706c0 2024-09-17T03:51:58.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.050+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f109c010830 con 0x7f10ac0706c0 2024-09-17T03:51:58.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.051+0000 7f10a2a00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 0x7f108807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:58.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.051+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f109c09b6f0 con 0x7f10ac0706c0 2024-09-17T03:51:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.052+0000 7f10a1400640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 0x7f108807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:58.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.052+0000 7f10a1400640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 0x7f108807b0a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1094034000 tx=0x7f1094004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:58.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.056+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f109c0634d0 con 0x7f10ac0706c0 2024-09-17T03:51:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.276+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f10ac1e4bd0 con 0x7f10ac0706c0 2024-09-17T03:51:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.276+0000 7f10a2a00640 1 -- 172.21.15.129:0/3478851904 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7f109c067f00 con 0x7f10ac0706c0 2024-09-17T03:51:58.276 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:51:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 msgr2=0x7f108807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 0x7f108807b0a0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1094034000 tx=0x7f1094004730 comp rx=0 tx=0).stop 2024-09-17T03:51:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 msgr2=0x7f10ac1e6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f109c00e790 tx=0x7f109c00ec50 comp rx=0 tx=0).stop 2024-09-17T03:51:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 shutdown_connections 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1088078b90 0x7f108807b0a0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ac0cd3e0 0x7f10ac148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ac0af780 0x7f10ac1f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 --2- 172.21.15.129:0/3478851904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ac0706c0 0x7f10ac1e6b20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 >> 172.21.15.129:0/3478851904 conn(0x7f10ac06d7d0 msgr2=0x7f10ac0742e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 shutdown_connections 2024-09-17T03:51:58.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58.280+0000 7f10b0e00640 1 -- 172.21.15.129:0/3478851904 wait complete. 2024-09-17T03:51:58.304 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:51:58.304 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:51:58.304 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:51:58.304 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:51:58.304 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:58.305 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:51:58.305 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:51:58.305 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:51:58.305 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:58.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.305 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:51:58.342 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:51:58.342 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:51:58.342 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:51:58.342 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.342 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:51:58.343 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42574sec preferred_lft 42574sec 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:51:58.344 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.345 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:51:58.345 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:51:58.345 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:51:58.345 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:51:58.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.346 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:51:58.436 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:51:58.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.440 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:51:58.476 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:51:58.476 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:51:58.477 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:51:58.477 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:51:58.477 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:51:58.477 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:51:58.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:51:58.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:51:58.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.674 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:51:58.710 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:51:58.710 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:51:58.711 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:58.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:51:58.807 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:51:58.807 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:58.843 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:51:58.843 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:51:58.843 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:58.919 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:51:58.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:58.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:59.114 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:59.114 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:59.114 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:59.134 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:59.134 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:59.134 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:59.134 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:51:59.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:51:59.167 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:51:59.168 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:51:59.168 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:51:59.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:51:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:51:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:51:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:51:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:51:59.172 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:51:59.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:51:59.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:51:59.229 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:51:59.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:51:59.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:51:59.325 INFO:teuthology.orchestra.run.smithi129.stdout:client5679 2024-09-17T03:51:59.325 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:51:59.325 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:51:59.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-09-17T03:51:59.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 -- 172.21.15.129:0/2959946152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f107406cde0 msgr2=0x7f107406d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:59.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 --2- 172.21.15.129:0/2959946152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f107406cde0 0x7f107406d1f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f106c005fd0 tx=0x7f106c02aa90 comp rx=0 tx=0).stop 2024-09-17T03:51:59.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 -- 172.21.15.129:0/2959946152 shutdown_connections 2024-09-17T03:51:59.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 --2- 172.21.15.129:0/2959946152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f107406cde0 0x7f107406d1f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 --2- 172.21.15.129:0/2959946152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 0x7f107415f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 -- 172.21.15.129:0/2959946152 >> 172.21.15.129:0/2959946152 conn(0x7f10740cdcb0 msgr2=0x7f10740ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:59.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 -- 172.21.15.129:0/2959946152 shutdown_connections 2024-09-17T03:51:59.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.628+0000 7f107b400640 1 -- 172.21.15.129:0/2959946152 wait complete. 2024-09-17T03:51:59.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.630+0000 7f107b400640 1 Processor -- start 2024-09-17T03:51:59.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.630+0000 7f107b400640 1 -- start start 2024-09-17T03:51:59.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f107406cde0 0x7f10740b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:59.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 0x7f107414ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10740b3350 con 0x7f10740701b0 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10740b31d0 con 0x7f107406cde0 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f107b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10740b34d0 con 0x7f107415c570 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49892/0 (socket says 172.21.15.129:49892) 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 -- 172.21.15.129:0/3377286424 learned_addr learned my addr 172.21.15.129:0/3377286424 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f1079400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 0x7f107414ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f1078a00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f107406cde0 0x7f10740b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 -- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 msgr2=0x7f107414ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 0x7f107414ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 -- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f107406cde0 msgr2=0x7f10740b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.632+0000 7f106be00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f107406cde0 0x7f10740b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f106be00640 1 -- 172.21.15.129:0/3377286424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10741485e0 con 0x7f10740701b0 2024-09-17T03:51:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f106be00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f105c00d650 tx=0x7f105c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c017070 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f105c009b10 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10741ebcb0 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c010460 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.633+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1074148b60 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.634+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10740b3a30 con 0x7f10740701b0 2024-09-17T03:51:59.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.636+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f105c01a070 con 0x7f10740701b0 2024-09-17T03:51:59.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.637+0000 7f1071600640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 0x7f105007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:51:59.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.637+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f105c09ad50 con 0x7f10740701b0 2024-09-17T03:51:59.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.638+0000 7f1078a00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 0x7f105007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:51:59.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.638+0000 7f1078a00640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 0x7f105007b0a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f107409f280 tx=0x7f1064009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:51:59.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.642+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f105c062b30 con 0x7f10740701b0 2024-09-17T03:51:59.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.824+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f107406db30 con 0x7f10740701b0 2024-09-17T03:51:59.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.829+0000 7f1071600640 1 -- 172.21.15.129:0/3377286424 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v38)=0 v38) ==== 124+0+0 (secure 0 0 0) 0x7f105c067560 con 0x7f10740701b0 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 msgr2=0x7f105007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 0x7f105007b0a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f107409f280 tx=0x7f1064009040 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 msgr2=0x7f107414a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f105c00d650 tx=0x7f105c00db10 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 shutdown_connections 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1050078b90 0x7f105007b0a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f107415c570 0x7f107414ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10740701b0 0x7f107414a5d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 --2- 172.21.15.129:0/3377286424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f107406cde0 0x7f10740b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:51:59.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 >> 172.21.15.129:0/3377286424 conn(0x7f10740cdcb0 msgr2=0x7f107415f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:51:59.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 shutdown_connections 2024-09-17T03:51:59.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:51:59.832+0000 7f107b400640 1 -- 172.21.15.129:0/3377286424 wait complete. 2024-09-17T03:51:59.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 1 2024-09-17T03:52:00.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 -- 172.21.15.129:0/294735146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070030 msgr2=0x7f64f4074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:00.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 --2- 172.21.15.129:0/294735146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070030 0x7f64f4074640 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f64e0005fd0 tx=0x7f64e002a670 comp rx=0 tx=0).stop 2024-09-17T03:52:00.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 -- 172.21.15.129:0/294735146 shutdown_connections 2024-09-17T03:52:00.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 --2- 172.21.15.129:0/294735146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070030 0x7f64f4074640 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 -- 172.21.15.129:0/294735146 >> 172.21.15.129:0/294735146 conn(0x7f64f40b1cb0 msgr2=0x7f64f40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:00.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 -- 172.21.15.129:0/294735146 shutdown_connections 2024-09-17T03:52:00.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.077+0000 7f64fb600640 1 -- 172.21.15.129:0/294735146 wait complete. 2024-09-17T03:52:00.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.079+0000 7f64fb600640 1 Processor -- start 2024-09-17T03:52:00.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.079+0000 7f64fb600640 1 -- start start 2024-09-17T03:52:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 0x7f64f409de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 0x7f64f415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f64f40b3c40 con 0x7f64f4070440 2024-09-17T03:52:00.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f64f40b3ac0 con 0x7f64f4070030 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64fb600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f64f40b3dc0 con 0x7f64f4073170 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49908/0 (socket says 172.21.15.129:49908) 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64f3e00640 1 -- 172.21.15.129:0/2804804639 learned_addr learned my addr 172.21.15.129:0/2804804639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64f8c00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 0x7f64f409de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.081+0000 7f64f9600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 0x7f64f415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 -- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 msgr2=0x7f64f415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 0x7f64f415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 -- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 msgr2=0x7f64f409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 0x7f64f409de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 -- 172.21.15.129:0/2804804639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f409ada0 con 0x7f64f4070440 2024-09-17T03:52:00.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f8c00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 0x7f64f409de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f9600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 0x7f64f415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f3e00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009230 tx=0x7f64e0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e003e030 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64e0003e00 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f409b080 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.082+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0035040 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.083+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f409b640 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.084+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f40b4300 con 0x7f64f4070440 2024-09-17T03:52:00.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.085+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f64e0041030 con 0x7f64f4070440 2024-09-17T03:52:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.086+0000 7f64f0c00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 0x7f64d407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.086+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f64e00bb6a0 con 0x7f64f4070440 2024-09-17T03:52:00.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.086+0000 7f64f8c00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 0x7f64d407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:00.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.087+0000 7f64f8c00640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 0x7f64d407b0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f64e4007fd0 tx=0x7f64e4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:00.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.091+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f64e0083a40 con 0x7f64f4070440 2024-09-17T03:52:00.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.272+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f64f415b2b0 con 0x7f64f4070440 2024-09-17T03:52:00.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.277+0000 7f64f0c00640 1 -- 172.21.15.129:0/2804804639 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v39)=0 v39) ==== 116+0+0 (secure 0 0 0) 0x7f64e0083340 con 0x7f64f4070440 2024-09-17T03:52:00.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 msgr2=0x7f64d407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 0x7f64d407b0d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f64e4007fd0 tx=0x7f64e4009040 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 msgr2=0x7f64f414b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f64e0009230 tx=0x7f64e0003910 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 shutdown_connections 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64d4078bc0 0x7f64d407b0d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64f4073170 0x7f64f415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64f4070440 0x7f64f414b4e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 --2- 172.21.15.129:0/2804804639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f4070030 0x7f64f409de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 >> 172.21.15.129:0/2804804639 conn(0x7f64f40b1cb0 msgr2=0x7f64f40ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 shutdown_connections 2024-09-17T03:52:00.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:00.279+0000 7f64fb600640 1 -- 172.21.15.129:0/2804804639 wait complete. 2024-09-17T03:52:10.303 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:10.303 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:10.344 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:52:10.344 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:52:10.344 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T03:52:10.344 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:10.344 DEBUG:teuthology.orchestra.run.smithi129:> n = 50 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T03:52:10.345 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T03:52:10.346 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T03:52:10.347 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:52:10.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:10.573 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:10.573 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:10.649 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:10.649 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:10.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 -- 172.21.15.129:0/3454841116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520073170 msgr2=0x7f152015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 --2- 172.21.15.129:0/3454841116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520073170 0x7f152015c780 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f15200710f0 tx=0x7f151002a560 comp rx=0 tx=0).stop 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 -- 172.21.15.129:0/3454841116 shutdown_connections 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 --2- 172.21.15.129:0/3454841116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520070440 0x7f1520074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 --2- 172.21.15.129:0/3454841116 >> v2:172.21.15.181:3300/0 conn(0x7f1520070030 0x7f1520074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 --2- 172.21.15.129:0/3454841116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520073170 0x7f152015c780 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 -- 172.21.15.129:0/3454841116 >> 172.21.15.129:0/3454841116 conn(0x7f15200b1cb0 msgr2=0x7f15200b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 -- 172.21.15.129:0/3454841116 shutdown_connections 2024-09-17T03:52:10.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.986+0000 7f1525400640 1 -- 172.21.15.129:0/3454841116 wait complete. 2024-09-17T03:52:10.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.988+0000 7f1525400640 1 Processor -- start 2024-09-17T03:52:10.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.989+0000 7f1525400640 1 -- start start 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 0x7f152014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 0x7f15201e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f152006f630 con 0x7f1520070030 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f152006f4b0 con 0x7f1520070440 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f1525400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f152006f7b0 con 0x7f1520073170 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f151e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f151e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47928/0 (socket says 172.21.15.129:47928) 2024-09-17T03:52:10.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.990+0000 7f151e800640 1 -- 172.21.15.129:0/2215719822 learned_addr learned my addr 172.21.15.129:0/2215719822 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151f200640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 0x7f15201e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1517e00640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 0x7f152014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 -- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 msgr2=0x7f15201e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 0x7f15201e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 -- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 msgr2=0x7f152014e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 0x7f152014e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 -- 172.21.15.129:0/2215719822 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152014eac0 con 0x7f1520070030 2024-09-17T03:52:10.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151f200640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 0x7f15201e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:10.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1517e00640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 0x7f152014e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:10.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f151e800640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1510034040 tx=0x7f15100039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151003e030 con 0x7f1520070030 2024-09-17T03:52:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15100047a0 con 0x7f1520070030 2024-09-17T03:52:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510035040 con 0x7f1520070030 2024-09-17T03:52:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.991+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15200cab50 con 0x7f1520070030 2024-09-17T03:52:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.992+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15200cb110 con 0x7f1520070030 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.992+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f152015c780 con 0x7f1520070030 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.994+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1510041030 con 0x7f1520070030 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.995+0000 7f1516a00640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 0x7f14fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.995+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f15100bb660 con 0x7f1520070030 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.995+0000 7f1517e00640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 0x7f14fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:10.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.996+0000 7f1517e00640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 0x7f14fc07b0a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15200af500 tx=0x7f1508005d10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:10.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:10.999+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1510083a90 con 0x7f1520070030 2024-09-17T03:52:11.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.196+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f15200c3860 con 0x7f1520070030 2024-09-17T03:52:11.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.196+0000 7f1516a00640 1 -- 172.21.15.129:0/2215719822 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f1510083390 con 0x7f1520070030 2024-09-17T03:52:11.197 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:52:11.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 msgr2=0x7f14fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 0x7f14fc07b0a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15200af500 tx=0x7f1508005d10 comp rx=0 tx=0).stop 2024-09-17T03:52:11.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 msgr2=0x7f15200d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f1510034040 tx=0x7f15100039e0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 shutdown_connections 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14fc078b90 0x7f14fc07b0a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1520073170 0x7f15201e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1520070440 0x7f152014e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 --2- 172.21.15.129:0/2215719822 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1520070030 0x7f15200d0ad0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 >> 172.21.15.129:0/2215719822 conn(0x7f15200b1cb0 msgr2=0x7f15200afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 shutdown_connections 2024-09-17T03:52:11.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.199+0000 7f1525400640 1 -- 172.21.15.129:0/2215719822 wait complete. 2024-09-17T03:52:11.221 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:52:11.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:11.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:11.370 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:52:11.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:11.370 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:52:11.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:11.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 -- 172.21.15.129:0/2281912287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6798004d90 msgr2=0x7f6798005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 --2- 172.21.15.129:0/2281912287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6798004d90 0x7f6798005220 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005fd0 tx=0x7f6798005830 comp rx=0 tx=0).stop 2024-09-17T03:52:11.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b2800640 1 -- 172.21.15.129:0/2281912287 reap_dead start 2024-09-17T03:52:11.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 -- 172.21.15.129:0/2281912287 shutdown_connections 2024-09-17T03:52:11.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 -- 172.21.15.129:0/2281912287 >> 172.21.15.129:0/2281912287 conn(0x7f67b409d6a0 msgr2=0x7f67b40d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:11.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 -- 172.21.15.129:0/2281912287 shutdown_connections 2024-09-17T03:52:11.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.677+0000 7f67b3200640 1 -- 172.21.15.129:0/2281912287 wait complete. 2024-09-17T03:52:11.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.679+0000 7f67b3200640 1 Processor -- start 2024-09-17T03:52:11.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.679+0000 7f67b3200640 1 -- start start 2024-09-17T03:52:11.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 0x7f67b4159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:11.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:11.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 0x7f67b415ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:11.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67b4167770 con 0x7f67b415c230 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f67b41675f0 con 0x7f67b40ca3f0 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f67b41678f0 con 0x7f6798004d90 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47952/0 (socket says 172.21.15.129:47952) 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 -- 172.21.15.129:0/704867466 learned_addr learned my addr 172.21.15.129:0/704867466 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 -- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 msgr2=0x7f67b4159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b1e00640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 0x7f67b4159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b2800640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 0x7f67b415ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 0x7f67b4159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 -- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 msgr2=0x7f67b415ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 0x7f67b415ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67a9400640 1 -- 172.21.15.129:0/704867466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b40ca800 con 0x7f67b415c230 2024-09-17T03:52:11.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b2800640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 0x7f67b415ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.681+0000 7f67b1e00640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 0x7f67b4159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67a9400640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f679c003f10 tx=0x7f679c0017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679c015070 con 0x7f67b415c230 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f679c011940 con 0x7f67b415c230 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b4159650 con 0x7f67b415c230 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f679c011d90 con 0x7f67b415c230 2024-09-17T03:52:11.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.682+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b41f5ba0 con 0x7f67b415c230 2024-09-17T03:52:11.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.683+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b40b9100 con 0x7f67b415c230 2024-09-17T03:52:11.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.684+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f679c018070 con 0x7f67b415c230 2024-09-17T03:52:11.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.685+0000 7f67aaa00640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 0x7f679007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:11.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.685+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f679c09bcc0 con 0x7f67b415c230 2024-09-17T03:52:11.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.686+0000 7f67b1e00640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 0x7f679007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:11.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.686+0000 7f67b1e00640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 0x7f679007b0a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f67a4000c00 tx=0x7f67a4001b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:11.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.690+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f679c0a2080 con 0x7f67b415c230 2024-09-17T03:52:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.913+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67b41f5650 con 0x7f67b415c230 2024-09-17T03:52:11.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.913+0000 7f67aaa00640 1 -- 172.21.15.129:0/704867466 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f67b40ca800 con 0x7f67b415c230 2024-09-17T03:52:11.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:11.914 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 msgr2=0x7f679007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 0x7f679007b0a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f67a4000c00 tx=0x7f67a4001b10 comp rx=0 tx=0).stop 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 msgr2=0x7f67b40c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f679c003f10 tx=0x7f679c0017f0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 shutdown_connections 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6790078b90 0x7f679007b0a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67b40ca3f0 0x7f67b415ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67b415c230 0x7f67b40c9eb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 --2- 172.21.15.129:0/704867466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6798004d90 0x7f67b4159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 >> 172.21.15.129:0/704867466 conn(0x7f67b409d6a0 msgr2=0x7f67b415c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 shutdown_connections 2024-09-17T03:52:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:11.916+0000 7f67b3200640 1 -- 172.21.15.129:0/704867466 wait complete. 2024-09-17T03:52:11.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dirfrag split /subdir 0/0 1 2024-09-17T03:52:12.179 DEBUG:tasks.cephfs.filesystem:_json_asok output empty 2024-09-17T03:52:12.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:12.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 -- 172.21.15.129:0/2752603763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0cc1e0 msgr2=0x7f9e3c0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 --2- 172.21.15.129:0/2752603763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0cc1e0 0x7f9e3c0af3c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c0cd710 tx=0x7f9e2c02a560 comp rx=0 tx=0).stop 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 -- 172.21.15.129:0/2752603763 shutdown_connections 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 --2- 172.21.15.129:0/2752603763 >> v2:172.21.15.181:3300/0 conn(0x7f9e3c0ccb50 0x7f9e3c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 --2- 172.21.15.129:0/2752603763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c15d8a0 0x7f9e3c0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 --2- 172.21.15.129:0/2752603763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0cc1e0 0x7f9e3c0af3c0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 -- 172.21.15.129:0/2752603763 >> 172.21.15.129:0/2752603763 conn(0x7f9e3c06d7d0 msgr2=0x7f9e3c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 -- 172.21.15.129:0/2752603763 shutdown_connections 2024-09-17T03:52:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.401+0000 7f9e42000640 1 -- 172.21.15.129:0/2752603763 wait complete. 2024-09-17T03:52:12.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.404+0000 7f9e42000640 1 Processor -- start 2024-09-17T03:52:12.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.404+0000 7f9e42000640 1 -- start start 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c0cc1e0 0x7f9e3c151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 0x7f9e3c1f9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0b4770 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e3c0b45f0 con 0x7f9e3c15d8a0 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e42000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3c1533a0 con 0x7f9e3c0cc1e0 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e32a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e32a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47970/0 (socket says 172.21.15.129:47970) 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.405+0000 7f9e32a00640 1 -- 172.21.15.129:0/1965411577 learned_addr learned my addr 172.21.15.129:0/1965411577 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e3b400640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c0cc1e0 0x7f9e3c151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e3be00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 0x7f9e3c1f9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 -- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c0cc1e0 msgr2=0x7f9e3c151ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:12.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c0cc1e0 0x7f9e3c151ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 -- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 msgr2=0x7f9e3c1f9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 0x7f9e3c1f9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 -- 172.21.15.129:0/1965411577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e3c09cb20 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e3be00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 0x7f9e3c1f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e32a00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c9a0 tx=0x7f9e2400ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24007bb0 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e24007e40 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3c09ce00 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.406+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2400f690 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.407+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3c09d3c0 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.408+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3c0ccf60 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.409+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e2400f830 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.410+0000 7f9e33e00640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 0x7f9e1807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:12.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.411+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f9e2409b4f0 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.411+0000 7f9e3b400640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 0x7f9e1807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:12.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.412+0000 7f9e3b400640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 0x7f9e1807b0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c033000 tx=0x7f9e2c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:12.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.415+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e24014030 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.635+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e3c070a60 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.636+0000 7f9e33e00640 1 -- 172.21.15.129:0/1965411577 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f9e24063920 con 0x7f9e3c0ccb50 2024-09-17T03:52:12.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:12.636 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:12.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 msgr2=0x7f9e1807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 0x7f9e1807b0d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c033000 tx=0x7f9e2c004730 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 msgr2=0x7f9e3c1f9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400c9a0 tx=0x7f9e2400ce60 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 shutdown_connections 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e18078bc0 0x7f9e1807b0d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3c15d8a0 0x7f9e3c1f9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e3c0ccb50 0x7f9e3c1f9450 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 --2- 172.21.15.129:0/1965411577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e3c0cc1e0 0x7f9e3c151ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 >> 172.21.15.129:0/1965411577 conn(0x7f9e3c06d7d0 msgr2=0x7f9e3c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.638+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 shutdown_connections 2024-09-17T03:52:12.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:12.639+0000 7f9e42000640 1 -- 172.21.15.129:0/1965411577 wait complete. 2024-09-17T03:52:12.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:52:12.918 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:52:12.919 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:52:12.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:52:13.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 -- 172.21.15.129:0/1124091070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 msgr2=0x7fe828074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:13.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 --2- 172.21.15.129:0/1124091070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe828074b80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe820005fd0 tx=0x7fe82002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 -- 172.21.15.129:0/1124091070 shutdown_connections 2024-09-17T03:52:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 --2- 172.21.15.129:0/1124091070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe828074b80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 --2- 172.21.15.129:0/1124091070 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828070030 0x7fe828074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 -- 172.21.15.129:0/1124091070 >> 172.21.15.129:0/1124091070 conn(0x7fe8280b1cb0 msgr2=0x7fe8280b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 -- 172.21.15.129:0/1124091070 shutdown_connections 2024-09-17T03:52:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.140+0000 7fe82f200640 1 -- 172.21.15.129:0/1124091070 wait complete. 2024-09-17T03:52:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.142+0000 7fe82f200640 1 Processor -- start 2024-09-17T03:52:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.142+0000 7fe82f200640 1 -- start start 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe82809de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828073170 0x7fe8281f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8280b3a90 con 0x7fe828070030 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe8280b3910 con 0x7fe828070440 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe8280b3c10 con 0x7fe828073170 2024-09-17T03:52:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe827e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe827e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47974/0 (socket says 172.21.15.129:47974) 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe827e00640 1 -- 172.21.15.129:0/1315761107 learned_addr learned my addr 172.21.15.129:0/1315761107 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe81f400640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe82809de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.144+0000 7fe82d200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828073170 0x7fe8281f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 -- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828073170 msgr2=0x7fe8281f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828073170 0x7fe8281f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 -- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 msgr2=0x7fe82809de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe82809de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 -- 172.21.15.129:0/1315761107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe828158680 con 0x7fe828070030 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe827e00640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c3d0 tx=0x7fe81800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818011790 con 0x7fe828070030 2024-09-17T03:52:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe828158960 con 0x7fe828070030 2024-09-17T03:52:13.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe818017070 con 0x7fe828070030 2024-09-17T03:52:13.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818011cc0 con 0x7fe828070030 2024-09-17T03:52:13.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.145+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8280cb240 con 0x7fe828070030 2024-09-17T03:52:13.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.146+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8280cb4a0 con 0x7fe828070030 2024-09-17T03:52:13.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.147+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe81801c460 con 0x7fe828070030 2024-09-17T03:52:13.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.149+0000 7fe824c00640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 0x7fe80407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:13.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.149+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fe81809aa80 con 0x7fe828070030 2024-09-17T03:52:13.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.150+0000 7fe81f400640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 0x7fe80407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:13.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.150+0000 7fe81f400640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 0x7fe80407b0a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe8280cbe30 tx=0x7fe810005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:13.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.154+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe81809ce20 con 0x7fe828070030 2024-09-17T03:52:13.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.371+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe8280b4130 con 0x7fe828070030 2024-09-17T03:52:13.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.371+0000 7fe824c00640 1 -- 172.21.15.129:0/1315761107 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) ==== 53+0+73 (secure 0 0 0) 0x7fe818062750 con 0x7fe828070030 2024-09-17T03:52:13.371 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.373+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 msgr2=0x7fe80407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.373+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 0x7fe80407b0a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe8280cbe30 tx=0x7fe810005f90 comp rx=0 tx=0).stop 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 msgr2=0x7fe828159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe81800c3d0 tx=0x7fe81800c890 comp rx=0 tx=0).stop 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 shutdown_connections 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe804078b90 0x7fe80407b0a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe828073170 0x7fe8281f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe828070440 0x7fe82809de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 --2- 172.21.15.129:0/1315761107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe828070030 0x7fe828159440 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 >> 172.21.15.129:0/1315761107 conn(0x7fe8280b1cb0 msgr2=0x7fe8280ad800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 shutdown_connections 2024-09-17T03:52:13.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13.374+0000 7fe82f200640 1 -- 172.21.15.129:0/1315761107 wait complete. 2024-09-17T03:52:13.397 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:52:13.398 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:13.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:13.398 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:52:13.434 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:52:13.435 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42559sec preferred_lft 42559sec 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.436 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:52:13.437 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:52:13.437 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:52:13.437 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:13.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:13.437 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:52:13.527 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:52:13.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:13.531 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:52:13.567 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:52:13.567 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:52:13.567 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:52:13.567 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:52:13.567 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:52:13.568 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:52:13.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:52:13.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:13.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:13.764 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:52:13.799 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:52:13.799 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:52:13.800 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:13.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:13.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:52:13.897 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:52:13.897 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:13.933 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:13.934 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:52:13.934 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:14.008 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:52:14.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:14.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:14.206 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:14.206 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:14.206 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:14.226 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:14.226 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:14.226 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:14.226 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:52:14.226 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:52:14.257 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:14.257 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:52:14.258 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:52:14.259 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:52:14.259 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:52:14.259 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:52:14.259 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:52:14.259 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:52:14.262 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:52:14.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:14.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:14.318 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:52:14.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:14 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:52:14.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:14.408 INFO:teuthology.orchestra.run.smithi129.stdout:client5210 2024-09-17T03:52:14.408 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:14.408 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:14.444 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:14.444 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> n = 75 2024-09-17T03:52:14.520 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T03:52:14.521 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:14.522 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T03:52:14.523 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:52:15.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:15.145 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:52:15.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:52:15.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:52:15.354 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:52:15.354 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:52:15.354 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:15.354 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:15.390 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:15.390 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:15.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:52:15.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 -- 172.21.15.129:0/2343035372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59e4004800 msgr2=0x7f59e4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 --2- 172.21.15.129:0/2343035372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59e4004800 0x7f59e4004c10 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001830 tx=0x7f59e4005830 comp rx=0 tx=0).stop 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05400640 1 -- 172.21.15.129:0/2343035372 reap_dead start 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 -- 172.21.15.129:0/2343035372 shutdown_connections 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 -- 172.21.15.129:0/2343035372 >> 172.21.15.129:0/2343035372 conn(0x7f5a0006d410 msgr2=0x7f5a000b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 -- 172.21.15.129:0/2343035372 shutdown_connections 2024-09-17T03:52:15.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.728+0000 7f5a05e00640 1 -- 172.21.15.129:0/2343035372 wait complete. 2024-09-17T03:52:15.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.730+0000 7f5a05e00640 1 Processor -- start 2024-09-17T03:52:15.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.731+0000 7f5a05e00640 1 -- start start 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59e4004800 0x7f5a001f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00144f60 0x7f5a00154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0015d700 con 0x7f5a001f2d90 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5a000cfc40 con 0x7f5a00144f60 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f5a05e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0015d880 con 0x7f59e4004800 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f59f7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f59f7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51670/0 (socket says 172.21.15.129:51670) 2024-09-17T03:52:15.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.732+0000 7f59f7e00640 1 -- 172.21.15.129:0/1949063830 learned_addr learned my addr 172.21.15.129:0/1949063830 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f5a04a00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59e4004800 0x7f5a001f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f5a05400640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00144f60 0x7f5a00154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 -- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59e4004800 msgr2=0x7f5a001f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59e4004800 0x7f5a001f7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 -- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00144f60 msgr2=0x7f5a00154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00144f60 0x7f5a00154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 -- 172.21.15.129:0/1949063830 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a001f79a0 con 0x7f5a001f2d90 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59f7e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001950 tx=0x7f59e8006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8043070 con 0x7f5a001f2d90 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f59e80038e0 con 0x7f5a001f2d90 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a00145370 con 0x7f5a001f2d90 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.733+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8003d30 con 0x7f5a001f2d90 2024-09-17T03:52:15.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.734+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a00159f50 con 0x7f5a001f2d90 2024-09-17T03:52:15.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.735+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0006ee70 con 0x7f5a001f2d90 2024-09-17T03:52:15.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.736+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f59e8046070 con 0x7f5a001f2d90 2024-09-17T03:52:15.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.737+0000 7f59fd600640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 0x7f59dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:15.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.737+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f59e80babd0 con 0x7f5a001f2d90 2024-09-17T03:52:15.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.737+0000 7f5a04a00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 0x7f59dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:15.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.738+0000 7f5a04a00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 0x7f59dc07b0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f59f0001eb0 tx=0x7f59f0000fe0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:15.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.741+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5a00145370 con 0x7f5a001f2d90 2024-09-17T03:52:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.938+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5a0006f220 con 0x7f5a001f2d90 2024-09-17T03:52:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.938+0000 7f59fd600640 1 -- 172.21.15.129:0/1949063830 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) ==== 80+0+0 (secure 0 0 0) 0x7f5a001f79a0 con 0x7f5a001f2d90 2024-09-17T03:52:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:52:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 msgr2=0x7f59dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 0x7f59dc07b0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f59f0001eb0 tx=0x7f59f0000fe0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 msgr2=0x7f5a00144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f59e8001950 tx=0x7f59e8006940 comp rx=0 tx=0).stop 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 shutdown_connections 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f59dc078bc0 0x7f59dc07b0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a00144f60 0x7f5a00154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a001f2d90 0x7f5a00144a20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 --2- 172.21.15.129:0/1949063830 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59e4004800 0x7f5a001f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 >> 172.21.15.129:0/1949063830 conn(0x7f5a0006d410 msgr2=0x7f5a00144310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 shutdown_connections 2024-09-17T03:52:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:15.941+0000 7f5a05e00640 1 -- 172.21.15.129:0/1949063830 wait complete. 2024-09-17T03:52:15.963 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:52:15.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:15.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:16.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:52:16.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:52:16.161 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:52:16.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:16.162 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:52:16.199 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:52:16.199 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:52:16.199 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:16.235 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:52:16.235 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:52:16.235 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:16.235 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:16.274 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:52:16.274 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:52:16.274 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:52:16.274 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:16.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:52:16.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:16.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 -- 172.21.15.129:0/4236319373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df815c110 msgr2=0x7f6df8166170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:16.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 --2- 172.21.15.129:0/4236319373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df815c110 0x7f6df8166170 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f6df0005fd0 tx=0x7f6df002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 -- 172.21.15.129:0/4236319373 shutdown_connections 2024-09-17T03:52:16.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 --2- 172.21.15.129:0/4236319373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df815c110 0x7f6df8166170 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 -- 172.21.15.129:0/4236319373 >> 172.21.15.129:0/4236319373 conn(0x7f6df806d870 msgr2=0x7f6df806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:16.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 -- 172.21.15.129:0/4236319373 shutdown_connections 2024-09-17T03:52:16.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.534+0000 7f6dfea00640 1 -- 172.21.15.129:0/4236319373 wait complete. 2024-09-17T03:52:16.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.536+0000 7f6dfea00640 1 Processor -- start 2024-09-17T03:52:16.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.536+0000 7f6dfea00640 1 -- start start 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6df80d0500 0x7f6df815a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6df815c110 0x7f6df815a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6df8166830 con 0x7f6df80cf3d0 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81666b0 con 0x7f6df815c110 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfea00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6df81669b0 con 0x7f6df80d0500 2024-09-17T03:52:16.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51696/0 (socket says 172.21.15.129:51696) 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 -- 172.21.15.129:0/3488774149 learned_addr learned my addr 172.21.15.129:0/3488774149 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfe000640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6df815c110 0x7f6df815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfcc00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6df80d0500 0x7f6df815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 -- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6df80d0500 msgr2=0x7f6df815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6df80d0500 0x7f6df815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 -- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6df815c110 msgr2=0x7f6df815a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6df815c110 0x7f6df815a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.538+0000 7f6dfd600640 1 -- 172.21.15.129:0/3488774149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df81f8d20 con 0x7f6df80cf3d0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6dfd600640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00b3f0 tx=0x7f6dec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec015020 con 0x7f6df80cf3d0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dec011920 con 0x7f6df80cf3d0 2024-09-17T03:52:16.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec011d70 con 0x7f6df80cf3d0 2024-09-17T03:52:16.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df81f9000 con 0x7f6df80cf3d0 2024-09-17T03:52:16.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.539+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df81f6500 con 0x7f6df80cf3d0 2024-09-17T03:52:16.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.540+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df815ba00 con 0x7f6df80cf3d0 2024-09-17T03:52:16.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.541+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6dec018020 con 0x7f6df80cf3d0 2024-09-17T03:52:16.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.542+0000 7f6df5600640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 0x7f6dd807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:16.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.542+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f6dec09a890 con 0x7f6df80cf3d0 2024-09-17T03:52:16.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.543+0000 7f6dfcc00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 0x7f6dd807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:16.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.543+0000 7f6dfcc00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 0x7f6dd807b0a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6de4007d60 tx=0x7f6de4007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:16.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.547+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6dec01f030 con 0x7f6df80cf3d0 2024-09-17T03:52:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.764+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6df80cfba0 con 0x7f6df80cf3d0 2024-09-17T03:52:16.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.764+0000 7f6df5600640 1 -- 172.21.15.129:0/3488774149 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f6dec062670 con 0x7f6df80cf3d0 2024-09-17T03:52:16.765 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:16.765 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:16.765 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 msgr2=0x7f6dd807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 0x7f6dd807b0a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6de4007d60 tx=0x7f6de4007cf0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 msgr2=0x7f6df8150da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00b3f0 tx=0x7f6dec00b8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 shutdown_connections 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dd8078b90 0x7f6dd807b0a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6df815c110 0x7f6df815a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6df80d0500 0x7f6df815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 --2- 172.21.15.129:0/3488774149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6df80cf3d0 0x7f6df8150da0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 >> 172.21.15.129:0/3488774149 conn(0x7f6df806d870 msgr2=0x7f6df815cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 shutdown_connections 2024-09-17T03:52:16.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:16.767+0000 7f6dfea00640 1 -- 172.21.15.129:0/3488774149 wait complete. 2024-09-17T03:52:16.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 -- 172.21.15.129:0/1845634687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0b6150 msgr2=0x7f3a2c0ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1845634687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0b6150 0x7f3a2c0ccce0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c074640 tx=0x7f3a2402a820 comp rx=0 tx=0).stop 2024-09-17T03:52:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 -- 172.21.15.129:0/1845634687 shutdown_connections 2024-09-17T03:52:17.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1845634687 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0b6150 0x7f3a2c0ccce0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1845634687 >> v2:172.21.15.181:3300/0 conn(0x7f3a2c0b6cd0 0x7f3a2c0b70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 -- 172.21.15.129:0/1845634687 >> 172.21.15.129:0/1845634687 conn(0x7f3a2c0cec00 msgr2=0x7f3a2c0cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 -- 172.21.15.129:0/1845634687 shutdown_connections 2024-09-17T03:52:17.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.013+0000 7f3a2be00640 1 -- 172.21.15.129:0/1845634687 wait complete. 2024-09-17T03:52:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.015+0000 7f3a2be00640 1 Processor -- start 2024-09-17T03:52:17.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.015+0000 7f3a2be00640 1 -- start start 2024-09-17T03:52:17.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a2c0b6150 0x7f3a2c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a2c0b6cd0 0x7f3a2c1f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c0cd9d0 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c0cd850 con 0x7f3a2c0b6cd0 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3a2c0cdb50 con 0x7f3a2c0b6150 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51706/0 (socket says 172.21.15.129:51706) 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 -- 172.21.15.129:0/1078241063 learned_addr learned my addr 172.21.15.129:0/1078241063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a22000640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a2c0b6cd0 0x7f3a2c1f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2aa00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a2c0b6150 0x7f3a2c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 -- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a2c0b6150 msgr2=0x7f3a2c154260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a2c0b6150 0x7f3a2c154260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 -- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a2c0b6cd0 msgr2=0x7f3a2c1f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a2c0b6cd0 0x7f3a2c1f6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.017+0000 7f3a2b400640 1 -- 172.21.15.129:0/1078241063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c1f7610 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a2b400640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009b30 tx=0x7f3a24032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24037040 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a24042070 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c1f78d0 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a24004280 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.018+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c0c7dd0 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.019+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2c0b11b0 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.021+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a24004420 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.022+0000 7f3a23400640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 0x7f3a0807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.022+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f3a240bbd00 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.022+0000 7f3a2aa00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 0x7f3a0807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.023+0000 7f3a2aa00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 0x7f3a0807b0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006fd0 tx=0x7f3a1c008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.026+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a24084130 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.245+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a2c0ce080 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.246+0000 7f3a23400640 1 -- 172.21.15.129:0/1078241063 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f3a24083a30 con 0x7f3a2c0cfcc0 2024-09-17T03:52:17.246 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:17.246 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:17.246 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:17.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 msgr2=0x7f3a0807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 0x7f3a0807b0d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c006fd0 tx=0x7f3a1c008040 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 msgr2=0x7f3a2c1f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009b30 tx=0x7f3a24032c90 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 shutdown_connections 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a08078bc0 0x7f3a0807b0d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a2c0cfcc0 0x7f3a2c1f70d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a2c0b6cd0 0x7f3a2c1f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 --2- 172.21.15.129:0/1078241063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a2c0b6150 0x7f3a2c154260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 >> 172.21.15.129:0/1078241063 conn(0x7f3a2c0cec00 msgr2=0x7f3a2c0b0920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 shutdown_connections 2024-09-17T03:52:17.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.248+0000 7f3a2be00640 1 -- 172.21.15.129:0/1078241063 wait complete. 2024-09-17T03:52:17.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 -- 172.21.15.129:0/2811135072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958162040 msgr2=0x7f095806cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 --2- 172.21.15.129:0/2811135072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958162040 0x7f095806cde0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f095806a210 tx=0x7f094802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 -- 172.21.15.129:0/2811135072 shutdown_connections 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 --2- 172.21.15.129:0/2811135072 >> v2:172.21.15.181:3300/0 conn(0x7f095806d320 0x7f0958074610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 --2- 172.21.15.129:0/2811135072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0958162040 0x7f095806cde0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 -- 172.21.15.129:0/2811135072 >> 172.21.15.129:0/2811135072 conn(0x7f09580cdcc0 msgr2=0x7f09580ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 -- 172.21.15.129:0/2811135072 shutdown_connections 2024-09-17T03:52:17.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.494+0000 7f095da00640 1 -- 172.21.15.129:0/2811135072 wait complete. 2024-09-17T03:52:17.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.496+0000 7f095da00640 1 Processor -- start 2024-09-17T03:52:17.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.496+0000 7f095da00640 1 -- start start 2024-09-17T03:52:17.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f095806d320 0x7f09580c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09580aef80 0x7f09580af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0958074cd0 con 0x7f09580b4570 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0958074b50 con 0x7f09580aef80 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f095da00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f09580b0310 con 0x7f095806d320 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f094e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f0956a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f095806d320 0x7f09580c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f094e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51726/0 (socket says 172.21.15.129:51726) 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f094e000640 1 -- 172.21.15.129:0/3318940206 learned_addr learned my addr 172.21.15.129:0/3318940206 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.498+0000 7f0957400640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09580aef80 0x7f09580af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 -- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f095806d320 msgr2=0x7f09580c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f095806d320 0x7f09580c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 -- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09580aef80 msgr2=0x7f09580af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09580aef80 0x7f09580af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 -- 172.21.15.129:0/3318940206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09580af8d0 con 0x7f09580b4570 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094e000640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f094000c3a0 tx=0x7f094000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940017070 con 0x7f09580b4570 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09400053c0 con 0x7f09580b4570 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09580ca3b0 con 0x7f09580b4570 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940003850 con 0x7f09580b4570 2024-09-17T03:52:17.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.499+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09580b11d0 con 0x7f09580b4570 2024-09-17T03:52:17.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.500+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09580ca120 con 0x7f09580b4570 2024-09-17T03:52:17.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.502+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f094001a070 con 0x7f09580b4570 2024-09-17T03:52:17.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.503+0000 7f094f400640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 0x7f093407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.503+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f0940099f50 con 0x7f09580b4570 2024-09-17T03:52:17.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.504+0000 7f0956a00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 0x7f093407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.504+0000 7f0956a00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 0x7f093407b0d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0948033000 tx=0x7f0948004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.508+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f095806d730 con 0x7f09580b4570 2024-09-17T03:52:17.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.691+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09580b41e0 con 0x7f09580b4570 2024-09-17T03:52:17.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.693+0000 7f094f400640 1 -- 172.21.15.129:0/3318940206 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16773 (secure 0 0 0) 0x7f0940061dd0 con 0x7f09580b4570 2024-09-17T03:52:17.693 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:17.693 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":56,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:52.164400+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:48.557870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:49.606077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:17.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 msgr2=0x7f093407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 0x7f093407b0d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0948033000 tx=0x7f0948004730 comp rx=0 tx=0).stop 2024-09-17T03:52:17.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 msgr2=0x7f0958153a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f094000c3a0 tx=0x7f094000c860 comp rx=0 tx=0).stop 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 shutdown_connections 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0934078bc0 0x7f093407b0d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09580aef80 0x7f09580af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09580b4570 0x7f0958153a10 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 --2- 172.21.15.129:0/3318940206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f095806d320 0x7f09580c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 >> 172.21.15.129:0/3318940206 conn(0x7f09580cdcc0 msgr2=0x7f0958162040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:17.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 shutdown_connections 2024-09-17T03:52:17.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.695+0000 7f095da00640 1 -- 172.21.15.129:0/3318940206 wait complete. 2024-09-17T03:52:17.722 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:52:17.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:17.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.943+0000 7faa20200640 1 -- 172.21.15.129:0/676526238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa00003a50 msgr2=0x7faa00003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.943+0000 7faa20200640 1 --2- 172.21.15.129:0/676526238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa00003a50 0x7faa00003ec0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c4a0 tx=0x7faa00005830 comp rx=0 tx=0).stop 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.943+0000 7faa20200640 1 -- 172.21.15.129:0/676526238 shutdown_connections 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.943+0000 7faa20200640 1 --2- 172.21.15.129:0/676526238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa00003a50 0x7faa00003ec0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.943+0000 7faa20200640 1 -- 172.21.15.129:0/676526238 >> 172.21.15.129:0/676526238 conn(0x7faa1806d7d0 msgr2=0x7faa180b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.944+0000 7faa20200640 1 -- 172.21.15.129:0/676526238 shutdown_connections 2024-09-17T03:52:17.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.944+0000 7faa20200640 1 -- 172.21.15.129:0/676526238 wait complete. 2024-09-17T03:52:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.946+0000 7faa20200640 1 Processor -- start 2024-09-17T03:52:17.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.946+0000 7faa20200640 1 -- start start 2024-09-17T03:52:17.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa00003a50 0x7faa1815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa1815ed30 0x7faa1815f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa180cd0e0 con 0x7faa181f5d30 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa180ccf60 con 0x7faa1815ed30 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.947+0000 7faa20200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faa180cd260 con 0x7faa00003a50 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51734/0 (socket says 172.21.15.129:51734) 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 -- 172.21.15.129:0/1416274656 learned_addr learned my addr 172.21.15.129:0/1416274656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:17.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1d800640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa00003a50 0x7faa1815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1e200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa1815ed30 0x7faa1815f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 -- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa00003a50 msgr2=0x7faa1815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa00003a50 0x7faa1815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 -- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa1815ed30 msgr2=0x7faa1815f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa1815ed30 0x7faa1815f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.948+0000 7faa1ce00640 1 -- 172.21.15.129:0/1416274656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1815f680 con 0x7faa181f5d30 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa1ce00640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa04003fb0 tx=0x7faa040016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04015070 con 0x7faa181f5d30 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa04011900 con 0x7faa181f5d30 2024-09-17T03:52:17.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04011d50 con 0x7faa181f5d30 2024-09-17T03:52:17.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa18072e30 con 0x7faa181f5d30 2024-09-17T03:52:17.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.949+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa181f6560 con 0x7faa181f5d30 2024-09-17T03:52:17.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.950+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa1809ae30 con 0x7faa181f5d30 2024-09-17T03:52:17.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.951+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faa04018070 con 0x7faa181f5d30 2024-09-17T03:52:17.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.952+0000 7faa15600640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 0x7fa9f807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:17.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.952+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7faa0409b8a0 con 0x7faa181f5d30 2024-09-17T03:52:17.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.953+0000 7faa1d800640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 0x7fa9f807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:17.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.953+0000 7faa1d800640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 0x7fa9f807b0a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa0c033000 tx=0x7faa180b4bb0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:17.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:17.956+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faa1815f680 con 0x7faa181f5d30 2024-09-17T03:52:18.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.173+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa180b43a0 con 0x7faa181f5d30 2024-09-17T03:52:18.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.173+0000 7faa15600640 1 -- 172.21.15.129:0/1416274656 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7faa040637b0 con 0x7faa181f5d30 2024-09-17T03:52:18.174 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:18.174 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:18.174 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 msgr2=0x7fa9f807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 0x7fa9f807b0a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa0c033000 tx=0x7faa180b4bb0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 msgr2=0x7faa18151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faa04003fb0 tx=0x7faa040016f0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 shutdown_connections 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9f8078b90 0x7fa9f807b0a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa1815ed30 0x7faa1815f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa181f5d30 0x7faa18151890 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 --2- 172.21.15.129:0/1416274656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa00003a50 0x7faa1815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 >> 172.21.15.129:0/1416274656 conn(0x7faa1806d7d0 msgr2=0x7faa1815aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:18.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 shutdown_connections 2024-09-17T03:52:18.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.176+0000 7faa20200640 1 -- 172.21.15.129:0/1416274656 wait complete. 2024-09-17T03:52:18.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.425+0000 7f92c9a00640 1 -- 172.21.15.129:0/3749510939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cd240 msgr2=0x7f92c40cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.425+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3749510939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cd240 0x7f92c40cc4c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f92ac009860 tx=0x7f92ac02a5a0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.426+0000 7f92c9a00640 1 -- 172.21.15.129:0/3749510939 shutdown_connections 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.426+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3749510939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cd240 0x7f92c40cc4c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.426+0000 7f92c9a00640 1 -- 172.21.15.129:0/3749510939 >> 172.21.15.129:0/3749510939 conn(0x7f92c40b1cb0 msgr2=0x7f92c40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.426+0000 7f92c9a00640 1 -- 172.21.15.129:0/3749510939 shutdown_connections 2024-09-17T03:52:18.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.426+0000 7f92c9a00640 1 -- 172.21.15.129:0/3749510939 wait complete. 2024-09-17T03:52:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.428+0000 7f92c9a00640 1 Processor -- start 2024-09-17T03:52:18.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.428+0000 7f92c9a00640 1 -- start start 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cf6f0 0x7f92c40ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c40af0c0 0x7f92c4161300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92c40cf340 con 0x7f92c40ce020 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92c40cf1c0 con 0x7f92c40af0c0 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.429+0000 7f92c9a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f92c40701b0 con 0x7f92c40cf6f0 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51738/0 (socket says 172.21.15.129:51738) 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 -- 172.21.15.129:0/3871519488 learned_addr learned my addr 172.21.15.129:0/3871519488 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:18.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92bbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cf6f0 0x7f92c40ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c3400640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c40af0c0 0x7f92c4161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 -- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cf6f0 msgr2=0x7f92c40ae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cf6f0 0x7f92c40ae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 -- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c40af0c0 msgr2=0x7f92c4161300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c40af0c0 0x7f92c4161300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 -- 172.21.15.129:0/3871519488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c409bc20 con 0x7f92c40ce020 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.430+0000 7f92c2a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c400 tx=0x7f92b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.431+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4011850 con 0x7f92c40ce020 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.431+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92b4017070 con 0x7f92c40ce020 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.431+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c409bee0 con 0x7f92c40ce020 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.431+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4011da0 con 0x7f92c40ce020 2024-09-17T03:52:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.431+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c409c4a0 con 0x7f92c40ce020 2024-09-17T03:52:18.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.432+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c40cc4c0 con 0x7f92c40ce020 2024-09-17T03:52:18.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.433+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f92b401c460 con 0x7f92c40ce020 2024-09-17T03:52:18.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.434+0000 7f92baa00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 0x7f92a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.434+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f92b409aa20 con 0x7f92c40ce020 2024-09-17T03:52:18.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.435+0000 7f92bbe00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 0x7f92a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.435+0000 7f92bbe00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 0x7f92a007b0a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f92ac034000 tx=0x7f92ac004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:18.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.438+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92b4062810 con 0x7f92c40ce020 2024-09-17T03:52:18.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.658+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92c415e910 con 0x7f92c40ce020 2024-09-17T03:52:18.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.659+0000 7f92baa00640 1 -- 172.21.15.129:0/3871519488 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5232 (secure 0 0 0) 0x7f92b4067240 con 0x7f92c40ce020 2024-09-17T03:52:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:18.659 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":35,"btime":"2024-09-17T03:51:54:177743+0000","default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:51:50.646668+0000","modified":"2024-09-17T03:51:53.666499+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5550},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5550":{"gid":5550,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/2835182709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2835182709},{"type":"v1","addr":"172.21.15.129:6834","nonce":2835182709}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5550,"qdb_cluster":[5550]},"id":4}]} 2024-09-17T03:52:18.659 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 35 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 msgr2=0x7f92a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 0x7f92a007b0a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f92ac034000 tx=0x7f92ac004730 comp rx=0 tx=0).stop 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 msgr2=0x7f92c41f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f92b400c400 tx=0x7f92b400c8c0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 shutdown_connections 2024-09-17T03:52:18.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a0078b90 0x7f92a007b0a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c40af0c0 0x7f92c4161300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c40cf6f0 0x7f92c40ae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 --2- 172.21.15.129:0/3871519488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c40ce020 0x7f92c41f5f50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 >> 172.21.15.129:0/3871519488 conn(0x7f92c40b1cb0 msgr2=0x7f92c4072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 shutdown_connections 2024-09-17T03:52:18.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.661+0000 7f92c9a00640 1 -- 172.21.15.129:0/3871519488 wait complete. 2024-09-17T03:52:18.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.906+0000 7fea14e00640 1 -- 172.21.15.129:0/3477976284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10070440 msgr2=0x7fea100b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.906+0000 7fea14e00640 1 --2- 172.21.15.129:0/3477976284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10070440 0x7fea100b3090 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fea08005fd0 tx=0x7fea0802a980 comp rx=0 tx=0).stop 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 -- 172.21.15.129:0/3477976284 shutdown_connections 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 --2- 172.21.15.129:0/3477976284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10070440 0x7fea100b3090 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 --2- 172.21.15.129:0/3477976284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10073170 0x7fea1015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 -- 172.21.15.129:0/3477976284 >> 172.21.15.129:0/3477976284 conn(0x7fea100b1cb0 msgr2=0x7fea100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:18.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 -- 172.21.15.129:0/3477976284 shutdown_connections 2024-09-17T03:52:18.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.907+0000 7fea14e00640 1 -- 172.21.15.129:0/3477976284 wait complete. 2024-09-17T03:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.909+0000 7fea14e00640 1 Processor -- start 2024-09-17T03:52:18.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.909+0000 7fea14e00640 1 -- start start 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.910+0000 7fea14e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10070030 0x7fea101f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.910+0000 7fea14e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea10070440 0x7fea101f9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.910+0000 7fea14e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea14e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea100b3b00 con 0x7fea10073170 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea14e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fea100b3980 con 0x7fea10070440 2024-09-17T03:52:18.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea14e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fea100b3c80 con 0x7fea10070030 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51752/0 (socket says 172.21.15.129:51752) 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 -- 172.21.15.129:0/2476007331 learned_addr learned my addr 172.21.15.129:0/2476007331 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0de00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10070030 0x7fea101f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0d400640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea10070440 0x7fea101f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 -- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10070030 msgr2=0x7fea101f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10070030 0x7fea101f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 -- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea10070440 msgr2=0x7fea101f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea10070440 0x7fea101f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.911+0000 7fea0e800640 1 -- 172.21.15.129:0/2476007331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea100cf320 con 0x7fea10073170 2024-09-17T03:52:18.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea0e800640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fea08009d90 tx=0x7fea08003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08034040 con 0x7fea10073170 2024-09-17T03:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea080045e0 con 0x7fea10073170 2024-09-17T03:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea101f6aa0 con 0x7fea10073170 2024-09-17T03:52:18.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea08045480 con 0x7fea10073170 2024-09-17T03:52:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.912+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea100cf5e0 con 0x7fea10073170 2024-09-17T03:52:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.913+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea100b41a0 con 0x7fea10073170 2024-09-17T03:52:18.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.914+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fea08037020 con 0x7fea10073170 2024-09-17T03:52:18.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.915+0000 7fea06000640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 0x7fe9f007b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:18.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.915+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fea080bc6d0 con 0x7fea10073170 2024-09-17T03:52:18.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.916+0000 7fea0de00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 0x7fe9f007b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:18.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.916+0000 7fea0de00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 0x7fe9f007b090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea00005fd0 tx=0x7fea0000a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:18.920+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fea08084430 con 0x7fea10073170 2024-09-17T03:52:19.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.104+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea100c7f90 con 0x7fea10073170 2024-09-17T03:52:19.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.105+0000 7fea06000640 1 -- 172.21.15.129:0/2476007331 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16773 (secure 0 0 0) 0x7fea08088e60 con 0x7fea10073170 2024-09-17T03:52:19.105 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:19.106 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":56,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:51:52.164400+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:51:48.557870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-17T03:51:49.606077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 msgr2=0x7fe9f007b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 0x7fe9f007b090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fea00005fd0 tx=0x7fea0000a250 comp rx=0 tx=0).stop 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 msgr2=0x7fea101f9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fea08009d90 tx=0x7fea08003f90 comp rx=0 tx=0).stop 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 shutdown_connections 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9f0078b80 0x7fe9f007b090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea10073170 0x7fea101f9650 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea10070440 0x7fea101f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 --2- 172.21.15.129:0/2476007331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea10070030 0x7fea101f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 >> 172.21.15.129:0/2476007331 conn(0x7fea100b1cb0 msgr2=0x7fea1015ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:19.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 shutdown_connections 2024-09-17T03:52:19.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.108+0000 7fea14e00640 1 -- 172.21.15.129:0/2476007331 wait complete. 2024-09-17T03:52:19.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:52:19.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 -- 172.21.15.129:0/2149261793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40704e0 msgr2=0x7fedc40cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 --2- 172.21.15.129:0/2149261793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40704e0 0x7fedc40cc830 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fedc40c71a0 tx=0x7fedbc02aa30 comp rx=0 tx=0).stop 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 -- 172.21.15.129:0/2149261793 shutdown_connections 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 --2- 172.21.15.129:0/2149261793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40704e0 0x7fedc40cc830 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 --2- 172.21.15.129:0/2149261793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40cad30 0x7fedc406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 --2- 172.21.15.129:0/2149261793 >> v2:172.21.15.181:3300/0 conn(0x7fedc40ca800 0x7fedc40cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 -- 172.21.15.129:0/2149261793 >> 172.21.15.129:0/2149261793 conn(0x7fedc40b1cb0 msgr2=0x7fedc40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 -- 172.21.15.129:0/2149261793 shutdown_connections 2024-09-17T03:52:19.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.351+0000 7fedcc200640 1 -- 172.21.15.129:0/2149261793 wait complete. 2024-09-17T03:52:19.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.353+0000 7fedcc200640 1 Processor -- start 2024-09-17T03:52:19.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.354+0000 7fedcc200640 1 -- start start 2024-09-17T03:52:19.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedc40704e0 0x7fedc40b3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 0x7fedc40c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fedc40ccf80 con 0x7fedc40cad30 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fedc40cec60 con 0x7fedc40704e0 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedcc200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fedc40cd100 con 0x7fedc40ca800 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedca200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedca200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51764/0 (socket says 172.21.15.129:51764) 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.355+0000 7fedca200640 1 -- 172.21.15.129:0/2153422084 learned_addr learned my addr 172.21.15.129:0/2153422084 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc9800640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedc40704e0 0x7fedc40b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc8e00640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 0x7fedc40c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 -- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 msgr2=0x7fedc40c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 0x7fedc40c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 -- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedc40704e0 msgr2=0x7fedc40b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedc40704e0 0x7fedc40b3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 -- 172.21.15.129:0/2153422084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc41f8c90 con 0x7fedc40cad30 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc8e00640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 0x7fedc40c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedca200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009d60 tx=0x7fedbc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc033050 con 0x7fedc40cad30 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fedbc004450 con 0x7fedc40cad30 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc41f8f70 con 0x7fedc40cad30 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.356+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc038450 con 0x7fedc40cad30 2024-09-17T03:52:19.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.357+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc41f9530 con 0x7fedc40cad30 2024-09-17T03:52:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.358+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc41f8a00 con 0x7fedc40cad30 2024-09-17T03:52:19.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.359+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fedbc036070 con 0x7fedc40cad30 2024-09-17T03:52:19.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.360+0000 7fedc1600640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 0x7feda407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.360+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7fedbc0bc1e0 con 0x7fedc40cad30 2024-09-17T03:52:19.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.360+0000 7fedc9800640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 0x7feda407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.361+0000 7fedc9800640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 0x7feda407b0a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fedc40cb250 tx=0x7fedb800b390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:19.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.364+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fedbc084590 con 0x7fedc40cad30 2024-09-17T03:52:19.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.580+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fedc40cd690 con 0x7fedc40cad30 2024-09-17T03:52:19.581 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:52:19.581+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:52:19.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.583+0000 7fedc1600640 1 -- 172.21.15.129:0/2153422084 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7fedbc083e90 con 0x7fedc40cad30 2024-09-17T03:52:19.583 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.586+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 msgr2=0x7feda407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.586+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 0x7feda407b0a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fedc40cb250 tx=0x7fedb800b390 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.586+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 msgr2=0x7fedc406ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.586+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009d60 tx=0x7fedbc003f90 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 shutdown_connections 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feda4078b90 0x7feda407b0a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedc40cad30 0x7fedc406ee90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedc40ca800 0x7fedc40c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 --2- 172.21.15.129:0/2153422084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedc40704e0 0x7fedc40b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 >> 172.21.15.129:0/2153422084 conn(0x7fedc40b1cb0 msgr2=0x7fedc4074400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 shutdown_connections 2024-09-17T03:52:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.587+0000 7fedcc200640 1 -- 172.21.15.129:0/2153422084 wait complete. 2024-09-17T03:52:19.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:52:19.726 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:52:19.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 -- 172.21.15.129:0/4048002700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 msgr2=0x7f468c1600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 --2- 172.21.15.129:0/4048002700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c1600f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f468c0d0ff0 tx=0x7f468402aa10 comp rx=0 tx=0).stop 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 -- 172.21.15.129:0/4048002700 shutdown_connections 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 --2- 172.21.15.129:0/4048002700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c1600f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 --2- 172.21.15.129:0/4048002700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c0b59a0 0x7f468c15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 --2- 172.21.15.129:0/4048002700 >> v2:172.21.15.181:3300/0 conn(0x7f468c153780 0x7f468c0b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 -- 172.21.15.129:0/4048002700 >> 172.21.15.129:0/4048002700 conn(0x7f468c06d870 msgr2=0x7f468c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 -- 172.21.15.129:0/4048002700 shutdown_connections 2024-09-17T03:52:19.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.843+0000 7f4692a00640 1 -- 172.21.15.129:0/4048002700 wait complete. 2024-09-17T03:52:19.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.845+0000 7f4692a00640 1 Processor -- start 2024-09-17T03:52:19.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.846+0000 7f4692a00640 1 -- start start 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c0b59a0 0x7f468c1f8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c153780 0x7f468c0ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f468c1609c0 con 0x7f468c15d2e0 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f468c160840 con 0x7f468c0b59a0 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f468c160b40 con 0x7f468c153780 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51782/0 (socket says 172.21.15.129:51782) 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.847+0000 7f4692000640 1 -- 172.21.15.129:0/98321683 learned_addr learned my addr 172.21.15.129:0/98321683 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4691600640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c0b59a0 0x7f468c1f8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4690c00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c153780 0x7f468c0ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 -- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c153780 msgr2=0x7f468c0ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c153780 0x7f468c0ceb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 -- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c0b59a0 msgr2=0x7f468c1f8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c0b59a0 0x7f468c1f8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 -- 172.21.15.129:0/98321683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f468c1f9e60 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.848+0000 7f4692000640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4684009d60 tx=0x7f4684003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.849+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684033050 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.849+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46840042d0 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.849+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684004720 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.849+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f468c1f95c0 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.849+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f468c1f9b80 con 0x7f468c15d2e0 2024-09-17T03:52:19.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.850+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f468c0b5460 con 0x7f468c15d2e0 2024-09-17T03:52:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.851+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4684036070 con 0x7f468c15d2e0 2024-09-17T03:52:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.852+0000 7f4689600640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 0x7f466c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.852+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7f46840bbd20 con 0x7f468c15d2e0 2024-09-17T03:52:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.852+0000 7f4691600640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 0x7f466c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:19.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.853+0000 7f4691600640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 0x7f466c07b0a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f468000bb50 tx=0x7f468000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:19.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:19.856+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4684049020 con 0x7f468c15d2e0 2024-09-17T03:52:20.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.077+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f468c1511b0 con 0x7f468c15d2e0 2024-09-17T03:52:20.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.585+0000 7f4689600640 1 -- 172.21.15.129:0/98321683 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7f4684083a20 con 0x7f468c15d2e0 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.589+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 msgr2=0x7f466c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.589+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 0x7f466c07b0a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f468000bb50 tx=0x7f468000b2e0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.589+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 msgr2=0x7f468c0c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.589+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f4684009d60 tx=0x7f4684003f10 comp rx=0 tx=0).stop 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 shutdown_connections 2024-09-17T03:52:20.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f466c078b90 0x7f466c07b0a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f468c15d2e0 0x7f468c0c92d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f468c153780 0x7f468c0ceb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 --2- 172.21.15.129:0/98321683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f468c0b59a0 0x7f468c1f8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 >> 172.21.15.129:0/98321683 conn(0x7f468c06d870 msgr2=0x7f468c152bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 shutdown_connections 2024-09-17T03:52:20.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.590+0000 7f4692a00640 1 -- 172.21.15.129:0/98321683 wait complete. 2024-09-17T03:52:20.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:52:20.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 -- 172.21.15.129:0/3066924685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 msgr2=0x7f811c0d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 --2- 172.21.15.129:0/3066924685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c0d0290 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f8104009830 tx=0x7f810402a510 comp rx=0 tx=0).stop 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 -- 172.21.15.129:0/3066924685 shutdown_connections 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 --2- 172.21.15.129:0/3066924685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c0d0290 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 -- 172.21.15.129:0/3066924685 >> 172.21.15.129:0/3066924685 conn(0x7f811c06cfc0 msgr2=0x7f811c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 -- 172.21.15.129:0/3066924685 shutdown_connections 2024-09-17T03:52:20.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.841+0000 7f811b200640 1 -- 172.21.15.129:0/3066924685 wait complete. 2024-09-17T03:52:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.843+0000 7f811b200640 1 Processor -- start 2024-09-17T03:52:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.844+0000 7f811b200640 1 -- start start 2024-09-17T03:52:20.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 0x7f811c0b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f811c0d07d0 0x7f811c0a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f811c074b20 con 0x7f811c142b80 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f811c0749a0 con 0x7f811c06f730 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f811c074ca0 con 0x7f811c0d07d0 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51806/0 (socket says 172.21.15.129:51806) 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.845+0000 7f811a800640 1 -- 172.21.15.129:0/1053361829 learned_addr learned my addr 172.21.15.129:0/1053361829 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f8119e00640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 0x7f811c0b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f8111400640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f811c0d07d0 0x7f811c0a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 -- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f811c0d07d0 msgr2=0x7f811c0a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f811c0d07d0 0x7f811c0a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 -- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 msgr2=0x7f811c0b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 0x7f811c0b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 -- 172.21.15.129:0/1053361829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f811c1f9e00 con 0x7f811c142b80 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f8119e00640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 0x7f811c0b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811a800640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f811400ba10 tx=0x7f811400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8114019070 con 0x7f811c142b80 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f811400cc30 con 0x7f811c142b80 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.846+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f811c0b4a90 con 0x7f811c142b80 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.847+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811400ce60 con 0x7f811c142b80 2024-09-17T03:52:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.847+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f811c0a9a60 con 0x7f811c142b80 2024-09-17T03:52:20.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.848+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f811c1f9b70 con 0x7f811c142b80 2024-09-17T03:52:20.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.849+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f811401c070 con 0x7f811c142b80 2024-09-17T03:52:20.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.850+0000 7f8112a00640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 0x7f80f807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:20.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.850+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5795+0+0 (secure 0 0 0) 0x7f811409a9b0 con 0x7f811c142b80 2024-09-17T03:52:20.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.850+0000 7f8119e00640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 0x7f80f807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:20.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.851+0000 7f8119e00640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 0x7f80f807b0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f811c15cb40 tx=0x7f810c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:20.854+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8114062d80 con 0x7f811c142b80 2024-09-17T03:52:21.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.039+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f811c0d0290 con 0x7f811c142b80 2024-09-17T03:52:21.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.895+0000 7f8112a00640 1 -- 172.21.15.129:0/1053361829 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) ==== 181+0+0 (secure 0 0 0) 0x7f8114062680 con 0x7f811c142b80 2024-09-17T03:52:21.895 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 msgr2=0x7f80f807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 0x7f80f807b0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f811c15cb40 tx=0x7f810c009040 comp rx=0 tx=0).stop 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 msgr2=0x7f811c13e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f811400ba10 tx=0x7f811400bed0 comp rx=0 tx=0).stop 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 shutdown_connections 2024-09-17T03:52:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f80f8078b90 0x7f80f807b0a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f811c142b80 0x7f811c13e890 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f811c0d07d0 0x7f811c0a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 --2- 172.21.15.129:0/1053361829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f811c06f730 0x7f811c0b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 >> 172.21.15.129:0/1053361829 conn(0x7f811c06cfc0 msgr2=0x7f811c144f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 shutdown_connections 2024-09-17T03:52:21.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:21.899+0000 7f811b200640 1 -- 172.21.15.129:0/1053361829 wait complete. 2024-09-17T03:52:21.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:52:22.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 -- 172.21.15.129:0/1501731409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 msgr2=0x7f6b9c0d0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:22.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1501731409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0d0270 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009830 tx=0x7f6b9002a5b0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 -- 172.21.15.129:0/1501731409 shutdown_connections 2024-09-17T03:52:22.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1501731409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0d0270 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 -- 172.21.15.129:0/1501731409 >> 172.21.15.129:0/1501731409 conn(0x7f6b9c06cfc0 msgr2=0x7f6b9c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:22.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 -- 172.21.15.129:0/1501731409 shutdown_connections 2024-09-17T03:52:22.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.154+0000 7f6ba2400640 1 -- 172.21.15.129:0/1501731409 wait complete. 2024-09-17T03:52:22.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.156+0000 7f6ba2400640 1 Processor -- start 2024-09-17T03:52:22.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.156+0000 7f6ba2400640 1 -- start start 2024-09-17T03:52:22.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 0x7f6b9c0cdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c073e30 con 0x7f6b9c15f650 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c073cb0 con 0x7f6b9c0b42d0 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba2400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c073fb0 con 0x7f6b9c0d07b0 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba1a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba1a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51812/0 (socket says 172.21.15.129:51812) 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba1a00640 1 -- 172.21.15.129:0/1222716448 learned_addr learned my addr 172.21.15.129:0/1222716448 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6ba1000640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.158+0000 7f6b9be00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 0x7f6b9c0cdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 -- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 msgr2=0x7f6b9c0cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 0x7f6b9c0cdbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 -- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 msgr2=0x7f6b9c0cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 -- 172.21.15.129:0/1222716448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c155da0 con 0x7f6b9c15f650 2024-09-17T03:52:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6b9be00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 0x7f6b9c0cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:22.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba1a00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400caa0 tx=0x7f6b9400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94016020 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b9400acb0 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b94012680 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c156080 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.159+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c158c60 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.161+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c0d0270 con 0x7f6b9c15f650 2024-09-17T03:52:22.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.162+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6b94019020 con 0x7f6b9c15f650 2024-09-17T03:52:22.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.163+0000 7f6b98c00640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 0x7f6b7c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.163+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5357+0+0 (secure 0 0 0) 0x7f6b9409b070 con 0x7f6b9c15f650 2024-09-17T03:52:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.163+0000 7f6ba1000640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 0x7f6b7c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:22.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.164+0000 7f6ba1000640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 0x7f6b7c07b0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b90034000 tx=0x7f6b90004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:22.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.168+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6b94020020 con 0x7f6b9c15f650 2024-09-17T03:52:22.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.357+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6b9c0af870 con 0x7f6b9c15f650 2024-09-17T03:52:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.902+0000 7f6b98c00640 1 -- 172.21.15.129:0/1222716448 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) ==== 169+0+0 (secure 0 0 0) 0x7f6b940636a0 con 0x7f6b9c15f650 2024-09-17T03:52:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 msgr2=0x7f6b7c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 0x7f6b7c07b0d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b90034000 tx=0x7f6b90004730 comp rx=0 tx=0).stop 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 msgr2=0x7f6b9c1fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400caa0 tx=0x7f6b9400cf60 comp rx=0 tx=0).stop 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 shutdown_connections 2024-09-17T03:52:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b7c078bc0 0x7f6b7c07b0d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b9c15f650 0x7f6b9c1fd280 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b9c0d07b0 0x7f6b9c0cdbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 --2- 172.21.15.129:0/1222716448 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b9c0b42d0 0x7f6b9c0cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 >> 172.21.15.129:0/1222716448 conn(0x7f6b9c06cfc0 msgr2=0x7f6b9c1619e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 shutdown_connections 2024-09-17T03:52:22.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:22.907+0000 7f6ba2400640 1 -- 172.21.15.129:0/1222716448 wait complete. 2024-09-17T03:52:22.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:52:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 -- 172.21.15.129:0/1575199122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac153780 msgr2=0x7fe9ac0b5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 --2- 172.21.15.129:0/1575199122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac153780 0x7fe9ac0b5460 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0b6020 tx=0x7fe99c02a560 comp rx=0 tx=0).stop 2024-09-17T03:52:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 -- 172.21.15.129:0/1575199122 shutdown_connections 2024-09-17T03:52:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 --2- 172.21.15.129:0/1575199122 >> v2:172.21.15.181:3300/0 conn(0x7fe9ac15d2e0 0x7fe9ac1600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 --2- 172.21.15.129:0/1575199122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 --2- 172.21.15.129:0/1575199122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac153780 0x7fe9ac0b5460 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 -- 172.21.15.129:0/1575199122 >> 172.21.15.129:0/1575199122 conn(0x7fe9ac06d870 msgr2=0x7fe9ac06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 -- 172.21.15.129:0/1575199122 shutdown_connections 2024-09-17T03:52:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.160+0000 7fe9abe00640 1 -- 172.21.15.129:0/1575199122 wait complete. 2024-09-17T03:52:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.162+0000 7fe9abe00640 1 Processor -- start 2024-09-17T03:52:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.162+0000 7fe9abe00640 1 -- start start 2024-09-17T03:52:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac0ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 0x7fe9ac0afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1609c0 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac160840 con 0x7fe9ac153780 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9abe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ac160b40 con 0x7fe9ac0b59a0 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51830/0 (socket says 172.21.15.129:51830) 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.164+0000 7fe9ab400640 1 -- 172.21.15.129:0/3267759182 learned_addr learned my addr 172.21.15.129:0/3267759182 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9aaa00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac0ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9a2000640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 0x7fe9ac0afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 -- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 msgr2=0x7fe9ac0ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac0ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 -- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 msgr2=0x7fe9ac0afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 0x7fe9ac0afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 -- 172.21.15.129:0/3267759182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac0b0810 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9a2000640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 0x7fe9ac0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9aaa00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9ab400640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400e960 tx=0x7fe9a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4004d20 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9a400cac0 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ac153d30 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4005020 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.165+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ac1542f0 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.166+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ac0b0580 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.168+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe9a40051c0 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.169+0000 7fe9a3400640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 0x7fe98807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.169+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fe9a409b390 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.169+0000 7fe9aaa00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 0x7fe98807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.170+0000 7fe9aaa00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 0x7fe98807b0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe99c033000 tx=0x7fe99c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:23.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.173+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe9a4016030 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.361+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7fe9ac0b5460 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.365+0000 7fe9a3400640 1 -- 172.21.15.129:0/3267759182 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v40) ==== 71+0+0 (secure 0 0 0) 0x7fe9a4063ac0 con 0x7fe9ac15d2e0 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.367+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 msgr2=0x7fe98807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.367+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 0x7fe98807b0d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe99c033000 tx=0x7fe99c004730 comp rx=0 tx=0).stop 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.367+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 msgr2=0x7fe9ac150730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.367+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400e960 tx=0x7fe9a400ee20 comp rx=0 tx=0).stop 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 shutdown_connections 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe988078bc0 0x7fe98807b0d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ac15d2e0 0x7fe9ac150730 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9ac153780 0x7fe9ac0afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 --2- 172.21.15.129:0/3267759182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ac0b59a0 0x7fe9ac0ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 >> 172.21.15.129:0/3267759182 conn(0x7fe9ac06d870 msgr2=0x7fe9ac15dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 shutdown_connections 2024-09-17T03:52:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.368+0000 7fe9abe00640 1 -- 172.21.15.129:0/3267759182 wait complete. 2024-09-17T03:52:23.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented' 2024-09-17T03:52:23.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 -- 172.21.15.129:0/4286546841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0af700 msgr2=0x7fa41c0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 --2- 172.21.15.129:0/4286546841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0af700 0x7fa41c0caa80 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa40c009830 tx=0x7fa40c02a5b0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 -- 172.21.15.129:0/4286546841 shutdown_connections 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 --2- 172.21.15.129:0/4286546841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0af700 0x7fa41c0caa80 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 -- 172.21.15.129:0/4286546841 >> 172.21.15.129:0/4286546841 conn(0x7fa41c06d870 msgr2=0x7fa41c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 -- 172.21.15.129:0/4286546841 shutdown_connections 2024-09-17T03:52:23.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.619+0000 7fa41b200640 1 -- 172.21.15.129:0/4286546841 wait complete. 2024-09-17T03:52:23.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.621+0000 7fa41b200640 1 Processor -- start 2024-09-17T03:52:23.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.621+0000 7fa41b200640 1 -- start start 2024-09-17T03:52:23.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa41c0735d0 0x7fa41c1544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa41c0af700 0x7fa41c151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c15d4b0 con 0x7fa41c0b54d0 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa41c15d330 con 0x7fa41c0735d0 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa41c15d630 con 0x7fa41c0af700 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51852/0 (socket says 172.21.15.129:51852) 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa41a800640 1 -- 172.21.15.129:0/1880090961 learned_addr learned my addr 172.21.15.129:0/1880090961 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa419e00640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa41c0735d0 0x7fa41c1544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.623+0000 7fa413e00640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa41c0af700 0x7fa41c151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 -- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa41c0af700 msgr2=0x7fa41c151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa41c0af700 0x7fa41c151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 -- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa41c0735d0 msgr2=0x7fa41c1544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa41c0735d0 0x7fa41c1544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 -- 172.21.15.129:0/1880090961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa41c0afc60 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41a800640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c700 tx=0x7fa41400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414003b70 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa414003d10 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41400aab0 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa41c0aff40 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.624+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa41c0b0500 con 0x7fa41c0b54d0 2024-09-17T03:52:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.625+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa41c154d40 con 0x7fa41c0b54d0 2024-09-17T03:52:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.627+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa41400ac50 con 0x7fa41c0b54d0 2024-09-17T03:52:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.627+0000 7fa412000640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 0x7fa3f807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.628+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fa41409a760 con 0x7fa41c0b54d0 2024-09-17T03:52:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.628+0000 7fa419e00640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 0x7fa3f807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.629+0000 7fa419e00640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 0x7fa3f807b0a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa40c034000 tx=0x7fa40c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:23.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.632+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa4140628e0 con 0x7fa41c0b54d0 2024-09-17T03:52:23.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.887+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]} v 0) -- 0x7fa41c0caa80 con 0x7fa41c0b54d0 2024-09-17T03:52:23.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.903+0000 7fa412000640 1 -- 172.21.15.129:0/1880090961 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_fragmented"]}]=0 v168) ==== 144+0+0 (secure 0 0 0) 0x7fa414067310 con 0x7fa41c0b54d0 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 msgr2=0x7fa3f807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 0x7fa3f807b0a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa40c034000 tx=0x7fa40c004730 comp rx=0 tx=0).stop 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 msgr2=0x7fa41c151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fa41400c700 tx=0x7fa41400cbc0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 shutdown_connections 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3f8078b90 0x7fa3f807b0a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa41c0b54d0 0x7fa41c151d40 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa41c0af700 0x7fa41c151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 --2- 172.21.15.129:0/1880090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa41c0735d0 0x7fa41c1544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:23.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 >> 172.21.15.129:0/1880090961 conn(0x7fa41c06d870 msgr2=0x7fa41c0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:23.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 shutdown_connections 2024-09-17T03:52:23.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:23.906+0000 7fa41b200640 1 -- 172.21.15.129:0/1880090961 wait complete. 2024-09-17T03:52:23.932 INFO:tasks.cephfs_test_runner:That fragmentation (forced) will allow more entries to be created. ... ok 2024-09-17T03:52:23.932 INFO:tasks.cephfs_test_runner:Starting test: test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) That unlinking fails when the stray directory fragment becomes too 2024-09-17T03:52:23.932 INFO:tasks.cephfs_test_runner:test_dirfrag_limit_strays (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:52:23.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:24.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 -- 172.21.15.129:0/3322337847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 msgr2=0x7f8c68074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 --2- 172.21.15.129:0/3322337847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c68074640 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8c680b4b40 tx=0x7f8c5802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 -- 172.21.15.129:0/3322337847 shutdown_connections 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 --2- 172.21.15.129:0/3322337847 >> v2:172.21.15.181:3300/0 conn(0x7f8c68070440 0x7f8c68074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 --2- 172.21.15.129:0/3322337847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c68074640 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 -- 172.21.15.129:0/3322337847 >> 172.21.15.129:0/3322337847 conn(0x7f8c680b1cb0 msgr2=0x7f8c680b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 -- 172.21.15.129:0/3322337847 shutdown_connections 2024-09-17T03:52:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.153+0000 7f8c70800640 1 -- 172.21.15.129:0/3322337847 wait complete. 2024-09-17T03:52:24.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.156+0000 7f8c70800640 1 Processor -- start 2024-09-17T03:52:24.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.156+0000 7f8c70800640 1 -- start start 2024-09-17T03:52:24.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c68070440 0x7f8c6814c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 0x7f8c6809d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c680b39f0 con 0x7f8c68070030 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8c680b3870 con 0x7f8c68073170 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c70800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8c680b3b70 con 0x7f8c68070440 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c6de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c6de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59562/0 (socket says 172.21.15.129:59562) 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.157+0000 7f8c6de00640 1 -- 172.21.15.129:0/4271577286 learned_addr learned my addr 172.21.15.129:0/4271577286 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6d400640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c68070440 0x7f8c6814c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6e800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 0x7f8c6809d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 -- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c68070440 msgr2=0x7f8c6814c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c68070440 0x7f8c6814c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 -- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 msgr2=0x7f8c6809d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 0x7f8c6809d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 -- 172.21.15.129:0/4271577286 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c6814c930 con 0x7f8c68070030 2024-09-17T03:52:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6e800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 0x7f8c6809d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c6de00640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00c3a0 tx=0x7f8c5c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c017070 con 0x7f8c68070030 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c5c0053c0 con 0x7f8c68070030 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c680c8820 con 0x7f8c68070030 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.158+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c003850 con 0x7f8c68070030 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.159+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c680c8b00 con 0x7f8c68070030 2024-09-17T03:52:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.160+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c680b4090 con 0x7f8c68070030 2024-09-17T03:52:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.161+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8c5c01a070 con 0x7f8c68070030 2024-09-17T03:52:24.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.162+0000 7f8c62000640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 0x7f8c4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.163+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f8c5c099d60 con 0x7f8c68070030 2024-09-17T03:52:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.163+0000 7f8c6d400640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 0x7f8c4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.164+0000 7f8c6d400640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 0x7f8c4c07b0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8c58033000 tx=0x7f8c58004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.167+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8c5c061e40 con 0x7f8c68070030 2024-09-17T03:52:24.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.351+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c680af8b0 con 0x7f8c68070030 2024-09-17T03:52:24.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.352+0000 7f8c62000640 1 -- 172.21.15.129:0/4271577286 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) ==== 74+0+13082 (secure 0 0 0) 0x7f8c5c066870 con 0x7f8c68070030 2024-09-17T03:52:24.352 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:24.353 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":59,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:52:22.897257+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/2835182709":"2024-09-18T03:52:19.582430+0000","172.21.15.129:6832/2835182709":"2024-09-18T03:52:19.582430+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 msgr2=0x7f8c4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 0x7f8c4c07b0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8c58033000 tx=0x7f8c58004730 comp rx=0 tx=0).stop 2024-09-17T03:52:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 msgr2=0x7f8c680ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00c3a0 tx=0x7f8c5c00c860 comp rx=0 tx=0).stop 2024-09-17T03:52:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 shutdown_connections 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c4c078b90 0x7f8c4c07b0a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c68073170 0x7f8c6809d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c68070440 0x7f8c6814c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 --2- 172.21.15.129:0/4271577286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c68070030 0x7f8c680ca0e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 >> 172.21.15.129:0/4271577286 conn(0x7f8c680b1cb0 msgr2=0x7f8c680ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 shutdown_connections 2024-09-17T03:52:24.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.355+0000 7f8c70800640 1 -- 172.21.15.129:0/4271577286 wait complete. 2024-09-17T03:52:24.382 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:52:24.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:52:24.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 -- 172.21.15.129:0/3994829230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40ca7b0 msgr2=0x7f16b40cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 --2- 172.21.15.129:0/3994829230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40ca7b0 0x7f16b40cabc0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f16ac005fd0 tx=0x7f16ac02a780 comp rx=0 tx=0).stop 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 -- 172.21.15.129:0/3994829230 shutdown_connections 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 --2- 172.21.15.129:0/3994829230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40ca7b0 0x7f16b40cabc0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 --2- 172.21.15.129:0/3994829230 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40b56a0 0x7f16b40704b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 -- 172.21.15.129:0/3994829230 >> 172.21.15.129:0/3994829230 conn(0x7f16b40d0160 msgr2=0x7f16b40d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 -- 172.21.15.129:0/3994829230 shutdown_connections 2024-09-17T03:52:24.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.606+0000 7f16bac00640 1 -- 172.21.15.129:0/3994829230 wait complete. 2024-09-17T03:52:24.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.609+0000 7f16bac00640 1 Processor -- start 2024-09-17T03:52:24.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.609+0000 7f16bac00640 1 -- start start 2024-09-17T03:52:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b40b56a0 0x7f16b4156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40ca7b0 0x7f16b409bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40cb700 con 0x7f16b40cc640 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40cb580 con 0x7f16b40b56a0 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.610+0000 7f16bac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f16b40cb880 con 0x7f16b40ca7b0 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59590/0 (socket says 172.21.15.129:59590) 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 -- 172.21.15.129:0/3013872878 learned_addr learned my addr 172.21.15.129:0/3013872878 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16ab400640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40ca7b0 0x7f16b409bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b3e00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b40b56a0 0x7f16b4156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 -- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40ca7b0 msgr2=0x7f16b409bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40ca7b0 0x7f16b409bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 -- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b40b56a0 msgr2=0x7f16b4156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b40b56a0 0x7f16b4156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 -- 172.21.15.129:0/3013872878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b41f5b20 con 0x7f16b40cc640 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.611+0000 7f16b8c00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f16ac034040 tx=0x7f16ac003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.612+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac040070 con 0x7f16b40cc640 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.612+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16ac003e50 con 0x7f16b40cc640 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.612+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac0047e0 con 0x7f16b40cc640 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.612+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b409b1d0 con 0x7f16b40cc640 2024-09-17T03:52:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.612+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b4156750 con 0x7f16b40cc640 2024-09-17T03:52:24.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.613+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b40cbe10 con 0x7f16b40cc640 2024-09-17T03:52:24.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.614+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16ac043070 con 0x7f16b40cc640 2024-09-17T03:52:24.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.615+0000 7f16b0c00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 0x7f169007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:24.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.615+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f16ac0bbc50 con 0x7f16b40cc640 2024-09-17T03:52:24.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.616+0000 7f16b3e00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 0x7f169007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:24.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.616+0000 7f16b3e00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 0x7f169007b0a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f16a4000f60 tx=0x7f16a4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:24.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.619+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16ac083dc0 con 0x7f16b40cc640 2024-09-17T03:52:24.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.802+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f16b40cfcd0 con 0x7f16b40cc640 2024-09-17T03:52:24.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.802+0000 7f16b0c00640 1 -- 172.21.15.129:0/3013872878 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) ==== 93+0+10 (secure 0 0 0) 0x7f16ac0887f0 con 0x7f16b40cc640 2024-09-17T03:52:24.803 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:52:24.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 msgr2=0x7f169007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 0x7f169007b0a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f16a4000f60 tx=0x7f16a4009040 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 msgr2=0x7f16b40cdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f16ac034040 tx=0x7f16ac003ce0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 shutdown_connections 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1690078b90 0x7f169007b0a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b40cc640 0x7f16b40cdd30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b40ca7b0 0x7f16b409bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 --2- 172.21.15.129:0/3013872878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b40b56a0 0x7f16b4156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 >> 172.21.15.129:0/3013872878 conn(0x7f16b40d0160 msgr2=0x7f16b40b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 shutdown_connections 2024-09-17T03:52:24.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:24.805+0000 7f16bac00640 1 -- 172.21.15.129:0/3013872878 wait complete. 2024-09-17T03:52:24.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-09-17T03:52:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.053+0000 7fa433200640 1 -- 172.21.15.129:0/1836017577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 msgr2=0x7fa42c0cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.053+0000 7fa433200640 1 --2- 172.21.15.129:0/1836017577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c0cfae0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009830 tx=0x7fa41c02a3f0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 -- 172.21.15.129:0/1836017577 shutdown_connections 2024-09-17T03:52:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 --2- 172.21.15.129:0/1836017577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 0x7fa42c0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 --2- 172.21.15.129:0/1836017577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c0cfae0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 -- 172.21.15.129:0/1836017577 >> 172.21.15.129:0/1836017577 conn(0x7fa42c06d7d0 msgr2=0x7fa42c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 -- 172.21.15.129:0/1836017577 shutdown_connections 2024-09-17T03:52:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.054+0000 7fa433200640 1 -- 172.21.15.129:0/1836017577 wait complete. 2024-09-17T03:52:25.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.056+0000 7fa433200640 1 Processor -- start 2024-09-17T03:52:25.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.056+0000 7fa433200640 1 -- start start 2024-09-17T03:52:25.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.057+0000 7fa433200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 0x7fa42c0749b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:25.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.057+0000 7fa433200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa433200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 0x7fa42c1456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa433200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c0cc760 con 0x7fa42c15a9b0 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa433200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c0cc5e0 con 0x7fa42c0cceb0 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa433200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa42c070f20 con 0x7fa42c15ef10 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59608/0 (socket says 172.21.15.129:59608) 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 -- 172.21.15.129:0/3174056152 learned_addr learned my addr 172.21.15.129:0/3174056152 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42be00640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 0x7fa42c0749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa431200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 0x7fa42c1456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 -- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 msgr2=0x7fa42c1456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 0x7fa42c1456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 -- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 msgr2=0x7fa42c0749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 0x7fa42c0749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42b400640 1 -- 172.21.15.129:0/3174056152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c1f57e0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.058+0000 7fa42be00640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 0x7fa42c0749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:52:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa42b400640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d740 tx=0x7fa41800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418017070 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa418009d20 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1f9520 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180106c0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.059+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1f9ae0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.060+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa42c06a5e0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.062+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa41801a070 con 0x7fa42c15a9b0 2024-09-17T03:52:25.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.063+0000 7fa423e00640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 0x7fa40c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:25.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.063+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fa41809aa70 con 0x7fa42c15a9b0 2024-09-17T03:52:25.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.064+0000 7fa42be00640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 0x7fa40c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:25.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.065+0000 7fa42be00640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 0x7fa40c07b0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa41c034000 tx=0x7fa41c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:25.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.068+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa42c06a5e0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.317+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) -- 0x7fa42c0cfae0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.911+0000 7fa423e00640 1 -- 172.21.15.129:0/3174056152 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v170) ==== 143+0+0 (secure 0 0 0) 0x7fa42c1f57e0 con 0x7fa42c15a9b0 2024-09-17T03:52:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 msgr2=0x7fa40c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:25.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 0x7fa40c07b0a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa41c034000 tx=0x7fa41c004730 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 msgr2=0x7fa42c148000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa41800d740 tx=0x7fa41800dc00 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 shutdown_connections 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa40c078b90 0x7fa40c07b0a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42c15ef10 0x7fa42c1456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa42c15a9b0 0x7fa42c148000 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 --2- 172.21.15.129:0/3174056152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa42c0cceb0 0x7fa42c0749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 >> 172.21.15.129:0/3174056152 conn(0x7fa42c06d7d0 msgr2=0x7fa42c0b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 shutdown_connections 2024-09-17T03:52:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:25.914+0000 7fa433200640 1 -- 172.21.15.129:0/3174056152 wait complete. 2024-09-17T03:52:25.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:52:26.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 -- 172.21.15.129:0/3567157242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf950 msgr2=0x7fef4006f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 --2- 172.21.15.129:0/3567157242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf950 0x7fef4006f3b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef38005fd0 tx=0x7fef3802aa10 comp rx=0 tx=0).stop 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 -- 172.21.15.129:0/3567157242 shutdown_connections 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 --2- 172.21.15.129:0/3567157242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf950 0x7fef4006f3b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 --2- 172.21.15.129:0/3567157242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef400cf540 0x7fef4006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 -- 172.21.15.129:0/3567157242 >> 172.21.15.129:0/3567157242 conn(0x7fef4006d410 msgr2=0x7fef400b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 -- 172.21.15.129:0/3567157242 shutdown_connections 2024-09-17T03:52:26.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.162+0000 7fef3fe00640 1 -- 172.21.15.129:0/3567157242 wait complete. 2024-09-17T03:52:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.164+0000 7fef3fe00640 1 Processor -- start 2024-09-17T03:52:26.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.165+0000 7fef3fe00640 1 -- start start 2024-09-17T03:52:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf540 0x7fef4015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef4015ca50 0x7fef401432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef4015de20 con 0x7fef400cf950 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef4015dca0 con 0x7fef400cf540 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3fe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fef4015dfa0 con 0x7fef4015ca50 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59614/0 (socket says 172.21.15.129:59614) 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.166+0000 7fef3e000640 1 -- 172.21.15.129:0/2760549357 learned_addr learned my addr 172.21.15.129:0/2760549357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3ea00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf540 0x7fef4015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3f400640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef4015ca50 0x7fef401432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 -- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef4015ca50 msgr2=0x7fef401432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef4015ca50 0x7fef401432c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 -- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf540 msgr2=0x7fef4015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf540 0x7fef4015adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 -- 172.21.15.129:0/2760549357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4015bf70 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef3e000640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea20 tx=0x7fef2c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.167+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c00cd80 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.168+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef2c004020 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.168+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef40158150 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.168+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2c004470 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.168+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef401586d0 con 0x7fef400cf950 2024-09-17T03:52:26.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.169+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4015e450 con 0x7fef400cf950 2024-09-17T03:52:26.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.170+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef2c0035f0 con 0x7fef400cf950 2024-09-17T03:52:26.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.171+0000 7fef36a00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 0x7fef2007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.172+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7fef2c09b1d0 con 0x7fef400cf950 2024-09-17T03:52:26.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.172+0000 7fef3ea00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 0x7fef2007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.173+0000 7fef3ea00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 0x7fef2007b0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fef40153ab0 tx=0x7fef30009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:26.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.176+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef2c0632c0 con 0x7fef400cf950 2024-09-17T03:52:26.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.359+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fef40153860 con 0x7fef400cf950 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.360+0000 7fef36a00640 1 -- 172.21.15.129:0/2760549357 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v40) ==== 59+0+2294 (secure 0 0 0) 0x7fef2c099e20 con 0x7fef400cf950 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:52:26.360 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:52:26.361 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:52:26.362 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:52:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 msgr2=0x7fef2007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 0x7fef2007b0d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fef40153ab0 tx=0x7fef30009040 comp rx=0 tx=0).stop 2024-09-17T03:52:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 msgr2=0x7fef400a9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fef2c00ea20 tx=0x7fef2c00eee0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 shutdown_connections 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef20078bc0 0x7fef2007b0d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef4015ca50 0x7fef401432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef400cf950 0x7fef400a9f10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 --2- 172.21.15.129:0/2760549357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef400cf540 0x7fef4015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 >> 172.21.15.129:0/2760549357 conn(0x7fef4006d410 msgr2=0x7fef4015e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 shutdown_connections 2024-09-17T03:52:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.362+0000 7fef3fe00640 1 -- 172.21.15.129:0/2760549357 wait complete. 2024-09-17T03:52:26.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 -- 172.21.15.129:0/2493148085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160040 msgr2=0x7ff28c160450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 --2- 172.21.15.129:0/2493148085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160040 0x7ff28c160450 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff278009830 tx=0x7ff27802a410 comp rx=0 tx=0).stop 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 -- 172.21.15.129:0/2493148085 shutdown_connections 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 --2- 172.21.15.129:0/2493148085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c160990 0x7ff28c0ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 --2- 172.21.15.129:0/2493148085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160040 0x7ff28c160450 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 -- 172.21.15.129:0/2493148085 >> 172.21.15.129:0/2493148085 conn(0x7ff28c06ee50 msgr2=0x7ff28c06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.610+0000 7ff292e00640 1 -- 172.21.15.129:0/2493148085 shutdown_connections 2024-09-17T03:52:26.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.611+0000 7ff292e00640 1 -- 172.21.15.129:0/2493148085 wait complete. 2024-09-17T03:52:26.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.613+0000 7ff292e00640 1 Processor -- start 2024-09-17T03:52:26.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.613+0000 7ff292e00640 1 -- start start 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 0x7ff28c1f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 0x7ff28c0b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c0cb160 con 0x7ff28c160040 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff28c0cafe0 con 0x7ff28c160990 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff292e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff28c0cb2e0 con 0x7ff28c15c580 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.614+0000 7ff291000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59616/0 (socket says 172.21.15.129:59616) 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 -- 172.21.15.129:0/1625898172 learned_addr learned my addr 172.21.15.129:0/1625898172 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:26.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291a00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 0x7ff28c1f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff292400640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 0x7ff28c0b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 -- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 msgr2=0x7ff28c1f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 0x7ff28c1f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 -- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 msgr2=0x7ff28c0b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 0x7ff28c0b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 -- 172.21.15.129:0/1625898172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff28c0a9730 con 0x7ff28c160040 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291a00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 0x7ff28c1f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff292400640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 0x7ff28c0b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff291000640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff27802aa00 tx=0x7ff278003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.615+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27803f030 con 0x7ff28c160040 2024-09-17T03:52:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.616+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff28c15f3c0 con 0x7ff28c160040 2024-09-17T03:52:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.616+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff278004510 con 0x7ff28c160040 2024-09-17T03:52:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.616+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff278037040 con 0x7ff28c160040 2024-09-17T03:52:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.616+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff28c1f5ae0 con 0x7ff28c160040 2024-09-17T03:52:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.617+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff28c0cb870 con 0x7ff28c160040 2024-09-17T03:52:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.618+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff278042070 con 0x7ff28c160040 2024-09-17T03:52:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.619+0000 7ff289600640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 0x7ff26c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.619+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7ff2780ba810 con 0x7ff28c160040 2024-09-17T03:52:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.619+0000 7ff291a00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 0x7ff26c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:26.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.620+0000 7ff291a00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 0x7ff26c07b0d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff2800078f0 tx=0x7ff280009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.623+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff278049070 con 0x7ff28c160040 2024-09-17T03:52:26.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.804+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff28c0b3f20 con 0x7ff28c160040 2024-09-17T03:52:26.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.808+0000 7ff289600640 1 -- 172.21.15.129:0/1625898172 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v41)=0 v41) ==== 122+0+0 (secure 0 0 0) 0x7ff278082f50 con 0x7ff28c160040 2024-09-17T03:52:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 msgr2=0x7ff26c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 0x7ff26c07b0d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff2800078f0 tx=0x7ff280009040 comp rx=0 tx=0).stop 2024-09-17T03:52:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 msgr2=0x7ff28c0c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7ff27802aa00 tx=0x7ff278003b60 comp rx=0 tx=0).stop 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 shutdown_connections 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff26c078bc0 0x7ff26c07b0d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff28c160990 0x7ff28c0b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff28c160040 0x7ff28c0c9740 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 --2- 172.21.15.129:0/1625898172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff28c15c580 0x7ff28c1f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 >> 172.21.15.129:0/1625898172 conn(0x7ff28c06ee50 msgr2=0x7ff28c0b0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 shutdown_connections 2024-09-17T03:52:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:26.811+0000 7ff292e00640 1 -- 172.21.15.129:0/1625898172 wait complete. 2024-09-17T03:52:26.835 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:52:26.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:52:26.924 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:52:27.028 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:52:27.028 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:27.028 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:27.064 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:52:27.064 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:52:27.064 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:27.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:52:27.101 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:27.101 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:27.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:52:27.140 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:52:27.140 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:27.177 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:52:27.177 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:52:27.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:27.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 -- 172.21.15.129:0/1590097147 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 msgr2=0x7f84540b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 --2- 172.21.15.129:0/1590097147 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f84540b3090 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f844c005fd0 tx=0x7f844c02aa90 comp rx=0 tx=0).stop 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 -- 172.21.15.129:0/1590097147 shutdown_connections 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 --2- 172.21.15.129:0/1590097147 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f84540b3090 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 --2- 172.21.15.129:0/1590097147 >> v2:172.21.15.181:3300/0 conn(0x7f8454073170 0x7f845415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 -- 172.21.15.129:0/1590097147 >> 172.21.15.129:0/1590097147 conn(0x7f84540b1cb0 msgr2=0x7f84540b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 -- 172.21.15.129:0/1590097147 shutdown_connections 2024-09-17T03:52:27.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.401+0000 7f845a600640 1 -- 172.21.15.129:0/1590097147 wait complete. 2024-09-17T03:52:27.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.403+0000 7f845a600640 1 Processor -- start 2024-09-17T03:52:27.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.403+0000 7f845a600640 1 -- start start 2024-09-17T03:52:27.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:27.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f845415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8454073170 0x7f845415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f84540b3a40 con 0x7f8454070030 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f84540b38c0 con 0x7f8454073170 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f845a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f84540b3bc0 con 0x7f8454070440 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59632/0 (socket says 172.21.15.129:59632) 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453400640 1 -- 172.21.15.129:0/3273376878 learned_addr learned my addr 172.21.15.129:0/3273376878 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453e00640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8454073170 0x7f845415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8452a00640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f845415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453400640 1 -- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 msgr2=0x7f845415df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.405+0000 7f8453400640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f845415df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f8453400640 1 -- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8454073170 msgr2=0x7f845415e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f8453400640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8454073170 0x7f845415e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f8453400640 1 -- 172.21.15.129:0/3273376878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84541f9610 con 0x7f8454070030 2024-09-17T03:52:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f8453400640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f844400ea20 tx=0x7f844400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844400cde0 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8444004020 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84541f98f0 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8444004470 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.406+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84541f9eb0 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.407+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84540b40e0 con 0x7f8454070030 2024-09-17T03:52:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.409+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f84440035f0 con 0x7f8454070030 2024-09-17T03:52:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.410+0000 7f844b400640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 0x7f843407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.410+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 4943+0+0 (secure 0 0 0) 0x7f844409b460 con 0x7f8454070030 2024-09-17T03:52:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.411+0000 7f8452a00640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 0x7f843407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:27.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.411+0000 7f8452a00640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 0x7f843407b0d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8440000f60 tx=0x7f8440009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:27.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.415+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8444063c10 con 0x7f8454070030 2024-09-17T03:52:27.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.635+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84540c3860 con 0x7f8454070030 2024-09-17T03:52:27.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.636+0000 7f844b400640 1 -- 172.21.15.129:0/3273376878 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+3920 (secure 0 0 0) 0x7f8444063510 con 0x7f8454070030 2024-09-17T03:52:27.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:27.636 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":37,"btime":"2024-09-17T03:52:20:584795+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5096,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1538354336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1538354336},{"type":"v1","addr":"172.21.15.181:6837","nonce":1538354336}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5102,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1381564114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1381564114},{"type":"v1","addr":"172.21.15.129:6835","nonce":1381564114}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/734565082","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":734565082},{"type":"v1","addr":"172.21.15.181:6836","nonce":734565082}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5772,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3876244301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3876244301},{"type":"v1","addr":"172.21.15.129:6834","nonce":3876244301}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[]} 2024-09-17T03:52:27.636 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 37 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 msgr2=0x7f843407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 0x7f843407b0d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8440000f60 tx=0x7f8440009040 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 msgr2=0x7f84540c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f844400ea20 tx=0x7f844400eee0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 shutdown_connections 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8434078bc0 0x7f843407b0d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8454073170 0x7f845415e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8454070440 0x7f845415df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.638+0000 7f845a600640 1 --2- 172.21.15.129:0/3273376878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8454070030 0x7f84540c95f0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.639+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 >> 172.21.15.129:0/3273376878 conn(0x7f84540b1cb0 msgr2=0x7f84541e8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:27.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.639+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 shutdown_connections 2024-09-17T03:52:27.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:27.639+0000 7f845a600640 1 -- 172.21.15.129:0/3273376878 wait complete. 2024-09-17T03:52:27.660 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:52:27.660 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:52:27.661 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:52:27.661 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:52:27.661 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:52:27.661 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:52:27.661 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:52:27.661 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:52:27.661 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:52:27.661 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:52:27.662 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:52:27.662 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:52:27.662 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:52:27.662 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:52:27.662 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:52:27.662 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:52:27.687 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:52:27.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:52:27.691 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:52:27.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:52:27.693 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:52:27.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:52:27.732 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:52:27.732 DEBUG:teuthology.parallel:result is None 2024-09-17T03:52:27.736 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:52:27.736 DEBUG:teuthology.parallel:result is None 2024-09-17T03:52:27.739 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:52:27.739 DEBUG:teuthology.parallel:result is None 2024-09-17T03:52:27.923 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:52:27.930 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:52:27.931 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:52:30.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-17T03:52:33.688 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:52:33.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:52:33.693 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:52:33.693 DEBUG:teuthology.parallel:result is None 2024-09-17T03:52:33.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:52:33.892 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:52:33.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 -- 172.21.15.129:0/323911314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c0ca5a0 msgr2=0x7f831c0cada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:33.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 --2- 172.21.15.129:0/323911314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c0ca5a0 0x7f831c0cada0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f831c0cda80 tx=0x7f831002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 -- 172.21.15.129:0/323911314 shutdown_connections 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 --2- 172.21.15.129:0/323911314 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 0x7f831c0b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 --2- 172.21.15.129:0/323911314 >> v2:172.21.15.181:3300/0 conn(0x7f831c15c4d0 0x7f831c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 --2- 172.21.15.129:0/323911314 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c0ca5a0 0x7f831c0cada0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 -- 172.21.15.129:0/323911314 >> 172.21.15.129:0/323911314 conn(0x7f831c06d870 msgr2=0x7f831c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 -- 172.21.15.129:0/323911314 shutdown_connections 2024-09-17T03:52:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.927+0000 7f8322a00640 1 -- 172.21.15.129:0/323911314 wait complete. 2024-09-17T03:52:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.929+0000 7f8322a00640 1 Processor -- start 2024-09-17T03:52:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.929+0000 7f8322a00640 1 -- start start 2024-09-17T03:52:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 0x7f831c15bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f831c0ca5a0 0x7f831c14c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:33.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f831c160510 con 0x7f831c15c4d0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f831c160390 con 0x7f831c0ca5a0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f831c160690 con 0x7f831c0b48c0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34898/0 (socket says 172.21.15.129:34898) 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 -- 172.21.15.129:0/845160347 learned_addr learned my addr 172.21.15.129:0/845160347 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8320c00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f831c0ca5a0 0x7f831c14c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8321600640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 0x7f831c15bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 -- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 msgr2=0x7f831c15bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 0x7f831c15bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 -- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f831c0ca5a0 msgr2=0x7f831c14c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:33.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f831c0ca5a0 0x7f831c14c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.931+0000 7f8322000640 1 -- 172.21.15.129:0/845160347 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f831c15aca0 con 0x7f831c15c4d0 2024-09-17T03:52:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8322000640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f831400eab0 tx=0x7f831400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831400ce40 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8314004b20 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f831c15af40 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8314005020 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.932+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f831c15b500 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.933+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f831c160bb0 con 0x7f831c15c4d0 2024-09-17T03:52:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.934+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f83140051c0 con 0x7f831c15c4d0 2024-09-17T03:52:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.935+0000 7f8319600640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 0x7f82fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.935+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 4943+0+0 (secure 0 0 0) 0x7f831409b4c0 con 0x7f831c15c4d0 2024-09-17T03:52:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.936+0000 7f8321600640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 0x7f82fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:33.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.936+0000 7f8321600640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 0x7f82fc07b0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8310033000 tx=0x7f8310004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:33.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:33.940+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8314063630 con 0x7f831c15c4d0 2024-09-17T03:52:34.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.134+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f831c06ddf0 con 0x7f831c15c4d0 2024-09-17T03:52:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.963+0000 7f8319600640 1 -- 172.21.15.129:0/845160347 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) ==== 97+0+0 (secure 0 0 0) 0x7f8314068060 con 0x7f831c15c4d0 2024-09-17T03:52:34.964 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.966+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 msgr2=0x7f82fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.966+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 0x7f82fc07b0a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8310033000 tx=0x7f8310004730 comp rx=0 tx=0).stop 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.966+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 msgr2=0x7f831c0b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.966+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f831400eab0 tx=0x7f831400ef70 comp rx=0 tx=0).stop 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 shutdown_connections 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82fc078b90 0x7f82fc07b0a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f831c15c4d0 0x7f831c0b1000 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f831c0ca5a0 0x7f831c14c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 --2- 172.21.15.129:0/845160347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f831c0b48c0 0x7f831c15bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 >> 172.21.15.129:0/845160347 conn(0x7f831c06d870 msgr2=0x7f831c15eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 shutdown_connections 2024-09-17T03:52:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:34.967+0000 7f8322a00640 1 -- 172.21.15.129:0/845160347 wait complete. 2024-09-17T03:52:34.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:52:35.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 -- 172.21.15.129:0/812007738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 msgr2=0x7efce400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 --2- 172.21.15.129:0/812007738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efce400b9c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005fd0 tx=0x7efce4005830 comp rx=0 tx=0).stop 2024-09-17T03:52:35.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 -- 172.21.15.129:0/812007738 shutdown_connections 2024-09-17T03:52:35.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 --2- 172.21.15.129:0/812007738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efce400b9c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 -- 172.21.15.129:0/812007738 >> 172.21.15.129:0/812007738 conn(0x7efd000b1cb0 msgr2=0x7efd000b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:35.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 -- 172.21.15.129:0/812007738 shutdown_connections 2024-09-17T03:52:35.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.218+0000 7efd05a00640 1 -- 172.21.15.129:0/812007738 wait complete. 2024-09-17T03:52:35.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.220+0000 7efd05a00640 1 Processor -- start 2024-09-17T03:52:35.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.221+0000 7efd05a00640 1 -- start start 2024-09-17T03:52:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 0x7efd001f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd001f6fd0 0x7efd0009c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd00074640 con 0x7efce4005280 2024-09-17T03:52:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efd0015cc60 con 0x7efd001f4150 2024-09-17T03:52:35.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efd05a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efd000747c0 con 0x7efd001f6fd0 2024-09-17T03:52:35.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efcfea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efcfea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34918/0 (socket says 172.21.15.129:34918) 2024-09-17T03:52:35.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.222+0000 7efcfea00640 1 -- 172.21.15.129:0/2115570936 learned_addr learned my addr 172.21.15.129:0/2115570936 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcf7e00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 0x7efd001f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 -- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd001f6fd0 msgr2=0x7efd0009c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd001f6fd0 0x7efd0009c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 -- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 msgr2=0x7efd001f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 0x7efd001f6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 -- 172.21.15.129:0/2115570936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd001f3420 con 0x7efce4005280 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcf7e00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 0x7efd001f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcfea00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efcf0003db0 tx=0x7efcf0001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:35.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0011700 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcf0016020 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0014eac0 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0011b60 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.223+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001f77c0 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.224+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00074d20 con 0x7efce4005280 2024-09-17T03:52:35.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.226+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efcf0011d00 con 0x7efce4005280 2024-09-17T03:52:35.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.227+0000 7efcf6a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 0x7efcdc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.227+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7efcf009b350 con 0x7efce4005280 2024-09-17T03:52:35.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.227+0000 7efcf7e00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 0x7efcdc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.228+0000 7efcf7e00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 0x7efcdc07b0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7efce8001240 tx=0x7efce8000f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:35.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.231+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efcf0020030 con 0x7efce4005280 2024-09-17T03:52:35.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.479+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7efd000c3860 con 0x7efce4005280 2024-09-17T03:52:35.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.479+0000 7efcf6a00640 1 -- 172.21.15.129:0/2115570936 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) ==== 80+0+5304 (secure 0 0 0) 0x7efcf0063ad0 con 0x7efce4005280 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:52:35.480 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.482 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.488 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:52:35.490 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.491 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:52:35.492 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:52:35.494 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:52:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 msgr2=0x7efcdc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 0x7efcdc07b0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7efce8001240 tx=0x7efce8000f30 comp rx=0 tx=0).stop 2024-09-17T03:52:35.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 msgr2=0x7efd0014e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efcf0003db0 tx=0x7efcf0001800 comp rx=0 tx=0).stop 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 shutdown_connections 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcdc078bc0 0x7efcdc07b0d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd001f6fd0 0x7efd0009c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd001f4150 0x7efd001f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 --2- 172.21.15.129:0/2115570936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce4005280 0x7efd0014e580 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 >> 172.21.15.129:0/2115570936 conn(0x7efd000b1cb0 msgr2=0x7efd000ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 shutdown_connections 2024-09-17T03:52:35.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.482+0000 7efd05a00640 1 -- 172.21.15.129:0/2115570936 wait complete. 2024-09-17T03:52:35.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:52:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.725+0000 7f420e000640 1 -- 172.21.15.129:0/3437811731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208070030 msgr2=0x7f4208074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.725+0000 7f420e000640 1 --2- 172.21.15.129:0/3437811731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208070030 0x7f4208074640 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f42080b4b40 tx=0x7f41f402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 -- 172.21.15.129:0/3437811731 shutdown_connections 2024-09-17T03:52:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 --2- 172.21.15.129:0/3437811731 >> v2:172.21.15.181:3300/0 conn(0x7f4208070440 0x7f4208074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 --2- 172.21.15.129:0/3437811731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208070030 0x7f4208074640 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 -- 172.21.15.129:0/3437811731 >> 172.21.15.129:0/3437811731 conn(0x7f42080b1cb0 msgr2=0x7f42080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 -- 172.21.15.129:0/3437811731 shutdown_connections 2024-09-17T03:52:35.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.726+0000 7f420e000640 1 -- 172.21.15.129:0/3437811731 wait complete. 2024-09-17T03:52:35.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.728+0000 7f420e000640 1 Processor -- start 2024-09-17T03:52:35.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.728+0000 7f420e000640 1 -- start start 2024-09-17T03:52:35.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4208070440 0x7f42081ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42080b39f0 con 0x7f4208073170 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42080b3870 con 0x7f4208070030 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.729+0000 7f420e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f42080b3b70 con 0x7f4208070440 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4206a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4208070440 0x7f42081ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:36084/0 (socket says 172.21.15.129:36084) 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34926/0 (socket says 172.21.15.129:34926) 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 -- 172.21.15.129:0/2870028909 learned_addr learned my addr 172.21.15.129:0/2870028909 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:35.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 -- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4208070440 msgr2=0x7f42081ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4208070440 0x7f42081ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 -- 172.21.15.129:0/2870028909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 msgr2=0x7f42080cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207400640 1 -- 172.21.15.129:0/2870028909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420809b9a0 con 0x7f4208070030 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.730+0000 7f4207e00640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:52:35.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f4207400640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41f800c3a0 tx=0x7f41f800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8017070 con 0x7f4208070030 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41f80053c0 con 0x7f4208070030 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420809bc80 con 0x7f4208070030 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8003850 con 0x7f4208070030 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.731+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420809c240 con 0x7f4208070030 2024-09-17T03:52:35.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.732+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42080b4090 con 0x7f4208070030 2024-09-17T03:52:35.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.734+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f41f801a070 con 0x7f4208070030 2024-09-17T03:52:35.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.735+0000 7f41ff400640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 0x7f41e807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:35.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.735+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f41f809a1d0 con 0x7f4208070030 2024-09-17T03:52:35.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.735+0000 7f4206a00640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 0x7f41e807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:35.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.736+0000 7f4206a00640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 0x7f41e807b0d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f41f4033000 tx=0x7f41f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:35.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.739+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f41f8021020 con 0x7f4208070030 2024-09-17T03:52:35.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.924+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f420809dd90 con 0x7f4208070030 2024-09-17T03:52:35.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.925+0000 7f41ff400640 1 -- 172.21.15.129:0/2870028909 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) ==== 84+0+65 (secure 0 0 0) 0x7f41f8062270 con 0x7f4208070030 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:52:35.926 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 msgr2=0x7f41e807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 0x7f41e807b0d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f41f4033000 tx=0x7f41f4004730 comp rx=0 tx=0).stop 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 msgr2=0x7f420809d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f41f800c3a0 tx=0x7f41f800c860 comp rx=0 tx=0).stop 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 shutdown_connections 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41e8078bc0 0x7f41e807b0d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4208073170 0x7f42080cfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4208070440 0x7f42081ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 --2- 172.21.15.129:0/2870028909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4208070030 0x7f420809d4c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 >> 172.21.15.129:0/2870028909 conn(0x7f42080b1cb0 msgr2=0x7f420815ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 shutdown_connections 2024-09-17T03:52:35.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:35.928+0000 7f420e000640 1 -- 172.21.15.129:0/2870028909 wait complete. 2024-09-17T03:52:35.951 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:52:35.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:52:36.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 -- 172.21.15.129:0/195997159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 msgr2=0x7f4fc4074500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/195997159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc4074500 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009830 tx=0x7f4fb002a510 comp rx=0 tx=0).stop 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 -- 172.21.15.129:0/195997159 shutdown_connections 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/195997159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc4074500 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/195997159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 -- 172.21.15.129:0/195997159 >> 172.21.15.129:0/195997159 conn(0x7f4fc406ee70 msgr2=0x7f4fc406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 -- 172.21.15.129:0/195997159 shutdown_connections 2024-09-17T03:52:36.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.177+0000 7f4fc3e00640 1 -- 172.21.15.129:0/195997159 wait complete. 2024-09-17T03:52:36.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.179+0000 7f4fc3e00640 1 Processor -- start 2024-09-17T03:52:36.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.179+0000 7f4fc3e00640 1 -- start start 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.180+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.180+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc41517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc40733d0 con 0x7f4fc4074a40 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc40b5650 con 0x7f4fc40af670 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc4073550 con 0x7f4fc415e6b0 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34940/0 (socket says 172.21.15.129:34940) 2024-09-17T03:52:36.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 -- 172.21.15.129:0/1497825920 learned_addr learned my addr 172.21.15.129:0/1497825920 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2000640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc41517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 -- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 msgr2=0x7f4fc40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3400640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 -- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 msgr2=0x7f4fc41517f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc41517f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 -- 172.21.15.129:0/1497825920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc41f8fa0 con 0x7f4fc4074a40 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc3400640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.181+0000 7f4fc2a00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e940 tx=0x7f4fb400ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.182+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400cd80 con 0x7f4fc4074a40 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.182+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc41f9280 con 0x7f4fc4074a40 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.182+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fb4010430 con 0x7f4fc4074a40 2024-09-17T03:52:36.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.182+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4010880 con 0x7f4fc4074a40 2024-09-17T03:52:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.182+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc41f9840 con 0x7f4fc4074a40 2024-09-17T03:52:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.183+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc41f9c20 con 0x7f4fc4074a40 2024-09-17T03:52:36.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.184+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4fb401f020 con 0x7f4fc4074a40 2024-09-17T03:52:36.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.185+0000 7f4fbaa00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 0x7f4fa407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:36.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.185+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f4fb409b2a0 con 0x7f4fc4074a40 2024-09-17T03:52:36.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.186+0000 7f4fc2000640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 0x7f4fa407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.186+0000 7f4fc2000640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 0x7f4fa407b0a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0034000 tx=0x7f4fb0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:36.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.189+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4fc41f9c20 con 0x7f4fc4074a40 2024-09-17T03:52:36.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.373+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4fc4074500 con 0x7f4fc4074a40 2024-09-17T03:52:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.972+0000 7f4fbaa00640 1 -- 172.21.15.129:0/1497825920 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) ==== 138+0+0 (secure 0 0 0) 0x7f4fb4063990 con 0x7f4fc4074a40 2024-09-17T03:52:36.973 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:52:36.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 msgr2=0x7f4fa407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 0x7f4fa407b0a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0034000 tx=0x7f4fb0004730 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 msgr2=0x7f4fc41459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e940 tx=0x7f4fb400ee00 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 shutdown_connections 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fa4078b90 0x7f4fa407b0a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fc415e6b0 0x7f4fc40b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fc40af670 0x7f4fc41517f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 --2- 172.21.15.129:0/1497825920 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fc4074a40 0x7f4fc41459d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.975+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 >> 172.21.15.129:0/1497825920 conn(0x7f4fc406ee70 msgr2=0x7f4fc4159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.976+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 shutdown_connections 2024-09-17T03:52:36.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:36.976+0000 7f4fc3e00640 1 -- 172.21.15.129:0/1497825920 wait complete. 2024-09-17T03:52:37.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:52:37.141 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-17T03:52:37.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 -- 172.21.15.129:0/228851096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240ca800 msgr2=0x7f22240cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:37.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 --2- 172.21.15.129:0/228851096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240ca800 0x7f22240cba00 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2224075080 tx=0x7f221802a560 comp rx=0 tx=0).stop 2024-09-17T03:52:37.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 -- 172.21.15.129:0/228851096 shutdown_connections 2024-09-17T03:52:37.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 --2- 172.21.15.129:0/228851096 >> v2:172.21.15.181:3300/0 conn(0x7f22240704e0 0x7f22240cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 --2- 172.21.15.129:0/228851096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240cad30 0x7f222406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 --2- 172.21.15.129:0/228851096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240ca800 0x7f22240cba00 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 -- 172.21.15.129:0/228851096 >> 172.21.15.129:0/228851096 conn(0x7f22240b1cb0 msgr2=0x7f22240b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:37.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 -- 172.21.15.129:0/228851096 shutdown_connections 2024-09-17T03:52:37.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.230+0000 7f222c200640 1 -- 172.21.15.129:0/228851096 wait complete. 2024-09-17T03:52:37.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.232+0000 7f222c200640 1 Processor -- start 2024-09-17T03:52:37.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.232+0000 7f222c200640 1 -- start start 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22240704e0 0x7f222415e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240ca800 0x7f22241f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22240ccf80 con 0x7f22240cad30 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f22240cec60 con 0x7f22240704e0 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222c200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f22240cd100 con 0x7f22240ca800 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34944/0 (socket says 172.21.15.129:34944) 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f222a200640 1 -- 172.21.15.129:0/3568902849 learned_addr learned my addr 172.21.15.129:0/3568902849 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:37.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f2229800640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22240704e0 0x7f222415e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.234+0000 7f2228e00640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240ca800 0x7f22241f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 -- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240ca800 msgr2=0x7f22241f35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240ca800 0x7f22241f35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 -- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22240704e0 msgr2=0x7f222415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22240704e0 0x7f222415e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 -- 172.21.15.129:0/3568902849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222415cfd0 con 0x7f22240cad30 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222a200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f221c00e960 tx=0x7f221c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c004d20 con 0x7f22240cad30 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f221c00cac0 con 0x7f22240cad30 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22241f3e40 con 0x7f22240cad30 2024-09-17T03:52:37.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c005020 con 0x7f22240cad30 2024-09-17T03:52:37.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.235+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f222415d240 con 0x7f22240cad30 2024-09-17T03:52:37.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.237+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22240cd690 con 0x7f22240cad30 2024-09-17T03:52:37.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.238+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f221c0051c0 con 0x7f22240cad30 2024-09-17T03:52:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.239+0000 7f2221600640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 0x7f220407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.239+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5227+0+0 (secure 0 0 0) 0x7f221c09b790 con 0x7f22240cad30 2024-09-17T03:52:37.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.239+0000 7f2229800640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 0x7f220407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:37.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.240+0000 7f2229800640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 0x7f220407b0a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2218033000 tx=0x7f2218004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:37.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.243+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f221c016030 con 0x7f22240cad30 2024-09-17T03:52:37.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.426+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f22240b23a0 con 0x7f22240cad30 2024-09-17T03:52:37.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.975+0000 7f2221600640 1 -- 172.21.15.129:0/3568902849 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) ==== 170+0+0 (secure 0 0 0) 0x7f221c063840 con 0x7f22240cad30 2024-09-17T03:52:37.976 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:52:37.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.980+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 msgr2=0x7f220407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:37.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.980+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 0x7f220407b0a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2218033000 tx=0x7f2218004730 comp rx=0 tx=0).stop 2024-09-17T03:52:37.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.980+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 msgr2=0x7f22240c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:37.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.980+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f221c00e960 tx=0x7f221c00ee20 comp rx=0 tx=0).stop 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 shutdown_connections 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2204078b90 0x7f220407b0a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22240cad30 0x7f22240c5ef0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22240ca800 0x7f22241f35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 --2- 172.21.15.129:0/3568902849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22240704e0 0x7f222415e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.981+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 >> 172.21.15.129:0/3568902849 conn(0x7f22240b1cb0 msgr2=0x7f222415b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.982+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 shutdown_connections 2024-09-17T03:52:37.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:37.982+0000 7f222c200640 1 -- 172.21.15.129:0/3568902849 wait complete. 2024-09-17T03:52:38.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:52:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 -- 172.21.15.129:0/21556518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cc4c0 msgr2=0x7f6ed00ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/21556518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cc4c0 0x7f6ed00ce730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8009be0 tx=0x7f6ec802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 -- 172.21.15.129:0/21556518 shutdown_connections 2024-09-17T03:52:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/21556518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cc4c0 0x7f6ed00ce730 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 -- 172.21.15.129:0/21556518 >> 172.21.15.129:0/21556518 conn(0x7f6ed00b1cb0 msgr2=0x7f6ed00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:38.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 -- 172.21.15.129:0/21556518 shutdown_connections 2024-09-17T03:52:38.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.250+0000 7f6ed5c00640 1 -- 172.21.15.129:0/21556518 wait complete. 2024-09-17T03:52:38.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.252+0000 7f6ed5c00640 1 Processor -- start 2024-09-17T03:52:38.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.252+0000 7f6ed5c00640 1 -- start start 2024-09-17T03:52:38.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed00cc4c0 0x7f6ed014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:38.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cf6f0 0x7f6ed01f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed00cf340 con 0x7f6ed01f5c90 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed00cf1c0 con 0x7f6ed00cc4c0 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ed5c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed00701b0 con 0x7f6ed00cf6f0 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34954/0 (socket says 172.21.15.129:34954) 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 -- 172.21.15.129:0/4092940304 learned_addr learned my addr 172.21.15.129:0/4092940304 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecea00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed00cc4c0 0x7f6ed014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ece000640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cf6f0 0x7f6ed01f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 -- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cf6f0 msgr2=0x7f6ed01f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cf6f0 0x7f6ed01f92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 -- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed00cc4c0 msgr2=0x7f6ed014e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:38.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed00cc4c0 0x7f6ed014e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.254+0000 7f6ecf400640 1 -- 172.21.15.129:0/4092940304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed0161750 con 0x7f6ed01f5c90 2024-09-17T03:52:38.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ecf400640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8035040 tx=0x7f6ec8003e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:38.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8042020 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ec8033b00 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8037040 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed01f9800 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.255+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed01f54d0 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.256+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed01f58d0 con 0x7f6ed01f5c90 2024-09-17T03:52:38.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.258+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6ec8045020 con 0x7f6ed01f5c90 2024-09-17T03:52:38.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.259+0000 7f6ec6a00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 0x7f6eb007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:38.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.259+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5609+0+0 (secure 0 0 0) 0x7f6ec80bbff0 con 0x7f6ed01f5c90 2024-09-17T03:52:38.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.259+0000 7f6ecea00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 0x7f6eb007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:38.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.260+0000 7f6ecea00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 0x7f6eb007b0a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001eb0 tx=0x7f6ec0001750 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:38.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.262+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ec804c030 con 0x7f6ed01f5c90 2024-09-17T03:52:38.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.482+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f6ed01619c0 con 0x7f6ed01f5c90 2024-09-17T03:52:38.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.987+0000 7f6ec6a00640 1 -- 172.21.15.129:0/4092940304 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) ==== 174+0+0 (secure 0 0 0) 0x7f6ed01f9800 con 0x7f6ed01f5c90 2024-09-17T03:52:38.987 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 10 and data pool 11 2024-09-17T03:52:38.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.990+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 msgr2=0x7f6eb007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:38.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.990+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 0x7f6eb007b0a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0001eb0 tx=0x7f6ec0001750 comp rx=0 tx=0).stop 2024-09-17T03:52:38.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.990+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 msgr2=0x7f6ed015f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:38.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.990+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8035040 tx=0x7f6ec8003e70 comp rx=0 tx=0).stop 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 shutdown_connections 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eb0078b90 0x7f6eb007b0a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ed01f5c90 0x7f6ed015f750 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ed00cf6f0 0x7f6ed01f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 --2- 172.21.15.129:0/4092940304 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ed00cc4c0 0x7f6ed014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.992+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 >> 172.21.15.129:0/4092940304 conn(0x7f6ed00b1cb0 msgr2=0x7f6ed00b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:38.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.993+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 shutdown_connections 2024-09-17T03:52:38.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:38.993+0000 7f6ed5c00640 1 -- 172.21.15.129:0/4092940304 wait complete. 2024-09-17T03:52:39.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 -- 172.21.15.129:0/2414986891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0d03d0 msgr2=0x7f38ac0b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 --2- 172.21.15.129:0/2414986891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0d03d0 0x7f38ac0b70e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3894005fd0 tx=0x7f389402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 -- 172.21.15.129:0/2414986891 shutdown_connections 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 --2- 172.21.15.129:0/2414986891 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0d03d0 0x7f38ac0b70e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 -- 172.21.15.129:0/2414986891 >> 172.21.15.129:0/2414986891 conn(0x7f38ac0cec00 msgr2=0x7f38ac0cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 -- 172.21.15.129:0/2414986891 shutdown_connections 2024-09-17T03:52:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.259+0000 7f38b2400640 1 -- 172.21.15.129:0/2414986891 wait complete. 2024-09-17T03:52:39.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.261+0000 7f38b2400640 1 Processor -- start 2024-09-17T03:52:39.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.261+0000 7f38b2400640 1 -- start start 2024-09-17T03:52:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38ac0b6cd0 0x7f38ac1f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38ac15c2f0 0x7f38ac0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac0cdba0 con 0x7f38ac0b6150 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac0cda20 con 0x7f38ac0b6cd0 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b2400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac0cdd20 con 0x7f38ac15c2f0 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34974/0 (socket says 172.21.15.129:34974) 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 -- 172.21.15.129:0/772290836 learned_addr learned my addr 172.21.15.129:0/772290836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 -- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38ac15c2f0 msgr2=0x7f38ac0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38ac15c2f0 0x7f38ac0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 -- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38ac0b6cd0 msgr2=0x7f38ac1f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38ac0b6cd0 0x7f38ac1f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.263+0000 7f38b1000640 1 -- 172.21.15.129:0/772290836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac14f330 con 0x7f38ac0b6150 2024-09-17T03:52:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38b1000640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c400 tx=0x7f389c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c017070 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f389c0053c0 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0038a0 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac1f54f0 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.264+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1f5ab0 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.265+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac09b970 con 0x7f38ac0b6150 2024-09-17T03:52:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.266+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f389c01a070 con 0x7f38ac0b6150 2024-09-17T03:52:39.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.267+0000 7f38a9600640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 0x7f388807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.267+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f389c09a160 con 0x7f38ac0b6150 2024-09-17T03:52:39.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.268+0000 7f38a3e00640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 0x7f388807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.268+0000 7f38a3e00640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 0x7f388807b0d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3894033000 tx=0x7f38940046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:39.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.272+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f38ac09b970 con 0x7f38ac0b6150 2024-09-17T03:52:39.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.455+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38ac0b70e0 con 0x7f38ac0b6150 2024-09-17T03:52:39.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.456+0000 7f38a9600640 1 -- 172.21.15.129:0/772290836 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16789 (secure 0 0 0) 0x7f389c062500 con 0x7f38ac0b6150 2024-09-17T03:52:39.456 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:39.456 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":67,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:52:38.484260+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:52:36.386165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-17T03:52:37.439087+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 msgr2=0x7f388807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 0x7f388807b0d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3894033000 tx=0x7f38940046c0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 msgr2=0x7f38ac0c82f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f389c00c400 tx=0x7f389c00c8c0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 shutdown_connections 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3888078bc0 0x7f388807b0d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38ac15c2f0 0x7f38ac0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38ac0b6cd0 0x7f38ac1f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 --2- 172.21.15.129:0/772290836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38ac0b6150 0x7f38ac0c82f0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 >> 172.21.15.129:0/772290836 conn(0x7f38ac0cec00 msgr2=0x7f38ac071140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.458+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 shutdown_connections 2024-09-17T03:52:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.459+0000 7f38b2400640 1 -- 172.21.15.129:0/772290836 wait complete. 2024-09-17T03:52:39.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 -- 172.21.15.129:0/3612145300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad006f580 msgr2=0x7fcad015d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 --2- 172.21.15.129:0/3612145300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad006f580 0x7fcad015d700 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcac8005fd0 tx=0x7fcac802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 -- 172.21.15.129:0/3612145300 shutdown_connections 2024-09-17T03:52:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 --2- 172.21.15.129:0/3612145300 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad006f580 0x7fcad015d700 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 -- 172.21.15.129:0/3612145300 >> 172.21.15.129:0/3612145300 conn(0x7fcad006d410 msgr2=0x7fcad00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 -- 172.21.15.129:0/3612145300 shutdown_connections 2024-09-17T03:52:39.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.701+0000 7fcad6400640 1 -- 172.21.15.129:0/3612145300 wait complete. 2024-09-17T03:52:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.703+0000 7fcad6400640 1 Processor -- start 2024-09-17T03:52:39.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.704+0000 7fcad6400640 1 -- start start 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcad00cf540 0x7fcad00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcad01ef2f0 0x7fcad01f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcad015e010 con 0x7fcad015ca50 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcad015de90 con 0x7fcad01ef2f0 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcad015e190 con 0x7fcad00cf540 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34990/0 (socket says 172.21.15.129:34990) 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcacfe00640 1 -- 172.21.15.129:0/2431144342 learned_addr learned my addr 172.21.15.129:0/2431144342 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.705+0000 7fcad5000640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcad00cf540 0x7fcad00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcad5a00640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcad01ef2f0 0x7fcad01f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 -- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcad00cf540 msgr2=0x7fcad00b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcad00cf540 0x7fcad00b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 -- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcad01ef2f0 msgr2=0x7fcad01f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcad01ef2f0 0x7fcad01f5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 -- 172.21.15.129:0/2431144342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad01f9150 con 0x7fcad015ca50 2024-09-17T03:52:39.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcacfe00640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c3c0 tx=0x7fcabc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.706+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc017070 con 0x7fcad015ca50 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.707+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcabc0053e0 con 0x7fcad015ca50 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.707+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcabc003870 con 0x7fcad015ca50 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.707+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad01f9430 con 0x7fcad015ca50 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.707+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad01f5700 con 0x7fcad015ca50 2024-09-17T03:52:39.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.708+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad01f54f0 con 0x7fcad015ca50 2024-09-17T03:52:39.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.709+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcabc01a070 con 0x7fcad015ca50 2024-09-17T03:52:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.710+0000 7fcaccc00640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 0x7fcab007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.710+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7fcabc09a0a0 con 0x7fcad015ca50 2024-09-17T03:52:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.710+0000 7fcad5000640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 0x7fcab007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.711+0000 7fcad5000640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 0x7fcab007b0a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcac0001e20 tx=0x7fcac0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.714+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcabc061df0 con 0x7fcad015ca50 2024-09-17T03:52:39.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.903+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcad015d700 con 0x7fcad015ca50 2024-09-17T03:52:39.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.904+0000 7fcaccc00640 1 -- 172.21.15.129:0/2431144342 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16789 (secure 0 0 0) 0x7fcabc066820 con 0x7fcad015ca50 2024-09-17T03:52:39.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:39.904 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":67,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:52:38.484260+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:52:36.386165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-17T03:52:37.439087+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 msgr2=0x7fcab007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 0x7fcab007b0a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcac0001e20 tx=0x7fcac0009040 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 msgr2=0x7fcad01407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcabc00c3c0 tx=0x7fcabc00c880 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 shutdown_connections 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcab0078b90 0x7fcab007b0a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcad01ef2f0 0x7fcad01f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcad015ca50 0x7fcad01407c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 --2- 172.21.15.129:0/2431144342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcad00cf540 0x7fcad00b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.906+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 >> 172.21.15.129:0/2431144342 conn(0x7fcad006d410 msgr2=0x7fcad0155210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:39.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.907+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 shutdown_connections 2024-09-17T03:52:39.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:39.907+0000 7fcad6400640 1 -- 172.21.15.129:0/2431144342 wait complete. 2024-09-17T03:52:39.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:52:40.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 -- 172.21.15.129:0/1014242288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 msgr2=0x7f34380ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:40.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 --2- 172.21.15.129:0/1014242288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f34380ccf60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f3438072bc0 tx=0x7f343002aa30 comp rx=0 tx=0).stop 2024-09-17T03:52:40.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 -- 172.21.15.129:0/1014242288 shutdown_connections 2024-09-17T03:52:40.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 --2- 172.21.15.129:0/1014242288 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f34380ccf60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:40.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 --2- 172.21.15.129:0/1014242288 >> v2:172.21.15.181:3300/0 conn(0x7f34380af3c0 0x7f34380b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:40.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 -- 172.21.15.129:0/1014242288 >> 172.21.15.129:0/1014242288 conn(0x7f343806d7d0 msgr2=0x7f34380b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:40.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 -- 172.21.15.129:0/1014242288 shutdown_connections 2024-09-17T03:52:40.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.158+0000 7f3440000640 1 -- 172.21.15.129:0/1014242288 wait complete. 2024-09-17T03:52:40.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.160+0000 7f3440000640 1 Processor -- start 2024-09-17T03:52:40.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.160+0000 7f3440000640 1 -- start start 2024-09-17T03:52:40.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34380af3c0 0x7f34380ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:40.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34380cc1e0 0x7f343814c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34381533a0 con 0x7f34380ccb50 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f34380af8e0 con 0x7f34380af3c0 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f3440000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3438153520 con 0x7f34380cc1e0 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35000/0 (socket says 172.21.15.129:35000) 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343e000640 1 -- 172.21.15.129:0/3538525392 learned_addr learned my addr 172.21.15.129:0/3538525392 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343d600640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34380af3c0 0x7f34380ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343cc00640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34380cc1e0 0x7f343814c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343e000640 1 -- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34380cc1e0 msgr2=0x7f343814c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.162+0000 7f343e000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34380cc1e0 0x7f343814c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:40.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f343e000640 1 -- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34380af3c0 msgr2=0x7f34380ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:40.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f343e000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34380af3c0 0x7f34380ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:40.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f343e000640 1 -- 172.21.15.129:0/3538525392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3438159410 con 0x7f34380ccb50 2024-09-17T03:52:40.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f343e000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f343002b020 tx=0x7f3430003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430033050 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3430004450 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34381f93f0 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3430004840 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.163+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34381f99b0 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.164+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3438153a40 con 0x7f34380ccb50 2024-09-17T03:52:40.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.165+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3430036070 con 0x7f34380ccb50 2024-09-17T03:52:40.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.166+0000 7f3435600640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 0x7f341807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:40.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.166+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f34300bc2c0 con 0x7f34380ccb50 2024-09-17T03:52:40.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.167+0000 7f343d600640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 0x7f341807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:40.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.168+0000 7f343d600640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 0x7f341807b0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f342c006fd0 tx=0x7f342c008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:40.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.171+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f34380cd4a0 con 0x7f34380ccb50 2024-09-17T03:52:40.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:40.393+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f34380b41a0 con 0x7f34380ccb50 2024-09-17T03:52:41.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.011+0000 7f3435600640 1 -- 172.21.15.129:0/3538525392 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) ==== 118+0+0 (secure 0 0 0) 0x7f34300846f0 con 0x7f34380ccb50 2024-09-17T03:52:41.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 msgr2=0x7f341807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:41.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 0x7f341807b0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f342c006fd0 tx=0x7f342c008040 comp rx=0 tx=0).stop 2024-09-17T03:52:41.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 msgr2=0x7f343814c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:41.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f343002b020 tx=0x7f3430003f10 comp rx=0 tx=0).stop 2024-09-17T03:52:41.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 shutdown_connections 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3418078b90 0x7f341807b0a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34380ccb50 0x7f343814c870 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34380cc1e0 0x7f343814c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 --2- 172.21.15.129:0/3538525392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34380af3c0 0x7f34380ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 >> 172.21.15.129:0/3538525392 conn(0x7f343806d7d0 msgr2=0x7f3438074730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 shutdown_connections 2024-09-17T03:52:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.014+0000 7f3440000640 1 -- 172.21.15.129:0/3538525392 wait complete. 2024-09-17T03:52:41.037 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:52:41.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:52:41.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.262+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3301100422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0d1100 msgr2=0x7fa0fc0b71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:41.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.262+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3301100422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0d1100 0x7fa0fc0b71c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4005fd0 tx=0x7fa0f402a540 comp rx=0 tx=0).stop 2024-09-17T03:52:41.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3301100422 shutdown_connections 2024-09-17T03:52:41.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3301100422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0d1100 0x7fa0fc0b71c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3301100422 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0cffa0 0x7fa0fc0b4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3301100422 >> 172.21.15.129:0/3301100422 conn(0x7fa0fc09d6b0 msgr2=0x7fa0fc0d4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:41.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3301100422 shutdown_connections 2024-09-17T03:52:41.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.263+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3301100422 wait complete. 2024-09-17T03:52:41.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.265+0000 7fa0fbe00640 1 Processor -- start 2024-09-17T03:52:41.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.265+0000 7fa0fbe00640 1 -- start start 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0fc0caf80 0x7fa0fc15dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0d1100 0x7fa0fc0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0fc0b7ae0 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0fc0b7960 con 0x7fa0fc0caf80 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fbe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa0fc0b7c60 con 0x7fa0fc0d1100 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35022/0 (socket says 172.21.15.129:35022) 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 -- 172.21.15.129:0/3595387783 learned_addr learned my addr 172.21.15.129:0/3595387783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:41.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0faa00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0fc0caf80 0x7fa0fc15dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fb400640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0d1100 0x7fa0fc0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 -- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0d1100 msgr2=0x7fa0fc0c8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0d1100 0x7fa0fc0c8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 -- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0fc0caf80 msgr2=0x7fa0fc15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0fc0caf80 0x7fa0fc15dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.267+0000 7fa0fa000640 1 -- 172.21.15.129:0/3595387783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0fc0cabe0 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0fa000640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800d740 tx=0x7fa0e800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8017070 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0e800de50 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0fc1624c0 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80099d0 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.268+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0fc162a80 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.269+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0fc0b4d90 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.270+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa0e8009b70 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.271+0000 7fa0f2a00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 0x7fa0dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:41.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.271+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fa0e809aff0 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.272+0000 7fa0faa00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 0x7fa0dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:41.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.273+0000 7fa0faa00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 0x7fa0dc07b0a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0cb3c0 tx=0x7fa0ec009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:41.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.276+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa0e8020020 con 0x7fa0fc0cffa0 2024-09-17T03:52:41.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:41.493+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa0fc0b6c80 con 0x7fa0fc0cffa0 2024-09-17T03:52:42.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.013+0000 7fa0f2a00640 1 -- 172.21.15.129:0/3595387783 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) ==== 122+0+0 (secure 0 0 0) 0x7fa0e80634d0 con 0x7fa0fc0cffa0 2024-09-17T03:52:42.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 msgr2=0x7fa0dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 0x7fa0dc07b0a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc0cb3c0 tx=0x7fa0ec009040 comp rx=0 tx=0).stop 2024-09-17T03:52:42.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 msgr2=0x7fa0fc0c8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800d740 tx=0x7fa0e800dc00 comp rx=0 tx=0).stop 2024-09-17T03:52:42.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 shutdown_connections 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0dc078b90 0x7fa0dc07b0a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0fc0d1100 0x7fa0fc0c8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0fc0cffa0 0x7fa0fc0c8500 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 --2- 172.21.15.129:0/3595387783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0fc0caf80 0x7fa0fc15dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 >> 172.21.15.129:0/3595387783 conn(0x7fa0fc09d6b0 msgr2=0x7fa0fc0d06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 shutdown_connections 2024-09-17T03:52:42.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.016+0000 7fa0fbe00640 1 -- 172.21.15.129:0/3595387783 wait complete. 2024-09-17T03:52:42.043 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:52:42.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d57400640 1 -- 172.21.15.129:0/233842679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 msgr2=0x7f2d380051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d57400640 1 --2- 172.21.15.129:0/233842679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d380051f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800c650 tx=0x7f2d38005830 comp rx=0 tx=0).stop 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d55400640 1 -- 172.21.15.129:0/233842679 reap_dead start 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d57400640 1 -- 172.21.15.129:0/233842679 shutdown_connections 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d57400640 1 -- 172.21.15.129:0/233842679 >> 172.21.15.129:0/233842679 conn(0x7f2d500b1cb0 msgr2=0x7f2d500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.263+0000 7f2d57400640 1 -- 172.21.15.129:0/233842679 shutdown_connections 2024-09-17T03:52:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.264+0000 7f2d57400640 1 -- 172.21.15.129:0/233842679 wait complete. 2024-09-17T03:52:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.266+0000 7f2d57400640 1 Processor -- start 2024-09-17T03:52:42.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.266+0000 7f2d57400640 1 -- start start 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d501f41c0 0x7f2d5009aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d501e5cb0 0x7f2d501e60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d500b37a0 con 0x7f2d38004d40 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5015cd20 con 0x7f2d501e5cb0 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d57400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2d500b3920 con 0x7f2d501f41c0 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.267+0000 7f2d54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35046/0 (socket says 172.21.15.129:35046) 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 -- 172.21.15.129:0/2117151615 learned_addr learned my addr 172.21.15.129:0/2117151615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d55400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d501e5cb0 0x7f2d501e60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d4fe00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d501f41c0 0x7f2d5009aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 -- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d501f41c0 msgr2=0x7f2d5009aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d501f41c0 0x7f2d5009aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 -- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d501e5cb0 msgr2=0x7f2d501e60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d501e5cb0 0x7f2d501e60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 -- 172.21.15.129:0/2117151615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d501e6e30 con 0x7f2d38004d40 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d54a00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2d44011ae0 tx=0x7f2d44009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.268+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4401e070 con 0x7f2d38004d40 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.269+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d44004460 con 0x7f2d38004d40 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.269+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d501e51d0 con 0x7f2d38004d40 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.269+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4400fc60 con 0x7f2d38004d40 2024-09-17T03:52:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.269+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d501e5790 con 0x7f2d38004d40 2024-09-17T03:52:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.270+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d500c3860 con 0x7f2d38004d40 2024-09-17T03:52:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.271+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2d44021070 con 0x7f2d38004d40 2024-09-17T03:52:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.272+0000 7f2d4cc00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 0x7f2d3007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.272+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f2d4409b790 con 0x7f2d38004d40 2024-09-17T03:52:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.273+0000 7f2d4fe00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 0x7f2d3007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.273+0000 7f2d4fe00640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 0x7f2d3007b0a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d5015bdd0 tx=0x7f2d3c006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:42.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.277+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2d44063630 con 0x7f2d38004d40 2024-09-17T03:52:42.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.493+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d500b3130 con 0x7f2d38004d40 2024-09-17T03:52:42.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.494+0000 7f2d4cc00640 1 -- 172.21.15.129:0/2117151615 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f2d44068060 con 0x7f2d38004d40 2024-09-17T03:52:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:42.494 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:52:42.495 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.496+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 msgr2=0x7f2d3007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.496+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 0x7f2d3007b0a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2d5015bdd0 tx=0x7f2d3c006b70 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.496+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 msgr2=0x7f2d5014b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.496+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f2d44011ae0 tx=0x7f2d44009e90 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 shutdown_connections 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d30078b90 0x7f2d3007b0a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d501e5cb0 0x7f2d501e60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d501f41c0 0x7f2d5009aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 --2- 172.21.15.129:0/2117151615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d38004d40 0x7f2d5014b670 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 >> 172.21.15.129:0/2117151615 conn(0x7f2d500b1cb0 msgr2=0x7f2d500ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 shutdown_connections 2024-09-17T03:52:42.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.497+0000 7f2d57400640 1 -- 172.21.15.129:0/2117151615 wait complete. 2024-09-17T03:52:42.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:52:42.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 -- 172.21.15.129:0/1338149941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ce020 msgr2=0x7f41080ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 --2- 172.21.15.129:0/1338149941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ce020 0x7f41080ce430 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4108071d60 tx=0x7f410002aac0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 -- 172.21.15.129:0/1338149941 shutdown_connections 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 --2- 172.21.15.129:0/1338149941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ce020 0x7f41080ce430 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 --2- 172.21.15.129:0/1338149941 >> v2:172.21.15.181:3300/0 conn(0x7f41080cf6f0 0x7f41080cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 -- 172.21.15.129:0/1338149941 >> 172.21.15.129:0/1338149941 conn(0x7f41080b1cb0 msgr2=0x7f41080b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 -- 172.21.15.129:0/1338149941 shutdown_connections 2024-09-17T03:52:42.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.741+0000 7f4110600640 1 -- 172.21.15.129:0/1338149941 wait complete. 2024-09-17T03:52:42.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.743+0000 7f4110600640 1 Processor -- start 2024-09-17T03:52:42.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.744+0000 7f4110600640 1 -- start start 2024-09-17T03:52:42.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 0x7f4108160470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 0x7f410809d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41080701b0 con 0x7f41080ccb30 2024-09-17T03:52:42.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f41080cf3d0 con 0x7f41080ce020 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f4110600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4108070330 con 0x7f41080cf6f0 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f410dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f410dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35070/0 (socket says 172.21.15.129:35070) 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.745+0000 7f410dc00640 1 -- 172.21.15.129:0/1152217528 learned_addr learned my addr 172.21.15.129:0/1152217528 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410e600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 0x7f410809d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410d200640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 0x7f4108160470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 -- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 msgr2=0x7f410809d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 0x7f410809d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 -- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 msgr2=0x7f4108160470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 0x7f4108160470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 -- 172.21.15.129:0/1152217528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410809dda0 con 0x7f41080ccb30 2024-09-17T03:52:42.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410e600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 0x7f410809d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410d200640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 0x7f4108160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f410dc00640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00d6d0 tx=0x7f40fc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc017070 con 0x7f41080ccb30 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40fc004310 con 0x7f41080ccb30 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41081f5b90 con 0x7f41080ccb30 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc009a80 con 0x7f41080ccb30 2024-09-17T03:52:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.746+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41081f6150 con 0x7f41080ccb30 2024-09-17T03:52:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.747+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41080ceae0 con 0x7f41080ccb30 2024-09-17T03:52:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.749+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f40fc0035f0 con 0x7f41080ccb30 2024-09-17T03:52:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.750+0000 7f4105600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 0x7f40e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.750+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f40fc09afc0 con 0x7f41080ccb30 2024-09-17T03:52:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.750+0000 7f410d200640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 0x7f40e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.751+0000 7f410d200640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 0x7f40e807b0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000f60 tx=0x7f40f4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:42.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.754+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f40fc063480 con 0x7f41080ccb30 2024-09-17T03:52:42.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.939+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41080c9f50 con 0x7f41080ccb30 2024-09-17T03:52:42.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.940+0000 7f4105600640 1 -- 172.21.15.129:0/1152217528 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16776 (secure 0 0 0) 0x7f40fc062d80 con 0x7f41080ccb30 2024-09-17T03:52:42.941 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:42.941 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":68,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:52:39.998558+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:52:36.386165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-17T03:52:37.439087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:52:42.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 msgr2=0x7f40e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 0x7f40e807b0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40f4000f60 tx=0x7f40f4009040 comp rx=0 tx=0).stop 2024-09-17T03:52:42.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 msgr2=0x7f410815cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:42.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00d6d0 tx=0x7f40fc00db90 comp rx=0 tx=0).stop 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 shutdown_connections 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40e8078b90 0x7f40e807b0a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41080cf6f0 0x7f410809d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41080ce020 0x7f4108160470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 --2- 172.21.15.129:0/1152217528 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41080ccb30 0x7f410815cab0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 >> 172.21.15.129:0/1152217528 conn(0x7f41080b1cb0 msgr2=0x7f41080b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 shutdown_connections 2024-09-17T03:52:42.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:42.943+0000 7f4110600640 1 -- 172.21.15.129:0/1152217528 wait complete. 2024-09-17T03:52:42.965 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:52:42.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:52:43.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 -- 172.21.15.129:0/3793885543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720073170 msgr2=0x7f072015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 --2- 172.21.15.129:0/3793885543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720073170 0x7f072015c780 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f07200710f0 tx=0x7f071002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 -- 172.21.15.129:0/3793885543 shutdown_connections 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 --2- 172.21.15.129:0/3793885543 >> v2:172.21.15.181:3300/0 conn(0x7f0720070030 0x7f0720074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 --2- 172.21.15.129:0/3793885543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720073170 0x7f072015c780 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 -- 172.21.15.129:0/3793885543 >> 172.21.15.129:0/3793885543 conn(0x7f07200b1cb0 msgr2=0x7f07200b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 -- 172.21.15.129:0/3793885543 shutdown_connections 2024-09-17T03:52:43.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.185+0000 7f0727200640 1 -- 172.21.15.129:0/3793885543 wait complete. 2024-09-17T03:52:43.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.187+0000 7f0727200640 1 Processor -- start 2024-09-17T03:52:43.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.188+0000 7f0727200640 1 -- start start 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 0x7f0720150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0720073170 0x7f07201e94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0720075050 con 0x7f0720070440 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0720074ed0 con 0x7f0720073170 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0727200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f072006f5a0 con 0x7f0720070030 2024-09-17T03:52:43.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0717400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0717400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35078/0 (socket says 172.21.15.129:35078) 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.189+0000 7f0717400640 1 -- 172.21.15.129:0/2456600684 learned_addr learned my addr 172.21.15.129:0/2456600684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0725200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0720073170 0x7f07201e94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f071fe00640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 0x7f0720150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 -- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 msgr2=0x7f0720150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 0x7f0720150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 -- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0720073170 msgr2=0x7f07201e94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0720073170 0x7f07201e94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 -- 172.21.15.129:0/2456600684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0720150620 con 0x7f0720070440 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f071fe00640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 0x7f0720150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0717400640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f070800d6d0 tx=0x7f070800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708017070 con 0x7f0720070440 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0708005310 con 0x7f0720070440 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07201f60a0 con 0x7f0720070440 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.190+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0708009820 con 0x7f0720070440 2024-09-17T03:52:43.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.191+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07201f6660 con 0x7f0720070440 2024-09-17T03:52:43.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.192+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07201f5bc0 con 0x7f0720070440 2024-09-17T03:52:43.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.193+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f07080099c0 con 0x7f0720070440 2024-09-17T03:52:43.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.194+0000 7f071cc00640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 0x7f06fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.194+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f070809af10 con 0x7f0720070440 2024-09-17T03:52:43.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.195+0000 7f071fe00640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 0x7f06fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.196+0000 7f071fe00640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 0x7f06fc07b0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0710033000 tx=0x7f0710004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:43.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.199+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f07201f5bc0 con 0x7f0720070440 2024-09-17T03:52:43.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.378+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f072006fae0 con 0x7f06fc078b90 2024-09-17T03:52:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.384+0000 7f071cc00640 1 -- 172.21.15.129:0/2456600684 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250824 (secure 0 0 0) 0x7f072006fae0 con 0x7f06fc078b90 2024-09-17T03:52:43.387 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 msgr2=0x7f06fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 0x7f06fc07b0a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0710033000 tx=0x7f0710004730 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 msgr2=0x7f07201500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f070800d6d0 tx=0x7f070800db90 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 shutdown_connections 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f06fc078b90 0x7f06fc07b0a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0720073170 0x7f07201e94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0720070440 0x7f07201500e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 --2- 172.21.15.129:0/2456600684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0720070030 0x7f0720150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 >> 172.21.15.129:0/2456600684 conn(0x7f07200b1cb0 msgr2=0x7f07200b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 shutdown_connections 2024-09-17T03:52:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.388+0000 7f0727200640 1 -- 172.21.15.129:0/2456600684 wait complete. 2024-09-17T03:52:43.410 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-17T03:52:42.343044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592686,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":136,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256936,"kb_used_data":7424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717784,"statfs":{"total":773094113280,"available":772831010816,"internally_reserved":0,"allocated":7602176,"data_stored":3818666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2318,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.366451"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028430+0000","last_change":"2024-09-17T03:52:39.012533+0000","last_active":"2024-09-17T03:52:39.028430+0000","last_peered":"2024-09-17T03:52:39.028430+0000","last_clean":"2024-09-17T03:52:39.028430+0000","last_became_active":"2024-09-17T03:52:39.012181+0000","last_became_peered":"2024-09-17T03:52:39.012181+0000","last_unstale":"2024-09-17T03:52:39.028430+0000","last_undegraded":"2024-09-17T03:52:39.028430+0000","last_fullsized":"2024-09-17T03:52:39.028430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.995252+0000","last_change":"2024-09-17T03:52:37.995252+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.995252+0000","last_undegraded":"2024-09-17T03:52:37.995252+0000","last_fullsized":"2024-09-17T03:52:37.995252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027134+0000","last_change":"2024-09-17T03:52:39.015268+0000","last_active":"2024-09-17T03:52:39.027134+0000","last_peered":"2024-09-17T03:52:39.027134+0000","last_clean":"2024-09-17T03:52:39.027134+0000","last_became_active":"2024-09-17T03:52:39.014959+0000","last_became_peered":"2024-09-17T03:52:39.014959+0000","last_unstale":"2024-09-17T03:52:39.027134+0000","last_undegraded":"2024-09-17T03:52:39.027134+0000","last_fullsized":"2024-09-17T03:52:39.027134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:48.400686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.026039+0000","last_change":"2024-09-17T03:52:39.022874+0000","last_active":"2024-09-17T03:52:39.026039+0000","last_peered":"2024-09-17T03:52:39.026039+0000","last_clean":"2024-09-17T03:52:39.026039+0000","last_became_active":"2024-09-17T03:52:39.022102+0000","last_became_peered":"2024-09-17T03:52:39.022102+0000","last_unstale":"2024-09-17T03:52:39.026039+0000","last_undegraded":"2024-09-17T03:52:39.026039+0000","last_fullsized":"2024-09-17T03:52:39.026039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:14.538536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.016705+0000","last_change":"2024-09-17T03:52:39.005338+0000","last_active":"2024-09-17T03:52:39.016705+0000","last_peered":"2024-09-17T03:52:39.016705+0000","last_clean":"2024-09-17T03:52:39.016705+0000","last_became_active":"2024-09-17T03:52:39.005016+0000","last_became_peered":"2024-09-17T03:52:39.005016+0000","last_unstale":"2024-09-17T03:52:39.016705+0000","last_undegraded":"2024-09-17T03:52:39.016705+0000","last_fullsized":"2024-09-17T03:52:39.016705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:39.136105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989194+0000","last_change":"2024-09-17T03:52:37.989194+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989194+0000","last_undegraded":"2024-09-17T03:52:37.989194+0000","last_fullsized":"2024-09-17T03:52:37.989194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030682+0000","last_change":"2024-09-17T03:52:39.026308+0000","last_active":"2024-09-17T03:52:39.030682+0000","last_peered":"2024-09-17T03:52:39.030682+0000","last_clean":"2024-09-17T03:52:39.030682+0000","last_became_active":"2024-09-17T03:52:39.025753+0000","last_became_peered":"2024-09-17T03:52:39.025753+0000","last_unstale":"2024-09-17T03:52:39.030682+0000","last_undegraded":"2024-09-17T03:52:39.030682+0000","last_fullsized":"2024-09-17T03:52:39.030682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989186+0000","last_change":"2024-09-17T03:52:37.989186+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989186+0000","last_undegraded":"2024-09-17T03:52:37.989186+0000","last_fullsized":"2024-09-17T03:52:37.989186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025209+0000","last_change":"2024-09-17T03:52:39.014691+0000","last_active":"2024-09-17T03:52:39.025209+0000","last_peered":"2024-09-17T03:52:39.025209+0000","last_clean":"2024-09-17T03:52:39.025209+0000","last_became_active":"2024-09-17T03:52:39.010250+0000","last_became_peered":"2024-09-17T03:52:39.010250+0000","last_unstale":"2024-09-17T03:52:39.025209+0000","last_undegraded":"2024-09-17T03:52:39.025209+0000","last_fullsized":"2024-09-17T03:52:39.025209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:28.603948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036735+0000","last_change":"2024-09-17T03:52:39.030566+0000","last_active":"2024-09-17T03:52:39.036735+0000","last_peered":"2024-09-17T03:52:39.036735+0000","last_clean":"2024-09-17T03:52:39.036735+0000","last_became_active":"2024-09-17T03:52:39.029885+0000","last_became_peered":"2024-09-17T03:52:39.029885+0000","last_unstale":"2024-09-17T03:52:39.036735+0000","last_undegraded":"2024-09-17T03:52:39.036735+0000","last_fullsized":"2024-09-17T03:52:39.036735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:15.311759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.032378+0000","last_change":"2024-09-17T03:52:39.027346+0000","last_active":"2024-09-17T03:52:39.032378+0000","last_peered":"2024-09-17T03:52:39.032378+0000","last_clean":"2024-09-17T03:52:39.032378+0000","last_became_active":"2024-09-17T03:52:39.025928+0000","last_became_peered":"2024-09-17T03:52:39.025928+0000","last_unstale":"2024-09-17T03:52:39.032378+0000","last_undegraded":"2024-09-17T03:52:39.032378+0000","last_fullsized":"2024-09-17T03:52:39.032378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:56.381402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024790+0000","last_change":"2024-09-17T03:52:39.018641+0000","last_active":"2024-09-17T03:52:39.024790+0000","last_peered":"2024-09-17T03:52:39.024790+0000","last_clean":"2024-09-17T03:52:39.024790+0000","last_became_active":"2024-09-17T03:52:39.018301+0000","last_became_peered":"2024-09-17T03:52:39.018301+0000","last_unstale":"2024-09-17T03:52:39.024790+0000","last_undegraded":"2024-09-17T03:52:39.024790+0000","last_fullsized":"2024-09-17T03:52:39.024790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:52.014834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989872+0000","last_change":"2024-09-17T03:52:37.989872+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989872+0000","last_undegraded":"2024-09-17T03:52:37.989872+0000","last_fullsized":"2024-09-17T03:52:37.989872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020941+0000","last_change":"2024-09-17T03:52:39.012957+0000","last_active":"2024-09-17T03:52:39.020941+0000","last_peered":"2024-09-17T03:52:39.020941+0000","last_clean":"2024-09-17T03:52:39.020941+0000","last_became_active":"2024-09-17T03:52:39.012674+0000","last_became_peered":"2024-09-17T03:52:39.012674+0000","last_unstale":"2024-09-17T03:52:39.020941+0000","last_undegraded":"2024-09-17T03:52:39.020941+0000","last_fullsized":"2024-09-17T03:52:39.020941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:36.707421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988197+0000","last_change":"2024-09-17T03:52:37.988197+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988197+0000","last_undegraded":"2024-09-17T03:52:37.988197+0000","last_fullsized":"2024-09-17T03:52:37.988197+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037382+0000","last_change":"2024-09-17T03:52:39.034015+0000","last_active":"2024-09-17T03:52:39.037382+0000","last_peered":"2024-09-17T03:52:39.037382+0000","last_clean":"2024-09-17T03:52:39.037382+0000","last_became_active":"2024-09-17T03:52:39.033201+0000","last_became_peered":"2024-09-17T03:52:39.033201+0000","last_unstale":"2024-09-17T03:52:39.037382+0000","last_undegraded":"2024-09-17T03:52:39.037382+0000","last_fullsized":"2024-09-17T03:52:39.037382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:10.085629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025244+0000","last_change":"2024-09-17T03:52:39.015002+0000","last_active":"2024-09-17T03:52:39.025244+0000","last_peered":"2024-09-17T03:52:39.025244+0000","last_clean":"2024-09-17T03:52:39.025244+0000","last_became_active":"2024-09-17T03:52:39.010548+0000","last_became_peered":"2024-09-17T03:52:39.010548+0000","last_unstale":"2024-09-17T03:52:39.025244+0000","last_undegraded":"2024-09-17T03:52:39.025244+0000","last_fullsized":"2024-09-17T03:52:39.025244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:19:23.870446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025955+0000","last_change":"2024-09-17T03:52:39.017861+0000","last_active":"2024-09-17T03:52:39.025955+0000","last_peered":"2024-09-17T03:52:39.025955+0000","last_clean":"2024-09-17T03:52:39.025955+0000","last_became_active":"2024-09-17T03:52:39.017375+0000","last_became_peered":"2024-09-17T03:52:39.017375+0000","last_unstale":"2024-09-17T03:52:39.025955+0000","last_undegraded":"2024-09-17T03:52:39.025955+0000","last_fullsized":"2024-09-17T03:52:39.025955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:41.879432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.994267+0000","last_change":"2024-09-17T03:52:37.994267+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.994267+0000","last_undegraded":"2024-09-17T03:52:37.994267+0000","last_fullsized":"2024-09-17T03:52:37.994267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988999+0000","last_change":"2024-09-17T03:52:37.988999+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988999+0000","last_undegraded":"2024-09-17T03:52:37.988999+0000","last_fullsized":"2024-09-17T03:52:37.988999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036307+0000","last_change":"2024-09-17T03:52:39.027699+0000","last_active":"2024-09-17T03:52:39.036307+0000","last_peered":"2024-09-17T03:52:39.036307+0000","last_clean":"2024-09-17T03:52:39.036307+0000","last_became_active":"2024-09-17T03:52:39.026422+0000","last_became_peered":"2024-09-17T03:52:39.026422+0000","last_unstale":"2024-09-17T03:52:39.036307+0000","last_undegraded":"2024-09-17T03:52:39.036307+0000","last_fullsized":"2024-09-17T03:52:39.036307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:09.586045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036731+0000","last_change":"2024-09-17T03:52:39.030179+0000","last_active":"2024-09-17T03:52:39.036731+0000","last_peered":"2024-09-17T03:52:39.036731+0000","last_clean":"2024-09-17T03:52:39.036731+0000","last_became_active":"2024-09-17T03:52:39.029845+0000","last_became_peered":"2024-09-17T03:52:39.029845+0000","last_unstale":"2024-09-17T03:52:39.036731+0000","last_undegraded":"2024-09-17T03:52:39.036731+0000","last_fullsized":"2024-09-17T03:52:39.036731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:59.212278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.017804+0000","last_change":"2024-09-17T03:52:39.013300+0000","last_active":"2024-09-17T03:52:39.017804+0000","last_peered":"2024-09-17T03:52:39.017804+0000","last_clean":"2024-09-17T03:52:39.017804+0000","last_became_active":"2024-09-17T03:52:39.012947+0000","last_became_peered":"2024-09-17T03:52:39.012947+0000","last_unstale":"2024-09-17T03:52:39.017804+0000","last_undegraded":"2024-09-17T03:52:39.017804+0000","last_fullsized":"2024-09-17T03:52:39.017804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:51.900838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.032253+0000","last_change":"2024-09-17T03:52:39.021402+0000","last_active":"2024-09-17T03:52:39.032253+0000","last_peered":"2024-09-17T03:52:39.032253+0000","last_clean":"2024-09-17T03:52:39.032253+0000","last_became_active":"2024-09-17T03:52:39.021091+0000","last_became_peered":"2024-09-17T03:52:39.021091+0000","last_unstale":"2024-09-17T03:52:39.032253+0000","last_undegraded":"2024-09-17T03:52:39.032253+0000","last_fullsized":"2024-09-17T03:52:39.032253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:25.145657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037049+0000","last_change":"2024-09-17T03:52:39.034509+0000","last_active":"2024-09-17T03:52:39.037049+0000","last_peered":"2024-09-17T03:52:39.037049+0000","last_clean":"2024-09-17T03:52:39.037049+0000","last_became_active":"2024-09-17T03:52:39.033549+0000","last_became_peered":"2024-09-17T03:52:39.033549+0000","last_unstale":"2024-09-17T03:52:39.037049+0000","last_undegraded":"2024-09-17T03:52:39.037049+0000","last_fullsized":"2024-09-17T03:52:39.037049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:04.194058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028726+0000","last_change":"2024-09-17T03:52:39.021427+0000","last_active":"2024-09-17T03:52:39.028726+0000","last_peered":"2024-09-17T03:52:39.028726+0000","last_clean":"2024-09-17T03:52:39.028726+0000","last_became_active":"2024-09-17T03:52:39.020901+0000","last_became_peered":"2024-09-17T03:52:39.020901+0000","last_unstale":"2024-09-17T03:52:39.028726+0000","last_undegraded":"2024-09-17T03:52:39.028726+0000","last_fullsized":"2024-09-17T03:52:39.028726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:27.084828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028134+0000","last_change":"2024-09-17T03:52:39.024384+0000","last_active":"2024-09-17T03:52:39.028134+0000","last_peered":"2024-09-17T03:52:39.028134+0000","last_clean":"2024-09-17T03:52:39.028134+0000","last_became_active":"2024-09-17T03:52:39.015719+0000","last_became_peered":"2024-09-17T03:52:39.015719+0000","last_unstale":"2024-09-17T03:52:39.028134+0000","last_undegraded":"2024-09-17T03:52:39.028134+0000","last_fullsized":"2024-09-17T03:52:39.028134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:00.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036787+0000","last_change":"2024-09-17T03:52:39.031532+0000","last_active":"2024-09-17T03:52:39.036787+0000","last_peered":"2024-09-17T03:52:39.036787+0000","last_clean":"2024-09-17T03:52:39.036787+0000","last_became_active":"2024-09-17T03:52:39.031235+0000","last_became_peered":"2024-09-17T03:52:39.031235+0000","last_unstale":"2024-09-17T03:52:39.036787+0000","last_undegraded":"2024-09-17T03:52:39.036787+0000","last_fullsized":"2024-09-17T03:52:39.036787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:51.694842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025053+0000","last_change":"2024-09-17T03:52:39.013885+0000","last_active":"2024-09-17T03:52:39.025053+0000","last_peered":"2024-09-17T03:52:39.025053+0000","last_clean":"2024-09-17T03:52:39.025053+0000","last_became_active":"2024-09-17T03:52:39.013584+0000","last_became_peered":"2024-09-17T03:52:39.013584+0000","last_unstale":"2024-09-17T03:52:39.025053+0000","last_undegraded":"2024-09-17T03:52:39.025053+0000","last_fullsized":"2024-09-17T03:52:39.025053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:11.385767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.019304+0000","last_change":"2024-09-17T03:52:39.016332+0000","last_active":"2024-09-17T03:52:39.019304+0000","last_peered":"2024-09-17T03:52:39.019304+0000","last_clean":"2024-09-17T03:52:39.019304+0000","last_became_active":"2024-09-17T03:52:39.015950+0000","last_became_peered":"2024-09-17T03:52:39.015950+0000","last_unstale":"2024-09-17T03:52:39.019304+0000","last_undegraded":"2024-09-17T03:52:39.019304+0000","last_fullsized":"2024-09-17T03:52:39.019304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:45.685964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028802+0000","last_change":"2024-09-17T03:52:39.026096+0000","last_active":"2024-09-17T03:52:39.028802+0000","last_peered":"2024-09-17T03:52:39.028802+0000","last_clean":"2024-09-17T03:52:39.028802+0000","last_became_active":"2024-09-17T03:52:39.025309+0000","last_became_peered":"2024-09-17T03:52:39.025309+0000","last_unstale":"2024-09-17T03:52:39.028802+0000","last_undegraded":"2024-09-17T03:52:39.028802+0000","last_fullsized":"2024-09-17T03:52:39.028802+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:37.598541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988679+0000","last_change":"2024-09-17T03:52:37.988679+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988679+0000","last_undegraded":"2024-09-17T03:52:37.988679+0000","last_fullsized":"2024-09-17T03:52:37.988679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030874+0000","last_change":"2024-09-17T03:52:39.023210+0000","last_active":"2024-09-17T03:52:39.030874+0000","last_peered":"2024-09-17T03:52:39.030874+0000","last_clean":"2024-09-17T03:52:39.030874+0000","last_became_active":"2024-09-17T03:52:39.022850+0000","last_became_peered":"2024-09-17T03:52:39.022850+0000","last_unstale":"2024-09-17T03:52:39.030874+0000","last_undegraded":"2024-09-17T03:52:39.030874+0000","last_fullsized":"2024-09-17T03:52:39.030874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.593260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030431+0000","last_change":"2024-09-17T03:52:38.004383+0000","last_active":"2024-09-17T03:52:39.030431+0000","last_peered":"2024-09-17T03:52:39.030431+0000","last_clean":"2024-09-17T03:52:39.030431+0000","last_became_active":"2024-09-17T03:52:38.003610+0000","last_became_peered":"2024-09-17T03:52:38.003610+0000","last_unstale":"2024-09-17T03:52:39.030431+0000","last_undegraded":"2024-09-17T03:52:39.030431+0000","last_fullsized":"2024-09-17T03:52:39.030431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:36.838884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.029038+0000","last_change":"2024-09-17T03:52:39.017710+0000","last_active":"2024-09-17T03:52:39.029038+0000","last_peered":"2024-09-17T03:52:39.029038+0000","last_clean":"2024-09-17T03:52:39.029038+0000","last_became_active":"2024-09-17T03:52:39.009010+0000","last_became_peered":"2024-09-17T03:52:39.009010+0000","last_unstale":"2024-09-17T03:52:39.029038+0000","last_undegraded":"2024-09-17T03:52:39.029038+0000","last_fullsized":"2024-09-17T03:52:39.029038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:36.016315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:37.996051+0000","last_change":"2024-09-17T03:52:37.987588+0000","last_active":"2024-09-17T03:52:37.996051+0000","last_peered":"2024-09-17T03:52:37.996051+0000","last_clean":"2024-09-17T03:52:37.996051+0000","last_became_active":"2024-09-17T03:52:37.986620+0000","last_became_peered":"2024-09-17T03:52:37.986620+0000","last_unstale":"2024-09-17T03:52:37.996051+0000","last_undegraded":"2024-09-17T03:52:37.996051+0000","last_fullsized":"2024-09-17T03:52:37.996051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:26.467016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.999829+0000","last_change":"2024-09-17T03:52:37.999829+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.999829+0000","last_undegraded":"2024-09-17T03:52:37.999829+0000","last_fullsized":"2024-09-17T03:52:37.999829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.033573+0000","last_change":"2024-09-17T03:52:38.008448+0000","last_active":"2024-09-17T03:52:39.033573+0000","last_peered":"2024-09-17T03:52:39.033573+0000","last_clean":"2024-09-17T03:52:39.033573+0000","last_became_active":"2024-09-17T03:52:38.008134+0000","last_became_peered":"2024-09-17T03:52:38.008134+0000","last_unstale":"2024-09-17T03:52:39.033573+0000","last_undegraded":"2024-09-17T03:52:39.033573+0000","last_fullsized":"2024-09-17T03:52:39.033573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:05.150798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.014491+0000","last_change":"2024-09-17T03:52:39.009452+0000","last_active":"2024-09-17T03:52:39.014491+0000","last_peered":"2024-09-17T03:52:39.014491+0000","last_clean":"2024-09-17T03:52:39.014491+0000","last_became_active":"2024-09-17T03:52:39.005725+0000","last_became_peered":"2024-09-17T03:52:39.005725+0000","last_unstale":"2024-09-17T03:52:39.014491+0000","last_undegraded":"2024-09-17T03:52:39.014491+0000","last_fullsized":"2024-09-17T03:52:39.014491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009240+0000","last_change":"2024-09-17T03:52:38.004751+0000","last_active":"2024-09-17T03:52:38.009240+0000","last_peered":"2024-09-17T03:52:38.009240+0000","last_clean":"2024-09-17T03:52:38.009240+0000","last_became_active":"2024-09-17T03:52:38.004436+0000","last_became_peered":"2024-09-17T03:52:38.004436+0000","last_unstale":"2024-09-17T03:52:38.009240+0000","last_undegraded":"2024-09-17T03:52:38.009240+0000","last_fullsized":"2024-09-17T03:52:38.009240+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:18.872377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027974+0000","last_change":"2024-09-17T03:52:39.023295+0000","last_active":"2024-09-17T03:52:39.027974+0000","last_peered":"2024-09-17T03:52:39.027974+0000","last_clean":"2024-09-17T03:52:39.027974+0000","last_became_active":"2024-09-17T03:52:39.022209+0000","last_became_peered":"2024-09-17T03:52:39.022209+0000","last_unstale":"2024-09-17T03:52:39.027974+0000","last_undegraded":"2024-09-17T03:52:39.027974+0000","last_fullsized":"2024-09-17T03:52:39.027974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:37.845837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036576+0000","last_change":"2024-09-17T03:52:38.004706+0000","last_active":"2024-09-17T03:52:39.036576+0000","last_peered":"2024-09-17T03:52:39.036576+0000","last_clean":"2024-09-17T03:52:39.036576+0000","last_became_active":"2024-09-17T03:52:38.004384+0000","last_became_peered":"2024-09-17T03:52:38.004384+0000","last_unstale":"2024-09-17T03:52:39.036576+0000","last_undegraded":"2024-09-17T03:52:39.036576+0000","last_fullsized":"2024-09-17T03:52:39.036576+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:24.947156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036975+0000","last_change":"2024-09-17T03:52:39.033871+0000","last_active":"2024-09-17T03:52:39.036975+0000","last_peered":"2024-09-17T03:52:39.036975+0000","last_clean":"2024-09-17T03:52:39.036975+0000","last_became_active":"2024-09-17T03:52:39.033350+0000","last_became_peered":"2024-09-17T03:52:39.033350+0000","last_unstale":"2024-09-17T03:52:39.036975+0000","last_undegraded":"2024-09-17T03:52:39.036975+0000","last_fullsized":"2024-09-17T03:52:39.036975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:28.281361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009976+0000","last_change":"2024-09-17T03:52:38.007787+0000","last_active":"2024-09-17T03:52:38.009976+0000","last_peered":"2024-09-17T03:52:38.009976+0000","last_clean":"2024-09-17T03:52:38.009976+0000","last_became_active":"2024-09-17T03:52:38.007287+0000","last_became_peered":"2024-09-17T03:52:38.007287+0000","last_unstale":"2024-09-17T03:52:38.009976+0000","last_undegraded":"2024-09-17T03:52:38.009976+0000","last_fullsized":"2024-09-17T03:52:38.009976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:37.943429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.031819+0000","last_change":"2024-09-17T03:52:39.026791+0000","last_active":"2024-09-17T03:52:39.031819+0000","last_peered":"2024-09-17T03:52:39.031819+0000","last_clean":"2024-09-17T03:52:39.031819+0000","last_became_active":"2024-09-17T03:52:39.026369+0000","last_became_peered":"2024-09-17T03:52:39.026369+0000","last_unstale":"2024-09-17T03:52:39.031819+0000","last_undegraded":"2024-09-17T03:52:39.031819+0000","last_fullsized":"2024-09-17T03:52:39.031819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:40.919599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998830+0000","last_change":"2024-09-17T03:52:38.004637+0000","last_active":"2024-09-17T03:52:38.998830+0000","last_peered":"2024-09-17T03:52:38.998830+0000","last_clean":"2024-09-17T03:52:38.998830+0000","last_became_active":"2024-09-17T03:52:38.004076+0000","last_became_peered":"2024-09-17T03:52:38.004076+0000","last_unstale":"2024-09-17T03:52:38.998830+0000","last_undegraded":"2024-09-17T03:52:38.998830+0000","last_fullsized":"2024-09-17T03:52:38.998830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:38.214053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987688+0000","last_change":"2024-09-17T03:52:37.987688+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987688+0000","last_undegraded":"2024-09-17T03:52:37.987688+0000","last_fullsized":"2024-09-17T03:52:37.987688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.997273+0000","last_change":"2024-09-17T03:52:37.992784+0000","last_active":"2024-09-17T03:52:38.997273+0000","last_peered":"2024-09-17T03:52:38.997273+0000","last_clean":"2024-09-17T03:52:38.997273+0000","last_became_active":"2024-09-17T03:52:37.992480+0000","last_became_peered":"2024-09-17T03:52:37.992480+0000","last_unstale":"2024-09-17T03:52:38.997273+0000","last_undegraded":"2024-09-17T03:52:38.997273+0000","last_fullsized":"2024-09-17T03:52:38.997273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:21.250972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.012794+0000","last_change":"2024-09-17T03:52:39.008286+0000","last_active":"2024-09-17T03:52:39.012794+0000","last_peered":"2024-09-17T03:52:39.012794+0000","last_clean":"2024-09-17T03:52:39.012794+0000","last_became_active":"2024-09-17T03:52:39.007952+0000","last_became_peered":"2024-09-17T03:52:39.007952+0000","last_unstale":"2024-09-17T03:52:39.012794+0000","last_undegraded":"2024-09-17T03:52:39.012794+0000","last_fullsized":"2024-09-17T03:52:39.012794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:10.085629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020500+0000","last_change":"2024-09-17T03:52:38.001450+0000","last_active":"2024-09-17T03:52:39.020500+0000","last_peered":"2024-09-17T03:52:39.020500+0000","last_clean":"2024-09-17T03:52:39.020500+0000","last_became_active":"2024-09-17T03:52:38.001124+0000","last_became_peered":"2024-09-17T03:52:38.001124+0000","last_unstale":"2024-09-17T03:52:39.020500+0000","last_undegraded":"2024-09-17T03:52:39.020500+0000","last_fullsized":"2024-09-17T03:52:39.020500+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:53.092676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030972+0000","last_change":"2024-09-17T03:52:39.027159+0000","last_active":"2024-09-17T03:52:39.030972+0000","last_peered":"2024-09-17T03:52:39.030972+0000","last_clean":"2024-09-17T03:52:39.030972+0000","last_became_active":"2024-09-17T03:52:39.026074+0000","last_became_peered":"2024-09-17T03:52:39.026074+0000","last_unstale":"2024-09-17T03:52:39.030972+0000","last_undegraded":"2024-09-17T03:52:39.030972+0000","last_fullsized":"2024-09-17T03:52:39.030972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:32.756863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009258+0000","last_change":"2024-09-17T03:52:38.004849+0000","last_active":"2024-09-17T03:52:38.009258+0000","last_peered":"2024-09-17T03:52:38.009258+0000","last_clean":"2024-09-17T03:52:38.009258+0000","last_became_active":"2024-09-17T03:52:38.002393+0000","last_became_peered":"2024-09-17T03:52:38.002393+0000","last_unstale":"2024-09-17T03:52:38.009258+0000","last_undegraded":"2024-09-17T03:52:38.009258+0000","last_fullsized":"2024-09-17T03:52:38.009258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:15.892290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.996175+0000","last_change":"2024-09-17T03:52:37.996175+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.996175+0000","last_undegraded":"2024-09-17T03:52:37.996175+0000","last_fullsized":"2024-09-17T03:52:37.996175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036085+0000","last_change":"2024-09-17T03:52:38.002263+0000","last_active":"2024-09-17T03:52:39.036085+0000","last_peered":"2024-09-17T03:52:39.036085+0000","last_clean":"2024-09-17T03:52:39.036085+0000","last_became_active":"2024-09-17T03:52:38.001712+0000","last_became_peered":"2024-09-17T03:52:38.001712+0000","last_unstale":"2024-09-17T03:52:39.036085+0000","last_undegraded":"2024-09-17T03:52:39.036085+0000","last_fullsized":"2024-09-17T03:52:39.036085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:53.008257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024340+0000","last_change":"2024-09-17T03:52:39.015946+0000","last_active":"2024-09-17T03:52:39.024340+0000","last_peered":"2024-09-17T03:52:39.024340+0000","last_clean":"2024-09-17T03:52:39.024340+0000","last_became_active":"2024-09-17T03:52:39.008987+0000","last_became_peered":"2024-09-17T03:52:39.008987+0000","last_unstale":"2024-09-17T03:52:39.024340+0000","last_undegraded":"2024-09-17T03:52:39.024340+0000","last_fullsized":"2024-09-17T03:52:39.024340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:19.490365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.040093+0000","last_change":"2024-09-17T03:52:38.000155+0000","last_active":"2024-09-17T03:52:39.040093+0000","last_peered":"2024-09-17T03:52:39.040093+0000","last_clean":"2024-09-17T03:52:39.040093+0000","last_became_active":"2024-09-17T03:52:37.999044+0000","last_became_peered":"2024-09-17T03:52:37.999044+0000","last_unstale":"2024-09-17T03:52:39.040093+0000","last_undegraded":"2024-09-17T03:52:39.040093+0000","last_fullsized":"2024-09-17T03:52:39.040093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:15.512221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.018466+0000","last_change":"2024-09-17T03:52:39.015628+0000","last_active":"2024-09-17T03:52:39.018466+0000","last_peered":"2024-09-17T03:52:39.018466+0000","last_clean":"2024-09-17T03:52:39.018466+0000","last_became_active":"2024-09-17T03:52:39.015286+0000","last_became_peered":"2024-09-17T03:52:39.015286+0000","last_unstale":"2024-09-17T03:52:39.018466+0000","last_undegraded":"2024-09-17T03:52:39.018466+0000","last_fullsized":"2024-09-17T03:52:39.018466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:49.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.034007+0000","last_change":"2024-09-17T03:52:38.001510+0000","last_active":"2024-09-17T03:52:39.034007+0000","last_peered":"2024-09-17T03:52:39.034007+0000","last_clean":"2024-09-17T03:52:39.034007+0000","last_became_active":"2024-09-17T03:52:38.001109+0000","last_became_peered":"2024-09-17T03:52:38.001109+0000","last_unstale":"2024-09-17T03:52:39.034007+0000","last_undegraded":"2024-09-17T03:52:39.034007+0000","last_fullsized":"2024-09-17T03:52:39.034007+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:53.377063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028648+0000","last_change":"2024-09-17T03:52:39.021149+0000","last_active":"2024-09-17T03:52:39.028648+0000","last_peered":"2024-09-17T03:52:39.028648+0000","last_clean":"2024-09-17T03:52:39.028648+0000","last_became_active":"2024-09-17T03:52:39.020787+0000","last_became_peered":"2024-09-17T03:52:39.020787+0000","last_unstale":"2024-09-17T03:52:39.028648+0000","last_undegraded":"2024-09-17T03:52:39.028648+0000","last_fullsized":"2024-09-17T03:52:39.028648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:53.816696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.007189+0000","last_change":"2024-09-17T03:52:38.000222+0000","last_active":"2024-09-17T03:52:38.007189+0000","last_peered":"2024-09-17T03:52:38.007189+0000","last_clean":"2024-09-17T03:52:38.007189+0000","last_became_active":"2024-09-17T03:52:37.998870+0000","last_became_peered":"2024-09-17T03:52:37.998870+0000","last_unstale":"2024-09-17T03:52:38.007189+0000","last_undegraded":"2024-09-17T03:52:38.007189+0000","last_fullsized":"2024-09-17T03:52:38.007189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:44.741147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987073+0000","last_change":"2024-09-17T03:52:37.987073+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987073+0000","last_undegraded":"2024-09-17T03:52:37.987073+0000","last_fullsized":"2024-09-17T03:52:37.987073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.004727+0000","last_change":"2024-09-17T03:52:37.993426+0000","last_active":"2024-09-17T03:52:39.004727+0000","last_peered":"2024-09-17T03:52:39.004727+0000","last_clean":"2024-09-17T03:52:39.004727+0000","last_became_active":"2024-09-17T03:52:37.992558+0000","last_became_peered":"2024-09-17T03:52:37.992558+0000","last_unstale":"2024-09-17T03:52:39.004727+0000","last_undegraded":"2024-09-17T03:52:39.004727+0000","last_fullsized":"2024-09-17T03:52:39.004727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:50.687889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028753+0000","last_change":"2024-09-17T03:52:39.020708+0000","last_active":"2024-09-17T03:52:39.028753+0000","last_peered":"2024-09-17T03:52:39.028753+0000","last_clean":"2024-09-17T03:52:39.028753+0000","last_became_active":"2024-09-17T03:52:39.019989+0000","last_became_peered":"2024-09-17T03:52:39.019989+0000","last_unstale":"2024-09-17T03:52:39.028753+0000","last_undegraded":"2024-09-17T03:52:39.028753+0000","last_fullsized":"2024-09-17T03:52:39.028753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:14.519601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037342+0000","last_change":"2024-09-17T03:52:38.000317+0000","last_active":"2024-09-17T03:52:39.037342+0000","last_peered":"2024-09-17T03:52:39.037342+0000","last_clean":"2024-09-17T03:52:39.037342+0000","last_became_active":"2024-09-17T03:52:37.999933+0000","last_became_peered":"2024-09-17T03:52:37.999933+0000","last_unstale":"2024-09-17T03:52:39.037342+0000","last_undegraded":"2024-09-17T03:52:39.037342+0000","last_fullsized":"2024-09-17T03:52:39.037342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:56.251630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.035161+0000","last_change":"2024-09-17T03:52:39.025015+0000","last_active":"2024-09-17T03:52:39.035161+0000","last_peered":"2024-09-17T03:52:39.035161+0000","last_clean":"2024-09-17T03:52:39.035161+0000","last_became_active":"2024-09-17T03:52:39.024596+0000","last_became_peered":"2024-09-17T03:52:39.024596+0000","last_unstale":"2024-09-17T03:52:39.035161+0000","last_undegraded":"2024-09-17T03:52:39.035161+0000","last_fullsized":"2024-09-17T03:52:39.035161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:05.782615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.010014+0000","last_change":"2024-09-17T03:52:38.007757+0000","last_active":"2024-09-17T03:52:38.010014+0000","last_peered":"2024-09-17T03:52:38.010014+0000","last_clean":"2024-09-17T03:52:38.010014+0000","last_became_active":"2024-09-17T03:52:38.007282+0000","last_became_peered":"2024-09-17T03:52:38.007282+0000","last_unstale":"2024-09-17T03:52:38.010014+0000","last_undegraded":"2024-09-17T03:52:38.010014+0000","last_fullsized":"2024-09-17T03:52:38.010014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:56.655759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020779+0000","last_change":"2024-09-17T03:52:39.007897+0000","last_active":"2024-09-17T03:52:39.020779+0000","last_peered":"2024-09-17T03:52:39.020779+0000","last_clean":"2024-09-17T03:52:39.020779+0000","last_became_active":"2024-09-17T03:52:39.007557+0000","last_became_peered":"2024-09-17T03:52:39.007557+0000","last_unstale":"2024-09-17T03:52:39.020779+0000","last_undegraded":"2024-09-17T03:52:39.020779+0000","last_fullsized":"2024-09-17T03:52:39.020779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.033549+0000","last_change":"2024-09-17T03:52:38.002206+0000","last_active":"2024-09-17T03:52:39.033549+0000","last_peered":"2024-09-17T03:52:39.033549+0000","last_clean":"2024-09-17T03:52:39.033549+0000","last_became_active":"2024-09-17T03:52:38.001872+0000","last_became_peered":"2024-09-17T03:52:38.001872+0000","last_unstale":"2024-09-17T03:52:39.033549+0000","last_undegraded":"2024-09-17T03:52:39.033549+0000","last_fullsized":"2024-09-17T03:52:39.033549+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:18.787765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024818+0000","last_change":"2024-09-17T03:52:39.018968+0000","last_active":"2024-09-17T03:52:39.024818+0000","last_peered":"2024-09-17T03:52:39.024818+0000","last_clean":"2024-09-17T03:52:39.024818+0000","last_became_active":"2024-09-17T03:52:39.018682+0000","last_became_peered":"2024-09-17T03:52:39.018682+0000","last_unstale":"2024-09-17T03:52:39.024818+0000","last_undegraded":"2024-09-17T03:52:39.024818+0000","last_fullsized":"2024-09-17T03:52:39.024818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:33:04.970156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.011962+0000","last_change":"2024-09-17T03:52:38.009348+0000","last_active":"2024-09-17T03:52:38.011962+0000","last_peered":"2024-09-17T03:52:38.011962+0000","last_clean":"2024-09-17T03:52:38.011962+0000","last_became_active":"2024-09-17T03:52:38.008926+0000","last_became_peered":"2024-09-17T03:52:38.008926+0000","last_unstale":"2024-09-17T03:52:38.011962+0000","last_undegraded":"2024-09-17T03:52:38.011962+0000","last_fullsized":"2024-09-17T03:52:38.011962+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:03:16.326145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028893+0000","last_change":"2024-09-17T03:52:39.021118+0000","last_active":"2024-09-17T03:52:39.028893+0000","last_peered":"2024-09-17T03:52:39.028893+0000","last_clean":"2024-09-17T03:52:39.028893+0000","last_became_active":"2024-09-17T03:52:39.020763+0000","last_became_peered":"2024-09-17T03:52:39.020763+0000","last_unstale":"2024-09-17T03:52:39.028893+0000","last_undegraded":"2024-09-17T03:52:39.028893+0000","last_fullsized":"2024-09-17T03:52:39.028893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:50.085185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.997610+0000","last_change":"2024-09-17T03:52:38.005886+0000","last_active":"2024-09-17T03:52:38.997610+0000","last_peered":"2024-09-17T03:52:38.997610+0000","last_clean":"2024-09-17T03:52:38.997610+0000","last_became_active":"2024-09-17T03:52:38.005550+0000","last_became_peered":"2024-09-17T03:52:38.005550+0000","last_unstale":"2024-09-17T03:52:38.997610+0000","last_undegraded":"2024-09-17T03:52:38.997610+0000","last_fullsized":"2024-09-17T03:52:38.997610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:36.591587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.014817+0000","last_change":"2024-09-17T03:52:39.009574+0000","last_active":"2024-09-17T03:52:39.014817+0000","last_peered":"2024-09-17T03:52:39.014817+0000","last_clean":"2024-09-17T03:52:39.014817+0000","last_became_active":"2024-09-17T03:52:39.009382+0000","last_became_peered":"2024-09-17T03:52:39.009382+0000","last_unstale":"2024-09-17T03:52:39.014817+0000","last_undegraded":"2024-09-17T03:52:39.014817+0000","last_fullsized":"2024-09-17T03:52:39.014817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:16.138436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009381+0000","last_change":"2024-09-17T03:52:38.006146+0000","last_active":"2024-09-17T03:52:38.009381+0000","last_peered":"2024-09-17T03:52:38.009381+0000","last_clean":"2024-09-17T03:52:38.009381+0000","last_became_active":"2024-09-17T03:52:38.005099+0000","last_became_peered":"2024-09-17T03:52:38.005099+0000","last_unstale":"2024-09-17T03:52:38.009381+0000","last_undegraded":"2024-09-17T03:52:38.009381+0000","last_fullsized":"2024-09-17T03:52:38.009381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:03.004454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027636+0000","last_change":"2024-09-17T03:52:39.020395+0000","last_active":"2024-09-17T03:52:39.027636+0000","last_peered":"2024-09-17T03:52:39.027636+0000","last_clean":"2024-09-17T03:52:39.027636+0000","last_became_active":"2024-09-17T03:52:39.019294+0000","last_became_peered":"2024-09-17T03:52:39.019294+0000","last_unstale":"2024-09-17T03:52:39.027636+0000","last_undegraded":"2024-09-17T03:52:39.027636+0000","last_fullsized":"2024-09-17T03:52:39.027636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:48.684398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.006539+0000","last_change":"2024-09-17T03:52:37.986744+0000","last_active":"2024-09-17T03:52:39.006539+0000","last_peered":"2024-09-17T03:52:39.006539+0000","last_clean":"2024-09-17T03:52:39.006539+0000","last_became_active":"2024-09-17T03:52:37.986385+0000","last_became_peered":"2024-09-17T03:52:37.986385+0000","last_unstale":"2024-09-17T03:52:39.006539+0000","last_undegraded":"2024-09-17T03:52:39.006539+0000","last_fullsized":"2024-09-17T03:52:39.006539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.274408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027469+0000","last_change":"2024-09-17T03:52:39.019842+0000","last_active":"2024-09-17T03:52:39.027469+0000","last_peered":"2024-09-17T03:52:39.027469+0000","last_clean":"2024-09-17T03:52:39.027469+0000","last_became_active":"2024-09-17T03:52:39.019548+0000","last_became_peered":"2024-09-17T03:52:39.019548+0000","last_unstale":"2024-09-17T03:52:39.027469+0000","last_undegraded":"2024-09-17T03:52:39.027469+0000","last_fullsized":"2024-09-17T03:52:39.027469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:28.088118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:37.995971+0000","last_change":"2024-09-17T03:52:37.987589+0000","last_active":"2024-09-17T03:52:37.995971+0000","last_peered":"2024-09-17T03:52:37.995971+0000","last_clean":"2024-09-17T03:52:37.995971+0000","last_became_active":"2024-09-17T03:52:37.986646+0000","last_became_peered":"2024-09-17T03:52:37.986646+0000","last_unstale":"2024-09-17T03:52:37.995971+0000","last_undegraded":"2024-09-17T03:52:37.995971+0000","last_fullsized":"2024-09-17T03:52:37.995971+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:18.770567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987171+0000","last_change":"2024-09-17T03:52:37.987171+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987171+0000","last_undegraded":"2024-09-17T03:52:37.987171+0000","last_fullsized":"2024-09-17T03:52:37.987171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009981+0000","last_change":"2024-09-17T03:52:38.007301+0000","last_active":"2024-09-17T03:52:38.009981+0000","last_peered":"2024-09-17T03:52:38.009981+0000","last_clean":"2024-09-17T03:52:38.009981+0000","last_became_active":"2024-09-17T03:52:38.006871+0000","last_became_peered":"2024-09-17T03:52:38.006871+0000","last_unstale":"2024-09-17T03:52:38.009981+0000","last_undegraded":"2024-09-17T03:52:38.009981+0000","last_fullsized":"2024-09-17T03:52:38.009981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:48:05.016401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.019091+0000","last_change":"2024-09-17T03:52:39.015261+0000","last_active":"2024-09-17T03:52:39.019091+0000","last_peered":"2024-09-17T03:52:39.019091+0000","last_clean":"2024-09-17T03:52:39.019091+0000","last_became_active":"2024-09-17T03:52:39.014907+0000","last_became_peered":"2024-09-17T03:52:39.014907+0000","last_unstale":"2024-09-17T03:52:39.019091+0000","last_undegraded":"2024-09-17T03:52:39.019091+0000","last_fullsized":"2024-09-17T03:52:39.019091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:04.194058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.003843+0000","last_change":"2024-09-17T03:52:37.993729+0000","last_active":"2024-09-17T03:52:39.003843+0000","last_peered":"2024-09-17T03:52:39.003843+0000","last_clean":"2024-09-17T03:52:39.003843+0000","last_became_active":"2024-09-17T03:52:37.991713+0000","last_became_peered":"2024-09-17T03:52:37.991713+0000","last_unstale":"2024-09-17T03:52:39.003843+0000","last_undegraded":"2024-09-17T03:52:39.003843+0000","last_fullsized":"2024-09-17T03:52:39.003843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:14.304806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.031113+0000","last_change":"2024-09-17T03:52:39.024919+0000","last_active":"2024-09-17T03:52:39.031113+0000","last_peered":"2024-09-17T03:52:39.031113+0000","last_clean":"2024-09-17T03:52:39.031113+0000","last_became_active":"2024-09-17T03:52:39.022961+0000","last_became_peered":"2024-09-17T03:52:39.022961+0000","last_unstale":"2024-09-17T03:52:39.031113+0000","last_undegraded":"2024-09-17T03:52:39.031113+0000","last_fullsized":"2024-09-17T03:52:39.031113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:30.540116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998737+0000","last_change":"2024-09-17T03:52:38.002436+0000","last_active":"2024-09-17T03:52:38.998737+0000","last_peered":"2024-09-17T03:52:38.998737+0000","last_clean":"2024-09-17T03:52:38.998737+0000","last_became_active":"2024-09-17T03:52:38.001380+0000","last_became_peered":"2024-09-17T03:52:38.001380+0000","last_unstale":"2024-09-17T03:52:38.998737+0000","last_undegraded":"2024-09-17T03:52:38.998737+0000","last_fullsized":"2024-09-17T03:52:38.998737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:16.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.017896+0000","last_change":"2024-09-17T03:52:39.013637+0000","last_active":"2024-09-17T03:52:39.017896+0000","last_peered":"2024-09-17T03:52:39.017896+0000","last_clean":"2024-09-17T03:52:39.017896+0000","last_became_active":"2024-09-17T03:52:39.013035+0000","last_became_peered":"2024-09-17T03:52:39.013035+0000","last_unstale":"2024-09-17T03:52:39.017896+0000","last_undegraded":"2024-09-17T03:52:39.017896+0000","last_fullsized":"2024-09-17T03:52:39.017896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:56:57.011981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.001930+0000","last_change":"2024-09-17T03:52:37.998007+0000","last_active":"2024-09-17T03:52:38.001930+0000","last_peered":"2024-09-17T03:52:38.001930+0000","last_clean":"2024-09-17T03:52:38.001930+0000","last_became_active":"2024-09-17T03:52:37.997670+0000","last_became_peered":"2024-09-17T03:52:37.997670+0000","last_unstale":"2024-09-17T03:52:38.001930+0000","last_undegraded":"2024-09-17T03:52:38.001930+0000","last_fullsized":"2024-09-17T03:52:38.001930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:14.989087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024301+0000","last_change":"2024-09-17T03:52:39.016581+0000","last_active":"2024-09-17T03:52:39.024301+0000","last_peered":"2024-09-17T03:52:39.024301+0000","last_clean":"2024-09-17T03:52:39.024301+0000","last_became_active":"2024-09-17T03:52:39.007886+0000","last_became_peered":"2024-09-17T03:52:39.007886+0000","last_unstale":"2024-09-17T03:52:39.024301+0000","last_undegraded":"2024-09-17T03:52:39.024301+0000","last_fullsized":"2024-09-17T03:52:39.024301+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:00.893129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998182+0000","last_change":"2024-09-17T03:52:37.991052+0000","last_active":"2024-09-17T03:52:38.998182+0000","last_peered":"2024-09-17T03:52:38.998182+0000","last_clean":"2024-09-17T03:52:38.998182+0000","last_became_active":"2024-09-17T03:52:37.990089+0000","last_became_peered":"2024-09-17T03:52:37.990089+0000","last_unstale":"2024-09-17T03:52:38.998182+0000","last_undegraded":"2024-09-17T03:52:38.998182+0000","last_fullsized":"2024-09-17T03:52:38.998182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:17.450063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988927+0000","last_change":"2024-09-17T03:52:37.988927+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988927+0000","last_undegraded":"2024-09-17T03:52:37.988927+0000","last_fullsized":"2024-09-17T03:52:37.988927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998673+0000","last_change":"2024-09-17T03:52:38.006002+0000","last_active":"2024-09-17T03:52:38.998673+0000","last_peered":"2024-09-17T03:52:38.998673+0000","last_clean":"2024-09-17T03:52:38.998673+0000","last_became_active":"2024-09-17T03:52:38.004725+0000","last_became_peered":"2024-09-17T03:52:38.004725+0000","last_unstale":"2024-09-17T03:52:38.998673+0000","last_undegraded":"2024-09-17T03:52:38.998673+0000","last_fullsized":"2024-09-17T03:52:38.998673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:05.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.015069+0000","last_change":"2024-09-17T03:52:39.010421+0000","last_active":"2024-09-17T03:52:39.015069+0000","last_peered":"2024-09-17T03:52:39.015069+0000","last_clean":"2024-09-17T03:52:39.015069+0000","last_became_active":"2024-09-17T03:52:39.010077+0000","last_became_peered":"2024-09-17T03:52:39.010077+0000","last_unstale":"2024-09-17T03:52:39.015069+0000","last_undegraded":"2024-09-17T03:52:39.015069+0000","last_fullsized":"2024-09-17T03:52:39.015069+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:28.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.016750+0000","last_change":"2024-09-17T03:52:37.998870+0000","last_active":"2024-09-17T03:52:39.016750+0000","last_peered":"2024-09-17T03:52:39.016750+0000","last_clean":"2024-09-17T03:52:39.016750+0000","last_became_active":"2024-09-17T03:52:37.998561+0000","last_became_peered":"2024-09-17T03:52:37.998561+0000","last_unstale":"2024-09-17T03:52:39.016750+0000","last_undegraded":"2024-09-17T03:52:39.016750+0000","last_fullsized":"2024-09-17T03:52:39.016750+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:58.681513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988483+0000","last_change":"2024-09-17T03:52:37.988483+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988483+0000","last_undegraded":"2024-09-17T03:52:37.988483+0000","last_fullsized":"2024-09-17T03:52:37.988483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009258+0000","last_change":"2024-09-17T03:52:38.004100+0000","last_active":"2024-09-17T03:52:38.009258+0000","last_peered":"2024-09-17T03:52:38.009258+0000","last_clean":"2024-09-17T03:52:38.009258+0000","last_became_active":"2024-09-17T03:52:38.000956+0000","last_became_peered":"2024-09-17T03:52:38.000956+0000","last_unstale":"2024-09-17T03:52:38.009258+0000","last_undegraded":"2024-09-17T03:52:38.009258+0000","last_fullsized":"2024-09-17T03:52:38.009258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:47.053179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024906+0000","last_change":"2024-09-17T03:52:39.013661+0000","last_active":"2024-09-17T03:52:39.024906+0000","last_peered":"2024-09-17T03:52:39.024906+0000","last_clean":"2024-09-17T03:52:39.024906+0000","last_became_active":"2024-09-17T03:52:39.013380+0000","last_became_peered":"2024-09-17T03:52:39.013380+0000","last_unstale":"2024-09-17T03:52:39.024906+0000","last_undegraded":"2024-09-17T03:52:39.024906+0000","last_fullsized":"2024-09-17T03:52:39.024906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:59.949458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.007029+0000","last_change":"2024-09-17T03:52:38.001315+0000","last_active":"2024-09-17T03:52:38.007029+0000","last_peered":"2024-09-17T03:52:38.007029+0000","last_clean":"2024-09-17T03:52:38.007029+0000","last_became_active":"2024-09-17T03:52:37.999404+0000","last_became_peered":"2024-09-17T03:52:37.999404+0000","last_unstale":"2024-09-17T03:52:38.007029+0000","last_undegraded":"2024-09-17T03:52:38.007029+0000","last_fullsized":"2024-09-17T03:52:38.007029+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:25:50.399405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":192,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.995809+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:52:38.995809+0000","last_peered":"2024-09-17T03:52:38.995809+0000","last_clean":"2024-09-17T03:52:38.995809+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:52:38.995809+0000","last_undegraded":"2024-09-17T03:52:38.995809+0000","last_fullsized":"2024-09-17T03:52:38.995809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2318,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":917873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509484,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":334280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":819200,"data_stored":335892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":336508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1376256,"data_stored":911598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:52:43.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 -- 172.21.15.129:0/3232075491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070440 msgr2=0x7ff720074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 --2- 172.21.15.129:0/3232075491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070440 0x7ff720074b80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7ff7200710f0 tx=0x7ff71802aa10 comp rx=0 tx=0).stop 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 -- 172.21.15.129:0/3232075491 shutdown_connections 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 --2- 172.21.15.129:0/3232075491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070440 0x7ff720074b80 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 --2- 172.21.15.129:0/3232075491 >> v2:172.21.15.181:3300/0 conn(0x7ff720070030 0x7ff720074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.670+0000 7ff727c00640 1 -- 172.21.15.129:0/3232075491 >> 172.21.15.129:0/3232075491 conn(0x7ff7200b1cb0 msgr2=0x7ff7200b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.671+0000 7ff727c00640 1 -- 172.21.15.129:0/3232075491 shutdown_connections 2024-09-17T03:52:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.671+0000 7ff727c00640 1 -- 172.21.15.129:0/3232075491 wait complete. 2024-09-17T03:52:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.673+0000 7ff727c00640 1 Processor -- start 2024-09-17T03:52:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.673+0000 7ff727c00640 1 -- start start 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff720070440 0x7ff720147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff720073170 0x7ff72014a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200b39f0 con 0x7ff720070030 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200b3870 con 0x7ff720073170 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.674+0000 7ff727c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff7200b3b70 con 0x7ff720070440 2024-09-17T03:52:43.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35106/0 (socket says 172.21.15.129:35106) 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 -- 172.21.15.129:0/1387360467 learned_addr learned my addr 172.21.15.129:0/1387360467 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff717e00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff720070440 0x7ff720147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff720073170 0x7ff72014a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 -- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff720070440 msgr2=0x7ff720147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff720070440 0x7ff720147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 -- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff720073170 msgr2=0x7ff72014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff720073170 0x7ff72014a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 -- 172.21.15.129:0/1387360467 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff720148690 con 0x7ff720070030 2024-09-17T03:52:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.675+0000 7ff725200640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b3f0 tx=0x7ff71000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.676+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710015020 con 0x7ff720070030 2024-09-17T03:52:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.676+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff710011920 con 0x7ff720070030 2024-09-17T03:52:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.676+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff710011d70 con 0x7ff720070030 2024-09-17T03:52:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.676+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7200d0560 con 0x7ff720070030 2024-09-17T03:52:43.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.676+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7200d0b20 con 0x7ff720070030 2024-09-17T03:52:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.677+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7200b40d0 con 0x7ff720070030 2024-09-17T03:52:43.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.678+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff710018020 con 0x7ff720070030 2024-09-17T03:52:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.679+0000 7ff71d600640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 0x7ff6fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.679+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7ff71009a9d0 con 0x7ff720070030 2024-09-17T03:52:43.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.680+0000 7ff717e00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 0x7ff6fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.680+0000 7ff717e00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 0x7ff6fc07b0a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff708005fd0 tx=0x7ff708005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:43.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.684+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff71001f030 con 0x7ff720070030 2024-09-17T03:52:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.863+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7200c3860 con 0x7ff6fc078b90 2024-09-17T03:52:43.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.869+0000 7ff71d600640 1 -- 172.21.15.129:0/1387360467 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250824 (secure 0 0 0) 0x7ff7200c3860 con 0x7ff6fc078b90 2024-09-17T03:52:43.872 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:43.873 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:52:43.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 msgr2=0x7ff6fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 0x7ff6fc07b0a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ff708005fd0 tx=0x7ff708005850 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 msgr2=0x7ff72009b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff71000b3f0 tx=0x7ff71000b8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 shutdown_connections 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6fc078b90 0x7ff6fc07b0a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff720073170 0x7ff72014a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff720070440 0x7ff720147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 --2- 172.21.15.129:0/1387360467 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff720070030 0x7ff72009b850 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 >> 172.21.15.129:0/1387360467 conn(0x7ff7200b1cb0 msgr2=0x7ff7200ae7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 shutdown_connections 2024-09-17T03:52:43.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:43.873+0000 7ff727c00640 1 -- 172.21.15.129:0/1387360467 wait complete. 2024-09-17T03:52:43.895 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-17T03:52:42.343044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592686,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":136,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":73,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":256936,"kb_used_data":7424,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717784,"statfs":{"total":773094113280,"available":772831010816,"internally_reserved":0,"allocated":7602176,"data_stored":3818666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2318,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.366451"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028430+0000","last_change":"2024-09-17T03:52:39.012533+0000","last_active":"2024-09-17T03:52:39.028430+0000","last_peered":"2024-09-17T03:52:39.028430+0000","last_clean":"2024-09-17T03:52:39.028430+0000","last_became_active":"2024-09-17T03:52:39.012181+0000","last_became_peered":"2024-09-17T03:52:39.012181+0000","last_unstale":"2024-09-17T03:52:39.028430+0000","last_undegraded":"2024-09-17T03:52:39.028430+0000","last_fullsized":"2024-09-17T03:52:39.028430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.995252+0000","last_change":"2024-09-17T03:52:37.995252+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.995252+0000","last_undegraded":"2024-09-17T03:52:37.995252+0000","last_fullsized":"2024-09-17T03:52:37.995252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027134+0000","last_change":"2024-09-17T03:52:39.015268+0000","last_active":"2024-09-17T03:52:39.027134+0000","last_peered":"2024-09-17T03:52:39.027134+0000","last_clean":"2024-09-17T03:52:39.027134+0000","last_became_active":"2024-09-17T03:52:39.014959+0000","last_became_peered":"2024-09-17T03:52:39.014959+0000","last_unstale":"2024-09-17T03:52:39.027134+0000","last_undegraded":"2024-09-17T03:52:39.027134+0000","last_fullsized":"2024-09-17T03:52:39.027134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:48.400686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.026039+0000","last_change":"2024-09-17T03:52:39.022874+0000","last_active":"2024-09-17T03:52:39.026039+0000","last_peered":"2024-09-17T03:52:39.026039+0000","last_clean":"2024-09-17T03:52:39.026039+0000","last_became_active":"2024-09-17T03:52:39.022102+0000","last_became_peered":"2024-09-17T03:52:39.022102+0000","last_unstale":"2024-09-17T03:52:39.026039+0000","last_undegraded":"2024-09-17T03:52:39.026039+0000","last_fullsized":"2024-09-17T03:52:39.026039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:14.538536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.016705+0000","last_change":"2024-09-17T03:52:39.005338+0000","last_active":"2024-09-17T03:52:39.016705+0000","last_peered":"2024-09-17T03:52:39.016705+0000","last_clean":"2024-09-17T03:52:39.016705+0000","last_became_active":"2024-09-17T03:52:39.005016+0000","last_became_peered":"2024-09-17T03:52:39.005016+0000","last_unstale":"2024-09-17T03:52:39.016705+0000","last_undegraded":"2024-09-17T03:52:39.016705+0000","last_fullsized":"2024-09-17T03:52:39.016705+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:39.136105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989194+0000","last_change":"2024-09-17T03:52:37.989194+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989194+0000","last_undegraded":"2024-09-17T03:52:37.989194+0000","last_fullsized":"2024-09-17T03:52:37.989194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030682+0000","last_change":"2024-09-17T03:52:39.026308+0000","last_active":"2024-09-17T03:52:39.030682+0000","last_peered":"2024-09-17T03:52:39.030682+0000","last_clean":"2024-09-17T03:52:39.030682+0000","last_became_active":"2024-09-17T03:52:39.025753+0000","last_became_peered":"2024-09-17T03:52:39.025753+0000","last_unstale":"2024-09-17T03:52:39.030682+0000","last_undegraded":"2024-09-17T03:52:39.030682+0000","last_fullsized":"2024-09-17T03:52:39.030682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989186+0000","last_change":"2024-09-17T03:52:37.989186+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989186+0000","last_undegraded":"2024-09-17T03:52:37.989186+0000","last_fullsized":"2024-09-17T03:52:37.989186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025209+0000","last_change":"2024-09-17T03:52:39.014691+0000","last_active":"2024-09-17T03:52:39.025209+0000","last_peered":"2024-09-17T03:52:39.025209+0000","last_clean":"2024-09-17T03:52:39.025209+0000","last_became_active":"2024-09-17T03:52:39.010250+0000","last_became_peered":"2024-09-17T03:52:39.010250+0000","last_unstale":"2024-09-17T03:52:39.025209+0000","last_undegraded":"2024-09-17T03:52:39.025209+0000","last_fullsized":"2024-09-17T03:52:39.025209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:28.603948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036735+0000","last_change":"2024-09-17T03:52:39.030566+0000","last_active":"2024-09-17T03:52:39.036735+0000","last_peered":"2024-09-17T03:52:39.036735+0000","last_clean":"2024-09-17T03:52:39.036735+0000","last_became_active":"2024-09-17T03:52:39.029885+0000","last_became_peered":"2024-09-17T03:52:39.029885+0000","last_unstale":"2024-09-17T03:52:39.036735+0000","last_undegraded":"2024-09-17T03:52:39.036735+0000","last_fullsized":"2024-09-17T03:52:39.036735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:15.311759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.032378+0000","last_change":"2024-09-17T03:52:39.027346+0000","last_active":"2024-09-17T03:52:39.032378+0000","last_peered":"2024-09-17T03:52:39.032378+0000","last_clean":"2024-09-17T03:52:39.032378+0000","last_became_active":"2024-09-17T03:52:39.025928+0000","last_became_peered":"2024-09-17T03:52:39.025928+0000","last_unstale":"2024-09-17T03:52:39.032378+0000","last_undegraded":"2024-09-17T03:52:39.032378+0000","last_fullsized":"2024-09-17T03:52:39.032378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:56.381402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024790+0000","last_change":"2024-09-17T03:52:39.018641+0000","last_active":"2024-09-17T03:52:39.024790+0000","last_peered":"2024-09-17T03:52:39.024790+0000","last_clean":"2024-09-17T03:52:39.024790+0000","last_became_active":"2024-09-17T03:52:39.018301+0000","last_became_peered":"2024-09-17T03:52:39.018301+0000","last_unstale":"2024-09-17T03:52:39.024790+0000","last_undegraded":"2024-09-17T03:52:39.024790+0000","last_fullsized":"2024-09-17T03:52:39.024790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:52.014834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.989872+0000","last_change":"2024-09-17T03:52:37.989872+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.989872+0000","last_undegraded":"2024-09-17T03:52:37.989872+0000","last_fullsized":"2024-09-17T03:52:37.989872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020941+0000","last_change":"2024-09-17T03:52:39.012957+0000","last_active":"2024-09-17T03:52:39.020941+0000","last_peered":"2024-09-17T03:52:39.020941+0000","last_clean":"2024-09-17T03:52:39.020941+0000","last_became_active":"2024-09-17T03:52:39.012674+0000","last_became_peered":"2024-09-17T03:52:39.012674+0000","last_unstale":"2024-09-17T03:52:39.020941+0000","last_undegraded":"2024-09-17T03:52:39.020941+0000","last_fullsized":"2024-09-17T03:52:39.020941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:36.707421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988197+0000","last_change":"2024-09-17T03:52:37.988197+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988197+0000","last_undegraded":"2024-09-17T03:52:37.988197+0000","last_fullsized":"2024-09-17T03:52:37.988197+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037382+0000","last_change":"2024-09-17T03:52:39.034015+0000","last_active":"2024-09-17T03:52:39.037382+0000","last_peered":"2024-09-17T03:52:39.037382+0000","last_clean":"2024-09-17T03:52:39.037382+0000","last_became_active":"2024-09-17T03:52:39.033201+0000","last_became_peered":"2024-09-17T03:52:39.033201+0000","last_unstale":"2024-09-17T03:52:39.037382+0000","last_undegraded":"2024-09-17T03:52:39.037382+0000","last_fullsized":"2024-09-17T03:52:39.037382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:10.085629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025244+0000","last_change":"2024-09-17T03:52:39.015002+0000","last_active":"2024-09-17T03:52:39.025244+0000","last_peered":"2024-09-17T03:52:39.025244+0000","last_clean":"2024-09-17T03:52:39.025244+0000","last_became_active":"2024-09-17T03:52:39.010548+0000","last_became_peered":"2024-09-17T03:52:39.010548+0000","last_unstale":"2024-09-17T03:52:39.025244+0000","last_undegraded":"2024-09-17T03:52:39.025244+0000","last_fullsized":"2024-09-17T03:52:39.025244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:19:23.870446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025955+0000","last_change":"2024-09-17T03:52:39.017861+0000","last_active":"2024-09-17T03:52:39.025955+0000","last_peered":"2024-09-17T03:52:39.025955+0000","last_clean":"2024-09-17T03:52:39.025955+0000","last_became_active":"2024-09-17T03:52:39.017375+0000","last_became_peered":"2024-09-17T03:52:39.017375+0000","last_unstale":"2024-09-17T03:52:39.025955+0000","last_undegraded":"2024-09-17T03:52:39.025955+0000","last_fullsized":"2024-09-17T03:52:39.025955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:41.879432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.994267+0000","last_change":"2024-09-17T03:52:37.994267+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.994267+0000","last_undegraded":"2024-09-17T03:52:37.994267+0000","last_fullsized":"2024-09-17T03:52:37.994267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988999+0000","last_change":"2024-09-17T03:52:37.988999+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988999+0000","last_undegraded":"2024-09-17T03:52:37.988999+0000","last_fullsized":"2024-09-17T03:52:37.988999+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036307+0000","last_change":"2024-09-17T03:52:39.027699+0000","last_active":"2024-09-17T03:52:39.036307+0000","last_peered":"2024-09-17T03:52:39.036307+0000","last_clean":"2024-09-17T03:52:39.036307+0000","last_became_active":"2024-09-17T03:52:39.026422+0000","last_became_peered":"2024-09-17T03:52:39.026422+0000","last_unstale":"2024-09-17T03:52:39.036307+0000","last_undegraded":"2024-09-17T03:52:39.036307+0000","last_fullsized":"2024-09-17T03:52:39.036307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:09.586045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036731+0000","last_change":"2024-09-17T03:52:39.030179+0000","last_active":"2024-09-17T03:52:39.036731+0000","last_peered":"2024-09-17T03:52:39.036731+0000","last_clean":"2024-09-17T03:52:39.036731+0000","last_became_active":"2024-09-17T03:52:39.029845+0000","last_became_peered":"2024-09-17T03:52:39.029845+0000","last_unstale":"2024-09-17T03:52:39.036731+0000","last_undegraded":"2024-09-17T03:52:39.036731+0000","last_fullsized":"2024-09-17T03:52:39.036731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:59.212278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.017804+0000","last_change":"2024-09-17T03:52:39.013300+0000","last_active":"2024-09-17T03:52:39.017804+0000","last_peered":"2024-09-17T03:52:39.017804+0000","last_clean":"2024-09-17T03:52:39.017804+0000","last_became_active":"2024-09-17T03:52:39.012947+0000","last_became_peered":"2024-09-17T03:52:39.012947+0000","last_unstale":"2024-09-17T03:52:39.017804+0000","last_undegraded":"2024-09-17T03:52:39.017804+0000","last_fullsized":"2024-09-17T03:52:39.017804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:51.900838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.032253+0000","last_change":"2024-09-17T03:52:39.021402+0000","last_active":"2024-09-17T03:52:39.032253+0000","last_peered":"2024-09-17T03:52:39.032253+0000","last_clean":"2024-09-17T03:52:39.032253+0000","last_became_active":"2024-09-17T03:52:39.021091+0000","last_became_peered":"2024-09-17T03:52:39.021091+0000","last_unstale":"2024-09-17T03:52:39.032253+0000","last_undegraded":"2024-09-17T03:52:39.032253+0000","last_fullsized":"2024-09-17T03:52:39.032253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:25.145657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037049+0000","last_change":"2024-09-17T03:52:39.034509+0000","last_active":"2024-09-17T03:52:39.037049+0000","last_peered":"2024-09-17T03:52:39.037049+0000","last_clean":"2024-09-17T03:52:39.037049+0000","last_became_active":"2024-09-17T03:52:39.033549+0000","last_became_peered":"2024-09-17T03:52:39.033549+0000","last_unstale":"2024-09-17T03:52:39.037049+0000","last_undegraded":"2024-09-17T03:52:39.037049+0000","last_fullsized":"2024-09-17T03:52:39.037049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:04.194058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028726+0000","last_change":"2024-09-17T03:52:39.021427+0000","last_active":"2024-09-17T03:52:39.028726+0000","last_peered":"2024-09-17T03:52:39.028726+0000","last_clean":"2024-09-17T03:52:39.028726+0000","last_became_active":"2024-09-17T03:52:39.020901+0000","last_became_peered":"2024-09-17T03:52:39.020901+0000","last_unstale":"2024-09-17T03:52:39.028726+0000","last_undegraded":"2024-09-17T03:52:39.028726+0000","last_fullsized":"2024-09-17T03:52:39.028726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:27.084828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028134+0000","last_change":"2024-09-17T03:52:39.024384+0000","last_active":"2024-09-17T03:52:39.028134+0000","last_peered":"2024-09-17T03:52:39.028134+0000","last_clean":"2024-09-17T03:52:39.028134+0000","last_became_active":"2024-09-17T03:52:39.015719+0000","last_became_peered":"2024-09-17T03:52:39.015719+0000","last_unstale":"2024-09-17T03:52:39.028134+0000","last_undegraded":"2024-09-17T03:52:39.028134+0000","last_fullsized":"2024-09-17T03:52:39.028134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:00.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036787+0000","last_change":"2024-09-17T03:52:39.031532+0000","last_active":"2024-09-17T03:52:39.036787+0000","last_peered":"2024-09-17T03:52:39.036787+0000","last_clean":"2024-09-17T03:52:39.036787+0000","last_became_active":"2024-09-17T03:52:39.031235+0000","last_became_peered":"2024-09-17T03:52:39.031235+0000","last_unstale":"2024-09-17T03:52:39.036787+0000","last_undegraded":"2024-09-17T03:52:39.036787+0000","last_fullsized":"2024-09-17T03:52:39.036787+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:51.694842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.025053+0000","last_change":"2024-09-17T03:52:39.013885+0000","last_active":"2024-09-17T03:52:39.025053+0000","last_peered":"2024-09-17T03:52:39.025053+0000","last_clean":"2024-09-17T03:52:39.025053+0000","last_became_active":"2024-09-17T03:52:39.013584+0000","last_became_peered":"2024-09-17T03:52:39.013584+0000","last_unstale":"2024-09-17T03:52:39.025053+0000","last_undegraded":"2024-09-17T03:52:39.025053+0000","last_fullsized":"2024-09-17T03:52:39.025053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:11.385767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.019304+0000","last_change":"2024-09-17T03:52:39.016332+0000","last_active":"2024-09-17T03:52:39.019304+0000","last_peered":"2024-09-17T03:52:39.019304+0000","last_clean":"2024-09-17T03:52:39.019304+0000","last_became_active":"2024-09-17T03:52:39.015950+0000","last_became_peered":"2024-09-17T03:52:39.015950+0000","last_unstale":"2024-09-17T03:52:39.019304+0000","last_undegraded":"2024-09-17T03:52:39.019304+0000","last_fullsized":"2024-09-17T03:52:39.019304+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:45.685964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028802+0000","last_change":"2024-09-17T03:52:39.026096+0000","last_active":"2024-09-17T03:52:39.028802+0000","last_peered":"2024-09-17T03:52:39.028802+0000","last_clean":"2024-09-17T03:52:39.028802+0000","last_became_active":"2024-09-17T03:52:39.025309+0000","last_became_peered":"2024-09-17T03:52:39.025309+0000","last_unstale":"2024-09-17T03:52:39.028802+0000","last_undegraded":"2024-09-17T03:52:39.028802+0000","last_fullsized":"2024-09-17T03:52:39.028802+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:37.598541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988679+0000","last_change":"2024-09-17T03:52:37.988679+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988679+0000","last_undegraded":"2024-09-17T03:52:37.988679+0000","last_fullsized":"2024-09-17T03:52:37.988679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030874+0000","last_change":"2024-09-17T03:52:39.023210+0000","last_active":"2024-09-17T03:52:39.030874+0000","last_peered":"2024-09-17T03:52:39.030874+0000","last_clean":"2024-09-17T03:52:39.030874+0000","last_became_active":"2024-09-17T03:52:39.022850+0000","last_became_peered":"2024-09-17T03:52:39.022850+0000","last_unstale":"2024-09-17T03:52:39.030874+0000","last_undegraded":"2024-09-17T03:52:39.030874+0000","last_fullsized":"2024-09-17T03:52:39.030874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:15.593260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030431+0000","last_change":"2024-09-17T03:52:38.004383+0000","last_active":"2024-09-17T03:52:39.030431+0000","last_peered":"2024-09-17T03:52:39.030431+0000","last_clean":"2024-09-17T03:52:39.030431+0000","last_became_active":"2024-09-17T03:52:38.003610+0000","last_became_peered":"2024-09-17T03:52:38.003610+0000","last_unstale":"2024-09-17T03:52:39.030431+0000","last_undegraded":"2024-09-17T03:52:39.030431+0000","last_fullsized":"2024-09-17T03:52:39.030431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:36.838884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.029038+0000","last_change":"2024-09-17T03:52:39.017710+0000","last_active":"2024-09-17T03:52:39.029038+0000","last_peered":"2024-09-17T03:52:39.029038+0000","last_clean":"2024-09-17T03:52:39.029038+0000","last_became_active":"2024-09-17T03:52:39.009010+0000","last_became_peered":"2024-09-17T03:52:39.009010+0000","last_unstale":"2024-09-17T03:52:39.029038+0000","last_undegraded":"2024-09-17T03:52:39.029038+0000","last_fullsized":"2024-09-17T03:52:39.029038+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:36.016315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:37.996051+0000","last_change":"2024-09-17T03:52:37.987588+0000","last_active":"2024-09-17T03:52:37.996051+0000","last_peered":"2024-09-17T03:52:37.996051+0000","last_clean":"2024-09-17T03:52:37.996051+0000","last_became_active":"2024-09-17T03:52:37.986620+0000","last_became_peered":"2024-09-17T03:52:37.986620+0000","last_unstale":"2024-09-17T03:52:37.996051+0000","last_undegraded":"2024-09-17T03:52:37.996051+0000","last_fullsized":"2024-09-17T03:52:37.996051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:51:26.467016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.999829+0000","last_change":"2024-09-17T03:52:37.999829+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.999829+0000","last_undegraded":"2024-09-17T03:52:37.999829+0000","last_fullsized":"2024-09-17T03:52:37.999829+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.033573+0000","last_change":"2024-09-17T03:52:38.008448+0000","last_active":"2024-09-17T03:52:39.033573+0000","last_peered":"2024-09-17T03:52:39.033573+0000","last_clean":"2024-09-17T03:52:39.033573+0000","last_became_active":"2024-09-17T03:52:38.008134+0000","last_became_peered":"2024-09-17T03:52:38.008134+0000","last_unstale":"2024-09-17T03:52:39.033573+0000","last_undegraded":"2024-09-17T03:52:39.033573+0000","last_fullsized":"2024-09-17T03:52:39.033573+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:05.150798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.014491+0000","last_change":"2024-09-17T03:52:39.009452+0000","last_active":"2024-09-17T03:52:39.014491+0000","last_peered":"2024-09-17T03:52:39.014491+0000","last_clean":"2024-09-17T03:52:39.014491+0000","last_became_active":"2024-09-17T03:52:39.005725+0000","last_became_peered":"2024-09-17T03:52:39.005725+0000","last_unstale":"2024-09-17T03:52:39.014491+0000","last_undegraded":"2024-09-17T03:52:39.014491+0000","last_fullsized":"2024-09-17T03:52:39.014491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009240+0000","last_change":"2024-09-17T03:52:38.004751+0000","last_active":"2024-09-17T03:52:38.009240+0000","last_peered":"2024-09-17T03:52:38.009240+0000","last_clean":"2024-09-17T03:52:38.009240+0000","last_became_active":"2024-09-17T03:52:38.004436+0000","last_became_peered":"2024-09-17T03:52:38.004436+0000","last_unstale":"2024-09-17T03:52:38.009240+0000","last_undegraded":"2024-09-17T03:52:38.009240+0000","last_fullsized":"2024-09-17T03:52:38.009240+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:23:18.872377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027974+0000","last_change":"2024-09-17T03:52:39.023295+0000","last_active":"2024-09-17T03:52:39.027974+0000","last_peered":"2024-09-17T03:52:39.027974+0000","last_clean":"2024-09-17T03:52:39.027974+0000","last_became_active":"2024-09-17T03:52:39.022209+0000","last_became_peered":"2024-09-17T03:52:39.022209+0000","last_unstale":"2024-09-17T03:52:39.027974+0000","last_undegraded":"2024-09-17T03:52:39.027974+0000","last_fullsized":"2024-09-17T03:52:39.027974+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:37.845837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036576+0000","last_change":"2024-09-17T03:52:38.004706+0000","last_active":"2024-09-17T03:52:39.036576+0000","last_peered":"2024-09-17T03:52:39.036576+0000","last_clean":"2024-09-17T03:52:39.036576+0000","last_became_active":"2024-09-17T03:52:38.004384+0000","last_became_peered":"2024-09-17T03:52:38.004384+0000","last_unstale":"2024-09-17T03:52:39.036576+0000","last_undegraded":"2024-09-17T03:52:39.036576+0000","last_fullsized":"2024-09-17T03:52:39.036576+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:24.947156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036975+0000","last_change":"2024-09-17T03:52:39.033871+0000","last_active":"2024-09-17T03:52:39.036975+0000","last_peered":"2024-09-17T03:52:39.036975+0000","last_clean":"2024-09-17T03:52:39.036975+0000","last_became_active":"2024-09-17T03:52:39.033350+0000","last_became_peered":"2024-09-17T03:52:39.033350+0000","last_unstale":"2024-09-17T03:52:39.036975+0000","last_undegraded":"2024-09-17T03:52:39.036975+0000","last_fullsized":"2024-09-17T03:52:39.036975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:28.281361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009976+0000","last_change":"2024-09-17T03:52:38.007787+0000","last_active":"2024-09-17T03:52:38.009976+0000","last_peered":"2024-09-17T03:52:38.009976+0000","last_clean":"2024-09-17T03:52:38.009976+0000","last_became_active":"2024-09-17T03:52:38.007287+0000","last_became_peered":"2024-09-17T03:52:38.007287+0000","last_unstale":"2024-09-17T03:52:38.009976+0000","last_undegraded":"2024-09-17T03:52:38.009976+0000","last_fullsized":"2024-09-17T03:52:38.009976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:37.943429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.031819+0000","last_change":"2024-09-17T03:52:39.026791+0000","last_active":"2024-09-17T03:52:39.031819+0000","last_peered":"2024-09-17T03:52:39.031819+0000","last_clean":"2024-09-17T03:52:39.031819+0000","last_became_active":"2024-09-17T03:52:39.026369+0000","last_became_peered":"2024-09-17T03:52:39.026369+0000","last_unstale":"2024-09-17T03:52:39.031819+0000","last_undegraded":"2024-09-17T03:52:39.031819+0000","last_fullsized":"2024-09-17T03:52:39.031819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:40.919599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998830+0000","last_change":"2024-09-17T03:52:38.004637+0000","last_active":"2024-09-17T03:52:38.998830+0000","last_peered":"2024-09-17T03:52:38.998830+0000","last_clean":"2024-09-17T03:52:38.998830+0000","last_became_active":"2024-09-17T03:52:38.004076+0000","last_became_peered":"2024-09-17T03:52:38.004076+0000","last_unstale":"2024-09-17T03:52:38.998830+0000","last_undegraded":"2024-09-17T03:52:38.998830+0000","last_fullsized":"2024-09-17T03:52:38.998830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:38.214053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987688+0000","last_change":"2024-09-17T03:52:37.987688+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987688+0000","last_undegraded":"2024-09-17T03:52:37.987688+0000","last_fullsized":"2024-09-17T03:52:37.987688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.997273+0000","last_change":"2024-09-17T03:52:37.992784+0000","last_active":"2024-09-17T03:52:38.997273+0000","last_peered":"2024-09-17T03:52:38.997273+0000","last_clean":"2024-09-17T03:52:38.997273+0000","last_became_active":"2024-09-17T03:52:37.992480+0000","last_became_peered":"2024-09-17T03:52:37.992480+0000","last_unstale":"2024-09-17T03:52:38.997273+0000","last_undegraded":"2024-09-17T03:52:38.997273+0000","last_fullsized":"2024-09-17T03:52:38.997273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:21.250972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.012794+0000","last_change":"2024-09-17T03:52:39.008286+0000","last_active":"2024-09-17T03:52:39.012794+0000","last_peered":"2024-09-17T03:52:39.012794+0000","last_clean":"2024-09-17T03:52:39.012794+0000","last_became_active":"2024-09-17T03:52:39.007952+0000","last_became_peered":"2024-09-17T03:52:39.007952+0000","last_unstale":"2024-09-17T03:52:39.012794+0000","last_undegraded":"2024-09-17T03:52:39.012794+0000","last_fullsized":"2024-09-17T03:52:39.012794+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:10.085629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020500+0000","last_change":"2024-09-17T03:52:38.001450+0000","last_active":"2024-09-17T03:52:39.020500+0000","last_peered":"2024-09-17T03:52:39.020500+0000","last_clean":"2024-09-17T03:52:39.020500+0000","last_became_active":"2024-09-17T03:52:38.001124+0000","last_became_peered":"2024-09-17T03:52:38.001124+0000","last_unstale":"2024-09-17T03:52:39.020500+0000","last_undegraded":"2024-09-17T03:52:39.020500+0000","last_fullsized":"2024-09-17T03:52:39.020500+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:53.092676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.030972+0000","last_change":"2024-09-17T03:52:39.027159+0000","last_active":"2024-09-17T03:52:39.030972+0000","last_peered":"2024-09-17T03:52:39.030972+0000","last_clean":"2024-09-17T03:52:39.030972+0000","last_became_active":"2024-09-17T03:52:39.026074+0000","last_became_peered":"2024-09-17T03:52:39.026074+0000","last_unstale":"2024-09-17T03:52:39.030972+0000","last_undegraded":"2024-09-17T03:52:39.030972+0000","last_fullsized":"2024-09-17T03:52:39.030972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:32.756863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009258+0000","last_change":"2024-09-17T03:52:38.004849+0000","last_active":"2024-09-17T03:52:38.009258+0000","last_peered":"2024-09-17T03:52:38.009258+0000","last_clean":"2024-09-17T03:52:38.009258+0000","last_became_active":"2024-09-17T03:52:38.002393+0000","last_became_peered":"2024-09-17T03:52:38.002393+0000","last_unstale":"2024-09-17T03:52:38.009258+0000","last_undegraded":"2024-09-17T03:52:38.009258+0000","last_fullsized":"2024-09-17T03:52:38.009258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:15.892290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.996175+0000","last_change":"2024-09-17T03:52:37.996175+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.996175+0000","last_undegraded":"2024-09-17T03:52:37.996175+0000","last_fullsized":"2024-09-17T03:52:37.996175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.036085+0000","last_change":"2024-09-17T03:52:38.002263+0000","last_active":"2024-09-17T03:52:39.036085+0000","last_peered":"2024-09-17T03:52:39.036085+0000","last_clean":"2024-09-17T03:52:39.036085+0000","last_became_active":"2024-09-17T03:52:38.001712+0000","last_became_peered":"2024-09-17T03:52:38.001712+0000","last_unstale":"2024-09-17T03:52:39.036085+0000","last_undegraded":"2024-09-17T03:52:39.036085+0000","last_fullsized":"2024-09-17T03:52:39.036085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:53.008257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024340+0000","last_change":"2024-09-17T03:52:39.015946+0000","last_active":"2024-09-17T03:52:39.024340+0000","last_peered":"2024-09-17T03:52:39.024340+0000","last_clean":"2024-09-17T03:52:39.024340+0000","last_became_active":"2024-09-17T03:52:39.008987+0000","last_became_peered":"2024-09-17T03:52:39.008987+0000","last_unstale":"2024-09-17T03:52:39.024340+0000","last_undegraded":"2024-09-17T03:52:39.024340+0000","last_fullsized":"2024-09-17T03:52:39.024340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:19.490365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.040093+0000","last_change":"2024-09-17T03:52:38.000155+0000","last_active":"2024-09-17T03:52:39.040093+0000","last_peered":"2024-09-17T03:52:39.040093+0000","last_clean":"2024-09-17T03:52:39.040093+0000","last_became_active":"2024-09-17T03:52:37.999044+0000","last_became_peered":"2024-09-17T03:52:37.999044+0000","last_unstale":"2024-09-17T03:52:39.040093+0000","last_undegraded":"2024-09-17T03:52:39.040093+0000","last_fullsized":"2024-09-17T03:52:39.040093+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:15.512221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.018466+0000","last_change":"2024-09-17T03:52:39.015628+0000","last_active":"2024-09-17T03:52:39.018466+0000","last_peered":"2024-09-17T03:52:39.018466+0000","last_clean":"2024-09-17T03:52:39.018466+0000","last_became_active":"2024-09-17T03:52:39.015286+0000","last_became_peered":"2024-09-17T03:52:39.015286+0000","last_unstale":"2024-09-17T03:52:39.018466+0000","last_undegraded":"2024-09-17T03:52:39.018466+0000","last_fullsized":"2024-09-17T03:52:39.018466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:49.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.034007+0000","last_change":"2024-09-17T03:52:38.001510+0000","last_active":"2024-09-17T03:52:39.034007+0000","last_peered":"2024-09-17T03:52:39.034007+0000","last_clean":"2024-09-17T03:52:39.034007+0000","last_became_active":"2024-09-17T03:52:38.001109+0000","last_became_peered":"2024-09-17T03:52:38.001109+0000","last_unstale":"2024-09-17T03:52:39.034007+0000","last_undegraded":"2024-09-17T03:52:39.034007+0000","last_fullsized":"2024-09-17T03:52:39.034007+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:53.377063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028648+0000","last_change":"2024-09-17T03:52:39.021149+0000","last_active":"2024-09-17T03:52:39.028648+0000","last_peered":"2024-09-17T03:52:39.028648+0000","last_clean":"2024-09-17T03:52:39.028648+0000","last_became_active":"2024-09-17T03:52:39.020787+0000","last_became_peered":"2024-09-17T03:52:39.020787+0000","last_unstale":"2024-09-17T03:52:39.028648+0000","last_undegraded":"2024-09-17T03:52:39.028648+0000","last_fullsized":"2024-09-17T03:52:39.028648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:53.816696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.007189+0000","last_change":"2024-09-17T03:52:38.000222+0000","last_active":"2024-09-17T03:52:38.007189+0000","last_peered":"2024-09-17T03:52:38.007189+0000","last_clean":"2024-09-17T03:52:38.007189+0000","last_became_active":"2024-09-17T03:52:37.998870+0000","last_became_peered":"2024-09-17T03:52:37.998870+0000","last_unstale":"2024-09-17T03:52:38.007189+0000","last_undegraded":"2024-09-17T03:52:38.007189+0000","last_fullsized":"2024-09-17T03:52:38.007189+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:44.741147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987073+0000","last_change":"2024-09-17T03:52:37.987073+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987073+0000","last_undegraded":"2024-09-17T03:52:37.987073+0000","last_fullsized":"2024-09-17T03:52:37.987073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.004727+0000","last_change":"2024-09-17T03:52:37.993426+0000","last_active":"2024-09-17T03:52:39.004727+0000","last_peered":"2024-09-17T03:52:39.004727+0000","last_clean":"2024-09-17T03:52:39.004727+0000","last_became_active":"2024-09-17T03:52:37.992558+0000","last_became_peered":"2024-09-17T03:52:37.992558+0000","last_unstale":"2024-09-17T03:52:39.004727+0000","last_undegraded":"2024-09-17T03:52:39.004727+0000","last_fullsized":"2024-09-17T03:52:39.004727+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:50.687889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028753+0000","last_change":"2024-09-17T03:52:39.020708+0000","last_active":"2024-09-17T03:52:39.028753+0000","last_peered":"2024-09-17T03:52:39.028753+0000","last_clean":"2024-09-17T03:52:39.028753+0000","last_became_active":"2024-09-17T03:52:39.019989+0000","last_became_peered":"2024-09-17T03:52:39.019989+0000","last_unstale":"2024-09-17T03:52:39.028753+0000","last_undegraded":"2024-09-17T03:52:39.028753+0000","last_fullsized":"2024-09-17T03:52:39.028753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:14.519601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":21,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.037342+0000","last_change":"2024-09-17T03:52:38.000317+0000","last_active":"2024-09-17T03:52:39.037342+0000","last_peered":"2024-09-17T03:52:39.037342+0000","last_clean":"2024-09-17T03:52:39.037342+0000","last_became_active":"2024-09-17T03:52:37.999933+0000","last_became_peered":"2024-09-17T03:52:37.999933+0000","last_unstale":"2024-09-17T03:52:39.037342+0000","last_undegraded":"2024-09-17T03:52:39.037342+0000","last_fullsized":"2024-09-17T03:52:39.037342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:56.251630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.035161+0000","last_change":"2024-09-17T03:52:39.025015+0000","last_active":"2024-09-17T03:52:39.035161+0000","last_peered":"2024-09-17T03:52:39.035161+0000","last_clean":"2024-09-17T03:52:39.035161+0000","last_became_active":"2024-09-17T03:52:39.024596+0000","last_became_peered":"2024-09-17T03:52:39.024596+0000","last_unstale":"2024-09-17T03:52:39.035161+0000","last_undegraded":"2024-09-17T03:52:39.035161+0000","last_fullsized":"2024-09-17T03:52:39.035161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:05.782615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.010014+0000","last_change":"2024-09-17T03:52:38.007757+0000","last_active":"2024-09-17T03:52:38.010014+0000","last_peered":"2024-09-17T03:52:38.010014+0000","last_clean":"2024-09-17T03:52:38.010014+0000","last_became_active":"2024-09-17T03:52:38.007282+0000","last_became_peered":"2024-09-17T03:52:38.007282+0000","last_unstale":"2024-09-17T03:52:38.010014+0000","last_undegraded":"2024-09-17T03:52:38.010014+0000","last_fullsized":"2024-09-17T03:52:38.010014+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:56.655759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.020779+0000","last_change":"2024-09-17T03:52:39.007897+0000","last_active":"2024-09-17T03:52:39.020779+0000","last_peered":"2024-09-17T03:52:39.020779+0000","last_clean":"2024-09-17T03:52:39.020779+0000","last_became_active":"2024-09-17T03:52:39.007557+0000","last_became_peered":"2024-09-17T03:52:39.007557+0000","last_unstale":"2024-09-17T03:52:39.020779+0000","last_undegraded":"2024-09-17T03:52:39.020779+0000","last_fullsized":"2024-09-17T03:52:39.020779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.033549+0000","last_change":"2024-09-17T03:52:38.002206+0000","last_active":"2024-09-17T03:52:39.033549+0000","last_peered":"2024-09-17T03:52:39.033549+0000","last_clean":"2024-09-17T03:52:39.033549+0000","last_became_active":"2024-09-17T03:52:38.001872+0000","last_became_peered":"2024-09-17T03:52:38.001872+0000","last_unstale":"2024-09-17T03:52:39.033549+0000","last_undegraded":"2024-09-17T03:52:39.033549+0000","last_fullsized":"2024-09-17T03:52:39.033549+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:18.787765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024818+0000","last_change":"2024-09-17T03:52:39.018968+0000","last_active":"2024-09-17T03:52:39.024818+0000","last_peered":"2024-09-17T03:52:39.024818+0000","last_clean":"2024-09-17T03:52:39.024818+0000","last_became_active":"2024-09-17T03:52:39.018682+0000","last_became_peered":"2024-09-17T03:52:39.018682+0000","last_unstale":"2024-09-17T03:52:39.024818+0000","last_undegraded":"2024-09-17T03:52:39.024818+0000","last_fullsized":"2024-09-17T03:52:39.024818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:33:04.970156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.011962+0000","last_change":"2024-09-17T03:52:38.009348+0000","last_active":"2024-09-17T03:52:38.011962+0000","last_peered":"2024-09-17T03:52:38.011962+0000","last_clean":"2024-09-17T03:52:38.011962+0000","last_became_active":"2024-09-17T03:52:38.008926+0000","last_became_peered":"2024-09-17T03:52:38.008926+0000","last_unstale":"2024-09-17T03:52:38.011962+0000","last_undegraded":"2024-09-17T03:52:38.011962+0000","last_fullsized":"2024-09-17T03:52:38.011962+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:03:16.326145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.028893+0000","last_change":"2024-09-17T03:52:39.021118+0000","last_active":"2024-09-17T03:52:39.028893+0000","last_peered":"2024-09-17T03:52:39.028893+0000","last_clean":"2024-09-17T03:52:39.028893+0000","last_became_active":"2024-09-17T03:52:39.020763+0000","last_became_peered":"2024-09-17T03:52:39.020763+0000","last_unstale":"2024-09-17T03:52:39.028893+0000","last_undegraded":"2024-09-17T03:52:39.028893+0000","last_fullsized":"2024-09-17T03:52:39.028893+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:50.085185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.997610+0000","last_change":"2024-09-17T03:52:38.005886+0000","last_active":"2024-09-17T03:52:38.997610+0000","last_peered":"2024-09-17T03:52:38.997610+0000","last_clean":"2024-09-17T03:52:38.997610+0000","last_became_active":"2024-09-17T03:52:38.005550+0000","last_became_peered":"2024-09-17T03:52:38.005550+0000","last_unstale":"2024-09-17T03:52:38.997610+0000","last_undegraded":"2024-09-17T03:52:38.997610+0000","last_fullsized":"2024-09-17T03:52:38.997610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:36.591587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.014817+0000","last_change":"2024-09-17T03:52:39.009574+0000","last_active":"2024-09-17T03:52:39.014817+0000","last_peered":"2024-09-17T03:52:39.014817+0000","last_clean":"2024-09-17T03:52:39.014817+0000","last_became_active":"2024-09-17T03:52:39.009382+0000","last_became_peered":"2024-09-17T03:52:39.009382+0000","last_unstale":"2024-09-17T03:52:39.014817+0000","last_undegraded":"2024-09-17T03:52:39.014817+0000","last_fullsized":"2024-09-17T03:52:39.014817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:16.138436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009381+0000","last_change":"2024-09-17T03:52:38.006146+0000","last_active":"2024-09-17T03:52:38.009381+0000","last_peered":"2024-09-17T03:52:38.009381+0000","last_clean":"2024-09-17T03:52:38.009381+0000","last_became_active":"2024-09-17T03:52:38.005099+0000","last_became_peered":"2024-09-17T03:52:38.005099+0000","last_unstale":"2024-09-17T03:52:38.009381+0000","last_undegraded":"2024-09-17T03:52:38.009381+0000","last_fullsized":"2024-09-17T03:52:38.009381+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:03.004454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027636+0000","last_change":"2024-09-17T03:52:39.020395+0000","last_active":"2024-09-17T03:52:39.027636+0000","last_peered":"2024-09-17T03:52:39.027636+0000","last_clean":"2024-09-17T03:52:39.027636+0000","last_became_active":"2024-09-17T03:52:39.019294+0000","last_became_peered":"2024-09-17T03:52:39.019294+0000","last_unstale":"2024-09-17T03:52:39.027636+0000","last_undegraded":"2024-09-17T03:52:39.027636+0000","last_fullsized":"2024-09-17T03:52:39.027636+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:48.684398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.006539+0000","last_change":"2024-09-17T03:52:37.986744+0000","last_active":"2024-09-17T03:52:39.006539+0000","last_peered":"2024-09-17T03:52:39.006539+0000","last_clean":"2024-09-17T03:52:39.006539+0000","last_became_active":"2024-09-17T03:52:37.986385+0000","last_became_peered":"2024-09-17T03:52:37.986385+0000","last_unstale":"2024-09-17T03:52:39.006539+0000","last_undegraded":"2024-09-17T03:52:39.006539+0000","last_fullsized":"2024-09-17T03:52:39.006539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.274408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.027469+0000","last_change":"2024-09-17T03:52:39.019842+0000","last_active":"2024-09-17T03:52:39.027469+0000","last_peered":"2024-09-17T03:52:39.027469+0000","last_clean":"2024-09-17T03:52:39.027469+0000","last_became_active":"2024-09-17T03:52:39.019548+0000","last_became_peered":"2024-09-17T03:52:39.019548+0000","last_unstale":"2024-09-17T03:52:39.027469+0000","last_undegraded":"2024-09-17T03:52:39.027469+0000","last_fullsized":"2024-09-17T03:52:39.027469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:28.088118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:37.995971+0000","last_change":"2024-09-17T03:52:37.987589+0000","last_active":"2024-09-17T03:52:37.995971+0000","last_peered":"2024-09-17T03:52:37.995971+0000","last_clean":"2024-09-17T03:52:37.995971+0000","last_became_active":"2024-09-17T03:52:37.986646+0000","last_became_peered":"2024-09-17T03:52:37.986646+0000","last_unstale":"2024-09-17T03:52:37.995971+0000","last_undegraded":"2024-09-17T03:52:37.995971+0000","last_fullsized":"2024-09-17T03:52:37.995971+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:18.770567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.987171+0000","last_change":"2024-09-17T03:52:37.987171+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.987171+0000","last_undegraded":"2024-09-17T03:52:37.987171+0000","last_fullsized":"2024-09-17T03:52:37.987171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009981+0000","last_change":"2024-09-17T03:52:38.007301+0000","last_active":"2024-09-17T03:52:38.009981+0000","last_peered":"2024-09-17T03:52:38.009981+0000","last_clean":"2024-09-17T03:52:38.009981+0000","last_became_active":"2024-09-17T03:52:38.006871+0000","last_became_peered":"2024-09-17T03:52:38.006871+0000","last_unstale":"2024-09-17T03:52:38.009981+0000","last_undegraded":"2024-09-17T03:52:38.009981+0000","last_fullsized":"2024-09-17T03:52:38.009981+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:48:05.016401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.019091+0000","last_change":"2024-09-17T03:52:39.015261+0000","last_active":"2024-09-17T03:52:39.019091+0000","last_peered":"2024-09-17T03:52:39.019091+0000","last_clean":"2024-09-17T03:52:39.019091+0000","last_became_active":"2024-09-17T03:52:39.014907+0000","last_became_peered":"2024-09-17T03:52:39.014907+0000","last_unstale":"2024-09-17T03:52:39.019091+0000","last_undegraded":"2024-09-17T03:52:39.019091+0000","last_fullsized":"2024-09-17T03:52:39.019091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:04.194058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.003843+0000","last_change":"2024-09-17T03:52:37.993729+0000","last_active":"2024-09-17T03:52:39.003843+0000","last_peered":"2024-09-17T03:52:39.003843+0000","last_clean":"2024-09-17T03:52:39.003843+0000","last_became_active":"2024-09-17T03:52:37.991713+0000","last_became_peered":"2024-09-17T03:52:37.991713+0000","last_unstale":"2024-09-17T03:52:39.003843+0000","last_undegraded":"2024-09-17T03:52:39.003843+0000","last_fullsized":"2024-09-17T03:52:39.003843+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:14.304806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.031113+0000","last_change":"2024-09-17T03:52:39.024919+0000","last_active":"2024-09-17T03:52:39.031113+0000","last_peered":"2024-09-17T03:52:39.031113+0000","last_clean":"2024-09-17T03:52:39.031113+0000","last_became_active":"2024-09-17T03:52:39.022961+0000","last_became_peered":"2024-09-17T03:52:39.022961+0000","last_unstale":"2024-09-17T03:52:39.031113+0000","last_undegraded":"2024-09-17T03:52:39.031113+0000","last_fullsized":"2024-09-17T03:52:39.031113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:30.540116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998737+0000","last_change":"2024-09-17T03:52:38.002436+0000","last_active":"2024-09-17T03:52:38.998737+0000","last_peered":"2024-09-17T03:52:38.998737+0000","last_clean":"2024-09-17T03:52:38.998737+0000","last_became_active":"2024-09-17T03:52:38.001380+0000","last_became_peered":"2024-09-17T03:52:38.001380+0000","last_unstale":"2024-09-17T03:52:38.998737+0000","last_undegraded":"2024-09-17T03:52:38.998737+0000","last_fullsized":"2024-09-17T03:52:38.998737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:16.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.017896+0000","last_change":"2024-09-17T03:52:39.013637+0000","last_active":"2024-09-17T03:52:39.017896+0000","last_peered":"2024-09-17T03:52:39.017896+0000","last_clean":"2024-09-17T03:52:39.017896+0000","last_became_active":"2024-09-17T03:52:39.013035+0000","last_became_peered":"2024-09-17T03:52:39.013035+0000","last_unstale":"2024-09-17T03:52:39.017896+0000","last_undegraded":"2024-09-17T03:52:39.017896+0000","last_fullsized":"2024-09-17T03:52:39.017896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:56:57.011981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.001930+0000","last_change":"2024-09-17T03:52:37.998007+0000","last_active":"2024-09-17T03:52:38.001930+0000","last_peered":"2024-09-17T03:52:38.001930+0000","last_clean":"2024-09-17T03:52:38.001930+0000","last_became_active":"2024-09-17T03:52:37.997670+0000","last_became_peered":"2024-09-17T03:52:37.997670+0000","last_unstale":"2024-09-17T03:52:38.001930+0000","last_undegraded":"2024-09-17T03:52:38.001930+0000","last_fullsized":"2024-09-17T03:52:38.001930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:28:14.989087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024301+0000","last_change":"2024-09-17T03:52:39.016581+0000","last_active":"2024-09-17T03:52:39.024301+0000","last_peered":"2024-09-17T03:52:39.024301+0000","last_clean":"2024-09-17T03:52:39.024301+0000","last_became_active":"2024-09-17T03:52:39.007886+0000","last_became_peered":"2024-09-17T03:52:39.007886+0000","last_unstale":"2024-09-17T03:52:39.024301+0000","last_undegraded":"2024-09-17T03:52:39.024301+0000","last_fullsized":"2024-09-17T03:52:39.024301+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:00.893129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998182+0000","last_change":"2024-09-17T03:52:37.991052+0000","last_active":"2024-09-17T03:52:38.998182+0000","last_peered":"2024-09-17T03:52:38.998182+0000","last_clean":"2024-09-17T03:52:38.998182+0000","last_became_active":"2024-09-17T03:52:37.990089+0000","last_became_peered":"2024-09-17T03:52:37.990089+0000","last_unstale":"2024-09-17T03:52:38.998182+0000","last_undegraded":"2024-09-17T03:52:38.998182+0000","last_fullsized":"2024-09-17T03:52:38.998182+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:17.450063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988927+0000","last_change":"2024-09-17T03:52:37.988927+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988927+0000","last_undegraded":"2024-09-17T03:52:37.988927+0000","last_fullsized":"2024-09-17T03:52:37.988927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.998673+0000","last_change":"2024-09-17T03:52:38.006002+0000","last_active":"2024-09-17T03:52:38.998673+0000","last_peered":"2024-09-17T03:52:38.998673+0000","last_clean":"2024-09-17T03:52:38.998673+0000","last_became_active":"2024-09-17T03:52:38.004725+0000","last_became_peered":"2024-09-17T03:52:38.004725+0000","last_unstale":"2024-09-17T03:52:38.998673+0000","last_undegraded":"2024-09-17T03:52:38.998673+0000","last_fullsized":"2024-09-17T03:52:38.998673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:05.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.015069+0000","last_change":"2024-09-17T03:52:39.010421+0000","last_active":"2024-09-17T03:52:39.015069+0000","last_peered":"2024-09-17T03:52:39.015069+0000","last_clean":"2024-09-17T03:52:39.015069+0000","last_became_active":"2024-09-17T03:52:39.010077+0000","last_became_peered":"2024-09-17T03:52:39.010077+0000","last_unstale":"2024-09-17T03:52:39.015069+0000","last_undegraded":"2024-09-17T03:52:39.015069+0000","last_fullsized":"2024-09-17T03:52:39.015069+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:28.741916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.016750+0000","last_change":"2024-09-17T03:52:37.998870+0000","last_active":"2024-09-17T03:52:39.016750+0000","last_peered":"2024-09-17T03:52:39.016750+0000","last_clean":"2024-09-17T03:52:39.016750+0000","last_became_active":"2024-09-17T03:52:37.998561+0000","last_became_peered":"2024-09-17T03:52:37.998561+0000","last_unstale":"2024-09-17T03:52:39.016750+0000","last_undegraded":"2024-09-17T03:52:39.016750+0000","last_fullsized":"2024-09-17T03:52:39.016750+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:58.681513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":66,"state":"creating+peering","last_fresh":"2024-09-17T03:52:37.988483+0000","last_change":"2024-09-17T03:52:37.988483+0000","last_active":"2024-09-17T03:52:37.973877+0000","last_peered":"2024-09-17T03:52:37.973877+0000","last_clean":"2024-09-17T03:52:37.973877+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:52:37.988483+0000","last_undegraded":"2024-09-17T03:52:37.988483+0000","last_fullsized":"2024-09-17T03:52:37.988483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.009258+0000","last_change":"2024-09-17T03:52:38.004100+0000","last_active":"2024-09-17T03:52:38.009258+0000","last_peered":"2024-09-17T03:52:38.009258+0000","last_clean":"2024-09-17T03:52:38.009258+0000","last_became_active":"2024-09-17T03:52:38.000956+0000","last_became_peered":"2024-09-17T03:52:38.000956+0000","last_unstale":"2024-09-17T03:52:38.009258+0000","last_undegraded":"2024-09-17T03:52:38.009258+0000","last_fullsized":"2024-09-17T03:52:38.009258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:58:47.053179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:39.024906+0000","last_change":"2024-09-17T03:52:39.013661+0000","last_active":"2024-09-17T03:52:39.024906+0000","last_peered":"2024-09-17T03:52:39.024906+0000","last_clean":"2024-09-17T03:52:39.024906+0000","last_became_active":"2024-09-17T03:52:39.013380+0000","last_became_peered":"2024-09-17T03:52:39.013380+0000","last_unstale":"2024-09-17T03:52:39.024906+0000","last_undegraded":"2024-09-17T03:52:39.024906+0000","last_fullsized":"2024-09-17T03:52:39.024906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:59.949458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.007029+0000","last_change":"2024-09-17T03:52:38.001315+0000","last_active":"2024-09-17T03:52:38.007029+0000","last_peered":"2024-09-17T03:52:38.007029+0000","last_clean":"2024-09-17T03:52:38.007029+0000","last_became_active":"2024-09-17T03:52:37.999404+0000","last_became_peered":"2024-09-17T03:52:37.999404+0000","last_unstale":"2024-09-17T03:52:38.007029+0000","last_undegraded":"2024-09-17T03:52:38.007029+0000","last_fullsized":"2024-09-17T03:52:38.007029+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:25:50.399405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":192,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-17T03:52:38.995809+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:52:38.995809+0000","last_peered":"2024-09-17T03:52:38.995809+0000","last_clean":"2024-09-17T03:52:38.995809+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:52:38.995809+0000","last_undegraded":"2024-09-17T03:52:38.995809+0000","last_fullsized":"2024-09-17T03:52:38.995809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2318,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28,"ondisk_log_size":28,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509481,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32544,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339296,"statfs":{"total":96636764160,"available":96603439104,"internally_reserved":0,"allocated":1388544,"data_stored":917873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509484,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509483,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31960,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339880,"statfs":{"total":96636764160,"available":96604037120,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509481,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31976,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339864,"statfs":{"total":96636764160,"available":96604020736,"internally_reserved":0,"allocated":806912,"data_stored":334280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":819200,"data_stored":335892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":336508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":794624,"data_stored":327505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32536,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339304,"statfs":{"total":96636764160,"available":96603447296,"internally_reserved":0,"allocated":1376256,"data_stored":911598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:52:43.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 -- 172.21.15.129:0/1610188949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 msgr2=0x7f2cdc0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/1610188949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0ccf60 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4005fd0 tx=0x7f2cd402a8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 -- 172.21.15.129:0/1610188949 shutdown_connections 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/1610188949 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0ccf60 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 -- 172.21.15.129:0/1610188949 >> 172.21.15.129:0/1610188949 conn(0x7f2cdc06d7d0 msgr2=0x7f2cdc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:44.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 -- 172.21.15.129:0/1610188949 shutdown_connections 2024-09-17T03:52:44.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.156+0000 7f2ce1a00640 1 -- 172.21.15.129:0/1610188949 wait complete. 2024-09-17T03:52:44.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.158+0000 7f2ce1a00640 1 Processor -- start 2024-09-17T03:52:44.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.158+0000 7f2ce1a00640 1 -- start start 2024-09-17T03:52:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.159+0000 7f2ce1a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cdc0af3c0 0x7f2cdc0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.159+0000 7f2ce1a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 0x7f2cdc14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.159+0000 7f2ce1a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2ce1a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc153520 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2ce1a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc1533a0 con 0x7f2cdc0af3c0 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2ce1a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc1536a0 con 0x7f2cdc0cc1e0 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36950/0 (socket says 172.21.15.129:36950) 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 -- 172.21.15.129:0/676005678 learned_addr learned my addr 172.21.15.129:0/676005678 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdaa00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cdc0af3c0 0x7f2cdc0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cda000640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 0x7f2cdc14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 -- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 msgr2=0x7f2cdc14e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 0x7f2cdc14e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 -- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cdc0af3c0 msgr2=0x7f2cdc0aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cdc0af3c0 0x7f2cdc0aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cdb400640 1 -- 172.21.15.129:0/676005678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0c95c0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.160+0000 7f2cda000640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 0x7f2cdc14e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:44.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2cdb400640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009cd0 tx=0x7f2cd40039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:44.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4031990 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2cd4031c20 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40332f0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc0c98a0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.161+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc0c9e60 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.162+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc0b2bc0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.163+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2cd4033490 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.164+0000 7f2cd2a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 0x7f2cbc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.164+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f2cd40bc2a0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.165+0000 7f2cdaa00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 0x7f2cbc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.165+0000 7f2cdaa00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 0x7f2cbc07b0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc000f60 tx=0x7f2ccc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:44.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.169+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2cdc0b2bc0 con 0x7f2cdc0ccb50 2024-09-17T03:52:44.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.349+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2cdc0c9370 con 0x7f2cbc078bc0 2024-09-17T03:52:44.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.355+0000 7f2cd2a00640 1 -- 172.21.15.129:0/676005678 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252046 (secure 0 0 0) 0x7f2cdc0c9370 con 0x7f2cbc078bc0 2024-09-17T03:52:44.358 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:44.359 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:52:44.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 msgr2=0x7f2cbc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 0x7f2cbc07b0d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc000f60 tx=0x7f2ccc009040 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 msgr2=0x7f2cdc0cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009cd0 tx=0x7f2cd40039f0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 shutdown_connections 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2cbc078bc0 0x7f2cbc07b0d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cdc0ccb50 0x7f2cdc0cb630 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cdc0cc1e0 0x7f2cdc14e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 --2- 172.21.15.129:0/676005678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cdc0af3c0 0x7f2cdc0aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 >> 172.21.15.129:0/676005678 conn(0x7f2cdc06d7d0 msgr2=0x7f2cdc0b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 shutdown_connections 2024-09-17T03:52:44.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.359+0000 7f2ce1a00640 1 -- 172.21.15.129:0/676005678 wait complete. 2024-09-17T03:52:44.384 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-17T03:52:44.343730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257164,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717556,"statfs":{"total":773094113280,"available":772830777344,"internally_reserved":0,"allocated":7806976,"data_stored":3910196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.365375"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289679+0000","last_change":"2024-09-17T03:52:39.012533+0000","last_active":"2024-09-17T03:52:40.289679+0000","last_peered":"2024-09-17T03:52:40.289679+0000","last_clean":"2024-09-17T03:52:40.289679+0000","last_became_active":"2024-09-17T03:52:39.012181+0000","last_became_peered":"2024-09-17T03:52:39.012181+0000","last_unstale":"2024-09-17T03:52:40.289679+0000","last_undegraded":"2024-09-17T03:52:40.289679+0000","last_fullsized":"2024-09-17T03:52:40.289679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:54:06.915985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291299+0000","last_change":"2024-09-17T03:52:39.024835+0000","last_active":"2024-09-17T03:52:40.291299+0000","last_peered":"2024-09-17T03:52:40.291299+0000","last_clean":"2024-09-17T03:52:40.291299+0000","last_became_active":"2024-09-17T03:52:39.023996+0000","last_became_peered":"2024-09-17T03:52:39.023996+0000","last_unstale":"2024-09-17T03:52:40.291299+0000","last_undegraded":"2024-09-17T03:52:40.291299+0000","last_fullsized":"2024-09-17T03:52:40.291299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:06.467449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288896+0000","last_change":"2024-09-17T03:52:39.015268+0000","last_active":"2024-09-17T03:52:40.288896+0000","last_peered":"2024-09-17T03:52:40.288896+0000","last_clean":"2024-09-17T03:52:40.288896+0000","last_became_active":"2024-09-17T03:52:39.014959+0000","last_became_peered":"2024-09-17T03:52:39.014959+0000","last_unstale":"2024-09-17T03:52:40.288896+0000","last_undegraded":"2024-09-17T03:52:40.288896+0000","last_fullsized":"2024-09-17T03:52:40.288896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:02:44.299798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007816+0000","last_change":"2024-09-17T03:52:39.022874+0000","last_active":"2024-09-17T03:52:40.007816+0000","last_peered":"2024-09-17T03:52:40.007816+0000","last_clean":"2024-09-17T03:52:40.007816+0000","last_became_active":"2024-09-17T03:52:39.022102+0000","last_became_peered":"2024-09-17T03:52:39.022102+0000","last_unstale":"2024-09-17T03:52:40.007816+0000","last_undegraded":"2024-09-17T03:52:40.007816+0000","last_fullsized":"2024-09-17T03:52:40.007816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:11.479509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008090+0000","last_change":"2024-09-17T03:52:39.005338+0000","last_active":"2024-09-17T03:52:40.008090+0000","last_peered":"2024-09-17T03:52:40.008090+0000","last_clean":"2024-09-17T03:52:40.008090+0000","last_became_active":"2024-09-17T03:52:39.005016+0000","last_became_peered":"2024-09-17T03:52:39.005016+0000","last_unstale":"2024-09-17T03:52:40.008090+0000","last_undegraded":"2024-09-17T03:52:40.008090+0000","last_fullsized":"2024-09-17T03:52:40.008090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:32.775243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290327+0000","last_change":"2024-09-17T03:52:39.018888+0000","last_active":"2024-09-17T03:52:40.290327+0000","last_peered":"2024-09-17T03:52:40.290327+0000","last_clean":"2024-09-17T03:52:40.290327+0000","last_became_active":"2024-09-17T03:52:39.018589+0000","last_became_peered":"2024-09-17T03:52:39.018589+0000","last_unstale":"2024-09-17T03:52:40.290327+0000","last_undegraded":"2024-09-17T03:52:40.290327+0000","last_fullsized":"2024-09-17T03:52:40.290327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:12.320186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007818+0000","last_change":"2024-09-17T03:52:39.026308+0000","last_active":"2024-09-17T03:52:40.007818+0000","last_peered":"2024-09-17T03:52:40.007818+0000","last_clean":"2024-09-17T03:52:40.007818+0000","last_became_active":"2024-09-17T03:52:39.025753+0000","last_became_peered":"2024-09-17T03:52:39.025753+0000","last_unstale":"2024-09-17T03:52:40.007818+0000","last_undegraded":"2024-09-17T03:52:40.007818+0000","last_fullsized":"2024-09-17T03:52:40.007818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289062+0000","last_change":"2024-09-17T03:52:39.013777+0000","last_active":"2024-09-17T03:52:40.289062+0000","last_peered":"2024-09-17T03:52:40.289062+0000","last_clean":"2024-09-17T03:52:40.289062+0000","last_became_active":"2024-09-17T03:52:39.013483+0000","last_became_peered":"2024-09-17T03:52:39.013483+0000","last_unstale":"2024-09-17T03:52:40.289062+0000","last_undegraded":"2024-09-17T03:52:40.289062+0000","last_fullsized":"2024-09-17T03:52:40.289062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:51.177952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009664+0000","last_change":"2024-09-17T03:52:39.014691+0000","last_active":"2024-09-17T03:52:40.009664+0000","last_peered":"2024-09-17T03:52:40.009664+0000","last_clean":"2024-09-17T03:52:40.009664+0000","last_became_active":"2024-09-17T03:52:39.010250+0000","last_became_peered":"2024-09-17T03:52:39.010250+0000","last_unstale":"2024-09-17T03:52:40.009664+0000","last_undegraded":"2024-09-17T03:52:40.009664+0000","last_fullsized":"2024-09-17T03:52:40.009664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:23.670201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290480+0000","last_change":"2024-09-17T03:52:39.030566+0000","last_active":"2024-09-17T03:52:40.290480+0000","last_peered":"2024-09-17T03:52:40.290480+0000","last_clean":"2024-09-17T03:52:40.290480+0000","last_became_active":"2024-09-17T03:52:39.029885+0000","last_became_peered":"2024-09-17T03:52:39.029885+0000","last_unstale":"2024-09-17T03:52:40.290480+0000","last_undegraded":"2024-09-17T03:52:40.290480+0000","last_fullsized":"2024-09-17T03:52:40.290480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:37.598541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288867+0000","last_change":"2024-09-17T03:52:39.027346+0000","last_active":"2024-09-17T03:52:40.288867+0000","last_peered":"2024-09-17T03:52:40.288867+0000","last_clean":"2024-09-17T03:52:40.288867+0000","last_became_active":"2024-09-17T03:52:39.025928+0000","last_became_peered":"2024-09-17T03:52:39.025928+0000","last_unstale":"2024-09-17T03:52:40.288867+0000","last_undegraded":"2024-09-17T03:52:40.288867+0000","last_fullsized":"2024-09-17T03:52:40.288867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:47.351765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007827+0000","last_change":"2024-09-17T03:52:39.018641+0000","last_active":"2024-09-17T03:52:40.007827+0000","last_peered":"2024-09-17T03:52:40.007827+0000","last_clean":"2024-09-17T03:52:40.007827+0000","last_became_active":"2024-09-17T03:52:39.018301+0000","last_became_peered":"2024-09-17T03:52:39.018301+0000","last_unstale":"2024-09-17T03:52:40.007827+0000","last_undegraded":"2024-09-17T03:52:40.007827+0000","last_fullsized":"2024-09-17T03:52:40.007827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:37.690837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287852+0000","last_change":"2024-09-17T03:52:39.022408+0000","last_active":"2024-09-17T03:52:40.287852+0000","last_peered":"2024-09-17T03:52:40.287852+0000","last_clean":"2024-09-17T03:52:40.287852+0000","last_became_active":"2024-09-17T03:52:39.021968+0000","last_became_peered":"2024-09-17T03:52:39.021968+0000","last_unstale":"2024-09-17T03:52:40.287852+0000","last_undegraded":"2024-09-17T03:52:40.287852+0000","last_fullsized":"2024-09-17T03:52:40.287852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:02.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009462+0000","last_change":"2024-09-17T03:52:39.012957+0000","last_active":"2024-09-17T03:52:40.009462+0000","last_peered":"2024-09-17T03:52:40.009462+0000","last_clean":"2024-09-17T03:52:40.009462+0000","last_became_active":"2024-09-17T03:52:39.012674+0000","last_became_peered":"2024-09-17T03:52:39.012674+0000","last_unstale":"2024-09-17T03:52:40.009462+0000","last_undegraded":"2024-09-17T03:52:40.009462+0000","last_fullsized":"2024-09-17T03:52:40.009462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.669111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287698+0000","last_change":"2024-09-17T03:52:39.007565+0000","last_active":"2024-09-17T03:52:40.287698+0000","last_peered":"2024-09-17T03:52:40.287698+0000","last_clean":"2024-09-17T03:52:40.287698+0000","last_became_active":"2024-09-17T03:52:39.007271+0000","last_became_peered":"2024-09-17T03:52:39.007271+0000","last_unstale":"2024-09-17T03:52:40.287698+0000","last_undegraded":"2024-09-17T03:52:40.287698+0000","last_fullsized":"2024-09-17T03:52:40.287698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:48:35.200854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288812+0000","last_change":"2024-09-17T03:52:39.034015+0000","last_active":"2024-09-17T03:52:40.288812+0000","last_peered":"2024-09-17T03:52:40.288812+0000","last_clean":"2024-09-17T03:52:40.288812+0000","last_became_active":"2024-09-17T03:52:39.033201+0000","last_became_peered":"2024-09-17T03:52:39.033201+0000","last_unstale":"2024-09-17T03:52:40.288812+0000","last_undegraded":"2024-09-17T03:52:40.288812+0000","last_fullsized":"2024-09-17T03:52:40.288812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:45.685964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009736+0000","last_change":"2024-09-17T03:52:39.015002+0000","last_active":"2024-09-17T03:52:40.009736+0000","last_peered":"2024-09-17T03:52:40.009736+0000","last_clean":"2024-09-17T03:52:40.009736+0000","last_became_active":"2024-09-17T03:52:39.010548+0000","last_became_peered":"2024-09-17T03:52:39.010548+0000","last_unstale":"2024-09-17T03:52:40.009736+0000","last_undegraded":"2024-09-17T03:52:40.009736+0000","last_fullsized":"2024-09-17T03:52:40.009736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:47.282513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008550+0000","last_change":"2024-09-17T03:52:39.017861+0000","last_active":"2024-09-17T03:52:40.008550+0000","last_peered":"2024-09-17T03:52:40.008550+0000","last_clean":"2024-09-17T03:52:40.008550+0000","last_became_active":"2024-09-17T03:52:39.017375+0000","last_became_peered":"2024-09-17T03:52:39.017375+0000","last_unstale":"2024-09-17T03:52:40.008550+0000","last_undegraded":"2024-09-17T03:52:40.008550+0000","last_fullsized":"2024-09-17T03:52:40.008550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:59.688466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290958+0000","last_change":"2024-09-17T03:52:39.024408+0000","last_active":"2024-09-17T03:52:40.290958+0000","last_peered":"2024-09-17T03:52:40.290958+0000","last_clean":"2024-09-17T03:52:40.290958+0000","last_became_active":"2024-09-17T03:52:39.024048+0000","last_became_peered":"2024-09-17T03:52:39.024048+0000","last_unstale":"2024-09-17T03:52:40.290958+0000","last_undegraded":"2024-09-17T03:52:40.290958+0000","last_fullsized":"2024-09-17T03:52:40.290958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:55.683703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288095+0000","last_change":"2024-09-17T03:52:39.008943+0000","last_active":"2024-09-17T03:52:40.288095+0000","last_peered":"2024-09-17T03:52:40.288095+0000","last_clean":"2024-09-17T03:52:40.288095+0000","last_became_active":"2024-09-17T03:52:39.008666+0000","last_became_peered":"2024-09-17T03:52:39.008666+0000","last_unstale":"2024-09-17T03:52:40.288095+0000","last_undegraded":"2024-09-17T03:52:40.288095+0000","last_fullsized":"2024-09-17T03:52:40.288095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:54.155207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289918+0000","last_change":"2024-09-17T03:52:39.027699+0000","last_active":"2024-09-17T03:52:40.289918+0000","last_peered":"2024-09-17T03:52:40.289918+0000","last_clean":"2024-09-17T03:52:40.289918+0000","last_became_active":"2024-09-17T03:52:39.026422+0000","last_became_peered":"2024-09-17T03:52:39.026422+0000","last_unstale":"2024-09-17T03:52:40.289918+0000","last_undegraded":"2024-09-17T03:52:40.289918+0000","last_fullsized":"2024-09-17T03:52:40.289918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:53.816696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290247+0000","last_change":"2024-09-17T03:52:39.030179+0000","last_active":"2024-09-17T03:52:40.290247+0000","last_peered":"2024-09-17T03:52:40.290247+0000","last_clean":"2024-09-17T03:52:40.290247+0000","last_became_active":"2024-09-17T03:52:39.029845+0000","last_became_peered":"2024-09-17T03:52:39.029845+0000","last_unstale":"2024-09-17T03:52:40.290247+0000","last_undegraded":"2024-09-17T03:52:40.290247+0000","last_fullsized":"2024-09-17T03:52:40.290247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:27.084828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289510+0000","last_change":"2024-09-17T03:52:39.013300+0000","last_active":"2024-09-17T03:52:40.289510+0000","last_peered":"2024-09-17T03:52:40.289510+0000","last_clean":"2024-09-17T03:52:40.289510+0000","last_became_active":"2024-09-17T03:52:39.012947+0000","last_became_peered":"2024-09-17T03:52:39.012947+0000","last_unstale":"2024-09-17T03:52:40.289510+0000","last_undegraded":"2024-09-17T03:52:40.289510+0000","last_fullsized":"2024-09-17T03:52:40.289510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:42.931837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288725+0000","last_change":"2024-09-17T03:52:39.021402+0000","last_active":"2024-09-17T03:52:40.288725+0000","last_peered":"2024-09-17T03:52:40.288725+0000","last_clean":"2024-09-17T03:52:40.288725+0000","last_became_active":"2024-09-17T03:52:39.021091+0000","last_became_peered":"2024-09-17T03:52:39.021091+0000","last_unstale":"2024-09-17T03:52:40.288725+0000","last_undegraded":"2024-09-17T03:52:40.288725+0000","last_fullsized":"2024-09-17T03:52:40.288725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:56.486110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.034509+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.033549+0000","last_became_peered":"2024-09-17T03:52:39.033549+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008078+0000","last_change":"2024-09-17T03:52:39.021427+0000","last_active":"2024-09-17T03:52:40.008078+0000","last_peered":"2024-09-17T03:52:40.008078+0000","last_clean":"2024-09-17T03:52:40.008078+0000","last_became_active":"2024-09-17T03:52:39.020901+0000","last_became_peered":"2024-09-17T03:52:39.020901+0000","last_unstale":"2024-09-17T03:52:40.008078+0000","last_undegraded":"2024-09-17T03:52:40.008078+0000","last_fullsized":"2024-09-17T03:52:40.008078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:51.900838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288900+0000","last_change":"2024-09-17T03:52:39.024384+0000","last_active":"2024-09-17T03:52:40.288900+0000","last_peered":"2024-09-17T03:52:40.288900+0000","last_clean":"2024-09-17T03:52:40.288900+0000","last_became_active":"2024-09-17T03:52:39.015719+0000","last_became_peered":"2024-09-17T03:52:39.015719+0000","last_unstale":"2024-09-17T03:52:40.288900+0000","last_undegraded":"2024-09-17T03:52:40.288900+0000","last_fullsized":"2024-09-17T03:52:40.288900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:27.198631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.031532+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.031235+0000","last_became_peered":"2024-09-17T03:52:39.031235+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:37.845837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288590+0000","last_change":"2024-09-17T03:52:39.013885+0000","last_active":"2024-09-17T03:52:40.288590+0000","last_peered":"2024-09-17T03:52:40.288590+0000","last_clean":"2024-09-17T03:52:40.288590+0000","last_became_active":"2024-09-17T03:52:39.013584+0000","last_became_peered":"2024-09-17T03:52:39.013584+0000","last_unstale":"2024-09-17T03:52:40.288590+0000","last_undegraded":"2024-09-17T03:52:40.288590+0000","last_fullsized":"2024-09-17T03:52:40.288590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:11.166875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007902+0000","last_change":"2024-09-17T03:52:39.016332+0000","last_active":"2024-09-17T03:52:40.007902+0000","last_peered":"2024-09-17T03:52:40.007902+0000","last_clean":"2024-09-17T03:52:40.007902+0000","last_became_active":"2024-09-17T03:52:39.015950+0000","last_became_peered":"2024-09-17T03:52:39.015950+0000","last_unstale":"2024-09-17T03:52:40.007902+0000","last_undegraded":"2024-09-17T03:52:40.007902+0000","last_fullsized":"2024-09-17T03:52:40.007902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008198+0000","last_change":"2024-09-17T03:52:39.026096+0000","last_active":"2024-09-17T03:52:40.008198+0000","last_peered":"2024-09-17T03:52:40.008198+0000","last_clean":"2024-09-17T03:52:40.008198+0000","last_became_active":"2024-09-17T03:52:39.025309+0000","last_became_peered":"2024-09-17T03:52:39.025309+0000","last_unstale":"2024-09-17T03:52:40.008198+0000","last_undegraded":"2024-09-17T03:52:40.008198+0000","last_fullsized":"2024-09-17T03:52:40.008198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:56:57.011981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289334+0000","last_change":"2024-09-17T03:52:39.019541+0000","last_active":"2024-09-17T03:52:40.289334+0000","last_peered":"2024-09-17T03:52:40.289334+0000","last_clean":"2024-09-17T03:52:40.289334+0000","last_became_active":"2024-09-17T03:52:39.016360+0000","last_became_peered":"2024-09-17T03:52:39.016360+0000","last_unstale":"2024-09-17T03:52:40.289334+0000","last_undegraded":"2024-09-17T03:52:40.289334+0000","last_fullsized":"2024-09-17T03:52:40.289334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:32.145164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289498+0000","last_change":"2024-09-17T03:52:39.023210+0000","last_active":"2024-09-17T03:52:40.289498+0000","last_peered":"2024-09-17T03:52:40.289498+0000","last_clean":"2024-09-17T03:52:40.289498+0000","last_became_active":"2024-09-17T03:52:39.022850+0000","last_became_peered":"2024-09-17T03:52:39.022850+0000","last_unstale":"2024-09-17T03:52:40.289498+0000","last_undegraded":"2024-09-17T03:52:40.289498+0000","last_fullsized":"2024-09-17T03:52:40.289498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:28.421892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291047+0000","last_change":"2024-09-17T03:52:38.004383+0000","last_active":"2024-09-17T03:52:40.291047+0000","last_peered":"2024-09-17T03:52:40.291047+0000","last_clean":"2024-09-17T03:52:40.291047+0000","last_became_active":"2024-09-17T03:52:38.003610+0000","last_became_peered":"2024-09-17T03:52:38.003610+0000","last_unstale":"2024-09-17T03:52:40.291047+0000","last_undegraded":"2024-09-17T03:52:40.291047+0000","last_fullsized":"2024-09-17T03:52:40.291047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:36.838884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009806+0000","last_change":"2024-09-17T03:52:39.017710+0000","last_active":"2024-09-17T03:52:40.009806+0000","last_peered":"2024-09-17T03:52:40.009806+0000","last_clean":"2024-09-17T03:52:40.009806+0000","last_became_active":"2024-09-17T03:52:39.009010+0000","last_became_peered":"2024-09-17T03:52:39.009010+0000","last_unstale":"2024-09-17T03:52:40.009806+0000","last_undegraded":"2024-09-17T03:52:40.009806+0000","last_fullsized":"2024-09-17T03:52:40.009806+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:50.092155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288354+0000","last_change":"2024-09-17T03:52:37.987588+0000","last_active":"2024-09-17T03:52:40.288354+0000","last_peered":"2024-09-17T03:52:40.288354+0000","last_clean":"2024-09-17T03:52:40.288354+0000","last_became_active":"2024-09-17T03:52:37.986620+0000","last_became_peered":"2024-09-17T03:52:37.986620+0000","last_unstale":"2024-09-17T03:52:40.288354+0000","last_undegraded":"2024-09-17T03:52:40.288354+0000","last_fullsized":"2024-09-17T03:52:40.288354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:58.890146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290564+0000","last_change":"2024-09-17T03:52:39.028972+0000","last_active":"2024-09-17T03:52:40.290564+0000","last_peered":"2024-09-17T03:52:40.290564+0000","last_clean":"2024-09-17T03:52:40.290564+0000","last_became_active":"2024-09-17T03:52:39.028419+0000","last_became_peered":"2024-09-17T03:52:39.028419+0000","last_unstale":"2024-09-17T03:52:40.290564+0000","last_undegraded":"2024-09-17T03:52:40.290564+0000","last_fullsized":"2024-09-17T03:52:40.290564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:56.889010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290134+0000","last_change":"2024-09-17T03:52:38.008448+0000","last_active":"2024-09-17T03:52:40.290134+0000","last_peered":"2024-09-17T03:52:40.290134+0000","last_clean":"2024-09-17T03:52:40.290134+0000","last_became_active":"2024-09-17T03:52:38.008134+0000","last_became_peered":"2024-09-17T03:52:38.008134+0000","last_unstale":"2024-09-17T03:52:40.290134+0000","last_undegraded":"2024-09-17T03:52:40.290134+0000","last_fullsized":"2024-09-17T03:52:40.290134+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:05.150798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289424+0000","last_change":"2024-09-17T03:52:39.009452+0000","last_active":"2024-09-17T03:52:40.289424+0000","last_peered":"2024-09-17T03:52:40.289424+0000","last_clean":"2024-09-17T03:52:40.289424+0000","last_became_active":"2024-09-17T03:52:39.005725+0000","last_became_peered":"2024-09-17T03:52:39.005725+0000","last_unstale":"2024-09-17T03:52:40.289424+0000","last_undegraded":"2024-09-17T03:52:40.289424+0000","last_fullsized":"2024-09-17T03:52:40.289424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:09.356525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290762+0000","last_change":"2024-09-17T03:52:38.004751+0000","last_active":"2024-09-17T03:52:40.290762+0000","last_peered":"2024-09-17T03:52:40.290762+0000","last_clean":"2024-09-17T03:52:40.290762+0000","last_became_active":"2024-09-17T03:52:38.004436+0000","last_became_peered":"2024-09-17T03:52:38.004436+0000","last_unstale":"2024-09-17T03:52:40.290762+0000","last_undegraded":"2024-09-17T03:52:40.290762+0000","last_fullsized":"2024-09-17T03:52:40.290762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:14.007022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008412+0000","last_change":"2024-09-17T03:52:39.023295+0000","last_active":"2024-09-17T03:52:40.008412+0000","last_peered":"2024-09-17T03:52:40.008412+0000","last_clean":"2024-09-17T03:52:40.008412+0000","last_became_active":"2024-09-17T03:52:39.022209+0000","last_became_peered":"2024-09-17T03:52:39.022209+0000","last_unstale":"2024-09-17T03:52:40.008412+0000","last_undegraded":"2024-09-17T03:52:40.008412+0000","last_fullsized":"2024-09-17T03:52:40.008412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:02.778195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288785+0000","last_change":"2024-09-17T03:52:38.004706+0000","last_active":"2024-09-17T03:52:40.288785+0000","last_peered":"2024-09-17T03:52:40.288785+0000","last_clean":"2024-09-17T03:52:40.288785+0000","last_became_active":"2024-09-17T03:52:38.004384+0000","last_became_peered":"2024-09-17T03:52:38.004384+0000","last_unstale":"2024-09-17T03:52:40.288785+0000","last_undegraded":"2024-09-17T03:52:40.288785+0000","last_fullsized":"2024-09-17T03:52:40.288785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:24.947156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290252+0000","last_change":"2024-09-17T03:52:39.033871+0000","last_active":"2024-09-17T03:52:40.290252+0000","last_peered":"2024-09-17T03:52:40.290252+0000","last_clean":"2024-09-17T03:52:40.290252+0000","last_became_active":"2024-09-17T03:52:39.033350+0000","last_became_peered":"2024-09-17T03:52:39.033350+0000","last_unstale":"2024-09-17T03:52:40.290252+0000","last_undegraded":"2024-09-17T03:52:40.290252+0000","last_fullsized":"2024-09-17T03:52:40.290252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290522+0000","last_change":"2024-09-17T03:52:38.007787+0000","last_active":"2024-09-17T03:52:40.290522+0000","last_peered":"2024-09-17T03:52:40.290522+0000","last_clean":"2024-09-17T03:52:40.290522+0000","last_became_active":"2024-09-17T03:52:38.007287+0000","last_became_peered":"2024-09-17T03:52:38.007287+0000","last_unstale":"2024-09-17T03:52:40.290522+0000","last_undegraded":"2024-09-17T03:52:40.290522+0000","last_fullsized":"2024-09-17T03:52:40.290522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:37.387663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007768+0000","last_change":"2024-09-17T03:52:39.026791+0000","last_active":"2024-09-17T03:52:40.007768+0000","last_peered":"2024-09-17T03:52:40.007768+0000","last_clean":"2024-09-17T03:52:40.007768+0000","last_became_active":"2024-09-17T03:52:39.026369+0000","last_became_peered":"2024-09-17T03:52:39.026369+0000","last_unstale":"2024-09-17T03:52:40.007768+0000","last_undegraded":"2024-09-17T03:52:40.007768+0000","last_fullsized":"2024-09-17T03:52:40.007768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:11.999239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007539+0000","last_change":"2024-09-17T03:52:38.004637+0000","last_active":"2024-09-17T03:52:40.007539+0000","last_peered":"2024-09-17T03:52:40.007539+0000","last_clean":"2024-09-17T03:52:40.007539+0000","last_became_active":"2024-09-17T03:52:38.004076+0000","last_became_peered":"2024-09-17T03:52:38.004076+0000","last_unstale":"2024-09-17T03:52:40.007539+0000","last_undegraded":"2024-09-17T03:52:40.007539+0000","last_fullsized":"2024-09-17T03:52:40.007539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:38.214053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287689+0000","last_change":"2024-09-17T03:52:39.000484+0000","last_active":"2024-09-17T03:52:40.287689+0000","last_peered":"2024-09-17T03:52:40.287689+0000","last_clean":"2024-09-17T03:52:40.287689+0000","last_became_active":"2024-09-17T03:52:39.000154+0000","last_became_peered":"2024-09-17T03:52:39.000154+0000","last_unstale":"2024-09-17T03:52:40.287689+0000","last_undegraded":"2024-09-17T03:52:40.287689+0000","last_fullsized":"2024-09-17T03:52:40.287689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008024+0000","last_change":"2024-09-17T03:52:37.992784+0000","last_active":"2024-09-17T03:52:40.008024+0000","last_peered":"2024-09-17T03:52:40.008024+0000","last_clean":"2024-09-17T03:52:40.008024+0000","last_became_active":"2024-09-17T03:52:37.992480+0000","last_became_peered":"2024-09-17T03:52:37.992480+0000","last_unstale":"2024-09-17T03:52:40.008024+0000","last_undegraded":"2024-09-17T03:52:40.008024+0000","last_fullsized":"2024-09-17T03:52:40.008024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:21.250972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008452+0000","last_change":"2024-09-17T03:52:39.008286+0000","last_active":"2024-09-17T03:52:40.008452+0000","last_peered":"2024-09-17T03:52:40.008452+0000","last_clean":"2024-09-17T03:52:40.008452+0000","last_became_active":"2024-09-17T03:52:39.007952+0000","last_became_peered":"2024-09-17T03:52:39.007952+0000","last_unstale":"2024-09-17T03:52:40.008452+0000","last_undegraded":"2024-09-17T03:52:40.008452+0000","last_fullsized":"2024-09-17T03:52:40.008452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:30.540116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290933+0000","last_change":"2024-09-17T03:52:38.001450+0000","last_active":"2024-09-17T03:52:40.290933+0000","last_peered":"2024-09-17T03:52:40.290933+0000","last_clean":"2024-09-17T03:52:40.290933+0000","last_became_active":"2024-09-17T03:52:38.001124+0000","last_became_peered":"2024-09-17T03:52:38.001124+0000","last_unstale":"2024-09-17T03:52:40.290933+0000","last_undegraded":"2024-09-17T03:52:40.290933+0000","last_fullsized":"2024-09-17T03:52:40.290933+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:53.092676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289507+0000","last_change":"2024-09-17T03:52:39.027159+0000","last_active":"2024-09-17T03:52:40.289507+0000","last_peered":"2024-09-17T03:52:40.289507+0000","last_clean":"2024-09-17T03:52:40.289507+0000","last_became_active":"2024-09-17T03:52:39.026074+0000","last_became_peered":"2024-09-17T03:52:39.026074+0000","last_unstale":"2024-09-17T03:52:40.289507+0000","last_undegraded":"2024-09-17T03:52:40.289507+0000","last_fullsized":"2024-09-17T03:52:40.289507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:28.966676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291290+0000","last_change":"2024-09-17T03:52:38.004849+0000","last_active":"2024-09-17T03:52:40.291290+0000","last_peered":"2024-09-17T03:52:40.291290+0000","last_clean":"2024-09-17T03:52:40.291290+0000","last_became_active":"2024-09-17T03:52:38.002393+0000","last_became_peered":"2024-09-17T03:52:38.002393+0000","last_unstale":"2024-09-17T03:52:40.291290+0000","last_undegraded":"2024-09-17T03:52:40.291290+0000","last_fullsized":"2024-09-17T03:52:40.291290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:06.095763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290594+0000","last_change":"2024-09-17T03:52:39.024541+0000","last_active":"2024-09-17T03:52:40.290594+0000","last_peered":"2024-09-17T03:52:40.290594+0000","last_clean":"2024-09-17T03:52:40.290594+0000","last_became_active":"2024-09-17T03:52:39.023962+0000","last_became_peered":"2024-09-17T03:52:39.023962+0000","last_unstale":"2024-09-17T03:52:40.290594+0000","last_undegraded":"2024-09-17T03:52:40.290594+0000","last_fullsized":"2024-09-17T03:52:40.290594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:05.325707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289078+0000","last_change":"2024-09-17T03:52:38.002263+0000","last_active":"2024-09-17T03:52:40.289078+0000","last_peered":"2024-09-17T03:52:40.289078+0000","last_clean":"2024-09-17T03:52:40.289078+0000","last_became_active":"2024-09-17T03:52:38.001712+0000","last_became_peered":"2024-09-17T03:52:38.001712+0000","last_unstale":"2024-09-17T03:52:40.289078+0000","last_undegraded":"2024-09-17T03:52:40.289078+0000","last_fullsized":"2024-09-17T03:52:40.289078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:53.008257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008530+0000","last_change":"2024-09-17T03:52:39.015946+0000","last_active":"2024-09-17T03:52:40.008530+0000","last_peered":"2024-09-17T03:52:40.008530+0000","last_clean":"2024-09-17T03:52:40.008530+0000","last_became_active":"2024-09-17T03:52:39.008987+0000","last_became_peered":"2024-09-17T03:52:39.008987+0000","last_unstale":"2024-09-17T03:52:40.008530+0000","last_undegraded":"2024-09-17T03:52:40.008530+0000","last_fullsized":"2024-09-17T03:52:40.008530+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:24.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008445+0000","last_change":"2024-09-17T03:52:38.000155+0000","last_active":"2024-09-17T03:52:40.008445+0000","last_peered":"2024-09-17T03:52:40.008445+0000","last_clean":"2024-09-17T03:52:40.008445+0000","last_became_active":"2024-09-17T03:52:37.999044+0000","last_became_peered":"2024-09-17T03:52:37.999044+0000","last_unstale":"2024-09-17T03:52:40.008445+0000","last_undegraded":"2024-09-17T03:52:40.008445+0000","last_fullsized":"2024-09-17T03:52:40.008445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:15.512221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289210+0000","last_change":"2024-09-17T03:52:39.015628+0000","last_active":"2024-09-17T03:52:40.289210+0000","last_peered":"2024-09-17T03:52:40.289210+0000","last_clean":"2024-09-17T03:52:40.289210+0000","last_became_active":"2024-09-17T03:52:39.015286+0000","last_became_peered":"2024-09-17T03:52:39.015286+0000","last_unstale":"2024-09-17T03:52:40.289210+0000","last_undegraded":"2024-09-17T03:52:40.289210+0000","last_fullsized":"2024-09-17T03:52:40.289210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:02.891865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007787+0000","last_change":"2024-09-17T03:52:38.001510+0000","last_active":"2024-09-17T03:52:40.007787+0000","last_peered":"2024-09-17T03:52:40.007787+0000","last_clean":"2024-09-17T03:52:40.007787+0000","last_became_active":"2024-09-17T03:52:38.001109+0000","last_became_peered":"2024-09-17T03:52:38.001109+0000","last_unstale":"2024-09-17T03:52:40.007787+0000","last_undegraded":"2024-09-17T03:52:40.007787+0000","last_fullsized":"2024-09-17T03:52:40.007787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:53.377063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008218+0000","last_change":"2024-09-17T03:52:39.021149+0000","last_active":"2024-09-17T03:52:40.008218+0000","last_peered":"2024-09-17T03:52:40.008218+0000","last_clean":"2024-09-17T03:52:40.008218+0000","last_became_active":"2024-09-17T03:52:39.020787+0000","last_became_peered":"2024-09-17T03:52:39.020787+0000","last_unstale":"2024-09-17T03:52:40.008218+0000","last_undegraded":"2024-09-17T03:52:40.008218+0000","last_fullsized":"2024-09-17T03:52:40.008218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:49.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290649+0000","last_change":"2024-09-17T03:52:38.000222+0000","last_active":"2024-09-17T03:52:40.290649+0000","last_peered":"2024-09-17T03:52:40.290649+0000","last_clean":"2024-09-17T03:52:40.290649+0000","last_became_active":"2024-09-17T03:52:37.998870+0000","last_became_peered":"2024-09-17T03:52:37.998870+0000","last_unstale":"2024-09-17T03:52:40.290649+0000","last_undegraded":"2024-09-17T03:52:40.290649+0000","last_fullsized":"2024-09-17T03:52:40.290649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:01.261692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287904+0000","last_change":"2024-09-17T03:52:39.015246+0000","last_active":"2024-09-17T03:52:40.287904+0000","last_peered":"2024-09-17T03:52:40.287904+0000","last_clean":"2024-09-17T03:52:40.287904+0000","last_became_active":"2024-09-17T03:52:39.014938+0000","last_became_peered":"2024-09-17T03:52:39.014938+0000","last_unstale":"2024-09-17T03:52:40.287904+0000","last_undegraded":"2024-09-17T03:52:40.287904+0000","last_fullsized":"2024-09-17T03:52:40.287904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:12.786786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007815+0000","last_change":"2024-09-17T03:52:37.993426+0000","last_active":"2024-09-17T03:52:40.007815+0000","last_peered":"2024-09-17T03:52:40.007815+0000","last_clean":"2024-09-17T03:52:40.007815+0000","last_became_active":"2024-09-17T03:52:37.992558+0000","last_became_peered":"2024-09-17T03:52:37.992558+0000","last_unstale":"2024-09-17T03:52:40.007815+0000","last_undegraded":"2024-09-17T03:52:40.007815+0000","last_fullsized":"2024-09-17T03:52:40.007815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:50.687889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009000+0000","last_change":"2024-09-17T03:52:39.020708+0000","last_active":"2024-09-17T03:52:40.009000+0000","last_peered":"2024-09-17T03:52:40.009000+0000","last_clean":"2024-09-17T03:52:40.009000+0000","last_became_active":"2024-09-17T03:52:39.019989+0000","last_became_peered":"2024-09-17T03:52:39.019989+0000","last_unstale":"2024-09-17T03:52:40.009000+0000","last_undegraded":"2024-09-17T03:52:40.009000+0000","last_fullsized":"2024-09-17T03:52:40.009000+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:26.316176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289749+0000","last_change":"2024-09-17T03:52:38.000317+0000","last_active":"2024-09-17T03:52:40.289749+0000","last_peered":"2024-09-17T03:52:40.289749+0000","last_clean":"2024-09-17T03:52:40.289749+0000","last_became_active":"2024-09-17T03:52:37.999933+0000","last_became_peered":"2024-09-17T03:52:37.999933+0000","last_unstale":"2024-09-17T03:52:40.289749+0000","last_undegraded":"2024-09-17T03:52:40.289749+0000","last_fullsized":"2024-09-17T03:52:40.289749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:56.251630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290593+0000","last_change":"2024-09-17T03:52:39.025015+0000","last_active":"2024-09-17T03:52:40.290593+0000","last_peered":"2024-09-17T03:52:40.290593+0000","last_clean":"2024-09-17T03:52:40.290593+0000","last_became_active":"2024-09-17T03:52:39.024596+0000","last_became_peered":"2024-09-17T03:52:39.024596+0000","last_unstale":"2024-09-17T03:52:40.290593+0000","last_undegraded":"2024-09-17T03:52:40.290593+0000","last_fullsized":"2024-09-17T03:52:40.290593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291013+0000","last_change":"2024-09-17T03:52:38.007757+0000","last_active":"2024-09-17T03:52:40.291013+0000","last_peered":"2024-09-17T03:52:40.291013+0000","last_clean":"2024-09-17T03:52:40.291013+0000","last_became_active":"2024-09-17T03:52:38.007282+0000","last_became_peered":"2024-09-17T03:52:38.007282+0000","last_unstale":"2024-09-17T03:52:40.291013+0000","last_undegraded":"2024-09-17T03:52:40.291013+0000","last_fullsized":"2024-09-17T03:52:40.291013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.565057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289931+0000","last_change":"2024-09-17T03:52:39.007897+0000","last_active":"2024-09-17T03:52:40.289931+0000","last_peered":"2024-09-17T03:52:40.289931+0000","last_clean":"2024-09-17T03:52:40.289931+0000","last_became_active":"2024-09-17T03:52:39.007557+0000","last_became_peered":"2024-09-17T03:52:39.007557+0000","last_unstale":"2024-09-17T03:52:40.289931+0000","last_undegraded":"2024-09-17T03:52:40.289931+0000","last_fullsized":"2024-09-17T03:52:40.289931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:22.330075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290700+0000","last_change":"2024-09-17T03:52:38.002206+0000","last_active":"2024-09-17T03:52:40.290700+0000","last_peered":"2024-09-17T03:52:40.290700+0000","last_clean":"2024-09-17T03:52:40.290700+0000","last_became_active":"2024-09-17T03:52:38.001872+0000","last_became_peered":"2024-09-17T03:52:38.001872+0000","last_unstale":"2024-09-17T03:52:40.290700+0000","last_undegraded":"2024-09-17T03:52:40.290700+0000","last_fullsized":"2024-09-17T03:52:40.290700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:18.787765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008257+0000","last_change":"2024-09-17T03:52:39.018968+0000","last_active":"2024-09-17T03:52:40.008257+0000","last_peered":"2024-09-17T03:52:40.008257+0000","last_clean":"2024-09-17T03:52:40.008257+0000","last_became_active":"2024-09-17T03:52:39.018682+0000","last_became_peered":"2024-09-17T03:52:39.018682+0000","last_unstale":"2024-09-17T03:52:40.008257+0000","last_undegraded":"2024-09-17T03:52:40.008257+0000","last_fullsized":"2024-09-17T03:52:40.008257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:06.157752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290986+0000","last_change":"2024-09-17T03:52:38.009348+0000","last_active":"2024-09-17T03:52:40.290986+0000","last_peered":"2024-09-17T03:52:40.290986+0000","last_clean":"2024-09-17T03:52:40.290986+0000","last_became_active":"2024-09-17T03:52:38.008926+0000","last_became_peered":"2024-09-17T03:52:38.008926+0000","last_unstale":"2024-09-17T03:52:40.290986+0000","last_undegraded":"2024-09-17T03:52:40.290986+0000","last_fullsized":"2024-09-17T03:52:40.290986+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:27.731425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008745+0000","last_change":"2024-09-17T03:52:39.021118+0000","last_active":"2024-09-17T03:52:40.008745+0000","last_peered":"2024-09-17T03:52:40.008745+0000","last_clean":"2024-09-17T03:52:40.008745+0000","last_became_active":"2024-09-17T03:52:39.020763+0000","last_became_peered":"2024-09-17T03:52:39.020763+0000","last_unstale":"2024-09-17T03:52:40.008745+0000","last_undegraded":"2024-09-17T03:52:40.008745+0000","last_fullsized":"2024-09-17T03:52:40.008745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:10.895566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289021+0000","last_change":"2024-09-17T03:52:38.005886+0000","last_active":"2024-09-17T03:52:40.289021+0000","last_peered":"2024-09-17T03:52:40.289021+0000","last_clean":"2024-09-17T03:52:40.289021+0000","last_became_active":"2024-09-17T03:52:38.005550+0000","last_became_peered":"2024-09-17T03:52:38.005550+0000","last_unstale":"2024-09-17T03:52:40.289021+0000","last_undegraded":"2024-09-17T03:52:40.289021+0000","last_fullsized":"2024-09-17T03:52:40.289021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:36.591587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288662+0000","last_change":"2024-09-17T03:52:39.009574+0000","last_active":"2024-09-17T03:52:40.288662+0000","last_peered":"2024-09-17T03:52:40.288662+0000","last_clean":"2024-09-17T03:52:40.288662+0000","last_became_active":"2024-09-17T03:52:39.009382+0000","last_became_peered":"2024-09-17T03:52:39.009382+0000","last_unstale":"2024-09-17T03:52:40.288662+0000","last_undegraded":"2024-09-17T03:52:40.288662+0000","last_fullsized":"2024-09-17T03:52:40.288662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:28.472361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290771+0000","last_change":"2024-09-17T03:52:38.006146+0000","last_active":"2024-09-17T03:52:40.290771+0000","last_peered":"2024-09-17T03:52:40.290771+0000","last_clean":"2024-09-17T03:52:40.290771+0000","last_became_active":"2024-09-17T03:52:38.005099+0000","last_became_peered":"2024-09-17T03:52:38.005099+0000","last_unstale":"2024-09-17T03:52:40.290771+0000","last_undegraded":"2024-09-17T03:52:40.290771+0000","last_fullsized":"2024-09-17T03:52:40.290771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:14.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288305+0000","last_change":"2024-09-17T03:52:39.020395+0000","last_active":"2024-09-17T03:52:40.288305+0000","last_peered":"2024-09-17T03:52:40.288305+0000","last_clean":"2024-09-17T03:52:40.288305+0000","last_became_active":"2024-09-17T03:52:39.019294+0000","last_became_peered":"2024-09-17T03:52:39.019294+0000","last_unstale":"2024-09-17T03:52:40.288305+0000","last_undegraded":"2024-09-17T03:52:40.288305+0000","last_fullsized":"2024-09-17T03:52:40.288305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:11.716349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008291+0000","last_change":"2024-09-17T03:52:37.986744+0000","last_active":"2024-09-17T03:52:40.008291+0000","last_peered":"2024-09-17T03:52:40.008291+0000","last_clean":"2024-09-17T03:52:40.008291+0000","last_became_active":"2024-09-17T03:52:37.986385+0000","last_became_peered":"2024-09-17T03:52:37.986385+0000","last_unstale":"2024-09-17T03:52:40.008291+0000","last_undegraded":"2024-09-17T03:52:40.008291+0000","last_fullsized":"2024-09-17T03:52:40.008291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.274408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288402+0000","last_change":"2024-09-17T03:52:39.019842+0000","last_active":"2024-09-17T03:52:40.288402+0000","last_peered":"2024-09-17T03:52:40.288402+0000","last_clean":"2024-09-17T03:52:40.288402+0000","last_became_active":"2024-09-17T03:52:39.019548+0000","last_became_peered":"2024-09-17T03:52:39.019548+0000","last_unstale":"2024-09-17T03:52:40.288402+0000","last_undegraded":"2024-09-17T03:52:40.288402+0000","last_fullsized":"2024-09-17T03:52:40.288402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:21.225549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287600+0000","last_change":"2024-09-17T03:52:37.987589+0000","last_active":"2024-09-17T03:52:40.287600+0000","last_peered":"2024-09-17T03:52:40.287600+0000","last_clean":"2024-09-17T03:52:40.287600+0000","last_became_active":"2024-09-17T03:52:37.986646+0000","last_became_peered":"2024-09-17T03:52:37.986646+0000","last_unstale":"2024-09-17T03:52:40.287600+0000","last_undegraded":"2024-09-17T03:52:40.287600+0000","last_fullsized":"2024-09-17T03:52:40.287600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:04.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288208+0000","last_change":"2024-09-17T03:52:39.017123+0000","last_active":"2024-09-17T03:52:40.288208+0000","last_peered":"2024-09-17T03:52:40.288208+0000","last_clean":"2024-09-17T03:52:40.288208+0000","last_became_active":"2024-09-17T03:52:39.016654+0000","last_became_peered":"2024-09-17T03:52:39.016654+0000","last_unstale":"2024-09-17T03:52:40.288208+0000","last_undegraded":"2024-09-17T03:52:40.288208+0000","last_fullsized":"2024-09-17T03:52:40.288208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:41.588250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291161+0000","last_change":"2024-09-17T03:52:38.007301+0000","last_active":"2024-09-17T03:52:40.291161+0000","last_peered":"2024-09-17T03:52:40.291161+0000","last_clean":"2024-09-17T03:52:40.291161+0000","last_became_active":"2024-09-17T03:52:38.006871+0000","last_became_peered":"2024-09-17T03:52:38.006871+0000","last_unstale":"2024-09-17T03:52:40.291161+0000","last_undegraded":"2024-09-17T03:52:40.291161+0000","last_fullsized":"2024-09-17T03:52:40.291161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:40.821800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008418+0000","last_change":"2024-09-17T03:52:39.015261+0000","last_active":"2024-09-17T03:52:40.008418+0000","last_peered":"2024-09-17T03:52:40.008418+0000","last_clean":"2024-09-17T03:52:40.008418+0000","last_became_active":"2024-09-17T03:52:39.014907+0000","last_became_peered":"2024-09-17T03:52:39.014907+0000","last_unstale":"2024-09-17T03:52:40.008418+0000","last_undegraded":"2024-09-17T03:52:40.008418+0000","last_fullsized":"2024-09-17T03:52:40.008418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:25.145657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008635+0000","last_change":"2024-09-17T03:52:37.993729+0000","last_active":"2024-09-17T03:52:40.008635+0000","last_peered":"2024-09-17T03:52:40.008635+0000","last_clean":"2024-09-17T03:52:40.008635+0000","last_became_active":"2024-09-17T03:52:37.991713+0000","last_became_peered":"2024-09-17T03:52:37.991713+0000","last_unstale":"2024-09-17T03:52:40.008635+0000","last_undegraded":"2024-09-17T03:52:40.008635+0000","last_fullsized":"2024-09-17T03:52:40.008635+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:14.304806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288837+0000","last_change":"2024-09-17T03:52:39.024919+0000","last_active":"2024-09-17T03:52:40.288837+0000","last_peered":"2024-09-17T03:52:40.288837+0000","last_clean":"2024-09-17T03:52:40.288837+0000","last_became_active":"2024-09-17T03:52:39.022961+0000","last_became_peered":"2024-09-17T03:52:39.022961+0000","last_unstale":"2024-09-17T03:52:40.288837+0000","last_undegraded":"2024-09-17T03:52:40.288837+0000","last_fullsized":"2024-09-17T03:52:40.288837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:47.897977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008905+0000","last_change":"2024-09-17T03:52:38.002436+0000","last_active":"2024-09-17T03:52:40.008905+0000","last_peered":"2024-09-17T03:52:40.008905+0000","last_clean":"2024-09-17T03:52:40.008905+0000","last_became_active":"2024-09-17T03:52:38.001380+0000","last_became_peered":"2024-09-17T03:52:38.001380+0000","last_unstale":"2024-09-17T03:52:40.008905+0000","last_undegraded":"2024-09-17T03:52:40.008905+0000","last_fullsized":"2024-09-17T03:52:40.008905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:16.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289539+0000","last_change":"2024-09-17T03:52:39.013637+0000","last_active":"2024-09-17T03:52:40.289539+0000","last_peered":"2024-09-17T03:52:40.289539+0000","last_clean":"2024-09-17T03:52:40.289539+0000","last_became_active":"2024-09-17T03:52:39.013035+0000","last_became_peered":"2024-09-17T03:52:39.013035+0000","last_unstale":"2024-09-17T03:52:40.289539+0000","last_undegraded":"2024-09-17T03:52:40.289539+0000","last_fullsized":"2024-09-17T03:52:40.289539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:39.874354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287877+0000","last_change":"2024-09-17T03:52:37.998007+0000","last_active":"2024-09-17T03:52:40.287877+0000","last_peered":"2024-09-17T03:52:40.287877+0000","last_clean":"2024-09-17T03:52:40.287877+0000","last_became_active":"2024-09-17T03:52:37.997670+0000","last_became_peered":"2024-09-17T03:52:37.997670+0000","last_unstale":"2024-09-17T03:52:40.287877+0000","last_undegraded":"2024-09-17T03:52:40.287877+0000","last_fullsized":"2024-09-17T03:52:40.287877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.236372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009681+0000","last_change":"2024-09-17T03:52:39.016581+0000","last_active":"2024-09-17T03:52:40.009681+0000","last_peered":"2024-09-17T03:52:40.009681+0000","last_clean":"2024-09-17T03:52:40.009681+0000","last_became_active":"2024-09-17T03:52:39.007886+0000","last_became_peered":"2024-09-17T03:52:39.007886+0000","last_unstale":"2024-09-17T03:52:40.009681+0000","last_undegraded":"2024-09-17T03:52:40.009681+0000","last_fullsized":"2024-09-17T03:52:40.009681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:13.021250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008230+0000","last_change":"2024-09-17T03:52:37.991052+0000","last_active":"2024-09-17T03:52:40.008230+0000","last_peered":"2024-09-17T03:52:40.008230+0000","last_clean":"2024-09-17T03:52:40.008230+0000","last_became_active":"2024-09-17T03:52:37.990089+0000","last_became_peered":"2024-09-17T03:52:37.990089+0000","last_unstale":"2024-09-17T03:52:40.008230+0000","last_undegraded":"2024-09-17T03:52:40.008230+0000","last_fullsized":"2024-09-17T03:52:40.008230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:17.450063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290527+0000","last_change":"2024-09-17T03:52:39.034798+0000","last_active":"2024-09-17T03:52:40.290527+0000","last_peered":"2024-09-17T03:52:40.290527+0000","last_clean":"2024-09-17T03:52:40.290527+0000","last_became_active":"2024-09-17T03:52:39.033757+0000","last_became_peered":"2024-09-17T03:52:39.033757+0000","last_unstale":"2024-09-17T03:52:40.290527+0000","last_undegraded":"2024-09-17T03:52:40.290527+0000","last_fullsized":"2024-09-17T03:52:40.290527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:40.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007401+0000","last_change":"2024-09-17T03:52:38.006002+0000","last_active":"2024-09-17T03:52:40.007401+0000","last_peered":"2024-09-17T03:52:40.007401+0000","last_clean":"2024-09-17T03:52:40.007401+0000","last_became_active":"2024-09-17T03:52:38.004725+0000","last_became_peered":"2024-09-17T03:52:38.004725+0000","last_unstale":"2024-09-17T03:52:40.007401+0000","last_undegraded":"2024-09-17T03:52:40.007401+0000","last_fullsized":"2024-09-17T03:52:40.007401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:05.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008222+0000","last_change":"2024-09-17T03:52:39.010421+0000","last_active":"2024-09-17T03:52:40.008222+0000","last_peered":"2024-09-17T03:52:40.008222+0000","last_clean":"2024-09-17T03:52:40.008222+0000","last_became_active":"2024-09-17T03:52:39.010077+0000","last_became_peered":"2024-09-17T03:52:39.010077+0000","last_unstale":"2024-09-17T03:52:40.008222+0000","last_undegraded":"2024-09-17T03:52:40.008222+0000","last_fullsized":"2024-09-17T03:52:40.008222+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:57.258583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289687+0000","last_change":"2024-09-17T03:52:37.998870+0000","last_active":"2024-09-17T03:52:40.289687+0000","last_peered":"2024-09-17T03:52:40.289687+0000","last_clean":"2024-09-17T03:52:40.289687+0000","last_became_active":"2024-09-17T03:52:37.998561+0000","last_became_peered":"2024-09-17T03:52:37.998561+0000","last_unstale":"2024-09-17T03:52:40.289687+0000","last_undegraded":"2024-09-17T03:52:40.289687+0000","last_fullsized":"2024-09-17T03:52:40.289687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:58.681513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290032+0000","last_change":"2024-09-17T03:52:39.022554+0000","last_active":"2024-09-17T03:52:40.290032+0000","last_peered":"2024-09-17T03:52:40.290032+0000","last_clean":"2024-09-17T03:52:40.290032+0000","last_became_active":"2024-09-17T03:52:39.022101+0000","last_became_peered":"2024-09-17T03:52:39.022101+0000","last_unstale":"2024-09-17T03:52:40.290032+0000","last_undegraded":"2024-09-17T03:52:40.290032+0000","last_fullsized":"2024-09-17T03:52:40.290032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:37.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289503+0000","last_change":"2024-09-17T03:52:38.004100+0000","last_active":"2024-09-17T03:52:40.289503+0000","last_peered":"2024-09-17T03:52:40.289503+0000","last_clean":"2024-09-17T03:52:40.289503+0000","last_became_active":"2024-09-17T03:52:38.000956+0000","last_became_peered":"2024-09-17T03:52:38.000956+0000","last_unstale":"2024-09-17T03:52:40.289503+0000","last_undegraded":"2024-09-17T03:52:40.289503+0000","last_fullsized":"2024-09-17T03:52:40.289503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:29.352497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288572+0000","last_change":"2024-09-17T03:52:39.013661+0000","last_active":"2024-09-17T03:52:40.288572+0000","last_peered":"2024-09-17T03:52:40.288572+0000","last_clean":"2024-09-17T03:52:40.288572+0000","last_became_active":"2024-09-17T03:52:39.013380+0000","last_became_peered":"2024-09-17T03:52:39.013380+0000","last_unstale":"2024-09-17T03:52:40.288572+0000","last_undegraded":"2024-09-17T03:52:40.288572+0000","last_fullsized":"2024-09-17T03:52:40.288572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:41.998544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290721+0000","last_change":"2024-09-17T03:52:38.001315+0000","last_active":"2024-09-17T03:52:40.290721+0000","last_peered":"2024-09-17T03:52:40.290721+0000","last_clean":"2024-09-17T03:52:40.290721+0000","last_became_active":"2024-09-17T03:52:37.999404+0000","last_became_peered":"2024-09-17T03:52:37.999404+0000","last_unstale":"2024-09-17T03:52:40.290721+0000","last_undegraded":"2024-09-17T03:52:40.290721+0000","last_fullsized":"2024-09-17T03:52:40.290721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:34:25.654563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288936+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:52:40.288936+0000","last_peered":"2024-09-17T03:52:40.288936+0000","last_clean":"2024-09-17T03:52:40.288936+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:52:40.288936+0000","last_undegraded":"2024-09-17T03:52:40.288936+0000","last_fullsized":"2024-09-17T03:52:40.288936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1421312,"data_stored":931564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509485,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":827392,"data_stored":340692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":831488,"data_stored":342192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":839680,"data_stored":342808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":340672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1413120,"data_stored":930948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:52:44.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T03:52:44.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 -- 172.21.15.129:0/776701697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 msgr2=0x7f6ae00b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 --2- 172.21.15.129:0/776701697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae00b3090 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8005fd0 tx=0x7f6ad802a840 comp rx=0 tx=0).stop 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 -- 172.21.15.129:0/776701697 shutdown_connections 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 --2- 172.21.15.129:0/776701697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae00b3090 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 --2- 172.21.15.129:0/776701697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 0x7f6ae015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 -- 172.21.15.129:0/776701697 >> 172.21.15.129:0/776701697 conn(0x7f6ae00b1cb0 msgr2=0x7f6ae00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 -- 172.21.15.129:0/776701697 shutdown_connections 2024-09-17T03:52:44.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.644+0000 7f6ae6800640 1 -- 172.21.15.129:0/776701697 wait complete. 2024-09-17T03:52:44.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.646+0000 7f6ae6800640 1 Processor -- start 2024-09-17T03:52:44.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.648+0000 7f6ae6800640 1 -- start start 2024-09-17T03:52:44.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 0x7f6ae01f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 0x7f6ae00c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae00b3b50 con 0x7f6ae0070440 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae00b39d0 con 0x7f6ae0070030 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae00b3cd0 con 0x7f6ae0073170 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36970/0 (socket says 172.21.15.129:36970) 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 -- 172.21.15.129:0/2648035170 learned_addr learned my addr 172.21.15.129:0/2648035170 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6adfe00640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 0x7f6ae00c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6adf400640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 0x7f6ae01f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 -- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 msgr2=0x7f6ae00c8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 0x7f6ae00c8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 -- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 msgr2=0x7f6ae01f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 0x7f6ae01f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 -- 172.21.15.129:0/2648035170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae00c7f30 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6adf400640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 0x7f6ae01f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad6a00640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c340 tx=0x7f6ac800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8017070 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ac80035f0 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae00c8210 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.649+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8003a40 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.650+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae00cb7f0 con 0x7f6ae0070440 2024-09-17T03:52:44.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.650+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae00c8450 con 0x7f6ae0070440 2024-09-17T03:52:44.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.652+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6ac801a070 con 0x7f6ae0070440 2024-09-17T03:52:44.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.653+0000 7f6ad7e00640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 0x7f6abc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:44.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.653+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6ac8099ec0 con 0x7f6ae0070440 2024-09-17T03:52:44.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.653+0000 7f6adf400640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 0x7f6abc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:44.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.654+0000 7f6adf400640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 0x7f6abc07b0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad000f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:44.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.657+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ae00c8450 con 0x7f6ae0070440 2024-09-17T03:52:44.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.914+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f6ae01f6820 con 0x7f6ae0070440 2024-09-17T03:52:44.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.914+0000 7f6ad7e00640 1 -- 172.21.15.129:0/2648035170 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1560 (secure 0 0 0) 0x7f6ac8061ca0 con 0x7f6ae0070440 2024-09-17T03:52:44.913 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:44.913 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":68,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":82},{"state_name":"creating+peering","count":15}],"num_pgs":97,"num_pools":3,"num_objects":18,"data_bytes":592686,"bytes_used":263102464,"bytes_avail":772831010816,"bytes_total":773094113280,"inactive_pgs_ratio":0.15463916957378387,"write_bytes_sec":2289,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"c","status":"up:active","gid":5865}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-17T03:52:20.338061+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.916+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 msgr2=0x7f6abc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.916+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 0x7f6abc07b0a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0005fd0 tx=0x7f6ad000f360 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.916+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 msgr2=0x7f6ae0153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.916+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c340 tx=0x7f6ac800c800 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 shutdown_connections 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6abc078b90 0x7f6abc07b0a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ae0073170 0x7f6ae00c8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ae0070440 0x7f6ae0153660 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 --2- 172.21.15.129:0/2648035170 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ae0070030 0x7f6ae01f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 >> 172.21.15.129:0/2648035170 conn(0x7f6ae00b1cb0 msgr2=0x7f6ae015ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:44.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 shutdown_connections 2024-09-17T03:52:44.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:44.917+0000 7f6ae6800640 1 -- 172.21.15.129:0/2648035170 wait complete. 2024-09-17T03:52:44.937 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T03:52:44.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:52:44.949 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:52:44.951+0000 7fc874800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:52:45.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 -- 172.21.15.129:0/2191188819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100cad30 msgr2=0x7f52100cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 --2- 172.21.15.129:0/2191188819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100cad30 0x7f52100cba00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f5208005fd0 tx=0x7f520802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 -- 172.21.15.129:0/2191188819 shutdown_connections 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 --2- 172.21.15.129:0/2191188819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100cad30 0x7f52100cba00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 -- 172.21.15.129:0/2191188819 >> 172.21.15.129:0/2191188819 conn(0x7f52100b1cb0 msgr2=0x7f52100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 -- 172.21.15.129:0/2191188819 shutdown_connections 2024-09-17T03:52:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.162+0000 7f5216000640 1 -- 172.21.15.129:0/2191188819 wait complete. 2024-09-17T03:52:45.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.164+0000 7f5216000640 1 Processor -- start 2024-09-17T03:52:45.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.165+0000 7f5216000640 1 -- start start 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 0x7f52101593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 0x7f52101f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5210074130 con 0x7f52100ca800 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5210073fb0 con 0x7f52100cad30 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f5216000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f52100742b0 con 0x7f52100c3c30 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f520f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f520f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36992/0 (socket says 172.21.15.129:36992) 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.166+0000 7f520f400640 1 -- 172.21.15.129:0/900635996 learned_addr learned my addr 172.21.15.129:0/900635996 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:45.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520ea00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 0x7f52101593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520fe00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 0x7f52101f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 -- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 msgr2=0x7f52101f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 0x7f52101f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 -- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 msgr2=0x7f52101593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 0x7f52101593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 -- 172.21.15.129:0/900635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52100c7ea0 con 0x7f52100ca800 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520fe00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 0x7f52101f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520ea00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 0x7f52101593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:45.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f520f400640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f520000c400 tx=0x7f520000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200011790 con 0x7f52100ca800 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5200017070 con 0x7f52100ca800 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52100c8180 con 0x7f52100ca800 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.167+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200011cc0 con 0x7f52100ca800 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.168+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52101f3250 con 0x7f52100ca800 2024-09-17T03:52:45.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.169+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52100747d0 con 0x7f52100ca800 2024-09-17T03:52:45.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.170+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f520001c460 con 0x7f52100ca800 2024-09-17T03:52:45.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.171+0000 7f5207400640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 0x7f51f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:45.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.171+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f520009abb0 con 0x7f52100ca800 2024-09-17T03:52:45.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.172+0000 7f520ea00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 0x7f51f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:45.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.173+0000 7f520ea00640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 0x7f51f007b0a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:45.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.176+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5200063010 con 0x7f52100ca800 2024-09-17T03:52:45.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.358+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5210159be0 con 0x7f51f0078b90 2024-09-17T03:52:45.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.364+0000 7f5207400640 1 -- 172.21.15.129:0/900635996 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252046 (secure 0 0 0) 0x7f5210159be0 con 0x7f51f0078b90 2024-09-17T03:52:45.365 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:45.367 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:52:45.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.367+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 msgr2=0x7f51f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:45.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.367+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 0x7f51f007b0a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006fd0 tx=0x7f51fc008040 comp rx=0 tx=0).stop 2024-09-17T03:52:45.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.367+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 msgr2=0x7f521009db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:45.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.367+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f520000c400 tx=0x7f520000c8c0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 shutdown_connections 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51f0078b90 0x7f51f007b0a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52100c3c30 0x7f52101f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52100cad30 0x7f52101593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 --2- 172.21.15.129:0/900635996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52100ca800 0x7f521009db20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 >> 172.21.15.129:0/900635996 conn(0x7f52100b1cb0 msgr2=0x7f521015b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 shutdown_connections 2024-09-17T03:52:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:45.368+0000 7f5216000640 1 -- 172.21.15.129:0/900635996 wait complete. 2024-09-17T03:52:45.391 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":162,"stamp":"2024-09-17T03:52:44.343730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257164,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717556,"statfs":{"total":773094113280,"available":772830777344,"internally_reserved":0,"allocated":7806976,"data_stored":3910196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.365375"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289679+0000","last_change":"2024-09-17T03:52:39.012533+0000","last_active":"2024-09-17T03:52:40.289679+0000","last_peered":"2024-09-17T03:52:40.289679+0000","last_clean":"2024-09-17T03:52:40.289679+0000","last_became_active":"2024-09-17T03:52:39.012181+0000","last_became_peered":"2024-09-17T03:52:39.012181+0000","last_unstale":"2024-09-17T03:52:40.289679+0000","last_undegraded":"2024-09-17T03:52:40.289679+0000","last_fullsized":"2024-09-17T03:52:40.289679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:54:06.915985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291299+0000","last_change":"2024-09-17T03:52:39.024835+0000","last_active":"2024-09-17T03:52:40.291299+0000","last_peered":"2024-09-17T03:52:40.291299+0000","last_clean":"2024-09-17T03:52:40.291299+0000","last_became_active":"2024-09-17T03:52:39.023996+0000","last_became_peered":"2024-09-17T03:52:39.023996+0000","last_unstale":"2024-09-17T03:52:40.291299+0000","last_undegraded":"2024-09-17T03:52:40.291299+0000","last_fullsized":"2024-09-17T03:52:40.291299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:06.467449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288896+0000","last_change":"2024-09-17T03:52:39.015268+0000","last_active":"2024-09-17T03:52:40.288896+0000","last_peered":"2024-09-17T03:52:40.288896+0000","last_clean":"2024-09-17T03:52:40.288896+0000","last_became_active":"2024-09-17T03:52:39.014959+0000","last_became_peered":"2024-09-17T03:52:39.014959+0000","last_unstale":"2024-09-17T03:52:40.288896+0000","last_undegraded":"2024-09-17T03:52:40.288896+0000","last_fullsized":"2024-09-17T03:52:40.288896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:02:44.299798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007816+0000","last_change":"2024-09-17T03:52:39.022874+0000","last_active":"2024-09-17T03:52:40.007816+0000","last_peered":"2024-09-17T03:52:40.007816+0000","last_clean":"2024-09-17T03:52:40.007816+0000","last_became_active":"2024-09-17T03:52:39.022102+0000","last_became_peered":"2024-09-17T03:52:39.022102+0000","last_unstale":"2024-09-17T03:52:40.007816+0000","last_undegraded":"2024-09-17T03:52:40.007816+0000","last_fullsized":"2024-09-17T03:52:40.007816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:11.479509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008090+0000","last_change":"2024-09-17T03:52:39.005338+0000","last_active":"2024-09-17T03:52:40.008090+0000","last_peered":"2024-09-17T03:52:40.008090+0000","last_clean":"2024-09-17T03:52:40.008090+0000","last_became_active":"2024-09-17T03:52:39.005016+0000","last_became_peered":"2024-09-17T03:52:39.005016+0000","last_unstale":"2024-09-17T03:52:40.008090+0000","last_undegraded":"2024-09-17T03:52:40.008090+0000","last_fullsized":"2024-09-17T03:52:40.008090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:32.775243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290327+0000","last_change":"2024-09-17T03:52:39.018888+0000","last_active":"2024-09-17T03:52:40.290327+0000","last_peered":"2024-09-17T03:52:40.290327+0000","last_clean":"2024-09-17T03:52:40.290327+0000","last_became_active":"2024-09-17T03:52:39.018589+0000","last_became_peered":"2024-09-17T03:52:39.018589+0000","last_unstale":"2024-09-17T03:52:40.290327+0000","last_undegraded":"2024-09-17T03:52:40.290327+0000","last_fullsized":"2024-09-17T03:52:40.290327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:12.320186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007818+0000","last_change":"2024-09-17T03:52:39.026308+0000","last_active":"2024-09-17T03:52:40.007818+0000","last_peered":"2024-09-17T03:52:40.007818+0000","last_clean":"2024-09-17T03:52:40.007818+0000","last_became_active":"2024-09-17T03:52:39.025753+0000","last_became_peered":"2024-09-17T03:52:39.025753+0000","last_unstale":"2024-09-17T03:52:40.007818+0000","last_undegraded":"2024-09-17T03:52:40.007818+0000","last_fullsized":"2024-09-17T03:52:40.007818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289062+0000","last_change":"2024-09-17T03:52:39.013777+0000","last_active":"2024-09-17T03:52:40.289062+0000","last_peered":"2024-09-17T03:52:40.289062+0000","last_clean":"2024-09-17T03:52:40.289062+0000","last_became_active":"2024-09-17T03:52:39.013483+0000","last_became_peered":"2024-09-17T03:52:39.013483+0000","last_unstale":"2024-09-17T03:52:40.289062+0000","last_undegraded":"2024-09-17T03:52:40.289062+0000","last_fullsized":"2024-09-17T03:52:40.289062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:51.177952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009664+0000","last_change":"2024-09-17T03:52:39.014691+0000","last_active":"2024-09-17T03:52:40.009664+0000","last_peered":"2024-09-17T03:52:40.009664+0000","last_clean":"2024-09-17T03:52:40.009664+0000","last_became_active":"2024-09-17T03:52:39.010250+0000","last_became_peered":"2024-09-17T03:52:39.010250+0000","last_unstale":"2024-09-17T03:52:40.009664+0000","last_undegraded":"2024-09-17T03:52:40.009664+0000","last_fullsized":"2024-09-17T03:52:40.009664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:23.670201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290480+0000","last_change":"2024-09-17T03:52:39.030566+0000","last_active":"2024-09-17T03:52:40.290480+0000","last_peered":"2024-09-17T03:52:40.290480+0000","last_clean":"2024-09-17T03:52:40.290480+0000","last_became_active":"2024-09-17T03:52:39.029885+0000","last_became_peered":"2024-09-17T03:52:39.029885+0000","last_unstale":"2024-09-17T03:52:40.290480+0000","last_undegraded":"2024-09-17T03:52:40.290480+0000","last_fullsized":"2024-09-17T03:52:40.290480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:37.598541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288867+0000","last_change":"2024-09-17T03:52:39.027346+0000","last_active":"2024-09-17T03:52:40.288867+0000","last_peered":"2024-09-17T03:52:40.288867+0000","last_clean":"2024-09-17T03:52:40.288867+0000","last_became_active":"2024-09-17T03:52:39.025928+0000","last_became_peered":"2024-09-17T03:52:39.025928+0000","last_unstale":"2024-09-17T03:52:40.288867+0000","last_undegraded":"2024-09-17T03:52:40.288867+0000","last_fullsized":"2024-09-17T03:52:40.288867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:47.351765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007827+0000","last_change":"2024-09-17T03:52:39.018641+0000","last_active":"2024-09-17T03:52:40.007827+0000","last_peered":"2024-09-17T03:52:40.007827+0000","last_clean":"2024-09-17T03:52:40.007827+0000","last_became_active":"2024-09-17T03:52:39.018301+0000","last_became_peered":"2024-09-17T03:52:39.018301+0000","last_unstale":"2024-09-17T03:52:40.007827+0000","last_undegraded":"2024-09-17T03:52:40.007827+0000","last_fullsized":"2024-09-17T03:52:40.007827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:37.690837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287852+0000","last_change":"2024-09-17T03:52:39.022408+0000","last_active":"2024-09-17T03:52:40.287852+0000","last_peered":"2024-09-17T03:52:40.287852+0000","last_clean":"2024-09-17T03:52:40.287852+0000","last_became_active":"2024-09-17T03:52:39.021968+0000","last_became_peered":"2024-09-17T03:52:39.021968+0000","last_unstale":"2024-09-17T03:52:40.287852+0000","last_undegraded":"2024-09-17T03:52:40.287852+0000","last_fullsized":"2024-09-17T03:52:40.287852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:02.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009462+0000","last_change":"2024-09-17T03:52:39.012957+0000","last_active":"2024-09-17T03:52:40.009462+0000","last_peered":"2024-09-17T03:52:40.009462+0000","last_clean":"2024-09-17T03:52:40.009462+0000","last_became_active":"2024-09-17T03:52:39.012674+0000","last_became_peered":"2024-09-17T03:52:39.012674+0000","last_unstale":"2024-09-17T03:52:40.009462+0000","last_undegraded":"2024-09-17T03:52:40.009462+0000","last_fullsized":"2024-09-17T03:52:40.009462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.669111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287698+0000","last_change":"2024-09-17T03:52:39.007565+0000","last_active":"2024-09-17T03:52:40.287698+0000","last_peered":"2024-09-17T03:52:40.287698+0000","last_clean":"2024-09-17T03:52:40.287698+0000","last_became_active":"2024-09-17T03:52:39.007271+0000","last_became_peered":"2024-09-17T03:52:39.007271+0000","last_unstale":"2024-09-17T03:52:40.287698+0000","last_undegraded":"2024-09-17T03:52:40.287698+0000","last_fullsized":"2024-09-17T03:52:40.287698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:48:35.200854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288812+0000","last_change":"2024-09-17T03:52:39.034015+0000","last_active":"2024-09-17T03:52:40.288812+0000","last_peered":"2024-09-17T03:52:40.288812+0000","last_clean":"2024-09-17T03:52:40.288812+0000","last_became_active":"2024-09-17T03:52:39.033201+0000","last_became_peered":"2024-09-17T03:52:39.033201+0000","last_unstale":"2024-09-17T03:52:40.288812+0000","last_undegraded":"2024-09-17T03:52:40.288812+0000","last_fullsized":"2024-09-17T03:52:40.288812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:45.685964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009736+0000","last_change":"2024-09-17T03:52:39.015002+0000","last_active":"2024-09-17T03:52:40.009736+0000","last_peered":"2024-09-17T03:52:40.009736+0000","last_clean":"2024-09-17T03:52:40.009736+0000","last_became_active":"2024-09-17T03:52:39.010548+0000","last_became_peered":"2024-09-17T03:52:39.010548+0000","last_unstale":"2024-09-17T03:52:40.009736+0000","last_undegraded":"2024-09-17T03:52:40.009736+0000","last_fullsized":"2024-09-17T03:52:40.009736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:47.282513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008550+0000","last_change":"2024-09-17T03:52:39.017861+0000","last_active":"2024-09-17T03:52:40.008550+0000","last_peered":"2024-09-17T03:52:40.008550+0000","last_clean":"2024-09-17T03:52:40.008550+0000","last_became_active":"2024-09-17T03:52:39.017375+0000","last_became_peered":"2024-09-17T03:52:39.017375+0000","last_unstale":"2024-09-17T03:52:40.008550+0000","last_undegraded":"2024-09-17T03:52:40.008550+0000","last_fullsized":"2024-09-17T03:52:40.008550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:59.688466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290958+0000","last_change":"2024-09-17T03:52:39.024408+0000","last_active":"2024-09-17T03:52:40.290958+0000","last_peered":"2024-09-17T03:52:40.290958+0000","last_clean":"2024-09-17T03:52:40.290958+0000","last_became_active":"2024-09-17T03:52:39.024048+0000","last_became_peered":"2024-09-17T03:52:39.024048+0000","last_unstale":"2024-09-17T03:52:40.290958+0000","last_undegraded":"2024-09-17T03:52:40.290958+0000","last_fullsized":"2024-09-17T03:52:40.290958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:55.683703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288095+0000","last_change":"2024-09-17T03:52:39.008943+0000","last_active":"2024-09-17T03:52:40.288095+0000","last_peered":"2024-09-17T03:52:40.288095+0000","last_clean":"2024-09-17T03:52:40.288095+0000","last_became_active":"2024-09-17T03:52:39.008666+0000","last_became_peered":"2024-09-17T03:52:39.008666+0000","last_unstale":"2024-09-17T03:52:40.288095+0000","last_undegraded":"2024-09-17T03:52:40.288095+0000","last_fullsized":"2024-09-17T03:52:40.288095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:54.155207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289918+0000","last_change":"2024-09-17T03:52:39.027699+0000","last_active":"2024-09-17T03:52:40.289918+0000","last_peered":"2024-09-17T03:52:40.289918+0000","last_clean":"2024-09-17T03:52:40.289918+0000","last_became_active":"2024-09-17T03:52:39.026422+0000","last_became_peered":"2024-09-17T03:52:39.026422+0000","last_unstale":"2024-09-17T03:52:40.289918+0000","last_undegraded":"2024-09-17T03:52:40.289918+0000","last_fullsized":"2024-09-17T03:52:40.289918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:53.816696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290247+0000","last_change":"2024-09-17T03:52:39.030179+0000","last_active":"2024-09-17T03:52:40.290247+0000","last_peered":"2024-09-17T03:52:40.290247+0000","last_clean":"2024-09-17T03:52:40.290247+0000","last_became_active":"2024-09-17T03:52:39.029845+0000","last_became_peered":"2024-09-17T03:52:39.029845+0000","last_unstale":"2024-09-17T03:52:40.290247+0000","last_undegraded":"2024-09-17T03:52:40.290247+0000","last_fullsized":"2024-09-17T03:52:40.290247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:27.084828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289510+0000","last_change":"2024-09-17T03:52:39.013300+0000","last_active":"2024-09-17T03:52:40.289510+0000","last_peered":"2024-09-17T03:52:40.289510+0000","last_clean":"2024-09-17T03:52:40.289510+0000","last_became_active":"2024-09-17T03:52:39.012947+0000","last_became_peered":"2024-09-17T03:52:39.012947+0000","last_unstale":"2024-09-17T03:52:40.289510+0000","last_undegraded":"2024-09-17T03:52:40.289510+0000","last_fullsized":"2024-09-17T03:52:40.289510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:42.931837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288725+0000","last_change":"2024-09-17T03:52:39.021402+0000","last_active":"2024-09-17T03:52:40.288725+0000","last_peered":"2024-09-17T03:52:40.288725+0000","last_clean":"2024-09-17T03:52:40.288725+0000","last_became_active":"2024-09-17T03:52:39.021091+0000","last_became_peered":"2024-09-17T03:52:39.021091+0000","last_unstale":"2024-09-17T03:52:40.288725+0000","last_undegraded":"2024-09-17T03:52:40.288725+0000","last_fullsized":"2024-09-17T03:52:40.288725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:56.486110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.034509+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.033549+0000","last_became_peered":"2024-09-17T03:52:39.033549+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008078+0000","last_change":"2024-09-17T03:52:39.021427+0000","last_active":"2024-09-17T03:52:40.008078+0000","last_peered":"2024-09-17T03:52:40.008078+0000","last_clean":"2024-09-17T03:52:40.008078+0000","last_became_active":"2024-09-17T03:52:39.020901+0000","last_became_peered":"2024-09-17T03:52:39.020901+0000","last_unstale":"2024-09-17T03:52:40.008078+0000","last_undegraded":"2024-09-17T03:52:40.008078+0000","last_fullsized":"2024-09-17T03:52:40.008078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:51.900838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288900+0000","last_change":"2024-09-17T03:52:39.024384+0000","last_active":"2024-09-17T03:52:40.288900+0000","last_peered":"2024-09-17T03:52:40.288900+0000","last_clean":"2024-09-17T03:52:40.288900+0000","last_became_active":"2024-09-17T03:52:39.015719+0000","last_became_peered":"2024-09-17T03:52:39.015719+0000","last_unstale":"2024-09-17T03:52:40.288900+0000","last_undegraded":"2024-09-17T03:52:40.288900+0000","last_fullsized":"2024-09-17T03:52:40.288900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:27.198631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.031532+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.031235+0000","last_became_peered":"2024-09-17T03:52:39.031235+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:37.845837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288590+0000","last_change":"2024-09-17T03:52:39.013885+0000","last_active":"2024-09-17T03:52:40.288590+0000","last_peered":"2024-09-17T03:52:40.288590+0000","last_clean":"2024-09-17T03:52:40.288590+0000","last_became_active":"2024-09-17T03:52:39.013584+0000","last_became_peered":"2024-09-17T03:52:39.013584+0000","last_unstale":"2024-09-17T03:52:40.288590+0000","last_undegraded":"2024-09-17T03:52:40.288590+0000","last_fullsized":"2024-09-17T03:52:40.288590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:11.166875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007902+0000","last_change":"2024-09-17T03:52:39.016332+0000","last_active":"2024-09-17T03:52:40.007902+0000","last_peered":"2024-09-17T03:52:40.007902+0000","last_clean":"2024-09-17T03:52:40.007902+0000","last_became_active":"2024-09-17T03:52:39.015950+0000","last_became_peered":"2024-09-17T03:52:39.015950+0000","last_unstale":"2024-09-17T03:52:40.007902+0000","last_undegraded":"2024-09-17T03:52:40.007902+0000","last_fullsized":"2024-09-17T03:52:40.007902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008198+0000","last_change":"2024-09-17T03:52:39.026096+0000","last_active":"2024-09-17T03:52:40.008198+0000","last_peered":"2024-09-17T03:52:40.008198+0000","last_clean":"2024-09-17T03:52:40.008198+0000","last_became_active":"2024-09-17T03:52:39.025309+0000","last_became_peered":"2024-09-17T03:52:39.025309+0000","last_unstale":"2024-09-17T03:52:40.008198+0000","last_undegraded":"2024-09-17T03:52:40.008198+0000","last_fullsized":"2024-09-17T03:52:40.008198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:56:57.011981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289334+0000","last_change":"2024-09-17T03:52:39.019541+0000","last_active":"2024-09-17T03:52:40.289334+0000","last_peered":"2024-09-17T03:52:40.289334+0000","last_clean":"2024-09-17T03:52:40.289334+0000","last_became_active":"2024-09-17T03:52:39.016360+0000","last_became_peered":"2024-09-17T03:52:39.016360+0000","last_unstale":"2024-09-17T03:52:40.289334+0000","last_undegraded":"2024-09-17T03:52:40.289334+0000","last_fullsized":"2024-09-17T03:52:40.289334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:32.145164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289498+0000","last_change":"2024-09-17T03:52:39.023210+0000","last_active":"2024-09-17T03:52:40.289498+0000","last_peered":"2024-09-17T03:52:40.289498+0000","last_clean":"2024-09-17T03:52:40.289498+0000","last_became_active":"2024-09-17T03:52:39.022850+0000","last_became_peered":"2024-09-17T03:52:39.022850+0000","last_unstale":"2024-09-17T03:52:40.289498+0000","last_undegraded":"2024-09-17T03:52:40.289498+0000","last_fullsized":"2024-09-17T03:52:40.289498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:28.421892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291047+0000","last_change":"2024-09-17T03:52:38.004383+0000","last_active":"2024-09-17T03:52:40.291047+0000","last_peered":"2024-09-17T03:52:40.291047+0000","last_clean":"2024-09-17T03:52:40.291047+0000","last_became_active":"2024-09-17T03:52:38.003610+0000","last_became_peered":"2024-09-17T03:52:38.003610+0000","last_unstale":"2024-09-17T03:52:40.291047+0000","last_undegraded":"2024-09-17T03:52:40.291047+0000","last_fullsized":"2024-09-17T03:52:40.291047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:36.838884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009806+0000","last_change":"2024-09-17T03:52:39.017710+0000","last_active":"2024-09-17T03:52:40.009806+0000","last_peered":"2024-09-17T03:52:40.009806+0000","last_clean":"2024-09-17T03:52:40.009806+0000","last_became_active":"2024-09-17T03:52:39.009010+0000","last_became_peered":"2024-09-17T03:52:39.009010+0000","last_unstale":"2024-09-17T03:52:40.009806+0000","last_undegraded":"2024-09-17T03:52:40.009806+0000","last_fullsized":"2024-09-17T03:52:40.009806+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:50.092155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288354+0000","last_change":"2024-09-17T03:52:37.987588+0000","last_active":"2024-09-17T03:52:40.288354+0000","last_peered":"2024-09-17T03:52:40.288354+0000","last_clean":"2024-09-17T03:52:40.288354+0000","last_became_active":"2024-09-17T03:52:37.986620+0000","last_became_peered":"2024-09-17T03:52:37.986620+0000","last_unstale":"2024-09-17T03:52:40.288354+0000","last_undegraded":"2024-09-17T03:52:40.288354+0000","last_fullsized":"2024-09-17T03:52:40.288354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:58.890146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290564+0000","last_change":"2024-09-17T03:52:39.028972+0000","last_active":"2024-09-17T03:52:40.290564+0000","last_peered":"2024-09-17T03:52:40.290564+0000","last_clean":"2024-09-17T03:52:40.290564+0000","last_became_active":"2024-09-17T03:52:39.028419+0000","last_became_peered":"2024-09-17T03:52:39.028419+0000","last_unstale":"2024-09-17T03:52:40.290564+0000","last_undegraded":"2024-09-17T03:52:40.290564+0000","last_fullsized":"2024-09-17T03:52:40.290564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:56.889010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290134+0000","last_change":"2024-09-17T03:52:38.008448+0000","last_active":"2024-09-17T03:52:40.290134+0000","last_peered":"2024-09-17T03:52:40.290134+0000","last_clean":"2024-09-17T03:52:40.290134+0000","last_became_active":"2024-09-17T03:52:38.008134+0000","last_became_peered":"2024-09-17T03:52:38.008134+0000","last_unstale":"2024-09-17T03:52:40.290134+0000","last_undegraded":"2024-09-17T03:52:40.290134+0000","last_fullsized":"2024-09-17T03:52:40.290134+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:05.150798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289424+0000","last_change":"2024-09-17T03:52:39.009452+0000","last_active":"2024-09-17T03:52:40.289424+0000","last_peered":"2024-09-17T03:52:40.289424+0000","last_clean":"2024-09-17T03:52:40.289424+0000","last_became_active":"2024-09-17T03:52:39.005725+0000","last_became_peered":"2024-09-17T03:52:39.005725+0000","last_unstale":"2024-09-17T03:52:40.289424+0000","last_undegraded":"2024-09-17T03:52:40.289424+0000","last_fullsized":"2024-09-17T03:52:40.289424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:09.356525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290762+0000","last_change":"2024-09-17T03:52:38.004751+0000","last_active":"2024-09-17T03:52:40.290762+0000","last_peered":"2024-09-17T03:52:40.290762+0000","last_clean":"2024-09-17T03:52:40.290762+0000","last_became_active":"2024-09-17T03:52:38.004436+0000","last_became_peered":"2024-09-17T03:52:38.004436+0000","last_unstale":"2024-09-17T03:52:40.290762+0000","last_undegraded":"2024-09-17T03:52:40.290762+0000","last_fullsized":"2024-09-17T03:52:40.290762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:14.007022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008412+0000","last_change":"2024-09-17T03:52:39.023295+0000","last_active":"2024-09-17T03:52:40.008412+0000","last_peered":"2024-09-17T03:52:40.008412+0000","last_clean":"2024-09-17T03:52:40.008412+0000","last_became_active":"2024-09-17T03:52:39.022209+0000","last_became_peered":"2024-09-17T03:52:39.022209+0000","last_unstale":"2024-09-17T03:52:40.008412+0000","last_undegraded":"2024-09-17T03:52:40.008412+0000","last_fullsized":"2024-09-17T03:52:40.008412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:02.778195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288785+0000","last_change":"2024-09-17T03:52:38.004706+0000","last_active":"2024-09-17T03:52:40.288785+0000","last_peered":"2024-09-17T03:52:40.288785+0000","last_clean":"2024-09-17T03:52:40.288785+0000","last_became_active":"2024-09-17T03:52:38.004384+0000","last_became_peered":"2024-09-17T03:52:38.004384+0000","last_unstale":"2024-09-17T03:52:40.288785+0000","last_undegraded":"2024-09-17T03:52:40.288785+0000","last_fullsized":"2024-09-17T03:52:40.288785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:24.947156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290252+0000","last_change":"2024-09-17T03:52:39.033871+0000","last_active":"2024-09-17T03:52:40.290252+0000","last_peered":"2024-09-17T03:52:40.290252+0000","last_clean":"2024-09-17T03:52:40.290252+0000","last_became_active":"2024-09-17T03:52:39.033350+0000","last_became_peered":"2024-09-17T03:52:39.033350+0000","last_unstale":"2024-09-17T03:52:40.290252+0000","last_undegraded":"2024-09-17T03:52:40.290252+0000","last_fullsized":"2024-09-17T03:52:40.290252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290522+0000","last_change":"2024-09-17T03:52:38.007787+0000","last_active":"2024-09-17T03:52:40.290522+0000","last_peered":"2024-09-17T03:52:40.290522+0000","last_clean":"2024-09-17T03:52:40.290522+0000","last_became_active":"2024-09-17T03:52:38.007287+0000","last_became_peered":"2024-09-17T03:52:38.007287+0000","last_unstale":"2024-09-17T03:52:40.290522+0000","last_undegraded":"2024-09-17T03:52:40.290522+0000","last_fullsized":"2024-09-17T03:52:40.290522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:37.387663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007768+0000","last_change":"2024-09-17T03:52:39.026791+0000","last_active":"2024-09-17T03:52:40.007768+0000","last_peered":"2024-09-17T03:52:40.007768+0000","last_clean":"2024-09-17T03:52:40.007768+0000","last_became_active":"2024-09-17T03:52:39.026369+0000","last_became_peered":"2024-09-17T03:52:39.026369+0000","last_unstale":"2024-09-17T03:52:40.007768+0000","last_undegraded":"2024-09-17T03:52:40.007768+0000","last_fullsized":"2024-09-17T03:52:40.007768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:11.999239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007539+0000","last_change":"2024-09-17T03:52:38.004637+0000","last_active":"2024-09-17T03:52:40.007539+0000","last_peered":"2024-09-17T03:52:40.007539+0000","last_clean":"2024-09-17T03:52:40.007539+0000","last_became_active":"2024-09-17T03:52:38.004076+0000","last_became_peered":"2024-09-17T03:52:38.004076+0000","last_unstale":"2024-09-17T03:52:40.007539+0000","last_undegraded":"2024-09-17T03:52:40.007539+0000","last_fullsized":"2024-09-17T03:52:40.007539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:38.214053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287689+0000","last_change":"2024-09-17T03:52:39.000484+0000","last_active":"2024-09-17T03:52:40.287689+0000","last_peered":"2024-09-17T03:52:40.287689+0000","last_clean":"2024-09-17T03:52:40.287689+0000","last_became_active":"2024-09-17T03:52:39.000154+0000","last_became_peered":"2024-09-17T03:52:39.000154+0000","last_unstale":"2024-09-17T03:52:40.287689+0000","last_undegraded":"2024-09-17T03:52:40.287689+0000","last_fullsized":"2024-09-17T03:52:40.287689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008024+0000","last_change":"2024-09-17T03:52:37.992784+0000","last_active":"2024-09-17T03:52:40.008024+0000","last_peered":"2024-09-17T03:52:40.008024+0000","last_clean":"2024-09-17T03:52:40.008024+0000","last_became_active":"2024-09-17T03:52:37.992480+0000","last_became_peered":"2024-09-17T03:52:37.992480+0000","last_unstale":"2024-09-17T03:52:40.008024+0000","last_undegraded":"2024-09-17T03:52:40.008024+0000","last_fullsized":"2024-09-17T03:52:40.008024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:21.250972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008452+0000","last_change":"2024-09-17T03:52:39.008286+0000","last_active":"2024-09-17T03:52:40.008452+0000","last_peered":"2024-09-17T03:52:40.008452+0000","last_clean":"2024-09-17T03:52:40.008452+0000","last_became_active":"2024-09-17T03:52:39.007952+0000","last_became_peered":"2024-09-17T03:52:39.007952+0000","last_unstale":"2024-09-17T03:52:40.008452+0000","last_undegraded":"2024-09-17T03:52:40.008452+0000","last_fullsized":"2024-09-17T03:52:40.008452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:30.540116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290933+0000","last_change":"2024-09-17T03:52:38.001450+0000","last_active":"2024-09-17T03:52:40.290933+0000","last_peered":"2024-09-17T03:52:40.290933+0000","last_clean":"2024-09-17T03:52:40.290933+0000","last_became_active":"2024-09-17T03:52:38.001124+0000","last_became_peered":"2024-09-17T03:52:38.001124+0000","last_unstale":"2024-09-17T03:52:40.290933+0000","last_undegraded":"2024-09-17T03:52:40.290933+0000","last_fullsized":"2024-09-17T03:52:40.290933+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:53.092676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289507+0000","last_change":"2024-09-17T03:52:39.027159+0000","last_active":"2024-09-17T03:52:40.289507+0000","last_peered":"2024-09-17T03:52:40.289507+0000","last_clean":"2024-09-17T03:52:40.289507+0000","last_became_active":"2024-09-17T03:52:39.026074+0000","last_became_peered":"2024-09-17T03:52:39.026074+0000","last_unstale":"2024-09-17T03:52:40.289507+0000","last_undegraded":"2024-09-17T03:52:40.289507+0000","last_fullsized":"2024-09-17T03:52:40.289507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:28.966676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291290+0000","last_change":"2024-09-17T03:52:38.004849+0000","last_active":"2024-09-17T03:52:40.291290+0000","last_peered":"2024-09-17T03:52:40.291290+0000","last_clean":"2024-09-17T03:52:40.291290+0000","last_became_active":"2024-09-17T03:52:38.002393+0000","last_became_peered":"2024-09-17T03:52:38.002393+0000","last_unstale":"2024-09-17T03:52:40.291290+0000","last_undegraded":"2024-09-17T03:52:40.291290+0000","last_fullsized":"2024-09-17T03:52:40.291290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:06.095763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290594+0000","last_change":"2024-09-17T03:52:39.024541+0000","last_active":"2024-09-17T03:52:40.290594+0000","last_peered":"2024-09-17T03:52:40.290594+0000","last_clean":"2024-09-17T03:52:40.290594+0000","last_became_active":"2024-09-17T03:52:39.023962+0000","last_became_peered":"2024-09-17T03:52:39.023962+0000","last_unstale":"2024-09-17T03:52:40.290594+0000","last_undegraded":"2024-09-17T03:52:40.290594+0000","last_fullsized":"2024-09-17T03:52:40.290594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:05.325707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289078+0000","last_change":"2024-09-17T03:52:38.002263+0000","last_active":"2024-09-17T03:52:40.289078+0000","last_peered":"2024-09-17T03:52:40.289078+0000","last_clean":"2024-09-17T03:52:40.289078+0000","last_became_active":"2024-09-17T03:52:38.001712+0000","last_became_peered":"2024-09-17T03:52:38.001712+0000","last_unstale":"2024-09-17T03:52:40.289078+0000","last_undegraded":"2024-09-17T03:52:40.289078+0000","last_fullsized":"2024-09-17T03:52:40.289078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:53.008257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008530+0000","last_change":"2024-09-17T03:52:39.015946+0000","last_active":"2024-09-17T03:52:40.008530+0000","last_peered":"2024-09-17T03:52:40.008530+0000","last_clean":"2024-09-17T03:52:40.008530+0000","last_became_active":"2024-09-17T03:52:39.008987+0000","last_became_peered":"2024-09-17T03:52:39.008987+0000","last_unstale":"2024-09-17T03:52:40.008530+0000","last_undegraded":"2024-09-17T03:52:40.008530+0000","last_fullsized":"2024-09-17T03:52:40.008530+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:24.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008445+0000","last_change":"2024-09-17T03:52:38.000155+0000","last_active":"2024-09-17T03:52:40.008445+0000","last_peered":"2024-09-17T03:52:40.008445+0000","last_clean":"2024-09-17T03:52:40.008445+0000","last_became_active":"2024-09-17T03:52:37.999044+0000","last_became_peered":"2024-09-17T03:52:37.999044+0000","last_unstale":"2024-09-17T03:52:40.008445+0000","last_undegraded":"2024-09-17T03:52:40.008445+0000","last_fullsized":"2024-09-17T03:52:40.008445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:15.512221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289210+0000","last_change":"2024-09-17T03:52:39.015628+0000","last_active":"2024-09-17T03:52:40.289210+0000","last_peered":"2024-09-17T03:52:40.289210+0000","last_clean":"2024-09-17T03:52:40.289210+0000","last_became_active":"2024-09-17T03:52:39.015286+0000","last_became_peered":"2024-09-17T03:52:39.015286+0000","last_unstale":"2024-09-17T03:52:40.289210+0000","last_undegraded":"2024-09-17T03:52:40.289210+0000","last_fullsized":"2024-09-17T03:52:40.289210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:02.891865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007787+0000","last_change":"2024-09-17T03:52:38.001510+0000","last_active":"2024-09-17T03:52:40.007787+0000","last_peered":"2024-09-17T03:52:40.007787+0000","last_clean":"2024-09-17T03:52:40.007787+0000","last_became_active":"2024-09-17T03:52:38.001109+0000","last_became_peered":"2024-09-17T03:52:38.001109+0000","last_unstale":"2024-09-17T03:52:40.007787+0000","last_undegraded":"2024-09-17T03:52:40.007787+0000","last_fullsized":"2024-09-17T03:52:40.007787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:53.377063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008218+0000","last_change":"2024-09-17T03:52:39.021149+0000","last_active":"2024-09-17T03:52:40.008218+0000","last_peered":"2024-09-17T03:52:40.008218+0000","last_clean":"2024-09-17T03:52:40.008218+0000","last_became_active":"2024-09-17T03:52:39.020787+0000","last_became_peered":"2024-09-17T03:52:39.020787+0000","last_unstale":"2024-09-17T03:52:40.008218+0000","last_undegraded":"2024-09-17T03:52:40.008218+0000","last_fullsized":"2024-09-17T03:52:40.008218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:49.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290649+0000","last_change":"2024-09-17T03:52:38.000222+0000","last_active":"2024-09-17T03:52:40.290649+0000","last_peered":"2024-09-17T03:52:40.290649+0000","last_clean":"2024-09-17T03:52:40.290649+0000","last_became_active":"2024-09-17T03:52:37.998870+0000","last_became_peered":"2024-09-17T03:52:37.998870+0000","last_unstale":"2024-09-17T03:52:40.290649+0000","last_undegraded":"2024-09-17T03:52:40.290649+0000","last_fullsized":"2024-09-17T03:52:40.290649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:01.261692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287904+0000","last_change":"2024-09-17T03:52:39.015246+0000","last_active":"2024-09-17T03:52:40.287904+0000","last_peered":"2024-09-17T03:52:40.287904+0000","last_clean":"2024-09-17T03:52:40.287904+0000","last_became_active":"2024-09-17T03:52:39.014938+0000","last_became_peered":"2024-09-17T03:52:39.014938+0000","last_unstale":"2024-09-17T03:52:40.287904+0000","last_undegraded":"2024-09-17T03:52:40.287904+0000","last_fullsized":"2024-09-17T03:52:40.287904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:12.786786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007815+0000","last_change":"2024-09-17T03:52:37.993426+0000","last_active":"2024-09-17T03:52:40.007815+0000","last_peered":"2024-09-17T03:52:40.007815+0000","last_clean":"2024-09-17T03:52:40.007815+0000","last_became_active":"2024-09-17T03:52:37.992558+0000","last_became_peered":"2024-09-17T03:52:37.992558+0000","last_unstale":"2024-09-17T03:52:40.007815+0000","last_undegraded":"2024-09-17T03:52:40.007815+0000","last_fullsized":"2024-09-17T03:52:40.007815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:50.687889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009000+0000","last_change":"2024-09-17T03:52:39.020708+0000","last_active":"2024-09-17T03:52:40.009000+0000","last_peered":"2024-09-17T03:52:40.009000+0000","last_clean":"2024-09-17T03:52:40.009000+0000","last_became_active":"2024-09-17T03:52:39.019989+0000","last_became_peered":"2024-09-17T03:52:39.019989+0000","last_unstale":"2024-09-17T03:52:40.009000+0000","last_undegraded":"2024-09-17T03:52:40.009000+0000","last_fullsized":"2024-09-17T03:52:40.009000+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:26.316176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289749+0000","last_change":"2024-09-17T03:52:38.000317+0000","last_active":"2024-09-17T03:52:40.289749+0000","last_peered":"2024-09-17T03:52:40.289749+0000","last_clean":"2024-09-17T03:52:40.289749+0000","last_became_active":"2024-09-17T03:52:37.999933+0000","last_became_peered":"2024-09-17T03:52:37.999933+0000","last_unstale":"2024-09-17T03:52:40.289749+0000","last_undegraded":"2024-09-17T03:52:40.289749+0000","last_fullsized":"2024-09-17T03:52:40.289749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:56.251630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290593+0000","last_change":"2024-09-17T03:52:39.025015+0000","last_active":"2024-09-17T03:52:40.290593+0000","last_peered":"2024-09-17T03:52:40.290593+0000","last_clean":"2024-09-17T03:52:40.290593+0000","last_became_active":"2024-09-17T03:52:39.024596+0000","last_became_peered":"2024-09-17T03:52:39.024596+0000","last_unstale":"2024-09-17T03:52:40.290593+0000","last_undegraded":"2024-09-17T03:52:40.290593+0000","last_fullsized":"2024-09-17T03:52:40.290593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291013+0000","last_change":"2024-09-17T03:52:38.007757+0000","last_active":"2024-09-17T03:52:40.291013+0000","last_peered":"2024-09-17T03:52:40.291013+0000","last_clean":"2024-09-17T03:52:40.291013+0000","last_became_active":"2024-09-17T03:52:38.007282+0000","last_became_peered":"2024-09-17T03:52:38.007282+0000","last_unstale":"2024-09-17T03:52:40.291013+0000","last_undegraded":"2024-09-17T03:52:40.291013+0000","last_fullsized":"2024-09-17T03:52:40.291013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.565057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289931+0000","last_change":"2024-09-17T03:52:39.007897+0000","last_active":"2024-09-17T03:52:40.289931+0000","last_peered":"2024-09-17T03:52:40.289931+0000","last_clean":"2024-09-17T03:52:40.289931+0000","last_became_active":"2024-09-17T03:52:39.007557+0000","last_became_peered":"2024-09-17T03:52:39.007557+0000","last_unstale":"2024-09-17T03:52:40.289931+0000","last_undegraded":"2024-09-17T03:52:40.289931+0000","last_fullsized":"2024-09-17T03:52:40.289931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:22.330075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290700+0000","last_change":"2024-09-17T03:52:38.002206+0000","last_active":"2024-09-17T03:52:40.290700+0000","last_peered":"2024-09-17T03:52:40.290700+0000","last_clean":"2024-09-17T03:52:40.290700+0000","last_became_active":"2024-09-17T03:52:38.001872+0000","last_became_peered":"2024-09-17T03:52:38.001872+0000","last_unstale":"2024-09-17T03:52:40.290700+0000","last_undegraded":"2024-09-17T03:52:40.290700+0000","last_fullsized":"2024-09-17T03:52:40.290700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:18.787765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008257+0000","last_change":"2024-09-17T03:52:39.018968+0000","last_active":"2024-09-17T03:52:40.008257+0000","last_peered":"2024-09-17T03:52:40.008257+0000","last_clean":"2024-09-17T03:52:40.008257+0000","last_became_active":"2024-09-17T03:52:39.018682+0000","last_became_peered":"2024-09-17T03:52:39.018682+0000","last_unstale":"2024-09-17T03:52:40.008257+0000","last_undegraded":"2024-09-17T03:52:40.008257+0000","last_fullsized":"2024-09-17T03:52:40.008257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:06.157752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290986+0000","last_change":"2024-09-17T03:52:38.009348+0000","last_active":"2024-09-17T03:52:40.290986+0000","last_peered":"2024-09-17T03:52:40.290986+0000","last_clean":"2024-09-17T03:52:40.290986+0000","last_became_active":"2024-09-17T03:52:38.008926+0000","last_became_peered":"2024-09-17T03:52:38.008926+0000","last_unstale":"2024-09-17T03:52:40.290986+0000","last_undegraded":"2024-09-17T03:52:40.290986+0000","last_fullsized":"2024-09-17T03:52:40.290986+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:27.731425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008745+0000","last_change":"2024-09-17T03:52:39.021118+0000","last_active":"2024-09-17T03:52:40.008745+0000","last_peered":"2024-09-17T03:52:40.008745+0000","last_clean":"2024-09-17T03:52:40.008745+0000","last_became_active":"2024-09-17T03:52:39.020763+0000","last_became_peered":"2024-09-17T03:52:39.020763+0000","last_unstale":"2024-09-17T03:52:40.008745+0000","last_undegraded":"2024-09-17T03:52:40.008745+0000","last_fullsized":"2024-09-17T03:52:40.008745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:10.895566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289021+0000","last_change":"2024-09-17T03:52:38.005886+0000","last_active":"2024-09-17T03:52:40.289021+0000","last_peered":"2024-09-17T03:52:40.289021+0000","last_clean":"2024-09-17T03:52:40.289021+0000","last_became_active":"2024-09-17T03:52:38.005550+0000","last_became_peered":"2024-09-17T03:52:38.005550+0000","last_unstale":"2024-09-17T03:52:40.289021+0000","last_undegraded":"2024-09-17T03:52:40.289021+0000","last_fullsized":"2024-09-17T03:52:40.289021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:36.591587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288662+0000","last_change":"2024-09-17T03:52:39.009574+0000","last_active":"2024-09-17T03:52:40.288662+0000","last_peered":"2024-09-17T03:52:40.288662+0000","last_clean":"2024-09-17T03:52:40.288662+0000","last_became_active":"2024-09-17T03:52:39.009382+0000","last_became_peered":"2024-09-17T03:52:39.009382+0000","last_unstale":"2024-09-17T03:52:40.288662+0000","last_undegraded":"2024-09-17T03:52:40.288662+0000","last_fullsized":"2024-09-17T03:52:40.288662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:28.472361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290771+0000","last_change":"2024-09-17T03:52:38.006146+0000","last_active":"2024-09-17T03:52:40.290771+0000","last_peered":"2024-09-17T03:52:40.290771+0000","last_clean":"2024-09-17T03:52:40.290771+0000","last_became_active":"2024-09-17T03:52:38.005099+0000","last_became_peered":"2024-09-17T03:52:38.005099+0000","last_unstale":"2024-09-17T03:52:40.290771+0000","last_undegraded":"2024-09-17T03:52:40.290771+0000","last_fullsized":"2024-09-17T03:52:40.290771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:14.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288305+0000","last_change":"2024-09-17T03:52:39.020395+0000","last_active":"2024-09-17T03:52:40.288305+0000","last_peered":"2024-09-17T03:52:40.288305+0000","last_clean":"2024-09-17T03:52:40.288305+0000","last_became_active":"2024-09-17T03:52:39.019294+0000","last_became_peered":"2024-09-17T03:52:39.019294+0000","last_unstale":"2024-09-17T03:52:40.288305+0000","last_undegraded":"2024-09-17T03:52:40.288305+0000","last_fullsized":"2024-09-17T03:52:40.288305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:11.716349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008291+0000","last_change":"2024-09-17T03:52:37.986744+0000","last_active":"2024-09-17T03:52:40.008291+0000","last_peered":"2024-09-17T03:52:40.008291+0000","last_clean":"2024-09-17T03:52:40.008291+0000","last_became_active":"2024-09-17T03:52:37.986385+0000","last_became_peered":"2024-09-17T03:52:37.986385+0000","last_unstale":"2024-09-17T03:52:40.008291+0000","last_undegraded":"2024-09-17T03:52:40.008291+0000","last_fullsized":"2024-09-17T03:52:40.008291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.274408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288402+0000","last_change":"2024-09-17T03:52:39.019842+0000","last_active":"2024-09-17T03:52:40.288402+0000","last_peered":"2024-09-17T03:52:40.288402+0000","last_clean":"2024-09-17T03:52:40.288402+0000","last_became_active":"2024-09-17T03:52:39.019548+0000","last_became_peered":"2024-09-17T03:52:39.019548+0000","last_unstale":"2024-09-17T03:52:40.288402+0000","last_undegraded":"2024-09-17T03:52:40.288402+0000","last_fullsized":"2024-09-17T03:52:40.288402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:21.225549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287600+0000","last_change":"2024-09-17T03:52:37.987589+0000","last_active":"2024-09-17T03:52:40.287600+0000","last_peered":"2024-09-17T03:52:40.287600+0000","last_clean":"2024-09-17T03:52:40.287600+0000","last_became_active":"2024-09-17T03:52:37.986646+0000","last_became_peered":"2024-09-17T03:52:37.986646+0000","last_unstale":"2024-09-17T03:52:40.287600+0000","last_undegraded":"2024-09-17T03:52:40.287600+0000","last_fullsized":"2024-09-17T03:52:40.287600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:04.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288208+0000","last_change":"2024-09-17T03:52:39.017123+0000","last_active":"2024-09-17T03:52:40.288208+0000","last_peered":"2024-09-17T03:52:40.288208+0000","last_clean":"2024-09-17T03:52:40.288208+0000","last_became_active":"2024-09-17T03:52:39.016654+0000","last_became_peered":"2024-09-17T03:52:39.016654+0000","last_unstale":"2024-09-17T03:52:40.288208+0000","last_undegraded":"2024-09-17T03:52:40.288208+0000","last_fullsized":"2024-09-17T03:52:40.288208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:41.588250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291161+0000","last_change":"2024-09-17T03:52:38.007301+0000","last_active":"2024-09-17T03:52:40.291161+0000","last_peered":"2024-09-17T03:52:40.291161+0000","last_clean":"2024-09-17T03:52:40.291161+0000","last_became_active":"2024-09-17T03:52:38.006871+0000","last_became_peered":"2024-09-17T03:52:38.006871+0000","last_unstale":"2024-09-17T03:52:40.291161+0000","last_undegraded":"2024-09-17T03:52:40.291161+0000","last_fullsized":"2024-09-17T03:52:40.291161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:40.821800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008418+0000","last_change":"2024-09-17T03:52:39.015261+0000","last_active":"2024-09-17T03:52:40.008418+0000","last_peered":"2024-09-17T03:52:40.008418+0000","last_clean":"2024-09-17T03:52:40.008418+0000","last_became_active":"2024-09-17T03:52:39.014907+0000","last_became_peered":"2024-09-17T03:52:39.014907+0000","last_unstale":"2024-09-17T03:52:40.008418+0000","last_undegraded":"2024-09-17T03:52:40.008418+0000","last_fullsized":"2024-09-17T03:52:40.008418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:25.145657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008635+0000","last_change":"2024-09-17T03:52:37.993729+0000","last_active":"2024-09-17T03:52:40.008635+0000","last_peered":"2024-09-17T03:52:40.008635+0000","last_clean":"2024-09-17T03:52:40.008635+0000","last_became_active":"2024-09-17T03:52:37.991713+0000","last_became_peered":"2024-09-17T03:52:37.991713+0000","last_unstale":"2024-09-17T03:52:40.008635+0000","last_undegraded":"2024-09-17T03:52:40.008635+0000","last_fullsized":"2024-09-17T03:52:40.008635+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:14.304806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288837+0000","last_change":"2024-09-17T03:52:39.024919+0000","last_active":"2024-09-17T03:52:40.288837+0000","last_peered":"2024-09-17T03:52:40.288837+0000","last_clean":"2024-09-17T03:52:40.288837+0000","last_became_active":"2024-09-17T03:52:39.022961+0000","last_became_peered":"2024-09-17T03:52:39.022961+0000","last_unstale":"2024-09-17T03:52:40.288837+0000","last_undegraded":"2024-09-17T03:52:40.288837+0000","last_fullsized":"2024-09-17T03:52:40.288837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:47.897977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008905+0000","last_change":"2024-09-17T03:52:38.002436+0000","last_active":"2024-09-17T03:52:40.008905+0000","last_peered":"2024-09-17T03:52:40.008905+0000","last_clean":"2024-09-17T03:52:40.008905+0000","last_became_active":"2024-09-17T03:52:38.001380+0000","last_became_peered":"2024-09-17T03:52:38.001380+0000","last_unstale":"2024-09-17T03:52:40.008905+0000","last_undegraded":"2024-09-17T03:52:40.008905+0000","last_fullsized":"2024-09-17T03:52:40.008905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:16.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289539+0000","last_change":"2024-09-17T03:52:39.013637+0000","last_active":"2024-09-17T03:52:40.289539+0000","last_peered":"2024-09-17T03:52:40.289539+0000","last_clean":"2024-09-17T03:52:40.289539+0000","last_became_active":"2024-09-17T03:52:39.013035+0000","last_became_peered":"2024-09-17T03:52:39.013035+0000","last_unstale":"2024-09-17T03:52:40.289539+0000","last_undegraded":"2024-09-17T03:52:40.289539+0000","last_fullsized":"2024-09-17T03:52:40.289539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:39.874354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287877+0000","last_change":"2024-09-17T03:52:37.998007+0000","last_active":"2024-09-17T03:52:40.287877+0000","last_peered":"2024-09-17T03:52:40.287877+0000","last_clean":"2024-09-17T03:52:40.287877+0000","last_became_active":"2024-09-17T03:52:37.997670+0000","last_became_peered":"2024-09-17T03:52:37.997670+0000","last_unstale":"2024-09-17T03:52:40.287877+0000","last_undegraded":"2024-09-17T03:52:40.287877+0000","last_fullsized":"2024-09-17T03:52:40.287877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.236372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009681+0000","last_change":"2024-09-17T03:52:39.016581+0000","last_active":"2024-09-17T03:52:40.009681+0000","last_peered":"2024-09-17T03:52:40.009681+0000","last_clean":"2024-09-17T03:52:40.009681+0000","last_became_active":"2024-09-17T03:52:39.007886+0000","last_became_peered":"2024-09-17T03:52:39.007886+0000","last_unstale":"2024-09-17T03:52:40.009681+0000","last_undegraded":"2024-09-17T03:52:40.009681+0000","last_fullsized":"2024-09-17T03:52:40.009681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:13.021250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008230+0000","last_change":"2024-09-17T03:52:37.991052+0000","last_active":"2024-09-17T03:52:40.008230+0000","last_peered":"2024-09-17T03:52:40.008230+0000","last_clean":"2024-09-17T03:52:40.008230+0000","last_became_active":"2024-09-17T03:52:37.990089+0000","last_became_peered":"2024-09-17T03:52:37.990089+0000","last_unstale":"2024-09-17T03:52:40.008230+0000","last_undegraded":"2024-09-17T03:52:40.008230+0000","last_fullsized":"2024-09-17T03:52:40.008230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:17.450063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290527+0000","last_change":"2024-09-17T03:52:39.034798+0000","last_active":"2024-09-17T03:52:40.290527+0000","last_peered":"2024-09-17T03:52:40.290527+0000","last_clean":"2024-09-17T03:52:40.290527+0000","last_became_active":"2024-09-17T03:52:39.033757+0000","last_became_peered":"2024-09-17T03:52:39.033757+0000","last_unstale":"2024-09-17T03:52:40.290527+0000","last_undegraded":"2024-09-17T03:52:40.290527+0000","last_fullsized":"2024-09-17T03:52:40.290527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:40.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007401+0000","last_change":"2024-09-17T03:52:38.006002+0000","last_active":"2024-09-17T03:52:40.007401+0000","last_peered":"2024-09-17T03:52:40.007401+0000","last_clean":"2024-09-17T03:52:40.007401+0000","last_became_active":"2024-09-17T03:52:38.004725+0000","last_became_peered":"2024-09-17T03:52:38.004725+0000","last_unstale":"2024-09-17T03:52:40.007401+0000","last_undegraded":"2024-09-17T03:52:40.007401+0000","last_fullsized":"2024-09-17T03:52:40.007401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:05.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008222+0000","last_change":"2024-09-17T03:52:39.010421+0000","last_active":"2024-09-17T03:52:40.008222+0000","last_peered":"2024-09-17T03:52:40.008222+0000","last_clean":"2024-09-17T03:52:40.008222+0000","last_became_active":"2024-09-17T03:52:39.010077+0000","last_became_peered":"2024-09-17T03:52:39.010077+0000","last_unstale":"2024-09-17T03:52:40.008222+0000","last_undegraded":"2024-09-17T03:52:40.008222+0000","last_fullsized":"2024-09-17T03:52:40.008222+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:57.258583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289687+0000","last_change":"2024-09-17T03:52:37.998870+0000","last_active":"2024-09-17T03:52:40.289687+0000","last_peered":"2024-09-17T03:52:40.289687+0000","last_clean":"2024-09-17T03:52:40.289687+0000","last_became_active":"2024-09-17T03:52:37.998561+0000","last_became_peered":"2024-09-17T03:52:37.998561+0000","last_unstale":"2024-09-17T03:52:40.289687+0000","last_undegraded":"2024-09-17T03:52:40.289687+0000","last_fullsized":"2024-09-17T03:52:40.289687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:58.681513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290032+0000","last_change":"2024-09-17T03:52:39.022554+0000","last_active":"2024-09-17T03:52:40.290032+0000","last_peered":"2024-09-17T03:52:40.290032+0000","last_clean":"2024-09-17T03:52:40.290032+0000","last_became_active":"2024-09-17T03:52:39.022101+0000","last_became_peered":"2024-09-17T03:52:39.022101+0000","last_unstale":"2024-09-17T03:52:40.290032+0000","last_undegraded":"2024-09-17T03:52:40.290032+0000","last_fullsized":"2024-09-17T03:52:40.290032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:37.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289503+0000","last_change":"2024-09-17T03:52:38.004100+0000","last_active":"2024-09-17T03:52:40.289503+0000","last_peered":"2024-09-17T03:52:40.289503+0000","last_clean":"2024-09-17T03:52:40.289503+0000","last_became_active":"2024-09-17T03:52:38.000956+0000","last_became_peered":"2024-09-17T03:52:38.000956+0000","last_unstale":"2024-09-17T03:52:40.289503+0000","last_undegraded":"2024-09-17T03:52:40.289503+0000","last_fullsized":"2024-09-17T03:52:40.289503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:29.352497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288572+0000","last_change":"2024-09-17T03:52:39.013661+0000","last_active":"2024-09-17T03:52:40.288572+0000","last_peered":"2024-09-17T03:52:40.288572+0000","last_clean":"2024-09-17T03:52:40.288572+0000","last_became_active":"2024-09-17T03:52:39.013380+0000","last_became_peered":"2024-09-17T03:52:39.013380+0000","last_unstale":"2024-09-17T03:52:40.288572+0000","last_undegraded":"2024-09-17T03:52:40.288572+0000","last_fullsized":"2024-09-17T03:52:40.288572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:41.998544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290721+0000","last_change":"2024-09-17T03:52:38.001315+0000","last_active":"2024-09-17T03:52:40.290721+0000","last_peered":"2024-09-17T03:52:40.290721+0000","last_clean":"2024-09-17T03:52:40.290721+0000","last_became_active":"2024-09-17T03:52:37.999404+0000","last_became_peered":"2024-09-17T03:52:37.999404+0000","last_unstale":"2024-09-17T03:52:40.290721+0000","last_undegraded":"2024-09-17T03:52:40.290721+0000","last_fullsized":"2024-09-17T03:52:40.290721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:34:25.654563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288936+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:52:40.288936+0000","last_peered":"2024-09-17T03:52:40.288936+0000","last_clean":"2024-09-17T03:52:40.288936+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:52:40.288936+0000","last_undegraded":"2024-09-17T03:52:40.288936+0000","last_fullsized":"2024-09-17T03:52:40.288936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1421312,"data_stored":931564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509485,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":827392,"data_stored":340692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":831488,"data_stored":342192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":839680,"data_stored":342808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":340672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1413120,"data_stored":930948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:52:46.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:52:46.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:52:48.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 -- 172.21.15.129:0/4030891926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80cad30 msgr2=0x7f25e806ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 --2- 172.21.15.129:0/4030891926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80cad30 0x7f25e806ffa0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f25d4005fd0 tx=0x7f25d402a530 comp rx=0 tx=0).stop 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 -- 172.21.15.129:0/4030891926 shutdown_connections 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 --2- 172.21.15.129:0/4030891926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80cad30 0x7f25e806ffa0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 --2- 172.21.15.129:0/4030891926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80ca800 0x7f25e80cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 -- 172.21.15.129:0/4030891926 >> 172.21.15.129:0/4030891926 conn(0x7f25e80b1cb0 msgr2=0x7f25e80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 -- 172.21.15.129:0/4030891926 shutdown_connections 2024-09-17T03:52:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.628+0000 7f25ef600640 1 -- 172.21.15.129:0/4030891926 wait complete. 2024-09-17T03:52:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.631+0000 7f25ef600640 1 Processor -- start 2024-09-17T03:52:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.631+0000 7f25ef600640 1 -- start start 2024-09-17T03:52:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25e80704e0 0x7f25e81f85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80cad30 0x7f25e815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25e80ccf40 con 0x7f25e80ca800 2024-09-17T03:52:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25e80747d0 con 0x7f25e80704e0 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.632+0000 7f25ef600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f25e80cd0c0 con 0x7f25e80cad30 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37018/0 (socket says 172.21.15.129:37018) 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 -- 172.21.15.129:0/3030795575 learned_addr learned my addr 172.21.15.129:0/3030795575 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25ecc00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25e80704e0 0x7f25e81f85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25ed600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80cad30 0x7f25e815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 -- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80cad30 msgr2=0x7f25e815e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80cad30 0x7f25e815e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 -- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25e80704e0 msgr2=0x7f25e81f85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25e80704e0 0x7f25e81f85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 -- 172.21.15.129:0/3030795575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e80ca440 con 0x7f25e80ca800 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.633+0000 7f25e7e00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006010 tx=0x7f25d40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.634+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d403e030 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.634+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25d40047a0 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.634+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e80c8190 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.634+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4035040 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.634+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e81f5ca0 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.635+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25e80add60 con 0x7f25e80ca800 2024-09-17T03:52:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.636+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f25d4041030 con 0x7f25e80ca800 2024-09-17T03:52:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.637+0000 7f25e4c00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 0x7f25c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.637+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f25d40bb5f0 con 0x7f25e80ca800 2024-09-17T03:52:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.638+0000 7f25ecc00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 0x7f25c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.638+0000 7f25ecc00640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 0x7f25c807b0a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25e80cb250 tx=0x7f25dc00a2d0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.642+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f25e80add60 con 0x7f25e80ca800 2024-09-17T03:52:48.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.821+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f25e80cd650 con 0x7f25c8078b90 2024-09-17T03:52:48.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.826+0000 7f25e4c00640 1 -- 172.21.15.129:0/3030795575 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252018 (secure 0 0 0) 0x7f25e80cd650 con 0x7f25c8078b90 2024-09-17T03:52:48.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:48.829 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:52:48.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.829+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 msgr2=0x7f25c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.829+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 0x7f25c807b0a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25e80cb250 tx=0x7f25dc00a2d0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.829+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 msgr2=0x7f25e80c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.829+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f25d4006010 tx=0x7f25d40039e0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 shutdown_connections 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25c8078b90 0x7f25c807b0a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25e80cad30 0x7f25e815e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25e80ca800 0x7f25e80c7870 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 --2- 172.21.15.129:0/3030795575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25e80704e0 0x7f25e81f85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 >> 172.21.15.129:0/3030795575 conn(0x7f25e80b1cb0 msgr2=0x7f25e80ad650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 shutdown_connections 2024-09-17T03:52:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:48.830+0000 7f25ef600640 1 -- 172.21.15.129:0/3030795575 wait complete. 2024-09-17T03:52:48.851 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-17T03:52:48.344936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":257164,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754717556,"statfs":{"total":773094113280,"available":772830777344,"internally_reserved":0,"allocated":7806976,"data_stored":3910196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.352572"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289679+0000","last_change":"2024-09-17T03:52:39.012533+0000","last_active":"2024-09-17T03:52:40.289679+0000","last_peered":"2024-09-17T03:52:40.289679+0000","last_clean":"2024-09-17T03:52:40.289679+0000","last_became_active":"2024-09-17T03:52:39.012181+0000","last_became_peered":"2024-09-17T03:52:39.012181+0000","last_unstale":"2024-09-17T03:52:40.289679+0000","last_undegraded":"2024-09-17T03:52:40.289679+0000","last_fullsized":"2024-09-17T03:52:40.289679+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:54:06.915985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291299+0000","last_change":"2024-09-17T03:52:39.024835+0000","last_active":"2024-09-17T03:52:40.291299+0000","last_peered":"2024-09-17T03:52:40.291299+0000","last_clean":"2024-09-17T03:52:40.291299+0000","last_became_active":"2024-09-17T03:52:39.023996+0000","last_became_peered":"2024-09-17T03:52:39.023996+0000","last_unstale":"2024-09-17T03:52:40.291299+0000","last_undegraded":"2024-09-17T03:52:40.291299+0000","last_fullsized":"2024-09-17T03:52:40.291299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:06.467449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288896+0000","last_change":"2024-09-17T03:52:39.015268+0000","last_active":"2024-09-17T03:52:40.288896+0000","last_peered":"2024-09-17T03:52:40.288896+0000","last_clean":"2024-09-17T03:52:40.288896+0000","last_became_active":"2024-09-17T03:52:39.014959+0000","last_became_peered":"2024-09-17T03:52:39.014959+0000","last_unstale":"2024-09-17T03:52:40.288896+0000","last_undegraded":"2024-09-17T03:52:40.288896+0000","last_fullsized":"2024-09-17T03:52:40.288896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:02:44.299798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007816+0000","last_change":"2024-09-17T03:52:39.022874+0000","last_active":"2024-09-17T03:52:40.007816+0000","last_peered":"2024-09-17T03:52:40.007816+0000","last_clean":"2024-09-17T03:52:40.007816+0000","last_became_active":"2024-09-17T03:52:39.022102+0000","last_became_peered":"2024-09-17T03:52:39.022102+0000","last_unstale":"2024-09-17T03:52:40.007816+0000","last_undegraded":"2024-09-17T03:52:40.007816+0000","last_fullsized":"2024-09-17T03:52:40.007816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:11.479509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008090+0000","last_change":"2024-09-17T03:52:39.005338+0000","last_active":"2024-09-17T03:52:40.008090+0000","last_peered":"2024-09-17T03:52:40.008090+0000","last_clean":"2024-09-17T03:52:40.008090+0000","last_became_active":"2024-09-17T03:52:39.005016+0000","last_became_peered":"2024-09-17T03:52:39.005016+0000","last_unstale":"2024-09-17T03:52:40.008090+0000","last_undegraded":"2024-09-17T03:52:40.008090+0000","last_fullsized":"2024-09-17T03:52:40.008090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:32.775243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290327+0000","last_change":"2024-09-17T03:52:39.018888+0000","last_active":"2024-09-17T03:52:40.290327+0000","last_peered":"2024-09-17T03:52:40.290327+0000","last_clean":"2024-09-17T03:52:40.290327+0000","last_became_active":"2024-09-17T03:52:39.018589+0000","last_became_peered":"2024-09-17T03:52:39.018589+0000","last_unstale":"2024-09-17T03:52:40.290327+0000","last_undegraded":"2024-09-17T03:52:40.290327+0000","last_fullsized":"2024-09-17T03:52:40.290327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:12.320186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007818+0000","last_change":"2024-09-17T03:52:39.026308+0000","last_active":"2024-09-17T03:52:40.007818+0000","last_peered":"2024-09-17T03:52:40.007818+0000","last_clean":"2024-09-17T03:52:40.007818+0000","last_became_active":"2024-09-17T03:52:39.025753+0000","last_became_peered":"2024-09-17T03:52:39.025753+0000","last_unstale":"2024-09-17T03:52:40.007818+0000","last_undegraded":"2024-09-17T03:52:40.007818+0000","last_fullsized":"2024-09-17T03:52:40.007818+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289062+0000","last_change":"2024-09-17T03:52:39.013777+0000","last_active":"2024-09-17T03:52:40.289062+0000","last_peered":"2024-09-17T03:52:40.289062+0000","last_clean":"2024-09-17T03:52:40.289062+0000","last_became_active":"2024-09-17T03:52:39.013483+0000","last_became_peered":"2024-09-17T03:52:39.013483+0000","last_unstale":"2024-09-17T03:52:40.289062+0000","last_undegraded":"2024-09-17T03:52:40.289062+0000","last_fullsized":"2024-09-17T03:52:40.289062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:51.177952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009664+0000","last_change":"2024-09-17T03:52:39.014691+0000","last_active":"2024-09-17T03:52:40.009664+0000","last_peered":"2024-09-17T03:52:40.009664+0000","last_clean":"2024-09-17T03:52:40.009664+0000","last_became_active":"2024-09-17T03:52:39.010250+0000","last_became_peered":"2024-09-17T03:52:39.010250+0000","last_unstale":"2024-09-17T03:52:40.009664+0000","last_undegraded":"2024-09-17T03:52:40.009664+0000","last_fullsized":"2024-09-17T03:52:40.009664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:23.670201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290480+0000","last_change":"2024-09-17T03:52:39.030566+0000","last_active":"2024-09-17T03:52:40.290480+0000","last_peered":"2024-09-17T03:52:40.290480+0000","last_clean":"2024-09-17T03:52:40.290480+0000","last_became_active":"2024-09-17T03:52:39.029885+0000","last_became_peered":"2024-09-17T03:52:39.029885+0000","last_unstale":"2024-09-17T03:52:40.290480+0000","last_undegraded":"2024-09-17T03:52:40.290480+0000","last_fullsized":"2024-09-17T03:52:40.290480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:37.598541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288867+0000","last_change":"2024-09-17T03:52:39.027346+0000","last_active":"2024-09-17T03:52:40.288867+0000","last_peered":"2024-09-17T03:52:40.288867+0000","last_clean":"2024-09-17T03:52:40.288867+0000","last_became_active":"2024-09-17T03:52:39.025928+0000","last_became_peered":"2024-09-17T03:52:39.025928+0000","last_unstale":"2024-09-17T03:52:40.288867+0000","last_undegraded":"2024-09-17T03:52:40.288867+0000","last_fullsized":"2024-09-17T03:52:40.288867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:47.351765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007827+0000","last_change":"2024-09-17T03:52:39.018641+0000","last_active":"2024-09-17T03:52:40.007827+0000","last_peered":"2024-09-17T03:52:40.007827+0000","last_clean":"2024-09-17T03:52:40.007827+0000","last_became_active":"2024-09-17T03:52:39.018301+0000","last_became_peered":"2024-09-17T03:52:39.018301+0000","last_unstale":"2024-09-17T03:52:40.007827+0000","last_undegraded":"2024-09-17T03:52:40.007827+0000","last_fullsized":"2024-09-17T03:52:40.007827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:37.690837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287852+0000","last_change":"2024-09-17T03:52:39.022408+0000","last_active":"2024-09-17T03:52:40.287852+0000","last_peered":"2024-09-17T03:52:40.287852+0000","last_clean":"2024-09-17T03:52:40.287852+0000","last_became_active":"2024-09-17T03:52:39.021968+0000","last_became_peered":"2024-09-17T03:52:39.021968+0000","last_unstale":"2024-09-17T03:52:40.287852+0000","last_undegraded":"2024-09-17T03:52:40.287852+0000","last_fullsized":"2024-09-17T03:52:40.287852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:02.552664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009462+0000","last_change":"2024-09-17T03:52:39.012957+0000","last_active":"2024-09-17T03:52:40.009462+0000","last_peered":"2024-09-17T03:52:40.009462+0000","last_clean":"2024-09-17T03:52:40.009462+0000","last_became_active":"2024-09-17T03:52:39.012674+0000","last_became_peered":"2024-09-17T03:52:39.012674+0000","last_unstale":"2024-09-17T03:52:40.009462+0000","last_undegraded":"2024-09-17T03:52:40.009462+0000","last_fullsized":"2024-09-17T03:52:40.009462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:26.669111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287698+0000","last_change":"2024-09-17T03:52:39.007565+0000","last_active":"2024-09-17T03:52:40.287698+0000","last_peered":"2024-09-17T03:52:40.287698+0000","last_clean":"2024-09-17T03:52:40.287698+0000","last_became_active":"2024-09-17T03:52:39.007271+0000","last_became_peered":"2024-09-17T03:52:39.007271+0000","last_unstale":"2024-09-17T03:52:40.287698+0000","last_undegraded":"2024-09-17T03:52:40.287698+0000","last_fullsized":"2024-09-17T03:52:40.287698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:48:35.200854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288812+0000","last_change":"2024-09-17T03:52:39.034015+0000","last_active":"2024-09-17T03:52:40.288812+0000","last_peered":"2024-09-17T03:52:40.288812+0000","last_clean":"2024-09-17T03:52:40.288812+0000","last_became_active":"2024-09-17T03:52:39.033201+0000","last_became_peered":"2024-09-17T03:52:39.033201+0000","last_unstale":"2024-09-17T03:52:40.288812+0000","last_undegraded":"2024-09-17T03:52:40.288812+0000","last_fullsized":"2024-09-17T03:52:40.288812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:45.685964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009736+0000","last_change":"2024-09-17T03:52:39.015002+0000","last_active":"2024-09-17T03:52:40.009736+0000","last_peered":"2024-09-17T03:52:40.009736+0000","last_clean":"2024-09-17T03:52:40.009736+0000","last_became_active":"2024-09-17T03:52:39.010548+0000","last_became_peered":"2024-09-17T03:52:39.010548+0000","last_unstale":"2024-09-17T03:52:40.009736+0000","last_undegraded":"2024-09-17T03:52:40.009736+0000","last_fullsized":"2024-09-17T03:52:40.009736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:47.282513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008550+0000","last_change":"2024-09-17T03:52:39.017861+0000","last_active":"2024-09-17T03:52:40.008550+0000","last_peered":"2024-09-17T03:52:40.008550+0000","last_clean":"2024-09-17T03:52:40.008550+0000","last_became_active":"2024-09-17T03:52:39.017375+0000","last_became_peered":"2024-09-17T03:52:39.017375+0000","last_unstale":"2024-09-17T03:52:40.008550+0000","last_undegraded":"2024-09-17T03:52:40.008550+0000","last_fullsized":"2024-09-17T03:52:40.008550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:59.688466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290958+0000","last_change":"2024-09-17T03:52:39.024408+0000","last_active":"2024-09-17T03:52:40.290958+0000","last_peered":"2024-09-17T03:52:40.290958+0000","last_clean":"2024-09-17T03:52:40.290958+0000","last_became_active":"2024-09-17T03:52:39.024048+0000","last_became_peered":"2024-09-17T03:52:39.024048+0000","last_unstale":"2024-09-17T03:52:40.290958+0000","last_undegraded":"2024-09-17T03:52:40.290958+0000","last_fullsized":"2024-09-17T03:52:40.290958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:55.683703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288095+0000","last_change":"2024-09-17T03:52:39.008943+0000","last_active":"2024-09-17T03:52:40.288095+0000","last_peered":"2024-09-17T03:52:40.288095+0000","last_clean":"2024-09-17T03:52:40.288095+0000","last_became_active":"2024-09-17T03:52:39.008666+0000","last_became_peered":"2024-09-17T03:52:39.008666+0000","last_unstale":"2024-09-17T03:52:40.288095+0000","last_undegraded":"2024-09-17T03:52:40.288095+0000","last_fullsized":"2024-09-17T03:52:40.288095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:24:54.155207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289918+0000","last_change":"2024-09-17T03:52:39.027699+0000","last_active":"2024-09-17T03:52:40.289918+0000","last_peered":"2024-09-17T03:52:40.289918+0000","last_clean":"2024-09-17T03:52:40.289918+0000","last_became_active":"2024-09-17T03:52:39.026422+0000","last_became_peered":"2024-09-17T03:52:39.026422+0000","last_unstale":"2024-09-17T03:52:40.289918+0000","last_undegraded":"2024-09-17T03:52:40.289918+0000","last_fullsized":"2024-09-17T03:52:40.289918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:53.816696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290247+0000","last_change":"2024-09-17T03:52:39.030179+0000","last_active":"2024-09-17T03:52:40.290247+0000","last_peered":"2024-09-17T03:52:40.290247+0000","last_clean":"2024-09-17T03:52:40.290247+0000","last_became_active":"2024-09-17T03:52:39.029845+0000","last_became_peered":"2024-09-17T03:52:39.029845+0000","last_unstale":"2024-09-17T03:52:40.290247+0000","last_undegraded":"2024-09-17T03:52:40.290247+0000","last_fullsized":"2024-09-17T03:52:40.290247+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:27.084828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289510+0000","last_change":"2024-09-17T03:52:39.013300+0000","last_active":"2024-09-17T03:52:40.289510+0000","last_peered":"2024-09-17T03:52:40.289510+0000","last_clean":"2024-09-17T03:52:40.289510+0000","last_became_active":"2024-09-17T03:52:39.012947+0000","last_became_peered":"2024-09-17T03:52:39.012947+0000","last_unstale":"2024-09-17T03:52:40.289510+0000","last_undegraded":"2024-09-17T03:52:40.289510+0000","last_fullsized":"2024-09-17T03:52:40.289510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:42.931837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288725+0000","last_change":"2024-09-17T03:52:39.021402+0000","last_active":"2024-09-17T03:52:40.288725+0000","last_peered":"2024-09-17T03:52:40.288725+0000","last_clean":"2024-09-17T03:52:40.288725+0000","last_became_active":"2024-09-17T03:52:39.021091+0000","last_became_peered":"2024-09-17T03:52:39.021091+0000","last_unstale":"2024-09-17T03:52:40.288725+0000","last_undegraded":"2024-09-17T03:52:40.288725+0000","last_fullsized":"2024-09-17T03:52:40.288725+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:54:56.486110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.034509+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.033549+0000","last_became_peered":"2024-09-17T03:52:39.033549+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:22.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008078+0000","last_change":"2024-09-17T03:52:39.021427+0000","last_active":"2024-09-17T03:52:40.008078+0000","last_peered":"2024-09-17T03:52:40.008078+0000","last_clean":"2024-09-17T03:52:40.008078+0000","last_became_active":"2024-09-17T03:52:39.020901+0000","last_became_peered":"2024-09-17T03:52:39.020901+0000","last_unstale":"2024-09-17T03:52:40.008078+0000","last_undegraded":"2024-09-17T03:52:40.008078+0000","last_fullsized":"2024-09-17T03:52:40.008078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:51.900838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288900+0000","last_change":"2024-09-17T03:52:39.024384+0000","last_active":"2024-09-17T03:52:40.288900+0000","last_peered":"2024-09-17T03:52:40.288900+0000","last_clean":"2024-09-17T03:52:40.288900+0000","last_became_active":"2024-09-17T03:52:39.015719+0000","last_became_peered":"2024-09-17T03:52:39.015719+0000","last_unstale":"2024-09-17T03:52:40.288900+0000","last_undegraded":"2024-09-17T03:52:40.288900+0000","last_fullsized":"2024-09-17T03:52:40.288900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:27.198631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290290+0000","last_change":"2024-09-17T03:52:39.031532+0000","last_active":"2024-09-17T03:52:40.290290+0000","last_peered":"2024-09-17T03:52:40.290290+0000","last_clean":"2024-09-17T03:52:40.290290+0000","last_became_active":"2024-09-17T03:52:39.031235+0000","last_became_peered":"2024-09-17T03:52:39.031235+0000","last_unstale":"2024-09-17T03:52:40.290290+0000","last_undegraded":"2024-09-17T03:52:40.290290+0000","last_fullsized":"2024-09-17T03:52:40.290290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:37.845837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288590+0000","last_change":"2024-09-17T03:52:39.013885+0000","last_active":"2024-09-17T03:52:40.288590+0000","last_peered":"2024-09-17T03:52:40.288590+0000","last_clean":"2024-09-17T03:52:40.288590+0000","last_became_active":"2024-09-17T03:52:39.013584+0000","last_became_peered":"2024-09-17T03:52:39.013584+0000","last_unstale":"2024-09-17T03:52:40.288590+0000","last_undegraded":"2024-09-17T03:52:40.288590+0000","last_fullsized":"2024-09-17T03:52:40.288590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:11.166875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007902+0000","last_change":"2024-09-17T03:52:39.016332+0000","last_active":"2024-09-17T03:52:40.007902+0000","last_peered":"2024-09-17T03:52:40.007902+0000","last_clean":"2024-09-17T03:52:40.007902+0000","last_became_active":"2024-09-17T03:52:39.015950+0000","last_became_peered":"2024-09-17T03:52:39.015950+0000","last_unstale":"2024-09-17T03:52:40.007902+0000","last_undegraded":"2024-09-17T03:52:40.007902+0000","last_fullsized":"2024-09-17T03:52:40.007902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008198+0000","last_change":"2024-09-17T03:52:39.026096+0000","last_active":"2024-09-17T03:52:40.008198+0000","last_peered":"2024-09-17T03:52:40.008198+0000","last_clean":"2024-09-17T03:52:40.008198+0000","last_became_active":"2024-09-17T03:52:39.025309+0000","last_became_peered":"2024-09-17T03:52:39.025309+0000","last_unstale":"2024-09-17T03:52:40.008198+0000","last_undegraded":"2024-09-17T03:52:40.008198+0000","last_fullsized":"2024-09-17T03:52:40.008198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:56:57.011981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289334+0000","last_change":"2024-09-17T03:52:39.019541+0000","last_active":"2024-09-17T03:52:40.289334+0000","last_peered":"2024-09-17T03:52:40.289334+0000","last_clean":"2024-09-17T03:52:40.289334+0000","last_became_active":"2024-09-17T03:52:39.016360+0000","last_became_peered":"2024-09-17T03:52:39.016360+0000","last_unstale":"2024-09-17T03:52:40.289334+0000","last_undegraded":"2024-09-17T03:52:40.289334+0000","last_fullsized":"2024-09-17T03:52:40.289334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:32.145164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289498+0000","last_change":"2024-09-17T03:52:39.023210+0000","last_active":"2024-09-17T03:52:40.289498+0000","last_peered":"2024-09-17T03:52:40.289498+0000","last_clean":"2024-09-17T03:52:40.289498+0000","last_became_active":"2024-09-17T03:52:39.022850+0000","last_became_peered":"2024-09-17T03:52:39.022850+0000","last_unstale":"2024-09-17T03:52:40.289498+0000","last_undegraded":"2024-09-17T03:52:40.289498+0000","last_fullsized":"2024-09-17T03:52:40.289498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:28.421892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":19,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291047+0000","last_change":"2024-09-17T03:52:38.004383+0000","last_active":"2024-09-17T03:52:40.291047+0000","last_peered":"2024-09-17T03:52:40.291047+0000","last_clean":"2024-09-17T03:52:40.291047+0000","last_became_active":"2024-09-17T03:52:38.003610+0000","last_became_peered":"2024-09-17T03:52:38.003610+0000","last_unstale":"2024-09-17T03:52:40.291047+0000","last_undegraded":"2024-09-17T03:52:40.291047+0000","last_fullsized":"2024-09-17T03:52:40.291047+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:36.838884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009806+0000","last_change":"2024-09-17T03:52:39.017710+0000","last_active":"2024-09-17T03:52:40.009806+0000","last_peered":"2024-09-17T03:52:40.009806+0000","last_clean":"2024-09-17T03:52:40.009806+0000","last_became_active":"2024-09-17T03:52:39.009010+0000","last_became_peered":"2024-09-17T03:52:39.009010+0000","last_unstale":"2024-09-17T03:52:40.009806+0000","last_undegraded":"2024-09-17T03:52:40.009806+0000","last_fullsized":"2024-09-17T03:52:40.009806+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:51:50.092155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288354+0000","last_change":"2024-09-17T03:52:37.987588+0000","last_active":"2024-09-17T03:52:40.288354+0000","last_peered":"2024-09-17T03:52:40.288354+0000","last_clean":"2024-09-17T03:52:40.288354+0000","last_became_active":"2024-09-17T03:52:37.986620+0000","last_became_peered":"2024-09-17T03:52:37.986620+0000","last_unstale":"2024-09-17T03:52:40.288354+0000","last_undegraded":"2024-09-17T03:52:40.288354+0000","last_fullsized":"2024-09-17T03:52:40.288354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:58.890146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290564+0000","last_change":"2024-09-17T03:52:39.028972+0000","last_active":"2024-09-17T03:52:40.290564+0000","last_peered":"2024-09-17T03:52:40.290564+0000","last_clean":"2024-09-17T03:52:40.290564+0000","last_became_active":"2024-09-17T03:52:39.028419+0000","last_became_peered":"2024-09-17T03:52:39.028419+0000","last_unstale":"2024-09-17T03:52:40.290564+0000","last_undegraded":"2024-09-17T03:52:40.290564+0000","last_fullsized":"2024-09-17T03:52:40.290564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:56.889010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290134+0000","last_change":"2024-09-17T03:52:38.008448+0000","last_active":"2024-09-17T03:52:40.290134+0000","last_peered":"2024-09-17T03:52:40.290134+0000","last_clean":"2024-09-17T03:52:40.290134+0000","last_became_active":"2024-09-17T03:52:38.008134+0000","last_became_peered":"2024-09-17T03:52:38.008134+0000","last_unstale":"2024-09-17T03:52:40.290134+0000","last_undegraded":"2024-09-17T03:52:40.290134+0000","last_fullsized":"2024-09-17T03:52:40.290134+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:05.150798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289424+0000","last_change":"2024-09-17T03:52:39.009452+0000","last_active":"2024-09-17T03:52:40.289424+0000","last_peered":"2024-09-17T03:52:40.289424+0000","last_clean":"2024-09-17T03:52:40.289424+0000","last_became_active":"2024-09-17T03:52:39.005725+0000","last_became_peered":"2024-09-17T03:52:39.005725+0000","last_unstale":"2024-09-17T03:52:40.289424+0000","last_undegraded":"2024-09-17T03:52:40.289424+0000","last_fullsized":"2024-09-17T03:52:40.289424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:09.356525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290762+0000","last_change":"2024-09-17T03:52:38.004751+0000","last_active":"2024-09-17T03:52:40.290762+0000","last_peered":"2024-09-17T03:52:40.290762+0000","last_clean":"2024-09-17T03:52:40.290762+0000","last_became_active":"2024-09-17T03:52:38.004436+0000","last_became_peered":"2024-09-17T03:52:38.004436+0000","last_unstale":"2024-09-17T03:52:40.290762+0000","last_undegraded":"2024-09-17T03:52:40.290762+0000","last_fullsized":"2024-09-17T03:52:40.290762+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:14.007022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008412+0000","last_change":"2024-09-17T03:52:39.023295+0000","last_active":"2024-09-17T03:52:40.008412+0000","last_peered":"2024-09-17T03:52:40.008412+0000","last_clean":"2024-09-17T03:52:40.008412+0000","last_became_active":"2024-09-17T03:52:39.022209+0000","last_became_peered":"2024-09-17T03:52:39.022209+0000","last_unstale":"2024-09-17T03:52:40.008412+0000","last_undegraded":"2024-09-17T03:52:40.008412+0000","last_fullsized":"2024-09-17T03:52:40.008412+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:02.778195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288785+0000","last_change":"2024-09-17T03:52:38.004706+0000","last_active":"2024-09-17T03:52:40.288785+0000","last_peered":"2024-09-17T03:52:40.288785+0000","last_clean":"2024-09-17T03:52:40.288785+0000","last_became_active":"2024-09-17T03:52:38.004384+0000","last_became_peered":"2024-09-17T03:52:38.004384+0000","last_unstale":"2024-09-17T03:52:40.288785+0000","last_undegraded":"2024-09-17T03:52:40.288785+0000","last_fullsized":"2024-09-17T03:52:40.288785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:24.947156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290252+0000","last_change":"2024-09-17T03:52:39.033871+0000","last_active":"2024-09-17T03:52:40.290252+0000","last_peered":"2024-09-17T03:52:40.290252+0000","last_clean":"2024-09-17T03:52:40.290252+0000","last_became_active":"2024-09-17T03:52:39.033350+0000","last_became_peered":"2024-09-17T03:52:39.033350+0000","last_unstale":"2024-09-17T03:52:40.290252+0000","last_undegraded":"2024-09-17T03:52:40.290252+0000","last_fullsized":"2024-09-17T03:52:40.290252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:29.422776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290522+0000","last_change":"2024-09-17T03:52:38.007787+0000","last_active":"2024-09-17T03:52:40.290522+0000","last_peered":"2024-09-17T03:52:40.290522+0000","last_clean":"2024-09-17T03:52:40.290522+0000","last_became_active":"2024-09-17T03:52:38.007287+0000","last_became_peered":"2024-09-17T03:52:38.007287+0000","last_unstale":"2024-09-17T03:52:40.290522+0000","last_undegraded":"2024-09-17T03:52:40.290522+0000","last_fullsized":"2024-09-17T03:52:40.290522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:37.387663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007768+0000","last_change":"2024-09-17T03:52:39.026791+0000","last_active":"2024-09-17T03:52:40.007768+0000","last_peered":"2024-09-17T03:52:40.007768+0000","last_clean":"2024-09-17T03:52:40.007768+0000","last_became_active":"2024-09-17T03:52:39.026369+0000","last_became_peered":"2024-09-17T03:52:39.026369+0000","last_unstale":"2024-09-17T03:52:40.007768+0000","last_undegraded":"2024-09-17T03:52:40.007768+0000","last_fullsized":"2024-09-17T03:52:40.007768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:11.999239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007539+0000","last_change":"2024-09-17T03:52:38.004637+0000","last_active":"2024-09-17T03:52:40.007539+0000","last_peered":"2024-09-17T03:52:40.007539+0000","last_clean":"2024-09-17T03:52:40.007539+0000","last_became_active":"2024-09-17T03:52:38.004076+0000","last_became_peered":"2024-09-17T03:52:38.004076+0000","last_unstale":"2024-09-17T03:52:40.007539+0000","last_undegraded":"2024-09-17T03:52:40.007539+0000","last_fullsized":"2024-09-17T03:52:40.007539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:38.214053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287689+0000","last_change":"2024-09-17T03:52:39.000484+0000","last_active":"2024-09-17T03:52:40.287689+0000","last_peered":"2024-09-17T03:52:40.287689+0000","last_clean":"2024-09-17T03:52:40.287689+0000","last_became_active":"2024-09-17T03:52:39.000154+0000","last_became_peered":"2024-09-17T03:52:39.000154+0000","last_unstale":"2024-09-17T03:52:40.287689+0000","last_undegraded":"2024-09-17T03:52:40.287689+0000","last_fullsized":"2024-09-17T03:52:40.287689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:57.983193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008024+0000","last_change":"2024-09-17T03:52:37.992784+0000","last_active":"2024-09-17T03:52:40.008024+0000","last_peered":"2024-09-17T03:52:40.008024+0000","last_clean":"2024-09-17T03:52:40.008024+0000","last_became_active":"2024-09-17T03:52:37.992480+0000","last_became_peered":"2024-09-17T03:52:37.992480+0000","last_unstale":"2024-09-17T03:52:40.008024+0000","last_undegraded":"2024-09-17T03:52:40.008024+0000","last_fullsized":"2024-09-17T03:52:40.008024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:21.250972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008452+0000","last_change":"2024-09-17T03:52:39.008286+0000","last_active":"2024-09-17T03:52:40.008452+0000","last_peered":"2024-09-17T03:52:40.008452+0000","last_clean":"2024-09-17T03:52:40.008452+0000","last_became_active":"2024-09-17T03:52:39.007952+0000","last_became_peered":"2024-09-17T03:52:39.007952+0000","last_unstale":"2024-09-17T03:52:40.008452+0000","last_undegraded":"2024-09-17T03:52:40.008452+0000","last_fullsized":"2024-09-17T03:52:40.008452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:30.540116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290933+0000","last_change":"2024-09-17T03:52:38.001450+0000","last_active":"2024-09-17T03:52:40.290933+0000","last_peered":"2024-09-17T03:52:40.290933+0000","last_clean":"2024-09-17T03:52:40.290933+0000","last_became_active":"2024-09-17T03:52:38.001124+0000","last_became_peered":"2024-09-17T03:52:38.001124+0000","last_unstale":"2024-09-17T03:52:40.290933+0000","last_undegraded":"2024-09-17T03:52:40.290933+0000","last_fullsized":"2024-09-17T03:52:40.290933+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:53.092676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289507+0000","last_change":"2024-09-17T03:52:39.027159+0000","last_active":"2024-09-17T03:52:40.289507+0000","last_peered":"2024-09-17T03:52:40.289507+0000","last_clean":"2024-09-17T03:52:40.289507+0000","last_became_active":"2024-09-17T03:52:39.026074+0000","last_became_peered":"2024-09-17T03:52:39.026074+0000","last_unstale":"2024-09-17T03:52:40.289507+0000","last_undegraded":"2024-09-17T03:52:40.289507+0000","last_fullsized":"2024-09-17T03:52:40.289507+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:28.966676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291290+0000","last_change":"2024-09-17T03:52:38.004849+0000","last_active":"2024-09-17T03:52:40.291290+0000","last_peered":"2024-09-17T03:52:40.291290+0000","last_clean":"2024-09-17T03:52:40.291290+0000","last_became_active":"2024-09-17T03:52:38.002393+0000","last_became_peered":"2024-09-17T03:52:38.002393+0000","last_unstale":"2024-09-17T03:52:40.291290+0000","last_undegraded":"2024-09-17T03:52:40.291290+0000","last_fullsized":"2024-09-17T03:52:40.291290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:06.095763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290594+0000","last_change":"2024-09-17T03:52:39.024541+0000","last_active":"2024-09-17T03:52:40.290594+0000","last_peered":"2024-09-17T03:52:40.290594+0000","last_clean":"2024-09-17T03:52:40.290594+0000","last_became_active":"2024-09-17T03:52:39.023962+0000","last_became_peered":"2024-09-17T03:52:39.023962+0000","last_unstale":"2024-09-17T03:52:40.290594+0000","last_undegraded":"2024-09-17T03:52:40.290594+0000","last_fullsized":"2024-09-17T03:52:40.290594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:05.325707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289078+0000","last_change":"2024-09-17T03:52:38.002263+0000","last_active":"2024-09-17T03:52:40.289078+0000","last_peered":"2024-09-17T03:52:40.289078+0000","last_clean":"2024-09-17T03:52:40.289078+0000","last_became_active":"2024-09-17T03:52:38.001712+0000","last_became_peered":"2024-09-17T03:52:38.001712+0000","last_unstale":"2024-09-17T03:52:40.289078+0000","last_undegraded":"2024-09-17T03:52:40.289078+0000","last_fullsized":"2024-09-17T03:52:40.289078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:53.008257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008530+0000","last_change":"2024-09-17T03:52:39.015946+0000","last_active":"2024-09-17T03:52:40.008530+0000","last_peered":"2024-09-17T03:52:40.008530+0000","last_clean":"2024-09-17T03:52:40.008530+0000","last_became_active":"2024-09-17T03:52:39.008987+0000","last_became_peered":"2024-09-17T03:52:39.008987+0000","last_unstale":"2024-09-17T03:52:40.008530+0000","last_undegraded":"2024-09-17T03:52:40.008530+0000","last_fullsized":"2024-09-17T03:52:40.008530+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:24.876376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":21,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008445+0000","last_change":"2024-09-17T03:52:38.000155+0000","last_active":"2024-09-17T03:52:40.008445+0000","last_peered":"2024-09-17T03:52:40.008445+0000","last_clean":"2024-09-17T03:52:40.008445+0000","last_became_active":"2024-09-17T03:52:37.999044+0000","last_became_peered":"2024-09-17T03:52:37.999044+0000","last_unstale":"2024-09-17T03:52:40.008445+0000","last_undegraded":"2024-09-17T03:52:40.008445+0000","last_fullsized":"2024-09-17T03:52:40.008445+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:15.512221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289210+0000","last_change":"2024-09-17T03:52:39.015628+0000","last_active":"2024-09-17T03:52:40.289210+0000","last_peered":"2024-09-17T03:52:40.289210+0000","last_clean":"2024-09-17T03:52:40.289210+0000","last_became_active":"2024-09-17T03:52:39.015286+0000","last_became_peered":"2024-09-17T03:52:39.015286+0000","last_unstale":"2024-09-17T03:52:40.289210+0000","last_undegraded":"2024-09-17T03:52:40.289210+0000","last_fullsized":"2024-09-17T03:52:40.289210+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:02.891865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007787+0000","last_change":"2024-09-17T03:52:38.001510+0000","last_active":"2024-09-17T03:52:40.007787+0000","last_peered":"2024-09-17T03:52:40.007787+0000","last_clean":"2024-09-17T03:52:40.007787+0000","last_became_active":"2024-09-17T03:52:38.001109+0000","last_became_peered":"2024-09-17T03:52:38.001109+0000","last_unstale":"2024-09-17T03:52:40.007787+0000","last_undegraded":"2024-09-17T03:52:40.007787+0000","last_fullsized":"2024-09-17T03:52:40.007787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:53.377063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008218+0000","last_change":"2024-09-17T03:52:39.021149+0000","last_active":"2024-09-17T03:52:40.008218+0000","last_peered":"2024-09-17T03:52:40.008218+0000","last_clean":"2024-09-17T03:52:40.008218+0000","last_became_active":"2024-09-17T03:52:39.020787+0000","last_became_peered":"2024-09-17T03:52:39.020787+0000","last_unstale":"2024-09-17T03:52:40.008218+0000","last_undegraded":"2024-09-17T03:52:40.008218+0000","last_fullsized":"2024-09-17T03:52:40.008218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:49.057250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290649+0000","last_change":"2024-09-17T03:52:38.000222+0000","last_active":"2024-09-17T03:52:40.290649+0000","last_peered":"2024-09-17T03:52:40.290649+0000","last_clean":"2024-09-17T03:52:40.290649+0000","last_became_active":"2024-09-17T03:52:37.998870+0000","last_became_peered":"2024-09-17T03:52:37.998870+0000","last_unstale":"2024-09-17T03:52:40.290649+0000","last_undegraded":"2024-09-17T03:52:40.290649+0000","last_fullsized":"2024-09-17T03:52:40.290649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:01.261692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287904+0000","last_change":"2024-09-17T03:52:39.015246+0000","last_active":"2024-09-17T03:52:40.287904+0000","last_peered":"2024-09-17T03:52:40.287904+0000","last_clean":"2024-09-17T03:52:40.287904+0000","last_became_active":"2024-09-17T03:52:39.014938+0000","last_became_peered":"2024-09-17T03:52:39.014938+0000","last_unstale":"2024-09-17T03:52:40.287904+0000","last_undegraded":"2024-09-17T03:52:40.287904+0000","last_fullsized":"2024-09-17T03:52:40.287904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:12.786786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007815+0000","last_change":"2024-09-17T03:52:37.993426+0000","last_active":"2024-09-17T03:52:40.007815+0000","last_peered":"2024-09-17T03:52:40.007815+0000","last_clean":"2024-09-17T03:52:40.007815+0000","last_became_active":"2024-09-17T03:52:37.992558+0000","last_became_peered":"2024-09-17T03:52:37.992558+0000","last_unstale":"2024-09-17T03:52:40.007815+0000","last_undegraded":"2024-09-17T03:52:40.007815+0000","last_fullsized":"2024-09-17T03:52:40.007815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:50.687889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009000+0000","last_change":"2024-09-17T03:52:39.020708+0000","last_active":"2024-09-17T03:52:40.009000+0000","last_peered":"2024-09-17T03:52:40.009000+0000","last_clean":"2024-09-17T03:52:40.009000+0000","last_became_active":"2024-09-17T03:52:39.019989+0000","last_became_peered":"2024-09-17T03:52:39.019989+0000","last_unstale":"2024-09-17T03:52:40.009000+0000","last_undegraded":"2024-09-17T03:52:40.009000+0000","last_fullsized":"2024-09-17T03:52:40.009000+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:26.316176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":23,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289749+0000","last_change":"2024-09-17T03:52:38.000317+0000","last_active":"2024-09-17T03:52:40.289749+0000","last_peered":"2024-09-17T03:52:40.289749+0000","last_clean":"2024-09-17T03:52:40.289749+0000","last_became_active":"2024-09-17T03:52:37.999933+0000","last_became_peered":"2024-09-17T03:52:37.999933+0000","last_unstale":"2024-09-17T03:52:40.289749+0000","last_undegraded":"2024-09-17T03:52:40.289749+0000","last_fullsized":"2024-09-17T03:52:40.289749+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:56.251630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290593+0000","last_change":"2024-09-17T03:52:39.025015+0000","last_active":"2024-09-17T03:52:40.290593+0000","last_peered":"2024-09-17T03:52:40.290593+0000","last_clean":"2024-09-17T03:52:40.290593+0000","last_became_active":"2024-09-17T03:52:39.024596+0000","last_became_peered":"2024-09-17T03:52:39.024596+0000","last_unstale":"2024-09-17T03:52:40.290593+0000","last_undegraded":"2024-09-17T03:52:40.290593+0000","last_fullsized":"2024-09-17T03:52:40.290593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:18.402266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291013+0000","last_change":"2024-09-17T03:52:38.007757+0000","last_active":"2024-09-17T03:52:40.291013+0000","last_peered":"2024-09-17T03:52:40.291013+0000","last_clean":"2024-09-17T03:52:40.291013+0000","last_became_active":"2024-09-17T03:52:38.007282+0000","last_became_peered":"2024-09-17T03:52:38.007282+0000","last_unstale":"2024-09-17T03:52:40.291013+0000","last_undegraded":"2024-09-17T03:52:40.291013+0000","last_fullsized":"2024-09-17T03:52:40.291013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:33.565057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289931+0000","last_change":"2024-09-17T03:52:39.007897+0000","last_active":"2024-09-17T03:52:40.289931+0000","last_peered":"2024-09-17T03:52:40.289931+0000","last_clean":"2024-09-17T03:52:40.289931+0000","last_became_active":"2024-09-17T03:52:39.007557+0000","last_became_peered":"2024-09-17T03:52:39.007557+0000","last_unstale":"2024-09-17T03:52:40.289931+0000","last_undegraded":"2024-09-17T03:52:40.289931+0000","last_fullsized":"2024-09-17T03:52:40.289931+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:22.330075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290700+0000","last_change":"2024-09-17T03:52:38.002206+0000","last_active":"2024-09-17T03:52:40.290700+0000","last_peered":"2024-09-17T03:52:40.290700+0000","last_clean":"2024-09-17T03:52:40.290700+0000","last_became_active":"2024-09-17T03:52:38.001872+0000","last_became_peered":"2024-09-17T03:52:38.001872+0000","last_unstale":"2024-09-17T03:52:40.290700+0000","last_undegraded":"2024-09-17T03:52:40.290700+0000","last_fullsized":"2024-09-17T03:52:40.290700+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:18.787765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008257+0000","last_change":"2024-09-17T03:52:39.018968+0000","last_active":"2024-09-17T03:52:40.008257+0000","last_peered":"2024-09-17T03:52:40.008257+0000","last_clean":"2024-09-17T03:52:40.008257+0000","last_became_active":"2024-09-17T03:52:39.018682+0000","last_became_peered":"2024-09-17T03:52:39.018682+0000","last_unstale":"2024-09-17T03:52:40.008257+0000","last_undegraded":"2024-09-17T03:52:40.008257+0000","last_fullsized":"2024-09-17T03:52:40.008257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:11:06.157752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290986+0000","last_change":"2024-09-17T03:52:38.009348+0000","last_active":"2024-09-17T03:52:40.290986+0000","last_peered":"2024-09-17T03:52:40.290986+0000","last_clean":"2024-09-17T03:52:40.290986+0000","last_became_active":"2024-09-17T03:52:38.008926+0000","last_became_peered":"2024-09-17T03:52:38.008926+0000","last_unstale":"2024-09-17T03:52:40.290986+0000","last_undegraded":"2024-09-17T03:52:40.290986+0000","last_fullsized":"2024-09-17T03:52:40.290986+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:27.731425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008745+0000","last_change":"2024-09-17T03:52:39.021118+0000","last_active":"2024-09-17T03:52:40.008745+0000","last_peered":"2024-09-17T03:52:40.008745+0000","last_clean":"2024-09-17T03:52:40.008745+0000","last_became_active":"2024-09-17T03:52:39.020763+0000","last_became_peered":"2024-09-17T03:52:39.020763+0000","last_unstale":"2024-09-17T03:52:40.008745+0000","last_undegraded":"2024-09-17T03:52:40.008745+0000","last_fullsized":"2024-09-17T03:52:40.008745+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:10.895566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289021+0000","last_change":"2024-09-17T03:52:38.005886+0000","last_active":"2024-09-17T03:52:40.289021+0000","last_peered":"2024-09-17T03:52:40.289021+0000","last_clean":"2024-09-17T03:52:40.289021+0000","last_became_active":"2024-09-17T03:52:38.005550+0000","last_became_peered":"2024-09-17T03:52:38.005550+0000","last_unstale":"2024-09-17T03:52:40.289021+0000","last_undegraded":"2024-09-17T03:52:40.289021+0000","last_fullsized":"2024-09-17T03:52:40.289021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:52:36.591587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288662+0000","last_change":"2024-09-17T03:52:39.009574+0000","last_active":"2024-09-17T03:52:40.288662+0000","last_peered":"2024-09-17T03:52:40.288662+0000","last_clean":"2024-09-17T03:52:40.288662+0000","last_became_active":"2024-09-17T03:52:39.009382+0000","last_became_peered":"2024-09-17T03:52:39.009382+0000","last_unstale":"2024-09-17T03:52:40.288662+0000","last_undegraded":"2024-09-17T03:52:40.288662+0000","last_fullsized":"2024-09-17T03:52:40.288662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:28.472361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290771+0000","last_change":"2024-09-17T03:52:38.006146+0000","last_active":"2024-09-17T03:52:40.290771+0000","last_peered":"2024-09-17T03:52:40.290771+0000","last_clean":"2024-09-17T03:52:40.290771+0000","last_became_active":"2024-09-17T03:52:38.005099+0000","last_became_peered":"2024-09-17T03:52:38.005099+0000","last_unstale":"2024-09-17T03:52:40.290771+0000","last_undegraded":"2024-09-17T03:52:40.290771+0000","last_fullsized":"2024-09-17T03:52:40.290771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:14.617776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288305+0000","last_change":"2024-09-17T03:52:39.020395+0000","last_active":"2024-09-17T03:52:40.288305+0000","last_peered":"2024-09-17T03:52:40.288305+0000","last_clean":"2024-09-17T03:52:40.288305+0000","last_became_active":"2024-09-17T03:52:39.019294+0000","last_became_peered":"2024-09-17T03:52:39.019294+0000","last_unstale":"2024-09-17T03:52:40.288305+0000","last_undegraded":"2024-09-17T03:52:40.288305+0000","last_fullsized":"2024-09-17T03:52:40.288305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:11.716349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008291+0000","last_change":"2024-09-17T03:52:37.986744+0000","last_active":"2024-09-17T03:52:40.008291+0000","last_peered":"2024-09-17T03:52:40.008291+0000","last_clean":"2024-09-17T03:52:40.008291+0000","last_became_active":"2024-09-17T03:52:37.986385+0000","last_became_peered":"2024-09-17T03:52:37.986385+0000","last_unstale":"2024-09-17T03:52:40.008291+0000","last_undegraded":"2024-09-17T03:52:40.008291+0000","last_fullsized":"2024-09-17T03:52:40.008291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:27.274408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288402+0000","last_change":"2024-09-17T03:52:39.019842+0000","last_active":"2024-09-17T03:52:40.288402+0000","last_peered":"2024-09-17T03:52:40.288402+0000","last_clean":"2024-09-17T03:52:40.288402+0000","last_became_active":"2024-09-17T03:52:39.019548+0000","last_became_peered":"2024-09-17T03:52:39.019548+0000","last_unstale":"2024-09-17T03:52:40.288402+0000","last_undegraded":"2024-09-17T03:52:40.288402+0000","last_fullsized":"2024-09-17T03:52:40.288402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:21.225549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287600+0000","last_change":"2024-09-17T03:52:37.987589+0000","last_active":"2024-09-17T03:52:40.287600+0000","last_peered":"2024-09-17T03:52:40.287600+0000","last_clean":"2024-09-17T03:52:40.287600+0000","last_became_active":"2024-09-17T03:52:37.986646+0000","last_became_peered":"2024-09-17T03:52:37.986646+0000","last_unstale":"2024-09-17T03:52:40.287600+0000","last_undegraded":"2024-09-17T03:52:40.287600+0000","last_fullsized":"2024-09-17T03:52:40.287600+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:04.484089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288208+0000","last_change":"2024-09-17T03:52:39.017123+0000","last_active":"2024-09-17T03:52:40.288208+0000","last_peered":"2024-09-17T03:52:40.288208+0000","last_clean":"2024-09-17T03:52:40.288208+0000","last_became_active":"2024-09-17T03:52:39.016654+0000","last_became_peered":"2024-09-17T03:52:39.016654+0000","last_unstale":"2024-09-17T03:52:40.288208+0000","last_undegraded":"2024-09-17T03:52:40.288208+0000","last_fullsized":"2024-09-17T03:52:40.288208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:41.588250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.291161+0000","last_change":"2024-09-17T03:52:38.007301+0000","last_active":"2024-09-17T03:52:40.291161+0000","last_peered":"2024-09-17T03:52:40.291161+0000","last_clean":"2024-09-17T03:52:40.291161+0000","last_became_active":"2024-09-17T03:52:38.006871+0000","last_became_peered":"2024-09-17T03:52:38.006871+0000","last_unstale":"2024-09-17T03:52:40.291161+0000","last_undegraded":"2024-09-17T03:52:40.291161+0000","last_fullsized":"2024-09-17T03:52:40.291161+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:40.821800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008418+0000","last_change":"2024-09-17T03:52:39.015261+0000","last_active":"2024-09-17T03:52:40.008418+0000","last_peered":"2024-09-17T03:52:40.008418+0000","last_clean":"2024-09-17T03:52:40.008418+0000","last_became_active":"2024-09-17T03:52:39.014907+0000","last_became_peered":"2024-09-17T03:52:39.014907+0000","last_unstale":"2024-09-17T03:52:40.008418+0000","last_undegraded":"2024-09-17T03:52:40.008418+0000","last_fullsized":"2024-09-17T03:52:40.008418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:25.145657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"67'1","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008635+0000","last_change":"2024-09-17T03:52:37.993729+0000","last_active":"2024-09-17T03:52:40.008635+0000","last_peered":"2024-09-17T03:52:40.008635+0000","last_clean":"2024-09-17T03:52:40.008635+0000","last_became_active":"2024-09-17T03:52:37.991713+0000","last_became_peered":"2024-09-17T03:52:37.991713+0000","last_unstale":"2024-09-17T03:52:40.008635+0000","last_undegraded":"2024-09-17T03:52:40.008635+0000","last_fullsized":"2024-09-17T03:52:40.008635+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:14.304806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288837+0000","last_change":"2024-09-17T03:52:39.024919+0000","last_active":"2024-09-17T03:52:40.288837+0000","last_peered":"2024-09-17T03:52:40.288837+0000","last_clean":"2024-09-17T03:52:40.288837+0000","last_became_active":"2024-09-17T03:52:39.022961+0000","last_became_peered":"2024-09-17T03:52:39.022961+0000","last_unstale":"2024-09-17T03:52:40.288837+0000","last_undegraded":"2024-09-17T03:52:40.288837+0000","last_fullsized":"2024-09-17T03:52:40.288837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:47.897977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008905+0000","last_change":"2024-09-17T03:52:38.002436+0000","last_active":"2024-09-17T03:52:40.008905+0000","last_peered":"2024-09-17T03:52:40.008905+0000","last_clean":"2024-09-17T03:52:40.008905+0000","last_became_active":"2024-09-17T03:52:38.001380+0000","last_became_peered":"2024-09-17T03:52:38.001380+0000","last_unstale":"2024-09-17T03:52:40.008905+0000","last_undegraded":"2024-09-17T03:52:40.008905+0000","last_fullsized":"2024-09-17T03:52:40.008905+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:16.020683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289539+0000","last_change":"2024-09-17T03:52:39.013637+0000","last_active":"2024-09-17T03:52:40.289539+0000","last_peered":"2024-09-17T03:52:40.289539+0000","last_clean":"2024-09-17T03:52:40.289539+0000","last_became_active":"2024-09-17T03:52:39.013035+0000","last_became_peered":"2024-09-17T03:52:39.013035+0000","last_unstale":"2024-09-17T03:52:40.289539+0000","last_undegraded":"2024-09-17T03:52:40.289539+0000","last_fullsized":"2024-09-17T03:52:40.289539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:39.874354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.287877+0000","last_change":"2024-09-17T03:52:37.998007+0000","last_active":"2024-09-17T03:52:40.287877+0000","last_peered":"2024-09-17T03:52:40.287877+0000","last_clean":"2024-09-17T03:52:40.287877+0000","last_became_active":"2024-09-17T03:52:37.997670+0000","last_became_peered":"2024-09-17T03:52:37.997670+0000","last_unstale":"2024-09-17T03:52:40.287877+0000","last_undegraded":"2024-09-17T03:52:40.287877+0000","last_fullsized":"2024-09-17T03:52:40.287877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.236372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.009681+0000","last_change":"2024-09-17T03:52:39.016581+0000","last_active":"2024-09-17T03:52:40.009681+0000","last_peered":"2024-09-17T03:52:40.009681+0000","last_clean":"2024-09-17T03:52:40.009681+0000","last_became_active":"2024-09-17T03:52:39.007886+0000","last_became_peered":"2024-09-17T03:52:39.007886+0000","last_unstale":"2024-09-17T03:52:40.009681+0000","last_undegraded":"2024-09-17T03:52:40.009681+0000","last_fullsized":"2024-09-17T03:52:40.009681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:13.021250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008230+0000","last_change":"2024-09-17T03:52:37.991052+0000","last_active":"2024-09-17T03:52:40.008230+0000","last_peered":"2024-09-17T03:52:40.008230+0000","last_clean":"2024-09-17T03:52:40.008230+0000","last_became_active":"2024-09-17T03:52:37.990089+0000","last_became_peered":"2024-09-17T03:52:37.990089+0000","last_unstale":"2024-09-17T03:52:40.008230+0000","last_undegraded":"2024-09-17T03:52:40.008230+0000","last_fullsized":"2024-09-17T03:52:40.008230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:17.450063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290527+0000","last_change":"2024-09-17T03:52:39.034798+0000","last_active":"2024-09-17T03:52:40.290527+0000","last_peered":"2024-09-17T03:52:40.290527+0000","last_clean":"2024-09-17T03:52:40.290527+0000","last_became_active":"2024-09-17T03:52:39.033757+0000","last_became_peered":"2024-09-17T03:52:39.033757+0000","last_unstale":"2024-09-17T03:52:40.290527+0000","last_undegraded":"2024-09-17T03:52:40.290527+0000","last_fullsized":"2024-09-17T03:52:40.290527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:40.489097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.007401+0000","last_change":"2024-09-17T03:52:38.006002+0000","last_active":"2024-09-17T03:52:40.007401+0000","last_peered":"2024-09-17T03:52:40.007401+0000","last_clean":"2024-09-17T03:52:40.007401+0000","last_became_active":"2024-09-17T03:52:38.004725+0000","last_became_peered":"2024-09-17T03:52:38.004725+0000","last_unstale":"2024-09-17T03:52:40.007401+0000","last_undegraded":"2024-09-17T03:52:40.007401+0000","last_fullsized":"2024-09-17T03:52:40.007401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:05.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.008222+0000","last_change":"2024-09-17T03:52:39.010421+0000","last_active":"2024-09-17T03:52:40.008222+0000","last_peered":"2024-09-17T03:52:40.008222+0000","last_clean":"2024-09-17T03:52:40.008222+0000","last_became_active":"2024-09-17T03:52:39.010077+0000","last_became_peered":"2024-09-17T03:52:39.010077+0000","last_unstale":"2024-09-17T03:52:40.008222+0000","last_undegraded":"2024-09-17T03:52:40.008222+0000","last_fullsized":"2024-09-17T03:52:40.008222+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:57.258583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289687+0000","last_change":"2024-09-17T03:52:37.998870+0000","last_active":"2024-09-17T03:52:40.289687+0000","last_peered":"2024-09-17T03:52:40.289687+0000","last_clean":"2024-09-17T03:52:40.289687+0000","last_became_active":"2024-09-17T03:52:37.998561+0000","last_became_peered":"2024-09-17T03:52:37.998561+0000","last_unstale":"2024-09-17T03:52:40.289687+0000","last_undegraded":"2024-09-17T03:52:40.289687+0000","last_fullsized":"2024-09-17T03:52:40.289687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:58.681513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290032+0000","last_change":"2024-09-17T03:52:39.022554+0000","last_active":"2024-09-17T03:52:40.290032+0000","last_peered":"2024-09-17T03:52:40.290032+0000","last_clean":"2024-09-17T03:52:40.290032+0000","last_became_active":"2024-09-17T03:52:39.022101+0000","last_became_peered":"2024-09-17T03:52:39.022101+0000","last_unstale":"2024-09-17T03:52:40.290032+0000","last_undegraded":"2024-09-17T03:52:40.290032+0000","last_fullsized":"2024-09-17T03:52:40.290032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:37.608298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.289503+0000","last_change":"2024-09-17T03:52:38.004100+0000","last_active":"2024-09-17T03:52:40.289503+0000","last_peered":"2024-09-17T03:52:40.289503+0000","last_clean":"2024-09-17T03:52:40.289503+0000","last_became_active":"2024-09-17T03:52:38.000956+0000","last_became_peered":"2024-09-17T03:52:38.000956+0000","last_unstale":"2024-09-17T03:52:40.289503+0000","last_undegraded":"2024-09-17T03:52:40.289503+0000","last_fullsized":"2024-09-17T03:52:40.289503+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:29.352497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":13,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288572+0000","last_change":"2024-09-17T03:52:39.013661+0000","last_active":"2024-09-17T03:52:40.288572+0000","last_peered":"2024-09-17T03:52:40.288572+0000","last_clean":"2024-09-17T03:52:40.288572+0000","last_became_active":"2024-09-17T03:52:39.013380+0000","last_became_peered":"2024-09-17T03:52:39.013380+0000","last_unstale":"2024-09-17T03:52:40.288572+0000","last_undegraded":"2024-09-17T03:52:40.288572+0000","last_fullsized":"2024-09-17T03:52:40.288572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:37.973877+0000","last_clean_scrub_stamp":"2024-09-17T03:52:37.973877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:41.998544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.290721+0000","last_change":"2024-09-17T03:52:38.001315+0000","last_active":"2024-09-17T03:52:40.290721+0000","last_peered":"2024-09-17T03:52:40.290721+0000","last_clean":"2024-09-17T03:52:40.290721+0000","last_became_active":"2024-09-17T03:52:37.999404+0000","last_became_peered":"2024-09-17T03:52:37.999404+0000","last_unstale":"2024-09-17T03:52:40.290721+0000","last_undegraded":"2024-09-17T03:52:40.290721+0000","last_fullsized":"2024-09-17T03:52:40.290721+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:52:36.966924+0000","last_clean_scrub_stamp":"2024-09-17T03:52:36.966924+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:34:25.654563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":194,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-17T03:52:40.288936+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:52:40.288936+0000","last_peered":"2024-09-17T03:52:40.288936+0000","last_clean":"2024-09-17T03:52:40.288936+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:52:40.288936+0000","last_undegraded":"2024-09-17T03:52:40.288936+0000","last_fullsized":"2024-09-17T03:52:40.288936+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509482,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32584,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339256,"statfs":{"total":96636764160,"available":96603398144,"internally_reserved":0,"allocated":1421312,"data_stored":931564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509485,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":823296,"data_stored":340650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509482,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":827392,"data_stored":340692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509484,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":831488,"data_stored":342192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509484,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":839680,"data_stored":342808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":827392,"data_stored":340672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542188,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1413120,"data_stored":930948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:52:48.855 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:52:48.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 -- 172.21.15.129:0/739397342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0070440 msgr2=0x7f00f0074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 --2- 172.21.15.129:0/739397342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0070440 0x7f00f0074b80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f00f00710f0 tx=0x7f00e802aac0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 -- 172.21.15.129:0/739397342 shutdown_connections 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 --2- 172.21.15.129:0/739397342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0070440 0x7f00f0074b80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 --2- 172.21.15.129:0/739397342 >> v2:172.21.15.181:3300/0 conn(0x7f00f0070030 0x7f00f0074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 -- 172.21.15.129:0/739397342 >> 172.21.15.129:0/739397342 conn(0x7f00f00b1cb0 msgr2=0x7f00f00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 -- 172.21.15.129:0/739397342 shutdown_connections 2024-09-17T03:52:49.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.121+0000 7f00f4c00640 1 -- 172.21.15.129:0/739397342 wait complete. 2024-09-17T03:52:49.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.124+0000 7f00f4c00640 1 Processor -- start 2024-09-17T03:52:49.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.124+0000 7f00f4c00640 1 -- start start 2024-09-17T03:52:49.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00f0070030 0x7f00f00cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 0x7f00f009b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00f00b3a40 con 0x7f00f0073170 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00f00b38c0 con 0x7f00f0070440 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00f4c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f00f00b3bc0 con 0x7f00f0070030 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37030/0 (socket says 172.21.15.129:37030) 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.125+0000 7f00ee800640 1 -- 172.21.15.129:0/2836504926 learned_addr learned my addr 172.21.15.129:0/2836504926 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:49.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ede00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00f0070030 0x7f00f00cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00e5400640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 0x7f00f009b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 -- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00f0070030 msgr2=0x7f00f00cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00f0070030 0x7f00f00cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 -- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 msgr2=0x7f00f009b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 0x7f00f009b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 -- 172.21.15.129:0/2836504926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f01f5e40 con 0x7f00f0073170 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00e5400640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 0x7f00f009b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00ee800640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f00e802b0b0 tx=0x7f00e8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8004250 con 0x7f00f0073170 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00e80043f0 con 0x7f00f0073170 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.126+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f01f6120 con 0x7f00f0073170 2024-09-17T03:52:49.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.127+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8004820 con 0x7f00f0073170 2024-09-17T03:52:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.127+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f01f66e0 con 0x7f00f0073170 2024-09-17T03:52:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.128+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f00b40e0 con 0x7f00f0073170 2024-09-17T03:52:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.129+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f00e8049430 con 0x7f00f0073170 2024-09-17T03:52:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.130+0000 7f00e6a00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 0x7f00cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.130+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f00e80bbc00 con 0x7f00f0073170 2024-09-17T03:52:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.130+0000 7f00ede00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 0x7f00cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.131+0000 7f00ede00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 0x7f00cc07b0a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00e00015e0 tx=0x7f00e0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.134+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f00e8032030 con 0x7f00f0073170 2024-09-17T03:52:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.391+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f00f00d0e50 con 0x7f00f0073170 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.395+0000 7f00e6a00640 1 -- 172.21.15.129:0/2836504926 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) ==== 190+0+147 (secure 0 0 0) 0x7f00e80840b0 con 0x7f00f0073170 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:52:49.394 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:52:49.395 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.397+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 msgr2=0x7f00cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.397+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 0x7f00cc07b0a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00e00015e0 tx=0x7f00e0009040 comp rx=0 tx=0).stop 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.397+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 msgr2=0x7f00f009cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.397+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f00e802b0b0 tx=0x7f00e8003f10 comp rx=0 tx=0).stop 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 shutdown_connections 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00cc078b90 0x7f00cc07b0a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00f0073170 0x7f00f009cf40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00f0070440 0x7f00f009b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 --2- 172.21.15.129:0/2836504926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00f0070030 0x7f00f00cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 >> 172.21.15.129:0/2836504926 conn(0x7f00f00b1cb0 msgr2=0x7f00f00aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 shutdown_connections 2024-09-17T03:52:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.398+0000 7f00f4c00640 1 -- 172.21.15.129:0/2836504926 wait complete. 2024-09-17T03:52:49.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 -- 172.21.15.129:0/394133917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 msgr2=0x7f556815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 --2- 172.21.15.129:0/394133917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815c780 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f55680b4b40 tx=0x7f555802a560 comp rx=0 tx=0).stop 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 -- 172.21.15.129:0/394133917 shutdown_connections 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 --2- 172.21.15.129:0/394133917 >> v2:172.21.15.181:3300/0 conn(0x7f5568070440 0x7f55680b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 --2- 172.21.15.129:0/394133917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815c780 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.642+0000 7f556ca00640 1 -- 172.21.15.129:0/394133917 >> 172.21.15.129:0/394133917 conn(0x7f55680b1cb0 msgr2=0x7f55680b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.643+0000 7f556ca00640 1 -- 172.21.15.129:0/394133917 shutdown_connections 2024-09-17T03:52:49.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.643+0000 7f556ca00640 1 -- 172.21.15.129:0/394133917 wait complete. 2024-09-17T03:52:49.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.645+0000 7f556ca00640 1 Processor -- start 2024-09-17T03:52:49.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.645+0000 7f556ca00640 1 -- start start 2024-09-17T03:52:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5568070030 0x7f5568152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5568070440 0x7f55680d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55680b3a40 con 0x7f5568073170 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55680b38c0 con 0x7f5568070030 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.646+0000 7f556ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f55680b3bc0 con 0x7f5568070440 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37044/0 (socket says 172.21.15.129:37044) 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 -- 172.21.15.129:0/1737622545 learned_addr learned my addr 172.21.15.129:0/1737622545 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5565e00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5568070030 0x7f5568152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5565400640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5568070440 0x7f55680d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 -- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5568070440 msgr2=0x7f55680d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5568070440 0x7f55680d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 -- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5568070030 msgr2=0x7f5568152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5568070030 0x7f5568152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 -- 172.21.15.129:0/1737622545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55680d0550 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f5566800640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9c0 tx=0x7f556000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.647+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560004d20 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.648+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f556000cb20 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.648+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560005020 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.648+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55680d0d40 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.648+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55681f7620 con 0x7f5568073170 2024-09-17T03:52:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.649+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55680b48a0 con 0x7f5568073170 2024-09-17T03:52:49.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.650+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f55600051c0 con 0x7f5568073170 2024-09-17T03:52:49.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.651+0000 7f555e000640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 0x7f554807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:49.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.651+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f556009b620 con 0x7f5568073170 2024-09-17T03:52:49.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.651+0000 7f5565e00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 0x7f554807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:49.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.652+0000 7f5565e00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 0x7f554807b0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5558033000 tx=0x7f5558004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:49.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.656+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5560016030 con 0x7f5568073170 2024-09-17T03:52:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.873+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5568051040 con 0x7f5568073170 2024-09-17T03:52:49.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.874+0000 7f555e000640 1 -- 172.21.15.129:0/1737622545 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f5560063a60 con 0x7f5568073170 2024-09-17T03:52:49.873 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:52:49.873 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:52:49.873 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:52:49.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 msgr2=0x7f554807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 0x7f554807b0a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5558033000 tx=0x7f5558004730 comp rx=0 tx=0).stop 2024-09-17T03:52:49.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 msgr2=0x7f556815f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f556000e9c0 tx=0x7f556000ee80 comp rx=0 tx=0).stop 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 shutdown_connections 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5548078b90 0x7f554807b0a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5568073170 0x7f556815f340 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5568070440 0x7f55680d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 --2- 172.21.15.129:0/1737622545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5568070030 0x7f5568152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 >> 172.21.15.129:0/1737622545 conn(0x7f55680b1cb0 msgr2=0x7f55680adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 shutdown_connections 2024-09-17T03:52:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:49.876+0000 7f556ca00640 1 -- 172.21.15.129:0/1737622545 wait complete. 2024-09-17T03:52:50.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:52:38.484197+0000', 'modified': '2024-09-17T03:52:41.494978+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5865}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5865': {'gid': 5865, 'name': 'c', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6835/2480264914', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 2480264914}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2480264914}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5865, 'qdb_cluster': [5865]} 2024-09-17T03:52:50.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:52:50.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:52:50.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 -- 172.21.15.129:0/3423544887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0070440 msgr2=0x7f32d0074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 --2- 172.21.15.129:0/3423544887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0070440 0x7f32d0074b80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009830 tx=0x7f32c802a530 comp rx=0 tx=0).stop 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 -- 172.21.15.129:0/3423544887 shutdown_connections 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 --2- 172.21.15.129:0/3423544887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0070440 0x7f32d0074b80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 --2- 172.21.15.129:0/3423544887 >> v2:172.21.15.181:3300/0 conn(0x7f32d0070030 0x7f32d0074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 -- 172.21.15.129:0/3423544887 >> 172.21.15.129:0/3423544887 conn(0x7f32d00b1cb0 msgr2=0x7f32d00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 -- 172.21.15.129:0/3423544887 shutdown_connections 2024-09-17T03:52:50.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.534+0000 7f32d4c00640 1 -- 172.21.15.129:0/3423544887 wait complete. 2024-09-17T03:52:50.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.536+0000 7f32d4c00640 1 Processor -- start 2024-09-17T03:52:50.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.536+0000 7f32d4c00640 1 -- start start 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32d0070030 0x7f32d014ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0073170 0x7f32d01f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00b3a40 con 0x7f32d0070440 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00b38c0 con 0x7f32d0070030 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32d4c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f32d00b3bc0 con 0x7f32d0073170 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37056/0 (socket says 172.21.15.129:37056) 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32c5400640 1 -- 172.21.15.129:0/1744442014 learned_addr learned my addr 172.21.15.129:0/1744442014 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.538+0000 7f32cde00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32d0070030 0x7f32d014ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32ce800640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0073170 0x7f32d01f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 -- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0073170 msgr2=0x7f32d01f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0073170 0x7f32d01f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 -- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32d0070030 msgr2=0x7f32d014ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32d0070030 0x7f32d014ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 -- 172.21.15.129:0/1744442014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d014b3a0 con 0x7f32d0070440 2024-09-17T03:52:50.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c5400640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e9f0 tx=0x7f32b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c6a00640 1 -- 172.21.15.129:0/1744442014 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b800cd80 con 0x7f32d0070440 2024-09-17T03:52:50.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c6a00640 1 -- 172.21.15.129:0/1744442014 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b8004020 con 0x7f32d0070440 2024-09-17T03:52:50.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32c6a00640 1 -- 172.21.15.129:0/1744442014 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8004470 con 0x7f32d0070440 2024-09-17T03:52:50.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d00cf070 con 0x7f32d0070440 2024-09-17T03:52:50.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.539+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d009b250 con 0x7f32d0070440 2024-09-17T03:52:50.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.543+0000 7f32c6a00640 1 -- 172.21.15.129:0/1744442014 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f32b80035f0 con 0x7f32d0070440 2024-09-17T03:52:50.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.544+0000 7f32dbc78740 1 Processor -- start 2024-09-17T03:52:50.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.544+0000 7f32c6a00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 0x7f32ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.544+0000 7f32c6a00640 1 -- 172.21.15.129:0/1744442014 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f32b809b6d0 con 0x7f32d0070440 2024-09-17T03:52:50.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.544+0000 7f32dbc78740 1 -- start start 2024-09-17T03:52:50.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.544+0000 7f32ce800640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 0x7f32ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.545+0000 7f32ce800640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 0x7f32ac07b0a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f32c8034000 tx=0x7f32c8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.545+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d847970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.545+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.545+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d456590 0x55e51d817f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32dbc78740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e51d7a1430 con 0x55e51d44e3b0 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e51d7915d0 con 0x55e51d477270 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55e51d780a00 con 0x55e51d456590 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37062/0 (socket says 172.21.15.129:37062) 2024-09-17T03:52:50.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 -- 172.21.15.129:0/1819595400 learned_addr learned my addr 172.21.15.129:0/1819595400 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32cde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d847970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32ce800640 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d456590 0x55e51d817f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 -- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d456590 msgr2=0x55e51d817f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d456590 0x55e51d817f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 -- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 msgr2=0x55e51d847970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d847970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32c5400640 1 -- 172.21.15.129:0/1819595400 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51d749520 con 0x55e51d44e3b0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.546+0000 7f32cde00640 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d847970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.547+0000 7f32c5400640 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f32b8063430 tx=0x7f32b8063270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a2bad86ea89118 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.547+0000 7f32be600640 1 -- 172.21.15.129:0/1819595400 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8062fb0 con 0x55e51d44e3b0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.547+0000 7f32be600640 1 -- 172.21.15.129:0/1819595400 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b8063ba0 con 0x55e51d44e3b0 2024-09-17T03:52:50.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.547+0000 7f32be600640 1 -- 172.21.15.129:0/1819595400 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8062900 con 0x55e51d44e3b0 2024-09-17T03:52:50.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 -- 172.21.15.129:0/1819595400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 msgr2=0x55e51d847eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f32b8063430 tx=0x7f32b8063270 comp rx=0 tx=0).stop 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 -- 172.21.15.129:0/1819595400 shutdown_connections 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d456590 0x55e51d817f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d44e3b0 0x55e51d847eb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1819595400 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d847970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 -- 172.21.15.129:0/1819595400 >> 172.21.15.129:0/1819595400 conn(0x55e51d46b3c0 msgr2=0x55e51d81d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 -- 172.21.15.129:0/1819595400 shutdown_connections 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.548+0000 7f32dbc78740 1 -- 172.21.15.129:0/1819595400 wait complete. 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.549+0000 7f32dbc78740 1 -- start start 2024-09-17T03:52:50.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.549+0000 7f32dbc78740 1 Processor -- start 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d879e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d456590 0x55e51d818430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e51d727330 con 0x55e51d477270 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e51d784810 con 0x55e51d456590 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.550+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55e51d4f1620 con 0x55e51d44e3b0 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32dbc78740 10 client.0 _ll_register_callbacks cb 0x55e51d72c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:52:50.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37078/0 (socket says 172.21.15.129:37078) 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 -- 172.21.15.129:0/2035746708 learned_addr learned my addr 172.21.15.129:0/2035746708 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32cde00640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d879e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32c5400640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d456590 0x55e51d818430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 -- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 msgr2=0x55e51d879e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d879e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 -- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d456590 msgr2=0x55e51d818430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d456590 0x55e51d818430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.551+0000 7f32ce800640 1 -- 172.21.15.129:0/2035746708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51d4e2850 con 0x55e51d477270 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32ce800640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f32c802ab20 tx=0x7f32c8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3057012c7ec0d13f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:52:50.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c804aa40 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f32c804abe0 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32b8063ba0 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32c804ade0 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32dbc78740 10 client.6018 fetch_fsmap learned FSMap version 45 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55e51d5596b0 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.552+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b8063ba0 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7f32c805b050 con 0x55e51d477270 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 10 client.6018 fetch_fsmap finished waiting for FSMap version 45 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 10 client.6018 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 20 client.6018 populate_metadata read hostname 'smithi129' 2024-09-17T03:52:50.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d435e40 0x55e51d85d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 4 client.6018 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:52:50.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 --> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e51d793520 con 0x55e51d435e40 2024-09-17T03:52:50.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32cde00640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d435e40 0x55e51d85d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32b2a00640 10 client.6018 ms_handle_connect on v2:172.21.15.129:6833/2480264914 2024-09-17T03:52:50.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.553+0000 7f32cde00640 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d435e40 0x55e51d85d0c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7279260d15b0e2a server_cookie=130d9ec4fa2f0924 in_seq=0 out_seq=0 2024-09-17T03:52:50.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.554+0000 7f32b2a00640 1 -- 172.21.15.129:0/2035746708 <== mds.0 v2:172.21.15.129:6833/2480264914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55e51d793520 con 0x55e51d435e40 2024-09-17T03:52:50.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.554+0000 7f32b2a00640 10 client.6018 handle_command_reply: tid=0 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.554+0000 7f32dbc78740 1 client.6018 shutdown 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 msgr2=0x55e51d872ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f32c802ab20 tx=0x7f32c8005f90 comp rx=0 tx=0).stop 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 shutdown_connections 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d435e40 0x55e51d85d0c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d477270 0x55e51d872ef0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d456590 0x55e51d818430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2035746708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d879e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32b2a00640 0 client.6018 ms_handle_reset on v2:172.21.15.129:6833/2480264914 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 >> 172.21.15.129:0/2035746708 conn(0x55e51d46b3c0 msgr2=0x55e51d862fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 shutdown_connections 2024-09-17T03:52:50.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 1 -- 172.21.15.129:0/2035746708 wait complete. 2024-09-17T03:52:50.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.555+0000 7f32dbc78740 20 client.6018 trim_cache size 0 max 16384 2024-09-17T03:52:50.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.574+0000 7f32dbc78740 1 Processor -- start 2024-09-17T03:52:50.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.574+0000 7f32dbc78740 1 -- start start 2024-09-17T03:52:50.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d44e3b0 0x55e51d817ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d477270 0x55e51d85d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00cc620 con 0x55e51d456590 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f32d00cb980 con 0x55e51d44e3b0 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.575+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f32d00cbfd0 con 0x55e51d477270 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37094/0 (socket says 172.21.15.129:37094) 2024-09-17T03:52:50.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 -- 172.21.15.129:0/1968342653 learned_addr learned my addr 172.21.15.129:0/1968342653 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 -- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d477270 msgr2=0x55e51d85d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d477270 0x55e51d85d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 -- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d44e3b0 msgr2=0x55e51d817ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d44e3b0 0x55e51d817ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 -- 172.21.15.129:0/1968342653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e51d81e650 con 0x55e51d456590 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32c5400640 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f32b8063430 tx=0x7f32b8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86689ad461b6596 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.576+0000 7f32bdc00640 1 -- 172.21.15.129:0/1968342653 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8062fb0 con 0x55e51d456590 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32bdc00640 1 -- 172.21.15.129:0/1968342653 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b8083070 con 0x55e51d456590 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32bdc00640 1 -- 172.21.15.129:0/1968342653 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8062960 con 0x55e51d456590 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 -- 172.21.15.129:0/1968342653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 msgr2=0x55e51d8474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f32b8063430 tx=0x7f32b8000bc0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 -- 172.21.15.129:0/1968342653 shutdown_connections 2024-09-17T03:52:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d477270 0x55e51d85d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8474a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 --2- 172.21.15.129:0/1968342653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d44e3b0 0x55e51d817ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 -- 172.21.15.129:0/1968342653 >> 172.21.15.129:0/1968342653 conn(0x55e51d46b3c0 msgr2=0x55e51d862750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 -- 172.21.15.129:0/1968342653 shutdown_connections 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.577+0000 7f32dbc78740 1 -- 172.21.15.129:0/1968342653 wait complete. 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.578+0000 7f32dbc78740 1 -- start start 2024-09-17T03:52:50.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.579+0000 7f32dbc78740 1 Processor -- start 2024-09-17T03:52:50.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d8cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d8177f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55e51d875f40 con 0x55e51d456590 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55e51d876d30 con 0x55e51d477270 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55e51d831060 con 0x55e51d44e3b0 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32dbc78740 10 client.0 _ll_register_callbacks cb 0x7f32c80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32c5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37110/0 (socket says 172.21.15.129:37110) 2024-09-17T03:52:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.580+0000 7f32c5400640 1 -- 172.21.15.129:0/2622194007 learned_addr learned my addr 172.21.15.129:0/2622194007 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32cde00640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d8cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 -- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 msgr2=0x55e51d8cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32ce800640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d8177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d8cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 -- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 msgr2=0x55e51d8177f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d8177f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 -- 172.21.15.129:0/2622194007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e51d817fa0 con 0x55e51d456590 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32cde00640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d8cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32ce800640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d8177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32c5400640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c6a0 tx=0x7f32b8003b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f3bd53b2592c0d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:52:50.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8062fb0 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f32b806a3a0 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32b8083070 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32b80634a0 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32dbc78740 10 client.6024 fetch_fsmap learned FSMap version 45 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.581+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) -- 0x55e51d836c00 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32b8083070 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 45) ==== 3502+0+0 (secure 0 0 0) 0x7f32b8079170 con 0x55e51d456590 2024-09-17T03:52:50.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 10 client.6024 fetch_fsmap finished waiting for FSMap version 45 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 10 client.6024 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 20 client.6024 populate_metadata read hostname 'smithi129' 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d835fd0 0x55e51d8363e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 4 client.6024 mds_command: new command op to 5865 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 --> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] -- command(tid 0: {"prefix": "status"}) -- 0x55e51d78efd0 con 0x55e51d835fd0 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.582+0000 7f32cde00640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d835fd0 0x55e51d8363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32cde00640 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d835fd0 0x55e51d8363e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33f13f577726ce5f server_cookie=6acc55adeeea8417 in_seq=0 out_seq=0 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32b2a00640 10 client.6024 ms_handle_connect on v2:172.21.15.129:6833/2480264914 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32b2a00640 1 -- 172.21.15.129:0/2622194007 <== mds.0 v2:172.21.15.129:6833/2480264914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55e51d78efd0 con 0x55e51d835fd0 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32b2a00640 10 client.6024 handle_command_reply: tid=0 2024-09-17T03:52:50.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 client.6024 shutdown 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 msgr2=0x55e51d8172b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f32b800c6a0 tx=0x7f32b8003b10 comp rx=0 tx=0).stop 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 shutdown_connections 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:6833/2480264914,v1:172.21.15.129:6835/2480264914] conn(0x55e51d835fd0 0x55e51d8363e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e51d477270 0x55e51d8177f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55e51d456590 0x55e51d8172b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32b2a00640 0 client.6024 ms_handle_reset on v2:172.21.15.129:6833/2480264914 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 --2- 172.21.15.129:0/2622194007 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55e51d44e3b0 0x55e51d8cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 >> 172.21.15.129:0/2622194007 conn(0x55e51d46b3c0 msgr2=0x55e51d8474a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 shutdown_connections 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.583+0000 7f32dbc78740 1 -- 172.21.15.129:0/2622194007 wait complete. 2024-09-17T03:52:50.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32dbc78740 20 client.6024 trim_cache size 0 max 16384 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 5865, 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.594052575999999, 2024-09-17T03:52:50.588 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 45, 2024-09-17T03:52:50.589 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 68, 2024-09-17T03:52:50.589 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 68, 2024-09-17T03:52:50.589 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 22.647382916000002 2024-09-17T03:52:50.589 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 msgr2=0x7f32ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 0x7f32ac07b0a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f32c8034000 tx=0x7f32c8004730 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 msgr2=0x7f32d014e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f32b800e9f0 tx=0x7f32b800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 shutdown_connections 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32ac078b90 0x7f32ac07b0a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32d0073170 0x7f32d01f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32d0070440 0x7f32d014e0e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 --2- 172.21.15.129:0/1744442014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32d0070030 0x7f32d014ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 >> 172.21.15.129:0/1744442014 conn(0x7f32d00b1cb0 msgr2=0x7f32d015ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 shutdown_connections 2024-09-17T03:52:50.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.584+0000 7f32d4c00640 1 -- 172.21.15.129:0/1744442014 wait complete. 2024-09-17T03:52:50.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:52:50.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 -- 172.21.15.129:0/1631166751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840ce020 msgr2=0x7f78840ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 --2- 172.21.15.129:0/1631166751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840ce020 0x7f78840ce430 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f7884071d60 tx=0x7f787c02aac0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 -- 172.21.15.129:0/1631166751 shutdown_connections 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 --2- 172.21.15.129:0/1631166751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840ce020 0x7f78840ce430 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 --2- 172.21.15.129:0/1631166751 >> v2:172.21.15.181:3300/0 conn(0x7f78840cf6f0 0x7f78840cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 -- 172.21.15.129:0/1631166751 >> 172.21.15.129:0/1631166751 conn(0x7f78840b1cb0 msgr2=0x7f78840b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 -- 172.21.15.129:0/1631166751 shutdown_connections 2024-09-17T03:52:50.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.827+0000 7f788b000640 1 -- 172.21.15.129:0/1631166751 wait complete. 2024-09-17T03:52:50.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.829+0000 7f788b000640 1 Processor -- start 2024-09-17T03:52:50.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.830+0000 7f788b000640 1 -- start start 2024-09-17T03:52:50.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 0x7f78841618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 0x7f788414e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78840701b0 con 0x7f78840cf6f0 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f78840cf3d0 con 0x7f78840ccb30 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f788b000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7884070330 con 0x7f78840ce020 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f7889000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f7889000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37126/0 (socket says 172.21.15.129:37126) 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.831+0000 7f7889000640 1 -- 172.21.15.129:0/378740038 learned_addr learned my addr 172.21.15.129:0/378740038 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7883e00640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 0x7f78841618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7883400640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 0x7f788414e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 -- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 msgr2=0x7f788414e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 0x7f788414e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 -- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 msgr2=0x7f78841618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:50.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 0x7f78841618c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 -- 172.21.15.129:0/378740038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7884160820 con 0x7f78840cf6f0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7883e00640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 0x7f78841618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7883400640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 0x7f788414e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f7889000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f787c009d60 tx=0x7f787c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c044070 con 0x7f78840cf6f0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f787c003e50 con 0x7f78840cf6f0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7884160b00 con 0x7f78840cf6f0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.832+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c0044b0 con 0x7f78840cf6f0 2024-09-17T03:52:50.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.833+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841610c0 con 0x7f78840cf6f0 2024-09-17T03:52:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.834+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78840ceb30 con 0x7f78840cf6f0 2024-09-17T03:52:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.835+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f787c004650 con 0x7f78840cf6f0 2024-09-17T03:52:50.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.836+0000 7f787be00640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:50.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.836+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f787c0bb3f0 con 0x7f78840cf6f0 2024-09-17T03:52:50.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.836+0000 7f7883e00640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:50.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.837+0000 7f7883e00640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7884071290 tx=0x7f7874005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:50.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:50.840+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f787c032030 con 0x7f78840cf6f0 2024-09-17T03:52:51.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.059+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7884151f70 con 0x7f78840cf6f0 2024-09-17T03:52:51.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.059+0000 7f787be00640 1 -- 172.21.15.129:0/378740038 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) ==== 53+0+73 (secure 0 0 0) 0x7f787c083820 con 0x7f78840cf6f0 2024-09-17T03:52:51.058 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.061+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 msgr2=0x7f786407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.061+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7884071290 tx=0x7f7874005f90 comp rx=0 tx=0).stop 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.061+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 msgr2=0x7f7884151a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.061+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f787c009d60 tx=0x7f787c003ce0 comp rx=0 tx=0).stop 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 shutdown_connections 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7864078b90 0x7f786407b0a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78840cf6f0 0x7f7884151a30 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78840ce020 0x7f788414e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 --2- 172.21.15.129:0/378740038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78840ccb30 0x7f78841618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:51.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 >> 172.21.15.129:0/378740038 conn(0x7f78840b1cb0 msgr2=0x7f78840b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:51.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 shutdown_connections 2024-09-17T03:52:51.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51.062+0000 7f788b000640 1 -- 172.21.15.129:0/378740038 wait complete. 2024-09-17T03:52:51.085 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:52:51.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:52:51.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:52:51.085 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:52:51.085 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:51.086 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:52:51.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:52:51.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:52:51.086 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:51.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.087 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:51.123 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42521sec preferred_lft 42521sec 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:52:51.124 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:52:51.125 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:52:51.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.126 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:52:51.217 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:52:51.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.220 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:52:51.256 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:52:51.256 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:52:51.256 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:52:51.256 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:52:51.256 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:52:51.257 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:52:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:52:51.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:51.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.453 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:52:51.489 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:52:51.489 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:52:51.489 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:51.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:52:51.585 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:52:51.585 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:51.620 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:52:51.620 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:52:51.621 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:51.696 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:52:51.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:51.892 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:51.892 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:51.892 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:51.913 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:51.913 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:51.913 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:51.913 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:52:51.913 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:52:51.942 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:52:51.943 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:52:51.943 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:52:51.943 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:52:51.943 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:52:51.944 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:52:51.949 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:52:51.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:52:51.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:52:52.006 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:52:52.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:52:52.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:52:52.101 INFO:teuthology.orchestra.run.smithi129.stdout:client5884 2024-09-17T03:52:52.102 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:52:52.102 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:52:52.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 10 2024-09-17T03:52:52.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 -- 172.21.15.129:0/1679220054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7068142b00 msgr2=0x7f706815ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 --2- 172.21.15.129:0/1679220054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7068142b00 0x7f706815ca80 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f7050009830 tx=0x7f705002a510 comp rx=0 tx=0).stop 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 -- 172.21.15.129:0/1679220054 shutdown_connections 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 --2- 172.21.15.129:0/1679220054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70680ced60 0x7f70680cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 --2- 172.21.15.129:0/1679220054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7068142b00 0x7f706815ca80 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 -- 172.21.15.129:0/1679220054 >> 172.21.15.129:0/1679220054 conn(0x7f706806d410 msgr2=0x7f70680b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 -- 172.21.15.129:0/1679220054 shutdown_connections 2024-09-17T03:52:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.403+0000 7f7067e00640 1 -- 172.21.15.129:0/1679220054 wait complete. 2024-09-17T03:52:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.405+0000 7f7067e00640 1 Processor -- start 2024-09-17T03:52:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.406+0000 7f7067e00640 1 -- start start 2024-09-17T03:52:52.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:52.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7068142b00 0x7f7068073e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70681600c0 0x7f706813c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f706806f9c0 con 0x7f70680ced60 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f706806f840 con 0x7f70681600c0 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7067e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f706806fb40 con 0x7f7068142b00 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7066a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7066a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37140/0 (socket says 172.21.15.129:37140) 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.407+0000 7f7066a00640 1 -- 172.21.15.129:0/1299122952 learned_addr learned my addr 172.21.15.129:0/1299122952 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f705e000640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7068142b00 0x7f7068073e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7067400640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70681600c0 0x7f706813c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 -- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7068142b00 msgr2=0x7f7068073e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7068142b00 0x7f7068073e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 -- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70681600c0 msgr2=0x7f706813c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70681600c0 0x7f706813c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 -- 172.21.15.129:0/1299122952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70680ca750 con 0x7f70680ced60 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f7066a00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f705800c3a0 tx=0x7f705800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058011790 con 0x7f70680ced60 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7058017070 con 0x7f70680ced60 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.408+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7058011d20 con 0x7f70680ced60 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.409+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70680caa30 con 0x7f70680ced60 2024-09-17T03:52:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.409+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70680caff0 con 0x7f70680ced60 2024-09-17T03:52:52.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.410+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f706806ffd0 con 0x7f70680ced60 2024-09-17T03:52:52.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.411+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f705801c460 con 0x7f70680ced60 2024-09-17T03:52:52.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.412+0000 7f705f400640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 0x7f704407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:52:52.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.412+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f705809ab00 con 0x7f70680ced60 2024-09-17T03:52:52.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.412+0000 7f705e000640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 0x7f704407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:52:52.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.413+0000 7f705e000640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 0x7f704407b0a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7050034000 tx=0x7f7050004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:52:52.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.416+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7058062fd0 con 0x7f70680ced60 2024-09-17T03:52:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.603+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f70680b48b0 con 0x7f70680ced60 2024-09-17T03:52:52.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.607+0000 7f705f400640 1 -- 172.21.15.129:0/1299122952 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v42)=0 v42) ==== 124+0+0 (secure 0 0 0) 0x7f70580628d0 con 0x7f70680ced60 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 msgr2=0x7f704407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 0x7f704407b0a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7050034000 tx=0x7f7050004730 comp rx=0 tx=0).stop 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 msgr2=0x7f7068143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f705800c3a0 tx=0x7f705800c860 comp rx=0 tx=0).stop 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 shutdown_connections 2024-09-17T03:52:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7044078b90 0x7f704407b0a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70681600c0 0x7f706813c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7068142b00 0x7f7068073e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 --2- 172.21.15.129:0/1299122952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70680ced60 0x7f7068143090 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 >> 172.21.15.129:0/1299122952 conn(0x7f706806d410 msgr2=0x7f7068072c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 shutdown_connections 2024-09-17T03:52:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:52:52.610+0000 7f7067e00640 1 -- 172.21.15.129:0/1299122952 wait complete. 2024-09-17T03:53:02.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.861+0000 7f727a600640 1 -- 172.21.15.129:0/2962885959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070030 msgr2=0x7f7274074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.861+0000 7f727a600640 1 --2- 172.21.15.129:0/2962885959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070030 0x7f7274074640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7260009860 tx=0x7f726002a5a0 comp rx=0 tx=0).stop 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.861+0000 7f727a600640 1 -- 172.21.15.129:0/2962885959 shutdown_connections 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.861+0000 7f727a600640 1 --2- 172.21.15.129:0/2962885959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070030 0x7f7274074640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.861+0000 7f727a600640 1 -- 172.21.15.129:0/2962885959 >> 172.21.15.129:0/2962885959 conn(0x7f72740b1cb0 msgr2=0x7f72740b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.862+0000 7f727a600640 1 -- 172.21.15.129:0/2962885959 shutdown_connections 2024-09-17T03:53:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.862+0000 7f727a600640 1 -- 172.21.15.129:0/2962885959 wait complete. 2024-09-17T03:53:02.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.864+0000 7f727a600640 1 Processor -- start 2024-09-17T03:53:02.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.864+0000 7f727a600640 1 -- start start 2024-09-17T03:53:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070440 0x7f727409cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 0x7f727415e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72740b3cf0 con 0x7f7274073170 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f72740b3b70 con 0x7f7274070030 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.865+0000 7f727a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f72740b3e70 con 0x7f7274070440 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 0x7f727415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:36984/0 (socket says 172.21.15.129:36984) 2024-09-17T03:53:02.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273400640 1 -- 172.21.15.129:0/2456083657 learned_addr learned my addr 172.21.15.129:0/2456083657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:02.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7272a00640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070440 0x7f727409cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 -- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070440 msgr2=0x7f727409cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070440 0x7f727409cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:02.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 -- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 msgr2=0x7f7274148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273e00640 1 -- 172.21.15.129:0/2456083657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f727409d130 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.866+0000 7f7273400640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f7273e00640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 0x7f727415e290 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f726c00c390 tx=0x7f726c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c011790 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f726c017070 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f726c011ce0 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f727415e7d0 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.867+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f727409d600 con 0x7f7274073170 2024-09-17T03:53:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.868+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7274074640 con 0x7f7274073170 2024-09-17T03:53:02.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.869+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f726c01c460 con 0x7f7274073170 2024-09-17T03:53:02.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.870+0000 7f726b400640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 0x7f725407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:02.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.870+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f726c09a920 con 0x7f7274073170 2024-09-17T03:53:02.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.871+0000 7f7273400640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 0x7f725407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:02.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.872+0000 7f7273400640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 0x7f725407b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7264000f80 tx=0x7f7264009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:02.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:02.875+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f726c062720 con 0x7f7274073170 2024-09-17T03:53:03.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.098+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72740c3860 con 0x7f7274073170 2024-09-17T03:53:03.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.099+0000 7f726b400640 1 -- 172.21.15.129:0/2456083657 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f726c067150 con 0x7f7274073170 2024-09-17T03:53:03.099 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:03.099 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:03.099 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 msgr2=0x7f725407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 0x7f725407b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7264000f80 tx=0x7f7264009040 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 msgr2=0x7f727415e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 0x7f727415e290 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f726c00c390 tx=0x7f726c00c850 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 shutdown_connections 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7254078b90 0x7f725407b0a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7274073170 0x7f727415e290 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7274070440 0x7f727409cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 --2- 172.21.15.129:0/2456083657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7274070030 0x7f7274148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 >> 172.21.15.129:0/2456083657 conn(0x7f72740b1cb0 msgr2=0x7f72740d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.101+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 shutdown_connections 2024-09-17T03:53:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.102+0000 7f727a600640 1 -- 172.21.15.129:0/2456083657 wait complete. 2024-09-17T03:53:03.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:53:03.365 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":0}} 2024-09-17T03:53:03.366 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 0 } } 2024-09-17T03:53:03.366 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:03.366 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> n = 100 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T03:53:03.402 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T03:53:03.403 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T03:53:03.404 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:53:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:Traceback (most recent call last): 2024-09-17T03:53:03.591 INFO:teuthology.orchestra.run.smithi129.stderr: File "", line 26, in 2024-09-17T03:53:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/subdir/file_10' 2024-09-17T03:53:03.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-09-17T03:53:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:53:03.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:03.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 -- 172.21.15.129:0/1099774224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 msgr2=0x7fce4815ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:03.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 --2- 172.21.15.129:0/1099774224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce4815ef50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fce34005fd0 tx=0x7fce3402a530 comp rx=0 tx=0).stop 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 -- 172.21.15.129:0/1099774224 shutdown_connections 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 --2- 172.21.15.129:0/1099774224 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce4815ef50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 --2- 172.21.15.129:0/1099774224 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce4815fbd0 0x7fce48152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 -- 172.21.15.129:0/1099774224 >> 172.21.15.129:0/1099774224 conn(0x7fce4806d410 msgr2=0x7fce480b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 -- 172.21.15.129:0/1099774224 shutdown_connections 2024-09-17T03:53:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.866+0000 7fce4f000640 1 -- 172.21.15.129:0/1099774224 wait complete. 2024-09-17T03:53:03.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.868+0000 7fce4f000640 1 Processor -- start 2024-09-17T03:53:03.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.869+0000 7fce4f000640 1 -- start start 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 0x7fce4814ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 0x7fce4815d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce4815c070 con 0x7fce48072c30 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fce4815f7e0 con 0x7fce48073040 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fce4815c1f0 con 0x7fce48150350 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34960/0 (socket says 172.21.15.129:34960) 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.870+0000 7fce4dc00640 1 -- 172.21.15.129:0/620275182 learned_addr learned my addr 172.21.15.129:0/620275182 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4d200640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 0x7fce4814ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 -- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 msgr2=0x7fce4815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4e600640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 0x7fce4815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 0x7fce4815d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 -- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 msgr2=0x7fce4814ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 0x7fce4814ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 -- 172.21.15.129:0/620275182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4814b2d0 con 0x7fce48072c30 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4d200640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 0x7fce4814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4e600640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 0x7fce4815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:03.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4dc00640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d700 tx=0x7fce3c00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:03.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c017070 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce3c005310 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4814ca20 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c009820 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.871+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4814cfe0 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.873+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4815c750 con 0x7fce48072c30 2024-09-17T03:53:03.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.874+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fce3c0099c0 con 0x7fce48072c30 2024-09-17T03:53:03.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.875+0000 7fce45600640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 0x7fce2807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:03.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.875+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fce3c09ad60 con 0x7fce48072c30 2024-09-17T03:53:03.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.875+0000 7fce4d200640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 0x7fce2807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:03.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.876+0000 7fce4d200640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 0x7fce2807b0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fce34033000 tx=0x7fce34004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:03.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:03.879+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fce4814b2d0 con 0x7fce48072c30 2024-09-17T03:53:04.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.098+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce481f3570 con 0x7fce48072c30 2024-09-17T03:53:04.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.099+0000 7fce45600640 1 -- 172.21.15.129:0/620275182 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fce3c0632c0 con 0x7fce48072c30 2024-09-17T03:53:04.099 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:04.099 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:04.099 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 msgr2=0x7fce2807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 0x7fce2807b0a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fce34033000 tx=0x7fce34004730 comp rx=0 tx=0).stop 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 msgr2=0x7fce48149f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d700 tx=0x7fce3c00dbc0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 shutdown_connections 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce28078b90 0x7fce2807b0a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce48150350 0x7fce4815d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce48073040 0x7fce4814ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 --2- 172.21.15.129:0/620275182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce48072c30 0x7fce48149f70 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.101+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 >> 172.21.15.129:0/620275182 conn(0x7fce4806d410 msgr2=0x7fce48068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:04.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.102+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 shutdown_connections 2024-09-17T03:53:04.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.102+0000 7fce4f000640 1 -- 172.21.15.129:0/620275182 wait complete. 2024-09-17T03:53:04.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:53:04.364 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":10}} 2024-09-17T03:53:04.364 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 10 } } 2024-09-17T03:53:04.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:04.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 -- 172.21.15.129:0/4013014254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 msgr2=0x7f8cc400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 --2- 172.21.15.129:0/4013014254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8cc400b9c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8001d90 tx=0x7f8cc4005830 comp rx=0 tx=0).stop 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 -- 172.21.15.129:0/4013014254 shutdown_connections 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 --2- 172.21.15.129:0/4013014254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8cc400b9c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 -- 172.21.15.129:0/4013014254 >> 172.21.15.129:0/4013014254 conn(0x7f8ce00b1cb0 msgr2=0x7f8ce00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 -- 172.21.15.129:0/4013014254 shutdown_connections 2024-09-17T03:53:04.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.586+0000 7f8ce7200640 1 -- 172.21.15.129:0/4013014254 wait complete. 2024-09-17T03:53:04.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.589+0000 7f8ce7200640 1 Processor -- start 2024-09-17T03:53:04.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.589+0000 7f8ce7200640 1 -- start start 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 0x7f8ce00cb320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 0x7f8ce00c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce0074a80 con 0x7f8cc4005250 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8ce0074900 con 0x7f8ce00cb860 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8ce7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8ce0074c00 con 0x7f8ce00c98f0 2024-09-17T03:53:04.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8cdfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8cdfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34976/0 (socket says 172.21.15.129:34976) 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.590+0000 7f8cdfe00640 1 -- 172.21.15.129:0/2362478851 learned_addr learned my addr 172.21.15.129:0/2362478851 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8ce5200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 0x7f8ce00c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 -- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 msgr2=0x7f8ce00cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdf400640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 0x7f8ce00cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 0x7f8ce00cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 -- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 msgr2=0x7f8ce00c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 0x7f8ce00c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 -- 172.21.15.129:0/2362478851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ce00cbc70 con 0x7f8cc4005250 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8ce5200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 0x7f8ce00c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdf400640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 0x7f8ce00cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cdfe00640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0006680 tx=0x7f8cd00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:04.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0011700 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cd0016020 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ce015eb60 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.591+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cd0011b40 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.592+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ce015f120 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.593+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ce009dc90 con 0x7f8cc4005250 2024-09-17T03:53:04.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.594+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8cd0011ce0 con 0x7f8cc4005250 2024-09-17T03:53:04.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.595+0000 7f8cd7e00640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 0x7f8cc007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:04.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.595+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f8cd009b7c0 con 0x7f8cc4005250 2024-09-17T03:53:04.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.596+0000 7f8cdf400640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 0x7f8cc007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:04.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.596+0000 7f8cdf400640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 0x7f8cc007b0a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009460 tx=0x7f8ccc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:04.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.600+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8cd0020020 con 0x7f8cc4005250 2024-09-17T03:53:04.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.820+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ce00c3860 con 0x7f8cc4005250 2024-09-17T03:53:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.821+0000 7f8cd7e00640 1 -- 172.21.15.129:0/2362478851 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f8ce00cbc70 con 0x7f8cc4005250 2024-09-17T03:53:04.821 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:04.821 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 msgr2=0x7f8cc007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 0x7f8cc007b0a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc009460 tx=0x7f8ccc005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 msgr2=0x7f8ce00c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0006680 tx=0x7f8cd00039c0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 shutdown_connections 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cc0078b90 0x7f8cc007b0a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ce00cb860 0x7f8ce00c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ce00c98f0 0x7f8ce00cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 --2- 172.21.15.129:0/2362478851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc4005250 0x7f8ce00c93b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 >> 172.21.15.129:0/2362478851 conn(0x7f8ce00b1cb0 msgr2=0x7f8ce00acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 shutdown_connections 2024-09-17T03:53:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:04.823+0000 7f8ce7200640 1 -- 172.21.15.129:0/2362478851 wait complete. 2024-09-17T03:53:04.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:53:05.087 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T03:53:05.087 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T03:53:05.087 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-09-17T03:53:10.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:10.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 -- 172.21.15.129:0/1974129188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 msgr2=0x7fe2800040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:10.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 --2- 172.21.15.129:0/1974129188 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe2800040b0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe28c0017a0 tx=0x7fe280005830 comp rx=0 tx=0).stop 2024-09-17T03:53:10.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a1600640 1 -- 172.21.15.129:0/1974129188 reap_dead start 2024-09-17T03:53:10.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 -- 172.21.15.129:0/1974129188 shutdown_connections 2024-09-17T03:53:10.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 -- 172.21.15.129:0/1974129188 >> 172.21.15.129:0/1974129188 conn(0x7fe29c06d870 msgr2=0x7fe29c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:10.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 -- 172.21.15.129:0/1974129188 shutdown_connections 2024-09-17T03:53:10.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.321+0000 7fe2a2000640 1 -- 172.21.15.129:0/1974129188 wait complete. 2024-09-17T03:53:10.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.323+0000 7fe2a2000640 1 Processor -- start 2024-09-17T03:53:10.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.323+0000 7fe2a2000640 1 -- start start 2024-09-17T03:53:10.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:10.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 0x7fe29c1f8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:10.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 0x7fe29c1409c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:10.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c1642e0 con 0x7fe280003c60 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c164160 con 0x7fe29c15a7e0 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c164460 con 0x7fe29c1f8f50 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34998/0 (socket says 172.21.15.129:34998) 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 -- 172.21.15.129:0/3526487443 learned_addr learned my addr 172.21.15.129:0/3526487443 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe293e00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 0x7fe29c1f8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a1600640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 0x7fe29c1409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 -- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 msgr2=0x7fe29c1409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 0x7fe29c1409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 -- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 msgr2=0x7fe29c1f8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 0x7fe29c1f8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a0c00640 1 -- 172.21.15.129:0/3526487443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c0b4010 con 0x7fe280003c60 2024-09-17T03:53:10.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe293e00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 0x7fe29c1f8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.325+0000 7fe2a1600640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 0x7fe29c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe2a0c00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000f60 tx=0x7fe28c006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c043070 con 0x7fe280003c60 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe28c0038e0 con 0x7fe280003c60 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003d30 con 0x7fe280003c60 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c0b42f0 con 0x7fe280003c60 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.326+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c15af80 con 0x7fe280003c60 2024-09-17T03:53:10.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.327+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c15ba00 con 0x7fe280003c60 2024-09-17T03:53:10.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.328+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe28c033480 con 0x7fe280003c60 2024-09-17T03:53:10.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.329+0000 7fe299600640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 0x7fe27807b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:10.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.329+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fe28c0bacd0 con 0x7fe280003c60 2024-09-17T03:53:10.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.330+0000 7fe293e00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 0x7fe27807b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:10.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.330+0000 7fe293e00640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 0x7fe27807b100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe284001a30 tx=0x7fe284001e00 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:10.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.334+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe28c083100 con 0x7fe280003c60 2024-09-17T03:53:10.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.555+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe29c06f0a0 con 0x7fe280003c60 2024-09-17T03:53:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.556+0000 7fe299600640 1 -- 172.21.15.129:0/3526487443 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fe28c082a00 con 0x7fe280003c60 2024-09-17T03:53:10.556 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:10.556 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:10.556 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 msgr2=0x7fe27807b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 0x7fe27807b100 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe284001a30 tx=0x7fe284001e00 comp rx=0 tx=0).stop 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 msgr2=0x7fe29c1440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000f60 tx=0x7fe28c006940 comp rx=0 tx=0).stop 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 shutdown_connections 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe278078bf0 0x7fe27807b100 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe29c1f8f50 0x7fe29c1409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe29c15a7e0 0x7fe29c1f8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 --2- 172.21.15.129:0/3526487443 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe280003c60 0x7fe29c1440c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.558+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 >> 172.21.15.129:0/3526487443 conn(0x7fe29c06d870 msgr2=0x7fe29c1553a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.559+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 shutdown_connections 2024-09-17T03:53:10.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:10.559+0000 7fe2a2000640 1 -- 172.21.15.129:0/3526487443 wait complete. 2024-09-17T03:53:10.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:53:10.828 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T03:53:10.828 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T03:53:10.828 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 10, waiting (timeout=60)... 2024-09-17T03:53:15.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 -- 172.21.15.129:0/847205187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070030 msgr2=0x7fefa0074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 --2- 172.21.15.129:0/847205187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070030 0x7fefa0074640 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fef88005fd0 tx=0x7fef8802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 -- 172.21.15.129:0/847205187 shutdown_connections 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 --2- 172.21.15.129:0/847205187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070030 0x7fefa0074640 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 -- 172.21.15.129:0/847205187 >> 172.21.15.129:0/847205187 conn(0x7fefa00b1cb0 msgr2=0x7fefa00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 -- 172.21.15.129:0/847205187 shutdown_connections 2024-09-17T03:53:16.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.061+0000 7fefa4c00640 1 -- 172.21.15.129:0/847205187 wait complete. 2024-09-17T03:53:16.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.064+0000 7fefa4c00640 1 Processor -- start 2024-09-17T03:53:16.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.064+0000 7fefa4c00640 1 -- start start 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefa0070030 0x7fefa00cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefa0073170 0x7fefa01e94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefa00b3c40 con 0x7fefa0070440 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fefa00b3ac0 con 0x7fefa0073170 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fefa4c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fefa00b3dc0 con 0x7fefa0070030 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fef95400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fef95400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56742/0 (socket says 172.21.15.129:56742) 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.065+0000 7fef95400640 1 -- 172.21.15.129:0/46433783 learned_addr learned my addr 172.21.15.129:0/46433783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef9e800640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefa0073170 0x7fefa01e94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef9de00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefa0070030 0x7fefa00cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 -- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefa0070030 msgr2=0x7fefa00cbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefa0070030 0x7fefa00cbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 -- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefa0073170 msgr2=0x7fefa01e94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefa0073170 0x7fefa01e94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 -- 172.21.15.129:0/46433783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa015b410 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef95400640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fef88006010 tx=0x7fef880038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.066+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8803e030 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.067+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef88003ca0 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.067+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88035040 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.067+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa00cb650 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.067+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa015af30 con 0x7fefa0070440 2024-09-17T03:53:16.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.068+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa015b160 con 0x7fefa0070440 2024-09-17T03:53:16.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.069+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef88041030 con 0x7fefa0070440 2024-09-17T03:53:16.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.070+0000 7fef96a00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 0x7fef7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.070+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5767+0+0 (secure 0 0 0) 0x7fef880bb4f0 con 0x7fefa0070440 2024-09-17T03:53:16.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.070+0000 7fef9de00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 0x7fef7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.071+0000 7fef9de00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 0x7fef7c07b0a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef90009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:16.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.074+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef88083200 con 0x7fefa0070440 2024-09-17T03:53:16.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:53:16.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.293+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefa00b4320 con 0x7fefa0070440 2024-09-17T03:53:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.293+0000 7fef96a00640 1 -- 172.21.15.129:0/46433783 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fef88087c30 con 0x7fefa0070440 2024-09-17T03:53:16.294 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:16.294 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:16.294 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:16.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 msgr2=0x7fef7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 0x7fef7c07b0a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef90009060 comp rx=0 tx=0).stop 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 msgr2=0x7fefa0157370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fef88006010 tx=0x7fef880038a0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 shutdown_connections 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef7c078b90 0x7fef7c07b0a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefa0073170 0x7fefa01e94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefa0070440 0x7fefa0157370 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 --2- 172.21.15.129:0/46433783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefa0070030 0x7fefa00cbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 >> 172.21.15.129:0/46433783 conn(0x7fefa00b1cb0 msgr2=0x7fefa015f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 shutdown_connections 2024-09-17T03:53:16.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.296+0000 7fefa4c00640 1 -- 172.21.15.129:0/46433783 wait complete. 2024-09-17T03:53:16.320 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:53:16.599 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":10}} 2024-09-17T03:53:16.599 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 10 } } 2024-09-17T03:53:16.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.825+0000 7fbaf7e00640 1 -- 172.21.15.129:0/2097332596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80af700 msgr2=0x7fbaf80caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.825+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/2097332596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80af700 0x7fbaf80caa80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80752d0 tx=0x7fbae802a560 comp rx=0 tx=0).stop 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 -- 172.21.15.129:0/2097332596 shutdown_connections 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/2097332596 >> v2:172.21.15.181:3300/0 conn(0x7fbaf80b54d0 0x7fbaf80b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/2097332596 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80735d0 0x7fbaf8153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/2097332596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80af700 0x7fbaf80caa80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 -- 172.21.15.129:0/2097332596 >> 172.21.15.129:0/2097332596 conn(0x7fbaf806d870 msgr2=0x7fbaf806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:16.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 -- 172.21.15.129:0/2097332596 shutdown_connections 2024-09-17T03:53:16.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.826+0000 7fbaf7e00640 1 -- 172.21.15.129:0/2097332596 wait complete. 2024-09-17T03:53:16.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.828+0000 7fbaf7e00640 1 Processor -- start 2024-09-17T03:53:16.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.828+0000 7fbaf7e00640 1 -- start start 2024-09-17T03:53:16.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 0x7fbaf80c4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80b54d0 0x7fbaf814e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf815d1d0 con 0x7fbaf80735d0 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf815d050 con 0x7fbaf80af700 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.829+0000 7fbaf7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbaf815d350 con 0x7fbaf80b54d0 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56758/0 (socket says 172.21.15.129:56758) 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 -- 172.21.15.129:0/3568018749 learned_addr learned my addr 172.21.15.129:0/3568018749 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaee000640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 0x7fbaf80c4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf7400640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80b54d0 0x7fbaf814e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 -- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80b54d0 msgr2=0x7fbaf814e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80b54d0 0x7fbaf814e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 -- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 msgr2=0x7fbaf80c4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:16.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 0x7fbaf80c4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:16.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaf6a00640 1 -- 172.21.15.129:0/3568018749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaf81f7170 con 0x7fbaf80735d0 2024-09-17T03:53:16.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.830+0000 7fbaee000640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 0x7fbaf80c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:16.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaf6a00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbae8034040 tx=0x7fbae8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:16.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae803e030 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbae8003e00 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaf81f7450 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae8035040 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.831+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaf81f7a10 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.832+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaf8155770 con 0x7fbaf80735d0 2024-09-17T03:53:16.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.833+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbae8041030 con 0x7fbaf80735d0 2024-09-17T03:53:16.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.834+0000 7fbaef400640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 0x7fbad407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:16.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.834+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7fbae80bb5a0 con 0x7fbaf80735d0 2024-09-17T03:53:16.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.835+0000 7fbaee000640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 0x7fbad407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:16.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.835+0000 7fbaee000640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 0x7fbad407b0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbaf815e6f0 tx=0x7fbae0005e30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:16.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:16.839+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbae8083990 con 0x7fbaf80735d0 2024-09-17T03:53:17.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.067+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaf815e030 con 0x7fbaf80735d0 2024-09-17T03:53:17.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.068+0000 7fbaef400640 1 -- 172.21.15.129:0/3568018749 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fbae8083290 con 0x7fbaf80735d0 2024-09-17T03:53:17.068 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:17.068 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:17.068 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 msgr2=0x7fbad407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 0x7fbad407b0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbaf815e6f0 tx=0x7fbae0005e30 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 msgr2=0x7fbaf81551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbae8034040 tx=0x7fbae8003910 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 shutdown_connections 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbad4078bc0 0x7fbad407b0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbaf80b54d0 0x7fbaf814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbaf80af700 0x7fbaf80c4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 --2- 172.21.15.129:0/3568018749 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbaf80735d0 0x7fbaf81551e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 >> 172.21.15.129:0/3568018749 conn(0x7fbaf806d870 msgr2=0x7fbaf815e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.070+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 shutdown_connections 2024-09-17T03:53:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17.071+0000 7fbaf7e00640 1 -- 172.21.15.129:0/3568018749 wait complete. 2024-09-17T03:53:17.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-09-17T03:53:17.343 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":10}} 2024-09-17T03:53:17.343 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 10 } } 2024-09-17T03:53:17.343 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:17.344 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:17.379 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> n = 10 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/subdir/file" 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T03:53:17.380 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> if True: 2024-09-17T03:53:17.381 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T03:53:17.382 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:53:17.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:53:17.586 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:53:17.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:17.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:17.806 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:53:17.806 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:17.806 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:17.807 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:17.842 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:17.842 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:17.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 -- 172.21.15.129:0/504831830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 msgr2=0x7fd548142b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 --2- 172.21.15.129:0/504831830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd548142b00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd5480cc690 tx=0x7fd53802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 -- 172.21.15.129:0/504831830 shutdown_connections 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 --2- 172.21.15.129:0/504831830 >> v2:172.21.15.181:3300/0 conn(0x7fd548072c30 0x7fd54815ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 --2- 172.21.15.129:0/504831830 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd548142b00 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 -- 172.21.15.129:0/504831830 >> 172.21.15.129:0/504831830 conn(0x7fd54806d410 msgr2=0x7fd5480b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:18.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 -- 172.21.15.129:0/504831830 shutdown_connections 2024-09-17T03:53:18.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.179+0000 7fd54e600640 1 -- 172.21.15.129:0/504831830 wait complete. 2024-09-17T03:53:18.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.181+0000 7fd54e600640 1 Processor -- start 2024-09-17T03:53:18.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.181+0000 7fd54e600640 1 -- start start 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd548072c30 0x7fd5480cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5480ced60 0x7fd5481f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd54806f9c0 con 0x7fd5481600c0 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd54806f840 con 0x7fd5480ced60 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd54806fb40 con 0x7fd548072c30 2024-09-17T03:53:18.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56782/0 (socket says 172.21.15.129:56782) 2024-09-17T03:53:18.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54dc00640 1 -- 172.21.15.129:0/2207240194 learned_addr learned my addr 172.21.15.129:0/2207240194 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:18.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd53fe00640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5480ced60 0x7fd5481f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54d200640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd548072c30 0x7fd5480cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54dc00640 1 -- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd548072c30 msgr2=0x7fd5480cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.183+0000 7fd54dc00640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd548072c30 0x7fd5480cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54dc00640 1 -- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5480ced60 msgr2=0x7fd5481f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54dc00640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5480ced60 0x7fd5481f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54dc00640 1 -- 172.21.15.129:0/2207240194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5480b0240 con 0x7fd5481600c0 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54dc00640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c730 tx=0x7fd54000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd540016020 con 0x7fd5481600c0 2024-09-17T03:53:18.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd540003990 con 0x7fd5481600c0 2024-09-17T03:53:18.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd540006e60 con 0x7fd5481600c0 2024-09-17T03:53:18.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5480b0520 con 0x7fd5481600c0 2024-09-17T03:53:18.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.184+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5480b0ae0 con 0x7fd5481600c0 2024-09-17T03:53:18.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.185+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd54806a740 con 0x7fd5481600c0 2024-09-17T03:53:18.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.186+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd5400050d0 con 0x7fd5481600c0 2024-09-17T03:53:18.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.187+0000 7fd545600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 0x7fd52407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.188+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7fd54009aa00 con 0x7fd5481600c0 2024-09-17T03:53:18.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.188+0000 7fd54d200640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 0x7fd52407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.189+0000 7fd54d200640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 0x7fd52407b0a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd538033000 tx=0x7fd538004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:18.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.192+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd54806a740 con 0x7fd5481600c0 2024-09-17T03:53:18.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.386+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd548070060 con 0x7fd5481600c0 2024-09-17T03:53:18.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.387+0000 7fd545600640 1 -- 172.21.15.129:0/2207240194 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) ==== 80+0+0 (secure 0 0 0) 0x7fd540020020 con 0x7fd5481600c0 2024-09-17T03:53:18.387 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:53:18.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 msgr2=0x7fd52407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 0x7fd52407b0a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd538033000 tx=0x7fd538004730 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 msgr2=0x7fd54813f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd54000c730 tx=0x7fd54000cb30 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 shutdown_connections 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd524078b90 0x7fd52407b0a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5481600c0 0x7fd54813f6f0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5480ced60 0x7fd5481f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 --2- 172.21.15.129:0/2207240194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd548072c30 0x7fd5480cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 >> 172.21.15.129:0/2207240194 conn(0x7fd54806d410 msgr2=0x7fd5480732c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.389+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 shutdown_connections 2024-09-17T03:53:18.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.390+0000 7fd54e600640 1 -- 172.21.15.129:0/2207240194 wait complete. 2024-09-17T03:53:18.414 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:53:18.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:18.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:18.514 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:53:18.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:18.514 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:53:18.558 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:18.558 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:18.558 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:18.596 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:18.596 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:18.596 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:18.596 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:18.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:53:18.633 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:18.633 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:18.633 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:18.672 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:18.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 -- 172.21.15.129:0/63306980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99806ee90 msgr2=0x7fc9980b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 --2- 172.21.15.129:0/63306980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99806ee90 0x7fc9980b47f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc988005fd0 tx=0x7fc98802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 -- 172.21.15.129:0/63306980 shutdown_connections 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 --2- 172.21.15.129:0/63306980 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9980b4d30 0x7fc9980ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 --2- 172.21.15.129:0/63306980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99806ee90 0x7fc9980b47f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 -- 172.21.15.129:0/63306980 >> 172.21.15.129:0/63306980 conn(0x7fc99806d7d0 msgr2=0x7fc9980b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 -- 172.21.15.129:0/63306980 shutdown_connections 2024-09-17T03:53:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.894+0000 7fc99e000640 1 -- 172.21.15.129:0/63306980 wait complete. 2024-09-17T03:53:18.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.897+0000 7fc99e000640 1 Processor -- start 2024-09-17T03:53:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.897+0000 7fc99e000640 1 -- start start 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc99806ee90 0x7fc9980aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9980b4d30 0x7fc99815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc99815feb0 con 0x7fc99806f2a0 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc99815fd30 con 0x7fc9980b4d30 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc99e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9980cd050 con 0x7fc99806ee90 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.898+0000 7fc996a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56788/0 (socket says 172.21.15.129:56788) 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 -- 172.21.15.129:0/3095855420 learned_addr learned my addr 172.21.15.129:0/3095855420 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc997400640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc99806ee90 0x7fc9980aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc997e00640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9980b4d30 0x7fc99815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 -- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc99806ee90 msgr2=0x7fc9980aebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc99806ee90 0x7fc9980aebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 -- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9980b4d30 msgr2=0x7fc99815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9980b4d30 0x7fc99815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 -- 172.21.15.129:0/3095855420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9980adb20 con 0x7fc99806f2a0 2024-09-17T03:53:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc996a00640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ea20 tx=0x7fc98400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.899+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98400ce40 con 0x7fc99806f2a0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.900+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc984004020 con 0x7fc99806f2a0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.900+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9980ade00 con 0x7fc99806f2a0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.900+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984004470 con 0x7fc99806f2a0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.900+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9980ae3c0 con 0x7fc99806f2a0 2024-09-17T03:53:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.901+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9980b47f0 con 0x7fc99806f2a0 2024-09-17T03:53:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.902+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9840035f0 con 0x7fc99806f2a0 2024-09-17T03:53:18.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.903+0000 7fc98f400640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 0x7fc97807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:18.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.903+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7fc98409b760 con 0x7fc99806f2a0 2024-09-17T03:53:18.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.903+0000 7fc997400640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 0x7fc97807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:18.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.904+0000 7fc997400640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 0x7fc97807b0a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc988033000 tx=0x7fc988004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:18.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:18.907+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc984063500 con 0x7fc99806f2a0 2024-09-17T03:53:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.132+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9980750f0 con 0x7fc99806f2a0 2024-09-17T03:53:19.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.133+0000 7fc98f400640 1 -- 172.21.15.129:0/3095855420 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7fc984067f30 con 0x7fc99806f2a0 2024-09-17T03:53:19.133 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:19.134 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:19.134 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 msgr2=0x7fc97807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 0x7fc97807b0a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc988033000 tx=0x7fc988004730 comp rx=0 tx=0).stop 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 msgr2=0x7fc998073ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fc98400ea20 tx=0x7fc98400eee0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 shutdown_connections 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc978078b90 0x7fc97807b0a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9980b4d30 0x7fc99815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99806f2a0 0x7fc998073ef0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 --2- 172.21.15.129:0/3095855420 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc99806ee90 0x7fc9980aebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.135+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 >> 172.21.15.129:0/3095855420 conn(0x7fc99806d7d0 msgr2=0x7fc99815e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.136+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 shutdown_connections 2024-09-17T03:53:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.136+0000 7fc99e000640 1 -- 172.21.15.129:0/3095855420 wait complete. 2024-09-17T03:53:19.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 -- 172.21.15.129:0/1406878562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cd240 msgr2=0x7f0a540cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 --2- 172.21.15.129:0/1406878562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cd240 0x7f0a540cc4c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0a5409ded0 tx=0x7f0a4002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 -- 172.21.15.129:0/1406878562 shutdown_connections 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 --2- 172.21.15.129:0/1406878562 >> v2:172.21.15.181:3300/0 conn(0x7f0a540ce020 0x7f0a540ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 --2- 172.21.15.129:0/1406878562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cd240 0x7f0a540cc4c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 -- 172.21.15.129:0/1406878562 >> 172.21.15.129:0/1406878562 conn(0x7f0a540b1cb0 msgr2=0x7f0a540b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:19.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 -- 172.21.15.129:0/1406878562 shutdown_connections 2024-09-17T03:53:19.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.380+0000 7f0a59800640 1 -- 172.21.15.129:0/1406878562 wait complete. 2024-09-17T03:53:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.382+0000 7f0a59800640 1 Processor -- start 2024-09-17T03:53:19.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.382+0000 7f0a59800640 1 -- start start 2024-09-17T03:53:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.383+0000 7f0a59800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a540ce020 0x7f0a541f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.383+0000 7f0a59800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.383+0000 7f0a59800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a540acf90 0x7f0a540ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a59800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540701b0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a59800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540cf3d0 con 0x7f0a540ce020 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a59800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0a54070330 con 0x7f0a540acf90 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56800/0 (socket says 172.21.15.129:56800) 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 -- 172.21.15.129:0/1279176770 learned_addr learned my addr 172.21.15.129:0/1279176770 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52a00640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a540ce020 0x7f0a541f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a53400640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a540acf90 0x7f0a540ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 -- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a540acf90 msgr2=0x7f0a540ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a540acf90 0x7f0a540ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 -- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a540ce020 msgr2=0x7f0a541f5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a540ce020 0x7f0a541f5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.384+0000 7f0a52000640 1 -- 172.21.15.129:0/1279176770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540ad8e0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a52000640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0a400099c0 tx=0x7f0a400039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4003e030 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a400047a0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40035040 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a541f87a0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.385+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a541f8d60 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.386+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a540ceae0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.387+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0a40041030 con 0x7f0a540cf6f0 2024-09-17T03:53:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.388+0000 7f0a4aa00640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 0x7f0a3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.388+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f0a400bb4c0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.389+0000 7f0a52a00640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 0x7f0a3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.390+0000 7f0a52a00640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 0x7f0a3407b0a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0a440012e0 tx=0x7f0a44009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:19.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.393+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a540ad8e0 con 0x7f0a540cf6f0 2024-09-17T03:53:19.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.612+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a540afe50 con 0x7f0a540cf6f0 2024-09-17T03:53:19.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.613+0000 7f0a4aa00640 1 -- 172.21.15.129:0/1279176770 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f0a40088e20 con 0x7f0a540cf6f0 2024-09-17T03:53:19.613 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:19.613 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:19.614 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 msgr2=0x7f0a3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 0x7f0a3407b0a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0a440012e0 tx=0x7f0a44009040 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 msgr2=0x7f0a541f57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0a400099c0 tx=0x7f0a400039e0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 shutdown_connections 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.615+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a34078b90 0x7f0a3407b0a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a540acf90 0x7f0a540ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a540cf6f0 0x7f0a541f57d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 --2- 172.21.15.129:0/1279176770 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a540ce020 0x7f0a541f5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 >> 172.21.15.129:0/1279176770 conn(0x7f0a540b1cb0 msgr2=0x7f0a540738a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 shutdown_connections 2024-09-17T03:53:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.616+0000 7f0a59800640 1 -- 172.21.15.129:0/1279176770 wait complete. 2024-09-17T03:53:19.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 -- 172.21.15.129:0/3117962054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc40cea10 msgr2=0x7fddc415c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3117962054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc40cea10 0x7fddc415c340 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fddb4005fd0 tx=0x7fddb402a540 comp rx=0 tx=0).stop 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 -- 172.21.15.129:0/3117962054 shutdown_connections 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3117962054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc40b4690 0x7fddc40b4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3117962054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc40cea10 0x7fddc415c340 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 -- 172.21.15.129:0/3117962054 >> 172.21.15.129:0/3117962054 conn(0x7fddc40cd6c0 msgr2=0x7fddc40cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 -- 172.21.15.129:0/3117962054 shutdown_connections 2024-09-17T03:53:19.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.859+0000 7fddc8a00640 1 -- 172.21.15.129:0/3117962054 wait complete. 2024-09-17T03:53:19.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.861+0000 7fddc8a00640 1 Processor -- start 2024-09-17T03:53:19.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.862+0000 7fddc8a00640 1 -- start start 2024-09-17T03:53:19.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc40b4690 0x7fddc41f8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 0x7fddc41f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fddc40b5370 con 0x7fddc4073930 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fddc40b51f0 con 0x7fddc40cea10 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fddc40b54f0 con 0x7fddc40b4690 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56818/0 (socket says 172.21.15.129:56818) 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.863+0000 7fddc2800640 1 -- 172.21.15.129:0/3921483565 learned_addr learned my addr 172.21.15.129:0/3921483565 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 -- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc40b4690 msgr2=0x7fddc41f8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc40b4690 0x7fddc41f8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 -- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 msgr2=0x7fddc41f91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:19.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc3200640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 0x7fddc41f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 0x7fddc41f91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:19.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 -- 172.21.15.129:0/3921483565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc4159130 con 0x7fddc4073930 2024-09-17T03:53:19.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc3200640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 0x7fddc41f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc2800640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006010 tx=0x7fddb40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb403e030 con 0x7fddc4073930 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddb4003d90 con 0x7fddc4073930 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc40c5a10 con 0x7fddc4073930 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.864+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4035040 con 0x7fddc4073930 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.865+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc40b04a0 con 0x7fddc4073930 2024-09-17T03:53:19.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.866+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc40b4aa0 con 0x7fddc4073930 2024-09-17T03:53:19.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.867+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fddb4041030 con 0x7fddc4073930 2024-09-17T03:53:19.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.868+0000 7fddbb400640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 0x7fdda007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:19.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.868+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7fddb40bb540 con 0x7fddc4073930 2024-09-17T03:53:19.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.868+0000 7fddb9e00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 0x7fdda007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:19.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.869+0000 7fddb9e00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 0x7fdda007b0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddac000f60 tx=0x7fddac0079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:19.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:19.872+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fddb4083940 con 0x7fddc4073930 2024-09-17T03:53:20.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.057+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddc406e000 con 0x7fddc4073930 2024-09-17T03:53:20.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.058+0000 7fddbb400640 1 -- 172.21.15.129:0/3921483565 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16874 (secure 0 0 0) 0x7fddb4083240 con 0x7fddc4073930 2024-09-17T03:53:20.058 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:20.059 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":70,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:16.668197+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:52:36.386165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-17T03:52:37.439087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2d","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.060+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 msgr2=0x7fdda007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.060+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 0x7fdda007b0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fddac000f60 tx=0x7fddac0079b0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.060+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 msgr2=0x7fddc41f6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.060+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fddb4006010 tx=0x7fddb40038a0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 shutdown_connections 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda0078bc0 0x7fdda007b0d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc40cea10 0x7fddc41f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc40b4690 0x7fddc41f8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 --2- 172.21.15.129:0/3921483565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc4073930 0x7fddc41f6100 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 >> 172.21.15.129:0/3921483565 conn(0x7fddc40cd6c0 msgr2=0x7fddc40b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 shutdown_connections 2024-09-17T03:53:20.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.061+0000 7fddc8a00640 1 -- 172.21.15.129:0/3921483565 wait complete. 2024-09-17T03:53:20.087 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:53:20.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.309+0000 7f4552000640 1 -- 172.21.15.129:0/1364386738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c0738b0 msgr2=0x7f454c167680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.309+0000 7f4552000640 1 --2- 172.21.15.129:0/1364386738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c0738b0 0x7f454c167680 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f4544005fd0 tx=0x7f454402aa10 comp rx=0 tx=0).stop 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.310+0000 7f4552000640 1 -- 172.21.15.129:0/1364386738 shutdown_connections 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.310+0000 7f4552000640 1 --2- 172.21.15.129:0/1364386738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c0738b0 0x7f454c167680 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.310+0000 7f4552000640 1 -- 172.21.15.129:0/1364386738 >> 172.21.15.129:0/1364386738 conn(0x7f454c06cde0 msgr2=0x7f454c06d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.310+0000 7f4552000640 1 -- 172.21.15.129:0/1364386738 shutdown_connections 2024-09-17T03:53:20.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.310+0000 7f4552000640 1 -- 172.21.15.129:0/1364386738 wait complete. 2024-09-17T03:53:20.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.312+0000 7f4552000640 1 Processor -- start 2024-09-17T03:53:20.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.312+0000 7f4552000640 1 -- start start 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f454c06fad0 0x7f454c1f9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f454c156c50 0x7f454c13a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f454c167f50 con 0x7f454c1f65b0 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f454c167dd0 con 0x7f454c06fad0 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.313+0000 7f4552000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f454c1680d0 con 0x7f454c156c50 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f454c06fad0 0x7f454c1f9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56838/0 (socket says 172.21.15.129:56838) 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 -- 172.21.15.129:0/4170952685 learned_addr learned my addr 172.21.15.129:0/4170952685 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454aa00640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f454c156c50 0x7f454c13a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 -- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f454c156c50 msgr2=0x7f454c13a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f454c156c50 0x7f454c13a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 -- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f454c06fad0 msgr2=0x7f454c1f9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f454c06fad0 0x7f454c1f9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.314+0000 7f454be00640 1 -- 172.21.15.129:0/4170952685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454c1f9e70 con 0x7f454c1f65b0 2024-09-17T03:53:20.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f454be00640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4544006010 tx=0x7f4544003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544046070 con 0x7f454c1f65b0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45440045c0 con 0x7f454c1f65b0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4544033070 con 0x7f454c1f65b0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454c15cc70 con 0x7f454c1f65b0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.315+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f454c1f5ad0 con 0x7f454c1f65b0 2024-09-17T03:53:20.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.316+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454c15d780 con 0x7f454c1f65b0 2024-09-17T03:53:20.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.317+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4544049070 con 0x7f454c1f65b0 2024-09-17T03:53:20.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.318+0000 7f4543400640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 0x7f452c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.318+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f45440bbff0 con 0x7f454c1f65b0 2024-09-17T03:53:20.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.319+0000 7f454b400640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 0x7f452c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.319+0000 7f454b400640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 0x7f452c07b0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f453c006fd0 tx=0x7f453c009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:20.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.323+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f454c1f9e70 con 0x7f454c1f65b0 2024-09-17T03:53:20.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.539+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f454c0ca890 con 0x7f454c1f65b0 2024-09-17T03:53:20.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.540+0000 7f4543400640 1 -- 172.21.15.129:0/4170952685 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f454403e070 con 0x7f454c1f65b0 2024-09-17T03:53:20.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:20.540 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:20.541 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.542+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 msgr2=0x7f452c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.542+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 0x7f452c07b0a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f453c006fd0 tx=0x7f453c009060 comp rx=0 tx=0).stop 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.542+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 msgr2=0x7f454c15b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.542+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4544006010 tx=0x7f4544003f10 comp rx=0 tx=0).stop 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 shutdown_connections 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f452c078b90 0x7f452c07b0a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f454c1f65b0 0x7f454c15b7d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f454c156c50 0x7f454c13a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 --2- 172.21.15.129:0/4170952685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f454c06fad0 0x7f454c1f9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 >> 172.21.15.129:0/4170952685 conn(0x7f454c06cde0 msgr2=0x7f454c1511d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:20.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 shutdown_connections 2024-09-17T03:53:20.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.543+0000 7f4552000640 1 -- 172.21.15.129:0/4170952685 wait complete. 2024-09-17T03:53:20.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:20.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 -- 172.21.15.129:0/3119578103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40b40a0 msgr2=0x7f6da4167450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 --2- 172.21.15.129:0/3119578103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40b40a0 0x7f6da4167450 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c005fd0 tx=0x7f6d9c02aa90 comp rx=0 tx=0).stop 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 -- 172.21.15.129:0/3119578103 shutdown_connections 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 --2- 172.21.15.129:0/3119578103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40b40a0 0x7f6da4167450 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 -- 172.21.15.129:0/3119578103 >> 172.21.15.129:0/3119578103 conn(0x7f6da40b1cb0 msgr2=0x7f6da40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 -- 172.21.15.129:0/3119578103 shutdown_connections 2024-09-17T03:53:20.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.789+0000 7f6dab000640 1 -- 172.21.15.129:0/3119578103 wait complete. 2024-09-17T03:53:20.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.791+0000 7f6dab000640 1 Processor -- start 2024-09-17T03:53:20.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.792+0000 7f6dab000640 1 -- start start 2024-09-17T03:53:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 0x7f6da415b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da40b40a0 0x7f6da4156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4167ba0 con 0x7f6da4070030 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4167d20 con 0x7f6da4073170 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6dab000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6da4167ea0 con 0x7f6da40b40a0 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6da3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6da3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56856/0 (socket says 172.21.15.129:56856) 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.793+0000 7f6da3e00640 1 -- 172.21.15.129:0/254852430 learned_addr learned my addr 172.21.15.129:0/254852430 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6d9be00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 0x7f6da415b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da9000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da40b40a0 0x7f6da4156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 -- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da40b40a0 msgr2=0x7f6da4156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da40b40a0 0x7f6da4156430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 -- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 msgr2=0x7f6da415b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:20.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 0x7f6da415b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 -- 172.21.15.129:0/254852430 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da415b9f0 con 0x7f6da4070030 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6d9be00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 0x7f6da415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da3e00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c3a0 tx=0x7f6d9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94017070 con 0x7f6da4070030 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d940053c0 con 0x7f6da4070030 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da41f3cb0 con 0x7f6da4070030 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94003850 con 0x7f6da4070030 2024-09-17T03:53:20.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.794+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41f4270 con 0x7f6da4070030 2024-09-17T03:53:20.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.795+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da4074640 con 0x7f6da4070030 2024-09-17T03:53:20.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.797+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6d9401a070 con 0x7f6da4070030 2024-09-17T03:53:20.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.798+0000 7f6da0c00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 0x7f6d8007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:20.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.798+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f6d9409a0c0 con 0x7f6da4070030 2024-09-17T03:53:20.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.798+0000 7f6d9be00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 0x7f6d8007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:20.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.799+0000 7f6d9be00640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 0x7f6d8007b0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6da4151d30 tx=0x7f6d8c008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:20.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:20.802+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6d940624b0 con 0x7f6da4070030 2024-09-17T03:53:21.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.019+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6da40c3860 con 0x7f6da4070030 2024-09-17T03:53:21.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.020+0000 7f6da0c00640 1 -- 172.21.15.129:0/254852430 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5237 (secure 0 0 0) 0x7f6d94061db0 con 0x7f6da4070030 2024-09-17T03:53:21.020 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:21.020 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":45,"btime":"2024-09-17T03:52:42:012384+0000","default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:52:38.484197+0000","modified":"2024-09-17T03:52:41.494978+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5865},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5865":{"gid":5865,"name":"c","rank":0,"incarnation":41,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/2480264914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2480264914},{"type":"v1","addr":"172.21.15.129:6835","nonce":2480264914}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5865,"qdb_cluster":[5865]},"id":5}]} 2024-09-17T03:53:21.021 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 45 2024-09-17T03:53:21.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 msgr2=0x7f6d8007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 0x7f6d8007b0d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6da4151d30 tx=0x7f6d8c008040 comp rx=0 tx=0).stop 2024-09-17T03:53:21.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 msgr2=0x7f6da414f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c3a0 tx=0x7f6d9400c860 comp rx=0 tx=0).stop 2024-09-17T03:53:21.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 shutdown_connections 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d80078bc0 0x7f6d8007b0d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da40b40a0 0x7f6da4156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da4073170 0x7f6da415b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 --2- 172.21.15.129:0/254852430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da4070030 0x7f6da414f280 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 >> 172.21.15.129:0/254852430 conn(0x7f6da40b1cb0 msgr2=0x7f6da40b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 shutdown_connections 2024-09-17T03:53:21.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.023+0000 7f6dab000640 1 -- 172.21.15.129:0/254852430 wait complete. 2024-09-17T03:53:21.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.269+0000 7f1514e00640 1 -- 172.21.15.129:0/2540871227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f4004800 msgr2=0x7f14f4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.269+0000 7f1514e00640 1 --2- 172.21.15.129:0/2540871227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14f4004800 0x7f14f4004c10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f1510072320 tx=0x7f14f4005830 comp rx=0 tx=0).stop 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.270+0000 7f150e800640 1 -- 172.21.15.129:0/2540871227 reap_dead start 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.270+0000 7f1514e00640 1 -- 172.21.15.129:0/2540871227 shutdown_connections 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.270+0000 7f1514e00640 1 -- 172.21.15.129:0/2540871227 >> 172.21.15.129:0/2540871227 conn(0x7f15100b1cb0 msgr2=0x7f15100b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.270+0000 7f1514e00640 1 -- 172.21.15.129:0/2540871227 shutdown_connections 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.270+0000 7f1514e00640 1 -- 172.21.15.129:0/2540871227 wait complete. 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.272+0000 7f1514e00640 1 Processor -- start 2024-09-17T03:53:21.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.272+0000 7f1514e00640 1 -- start start 2024-09-17T03:53:21.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f4004800 0x7f15100cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 0x7f15101545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f151006faf0 con 0x7f15101f98e0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f151006f970 con 0x7f14f4004800 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f1514e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f151006fc70 con 0x7f15101541d0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56874/0 (socket says 172.21.15.129:56874) 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 -- 172.21.15.129:0/1239895194 learned_addr learned my addr 172.21.15.129:0/1239895194 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150de00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f4004800 0x7f15100cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150e800640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 0x7f15101545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 -- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 msgr2=0x7f15101545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 0x7f15101545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 -- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f4004800 msgr2=0x7f15100cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f4004800 0x7f15100cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150d400640 1 -- 172.21.15.129:0/1239895194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151009aad0 con 0x7f15101f98e0 2024-09-17T03:53:21.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.274+0000 7f150e800640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 0x7f15101545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f150d400640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f14fc036040 tx=0x7f14fc033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc038040 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14fc004020 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151009adb0 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc004450 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.275+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15100c7e90 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.276+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151015c780 con 0x7f15101f98e0 2024-09-17T03:53:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.277+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14fc045020 con 0x7f15101f98e0 2024-09-17T03:53:21.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.278+0000 7f1506000640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 0x7f14f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.278+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f14fc0bb7a0 con 0x7f15101f98e0 2024-09-17T03:53:21.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.279+0000 7f150de00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 0x7f14f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.280+0000 7f150de00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 0x7f14f007b0a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1500009b90 tx=0x7f1500005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:21.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.283+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14fc083b00 con 0x7f15101f98e0 2024-09-17T03:53:21.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.465+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15100c3860 con 0x7f15101f98e0 2024-09-17T03:53:21.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.467+0000 7f1506000640 1 -- 172.21.15.129:0/1239895194 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) ==== 74+0+16874 (secure 0 0 0) 0x7f14fc083400 con 0x7f15101f98e0 2024-09-17T03:53:21.467 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:21.467 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":70,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:16.668197+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:52:36.386165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-17T03:52:37.439087+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.2d","mappings":[{"from":6,"to":0}]},{"pgid":"11.30","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:21.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 msgr2=0x7f14f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 0x7f14f007b0a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1500009b90 tx=0x7f1500005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:21.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 msgr2=0x7f15100d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f14fc036040 tx=0x7f14fc033670 comp rx=0 tx=0).stop 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 shutdown_connections 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14f0078b90 0x7f14f007b0a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15101541d0 0x7f15101545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15101f98e0 0x7f15100d0490 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 --2- 172.21.15.129:0/1239895194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14f4004800 0x7f15100cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 >> 172.21.15.129:0/1239895194 conn(0x7f15100b1cb0 msgr2=0x7f15100b13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 shutdown_connections 2024-09-17T03:53:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.469+0000 7f1514e00640 1 -- 172.21.15.129:0/1239895194 wait complete. 2024-09-17T03:53:21.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:53:21.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 -- 172.21.15.129:0/2469107640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 msgr2=0x7f68f80b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 --2- 172.21.15.129:0/2469107640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f80b3db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f68e80098e0 tx=0x7f68e802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 -- 172.21.15.129:0/2469107640 shutdown_connections 2024-09-17T03:53:21.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 --2- 172.21.15.129:0/2469107640 >> v2:172.21.15.181:3300/0 conn(0x7f68f80b8500 0x7f68f80b8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 --2- 172.21.15.129:0/2469107640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f80b3db0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 -- 172.21.15.129:0/2469107640 >> 172.21.15.129:0/2469107640 conn(0x7f68f808bfc0 msgr2=0x7f68f809d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:21.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 -- 172.21.15.129:0/2469107640 shutdown_connections 2024-09-17T03:53:21.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.713+0000 7f68fd200640 1 -- 172.21.15.129:0/2469107640 wait complete. 2024-09-17T03:53:21.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.715+0000 7f68fd200640 1 Processor -- start 2024-09-17T03:53:21.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.715+0000 7f68fd200640 1 -- start start 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f81d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f80d43c0 0x7f68f806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68f80b9290 con 0x7f68f80b8500 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f68f80b9110 con 0x7f68f80d43c0 2024-09-17T03:53:21.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68fd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f68f80b9410 con 0x7f68f80b39a0 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56882/0 (socket says 172.21.15.129:56882) 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5400640 1 -- 172.21.15.129:0/3413644626 learned_addr learned my addr 172.21.15.129:0/3413644626 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5e00640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f81d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f6800640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f80d43c0 0x7f68f806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5400640 1 -- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 msgr2=0x7f68f81d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.717+0000 7f68f5400640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f81d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68f5400640 1 -- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f80d43c0 msgr2=0x7f68f806ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68f5400640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f80d43c0 0x7f68f806ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68f5400640 1 -- 172.21.15.129:0/3413644626 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f81d2200 con 0x7f68f80b8500 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68f5400640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c730 tx=0x7f68e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e40050d0 con 0x7f68f80b8500 2024-09-17T03:53:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68e4005290 con 0x7f68f80b8500 2024-09-17T03:53:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f80ca060 con 0x7f68f80b8500 2024-09-17T03:53:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e400a8a0 con 0x7f68f80b8500 2024-09-17T03:53:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.718+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f80ca620 con 0x7f68f80b8500 2024-09-17T03:53:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.719+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f80b3db0 con 0x7f68f80b8500 2024-09-17T03:53:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.720+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f68e400aa40 con 0x7f68f80b8500 2024-09-17T03:53:21.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.722+0000 7f68ee000640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 0x7f68d807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:21.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.722+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 5767+0+0 (secure 0 0 0) 0x7f68e409a700 con 0x7f68f80b8500 2024-09-17T03:53:21.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.722+0000 7f68f5e00640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 0x7f68d807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:21.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.723+0000 7f68f5e00640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 0x7f68d807b0a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f68e8034000 tx=0x7f68e8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:21.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.726+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f68f80d3460 con 0x7f68f80b8500 2024-09-17T03:53:21.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.945+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f68f806cde0 con 0x7f68f80b8500 2024-09-17T03:53:21.946 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:53:21.946+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:53:21.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.949+0000 7f68ee000640 1 -- 172.21.15.129:0/3413644626 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) ==== 161+0+0 (secure 0 0 0) 0x7f68e4062530 con 0x7f68f80b8500 2024-09-17T03:53:21.949 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:53:21.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 msgr2=0x7f68d807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 0x7f68d807b0a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f68e8034000 tx=0x7f68e8004730 comp rx=0 tx=0).stop 2024-09-17T03:53:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 msgr2=0x7f68f81d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f68e400c730 tx=0x7f68e400cbf0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 shutdown_connections 2024-09-17T03:53:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68d8078b90 0x7f68d807b0a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68f80d43c0 0x7f68f806ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68f80b8500 0x7f68f81d1cc0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 --2- 172.21.15.129:0/3413644626 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68f80b39a0 0x7f68f81d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 >> 172.21.15.129:0/3413644626 conn(0x7f68f808bfc0 msgr2=0x7f68f80cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 shutdown_connections 2024-09-17T03:53:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:21.953+0000 7f68fd200640 1 -- 172.21.15.129:0/3413644626 wait complete. 2024-09-17T03:53:21.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:53:22.093 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:53:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 -- 172.21.15.129:0/331540270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 msgr2=0x7f328000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 --2- 172.21.15.129:0/331540270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f328000b9c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3294005fd0 tx=0x7f3280005830 comp rx=0 tx=0).stop 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f329be00640 1 -- 172.21.15.129:0/331540270 reap_dead start 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 -- 172.21.15.129:0/331540270 shutdown_connections 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 -- 172.21.15.129:0/331540270 >> 172.21.15.129:0/331540270 conn(0x7f329c06d870 msgr2=0x7f329c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 -- 172.21.15.129:0/331540270 shutdown_connections 2024-09-17T03:53:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.213+0000 7f32a1200640 1 -- 172.21.15.129:0/331540270 wait complete. 2024-09-17T03:53:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.216+0000 7f32a1200640 1 Processor -- start 2024-09-17T03:53:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.216+0000 7f32a1200640 1 -- start start 2024-09-17T03:53:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f329c0b0d60 0x7f329c1f8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f329c0b1170 0x7f329c1f8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f329c0cae10 con 0x7f32800051a0 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f329c0cac90 con 0x7f329c0b1170 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f32a1200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f329c15ce40 con 0x7f329c0b0d60 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f329b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f329b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56904/0 (socket says 172.21.15.129:56904) 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.217+0000 7f329b400640 1 -- 172.21.15.129:0/2449168543 learned_addr learned my addr 172.21.15.129:0/2449168543 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329be00640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f329c0b1170 0x7f329c1f8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f3292a00640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f329c0b0d60 0x7f329c1f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 -- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f329c0b0d60 msgr2=0x7f329c1f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f329c0b0d60 0x7f329c1f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 -- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f329c0b1170 msgr2=0x7f329c1f8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f329c0b1170 0x7f329c1f8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 -- 172.21.15.129:0/2449168543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f329c0b1730 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f329b400640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f328c004e00 tx=0x7f328c001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c0107d0 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.218+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f328c016070 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.219+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328c010d00 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.219+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329c0b19d0 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.219+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f329c0b1f90 con 0x7f32800051a0 2024-09-17T03:53:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.220+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f329c0b54d0 con 0x7f32800051a0 2024-09-17T03:53:22.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.221+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f328c01b460 con 0x7f32800051a0 2024-09-17T03:53:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.222+0000 7f3293e00640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 0x7f327807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.222+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7f328c09b700 con 0x7f32800051a0 2024-09-17T03:53:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.223+0000 7f3292a00640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 0x7f327807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:22.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.223+0000 7f3292a00640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 0x7f327807b0a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3284001050 tx=0x7f3284006720 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:22.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.226+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f328c063aa0 con 0x7f32800051a0 2024-09-17T03:53:22.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.446+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f329c15c740 con 0x7f32800051a0 2024-09-17T03:53:22.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.958+0000 7f3293e00640 1 -- 172.21.15.129:0/2449168543 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) ==== 104+0+0 (secure 0 0 0) 0x7f328c0633a0 con 0x7f32800051a0 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 msgr2=0x7f327807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 0x7f327807b0a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3284001050 tx=0x7f3284006720 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 msgr2=0x7f329c1560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f328c004e00 tx=0x7f328c001800 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 shutdown_connections 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3278078b90 0x7f327807b0a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f329c0b1170 0x7f329c1f8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f329c0b0d60 0x7f329c1f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 --2- 172.21.15.129:0/2449168543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32800051a0 0x7f329c1560a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.960+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 >> 172.21.15.129:0/2449168543 conn(0x7f329c06d870 msgr2=0x7f329c0ce840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:22.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.961+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 shutdown_connections 2024-09-17T03:53:22.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:22.961+0000 7f32a1200640 1 -- 172.21.15.129:0/2449168543 wait complete. 2024-09-17T03:53:22.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:53:23.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 -- 172.21.15.129:0/3489799358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 msgr2=0x7fbde0073820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 --2- 172.21.15.129:0/3489799358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde0073820 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0005fd0 tx=0x7fbdd002a4a0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 -- 172.21.15.129:0/3489799358 shutdown_connections 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 --2- 172.21.15.129:0/3489799358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 0x7fbde00cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 --2- 172.21.15.129:0/3489799358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde0073820 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 -- 172.21.15.129:0/3489799358 >> 172.21.15.129:0/3489799358 conn(0x7fbde00d0130 msgr2=0x7fbde00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 -- 172.21.15.129:0/3489799358 shutdown_connections 2024-09-17T03:53:23.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.208+0000 7fbde5400640 1 -- 172.21.15.129:0/3489799358 wait complete. 2024-09-17T03:53:23.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.210+0000 7fbde5400640 1 Processor -- start 2024-09-17T03:53:23.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.210+0000 7fbde5400640 1 -- start start 2024-09-17T03:53:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde00cb300 0x7fbde01f6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 0x7fbde00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbde006f640 con 0x7fbde00cc560 2024-09-17T03:53:23.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbde006f4c0 con 0x7fbde00cb300 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbde5400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbde006f7c0 con 0x7fbde015fb80 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbdd5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbdd5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56922/0 (socket says 172.21.15.129:56922) 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbdd5e00640 1 -- 172.21.15.129:0/2213613574 learned_addr learned my addr 172.21.15.129:0/2213613574 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbdde800640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde00cb300 0x7fbde01f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:23.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.212+0000 7fbddf200640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 0x7fbde00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 -- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 msgr2=0x7fbde00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 0x7fbde00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 -- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde00cb300 msgr2=0x7fbde01f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde00cb300 0x7fbde01f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 -- 172.21.15.129:0/2213613574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde00b40d0 con 0x7fbde00cc560 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd5e00640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c9a0 tx=0x7fbdc800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8007b50 con 0x7fbde00cc560 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbdc8007de0 con 0x7fbde00cc560 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde00c93d0 con 0x7fbde00cc560 2024-09-17T03:53:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8005280 con 0x7fbde00cc560 2024-09-17T03:53:23.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.213+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde00c9990 con 0x7fbde00cc560 2024-09-17T03:53:23.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.214+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde015ca90 con 0x7fbde00cc560 2024-09-17T03:53:23.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.216+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbdc8005420 con 0x7fbde00cc560 2024-09-17T03:53:23.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.217+0000 7fbdd7400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:23.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.217+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5853+0+0 (secure 0 0 0) 0x7fbdc809b180 con 0x7fbde00cc560 2024-09-17T03:53:23.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.217+0000 7fbdde800640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:23.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.218+0000 7fbdde800640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0033000 tx=0x7fbdd0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:23.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.221+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbdc8014030 con 0x7fbde00cc560 2024-09-17T03:53:23.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.402+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbde00d0790 con 0x7fbde00cc560 2024-09-17T03:53:23.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.961+0000 7fbdd7400640 1 -- 172.21.15.129:0/2213613574 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) ==== 181+0+0 (secure 0 0 0) 0x7fbdc8062ed0 con 0x7fbde00cc560 2024-09-17T03:53:23.962 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:53:23.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 msgr2=0x7fbdbc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0033000 tx=0x7fbdd0004730 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 msgr2=0x7fbde015c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c9a0 tx=0x7fbdc800ce60 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 shutdown_connections 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdbc078b90 0x7fbdbc07b0a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbde015fb80 0x7fbde00c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbde00cc560 0x7fbde015c550 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 --2- 172.21.15.129:0/2213613574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbde00cb300 0x7fbde01f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.965+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 >> 172.21.15.129:0/2213613574 conn(0x7fbde00d0130 msgr2=0x7fbde00af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.966+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 shutdown_connections 2024-09-17T03:53:23.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:23.966+0000 7fbde5400640 1 -- 172.21.15.129:0/2213613574 wait complete. 2024-09-17T03:53:23.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.219+0000 7f7b06a00640 1 -- 172.21.15.129:0/1793951726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae4004d40 msgr2=0x7f7ae4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.219+0000 7f7b06a00640 1 --2- 172.21.15.129:0/1793951726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ae4004d40 0x7f7ae4005150 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f7af8005fd0 tx=0x7f7ae4005830 comp rx=0 tx=0).stop 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.219+0000 7f7b04a00640 1 -- 172.21.15.129:0/1793951726 reap_dead start 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.219+0000 7f7b06a00640 1 -- 172.21.15.129:0/1793951726 shutdown_connections 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.219+0000 7f7b06a00640 1 -- 172.21.15.129:0/1793951726 >> 172.21.15.129:0/1793951726 conn(0x7f7b00073d90 msgr2=0x7f7b000741a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.220+0000 7f7b06a00640 1 -- 172.21.15.129:0/1793951726 shutdown_connections 2024-09-17T03:53:24.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.220+0000 7f7b06a00640 1 -- 172.21.15.129:0/1793951726 wait complete. 2024-09-17T03:53:24.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.222+0000 7f7b06a00640 1 Processor -- start 2024-09-17T03:53:24.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.222+0000 7f7b06a00640 1 -- start start 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 0x7f7b001f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 0x7f7b001f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b000b0660 con 0x7f7b001f2e30 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b000b04e0 con 0x7f7b00145e20 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.223+0000 7f7b06a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7b000cd2a0 con 0x7f7ae4004d40 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55630/0 (socket says 172.21.15.129:55630) 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 -- 172.21.15.129:0/3516136647 learned_addr learned my addr 172.21.15.129:0/3516136647 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7affe00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 0x7f7b001f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:24.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7b04a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 0x7f7b001f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 -- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 msgr2=0x7f7b001f28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 0x7f7b001f28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 -- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 msgr2=0x7f7b001f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 0x7f7b001f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 -- 172.21.15.129:0/3516136647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b001f3240 con 0x7f7b001f2e30 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7b04a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 0x7f7b001f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7affe00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 0x7f7b001f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7af7400640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8006da0 tx=0x7f7ae80049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.224+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8015020 con 0x7f7b001f2e30 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.225+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ae8011980 con 0x7f7b001f2e30 2024-09-17T03:53:24.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.225+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8011dd0 con 0x7f7b001f2e30 2024-09-17T03:53:24.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.225+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b00146230 con 0x7f7b001f2e30 2024-09-17T03:53:24.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.225+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b00155590 con 0x7f7b001f2e30 2024-09-17T03:53:24.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.226+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b00155000 con 0x7f7b001f2e30 2024-09-17T03:53:24.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.227+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7ae8018020 con 0x7f7b001f2e30 2024-09-17T03:53:24.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.228+0000 7f7afcc00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 0x7f7adc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:24.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.229+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5415+0+0 (secure 0 0 0) 0x7f7ae809b530 con 0x7f7b001f2e30 2024-09-17T03:53:24.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.229+0000 7f7affe00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 0x7f7adc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:24.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.229+0000 7f7affe00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 0x7f7adc07b0a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009be0 tx=0x7f7af0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:24.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.233+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7ae801f030 con 0x7f7b001f2e30 2024-09-17T03:53:24.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.416+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7b00160310 con 0x7f7b001f2e30 2024-09-17T03:53:24.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.970+0000 7f7afcc00640 1 -- 172.21.15.129:0/3516136647 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) ==== 169+0+0 (secure 0 0 0) 0x7f7b000cda50 con 0x7f7b001f2e30 2024-09-17T03:53:24.970 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:53:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 msgr2=0x7f7adc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 0x7f7adc07b0a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009be0 tx=0x7f7af0005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 msgr2=0x7f7b001458e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8006da0 tx=0x7f7ae80049d0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 shutdown_connections 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7adc078b90 0x7f7adc07b0a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b00145e20 0x7f7b001f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b001f2e30 0x7f7b001458e0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 --2- 172.21.15.129:0/3516136647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ae4004d40 0x7f7b001f28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.974+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 >> 172.21.15.129:0/3516136647 conn(0x7f7b00073d90 msgr2=0x7f7b00070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.975+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 shutdown_connections 2024-09-17T03:53:24.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:24.975+0000 7f7b06a00640 1 -- 172.21.15.129:0/3516136647 wait complete. 2024-09-17T03:53:25.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:53:25.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 -- 172.21.15.129:0/3155621382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80b5470 msgr2=0x7faea80b9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 --2- 172.21.15.129:0/3155621382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80b5470 0x7faea80b9100 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7faea80d33a0 tx=0x7fae9c02a560 comp rx=0 tx=0).stop 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 -- 172.21.15.129:0/3155621382 shutdown_connections 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 --2- 172.21.15.129:0/3155621382 >> v2:172.21.15.181:3300/0 conn(0x7faea80d2820 0x7faea80b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 --2- 172.21.15.129:0/3155621382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80b5470 0x7faea80b9100 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 -- 172.21.15.129:0/3155621382 >> 172.21.15.129:0/3155621382 conn(0x7faea809d6a0 msgr2=0x7faea80d4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 -- 172.21.15.129:0/3155621382 shutdown_connections 2024-09-17T03:53:25.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.227+0000 7faeaec00640 1 -- 172.21.15.129:0/3155621382 wait complete. 2024-09-17T03:53:25.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.229+0000 7faeaec00640 1 Processor -- start 2024-09-17T03:53:25.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.229+0000 7faeaec00640 1 -- start start 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faea80b5470 0x7faea814e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faea80b9640 0x7faea814e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faea80b7f30 con 0x7faea80d2820 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faea80b7db0 con 0x7faea80b5470 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeaec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faea80b80b0 con 0x7faea80b9640 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeae200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faead800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faea80b5470 0x7faea814e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeae200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55646/0 (socket says 172.21.15.129:55646) 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeae200640 1 -- 172.21.15.129:0/3024966581 learned_addr learned my addr 172.21.15.129:0/3024966581 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.231+0000 7faeace00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faea80b9640 0x7faea814e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 -- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faea80b9640 msgr2=0x7faea814e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faea80b9640 0x7faea814e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 -- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faea80b5470 msgr2=0x7faea814e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faea80b5470 0x7faea814e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 -- 172.21.15.129:0/3024966581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea814e5a0 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faeae200640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7faea000ca80 tx=0x7faea000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.232+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0007d90 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.233+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faea0004c30 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.233+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea815ef30 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.233+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea0005280 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.233+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea815f4f0 con 0x7faea80d2820 2024-09-17T03:53:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.234+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea80b85f0 con 0x7faea80d2820 2024-09-17T03:53:25.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.235+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faea0005420 con 0x7faea80d2820 2024-09-17T03:53:25.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.236+0000 7faea5600640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 0x7fae8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.236+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7faea009b420 con 0x7faea80d2820 2024-09-17T03:53:25.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.237+0000 7faead800640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 0x7fae8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.237+0000 7faead800640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 0x7fae8807b0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae9c033000 tx=0x7fae9c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:25.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.240+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faea0014070 con 0x7faea80d2820 2024-09-17T03:53:25.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.434+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7faea80cb120 con 0x7faea80d2820 2024-09-17T03:53:25.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.439+0000 7faea5600640 1 -- 172.21.15.129:0/3024966581 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v43) ==== 71+0+0 (secure 0 0 0) 0x7faea0063470 con 0x7faea80d2820 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.441+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 msgr2=0x7fae8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.441+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 0x7fae8807b0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fae9c033000 tx=0x7fae9c004730 comp rx=0 tx=0).stop 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.441+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 msgr2=0x7faea814edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.441+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7faea000ca80 tx=0x7faea000cf40 comp rx=0 tx=0).stop 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 shutdown_connections 2024-09-17T03:53:25.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae88078b90 0x7fae8807b0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faea80d2820 0x7faea814edd0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faea80b9640 0x7faea814e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 --2- 172.21.15.129:0/3024966581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faea80b5470 0x7faea814e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 >> 172.21.15.129:0/3024966581 conn(0x7faea809d6a0 msgr2=0x7faea80d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 shutdown_connections 2024-09-17T03:53:25.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.442+0000 7faeaec00640 1 -- 172.21.15.129:0/3024966581 wait complete. 2024-09-17T03:53:25.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays' 2024-09-17T03:53:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 -- 172.21.15.129:0/2366700302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 msgr2=0x7fb16c15ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 --2- 172.21.15.129:0/2366700302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c15ba70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0d11e0 tx=0x7fb16002a560 comp rx=0 tx=0).stop 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 -- 172.21.15.129:0/2366700302 shutdown_connections 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 --2- 172.21.15.129:0/2366700302 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 --2- 172.21.15.129:0/2366700302 >> v2:172.21.15.181:3300/0 conn(0x7fb16c15bfb0 0x7fb16c0ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 --2- 172.21.15.129:0/2366700302 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c15ba70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 -- 172.21.15.129:0/2366700302 >> 172.21.15.129:0/2366700302 conn(0x7fb16c0cd670 msgr2=0x7fb16c0d0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 -- 172.21.15.129:0/2366700302 shutdown_connections 2024-09-17T03:53:25.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.683+0000 7fb174800640 1 -- 172.21.15.129:0/2366700302 wait complete. 2024-09-17T03:53:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.685+0000 7fb174800640 1 Processor -- start 2024-09-17T03:53:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.685+0000 7fb174800640 1 -- start start 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c1f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 0x7fb16c0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c0707b0 con 0x7fb16c0748c0 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c070630 con 0x7fb16c15bfb0 2024-09-17T03:53:25.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb174800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb16c070930 con 0x7fb16c074cd0 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55656/0 (socket says 172.21.15.129:55656) 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 -- 172.21.15.129:0/257763996 learned_addr learned my addr 172.21.15.129:0/257763996 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171400640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c1f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb172800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 0x7fb16c0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 -- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 msgr2=0x7fb16c1f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c1f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 -- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 msgr2=0x7fb16c0c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 0x7fb16c0c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171e00640 1 -- 172.21.15.129:0/257763996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c148510 con 0x7fb16c0748c0 2024-09-17T03:53:25.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb171400640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c1f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.687+0000 7fb172800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 0x7fb16c0c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb171e00640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb160034040 tx=0x7fb1600039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16003e030 con 0x7fb16c0748c0 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1600047a0 con 0x7fb16c0748c0 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c06fee0 con 0x7fb16c0748c0 2024-09-17T03:53:25.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160035040 con 0x7fb16c0748c0 2024-09-17T03:53:25.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.688+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c0b0030 con 0x7fb16c0748c0 2024-09-17T03:53:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.689+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c0cfc80 con 0x7fb16c0748c0 2024-09-17T03:53:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.690+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb160041030 con 0x7fb16c0748c0 2024-09-17T03:53:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.691+0000 7fb166000640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 0x7fb15007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:25.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.691+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7fb1600bb2f0 con 0x7fb16c0748c0 2024-09-17T03:53:25.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.691+0000 7fb171400640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 0x7fb15007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:25.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.692+0000 7fb171400640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 0x7fb15007b0a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb15c005fd0 tx=0x7fb15c005d40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:25.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.695+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb16c0cfc80 con 0x7fb16c0748c0 2024-09-17T03:53:25.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.947+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]} v 0) -- 0x7fb16c070e50 con 0x7fb16c0748c0 2024-09-17T03:53:25.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.972+0000 7fb166000640 1 -- 172.21.15.129:0/257763996 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_dirfrag_limit_strays"]}]=0 v217) ==== 140+0+0 (secure 0 0 0) 0x7fb16c148510 con 0x7fb16c0748c0 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 msgr2=0x7fb15007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 0x7fb15007b0a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb15c005fd0 tx=0x7fb15c005d40 comp rx=0 tx=0).stop 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 msgr2=0x7fb16c147fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fb160034040 tx=0x7fb1600039e0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 shutdown_connections 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb150078b90 0x7fb15007b0a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb16c15bfb0 0x7fb16c0c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb16c074cd0 0x7fb16c1f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 --2- 172.21.15.129:0/257763996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb16c0748c0 0x7fb16c147fd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 >> 172.21.15.129:0/257763996 conn(0x7fb16c0cd670 msgr2=0x7fb16c1595d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 shutdown_connections 2024-09-17T03:53:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:25.975+0000 7fb174800640 1 -- 172.21.15.129:0/257763996 wait complete. 2024-09-17T03:53:26.001 INFO:tasks.cephfs_test_runner:That unlinking fails when the stray directory fragment becomes too ... ok 2024-09-17T03:53:26.001 INFO:tasks.cephfs_test_runner:Starting test: test_fancy_layout (tasks.cephfs.test_strays.TestStrays) purge stray file with fancy layout 2024-09-17T03:53:26.001 INFO:tasks.cephfs_test_runner:test_fancy_layout (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:53:26.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 -- 172.21.15.129:0/1290366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200d2af0 msgr2=0x7f6f200b5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 --2- 172.21.15.129:0/1290366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200d2af0 0x7f6f200b5560 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f6f14005fd0 tx=0x7f6f1402a540 comp rx=0 tx=0).stop 2024-09-17T03:53:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 -- 172.21.15.129:0/1290366076 shutdown_connections 2024-09-17T03:53:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 --2- 172.21.15.129:0/1290366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200d2af0 0x7f6f200b5560 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 -- 172.21.15.129:0/1290366076 >> 172.21.15.129:0/1290366076 conn(0x7f6f2009d690 msgr2=0x7f6f200d4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:26.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 -- 172.21.15.129:0/1290366076 shutdown_connections 2024-09-17T03:53:26.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.222+0000 7f6f27000640 1 -- 172.21.15.129:0/1290366076 wait complete. 2024-09-17T03:53:26.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.224+0000 7f6f27000640 1 Processor -- start 2024-09-17T03:53:26.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.224+0000 7f6f27000640 1 -- start start 2024-09-17T03:53:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f200d0260 0x7f6f200cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 0x7f6f200d1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6f20161a50 con 0x7f6f200b3b60 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6f200d08c0 con 0x7f6f200d2af0 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f27000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6f20161bd0 con 0x7f6f200d0260 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55662/0 (socket says 172.21.15.129:55662) 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 -- 172.21.15.129:0/3924513752 learned_addr learned my addr 172.21.15.129:0/3924513752 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f26600640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 0x7f6f200d1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25200640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f200d0260 0x7f6f200cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 -- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f200d0260 msgr2=0x7f6f200cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f200d0260 0x7f6f200cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 -- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 msgr2=0x7f6f200d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 0x7f6f200d1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f25c00640 1 -- 172.21.15.129:0/3924513752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f200cca70 con 0x7f6f200b3b60 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.226+0000 7f6f26600640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 0x7f6f200d1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f25c00640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f14034040 tx=0x7f6f140039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1403e030 con 0x7f6f200b3b60 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f140047a0 con 0x7f6f200b3b60 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f200ccd50 con 0x7f6f200b3b60 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f14035040 con 0x7f6f200b3b60 2024-09-17T03:53:26.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.227+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f200cd310 con 0x7f6f200b3b60 2024-09-17T03:53:26.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.228+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f200ce5f0 con 0x7f6f200b3b60 2024-09-17T03:53:26.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.229+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6f14041030 con 0x7f6f200b3b60 2024-09-17T03:53:26.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.230+0000 7f6f1d600640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 0x7f6f0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.230+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f6f140bb390 con 0x7f6f200b3b60 2024-09-17T03:53:26.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.231+0000 7f6f25200640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 0x7f6f0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.232+0000 7f6f25200640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 0x7f6f0007b0a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6f200b5170 tx=0x7f6f0c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:26.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.235+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6f200ce5f0 con 0x7f6f200b3b60 2024-09-17T03:53:26.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.419+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f201620b0 con 0x7f6f200b3b60 2024-09-17T03:53:26.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.420+0000 7f6f1d600640 1 -- 172.21.15.129:0/3924513752 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+13083 (secure 0 0 0) 0x7f6f14083a30 con 0x7f6f200b3b60 2024-09-17T03:53:26.420 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:26.420 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":73,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:24.963096+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/2480264914":"2024-09-18T03:53:21.947307+0000","172.21.15.129:6833/2480264914":"2024-09-18T03:53:21.947307+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 msgr2=0x7f6f0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 0x7f6f0007b0a0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6f200b5170 tx=0x7f6f0c009040 comp rx=0 tx=0).stop 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 msgr2=0x7f6f201508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f6f14034040 tx=0x7f6f140039e0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 shutdown_connections 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f00078b90 0x7f6f0007b0a0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f200d2af0 0x7f6f200d1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f200d0260 0x7f6f200cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 --2- 172.21.15.129:0/3924513752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f200b3b60 0x7f6f201508f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.422+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 >> 172.21.15.129:0/3924513752 conn(0x7f6f2009d690 msgr2=0x7f6f2006dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:26.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.423+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 shutdown_connections 2024-09-17T03:53:26.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.423+0000 7f6f27000640 1 -- 172.21.15.129:0/3924513752 wait complete. 2024-09-17T03:53:26.444 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:53:26.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:53:26.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.667+0000 7f4f34a00640 1 -- 172.21.15.129:0/739932500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 msgr2=0x7f4f300b56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.667+0000 7f4f34a00640 1 --2- 172.21.15.129:0/739932500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f300b56a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f4f30074570 tx=0x7f4f2002a440 comp rx=0 tx=0).stop 2024-09-17T03:53:26.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 -- 172.21.15.129:0/739932500 shutdown_connections 2024-09-17T03:53:26.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 --2- 172.21.15.129:0/739932500 >> v2:172.21.15.181:3300/0 conn(0x7f4f300ca7b0 0x7f4f300cabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 --2- 172.21.15.129:0/739932500 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300704b0 0x7f4f300b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 --2- 172.21.15.129:0/739932500 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f300b56a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 -- 172.21.15.129:0/739932500 >> 172.21.15.129:0/739932500 conn(0x7f4f300d0160 msgr2=0x7f4f300d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:26.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 -- 172.21.15.129:0/739932500 shutdown_connections 2024-09-17T03:53:26.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.668+0000 7f4f34a00640 1 -- 172.21.15.129:0/739932500 wait complete. 2024-09-17T03:53:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.670+0000 7f4f34a00640 1 Processor -- start 2024-09-17T03:53:26.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.670+0000 7f4f34a00640 1 -- start start 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.671+0000 7f4f34a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f300704b0 0x7f4f30156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f34a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300ca7b0 0x7f4f3009b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f34a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f34a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300cb4f0 con 0x7f4f300cc640 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f34a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300cb370 con 0x7f4f300704b0 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f34a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4f300cb670 con 0x7f4f300ca7b0 2024-09-17T03:53:26.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55684/0 (socket says 172.21.15.129:55684) 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 -- 172.21.15.129:0/2715769089 learned_addr learned my addr 172.21.15.129:0/2715769089 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2d400640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300ca7b0 0x7f4f3009b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2de00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f300704b0 0x7f4f30156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 -- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300ca7b0 msgr2=0x7f4f3009b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300ca7b0 0x7f4f3009b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 -- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f300704b0 msgr2=0x7f4f30156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f300704b0 0x7f4f30156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:26.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.672+0000 7f4f2e800640 1 -- 172.21.15.129:0/2715769089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f301f35b0 con 0x7f4f300cc640 2024-09-17T03:53:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f2e800640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800dac0 tx=0x7f4f2800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28017070 con 0x7f4f300cc640 2024-09-17T03:53:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f28003970 con 0x7f4f300cc640 2024-09-17T03:53:26.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f301f3890 con 0x7f4f300cc640 2024-09-17T03:53:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28003dc0 con 0x7f4f300cc640 2024-09-17T03:53:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.673+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f301f3e50 con 0x7f4f300cc640 2024-09-17T03:53:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.674+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f300cbc00 con 0x7f4f300cc640 2024-09-17T03:53:26.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.675+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4f28003f60 con 0x7f4f300cc640 2024-09-17T03:53:26.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.676+0000 7f4f26000640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 0x7f4f1007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:26.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.676+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f4f2809af40 con 0x7f4f300cc640 2024-09-17T03:53:26.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.677+0000 7f4f2de00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 0x7f4f1007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:26.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.677+0000 7f4f2de00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 0x7f4f1007b0a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4f20033000 tx=0x7f4f20004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:26.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.680+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4f28063670 con 0x7f4f300cc640 2024-09-17T03:53:26.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.864+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f300add20 con 0x7f4f300cc640 2024-09-17T03:53:27.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.865+0000 7f4f26000640 1 -- 172.21.15.129:0/2715769089 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f4f28062f70 con 0x7f4f300cc640 2024-09-17T03:53:27.122 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:53:27.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 msgr2=0x7f4f1007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 0x7f4f1007b0a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4f20033000 tx=0x7f4f20004730 comp rx=0 tx=0).stop 2024-09-17T03:53:27.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 msgr2=0x7f4f30159c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4f2800dac0 tx=0x7f4f2800df80 comp rx=0 tx=0).stop 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 shutdown_connections 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f10078b90 0x7f4f1007b0a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f300cc640 0x7f4f30159c80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f300ca7b0 0x7f4f3009b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 --2- 172.21.15.129:0/2715769089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f300704b0 0x7f4f30156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 >> 172.21.15.129:0/2715769089 conn(0x7f4f300d0160 msgr2=0x7f4f300aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 shutdown_connections 2024-09-17T03:53:27.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:26.867+0000 7f4f34a00640 1 -- 172.21.15.129:0/2715769089 wait complete. 2024-09-17T03:53:27.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-09-17T03:53:27.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 -- 172.21.15.129:0/4090697428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342806f580 msgr2=0x7f342815d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 --2- 172.21.15.129:0/4090697428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342806f580 0x7f342815d700 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f34280b5220 tx=0x7f342002aa30 comp rx=0 tx=0).stop 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 -- 172.21.15.129:0/4090697428 shutdown_connections 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 --2- 172.21.15.129:0/4090697428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342806f580 0x7f342815d700 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 --2- 172.21.15.129:0/4090697428 >> v2:172.21.15.181:3300/0 conn(0x7f342815ca50 0x7f34280cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 -- 172.21.15.129:0/4090697428 >> 172.21.15.129:0/4090697428 conn(0x7f342806d410 msgr2=0x7f34280b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 -- 172.21.15.129:0/4090697428 shutdown_connections 2024-09-17T03:53:27.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.346+0000 7f342ee00640 1 -- 172.21.15.129:0/4090697428 wait complete. 2024-09-17T03:53:27.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.348+0000 7f342ee00640 1 Processor -- start 2024-09-17T03:53:27.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.349+0000 7f342ee00640 1 -- start start 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34280cf540 0x7f34280cb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34281545a0 0x7f34281f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f342815ddc0 con 0x7f342815ca50 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f342815dc40 con 0x7f34280cf540 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342ee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f342815df40 con 0x7f34281545a0 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:27.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55714/0 (socket says 172.21.15.129:55714) 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.350+0000 7f342d000640 1 -- 172.21.15.129:0/2309093272 learned_addr learned my addr 172.21.15.129:0/2309093272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342da00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34280cf540 0x7f34280cb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342e400640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34281545a0 0x7f34281f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 -- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34281545a0 msgr2=0x7f34281f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34281545a0 0x7f34281f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 -- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34280cf540 msgr2=0x7f34280cb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34280cf540 0x7f34280cb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 -- 172.21.15.129:0/2309093272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34281f9c80 con 0x7f342815ca50 2024-09-17T03:53:27.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342d000640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f341400c390 tx=0x7f341400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:27.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414017070 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34140053c0 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3428154c20 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3414003850 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.351+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34281551e0 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.353+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342815e3f0 con 0x7f342815ca50 2024-09-17T03:53:27.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.354+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f341401a070 con 0x7f342815ca50 2024-09-17T03:53:27.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.355+0000 7f3425600640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 0x7f340807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:27.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.355+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f3414099e40 con 0x7f342815ca50 2024-09-17T03:53:27.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.356+0000 7f342da00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 0x7f340807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:27.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.357+0000 7f342da00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 0x7f340807b0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f341c009ac0 tx=0x7f341c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:27.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.360+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f34281f9c80 con 0x7f342815ca50 2024-09-17T03:53:27.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.610+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) -- 0x7f34280739e0 con 0x7f342815ca50 2024-09-17T03:53:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.981+0000 7f3425600640 1 -- 172.21.15.129:0/2309093272 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v219) ==== 135+0+0 (secure 0 0 0) 0x7f3414062580 con 0x7f342815ca50 2024-09-17T03:53:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 msgr2=0x7f340807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 0x7f340807b0a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f341c009ac0 tx=0x7f341c005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 msgr2=0x7f3428154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f341400c390 tx=0x7f341400c850 comp rx=0 tx=0).stop 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 shutdown_connections 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3408078b90 0x7f340807b0a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34281545a0 0x7f34281f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342815ca50 0x7f3428154060 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 --2- 172.21.15.129:0/2309093272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34280cf540 0x7f34280cb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 >> 172.21.15.129:0/2309093272 conn(0x7f342806d410 msgr2=0x7f34281455c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 shutdown_connections 2024-09-17T03:53:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:27.984+0000 7f342ee00640 1 -- 172.21.15.129:0/2309093272 wait complete. 2024-09-17T03:53:28.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.229+0000 7f52cb400640 1 -- 172.21.15.129:0/890245366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 msgr2=0x7f52ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.229+0000 7f52cb400640 1 --2- 172.21.15.129:0/890245366 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52ac004c10 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f52c4072320 tx=0x7f52ac005830 comp rx=0 tx=0).stop 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.229+0000 7f52c9400640 1 -- 172.21.15.129:0/890245366 reap_dead start 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.230+0000 7f52cb400640 1 -- 172.21.15.129:0/890245366 shutdown_connections 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.230+0000 7f52cb400640 1 -- 172.21.15.129:0/890245366 >> 172.21.15.129:0/890245366 conn(0x7f52c40b1cb0 msgr2=0x7f52c40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.230+0000 7f52cb400640 1 -- 172.21.15.129:0/890245366 shutdown_connections 2024-09-17T03:53:28.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.230+0000 7f52cb400640 1 -- 172.21.15.129:0/890245366 wait complete. 2024-09-17T03:53:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.232+0000 7f52cb400640 1 Processor -- start 2024-09-17T03:53:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.232+0000 7f52cb400640 1 -- start start 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52c409aa30 0x7f52c409d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52c415b450 0x7f52c409d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52c40b37a0 con 0x7f52ac004800 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4075070 con 0x7f52c409aa30 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.233+0000 7f52cb400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f52c40b3920 con 0x7f52c415b450 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55728/0 (socket says 172.21.15.129:55728) 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 -- 172.21.15.129:0/2213176832 learned_addr learned my addr 172.21.15.129:0/2213176832 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c3e00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52c409aa30 0x7f52c409d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c9400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52c415b450 0x7f52c409d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 -- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52c415b450 msgr2=0x7f52c409d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52c415b450 0x7f52c409d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 -- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52c409aa30 msgr2=0x7f52c409d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52c409aa30 0x7f52c409d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 -- 172.21.15.129:0/2213176832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c41f7130 con 0x7f52ac004800 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.234+0000 7f52c8a00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006630 tx=0x7f52b80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.235+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8015070 con 0x7f52ac004800 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.235+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52b8011920 con 0x7f52ac004800 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.235+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b8011d70 con 0x7f52ac004800 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.235+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c41f7410 con 0x7f52ac004800 2024-09-17T03:53:28.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.235+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c41f79d0 con 0x7f52ac004800 2024-09-17T03:53:28.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.236+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c40b3eb0 con 0x7f52ac004800 2024-09-17T03:53:28.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.237+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52b8018070 con 0x7f52ac004800 2024-09-17T03:53:28.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.238+0000 7f52c0c00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 0x7f52a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.238+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f52b809b7b0 con 0x7f52ac004800 2024-09-17T03:53:28.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.239+0000 7f52c3e00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 0x7f52a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.239+0000 7f52c3e00640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 0x7f52a407b0d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f52b0035000 tx=0x7f52b00046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.243+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f52b80638a0 con 0x7f52ac004800 2024-09-17T03:53:28.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.427+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f52c40c5270 con 0x7f52ac004800 2024-09-17T03:53:28.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.427+0000 7f52c0c00640 1 -- 172.21.15.129:0/2213176832 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v43) ==== 59+0+2294 (secure 0 0 0) 0x7f52b80682d0 con 0x7f52ac004800 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:53:28.428 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:53:28.429 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:53:28.430 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 msgr2=0x7f52a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 0x7f52a407b0d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f52b0035000 tx=0x7f52b00046c0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 msgr2=0x7f52c4153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006630 tx=0x7f52b80039c0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 shutdown_connections 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52a4078bc0 0x7f52a407b0d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52c415b450 0x7f52c409d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52c409aa30 0x7f52c409d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 --2- 172.21.15.129:0/2213176832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52ac004800 0x7f52c4153430 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 >> 172.21.15.129:0/2213176832 conn(0x7f52c40b1cb0 msgr2=0x7f52c40ae9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:28.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 shutdown_connections 2024-09-17T03:53:28.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.430+0000 7f52cb400640 1 -- 172.21.15.129:0/2213176832 wait complete. 2024-09-17T03:53:28.453 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 -- 172.21.15.129:0/662749996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220073170 msgr2=0x7f822015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 --2- 172.21.15.129:0/662749996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220073170 0x7f822015c780 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8210005fd0 tx=0x7f821002a540 comp rx=0 tx=0).stop 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 -- 172.21.15.129:0/662749996 shutdown_connections 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 --2- 172.21.15.129:0/662749996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070030 0x7f8220074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 --2- 172.21.15.129:0/662749996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220073170 0x7f822015c780 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 -- 172.21.15.129:0/662749996 >> 172.21.15.129:0/662749996 conn(0x7f82200b1cb0 msgr2=0x7f82200b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 -- 172.21.15.129:0/662749996 shutdown_connections 2024-09-17T03:53:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.676+0000 7f8227000640 1 -- 172.21.15.129:0/662749996 wait complete. 2024-09-17T03:53:28.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.678+0000 7f8227000640 1 Processor -- start 2024-09-17T03:53:28.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.679+0000 7f8227000640 1 -- start start 2024-09-17T03:53:28.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220070030 0x7f8220148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070440 0x7f82201e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f822006f5a0 con 0x7f8220073170 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8220074f80 con 0x7f8220070030 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8227000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f822006f720 con 0x7f8220070440 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8225000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8225000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55744/0 (socket says 172.21.15.129:55744) 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.680+0000 7f8225000640 1 -- 172.21.15.129:0/3187532253 learned_addr learned my addr 172.21.15.129:0/3187532253 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f821fe00640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220070030 0x7f8220148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f821f400640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070440 0x7f82201e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 -- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070440 msgr2=0x7f82201e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070440 0x7f82201e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 -- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220070030 msgr2=0x7f8220148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220070030 0x7f8220148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 -- 172.21.15.129:0/3187532253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822015b520 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8225000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8218002230 tx=0x7f821800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218003b70 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8218003d10 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218005330 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.681+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82201488c0 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.682+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82201f7650 con 0x7f8220073170 2024-09-17T03:53:28.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.683+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82201f70e0 con 0x7f8220073170 2024-09-17T03:53:28.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.684+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f82180056e0 con 0x7f8220073170 2024-09-17T03:53:28.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.685+0000 7f8217e00640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 0x7f820007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.685+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f821809a880 con 0x7f8220073170 2024-09-17T03:53:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.685+0000 7f821fe00640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 0x7f820007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:28.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.686+0000 7f821fe00640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 0x7f820007b0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8210033000 tx=0x7f8210004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:28.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.689+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8218062a00 con 0x7f8220073170 2024-09-17T03:53:28.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.876+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f822006fc40 con 0x7f8220073170 2024-09-17T03:53:28.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.880+0000 7f8217e00640 1 -- 172.21.15.129:0/3187532253 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f8218067430 con 0x7f8220073170 2024-09-17T03:53:28.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.882+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 msgr2=0x7f820007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.882+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 0x7f820007b0d0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8210033000 tx=0x7f8210004730 comp rx=0 tx=0).stop 2024-09-17T03:53:28.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.882+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 msgr2=0x7f82201e55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:28.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.882+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f8218002230 tx=0x7f821800cb90 comp rx=0 tx=0).stop 2024-09-17T03:53:28.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 shutdown_connections 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8200078bc0 0x7f820007b0d0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8220073170 0x7f82201e55f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8220070440 0x7f82201e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 --2- 172.21.15.129:0/3187532253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8220070030 0x7f8220148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 >> 172.21.15.129:0/3187532253 conn(0x7f82200b1cb0 msgr2=0x7f82200b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 shutdown_connections 2024-09-17T03:53:28.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:28.883+0000 7f8227000640 1 -- 172.21.15.129:0/3187532253 wait complete. 2024-09-17T03:53:28.907 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:53:28.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:28.997 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:29.100 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:53:29.100 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:29.100 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:29.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:53:29.136 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:29.136 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:29.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:29.173 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:29.173 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:29.211 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:53:29.211 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:29.211 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:29.249 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:29.249 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:53:29.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.471+0000 7f06a3200640 1 -- 172.21.15.129:0/961184047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0ccb50 msgr2=0x7f069c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.471+0000 7f06a3200640 1 --2- 172.21.15.129:0/961184047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0ccb50 0x7f069c0ccf60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0694005fd0 tx=0x7f069402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 -- 172.21.15.129:0/961184047 shutdown_connections 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 --2- 172.21.15.129:0/961184047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0ccb50 0x7f069c0ccf60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 --2- 172.21.15.129:0/961184047 >> v2:172.21.15.181:3300/0 conn(0x7f069c0cc1e0 0x7f069c0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 -- 172.21.15.129:0/961184047 >> 172.21.15.129:0/961184047 conn(0x7f069c06d7d0 msgr2=0x7f069c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:29.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 -- 172.21.15.129:0/961184047 shutdown_connections 2024-09-17T03:53:29.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.472+0000 7f06a3200640 1 -- 172.21.15.129:0/961184047 wait complete. 2024-09-17T03:53:29.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.474+0000 7f06a3200640 1 Processor -- start 2024-09-17T03:53:29.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.474+0000 7f06a3200640 1 -- start start 2024-09-17T03:53:29.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.475+0000 7f06a3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0cc1e0 0x7f069c1fa160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:29.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:29.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069c15d8a0 0x7f069c1f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:29.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f069c1533a0 con 0x7f069c0ccb50 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f069c153520 con 0x7f069c15d8a0 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f069c1536a0 con 0x7f069c0cc1e0 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55746/0 (socket says 172.21.15.129:55746) 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0cc1e0 0x7f069c1fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 -- 172.21.15.129:0/2026551230 learned_addr learned my addr 172.21.15.129:0/2026551230 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f06a1200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069c15d8a0 0x7f069c1f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 -- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0cc1e0 msgr2=0x7f069c1fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0cc1e0 0x7f069c1fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 -- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069c15d8a0 msgr2=0x7f069c1f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069c15d8a0 0x7f069c1f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.476+0000 7f069b400640 1 -- 172.21.15.129:0/2026551230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069c1f5f80 con 0x7f069c0ccb50 2024-09-17T03:53:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f069b400640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f068800c770 tx=0x7f068800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:29.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688003b70 con 0x7f069c0ccb50 2024-09-17T03:53:29.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0688003d30 con 0x7f069c0ccb50 2024-09-17T03:53:29.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f069c09d690 con 0x7f069c0ccb50 2024-09-17T03:53:29.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068800aa50 con 0x7f069c0ccb50 2024-09-17T03:53:29.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.477+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069c1f9680 con 0x7f069c0ccb50 2024-09-17T03:53:29.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.478+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f069c09d3f0 con 0x7f069c0ccb50 2024-09-17T03:53:29.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.479+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f068800abf0 con 0x7f069c0ccb50 2024-09-17T03:53:29.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.481+0000 7f0693e00640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 0x7f067c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:29.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.481+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f068809a6e0 con 0x7f069c0ccb50 2024-09-17T03:53:29.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.481+0000 7f069be00640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 0x7f067c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:29.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.482+0000 7f069be00640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 0x7f067c07b0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f068c0095f0 tx=0x7f068c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:29.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.486+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f069c09d3f0 con 0x7f069c0ccb50 2024-09-17T03:53:29.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.705+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f069c153bc0 con 0x7f069c0ccb50 2024-09-17T03:53:29.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.706+0000 7f0693e00640 1 -- 172.21.15.129:0/2026551230 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+3923 (secure 0 0 0) 0x7f069c0cd4a0 con 0x7f069c0ccb50 2024-09-17T03:53:29.706 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:29.706 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":47,"btime":"2024-09-17T03:53:22:951728+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5321,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3032818029","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3032818029},{"type":"v1","addr":"172.21.15.181:6837","nonce":3032818029}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":40},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1906215362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1906215362},{"type":"v1","addr":"172.21.15.181:6835","nonce":1906215362}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":5862,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/4061349733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":4061349733},{"type":"v1","addr":"172.21.15.129:6834","nonce":4061349733}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6141,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1345181219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1345181219},{"type":"v1","addr":"172.21.15.129:6835","nonce":1345181219}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":47}],"filesystems":[]} 2024-09-17T03:53:29.706 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 47 2024-09-17T03:53:29.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 msgr2=0x7f067c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 0x7f067c07b0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f068c0095f0 tx=0x7f068c005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 msgr2=0x7f069c074990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f068800c770 tx=0x7f068800cc30 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 shutdown_connections 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f067c078bc0 0x7f067c07b0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f069c15d8a0 0x7f069c1f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f069c0ccb50 0x7f069c074990 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 --2- 172.21.15.129:0/2026551230 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f069c0cc1e0 0x7f069c1fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 >> 172.21.15.129:0/2026551230 conn(0x7f069c06d7d0 msgr2=0x7f069c0b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:29.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 shutdown_connections 2024-09-17T03:53:29.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:29.708+0000 7f06a3200640 1 -- 172.21.15.129:0/2026551230 wait complete. 2024-09-17T03:53:29.730 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:53:29.730 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:53:29.730 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:53:29.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:53:29.731 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:53:29.731 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:53:29.731 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:53:29.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:53:29.731 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:53:29.731 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:53:29.732 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:53:29.732 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:53:29.732 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:53:29.732 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:53:29.732 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:53:29.732 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:53:29.754 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:53:29.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:53:29.755 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:53:29.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:53:29.756 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:53:29.756 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:53:29.757 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:53:29.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:53:29.800 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:53:29.800 DEBUG:teuthology.parallel:result is None 2024-09-17T03:53:29.802 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:53:29.802 DEBUG:teuthology.parallel:result is None 2024-09-17T03:53:29.803 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:53:29.803 DEBUG:teuthology.parallel:result is None 2024-09-17T03:53:29.805 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:53:29.805 DEBUG:teuthology.parallel:result is None 2024-09-17T03:53:29.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:53:29.998 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:53:29.998 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:53:30.000 INFO:tasks.ceph.mds.d.smithi181.stderr:2024-09-17T03:53:30.000+0000 7f6533200640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-17T03:53:30.000 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:53:30.001 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:53:30.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 -- 172.21.15.129:0/2762832704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 msgr2=0x7f7b7006f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:30.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2762832704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b7006f1a0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f7b700d1c50 tx=0x7f7b6402a820 comp rx=0 tx=0).stop 2024-09-17T03:53:30.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 -- 172.21.15.129:0/2762832704 shutdown_connections 2024-09-17T03:53:30.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2762832704 >> v2:172.21.15.181:3300/0 conn(0x7f7b700d08e0 0x7f7b7015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:30.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2762832704 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b7006f1a0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:30.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 -- 172.21.15.129:0/2762832704 >> 172.21.15.129:0/2762832704 conn(0x7f7b7006d4f0 msgr2=0x7f7b70071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:30.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 -- 172.21.15.129:0/2762832704 shutdown_connections 2024-09-17T03:53:30.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.040+0000 7f7b77e00640 1 -- 172.21.15.129:0/2762832704 wait complete. 2024-09-17T03:53:30.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.042+0000 7f7b77e00640 1 Processor -- start 2024-09-17T03:53:30.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.042+0000 7f7b77e00640 1 -- start start 2024-09-17T03:53:30.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b70156330 0x7f7b700c5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 0x7f7b7015a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b70160230 con 0x7f7b700cf050 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b701600b0 con 0x7f7b70156330 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b77e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7b701603b0 con 0x7f7b700c6340 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55776/0 (socket says 172.21.15.129:55776) 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 -- 172.21.15.129:0/2550475308 learned_addr learned my addr 172.21.15.129:0/2550475308 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 0x7f7b7015a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b74a00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b70156330 0x7f7b700c5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 -- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 msgr2=0x7f7b7015a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:30.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 0x7f7b7015a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 -- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b70156330 msgr2=0x7f7b700c5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b70156330 0x7f7b700c5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75400640 1 -- 172.21.15.129:0/2550475308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b700c6750 con 0x7f7b700cf050 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.044+0000 7f7b75e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 0x7f7b7015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b75400640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000f80 tx=0x7f7b64032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6403f020 con 0x7f7b700cf050 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b64004040 con 0x7f7b700cf050 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64037040 con 0x7f7b700cf050 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b701f9dd0 con 0x7f7b700cf050 2024-09-17T03:53:30.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.045+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7015c1a0 con 0x7f7b700cf050 2024-09-17T03:53:30.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.046+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b701fa2f0 con 0x7f7b700cf050 2024-09-17T03:53:30.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.048+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7b64042020 con 0x7f7b700cf050 2024-09-17T03:53:30.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.049+0000 7f7b6d600640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 0x7f7b5007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:30.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.049+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4943+0+0 (secure 0 0 0) 0x7f7b640bb170 con 0x7f7b700cf050 2024-09-17T03:53:30.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.049+0000 7f7b74a00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 0x7f7b5007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:30.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.050+0000 7f7b74a00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 0x7f7b5007b0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b700c5c60 tx=0x7f7b5c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:30.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.053+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7b701fa2f0 con 0x7f7b700cf050 2024-09-17T03:53:30.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:30.246+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7b700cf460 con 0x7f7b700cf050 2024-09-17T03:53:31.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.004+0000 7f7b6d600640 1 -- 172.21.15.129:0/2550475308 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v75) ==== 97+0+0 (secure 0 0 0) 0x7f7b64049030 con 0x7f7b700cf050 2024-09-17T03:53:31.004 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 msgr2=0x7f7b5007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 0x7f7b5007b0d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7b700c5c60 tx=0x7f7b5c009040 comp rx=0 tx=0).stop 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 msgr2=0x7f7b701f93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f7b64000f80 tx=0x7f7b64032d10 comp rx=0 tx=0).stop 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 shutdown_connections 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b50078bc0 0x7f7b5007b0d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b700c6340 0x7f7b7015a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b70156330 0x7f7b700c5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 --2- 172.21.15.129:0/2550475308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b700cf050 0x7f7b701f93f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.006+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 >> 172.21.15.129:0/2550475308 conn(0x7f7b7006d4f0 msgr2=0x7f7b701625e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:31.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.007+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 shutdown_connections 2024-09-17T03:53:31.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.007+0000 7f7b77e00640 1 -- 172.21.15.129:0/2550475308 wait complete. 2024-09-17T03:53:31.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:53:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 -- 172.21.15.129:0/727082694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00cd860 msgr2=0x7f57b00b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 --2- 172.21.15.129:0/727082694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00cd860 0x7f57b00b4700 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f57a0009820 tx=0x7f57a002a5a0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 -- 172.21.15.129:0/727082694 shutdown_connections 2024-09-17T03:53:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 --2- 172.21.15.129:0/727082694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00cd860 0x7f57b00b4700 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 -- 172.21.15.129:0/727082694 >> 172.21.15.129:0/727082694 conn(0x7f57b00b3be0 msgr2=0x7f57b00b3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 -- 172.21.15.129:0/727082694 shutdown_connections 2024-09-17T03:53:31.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.257+0000 7f57b5000640 1 -- 172.21.15.129:0/727082694 wait complete. 2024-09-17T03:53:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.259+0000 7f57b5000640 1 Processor -- start 2024-09-17T03:53:31.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.259+0000 7f57b5000640 1 -- start start 2024-09-17T03:53:31.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57b009aa30 0x7f57b015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b00cd860 0x7f57b00b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57b00d0b70 con 0x7f57b00b6e20 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f57b00d09f0 con 0x7f57b00cd860 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57b5000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f57b00d0cf0 con 0x7f57b009aa30 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55796/0 (socket says 172.21.15.129:55796) 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 -- 172.21.15.129:0/1612958029 learned_addr learned my addr 172.21.15.129:0/1612958029 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57af400640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57b009aa30 0x7f57b015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57afe00640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b00cd860 0x7f57b00b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 -- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57b009aa30 msgr2=0x7f57b015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57b009aa30 0x7f57b015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 -- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b00cd860 msgr2=0x7f57b00b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b00cd860 0x7f57b00b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.261+0000 7f57a6a00640 1 -- 172.21.15.129:0/1612958029 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57b01f90d0 con 0x7f57b00b6e20 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57a6a00640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f579800c400 tx=0x7f579800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798011790 con 0x7f57b00b6e20 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5798017070 con 0x7f57b00b6e20 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57b01f93b0 con 0x7f57b00b6e20 2024-09-17T03:53:31.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5798011ca0 con 0x7f57b00b6e20 2024-09-17T03:53:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.262+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57b01f9970 con 0x7f57b00b6e20 2024-09-17T03:53:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.263+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57b015a740 con 0x7f57b00b6e20 2024-09-17T03:53:31.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.264+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f579801c460 con 0x7f57b00b6e20 2024-09-17T03:53:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.265+0000 7f57a7e00640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 0x7f578c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.265+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f579809a7c0 con 0x7f57b00b6e20 2024-09-17T03:53:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.266+0000 7f57af400640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 0x7f578c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.267+0000 7f57af400640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 0x7f578c07b0a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034000 tx=0x7f57a0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:31.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.270+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5798062fd0 con 0x7f57b00b6e20 2024-09-17T03:53:31.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.519+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f57b00b4700 con 0x7f57b00b6e20 2024-09-17T03:53:31.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.520+0000 7f57a7e00640 1 -- 172.21.15.129:0/1612958029 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v30) ==== 80+0+5304 (secure 0 0 0) 0x7f57980628d0 con 0x7f57b00b6e20 2024-09-17T03:53:31.519 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:31.519 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.520 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:53:31.522 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.523 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.524 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:53:31.525 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:53:31.526 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.528 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.529 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:53:31.530 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:53:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:53:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.532 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:53:31.533 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.535 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.536 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.536 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.536 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:53:31.538 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.539 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:53:31.540 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:53:31.542 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:53:31.543 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:53:31.543 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:53:31.543 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:53:31.544 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:53:31.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.544 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:53:31.544 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:53:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 msgr2=0x7f578c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 0x7f578c07b0a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f57a0034000 tx=0x7f57a0004730 comp rx=0 tx=0).stop 2024-09-17T03:53:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 msgr2=0x7f57b0155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f579800c400 tx=0x7f579800c8c0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 shutdown_connections 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f578c078b90 0x7f578c07b0a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57b00cd860 0x7f57b00b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57b00b6e20 0x7f57b0155ce0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 --2- 172.21.15.129:0/1612958029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57b009aa30 0x7f57b015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.522+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 >> 172.21.15.129:0/1612958029 conn(0x7f57b00b3be0 msgr2=0x7f57b00710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:31.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.523+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 shutdown_connections 2024-09-17T03:53:31.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.523+0000 7f57b5000640 1 -- 172.21.15.129:0/1612958029 wait complete. 2024-09-17T03:53:31.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:53:31.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 -- 172.21.15.129:0/514325099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 msgr2=0x7f220415ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 --2- 172.21.15.129:0/514325099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220415ef50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009830 tx=0x7f21ec02a530 comp rx=0 tx=0).stop 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 -- 172.21.15.129:0/514325099 shutdown_connections 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 --2- 172.21.15.129:0/514325099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220415ef50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 -- 172.21.15.129:0/514325099 >> 172.21.15.129:0/514325099 conn(0x7f220406d410 msgr2=0x7f22040b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 -- 172.21.15.129:0/514325099 shutdown_connections 2024-09-17T03:53:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.770+0000 7f2203200640 1 -- 172.21.15.129:0/514325099 wait complete. 2024-09-17T03:53:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.773+0000 7f2203200640 1 Processor -- start 2024-09-17T03:53:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.773+0000 7f2203200640 1 -- start start 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2204073040 0x7f220414fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 0x7f2204154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22040b4200 con 0x7f2204072c30 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f22040b4080 con 0x7f22041f4600 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2203200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f22040b4380 con 0x7f2204073040 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.774+0000 7f2201e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55820/0 (socket says 172.21.15.129:55820) 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 -- 172.21.15.129:0/1308366076 learned_addr learned my addr 172.21.15.129:0/1308366076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f21f9400640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2204073040 0x7f220414fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2202800640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 0x7f2204154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 -- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2204073040 msgr2=0x7f220414fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2204073040 0x7f220414fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 -- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 msgr2=0x7f2204154600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 0x7f2204154600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 -- 172.21.15.129:0/1308366076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220414bcc0 con 0x7f2204072c30 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2202800640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 0x7f2204154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.775+0000 7f2201e00640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f21f400c400 tx=0x7f21f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.776+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4011790 con 0x7f2204072c30 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.776+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21f4017070 con 0x7f2204072c30 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.776+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4011cc0 con 0x7f2204072c30 2024-09-17T03:53:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.776+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22041f3b20 con 0x7f2204072c30 2024-09-17T03:53:31.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.776+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22041501b0 con 0x7f2204072c30 2024-09-17T03:53:31.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.777+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22041f3910 con 0x7f2204072c30 2024-09-17T03:53:31.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.778+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f21f401c460 con 0x7f2204072c30 2024-09-17T03:53:31.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.779+0000 7f21faa00640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 0x7f21e007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:31.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.779+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f21f409a840 con 0x7f2204072c30 2024-09-17T03:53:31.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.779+0000 7f21f9400640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 0x7f21e007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.780+0000 7f21f9400640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 0x7f21e007b0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f21ec034000 tx=0x7f21ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.783+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f21f4062f40 con 0x7f2204072c30 2024-09-17T03:53:31.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.967+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f22040b4830 con 0x7f2204072c30 2024-09-17T03:53:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.968+0000 7f21faa00640 1 -- 172.21.15.129:0/1308366076 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) ==== 84+0+65 (secure 0 0 0) 0x7f21f4062840 con 0x7f2204072c30 2024-09-17T03:53:31.967 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:31.967 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:53:31.967 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:31.967 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:53:31.968 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:53:31.968 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:31.968 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:53:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 msgr2=0x7f21e007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 0x7f21e007b0d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f21ec034000 tx=0x7f21ec004730 comp rx=0 tx=0).stop 2024-09-17T03:53:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 msgr2=0x7f220414b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f21f400c400 tx=0x7f21f400c8c0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 shutdown_connections 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21e0078bc0 0x7f21e007b0d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22041f4600 0x7f2204154600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2204073040 0x7f220414fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 --2- 172.21.15.129:0/1308366076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2204072c30 0x7f220414b350 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 >> 172.21.15.129:0/1308366076 conn(0x7f220406d410 msgr2=0x7f220415e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 shutdown_connections 2024-09-17T03:53:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:31.970+0000 7f2203200640 1 -- 172.21.15.129:0/1308366076 wait complete. 2024-09-17T03:53:31.992 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:53:31.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:53:32.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 -- 172.21.15.129:0/1249010006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a24073590 msgr2=0x7f2a240cc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 --2- 172.21.15.129:0/1249010006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a24073590 0x7f2a240cc610 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f2a24071190 tx=0x7f2a1002a560 comp rx=0 tx=0).stop 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 -- 172.21.15.129:0/1249010006 shutdown_connections 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 --2- 172.21.15.129:0/1249010006 >> v2:172.21.15.181:3300/0 conn(0x7f2a240b5310 0x7f2a240b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 --2- 172.21.15.129:0/1249010006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a24073590 0x7f2a240cc610 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 --2- 172.21.15.129:0/1249010006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24074d80 0x7f2a2415d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 -- 172.21.15.129:0/1249010006 >> 172.21.15.129:0/1249010006 conn(0x7f2a2406ee70 msgr2=0x7f2a2406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 -- 172.21.15.129:0/1249010006 shutdown_connections 2024-09-17T03:53:32.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.220+0000 7f2a2a000640 1 -- 172.21.15.129:0/1249010006 wait complete. 2024-09-17T03:53:32.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.222+0000 7f2a2a000640 1 Processor -- start 2024-09-17T03:53:32.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.223+0000 7f2a2a000640 1 -- start start 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24073590 0x7f2a241331c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a24074d80 0x7f2a2415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a240cf130 con 0x7f2a240b5310 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a240cefb0 con 0x7f2a24074d80 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a2a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2a240cf2b0 con 0x7f2a24073590 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a29600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a29600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55824/0 (socket says 172.21.15.129:55824) 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.224+0000 7f2a29600640 1 -- 172.21.15.129:0/2762166019 learned_addr learned my addr 172.21.15.129:0/2762166019 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a23e00640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a24074d80 0x7f2a2415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a28c00640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24073590 0x7f2a241331c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 -- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24073590 msgr2=0x7f2a241331c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24073590 0x7f2a241331c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 -- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a24074d80 msgr2=0x7f2a2415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a24074d80 0x7f2a2415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 -- 172.21.15.129:0/2762166019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a241f8750 con 0x7f2a240b5310 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a29600640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e9c0 tx=0x7f2a1c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c004d20 con 0x7f2a240b5310 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a1c00cac0 con 0x7f2a240b5310 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c005020 con 0x7f2a240b5310 2024-09-17T03:53:32.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.225+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a241f89f0 con 0x7f2a240b5310 2024-09-17T03:53:32.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.226+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a241f8fb0 con 0x7f2a240b5310 2024-09-17T03:53:32.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.227+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2415c0a0 con 0x7f2a240b5310 2024-09-17T03:53:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.228+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a1c0051c0 con 0x7f2a240b5310 2024-09-17T03:53:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.229+0000 7f2a20c00640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 0x7f2a0407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.229+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4857+0+0 (secure 0 0 0) 0x7f2a1c09b100 con 0x7f2a240b5310 2024-09-17T03:53:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.230+0000 7f2a28c00640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 0x7f2a0407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.231+0000 7f2a28c00640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 0x7f2a0407b0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2a14005fd0 tx=0x7f2a14005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:32.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.234+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a1c016030 con 0x7f2a240b5310 2024-09-17T03:53:32.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:32.419+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2a2406a170 con 0x7f2a240b5310 2024-09-17T03:53:33.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.014+0000 7f2a20c00640 1 -- 172.21.15.129:0/2762166019 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) ==== 138+0+0 (secure 0 0 0) 0x7f2a2406a170 con 0x7f2a240b5310 2024-09-17T03:53:33.013 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:53:33.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 msgr2=0x7f2a0407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:33.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 0x7f2a0407b0d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2a14005fd0 tx=0x7f2a14005850 comp rx=0 tx=0).stop 2024-09-17T03:53:33.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 msgr2=0x7f2a241e8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c00e9c0 tx=0x7f2a1c00ee80 comp rx=0 tx=0).stop 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 shutdown_connections 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a04078bc0 0x7f2a0407b0d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a240b5310 0x7f2a241e8d30 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a24074d80 0x7f2a2415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 --2- 172.21.15.129:0/2762166019 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a24073590 0x7f2a241331c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 >> 172.21.15.129:0/2762166019 conn(0x7f2a2406ee70 msgr2=0x7f2a24155f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 shutdown_connections 2024-09-17T03:53:33.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.017+0000 7f2a2a000640 1 -- 172.21.15.129:0/2762166019 wait complete. 2024-09-17T03:53:33.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.277+0000 7f1b2a800640 1 -- 172.21.15.129:0/2086887743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 msgr2=0x7f1b2415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.277+0000 7f1b2a800640 1 --2- 172.21.15.129:0/2086887743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b2415c780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1b14009830 tx=0x7f1b1402a480 comp rx=0 tx=0).stop 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.277+0000 7f1b2a800640 1 -- 172.21.15.129:0/2086887743 shutdown_connections 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.277+0000 7f1b2a800640 1 --2- 172.21.15.129:0/2086887743 >> v2:172.21.15.181:3300/0 conn(0x7f1b24070440 0x7f1b240b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.277+0000 7f1b2a800640 1 --2- 172.21.15.129:0/2086887743 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b2415c780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.278+0000 7f1b2a800640 1 -- 172.21.15.129:0/2086887743 >> 172.21.15.129:0/2086887743 conn(0x7f1b240b1cb0 msgr2=0x7f1b240b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.278+0000 7f1b2a800640 1 -- 172.21.15.129:0/2086887743 shutdown_connections 2024-09-17T03:53:33.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.278+0000 7f1b2a800640 1 -- 172.21.15.129:0/2086887743 wait complete. 2024-09-17T03:53:33.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.280+0000 7f1b2a800640 1 Processor -- start 2024-09-17T03:53:33.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.280+0000 7f1b2a800640 1 -- start start 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b24070030 0x7f1b24147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b240d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240b3b00 con 0x7f1b24070440 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1b240b3980 con 0x7f1b24070030 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.281+0000 7f1b2a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1b240b3c80 con 0x7f1b24073170 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55828/0 (socket says 172.21.15.129:55828) 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 -- 172.21.15.129:0/4129976253 learned_addr learned my addr 172.21.15.129:0/4129976253 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b23e00640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b240d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b23400640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b24070030 0x7f1b24147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:33.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 -- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 msgr2=0x7f1b240d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b240d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 -- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b24070030 msgr2=0x7f1b24147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b24070030 0x7f1b24147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 -- 172.21.15.129:0/4129976253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b24159a00 con 0x7f1b24070440 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.282+0000 7f1b1aa00640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c3c0 tx=0x7f1b0c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.283+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c017070 con 0x7f1b24070440 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.283+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1b0c0053c0 con 0x7f1b24070440 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.283+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0c003850 con 0x7f1b24070440 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.283+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b241e52c0 con 0x7f1b24070440 2024-09-17T03:53:33.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.283+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b240cf2d0 con 0x7f1b24070440 2024-09-17T03:53:33.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.284+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b240b41a0 con 0x7f1b24070440 2024-09-17T03:53:33.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.285+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1b0c01a070 con 0x7f1b24070440 2024-09-17T03:53:33.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.286+0000 7f1b1be00640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 0x7f1b0007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:33.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.286+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 5227+0+0 (secure 0 0 0) 0x7f1b0c099dd0 con 0x7f1b24070440 2024-09-17T03:53:33.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.287+0000 7f1b23400640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 0x7f1b0007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:33.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.287+0000 7f1b23400640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 0x7f1b0007b0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:33.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.291+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1b24159a00 con 0x7f1b24070440 2024-09-17T03:53:33.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:33.475+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1b240c3860 con 0x7f1b24070440 2024-09-17T03:53:34.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.017+0000 7f1b1be00640 1 -- 172.21.15.129:0/4129976253 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) ==== 170+0+0 (secure 0 0 0) 0x7f1b0c061d00 con 0x7f1b24070440 2024-09-17T03:53:34.016 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.020+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 msgr2=0x7f1b0007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.020+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 0x7f1b0007b0d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1b14034000 tx=0x7f1b14004730 comp rx=0 tx=0).stop 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.020+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 msgr2=0x7f1b240cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.020+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c00c3c0 tx=0x7f1b0c00c880 comp rx=0 tx=0).stop 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 shutdown_connections 2024-09-17T03:53:34.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b00078bc0 0x7f1b0007b0d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1b24073170 0x7f1b240d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1b24070440 0x7f1b240cfad0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 --2- 172.21.15.129:0/4129976253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b24070030 0x7f1b24147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 >> 172.21.15.129:0/4129976253 conn(0x7f1b240b1cb0 msgr2=0x7f1b2415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 shutdown_connections 2024-09-17T03:53:34.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.021+0000 7f1b2a800640 1 -- 172.21.15.129:0/4129976253 wait complete. 2024-09-17T03:53:34.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 -- 172.21.15.129:0/1938389327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 msgr2=0x7f84280cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 --2- 172.21.15.129:0/1938389327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cc830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f841c009830 tx=0x7f841c02a480 comp rx=0 tx=0).stop 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 -- 172.21.15.129:0/1938389327 shutdown_connections 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 --2- 172.21.15.129:0/1938389327 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 0x7f842806ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 --2- 172.21.15.129:0/1938389327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cc830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 -- 172.21.15.129:0/1938389327 >> 172.21.15.129:0/1938389327 conn(0x7f842806d410 msgr2=0x7f84280b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:34.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 -- 172.21.15.129:0/1938389327 shutdown_connections 2024-09-17T03:53:34.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.283+0000 7f842e800640 1 -- 172.21.15.129:0/1938389327 wait complete. 2024-09-17T03:53:34.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.285+0000 7f842e800640 1 Processor -- start 2024-09-17T03:53:34.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.285+0000 7f842e800640 1 -- start start 2024-09-17T03:53:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 0x7f8428150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:34.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:34.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:34.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f842815de10 con 0x7f84280cf950 2024-09-17T03:53:34.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f842815df90 con 0x7f842815ca50 2024-09-17T03:53:34.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f842815e110 con 0x7f84280cf540 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48982/0 (socket says 172.21.15.129:48982) 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 -- 172.21.15.129:0/834886052 learned_addr learned my addr 172.21.15.129:0/834886052 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842d400640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 0x7f8428150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842de00640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 -- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 msgr2=0x7f8428150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 0x7f8428150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 -- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 msgr2=0x7f84280cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.287+0000 7f842ca00640 1 -- 172.21.15.129:0/834886052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8428157100 con 0x7f84280cf950 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f842ca00640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f841400e9c0 tx=0x7f841400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841400ccc0 con 0x7f84280cf950 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8414004020 con 0x7f84280cf950 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841400ce60 con 0x7f84280cf950 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8428156270 con 0x7f84280cf950 2024-09-17T03:53:34.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.288+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8428158910 con 0x7f84280cf950 2024-09-17T03:53:34.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.289+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8428155fe0 con 0x7f84280cf950 2024-09-17T03:53:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.291+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f84140035f0 con 0x7f84280cf950 2024-09-17T03:53:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.292+0000 7f8425600640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.292+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5609+0+0 (secure 0 0 0) 0x7f841409b850 con 0x7f84280cf950 2024-09-17T03:53:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.292+0000 7f842d400640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:34.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.293+0000 7f842d400640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f841c034000 tx=0x7f841c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:34.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.295+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8414063d60 con 0x7f84280cf950 2024-09-17T03:53:34.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:34.515+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f8428070f00 con 0x7f84280cf950 2024-09-17T03:53:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.033+0000 7f8425600640 1 -- 172.21.15.129:0/834886052 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v50) ==== 174+0+0 (secure 0 0 0) 0x7f8428070f00 con 0x7f84280cf950 2024-09-17T03:53:35.032 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 12 and data pool 13 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.035+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 msgr2=0x7f840807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.035+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f841c034000 tx=0x7f841c004730 comp rx=0 tx=0).stop 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.035+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 msgr2=0x7f8428155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.035+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f841400e9c0 tx=0x7f841400ee80 comp rx=0 tx=0).stop 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 shutdown_connections 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8408078b90 0x7f840807b0a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f842815ca50 0x7f84280cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84280cf950 0x7f8428155890 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 --2- 172.21.15.129:0/834886052 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84280cf540 0x7f8428150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 >> 172.21.15.129:0/834886052 conn(0x7f842806d410 msgr2=0x7f84280b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 shutdown_connections 2024-09-17T03:53:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.036+0000 7f842e800640 1 -- 172.21.15.129:0/834886052 wait complete. 2024-09-17T03:53:35.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 -- 172.21.15.129:0/61907926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3374005350 msgr2=0x7f337400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 --2- 172.21.15.129:0/61907926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3374005350 0x7f337400b9c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f3388001d90 tx=0x7f3374005830 comp rx=0 tx=0).stop 2024-09-17T03:53:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 -- 172.21.15.129:0/61907926 shutdown_connections 2024-09-17T03:53:35.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 --2- 172.21.15.129:0/61907926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3374005350 0x7f337400b9c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 -- 172.21.15.129:0/61907926 >> 172.21.15.129:0/61907926 conn(0x7f339006d410 msgr2=0x7f33900b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 -- 172.21.15.129:0/61907926 shutdown_connections 2024-09-17T03:53:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.310+0000 7f338f200640 1 -- 172.21.15.129:0/61907926 wait complete. 2024-09-17T03:53:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.312+0000 7f338f200640 1 Processor -- start 2024-09-17T03:53:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.312+0000 7f338f200640 1 -- start start 2024-09-17T03:53:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3374005350 0x7f3390153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f33901449a0 0x7f339013e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f33900ccbc0 con 0x7f33900b0b70 2024-09-17T03:53:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f33900cca40 con 0x7f3374005350 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f33900ccd40 con 0x7f33901449a0 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48990/0 (socket says 172.21.15.129:48990) 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338e800640 1 -- 172.21.15.129:0/3529180072 learned_addr learned my addr 172.21.15.129:0/3529180072 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338de00640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3374005350 0x7f3390153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.314+0000 7f338d400640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f33901449a0 0x7f339013e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 -- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f33901449a0 msgr2=0x7f339013e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f33901449a0 0x7f339013e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 -- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3374005350 msgr2=0x7f3390153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3374005350 0x7f3390153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 -- 172.21.15.129:0/3529180072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33900b14c0 con 0x7f33900b0b70 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338e800640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f338802ac80 tx=0x7f3388006d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388003d80 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3388005a90 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33901f8870 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388005ec0 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.315+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33901f8e30 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.316+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f339006f0c0 con 0x7f33900b0b70 2024-09-17T03:53:35.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.318+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3388006060 con 0x7f33900b0b70 2024-09-17T03:53:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.319+0000 7f3386000640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 0x7f337007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.319+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f33880ba7c0 con 0x7f33900b0b70 2024-09-17T03:53:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.319+0000 7f338de00640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 0x7f337007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.320+0000 7f338de00640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 0x7f337007b0a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3380009b10 tx=0x7f3380005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:35.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.323+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f33900b14c0 con 0x7f33900b0b70 2024-09-17T03:53:35.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.505+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f339006fb50 con 0x7f33900b0b70 2024-09-17T03:53:35.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.507+0000 7f3386000640 1 -- 172.21.15.129:0/3529180072 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16781 (secure 0 0 0) 0x7f3388082ae0 con 0x7f33900b0b70 2024-09-17T03:53:35.506 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:35.506 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:34.515744+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:53:32.430935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-17T03:53:33.487308+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 msgr2=0x7f337007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 0x7f337007b0a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3380009b10 tx=0x7f3380005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 msgr2=0x7f33900b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f338802ac80 tx=0x7f3388006d20 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 shutdown_connections 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3370078b90 0x7f337007b0a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f33900b0b70 0x7f33900b0f80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f33901449a0 0x7f339013e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 --2- 172.21.15.129:0/3529180072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3374005350 0x7f3390153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 >> 172.21.15.129:0/3529180072 conn(0x7f339006d410 msgr2=0x7f33900ae2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 shutdown_connections 2024-09-17T03:53:35.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.509+0000 7f338f200640 1 -- 172.21.15.129:0/3529180072 wait complete. 2024-09-17T03:53:35.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 -- 172.21.15.129:0/2402362883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 msgr2=0x7f8b4806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 --2- 172.21.15.129:0/2402362883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b4806ee70 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f8b34005fd0 tx=0x7f8b3402b9a0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 -- 172.21.15.129:0/2402362883 shutdown_connections 2024-09-17T03:53:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 --2- 172.21.15.129:0/2402362883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b4806ee70 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 -- 172.21.15.129:0/2402362883 >> 172.21.15.129:0/2402362883 conn(0x7f8b4806d410 msgr2=0x7f8b480b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 -- 172.21.15.129:0/2402362883 shutdown_connections 2024-09-17T03:53:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.755+0000 7f8b4e800640 1 -- 172.21.15.129:0/2402362883 wait complete. 2024-09-17T03:53:35.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.757+0000 7f8b4e800640 1 Processor -- start 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.757+0000 7f8b4e800640 1 -- start start 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4806f3b0 0x7f8b481f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 0x7f8b481485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480b4560 con 0x7f8b48151640 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480b43e0 con 0x7f8b4806f3b0 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8b480b46e0 con 0x7f8b4815b8f0 2024-09-17T03:53:35.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49004/0 (socket says 172.21.15.129:49004) 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4ca00640 1 -- 172.21.15.129:0/3709548801 learned_addr learned my addr 172.21.15.129:0/3709548801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4d400640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4806f3b0 0x7f8b481f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.759+0000 7f8b4de00640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 0x7f8b481485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 -- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 msgr2=0x7f8b481485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 0x7f8b481485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 -- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4806f3b0 msgr2=0x7f8b481f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4806f3b0 0x7f8b481f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 -- 172.21.15.129:0/3709548801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b481f7970 con 0x7f8b48151640 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4de00640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 0x7f8b481485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4ca00640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400ab00 tx=0x7f8b34030920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34034070 con 0x7f8b48151640 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b340035f0 con 0x7f8b48151640 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34003a40 con 0x7f8b48151640 2024-09-17T03:53:35.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b48159780 con 0x7f8b48151640 2024-09-17T03:53:35.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.760+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b48159d40 con 0x7f8b48151640 2024-09-17T03:53:35.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.761+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b480b41d0 con 0x7f8b48151640 2024-09-17T03:53:35.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.763+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b34003be0 con 0x7f8b48151640 2024-09-17T03:53:35.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.763+0000 7f8b45600640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 0x7f8b2807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:35.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.764+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5709+0+0 (secure 0 0 0) 0x7f8b340bc220 con 0x7f8b48151640 2024-09-17T03:53:35.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.764+0000 7f8b4d400640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 0x7f8b2807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:35.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.765+0000 7f8b4d400640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 0x7f8b2807b0a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c000f60 tx=0x7f8b3c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:35.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.768+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8b480b41d0 con 0x7f8b48151640 2024-09-17T03:53:35.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.952+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b480cc370 con 0x7f8b48151640 2024-09-17T03:53:35.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.953+0000 7f8b45600640 1 -- 172.21.15.129:0/3709548801 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) ==== 74+0+16781 (secure 0 0 0) 0x7f8b340845c0 con 0x7f8b48151640 2024-09-17T03:53:35.952 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:35.952 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:34.515744+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:53:32.430935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-17T03:53:33.487308+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:35.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 msgr2=0x7f8b2807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 0x7f8b2807b0a0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c000f60 tx=0x7f8b3c009040 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 msgr2=0x7f8b480c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400ab00 tx=0x7f8b34030920 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 shutdown_connections 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b28078b90 0x7f8b2807b0a0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b4815b8f0 0x7f8b481485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b48151640 0x7f8b480c65d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 --2- 172.21.15.129:0/3709548801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b4806f3b0 0x7f8b481f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.955+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 >> 172.21.15.129:0/3709548801 conn(0x7f8b4806d410 msgr2=0x7f8b4815ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.956+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 shutdown_connections 2024-09-17T03:53:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:35.956+0000 7f8b4e800640 1 -- 172.21.15.129:0/3709548801 wait complete. 2024-09-17T03:53:35.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:53:36.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2954737418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160990 msgr2=0x7f3e180ca5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2954737418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160990 0x7f3e180ca5e0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3e18161100 tx=0x7f3e1002aa30 comp rx=0 tx=0).stop 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2954737418 shutdown_connections 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2954737418 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160990 0x7f3e180ca5e0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2954737418 >> v2:172.21.15.181:3300/0 conn(0x7f3e18160040 0x7f3e18160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2954737418 >> 172.21.15.129:0/2954737418 conn(0x7f3e1806ee50 msgr2=0x7f3e1806f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2954737418 shutdown_connections 2024-09-17T03:53:36.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.205+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2954737418 wait complete. 2024-09-17T03:53:36.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.207+0000 7f3e1ca00640 1 Processor -- start 2024-09-17T03:53:36.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.208+0000 7f3e1ca00640 1 -- start start 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 0x7f3e1814bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 0x7f3e180b3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e180cb160 con 0x7f3e18160040 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e180cafe0 con 0x7f3e18160990 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e1ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3e180cb2e0 con 0x7f3e1815c580 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e0ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e0ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49024/0 (socket says 172.21.15.129:49024) 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.209+0000 7f3e0ea00640 1 -- 172.21.15.129:0/2726493477 learned_addr learned my addr 172.21.15.129:0/2726493477 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:36.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e17400640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 0x7f3e1814bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e17e00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 0x7f3e180b3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 -- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 msgr2=0x7f3e1814bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 0x7f3e1814bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 -- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 msgr2=0x7f3e180b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 0x7f3e180b3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 -- 172.21.15.129:0/2726493477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e181f6de0 con 0x7f3e18160040 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e17e00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 0x7f3e180b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e17400640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 0x7f3e1814bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:36.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0ea00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000e790 tx=0x7f3e0000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e0000eea0 con 0x7f3e18160040 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e0000cdb0 con 0x7f3e18160040 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00010690 con 0x7f3e18160040 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.210+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e181f70c0 con 0x7f3e18160040 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.211+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e181f7680 con 0x7f3e18160040 2024-09-17T03:53:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.211+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e180cb800 con 0x7f3e18160040 2024-09-17T03:53:36.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.213+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3e00010830 con 0x7f3e18160040 2024-09-17T03:53:36.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.214+0000 7f3e0fe00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 0x7f3df407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:36.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.214+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f3e0009b720 con 0x7f3e18160040 2024-09-17T03:53:36.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.214+0000 7f3e17400640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 0x7f3df407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:36.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.215+0000 7f3e17400640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 0x7f3df407b0a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3e08000f60 tx=0x7f3e08009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:36.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.218+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3e00063be0 con 0x7f3e18160040 2024-09-17T03:53:36.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:36.436+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3e180b05d0 con 0x7f3e18160040 2024-09-17T03:53:37.016 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:53:37.017+0000 7f1a0e000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:53:37.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.061+0000 7f3e0fe00640 1 -- 172.21.15.129:0/2726493477 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v53) ==== 118+0+0 (secure 0 0 0) 0x7f3e000634e0 con 0x7f3e18160040 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 msgr2=0x7f3df407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 0x7f3df407b0a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3e08000f60 tx=0x7f3e08009040 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 msgr2=0x7f3e180b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000e790 tx=0x7f3e0000ec50 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 shutdown_connections 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3df4078b90 0x7f3df407b0a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e18160990 0x7f3e180b3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e18160040 0x7f3e180b39c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 --2- 172.21.15.129:0/2726493477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e1815c580 0x7f3e1814bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.063+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 >> 172.21.15.129:0/2726493477 conn(0x7f3e1806ee50 msgr2=0x7f3e180b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.064+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 shutdown_connections 2024-09-17T03:53:37.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.064+0000 7f3e1ca00640 1 -- 172.21.15.129:0/2726493477 wait complete. 2024-09-17T03:53:37.088 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:53:37.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:53:37.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.313+0000 7f66f7e00640 1 -- 172.21.15.129:0/4241717005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070440 msgr2=0x7f66f0074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:37.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.313+0000 7f66f7e00640 1 --2- 172.21.15.129:0/4241717005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070440 0x7f66f0074b80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f66f00710f0 tx=0x7f66e802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 -- 172.21.15.129:0/4241717005 shutdown_connections 2024-09-17T03:53:37.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 --2- 172.21.15.129:0/4241717005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070440 0x7f66f0074b80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 --2- 172.21.15.129:0/4241717005 >> v2:172.21.15.181:3300/0 conn(0x7f66f0070030 0x7f66f0074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 -- 172.21.15.129:0/4241717005 >> 172.21.15.129:0/4241717005 conn(0x7f66f00b1cb0 msgr2=0x7f66f00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:37.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 -- 172.21.15.129:0/4241717005 shutdown_connections 2024-09-17T03:53:37.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.314+0000 7f66f7e00640 1 -- 172.21.15.129:0/4241717005 wait complete. 2024-09-17T03:53:37.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.316+0000 7f66f7e00640 1 Processor -- start 2024-09-17T03:53:37.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.316+0000 7f66f7e00640 1 -- start start 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66f0070440 0x7f66f015af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f0073170 0x7f66f01f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00b39f0 con 0x7f66f0070030 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00b3870 con 0x7f66f0073170 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.317+0000 7f66f7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f66f00b3b70 con 0x7f66f0070440 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49050/0 (socket says 172.21.15.129:49050) 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 -- 172.21.15.129:0/3236260936 learned_addr learned my addr 172.21.15.129:0/3236260936 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f0073170 0x7f66f01f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:37.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f4a00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66f0070440 0x7f66f015af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 -- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66f0070440 msgr2=0x7f66f015af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66f0070440 0x7f66f015af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 -- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f0073170 msgr2=0x7f66f01f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f0073170 0x7f66f01f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 -- 172.21.15.129:0/3236260936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f01f5880 con 0x7f66f0070030 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.318+0000 7f66f5400640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c3f0 tx=0x7f66e400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.319+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4017070 con 0x7f66f0070030 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.319+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66e40035f0 con 0x7f66f0070030 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.319+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f009ad10 con 0x7f66f0070030 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.319+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4003a40 con 0x7f66f0070030 2024-09-17T03:53:37.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.319+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f01f5b40 con 0x7f66f0070030 2024-09-17T03:53:37.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.320+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f00b4110 con 0x7f66f0070030 2024-09-17T03:53:37.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.321+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f66e401a070 con 0x7f66f0070030 2024-09-17T03:53:37.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.322+0000 7f66ed600640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 0x7f66d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:37.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.322+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f66e409a060 con 0x7f66f0070030 2024-09-17T03:53:37.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.323+0000 7f66f4a00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 0x7f66d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:37.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.323+0000 7f66f4a00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 0x7f66d007b0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f66f0072de0 tx=0x7f66dc008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:37.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.326+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f66e4062490 con 0x7f66f0070030 2024-09-17T03:53:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:37.543+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f66f00c3860 con 0x7f66f0070030 2024-09-17T03:53:38.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.069+0000 7f66ed600640 1 -- 172.21.15.129:0/3236260936 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v54) ==== 122+0+0 (secure 0 0 0) 0x7f66e4061d90 con 0x7f66f0070030 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.071+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 msgr2=0x7f66d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.071+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 0x7f66d007b0d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f66f0072de0 tx=0x7f66dc008040 comp rx=0 tx=0).stop 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.071+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 msgr2=0x7f66f009d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.071+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c3f0 tx=0x7f66e400c8b0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 shutdown_connections 2024-09-17T03:53:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66d0078bc0 0x7f66d007b0d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66f0073170 0x7f66f01f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66f0070440 0x7f66f015af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 --2- 172.21.15.129:0/3236260936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66f0070030 0x7f66f009d970 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 >> 172.21.15.129:0/3236260936 conn(0x7f66f00b1cb0 msgr2=0x7f66f00b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 shutdown_connections 2024-09-17T03:53:38.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.072+0000 7f66f7e00640 1 -- 172.21.15.129:0/3236260936 wait complete. 2024-09-17T03:53:38.095 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:53:38.095 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 -- 172.21.15.129:0/3876058404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 msgr2=0x7f15700cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 --2- 172.21.15.129:0/3876058404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700cf170 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f15700726e0 tx=0x7f156802aac0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 -- 172.21.15.129:0/3876058404 shutdown_connections 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 --2- 172.21.15.129:0/3876058404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700cf170 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 --2- 172.21.15.129:0/3876058404 >> v2:172.21.15.181:3300/0 conn(0x7f15701600c0 0x7f1570142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 -- 172.21.15.129:0/3876058404 >> 172.21.15.129:0/3876058404 conn(0x7f157006d410 msgr2=0x7f15700b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:38.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 -- 172.21.15.129:0/3876058404 shutdown_connections 2024-09-17T03:53:38.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.315+0000 7f1575800640 1 -- 172.21.15.129:0/3876058404 wait complete. 2024-09-17T03:53:38.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.317+0000 7f1575800640 1 Processor -- start 2024-09-17T03:53:38.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.317+0000 7f1575800640 1 -- start start 2024-09-17T03:53:38.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 0x7f15701f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15701600c0 0x7f15700a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f157006f9c0 con 0x7f15700ced60 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f157006f840 con 0x7f15701600c0 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1575800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f157006fb40 con 0x7f1570072c30 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1567400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1567400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49064/0 (socket says 172.21.15.129:49064) 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1567400640 1 -- 172.21.15.129:0/3484746967 learned_addr learned my addr 172.21.15.129:0/3484746967 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f156fe00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 0x7f15701f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.319+0000 7f1574e00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15701600c0 0x7f15700a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 -- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 msgr2=0x7f15701f7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 0x7f15701f7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 -- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15701600c0 msgr2=0x7f15700a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15701600c0 0x7f15700a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 -- 172.21.15.129:0/3484746967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15700cad40 con 0x7f15700ced60 2024-09-17T03:53:38.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f156fe00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 0x7f15701f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:38.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1567400640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f155800c370 tx=0x7f155800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15580052d0 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1558007020 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558007470 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15700cb020 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.320+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15700cb5e0 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.321+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1570070060 con 0x7f15700ced60 2024-09-17T03:53:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.323+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f15580067d0 con 0x7f15700ced60 2024-09-17T03:53:38.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.324+0000 7f156cc00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 0x7f154c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.324+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f155809a1c0 con 0x7f15700ced60 2024-09-17T03:53:38.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.324+0000 7f156fe00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 0x7f154c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.325+0000 7f156fe00640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 0x7f154c07b0a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1560009b80 tx=0x7f1560005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:38.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.328+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1558061f10 con 0x7f15700ced60 2024-09-17T03:53:38.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.546+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15700caad0 con 0x7f15700ced60 2024-09-17T03:53:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.547+0000 7f156cc00640 1 -- 172.21.15.129:0/3484746967 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7f1558066940 con 0x7f15700ced60 2024-09-17T03:53:38.546 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:38.546 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 msgr2=0x7f154c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 0x7f154c07b0a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1560009b80 tx=0x7f1560005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:38.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 msgr2=0x7f15700ca280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f155800c370 tx=0x7f155800c830 comp rx=0 tx=0).stop 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 shutdown_connections 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f154c078b90 0x7f154c07b0a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15701600c0 0x7f15700a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15700ced60 0x7f15700ca280 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.549+0000 7f1575800640 1 --2- 172.21.15.129:0/3484746967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1570072c30 0x7f15701f7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.550+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 >> 172.21.15.129:0/3484746967 conn(0x7f157006d410 msgr2=0x7f15700b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.550+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 shutdown_connections 2024-09-17T03:53:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.550+0000 7f1575800640 1 -- 172.21.15.129:0/3484746967 wait complete. 2024-09-17T03:53:38.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:38.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 -- 172.21.15.129:0/1285475320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d80d0500 msgr2=0x7f49d80cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 --2- 172.21.15.129:0/1285475320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d80d0500 0x7f49d80cecc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f49d8070e50 tx=0x7f49cc02aa10 comp rx=0 tx=0).stop 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 -- 172.21.15.129:0/1285475320 shutdown_connections 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 --2- 172.21.15.129:0/1285475320 >> v2:172.21.15.181:3300/0 conn(0x7f49d815bf40 0x7f49d815eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 --2- 172.21.15.129:0/1285475320 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80cf200 0x7f49d815ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 --2- 172.21.15.129:0/1285475320 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d80d0500 0x7f49d80cecc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 -- 172.21.15.129:0/1285475320 >> 172.21.15.129:0/1285475320 conn(0x7f49d806d870 msgr2=0x7f49d806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 -- 172.21.15.129:0/1285475320 shutdown_connections 2024-09-17T03:53:38.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.790+0000 7f49dee00640 1 -- 172.21.15.129:0/1285475320 wait complete. 2024-09-17T03:53:38.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.792+0000 7f49dee00640 1 Processor -- start 2024-09-17T03:53:38.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.793+0000 7f49dee00640 1 -- start start 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49d80cf200 0x7f49d81f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80d0500 0x7f49d8073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f49d815f530 con 0x7f49d815bf40 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f49d815c560 con 0x7f49d80cf200 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f49d815f6b0 con 0x7f49d80d0500 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49de400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49de400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49084/0 (socket says 172.21.15.129:49084) 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49de400640 1 -- 172.21.15.129:0/3868306918 learned_addr learned my addr 172.21.15.129:0/3868306918 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.794+0000 7f49dda00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49d80cf200 0x7f49d81f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49dd000640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80d0500 0x7f49d8073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 -- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80d0500 msgr2=0x7f49d8073ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80d0500 0x7f49d8073ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 -- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49d80cf200 msgr2=0x7f49d81f76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49d80cf200 0x7f49d81f76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 -- 172.21.15.129:0/3868306918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d81f36b0 con 0x7f49d815bf40 2024-09-17T03:53:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49de400640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f49d000d620 tx=0x7f49d000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0017070 con 0x7f49d815bf40 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.795+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49d0005310 con 0x7f49d815bf40 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.796+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00047a0 con 0x7f49d815bf40 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.796+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d81f39a0 con 0x7f49d815bf40 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.796+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d81f3f60 con 0x7f49d815bf40 2024-09-17T03:53:38.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.797+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d81f4620 con 0x7f49d815bf40 2024-09-17T03:53:38.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.798+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f49d0004940 con 0x7f49d815bf40 2024-09-17T03:53:38.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.799+0000 7f49d5600640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 0x7f49b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:38.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.799+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f49d009ab80 con 0x7f49d815bf40 2024-09-17T03:53:38.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.800+0000 7f49dda00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 0x7f49b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:38.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.800+0000 7f49dda00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 0x7f49b807b0a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f49cc035000 tx=0x7f49cc03b040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.803+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f49d81f4620 con 0x7f49d815bf40 2024-09-17T03:53:38.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.989+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49d815fb60 con 0x7f49d815bf40 2024-09-17T03:53:38.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.990+0000 7f49d5600640 1 -- 172.21.15.129:0/3868306918 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16768 (secure 0 0 0) 0x7f49d0062a00 con 0x7f49d815bf40 2024-09-17T03:53:38.989 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:38.989 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":81,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:36.047969+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:53:32.430935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-17T03:53:33.487308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.992+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 msgr2=0x7f49b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.992+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 0x7f49b807b0a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f49cc035000 tx=0x7f49cc03b040 comp rx=0 tx=0).stop 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.992+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 msgr2=0x7f49d80743e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.992+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f49d000d620 tx=0x7f49d000dae0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 shutdown_connections 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49b8078b90 0x7f49b807b0a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49d815bf40 0x7f49d80743e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49d80d0500 0x7f49d8073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 --2- 172.21.15.129:0/3868306918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49d80cf200 0x7f49d81f76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:38.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 >> 172.21.15.129:0/3868306918 conn(0x7f49d806d870 msgr2=0x7f49d80b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:38.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 shutdown_connections 2024-09-17T03:53:38.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:38.993+0000 7f49dee00640 1 -- 172.21.15.129:0/3868306918 wait complete. 2024-09-17T03:53:39.014 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:53:39.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 -- 172.21.15.129:0/2924158961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b8fd0 msgr2=0x7f84ec0b93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 --2- 172.21.15.129:0/2924158961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b8fd0 0x7f84ec0b93e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f84e4005fd0 tx=0x7f84e402a820 comp rx=0 tx=0).stop 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 -- 172.21.15.129:0/2924158961 shutdown_connections 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 --2- 172.21.15.129:0/2924158961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b8fd0 0x7f84ec0b93e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 --2- 172.21.15.129:0/2924158961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec162b50 0x7f84ec0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 -- 172.21.15.129:0/2924158961 >> 172.21.15.129:0/2924158961 conn(0x7f84ec09d6c0 msgr2=0x7f84ec0d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 -- 172.21.15.129:0/2924158961 shutdown_connections 2024-09-17T03:53:39.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.236+0000 7f84f2a00640 1 -- 172.21.15.129:0/2924158961 wait complete. 2024-09-17T03:53:39.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.239+0000 7f84f2a00640 1 Processor -- start 2024-09-17T03:53:39.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.239+0000 7f84f2a00640 1 -- start start 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec0b8fd0 0x7f84ec15ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b9bd0 0x7f84ec0d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec0b5360 con 0x7f84ec162b50 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f84ec0b51e0 con 0x7f84ec0b9bd0 2024-09-17T03:53:39.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f84ec0b54e0 con 0x7f84ec0b8fd0 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.240+0000 7f84f2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49090/0 (socket says 172.21.15.129:49090) 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 -- 172.21.15.129:0/1163546950 learned_addr learned my addr 172.21.15.129:0/1163546950 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f0c00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b9bd0 0x7f84ec0d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f1600640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec0b8fd0 0x7f84ec15ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 -- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec0b8fd0 msgr2=0x7f84ec15ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec0b8fd0 0x7f84ec15ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 -- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b9bd0 msgr2=0x7f84ec0d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b9bd0 0x7f84ec0d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 -- 172.21.15.129:0/1163546950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ec0c8240 con 0x7f84ec162b50 2024-09-17T03:53:39.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.241+0000 7f84f2000640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f84e4006010 tx=0x7f84e4032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.242+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4037040 con 0x7f84ec162b50 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.242+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f84e4042070 con 0x7f84ec162b50 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.242+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84ec0c8520 con 0x7f84ec162b50 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.242+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e40042a0 con 0x7f84ec162b50 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.242+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84ec0c8ae0 con 0x7f84ec162b50 2024-09-17T03:53:39.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.243+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84ec08cb00 con 0x7f84ec162b50 2024-09-17T03:53:39.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.244+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f84e4004440 con 0x7f84ec162b50 2024-09-17T03:53:39.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.245+0000 7f84e9600640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 0x7f84cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.245+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f84e40bba60 con 0x7f84ec162b50 2024-09-17T03:53:39.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.246+0000 7f84f1600640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 0x7f84cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.246+0000 7f84f1600640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 0x7f84cc07b0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f84e0001720 tx=0x7f84e0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:39.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.250+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f84ec08cb00 con 0x7f84ec162b50 2024-09-17T03:53:39.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.431+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f84ec0b59b0 con 0x7f84cc078bc0 2024-09-17T03:53:39.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.436+0000 7f84e9600640 1 -- 172.21.15.129:0/1163546950 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248034 (secure 0 0 0) 0x7f84ec0b59b0 con 0x7f84cc078bc0 2024-09-17T03:53:39.438 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:39.440 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 msgr2=0x7f84cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 0x7f84cc07b0d0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f84e0001720 tx=0x7f84e0009040 comp rx=0 tx=0).stop 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 msgr2=0x7f84ec0d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f84e4006010 tx=0x7f84e4032c90 comp rx=0 tx=0).stop 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 shutdown_connections 2024-09-17T03:53:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f84cc078bc0 0x7f84cc07b0d0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.440+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84ec162b50 0x7f84ec0d07d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.441+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84ec0b9bd0 0x7f84ec0d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.441+0000 7f84f2a00640 1 --2- 172.21.15.129:0/1163546950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84ec0b8fd0 0x7f84ec15ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.441+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 >> 172.21.15.129:0/1163546950 conn(0x7f84ec09d6c0 msgr2=0x7f84ec1617d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.441+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 shutdown_connections 2024-09-17T03:53:39.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.441+0000 7f84f2a00640 1 -- 172.21.15.129:0/1163546950 wait complete. 2024-09-17T03:53:39.465 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-17T03:53:38.359735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":131,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258184,"kb_used_data":8672,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716536,"statfs":{"total":773094113280,"available":772829732864,"internally_reserved":0,"allocated":8880128,"data_stored":4392868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.344257"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178823+0000","last_change":"2024-09-17T03:53:35.072458+0000","last_active":"2024-09-17T03:53:36.178823+0000","last_peered":"2024-09-17T03:53:36.178823+0000","last_clean":"2024-09-17T03:53:36.178823+0000","last_became_active":"2024-09-17T03:53:35.070660+0000","last_became_peered":"2024-09-17T03:53:35.070660+0000","last_unstale":"2024-09-17T03:53:36.178823+0000","last_undegraded":"2024-09-17T03:53:36.178823+0000","last_fullsized":"2024-09-17T03:53:36.178823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.505206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028869+0000","last_change":"2024-09-17T03:53:34.028869+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028869+0000","last_undegraded":"2024-09-17T03:53:34.028869+0000","last_fullsized":"2024-09-17T03:53:34.028869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033363+0000","last_change":"2024-09-17T03:53:34.033363+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033363+0000","last_undegraded":"2024-09-17T03:53:34.033363+0000","last_fullsized":"2024-09-17T03:53:34.033363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034036+0000","last_change":"2024-09-17T03:53:34.034036+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034036+0000","last_undegraded":"2024-09-17T03:53:34.034036+0000","last_fullsized":"2024-09-17T03:53:34.034036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031012+0000","last_change":"2024-09-17T03:53:34.031012+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031012+0000","last_undegraded":"2024-09-17T03:53:34.031012+0000","last_fullsized":"2024-09-17T03:53:34.031012+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.029155+0000","last_change":"2024-09-17T03:53:34.029155+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.029155+0000","last_undegraded":"2024-09-17T03:53:34.029155+0000","last_fullsized":"2024-09-17T03:53:34.029155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033193+0000","last_change":"2024-09-17T03:53:34.033193+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033193+0000","last_undegraded":"2024-09-17T03:53:34.033193+0000","last_fullsized":"2024-09-17T03:53:34.033193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027576+0000","last_change":"2024-09-17T03:53:34.027576+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027576+0000","last_undegraded":"2024-09-17T03:53:34.027576+0000","last_fullsized":"2024-09-17T03:53:34.027576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026926+0000","last_change":"2024-09-17T03:53:34.026926+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026926+0000","last_undegraded":"2024-09-17T03:53:34.026926+0000","last_fullsized":"2024-09-17T03:53:34.026926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178936+0000","last_change":"2024-09-17T03:53:35.075213+0000","last_active":"2024-09-17T03:53:36.178936+0000","last_peered":"2024-09-17T03:53:36.178936+0000","last_clean":"2024-09-17T03:53:36.178936+0000","last_became_active":"2024-09-17T03:53:35.074344+0000","last_became_peered":"2024-09-17T03:53:35.074344+0000","last_unstale":"2024-09-17T03:53:36.178936+0000","last_undegraded":"2024-09-17T03:53:36.178936+0000","last_fullsized":"2024-09-17T03:53:36.178936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:02.543171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.025977+0000","last_change":"2024-09-17T03:53:34.025977+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.025977+0000","last_undegraded":"2024-09-17T03:53:34.025977+0000","last_fullsized":"2024-09-17T03:53:34.025977+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178709+0000","last_change":"2024-09-17T03:53:35.073592+0000","last_active":"2024-09-17T03:53:36.178709+0000","last_peered":"2024-09-17T03:53:36.178709+0000","last_clean":"2024-09-17T03:53:36.178709+0000","last_became_active":"2024-09-17T03:53:35.073318+0000","last_became_peered":"2024-09-17T03:53:35.073318+0000","last_unstale":"2024-09-17T03:53:36.178709+0000","last_undegraded":"2024-09-17T03:53:36.178709+0000","last_fullsized":"2024-09-17T03:53:36.178709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:26.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032082+0000","last_change":"2024-09-17T03:53:34.032082+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032082+0000","last_undegraded":"2024-09-17T03:53:34.032082+0000","last_fullsized":"2024-09-17T03:53:34.032082+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036001+0000","last_change":"2024-09-17T03:53:34.036001+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036001+0000","last_undegraded":"2024-09-17T03:53:34.036001+0000","last_fullsized":"2024-09-17T03:53:34.036001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027512+0000","last_change":"2024-09-17T03:53:34.027512+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027512+0000","last_undegraded":"2024-09-17T03:53:34.027512+0000","last_fullsized":"2024-09-17T03:53:34.027512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034599+0000","last_change":"2024-09-17T03:53:34.034599+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034599+0000","last_undegraded":"2024-09-17T03:53:34.034599+0000","last_fullsized":"2024-09-17T03:53:34.034599+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.030598+0000","last_change":"2024-09-17T03:53:34.030598+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.030598+0000","last_undegraded":"2024-09-17T03:53:34.030598+0000","last_fullsized":"2024-09-17T03:53:34.030598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037042+0000","last_change":"2024-09-17T03:53:34.037042+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037042+0000","last_undegraded":"2024-09-17T03:53:34.037042+0000","last_fullsized":"2024-09-17T03:53:34.037042+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529477+0000","last_change":"2024-09-17T03:53:35.077049+0000","last_active":"2024-09-17T03:53:36.529477+0000","last_peered":"2024-09-17T03:53:36.529477+0000","last_clean":"2024-09-17T03:53:36.529477+0000","last_became_active":"2024-09-17T03:53:35.076178+0000","last_became_peered":"2024-09-17T03:53:35.076178+0000","last_unstale":"2024-09-17T03:53:36.529477+0000","last_undegraded":"2024-09-17T03:53:36.529477+0000","last_fullsized":"2024-09-17T03:53:36.529477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033561+0000","last_change":"2024-09-17T03:53:34.033561+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033561+0000","last_undegraded":"2024-09-17T03:53:34.033561+0000","last_fullsized":"2024-09-17T03:53:34.033561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034737+0000","last_change":"2024-09-17T03:53:34.034737+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034737+0000","last_undegraded":"2024-09-17T03:53:34.034737+0000","last_fullsized":"2024-09-17T03:53:34.034737+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.029254+0000","last_change":"2024-09-17T03:53:34.029254+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.029254+0000","last_undegraded":"2024-09-17T03:53:34.029254+0000","last_fullsized":"2024-09-17T03:53:34.029254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178654+0000","last_change":"2024-09-17T03:53:35.068757+0000","last_active":"2024-09-17T03:53:36.178654+0000","last_peered":"2024-09-17T03:53:36.178654+0000","last_clean":"2024-09-17T03:53:36.178654+0000","last_became_active":"2024-09-17T03:53:35.068235+0000","last_became_peered":"2024-09-17T03:53:35.068235+0000","last_unstale":"2024-09-17T03:53:36.178654+0000","last_undegraded":"2024-09-17T03:53:36.178654+0000","last_fullsized":"2024-09-17T03:53:36.178654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:16.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037547+0000","last_change":"2024-09-17T03:53:34.037547+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037547+0000","last_undegraded":"2024-09-17T03:53:34.037547+0000","last_fullsized":"2024-09-17T03:53:34.037547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027005+0000","last_change":"2024-09-17T03:53:34.027005+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027005+0000","last_undegraded":"2024-09-17T03:53:34.027005+0000","last_fullsized":"2024-09-17T03:53:34.027005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033556+0000","last_change":"2024-09-17T03:53:34.033556+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033556+0000","last_undegraded":"2024-09-17T03:53:34.033556+0000","last_fullsized":"2024-09-17T03:53:34.033556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034217+0000","last_change":"2024-09-17T03:53:34.034217+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034217+0000","last_undegraded":"2024-09-17T03:53:34.034217+0000","last_fullsized":"2024-09-17T03:53:34.034217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178706+0000","last_change":"2024-09-17T03:53:35.067698+0000","last_active":"2024-09-17T03:53:36.178706+0000","last_peered":"2024-09-17T03:53:36.178706+0000","last_clean":"2024-09-17T03:53:36.178706+0000","last_became_active":"2024-09-17T03:53:35.066584+0000","last_became_peered":"2024-09-17T03:53:35.066584+0000","last_unstale":"2024-09-17T03:53:36.178706+0000","last_undegraded":"2024-09-17T03:53:36.178706+0000","last_fullsized":"2024-09-17T03:53:36.178706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:57.060788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026812+0000","last_change":"2024-09-17T03:53:34.026812+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026812+0000","last_undegraded":"2024-09-17T03:53:34.026812+0000","last_fullsized":"2024-09-17T03:53:34.026812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028702+0000","last_change":"2024-09-17T03:53:34.028702+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028702+0000","last_undegraded":"2024-09-17T03:53:34.028702+0000","last_fullsized":"2024-09-17T03:53:34.028702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036230+0000","last_change":"2024-09-17T03:53:34.036230+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036230+0000","last_undegraded":"2024-09-17T03:53:34.036230+0000","last_fullsized":"2024-09-17T03:53:34.036230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178515+0000","last_change":"2024-09-17T03:53:35.062010+0000","last_active":"2024-09-17T03:53:36.178515+0000","last_peered":"2024-09-17T03:53:36.178515+0000","last_clean":"2024-09-17T03:53:36.178515+0000","last_became_active":"2024-09-17T03:53:35.059874+0000","last_became_peered":"2024-09-17T03:53:35.059874+0000","last_unstale":"2024-09-17T03:53:36.178515+0000","last_undegraded":"2024-09-17T03:53:36.178515+0000","last_fullsized":"2024-09-17T03:53:36.178515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:26.933349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036366+0000","last_change":"2024-09-17T03:53:34.036366+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036366+0000","last_undegraded":"2024-09-17T03:53:34.036366+0000","last_fullsized":"2024-09-17T03:53:34.036366+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178966+0000","last_change":"2024-09-17T03:53:34.042427+0000","last_active":"2024-09-17T03:53:36.178966+0000","last_peered":"2024-09-17T03:53:36.178966+0000","last_clean":"2024-09-17T03:53:36.178966+0000","last_became_active":"2024-09-17T03:53:34.042117+0000","last_became_peered":"2024-09-17T03:53:34.042117+0000","last_unstale":"2024-09-17T03:53:36.178966+0000","last_undegraded":"2024-09-17T03:53:36.178966+0000","last_fullsized":"2024-09-17T03:53:36.178966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:57.275556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178860+0000","last_change":"2024-09-17T03:53:35.075209+0000","last_active":"2024-09-17T03:53:36.178860+0000","last_peered":"2024-09-17T03:53:36.178860+0000","last_clean":"2024-09-17T03:53:36.178860+0000","last_became_active":"2024-09-17T03:53:35.074557+0000","last_became_peered":"2024-09-17T03:53:35.074557+0000","last_unstale":"2024-09-17T03:53:36.178860+0000","last_undegraded":"2024-09-17T03:53:36.178860+0000","last_fullsized":"2024-09-17T03:53:36.178860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:26.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049450+0000","last_change":"2024-09-17T03:53:34.047657+0000","last_active":"2024-09-17T03:53:34.049450+0000","last_peered":"2024-09-17T03:53:34.049450+0000","last_clean":"2024-09-17T03:53:34.049450+0000","last_became_active":"2024-09-17T03:53:34.047290+0000","last_became_peered":"2024-09-17T03:53:34.047290+0000","last_unstale":"2024-09-17T03:53:34.049450+0000","last_undegraded":"2024-09-17T03:53:34.049450+0000","last_fullsized":"2024-09-17T03:53:34.049450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:32:37.965629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178967+0000","last_change":"2024-09-17T03:53:35.068246+0000","last_active":"2024-09-17T03:53:36.178967+0000","last_peered":"2024-09-17T03:53:36.178967+0000","last_clean":"2024-09-17T03:53:36.178967+0000","last_became_active":"2024-09-17T03:53:35.066866+0000","last_became_peered":"2024-09-17T03:53:35.066866+0000","last_unstale":"2024-09-17T03:53:36.178967+0000","last_undegraded":"2024-09-17T03:53:36.178967+0000","last_fullsized":"2024-09-17T03:53:36.178967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:59.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529312+0000","last_change":"2024-09-17T03:53:34.047098+0000","last_active":"2024-09-17T03:53:36.529312+0000","last_peered":"2024-09-17T03:53:36.529312+0000","last_clean":"2024-09-17T03:53:36.529312+0000","last_became_active":"2024-09-17T03:53:34.046816+0000","last_became_peered":"2024-09-17T03:53:34.046816+0000","last_unstale":"2024-09-17T03:53:36.529312+0000","last_undegraded":"2024-09-17T03:53:36.529312+0000","last_fullsized":"2024-09-17T03:53:36.529312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:25.856611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032835+0000","last_change":"2024-09-17T03:53:34.032835+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032835+0000","last_undegraded":"2024-09-17T03:53:34.032835+0000","last_fullsized":"2024-09-17T03:53:34.032835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529204+0000","last_change":"2024-09-17T03:53:34.047320+0000","last_active":"2024-09-17T03:53:36.529204+0000","last_peered":"2024-09-17T03:53:36.529204+0000","last_clean":"2024-09-17T03:53:36.529204+0000","last_became_active":"2024-09-17T03:53:34.045973+0000","last_became_peered":"2024-09-17T03:53:34.045973+0000","last_unstale":"2024-09-17T03:53:36.529204+0000","last_undegraded":"2024-09-17T03:53:36.529204+0000","last_fullsized":"2024-09-17T03:53:36.529204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:03.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028186+0000","last_change":"2024-09-17T03:53:34.028186+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028186+0000","last_undegraded":"2024-09-17T03:53:34.028186+0000","last_fullsized":"2024-09-17T03:53:34.028186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049780+0000","last_change":"2024-09-17T03:53:34.046521+0000","last_active":"2024-09-17T03:53:34.049780+0000","last_peered":"2024-09-17T03:53:34.049780+0000","last_clean":"2024-09-17T03:53:34.049780+0000","last_became_active":"2024-09-17T03:53:34.046151+0000","last_became_peered":"2024-09-17T03:53:34.046151+0000","last_unstale":"2024-09-17T03:53:34.049780+0000","last_undegraded":"2024-09-17T03:53:34.049780+0000","last_fullsized":"2024-09-17T03:53:34.049780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:01.949777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.030392+0000","last_change":"2024-09-17T03:53:34.030392+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.030392+0000","last_undegraded":"2024-09-17T03:53:34.030392+0000","last_fullsized":"2024-09-17T03:53:34.030392+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048684+0000","last_change":"2024-09-17T03:53:34.045410+0000","last_active":"2024-09-17T03:53:34.048684+0000","last_peered":"2024-09-17T03:53:34.048684+0000","last_clean":"2024-09-17T03:53:34.048684+0000","last_became_active":"2024-09-17T03:53:34.045030+0000","last_became_peered":"2024-09-17T03:53:34.045030+0000","last_unstale":"2024-09-17T03:53:34.048684+0000","last_undegraded":"2024-09-17T03:53:34.048684+0000","last_fullsized":"2024-09-17T03:53:34.048684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:23.455684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031612+0000","last_change":"2024-09-17T03:53:34.031612+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031612+0000","last_undegraded":"2024-09-17T03:53:34.031612+0000","last_fullsized":"2024-09-17T03:53:34.031612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528363+0000","last_change":"2024-09-17T03:53:34.040513+0000","last_active":"2024-09-17T03:53:36.528363+0000","last_peered":"2024-09-17T03:53:36.528363+0000","last_clean":"2024-09-17T03:53:36.528363+0000","last_became_active":"2024-09-17T03:53:34.039387+0000","last_became_peered":"2024-09-17T03:53:34.039387+0000","last_unstale":"2024-09-17T03:53:36.528363+0000","last_undegraded":"2024-09-17T03:53:36.528363+0000","last_fullsized":"2024-09-17T03:53:36.528363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:35.514048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024952+0000","last_change":"2024-09-17T03:53:34.024952+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024952+0000","last_undegraded":"2024-09-17T03:53:34.024952+0000","last_fullsized":"2024-09-17T03:53:34.024952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.045346+0000","last_change":"2024-09-17T03:53:34.042323+0000","last_active":"2024-09-17T03:53:34.045346+0000","last_peered":"2024-09-17T03:53:34.045346+0000","last_clean":"2024-09-17T03:53:34.045346+0000","last_became_active":"2024-09-17T03:53:34.041995+0000","last_became_peered":"2024-09-17T03:53:34.041995+0000","last_unstale":"2024-09-17T03:53:34.045346+0000","last_undegraded":"2024-09-17T03:53:34.045346+0000","last_fullsized":"2024-09-17T03:53:34.045346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:32.733482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024710+0000","last_change":"2024-09-17T03:53:34.024710+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024710+0000","last_undegraded":"2024-09-17T03:53:34.024710+0000","last_fullsized":"2024-09-17T03:53:34.024710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179093+0000","last_change":"2024-09-17T03:53:34.041277+0000","last_active":"2024-09-17T03:53:36.179093+0000","last_peered":"2024-09-17T03:53:36.179093+0000","last_clean":"2024-09-17T03:53:36.179093+0000","last_became_active":"2024-09-17T03:53:34.040555+0000","last_became_peered":"2024-09-17T03:53:34.040555+0000","last_unstale":"2024-09-17T03:53:36.179093+0000","last_undegraded":"2024-09-17T03:53:36.179093+0000","last_fullsized":"2024-09-17T03:53:36.179093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:05.315114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024763+0000","last_change":"2024-09-17T03:53:34.024763+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024763+0000","last_undegraded":"2024-09-17T03:53:34.024763+0000","last_fullsized":"2024-09-17T03:53:34.024763+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178778+0000","last_change":"2024-09-17T03:53:34.047020+0000","last_active":"2024-09-17T03:53:36.178778+0000","last_peered":"2024-09-17T03:53:36.178778+0000","last_clean":"2024-09-17T03:53:36.178778+0000","last_became_active":"2024-09-17T03:53:34.045487+0000","last_became_peered":"2024-09-17T03:53:34.045487+0000","last_unstale":"2024-09-17T03:53:36.178778+0000","last_undegraded":"2024-09-17T03:53:36.178778+0000","last_fullsized":"2024-09-17T03:53:36.178778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:03.205170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529300+0000","last_change":"2024-09-17T03:53:35.074637+0000","last_active":"2024-09-17T03:53:36.529300+0000","last_peered":"2024-09-17T03:53:36.529300+0000","last_clean":"2024-09-17T03:53:36.529300+0000","last_became_active":"2024-09-17T03:53:35.074086+0000","last_became_peered":"2024-09-17T03:53:35.074086+0000","last_unstale":"2024-09-17T03:53:36.529300+0000","last_undegraded":"2024-09-17T03:53:36.529300+0000","last_fullsized":"2024-09-17T03:53:36.529300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.027879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178381+0000","last_change":"2024-09-17T03:53:34.045567+0000","last_active":"2024-09-17T03:53:36.178381+0000","last_peered":"2024-09-17T03:53:36.178381+0000","last_clean":"2024-09-17T03:53:36.178381+0000","last_became_active":"2024-09-17T03:53:34.045029+0000","last_became_peered":"2024-09-17T03:53:34.045029+0000","last_unstale":"2024-09-17T03:53:36.178381+0000","last_undegraded":"2024-09-17T03:53:36.178381+0000","last_fullsized":"2024-09-17T03:53:36.178381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:38.299845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033101+0000","last_change":"2024-09-17T03:53:34.033101+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033101+0000","last_undegraded":"2024-09-17T03:53:34.033101+0000","last_fullsized":"2024-09-17T03:53:34.033101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.038363+0000","last_change":"2024-09-17T03:53:34.033143+0000","last_active":"2024-09-17T03:53:34.038363+0000","last_peered":"2024-09-17T03:53:34.038363+0000","last_clean":"2024-09-17T03:53:34.038363+0000","last_became_active":"2024-09-17T03:53:34.031770+0000","last_became_peered":"2024-09-17T03:53:34.031770+0000","last_unstale":"2024-09-17T03:53:34.038363+0000","last_undegraded":"2024-09-17T03:53:34.038363+0000","last_fullsized":"2024-09-17T03:53:34.038363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:20.987901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529438+0000","last_change":"2024-09-17T03:53:35.062155+0000","last_active":"2024-09-17T03:53:36.529438+0000","last_peered":"2024-09-17T03:53:36.529438+0000","last_clean":"2024-09-17T03:53:36.529438+0000","last_became_active":"2024-09-17T03:53:35.061536+0000","last_became_peered":"2024-09-17T03:53:35.061536+0000","last_unstale":"2024-09-17T03:53:36.529438+0000","last_undegraded":"2024-09-17T03:53:36.529438+0000","last_fullsized":"2024-09-17T03:53:36.529438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:36.274547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.045382+0000","last_change":"2024-09-17T03:53:34.043064+0000","last_active":"2024-09-17T03:53:34.045382+0000","last_peered":"2024-09-17T03:53:34.045382+0000","last_clean":"2024-09-17T03:53:34.045382+0000","last_became_active":"2024-09-17T03:53:34.038696+0000","last_became_peered":"2024-09-17T03:53:34.038696+0000","last_unstale":"2024-09-17T03:53:34.045382+0000","last_undegraded":"2024-09-17T03:53:34.045382+0000","last_fullsized":"2024-09-17T03:53:34.045382+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:40.436716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031491+0000","last_change":"2024-09-17T03:53:34.031491+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031491+0000","last_undegraded":"2024-09-17T03:53:34.031491+0000","last_fullsized":"2024-09-17T03:53:34.031491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050885+0000","last_change":"2024-09-17T03:53:34.048029+0000","last_active":"2024-09-17T03:53:34.050885+0000","last_peered":"2024-09-17T03:53:34.050885+0000","last_clean":"2024-09-17T03:53:34.050885+0000","last_became_active":"2024-09-17T03:53:34.046461+0000","last_became_peered":"2024-09-17T03:53:34.046461+0000","last_unstale":"2024-09-17T03:53:34.050885+0000","last_undegraded":"2024-09-17T03:53:34.050885+0000","last_fullsized":"2024-09-17T03:53:34.050885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:27.848560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026428+0000","last_change":"2024-09-17T03:53:34.026428+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026428+0000","last_undegraded":"2024-09-17T03:53:34.026428+0000","last_fullsized":"2024-09-17T03:53:34.026428+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048911+0000","last_change":"2024-09-17T03:53:34.045446+0000","last_active":"2024-09-17T03:53:34.048911+0000","last_peered":"2024-09-17T03:53:34.048911+0000","last_clean":"2024-09-17T03:53:34.048911+0000","last_became_active":"2024-09-17T03:53:34.045103+0000","last_became_peered":"2024-09-17T03:53:34.045103+0000","last_unstale":"2024-09-17T03:53:34.048911+0000","last_undegraded":"2024-09-17T03:53:34.048911+0000","last_fullsized":"2024-09-17T03:53:34.048911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:49.049002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027698+0000","last_change":"2024-09-17T03:53:34.027698+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027698+0000","last_undegraded":"2024-09-17T03:53:34.027698+0000","last_fullsized":"2024-09-17T03:53:34.027698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528873+0000","last_change":"2024-09-17T03:53:34.043445+0000","last_active":"2024-09-17T03:53:36.528873+0000","last_peered":"2024-09-17T03:53:36.528873+0000","last_clean":"2024-09-17T03:53:36.528873+0000","last_became_active":"2024-09-17T03:53:34.042920+0000","last_became_peered":"2024-09-17T03:53:34.042920+0000","last_unstale":"2024-09-17T03:53:36.528873+0000","last_undegraded":"2024-09-17T03:53:36.528873+0000","last_fullsized":"2024-09-17T03:53:36.528873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:42.885526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032446+0000","last_change":"2024-09-17T03:53:34.032446+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032446+0000","last_undegraded":"2024-09-17T03:53:34.032446+0000","last_fullsized":"2024-09-17T03:53:34.032446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179205+0000","last_change":"2024-09-17T03:53:34.041969+0000","last_active":"2024-09-17T03:53:36.179205+0000","last_peered":"2024-09-17T03:53:36.179205+0000","last_clean":"2024-09-17T03:53:36.179205+0000","last_became_active":"2024-09-17T03:53:34.040090+0000","last_became_peered":"2024-09-17T03:53:34.040090+0000","last_unstale":"2024-09-17T03:53:36.179205+0000","last_undegraded":"2024-09-17T03:53:36.179205+0000","last_fullsized":"2024-09-17T03:53:36.179205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:12.790859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179104+0000","last_change":"2024-09-17T03:53:35.071124+0000","last_active":"2024-09-17T03:53:36.179104+0000","last_peered":"2024-09-17T03:53:36.179104+0000","last_clean":"2024-09-17T03:53:36.179104+0000","last_became_active":"2024-09-17T03:53:35.070636+0000","last_became_peered":"2024-09-17T03:53:35.070636+0000","last_unstale":"2024-09-17T03:53:36.179104+0000","last_undegraded":"2024-09-17T03:53:36.179104+0000","last_fullsized":"2024-09-17T03:53:36.179104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:18.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529575+0000","last_change":"2024-09-17T03:53:34.049033+0000","last_active":"2024-09-17T03:53:36.529575+0000","last_peered":"2024-09-17T03:53:36.529575+0000","last_clean":"2024-09-17T03:53:36.529575+0000","last_became_active":"2024-09-17T03:53:34.048596+0000","last_became_peered":"2024-09-17T03:53:34.048596+0000","last_unstale":"2024-09-17T03:53:36.529575+0000","last_undegraded":"2024-09-17T03:53:36.529575+0000","last_fullsized":"2024-09-17T03:53:36.529575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:25.851358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032593+0000","last_change":"2024-09-17T03:53:34.032593+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032593+0000","last_undegraded":"2024-09-17T03:53:34.032593+0000","last_fullsized":"2024-09-17T03:53:34.032593+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529714+0000","last_change":"2024-09-17T03:53:34.040065+0000","last_active":"2024-09-17T03:53:36.529714+0000","last_peered":"2024-09-17T03:53:36.529714+0000","last_clean":"2024-09-17T03:53:36.529714+0000","last_became_active":"2024-09-17T03:53:34.039675+0000","last_became_peered":"2024-09-17T03:53:34.039675+0000","last_unstale":"2024-09-17T03:53:36.529714+0000","last_undegraded":"2024-09-17T03:53:36.529714+0000","last_fullsized":"2024-09-17T03:53:36.529714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:40.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.035557+0000","last_change":"2024-09-17T03:53:34.035557+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.035557+0000","last_undegraded":"2024-09-17T03:53:34.035557+0000","last_fullsized":"2024-09-17T03:53:34.035557+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050120+0000","last_change":"2024-09-17T03:53:34.042295+0000","last_active":"2024-09-17T03:53:34.050120+0000","last_peered":"2024-09-17T03:53:34.050120+0000","last_clean":"2024-09-17T03:53:34.050120+0000","last_became_active":"2024-09-17T03:53:34.041992+0000","last_became_peered":"2024-09-17T03:53:34.041992+0000","last_unstale":"2024-09-17T03:53:34.050120+0000","last_undegraded":"2024-09-17T03:53:34.050120+0000","last_fullsized":"2024-09-17T03:53:34.050120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:10.329340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027458+0000","last_change":"2024-09-17T03:53:34.027458+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027458+0000","last_undegraded":"2024-09-17T03:53:34.027458+0000","last_fullsized":"2024-09-17T03:53:34.027458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.054928+0000","last_change":"2024-09-17T03:53:34.050809+0000","last_active":"2024-09-17T03:53:34.054928+0000","last_peered":"2024-09-17T03:53:34.054928+0000","last_clean":"2024-09-17T03:53:34.054928+0000","last_became_active":"2024-09-17T03:53:34.050498+0000","last_became_peered":"2024-09-17T03:53:34.050498+0000","last_unstale":"2024-09-17T03:53:34.054928+0000","last_undegraded":"2024-09-17T03:53:34.054928+0000","last_fullsized":"2024-09-17T03:53:34.054928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:13:23.876004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037100+0000","last_change":"2024-09-17T03:53:34.037100+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037100+0000","last_undegraded":"2024-09-17T03:53:34.037100+0000","last_fullsized":"2024-09-17T03:53:34.037100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049807+0000","last_change":"2024-09-17T03:53:34.046054+0000","last_active":"2024-09-17T03:53:34.049807+0000","last_peered":"2024-09-17T03:53:34.049807+0000","last_clean":"2024-09-17T03:53:34.049807+0000","last_became_active":"2024-09-17T03:53:34.045718+0000","last_became_peered":"2024-09-17T03:53:34.045718+0000","last_unstale":"2024-09-17T03:53:34.049807+0000","last_undegraded":"2024-09-17T03:53:34.049807+0000","last_fullsized":"2024-09-17T03:53:34.049807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:11.308714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.025130+0000","last_change":"2024-09-17T03:53:34.025130+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.025130+0000","last_undegraded":"2024-09-17T03:53:34.025130+0000","last_fullsized":"2024-09-17T03:53:34.025130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050129+0000","last_change":"2024-09-17T03:53:34.045835+0000","last_active":"2024-09-17T03:53:34.050129+0000","last_peered":"2024-09-17T03:53:34.050129+0000","last_clean":"2024-09-17T03:53:34.050129+0000","last_became_active":"2024-09-17T03:53:34.045329+0000","last_became_peered":"2024-09-17T03:53:34.045329+0000","last_unstale":"2024-09-17T03:53:34.050129+0000","last_undegraded":"2024-09-17T03:53:34.050129+0000","last_fullsized":"2024-09-17T03:53:34.050129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:36.472408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.035850+0000","last_change":"2024-09-17T03:53:34.035850+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.035850+0000","last_undegraded":"2024-09-17T03:53:34.035850+0000","last_fullsized":"2024-09-17T03:53:34.035850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.046315+0000","last_change":"2024-09-17T03:53:34.043457+0000","last_active":"2024-09-17T03:53:34.046315+0000","last_peered":"2024-09-17T03:53:34.046315+0000","last_clean":"2024-09-17T03:53:34.046315+0000","last_became_active":"2024-09-17T03:53:34.043142+0000","last_became_peered":"2024-09-17T03:53:34.043142+0000","last_unstale":"2024-09-17T03:53:34.046315+0000","last_undegraded":"2024-09-17T03:53:34.046315+0000","last_fullsized":"2024-09-17T03:53:34.046315+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:08.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033549+0000","last_change":"2024-09-17T03:53:34.033549+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033549+0000","last_undegraded":"2024-09-17T03:53:34.033549+0000","last_fullsized":"2024-09-17T03:53:34.033549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049609+0000","last_change":"2024-09-17T03:53:34.047344+0000","last_active":"2024-09-17T03:53:34.049609+0000","last_peered":"2024-09-17T03:53:34.049609+0000","last_clean":"2024-09-17T03:53:34.049609+0000","last_became_active":"2024-09-17T03:53:34.041909+0000","last_became_peered":"2024-09-17T03:53:34.041909+0000","last_unstale":"2024-09-17T03:53:34.049609+0000","last_undegraded":"2024-09-17T03:53:34.049609+0000","last_fullsized":"2024-09-17T03:53:34.049609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:50.302474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179022+0000","last_change":"2024-09-17T03:53:35.060992+0000","last_active":"2024-09-17T03:53:36.179022+0000","last_peered":"2024-09-17T03:53:36.179022+0000","last_clean":"2024-09-17T03:53:36.179022+0000","last_became_active":"2024-09-17T03:53:35.060339+0000","last_became_peered":"2024-09-17T03:53:35.060339+0000","last_unstale":"2024-09-17T03:53:36.179022+0000","last_undegraded":"2024-09-17T03:53:36.179022+0000","last_fullsized":"2024-09-17T03:53:36.179022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:48.433949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178476+0000","last_change":"2024-09-17T03:53:34.042515+0000","last_active":"2024-09-17T03:53:36.178476+0000","last_peered":"2024-09-17T03:53:36.178476+0000","last_clean":"2024-09-17T03:53:36.178476+0000","last_became_active":"2024-09-17T03:53:34.041350+0000","last_became_peered":"2024-09-17T03:53:34.041350+0000","last_unstale":"2024-09-17T03:53:36.178476+0000","last_undegraded":"2024-09-17T03:53:36.178476+0000","last_fullsized":"2024-09-17T03:53:36.178476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:54.121078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031367+0000","last_change":"2024-09-17T03:53:34.031367+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031367+0000","last_undegraded":"2024-09-17T03:53:34.031367+0000","last_fullsized":"2024-09-17T03:53:34.031367+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179021+0000","last_change":"2024-09-17T03:53:34.043860+0000","last_active":"2024-09-17T03:53:36.179021+0000","last_peered":"2024-09-17T03:53:36.179021+0000","last_clean":"2024-09-17T03:53:36.179021+0000","last_became_active":"2024-09-17T03:53:34.043502+0000","last_became_peered":"2024-09-17T03:53:34.043502+0000","last_unstale":"2024-09-17T03:53:36.179021+0000","last_undegraded":"2024-09-17T03:53:36.179021+0000","last_fullsized":"2024-09-17T03:53:36.179021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:02.362312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024331+0000","last_change":"2024-09-17T03:53:34.024331+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024331+0000","last_undegraded":"2024-09-17T03:53:34.024331+0000","last_fullsized":"2024-09-17T03:53:34.024331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529388+0000","last_change":"2024-09-17T03:53:34.037735+0000","last_active":"2024-09-17T03:53:36.529388+0000","last_peered":"2024-09-17T03:53:36.529388+0000","last_clean":"2024-09-17T03:53:36.529388+0000","last_became_active":"2024-09-17T03:53:34.037448+0000","last_became_peered":"2024-09-17T03:53:34.037448+0000","last_unstale":"2024-09-17T03:53:36.529388+0000","last_undegraded":"2024-09-17T03:53:36.529388+0000","last_fullsized":"2024-09-17T03:53:36.529388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:52.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529554+0000","last_change":"2024-09-17T03:53:35.071423+0000","last_active":"2024-09-17T03:53:36.529554+0000","last_peered":"2024-09-17T03:53:36.529554+0000","last_clean":"2024-09-17T03:53:36.529554+0000","last_became_active":"2024-09-17T03:53:35.071008+0000","last_became_peered":"2024-09-17T03:53:35.071008+0000","last_unstale":"2024-09-17T03:53:36.529554+0000","last_undegraded":"2024-09-17T03:53:36.529554+0000","last_fullsized":"2024-09-17T03:53:36.529554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:43.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049058+0000","last_change":"2024-09-17T03:53:34.045408+0000","last_active":"2024-09-17T03:53:34.049058+0000","last_peered":"2024-09-17T03:53:34.049058+0000","last_clean":"2024-09-17T03:53:34.049058+0000","last_became_active":"2024-09-17T03:53:34.045021+0000","last_became_peered":"2024-09-17T03:53:34.045021+0000","last_unstale":"2024-09-17T03:53:34.049058+0000","last_undegraded":"2024-09-17T03:53:34.049058+0000","last_fullsized":"2024-09-17T03:53:34.049058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:34.908146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031016+0000","last_change":"2024-09-17T03:53:34.031016+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031016+0000","last_undegraded":"2024-09-17T03:53:34.031016+0000","last_fullsized":"2024-09-17T03:53:34.031016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178647+0000","last_change":"2024-09-17T03:53:34.041865+0000","last_active":"2024-09-17T03:53:36.178647+0000","last_peered":"2024-09-17T03:53:36.178647+0000","last_clean":"2024-09-17T03:53:36.178647+0000","last_became_active":"2024-09-17T03:53:34.041169+0000","last_became_peered":"2024-09-17T03:53:34.041169+0000","last_unstale":"2024-09-17T03:53:36.178647+0000","last_undegraded":"2024-09-17T03:53:36.178647+0000","last_fullsized":"2024-09-17T03:53:36.178647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:16.044280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178489+0000","last_change":"2024-09-17T03:53:35.064940+0000","last_active":"2024-09-17T03:53:36.178489+0000","last_peered":"2024-09-17T03:53:36.178489+0000","last_clean":"2024-09-17T03:53:36.178489+0000","last_became_active":"2024-09-17T03:53:35.064574+0000","last_became_peered":"2024-09-17T03:53:35.064574+0000","last_unstale":"2024-09-17T03:53:36.178489+0000","last_undegraded":"2024-09-17T03:53:36.178489+0000","last_fullsized":"2024-09-17T03:53:36.178489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:08.160089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049928+0000","last_change":"2024-09-17T03:53:34.048127+0000","last_active":"2024-09-17T03:53:34.049928+0000","last_peered":"2024-09-17T03:53:34.049928+0000","last_clean":"2024-09-17T03:53:34.049928+0000","last_became_active":"2024-09-17T03:53:34.047768+0000","last_became_peered":"2024-09-17T03:53:34.047768+0000","last_unstale":"2024-09-17T03:53:34.049928+0000","last_undegraded":"2024-09-17T03:53:34.049928+0000","last_fullsized":"2024-09-17T03:53:34.049928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:24.000468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027124+0000","last_change":"2024-09-17T03:53:34.027124+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027124+0000","last_undegraded":"2024-09-17T03:53:34.027124+0000","last_fullsized":"2024-09-17T03:53:34.027124+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048868+0000","last_change":"2024-09-17T03:53:34.046914+0000","last_active":"2024-09-17T03:53:34.048868+0000","last_peered":"2024-09-17T03:53:34.048868+0000","last_clean":"2024-09-17T03:53:34.048868+0000","last_became_active":"2024-09-17T03:53:34.046056+0000","last_became_peered":"2024-09-17T03:53:34.046056+0000","last_unstale":"2024-09-17T03:53:34.048868+0000","last_undegraded":"2024-09-17T03:53:34.048868+0000","last_fullsized":"2024-09-17T03:53:34.048868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:21.605697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":215,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.036269+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:53:34.036269+0000","last_peered":"2024-09-17T03:53:34.036269+0000","last_clean":"2024-09-17T03:53:34.036269+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:53:34.036269+0000","last_undegraded":"2024-09-17T03:53:34.036269+0000","last_fullsized":"2024-09-17T03:53:34.036269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509492,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1544192,"data_stored":986956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509494,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542197,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":999424,"data_stored":416632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542198,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:53:39.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:53:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 -- 172.21.15.129:0/3016704539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244073040 msgr2=0x7fb24415f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 --2- 172.21.15.129:0/3016704539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244073040 0x7fb24415f490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb244070fa0 tx=0x7fb23c02aa30 comp rx=0 tx=0).stop 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 -- 172.21.15.129:0/3016704539 shutdown_connections 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 --2- 172.21.15.129:0/3016704539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244073040 0x7fb24415f490 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 --2- 172.21.15.129:0/3016704539 >> v2:172.21.15.181:3300/0 conn(0x7fb244072c30 0x7fb24415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 --2- 172.21.15.129:0/3016704539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb24415fbd0 0x7fb244152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 -- 172.21.15.129:0/3016704539 >> 172.21.15.129:0/3016704539 conn(0x7fb24406d410 msgr2=0x7fb2440b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 -- 172.21.15.129:0/3016704539 shutdown_connections 2024-09-17T03:53:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.727+0000 7fb249a00640 1 -- 172.21.15.129:0/3016704539 wait complete. 2024-09-17T03:53:39.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.729+0000 7fb249a00640 1 Processor -- start 2024-09-17T03:53:39.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.729+0000 7fb249a00640 1 -- start start 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb244073040 0x7fb244141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb244141ba0 0x7fb2440cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb24415c1e0 con 0x7fb244072c30 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb24415c060 con 0x7fb244073040 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb249a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb24415c360 con 0x7fb244141ba0 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb243e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb243e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49112/0 (socket says 172.21.15.129:49112) 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.731+0000 7fb243e00640 1 -- 172.21.15.129:0/243678419 learned_addr learned my addr 172.21.15.129:0/243678419 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb23be00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb244073040 0x7fb244141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb249000640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb244141ba0 0x7fb2440cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 -- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb244141ba0 msgr2=0x7fb2440cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb244141ba0 0x7fb2440cfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 -- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb244073040 msgr2=0x7fb244141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb244073040 0x7fb244141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 -- 172.21.15.129:0/243678419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2441f65c0 con 0x7fb244072c30 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb243e00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb23400e7f0 tx=0x7fb23400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb23400ccc0 con 0x7fb244072c30 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb23400ce80 con 0x7fb244072c30 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2440cf0d0 con 0x7fb244072c30 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234005280 con 0x7fb244072c30 2024-09-17T03:53:39.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.732+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb244156640 con 0x7fb244072c30 2024-09-17T03:53:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.733+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2441568a0 con 0x7fb244072c30 2024-09-17T03:53:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.735+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb234005420 con 0x7fb244072c30 2024-09-17T03:53:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.736+0000 7fb240c00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 0x7fb22007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:39.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.736+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb23409b5f0 con 0x7fb244072c30 2024-09-17T03:53:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.736+0000 7fb23be00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 0x7fb22007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:39.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.737+0000 7fb23be00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 0x7fb22007b0a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb2441507d0 tx=0x7fb22c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:39.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.740+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb234063b30 con 0x7fb244072c30 2024-09-17T03:53:39.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.919+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2441587e0 con 0x7fb220078b90 2024-09-17T03:53:39.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.925+0000 7fb240c00640 1 -- 172.21.15.129:0/243678419 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248034 (secure 0 0 0) 0x7fb22c011af0 con 0x7fb220078b90 2024-09-17T03:53:39.926 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:39.928 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 msgr2=0x7fb22007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 0x7fb22007b0a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb2441507d0 tx=0x7fb22c005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 msgr2=0x7fb244156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb23400e7f0 tx=0x7fb23400ecb0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 shutdown_connections 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb220078b90 0x7fb22007b0a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb244141ba0 0x7fb2440cfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb244073040 0x7fb244141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 --2- 172.21.15.129:0/243678419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb244072c30 0x7fb244156000 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 >> 172.21.15.129:0/243678419 conn(0x7fb24406d410 msgr2=0x7fb24415c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.928+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 shutdown_connections 2024-09-17T03:53:39.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:39.929+0000 7fb249a00640 1 -- 172.21.15.129:0/243678419 wait complete. 2024-09-17T03:53:39.950 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-17T03:53:38.359735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591062,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":131,"num_write_kb":1631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258184,"kb_used_data":8672,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716536,"statfs":{"total":773094113280,"available":772829732864,"internally_reserved":0,"allocated":8880128,"data_stored":4392868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":694,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.344257"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178823+0000","last_change":"2024-09-17T03:53:35.072458+0000","last_active":"2024-09-17T03:53:36.178823+0000","last_peered":"2024-09-17T03:53:36.178823+0000","last_clean":"2024-09-17T03:53:36.178823+0000","last_became_active":"2024-09-17T03:53:35.070660+0000","last_became_peered":"2024-09-17T03:53:35.070660+0000","last_unstale":"2024-09-17T03:53:36.178823+0000","last_undegraded":"2024-09-17T03:53:36.178823+0000","last_fullsized":"2024-09-17T03:53:36.178823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.505206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028869+0000","last_change":"2024-09-17T03:53:34.028869+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028869+0000","last_undegraded":"2024-09-17T03:53:34.028869+0000","last_fullsized":"2024-09-17T03:53:34.028869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033363+0000","last_change":"2024-09-17T03:53:34.033363+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033363+0000","last_undegraded":"2024-09-17T03:53:34.033363+0000","last_fullsized":"2024-09-17T03:53:34.033363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034036+0000","last_change":"2024-09-17T03:53:34.034036+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034036+0000","last_undegraded":"2024-09-17T03:53:34.034036+0000","last_fullsized":"2024-09-17T03:53:34.034036+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031012+0000","last_change":"2024-09-17T03:53:34.031012+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031012+0000","last_undegraded":"2024-09-17T03:53:34.031012+0000","last_fullsized":"2024-09-17T03:53:34.031012+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.029155+0000","last_change":"2024-09-17T03:53:34.029155+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.029155+0000","last_undegraded":"2024-09-17T03:53:34.029155+0000","last_fullsized":"2024-09-17T03:53:34.029155+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033193+0000","last_change":"2024-09-17T03:53:34.033193+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033193+0000","last_undegraded":"2024-09-17T03:53:34.033193+0000","last_fullsized":"2024-09-17T03:53:34.033193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027576+0000","last_change":"2024-09-17T03:53:34.027576+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027576+0000","last_undegraded":"2024-09-17T03:53:34.027576+0000","last_fullsized":"2024-09-17T03:53:34.027576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026926+0000","last_change":"2024-09-17T03:53:34.026926+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026926+0000","last_undegraded":"2024-09-17T03:53:34.026926+0000","last_fullsized":"2024-09-17T03:53:34.026926+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178936+0000","last_change":"2024-09-17T03:53:35.075213+0000","last_active":"2024-09-17T03:53:36.178936+0000","last_peered":"2024-09-17T03:53:36.178936+0000","last_clean":"2024-09-17T03:53:36.178936+0000","last_became_active":"2024-09-17T03:53:35.074344+0000","last_became_peered":"2024-09-17T03:53:35.074344+0000","last_unstale":"2024-09-17T03:53:36.178936+0000","last_undegraded":"2024-09-17T03:53:36.178936+0000","last_fullsized":"2024-09-17T03:53:36.178936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:02.543171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.025977+0000","last_change":"2024-09-17T03:53:34.025977+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.025977+0000","last_undegraded":"2024-09-17T03:53:34.025977+0000","last_fullsized":"2024-09-17T03:53:34.025977+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178709+0000","last_change":"2024-09-17T03:53:35.073592+0000","last_active":"2024-09-17T03:53:36.178709+0000","last_peered":"2024-09-17T03:53:36.178709+0000","last_clean":"2024-09-17T03:53:36.178709+0000","last_became_active":"2024-09-17T03:53:35.073318+0000","last_became_peered":"2024-09-17T03:53:35.073318+0000","last_unstale":"2024-09-17T03:53:36.178709+0000","last_undegraded":"2024-09-17T03:53:36.178709+0000","last_fullsized":"2024-09-17T03:53:36.178709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:26.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032082+0000","last_change":"2024-09-17T03:53:34.032082+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032082+0000","last_undegraded":"2024-09-17T03:53:34.032082+0000","last_fullsized":"2024-09-17T03:53:34.032082+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036001+0000","last_change":"2024-09-17T03:53:34.036001+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036001+0000","last_undegraded":"2024-09-17T03:53:34.036001+0000","last_fullsized":"2024-09-17T03:53:34.036001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027512+0000","last_change":"2024-09-17T03:53:34.027512+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027512+0000","last_undegraded":"2024-09-17T03:53:34.027512+0000","last_fullsized":"2024-09-17T03:53:34.027512+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034599+0000","last_change":"2024-09-17T03:53:34.034599+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034599+0000","last_undegraded":"2024-09-17T03:53:34.034599+0000","last_fullsized":"2024-09-17T03:53:34.034599+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.030598+0000","last_change":"2024-09-17T03:53:34.030598+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.030598+0000","last_undegraded":"2024-09-17T03:53:34.030598+0000","last_fullsized":"2024-09-17T03:53:34.030598+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037042+0000","last_change":"2024-09-17T03:53:34.037042+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037042+0000","last_undegraded":"2024-09-17T03:53:34.037042+0000","last_fullsized":"2024-09-17T03:53:34.037042+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529477+0000","last_change":"2024-09-17T03:53:35.077049+0000","last_active":"2024-09-17T03:53:36.529477+0000","last_peered":"2024-09-17T03:53:36.529477+0000","last_clean":"2024-09-17T03:53:36.529477+0000","last_became_active":"2024-09-17T03:53:35.076178+0000","last_became_peered":"2024-09-17T03:53:35.076178+0000","last_unstale":"2024-09-17T03:53:36.529477+0000","last_undegraded":"2024-09-17T03:53:36.529477+0000","last_fullsized":"2024-09-17T03:53:36.529477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033561+0000","last_change":"2024-09-17T03:53:34.033561+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033561+0000","last_undegraded":"2024-09-17T03:53:34.033561+0000","last_fullsized":"2024-09-17T03:53:34.033561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034737+0000","last_change":"2024-09-17T03:53:34.034737+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034737+0000","last_undegraded":"2024-09-17T03:53:34.034737+0000","last_fullsized":"2024-09-17T03:53:34.034737+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.029254+0000","last_change":"2024-09-17T03:53:34.029254+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.029254+0000","last_undegraded":"2024-09-17T03:53:34.029254+0000","last_fullsized":"2024-09-17T03:53:34.029254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178654+0000","last_change":"2024-09-17T03:53:35.068757+0000","last_active":"2024-09-17T03:53:36.178654+0000","last_peered":"2024-09-17T03:53:36.178654+0000","last_clean":"2024-09-17T03:53:36.178654+0000","last_became_active":"2024-09-17T03:53:35.068235+0000","last_became_peered":"2024-09-17T03:53:35.068235+0000","last_unstale":"2024-09-17T03:53:36.178654+0000","last_undegraded":"2024-09-17T03:53:36.178654+0000","last_fullsized":"2024-09-17T03:53:36.178654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:16.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037547+0000","last_change":"2024-09-17T03:53:34.037547+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037547+0000","last_undegraded":"2024-09-17T03:53:34.037547+0000","last_fullsized":"2024-09-17T03:53:34.037547+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027005+0000","last_change":"2024-09-17T03:53:34.027005+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027005+0000","last_undegraded":"2024-09-17T03:53:34.027005+0000","last_fullsized":"2024-09-17T03:53:34.027005+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033556+0000","last_change":"2024-09-17T03:53:34.033556+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033556+0000","last_undegraded":"2024-09-17T03:53:34.033556+0000","last_fullsized":"2024-09-17T03:53:34.033556+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.034217+0000","last_change":"2024-09-17T03:53:34.034217+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.034217+0000","last_undegraded":"2024-09-17T03:53:34.034217+0000","last_fullsized":"2024-09-17T03:53:34.034217+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178706+0000","last_change":"2024-09-17T03:53:35.067698+0000","last_active":"2024-09-17T03:53:36.178706+0000","last_peered":"2024-09-17T03:53:36.178706+0000","last_clean":"2024-09-17T03:53:36.178706+0000","last_became_active":"2024-09-17T03:53:35.066584+0000","last_became_peered":"2024-09-17T03:53:35.066584+0000","last_unstale":"2024-09-17T03:53:36.178706+0000","last_undegraded":"2024-09-17T03:53:36.178706+0000","last_fullsized":"2024-09-17T03:53:36.178706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:57.060788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026812+0000","last_change":"2024-09-17T03:53:34.026812+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026812+0000","last_undegraded":"2024-09-17T03:53:34.026812+0000","last_fullsized":"2024-09-17T03:53:34.026812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028702+0000","last_change":"2024-09-17T03:53:34.028702+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028702+0000","last_undegraded":"2024-09-17T03:53:34.028702+0000","last_fullsized":"2024-09-17T03:53:34.028702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036230+0000","last_change":"2024-09-17T03:53:34.036230+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036230+0000","last_undegraded":"2024-09-17T03:53:34.036230+0000","last_fullsized":"2024-09-17T03:53:34.036230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178515+0000","last_change":"2024-09-17T03:53:35.062010+0000","last_active":"2024-09-17T03:53:36.178515+0000","last_peered":"2024-09-17T03:53:36.178515+0000","last_clean":"2024-09-17T03:53:36.178515+0000","last_became_active":"2024-09-17T03:53:35.059874+0000","last_became_peered":"2024-09-17T03:53:35.059874+0000","last_unstale":"2024-09-17T03:53:36.178515+0000","last_undegraded":"2024-09-17T03:53:36.178515+0000","last_fullsized":"2024-09-17T03:53:36.178515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:26.933349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.036366+0000","last_change":"2024-09-17T03:53:34.036366+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.036366+0000","last_undegraded":"2024-09-17T03:53:34.036366+0000","last_fullsized":"2024-09-17T03:53:34.036366+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178966+0000","last_change":"2024-09-17T03:53:34.042427+0000","last_active":"2024-09-17T03:53:36.178966+0000","last_peered":"2024-09-17T03:53:36.178966+0000","last_clean":"2024-09-17T03:53:36.178966+0000","last_became_active":"2024-09-17T03:53:34.042117+0000","last_became_peered":"2024-09-17T03:53:34.042117+0000","last_unstale":"2024-09-17T03:53:36.178966+0000","last_undegraded":"2024-09-17T03:53:36.178966+0000","last_fullsized":"2024-09-17T03:53:36.178966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:57.275556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178860+0000","last_change":"2024-09-17T03:53:35.075209+0000","last_active":"2024-09-17T03:53:36.178860+0000","last_peered":"2024-09-17T03:53:36.178860+0000","last_clean":"2024-09-17T03:53:36.178860+0000","last_became_active":"2024-09-17T03:53:35.074557+0000","last_became_peered":"2024-09-17T03:53:35.074557+0000","last_unstale":"2024-09-17T03:53:36.178860+0000","last_undegraded":"2024-09-17T03:53:36.178860+0000","last_fullsized":"2024-09-17T03:53:36.178860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:26.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049450+0000","last_change":"2024-09-17T03:53:34.047657+0000","last_active":"2024-09-17T03:53:34.049450+0000","last_peered":"2024-09-17T03:53:34.049450+0000","last_clean":"2024-09-17T03:53:34.049450+0000","last_became_active":"2024-09-17T03:53:34.047290+0000","last_became_peered":"2024-09-17T03:53:34.047290+0000","last_unstale":"2024-09-17T03:53:34.049450+0000","last_undegraded":"2024-09-17T03:53:34.049450+0000","last_fullsized":"2024-09-17T03:53:34.049450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:32:37.965629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178967+0000","last_change":"2024-09-17T03:53:35.068246+0000","last_active":"2024-09-17T03:53:36.178967+0000","last_peered":"2024-09-17T03:53:36.178967+0000","last_clean":"2024-09-17T03:53:36.178967+0000","last_became_active":"2024-09-17T03:53:35.066866+0000","last_became_peered":"2024-09-17T03:53:35.066866+0000","last_unstale":"2024-09-17T03:53:36.178967+0000","last_undegraded":"2024-09-17T03:53:36.178967+0000","last_fullsized":"2024-09-17T03:53:36.178967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:59.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529312+0000","last_change":"2024-09-17T03:53:34.047098+0000","last_active":"2024-09-17T03:53:36.529312+0000","last_peered":"2024-09-17T03:53:36.529312+0000","last_clean":"2024-09-17T03:53:36.529312+0000","last_became_active":"2024-09-17T03:53:34.046816+0000","last_became_peered":"2024-09-17T03:53:34.046816+0000","last_unstale":"2024-09-17T03:53:36.529312+0000","last_undegraded":"2024-09-17T03:53:36.529312+0000","last_fullsized":"2024-09-17T03:53:36.529312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:25.856611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032835+0000","last_change":"2024-09-17T03:53:34.032835+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032835+0000","last_undegraded":"2024-09-17T03:53:34.032835+0000","last_fullsized":"2024-09-17T03:53:34.032835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529204+0000","last_change":"2024-09-17T03:53:34.047320+0000","last_active":"2024-09-17T03:53:36.529204+0000","last_peered":"2024-09-17T03:53:36.529204+0000","last_clean":"2024-09-17T03:53:36.529204+0000","last_became_active":"2024-09-17T03:53:34.045973+0000","last_became_peered":"2024-09-17T03:53:34.045973+0000","last_unstale":"2024-09-17T03:53:36.529204+0000","last_undegraded":"2024-09-17T03:53:36.529204+0000","last_fullsized":"2024-09-17T03:53:36.529204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:03.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.028186+0000","last_change":"2024-09-17T03:53:34.028186+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.028186+0000","last_undegraded":"2024-09-17T03:53:34.028186+0000","last_fullsized":"2024-09-17T03:53:34.028186+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049780+0000","last_change":"2024-09-17T03:53:34.046521+0000","last_active":"2024-09-17T03:53:34.049780+0000","last_peered":"2024-09-17T03:53:34.049780+0000","last_clean":"2024-09-17T03:53:34.049780+0000","last_became_active":"2024-09-17T03:53:34.046151+0000","last_became_peered":"2024-09-17T03:53:34.046151+0000","last_unstale":"2024-09-17T03:53:34.049780+0000","last_undegraded":"2024-09-17T03:53:34.049780+0000","last_fullsized":"2024-09-17T03:53:34.049780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:01.949777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.030392+0000","last_change":"2024-09-17T03:53:34.030392+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.030392+0000","last_undegraded":"2024-09-17T03:53:34.030392+0000","last_fullsized":"2024-09-17T03:53:34.030392+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048684+0000","last_change":"2024-09-17T03:53:34.045410+0000","last_active":"2024-09-17T03:53:34.048684+0000","last_peered":"2024-09-17T03:53:34.048684+0000","last_clean":"2024-09-17T03:53:34.048684+0000","last_became_active":"2024-09-17T03:53:34.045030+0000","last_became_peered":"2024-09-17T03:53:34.045030+0000","last_unstale":"2024-09-17T03:53:34.048684+0000","last_undegraded":"2024-09-17T03:53:34.048684+0000","last_fullsized":"2024-09-17T03:53:34.048684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:23.455684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031612+0000","last_change":"2024-09-17T03:53:34.031612+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031612+0000","last_undegraded":"2024-09-17T03:53:34.031612+0000","last_fullsized":"2024-09-17T03:53:34.031612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528363+0000","last_change":"2024-09-17T03:53:34.040513+0000","last_active":"2024-09-17T03:53:36.528363+0000","last_peered":"2024-09-17T03:53:36.528363+0000","last_clean":"2024-09-17T03:53:36.528363+0000","last_became_active":"2024-09-17T03:53:34.039387+0000","last_became_peered":"2024-09-17T03:53:34.039387+0000","last_unstale":"2024-09-17T03:53:36.528363+0000","last_undegraded":"2024-09-17T03:53:36.528363+0000","last_fullsized":"2024-09-17T03:53:36.528363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:35.514048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024952+0000","last_change":"2024-09-17T03:53:34.024952+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024952+0000","last_undegraded":"2024-09-17T03:53:34.024952+0000","last_fullsized":"2024-09-17T03:53:34.024952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.045346+0000","last_change":"2024-09-17T03:53:34.042323+0000","last_active":"2024-09-17T03:53:34.045346+0000","last_peered":"2024-09-17T03:53:34.045346+0000","last_clean":"2024-09-17T03:53:34.045346+0000","last_became_active":"2024-09-17T03:53:34.041995+0000","last_became_peered":"2024-09-17T03:53:34.041995+0000","last_unstale":"2024-09-17T03:53:34.045346+0000","last_undegraded":"2024-09-17T03:53:34.045346+0000","last_fullsized":"2024-09-17T03:53:34.045346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:32.733482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024710+0000","last_change":"2024-09-17T03:53:34.024710+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024710+0000","last_undegraded":"2024-09-17T03:53:34.024710+0000","last_fullsized":"2024-09-17T03:53:34.024710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179093+0000","last_change":"2024-09-17T03:53:34.041277+0000","last_active":"2024-09-17T03:53:36.179093+0000","last_peered":"2024-09-17T03:53:36.179093+0000","last_clean":"2024-09-17T03:53:36.179093+0000","last_became_active":"2024-09-17T03:53:34.040555+0000","last_became_peered":"2024-09-17T03:53:34.040555+0000","last_unstale":"2024-09-17T03:53:36.179093+0000","last_undegraded":"2024-09-17T03:53:36.179093+0000","last_fullsized":"2024-09-17T03:53:36.179093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:05.315114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024763+0000","last_change":"2024-09-17T03:53:34.024763+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024763+0000","last_undegraded":"2024-09-17T03:53:34.024763+0000","last_fullsized":"2024-09-17T03:53:34.024763+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178778+0000","last_change":"2024-09-17T03:53:34.047020+0000","last_active":"2024-09-17T03:53:36.178778+0000","last_peered":"2024-09-17T03:53:36.178778+0000","last_clean":"2024-09-17T03:53:36.178778+0000","last_became_active":"2024-09-17T03:53:34.045487+0000","last_became_peered":"2024-09-17T03:53:34.045487+0000","last_unstale":"2024-09-17T03:53:36.178778+0000","last_undegraded":"2024-09-17T03:53:36.178778+0000","last_fullsized":"2024-09-17T03:53:36.178778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:03.205170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529300+0000","last_change":"2024-09-17T03:53:35.074637+0000","last_active":"2024-09-17T03:53:36.529300+0000","last_peered":"2024-09-17T03:53:36.529300+0000","last_clean":"2024-09-17T03:53:36.529300+0000","last_became_active":"2024-09-17T03:53:35.074086+0000","last_became_peered":"2024-09-17T03:53:35.074086+0000","last_unstale":"2024-09-17T03:53:36.529300+0000","last_undegraded":"2024-09-17T03:53:36.529300+0000","last_fullsized":"2024-09-17T03:53:36.529300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.027879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178381+0000","last_change":"2024-09-17T03:53:34.045567+0000","last_active":"2024-09-17T03:53:36.178381+0000","last_peered":"2024-09-17T03:53:36.178381+0000","last_clean":"2024-09-17T03:53:36.178381+0000","last_became_active":"2024-09-17T03:53:34.045029+0000","last_became_peered":"2024-09-17T03:53:34.045029+0000","last_unstale":"2024-09-17T03:53:36.178381+0000","last_undegraded":"2024-09-17T03:53:36.178381+0000","last_fullsized":"2024-09-17T03:53:36.178381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:38.299845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033101+0000","last_change":"2024-09-17T03:53:34.033101+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033101+0000","last_undegraded":"2024-09-17T03:53:34.033101+0000","last_fullsized":"2024-09-17T03:53:34.033101+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.038363+0000","last_change":"2024-09-17T03:53:34.033143+0000","last_active":"2024-09-17T03:53:34.038363+0000","last_peered":"2024-09-17T03:53:34.038363+0000","last_clean":"2024-09-17T03:53:34.038363+0000","last_became_active":"2024-09-17T03:53:34.031770+0000","last_became_peered":"2024-09-17T03:53:34.031770+0000","last_unstale":"2024-09-17T03:53:34.038363+0000","last_undegraded":"2024-09-17T03:53:34.038363+0000","last_fullsized":"2024-09-17T03:53:34.038363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:20.987901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529438+0000","last_change":"2024-09-17T03:53:35.062155+0000","last_active":"2024-09-17T03:53:36.529438+0000","last_peered":"2024-09-17T03:53:36.529438+0000","last_clean":"2024-09-17T03:53:36.529438+0000","last_became_active":"2024-09-17T03:53:35.061536+0000","last_became_peered":"2024-09-17T03:53:35.061536+0000","last_unstale":"2024-09-17T03:53:36.529438+0000","last_undegraded":"2024-09-17T03:53:36.529438+0000","last_fullsized":"2024-09-17T03:53:36.529438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:36.274547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.045382+0000","last_change":"2024-09-17T03:53:34.043064+0000","last_active":"2024-09-17T03:53:34.045382+0000","last_peered":"2024-09-17T03:53:34.045382+0000","last_clean":"2024-09-17T03:53:34.045382+0000","last_became_active":"2024-09-17T03:53:34.038696+0000","last_became_peered":"2024-09-17T03:53:34.038696+0000","last_unstale":"2024-09-17T03:53:34.045382+0000","last_undegraded":"2024-09-17T03:53:34.045382+0000","last_fullsized":"2024-09-17T03:53:34.045382+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:40.436716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031491+0000","last_change":"2024-09-17T03:53:34.031491+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031491+0000","last_undegraded":"2024-09-17T03:53:34.031491+0000","last_fullsized":"2024-09-17T03:53:34.031491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050885+0000","last_change":"2024-09-17T03:53:34.048029+0000","last_active":"2024-09-17T03:53:34.050885+0000","last_peered":"2024-09-17T03:53:34.050885+0000","last_clean":"2024-09-17T03:53:34.050885+0000","last_became_active":"2024-09-17T03:53:34.046461+0000","last_became_peered":"2024-09-17T03:53:34.046461+0000","last_unstale":"2024-09-17T03:53:34.050885+0000","last_undegraded":"2024-09-17T03:53:34.050885+0000","last_fullsized":"2024-09-17T03:53:34.050885+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:27.848560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.026428+0000","last_change":"2024-09-17T03:53:34.026428+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.026428+0000","last_undegraded":"2024-09-17T03:53:34.026428+0000","last_fullsized":"2024-09-17T03:53:34.026428+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048911+0000","last_change":"2024-09-17T03:53:34.045446+0000","last_active":"2024-09-17T03:53:34.048911+0000","last_peered":"2024-09-17T03:53:34.048911+0000","last_clean":"2024-09-17T03:53:34.048911+0000","last_became_active":"2024-09-17T03:53:34.045103+0000","last_became_peered":"2024-09-17T03:53:34.045103+0000","last_unstale":"2024-09-17T03:53:34.048911+0000","last_undegraded":"2024-09-17T03:53:34.048911+0000","last_fullsized":"2024-09-17T03:53:34.048911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:49.049002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027698+0000","last_change":"2024-09-17T03:53:34.027698+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027698+0000","last_undegraded":"2024-09-17T03:53:34.027698+0000","last_fullsized":"2024-09-17T03:53:34.027698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528873+0000","last_change":"2024-09-17T03:53:34.043445+0000","last_active":"2024-09-17T03:53:36.528873+0000","last_peered":"2024-09-17T03:53:36.528873+0000","last_clean":"2024-09-17T03:53:36.528873+0000","last_became_active":"2024-09-17T03:53:34.042920+0000","last_became_peered":"2024-09-17T03:53:34.042920+0000","last_unstale":"2024-09-17T03:53:36.528873+0000","last_undegraded":"2024-09-17T03:53:36.528873+0000","last_fullsized":"2024-09-17T03:53:36.528873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:42.885526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032446+0000","last_change":"2024-09-17T03:53:34.032446+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032446+0000","last_undegraded":"2024-09-17T03:53:34.032446+0000","last_fullsized":"2024-09-17T03:53:34.032446+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179205+0000","last_change":"2024-09-17T03:53:34.041969+0000","last_active":"2024-09-17T03:53:36.179205+0000","last_peered":"2024-09-17T03:53:36.179205+0000","last_clean":"2024-09-17T03:53:36.179205+0000","last_became_active":"2024-09-17T03:53:34.040090+0000","last_became_peered":"2024-09-17T03:53:34.040090+0000","last_unstale":"2024-09-17T03:53:36.179205+0000","last_undegraded":"2024-09-17T03:53:36.179205+0000","last_fullsized":"2024-09-17T03:53:36.179205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:12.790859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179104+0000","last_change":"2024-09-17T03:53:35.071124+0000","last_active":"2024-09-17T03:53:36.179104+0000","last_peered":"2024-09-17T03:53:36.179104+0000","last_clean":"2024-09-17T03:53:36.179104+0000","last_became_active":"2024-09-17T03:53:35.070636+0000","last_became_peered":"2024-09-17T03:53:35.070636+0000","last_unstale":"2024-09-17T03:53:36.179104+0000","last_undegraded":"2024-09-17T03:53:36.179104+0000","last_fullsized":"2024-09-17T03:53:36.179104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:18.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529575+0000","last_change":"2024-09-17T03:53:34.049033+0000","last_active":"2024-09-17T03:53:36.529575+0000","last_peered":"2024-09-17T03:53:36.529575+0000","last_clean":"2024-09-17T03:53:36.529575+0000","last_became_active":"2024-09-17T03:53:34.048596+0000","last_became_peered":"2024-09-17T03:53:34.048596+0000","last_unstale":"2024-09-17T03:53:36.529575+0000","last_undegraded":"2024-09-17T03:53:36.529575+0000","last_fullsized":"2024-09-17T03:53:36.529575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:25.851358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.032593+0000","last_change":"2024-09-17T03:53:34.032593+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.032593+0000","last_undegraded":"2024-09-17T03:53:34.032593+0000","last_fullsized":"2024-09-17T03:53:34.032593+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529714+0000","last_change":"2024-09-17T03:53:34.040065+0000","last_active":"2024-09-17T03:53:36.529714+0000","last_peered":"2024-09-17T03:53:36.529714+0000","last_clean":"2024-09-17T03:53:36.529714+0000","last_became_active":"2024-09-17T03:53:34.039675+0000","last_became_peered":"2024-09-17T03:53:34.039675+0000","last_unstale":"2024-09-17T03:53:36.529714+0000","last_undegraded":"2024-09-17T03:53:36.529714+0000","last_fullsized":"2024-09-17T03:53:36.529714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:40.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.035557+0000","last_change":"2024-09-17T03:53:34.035557+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.035557+0000","last_undegraded":"2024-09-17T03:53:34.035557+0000","last_fullsized":"2024-09-17T03:53:34.035557+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050120+0000","last_change":"2024-09-17T03:53:34.042295+0000","last_active":"2024-09-17T03:53:34.050120+0000","last_peered":"2024-09-17T03:53:34.050120+0000","last_clean":"2024-09-17T03:53:34.050120+0000","last_became_active":"2024-09-17T03:53:34.041992+0000","last_became_peered":"2024-09-17T03:53:34.041992+0000","last_unstale":"2024-09-17T03:53:34.050120+0000","last_undegraded":"2024-09-17T03:53:34.050120+0000","last_fullsized":"2024-09-17T03:53:34.050120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:10.329340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027458+0000","last_change":"2024-09-17T03:53:34.027458+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027458+0000","last_undegraded":"2024-09-17T03:53:34.027458+0000","last_fullsized":"2024-09-17T03:53:34.027458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.054928+0000","last_change":"2024-09-17T03:53:34.050809+0000","last_active":"2024-09-17T03:53:34.054928+0000","last_peered":"2024-09-17T03:53:34.054928+0000","last_clean":"2024-09-17T03:53:34.054928+0000","last_became_active":"2024-09-17T03:53:34.050498+0000","last_became_peered":"2024-09-17T03:53:34.050498+0000","last_unstale":"2024-09-17T03:53:34.054928+0000","last_undegraded":"2024-09-17T03:53:34.054928+0000","last_fullsized":"2024-09-17T03:53:34.054928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:13:23.876004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.037100+0000","last_change":"2024-09-17T03:53:34.037100+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.037100+0000","last_undegraded":"2024-09-17T03:53:34.037100+0000","last_fullsized":"2024-09-17T03:53:34.037100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049807+0000","last_change":"2024-09-17T03:53:34.046054+0000","last_active":"2024-09-17T03:53:34.049807+0000","last_peered":"2024-09-17T03:53:34.049807+0000","last_clean":"2024-09-17T03:53:34.049807+0000","last_became_active":"2024-09-17T03:53:34.045718+0000","last_became_peered":"2024-09-17T03:53:34.045718+0000","last_unstale":"2024-09-17T03:53:34.049807+0000","last_undegraded":"2024-09-17T03:53:34.049807+0000","last_fullsized":"2024-09-17T03:53:34.049807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:11.308714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.025130+0000","last_change":"2024-09-17T03:53:34.025130+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.025130+0000","last_undegraded":"2024-09-17T03:53:34.025130+0000","last_fullsized":"2024-09-17T03:53:34.025130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.050129+0000","last_change":"2024-09-17T03:53:34.045835+0000","last_active":"2024-09-17T03:53:34.050129+0000","last_peered":"2024-09-17T03:53:34.050129+0000","last_clean":"2024-09-17T03:53:34.050129+0000","last_became_active":"2024-09-17T03:53:34.045329+0000","last_became_peered":"2024-09-17T03:53:34.045329+0000","last_unstale":"2024-09-17T03:53:34.050129+0000","last_undegraded":"2024-09-17T03:53:34.050129+0000","last_fullsized":"2024-09-17T03:53:34.050129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:36.472408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.035850+0000","last_change":"2024-09-17T03:53:34.035850+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.035850+0000","last_undegraded":"2024-09-17T03:53:34.035850+0000","last_fullsized":"2024-09-17T03:53:34.035850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.046315+0000","last_change":"2024-09-17T03:53:34.043457+0000","last_active":"2024-09-17T03:53:34.046315+0000","last_peered":"2024-09-17T03:53:34.046315+0000","last_clean":"2024-09-17T03:53:34.046315+0000","last_became_active":"2024-09-17T03:53:34.043142+0000","last_became_peered":"2024-09-17T03:53:34.043142+0000","last_unstale":"2024-09-17T03:53:34.046315+0000","last_undegraded":"2024-09-17T03:53:34.046315+0000","last_fullsized":"2024-09-17T03:53:34.046315+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:08.135213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.033549+0000","last_change":"2024-09-17T03:53:34.033549+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.033549+0000","last_undegraded":"2024-09-17T03:53:34.033549+0000","last_fullsized":"2024-09-17T03:53:34.033549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049609+0000","last_change":"2024-09-17T03:53:34.047344+0000","last_active":"2024-09-17T03:53:34.049609+0000","last_peered":"2024-09-17T03:53:34.049609+0000","last_clean":"2024-09-17T03:53:34.049609+0000","last_became_active":"2024-09-17T03:53:34.041909+0000","last_became_peered":"2024-09-17T03:53:34.041909+0000","last_unstale":"2024-09-17T03:53:34.049609+0000","last_undegraded":"2024-09-17T03:53:34.049609+0000","last_fullsized":"2024-09-17T03:53:34.049609+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:50.302474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179022+0000","last_change":"2024-09-17T03:53:35.060992+0000","last_active":"2024-09-17T03:53:36.179022+0000","last_peered":"2024-09-17T03:53:36.179022+0000","last_clean":"2024-09-17T03:53:36.179022+0000","last_became_active":"2024-09-17T03:53:35.060339+0000","last_became_peered":"2024-09-17T03:53:35.060339+0000","last_unstale":"2024-09-17T03:53:36.179022+0000","last_undegraded":"2024-09-17T03:53:36.179022+0000","last_fullsized":"2024-09-17T03:53:36.179022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:48.433949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178476+0000","last_change":"2024-09-17T03:53:34.042515+0000","last_active":"2024-09-17T03:53:36.178476+0000","last_peered":"2024-09-17T03:53:36.178476+0000","last_clean":"2024-09-17T03:53:36.178476+0000","last_became_active":"2024-09-17T03:53:34.041350+0000","last_became_peered":"2024-09-17T03:53:34.041350+0000","last_unstale":"2024-09-17T03:53:36.178476+0000","last_undegraded":"2024-09-17T03:53:36.178476+0000","last_fullsized":"2024-09-17T03:53:36.178476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:54.121078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031367+0000","last_change":"2024-09-17T03:53:34.031367+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031367+0000","last_undegraded":"2024-09-17T03:53:34.031367+0000","last_fullsized":"2024-09-17T03:53:34.031367+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179021+0000","last_change":"2024-09-17T03:53:34.043860+0000","last_active":"2024-09-17T03:53:36.179021+0000","last_peered":"2024-09-17T03:53:36.179021+0000","last_clean":"2024-09-17T03:53:36.179021+0000","last_became_active":"2024-09-17T03:53:34.043502+0000","last_became_peered":"2024-09-17T03:53:34.043502+0000","last_unstale":"2024-09-17T03:53:36.179021+0000","last_undegraded":"2024-09-17T03:53:36.179021+0000","last_fullsized":"2024-09-17T03:53:36.179021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:02.362312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.024331+0000","last_change":"2024-09-17T03:53:34.024331+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.024331+0000","last_undegraded":"2024-09-17T03:53:34.024331+0000","last_fullsized":"2024-09-17T03:53:34.024331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529388+0000","last_change":"2024-09-17T03:53:34.037735+0000","last_active":"2024-09-17T03:53:36.529388+0000","last_peered":"2024-09-17T03:53:36.529388+0000","last_clean":"2024-09-17T03:53:36.529388+0000","last_became_active":"2024-09-17T03:53:34.037448+0000","last_became_peered":"2024-09-17T03:53:34.037448+0000","last_unstale":"2024-09-17T03:53:36.529388+0000","last_undegraded":"2024-09-17T03:53:36.529388+0000","last_fullsized":"2024-09-17T03:53:36.529388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:52.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529554+0000","last_change":"2024-09-17T03:53:35.071423+0000","last_active":"2024-09-17T03:53:36.529554+0000","last_peered":"2024-09-17T03:53:36.529554+0000","last_clean":"2024-09-17T03:53:36.529554+0000","last_became_active":"2024-09-17T03:53:35.071008+0000","last_became_peered":"2024-09-17T03:53:35.071008+0000","last_unstale":"2024-09-17T03:53:36.529554+0000","last_undegraded":"2024-09-17T03:53:36.529554+0000","last_fullsized":"2024-09-17T03:53:36.529554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:43.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049058+0000","last_change":"2024-09-17T03:53:34.045408+0000","last_active":"2024-09-17T03:53:34.049058+0000","last_peered":"2024-09-17T03:53:34.049058+0000","last_clean":"2024-09-17T03:53:34.049058+0000","last_became_active":"2024-09-17T03:53:34.045021+0000","last_became_peered":"2024-09-17T03:53:34.045021+0000","last_unstale":"2024-09-17T03:53:34.049058+0000","last_undegraded":"2024-09-17T03:53:34.049058+0000","last_fullsized":"2024-09-17T03:53:34.049058+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:34.908146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.031016+0000","last_change":"2024-09-17T03:53:34.031016+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.031016+0000","last_undegraded":"2024-09-17T03:53:34.031016+0000","last_fullsized":"2024-09-17T03:53:34.031016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178647+0000","last_change":"2024-09-17T03:53:34.041865+0000","last_active":"2024-09-17T03:53:36.178647+0000","last_peered":"2024-09-17T03:53:36.178647+0000","last_clean":"2024-09-17T03:53:36.178647+0000","last_became_active":"2024-09-17T03:53:34.041169+0000","last_became_peered":"2024-09-17T03:53:34.041169+0000","last_unstale":"2024-09-17T03:53:36.178647+0000","last_undegraded":"2024-09-17T03:53:36.178647+0000","last_fullsized":"2024-09-17T03:53:36.178647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:16.044280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178489+0000","last_change":"2024-09-17T03:53:35.064940+0000","last_active":"2024-09-17T03:53:36.178489+0000","last_peered":"2024-09-17T03:53:36.178489+0000","last_clean":"2024-09-17T03:53:36.178489+0000","last_became_active":"2024-09-17T03:53:35.064574+0000","last_became_peered":"2024-09-17T03:53:35.064574+0000","last_unstale":"2024-09-17T03:53:36.178489+0000","last_undegraded":"2024-09-17T03:53:36.178489+0000","last_fullsized":"2024-09-17T03:53:36.178489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:08.160089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.049928+0000","last_change":"2024-09-17T03:53:34.048127+0000","last_active":"2024-09-17T03:53:34.049928+0000","last_peered":"2024-09-17T03:53:34.049928+0000","last_clean":"2024-09-17T03:53:34.049928+0000","last_became_active":"2024-09-17T03:53:34.047768+0000","last_became_peered":"2024-09-17T03:53:34.047768+0000","last_unstale":"2024-09-17T03:53:34.049928+0000","last_undegraded":"2024-09-17T03:53:34.049928+0000","last_fullsized":"2024-09-17T03:53:34.049928+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:24.000468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":79,"state":"creating+peering","last_fresh":"2024-09-17T03:53:34.027124+0000","last_change":"2024-09-17T03:53:34.027124+0000","last_active":"2024-09-17T03:53:34.014366+0000","last_peered":"2024-09-17T03:53:34.014366+0000","last_clean":"2024-09-17T03:53:34.014366+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:53:34.027124+0000","last_undegraded":"2024-09-17T03:53:34.027124+0000","last_fullsized":"2024-09-17T03:53:34.027124+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.048868+0000","last_change":"2024-09-17T03:53:34.046914+0000","last_active":"2024-09-17T03:53:34.048868+0000","last_peered":"2024-09-17T03:53:34.048868+0000","last_clean":"2024-09-17T03:53:34.048868+0000","last_became_active":"2024-09-17T03:53:34.046056+0000","last_became_peered":"2024-09-17T03:53:34.046056+0000","last_unstale":"2024-09-17T03:53:34.048868+0000","last_undegraded":"2024-09-17T03:53:34.048868+0000","last_fullsized":"2024-09-17T03:53:34.048868+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:21.605697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":215,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-17T03:53:34.036269+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:53:34.036269+0000","last_peered":"2024-09-17T03:53:34.036269+0000","last_clean":"2024-09-17T03:53:34.036269+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:53:34.036269+0000","last_undegraded":"2024-09-17T03:53:34.036269+0000","last_fullsized":"2024-09-17T03:53:34.036269+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":694,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22,"ondisk_log_size":22,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509492,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1544192,"data_stored":986956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509495,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509492,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509494,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509494,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":950272,"data_stored":396588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542197,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":999424,"data_stored":416632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542198,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:53:39.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:53:40.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1069408021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 msgr2=0x7f9bec162850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1069408021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec162850 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4009820 tx=0x7f9bd402a480 comp rx=0 tx=0).stop 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1069408021 shutdown_connections 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1069408021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec162850 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1069408021 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 0x7f9bec0b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1069408021 >> 172.21.15.129:0/1069408021 conn(0x7f9bec09d6c0 msgr2=0x7f9bec0d4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1069408021 shutdown_connections 2024-09-17T03:53:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.214+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1069408021 wait complete. 2024-09-17T03:53:40.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.216+0000 7f9bf0a00640 1 Processor -- start 2024-09-17T03:53:40.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.216+0000 7f9bf0a00640 1 -- start start 2024-09-17T03:53:40.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 0x7f9bec0b24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec0cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec0b5360 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9bec0b51e0 con 0x7f9bec162440 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bf0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9bec0b54e0 con 0x7f9bec0b9bd0 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9beb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9beb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49114/0 (socket says 172.21.15.129:49114) 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9beb400640 1 -- 172.21.15.129:0/1047180146 learned_addr learned my addr 172.21.15.129:0/1047180146 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9be2a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 0x7f9bec0b24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.218+0000 7f9bebe00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec0cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 -- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 msgr2=0x7f9bec0b24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 0x7f9bec0b24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 -- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 msgr2=0x7f9bec0cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec0cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 -- 172.21.15.129:0/1047180146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bec0caa40 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9beb400640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e9f0 tx=0x7f9bdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc00cd80 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9bdc004020 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bec0cbc20 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc004470 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.219+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bec0c9eb0 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.220+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bec08cb00 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.222+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9bdc0035f0 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.222+0000 7f9be3e00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 0x7f9bc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.223+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f9bdc09b840 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.223+0000 7f9be2a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 0x7f9bc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.224+0000 7f9be2a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 0x7f9bc807b0a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4034000 tx=0x7f9bd4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:40.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.227+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9bec08cb00 con 0x7f9bec0b8fd0 2024-09-17T03:53:40.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.409+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9bec0cad20 con 0x7f9bc8078b90 2024-09-17T03:53:40.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.413+0000 7f9be3e00640 1 -- 172.21.15.129:0/1047180146 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252064 (secure 0 0 0) 0x7f9bec0cad20 con 0x7f9bc8078b90 2024-09-17T03:53:40.415 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:40.416 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.417+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 msgr2=0x7f9bc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.417+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 0x7f9bc807b0a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9bd4034000 tx=0x7f9bd4004730 comp rx=0 tx=0).stop 2024-09-17T03:53:40.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.417+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 msgr2=0x7f9bec0b1f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.417+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc00e9f0 tx=0x7f9bdc00eeb0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 shutdown_connections 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9bc8078b90 0x7f9bc807b0a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9bec162440 0x7f9bec0cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9bec0b9bd0 0x7f9bec0b24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 --2- 172.21.15.129:0/1047180146 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9bec0b8fd0 0x7f9bec0b1f70 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 >> 172.21.15.129:0/1047180146 conn(0x7f9bec09d6c0 msgr2=0x7f9bec06eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 shutdown_connections 2024-09-17T03:53:40.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.418+0000 7f9bf0a00640 1 -- 172.21.15.129:0/1047180146 wait complete. 2024-09-17T03:53:40.442 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-17T03:53:40.360291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258452,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716268,"statfs":{"total":773094113280,"available":772829458432,"internally_reserved":0,"allocated":9150464,"data_stored":4513060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.339125"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178823+0000","last_change":"2024-09-17T03:53:35.072458+0000","last_active":"2024-09-17T03:53:36.178823+0000","last_peered":"2024-09-17T03:53:36.178823+0000","last_clean":"2024-09-17T03:53:36.178823+0000","last_became_active":"2024-09-17T03:53:35.070660+0000","last_became_peered":"2024-09-17T03:53:35.070660+0000","last_unstale":"2024-09-17T03:53:36.178823+0000","last_undegraded":"2024-09-17T03:53:36.178823+0000","last_fullsized":"2024-09-17T03:53:36.178823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.505206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058029+0000","last_change":"2024-09-17T03:53:35.067959+0000","last_active":"2024-09-17T03:53:36.058029+0000","last_peered":"2024-09-17T03:53:36.058029+0000","last_clean":"2024-09-17T03:53:36.058029+0000","last_became_active":"2024-09-17T03:53:35.067646+0000","last_became_peered":"2024-09-17T03:53:35.067646+0000","last_unstale":"2024-09-17T03:53:36.058029+0000","last_undegraded":"2024-09-17T03:53:36.058029+0000","last_fullsized":"2024-09-17T03:53:36.058029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:55.989947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249115+0000","last_change":"2024-09-17T03:53:35.073278+0000","last_active":"2024-09-17T03:53:36.249115+0000","last_peered":"2024-09-17T03:53:36.249115+0000","last_clean":"2024-09-17T03:53:36.249115+0000","last_became_active":"2024-09-17T03:53:35.072943+0000","last_became_peered":"2024-09-17T03:53:35.072943+0000","last_unstale":"2024-09-17T03:53:36.249115+0000","last_undegraded":"2024-09-17T03:53:36.249115+0000","last_fullsized":"2024-09-17T03:53:36.249115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528171+0000","last_change":"2024-09-17T03:53:35.063690+0000","last_active":"2024-09-17T03:53:36.528171+0000","last_peered":"2024-09-17T03:53:36.528171+0000","last_clean":"2024-09-17T03:53:36.528171+0000","last_became_active":"2024-09-17T03:53:35.062877+0000","last_became_peered":"2024-09-17T03:53:35.062877+0000","last_unstale":"2024-09-17T03:53:36.528171+0000","last_undegraded":"2024-09-17T03:53:36.528171+0000","last_fullsized":"2024-09-17T03:53:36.528171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:42.690776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058612+0000","last_change":"2024-09-17T03:53:35.054924+0000","last_active":"2024-09-17T03:53:36.058612+0000","last_peered":"2024-09-17T03:53:36.058612+0000","last_clean":"2024-09-17T03:53:36.058612+0000","last_became_active":"2024-09-17T03:53:35.053852+0000","last_became_peered":"2024-09-17T03:53:35.053852+0000","last_unstale":"2024-09-17T03:53:36.058612+0000","last_undegraded":"2024-09-17T03:53:36.058612+0000","last_fullsized":"2024-09-17T03:53:36.058612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:49.257827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058725+0000","last_change":"2024-09-17T03:53:35.067059+0000","last_active":"2024-09-17T03:53:36.058725+0000","last_peered":"2024-09-17T03:53:36.058725+0000","last_clean":"2024-09-17T03:53:36.058725+0000","last_became_active":"2024-09-17T03:53:35.066734+0000","last_became_peered":"2024-09-17T03:53:35.066734+0000","last_unstale":"2024-09-17T03:53:36.058725+0000","last_undegraded":"2024-09-17T03:53:36.058725+0000","last_fullsized":"2024-09-17T03:53:36.058725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:54.297031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528126+0000","last_change":"2024-09-17T03:53:35.059267+0000","last_active":"2024-09-17T03:53:36.528126+0000","last_peered":"2024-09-17T03:53:36.528126+0000","last_clean":"2024-09-17T03:53:36.528126+0000","last_became_active":"2024-09-17T03:53:35.058966+0000","last_became_peered":"2024-09-17T03:53:35.058966+0000","last_unstale":"2024-09-17T03:53:36.528126+0000","last_undegraded":"2024-09-17T03:53:36.528126+0000","last_fullsized":"2024-09-17T03:53:36.528126+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:17:51.318108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058914+0000","last_change":"2024-09-17T03:53:35.051417+0000","last_active":"2024-09-17T03:53:36.058914+0000","last_peered":"2024-09-17T03:53:36.058914+0000","last_clean":"2024-09-17T03:53:36.058914+0000","last_became_active":"2024-09-17T03:53:35.051082+0000","last_became_peered":"2024-09-17T03:53:35.051082+0000","last_unstale":"2024-09-17T03:53:36.058914+0000","last_undegraded":"2024-09-17T03:53:36.058914+0000","last_fullsized":"2024-09-17T03:53:36.058914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:40.340287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058871+0000","last_change":"2024-09-17T03:53:35.076596+0000","last_active":"2024-09-17T03:53:36.058871+0000","last_peered":"2024-09-17T03:53:36.058871+0000","last_clean":"2024-09-17T03:53:36.058871+0000","last_became_active":"2024-09-17T03:53:35.076237+0000","last_became_peered":"2024-09-17T03:53:35.076237+0000","last_unstale":"2024-09-17T03:53:36.058871+0000","last_undegraded":"2024-09-17T03:53:36.058871+0000","last_fullsized":"2024-09-17T03:53:36.058871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:49.104278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178936+0000","last_change":"2024-09-17T03:53:35.075213+0000","last_active":"2024-09-17T03:53:36.178936+0000","last_peered":"2024-09-17T03:53:36.178936+0000","last_clean":"2024-09-17T03:53:36.178936+0000","last_became_active":"2024-09-17T03:53:35.074344+0000","last_became_peered":"2024-09-17T03:53:35.074344+0000","last_unstale":"2024-09-17T03:53:36.178936+0000","last_undegraded":"2024-09-17T03:53:36.178936+0000","last_fullsized":"2024-09-17T03:53:36.178936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:02.543171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057911+0000","last_change":"2024-09-17T03:53:35.060229+0000","last_active":"2024-09-17T03:53:36.057911+0000","last_peered":"2024-09-17T03:53:36.057911+0000","last_clean":"2024-09-17T03:53:36.057911+0000","last_became_active":"2024-09-17T03:53:35.059966+0000","last_became_peered":"2024-09-17T03:53:35.059966+0000","last_unstale":"2024-09-17T03:53:36.057911+0000","last_undegraded":"2024-09-17T03:53:36.057911+0000","last_fullsized":"2024-09-17T03:53:36.057911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:50.055699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178709+0000","last_change":"2024-09-17T03:53:35.073592+0000","last_active":"2024-09-17T03:53:36.178709+0000","last_peered":"2024-09-17T03:53:36.178709+0000","last_clean":"2024-09-17T03:53:36.178709+0000","last_became_active":"2024-09-17T03:53:35.073318+0000","last_became_peered":"2024-09-17T03:53:35.073318+0000","last_unstale":"2024-09-17T03:53:36.178709+0000","last_undegraded":"2024-09-17T03:53:36.178709+0000","last_fullsized":"2024-09-17T03:53:36.178709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:26.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249029+0000","last_change":"2024-09-17T03:53:35.060095+0000","last_active":"2024-09-17T03:53:36.249029+0000","last_peered":"2024-09-17T03:53:36.249029+0000","last_clean":"2024-09-17T03:53:36.249029+0000","last_became_active":"2024-09-17T03:53:35.059471+0000","last_became_peered":"2024-09-17T03:53:35.059471+0000","last_unstale":"2024-09-17T03:53:36.249029+0000","last_undegraded":"2024-09-17T03:53:36.249029+0000","last_fullsized":"2024-09-17T03:53:36.249029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528250+0000","last_change":"2024-09-17T03:53:35.080649+0000","last_active":"2024-09-17T03:53:36.528250+0000","last_peered":"2024-09-17T03:53:36.528250+0000","last_clean":"2024-09-17T03:53:36.528250+0000","last_became_active":"2024-09-17T03:53:35.080294+0000","last_became_peered":"2024-09-17T03:53:35.080294+0000","last_unstale":"2024-09-17T03:53:36.528250+0000","last_undegraded":"2024-09-17T03:53:36.528250+0000","last_fullsized":"2024-09-17T03:53:36.528250+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:33.740179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528771+0000","last_change":"2024-09-17T03:53:35.059152+0000","last_active":"2024-09-17T03:53:36.528771+0000","last_peered":"2024-09-17T03:53:36.528771+0000","last_clean":"2024-09-17T03:53:36.528771+0000","last_became_active":"2024-09-17T03:53:35.058509+0000","last_became_peered":"2024-09-17T03:53:35.058509+0000","last_unstale":"2024-09-17T03:53:36.528771+0000","last_undegraded":"2024-09-17T03:53:36.528771+0000","last_fullsized":"2024-09-17T03:53:36.528771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:47.079665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058072+0000","last_change":"2024-09-17T03:53:35.070836+0000","last_active":"2024-09-17T03:53:36.058072+0000","last_peered":"2024-09-17T03:53:36.058072+0000","last_clean":"2024-09-17T03:53:36.058072+0000","last_became_active":"2024-09-17T03:53:35.065693+0000","last_became_peered":"2024-09-17T03:53:35.065693+0000","last_unstale":"2024-09-17T03:53:36.058072+0000","last_undegraded":"2024-09-17T03:53:36.058072+0000","last_fullsized":"2024-09-17T03:53:36.058072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:50.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059504+0000","last_change":"2024-09-17T03:53:35.070366+0000","last_active":"2024-09-17T03:53:36.059504+0000","last_peered":"2024-09-17T03:53:36.059504+0000","last_clean":"2024-09-17T03:53:36.059504+0000","last_became_active":"2024-09-17T03:53:35.070062+0000","last_became_peered":"2024-09-17T03:53:35.070062+0000","last_unstale":"2024-09-17T03:53:36.059504+0000","last_undegraded":"2024-09-17T03:53:36.059504+0000","last_fullsized":"2024-09-17T03:53:36.059504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:45.603585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528047+0000","last_change":"2024-09-17T03:53:35.065367+0000","last_active":"2024-09-17T03:53:36.528047+0000","last_peered":"2024-09-17T03:53:36.528047+0000","last_clean":"2024-09-17T03:53:36.528047+0000","last_became_active":"2024-09-17T03:53:35.065042+0000","last_became_peered":"2024-09-17T03:53:35.065042+0000","last_unstale":"2024-09-17T03:53:36.528047+0000","last_undegraded":"2024-09-17T03:53:36.528047+0000","last_fullsized":"2024-09-17T03:53:36.528047+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:01.776287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529477+0000","last_change":"2024-09-17T03:53:35.077049+0000","last_active":"2024-09-17T03:53:36.529477+0000","last_peered":"2024-09-17T03:53:36.529477+0000","last_clean":"2024-09-17T03:53:36.529477+0000","last_became_active":"2024-09-17T03:53:35.076178+0000","last_became_peered":"2024-09-17T03:53:35.076178+0000","last_unstale":"2024-09-17T03:53:36.529477+0000","last_undegraded":"2024-09-17T03:53:36.529477+0000","last_fullsized":"2024-09-17T03:53:36.529477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528413+0000","last_change":"2024-09-17T03:53:35.070151+0000","last_active":"2024-09-17T03:53:36.528413+0000","last_peered":"2024-09-17T03:53:36.528413+0000","last_clean":"2024-09-17T03:53:36.528413+0000","last_became_active":"2024-09-17T03:53:35.069825+0000","last_became_peered":"2024-09-17T03:53:35.069825+0000","last_unstale":"2024-09-17T03:53:36.528413+0000","last_undegraded":"2024-09-17T03:53:36.528413+0000","last_fullsized":"2024-09-17T03:53:36.528413+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:41.443412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528656+0000","last_change":"2024-09-17T03:53:35.077741+0000","last_active":"2024-09-17T03:53:36.528656+0000","last_peered":"2024-09-17T03:53:36.528656+0000","last_clean":"2024-09-17T03:53:36.528656+0000","last_became_active":"2024-09-17T03:53:35.077437+0000","last_became_peered":"2024-09-17T03:53:35.077437+0000","last_unstale":"2024-09-17T03:53:36.528656+0000","last_undegraded":"2024-09-17T03:53:36.528656+0000","last_fullsized":"2024-09-17T03:53:36.528656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:51.309170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057255+0000","last_change":"2024-09-17T03:53:35.057094+0000","last_active":"2024-09-17T03:53:36.057255+0000","last_peered":"2024-09-17T03:53:36.057255+0000","last_clean":"2024-09-17T03:53:36.057255+0000","last_became_active":"2024-09-17T03:53:35.056803+0000","last_became_peered":"2024-09-17T03:53:35.056803+0000","last_unstale":"2024-09-17T03:53:36.057255+0000","last_undegraded":"2024-09-17T03:53:36.057255+0000","last_fullsized":"2024-09-17T03:53:36.057255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:43.810445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178654+0000","last_change":"2024-09-17T03:53:35.068757+0000","last_active":"2024-09-17T03:53:36.178654+0000","last_peered":"2024-09-17T03:53:36.178654+0000","last_clean":"2024-09-17T03:53:36.178654+0000","last_became_active":"2024-09-17T03:53:35.068235+0000","last_became_peered":"2024-09-17T03:53:35.068235+0000","last_unstale":"2024-09-17T03:53:36.178654+0000","last_undegraded":"2024-09-17T03:53:36.178654+0000","last_fullsized":"2024-09-17T03:53:36.178654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:16.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527840+0000","last_change":"2024-09-17T03:53:35.058419+0000","last_active":"2024-09-17T03:53:36.527840+0000","last_peered":"2024-09-17T03:53:36.527840+0000","last_clean":"2024-09-17T03:53:36.527840+0000","last_became_active":"2024-09-17T03:53:35.057357+0000","last_became_peered":"2024-09-17T03:53:35.057357+0000","last_unstale":"2024-09-17T03:53:36.527840+0000","last_undegraded":"2024-09-17T03:53:36.527840+0000","last_fullsized":"2024-09-17T03:53:36.527840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:46.020805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057970+0000","last_change":"2024-09-17T03:53:35.067946+0000","last_active":"2024-09-17T03:53:36.057970+0000","last_peered":"2024-09-17T03:53:36.057970+0000","last_clean":"2024-09-17T03:53:36.057970+0000","last_became_active":"2024-09-17T03:53:35.067630+0000","last_became_peered":"2024-09-17T03:53:35.067630+0000","last_unstale":"2024-09-17T03:53:36.057970+0000","last_undegraded":"2024-09-17T03:53:36.057970+0000","last_fullsized":"2024-09-17T03:53:36.057970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:12.178924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249169+0000","last_change":"2024-09-17T03:53:35.064329+0000","last_active":"2024-09-17T03:53:36.249169+0000","last_peered":"2024-09-17T03:53:36.249169+0000","last_clean":"2024-09-17T03:53:36.249169+0000","last_became_active":"2024-09-17T03:53:35.062761+0000","last_became_peered":"2024-09-17T03:53:35.062761+0000","last_unstale":"2024-09-17T03:53:36.249169+0000","last_undegraded":"2024-09-17T03:53:36.249169+0000","last_fullsized":"2024-09-17T03:53:36.249169+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059290+0000","last_change":"2024-09-17T03:53:35.057286+0000","last_active":"2024-09-17T03:53:36.059290+0000","last_peered":"2024-09-17T03:53:36.059290+0000","last_clean":"2024-09-17T03:53:36.059290+0000","last_became_active":"2024-09-17T03:53:35.056910+0000","last_became_peered":"2024-09-17T03:53:35.056910+0000","last_unstale":"2024-09-17T03:53:36.059290+0000","last_undegraded":"2024-09-17T03:53:36.059290+0000","last_fullsized":"2024-09-17T03:53:36.059290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:48.490076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178706+0000","last_change":"2024-09-17T03:53:35.067698+0000","last_active":"2024-09-17T03:53:36.178706+0000","last_peered":"2024-09-17T03:53:36.178706+0000","last_clean":"2024-09-17T03:53:36.178706+0000","last_became_active":"2024-09-17T03:53:35.066584+0000","last_became_peered":"2024-09-17T03:53:35.066584+0000","last_unstale":"2024-09-17T03:53:36.178706+0000","last_undegraded":"2024-09-17T03:53:36.178706+0000","last_fullsized":"2024-09-17T03:53:36.178706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:57.060788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057393+0000","last_change":"2024-09-17T03:53:35.065693+0000","last_active":"2024-09-17T03:53:36.057393+0000","last_peered":"2024-09-17T03:53:36.057393+0000","last_clean":"2024-09-17T03:53:36.057393+0000","last_became_active":"2024-09-17T03:53:35.065403+0000","last_became_peered":"2024-09-17T03:53:35.065403+0000","last_unstale":"2024-09-17T03:53:36.057393+0000","last_undegraded":"2024-09-17T03:53:36.057393+0000","last_fullsized":"2024-09-17T03:53:36.057393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:50.272364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057291+0000","last_change":"2024-09-17T03:53:35.075375+0000","last_active":"2024-09-17T03:53:36.057291+0000","last_peered":"2024-09-17T03:53:36.057291+0000","last_clean":"2024-09-17T03:53:36.057291+0000","last_became_active":"2024-09-17T03:53:35.074718+0000","last_became_peered":"2024-09-17T03:53:35.074718+0000","last_unstale":"2024-09-17T03:53:36.057291+0000","last_undegraded":"2024-09-17T03:53:36.057291+0000","last_fullsized":"2024-09-17T03:53:36.057291+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:39.360722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058059+0000","last_change":"2024-09-17T03:53:35.069885+0000","last_active":"2024-09-17T03:53:36.058059+0000","last_peered":"2024-09-17T03:53:36.058059+0000","last_clean":"2024-09-17T03:53:36.058059+0000","last_became_active":"2024-09-17T03:53:35.069558+0000","last_became_peered":"2024-09-17T03:53:35.069558+0000","last_unstale":"2024-09-17T03:53:36.058059+0000","last_undegraded":"2024-09-17T03:53:36.058059+0000","last_fullsized":"2024-09-17T03:53:36.058059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:25.141426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178515+0000","last_change":"2024-09-17T03:53:35.062010+0000","last_active":"2024-09-17T03:53:36.178515+0000","last_peered":"2024-09-17T03:53:36.178515+0000","last_clean":"2024-09-17T03:53:36.178515+0000","last_became_active":"2024-09-17T03:53:35.059874+0000","last_became_peered":"2024-09-17T03:53:35.059874+0000","last_unstale":"2024-09-17T03:53:36.178515+0000","last_undegraded":"2024-09-17T03:53:36.178515+0000","last_fullsized":"2024-09-17T03:53:36.178515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:26.933349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060228+0000","last_change":"2024-09-17T03:53:35.065637+0000","last_active":"2024-09-17T03:53:36.060228+0000","last_peered":"2024-09-17T03:53:36.060228+0000","last_clean":"2024-09-17T03:53:36.060228+0000","last_became_active":"2024-09-17T03:53:35.065252+0000","last_became_peered":"2024-09-17T03:53:35.065252+0000","last_unstale":"2024-09-17T03:53:36.060228+0000","last_undegraded":"2024-09-17T03:53:36.060228+0000","last_fullsized":"2024-09-17T03:53:36.060228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178966+0000","last_change":"2024-09-17T03:53:34.042427+0000","last_active":"2024-09-17T03:53:36.178966+0000","last_peered":"2024-09-17T03:53:36.178966+0000","last_clean":"2024-09-17T03:53:36.178966+0000","last_became_active":"2024-09-17T03:53:34.042117+0000","last_became_peered":"2024-09-17T03:53:34.042117+0000","last_unstale":"2024-09-17T03:53:36.178966+0000","last_undegraded":"2024-09-17T03:53:36.178966+0000","last_fullsized":"2024-09-17T03:53:36.178966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:57.275556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178860+0000","last_change":"2024-09-17T03:53:35.075209+0000","last_active":"2024-09-17T03:53:36.178860+0000","last_peered":"2024-09-17T03:53:36.178860+0000","last_clean":"2024-09-17T03:53:36.178860+0000","last_became_active":"2024-09-17T03:53:35.074557+0000","last_became_peered":"2024-09-17T03:53:35.074557+0000","last_unstale":"2024-09-17T03:53:36.178860+0000","last_undegraded":"2024-09-17T03:53:36.178860+0000","last_fullsized":"2024-09-17T03:53:36.178860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:26.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060393+0000","last_change":"2024-09-17T03:53:34.047657+0000","last_active":"2024-09-17T03:53:36.060393+0000","last_peered":"2024-09-17T03:53:36.060393+0000","last_clean":"2024-09-17T03:53:36.060393+0000","last_became_active":"2024-09-17T03:53:34.047290+0000","last_became_peered":"2024-09-17T03:53:34.047290+0000","last_unstale":"2024-09-17T03:53:36.060393+0000","last_undegraded":"2024-09-17T03:53:36.060393+0000","last_fullsized":"2024-09-17T03:53:36.060393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:25.205958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178967+0000","last_change":"2024-09-17T03:53:35.068246+0000","last_active":"2024-09-17T03:53:36.178967+0000","last_peered":"2024-09-17T03:53:36.178967+0000","last_clean":"2024-09-17T03:53:36.178967+0000","last_became_active":"2024-09-17T03:53:35.066866+0000","last_became_peered":"2024-09-17T03:53:35.066866+0000","last_unstale":"2024-09-17T03:53:36.178967+0000","last_undegraded":"2024-09-17T03:53:36.178967+0000","last_fullsized":"2024-09-17T03:53:36.178967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:59.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529312+0000","last_change":"2024-09-17T03:53:34.047098+0000","last_active":"2024-09-17T03:53:36.529312+0000","last_peered":"2024-09-17T03:53:36.529312+0000","last_clean":"2024-09-17T03:53:36.529312+0000","last_became_active":"2024-09-17T03:53:34.046816+0000","last_became_peered":"2024-09-17T03:53:34.046816+0000","last_unstale":"2024-09-17T03:53:36.529312+0000","last_undegraded":"2024-09-17T03:53:36.529312+0000","last_fullsized":"2024-09-17T03:53:36.529312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:25.856611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249238+0000","last_change":"2024-09-17T03:53:35.067274+0000","last_active":"2024-09-17T03:53:36.249238+0000","last_peered":"2024-09-17T03:53:36.249238+0000","last_clean":"2024-09-17T03:53:36.249238+0000","last_became_active":"2024-09-17T03:53:35.066895+0000","last_became_peered":"2024-09-17T03:53:35.066895+0000","last_unstale":"2024-09-17T03:53:36.249238+0000","last_undegraded":"2024-09-17T03:53:36.249238+0000","last_fullsized":"2024-09-17T03:53:36.249238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529204+0000","last_change":"2024-09-17T03:53:34.047320+0000","last_active":"2024-09-17T03:53:36.529204+0000","last_peered":"2024-09-17T03:53:36.529204+0000","last_clean":"2024-09-17T03:53:36.529204+0000","last_became_active":"2024-09-17T03:53:34.045973+0000","last_became_peered":"2024-09-17T03:53:34.045973+0000","last_unstale":"2024-09-17T03:53:36.529204+0000","last_undegraded":"2024-09-17T03:53:36.529204+0000","last_fullsized":"2024-09-17T03:53:36.529204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:03.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057495+0000","last_change":"2024-09-17T03:53:35.075107+0000","last_active":"2024-09-17T03:53:36.057495+0000","last_peered":"2024-09-17T03:53:36.057495+0000","last_clean":"2024-09-17T03:53:36.057495+0000","last_became_active":"2024-09-17T03:53:35.069999+0000","last_became_peered":"2024-09-17T03:53:35.069999+0000","last_unstale":"2024-09-17T03:53:36.057495+0000","last_undegraded":"2024-09-17T03:53:36.057495+0000","last_fullsized":"2024-09-17T03:53:36.057495+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:15.911260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059049+0000","last_change":"2024-09-17T03:53:34.046521+0000","last_active":"2024-09-17T03:53:36.059049+0000","last_peered":"2024-09-17T03:53:36.059049+0000","last_clean":"2024-09-17T03:53:36.059049+0000","last_became_active":"2024-09-17T03:53:34.046151+0000","last_became_peered":"2024-09-17T03:53:34.046151+0000","last_unstale":"2024-09-17T03:53:36.059049+0000","last_undegraded":"2024-09-17T03:53:36.059049+0000","last_fullsized":"2024-09-17T03:53:36.059049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:04.714385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058821+0000","last_change":"2024-09-17T03:53:35.069264+0000","last_active":"2024-09-17T03:53:36.058821+0000","last_peered":"2024-09-17T03:53:36.058821+0000","last_clean":"2024-09-17T03:53:36.058821+0000","last_became_active":"2024-09-17T03:53:35.065458+0000","last_became_peered":"2024-09-17T03:53:35.065458+0000","last_unstale":"2024-09-17T03:53:36.058821+0000","last_undegraded":"2024-09-17T03:53:36.058821+0000","last_fullsized":"2024-09-17T03:53:36.058821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:16.559380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058298+0000","last_change":"2024-09-17T03:53:34.045410+0000","last_active":"2024-09-17T03:53:36.058298+0000","last_peered":"2024-09-17T03:53:36.058298+0000","last_clean":"2024-09-17T03:53:36.058298+0000","last_became_active":"2024-09-17T03:53:34.045030+0000","last_became_peered":"2024-09-17T03:53:34.045030+0000","last_unstale":"2024-09-17T03:53:36.058298+0000","last_undegraded":"2024-09-17T03:53:36.058298+0000","last_fullsized":"2024-09-17T03:53:36.058298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248910+0000","last_change":"2024-09-17T03:53:35.075625+0000","last_active":"2024-09-17T03:53:36.248910+0000","last_peered":"2024-09-17T03:53:36.248910+0000","last_clean":"2024-09-17T03:53:36.248910+0000","last_became_active":"2024-09-17T03:53:35.075285+0000","last_became_peered":"2024-09-17T03:53:35.075285+0000","last_unstale":"2024-09-17T03:53:36.248910+0000","last_undegraded":"2024-09-17T03:53:36.248910+0000","last_fullsized":"2024-09-17T03:53:36.248910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528363+0000","last_change":"2024-09-17T03:53:34.040513+0000","last_active":"2024-09-17T03:53:36.528363+0000","last_peered":"2024-09-17T03:53:36.528363+0000","last_clean":"2024-09-17T03:53:36.528363+0000","last_became_active":"2024-09-17T03:53:34.039387+0000","last_became_peered":"2024-09-17T03:53:34.039387+0000","last_unstale":"2024-09-17T03:53:36.528363+0000","last_undegraded":"2024-09-17T03:53:36.528363+0000","last_fullsized":"2024-09-17T03:53:36.528363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:35.514048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528897+0000","last_change":"2024-09-17T03:53:35.058824+0000","last_active":"2024-09-17T03:53:36.528897+0000","last_peered":"2024-09-17T03:53:36.528897+0000","last_clean":"2024-09-17T03:53:36.528897+0000","last_became_active":"2024-09-17T03:53:35.058053+0000","last_became_peered":"2024-09-17T03:53:35.058053+0000","last_unstale":"2024-09-17T03:53:36.528897+0000","last_undegraded":"2024-09-17T03:53:36.528897+0000","last_fullsized":"2024-09-17T03:53:36.528897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:39.813037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057165+0000","last_change":"2024-09-17T03:53:34.042323+0000","last_active":"2024-09-17T03:53:36.057165+0000","last_peered":"2024-09-17T03:53:36.057165+0000","last_clean":"2024-09-17T03:53:36.057165+0000","last_became_active":"2024-09-17T03:53:34.041995+0000","last_became_peered":"2024-09-17T03:53:34.041995+0000","last_unstale":"2024-09-17T03:53:36.057165+0000","last_undegraded":"2024-09-17T03:53:36.057165+0000","last_fullsized":"2024-09-17T03:53:36.057165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:54.594238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.530067+0000","last_change":"2024-09-17T03:53:35.066310+0000","last_active":"2024-09-17T03:53:36.530067+0000","last_peered":"2024-09-17T03:53:36.530067+0000","last_clean":"2024-09-17T03:53:36.530067+0000","last_became_active":"2024-09-17T03:53:35.065211+0000","last_became_peered":"2024-09-17T03:53:35.065211+0000","last_unstale":"2024-09-17T03:53:36.530067+0000","last_undegraded":"2024-09-17T03:53:36.530067+0000","last_fullsized":"2024-09-17T03:53:36.530067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:11.925576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179093+0000","last_change":"2024-09-17T03:53:34.041277+0000","last_active":"2024-09-17T03:53:36.179093+0000","last_peered":"2024-09-17T03:53:36.179093+0000","last_clean":"2024-09-17T03:53:36.179093+0000","last_became_active":"2024-09-17T03:53:34.040555+0000","last_became_peered":"2024-09-17T03:53:34.040555+0000","last_unstale":"2024-09-17T03:53:36.179093+0000","last_undegraded":"2024-09-17T03:53:36.179093+0000","last_fullsized":"2024-09-17T03:53:36.179093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:05.315114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059958+0000","last_change":"2024-09-17T03:53:35.058829+0000","last_active":"2024-09-17T03:53:36.059958+0000","last_peered":"2024-09-17T03:53:36.059958+0000","last_clean":"2024-09-17T03:53:36.059958+0000","last_became_active":"2024-09-17T03:53:35.058494+0000","last_became_peered":"2024-09-17T03:53:35.058494+0000","last_unstale":"2024-09-17T03:53:36.059958+0000","last_undegraded":"2024-09-17T03:53:36.059958+0000","last_fullsized":"2024-09-17T03:53:36.059958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:09.497624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178778+0000","last_change":"2024-09-17T03:53:34.047020+0000","last_active":"2024-09-17T03:53:36.178778+0000","last_peered":"2024-09-17T03:53:36.178778+0000","last_clean":"2024-09-17T03:53:36.178778+0000","last_became_active":"2024-09-17T03:53:34.045487+0000","last_became_peered":"2024-09-17T03:53:34.045487+0000","last_unstale":"2024-09-17T03:53:36.178778+0000","last_undegraded":"2024-09-17T03:53:36.178778+0000","last_fullsized":"2024-09-17T03:53:36.178778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:03.205170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529300+0000","last_change":"2024-09-17T03:53:35.074637+0000","last_active":"2024-09-17T03:53:36.529300+0000","last_peered":"2024-09-17T03:53:36.529300+0000","last_clean":"2024-09-17T03:53:36.529300+0000","last_became_active":"2024-09-17T03:53:35.074086+0000","last_became_peered":"2024-09-17T03:53:35.074086+0000","last_unstale":"2024-09-17T03:53:36.529300+0000","last_undegraded":"2024-09-17T03:53:36.529300+0000","last_fullsized":"2024-09-17T03:53:36.529300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.027879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178381+0000","last_change":"2024-09-17T03:53:34.045567+0000","last_active":"2024-09-17T03:53:36.178381+0000","last_peered":"2024-09-17T03:53:36.178381+0000","last_clean":"2024-09-17T03:53:36.178381+0000","last_became_active":"2024-09-17T03:53:34.045029+0000","last_became_peered":"2024-09-17T03:53:34.045029+0000","last_unstale":"2024-09-17T03:53:36.178381+0000","last_undegraded":"2024-09-17T03:53:36.178381+0000","last_fullsized":"2024-09-17T03:53:36.178381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:38.299845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059436+0000","last_change":"2024-09-17T03:53:35.078726+0000","last_active":"2024-09-17T03:53:36.059436+0000","last_peered":"2024-09-17T03:53:36.059436+0000","last_clean":"2024-09-17T03:53:36.059436+0000","last_became_active":"2024-09-17T03:53:35.078369+0000","last_became_peered":"2024-09-17T03:53:35.078369+0000","last_unstale":"2024-09-17T03:53:36.059436+0000","last_undegraded":"2024-09-17T03:53:36.059436+0000","last_fullsized":"2024-09-17T03:53:36.059436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059227+0000","last_change":"2024-09-17T03:53:34.033143+0000","last_active":"2024-09-17T03:53:36.059227+0000","last_peered":"2024-09-17T03:53:36.059227+0000","last_clean":"2024-09-17T03:53:36.059227+0000","last_became_active":"2024-09-17T03:53:34.031770+0000","last_became_peered":"2024-09-17T03:53:34.031770+0000","last_unstale":"2024-09-17T03:53:36.059227+0000","last_undegraded":"2024-09-17T03:53:36.059227+0000","last_fullsized":"2024-09-17T03:53:36.059227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529438+0000","last_change":"2024-09-17T03:53:35.062155+0000","last_active":"2024-09-17T03:53:36.529438+0000","last_peered":"2024-09-17T03:53:36.529438+0000","last_clean":"2024-09-17T03:53:36.529438+0000","last_became_active":"2024-09-17T03:53:35.061536+0000","last_became_peered":"2024-09-17T03:53:35.061536+0000","last_unstale":"2024-09-17T03:53:36.529438+0000","last_undegraded":"2024-09-17T03:53:36.529438+0000","last_fullsized":"2024-09-17T03:53:36.529438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:36.274547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057583+0000","last_change":"2024-09-17T03:53:34.043064+0000","last_active":"2024-09-17T03:53:36.057583+0000","last_peered":"2024-09-17T03:53:36.057583+0000","last_clean":"2024-09-17T03:53:36.057583+0000","last_became_active":"2024-09-17T03:53:34.038696+0000","last_became_peered":"2024-09-17T03:53:34.038696+0000","last_unstale":"2024-09-17T03:53:36.057583+0000","last_undegraded":"2024-09-17T03:53:36.057583+0000","last_fullsized":"2024-09-17T03:53:36.057583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:29.561790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249364+0000","last_change":"2024-09-17T03:53:35.063772+0000","last_active":"2024-09-17T03:53:36.249364+0000","last_peered":"2024-09-17T03:53:36.249364+0000","last_clean":"2024-09-17T03:53:36.249364+0000","last_became_active":"2024-09-17T03:53:35.063400+0000","last_became_peered":"2024-09-17T03:53:35.063400+0000","last_unstale":"2024-09-17T03:53:36.249364+0000","last_undegraded":"2024-09-17T03:53:36.249364+0000","last_fullsized":"2024-09-17T03:53:36.249364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:14.277688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529588+0000","last_change":"2024-09-17T03:53:34.048029+0000","last_active":"2024-09-17T03:53:36.529588+0000","last_peered":"2024-09-17T03:53:36.529588+0000","last_clean":"2024-09-17T03:53:36.529588+0000","last_became_active":"2024-09-17T03:53:34.046461+0000","last_became_peered":"2024-09-17T03:53:34.046461+0000","last_unstale":"2024-09-17T03:53:36.529588+0000","last_undegraded":"2024-09-17T03:53:36.529588+0000","last_fullsized":"2024-09-17T03:53:36.529588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:07.500302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057588+0000","last_change":"2024-09-17T03:53:35.062083+0000","last_active":"2024-09-17T03:53:36.057588+0000","last_peered":"2024-09-17T03:53:36.057588+0000","last_clean":"2024-09-17T03:53:36.057588+0000","last_became_active":"2024-09-17T03:53:35.061800+0000","last_became_peered":"2024-09-17T03:53:35.061800+0000","last_unstale":"2024-09-17T03:53:36.057588+0000","last_undegraded":"2024-09-17T03:53:36.057588+0000","last_fullsized":"2024-09-17T03:53:36.057588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:49.841803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059500+0000","last_change":"2024-09-17T03:53:34.045446+0000","last_active":"2024-09-17T03:53:36.059500+0000","last_peered":"2024-09-17T03:53:36.059500+0000","last_clean":"2024-09-17T03:53:36.059500+0000","last_became_active":"2024-09-17T03:53:34.045103+0000","last_became_peered":"2024-09-17T03:53:34.045103+0000","last_unstale":"2024-09-17T03:53:36.059500+0000","last_undegraded":"2024-09-17T03:53:36.059500+0000","last_fullsized":"2024-09-17T03:53:36.059500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057116+0000","last_change":"2024-09-17T03:53:35.068169+0000","last_active":"2024-09-17T03:53:36.057116+0000","last_peered":"2024-09-17T03:53:36.057116+0000","last_clean":"2024-09-17T03:53:36.057116+0000","last_became_active":"2024-09-17T03:53:35.067861+0000","last_became_peered":"2024-09-17T03:53:35.067861+0000","last_unstale":"2024-09-17T03:53:36.057116+0000","last_undegraded":"2024-09-17T03:53:36.057116+0000","last_fullsized":"2024-09-17T03:53:36.057116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:58.932353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528873+0000","last_change":"2024-09-17T03:53:34.043445+0000","last_active":"2024-09-17T03:53:36.528873+0000","last_peered":"2024-09-17T03:53:36.528873+0000","last_clean":"2024-09-17T03:53:36.528873+0000","last_became_active":"2024-09-17T03:53:34.042920+0000","last_became_peered":"2024-09-17T03:53:34.042920+0000","last_unstale":"2024-09-17T03:53:36.528873+0000","last_undegraded":"2024-09-17T03:53:36.528873+0000","last_fullsized":"2024-09-17T03:53:36.528873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:42.885526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249340+0000","last_change":"2024-09-17T03:53:35.061631+0000","last_active":"2024-09-17T03:53:36.249340+0000","last_peered":"2024-09-17T03:53:36.249340+0000","last_clean":"2024-09-17T03:53:36.249340+0000","last_became_active":"2024-09-17T03:53:35.061273+0000","last_became_peered":"2024-09-17T03:53:35.061273+0000","last_unstale":"2024-09-17T03:53:36.249340+0000","last_undegraded":"2024-09-17T03:53:36.249340+0000","last_fullsized":"2024-09-17T03:53:36.249340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:08.506998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179205+0000","last_change":"2024-09-17T03:53:34.041969+0000","last_active":"2024-09-17T03:53:36.179205+0000","last_peered":"2024-09-17T03:53:36.179205+0000","last_clean":"2024-09-17T03:53:36.179205+0000","last_became_active":"2024-09-17T03:53:34.040090+0000","last_became_peered":"2024-09-17T03:53:34.040090+0000","last_unstale":"2024-09-17T03:53:36.179205+0000","last_undegraded":"2024-09-17T03:53:36.179205+0000","last_fullsized":"2024-09-17T03:53:36.179205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:12.790859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179104+0000","last_change":"2024-09-17T03:53:35.071124+0000","last_active":"2024-09-17T03:53:36.179104+0000","last_peered":"2024-09-17T03:53:36.179104+0000","last_clean":"2024-09-17T03:53:36.179104+0000","last_became_active":"2024-09-17T03:53:35.070636+0000","last_became_peered":"2024-09-17T03:53:35.070636+0000","last_unstale":"2024-09-17T03:53:36.179104+0000","last_undegraded":"2024-09-17T03:53:36.179104+0000","last_fullsized":"2024-09-17T03:53:36.179104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:18.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529575+0000","last_change":"2024-09-17T03:53:34.049033+0000","last_active":"2024-09-17T03:53:36.529575+0000","last_peered":"2024-09-17T03:53:36.529575+0000","last_clean":"2024-09-17T03:53:36.529575+0000","last_became_active":"2024-09-17T03:53:34.048596+0000","last_became_peered":"2024-09-17T03:53:34.048596+0000","last_unstale":"2024-09-17T03:53:36.529575+0000","last_undegraded":"2024-09-17T03:53:36.529575+0000","last_fullsized":"2024-09-17T03:53:36.529575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:25.851358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059564+0000","last_change":"2024-09-17T03:53:35.052022+0000","last_active":"2024-09-17T03:53:36.059564+0000","last_peered":"2024-09-17T03:53:36.059564+0000","last_clean":"2024-09-17T03:53:36.059564+0000","last_became_active":"2024-09-17T03:53:35.051704+0000","last_became_peered":"2024-09-17T03:53:35.051704+0000","last_unstale":"2024-09-17T03:53:36.059564+0000","last_undegraded":"2024-09-17T03:53:36.059564+0000","last_fullsized":"2024-09-17T03:53:36.059564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529714+0000","last_change":"2024-09-17T03:53:34.040065+0000","last_active":"2024-09-17T03:53:36.529714+0000","last_peered":"2024-09-17T03:53:36.529714+0000","last_clean":"2024-09-17T03:53:36.529714+0000","last_became_active":"2024-09-17T03:53:34.039675+0000","last_became_peered":"2024-09-17T03:53:34.039675+0000","last_unstale":"2024-09-17T03:53:36.529714+0000","last_undegraded":"2024-09-17T03:53:36.529714+0000","last_fullsized":"2024-09-17T03:53:36.529714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:40.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528473+0000","last_change":"2024-09-17T03:53:35.068226+0000","last_active":"2024-09-17T03:53:36.528473+0000","last_peered":"2024-09-17T03:53:36.528473+0000","last_clean":"2024-09-17T03:53:36.528473+0000","last_became_active":"2024-09-17T03:53:35.067943+0000","last_became_peered":"2024-09-17T03:53:35.067943+0000","last_unstale":"2024-09-17T03:53:36.528473+0000","last_undegraded":"2024-09-17T03:53:36.528473+0000","last_fullsized":"2024-09-17T03:53:36.528473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:09.141910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529817+0000","last_change":"2024-09-17T03:53:34.042295+0000","last_active":"2024-09-17T03:53:36.529817+0000","last_peered":"2024-09-17T03:53:36.529817+0000","last_clean":"2024-09-17T03:53:36.529817+0000","last_became_active":"2024-09-17T03:53:34.041992+0000","last_became_peered":"2024-09-17T03:53:34.041992+0000","last_unstale":"2024-09-17T03:53:36.529817+0000","last_undegraded":"2024-09-17T03:53:36.529817+0000","last_fullsized":"2024-09-17T03:53:36.529817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:16.446587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058700+0000","last_change":"2024-09-17T03:53:35.063084+0000","last_active":"2024-09-17T03:53:36.058700+0000","last_peered":"2024-09-17T03:53:36.058700+0000","last_clean":"2024-09-17T03:53:36.058700+0000","last_became_active":"2024-09-17T03:53:35.062759+0000","last_became_peered":"2024-09-17T03:53:35.062759+0000","last_unstale":"2024-09-17T03:53:36.058700+0000","last_undegraded":"2024-09-17T03:53:36.058700+0000","last_fullsized":"2024-09-17T03:53:36.058700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:22.612393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528628+0000","last_change":"2024-09-17T03:53:34.050809+0000","last_active":"2024-09-17T03:53:36.528628+0000","last_peered":"2024-09-17T03:53:36.528628+0000","last_clean":"2024-09-17T03:53:36.528628+0000","last_became_active":"2024-09-17T03:53:34.050498+0000","last_became_peered":"2024-09-17T03:53:34.050498+0000","last_unstale":"2024-09-17T03:53:36.528628+0000","last_undegraded":"2024-09-17T03:53:36.528628+0000","last_fullsized":"2024-09-17T03:53:36.528628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:01.191543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527911+0000","last_change":"2024-09-17T03:53:35.059466+0000","last_active":"2024-09-17T03:53:36.527911+0000","last_peered":"2024-09-17T03:53:36.527911+0000","last_clean":"2024-09-17T03:53:36.527911+0000","last_became_active":"2024-09-17T03:53:35.059160+0000","last_became_peered":"2024-09-17T03:53:35.059160+0000","last_unstale":"2024-09-17T03:53:36.527911+0000","last_undegraded":"2024-09-17T03:53:36.527911+0000","last_fullsized":"2024-09-17T03:53:36.527911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:15.257783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527972+0000","last_change":"2024-09-17T03:53:34.046054+0000","last_active":"2024-09-17T03:53:36.527972+0000","last_peered":"2024-09-17T03:53:36.527972+0000","last_clean":"2024-09-17T03:53:36.527972+0000","last_became_active":"2024-09-17T03:53:34.045718+0000","last_became_peered":"2024-09-17T03:53:34.045718+0000","last_unstale":"2024-09-17T03:53:36.527972+0000","last_undegraded":"2024-09-17T03:53:36.527972+0000","last_fullsized":"2024-09-17T03:53:36.527972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:09.572328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059323+0000","last_change":"2024-09-17T03:53:35.068451+0000","last_active":"2024-09-17T03:53:36.059323+0000","last_peered":"2024-09-17T03:53:36.059323+0000","last_clean":"2024-09-17T03:53:36.059323+0000","last_became_active":"2024-09-17T03:53:35.067500+0000","last_became_peered":"2024-09-17T03:53:35.067500+0000","last_unstale":"2024-09-17T03:53:36.059323+0000","last_undegraded":"2024-09-17T03:53:36.059323+0000","last_fullsized":"2024-09-17T03:53:36.059323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:10.103551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528173+0000","last_change":"2024-09-17T03:53:34.045835+0000","last_active":"2024-09-17T03:53:36.528173+0000","last_peered":"2024-09-17T03:53:36.528173+0000","last_clean":"2024-09-17T03:53:36.528173+0000","last_became_active":"2024-09-17T03:53:34.045329+0000","last_became_peered":"2024-09-17T03:53:34.045329+0000","last_unstale":"2024-09-17T03:53:36.528173+0000","last_undegraded":"2024-09-17T03:53:36.528173+0000","last_fullsized":"2024-09-17T03:53:36.528173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:07.278692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060312+0000","last_change":"2024-09-17T03:53:35.045510+0000","last_active":"2024-09-17T03:53:36.060312+0000","last_peered":"2024-09-17T03:53:36.060312+0000","last_clean":"2024-09-17T03:53:36.060312+0000","last_became_active":"2024-09-17T03:53:35.044996+0000","last_became_peered":"2024-09-17T03:53:35.044996+0000","last_unstale":"2024-09-17T03:53:36.060312+0000","last_undegraded":"2024-09-17T03:53:36.060312+0000","last_fullsized":"2024-09-17T03:53:36.060312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058140+0000","last_change":"2024-09-17T03:53:34.043457+0000","last_active":"2024-09-17T03:53:36.058140+0000","last_peered":"2024-09-17T03:53:36.058140+0000","last_clean":"2024-09-17T03:53:36.058140+0000","last_became_active":"2024-09-17T03:53:34.043142+0000","last_became_peered":"2024-09-17T03:53:34.043142+0000","last_unstale":"2024-09-17T03:53:36.058140+0000","last_undegraded":"2024-09-17T03:53:36.058140+0000","last_fullsized":"2024-09-17T03:53:36.058140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:14.453153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249615+0000","last_change":"2024-09-17T03:53:35.077390+0000","last_active":"2024-09-17T03:53:36.249615+0000","last_peered":"2024-09-17T03:53:36.249615+0000","last_clean":"2024-09-17T03:53:36.249615+0000","last_became_active":"2024-09-17T03:53:35.077039+0000","last_became_peered":"2024-09-17T03:53:35.077039+0000","last_unstale":"2024-09-17T03:53:36.249615+0000","last_undegraded":"2024-09-17T03:53:36.249615+0000","last_fullsized":"2024-09-17T03:53:36.249615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:34.715128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058032+0000","last_change":"2024-09-17T03:53:34.047344+0000","last_active":"2024-09-17T03:53:36.058032+0000","last_peered":"2024-09-17T03:53:36.058032+0000","last_clean":"2024-09-17T03:53:36.058032+0000","last_became_active":"2024-09-17T03:53:34.041909+0000","last_became_peered":"2024-09-17T03:53:34.041909+0000","last_unstale":"2024-09-17T03:53:36.058032+0000","last_undegraded":"2024-09-17T03:53:36.058032+0000","last_fullsized":"2024-09-17T03:53:36.058032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:45.691778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179022+0000","last_change":"2024-09-17T03:53:35.060992+0000","last_active":"2024-09-17T03:53:36.179022+0000","last_peered":"2024-09-17T03:53:36.179022+0000","last_clean":"2024-09-17T03:53:36.179022+0000","last_became_active":"2024-09-17T03:53:35.060339+0000","last_became_peered":"2024-09-17T03:53:35.060339+0000","last_unstale":"2024-09-17T03:53:36.179022+0000","last_undegraded":"2024-09-17T03:53:36.179022+0000","last_fullsized":"2024-09-17T03:53:36.179022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:48.433949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178476+0000","last_change":"2024-09-17T03:53:34.042515+0000","last_active":"2024-09-17T03:53:36.178476+0000","last_peered":"2024-09-17T03:53:36.178476+0000","last_clean":"2024-09-17T03:53:36.178476+0000","last_became_active":"2024-09-17T03:53:34.041350+0000","last_became_peered":"2024-09-17T03:53:34.041350+0000","last_unstale":"2024-09-17T03:53:36.178476+0000","last_undegraded":"2024-09-17T03:53:36.178476+0000","last_fullsized":"2024-09-17T03:53:36.178476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:54.121078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059705+0000","last_change":"2024-09-17T03:53:35.078734+0000","last_active":"2024-09-17T03:53:36.059705+0000","last_peered":"2024-09-17T03:53:36.059705+0000","last_clean":"2024-09-17T03:53:36.059705+0000","last_became_active":"2024-09-17T03:53:35.078432+0000","last_became_peered":"2024-09-17T03:53:35.078432+0000","last_unstale":"2024-09-17T03:53:36.059705+0000","last_undegraded":"2024-09-17T03:53:36.059705+0000","last_fullsized":"2024-09-17T03:53:36.059705+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179021+0000","last_change":"2024-09-17T03:53:34.043860+0000","last_active":"2024-09-17T03:53:36.179021+0000","last_peered":"2024-09-17T03:53:36.179021+0000","last_clean":"2024-09-17T03:53:36.179021+0000","last_became_active":"2024-09-17T03:53:34.043502+0000","last_became_peered":"2024-09-17T03:53:34.043502+0000","last_unstale":"2024-09-17T03:53:36.179021+0000","last_undegraded":"2024-09-17T03:53:36.179021+0000","last_fullsized":"2024-09-17T03:53:36.179021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:02.362312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058992+0000","last_change":"2024-09-17T03:53:35.064082+0000","last_active":"2024-09-17T03:53:36.058992+0000","last_peered":"2024-09-17T03:53:36.058992+0000","last_clean":"2024-09-17T03:53:36.058992+0000","last_became_active":"2024-09-17T03:53:35.063675+0000","last_became_peered":"2024-09-17T03:53:35.063675+0000","last_unstale":"2024-09-17T03:53:36.058992+0000","last_undegraded":"2024-09-17T03:53:36.058992+0000","last_fullsized":"2024-09-17T03:53:36.058992+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:11.219352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529388+0000","last_change":"2024-09-17T03:53:34.037735+0000","last_active":"2024-09-17T03:53:36.529388+0000","last_peered":"2024-09-17T03:53:36.529388+0000","last_clean":"2024-09-17T03:53:36.529388+0000","last_became_active":"2024-09-17T03:53:34.037448+0000","last_became_peered":"2024-09-17T03:53:34.037448+0000","last_unstale":"2024-09-17T03:53:36.529388+0000","last_undegraded":"2024-09-17T03:53:36.529388+0000","last_fullsized":"2024-09-17T03:53:36.529388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:52.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529554+0000","last_change":"2024-09-17T03:53:35.071423+0000","last_active":"2024-09-17T03:53:36.529554+0000","last_peered":"2024-09-17T03:53:36.529554+0000","last_clean":"2024-09-17T03:53:36.529554+0000","last_became_active":"2024-09-17T03:53:35.071008+0000","last_became_peered":"2024-09-17T03:53:35.071008+0000","last_unstale":"2024-09-17T03:53:36.529554+0000","last_undegraded":"2024-09-17T03:53:36.529554+0000","last_fullsized":"2024-09-17T03:53:36.529554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:43.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058428+0000","last_change":"2024-09-17T03:53:34.045408+0000","last_active":"2024-09-17T03:53:36.058428+0000","last_peered":"2024-09-17T03:53:36.058428+0000","last_clean":"2024-09-17T03:53:36.058428+0000","last_became_active":"2024-09-17T03:53:34.045021+0000","last_became_peered":"2024-09-17T03:53:34.045021+0000","last_unstale":"2024-09-17T03:53:36.058428+0000","last_undegraded":"2024-09-17T03:53:36.058428+0000","last_fullsized":"2024-09-17T03:53:36.058428+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248920+0000","last_change":"2024-09-17T03:53:35.070768+0000","last_active":"2024-09-17T03:53:36.248920+0000","last_peered":"2024-09-17T03:53:36.248920+0000","last_clean":"2024-09-17T03:53:36.248920+0000","last_became_active":"2024-09-17T03:53:35.070352+0000","last_became_peered":"2024-09-17T03:53:35.070352+0000","last_unstale":"2024-09-17T03:53:36.248920+0000","last_undegraded":"2024-09-17T03:53:36.248920+0000","last_fullsized":"2024-09-17T03:53:36.248920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:11.336036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178647+0000","last_change":"2024-09-17T03:53:34.041865+0000","last_active":"2024-09-17T03:53:36.178647+0000","last_peered":"2024-09-17T03:53:36.178647+0000","last_clean":"2024-09-17T03:53:36.178647+0000","last_became_active":"2024-09-17T03:53:34.041169+0000","last_became_peered":"2024-09-17T03:53:34.041169+0000","last_unstale":"2024-09-17T03:53:36.178647+0000","last_undegraded":"2024-09-17T03:53:36.178647+0000","last_fullsized":"2024-09-17T03:53:36.178647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:16.044280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178489+0000","last_change":"2024-09-17T03:53:35.064940+0000","last_active":"2024-09-17T03:53:36.178489+0000","last_peered":"2024-09-17T03:53:36.178489+0000","last_clean":"2024-09-17T03:53:36.178489+0000","last_became_active":"2024-09-17T03:53:35.064574+0000","last_became_peered":"2024-09-17T03:53:35.064574+0000","last_unstale":"2024-09-17T03:53:36.178489+0000","last_undegraded":"2024-09-17T03:53:36.178489+0000","last_fullsized":"2024-09-17T03:53:36.178489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:08.160089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057951+0000","last_change":"2024-09-17T03:53:34.048127+0000","last_active":"2024-09-17T03:53:36.057951+0000","last_peered":"2024-09-17T03:53:36.057951+0000","last_clean":"2024-09-17T03:53:36.057951+0000","last_became_active":"2024-09-17T03:53:34.047768+0000","last_became_peered":"2024-09-17T03:53:34.047768+0000","last_unstale":"2024-09-17T03:53:36.057951+0000","last_undegraded":"2024-09-17T03:53:36.057951+0000","last_fullsized":"2024-09-17T03:53:36.057951+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:17.363867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057698+0000","last_change":"2024-09-17T03:53:35.051987+0000","last_active":"2024-09-17T03:53:36.057698+0000","last_peered":"2024-09-17T03:53:36.057698+0000","last_clean":"2024-09-17T03:53:36.057698+0000","last_became_active":"2024-09-17T03:53:35.049620+0000","last_became_peered":"2024-09-17T03:53:35.049620+0000","last_unstale":"2024-09-17T03:53:36.057698+0000","last_undegraded":"2024-09-17T03:53:36.057698+0000","last_fullsized":"2024-09-17T03:53:36.057698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:21.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249368+0000","last_change":"2024-09-17T03:53:34.046914+0000","last_active":"2024-09-17T03:53:36.249368+0000","last_peered":"2024-09-17T03:53:36.249368+0000","last_clean":"2024-09-17T03:53:36.249368+0000","last_became_active":"2024-09-17T03:53:34.046056+0000","last_became_peered":"2024-09-17T03:53:34.046056+0000","last_unstale":"2024-09-17T03:53:36.249368+0000","last_undegraded":"2024-09-17T03:53:36.249368+0000","last_fullsized":"2024-09-17T03:53:36.249368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:53.290335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":219,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249351+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:53:36.249351+0000","last_peered":"2024-09-17T03:53:36.249351+0000","last_clean":"2024-09-17T03:53:36.249351+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:53:36.249351+0000","last_undegraded":"2024-09-17T03:53:36.249351+0000","last_fullsized":"2024-09-17T03:53:36.249351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509493,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509496,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":417572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509495,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":415938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509493,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":417550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":416520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509495,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":416132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542197,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":999424,"data_stored":416632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542198,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:53:40.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T03:53:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 -- 172.21.15.129:0/2985873861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002c003c10 msgr2=0x7f002c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 --2- 172.21.15.129:0/2985873861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f002c003c10 0x7f002c004060 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f0038006f90 tx=0x7f002c005830 comp rx=0 tx=0).stop 2024-09-17T03:53:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004cc00640 1 -- 172.21.15.129:0/2985873861 reap_dead start 2024-09-17T03:53:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 -- 172.21.15.129:0/2985873861 shutdown_connections 2024-09-17T03:53:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 -- 172.21.15.129:0/2985873861 >> 172.21.15.129:0/2985873861 conn(0x7f004806d410 msgr2=0x7f00480b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 -- 172.21.15.129:0/2985873861 shutdown_connections 2024-09-17T03:53:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.703+0000 7f004d600640 1 -- 172.21.15.129:0/2985873861 wait complete. 2024-09-17T03:53:40.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.705+0000 7f004d600640 1 Processor -- start 2024-09-17T03:53:40.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.705+0000 7f004d600640 1 -- start start 2024-09-17T03:53:40.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 0x7f00480c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00481e8f00 0x7f00481e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 0x7f0048143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00480b3c80 con 0x7f0048143610 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f004815f8a0 con 0x7f00481e8f00 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f00480b3e00 con 0x7f002c003c10 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 0x7f0048143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f0047e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 0x7f00480c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f0047e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 0x7f00480c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:53278/0 (socket says 172.21.15.129:53278) 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f0047e00640 1 -- 172.21.15.129:0/3238012129 learned_addr learned my addr 172.21.15.129:0/3238012129 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f0047400640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00481e8f00 0x7f00481e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 -- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 msgr2=0x7f00480c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 0x7f00480c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 -- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00481e8f00 msgr2=0x7f00481e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00481e8f00 0x7f00481e9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.707+0000 7f004cc00640 1 -- 172.21.15.129:0/3238012129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00480c6950 con 0x7f0048143610 2024-09-17T03:53:40.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f004cc00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 0x7f0048143a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0048070fa0 tx=0x7f0040001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040011870 con 0x7f0048143610 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0040017070 con 0x7f0048143610 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481e9850 con 0x7f0048143610 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0040011de0 con 0x7f0048143610 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.708+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00481f94b0 con 0x7f0048143610 2024-09-17T03:53:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.709+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004815fbd0 con 0x7f0048143610 2024-09-17T03:53:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.711+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f004001c460 con 0x7f0048143610 2024-09-17T03:53:40.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.712+0000 7f003fe00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 0x7f002807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:40.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.712+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f004009b480 con 0x7f0048143610 2024-09-17T03:53:40.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.713+0000 7f0047e00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 0x7f002807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:40.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.713+0000 7f0047e00640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 0x7f002807b0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0038000f60 tx=0x7f0048073a30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:40.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.717+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f00481e9850 con 0x7f0048143610 2024-09-17T03:53:40.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.968+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f00480c66c0 con 0x7f0048143610 2024-09-17T03:53:40.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.968+0000 7f003fe00640 1 -- 172.21.15.129:0/3238012129 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1430 (secure 0 0 0) 0x7f0040068e20 con 0x7f0048143610 2024-09-17T03:53:40.968 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:40.968 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":272,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":81,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":49},{"state_name":"active+clean","count":48}],"num_pgs":97,"num_pools":3,"num_objects":14,"data_bytes":591062,"bytes_used":264380416,"bytes_avail":772829732864,"bytes_total":773094113280,"inactive_pgs_ratio":0.50515460968017578,"write_bytes_sec":574,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":6049}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-17T03:53:30.357789+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 msgr2=0x7f002807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 0x7f002807b0d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0038000f60 tx=0x7f0048073a30 comp rx=0 tx=0).stop 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 msgr2=0x7f0048143a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 0x7f0048143a20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f0048070fa0 tx=0x7f0040001b80 comp rx=0 tx=0).stop 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 shutdown_connections 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0028078bc0 0x7f002807b0d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0048143610 0x7f0048143a20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00481e8f00 0x7f00481e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 --2- 172.21.15.129:0/3238012129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f002c003c10 0x7f00480c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:40.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 >> 172.21.15.129:0/3238012129 conn(0x7f004806d410 msgr2=0x7f0048068e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:40.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 shutdown_connections 2024-09-17T03:53:40.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:40.971+0000 7f004d600640 1 -- 172.21.15.129:0/3238012129 wait complete. 2024-09-17T03:53:40.994 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T03:53:40.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:53:41.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 -- 172.21.15.129:0/561976631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 msgr2=0x7fa62c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:41.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 --2- 172.21.15.129:0/561976631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c0ccf60 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa62402aa90 comp rx=0 tx=0).stop 2024-09-17T03:53:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 -- 172.21.15.129:0/561976631 shutdown_connections 2024-09-17T03:53:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 --2- 172.21.15.129:0/561976631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c0ccf60 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 --2- 172.21.15.129:0/561976631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 0x7fa62c0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 -- 172.21.15.129:0/561976631 >> 172.21.15.129:0/561976631 conn(0x7fa62c06d7d0 msgr2=0x7fa62c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 -- 172.21.15.129:0/561976631 shutdown_connections 2024-09-17T03:53:41.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.217+0000 7fa633400640 1 -- 172.21.15.129:0/561976631 wait complete. 2024-09-17T03:53:41.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.220+0000 7fa633400640 1 Processor -- start 2024-09-17T03:53:41.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.220+0000 7fa633400640 1 -- start start 2024-09-17T03:53:41.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 0x7fa62c1f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 0x7fa62c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c153520 con 0x7fa62c0ccb50 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1533a0 con 0x7fa62c15d8a0 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa633400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c1536a0 con 0x7fa62c0cc1e0 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa62be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa62be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49144/0 (socket says 172.21.15.129:49144) 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.221+0000 7fa62be00640 1 -- 172.21.15.129:0/1143291090 learned_addr learned my addr 172.21.15.129:0/1143291090 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa630a00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 0x7fa62c1f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa631400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 0x7fa62c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 -- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 msgr2=0x7fa62c1f9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:41.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 0x7fa62c1f9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 -- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 msgr2=0x7fa62c1587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 0x7fa62c1587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 -- 172.21.15.129:0/1143291090 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c15a5e0 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa631400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 0x7fa62c1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa62be00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa61800c3a0 tx=0x7fa61800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618017070 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa6180053c0 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa618003850 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c159a50 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.222+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c158120 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.224+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c153b50 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.225+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa61801a070 con 0x7fa62c0ccb50 2024-09-17T03:53:41.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.226+0000 7fa628c00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 0x7fa60c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:41.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.226+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fa61809a180 con 0x7fa62c0ccb50 2024-09-17T03:53:41.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.226+0000 7fa630a00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 0x7fa60c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:41.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.227+0000 7fa630a00640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 0x7fa60c07b0d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa620009ac0 tx=0x7fa620005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:41.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.230+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa62c0cd4a0 con 0x7fa62c0ccb50 2024-09-17T03:53:41.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.412+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa62c156e90 con 0x7fa60c078bc0 2024-09-17T03:53:41.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.418+0000 7fa628c00640 1 -- 172.21.15.129:0/1143291090 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252064 (secure 0 0 0) 0x7fa62c156e90 con 0x7fa60c078bc0 2024-09-17T03:53:41.420 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:41.421 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:53:41.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 msgr2=0x7fa60c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 0x7fa60c07b0d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa620009ac0 tx=0x7fa620005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 msgr2=0x7fa62c1e9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fa61800c3a0 tx=0x7fa61800c860 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 shutdown_connections 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa60c078bc0 0x7fa60c07b0d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa62c15d8a0 0x7fa62c1587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa62c0ccb50 0x7fa62c1e9520 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 --2- 172.21.15.129:0/1143291090 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa62c0cc1e0 0x7fa62c1f9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.422+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 >> 172.21.15.129:0/1143291090 conn(0x7fa62c06d7d0 msgr2=0x7fa62c0acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.423+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 shutdown_connections 2024-09-17T03:53:41.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:41.423+0000 7fa633400640 1 -- 172.21.15.129:0/1143291090 wait complete. 2024-09-17T03:53:41.444 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-17T03:53:40.360291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":174,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258452,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716268,"statfs":{"total":773094113280,"available":772829458432,"internally_reserved":0,"allocated":9150464,"data_stored":4513060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.339125"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178823+0000","last_change":"2024-09-17T03:53:35.072458+0000","last_active":"2024-09-17T03:53:36.178823+0000","last_peered":"2024-09-17T03:53:36.178823+0000","last_clean":"2024-09-17T03:53:36.178823+0000","last_became_active":"2024-09-17T03:53:35.070660+0000","last_became_peered":"2024-09-17T03:53:35.070660+0000","last_unstale":"2024-09-17T03:53:36.178823+0000","last_undegraded":"2024-09-17T03:53:36.178823+0000","last_fullsized":"2024-09-17T03:53:36.178823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.505206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058029+0000","last_change":"2024-09-17T03:53:35.067959+0000","last_active":"2024-09-17T03:53:36.058029+0000","last_peered":"2024-09-17T03:53:36.058029+0000","last_clean":"2024-09-17T03:53:36.058029+0000","last_became_active":"2024-09-17T03:53:35.067646+0000","last_became_peered":"2024-09-17T03:53:35.067646+0000","last_unstale":"2024-09-17T03:53:36.058029+0000","last_undegraded":"2024-09-17T03:53:36.058029+0000","last_fullsized":"2024-09-17T03:53:36.058029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:55.989947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249115+0000","last_change":"2024-09-17T03:53:35.073278+0000","last_active":"2024-09-17T03:53:36.249115+0000","last_peered":"2024-09-17T03:53:36.249115+0000","last_clean":"2024-09-17T03:53:36.249115+0000","last_became_active":"2024-09-17T03:53:35.072943+0000","last_became_peered":"2024-09-17T03:53:35.072943+0000","last_unstale":"2024-09-17T03:53:36.249115+0000","last_undegraded":"2024-09-17T03:53:36.249115+0000","last_fullsized":"2024-09-17T03:53:36.249115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528171+0000","last_change":"2024-09-17T03:53:35.063690+0000","last_active":"2024-09-17T03:53:36.528171+0000","last_peered":"2024-09-17T03:53:36.528171+0000","last_clean":"2024-09-17T03:53:36.528171+0000","last_became_active":"2024-09-17T03:53:35.062877+0000","last_became_peered":"2024-09-17T03:53:35.062877+0000","last_unstale":"2024-09-17T03:53:36.528171+0000","last_undegraded":"2024-09-17T03:53:36.528171+0000","last_fullsized":"2024-09-17T03:53:36.528171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:42.690776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058612+0000","last_change":"2024-09-17T03:53:35.054924+0000","last_active":"2024-09-17T03:53:36.058612+0000","last_peered":"2024-09-17T03:53:36.058612+0000","last_clean":"2024-09-17T03:53:36.058612+0000","last_became_active":"2024-09-17T03:53:35.053852+0000","last_became_peered":"2024-09-17T03:53:35.053852+0000","last_unstale":"2024-09-17T03:53:36.058612+0000","last_undegraded":"2024-09-17T03:53:36.058612+0000","last_fullsized":"2024-09-17T03:53:36.058612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:49.257827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058725+0000","last_change":"2024-09-17T03:53:35.067059+0000","last_active":"2024-09-17T03:53:36.058725+0000","last_peered":"2024-09-17T03:53:36.058725+0000","last_clean":"2024-09-17T03:53:36.058725+0000","last_became_active":"2024-09-17T03:53:35.066734+0000","last_became_peered":"2024-09-17T03:53:35.066734+0000","last_unstale":"2024-09-17T03:53:36.058725+0000","last_undegraded":"2024-09-17T03:53:36.058725+0000","last_fullsized":"2024-09-17T03:53:36.058725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:54.297031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528126+0000","last_change":"2024-09-17T03:53:35.059267+0000","last_active":"2024-09-17T03:53:36.528126+0000","last_peered":"2024-09-17T03:53:36.528126+0000","last_clean":"2024-09-17T03:53:36.528126+0000","last_became_active":"2024-09-17T03:53:35.058966+0000","last_became_peered":"2024-09-17T03:53:35.058966+0000","last_unstale":"2024-09-17T03:53:36.528126+0000","last_undegraded":"2024-09-17T03:53:36.528126+0000","last_fullsized":"2024-09-17T03:53:36.528126+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:17:51.318108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058914+0000","last_change":"2024-09-17T03:53:35.051417+0000","last_active":"2024-09-17T03:53:36.058914+0000","last_peered":"2024-09-17T03:53:36.058914+0000","last_clean":"2024-09-17T03:53:36.058914+0000","last_became_active":"2024-09-17T03:53:35.051082+0000","last_became_peered":"2024-09-17T03:53:35.051082+0000","last_unstale":"2024-09-17T03:53:36.058914+0000","last_undegraded":"2024-09-17T03:53:36.058914+0000","last_fullsized":"2024-09-17T03:53:36.058914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:40.340287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058871+0000","last_change":"2024-09-17T03:53:35.076596+0000","last_active":"2024-09-17T03:53:36.058871+0000","last_peered":"2024-09-17T03:53:36.058871+0000","last_clean":"2024-09-17T03:53:36.058871+0000","last_became_active":"2024-09-17T03:53:35.076237+0000","last_became_peered":"2024-09-17T03:53:35.076237+0000","last_unstale":"2024-09-17T03:53:36.058871+0000","last_undegraded":"2024-09-17T03:53:36.058871+0000","last_fullsized":"2024-09-17T03:53:36.058871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:49.104278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178936+0000","last_change":"2024-09-17T03:53:35.075213+0000","last_active":"2024-09-17T03:53:36.178936+0000","last_peered":"2024-09-17T03:53:36.178936+0000","last_clean":"2024-09-17T03:53:36.178936+0000","last_became_active":"2024-09-17T03:53:35.074344+0000","last_became_peered":"2024-09-17T03:53:35.074344+0000","last_unstale":"2024-09-17T03:53:36.178936+0000","last_undegraded":"2024-09-17T03:53:36.178936+0000","last_fullsized":"2024-09-17T03:53:36.178936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:02.543171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057911+0000","last_change":"2024-09-17T03:53:35.060229+0000","last_active":"2024-09-17T03:53:36.057911+0000","last_peered":"2024-09-17T03:53:36.057911+0000","last_clean":"2024-09-17T03:53:36.057911+0000","last_became_active":"2024-09-17T03:53:35.059966+0000","last_became_peered":"2024-09-17T03:53:35.059966+0000","last_unstale":"2024-09-17T03:53:36.057911+0000","last_undegraded":"2024-09-17T03:53:36.057911+0000","last_fullsized":"2024-09-17T03:53:36.057911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:50.055699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178709+0000","last_change":"2024-09-17T03:53:35.073592+0000","last_active":"2024-09-17T03:53:36.178709+0000","last_peered":"2024-09-17T03:53:36.178709+0000","last_clean":"2024-09-17T03:53:36.178709+0000","last_became_active":"2024-09-17T03:53:35.073318+0000","last_became_peered":"2024-09-17T03:53:35.073318+0000","last_unstale":"2024-09-17T03:53:36.178709+0000","last_undegraded":"2024-09-17T03:53:36.178709+0000","last_fullsized":"2024-09-17T03:53:36.178709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:26.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249029+0000","last_change":"2024-09-17T03:53:35.060095+0000","last_active":"2024-09-17T03:53:36.249029+0000","last_peered":"2024-09-17T03:53:36.249029+0000","last_clean":"2024-09-17T03:53:36.249029+0000","last_became_active":"2024-09-17T03:53:35.059471+0000","last_became_peered":"2024-09-17T03:53:35.059471+0000","last_unstale":"2024-09-17T03:53:36.249029+0000","last_undegraded":"2024-09-17T03:53:36.249029+0000","last_fullsized":"2024-09-17T03:53:36.249029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528250+0000","last_change":"2024-09-17T03:53:35.080649+0000","last_active":"2024-09-17T03:53:36.528250+0000","last_peered":"2024-09-17T03:53:36.528250+0000","last_clean":"2024-09-17T03:53:36.528250+0000","last_became_active":"2024-09-17T03:53:35.080294+0000","last_became_peered":"2024-09-17T03:53:35.080294+0000","last_unstale":"2024-09-17T03:53:36.528250+0000","last_undegraded":"2024-09-17T03:53:36.528250+0000","last_fullsized":"2024-09-17T03:53:36.528250+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:33.740179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528771+0000","last_change":"2024-09-17T03:53:35.059152+0000","last_active":"2024-09-17T03:53:36.528771+0000","last_peered":"2024-09-17T03:53:36.528771+0000","last_clean":"2024-09-17T03:53:36.528771+0000","last_became_active":"2024-09-17T03:53:35.058509+0000","last_became_peered":"2024-09-17T03:53:35.058509+0000","last_unstale":"2024-09-17T03:53:36.528771+0000","last_undegraded":"2024-09-17T03:53:36.528771+0000","last_fullsized":"2024-09-17T03:53:36.528771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:47.079665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058072+0000","last_change":"2024-09-17T03:53:35.070836+0000","last_active":"2024-09-17T03:53:36.058072+0000","last_peered":"2024-09-17T03:53:36.058072+0000","last_clean":"2024-09-17T03:53:36.058072+0000","last_became_active":"2024-09-17T03:53:35.065693+0000","last_became_peered":"2024-09-17T03:53:35.065693+0000","last_unstale":"2024-09-17T03:53:36.058072+0000","last_undegraded":"2024-09-17T03:53:36.058072+0000","last_fullsized":"2024-09-17T03:53:36.058072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:50.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059504+0000","last_change":"2024-09-17T03:53:35.070366+0000","last_active":"2024-09-17T03:53:36.059504+0000","last_peered":"2024-09-17T03:53:36.059504+0000","last_clean":"2024-09-17T03:53:36.059504+0000","last_became_active":"2024-09-17T03:53:35.070062+0000","last_became_peered":"2024-09-17T03:53:35.070062+0000","last_unstale":"2024-09-17T03:53:36.059504+0000","last_undegraded":"2024-09-17T03:53:36.059504+0000","last_fullsized":"2024-09-17T03:53:36.059504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:45.603585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528047+0000","last_change":"2024-09-17T03:53:35.065367+0000","last_active":"2024-09-17T03:53:36.528047+0000","last_peered":"2024-09-17T03:53:36.528047+0000","last_clean":"2024-09-17T03:53:36.528047+0000","last_became_active":"2024-09-17T03:53:35.065042+0000","last_became_peered":"2024-09-17T03:53:35.065042+0000","last_unstale":"2024-09-17T03:53:36.528047+0000","last_undegraded":"2024-09-17T03:53:36.528047+0000","last_fullsized":"2024-09-17T03:53:36.528047+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:01.776287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529477+0000","last_change":"2024-09-17T03:53:35.077049+0000","last_active":"2024-09-17T03:53:36.529477+0000","last_peered":"2024-09-17T03:53:36.529477+0000","last_clean":"2024-09-17T03:53:36.529477+0000","last_became_active":"2024-09-17T03:53:35.076178+0000","last_became_peered":"2024-09-17T03:53:35.076178+0000","last_unstale":"2024-09-17T03:53:36.529477+0000","last_undegraded":"2024-09-17T03:53:36.529477+0000","last_fullsized":"2024-09-17T03:53:36.529477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528413+0000","last_change":"2024-09-17T03:53:35.070151+0000","last_active":"2024-09-17T03:53:36.528413+0000","last_peered":"2024-09-17T03:53:36.528413+0000","last_clean":"2024-09-17T03:53:36.528413+0000","last_became_active":"2024-09-17T03:53:35.069825+0000","last_became_peered":"2024-09-17T03:53:35.069825+0000","last_unstale":"2024-09-17T03:53:36.528413+0000","last_undegraded":"2024-09-17T03:53:36.528413+0000","last_fullsized":"2024-09-17T03:53:36.528413+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:41.443412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528656+0000","last_change":"2024-09-17T03:53:35.077741+0000","last_active":"2024-09-17T03:53:36.528656+0000","last_peered":"2024-09-17T03:53:36.528656+0000","last_clean":"2024-09-17T03:53:36.528656+0000","last_became_active":"2024-09-17T03:53:35.077437+0000","last_became_peered":"2024-09-17T03:53:35.077437+0000","last_unstale":"2024-09-17T03:53:36.528656+0000","last_undegraded":"2024-09-17T03:53:36.528656+0000","last_fullsized":"2024-09-17T03:53:36.528656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:51.309170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057255+0000","last_change":"2024-09-17T03:53:35.057094+0000","last_active":"2024-09-17T03:53:36.057255+0000","last_peered":"2024-09-17T03:53:36.057255+0000","last_clean":"2024-09-17T03:53:36.057255+0000","last_became_active":"2024-09-17T03:53:35.056803+0000","last_became_peered":"2024-09-17T03:53:35.056803+0000","last_unstale":"2024-09-17T03:53:36.057255+0000","last_undegraded":"2024-09-17T03:53:36.057255+0000","last_fullsized":"2024-09-17T03:53:36.057255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:43.810445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178654+0000","last_change":"2024-09-17T03:53:35.068757+0000","last_active":"2024-09-17T03:53:36.178654+0000","last_peered":"2024-09-17T03:53:36.178654+0000","last_clean":"2024-09-17T03:53:36.178654+0000","last_became_active":"2024-09-17T03:53:35.068235+0000","last_became_peered":"2024-09-17T03:53:35.068235+0000","last_unstale":"2024-09-17T03:53:36.178654+0000","last_undegraded":"2024-09-17T03:53:36.178654+0000","last_fullsized":"2024-09-17T03:53:36.178654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:16.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527840+0000","last_change":"2024-09-17T03:53:35.058419+0000","last_active":"2024-09-17T03:53:36.527840+0000","last_peered":"2024-09-17T03:53:36.527840+0000","last_clean":"2024-09-17T03:53:36.527840+0000","last_became_active":"2024-09-17T03:53:35.057357+0000","last_became_peered":"2024-09-17T03:53:35.057357+0000","last_unstale":"2024-09-17T03:53:36.527840+0000","last_undegraded":"2024-09-17T03:53:36.527840+0000","last_fullsized":"2024-09-17T03:53:36.527840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:46.020805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057970+0000","last_change":"2024-09-17T03:53:35.067946+0000","last_active":"2024-09-17T03:53:36.057970+0000","last_peered":"2024-09-17T03:53:36.057970+0000","last_clean":"2024-09-17T03:53:36.057970+0000","last_became_active":"2024-09-17T03:53:35.067630+0000","last_became_peered":"2024-09-17T03:53:35.067630+0000","last_unstale":"2024-09-17T03:53:36.057970+0000","last_undegraded":"2024-09-17T03:53:36.057970+0000","last_fullsized":"2024-09-17T03:53:36.057970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:12.178924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249169+0000","last_change":"2024-09-17T03:53:35.064329+0000","last_active":"2024-09-17T03:53:36.249169+0000","last_peered":"2024-09-17T03:53:36.249169+0000","last_clean":"2024-09-17T03:53:36.249169+0000","last_became_active":"2024-09-17T03:53:35.062761+0000","last_became_peered":"2024-09-17T03:53:35.062761+0000","last_unstale":"2024-09-17T03:53:36.249169+0000","last_undegraded":"2024-09-17T03:53:36.249169+0000","last_fullsized":"2024-09-17T03:53:36.249169+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059290+0000","last_change":"2024-09-17T03:53:35.057286+0000","last_active":"2024-09-17T03:53:36.059290+0000","last_peered":"2024-09-17T03:53:36.059290+0000","last_clean":"2024-09-17T03:53:36.059290+0000","last_became_active":"2024-09-17T03:53:35.056910+0000","last_became_peered":"2024-09-17T03:53:35.056910+0000","last_unstale":"2024-09-17T03:53:36.059290+0000","last_undegraded":"2024-09-17T03:53:36.059290+0000","last_fullsized":"2024-09-17T03:53:36.059290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:48.490076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178706+0000","last_change":"2024-09-17T03:53:35.067698+0000","last_active":"2024-09-17T03:53:36.178706+0000","last_peered":"2024-09-17T03:53:36.178706+0000","last_clean":"2024-09-17T03:53:36.178706+0000","last_became_active":"2024-09-17T03:53:35.066584+0000","last_became_peered":"2024-09-17T03:53:35.066584+0000","last_unstale":"2024-09-17T03:53:36.178706+0000","last_undegraded":"2024-09-17T03:53:36.178706+0000","last_fullsized":"2024-09-17T03:53:36.178706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:57.060788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057393+0000","last_change":"2024-09-17T03:53:35.065693+0000","last_active":"2024-09-17T03:53:36.057393+0000","last_peered":"2024-09-17T03:53:36.057393+0000","last_clean":"2024-09-17T03:53:36.057393+0000","last_became_active":"2024-09-17T03:53:35.065403+0000","last_became_peered":"2024-09-17T03:53:35.065403+0000","last_unstale":"2024-09-17T03:53:36.057393+0000","last_undegraded":"2024-09-17T03:53:36.057393+0000","last_fullsized":"2024-09-17T03:53:36.057393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:50.272364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057291+0000","last_change":"2024-09-17T03:53:35.075375+0000","last_active":"2024-09-17T03:53:36.057291+0000","last_peered":"2024-09-17T03:53:36.057291+0000","last_clean":"2024-09-17T03:53:36.057291+0000","last_became_active":"2024-09-17T03:53:35.074718+0000","last_became_peered":"2024-09-17T03:53:35.074718+0000","last_unstale":"2024-09-17T03:53:36.057291+0000","last_undegraded":"2024-09-17T03:53:36.057291+0000","last_fullsized":"2024-09-17T03:53:36.057291+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:39.360722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058059+0000","last_change":"2024-09-17T03:53:35.069885+0000","last_active":"2024-09-17T03:53:36.058059+0000","last_peered":"2024-09-17T03:53:36.058059+0000","last_clean":"2024-09-17T03:53:36.058059+0000","last_became_active":"2024-09-17T03:53:35.069558+0000","last_became_peered":"2024-09-17T03:53:35.069558+0000","last_unstale":"2024-09-17T03:53:36.058059+0000","last_undegraded":"2024-09-17T03:53:36.058059+0000","last_fullsized":"2024-09-17T03:53:36.058059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:25.141426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178515+0000","last_change":"2024-09-17T03:53:35.062010+0000","last_active":"2024-09-17T03:53:36.178515+0000","last_peered":"2024-09-17T03:53:36.178515+0000","last_clean":"2024-09-17T03:53:36.178515+0000","last_became_active":"2024-09-17T03:53:35.059874+0000","last_became_peered":"2024-09-17T03:53:35.059874+0000","last_unstale":"2024-09-17T03:53:36.178515+0000","last_undegraded":"2024-09-17T03:53:36.178515+0000","last_fullsized":"2024-09-17T03:53:36.178515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:26.933349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060228+0000","last_change":"2024-09-17T03:53:35.065637+0000","last_active":"2024-09-17T03:53:36.060228+0000","last_peered":"2024-09-17T03:53:36.060228+0000","last_clean":"2024-09-17T03:53:36.060228+0000","last_became_active":"2024-09-17T03:53:35.065252+0000","last_became_peered":"2024-09-17T03:53:35.065252+0000","last_unstale":"2024-09-17T03:53:36.060228+0000","last_undegraded":"2024-09-17T03:53:36.060228+0000","last_fullsized":"2024-09-17T03:53:36.060228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178966+0000","last_change":"2024-09-17T03:53:34.042427+0000","last_active":"2024-09-17T03:53:36.178966+0000","last_peered":"2024-09-17T03:53:36.178966+0000","last_clean":"2024-09-17T03:53:36.178966+0000","last_became_active":"2024-09-17T03:53:34.042117+0000","last_became_peered":"2024-09-17T03:53:34.042117+0000","last_unstale":"2024-09-17T03:53:36.178966+0000","last_undegraded":"2024-09-17T03:53:36.178966+0000","last_fullsized":"2024-09-17T03:53:36.178966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:57.275556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178860+0000","last_change":"2024-09-17T03:53:35.075209+0000","last_active":"2024-09-17T03:53:36.178860+0000","last_peered":"2024-09-17T03:53:36.178860+0000","last_clean":"2024-09-17T03:53:36.178860+0000","last_became_active":"2024-09-17T03:53:35.074557+0000","last_became_peered":"2024-09-17T03:53:35.074557+0000","last_unstale":"2024-09-17T03:53:36.178860+0000","last_undegraded":"2024-09-17T03:53:36.178860+0000","last_fullsized":"2024-09-17T03:53:36.178860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:26.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060393+0000","last_change":"2024-09-17T03:53:34.047657+0000","last_active":"2024-09-17T03:53:36.060393+0000","last_peered":"2024-09-17T03:53:36.060393+0000","last_clean":"2024-09-17T03:53:36.060393+0000","last_became_active":"2024-09-17T03:53:34.047290+0000","last_became_peered":"2024-09-17T03:53:34.047290+0000","last_unstale":"2024-09-17T03:53:36.060393+0000","last_undegraded":"2024-09-17T03:53:36.060393+0000","last_fullsized":"2024-09-17T03:53:36.060393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:25.205958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178967+0000","last_change":"2024-09-17T03:53:35.068246+0000","last_active":"2024-09-17T03:53:36.178967+0000","last_peered":"2024-09-17T03:53:36.178967+0000","last_clean":"2024-09-17T03:53:36.178967+0000","last_became_active":"2024-09-17T03:53:35.066866+0000","last_became_peered":"2024-09-17T03:53:35.066866+0000","last_unstale":"2024-09-17T03:53:36.178967+0000","last_undegraded":"2024-09-17T03:53:36.178967+0000","last_fullsized":"2024-09-17T03:53:36.178967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:59.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529312+0000","last_change":"2024-09-17T03:53:34.047098+0000","last_active":"2024-09-17T03:53:36.529312+0000","last_peered":"2024-09-17T03:53:36.529312+0000","last_clean":"2024-09-17T03:53:36.529312+0000","last_became_active":"2024-09-17T03:53:34.046816+0000","last_became_peered":"2024-09-17T03:53:34.046816+0000","last_unstale":"2024-09-17T03:53:36.529312+0000","last_undegraded":"2024-09-17T03:53:36.529312+0000","last_fullsized":"2024-09-17T03:53:36.529312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:25.856611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249238+0000","last_change":"2024-09-17T03:53:35.067274+0000","last_active":"2024-09-17T03:53:36.249238+0000","last_peered":"2024-09-17T03:53:36.249238+0000","last_clean":"2024-09-17T03:53:36.249238+0000","last_became_active":"2024-09-17T03:53:35.066895+0000","last_became_peered":"2024-09-17T03:53:35.066895+0000","last_unstale":"2024-09-17T03:53:36.249238+0000","last_undegraded":"2024-09-17T03:53:36.249238+0000","last_fullsized":"2024-09-17T03:53:36.249238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529204+0000","last_change":"2024-09-17T03:53:34.047320+0000","last_active":"2024-09-17T03:53:36.529204+0000","last_peered":"2024-09-17T03:53:36.529204+0000","last_clean":"2024-09-17T03:53:36.529204+0000","last_became_active":"2024-09-17T03:53:34.045973+0000","last_became_peered":"2024-09-17T03:53:34.045973+0000","last_unstale":"2024-09-17T03:53:36.529204+0000","last_undegraded":"2024-09-17T03:53:36.529204+0000","last_fullsized":"2024-09-17T03:53:36.529204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:03.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057495+0000","last_change":"2024-09-17T03:53:35.075107+0000","last_active":"2024-09-17T03:53:36.057495+0000","last_peered":"2024-09-17T03:53:36.057495+0000","last_clean":"2024-09-17T03:53:36.057495+0000","last_became_active":"2024-09-17T03:53:35.069999+0000","last_became_peered":"2024-09-17T03:53:35.069999+0000","last_unstale":"2024-09-17T03:53:36.057495+0000","last_undegraded":"2024-09-17T03:53:36.057495+0000","last_fullsized":"2024-09-17T03:53:36.057495+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:15.911260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059049+0000","last_change":"2024-09-17T03:53:34.046521+0000","last_active":"2024-09-17T03:53:36.059049+0000","last_peered":"2024-09-17T03:53:36.059049+0000","last_clean":"2024-09-17T03:53:36.059049+0000","last_became_active":"2024-09-17T03:53:34.046151+0000","last_became_peered":"2024-09-17T03:53:34.046151+0000","last_unstale":"2024-09-17T03:53:36.059049+0000","last_undegraded":"2024-09-17T03:53:36.059049+0000","last_fullsized":"2024-09-17T03:53:36.059049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:04.714385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058821+0000","last_change":"2024-09-17T03:53:35.069264+0000","last_active":"2024-09-17T03:53:36.058821+0000","last_peered":"2024-09-17T03:53:36.058821+0000","last_clean":"2024-09-17T03:53:36.058821+0000","last_became_active":"2024-09-17T03:53:35.065458+0000","last_became_peered":"2024-09-17T03:53:35.065458+0000","last_unstale":"2024-09-17T03:53:36.058821+0000","last_undegraded":"2024-09-17T03:53:36.058821+0000","last_fullsized":"2024-09-17T03:53:36.058821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:16.559380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058298+0000","last_change":"2024-09-17T03:53:34.045410+0000","last_active":"2024-09-17T03:53:36.058298+0000","last_peered":"2024-09-17T03:53:36.058298+0000","last_clean":"2024-09-17T03:53:36.058298+0000","last_became_active":"2024-09-17T03:53:34.045030+0000","last_became_peered":"2024-09-17T03:53:34.045030+0000","last_unstale":"2024-09-17T03:53:36.058298+0000","last_undegraded":"2024-09-17T03:53:36.058298+0000","last_fullsized":"2024-09-17T03:53:36.058298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248910+0000","last_change":"2024-09-17T03:53:35.075625+0000","last_active":"2024-09-17T03:53:36.248910+0000","last_peered":"2024-09-17T03:53:36.248910+0000","last_clean":"2024-09-17T03:53:36.248910+0000","last_became_active":"2024-09-17T03:53:35.075285+0000","last_became_peered":"2024-09-17T03:53:35.075285+0000","last_unstale":"2024-09-17T03:53:36.248910+0000","last_undegraded":"2024-09-17T03:53:36.248910+0000","last_fullsized":"2024-09-17T03:53:36.248910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528363+0000","last_change":"2024-09-17T03:53:34.040513+0000","last_active":"2024-09-17T03:53:36.528363+0000","last_peered":"2024-09-17T03:53:36.528363+0000","last_clean":"2024-09-17T03:53:36.528363+0000","last_became_active":"2024-09-17T03:53:34.039387+0000","last_became_peered":"2024-09-17T03:53:34.039387+0000","last_unstale":"2024-09-17T03:53:36.528363+0000","last_undegraded":"2024-09-17T03:53:36.528363+0000","last_fullsized":"2024-09-17T03:53:36.528363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:35.514048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528897+0000","last_change":"2024-09-17T03:53:35.058824+0000","last_active":"2024-09-17T03:53:36.528897+0000","last_peered":"2024-09-17T03:53:36.528897+0000","last_clean":"2024-09-17T03:53:36.528897+0000","last_became_active":"2024-09-17T03:53:35.058053+0000","last_became_peered":"2024-09-17T03:53:35.058053+0000","last_unstale":"2024-09-17T03:53:36.528897+0000","last_undegraded":"2024-09-17T03:53:36.528897+0000","last_fullsized":"2024-09-17T03:53:36.528897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:39.813037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057165+0000","last_change":"2024-09-17T03:53:34.042323+0000","last_active":"2024-09-17T03:53:36.057165+0000","last_peered":"2024-09-17T03:53:36.057165+0000","last_clean":"2024-09-17T03:53:36.057165+0000","last_became_active":"2024-09-17T03:53:34.041995+0000","last_became_peered":"2024-09-17T03:53:34.041995+0000","last_unstale":"2024-09-17T03:53:36.057165+0000","last_undegraded":"2024-09-17T03:53:36.057165+0000","last_fullsized":"2024-09-17T03:53:36.057165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:54.594238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.530067+0000","last_change":"2024-09-17T03:53:35.066310+0000","last_active":"2024-09-17T03:53:36.530067+0000","last_peered":"2024-09-17T03:53:36.530067+0000","last_clean":"2024-09-17T03:53:36.530067+0000","last_became_active":"2024-09-17T03:53:35.065211+0000","last_became_peered":"2024-09-17T03:53:35.065211+0000","last_unstale":"2024-09-17T03:53:36.530067+0000","last_undegraded":"2024-09-17T03:53:36.530067+0000","last_fullsized":"2024-09-17T03:53:36.530067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:11.925576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179093+0000","last_change":"2024-09-17T03:53:34.041277+0000","last_active":"2024-09-17T03:53:36.179093+0000","last_peered":"2024-09-17T03:53:36.179093+0000","last_clean":"2024-09-17T03:53:36.179093+0000","last_became_active":"2024-09-17T03:53:34.040555+0000","last_became_peered":"2024-09-17T03:53:34.040555+0000","last_unstale":"2024-09-17T03:53:36.179093+0000","last_undegraded":"2024-09-17T03:53:36.179093+0000","last_fullsized":"2024-09-17T03:53:36.179093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:05.315114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059958+0000","last_change":"2024-09-17T03:53:35.058829+0000","last_active":"2024-09-17T03:53:36.059958+0000","last_peered":"2024-09-17T03:53:36.059958+0000","last_clean":"2024-09-17T03:53:36.059958+0000","last_became_active":"2024-09-17T03:53:35.058494+0000","last_became_peered":"2024-09-17T03:53:35.058494+0000","last_unstale":"2024-09-17T03:53:36.059958+0000","last_undegraded":"2024-09-17T03:53:36.059958+0000","last_fullsized":"2024-09-17T03:53:36.059958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:09.497624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178778+0000","last_change":"2024-09-17T03:53:34.047020+0000","last_active":"2024-09-17T03:53:36.178778+0000","last_peered":"2024-09-17T03:53:36.178778+0000","last_clean":"2024-09-17T03:53:36.178778+0000","last_became_active":"2024-09-17T03:53:34.045487+0000","last_became_peered":"2024-09-17T03:53:34.045487+0000","last_unstale":"2024-09-17T03:53:36.178778+0000","last_undegraded":"2024-09-17T03:53:36.178778+0000","last_fullsized":"2024-09-17T03:53:36.178778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:03.205170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529300+0000","last_change":"2024-09-17T03:53:35.074637+0000","last_active":"2024-09-17T03:53:36.529300+0000","last_peered":"2024-09-17T03:53:36.529300+0000","last_clean":"2024-09-17T03:53:36.529300+0000","last_became_active":"2024-09-17T03:53:35.074086+0000","last_became_peered":"2024-09-17T03:53:35.074086+0000","last_unstale":"2024-09-17T03:53:36.529300+0000","last_undegraded":"2024-09-17T03:53:36.529300+0000","last_fullsized":"2024-09-17T03:53:36.529300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.027879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178381+0000","last_change":"2024-09-17T03:53:34.045567+0000","last_active":"2024-09-17T03:53:36.178381+0000","last_peered":"2024-09-17T03:53:36.178381+0000","last_clean":"2024-09-17T03:53:36.178381+0000","last_became_active":"2024-09-17T03:53:34.045029+0000","last_became_peered":"2024-09-17T03:53:34.045029+0000","last_unstale":"2024-09-17T03:53:36.178381+0000","last_undegraded":"2024-09-17T03:53:36.178381+0000","last_fullsized":"2024-09-17T03:53:36.178381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:38.299845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059436+0000","last_change":"2024-09-17T03:53:35.078726+0000","last_active":"2024-09-17T03:53:36.059436+0000","last_peered":"2024-09-17T03:53:36.059436+0000","last_clean":"2024-09-17T03:53:36.059436+0000","last_became_active":"2024-09-17T03:53:35.078369+0000","last_became_peered":"2024-09-17T03:53:35.078369+0000","last_unstale":"2024-09-17T03:53:36.059436+0000","last_undegraded":"2024-09-17T03:53:36.059436+0000","last_fullsized":"2024-09-17T03:53:36.059436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059227+0000","last_change":"2024-09-17T03:53:34.033143+0000","last_active":"2024-09-17T03:53:36.059227+0000","last_peered":"2024-09-17T03:53:36.059227+0000","last_clean":"2024-09-17T03:53:36.059227+0000","last_became_active":"2024-09-17T03:53:34.031770+0000","last_became_peered":"2024-09-17T03:53:34.031770+0000","last_unstale":"2024-09-17T03:53:36.059227+0000","last_undegraded":"2024-09-17T03:53:36.059227+0000","last_fullsized":"2024-09-17T03:53:36.059227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529438+0000","last_change":"2024-09-17T03:53:35.062155+0000","last_active":"2024-09-17T03:53:36.529438+0000","last_peered":"2024-09-17T03:53:36.529438+0000","last_clean":"2024-09-17T03:53:36.529438+0000","last_became_active":"2024-09-17T03:53:35.061536+0000","last_became_peered":"2024-09-17T03:53:35.061536+0000","last_unstale":"2024-09-17T03:53:36.529438+0000","last_undegraded":"2024-09-17T03:53:36.529438+0000","last_fullsized":"2024-09-17T03:53:36.529438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:36.274547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057583+0000","last_change":"2024-09-17T03:53:34.043064+0000","last_active":"2024-09-17T03:53:36.057583+0000","last_peered":"2024-09-17T03:53:36.057583+0000","last_clean":"2024-09-17T03:53:36.057583+0000","last_became_active":"2024-09-17T03:53:34.038696+0000","last_became_peered":"2024-09-17T03:53:34.038696+0000","last_unstale":"2024-09-17T03:53:36.057583+0000","last_undegraded":"2024-09-17T03:53:36.057583+0000","last_fullsized":"2024-09-17T03:53:36.057583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:29.561790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249364+0000","last_change":"2024-09-17T03:53:35.063772+0000","last_active":"2024-09-17T03:53:36.249364+0000","last_peered":"2024-09-17T03:53:36.249364+0000","last_clean":"2024-09-17T03:53:36.249364+0000","last_became_active":"2024-09-17T03:53:35.063400+0000","last_became_peered":"2024-09-17T03:53:35.063400+0000","last_unstale":"2024-09-17T03:53:36.249364+0000","last_undegraded":"2024-09-17T03:53:36.249364+0000","last_fullsized":"2024-09-17T03:53:36.249364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:14.277688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529588+0000","last_change":"2024-09-17T03:53:34.048029+0000","last_active":"2024-09-17T03:53:36.529588+0000","last_peered":"2024-09-17T03:53:36.529588+0000","last_clean":"2024-09-17T03:53:36.529588+0000","last_became_active":"2024-09-17T03:53:34.046461+0000","last_became_peered":"2024-09-17T03:53:34.046461+0000","last_unstale":"2024-09-17T03:53:36.529588+0000","last_undegraded":"2024-09-17T03:53:36.529588+0000","last_fullsized":"2024-09-17T03:53:36.529588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:07.500302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057588+0000","last_change":"2024-09-17T03:53:35.062083+0000","last_active":"2024-09-17T03:53:36.057588+0000","last_peered":"2024-09-17T03:53:36.057588+0000","last_clean":"2024-09-17T03:53:36.057588+0000","last_became_active":"2024-09-17T03:53:35.061800+0000","last_became_peered":"2024-09-17T03:53:35.061800+0000","last_unstale":"2024-09-17T03:53:36.057588+0000","last_undegraded":"2024-09-17T03:53:36.057588+0000","last_fullsized":"2024-09-17T03:53:36.057588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:49.841803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059500+0000","last_change":"2024-09-17T03:53:34.045446+0000","last_active":"2024-09-17T03:53:36.059500+0000","last_peered":"2024-09-17T03:53:36.059500+0000","last_clean":"2024-09-17T03:53:36.059500+0000","last_became_active":"2024-09-17T03:53:34.045103+0000","last_became_peered":"2024-09-17T03:53:34.045103+0000","last_unstale":"2024-09-17T03:53:36.059500+0000","last_undegraded":"2024-09-17T03:53:36.059500+0000","last_fullsized":"2024-09-17T03:53:36.059500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057116+0000","last_change":"2024-09-17T03:53:35.068169+0000","last_active":"2024-09-17T03:53:36.057116+0000","last_peered":"2024-09-17T03:53:36.057116+0000","last_clean":"2024-09-17T03:53:36.057116+0000","last_became_active":"2024-09-17T03:53:35.067861+0000","last_became_peered":"2024-09-17T03:53:35.067861+0000","last_unstale":"2024-09-17T03:53:36.057116+0000","last_undegraded":"2024-09-17T03:53:36.057116+0000","last_fullsized":"2024-09-17T03:53:36.057116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:58.932353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528873+0000","last_change":"2024-09-17T03:53:34.043445+0000","last_active":"2024-09-17T03:53:36.528873+0000","last_peered":"2024-09-17T03:53:36.528873+0000","last_clean":"2024-09-17T03:53:36.528873+0000","last_became_active":"2024-09-17T03:53:34.042920+0000","last_became_peered":"2024-09-17T03:53:34.042920+0000","last_unstale":"2024-09-17T03:53:36.528873+0000","last_undegraded":"2024-09-17T03:53:36.528873+0000","last_fullsized":"2024-09-17T03:53:36.528873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:42.885526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249340+0000","last_change":"2024-09-17T03:53:35.061631+0000","last_active":"2024-09-17T03:53:36.249340+0000","last_peered":"2024-09-17T03:53:36.249340+0000","last_clean":"2024-09-17T03:53:36.249340+0000","last_became_active":"2024-09-17T03:53:35.061273+0000","last_became_peered":"2024-09-17T03:53:35.061273+0000","last_unstale":"2024-09-17T03:53:36.249340+0000","last_undegraded":"2024-09-17T03:53:36.249340+0000","last_fullsized":"2024-09-17T03:53:36.249340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:08.506998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179205+0000","last_change":"2024-09-17T03:53:34.041969+0000","last_active":"2024-09-17T03:53:36.179205+0000","last_peered":"2024-09-17T03:53:36.179205+0000","last_clean":"2024-09-17T03:53:36.179205+0000","last_became_active":"2024-09-17T03:53:34.040090+0000","last_became_peered":"2024-09-17T03:53:34.040090+0000","last_unstale":"2024-09-17T03:53:36.179205+0000","last_undegraded":"2024-09-17T03:53:36.179205+0000","last_fullsized":"2024-09-17T03:53:36.179205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:12.790859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179104+0000","last_change":"2024-09-17T03:53:35.071124+0000","last_active":"2024-09-17T03:53:36.179104+0000","last_peered":"2024-09-17T03:53:36.179104+0000","last_clean":"2024-09-17T03:53:36.179104+0000","last_became_active":"2024-09-17T03:53:35.070636+0000","last_became_peered":"2024-09-17T03:53:35.070636+0000","last_unstale":"2024-09-17T03:53:36.179104+0000","last_undegraded":"2024-09-17T03:53:36.179104+0000","last_fullsized":"2024-09-17T03:53:36.179104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:18.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529575+0000","last_change":"2024-09-17T03:53:34.049033+0000","last_active":"2024-09-17T03:53:36.529575+0000","last_peered":"2024-09-17T03:53:36.529575+0000","last_clean":"2024-09-17T03:53:36.529575+0000","last_became_active":"2024-09-17T03:53:34.048596+0000","last_became_peered":"2024-09-17T03:53:34.048596+0000","last_unstale":"2024-09-17T03:53:36.529575+0000","last_undegraded":"2024-09-17T03:53:36.529575+0000","last_fullsized":"2024-09-17T03:53:36.529575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:25.851358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059564+0000","last_change":"2024-09-17T03:53:35.052022+0000","last_active":"2024-09-17T03:53:36.059564+0000","last_peered":"2024-09-17T03:53:36.059564+0000","last_clean":"2024-09-17T03:53:36.059564+0000","last_became_active":"2024-09-17T03:53:35.051704+0000","last_became_peered":"2024-09-17T03:53:35.051704+0000","last_unstale":"2024-09-17T03:53:36.059564+0000","last_undegraded":"2024-09-17T03:53:36.059564+0000","last_fullsized":"2024-09-17T03:53:36.059564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529714+0000","last_change":"2024-09-17T03:53:34.040065+0000","last_active":"2024-09-17T03:53:36.529714+0000","last_peered":"2024-09-17T03:53:36.529714+0000","last_clean":"2024-09-17T03:53:36.529714+0000","last_became_active":"2024-09-17T03:53:34.039675+0000","last_became_peered":"2024-09-17T03:53:34.039675+0000","last_unstale":"2024-09-17T03:53:36.529714+0000","last_undegraded":"2024-09-17T03:53:36.529714+0000","last_fullsized":"2024-09-17T03:53:36.529714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:40.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528473+0000","last_change":"2024-09-17T03:53:35.068226+0000","last_active":"2024-09-17T03:53:36.528473+0000","last_peered":"2024-09-17T03:53:36.528473+0000","last_clean":"2024-09-17T03:53:36.528473+0000","last_became_active":"2024-09-17T03:53:35.067943+0000","last_became_peered":"2024-09-17T03:53:35.067943+0000","last_unstale":"2024-09-17T03:53:36.528473+0000","last_undegraded":"2024-09-17T03:53:36.528473+0000","last_fullsized":"2024-09-17T03:53:36.528473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:09.141910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529817+0000","last_change":"2024-09-17T03:53:34.042295+0000","last_active":"2024-09-17T03:53:36.529817+0000","last_peered":"2024-09-17T03:53:36.529817+0000","last_clean":"2024-09-17T03:53:36.529817+0000","last_became_active":"2024-09-17T03:53:34.041992+0000","last_became_peered":"2024-09-17T03:53:34.041992+0000","last_unstale":"2024-09-17T03:53:36.529817+0000","last_undegraded":"2024-09-17T03:53:36.529817+0000","last_fullsized":"2024-09-17T03:53:36.529817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:16.446587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058700+0000","last_change":"2024-09-17T03:53:35.063084+0000","last_active":"2024-09-17T03:53:36.058700+0000","last_peered":"2024-09-17T03:53:36.058700+0000","last_clean":"2024-09-17T03:53:36.058700+0000","last_became_active":"2024-09-17T03:53:35.062759+0000","last_became_peered":"2024-09-17T03:53:35.062759+0000","last_unstale":"2024-09-17T03:53:36.058700+0000","last_undegraded":"2024-09-17T03:53:36.058700+0000","last_fullsized":"2024-09-17T03:53:36.058700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:22.612393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528628+0000","last_change":"2024-09-17T03:53:34.050809+0000","last_active":"2024-09-17T03:53:36.528628+0000","last_peered":"2024-09-17T03:53:36.528628+0000","last_clean":"2024-09-17T03:53:36.528628+0000","last_became_active":"2024-09-17T03:53:34.050498+0000","last_became_peered":"2024-09-17T03:53:34.050498+0000","last_unstale":"2024-09-17T03:53:36.528628+0000","last_undegraded":"2024-09-17T03:53:36.528628+0000","last_fullsized":"2024-09-17T03:53:36.528628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:01.191543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527911+0000","last_change":"2024-09-17T03:53:35.059466+0000","last_active":"2024-09-17T03:53:36.527911+0000","last_peered":"2024-09-17T03:53:36.527911+0000","last_clean":"2024-09-17T03:53:36.527911+0000","last_became_active":"2024-09-17T03:53:35.059160+0000","last_became_peered":"2024-09-17T03:53:35.059160+0000","last_unstale":"2024-09-17T03:53:36.527911+0000","last_undegraded":"2024-09-17T03:53:36.527911+0000","last_fullsized":"2024-09-17T03:53:36.527911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:15.257783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527972+0000","last_change":"2024-09-17T03:53:34.046054+0000","last_active":"2024-09-17T03:53:36.527972+0000","last_peered":"2024-09-17T03:53:36.527972+0000","last_clean":"2024-09-17T03:53:36.527972+0000","last_became_active":"2024-09-17T03:53:34.045718+0000","last_became_peered":"2024-09-17T03:53:34.045718+0000","last_unstale":"2024-09-17T03:53:36.527972+0000","last_undegraded":"2024-09-17T03:53:36.527972+0000","last_fullsized":"2024-09-17T03:53:36.527972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:09.572328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059323+0000","last_change":"2024-09-17T03:53:35.068451+0000","last_active":"2024-09-17T03:53:36.059323+0000","last_peered":"2024-09-17T03:53:36.059323+0000","last_clean":"2024-09-17T03:53:36.059323+0000","last_became_active":"2024-09-17T03:53:35.067500+0000","last_became_peered":"2024-09-17T03:53:35.067500+0000","last_unstale":"2024-09-17T03:53:36.059323+0000","last_undegraded":"2024-09-17T03:53:36.059323+0000","last_fullsized":"2024-09-17T03:53:36.059323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:10.103551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528173+0000","last_change":"2024-09-17T03:53:34.045835+0000","last_active":"2024-09-17T03:53:36.528173+0000","last_peered":"2024-09-17T03:53:36.528173+0000","last_clean":"2024-09-17T03:53:36.528173+0000","last_became_active":"2024-09-17T03:53:34.045329+0000","last_became_peered":"2024-09-17T03:53:34.045329+0000","last_unstale":"2024-09-17T03:53:36.528173+0000","last_undegraded":"2024-09-17T03:53:36.528173+0000","last_fullsized":"2024-09-17T03:53:36.528173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:07.278692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060312+0000","last_change":"2024-09-17T03:53:35.045510+0000","last_active":"2024-09-17T03:53:36.060312+0000","last_peered":"2024-09-17T03:53:36.060312+0000","last_clean":"2024-09-17T03:53:36.060312+0000","last_became_active":"2024-09-17T03:53:35.044996+0000","last_became_peered":"2024-09-17T03:53:35.044996+0000","last_unstale":"2024-09-17T03:53:36.060312+0000","last_undegraded":"2024-09-17T03:53:36.060312+0000","last_fullsized":"2024-09-17T03:53:36.060312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058140+0000","last_change":"2024-09-17T03:53:34.043457+0000","last_active":"2024-09-17T03:53:36.058140+0000","last_peered":"2024-09-17T03:53:36.058140+0000","last_clean":"2024-09-17T03:53:36.058140+0000","last_became_active":"2024-09-17T03:53:34.043142+0000","last_became_peered":"2024-09-17T03:53:34.043142+0000","last_unstale":"2024-09-17T03:53:36.058140+0000","last_undegraded":"2024-09-17T03:53:36.058140+0000","last_fullsized":"2024-09-17T03:53:36.058140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:14.453153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249615+0000","last_change":"2024-09-17T03:53:35.077390+0000","last_active":"2024-09-17T03:53:36.249615+0000","last_peered":"2024-09-17T03:53:36.249615+0000","last_clean":"2024-09-17T03:53:36.249615+0000","last_became_active":"2024-09-17T03:53:35.077039+0000","last_became_peered":"2024-09-17T03:53:35.077039+0000","last_unstale":"2024-09-17T03:53:36.249615+0000","last_undegraded":"2024-09-17T03:53:36.249615+0000","last_fullsized":"2024-09-17T03:53:36.249615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:34.715128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058032+0000","last_change":"2024-09-17T03:53:34.047344+0000","last_active":"2024-09-17T03:53:36.058032+0000","last_peered":"2024-09-17T03:53:36.058032+0000","last_clean":"2024-09-17T03:53:36.058032+0000","last_became_active":"2024-09-17T03:53:34.041909+0000","last_became_peered":"2024-09-17T03:53:34.041909+0000","last_unstale":"2024-09-17T03:53:36.058032+0000","last_undegraded":"2024-09-17T03:53:36.058032+0000","last_fullsized":"2024-09-17T03:53:36.058032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:45.691778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179022+0000","last_change":"2024-09-17T03:53:35.060992+0000","last_active":"2024-09-17T03:53:36.179022+0000","last_peered":"2024-09-17T03:53:36.179022+0000","last_clean":"2024-09-17T03:53:36.179022+0000","last_became_active":"2024-09-17T03:53:35.060339+0000","last_became_peered":"2024-09-17T03:53:35.060339+0000","last_unstale":"2024-09-17T03:53:36.179022+0000","last_undegraded":"2024-09-17T03:53:36.179022+0000","last_fullsized":"2024-09-17T03:53:36.179022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:48.433949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178476+0000","last_change":"2024-09-17T03:53:34.042515+0000","last_active":"2024-09-17T03:53:36.178476+0000","last_peered":"2024-09-17T03:53:36.178476+0000","last_clean":"2024-09-17T03:53:36.178476+0000","last_became_active":"2024-09-17T03:53:34.041350+0000","last_became_peered":"2024-09-17T03:53:34.041350+0000","last_unstale":"2024-09-17T03:53:36.178476+0000","last_undegraded":"2024-09-17T03:53:36.178476+0000","last_fullsized":"2024-09-17T03:53:36.178476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:54.121078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059705+0000","last_change":"2024-09-17T03:53:35.078734+0000","last_active":"2024-09-17T03:53:36.059705+0000","last_peered":"2024-09-17T03:53:36.059705+0000","last_clean":"2024-09-17T03:53:36.059705+0000","last_became_active":"2024-09-17T03:53:35.078432+0000","last_became_peered":"2024-09-17T03:53:35.078432+0000","last_unstale":"2024-09-17T03:53:36.059705+0000","last_undegraded":"2024-09-17T03:53:36.059705+0000","last_fullsized":"2024-09-17T03:53:36.059705+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179021+0000","last_change":"2024-09-17T03:53:34.043860+0000","last_active":"2024-09-17T03:53:36.179021+0000","last_peered":"2024-09-17T03:53:36.179021+0000","last_clean":"2024-09-17T03:53:36.179021+0000","last_became_active":"2024-09-17T03:53:34.043502+0000","last_became_peered":"2024-09-17T03:53:34.043502+0000","last_unstale":"2024-09-17T03:53:36.179021+0000","last_undegraded":"2024-09-17T03:53:36.179021+0000","last_fullsized":"2024-09-17T03:53:36.179021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:02.362312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058992+0000","last_change":"2024-09-17T03:53:35.064082+0000","last_active":"2024-09-17T03:53:36.058992+0000","last_peered":"2024-09-17T03:53:36.058992+0000","last_clean":"2024-09-17T03:53:36.058992+0000","last_became_active":"2024-09-17T03:53:35.063675+0000","last_became_peered":"2024-09-17T03:53:35.063675+0000","last_unstale":"2024-09-17T03:53:36.058992+0000","last_undegraded":"2024-09-17T03:53:36.058992+0000","last_fullsized":"2024-09-17T03:53:36.058992+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:11.219352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529388+0000","last_change":"2024-09-17T03:53:34.037735+0000","last_active":"2024-09-17T03:53:36.529388+0000","last_peered":"2024-09-17T03:53:36.529388+0000","last_clean":"2024-09-17T03:53:36.529388+0000","last_became_active":"2024-09-17T03:53:34.037448+0000","last_became_peered":"2024-09-17T03:53:34.037448+0000","last_unstale":"2024-09-17T03:53:36.529388+0000","last_undegraded":"2024-09-17T03:53:36.529388+0000","last_fullsized":"2024-09-17T03:53:36.529388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:52.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529554+0000","last_change":"2024-09-17T03:53:35.071423+0000","last_active":"2024-09-17T03:53:36.529554+0000","last_peered":"2024-09-17T03:53:36.529554+0000","last_clean":"2024-09-17T03:53:36.529554+0000","last_became_active":"2024-09-17T03:53:35.071008+0000","last_became_peered":"2024-09-17T03:53:35.071008+0000","last_unstale":"2024-09-17T03:53:36.529554+0000","last_undegraded":"2024-09-17T03:53:36.529554+0000","last_fullsized":"2024-09-17T03:53:36.529554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:43.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058428+0000","last_change":"2024-09-17T03:53:34.045408+0000","last_active":"2024-09-17T03:53:36.058428+0000","last_peered":"2024-09-17T03:53:36.058428+0000","last_clean":"2024-09-17T03:53:36.058428+0000","last_became_active":"2024-09-17T03:53:34.045021+0000","last_became_peered":"2024-09-17T03:53:34.045021+0000","last_unstale":"2024-09-17T03:53:36.058428+0000","last_undegraded":"2024-09-17T03:53:36.058428+0000","last_fullsized":"2024-09-17T03:53:36.058428+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248920+0000","last_change":"2024-09-17T03:53:35.070768+0000","last_active":"2024-09-17T03:53:36.248920+0000","last_peered":"2024-09-17T03:53:36.248920+0000","last_clean":"2024-09-17T03:53:36.248920+0000","last_became_active":"2024-09-17T03:53:35.070352+0000","last_became_peered":"2024-09-17T03:53:35.070352+0000","last_unstale":"2024-09-17T03:53:36.248920+0000","last_undegraded":"2024-09-17T03:53:36.248920+0000","last_fullsized":"2024-09-17T03:53:36.248920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:11.336036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178647+0000","last_change":"2024-09-17T03:53:34.041865+0000","last_active":"2024-09-17T03:53:36.178647+0000","last_peered":"2024-09-17T03:53:36.178647+0000","last_clean":"2024-09-17T03:53:36.178647+0000","last_became_active":"2024-09-17T03:53:34.041169+0000","last_became_peered":"2024-09-17T03:53:34.041169+0000","last_unstale":"2024-09-17T03:53:36.178647+0000","last_undegraded":"2024-09-17T03:53:36.178647+0000","last_fullsized":"2024-09-17T03:53:36.178647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:16.044280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178489+0000","last_change":"2024-09-17T03:53:35.064940+0000","last_active":"2024-09-17T03:53:36.178489+0000","last_peered":"2024-09-17T03:53:36.178489+0000","last_clean":"2024-09-17T03:53:36.178489+0000","last_became_active":"2024-09-17T03:53:35.064574+0000","last_became_peered":"2024-09-17T03:53:35.064574+0000","last_unstale":"2024-09-17T03:53:36.178489+0000","last_undegraded":"2024-09-17T03:53:36.178489+0000","last_fullsized":"2024-09-17T03:53:36.178489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:08.160089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057951+0000","last_change":"2024-09-17T03:53:34.048127+0000","last_active":"2024-09-17T03:53:36.057951+0000","last_peered":"2024-09-17T03:53:36.057951+0000","last_clean":"2024-09-17T03:53:36.057951+0000","last_became_active":"2024-09-17T03:53:34.047768+0000","last_became_peered":"2024-09-17T03:53:34.047768+0000","last_unstale":"2024-09-17T03:53:36.057951+0000","last_undegraded":"2024-09-17T03:53:36.057951+0000","last_fullsized":"2024-09-17T03:53:36.057951+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:17.363867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057698+0000","last_change":"2024-09-17T03:53:35.051987+0000","last_active":"2024-09-17T03:53:36.057698+0000","last_peered":"2024-09-17T03:53:36.057698+0000","last_clean":"2024-09-17T03:53:36.057698+0000","last_became_active":"2024-09-17T03:53:35.049620+0000","last_became_peered":"2024-09-17T03:53:35.049620+0000","last_unstale":"2024-09-17T03:53:36.057698+0000","last_undegraded":"2024-09-17T03:53:36.057698+0000","last_fullsized":"2024-09-17T03:53:36.057698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:21.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249368+0000","last_change":"2024-09-17T03:53:34.046914+0000","last_active":"2024-09-17T03:53:36.249368+0000","last_peered":"2024-09-17T03:53:36.249368+0000","last_clean":"2024-09-17T03:53:36.249368+0000","last_became_active":"2024-09-17T03:53:34.046056+0000","last_became_peered":"2024-09-17T03:53:34.046056+0000","last_unstale":"2024-09-17T03:53:36.249368+0000","last_undegraded":"2024-09-17T03:53:36.249368+0000","last_fullsized":"2024-09-17T03:53:36.249368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:53.290335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":219,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249351+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:53:36.249351+0000","last_peered":"2024-09-17T03:53:36.249351+0000","last_clean":"2024-09-17T03:53:36.249351+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:53:36.249351+0000","last_undegraded":"2024-09-17T03:53:36.249351+0000","last_fullsized":"2024-09-17T03:53:36.249351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509493,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509496,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":417572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509495,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":415938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509493,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":417550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509495,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":416520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509495,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":416132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542197,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":999424,"data_stored":416632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542198,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:53:44.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:53:44.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 -- 172.21.15.129:0/3463627950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c4005250 msgr2=0x7fb9c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 --2- 172.21.15.129:0/3463627950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c4005250 0x7fb9c400b9c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fb9d8005fd0 tx=0x7fb9c4005830 comp rx=0 tx=0).stop 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 -- 172.21.15.129:0/3463627950 shutdown_connections 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 --2- 172.21.15.129:0/3463627950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9c4005250 0x7fb9c400b9c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 -- 172.21.15.129:0/3463627950 >> 172.21.15.129:0/3463627950 conn(0x7fb9e00b1cb0 msgr2=0x7fb9e00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 -- 172.21.15.129:0/3463627950 shutdown_connections 2024-09-17T03:53:44.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.680+0000 7fb9e7600640 1 -- 172.21.15.129:0/3463627950 wait complete. 2024-09-17T03:53:44.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.682+0000 7fb9e7600640 1 Processor -- start 2024-09-17T03:53:44.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.683+0000 7fb9e7600640 1 -- start start 2024-09-17T03:53:44.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 0x7fb9e01f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:44.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:44.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9e015ef50 0x7fb9e015f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e0074640 con 0x7fb9e00c80d0 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb9e015cc00 con 0x7fb9c4005250 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9e7600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb9e00747c0 con 0x7fb9e015ef50 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38904/0 (socket says 172.21.15.129:38904) 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.684+0000 7fb9d7e00640 1 -- 172.21.15.129:0/1839655763 learned_addr learned my addr 172.21.15.129:0/1839655763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9e4c00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 0x7fb9e01f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9e5600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9e015ef50 0x7fb9e015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 -- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9e015ef50 msgr2=0x7fb9e015f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9e015ef50 0x7fb9e015f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 -- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 msgr2=0x7fb9e01f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 0x7fb9e01f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 -- 172.21.15.129:0/1839655763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9e015f8a0 con 0x7fb9e00c80d0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9e4c00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 0x7fb9e01f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9d7e00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8006480 tx=0x7fb9c80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8015070 con 0x7fb9e00c80d0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9c8011880 con 0x7fb9e00c80d0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8011cd0 con 0x7fb9e00c80d0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9e00cc1d0 con 0x7fb9e00c80d0 2024-09-17T03:53:44.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.685+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9e00cc750 con 0x7fb9e00c80d0 2024-09-17T03:53:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.686+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9e0074d20 con 0x7fb9e00c80d0 2024-09-17T03:53:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.688+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb9c8018070 con 0x7fb9e00c80d0 2024-09-17T03:53:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.689+0000 7fb9dd600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 0x7fb9bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:44.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.689+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb9c809bae0 con 0x7fb9e00c80d0 2024-09-17T03:53:44.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.689+0000 7fb9e4c00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 0x7fb9bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:44.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.690+0000 7fb9e4c00640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 0x7fb9bc07b0a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d0006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:44.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.693+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb9e015f8a0 con 0x7fb9e00c80d0 2024-09-17T03:53:44.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.872+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9e00c8a50 con 0x7fb9bc078b90 2024-09-17T03:53:44.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.877+0000 7fb9dd600640 1 -- 172.21.15.129:0/1839655763 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251952 (secure 0 0 0) 0x7fb9e00c8a50 con 0x7fb9bc078b90 2024-09-17T03:53:44.880 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:44.881 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 msgr2=0x7fb9bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 0x7fb9bc07b0a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0000c00 tx=0x7fb9d0006b70 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 msgr2=0x7fb9e00cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb9c8006480 tx=0x7fb9c80039c0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 shutdown_connections 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9bc078b90 0x7fb9bc07b0a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9e015ef50 0x7fb9e015f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9e00c80d0 0x7fb9e00cb170 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 --2- 172.21.15.129:0/1839655763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9c4005250 0x7fb9e01f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 >> 172.21.15.129:0/1839655763 conn(0x7fb9e00b1cb0 msgr2=0x7fb9e015a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 shutdown_connections 2024-09-17T03:53:44.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:44.881+0000 7fb9e7600640 1 -- 172.21.15.129:0/1839655763 wait complete. 2024-09-17T03:53:44.905 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-17T03:53:44.361480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258452,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716268,"statfs":{"total":773094113280,"available":772829458432,"internally_reserved":0,"allocated":9150464,"data_stored":4513060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.325860"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178823+0000","last_change":"2024-09-17T03:53:35.072458+0000","last_active":"2024-09-17T03:53:36.178823+0000","last_peered":"2024-09-17T03:53:36.178823+0000","last_clean":"2024-09-17T03:53:36.178823+0000","last_became_active":"2024-09-17T03:53:35.070660+0000","last_became_peered":"2024-09-17T03:53:35.070660+0000","last_unstale":"2024-09-17T03:53:36.178823+0000","last_undegraded":"2024-09-17T03:53:36.178823+0000","last_fullsized":"2024-09-17T03:53:36.178823+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:56.505206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058029+0000","last_change":"2024-09-17T03:53:35.067959+0000","last_active":"2024-09-17T03:53:36.058029+0000","last_peered":"2024-09-17T03:53:36.058029+0000","last_clean":"2024-09-17T03:53:36.058029+0000","last_became_active":"2024-09-17T03:53:35.067646+0000","last_became_peered":"2024-09-17T03:53:35.067646+0000","last_unstale":"2024-09-17T03:53:36.058029+0000","last_undegraded":"2024-09-17T03:53:36.058029+0000","last_fullsized":"2024-09-17T03:53:36.058029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:55.989947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249115+0000","last_change":"2024-09-17T03:53:35.073278+0000","last_active":"2024-09-17T03:53:36.249115+0000","last_peered":"2024-09-17T03:53:36.249115+0000","last_clean":"2024-09-17T03:53:36.249115+0000","last_became_active":"2024-09-17T03:53:35.072943+0000","last_became_peered":"2024-09-17T03:53:35.072943+0000","last_unstale":"2024-09-17T03:53:36.249115+0000","last_undegraded":"2024-09-17T03:53:36.249115+0000","last_fullsized":"2024-09-17T03:53:36.249115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528171+0000","last_change":"2024-09-17T03:53:35.063690+0000","last_active":"2024-09-17T03:53:36.528171+0000","last_peered":"2024-09-17T03:53:36.528171+0000","last_clean":"2024-09-17T03:53:36.528171+0000","last_became_active":"2024-09-17T03:53:35.062877+0000","last_became_peered":"2024-09-17T03:53:35.062877+0000","last_unstale":"2024-09-17T03:53:36.528171+0000","last_undegraded":"2024-09-17T03:53:36.528171+0000","last_fullsized":"2024-09-17T03:53:36.528171+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:42.690776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058612+0000","last_change":"2024-09-17T03:53:35.054924+0000","last_active":"2024-09-17T03:53:36.058612+0000","last_peered":"2024-09-17T03:53:36.058612+0000","last_clean":"2024-09-17T03:53:36.058612+0000","last_became_active":"2024-09-17T03:53:35.053852+0000","last_became_peered":"2024-09-17T03:53:35.053852+0000","last_unstale":"2024-09-17T03:53:36.058612+0000","last_undegraded":"2024-09-17T03:53:36.058612+0000","last_fullsized":"2024-09-17T03:53:36.058612+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:49.257827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058725+0000","last_change":"2024-09-17T03:53:35.067059+0000","last_active":"2024-09-17T03:53:36.058725+0000","last_peered":"2024-09-17T03:53:36.058725+0000","last_clean":"2024-09-17T03:53:36.058725+0000","last_became_active":"2024-09-17T03:53:35.066734+0000","last_became_peered":"2024-09-17T03:53:35.066734+0000","last_unstale":"2024-09-17T03:53:36.058725+0000","last_undegraded":"2024-09-17T03:53:36.058725+0000","last_fullsized":"2024-09-17T03:53:36.058725+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:54.297031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528126+0000","last_change":"2024-09-17T03:53:35.059267+0000","last_active":"2024-09-17T03:53:36.528126+0000","last_peered":"2024-09-17T03:53:36.528126+0000","last_clean":"2024-09-17T03:53:36.528126+0000","last_became_active":"2024-09-17T03:53:35.058966+0000","last_became_peered":"2024-09-17T03:53:35.058966+0000","last_unstale":"2024-09-17T03:53:36.528126+0000","last_undegraded":"2024-09-17T03:53:36.528126+0000","last_fullsized":"2024-09-17T03:53:36.528126+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:17:51.318108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058914+0000","last_change":"2024-09-17T03:53:35.051417+0000","last_active":"2024-09-17T03:53:36.058914+0000","last_peered":"2024-09-17T03:53:36.058914+0000","last_clean":"2024-09-17T03:53:36.058914+0000","last_became_active":"2024-09-17T03:53:35.051082+0000","last_became_peered":"2024-09-17T03:53:35.051082+0000","last_unstale":"2024-09-17T03:53:36.058914+0000","last_undegraded":"2024-09-17T03:53:36.058914+0000","last_fullsized":"2024-09-17T03:53:36.058914+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:40.340287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058871+0000","last_change":"2024-09-17T03:53:35.076596+0000","last_active":"2024-09-17T03:53:36.058871+0000","last_peered":"2024-09-17T03:53:36.058871+0000","last_clean":"2024-09-17T03:53:36.058871+0000","last_became_active":"2024-09-17T03:53:35.076237+0000","last_became_peered":"2024-09-17T03:53:35.076237+0000","last_unstale":"2024-09-17T03:53:36.058871+0000","last_undegraded":"2024-09-17T03:53:36.058871+0000","last_fullsized":"2024-09-17T03:53:36.058871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:49.104278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178936+0000","last_change":"2024-09-17T03:53:35.075213+0000","last_active":"2024-09-17T03:53:36.178936+0000","last_peered":"2024-09-17T03:53:36.178936+0000","last_clean":"2024-09-17T03:53:36.178936+0000","last_became_active":"2024-09-17T03:53:35.074344+0000","last_became_peered":"2024-09-17T03:53:35.074344+0000","last_unstale":"2024-09-17T03:53:36.178936+0000","last_undegraded":"2024-09-17T03:53:36.178936+0000","last_fullsized":"2024-09-17T03:53:36.178936+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:45:02.543171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057911+0000","last_change":"2024-09-17T03:53:35.060229+0000","last_active":"2024-09-17T03:53:36.057911+0000","last_peered":"2024-09-17T03:53:36.057911+0000","last_clean":"2024-09-17T03:53:36.057911+0000","last_became_active":"2024-09-17T03:53:35.059966+0000","last_became_peered":"2024-09-17T03:53:35.059966+0000","last_unstale":"2024-09-17T03:53:36.057911+0000","last_undegraded":"2024-09-17T03:53:36.057911+0000","last_fullsized":"2024-09-17T03:53:36.057911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:50.055699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178709+0000","last_change":"2024-09-17T03:53:35.073592+0000","last_active":"2024-09-17T03:53:36.178709+0000","last_peered":"2024-09-17T03:53:36.178709+0000","last_clean":"2024-09-17T03:53:36.178709+0000","last_became_active":"2024-09-17T03:53:35.073318+0000","last_became_peered":"2024-09-17T03:53:35.073318+0000","last_unstale":"2024-09-17T03:53:36.178709+0000","last_undegraded":"2024-09-17T03:53:36.178709+0000","last_fullsized":"2024-09-17T03:53:36.178709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:26.858055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249029+0000","last_change":"2024-09-17T03:53:35.060095+0000","last_active":"2024-09-17T03:53:36.249029+0000","last_peered":"2024-09-17T03:53:36.249029+0000","last_clean":"2024-09-17T03:53:36.249029+0000","last_became_active":"2024-09-17T03:53:35.059471+0000","last_became_peered":"2024-09-17T03:53:35.059471+0000","last_unstale":"2024-09-17T03:53:36.249029+0000","last_undegraded":"2024-09-17T03:53:36.249029+0000","last_fullsized":"2024-09-17T03:53:36.249029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528250+0000","last_change":"2024-09-17T03:53:35.080649+0000","last_active":"2024-09-17T03:53:36.528250+0000","last_peered":"2024-09-17T03:53:36.528250+0000","last_clean":"2024-09-17T03:53:36.528250+0000","last_became_active":"2024-09-17T03:53:35.080294+0000","last_became_peered":"2024-09-17T03:53:35.080294+0000","last_unstale":"2024-09-17T03:53:36.528250+0000","last_undegraded":"2024-09-17T03:53:36.528250+0000","last_fullsized":"2024-09-17T03:53:36.528250+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:33.740179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528771+0000","last_change":"2024-09-17T03:53:35.059152+0000","last_active":"2024-09-17T03:53:36.528771+0000","last_peered":"2024-09-17T03:53:36.528771+0000","last_clean":"2024-09-17T03:53:36.528771+0000","last_became_active":"2024-09-17T03:53:35.058509+0000","last_became_peered":"2024-09-17T03:53:35.058509+0000","last_unstale":"2024-09-17T03:53:36.528771+0000","last_undegraded":"2024-09-17T03:53:36.528771+0000","last_fullsized":"2024-09-17T03:53:36.528771+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:47.079665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058072+0000","last_change":"2024-09-17T03:53:35.070836+0000","last_active":"2024-09-17T03:53:36.058072+0000","last_peered":"2024-09-17T03:53:36.058072+0000","last_clean":"2024-09-17T03:53:36.058072+0000","last_became_active":"2024-09-17T03:53:35.065693+0000","last_became_peered":"2024-09-17T03:53:35.065693+0000","last_unstale":"2024-09-17T03:53:36.058072+0000","last_undegraded":"2024-09-17T03:53:36.058072+0000","last_fullsized":"2024-09-17T03:53:36.058072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:50.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059504+0000","last_change":"2024-09-17T03:53:35.070366+0000","last_active":"2024-09-17T03:53:36.059504+0000","last_peered":"2024-09-17T03:53:36.059504+0000","last_clean":"2024-09-17T03:53:36.059504+0000","last_became_active":"2024-09-17T03:53:35.070062+0000","last_became_peered":"2024-09-17T03:53:35.070062+0000","last_unstale":"2024-09-17T03:53:36.059504+0000","last_undegraded":"2024-09-17T03:53:36.059504+0000","last_fullsized":"2024-09-17T03:53:36.059504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:45.603585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528047+0000","last_change":"2024-09-17T03:53:35.065367+0000","last_active":"2024-09-17T03:53:36.528047+0000","last_peered":"2024-09-17T03:53:36.528047+0000","last_clean":"2024-09-17T03:53:36.528047+0000","last_became_active":"2024-09-17T03:53:35.065042+0000","last_became_peered":"2024-09-17T03:53:35.065042+0000","last_unstale":"2024-09-17T03:53:36.528047+0000","last_undegraded":"2024-09-17T03:53:36.528047+0000","last_fullsized":"2024-09-17T03:53:36.528047+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:01.776287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529477+0000","last_change":"2024-09-17T03:53:35.077049+0000","last_active":"2024-09-17T03:53:36.529477+0000","last_peered":"2024-09-17T03:53:36.529477+0000","last_clean":"2024-09-17T03:53:36.529477+0000","last_became_active":"2024-09-17T03:53:35.076178+0000","last_became_peered":"2024-09-17T03:53:35.076178+0000","last_unstale":"2024-09-17T03:53:36.529477+0000","last_undegraded":"2024-09-17T03:53:36.529477+0000","last_fullsized":"2024-09-17T03:53:36.529477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.223788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528413+0000","last_change":"2024-09-17T03:53:35.070151+0000","last_active":"2024-09-17T03:53:36.528413+0000","last_peered":"2024-09-17T03:53:36.528413+0000","last_clean":"2024-09-17T03:53:36.528413+0000","last_became_active":"2024-09-17T03:53:35.069825+0000","last_became_peered":"2024-09-17T03:53:35.069825+0000","last_unstale":"2024-09-17T03:53:36.528413+0000","last_undegraded":"2024-09-17T03:53:36.528413+0000","last_fullsized":"2024-09-17T03:53:36.528413+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:41.443412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528656+0000","last_change":"2024-09-17T03:53:35.077741+0000","last_active":"2024-09-17T03:53:36.528656+0000","last_peered":"2024-09-17T03:53:36.528656+0000","last_clean":"2024-09-17T03:53:36.528656+0000","last_became_active":"2024-09-17T03:53:35.077437+0000","last_became_peered":"2024-09-17T03:53:35.077437+0000","last_unstale":"2024-09-17T03:53:36.528656+0000","last_undegraded":"2024-09-17T03:53:36.528656+0000","last_fullsized":"2024-09-17T03:53:36.528656+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:51.309170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057255+0000","last_change":"2024-09-17T03:53:35.057094+0000","last_active":"2024-09-17T03:53:36.057255+0000","last_peered":"2024-09-17T03:53:36.057255+0000","last_clean":"2024-09-17T03:53:36.057255+0000","last_became_active":"2024-09-17T03:53:35.056803+0000","last_became_peered":"2024-09-17T03:53:35.056803+0000","last_unstale":"2024-09-17T03:53:36.057255+0000","last_undegraded":"2024-09-17T03:53:36.057255+0000","last_fullsized":"2024-09-17T03:53:36.057255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:43.810445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178654+0000","last_change":"2024-09-17T03:53:35.068757+0000","last_active":"2024-09-17T03:53:36.178654+0000","last_peered":"2024-09-17T03:53:36.178654+0000","last_clean":"2024-09-17T03:53:36.178654+0000","last_became_active":"2024-09-17T03:53:35.068235+0000","last_became_peered":"2024-09-17T03:53:35.068235+0000","last_unstale":"2024-09-17T03:53:36.178654+0000","last_undegraded":"2024-09-17T03:53:36.178654+0000","last_fullsized":"2024-09-17T03:53:36.178654+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:16.303914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527840+0000","last_change":"2024-09-17T03:53:35.058419+0000","last_active":"2024-09-17T03:53:36.527840+0000","last_peered":"2024-09-17T03:53:36.527840+0000","last_clean":"2024-09-17T03:53:36.527840+0000","last_became_active":"2024-09-17T03:53:35.057357+0000","last_became_peered":"2024-09-17T03:53:35.057357+0000","last_unstale":"2024-09-17T03:53:36.527840+0000","last_undegraded":"2024-09-17T03:53:36.527840+0000","last_fullsized":"2024-09-17T03:53:36.527840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:46.020805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057970+0000","last_change":"2024-09-17T03:53:35.067946+0000","last_active":"2024-09-17T03:53:36.057970+0000","last_peered":"2024-09-17T03:53:36.057970+0000","last_clean":"2024-09-17T03:53:36.057970+0000","last_became_active":"2024-09-17T03:53:35.067630+0000","last_became_peered":"2024-09-17T03:53:35.067630+0000","last_unstale":"2024-09-17T03:53:36.057970+0000","last_undegraded":"2024-09-17T03:53:36.057970+0000","last_fullsized":"2024-09-17T03:53:36.057970+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:12.178924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249169+0000","last_change":"2024-09-17T03:53:35.064329+0000","last_active":"2024-09-17T03:53:36.249169+0000","last_peered":"2024-09-17T03:53:36.249169+0000","last_clean":"2024-09-17T03:53:36.249169+0000","last_became_active":"2024-09-17T03:53:35.062761+0000","last_became_peered":"2024-09-17T03:53:35.062761+0000","last_unstale":"2024-09-17T03:53:36.249169+0000","last_undegraded":"2024-09-17T03:53:36.249169+0000","last_fullsized":"2024-09-17T03:53:36.249169+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059290+0000","last_change":"2024-09-17T03:53:35.057286+0000","last_active":"2024-09-17T03:53:36.059290+0000","last_peered":"2024-09-17T03:53:36.059290+0000","last_clean":"2024-09-17T03:53:36.059290+0000","last_became_active":"2024-09-17T03:53:35.056910+0000","last_became_peered":"2024-09-17T03:53:35.056910+0000","last_unstale":"2024-09-17T03:53:36.059290+0000","last_undegraded":"2024-09-17T03:53:36.059290+0000","last_fullsized":"2024-09-17T03:53:36.059290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:48.490076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178706+0000","last_change":"2024-09-17T03:53:35.067698+0000","last_active":"2024-09-17T03:53:36.178706+0000","last_peered":"2024-09-17T03:53:36.178706+0000","last_clean":"2024-09-17T03:53:36.178706+0000","last_became_active":"2024-09-17T03:53:35.066584+0000","last_became_peered":"2024-09-17T03:53:35.066584+0000","last_unstale":"2024-09-17T03:53:36.178706+0000","last_undegraded":"2024-09-17T03:53:36.178706+0000","last_fullsized":"2024-09-17T03:53:36.178706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:57.060788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057393+0000","last_change":"2024-09-17T03:53:35.065693+0000","last_active":"2024-09-17T03:53:36.057393+0000","last_peered":"2024-09-17T03:53:36.057393+0000","last_clean":"2024-09-17T03:53:36.057393+0000","last_became_active":"2024-09-17T03:53:35.065403+0000","last_became_peered":"2024-09-17T03:53:35.065403+0000","last_unstale":"2024-09-17T03:53:36.057393+0000","last_undegraded":"2024-09-17T03:53:36.057393+0000","last_fullsized":"2024-09-17T03:53:36.057393+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:50.272364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057291+0000","last_change":"2024-09-17T03:53:35.075375+0000","last_active":"2024-09-17T03:53:36.057291+0000","last_peered":"2024-09-17T03:53:36.057291+0000","last_clean":"2024-09-17T03:53:36.057291+0000","last_became_active":"2024-09-17T03:53:35.074718+0000","last_became_peered":"2024-09-17T03:53:35.074718+0000","last_unstale":"2024-09-17T03:53:36.057291+0000","last_undegraded":"2024-09-17T03:53:36.057291+0000","last_fullsized":"2024-09-17T03:53:36.057291+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:39.360722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058059+0000","last_change":"2024-09-17T03:53:35.069885+0000","last_active":"2024-09-17T03:53:36.058059+0000","last_peered":"2024-09-17T03:53:36.058059+0000","last_clean":"2024-09-17T03:53:36.058059+0000","last_became_active":"2024-09-17T03:53:35.069558+0000","last_became_peered":"2024-09-17T03:53:35.069558+0000","last_unstale":"2024-09-17T03:53:36.058059+0000","last_undegraded":"2024-09-17T03:53:36.058059+0000","last_fullsized":"2024-09-17T03:53:36.058059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:25.141426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178515+0000","last_change":"2024-09-17T03:53:35.062010+0000","last_active":"2024-09-17T03:53:36.178515+0000","last_peered":"2024-09-17T03:53:36.178515+0000","last_clean":"2024-09-17T03:53:36.178515+0000","last_became_active":"2024-09-17T03:53:35.059874+0000","last_became_peered":"2024-09-17T03:53:35.059874+0000","last_unstale":"2024-09-17T03:53:36.178515+0000","last_undegraded":"2024-09-17T03:53:36.178515+0000","last_fullsized":"2024-09-17T03:53:36.178515+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:26.933349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060228+0000","last_change":"2024-09-17T03:53:35.065637+0000","last_active":"2024-09-17T03:53:36.060228+0000","last_peered":"2024-09-17T03:53:36.060228+0000","last_clean":"2024-09-17T03:53:36.060228+0000","last_became_active":"2024-09-17T03:53:35.065252+0000","last_became_peered":"2024-09-17T03:53:35.065252+0000","last_unstale":"2024-09-17T03:53:36.060228+0000","last_undegraded":"2024-09-17T03:53:36.060228+0000","last_fullsized":"2024-09-17T03:53:36.060228+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:17.453283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"80'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178966+0000","last_change":"2024-09-17T03:53:34.042427+0000","last_active":"2024-09-17T03:53:36.178966+0000","last_peered":"2024-09-17T03:53:36.178966+0000","last_clean":"2024-09-17T03:53:36.178966+0000","last_became_active":"2024-09-17T03:53:34.042117+0000","last_became_peered":"2024-09-17T03:53:34.042117+0000","last_unstale":"2024-09-17T03:53:36.178966+0000","last_undegraded":"2024-09-17T03:53:36.178966+0000","last_fullsized":"2024-09-17T03:53:36.178966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:57.275556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178860+0000","last_change":"2024-09-17T03:53:35.075209+0000","last_active":"2024-09-17T03:53:36.178860+0000","last_peered":"2024-09-17T03:53:36.178860+0000","last_clean":"2024-09-17T03:53:36.178860+0000","last_became_active":"2024-09-17T03:53:35.074557+0000","last_became_peered":"2024-09-17T03:53:35.074557+0000","last_unstale":"2024-09-17T03:53:36.178860+0000","last_undegraded":"2024-09-17T03:53:36.178860+0000","last_fullsized":"2024-09-17T03:53:36.178860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:26.363909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060393+0000","last_change":"2024-09-17T03:53:34.047657+0000","last_active":"2024-09-17T03:53:36.060393+0000","last_peered":"2024-09-17T03:53:36.060393+0000","last_clean":"2024-09-17T03:53:36.060393+0000","last_became_active":"2024-09-17T03:53:34.047290+0000","last_became_peered":"2024-09-17T03:53:34.047290+0000","last_unstale":"2024-09-17T03:53:36.060393+0000","last_undegraded":"2024-09-17T03:53:36.060393+0000","last_fullsized":"2024-09-17T03:53:36.060393+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:25.205958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178967+0000","last_change":"2024-09-17T03:53:35.068246+0000","last_active":"2024-09-17T03:53:36.178967+0000","last_peered":"2024-09-17T03:53:36.178967+0000","last_clean":"2024-09-17T03:53:36.178967+0000","last_became_active":"2024-09-17T03:53:35.066866+0000","last_became_peered":"2024-09-17T03:53:35.066866+0000","last_unstale":"2024-09-17T03:53:36.178967+0000","last_undegraded":"2024-09-17T03:53:36.178967+0000","last_fullsized":"2024-09-17T03:53:36.178967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:59.503515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529312+0000","last_change":"2024-09-17T03:53:34.047098+0000","last_active":"2024-09-17T03:53:36.529312+0000","last_peered":"2024-09-17T03:53:36.529312+0000","last_clean":"2024-09-17T03:53:36.529312+0000","last_became_active":"2024-09-17T03:53:34.046816+0000","last_became_peered":"2024-09-17T03:53:34.046816+0000","last_unstale":"2024-09-17T03:53:36.529312+0000","last_undegraded":"2024-09-17T03:53:36.529312+0000","last_fullsized":"2024-09-17T03:53:36.529312+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:25.856611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249238+0000","last_change":"2024-09-17T03:53:35.067274+0000","last_active":"2024-09-17T03:53:36.249238+0000","last_peered":"2024-09-17T03:53:36.249238+0000","last_clean":"2024-09-17T03:53:36.249238+0000","last_became_active":"2024-09-17T03:53:35.066895+0000","last_became_peered":"2024-09-17T03:53:35.066895+0000","last_unstale":"2024-09-17T03:53:36.249238+0000","last_undegraded":"2024-09-17T03:53:36.249238+0000","last_fullsized":"2024-09-17T03:53:36.249238+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529204+0000","last_change":"2024-09-17T03:53:34.047320+0000","last_active":"2024-09-17T03:53:36.529204+0000","last_peered":"2024-09-17T03:53:36.529204+0000","last_clean":"2024-09-17T03:53:36.529204+0000","last_became_active":"2024-09-17T03:53:34.045973+0000","last_became_peered":"2024-09-17T03:53:34.045973+0000","last_unstale":"2024-09-17T03:53:36.529204+0000","last_undegraded":"2024-09-17T03:53:36.529204+0000","last_fullsized":"2024-09-17T03:53:36.529204+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:05:03.153785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057495+0000","last_change":"2024-09-17T03:53:35.075107+0000","last_active":"2024-09-17T03:53:36.057495+0000","last_peered":"2024-09-17T03:53:36.057495+0000","last_clean":"2024-09-17T03:53:36.057495+0000","last_became_active":"2024-09-17T03:53:35.069999+0000","last_became_peered":"2024-09-17T03:53:35.069999+0000","last_unstale":"2024-09-17T03:53:36.057495+0000","last_undegraded":"2024-09-17T03:53:36.057495+0000","last_fullsized":"2024-09-17T03:53:36.057495+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:15.911260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059049+0000","last_change":"2024-09-17T03:53:34.046521+0000","last_active":"2024-09-17T03:53:36.059049+0000","last_peered":"2024-09-17T03:53:36.059049+0000","last_clean":"2024-09-17T03:53:36.059049+0000","last_became_active":"2024-09-17T03:53:34.046151+0000","last_became_peered":"2024-09-17T03:53:34.046151+0000","last_unstale":"2024-09-17T03:53:36.059049+0000","last_undegraded":"2024-09-17T03:53:36.059049+0000","last_fullsized":"2024-09-17T03:53:36.059049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:04.714385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058821+0000","last_change":"2024-09-17T03:53:35.069264+0000","last_active":"2024-09-17T03:53:36.058821+0000","last_peered":"2024-09-17T03:53:36.058821+0000","last_clean":"2024-09-17T03:53:36.058821+0000","last_became_active":"2024-09-17T03:53:35.065458+0000","last_became_peered":"2024-09-17T03:53:35.065458+0000","last_unstale":"2024-09-17T03:53:36.058821+0000","last_undegraded":"2024-09-17T03:53:36.058821+0000","last_fullsized":"2024-09-17T03:53:36.058821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:16.559380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058298+0000","last_change":"2024-09-17T03:53:34.045410+0000","last_active":"2024-09-17T03:53:36.058298+0000","last_peered":"2024-09-17T03:53:36.058298+0000","last_clean":"2024-09-17T03:53:36.058298+0000","last_became_active":"2024-09-17T03:53:34.045030+0000","last_became_peered":"2024-09-17T03:53:34.045030+0000","last_unstale":"2024-09-17T03:53:36.058298+0000","last_undegraded":"2024-09-17T03:53:36.058298+0000","last_fullsized":"2024-09-17T03:53:36.058298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248910+0000","last_change":"2024-09-17T03:53:35.075625+0000","last_active":"2024-09-17T03:53:36.248910+0000","last_peered":"2024-09-17T03:53:36.248910+0000","last_clean":"2024-09-17T03:53:36.248910+0000","last_became_active":"2024-09-17T03:53:35.075285+0000","last_became_peered":"2024-09-17T03:53:35.075285+0000","last_unstale":"2024-09-17T03:53:36.248910+0000","last_undegraded":"2024-09-17T03:53:36.248910+0000","last_fullsized":"2024-09-17T03:53:36.248910+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528363+0000","last_change":"2024-09-17T03:53:34.040513+0000","last_active":"2024-09-17T03:53:36.528363+0000","last_peered":"2024-09-17T03:53:36.528363+0000","last_clean":"2024-09-17T03:53:36.528363+0000","last_became_active":"2024-09-17T03:53:34.039387+0000","last_became_peered":"2024-09-17T03:53:34.039387+0000","last_unstale":"2024-09-17T03:53:36.528363+0000","last_undegraded":"2024-09-17T03:53:36.528363+0000","last_fullsized":"2024-09-17T03:53:36.528363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:35.514048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528897+0000","last_change":"2024-09-17T03:53:35.058824+0000","last_active":"2024-09-17T03:53:36.528897+0000","last_peered":"2024-09-17T03:53:36.528897+0000","last_clean":"2024-09-17T03:53:36.528897+0000","last_became_active":"2024-09-17T03:53:35.058053+0000","last_became_peered":"2024-09-17T03:53:35.058053+0000","last_unstale":"2024-09-17T03:53:36.528897+0000","last_undegraded":"2024-09-17T03:53:36.528897+0000","last_fullsized":"2024-09-17T03:53:36.528897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:39.813037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057165+0000","last_change":"2024-09-17T03:53:34.042323+0000","last_active":"2024-09-17T03:53:36.057165+0000","last_peered":"2024-09-17T03:53:36.057165+0000","last_clean":"2024-09-17T03:53:36.057165+0000","last_became_active":"2024-09-17T03:53:34.041995+0000","last_became_peered":"2024-09-17T03:53:34.041995+0000","last_unstale":"2024-09-17T03:53:36.057165+0000","last_undegraded":"2024-09-17T03:53:36.057165+0000","last_fullsized":"2024-09-17T03:53:36.057165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:38:54.594238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.530067+0000","last_change":"2024-09-17T03:53:35.066310+0000","last_active":"2024-09-17T03:53:36.530067+0000","last_peered":"2024-09-17T03:53:36.530067+0000","last_clean":"2024-09-17T03:53:36.530067+0000","last_became_active":"2024-09-17T03:53:35.065211+0000","last_became_peered":"2024-09-17T03:53:35.065211+0000","last_unstale":"2024-09-17T03:53:36.530067+0000","last_undegraded":"2024-09-17T03:53:36.530067+0000","last_fullsized":"2024-09-17T03:53:36.530067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:11.925576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179093+0000","last_change":"2024-09-17T03:53:34.041277+0000","last_active":"2024-09-17T03:53:36.179093+0000","last_peered":"2024-09-17T03:53:36.179093+0000","last_clean":"2024-09-17T03:53:36.179093+0000","last_became_active":"2024-09-17T03:53:34.040555+0000","last_became_peered":"2024-09-17T03:53:34.040555+0000","last_unstale":"2024-09-17T03:53:36.179093+0000","last_undegraded":"2024-09-17T03:53:36.179093+0000","last_fullsized":"2024-09-17T03:53:36.179093+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:09:05.315114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059958+0000","last_change":"2024-09-17T03:53:35.058829+0000","last_active":"2024-09-17T03:53:36.059958+0000","last_peered":"2024-09-17T03:53:36.059958+0000","last_clean":"2024-09-17T03:53:36.059958+0000","last_became_active":"2024-09-17T03:53:35.058494+0000","last_became_peered":"2024-09-17T03:53:35.058494+0000","last_unstale":"2024-09-17T03:53:36.059958+0000","last_undegraded":"2024-09-17T03:53:36.059958+0000","last_fullsized":"2024-09-17T03:53:36.059958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:09.497624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178778+0000","last_change":"2024-09-17T03:53:34.047020+0000","last_active":"2024-09-17T03:53:36.178778+0000","last_peered":"2024-09-17T03:53:36.178778+0000","last_clean":"2024-09-17T03:53:36.178778+0000","last_became_active":"2024-09-17T03:53:34.045487+0000","last_became_peered":"2024-09-17T03:53:34.045487+0000","last_unstale":"2024-09-17T03:53:36.178778+0000","last_undegraded":"2024-09-17T03:53:36.178778+0000","last_fullsized":"2024-09-17T03:53:36.178778+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:03.205170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529300+0000","last_change":"2024-09-17T03:53:35.074637+0000","last_active":"2024-09-17T03:53:36.529300+0000","last_peered":"2024-09-17T03:53:36.529300+0000","last_clean":"2024-09-17T03:53:36.529300+0000","last_became_active":"2024-09-17T03:53:35.074086+0000","last_became_peered":"2024-09-17T03:53:35.074086+0000","last_unstale":"2024-09-17T03:53:36.529300+0000","last_undegraded":"2024-09-17T03:53:36.529300+0000","last_fullsized":"2024-09-17T03:53:36.529300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:11.027879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"80'3","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178381+0000","last_change":"2024-09-17T03:53:34.045567+0000","last_active":"2024-09-17T03:53:36.178381+0000","last_peered":"2024-09-17T03:53:36.178381+0000","last_clean":"2024-09-17T03:53:36.178381+0000","last_became_active":"2024-09-17T03:53:34.045029+0000","last_became_peered":"2024-09-17T03:53:34.045029+0000","last_unstale":"2024-09-17T03:53:36.178381+0000","last_undegraded":"2024-09-17T03:53:36.178381+0000","last_fullsized":"2024-09-17T03:53:36.178381+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:18:38.299845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059436+0000","last_change":"2024-09-17T03:53:35.078726+0000","last_active":"2024-09-17T03:53:36.059436+0000","last_peered":"2024-09-17T03:53:36.059436+0000","last_clean":"2024-09-17T03:53:36.059436+0000","last_became_active":"2024-09-17T03:53:35.078369+0000","last_became_peered":"2024-09-17T03:53:35.078369+0000","last_unstale":"2024-09-17T03:53:36.059436+0000","last_undegraded":"2024-09-17T03:53:36.059436+0000","last_fullsized":"2024-09-17T03:53:36.059436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:30:03.736633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"80'6","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059227+0000","last_change":"2024-09-17T03:53:34.033143+0000","last_active":"2024-09-17T03:53:36.059227+0000","last_peered":"2024-09-17T03:53:36.059227+0000","last_clean":"2024-09-17T03:53:36.059227+0000","last_became_active":"2024-09-17T03:53:34.031770+0000","last_became_peered":"2024-09-17T03:53:34.031770+0000","last_unstale":"2024-09-17T03:53:36.059227+0000","last_undegraded":"2024-09-17T03:53:36.059227+0000","last_fullsized":"2024-09-17T03:53:36.059227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529438+0000","last_change":"2024-09-17T03:53:35.062155+0000","last_active":"2024-09-17T03:53:36.529438+0000","last_peered":"2024-09-17T03:53:36.529438+0000","last_clean":"2024-09-17T03:53:36.529438+0000","last_became_active":"2024-09-17T03:53:35.061536+0000","last_became_peered":"2024-09-17T03:53:35.061536+0000","last_unstale":"2024-09-17T03:53:36.529438+0000","last_undegraded":"2024-09-17T03:53:36.529438+0000","last_fullsized":"2024-09-17T03:53:36.529438+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:36.274547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057583+0000","last_change":"2024-09-17T03:53:34.043064+0000","last_active":"2024-09-17T03:53:36.057583+0000","last_peered":"2024-09-17T03:53:36.057583+0000","last_clean":"2024-09-17T03:53:36.057583+0000","last_became_active":"2024-09-17T03:53:34.038696+0000","last_became_peered":"2024-09-17T03:53:34.038696+0000","last_unstale":"2024-09-17T03:53:36.057583+0000","last_undegraded":"2024-09-17T03:53:36.057583+0000","last_fullsized":"2024-09-17T03:53:36.057583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:29.561790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249364+0000","last_change":"2024-09-17T03:53:35.063772+0000","last_active":"2024-09-17T03:53:36.249364+0000","last_peered":"2024-09-17T03:53:36.249364+0000","last_clean":"2024-09-17T03:53:36.249364+0000","last_became_active":"2024-09-17T03:53:35.063400+0000","last_became_peered":"2024-09-17T03:53:35.063400+0000","last_unstale":"2024-09-17T03:53:36.249364+0000","last_undegraded":"2024-09-17T03:53:36.249364+0000","last_fullsized":"2024-09-17T03:53:36.249364+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:14.277688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529588+0000","last_change":"2024-09-17T03:53:34.048029+0000","last_active":"2024-09-17T03:53:36.529588+0000","last_peered":"2024-09-17T03:53:36.529588+0000","last_clean":"2024-09-17T03:53:36.529588+0000","last_became_active":"2024-09-17T03:53:34.046461+0000","last_became_peered":"2024-09-17T03:53:34.046461+0000","last_unstale":"2024-09-17T03:53:36.529588+0000","last_undegraded":"2024-09-17T03:53:36.529588+0000","last_fullsized":"2024-09-17T03:53:36.529588+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:07.500302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057588+0000","last_change":"2024-09-17T03:53:35.062083+0000","last_active":"2024-09-17T03:53:36.057588+0000","last_peered":"2024-09-17T03:53:36.057588+0000","last_clean":"2024-09-17T03:53:36.057588+0000","last_became_active":"2024-09-17T03:53:35.061800+0000","last_became_peered":"2024-09-17T03:53:35.061800+0000","last_unstale":"2024-09-17T03:53:36.057588+0000","last_undegraded":"2024-09-17T03:53:36.057588+0000","last_fullsized":"2024-09-17T03:53:36.057588+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:18:49.841803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059500+0000","last_change":"2024-09-17T03:53:34.045446+0000","last_active":"2024-09-17T03:53:36.059500+0000","last_peered":"2024-09-17T03:53:36.059500+0000","last_clean":"2024-09-17T03:53:36.059500+0000","last_became_active":"2024-09-17T03:53:34.045103+0000","last_became_peered":"2024-09-17T03:53:34.045103+0000","last_unstale":"2024-09-17T03:53:36.059500+0000","last_undegraded":"2024-09-17T03:53:36.059500+0000","last_fullsized":"2024-09-17T03:53:36.059500+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:13.853299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057116+0000","last_change":"2024-09-17T03:53:35.068169+0000","last_active":"2024-09-17T03:53:36.057116+0000","last_peered":"2024-09-17T03:53:36.057116+0000","last_clean":"2024-09-17T03:53:36.057116+0000","last_became_active":"2024-09-17T03:53:35.067861+0000","last_became_peered":"2024-09-17T03:53:35.067861+0000","last_unstale":"2024-09-17T03:53:36.057116+0000","last_undegraded":"2024-09-17T03:53:36.057116+0000","last_fullsized":"2024-09-17T03:53:36.057116+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:58.932353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"80'8","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528873+0000","last_change":"2024-09-17T03:53:34.043445+0000","last_active":"2024-09-17T03:53:36.528873+0000","last_peered":"2024-09-17T03:53:36.528873+0000","last_clean":"2024-09-17T03:53:36.528873+0000","last_became_active":"2024-09-17T03:53:34.042920+0000","last_became_peered":"2024-09-17T03:53:34.042920+0000","last_unstale":"2024-09-17T03:53:36.528873+0000","last_undegraded":"2024-09-17T03:53:36.528873+0000","last_fullsized":"2024-09-17T03:53:36.528873+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:42.885526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249340+0000","last_change":"2024-09-17T03:53:35.061631+0000","last_active":"2024-09-17T03:53:36.249340+0000","last_peered":"2024-09-17T03:53:36.249340+0000","last_clean":"2024-09-17T03:53:36.249340+0000","last_became_active":"2024-09-17T03:53:35.061273+0000","last_became_peered":"2024-09-17T03:53:35.061273+0000","last_unstale":"2024-09-17T03:53:36.249340+0000","last_undegraded":"2024-09-17T03:53:36.249340+0000","last_fullsized":"2024-09-17T03:53:36.249340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:08.506998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179205+0000","last_change":"2024-09-17T03:53:34.041969+0000","last_active":"2024-09-17T03:53:36.179205+0000","last_peered":"2024-09-17T03:53:36.179205+0000","last_clean":"2024-09-17T03:53:36.179205+0000","last_became_active":"2024-09-17T03:53:34.040090+0000","last_became_peered":"2024-09-17T03:53:34.040090+0000","last_unstale":"2024-09-17T03:53:36.179205+0000","last_undegraded":"2024-09-17T03:53:36.179205+0000","last_fullsized":"2024-09-17T03:53:36.179205+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:12.790859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179104+0000","last_change":"2024-09-17T03:53:35.071124+0000","last_active":"2024-09-17T03:53:36.179104+0000","last_peered":"2024-09-17T03:53:36.179104+0000","last_clean":"2024-09-17T03:53:36.179104+0000","last_became_active":"2024-09-17T03:53:35.070636+0000","last_became_peered":"2024-09-17T03:53:35.070636+0000","last_unstale":"2024-09-17T03:53:36.179104+0000","last_undegraded":"2024-09-17T03:53:36.179104+0000","last_fullsized":"2024-09-17T03:53:36.179104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:35:18.697811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529575+0000","last_change":"2024-09-17T03:53:34.049033+0000","last_active":"2024-09-17T03:53:36.529575+0000","last_peered":"2024-09-17T03:53:36.529575+0000","last_clean":"2024-09-17T03:53:36.529575+0000","last_became_active":"2024-09-17T03:53:34.048596+0000","last_became_peered":"2024-09-17T03:53:34.048596+0000","last_unstale":"2024-09-17T03:53:36.529575+0000","last_undegraded":"2024-09-17T03:53:36.529575+0000","last_fullsized":"2024-09-17T03:53:36.529575+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:25.851358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059564+0000","last_change":"2024-09-17T03:53:35.052022+0000","last_active":"2024-09-17T03:53:36.059564+0000","last_peered":"2024-09-17T03:53:36.059564+0000","last_clean":"2024-09-17T03:53:36.059564+0000","last_became_active":"2024-09-17T03:53:35.051704+0000","last_became_peered":"2024-09-17T03:53:35.051704+0000","last_unstale":"2024-09-17T03:53:36.059564+0000","last_undegraded":"2024-09-17T03:53:36.059564+0000","last_fullsized":"2024-09-17T03:53:36.059564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:43.978056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529714+0000","last_change":"2024-09-17T03:53:34.040065+0000","last_active":"2024-09-17T03:53:36.529714+0000","last_peered":"2024-09-17T03:53:36.529714+0000","last_clean":"2024-09-17T03:53:36.529714+0000","last_became_active":"2024-09-17T03:53:34.039675+0000","last_became_peered":"2024-09-17T03:53:34.039675+0000","last_unstale":"2024-09-17T03:53:36.529714+0000","last_undegraded":"2024-09-17T03:53:36.529714+0000","last_fullsized":"2024-09-17T03:53:36.529714+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:40.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528473+0000","last_change":"2024-09-17T03:53:35.068226+0000","last_active":"2024-09-17T03:53:36.528473+0000","last_peered":"2024-09-17T03:53:36.528473+0000","last_clean":"2024-09-17T03:53:36.528473+0000","last_became_active":"2024-09-17T03:53:35.067943+0000","last_became_peered":"2024-09-17T03:53:35.067943+0000","last_unstale":"2024-09-17T03:53:36.528473+0000","last_undegraded":"2024-09-17T03:53:36.528473+0000","last_fullsized":"2024-09-17T03:53:36.528473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:09.141910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529817+0000","last_change":"2024-09-17T03:53:34.042295+0000","last_active":"2024-09-17T03:53:36.529817+0000","last_peered":"2024-09-17T03:53:36.529817+0000","last_clean":"2024-09-17T03:53:36.529817+0000","last_became_active":"2024-09-17T03:53:34.041992+0000","last_became_peered":"2024-09-17T03:53:34.041992+0000","last_unstale":"2024-09-17T03:53:36.529817+0000","last_undegraded":"2024-09-17T03:53:36.529817+0000","last_fullsized":"2024-09-17T03:53:36.529817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:16.446587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058700+0000","last_change":"2024-09-17T03:53:35.063084+0000","last_active":"2024-09-17T03:53:36.058700+0000","last_peered":"2024-09-17T03:53:36.058700+0000","last_clean":"2024-09-17T03:53:36.058700+0000","last_became_active":"2024-09-17T03:53:35.062759+0000","last_became_peered":"2024-09-17T03:53:35.062759+0000","last_unstale":"2024-09-17T03:53:36.058700+0000","last_undegraded":"2024-09-17T03:53:36.058700+0000","last_fullsized":"2024-09-17T03:53:36.058700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:22.612393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528628+0000","last_change":"2024-09-17T03:53:34.050809+0000","last_active":"2024-09-17T03:53:36.528628+0000","last_peered":"2024-09-17T03:53:36.528628+0000","last_clean":"2024-09-17T03:53:36.528628+0000","last_became_active":"2024-09-17T03:53:34.050498+0000","last_became_peered":"2024-09-17T03:53:34.050498+0000","last_unstale":"2024-09-17T03:53:36.528628+0000","last_undegraded":"2024-09-17T03:53:36.528628+0000","last_fullsized":"2024-09-17T03:53:36.528628+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:01.191543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527911+0000","last_change":"2024-09-17T03:53:35.059466+0000","last_active":"2024-09-17T03:53:36.527911+0000","last_peered":"2024-09-17T03:53:36.527911+0000","last_clean":"2024-09-17T03:53:36.527911+0000","last_became_active":"2024-09-17T03:53:35.059160+0000","last_became_peered":"2024-09-17T03:53:35.059160+0000","last_unstale":"2024-09-17T03:53:36.527911+0000","last_undegraded":"2024-09-17T03:53:36.527911+0000","last_fullsized":"2024-09-17T03:53:36.527911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:15.257783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.527972+0000","last_change":"2024-09-17T03:53:34.046054+0000","last_active":"2024-09-17T03:53:36.527972+0000","last_peered":"2024-09-17T03:53:36.527972+0000","last_clean":"2024-09-17T03:53:36.527972+0000","last_became_active":"2024-09-17T03:53:34.045718+0000","last_became_peered":"2024-09-17T03:53:34.045718+0000","last_unstale":"2024-09-17T03:53:36.527972+0000","last_undegraded":"2024-09-17T03:53:36.527972+0000","last_fullsized":"2024-09-17T03:53:36.527972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:09.572328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059323+0000","last_change":"2024-09-17T03:53:35.068451+0000","last_active":"2024-09-17T03:53:36.059323+0000","last_peered":"2024-09-17T03:53:36.059323+0000","last_clean":"2024-09-17T03:53:36.059323+0000","last_became_active":"2024-09-17T03:53:35.067500+0000","last_became_peered":"2024-09-17T03:53:35.067500+0000","last_unstale":"2024-09-17T03:53:36.059323+0000","last_undegraded":"2024-09-17T03:53:36.059323+0000","last_fullsized":"2024-09-17T03:53:36.059323+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:10.103551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.528173+0000","last_change":"2024-09-17T03:53:34.045835+0000","last_active":"2024-09-17T03:53:36.528173+0000","last_peered":"2024-09-17T03:53:36.528173+0000","last_clean":"2024-09-17T03:53:36.528173+0000","last_became_active":"2024-09-17T03:53:34.045329+0000","last_became_peered":"2024-09-17T03:53:34.045329+0000","last_unstale":"2024-09-17T03:53:36.528173+0000","last_undegraded":"2024-09-17T03:53:36.528173+0000","last_fullsized":"2024-09-17T03:53:36.528173+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:07.278692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.060312+0000","last_change":"2024-09-17T03:53:35.045510+0000","last_active":"2024-09-17T03:53:36.060312+0000","last_peered":"2024-09-17T03:53:36.060312+0000","last_clean":"2024-09-17T03:53:36.060312+0000","last_became_active":"2024-09-17T03:53:35.044996+0000","last_became_peered":"2024-09-17T03:53:35.044996+0000","last_unstale":"2024-09-17T03:53:36.060312+0000","last_undegraded":"2024-09-17T03:53:36.060312+0000","last_fullsized":"2024-09-17T03:53:36.060312+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:08.285388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"80'2","reported_seq":17,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058140+0000","last_change":"2024-09-17T03:53:34.043457+0000","last_active":"2024-09-17T03:53:36.058140+0000","last_peered":"2024-09-17T03:53:36.058140+0000","last_clean":"2024-09-17T03:53:36.058140+0000","last_became_active":"2024-09-17T03:53:34.043142+0000","last_became_peered":"2024-09-17T03:53:34.043142+0000","last_unstale":"2024-09-17T03:53:36.058140+0000","last_undegraded":"2024-09-17T03:53:36.058140+0000","last_fullsized":"2024-09-17T03:53:36.058140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:14.453153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249615+0000","last_change":"2024-09-17T03:53:35.077390+0000","last_active":"2024-09-17T03:53:36.249615+0000","last_peered":"2024-09-17T03:53:36.249615+0000","last_clean":"2024-09-17T03:53:36.249615+0000","last_became_active":"2024-09-17T03:53:35.077039+0000","last_became_peered":"2024-09-17T03:53:35.077039+0000","last_unstale":"2024-09-17T03:53:36.249615+0000","last_undegraded":"2024-09-17T03:53:36.249615+0000","last_fullsized":"2024-09-17T03:53:36.249615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:36:34.715128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"81'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058032+0000","last_change":"2024-09-17T03:53:34.047344+0000","last_active":"2024-09-17T03:53:36.058032+0000","last_peered":"2024-09-17T03:53:36.058032+0000","last_clean":"2024-09-17T03:53:36.058032+0000","last_became_active":"2024-09-17T03:53:34.041909+0000","last_became_peered":"2024-09-17T03:53:34.041909+0000","last_unstale":"2024-09-17T03:53:36.058032+0000","last_undegraded":"2024-09-17T03:53:36.058032+0000","last_fullsized":"2024-09-17T03:53:36.058032+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:45.691778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179022+0000","last_change":"2024-09-17T03:53:35.060992+0000","last_active":"2024-09-17T03:53:36.179022+0000","last_peered":"2024-09-17T03:53:36.179022+0000","last_clean":"2024-09-17T03:53:36.179022+0000","last_became_active":"2024-09-17T03:53:35.060339+0000","last_became_peered":"2024-09-17T03:53:35.060339+0000","last_unstale":"2024-09-17T03:53:36.179022+0000","last_undegraded":"2024-09-17T03:53:36.179022+0000","last_fullsized":"2024-09-17T03:53:36.179022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:48.433949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178476+0000","last_change":"2024-09-17T03:53:34.042515+0000","last_active":"2024-09-17T03:53:36.178476+0000","last_peered":"2024-09-17T03:53:36.178476+0000","last_clean":"2024-09-17T03:53:36.178476+0000","last_became_active":"2024-09-17T03:53:34.041350+0000","last_became_peered":"2024-09-17T03:53:34.041350+0000","last_unstale":"2024-09-17T03:53:36.178476+0000","last_undegraded":"2024-09-17T03:53:36.178476+0000","last_fullsized":"2024-09-17T03:53:36.178476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:54.121078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.059705+0000","last_change":"2024-09-17T03:53:35.078734+0000","last_active":"2024-09-17T03:53:36.059705+0000","last_peered":"2024-09-17T03:53:36.059705+0000","last_clean":"2024-09-17T03:53:36.059705+0000","last_became_active":"2024-09-17T03:53:35.078432+0000","last_became_peered":"2024-09-17T03:53:35.078432+0000","last_unstale":"2024-09-17T03:53:36.059705+0000","last_undegraded":"2024-09-17T03:53:36.059705+0000","last_fullsized":"2024-09-17T03:53:36.059705+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:20.224088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.179021+0000","last_change":"2024-09-17T03:53:34.043860+0000","last_active":"2024-09-17T03:53:36.179021+0000","last_peered":"2024-09-17T03:53:36.179021+0000","last_clean":"2024-09-17T03:53:36.179021+0000","last_became_active":"2024-09-17T03:53:34.043502+0000","last_became_peered":"2024-09-17T03:53:34.043502+0000","last_unstale":"2024-09-17T03:53:36.179021+0000","last_undegraded":"2024-09-17T03:53:36.179021+0000","last_fullsized":"2024-09-17T03:53:36.179021+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:02.362312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058992+0000","last_change":"2024-09-17T03:53:35.064082+0000","last_active":"2024-09-17T03:53:36.058992+0000","last_peered":"2024-09-17T03:53:36.058992+0000","last_clean":"2024-09-17T03:53:36.058992+0000","last_became_active":"2024-09-17T03:53:35.063675+0000","last_became_peered":"2024-09-17T03:53:35.063675+0000","last_unstale":"2024-09-17T03:53:36.058992+0000","last_undegraded":"2024-09-17T03:53:36.058992+0000","last_fullsized":"2024-09-17T03:53:36.058992+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:11.219352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529388+0000","last_change":"2024-09-17T03:53:34.037735+0000","last_active":"2024-09-17T03:53:36.529388+0000","last_peered":"2024-09-17T03:53:36.529388+0000","last_clean":"2024-09-17T03:53:36.529388+0000","last_became_active":"2024-09-17T03:53:34.037448+0000","last_became_peered":"2024-09-17T03:53:34.037448+0000","last_unstale":"2024-09-17T03:53:36.529388+0000","last_undegraded":"2024-09-17T03:53:36.529388+0000","last_fullsized":"2024-09-17T03:53:36.529388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:52.428087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.529554+0000","last_change":"2024-09-17T03:53:35.071423+0000","last_active":"2024-09-17T03:53:36.529554+0000","last_peered":"2024-09-17T03:53:36.529554+0000","last_clean":"2024-09-17T03:53:36.529554+0000","last_became_active":"2024-09-17T03:53:35.071008+0000","last_became_peered":"2024-09-17T03:53:35.071008+0000","last_unstale":"2024-09-17T03:53:36.529554+0000","last_undegraded":"2024-09-17T03:53:36.529554+0000","last_fullsized":"2024-09-17T03:53:36.529554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:43.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.058428+0000","last_change":"2024-09-17T03:53:34.045408+0000","last_active":"2024-09-17T03:53:36.058428+0000","last_peered":"2024-09-17T03:53:36.058428+0000","last_clean":"2024-09-17T03:53:36.058428+0000","last_became_active":"2024-09-17T03:53:34.045021+0000","last_became_peered":"2024-09-17T03:53:34.045021+0000","last_unstale":"2024-09-17T03:53:36.058428+0000","last_undegraded":"2024-09-17T03:53:36.058428+0000","last_fullsized":"2024-09-17T03:53:36.058428+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:13:48.651492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.248920+0000","last_change":"2024-09-17T03:53:35.070768+0000","last_active":"2024-09-17T03:53:36.248920+0000","last_peered":"2024-09-17T03:53:36.248920+0000","last_clean":"2024-09-17T03:53:36.248920+0000","last_became_active":"2024-09-17T03:53:35.070352+0000","last_became_peered":"2024-09-17T03:53:35.070352+0000","last_unstale":"2024-09-17T03:53:36.248920+0000","last_undegraded":"2024-09-17T03:53:36.248920+0000","last_fullsized":"2024-09-17T03:53:36.248920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:11.336036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178647+0000","last_change":"2024-09-17T03:53:34.041865+0000","last_active":"2024-09-17T03:53:36.178647+0000","last_peered":"2024-09-17T03:53:36.178647+0000","last_clean":"2024-09-17T03:53:36.178647+0000","last_became_active":"2024-09-17T03:53:34.041169+0000","last_became_peered":"2024-09-17T03:53:34.041169+0000","last_unstale":"2024-09-17T03:53:36.178647+0000","last_undegraded":"2024-09-17T03:53:36.178647+0000","last_fullsized":"2024-09-17T03:53:36.178647+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:16.044280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.178489+0000","last_change":"2024-09-17T03:53:35.064940+0000","last_active":"2024-09-17T03:53:36.178489+0000","last_peered":"2024-09-17T03:53:36.178489+0000","last_clean":"2024-09-17T03:53:36.178489+0000","last_became_active":"2024-09-17T03:53:35.064574+0000","last_became_peered":"2024-09-17T03:53:35.064574+0000","last_unstale":"2024-09-17T03:53:36.178489+0000","last_undegraded":"2024-09-17T03:53:36.178489+0000","last_fullsized":"2024-09-17T03:53:36.178489+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:23:08.160089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057951+0000","last_change":"2024-09-17T03:53:34.048127+0000","last_active":"2024-09-17T03:53:36.057951+0000","last_peered":"2024-09-17T03:53:36.057951+0000","last_clean":"2024-09-17T03:53:36.057951+0000","last_became_active":"2024-09-17T03:53:34.047768+0000","last_became_peered":"2024-09-17T03:53:34.047768+0000","last_unstale":"2024-09-17T03:53:36.057951+0000","last_undegraded":"2024-09-17T03:53:36.057951+0000","last_fullsized":"2024-09-17T03:53:36.057951+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:29:17.363867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.057698+0000","last_change":"2024-09-17T03:53:35.051987+0000","last_active":"2024-09-17T03:53:36.057698+0000","last_peered":"2024-09-17T03:53:36.057698+0000","last_clean":"2024-09-17T03:53:36.057698+0000","last_became_active":"2024-09-17T03:53:35.049620+0000","last_became_peered":"2024-09-17T03:53:35.049620+0000","last_unstale":"2024-09-17T03:53:36.057698+0000","last_undegraded":"2024-09-17T03:53:36.057698+0000","last_fullsized":"2024-09-17T03:53:36.057698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:34.014366+0000","last_clean_scrub_stamp":"2024-09-17T03:53:34.014366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:21.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"80'1","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249368+0000","last_change":"2024-09-17T03:53:34.046914+0000","last_active":"2024-09-17T03:53:36.249368+0000","last_peered":"2024-09-17T03:53:36.249368+0000","last_clean":"2024-09-17T03:53:36.249368+0000","last_became_active":"2024-09-17T03:53:34.046056+0000","last_became_peered":"2024-09-17T03:53:34.046056+0000","last_unstale":"2024-09-17T03:53:36.249368+0000","last_undegraded":"2024-09-17T03:53:36.249368+0000","last_fullsized":"2024-09-17T03:53:36.249368+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:53:33.007669+0000","last_clean_scrub_stamp":"2024-09-17T03:53:33.007669+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:53.290335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":219,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-17T03:53:36.249351+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:53:36.249351+0000","last_peered":"2024-09-17T03:53:36.249351+0000","last_clean":"2024-09-17T03:53:36.249351+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:53:36.249351+0000","last_undegraded":"2024-09-17T03:53:36.249351+0000","last_fullsized":"2024-09-17T03:53:36.249351+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509494,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509497,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1003520,"data_stored":417572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509496,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":415938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509494,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":417550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509496,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":416520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509496,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32164,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339676,"statfs":{"total":96636764160,"available":96603828224,"internally_reserved":0,"allocated":999424,"data_stored":416132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542198,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":999424,"data_stored":416632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32740,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339100,"statfs":{"total":96636764160,"available":96603238400,"internally_reserved":0,"allocated":1585152,"data_stored":1006340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:53:44.910 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:53:44.910 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 -- 172.21.15.129:0/4030355457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccb50 msgr2=0x7f520c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 --2- 172.21.15.129:0/4030355457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccb50 0x7f520c0ccf60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5204005fd0 tx=0x7f520402aa90 comp rx=0 tx=0).stop 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 -- 172.21.15.129:0/4030355457 shutdown_connections 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 --2- 172.21.15.129:0/4030355457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccb50 0x7f520c0ccf60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 --2- 172.21.15.129:0/4030355457 >> v2:172.21.15.181:3300/0 conn(0x7f520c0af3c0 0x7f520c0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 -- 172.21.15.129:0/4030355457 >> 172.21.15.129:0/4030355457 conn(0x7f520c06d7d0 msgr2=0x7f520c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 -- 172.21.15.129:0/4030355457 shutdown_connections 2024-09-17T03:53:45.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.174+0000 7f5212e00640 1 -- 172.21.15.129:0/4030355457 wait complete. 2024-09-17T03:53:45.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.177+0000 7f5212e00640 1 Processor -- start 2024-09-17T03:53:45.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.177+0000 7f5212e00640 1 -- start start 2024-09-17T03:53:45.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0af3c0 0x7f520c1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c0ccb50 0x7f520c15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f520c153520 con 0x7f520c0cc1e0 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1533a0 con 0x7f520c0ccb50 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f5212e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f520c1536a0 con 0x7f520c0af3c0 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.178+0000 7f520b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38916/0 (socket says 172.21.15.129:38916) 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 -- 172.21.15.129:0/1196814607 learned_addr learned my addr 172.21.15.129:0/1196814607 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520be00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0af3c0 0x7f520c1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f5210e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c0ccb50 0x7f520c15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 -- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0af3c0 msgr2=0x7f520c1527a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0af3c0 0x7f520c1527a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 -- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c0ccb50 msgr2=0x7f520c15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c0ccb50 0x7f520c15e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 -- 172.21.15.129:0/1196814607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c153070 con 0x7f520c0cc1e0 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f520b400640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e9f0 tx=0x7f51f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.179+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800cd80 con 0x7f520c0cc1e0 2024-09-17T03:53:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.180+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51f8004020 con 0x7f520c0cc1e0 2024-09-17T03:53:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.180+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c1f4bd0 con 0x7f520c0cc1e0 2024-09-17T03:53:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.180+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004470 con 0x7f520c0cc1e0 2024-09-17T03:53:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.180+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1f4f80 con 0x7f520c0cc1e0 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.181+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c1e7bd0 con 0x7f520c0cc1e0 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.182+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f51f80035f0 con 0x7f520c0cc1e0 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.184+0000 7f5203e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 0x7f51ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.184+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f51f809b7e0 con 0x7f520c0cc1e0 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.184+0000 7f520be00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 0x7f51ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.185+0000 7f520be00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 0x7f51ec07b0a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0015e0 tx=0x7f51fc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:45.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.188+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f51f8063b80 con 0x7f520c0cc1e0 2024-09-17T03:53:45.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.439+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f520c0ccf60 con 0x7f520c0cc1e0 2024-09-17T03:53:45.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.444+0000 7f5203e00640 1 -- 172.21.15.129:0/1196814607 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v31) ==== 190+0+147 (secure 0 0 0) 0x7f520c0ccf60 con 0x7f520c0cc1e0 2024-09-17T03:53:45.445 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:53:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:53:45.445 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:53:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:53:45.446 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:53:45.446 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 msgr2=0x7f51ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 0x7f51ec07b0a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0015e0 tx=0x7f51fc009040 comp rx=0 tx=0).stop 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 msgr2=0x7f520c1f4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e9f0 tx=0x7f51f800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 shutdown_connections 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078b90 0x7f51ec07b0a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c0ccb50 0x7f520c15e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cc1e0 0x7f520c1f4690 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 --2- 172.21.15.129:0/1196814607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0af3c0 0x7f520c1527a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 >> 172.21.15.129:0/1196814607 conn(0x7f520c06d7d0 msgr2=0x7f520c0b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 shutdown_connections 2024-09-17T03:53:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.447+0000 7f5212e00640 1 -- 172.21.15.129:0/1196814607 wait complete. 2024-09-17T03:53:45.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:45.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 -- 172.21.15.129:0/285608553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070030 msgr2=0x7fb988074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 --2- 172.21.15.129:0/285608553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070030 0x7fb988074640 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fb9880b4b40 tx=0x7fb97002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 -- 172.21.15.129:0/285608553 shutdown_connections 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 --2- 172.21.15.129:0/285608553 >> v2:172.21.15.181:3300/0 conn(0x7fb988070440 0x7fb988074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 --2- 172.21.15.129:0/285608553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070030 0x7fb988074640 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 -- 172.21.15.129:0/285608553 >> 172.21.15.129:0/285608553 conn(0x7fb9880b1cb0 msgr2=0x7fb9880b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 -- 172.21.15.129:0/285608553 shutdown_connections 2024-09-17T03:53:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.689+0000 7fb98ea00640 1 -- 172.21.15.129:0/285608553 wait complete. 2024-09-17T03:53:45.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.691+0000 7fb98ea00640 1 Processor -- start 2024-09-17T03:53:45.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.691+0000 7fb98ea00640 1 -- start start 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb988070030 0x7fb98814ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 0x7fb9881f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880b39f0 con 0x7fb988070440 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880b3870 con 0x7fb988073170 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ea00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb9880b3b70 con 0x7fb988070030 2024-09-17T03:53:45.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb97f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb97f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38934/0 (socket says 172.21.15.129:38934) 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb97f400640 1 -- 172.21.15.129:0/4151505972 learned_addr learned my addr 172.21.15.129:0/4151505972 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb98ca00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 0x7fb9881f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.693+0000 7fb987e00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb988070030 0x7fb98814ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 -- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb988070030 msgr2=0x7fb98814ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb988070030 0x7fb98814ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 -- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 msgr2=0x7fb9881f9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 0x7fb9881f9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 -- 172.21.15.129:0/4151505972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9881f3b60 con 0x7fb988070440 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb98ca00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 0x7fb9881f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb97f400640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb970034040 tx=0x7fb970003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb97003e030 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb970003e00 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb98814fa60 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb970035040 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.694+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9880cf530 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.695+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98814f7d0 con 0x7fb988070440 2024-09-17T03:53:45.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.697+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb970041030 con 0x7fb988070440 2024-09-17T03:53:45.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.698+0000 7fb984c00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 0x7fb96407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:45.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.698+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb9700bb5b0 con 0x7fb988070440 2024-09-17T03:53:45.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.699+0000 7fb987e00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 0x7fb96407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:45.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.699+0000 7fb987e00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 0x7fb96407b0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb9780015e0 tx=0x7fb978009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.703+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb970083950 con 0x7fb988070440 2024-09-17T03:53:45.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.922+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9880cf070 con 0x7fb988070440 2024-09-17T03:53:45.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.923+0000 7fb984c00640 1 -- 172.21.15.129:0/4151505972 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7fb9880cf070 con 0x7fb988070440 2024-09-17T03:53:45.923 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:45.923 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:45.923 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 msgr2=0x7fb96407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 0x7fb96407b0d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb9780015e0 tx=0x7fb978009040 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 msgr2=0x7fb9881f9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fb970034040 tx=0x7fb970003910 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 shutdown_connections 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb964078bc0 0x7fb96407b0d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb988073170 0x7fb9881f9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb988070440 0x7fb9881f9510 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 --2- 172.21.15.129:0/4151505972 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb988070030 0x7fb98814ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.925+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 >> 172.21.15.129:0/4151505972 conn(0x7fb9880b1cb0 msgr2=0x7fb9880cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.926+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 shutdown_connections 2024-09-17T03:53:45.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:45.926+0000 7fb98ea00640 1 -- 172.21.15.129:0/4151505972 wait complete. 2024-09-17T03:53:46.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:53:34.515705+0000', 'modified': '2024-09-17T03:53:37.544115+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6049': {'gid': 6049, 'name': 'c', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/915087149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 915087149}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 915087149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6049, 'qdb_cluster': [6049]} 2024-09-17T03:53:46.351 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:53:46.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:53:46.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:53:46.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:53:46.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.578+0000 7fd293800640 1 -- 172.21.15.129:0/822516588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd274004800 msgr2=0x7fd274004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.578+0000 7fd293800640 1 --2- 172.21.15.129:0/822516588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd274004800 0x7fd274004c10 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd278009b50 tx=0x7fd274005830 comp rx=0 tx=0).stop 2024-09-17T03:53:46.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.578+0000 7fd293800640 1 -- 172.21.15.129:0/822516588 shutdown_connections 2024-09-17T03:53:46.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.579+0000 7fd293800640 1 --2- 172.21.15.129:0/822516588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd274004800 0x7fd274004c10 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.579+0000 7fd293800640 1 -- 172.21.15.129:0/822516588 >> 172.21.15.129:0/822516588 conn(0x7fd28c06d7d0 msgr2=0x7fd28c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.579+0000 7fd293800640 1 -- 172.21.15.129:0/822516588 shutdown_connections 2024-09-17T03:53:46.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.579+0000 7fd293800640 1 -- 172.21.15.129:0/822516588 wait complete. 2024-09-17T03:53:46.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.581+0000 7fd293800640 1 Processor -- start 2024-09-17T03:53:46.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.581+0000 7fd293800640 1 -- start start 2024-09-17T03:53:46.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd274004800 0x7fd28c148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd28c09aa30 0x7fd28c1580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c15dc30 con 0x7fd28c156bb0 2024-09-17T03:53:46.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c15dab0 con 0x7fd274004800 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.582+0000 7fd293800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c15ddb0 con 0x7fd28c09aa30 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38946/0 (socket says 172.21.15.129:38946) 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 -- 172.21.15.129:0/1189456787 learned_addr learned my addr 172.21.15.129:0/1189456787 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd290e00640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd274004800 0x7fd28c148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd28be00640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd28c09aa30 0x7fd28c1580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 -- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd28c09aa30 msgr2=0x7fd28c1580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd28c09aa30 0x7fd28c1580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 -- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd274004800 msgr2=0x7fd28c148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd274004800 0x7fd28c148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 -- 172.21.15.129:0/1189456787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c149100 con 0x7fd28c156bb0 2024-09-17T03:53:46.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.583+0000 7fd291800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0cd710 tx=0x7fd284006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.584+0000 7fd288c00640 1 -- 172.21.15.129:0/1189456787 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28400fc90 con 0x7fd28c156bb0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.584+0000 7fd288c00640 1 -- 172.21.15.129:0/1189456787 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd284018070 con 0x7fd28c156bb0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.584+0000 7fd288c00640 1 -- 172.21.15.129:0/1189456787 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284011690 con 0x7fd28c156bb0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.584+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c155b10 con 0x7fd28c156bb0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.584+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c1560d0 con 0x7fd28c156bb0 2024-09-17T03:53:46.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.586+0000 7fd288c00640 1 -- 172.21.15.129:0/1189456787 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd284011830 con 0x7fd28c156bb0 2024-09-17T03:53:46.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.587+0000 7fd288c00640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 0x7fd26c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.587+0000 7fd288c00640 1 -- 172.21.15.129:0/1189456787 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fd28409b460 con 0x7fd28c156bb0 2024-09-17T03:53:46.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.588+0000 7fd29a9c0740 1 Processor -- start 2024-09-17T03:53:46.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.588+0000 7fd290e00640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 0x7fd26c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.588+0000 7fd29a9c0740 1 -- start start 2024-09-17T03:53:46.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.588+0000 7fd290e00640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 0x7fd26c07b0a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd280000c00 tx=0x7fd280006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfed5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfea5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55afdfe2f430 con 0x55afdfb05270 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55afdfe1f5d0 con 0x55afdfadc3b0 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55afdfe0ea00 con 0x55afdfae4590 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd290e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd290e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38960/0 (socket says 172.21.15.129:38960) 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.589+0000 7fd290e00640 1 -- 172.21.15.129:0/2893602761 learned_addr learned my addr 172.21.15.129:0/2893602761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd28be00640 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfed5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd291800640 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfea5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 -- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 msgr2=0x55afdfea5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfea5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 -- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 msgr2=0x55afdfed5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfed5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 -- 172.21.15.129:0/2893602761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afdfdd7520 con 0x55afdfb05270 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd290e00640 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd2800129b0 tx=0x7fd280015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6c37160d7c0264 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd27d600640 1 -- 172.21.15.129:0/2893602761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28000f960 con 0x55afdfb05270 2024-09-17T03:53:46.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd27d600640 1 -- 172.21.15.129:0/2893602761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd280024020 con 0x55afdfb05270 2024-09-17T03:53:46.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.590+0000 7fd27d600640 1 -- 172.21.15.129:0/2893602761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28000fd20 con 0x55afdfb05270 2024-09-17T03:53:46.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2893602761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 msgr2=0x55afdfed5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fd2800129b0 tx=0x7fd280015000 comp rx=0 tx=0).stop 2024-09-17T03:53:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2893602761 shutdown_connections 2024-09-17T03:53:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfea5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfed5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2893602761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdfed5970 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2893602761 >> 172.21.15.129:0/2893602761 conn(0x55afdfaf93c0 msgr2=0x55afdfeab970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2893602761 shutdown_connections 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.591+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2893602761 wait complete. 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.592+0000 7fd29a9c0740 1 -- start start 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.593+0000 7fd29a9c0740 1 Processor -- start 2024-09-17T03:53:46.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfed2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfb05270 0x55afdfed3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55afdfdb5330 con 0x55afdfadc3b0 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55afdfe12810 con 0x55afdfae4590 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55afdfb7f620 con 0x55afdfb05270 2024-09-17T03:53:46.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd29a9c0740 10 client.0 _ll_register_callbacks cb 0x55afdfdba420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd290e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd290e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38974/0 (socket says 172.21.15.129:38974) 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.594+0000 7fd290e00640 1 -- 172.21.15.129:0/2567881270 learned_addr learned my addr 172.21.15.129:0/2567881270 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd28be00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfed2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd291800640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfb05270 0x55afdfed3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 -- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfb05270 msgr2=0x55afdfed3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfb05270 0x55afdfed3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 -- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 msgr2=0x55afdfed2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfed2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 -- 172.21.15.129:0/2567881270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afdfb70850 con 0x55afdfadc3b0 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd290e00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd280006500 tx=0x7fd28000fd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72a9b537fd00dd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd272000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.595+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280023070 con 0x55afdfadc3b0 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fd280029840 con 0x55afdfadc3b0 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd280024020 con 0x55afdfadc3b0 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd28003b050 con 0x55afdfadc3b0 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 10 client.6369 fetch_fsmap learned FSMap version 54 2024-09-17T03:53:46.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) -- 0x55afdfbe76b0 con 0x55afdfadc3b0 2024-09-17T03:53:46.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd280024020 con 0x55afdfadc3b0 2024-09-17T03:53:46.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 3502+0+0 (secure 0 0 0) 0x7fd28003e020 con 0x55afdfadc3b0 2024-09-17T03:53:46.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 10 client.6369 fetch_fsmap finished waiting for FSMap version 54 2024-09-17T03:53:46.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 10 client.6369 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:53:46.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 20 client.6369 populate_metadata read hostname 'smithi129' 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfac3e40 0x55afdfecb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 4 client.6369 mds_command: new command op to 6049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.596+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 --> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afdfe21520 con 0x55afdfac3e40 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.597+0000 7fd28be00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfac3e40 0x55afdfecb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.597+0000 7fd28be00640 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfac3e40 0x55afdfecb7f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=229d6bb58f9fa9c5 server_cookie=f2eb7b0db80d875c in_seq=0 out_seq=0 2024-09-17T03:53:46.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.597+0000 7fd272000640 10 client.6369 ms_handle_connect on v2:172.21.15.129:6833/915087149 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd272000640 1 -- 172.21.15.129:0/2567881270 <== mds.0 v2:172.21.15.129:6833/915087149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55afdfe21520 con 0x55afdfac3e40 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd272000640 10 client.6369 handle_command_reply: tid=0 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 client.6369 shutdown 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 msgr2=0x55afdfece970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fd280006500 tx=0x7fd28000fd50 comp rx=0 tx=0).stop 2024-09-17T03:53:46.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 shutdown_connections 2024-09-17T03:53:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfb05270 0x55afdfed3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfac3e40 0x55afdfecb7f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfed2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/2567881270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfadc3b0 0x55afdfece970 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd272000640 0 client.6369 ms_handle_reset on v2:172.21.15.129:6833/915087149 2024-09-17T03:53:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.598+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 >> 172.21.15.129:0/2567881270 conn(0x55afdfaf93c0 msgr2=0x55afdfea5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.599+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 shutdown_connections 2024-09-17T03:53:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.599+0000 7fd29a9c0740 1 -- 172.21.15.129:0/2567881270 wait complete. 2024-09-17T03:53:46.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.599+0000 7fd29a9c0740 20 client.6369 trim_cache size 0 max 16384 2024-09-17T03:53:46.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.617+0000 7fd29a9c0740 1 Processor -- start 2024-09-17T03:53:46.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.618+0000 7fd29a9c0740 1 -- start start 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfee87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfee8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c09b490 con 0x55afdfb05270 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c1f4440 con 0x55afdfadc3b0 2024-09-17T03:53:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c09ae40 con 0x55afdfae4590 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38982/0 (socket says 172.21.15.129:38982) 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.619+0000 7fd291800640 1 -- 172.21.15.129:0/1917016226 learned_addr learned my addr 172.21.15.129:0/1917016226 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd290e00640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfee87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd28be00640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfee8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 -- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 msgr2=0x55afdfee8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfee8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 -- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 msgr2=0x55afdfee87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfee87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 -- 172.21.15.129:0/1917016226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afdfeac650 con 0x55afdfb05270 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd290e00640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfee87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:46.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd28be00640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfee8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd291800640 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd284060420 tx=0x7fd284067040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5dfc46f7450749 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd27cc00640 1 -- 172.21.15.129:0/1917016226 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284062cc0 con 0x55afdfb05270 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd27cc00640 1 -- 172.21.15.129:0/1917016226 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd28404e030 con 0x55afdfb05270 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.620+0000 7fd27cc00640 1 -- 172.21.15.129:0/1917016226 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28406c8b0 con 0x55afdfb05270 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1917016226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 msgr2=0x55afdff0a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fd284060420 tx=0x7fd284067040 comp rx=0 tx=0).stop 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1917016226 shutdown_connections 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff0a1d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfae4590 0x55afdfee8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1917016226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfadc3b0 0x55afdfee87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1917016226 >> 172.21.15.129:0/1917016226 conn(0x55afdfaf93c0 msgr2=0x55afdfea7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1917016226 shutdown_connections 2024-09-17T03:53:46.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.621+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1917016226 wait complete. 2024-09-17T03:53:46.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.622+0000 7fd29a9c0740 1 -- start start 2024-09-17T03:53:46.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.622+0000 7fd29a9c0740 1 Processor -- start 2024-09-17T03:53:46.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfadc3b0 0x55afdff69880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfeffff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55afdfed2490 con 0x55afdfb05270 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55afdfed2090 con 0x55afdfae4590 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55afdfecfa60 con 0x55afdfadc3b0 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd29a9c0740 10 client.0 _ll_register_callbacks cb 0x7fd280012880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38988/0 (socket says 172.21.15.129:38988) 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 -- 172.21.15.129:0/1119616637 learned_addr learned my addr 172.21.15.129:0/1119616637 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd28be00640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfeffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd290e00640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfadc3b0 0x55afdff69880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 -- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfadc3b0 msgr2=0x55afdff69880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfadc3b0 0x55afdff69880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 -- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 msgr2=0x55afdfeffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfeffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.624+0000 7fd291800640 1 -- 172.21.15.129:0/1119616637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afdff03290 con 0x55afdfb05270 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd291800640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd284063140 tx=0x7fd28406ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0c7cd25d7be8182 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284081020 con 0x55afdfb05270 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fd284011b20 con 0x55afdfb05270 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd28404e030 con 0x55afdfb05270 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) ==== 24+0+0 (secure 0 0 0) 0x7fd284062cc0 con 0x55afdfb05270 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd29a9c0740 10 client.6375 fetch_fsmap learned FSMap version 54 2024-09-17T03:53:46.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=2+}) -- 0x55afdff03850 con 0x55afdfb05270 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd28404e030 con 0x55afdfb05270 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.625+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 54) ==== 3502+0+0 (secure 0 0 0) 0x7fd28404aaf0 con 0x55afdfb05270 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 10 client.6375 fetch_fsmap finished waiting for FSMap version 54 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 10 client.6375 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 20 client.6375 populate_metadata read hostname 'smithi129' 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfef0f60 0x55afdff72e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 4 client.6375 mds_command: new command op to 6049 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 --> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] -- command(tid 0: {"prefix": "status"}) -- 0x55afdfe1cfd0 con 0x55afdfef0f60 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd290e00640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfef0f60 0x55afdff72e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd290e00640 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfef0f60 0x55afdff72e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6941dbbea07db4c7 server_cookie=22ca1e887c0d53f1 in_seq=0 out_seq=0 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.626+0000 7fd272000640 10 client.6375 ms_handle_connect on v2:172.21.15.129:6833/915087149 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd272000640 1 -- 172.21.15.129:0/1119616637 <== mds.0 v2:172.21.15.129:6833/915087149 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55afdfe1cfd0 con 0x55afdfef0f60 2024-09-17T03:53:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd272000640 10 client.6375 handle_command_reply: tid=0 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 client.6375 shutdown 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 msgr2=0x55afdff70160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd284063140 tx=0x7fd28406ca50 comp rx=0 tx=0).stop 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 shutdown_connections 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55afdfb05270 0x55afdff70160 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.129:6833/915087149,v1:172.21.15.129:6835/915087149] conn(0x55afdfef0f60 0x55afdff72e40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55afdfae4590 0x55afdfeffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 --2- 172.21.15.129:0/1119616637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55afdfadc3b0 0x55afdff69880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd272000640 0 client.6375 ms_handle_reset on v2:172.21.15.129:6833/915087149 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 >> 172.21.15.129:0/1119616637 conn(0x55afdfaf93c0 msgr2=0x55afdfee87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 shutdown_connections 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 1 -- 172.21.15.129:0/1119616637 wait complete. 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.627+0000 7fd29a9c0740 20 client.6375 trim_cache size 0 max 16384 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:53:46.636 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 6049, 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.591595469, 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 54, 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 81, 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 81, 2024-09-17T03:53:46.637 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.620240901999999 2024-09-17T03:53:46.638 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:53:46.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 msgr2=0x7fd26c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 0x7fd26c07b0a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd280000c00 tx=0x7fd280006b70 comp rx=0 tx=0).stop 2024-09-17T03:53:46.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 msgr2=0x7fd28c1f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0cd710 tx=0x7fd284006720 comp rx=0 tx=0).stop 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 shutdown_connections 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd26c078b90 0x7fd26c07b0a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd28c156bb0 0x7fd28c1f7040 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd28c09aa30 0x7fd28c1580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 --2- 172.21.15.129:0/1189456787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd274004800 0x7fd28c148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 >> 172.21.15.129:0/1189456787 conn(0x7fd28c06d7d0 msgr2=0x7fd28c15eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 shutdown_connections 2024-09-17T03:53:46.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.628+0000 7fd293800640 1 -- 172.21.15.129:0/1189456787 wait complete. 2024-09-17T03:53:46.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 -- 172.21.15.129:0/3118172028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c06f0a0 msgr2=0x7fa20c0b7950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 --2- 172.21.15.129:0/3118172028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c06f0a0 0x7fa20c0b7950 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8005fd0 tx=0x7fa1f802a670 comp rx=0 tx=0).stop 2024-09-17T03:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 -- 172.21.15.129:0/3118172028 shutdown_connections 2024-09-17T03:53:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 --2- 172.21.15.129:0/3118172028 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c06f0a0 0x7fa20c0b7950 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 -- 172.21.15.129:0/3118172028 >> 172.21.15.129:0/3118172028 conn(0x7fa20c0cec00 msgr2=0x7fa20c0cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:46.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 -- 172.21.15.129:0/3118172028 shutdown_connections 2024-09-17T03:53:46.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.876+0000 7fa212a00640 1 -- 172.21.15.129:0/3118172028 wait complete. 2024-09-17T03:53:46.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.878+0000 7fa212a00640 1 Processor -- start 2024-09-17T03:53:46.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.878+0000 7fa212a00640 1 -- start start 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa20c14fe20 0x7fa20c1f88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa20c1f8df0 0x7fa20c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c0d0f50 con 0x7fa20c0b1190 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa20c0d0dd0 con 0x7fa20c14fe20 2024-09-17T03:53:46.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa212a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa20c0d10d0 con 0x7fa20c1f8df0 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39002/0 (socket says 172.21.15.129:39002) 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 -- 172.21.15.129:0/1212945215 learned_addr learned my addr 172.21.15.129:0/1212945215 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 -- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa20c1f8df0 msgr2=0x7fa20c0b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa20c1f8df0 0x7fa20c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.880+0000 7fa211600640 1 -- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa20c14fe20 msgr2=0x7fa20c1f88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa211600640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa20c14fe20 0x7fa20c1f88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa211600640 1 -- 172.21.15.129:0/1212945215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20c0cc120 con 0x7fa20c0b1190 2024-09-17T03:53:46.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa211600640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa20000c3f0 tx=0x7fa20000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200017070 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2000035f0 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20c1f9200 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200003a40 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.881+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa20c0b3960 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.882+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa20c1630e0 con 0x7fa20c0b1190 2024-09-17T03:53:46.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.884+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa20001a070 con 0x7fa20c0b1190 2024-09-17T03:53:46.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.885+0000 7fa209600640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 0x7fa1ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:46.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.885+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fa20009a150 con 0x7fa20c0b1190 2024-09-17T03:53:46.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.885+0000 7fa210c00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 0x7fa1ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:46.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.886+0000 7fa210c00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 0x7fa1ec07b0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8033000 tx=0x7fa1f80046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:46.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:46.889+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa20c1f9200 con 0x7fa20c0b1190 2024-09-17T03:53:47.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.109+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa20c0cf9f0 con 0x7fa20c0b1190 2024-09-17T03:53:47.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.110+0000 7fa209600640 1 -- 172.21.15.129:0/1212945215 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v54) ==== 53+0+73 (secure 0 0 0) 0x7fa2000624f0 con 0x7fa20c0b1190 2024-09-17T03:53:47.110 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 msgr2=0x7fa1ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 0x7fa1ec07b0a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8033000 tx=0x7fa1f80046c0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 msgr2=0x7fa20c0cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa20000c3f0 tx=0x7fa20000c8b0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 shutdown_connections 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1ec078b90 0x7fa1ec07b0a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa20c1f8df0 0x7fa20c0b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa20c14fe20 0x7fa20c1f88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 --2- 172.21.15.129:0/1212945215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa20c0b1190 0x7fa20c0cbbe0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:47.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 >> 172.21.15.129:0/1212945215 conn(0x7fa20c0cec00 msgr2=0x7fa20c0b0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:47.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 shutdown_connections 2024-09-17T03:53:47.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47.112+0000 7fa212a00640 1 -- 172.21.15.129:0/1212945215 wait complete. 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:53:47.135 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:53:47.136 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:53:47.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.136 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.172 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42465sec preferred_lft 42465sec 2024-09-17T03:53:47.173 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:53:47.174 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.174 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:53:47.174 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:53:47.175 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:53:47.175 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.175 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:53:47.175 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:53:47.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.177 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:53:47.265 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:53:47.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.268 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:53:47.305 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:53:47.305 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:53:47.305 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:53:47.305 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:53:47.305 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:53:47.305 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:53:47.306 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:53:47.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:53:47.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:53:47.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:53:47.306 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:53:47.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:53:47.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:53:47.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.501 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:53:47.536 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:53:47.536 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:53:47.537 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:53:47.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:53:47.633 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:53:47.633 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:47.669 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:53:47.669 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:53:47.670 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:47.744 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:53:47.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:53:47.937 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:53:47.938 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:53:47.938 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:53:47.958 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:53:47.958 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:53:47.958 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:53:47.958 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:53:47.958 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:53:47.992 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:53:47.993 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:53:47.994 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:53:47.994 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:53:47.994 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:53:47.994 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:53:47.994 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:53:47.997 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:53:47.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:47.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:48.053 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:53:48.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:53:48.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:53:48.151 INFO:teuthology.orchestra.run.smithi129.stdout:client6205 2024-09-17T03:53:48.151 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:48.151 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:48.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:48.187 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch fancy_layout_file) 2024-09-17T03:53:48.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:48.268 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout -v 'stripe_unit=1048576 stripe_count=4 object_size=8388608' fancy_layout_file) 2024-09-17T03:53:48.365 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:48.365 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:48.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:48.403 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=fancy_layout_file bs=1M conv=fdatasync count=35 seek=0) 2024-09-17T03:53:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:35+0 records in 2024-09-17T03:53:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:35+0 records out 2024-09-17T03:53:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:36700160 bytes (37 MB, 35 MiB) copied, 0.300737 s, 122 MB/s 2024-09-17T03:53:48.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:48.783 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f fancy_layout_file) 2024-09-17T03:53:48.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 -- 172.21.15.129:0/3878430330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980cc1e0 msgr2=0x7fd6980af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 --2- 172.21.15.129:0/3878430330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980cc1e0 0x7fd6980af3c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd688009830 tx=0x7fd68802a510 comp rx=0 tx=0).stop 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 -- 172.21.15.129:0/3878430330 shutdown_connections 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 --2- 172.21.15.129:0/3878430330 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6980ccb50 0x7fd6980ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 --2- 172.21.15.129:0/3878430330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980cc1e0 0x7fd6980af3c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 -- 172.21.15.129:0/3878430330 >> 172.21.15.129:0/3878430330 conn(0x7fd69806d7d0 msgr2=0x7fd6980b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 -- 172.21.15.129:0/3878430330 shutdown_connections 2024-09-17T03:53:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.088+0000 7fd69d800640 1 -- 172.21.15.129:0/3878430330 wait complete. 2024-09-17T03:53:49.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.091+0000 7fd69d800640 1 Processor -- start 2024-09-17T03:53:49.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.091+0000 7fd69d800640 1 -- start start 2024-09-17T03:53:49.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 0x7fd69815e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd69815d8a0 0x7fd6980b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981533a0 con 0x7fd6980cc1e0 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd698153520 con 0x7fd6980ccb50 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd69d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd6981536a0 con 0x7fd69815d8a0 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd696a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd696a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39014/0 (socket says 172.21.15.129:39014) 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.092+0000 7fd696a00640 1 -- 172.21.15.129:0/2336929847 learned_addr learned my addr 172.21.15.129:0/2336929847 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd697400640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd69815d8a0 0x7fd6980b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696000640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 0x7fd69815e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 -- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd69815d8a0 msgr2=0x7fd6980b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd69815d8a0 0x7fd6980b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 -- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 msgr2=0x7fd69815e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 0x7fd69815e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 -- 172.21.15.129:0/2336929847 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6981f4980 con 0x7fd6980cc1e0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696000640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 0x7fd69815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd696a00640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd68802ab00 tx=0x7fd688033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688005c90 con 0x7fd6980cc1e0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6981f4c60 con 0x7fd6980cc1e0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd688031e50 con 0x7fd6980cc1e0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.093+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688005260 con 0x7fd6980cc1e0 2024-09-17T03:53:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.094+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69815e3b0 con 0x7fd6980cc1e0 2024-09-17T03:53:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.095+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69815dec0 con 0x7fd6980cc1e0 2024-09-17T03:53:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.096+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd688005400 con 0x7fd6980cc1e0 2024-09-17T03:53:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.097+0000 7fd68ea00640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 0x7fd67807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.097+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fd6880bbdd0 con 0x7fd6980cc1e0 2024-09-17T03:53:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.097+0000 7fd696000640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 0x7fd67807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.098+0000 7fd696000640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 0x7fd67807b0a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd698157340 tx=0x7fd684009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:49.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.101+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd68803f030 con 0x7fd6980cc1e0 2024-09-17T03:53:49.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.321+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd698153bc0 con 0x7fd6980cc1e0 2024-09-17T03:53:49.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.322+0000 7fd68ea00640 1 -- 172.21.15.129:0/2336929847 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7fd688084180 con 0x7fd6980cc1e0 2024-09-17T03:53:49.322 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:49.322 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:49.322 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:49.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 msgr2=0x7fd67807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 0x7fd67807b0a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd698157340 tx=0x7fd684009040 comp rx=0 tx=0).stop 2024-09-17T03:53:49.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 msgr2=0x7fd69809aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd68802ab00 tx=0x7fd688033c20 comp rx=0 tx=0).stop 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 shutdown_connections 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd678078b90 0x7fd67807b0a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd69815d8a0 0x7fd6980b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6980ccb50 0x7fd69815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 --2- 172.21.15.129:0/2336929847 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6980cc1e0 0x7fd69809aa30 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 >> 172.21.15.129:0/2336929847 conn(0x7fd69806d7d0 msgr2=0x7fd69814dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 shutdown_connections 2024-09-17T03:53:49.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.324+0000 7fd69d800640 1 -- 172.21.15.129:0/2336929847 wait complete. 2024-09-17T03:53:49.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:53:49.597 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:53:49.598 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:53:49.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.815+0000 7f9e38c00640 1 -- 172.21.15.129:0/759227039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 msgr2=0x7f9e34074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.815+0000 7f9e38c00640 1 --2- 172.21.15.129:0/759227039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e34074640 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f9e34070e70 tx=0x7f9e2002a4d0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.815+0000 7f9e38c00640 1 -- 172.21.15.129:0/759227039 shutdown_connections 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.815+0000 7f9e38c00640 1 --2- 172.21.15.129:0/759227039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e34074640 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.815+0000 7f9e38c00640 1 --2- 172.21.15.129:0/759227039 >> v2:172.21.15.181:3300/0 conn(0x7f9e34073170 0x7f9e3415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.816+0000 7f9e38c00640 1 -- 172.21.15.129:0/759227039 >> 172.21.15.129:0/759227039 conn(0x7f9e340b1cb0 msgr2=0x7f9e340b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:49.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.816+0000 7f9e38c00640 1 -- 172.21.15.129:0/759227039 shutdown_connections 2024-09-17T03:53:49.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.816+0000 7f9e38c00640 1 -- 172.21.15.129:0/759227039 wait complete. 2024-09-17T03:53:49.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.818+0000 7f9e38c00640 1 Processor -- start 2024-09-17T03:53:49.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.818+0000 7f9e38c00640 1 -- start start 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e34070440 0x7f9e3414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e34073170 0x7f9e341f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e34075050 con 0x7f9e34070030 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e34074ed0 con 0x7f9e34070440 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.819+0000 7f9e38c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e3406f5a0 con 0x7f9e34073170 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39046/0 (socket says 172.21.15.129:39046) 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 -- 172.21.15.129:0/1432738258 learned_addr learned my addr 172.21.15.129:0/1432738258 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31400640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e34070440 0x7f9e3414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e32800640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e34073170 0x7f9e341f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 -- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e34073170 msgr2=0x7f9e341f9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e34073170 0x7f9e341f9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 -- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e34070440 msgr2=0x7f9e3414f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e34070440 0x7f9e3414f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 -- 172.21.15.129:0/1432738258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e34152800 con 0x7f9e34070030 2024-09-17T03:53:49.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.820+0000 7f9e31e00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d700 tx=0x7f9e2400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.821+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24017070 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.821+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e24005310 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.821+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e3409c0c0 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.821+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24009820 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.821+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e3409c680 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.822+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e3406fac0 con 0x7f9e34070030 2024-09-17T03:53:49.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.823+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e240099c0 con 0x7f9e34070030 2024-09-17T03:53:49.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.824+0000 7f9e2a000640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 0x7f9e1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:49.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.824+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f9e2409ae70 con 0x7f9e34070030 2024-09-17T03:53:49.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.825+0000 7f9e31400640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 0x7f9e1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:49.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.826+0000 7f9e31400640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 0x7f9e1407b0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9e20033000 tx=0x7f9e20004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:49.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:49.829+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e24062bc0 con 0x7f9e34070030 2024-09-17T03:53:50.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.084+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f9e340c3860 con 0x7f9e34070030 2024-09-17T03:53:50.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.085+0000 7f9e2a000640 1 -- 172.21.15.129:0/1432738258 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1653 (secure 0 0 0) 0x7f9e240675f0 con 0x7f9e34070030 2024-09-17T03:53:50.085 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:50.086 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:53:50.086 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:53:50.086 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T03:53:50.086 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772829458432, 2024-09-17T03:53:50.086 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 264654848, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 264654848, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00034233200130984187, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:50.087 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T03:53:50.088 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T03:53:50.088 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T03:53:50.088 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772829458432, 2024-09-17T03:53:50.088 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 264654848, 2024-09-17T03:53:50.088 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 264654848, 2024-09-17T03:53:50.089 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00034233200130984187 2024-09-17T03:53:50.089 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:50.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:50.089 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T03:53:50.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T03:53:50.090 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T03:53:50.091 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T03:53:50.091 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6179307067432092e-06, 2024-09-17T03:53:50.091 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367085584384 2024-09-17T03:53:50.091 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:50.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:50.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:50.092 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T03:53:50.092 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 12, 2024-09-17T03:53:50.092 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 2410, 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 22, 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 64, 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 65536, 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 8.9265277836148016e-08, 2024-09-17T03:53:50.093 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367085584384 2024-09-17T03:53:50.094 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:50.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:53:50.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:53:50.094 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T03:53:50.094 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 13, 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 0, 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 0, 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 0, 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 0, 2024-09-17T03:53:50.095 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 0, 2024-09-17T03:53:50.096 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367085584384 2024-09-17T03:53:50.096 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:50.096 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:53:50.096 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:53:50.096 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:53:50.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 msgr2=0x7f9e1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 0x7f9e1407b0a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9e20033000 tx=0x7f9e20004730 comp rx=0 tx=0).stop 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 msgr2=0x7f9e340cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400d700 tx=0x7f9e2400dbc0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 shutdown_connections 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e14078b90 0x7f9e1407b0a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.087+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e34073170 0x7f9e341f9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.088+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e34070440 0x7f9e3414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.088+0000 7f9e38c00640 1 --2- 172.21.15.129:0/1432738258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e34070030 0x7f9e340cc300 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.088+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 >> 172.21.15.129:0/1432738258 conn(0x7f9e340b1cb0 msgr2=0x7f9e340b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:50.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.088+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 shutdown_connections 2024-09-17T03:53:50.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.088+0000 7f9e38c00640 1 -- 172.21.15.129:0/1432738258 wait complete. 2024-09-17T03:53:50.111 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-17T03:53:50.111 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:53:50.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:50.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:53:50.311 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:53:50.312 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:50.312 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:50.312 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:50.348 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:50.348 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:50.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:53:50.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3535285655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4073170 msgr2=0x7f1ab415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3535285655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4073170 0x7f1ab415c780 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40710f0 tx=0x7f1aa402a560 comp rx=0 tx=0).stop 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3535285655 shutdown_connections 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3535285655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 0x7f1ab4074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3535285655 >> v2:172.21.15.181:3300/0 conn(0x7f1ab4070030 0x7f1ab4074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3535285655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4073170 0x7f1ab415c780 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3535285655 >> 172.21.15.129:0/3535285655 conn(0x7f1ab40b1cb0 msgr2=0x7f1ab40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3535285655 shutdown_connections 2024-09-17T03:53:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.685+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3535285655 wait complete. 2024-09-17T03:53:50.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.687+0000 7f1ab9e00640 1 Processor -- start 2024-09-17T03:53:50.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.688+0000 7f1ab9e00640 1 -- start start 2024-09-17T03:53:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 0x7f1ab41f3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4073170 0x7f1ab40cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab406f640 con 0x7f1ab4070030 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab406f4c0 con 0x7f1ab4073170 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab9e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab406f7c0 con 0x7f1ab4070440 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39072/0 (socket says 172.21.15.129:39072) 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab2a00640 1 -- 172.21.15.129:0/3111007358 learned_addr learned my addr 172.21.15.129:0/3111007358 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab2000640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 0x7f1ab41f3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.689+0000 7f1ab3400640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4073170 0x7f1ab40cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 -- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 msgr2=0x7f1ab41f3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 0x7f1ab41f3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 -- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4073170 msgr2=0x7f1ab40cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4073170 0x7f1ab40cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 -- 172.21.15.129:0/3111007358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab40d0f10 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab2a00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000c00 tx=0x7f1aa4003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa403e030 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1aa4003e00 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab409ddc0 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.690+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4035040 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.691+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab41f4b60 con 0x7f1ab4070030 2024-09-17T03:53:50.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.692+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab415c780 con 0x7f1ab4070030 2024-09-17T03:53:50.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.693+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1aa4041030 con 0x7f1ab4070030 2024-09-17T03:53:50.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.694+0000 7f1aab400640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 0x7f1a9407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:50.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.695+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f1aa40bb4d0 con 0x7f1ab4070030 2024-09-17T03:53:50.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.695+0000 7f1ab2000640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 0x7f1a9407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:50.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.695+0000 7f1ab2000640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 0x7f1a9407b0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1ab41f1340 tx=0x7f1aa000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:50.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.699+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1aa4083220 con 0x7f1ab4070030 2024-09-17T03:53:50.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.894+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1ab40c3860 con 0x7f1ab4070030 2024-09-17T03:53:50.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.894+0000 7f1aab400640 1 -- 172.21.15.129:0/3111007358 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) ==== 80+0+0 (secure 0 0 0) 0x7f1aa4087c50 con 0x7f1ab4070030 2024-09-17T03:53:50.894 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:53:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 msgr2=0x7f1a9407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 0x7f1a9407b0d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1ab41f1340 tx=0x7f1aa000b2e0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 msgr2=0x7f1ab40d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1aa4000c00 tx=0x7f1aa4003910 comp rx=0 tx=0).stop 2024-09-17T03:53:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 shutdown_connections 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a94078bc0 0x7f1a9407b0d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4073170 0x7f1ab40cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ab4070440 0x7f1ab41f3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 --2- 172.21.15.129:0/3111007358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4070030 0x7f1ab40d09d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 >> 172.21.15.129:0/3111007358 conn(0x7f1ab40b1cb0 msgr2=0x7f1ab415ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 shutdown_connections 2024-09-17T03:53:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:50.897+0000 7f1ab9e00640 1 -- 172.21.15.129:0/3111007358 wait complete. 2024-09-17T03:53:50.919 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:53:50.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:50.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:53:50.983 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:53:50.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:53:50.984 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:53:51.019 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:51.020 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:51.020 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:51.061 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:51.062 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:51.062 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:53:51.062 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:51.099 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:53:51.099 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:53:51.099 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:53:51.099 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:53:51.140 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:53:51.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.361+0000 7f8b48400640 1 -- 172.21.15.129:0/1118864104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 msgr2=0x7f8b400ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.361+0000 7f8b48400640 1 --2- 172.21.15.129:0/1118864104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b400ccf40 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8b40071d60 tx=0x7f8b2c02a560 comp rx=0 tx=0).stop 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.361+0000 7f8b48400640 1 -- 172.21.15.129:0/1118864104 shutdown_connections 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.362+0000 7f8b48400640 1 --2- 172.21.15.129:0/1118864104 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b400ccf40 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.362+0000 7f8b48400640 1 --2- 172.21.15.129:0/1118864104 >> v2:172.21.15.181:3300/0 conn(0x7f8b400cf6f0 0x7f8b400cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.362+0000 7f8b48400640 1 -- 172.21.15.129:0/1118864104 >> 172.21.15.129:0/1118864104 conn(0x7f8b400b1cb0 msgr2=0x7f8b400b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:51.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.362+0000 7f8b48400640 1 -- 172.21.15.129:0/1118864104 shutdown_connections 2024-09-17T03:53:51.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.362+0000 7f8b48400640 1 -- 172.21.15.129:0/1118864104 wait complete. 2024-09-17T03:53:51.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.364+0000 7f8b48400640 1 Processor -- start 2024-09-17T03:53:51.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.364+0000 7f8b48400640 1 -- start start 2024-09-17T03:53:51.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b400ce020 0x7f8b400734c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b400cf6f0 0x7f8b40073a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400cecc0 con 0x7f8b400ccb30 2024-09-17T03:53:51.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8b400ceb40 con 0x7f8b400cf6f0 2024-09-17T03:53:51.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.365+0000 7f8b48400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8b400cf1c0 con 0x7f8b400ce020 2024-09-17T03:53:51.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39094/0 (socket says 172.21.15.129:39094) 2024-09-17T03:53:51.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 -- 172.21.15.129:0/839715541 learned_addr learned my addr 172.21.15.129:0/839715541 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:51.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b46400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b400cf6f0 0x7f8b40073a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45000640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b400ce020 0x7f8b400734c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 -- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b400ce020 msgr2=0x7f8b400734c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b400ce020 0x7f8b400734c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 -- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b400cf6f0 msgr2=0x7f8b40073a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b400cf6f0 0x7f8b40073a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 -- 172.21.15.129:0/839715541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b401f94a0 con 0x7f8b400ccb30 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.366+0000 7f8b45a00640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400d750 tx=0x7f8b3400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:51.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.367+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34017070 con 0x7f8b400ccb30 2024-09-17T03:53:51.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.367+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b34009b20 con 0x7f8b400ccb30 2024-09-17T03:53:51.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.367+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34010460 con 0x7f8b400ccb30 2024-09-17T03:53:51.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.367+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b401f9780 con 0x7f8b400ccb30 2024-09-17T03:53:51.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.367+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b401f9d40 con 0x7f8b400ccb30 2024-09-17T03:53:51.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.368+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b40070290 con 0x7f8b400ccb30 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.369+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b3401a070 con 0x7f8b400ccb30 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.370+0000 7f8b3d600640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 0x7f8b2007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.370+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f8b3409ae30 con 0x7f8b400ccb30 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.371+0000 7f8b45000640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 0x7f8b2007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.371+0000 7f8b45000640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 0x7f8b2007b0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c033000 tx=0x7f8b2c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:51.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.374+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8b34062c20 con 0x7f8b400ccb30 2024-09-17T03:53:51.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.594+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b400b56b0 con 0x7f8b400ccb30 2024-09-17T03:53:51.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.594+0000 7f8b3d600640 1 -- 172.21.15.129:0/839715541 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7f8b34067650 con 0x7f8b400ccb30 2024-09-17T03:53:51.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:51.595 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:51.595 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 msgr2=0x7f8b2007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 0x7f8b2007b0d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c033000 tx=0x7f8b2c004730 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 msgr2=0x7f8b4015dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400d750 tx=0x7f8b3400dc10 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 shutdown_connections 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b20078bc0 0x7f8b2007b0d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b400cf6f0 0x7f8b40073a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b400ce020 0x7f8b400734c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 --2- 172.21.15.129:0/839715541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b400ccb30 0x7f8b4015dc40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 >> 172.21.15.129:0/839715541 conn(0x7f8b400b1cb0 msgr2=0x7f8b400ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 shutdown_connections 2024-09-17T03:53:51.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.597+0000 7f8b48400640 1 -- 172.21.15.129:0/839715541 wait complete. 2024-09-17T03:53:51.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:51.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 -- 172.21.15.129:0/1992310363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 msgr2=0x7fe6b40b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/1992310363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40b3090 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac005fd0 tx=0x7fe6ac02aa10 comp rx=0 tx=0).stop 2024-09-17T03:53:51.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 -- 172.21.15.129:0/1992310363 shutdown_connections 2024-09-17T03:53:51.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/1992310363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40b3090 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/1992310363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4073170 0x7fe6b415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 -- 172.21.15.129:0/1992310363 >> 172.21.15.129:0/1992310363 conn(0x7fe6b40b1cb0 msgr2=0x7fe6b40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:51.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 -- 172.21.15.129:0/1992310363 shutdown_connections 2024-09-17T03:53:51.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.839+0000 7fe6bbe00640 1 -- 172.21.15.129:0/1992310363 wait complete. 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.841+0000 7fe6bbe00640 1 Processor -- start 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.841+0000 7fe6bbe00640 1 -- start start 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4070030 0x7fe6b409dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b40b3b00 con 0x7fe6b4073170 2024-09-17T03:53:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b40b3980 con 0x7fe6b4070440 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6bbe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b40b3c80 con 0x7fe6b4070030 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39104/0 (socket says 172.21.15.129:39104) 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6b9e00640 1 -- 172.21.15.129:0/410309793 learned_addr learned my addr 172.21.15.129:0/410309793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6b8a00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.843+0000 7fe6b9400640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4070030 0x7fe6b409dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 -- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4070030 msgr2=0x7fe6b409dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4070030 0x7fe6b409dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 -- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 msgr2=0x7fe6b40ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 -- 172.21.15.129:0/410309793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b40c9790 con 0x7fe6b4073170 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b9e00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009d90 tx=0x7fe6ac003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac033050 con 0x7fe6b4073170 2024-09-17T03:53:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6ac004450 con 0x7fe6b4073170 2024-09-17T03:53:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b40c89d0 con 0x7fe6b4073170 2024-09-17T03:53:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac004860 con 0x7fe6b4073170 2024-09-17T03:53:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.844+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b40c8f90 con 0x7fe6b4073170 2024-09-17T03:53:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.845+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b40b41a0 con 0x7fe6b4073170 2024-09-17T03:53:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.847+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe6ac036070 con 0x7fe6b4073170 2024-09-17T03:53:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.848+0000 7fe6b1600640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 0x7fe69407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.848+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fe6ac0bbfe0 con 0x7fe6b4073170 2024-09-17T03:53:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.848+0000 7fe6b9400640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 0x7fe69407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.849+0000 7fe6b9400640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 0x7fe69407b0d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009ac0 tx=0x7fe6a8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:51.852+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe6ac084470 con 0x7fe6b4073170 2024-09-17T03:53:52.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.070+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6b40c3860 con 0x7fe6b4073170 2024-09-17T03:53:52.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.071+0000 7fe6b1600640 1 -- 172.21.15.129:0/410309793 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7fe6ac083d70 con 0x7fe6b4073170 2024-09-17T03:53:52.071 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:52.071 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:52.072 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 msgr2=0x7fe69407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 0x7fe69407b0d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8009ac0 tx=0x7fe6a8005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:52.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 msgr2=0x7fe6b40cacb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac009d90 tx=0x7fe6ac003f10 comp rx=0 tx=0).stop 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 shutdown_connections 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe694078bc0 0x7fe69407b0d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6b4073170 0x7fe6b40cacb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6b4070440 0x7fe6b40ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 --2- 172.21.15.129:0/410309793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6b4070030 0x7fe6b409dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 >> 172.21.15.129:0/410309793 conn(0x7fe6b40b1cb0 msgr2=0x7fe6b415ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 shutdown_connections 2024-09-17T03:53:52.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.074+0000 7fe6bbe00640 1 -- 172.21.15.129:0/410309793 wait complete. 2024-09-17T03:53:52.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:52.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 -- 172.21.15.129:0/3083230267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 msgr2=0x7ff9a015f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/3083230267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015f490 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff9a015ee80 tx=0x7ff99802aa30 comp rx=0 tx=0).stop 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 -- 172.21.15.129:0/3083230267 shutdown_connections 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/3083230267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015f490 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/3083230267 >> v2:172.21.15.181:3300/0 conn(0x7ff9a0072c30 0x7ff9a015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 -- 172.21.15.129:0/3083230267 >> 172.21.15.129:0/3083230267 conn(0x7ff9a006d410 msgr2=0x7ff9a00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 -- 172.21.15.129:0/3083230267 shutdown_connections 2024-09-17T03:53:52.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.321+0000 7ff9a6c00640 1 -- 172.21.15.129:0/3083230267 wait complete. 2024-09-17T03:53:52.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.323+0000 7ff9a6c00640 1 Processor -- start 2024-09-17T03:53:52.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.323+0000 7ff9a6c00640 1 -- start start 2024-09-17T03:53:52.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 0x7ff9a0159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 0x7ff9a0154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a00b3fa0 con 0x7ff9a0073040 2024-09-17T03:53:52.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a00b3e20 con 0x7ff9a0154110 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a00b4120 con 0x7ff9a0072c30 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39124/0 (socket says 172.21.15.129:39124) 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.325+0000 7ff9a4e00640 1 -- 172.21.15.129:0/2070031410 learned_addr learned my addr 172.21.15.129:0/2070031410 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a6200640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 0x7ff9a0154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 -- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 msgr2=0x7ff9a0159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a5800640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 0x7ff9a0159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 0x7ff9a0159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 -- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 msgr2=0x7ff9a0154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 0x7ff9a0154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 -- 172.21.15.129:0/2070031410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a01f9e10 con 0x7ff9a0073040 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a6200640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 0x7ff9a0154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a5800640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 0x7ff9a0159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:52.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a4e00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e790 tx=0x7ff98c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c00eea0 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff98c00ccc0 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98c005280 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a01f9290 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.326+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a01f9850 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.327+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a01f9000 con 0x7ff9a0073040 2024-09-17T03:53:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.329+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff98c005420 con 0x7ff9a0073040 2024-09-17T03:53:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.330+0000 7ff99d600640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 0x7ff98007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.330+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7ff98c09b680 con 0x7ff9a0073040 2024-09-17T03:53:52.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.331+0000 7ff9a5800640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 0x7ff98007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.331+0000 7ff9a5800640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 0x7ff98007b0a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff994000f60 tx=0x7ff994009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:52.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.335+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff98c063ab0 con 0x7ff9a0073040 2024-09-17T03:53:52.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.518+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9a015fbd0 con 0x7ff9a0073040 2024-09-17T03:53:52.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.519+0000 7ff99d600640 1 -- 172.21.15.129:0/2070031410 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16768 (secure 0 0 0) 0x7ff98c0633b0 con 0x7ff9a0073040 2024-09-17T03:53:52.520 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:52.520 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":81,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:36.047969+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:53:32.430935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-17T03:53:33.487308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:52.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 msgr2=0x7ff98007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 0x7ff98007b0a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff994000f60 tx=0x7ff994009040 comp rx=0 tx=0).stop 2024-09-17T03:53:52.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 msgr2=0x7ff9a015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00e790 tx=0x7ff98c00ec50 comp rx=0 tx=0).stop 2024-09-17T03:53:52.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 shutdown_connections 2024-09-17T03:53:52.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff980078b90 0x7ff98007b0a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9a0154110 0x7ff9a0154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9a0073040 0x7ff9a015abb0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 --2- 172.21.15.129:0/2070031410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9a0072c30 0x7ff9a0159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 >> 172.21.15.129:0/2070031410 conn(0x7ff9a006d410 msgr2=0x7ff9a00cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:52.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 shutdown_connections 2024-09-17T03:53:52.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.522+0000 7ff9a6c00640 1 -- 172.21.15.129:0/2070031410 wait complete. 2024-09-17T03:53:52.545 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:53:52.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 -- 172.21.15.129:0/3901247524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad815e210 msgr2=0x7fbad8073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 --2- 172.21.15.129:0/3901247524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad815e210 0x7fbad8073160 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbac8005fd0 tx=0x7fbac802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 -- 172.21.15.129:0/3901247524 shutdown_connections 2024-09-17T03:53:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 --2- 172.21.15.129:0/3901247524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815a490 0x7fbad80cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 --2- 172.21.15.129:0/3901247524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad815e210 0x7fbad8073160 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 -- 172.21.15.129:0/3901247524 >> 172.21.15.129:0/3901247524 conn(0x7fbad806d870 msgr2=0x7fbad806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 -- 172.21.15.129:0/3901247524 shutdown_connections 2024-09-17T03:53:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.763+0000 7fbadd600640 1 -- 172.21.15.129:0/3901247524 wait complete. 2024-09-17T03:53:52.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.765+0000 7fbadd600640 1 Processor -- start 2024-09-17T03:53:52.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.765+0000 7fbadd600640 1 -- start start 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbad815a490 0x7fbad81493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 0x7fbad81663a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbad80cb860 con 0x7fbad80d01f0 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbad815aab0 con 0x7fbad815a490 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbadd600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbad8070d90 con 0x7fbad815e210 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbad7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39138/0 (socket says 172.21.15.129:39138) 2024-09-17T03:53:52.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.767+0000 7fbad7e00640 1 -- 172.21.15.129:0/921997396 learned_addr learned my addr 172.21.15.129:0/921997396 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbacf400640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbad815a490 0x7fbad81493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbadcc00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 0x7fbad81663a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 -- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 msgr2=0x7fbad81663a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 0x7fbad81663a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 -- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbad815a490 msgr2=0x7fbad81493a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbad815a490 0x7fbad81493a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 -- 172.21.15.129:0/921997396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad8155180 con 0x7fbad80d01f0 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbadcc00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 0x7fbad81663a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:52.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad7e00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbac8034040 tx=0x7fbac80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:52.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac803e030 con 0x7fbad80d01f0 2024-09-17T03:53:52.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbac8003ae0 con 0x7fbad80d01f0 2024-09-17T03:53:52.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad814ddc0 con 0x7fbad80d01f0 2024-09-17T03:53:52.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.768+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8035040 con 0x7fbad80d01f0 2024-09-17T03:53:52.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.769+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad814e380 con 0x7fbad80d01f0 2024-09-17T03:53:52.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.769+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad80cfcb0 con 0x7fbad80d01f0 2024-09-17T03:53:52.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.771+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbac8041030 con 0x7fbad80d01f0 2024-09-17T03:53:52.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.772+0000 7fbad4c00640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 0x7fbab407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:52.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.772+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fbac80bb5f0 con 0x7fbad80d01f0 2024-09-17T03:53:52.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.773+0000 7fbacf400640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 0x7fbab407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:52.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.773+0000 7fbacf400640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 0x7fbab407b0a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbad80cb300 tx=0x7fbac0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:52.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.777+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbac8083a20 con 0x7fbad80d01f0 2024-09-17T03:53:52.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.996+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbad81576f0 con 0x7fbad80d01f0 2024-09-17T03:53:52.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.997+0000 7fbad4c00640 1 -- 172.21.15.129:0/921997396 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7fbac8083320 con 0x7fbad80d01f0 2024-09-17T03:53:52.997 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:52.997 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:52.998 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.999+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 msgr2=0x7fbab407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.999+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 0x7fbab407b0a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbad80cb300 tx=0x7fbac0005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.999+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 msgr2=0x7fbad81f94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:52.999+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fbac8034040 tx=0x7fbac80035f0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 shutdown_connections 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbab4078b90 0x7fbab407b0a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbad815e210 0x7fbad81663a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbad815a490 0x7fbad81493a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 --2- 172.21.15.129:0/921997396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbad80d01f0 0x7fbad81f94d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 >> 172.21.15.129:0/921997396 conn(0x7fbad806d870 msgr2=0x7fbad815e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:53.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 shutdown_connections 2024-09-17T03:53:53.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.000+0000 7fbadd600640 1 -- 172.21.15.129:0/921997396 wait complete. 2024-09-17T03:53:53.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:53:53.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 -- 172.21.15.129:0/3671515651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88073170 msgr2=0x7eff8815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 --2- 172.21.15.129:0/3671515651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88073170 0x7eff8815c780 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7eff78005fd0 tx=0x7eff7802a530 comp rx=0 tx=0).stop 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 -- 172.21.15.129:0/3671515651 shutdown_connections 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 --2- 172.21.15.129:0/3671515651 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88070440 0x7eff880b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 --2- 172.21.15.129:0/3671515651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88073170 0x7eff8815c780 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 -- 172.21.15.129:0/3671515651 >> 172.21.15.129:0/3671515651 conn(0x7eff880b1cb0 msgr2=0x7eff880b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 -- 172.21.15.129:0/3671515651 shutdown_connections 2024-09-17T03:53:53.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.246+0000 7eff8fa00640 1 -- 172.21.15.129:0/3671515651 wait complete. 2024-09-17T03:53:53.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.248+0000 7eff8fa00640 1 Processor -- start 2024-09-17T03:53:53.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.248+0000 7eff8fa00640 1 -- start start 2024-09-17T03:53:53.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 0x7eff8814e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 0x7eff881f7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff880b3b00 con 0x7eff88070440 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7eff880b3980 con 0x7eff88070030 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7eff880b3c80 con 0x7eff88073170 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff7fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff7fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39160/0 (socket says 172.21.15.129:39160) 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff7fe00640 1 -- 172.21.15.129:0/2423918303 learned_addr learned my addr 172.21.15.129:0/2423918303 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8da00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 0x7eff881f7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.250+0000 7eff8d000640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 0x7eff8814e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 -- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 msgr2=0x7eff881f7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 0x7eff881f7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 -- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 msgr2=0x7eff8814e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 0x7eff8814e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 -- 172.21.15.129:0/2423918303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff880cf2a0 con 0x7eff88070440 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff8d000640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 0x7eff8814e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff8da00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 0x7eff881f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:53.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff7fe00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b3c0 tx=0x7eff7000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70011830 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff70016070 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70011d60 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff881e5b90 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.251+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8809dc60 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.253+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff880b4150 con 0x7eff88070440 2024-09-17T03:53:53.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.254+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7eff7001b460 con 0x7eff88070440 2024-09-17T03:53:53.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.255+0000 7eff85600640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 0x7eff6407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.255+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7eff7009a840 con 0x7eff88070440 2024-09-17T03:53:53.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.256+0000 7eff8d000640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 0x7eff6407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.257+0000 7eff8d000640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 0x7eff6407b0a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7eff78033000 tx=0x7eff78004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:53.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.260+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7eff70020020 con 0x7eff88070440 2024-09-17T03:53:53.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.480+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff880c3860 con 0x7eff88070440 2024-09-17T03:53:53.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.481+0000 7eff85600640 1 -- 172.21.15.129:0/2423918303 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5234 (secure 0 0 0) 0x7eff880cf2a0 con 0x7eff88070440 2024-09-17T03:53:53.481 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:53.481 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":54,"btime":"2024-09-17T03:53:38:061677+0000","default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:53:34.515705+0000","modified":"2024-09-17T03:53:37.544115+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6049":{"gid":6049,"name":"c","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/915087149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":915087149},{"type":"v1","addr":"172.21.15.129:6835","nonce":915087149}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6049,"qdb_cluster":[6049]},"id":6}]} 2024-09-17T03:53:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 54 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.483+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 msgr2=0x7eff6407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.483+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 0x7eff6407b0a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7eff78033000 tx=0x7eff78004730 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.483+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 msgr2=0x7eff881f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.483+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7eff7000b3c0 tx=0x7eff7000b880 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 shutdown_connections 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff64078b90 0x7eff6407b0a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff88073170 0x7eff881f7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff88070440 0x7eff881f6f00 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 --2- 172.21.15.129:0/2423918303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff88070030 0x7eff8814e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 >> 172.21.15.129:0/2423918303 conn(0x7eff880b1cb0 msgr2=0x7eff880ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:53.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 shutdown_connections 2024-09-17T03:53:53.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.484+0000 7eff8fa00640 1 -- 172.21.15.129:0/2423918303 wait complete. 2024-09-17T03:53:53.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 -- 172.21.15.129:0/3064830213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cd240 msgr2=0x7f52e40cf010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 --2- 172.21.15.129:0/3064830213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cd240 0x7f52e40cf010 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f52cc00a940 tx=0x7f52cc02b7f0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 -- 172.21.15.129:0/3064830213 shutdown_connections 2024-09-17T03:53:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 --2- 172.21.15.129:0/3064830213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cd240 0x7f52e40cf010 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 -- 172.21.15.129:0/3064830213 >> 172.21.15.129:0/3064830213 conn(0x7f52e40b1cb0 msgr2=0x7f52e40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 -- 172.21.15.129:0/3064830213 shutdown_connections 2024-09-17T03:53:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.726+0000 7f52e8a00640 1 -- 172.21.15.129:0/3064830213 wait complete. 2024-09-17T03:53:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.728+0000 7f52e8a00640 1 Processor -- start 2024-09-17T03:53:53.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.729+0000 7f52e8a00640 1 -- start start 2024-09-17T03:53:53.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cfb90 0x7f52e4147100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 0x7f52e4147640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52e40710a0 con 0x7f52e40cf660 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f52e4070f20 con 0x7f52e40cfb90 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f52e40b36c0 con 0x7f52e414abe0 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39162/0 (socket says 172.21.15.129:39162) 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.730+0000 7f52e1e00640 1 -- 172.21.15.129:0/2122707719 learned_addr learned my addr 172.21.15.129:0/2122707719 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52dbe00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cfb90 0x7f52e4147100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e2800640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 0x7f52e4147640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 -- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 msgr2=0x7f52e4147640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 0x7f52e4147640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 -- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cfb90 msgr2=0x7f52e4147100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cfb90 0x7f52e4147100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 -- 172.21.15.129:0/2122707719 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e409cee0 con 0x7f52e40cf660 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e2800640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 0x7f52e4147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e1e00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c400 tx=0x7f52d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4017070 con 0x7f52e40cf660 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52d40053c0 con 0x7f52e40cf660 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e409d1c0 con 0x7f52e40cf660 2024-09-17T03:53:53.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.731+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4003830 con 0x7f52e40cf660 2024-09-17T03:53:53.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.732+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e409d780 con 0x7f52e40cf660 2024-09-17T03:53:53.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.732+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e409deb0 con 0x7f52e40cf660 2024-09-17T03:53:53.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.734+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52d401a070 con 0x7f52e40cf660 2024-09-17T03:53:53.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.735+0000 7f52da000640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 0x7f52c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:53.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.735+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f52d409a1e0 con 0x7f52e40cf660 2024-09-17T03:53:53.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.735+0000 7f52dbe00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 0x7f52c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:53.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.736+0000 7f52dbe00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 0x7f52c007b0a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52cc035000 tx=0x7f52cc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:53.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.739+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f52e409deb0 con 0x7f52e40cf660 2024-09-17T03:53:53.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.927+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f52e40b2160 con 0x7f52e40cf660 2024-09-17T03:53:53.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.928+0000 7f52da000640 1 -- 172.21.15.129:0/2122707719 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16768 (secure 0 0 0) 0x7f52d4062580 con 0x7f52e40cf660 2024-09-17T03:53:53.929 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:53.929 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":81,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:36.047969+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:53:32.430935+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-17T03:53:33.487308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 msgr2=0x7f52c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 0x7f52c007b0a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52cc035000 tx=0x7f52cc004730 comp rx=0 tx=0).stop 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 msgr2=0x7f52e415dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c400 tx=0x7f52d400c8c0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 shutdown_connections 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52c0078b90 0x7f52c007b0a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52e414abe0 0x7f52e4147640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52e40cfb90 0x7f52e4147100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 --2- 172.21.15.129:0/2122707719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52e40cf660 0x7f52e415dc80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:53.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 >> 172.21.15.129:0/2122707719 conn(0x7f52e40b1cb0 msgr2=0x7f52e40b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:53.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 shutdown_connections 2024-09-17T03:53:53.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:53.931+0000 7f52e8a00640 1 -- 172.21.15.129:0/2122707719 wait complete. 2024-09-17T03:53:53.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:53:54.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 -- 172.21.15.129:0/1253796125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28070440 msgr2=0x7f4b28074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 --2- 172.21.15.129:0/1253796125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28070440 0x7f4b28074b80 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4b20005fd0 tx=0x7f4b2002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 -- 172.21.15.129:0/1253796125 shutdown_connections 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 --2- 172.21.15.129:0/1253796125 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28070440 0x7f4b28074b80 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 --2- 172.21.15.129:0/1253796125 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070030 0x7f4b28074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 -- 172.21.15.129:0/1253796125 >> 172.21.15.129:0/1253796125 conn(0x7f4b280b1cb0 msgr2=0x7f4b280b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 -- 172.21.15.129:0/1253796125 shutdown_connections 2024-09-17T03:53:54.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.176+0000 7f4b2d400640 1 -- 172.21.15.129:0/1253796125 wait complete. 2024-09-17T03:53:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.178+0000 7f4b2d400640 1 Processor -- start 2024-09-17T03:53:54.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.179+0000 7f4b2d400640 1 -- start start 2024-09-17T03:53:54.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070440 0x7f4b28159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28073170 0x7f4b2809aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b280b3a90 con 0x7f4b28070030 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4b280b3910 con 0x7f4b28073170 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b2d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4b280b3c10 con 0x7f4b28070440 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b26800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b26800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44516/0 (socket says 172.21.15.129:44516) 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b26800640 1 -- 172.21.15.129:0/3095503050 learned_addr learned my addr 172.21.15.129:0/3095503050 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b27200640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28073170 0x7f4b2809aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.180+0000 7f4b25e00640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070440 0x7f4b28159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 -- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070440 msgr2=0x7f4b28159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070440 0x7f4b28159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 -- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28073170 msgr2=0x7f4b2809aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28073170 0x7f4b2809aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 -- 172.21.15.129:0/3095503050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b280d0ef0 con 0x7f4b28070030 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b26800640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c3a0 tx=0x7f4b1800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011790 con 0x7f4b28070030 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b18017070 con 0x7f4b28070030 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.181+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b280ca880 con 0x7f4b28070030 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.182+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18011cc0 con 0x7f4b28070030 2024-09-17T03:53:54.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.182+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b280cae40 con 0x7f4b28070030 2024-09-17T03:53:54.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.183+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b280b4130 con 0x7f4b28070030 2024-09-17T03:53:54.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.184+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4b1801c460 con 0x7f4b28070030 2024-09-17T03:53:54.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.185+0000 7f4b1ea00640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 0x7f4b0807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.185+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f4b1809aa30 con 0x7f4b28070030 2024-09-17T03:53:54.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.186+0000 7f4b25e00640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 0x7f4b0807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.187+0000 7f4b25e00640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 0x7f4b0807b0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b280c9fc0 tx=0x7f4b14005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:54.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.190+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4b18062780 con 0x7f4b28070030 2024-09-17T03:53:54.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.404+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4b280c3860 con 0x7f4b28070030 2024-09-17T03:53:54.405 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:53:54.405+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:53:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.407+0000 7f4b1ea00640 1 -- 172.21.15.129:0/3095503050 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v55) ==== 161+0+0 (secure 0 0 0) 0x7f4b180671b0 con 0x7f4b28070030 2024-09-17T03:53:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:53:54.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.411+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 msgr2=0x7f4b0807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.411+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 0x7f4b0807b0d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4b280c9fc0 tx=0x7f4b14005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.411+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 msgr2=0x7f4b280cb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.411+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c3a0 tx=0x7f4b1800c860 comp rx=0 tx=0).stop 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 shutdown_connections 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b08078bc0 0x7f4b0807b0d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b28073170 0x7f4b2809aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b28070440 0x7f4b28159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 --2- 172.21.15.129:0/3095503050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b28070030 0x7f4b280cb640 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 >> 172.21.15.129:0/3095503050 conn(0x7f4b280b1cb0 msgr2=0x7f4b2815eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 shutdown_connections 2024-09-17T03:53:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.412+0000 7f4b2d400640 1 -- 172.21.15.129:0/3095503050 wait complete. 2024-09-17T03:53:54.419 INFO:tasks.ceph.mds.c.smithi129.stderr: -4483> 2024-09-17T03:53:37.017+0000 7f1a0e000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:53:54.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:53:54.543 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.661+0000 7f5aa1000640 1 -- 172.21.15.129:0/3332435728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c0b3e20 msgr2=0x7f5a9c0b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.661+0000 7f5aa1000640 1 --2- 172.21.15.129:0/3332435728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c0b3e20 0x7f5a9c0b4230 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c15ef30 tx=0x7f5a8802a480 comp rx=0 tx=0).stop 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 -- 172.21.15.129:0/3332435728 shutdown_connections 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 --2- 172.21.15.129:0/3332435728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c0b3e20 0x7f5a9c0b4230 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 --2- 172.21.15.129:0/3332435728 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c15e950 0x7f5a9c0d0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 -- 172.21.15.129:0/3332435728 >> 172.21.15.129:0/3332435728 conn(0x7f5a9c0cee20 msgr2=0x7f5a9c0cf230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:54.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 -- 172.21.15.129:0/3332435728 shutdown_connections 2024-09-17T03:53:54.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.662+0000 7f5aa1000640 1 -- 172.21.15.129:0/3332435728 wait complete. 2024-09-17T03:53:54.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.664+0000 7f5aa1000640 1 Processor -- start 2024-09-17T03:53:54.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.664+0000 7f5aa1000640 1 -- start start 2024-09-17T03:53:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a9c0727e0 0x7f5a9c1f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c0b3e20 0x7f5a9c14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c0ce800 con 0x7f5a9c15e950 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9c0ce680 con 0x7f5a9c0727e0 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.665+0000 7f5aa1000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5a9c073300 con 0x7f5a9c0b3e20 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44556/0 (socket says 172.21.15.129:44556) 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 -- 172.21.15.129:0/1321991535 learned_addr learned my addr 172.21.15.129:0/1321991535 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:54.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a99e00640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a9c0727e0 0x7f5a9c1f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a99400640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c0b3e20 0x7f5a9c14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 -- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c0b3e20 msgr2=0x7f5a9c14b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c0b3e20 0x7f5a9c14b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 -- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a9c0727e0 msgr2=0x7f5a9c1f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a9c0727e0 0x7f5a9c1f5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 -- 172.21.15.129:0/1321991535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a9c14bcf0 con 0x7f5a9c15e950 2024-09-17T03:53:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.666+0000 7f5a9a800640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5a9400c730 tx=0x7f5a9400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.667+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a94003b70 con 0x7f5a9c15e950 2024-09-17T03:53:54.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.667+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a94003d30 con 0x7f5a9c15e950 2024-09-17T03:53:54.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.667+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9c1f6560 con 0x7f5a9c15e950 2024-09-17T03:53:54.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.667+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a940052d0 con 0x7f5a9c15e950 2024-09-17T03:53:54.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.667+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9c1f6b20 con 0x7f5a9c15e950 2024-09-17T03:53:54.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.668+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a9c0b4230 con 0x7f5a9c15e950 2024-09-17T03:53:54.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.669+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5a94005680 con 0x7f5a9c15e950 2024-09-17T03:53:54.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.670+0000 7f5a92000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 0x7f5a7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:54.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.670+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7f5a9409ac90 con 0x7f5a9c15e950 2024-09-17T03:53:54.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.671+0000 7f5a99e00640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 0x7f5a7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:54.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.671+0000 7f5a99e00640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 0x7f5a7c07b0a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009b30 tx=0x7f5a8c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:54.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.675+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5a9c0b4770 con 0x7f5a9c15e950 2024-09-17T03:53:54.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:54.894+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5a9c1f62f0 con 0x7f5a9c15e950 2024-09-17T03:53:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.409+0000 7f5a92000640 1 -- 172.21.15.129:0/1321991535 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v56) ==== 104+0+0 (secure 0 0 0) 0x7f5a94062980 con 0x7f5a9c15e950 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.412+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 msgr2=0x7f5a7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.412+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 0x7f5a7c07b0a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c009b30 tx=0x7f5a8c005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 msgr2=0x7f5a9c0af2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f5a9400c730 tx=0x7f5a9400cbf0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 shutdown_connections 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a7c078b90 0x7f5a7c07b0a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a9c15e950 0x7f5a9c0af2f0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a9c0b3e20 0x7f5a9c14b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 --2- 172.21.15.129:0/1321991535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a9c0727e0 0x7f5a9c1f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 >> 172.21.15.129:0/1321991535 conn(0x7f5a9c0cee20 msgr2=0x7f5a9c15af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 shutdown_connections 2024-09-17T03:53:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.413+0000 7f5aa1000640 1 -- 172.21.15.129:0/1321991535 wait complete. 2024-09-17T03:53:55.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:53:55.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.660+0000 7ff6fd000640 1 -- 172.21.15.129:0/3275852973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 msgr2=0x7ff6f80b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:55.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.660+0000 7ff6fd000640 1 --2- 172.21.15.129:0/3275852973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f80b2f40 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff6f80cb560 tx=0x7ff6e802a410 comp rx=0 tx=0).stop 2024-09-17T03:53:55.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 -- 172.21.15.129:0/3275852973 shutdown_connections 2024-09-17T03:53:55.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 --2- 172.21.15.129:0/3275852973 >> v2:172.21.15.181:3300/0 conn(0x7ff6f80b3480 0x7ff6f8072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 --2- 172.21.15.129:0/3275852973 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f80b2f40 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 -- 172.21.15.129:0/3275852973 >> 172.21.15.129:0/3275852973 conn(0x7ff6f80b1c40 msgr2=0x7ff6f80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:55.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 -- 172.21.15.129:0/3275852973 shutdown_connections 2024-09-17T03:53:55.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.661+0000 7ff6fd000640 1 -- 172.21.15.129:0/3275852973 wait complete. 2024-09-17T03:53:55.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.663+0000 7ff6fd000640 1 Processor -- start 2024-09-17T03:53:55.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.663+0000 7ff6fd000640 1 -- start start 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.664+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6f80733c0 0x7ff6f81f9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.664+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.664+0000 7ff6fd000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f80b3480 0x7ff6f81f8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6fd000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8074060 con 0x7ff6f80748f0 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6fd000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6f8073ee0 con 0x7ff6f80b3480 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6fd000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff6f806fe50 con 0x7ff6f80733c0 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44586/0 (socket says 172.21.15.129:44586) 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 -- 172.21.15.129:0/176909502 learned_addr learned my addr 172.21.15.129:0/176909502 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:55.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6f5e00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6f80733c0 0x7ff6f81f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6f6800640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f80b3480 0x7ff6f81f8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 -- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6f80733c0 msgr2=0x7ff6f81f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6f80733c0 0x7ff6f81f9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 -- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f80b3480 msgr2=0x7ff6f81f8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f80b3480 0x7ff6f81f8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.665+0000 7ff6efe00640 1 -- 172.21.15.129:0/176909502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6f809cbc0 con 0x7ff6f80748f0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6efe00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000e7f0 tx=0x7ff6e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e000eea0 con 0x7ff6f80748f0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6e000ccc0 con 0x7ff6f80748f0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6f809cea0 con 0x7ff6f80748f0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e0004280 con 0x7ff6f80748f0 2024-09-17T03:53:55.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.666+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6f809d460 con 0x7ff6f80748f0 2024-09-17T03:53:55.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.667+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6f80741e0 con 0x7ff6f80748f0 2024-09-17T03:53:55.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.668+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff6e00035f0 con 0x7ff6f80748f0 2024-09-17T03:53:55.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.669+0000 7ff6ee000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 0x7ff6d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:55.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.669+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5795+0+0 (secure 0 0 0) 0x7ff6e009b630 con 0x7ff6f80748f0 2024-09-17T03:53:55.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.670+0000 7ff6f5e00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 0x7ff6d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:55.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.671+0000 7ff6f5e00640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 0x7ff6d407b0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8034000 tx=0x7ff6e8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:55.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.674+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff6f80741e0 con 0x7ff6f80748f0 2024-09-17T03:53:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:55.857+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff6f80b2f40 con 0x7ff6f80748f0 2024-09-17T03:53:56.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.412+0000 7ff6ee000640 1 -- 172.21.15.129:0/176909502 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) ==== 181+0+0 (secure 0 0 0) 0x7ff6e0063320 con 0x7ff6f80748f0 2024-09-17T03:53:56.412 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:53:56.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.415+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 msgr2=0x7ff6d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:56.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.415+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 0x7ff6d407b0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6e8034000 tx=0x7ff6e8004730 comp rx=0 tx=0).stop 2024-09-17T03:53:56.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.415+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 msgr2=0x7ff6f8150500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:56.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.415+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff6e000e7f0 tx=0x7ff6e000ecb0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 shutdown_connections 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6d4078b90 0x7ff6d407b0a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6f80b3480 0x7ff6f81f8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6f80748f0 0x7ff6f8150500 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 --2- 172.21.15.129:0/176909502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6f80733c0 0x7ff6f81f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 >> 172.21.15.129:0/176909502 conn(0x7ff6f80b1c40 msgr2=0x7ff6f80c9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 shutdown_connections 2024-09-17T03:53:56.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.416+0000 7ff6fd000640 1 -- 172.21.15.129:0/176909502 wait complete. 2024-09-17T03:53:56.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:53:56.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 -- 172.21.15.129:0/2990564167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0504004800 msgr2=0x7f0504004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:56.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 --2- 172.21.15.129:0/2990564167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0504004800 0x7f0504004c10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f051c0c3a50 tx=0x7f0504005830 comp rx=0 tx=0).stop 2024-09-17T03:53:56.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 -- 172.21.15.129:0/2990564167 shutdown_connections 2024-09-17T03:53:56.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 --2- 172.21.15.129:0/2990564167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0504004800 0x7f0504004c10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 -- 172.21.15.129:0/2990564167 >> 172.21.15.129:0/2990564167 conn(0x7f051c0b1c40 msgr2=0x7f051c0b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:56.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 -- 172.21.15.129:0/2990564167 shutdown_connections 2024-09-17T03:53:56.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.669+0000 7f0524800640 1 -- 172.21.15.129:0/2990564167 wait complete. 2024-09-17T03:53:56.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.671+0000 7f0524800640 1 Processor -- start 2024-09-17T03:53:56.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.671+0000 7f0524800640 1 -- start start 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0504004800 0x7f051c0c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f051c15c6c0 0x7f051c09cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f051c0b30c0 con 0x7f051c0cf330 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f051c0b2f40 con 0x7f0504004800 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0524800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f051c0b3240 con 0x7f051c15c6c0 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:56.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0504004800 0x7f051c0c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:56.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44600/0 (socket says 172.21.15.129:44600) 2024-09-17T03:53:56.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521400640 1 -- 172.21.15.129:0/3520364826 learned_addr learned my addr 172.21.15.129:0/3520364826 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:56.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0522800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f051c15c6c0 0x7f051c09cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521400640 1 -- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f051c15c6c0 msgr2=0x7f051c09cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.673+0000 7f0521400640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f051c15c6c0 0x7f051c09cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0521400640 1 -- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0504004800 msgr2=0x7f051c0c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0521400640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0504004800 0x7f051c0c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0521400640 1 -- 172.21.15.129:0/3520364826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051c0cdd60 con 0x7f051c0cf330 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0521400640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f050c009c10 tx=0x7f050c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c038040 con 0x7f051c0cf330 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f050c043070 con 0x7f051c0cf330 2024-09-17T03:53:56.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050c033bd0 con 0x7f051c0cf330 2024-09-17T03:53:56.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051c0cfad0 con 0x7f051c0cf330 2024-09-17T03:53:56.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.674+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051c1f9500 con 0x7f051c0cf330 2024-09-17T03:53:56.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.675+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f051c0b1a40 con 0x7f051c0cf330 2024-09-17T03:53:56.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.677+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f050c048460 con 0x7f051c0cf330 2024-09-17T03:53:56.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.678+0000 7f0516000640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 0x7f050007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:56.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.678+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5357+0+0 (secure 0 0 0) 0x7f050c0bbcf0 con 0x7f051c0cf330 2024-09-17T03:53:56.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.678+0000 7f0521e00640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 0x7f050007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:56.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.679+0000 7f0521e00640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 0x7f050007b0d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0510001860 tx=0x7f0510006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:56.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.682+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f051c0b1a40 con 0x7f051c0cf330 2024-09-17T03:53:56.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:56.869+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f051c050f60 con 0x7f051c0cf330 2024-09-17T03:53:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.421+0000 7f0516000640 1 -- 172.21.15.129:0/3520364826 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) ==== 169+0+0 (secure 0 0 0) 0x7f051c050f60 con 0x7f051c0cf330 2024-09-17T03:53:57.422 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:53:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 msgr2=0x7f050007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 0x7f050007b0d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0510001860 tx=0x7f0510006040 comp rx=0 tx=0).stop 2024-09-17T03:53:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 msgr2=0x7f051c15a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f050c009c10 tx=0x7f050c033670 comp rx=0 tx=0).stop 2024-09-17T03:53:57.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 shutdown_connections 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0500078bc0 0x7f050007b0d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f051c15c6c0 0x7f051c09cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f051c0cf330 0x7f051c15a5f0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 --2- 172.21.15.129:0/3520364826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0504004800 0x7f051c0c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 >> 172.21.15.129:0/3520364826 conn(0x7f051c0b1c40 msgr2=0x7f051c073200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 shutdown_connections 2024-09-17T03:53:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.424+0000 7f0524800640 1 -- 172.21.15.129:0/3520364826 wait complete. 2024-09-17T03:53:57.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 -- 172.21.15.129:0/1129191163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ccb30 msgr2=0x7f12c40ccf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 --2- 172.21.15.129:0/1129191163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ccb30 0x7f12c40ccf40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f12c4072be0 tx=0x7f12ac02a560 comp rx=0 tx=0).stop 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 -- 172.21.15.129:0/1129191163 shutdown_connections 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 --2- 172.21.15.129:0/1129191163 >> v2:172.21.15.181:3300/0 conn(0x7f12c40ce020 0x7f12c40ce430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 --2- 172.21.15.129:0/1129191163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ccb30 0x7f12c40ccf40 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 --2- 172.21.15.129:0/1129191163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40cf6f0 0x7f12c40cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 -- 172.21.15.129:0/1129191163 >> 172.21.15.129:0/1129191163 conn(0x7f12c40b1cb0 msgr2=0x7f12c40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:57.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 -- 172.21.15.129:0/1129191163 shutdown_connections 2024-09-17T03:53:57.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.676+0000 7f12c8a00640 1 -- 172.21.15.129:0/1129191163 wait complete. 2024-09-17T03:53:57.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.679+0000 7f12c8a00640 1 Processor -- start 2024-09-17T03:53:57.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.679+0000 7f12c8a00640 1 -- start start 2024-09-17T03:53:57.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 0x7f12c41615b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 0x7f12c414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f12c40cf340 con 0x7f12c40ce020 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f12c40cf1c0 con 0x7f12c40cf6f0 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.680+0000 7f12c8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f12c40701b0 con 0x7f12c40ccb30 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44618/0 (socket says 172.21.15.129:44618) 2024-09-17T03:53:57.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 -- 172.21.15.129:0/3732222177 learned_addr learned my addr 172.21.15.129:0/3732222177 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12c2800640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 0x7f12c414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 -- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 msgr2=0x7f12c41615b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12c1e00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 0x7f12c41615b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 0x7f12c41615b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 -- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 msgr2=0x7f12c414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 0x7f12c414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 -- 172.21.15.129:0/3732222177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12c40c3ae0 con 0x7f12c40ce020 2024-09-17T03:53:57.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12c2800640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 0x7f12c414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12c1e00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 0x7f12c41615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.681+0000 7f12bbe00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0099c0 tx=0x7f12ac0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.682+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac03e030 con 0x7f12c40ce020 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.682+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f12ac0047a0 con 0x7f12c40ce020 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.682+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12c40c3dc0 con 0x7f12c40ce020 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.682+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ac035040 con 0x7f12c40ce020 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.682+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12c4161090 con 0x7f12c40ce020 2024-09-17T03:53:57.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.683+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12c41fa1c0 con 0x7f12c40ce020 2024-09-17T03:53:57.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.684+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f12ac041030 con 0x7f12c40ce020 2024-09-17T03:53:57.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.685+0000 7f12ba000640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 0x7f12a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:57.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.685+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f12ac0bb3b0 con 0x7f12c40ce020 2024-09-17T03:53:57.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.686+0000 7f12c1e00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 0x7f12a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:57.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.686+0000 7f12c1e00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 0x7f12a007b0a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12c4071290 tx=0x7f12b4008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:57.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.690+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f12ac083a50 con 0x7f12c40ce020 2024-09-17T03:53:57.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.873+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f12c40ceb00 con 0x7f12c40ce020 2024-09-17T03:53:57.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.877+0000 7f12ba000640 1 -- 172.21.15.129:0/3732222177 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v45) ==== 71+0+0 (secure 0 0 0) 0x7f12ac08c020 con 0x7f12c40ce020 2024-09-17T03:53:57.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 msgr2=0x7f12a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 0x7f12a007b0a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12c4071290 tx=0x7f12b4008040 comp rx=0 tx=0).stop 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 msgr2=0x7f12c414ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f12ac0099c0 tx=0x7f12ac0039e0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 shutdown_connections 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f12a0078b90 0x7f12a007b0a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f12c40cf6f0 0x7f12c414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f12c40ce020 0x7f12c414ecc0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 --2- 172.21.15.129:0/3732222177 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f12c40ccb30 0x7f12c41615b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:57.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 >> 172.21.15.129:0/3732222177 conn(0x7f12c40b1cb0 msgr2=0x7f12c40b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:57.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 shutdown_connections 2024-09-17T03:53:57.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:57.880+0000 7f12c8a00640 1 -- 172.21.15.129:0/3732222177 wait complete. 2024-09-17T03:53:57.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout' 2024-09-17T03:53:58.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 -- 172.21.15.129:0/4292986667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 msgr2=0x7f70f0073280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 --2- 172.21.15.129:0/4292986667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f0073280 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f70f006e680 tx=0x7f70e402aac0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 -- 172.21.15.129:0/4292986667 shutdown_connections 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 --2- 172.21.15.129:0/4292986667 >> v2:172.21.15.181:3300/0 conn(0x7f70f00b4900 0x7f70f00737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 --2- 172.21.15.129:0/4292986667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f0073280 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 -- 172.21.15.129:0/4292986667 >> 172.21.15.129:0/4292986667 conn(0x7f70f00b2160 msgr2=0x7f70f00b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 -- 172.21.15.129:0/4292986667 shutdown_connections 2024-09-17T03:53:58.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.131+0000 7f70f8200640 1 -- 172.21.15.129:0/4292986667 wait complete. 2024-09-17T03:53:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.134+0000 7f70f8200640 1 Processor -- start 2024-09-17T03:53:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.134+0000 7f70f8200640 1 -- start start 2024-09-17T03:53:58.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.136+0000 7f70f8200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.136+0000 7f70f8200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70f00b4900 0x7f70f0136590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f8200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f0146200 0x7f70f0136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f8200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70f0160a00 con 0x7f70f0071420 2024-09-17T03:53:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f8200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70f00cdb90 con 0x7f70f0146200 2024-09-17T03:53:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f8200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70f0160b80 con 0x7f70f00b4900 2024-09-17T03:53:58.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44636/0 (socket says 172.21.15.129:44636) 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f5800640 1 -- 172.21.15.129:0/2792996293 learned_addr learned my addr 172.21.15.129:0/2792996293 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f6200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f0146200 0x7f70f0136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.137+0000 7f70f4e00640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70f00b4900 0x7f70f0136590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 -- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70f00b4900 msgr2=0x7f70f0136590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70f00b4900 0x7f70f0136590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 -- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f0146200 msgr2=0x7f70f0136ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f0146200 0x7f70f0136ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 -- 172.21.15.129:0/2792996293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f00c7290 con 0x7f70f0071420 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f5800640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009d60 tx=0x7f70e4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4004250 con 0x7f70f0071420 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70e40043f0 con 0x7f70f0071420 2024-09-17T03:53:58.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.138+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f013a270 con 0x7f70f0071420 2024-09-17T03:53:58.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.139+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4004820 con 0x7f70f0071420 2024-09-17T03:53:58.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.139+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f00c7830 con 0x7f70f0071420 2024-09-17T03:53:58.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.140+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f01610a0 con 0x7f70f0071420 2024-09-17T03:53:58.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.141+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70e4036400 con 0x7f70f0071420 2024-09-17T03:53:58.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.142+0000 7f70ed600640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 0x7f70d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.142+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f70e40bba90 con 0x7f70f0071420 2024-09-17T03:53:58.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.143+0000 7f70f4e00640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 0x7f70d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.144+0000 7f70f4e00640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 0x7f70d007b0a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f70f00cc680 tx=0x7f70dc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.146+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f70e4032030 con 0x7f70f0071420 2024-09-17T03:53:58.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.398+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]} v 0) -- 0x7f70f00b2ab0 con 0x7f70f0071420 2024-09-17T03:53:58.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.423+0000 7f70ed600640 1 -- 172.21.15.129:0/2792996293 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_fancy_layout"]}]=0 v246) ==== 132+0+0 (secure 0 0 0) 0x7f70e4084130 con 0x7f70f0071420 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.425+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 msgr2=0x7f70d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.425+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 0x7f70d007b0a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f70f00cc680 tx=0x7f70dc009040 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.425+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 msgr2=0x7f70f01fa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.425+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009d60 tx=0x7f70e4003f10 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 shutdown_connections 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70d0078b90 0x7f70d007b0a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70f0146200 0x7f70f0136ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70f00b4900 0x7f70f0136590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 --2- 172.21.15.129:0/2792996293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70f0071420 0x7f70f01fa090 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 >> 172.21.15.129:0/2792996293 conn(0x7f70f00b2160 msgr2=0x7f70f00d04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:58.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 shutdown_connections 2024-09-17T03:53:58.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.426+0000 7f70f8200640 1 -- 172.21.15.129:0/2792996293 wait complete. 2024-09-17T03:53:58.448 INFO:tasks.cephfs_test_runner:purge stray file with fancy layout ... ok 2024-09-17T03:53:58.448 INFO:tasks.cephfs_test_runner:Starting test: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:53:58.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 -- 172.21.15.129:0/3634395476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf950 msgr2=0x7f8f3406f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 --2- 172.21.15.129:0/3634395476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf950 0x7f8f3406f3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c005fd0 tx=0x7f8f2c02aa00 comp rx=0 tx=0).stop 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 -- 172.21.15.129:0/3634395476 shutdown_connections 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 --2- 172.21.15.129:0/3634395476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf950 0x7f8f3406f3b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 --2- 172.21.15.129:0/3634395476 >> v2:172.21.15.181:3300/0 conn(0x7f8f340cf540 0x7f8f3406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 -- 172.21.15.129:0/3634395476 >> 172.21.15.129:0/3634395476 conn(0x7f8f3406d410 msgr2=0x7f8f340b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 -- 172.21.15.129:0/3634395476 shutdown_connections 2024-09-17T03:53:58.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.667+0000 7f8f3a000640 1 -- 172.21.15.129:0/3634395476 wait complete. 2024-09-17T03:53:58.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.670+0000 7f8f3a000640 1 Processor -- start 2024-09-17T03:53:58.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.670+0000 7f8f3a000640 1 -- start start 2024-09-17T03:53:58.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf540 0x7f8f341f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f340cf950 0x7f8f340ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3415ddd0 con 0x7f8f3415ca50 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3415dc50 con 0x7f8f340cf950 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.671+0000 7f8f3a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3415df50 con 0x7f8f340cf540 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44648/0 (socket says 172.21.15.129:44648) 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 -- 172.21.15.129:0/2609627161 learned_addr learned my addr 172.21.15.129:0/2609627161 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:58.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f38c00640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf540 0x7f8f341f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f2be00640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f340cf950 0x7f8f340ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 -- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf540 msgr2=0x7f8f341f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf540 0x7f8f341f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 -- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f340cf950 msgr2=0x7f8f340ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f340cf950 0x7f8f340ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 -- 172.21.15.129:0/2609627161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f34157e50 con 0x7f8f3415ca50 2024-09-17T03:53:58.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.672+0000 7f8f39600640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c006010 tx=0x7f8f2c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.673+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c046070 con 0x7f8f3415ca50 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.673+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f2c004540 con 0x7f8f3415ca50 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.673+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f34158dd0 con 0x7f8f3415ca50 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.673+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c033070 con 0x7f8f3415ca50 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.673+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f34159390 con 0x7f8f3415ca50 2024-09-17T03:53:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.674+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f3415aef0 con 0x7f8f3415ca50 2024-09-17T03:53:58.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.675+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8f2c049070 con 0x7f8f3415ca50 2024-09-17T03:53:58.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.676+0000 7f8f31600640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 0x7f8f1007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:58.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.676+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f8f2c0bbae0 con 0x7f8f3415ca50 2024-09-17T03:53:58.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.677+0000 7f8f38c00640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 0x7f8f1007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:58.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.677+0000 7f8f38c00640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 0x7f8f1007b0a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8f240016c0 tx=0x7f8f24009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:58.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.680+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8f2c03e070 con 0x7f8f3415ca50 2024-09-17T03:53:58.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.865+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f3415e400 con 0x7f8f3415ca50 2024-09-17T03:53:58.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.866+0000 7f8f31600640 1 -- 172.21.15.129:0/2609627161 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+13081 (secure 0 0 0) 0x7f8f2c083b40 con 0x7f8f3415ca50 2024-09-17T03:53:58.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:53:58.867 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":84,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:53:57.413678+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/915087149":"2024-09-18T03:53:54.406148+0000","172.21.15.129:6833/915087149":"2024-09-18T03:53:54.406148+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 msgr2=0x7f8f1007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 0x7f8f1007b0a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8f240016c0 tx=0x7f8f24009040 comp rx=0 tx=0).stop 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 msgr2=0x7f8f34157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c006010 tx=0x7f8f2c003f10 comp rx=0 tx=0).stop 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 shutdown_connections 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f10078b90 0x7f8f1007b0a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f3415ca50 0x7f8f34157060 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f340cf950 0x7f8f340ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 --2- 172.21.15.129:0/2609627161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f340cf540 0x7f8f341f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 >> 172.21.15.129:0/2609627161 conn(0x7f8f3406d410 msgr2=0x7f8f34156270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 shutdown_connections 2024-09-17T03:53:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:58.869+0000 7f8f3a000640 1 -- 172.21.15.129:0/2609627161 wait complete. 2024-09-17T03:53:58.896 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:53:58.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:53:59.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 -- 172.21.15.129:0/3382340081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4554004e60 msgr2=0x7f45540052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 --2- 172.21.15.129:0/3382340081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4554004e60 0x7f45540052b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f45700745e0 tx=0x7f4554005830 comp rx=0 tx=0).stop 2024-09-17T03:53:59.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f456f200640 1 -- 172.21.15.129:0/3382340081 reap_dead start 2024-09-17T03:53:59.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 -- 172.21.15.129:0/3382340081 shutdown_connections 2024-09-17T03:53:59.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 -- 172.21.15.129:0/3382340081 >> 172.21.15.129:0/3382340081 conn(0x7f45700d0160 msgr2=0x7f45700d0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:59.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 -- 172.21.15.129:0/3382340081 shutdown_connections 2024-09-17T03:53:59.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.121+0000 7f4575400640 1 -- 172.21.15.129:0/3382340081 wait complete. 2024-09-17T03:53:59.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.123+0000 7f4575400640 1 Processor -- start 2024-09-17T03:53:59.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.123+0000 7f4575400640 1 -- start start 2024-09-17T03:53:59.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 0x7f45700c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f457015b430 0x7f457015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f45700cb370 con 0x7f457015bd80 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f45700b59f0 con 0x7f4554004e60 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4575400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f45700cb4f0 con 0x7f457015b430 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44666/0 (socket says 172.21.15.129:44666) 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 -- 172.21.15.129:0/3028733244 learned_addr learned my addr 172.21.15.129:0/3028733244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:59.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456e800640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 0x7f45700c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f4565e00640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f457015b430 0x7f457015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 -- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f457015b430 msgr2=0x7f457015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f457015b430 0x7f457015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 -- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 msgr2=0x7f45700c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 0x7f45700c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456f200640 1 -- 172.21.15.129:0/3028733244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4570156f80 con 0x7f457015bd80 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.125+0000 7f456e800640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 0x7f45700c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f456f200640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f456802b680 tx=0x7f4568001c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4568039070 con 0x7f457015bd80 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4568003970 con 0x7f457015bd80 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4570159aa0 con 0x7f457015bd80 2024-09-17T03:53:59.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4568003d60 con 0x7f457015bd80 2024-09-17T03:53:59.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.126+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f457015ac30 con 0x7f457015bd80 2024-09-17T03:53:59.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.127+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45700cb9e0 con 0x7f457015bd80 2024-09-17T03:53:59.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.128+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4568006140 con 0x7f457015bd80 2024-09-17T03:53:59.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.129+0000 7f4567400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 0x7f454c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.129+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f45680ba420 con 0x7f457015bd80 2024-09-17T03:53:59.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.130+0000 7f456e800640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 0x7f454c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.130+0000 7f456e800640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 0x7f454c07b0a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4560009b60 tx=0x7f4560005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:59.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.134+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4570156f80 con 0x7f457015bd80 2024-09-17T03:53:59.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.315+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4570159830 con 0x7f457015bd80 2024-09-17T03:53:59.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.315+0000 7f4567400640 1 -- 172.21.15.129:0/3028733244 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) ==== 93+0+10 (secure 0 0 0) 0x7f45700cc440 con 0x7f457015bd80 2024-09-17T03:53:59.315 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:53:59.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 msgr2=0x7f454c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 0x7f454c07b0a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4560009b60 tx=0x7f4560005f90 comp rx=0 tx=0).stop 2024-09-17T03:53:59.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 msgr2=0x7f45701f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f456802b680 tx=0x7f4568001c50 comp rx=0 tx=0).stop 2024-09-17T03:53:59.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 shutdown_connections 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f454c078b90 0x7f454c07b0a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f457015bd80 0x7f45701f8e30 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f457015b430 0x7f457015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 --2- 172.21.15.129:0/3028733244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4554004e60 0x7f45700c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 >> 172.21.15.129:0/3028733244 conn(0x7f45700d0160 msgr2=0x7f457015fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:59.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 shutdown_connections 2024-09-17T03:53:59.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.318+0000 7f4575400640 1 -- 172.21.15.129:0/3028733244 wait complete. 2024-09-17T03:53:59.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-09-17T03:53:59.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 -- 172.21.15.129:0/4081625842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 msgr2=0x7fd1440cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 --2- 172.21.15.129:0/4081625842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440cc830 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd138009830 tx=0x7fd13802a480 comp rx=0 tx=0).stop 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 -- 172.21.15.129:0/4081625842 shutdown_connections 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 --2- 172.21.15.129:0/4081625842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 0x7fd14406ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 --2- 172.21.15.129:0/4081625842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440cc830 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 -- 172.21.15.129:0/4081625842 >> 172.21.15.129:0/4081625842 conn(0x7fd14406d410 msgr2=0x7fd1440b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:53:59.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 -- 172.21.15.129:0/4081625842 shutdown_connections 2024-09-17T03:53:59.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.562+0000 7fd14a800640 1 -- 172.21.15.129:0/4081625842 wait complete. 2024-09-17T03:53:59.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.564+0000 7fd14a800640 1 Processor -- start 2024-09-17T03:53:59.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.565+0000 7fd14a800640 1 -- start start 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 0x7fd144153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1440cf950 0x7fd1441472b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd14415de10 con 0x7fd14415ca50 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd14415df90 con 0x7fd1440cf950 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd14a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd14415e110 con 0x7fd1440cf540 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd149e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd149e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44686/0 (socket says 172.21.15.129:44686) 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd149e00640 1 -- 172.21.15.129:0/2865384489 learned_addr learned my addr 172.21.15.129:0/2865384489 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:53:59.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd149400640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 0x7fd144153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.566+0000 7fd148a00640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1440cf950 0x7fd1441472b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 -- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 msgr2=0x7fd144153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 0x7fd144153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 -- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1440cf950 msgr2=0x7fd1441472b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1440cf950 0x7fd1441472b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 -- 172.21.15.129:0/2865384489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1441563a0 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd149e00640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00caa0 tx=0x7fd13c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c016020 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd13c00abd0 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd144156680 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c012420 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.567+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd144156c40 con 0x7fd14415ca50 2024-09-17T03:53:59.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.568+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14406ee70 con 0x7fd14415ca50 2024-09-17T03:53:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.570+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd13c019020 con 0x7fd14415ca50 2024-09-17T03:53:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.570+0000 7fd141600640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:53:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.571+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7fd13c09ad60 con 0x7fd14415ca50 2024-09-17T03:53:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.571+0000 7fd149400640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:53:59.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.572+0000 7fd149400640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd138034000 tx=0x7fd138004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:53:59.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.575+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd13c020020 con 0x7fd14415ca50 2024-09-17T03:53:59.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:53:59.826+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) -- 0x7fd1441541e0 con 0x7fd14415ca50 2024-09-17T03:54:00.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.431+0000 7fd141600640 1 -- 172.21.15.129:0/2865384489 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v248) ==== 137+0+0 (secure 0 0 0) 0x7fd13c062f50 con 0x7fd14415ca50 2024-09-17T03:54:00.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 msgr2=0x7fd12407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fd138034000 tx=0x7fd138004730 comp rx=0 tx=0).stop 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 msgr2=0x7fd1440b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd13c00caa0 tx=0x7fd13c00cf60 comp rx=0 tx=0).stop 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 shutdown_connections 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd124078b90 0x7fd12407b0a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd14415ca50 0x7fd1440b0c70 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1440cf950 0x7fd1441472b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 --2- 172.21.15.129:0/2865384489 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1440cf540 0x7fd144153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 >> 172.21.15.129:0/2865384489 conn(0x7fd14406d410 msgr2=0x7fd144074040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:00.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 shutdown_connections 2024-09-17T03:54:00.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.434+0000 7fd14a800640 1 -- 172.21.15.129:0/2865384489 wait complete. 2024-09-17T03:54:00.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:54:00.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 -- 172.21.15.129:0/1219163475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14070440 msgr2=0x7f9e140b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 --2- 172.21.15.129:0/1219163475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14070440 0x7f9e140b3090 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9e14070e70 tx=0x7f9e0c02a4a0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 -- 172.21.15.129:0/1219163475 shutdown_connections 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 --2- 172.21.15.129:0/1219163475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14070440 0x7f9e140b3090 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 --2- 172.21.15.129:0/1219163475 >> v2:172.21.15.181:3300/0 conn(0x7f9e14073170 0x7f9e1415c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 -- 172.21.15.129:0/1219163475 >> 172.21.15.129:0/1219163475 conn(0x7f9e140b1cb0 msgr2=0x7f9e140b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 -- 172.21.15.129:0/1219163475 shutdown_connections 2024-09-17T03:54:00.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.682+0000 7f9e1c000640 1 -- 172.21.15.129:0/1219163475 wait complete. 2024-09-17T03:54:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.685+0000 7f9e1c000640 1 Processor -- start 2024-09-17T03:54:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.685+0000 7f9e1c000640 1 -- start start 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e14070030 0x7f9e14155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e14070440 0x7f9e140d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e140b3a40 con 0x7f9e14073170 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e140b38c0 con 0x7f9e14070440 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.686+0000 7f9e1c000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e140b3bc0 con 0x7f9e14070030 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:00.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44698/0 (socket says 172.21.15.129:44698) 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 -- 172.21.15.129:0/152103936 learned_addr learned my addr 172.21.15.129:0/152103936 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e19600640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e14070030 0x7f9e14155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e18c00640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e14070440 0x7f9e140d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 -- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e14070030 msgr2=0x7f9e14155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e14070030 0x7f9e14155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 -- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e14070440 msgr2=0x7f9e140d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e14070440 0x7f9e140d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 -- 172.21.15.129:0/152103936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1415b940 con 0x7f9e14073170 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.687+0000 7f9e1a000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0098f0 tx=0x7f9e0c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.688+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c005c90 con 0x7f9e14073170 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.688+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e0c031e50 con 0x7f9e14073170 2024-09-17T03:54:00.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.688+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e140c8b80 con 0x7f9e14073170 2024-09-17T03:54:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.688+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e0c005240 con 0x7f9e14073170 2024-09-17T03:54:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.688+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e140c9140 con 0x7f9e14073170 2024-09-17T03:54:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.689+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e140b40e0 con 0x7f9e14073170 2024-09-17T03:54:00.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.690+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e0c0053e0 con 0x7f9e14073170 2024-09-17T03:54:00.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.691+0000 7f9e11600640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 0x7f9df407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:00.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.691+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f9e0c0bba70 con 0x7f9e14073170 2024-09-17T03:54:00.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.692+0000 7f9e19600640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 0x7f9df407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:00.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.692+0000 7f9e19600640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 0x7f9df407b0a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e08005fd0 tx=0x7f9e08009500 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:00.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.696+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e0c03f030 con 0x7f9e14073170 2024-09-17T03:54:00.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.878+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f9e140b56d0 con 0x7f9e14073170 2024-09-17T03:54:00.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.878+0000 7f9e11600640 1 -- 172.21.15.129:0/152103936 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v45) ==== 59+0+2294 (secure 0 0 0) 0x7f9e0c031770 con 0x7f9e14073170 2024-09-17T03:54:00.878 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:54:00.878 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:54:00.879 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:54:00.880 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:54:00.881 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 msgr2=0x7f9df407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 0x7f9df407b0a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e08005fd0 tx=0x7f9e08009500 comp rx=0 tx=0).stop 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 msgr2=0x7f9e140ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c0098f0 tx=0x7f9e0c033c20 comp rx=0 tx=0).stop 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 shutdown_connections 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9df4078b90 0x7f9df407b0a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e14073170 0x7f9e140ce810 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e14070440 0x7f9e140d08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 --2- 172.21.15.129:0/152103936 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e14070030 0x7f9e14155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:00.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 >> 172.21.15.129:0/152103936 conn(0x7f9e140b1cb0 msgr2=0x7f9e140ad670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:00.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 shutdown_connections 2024-09-17T03:54:00.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:00.881+0000 7f9e1c000640 1 -- 172.21.15.129:0/152103936 wait complete. 2024-09-17T03:54:00.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:54:01.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 -- 172.21.15.129:0/3539596552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 msgr2=0x7fa8e8005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 --2- 172.21.15.129:0/3539596552 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa8e8005220 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa90415c9b0 tx=0x7fa8e8005830 comp rx=0 tx=0).stop 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa902800640 1 -- 172.21.15.129:0/3539596552 reap_dead start 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 -- 172.21.15.129:0/3539596552 shutdown_connections 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 -- 172.21.15.129:0/3539596552 >> 172.21.15.129:0/3539596552 conn(0x7fa90406d410 msgr2=0x7fa9040b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 -- 172.21.15.129:0/3539596552 shutdown_connections 2024-09-17T03:54:01.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.132+0000 7fa903200640 1 -- 172.21.15.129:0/3539596552 wait complete. 2024-09-17T03:54:01.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.134+0000 7fa903200640 1 Processor -- start 2024-09-17T03:54:01.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.135+0000 7fa903200640 1 -- start start 2024-09-17T03:54:01.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa90415e3f0 0x7fa9041e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa90415e800 0x7fa9041e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa9040b3c80 con 0x7fa8e8004d90 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa90415f870 con 0x7fa90415e800 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa903200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa9040b3e00 con 0x7fa90415e3f0 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa901e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa901e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44722/0 (socket says 172.21.15.129:44722) 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.136+0000 7fa901e00640 1 -- 172.21.15.129:0/3298350570 learned_addr learned my addr 172.21.15.129:0/3298350570 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901400640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa90415e3f0 0x7fa9041e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa902800640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa90415e800 0x7fa9041e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 -- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa90415e3f0 msgr2=0x7fa9041e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa90415e3f0 0x7fa9041e8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 -- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa90415e800 msgr2=0x7fa9041e9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa90415e800 0x7fa9041e9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 -- 172.21.15.129:0/3298350570 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90414c890 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa901e00640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006140 tx=0x7fa8f4006600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4015070 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa8f4006850 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9041f8df0 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.137+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f4011430 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.138+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9041f93b0 con 0x7fa8e8004d90 2024-09-17T03:54:01.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.139+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa904077030 con 0x7fa8e8004d90 2024-09-17T03:54:01.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.140+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa8f40115d0 con 0x7fa8e8004d90 2024-09-17T03:54:01.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.141+0000 7fa8fa000640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 0x7fa8e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.141+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7fa8f409b9c0 con 0x7fa8e8004d90 2024-09-17T03:54:01.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.141+0000 7fa901400640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 0x7fa8e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.142+0000 7fa901400640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 0x7fa8e407b0a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa90414cb90 tx=0x7fa8f0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:01.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.145+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa904077030 con 0x7fa8e8004d90 2024-09-17T03:54:01.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.325+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fa90415fbd0 con 0x7fa8e8004d90 2024-09-17T03:54:01.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.329+0000 7fa8fa000640 1 -- 172.21.15.129:0/3298350570 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) ==== 122+0+0 (secure 0 0 0) 0x7fa8f4064180 con 0x7fa8e8004d90 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 msgr2=0x7fa8e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 0x7fa8e407b0a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa90414cb90 tx=0x7fa8f0005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 msgr2=0x7fa904156980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4006140 tx=0x7fa8f4006600 comp rx=0 tx=0).stop 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 shutdown_connections 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa8e4078b90 0x7fa8e407b0a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa90415e800 0x7fa9041e9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa90415e3f0 0x7fa9041e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 --2- 172.21.15.129:0/3298350570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa8e8004d90 0x7fa904156980 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 >> 172.21.15.129:0/3298350570 conn(0x7fa90406d410 msgr2=0x7fa90406f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:01.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 shutdown_connections 2024-09-17T03:54:01.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.332+0000 7fa903200640 1 -- 172.21.15.129:0/3298350570 wait complete. 2024-09-17T03:54:01.356 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:54:01.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:54:01.444 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:54:01.534 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:54:01.534 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:01.534 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:01.570 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:54:01.570 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:54:01.570 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:01.607 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:54:01.608 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:01.608 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:01.645 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:54:01.645 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:54:01.645 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:01.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:54:01.684 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:54:01.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.904+0000 7f3bcf200640 1 -- 172.21.15.129:0/126127143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8073170 msgr2=0x7f3bc815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.904+0000 7f3bcf200640 1 --2- 172.21.15.129:0/126127143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8073170 0x7f3bc815c780 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80b4b40 tx=0x7f3bb802a560 comp rx=0 tx=0).stop 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 -- 172.21.15.129:0/126127143 shutdown_connections 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 --2- 172.21.15.129:0/126127143 >> v2:172.21.15.181:3300/0 conn(0x7f3bc8070440 0x7f3bc80b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 --2- 172.21.15.129:0/126127143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8073170 0x7f3bc815c780 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 -- 172.21.15.129:0/126127143 >> 172.21.15.129:0/126127143 conn(0x7f3bc80b1cb0 msgr2=0x7f3bc80b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:01.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 -- 172.21.15.129:0/126127143 shutdown_connections 2024-09-17T03:54:01.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.905+0000 7f3bcf200640 1 -- 172.21.15.129:0/126127143 wait complete. 2024-09-17T03:54:01.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.907+0000 7f3bcf200640 1 Processor -- start 2024-09-17T03:54:01.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.907+0000 7f3bcf200640 1 -- start start 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3bc8070440 0x7f3bc814b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc8073170 0x7f3bc809b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b3a40 con 0x7f3bc8070030 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b38c0 con 0x7f3bc8073170 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.908+0000 7f3bcf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc80b3bc0 con 0x7f3bc8070440 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44740/0 (socket says 172.21.15.129:44740) 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 -- 172.21.15.129:0/2319407531 learned_addr learned my addr 172.21.15.129:0/2319407531 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 -- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3bc8070440 msgr2=0x7f3bc814b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3bc8070440 0x7f3bc814b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 -- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc8073170 msgr2=0x7f3bc809b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc8073170 0x7f3bc809b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 -- 172.21.15.129:0/2319407531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc814bc80 con 0x7f3bc8070030 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.909+0000 7f3bc7e00640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8034040 tx=0x7f3bb80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.910+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb803e030 con 0x7f3bc8070030 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.910+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bb8003db0 con 0x7f3bc8070030 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.910+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8035040 con 0x7f3bc8070030 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.910+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc80c3b20 con 0x7f3bc8070030 2024-09-17T03:54:01.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.910+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc80c40e0 con 0x7f3bc8070030 2024-09-17T03:54:01.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.911+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc80b40e0 con 0x7f3bc8070030 2024-09-17T03:54:01.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.912+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3bb8041030 con 0x7f3bc8070030 2024-09-17T03:54:01.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.913+0000 7f3bbfe00640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 0x7f3ba807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:01.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.913+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f3bb80bb510 con 0x7f3bc8070030 2024-09-17T03:54:01.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.914+0000 7f3bc7400640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 0x7f3ba807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:01.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.914+0000 7f3bc7400640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 0x7f3ba807b0a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4000f60 tx=0x7f3bb40079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:01.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:01.918+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3bb8083c50 con 0x7f3bc8070030 2024-09-17T03:54:02.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.136+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bc80c3860 con 0x7f3bc8070030 2024-09-17T03:54:02.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.136+0000 7f3bbfe00640 1 -- 172.21.15.129:0/2319407531 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+3923 (secure 0 0 0) 0x7f3bb8083550 con 0x7f3bc8070030 2024-09-17T03:54:02.136 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:02.137 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":56,"btime":"2024-09-17T03:53:55:408800+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5519,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2357419611","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2357419611},{"type":"v1","addr":"172.21.15.181:6837","nonce":2357419611}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":48},{"gid":6040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4054488776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4054488776},{"type":"v1","addr":"172.21.15.181:6836","nonce":4054488776}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6046,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1219825057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1219825057},{"type":"v1","addr":"172.21.15.129:6834","nonce":1219825057}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49},{"gid":6450,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1792133616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1792133616},{"type":"v1","addr":"172.21.15.129:6835","nonce":1792133616}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[]} 2024-09-17T03:54:02.137 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 56 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 msgr2=0x7f3ba807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 0x7f3ba807b0a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4000f60 tx=0x7f3bb40079b0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 msgr2=0x7f3bc81f74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8034040 tx=0x7f3bb80038a0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 shutdown_connections 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ba8078b90 0x7f3ba807b0a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3bc8073170 0x7f3bc809b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3bc8070440 0x7f3bc814b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 --2- 172.21.15.129:0/2319407531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3bc8070030 0x7f3bc81f74e0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 >> 172.21.15.129:0/2319407531 conn(0x7f3bc80b1cb0 msgr2=0x7f3bc80ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:02.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 shutdown_connections 2024-09-17T03:54:02.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.139+0000 7f3bcf200640 1 -- 172.21.15.129:0/2319407531 wait complete. 2024-09-17T03:54:02.161 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:54:02.161 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:54:02.162 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:54:02.162 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:02.162 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:54:02.162 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:54:02.162 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:54:02.162 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:02.162 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:54:02.163 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:54:02.163 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:54:02.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:02.163 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:54:02.163 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:54:02.163 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:54:02.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:02.185 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:54:02.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:54:02.187 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:54:02.187 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:54:02.188 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:54:02.188 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:54:02.189 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:54:02.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:54:02.232 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:54:02.233 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:02.234 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:54:02.234 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:02.235 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:54:02.235 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:02.237 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:54:02.237 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:02.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:54:02.429 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:54:02.430 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:54:02.431 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:54:02.432 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 -- 172.21.15.129:0/503858153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 msgr2=0x7f112406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 --2- 172.21.15.129:0/503858153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f112406ee70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1110005fd0 tx=0x7f111002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 -- 172.21.15.129:0/503858153 shutdown_connections 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 --2- 172.21.15.129:0/503858153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f112406ee70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 --2- 172.21.15.129:0/503858153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 0x7f1124073160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 -- 172.21.15.129:0/503858153 >> 172.21.15.129:0/503858153 conn(0x7f112406d410 msgr2=0x7f11240b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 -- 172.21.15.129:0/503858153 shutdown_connections 2024-09-17T03:54:02.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.474+0000 7f1123e00640 1 -- 172.21.15.129:0/503858153 wait complete. 2024-09-17T03:54:02.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.476+0000 7f1123e00640 1 Processor -- start 2024-09-17T03:54:02.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.476+0000 7f1123e00640 1 -- start start 2024-09-17T03:54:02.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 0x7f11240c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:02.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f1124153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:02.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 0x7f112413d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:02.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f11240cf790 con 0x7f112406f3b0 2024-09-17T03:54:02.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1124070f70 con 0x7f1124151640 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f11240cf910 con 0x7f112415b8f0 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 0x7f11240c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 0x7f11240c4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44760/0 (socket says 172.21.15.129:44760) 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122a00640 1 -- 172.21.15.129:0/2657965097 learned_addr learned my addr 172.21.15.129:0/2657965097 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f1124153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1123400640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 0x7f112413d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 -- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 msgr2=0x7f112413d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 0x7f112413d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 -- 172.21.15.129:0/2657965097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 msgr2=0x7f11240c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 0x7f11240c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:02.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.478+0000 7f1122000640 1 -- 172.21.15.129:0/2657965097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11240ce250 con 0x7f1124151640 2024-09-17T03:54:02.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f1122000640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f1124153f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f11240b2ae0 tx=0x7f11100035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:02.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111003e030 con 0x7f1124151640 2024-09-17T03:54:02.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1110003b00 con 0x7f1124151640 2024-09-17T03:54:02.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11240ce530 con 0x7f1124151640 2024-09-17T03:54:02.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110035040 con 0x7f1124151640 2024-09-17T03:54:02.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.479+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112413cb30 con 0x7f1124151640 2024-09-17T03:54:02.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.481+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11240cdfc0 con 0x7f1124151640 2024-09-17T03:54:02.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.482+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1110041030 con 0x7f1124151640 2024-09-17T03:54:02.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.483+0000 7f111aa00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 0x7f110407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:02.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.483+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 4943+0+0 (secure 0 0 0) 0x7f11100bacf0 con 0x7f1124151640 2024-09-17T03:54:02.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.483+0000 7f1122a00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 0x7f110407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:02.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.484+0000 7f1122a00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 0x7f110407b0a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1114009c70 tx=0x7f1114005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:02.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.488+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11100bf070 con 0x7f1124151640 2024-09-17T03:54:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:02.682+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f112406ee70 con 0x7f1124151640 2024-09-17T03:54:03.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.458+0000 7f111aa00640 1 -- 172.21.15.129:0/2657965097 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v86) ==== 97+0+0 (secure 0 0 0) 0x7f1110082f00 con 0x7f1124151640 2024-09-17T03:54:03.458 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.461+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 msgr2=0x7f110407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.461+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 0x7f110407b0a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1114009c70 tx=0x7f1114005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.461+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 msgr2=0x7f1124153f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.461+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f1124153f10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f11240b2ae0 tx=0x7f11100035f0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 shutdown_connections 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1104078b90 0x7f110407b0a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f112415b8f0 0x7f112413d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1124151640 0x7f1124153f10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 --2- 172.21.15.129:0/2657965097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f112406f3b0 0x7f11240c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 >> 172.21.15.129:0/2657965097 conn(0x7f112406d410 msgr2=0x7f11240d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:03.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 shutdown_connections 2024-09-17T03:54:03.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.462+0000 7f1123e00640 1 -- 172.21.15.129:0/2657965097 wait complete. 2024-09-17T03:54:03.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:54:03.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 -- 172.21.15.129:0/767594079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1940004800 msgr2=0x7f1940004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 --2- 172.21.15.129:0/767594079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1940004800 0x7f1940004c10 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f194400c610 tx=0x7f1940005830 comp rx=0 tx=0).stop 2024-09-17T03:54:03.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 -- 172.21.15.129:0/767594079 shutdown_connections 2024-09-17T03:54:03.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 --2- 172.21.15.129:0/767594079 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1940004800 0x7f1940004c10 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 -- 172.21.15.129:0/767594079 >> 172.21.15.129:0/767594079 conn(0x7f195806d870 msgr2=0x7f195806ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:03.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 -- 172.21.15.129:0/767594079 shutdown_connections 2024-09-17T03:54:03.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.715+0000 7f195e000640 1 -- 172.21.15.129:0/767594079 wait complete. 2024-09-17T03:54:03.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.717+0000 7f195e000640 1 Processor -- start 2024-09-17T03:54:03.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.718+0000 7f195e000640 1 -- start start 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 0x7f19581f8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19581f9100 0x7f195815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19580cee40 con 0x7f1940004800 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19580cecc0 con 0x7f19581f5270 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f19580cefc0 con 0x7f19581f9100 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44780/0 (socket says 172.21.15.129:44780) 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.719+0000 7f195cc00640 1 -- 172.21.15.129:0/3706911009 learned_addr learned my addr 172.21.15.129:0/3706911009 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:03.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f1957e00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 0x7f19581f8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195d600640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19581f9100 0x7f195815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 -- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19581f9100 msgr2=0x7f195815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19581f9100 0x7f195815a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 -- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 msgr2=0x7f19581f8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 0x7f19581f8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 -- 172.21.15.129:0/3706911009 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19581f9890 con 0x7f1940004800 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f1957e00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 0x7f19581f8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195cc00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f194c009fd0 tx=0x7f194c011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c009720 con 0x7f1940004800 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f194c00fd30 con 0x7f1940004800 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195813ae20 con 0x7f1940004800 2024-09-17T03:54:03.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.720+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c005280 con 0x7f1940004800 2024-09-17T03:54:03.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.721+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19580c98e0 con 0x7f1940004800 2024-09-17T03:54:03.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.722+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195815bcf0 con 0x7f1940004800 2024-09-17T03:54:03.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.723+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f194c020020 con 0x7f1940004800 2024-09-17T03:54:03.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.724+0000 7f1954c00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 0x7f193807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:03.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.724+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f194c09b2e0 con 0x7f1940004800 2024-09-17T03:54:03.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.724+0000 7f1957e00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 0x7f193807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:03.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.725+0000 7f1957e00640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 0x7f193807b0a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1944033000 tx=0x7f1944001bc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:03.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.728+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f194c0639d0 con 0x7f1940004800 2024-09-17T03:54:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.977+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f19580b1ba0 con 0x7f1940004800 2024-09-17T03:54:03.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.978+0000 7f1954c00640 1 -- 172.21.15.129:0/3706911009 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) ==== 80+0+5304 (secure 0 0 0) 0x7f194c0632d0 con 0x7f1940004800 2024-09-17T03:54:03.978 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:03.979 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:54:03.979 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:54:03.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.979 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:54:03.979 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.980 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.981 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:54:03.982 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:54:03.984 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:54:03.985 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.986 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.987 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.988 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.990 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:54:03.992 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.995 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:54:03.996 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:54:03.997 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:54:03.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 msgr2=0x7f193807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 0x7f193807b0a0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1944033000 tx=0x7f1944001bc0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 msgr2=0x7f19581f4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f194c009fd0 tx=0x7f194c011ee0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 shutdown_connections 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1938078b90 0x7f193807b0a0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19581f9100 0x7f195815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19581f5270 0x7f19581f8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 --2- 172.21.15.129:0/3706911009 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1940004800 0x7f19581f4d30 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 >> 172.21.15.129:0/3706911009 conn(0x7f195806d870 msgr2=0x7f19580cc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 shutdown_connections 2024-09-17T03:54:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:03.981+0000 7f195e000640 1 -- 172.21.15.129:0/3706911009 wait complete. 2024-09-17T03:54:04.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:54:04.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 -- 172.21.15.129:0/1625574276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 msgr2=0x7f6ecc0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/1625574276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc0ccf60 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4005fd0 tx=0x7f6ec402a8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 -- 172.21.15.129:0/1625574276 shutdown_connections 2024-09-17T03:54:04.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/1625574276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc0ccf60 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/1625574276 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 0x7f6ecc0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 -- 172.21.15.129:0/1625574276 >> 172.21.15.129:0/1625574276 conn(0x7f6ecc06d7d0 msgr2=0x7f6ecc0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:04.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 -- 172.21.15.129:0/1625574276 shutdown_connections 2024-09-17T03:54:04.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.222+0000 7f6ed2c00640 1 -- 172.21.15.129:0/1625574276 wait complete. 2024-09-17T03:54:04.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.225+0000 7f6ed2c00640 1 Processor -- start 2024-09-17T03:54:04.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.225+0000 7f6ed2c00640 1 -- start start 2024-09-17T03:54:04.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 0x7f6ecc152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ecc15d8a0 0x7f6ecc09d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc153520 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc1533a0 con 0x7f6ecc15d8a0 2024-09-17T03:54:04.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ed2c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ecc1536a0 con 0x7f6ecc0cc1e0 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.226+0000 7f6ecb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 0x7f6ecc152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ed0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ecc15d8a0 0x7f6ecc09d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57704/0 (socket says 172.21.15.129:57704) 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 -- 172.21.15.129:0/2175153222 learned_addr learned my addr 172.21.15.129:0/2175153222 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:04.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 -- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 msgr2=0x7f6ecc152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 0x7f6ecc152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 -- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ecc15d8a0 msgr2=0x7f6ecc09d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ecc15d8a0 0x7f6ecc09d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 -- 172.21.15.129:0/2175153222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ecc1f6e00 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.227+0000 7f6ecb400640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800c3f0 tx=0x7f6eb800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.228+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8017070 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.228+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6eb80035f0 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.228+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ecc09b450 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.228+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8003a40 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.228+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ecc1f6920 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.229+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ecc0c9370 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.230+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6eb801a070 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.231+0000 7f6ec3e00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 0x7f6eac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.231+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7f6eb8099bc0 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.231+0000 7f6ecbe00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 0x7f6eac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.232+0000 7f6ecbe00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 0x7f6eac07b0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00bb50 tx=0x7f6ebc00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:04.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.235+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ecc0cd4a0 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.419+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6ecc153b50 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.420+0000 7f6ec3e00640 1 -- 172.21.15.129:0/2175153222 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v88) ==== 84+0+65 (secure 0 0 0) 0x7f6eb8061d90 con 0x7f6ecc0ccb50 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:54:04.421 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 msgr2=0x7f6eac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 0x7f6eac07b0d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00bb50 tx=0x7f6ebc00b2e0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 msgr2=0x7f6ecc09cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6eb800c3f0 tx=0x7f6eb800c8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 shutdown_connections 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6eac078bc0 0x7f6eac07b0d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ecc15d8a0 0x7f6ecc09d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ecc0ccb50 0x7f6ecc09cf40 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 --2- 172.21.15.129:0/2175153222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ecc0cc1e0 0x7f6ecc152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.422+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 >> 172.21.15.129:0/2175153222 conn(0x7f6ecc06d7d0 msgr2=0x7f6ecc15aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:04.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.423+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 shutdown_connections 2024-09-17T03:54:04.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.423+0000 7f6ed2c00640 1 -- 172.21.15.129:0/2175153222 wait complete. 2024-09-17T03:54:04.446 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:54:04.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 -- 172.21.15.129:0/3328185737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb610004d40 msgr2=0x7fb610005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 --2- 172.21.15.129:0/3328185737 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb610004d40 0x7fb610005150 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb624002230 tx=0x7fb610005830 comp rx=0 tx=0).stop 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb62e800640 1 -- 172.21.15.129:0/3328185737 reap_dead start 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 -- 172.21.15.129:0/3328185737 shutdown_connections 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 -- 172.21.15.129:0/3328185737 >> 172.21.15.129:0/3328185737 conn(0x7fb6280d0130 msgr2=0x7fb6280d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 -- 172.21.15.129:0/3328185737 shutdown_connections 2024-09-17T03:54:04.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.672+0000 7fb630800640 1 -- 172.21.15.129:0/3328185737 wait complete. 2024-09-17T03:54:04.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.675+0000 7fb630800640 1 Processor -- start 2024-09-17T03:54:04.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.675+0000 7fb630800640 1 -- start start 2024-09-17T03:54:04.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb610004d40 0x7fb62809d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb628154590 0x7fb6280b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6280707b0 con 0x7fb6281582d0 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb628070630 con 0x7fb610004d40 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb630800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb6280ca990 con 0x7fb628154590 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb62d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb62d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57728/0 (socket says 172.21.15.129:57728) 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.676+0000 7fb62d400640 1 -- 172.21.15.129:0/3906574643 learned_addr learned my addr 172.21.15.129:0/3906574643 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62e800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb628154590 0x7fb6280b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62de00640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb610004d40 0x7fb62809d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 -- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb628154590 msgr2=0x7fb6280b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb628154590 0x7fb6280b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 -- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb610004d40 msgr2=0x7fb62809d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb610004d40 0x7fb62809d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:04.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 -- 172.21.15.129:0/3906574643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb628154de0 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb62d400640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb618011ae0 tx=0x7fb618009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61801e070 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb618004460 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61800fc60 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.677+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6280b5110 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.678+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb628155df0 con 0x7fb6281582d0 2024-09-17T03:54:04.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.679+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6280d05d0 con 0x7fb6281582d0 2024-09-17T03:54:04.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.680+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb618021070 con 0x7fb6281582d0 2024-09-17T03:54:04.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.681+0000 7fb622000640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 0x7fb60c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:04.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.681+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 4857+0+0 (secure 0 0 0) 0x7fb61809b510 con 0x7fb6281582d0 2024-09-17T03:54:04.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.682+0000 7fb62de00640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 0x7fb60c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:04.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.682+0000 7fb62de00640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 0x7fb60c07b0a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb628070da0 tx=0x7fb61c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:04.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.685+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb6280d05d0 con 0x7fb6281582d0 2024-09-17T03:54:04.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:04.868+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb628157f10 con 0x7fb6281582d0 2024-09-17T03:54:05.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.470+0000 7fb622000640 1 -- 172.21.15.129:0/3906574643 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v89) ==== 138+0+0 (secure 0 0 0) 0x7fb6180635b0 con 0x7fb6281582d0 2024-09-17T03:54:05.470 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 msgr2=0x7fb60c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 0x7fb60c07b0a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb628070da0 tx=0x7fb61c005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 msgr2=0x7fb6280b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb618011ae0 tx=0x7fb618009e40 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 shutdown_connections 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb60c078b90 0x7fb60c07b0a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb628154590 0x7fb6280b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6281582d0 0x7fb6280b4690 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 --2- 172.21.15.129:0/3906574643 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb610004d40 0x7fb62809d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 >> 172.21.15.129:0/3906574643 conn(0x7fb6280d0130 msgr2=0x7fb6280acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:05.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 shutdown_connections 2024-09-17T03:54:05.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.473+0000 7fb630800640 1 -- 172.21.15.129:0/3906574643 wait complete. 2024-09-17T03:54:05.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.732+0000 7eff77200640 1 -- 172.21.15.129:0/2344001916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 msgr2=0x7eff5c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.732+0000 7eff77200640 1 --2- 172.21.15.129:0/2344001916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff5c003e70 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7eff6800c610 tx=0x7eff5c005830 comp rx=0 tx=0).stop 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.732+0000 7eff76800640 1 -- 172.21.15.129:0/2344001916 reap_dead start 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.732+0000 7eff77200640 1 -- 172.21.15.129:0/2344001916 shutdown_connections 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.732+0000 7eff77200640 1 -- 172.21.15.129:0/2344001916 >> 172.21.15.129:0/2344001916 conn(0x7eff7806ee50 msgr2=0x7eff7806f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.733+0000 7eff77200640 1 -- 172.21.15.129:0/2344001916 shutdown_connections 2024-09-17T03:54:05.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.733+0000 7eff77200640 1 -- 172.21.15.129:0/2344001916 wait complete. 2024-09-17T03:54:05.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.735+0000 7eff77200640 1 Processor -- start 2024-09-17T03:54:05.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.735+0000 7eff77200640 1 -- start start 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff781f69f0 0x7eff781f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 0x7eff780c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff780caeb0 con 0x7eff5c003a00 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7eff780cad30 con 0x7eff781f69f0 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.736+0000 7eff77200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7eff780cb030 con 0x7eff780c87c0 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:05.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57750/0 (socket says 172.21.15.129:57750) 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 -- 172.21.15.129:0/1755375064 learned_addr learned my addr 172.21.15.129:0/1755375064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff6d400640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff781f69f0 0x7eff781f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff76800640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 0x7eff780c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 -- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 msgr2=0x7eff780c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 0x7eff780c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 -- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff781f69f0 msgr2=0x7eff781f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff781f69f0 0x7eff781f6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 -- 172.21.15.129:0/1755375064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff780c9110 con 0x7eff5c003a00 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff76800640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 0x7eff780c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.737+0000 7eff75e00640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7eff6800c610 tx=0x7eff68004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.738+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6802bc10 con 0x7eff5c003a00 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.738+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff6802bdb0 con 0x7eff5c003a00 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.738+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff68006320 con 0x7eff5c003a00 2024-09-17T03:54:05.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.738+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff781f7760 con 0x7eff5c003a00 2024-09-17T03:54:05.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.738+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff781f47f0 con 0x7eff5c003a00 2024-09-17T03:54:05.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.739+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff780cb5c0 con 0x7eff5c003a00 2024-09-17T03:54:05.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.741+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7eff680064c0 con 0x7eff5c003a00 2024-09-17T03:54:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.742+0000 7eff6ea00640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 0x7eff5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.742+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5227+0+0 (secure 0 0 0) 0x7eff680bb660 con 0x7eff5c003a00 2024-09-17T03:54:05.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.743+0000 7eff6d400640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 0x7eff5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:05.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.743+0000 7eff6d400640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 0x7eff5407b0a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7eff7815f910 tx=0x7eff60005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:05.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.746+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7eff780c9110 con 0x7eff5c003a00 2024-09-17T03:54:05.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:05.929+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7eff780b17a0 con 0x7eff5c003a00 2024-09-17T03:54:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.476+0000 7eff6ea00640 1 -- 172.21.15.129:0/1755375064 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v90) ==== 170+0+0 (secure 0 0 0) 0x7eff68083c60 con 0x7eff5c003a00 2024-09-17T03:54:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:54:06.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 msgr2=0x7eff5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 0x7eff5407b0a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7eff7815f910 tx=0x7eff60005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 msgr2=0x7eff7814e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7eff6800c610 tx=0x7eff68004c30 comp rx=0 tx=0).stop 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 shutdown_connections 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff54078b90 0x7eff5407b0a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff780c87c0 0x7eff780c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff781f69f0 0x7eff781f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 --2- 172.21.15.129:0/1755375064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff5c003a00 0x7eff7814e8e0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 >> 172.21.15.129:0/1755375064 conn(0x7eff7806ee50 msgr2=0x7eff780d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:06.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.480+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 shutdown_connections 2024-09-17T03:54:06.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.481+0000 7eff77200640 1 -- 172.21.15.129:0/1755375064 wait complete. 2024-09-17T03:54:06.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 -- 172.21.15.129:0/1493908014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00b96b0 msgr2=0x7f1dd00d0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 --2- 172.21.15.129:0/1493908014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00b96b0 0x7f1dd00d0de0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0009850 tx=0x7f1dc002a480 comp rx=0 tx=0).stop 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 -- 172.21.15.129:0/1493908014 shutdown_connections 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 --2- 172.21.15.129:0/1493908014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00b96b0 0x7f1dd00d0de0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 -- 172.21.15.129:0/1493908014 >> 172.21.15.129:0/1493908014 conn(0x7f1dd009d680 msgr2=0x7f1dd00d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 -- 172.21.15.129:0/1493908014 shutdown_connections 2024-09-17T03:54:06.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.751+0000 7f1dd6000640 1 -- 172.21.15.129:0/1493908014 wait complete. 2024-09-17T03:54:06.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.754+0000 7f1dd6000640 1 Processor -- start 2024-09-17T03:54:06.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.754+0000 7f1dd6000640 1 -- start start 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dd00b42f0 0x7f1dd0169a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00d1320 0x7f1dd0166a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd00b5040 con 0x7f1dd00b96b0 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1dd00b4ec0 con 0x7f1dd00b42f0 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dd6000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1dd016a0a0 con 0x7f1dd00d1320 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.755+0000 7f1dc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:06.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57758/0 (socket says 172.21.15.129:57758) 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 -- 172.21.15.129:0/4255016033 learned_addr learned my addr 172.21.15.129:0/4255016033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dd5600640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00d1320 0x7f1dd0166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dd4c00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dd00b42f0 0x7f1dd0169a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 -- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00d1320 msgr2=0x7f1dd0166a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00d1320 0x7f1dd0166a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 -- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dd00b42f0 msgr2=0x7f1dd0169a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dd00b42f0 0x7f1dd0169a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:06.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 -- 172.21.15.129:0/4255016033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dd015c9b0 con 0x7f1dd00b96b0 2024-09-17T03:54:06.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.756+0000 7f1dc7e00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1db800c3c0 tx=0x7f1db800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:06.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.757+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8017070 con 0x7f1dd00b96b0 2024-09-17T03:54:06.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.757+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1db80053c0 con 0x7f1dd00b96b0 2024-09-17T03:54:06.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.757+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dd0168a30 con 0x7f1dd00b96b0 2024-09-17T03:54:06.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.757+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db8003850 con 0x7f1dd00b96b0 2024-09-17T03:54:06.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.757+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dd0168cd0 con 0x7f1dd00b96b0 2024-09-17T03:54:06.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.758+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dd0167340 con 0x7f1dd00b96b0 2024-09-17T03:54:06.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.759+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1db801a070 con 0x7f1dd00b96b0 2024-09-17T03:54:06.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.760+0000 7f1dcd600640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 0x7f1dac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:06.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.760+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 5609+0+0 (secure 0 0 0) 0x7f1db8099f70 con 0x7f1dd00b96b0 2024-09-17T03:54:06.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.760+0000 7f1dd4c00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 0x7f1dac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:06.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.761+0000 7f1dd4c00640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 0x7f1dac07b0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0034000 tx=0x7f1dc0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:06.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.765+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1db8061db0 con 0x7f1dd00b96b0 2024-09-17T03:54:06.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:06.983+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f1dd00d0de0 con 0x7f1dd00b96b0 2024-09-17T03:54:07.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.478+0000 7f1dcd600640 1 -- 172.21.15.129:0/4255016033 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v59) ==== 174+0+0 (secure 0 0 0) 0x7f1db80667e0 con 0x7f1dd00b96b0 2024-09-17T03:54:07.478 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 14 and data pool 15 2024-09-17T03:54:07.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 msgr2=0x7f1dac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 0x7f1dac07b0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0034000 tx=0x7f1dc0004730 comp rx=0 tx=0).stop 2024-09-17T03:54:07.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 msgr2=0x7f1dd015c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f1db800c3c0 tx=0x7f1db800c880 comp rx=0 tx=0).stop 2024-09-17T03:54:07.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 shutdown_connections 2024-09-17T03:54:07.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dac078bc0 0x7f1dac07b0d0 secure :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0034000 tx=0x7f1dc0004730 comp rx=0 tx=0).stop 2024-09-17T03:54:07.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dd00d1320 0x7f1dd0166a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dd00b96b0 0x7f1dd015c470 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 --2- 172.21.15.129:0/4255016033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dd00b42f0 0x7f1dd0169a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 >> 172.21.15.129:0/4255016033 conn(0x7f1dd009d680 msgr2=0x7f1dd00d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:07.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.482+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 shutdown_connections 2024-09-17T03:54:07.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.483+0000 7f1dd6000640 1 -- 172.21.15.129:0/4255016033 wait complete. 2024-09-17T03:54:07.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:54:07.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.756+0000 7fae3a400640 1 -- 172.21.15.129:0/1384628470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae3415f660 msgr2=0x7fae340b3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.756+0000 7fae3a400640 1 --2- 172.21.15.129:0/1384628470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae3415f660 0x7fae340b3570 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae2c005fd0 tx=0x7fae2c02aa10 comp rx=0 tx=0).stop 2024-09-17T03:54:07.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 -- 172.21.15.129:0/1384628470 shutdown_connections 2024-09-17T03:54:07.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 --2- 172.21.15.129:0/1384628470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae3415f660 0x7fae340b3570 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 --2- 172.21.15.129:0/1384628470 >> v2:172.21.15.181:3300/0 conn(0x7fae3415fbd0 0x7fae34152360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 -- 172.21.15.129:0/1384628470 >> 172.21.15.129:0/1384628470 conn(0x7fae3406d410 msgr2=0x7fae340b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:07.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 -- 172.21.15.129:0/1384628470 shutdown_connections 2024-09-17T03:54:07.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.757+0000 7fae3a400640 1 -- 172.21.15.129:0/1384628470 wait complete. 2024-09-17T03:54:07.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.759+0000 7fae3a400640 1 Processor -- start 2024-09-17T03:54:07.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.759+0000 7fae3a400640 1 -- start start 2024-09-17T03:54:07.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 0x7fae340c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:07.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae341f9330 0x7fae3414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae340b3fe0 con 0x7fae340ca000 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae340b3e60 con 0x7fae34072c30 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.760+0000 7fae3a400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fae340b4160 con 0x7fae341f9330 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 0x7fae340c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57764/0 (socket says 172.21.15.129:57764) 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39a00640 1 -- 172.21.15.129:0/1992757319 learned_addr learned my addr 172.21.15.129:0/1992757319 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae33e00640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae341f9330 0x7fae3414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 -- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae341f9330 msgr2=0x7fae3414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae341f9330 0x7fae3414d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 -- 172.21.15.129:0/1992757319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 msgr2=0x7fae340ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39000640 1 -- 172.21.15.129:0/1992757319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae340ca950 con 0x7fae34072c30 2024-09-17T03:54:07.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.761+0000 7fae39a00640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:54:07.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae39000640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 0x7fae340c9380 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e9f0 tx=0x7fae2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:07.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800ccc0 con 0x7fae34072c30 2024-09-17T03:54:07.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae28004020 con 0x7fae34072c30 2024-09-17T03:54:07.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800ce60 con 0x7fae34072c30 2024-09-17T03:54:07.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3415e5b0 con 0x7fae34072c30 2024-09-17T03:54:07.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.762+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae341f8b30 con 0x7fae34072c30 2024-09-17T03:54:07.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.763+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae340b4680 con 0x7fae34072c30 2024-09-17T03:54:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.765+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fae2803bd10 con 0x7fae34072c30 2024-09-17T03:54:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.765+0000 7fae30c00640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 0x7fae1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:07.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.766+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5709+0+0 (secure 0 0 0) 0x7fae2809b5e0 con 0x7fae34072c30 2024-09-17T03:54:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.766+0000 7fae33e00640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 0x7fae1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:07.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.767+0000 7fae33e00640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 0x7fae1407b0a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fae341ec3e0 tx=0x7fae20009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:07.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.770+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fae340ca950 con 0x7fae34072c30 2024-09-17T03:54:07.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.955+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae340750a0 con 0x7fae34072c30 2024-09-17T03:54:07.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.956+0000 7fae30c00640 1 -- 172.21.15.129:0/1992757319 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16729 (secure 0 0 0) 0x7fae280633b0 con 0x7fae34072c30 2024-09-17T03:54:07.957 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:07.957 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":91,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:06.985293+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:54:04.881426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-17T03:54:05.942429+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 msgr2=0x7fae1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 0x7fae1407b0a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fae341ec3e0 tx=0x7fae20009040 comp rx=0 tx=0).stop 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 msgr2=0x7fae340c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 0x7fae340c9380 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fae2800e9f0 tx=0x7fae2800eeb0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 shutdown_connections 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae14078b90 0x7fae1407b0a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae340ca000 0x7fae340ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae341f9330 0x7fae3414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 --2- 172.21.15.129:0/1992757319 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae34072c30 0x7fae340c9380 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 >> 172.21.15.129:0/1992757319 conn(0x7fae3406d410 msgr2=0x7fae3415cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 shutdown_connections 2024-09-17T03:54:07.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:07.959+0000 7fae3a400640 1 -- 172.21.15.129:0/1992757319 wait complete. 2024-09-17T03:54:07.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:54:08.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 -- 172.21.15.129:0/1635644099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec0ccb50 msgr2=0x7fb7ec0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/1635644099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec0ccf60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec071d60 tx=0x7fb7e402aa30 comp rx=0 tx=0).stop 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 -- 172.21.15.129:0/1635644099 shutdown_connections 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/1635644099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec0ccf60 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/1635644099 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec15d8a0 0x7fb7ec0b40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/1635644099 >> v2:172.21.15.181:3300/0 conn(0x7fb7ec0cc1e0 0x7fb7ec0af3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 -- 172.21.15.129:0/1635644099 >> 172.21.15.129:0/1635644099 conn(0x7fb7ec06d7d0 msgr2=0x7fb7ec0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 -- 172.21.15.129:0/1635644099 shutdown_connections 2024-09-17T03:54:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.202+0000 7fb7f3e00640 1 -- 172.21.15.129:0/1635644099 wait complete. 2024-09-17T03:54:08.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.204+0000 7fb7f3e00640 1 Processor -- start 2024-09-17T03:54:08.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.204+0000 7fb7f3e00640 1 -- start start 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec0cc1e0 0x7fb7ec150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec1e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0b4770 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb7ec0b45f0 con 0x7fb7ec0ccb50 2024-09-17T03:54:08.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb7ec1533a0 con 0x7fb7ec0cc1e0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57778/0 (socket says 172.21.15.129:57778) 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 -- 172.21.15.129:0/3432603656 learned_addr learned my addr 172.21.15.129:0/3432603656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1400640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec0cc1e0 0x7fb7ec150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f0a00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec1e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 -- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec0cc1e0 msgr2=0x7fb7ec150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec0cc1e0 0x7fb7ec150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 -- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0ccb50 msgr2=0x7fb7ec1e9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.206+0000 7fb7f1e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec1e9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7f1e00640 1 -- 172.21.15.129:0/3432603656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7ec1f9aa0 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7f1e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb7e402b020 tx=0x7fb7e4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:08.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4033050 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7e4004450 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4038450 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7ec1f9f80 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.207+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7ec09adb0 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.208+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7ec0ccf60 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.209+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb7e4036070 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.210+0000 7fb7e9600640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 0x7fb7cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.210+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 5709+0+0 (secure 0 0 0) 0x7fb7e40bc090 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.211+0000 7fb7f1400640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 0x7fb7cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.211+0000 7fb7f1400640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 0x7fb7cc07b0a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009ab0 tx=0x7fb7e0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:08.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.215+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb7ec1f9aa0 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.398+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7ec1f9cd0 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.399+0000 7fb7e9600640 1 -- 172.21.15.129:0/3432603656 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+16729 (secure 0 0 0) 0x7fb7e4083df0 con 0x7fb7ec15d8a0 2024-09-17T03:54:08.400 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:08.400 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":91,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:06.985293+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:54:04.881426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-17T03:54:05.942429+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:54:08.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 msgr2=0x7fb7cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 0x7fb7cc07b0a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0009ab0 tx=0x7fb7e0005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:08.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 msgr2=0x7fb7ec1e9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb7e402b020 tx=0x7fb7e4003f10 comp rx=0 tx=0).stop 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 shutdown_connections 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7cc078b90 0x7fb7cc07b0a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7ec15d8a0 0x7fb7ec1e9850 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7ec0ccb50 0x7fb7ec1e9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 --2- 172.21.15.129:0/3432603656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7ec0cc1e0 0x7fb7ec150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 >> 172.21.15.129:0/3432603656 conn(0x7fb7ec06d7d0 msgr2=0x7fb7ec06f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 shutdown_connections 2024-09-17T03:54:08.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.402+0000 7fb7f3e00640 1 -- 172.21.15.129:0/3432603656 wait complete. 2024-09-17T03:54:08.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 -- 172.21.15.129:0/1593747138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 msgr2=0x7fd750074b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 --2- 172.21.15.129:0/1593747138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd750074b80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fd750070e70 tx=0x7fd74802a980 comp rx=0 tx=0).stop 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 -- 172.21.15.129:0/1593747138 shutdown_connections 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 --2- 172.21.15.129:0/1593747138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd750074b80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 --2- 172.21.15.129:0/1593747138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750070030 0x7fd750074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 --2- 172.21.15.129:0/1593747138 >> v2:172.21.15.181:3300/0 conn(0x7fd750073170 0x7fd75015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 -- 172.21.15.129:0/1593747138 >> 172.21.15.129:0/1593747138 conn(0x7fd7500b1cb0 msgr2=0x7fd7500b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.657+0000 7fd756000640 1 -- 172.21.15.129:0/1593747138 shutdown_connections 2024-09-17T03:54:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.658+0000 7fd756000640 1 -- 172.21.15.129:0/1593747138 wait complete. 2024-09-17T03:54:08.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.660+0000 7fd756000640 1 Processor -- start 2024-09-17T03:54:08.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.660+0000 7fd756000640 1 -- start start 2024-09-17T03:54:08.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd750070030 0x7fd75009db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 0x7fd7500d0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd75006f690 con 0x7fd750070440 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd75006f510 con 0x7fd750070030 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd756000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd75006f810 con 0x7fd750073170 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.661+0000 7fd74ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57790/0 (socket says 172.21.15.129:57790) 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 -- 172.21.15.129:0/2819740495 learned_addr learned my addr 172.21.15.129:0/2819740495 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74f400640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd750070030 0x7fd75009db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74fe00640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 0x7fd7500d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 -- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 msgr2=0x7fd7500d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 0x7fd7500d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 -- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd750070030 msgr2=0x7fd75009db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd750070030 0x7fd75009db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 -- 172.21.15.129:0/2819740495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7500cf210 con 0x7fd750070440 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74fe00640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 0x7fd7500d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd74ea00640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c970 tx=0x7fd73c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c007bb0 con 0x7fd750070440 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd73c007e40 con 0x7fd750070440 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.662+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7500cf4f0 con 0x7fd750070440 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.663+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd73c00f6b0 con 0x7fd750070440 2024-09-17T03:54:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.663+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7500cfab0 con 0x7fd750070440 2024-09-17T03:54:08.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.664+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd75015c780 con 0x7fd750070440 2024-09-17T03:54:08.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.665+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd73c00f850 con 0x7fd750070440 2024-09-17T03:54:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.666+0000 7fd747400640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 0x7fd73007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.666+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd73c09b5a0 con 0x7fd750070440 2024-09-17T03:54:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.666+0000 7fd74f400640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 0x7fd73007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.667+0000 7fd74f400640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 0x7fd73007b0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd74000ca40 tx=0x7fd74000c420 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.670+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd73c014030 con 0x7fd750070440 2024-09-17T03:54:08.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:08.893+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd7500c3860 con 0x7fd750070440 2024-09-17T03:54:09.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.498+0000 7fd747400640 1 -- 172.21.15.129:0/2819740495 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) ==== 118+0+0 (secure 0 0 0) 0x7fd73c063a00 con 0x7fd750070440 2024-09-17T03:54:09.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 msgr2=0x7fd73007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:09.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 0x7fd73007b0d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd74000ca40 tx=0x7fd74000c420 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 msgr2=0x7fd7500cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fd73c00c970 tx=0x7fd73c00ce30 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 shutdown_connections 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd730078bc0 0x7fd73007b0d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd750073170 0x7fd7500d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd750070440 0x7fd7500cffd0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 --2- 172.21.15.129:0/2819740495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd750070030 0x7fd75009db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 >> 172.21.15.129:0/2819740495 conn(0x7fd7500b1cb0 msgr2=0x7fd7500afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 shutdown_connections 2024-09-17T03:54:09.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.501+0000 7fd756000640 1 -- 172.21.15.129:0/2819740495 wait complete. 2024-09-17T03:54:09.525 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:54:09.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:54:09.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 -- 172.21.15.129:0/2228745497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0070030 msgr2=0x7f0de0074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:09.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 --2- 172.21.15.129:0/2228745497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0070030 0x7f0de0074640 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009860 tx=0x7f0dcc02a5a0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 -- 172.21.15.129:0/2228745497 shutdown_connections 2024-09-17T03:54:09.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 --2- 172.21.15.129:0/2228745497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0070030 0x7f0de0074640 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 -- 172.21.15.129:0/2228745497 >> 172.21.15.129:0/2228745497 conn(0x7f0de00b1cb0 msgr2=0x7f0de00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:09.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 -- 172.21.15.129:0/2228745497 shutdown_connections 2024-09-17T03:54:09.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.752+0000 7f0de8200640 1 -- 172.21.15.129:0/2228745497 wait complete. 2024-09-17T03:54:09.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.754+0000 7f0de8200640 1 Processor -- start 2024-09-17T03:54:09.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.754+0000 7f0de8200640 1 -- start start 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.755+0000 7f0de8200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de8200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 0x7f0de0147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de8200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 0x7f0de014adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de8200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0de00b3cf0 con 0x7f0de0070030 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de8200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0de00b3b70 con 0x7f0de0070440 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de8200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0de00b3e70 con 0x7f0de0073170 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:09.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57802/0 (socket says 172.21.15.129:57802) 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 -- 172.21.15.129:0/2184709491 learned_addr learned my addr 172.21.15.129:0/2184709491 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de4e00640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 0x7f0de0147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de6200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 0x7f0de014adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 -- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 msgr2=0x7f0de014adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 0x7f0de014adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 -- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 msgr2=0x7f0de0147c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 0x7f0de0147c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de5800640 1 -- 172.21.15.129:0/2184709491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de014a610 con 0x7f0de0070030 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de4e00640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 0x7f0de0147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.756+0000 7f0de6200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 0x7f0de014adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0de5800640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c440 tx=0x7f0dd400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:09.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4016020 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de009c8e0 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0dd4012920 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4012d70 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.757+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de009cea0 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.758+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de0074640 con 0x7f0de0070030 2024-09-17T03:54:09.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.759+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0dd4019020 con 0x7f0de0070030 2024-09-17T03:54:09.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.760+0000 7f0ddd600640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 0x7f0dc007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:09.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.760+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f0dd409aa40 con 0x7f0de0070030 2024-09-17T03:54:09.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.761+0000 7f0de4e00640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 0x7f0dc007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:09.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.762+0000 7f0de4e00640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 0x7f0dc007b0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc034000 tx=0x7f0dcc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:09.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.765+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0dd4020030 con 0x7f0de0070030 2024-09-17T03:54:09.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:09.988+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0de00c3860 con 0x7f0de0070030 2024-09-17T03:54:10.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.512+0000 7f0ddd600640 1 -- 172.21.15.129:0/2184709491 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v63) ==== 122+0+0 (secure 0 0 0) 0x7f0dd4062e90 con 0x7f0de0070030 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 msgr2=0x7f0dc007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 0x7f0dc007b0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc034000 tx=0x7f0dcc004730 comp rx=0 tx=0).stop 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 msgr2=0x7f0de009c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400c440 tx=0x7f0dd400c900 comp rx=0 tx=0).stop 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 shutdown_connections 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0dc0078bc0 0x7f0dc007b0d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0de0073170 0x7f0de014adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0de0070440 0x7f0de0147c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 --2- 172.21.15.129:0/2184709491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0de0070030 0x7f0de009c0b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 >> 172.21.15.129:0/2184709491 conn(0x7f0de00b1cb0 msgr2=0x7f0de00acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 shutdown_connections 2024-09-17T03:54:10.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.515+0000 7f0de8200640 1 -- 172.21.15.129:0/2184709491 wait complete. 2024-09-17T03:54:10.538 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:54:10.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 -- 172.21.15.129:0/3059290694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0360004800 msgr2=0x7f0360004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 --2- 172.21.15.129:0/3059290694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0360004800 0x7f0360004c10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0364009b50 tx=0x7f0360005830 comp rx=0 tx=0).stop 2024-09-17T03:54:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0380e00640 1 -- 172.21.15.129:0/3059290694 reap_dead start 2024-09-17T03:54:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 -- 172.21.15.129:0/3059290694 shutdown_connections 2024-09-17T03:54:10.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 -- 172.21.15.129:0/3059290694 >> 172.21.15.129:0/3059290694 conn(0x7f037c06d870 msgr2=0x7f037c06ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:10.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 -- 172.21.15.129:0/3059290694 shutdown_connections 2024-09-17T03:54:10.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.760+0000 7f0381800640 1 -- 172.21.15.129:0/3059290694 wait complete. 2024-09-17T03:54:10.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.762+0000 7f0381800640 1 Processor -- start 2024-09-17T03:54:10.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.762+0000 7f0381800640 1 -- start start 2024-09-17T03:54:10.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 0x7f037c1499f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:10.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f037c1f9700 0x7f037c1f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:10.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:10.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15bb80 con 0x7f037c1f9280 2024-09-17T03:54:10.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f037c15ba00 con 0x7f037c1f9700 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0381800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f037c15bd00 con 0x7f0360004800 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0380e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f037be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 0x7f037c1499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f037be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 0x7f037c1499f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:58524/0 (socket says 172.21.15.129:58524) 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f037be00640 1 -- 172.21.15.129:0/2660516526 learned_addr learned my addr 172.21.15.129:0/2660516526 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0380e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57818/0 (socket says 172.21.15.129:57818) 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.764+0000 7f0373400640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f037c1f9700 0x7f037c1f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 -- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 msgr2=0x7f037c1499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 0x7f037c1499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 -- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f037c1f9700 msgr2=0x7f037c1f8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f037c1f9700 0x7f037c1f8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 -- 172.21.15.129:0/2660516526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f037c149f30 con 0x7f037c1f9280 2024-09-17T03:54:10.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0380e00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f037c070e50 tx=0x7f037400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037400eea0 con 0x7f037c1f9280 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0374015070 con 0x7f037c1f9280 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0374003dd0 con 0x7f037c1f9280 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037c0c9df0 con 0x7f037c1f9280 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.765+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037c0ca370 con 0x7f037c1f9280 2024-09-17T03:54:10.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.766+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f037c074fc0 con 0x7f037c1f9280 2024-09-17T03:54:10.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.767+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0374004030 con 0x7f037c1f9280 2024-09-17T03:54:10.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.768+0000 7f0378c00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 0x7f035807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:10.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.769+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f037409b9d0 con 0x7f037c1f9280 2024-09-17T03:54:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.769+0000 7f037be00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 0x7f035807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:10.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.770+0000 7f037be00640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 0x7f035807b0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f036c001eb0 tx=0x7f036c006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:10.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.773+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f037c074fc0 con 0x7f037c1f9280 2024-09-17T03:54:10.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.992+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f037c073e20 con 0x7f037c1f9280 2024-09-17T03:54:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.993+0000 7f0378c00640 1 -- 172.21.15.129:0/2660516526 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5234 (secure 0 0 0) 0x7f037c149f30 con 0x7f037c1f9280 2024-09-17T03:54:10.993 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:10.993 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":63,"btime":"2024-09-17T03:54:10:507817+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3749438691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3749438691},{"type":"v1","addr":"172.21.15.181:6836","nonce":3749438691}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":5726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2740088493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2740088493},{"type":"v1","addr":"172.21.15.181:6837","nonce":2740088493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":6358,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/118663331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":118663331},{"type":"v1","addr":"172.21.15.129:6835","nonce":118663331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:09.990095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1986086177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1986086177},{"type":"v1","addr":"172.21.15.129:6834","nonce":1986086177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6546,"qdb_cluster":[6546]},"id":7}]} 2024-09-17T03:54:10.993 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 63 2024-09-17T03:54:10.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 msgr2=0x7f035807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 0x7f035807b0d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f036c001eb0 tx=0x7f036c006040 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 msgr2=0x7f037c1518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f037c070e50 tx=0x7f037400e720 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 shutdown_connections 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0358078bc0 0x7f035807b0d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f037c1f9280 0x7f037c1518e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f037c1f9700 0x7f037c1f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.995+0000 7f0381800640 1 --2- 172.21.15.129:0/2660516526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0360004800 0x7f037c1499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.996+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 >> 172.21.15.129:0/2660516526 conn(0x7f037c06d870 msgr2=0x7f037c0b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.996+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 shutdown_connections 2024-09-17T03:54:10.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:10.996+0000 7f0381800640 1 -- 172.21.15.129:0/2660516526 wait complete. 2024-09-17T03:54:11.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:54:11.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 -- 172.21.15.129:0/3312828145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 msgr2=0x7f4d6015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3312828145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d6015c780 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f4d600b4b40 tx=0x7f4d5002a980 comp rx=0 tx=0).stop 2024-09-17T03:54:11.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 -- 172.21.15.129:0/3312828145 shutdown_connections 2024-09-17T03:54:11.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3312828145 >> v2:172.21.15.181:3300/0 conn(0x7f4d60070440 0x7f4d60074b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3312828145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 0x7f4d60074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3312828145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d6015c780 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 -- 172.21.15.129:0/3312828145 >> 172.21.15.129:0/3312828145 conn(0x7f4d600b1cb0 msgr2=0x7f4d600b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:11.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 -- 172.21.15.129:0/3312828145 shutdown_connections 2024-09-17T03:54:11.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.240+0000 7f4d66e00640 1 -- 172.21.15.129:0/3312828145 wait complete. 2024-09-17T03:54:11.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.243+0000 7f4d66e00640 1 Processor -- start 2024-09-17T03:54:11.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.243+0000 7f4d66e00640 1 -- start start 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 0x7f4d601f9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d60070440 0x7f4d6009dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6006f690 con 0x7f4d60073170 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6006f510 con 0x7f4d60070440 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d66e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6006f810 con 0x7f4d60070030 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d64e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d64e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57832/0 (socket says 172.21.15.129:57832) 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.244+0000 7f4d64e00640 1 -- 172.21.15.129:0/3366912289 learned_addr learned my addr 172.21.15.129:0/3366912289 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d5f400640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d60070440 0x7f4d6009dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 0x7f4d601f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 -- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 msgr2=0x7f4d601f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 0x7f4d601f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 -- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d60070440 msgr2=0x7f4d6009dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d60070440 0x7f4d6009dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 -- 172.21.15.129:0/3366912289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6009e140 con 0x7f4d60073170 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d64e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800e6d0 tx=0x7f4d5800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5800cab0 con 0x7f4d60073170 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d5800cc50 con 0x7f4d60073170 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58005280 con 0x7f4d60073170 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.245+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d601f9fb0 con 0x7f4d60073170 2024-09-17T03:54:11.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.246+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d601f9cd0 con 0x7f4d60073170 2024-09-17T03:54:11.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.247+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d6015c780 con 0x7f4d60073170 2024-09-17T03:54:11.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.248+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4d58005420 con 0x7f4d60073170 2024-09-17T03:54:11.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.249+0000 7f4d57e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 0x7f4d4007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.249+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f4d5809b6f0 con 0x7f4d60073170 2024-09-17T03:54:11.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.249+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 0x7f4d4007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.250+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 0x7f4d4007b0d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4d50035000 tx=0x7f4d50004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:11.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.253+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4d5800ce20 con 0x7f4d60073170 2024-09-17T03:54:11.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.444+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d600c3860 con 0x7f4d60073170 2024-09-17T03:54:11.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.445+0000 7f4d57e00640 1 -- 172.21.15.129:0/3366912289 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+16716 (secure 0 0 0) 0x7f4d58063a90 con 0x7f4d60073170 2024-09-17T03:54:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:11.448 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":92,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:08.489212+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:54:04.881426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-17T03:54:05.942429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:54:11.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 msgr2=0x7f4d4007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 0x7f4d4007b0d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4d50035000 tx=0x7f4d50004ab0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 msgr2=0x7f4d600d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4d5800e6d0 tx=0x7f4d5800eb90 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 shutdown_connections 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d40078bc0 0x7f4d4007b0d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d60073170 0x7f4d600d0330 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d60070440 0x7f4d6009dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 --2- 172.21.15.129:0/3366912289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d60070030 0x7f4d601f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 >> 172.21.15.129:0/3366912289 conn(0x7f4d600b1cb0 msgr2=0x7f4d6015f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 shutdown_connections 2024-09-17T03:54:11.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.448+0000 7f4d66e00640 1 -- 172.21.15.129:0/3366912289 wait complete. 2024-09-17T03:54:11.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:54:11.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 -- 172.21.15.129:0/1911742803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0cc1e0 msgr2=0x7f65ec0af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 --2- 172.21.15.129:0/1911742803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0cc1e0 0x7f65ec0af3c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f65e0005fd0 tx=0x7f65e002a530 comp rx=0 tx=0).stop 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 -- 172.21.15.129:0/1911742803 shutdown_connections 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 --2- 172.21.15.129:0/1911742803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0ccb50 0x7f65ec0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 --2- 172.21.15.129:0/1911742803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0cc1e0 0x7f65ec0af3c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 -- 172.21.15.129:0/1911742803 >> 172.21.15.129:0/1911742803 conn(0x7f65ec06d7d0 msgr2=0x7f65ec0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:11.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 -- 172.21.15.129:0/1911742803 shutdown_connections 2024-09-17T03:54:11.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.691+0000 7f65f3800640 1 -- 172.21.15.129:0/1911742803 wait complete. 2024-09-17T03:54:11.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.693+0000 7f65f3800640 1 Processor -- start 2024-09-17T03:54:11.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.693+0000 7f65f3800640 1 -- start start 2024-09-17T03:54:11.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.694+0000 7f65f3800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 0x7f65ec1f59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f3800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f3800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65ec15d8a0 0x7f65ec09c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f3800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec1533a0 con 0x7f65ec0ccb50 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f3800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f65ec153520 con 0x7f65ec15d8a0 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f3800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f65ec1536a0 con 0x7f65ec0cc1e0 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 0x7f65ec1f59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 0x7f65ec1f59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:58552/0 (socket says 172.21.15.129:58552) 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f0e00640 1 -- 172.21.15.129:0/2321900126 learned_addr learned my addr 172.21.15.129:0/2321900126 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57840/0 (socket says 172.21.15.129:57840) 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65f1800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65ec15d8a0 0x7f65ec09c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 -- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 msgr2=0x7f65ec1f59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 0x7f65ec1f59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 -- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65ec15d8a0 msgr2=0x7f65ec09c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65ec15d8a0 0x7f65ec09c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.695+0000 7f65ebe00640 1 -- 172.21.15.129:0/2321900126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ec09b8b0 con 0x7f65ec0ccb50 2024-09-17T03:54:11.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65ebe00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f65d800b410 tx=0x7f65d800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8015030 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65d8004020 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ec09bb90 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65d8011630 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.696+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ec09c150 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.697+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ec1f6c80 con 0x7f65ec0ccb50 2024-09-17T03:54:11.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.698+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f65d8018070 con 0x7f65ec0ccb50 2024-09-17T03:54:11.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.699+0000 7f65e8c00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 0x7f65cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:11.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.699+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f65d809a710 con 0x7f65ec0ccb50 2024-09-17T03:54:11.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.700+0000 7f65f0e00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 0x7f65cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:11.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.701+0000 7f65f0e00640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 0x7f65cc07b0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009230 tx=0x7f65e0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.704+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f65d8062470 con 0x7f65ec0ccb50 2024-09-17T03:54:11.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.884+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65ec1f6f10 con 0x7f65cc078bc0 2024-09-17T03:54:11.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.890+0000 7f65e8c00640 1 -- 172.21.15.129:0/2321900126 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252070 (secure 0 0 0) 0x7f65ec1f6f10 con 0x7f65cc078bc0 2024-09-17T03:54:11.893 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:11.894 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 msgr2=0x7f65cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 0x7f65cc07b0d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009230 tx=0x7f65e0004730 comp rx=0 tx=0).stop 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 msgr2=0x7f65ec072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f65d800b410 tx=0x7f65d800b8d0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 shutdown_connections 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f65cc078bc0 0x7f65cc07b0d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65ec15d8a0 0x7f65ec09c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65ec0ccb50 0x7f65ec072e30 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 --2- 172.21.15.129:0/2321900126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65ec0cc1e0 0x7f65ec1f59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:11.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 >> 172.21.15.129:0/2321900126 conn(0x7f65ec06d7d0 msgr2=0x7f65ec15b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:11.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 shutdown_connections 2024-09-17T03:54:11.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:11.894+0000 7f65f3800640 1 -- 172.21.15.129:0/2321900126 wait complete. 2024-09-17T03:54:11.920 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-17T03:54:10.369436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263688,"kb_used_data":9992,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711032,"statfs":{"total":773094113280,"available":772824096768,"internally_reserved":0,"allocated":10231808,"data_stored":4986578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.894540"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524290+0000","last_change":"2024-09-17T03:54:07.514193+0000","last_active":"2024-09-17T03:54:07.524290+0000","last_peered":"2024-09-17T03:54:07.524290+0000","last_clean":"2024-09-17T03:54:07.524290+0000","last_became_active":"2024-09-17T03:54:07.513894+0000","last_became_peered":"2024-09-17T03:54:07.513894+0000","last_unstale":"2024-09-17T03:54:07.524290+0000","last_undegraded":"2024-09-17T03:54:07.524290+0000","last_fullsized":"2024-09-17T03:54:07.524290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:11.829301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523618+0000","last_change":"2024-09-17T03:54:07.520952+0000","last_active":"2024-09-17T03:54:07.523618+0000","last_peered":"2024-09-17T03:54:07.523618+0000","last_clean":"2024-09-17T03:54:07.523618+0000","last_became_active":"2024-09-17T03:54:07.520678+0000","last_became_peered":"2024-09-17T03:54:07.520678+0000","last_unstale":"2024-09-17T03:54:07.523618+0000","last_undegraded":"2024-09-17T03:54:07.523618+0000","last_fullsized":"2024-09-17T03:54:07.523618+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:28.803341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524332+0000","last_change":"2024-09-17T03:54:07.514034+0000","last_active":"2024-09-17T03:54:07.524332+0000","last_peered":"2024-09-17T03:54:07.524332+0000","last_clean":"2024-09-17T03:54:07.524332+0000","last_became_active":"2024-09-17T03:54:07.513744+0000","last_became_peered":"2024-09-17T03:54:07.513744+0000","last_unstale":"2024-09-17T03:54:07.524332+0000","last_undegraded":"2024-09-17T03:54:07.524332+0000","last_fullsized":"2024-09-17T03:54:07.524332+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:08.025468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503865+0000","last_change":"2024-09-17T03:54:07.513408+0000","last_active":"2024-09-17T03:54:08.503865+0000","last_peered":"2024-09-17T03:54:08.503865+0000","last_clean":"2024-09-17T03:54:08.503865+0000","last_became_active":"2024-09-17T03:54:07.508028+0000","last_became_peered":"2024-09-17T03:54:07.508028+0000","last_unstale":"2024-09-17T03:54:08.503865+0000","last_undegraded":"2024-09-17T03:54:08.503865+0000","last_fullsized":"2024-09-17T03:54:08.503865+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:11.607364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512093+0000","last_change":"2024-09-17T03:54:07.507443+0000","last_active":"2024-09-17T03:54:07.512093+0000","last_peered":"2024-09-17T03:54:07.512093+0000","last_clean":"2024-09-17T03:54:07.512093+0000","last_became_active":"2024-09-17T03:54:07.504889+0000","last_became_peered":"2024-09-17T03:54:07.504889+0000","last_unstale":"2024-09-17T03:54:07.512093+0000","last_undegraded":"2024-09-17T03:54:07.512093+0000","last_fullsized":"2024-09-17T03:54:07.512093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:47.689693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515758+0000","last_change":"2024-09-17T03:54:07.511178+0000","last_active":"2024-09-17T03:54:07.515758+0000","last_peered":"2024-09-17T03:54:07.515758+0000","last_clean":"2024-09-17T03:54:07.515758+0000","last_became_active":"2024-09-17T03:54:07.508236+0000","last_became_peered":"2024-09-17T03:54:07.508236+0000","last_unstale":"2024-09-17T03:54:07.515758+0000","last_undegraded":"2024-09-17T03:54:07.515758+0000","last_fullsized":"2024-09-17T03:54:07.515758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:54.495394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.508372+0000","last_change":"2024-09-17T03:54:07.499852+0000","last_active":"2024-09-17T03:54:07.508372+0000","last_peered":"2024-09-17T03:54:07.508372+0000","last_clean":"2024-09-17T03:54:07.508372+0000","last_became_active":"2024-09-17T03:54:07.499500+0000","last_became_peered":"2024-09-17T03:54:07.499500+0000","last_unstale":"2024-09-17T03:54:07.508372+0000","last_undegraded":"2024-09-17T03:54:07.508372+0000","last_fullsized":"2024-09-17T03:54:07.508372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:02.238278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523423+0000","last_change":"2024-09-17T03:54:07.517037+0000","last_active":"2024-09-17T03:54:07.523423+0000","last_peered":"2024-09-17T03:54:07.523423+0000","last_clean":"2024-09-17T03:54:07.523423+0000","last_became_active":"2024-09-17T03:54:07.516682+0000","last_became_peered":"2024-09-17T03:54:07.516682+0000","last_unstale":"2024-09-17T03:54:07.523423+0000","last_undegraded":"2024-09-17T03:54:07.523423+0000","last_fullsized":"2024-09-17T03:54:07.523423+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:38:11.953720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513328+0000","last_change":"2024-09-17T03:54:07.507628+0000","last_active":"2024-09-17T03:54:07.513328+0000","last_peered":"2024-09-17T03:54:07.513328+0000","last_clean":"2024-09-17T03:54:07.513328+0000","last_became_active":"2024-09-17T03:54:07.507270+0000","last_became_peered":"2024-09-17T03:54:07.507270+0000","last_unstale":"2024-09-17T03:54:07.513328+0000","last_undegraded":"2024-09-17T03:54:07.513328+0000","last_fullsized":"2024-09-17T03:54:07.513328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:46.163515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524592+0000","last_change":"2024-09-17T03:54:07.522341+0000","last_active":"2024-09-17T03:54:07.524592+0000","last_peered":"2024-09-17T03:54:07.524592+0000","last_clean":"2024-09-17T03:54:07.524592+0000","last_became_active":"2024-09-17T03:54:07.521913+0000","last_became_peered":"2024-09-17T03:54:07.521913+0000","last_unstale":"2024-09-17T03:54:07.524592+0000","last_undegraded":"2024-09-17T03:54:07.524592+0000","last_fullsized":"2024-09-17T03:54:07.524592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:09.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514069+0000","last_change":"2024-09-17T03:54:07.507184+0000","last_active":"2024-09-17T03:54:07.514069+0000","last_peered":"2024-09-17T03:54:07.514069+0000","last_clean":"2024-09-17T03:54:07.514069+0000","last_became_active":"2024-09-17T03:54:07.503029+0000","last_became_peered":"2024-09-17T03:54:07.503029+0000","last_unstale":"2024-09-17T03:54:07.514069+0000","last_undegraded":"2024-09-17T03:54:07.514069+0000","last_fullsized":"2024-09-17T03:54:07.514069+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:26.434816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512662+0000","last_change":"2024-09-17T03:54:07.502612+0000","last_active":"2024-09-17T03:54:07.512662+0000","last_peered":"2024-09-17T03:54:07.512662+0000","last_clean":"2024-09-17T03:54:07.512662+0000","last_became_active":"2024-09-17T03:54:07.502316+0000","last_became_peered":"2024-09-17T03:54:07.502316+0000","last_unstale":"2024-09-17T03:54:07.512662+0000","last_undegraded":"2024-09-17T03:54:07.512662+0000","last_fullsized":"2024-09-17T03:54:07.512662+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:52.268162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503311+0000","last_change":"2024-09-17T03:54:07.501178+0000","last_active":"2024-09-17T03:54:08.503311+0000","last_peered":"2024-09-17T03:54:08.503311+0000","last_clean":"2024-09-17T03:54:08.503311+0000","last_became_active":"2024-09-17T03:54:07.500894+0000","last_became_peered":"2024-09-17T03:54:07.500894+0000","last_unstale":"2024-09-17T03:54:08.503311+0000","last_undegraded":"2024-09-17T03:54:08.503311+0000","last_fullsized":"2024-09-17T03:54:08.503311+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:37.646246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502164+0000","last_change":"2024-09-17T03:54:07.498806+0000","last_active":"2024-09-17T03:54:08.502164+0000","last_peered":"2024-09-17T03:54:08.502164+0000","last_clean":"2024-09-17T03:54:08.502164+0000","last_became_active":"2024-09-17T03:54:07.498490+0000","last_became_peered":"2024-09-17T03:54:07.498490+0000","last_unstale":"2024-09-17T03:54:08.502164+0000","last_undegraded":"2024-09-17T03:54:08.502164+0000","last_fullsized":"2024-09-17T03:54:08.502164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:30.836203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502852+0000","last_change":"2024-09-17T03:54:07.509867+0000","last_active":"2024-09-17T03:54:08.502852+0000","last_peered":"2024-09-17T03:54:08.502852+0000","last_clean":"2024-09-17T03:54:08.502852+0000","last_became_active":"2024-09-17T03:54:07.509574+0000","last_became_peered":"2024-09-17T03:54:07.509574+0000","last_unstale":"2024-09-17T03:54:08.502852+0000","last_undegraded":"2024-09-17T03:54:08.502852+0000","last_fullsized":"2024-09-17T03:54:08.502852+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:41.937363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515891+0000","last_change":"2024-09-17T03:54:07.513453+0000","last_active":"2024-09-17T03:54:07.515891+0000","last_peered":"2024-09-17T03:54:07.515891+0000","last_clean":"2024-09-17T03:54:07.515891+0000","last_became_active":"2024-09-17T03:54:07.512032+0000","last_became_peered":"2024-09-17T03:54:07.512032+0000","last_unstale":"2024-09-17T03:54:07.515891+0000","last_undegraded":"2024-09-17T03:54:07.515891+0000","last_fullsized":"2024-09-17T03:54:07.515891+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:36.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523668+0000","last_change":"2024-09-17T03:54:07.518054+0000","last_active":"2024-09-17T03:54:07.523668+0000","last_peered":"2024-09-17T03:54:07.523668+0000","last_clean":"2024-09-17T03:54:07.523668+0000","last_became_active":"2024-09-17T03:54:07.517620+0000","last_became_peered":"2024-09-17T03:54:07.517620+0000","last_unstale":"2024-09-17T03:54:07.523668+0000","last_undegraded":"2024-09-17T03:54:07.523668+0000","last_fullsized":"2024-09-17T03:54:07.523668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:44.811654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503970+0000","last_change":"2024-09-17T03:54:07.506526+0000","last_active":"2024-09-17T03:54:08.503970+0000","last_peered":"2024-09-17T03:54:08.503970+0000","last_clean":"2024-09-17T03:54:08.503970+0000","last_became_active":"2024-09-17T03:54:07.506199+0000","last_became_peered":"2024-09-17T03:54:07.506199+0000","last_unstale":"2024-09-17T03:54:08.503970+0000","last_undegraded":"2024-09-17T03:54:08.503970+0000","last_fullsized":"2024-09-17T03:54:08.503970+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:28.642053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502080+0000","last_change":"2024-09-17T03:54:07.509480+0000","last_active":"2024-09-17T03:54:08.502080+0000","last_peered":"2024-09-17T03:54:08.502080+0000","last_clean":"2024-09-17T03:54:08.502080+0000","last_became_active":"2024-09-17T03:54:07.508967+0000","last_became_peered":"2024-09-17T03:54:07.508967+0000","last_unstale":"2024-09-17T03:54:08.502080+0000","last_undegraded":"2024-09-17T03:54:08.502080+0000","last_fullsized":"2024-09-17T03:54:08.502080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:19.393480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513474+0000","last_change":"2024-09-17T03:54:07.505671+0000","last_active":"2024-09-17T03:54:07.513474+0000","last_peered":"2024-09-17T03:54:07.513474+0000","last_clean":"2024-09-17T03:54:07.513474+0000","last_became_active":"2024-09-17T03:54:07.505347+0000","last_became_peered":"2024-09-17T03:54:07.505347+0000","last_unstale":"2024-09-17T03:54:07.513474+0000","last_undegraded":"2024-09-17T03:54:07.513474+0000","last_fullsized":"2024-09-17T03:54:07.513474+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:10.052345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502216+0000","last_change":"2024-09-17T03:54:07.501099+0000","last_active":"2024-09-17T03:54:08.502216+0000","last_peered":"2024-09-17T03:54:08.502216+0000","last_clean":"2024-09-17T03:54:08.502216+0000","last_became_active":"2024-09-17T03:54:07.500794+0000","last_became_peered":"2024-09-17T03:54:07.500794+0000","last_unstale":"2024-09-17T03:54:08.502216+0000","last_undegraded":"2024-09-17T03:54:08.502216+0000","last_fullsized":"2024-09-17T03:54:08.502216+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:40.214091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524068+0000","last_change":"2024-09-17T03:54:07.519854+0000","last_active":"2024-09-17T03:54:07.524068+0000","last_peered":"2024-09-17T03:54:07.524068+0000","last_clean":"2024-09-17T03:54:07.524068+0000","last_became_active":"2024-09-17T03:54:07.519578+0000","last_became_peered":"2024-09-17T03:54:07.519578+0000","last_unstale":"2024-09-17T03:54:07.524068+0000","last_undegraded":"2024-09-17T03:54:07.524068+0000","last_fullsized":"2024-09-17T03:54:07.524068+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:53.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523275+0000","last_change":"2024-09-17T03:54:07.516524+0000","last_active":"2024-09-17T03:54:07.523275+0000","last_peered":"2024-09-17T03:54:07.523275+0000","last_clean":"2024-09-17T03:54:07.523275+0000","last_became_active":"2024-09-17T03:54:07.516214+0000","last_became_peered":"2024-09-17T03:54:07.516214+0000","last_unstale":"2024-09-17T03:54:07.523275+0000","last_undegraded":"2024-09-17T03:54:07.523275+0000","last_fullsized":"2024-09-17T03:54:07.523275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:41.990100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513687+0000","last_change":"2024-09-17T03:54:07.502148+0000","last_active":"2024-09-17T03:54:07.513687+0000","last_peered":"2024-09-17T03:54:07.513687+0000","last_clean":"2024-09-17T03:54:07.513687+0000","last_became_active":"2024-09-17T03:54:07.501830+0000","last_became_peered":"2024-09-17T03:54:07.501830+0000","last_unstale":"2024-09-17T03:54:07.513687+0000","last_undegraded":"2024-09-17T03:54:07.513687+0000","last_fullsized":"2024-09-17T03:54:07.513687+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:24.253089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.517837+0000","last_change":"2024-09-17T03:54:07.510557+0000","last_active":"2024-09-17T03:54:07.517837+0000","last_peered":"2024-09-17T03:54:07.517837+0000","last_clean":"2024-09-17T03:54:07.517837+0000","last_became_active":"2024-09-17T03:54:07.510166+0000","last_became_peered":"2024-09-17T03:54:07.510166+0000","last_unstale":"2024-09-17T03:54:07.517837+0000","last_undegraded":"2024-09-17T03:54:07.517837+0000","last_fullsized":"2024-09-17T03:54:07.517837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:44.143436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521181+0000","last_change":"2024-09-17T03:54:07.512465+0000","last_active":"2024-09-17T03:54:07.521181+0000","last_peered":"2024-09-17T03:54:07.521181+0000","last_clean":"2024-09-17T03:54:07.521181+0000","last_became_active":"2024-09-17T03:54:07.509547+0000","last_became_peered":"2024-09-17T03:54:07.509547+0000","last_unstale":"2024-09-17T03:54:07.521181+0000","last_undegraded":"2024-09-17T03:54:07.521181+0000","last_fullsized":"2024-09-17T03:54:07.521181+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:49.842716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514169+0000","last_change":"2024-09-17T03:54:07.508577+0000","last_active":"2024-09-17T03:54:07.514169+0000","last_peered":"2024-09-17T03:54:07.514169+0000","last_clean":"2024-09-17T03:54:07.514169+0000","last_became_active":"2024-09-17T03:54:07.507537+0000","last_became_peered":"2024-09-17T03:54:07.507537+0000","last_unstale":"2024-09-17T03:54:07.514169+0000","last_undegraded":"2024-09-17T03:54:07.514169+0000","last_fullsized":"2024-09-17T03:54:07.514169+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:11.333690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512864+0000","last_change":"2024-09-17T03:54:07.504594+0000","last_active":"2024-09-17T03:54:07.512864+0000","last_peered":"2024-09-17T03:54:07.512864+0000","last_clean":"2024-09-17T03:54:07.512864+0000","last_became_active":"2024-09-17T03:54:07.503195+0000","last_became_peered":"2024-09-17T03:54:07.503195+0000","last_unstale":"2024-09-17T03:54:07.512864+0000","last_undegraded":"2024-09-17T03:54:07.512864+0000","last_fullsized":"2024-09-17T03:54:07.512864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:04.407555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.525955+0000","last_change":"2024-09-17T03:54:07.525228+0000","last_active":"2024-09-17T03:54:07.525955+0000","last_peered":"2024-09-17T03:54:07.525955+0000","last_clean":"2024-09-17T03:54:07.525955+0000","last_became_active":"2024-09-17T03:54:07.524909+0000","last_became_peered":"2024-09-17T03:54:07.524909+0000","last_unstale":"2024-09-17T03:54:07.525955+0000","last_undegraded":"2024-09-17T03:54:07.525955+0000","last_fullsized":"2024-09-17T03:54:07.525955+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:44.791790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501966+0000","last_change":"2024-09-17T03:54:07.500582+0000","last_active":"2024-09-17T03:54:08.501966+0000","last_peered":"2024-09-17T03:54:08.501966+0000","last_clean":"2024-09-17T03:54:08.501966+0000","last_became_active":"2024-09-17T03:54:07.499821+0000","last_became_peered":"2024-09-17T03:54:07.499821+0000","last_unstale":"2024-09-17T03:54:08.501966+0000","last_undegraded":"2024-09-17T03:54:08.501966+0000","last_fullsized":"2024-09-17T03:54:08.501966+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:58.669908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503197+0000","last_change":"2024-09-17T03:54:07.499625+0000","last_active":"2024-09-17T03:54:08.503197+0000","last_peered":"2024-09-17T03:54:08.503197+0000","last_clean":"2024-09-17T03:54:08.503197+0000","last_became_active":"2024-09-17T03:54:07.499273+0000","last_became_peered":"2024-09-17T03:54:07.499273+0000","last_unstale":"2024-09-17T03:54:08.503197+0000","last_undegraded":"2024-09-17T03:54:08.503197+0000","last_fullsized":"2024-09-17T03:54:08.503197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:42.604448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526407+0000","last_change":"2024-09-17T03:54:07.519342+0000","last_active":"2024-09-17T03:54:07.526407+0000","last_peered":"2024-09-17T03:54:07.526407+0000","last_clean":"2024-09-17T03:54:07.526407+0000","last_became_active":"2024-09-17T03:54:07.519045+0000","last_became_peered":"2024-09-17T03:54:07.519045+0000","last_unstale":"2024-09-17T03:54:07.526407+0000","last_undegraded":"2024-09-17T03:54:07.526407+0000","last_fullsized":"2024-09-17T03:54:07.526407+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:59.727134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503191+0000","last_change":"2024-09-17T03:54:07.497783+0000","last_active":"2024-09-17T03:54:08.503191+0000","last_peered":"2024-09-17T03:54:08.503191+0000","last_clean":"2024-09-17T03:54:08.503191+0000","last_became_active":"2024-09-17T03:54:07.497442+0000","last_became_peered":"2024-09-17T03:54:07.497442+0000","last_unstale":"2024-09-17T03:54:08.503191+0000","last_undegraded":"2024-09-17T03:54:08.503191+0000","last_fullsized":"2024-09-17T03:54:08.503191+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:26.754285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.527202+0000","last_change":"2024-09-17T03:54:06.504697+0000","last_active":"2024-09-17T03:54:07.527202+0000","last_peered":"2024-09-17T03:54:07.527202+0000","last_clean":"2024-09-17T03:54:07.527202+0000","last_became_active":"2024-09-17T03:54:06.504343+0000","last_became_peered":"2024-09-17T03:54:06.504343+0000","last_unstale":"2024-09-17T03:54:07.527202+0000","last_undegraded":"2024-09-17T03:54:07.527202+0000","last_fullsized":"2024-09-17T03:54:07.527202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:08.795420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.504085+0000","last_change":"2024-09-17T03:54:07.503853+0000","last_active":"2024-09-17T03:54:08.504085+0000","last_peered":"2024-09-17T03:54:08.504085+0000","last_clean":"2024-09-17T03:54:08.504085+0000","last_became_active":"2024-09-17T03:54:07.502599+0000","last_became_peered":"2024-09-17T03:54:07.502599+0000","last_unstale":"2024-09-17T03:54:08.504085+0000","last_undegraded":"2024-09-17T03:54:08.504085+0000","last_fullsized":"2024-09-17T03:54:08.504085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:37.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524893+0000","last_change":"2024-09-17T03:54:06.504622+0000","last_active":"2024-09-17T03:54:07.524893+0000","last_peered":"2024-09-17T03:54:07.524893+0000","last_clean":"2024-09-17T03:54:07.524893+0000","last_became_active":"2024-09-17T03:54:06.504295+0000","last_became_peered":"2024-09-17T03:54:06.504295+0000","last_unstale":"2024-09-17T03:54:07.524893+0000","last_undegraded":"2024-09-17T03:54:07.524893+0000","last_fullsized":"2024-09-17T03:54:07.524893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:12.976269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524136+0000","last_change":"2024-09-17T03:54:07.513146+0000","last_active":"2024-09-17T03:54:07.524136+0000","last_peered":"2024-09-17T03:54:07.524136+0000","last_clean":"2024-09-17T03:54:07.524136+0000","last_became_active":"2024-09-17T03:54:07.512081+0000","last_became_peered":"2024-09-17T03:54:07.512081+0000","last_unstale":"2024-09-17T03:54:07.524136+0000","last_undegraded":"2024-09-17T03:54:07.524136+0000","last_fullsized":"2024-09-17T03:54:07.524136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:56.818324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521720+0000","last_change":"2024-09-17T03:54:06.495577+0000","last_active":"2024-09-17T03:54:07.521720+0000","last_peered":"2024-09-17T03:54:07.521720+0000","last_clean":"2024-09-17T03:54:07.521720+0000","last_became_active":"2024-09-17T03:54:06.495026+0000","last_became_peered":"2024-09-17T03:54:06.495026+0000","last_unstale":"2024-09-17T03:54:07.521720+0000","last_undegraded":"2024-09-17T03:54:07.521720+0000","last_fullsized":"2024-09-17T03:54:07.521720+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:20.756533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521344+0000","last_change":"2024-09-17T03:54:07.516993+0000","last_active":"2024-09-17T03:54:07.521344+0000","last_peered":"2024-09-17T03:54:07.521344+0000","last_clean":"2024-09-17T03:54:07.521344+0000","last_became_active":"2024-09-17T03:54:07.516543+0000","last_became_peered":"2024-09-17T03:54:07.516543+0000","last_unstale":"2024-09-17T03:54:07.521344+0000","last_undegraded":"2024-09-17T03:54:07.521344+0000","last_fullsized":"2024-09-17T03:54:07.521344+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:26.434816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.522005+0000","last_change":"2024-09-17T03:54:06.498040+0000","last_active":"2024-09-17T03:54:07.522005+0000","last_peered":"2024-09-17T03:54:07.522005+0000","last_clean":"2024-09-17T03:54:07.522005+0000","last_became_active":"2024-09-17T03:54:06.497769+0000","last_became_peered":"2024-09-17T03:54:06.497769+0000","last_unstale":"2024-09-17T03:54:07.522005+0000","last_undegraded":"2024-09-17T03:54:07.522005+0000","last_fullsized":"2024-09-17T03:54:07.522005+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:57.329153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503806+0000","last_change":"2024-09-17T03:54:07.508884+0000","last_active":"2024-09-17T03:54:08.503806+0000","last_peered":"2024-09-17T03:54:08.503806+0000","last_clean":"2024-09-17T03:54:08.503806+0000","last_became_active":"2024-09-17T03:54:07.508533+0000","last_became_peered":"2024-09-17T03:54:07.508533+0000","last_unstale":"2024-09-17T03:54:08.503806+0000","last_undegraded":"2024-09-17T03:54:08.503806+0000","last_fullsized":"2024-09-17T03:54:08.503806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:28.733545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524946+0000","last_change":"2024-09-17T03:54:06.505075+0000","last_active":"2024-09-17T03:54:07.524946+0000","last_peered":"2024-09-17T03:54:07.524946+0000","last_clean":"2024-09-17T03:54:07.524946+0000","last_became_active":"2024-09-17T03:54:06.504781+0000","last_became_peered":"2024-09-17T03:54:06.504781+0000","last_unstale":"2024-09-17T03:54:07.524946+0000","last_undegraded":"2024-09-17T03:54:07.524946+0000","last_fullsized":"2024-09-17T03:54:07.524946+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:42.673956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514371+0000","last_change":"2024-09-17T03:54:07.509678+0000","last_active":"2024-09-17T03:54:07.514371+0000","last_peered":"2024-09-17T03:54:07.514371+0000","last_clean":"2024-09-17T03:54:07.514371+0000","last_became_active":"2024-09-17T03:54:07.507855+0000","last_became_peered":"2024-09-17T03:54:07.507855+0000","last_unstale":"2024-09-17T03:54:07.514371+0000","last_undegraded":"2024-09-17T03:54:07.514371+0000","last_fullsized":"2024-09-17T03:54:07.514371+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:34:14.900411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526527+0000","last_change":"2024-09-17T03:54:06.504445+0000","last_active":"2024-09-17T03:54:07.526527+0000","last_peered":"2024-09-17T03:54:07.526527+0000","last_clean":"2024-09-17T03:54:07.526527+0000","last_became_active":"2024-09-17T03:54:06.503430+0000","last_became_peered":"2024-09-17T03:54:06.503430+0000","last_unstale":"2024-09-17T03:54:07.526527+0000","last_undegraded":"2024-09-17T03:54:07.526527+0000","last_fullsized":"2024-09-17T03:54:07.526527+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:46.687044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502901+0000","last_change":"2024-09-17T03:54:07.502148+0000","last_active":"2024-09-17T03:54:08.502901+0000","last_peered":"2024-09-17T03:54:08.502901+0000","last_clean":"2024-09-17T03:54:08.502901+0000","last_became_active":"2024-09-17T03:54:07.499812+0000","last_became_peered":"2024-09-17T03:54:07.499812+0000","last_unstale":"2024-09-17T03:54:08.502901+0000","last_undegraded":"2024-09-17T03:54:08.502901+0000","last_fullsized":"2024-09-17T03:54:08.502901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:10.977290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.492994+0000","last_change":"2024-09-17T03:54:06.505126+0000","last_active":"2024-09-17T03:54:07.492994+0000","last_peered":"2024-09-17T03:54:07.492994+0000","last_clean":"2024-09-17T03:54:07.492994+0000","last_became_active":"2024-09-17T03:54:06.504801+0000","last_became_peered":"2024-09-17T03:54:06.504801+0000","last_unstale":"2024-09-17T03:54:07.492994+0000","last_undegraded":"2024-09-17T03:54:07.492994+0000","last_fullsized":"2024-09-17T03:54:07.492994+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:08.265326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521030+0000","last_change":"2024-09-17T03:54:07.515697+0000","last_active":"2024-09-17T03:54:07.521030+0000","last_peered":"2024-09-17T03:54:07.521030+0000","last_clean":"2024-09-17T03:54:07.521030+0000","last_became_active":"2024-09-17T03:54:07.515284+0000","last_became_peered":"2024-09-17T03:54:07.515284+0000","last_unstale":"2024-09-17T03:54:07.521030+0000","last_undegraded":"2024-09-17T03:54:07.521030+0000","last_fullsized":"2024-09-17T03:54:07.521030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:24.253089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.503576+0000","last_change":"2024-09-17T03:54:06.502158+0000","last_active":"2024-09-17T03:54:07.503576+0000","last_peered":"2024-09-17T03:54:07.503576+0000","last_clean":"2024-09-17T03:54:07.503576+0000","last_became_active":"2024-09-17T03:54:06.501853+0000","last_became_peered":"2024-09-17T03:54:06.501853+0000","last_unstale":"2024-09-17T03:54:07.503576+0000","last_undegraded":"2024-09-17T03:54:07.503576+0000","last_fullsized":"2024-09-17T03:54:07.503576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:20.558882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523354+0000","last_change":"2024-09-17T03:54:07.517010+0000","last_active":"2024-09-17T03:54:07.523354+0000","last_peered":"2024-09-17T03:54:07.523354+0000","last_clean":"2024-09-17T03:54:07.523354+0000","last_became_active":"2024-09-17T03:54:07.516714+0000","last_became_peered":"2024-09-17T03:54:07.516714+0000","last_unstale":"2024-09-17T03:54:07.523354+0000","last_undegraded":"2024-09-17T03:54:07.523354+0000","last_fullsized":"2024-09-17T03:54:07.523354+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:06.198432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.506973+0000","last_change":"2024-09-17T03:54:06.506662+0000","last_active":"2024-09-17T03:54:07.506973+0000","last_peered":"2024-09-17T03:54:07.506973+0000","last_clean":"2024-09-17T03:54:07.506973+0000","last_became_active":"2024-09-17T03:54:06.506373+0000","last_became_peered":"2024-09-17T03:54:06.506373+0000","last_unstale":"2024-09-17T03:54:07.506973+0000","last_undegraded":"2024-09-17T03:54:07.506973+0000","last_fullsized":"2024-09-17T03:54:07.506973+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:41.558155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521580+0000","last_change":"2024-09-17T03:54:07.517161+0000","last_active":"2024-09-17T03:54:07.521580+0000","last_peered":"2024-09-17T03:54:07.521580+0000","last_clean":"2024-09-17T03:54:07.521580+0000","last_became_active":"2024-09-17T03:54:07.516751+0000","last_became_peered":"2024-09-17T03:54:07.516751+0000","last_unstale":"2024-09-17T03:54:07.521580+0000","last_undegraded":"2024-09-17T03:54:07.521580+0000","last_fullsized":"2024-09-17T03:54:07.521580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:11.333690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503868+0000","last_change":"2024-09-17T03:54:06.503897+0000","last_active":"2024-09-17T03:54:08.503868+0000","last_peered":"2024-09-17T03:54:08.503868+0000","last_clean":"2024-09-17T03:54:08.503868+0000","last_became_active":"2024-09-17T03:54:06.503598+0000","last_became_peered":"2024-09-17T03:54:06.503598+0000","last_unstale":"2024-09-17T03:54:08.503868+0000","last_undegraded":"2024-09-17T03:54:08.503868+0000","last_fullsized":"2024-09-17T03:54:08.503868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:01.235629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.509662+0000","last_change":"2024-09-17T03:54:07.502593+0000","last_active":"2024-09-17T03:54:07.509662+0000","last_peered":"2024-09-17T03:54:07.509662+0000","last_clean":"2024-09-17T03:54:07.509662+0000","last_became_active":"2024-09-17T03:54:07.502243+0000","last_became_peered":"2024-09-17T03:54:07.502243+0000","last_unstale":"2024-09-17T03:54:07.509662+0000","last_undegraded":"2024-09-17T03:54:07.509662+0000","last_fullsized":"2024-09-17T03:54:07.509662+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:17.600556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.520269+0000","last_change":"2024-09-17T03:54:06.499973+0000","last_active":"2024-09-17T03:54:07.520269+0000","last_peered":"2024-09-17T03:54:07.520269+0000","last_clean":"2024-09-17T03:54:07.520269+0000","last_became_active":"2024-09-17T03:54:06.499672+0000","last_became_peered":"2024-09-17T03:54:06.499672+0000","last_unstale":"2024-09-17T03:54:07.520269+0000","last_undegraded":"2024-09-17T03:54:07.520269+0000","last_fullsized":"2024-09-17T03:54:07.520269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:09.049696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524327+0000","last_change":"2024-09-17T03:54:07.519883+0000","last_active":"2024-09-17T03:54:07.524327+0000","last_peered":"2024-09-17T03:54:07.524327+0000","last_clean":"2024-09-17T03:54:07.524327+0000","last_became_active":"2024-09-17T03:54:07.519107+0000","last_became_peered":"2024-09-17T03:54:07.519107+0000","last_unstale":"2024-09-17T03:54:07.524327+0000","last_undegraded":"2024-09-17T03:54:07.524327+0000","last_fullsized":"2024-09-17T03:54:07.524327+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:24.838940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.527612+0000","last_change":"2024-09-17T03:54:06.497042+0000","last_active":"2024-09-17T03:54:07.527612+0000","last_peered":"2024-09-17T03:54:07.527612+0000","last_clean":"2024-09-17T03:54:07.527612+0000","last_became_active":"2024-09-17T03:54:06.495801+0000","last_became_peered":"2024-09-17T03:54:06.495801+0000","last_unstale":"2024-09-17T03:54:07.527612+0000","last_undegraded":"2024-09-17T03:54:07.527612+0000","last_fullsized":"2024-09-17T03:54:07.527612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:30.769908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503430+0000","last_change":"2024-09-17T03:54:07.491211+0000","last_active":"2024-09-17T03:54:08.503430+0000","last_peered":"2024-09-17T03:54:08.503430+0000","last_clean":"2024-09-17T03:54:08.503430+0000","last_became_active":"2024-09-17T03:54:07.490916+0000","last_became_peered":"2024-09-17T03:54:07.490916+0000","last_unstale":"2024-09-17T03:54:08.503430+0000","last_undegraded":"2024-09-17T03:54:08.503430+0000","last_fullsized":"2024-09-17T03:54:08.503430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:56.420057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502532+0000","last_change":"2024-09-17T03:54:06.498677+0000","last_active":"2024-09-17T03:54:08.502532+0000","last_peered":"2024-09-17T03:54:08.502532+0000","last_clean":"2024-09-17T03:54:08.502532+0000","last_became_active":"2024-09-17T03:54:06.498351+0000","last_became_peered":"2024-09-17T03:54:06.498351+0000","last_unstale":"2024-09-17T03:54:08.502532+0000","last_undegraded":"2024-09-17T03:54:08.502532+0000","last_fullsized":"2024-09-17T03:54:08.502532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:00.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526177+0000","last_change":"2024-09-17T03:54:07.513062+0000","last_active":"2024-09-17T03:54:07.526177+0000","last_peered":"2024-09-17T03:54:07.526177+0000","last_clean":"2024-09-17T03:54:07.526177+0000","last_became_active":"2024-09-17T03:54:07.512777+0000","last_became_peered":"2024-09-17T03:54:07.512777+0000","last_unstale":"2024-09-17T03:54:07.526177+0000","last_undegraded":"2024-09-17T03:54:07.526177+0000","last_fullsized":"2024-09-17T03:54:07.526177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:44:50.008384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502333+0000","last_change":"2024-09-17T03:54:06.508013+0000","last_active":"2024-09-17T03:54:08.502333+0000","last_peered":"2024-09-17T03:54:08.502333+0000","last_clean":"2024-09-17T03:54:08.502333+0000","last_became_active":"2024-09-17T03:54:06.507685+0000","last_became_peered":"2024-09-17T03:54:06.507685+0000","last_unstale":"2024-09-17T03:54:08.502333+0000","last_undegraded":"2024-09-17T03:54:08.502333+0000","last_fullsized":"2024-09-17T03:54:08.502333+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:58.035209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513376+0000","last_change":"2024-09-17T03:54:07.505967+0000","last_active":"2024-09-17T03:54:07.513376+0000","last_peered":"2024-09-17T03:54:07.513376+0000","last_clean":"2024-09-17T03:54:07.513376+0000","last_became_active":"2024-09-17T03:54:07.505594+0000","last_became_peered":"2024-09-17T03:54:07.505594+0000","last_unstale":"2024-09-17T03:54:07.513376+0000","last_undegraded":"2024-09-17T03:54:07.513376+0000","last_fullsized":"2024-09-17T03:54:07.513376+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:02.238278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.494886+0000","last_change":"2024-09-17T03:54:06.504531+0000","last_active":"2024-09-17T03:54:07.494886+0000","last_peered":"2024-09-17T03:54:07.494886+0000","last_clean":"2024-09-17T03:54:07.494886+0000","last_became_active":"2024-09-17T03:54:06.504133+0000","last_became_peered":"2024-09-17T03:54:06.504133+0000","last_unstale":"2024-09-17T03:54:07.494886+0000","last_undegraded":"2024-09-17T03:54:07.494886+0000","last_fullsized":"2024-09-17T03:54:07.494886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:25.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501666+0000","last_change":"2024-09-17T03:54:07.499866+0000","last_active":"2024-09-17T03:54:08.501666+0000","last_peered":"2024-09-17T03:54:08.501666+0000","last_clean":"2024-09-17T03:54:08.501666+0000","last_became_active":"2024-09-17T03:54:07.499536+0000","last_became_peered":"2024-09-17T03:54:07.499536+0000","last_unstale":"2024-09-17T03:54:08.501666+0000","last_undegraded":"2024-09-17T03:54:08.501666+0000","last_fullsized":"2024-09-17T03:54:08.501666+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:38.178335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.529990+0000","last_change":"2024-09-17T03:54:06.500148+0000","last_active":"2024-09-17T03:54:07.529990+0000","last_peered":"2024-09-17T03:54:07.529990+0000","last_clean":"2024-09-17T03:54:07.529990+0000","last_became_active":"2024-09-17T03:54:06.499831+0000","last_became_peered":"2024-09-17T03:54:06.499831+0000","last_unstale":"2024-09-17T03:54:07.529990+0000","last_undegraded":"2024-09-17T03:54:07.529990+0000","last_fullsized":"2024-09-17T03:54:07.529990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:11.794891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512730+0000","last_change":"2024-09-17T03:54:07.506127+0000","last_active":"2024-09-17T03:54:07.512730+0000","last_peered":"2024-09-17T03:54:07.512730+0000","last_clean":"2024-09-17T03:54:07.512730+0000","last_became_active":"2024-09-17T03:54:07.505432+0000","last_became_peered":"2024-09-17T03:54:07.505432+0000","last_unstale":"2024-09-17T03:54:07.512730+0000","last_undegraded":"2024-09-17T03:54:07.512730+0000","last_fullsized":"2024-09-17T03:54:07.512730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:34.280281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.518852+0000","last_change":"2024-09-17T03:54:06.503203+0000","last_active":"2024-09-17T03:54:07.518852+0000","last_peered":"2024-09-17T03:54:07.518852+0000","last_clean":"2024-09-17T03:54:07.518852+0000","last_became_active":"2024-09-17T03:54:06.502793+0000","last_became_peered":"2024-09-17T03:54:06.502793+0000","last_unstale":"2024-09-17T03:54:07.518852+0000","last_undegraded":"2024-09-17T03:54:07.518852+0000","last_fullsized":"2024-09-17T03:54:07.518852+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:29:01.591052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503489+0000","last_change":"2024-09-17T03:54:07.505611+0000","last_active":"2024-09-17T03:54:08.503489+0000","last_peered":"2024-09-17T03:54:08.503489+0000","last_clean":"2024-09-17T03:54:08.503489+0000","last_became_active":"2024-09-17T03:54:07.505272+0000","last_became_peered":"2024-09-17T03:54:07.505272+0000","last_unstale":"2024-09-17T03:54:08.503489+0000","last_undegraded":"2024-09-17T03:54:08.503489+0000","last_fullsized":"2024-09-17T03:54:08.503489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:44.053641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.518735+0000","last_change":"2024-09-17T03:54:06.502802+0000","last_active":"2024-09-17T03:54:07.518735+0000","last_peered":"2024-09-17T03:54:07.518735+0000","last_clean":"2024-09-17T03:54:07.518735+0000","last_became_active":"2024-09-17T03:54:06.502284+0000","last_became_peered":"2024-09-17T03:54:06.502284+0000","last_unstale":"2024-09-17T03:54:07.518735+0000","last_undegraded":"2024-09-17T03:54:07.518735+0000","last_fullsized":"2024-09-17T03:54:07.518735+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:00.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512907+0000","last_change":"2024-09-17T03:54:07.501125+0000","last_active":"2024-09-17T03:54:07.512907+0000","last_peered":"2024-09-17T03:54:07.512907+0000","last_clean":"2024-09-17T03:54:07.512907+0000","last_became_active":"2024-09-17T03:54:07.500627+0000","last_became_peered":"2024-09-17T03:54:07.500627+0000","last_unstale":"2024-09-17T03:54:07.512907+0000","last_undegraded":"2024-09-17T03:54:07.512907+0000","last_fullsized":"2024-09-17T03:54:07.512907+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:46.163515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.511329+0000","last_change":"2024-09-17T03:54:06.502310+0000","last_active":"2024-09-17T03:54:07.511329+0000","last_peered":"2024-09-17T03:54:07.511329+0000","last_clean":"2024-09-17T03:54:07.511329+0000","last_became_active":"2024-09-17T03:54:06.501977+0000","last_became_peered":"2024-09-17T03:54:06.501977+0000","last_unstale":"2024-09-17T03:54:07.511329+0000","last_undegraded":"2024-09-17T03:54:07.511329+0000","last_fullsized":"2024-09-17T03:54:07.511329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:04.752019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502210+0000","last_change":"2024-09-17T03:54:07.491289+0000","last_active":"2024-09-17T03:54:08.502210+0000","last_peered":"2024-09-17T03:54:08.502210+0000","last_clean":"2024-09-17T03:54:08.502210+0000","last_became_active":"2024-09-17T03:54:07.490955+0000","last_became_peered":"2024-09-17T03:54:07.490955+0000","last_unstale":"2024-09-17T03:54:08.502210+0000","last_undegraded":"2024-09-17T03:54:08.502210+0000","last_fullsized":"2024-09-17T03:54:08.502210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:24.805323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521658+0000","last_change":"2024-09-17T03:54:06.493771+0000","last_active":"2024-09-17T03:54:07.521658+0000","last_peered":"2024-09-17T03:54:07.521658+0000","last_clean":"2024-09-17T03:54:07.521658+0000","last_became_active":"2024-09-17T03:54:06.493049+0000","last_became_peered":"2024-09-17T03:54:06.493049+0000","last_unstale":"2024-09-17T03:54:07.521658+0000","last_undegraded":"2024-09-17T03:54:07.521658+0000","last_fullsized":"2024-09-17T03:54:07.521658+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:55.728832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502462+0000","last_change":"2024-09-17T03:54:07.511066+0000","last_active":"2024-09-17T03:54:08.502462+0000","last_peered":"2024-09-17T03:54:08.502462+0000","last_clean":"2024-09-17T03:54:08.502462+0000","last_became_active":"2024-09-17T03:54:07.509580+0000","last_became_peered":"2024-09-17T03:54:07.509580+0000","last_unstale":"2024-09-17T03:54:08.502462+0000","last_undegraded":"2024-09-17T03:54:08.502462+0000","last_fullsized":"2024-09-17T03:54:08.502462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:55.069646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503631+0000","last_change":"2024-09-17T03:54:06.507942+0000","last_active":"2024-09-17T03:54:08.503631+0000","last_peered":"2024-09-17T03:54:08.503631+0000","last_clean":"2024-09-17T03:54:08.503631+0000","last_became_active":"2024-09-17T03:54:06.507546+0000","last_became_peered":"2024-09-17T03:54:06.507546+0000","last_unstale":"2024-09-17T03:54:08.503631+0000","last_undegraded":"2024-09-17T03:54:08.503631+0000","last_fullsized":"2024-09-17T03:54:08.503631+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:39.406371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501590+0000","last_change":"2024-09-17T03:54:07.493181+0000","last_active":"2024-09-17T03:54:08.501590+0000","last_peered":"2024-09-17T03:54:08.501590+0000","last_clean":"2024-09-17T03:54:08.501590+0000","last_became_active":"2024-09-17T03:54:07.492903+0000","last_became_peered":"2024-09-17T03:54:07.492903+0000","last_unstale":"2024-09-17T03:54:08.501590+0000","last_undegraded":"2024-09-17T03:54:08.501590+0000","last_fullsized":"2024-09-17T03:54:08.501590+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:47.317249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503313+0000","last_change":"2024-09-17T03:54:06.501426+0000","last_active":"2024-09-17T03:54:08.503313+0000","last_peered":"2024-09-17T03:54:08.503313+0000","last_clean":"2024-09-17T03:54:08.503313+0000","last_became_active":"2024-09-17T03:54:06.500781+0000","last_became_peered":"2024-09-17T03:54:06.500781+0000","last_unstale":"2024-09-17T03:54:08.503313+0000","last_undegraded":"2024-09-17T03:54:08.503313+0000","last_fullsized":"2024-09-17T03:54:08.503313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:24.871710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.516145+0000","last_change":"2024-09-17T03:54:07.510555+0000","last_active":"2024-09-17T03:54:07.516145+0000","last_peered":"2024-09-17T03:54:07.516145+0000","last_clean":"2024-09-17T03:54:07.516145+0000","last_became_active":"2024-09-17T03:54:07.510199+0000","last_became_peered":"2024-09-17T03:54:07.510199+0000","last_unstale":"2024-09-17T03:54:07.516145+0000","last_undegraded":"2024-09-17T03:54:07.516145+0000","last_fullsized":"2024-09-17T03:54:07.516145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:05.071652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.510622+0000","last_change":"2024-09-17T03:54:06.499842+0000","last_active":"2024-09-17T03:54:07.510622+0000","last_peered":"2024-09-17T03:54:07.510622+0000","last_clean":"2024-09-17T03:54:07.510622+0000","last_became_active":"2024-09-17T03:54:06.499557+0000","last_became_peered":"2024-09-17T03:54:06.499557+0000","last_unstale":"2024-09-17T03:54:07.510622+0000","last_undegraded":"2024-09-17T03:54:07.510622+0000","last_fullsized":"2024-09-17T03:54:07.510622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:04.069003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523593+0000","last_change":"2024-09-17T03:54:07.517762+0000","last_active":"2024-09-17T03:54:07.523593+0000","last_peered":"2024-09-17T03:54:07.523593+0000","last_clean":"2024-09-17T03:54:07.523593+0000","last_became_active":"2024-09-17T03:54:07.517410+0000","last_became_peered":"2024-09-17T03:54:07.517410+0000","last_unstale":"2024-09-17T03:54:07.523593+0000","last_undegraded":"2024-09-17T03:54:07.523593+0000","last_fullsized":"2024-09-17T03:54:07.523593+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:59.323939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503759+0000","last_change":"2024-09-17T03:54:06.500255+0000","last_active":"2024-09-17T03:54:08.503759+0000","last_peered":"2024-09-17T03:54:08.503759+0000","last_clean":"2024-09-17T03:54:08.503759+0000","last_became_active":"2024-09-17T03:54:06.499799+0000","last_became_peered":"2024-09-17T03:54:06.499799+0000","last_unstale":"2024-09-17T03:54:08.503759+0000","last_undegraded":"2024-09-17T03:54:08.503759+0000","last_fullsized":"2024-09-17T03:54:08.503759+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:19.457347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526361+0000","last_change":"2024-09-17T03:54:07.520359+0000","last_active":"2024-09-17T03:54:07.526361+0000","last_peered":"2024-09-17T03:54:07.526361+0000","last_clean":"2024-09-17T03:54:07.526361+0000","last_became_active":"2024-09-17T03:54:07.515025+0000","last_became_peered":"2024-09-17T03:54:07.515025+0000","last_unstale":"2024-09-17T03:54:07.526361+0000","last_undegraded":"2024-09-17T03:54:07.526361+0000","last_fullsized":"2024-09-17T03:54:07.526361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:43.793289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515661+0000","last_change":"2024-09-17T03:54:06.497203+0000","last_active":"2024-09-17T03:54:07.515661+0000","last_peered":"2024-09-17T03:54:07.515661+0000","last_clean":"2024-09-17T03:54:07.515661+0000","last_became_active":"2024-09-17T03:54:06.496848+0000","last_became_peered":"2024-09-17T03:54:06.496848+0000","last_unstale":"2024-09-17T03:54:07.515661+0000","last_undegraded":"2024-09-17T03:54:07.515661+0000","last_fullsized":"2024-09-17T03:54:07.515661+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:34.273653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524100+0000","last_change":"2024-09-17T03:54:07.511643+0000","last_active":"2024-09-17T03:54:07.524100+0000","last_peered":"2024-09-17T03:54:07.524100+0000","last_clean":"2024-09-17T03:54:07.524100+0000","last_became_active":"2024-09-17T03:54:07.507566+0000","last_became_peered":"2024-09-17T03:54:07.507566+0000","last_unstale":"2024-09-17T03:54:07.524100+0000","last_undegraded":"2024-09-17T03:54:07.524100+0000","last_fullsized":"2024-09-17T03:54:07.524100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:41.954877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.511502+0000","last_change":"2024-09-17T03:54:06.500628+0000","last_active":"2024-09-17T03:54:07.511502+0000","last_peered":"2024-09-17T03:54:07.511502+0000","last_clean":"2024-09-17T03:54:07.511502+0000","last_became_active":"2024-09-17T03:54:06.500329+0000","last_became_peered":"2024-09-17T03:54:06.500329+0000","last_unstale":"2024-09-17T03:54:07.511502+0000","last_undegraded":"2024-09-17T03:54:07.511502+0000","last_fullsized":"2024-09-17T03:54:07.511502+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:25.765071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512804+0000","last_change":"2024-09-17T03:54:07.503041+0000","last_active":"2024-09-17T03:54:07.512804+0000","last_peered":"2024-09-17T03:54:07.512804+0000","last_clean":"2024-09-17T03:54:07.512804+0000","last_became_active":"2024-09-17T03:54:07.502697+0000","last_became_peered":"2024-09-17T03:54:07.502697+0000","last_unstale":"2024-09-17T03:54:07.512804+0000","last_undegraded":"2024-09-17T03:54:07.512804+0000","last_fullsized":"2024-09-17T03:54:07.512804+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:43:52.796826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.494167+0000","last_change":"2024-09-17T03:54:06.500838+0000","last_active":"2024-09-17T03:54:07.494167+0000","last_peered":"2024-09-17T03:54:07.494167+0000","last_clean":"2024-09-17T03:54:07.494167+0000","last_became_active":"2024-09-17T03:54:06.500220+0000","last_became_peered":"2024-09-17T03:54:06.500220+0000","last_unstale":"2024-09-17T03:54:07.494167+0000","last_undegraded":"2024-09-17T03:54:07.494167+0000","last_fullsized":"2024-09-17T03:54:07.494167+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:58.815768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502334+0000","last_change":"2024-09-17T03:54:07.504759+0000","last_active":"2024-09-17T03:54:08.502334+0000","last_peered":"2024-09-17T03:54:08.502334+0000","last_clean":"2024-09-17T03:54:08.502334+0000","last_became_active":"2024-09-17T03:54:07.504356+0000","last_became_peered":"2024-09-17T03:54:07.504356+0000","last_unstale":"2024-09-17T03:54:08.502334+0000","last_undegraded":"2024-09-17T03:54:08.502334+0000","last_fullsized":"2024-09-17T03:54:08.502334+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:49.216622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.510787+0000","last_change":"2024-09-17T03:54:06.494509+0000","last_active":"2024-09-17T03:54:07.510787+0000","last_peered":"2024-09-17T03:54:07.510787+0000","last_clean":"2024-09-17T03:54:07.510787+0000","last_became_active":"2024-09-17T03:54:06.493460+0000","last_became_peered":"2024-09-17T03:54:06.493460+0000","last_unstale":"2024-09-17T03:54:07.510787+0000","last_undegraded":"2024-09-17T03:54:07.510787+0000","last_fullsized":"2024-09-17T03:54:07.510787+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:43.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.506577+0000","last_change":"2024-09-17T03:54:07.501214+0000","last_active":"2024-09-17T03:54:07.506577+0000","last_peered":"2024-09-17T03:54:07.506577+0000","last_clean":"2024-09-17T03:54:07.506577+0000","last_became_active":"2024-09-17T03:54:07.500889+0000","last_became_peered":"2024-09-17T03:54:07.500889+0000","last_unstale":"2024-09-17T03:54:07.506577+0000","last_undegraded":"2024-09-17T03:54:07.506577+0000","last_fullsized":"2024-09-17T03:54:07.506577+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:13.978918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.493753+0000","last_change":"2024-09-17T03:54:06.506480+0000","last_active":"2024-09-17T03:54:07.493753+0000","last_peered":"2024-09-17T03:54:07.493753+0000","last_clean":"2024-09-17T03:54:07.493753+0000","last_became_active":"2024-09-17T03:54:06.506044+0000","last_became_peered":"2024-09-17T03:54:06.506044+0000","last_unstale":"2024-09-17T03:54:07.493753+0000","last_undegraded":"2024-09-17T03:54:07.493753+0000","last_fullsized":"2024-09-17T03:54:07.493753+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:54.055005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501628+0000","last_change":"2024-09-17T03:54:07.504147+0000","last_active":"2024-09-17T03:54:08.501628+0000","last_peered":"2024-09-17T03:54:08.501628+0000","last_clean":"2024-09-17T03:54:08.501628+0000","last_became_active":"2024-09-17T03:54:07.503070+0000","last_became_peered":"2024-09-17T03:54:07.503070+0000","last_unstale":"2024-09-17T03:54:08.501628+0000","last_undegraded":"2024-09-17T03:54:08.501628+0000","last_fullsized":"2024-09-17T03:54:08.501628+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:22.115441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503330+0000","last_change":"2024-09-17T03:54:06.500713+0000","last_active":"2024-09-17T03:54:08.503330+0000","last_peered":"2024-09-17T03:54:08.503330+0000","last_clean":"2024-09-17T03:54:08.503330+0000","last_became_active":"2024-09-17T03:54:06.500081+0000","last_became_peered":"2024-09-17T03:54:06.500081+0000","last_unstale":"2024-09-17T03:54:08.503330+0000","last_undegraded":"2024-09-17T03:54:08.503330+0000","last_fullsized":"2024-09-17T03:54:08.503330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:09.049696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513064+0000","last_change":"2024-09-17T03:54:07.508155+0000","last_active":"2024-09-17T03:54:07.513064+0000","last_peered":"2024-09-17T03:54:07.513064+0000","last_clean":"2024-09-17T03:54:07.513064+0000","last_became_active":"2024-09-17T03:54:07.503157+0000","last_became_peered":"2024-09-17T03:54:07.503157+0000","last_unstale":"2024-09-17T03:54:07.513064+0000","last_undegraded":"2024-09-17T03:54:07.513064+0000","last_fullsized":"2024-09-17T03:54:07.513064+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:56.731481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501663+0000","last_change":"2024-09-17T03:54:06.503324+0000","last_active":"2024-09-17T03:54:08.501663+0000","last_peered":"2024-09-17T03:54:08.501663+0000","last_clean":"2024-09-17T03:54:08.501663+0000","last_became_active":"2024-09-17T03:54:06.503001+0000","last_became_peered":"2024-09-17T03:54:06.503001+0000","last_unstale":"2024-09-17T03:54:08.501663+0000","last_undegraded":"2024-09-17T03:54:08.501663+0000","last_fullsized":"2024-09-17T03:54:08.501663+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:43.088353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503432+0000","last_change":"2024-09-17T03:54:07.498775+0000","last_active":"2024-09-17T03:54:08.503432+0000","last_peered":"2024-09-17T03:54:08.503432+0000","last_clean":"2024-09-17T03:54:08.503432+0000","last_became_active":"2024-09-17T03:54:07.498457+0000","last_became_peered":"2024-09-17T03:54:07.498457+0000","last_unstale":"2024-09-17T03:54:08.503432+0000","last_undegraded":"2024-09-17T03:54:08.503432+0000","last_fullsized":"2024-09-17T03:54:08.503432+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:21.561531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.522643+0000","last_change":"2024-09-17T03:54:06.502684+0000","last_active":"2024-09-17T03:54:07.522643+0000","last_peered":"2024-09-17T03:54:07.522643+0000","last_clean":"2024-09-17T03:54:07.522643+0000","last_became_active":"2024-09-17T03:54:06.502209+0000","last_became_peered":"2024-09-17T03:54:06.502209+0000","last_unstale":"2024-09-17T03:54:07.522643+0000","last_undegraded":"2024-09-17T03:54:07.522643+0000","last_fullsized":"2024-09-17T03:54:07.522643+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:51.265513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":238,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.484508+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:54:07.484508+0000","last_peered":"2024-09-17T03:54:07.484508+0000","last_clean":"2024-09-17T03:54:07.484508+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:54:07.484508+0000","last_undegraded":"2024-09-17T03:54:07.484508+0000","last_fullsized":"2024-09-17T03:54:07.484508+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1720320,"data_stored":1063318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509502,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1134592,"data_stored":479208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509501,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1138688,"data_stored":474574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1146880,"data_stored":480772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509501,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1134592,"data_stored":479172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509501,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1118208,"data_stored":472838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":472908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542204,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1716224,"data_stored":1063788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:54:11.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:54:12.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 -- 172.21.15.129:0/2596004533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300cad30 msgr2=0x7f22300cba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 --2- 172.21.15.129:0/2596004533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300cad30 0x7f22300cba00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f2228005fd0 tx=0x7f222802a940 comp rx=0 tx=0).stop 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 -- 172.21.15.129:0/2596004533 shutdown_connections 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 --2- 172.21.15.129:0/2596004533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300cad30 0x7f22300cba00 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 -- 172.21.15.129:0/2596004533 >> 172.21.15.129:0/2596004533 conn(0x7f22300b1cb0 msgr2=0x7f22300b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 -- 172.21.15.129:0/2596004533 shutdown_connections 2024-09-17T03:54:12.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.179+0000 7f2235000640 1 -- 172.21.15.129:0/2596004533 wait complete. 2024-09-17T03:54:12.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.181+0000 7f2235000640 1 Processor -- start 2024-09-17T03:54:12.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.182+0000 7f2235000640 1 -- start start 2024-09-17T03:54:12.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22300ca800 0x7f2230159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22300cad30 0x7f22300b0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f22300740c0 con 0x7f22300b1490 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2230073f40 con 0x7f22300cad30 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f2235000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2230074240 con 0x7f22300ca800 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f222e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f222e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57846/0 (socket says 172.21.15.129:57846) 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.183+0000 7f222e800640 1 -- 172.21.15.129:0/2765947156 learned_addr learned my addr 172.21.15.129:0/2765947156 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f2227e00640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22300cad30 0x7f22300b0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222de00640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22300ca800 0x7f2230159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 -- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22300ca800 msgr2=0x7f2230159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22300ca800 0x7f2230159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 -- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22300cad30 msgr2=0x7f22300b0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22300cad30 0x7f22300b0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 -- 172.21.15.129:0/2765947156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22301f8e10 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f222e800640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2228009cd0 tx=0x7f2228031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222803e070 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2228004540 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2228043460 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.184+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22301f90f0 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.185+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22301f96b0 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.186+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22300747d0 con 0x7f22300b1490 2024-09-17T03:54:12.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.187+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2228043600 con 0x7f22300b1490 2024-09-17T03:54:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.188+0000 7f2226000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 0x7f220c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.188+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f22280bca50 con 0x7f22300b1490 2024-09-17T03:54:12.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.188+0000 7f222de00640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 0x7f220c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.189+0000 7f222de00640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 0x7f220c07b0a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2220000f60 tx=0x7f2220009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:12.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.192+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f222804a020 con 0x7f22300b1490 2024-09-17T03:54:12.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.374+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f22300b19d0 con 0x7f220c078b90 2024-09-17T03:54:12.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.379+0000 7f2226000640 1 -- 172.21.15.129:0/2765947156 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252070 (secure 0 0 0) 0x7f22300b19d0 con 0x7f220c078b90 2024-09-17T03:54:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:12.383 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 msgr2=0x7f220c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 0x7f220c07b0a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2220000f60 tx=0x7f2220009040 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 msgr2=0x7f22300adeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f2228009cd0 tx=0x7f2228031e20 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 shutdown_connections 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f220c078b90 0x7f220c07b0a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f22300b1490 0x7f22300adeb0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22300cad30 0x7f22300b0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 --2- 172.21.15.129:0/2765947156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f22300ca800 0x7f2230159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 >> 172.21.15.129:0/2765947156 conn(0x7f22300b1cb0 msgr2=0x7f223015b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 shutdown_connections 2024-09-17T03:54:12.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.384+0000 7f2235000640 1 -- 172.21.15.129:0/2765947156 wait complete. 2024-09-17T03:54:12.406 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-17T03:54:12.370166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263688,"kb_used_data":9992,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754711032,"statfs":{"total":773094113280,"available":772824096768,"internally_reserved":0,"allocated":10231808,"data_stored":4986578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":259575344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001932"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524290+0000","last_change":"2024-09-17T03:54:07.514193+0000","last_active":"2024-09-17T03:54:07.524290+0000","last_peered":"2024-09-17T03:54:07.524290+0000","last_clean":"2024-09-17T03:54:07.524290+0000","last_became_active":"2024-09-17T03:54:07.513894+0000","last_became_peered":"2024-09-17T03:54:07.513894+0000","last_unstale":"2024-09-17T03:54:07.524290+0000","last_undegraded":"2024-09-17T03:54:07.524290+0000","last_fullsized":"2024-09-17T03:54:07.524290+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:20:11.829301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523618+0000","last_change":"2024-09-17T03:54:07.520952+0000","last_active":"2024-09-17T03:54:07.523618+0000","last_peered":"2024-09-17T03:54:07.523618+0000","last_clean":"2024-09-17T03:54:07.523618+0000","last_became_active":"2024-09-17T03:54:07.520678+0000","last_became_peered":"2024-09-17T03:54:07.520678+0000","last_unstale":"2024-09-17T03:54:07.523618+0000","last_undegraded":"2024-09-17T03:54:07.523618+0000","last_fullsized":"2024-09-17T03:54:07.523618+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:28.803341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524332+0000","last_change":"2024-09-17T03:54:07.514034+0000","last_active":"2024-09-17T03:54:07.524332+0000","last_peered":"2024-09-17T03:54:07.524332+0000","last_clean":"2024-09-17T03:54:07.524332+0000","last_became_active":"2024-09-17T03:54:07.513744+0000","last_became_peered":"2024-09-17T03:54:07.513744+0000","last_unstale":"2024-09-17T03:54:07.524332+0000","last_undegraded":"2024-09-17T03:54:07.524332+0000","last_fullsized":"2024-09-17T03:54:07.524332+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:08.025468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503865+0000","last_change":"2024-09-17T03:54:07.513408+0000","last_active":"2024-09-17T03:54:08.503865+0000","last_peered":"2024-09-17T03:54:08.503865+0000","last_clean":"2024-09-17T03:54:08.503865+0000","last_became_active":"2024-09-17T03:54:07.508028+0000","last_became_peered":"2024-09-17T03:54:07.508028+0000","last_unstale":"2024-09-17T03:54:08.503865+0000","last_undegraded":"2024-09-17T03:54:08.503865+0000","last_fullsized":"2024-09-17T03:54:08.503865+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:11.607364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512093+0000","last_change":"2024-09-17T03:54:07.507443+0000","last_active":"2024-09-17T03:54:07.512093+0000","last_peered":"2024-09-17T03:54:07.512093+0000","last_clean":"2024-09-17T03:54:07.512093+0000","last_became_active":"2024-09-17T03:54:07.504889+0000","last_became_peered":"2024-09-17T03:54:07.504889+0000","last_unstale":"2024-09-17T03:54:07.512093+0000","last_undegraded":"2024-09-17T03:54:07.512093+0000","last_fullsized":"2024-09-17T03:54:07.512093+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:47.689693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515758+0000","last_change":"2024-09-17T03:54:07.511178+0000","last_active":"2024-09-17T03:54:07.515758+0000","last_peered":"2024-09-17T03:54:07.515758+0000","last_clean":"2024-09-17T03:54:07.515758+0000","last_became_active":"2024-09-17T03:54:07.508236+0000","last_became_peered":"2024-09-17T03:54:07.508236+0000","last_unstale":"2024-09-17T03:54:07.515758+0000","last_undegraded":"2024-09-17T03:54:07.515758+0000","last_fullsized":"2024-09-17T03:54:07.515758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:54.495394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.508372+0000","last_change":"2024-09-17T03:54:07.499852+0000","last_active":"2024-09-17T03:54:07.508372+0000","last_peered":"2024-09-17T03:54:07.508372+0000","last_clean":"2024-09-17T03:54:07.508372+0000","last_became_active":"2024-09-17T03:54:07.499500+0000","last_became_peered":"2024-09-17T03:54:07.499500+0000","last_unstale":"2024-09-17T03:54:07.508372+0000","last_undegraded":"2024-09-17T03:54:07.508372+0000","last_fullsized":"2024-09-17T03:54:07.508372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:02.238278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523423+0000","last_change":"2024-09-17T03:54:07.517037+0000","last_active":"2024-09-17T03:54:07.523423+0000","last_peered":"2024-09-17T03:54:07.523423+0000","last_clean":"2024-09-17T03:54:07.523423+0000","last_became_active":"2024-09-17T03:54:07.516682+0000","last_became_peered":"2024-09-17T03:54:07.516682+0000","last_unstale":"2024-09-17T03:54:07.523423+0000","last_undegraded":"2024-09-17T03:54:07.523423+0000","last_fullsized":"2024-09-17T03:54:07.523423+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:38:11.953720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513328+0000","last_change":"2024-09-17T03:54:07.507628+0000","last_active":"2024-09-17T03:54:07.513328+0000","last_peered":"2024-09-17T03:54:07.513328+0000","last_clean":"2024-09-17T03:54:07.513328+0000","last_became_active":"2024-09-17T03:54:07.507270+0000","last_became_peered":"2024-09-17T03:54:07.507270+0000","last_unstale":"2024-09-17T03:54:07.513328+0000","last_undegraded":"2024-09-17T03:54:07.513328+0000","last_fullsized":"2024-09-17T03:54:07.513328+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:46.163515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524592+0000","last_change":"2024-09-17T03:54:07.522341+0000","last_active":"2024-09-17T03:54:07.524592+0000","last_peered":"2024-09-17T03:54:07.524592+0000","last_clean":"2024-09-17T03:54:07.524592+0000","last_became_active":"2024-09-17T03:54:07.521913+0000","last_became_peered":"2024-09-17T03:54:07.521913+0000","last_unstale":"2024-09-17T03:54:07.524592+0000","last_undegraded":"2024-09-17T03:54:07.524592+0000","last_fullsized":"2024-09-17T03:54:07.524592+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:09.798069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514069+0000","last_change":"2024-09-17T03:54:07.507184+0000","last_active":"2024-09-17T03:54:07.514069+0000","last_peered":"2024-09-17T03:54:07.514069+0000","last_clean":"2024-09-17T03:54:07.514069+0000","last_became_active":"2024-09-17T03:54:07.503029+0000","last_became_peered":"2024-09-17T03:54:07.503029+0000","last_unstale":"2024-09-17T03:54:07.514069+0000","last_undegraded":"2024-09-17T03:54:07.514069+0000","last_fullsized":"2024-09-17T03:54:07.514069+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:26.434816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512662+0000","last_change":"2024-09-17T03:54:07.502612+0000","last_active":"2024-09-17T03:54:07.512662+0000","last_peered":"2024-09-17T03:54:07.512662+0000","last_clean":"2024-09-17T03:54:07.512662+0000","last_became_active":"2024-09-17T03:54:07.502316+0000","last_became_peered":"2024-09-17T03:54:07.502316+0000","last_unstale":"2024-09-17T03:54:07.512662+0000","last_undegraded":"2024-09-17T03:54:07.512662+0000","last_fullsized":"2024-09-17T03:54:07.512662+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:52.268162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503311+0000","last_change":"2024-09-17T03:54:07.501178+0000","last_active":"2024-09-17T03:54:08.503311+0000","last_peered":"2024-09-17T03:54:08.503311+0000","last_clean":"2024-09-17T03:54:08.503311+0000","last_became_active":"2024-09-17T03:54:07.500894+0000","last_became_peered":"2024-09-17T03:54:07.500894+0000","last_unstale":"2024-09-17T03:54:08.503311+0000","last_undegraded":"2024-09-17T03:54:08.503311+0000","last_fullsized":"2024-09-17T03:54:08.503311+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:37.646246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502164+0000","last_change":"2024-09-17T03:54:07.498806+0000","last_active":"2024-09-17T03:54:08.502164+0000","last_peered":"2024-09-17T03:54:08.502164+0000","last_clean":"2024-09-17T03:54:08.502164+0000","last_became_active":"2024-09-17T03:54:07.498490+0000","last_became_peered":"2024-09-17T03:54:07.498490+0000","last_unstale":"2024-09-17T03:54:08.502164+0000","last_undegraded":"2024-09-17T03:54:08.502164+0000","last_fullsized":"2024-09-17T03:54:08.502164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:30.836203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502852+0000","last_change":"2024-09-17T03:54:07.509867+0000","last_active":"2024-09-17T03:54:08.502852+0000","last_peered":"2024-09-17T03:54:08.502852+0000","last_clean":"2024-09-17T03:54:08.502852+0000","last_became_active":"2024-09-17T03:54:07.509574+0000","last_became_peered":"2024-09-17T03:54:07.509574+0000","last_unstale":"2024-09-17T03:54:08.502852+0000","last_undegraded":"2024-09-17T03:54:08.502852+0000","last_fullsized":"2024-09-17T03:54:08.502852+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:41.937363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515891+0000","last_change":"2024-09-17T03:54:07.513453+0000","last_active":"2024-09-17T03:54:07.515891+0000","last_peered":"2024-09-17T03:54:07.515891+0000","last_clean":"2024-09-17T03:54:07.515891+0000","last_became_active":"2024-09-17T03:54:07.512032+0000","last_became_peered":"2024-09-17T03:54:07.512032+0000","last_unstale":"2024-09-17T03:54:07.515891+0000","last_undegraded":"2024-09-17T03:54:07.515891+0000","last_fullsized":"2024-09-17T03:54:07.515891+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:36.809386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523668+0000","last_change":"2024-09-17T03:54:07.518054+0000","last_active":"2024-09-17T03:54:07.523668+0000","last_peered":"2024-09-17T03:54:07.523668+0000","last_clean":"2024-09-17T03:54:07.523668+0000","last_became_active":"2024-09-17T03:54:07.517620+0000","last_became_peered":"2024-09-17T03:54:07.517620+0000","last_unstale":"2024-09-17T03:54:07.523668+0000","last_undegraded":"2024-09-17T03:54:07.523668+0000","last_fullsized":"2024-09-17T03:54:07.523668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:44.811654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503970+0000","last_change":"2024-09-17T03:54:07.506526+0000","last_active":"2024-09-17T03:54:08.503970+0000","last_peered":"2024-09-17T03:54:08.503970+0000","last_clean":"2024-09-17T03:54:08.503970+0000","last_became_active":"2024-09-17T03:54:07.506199+0000","last_became_peered":"2024-09-17T03:54:07.506199+0000","last_unstale":"2024-09-17T03:54:08.503970+0000","last_undegraded":"2024-09-17T03:54:08.503970+0000","last_fullsized":"2024-09-17T03:54:08.503970+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:28.642053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502080+0000","last_change":"2024-09-17T03:54:07.509480+0000","last_active":"2024-09-17T03:54:08.502080+0000","last_peered":"2024-09-17T03:54:08.502080+0000","last_clean":"2024-09-17T03:54:08.502080+0000","last_became_active":"2024-09-17T03:54:07.508967+0000","last_became_peered":"2024-09-17T03:54:07.508967+0000","last_unstale":"2024-09-17T03:54:08.502080+0000","last_undegraded":"2024-09-17T03:54:08.502080+0000","last_fullsized":"2024-09-17T03:54:08.502080+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:19.393480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513474+0000","last_change":"2024-09-17T03:54:07.505671+0000","last_active":"2024-09-17T03:54:07.513474+0000","last_peered":"2024-09-17T03:54:07.513474+0000","last_clean":"2024-09-17T03:54:07.513474+0000","last_became_active":"2024-09-17T03:54:07.505347+0000","last_became_peered":"2024-09-17T03:54:07.505347+0000","last_unstale":"2024-09-17T03:54:07.513474+0000","last_undegraded":"2024-09-17T03:54:07.513474+0000","last_fullsized":"2024-09-17T03:54:07.513474+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:10.052345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502216+0000","last_change":"2024-09-17T03:54:07.501099+0000","last_active":"2024-09-17T03:54:08.502216+0000","last_peered":"2024-09-17T03:54:08.502216+0000","last_clean":"2024-09-17T03:54:08.502216+0000","last_became_active":"2024-09-17T03:54:07.500794+0000","last_became_peered":"2024-09-17T03:54:07.500794+0000","last_unstale":"2024-09-17T03:54:08.502216+0000","last_undegraded":"2024-09-17T03:54:08.502216+0000","last_fullsized":"2024-09-17T03:54:08.502216+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:40.214091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524068+0000","last_change":"2024-09-17T03:54:07.519854+0000","last_active":"2024-09-17T03:54:07.524068+0000","last_peered":"2024-09-17T03:54:07.524068+0000","last_clean":"2024-09-17T03:54:07.524068+0000","last_became_active":"2024-09-17T03:54:07.519578+0000","last_became_peered":"2024-09-17T03:54:07.519578+0000","last_unstale":"2024-09-17T03:54:07.524068+0000","last_undegraded":"2024-09-17T03:54:07.524068+0000","last_fullsized":"2024-09-17T03:54:07.524068+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:53.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523275+0000","last_change":"2024-09-17T03:54:07.516524+0000","last_active":"2024-09-17T03:54:07.523275+0000","last_peered":"2024-09-17T03:54:07.523275+0000","last_clean":"2024-09-17T03:54:07.523275+0000","last_became_active":"2024-09-17T03:54:07.516214+0000","last_became_peered":"2024-09-17T03:54:07.516214+0000","last_unstale":"2024-09-17T03:54:07.523275+0000","last_undegraded":"2024-09-17T03:54:07.523275+0000","last_fullsized":"2024-09-17T03:54:07.523275+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:41.990100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513687+0000","last_change":"2024-09-17T03:54:07.502148+0000","last_active":"2024-09-17T03:54:07.513687+0000","last_peered":"2024-09-17T03:54:07.513687+0000","last_clean":"2024-09-17T03:54:07.513687+0000","last_became_active":"2024-09-17T03:54:07.501830+0000","last_became_peered":"2024-09-17T03:54:07.501830+0000","last_unstale":"2024-09-17T03:54:07.513687+0000","last_undegraded":"2024-09-17T03:54:07.513687+0000","last_fullsized":"2024-09-17T03:54:07.513687+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:24.253089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.517837+0000","last_change":"2024-09-17T03:54:07.510557+0000","last_active":"2024-09-17T03:54:07.517837+0000","last_peered":"2024-09-17T03:54:07.517837+0000","last_clean":"2024-09-17T03:54:07.517837+0000","last_became_active":"2024-09-17T03:54:07.510166+0000","last_became_peered":"2024-09-17T03:54:07.510166+0000","last_unstale":"2024-09-17T03:54:07.517837+0000","last_undegraded":"2024-09-17T03:54:07.517837+0000","last_fullsized":"2024-09-17T03:54:07.517837+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:44.143436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521181+0000","last_change":"2024-09-17T03:54:07.512465+0000","last_active":"2024-09-17T03:54:07.521181+0000","last_peered":"2024-09-17T03:54:07.521181+0000","last_clean":"2024-09-17T03:54:07.521181+0000","last_became_active":"2024-09-17T03:54:07.509547+0000","last_became_peered":"2024-09-17T03:54:07.509547+0000","last_unstale":"2024-09-17T03:54:07.521181+0000","last_undegraded":"2024-09-17T03:54:07.521181+0000","last_fullsized":"2024-09-17T03:54:07.521181+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:49.842716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514169+0000","last_change":"2024-09-17T03:54:07.508577+0000","last_active":"2024-09-17T03:54:07.514169+0000","last_peered":"2024-09-17T03:54:07.514169+0000","last_clean":"2024-09-17T03:54:07.514169+0000","last_became_active":"2024-09-17T03:54:07.507537+0000","last_became_peered":"2024-09-17T03:54:07.507537+0000","last_unstale":"2024-09-17T03:54:07.514169+0000","last_undegraded":"2024-09-17T03:54:07.514169+0000","last_fullsized":"2024-09-17T03:54:07.514169+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:11.333690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512864+0000","last_change":"2024-09-17T03:54:07.504594+0000","last_active":"2024-09-17T03:54:07.512864+0000","last_peered":"2024-09-17T03:54:07.512864+0000","last_clean":"2024-09-17T03:54:07.512864+0000","last_became_active":"2024-09-17T03:54:07.503195+0000","last_became_peered":"2024-09-17T03:54:07.503195+0000","last_unstale":"2024-09-17T03:54:07.512864+0000","last_undegraded":"2024-09-17T03:54:07.512864+0000","last_fullsized":"2024-09-17T03:54:07.512864+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:04.407555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.525955+0000","last_change":"2024-09-17T03:54:07.525228+0000","last_active":"2024-09-17T03:54:07.525955+0000","last_peered":"2024-09-17T03:54:07.525955+0000","last_clean":"2024-09-17T03:54:07.525955+0000","last_became_active":"2024-09-17T03:54:07.524909+0000","last_became_peered":"2024-09-17T03:54:07.524909+0000","last_unstale":"2024-09-17T03:54:07.525955+0000","last_undegraded":"2024-09-17T03:54:07.525955+0000","last_fullsized":"2024-09-17T03:54:07.525955+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:44.791790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501966+0000","last_change":"2024-09-17T03:54:07.500582+0000","last_active":"2024-09-17T03:54:08.501966+0000","last_peered":"2024-09-17T03:54:08.501966+0000","last_clean":"2024-09-17T03:54:08.501966+0000","last_became_active":"2024-09-17T03:54:07.499821+0000","last_became_peered":"2024-09-17T03:54:07.499821+0000","last_unstale":"2024-09-17T03:54:08.501966+0000","last_undegraded":"2024-09-17T03:54:08.501966+0000","last_fullsized":"2024-09-17T03:54:08.501966+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:58.669908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503197+0000","last_change":"2024-09-17T03:54:07.499625+0000","last_active":"2024-09-17T03:54:08.503197+0000","last_peered":"2024-09-17T03:54:08.503197+0000","last_clean":"2024-09-17T03:54:08.503197+0000","last_became_active":"2024-09-17T03:54:07.499273+0000","last_became_peered":"2024-09-17T03:54:07.499273+0000","last_unstale":"2024-09-17T03:54:08.503197+0000","last_undegraded":"2024-09-17T03:54:08.503197+0000","last_fullsized":"2024-09-17T03:54:08.503197+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:42.604448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526407+0000","last_change":"2024-09-17T03:54:07.519342+0000","last_active":"2024-09-17T03:54:07.526407+0000","last_peered":"2024-09-17T03:54:07.526407+0000","last_clean":"2024-09-17T03:54:07.526407+0000","last_became_active":"2024-09-17T03:54:07.519045+0000","last_became_peered":"2024-09-17T03:54:07.519045+0000","last_unstale":"2024-09-17T03:54:07.526407+0000","last_undegraded":"2024-09-17T03:54:07.526407+0000","last_fullsized":"2024-09-17T03:54:07.526407+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:59.727134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503191+0000","last_change":"2024-09-17T03:54:07.497783+0000","last_active":"2024-09-17T03:54:08.503191+0000","last_peered":"2024-09-17T03:54:08.503191+0000","last_clean":"2024-09-17T03:54:08.503191+0000","last_became_active":"2024-09-17T03:54:07.497442+0000","last_became_peered":"2024-09-17T03:54:07.497442+0000","last_unstale":"2024-09-17T03:54:08.503191+0000","last_undegraded":"2024-09-17T03:54:08.503191+0000","last_fullsized":"2024-09-17T03:54:08.503191+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:26.754285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"91'4","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.527202+0000","last_change":"2024-09-17T03:54:06.504697+0000","last_active":"2024-09-17T03:54:07.527202+0000","last_peered":"2024-09-17T03:54:07.527202+0000","last_clean":"2024-09-17T03:54:07.527202+0000","last_became_active":"2024-09-17T03:54:06.504343+0000","last_became_peered":"2024-09-17T03:54:06.504343+0000","last_unstale":"2024-09-17T03:54:07.527202+0000","last_undegraded":"2024-09-17T03:54:07.527202+0000","last_fullsized":"2024-09-17T03:54:07.527202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:08.795420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.504085+0000","last_change":"2024-09-17T03:54:07.503853+0000","last_active":"2024-09-17T03:54:08.504085+0000","last_peered":"2024-09-17T03:54:08.504085+0000","last_clean":"2024-09-17T03:54:08.504085+0000","last_became_active":"2024-09-17T03:54:07.502599+0000","last_became_peered":"2024-09-17T03:54:07.502599+0000","last_unstale":"2024-09-17T03:54:08.504085+0000","last_undegraded":"2024-09-17T03:54:08.504085+0000","last_fullsized":"2024-09-17T03:54:08.504085+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:37.518932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524893+0000","last_change":"2024-09-17T03:54:06.504622+0000","last_active":"2024-09-17T03:54:07.524893+0000","last_peered":"2024-09-17T03:54:07.524893+0000","last_clean":"2024-09-17T03:54:07.524893+0000","last_became_active":"2024-09-17T03:54:06.504295+0000","last_became_peered":"2024-09-17T03:54:06.504295+0000","last_unstale":"2024-09-17T03:54:07.524893+0000","last_undegraded":"2024-09-17T03:54:07.524893+0000","last_fullsized":"2024-09-17T03:54:07.524893+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:12.976269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524136+0000","last_change":"2024-09-17T03:54:07.513146+0000","last_active":"2024-09-17T03:54:07.524136+0000","last_peered":"2024-09-17T03:54:07.524136+0000","last_clean":"2024-09-17T03:54:07.524136+0000","last_became_active":"2024-09-17T03:54:07.512081+0000","last_became_peered":"2024-09-17T03:54:07.512081+0000","last_unstale":"2024-09-17T03:54:07.524136+0000","last_undegraded":"2024-09-17T03:54:07.524136+0000","last_fullsized":"2024-09-17T03:54:07.524136+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:56.818324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521720+0000","last_change":"2024-09-17T03:54:06.495577+0000","last_active":"2024-09-17T03:54:07.521720+0000","last_peered":"2024-09-17T03:54:07.521720+0000","last_clean":"2024-09-17T03:54:07.521720+0000","last_became_active":"2024-09-17T03:54:06.495026+0000","last_became_peered":"2024-09-17T03:54:06.495026+0000","last_unstale":"2024-09-17T03:54:07.521720+0000","last_undegraded":"2024-09-17T03:54:07.521720+0000","last_fullsized":"2024-09-17T03:54:07.521720+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:20.756533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521344+0000","last_change":"2024-09-17T03:54:07.516993+0000","last_active":"2024-09-17T03:54:07.521344+0000","last_peered":"2024-09-17T03:54:07.521344+0000","last_clean":"2024-09-17T03:54:07.521344+0000","last_became_active":"2024-09-17T03:54:07.516543+0000","last_became_peered":"2024-09-17T03:54:07.516543+0000","last_unstale":"2024-09-17T03:54:07.521344+0000","last_undegraded":"2024-09-17T03:54:07.521344+0000","last_fullsized":"2024-09-17T03:54:07.521344+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:26.434816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.522005+0000","last_change":"2024-09-17T03:54:06.498040+0000","last_active":"2024-09-17T03:54:07.522005+0000","last_peered":"2024-09-17T03:54:07.522005+0000","last_clean":"2024-09-17T03:54:07.522005+0000","last_became_active":"2024-09-17T03:54:06.497769+0000","last_became_peered":"2024-09-17T03:54:06.497769+0000","last_unstale":"2024-09-17T03:54:07.522005+0000","last_undegraded":"2024-09-17T03:54:07.522005+0000","last_fullsized":"2024-09-17T03:54:07.522005+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:57.329153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503806+0000","last_change":"2024-09-17T03:54:07.508884+0000","last_active":"2024-09-17T03:54:08.503806+0000","last_peered":"2024-09-17T03:54:08.503806+0000","last_clean":"2024-09-17T03:54:08.503806+0000","last_became_active":"2024-09-17T03:54:07.508533+0000","last_became_peered":"2024-09-17T03:54:07.508533+0000","last_unstale":"2024-09-17T03:54:08.503806+0000","last_undegraded":"2024-09-17T03:54:08.503806+0000","last_fullsized":"2024-09-17T03:54:08.503806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:28.733545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524946+0000","last_change":"2024-09-17T03:54:06.505075+0000","last_active":"2024-09-17T03:54:07.524946+0000","last_peered":"2024-09-17T03:54:07.524946+0000","last_clean":"2024-09-17T03:54:07.524946+0000","last_became_active":"2024-09-17T03:54:06.504781+0000","last_became_peered":"2024-09-17T03:54:06.504781+0000","last_unstale":"2024-09-17T03:54:07.524946+0000","last_undegraded":"2024-09-17T03:54:07.524946+0000","last_fullsized":"2024-09-17T03:54:07.524946+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:42.673956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.514371+0000","last_change":"2024-09-17T03:54:07.509678+0000","last_active":"2024-09-17T03:54:07.514371+0000","last_peered":"2024-09-17T03:54:07.514371+0000","last_clean":"2024-09-17T03:54:07.514371+0000","last_became_active":"2024-09-17T03:54:07.507855+0000","last_became_peered":"2024-09-17T03:54:07.507855+0000","last_unstale":"2024-09-17T03:54:07.514371+0000","last_undegraded":"2024-09-17T03:54:07.514371+0000","last_fullsized":"2024-09-17T03:54:07.514371+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:34:14.900411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526527+0000","last_change":"2024-09-17T03:54:06.504445+0000","last_active":"2024-09-17T03:54:07.526527+0000","last_peered":"2024-09-17T03:54:07.526527+0000","last_clean":"2024-09-17T03:54:07.526527+0000","last_became_active":"2024-09-17T03:54:06.503430+0000","last_became_peered":"2024-09-17T03:54:06.503430+0000","last_unstale":"2024-09-17T03:54:07.526527+0000","last_undegraded":"2024-09-17T03:54:07.526527+0000","last_fullsized":"2024-09-17T03:54:07.526527+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:46.687044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502901+0000","last_change":"2024-09-17T03:54:07.502148+0000","last_active":"2024-09-17T03:54:08.502901+0000","last_peered":"2024-09-17T03:54:08.502901+0000","last_clean":"2024-09-17T03:54:08.502901+0000","last_became_active":"2024-09-17T03:54:07.499812+0000","last_became_peered":"2024-09-17T03:54:07.499812+0000","last_unstale":"2024-09-17T03:54:08.502901+0000","last_undegraded":"2024-09-17T03:54:08.502901+0000","last_fullsized":"2024-09-17T03:54:08.502901+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:10.977290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.492994+0000","last_change":"2024-09-17T03:54:06.505126+0000","last_active":"2024-09-17T03:54:07.492994+0000","last_peered":"2024-09-17T03:54:07.492994+0000","last_clean":"2024-09-17T03:54:07.492994+0000","last_became_active":"2024-09-17T03:54:06.504801+0000","last_became_peered":"2024-09-17T03:54:06.504801+0000","last_unstale":"2024-09-17T03:54:07.492994+0000","last_undegraded":"2024-09-17T03:54:07.492994+0000","last_fullsized":"2024-09-17T03:54:07.492994+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:08.265326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521030+0000","last_change":"2024-09-17T03:54:07.515697+0000","last_active":"2024-09-17T03:54:07.521030+0000","last_peered":"2024-09-17T03:54:07.521030+0000","last_clean":"2024-09-17T03:54:07.521030+0000","last_became_active":"2024-09-17T03:54:07.515284+0000","last_became_peered":"2024-09-17T03:54:07.515284+0000","last_unstale":"2024-09-17T03:54:07.521030+0000","last_undegraded":"2024-09-17T03:54:07.521030+0000","last_fullsized":"2024-09-17T03:54:07.521030+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:24.253089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.503576+0000","last_change":"2024-09-17T03:54:06.502158+0000","last_active":"2024-09-17T03:54:07.503576+0000","last_peered":"2024-09-17T03:54:07.503576+0000","last_clean":"2024-09-17T03:54:07.503576+0000","last_became_active":"2024-09-17T03:54:06.501853+0000","last_became_peered":"2024-09-17T03:54:06.501853+0000","last_unstale":"2024-09-17T03:54:07.503576+0000","last_undegraded":"2024-09-17T03:54:07.503576+0000","last_fullsized":"2024-09-17T03:54:07.503576+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:20.558882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523354+0000","last_change":"2024-09-17T03:54:07.517010+0000","last_active":"2024-09-17T03:54:07.523354+0000","last_peered":"2024-09-17T03:54:07.523354+0000","last_clean":"2024-09-17T03:54:07.523354+0000","last_became_active":"2024-09-17T03:54:07.516714+0000","last_became_peered":"2024-09-17T03:54:07.516714+0000","last_unstale":"2024-09-17T03:54:07.523354+0000","last_undegraded":"2024-09-17T03:54:07.523354+0000","last_fullsized":"2024-09-17T03:54:07.523354+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:06.198432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.506973+0000","last_change":"2024-09-17T03:54:06.506662+0000","last_active":"2024-09-17T03:54:07.506973+0000","last_peered":"2024-09-17T03:54:07.506973+0000","last_clean":"2024-09-17T03:54:07.506973+0000","last_became_active":"2024-09-17T03:54:06.506373+0000","last_became_peered":"2024-09-17T03:54:06.506373+0000","last_unstale":"2024-09-17T03:54:07.506973+0000","last_undegraded":"2024-09-17T03:54:07.506973+0000","last_fullsized":"2024-09-17T03:54:07.506973+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:41.558155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521580+0000","last_change":"2024-09-17T03:54:07.517161+0000","last_active":"2024-09-17T03:54:07.521580+0000","last_peered":"2024-09-17T03:54:07.521580+0000","last_clean":"2024-09-17T03:54:07.521580+0000","last_became_active":"2024-09-17T03:54:07.516751+0000","last_became_peered":"2024-09-17T03:54:07.516751+0000","last_unstale":"2024-09-17T03:54:07.521580+0000","last_undegraded":"2024-09-17T03:54:07.521580+0000","last_fullsized":"2024-09-17T03:54:07.521580+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:50:11.333690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503868+0000","last_change":"2024-09-17T03:54:06.503897+0000","last_active":"2024-09-17T03:54:08.503868+0000","last_peered":"2024-09-17T03:54:08.503868+0000","last_clean":"2024-09-17T03:54:08.503868+0000","last_became_active":"2024-09-17T03:54:06.503598+0000","last_became_peered":"2024-09-17T03:54:06.503598+0000","last_unstale":"2024-09-17T03:54:08.503868+0000","last_undegraded":"2024-09-17T03:54:08.503868+0000","last_fullsized":"2024-09-17T03:54:08.503868+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:01.235629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.509662+0000","last_change":"2024-09-17T03:54:07.502593+0000","last_active":"2024-09-17T03:54:07.509662+0000","last_peered":"2024-09-17T03:54:07.509662+0000","last_clean":"2024-09-17T03:54:07.509662+0000","last_became_active":"2024-09-17T03:54:07.502243+0000","last_became_peered":"2024-09-17T03:54:07.502243+0000","last_unstale":"2024-09-17T03:54:07.509662+0000","last_undegraded":"2024-09-17T03:54:07.509662+0000","last_fullsized":"2024-09-17T03:54:07.509662+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:17.600556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"91'3","reported_seq":16,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.520269+0000","last_change":"2024-09-17T03:54:06.499973+0000","last_active":"2024-09-17T03:54:07.520269+0000","last_peered":"2024-09-17T03:54:07.520269+0000","last_clean":"2024-09-17T03:54:07.520269+0000","last_became_active":"2024-09-17T03:54:06.499672+0000","last_became_peered":"2024-09-17T03:54:06.499672+0000","last_unstale":"2024-09-17T03:54:07.520269+0000","last_undegraded":"2024-09-17T03:54:07.520269+0000","last_fullsized":"2024-09-17T03:54:07.520269+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:09.049696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524327+0000","last_change":"2024-09-17T03:54:07.519883+0000","last_active":"2024-09-17T03:54:07.524327+0000","last_peered":"2024-09-17T03:54:07.524327+0000","last_clean":"2024-09-17T03:54:07.524327+0000","last_became_active":"2024-09-17T03:54:07.519107+0000","last_became_peered":"2024-09-17T03:54:07.519107+0000","last_unstale":"2024-09-17T03:54:07.524327+0000","last_undegraded":"2024-09-17T03:54:07.524327+0000","last_fullsized":"2024-09-17T03:54:07.524327+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:24.838940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"91'6","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.527612+0000","last_change":"2024-09-17T03:54:06.497042+0000","last_active":"2024-09-17T03:54:07.527612+0000","last_peered":"2024-09-17T03:54:07.527612+0000","last_clean":"2024-09-17T03:54:07.527612+0000","last_became_active":"2024-09-17T03:54:06.495801+0000","last_became_peered":"2024-09-17T03:54:06.495801+0000","last_unstale":"2024-09-17T03:54:07.527612+0000","last_undegraded":"2024-09-17T03:54:07.527612+0000","last_fullsized":"2024-09-17T03:54:07.527612+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:30.769908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503430+0000","last_change":"2024-09-17T03:54:07.491211+0000","last_active":"2024-09-17T03:54:08.503430+0000","last_peered":"2024-09-17T03:54:08.503430+0000","last_clean":"2024-09-17T03:54:08.503430+0000","last_became_active":"2024-09-17T03:54:07.490916+0000","last_became_peered":"2024-09-17T03:54:07.490916+0000","last_unstale":"2024-09-17T03:54:08.503430+0000","last_undegraded":"2024-09-17T03:54:08.503430+0000","last_fullsized":"2024-09-17T03:54:08.503430+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:56.420057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"91'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502532+0000","last_change":"2024-09-17T03:54:06.498677+0000","last_active":"2024-09-17T03:54:08.502532+0000","last_peered":"2024-09-17T03:54:08.502532+0000","last_clean":"2024-09-17T03:54:08.502532+0000","last_became_active":"2024-09-17T03:54:06.498351+0000","last_became_peered":"2024-09-17T03:54:06.498351+0000","last_unstale":"2024-09-17T03:54:08.502532+0000","last_undegraded":"2024-09-17T03:54:08.502532+0000","last_fullsized":"2024-09-17T03:54:08.502532+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:00.251956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526177+0000","last_change":"2024-09-17T03:54:07.513062+0000","last_active":"2024-09-17T03:54:07.526177+0000","last_peered":"2024-09-17T03:54:07.526177+0000","last_clean":"2024-09-17T03:54:07.526177+0000","last_became_active":"2024-09-17T03:54:07.512777+0000","last_became_peered":"2024-09-17T03:54:07.512777+0000","last_unstale":"2024-09-17T03:54:07.526177+0000","last_undegraded":"2024-09-17T03:54:07.526177+0000","last_fullsized":"2024-09-17T03:54:07.526177+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:44:50.008384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502333+0000","last_change":"2024-09-17T03:54:06.508013+0000","last_active":"2024-09-17T03:54:08.502333+0000","last_peered":"2024-09-17T03:54:08.502333+0000","last_clean":"2024-09-17T03:54:08.502333+0000","last_became_active":"2024-09-17T03:54:06.507685+0000","last_became_peered":"2024-09-17T03:54:06.507685+0000","last_unstale":"2024-09-17T03:54:08.502333+0000","last_undegraded":"2024-09-17T03:54:08.502333+0000","last_fullsized":"2024-09-17T03:54:08.502333+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:58.035209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513376+0000","last_change":"2024-09-17T03:54:07.505967+0000","last_active":"2024-09-17T03:54:07.513376+0000","last_peered":"2024-09-17T03:54:07.513376+0000","last_clean":"2024-09-17T03:54:07.513376+0000","last_became_active":"2024-09-17T03:54:07.505594+0000","last_became_peered":"2024-09-17T03:54:07.505594+0000","last_unstale":"2024-09-17T03:54:07.513376+0000","last_undegraded":"2024-09-17T03:54:07.513376+0000","last_fullsized":"2024-09-17T03:54:07.513376+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:05:02.238278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.494886+0000","last_change":"2024-09-17T03:54:06.504531+0000","last_active":"2024-09-17T03:54:07.494886+0000","last_peered":"2024-09-17T03:54:07.494886+0000","last_clean":"2024-09-17T03:54:07.494886+0000","last_became_active":"2024-09-17T03:54:06.504133+0000","last_became_peered":"2024-09-17T03:54:06.504133+0000","last_unstale":"2024-09-17T03:54:07.494886+0000","last_undegraded":"2024-09-17T03:54:07.494886+0000","last_fullsized":"2024-09-17T03:54:07.494886+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:25.690889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501666+0000","last_change":"2024-09-17T03:54:07.499866+0000","last_active":"2024-09-17T03:54:08.501666+0000","last_peered":"2024-09-17T03:54:08.501666+0000","last_clean":"2024-09-17T03:54:08.501666+0000","last_became_active":"2024-09-17T03:54:07.499536+0000","last_became_peered":"2024-09-17T03:54:07.499536+0000","last_unstale":"2024-09-17T03:54:08.501666+0000","last_undegraded":"2024-09-17T03:54:08.501666+0000","last_fullsized":"2024-09-17T03:54:08.501666+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:38.178335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"91'8","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.529990+0000","last_change":"2024-09-17T03:54:06.500148+0000","last_active":"2024-09-17T03:54:07.529990+0000","last_peered":"2024-09-17T03:54:07.529990+0000","last_clean":"2024-09-17T03:54:07.529990+0000","last_became_active":"2024-09-17T03:54:06.499831+0000","last_became_peered":"2024-09-17T03:54:06.499831+0000","last_unstale":"2024-09-17T03:54:07.529990+0000","last_undegraded":"2024-09-17T03:54:07.529990+0000","last_fullsized":"2024-09-17T03:54:07.529990+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:11.794891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512730+0000","last_change":"2024-09-17T03:54:07.506127+0000","last_active":"2024-09-17T03:54:07.512730+0000","last_peered":"2024-09-17T03:54:07.512730+0000","last_clean":"2024-09-17T03:54:07.512730+0000","last_became_active":"2024-09-17T03:54:07.505432+0000","last_became_peered":"2024-09-17T03:54:07.505432+0000","last_unstale":"2024-09-17T03:54:07.512730+0000","last_undegraded":"2024-09-17T03:54:07.512730+0000","last_fullsized":"2024-09-17T03:54:07.512730+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:34.280281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.518852+0000","last_change":"2024-09-17T03:54:06.503203+0000","last_active":"2024-09-17T03:54:07.518852+0000","last_peered":"2024-09-17T03:54:07.518852+0000","last_clean":"2024-09-17T03:54:07.518852+0000","last_became_active":"2024-09-17T03:54:06.502793+0000","last_became_peered":"2024-09-17T03:54:06.502793+0000","last_unstale":"2024-09-17T03:54:07.518852+0000","last_undegraded":"2024-09-17T03:54:07.518852+0000","last_fullsized":"2024-09-17T03:54:07.518852+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:29:01.591052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503489+0000","last_change":"2024-09-17T03:54:07.505611+0000","last_active":"2024-09-17T03:54:08.503489+0000","last_peered":"2024-09-17T03:54:08.503489+0000","last_clean":"2024-09-17T03:54:08.503489+0000","last_became_active":"2024-09-17T03:54:07.505272+0000","last_became_peered":"2024-09-17T03:54:07.505272+0000","last_unstale":"2024-09-17T03:54:08.503489+0000","last_undegraded":"2024-09-17T03:54:08.503489+0000","last_fullsized":"2024-09-17T03:54:08.503489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:44.053641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.518735+0000","last_change":"2024-09-17T03:54:06.502802+0000","last_active":"2024-09-17T03:54:07.518735+0000","last_peered":"2024-09-17T03:54:07.518735+0000","last_clean":"2024-09-17T03:54:07.518735+0000","last_became_active":"2024-09-17T03:54:06.502284+0000","last_became_peered":"2024-09-17T03:54:06.502284+0000","last_unstale":"2024-09-17T03:54:07.518735+0000","last_undegraded":"2024-09-17T03:54:07.518735+0000","last_fullsized":"2024-09-17T03:54:07.518735+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:00.902263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512907+0000","last_change":"2024-09-17T03:54:07.501125+0000","last_active":"2024-09-17T03:54:07.512907+0000","last_peered":"2024-09-17T03:54:07.512907+0000","last_clean":"2024-09-17T03:54:07.512907+0000","last_became_active":"2024-09-17T03:54:07.500627+0000","last_became_peered":"2024-09-17T03:54:07.500627+0000","last_unstale":"2024-09-17T03:54:07.512907+0000","last_undegraded":"2024-09-17T03:54:07.512907+0000","last_fullsized":"2024-09-17T03:54:07.512907+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:46.163515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.511329+0000","last_change":"2024-09-17T03:54:06.502310+0000","last_active":"2024-09-17T03:54:07.511329+0000","last_peered":"2024-09-17T03:54:07.511329+0000","last_clean":"2024-09-17T03:54:07.511329+0000","last_became_active":"2024-09-17T03:54:06.501977+0000","last_became_peered":"2024-09-17T03:54:06.501977+0000","last_unstale":"2024-09-17T03:54:07.511329+0000","last_undegraded":"2024-09-17T03:54:07.511329+0000","last_fullsized":"2024-09-17T03:54:07.511329+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:00:04.752019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502210+0000","last_change":"2024-09-17T03:54:07.491289+0000","last_active":"2024-09-17T03:54:08.502210+0000","last_peered":"2024-09-17T03:54:08.502210+0000","last_clean":"2024-09-17T03:54:08.502210+0000","last_became_active":"2024-09-17T03:54:07.490955+0000","last_became_peered":"2024-09-17T03:54:07.490955+0000","last_unstale":"2024-09-17T03:54:08.502210+0000","last_undegraded":"2024-09-17T03:54:08.502210+0000","last_fullsized":"2024-09-17T03:54:08.502210+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:24.805323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"91'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.521658+0000","last_change":"2024-09-17T03:54:06.493771+0000","last_active":"2024-09-17T03:54:07.521658+0000","last_peered":"2024-09-17T03:54:07.521658+0000","last_clean":"2024-09-17T03:54:07.521658+0000","last_became_active":"2024-09-17T03:54:06.493049+0000","last_became_peered":"2024-09-17T03:54:06.493049+0000","last_unstale":"2024-09-17T03:54:07.521658+0000","last_undegraded":"2024-09-17T03:54:07.521658+0000","last_fullsized":"2024-09-17T03:54:07.521658+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:55.728832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502462+0000","last_change":"2024-09-17T03:54:07.511066+0000","last_active":"2024-09-17T03:54:08.502462+0000","last_peered":"2024-09-17T03:54:08.502462+0000","last_clean":"2024-09-17T03:54:08.502462+0000","last_became_active":"2024-09-17T03:54:07.509580+0000","last_became_peered":"2024-09-17T03:54:07.509580+0000","last_unstale":"2024-09-17T03:54:08.502462+0000","last_undegraded":"2024-09-17T03:54:08.502462+0000","last_fullsized":"2024-09-17T03:54:08.502462+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:55.069646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503631+0000","last_change":"2024-09-17T03:54:06.507942+0000","last_active":"2024-09-17T03:54:08.503631+0000","last_peered":"2024-09-17T03:54:08.503631+0000","last_clean":"2024-09-17T03:54:08.503631+0000","last_became_active":"2024-09-17T03:54:06.507546+0000","last_became_peered":"2024-09-17T03:54:06.507546+0000","last_unstale":"2024-09-17T03:54:08.503631+0000","last_undegraded":"2024-09-17T03:54:08.503631+0000","last_fullsized":"2024-09-17T03:54:08.503631+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:39.406371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501590+0000","last_change":"2024-09-17T03:54:07.493181+0000","last_active":"2024-09-17T03:54:08.501590+0000","last_peered":"2024-09-17T03:54:08.501590+0000","last_clean":"2024-09-17T03:54:08.501590+0000","last_became_active":"2024-09-17T03:54:07.492903+0000","last_became_peered":"2024-09-17T03:54:07.492903+0000","last_unstale":"2024-09-17T03:54:08.501590+0000","last_undegraded":"2024-09-17T03:54:08.501590+0000","last_fullsized":"2024-09-17T03:54:08.501590+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:58:47.317249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503313+0000","last_change":"2024-09-17T03:54:06.501426+0000","last_active":"2024-09-17T03:54:08.503313+0000","last_peered":"2024-09-17T03:54:08.503313+0000","last_clean":"2024-09-17T03:54:08.503313+0000","last_became_active":"2024-09-17T03:54:06.500781+0000","last_became_peered":"2024-09-17T03:54:06.500781+0000","last_unstale":"2024-09-17T03:54:08.503313+0000","last_undegraded":"2024-09-17T03:54:08.503313+0000","last_fullsized":"2024-09-17T03:54:08.503313+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:24.871710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.516145+0000","last_change":"2024-09-17T03:54:07.510555+0000","last_active":"2024-09-17T03:54:07.516145+0000","last_peered":"2024-09-17T03:54:07.516145+0000","last_clean":"2024-09-17T03:54:07.516145+0000","last_became_active":"2024-09-17T03:54:07.510199+0000","last_became_peered":"2024-09-17T03:54:07.510199+0000","last_unstale":"2024-09-17T03:54:07.516145+0000","last_undegraded":"2024-09-17T03:54:07.516145+0000","last_fullsized":"2024-09-17T03:54:07.516145+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:05.071652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.510622+0000","last_change":"2024-09-17T03:54:06.499842+0000","last_active":"2024-09-17T03:54:07.510622+0000","last_peered":"2024-09-17T03:54:07.510622+0000","last_clean":"2024-09-17T03:54:07.510622+0000","last_became_active":"2024-09-17T03:54:06.499557+0000","last_became_peered":"2024-09-17T03:54:06.499557+0000","last_unstale":"2024-09-17T03:54:07.510622+0000","last_undegraded":"2024-09-17T03:54:07.510622+0000","last_fullsized":"2024-09-17T03:54:07.510622+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:04.069003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.523593+0000","last_change":"2024-09-17T03:54:07.517762+0000","last_active":"2024-09-17T03:54:07.523593+0000","last_peered":"2024-09-17T03:54:07.523593+0000","last_clean":"2024-09-17T03:54:07.523593+0000","last_became_active":"2024-09-17T03:54:07.517410+0000","last_became_peered":"2024-09-17T03:54:07.517410+0000","last_unstale":"2024-09-17T03:54:07.523593+0000","last_undegraded":"2024-09-17T03:54:07.523593+0000","last_fullsized":"2024-09-17T03:54:07.523593+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:59.323939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"91'2","reported_seq":17,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503759+0000","last_change":"2024-09-17T03:54:06.500255+0000","last_active":"2024-09-17T03:54:08.503759+0000","last_peered":"2024-09-17T03:54:08.503759+0000","last_clean":"2024-09-17T03:54:08.503759+0000","last_became_active":"2024-09-17T03:54:06.499799+0000","last_became_peered":"2024-09-17T03:54:06.499799+0000","last_unstale":"2024-09-17T03:54:08.503759+0000","last_undegraded":"2024-09-17T03:54:08.503759+0000","last_fullsized":"2024-09-17T03:54:08.503759+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:19.457347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.526361+0000","last_change":"2024-09-17T03:54:07.520359+0000","last_active":"2024-09-17T03:54:07.526361+0000","last_peered":"2024-09-17T03:54:07.526361+0000","last_clean":"2024-09-17T03:54:07.526361+0000","last_became_active":"2024-09-17T03:54:07.515025+0000","last_became_peered":"2024-09-17T03:54:07.515025+0000","last_unstale":"2024-09-17T03:54:07.526361+0000","last_undegraded":"2024-09-17T03:54:07.526361+0000","last_fullsized":"2024-09-17T03:54:07.526361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:43.793289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.515661+0000","last_change":"2024-09-17T03:54:06.497203+0000","last_active":"2024-09-17T03:54:07.515661+0000","last_peered":"2024-09-17T03:54:07.515661+0000","last_clean":"2024-09-17T03:54:07.515661+0000","last_became_active":"2024-09-17T03:54:06.496848+0000","last_became_peered":"2024-09-17T03:54:06.496848+0000","last_unstale":"2024-09-17T03:54:07.515661+0000","last_undegraded":"2024-09-17T03:54:07.515661+0000","last_fullsized":"2024-09-17T03:54:07.515661+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:34.273653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.524100+0000","last_change":"2024-09-17T03:54:07.511643+0000","last_active":"2024-09-17T03:54:07.524100+0000","last_peered":"2024-09-17T03:54:07.524100+0000","last_clean":"2024-09-17T03:54:07.524100+0000","last_became_active":"2024-09-17T03:54:07.507566+0000","last_became_peered":"2024-09-17T03:54:07.507566+0000","last_unstale":"2024-09-17T03:54:07.524100+0000","last_undegraded":"2024-09-17T03:54:07.524100+0000","last_fullsized":"2024-09-17T03:54:07.524100+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:41.954877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.511502+0000","last_change":"2024-09-17T03:54:06.500628+0000","last_active":"2024-09-17T03:54:07.511502+0000","last_peered":"2024-09-17T03:54:07.511502+0000","last_clean":"2024-09-17T03:54:07.511502+0000","last_became_active":"2024-09-17T03:54:06.500329+0000","last_became_peered":"2024-09-17T03:54:06.500329+0000","last_unstale":"2024-09-17T03:54:07.511502+0000","last_undegraded":"2024-09-17T03:54:07.511502+0000","last_fullsized":"2024-09-17T03:54:07.511502+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:25.765071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.512804+0000","last_change":"2024-09-17T03:54:07.503041+0000","last_active":"2024-09-17T03:54:07.512804+0000","last_peered":"2024-09-17T03:54:07.512804+0000","last_clean":"2024-09-17T03:54:07.512804+0000","last_became_active":"2024-09-17T03:54:07.502697+0000","last_became_peered":"2024-09-17T03:54:07.502697+0000","last_unstale":"2024-09-17T03:54:07.512804+0000","last_undegraded":"2024-09-17T03:54:07.512804+0000","last_fullsized":"2024-09-17T03:54:07.512804+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:43:52.796826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.494167+0000","last_change":"2024-09-17T03:54:06.500838+0000","last_active":"2024-09-17T03:54:07.494167+0000","last_peered":"2024-09-17T03:54:07.494167+0000","last_clean":"2024-09-17T03:54:07.494167+0000","last_became_active":"2024-09-17T03:54:06.500220+0000","last_became_peered":"2024-09-17T03:54:06.500220+0000","last_unstale":"2024-09-17T03:54:07.494167+0000","last_undegraded":"2024-09-17T03:54:07.494167+0000","last_fullsized":"2024-09-17T03:54:07.494167+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:58.815768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.502334+0000","last_change":"2024-09-17T03:54:07.504759+0000","last_active":"2024-09-17T03:54:08.502334+0000","last_peered":"2024-09-17T03:54:08.502334+0000","last_clean":"2024-09-17T03:54:08.502334+0000","last_became_active":"2024-09-17T03:54:07.504356+0000","last_became_peered":"2024-09-17T03:54:07.504356+0000","last_unstale":"2024-09-17T03:54:08.502334+0000","last_undegraded":"2024-09-17T03:54:08.502334+0000","last_fullsized":"2024-09-17T03:54:08.502334+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:49.216622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.510787+0000","last_change":"2024-09-17T03:54:06.494509+0000","last_active":"2024-09-17T03:54:07.510787+0000","last_peered":"2024-09-17T03:54:07.510787+0000","last_clean":"2024-09-17T03:54:07.510787+0000","last_became_active":"2024-09-17T03:54:06.493460+0000","last_became_peered":"2024-09-17T03:54:06.493460+0000","last_unstale":"2024-09-17T03:54:07.510787+0000","last_undegraded":"2024-09-17T03:54:07.510787+0000","last_fullsized":"2024-09-17T03:54:07.510787+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:43.140787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.506577+0000","last_change":"2024-09-17T03:54:07.501214+0000","last_active":"2024-09-17T03:54:07.506577+0000","last_peered":"2024-09-17T03:54:07.506577+0000","last_clean":"2024-09-17T03:54:07.506577+0000","last_became_active":"2024-09-17T03:54:07.500889+0000","last_became_peered":"2024-09-17T03:54:07.500889+0000","last_unstale":"2024-09-17T03:54:07.506577+0000","last_undegraded":"2024-09-17T03:54:07.506577+0000","last_fullsized":"2024-09-17T03:54:07.506577+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:13.978918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":13,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.493753+0000","last_change":"2024-09-17T03:54:06.506480+0000","last_active":"2024-09-17T03:54:07.493753+0000","last_peered":"2024-09-17T03:54:07.493753+0000","last_clean":"2024-09-17T03:54:07.493753+0000","last_became_active":"2024-09-17T03:54:06.506044+0000","last_became_peered":"2024-09-17T03:54:06.506044+0000","last_unstale":"2024-09-17T03:54:07.493753+0000","last_undegraded":"2024-09-17T03:54:07.493753+0000","last_fullsized":"2024-09-17T03:54:07.493753+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:54.055005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501628+0000","last_change":"2024-09-17T03:54:07.504147+0000","last_active":"2024-09-17T03:54:08.501628+0000","last_peered":"2024-09-17T03:54:08.501628+0000","last_clean":"2024-09-17T03:54:08.501628+0000","last_became_active":"2024-09-17T03:54:07.503070+0000","last_became_peered":"2024-09-17T03:54:07.503070+0000","last_unstale":"2024-09-17T03:54:08.501628+0000","last_undegraded":"2024-09-17T03:54:08.501628+0000","last_fullsized":"2024-09-17T03:54:08.501628+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:22.115441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503330+0000","last_change":"2024-09-17T03:54:06.500713+0000","last_active":"2024-09-17T03:54:08.503330+0000","last_peered":"2024-09-17T03:54:08.503330+0000","last_clean":"2024-09-17T03:54:08.503330+0000","last_became_active":"2024-09-17T03:54:06.500081+0000","last_became_peered":"2024-09-17T03:54:06.500081+0000","last_unstale":"2024-09-17T03:54:08.503330+0000","last_undegraded":"2024-09-17T03:54:08.503330+0000","last_fullsized":"2024-09-17T03:54:08.503330+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:06:09.049696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.513064+0000","last_change":"2024-09-17T03:54:07.508155+0000","last_active":"2024-09-17T03:54:07.513064+0000","last_peered":"2024-09-17T03:54:07.513064+0000","last_clean":"2024-09-17T03:54:07.513064+0000","last_became_active":"2024-09-17T03:54:07.503157+0000","last_became_peered":"2024-09-17T03:54:07.503157+0000","last_unstale":"2024-09-17T03:54:07.513064+0000","last_undegraded":"2024-09-17T03:54:07.513064+0000","last_fullsized":"2024-09-17T03:54:07.513064+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:56.731481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":15,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.501663+0000","last_change":"2024-09-17T03:54:06.503324+0000","last_active":"2024-09-17T03:54:08.501663+0000","last_peered":"2024-09-17T03:54:08.501663+0000","last_clean":"2024-09-17T03:54:08.501663+0000","last_became_active":"2024-09-17T03:54:06.503001+0000","last_became_peered":"2024-09-17T03:54:06.503001+0000","last_unstale":"2024-09-17T03:54:08.501663+0000","last_undegraded":"2024-09-17T03:54:08.501663+0000","last_fullsized":"2024-09-17T03:54:08.501663+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:43.088353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-17T03:54:08.503432+0000","last_change":"2024-09-17T03:54:07.498775+0000","last_active":"2024-09-17T03:54:08.503432+0000","last_peered":"2024-09-17T03:54:08.503432+0000","last_clean":"2024-09-17T03:54:08.503432+0000","last_became_active":"2024-09-17T03:54:07.498457+0000","last_became_peered":"2024-09-17T03:54:07.498457+0000","last_unstale":"2024-09-17T03:54:08.503432+0000","last_undegraded":"2024-09-17T03:54:08.503432+0000","last_fullsized":"2024-09-17T03:54:08.503432+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:06.471619+0000","last_clean_scrub_stamp":"2024-09-17T03:54:06.471619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:21.561531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"91'1","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.522643+0000","last_change":"2024-09-17T03:54:06.502684+0000","last_active":"2024-09-17T03:54:07.522643+0000","last_peered":"2024-09-17T03:54:07.522643+0000","last_clean":"2024-09-17T03:54:07.522643+0000","last_became_active":"2024-09-17T03:54:06.502209+0000","last_became_peered":"2024-09-17T03:54:06.502209+0000","last_unstale":"2024-09-17T03:54:07.522643+0000","last_undegraded":"2024-09-17T03:54:07.522643+0000","last_fullsized":"2024-09-17T03:54:07.522643+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:54:05.468970+0000","last_clean_scrub_stamp":"2024-09-17T03:54:05.468970+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:51.265513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":238,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-17T03:54:07.484508+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:54:07.484508+0000","last_peered":"2024-09-17T03:54:07.484508+0000","last_clean":"2024-09-17T03:54:07.484508+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:54:07.484508+0000","last_undegraded":"2024-09-17T03:54:07.484508+0000","last_fullsized":"2024-09-17T03:54:07.484508+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509499,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32876,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338964,"statfs":{"total":96636764160,"available":96603099136,"internally_reserved":0,"allocated":1720320,"data_stored":1063318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509502,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1134592,"data_stored":479208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509501,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1138688,"data_stored":474574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509499,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32308,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339532,"statfs":{"total":96636764160,"available":96603680768,"internally_reserved":0,"allocated":1146880,"data_stored":480772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509501,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1134592,"data_stored":479172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509501,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1118208,"data_stored":472838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542203,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1122304,"data_stored":472908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542204,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1676,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1716224,"data_stored":1063788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:54:12.410 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:54:12.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.676+0000 7feff2c00640 1 -- 172.21.15.129:0/1728625342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1536a0 msgr2=0x7fefec15d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.676+0000 7feff2c00640 1 --2- 172.21.15.129:0/1728625342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1536a0 0x7fefec15d430 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fefec0af630 tx=0x7fefe002a530 comp rx=0 tx=0).stop 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 -- 172.21.15.129:0/1728625342 shutdown_connections 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 --2- 172.21.15.129:0/1728625342 >> v2:172.21.15.181:3300/0 conn(0x7fefec0cd0b0 0x7fefec1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 --2- 172.21.15.129:0/1728625342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec1536a0 0x7fefec15d430 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 -- 172.21.15.129:0/1728625342 >> 172.21.15.129:0/1728625342 conn(0x7fefec06ee50 msgr2=0x7fefec06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:12.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 -- 172.21.15.129:0/1728625342 shutdown_connections 2024-09-17T03:54:12.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.677+0000 7feff2c00640 1 -- 172.21.15.129:0/1728625342 wait complete. 2024-09-17T03:54:12.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.679+0000 7feff2c00640 1 Processor -- start 2024-09-17T03:54:12.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.679+0000 7feff2c00640 1 -- start start 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.680+0000 7feff2c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 0x7fefec150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.680+0000 7feff2c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff2c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefec1f85e0 0x7fefec1f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff2c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefec0d0290 con 0x7fefec15ffa0 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff2c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fefec0d0110 con 0x7fefec1536a0 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff2c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fefec0d0410 con 0x7fefec1f85e0 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 0x7fefec150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57862/0 (socket says 172.21.15.129:57862) 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 0x7fefec150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:49784/0 (socket says 172.21.15.129:49784) 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 -- 172.21.15.129:0/4265971228 learned_addr learned my addr 172.21.15.129:0/4265971228 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff2200640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefec1f85e0 0x7fefec1f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 -- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefec1f85e0 msgr2=0x7fefec1f89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefec1f85e0 0x7fefec1f89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 -- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 msgr2=0x7fefec150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 0x7fefec150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.681+0000 7feff0e00640 1 -- 172.21.15.129:0/4265971228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefec1517d0 con 0x7fefec15ffa0 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7feff0e00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c730 tx=0x7fefd800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8016020 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd800a950 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd800ada0 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefec151ab0 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.682+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefec0b08e0 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.683+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefec151540 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.684+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fefd8019020 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.685+0000 7fefe9600640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 0x7fefcc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.685+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fefd809aae0 con 0x7fefec15ffa0 2024-09-17T03:54:12.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.686+0000 7feff1800640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 0x7fefcc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:12.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.687+0000 7feff1800640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 0x7fefcc07b0d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fefe0034000 tx=0x7fefe0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:12.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.690+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fefd8020020 con 0x7fefec15ffa0 2024-09-17T03:54:12.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.941+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fefec0b4cd0 con 0x7fefec15ffa0 2024-09-17T03:54:12.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.944+0000 7fefe9600640 1 -- 172.21.15.129:0/4265971228 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v32) ==== 190+0+147 (secure 0 0 0) 0x7fefec15c9b0 con 0x7fefec15ffa0 2024-09-17T03:54:12.944 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:54:12.944 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:54:12.944 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:54:12.945 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:54:12.945 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:54:12.945 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.946+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 msgr2=0x7fefcc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 0x7fefcc07b0d0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fefe0034000 tx=0x7fefe0004730 comp rx=0 tx=0).stop 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 msgr2=0x7fefec156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c730 tx=0x7fefd800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 shutdown_connections 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefcc078bc0 0x7fefcc07b0d0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefec1f85e0 0x7fefec1f89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefec15ffa0 0x7fefec156680 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 --2- 172.21.15.129:0/4265971228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefec1536a0 0x7fefec150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:12.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 >> 172.21.15.129:0/4265971228 conn(0x7fefec06ee50 msgr2=0x7fefec0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:12.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 shutdown_connections 2024-09-17T03:54:12.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:12.947+0000 7feff2c00640 1 -- 172.21.15.129:0/4265971228 wait complete. 2024-09-17T03:54:12.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:13.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 -- 172.21.15.129:0/4241842587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca415ca50 msgr2=0x7f7ca40cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:13.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/4241842587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca415ca50 0x7f7ca40cc830 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40b25f0 tx=0x7f7c9402a560 comp rx=0 tx=0).stop 2024-09-17T03:54:13.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 -- 172.21.15.129:0/4241842587 shutdown_connections 2024-09-17T03:54:13.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/4241842587 >> v2:172.21.15.181:3300/0 conn(0x7f7ca406f580 0x7f7ca415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/4241842587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca415ca50 0x7f7ca40cc830 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 -- 172.21.15.129:0/4241842587 >> 172.21.15.129:0/4241842587 conn(0x7f7ca406d410 msgr2=0x7f7ca40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:13.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 -- 172.21.15.129:0/4241842587 shutdown_connections 2024-09-17T03:54:13.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.191+0000 7f7ca3e00640 1 -- 172.21.15.129:0/4241842587 wait complete. 2024-09-17T03:54:13.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.193+0000 7f7ca3e00640 1 Processor -- start 2024-09-17T03:54:13.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.193+0000 7f7ca3e00640 1 -- start start 2024-09-17T03:54:13.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.194+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca415ca50 0x7f7ca413f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ca413fb80 0x7f7ca413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca415ddc0 con 0x7f7ca40cf540 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca415dc40 con 0x7f7ca415ca50 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7ca415df40 con 0x7f7ca413fb80 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57880/0 (socket says 172.21.15.129:57880) 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 -- 172.21.15.129:0/2338616728 learned_addr learned my addr 172.21.15.129:0/2338616728 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7c9a000640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca415ca50 0x7f7ca413f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca3400640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ca413fb80 0x7f7ca413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 -- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ca413fb80 msgr2=0x7f7ca413ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:13.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ca413fb80 0x7f7ca413ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 -- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca415ca50 msgr2=0x7f7ca413f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:13.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca415ca50 0x7f7ca413f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.195+0000 7f7ca2a00640 1 -- 172.21.15.129:0/2338616728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca413cd60 con 0x7f7ca40cf540 2024-09-17T03:54:13.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7ca2a00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7c94009850 tx=0x7f7c94003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9403e030 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c94003e00 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c94035040 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca40af2f0 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.196+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca413eb60 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.197+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca415e3f0 con 0x7f7ca40cf540 2024-09-17T03:54:13.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.198+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7c94041030 con 0x7f7ca40cf540 2024-09-17T03:54:13.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.199+0000 7f7c9b400640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 0x7f7c8007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:13.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.199+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f7c940bb420 con 0x7f7ca40cf540 2024-09-17T03:54:13.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.200+0000 7f7c9a000640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 0x7f7c8007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:13.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.200+0000 7f7c9a000640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 0x7f7c8007b0d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4156f00 tx=0x7f7c8c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:13.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.204+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c94083210 con 0x7f7ca40cf540 2024-09-17T03:54:13.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.422+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ca406fb50 con 0x7f7ca40cf540 2024-09-17T03:54:13.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.423+0000 7f7c9b400640 1 -- 172.21.15.129:0/2338616728 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+5234 (secure 0 0 0) 0x7f7c94087c40 con 0x7f7ca40cf540 2024-09-17T03:54:13.423 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:13.423 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":63,"btime":"2024-09-17T03:54:10:507817+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5723,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3749438691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3749438691},{"type":"v1","addr":"172.21.15.181:6836","nonce":3749438691}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":5726,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2740088493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2740088493},{"type":"v1","addr":"172.21.15.181:6837","nonce":2740088493}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58},{"gid":6358,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/118663331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":118663331},{"type":"v1","addr":"172.21.15.129:6835","nonce":118663331}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":58}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:09.990095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"a","rank":0,"incarnation":59,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1986086177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1986086177},{"type":"v1","addr":"172.21.15.129:6834","nonce":1986086177}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6546,"qdb_cluster":[6546]},"id":7}]} 2024-09-17T03:54:13.423 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 63 2024-09-17T03:54:13.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 msgr2=0x7f7c8007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:13.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 0x7f7c8007b0d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4156f00 tx=0x7f7c8c009040 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 msgr2=0x7f7ca40ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7c94009850 tx=0x7f7c94003910 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 shutdown_connections 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c80078bc0 0x7f7c8007b0d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ca413fb80 0x7f7ca413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ca415ca50 0x7f7ca413f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 --2- 172.21.15.129:0/2338616728 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ca40cf540 0x7f7ca40ae710 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 >> 172.21.15.129:0/2338616728 conn(0x7f7ca406d410 msgr2=0x7f7ca415f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 shutdown_connections 2024-09-17T03:54:13.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:13.425+0000 7f7ca3e00640 1 -- 172.21.15.129:0/2338616728 wait complete. 2024-09-17T03:54:13.460 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T03:54:13.460+0000 7fbf1f800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:54:13.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 63, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:54:06.985254+0000', 'modified': '2024-09-17T03:54:09.990095+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6546}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6546': {'gid': 6546, 'name': 'a', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1986086177', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 1986086177}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1986086177}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6546, 'qdb_cluster': [6546]} 2024-09-17T03:54:13.852 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:54:13.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T03:54:14.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 -- 172.21.15.129:0/3639138864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 msgr2=0x7f3d6015c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 --2- 172.21.15.129:0/3639138864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6015c780 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3d50005fd0 tx=0x7f3d5002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 -- 172.21.15.129:0/3639138864 shutdown_connections 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 --2- 172.21.15.129:0/3639138864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070440 0x7f3d600b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 --2- 172.21.15.129:0/3639138864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6015c780 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 -- 172.21.15.129:0/3639138864 >> 172.21.15.129:0/3639138864 conn(0x7f3d600b1cb0 msgr2=0x7f3d600b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 -- 172.21.15.129:0/3639138864 shutdown_connections 2024-09-17T03:54:14.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.075+0000 7f3d67000640 1 -- 172.21.15.129:0/3639138864 wait complete. 2024-09-17T03:54:14.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.077+0000 7f3d67000640 1 Processor -- start 2024-09-17T03:54:14.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.078+0000 7f3d67000640 1 -- start start 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070030 0x7f3d601e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d60070440 0x7f3d601f3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600b3b00 con 0x7f3d60073170 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d600b3980 con 0x7f3d60070440 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d67000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d600b3c80 con 0x7f3d60070030 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37138/0 (socket says 172.21.15.129:37138) 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.079+0000 7f3d65000640 1 -- 172.21.15.129:0/1718727001 learned_addr learned my addr 172.21.15.129:0/1718727001 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d57400640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d60070440 0x7f3d601f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070030 0x7f3d601e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 -- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070030 msgr2=0x7f3d601e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070030 0x7f3d601e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 -- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d60070440 msgr2=0x7f3d601f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d60070440 0x7f3d601f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 -- 172.21.15.129:0/1718727001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d601e5a20 con 0x7f3d60073170 2024-09-17T03:54:14.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d65000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800e9c0 tx=0x7f3d5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d5cc00640 1 -- 172.21.15.129:0/1718727001 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58004d20 con 0x7f3d60073170 2024-09-17T03:54:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d5cc00640 1 -- 172.21.15.129:0/1718727001 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d5800cb20 con 0x7f3d60073170 2024-09-17T03:54:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d5cc00640 1 -- 172.21.15.129:0/1718727001 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58005020 con 0x7f3d60073170 2024-09-17T03:54:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d601e5d00 con 0x7f3d60073170 2024-09-17T03:54:14.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.080+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d601e62c0 con 0x7f3d60073170 2024-09-17T03:54:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.084+0000 7f3d5cc00640 1 -- 172.21.15.129:0/1718727001 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d580051c0 con 0x7f3d60073170 2024-09-17T03:54:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.085+0000 7f3d5cc00640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 0x7f3d3c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.085+0000 7f3d5cc00640 1 -- 172.21.15.129:0/1718727001 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3d5809b5b0 con 0x7f3d60073170 2024-09-17T03:54:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.085+0000 7f3d6e074740 1 Processor -- start 2024-09-17T03:54:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.085+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 0x7f3d3c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.085+0000 7f3d6e074740 1 -- start start 2024-09-17T03:54:14.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.086+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 0x7f3d3c07b0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3d50033000 tx=0x7f3d50004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548a86970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548a86eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c5489e0430 con 0x55c548695590 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c5489d05d0 con 0x55c5486b6270 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55c5489bfa00 con 0x55c54868d3b0 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37140/0 (socket says 172.21.15.129:37140) 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d5fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548a86970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d65000640 1 -- 172.21.15.129:0/2229178368 learned_addr learned my addr 172.21.15.129:0/2229178368 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.087+0000 7f3d57400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548a86eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 -- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 msgr2=0x55c548a86eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548a86eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 -- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 msgr2=0x55c548a86970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548a86970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 -- 172.21.15.129:0/2229178368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c548988520 con 0x55c548695590 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d57400640 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548a86eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548a86970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d65000640 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3d58063310 tx=0x7f3d58062e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29a56999af05943 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d54c00640 1 -- 172.21.15.129:0/2229178368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5801f040 con 0x55c548695590 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d54c00640 1 -- 172.21.15.129:0/2229178368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d58067e60 con 0x55c548695590 2024-09-17T03:54:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.088+0000 7f3d54c00640 1 -- 172.21.15.129:0/2229178368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d580627e0 con 0x55c548695590 2024-09-17T03:54:14.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 -- 172.21.15.129:0/2229178368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 msgr2=0x55c548a56f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f3d58063310 tx=0x7f3d58062e90 comp rx=0 tx=0).stop 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 -- 172.21.15.129:0/2229178368 shutdown_connections 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548a56f60 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548a86eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2229178368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548a86970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 -- 172.21.15.129:0/2229178368 >> 172.21.15.129:0/2229178368 conn(0x55c5486aa3c0 msgr2=0x55c548a5c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 -- 172.21.15.129:0/2229178368 shutdown_connections 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.089+0000 7f3d6e074740 1 -- 172.21.15.129:0/2229178368 wait complete. 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.090+0000 7f3d6e074740 1 -- start start 2024-09-17T03:54:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.091+0000 7f3d6e074740 1 Processor -- start 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c54868d3b0 0x55c548a7c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c5486b6270 0x55c548ab26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c548966330 con 0x55c548695590 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c5489c3810 con 0x55c54868d3b0 2024-09-17T03:54:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55c548730620 con 0x55c5486b6270 2024-09-17T03:54:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d6e074740 10 client.0 _ll_register_callbacks cb 0x55c54896b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:54:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d57400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d57400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37142/0 (socket says 172.21.15.129:37142) 2024-09-17T03:54:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.092+0000 7f3d57400640 1 -- 172.21.15.129:0/3112147085 learned_addr learned my addr 172.21.15.129:0/3112147085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c54868d3b0 0x55c548a7c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d65000640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c5486b6270 0x55c548ab26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 -- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c5486b6270 msgr2=0x55c548ab26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c5486b6270 0x55c548ab26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 -- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c54868d3b0 msgr2=0x55c548a7c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c54868d3b0 0x55c548a7c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 -- 172.21.15.129:0/3112147085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c548721850 con 0x55c548695590 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d57400640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3d48001d10 tx=0x7f3d4800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=404f201e144feb3c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d4d600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d480052d0 con 0x55c548695590 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3d48019b30 con 0x55c548695590 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.093+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d58067e60 con 0x55c548695590 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f3d48005470 con 0x55c548695590 2024-09-17T03:54:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 10 client.6660 fetch_fsmap learned FSMap version 63 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x55c5487986b0 con 0x55c548695590 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d58067e60 con 0x55c548695590 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 63) ==== 3502+0+0 (secure 0 0 0) 0x7f3d48007020 con 0x55c548695590 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 10 client.6660 fetch_fsmap finished waiting for FSMap version 63 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 10 client.6660 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 20 client.6660 populate_metadata read hostname 'smithi129' 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548674e40 0x55c548ab4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 4 client.6660 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 --> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5489d2520 con 0x55c548674e40 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.094+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548674e40 0x55c548ab4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.095+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548674e40 0x55c548ab4740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a3f06467674fbc server_cookie=5fbca0b732c5c458 in_seq=0 out_seq=0 2024-09-17T03:54:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.095+0000 7f3d4d600640 10 client.6660 ms_handle_connect on v2:172.21.15.129:6833/1986086177 2024-09-17T03:54:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d4d600640 1 -- 172.21.15.129:0/3112147085 <== mds.0 v2:172.21.15.129:6833/1986086177 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55c5489d2520 con 0x55c548674e40 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d4d600640 10 client.6660 handle_command_reply: tid=0 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 client.6660 shutdown 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 msgr2=0x55c548ab2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3d48001d10 tx=0x7f3d4800e530 comp rx=0 tx=0).stop 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 shutdown_connections 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c5486b6270 0x55c548ab26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548674e40 0x55c548ab4740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c548695590 0x55c548ab2180 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 --2- 172.21.15.129:0/3112147085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c54868d3b0 0x55c548a7c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 >> 172.21.15.129:0/3112147085 conn(0x55c5486aa3c0 msgr2=0x55c548aa1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d4d600640 0 client.6660 ms_handle_reset on v2:172.21.15.129:6833/1986086177 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 shutdown_connections 2024-09-17T03:54:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 1 -- 172.21.15.129:0/3112147085 wait complete. 2024-09-17T03:54:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.096+0000 7f3d6e074740 20 client.6660 trim_cache size 0 max 16384 2024-09-17T03:54:14.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.115+0000 7f3d6e074740 1 Processor -- start 2024-09-17T03:54:14.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.115+0000 7f3d6e074740 1 -- start start 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548aa4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 0x55c548aa5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601fef90 con 0x55c5486b6270 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601fe410 con 0x55c548695590 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d601fe940 con 0x55c54868d3b0 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37146/0 (socket says 172.21.15.129:37146) 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 -- 172.21.15.129:0/2842635354 learned_addr learned my addr 172.21.15.129:0/2842635354 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548aa4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 -- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 msgr2=0x55c548aa4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d57400640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 0x55c548aa5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548aa4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 -- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 msgr2=0x55c548aa5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 0x55c548aa5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d65000640 1 -- 172.21.15.129:0/2842635354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c548a5d650 con 0x55c5486b6270 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548aa4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.117+0000 7f3d57400640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 0x55c548aa5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d65000640 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f3d58063310 tx=0x7f3d580999b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f221d7df4f5689a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d4fe00640 1 -- 172.21.15.129:0/2842635354 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5801f040 con 0x55c5486b6270 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d4fe00640 1 -- 172.21.15.129:0/2842635354 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d58067e60 con 0x55c5486b6270 2024-09-17T03:54:14.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d4fe00640 1 -- 172.21.15.129:0/2842635354 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58062840 con 0x55c5486b6270 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d6e074740 1 -- 172.21.15.129:0/2842635354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 msgr2=0x55c548aba770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.118+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f3d58063310 tx=0x7f3d580999b0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 -- 172.21.15.129:0/2842635354 shutdown_connections 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c5486b6270 0x55c548aba770 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c548695590 0x55c548aa5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 --2- 172.21.15.129:0/2842635354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c54868d3b0 0x55c548aa4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 -- 172.21.15.129:0/2842635354 >> 172.21.15.129:0/2842635354 conn(0x55c5486aa3c0 msgr2=0x55c548ab2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 -- 172.21.15.129:0/2842635354 shutdown_connections 2024-09-17T03:54:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.119+0000 7f3d6e074740 1 -- 172.21.15.129:0/2842635354 wait complete. 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.120+0000 7f3d6e074740 1 -- start start 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.120+0000 7f3d6e074740 1 Processor -- start 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c548695590 0x55c548b25950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548b25e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c548a7d410 con 0x55c54868d3b0 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c548a80eb0 con 0x55c5486b6270 2024-09-17T03:54:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55c548ab10b0 con 0x55c548695590 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d6e074740 10 client.0 _ll_register_callbacks cb 0x7f3d480019b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.121+0000 7f3d5fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37158/0 (socket says 172.21.15.129:37158) 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 -- 172.21.15.129:0/591301836 learned_addr learned my addr 172.21.15.129:0/591301836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d57400640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c548695590 0x55c548b25950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d65000640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548b25e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 -- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c548695590 msgr2=0x55c548b25950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c548695590 0x55c548b25950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 -- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 msgr2=0x55c548b25e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548b25e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 -- 172.21.15.129:0/591301836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c548a57850 con 0x55c54868d3b0 2024-09-17T03:54:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d65000640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548b25e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d5fe00640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d50009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0bd885bf393f59c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d4d600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.122+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50056060 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f3d50004e70 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d58067e60 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) ==== 24+0+0 (secure 0 0 0) 0x7f3d5005c050 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 10 client.6666 fetch_fsmap learned FSMap version 63 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) -- 0x55c548abb310 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d58067e60 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 63) ==== 3502+0+0 (secure 0 0 0) 0x7f3d5005f020 con 0x55c54868d3b0 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 10 client.6666 fetch_fsmap finished waiting for FSMap version 63 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 10 client.6666 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:54:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 20 client.6666 populate_metadata read hostname 'smithi129' 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548ab2a50 0x55c548b29590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 4 client.6666 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 --> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] -- command(tid 0: {"prefix": "status"}) -- 0x55c5489cdfd0 con 0x55c548ab2a50 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.123+0000 7f3d57400640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548ab2a50 0x55c548b29590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d4d600640 10 client.6666 ms_handle_connect on v2:172.21.15.129:6833/1986086177 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d57400640 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548ab2a50 0x55c548b29590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e0119a65c72501e server_cookie=9fd238b9c5ce033f in_seq=0 out_seq=0 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d4d600640 1 -- 172.21.15.129:0/591301836 <== mds.0 v2:172.21.15.129:6833/1986086177 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55c5489cdfd0 con 0x55c548ab2a50 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d4d600640 10 client.6666 handle_command_reply: tid=0 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d6e074740 1 client.6666 shutdown 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 msgr2=0x55c548ac4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.124+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3d50006010 tx=0x7f3d50009260 comp rx=0 tx=0).stop 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 shutdown_connections 2024-09-17T03:54:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c5486b6270 0x55c548b25e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c548695590 0x55c548b25950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:6833/1986086177,v1:172.21.15.129:6834/1986086177] conn(0x55c548ab2a50 0x55c548b29590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 --2- 172.21.15.129:0/591301836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c54868d3b0 0x55c548ac4600 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d4d600640 0 client.6666 ms_handle_reset on v2:172.21.15.129:6833/1986086177 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 >> 172.21.15.129:0/591301836 conn(0x55c5486aa3c0 msgr2=0x55c548aba770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 shutdown_connections 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 1 -- 172.21.15.129:0/591301836 wait complete. 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d6e074740 20 client.6666 trim_cache size 0 max 16384 2024-09-17T03:54:14.129 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 6546, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.6449171890000001, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 63, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 92, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 92, 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.687908322 2024-09-17T03:54:14.130 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 msgr2=0x7f3d3c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 0x7f3d3c07b0a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3d50033000 tx=0x7f3d50004730 comp rx=0 tx=0).stop 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 msgr2=0x7f3d6009d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800e9c0 tx=0x7f3d5800ee80 comp rx=0 tx=0).stop 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 shutdown_connections 2024-09-17T03:54:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.125+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d3c078b90 0x7f3d3c07b0a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d60073170 0x7f3d6009d680 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d60070440 0x7f3d601f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 --2- 172.21.15.129:0/1718727001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d60070030 0x7f3d601e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 >> 172.21.15.129:0/1718727001 conn(0x7f3d600b1cb0 msgr2=0x7f3d600aec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 shutdown_connections 2024-09-17T03:54:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.126+0000 7f3d67000640 1 -- 172.21.15.129:0/1718727001 wait complete. 2024-09-17T03:54:14.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:54:14.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 -- 172.21.15.129:0/600694085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070440 msgr2=0x7f45b00b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 --2- 172.21.15.129:0/600694085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070440 0x7f45b00b3090 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f45a8005fd0 tx=0x7f45a802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 -- 172.21.15.129:0/600694085 shutdown_connections 2024-09-17T03:54:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 --2- 172.21.15.129:0/600694085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070440 0x7f45b00b3090 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 --2- 172.21.15.129:0/600694085 >> v2:172.21.15.181:3300/0 conn(0x7f45b0073170 0x7f45b015c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 -- 172.21.15.129:0/600694085 >> 172.21.15.129:0/600694085 conn(0x7f45b00b1cb0 msgr2=0x7f45b00b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 -- 172.21.15.129:0/600694085 shutdown_connections 2024-09-17T03:54:14.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.369+0000 7f45b4a00640 1 -- 172.21.15.129:0/600694085 wait complete. 2024-09-17T03:54:14.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.371+0000 7f45b4a00640 1 Processor -- start 2024-09-17T03:54:14.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.372+0000 7f45b4a00640 1 -- start start 2024-09-17T03:54:14.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070030 0x7f45b01e7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b0073170 0x7f45b00cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f45b00b3b00 con 0x7f45b0070440 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f45b00b3980 con 0x7f45b0073170 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45b4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f45b00b3c80 con 0x7f45b0070030 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37160/0 (socket says 172.21.15.129:37160) 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45ad400640 1 -- 172.21.15.129:0/2321478049 learned_addr learned my addr 172.21.15.129:0/2321478049 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45ade00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070030 0x7f45b01e7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.373+0000 7f45ae800640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b0073170 0x7f45b00cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 -- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070030 msgr2=0x7f45b01e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070030 0x7f45b01e7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 -- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b0073170 msgr2=0x7f45b00cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b0073170 0x7f45b00cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 -- 172.21.15.129:0/2321478049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b015ee90 con 0x7f45b0070440 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45ad400640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f459c00c430 tx=0x7f459c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c011790 con 0x7f45b0070440 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f459c017070 con 0x7f45b0070440 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b00d02a0 con 0x7f45b0070440 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c011cc0 con 0x7f45b0070440 2024-09-17T03:54:14.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.374+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b00d0860 con 0x7f45b0070440 2024-09-17T03:54:14.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.375+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b00b4130 con 0x7f45b0070440 2024-09-17T03:54:14.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.377+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f459c01c460 con 0x7f45b0070440 2024-09-17T03:54:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.378+0000 7f45a6000640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 0x7f459007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.378+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f459c09a740 con 0x7f45b0070440 2024-09-17T03:54:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.378+0000 7f45ade00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 0x7f459007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.379+0000 7f45ade00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 0x7f459007b0a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009b30 tx=0x7f45a0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:14.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.382+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f459c021020 con 0x7f45b0070440 2024-09-17T03:54:14.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.606+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f45b00c3860 con 0x7f45b0070440 2024-09-17T03:54:14.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.607+0000 7f45a6000640 1 -- 172.21.15.129:0/2321478049 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v63) ==== 53+0+73 (secure 0 0 0) 0x7f459c0625e0 con 0x7f45b0070440 2024-09-17T03:54:14.607 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 msgr2=0x7f459007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 0x7f459007b0a0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009b30 tx=0x7f45a0005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 msgr2=0x7f45b015e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f459c00c430 tx=0x7f459c00c8f0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 shutdown_connections 2024-09-17T03:54:14.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4590078b90 0x7f459007b0a0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45b0073170 0x7f45b00cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f45b0070440 0x7f45b015e950 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 --2- 172.21.15.129:0/2321478049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f45b0070030 0x7f45b01e7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 >> 172.21.15.129:0/2321478049 conn(0x7f45b00b1cb0 msgr2=0x7f45b01ea780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 shutdown_connections 2024-09-17T03:54:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14.609+0000 7f45b4a00640 1 -- 172.21.15.129:0/2321478049 wait complete. 2024-09-17T03:54:14.633 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:54:14.634 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:54:14.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:14.635 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:54:14.670 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:54:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:54:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:54:14.671 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42437sec preferred_lft 42437sec 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.672 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:54:14.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:14.673 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:54:14.764 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:54:14.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:14.767 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:54:14.803 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:54:14.803 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:54:14.803 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:54:14.803 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:54:14.803 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:54:14.804 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:54:14.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:54:14.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:54:15.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:15.001 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:54:15.036 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:54:15.036 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:54:15.036 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:54:15.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:15.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:54:15.133 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:54:15.133 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:54:15.168 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:54:15.168 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:54:15.169 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:54:15.244 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:54:15.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:15.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:54:15.438 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:54:15.439 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:54:15.439 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:54:15.459 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:54:15.459 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:54:15.459 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:54:15.459 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:54:15.459 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:54:15.492 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:54:15.492 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:54:15.492 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:54:15.493 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:54:15.493 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:54:15.493 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:54:15.493 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:54:15.493 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:54:15.494 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:54:15.494 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:54:15.494 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:54:15.494 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:54:15.494 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:54:15.497 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:54:15.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:15.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:54:15.555 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:54:15.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:15 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:54:15.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:54:15.650 INFO:teuthology.orchestra.run.smithi129.stdout:client6675 2024-09-17T03:54:15.650 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:15.650 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:15.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:15.730 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:15.824 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:15.825 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:15.826 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.827 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:15.828 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.829 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.830 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:15.831 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.832 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:15.833 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.834 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.835 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:15.836 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:15.836 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:15.836 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:15.836 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:15.837 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 6 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.838 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.839 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:15.840 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:15.841 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:15.842 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:15.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:15.844 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.845 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.846 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:15.847 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.848 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:15.849 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:15.850 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:15.850 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:15.850 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:15.850 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 6 2024-09-17T03:54:15.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:15.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:15.960 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:15.960 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:15.960 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:15.960 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:15.961 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:15.962 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:15.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.964 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:15.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.966 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:15.967 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.968 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:15.969 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:15.970 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:15.971 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 6 2024-09-17T03:54:15.972 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:15.973 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:15.974 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:15.975 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:15.976 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:15.977 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.978 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:15.979 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.980 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:15.981 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:15.982 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:15.983 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 6 2024-09-17T03:54:15.984 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:15.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 -- 172.21.15.129:0/1390192828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 msgr2=0x7f417806fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 --2- 172.21.15.129:0/1390192828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f417806fe10 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4168005fd0 tx=0x7f416802a540 comp rx=0 tx=0).stop 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 -- 172.21.15.129:0/1390192828 shutdown_connections 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 --2- 172.21.15.129:0/1390192828 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178070350 0x7f4178072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 --2- 172.21.15.129:0/1390192828 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f417806fe10 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 -- 172.21.15.129:0/1390192828 >> 172.21.15.129:0/1390192828 conn(0x7f41780b2060 msgr2=0x7f41780b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 -- 172.21.15.129:0/1390192828 shutdown_connections 2024-09-17T03:54:16.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.212+0000 7f417d800640 1 -- 172.21.15.129:0/1390192828 wait complete. 2024-09-17T03:54:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.215+0000 7f417d800640 1 Processor -- start 2024-09-17T03:54:16.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.215+0000 7f417d800640 1 -- start start 2024-09-17T03:54:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178072bc0 0x7f41781e93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f41781e9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f417815f9f0 con 0x7f4178070350 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f41780b5590 con 0x7f417815b770 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.216+0000 7f417d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f417815fb70 con 0x7f4178072bc0 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37174/0 (socket says 172.21.15.129:37174) 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 -- 172.21.15.129:0/2535174202 learned_addr learned my addr 172.21.15.129:0/2535174202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:16.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176000640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178072bc0 0x7f41781e93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4177400640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f41781e9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 -- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178072bc0 msgr2=0x7f41781e93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178072bc0 0x7f41781e93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 -- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 msgr2=0x7f41781e9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f41781e9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 -- 172.21.15.129:0/2535174202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4178152800 con 0x7f4178070350 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.217+0000 7f4176a00640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4168000c00 tx=0x7f41680038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.218+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416803e030 con 0x7f4178070350 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.218+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4168003db0 con 0x7f4178070350 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.218+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4178151ed0 con 0x7f4178070350 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.218+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168035040 con 0x7f4178070350 2024-09-17T03:54:16.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.218+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4178152a70 con 0x7f4178070350 2024-09-17T03:54:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.219+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41781600a0 con 0x7f4178070350 2024-09-17T03:54:16.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.220+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4168041030 con 0x7f4178070350 2024-09-17T03:54:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.222+0000 7f416ea00640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 0x7f415407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.222+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f41680bb4b0 con 0x7f4178070350 2024-09-17T03:54:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.222+0000 7f4176000640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 0x7f415407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.223+0000 7f4176000640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 0x7f415407b0a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f417815afd0 tx=0x7f4160005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.226+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4168083200 con 0x7f4178070350 2024-09-17T03:54:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.409+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4178152de0 con 0x7f4178070350 2024-09-17T03:54:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.409+0000 7f416ea00640 1 -- 172.21.15.129:0/2535174202 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f4168087c30 con 0x7f4178070350 2024-09-17T03:54:16.410 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 msgr2=0x7f415407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 0x7f415407b0a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f417815afd0 tx=0x7f4160005f30 comp rx=0 tx=0).stop 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 msgr2=0x7f41781516f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f4168000c00 tx=0x7f41680038a0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 shutdown_connections 2024-09-17T03:54:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4154078b90 0x7f415407b0a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f417815b770 0x7f41781e9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4178072bc0 0x7f41781e93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 --2- 172.21.15.129:0/2535174202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4178070350 0x7f41781516f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 >> 172.21.15.129:0/2535174202 conn(0x7f41780b2060 msgr2=0x7f41780aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 shutdown_connections 2024-09-17T03:54:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.412+0000 7f417d800640 1 -- 172.21.15.129:0/2535174202 wait complete. 2024-09-17T03:54:16.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:16.439 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:16.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:54:16.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.535 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.536 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:16.537 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:16.538 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.539 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.540 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:16.541 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:16.542 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:16.542 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:16.542 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:16.543 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:16.544 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:16.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:16.546 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:16.547 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 6 2024-09-17T03:54:16.548 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:16.548 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.549 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:16.550 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:16.551 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.552 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:16.553 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.554 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:16.555 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:16.556 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:16.557 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:16.558 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:16.560 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:16.560 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.560 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:16.560 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:16.560 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:16.561 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:16.562 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:16.564 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:16.565 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:16.565 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:16.565 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 6 2024-09-17T03:54:16.565 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:16.565 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops_per_pg = 1562500.0 2024-09-17T03:54:16.568 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:54:16.568 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:16.569 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:54:16.569 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:16.569 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:54:16.569 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:16.569 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:54:16.569 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:54:16.601 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:54:16.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T03:54:16.602 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:54:16.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-17T03:54:16.606 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:54:16.606 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T03:54:16.607 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:54:16.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 -- 172.21.15.129:0/2803691126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 msgr2=0x7f2d6815f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 --2- 172.21.15.129:0/2803691126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d6815f560 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009830 tx=0x7f2d5402a510 comp rx=0 tx=0).stop 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 -- 172.21.15.129:0/2803691126 shutdown_connections 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 --2- 172.21.15.129:0/2803691126 >> v2:172.21.15.181:3300/0 conn(0x7f2d6815faa0 0x7f2d680b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 --2- 172.21.15.129:0/2803691126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d6815f560 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.877+0000 7f2d67e00640 1 -- 172.21.15.129:0/2803691126 >> 172.21.15.129:0/2803691126 conn(0x7f2d6806d560 msgr2=0x7f2d680b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.878+0000 7f2d67e00640 1 -- 172.21.15.129:0/2803691126 shutdown_connections 2024-09-17T03:54:16.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.878+0000 7f2d67e00640 1 -- 172.21.15.129:0/2803691126 wait complete. 2024-09-17T03:54:16.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.880+0000 7f2d67e00640 1 Processor -- start 2024-09-17T03:54:16.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.881+0000 7f2d67e00640 1 -- start start 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 -- 172.21.15.129:0/392066508 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 msgr2=0x7f17f815ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 --2- 172.21.15.129:0/392066508 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f815ed50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f17f80b3b50 tx=0x7f17e802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 -- 172.21.15.129:0/392066508 shutdown_connections 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 --2- 172.21.15.129:0/392066508 >> v2:172.21.15.181:3300/0 conn(0x7f17f80b3180 0x7f17f80cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 --2- 172.21.15.129:0/392066508 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f815ed50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 -- 172.21.15.129:0/392066508 >> 172.21.15.129:0/392066508 conn(0x7f17f80b6710 msgr2=0x7f17f806e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 -- 172.21.15.129:0/392066508 shutdown_connections 2024-09-17T03:54:16.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 -- 172.21.15.129:0/3978751172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00b3180 msgr2=0x7fd0f00cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 --2- 172.21.15.129:0/3978751172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00b3180 0x7fd0f00cfd60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd0f00b3b50 tx=0x7fd0e802a890 comp rx=0 tx=0).stop 2024-09-17T03:54:16.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7f17fca00640 1 -- 172.21.15.129:0/392066508 wait complete. 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 -- 172.21.15.129:0/3978751172 shutdown_connections 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 --2- 172.21.15.129:0/3978751172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00b3180 0x7fd0f00cfd60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 --2- 172.21.15.129:0/3978751172 >> v2:172.21.15.181:3300/0 conn(0x7fd0f006d7e0 0x7fd0f015ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.882+0000 7fd0ef200640 1 -- 172.21.15.129:0/3978751172 >> 172.21.15.129:0/3978751172 conn(0x7fd0f00b6710 msgr2=0x7fd0f006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7fd0ef200640 1 -- 172.21.15.129:0/3978751172 shutdown_connections 2024-09-17T03:54:16.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7fd0ef200640 1 -- 172.21.15.129:0/3978751172 wait complete. 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 0x7f2d681571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d680ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 0x7f2d681539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d680b39d0 con 0x7f2d6806f930 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2d680b3850 con 0x7f2d6815faa0 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2d680b3b50 con 0x7f2d6815f150 2024-09-17T03:54:16.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 -- 172.21.15.129:0/1929449262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 msgr2=0x7f43e006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 --2- 172.21.15.129:0/1929449262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e006d270 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f43cc009830 tx=0x7f43cc02a510 comp rx=0 tx=0).stop 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 -- 172.21.15.129:0/1929449262 shutdown_connections 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 --2- 172.21.15.129:0/1929449262 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00d3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 --2- 172.21.15.129:0/1929449262 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e006d270 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 -- 172.21.15.129:0/1929449262 >> 172.21.15.129:0/1929449262 conn(0x7f43e00d1770 msgr2=0x7f43e00d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:16.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 -- 172.21.15.129:0/1929449262 shutdown_connections 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d66a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 0x7f2d681571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d66000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d680ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d66a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 0x7f2d681571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37190/0 (socket says 172.21.15.129:37190) 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d66a00640 1 -- 172.21.15.129:0/3332342559 learned_addr learned my addr 172.21.15.129:0/3332342559 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f2d67400640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 0x7f2d681539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.883+0000 7f43e6c00640 1 -- 172.21.15.129:0/1929449262 wait complete. 2024-09-17T03:54:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 -- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 msgr2=0x7f2d680ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d680ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 -- 172.21.15.129:0/3332342559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 msgr2=0x7f2d681571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 0x7f2d681571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 -- 172.21.15.129:0/3332342559 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d680c9480 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.884+0000 7f2d67400640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 0x7f2d681539a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000eb50 tx=0x7f2d6000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.885+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60003b70 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.885+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d60003d10 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.885+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60004280 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.885+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d681f57a0 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.885+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d681f5d60 con 0x7f2d6815faa0 2024-09-17T03:54:16.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7f43e6c00640 1 Processor -- start 2024-09-17T03:54:16.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6815f560 con 0x7f2d6815faa0 2024-09-17T03:54:16.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7f43e6c00640 1 -- start start 2024-09-17T03:54:16.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7fd0ef200640 1 Processor -- start 2024-09-17T03:54:16.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7fd0ef200640 1 -- start start 2024-09-17T03:54:16.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.886+0000 7f17fca00640 1 Processor -- start 2024-09-17T03:54:16.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.887+0000 7f17fca00640 1 -- start start 2024-09-17T03:54:16.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.887+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2d60004b40 con 0x7f2d6815faa0 2024-09-17T03:54:16.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00d4090 0x7fd0f01f8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f01f9290 0x7fd0f015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f00b81e0 con 0x7fd0f01f9290 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f00b8060 con 0x7fd0f00b3180 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ef200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f00b8360 con 0x7fd0f00d4090 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00aafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e0154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43e015eb00 0x7f43e01f9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43e00d3c40 con 0x7f43e015eb00 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43e00d3ac0 con 0x7f43e00b7c10 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f43e00d3dc0 con 0x7f43e006d7b0 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ede00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:43770/0 (socket says 172.21.15.129:43770) 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ede00640 1 -- 172.21.15.129:0/1573235227 learned_addr learned my addr 172.21.15.129:0/1573235227 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:16.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ed400640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00d4090 0x7fd0f01f8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f17fca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 0x7f17f81fe320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f17fca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7fd0ee800640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f01f9290 0x7fd0f015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17fca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17fca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17f80b8250 con 0x7f17f806d0d0 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17fca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17f80b80d0 con 0x7f17f80b3180 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17fca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17f80b83d0 con 0x7f17f80d4090 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 -- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00d4090 msgr2=0x7fd0f01f8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00d4090 0x7fd0f01f8d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 -- 172.21.15.129:0/1573235227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f01f9290 msgr2=0x7fd0f015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f01f9290 0x7fd0f015a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 -- 172.21.15.129:0/1573235227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f0161380 con 0x7fd0f00b3180 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e5800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00aafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e4e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e0154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e5800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00aafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:59600/0 (socket says 172.21.15.129:59600) 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.888+0000 7f43e5800640 1 -- 172.21.15.129:0/2159645105 learned_addr learned my addr 172.21.15.129:0/2159645105 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f2d5f400640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 0x7f2d4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e6200640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43e015eb00 0x7f43e01f9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 -- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 msgr2=0x7f43e00aafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00aafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 -- 172.21.15.129:0/2159645105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43e015eb00 msgr2=0x7f43e01f9240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43e015eb00 0x7f43e01f9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 -- 172.21.15.129:0/2159645105 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e00ab510 con 0x7f43e00b7c10 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:43794/0 (socket says 172.21.15.129:43794) 2024-09-17T03:54:16.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17f6a00640 1 -- 172.21.15.129:0/3559524842 learned_addr learned my addr 172.21.15.129:0/3559524842 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:16.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f2d6009bbe0 con 0x7f2d6815faa0 2024-09-17T03:54:16.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17f7e00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f17f7400640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 0x7f17f81fe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7fd0ede00640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000c390 tx=0x7fd0e000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f43e4e00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e0154b60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f43cc031000 tx=0x7f43cc033ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.889+0000 7f2d66a00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 0x7f2d4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 -- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 msgr2=0x7f17f80cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f80cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 -- 172.21.15.129:0/3559524842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 msgr2=0x7f17f81fe320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 0x7f17f81fe320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 -- 172.21.15.129:0/3559524842 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f80cb8b0 con 0x7f17f80b3180 2024-09-17T03:54:16.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f7400640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 0x7f17f81fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f17f6a00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17e400e6a0 tx=0x7f17e400eb60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.890+0000 7f2d66a00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 0x7f2d4807b0a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2d58007fd0 tx=0x7f2d58001450 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0011790 con 0x7fd0f00b3180 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f00c8c20 con 0x7fd0f00b3180 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0e0017070 con 0x7fd0f00b3180 2024-09-17T03:54:16.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0011ce0 con 0x7fd0f00b3180 2024-09-17T03:54:16.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc031b50 con 0x7f43e00b7c10 2024-09-17T03:54:16.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004220 con 0x7f17f80b3180 2024-09-17T03:54:16.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17e40043c0 con 0x7f17f80b3180 2024-09-17T03:54:16.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e400cb10 con 0x7f17f80b3180 2024-09-17T03:54:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f00c91e0 con 0x7fd0f00b3180 2024-09-17T03:54:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43cc031e00 con 0x7f43e00b7c10 2024-09-17T03:54:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f80cbb90 con 0x7f17f80b3180 2024-09-17T03:54:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81fde00 con 0x7f17f80b3180 2024-09-17T03:54:16.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2d60063a40 con 0x7f2d6815faa0 2024-09-17T03:54:16.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e01f5c70 con 0x7f43e00b7c10 2024-09-17T03:54:16.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc033330 con 0x7f43e00b7c10 2024-09-17T03:54:16.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.892+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e00cbc00 con 0x7f43e00b7c10 2024-09-17T03:54:16.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.894+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f00b8810 con 0x7fd0f00b3180 2024-09-17T03:54:16.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.894+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f80b8910 con 0x7f17f80b3180 2024-09-17T03:54:16.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.895+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43bc005290 con 0x7f43e00b7c10 2024-09-17T03:54:16.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.895+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0e001c460 con 0x7fd0f00b3180 2024-09-17T03:54:16.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.895+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f43cc0334d0 con 0x7f43e00b7c10 2024-09-17T03:54:16.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.896+0000 7f43dd600640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 0x7f43c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.896+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f43cc0bbda0 con 0x7f43e00b7c10 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7f43e5800640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 0x7f43c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f17e400ccb0 con 0x7f17f80b3180 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7fd0e6000640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 0x7fd0d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7fd0e009a880 con 0x7fd0f00b3180 2024-09-17T03:54:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7fd0ed400640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 0x7fd0d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.897+0000 7f43e5800640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 0x7f43c007b0a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f43d40016c0 tx=0x7f43d4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.898+0000 7fd0ed400640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 0x7fd0d007b0a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd0f01f30e0 tx=0x7fd0dc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.898+0000 7f17ef400640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 0x7f17d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:16.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.898+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 5709+0+0 (secure 0 0 0) 0x7f17e409b500 con 0x7f17f80b3180 2024-09-17T03:54:16.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.898+0000 7f17f7400640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 0x7f17d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:16.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.899+0000 7f17f7400640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 0x7f17d807b0d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17e8033000 tx=0x7f17e8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:16.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.901+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0e0021020 con 0x7fd0f00b3180 2024-09-17T03:54:16.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.906+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f43cc04a030 con 0x7f43e00b7c10 2024-09-17T03:54:16.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:16.911+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f17e4063930 con 0x7f17f80b3180 2024-09-17T03:54:17.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.174+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7f43bc005580 con 0x7f43e00b7c10 2024-09-17T03:54:17.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.175+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7f17f806da60 con 0x7f17f80b3180 2024-09-17T03:54:17.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.193+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7f2d6815ece0 con 0x7f2d6815faa0 2024-09-17T03:54:17.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.208+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fd0f00b99f0 con 0x7fd0f00b3180 2024-09-17T03:54:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.543+0000 7f43dd600640 1 -- 172.21.15.129:0/2159645105 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5723 v64) ==== 95+0+0 (secure 0 0 0) 0x7f43cc03f030 con 0x7f43e00b7c10 2024-09-17T03:54:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 5723 2024-09-17T03:54:17.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 msgr2=0x7f43c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 0x7f43c007b0a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f43d40016c0 tx=0x7f43d4009040 comp rx=0 tx=0).stop 2024-09-17T03:54:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 msgr2=0x7f43e0154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e0154b60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f43cc031000 tx=0x7f43cc033ba0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 shutdown_connections 2024-09-17T03:54:17.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43c0078b90 0x7f43c007b0a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43e015eb00 0x7f43e01f9240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43e00b7c10 0x7f43e0154b60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.546+0000 7f43e6c00640 1 --2- 172.21.15.129:0/2159645105 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43e006d7b0 0x7f43e00aafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.547+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 >> 172.21.15.129:0/2159645105 conn(0x7f43e00d1770 msgr2=0x7f43e00b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.547+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 shutdown_connections 2024-09-17T03:54:17.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.547+0000 7f43e6c00640 1 -- 172.21.15.129:0/2159645105 wait complete. 2024-09-17T03:54:17.549 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:54:17.549+0000 7f409c400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:54:17.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.556+0000 7f17ef400640 1 -- 172.21.15.129:0/3559524842 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6358 v65) ==== 95+0+0 (secure 0 0 0) 0x7f17e4063230 con 0x7f17f80b3180 2024-09-17T03:54:17.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.556+0000 7f2d5f400640 1 -- 172.21.15.129:0/3332342559 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6546 v65) ==== 95+0+0 (secure 0 0 0) 0x7f2d60068470 con 0x7f2d6815faa0 2024-09-17T03:54:17.556 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6546 2024-09-17T03:54:17.556 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6358 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.559+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 msgr2=0x7f2d4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.559+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 0x7f2d4807b0a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2d58007fd0 tx=0x7f2d58001450 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 msgr2=0x7f2d681539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 0x7f2d681539a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f2d6000eb50 tx=0x7f2d6000c6a0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 shutdown_connections 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d48078b90 0x7f2d4807b0a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d6815faa0 0x7f2d681539a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d6815f150 0x7f2d680ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 --2- 172.21.15.129:0/3332342559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d6806f930 0x7f2d681571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 >> 172.21.15.129:0/3332342559 conn(0x7f2d6806d560 msgr2=0x7f2d681591f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:17.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 shutdown_connections 2024-09-17T03:54:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f2d67e00640 1 -- 172.21.15.129:0/3332342559 wait complete. 2024-09-17T03:54:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 msgr2=0x7f17d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 0x7f17d807b0d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f17e8033000 tx=0x7f17e8004730 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 msgr2=0x7f17f81fac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.560+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17e400e6a0 tx=0x7f17e400eb60 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 shutdown_connections 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078bc0 0x7f17d807b0d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80d4090 0x7f17f80cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f80b3180 0x7f17f81fac60 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 --2- 172.21.15.129:0/3559524842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806d0d0 0x7f17f81fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 >> 172.21.15.129:0/3559524842 conn(0x7f17f80b6710 msgr2=0x7f17f80b8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 shutdown_connections 2024-09-17T03:54:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:17.561+0000 7f17fca00640 1 -- 172.21.15.129:0/3559524842 wait complete. 2024-09-17T03:54:17.575 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:54:17.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:54:17.580 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:54:17.580 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:17.587 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:54:17.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:54:17.591 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:54:17.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:54:17.636 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:54:17.637 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:17.639 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:54:17.640 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:17.770 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:54:17.832 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:54:17.832 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:54:18.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.559+0000 7fd0e6000640 1 -- 172.21.15.129:0/1573235227 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5726 v66) ==== 95+0+0 (secure 0 0 0) 0x7fd0e0062d40 con 0x7fd0f00b3180 2024-09-17T03:54:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 5726 2024-09-17T03:54:18.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.562+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 msgr2=0x7fd0d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:18.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.562+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 0x7fd0d007b0a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fd0f01f30e0 tx=0x7fd0dc005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:18.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.562+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 msgr2=0x7fd0f0161fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.562+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000c390 tx=0x7fd0e000c850 comp rx=0 tx=0).stop 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 shutdown_connections 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0d0078b90 0x7fd0d007b0a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0f01f9290 0x7fd0f015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0f00d4090 0x7fd0f01f8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 --2- 172.21.15.129:0/1573235227 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0f00b3180 0x7fd0f0161fe0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 >> 172.21.15.129:0/1573235227 conn(0x7fd0f00b6710 msgr2=0x7fd0f015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 shutdown_connections 2024-09-17T03:54:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.563+0000 7fd0ef200640 1 -- 172.21.15.129:0/1573235227 wait complete. 2024-09-17T03:54:18.592 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:54:18.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:54:18.596 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:54:18.596 DEBUG:teuthology.parallel:result is None 2024-09-17T03:54:18.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:18.786 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 -- 172.21.15.129:0/3839355171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 msgr2=0x7f0f70072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 --2- 172.21.15.129:0/3839355171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f70072d70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c009830 tx=0x7f0f5c02a480 comp rx=0 tx=0).stop 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 -- 172.21.15.129:0/3839355171 shutdown_connections 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 --2- 172.21.15.129:0/3839355171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f70072d70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 --2- 172.21.15.129:0/3839355171 >> v2:172.21.15.181:3300/0 conn(0x7f0f700cd190 0x7f0f7015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 -- 172.21.15.129:0/3839355171 >> 172.21.15.129:0/3839355171 conn(0x7f0f700b23c0 msgr2=0x7f0f700b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.857+0000 7f0f78400640 1 -- 172.21.15.129:0/3839355171 shutdown_connections 2024-09-17T03:54:18.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.858+0000 7f0f78400640 1 -- 172.21.15.129:0/3839355171 wait complete. 2024-09-17T03:54:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.860+0000 7f0f78400640 1 Processor -- start 2024-09-17T03:54:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.860+0000 7f0f78400640 1 -- start start 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f700b59d0 0x7f0f7015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f700cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0f70070060 con 0x7f0f700cd190 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7006fee0 con 0x7f0f700b59d0 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.861+0000 7f0f78400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0f700701e0 con 0x7f0f700cfd80 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37246/0 (socket says 172.21.15.129:37246) 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 -- 172.21.15.129:0/4160444681 learned_addr learned my addr 172.21.15.129:0/4160444681 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f76400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f700cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75a00640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f700b59d0 0x7f0f7015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 -- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 msgr2=0x7f0f700cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f700cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 -- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f700b59d0 msgr2=0x7f0f7015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f700b59d0 0x7f0f7015c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.862+0000 7f0f75000640 1 -- 172.21.15.129:0/4160444681 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f700b57c0 con 0x7f0f700cd190 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f75000640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000c00 tx=0x7f0f5c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c005c90 con 0x7f0f700cd190 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f5c031e50 con 0x7f0f700cd190 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f5c033310 con 0x7f0f700cd190 2024-09-17T03:54:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f700b3940 con 0x7f0f700cd190 2024-09-17T03:54:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.863+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f700b3c00 con 0x7f0f700cd190 2024-09-17T03:54:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.864+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f7015cf70 con 0x7f0f700cd190 2024-09-17T03:54:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.865+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0f5c0334b0 con 0x7f0f700cd190 2024-09-17T03:54:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.866+0000 7f0f6d600640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 0x7f0f5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.866+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f0f5c0bbf80 con 0x7f0f700cd190 2024-09-17T03:54:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.867+0000 7f0f75a00640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 0x7f0f5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.867+0000 7f0f75a00640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 0x7f0f5007b0a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0f640095f0 tx=0x7f0f64005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:18.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:18.871+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0f5c03f030 con 0x7f0f700cd190 2024-09-17T03:54:19.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.091+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f70145900 con 0x7f0f700cd190 2024-09-17T03:54:19.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.091+0000 7f0f6d600640 1 -- 172.21.15.129:0/4160444681 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+4400 (secure 0 0 0) 0x7f0f5c083d10 con 0x7f0f700cd190 2024-09-17T03:54:19.091 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:19.091 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":67,"btime":"2024-09-17T03:54:18:558033+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:18.558009+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-09-17T03:54:19.091 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 67 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 msgr2=0x7f0f5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 0x7f0f5007b0a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0f640095f0 tx=0x7f0f64005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 msgr2=0x7f0f700cb480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c000c00 tx=0x7f0f5c033c20 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 shutdown_connections 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f50078b90 0x7f0f5007b0a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f700cfd80 0x7f0f700cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f700cd190 0x7f0f700cb480 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 --2- 172.21.15.129:0/4160444681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f700b59d0 0x7f0f7015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 >> 172.21.15.129:0/4160444681 conn(0x7f0f700b23c0 msgr2=0x7f0f700b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:19.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 shutdown_connections 2024-09-17T03:54:19.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:19.094+0000 7f0f78400640 1 -- 172.21.15.129:0/4160444681 wait complete. 2024-09-17T03:54:19.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:54:06.985254+0000', 'modified': '2024-09-17T03:54:18.558009+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5876}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5876': {'gid': 5876, 'name': 'c', 'rank': 0, 'incarnation': 67, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.129:6835/3654208591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3654208591}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3654208591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T03:54:19.519 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5876:up:replay 2024-09-17T03:54:20.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 -- 172.21.15.129:0/1661032365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0b59d0 msgr2=0x7ff93c0b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 --2- 172.21.15.129:0/1661032365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0b59d0 0x7ff93c0b5de0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ff934005fd0 tx=0x7ff93402aa10 comp rx=0 tx=0).stop 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 -- 172.21.15.129:0/1661032365 shutdown_connections 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 --2- 172.21.15.129:0/1661032365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0b59d0 0x7ff93c0b5de0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 --2- 172.21.15.129:0/1661032365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cd190 0x7ff93c15cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 -- 172.21.15.129:0/1661032365 >> 172.21.15.129:0/1661032365 conn(0x7ff93c0b23c0 msgr2=0x7ff93c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 -- 172.21.15.129:0/1661032365 shutdown_connections 2024-09-17T03:54:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.751+0000 7ff941e00640 1 -- 172.21.15.129:0/1661032365 wait complete. 2024-09-17T03:54:20.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.753+0000 7ff941e00640 1 Processor -- start 2024-09-17T03:54:20.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.754+0000 7ff941e00640 1 -- start start 2024-09-17T03:54:20.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff93c0b59d0 0x7ff93c143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:20.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:20.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cfd80 0x7ff93c0735a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:20.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c06fe50 con 0x7ff93c0cd190 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c06fcd0 con 0x7ff93c0b59d0 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff941e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff93c06ffd0 con 0x7ff93c0cfd80 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff93a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff93a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37260/0 (socket says 172.21.15.129:37260) 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff93a000640 1 -- 172.21.15.129:0/140417656 learned_addr learned my addr 172.21.15.129:0/140417656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.755+0000 7ff93aa00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff93c0b59d0 0x7ff93c143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93b400640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cfd80 0x7ff93c0735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 -- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cfd80 msgr2=0x7ff93c0735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cfd80 0x7ff93c0735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 -- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff93c0b59d0 msgr2=0x7ff93c143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff93c0b59d0 0x7ff93c143790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 -- 172.21.15.129:0/140417656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93c0c5630 con 0x7ff93c0cd190 2024-09-17T03:54:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff93a000640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff92800c3a0 tx=0x7ff92800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928017070 con 0x7ff93c0cd190 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9280053c0 con 0x7ff93c0cd190 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93c1f9120 con 0x7ff93c0cd190 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.756+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928003850 con 0x7ff93c0cd190 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.757+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff93c1f9900 con 0x7ff93c0cd190 2024-09-17T03:54:20.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.758+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff93c1f9390 con 0x7ff93c0cd190 2024-09-17T03:54:20.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.759+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff92801a070 con 0x7ff93c0cd190 2024-09-17T03:54:20.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.760+0000 7ff933400640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 0x7ff91c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.761+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7ff92809a070 con 0x7ff93c0cd190 2024-09-17T03:54:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.761+0000 7ff93aa00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 0x7ff91c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.761+0000 7ff93aa00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 0x7ff91c07b0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff92c004630 tx=0x7ff92c00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:20.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.765+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff928061df0 con 0x7ff93c0cd190 2024-09-17T03:54:20.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.985+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff93c070510 con 0x7ff93c0cd190 2024-09-17T03:54:20.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.985+0000 7ff933400640 1 -- 172.21.15.129:0/140417656 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5228 (secure 0 0 0) 0x7ff928066820 con 0x7ff93c0cd190 2024-09-17T03:54:20.985 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:20.985 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":69,"btime":"2024-09-17T03:54:20:570874+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:19.580155+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-09-17T03:54:20.985 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 69 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.988+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 msgr2=0x7ff91c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.988+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 0x7ff91c07b0d0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff92c004630 tx=0x7ff92c00b2e0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.988+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 msgr2=0x7ff93c15c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.988+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ff92800c3a0 tx=0x7ff92800c860 comp rx=0 tx=0).stop 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 shutdown_connections 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff91c078bc0 0x7ff91c07b0d0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff93c0cfd80 0x7ff93c0735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff93c0cd190 0x7ff93c15c1e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 --2- 172.21.15.129:0/140417656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff93c0b59d0 0x7ff93c143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:20.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 >> 172.21.15.129:0/140417656 conn(0x7ff93c0b23c0 msgr2=0x7ff93c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:20.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 shutdown_connections 2024-09-17T03:54:20.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:20.989+0000 7ff941e00640 1 -- 172.21.15.129:0/140417656 wait complete. 2024-09-17T03:54:21.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:54:06.985254+0000', 'modified': '2024-09-17T03:54:19.580155+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5876}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5876': {'gid': 5876, 'name': 'c', 'rank': 0, 'incarnation': 67, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.129:6835/3654208591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3654208591}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3654208591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T03:54:21.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5876:up:rejoin 2024-09-17T03:54:22.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.670+0000 7fc9bd200640 1 -- 172.21.15.129:0/1562348917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99c003a30 msgr2=0x7fc99c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.670+0000 7fc9bd200640 1 --2- 172.21.15.129:0/1562348917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc99c003a30 0x7fc99c003ea0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8006fc0 tx=0x7fc99c005830 comp rx=0 tx=0).stop 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.670+0000 7fc9b6800640 1 -- 172.21.15.129:0/1562348917 reap_dead start 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.671+0000 7fc9bd200640 1 -- 172.21.15.129:0/1562348917 shutdown_connections 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.671+0000 7fc9bd200640 1 -- 172.21.15.129:0/1562348917 >> 172.21.15.129:0/1562348917 conn(0x7fc9b80b23c0 msgr2=0x7fc9b80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.671+0000 7fc9bd200640 1 -- 172.21.15.129:0/1562348917 shutdown_connections 2024-09-17T03:54:22.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.671+0000 7fc9bd200640 1 -- 172.21.15.129:0/1562348917 wait complete. 2024-09-17T03:54:22.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.673+0000 7fc9bd200640 1 Processor -- start 2024-09-17T03:54:22.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.673+0000 7fc9bd200640 1 -- start start 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99c003a30 0x7fc9b80cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9b8143480 0x7fc9b80aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b80b6170 con 0x7fc9b815c400 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b80b5ff0 con 0x7fc99c003a30 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.674+0000 7fc9bd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9b806fcd0 con 0x7fc9b8143480 2024-09-17T03:54:22.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37276/0 (socket says 172.21.15.129:37276) 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 -- 172.21.15.129:0/499014470 learned_addr learned my addr 172.21.15.129:0/499014470 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b5e00640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99c003a30 0x7fc9b80cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9afe00640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9b8143480 0x7fc9b80aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 -- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9b8143480 msgr2=0x7fc9b80aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9b8143480 0x7fc9b80aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 -- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99c003a30 msgr2=0x7fc9b80cb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99c003a30 0x7fc9b80cb140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 -- 172.21.15.129:0/499014470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b80b3a10 con 0x7fc9b815c400 2024-09-17T03:54:22.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.675+0000 7fc9b6800640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80ced30 tx=0x7fc9b000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.676+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0004b20 con 0x7fc9b815c400 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.676+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9b0004cc0 con 0x7fc9b815c400 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.676+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0003bd0 con 0x7fc9b815c400 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.676+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b80b3cf0 con 0x7fc9b815c400 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.676+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b80af480 con 0x7fc9b815c400 2024-09-17T03:54:22.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.677+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b80cb700 con 0x7fc9b815c400 2024-09-17T03:54:22.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.678+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9b0004030 con 0x7fc9b815c400 2024-09-17T03:54:22.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.679+0000 7fc9ae000640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 0x7fc99407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:22.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.679+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fc9b009b1d0 con 0x7fc9b815c400 2024-09-17T03:54:22.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.680+0000 7fc9b5e00640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 0x7fc99407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:22.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.680+0000 7fc9b5e00640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 0x7fc99407b0a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8033000 tx=0x7fc9a8038040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:22.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.683+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9b0062f50 con 0x7fc9b815c400 2024-09-17T03:54:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.903+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9b8070260 con 0x7fc9b815c400 2024-09-17T03:54:22.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.903+0000 7fc9ae000640 1 -- 172.21.15.129:0/499014470 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7fc9b0067980 con 0x7fc9b815c400 2024-09-17T03:54:22.902 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:22.903 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:22.903 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 msgr2=0x7fc99407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 0x7fc99407b0a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8033000 tx=0x7fc9a8038040 comp rx=0 tx=0).stop 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 msgr2=0x7fc9b80b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80ced30 tx=0x7fc9b000f3c0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 shutdown_connections 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc994078b90 0x7fc99407b0a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9b815c400 0x7fc9b80b47d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9b8143480 0x7fc9b80aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 --2- 172.21.15.129:0/499014470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc99c003a30 0x7fc9b80cb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 >> 172.21.15.129:0/499014470 conn(0x7fc9b80b23c0 msgr2=0x7fc9b81604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 shutdown_connections 2024-09-17T03:54:22.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:22.906+0000 7fc9bd200640 1 -- 172.21.15.129:0/499014470 wait complete. 2024-09-17T03:54:23.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:54:06.985254+0000', 'modified': '2024-09-17T03:54:22.584225+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 95, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5876}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5876': {'gid': 5876, 'name': 'c', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.129:6835/3654208591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3654208591}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3654208591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5876, 'qdb_cluster': [5876]} 2024-09-17T03:54:23.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:54:23.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:54:23.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 -- 172.21.15.129:0/2985482821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 msgr2=0x7f5844004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 --2- 172.21.15.129:0/2985482821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5844004c10 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f58480016c0 tx=0x7f5844005830 comp rx=0 tx=0).stop 2024-09-17T03:54:23.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f585fe00640 1 -- 172.21.15.129:0/2985482821 reap_dead start 2024-09-17T03:54:23.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 -- 172.21.15.129:0/2985482821 shutdown_connections 2024-09-17T03:54:23.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 -- 172.21.15.129:0/2985482821 >> 172.21.15.129:0/2985482821 conn(0x7f586006d560 msgr2=0x7f58600b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 -- 172.21.15.129:0/2985482821 shutdown_connections 2024-09-17T03:54:23.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.561+0000 7f5865200640 1 -- 172.21.15.129:0/2985482821 wait complete. 2024-09-17T03:54:23.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.564+0000 7f5865200640 1 Processor -- start 2024-09-17T03:54:23.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.564+0000 7f5865200640 1 -- start start 2024-09-17T03:54:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58601f8e10 0x7f58601f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58601508a0 0x7f5860150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5860074b40 con 0x7f5844004800 2024-09-17T03:54:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f58600749c0 con 0x7f58601508a0 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.565+0000 7f5865200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5860074cc0 con 0x7f58601f8e10 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37282/0 (socket says 172.21.15.129:37282) 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 -- 172.21.15.129:0/1206373200 learned_addr learned my addr 172.21.15.129:0/1206373200 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f5856a00640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58601f8e10 0x7f58601f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585fe00640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58601508a0 0x7f5860150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 -- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58601f8e10 msgr2=0x7f58601f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58601f8e10 0x7f58601f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 -- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58601508a0 msgr2=0x7f5860150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58601508a0 0x7f5860150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 -- 172.21.15.129:0/1206373200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58601511f0 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.566+0000 7f585f400640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5850011ae0 tx=0x7f5850009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.567+0000 7f5857e00640 1 -- 172.21.15.129:0/1206373200 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585001e070 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.567+0000 7f5857e00640 1 -- 172.21.15.129:0/1206373200 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585000fe70 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.567+0000 7f5857e00640 1 -- 172.21.15.129:0/1206373200 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850004290 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.567+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5860152c90 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.567+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5860153480 con 0x7f5844004800 2024-09-17T03:54:23.568 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:54:23.568+0000 7f1bd3800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:54:23.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.569+0000 7f5857e00640 1 -- 172.21.15.129:0/1206373200 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5850021070 con 0x7f5844004800 2024-09-17T03:54:23.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.570+0000 7f5857e00640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 0x7f583c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.571+0000 7f5857e00640 1 -- 172.21.15.129:0/1206373200 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f585009b760 con 0x7f5844004800 2024-09-17T03:54:23.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.571+0000 7f5856a00640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 0x7f583c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.571+0000 7f586d89a740 1 Processor -- start 2024-09-17T03:54:23.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.571+0000 7f586d89a740 1 -- start start 2024-09-17T03:54:23.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.572+0000 7f5856a00640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 0x7f583c07b0a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5848032000 tx=0x7f5848001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2f0bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b123b0 0x55ace2f0c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ace2e65430 con 0x55ace2b1a590 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ace2e555d0 con 0x55ace2b123b0 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ace2e44a00 con 0x55ace2b3b270 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37298/0 (socket says 172.21.15.129:37298) 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 -- 172.21.15.129:0/694143208 learned_addr learned my addr 172.21.15.129:0/694143208 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f5856a00640 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b123b0 0x55ace2f0c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585f400640 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2f0bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 -- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 msgr2=0x55ace2f0bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2f0bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 -- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b123b0 msgr2=0x55ace2f0c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b123b0 0x55ace2f0c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.573+0000 7f585fe00640 1 -- 172.21.15.129:0/694143208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ace2e0d520 con 0x55ace2b1a590 2024-09-17T03:54:23.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.574+0000 7f585fe00640 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5858006fd0 tx=0x7f5858003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e7cee96de35f63a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.574+0000 7f584fe00640 1 -- 172.21.15.129:0/694143208 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858006830 con 0x55ace2b1a590 2024-09-17T03:54:23.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.574+0000 7f584fe00640 1 -- 172.21.15.129:0/694143208 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5858011430 con 0x55ace2b1a590 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.574+0000 7f584fe00640 1 -- 172.21.15.129:0/694143208 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858011800 con 0x55ace2b1a590 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 -- 172.21.15.129:0/694143208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 msgr2=0x55ace2edc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5858006fd0 tx=0x7f5858003d40 comp rx=0 tx=0).stop 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 -- 172.21.15.129:0/694143208 shutdown_connections 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edc150 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b123b0 0x55ace2f0c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 --2- 172.21.15.129:0/694143208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2f0bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 -- 172.21.15.129:0/694143208 >> 172.21.15.129:0/694143208 conn(0x55ace2b2f3c0 msgr2=0x55ace2ee1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 -- 172.21.15.129:0/694143208 shutdown_connections 2024-09-17T03:54:23.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.575+0000 7f586d89a740 1 -- 172.21.15.129:0/694143208 wait complete. 2024-09-17T03:54:23.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.576+0000 7f586d89a740 1 -- start start 2024-09-17T03:54:23.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.576+0000 7f586d89a740 1 Processor -- start 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b1a590 0x55ace2f21010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2f29ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ace2deb330 con 0x55ace2b123b0 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ace2e48810 con 0x55ace2b3b270 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ace2bb5620 con 0x55ace2b1a590 2024-09-17T03:54:23.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f586d89a740 10 client.0 _ll_register_callbacks cb 0x55ace2df0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37304/0 (socket says 172.21.15.129:37304) 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f585f400640 1 -- 172.21.15.129:0/3858388250 learned_addr learned my addr 172.21.15.129:0/3858388250 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f585fe00640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2f29ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.578+0000 7f5856a00640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b1a590 0x55ace2f21010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 -- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b1a590 msgr2=0x55ace2f21010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b1a590 0x55ace2f21010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 -- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 msgr2=0x55ace2f29ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2f29ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 -- 172.21.15.129:0/3858388250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ace2ba6850 con 0x55ace2b123b0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585fe00640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2f29ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f585f400640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5850060940 tx=0x7f58500635b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbd408466b93567 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f584cc00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850089070 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f5850062930 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5858011430 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f5850084050 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f586d89a740 10 client.6750 fetch_fsmap learned FSMap version 71 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.579+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) -- 0x55ace2c1d6b0 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5858011430 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 71) ==== 3502+0+0 (secure 0 0 0) 0x7f585006a310 con 0x55ace2b123b0 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 10 client.6750 fetch_fsmap finished waiting for FSMap version 71 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 10 client.6750 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 20 client.6750 populate_metadata read hostname 'smithi129' 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2af9e40 0x55ace2f38ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 4 client.6750 mds_command: new command op to 5876 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:54:23.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 --> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ace2e57520 con 0x55ace2af9e40 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.580+0000 7f5856a00640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2af9e40 0x55ace2f38ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.581+0000 7f584cc00640 10 client.6750 ms_handle_connect on v2:172.21.15.129:6833/3654208591 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.581+0000 7f5856a00640 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2af9e40 0x55ace2f38ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10785fc05bcb81b1 server_cookie=3a47ac0885c0d309 in_seq=0 out_seq=0 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f584cc00640 1 -- 172.21.15.129:0/3858388250 <== mds.0 v2:172.21.15.129:6833/3654208591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ace2e57520 con 0x55ace2af9e40 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f584cc00640 10 client.6750 handle_command_reply: tid=0 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 client.6750 shutdown 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 msgr2=0x55ace2f02240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f5850060940 tx=0x7f58500635b0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 shutdown_connections 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2af9e40 0x55ace2f38ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2f29ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b1a590 0x55ace2f21010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f584cc00640 0 client.6750 ms_handle_reset on v2:172.21.15.129:6833/3654208591 2024-09-17T03:54:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 --2- 172.21.15.129:0/3858388250 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f02240 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 >> 172.21.15.129:0/3858388250 conn(0x55ace2b2f3c0 msgr2=0x55ace2edd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 shutdown_connections 2024-09-17T03:54:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 1 -- 172.21.15.129:0/3858388250 wait complete. 2024-09-17T03:54:23.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.582+0000 7f586d89a740 20 client.6750 trim_cache size 0 max 16384 2024-09-17T03:54:23.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.602+0000 7f586d89a740 1 Processor -- start 2024-09-17T03:54:23.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.602+0000 7f586d89a740 1 -- start start 2024-09-17T03:54:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b123b0 0x55ace2ee1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2edc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58602016d0 con 0x55ace2b1a590 2024-09-17T03:54:23.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f58601f6320 con 0x55ace2b3b270 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.603+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5860201080 con 0x55ace2b123b0 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37306/0 (socket says 172.21.15.129:37306) 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 -- 172.21.15.129:0/3661664192 learned_addr learned my addr 172.21.15.129:0/3661664192 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f585f400640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b123b0 0x55ace2ee1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f585fe00640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2edc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 -- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b123b0 msgr2=0x55ace2ee1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b123b0 0x55ace2ee1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 -- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 msgr2=0x55ace2edc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2edc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 -- 172.21.15.129:0/3661664192 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ace2f354d0 con 0x55ace2b1a590 2024-09-17T03:54:23.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f585fe00640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2edc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:23.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.604+0000 7f5856a00640 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f58480017b0 tx=0x7f584803a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3db65f83a998eac2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.605+0000 7f584f400640 1 -- 172.21.15.129:0/3661664192 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848038040 con 0x55ace2b1a590 2024-09-17T03:54:23.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.605+0000 7f584f400640 1 -- 172.21.15.129:0/3661664192 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5848043040 con 0x55ace2b1a590 2024-09-17T03:54:23.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.605+0000 7f584f400640 1 -- 172.21.15.129:0/3661664192 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f584804b890 con 0x55ace2b1a590 2024-09-17T03:54:23.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.605+0000 7f586d89a740 1 -- 172.21.15.129:0/3661664192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 msgr2=0x55ace2edbe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.605+0000 7f586d89a740 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f58480017b0 tx=0x7f584803a000 comp rx=0 tx=0).stop 2024-09-17T03:54:23.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 -- 172.21.15.129:0/3661664192 shutdown_connections 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b3b270 0x55ace2edc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b1a590 0x55ace2edbe40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 --2- 172.21.15.129:0/3661664192 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b123b0 0x55ace2ee1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 -- 172.21.15.129:0/3661664192 >> 172.21.15.129:0/3661664192 conn(0x55ace2b2f3c0 msgr2=0x55ace2f05d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 -- 172.21.15.129:0/3661664192 shutdown_connections 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.606+0000 7f586d89a740 1 -- 172.21.15.129:0/3661664192 wait complete. 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.607+0000 7f586d89a740 1 -- start start 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.607+0000 7f586d89a740 1 Processor -- start 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b1a590 0x55ace2f0bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2fa9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ace2f027d0 con 0x55ace2b123b0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ace2f05850 con 0x55ace2b1a590 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ace2ef2ff0 con 0x55ace2b3b270 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.608+0000 7f586d89a740 10 client.0 _ll_register_callbacks cb 0x7f5850063020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37310/0 (socket says 172.21.15.129:37310) 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 -- 172.21.15.129:0/2694055741 learned_addr learned my addr 172.21.15.129:0/2694055741 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585fe00640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2fa9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f5856a00640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b1a590 0x55ace2f0bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 -- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 msgr2=0x55ace2fa9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2fa9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 -- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b1a590 msgr2=0x55ace2f0bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b1a590 0x55ace2f0bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 -- 172.21.15.129:0/2694055741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ace2edbd70 con 0x55ace2b123b0 2024-09-17T03:54:23.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f585f400640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f58500633d0 tx=0x7f5850062ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55b727b1806bf5c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f584cc00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850089040 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.609+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f5850089de0 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5848043040 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=71) ==== 24+0+0 (secure 0 0 0) 0x7f5850096050 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 10 client.6756 fetch_fsmap learned FSMap version 71 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=71,monmap=2+}) -- 0x55ace2f0c2e0 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5848043040 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 71) ==== 3502+0+0 (secure 0 0 0) 0x7f5850099020 con 0x55ace2b123b0 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 10 client.6756 fetch_fsmap finished waiting for FSMap version 71 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 10 client.6756 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 20 client.6756 populate_metadata read hostname 'smithi129' 2024-09-17T03:54:23.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2edc620 0x55ace2f091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:23.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 4 client.6756 mds_command: new command op to 5876 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:54:23.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 --> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] -- command(tid 0: {"prefix": "status"}) -- 0x55ace2ee11a0 con 0x55ace2edc620 2024-09-17T03:54:23.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.610+0000 7f5856a00640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2edc620 0x55ace2f091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f584cc00640 10 client.6756 ms_handle_connect on v2:172.21.15.129:6833/3654208591 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f5856a00640 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2edc620 0x55ace2f091d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da7314ce08ecde1c server_cookie=52b183f8bddc7cf in_seq=0 out_seq=0 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f584cc00640 1 -- 172.21.15.129:0/2694055741 <== mds.0 v2:172.21.15.129:6833/3654208591 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ace2ee11a0 con 0x55ace2edc620 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f584cc00640 10 client.6756 handle_command_reply: tid=0 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f586d89a740 1 client.6756 shutdown 2024-09-17T03:54:23.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 msgr2=0x55ace2f8bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.611+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f58500633d0 tx=0x7f5850062ef0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 shutdown_connections 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:6833/3654208591,v1:172.21.15.129:6835/3654208591] conn(0x55ace2edc620 0x55ace2f091d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ace2b3b270 0x55ace2fa9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ace2b1a590 0x55ace2f0bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f584cc00640 0 client.6756 ms_handle_reset on v2:172.21.15.129:6833/3654208591 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 --2- 172.21.15.129:0/2694055741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ace2b123b0 0x55ace2f8bf20 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 >> 172.21.15.129:0/2694055741 conn(0x55ace2b2f3c0 msgr2=0x55ace2edcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 shutdown_connections 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 1 -- 172.21.15.129:0/2694055741 wait complete. 2024-09-17T03:54:23.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f586d89a740 20 client.6756 trim_cache size 0 max 16384 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 5876, 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 5.0452764940000003, 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 71, 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 96, 2024-09-17T03:54:23.616 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 96, 2024-09-17T03:54:23.617 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 5.7748936469999999 2024-09-17T03:54:23.617 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:54:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 msgr2=0x7f583c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 0x7f583c07b0a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5848032000 tx=0x7f5848001190 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 msgr2=0x7f5860152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.612+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5850011ae0 tx=0x7f5850009e90 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 shutdown_connections 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f583c078b90 0x7f583c07b0a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58601508a0 0x7f5860150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58601f8e10 0x7f58601f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 --2- 172.21.15.129:0/1206373200 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5844004800 0x7f5860152750 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 >> 172.21.15.129:0/1206373200 conn(0x7f586006d560 msgr2=0x7f58600b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 shutdown_connections 2024-09-17T03:54:23.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:23.613+0000 7f5865200640 1 -- 172.21.15.129:0/1206373200 wait complete. 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> subdir = "delete_me" 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> size_unit = 1024 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> file_multiplier = 200 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-09-17T03:54:23.634 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(0, file_multiplier): 2024-09-17T03:54:23.635 DEBUG:teuthology.orchestra.run.smithi129:> for size in range(0, 16*size_unit, size_unit): 2024-09-17T03:54:23.635 DEBUG:teuthology.orchestra.run.smithi129:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-09-17T03:54:23.635 DEBUG:teuthology.orchestra.run.smithi129:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-09-17T03:54:23.635 DEBUG:teuthology.orchestra.run.smithi129:> f.write(size * '"'"'x'"'"') 2024-09-17T03:54:23.635 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:54:28.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:54:28.042 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_ops 2024-09-17T03:54:28.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:28.043 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-09-17T03:54:28.288 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_max_purge_ops":"100000000"} 2024-09-17T03:54:28.289 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "100000000" } 2024-09-17T03:54:28.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2024-09-17T03:54:28.525 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_max_purge_files":"6"} 2024-09-17T03:54:28.526 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "6" } 2024-09-17T03:54:28.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 -- 172.21.15.129:0/3667841608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab8162850 msgr2=0x7f7ab8162c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 --2- 172.21.15.129:0/3667841608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab8162850 0x7f7ab8162c60 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0005fd0 tx=0x7f7aa002a540 comp rx=0 tx=0).stop 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 -- 172.21.15.129:0/3667841608 shutdown_connections 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 --2- 172.21.15.129:0/3667841608 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ab80d28a0 0x7f7ab80d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 --2- 172.21.15.129:0/3667841608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab8162850 0x7f7ab8162c60 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 -- 172.21.15.129:0/3667841608 >> 172.21.15.129:0/3667841608 conn(0x7f7ab80d4b70 msgr2=0x7f7ab80d4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 -- 172.21.15.129:0/3667841608 shutdown_connections 2024-09-17T03:54:28.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.764+0000 7f7ab7200640 1 -- 172.21.15.129:0/3667841608 wait complete. 2024-09-17T03:54:28.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.768+0000 7f7ab7200640 1 Processor -- start 2024-09-17T03:54:28.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.768+0000 7f7ab7200640 1 -- start start 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.770+0000 7f7ab7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ab80d28a0 0x7f7ab80c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 0x7f7ab8160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab80d38e0 con 0x7f7ab806ea50 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab80d3760 con 0x7f7ab8162850 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab815e820 con 0x7f7ab80d28a0 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57198/0 (socket says 172.21.15.129:57198) 2024-09-17T03:54:28.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 -- 172.21.15.129:0/4092942549 learned_addr learned my addr 172.21.15.129:0/4092942549 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 -- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ab80d28a0 msgr2=0x7f7ab80c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ab80d28a0 0x7f7ab80c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 -- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 msgr2=0x7f7ab8160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab6800640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 0x7f7ab8160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 0x7f7ab8160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab5e00640 1 -- 172.21.15.129:0/4092942549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab81f54e0 con 0x7f7ab806ea50 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.771+0000 7f7ab6800640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 0x7f7ab8160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7ab5e00640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800c3a0 tx=0x7f7aa800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8017070 con 0x7f7ab806ea50 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7aa80053c0 con 0x7f7ab806ea50 2024-09-17T03:54:28.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab81f57a0 con 0x7f7ab806ea50 2024-09-17T03:54:28.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8003850 con 0x7f7ab806ea50 2024-09-17T03:54:28.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.772+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab81f5d60 con 0x7f7ab806ea50 2024-09-17T03:54:28.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.773+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab80d2cb0 con 0x7f7ab806ea50 2024-09-17T03:54:28.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.775+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7aa801a070 con 0x7f7ab806ea50 2024-09-17T03:54:28.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.776+0000 7f7aae000640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 0x7f7a9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:28.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.776+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f7aa809a180 con 0x7f7ab806ea50 2024-09-17T03:54:28.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.776+0000 7f7aafe00640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 0x7f7a9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:28.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.777+0000 7f7aafe00640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 0x7f7a9407b0a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0033000 tx=0x7f7aa00046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:28.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:28.782+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7aa8062550 con 0x7f7ab806ea50 2024-09-17T03:54:29.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.063+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ab806a410 con 0x7f7ab806ea50 2024-09-17T03:54:29.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.064+0000 7f7aae000640 1 -- 172.21.15.129:0/4092942549 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f7aa8061e50 con 0x7f7ab806ea50 2024-09-17T03:54:29.064 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:29.064 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:29.064 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:29.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 msgr2=0x7f7a9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:29.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 0x7f7a9407b0a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0033000 tx=0x7f7aa00046c0 comp rx=0 tx=0).stop 2024-09-17T03:54:29.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 msgr2=0x7f7ab80cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:29.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f7aa800c3a0 tx=0x7f7aa800c860 comp rx=0 tx=0).stop 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 shutdown_connections 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7a94078b90 0x7f7a9407b0a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ab8162850 0x7f7ab8160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ab80d28a0 0x7f7ab80c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 --2- 172.21.15.129:0/4092942549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ab806ea50 0x7f7ab80cd5b0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 >> 172.21.15.129:0/4092942549 conn(0x7f7ab80d4b70 msgr2=0x7f7ab815de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 shutdown_connections 2024-09-17T03:54:29.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:29.067+0000 7f7ab7200640 1 -- 172.21.15.129:0/4092942549 wait complete. 2024-09-17T03:54:29.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:29.435 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":25,"msgr_send_messages":25,"msgr_recv_bytes":15887,"msgr_send_bytes":4226606,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.024854337,"msgr_running_send_time":0.011206356,"msgr_running_recv_time":0.008084924,"msgr_running_fast_dispatch_time":0.002555338,"msgr_send_messages_queue_lat":{"avgcount":24,"sum":0.012339408,"avgtime":0.000514142},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":4364,"msgr_send_messages":10998,"msgr_recv_bytes":1616185,"msgr_send_bytes":6903745,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.882947669,"msgr_running_send_time":0.664819817,"msgr_running_recv_time":0.262597881,"msgr_running_fast_dispatch_time":0.001984667,"msgr_send_messages_queue_lat":{"avgcount":10997,"sum":0.429744479,"avgtime":0.000039078},"msgr_handle_ack_lat":{"avgcount":10552,"sum":0.002194319,"avgtime":0.000000207},"msgr_recv_encrypted_bytes":119488,"msgr_send_encrypted_bytes":6160},"AsyncMessenger::Worker-2":{"msgr_recv_messages":25,"msgr_send_messages":30,"msgr_recv_bytes":11463,"msgr_send_bytes":2990389,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.016135746,"msgr_running_send_time":0.010072229,"msgr_running_recv_time":0.003392078,"msgr_running_fast_dispatch_time":0.002251768,"msgr_send_messages_queue_lat":{"avgcount":29,"sum":0.007135703,"avgtime":0.000246058},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":61488},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":623,"complete_latency":{"avgcount":74,"sum":1.016374391,"avgtime":0.013734789}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000140359,"avgtime":0.000035089}},"mds":{"request":3762,"reply":3760,"reply_latency":{"avgcount":3760,"sum":4.276313824,"avgtime":0.001137317},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":3763,"inodes_top":0,"inodes_bottom":3763,"inodes_pin_tail":0,"inodes_pinned":3763,"inodes_expired":0,"inodes_with_caps":3202,"caps":3202,"subtrees":2,"traverse":3762,"traverse_hit":3762,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":3013,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":3479,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":535,"handle_client_caps_dirty":531,"handle_client_cap_release":0,"process_request_cap_release":560},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":552,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":552,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":4293,"evex":0,"evtrm":0,"ev":4296,"evexg":0,"evexd":0,"segadd":4,"segex":0,"segtrm":0,"seg":6,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":13836357,"rdpos":4195244,"jlat":{"avgcount":13,"sum":0.091801829,"avgtime":0.007061679},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":3763,"dn+":3763,"dn-":0,"cap":3202,"cap+":3202,"cap-":0,"rss":148948,"heap":381505536},"mds_server":{"dispatch_client_request":3762,"dispatch_server_request":0,"handle_client_request":3762,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":552,"sum":0.894381601,"avgtime":0.001620256},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3629,"average_load":3629,"avg_session_uptime":797,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":425221,"buffer_anon_items":3158,"buffer_meta_bytes":2200,"buffer_meta_items":25,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10401995,"mds_co_items":105877,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":64,"op_send_bytes":8647220,"op_resend":0,"op_reply":64,"op_latency":{"avgcount":64,"sum":0.172996274,"avgtime":0.002703066},"op_inflight":0,"oplen_avg":{"avgcount":64,"sum":115},"op":64,"op_r":36,"op_w":28,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":13,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":7,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":5092,"max":104857600,"get_started":0,"get":4410,"get_sum":1312894,"get_or_fail_fail":0,"get_or_fail_success":4410,"take":0,"take_sum":0,"put":4391,"put_sum":1307802,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":8648155,"put":64,"put_sum":8648155,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":64,"take_sum":64,"put":64,"put_sum":64,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":1253445,"max":3758096384,"get_started":0,"get":4293,"get_sum":9640854,"get_or_fail_fail":0,"get_or_fail_success":4293,"take":0,"take_sum":0,"put":13,"put_sum":8387409,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:29.438 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 15887, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.002555338, "msgr_running_recv_time": 0.008084924, "msgr_running_send_time": 0.011206356, "msgr_running_total_time": 0.024854337, "msgr_send_bytes": 4226606, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 25, "msgr_send_messages_queue_lat": { "avgcount": 24, "avgtime": 0.000514142, "sum": 0.012339408 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 10552, "avgtime": 2.07e-07, "sum": 0.002194319 }, "msgr_recv_bytes": 1616185, "msgr_recv_encrypted_bytes": 119488, "msgr_recv_messages": 4364, "msgr_running_fast_dispatch_time": 0.001984667, "msgr_running_recv_time": 0.262597881, "msgr_running_send_time": 0.664819817, "msgr_running_total_time": 0.882947669, "msgr_send_bytes": 6903745, "msgr_send_encrypted_bytes": 6160, "msgr_send_messages": 10998, "msgr_send_messages_queue_lat": { "avgcount": 10997, "avgtime": 3.9078e-05, "sum": 0.429744479 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 11463, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.002251768, "msgr_running_recv_time": 0.003392078, "msgr_running_send_time": 0.010072229, "msgr_running_total_time": 0.016135746, "msgr_send_bytes": 2990389, "msgr_send_encrypted_bytes": 61488, "msgr_send_messages": 30, "msgr_send_messages_queue_lat": { "avgcount": 29, "avgtime": 0.000246058, "sum": 0.007135703 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 74, "avgtime": 0.013734789, "sum": 1.016374391 }, "queue_len": 623 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 3.5089e-05, "sum": 0.000140359 }, "queue_len": 0 }, "mds": { "caps": 3202, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3479, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 535, "handle_client_caps_dirty": 531, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3763, "inodes_bottom": 3763, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3763, "inodes_top": 0, "inodes_with_caps": 3202, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 560, "q": 0, "reply": 3760, "reply_latency": { "avgcount": 3760, "avgtime": 0.001137317, "sum": 4.276313824 }, "request": 3762, "root_rbytes": 0, "root_rfiles": 3013, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3762, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3762, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 552, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 552, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 4296, "evadd": 4293, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 13, "avgtime": 0.007061679, "sum": 0.091801829 }, "rdpos": 4195244, "replayed": 3, "seg": 6, "segadd": 4, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 13836357 }, "mds_mem": { "cap": 3202, "cap+": 3202, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 3763, "dn+": 3763, "dn-": 0, "heap": 381505536, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 148948 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3762, "dispatch_server_request": 0, "handle_client_request": 3762, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 552, "avgtime": 0.001620256, "sum": 0.894381601 } }, "mds_sessions": { "average_load": 3629, "avg_session_uptime": 797, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3629 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 425221, "buffer_anon_items": 3158, "buffer_meta_bytes": 2200, "buffer_meta_items": 25, "mds_co_bytes": 10401995, "mds_co_items": 105877, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 0, "omap_rd": 48, "omap_wr": 11, "op": 64, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 64, "avgtime": 0.002703066, "sum": 0.172996274 }, "op_pg": 0, "op_r": 36, "op_reply": 64, "op_resend": 0, "op_rmw": 0, "op_send": 64, "op_send_bytes": 8647220, "op_w": 28, "oplen_avg": { "avgcount": 64, "sum": 115 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 7, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 13, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 4410, "get_or_fail_fail": 0, "get_or_fail_success": 4410, "get_started": 0, "get_sum": 1312894, "max": 104857600, "put": 4391, "put_sum": 1307802, "take": 0, "take_sum": 0, "val": 5092, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 64, "put_sum": 8648155, "take": 64, "take_sum": 8648155, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 64, "put_sum": 64, "take": 64, "take_sum": 64, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 4293, "get_or_fail_fail": 0, "get_or_fail_success": 4293, "get_started": 0, "get_sum": 9640854, "max": 3758096384, "put": 13, "put_sum": 8387409, "take": 0, "take_sum": 0, "val": 1253445, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:29.438 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/552, 0/552 2024-09-17T03:54:30.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 -- 172.21.15.129:0/400262306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0b59d0 msgr2=0x7f4d3c0b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 --2- 172.21.15.129:0/400262306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c0b5de0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f4d34005fd0 tx=0x7f4d3402aa10 comp rx=0 tx=0).stop 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 -- 172.21.15.129:0/400262306 shutdown_connections 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 --2- 172.21.15.129:0/400262306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c0b5de0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 --2- 172.21.15.129:0/400262306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cd190 0x7f4d3c15cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 -- 172.21.15.129:0/400262306 >> 172.21.15.129:0/400262306 conn(0x7f4d3c0b23c0 msgr2=0x7f4d3c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 -- 172.21.15.129:0/400262306 shutdown_connections 2024-09-17T03:54:30.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.734+0000 7f4d41a00640 1 -- 172.21.15.129:0/400262306 wait complete. 2024-09-17T03:54:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.738+0000 7f4d41a00640 1 Processor -- start 2024-09-17T03:54:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.738+0000 7f4d41a00640 1 -- start start 2024-09-17T03:54:30.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c1f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:30.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cfd80 0x7f4d3c13d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c06fe50 con 0x7f4d3c0cd190 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c06fcd0 con 0x7f4d3c0b59d0 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d41a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c06ffd0 con 0x7f4d3c0cfd80 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d3a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d3a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57222/0 (socket says 172.21.15.129:57222) 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d3a000640 1 -- 172.21.15.129:0/4149049359 learned_addr learned my addr 172.21.15.129:0/4149049359 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d3aa00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c1f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.740+0000 7f4d3b400640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cfd80 0x7f4d3c13d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 -- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cfd80 msgr2=0x7f4d3c13d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cfd80 0x7f4d3c13d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:30.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 -- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d3c0b59d0 msgr2=0x7f4d3c1f5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:30.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c1f5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:30.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 -- 172.21.15.129:0/4149049359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c1f6390 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d3a000640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800c3f0 tx=0x7f4d2800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28017070 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d280035f0 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28003a40 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c1f5760 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.741+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c13c860 con 0x7f4d3c0cd190 2024-09-17T03:54:30.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.743+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3c070530 con 0x7f4d3c0cd190 2024-09-17T03:54:30.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.744+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4d2801a070 con 0x7f4d3c0cd190 2024-09-17T03:54:30.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.745+0000 7f4d32a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 0x7f4d1c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:30.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.745+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f4d2809a060 con 0x7f4d3c0cd190 2024-09-17T03:54:30.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.745+0000 7f4d3aa00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 0x7f4d1c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:30.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.746+0000 7f4d3aa00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 0x7f4d1c07b0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009b30 tx=0x7f4d2c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:30.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:30.750+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4d3c1f5760 con 0x7f4d3c0cd190 2024-09-17T03:54:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.087+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d3c13c2a0 con 0x7f4d3c0cd190 2024-09-17T03:54:31.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.088+0000 7f4d32a00640 1 -- 172.21.15.129:0/4149049359 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f4d28061e20 con 0x7f4d3c0cd190 2024-09-17T03:54:31.088 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:31.088 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:31.088 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 msgr2=0x7f4d1c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 0x7f4d1c07b0d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009b30 tx=0x7f4d2c005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 msgr2=0x7f4d3c13b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:31.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f4d2800c3f0 tx=0x7f4d2800c8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 shutdown_connections 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d1c078bc0 0x7f4d1c07b0d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d3c0cfd80 0x7f4d3c13d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d3c0cd190 0x7f4d3c13b9a0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 --2- 172.21.15.129:0/4149049359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d3c0b59d0 0x7f4d3c1f5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 >> 172.21.15.129:0/4149049359 conn(0x7f4d3c0b23c0 msgr2=0x7f4d3c0aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 shutdown_connections 2024-09-17T03:54:31.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:31.090+0000 7f4d41a00640 1 -- 172.21.15.129:0/4149049359 wait complete. 2024-09-17T03:54:31.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:31.355 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":69,"msgr_send_messages":69,"msgr_recv_bytes":27047,"msgr_send_bytes":4240934,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.032112836,"msgr_running_send_time":0.015787693,"msgr_running_recv_time":0.009175562,"msgr_running_fast_dispatch_time":0.003676049,"msgr_send_messages_queue_lat":{"avgcount":68,"sum":0.016193852,"avgtime":0.000238144},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":6295,"msgr_send_messages":13670,"msgr_recv_bytes":2275028,"msgr_send_bytes":12027240,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.095150604,"msgr_running_send_time":0.815055059,"msgr_running_recv_time":10.015462093,"msgr_running_fast_dispatch_time":0.004887013,"msgr_send_messages_queue_lat":{"avgcount":13669,"sum":0.681440010,"avgtime":0.000049852},"msgr_handle_ack_lat":{"avgcount":11391,"sum":0.002414180,"avgtime":0.000000211},"msgr_recv_encrypted_bytes":119760,"msgr_send_encrypted_bytes":6800},"AsyncMessenger::Worker-2":{"msgr_recv_messages":67,"msgr_send_messages":72,"msgr_recv_bytes":22107,"msgr_send_bytes":3004237,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.019968757,"msgr_running_send_time":0.011715095,"msgr_running_recv_time":0.004280557,"msgr_running_fast_dispatch_time":0.003237248,"msgr_send_messages_queue_lat":{"avgcount":71,"sum":0.009556192,"avgtime":0.000134594},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":61488},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":80,"sum":2.787800956,"avgtime":0.034847511}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":192,"sum":0.183876345,"avgtime":0.000957689}},"mds":{"request":4114,"reply":4112,"reply_latency":{"avgcount":4112,"sum":5.607368117,"avgtime":0.001363659},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4115,"inodes_top":0,"inodes_bottom":4115,"inodes_pin_tail":0,"inodes_pinned":4115,"inodes_expired":0,"inodes_with_caps":3053,"caps":3053,"subtrees":2,"traverse":4114,"traverse_hit":4114,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2647,"root_rbytes":4544512,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":3943,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2058,"handle_client_caps_dirty":1818,"handle_client_cap_release":33,"process_request_cap_release":912},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":796,"num_strays_delayed":41,"num_strays_enqueuing":0,"strays_created":904,"strays_enqueued":108,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":6302,"evex":0,"evtrm":0,"ev":6305,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":18189674,"rdpos":4195244,"jlat":{"avgcount":14,"sum":0.125703677,"avgtime":0.008978834},"replayed":3},"mds_mem":{"ino":3214,"ino+":3214,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":4115,"dn+":4115,"dn-":0,"cap":3053,"cap+":3202,"cap-":149,"rss":175188,"heap":408899584},"mds_server":{"dispatch_client_request":4114,"dispatch_server_request":0,"handle_client_request":4114,"handle_client_session":0,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":904,"sum":2.225435894,"avgtime":0.002461765},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":3912,"average_load":3912,"avg_session_uptime":797,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":425221,"buffer_anon_items":3158,"buffer_meta_bytes":2200,"buffer_meta_items":25,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":10401995,"mds_co_items":105877,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":188,"op_send_bytes":12889230,"op_resend":0,"op_reply":188,"op_latency":{"avgcount":188,"sum":0.555988025,"avgtime":0.002957383},"op_inflight":0,"oplen_avg":{"avgcount":188,"sum":243},"op":188,"op_r":38,"op_w":150,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":10,"osdop_write":16,"osdop_writefull":3,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":122,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":2},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":108,"pq_executed":108,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":6427,"get_sum":1842231,"get_or_fail_fail":0,"get_or_fail_success":6427,"take":0,"take_sum":0,"put":6427,"put_sum":1842231,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":188,"take_sum":12901073,"put":188,"put_sum":12901073,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":188,"take_sum":188,"put":188,"put_sum":188,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":108,"get_sum":10908,"get_or_fail_fail":0,"get_or_fail_success":108,"take":0,"take_sum":0,"put":2,"put_sum":10908,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":1412458,"max":3758096384,"get_started":0,"get":6302,"get_sum":13994171,"get_or_fail_fail":0,"get_or_fail_success":6302,"take":0,"take_sum":0,"put":14,"put_sum":12581713,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:31.357 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 27047, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 69, "msgr_running_fast_dispatch_time": 0.003676049, "msgr_running_recv_time": 0.009175562, "msgr_running_send_time": 0.015787693, "msgr_running_total_time": 0.032112836, "msgr_send_bytes": 4240934, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 69, "msgr_send_messages_queue_lat": { "avgcount": 68, "avgtime": 0.000238144, "sum": 0.016193852 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 11391, "avgtime": 2.11e-07, "sum": 0.00241418 }, "msgr_recv_bytes": 2275028, "msgr_recv_encrypted_bytes": 119760, "msgr_recv_messages": 6295, "msgr_running_fast_dispatch_time": 0.004887013, "msgr_running_recv_time": 10.015462093, "msgr_running_send_time": 0.815055059, "msgr_running_total_time": 1.095150604, "msgr_send_bytes": 12027240, "msgr_send_encrypted_bytes": 6800, "msgr_send_messages": 13670, "msgr_send_messages_queue_lat": { "avgcount": 13669, "avgtime": 4.9852e-05, "sum": 0.68144001 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 22107, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 67, "msgr_running_fast_dispatch_time": 0.003237248, "msgr_running_recv_time": 0.004280557, "msgr_running_send_time": 0.011715095, "msgr_running_total_time": 0.019968757, "msgr_send_bytes": 3004237, "msgr_send_encrypted_bytes": 61488, "msgr_send_messages": 72, "msgr_send_messages_queue_lat": { "avgcount": 71, "avgtime": 0.000134594, "sum": 0.009556192 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 80, "avgtime": 0.034847511, "sum": 2.787800956 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 192, "avgtime": 0.000957689, "sum": 0.183876345 }, "queue_len": 0 }, "mds": { "caps": 3053, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3943, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 33, "handle_client_caps": 2058, "handle_client_caps_dirty": 1818, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4115, "inodes_bottom": 4115, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4115, "inodes_top": 0, "inodes_with_caps": 3053, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 912, "q": 0, "reply": 4112, "reply_latency": { "avgcount": 4112, "avgtime": 0.001363659, "sum": 5.607368117 }, "request": 4114, "root_rbytes": 4544512, "root_rfiles": 2647, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4114, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4114, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 796, "num_strays_delayed": 41, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 904, "strays_enqueued": 108, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 6305, "evadd": 6302, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 14, "avgtime": 0.008978834, "sum": 0.125703677 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 18189674 }, "mds_mem": { "cap": 3053, "cap+": 3202, "cap-": 149, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4115, "dn+": 4115, "dn-": 0, "heap": 408899584, "ino": 3214, "ino+": 3214, "ino-": 0, "rss": 175188 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4114, "dispatch_server_request": 0, "handle_client_request": 4114, "handle_client_session": 0, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 904, "avgtime": 0.002461765, "sum": 2.225435894 } }, "mds_sessions": { "average_load": 3912, "avg_session_uptime": 797, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 3912 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 425221, "buffer_anon_items": 3158, "buffer_meta_bytes": 2200, "buffer_meta_items": 25, "mds_co_bytes": 10401995, "mds_co_items": 105877, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 2, "omap_rd": 48, "omap_wr": 15, "op": 188, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 188, "avgtime": 0.002957383, "sum": 0.555988025 }, "op_pg": 0, "op_r": 38, "op_reply": 188, "op_resend": 0, "op_rmw": 0, "op_send": 188, "op_send_bytes": 12889230, "op_w": 150, "oplen_avg": { "avgcount": 188, "sum": 243 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 122, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 10, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 16, "osdop_writefull": 3, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 108, "pq_executed_ops": 108, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 6427, "get_or_fail_fail": 0, "get_or_fail_success": 6427, "get_started": 0, "get_sum": 1842231, "max": 104857600, "put": 6427, "put_sum": 1842231, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 188, "put_sum": 12901073, "take": 188, "take_sum": 12901073, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 188, "put_sum": 188, "take": 188, "take_sum": 188, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 108, "get_or_fail_fail": 0, "get_or_fail_success": 108, "get_started": 0, "get_sum": 10908, "max": 3758096384, "put": 2, "put_sum": 10908, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 6302, "get_or_fail_fail": 0, "get_or_fail_success": 6302, "get_started": 0, "get_sum": 13994171, "max": 3758096384, "put": 14, "put_sum": 12581713, "take": 0, "take_sum": 0, "val": 1412458, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:31.357 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/796, 108/904 2024-09-17T03:54:32.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:32.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.609+0000 7f99c7200640 1 -- 172.21.15.129:0/2286987903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c015f110 msgr2=0x7f99c0141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.609+0000 7f99c7200640 1 --2- 172.21.15.129:0/2286987903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c015f110 0x7f99c0141800 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f99c0159710 tx=0x7f99b002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 -- 172.21.15.129:0/2286987903 shutdown_connections 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 --2- 172.21.15.129:0/2286987903 >> v2:172.21.15.181:3300/0 conn(0x7f99c00aeca0 0x7f99c006f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 --2- 172.21.15.129:0/2286987903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c015f110 0x7f99c0141800 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 -- 172.21.15.129:0/2286987903 >> 172.21.15.129:0/2286987903 conn(0x7f99c00b23c0 msgr2=0x7f99c00b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:32.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 -- 172.21.15.129:0/2286987903 shutdown_connections 2024-09-17T03:54:32.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.610+0000 7f99c7200640 1 -- 172.21.15.129:0/2286987903 wait complete. 2024-09-17T03:54:32.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.612+0000 7f99c7200640 1 Processor -- start 2024-09-17T03:54:32.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.613+0000 7f99c7200640 1 -- start start 2024-09-17T03:54:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.614+0000 7f99c7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:32.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.614+0000 7f99c7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 0x7f99c013e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99c7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99c015f110 0x7f99c01468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99c7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99c00b5d30 con 0x7f99c00ae890 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99c7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f99c0070060 con 0x7f99c015f110 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99c7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f99c00b5eb0 con 0x7f99c00aeca0 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57244/0 (socket says 172.21.15.129:57244) 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 -- 172.21.15.129:0/1699191624 learned_addr learned my addr 172.21.15.129:0/1699191624 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:32.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99c5200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99c015f110 0x7f99c01468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99b7e00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 0x7f99c013e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 -- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 msgr2=0x7f99c013e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 0x7f99c013e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 -- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99c015f110 msgr2=0x7f99c01468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99c015f110 0x7f99c01468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99bfe00640 1 -- 172.21.15.129:0/1699191624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c0157b00 con 0x7f99c00ae890 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.615+0000 7f99b7e00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 0x7f99c013e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99bfe00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f99b0000c00 tx=0x7f99b0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b003e030 con 0x7f99c00ae890 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99b0003e00 con 0x7f99c00ae890 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c0157d80 con 0x7f99c00ae890 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b0035040 con 0x7f99c00ae890 2024-09-17T03:54:32.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.616+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c0158340 con 0x7f99c00ae890 2024-09-17T03:54:32.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.618+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c00766e0 con 0x7f99c00ae890 2024-09-17T03:54:32.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.619+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f99b0041030 con 0x7f99c00ae890 2024-09-17T03:54:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.621+0000 7f99bcc00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 0x7f999c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.621+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f99b00bb430 con 0x7f99c00ae890 2024-09-17T03:54:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.621+0000 7f99b7e00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 0x7f999c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:32.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.622+0000 7f99b7e00640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 0x7f999c07b0a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f99a8007fd0 tx=0x7f99a8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:32.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.626+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f99c00766e0 con 0x7f99c00ae890 2024-09-17T03:54:32.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.922+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99c006f710 con 0x7f99c00ae890 2024-09-17T03:54:32.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.923+0000 7f99bcc00640 1 -- 172.21.15.129:0/1699191624 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f99c0157b00 con 0x7f99c00ae890 2024-09-17T03:54:32.923 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:32.923 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:32.923 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 msgr2=0x7f999c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 0x7f999c07b0a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f99a8007fd0 tx=0x7f99a8009040 comp rx=0 tx=0).stop 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 msgr2=0x7f99c0149520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f99b0000c00 tx=0x7f99b0003910 comp rx=0 tx=0).stop 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 shutdown_connections 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f999c078b90 0x7f999c07b0a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99c015f110 0x7f99c01468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99c00aeca0 0x7f99c013e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 --2- 172.21.15.129:0/1699191624 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99c00ae890 0x7f99c0149520 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 >> 172.21.15.129:0/1699191624 conn(0x7f99c00b23c0 msgr2=0x7f99c0074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 shutdown_connections 2024-09-17T03:54:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:32.926+0000 7f99c7200640 1 -- 172.21.15.129:0/1699191624 wait complete. 2024-09-17T03:54:32.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:33.206 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":168,"msgr_send_messages":168,"msgr_recv_bytes":52185,"msgr_send_bytes":4273200,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.046669082,"msgr_running_send_time":0.023468546,"msgr_running_recv_time":0.011943509,"msgr_running_fast_dispatch_time":0.006891839,"msgr_send_messages_queue_lat":{"avgcount":167,"sum":0.024174677,"avgtime":0.000144758},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7439,"msgr_send_messages":16264,"msgr_recv_bytes":2659014,"msgr_send_bytes":17352962,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.300907031,"msgr_running_send_time":0.971059666,"msgr_running_recv_time":10.083834440,"msgr_running_fast_dispatch_time":0.010008268,"msgr_send_messages_queue_lat":{"avgcount":16263,"sum":0.842656057,"avgtime":0.000051814},"msgr_handle_ack_lat":{"avgcount":12057,"sum":0.002773535,"avgtime":0.000000230},"msgr_recv_encrypted_bytes":119760,"msgr_send_encrypted_bytes":6800},"AsyncMessenger::Worker-2":{"msgr_recv_messages":127,"msgr_send_messages":132,"msgr_recv_bytes":37331,"msgr_send_bytes":3023961,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.026613212,"msgr_running_send_time":0.014758006,"msgr_running_recv_time":0.005606636,"msgr_running_fast_dispatch_time":0.004898901,"msgr_send_messages_queue_lat":{"avgcount":131,"sum":0.015380360,"avgtime":0.000117407},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":61488},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1691,"complete_latency":{"avgcount":84,"sum":2.789106672,"avgtime":0.033203650}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":541,"sum":0.288101715,"avgtime":0.000532535}},"mds":{"request":4754,"reply":4751,"reply_latency":{"avgcount":4751,"sum":6.487977873,"avgtime":0.001365602},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4605,"inodes_top":0,"inodes_bottom":4605,"inodes_pin_tail":0,"inodes_pinned":4605,"inodes_expired":0,"inodes_with_caps":2584,"caps":2584,"subtrees":2,"traverse":4754,"traverse_hit":4754,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":2149,"root_rbytes":7583744,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":4722,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2389,"handle_client_caps_dirty":2149,"handle_client_cap_release":134,"process_request_cap_release":1552},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1226,"num_strays_delayed":301,"num_strays_enqueuing":0,"strays_created":1543,"strays_enqueued":317,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":7588,"evex":0,"evtrm":0,"ev":7591,"evexg":0,"evexd":0,"segadd":8,"segex":0,"segtrm":0,"seg":10,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21251878,"rdpos":4195244,"jlat":{"avgcount":15,"sum":0.175036403,"avgtime":0.011669093},"replayed":3},"mds_mem":{"ino":3065,"ino+":3214,"ino-":149,"dir":13,"dir+":13,"dir-":0,"dn":4605,"dn+":4754,"dn-":149,"cap":2584,"cap+":3202,"cap-":618,"rss":179412,"heap":413093888},"mds_server":{"dispatch_client_request":4754,"dispatch_server_request":0,"handle_client_request":4754,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1543,"sum":3.106045650,"avgtime":0.002012991},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":4443,"average_load":4443,"avg_session_uptime":802,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5199665,"buffer_anon_items":5011,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15065621,"mds_co_items":209455,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":416,"op_send_bytes":17388006,"op_resend":0,"op_reply":416,"op_latency":{"avgcount":416,"sum":1.334971239,"avgtime":0.003209065},"op_inflight":0,"oplen_avg":{"avgcount":416,"sum":473},"op":416,"op_r":44,"op_w":372,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":16,"osdop_write":24,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":332,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":18,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":317,"pq_executed":317,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":251,"max":104857600,"get_started":0,"get":7730,"get_sum":2166509,"get_or_fail_fail":0,"get_or_fail_success":7730,"take":0,"take_sum":0,"put":7729,"put_sum":2166258,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":416,"take_sum":17420958,"put":416,"put_sum":17420958,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":416,"take_sum":416,"put":416,"put_sum":416,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":317,"get_sum":32017,"get_or_fail_fail":0,"get_or_fail_success":317,"take":0,"take_sum":0,"put":8,"put_sum":32017,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":6553,"max":3758096384,"get_started":0,"get":7588,"get_sum":17056375,"get_or_fail_fail":0,"get_or_fail_success":7588,"take":0,"take_sum":0,"put":16,"put_sum":17049822,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:33.208 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 52185, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 168, "msgr_running_fast_dispatch_time": 0.006891839, "msgr_running_recv_time": 0.011943509, "msgr_running_send_time": 0.023468546, "msgr_running_total_time": 0.046669082, "msgr_send_bytes": 4273200, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 168, "msgr_send_messages_queue_lat": { "avgcount": 167, "avgtime": 0.000144758, "sum": 0.024174677 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 12057, "avgtime": 2.3e-07, "sum": 0.002773535 }, "msgr_recv_bytes": 2659014, "msgr_recv_encrypted_bytes": 119760, "msgr_recv_messages": 7439, "msgr_running_fast_dispatch_time": 0.010008268, "msgr_running_recv_time": 10.08383444, "msgr_running_send_time": 0.971059666, "msgr_running_total_time": 1.300907031, "msgr_send_bytes": 17352962, "msgr_send_encrypted_bytes": 6800, "msgr_send_messages": 16264, "msgr_send_messages_queue_lat": { "avgcount": 16263, "avgtime": 5.1814e-05, "sum": 0.842656057 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 37331, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 127, "msgr_running_fast_dispatch_time": 0.004898901, "msgr_running_recv_time": 0.005606636, "msgr_running_send_time": 0.014758006, "msgr_running_total_time": 0.026613212, "msgr_send_bytes": 3023961, "msgr_send_encrypted_bytes": 61488, "msgr_send_messages": 132, "msgr_send_messages_queue_lat": { "avgcount": 131, "avgtime": 0.000117407, "sum": 0.01538036 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 84, "avgtime": 0.03320365, "sum": 2.789106672 }, "queue_len": 1691 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 541, "avgtime": 0.000532535, "sum": 0.288101715 }, "queue_len": 0 }, "mds": { "caps": 2584, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4722, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 134, "handle_client_caps": 2389, "handle_client_caps_dirty": 2149, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4605, "inodes_bottom": 4605, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4605, "inodes_top": 0, "inodes_with_caps": 2584, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1552, "q": 0, "reply": 4751, "reply_latency": { "avgcount": 4751, "avgtime": 0.001365602, "sum": 6.487977873 }, "request": 4754, "root_rbytes": 7583744, "root_rfiles": 2149, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 4754, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 4754, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1226, "num_strays_delayed": 301, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1543, "strays_enqueued": 317, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 7591, "evadd": 7588, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 15, "avgtime": 0.011669093, "sum": 0.175036403 }, "rdpos": 4195244, "replayed": 3, "seg": 10, "segadd": 8, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21251878 }, "mds_mem": { "cap": 2584, "cap+": 3202, "cap-": 618, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4605, "dn+": 4754, "dn-": 149, "heap": 413093888, "ino": 3065, "ino+": 3214, "ino-": 149, "rss": 179412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 4754, "dispatch_server_request": 0, "handle_client_request": 4754, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1543, "avgtime": 0.002012991, "sum": 3.10604565 } }, "mds_sessions": { "average_load": 4443, "avg_session_uptime": 802, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 4443 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5199665, "buffer_anon_items": 5011, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15065621, "mds_co_items": 209455, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 3, "omap_rd": 48, "omap_wr": 18, "op": 416, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 416, "avgtime": 0.003209065, "sum": 1.334971239 }, "op_pg": 0, "op_r": 44, "op_reply": 416, "op_resend": 0, "op_rmw": 0, "op_send": 416, "op_send_bytes": 17388006, "op_w": 372, "oplen_avg": { "avgcount": 416, "sum": 473 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 332, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 16, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 24, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 317, "pq_executed_ops": 317, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 7730, "get_or_fail_fail": 0, "get_or_fail_success": 7730, "get_started": 0, "get_sum": 2166509, "max": 104857600, "put": 7729, "put_sum": 2166258, "take": 0, "take_sum": 0, "val": 251, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 416, "put_sum": 17420958, "take": 416, "take_sum": 17420958, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 416, "put_sum": 416, "take": 416, "take_sum": 416, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 317, "get_or_fail_fail": 0, "get_or_fail_success": 317, "get_started": 0, "get_sum": 32017, "max": 3758096384, "put": 8, "put_sum": 32017, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 7588, "get_or_fail_fail": 0, "get_or_fail_success": 7588, "get_started": 0, "get_sum": 17056375, "max": 3758096384, "put": 16, "put_sum": 17049822, "take": 0, "take_sum": 0, "val": 6553, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:33.209 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1226, 317/1543 2024-09-17T03:54:34.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:34.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 -- 172.21.15.129:0/1082697702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8415cf70 msgr2=0x7fcd84072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:34.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/1082697702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8415cf70 0x7fcd84072d70 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009830 tx=0x7fcd6c02a530 comp rx=0 tx=0).stop 2024-09-17T03:54:34.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 -- 172.21.15.129:0/1082697702 shutdown_connections 2024-09-17T03:54:34.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/1082697702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd8415cf70 0x7fcd84072d70 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 -- 172.21.15.129:0/1082697702 >> 172.21.15.129:0/1082697702 conn(0x7fcd840b23c0 msgr2=0x7fcd840b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:34.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 -- 172.21.15.129:0/1082697702 shutdown_connections 2024-09-17T03:54:34.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.447+0000 7fcd8aa00640 1 -- 172.21.15.129:0/1082697702 wait complete. 2024-09-17T03:54:34.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.450+0000 7fcd8aa00640 1 Processor -- start 2024-09-17T03:54:34.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.450+0000 7fcd8aa00640 1 -- start start 2024-09-17T03:54:34.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:34.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd840cd190 0x7fcd8413ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd8415cf70 0x7fcd840ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcd84070060 con 0x7fcd840b59d0 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcd8406fee0 con 0x7fcd8415cf70 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.451+0000 7fcd8aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcd840701e0 con 0x7fcd840cd190 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49536/0 (socket says 172.21.15.129:49536) 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 -- 172.21.15.129:0/2551860658 learned_addr learned my addr 172.21.15.129:0/2551860658 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd7be00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd840cd190 0x7fcd8413ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd88a00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd8415cf70 0x7fcd840ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 -- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd840cd190 msgr2=0x7fcd8413ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd840cd190 0x7fcd8413ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 -- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd8415cf70 msgr2=0x7fcd840ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:34.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd8415cf70 0x7fcd840ac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.452+0000 7fcd83e00640 1 -- 172.21.15.129:0/2551860658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd841f3900 con 0x7fcd840b59d0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd83e00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c370 tx=0x7fcd7400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74017070 con 0x7fcd840b59d0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcd740035f0 con 0x7fcd840b59d0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd841f3be0 con 0x7fcd840b59d0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd74003a40 con 0x7fcd840b59d0 2024-09-17T03:54:34.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.453+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd841f41a0 con 0x7fcd840b59d0 2024-09-17T03:54:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.454+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd8415e0d0 con 0x7fcd840b59d0 2024-09-17T03:54:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.456+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcd7401a070 con 0x7fcd840b59d0 2024-09-17T03:54:34.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.457+0000 7fcd80c00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 0x7fcd6007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.457+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fcd74099ef0 con 0x7fcd840b59d0 2024-09-17T03:54:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.458+0000 7fcd7be00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 0x7fcd6007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:34.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.458+0000 7fcd7be00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 0x7fcd6007b0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c034000 tx=0x7fcd6c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:34.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.462+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcd74061d10 con 0x7fcd840b59d0 2024-09-17T03:54:34.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.722+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd840aa4e0 con 0x7fcd840b59d0 2024-09-17T03:54:34.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.723+0000 7fcd80c00640 1 -- 172.21.15.129:0/2551860658 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7fcd74066740 con 0x7fcd840b59d0 2024-09-17T03:54:34.723 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:34.723 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:34.724 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 msgr2=0x7fcd6007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 0x7fcd6007b0d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c034000 tx=0x7fcd6c004730 comp rx=0 tx=0).stop 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 msgr2=0x7fcd841f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c370 tx=0x7fcd7400c830 comp rx=0 tx=0).stop 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 shutdown_connections 2024-09-17T03:54:34.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcd60078bc0 0x7fcd6007b0d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcd8415cf70 0x7fcd840ac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcd840cd190 0x7fcd8413ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:34.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 --2- 172.21.15.129:0/2551860658 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcd840b59d0 0x7fcd841f7470 secure :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c370 tx=0x7fcd7400c830 comp rx=0 tx=0).stop 2024-09-17T03:54:34.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.728+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 >> 172.21.15.129:0/2551860658 conn(0x7fcd840b23c0 msgr2=0x7fcd841604c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.729+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 shutdown_connections 2024-09-17T03:54:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:34.729+0000 7fcd8aa00640 1 -- 172.21.15.129:0/2551860658 wait complete. 2024-09-17T03:54:34.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:35.152 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":364,"msgr_send_messages":364,"msgr_recv_bytes":101961,"msgr_send_bytes":4337088,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.070650754,"msgr_running_send_time":0.035833361,"msgr_running_recv_time":0.017257410,"msgr_running_fast_dispatch_time":0.011773768,"msgr_send_messages_queue_lat":{"avgcount":363,"sum":0.038686468,"avgtime":0.000106574},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8884,"msgr_send_messages":18694,"msgr_recv_bytes":3182391,"msgr_send_bytes":22233904,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.544243501,"msgr_running_send_time":1.147574291,"msgr_running_recv_time":10.164988715,"msgr_running_fast_dispatch_time":0.017168260,"msgr_send_messages_queue_lat":{"avgcount":18688,"sum":1.084266789,"avgtime":0.000058019},"msgr_handle_ack_lat":{"avgcount":13832,"sum":0.003329116,"avgtime":0.000000240},"msgr_recv_encrypted_bytes":120032,"msgr_send_encrypted_bytes":7440},"AsyncMessenger::Worker-2":{"msgr_recv_messages":276,"msgr_send_messages":282,"msgr_recv_bytes":75161,"msgr_send_bytes":3074539,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.041663206,"msgr_running_send_time":0.021447271,"msgr_running_recv_time":0.009075676,"msgr_running_fast_dispatch_time":0.008637487,"msgr_send_messages_queue_lat":{"avgcount":281,"sum":0.026654181,"avgtime":0.000094854},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":63328},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1010,"complete_latency":{"avgcount":95,"sum":4.677788410,"avgtime":0.049239878}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1363,"sum":0.700432825,"avgtime":0.000513890}},"mds":{"request":5584,"reply":5582,"reply_latency":{"avgcount":5582,"sum":8.057663014,"avgtime":0.001443508},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4967,"inodes_top":0,"inodes_bottom":4967,"inodes_pin_tail":0,"inodes_pinned":4967,"inodes_expired":0,"inodes_with_caps":2273,"caps":2273,"subtrees":2,"traverse":5584,"traverse_hit":5584,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1320,"root_rbytes":4855808,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":5297,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2853,"handle_client_caps_dirty":2594,"handle_client_cap_release":136,"process_request_cap_release":2382},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1586,"num_strays_delayed":128,"num_strays_enqueuing":0,"strays_created":2374,"strays_enqueued":788,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9499,"evex":0,"evtrm":0,"ev":9502,"evexg":0,"evexd":0,"segadd":10,"segex":0,"segtrm":0,"seg":12,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":25434028,"rdpos":4195244,"jlat":{"avgcount":19,"sum":0.832457747,"avgtime":0.043813565},"replayed":3},"mds_mem":{"ino":2596,"ino+":3214,"ino-":618,"dir":13,"dir+":13,"dir-":0,"dn":4967,"dn+":5585,"dn-":618,"cap":2273,"cap+":3202,"cap-":929,"rss":179412,"heap":413093888},"mds_server":{"dispatch_client_request":5584,"dispatch_server_request":0,"handle_client_request":5584,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2374,"sum":4.675730791,"avgtime":0.001969558},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5175,"average_load":5175,"avg_session_uptime":802,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5199665,"buffer_anon_items":5011,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15065621,"mds_co_items":209455,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":907,"op_send_bytes":21422574,"op_resend":0,"op_reply":907,"op_latency":{"avgcount":907,"sum":2.527945535,"avgtime":0.002787150},"op_inflight":0,"oplen_avg":{"avgcount":907,"sum":968},"op":907,"op_r":50,"op_w":857,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":22,"osdop_write":33,"osdop_writefull":7,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":804,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":22,"omap_rd":48,"omap_del":5},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":788,"pq_executed":788,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":9520,"get_sum":2637108,"get_or_fail_fail":0,"get_or_fail_success":9520,"take":0,"take_sum":0,"put":9520,"put_sum":2637108,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":907,"take_sum":21503097,"put":907,"put_sum":21503097,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":907,"take_sum":907,"put":907,"put_sum":907,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":788,"get_sum":79588,"get_or_fail_fail":0,"get_or_fail_success":788,"take":0,"take_sum":0,"put":14,"put_sum":79588,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":268204,"max":3758096384,"get_started":0,"get":9499,"get_sum":21238525,"get_or_fail_fail":0,"get_or_fail_success":9499,"take":0,"take_sum":0,"put":19,"put_sum":20970321,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:35.153 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 101961, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 364, "msgr_running_fast_dispatch_time": 0.011773768, "msgr_running_recv_time": 0.01725741, "msgr_running_send_time": 0.035833361, "msgr_running_total_time": 0.070650754, "msgr_send_bytes": 4337088, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 364, "msgr_send_messages_queue_lat": { "avgcount": 363, "avgtime": 0.000106574, "sum": 0.038686468 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13832, "avgtime": 2.4e-07, "sum": 0.003329116 }, "msgr_recv_bytes": 3182391, "msgr_recv_encrypted_bytes": 120032, "msgr_recv_messages": 8884, "msgr_running_fast_dispatch_time": 0.01716826, "msgr_running_recv_time": 10.164988715, "msgr_running_send_time": 1.147574291, "msgr_running_total_time": 1.544243501, "msgr_send_bytes": 22233904, "msgr_send_encrypted_bytes": 7440, "msgr_send_messages": 18694, "msgr_send_messages_queue_lat": { "avgcount": 18688, "avgtime": 5.8019e-05, "sum": 1.084266789 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 75161, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 276, "msgr_running_fast_dispatch_time": 0.008637487, "msgr_running_recv_time": 0.009075676, "msgr_running_send_time": 0.021447271, "msgr_running_total_time": 0.041663206, "msgr_send_bytes": 3074539, "msgr_send_encrypted_bytes": 63328, "msgr_send_messages": 282, "msgr_send_messages_queue_lat": { "avgcount": 281, "avgtime": 9.4854e-05, "sum": 0.026654181 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 95, "avgtime": 0.049239878, "sum": 4.67778841 }, "queue_len": 1010 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1363, "avgtime": 0.00051389, "sum": 0.700432825 }, "queue_len": 0 }, "mds": { "caps": 2273, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 5297, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 136, "handle_client_caps": 2853, "handle_client_caps_dirty": 2594, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4967, "inodes_bottom": 4967, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4967, "inodes_top": 0, "inodes_with_caps": 2273, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2382, "q": 0, "reply": 5582, "reply_latency": { "avgcount": 5582, "avgtime": 0.001443508, "sum": 8.057663014 }, "request": 5584, "root_rbytes": 4855808, "root_rfiles": 1320, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5584, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5584, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1586, "num_strays_delayed": 128, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2374, "strays_enqueued": 788, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 9502, "evadd": 9499, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 19, "avgtime": 0.043813565, "sum": 0.832457747 }, "rdpos": 4195244, "replayed": 3, "seg": 12, "segadd": 10, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 25434028 }, "mds_mem": { "cap": 2273, "cap+": 3202, "cap-": 929, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4967, "dn+": 5585, "dn-": 618, "heap": 413093888, "ino": 2596, "ino+": 3214, "ino-": 618, "rss": 179412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5584, "dispatch_server_request": 0, "handle_client_request": 5584, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2374, "avgtime": 0.001969558, "sum": 4.675730791 } }, "mds_sessions": { "average_load": 5175, "avg_session_uptime": 802, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5175 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5199665, "buffer_anon_items": 5011, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15065621, "mds_co_items": 209455, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 5, "omap_rd": 48, "omap_wr": 22, "op": 907, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 907, "avgtime": 0.00278715, "sum": 2.527945535 }, "op_pg": 0, "op_r": 50, "op_reply": 907, "op_resend": 0, "op_rmw": 0, "op_send": 907, "op_send_bytes": 21422574, "op_w": 857, "oplen_avg": { "avgcount": 907, "sum": 968 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 804, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 22, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 33, "osdop_writefull": 7, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 788, "pq_executed_ops": 788, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 9520, "get_or_fail_fail": 0, "get_or_fail_success": 9520, "get_started": 0, "get_sum": 2637108, "max": 104857600, "put": 9520, "put_sum": 2637108, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 907, "put_sum": 21503097, "take": 907, "take_sum": 21503097, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 907, "put_sum": 907, "take": 907, "take_sum": 907, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 788, "get_or_fail_fail": 0, "get_or_fail_success": 788, "get_started": 0, "get_sum": 79588, "max": 3758096384, "put": 14, "put_sum": 79588, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 9499, "get_or_fail_fail": 0, "get_or_fail_success": 9499, "get_started": 0, "get_sum": 21238525, "max": 3758096384, "put": 19, "put_sum": 20970321, "take": 0, "take_sum": 0, "val": 268204, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:35.153 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1586, 788/2374 2024-09-17T03:54:36.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 -- 172.21.15.129:0/1722695938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c0cf2e0 msgr2=0x7f0f4c0d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1722695938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c0cf2e0 0x7f0f4c0d2290 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c005fd0 tx=0x7f0f3c02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 -- 172.21.15.129:0/1722695938 shutdown_connections 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1722695938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c06d270 0x7f0f4c06d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1722695938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c0cf2e0 0x7f0f4c0d2290 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 -- 172.21.15.129:0/1722695938 >> 172.21.15.129:0/1722695938 conn(0x7f0f4c0d1770 msgr2=0x7f0f4c0d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 -- 172.21.15.129:0/1722695938 shutdown_connections 2024-09-17T03:54:36.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.378+0000 7f0f50c00640 1 -- 172.21.15.129:0/1722695938 wait complete. 2024-09-17T03:54:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.381+0000 7f0f50c00640 1 Processor -- start 2024-09-17T03:54:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.381+0000 7f0f50c00640 1 -- start start 2024-09-17T03:54:36.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c0cf2e0 0x7f0f4c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f4c15ea40 0x7f0f4c0ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c0d3be0 con 0x7f0f4c06d270 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0f4c0d3a60 con 0x7f0f4c15ea40 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.382+0000 7f0f50c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0f4c0d3d60 con 0x7f0f4c0cf2e0 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49546/0 (socket says 172.21.15.129:49546) 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 -- 172.21.15.129:0/1499137879 learned_addr learned my addr 172.21.15.129:0/1499137879 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4aa00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c0cf2e0 0x7f0f4c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4be00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f4c15ea40 0x7f0f4c0ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 -- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c0cf2e0 msgr2=0x7f0f4c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c0cf2e0 0x7f0f4c15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 -- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f4c15ea40 msgr2=0x7f0f4c0ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f4c15ea40 0x7f0f4c0ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 -- 172.21.15.129:0/1499137879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f4c1f6e50 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.383+0000 7f0f4b400640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c034040 tx=0x7f0f3c0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.384+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c03e030 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.384+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f3c003ae0 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.384+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f4c1fa3d0 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.384+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c035040 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.384+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f4c1f6370 con 0x7f0f4c06d270 2024-09-17T03:54:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.385+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f4c0d4240 con 0x7f0f4c06d270 2024-09-17T03:54:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.386+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0f3c041030 con 0x7f0f4c06d270 2024-09-17T03:54:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.387+0000 7f0f43400640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 0x7f0f2c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.387+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f0f3c0bb290 con 0x7f0f4c06d270 2024-09-17T03:54:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.387+0000 7f0f4aa00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 0x7f0f2c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:36.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.388+0000 7f0f4aa00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 0x7f0f2c07b0a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c15c940 tx=0x7f0f38009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.391+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0f3c083010 con 0x7f0f4c06d270 2024-09-17T03:54:36.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.622+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f4c0c7e60 con 0x7f0f4c06d270 2024-09-17T03:54:36.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.623+0000 7f0f43400640 1 -- 172.21.15.129:0/1499137879 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f0f3c087a40 con 0x7f0f4c06d270 2024-09-17T03:54:36.623 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:36.623 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:36.624 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 msgr2=0x7f0f2c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 0x7f0f2c07b0a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c15c940 tx=0x7f0f38009040 comp rx=0 tx=0).stop 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 msgr2=0x7f0f4c0cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c034040 tx=0x7f0f3c0035f0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 shutdown_connections 2024-09-17T03:54:36.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f2c078b90 0x7f0f2c07b0a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f4c15ea40 0x7f0f4c0ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f4c0cf2e0 0x7f0f4c15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 --2- 172.21.15.129:0/1499137879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f4c06d270 0x7f0f4c0cc120 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 >> 172.21.15.129:0/1499137879 conn(0x7f0f4c0d1770 msgr2=0x7f0f4c075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 shutdown_connections 2024-09-17T03:54:36.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:36.626+0000 7f0f50c00640 1 -- 172.21.15.129:0/1499137879 wait complete. 2024-09-17T03:54:36.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:36.900 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":442,"msgr_send_messages":442,"msgr_recv_bytes":121773,"msgr_send_bytes":4362516,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.081303063,"msgr_running_send_time":0.040778070,"msgr_running_recv_time":0.019946862,"msgr_running_fast_dispatch_time":0.014212722,"msgr_send_messages_queue_lat":{"avgcount":441,"sum":0.044378730,"avgtime":0.000100632},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10532,"msgr_send_messages":20250,"msgr_recv_bytes":3780771,"msgr_send_bytes":22739282,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.679340886,"msgr_running_send_time":1.221234875,"msgr_running_recv_time":47.406164032,"msgr_running_fast_dispatch_time":0.019294356,"msgr_send_messages_queue_lat":{"avgcount":20249,"sum":1.539842310,"avgtime":0.000076045},"msgr_handle_ack_lat":{"avgcount":13834,"sum":0.003358172,"avgtime":0.000000242},"msgr_recv_encrypted_bytes":120032,"msgr_send_encrypted_bytes":7440},"AsyncMessenger::Worker-2":{"msgr_recv_messages":334,"msgr_send_messages":340,"msgr_recv_bytes":89877,"msgr_send_bytes":3093611,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.048947824,"msgr_running_send_time":0.024358928,"msgr_running_recv_time":0.010660225,"msgr_running_fast_dispatch_time":0.010946059,"msgr_send_messages_queue_lat":{"avgcount":339,"sum":0.030973264,"avgtime":0.000091366},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":63328},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":98,"sum":5.675844966,"avgtime":0.057916785}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1706,"sum":0.827770526,"avgtime":0.000485211}},"mds":{"request":5585,"reply":5583,"reply_latency":{"avgcount":5583,"sum":8.248641438,"avgtime":0.001477456},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4798,"inodes_top":0,"inodes_bottom":4798,"inodes_pin_tail":0,"inodes_pinned":4798,"inodes_expired":0,"inodes_with_caps":1887,"caps":1887,"subtrees":2,"traverse":5585,"traverse_hit":5585,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1043,"root_rbytes":3853312,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":6212,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4417,"handle_client_caps_dirty":2991,"handle_client_cap_release":148,"process_request_cap_release":2383},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1396,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":2375,"strays_enqueued":979,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":11106,"evex":0,"evtrm":0,"ev":11109,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28401824,"rdpos":4195244,"jlat":{"avgcount":19,"sum":0.832457747,"avgtime":0.043813565},"replayed":3},"mds_mem":{"ino":2426,"ino+":3214,"ino-":788,"dir":13,"dir+":13,"dir-":0,"dn":4798,"dn+":5586,"dn-":788,"cap":1253,"cap+":3202,"cap-":1949,"rss":179412,"heap":413093888},"mds_server":{"dispatch_client_request":5585,"dispatch_server_request":0,"handle_client_request":5585,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2375,"sum":4.866709215,"avgtime":0.002049140},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5151,"average_load":5151,"avg_session_uptime":802,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":5199665,"buffer_anon_items":5011,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":15065621,"mds_co_items":209455,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1114,"op_send_bytes":21487928,"op_resend":0,"op_reply":1114,"op_latency":{"avgcount":1114,"sum":3.024934825,"avgtime":0.002715381},"op_inflight":0,"oplen_avg":{"avgcount":1114,"sum":1179},"op":1114,"op_r":56,"op_w":1058,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":28,"osdop_write":39,"osdop_writefull":9,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":995,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":26,"omap_rd":48,"omap_del":7},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":979,"pq_executed":979,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11304,"get_sum":3134688,"get_or_fail_fail":0,"get_or_fail_success":11304,"take":0,"take_sum":0,"put":11304,"put_sum":3134688,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1114,"take_sum":21587742,"put":1114,"put_sum":21587742,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1114,"take_sum":1114,"put":1114,"put_sum":1114,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":979,"get_sum":98879,"get_or_fail_fail":0,"get_or_fail_success":979,"take":0,"take_sum":0,"put":20,"put_sum":98879,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":3236000,"max":3758096384,"get_started":0,"get":11106,"get_sum":24206321,"get_or_fail_fail":0,"get_or_fail_success":11106,"take":0,"take_sum":0,"put":19,"put_sum":20970321,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:36.903 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 121773, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 442, "msgr_running_fast_dispatch_time": 0.014212722, "msgr_running_recv_time": 0.019946862, "msgr_running_send_time": 0.04077807, "msgr_running_total_time": 0.081303063, "msgr_send_bytes": 4362516, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 442, "msgr_send_messages_queue_lat": { "avgcount": 441, "avgtime": 0.000100632, "sum": 0.04437873 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13834, "avgtime": 2.42e-07, "sum": 0.003358172 }, "msgr_recv_bytes": 3780771, "msgr_recv_encrypted_bytes": 120032, "msgr_recv_messages": 10532, "msgr_running_fast_dispatch_time": 0.019294356, "msgr_running_recv_time": 47.406164032, "msgr_running_send_time": 1.221234875, "msgr_running_total_time": 1.679340886, "msgr_send_bytes": 22739282, "msgr_send_encrypted_bytes": 7440, "msgr_send_messages": 20250, "msgr_send_messages_queue_lat": { "avgcount": 20249, "avgtime": 7.6045e-05, "sum": 1.53984231 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 89877, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 334, "msgr_running_fast_dispatch_time": 0.010946059, "msgr_running_recv_time": 0.010660225, "msgr_running_send_time": 0.024358928, "msgr_running_total_time": 0.048947824, "msgr_send_bytes": 3093611, "msgr_send_encrypted_bytes": 63328, "msgr_send_messages": 340, "msgr_send_messages_queue_lat": { "avgcount": 339, "avgtime": 9.1366e-05, "sum": 0.030973264 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 98, "avgtime": 0.057916785, "sum": 5.675844966 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1706, "avgtime": 0.000485211, "sum": 0.827770526 }, "queue_len": 0 }, "mds": { "caps": 1887, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6212, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 148, "handle_client_caps": 4417, "handle_client_caps_dirty": 2991, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4798, "inodes_bottom": 4798, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4798, "inodes_top": 0, "inodes_with_caps": 1887, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2383, "q": 0, "reply": 5583, "reply_latency": { "avgcount": 5583, "avgtime": 0.001477456, "sum": 8.248641438 }, "request": 5585, "root_rbytes": 3853312, "root_rfiles": 1043, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5585, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5585, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1396, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2375, "strays_enqueued": 979, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 11109, "evadd": 11106, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 19, "avgtime": 0.043813565, "sum": 0.832457747 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28401824 }, "mds_mem": { "cap": 1253, "cap+": 3202, "cap-": 1949, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4798, "dn+": 5586, "dn-": 788, "heap": 413093888, "ino": 2426, "ino+": 3214, "ino-": 788, "rss": 179412 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5585, "dispatch_server_request": 0, "handle_client_request": 5585, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2375, "avgtime": 0.00204914, "sum": 4.866709215 } }, "mds_sessions": { "average_load": 5151, "avg_session_uptime": 802, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5151 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 5199665, "buffer_anon_items": 5011, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 15065621, "mds_co_items": 209455, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 7, "omap_rd": 48, "omap_wr": 26, "op": 1114, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1114, "avgtime": 0.002715381, "sum": 3.024934825 }, "op_pg": 0, "op_r": 56, "op_reply": 1114, "op_resend": 0, "op_rmw": 0, "op_send": 1114, "op_send_bytes": 21487928, "op_w": 1058, "oplen_avg": { "avgcount": 1114, "sum": 1179 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 995, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 39, "osdop_writefull": 9, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 979, "pq_executed_ops": 979, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11304, "get_or_fail_fail": 0, "get_or_fail_success": 11304, "get_started": 0, "get_sum": 3134688, "max": 104857600, "put": 11304, "put_sum": 3134688, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1114, "put_sum": 21587742, "take": 1114, "take_sum": 21587742, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1114, "put_sum": 1114, "take": 1114, "take_sum": 1114, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 979, "get_or_fail_fail": 0, "get_or_fail_success": 979, "get_started": 0, "get_sum": 98879, "max": 3758096384, "put": 20, "put_sum": 98879, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 11106, "get_or_fail_fail": 0, "get_or_fail_success": 11106, "get_started": 0, "get_sum": 24206321, "max": 3758096384, "put": 19, "put_sum": 20970321, "take": 0, "take_sum": 0, "val": 3236000, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:36.903 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1396, 979/2375 2024-09-17T03:54:37.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:38.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 -- 172.21.15.129:0/3130817592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 msgr2=0x7f2640142370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 --2- 172.21.15.129:0/3130817592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f2640142370 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2638005fd0 tx=0x7f263802aa90 comp rx=0 tx=0).stop 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 -- 172.21.15.129:0/3130817592 shutdown_connections 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 --2- 172.21.15.129:0/3130817592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f2640142370 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 --2- 172.21.15.129:0/3130817592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 0x7f264006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 -- 172.21.15.129:0/3130817592 >> 172.21.15.129:0/3130817592 conn(0x7f264006d560 msgr2=0x7f26400b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 -- 172.21.15.129:0/3130817592 shutdown_connections 2024-09-17T03:54:38.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.129+0000 7f263f200640 1 -- 172.21.15.129:0/3130817592 wait complete. 2024-09-17T03:54:38.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.132+0000 7f263f200640 1 Processor -- start 2024-09-17T03:54:38.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.132+0000 7f263f200640 1 -- start start 2024-09-17T03:54:38.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f26400ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 0x7f26401f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2640074ec0 con 0x7f264015da40 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2640074d40 con 0x7f2640141f60 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2640075040 con 0x7f2640156f90 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.133+0000 7f263e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49550/0 (socket says 172.21.15.129:49550) 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 -- 172.21.15.129:0/2153836442 learned_addr learned my addr 172.21.15.129:0/2153836442 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f2635400640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 0x7f26401f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:38.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263de00640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f26400ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 -- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 msgr2=0x7f26401f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 0x7f26401f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 -- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 msgr2=0x7f26400ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f26400ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.134+0000 7f263e800640 1 -- 172.21.15.129:0/2153836442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26400d0890 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f263e800640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2638009d90 tx=0x7f2638003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2638044070 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2638003e00 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26400d0b70 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26380044b0 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.135+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26400d03e0 con 0x7f264015da40 2024-09-17T03:54:38.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.136+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26401e3910 con 0x7f264015da40 2024-09-17T03:54:38.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.138+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2638047070 con 0x7f264015da40 2024-09-17T03:54:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.139+0000 7f2636a00640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 0x7f261c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.139+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f26380bbba0 con 0x7f264015da40 2024-09-17T03:54:38.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.140+0000 7f263de00640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 0x7f261c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:38.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.140+0000 7f263de00640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 0x7f261c07b0a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2630001720 tx=0x7f2630009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:38.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.144+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2638032030 con 0x7f264015da40 2024-09-17T03:54:38.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.372+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2640075560 con 0x7f264015da40 2024-09-17T03:54:38.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.373+0000 7f2636a00640 1 -- 172.21.15.129:0/2153836442 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f26380098b0 con 0x7f264015da40 2024-09-17T03:54:38.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:38.373 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:38.373 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 msgr2=0x7f261c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 0x7f261c07b0a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2630001720 tx=0x7f2630009040 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 msgr2=0x7f26401f3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f2638009d90 tx=0x7f2638003c10 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 shutdown_connections 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f261c078b90 0x7f261c07b0a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f264015da40 0x7f26401f3d60 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2640156f90 0x7f26401f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.375+0000 7f263f200640 1 --2- 172.21.15.129:0/2153836442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2640141f60 0x7f26400ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.376+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 >> 172.21.15.129:0/2153836442 conn(0x7f264006d560 msgr2=0x7f26400b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:38.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.376+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 shutdown_connections 2024-09-17T03:54:38.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:38.376+0000 7f263f200640 1 -- 172.21.15.129:0/2153836442 wait complete. 2024-09-17T03:54:38.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:38.646 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":442,"msgr_send_messages":442,"msgr_recv_bytes":121773,"msgr_send_bytes":4362516,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.081303063,"msgr_running_send_time":0.040778070,"msgr_running_recv_time":0.019946862,"msgr_running_fast_dispatch_time":0.014212722,"msgr_send_messages_queue_lat":{"avgcount":441,"sum":0.044378730,"avgtime":0.000100632},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10534,"msgr_send_messages":20841,"msgr_recv_bytes":3781377,"msgr_send_bytes":22931020,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.703154389,"msgr_running_send_time":1.241909830,"msgr_running_recv_time":47.406236151,"msgr_running_fast_dispatch_time":0.019330881,"msgr_send_messages_queue_lat":{"avgcount":20840,"sum":1.553441697,"avgtime":0.000074541},"msgr_handle_ack_lat":{"avgcount":13834,"sum":0.003358172,"avgtime":0.000000242},"msgr_recv_encrypted_bytes":120304,"msgr_send_encrypted_bytes":8080},"AsyncMessenger::Worker-2":{"msgr_recv_messages":337,"msgr_send_messages":344,"msgr_recv_bytes":90615,"msgr_send_bytes":3163683,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.060957859,"msgr_running_send_time":0.034729114,"msgr_running_recv_time":0.010744837,"msgr_running_fast_dispatch_time":0.012471469,"msgr_send_messages_queue_lat":{"avgcount":343,"sum":0.046694614,"avgtime":0.000136135},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":65168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":103,"sum":5.942379223,"avgtime":0.057693002}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1706,"sum":0.827770526,"avgtime":0.000485211}},"mds":{"request":5585,"reply":5583,"reply_latency":{"avgcount":5583,"sum":8.248641438,"avgtime":0.001477456},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4798,"inodes_top":0,"inodes_bottom":4798,"inodes_pin_tail":0,"inodes_pinned":4798,"inodes_expired":0,"inodes_with_caps":1253,"caps":1253,"subtrees":2,"traverse":5585,"traverse_hit":5585,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1043,"root_rbytes":3853312,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":6322,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4417,"handle_client_caps_dirty":2991,"handle_client_cap_release":148,"process_request_cap_release":2383},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":1396,"num_strays_delayed":970,"num_strays_enqueuing":0,"strays_created":2375,"strays_enqueued":979,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":11107,"evex":0,"evtrm":0,"ev":11110,"evexg":0,"evexd":0,"segadd":11,"segex":0,"segtrm":0,"seg":13,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28407066,"rdpos":4195244,"jlat":{"avgcount":21,"sum":0.889199016,"avgtime":0.042342810},"replayed":3},"mds_mem":{"ino":2426,"ino+":3214,"ino-":788,"dir":13,"dir+":13,"dir-":0,"dn":4798,"dn+":5586,"dn-":788,"cap":1253,"cap+":3202,"cap-":1949,"rss":177216,"heap":413093888},"mds_server":{"dispatch_client_request":5585,"dispatch_server_request":0,"handle_client_request":5585,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2375,"sum":4.866709215,"avgtime":0.002049140},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5151,"average_load":5151,"avg_session_uptime":807,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":26164311,"buffer_anon_items":8979,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13145938,"mds_co_items":217770,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1118,"op_send_bytes":24732041,"op_resend":0,"op_reply":1118,"op_latency":{"avgcount":1118,"sum":3.090933403,"avgtime":0.002764698},"op_inflight":0,"oplen_avg":{"avgcount":1118,"sum":1185},"op":1118,"op_r":56,"op_w":1062,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":28,"osdop_write":41,"osdop_writefull":10,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":995,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":28,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":979,"pq_executed":979,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":11309,"get_sum":3135530,"get_or_fail_fail":0,"get_or_fail_success":11309,"take":0,"take_sum":0,"put":11309,"put_sum":3135530,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1118,"take_sum":24831855,"put":1118,"put_sum":24831855,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1118,"take_sum":1118,"put":1118,"put_sum":1118,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":979,"get_sum":98879,"get_or_fail_fail":0,"get_or_fail_success":979,"take":0,"take_sum":0,"put":20,"put_sum":98879,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":0,"max":3758096384,"get_started":0,"get":11107,"get_sum":24211563,"get_or_fail_fail":0,"get_or_fail_success":11107,"take":0,"take_sum":0,"put":21,"put_sum":24211563,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:38.648 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 121773, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 442, "msgr_running_fast_dispatch_time": 0.014212722, "msgr_running_recv_time": 0.019946862, "msgr_running_send_time": 0.04077807, "msgr_running_total_time": 0.081303063, "msgr_send_bytes": 4362516, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 442, "msgr_send_messages_queue_lat": { "avgcount": 441, "avgtime": 0.000100632, "sum": 0.04437873 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13834, "avgtime": 2.42e-07, "sum": 0.003358172 }, "msgr_recv_bytes": 3781377, "msgr_recv_encrypted_bytes": 120304, "msgr_recv_messages": 10534, "msgr_running_fast_dispatch_time": 0.019330881, "msgr_running_recv_time": 47.406236151, "msgr_running_send_time": 1.24190983, "msgr_running_total_time": 1.703154389, "msgr_send_bytes": 22931020, "msgr_send_encrypted_bytes": 8080, "msgr_send_messages": 20841, "msgr_send_messages_queue_lat": { "avgcount": 20840, "avgtime": 7.4541e-05, "sum": 1.553441697 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 90615, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 337, "msgr_running_fast_dispatch_time": 0.012471469, "msgr_running_recv_time": 0.010744837, "msgr_running_send_time": 0.034729114, "msgr_running_total_time": 0.060957859, "msgr_send_bytes": 3163683, "msgr_send_encrypted_bytes": 65168, "msgr_send_messages": 344, "msgr_send_messages_queue_lat": { "avgcount": 343, "avgtime": 0.000136135, "sum": 0.046694614 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 103, "avgtime": 0.057693002, "sum": 5.942379223 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1706, "avgtime": 0.000485211, "sum": 0.827770526 }, "queue_len": 0 }, "mds": { "caps": 1253, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6322, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 148, "handle_client_caps": 4417, "handle_client_caps_dirty": 2991, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4798, "inodes_bottom": 4798, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4798, "inodes_top": 0, "inodes_with_caps": 1253, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2383, "q": 0, "reply": 5583, "reply_latency": { "avgcount": 5583, "avgtime": 0.001477456, "sum": 8.248641438 }, "request": 5585, "root_rbytes": 3853312, "root_rfiles": 1043, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5585, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5585, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 1396, "num_strays_delayed": 970, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2375, "strays_enqueued": 979, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 11110, "evadd": 11107, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 21, "avgtime": 0.04234281, "sum": 0.889199016 }, "rdpos": 4195244, "replayed": 3, "seg": 13, "segadd": 11, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28407066 }, "mds_mem": { "cap": 1253, "cap+": 3202, "cap-": 1949, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4798, "dn+": 5586, "dn-": 788, "heap": 413093888, "ino": 2426, "ino+": 3214, "ino-": 788, "rss": 177216 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5585, "dispatch_server_request": 0, "handle_client_request": 5585, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2375, "avgtime": 0.00204914, "sum": 4.866709215 } }, "mds_sessions": { "average_load": 5151, "avg_session_uptime": 807, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5151 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 26164311, "buffer_anon_items": 8979, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13145938, "mds_co_items": 217770, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 28, "op": 1118, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1118, "avgtime": 0.002764698, "sum": 3.090933403 }, "op_pg": 0, "op_r": 56, "op_reply": 1118, "op_resend": 0, "op_rmw": 0, "op_send": 1118, "op_send_bytes": 24732041, "op_w": 1062, "oplen_avg": { "avgcount": 1118, "sum": 1185 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 995, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 41, "osdop_writefull": 10, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 979, "pq_executed_ops": 979, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 11309, "get_or_fail_fail": 0, "get_or_fail_success": 11309, "get_started": 0, "get_sum": 3135530, "max": 104857600, "put": 11309, "put_sum": 3135530, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1118, "put_sum": 24831855, "take": 1118, "take_sum": 24831855, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1118, "put_sum": 1118, "take": 1118, "take_sum": 1118, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 979, "get_or_fail_fail": 0, "get_or_fail_success": 979, "get_started": 0, "get_sum": 98879, "max": 3758096384, "put": 20, "put_sum": 98879, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 11107, "get_or_fail_fail": 0, "get_or_fail_success": 11107, "get_started": 0, "get_sum": 24211563, "max": 3758096384, "put": 21, "put_sum": 24211563, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:38.648 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/1396, 979/2375 2024-09-17T03:54:39.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 -- 172.21.15.129:0/690884789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c40d2930 msgr2=0x7f10c40b6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 --2- 172.21.15.129:0/690884789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c40d2930 0x7f10c40b6050 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f10c408da90 tx=0x7f10b002a440 comp rx=0 tx=0).stop 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 -- 172.21.15.129:0/690884789 shutdown_connections 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 --2- 172.21.15.129:0/690884789 >> v2:172.21.15.181:3300/0 conn(0x7f10c40b6590 0x7f10c40cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 --2- 172.21.15.129:0/690884789 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c40d2930 0x7f10c40b6050 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 --2- 172.21.15.129:0/690884789 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c415eda0 0x7f10c406dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:39.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 -- 172.21.15.129:0/690884789 >> 172.21.15.129:0/690884789 conn(0x7f10c40d1230 msgr2=0x7f10c40d1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:39.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.875+0000 7f10c3e00640 1 -- 172.21.15.129:0/690884789 shutdown_connections 2024-09-17T03:54:39.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.876+0000 7f10c3e00640 1 -- 172.21.15.129:0/690884789 wait complete. 2024-09-17T03:54:39.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.878+0000 7f10c3e00640 1 Processor -- start 2024-09-17T03:54:39.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.878+0000 7f10c3e00640 1 -- start start 2024-09-17T03:54:39.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 0x7f10c4154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:39.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:39.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:39.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10c40cfb60 con 0x7f10c415eda0 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10c40cf9e0 con 0x7f10c40b6590 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.879+0000 7f10c3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10c40cfce0 con 0x7f10c40d2930 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 0x7f10c4154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:59100/0 (socket says 172.21.15.129:59100) 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 0x7f10c4154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:34250/0 (socket says 172.21.15.129:34250) 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 -- 172.21.15.129:0/1747978544 learned_addr learned my addr 172.21.15.129:0/1747978544 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49574/0 (socket says 172.21.15.129:49574) 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 -- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 msgr2=0x7f10c4154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 0x7f10c4154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:39.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 -- 172.21.15.129:0/1747978544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 msgr2=0x7f10c4151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:39.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:39.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c2000640 1 -- 172.21.15.129:0/1747978544 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c41f7a70 con 0x7f10c40d2930 2024-09-17T03:54:39.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.880+0000 7f10c3400640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:54:39.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10c2000640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f10b0000c00 tx=0x7f10b00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b003e030 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10b0003ca0 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0035040 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c41526c0 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.881+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c4150970 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.883+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10c40d0200 con 0x7f10c40d2930 2024-09-17T03:54:39.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.883+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10b0041030 con 0x7f10c40d2930 2024-09-17T03:54:39.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.884+0000 7f10baa00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 0x7f10a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:39.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.884+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f10b00bb360 con 0x7f10c40d2930 2024-09-17T03:54:39.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.885+0000 7f10c2a00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 0x7f10a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:39.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.886+0000 7f10c2a00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 0x7f10a407b0a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009ab0 tx=0x7f10b4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:39.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:39.890+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10b0083170 con 0x7f10c40d2930 2024-09-17T03:54:40.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.116+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10c4154900 con 0x7f10c40d2930 2024-09-17T03:54:40.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.117+0000 7f10baa00640 1 -- 172.21.15.129:0/1747978544 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f10b0087ba0 con 0x7f10c40d2930 2024-09-17T03:54:40.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:40.118 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:40.118 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 msgr2=0x7f10a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 0x7f10a407b0a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009ab0 tx=0x7f10b4005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 msgr2=0x7f10c4150430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f10b0000c00 tx=0x7f10b00038a0 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 shutdown_connections 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10a4078b90 0x7f10a407b0a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10c415eda0 0x7f10c4151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10c40d2930 0x7f10c4150430 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 --2- 172.21.15.129:0/1747978544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10c40b6590 0x7f10c4154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 >> 172.21.15.129:0/1747978544 conn(0x7f10c40d1230 msgr2=0x7f10c40b9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:40.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 shutdown_connections 2024-09-17T03:54:40.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:40.120+0000 7f10c3e00640 1 -- 172.21.15.129:0/1747978544 wait complete. 2024-09-17T03:54:40.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:40.396 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":881,"msgr_send_messages":881,"msgr_recv_bytes":233279,"msgr_send_bytes":4505630,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.130581881,"msgr_running_send_time":0.063851949,"msgr_running_recv_time":0.029946742,"msgr_running_fast_dispatch_time":0.027621332,"msgr_send_messages_queue_lat":{"avgcount":880,"sum":0.068617785,"avgtime":0.000077974},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10793,"msgr_send_messages":21100,"msgr_recv_bytes":3945081,"msgr_send_bytes":23113414,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.734205737,"msgr_running_send_time":1.257592954,"msgr_running_recv_time":47.412483446,"msgr_running_fast_dispatch_time":0.026705786,"msgr_send_messages_queue_lat":{"avgcount":21099,"sum":1.571254584,"avgtime":0.000074470},"msgr_handle_ack_lat":{"avgcount":13834,"sum":0.003358172,"avgtime":0.000000242},"msgr_recv_encrypted_bytes":120304,"msgr_send_encrypted_bytes":8080},"AsyncMessenger::Worker-2":{"msgr_recv_messages":617,"msgr_send_messages":624,"msgr_recv_bytes":161727,"msgr_send_bytes":3255045,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.082120080,"msgr_running_send_time":0.044234316,"msgr_running_recv_time":0.015464732,"msgr_running_fast_dispatch_time":0.017898197,"msgr_send_messages_queue_lat":{"avgcount":623,"sum":0.054531840,"avgtime":0.000087531},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":65168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":104,"sum":5.942386460,"avgtime":0.057138331}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3421,"sum":0.927888504,"avgtime":0.000271233}},"mds":{"request":5585,"reply":5583,"reply_latency":{"avgcount":5583,"sum":8.248641438,"avgtime":0.001477456},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4798,"inodes_top":0,"inodes_bottom":4798,"inodes_pin_tail":0,"inodes_pinned":4798,"inodes_expired":0,"inodes_with_caps":1253,"caps":1253,"subtrees":2,"traverse":5585,"traverse_hit":5585,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1043,"root_rbytes":3853312,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":6322,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4417,"handle_client_caps_dirty":2991,"handle_client_cap_release":148,"process_request_cap_release":2383},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":426,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":2375,"strays_enqueued":1949,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":12078,"evex":0,"evtrm":0,"ev":12081,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":14,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28988144,"rdpos":4195244,"jlat":{"avgcount":21,"sum":0.889199016,"avgtime":0.042342810},"replayed":3},"mds_mem":{"ino":2235,"ino+":3214,"ino-":979,"dir":13,"dir+":13,"dir-":0,"dn":4607,"dn+":5586,"dn-":979,"cap":1253,"cap+":3202,"cap-":1949,"rss":177984,"heap":413093888},"mds_server":{"dispatch_client_request":5585,"dispatch_server_request":0,"handle_client_request":5585,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2375,"sum":4.866709215,"avgtime":0.002049140},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5151,"average_load":5151,"avg_session_uptime":807,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":26164311,"buffer_anon_items":8979,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13145938,"mds_co_items":217770,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":2096,"op_send_bytes":24830143,"op_resend":0,"op_reply":2096,"op_latency":{"avgcount":2096,"sum":5.406883509,"avgtime":0.002579619},"op_inflight":0,"oplen_avg":{"avgcount":2096,"sum":2163},"op":2096,"op_r":59,"op_w":2037,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":31,"osdop_write":44,"osdop_writefull":11,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1965,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":29,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1949,"pq_executed":1949,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12287,"get_sum":3393832,"get_or_fail_fail":0,"get_or_fail_success":12287,"take":0,"take_sum":0,"put":12287,"put_sum":3393832,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2096,"take_sum":25027927,"put":2096,"put_sum":25027927,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2096,"take_sum":2096,"put":2096,"put_sum":2096,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1949,"get_sum":196849,"get_or_fail_fail":0,"get_or_fail_success":1949,"take":0,"take_sum":0,"put":23,"put_sum":196849,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":581078,"max":3758096384,"get_started":0,"get":12078,"get_sum":24792641,"get_or_fail_fail":0,"get_or_fail_success":12078,"take":0,"take_sum":0,"put":21,"put_sum":24211563,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:40.397 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 233279, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 881, "msgr_running_fast_dispatch_time": 0.027621332, "msgr_running_recv_time": 0.029946742, "msgr_running_send_time": 0.063851949, "msgr_running_total_time": 0.130581881, "msgr_send_bytes": 4505630, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 881, "msgr_send_messages_queue_lat": { "avgcount": 880, "avgtime": 7.7974e-05, "sum": 0.068617785 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13834, "avgtime": 2.42e-07, "sum": 0.003358172 }, "msgr_recv_bytes": 3945081, "msgr_recv_encrypted_bytes": 120304, "msgr_recv_messages": 10793, "msgr_running_fast_dispatch_time": 0.026705786, "msgr_running_recv_time": 47.412483446, "msgr_running_send_time": 1.257592954, "msgr_running_total_time": 1.734205737, "msgr_send_bytes": 23113414, "msgr_send_encrypted_bytes": 8080, "msgr_send_messages": 21100, "msgr_send_messages_queue_lat": { "avgcount": 21099, "avgtime": 7.447e-05, "sum": 1.571254584 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 161727, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 617, "msgr_running_fast_dispatch_time": 0.017898197, "msgr_running_recv_time": 0.015464732, "msgr_running_send_time": 0.044234316, "msgr_running_total_time": 0.08212008, "msgr_send_bytes": 3255045, "msgr_send_encrypted_bytes": 65168, "msgr_send_messages": 624, "msgr_send_messages_queue_lat": { "avgcount": 623, "avgtime": 8.7531e-05, "sum": 0.05453184 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 104, "avgtime": 0.057138331, "sum": 5.94238646 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3421, "avgtime": 0.000271233, "sum": 0.927888504 }, "queue_len": 0 }, "mds": { "caps": 1253, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6322, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 148, "handle_client_caps": 4417, "handle_client_caps_dirty": 2991, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4798, "inodes_bottom": 4798, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4798, "inodes_top": 0, "inodes_with_caps": 1253, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2383, "q": 0, "reply": 5583, "reply_latency": { "avgcount": 5583, "avgtime": 0.001477456, "sum": 8.248641438 }, "request": 5585, "root_rbytes": 3853312, "root_rfiles": 1043, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5585, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5585, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 426, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2375, "strays_enqueued": 1949, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 12081, "evadd": 12078, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 21, "avgtime": 0.04234281, "sum": 0.889199016 }, "rdpos": 4195244, "replayed": 3, "seg": 14, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28988144 }, "mds_mem": { "cap": 1253, "cap+": 3202, "cap-": 1949, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4607, "dn+": 5586, "dn-": 979, "heap": 413093888, "ino": 2235, "ino+": 3214, "ino-": 979, "rss": 177984 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5585, "dispatch_server_request": 0, "handle_client_request": 5585, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2375, "avgtime": 0.00204914, "sum": 4.866709215 } }, "mds_sessions": { "average_load": 5151, "avg_session_uptime": 807, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5151 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 26164311, "buffer_anon_items": 8979, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13145938, "mds_co_items": 217770, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 29, "op": 2096, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 2096, "avgtime": 0.002579619, "sum": 5.406883509 }, "op_pg": 0, "op_r": 59, "op_reply": 2096, "op_resend": 0, "op_rmw": 0, "op_send": 2096, "op_send_bytes": 24830143, "op_w": 2037, "oplen_avg": { "avgcount": 2096, "sum": 2163 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1965, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 44, "osdop_writefull": 11, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1949, "pq_executed_ops": 1949, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12287, "get_or_fail_fail": 0, "get_or_fail_success": 12287, "get_started": 0, "get_sum": 3393832, "max": 104857600, "put": 12287, "put_sum": 3393832, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2096, "put_sum": 25027927, "take": 2096, "take_sum": 25027927, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2096, "put_sum": 2096, "take": 2096, "take_sum": 2096, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1949, "get_or_fail_fail": 0, "get_or_fail_success": 1949, "get_started": 0, "get_sum": 196849, "max": 3758096384, "put": 23, "put_sum": 196849, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 12078, "get_or_fail_fail": 0, "get_or_fail_success": 12078, "get_started": 0, "get_sum": 24792641, "max": 3758096384, "put": 21, "put_sum": 24211563, "take": 0, "take_sum": 0, "val": 581078, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:40.398 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/426, 1949/2375 2024-09-17T03:54:41.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 -- 172.21.15.129:0/1959821746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 msgr2=0x7fdab00b7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 --2- 172.21.15.129:0/1959821746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00b7c10 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fdab00b7290 tx=0x7fda9c02a560 comp rx=0 tx=0).stop 2024-09-17T03:54:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 -- 172.21.15.129:0/1959821746 shutdown_connections 2024-09-17T03:54:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 --2- 172.21.15.129:0/1959821746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00b7c10 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 --2- 172.21.15.129:0/1959821746 >> v2:172.21.15.181:3300/0 conn(0x7fdab00cf2e0 0x7fdab00d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 -- 172.21.15.129:0/1959821746 >> 172.21.15.129:0/1959821746 conn(0x7fdab00d1770 msgr2=0x7fdab00d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 -- 172.21.15.129:0/1959821746 shutdown_connections 2024-09-17T03:54:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.622+0000 7fdab6400640 1 -- 172.21.15.129:0/1959821746 wait complete. 2024-09-17T03:54:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.624+0000 7fdab6400640 1 Processor -- start 2024-09-17T03:54:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.624+0000 7fdab6400640 1 -- start start 2024-09-17T03:54:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdab00b8150 0x7fdab015cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab00cf2e0 0x7fdab014dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00d3be0 con 0x7fdab015ea40 2024-09-17T03:54:41.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00d3a60 con 0x7fdab00cf2e0 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdab00d3d60 con 0x7fdab00b8150 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49600/0 (socket says 172.21.15.129:49600) 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab5a00640 1 -- 172.21.15.129:0/3855344924 learned_addr learned my addr 172.21.15.129:0/3855344924 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdab5000640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdab00b8150 0x7fdab015cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.626+0000 7fdaafe00640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab00cf2e0 0x7fdab014dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 -- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdab00b8150 msgr2=0x7fdab015cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdab00b8150 0x7fdab015cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 -- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab00cf2e0 msgr2=0x7fdab014dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab00cf2e0 0x7fdab014dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 -- 172.21.15.129:0/3855344924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab015d4f0 con 0x7fdab015ea40 2024-09-17T03:54:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab5a00640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800b9e0 tx=0x7fdaa800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8003b70 con 0x7fdab015ea40 2024-09-17T03:54:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdaa8003e00 con 0x7fdab015ea40 2024-09-17T03:54:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab014e010 con 0x7fdab015ea40 2024-09-17T03:54:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa800cb80 con 0x7fdab015ea40 2024-09-17T03:54:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.627+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab01f98d0 con 0x7fdab015ea40 2024-09-17T03:54:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.629+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab00d4230 con 0x7fdab015ea40 2024-09-17T03:54:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.630+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdaa8004000 con 0x7fdab015ea40 2024-09-17T03:54:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.631+0000 7fdaacc00640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 0x7fda9007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:41.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.631+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fdaa809ab50 con 0x7fdab015ea40 2024-09-17T03:54:41.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.631+0000 7fdab5000640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 0x7fda9007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:41.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.632+0000 7fdab5000640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 0x7fda9007b0d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4004630 tx=0x7fdaa400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.636+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdaa8062970 con 0x7fdab015ea40 2024-09-17T03:54:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.862+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdab015e180 con 0x7fdab015ea40 2024-09-17T03:54:41.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.863+0000 7fdaacc00640 1 -- 172.21.15.129:0/3855344924 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7fdaa80673a0 con 0x7fdab015ea40 2024-09-17T03:54:41.863 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:41.863 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:41.864 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.865+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 msgr2=0x7fda9007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.865+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 0x7fda9007b0d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4004630 tx=0x7fdaa400b2e0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 msgr2=0x7fdab00cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdaa800b9e0 tx=0x7fdaa800bea0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 shutdown_connections 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda90078bc0 0x7fda9007b0d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdab015ea40 0x7fdab00cde30 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdab00cf2e0 0x7fdab014dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 --2- 172.21.15.129:0/3855344924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdab00b8150 0x7fdab015cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:41.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 >> 172.21.15.129:0/3855344924 conn(0x7fdab00d1770 msgr2=0x7fdab0161400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:41.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 shutdown_connections 2024-09-17T03:54:41.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:41.866+0000 7fdab6400640 1 -- 172.21.15.129:0/3855344924 wait complete. 2024-09-17T03:54:41.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:42.133 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":881,"msgr_send_messages":881,"msgr_recv_bytes":233279,"msgr_send_bytes":4505630,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.130581881,"msgr_running_send_time":0.063851949,"msgr_running_recv_time":0.029946742,"msgr_running_fast_dispatch_time":0.027621332,"msgr_send_messages_queue_lat":{"avgcount":880,"sum":0.068617785,"avgtime":0.000077974},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":10794,"msgr_send_messages":21101,"msgr_recv_bytes":3945353,"msgr_send_bytes":23114054,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.734415734,"msgr_running_send_time":1.257719893,"msgr_running_recv_time":47.412560150,"msgr_running_fast_dispatch_time":0.026705786,"msgr_send_messages_queue_lat":{"avgcount":21100,"sum":1.571300400,"avgtime":0.000074469},"msgr_handle_ack_lat":{"avgcount":13834,"sum":0.003358172,"avgtime":0.000000242},"msgr_recv_encrypted_bytes":120576,"msgr_send_encrypted_bytes":8720},"AsyncMessenger::Worker-2":{"msgr_recv_messages":617,"msgr_send_messages":624,"msgr_recv_bytes":161727,"msgr_send_bytes":3255045,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.082120080,"msgr_running_send_time":0.044234316,"msgr_running_recv_time":0.015464732,"msgr_running_fast_dispatch_time":0.017898197,"msgr_send_messages_queue_lat":{"avgcount":623,"sum":0.054531840,"avgtime":0.000087531},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":65168},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":104,"sum":5.942386460,"avgtime":0.057138331}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3421,"sum":0.927888504,"avgtime":0.000271233}},"mds":{"request":5585,"reply":5583,"reply_latency":{"avgcount":5583,"sum":8.248641438,"avgtime":0.001477456},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4798,"inodes_top":0,"inodes_bottom":4798,"inodes_pin_tail":0,"inodes_pinned":4798,"inodes_expired":0,"inodes_with_caps":1253,"caps":1253,"subtrees":2,"traverse":5585,"traverse_hit":5585,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1043,"root_rbytes":3853312,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":6322,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4417,"handle_client_caps_dirty":2991,"handle_client_cap_release":148,"process_request_cap_release":2383},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":426,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":2375,"strays_enqueued":1949,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":12078,"evex":0,"evtrm":0,"ev":12081,"evexg":0,"evexd":0,"segadd":12,"segex":0,"segtrm":0,"seg":14,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":28988144,"rdpos":4195244,"jlat":{"avgcount":21,"sum":0.889199016,"avgtime":0.042342810},"replayed":3},"mds_mem":{"ino":2235,"ino+":3214,"ino-":979,"dir":13,"dir+":13,"dir-":0,"dn":4607,"dn+":5586,"dn-":979,"cap":1253,"cap+":3202,"cap-":1949,"rss":177984,"heap":413093888},"mds_server":{"dispatch_client_request":5585,"dispatch_server_request":0,"handle_client_request":5585,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":2375,"sum":4.866709215,"avgtime":0.002049140},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5151,"average_load":5151,"avg_session_uptime":807,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":26164311,"buffer_anon_items":8979,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":13145938,"mds_co_items":217770,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":2096,"op_send_bytes":24830143,"op_resend":0,"op_reply":2096,"op_latency":{"avgcount":2096,"sum":5.406883509,"avgtime":0.002579619},"op_inflight":0,"oplen_avg":{"avgcount":2096,"sum":2163},"op":2096,"op_r":59,"op_w":2037,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":31,"osdop_write":44,"osdop_writefull":11,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1965,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":29,"omap_rd":48,"omap_del":8},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":1949,"pq_executed":1949,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12288,"get_sum":3393962,"get_or_fail_fail":0,"get_or_fail_success":12288,"take":0,"take_sum":0,"put":12288,"put_sum":3393962,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2096,"take_sum":25027927,"put":2096,"put_sum":25027927,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2096,"take_sum":2096,"put":2096,"put_sum":2096,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1949,"get_sum":196849,"get_or_fail_fail":0,"get_or_fail_success":1949,"take":0,"take_sum":0,"put":23,"put_sum":196849,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":581078,"max":3758096384,"get_started":0,"get":12078,"get_sum":24792641,"get_or_fail_fail":0,"get_or_fail_success":12078,"take":0,"take_sum":0,"put":21,"put_sum":24211563,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:42.135 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 233279, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 881, "msgr_running_fast_dispatch_time": 0.027621332, "msgr_running_recv_time": 0.029946742, "msgr_running_send_time": 0.063851949, "msgr_running_total_time": 0.130581881, "msgr_send_bytes": 4505630, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 881, "msgr_send_messages_queue_lat": { "avgcount": 880, "avgtime": 7.7974e-05, "sum": 0.068617785 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 13834, "avgtime": 2.42e-07, "sum": 0.003358172 }, "msgr_recv_bytes": 3945353, "msgr_recv_encrypted_bytes": 120576, "msgr_recv_messages": 10794, "msgr_running_fast_dispatch_time": 0.026705786, "msgr_running_recv_time": 47.41256015, "msgr_running_send_time": 1.257719893, "msgr_running_total_time": 1.734415734, "msgr_send_bytes": 23114054, "msgr_send_encrypted_bytes": 8720, "msgr_send_messages": 21101, "msgr_send_messages_queue_lat": { "avgcount": 21100, "avgtime": 7.4469e-05, "sum": 1.5713004 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 161727, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 617, "msgr_running_fast_dispatch_time": 0.017898197, "msgr_running_recv_time": 0.015464732, "msgr_running_send_time": 0.044234316, "msgr_running_total_time": 0.08212008, "msgr_send_bytes": 3255045, "msgr_send_encrypted_bytes": 65168, "msgr_send_messages": 624, "msgr_send_messages_queue_lat": { "avgcount": 623, "avgtime": 8.7531e-05, "sum": 0.05453184 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 104, "avgtime": 0.057138331, "sum": 5.94238646 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3421, "avgtime": 0.000271233, "sum": 0.927888504 }, "queue_len": 0 }, "mds": { "caps": 1253, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 6322, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 148, "handle_client_caps": 4417, "handle_client_caps_dirty": 2991, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4798, "inodes_bottom": 4798, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4798, "inodes_top": 0, "inodes_with_caps": 1253, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 2383, "q": 0, "reply": 5583, "reply_latency": { "avgcount": 5583, "avgtime": 0.001477456, "sum": 8.248641438 }, "request": 5585, "root_rbytes": 3853312, "root_rfiles": 1043, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 5585, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 5585, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 426, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 2375, "strays_enqueued": 1949, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 12081, "evadd": 12078, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 21, "avgtime": 0.04234281, "sum": 0.889199016 }, "rdpos": 4195244, "replayed": 3, "seg": 14, "segadd": 12, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 28988144 }, "mds_mem": { "cap": 1253, "cap+": 3202, "cap-": 1949, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4607, "dn+": 5586, "dn-": 979, "heap": 413093888, "ino": 2235, "ino+": 3214, "ino-": 979, "rss": 177984 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 5585, "dispatch_server_request": 0, "handle_client_request": 5585, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 2375, "avgtime": 0.00204914, "sum": 4.866709215 } }, "mds_sessions": { "average_load": 5151, "avg_session_uptime": 807, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5151 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 26164311, "buffer_anon_items": 8979, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 13145938, "mds_co_items": 217770, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 8, "omap_rd": 48, "omap_wr": 29, "op": 2096, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 2096, "avgtime": 0.002579619, "sum": 5.406883509 }, "op_pg": 0, "op_r": 59, "op_reply": 2096, "op_resend": 0, "op_rmw": 0, "op_send": 2096, "op_send_bytes": 24830143, "op_w": 2037, "oplen_avg": { "avgcount": 2096, "sum": 2163 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1965, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 44, "osdop_writefull": 11, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1949, "pq_executed_ops": 1949, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12288, "get_or_fail_fail": 0, "get_or_fail_success": 12288, "get_started": 0, "get_sum": 3393962, "max": 104857600, "put": 12288, "put_sum": 3393962, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2096, "put_sum": 25027927, "take": 2096, "take_sum": 25027927, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2096, "put_sum": 2096, "take": 2096, "take_sum": 2096, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1949, "get_or_fail_fail": 0, "get_or_fail_success": 1949, "get_started": 0, "get_sum": 196849, "max": 3758096384, "put": 23, "put_sum": 196849, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 12078, "get_or_fail_fail": 0, "get_or_fail_success": 12078, "get_started": 0, "get_sum": 24792641, "max": 3758096384, "put": 21, "put_sum": 24211563, "take": 0, "take_sum": 0, "val": 581078, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:42.135 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/426, 1949/2375 2024-09-17T03:54:43.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.381+0000 7faabec00640 1 -- 172.21.15.129:0/3140209610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa9c003c30 msgr2=0x7faa9c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.381+0000 7faabec00640 1 --2- 172.21.15.129:0/3140209610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa9c003c30 0x7faa9c004080 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7faaa8001830 tx=0x7faa9c005830 comp rx=0 tx=0).stop 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.382+0000 7faabec00640 1 -- 172.21.15.129:0/3140209610 shutdown_connections 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.382+0000 7faabec00640 1 --2- 172.21.15.129:0/3140209610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa9c003c30 0x7faa9c004080 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.382+0000 7faabec00640 1 -- 172.21.15.129:0/3140209610 >> 172.21.15.129:0/3140209610 conn(0x7faab80b4370 msgr2=0x7faab80b4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:43.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.382+0000 7faabec00640 1 -- 172.21.15.129:0/3140209610 shutdown_connections 2024-09-17T03:54:43.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.382+0000 7faabec00640 1 -- 172.21.15.129:0/3140209610 wait complete. 2024-09-17T03:54:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.384+0000 7faabec00640 1 Processor -- start 2024-09-17T03:54:43.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.384+0000 7faabec00640 1 -- start start 2024-09-17T03:54:43.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faab81f6c50 0x7faab81f7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faab81f75a0 0x7faab81f9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faab80b7ff0 con 0x7faa9c003c30 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faab80b7e70 con 0x7faab81f6c50 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faab8160cb0 con 0x7faab81f75a0 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faab7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faab7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49618/0 (socket says 172.21.15.129:49618) 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faab7e00640 1 -- 172.21.15.129:0/1696318391 learned_addr learned my addr 172.21.15.129:0/1696318391 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faab7400640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faab81f6c50 0x7faab81f7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.387+0000 7faabcc00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faab81f75a0 0x7faab81f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 -- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faab81f75a0 msgr2=0x7faab81f9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faab81f75a0 0x7faab81f9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 -- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faab81f6c50 msgr2=0x7faab81f7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faab81f6c50 0x7faab81f7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 -- 172.21.15.129:0/1696318391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab81eb720 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faab7e00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faaa8001950 tx=0x7faaa80060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8043070 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faaa80038e0 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab8160550 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.388+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8003cd0 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.389+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab815e4d0 con 0x7faa9c003c30 2024-09-17T03:54:43.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.390+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab8075310 con 0x7faa9c003c30 2024-09-17T03:54:43.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.391+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faaa8033480 con 0x7faa9c003c30 2024-09-17T03:54:43.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.392+0000 7faaafe00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 0x7faa9407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:43.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.393+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7faaa80bae40 con 0x7faa9c003c30 2024-09-17T03:54:43.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.393+0000 7faab7400640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 0x7faa9407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:43.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.394+0000 7faab7400640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 0x7faa9407b0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faab809d3f0 tx=0x7faaa0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:43.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.398+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faab8075310 con 0x7faa9c003c30 2024-09-17T03:54:43.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.682+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faab80cd460 con 0x7faa9c003c30 2024-09-17T03:54:43.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.683+0000 7faaafe00640 1 -- 172.21.15.129:0/1696318391 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7faaa80831c0 con 0x7faa9c003c30 2024-09-17T03:54:43.684 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:43.684 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:43.684 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 msgr2=0x7faa9407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 0x7faa9407b0d0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faab809d3f0 tx=0x7faaa0005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 msgr2=0x7faab815b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7faaa8001950 tx=0x7faaa80060e0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 shutdown_connections 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa94078bc0 0x7faa9407b0d0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faab81f75a0 0x7faab81f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faab81f6c50 0x7faab81f7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 --2- 172.21.15.129:0/1696318391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa9c003c30 0x7faab815b510 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 >> 172.21.15.129:0/1696318391 conn(0x7faab80b4370 msgr2=0x7faab80cc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 shutdown_connections 2024-09-17T03:54:43.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:43.686+0000 7faabec00640 1 -- 172.21.15.129:0/1696318391 wait complete. 2024-09-17T03:54:43.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:44.050 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1159,"msgr_send_messages":1162,"msgr_recv_bytes":303883,"msgr_send_bytes":4597228,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.170051364,"msgr_running_send_time":0.084535464,"msgr_running_recv_time":0.036752650,"msgr_running_fast_dispatch_time":0.036711582,"msgr_send_messages_queue_lat":{"avgcount":1161,"sum":0.091963991,"avgtime":0.000079211},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12339,"msgr_send_messages":23502,"msgr_recv_bytes":4434927,"msgr_send_bytes":27520914,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":1.956592000,"msgr_running_send_time":1.424880276,"msgr_running_recv_time":47.768162495,"msgr_running_fast_dispatch_time":0.034021591,"msgr_send_messages_queue_lat":{"avgcount":23500,"sum":1.728080558,"avgtime":0.000073535},"msgr_handle_ack_lat":{"avgcount":15371,"sum":0.003899262,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":120576,"msgr_send_encrypted_bytes":8720},"AsyncMessenger::Worker-2":{"msgr_recv_messages":785,"msgr_send_messages":794,"msgr_recv_bytes":204375,"msgr_send_bytes":4264781,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.104725962,"msgr_running_send_time":0.053745038,"msgr_running_recv_time":0.020598139,"msgr_running_fast_dispatch_time":0.023804200,"msgr_send_messages_queue_lat":{"avgcount":793,"sum":0.065380567,"avgtime":0.000082447},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":67008},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":677,"complete_latency":{"avgcount":116,"sum":7.210557734,"avgtime":0.062159980}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4443,"sum":1.305638628,"avgtime":0.000293864}},"mds":{"request":6411,"reply":6408,"reply_latency":{"avgcount":6408,"sum":9.502335023,"avgtime":0.001482886},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":4037,"inodes_top":0,"inodes_bottom":4037,"inodes_pin_tail":0,"inodes_pinned":4037,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":218,"root_rbytes":1675264,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":7,"ceph_cap_op_grant":7028,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4453,"handle_client_caps_dirty":3000,"handle_client_cap_release":656,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":475,"num_strays_delayed":340,"num_strays_enqueuing":0,"strays_created":3200,"strays_enqueued":2725,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14280,"evex":0,"evtrm":0,"ev":14283,"evexg":0,"evexd":0,"segadd":15,"segex":0,"segtrm":0,"seg":17,"segmjr":3,"segexg":0,"segexd":0,"expos":4194304,"wrpos":32893231,"rdpos":4195244,"jlat":{"avgcount":28,"sum":1.028655301,"avgtime":0.036737689},"replayed":3},"mds_mem":{"ino":839,"ino+":3214,"ino-":2375,"dir":13,"dir+":13,"dir-":0,"dn":4037,"dn+":6412,"dn-":2375,"cap":137,"cap+":3202,"cap-":3065,"rss":177984,"heap":413093888},"mds_server":{"dispatch_client_request":6412,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5488,"average_load":5488,"avg_session_uptime":812,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8776288,"buffer_anon_items":3632,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":6960268,"mds_co_items":104794,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":6,"op_laggy":0,"op_send":2713,"op_send_bytes":29382395,"op_resend":0,"op_reply":2707,"op_latency":{"avgcount":2707,"sum":6.917850974,"avgtime":0.002555541},"op_inflight":4,"oplen_avg":{"avgcount":2713,"sum":2784},"op":2713,"op_r":65,"op_w":2648,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":37,"osdop_write":57,"osdop_writefull":12,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2560,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":33,"omap_rd":48,"omap_del":10},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":6,"pq_executing_ops_high_water":6,"pq_executing":6,"pq_executing_high_water":6,"pq_executed_ops":2537,"pq_executed":2537,"pq_item_in_journal":190},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14279,"get_sum":3839678,"get_or_fail_fail":0,"get_or_fail_success":14279,"take":0,"take_sum":0,"put":14279,"put_sum":3839678,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2713,"take_sum":29658555,"put":2707,"put_sum":29658555,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":6,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":2713,"take_sum":2713,"put":2707,"put_sum":2707,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":2725,"get_sum":275225,"get_or_fail_fail":0,"get_or_fail_success":2725,"take":0,"take_sum":0,"put":29,"put_sum":275225,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":25158,"max":3758096384,"get_started":0,"get":14280,"get_sum":28697728,"get_or_fail_fail":0,"get_or_fail_success":14280,"take":0,"take_sum":0,"put":28,"put_sum":28672570,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:44.052 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 303883, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1159, "msgr_running_fast_dispatch_time": 0.036711582, "msgr_running_recv_time": 0.03675265, "msgr_running_send_time": 0.084535464, "msgr_running_total_time": 0.170051364, "msgr_send_bytes": 4597228, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1162, "msgr_send_messages_queue_lat": { "avgcount": 1161, "avgtime": 7.9211e-05, "sum": 0.091963991 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15371, "avgtime": 2.53e-07, "sum": 0.003899262 }, "msgr_recv_bytes": 4434927, "msgr_recv_encrypted_bytes": 120576, "msgr_recv_messages": 12339, "msgr_running_fast_dispatch_time": 0.034021591, "msgr_running_recv_time": 47.768162495, "msgr_running_send_time": 1.424880276, "msgr_running_total_time": 1.956592, "msgr_send_bytes": 27520914, "msgr_send_encrypted_bytes": 8720, "msgr_send_messages": 23502, "msgr_send_messages_queue_lat": { "avgcount": 23500, "avgtime": 7.3535e-05, "sum": 1.728080558 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 204375, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 785, "msgr_running_fast_dispatch_time": 0.0238042, "msgr_running_recv_time": 0.020598139, "msgr_running_send_time": 0.053745038, "msgr_running_total_time": 0.104725962, "msgr_send_bytes": 4264781, "msgr_send_encrypted_bytes": 67008, "msgr_send_messages": 794, "msgr_send_messages_queue_lat": { "avgcount": 793, "avgtime": 8.2447e-05, "sum": 0.065380567 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 116, "avgtime": 0.06215998, "sum": 7.210557734 }, "queue_len": 677 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4443, "avgtime": 0.000293864, "sum": 1.305638628 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7028, "ceph_cap_op_revoke": 7, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 656, "handle_client_caps": 4453, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 4037, "inodes_bottom": 4037, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 4037, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6408, "reply_latency": { "avgcount": 6408, "avgtime": 0.001482886, "sum": 9.502335023 }, "request": 6411, "root_rbytes": 1675264, "root_rfiles": 218, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 475, "num_strays_delayed": 340, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3200, "strays_enqueued": 2725, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 14283, "evadd": 14280, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 28, "avgtime": 0.036737689, "sum": 1.028655301 }, "rdpos": 4195244, "replayed": 3, "seg": 17, "segadd": 15, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 3, "segtrm": 0, "wrpos": 32893231 }, "mds_mem": { "cap": 137, "cap+": 3202, "cap-": 3065, "dir": 13, "dir+": 13, "dir-": 0, "dn": 4037, "dn+": 6412, "dn-": 2375, "heap": 413093888, "ino": 839, "ino+": 3214, "ino-": 2375, "rss": 177984 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6412, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5488, "avg_session_uptime": 812, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5488 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8776288, "buffer_anon_items": 3632, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 6960268, "mds_co_items": 104794, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 10, "omap_rd": 48, "omap_wr": 33, "op": 2713, "op_Infinitylight": 4, "op_active": 6, "op_laggy": 0, "op_latency": { "avgcount": 2707, "avgtime": 0.002555541, "sum": 6.917850974 }, "op_pg": 0, "op_r": 65, "op_reply": 2707, "op_resend": 0, "op_rmw": 0, "op_send": 2713, "op_send_bytes": 29382395, "op_w": 2648, "oplen_avg": { "avgcount": 2713, "sum": 2784 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 2560, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 37, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 57, "osdop_writefull": 12, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 2537, "pq_executed_ops": 2537, "pq_executing": 6, "pq_executing_high_water": 6, "pq_executing_ops": 6, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 190 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14279, "get_or_fail_fail": 0, "get_or_fail_success": 14279, "get_started": 0, "get_sum": 3839678, "max": 104857600, "put": 14279, "put_sum": 3839678, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 2707, "put_sum": 29658555, "take": 2713, "take_sum": 29658555, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 2707, "put_sum": 2707, "take": 2713, "take_sum": 2713, "val": 6, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 2725, "get_or_fail_fail": 0, "get_or_fail_success": 2725, "get_started": 0, "get_sum": 275225, "max": 3758096384, "put": 29, "put_sum": 275225, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14280, "get_or_fail_fail": 0, "get_or_fail_success": 14280, "get_started": 0, "get_sum": 28697728, "max": 3758096384, "put": 28, "put_sum": 28672570, "take": 0, "take_sum": 0, "val": 25158, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:44.052 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/475, 2537/3200 2024-09-17T03:54:44.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 -- 172.21.15.129:0/2000489536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c0b4340 msgr2=0x7feb1c06f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 --2- 172.21.15.129:0/2000489536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c0b4340 0x7feb1c06f710 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0cd1f0 tx=0x7feb0c02a560 comp rx=0 tx=0).stop 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 -- 172.21.15.129:0/2000489536 shutdown_connections 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 --2- 172.21.15.129:0/2000489536 >> v2:172.21.15.181:3300/0 conn(0x7feb1c151e30 0x7feb1c0726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 --2- 172.21.15.129:0/2000489536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c0b4340 0x7feb1c06f710 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.337+0000 7feb20e00640 1 -- 172.21.15.129:0/2000489536 >> 172.21.15.129:0/2000489536 conn(0x7feb1c0b23c0 msgr2=0x7feb1c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:44.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.338+0000 7feb20e00640 1 -- 172.21.15.129:0/2000489536 shutdown_connections 2024-09-17T03:54:44.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.338+0000 7feb20e00640 1 -- 172.21.15.129:0/2000489536 wait complete. 2024-09-17T03:54:44.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.340+0000 7feb20e00640 1 Processor -- start 2024-09-17T03:54:44.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.340+0000 7feb20e00640 1 -- start start 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 0x7feb1c154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1c151e30 0x7feb1c1e8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c073030 con 0x7feb1c06fc50 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c072eb0 con 0x7feb1c151e30 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.341+0000 7feb20e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c15b6e0 con 0x7feb1c0b4340 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47302/0 (socket says 172.21.15.129:47302) 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 -- 172.21.15.129:0/2379058130 learned_addr learned my addr 172.21.15.129:0/2379058130 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:44.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb1a800640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1c151e30 0x7feb1c1e8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19400640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 0x7feb1c154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 -- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 msgr2=0x7feb1c154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 0x7feb1c154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 -- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1c151e30 msgr2=0x7feb1c1e8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1c151e30 0x7feb1c1e8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19e00640 1 -- 172.21.15.129:0/2379058130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c1f3970 con 0x7feb1c06fc50 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.342+0000 7feb19400640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 0x7feb1c154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb19e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7feb0c034040 tx=0x7feb0c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c03e030 con 0x7feb1c06fc50 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb0c0047a0 con 0x7feb1c06fc50 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1c1f3e40 con 0x7feb1c06fc50 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c035040 con 0x7feb1c06fc50 2024-09-17T03:54:44.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.343+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1c1f4400 con 0x7feb1c06fc50 2024-09-17T03:54:44.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.344+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1c06f710 con 0x7feb1c06fc50 2024-09-17T03:54:44.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.345+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feb0c041030 con 0x7feb1c06fc50 2024-09-17T03:54:44.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.346+0000 7feb12000640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 0x7feafc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:44.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.346+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7feb0c0bb3a0 con 0x7feb1c06fc50 2024-09-17T03:54:44.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.347+0000 7feb19400640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 0x7feafc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:44.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.347+0000 7feb19400640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 0x7feafc07b0d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb1c15d810 tx=0x7feb08009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:44.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.351+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feb0c083120 con 0x7feb1c06fc50 2024-09-17T03:54:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.571+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb1c0af890 con 0x7feb1c06fc50 2024-09-17T03:54:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.572+0000 7feb12000640 1 -- 172.21.15.129:0/2379058130 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7feb0c087b50 con 0x7feb1c06fc50 2024-09-17T03:54:44.572 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:44.572 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 msgr2=0x7feafc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 0x7feafc07b0d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7feb1c15d810 tx=0x7feb08009040 comp rx=0 tx=0).stop 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 msgr2=0x7feb1c139f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7feb0c034040 tx=0x7feb0c0039e0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 shutdown_connections 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feafc078bc0 0x7feafc07b0d0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb1c151e30 0x7feb1c1e8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb1c0b4340 0x7feb1c154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 --2- 172.21.15.129:0/2379058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb1c06fc50 0x7feb1c139f90 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 >> 172.21.15.129:0/2379058130 conn(0x7feb1c0b23c0 msgr2=0x7feb1c150180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 shutdown_connections 2024-09-17T03:54:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:44.574+0000 7feb20e00640 1 -- 172.21.15.129:0/2379058130 wait complete. 2024-09-17T03:54:44.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:54:44.962 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:54:44.962 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:54:45.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:45.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 -- 172.21.15.129:0/1860659058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3015f880 msgr2=0x7f0c3015fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 --2- 172.21.15.129:0/1860659058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3015f880 0x7f0c3015fc90 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0c18005fd0 tx=0x7f0c1802a530 comp rx=0 tx=0).stop 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 -- 172.21.15.129:0/1860659058 shutdown_connections 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 --2- 172.21.15.129:0/1860659058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c300cedf0 0x7f0c300cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 --2- 172.21.15.129:0/1860659058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3015f880 0x7f0c3015fc90 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 -- 172.21.15.129:0/1860659058 >> 172.21.15.129:0/1860659058 conn(0x7f0c300b23c0 msgr2=0x7f0c300b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 -- 172.21.15.129:0/1860659058 shutdown_connections 2024-09-17T03:54:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.272+0000 7f0c37000640 1 -- 172.21.15.129:0/1860659058 wait complete. 2024-09-17T03:54:45.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.274+0000 7f0c37000640 1 Processor -- start 2024-09-17T03:54:45.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.274+0000 7f0c37000640 1 -- start start 2024-09-17T03:54:45.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:45.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300cedf0 0x7f0c30144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:45.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3015f880 0x7f0c300726a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:45.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0c300cfad0 con 0x7f0c300737b0 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c300cf950 con 0x7f0c300cedf0 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c37000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0c300cfc50 con 0x7f0c3015f880 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47324/0 (socket says 172.21.15.129:47324) 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c2fe00640 1 -- 172.21.15.129:0/3754619726 learned_addr learned my addr 172.21.15.129:0/3754619726 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c27400640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300cedf0 0x7f0c30144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.276+0000 7f0c35000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3015f880 0x7f0c300726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 -- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3015f880 msgr2=0x7f0c300726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:45.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3015f880 0x7f0c300726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 -- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300cedf0 msgr2=0x7f0c30144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300cedf0 0x7f0c30144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 -- 172.21.15.129:0/3754619726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c301f9070 con 0x7f0c300737b0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2fe00640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000b3f0 tx=0x7f0c2000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20015020 con 0x7f0c300737b0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c20011920 con 0x7f0c300737b0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20011d70 con 0x7f0c300737b0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c301453f0 con 0x7f0c300737b0 2024-09-17T03:54:45.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.277+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c301f9850 con 0x7f0c300737b0 2024-09-17T03:54:45.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.279+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c300cf200 con 0x7f0c300737b0 2024-09-17T03:54:45.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.280+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0c20018020 con 0x7f0c300737b0 2024-09-17T03:54:45.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.281+0000 7f0c2cc00640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 0x7f0c0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:45.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.281+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f0c2009a8c0 con 0x7f0c300737b0 2024-09-17T03:54:45.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.281+0000 7f0c27400640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 0x7f0c0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:45.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.282+0000 7f0c27400640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 0x7f0c0c07b0a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c18033000 tx=0x7f0c18004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:45.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.285+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0c2001f030 con 0x7f0c300737b0 2024-09-17T03:54:45.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.501+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c300cd350 con 0x7f0c300737b0 2024-09-17T03:54:45.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.502+0000 7f0c2cc00640 1 -- 172.21.15.129:0/3754619726 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f0c200626c0 con 0x7f0c300737b0 2024-09-17T03:54:45.502 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:45.502 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:45.503 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.504+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 msgr2=0x7f0c0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.504+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 0x7f0c0c07b0a0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0c18033000 tx=0x7f0c18004730 comp rx=0 tx=0).stop 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 msgr2=0x7f0c30158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000b3f0 tx=0x7f0c2000b8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 shutdown_connections 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c0c078b90 0x7f0c0c07b0a0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3015f880 0x7f0c300726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c300cedf0 0x7f0c30144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 --2- 172.21.15.129:0/3754619726 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c300737b0 0x7f0c30158e10 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 >> 172.21.15.129:0/3754619726 conn(0x7f0c300b23c0 msgr2=0x7f0c300b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 shutdown_connections 2024-09-17T03:54:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:45.505+0000 7f0c37000640 1 -- 172.21.15.129:0/3754619726 wait complete. 2024-09-17T03:54:45.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:45.770 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1426,"msgr_send_messages":1426,"msgr_recv_bytes":376864,"msgr_send_bytes":4702314,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.203367204,"msgr_running_send_time":0.101821624,"msgr_running_recv_time":0.043501000,"msgr_running_fast_dispatch_time":0.044151982,"msgr_send_messages_queue_lat":{"avgcount":1425,"sum":0.268591193,"avgtime":0.000188485},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12579,"msgr_send_messages":24007,"msgr_recv_bytes":4534396,"msgr_send_bytes":28047362,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":2.009445019,"msgr_running_send_time":1.461130876,"msgr_running_recv_time":47.774851168,"msgr_running_fast_dispatch_time":0.041265630,"msgr_send_messages_queue_lat":{"avgcount":24006,"sum":1.822910232,"avgtime":0.000075935},"msgr_handle_ack_lat":{"avgcount":15522,"sum":0.003934829,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":120848,"msgr_send_encrypted_bytes":9360},"AsyncMessenger::Worker-2":{"msgr_recv_messages":994,"msgr_send_messages":1002,"msgr_recv_bytes":261209,"msgr_send_bytes":4344389,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.129088360,"msgr_running_send_time":0.065191703,"msgr_running_recv_time":0.025755798,"msgr_running_fast_dispatch_time":0.029862850,"msgr_send_messages_queue_lat":{"avgcount":1001,"sum":0.186225983,"avgtime":0.000186039},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":67008},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":148,"sum":7.446663242,"avgtime":0.050315292}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5347,"sum":1.357045892,"avgtime":0.000253795}},"mds":{"request":6411,"reply":6409,"reply_latency":{"avgcount":6409,"sum":9.770064755,"avgtime":0.001524428},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3687,"inodes_top":0,"inodes_bottom":3687,"inodes_pin_tail":0,"inodes_pinned":3687,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":7162,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4455,"handle_client_caps_dirty":3000,"handle_client_cap_release":656,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":136,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3065,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14625,"evex":14625,"evtrm":14625,"ev":14627,"evexg":0,"evexd":14624,"segadd":17,"segex":18,"segtrm":18,"seg":1,"segmjr":1,"segexg":0,"segexd":17,"expos":33099913,"wrpos":33103081,"rdpos":4195244,"jlat":{"avgcount":31,"sum":1.062933660,"avgtime":0.034288182},"replayed":3},"mds_mem":{"ino":149,"ino+":3214,"ino-":3065,"dir":13,"dir+":13,"dir-":0,"dn":147,"dn+":6412,"dn-":6265,"cap":137,"cap+":3202,"cap-":3065,"rss":178112,"heap":413093888},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.267729732,"avgtime":0.267729732},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5472,"average_load":5472,"avg_session_uptime":812,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8776288,"buffer_anon_items":3632,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":6960268,"mds_co_items":104794,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3419,"op_send_bytes":29789950,"op_resend":0,"op_reply":3419,"op_latency":{"avgcount":3419,"sum":11.074761444,"avgtime":0.003239181},"op_inflight":0,"oplen_avg":{"avgcount":3419,"sum":3802},"op":3419,"op_r":68,"op_w":3351,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":147,"osdop_read":40,"osdop_write":63,"osdop_writefull":19,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3088,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":294,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3065,"pq_executed":3065,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14995,"get_sum":4004518,"get_or_fail_fail":0,"get_or_fail_success":14995,"take":0,"take_sum":0,"put":14995,"put_sum":4004518,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":30100450,"put":3419,"put_sum":30100450,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":3419,"put":3419,"put_sum":3419,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3065,"get_sum":309565,"get_or_fail_fail":0,"get_or_fail_success":3065,"take":0,"take_sum":0,"put":32,"put_sum":309565,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":0,"max":3758096384,"get_started":0,"get":14625,"get_sum":28907578,"get_or_fail_fail":0,"get_or_fail_success":14625,"take":0,"take_sum":0,"put":31,"put_sum":28907578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:45.771 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 376864, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.044151982, "msgr_running_recv_time": 0.043501, "msgr_running_send_time": 0.101821624, "msgr_running_total_time": 0.203367204, "msgr_send_bytes": 4702314, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1426, "msgr_send_messages_queue_lat": { "avgcount": 1425, "avgtime": 0.000188485, "sum": 0.268591193 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15522, "avgtime": 2.53e-07, "sum": 0.003934829 }, "msgr_recv_bytes": 4534396, "msgr_recv_encrypted_bytes": 120848, "msgr_recv_messages": 12579, "msgr_running_fast_dispatch_time": 0.04126563, "msgr_running_recv_time": 47.774851168, "msgr_running_send_time": 1.461130876, "msgr_running_total_time": 2.009445019, "msgr_send_bytes": 28047362, "msgr_send_encrypted_bytes": 9360, "msgr_send_messages": 24007, "msgr_send_messages_queue_lat": { "avgcount": 24006, "avgtime": 7.5935e-05, "sum": 1.822910232 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 261209, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 994, "msgr_running_fast_dispatch_time": 0.02986285, "msgr_running_recv_time": 0.025755798, "msgr_running_send_time": 0.065191703, "msgr_running_total_time": 0.12908836, "msgr_send_bytes": 4344389, "msgr_send_encrypted_bytes": 67008, "msgr_send_messages": 1002, "msgr_send_messages_queue_lat": { "avgcount": 1001, "avgtime": 0.000186039, "sum": 0.186225983 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 148, "avgtime": 0.050315292, "sum": 7.446663242 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5347, "avgtime": 0.000253795, "sum": 1.357045892 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7162, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 656, "handle_client_caps": 4455, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3687, "inodes_bottom": 3687, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3687, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.001524428, "sum": 9.770064755 }, "request": 6411, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 136, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3065, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 14627, "evadd": 14625, "evex": 14625, "evexd": 14624, "evexg": 0, "evlrg": 0, "evtrm": 14625, "expos": 33099913, "jlat": { "avgcount": 31, "avgtime": 0.034288182, "sum": 1.06293366 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 18, "wrpos": 33103081 }, "mds_mem": { "cap": 137, "cap+": 3202, "cap-": 3065, "dir": 13, "dir+": 13, "dir-": 0, "dn": 147, "dn+": 6412, "dn-": 6265, "heap": 413093888, "ino": 149, "ino+": 3214, "ino-": 3065, "rss": 178112 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.267729732, "sum": 0.267729732 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5472, "avg_session_uptime": 812, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5472 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8776288, "buffer_anon_items": 3632, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 6960268, "mds_co_items": 104794, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 53, "op": 3419, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3419, "avgtime": 0.003239181, "sum": 11.074761444 }, "op_pg": 0, "op_r": 68, "op_reply": 3419, "op_resend": 0, "op_rmw": 0, "op_send": 3419, "op_send_bytes": 29789950, "op_w": 3351, "oplen_avg": { "avgcount": 3419, "sum": 3802 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 147, "osdop_delete": 3088, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 40, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 294, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 63, "osdop_writefull": 19, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3065, "pq_executed_ops": 3065, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14995, "get_or_fail_fail": 0, "get_or_fail_success": 14995, "get_started": 0, "get_sum": 4004518, "max": 104857600, "put": 14995, "put_sum": 4004518, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3419, "put_sum": 30100450, "take": 3419, "take_sum": 30100450, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3419, "put_sum": 3419, "take": 3419, "take_sum": 3419, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3065, "get_or_fail_fail": 0, "get_or_fail_success": 3065, "get_started": 0, "get_sum": 309565, "max": 3758096384, "put": 32, "put_sum": 309565, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14625, "get_or_fail_fail": 0, "get_or_fail_success": 14625, "get_started": 0, "get_sum": 28907578, "max": 3758096384, "put": 31, "put_sum": 28907578, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:45.772 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/136, 3065/3201 2024-09-17T03:54:46.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:54:46.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:54:46.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 -- 172.21.15.129:0/1982102430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 msgr2=0x7f46cc151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 --2- 172.21.15.129:0/1982102430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc151720 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0b67e0 tx=0x7f46b802ade0 comp rx=0 tx=0).stop 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 -- 172.21.15.129:0/1982102430 shutdown_connections 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 --2- 172.21.15.129:0/1982102430 >> v2:172.21.15.181:3300/0 conn(0x7f46cc151c60 0x7f46cc0726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 --2- 172.21.15.129:0/1982102430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc151720 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:46.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 --2- 172.21.15.129:0/1982102430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc0b4340 0x7f46cc06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.995+0000 7f46d1800640 1 -- 172.21.15.129:0/1982102430 >> 172.21.15.129:0/1982102430 conn(0x7f46cc0b23c0 msgr2=0x7f46cc0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.996+0000 7f46d1800640 1 -- 172.21.15.129:0/1982102430 shutdown_connections 2024-09-17T03:54:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.996+0000 7f46d1800640 1 -- 172.21.15.129:0/1982102430 wait complete. 2024-09-17T03:54:46.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.998+0000 7f46d1800640 1 Processor -- start 2024-09-17T03:54:46.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.998+0000 7f46d1800640 1 -- start start 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46cc0b4340 0x7f46cc1eaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc151c60 0x7f46cc0b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc072f70 con 0x7f46cc06fc50 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc072df0 con 0x7f46cc0b4340 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46d1800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc0730f0 con 0x7f46cc151c60 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:46.999+0000 7f46caa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:47.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47350/0 (socket says 172.21.15.129:47350) 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 -- 172.21.15.129:0/3720510386 learned_addr learned my addr 172.21.15.129:0/3720510386 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46ca000640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46cc0b4340 0x7f46cc1eaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46cb400640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc151c60 0x7f46cc0b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 -- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc151c60 msgr2=0x7f46cc0b09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc151c60 0x7f46cc0b09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 -- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46cc0b4340 msgr2=0x7f46cc1eaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46cc0b4340 0x7f46cc1eaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 -- 172.21.15.129:0/3720510386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc1f9ac0 con 0x7f46cc06fc50 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.000+0000 7f46caa00640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c970 tx=0x7f46bc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.001+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc007bb0 con 0x7f46cc06fc50 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.001+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46bc007e40 con 0x7f46cc06fc50 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.001+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc00f6b0 con 0x7f46cc06fc50 2024-09-17T03:54:47.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.001+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc1f9500 con 0x7f46cc06fc50 2024-09-17T03:54:47.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.001+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc1e8970 con 0x7f46cc06fc50 2024-09-17T03:54:47.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.002+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc06f710 con 0x7f46cc06fc50 2024-09-17T03:54:47.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.003+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f46bc00f850 con 0x7f46cc06fc50 2024-09-17T03:54:47.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.004+0000 7f46c2a00640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 0x7f46ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:47.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.005+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f46bc09b540 con 0x7f46cc06fc50 2024-09-17T03:54:47.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.005+0000 7f46ca000640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 0x7f46ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:47.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.006+0000 7f46ca000640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 0x7f46ac07b0a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f46b8034000 tx=0x7f46b8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:47.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.009+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f46bc014030 con 0x7f46cc06fc50 2024-09-17T03:54:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.227+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46cc15f440 con 0x7f46cc06fc50 2024-09-17T03:54:47.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.228+0000 7f46c2a00640 1 -- 172.21.15.129:0/3720510386 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f46bc063890 con 0x7f46cc06fc50 2024-09-17T03:54:47.228 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:47.228 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:47.228 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:47.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 msgr2=0x7f46ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:47.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 0x7f46ac07b0a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f46b8034000 tx=0x7f46b8004730 comp rx=0 tx=0).stop 2024-09-17T03:54:47.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 msgr2=0x7f46cc1f8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:47.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00c970 tx=0x7f46bc00ce30 comp rx=0 tx=0).stop 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 shutdown_connections 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46ac078b90 0x7f46ac07b0a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46cc151c60 0x7f46cc0b09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46cc0b4340 0x7f46cc1eaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 --2- 172.21.15.129:0/3720510386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46cc06fc50 0x7f46cc1f8b90 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 >> 172.21.15.129:0/3720510386 conn(0x7f46cc0b23c0 msgr2=0x7f46cc06eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 shutdown_connections 2024-09-17T03:54:47.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:47.230+0000 7f46d1800640 1 -- 172.21.15.129:0/3720510386 wait complete. 2024-09-17T03:54:47.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:47.492 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1426,"msgr_send_messages":1426,"msgr_recv_bytes":376864,"msgr_send_bytes":4702314,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.203367204,"msgr_running_send_time":0.101821624,"msgr_running_recv_time":0.043501000,"msgr_running_fast_dispatch_time":0.044151982,"msgr_send_messages_queue_lat":{"avgcount":1425,"sum":0.268591193,"avgtime":0.000188485},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12581,"msgr_send_messages":24007,"msgr_recv_bytes":4535200,"msgr_send_bytes":28047362,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":2.009728553,"msgr_running_send_time":1.461246104,"msgr_running_recv_time":47.774924213,"msgr_running_fast_dispatch_time":0.041354742,"msgr_send_messages_queue_lat":{"avgcount":24006,"sum":1.822910232,"avgtime":0.000075935},"msgr_handle_ack_lat":{"avgcount":15522,"sum":0.003934829,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":120848,"msgr_send_encrypted_bytes":9360},"AsyncMessenger::Worker-2":{"msgr_recv_messages":994,"msgr_send_messages":1002,"msgr_recv_bytes":261209,"msgr_send_bytes":4344389,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.129088360,"msgr_running_send_time":0.065191703,"msgr_running_recv_time":0.025755798,"msgr_running_fast_dispatch_time":0.029862850,"msgr_send_messages_queue_lat":{"avgcount":1001,"sum":0.186225983,"avgtime":0.000186039},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":67008},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":148,"sum":7.446663242,"avgtime":0.050315292}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5347,"sum":1.357045892,"avgtime":0.000253795}},"mds":{"request":6411,"reply":6409,"reply_latency":{"avgcount":6409,"sum":9.770064755,"avgtime":0.001524428},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":3687,"inodes_top":0,"inodes_bottom":3687,"inodes_pin_tail":0,"inodes_pinned":3687,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":7162,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4455,"handle_client_caps_dirty":3000,"handle_client_cap_release":656,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":136,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3065,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14625,"evex":14625,"evtrm":14625,"ev":14627,"evexg":0,"evexd":14624,"segadd":17,"segex":18,"segtrm":18,"seg":1,"segmjr":1,"segexg":0,"segexd":17,"expos":33099913,"wrpos":33103081,"rdpos":4195244,"jlat":{"avgcount":31,"sum":1.062933660,"avgtime":0.034288182},"replayed":3},"mds_mem":{"ino":149,"ino+":3214,"ino-":3065,"dir":13,"dir+":13,"dir-":0,"dn":147,"dn+":6412,"dn-":6265,"cap":137,"cap+":3202,"cap-":3065,"rss":156588,"heap":413093888},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.267729732,"avgtime":0.267729732},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5472,"average_load":5472,"avg_session_uptime":812,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":8776288,"buffer_anon_items":3632,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":6960268,"mds_co_items":104794,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3419,"op_send_bytes":29789950,"op_resend":0,"op_reply":3419,"op_latency":{"avgcount":3419,"sum":11.074761444,"avgtime":0.003239181},"op_inflight":0,"oplen_avg":{"avgcount":3419,"sum":3802},"op":3419,"op_r":68,"op_w":3351,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":147,"osdop_read":40,"osdop_write":63,"osdop_writefull":19,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3088,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":294,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3065,"pq_executed":3065,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14997,"get_sum":4005174,"get_or_fail_fail":0,"get_or_fail_success":14997,"take":0,"take_sum":0,"put":14997,"put_sum":4005174,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":30100450,"put":3419,"put_sum":30100450,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":3419,"put":3419,"put_sum":3419,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3065,"get_sum":309565,"get_or_fail_fail":0,"get_or_fail_success":3065,"take":0,"take_sum":0,"put":32,"put_sum":309565,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":0,"max":3758096384,"get_started":0,"get":14625,"get_sum":28907578,"get_or_fail_fail":0,"get_or_fail_success":14625,"take":0,"take_sum":0,"put":31,"put_sum":28907578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:47.494 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 376864, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.044151982, "msgr_running_recv_time": 0.043501, "msgr_running_send_time": 0.101821624, "msgr_running_total_time": 0.203367204, "msgr_send_bytes": 4702314, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1426, "msgr_send_messages_queue_lat": { "avgcount": 1425, "avgtime": 0.000188485, "sum": 0.268591193 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15522, "avgtime": 2.53e-07, "sum": 0.003934829 }, "msgr_recv_bytes": 4535200, "msgr_recv_encrypted_bytes": 120848, "msgr_recv_messages": 12581, "msgr_running_fast_dispatch_time": 0.041354742, "msgr_running_recv_time": 47.774924213, "msgr_running_send_time": 1.461246104, "msgr_running_total_time": 2.009728553, "msgr_send_bytes": 28047362, "msgr_send_encrypted_bytes": 9360, "msgr_send_messages": 24007, "msgr_send_messages_queue_lat": { "avgcount": 24006, "avgtime": 7.5935e-05, "sum": 1.822910232 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 261209, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 994, "msgr_running_fast_dispatch_time": 0.02986285, "msgr_running_recv_time": 0.025755798, "msgr_running_send_time": 0.065191703, "msgr_running_total_time": 0.12908836, "msgr_send_bytes": 4344389, "msgr_send_encrypted_bytes": 67008, "msgr_send_messages": 1002, "msgr_send_messages_queue_lat": { "avgcount": 1001, "avgtime": 0.000186039, "sum": 0.186225983 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 148, "avgtime": 0.050315292, "sum": 7.446663242 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5347, "avgtime": 0.000253795, "sum": 1.357045892 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7162, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 656, "handle_client_caps": 4455, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 3687, "inodes_bottom": 3687, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 3687, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.001524428, "sum": 9.770064755 }, "request": 6411, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 136, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3065, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 14627, "evadd": 14625, "evex": 14625, "evexd": 14624, "evexg": 0, "evlrg": 0, "evtrm": 14625, "expos": 33099913, "jlat": { "avgcount": 31, "avgtime": 0.034288182, "sum": 1.06293366 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 18, "wrpos": 33103081 }, "mds_mem": { "cap": 137, "cap+": 3202, "cap-": 3065, "dir": 13, "dir+": 13, "dir-": 0, "dn": 147, "dn+": 6412, "dn-": 6265, "heap": 413093888, "ino": 149, "ino+": 3214, "ino-": 3065, "rss": 156588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.267729732, "sum": 0.267729732 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5472, "avg_session_uptime": 812, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5472 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 8776288, "buffer_anon_items": 3632, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 6960268, "mds_co_items": 104794, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 53, "op": 3419, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3419, "avgtime": 0.003239181, "sum": 11.074761444 }, "op_pg": 0, "op_r": 68, "op_reply": 3419, "op_resend": 0, "op_rmw": 0, "op_send": 3419, "op_send_bytes": 29789950, "op_w": 3351, "oplen_avg": { "avgcount": 3419, "sum": 3802 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 147, "osdop_delete": 3088, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 40, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 294, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 63, "osdop_writefull": 19, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3065, "pq_executed_ops": 3065, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14997, "get_or_fail_fail": 0, "get_or_fail_success": 14997, "get_started": 0, "get_sum": 4005174, "max": 104857600, "put": 14997, "put_sum": 4005174, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3419, "put_sum": 30100450, "take": 3419, "take_sum": 30100450, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3419, "put_sum": 3419, "take": 3419, "take_sum": 3419, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3065, "get_or_fail_fail": 0, "get_or_fail_success": 3065, "get_started": 0, "get_sum": 309565, "max": 3758096384, "put": 32, "put_sum": 309565, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14625, "get_or_fail_fail": 0, "get_or_fail_success": 14625, "get_started": 0, "get_sum": 28907578, "max": 3758096384, "put": 31, "put_sum": 28907578, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:47.494 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/136, 3065/3201 2024-09-17T03:54:48.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:48.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 -- 172.21.15.129:0/2070411788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 msgr2=0x7febd0003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 --2- 172.21.15.129:0/2070411788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febd0003ee0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7febdc006fd0 tx=0x7febd0005830 comp rx=0 tx=0).stop 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 -- 172.21.15.129:0/2070411788 shutdown_connections 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 --2- 172.21.15.129:0/2070411788 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febd0003ee0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 -- 172.21.15.129:0/2070411788 >> 172.21.15.129:0/2070411788 conn(0x7febe806d560 msgr2=0x7febe80b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 -- 172.21.15.129:0/2070411788 shutdown_connections 2024-09-17T03:54:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.723+0000 7febef000640 1 -- 172.21.15.129:0/2070411788 wait complete. 2024-09-17T03:54:48.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.725+0000 7febef000640 1 Processor -- start 2024-09-17T03:54:48.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.726+0000 7febef000640 1 -- start start 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 0x7febe815f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febe815fdb0 0x7febe81f8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7febe81424f0 con 0x7febd0003a50 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7febe8142370 con 0x7febe815fdb0 2024-09-17T03:54:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febef000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7febe8142670 con 0x7febe809da40 2024-09-17T03:54:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febedc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febedc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47366/0 (socket says 172.21.15.129:47366) 2024-09-17T03:54:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febedc00640 1 -- 172.21.15.129:0/3991952143 learned_addr learned my addr 172.21.15.129:0/3991952143 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febed200640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 0x7febe815f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.727+0000 7febee600640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febe815fdb0 0x7febe81f8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 -- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 msgr2=0x7febe815f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 0x7febe815f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 -- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febe815fdb0 msgr2=0x7febe81f8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febe815fdb0 0x7febe81f8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 -- 172.21.15.129:0/3991952143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe81601c0 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febed200640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 0x7febe815f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febedc00640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7febdc000c00 tx=0x7febdc001ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc031040 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febdc006540 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc003a60 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe8161120 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.728+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe8152d70 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.729+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe8072390 con 0x7febd0003a50 2024-09-17T03:54:48.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.731+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7febdc034070 con 0x7febd0003a50 2024-09-17T03:54:48.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.732+0000 7febe5600640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 0x7febc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:48.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.732+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7febdc0ba830 con 0x7febd0003a50 2024-09-17T03:54:48.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.732+0000 7febed200640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 0x7febc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:48.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.733+0000 7febed200640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 0x7febc807b0a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7febd4005eb0 tx=0x7febd4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:48.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.736+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7febdc047070 con 0x7febd0003a50 2024-09-17T03:54:48.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.956+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febe80afe80 con 0x7febd0003a50 2024-09-17T03:54:48.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.957+0000 7febe5600640 1 -- 172.21.15.129:0/3991952143 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7febe81601c0 con 0x7febd0003a50 2024-09-17T03:54:48.957 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:48.957 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:48.957 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:48.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 msgr2=0x7febc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:48.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 0x7febc807b0a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7febd4005eb0 tx=0x7febd4005f90 comp rx=0 tx=0).stop 2024-09-17T03:54:48.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 msgr2=0x7febe81f8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:48.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7febdc000c00 tx=0x7febdc001ce0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 shutdown_connections 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febc8078b90 0x7febc807b0a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7febe815fdb0 0x7febe81f8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7febe809da40 0x7febe815f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 --2- 172.21.15.129:0/3991952143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7febd0003a50 0x7febe81f8660 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 >> 172.21.15.129:0/3991952143 conn(0x7febe806d560 msgr2=0x7febe80b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 shutdown_connections 2024-09-17T03:54:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:48.959+0000 7febef000640 1 -- 172.21.15.129:0/3991952143 wait complete. 2024-09-17T03:54:48.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:49.219 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1426,"msgr_send_messages":1426,"msgr_recv_bytes":376864,"msgr_send_bytes":4702314,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.203367204,"msgr_running_send_time":0.101821624,"msgr_running_recv_time":0.043501000,"msgr_running_fast_dispatch_time":0.044151982,"msgr_send_messages_queue_lat":{"avgcount":1425,"sum":0.268591193,"avgtime":0.000188485},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12583,"msgr_send_messages":24007,"msgr_recv_bytes":4536004,"msgr_send_bytes":28047362,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":2.009990731,"msgr_running_send_time":1.461361096,"msgr_running_recv_time":47.774993337,"msgr_running_fast_dispatch_time":0.041426841,"msgr_send_messages_queue_lat":{"avgcount":24006,"sum":1.822910232,"avgtime":0.000075935},"msgr_handle_ack_lat":{"avgcount":15522,"sum":0.003934829,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":120848,"msgr_send_encrypted_bytes":9360},"AsyncMessenger::Worker-2":{"msgr_recv_messages":994,"msgr_send_messages":1003,"msgr_recv_bytes":261209,"msgr_send_bytes":4346229,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.129157396,"msgr_running_send_time":0.065251921,"msgr_running_recv_time":0.025755798,"msgr_running_fast_dispatch_time":0.029862850,"msgr_send_messages_queue_lat":{"avgcount":1002,"sum":0.186259237,"avgtime":0.000185887},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":68848},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":148,"sum":7.446663242,"avgtime":0.050315292}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5347,"sum":1.357045892,"avgtime":0.000253795}},"mds":{"request":6411,"reply":6409,"reply_latency":{"avgcount":6409,"sum":9.770064755,"avgtime":0.001524428},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":147,"inodes_top":0,"inodes_bottom":147,"inodes_pin_tail":0,"inodes_pinned":147,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":7162,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4455,"handle_client_caps_dirty":3000,"handle_client_cap_release":656,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":136,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3065,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14625,"evex":14625,"evtrm":14625,"ev":14627,"evexg":0,"evexd":14624,"segadd":17,"segex":18,"segtrm":18,"seg":1,"segmjr":1,"segexg":0,"segexd":17,"expos":33099913,"wrpos":33103081,"rdpos":4195244,"jlat":{"avgcount":31,"sum":1.062933660,"avgtime":0.034288182},"replayed":3},"mds_mem":{"ino":149,"ino+":3214,"ino-":3065,"dir":13,"dir+":13,"dir-":0,"dn":147,"dn+":6412,"dn-":6265,"cap":137,"cap+":3202,"cap-":3065,"rss":156588,"heap":413093888},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":1,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.267729732,"avgtime":0.267729732},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5472,"average_load":5472,"avg_session_uptime":817,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248438,"buffer_anon_items":227,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":526105,"mds_co_items":10821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3419,"op_send_bytes":29789950,"op_resend":0,"op_reply":3419,"op_latency":{"avgcount":3419,"sum":11.074761444,"avgtime":0.003239181},"op_inflight":0,"oplen_avg":{"avgcount":3419,"sum":3802},"op":3419,"op_r":68,"op_w":3351,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":147,"osdop_read":40,"osdop_write":63,"osdop_writefull":19,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3088,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":294,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":53,"omap_rd":48,"omap_del":13},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3065,"pq_executed":3065,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":14999,"get_sum":4005830,"get_or_fail_fail":0,"get_or_fail_success":14999,"take":0,"take_sum":0,"put":14999,"put_sum":4005830,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":30100450,"put":3419,"put_sum":30100450,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3419,"take_sum":3419,"put":3419,"put_sum":3419,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3065,"get_sum":309565,"get_or_fail_fail":0,"get_or_fail_success":3065,"take":0,"take_sum":0,"put":32,"put_sum":309565,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":0,"max":3758096384,"get_started":0,"get":14625,"get_sum":28907578,"get_or_fail_fail":0,"get_or_fail_success":14625,"take":0,"take_sum":0,"put":31,"put_sum":28907578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:49.220 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 376864, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1426, "msgr_running_fast_dispatch_time": 0.044151982, "msgr_running_recv_time": 0.043501, "msgr_running_send_time": 0.101821624, "msgr_running_total_time": 0.203367204, "msgr_send_bytes": 4702314, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1426, "msgr_send_messages_queue_lat": { "avgcount": 1425, "avgtime": 0.000188485, "sum": 0.268591193 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15522, "avgtime": 2.53e-07, "sum": 0.003934829 }, "msgr_recv_bytes": 4536004, "msgr_recv_encrypted_bytes": 120848, "msgr_recv_messages": 12583, "msgr_running_fast_dispatch_time": 0.041426841, "msgr_running_recv_time": 47.774993337, "msgr_running_send_time": 1.461361096, "msgr_running_total_time": 2.009990731, "msgr_send_bytes": 28047362, "msgr_send_encrypted_bytes": 9360, "msgr_send_messages": 24007, "msgr_send_messages_queue_lat": { "avgcount": 24006, "avgtime": 7.5935e-05, "sum": 1.822910232 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 261209, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 994, "msgr_running_fast_dispatch_time": 0.02986285, "msgr_running_recv_time": 0.025755798, "msgr_running_send_time": 0.065251921, "msgr_running_total_time": 0.129157396, "msgr_send_bytes": 4346229, "msgr_send_encrypted_bytes": 68848, "msgr_send_messages": 1003, "msgr_send_messages_queue_lat": { "avgcount": 1002, "avgtime": 0.000185887, "sum": 0.186259237 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 148, "avgtime": 0.050315292, "sum": 7.446663242 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5347, "avgtime": 0.000253795, "sum": 1.357045892 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7162, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 656, "handle_client_caps": 4455, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 147, "inodes_bottom": 147, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 147, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.001524428, "sum": 9.770064755 }, "request": 6411, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 136, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3065, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 14627, "evadd": 14625, "evex": 14625, "evexd": 14624, "evexg": 0, "evlrg": 0, "evtrm": 14625, "expos": 33099913, "jlat": { "avgcount": 31, "avgtime": 0.034288182, "sum": 1.06293366 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 18, "wrpos": 33103081 }, "mds_mem": { "cap": 137, "cap+": 3202, "cap-": 3065, "dir": 13, "dir+": 13, "dir-": 0, "dn": 147, "dn+": 6412, "dn-": 6265, "heap": 413093888, "ino": 149, "ino+": 3214, "ino-": 3065, "rss": 156588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 1, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.267729732, "sum": 0.267729732 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5472, "avg_session_uptime": 817, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5472 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248438, "buffer_anon_items": 227, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 526105, "mds_co_items": 10821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 13, "omap_rd": 48, "omap_wr": 53, "op": 3419, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3419, "avgtime": 0.003239181, "sum": 11.074761444 }, "op_pg": 0, "op_r": 68, "op_reply": 3419, "op_resend": 0, "op_rmw": 0, "op_send": 3419, "op_send_bytes": 29789950, "op_w": 3351, "oplen_avg": { "avgcount": 3419, "sum": 3802 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 147, "osdop_delete": 3088, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 40, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 294, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 63, "osdop_writefull": 19, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3065, "pq_executed_ops": 3065, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 14999, "get_or_fail_fail": 0, "get_or_fail_success": 14999, "get_started": 0, "get_sum": 4005830, "max": 104857600, "put": 14999, "put_sum": 4005830, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3419, "put_sum": 30100450, "take": 3419, "take_sum": 30100450, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3419, "put_sum": 3419, "take": 3419, "take_sum": 3419, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3065, "get_or_fail_fail": 0, "get_or_fail_success": 3065, "get_started": 0, "get_sum": 309565, "max": 3758096384, "put": 32, "put_sum": 309565, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14625, "get_or_fail_fail": 0, "get_or_fail_success": 14625, "get_started": 0, "get_sum": 28907578, "max": 3758096384, "put": 31, "put_sum": 28907578, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:49.220 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/136, 3065/3201 2024-09-17T03:54:50.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b36800640 1 -- 172.21.15.129:0/2713356901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 msgr2=0x7f2b14003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b36800640 1 --2- 172.21.15.129:0/2713356901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b14003ec0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009b50 tx=0x7f2b14005830 comp rx=0 tx=0).stop 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b2fe00640 1 -- 172.21.15.129:0/2713356901 reap_dead start 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b36800640 1 -- 172.21.15.129:0/2713356901 shutdown_connections 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b36800640 1 -- 172.21.15.129:0/2713356901 >> 172.21.15.129:0/2713356901 conn(0x7f2b300b23c0 msgr2=0x7f2b300b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.443+0000 7f2b36800640 1 -- 172.21.15.129:0/2713356901 shutdown_connections 2024-09-17T03:54:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.444+0000 7f2b36800640 1 -- 172.21.15.129:0/2713356901 wait complete. 2024-09-17T03:54:50.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.446+0000 7f2b36800640 1 Processor -- start 2024-09-17T03:54:50.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.446+0000 7f2b36800640 1 -- start start 2024-09-17T03:54:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b30145d90 0x7f2b30153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b301540a0 0x7f2b301f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2b300cf590 con 0x7f2b14003a50 2024-09-17T03:54:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2b300cf410 con 0x7f2b30145d90 2024-09-17T03:54:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b36800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2b300cf710 con 0x7f2b301540a0 2024-09-17T03:54:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.447+0000 7f2b2f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47396/0 (socket says 172.21.15.129:47396) 2024-09-17T03:54:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 -- 172.21.15.129:0/1882462571 learned_addr learned my addr 172.21.15.129:0/1882462571 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b26a00640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b30145d90 0x7f2b30153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2fe00640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b301540a0 0x7f2b301f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 -- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b301540a0 msgr2=0x7f2b301f8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b301540a0 0x7f2b301f8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 -- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b30145d90 msgr2=0x7f2b30153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:50.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b30145d90 0x7f2b30153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 -- 172.21.15.129:0/1882462571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b301544b0 con 0x7f2b14003a50 2024-09-17T03:54:50.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b2f400640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009a70 tx=0x7f2b20033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:50.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.448+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20038040 con 0x7f2b14003a50 2024-09-17T03:54:50.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.449+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b20004110 con 0x7f2b14003a50 2024-09-17T03:54:50.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.449+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b200337e0 con 0x7f2b14003a50 2024-09-17T03:54:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.449+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b30156f40 con 0x7f2b14003a50 2024-09-17T03:54:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.449+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b301574c0 con 0x7f2b14003a50 2024-09-17T03:54:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.450+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b30073bc0 con 0x7f2b14003a50 2024-09-17T03:54:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.451+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2b20045020 con 0x7f2b14003a50 2024-09-17T03:54:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.452+0000 7f2b27e00640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 0x7f2b0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.452+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f2b200bb8a0 con 0x7f2b14003a50 2024-09-17T03:54:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.453+0000 7f2b26a00640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 0x7f2b0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.453+0000 7f2b26a00640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 0x7f2b0c07b0a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2b18001a30 tx=0x7f2b18001110 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.456+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2b301544b0 con 0x7f2b14003a50 2024-09-17T03:54:50.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.675+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b30069b10 con 0x7f2b14003a50 2024-09-17T03:54:50.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.676+0000 7f2b27e00640 1 -- 172.21.15.129:0/1882462571 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f2b20083be0 con 0x7f2b14003a50 2024-09-17T03:54:50.676 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:50.676 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:50.677 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:50.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 msgr2=0x7f2b0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 0x7f2b0c07b0a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2b18001a30 tx=0x7f2b18001110 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 msgr2=0x7f2b30155c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2b20009a70 tx=0x7f2b20033670 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 shutdown_connections 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b0c078b90 0x7f2b0c07b0a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b301540a0 0x7f2b301f8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b30145d90 0x7f2b30153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 --2- 172.21.15.129:0/1882462571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b14003a50 0x7f2b30155c20 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.678+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 >> 172.21.15.129:0/1882462571 conn(0x7f2b300b23c0 msgr2=0x7f2b300b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.679+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 shutdown_connections 2024-09-17T03:54:50.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:50.679+0000 7f2b36800640 1 -- 172.21.15.129:0/1882462571 wait complete. 2024-09-17T03:54:50.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:50.940 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1482,"msgr_send_messages":1482,"msgr_recv_bytes":391088,"msgr_send_bytes":4720570,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.209685891,"msgr_running_send_time":0.104881057,"msgr_running_recv_time":0.044872171,"msgr_running_fast_dispatch_time":0.045614332,"msgr_send_messages_queue_lat":{"avgcount":1481,"sum":0.271024534,"avgtime":0.000183001},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12635,"msgr_send_messages":24056,"msgr_recv_bytes":4566386,"msgr_send_bytes":28077304,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":2.016464086,"msgr_running_send_time":1.464525682,"msgr_running_recv_time":47.776669593,"msgr_running_fast_dispatch_time":0.042898000,"msgr_send_messages_queue_lat":{"avgcount":24055,"sum":1.824958777,"avgtime":0.000075866},"msgr_handle_ack_lat":{"avgcount":15523,"sum":0.003935424,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":121120,"msgr_send_encrypted_bytes":10000},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1035,"msgr_send_messages":1044,"msgr_recv_bytes":271615,"msgr_send_bytes":4359677,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.133342580,"msgr_running_send_time":0.067147503,"msgr_running_recv_time":0.026675498,"msgr_running_fast_dispatch_time":0.030910844,"msgr_send_messages_queue_lat":{"avgcount":1043,"sum":0.187861830,"avgtime":0.000180116},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":68848},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":149,"sum":7.446672452,"avgtime":0.049977667}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5604,"sum":1.371909955,"avgtime":0.000244809}},"mds":{"request":6411,"reply":6409,"reply_latency":{"avgcount":6409,"sum":9.770064755,"avgtime":0.001524428},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":147,"inodes_top":0,"inodes_bottom":147,"inodes_pin_tail":0,"inodes_pinned":147,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":7162,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4455,"handle_client_caps_dirty":3000,"handle_client_cap_release":657,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14761,"evex":14625,"evtrm":14625,"ev":139,"evexg":0,"evexd":14624,"segadd":17,"segex":18,"segtrm":18,"seg":1,"segmjr":1,"segexg":0,"segexd":17,"expos":33099913,"wrpos":33185394,"rdpos":4195244,"jlat":{"avgcount":31,"sum":1.062933660,"avgtime":0.034288182},"replayed":3},"mds_mem":{"ino":149,"ino+":3214,"ino-":3065,"dir":12,"dir+":13,"dir-":1,"dn":147,"dn+":6412,"dn-":6265,"cap":1,"cap+":3202,"cap-":3201,"rss":156588,"heap":413093888},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.267729732,"avgtime":0.267729732},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5472,"average_load":5472,"avg_session_uptime":817,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248438,"buffer_anon_items":227,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":526105,"mds_co_items":10821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3563,"op_send_bytes":29803893,"op_resend":0,"op_reply":3563,"op_latency":{"avgcount":3563,"sum":11.374754980,"avgtime":0.003192465},"op_inflight":0,"oplen_avg":{"avgcount":3563,"sum":3948},"op":3563,"op_r":71,"op_w":3492,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":147,"osdop_read":43,"osdop_write":66,"osdop_writefull":20,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3224,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":294,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":14},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15148,"get_sum":4047444,"get_or_fail_fail":0,"get_or_fail_success":15148,"take":0,"take_sum":0,"put":15148,"put_sum":4047444,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3563,"take_sum":30128129,"put":3563,"put_sum":30128129,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3563,"take_sum":3563,"put":3563,"put_sum":3563,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":35,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":82313,"max":3758096384,"get_started":0,"get":14761,"get_sum":28989891,"get_or_fail_fail":0,"get_or_fail_success":14761,"take":0,"take_sum":0,"put":31,"put_sum":28907578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:50.942 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 391088, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1482, "msgr_running_fast_dispatch_time": 0.045614332, "msgr_running_recv_time": 0.044872171, "msgr_running_send_time": 0.104881057, "msgr_running_total_time": 0.209685891, "msgr_send_bytes": 4720570, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1482, "msgr_send_messages_queue_lat": { "avgcount": 1481, "avgtime": 0.000183001, "sum": 0.271024534 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15523, "avgtime": 2.53e-07, "sum": 0.003935424 }, "msgr_recv_bytes": 4566386, "msgr_recv_encrypted_bytes": 121120, "msgr_recv_messages": 12635, "msgr_running_fast_dispatch_time": 0.042898, "msgr_running_recv_time": 47.776669593, "msgr_running_send_time": 1.464525682, "msgr_running_total_time": 2.016464086, "msgr_send_bytes": 28077304, "msgr_send_encrypted_bytes": 10000, "msgr_send_messages": 24056, "msgr_send_messages_queue_lat": { "avgcount": 24055, "avgtime": 7.5866e-05, "sum": 1.824958777 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 271615, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 1035, "msgr_running_fast_dispatch_time": 0.030910844, "msgr_running_recv_time": 0.026675498, "msgr_running_send_time": 0.067147503, "msgr_running_total_time": 0.13334258, "msgr_send_bytes": 4359677, "msgr_send_encrypted_bytes": 68848, "msgr_send_messages": 1044, "msgr_send_messages_queue_lat": { "avgcount": 1043, "avgtime": 0.000180116, "sum": 0.18786183 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 149, "avgtime": 0.049977667, "sum": 7.446672452 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5604, "avgtime": 0.000244809, "sum": 1.371909955 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7162, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 657, "handle_client_caps": 4455, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 147, "inodes_bottom": 147, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 147, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.001524428, "sum": 9.770064755 }, "request": 6411, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 139, "evadd": 14761, "evex": 14625, "evexd": 14624, "evexg": 0, "evlrg": 0, "evtrm": 14625, "expos": 33099913, "jlat": { "avgcount": 31, "avgtime": 0.034288182, "sum": 1.06293366 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 18, "wrpos": 33185394 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 147, "dn+": 6412, "dn-": 6265, "heap": 413093888, "ino": 149, "ino+": 3214, "ino-": 3065, "rss": 156588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.267729732, "sum": 0.267729732 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5472, "avg_session_uptime": 817, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5472 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248438, "buffer_anon_items": 227, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 526105, "mds_co_items": 10821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 14, "omap_rd": 48, "omap_wr": 55, "op": 3563, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3563, "avgtime": 0.003192465, "sum": 11.37475498 }, "op_pg": 0, "op_r": 71, "op_reply": 3563, "op_resend": 0, "op_rmw": 0, "op_send": 3563, "op_send_bytes": 29803893, "op_w": 3492, "oplen_avg": { "avgcount": 3563, "sum": 3948 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 147, "osdop_delete": 3224, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 43, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 294, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 66, "osdop_writefull": 20, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15148, "get_or_fail_fail": 0, "get_or_fail_success": 15148, "get_started": 0, "get_sum": 4047444, "max": 104857600, "put": 15148, "put_sum": 4047444, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3563, "put_sum": 30128129, "take": 3563, "take_sum": 30128129, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3563, "put_sum": 3563, "take": 3563, "take_sum": 3563, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 35, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14761, "get_or_fail_fail": 0, "get_or_fail_success": 14761, "get_started": 0, "get_sum": 28989891, "max": 3758096384, "put": 31, "put_sum": 28907578, "take": 0, "take_sum": 0, "val": 82313, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:50.942 INFO:tasks.cephfs.test_strays:Complete purge in 12 seconds 2024-09-17T03:54:50.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.162+0000 7fb1ae600640 1 -- 172.21.15.129:0/132982534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a815cf70 msgr2=0x7fb1a8072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.162+0000 7fb1ae600640 1 --2- 172.21.15.129:0/132982534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a815cf70 0x7fb1a8072d70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb194009860 tx=0x7fb19402a4a0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.163+0000 7fb1ae600640 1 -- 172.21.15.129:0/132982534 shutdown_connections 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.163+0000 7fb1ae600640 1 --2- 172.21.15.129:0/132982534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a815cf70 0x7fb1a8072d70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.163+0000 7fb1ae600640 1 -- 172.21.15.129:0/132982534 >> 172.21.15.129:0/132982534 conn(0x7fb1a80b23c0 msgr2=0x7fb1a80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.163+0000 7fb1ae600640 1 -- 172.21.15.129:0/132982534 shutdown_connections 2024-09-17T03:54:51.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.163+0000 7fb1ae600640 1 -- 172.21.15.129:0/132982534 wait complete. 2024-09-17T03:54:51.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.165+0000 7fb1ae600640 1 Processor -- start 2024-09-17T03:54:51.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.165+0000 7fb1ae600640 1 -- start start 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 0x7fb1a81f9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a80cd190 0x7fb1a815c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a80700a0 con 0x7fb1a815cf70 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a806ff20 con 0x7fb1a80b59d0 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.166+0000 7fb1ae600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a8070220 con 0x7fb1a80cd190 2024-09-17T03:54:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47400/0 (socket says 172.21.15.129:47400) 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 -- 172.21.15.129:0/4114041872 learned_addr learned my addr 172.21.15.129:0/4114041872 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a6a00640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a80cd190 0x7fb1a815c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7400640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 0x7fb1a81f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 -- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a80cd190 msgr2=0x7fb1a815c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a80cd190 0x7fb1a815c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 -- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 msgr2=0x7fb1a81f9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 0x7fb1a81f9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 -- 172.21.15.129:0/4114041872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a81f9ca0 con 0x7fb1a815cf70 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7400640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 0x7fb1a81f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.167+0000 7fb1a7e00640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0002230 tx=0x7fb1a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:51.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.168+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0016020 con 0x7fb1a815cf70 2024-09-17T03:54:51.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.168+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1a000ae30 con 0x7fb1a815cf70 2024-09-17T03:54:51.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.168+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0012680 con 0x7fb1a815cf70 2024-09-17T03:54:51.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.168+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a81386b0 con 0x7fb1a815cf70 2024-09-17T03:54:51.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.168+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a8138c70 con 0x7fb1a815cf70 2024-09-17T03:54:51.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.169+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a8138420 con 0x7fb1a815cf70 2024-09-17T03:54:51.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.170+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb1a0019020 con 0x7fb1a815cf70 2024-09-17T03:54:51.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.172+0000 7fb19f400640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 0x7fb18807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:51.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.172+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fb1a009b450 con 0x7fb1a815cf70 2024-09-17T03:54:51.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.172+0000 7fb1a7400640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 0x7fb18807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:51.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.173+0000 7fb1a7400640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 0x7fb18807b0a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb198001120 tx=0x7fb198009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:51.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.176+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb1a0020020 con 0x7fb1a815cf70 2024-09-17T03:54:51.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.392+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1a8145490 con 0x7fb1a815cf70 2024-09-17T03:54:51.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.393+0000 7fb19f400640 1 -- 172.21.15.129:0/4114041872 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7fb1a0063820 con 0x7fb1a815cf70 2024-09-17T03:54:51.393 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:51.393 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:51.393 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 msgr2=0x7fb18807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 0x7fb18807b0a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb198001120 tx=0x7fb198009040 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 msgr2=0x7fb1a80b0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0002230 tx=0x7fb1a000a430 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 shutdown_connections 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb188078b90 0x7fb18807b0a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1a815cf70 0x7fb1a80b0d40 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1a80cd190 0x7fb1a815c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 --2- 172.21.15.129:0/4114041872 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1a80b59d0 0x7fb1a81f9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.395+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 >> 172.21.15.129:0/4114041872 conn(0x7fb1a80b23c0 msgr2=0x7fb1a80ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:51.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.396+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 shutdown_connections 2024-09-17T03:54:51.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:51.396+0000 7fb1ae600640 1 -- 172.21.15.129:0/4114041872 wait complete. 2024-09-17T03:54:51.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T03:54:51.657 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1482,"msgr_send_messages":1482,"msgr_recv_bytes":391088,"msgr_send_bytes":4720570,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.209685891,"msgr_running_send_time":0.104881057,"msgr_running_recv_time":0.044872171,"msgr_running_fast_dispatch_time":0.045614332,"msgr_send_messages_queue_lat":{"avgcount":1481,"sum":0.271024534,"avgtime":0.000183001},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000276,"avgtime":0.000000138},"msgr_recv_encrypted_bytes":1872,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":12635,"msgr_send_messages":24056,"msgr_recv_bytes":4566386,"msgr_send_bytes":28077304,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":2.016540961,"msgr_running_send_time":1.464556496,"msgr_running_recv_time":47.776688100,"msgr_running_fast_dispatch_time":0.042898000,"msgr_send_messages_queue_lat":{"avgcount":24055,"sum":1.824958777,"avgtime":0.000075866},"msgr_handle_ack_lat":{"avgcount":15523,"sum":0.003935424,"avgtime":0.000000253},"msgr_recv_encrypted_bytes":121120,"msgr_send_encrypted_bytes":10000},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1035,"msgr_send_messages":1044,"msgr_recv_bytes":271615,"msgr_send_bytes":4359677,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.133342580,"msgr_running_send_time":0.067147503,"msgr_running_recv_time":0.026675498,"msgr_running_fast_dispatch_time":0.030910844,"msgr_send_messages_queue_lat":{"avgcount":1043,"sum":0.187861830,"avgtime":0.000180116},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":68848},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":149,"sum":7.446672452,"avgtime":0.049977667}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":5604,"sum":1.371909955,"avgtime":0.000244809}},"mds":{"request":6411,"reply":6409,"reply_latency":{"avgcount":6409,"sum":9.770064755,"avgtime":0.001524428},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":147,"inodes_top":0,"inodes_bottom":147,"inodes_pin_tail":0,"inodes_pinned":147,"inodes_expired":0,"inodes_with_caps":137,"caps":137,"subtrees":2,"traverse":6412,"traverse_hit":6411,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":9,"ceph_cap_op_grant":7162,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4455,"handle_client_caps_dirty":3000,"handle_client_cap_release":657,"process_request_cap_release":3210},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3201,"strays_enqueued":3201,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":14761,"evex":14625,"evtrm":14625,"ev":139,"evexg":0,"evexd":14624,"segadd":17,"segex":18,"segtrm":18,"seg":1,"segmjr":1,"segexg":0,"segexd":17,"expos":33099913,"wrpos":33185394,"rdpos":4195244,"jlat":{"avgcount":31,"sum":1.062933660,"avgtime":0.034288182},"replayed":3},"mds_mem":{"ino":149,"ino+":3214,"ino-":3065,"dir":12,"dir+":13,"dir-":1,"dn":147,"dn+":6412,"dn-":6265,"cap":1,"cap+":3202,"cap-":3201,"rss":156588,"heap":413093888},"mds_server":{"dispatch_client_request":6415,"dispatch_server_request":0,"handle_client_request":6411,"handle_client_session":2,"handle_peer_request":0,"req_create_latency":{"avgcount":3200,"sum":3.272573244,"avgtime":0.001022679},"req_getattr_latency":{"avgcount":1,"sum":0.000344132,"avgtime":0.000344132},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003241189,"avgtime":0.003241189},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":6,"sum":0.105773658,"avgtime":0.017628943},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.267729732,"avgtime":0.267729732},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":3200,"sum":6.120402800,"avgtime":0.001912625},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":5472,"average_load":5472,"avg_session_uptime":817,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248438,"buffer_anon_items":227,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5544,"osdmap_items":90,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":526105,"mds_co_items":10821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":3563,"op_send_bytes":29803893,"op_resend":0,"op_reply":3563,"op_latency":{"avgcount":3563,"sum":11.374754980,"avgtime":0.003192465},"op_inflight":0,"oplen_avg":{"avgcount":3563,"sum":3948},"op":3563,"op_r":71,"op_w":3492,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":147,"osdop_read":43,"osdop_write":66,"osdop_writefull":20,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3224,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":294,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":0,"map_full":0,"map_inc":0,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":55,"omap_rd":48,"omap_del":14},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":6,"pq_executing":0,"pq_executing_high_water":6,"pq_executed_ops":3201,"pq_executed":3201,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":15148,"get_sum":4047444,"get_or_fail_fail":0,"get_or_fail_success":15148,"take":0,"take_sum":0,"put":15148,"put_sum":4047444,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3563,"take_sum":30128129,"put":3563,"put_sum":30128129,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3563,"take_sum":3563,"put":3563,"put_sum":3563,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":3201,"get_sum":323301,"get_or_fail_fail":0,"get_or_fail_success":3201,"take":0,"take_sum":0,"put":35,"put_sum":323301,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5635498a9a00":{"val":82313,"max":3758096384,"get_started":0,"get":14761,"get_sum":28989891,"get_or_fail_fail":0,"get_or_fail_success":14761,"take":0,"take_sum":0,"put":31,"put_sum":28907578,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T03:54:51.658 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.38e-07, "sum": 2.76e-07 }, "msgr_recv_bytes": 391088, "msgr_recv_encrypted_bytes": 1872, "msgr_recv_messages": 1482, "msgr_running_fast_dispatch_time": 0.045614332, "msgr_running_recv_time": 0.044872171, "msgr_running_send_time": 0.104881057, "msgr_running_total_time": 0.209685891, "msgr_send_bytes": 4720570, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 1482, "msgr_send_messages_queue_lat": { "avgcount": 1481, "avgtime": 0.000183001, "sum": 0.271024534 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 15523, "avgtime": 2.53e-07, "sum": 0.003935424 }, "msgr_recv_bytes": 4566386, "msgr_recv_encrypted_bytes": 121120, "msgr_recv_messages": 12635, "msgr_running_fast_dispatch_time": 0.042898, "msgr_running_recv_time": 47.7766881, "msgr_running_send_time": 1.464556496, "msgr_running_total_time": 2.016540961, "msgr_send_bytes": 28077304, "msgr_send_encrypted_bytes": 10000, "msgr_send_messages": 24056, "msgr_send_messages_queue_lat": { "avgcount": 24055, "avgtime": 7.5866e-05, "sum": 1.824958777 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 271615, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 1035, "msgr_running_fast_dispatch_time": 0.030910844, "msgr_running_recv_time": 0.026675498, "msgr_running_send_time": 0.067147503, "msgr_running_total_time": 0.13334258, "msgr_send_bytes": 4359677, "msgr_send_encrypted_bytes": 68848, "msgr_send_messages": 1044, "msgr_send_messages_queue_lat": { "avgcount": 1043, "avgtime": 0.000180116, "sum": 0.18786183 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 149, "avgtime": 0.049977667, "sum": 7.446672452 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 5604, "avgtime": 0.000244809, "sum": 1.371909955 }, "queue_len": 0 }, "mds": { "caps": 137, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 7162, "ceph_cap_op_revoke": 9, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 657, "handle_client_caps": 4455, "handle_client_caps_dirty": 3000, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 147, "inodes_bottom": 147, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 147, "inodes_top": 0, "inodes_with_caps": 137, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 3210, "q": 0, "reply": 6409, "reply_latency": { "avgcount": 6409, "avgtime": 0.001524428, "sum": 9.770064755 }, "request": 6411, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 6412, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 6411, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3201, "strays_enqueued": 3201, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 139, "evadd": 14761, "evex": 14625, "evexd": 14624, "evexg": 0, "evlrg": 0, "evtrm": 14625, "expos": 33099913, "jlat": { "avgcount": 31, "avgtime": 0.034288182, "sum": 1.06293366 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 17, "segex": 18, "segexd": 17, "segexg": 0, "segmjr": 1, "segtrm": 18, "wrpos": 33185394 }, "mds_mem": { "cap": 1, "cap+": 3202, "cap-": 3201, "dir": 12, "dir+": 13, "dir-": 1, "dn": 147, "dn+": 6412, "dn-": 6265, "heap": 413093888, "ino": 149, "ino+": 3214, "ino-": 3065, "rss": 156588 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 6415, "dispatch_server_request": 0, "handle_client_request": 6411, "handle_client_session": 2, "handle_peer_request": 0, "req_create_latency": { "avgcount": 3200, "avgtime": 0.001022679, "sum": 3.272573244 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000344132, "sum": 0.000344132 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.003241189, "sum": 0.003241189 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 6, "avgtime": 0.017628943, "sum": 0.105773658 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.267729732, "sum": 0.267729732 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 3200, "avgtime": 0.001912625, "sum": 6.1204028 } }, "mds_sessions": { "average_load": 5472, "avg_session_uptime": 817, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 5472 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248438, "buffer_anon_items": 227, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 526105, "mds_co_items": 10821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5544, "osdmap_items": 90, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 0, "map_full": 0, "map_inc": 0, "omap_del": 14, "omap_rd": 48, "omap_wr": 55, "op": 3563, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 3563, "avgtime": 0.003192465, "sum": 11.37475498 }, "op_pg": 0, "op_r": 71, "op_reply": 3563, "op_resend": 0, "op_rmw": 0, "op_send": 3563, "op_send_bytes": 29803893, "op_w": 3492, "oplen_avg": { "avgcount": 3563, "sum": 3948 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 147, "osdop_delete": 3224, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 43, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 294, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 66, "osdop_writefull": 20, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 3201, "pq_executed_ops": 3201, "pq_executing": 0, "pq_executing_high_water": 6, "pq_executing_ops": 0, "pq_executing_ops_high_water": 6, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 15148, "get_or_fail_fail": 0, "get_or_fail_success": 15148, "get_started": 0, "get_sum": 4047444, "max": 104857600, "put": 15148, "put_sum": 4047444, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3563, "put_sum": 30128129, "take": 3563, "take_sum": 30128129, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3563, "put_sum": 3563, "take": 3563, "take_sum": 3563, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 3201, "get_or_fail_fail": 0, "get_or_fail_success": 3201, "get_started": 0, "get_sum": 323301, "max": 3758096384, "put": 35, "put_sum": 323301, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5635498a9a00": { "get": 14761, "get_or_fail_fail": 0, "get_or_fail_success": 14761, "get_started": 0, "get_sum": 28989891, "max": 3758096384, "put": 31, "put_sum": 28907578, "take": 0, "take_sum": 0, "val": 82313, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T03:54:51.659 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:54:51.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:54:51.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:54:51.852 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:54:51.852 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:54:51.852 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:51.852 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:51.889 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:51.889 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:51.965 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.224+0000 7f3b42600640 1 -- 172.21.15.129:0/2330567876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b20005320 msgr2=0x7f3b2000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.224+0000 7f3b42600640 1 --2- 172.21.15.129:0/2330567876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b20005320 0x7f3b2000b9c0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400c610 tx=0x7f3b20005830 comp rx=0 tx=0).stop 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.224+0000 7f3b3be00640 1 -- 172.21.15.129:0/2330567876 reap_dead start 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.225+0000 7f3b42600640 1 -- 172.21.15.129:0/2330567876 shutdown_connections 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.225+0000 7f3b42600640 1 -- 172.21.15.129:0/2330567876 >> 172.21.15.129:0/2330567876 conn(0x7f3b3c0b23c0 msgr2=0x7f3b3c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:52.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.225+0000 7f3b42600640 1 -- 172.21.15.129:0/2330567876 shutdown_connections 2024-09-17T03:54:52.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.225+0000 7f3b42600640 1 -- 172.21.15.129:0/2330567876 wait complete. 2024-09-17T03:54:52.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.227+0000 7f3b42600640 1 Processor -- start 2024-09-17T03:54:52.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.227+0000 7f3b42600640 1 -- start start 2024-09-17T03:54:52.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b20005320 0x7f3b3c0ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b3c0b0430 0x7f3b3c0b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c0733a0 con 0x7f3b3c13f080 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c073220 con 0x7f3b20005320 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.228+0000 7f3b42600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c06fcd0 con 0x7f3b3c0b0430 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47408/0 (socket says 172.21.15.129:47408) 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 -- 172.21.15.129:0/1236001995 learned_addr learned my addr 172.21.15.129:0/1236001995 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b3b400640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b20005320 0x7f3b3c0ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b3be00640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b3c0b0430 0x7f3b3c0b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 -- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b3c0b0430 msgr2=0x7f3b3c0b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b3c0b0430 0x7f3b3c0b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 -- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b20005320 msgr2=0x7f3b3c0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b20005320 0x7f3b3c0ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 -- 172.21.15.129:0/1236001995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c0ab5f0 con 0x7f3b3c13f080 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.229+0000 7f3b33e00640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3b24011a60 tx=0x7f3b24011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.230+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b2401f070 con 0x7f3b3c13f080 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.230+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b24012450 con 0x7f3b3c13f080 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.230+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b3c0ab8d0 con 0x7f3b3c13f080 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.230+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24012860 con 0x7f3b3c13f080 2024-09-17T03:54:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.230+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b3c0abe90 con 0x7f3b3c13f080 2024-09-17T03:54:52.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.231+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b3c0ab360 con 0x7f3b3c13f080 2024-09-17T03:54:52.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.232+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3b24022070 con 0x7f3b3c13f080 2024-09-17T03:54:52.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.233+0000 7f3b33400640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 0x7f3b1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.233+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f3b2409ba30 con 0x7f3b3c13f080 2024-09-17T03:54:52.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.233+0000 7f3b3b400640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 0x7f3b1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.234+0000 7f3b3b400640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 0x7f3b1807b0a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c006fd0 tx=0x7f3b2c006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:52.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.237+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3b24063720 con 0x7f3b3c13f080 2024-09-17T03:54:52.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.431+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3b3c1462b0 con 0x7f3b3c13f080 2024-09-17T03:54:52.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.431+0000 7f3b33400640 1 -- 172.21.15.129:0/1236001995 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v96) ==== 80+0+124 (secure 0 0 0) 0x7f3b3c1462b0 con 0x7f3b3c13f080 2024-09-17T03:54:52.432 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6834/1986086177 2024-09-18T03:54:17.549442+0000 2024-09-17T03:54:52.432 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6833/1986086177 2024-09-18T03:54:17.549442+0000 2024-09-17T03:54:52.432 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 msgr2=0x7f3b1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 0x7f3b1807b0a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c006fd0 tx=0x7f3b2c006b10 comp rx=0 tx=0).stop 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 msgr2=0x7f3b3c0afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f3b24011a60 tx=0x7f3b24011f20 comp rx=0 tx=0).stop 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 shutdown_connections 2024-09-17T03:54:52.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b18078b90 0x7f3b1807b0a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b3c0b0430 0x7f3b3c0b0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b3c13f080 0x7f3b3c0afef0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 --2- 172.21.15.129:0/1236001995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b20005320 0x7f3b3c0ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 >> 172.21.15.129:0/1236001995 conn(0x7f3b3c0b23c0 msgr2=0x7f3b3c0cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 shutdown_connections 2024-09-17T03:54:52.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.434+0000 7f3b42600640 1 -- 172.21.15.129:0/1236001995 wait complete. 2024-09-17T03:54:52.458 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:54:52.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:52.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:54:52.571 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:54:52.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:54:52.571 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:54:52.614 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:54:52.614 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:54:52.614 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:52.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:54:52.651 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:54:52.651 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:54:52.651 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:52.689 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:54:52.689 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:54:52.689 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:54:52.689 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:54:52.727 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:54:52.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 -- 172.21.15.129:0/2298431758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cd190 msgr2=0x7f819015cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 --2- 172.21.15.129:0/2298431758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cd190 0x7f819015cf70 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8180005fd0 tx=0x7f818002a4b0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 -- 172.21.15.129:0/2298431758 shutdown_connections 2024-09-17T03:54:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 --2- 172.21.15.129:0/2298431758 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 0x7f81900b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 --2- 172.21.15.129:0/2298431758 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cd190 0x7f819015cf70 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 -- 172.21.15.129:0/2298431758 >> 172.21.15.129:0/2298431758 conn(0x7f81900b23c0 msgr2=0x7f81900b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 -- 172.21.15.129:0/2298431758 shutdown_connections 2024-09-17T03:54:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.950+0000 7f8197a00640 1 -- 172.21.15.129:0/2298431758 wait complete. 2024-09-17T03:54:52.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.952+0000 7f8197a00640 1 Processor -- start 2024-09-17T03:54:52.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.952+0000 7f8197a00640 1 -- start start 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 0x7f81900b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 0x7f819015bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f819006fe50 con 0x7f81900cfd80 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f819006fcd0 con 0x7f81900cd190 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8197a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f819006ffd0 con 0x7f81900b59d0 2024-09-17T03:54:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8195a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8195a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47420/0 (socket says 172.21.15.129:47420) 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8195a00640 1 -- 172.21.15.129:0/1578890325 learned_addr learned my addr 172.21.15.129:0/1578890325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f8195000640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 0x7f81900b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.954+0000 7f818fe00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 0x7f819015bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 -- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 msgr2=0x7f81900b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 0x7f81900b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 -- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 msgr2=0x7f819015bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 0x7f819015bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 -- 172.21.15.129:0/1578890325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81901f9650 con 0x7f81900cfd80 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f818fe00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 0x7f819015bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8195a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f818800cad0 tx=0x7f818800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818800ae20 con 0x7f81900cfd80 2024-09-17T03:54:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f818800f540 con 0x7f81900cfd80 2024-09-17T03:54:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818800f990 con 0x7f81900cfd80 2024-09-17T03:54:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81901f9930 con 0x7f81900cfd80 2024-09-17T03:54:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.955+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81901f9ef0 con 0x7f81900cfd80 2024-09-17T03:54:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.956+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8190070530 con 0x7f81900cfd80 2024-09-17T03:54:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.957+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f818800fb30 con 0x7f81900cfd80 2024-09-17T03:54:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.958+0000 7f818cc00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 0x7f817007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.958+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f818809b8b0 con 0x7f81900cfd80 2024-09-17T03:54:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.959+0000 7f8195000640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 0x7f817007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.959+0000 7f8195000640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 0x7f817007b0a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8180033000 tx=0x7f8180004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:52.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:52.963+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8188014030 con 0x7f81900cfd80 2024-09-17T03:54:53.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.182+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f819015e490 con 0x7f81900cfd80 2024-09-17T03:54:53.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.182+0000 7f818cc00640 1 -- 172.21.15.129:0/1578890325 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f818800fe20 con 0x7f81900cfd80 2024-09-17T03:54:53.182 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:53.183 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:53.183 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:53.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 msgr2=0x7f817007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 0x7f817007b0a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8180033000 tx=0x7f8180004730 comp rx=0 tx=0).stop 2024-09-17T03:54:53.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 msgr2=0x7f81900ae890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f818800cad0 tx=0x7f818800cf90 comp rx=0 tx=0).stop 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 shutdown_connections 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8170078b90 0x7f817007b0a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81900cfd80 0x7f81900ae890 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81900cd190 0x7f819015bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 --2- 172.21.15.129:0/1578890325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81900b59d0 0x7f81900b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 >> 172.21.15.129:0/1578890325 conn(0x7f81900b23c0 msgr2=0x7f81900b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 shutdown_connections 2024-09-17T03:54:53.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.185+0000 7f8197a00640 1 -- 172.21.15.129:0/1578890325 wait complete. 2024-09-17T03:54:53.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 -- 172.21.15.129:0/879489160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c00b59d0 msgr2=0x7f29c00b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 --2- 172.21.15.129:0/879489160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c00b59d0 0x7f29c00b5de0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f29c00d0490 tx=0x7f29b802a560 comp rx=0 tx=0).stop 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 -- 172.21.15.129:0/879489160 shutdown_connections 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 --2- 172.21.15.129:0/879489160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c00b59d0 0x7f29c00b5de0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 --2- 172.21.15.129:0/879489160 >> v2:172.21.15.181:3300/0 conn(0x7f29c015cf70 0x7f29c0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 -- 172.21.15.129:0/879489160 >> 172.21.15.129:0/879489160 conn(0x7f29c00b23c0 msgr2=0x7f29c00b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 -- 172.21.15.129:0/879489160 shutdown_connections 2024-09-17T03:54:53.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.431+0000 7f29c7000640 1 -- 172.21.15.129:0/879489160 wait complete. 2024-09-17T03:54:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.433+0000 7f29c7000640 1 Processor -- start 2024-09-17T03:54:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.434+0000 7f29c7000640 1 -- start start 2024-09-17T03:54:53.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 0x7f29c013efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 0x7f29c01f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29c006fe50 con 0x7f29c015cf70 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29c006fcd0 con 0x7f29c00cd190 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c7000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f29c006ffd0 con 0x7f29c00b59d0 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47444/0 (socket says 172.21.15.129:47444) 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.435+0000 7f29c5000640 1 -- 172.21.15.129:0/4102865996 learned_addr learned my addr 172.21.15.129:0/4102865996 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29b7400640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 0x7f29c01f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29bfe00640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 0x7f29c013efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 -- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 msgr2=0x7f29c013efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 0x7f29c013efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 -- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 msgr2=0x7f29c01f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 0x7f29c01f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 -- 172.21.15.129:0/4102865996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c01f6aa0 con 0x7f29c015cf70 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29b7400640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 0x7f29c01f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29bfe00640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 0x7f29c013efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c5000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f29b8034040 tx=0x7f29b80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b803e030 con 0x7f29c015cf70 2024-09-17T03:54:53.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29b8003dd0 con 0x7f29c015cf70 2024-09-17T03:54:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8035040 con 0x7f29c015cf70 2024-09-17T03:54:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.436+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c0135040 con 0x7f29c015cf70 2024-09-17T03:54:53.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.437+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c0134560 con 0x7f29c015cf70 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.438+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c013f4e0 con 0x7f29c015cf70 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.439+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f29b8041030 con 0x7f29c015cf70 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.440+0000 7f29bcc00640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 0x7f299c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.441+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f29b80bb470 con 0x7f29c015cf70 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.441+0000 7f29bfe00640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 0x7f299c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.442+0000 7f29bfe00640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 0x7f299c07b0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f29b0001d10 tx=0x7f29b0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:53.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.445+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f29c01f6aa0 con 0x7f29c015cf70 2024-09-17T03:54:53.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.661+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29c00704c0 con 0x7f29c015cf70 2024-09-17T03:54:53.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.661+0000 7f29bcc00640 1 -- 172.21.15.129:0/4102865996 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f29b80837c0 con 0x7f29c015cf70 2024-09-17T03:54:53.662 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:53.662 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:53.662 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 msgr2=0x7f299c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 0x7f299c07b0d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f29b0001d10 tx=0x7f29b0009040 comp rx=0 tx=0).stop 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 msgr2=0x7f29c015e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f29b8034040 tx=0x7f29b80038a0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 shutdown_connections 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f299c078bc0 0x7f299c07b0d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29c015cf70 0x7f29c015e0f0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29c00cd190 0x7f29c01f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 --2- 172.21.15.129:0/4102865996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29c00b59d0 0x7f29c013efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 >> 172.21.15.129:0/4102865996 conn(0x7f29c00b23c0 msgr2=0x7f29c00b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:53.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 shutdown_connections 2024-09-17T03:54:53.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.664+0000 7f29c7000640 1 -- 172.21.15.129:0/4102865996 wait complete. 2024-09-17T03:54:53.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:54:53.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 -- 172.21.15.129:0/3526421184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c15da40 msgr2=0x7fe34c06e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 --2- 172.21.15.129:0/3526421184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c15da40 0x7fe34c06e310 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0b2860 tx=0x7fe33c02a5b0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 -- 172.21.15.129:0/3526421184 shutdown_connections 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 --2- 172.21.15.129:0/3526421184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c15da40 0x7fe34c06e310 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 -- 172.21.15.129:0/3526421184 >> 172.21.15.129:0/3526421184 conn(0x7fe34c06d560 msgr2=0x7fe34c0b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 -- 172.21.15.129:0/3526421184 shutdown_connections 2024-09-17T03:54:53.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.907+0000 7fe351800640 1 -- 172.21.15.129:0/3526421184 wait complete. 2024-09-17T03:54:53.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.909+0000 7fe351800640 1 Processor -- start 2024-09-17T03:54:53.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.910+0000 7fe351800640 1 -- start start 2024-09-17T03:54:53.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c141f60 0x7fe34c160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 0x7fe34c09cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c075110 con 0x7fe34c156f90 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe34c074f90 con 0x7fe34c141f60 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe351800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe34c075290 con 0x7fe34c15da40 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe34b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe34b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33768/0 (socket says 172.21.15.129:33768) 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe34b400640 1 -- 172.21.15.129:0/3575333899 learned_addr learned my addr 172.21.15.129:0/3575333899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.911+0000 7fe34be00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c141f60 0x7fe34c160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe350e00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 0x7fe34c09cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 -- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 msgr2=0x7fe34c09cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 0x7fe34c09cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 -- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c141f60 msgr2=0x7fe34c160d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:53.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c141f60 0x7fe34c160d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 -- 172.21.15.129:0/3575333899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34c1e6270 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe350e00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 0x7fe34c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe34b400640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c460 tx=0x7fe33800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338016020 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe338012a00 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338012e50 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.912+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34c1e6550 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.913+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe34c150030 con 0x7fe34c156f90 2024-09-17T03:54:53.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.914+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe34c06e310 con 0x7fe34c156f90 2024-09-17T03:54:53.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.915+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe338019020 con 0x7fe34c156f90 2024-09-17T03:54:53.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.916+0000 7fe343e00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 0x7fe32c07b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:53.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.917+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fe33809aa20 con 0x7fe34c156f90 2024-09-17T03:54:53.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.917+0000 7fe34be00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 0x7fe32c07b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:53.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.918+0000 7fe34be00640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 0x7fe32c07b240 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe33c034000 tx=0x7fe33c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:53.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:53.921+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe338020030 con 0x7fe34c156f90 2024-09-17T03:54:54.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.105+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe34c0b3630 con 0x7fe34c156f90 2024-09-17T03:54:54.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.106+0000 7fe343e00640 1 -- 172.21.15.129:0/3575333899 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16847 (secure 0 0 0) 0x7fe338062d60 con 0x7fe34c156f90 2024-09-17T03:54:54.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:54.107 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":96,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:17.557566+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:54:04.881426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-17T03:54:05.942429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/1986086177":"2024-09-18T03:54:17.549442+0000","172.21.15.129:6833/1986086177":"2024-09-18T03:54:17.549442+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:54:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 msgr2=0x7fe32c07b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 0x7fe32c07b240 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe33c034000 tx=0x7fe33c004730 comp rx=0 tx=0).stop 2024-09-17T03:54:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 msgr2=0x7fe34c150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fe33800c460 tx=0x7fe33800c920 comp rx=0 tx=0).stop 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 shutdown_connections 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe32c078d30 0x7fe32c07b240 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe34c15da40 0x7fe34c09cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe34c156f90 0x7fe34c150550 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 --2- 172.21.15.129:0/3575333899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe34c141f60 0x7fe34c160d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 >> 172.21.15.129:0/3575333899 conn(0x7fe34c06d560 msgr2=0x7fe34c0b4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 shutdown_connections 2024-09-17T03:54:54.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.109+0000 7fe351800640 1 -- 172.21.15.129:0/3575333899 wait complete. 2024-09-17T03:54:54.131 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:54:54.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:54.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 -- 172.21.15.129:0/1876046648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d0c003a50 msgr2=0x7f8d0c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 --2- 172.21.15.129:0/1876046648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d0c003a50 0x7f8d0c003ee0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8d180016c0 tx=0x7f8d0c005830 comp rx=0 tx=0).stop 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 -- 172.21.15.129:0/1876046648 shutdown_connections 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 --2- 172.21.15.129:0/1876046648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d0c003a50 0x7f8d0c003ee0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 -- 172.21.15.129:0/1876046648 >> 172.21.15.129:0/1876046648 conn(0x7f8d2806d560 msgr2=0x7f8d280b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 -- 172.21.15.129:0/1876046648 shutdown_connections 2024-09-17T03:54:54.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.354+0000 7f8d2d800640 1 -- 172.21.15.129:0/1876046648 wait complete. 2024-09-17T03:54:54.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.357+0000 7f8d2d800640 1 Processor -- start 2024-09-17T03:54:54.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.357+0000 7f8d2d800640 1 -- start start 2024-09-17T03:54:54.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d0c003a50 0x7f8d28150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 0x7f8d28150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d28072430 con 0x7f8d2815e1f0 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8d280722b0 con 0x7f8d280b1520 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8d280725b0 con 0x7f8d0c003a50 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33792/0 (socket says 172.21.15.129:33792) 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.358+0000 7f8d2ce00640 1 -- 172.21.15.129:0/3014511803 learned_addr learned my addr 172.21.15.129:0/3014511803 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d27400640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 0x7f8d28150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d27e00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d0c003a50 0x7f8d28150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 -- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d0c003a50 msgr2=0x7f8d28150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d0c003a50 0x7f8d28150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 -- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 msgr2=0x7f8d28150b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 0x7f8d28150b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 -- 172.21.15.129:0/3014511803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d28151080 con 0x7f8d2815e1f0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d27400640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 0x7f8d28150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2ce00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8d20005fd0 tx=0x7f8d2000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20003d80 con 0x7f8d2815e1f0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d20004b20 con 0x7f8d2815e1f0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d280b3d10 con 0x7f8d2815e1f0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.359+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20004030 con 0x7f8d2815e1f0 2024-09-17T03:54:54.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.360+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d281f6b20 con 0x7f8d2815e1f0 2024-09-17T03:54:54.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.361+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d281420d0 con 0x7f8d2815e1f0 2024-09-17T03:54:54.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.362+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8d200041d0 con 0x7f8d2815e1f0 2024-09-17T03:54:54.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.363+0000 7f8d1fe00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 0x7f8d0807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.363+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f8d2009bd80 con 0x7f8d2815e1f0 2024-09-17T03:54:54.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.363+0000 7f8d27e00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 0x7f8d0807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.364+0000 7f8d27e00640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 0x7f8d0807b0a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8d18032000 tx=0x7f8d18001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:54.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.367+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8d20020020 con 0x7f8d2815e1f0 2024-09-17T03:54:54.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.590+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d280b0840 con 0x7f8d2815e1f0 2024-09-17T03:54:54.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.590+0000 7f8d1fe00640 1 -- 172.21.15.129:0/3014511803 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f8d20064040 con 0x7f8d2815e1f0 2024-09-17T03:54:54.590 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:54.591 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:54.591 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 msgr2=0x7f8d0807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 0x7f8d0807b0a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8d18032000 tx=0x7f8d18001190 comp rx=0 tx=0).stop 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 msgr2=0x7f8d2815e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f8d20005fd0 tx=0x7f8d2000f790 comp rx=0 tx=0).stop 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 shutdown_connections 2024-09-17T03:54:54.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d08078b90 0x7f8d0807b0a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d2815e1f0 0x7f8d2815e600 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d280b1520 0x7f8d28150b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 --2- 172.21.15.129:0/3014511803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d0c003a50 0x7f8d28150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 >> 172.21.15.129:0/3014511803 conn(0x7f8d2806d560 msgr2=0x7f8d280b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 shutdown_connections 2024-09-17T03:54:54.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.593+0000 7f8d2d800640 1 -- 172.21.15.129:0/3014511803 wait complete. 2024-09-17T03:54:54.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.839+0000 7f0087200640 1 -- 172.21.15.129:0/823046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00880ad0e0 msgr2=0x7f00880ad4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.839+0000 7f0087200640 1 --2- 172.21.15.129:0/823046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00880ad0e0 0x7f00880ad4f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f0080005fd0 tx=0x7f008002a940 comp rx=0 tx=0).stop 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.839+0000 7f0087200640 1 -- 172.21.15.129:0/823046860 shutdown_connections 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.839+0000 7f0087200640 1 --2- 172.21.15.129:0/823046860 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00880ad0e0 0x7f00880ad4f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.839+0000 7f0087200640 1 -- 172.21.15.129:0/823046860 >> 172.21.15.129:0/823046860 conn(0x7f008806d560 msgr2=0x7f00880b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.840+0000 7f0087200640 1 -- 172.21.15.129:0/823046860 shutdown_connections 2024-09-17T03:54:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.840+0000 7f0087200640 1 -- 172.21.15.129:0/823046860 wait complete. 2024-09-17T03:54:54.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.842+0000 7f0087200640 1 Processor -- start 2024-09-17T03:54:54.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.842+0000 7f0087200640 1 -- start start 2024-09-17T03:54:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 0x7f0088159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 0x7f0088159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00880addc0 con 0x7f00881525b0 2024-09-17T03:54:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00880adc40 con 0x7f008806f420 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.843+0000 7f0087200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f00880adf40 con 0x7f00880ad0e0 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33808/0 (socket says 172.21.15.129:33808) 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 -- 172.21.15.129:0/3312865378 learned_addr learned my addr 172.21.15.129:0/3312865378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0085e00640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 0x7f0088159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 -- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 msgr2=0x7f0088159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f007d400640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 0x7f0088159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 0x7f0088159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 -- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 msgr2=0x7f0088159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 0x7f0088159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 -- 172.21.15.129:0/3312865378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00880cf170 con 0x7f00881525b0 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0085e00640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 0x7f0088159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:54.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f007d400640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 0x7f0088159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.844+0000 7f0086800640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f008002af30 tx=0x7f0080031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.845+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008003e070 con 0x7f00881525b0 2024-09-17T03:54:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.845+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0080004560 con 0x7f00881525b0 2024-09-17T03:54:54.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.845+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00880cf450 con 0x7f00881525b0 2024-09-17T03:54:54.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.845+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080043460 con 0x7f00881525b0 2024-09-17T03:54:54.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.845+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00880cfa10 con 0x7f00881525b0 2024-09-17T03:54:54.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.846+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00880ad4f0 con 0x7f00881525b0 2024-09-17T03:54:54.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.847+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f008003a440 con 0x7f00881525b0 2024-09-17T03:54:54.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.848+0000 7f007ea00640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 0x7f006407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:54.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.848+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f00800bc900 con 0x7f00881525b0 2024-09-17T03:54:54.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.849+0000 7f0085e00640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 0x7f006407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:54.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.849+0000 7f0085e00640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 0x7f006407b0a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00780046b0 tx=0x7f0078009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:54.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:54.853+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0080084c40 con 0x7f00881525b0 2024-09-17T03:54:55.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.069+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00880750e0 con 0x7f00881525b0 2024-09-17T03:54:55.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.069+0000 7f007ea00640 1 -- 172.21.15.129:0/3312865378 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5235 (secure 0 0 0) 0x7f0080084540 con 0x7f00881525b0 2024-09-17T03:54:55.070 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:55.070 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":71,"btime":"2024-09-17T03:54:22:584230+0000","default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:54:06.985254+0000","modified":"2024-09-17T03:54:22.584225+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":95,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5876},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5876":{"gid":5876,"name":"c","rank":0,"incarnation":67,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3654208591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3654208591},{"type":"v1","addr":"172.21.15.129:6835","nonce":3654208591}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5876,"qdb_cluster":[5876]},"id":7}]} 2024-09-17T03:54:55.070 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 71 2024-09-17T03:54:55.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 msgr2=0x7f006407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 0x7f006407b0a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f00780046b0 tx=0x7f0078009040 comp rx=0 tx=0).stop 2024-09-17T03:54:55.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 msgr2=0x7f00881f58c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f008002af30 tx=0x7f0080031ea0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 shutdown_connections 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0064078b90 0x7f006407b0a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00881525b0 0x7f00881f58c0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00880ad0e0 0x7f0088159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 --2- 172.21.15.129:0/3312865378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f008806f420 0x7f0088159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 >> 172.21.15.129:0/3312865378 conn(0x7f008806d560 msgr2=0x7f00880742d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 shutdown_connections 2024-09-17T03:54:55.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.072+0000 7f0087200640 1 -- 172.21.15.129:0/3312865378 wait complete. 2024-09-17T03:54:55.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.316+0000 7fabefc00640 1 -- 172.21.15.129:0/1646219533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe815f880 msgr2=0x7fabe815fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.316+0000 7fabefc00640 1 --2- 172.21.15.129:0/1646219533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe815f880 0x7fabe815fc90 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009830 tx=0x7fabd402a530 comp rx=0 tx=0).stop 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.316+0000 7fabefc00640 1 -- 172.21.15.129:0/1646219533 shutdown_connections 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.316+0000 7fabefc00640 1 --2- 172.21.15.129:0/1646219533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe815f880 0x7fabe815fc90 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.316+0000 7fabefc00640 1 -- 172.21.15.129:0/1646219533 >> 172.21.15.129:0/1646219533 conn(0x7fabe80b23c0 msgr2=0x7fabe80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.317+0000 7fabefc00640 1 -- 172.21.15.129:0/1646219533 shutdown_connections 2024-09-17T03:54:55.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.317+0000 7fabefc00640 1 -- 172.21.15.129:0/1646219533 wait complete. 2024-09-17T03:54:55.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.319+0000 7fabefc00640 1 Processor -- start 2024-09-17T03:54:55.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.319+0000 7fabefc00640 1 -- start start 2024-09-17T03:54:55.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe80737b0 0x7fabe8148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabe815f880 0x7fabe8145c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabe80cfca0 con 0x7fabe80cedf0 2024-09-17T03:54:55.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fabe80cfb20 con 0x7fabe80737b0 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabefc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fabe80cfe20 con 0x7fabe815f880 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.320+0000 7fabe7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33836/0 (socket says 172.21.15.129:33836) 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 -- 172.21.15.129:0/172161441 learned_addr learned my addr 172.21.15.129:0/172161441 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabed200640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe80737b0 0x7fabe8148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabedc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabe815f880 0x7fabe8145c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 -- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabe815f880 msgr2=0x7fabe8145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabe815f880 0x7fabe8145c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 -- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe80737b0 msgr2=0x7fabe8148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe80737b0 0x7fabe8148ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 -- 172.21.15.129:0/172161441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe8146160 con 0x7fabe80cedf0 2024-09-17T03:54:55.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.321+0000 7fabe7e00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009950 tx=0x7fabd4033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:55.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.322+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4031b60 con 0x7fabe80cedf0 2024-09-17T03:54:55.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.322+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabd4031df0 con 0x7fabe80cedf0 2024-09-17T03:54:55.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.322+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4005280 con 0x7fabe80cedf0 2024-09-17T03:54:55.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.322+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe8156430 con 0x7fabe80cedf0 2024-09-17T03:54:55.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.322+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe81569f0 con 0x7fabe80cedf0 2024-09-17T03:54:55.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.323+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe81493e0 con 0x7fabe80cedf0 2024-09-17T03:54:55.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.324+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fabd4005420 con 0x7fabe80cedf0 2024-09-17T03:54:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.325+0000 7fabe4c00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 0x7fabc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.325+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7fabd40bbe80 con 0x7fabe80cedf0 2024-09-17T03:54:55.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.325+0000 7fabed200640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 0x7fabc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.326+0000 7fabed200640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 0x7fabc807b0a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fabdc000f60 tx=0x7fabdc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:55.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.330+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fabd403f030 con 0x7fabe80cedf0 2024-09-17T03:54:55.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.518+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabe815fc90 con 0x7fabe80cedf0 2024-09-17T03:54:55.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.519+0000 7fabe4c00640 1 -- 172.21.15.129:0/172161441 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16847 (secure 0 0 0) 0x7fabd4031770 con 0x7fabe80cedf0 2024-09-17T03:54:55.520 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:55.520 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":96,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:17.557566+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:54:04.881426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-17T03:54:05.942429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/1986086177":"2024-09-18T03:54:17.549442+0000","172.21.15.129:6833/1986086177":"2024-09-18T03:54:17.549442+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:54:55.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 msgr2=0x7fabc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 0x7fabc807b0a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fabdc000f60 tx=0x7fabdc009040 comp rx=0 tx=0).stop 2024-09-17T03:54:55.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 msgr2=0x7fabe8146430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009950 tx=0x7fabd4033c10 comp rx=0 tx=0).stop 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 shutdown_connections 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabc8078b90 0x7fabc807b0a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabe815f880 0x7fabe8145c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabe80cedf0 0x7fabe8146430 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 --2- 172.21.15.129:0/172161441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabe80737b0 0x7fabe8148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 >> 172.21.15.129:0/172161441 conn(0x7fabe80b23c0 msgr2=0x7fabe80cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 shutdown_connections 2024-09-17T03:54:55.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.522+0000 7fabefc00640 1 -- 172.21.15.129:0/172161441 wait complete. 2024-09-17T03:54:55.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:54:55.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 -- 172.21.15.129:0/2809445128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0b59d0 msgr2=0x7f437c0b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 --2- 172.21.15.129:0/2809445128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0b59d0 0x7f437c0b5de0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f437c0b6780 tx=0x7f437402a980 comp rx=0 tx=0).stop 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 -- 172.21.15.129:0/2809445128 shutdown_connections 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 --2- 172.21.15.129:0/2809445128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0b59d0 0x7f437c0b5de0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 --2- 172.21.15.129:0/2809445128 >> v2:172.21.15.181:3300/0 conn(0x7f437c0cfd80 0x7f437c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 --2- 172.21.15.129:0/2809445128 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cd190 0x7f437c15cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 -- 172.21.15.129:0/2809445128 >> 172.21.15.129:0/2809445128 conn(0x7f437c0b23c0 msgr2=0x7f437c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:55.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 -- 172.21.15.129:0/2809445128 shutdown_connections 2024-09-17T03:54:55.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.768+0000 7f4382400640 1 -- 172.21.15.129:0/2809445128 wait complete. 2024-09-17T03:54:55.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.770+0000 7f4382400640 1 Processor -- start 2024-09-17T03:54:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.771+0000 7f4382400640 1 -- start start 2024-09-17T03:54:55.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f437c0b59d0 0x7f437c0c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 0x7f437c0ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f437c06fe50 con 0x7f437c0cd190 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f437c06fcd0 con 0x7f437c0b59d0 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4382400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f437c06ffd0 con 0x7f437c0cfd80 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4372a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4372a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33850/0 (socket says 172.21.15.129:33850) 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.772+0000 7f4372a00640 1 -- 172.21.15.129:0/186538566 learned_addr learned my addr 172.21.15.129:0/186538566 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f437b400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f437c0b59d0 0x7f437c0c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f437be00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 0x7f437c0ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 -- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 msgr2=0x7f437c0ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 0x7f437c0ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 -- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f437c0b59d0 msgr2=0x7f437c0c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f437c0b59d0 0x7f437c0c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 -- 172.21.15.129:0/186538566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c0c5840 con 0x7f437c0cd190 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f437be00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 0x7f437c0ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4372a00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f436400e7c0 tx=0x7f436400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:55.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436400ccc0 con 0x7f437c0cd190 2024-09-17T03:54:55.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f436400ee70 con 0x7f437c0cd190 2024-09-17T03:54:55.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364010690 con 0x7f437c0cd190 2024-09-17T03:54:55.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.773+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c0c5b20 con 0x7f437c0cd190 2024-09-17T03:54:55.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.774+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c0c60e0 con 0x7f437c0cd190 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.775+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437c0704f0 con 0x7f437c0cd190 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.776+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4364010830 con 0x7f437c0cd190 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.777+0000 7f4373e00640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.777+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5795+0+0 (secure 0 0 0) 0x7f436409b800 con 0x7f437c0cd190 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.777+0000 7f437b400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:55.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.778+0000 7f437b400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f436c00bb00 tx=0x7f436c005e30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:55.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:55.781+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4364063b40 con 0x7f437c0cd190 2024-09-17T03:54:56.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.000+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f437c0c7850 con 0x7f437c0cd190 2024-09-17T03:54:56.001 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:54:56.001+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:54:56.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.003+0000 7f4373e00640 1 -- 172.21.15.129:0/186538566 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v72) ==== 161+0+0 (secure 0 0 0) 0x7f4364063440 con 0x7f437c0cd190 2024-09-17T03:54:56.003 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:54:56.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 msgr2=0x7f435807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f436c00bb00 tx=0x7f436c005e30 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 msgr2=0x7f437c0af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f436400e7c0 tx=0x7f436400ec80 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 shutdown_connections 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f437c0cfd80 0x7f437c0ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437c0cd190 0x7f437c0af040 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 --2- 172.21.15.129:0/186538566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f437c0b59d0 0x7f437c0c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 >> 172.21.15.129:0/186538566 conn(0x7f437c0b23c0 msgr2=0x7f437c0b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 shutdown_connections 2024-09-17T03:54:56.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.007+0000 7f4382400640 1 -- 172.21.15.129:0/186538566 wait complete. 2024-09-17T03:54:56.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:54:56.162 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:54:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 -- 172.21.15.129:0/1138999592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00b59d0 msgr2=0x7f52a00b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 --2- 172.21.15.129:0/1138999592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00b59d0 0x7f52a00b5de0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f52a00d0490 tx=0x7f529802aa30 comp rx=0 tx=0).stop 2024-09-17T03:54:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 -- 172.21.15.129:0/1138999592 shutdown_connections 2024-09-17T03:54:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 --2- 172.21.15.129:0/1138999592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00b59d0 0x7f52a00b5de0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 --2- 172.21.15.129:0/1138999592 >> v2:172.21.15.181:3300/0 conn(0x7f52a015cf70 0x7f52a0072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 -- 172.21.15.129:0/1138999592 >> 172.21.15.129:0/1138999592 conn(0x7f52a00b23c0 msgr2=0x7f52a00b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:56.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 -- 172.21.15.129:0/1138999592 shutdown_connections 2024-09-17T03:54:56.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.274+0000 7f52a4a00640 1 -- 172.21.15.129:0/1138999592 wait complete. 2024-09-17T03:54:56.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.276+0000 7f52a4a00640 1 Processor -- start 2024-09-17T03:54:56.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.276+0000 7f52a4a00640 1 -- start start 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.277+0000 7f52a4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 0x7f52a00cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.277+0000 7f52a4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f52a4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52a015cf70 0x7f52a0131900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f52a4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52a006fe50 con 0x7f52a00cd190 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f52a4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f52a006fcd0 con 0x7f52a015cf70 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f52a4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f52a006ffd0 con 0x7f52a00b59d0 2024-09-17T03:54:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33866/0 (socket says 172.21.15.129:33866) 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 -- 172.21.15.129:0/867441963 learned_addr learned my addr 172.21.15.129:0/867441963 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f529e800640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52a015cf70 0x7f52a0131900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f529de00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 0x7f52a00cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 -- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 msgr2=0x7f52a00cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 0x7f52a00cc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 -- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52a015cf70 msgr2=0x7f52a0131900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52a015cf70 0x7f52a0131900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:56.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 -- 172.21.15.129:0/867441963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a01e7ae0 con 0x7f52a00cd190 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f529de00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 0x7f52a00cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.278+0000 7f5297e00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f528800e790 tx=0x7f528800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.279+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528800eea0 con 0x7f52a00cd190 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.279+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f528800cdb0 con 0x7f52a00cd190 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.279+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a01308e0 con 0x7f52a00cd190 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.279+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5288010690 con 0x7f52a00cd190 2024-09-17T03:54:56.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.279+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a01e7dc0 con 0x7f52a00cd190 2024-09-17T03:54:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.280+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a01e7850 con 0x7f52a00cd190 2024-09-17T03:54:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.281+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5288010830 con 0x7f52a00cd190 2024-09-17T03:54:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.282+0000 7f5296000640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 0x7f527c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.282+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5881+0+0 (secure 0 0 0) 0x7f528809b8e0 con 0x7f52a00cd190 2024-09-17T03:54:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.283+0000 7f529de00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 0x7f527c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:56.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.283+0000 7f529de00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 0x7f527c07b0a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f52900016a0 tx=0x7f5290009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.287+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5288063bd0 con 0x7f52a00cd190 2024-09-17T03:54:56.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:56.508+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f52a00704c0 con 0x7f52a00cd190 2024-09-17T03:54:57.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.010+0000 7f5296000640 1 -- 172.21.15.129:0/867441963 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v73) ==== 104+0+0 (secure 0 0 0) 0x7f52880634d0 con 0x7f52a00cd190 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 msgr2=0x7f527c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 0x7f527c07b0a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f52900016a0 tx=0x7f5290009040 comp rx=0 tx=0).stop 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 msgr2=0x7f52a01313c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f528800e790 tx=0x7f528800ec50 comp rx=0 tx=0).stop 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 shutdown_connections 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f527c078b90 0x7f527c07b0a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f52a015cf70 0x7f52a0131900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52a00cd190 0x7f52a01313c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.013+0000 7f52a4a00640 1 --2- 172.21.15.129:0/867441963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52a00b59d0 0x7f52a00cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.014+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 >> 172.21.15.129:0/867441963 conn(0x7f52a00b23c0 msgr2=0x7f52a01450f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:57.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.014+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 shutdown_connections 2024-09-17T03:54:57.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.014+0000 7f52a4a00640 1 -- 172.21.15.129:0/867441963 wait complete. 2024-09-17T03:54:57.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:54:57.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 -- 172.21.15.129:0/836661805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cd190 msgr2=0x7fd65015cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:57.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 --2- 172.21.15.129:0/836661805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cd190 0x7fd65015cf70 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd640005fd0 tx=0x7fd64002a820 comp rx=0 tx=0).stop 2024-09-17T03:54:57.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 -- 172.21.15.129:0/836661805 shutdown_connections 2024-09-17T03:54:57.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 --2- 172.21.15.129:0/836661805 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cd190 0x7fd65015cf70 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 -- 172.21.15.129:0/836661805 >> 172.21.15.129:0/836661805 conn(0x7fd6500b23c0 msgr2=0x7fd6500b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:57.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 -- 172.21.15.129:0/836661805 shutdown_connections 2024-09-17T03:54:57.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.263+0000 7fd657200640 1 -- 172.21.15.129:0/836661805 wait complete. 2024-09-17T03:54:57.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.265+0000 7fd657200640 1 Processor -- start 2024-09-17T03:54:57.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.265+0000 7fd657200640 1 -- start start 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6500b59d0 0x7fd6500b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6500cd190 0x7fd6500cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd650070060 con 0x7fd6500cfd80 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd65006fee0 con 0x7fd6500cd190 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd657200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd6500701e0 con 0x7fd6500b59d0 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd655200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd655200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33888/0 (socket says 172.21.15.129:33888) 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.267+0000 7fd655200640 1 -- 172.21.15.129:0/2628718461 learned_addr learned my addr 172.21.15.129:0/2628718461 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:57.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd64fe00640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6500b59d0 0x7fd6500b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd64f400640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6500cd190 0x7fd6500cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 -- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6500b59d0 msgr2=0x7fd6500b37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6500b59d0 0x7fd6500b37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 -- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6500cd190 msgr2=0x7fd6500cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6500cd190 0x7fd6500cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 -- 172.21.15.129:0/2628718461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6500b3d00 con 0x7fd6500cfd80 2024-09-17T03:54:57.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd655200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd64800c3f0 tx=0x7fd64800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648017070 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6480053c0 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6500ad470 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648003850 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.268+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6500ad710 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.270+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6500706d0 con 0x7fd6500cfd80 2024-09-17T03:54:57.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.271+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd64801a070 con 0x7fd6500cfd80 2024-09-17T03:54:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.272+0000 7fd647e00640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 0x7fd63007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.272+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5881+0+0 (secure 0 0 0) 0x7fd64809a250 con 0x7fd6500cfd80 2024-09-17T03:54:57.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.273+0000 7fd64fe00640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 0x7fd63007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:57.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.274+0000 7fd64fe00640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 0x7fd63007b0d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd640034000 tx=0x7fd640004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:57.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.277+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd648062600 con 0x7fd6500cfd80 2024-09-17T03:54:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:57.459+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd6500cb900 con 0x7fd6500cfd80 2024-09-17T03:54:58.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.012+0000 7fd647e00640 1 -- 172.21.15.129:0/2628718461 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) ==== 181+0+0 (secure 0 0 0) 0x7fd648061f00 con 0x7fd6500cfd80 2024-09-17T03:54:58.012 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:54:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 msgr2=0x7fd63007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 0x7fd63007b0d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd640034000 tx=0x7fd640004ab0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 msgr2=0x7fd6500aedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:58.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fd64800c3f0 tx=0x7fd64800c8b0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 shutdown_connections 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd630078bc0 0x7fd63007b0d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6500cfd80 0x7fd6500aedd0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6500cd190 0x7fd6500cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 --2- 172.21.15.129:0/2628718461 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6500b59d0 0x7fd6500b37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 >> 172.21.15.129:0/2628718461 conn(0x7fd6500b23c0 msgr2=0x7fd650075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 shutdown_connections 2024-09-17T03:54:58.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.016+0000 7fd657200640 1 -- 172.21.15.129:0/2628718461 wait complete. 2024-09-17T03:54:58.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 -- 172.21.15.129:0/4078433407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0cd190 msgr2=0x7fdaec15cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 --2- 172.21.15.129:0/4078433407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0cd190 0x7fdaec15cf70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdae0009830 tx=0x7fdae002a480 comp rx=0 tx=0).stop 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 -- 172.21.15.129:0/4078433407 shutdown_connections 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 --2- 172.21.15.129:0/4078433407 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0b59d0 0x7fdaec0b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 --2- 172.21.15.129:0/4078433407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0cd190 0x7fdaec15cf70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 -- 172.21.15.129:0/4078433407 >> 172.21.15.129:0/4078433407 conn(0x7fdaec0b23c0 msgr2=0x7fdaec0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.266+0000 7fdaf3600640 1 -- 172.21.15.129:0/4078433407 shutdown_connections 2024-09-17T03:54:58.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.267+0000 7fdaf3600640 1 -- 172.21.15.129:0/4078433407 wait complete. 2024-09-17T03:54:58.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.269+0000 7fdaf3600640 1 Processor -- start 2024-09-17T03:54:58.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.269+0000 7fdaf3600640 1 -- start start 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0b59d0 0x7fdaec15c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0cfd80 0x7fdaec0cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec06fe50 con 0x7fdaec0cd190 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec06fcd0 con 0x7fdaec0b59d0 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.270+0000 7fdaf3600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdaec06ffd0 con 0x7fdaec0cfd80 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33902/0 (socket says 172.21.15.129:33902) 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 -- 172.21.15.129:0/531545691 learned_addr learned my addr 172.21.15.129:0/531545691 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:58.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaf1600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0cfd80 0x7fdaec0cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaf0c00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0b59d0 0x7fdaec15c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 -- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0cfd80 msgr2=0x7fdaec0cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0cfd80 0x7fdaec0cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 -- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0b59d0 msgr2=0x7fdaec15c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0b59d0 0x7fdaec15c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:58.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 -- 172.21.15.129:0/531545691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec0cc640 con 0x7fdaec0cd190 2024-09-17T03:54:58.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.271+0000 7fdaebe00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c400 tx=0x7fdad800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:58.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.272+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011790 con 0x7fdaec0cd190 2024-09-17T03:54:58.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.272+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdad8017070 con 0x7fdaec0cd190 2024-09-17T03:54:58.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.272+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8011c60 con 0x7fdaec0cd190 2024-09-17T03:54:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.272+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaec0c56d0 con 0x7fdaec0cd190 2024-09-17T03:54:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.272+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaec0c5c90 con 0x7fdaec0cd190 2024-09-17T03:54:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.273+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaec0cc390 con 0x7fdaec0cd190 2024-09-17T03:54:58.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.274+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdad801c460 con 0x7fdaec0cd190 2024-09-17T03:54:58.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.275+0000 7fdae8c00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 0x7fdacc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:58.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.275+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5443+0+0 (secure 0 0 0) 0x7fdad809a9a0 con 0x7fdaec0cd190 2024-09-17T03:54:58.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.275+0000 7fdaf0c00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 0x7fdacc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:58.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.276+0000 7fdaf0c00640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 0x7fdacc07b0d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdae0034000 tx=0x7fdae0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:58.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.279+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdad80627f0 con 0x7fdaec0cd190 2024-09-17T03:54:58.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:58.463+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fdaec0704f0 con 0x7fdaec0cd190 2024-09-17T03:54:59.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.020+0000 7fdae8c00640 1 -- 172.21.15.129:0/531545691 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) ==== 169+0+0 (secure 0 0 0) 0x7fdad8067220 con 0x7fdaec0cd190 2024-09-17T03:54:59.020 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:54:59.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.024+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 msgr2=0x7fdacc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:59.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.024+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 0x7fdacc07b0d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdae0034000 tx=0x7fdae0004730 comp rx=0 tx=0).stop 2024-09-17T03:54:59.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.024+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 msgr2=0x7fdaec0cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.024+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c400 tx=0x7fdad800c8c0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 shutdown_connections 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdacc078bc0 0x7fdacc07b0d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaec0cfd80 0x7fdaec0cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaec0cd190 0x7fdaec0cb8a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 --2- 172.21.15.129:0/531545691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdaec0b59d0 0x7fdaec15c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 >> 172.21.15.129:0/531545691 conn(0x7fdaec0b23c0 msgr2=0x7fdaec075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 shutdown_connections 2024-09-17T03:54:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.025+0000 7fdaf3600640 1 -- 172.21.15.129:0/531545691 wait complete. 2024-09-17T03:54:59.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:59.171 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 6 2024-09-17T03:54:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:59.174 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:59.174 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.175 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.176 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.176 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:59.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.176 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:59.177 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:59.177 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.177 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.178 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.178 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.179 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.180 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.180 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.180 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:59.181 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:59.182 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.183 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.184 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.185 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:59.186 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.187 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:59.188 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.188 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.188 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.188 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.189 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.190 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:59.191 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:59.192 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:59.192 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:59.192 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:59.192 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 6 2024-09-17T03:54:59.192 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 100000000 2024-09-17T03:54:59.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.306 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.307 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.308 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.309 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.310 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.311 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.312 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.313 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.314 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:59.315 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:59.316 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:59.317 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 6 2024-09-17T03:54:59.323 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:59.324 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.325 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.326 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:59.327 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:59.328 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:59.329 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.330 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.331 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.332 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.333 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:59.334 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:59.335 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:59.335 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:59.335 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:59.335 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 6 2024-09-17T03:54:59.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T03:54:59.686 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T03:54:59.686 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T03:54:59.686 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.686 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T03:54:59.686 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.687 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.688 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T03:54:59.689 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.690 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T03:54:59.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.692 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.693 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.694 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.695 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T03:54:59.696 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T03:54:59.697 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.698 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T03:54:59.699 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T03:54:59.700 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T03:54:59.701 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T03:54:59.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T03:54:59.703 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.704 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T03:54:59.705 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.706 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T03:54:59.707 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T03:54:59.708 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T03:54:59.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:54:59.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 -- 172.21.15.129:0/30909503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0564099570 msgr2=0x7f056406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 --2- 172.21.15.129:0/30909503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0564099570 0x7f056406eed0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f05640b3450 tx=0x7f055802a4d0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 -- 172.21.15.129:0/30909503 shutdown_connections 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 --2- 172.21.15.129:0/30909503 >> v2:172.21.15.181:3300/0 conn(0x7f05640cf9c0 0x7f05640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 --2- 172.21.15.129:0/30909503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0564099570 0x7f056406eed0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 -- 172.21.15.129:0/30909503 >> 172.21.15.129:0/30909503 conn(0x7f056406d410 msgr2=0x7f05640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 -- 172.21.15.129:0/30909503 shutdown_connections 2024-09-17T03:54:59.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.943+0000 7f056b800640 1 -- 172.21.15.129:0/30909503 wait complete. 2024-09-17T03:54:59.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.945+0000 7f056b800640 1 Processor -- start 2024-09-17T03:54:59.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.946+0000 7f056b800640 1 -- start start 2024-09-17T03:54:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f056406f410 0x7f05640c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:59.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0564099570 0x7f05640743a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0564160ba0 con 0x7f05640cf9c0 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0564160a20 con 0x7f0564099570 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f056b800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0564160d20 con 0x7f056406f410 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f0569800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f0569800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33926/0 (socket says 172.21.15.129:33926) 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f0569800640 1 -- 172.21.15.129:0/847673720 learned_addr learned my addr 172.21.15.129:0/847673720 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.947+0000 7f0568e00640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f056406f410 0x7f05640c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0563e00640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0564099570 0x7f05640743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 -- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f056406f410 msgr2=0x7f05640c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f056406f410 0x7f05640c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 -- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0564099570 msgr2=0x7f05640743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:54:59.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0564099570 0x7f05640743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 -- 172.21.15.129:0/847673720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f056415d020 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0569800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f055c00b9e0 tx=0x7f055c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c019070 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f055c00cbd0 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f056415d2c0 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.948+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f055c014440 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.949+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05640748e0 con 0x7f05640cf9c0 2024-09-17T03:54:59.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.950+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05641612c0 con 0x7f05640cf9c0 2024-09-17T03:54:59.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.951+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f055c01c070 con 0x7f05640cf9c0 2024-09-17T03:54:59.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.952+0000 7f0560c00640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 0x7f054407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:54:59.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.952+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f055c09a710 con 0x7f05640cf9c0 2024-09-17T03:54:59.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.952+0000 7f0568e00640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 0x7f054407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:54:59.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.953+0000 7f0568e00640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 0x7f054407b0a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0558033000 tx=0x7f0558004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:54:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:54:59.956+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f055c0627a0 con 0x7f05640cf9c0 2024-09-17T03:55:00.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.142+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f056406a140 con 0x7f05640cf9c0 2024-09-17T03:55:00.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.146+0000 7f0560c00640 1 -- 172.21.15.129:0/847673720 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v47) ==== 71+0+0 (secure 0 0 0) 0x7f055c0671d0 con 0x7f05640cf9c0 2024-09-17T03:55:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 msgr2=0x7f054407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 0x7f054407b0a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0558033000 tx=0x7f0558004730 comp rx=0 tx=0).stop 2024-09-17T03:55:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 msgr2=0x7f056415ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f055c00b9e0 tx=0x7f055c00bea0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 shutdown_connections 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0544078b90 0x7f054407b0a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05640cf9c0 0x7f056415ea40 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0564099570 0x7f05640743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 --2- 172.21.15.129:0/847673720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f056406f410 0x7f05640c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 >> 172.21.15.129:0/847673720 conn(0x7f056406d410 msgr2=0x7f05640d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 shutdown_connections 2024-09-17T03:55:00.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.149+0000 7f056b800640 1 -- 172.21.15.129:0/847673720 wait complete. 2024-09-17T03:55:00.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle' 2024-09-17T03:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 -- 172.21.15.129:0/941619072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202415f820 msgr2=0x7f202415fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:00.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 --2- 172.21.15.129:0/941619072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202415f820 0x7f202415fc30 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f20240731d0 tx=0x7f200c02a560 comp rx=0 tx=0).stop 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 -- 172.21.15.129:0/941619072 shutdown_connections 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 --2- 172.21.15.129:0/941619072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f202415f820 0x7f202415fc30 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 --2- 172.21.15.129:0/941619072 >> v2:172.21.15.181:3300/0 conn(0x7f202415b8b0 0x7f202415f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 -- 172.21.15.129:0/941619072 >> 172.21.15.129:0/941619072 conn(0x7f20240d0a70 msgr2=0x7f20240d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 -- 172.21.15.129:0/941619072 shutdown_connections 2024-09-17T03:55:00.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.394+0000 7f202bc00640 1 -- 172.21.15.129:0/941619072 wait complete. 2024-09-17T03:55:00.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.396+0000 7f202bc00640 1 Processor -- start 2024-09-17T03:55:00.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.396+0000 7f202bc00640 1 -- start start 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202415b8b0 0x7f20240b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f202415f820 0x7f20240b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20240c42c0 con 0x7f20240cb0c0 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20240c4140 con 0x7f202415b8b0 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f202bc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f20240cbbe0 con 0x7f202415f820 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33942/0 (socket says 172.21.15.129:33942) 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 -- 172.21.15.129:0/3194268202 learned_addr learned my addr 172.21.15.129:0/3194268202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f201be00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202415b8b0 0x7f20240b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:00.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029c00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f202415f820 0x7f20240b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 -- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f202415f820 msgr2=0x7f20240b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f202415f820 0x7f20240b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 -- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202415b8b0 msgr2=0x7f20240b6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202415b8b0 0x7f20240b6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.398+0000 7f2029200640 1 -- 172.21.15.129:0/3194268202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f202415dff0 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f2029200640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f201400d6d0 tx=0x7f201400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014017070 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2014005330 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014009820 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202415e2d0 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.399+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202415e890 con 0x7f20240cb0c0 2024-09-17T03:55:00.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.400+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f202415f2e0 con 0x7f20240cb0c0 2024-09-17T03:55:00.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.401+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f20140099c0 con 0x7f20240cb0c0 2024-09-17T03:55:00.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.402+0000 7f2021600640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 0x7f200007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:00.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.402+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f201409abe0 con 0x7f20240cb0c0 2024-09-17T03:55:00.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.403+0000 7f201be00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 0x7f200007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:00.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.403+0000 7f201be00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 0x7f200007b0a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f200c033000 tx=0x7f200c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:00.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.407+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2014062bd0 con 0x7f20240cb0c0 2024-09-17T03:55:00.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:00.660+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]} v 0) -- 0x7f20240d0430 con 0x7f20240cb0c0 2024-09-17T03:55:01.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.024+0000 7f2021600640 1 -- 172.21.15.129:0/3194268202 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_files_throttle"]}]=0 v297) ==== 134+0+0 (secure 0 0 0) 0x7f2014067600 con 0x7f20240cb0c0 2024-09-17T03:55:01.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 msgr2=0x7f200007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 0x7f200007b0a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f200c033000 tx=0x7f200c004730 comp rx=0 tx=0).stop 2024-09-17T03:55:01.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 msgr2=0x7f20241f5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f201400d6d0 tx=0x7f201400db90 comp rx=0 tx=0).stop 2024-09-17T03:55:01.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 shutdown_connections 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2000078b90 0x7f200007b0a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f202415f820 0x7f20240b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f202415b8b0 0x7f20240b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 --2- 172.21.15.129:0/3194268202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20240cb0c0 0x7f20241f5540 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 >> 172.21.15.129:0/3194268202 conn(0x7f20240d0a70 msgr2=0x7f20240cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 shutdown_connections 2024-09-17T03:55:01.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.027+0000 7f202bc00640 1 -- 172.21.15.129:0/3194268202 wait complete. 2024-09-17T03:55:01.049 INFO:tasks.cephfs_test_runner:test_files_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-09-17T03:55:01.049 INFO:tasks.cephfs_test_runner:Starting test: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) That removal of primary dentry of hardlinked inode results 2024-09-17T03:55:01.050 INFO:tasks.cephfs_test_runner:test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:55:01.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 -- 172.21.15.129:0/1225966885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 msgr2=0x7f92101609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 --2- 172.21.15.129:0/1225966885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101609c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f9208005fd0 tx=0x7f920802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 -- 172.21.15.129:0/1225966885 shutdown_connections 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 --2- 172.21.15.129:0/1225966885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101609c0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 --2- 172.21.15.129:0/1225966885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100b4140 0x7f92100ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 -- 172.21.15.129:0/1225966885 >> 172.21.15.129:0/1225966885 conn(0x7f921006d410 msgr2=0x7f92100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 -- 172.21.15.129:0/1225966885 shutdown_connections 2024-09-17T03:55:01.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.272+0000 7f9215c00640 1 -- 172.21.15.129:0/1225966885 wait complete. 2024-09-17T03:55:01.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.275+0000 7f9215c00640 1 Processor -- start 2024-09-17T03:55:01.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.275+0000 7f9215c00640 1 -- start start 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100705a0 0x7f921015e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92100b4140 0x7f92101f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9210160260 con 0x7f92100cd1c0 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92101600e0 con 0x7f92100b4140 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.276+0000 7f9215c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9210160f00 con 0x7f92100705a0 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33958/0 (socket says 172.21.15.129:33958) 2024-09-17T03:55:01.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 -- 172.21.15.129:0/3403360069 learned_addr learned my addr 172.21.15.129:0/3403360069 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f9206000640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92100b4140 0x7f92101f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920ea00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100705a0 0x7f921015e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 -- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100705a0 msgr2=0x7f921015e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100705a0 0x7f921015e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 -- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92100b4140 msgr2=0x7f92101f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92100b4140 0x7f92101f5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 -- 172.21.15.129:0/3403360069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92101f5950 con 0x7f92100cd1c0 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.277+0000 7f920f400640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9208009cd0 tx=0x7f9208031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:01.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.278+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208031990 con 0x7f92100cd1c0 2024-09-17T03:55:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.278+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9208031c20 con 0x7f92100cd1c0 2024-09-17T03:55:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.278+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9208033070 con 0x7f92100cd1c0 2024-09-17T03:55:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.278+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92100c4cf0 con 0x7f92100cd1c0 2024-09-17T03:55:01.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.278+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92100c52b0 con 0x7f92100cd1c0 2024-09-17T03:55:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.279+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92101f9e30 con 0x7f92100cd1c0 2024-09-17T03:55:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.280+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9208033210 con 0x7f92100cd1c0 2024-09-17T03:55:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.281+0000 7f9207400640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 0x7f91ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.281+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f92080bbe70 con 0x7f92100cd1c0 2024-09-17T03:55:01.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.282+0000 7f920ea00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 0x7f91ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.282+0000 7f920ea00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 0x7f91ec07b0a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9200007d60 tx=0x7f9200007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:01.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.285+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f920803f030 con 0x7f92100cd1c0 2024-09-17T03:55:01.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.469+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92100ccc80 con 0x7f92100cd1c0 2024-09-17T03:55:01.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.470+0000 7f9207400640 1 -- 172.21.15.129:0/3403360069 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+13215 (secure 0 0 0) 0x7f9208083f80 con 0x7f92100cd1c0 2024-09-17T03:55:01.470 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:01.470 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":99,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:54:59.013727+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/3654208591":"2024-09-18T03:54:56.002413+0000","172.21.15.129:6833/3654208591":"2024-09-18T03:54:56.002413+0000","172.21.15.129:6834/1986086177":"2024-09-18T03:54:17.549442+0000","172.21.15.129:6833/1986086177":"2024-09-18T03:54:17.549442+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 msgr2=0x7f91ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 0x7f91ec07b0a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9200007d60 tx=0x7f9200007cf0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 msgr2=0x7f92101f9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f9208009cd0 tx=0x7f9208031700 comp rx=0 tx=0).stop 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 shutdown_connections 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.472+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f91ec078b90 0x7f91ec07b0a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92100cd1c0 0x7f92101f9100 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92100b4140 0x7f92101f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 --2- 172.21.15.129:0/3403360069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92100705a0 0x7f921015e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 >> 172.21.15.129:0/3403360069 conn(0x7f921006d410 msgr2=0x7f921006eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:01.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 shutdown_connections 2024-09-17T03:55:01.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.473+0000 7f9215c00640 1 -- 172.21.15.129:0/3403360069 wait complete. 2024-09-17T03:55:01.496 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:55:01.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:55:01.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 -- 172.21.15.129:0/397807514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c0b49a0 msgr2=0x7fc44c0cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 --2- 172.21.15.129:0/397807514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c0b49a0 0x7fc44c0cc1a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fc444005fd0 tx=0x7fc44402a820 comp rx=0 tx=0).stop 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 -- 172.21.15.129:0/397807514 shutdown_connections 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 --2- 172.21.15.129:0/397807514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c0b49a0 0x7fc44c0cc1a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 -- 172.21.15.129:0/397807514 >> 172.21.15.129:0/397807514 conn(0x7fc44c0ccc40 msgr2=0x7fc44c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 -- 172.21.15.129:0/397807514 shutdown_connections 2024-09-17T03:55:01.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.719+0000 7fc452e00640 1 -- 172.21.15.129:0/397807514 wait complete. 2024-09-17T03:55:01.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.721+0000 7fc452e00640 1 Processor -- start 2024-09-17T03:55:01.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.721+0000 7fc452e00640 1 -- start start 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc44c0b49a0 0x7fc44c1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c1591f0 0x7fc44c159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c167f30 con 0x7fc44c074580 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c0cca00 con 0x7fc44c1591f0 2024-09-17T03:55:01.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc452e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c1680b0 con 0x7fc44c0b49a0 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33972/0 (socket says 172.21.15.129:33972) 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 -- 172.21.15.129:0/545339004 learned_addr learned my addr 172.21.15.129:0/545339004 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc450e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c1591f0 0x7fc44c159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44b400640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc44c0b49a0 0x7fc44c1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 -- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc44c0b49a0 msgr2=0x7fc44c1f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc44c0b49a0 0x7fc44c1f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 -- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c1591f0 msgr2=0x7fc44c159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c1591f0 0x7fc44c159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.723+0000 7fc44be00640 1 -- 172.21.15.129:0/545339004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c159b40 con 0x7fc44c074580 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc44be00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b3f0 tx=0x7fc43c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:01.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c015020 con 0x7fc44c074580 2024-09-17T03:55:01.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc43c011920 con 0x7fc44c074580 2024-09-17T03:55:01.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c011d70 con 0x7fc44c074580 2024-09-17T03:55:01.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44c14f740 con 0x7fc44c074580 2024-09-17T03:55:01.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.724+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44c1f4100 con 0x7fc44c074580 2024-09-17T03:55:01.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.725+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44c0b3f70 con 0x7fc44c074580 2024-09-17T03:55:01.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.726+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc43c018020 con 0x7fc44c074580 2024-09-17T03:55:01.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.728+0000 7fc443e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 0x7fc42c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:01.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.728+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7fc43c09a9a0 con 0x7fc44c074580 2024-09-17T03:55:01.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.729+0000 7fc44b400640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 0x7fc42c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:01.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.729+0000 7fc44b400640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 0x7fc42c07b0d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc44c1557e0 tx=0x7fc438007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:01.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.733+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc43c01f020 con 0x7fc44c074580 2024-09-17T03:55:01.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.916+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc44c09db20 con 0x7fc44c074580 2024-09-17T03:55:01.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.917+0000 7fc443e00640 1 -- 172.21.15.129:0/545339004 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7fc44c159b40 con 0x7fc44c074580 2024-09-17T03:55:01.917 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:55:01.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 msgr2=0x7fc42c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 0x7fc42c07b0d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc44c1557e0 tx=0x7fc438007e90 comp rx=0 tx=0).stop 2024-09-17T03:55:01.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 msgr2=0x7fc44c14f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00b3f0 tx=0x7fc43c00b8b0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 shutdown_connections 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc42c078bc0 0x7fc42c07b0d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc44c1591f0 0x7fc44c159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc44c0b49a0 0x7fc44c1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 --2- 172.21.15.129:0/545339004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc44c074580 0x7fc44c14f200 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 >> 172.21.15.129:0/545339004 conn(0x7fc44c0ccc40 msgr2=0x7fc44c0b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 shutdown_connections 2024-09-17T03:55:01.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:01.919+0000 7fc452e00640 1 -- 172.21.15.129:0/545339004 wait complete. 2024-09-17T03:55:01.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.162+0000 7f778ec00640 1 -- 172.21.15.129:0/993284092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77880705a0 msgr2=0x7f77880b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.162+0000 7f778ec00640 1 --2- 172.21.15.129:0/993284092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77880705a0 0x7f77880b4140 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7778009830 tx=0x7f777802a510 comp rx=0 tx=0).stop 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.163+0000 7f778ec00640 1 -- 172.21.15.129:0/993284092 shutdown_connections 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.163+0000 7f778ec00640 1 --2- 172.21.15.129:0/993284092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77880705a0 0x7f77880b4140 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.163+0000 7f778ec00640 1 -- 172.21.15.129:0/993284092 >> 172.21.15.129:0/993284092 conn(0x7f778806d410 msgr2=0x7f77880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:02.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.163+0000 7f778ec00640 1 -- 172.21.15.129:0/993284092 shutdown_connections 2024-09-17T03:55:02.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.163+0000 7f778ec00640 1 -- 172.21.15.129:0/993284092 wait complete. 2024-09-17T03:55:02.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.165+0000 7f778ec00640 1 Processor -- start 2024-09-17T03:55:02.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.165+0000 7f778ec00640 1 -- start start 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.166+0000 7f778ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f778806eed0 0x7f77880c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.166+0000 7f778ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77880705a0 0x7f77880cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f778806f2e0 con 0x7f77881f8de0 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77881612e0 con 0x7f77880705a0 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f778806f460 con 0x7f778806eed0 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:33986/0 (socket says 172.21.15.129:33986) 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 -- 172.21.15.129:0/3363846581 learned_addr learned my addr 172.21.15.129:0/3363846581 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:02.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f7787e00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f778806eed0 0x7f77880c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f7787400640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77880705a0 0x7f77880cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 -- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f778806eed0 msgr2=0x7f77880c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f778806eed0 0x7f77880c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 -- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77880705a0 msgr2=0x7f77880cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77880705a0 0x7f77880cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:02.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.167+0000 7f778cc00640 1 -- 172.21.15.129:0/3363846581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77881f94c0 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f778cc00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f778000c730 tx=0x7f778000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7780003b70 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7780003d30 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77800052d0 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77881f97a0 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.168+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77881f5990 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.169+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77880b4140 con 0x7f77881f8de0 2024-09-17T03:55:02.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.170+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7780005470 con 0x7f77881f8de0 2024-09-17T03:55:02.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.171+0000 7f777fe00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 0x7f776807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:02.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.171+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f778009a740 con 0x7f77881f8de0 2024-09-17T03:55:02.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.172+0000 7f7787e00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 0x7f776807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:02.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.172+0000 7f7787e00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 0x7f776807b0a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7778034000 tx=0x7f7778004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:02.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.176+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7780062730 con 0x7f77881f8de0 2024-09-17T03:55:02.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:02.427+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) -- 0x7f77881f9250 con 0x7f77881f8de0 2024-09-17T03:55:03.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.031+0000 7f777fe00640 1 -- 172.21.15.129:0/3363846581 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v299) ==== 145+0+0 (secure 0 0 0) 0x7f7780067160 con 0x7f77881f8de0 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 msgr2=0x7f776807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 0x7f776807b0a0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7778034000 tx=0x7f7778004730 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 msgr2=0x7f778815d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f778000c730 tx=0x7f778000cbf0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 shutdown_connections 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7768078b90 0x7f776807b0a0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77881f8de0 0x7f778815d980 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77880705a0 0x7f77880cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 --2- 172.21.15.129:0/3363846581 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f778806eed0 0x7f77880c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.034+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 >> 172.21.15.129:0/3363846581 conn(0x7f778806d410 msgr2=0x7f77880ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.035+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 shutdown_connections 2024-09-17T03:55:03.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.035+0000 7f778ec00640 1 -- 172.21.15.129:0/3363846581 wait complete. 2024-09-17T03:55:03.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 -- 172.21.15.129:0/3182297184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 msgr2=0x7f1760003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 --2- 172.21.15.129:0/3182297184 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f1760003f10 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f17780b4450 tx=0x7f1760005830 comp rx=0 tx=0).stop 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177e600640 1 -- 172.21.15.129:0/3182297184 reap_dead start 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 -- 172.21.15.129:0/3182297184 shutdown_connections 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 -- 172.21.15.129:0/3182297184 >> 172.21.15.129:0/3182297184 conn(0x7f177806cfc0 msgr2=0x7f177806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 -- 172.21.15.129:0/3182297184 shutdown_connections 2024-09-17T03:55:03.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.282+0000 7f177f000640 1 -- 172.21.15.129:0/3182297184 wait complete. 2024-09-17T03:55:03.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.285+0000 7f177f000640 1 Processor -- start 2024-09-17T03:55:03.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.285+0000 7f177f000640 1 -- start start 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 0x7f17781f6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17781f3a90 0x7f17781f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17780d0440 con 0x7f1760003aa0 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17780d02c0 con 0x7f17781f3a90 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17780d05c0 con 0x7f1778141890 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34006/0 (socket says 172.21.15.129:34006) 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.286+0000 7f177dc00640 1 -- 172.21.15.129:0/3656277967 learned_addr learned my addr 172.21.15.129:0/3656277967 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177e600640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17781f3a90 0x7f17781f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177d200640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 0x7f17781f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 -- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 msgr2=0x7f17781f6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 0x7f17781f6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 -- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17781f3a90 msgr2=0x7f17781f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17781f3a90 0x7f17781f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 -- 172.21.15.129:0/3656277967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17781573d0 con 0x7f1760003aa0 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177d200640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 0x7f17781f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:03.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177dc00640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f176c02b2f0 tx=0x7f176c00c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c039070 con 0x7f1760003aa0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f176c02b600 con 0x7f1760003aa0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c02ba50 con 0x7f1760003aa0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.287+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17781576b0 con 0x7f1760003aa0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.288+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1778141d40 con 0x7f1760003aa0 2024-09-17T03:55:03.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.289+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1778141fa0 con 0x7f1760003aa0 2024-09-17T03:55:03.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.290+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f176c02bbf0 con 0x7f1760003aa0 2024-09-17T03:55:03.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.291+0000 7f1775600640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 0x7f175807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.291+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f176c0bacd0 con 0x7f1760003aa0 2024-09-17T03:55:03.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.292+0000 7f177d200640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 0x7f175807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.292+0000 7f177d200640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 0x7f175807b0a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f17780a96e0 tx=0x7f1764005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:03.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.296+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f17780d0c10 con 0x7f1760003aa0 2024-09-17T03:55:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.485+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f177815d660 con 0x7f1760003aa0 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.486+0000 7f1775600640 1 -- 172.21.15.129:0/3656277967 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v47) ==== 59+0+2294 (secure 0 0 0) 0x7f176c0833a0 con 0x7f1760003aa0 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:55:03.486 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:55:03.487 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:55:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:55:03.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 msgr2=0x7f175807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 0x7f175807b0a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f17780a96e0 tx=0x7f1764005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 msgr2=0x7f17781f3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f176c02b2f0 tx=0x7f176c00c2c0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 shutdown_connections 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1758078b90 0x7f175807b0a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17781f3a90 0x7f17781f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1778141890 0x7f17781f6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 --2- 172.21.15.129:0/3656277967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1760003aa0 0x7f17781f3550 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 >> 172.21.15.129:0/3656277967 conn(0x7f177806cfc0 msgr2=0x7f17780b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 shutdown_connections 2024-09-17T03:55:03.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.489+0000 7f177f000640 1 -- 172.21.15.129:0/3656277967 wait complete. 2024-09-17T03:55:03.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:55:03.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.738+0000 7ff77a600640 1 -- 172.21.15.129:0/2643749956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff774073350 msgr2=0x7ff7740ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.738+0000 7ff77a600640 1 --2- 172.21.15.129:0/2643749956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff774073350 0x7ff7740ce480 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff76c005fd0 tx=0x7ff76c02a830 comp rx=0 tx=0).stop 2024-09-17T03:55:03.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 -- 172.21.15.129:0/2643749956 shutdown_connections 2024-09-17T03:55:03.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 --2- 172.21.15.129:0/2643749956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff774073350 0x7ff7740ce480 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 --2- 172.21.15.129:0/2643749956 >> v2:172.21.15.181:3300/0 conn(0x7ff7740cb080 0x7ff774072e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 -- 172.21.15.129:0/2643749956 >> 172.21.15.129:0/2643749956 conn(0x7ff7740d00b0 msgr2=0x7ff7740d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:03.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 -- 172.21.15.129:0/2643749956 shutdown_connections 2024-09-17T03:55:03.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.739+0000 7ff77a600640 1 -- 172.21.15.129:0/2643749956 wait complete. 2024-09-17T03:55:03.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.742+0000 7ff77a600640 1 Processor -- start 2024-09-17T03:55:03.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.742+0000 7ff77a600640 1 -- start start 2024-09-17T03:55:03.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.743+0000 7ff77a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 0x7ff77409c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.743+0000 7ff77a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 0x7ff7741f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740ced50 con 0x7ff77415c7b0 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff7740cebd0 con 0x7ff774073350 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff7740ceed0 con 0x7ff7740cb080 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 0x7ff77409c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff76be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 0x7ff7741f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff76be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 0x7ff7741f7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:55770/0 (socket says 172.21.15.129:55770) 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 0x7ff77409c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:56326/0 (socket says 172.21.15.129:56326) 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff76be00640 1 -- 172.21.15.129:0/1707838511 learned_addr learned my addr 172.21.15.129:0/1707838511 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34014/0 (socket says 172.21.15.129:34014) 2024-09-17T03:55:03.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 -- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 msgr2=0x7ff7741f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 0x7ff7741f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 -- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 msgr2=0x7ff77409c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 0x7ff77409c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 -- 172.21.15.129:0/1707838511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7741f3730 con 0x7ff77415c7b0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff779c00640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009000 tx=0x7ff76c032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c041020 con 0x7ff77415c7b0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff76c032e50 con 0x7ff77415c7b0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c036040 con 0x7ff77415c7b0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7741f3a10 con 0x7ff77415c7b0 2024-09-17T03:55:03.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.744+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741f3fd0 con 0x7ff77415c7b0 2024-09-17T03:55:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.745+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7740d0f50 con 0x7ff77415c7b0 2024-09-17T03:55:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.747+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff76c044020 con 0x7ff77415c7b0 2024-09-17T03:55:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.747+0000 7ff771600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 0x7ff75007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:03.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.748+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7ff76c0bb8c0 con 0x7ff77415c7b0 2024-09-17T03:55:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.748+0000 7ff779200640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 0x7ff75007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:03.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.749+0000 7ff779200640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 0x7ff75007b0d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff764007a40 tx=0x7ff764010040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:03.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.752+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff76c04b030 con 0x7ff77415c7b0 2024-09-17T03:55:03.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.936+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff77409cda0 con 0x7ff77415c7b0 2024-09-17T03:55:03.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.940+0000 7ff771600640 1 -- 172.21.15.129:0/1707838511 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) ==== 122+0+0 (secure 0 0 0) 0x7ff76c083940 con 0x7ff77415c7b0 2024-09-17T03:55:03.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 msgr2=0x7ff75007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 0x7ff75007b0d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff764007a40 tx=0x7ff764010040 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 msgr2=0x7ff7740738a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7ff76c009000 tx=0x7ff76c032c60 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 shutdown_connections 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff750078bc0 0x7ff75007b0d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff77415c7b0 0x7ff7740738a0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7740cb080 0x7ff7741f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 --2- 172.21.15.129:0/1707838511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff774073350 0x7ff77409c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 >> 172.21.15.129:0/1707838511 conn(0x7ff7740d00b0 msgr2=0x7ff7740ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 shutdown_connections 2024-09-17T03:55:03.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:03.943+0000 7ff77a600640 1 -- 172.21.15.129:0/1707838511 wait complete. 2024-09-17T03:55:03.969 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:55:03.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:55:04.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:55:04.164 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:55:04.219 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:04.219 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:04.256 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:55:04.256 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:55:04.256 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:04.296 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:55:04.296 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:04.297 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:04.333 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:55:04.334 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:55:04.334 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:04.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:55:04.386 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:55:04.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.613+0000 7f5f0ec00640 1 -- 172.21.15.129:0/3866829162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815d7e0 msgr2=0x7f5f080cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.613+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/3866829162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815d7e0 0x7f5f080cee00 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4005fd0 tx=0x7f5ef402a540 comp rx=0 tx=0).stop 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 -- 172.21.15.129:0/3866829162 shutdown_connections 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/3866829162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815d7e0 0x7f5f080cee00 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/3866829162 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815c8a0 0x7f5f080ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 -- 172.21.15.129:0/3866829162 >> 172.21.15.129:0/3866829162 conn(0x7f5f08069330 msgr2=0x7f5f08071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 -- 172.21.15.129:0/3866829162 shutdown_connections 2024-09-17T03:55:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.614+0000 7f5f0ec00640 1 -- 172.21.15.129:0/3866829162 wait complete. 2024-09-17T03:55:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.616+0000 7f5f0ec00640 1 Processor -- start 2024-09-17T03:55:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.616+0000 7f5f0ec00640 1 -- start start 2024-09-17T03:55:04.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f080cf340 0x7f5f0814a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:04.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:04.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815d7e0 0x7f5f081e9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f080b4580 con 0x7f5f0815c8a0 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0815dd30 con 0x7f5f080cf340 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5f080b4700 con 0x7f5f0815d7e0 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55416/0 (socket says 172.21.15.129:55416) 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 -- 172.21.15.129:0/2841365137 learned_addr learned my addr 172.21.15.129:0/2841365137 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0d800640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f080cf340 0x7f5f0814a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0e200640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815d7e0 0x7f5f081e9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 -- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815d7e0 msgr2=0x7f5f081e9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815d7e0 0x7f5f081e9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 -- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f080cf340 msgr2=0x7f5f0814a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f080cf340 0x7f5f0814a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.618+0000 7f5f0ce00640 1 -- 172.21.15.129:0/2841365137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f081474d0 con 0x7f5f0815c8a0 2024-09-17T03:55:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f0ce00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4034040 tx=0x7f5ef4004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef403e030 con 0x7f5f0815c8a0 2024-09-17T03:55:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ef40047a0 con 0x7f5f0815c8a0 2024-09-17T03:55:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4035040 con 0x7f5f0815c8a0 2024-09-17T03:55:04.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f080cc300 con 0x7f5f0815c8a0 2024-09-17T03:55:04.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.619+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f080c9800 con 0x7f5f0815c8a0 2024-09-17T03:55:04.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.620+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f080d2e20 con 0x7f5f0815c8a0 2024-09-17T03:55:04.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.622+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5ef4041030 con 0x7f5f0815c8a0 2024-09-17T03:55:04.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.623+0000 7f5f05600640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 0x7f5ee807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:04.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.623+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f5ef40bb560 con 0x7f5f0815c8a0 2024-09-17T03:55:04.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.623+0000 7f5f0d800640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 0x7f5ee807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:04.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.624+0000 7f5f0d800640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 0x7f5ee807b0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009b30 tx=0x7f5efc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:04.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.626+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5f080d2e20 con 0x7f5f0815c8a0 2024-09-17T03:55:04.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.846+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f080cc090 con 0x7f5f0815c8a0 2024-09-17T03:55:04.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.847+0000 7f5f05600640 1 -- 172.21.15.129:0/2841365137 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+3920 (secure 0 0 0) 0x7f5ef40b9e20 con 0x7f5f0815c8a0 2024-09-17T03:55:04.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:04.846 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":73,"btime":"2024-09-17T03:54:57:005337+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/977755666","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":977755666},{"type":"v1","addr":"172.21.15.181:6835","nonce":977755666}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":5873,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1393925792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1393925792},{"type":"v1","addr":"172.21.15.129:6834","nonce":1393925792}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1026314889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1026314889},{"type":"v1","addr":"172.21.15.181:6837","nonce":1026314889}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68},{"gid":6909,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2789223749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2789223749},{"type":"v1","addr":"172.21.15.129:6835","nonce":2789223749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":73}],"filesystems":[]} 2024-09-17T03:55:04.846 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 73 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 msgr2=0x7f5ee807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 0x7f5ee807b0a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009b30 tx=0x7f5efc005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 msgr2=0x7f5f081e9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4034040 tx=0x7f5ef4004c80 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 shutdown_connections 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ee8078b90 0x7f5ee807b0a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f0815d7e0 0x7f5f081e9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f0815c8a0 0x7f5f081e9210 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 --2- 172.21.15.129:0/2841365137 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f080cf340 0x7f5f0814a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 >> 172.21.15.129:0/2841365137 conn(0x7f5f08069330 msgr2=0x7f5f080b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:04.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.849+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 shutdown_connections 2024-09-17T03:55:04.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:04.850+0000 7f5f0ec00640 1 -- 172.21.15.129:0/2841365137 wait complete. 2024-09-17T03:55:04.871 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:55:04.872 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:55:04.872 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:55:04.872 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:04.872 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:55:04.872 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:55:04.872 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:55:04.872 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:04.873 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:55:04.873 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:55:04.873 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:55:04.873 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:04.873 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:55:04.873 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:55:04.874 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:55:04.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:04.892 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:55:04.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:55:04.893 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:55:04.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:55:04.896 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:55:04.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:55:04.897 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:55:04.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:55:04.937 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:55:04.937 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:04.939 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:55:04.939 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:04.941 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:55:04.941 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:04.944 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:55:04.944 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:04.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:55:05.135 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:55:05.136 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:55:05.138 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:55:05.139 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:55:05.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 -- 172.21.15.129:0/1197392306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 msgr2=0x7f40d40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:05.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 --2- 172.21.15.129:0/1197392306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d40ccc80 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f40d4072e00 tx=0x7f40c002a540 comp rx=0 tx=0).stop 2024-09-17T03:55:05.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 -- 172.21.15.129:0/1197392306 shutdown_connections 2024-09-17T03:55:05.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 --2- 172.21.15.129:0/1197392306 >> v2:172.21.15.181:3300/0 conn(0x7f40d40cd1c0 0x7f40d41609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:05.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 --2- 172.21.15.129:0/1197392306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d40ccc80 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:05.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 -- 172.21.15.129:0/1197392306 >> 172.21.15.129:0/1197392306 conn(0x7f40d406d410 msgr2=0x7f40d40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:05.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 -- 172.21.15.129:0/1197392306 shutdown_connections 2024-09-17T03:55:05.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.179+0000 7f40dba00640 1 -- 172.21.15.129:0/1197392306 wait complete. 2024-09-17T03:55:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.182+0000 7f40dba00640 1 Processor -- start 2024-09-17T03:55:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.182+0000 7f40dba00640 1 -- start start 2024-09-17T03:55:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 0x7f40d4159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:05.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d40cd1c0 0x7f40d41f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40d4160260 con 0x7f40d40b4140 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f40d41600e0 con 0x7f40d40cd1c0 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40dba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f40d4160f00 con 0x7f40d40705a0 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40d3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55458/0 (socket says 172.21.15.129:55458) 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.183+0000 7f40d3e00640 1 -- 172.21.15.129:0/791424871 learned_addr learned my addr 172.21.15.129:0/791424871 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d9a00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d40cd1c0 0x7f40d41f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d9000640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 0x7f40d4159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 -- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 msgr2=0x7f40d4159720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 0x7f40d4159720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 -- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d40cd1c0 msgr2=0x7f40d41f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d40cd1c0 0x7f40d41f9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 -- 172.21.15.129:0/791424871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d4159c60 con 0x7f40d40b4140 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d9000640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 0x7f40d4159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:05.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d3e00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f40c002ab30 tx=0x7f40c0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0031d60 con 0x7f40d40b4140 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.184+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40c0033e10 con 0x7f40d40b4140 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.185+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d415c040 con 0x7f40d40b4140 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.185+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0005260 con 0x7f40d40b4140 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.185+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d415c600 con 0x7f40d40b4140 2024-09-17T03:55:05.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.186+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d40ccc80 con 0x7f40d40b4140 2024-09-17T03:55:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.187+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f40c0005400 con 0x7f40d40b4140 2024-09-17T03:55:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.189+0000 7f40d0c00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 0x7f40b407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.189+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5029+0+0 (secure 0 0 0) 0x7f40c00bbff0 con 0x7f40d40b4140 2024-09-17T03:55:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.189+0000 7f40d9000640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 0x7f40b407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.190+0000 7f40d9000640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 0x7f40b407b0d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40c8000f60 tx=0x7f40c8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:05.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.192+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f40c003f030 con 0x7f40d40b4140 2024-09-17T03:55:05.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:05.388+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f40d406a140 con 0x7f40d40b4140 2024-09-17T03:55:06.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.051+0000 7f40d0c00640 1 -- 172.21.15.129:0/791424871 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v101) ==== 97+0+0 (secure 0 0 0) 0x7f40c0005d00 con 0x7f40d40b4140 2024-09-17T03:55:06.050 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 msgr2=0x7f40b407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 0x7f40b407b0d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f40c8000f60 tx=0x7f40c8009040 comp rx=0 tx=0).stop 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 msgr2=0x7f40d41f9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f40c002ab30 tx=0x7f40c0033c20 comp rx=0 tx=0).stop 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 shutdown_connections 2024-09-17T03:55:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f40b4078bc0 0x7f40b407b0d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40d40cd1c0 0x7f40d41f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40d40b4140 0x7f40d41f9410 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 --2- 172.21.15.129:0/791424871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40d40705a0 0x7f40d4159720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 >> 172.21.15.129:0/791424871 conn(0x7f40d406d410 msgr2=0x7f40d40adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 shutdown_connections 2024-09-17T03:55:06.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.054+0000 7f40dba00640 1 -- 172.21.15.129:0/791424871 wait complete. 2024-09-17T03:55:06.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:55:06.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 -- 172.21.15.129:0/630532281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 msgr2=0x7f782c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 --2- 172.21.15.129:0/630532281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0ccc80 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f7814009830 tx=0x7f781402a5c0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 -- 172.21.15.129:0/630532281 shutdown_connections 2024-09-17T03:55:06.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 --2- 172.21.15.129:0/630532281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0ccc80 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 -- 172.21.15.129:0/630532281 >> 172.21.15.129:0/630532281 conn(0x7f782c06d410 msgr2=0x7f782c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:06.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 -- 172.21.15.129:0/630532281 shutdown_connections 2024-09-17T03:55:06.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.301+0000 7f7833600640 1 -- 172.21.15.129:0/630532281 wait complete. 2024-09-17T03:55:06.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.303+0000 7f7833600640 1 Processor -- start 2024-09-17T03:55:06.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.304+0000 7f7833600640 1 -- start start 2024-09-17T03:55:06.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 0x7f782c1f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 0x7f782c15a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f782c160260 con 0x7f782c0b4140 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f782c1600e0 con 0x7f782c0cd1c0 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7833600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f782c1610d0 con 0x7f782c0705a0 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7823e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7823e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55474/0 (socket says 172.21.15.129:55474) 2024-09-17T03:55:06.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.305+0000 7f7823e00640 1 -- 172.21.15.129:0/1778467767 learned_addr learned my addr 172.21.15.129:0/1778467767 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:06.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7831600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 0x7f782c15a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7830c00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 0x7f782c1f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 -- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 msgr2=0x7f782c1f9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 0x7f782c1f9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 -- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 msgr2=0x7f782c15a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 0x7f782c15a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 -- 172.21.15.129:0/1778467767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782c1f6d90 con 0x7f782c0b4140 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7831600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 0x7f782c15a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7830c00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 0x7f782c1f9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:06.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7823e00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:06.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814031d60 con 0x7f782c0b4140 2024-09-17T03:55:06.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7814033070 con 0x7f782c0b4140 2024-09-17T03:55:06.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782c1f5fd0 con 0x7f782c0b4140 2024-09-17T03:55:06.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78140334c0 con 0x7f782c0b4140 2024-09-17T03:55:06.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.306+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f782c1f6590 con 0x7f782c0b4140 2024-09-17T03:55:06.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.308+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f782c0c4780 con 0x7f782c0b4140 2024-09-17T03:55:06.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.309+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7814033660 con 0x7f782c0b4140 2024-09-17T03:55:06.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.310+0000 7f7829600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 0x7f780807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.310+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 4857+0+0 (secure 0 0 0) 0x7f78140bc0b0 con 0x7f782c0b4140 2024-09-17T03:55:06.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.310+0000 7f7830c00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 0x7f780807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.311+0000 7f7830c00640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 0x7f780807b0d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f60 tx=0x7f781c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:06.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.314+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f781403f030 con 0x7f782c0b4140 2024-09-17T03:55:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.564+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f782c0ccc80 con 0x7f782c0b4140 2024-09-17T03:55:06.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.565+0000 7f7829600640 1 -- 172.21.15.129:0/1778467767 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v32) ==== 80+0+5304 (secure 0 0 0) 0x7f78140847a0 con 0x7f782c0b4140 2024-09-17T03:55:06.564 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:06.564 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:55:06.565 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.569 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:55:06.570 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:55:06.572 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.573 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:55:06.575 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.576 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:55:06.577 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.579 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.580 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:06.581 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:55:06.581 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 msgr2=0x7f780807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 0x7f780807b0d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f781c000f60 tx=0x7f781c009040 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 msgr2=0x7f782c0998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f7814000c00 tx=0x7f7814033d60 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 shutdown_connections 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7808078bc0 0x7f780807b0d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f782c0cd1c0 0x7f782c15a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f782c0b4140 0x7f782c0998b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.567+0000 7f7833600640 1 --2- 172.21.15.129:0/1778467767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f782c0705a0 0x7f782c1f9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.568+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 >> 172.21.15.129:0/1778467767 conn(0x7f782c06d410 msgr2=0x7f782c06eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:06.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.568+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 shutdown_connections 2024-09-17T03:55:06.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.568+0000 7f7833600640 1 -- 172.21.15.129:0/1778467767 wait complete. 2024-09-17T03:55:06.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:55:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 -- 172.21.15.129:0/2371963784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec140705a0 msgr2=0x7fec140b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 --2- 172.21.15.129:0/2371963784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec140705a0 0x7fec140b4140 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fec04009830 tx=0x7fec0402a5b0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 -- 172.21.15.129:0/2371963784 shutdown_connections 2024-09-17T03:55:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 --2- 172.21.15.129:0/2371963784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec140705a0 0x7fec140b4140 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 -- 172.21.15.129:0/2371963784 >> 172.21.15.129:0/2371963784 conn(0x7fec1406d410 msgr2=0x7fec140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 -- 172.21.15.129:0/2371963784 shutdown_connections 2024-09-17T03:55:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.810+0000 7fec1b800640 1 -- 172.21.15.129:0/2371963784 wait complete. 2024-09-17T03:55:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.812+0000 7fec1b800640 1 Processor -- start 2024-09-17T03:55:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.812+0000 7fec1b800640 1 -- start start 2024-09-17T03:55:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 0x7fec141f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 0x7fec14159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec1406f2e0 con 0x7fec1406eed0 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fec141612e0 con 0x7fec14155480 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec1b800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fec1406f460 con 0x7fec140705a0 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.814+0000 7fec18e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55484/0 (socket says 172.21.15.129:55484) 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 -- 172.21.15.129:0/4246550639 learned_addr learned my addr 172.21.15.129:0/4246550639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec19800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 0x7fec14159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec0be00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 0x7fec141f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 -- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 msgr2=0x7fec141f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 0x7fec141f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 -- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 msgr2=0x7fec14159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 0x7fec14159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 -- 172.21.15.129:0/4246550639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec140c65a0 con 0x7fec1406eed0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec19800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 0x7fec14159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec0be00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 0x7fec141f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec18e00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec04033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.815+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04031d60 con 0x7fec1406eed0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.816+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec04033070 con 0x7fec1406eed0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.816+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec141f7ad0 con 0x7fec1406eed0 2024-09-17T03:55:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.816+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec040334c0 con 0x7fec1406eed0 2024-09-17T03:55:06.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.816+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec14159680 con 0x7fec1406eed0 2024-09-17T03:55:06.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.817+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec14070190 con 0x7fec1406eed0 2024-09-17T03:55:06.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.818+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fec04033660 con 0x7fec1406eed0 2024-09-17T03:55:06.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.819+0000 7fec11600640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 0x7febf007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:06.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.819+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 4857+0+0 (secure 0 0 0) 0x7fec040bbfe0 con 0x7fec1406eed0 2024-09-17T03:55:06.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.819+0000 7fec0be00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 0x7febf007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:06.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.820+0000 7fec0be00640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 0x7febf007b0a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fec140c6a90 tx=0x7febfc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:06.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:06.823+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec0403f030 con 0x7fec1406eed0 2024-09-17T03:55:07.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.009+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fec140b4140 con 0x7fec1406eed0 2024-09-17T03:55:07.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.009+0000 7fec11600640 1 -- 172.21.15.129:0/4246550639 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v103) ==== 84+0+65 (secure 0 0 0) 0x7fec040847a0 con 0x7fec1406eed0 2024-09-17T03:55:07.008 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:07.009 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 msgr2=0x7febf007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 0x7febf007b0a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fec140c6a90 tx=0x7febfc009040 comp rx=0 tx=0).stop 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 msgr2=0x7fec140c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec04033d60 comp rx=0 tx=0).stop 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 shutdown_connections 2024-09-17T03:55:07.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febf0078b90 0x7febf007b0a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec14155480 0x7fec14159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec140705a0 0x7fec141f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 --2- 172.21.15.129:0/4246550639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec1406eed0 0x7fec140c8370 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 >> 172.21.15.129:0/4246550639 conn(0x7fec1406d410 msgr2=0x7fec141609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 shutdown_connections 2024-09-17T03:55:07.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.012+0000 7fec1b800640 1 -- 172.21.15.129:0/4246550639 wait complete. 2024-09-17T03:55:07.033 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:55:07.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 -- 172.21.15.129:0/2631340148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6415ca40 msgr2=0x7efd640cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 --2- 172.21.15.129:0/2631340148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6415ca40 0x7efd640cc990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7efd64153b30 tx=0x7efd5402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 -- 172.21.15.129:0/2631340148 shutdown_connections 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 --2- 172.21.15.129:0/2631340148 >> v2:172.21.15.181:3300/0 conn(0x7efd640cced0 0x7efd64160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 --2- 172.21.15.129:0/2631340148 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd6415ca40 0x7efd640cc990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 -- 172.21.15.129:0/2631340148 >> 172.21.15.129:0/2631340148 conn(0x7efd640b1c50 msgr2=0x7efd640b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 -- 172.21.15.129:0/2631340148 shutdown_connections 2024-09-17T03:55:07.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.261+0000 7efd69600640 1 -- 172.21.15.129:0/2631340148 wait complete. 2024-09-17T03:55:07.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.264+0000 7efd69600640 1 Processor -- start 2024-09-17T03:55:07.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.264+0000 7efd69600640 1 -- start start 2024-09-17T03:55:07.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd6415ca40 0x7efd640c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 0x7efd6415a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd6406f170 con 0x7efd640cced0 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efd640c6820 con 0x7efd6415ca40 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd69600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efd6406f2f0 con 0x7efd64160aa0 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.265+0000 7efd63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55512/0 (socket says 172.21.15.129:55512) 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 -- 172.21.15.129:0/1671200120 learned_addr learned my addr 172.21.15.129:0/1671200120 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd5b400640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd6415ca40 0x7efd640c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd68c00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 0x7efd6415a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 -- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 msgr2=0x7efd6415a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:07.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 0x7efd6415a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 -- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd6415ca40 msgr2=0x7efd640c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd6415ca40 0x7efd640c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 -- 172.21.15.129:0/1671200120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd641f33b0 con 0x7efd640cced0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd68c00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 0x7efd6415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.266+0000 7efd63e00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7efd5402abc0 tx=0x7efd54031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.267+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54033c00 con 0x7efd640cced0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.267+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd54031950 con 0x7efd640cced0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.267+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd641f3690 con 0x7efd640cced0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.267+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd54031d40 con 0x7efd640cced0 2024-09-17T03:55:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.267+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd641f3c50 con 0x7efd640cced0 2024-09-17T03:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.268+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6406f810 con 0x7efd640cced0 2024-09-17T03:55:07.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.269+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efd54045020 con 0x7efd640cced0 2024-09-17T03:55:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.270+0000 7efd60c00640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 0x7efd4007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.270+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 4857+0+0 (secure 0 0 0) 0x7efd540badc0 con 0x7efd640cced0 2024-09-17T03:55:07.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.270+0000 7efd5b400640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 0x7efd4007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:07.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.271+0000 7efd5b400640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 0x7efd4007b0a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efd640b6180 tx=0x7efd4c00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:07.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.274+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efd5404c020 con 0x7efd640cced0 2024-09-17T03:55:07.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:07.457+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7efd640c4f90 con 0x7efd640cced0 2024-09-17T03:55:08.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.066+0000 7efd60c00640 1 -- 172.21.15.129:0/1671200120 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v104) ==== 138+0+0 (secure 0 0 0) 0x7efd540834b0 con 0x7efd640cced0 2024-09-17T03:55:08.065 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.069+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 msgr2=0x7efd4007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.069+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 0x7efd4007b0a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7efd640b6180 tx=0x7efd4c00b2e0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.069+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 msgr2=0x7efd641588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.069+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7efd5402abc0 tx=0x7efd54031630 comp rx=0 tx=0).stop 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 shutdown_connections 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd40078b90 0x7efd4007b0a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd64160aa0 0x7efd6415a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd6415ca40 0x7efd640c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 --2- 172.21.15.129:0/1671200120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd640cced0 0x7efd641588c0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 >> 172.21.15.129:0/1671200120 conn(0x7efd640b1c50 msgr2=0x7efd6415bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:08.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 shutdown_connections 2024-09-17T03:55:08.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.070+0000 7efd69600640 1 -- 172.21.15.129:0/1671200120 wait complete. 2024-09-17T03:55:08.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:55:08.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 -- 172.21.15.129:0/3445162676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4003a50 msgr2=0x7f1ab4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:08.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/3445162676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4003a50 0x7f1ab4003ec0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0009b50 tx=0x7f1ab4005830 comp rx=0 tx=0).stop 2024-09-17T03:55:08.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 -- 172.21.15.129:0/3445162676 shutdown_connections 2024-09-17T03:55:08.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/3445162676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ab4003a50 0x7f1ab4003ec0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 -- 172.21.15.129:0/3445162676 >> 172.21.15.129:0/3445162676 conn(0x7f1ad006d7a0 msgr2=0x7f1ad00b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:08.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 -- 172.21.15.129:0/3445162676 shutdown_connections 2024-09-17T03:55:08.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.333+0000 7f1ad6c00640 1 -- 172.21.15.129:0/3445162676 wait complete. 2024-09-17T03:55:08.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.335+0000 7f1ad6c00640 1 Processor -- start 2024-09-17T03:55:08.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.336+0000 7f1ad6c00640 1 -- start start 2024-09-17T03:55:08.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 0x7f1ad01f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:08.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 0x7f1ad01f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:08.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:08.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00b2cb0 con 0x7f1ad01e6260 2024-09-17T03:55:08.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad00b2b30 con 0x7f1ab4003a50 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1ad00703b0 con 0x7f1ad01f0d50 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55532/0 (socket says 172.21.15.129:55532) 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1ad4c00640 1 -- 172.21.15.129:0/1706825974 learned_addr learned my addr 172.21.15.129:0/1706825974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1acfe00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 0x7f1ad01f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.337+0000 7f1acf400640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 0x7f1ad01f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:08.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 -- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 msgr2=0x7f1ad01f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 0x7f1ad01f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 -- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 msgr2=0x7f1ad01f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 0x7f1ad01f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 -- 172.21.15.129:0/1706825974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad01e6bb0 con 0x7f1ad01e6260 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1acfe00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 0x7f1ad01f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1acf400640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 0x7f1ad01f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad4c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006ddf0 tx=0x7f1ac80019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:08.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8011870 con 0x7f1ad01e6260 2024-09-17T03:55:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ac8017070 con 0x7f1ad01e6260 2024-09-17T03:55:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01f4960 con 0x7f1ad01e6260 2024-09-17T03:55:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8011dc0 con 0x7f1ad01e6260 2024-09-17T03:55:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.338+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad01f4f20 con 0x7f1ad01e6260 2024-09-17T03:55:08.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.339+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0070820 con 0x7f1ad01e6260 2024-09-17T03:55:08.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.341+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1ac801c460 con 0x7f1ad01e6260 2024-09-17T03:55:08.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.342+0000 7f1ac7e00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 0x7f1ab007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:08.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.342+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 5227+0+0 (secure 0 0 0) 0x7f1ac809bd90 con 0x7f1ad01e6260 2024-09-17T03:55:08.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.342+0000 7f1acfe00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 0x7f1ab007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:08.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.343+0000 7f1acfe00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 0x7f1ab007b0a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0035000 tx=0x7f1ac00046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:08.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.346+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1ad01e6bb0 con 0x7f1ad01e6260 2024-09-17T03:55:08.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:08.532+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f1ad01f46f0 con 0x7f1ad01e6260 2024-09-17T03:55:09.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.075+0000 7f1ac7e00640 1 -- 172.21.15.129:0/1706825974 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v105) ==== 170+0+0 (secure 0 0 0) 0x7f1ac8064520 con 0x7f1ad01e6260 2024-09-17T03:55:09.075 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:55:09.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.079+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 msgr2=0x7f1ab007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:09.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.079+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 0x7f1ab007b0a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0035000 tx=0x7f1ac00046c0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.079+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 msgr2=0x7f1ad01e6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:09.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.079+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f1ad006ddf0 tx=0x7f1ac80019b0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 shutdown_connections 2024-09-17T03:55:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ab0078b90 0x7f1ab007b0a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ad01e6260 0x7f1ad01e6670 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ad01f0d50 0x7f1ad01f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 --2- 172.21.15.129:0/1706825974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ab4003a50 0x7f1ad01f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 >> 172.21.15.129:0/1706825974 conn(0x7f1ad006d7a0 msgr2=0x7f1ad015e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:09.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 shutdown_connections 2024-09-17T03:55:09.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.080+0000 7f1ad6c00640 1 -- 172.21.15.129:0/1706825974 wait complete. 2024-09-17T03:55:09.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 -- 172.21.15.129:0/685026809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 msgr2=0x7f35040d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 --2- 172.21.15.129:0/685026809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040d0210 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f35040d43a0 tx=0x7f34f802a4d0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 -- 172.21.15.129:0/685026809 shutdown_connections 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 --2- 172.21.15.129:0/685026809 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d0f00 0x7f35040d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 --2- 172.21.15.129:0/685026809 >> v2:172.21.15.181:3300/0 conn(0x7f35040d2f60 0x7f3504162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 --2- 172.21.15.129:0/685026809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040d0210 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 -- 172.21.15.129:0/685026809 >> 172.21.15.129:0/685026809 conn(0x7f350408bff0 msgr2=0x7f350409d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:09.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 -- 172.21.15.129:0/685026809 shutdown_connections 2024-09-17T03:55:09.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.351+0000 7f350b200640 1 -- 172.21.15.129:0/685026809 wait complete. 2024-09-17T03:55:09.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.354+0000 7f350b200640 1 Processor -- start 2024-09-17T03:55:09.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.354+0000 7f350b200640 1 -- start start 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35040d0f00 0x7f35041f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d2f60 0x7f350415de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f35040b42b0 con 0x7f35040b3480 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f35040b4130 con 0x7f35040d0f00 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.355+0000 7f350b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f35040b4430 con 0x7f35040d2f60 2024-09-17T03:55:09.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55546/0 (socket says 172.21.15.129:55546) 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 -- 172.21.15.129:0/1695005757 learned_addr learned my addr 172.21.15.129:0/1695005757 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509400640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35040d0f00 0x7f35041f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f350a800640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d2f60 0x7f350415de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 -- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d2f60 msgr2=0x7f350415de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d2f60 0x7f350415de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 -- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35040d0f00 msgr2=0x7f35041f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35040d0f00 0x7f35041f6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:09.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 -- 172.21.15.129:0/1695005757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35041f58f0 con 0x7f35040b3480 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.356+0000 7f3509e00640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f34f8034040 tx=0x7f34f80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.357+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f803e030 con 0x7f35040b3480 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.357+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34f8003ca0 con 0x7f35040b3480 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.357+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35040cb690 con 0x7f35040b3480 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.357+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8035040 con 0x7f35040b3480 2024-09-17T03:55:09.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.357+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35041f5b60 con 0x7f35040b3480 2024-09-17T03:55:09.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.358+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35040b49c0 con 0x7f35040b3480 2024-09-17T03:55:09.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.359+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f34f8041030 con 0x7f35040b3480 2024-09-17T03:55:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.360+0000 7f34fe000640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 0x7f34e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.360+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 5609+0+0 (secure 0 0 0) 0x7f34f80bb2e0 con 0x7f35040b3480 2024-09-17T03:55:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.361+0000 7f3509400640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 0x7f34e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:09.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.362+0000 7f3509400640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 0x7f34e807b0a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35040b93b0 tx=0x7f34f4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:09.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.366+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f34f8083090 con 0x7f35040b3480 2024-09-17T03:55:09.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:09.588+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f35040b4e20 con 0x7f35040b3480 2024-09-17T03:55:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.078+0000 7f34fe000640 1 -- 172.21.15.129:0/1695005757 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v76) ==== 174+0+0 (secure 0 0 0) 0x7f34f80bb0c0 con 0x7f35040b3480 2024-09-17T03:55:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 16 and data pool 17 2024-09-17T03:55:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.085+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 msgr2=0x7f34e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.085+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 0x7f34e807b0a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f35040b93b0 tx=0x7f34f4005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.085+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 msgr2=0x7f35040cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.085+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f34f8034040 tx=0x7f34f80038a0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 shutdown_connections 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34e8078b90 0x7f34e807b0a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35040d2f60 0x7f350415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35040d0f00 0x7f35041f6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 --2- 172.21.15.129:0/1695005757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35040b3480 0x7f35040cabc0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.088+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 >> 172.21.15.129:0/1695005757 conn(0x7f350408bff0 msgr2=0x7f350406ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:10.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.089+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 shutdown_connections 2024-09-17T03:55:10.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.089+0000 7f350b200640 1 -- 172.21.15.129:0/1695005757 wait complete. 2024-09-17T03:55:10.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 -- 172.21.15.129:0/3921965599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67900052a0 msgr2=0x7f679000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 --2- 172.21.15.129:0/3921965599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67900052a0 0x7f679000b9c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f67a80cdaf0 tx=0x7f6790005830 comp rx=0 tx=0).stop 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 -- 172.21.15.129:0/3921965599 shutdown_connections 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 --2- 172.21.15.129:0/3921965599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67900052a0 0x7f679000b9c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 -- 172.21.15.129:0/3921965599 >> 172.21.15.129:0/3921965599 conn(0x7f67a806d410 msgr2=0x7f67a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 -- 172.21.15.129:0/3921965599 shutdown_connections 2024-09-17T03:55:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.366+0000 7f67b0800640 1 -- 172.21.15.129:0/3921965599 wait complete. 2024-09-17T03:55:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.369+0000 7f67b0800640 1 Processor -- start 2024-09-17T03:55:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.369+0000 7f67b0800640 1 -- start start 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67900052a0 0x7f67a80c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67a8073bd0 0x7f67a815d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67a8160260 con 0x7f67a8159230 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f67a81600e0 con 0x7f67a8073bd0 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67b0800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f67a806eed0 con 0x7f67900052a0 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67ade00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67900052a0 0x7f67a80c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67ad400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55558/0 (socket says 172.21.15.129:55558) 2024-09-17T03:55:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.370+0000 7f67ad400640 1 -- 172.21.15.129:0/3776506638 learned_addr learned my addr 172.21.15.129:0/3776506638 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ae800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67a8073bd0 0x7f67a815d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 -- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67900052a0 msgr2=0x7f67a80c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67900052a0 0x7f67a80c96c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 -- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67a8073bd0 msgr2=0x7f67a815d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67a8073bd0 0x7f67a815d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 -- 172.21.15.129:0/3776506638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67a8073fe0 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67ad400640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6798004ca0 tx=0x7f6798001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798015020 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.371+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6798011750 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.372+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6798011ba0 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.372+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67a815e790 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.372+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67a81f90e0 con 0x7f67a8159230 2024-09-17T03:55:10.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.373+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67a806f3e0 con 0x7f67a8159230 2024-09-17T03:55:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.374+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6798018020 con 0x7f67a8159230 2024-09-17T03:55:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.375+0000 7f67a2000640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 0x7f678c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.375+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5709+0+0 (secure 0 0 0) 0x7f679809b480 con 0x7f67a8159230 2024-09-17T03:55:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.375+0000 7f67ade00640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 0x7f678c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.376+0000 7f67ade00640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 0x7f678c07b0a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f679c001cc0 tx=0x7f679c006720 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.379+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f679801f020 con 0x7f67a8159230 2024-09-17T03:55:10.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.567+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f67a806a140 con 0x7f67a8159230 2024-09-17T03:55:10.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.568+0000 7f67a2000640 1 -- 172.21.15.129:0/3776506638 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16748 (secure 0 0 0) 0x7f67a8073fe0 con 0x7f67a8159230 2024-09-17T03:55:10.567 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:10.567 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":106,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:55:09.589688+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:55:07.469103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-17T03:55:08.543736+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 msgr2=0x7f678c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 0x7f678c07b0a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f679c001cc0 tx=0x7f679c006720 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 msgr2=0x7f67a8073690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6798004ca0 tx=0x7f6798001630 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 shutdown_connections 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f678c078b90 0x7f678c07b0a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67a8073bd0 0x7f67a815d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67a8159230 0x7f67a8073690 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 --2- 172.21.15.129:0/3776506638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67900052a0 0x7f67a80c96c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.570+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 >> 172.21.15.129:0/3776506638 conn(0x7f67a806d410 msgr2=0x7f67a80ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.571+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 shutdown_connections 2024-09-17T03:55:10.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.571+0000 7f67b0800640 1 -- 172.21.15.129:0/3776506638 wait complete. 2024-09-17T03:55:10.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:55:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 -- 172.21.15.129:0/324594774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0cd390 msgr2=0x7f179c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 --2- 172.21.15.129:0/324594774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0cd390 0x7f179c1609c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f179c0d1370 tx=0x7f179402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 -- 172.21.15.129:0/324594774 shutdown_connections 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 --2- 172.21.15.129:0/324594774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0cd390 0x7f179c1609c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 --2- 172.21.15.129:0/324594774 >> v2:172.21.15.181:3300/0 conn(0x7f179c0705a0 0x7f179c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 -- 172.21.15.129:0/324594774 >> 172.21.15.129:0/324594774 conn(0x7f179c06d410 msgr2=0x7f179c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 -- 172.21.15.129:0/324594774 shutdown_connections 2024-09-17T03:55:10.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.823+0000 7f17a2a00640 1 -- 172.21.15.129:0/324594774 wait complete. 2024-09-17T03:55:10.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.825+0000 7f17a2a00640 1 Processor -- start 2024-09-17T03:55:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.826+0000 7f17a2a00640 1 -- start start 2024-09-17T03:55:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f179c06eed0 0x7f179c08c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f179c159b00 0x7f179c1f88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f179c160250 con 0x7f179c0705a0 2024-09-17T03:55:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f179c06f4f0 con 0x7f179c06eed0 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f17a2a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f179c160f00 con 0x7f179c159b00 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f1793400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.827+0000 7f1793400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55572/0 (socket says 172.21.15.129:55572) 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 -- 172.21.15.129:0/1491932145 learned_addr learned my addr 172.21.15.129:0/1491932145 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f179be00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f179c06eed0 0x7f179c08c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f17a0a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f179c159b00 0x7f179c1f88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 -- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f179c159b00 msgr2=0x7f179c1f88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f179c159b00 0x7f179c1f88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 -- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f179c06eed0 msgr2=0x7f179c08c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f179c06eed0 0x7f179c08c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 -- 172.21.15.129:0/1491932145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f179c15a2a0 con 0x7f179c0705a0 2024-09-17T03:55:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.828+0000 7f1793400640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f178400c760 tx=0x7f178400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:10.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.829+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1784003b70 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.829+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1784003d10 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.829+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178400aab0 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.829+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179c15a580 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.829+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f179c1f7b00 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.830+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f179c0b4140 con 0x7f179c0705a0 2024-09-17T03:55:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.831+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f178400ac50 con 0x7f179c0705a0 2024-09-17T03:55:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.833+0000 7f1798c00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 0x7f177807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:10.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.833+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5709+0+0 (secure 0 0 0) 0x7f178409aa20 con 0x7f179c0705a0 2024-09-17T03:55:10.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.833+0000 7f179be00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 0x7f177807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:10.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.834+0000 7f179be00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 0x7f177807b0a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f178c009b30 tx=0x7f178c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:10.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:10.837+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1784062890 con 0x7f179c0705a0 2024-09-17T03:55:11.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.030+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f179c15a030 con 0x7f179c0705a0 2024-09-17T03:55:11.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.031+0000 7f1798c00640 1 -- 172.21.15.129:0/1491932145 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+16748 (secure 0 0 0) 0x7f17840672c0 con 0x7f179c0705a0 2024-09-17T03:55:11.031 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:11.032 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":106,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:55:09.589688+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:55:07.469103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-17T03:55:08.543736+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:55:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 msgr2=0x7f177807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 0x7f177807b0a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f178c009b30 tx=0x7f178c005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 msgr2=0x7f179c15e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f178400c760 tx=0x7f178400cc20 comp rx=0 tx=0).stop 2024-09-17T03:55:11.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 shutdown_connections 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1778078b90 0x7f177807b0a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f179c159b00 0x7f179c1f88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f179c0705a0 0x7f179c15e0c0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 --2- 172.21.15.129:0/1491932145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f179c06eed0 0x7f179c08c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 >> 172.21.15.129:0/1491932145 conn(0x7f179c06d410 msgr2=0x7f179c0ad330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 shutdown_connections 2024-09-17T03:55:11.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.034+0000 7f17a2a00640 1 -- 172.21.15.129:0/1491932145 wait complete. 2024-09-17T03:55:11.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:55:11.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 -- 172.21.15.129:0/2900012695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880ce500 msgr2=0x7f0b880ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:11.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/2900012695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880ce500 0x7f0b880ce910 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f0b880d1290 tx=0x7f0b7002a560 comp rx=0 tx=0).stop 2024-09-17T03:55:11.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 -- 172.21.15.129:0/2900012695 shutdown_connections 2024-09-17T03:55:11.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/2900012695 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880b4220 0x7f0b880b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/2900012695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880ce500 0x7f0b880ce910 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.292+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/2900012695 >> v2:172.21.15.181:3300/0 conn(0x7f0b880b2650 0x7f0b880cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.293+0000 7f0b8dc00640 1 -- 172.21.15.129:0/2900012695 >> 172.21.15.129:0/2900012695 conn(0x7f0b880ccc40 msgr2=0x7f0b880cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:11.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.293+0000 7f0b8dc00640 1 -- 172.21.15.129:0/2900012695 shutdown_connections 2024-09-17T03:55:11.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.293+0000 7f0b8dc00640 1 -- 172.21.15.129:0/2900012695 wait complete. 2024-09-17T03:55:11.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.295+0000 7f0b8dc00640 1 Processor -- start 2024-09-17T03:55:11.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.295+0000 7f0b8dc00640 1 -- start start 2024-09-17T03:55:11.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b880b2650 0x7f0b88154540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:11.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:11.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880ce500 0x7f0b88151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:11.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880ca9f0 con 0x7f0b880b4220 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880cab70 con 0x7f0b880b2650 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.296+0000 7f0b8dc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0b880cacf0 con 0x7f0b880ce500 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55584/0 (socket says 172.21.15.129:55584) 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 -- 172.21.15.129:0/3256934623 learned_addr learned my addr 172.21.15.129:0/3256934623 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b87400640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880ce500 0x7f0b88151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b86a00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b880b2650 0x7f0b88154540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 -- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880ce500 msgr2=0x7f0b88151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880ce500 0x7f0b88151190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 -- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b880b2650 msgr2=0x7f0b88154540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b880b2650 0x7f0b88154540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:11.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.297+0000 7f0b7e000640 1 -- 172.21.15.129:0/3256934623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b88154a80 con 0x7f0b880b4220 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b7e000640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0b70034040 tx=0x7f0b70003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7003e030 con 0x7f0b880b4220 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b70003e00 con 0x7f0b880b4220 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8809af60 con 0x7f0b880b4220 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b70035040 con 0x7f0b880b4220 2024-09-17T03:55:11.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.298+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b8809b520 con 0x7f0b880b4220 2024-09-17T03:55:11.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.299+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b880cb1a0 con 0x7f0b880b4220 2024-09-17T03:55:11.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.300+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0b70041030 con 0x7f0b880b4220 2024-09-17T03:55:11.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.301+0000 7f0b7f400640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 0x7f0b6407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:11.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.301+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f0b700bb500 con 0x7f0b880b4220 2024-09-17T03:55:11.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.302+0000 7f0b86a00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 0x7f0b6407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:11.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.302+0000 7f0b86a00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 0x7f0b6407b0d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0b880726b0 tx=0x7f0b7800b260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:11.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.305+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0b88154a80 con 0x7f0b880b4220 2024-09-17T03:55:11.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:11.532+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0b880b2430 con 0x7f0b880b4220 2024-09-17T03:55:12.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.283+0000 7f0b7f400640 1 -- 172.21.15.129:0/3256934623 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v79) ==== 118+0+0 (secure 0 0 0) 0x7f0b700832e0 con 0x7f0b880b4220 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 msgr2=0x7f0b6407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 0x7f0b6407b0d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0b880726b0 tx=0x7f0b7800b260 comp rx=0 tx=0).stop 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 msgr2=0x7f0b88151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f0b70034040 tx=0x7f0b70003910 comp rx=0 tx=0).stop 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 shutdown_connections 2024-09-17T03:55:12.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b64078bc0 0x7f0b6407b0d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b880ce500 0x7f0b88151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b880b4220 0x7f0b88151aa0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 --2- 172.21.15.129:0/3256934623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b880b2650 0x7f0b88154540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 >> 172.21.15.129:0/3256934623 conn(0x7f0b880ccc40 msgr2=0x7f0b880b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 shutdown_connections 2024-09-17T03:55:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.288+0000 7f0b8dc00640 1 -- 172.21.15.129:0/3256934623 wait complete. 2024-09-17T03:55:12.317 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:55:12.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 -- 172.21.15.129:0/431204730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 msgr2=0x7fd120162000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 --2- 172.21.15.129:0/431204730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120162000 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fd1200d56a0 tx=0x7fd10c02a4d0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 -- 172.21.15.129:0/431204730 shutdown_connections 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 --2- 172.21.15.129:0/431204730 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200d0f00 0x7fd1200d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 --2- 172.21.15.129:0/431204730 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120162000 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 --2- 172.21.15.129:0/431204730 >> v2:172.21.15.181:3300/0 conn(0x7fd1200b3480 0x7fd1200d0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 -- 172.21.15.129:0/431204730 >> 172.21.15.129:0/431204730 conn(0x7fd12008bff0 msgr2=0x7fd12009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 -- 172.21.15.129:0/431204730 shutdown_connections 2024-09-17T03:55:12.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.546+0000 7fd124a00640 1 -- 172.21.15.129:0/431204730 wait complete. 2024-09-17T03:55:12.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.549+0000 7fd124a00640 1 Processor -- start 2024-09-17T03:55:12.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.549+0000 7fd124a00640 1 -- start start 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200b3480 0x7fd12014e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1200d0f00 0x7fd120150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200b42b0 con 0x7fd1200d2f60 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd1200b4130 con 0x7fd1200d0f00 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd124a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd1200b4430 con 0x7fd1200b3480 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55604/0 (socket says 172.21.15.129:55604) 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 -- 172.21.15.129:0/928704361 learned_addr learned my addr 172.21.15.129:0/928704361 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11f400640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200b3480 0x7fd12014e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11ea00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1200d0f00 0x7fd120150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 -- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200b3480 msgr2=0x7fd12014e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200b3480 0x7fd12014e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 -- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1200d0f00 msgr2=0x7fd120150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1200d0f00 0x7fd120150d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.551+0000 7fd11fe00640 1 -- 172.21.15.129:0/928704361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1200b7620 con 0x7fd1200d2f60 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd11fe00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd11800ea40 tx=0x7fd11800ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11800ce40 con 0x7fd1200d2f60 2024-09-17T03:55:12.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd118004c10 con 0x7fd1200d2f60 2024-09-17T03:55:12.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd118005020 con 0x7fd1200d2f60 2024-09-17T03:55:12.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1201f9b80 con 0x7fd1200d2f60 2024-09-17T03:55:12.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.552+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1201f9e40 con 0x7fd1200d2f60 2024-09-17T03:55:12.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.554+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1200b49c0 con 0x7fd1200d2f60 2024-09-17T03:55:12.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.555+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd1180051c0 con 0x7fd1200d2f60 2024-09-17T03:55:12.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.555+0000 7fd117400640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 0x7fd10007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:12.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.556+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fd11809ba20 con 0x7fd1200d2f60 2024-09-17T03:55:12.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.556+0000 7fd11f400640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 0x7fd10007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:12.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.557+0000 7fd11f400640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 0x7fd10007b0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd110005fd0 tx=0x7fd11000b260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:12.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.560+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd118063810 con 0x7fd1200d2f60 2024-09-17T03:55:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:12.782+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd1201fa080 con 0x7fd1200d2f60 2024-09-17T03:55:13.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.291+0000 7fd117400640 1 -- 172.21.15.129:0/928704361 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v80) ==== 122+0+0 (secure 0 0 0) 0x7fd118068240 con 0x7fd1200d2f60 2024-09-17T03:55:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 msgr2=0x7fd10007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 0x7fd10007b0a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd110005fd0 tx=0x7fd11000b260 comp rx=0 tx=0).stop 2024-09-17T03:55:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 msgr2=0x7fd120151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fd11800ea40 tx=0x7fd11800ef00 comp rx=0 tx=0).stop 2024-09-17T03:55:13.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 shutdown_connections 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd100078b90 0x7fd10007b0a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1200d2f60 0x7fd120151290 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1200d0f00 0x7fd120150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 --2- 172.21.15.129:0/928704361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1200b3480 0x7fd12014e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.294+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 >> 172.21.15.129:0/928704361 conn(0x7fd12008bff0 msgr2=0x7fd1200b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:13.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.295+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 shutdown_connections 2024-09-17T03:55:13.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.295+0000 7fd124a00640 1 -- 172.21.15.129:0/928704361 wait complete. 2024-09-17T03:55:13.321 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:55:13.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 -- 172.21.15.129:0/1303913705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 msgr2=0x7f3d780ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/1303913705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780ccd20 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f3d70005fd0 tx=0x7f3d7002aa90 comp rx=0 tx=0).stop 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 -- 172.21.15.129:0/1303913705 shutdown_connections 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/1303913705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780ccd20 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/1303913705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d780cc3e0 0x7f3d7815f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 -- 172.21.15.129:0/1303913705 >> 172.21.15.129:0/1303913705 conn(0x7f3d7806d410 msgr2=0x7f3d780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 -- 172.21.15.129:0/1303913705 shutdown_connections 2024-09-17T03:55:13.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.545+0000 7f3d7ec00640 1 -- 172.21.15.129:0/1303913705 wait complete. 2024-09-17T03:55:13.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.548+0000 7f3d7ec00640 1 Processor -- start 2024-09-17T03:55:13.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.548+0000 7f3d7ec00640 1 -- start start 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d78070a50 0x7f3d78159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d780cc3e0 0x7f3d780991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d780af070 con 0x7f3d780cc910 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7815f7b0 con 0x7f3d780cc3e0 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.549+0000 7f3d7ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d780af1f0 con 0x7f3d78070a50 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55620/0 (socket says 172.21.15.129:55620) 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 -- 172.21.15.129:0/4247668663 learned_addr learned my addr 172.21.15.129:0/4247668663 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d77e00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d78070a50 0x7f3d78159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d77400640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d780cc3e0 0x7f3d780991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 -- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d78070a50 msgr2=0x7f3d78159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d78070a50 0x7f3d78159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 -- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d780cc3e0 msgr2=0x7f3d780991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d780cc3e0 0x7f3d780991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 -- 172.21.15.129:0/4247668663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d78159fb0 con 0x7f3d780cc910 2024-09-17T03:55:13.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d7cc00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009d90 tx=0x7f3d70003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.550+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70044070 con 0x7f3d780cc910 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.551+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d70004040 con 0x7f3d780cc910 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.551+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70004430 con 0x7f3d780cc910 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.551+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d78099730 con 0x7f3d780cc910 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.551+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7815a7a0 con 0x7f3d780cc910 2024-09-17T03:55:13.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.552+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d780af6e0 con 0x7f3d780cc910 2024-09-17T03:55:13.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.553+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d70047070 con 0x7f3d780cc910 2024-09-17T03:55:13.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.554+0000 7f3d6fe00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 0x7f3d5807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:13.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.554+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f3d700bbbd0 con 0x7f3d780cc910 2024-09-17T03:55:13.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.555+0000 7f3d77e00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 0x7f3d5807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:13.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.555+0000 7f3d77e00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 0x7f3d5807b0a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3d68001720 tx=0x7f3d68009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:13.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.558+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d70032030 con 0x7f3d780cc910 2024-09-17T03:55:13.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.775+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d7815a250 con 0x7f3d780cc910 2024-09-17T03:55:13.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.776+0000 7f3d6fe00640 1 -- 172.21.15.129:0/4247668663 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5228 (secure 0 0 0) 0x7f3d70083920 con 0x7f3d780cc910 2024-09-17T03:55:13.775 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:13.775 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"btime":"2024-09-17T03:55:13:284539+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/4006656424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4006656424},{"type":"v1","addr":"172.21.15.129:6835","nonce":4006656424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/473985729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":473985729},{"type":"v1","addr":"172.21.15.181:6836","nonce":473985729}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/826515162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":826515162},{"type":"v1","addr":"172.21.15.181:6837","nonce":826515162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:12.783578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7005":{"gid":7005,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/153768962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":153768962},{"type":"v1","addr":"172.21.15.129:6834","nonce":153768962}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7005,"qdb_cluster":[7005]},"id":8}]} 2024-09-17T03:55:13.775 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 80 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 msgr2=0x7f3d5807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 0x7f3d5807b0a0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3d68001720 tx=0x7f3d68009040 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 msgr2=0x7f3d780afc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009d90 tx=0x7f3d70003ce0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 shutdown_connections 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d58078b90 0x7f3d5807b0a0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d780cc910 0x7f3d780afc50 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d780cc3e0 0x7f3d780991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 --2- 172.21.15.129:0/4247668663 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d78070a50 0x7f3d78159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 >> 172.21.15.129:0/4247668663 conn(0x7f3d7806d410 msgr2=0x7f3d780ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:13.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.778+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 shutdown_connections 2024-09-17T03:55:13.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:13.779+0000 7f3d7ec00640 1 -- 172.21.15.129:0/4247668663 wait complete. 2024-09-17T03:55:13.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 -- 172.21.15.129:0/1915617362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440cfb60 msgr2=0x7f2344160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 --2- 172.21.15.129:0/1915617362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440cfb60 0x7f2344160cf0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2334009830 tx=0x7f233402a480 comp rx=0 tx=0).stop 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 -- 172.21.15.129:0/1915617362 shutdown_connections 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 --2- 172.21.15.129:0/1915617362 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440d09d0 0x7f23440b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 --2- 172.21.15.129:0/1915617362 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440cfb60 0x7f2344160cf0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 -- 172.21.15.129:0/1915617362 >> 172.21.15.129:0/1915617362 conn(0x7f234406cfc0 msgr2=0x7f234406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 -- 172.21.15.129:0/1915617362 shutdown_connections 2024-09-17T03:55:14.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.025+0000 7f234a000640 1 -- 172.21.15.129:0/1915617362 wait complete. 2024-09-17T03:55:14.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.028+0000 7f234a000640 1 Processor -- start 2024-09-17T03:55:14.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.028+0000 7f234a000640 1 -- start start 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440cfb60 0x7f2344072d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2344073270 0x7f234415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f234415d270 con 0x7f23440b46e0 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f234415d0f0 con 0x7f2344073270 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f234a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f234415d3f0 con 0x7f23440cfb60 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f2348c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f2348c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53088/0 (socket says 172.21.15.129:53088) 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.029+0000 7f2348c00640 1 -- 172.21.15.129:0/1561590733 learned_addr learned my addr 172.21.15.129:0/1561590733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f233be00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440cfb60 0x7f2344072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2349600640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2344073270 0x7f234415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 -- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440cfb60 msgr2=0x7f2344072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440cfb60 0x7f2344072d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 -- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2344073270 msgr2=0x7f234415b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2344073270 0x7f234415b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 -- 172.21.15.129:0/1561590733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23440c53f0 con 0x7f23440b46e0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2348c00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f233402aa70 tx=0x7f2334033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.030+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334005c90 con 0x7f23440b46e0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.031+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2334031e50 con 0x7f23440b46e0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.031+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2334005260 con 0x7f23440b46e0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.031+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23440c56d0 con 0x7f23440b46e0 2024-09-17T03:55:14.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.031+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23440c5c90 con 0x7f23440b46e0 2024-09-17T03:55:14.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.032+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f234415d8a0 con 0x7f23440b46e0 2024-09-17T03:55:14.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.033+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2334005400 con 0x7f23440b46e0 2024-09-17T03:55:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.034+0000 7f2341600640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 0x7f232007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.034+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f23340bbde0 con 0x7f23440b46e0 2024-09-17T03:55:14.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.035+0000 7f233be00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 0x7f232007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.035+0000 7f233be00640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 0x7f232007b0d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f232c007fd0 tx=0x7f232c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.039+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f233403f030 con 0x7f23440b46e0 2024-09-17T03:55:14.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.222+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f234406a140 con 0x7f23440b46e0 2024-09-17T03:55:14.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.224+0000 7f2341600640 1 -- 172.21.15.129:0/1561590733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) ==== 74+0+16735 (secure 0 0 0) 0x7f2334084220 con 0x7f23440b46e0 2024-09-17T03:55:14.223 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:14.223 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:55:11.091744+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:55:07.469103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-17T03:55:08.543736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:55:14.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 msgr2=0x7f232007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 0x7f232007b0d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f232c007fd0 tx=0x7f232c009040 comp rx=0 tx=0).stop 2024-09-17T03:55:14.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 msgr2=0x7f23440c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f233402aa70 tx=0x7f2334033c20 comp rx=0 tx=0).stop 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 shutdown_connections 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2320078bc0 0x7f232007b0d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2344073270 0x7f234415b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23440cfb60 0x7f2344072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 --2- 172.21.15.129:0/1561590733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23440b46e0 0x7f23440c6490 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 >> 172.21.15.129:0/1561590733 conn(0x7f234406cfc0 msgr2=0x7f23441534f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 shutdown_connections 2024-09-17T03:55:14.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.226+0000 7f234a000640 1 -- 172.21.15.129:0/1561590733 wait complete. 2024-09-17T03:55:14.247 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:55:14.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.469+0000 7fb081400640 1 -- 172.21.15.129:0/1548125886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb060003a00 msgr2=0x7fb060003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.469+0000 7fb081400640 1 --2- 172.21.15.129:0/1548125886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb060003a00 0x7fb060003e30 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009b50 tx=0x7fb060005830 comp rx=0 tx=0).stop 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.469+0000 7fb081400640 1 -- 172.21.15.129:0/1548125886 shutdown_connections 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.469+0000 7fb081400640 1 --2- 172.21.15.129:0/1548125886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb060003a00 0x7fb060003e30 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.469+0000 7fb081400640 1 -- 172.21.15.129:0/1548125886 >> 172.21.15.129:0/1548125886 conn(0x7fb07c06d410 msgr2=0x7fb07c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.470+0000 7fb081400640 1 -- 172.21.15.129:0/1548125886 shutdown_connections 2024-09-17T03:55:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.470+0000 7fb081400640 1 -- 172.21.15.129:0/1548125886 wait complete. 2024-09-17T03:55:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.472+0000 7fb081400640 1 Processor -- start 2024-09-17T03:55:14.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.472+0000 7fb081400640 1 -- start start 2024-09-17T03:55:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb060003a00 0x7fb07c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb07c156fe0 0x7fb07c073230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c0b4140 con 0x7fb07c157e50 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c06f4f0 con 0x7fb07c156fe0 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.473+0000 7fb081400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb07c0b42c0 con 0x7fb060003a00 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53116/0 (socket says 172.21.15.129:53116) 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 -- 172.21.15.129:0/3436540837 learned_addr learned my addr 172.21.15.129:0/3436540837 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 -- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb060003a00 msgr2=0x7fb07c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb060003a00 0x7fb07c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 -- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb07c156fe0 msgr2=0x7fb07c073230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb07c156fe0 0x7fb07c073230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 -- 172.21.15.129:0/3436540837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c15ca50 con 0x7fb07c157e50 2024-09-17T03:55:14.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb079e00640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb068006600 tx=0x7fb0680039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068015070 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb068011900 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068011d50 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.474+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c15cd30 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.475+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07c158290 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.476+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07c0ccc80 con 0x7fb07c157e50 2024-09-17T03:55:14.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.477+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb068018070 con 0x7fb07c157e50 2024-09-17T03:55:14.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.478+0000 7fb072a00640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 0x7fb05c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.478+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fb06809bc10 con 0x7fb07c157e50 2024-09-17T03:55:14.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.479+0000 7fb07a800640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 0x7fb05c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.480+0000 7fb07a800640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 0x7fb05c07b0a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb06c035000 tx=0x7fb06c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.483+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb068063fb0 con 0x7fb07c157e50 2024-09-17T03:55:14.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.667+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb07c15c7e0 con 0x7fb05c078b90 2024-09-17T03:55:14.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.673+0000 7fb072a00640 1 -- 172.21.15.129:0/3436540837 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250957 (secure 0 0 0) 0x7fb06c07f460 con 0x7fb05c078b90 2024-09-17T03:55:14.675 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:14.677 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:55:14.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 msgr2=0x7fb05c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 0x7fb05c07b0a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb06c035000 tx=0x7fb06c004730 comp rx=0 tx=0).stop 2024-09-17T03:55:14.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 msgr2=0x7fb07c15aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fb068006600 tx=0x7fb0680039c0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 shutdown_connections 2024-09-17T03:55:14.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb05c078b90 0x7fb05c07b0a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb07c156fe0 0x7fb07c073230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb07c157e50 0x7fb07c15aa00 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 --2- 172.21.15.129:0/3436540837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb060003a00 0x7fb07c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 >> 172.21.15.129:0/3436540837 conn(0x7fb07c06d410 msgr2=0x7fb07c0ae220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 shutdown_connections 2024-09-17T03:55:14.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.677+0000 7fb081400640 1 -- 172.21.15.129:0/3436540837 wait complete. 2024-09-17T03:55:14.698 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-17T03:55:14.387295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":118,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302584,"kb_used_data":11428,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672136,"statfs":{"total":773094113280,"available":772784267264,"internally_reserved":0,"allocated":11702272,"data_stored":5658868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001744"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424841+0000","last_change":"2024-09-17T03:55:10.112479+0000","last_active":"2024-09-17T03:55:11.424841+0000","last_peered":"2024-09-17T03:55:11.424841+0000","last_clean":"2024-09-17T03:55:11.424841+0000","last_became_active":"2024-09-17T03:55:10.112184+0000","last_became_peered":"2024-09-17T03:55:10.112184+0000","last_unstale":"2024-09-17T03:55:11.424841+0000","last_undegraded":"2024-09-17T03:55:11.424841+0000","last_fullsized":"2024-09-17T03:55:11.424841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:12.328355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425890+0000","last_change":"2024-09-17T03:55:10.123473+0000","last_active":"2024-09-17T03:55:11.425890+0000","last_peered":"2024-09-17T03:55:11.425890+0000","last_clean":"2024-09-17T03:55:11.425890+0000","last_became_active":"2024-09-17T03:55:10.123217+0000","last_became_peered":"2024-09-17T03:55:10.123217+0000","last_unstale":"2024-09-17T03:55:11.425890+0000","last_undegraded":"2024-09-17T03:55:11.425890+0000","last_fullsized":"2024-09-17T03:55:11.425890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:50.947437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425736+0000","last_change":"2024-09-17T03:55:10.103667+0000","last_active":"2024-09-17T03:55:11.425736+0000","last_peered":"2024-09-17T03:55:11.425736+0000","last_clean":"2024-09-17T03:55:11.425736+0000","last_became_active":"2024-09-17T03:55:10.103336+0000","last_became_peered":"2024-09-17T03:55:10.103336+0000","last_unstale":"2024-09-17T03:55:11.425736+0000","last_undegraded":"2024-09-17T03:55:11.425736+0000","last_fullsized":"2024-09-17T03:55:11.425736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:33.517751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424369+0000","last_change":"2024-09-17T03:55:10.109342+0000","last_active":"2024-09-17T03:55:11.424369+0000","last_peered":"2024-09-17T03:55:11.424369+0000","last_clean":"2024-09-17T03:55:11.424369+0000","last_became_active":"2024-09-17T03:55:10.109013+0000","last_became_peered":"2024-09-17T03:55:10.109013+0000","last_unstale":"2024-09-17T03:55:11.424369+0000","last_undegraded":"2024-09-17T03:55:11.424369+0000","last_fullsized":"2024-09-17T03:55:11.424369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:01.880305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.093607+0000","last_change":"2024-09-17T03:55:09.093607+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.093607+0000","last_undegraded":"2024-09-17T03:55:09.093607+0000","last_fullsized":"2024-09-17T03:55:09.093607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.090308+0000","last_change":"2024-09-17T03:55:09.090308+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.090308+0000","last_undegraded":"2024-09-17T03:55:09.090308+0000","last_fullsized":"2024-09-17T03:55:09.090308+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104179+0000","last_change":"2024-09-17T03:55:10.096583+0000","last_active":"2024-09-17T03:55:11.104179+0000","last_peered":"2024-09-17T03:55:11.104179+0000","last_clean":"2024-09-17T03:55:11.104179+0000","last_became_active":"2024-09-17T03:55:10.096289+0000","last_became_peered":"2024-09-17T03:55:10.096289+0000","last_unstale":"2024-09-17T03:55:11.104179+0000","last_undegraded":"2024-09-17T03:55:11.104179+0000","last_fullsized":"2024-09-17T03:55:11.104179+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424746+0000","last_change":"2024-09-17T03:55:10.116515+0000","last_active":"2024-09-17T03:55:11.424746+0000","last_peered":"2024-09-17T03:55:11.424746+0000","last_clean":"2024-09-17T03:55:11.424746+0000","last_became_active":"2024-09-17T03:55:10.116161+0000","last_became_peered":"2024-09-17T03:55:10.116161+0000","last_unstale":"2024-09-17T03:55:11.424746+0000","last_undegraded":"2024-09-17T03:55:11.424746+0000","last_fullsized":"2024-09-17T03:55:11.424746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.433598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424969+0000","last_change":"2024-09-17T03:55:10.113065+0000","last_active":"2024-09-17T03:55:11.424969+0000","last_peered":"2024-09-17T03:55:11.424969+0000","last_clean":"2024-09-17T03:55:11.424969+0000","last_became_active":"2024-09-17T03:55:10.111008+0000","last_became_peered":"2024-09-17T03:55:10.111008+0000","last_unstale":"2024-09-17T03:55:11.424969+0000","last_undegraded":"2024-09-17T03:55:11.424969+0000","last_fullsized":"2024-09-17T03:55:11.424969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:54.618141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424625+0000","last_change":"2024-09-17T03:55:10.105708+0000","last_active":"2024-09-17T03:55:11.424625+0000","last_peered":"2024-09-17T03:55:11.424625+0000","last_clean":"2024-09-17T03:55:11.424625+0000","last_became_active":"2024-09-17T03:55:10.105368+0000","last_became_peered":"2024-09-17T03:55:10.105368+0000","last_unstale":"2024-09-17T03:55:11.424625+0000","last_undegraded":"2024-09-17T03:55:11.424625+0000","last_fullsized":"2024-09-17T03:55:11.424625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:19.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425340+0000","last_change":"2024-09-17T03:55:10.107346+0000","last_active":"2024-09-17T03:55:11.425340+0000","last_peered":"2024-09-17T03:55:11.425340+0000","last_clean":"2024-09-17T03:55:11.425340+0000","last_became_active":"2024-09-17T03:55:10.106955+0000","last_became_peered":"2024-09-17T03:55:10.106955+0000","last_unstale":"2024-09-17T03:55:11.425340+0000","last_undegraded":"2024-09-17T03:55:11.425340+0000","last_fullsized":"2024-09-17T03:55:11.425340+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:29.542513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424410+0000","last_change":"2024-09-17T03:55:10.108582+0000","last_active":"2024-09-17T03:55:11.424410+0000","last_peered":"2024-09-17T03:55:11.424410+0000","last_clean":"2024-09-17T03:55:11.424410+0000","last_became_active":"2024-09-17T03:55:10.105982+0000","last_became_peered":"2024-09-17T03:55:10.105982+0000","last_unstale":"2024-09-17T03:55:11.424410+0000","last_undegraded":"2024-09-17T03:55:11.424410+0000","last_fullsized":"2024-09-17T03:55:11.424410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:31.031639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425062+0000","last_change":"2024-09-17T03:55:10.109256+0000","last_active":"2024-09-17T03:55:11.425062+0000","last_peered":"2024-09-17T03:55:11.425062+0000","last_clean":"2024-09-17T03:55:11.425062+0000","last_became_active":"2024-09-17T03:55:10.108832+0000","last_became_peered":"2024-09-17T03:55:10.108832+0000","last_unstale":"2024-09-17T03:55:11.425062+0000","last_undegraded":"2024-09-17T03:55:11.425062+0000","last_fullsized":"2024-09-17T03:55:11.425062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:16.522311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424801+0000","last_change":"2024-09-17T03:55:10.112919+0000","last_active":"2024-09-17T03:55:11.424801+0000","last_peered":"2024-09-17T03:55:11.424801+0000","last_clean":"2024-09-17T03:55:11.424801+0000","last_became_active":"2024-09-17T03:55:10.112583+0000","last_became_peered":"2024-09-17T03:55:10.112583+0000","last_unstale":"2024-09-17T03:55:11.424801+0000","last_undegraded":"2024-09-17T03:55:11.424801+0000","last_fullsized":"2024-09-17T03:55:11.424801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:27.443267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087537+0000","last_change":"2024-09-17T03:55:09.087537+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087537+0000","last_undegraded":"2024-09-17T03:55:09.087537+0000","last_fullsized":"2024-09-17T03:55:09.087537+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423770+0000","last_change":"2024-09-17T03:55:10.094106+0000","last_active":"2024-09-17T03:55:11.423770+0000","last_peered":"2024-09-17T03:55:11.423770+0000","last_clean":"2024-09-17T03:55:11.423770+0000","last_became_active":"2024-09-17T03:55:10.093763+0000","last_became_peered":"2024-09-17T03:55:10.093763+0000","last_unstale":"2024-09-17T03:55:11.423770+0000","last_undegraded":"2024-09-17T03:55:11.423770+0000","last_fullsized":"2024-09-17T03:55:11.423770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.835238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426449+0000","last_change":"2024-09-17T03:55:10.101103+0000","last_active":"2024-09-17T03:55:11.426449+0000","last_peered":"2024-09-17T03:55:11.426449+0000","last_clean":"2024-09-17T03:55:11.426449+0000","last_became_active":"2024-09-17T03:55:10.100775+0000","last_became_peered":"2024-09-17T03:55:10.100775+0000","last_unstale":"2024-09-17T03:55:11.426449+0000","last_undegraded":"2024-09-17T03:55:11.426449+0000","last_fullsized":"2024-09-17T03:55:11.426449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426439+0000","last_change":"2024-09-17T03:55:10.100108+0000","last_active":"2024-09-17T03:55:11.426439+0000","last_peered":"2024-09-17T03:55:11.426439+0000","last_clean":"2024-09-17T03:55:11.426439+0000","last_became_active":"2024-09-17T03:55:10.099786+0000","last_became_peered":"2024-09-17T03:55:10.099786+0000","last_unstale":"2024-09-17T03:55:11.426439+0000","last_undegraded":"2024-09-17T03:55:11.426439+0000","last_fullsized":"2024-09-17T03:55:11.426439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:00.641678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425612+0000","last_change":"2024-09-17T03:55:10.109163+0000","last_active":"2024-09-17T03:55:11.425612+0000","last_peered":"2024-09-17T03:55:11.425612+0000","last_clean":"2024-09-17T03:55:11.425612+0000","last_became_active":"2024-09-17T03:55:10.108810+0000","last_became_peered":"2024-09-17T03:55:10.108810+0000","last_unstale":"2024-09-17T03:55:11.425612+0000","last_undegraded":"2024-09-17T03:55:11.425612+0000","last_fullsized":"2024-09-17T03:55:11.425612+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:32.706728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426258+0000","last_change":"2024-09-17T03:55:10.102425+0000","last_active":"2024-09-17T03:55:11.426258+0000","last_peered":"2024-09-17T03:55:11.426258+0000","last_clean":"2024-09-17T03:55:11.426258+0000","last_became_active":"2024-09-17T03:55:10.102086+0000","last_became_peered":"2024-09-17T03:55:10.102086+0000","last_unstale":"2024-09-17T03:55:11.426258+0000","last_undegraded":"2024-09-17T03:55:11.426258+0000","last_fullsized":"2024-09-17T03:55:11.426258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426586+0000","last_change":"2024-09-17T03:55:10.100194+0000","last_active":"2024-09-17T03:55:11.426586+0000","last_peered":"2024-09-17T03:55:11.426586+0000","last_clean":"2024-09-17T03:55:11.426586+0000","last_became_active":"2024-09-17T03:55:10.099857+0000","last_became_peered":"2024-09-17T03:55:10.099857+0000","last_unstale":"2024-09-17T03:55:11.426586+0000","last_undegraded":"2024-09-17T03:55:11.426586+0000","last_fullsized":"2024-09-17T03:55:11.426586+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085847+0000","last_change":"2024-09-17T03:55:09.085847+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085847+0000","last_undegraded":"2024-09-17T03:55:09.085847+0000","last_fullsized":"2024-09-17T03:55:09.085847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425299+0000","last_change":"2024-09-17T03:55:10.097618+0000","last_active":"2024-09-17T03:55:11.425299+0000","last_peered":"2024-09-17T03:55:11.425299+0000","last_clean":"2024-09-17T03:55:11.425299+0000","last_became_active":"2024-09-17T03:55:10.097328+0000","last_became_peered":"2024-09-17T03:55:10.097328+0000","last_unstale":"2024-09-17T03:55:11.425299+0000","last_undegraded":"2024-09-17T03:55:11.425299+0000","last_fullsized":"2024-09-17T03:55:11.425299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:23.303493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085796+0000","last_change":"2024-09-17T03:55:09.085796+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085796+0000","last_undegraded":"2024-09-17T03:55:09.085796+0000","last_fullsized":"2024-09-17T03:55:09.085796+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426752+0000","last_change":"2024-09-17T03:55:10.098678+0000","last_active":"2024-09-17T03:55:11.426752+0000","last_peered":"2024-09-17T03:55:11.426752+0000","last_clean":"2024-09-17T03:55:11.426752+0000","last_became_active":"2024-09-17T03:55:10.098318+0000","last_became_peered":"2024-09-17T03:55:10.098318+0000","last_unstale":"2024-09-17T03:55:11.426752+0000","last_undegraded":"2024-09-17T03:55:11.426752+0000","last_fullsized":"2024-09-17T03:55:11.426752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424684+0000","last_change":"2024-09-17T03:55:10.102840+0000","last_active":"2024-09-17T03:55:11.424684+0000","last_peered":"2024-09-17T03:55:11.424684+0000","last_clean":"2024-09-17T03:55:11.424684+0000","last_became_active":"2024-09-17T03:55:10.100850+0000","last_became_peered":"2024-09-17T03:55:10.100850+0000","last_unstale":"2024-09-17T03:55:11.424684+0000","last_undegraded":"2024-09-17T03:55:11.424684+0000","last_fullsized":"2024-09-17T03:55:11.424684+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:17.813222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087802+0000","last_change":"2024-09-17T03:55:09.087802+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087802+0000","last_undegraded":"2024-09-17T03:55:09.087802+0000","last_fullsized":"2024-09-17T03:55:09.087802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089564+0000","last_change":"2024-09-17T03:55:09.089564+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089564+0000","last_undegraded":"2024-09-17T03:55:09.089564+0000","last_fullsized":"2024-09-17T03:55:09.089564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425371+0000","last_change":"2024-09-17T03:55:10.103912+0000","last_active":"2024-09-17T03:55:11.425371+0000","last_peered":"2024-09-17T03:55:11.425371+0000","last_clean":"2024-09-17T03:55:11.425371+0000","last_became_active":"2024-09-17T03:55:10.103569+0000","last_became_peered":"2024-09-17T03:55:10.103569+0000","last_unstale":"2024-09-17T03:55:11.425371+0000","last_undegraded":"2024-09-17T03:55:11.425371+0000","last_fullsized":"2024-09-17T03:55:11.425371+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:36.412867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104245+0000","last_change":"2024-09-17T03:55:10.111756+0000","last_active":"2024-09-17T03:55:11.104245+0000","last_peered":"2024-09-17T03:55:11.104245+0000","last_clean":"2024-09-17T03:55:11.104245+0000","last_became_active":"2024-09-17T03:55:10.111350+0000","last_became_peered":"2024-09-17T03:55:10.111350+0000","last_unstale":"2024-09-17T03:55:11.104245+0000","last_undegraded":"2024-09-17T03:55:11.104245+0000","last_fullsized":"2024-09-17T03:55:11.104245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423827+0000","last_change":"2024-09-17T03:55:10.110031+0000","last_active":"2024-09-17T03:55:11.423827+0000","last_peered":"2024-09-17T03:55:11.423827+0000","last_clean":"2024-09-17T03:55:11.423827+0000","last_became_active":"2024-09-17T03:55:10.106150+0000","last_became_peered":"2024-09-17T03:55:10.106150+0000","last_unstale":"2024-09-17T03:55:11.423827+0000","last_undegraded":"2024-09-17T03:55:11.423827+0000","last_fullsized":"2024-09-17T03:55:11.423827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:15.416205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085837+0000","last_change":"2024-09-17T03:55:09.085837+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085837+0000","last_undegraded":"2024-09-17T03:55:09.085837+0000","last_fullsized":"2024-09-17T03:55:09.085837+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":19,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104474+0000","last_change":"2024-09-17T03:55:09.098888+0000","last_active":"2024-09-17T03:55:11.104474+0000","last_peered":"2024-09-17T03:55:11.104474+0000","last_clean":"2024-09-17T03:55:11.104474+0000","last_became_active":"2024-09-17T03:55:09.098488+0000","last_became_peered":"2024-09-17T03:55:09.098488+0000","last_unstale":"2024-09-17T03:55:11.104474+0000","last_undegraded":"2024-09-17T03:55:11.104474+0000","last_fullsized":"2024-09-17T03:55:11.104474+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:43.529970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426303+0000","last_change":"2024-09-17T03:55:10.098038+0000","last_active":"2024-09-17T03:55:11.426303+0000","last_peered":"2024-09-17T03:55:11.426303+0000","last_clean":"2024-09-17T03:55:11.426303+0000","last_became_active":"2024-09-17T03:55:10.097706+0000","last_became_peered":"2024-09-17T03:55:10.097706+0000","last_unstale":"2024-09-17T03:55:11.426303+0000","last_undegraded":"2024-09-17T03:55:11.426303+0000","last_fullsized":"2024-09-17T03:55:11.426303+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:48.538101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424685+0000","last_change":"2024-09-17T03:55:09.102970+0000","last_active":"2024-09-17T03:55:11.424685+0000","last_peered":"2024-09-17T03:55:11.424685+0000","last_clean":"2024-09-17T03:55:11.424685+0000","last_became_active":"2024-09-17T03:55:09.102558+0000","last_became_peered":"2024-09-17T03:55:09.102558+0000","last_unstale":"2024-09-17T03:55:11.424685+0000","last_undegraded":"2024-09-17T03:55:11.424685+0000","last_fullsized":"2024-09-17T03:55:11.424685+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:26.840025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084952+0000","last_change":"2024-09-17T03:55:09.084952+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084952+0000","last_undegraded":"2024-09-17T03:55:09.084952+0000","last_fullsized":"2024-09-17T03:55:09.084952+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109001+0000","last_change":"2024-09-17T03:55:09.106007+0000","last_active":"2024-09-17T03:55:09.109001+0000","last_peered":"2024-09-17T03:55:09.109001+0000","last_clean":"2024-09-17T03:55:09.109001+0000","last_became_active":"2024-09-17T03:55:09.105510+0000","last_became_peered":"2024-09-17T03:55:09.105510+0000","last_unstale":"2024-09-17T03:55:09.109001+0000","last_undegraded":"2024-09-17T03:55:09.109001+0000","last_fullsized":"2024-09-17T03:55:09.109001+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:19.653445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425907+0000","last_change":"2024-09-17T03:55:10.111580+0000","last_active":"2024-09-17T03:55:11.425907+0000","last_peered":"2024-09-17T03:55:11.425907+0000","last_clean":"2024-09-17T03:55:11.425907+0000","last_became_active":"2024-09-17T03:55:10.111121+0000","last_became_peered":"2024-09-17T03:55:10.111121+0000","last_unstale":"2024-09-17T03:55:11.425907+0000","last_undegraded":"2024-09-17T03:55:11.425907+0000","last_fullsized":"2024-09-17T03:55:11.425907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112556+0000","last_change":"2024-09-17T03:55:09.109395+0000","last_active":"2024-09-17T03:55:09.112556+0000","last_peered":"2024-09-17T03:55:09.112556+0000","last_clean":"2024-09-17T03:55:09.112556+0000","last_became_active":"2024-09-17T03:55:09.107581+0000","last_became_peered":"2024-09-17T03:55:09.107581+0000","last_unstale":"2024-09-17T03:55:09.112556+0000","last_undegraded":"2024-09-17T03:55:09.112556+0000","last_fullsized":"2024-09-17T03:55:09.112556+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:51.503143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423981+0000","last_change":"2024-09-17T03:55:10.115350+0000","last_active":"2024-09-17T03:55:11.423981+0000","last_peered":"2024-09-17T03:55:11.423981+0000","last_clean":"2024-09-17T03:55:11.423981+0000","last_became_active":"2024-09-17T03:55:10.111606+0000","last_became_peered":"2024-09-17T03:55:10.111606+0000","last_unstale":"2024-09-17T03:55:11.423981+0000","last_undegraded":"2024-09-17T03:55:11.423981+0000","last_fullsized":"2024-09-17T03:55:11.423981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:26.916486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104020+0000","last_change":"2024-09-17T03:55:09.100552+0000","last_active":"2024-09-17T03:55:11.104020+0000","last_peered":"2024-09-17T03:55:11.104020+0000","last_clean":"2024-09-17T03:55:11.104020+0000","last_became_active":"2024-09-17T03:55:09.100230+0000","last_became_peered":"2024-09-17T03:55:09.100230+0000","last_unstale":"2024-09-17T03:55:11.104020+0000","last_undegraded":"2024-09-17T03:55:11.104020+0000","last_fullsized":"2024-09-17T03:55:11.104020+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:26.431547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425676+0000","last_change":"2024-09-17T03:55:10.125057+0000","last_active":"2024-09-17T03:55:11.425676+0000","last_peered":"2024-09-17T03:55:11.425676+0000","last_clean":"2024-09-17T03:55:11.425676+0000","last_became_active":"2024-09-17T03:55:10.124788+0000","last_became_peered":"2024-09-17T03:55:10.124788+0000","last_unstale":"2024-09-17T03:55:11.425676+0000","last_undegraded":"2024-09-17T03:55:11.425676+0000","last_fullsized":"2024-09-17T03:55:11.425676+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:49.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.103014+0000","last_change":"2024-09-17T03:55:09.098297+0000","last_active":"2024-09-17T03:55:09.103014+0000","last_peered":"2024-09-17T03:55:09.103014+0000","last_clean":"2024-09-17T03:55:09.103014+0000","last_became_active":"2024-09-17T03:55:09.097970+0000","last_became_peered":"2024-09-17T03:55:09.097970+0000","last_unstale":"2024-09-17T03:55:09.103014+0000","last_undegraded":"2024-09-17T03:55:09.103014+0000","last_fullsized":"2024-09-17T03:55:09.103014+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:08:34.560601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424050+0000","last_change":"2024-09-17T03:55:10.122441+0000","last_active":"2024-09-17T03:55:11.424050+0000","last_peered":"2024-09-17T03:55:11.424050+0000","last_clean":"2024-09-17T03:55:11.424050+0000","last_became_active":"2024-09-17T03:55:10.121881+0000","last_became_peered":"2024-09-17T03:55:10.121881+0000","last_unstale":"2024-09-17T03:55:11.424050+0000","last_undegraded":"2024-09-17T03:55:11.424050+0000","last_fullsized":"2024-09-17T03:55:11.424050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.313711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425520+0000","last_change":"2024-09-17T03:55:09.101386+0000","last_active":"2024-09-17T03:55:11.425520+0000","last_peered":"2024-09-17T03:55:11.425520+0000","last_clean":"2024-09-17T03:55:11.425520+0000","last_became_active":"2024-09-17T03:55:09.101041+0000","last_became_peered":"2024-09-17T03:55:09.101041+0000","last_unstale":"2024-09-17T03:55:11.425520+0000","last_undegraded":"2024-09-17T03:55:11.425520+0000","last_fullsized":"2024-09-17T03:55:11.425520+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:07.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104041+0000","last_change":"2024-09-17T03:55:10.097916+0000","last_active":"2024-09-17T03:55:11.104041+0000","last_peered":"2024-09-17T03:55:11.104041+0000","last_clean":"2024-09-17T03:55:11.104041+0000","last_became_active":"2024-09-17T03:55:10.097595+0000","last_became_peered":"2024-09-17T03:55:10.097595+0000","last_unstale":"2024-09-17T03:55:11.104041+0000","last_undegraded":"2024-09-17T03:55:11.104041+0000","last_fullsized":"2024-09-17T03:55:11.104041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423852+0000","last_change":"2024-09-17T03:55:09.093148+0000","last_active":"2024-09-17T03:55:11.423852+0000","last_peered":"2024-09-17T03:55:11.423852+0000","last_clean":"2024-09-17T03:55:11.423852+0000","last_became_active":"2024-09-17T03:55:09.092860+0000","last_became_peered":"2024-09-17T03:55:09.092860+0000","last_unstale":"2024-09-17T03:55:11.423852+0000","last_undegraded":"2024-09-17T03:55:11.423852+0000","last_fullsized":"2024-09-17T03:55:11.423852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:11.814702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104560+0000","last_change":"2024-09-17T03:55:10.102328+0000","last_active":"2024-09-17T03:55:11.104560+0000","last_peered":"2024-09-17T03:55:11.104560+0000","last_clean":"2024-09-17T03:55:11.104560+0000","last_became_active":"2024-09-17T03:55:10.101937+0000","last_became_peered":"2024-09-17T03:55:10.101937+0000","last_unstale":"2024-09-17T03:55:11.104560+0000","last_undegraded":"2024-09-17T03:55:11.104560+0000","last_fullsized":"2024-09-17T03:55:11.104560+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:53.259242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112624+0000","last_change":"2024-09-17T03:55:09.110979+0000","last_active":"2024-09-17T03:55:09.112624+0000","last_peered":"2024-09-17T03:55:09.112624+0000","last_clean":"2024-09-17T03:55:09.112624+0000","last_became_active":"2024-09-17T03:55:09.110640+0000","last_became_peered":"2024-09-17T03:55:09.110640+0000","last_unstale":"2024-09-17T03:55:09.112624+0000","last_undegraded":"2024-09-17T03:55:09.112624+0000","last_fullsized":"2024-09-17T03:55:09.112624+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:42.556858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426564+0000","last_change":"2024-09-17T03:55:10.105276+0000","last_active":"2024-09-17T03:55:11.426564+0000","last_peered":"2024-09-17T03:55:11.426564+0000","last_clean":"2024-09-17T03:55:11.426564+0000","last_became_active":"2024-09-17T03:55:10.104973+0000","last_became_peered":"2024-09-17T03:55:10.104973+0000","last_unstale":"2024-09-17T03:55:11.426564+0000","last_undegraded":"2024-09-17T03:55:11.426564+0000","last_fullsized":"2024-09-17T03:55:11.426564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:37.836511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.105305+0000","last_change":"2024-09-17T03:55:09.100586+0000","last_active":"2024-09-17T03:55:09.105305+0000","last_peered":"2024-09-17T03:55:09.105305+0000","last_clean":"2024-09-17T03:55:09.105305+0000","last_became_active":"2024-09-17T03:55:09.100303+0000","last_became_peered":"2024-09-17T03:55:09.100303+0000","last_unstale":"2024-09-17T03:55:09.105305+0000","last_undegraded":"2024-09-17T03:55:09.105305+0000","last_fullsized":"2024-09-17T03:55:09.105305+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:36.891586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424336+0000","last_change":"2024-09-17T03:55:09.101674+0000","last_active":"2024-09-17T03:55:11.424336+0000","last_peered":"2024-09-17T03:55:11.424336+0000","last_clean":"2024-09-17T03:55:11.424336+0000","last_became_active":"2024-09-17T03:55:09.101311+0000","last_became_peered":"2024-09-17T03:55:09.101311+0000","last_unstale":"2024-09-17T03:55:11.424336+0000","last_undegraded":"2024-09-17T03:55:11.424336+0000","last_fullsized":"2024-09-17T03:55:11.424336+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:43.324925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.105085+0000","last_change":"2024-09-17T03:55:10.100605+0000","last_active":"2024-09-17T03:55:11.105085+0000","last_peered":"2024-09-17T03:55:11.105085+0000","last_clean":"2024-09-17T03:55:11.105085+0000","last_became_active":"2024-09-17T03:55:10.100264+0000","last_became_peered":"2024-09-17T03:55:10.100264+0000","last_unstale":"2024-09-17T03:55:11.105085+0000","last_undegraded":"2024-09-17T03:55:11.105085+0000","last_fullsized":"2024-09-17T03:55:11.105085+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:31.420107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":21,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424538+0000","last_change":"2024-09-17T03:55:09.102173+0000","last_active":"2024-09-17T03:55:11.424538+0000","last_peered":"2024-09-17T03:55:11.424538+0000","last_clean":"2024-09-17T03:55:11.424538+0000","last_became_active":"2024-09-17T03:55:09.101451+0000","last_became_peered":"2024-09-17T03:55:09.101451+0000","last_unstale":"2024-09-17T03:55:11.424538+0000","last_undegraded":"2024-09-17T03:55:11.424538+0000","last_fullsized":"2024-09-17T03:55:11.424538+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:16.929007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085389+0000","last_change":"2024-09-17T03:55:09.085389+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085389+0000","last_undegraded":"2024-09-17T03:55:09.085389+0000","last_fullsized":"2024-09-17T03:55:09.085389+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.107231+0000","last_change":"2024-09-17T03:55:09.101519+0000","last_active":"2024-09-17T03:55:09.107231+0000","last_peered":"2024-09-17T03:55:09.107231+0000","last_clean":"2024-09-17T03:55:09.107231+0000","last_became_active":"2024-09-17T03:55:09.101211+0000","last_became_peered":"2024-09-17T03:55:09.101211+0000","last_unstale":"2024-09-17T03:55:09.107231+0000","last_undegraded":"2024-09-17T03:55:09.107231+0000","last_fullsized":"2024-09-17T03:55:09.107231+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:29.559882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.088070+0000","last_change":"2024-09-17T03:55:09.088070+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.088070+0000","last_undegraded":"2024-09-17T03:55:09.088070+0000","last_fullsized":"2024-09-17T03:55:09.088070+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424612+0000","last_change":"2024-09-17T03:55:09.103112+0000","last_active":"2024-09-17T03:55:11.424612+0000","last_peered":"2024-09-17T03:55:11.424612+0000","last_clean":"2024-09-17T03:55:11.424612+0000","last_became_active":"2024-09-17T03:55:09.102182+0000","last_became_peered":"2024-09-17T03:55:09.102182+0000","last_unstale":"2024-09-17T03:55:11.424612+0000","last_undegraded":"2024-09-17T03:55:11.424612+0000","last_fullsized":"2024-09-17T03:55:11.424612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:53:55.298078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084668+0000","last_change":"2024-09-17T03:55:09.084668+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084668+0000","last_undegraded":"2024-09-17T03:55:09.084668+0000","last_fullsized":"2024-09-17T03:55:09.084668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112617+0000","last_change":"2024-09-17T03:55:09.110547+0000","last_active":"2024-09-17T03:55:09.112617+0000","last_peered":"2024-09-17T03:55:09.112617+0000","last_clean":"2024-09-17T03:55:09.112617+0000","last_became_active":"2024-09-17T03:55:09.109930+0000","last_became_peered":"2024-09-17T03:55:09.109930+0000","last_unstale":"2024-09-17T03:55:09.112617+0000","last_undegraded":"2024-09-17T03:55:09.112617+0000","last_fullsized":"2024-09-17T03:55:09.112617+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:42.335248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.095435+0000","last_change":"2024-09-17T03:55:09.095435+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.095435+0000","last_undegraded":"2024-09-17T03:55:09.095435+0000","last_fullsized":"2024-09-17T03:55:09.095435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424999+0000","last_change":"2024-09-17T03:55:09.102865+0000","last_active":"2024-09-17T03:55:11.424999+0000","last_peered":"2024-09-17T03:55:11.424999+0000","last_clean":"2024-09-17T03:55:11.424999+0000","last_became_active":"2024-09-17T03:55:09.100408+0000","last_became_peered":"2024-09-17T03:55:09.100408+0000","last_unstale":"2024-09-17T03:55:11.424999+0000","last_undegraded":"2024-09-17T03:55:11.424999+0000","last_fullsized":"2024-09-17T03:55:11.424999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:29.269455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085989+0000","last_change":"2024-09-17T03:55:09.085989+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085989+0000","last_undegraded":"2024-09-17T03:55:09.085989+0000","last_fullsized":"2024-09-17T03:55:09.085989+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109499+0000","last_change":"2024-09-17T03:55:09.106143+0000","last_active":"2024-09-17T03:55:09.109499+0000","last_peered":"2024-09-17T03:55:09.109499+0000","last_clean":"2024-09-17T03:55:09.109499+0000","last_became_active":"2024-09-17T03:55:09.105539+0000","last_became_peered":"2024-09-17T03:55:09.105539+0000","last_unstale":"2024-09-17T03:55:09.109499+0000","last_undegraded":"2024-09-17T03:55:09.109499+0000","last_fullsized":"2024-09-17T03:55:09.109499+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:18.027915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425055+0000","last_change":"2024-09-17T03:55:10.118502+0000","last_active":"2024-09-17T03:55:11.425055+0000","last_peered":"2024-09-17T03:55:11.425055+0000","last_clean":"2024-09-17T03:55:11.425055+0000","last_became_active":"2024-09-17T03:55:10.118276+0000","last_became_peered":"2024-09-17T03:55:10.118276+0000","last_unstale":"2024-09-17T03:55:11.425055+0000","last_undegraded":"2024-09-17T03:55:11.425055+0000","last_fullsized":"2024-09-17T03:55:11.425055+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:23.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425392+0000","last_change":"2024-09-17T03:55:09.099409+0000","last_active":"2024-09-17T03:55:11.425392+0000","last_peered":"2024-09-17T03:55:11.425392+0000","last_clean":"2024-09-17T03:55:11.425392+0000","last_became_active":"2024-09-17T03:55:09.097894+0000","last_became_peered":"2024-09-17T03:55:09.097894+0000","last_unstale":"2024-09-17T03:55:11.425392+0000","last_undegraded":"2024-09-17T03:55:11.425392+0000","last_fullsized":"2024-09-17T03:55:11.425392+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:30.592901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426005+0000","last_change":"2024-09-17T03:55:10.101315+0000","last_active":"2024-09-17T03:55:11.426005+0000","last_peered":"2024-09-17T03:55:11.426005+0000","last_clean":"2024-09-17T03:55:11.426005+0000","last_became_active":"2024-09-17T03:55:10.100936+0000","last_became_peered":"2024-09-17T03:55:10.100936+0000","last_unstale":"2024-09-17T03:55:11.426005+0000","last_undegraded":"2024-09-17T03:55:11.426005+0000","last_fullsized":"2024-09-17T03:55:11.426005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:51.665179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104091+0000","last_change":"2024-09-17T03:55:09.095974+0000","last_active":"2024-09-17T03:55:11.104091+0000","last_peered":"2024-09-17T03:55:11.104091+0000","last_clean":"2024-09-17T03:55:11.104091+0000","last_became_active":"2024-09-17T03:55:09.095632+0000","last_became_peered":"2024-09-17T03:55:09.095632+0000","last_unstale":"2024-09-17T03:55:11.104091+0000","last_undegraded":"2024-09-17T03:55:11.104091+0000","last_fullsized":"2024-09-17T03:55:11.104091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.401993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424900+0000","last_change":"2024-09-17T03:55:10.104348+0000","last_active":"2024-09-17T03:55:11.424900+0000","last_peered":"2024-09-17T03:55:11.424900+0000","last_clean":"2024-09-17T03:55:11.424900+0000","last_became_active":"2024-09-17T03:55:10.104019+0000","last_became_peered":"2024-09-17T03:55:10.104019+0000","last_unstale":"2024-09-17T03:55:11.424900+0000","last_undegraded":"2024-09-17T03:55:11.424900+0000","last_fullsized":"2024-09-17T03:55:11.424900+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.618440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.108346+0000","last_change":"2024-09-17T03:55:09.107009+0000","last_active":"2024-09-17T03:55:09.108346+0000","last_peered":"2024-09-17T03:55:09.108346+0000","last_clean":"2024-09-17T03:55:09.108346+0000","last_became_active":"2024-09-17T03:55:09.106706+0000","last_became_peered":"2024-09-17T03:55:09.106706+0000","last_unstale":"2024-09-17T03:55:09.108346+0000","last_undegraded":"2024-09-17T03:55:09.108346+0000","last_fullsized":"2024-09-17T03:55:09.108346+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:13.263489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424019+0000","last_change":"2024-09-17T03:55:10.101801+0000","last_active":"2024-09-17T03:55:11.424019+0000","last_peered":"2024-09-17T03:55:11.424019+0000","last_clean":"2024-09-17T03:55:11.424019+0000","last_became_active":"2024-09-17T03:55:10.101240+0000","last_became_peered":"2024-09-17T03:55:10.101240+0000","last_unstale":"2024-09-17T03:55:11.424019+0000","last_undegraded":"2024-09-17T03:55:11.424019+0000","last_fullsized":"2024-09-17T03:55:11.424019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:08.177233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109573+0000","last_change":"2024-09-17T03:55:09.105469+0000","last_active":"2024-09-17T03:55:09.109573+0000","last_peered":"2024-09-17T03:55:09.109573+0000","last_clean":"2024-09-17T03:55:09.109573+0000","last_became_active":"2024-09-17T03:55:09.105074+0000","last_became_peered":"2024-09-17T03:55:09.105074+0000","last_unstale":"2024-09-17T03:55:09.109573+0000","last_undegraded":"2024-09-17T03:55:09.109573+0000","last_fullsized":"2024-09-17T03:55:09.109573+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:37.786492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425201+0000","last_change":"2024-09-17T03:55:10.110424+0000","last_active":"2024-09-17T03:55:11.425201+0000","last_peered":"2024-09-17T03:55:11.425201+0000","last_clean":"2024-09-17T03:55:11.425201+0000","last_became_active":"2024-09-17T03:55:10.110057+0000","last_became_peered":"2024-09-17T03:55:10.110057+0000","last_unstale":"2024-09-17T03:55:11.425201+0000","last_undegraded":"2024-09-17T03:55:11.425201+0000","last_fullsized":"2024-09-17T03:55:11.425201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:40.073222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.111837+0000","last_change":"2024-09-17T03:55:09.109486+0000","last_active":"2024-09-17T03:55:09.111837+0000","last_peered":"2024-09-17T03:55:09.111837+0000","last_clean":"2024-09-17T03:55:09.111837+0000","last_became_active":"2024-09-17T03:55:09.108902+0000","last_became_peered":"2024-09-17T03:55:09.108902+0000","last_unstale":"2024-09-17T03:55:09.111837+0000","last_undegraded":"2024-09-17T03:55:09.111837+0000","last_fullsized":"2024-09-17T03:55:09.111837+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:45.582374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084094+0000","last_change":"2024-09-17T03:55:09.084094+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084094+0000","last_undegraded":"2024-09-17T03:55:09.084094+0000","last_fullsized":"2024-09-17T03:55:09.084094+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423957+0000","last_change":"2024-09-17T03:55:09.089727+0000","last_active":"2024-09-17T03:55:11.423957+0000","last_peered":"2024-09-17T03:55:11.423957+0000","last_clean":"2024-09-17T03:55:11.423957+0000","last_became_active":"2024-09-17T03:55:09.089416+0000","last_became_peered":"2024-09-17T03:55:09.089416+0000","last_unstale":"2024-09-17T03:55:11.423957+0000","last_undegraded":"2024-09-17T03:55:11.423957+0000","last_fullsized":"2024-09-17T03:55:11.423957+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:19.704845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426516+0000","last_change":"2024-09-17T03:55:10.106649+0000","last_active":"2024-09-17T03:55:11.426516+0000","last_peered":"2024-09-17T03:55:11.426516+0000","last_clean":"2024-09-17T03:55:11.426516+0000","last_became_active":"2024-09-17T03:55:10.106347+0000","last_became_peered":"2024-09-17T03:55:10.106347+0000","last_unstale":"2024-09-17T03:55:11.426516+0000","last_undegraded":"2024-09-17T03:55:11.426516+0000","last_fullsized":"2024-09-17T03:55:11.426516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:21.848148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.110091+0000","last_change":"2024-09-17T03:55:09.106794+0000","last_active":"2024-09-17T03:55:09.110091+0000","last_peered":"2024-09-17T03:55:09.110091+0000","last_clean":"2024-09-17T03:55:09.110091+0000","last_became_active":"2024-09-17T03:55:09.106461+0000","last_became_peered":"2024-09-17T03:55:09.106461+0000","last_unstale":"2024-09-17T03:55:09.110091+0000","last_undegraded":"2024-09-17T03:55:09.110091+0000","last_fullsized":"2024-09-17T03:55:09.110091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:22.706399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424964+0000","last_change":"2024-09-17T03:55:10.111117+0000","last_active":"2024-09-17T03:55:11.424964+0000","last_peered":"2024-09-17T03:55:11.424964+0000","last_clean":"2024-09-17T03:55:11.424964+0000","last_became_active":"2024-09-17T03:55:10.110769+0000","last_became_peered":"2024-09-17T03:55:10.110769+0000","last_unstale":"2024-09-17T03:55:11.424964+0000","last_undegraded":"2024-09-17T03:55:11.424964+0000","last_fullsized":"2024-09-17T03:55:11.424964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:34.220767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426576+0000","last_change":"2024-09-17T03:55:09.091992+0000","last_active":"2024-09-17T03:55:11.426576+0000","last_peered":"2024-09-17T03:55:11.426576+0000","last_clean":"2024-09-17T03:55:11.426576+0000","last_became_active":"2024-09-17T03:55:09.091655+0000","last_became_peered":"2024-09-17T03:55:09.091655+0000","last_unstale":"2024-09-17T03:55:11.426576+0000","last_undegraded":"2024-09-17T03:55:11.426576+0000","last_fullsized":"2024-09-17T03:55:11.426576+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:14.155619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424655+0000","last_change":"2024-09-17T03:55:10.125141+0000","last_active":"2024-09-17T03:55:11.424655+0000","last_peered":"2024-09-17T03:55:11.424655+0000","last_clean":"2024-09-17T03:55:11.424655+0000","last_became_active":"2024-09-17T03:55:10.124839+0000","last_became_peered":"2024-09-17T03:55:10.124839+0000","last_unstale":"2024-09-17T03:55:11.424655+0000","last_undegraded":"2024-09-17T03:55:11.424655+0000","last_fullsized":"2024-09-17T03:55:11.424655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:49.605272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424463+0000","last_change":"2024-09-17T03:55:09.092456+0000","last_active":"2024-09-17T03:55:11.424463+0000","last_peered":"2024-09-17T03:55:11.424463+0000","last_clean":"2024-09-17T03:55:11.424463+0000","last_became_active":"2024-09-17T03:55:09.092133+0000","last_became_peered":"2024-09-17T03:55:09.092133+0000","last_unstale":"2024-09-17T03:55:11.424463+0000","last_undegraded":"2024-09-17T03:55:11.424463+0000","last_fullsized":"2024-09-17T03:55:11.424463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:29.592608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425399+0000","last_change":"2024-09-17T03:55:10.105263+0000","last_active":"2024-09-17T03:55:11.425399+0000","last_peered":"2024-09-17T03:55:11.425399+0000","last_clean":"2024-09-17T03:55:11.425399+0000","last_became_active":"2024-09-17T03:55:10.104878+0000","last_became_peered":"2024-09-17T03:55:10.104878+0000","last_unstale":"2024-09-17T03:55:11.425399+0000","last_undegraded":"2024-09-17T03:55:11.425399+0000","last_fullsized":"2024-09-17T03:55:11.425399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:29.686662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.104632+0000","last_change":"2024-09-17T03:55:09.101334+0000","last_active":"2024-09-17T03:55:09.104632+0000","last_peered":"2024-09-17T03:55:09.104632+0000","last_clean":"2024-09-17T03:55:09.104632+0000","last_became_active":"2024-09-17T03:55:09.101016+0000","last_became_peered":"2024-09-17T03:55:09.101016+0000","last_unstale":"2024-09-17T03:55:09.104632+0000","last_undegraded":"2024-09-17T03:55:09.104632+0000","last_fullsized":"2024-09-17T03:55:09.104632+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:30.035885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425931+0000","last_change":"2024-09-17T03:55:10.097057+0000","last_active":"2024-09-17T03:55:11.425931+0000","last_peered":"2024-09-17T03:55:11.425931+0000","last_clean":"2024-09-17T03:55:11.425931+0000","last_became_active":"2024-09-17T03:55:10.096210+0000","last_became_peered":"2024-09-17T03:55:10.096210+0000","last_unstale":"2024-09-17T03:55:11.425931+0000","last_undegraded":"2024-09-17T03:55:11.425931+0000","last_fullsized":"2024-09-17T03:55:11.425931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:04.260938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104187+0000","last_change":"2024-09-17T03:55:09.099774+0000","last_active":"2024-09-17T03:55:11.104187+0000","last_peered":"2024-09-17T03:55:11.104187+0000","last_clean":"2024-09-17T03:55:11.104187+0000","last_became_active":"2024-09-17T03:55:09.099422+0000","last_became_peered":"2024-09-17T03:55:09.099422+0000","last_unstale":"2024-09-17T03:55:11.104187+0000","last_undegraded":"2024-09-17T03:55:11.104187+0000","last_fullsized":"2024-09-17T03:55:11.104187+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:56.088001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426331+0000","last_change":"2024-09-17T03:55:10.112375+0000","last_active":"2024-09-17T03:55:11.426331+0000","last_peered":"2024-09-17T03:55:11.426331+0000","last_clean":"2024-09-17T03:55:11.426331+0000","last_became_active":"2024-09-17T03:55:10.112033+0000","last_became_peered":"2024-09-17T03:55:10.112033+0000","last_unstale":"2024-09-17T03:55:11.426331+0000","last_undegraded":"2024-09-17T03:55:11.426331+0000","last_fullsized":"2024-09-17T03:55:11.426331+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426160+0000","last_change":"2024-09-17T03:55:09.103012+0000","last_active":"2024-09-17T03:55:11.426160+0000","last_peered":"2024-09-17T03:55:11.426160+0000","last_clean":"2024-09-17T03:55:11.426160+0000","last_became_active":"2024-09-17T03:55:09.102701+0000","last_became_peered":"2024-09-17T03:55:09.102701+0000","last_unstale":"2024-09-17T03:55:11.426160+0000","last_undegraded":"2024-09-17T03:55:11.426160+0000","last_fullsized":"2024-09-17T03:55:11.426160+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:02.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425468+0000","last_change":"2024-09-17T03:55:10.113187+0000","last_active":"2024-09-17T03:55:11.425468+0000","last_peered":"2024-09-17T03:55:11.425468+0000","last_clean":"2024-09-17T03:55:11.425468+0000","last_became_active":"2024-09-17T03:55:10.112812+0000","last_became_peered":"2024-09-17T03:55:10.112812+0000","last_unstale":"2024-09-17T03:55:11.425468+0000","last_undegraded":"2024-09-17T03:55:11.425468+0000","last_fullsized":"2024-09-17T03:55:11.425468+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:18.693960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112065+0000","last_change":"2024-09-17T03:55:09.108670+0000","last_active":"2024-09-17T03:55:09.112065+0000","last_peered":"2024-09-17T03:55:09.112065+0000","last_clean":"2024-09-17T03:55:09.112065+0000","last_became_active":"2024-09-17T03:55:09.106248+0000","last_became_peered":"2024-09-17T03:55:09.106248+0000","last_unstale":"2024-09-17T03:55:09.112065+0000","last_undegraded":"2024-09-17T03:55:09.112065+0000","last_fullsized":"2024-09-17T03:55:09.112065+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:54.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423930+0000","last_change":"2024-09-17T03:55:10.126012+0000","last_active":"2024-09-17T03:55:11.423930+0000","last_peered":"2024-09-17T03:55:11.423930+0000","last_clean":"2024-09-17T03:55:11.423930+0000","last_became_active":"2024-09-17T03:55:10.125592+0000","last_became_peered":"2024-09-17T03:55:10.125592+0000","last_unstale":"2024-09-17T03:55:11.423930+0000","last_undegraded":"2024-09-17T03:55:11.423930+0000","last_fullsized":"2024-09-17T03:55:11.423930+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:17.583329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424054+0000","last_change":"2024-09-17T03:55:09.096029+0000","last_active":"2024-09-17T03:55:11.424054+0000","last_peered":"2024-09-17T03:55:11.424054+0000","last_clean":"2024-09-17T03:55:11.424054+0000","last_became_active":"2024-09-17T03:55:09.091782+0000","last_became_peered":"2024-09-17T03:55:09.091782+0000","last_unstale":"2024-09-17T03:55:11.424054+0000","last_undegraded":"2024-09-17T03:55:11.424054+0000","last_fullsized":"2024-09-17T03:55:11.424054+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:17.583743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085271+0000","last_change":"2024-09-17T03:55:09.085271+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085271+0000","last_undegraded":"2024-09-17T03:55:09.085271+0000","last_fullsized":"2024-09-17T03:55:09.085271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104276+0000","last_change":"2024-09-17T03:55:09.096214+0000","last_active":"2024-09-17T03:55:11.104276+0000","last_peered":"2024-09-17T03:55:11.104276+0000","last_clean":"2024-09-17T03:55:11.104276+0000","last_became_active":"2024-09-17T03:55:09.095625+0000","last_became_peered":"2024-09-17T03:55:09.095625+0000","last_unstale":"2024-09-17T03:55:11.104276+0000","last_undegraded":"2024-09-17T03:55:11.104276+0000","last_fullsized":"2024-09-17T03:55:11.104276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:12.569897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":265,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425604+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:55:11.425604+0000","last_peered":"2024-09-17T03:55:11.425604+0000","last_clean":"2024-09-17T03:55:11.425604+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:55:11.425604+0000","last_undegraded":"2024-09-17T03:55:11.425604+0000","last_fullsized":"2024-09-17T03:55:11.425604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1157630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":567250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509511,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1327104,"data_stored":566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542217,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":1158738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:55:14.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:55:14.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2875734565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c41602b0 msgr2=0x7fe9c40cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2875734565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cee20 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe9c40ce910 tx=0x7fe9b002a560 comp rx=0 tx=0).stop 2024-09-17T03:55:14.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2875734565 shutdown_connections 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2875734565 >> v2:172.21.15.181:3300/0 conn(0x7fe9c41606c0 0x7fe9c40cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2875734565 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cee20 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2875734565 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c415a220 0x7fe9c415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2875734565 >> 172.21.15.129:0/2875734565 conn(0x7fe9c4069150 msgr2=0x7fe9c4069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2875734565 shutdown_connections 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.963+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2875734565 wait complete. 2024-09-17T03:55:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.965+0000 7fe9c3e00640 1 Processor -- start 2024-09-17T03:55:14.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.965+0000 7fe9c3e00640 1 -- start start 2024-09-17T03:55:14.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.966+0000 7fe9c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.966+0000 7fe9c3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 0x7fe9c406def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c40d00d0 con 0x7fe9c415a220 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c40cff50 con 0x7fe9c41602b0 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c40d0250 con 0x7fe9c41606c0 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53140/0 (socket says 172.21.15.129:53140) 2024-09-17T03:55:14.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 -- 172.21.15.129:0/2421147083 learned_addr learned my addr 172.21.15.129:0/2421147083 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3400640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 0x7fe9c406def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2000640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 -- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 msgr2=0x7fe9c406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 0x7fe9c406def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 -- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 msgr2=0x7fe9c40cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:14.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2a00640 1 -- 172.21.15.129:0/2421147083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c40cbca0 con 0x7fe9c415a220 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c3400640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 0x7fe9c406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.967+0000 7fe9c2000640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9c2a00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c940 tx=0x7fe9b400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4007bb0 con 0x7fe9c415a220 2024-09-17T03:55:14.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9b4007e40 con 0x7fe9c415a220 2024-09-17T03:55:14.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c40c95d0 con 0x7fe9c415a220 2024-09-17T03:55:14.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400f670 con 0x7fe9c415a220 2024-09-17T03:55:14.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.968+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c40cc6a0 con 0x7fe9c415a220 2024-09-17T03:55:14.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.969+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c40d07e0 con 0x7fe9c415a220 2024-09-17T03:55:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.970+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe9b400f810 con 0x7fe9c415a220 2024-09-17T03:55:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.971+0000 7fe9baa00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 0x7fe9a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.971+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fe9b409b510 con 0x7fe9c415a220 2024-09-17T03:55:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.972+0000 7fe9c2000640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 0x7fe9a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.972+0000 7fe9c2000640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 0x7fe9a407b0a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0033000 tx=0x7fe9b0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:14.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:14.976+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe9b4014030 con 0x7fe9c415a220 2024-09-17T03:55:15.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.158+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9c406a130 con 0x7fe9a4078b90 2024-09-17T03:55:15.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.163+0000 7fe9baa00640 1 -- 172.21.15.129:0/2421147083 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250957 (secure 0 0 0) 0x7fe9c406a130 con 0x7fe9a4078b90 2024-09-17T03:55:15.166 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:15.168 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:55:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.167+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 msgr2=0x7fe9a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.167+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 0x7fe9a407b0a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0033000 tx=0x7fe9b0004730 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.167+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 msgr2=0x7fe9c40b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.167+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe9b400c940 tx=0x7fe9b400ce00 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 shutdown_connections 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9a4078b90 0x7fe9a407b0a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9c41606c0 0x7fe9c406def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9c41602b0 0x7fe9c40cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 --2- 172.21.15.129:0/2421147083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9c415a220 0x7fe9c40b0e70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 >> 172.21.15.129:0/2421147083 conn(0x7fe9c4069150 msgr2=0x7fe9c415f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 shutdown_connections 2024-09-17T03:55:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.168+0000 7fe9c3e00640 1 -- 172.21.15.129:0/2421147083 wait complete. 2024-09-17T03:55:15.189 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-17T03:55:14.387295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":118,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302584,"kb_used_data":11428,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672136,"statfs":{"total":773094113280,"available":772784267264,"internally_reserved":0,"allocated":11702272,"data_stored":5658868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001744"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424841+0000","last_change":"2024-09-17T03:55:10.112479+0000","last_active":"2024-09-17T03:55:11.424841+0000","last_peered":"2024-09-17T03:55:11.424841+0000","last_clean":"2024-09-17T03:55:11.424841+0000","last_became_active":"2024-09-17T03:55:10.112184+0000","last_became_peered":"2024-09-17T03:55:10.112184+0000","last_unstale":"2024-09-17T03:55:11.424841+0000","last_undegraded":"2024-09-17T03:55:11.424841+0000","last_fullsized":"2024-09-17T03:55:11.424841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:12.328355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425890+0000","last_change":"2024-09-17T03:55:10.123473+0000","last_active":"2024-09-17T03:55:11.425890+0000","last_peered":"2024-09-17T03:55:11.425890+0000","last_clean":"2024-09-17T03:55:11.425890+0000","last_became_active":"2024-09-17T03:55:10.123217+0000","last_became_peered":"2024-09-17T03:55:10.123217+0000","last_unstale":"2024-09-17T03:55:11.425890+0000","last_undegraded":"2024-09-17T03:55:11.425890+0000","last_fullsized":"2024-09-17T03:55:11.425890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:50.947437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425736+0000","last_change":"2024-09-17T03:55:10.103667+0000","last_active":"2024-09-17T03:55:11.425736+0000","last_peered":"2024-09-17T03:55:11.425736+0000","last_clean":"2024-09-17T03:55:11.425736+0000","last_became_active":"2024-09-17T03:55:10.103336+0000","last_became_peered":"2024-09-17T03:55:10.103336+0000","last_unstale":"2024-09-17T03:55:11.425736+0000","last_undegraded":"2024-09-17T03:55:11.425736+0000","last_fullsized":"2024-09-17T03:55:11.425736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:33.517751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424369+0000","last_change":"2024-09-17T03:55:10.109342+0000","last_active":"2024-09-17T03:55:11.424369+0000","last_peered":"2024-09-17T03:55:11.424369+0000","last_clean":"2024-09-17T03:55:11.424369+0000","last_became_active":"2024-09-17T03:55:10.109013+0000","last_became_peered":"2024-09-17T03:55:10.109013+0000","last_unstale":"2024-09-17T03:55:11.424369+0000","last_undegraded":"2024-09-17T03:55:11.424369+0000","last_fullsized":"2024-09-17T03:55:11.424369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:01.880305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.093607+0000","last_change":"2024-09-17T03:55:09.093607+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.093607+0000","last_undegraded":"2024-09-17T03:55:09.093607+0000","last_fullsized":"2024-09-17T03:55:09.093607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.090308+0000","last_change":"2024-09-17T03:55:09.090308+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.090308+0000","last_undegraded":"2024-09-17T03:55:09.090308+0000","last_fullsized":"2024-09-17T03:55:09.090308+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104179+0000","last_change":"2024-09-17T03:55:10.096583+0000","last_active":"2024-09-17T03:55:11.104179+0000","last_peered":"2024-09-17T03:55:11.104179+0000","last_clean":"2024-09-17T03:55:11.104179+0000","last_became_active":"2024-09-17T03:55:10.096289+0000","last_became_peered":"2024-09-17T03:55:10.096289+0000","last_unstale":"2024-09-17T03:55:11.104179+0000","last_undegraded":"2024-09-17T03:55:11.104179+0000","last_fullsized":"2024-09-17T03:55:11.104179+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424746+0000","last_change":"2024-09-17T03:55:10.116515+0000","last_active":"2024-09-17T03:55:11.424746+0000","last_peered":"2024-09-17T03:55:11.424746+0000","last_clean":"2024-09-17T03:55:11.424746+0000","last_became_active":"2024-09-17T03:55:10.116161+0000","last_became_peered":"2024-09-17T03:55:10.116161+0000","last_unstale":"2024-09-17T03:55:11.424746+0000","last_undegraded":"2024-09-17T03:55:11.424746+0000","last_fullsized":"2024-09-17T03:55:11.424746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.433598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424969+0000","last_change":"2024-09-17T03:55:10.113065+0000","last_active":"2024-09-17T03:55:11.424969+0000","last_peered":"2024-09-17T03:55:11.424969+0000","last_clean":"2024-09-17T03:55:11.424969+0000","last_became_active":"2024-09-17T03:55:10.111008+0000","last_became_peered":"2024-09-17T03:55:10.111008+0000","last_unstale":"2024-09-17T03:55:11.424969+0000","last_undegraded":"2024-09-17T03:55:11.424969+0000","last_fullsized":"2024-09-17T03:55:11.424969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:54.618141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424625+0000","last_change":"2024-09-17T03:55:10.105708+0000","last_active":"2024-09-17T03:55:11.424625+0000","last_peered":"2024-09-17T03:55:11.424625+0000","last_clean":"2024-09-17T03:55:11.424625+0000","last_became_active":"2024-09-17T03:55:10.105368+0000","last_became_peered":"2024-09-17T03:55:10.105368+0000","last_unstale":"2024-09-17T03:55:11.424625+0000","last_undegraded":"2024-09-17T03:55:11.424625+0000","last_fullsized":"2024-09-17T03:55:11.424625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:19.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425340+0000","last_change":"2024-09-17T03:55:10.107346+0000","last_active":"2024-09-17T03:55:11.425340+0000","last_peered":"2024-09-17T03:55:11.425340+0000","last_clean":"2024-09-17T03:55:11.425340+0000","last_became_active":"2024-09-17T03:55:10.106955+0000","last_became_peered":"2024-09-17T03:55:10.106955+0000","last_unstale":"2024-09-17T03:55:11.425340+0000","last_undegraded":"2024-09-17T03:55:11.425340+0000","last_fullsized":"2024-09-17T03:55:11.425340+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:29.542513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424410+0000","last_change":"2024-09-17T03:55:10.108582+0000","last_active":"2024-09-17T03:55:11.424410+0000","last_peered":"2024-09-17T03:55:11.424410+0000","last_clean":"2024-09-17T03:55:11.424410+0000","last_became_active":"2024-09-17T03:55:10.105982+0000","last_became_peered":"2024-09-17T03:55:10.105982+0000","last_unstale":"2024-09-17T03:55:11.424410+0000","last_undegraded":"2024-09-17T03:55:11.424410+0000","last_fullsized":"2024-09-17T03:55:11.424410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:31.031639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425062+0000","last_change":"2024-09-17T03:55:10.109256+0000","last_active":"2024-09-17T03:55:11.425062+0000","last_peered":"2024-09-17T03:55:11.425062+0000","last_clean":"2024-09-17T03:55:11.425062+0000","last_became_active":"2024-09-17T03:55:10.108832+0000","last_became_peered":"2024-09-17T03:55:10.108832+0000","last_unstale":"2024-09-17T03:55:11.425062+0000","last_undegraded":"2024-09-17T03:55:11.425062+0000","last_fullsized":"2024-09-17T03:55:11.425062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:16.522311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424801+0000","last_change":"2024-09-17T03:55:10.112919+0000","last_active":"2024-09-17T03:55:11.424801+0000","last_peered":"2024-09-17T03:55:11.424801+0000","last_clean":"2024-09-17T03:55:11.424801+0000","last_became_active":"2024-09-17T03:55:10.112583+0000","last_became_peered":"2024-09-17T03:55:10.112583+0000","last_unstale":"2024-09-17T03:55:11.424801+0000","last_undegraded":"2024-09-17T03:55:11.424801+0000","last_fullsized":"2024-09-17T03:55:11.424801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:27.443267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087537+0000","last_change":"2024-09-17T03:55:09.087537+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087537+0000","last_undegraded":"2024-09-17T03:55:09.087537+0000","last_fullsized":"2024-09-17T03:55:09.087537+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423770+0000","last_change":"2024-09-17T03:55:10.094106+0000","last_active":"2024-09-17T03:55:11.423770+0000","last_peered":"2024-09-17T03:55:11.423770+0000","last_clean":"2024-09-17T03:55:11.423770+0000","last_became_active":"2024-09-17T03:55:10.093763+0000","last_became_peered":"2024-09-17T03:55:10.093763+0000","last_unstale":"2024-09-17T03:55:11.423770+0000","last_undegraded":"2024-09-17T03:55:11.423770+0000","last_fullsized":"2024-09-17T03:55:11.423770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.835238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426449+0000","last_change":"2024-09-17T03:55:10.101103+0000","last_active":"2024-09-17T03:55:11.426449+0000","last_peered":"2024-09-17T03:55:11.426449+0000","last_clean":"2024-09-17T03:55:11.426449+0000","last_became_active":"2024-09-17T03:55:10.100775+0000","last_became_peered":"2024-09-17T03:55:10.100775+0000","last_unstale":"2024-09-17T03:55:11.426449+0000","last_undegraded":"2024-09-17T03:55:11.426449+0000","last_fullsized":"2024-09-17T03:55:11.426449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426439+0000","last_change":"2024-09-17T03:55:10.100108+0000","last_active":"2024-09-17T03:55:11.426439+0000","last_peered":"2024-09-17T03:55:11.426439+0000","last_clean":"2024-09-17T03:55:11.426439+0000","last_became_active":"2024-09-17T03:55:10.099786+0000","last_became_peered":"2024-09-17T03:55:10.099786+0000","last_unstale":"2024-09-17T03:55:11.426439+0000","last_undegraded":"2024-09-17T03:55:11.426439+0000","last_fullsized":"2024-09-17T03:55:11.426439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:00.641678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425612+0000","last_change":"2024-09-17T03:55:10.109163+0000","last_active":"2024-09-17T03:55:11.425612+0000","last_peered":"2024-09-17T03:55:11.425612+0000","last_clean":"2024-09-17T03:55:11.425612+0000","last_became_active":"2024-09-17T03:55:10.108810+0000","last_became_peered":"2024-09-17T03:55:10.108810+0000","last_unstale":"2024-09-17T03:55:11.425612+0000","last_undegraded":"2024-09-17T03:55:11.425612+0000","last_fullsized":"2024-09-17T03:55:11.425612+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:32.706728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426258+0000","last_change":"2024-09-17T03:55:10.102425+0000","last_active":"2024-09-17T03:55:11.426258+0000","last_peered":"2024-09-17T03:55:11.426258+0000","last_clean":"2024-09-17T03:55:11.426258+0000","last_became_active":"2024-09-17T03:55:10.102086+0000","last_became_peered":"2024-09-17T03:55:10.102086+0000","last_unstale":"2024-09-17T03:55:11.426258+0000","last_undegraded":"2024-09-17T03:55:11.426258+0000","last_fullsized":"2024-09-17T03:55:11.426258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426586+0000","last_change":"2024-09-17T03:55:10.100194+0000","last_active":"2024-09-17T03:55:11.426586+0000","last_peered":"2024-09-17T03:55:11.426586+0000","last_clean":"2024-09-17T03:55:11.426586+0000","last_became_active":"2024-09-17T03:55:10.099857+0000","last_became_peered":"2024-09-17T03:55:10.099857+0000","last_unstale":"2024-09-17T03:55:11.426586+0000","last_undegraded":"2024-09-17T03:55:11.426586+0000","last_fullsized":"2024-09-17T03:55:11.426586+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085847+0000","last_change":"2024-09-17T03:55:09.085847+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085847+0000","last_undegraded":"2024-09-17T03:55:09.085847+0000","last_fullsized":"2024-09-17T03:55:09.085847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425299+0000","last_change":"2024-09-17T03:55:10.097618+0000","last_active":"2024-09-17T03:55:11.425299+0000","last_peered":"2024-09-17T03:55:11.425299+0000","last_clean":"2024-09-17T03:55:11.425299+0000","last_became_active":"2024-09-17T03:55:10.097328+0000","last_became_peered":"2024-09-17T03:55:10.097328+0000","last_unstale":"2024-09-17T03:55:11.425299+0000","last_undegraded":"2024-09-17T03:55:11.425299+0000","last_fullsized":"2024-09-17T03:55:11.425299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:23.303493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085796+0000","last_change":"2024-09-17T03:55:09.085796+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085796+0000","last_undegraded":"2024-09-17T03:55:09.085796+0000","last_fullsized":"2024-09-17T03:55:09.085796+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426752+0000","last_change":"2024-09-17T03:55:10.098678+0000","last_active":"2024-09-17T03:55:11.426752+0000","last_peered":"2024-09-17T03:55:11.426752+0000","last_clean":"2024-09-17T03:55:11.426752+0000","last_became_active":"2024-09-17T03:55:10.098318+0000","last_became_peered":"2024-09-17T03:55:10.098318+0000","last_unstale":"2024-09-17T03:55:11.426752+0000","last_undegraded":"2024-09-17T03:55:11.426752+0000","last_fullsized":"2024-09-17T03:55:11.426752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424684+0000","last_change":"2024-09-17T03:55:10.102840+0000","last_active":"2024-09-17T03:55:11.424684+0000","last_peered":"2024-09-17T03:55:11.424684+0000","last_clean":"2024-09-17T03:55:11.424684+0000","last_became_active":"2024-09-17T03:55:10.100850+0000","last_became_peered":"2024-09-17T03:55:10.100850+0000","last_unstale":"2024-09-17T03:55:11.424684+0000","last_undegraded":"2024-09-17T03:55:11.424684+0000","last_fullsized":"2024-09-17T03:55:11.424684+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:17.813222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087802+0000","last_change":"2024-09-17T03:55:09.087802+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087802+0000","last_undegraded":"2024-09-17T03:55:09.087802+0000","last_fullsized":"2024-09-17T03:55:09.087802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089564+0000","last_change":"2024-09-17T03:55:09.089564+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089564+0000","last_undegraded":"2024-09-17T03:55:09.089564+0000","last_fullsized":"2024-09-17T03:55:09.089564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425371+0000","last_change":"2024-09-17T03:55:10.103912+0000","last_active":"2024-09-17T03:55:11.425371+0000","last_peered":"2024-09-17T03:55:11.425371+0000","last_clean":"2024-09-17T03:55:11.425371+0000","last_became_active":"2024-09-17T03:55:10.103569+0000","last_became_peered":"2024-09-17T03:55:10.103569+0000","last_unstale":"2024-09-17T03:55:11.425371+0000","last_undegraded":"2024-09-17T03:55:11.425371+0000","last_fullsized":"2024-09-17T03:55:11.425371+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:36.412867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104245+0000","last_change":"2024-09-17T03:55:10.111756+0000","last_active":"2024-09-17T03:55:11.104245+0000","last_peered":"2024-09-17T03:55:11.104245+0000","last_clean":"2024-09-17T03:55:11.104245+0000","last_became_active":"2024-09-17T03:55:10.111350+0000","last_became_peered":"2024-09-17T03:55:10.111350+0000","last_unstale":"2024-09-17T03:55:11.104245+0000","last_undegraded":"2024-09-17T03:55:11.104245+0000","last_fullsized":"2024-09-17T03:55:11.104245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423827+0000","last_change":"2024-09-17T03:55:10.110031+0000","last_active":"2024-09-17T03:55:11.423827+0000","last_peered":"2024-09-17T03:55:11.423827+0000","last_clean":"2024-09-17T03:55:11.423827+0000","last_became_active":"2024-09-17T03:55:10.106150+0000","last_became_peered":"2024-09-17T03:55:10.106150+0000","last_unstale":"2024-09-17T03:55:11.423827+0000","last_undegraded":"2024-09-17T03:55:11.423827+0000","last_fullsized":"2024-09-17T03:55:11.423827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:15.416205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085837+0000","last_change":"2024-09-17T03:55:09.085837+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085837+0000","last_undegraded":"2024-09-17T03:55:09.085837+0000","last_fullsized":"2024-09-17T03:55:09.085837+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":19,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104474+0000","last_change":"2024-09-17T03:55:09.098888+0000","last_active":"2024-09-17T03:55:11.104474+0000","last_peered":"2024-09-17T03:55:11.104474+0000","last_clean":"2024-09-17T03:55:11.104474+0000","last_became_active":"2024-09-17T03:55:09.098488+0000","last_became_peered":"2024-09-17T03:55:09.098488+0000","last_unstale":"2024-09-17T03:55:11.104474+0000","last_undegraded":"2024-09-17T03:55:11.104474+0000","last_fullsized":"2024-09-17T03:55:11.104474+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:43.529970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426303+0000","last_change":"2024-09-17T03:55:10.098038+0000","last_active":"2024-09-17T03:55:11.426303+0000","last_peered":"2024-09-17T03:55:11.426303+0000","last_clean":"2024-09-17T03:55:11.426303+0000","last_became_active":"2024-09-17T03:55:10.097706+0000","last_became_peered":"2024-09-17T03:55:10.097706+0000","last_unstale":"2024-09-17T03:55:11.426303+0000","last_undegraded":"2024-09-17T03:55:11.426303+0000","last_fullsized":"2024-09-17T03:55:11.426303+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:48.538101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424685+0000","last_change":"2024-09-17T03:55:09.102970+0000","last_active":"2024-09-17T03:55:11.424685+0000","last_peered":"2024-09-17T03:55:11.424685+0000","last_clean":"2024-09-17T03:55:11.424685+0000","last_became_active":"2024-09-17T03:55:09.102558+0000","last_became_peered":"2024-09-17T03:55:09.102558+0000","last_unstale":"2024-09-17T03:55:11.424685+0000","last_undegraded":"2024-09-17T03:55:11.424685+0000","last_fullsized":"2024-09-17T03:55:11.424685+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:26.840025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084952+0000","last_change":"2024-09-17T03:55:09.084952+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084952+0000","last_undegraded":"2024-09-17T03:55:09.084952+0000","last_fullsized":"2024-09-17T03:55:09.084952+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109001+0000","last_change":"2024-09-17T03:55:09.106007+0000","last_active":"2024-09-17T03:55:09.109001+0000","last_peered":"2024-09-17T03:55:09.109001+0000","last_clean":"2024-09-17T03:55:09.109001+0000","last_became_active":"2024-09-17T03:55:09.105510+0000","last_became_peered":"2024-09-17T03:55:09.105510+0000","last_unstale":"2024-09-17T03:55:09.109001+0000","last_undegraded":"2024-09-17T03:55:09.109001+0000","last_fullsized":"2024-09-17T03:55:09.109001+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:19.653445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425907+0000","last_change":"2024-09-17T03:55:10.111580+0000","last_active":"2024-09-17T03:55:11.425907+0000","last_peered":"2024-09-17T03:55:11.425907+0000","last_clean":"2024-09-17T03:55:11.425907+0000","last_became_active":"2024-09-17T03:55:10.111121+0000","last_became_peered":"2024-09-17T03:55:10.111121+0000","last_unstale":"2024-09-17T03:55:11.425907+0000","last_undegraded":"2024-09-17T03:55:11.425907+0000","last_fullsized":"2024-09-17T03:55:11.425907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112556+0000","last_change":"2024-09-17T03:55:09.109395+0000","last_active":"2024-09-17T03:55:09.112556+0000","last_peered":"2024-09-17T03:55:09.112556+0000","last_clean":"2024-09-17T03:55:09.112556+0000","last_became_active":"2024-09-17T03:55:09.107581+0000","last_became_peered":"2024-09-17T03:55:09.107581+0000","last_unstale":"2024-09-17T03:55:09.112556+0000","last_undegraded":"2024-09-17T03:55:09.112556+0000","last_fullsized":"2024-09-17T03:55:09.112556+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:51.503143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423981+0000","last_change":"2024-09-17T03:55:10.115350+0000","last_active":"2024-09-17T03:55:11.423981+0000","last_peered":"2024-09-17T03:55:11.423981+0000","last_clean":"2024-09-17T03:55:11.423981+0000","last_became_active":"2024-09-17T03:55:10.111606+0000","last_became_peered":"2024-09-17T03:55:10.111606+0000","last_unstale":"2024-09-17T03:55:11.423981+0000","last_undegraded":"2024-09-17T03:55:11.423981+0000","last_fullsized":"2024-09-17T03:55:11.423981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:26.916486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104020+0000","last_change":"2024-09-17T03:55:09.100552+0000","last_active":"2024-09-17T03:55:11.104020+0000","last_peered":"2024-09-17T03:55:11.104020+0000","last_clean":"2024-09-17T03:55:11.104020+0000","last_became_active":"2024-09-17T03:55:09.100230+0000","last_became_peered":"2024-09-17T03:55:09.100230+0000","last_unstale":"2024-09-17T03:55:11.104020+0000","last_undegraded":"2024-09-17T03:55:11.104020+0000","last_fullsized":"2024-09-17T03:55:11.104020+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:26.431547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425676+0000","last_change":"2024-09-17T03:55:10.125057+0000","last_active":"2024-09-17T03:55:11.425676+0000","last_peered":"2024-09-17T03:55:11.425676+0000","last_clean":"2024-09-17T03:55:11.425676+0000","last_became_active":"2024-09-17T03:55:10.124788+0000","last_became_peered":"2024-09-17T03:55:10.124788+0000","last_unstale":"2024-09-17T03:55:11.425676+0000","last_undegraded":"2024-09-17T03:55:11.425676+0000","last_fullsized":"2024-09-17T03:55:11.425676+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:49.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.103014+0000","last_change":"2024-09-17T03:55:09.098297+0000","last_active":"2024-09-17T03:55:09.103014+0000","last_peered":"2024-09-17T03:55:09.103014+0000","last_clean":"2024-09-17T03:55:09.103014+0000","last_became_active":"2024-09-17T03:55:09.097970+0000","last_became_peered":"2024-09-17T03:55:09.097970+0000","last_unstale":"2024-09-17T03:55:09.103014+0000","last_undegraded":"2024-09-17T03:55:09.103014+0000","last_fullsized":"2024-09-17T03:55:09.103014+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:08:34.560601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424050+0000","last_change":"2024-09-17T03:55:10.122441+0000","last_active":"2024-09-17T03:55:11.424050+0000","last_peered":"2024-09-17T03:55:11.424050+0000","last_clean":"2024-09-17T03:55:11.424050+0000","last_became_active":"2024-09-17T03:55:10.121881+0000","last_became_peered":"2024-09-17T03:55:10.121881+0000","last_unstale":"2024-09-17T03:55:11.424050+0000","last_undegraded":"2024-09-17T03:55:11.424050+0000","last_fullsized":"2024-09-17T03:55:11.424050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.313711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425520+0000","last_change":"2024-09-17T03:55:09.101386+0000","last_active":"2024-09-17T03:55:11.425520+0000","last_peered":"2024-09-17T03:55:11.425520+0000","last_clean":"2024-09-17T03:55:11.425520+0000","last_became_active":"2024-09-17T03:55:09.101041+0000","last_became_peered":"2024-09-17T03:55:09.101041+0000","last_unstale":"2024-09-17T03:55:11.425520+0000","last_undegraded":"2024-09-17T03:55:11.425520+0000","last_fullsized":"2024-09-17T03:55:11.425520+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:07.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104041+0000","last_change":"2024-09-17T03:55:10.097916+0000","last_active":"2024-09-17T03:55:11.104041+0000","last_peered":"2024-09-17T03:55:11.104041+0000","last_clean":"2024-09-17T03:55:11.104041+0000","last_became_active":"2024-09-17T03:55:10.097595+0000","last_became_peered":"2024-09-17T03:55:10.097595+0000","last_unstale":"2024-09-17T03:55:11.104041+0000","last_undegraded":"2024-09-17T03:55:11.104041+0000","last_fullsized":"2024-09-17T03:55:11.104041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423852+0000","last_change":"2024-09-17T03:55:09.093148+0000","last_active":"2024-09-17T03:55:11.423852+0000","last_peered":"2024-09-17T03:55:11.423852+0000","last_clean":"2024-09-17T03:55:11.423852+0000","last_became_active":"2024-09-17T03:55:09.092860+0000","last_became_peered":"2024-09-17T03:55:09.092860+0000","last_unstale":"2024-09-17T03:55:11.423852+0000","last_undegraded":"2024-09-17T03:55:11.423852+0000","last_fullsized":"2024-09-17T03:55:11.423852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:11.814702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104560+0000","last_change":"2024-09-17T03:55:10.102328+0000","last_active":"2024-09-17T03:55:11.104560+0000","last_peered":"2024-09-17T03:55:11.104560+0000","last_clean":"2024-09-17T03:55:11.104560+0000","last_became_active":"2024-09-17T03:55:10.101937+0000","last_became_peered":"2024-09-17T03:55:10.101937+0000","last_unstale":"2024-09-17T03:55:11.104560+0000","last_undegraded":"2024-09-17T03:55:11.104560+0000","last_fullsized":"2024-09-17T03:55:11.104560+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:53.259242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112624+0000","last_change":"2024-09-17T03:55:09.110979+0000","last_active":"2024-09-17T03:55:09.112624+0000","last_peered":"2024-09-17T03:55:09.112624+0000","last_clean":"2024-09-17T03:55:09.112624+0000","last_became_active":"2024-09-17T03:55:09.110640+0000","last_became_peered":"2024-09-17T03:55:09.110640+0000","last_unstale":"2024-09-17T03:55:09.112624+0000","last_undegraded":"2024-09-17T03:55:09.112624+0000","last_fullsized":"2024-09-17T03:55:09.112624+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:42.556858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426564+0000","last_change":"2024-09-17T03:55:10.105276+0000","last_active":"2024-09-17T03:55:11.426564+0000","last_peered":"2024-09-17T03:55:11.426564+0000","last_clean":"2024-09-17T03:55:11.426564+0000","last_became_active":"2024-09-17T03:55:10.104973+0000","last_became_peered":"2024-09-17T03:55:10.104973+0000","last_unstale":"2024-09-17T03:55:11.426564+0000","last_undegraded":"2024-09-17T03:55:11.426564+0000","last_fullsized":"2024-09-17T03:55:11.426564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:37.836511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.105305+0000","last_change":"2024-09-17T03:55:09.100586+0000","last_active":"2024-09-17T03:55:09.105305+0000","last_peered":"2024-09-17T03:55:09.105305+0000","last_clean":"2024-09-17T03:55:09.105305+0000","last_became_active":"2024-09-17T03:55:09.100303+0000","last_became_peered":"2024-09-17T03:55:09.100303+0000","last_unstale":"2024-09-17T03:55:09.105305+0000","last_undegraded":"2024-09-17T03:55:09.105305+0000","last_fullsized":"2024-09-17T03:55:09.105305+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:36.891586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424336+0000","last_change":"2024-09-17T03:55:09.101674+0000","last_active":"2024-09-17T03:55:11.424336+0000","last_peered":"2024-09-17T03:55:11.424336+0000","last_clean":"2024-09-17T03:55:11.424336+0000","last_became_active":"2024-09-17T03:55:09.101311+0000","last_became_peered":"2024-09-17T03:55:09.101311+0000","last_unstale":"2024-09-17T03:55:11.424336+0000","last_undegraded":"2024-09-17T03:55:11.424336+0000","last_fullsized":"2024-09-17T03:55:11.424336+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:43.324925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.105085+0000","last_change":"2024-09-17T03:55:10.100605+0000","last_active":"2024-09-17T03:55:11.105085+0000","last_peered":"2024-09-17T03:55:11.105085+0000","last_clean":"2024-09-17T03:55:11.105085+0000","last_became_active":"2024-09-17T03:55:10.100264+0000","last_became_peered":"2024-09-17T03:55:10.100264+0000","last_unstale":"2024-09-17T03:55:11.105085+0000","last_undegraded":"2024-09-17T03:55:11.105085+0000","last_fullsized":"2024-09-17T03:55:11.105085+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:31.420107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":21,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424538+0000","last_change":"2024-09-17T03:55:09.102173+0000","last_active":"2024-09-17T03:55:11.424538+0000","last_peered":"2024-09-17T03:55:11.424538+0000","last_clean":"2024-09-17T03:55:11.424538+0000","last_became_active":"2024-09-17T03:55:09.101451+0000","last_became_peered":"2024-09-17T03:55:09.101451+0000","last_unstale":"2024-09-17T03:55:11.424538+0000","last_undegraded":"2024-09-17T03:55:11.424538+0000","last_fullsized":"2024-09-17T03:55:11.424538+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:16.929007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085389+0000","last_change":"2024-09-17T03:55:09.085389+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085389+0000","last_undegraded":"2024-09-17T03:55:09.085389+0000","last_fullsized":"2024-09-17T03:55:09.085389+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.107231+0000","last_change":"2024-09-17T03:55:09.101519+0000","last_active":"2024-09-17T03:55:09.107231+0000","last_peered":"2024-09-17T03:55:09.107231+0000","last_clean":"2024-09-17T03:55:09.107231+0000","last_became_active":"2024-09-17T03:55:09.101211+0000","last_became_peered":"2024-09-17T03:55:09.101211+0000","last_unstale":"2024-09-17T03:55:09.107231+0000","last_undegraded":"2024-09-17T03:55:09.107231+0000","last_fullsized":"2024-09-17T03:55:09.107231+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:29.559882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.088070+0000","last_change":"2024-09-17T03:55:09.088070+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.088070+0000","last_undegraded":"2024-09-17T03:55:09.088070+0000","last_fullsized":"2024-09-17T03:55:09.088070+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424612+0000","last_change":"2024-09-17T03:55:09.103112+0000","last_active":"2024-09-17T03:55:11.424612+0000","last_peered":"2024-09-17T03:55:11.424612+0000","last_clean":"2024-09-17T03:55:11.424612+0000","last_became_active":"2024-09-17T03:55:09.102182+0000","last_became_peered":"2024-09-17T03:55:09.102182+0000","last_unstale":"2024-09-17T03:55:11.424612+0000","last_undegraded":"2024-09-17T03:55:11.424612+0000","last_fullsized":"2024-09-17T03:55:11.424612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:53:55.298078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084668+0000","last_change":"2024-09-17T03:55:09.084668+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084668+0000","last_undegraded":"2024-09-17T03:55:09.084668+0000","last_fullsized":"2024-09-17T03:55:09.084668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112617+0000","last_change":"2024-09-17T03:55:09.110547+0000","last_active":"2024-09-17T03:55:09.112617+0000","last_peered":"2024-09-17T03:55:09.112617+0000","last_clean":"2024-09-17T03:55:09.112617+0000","last_became_active":"2024-09-17T03:55:09.109930+0000","last_became_peered":"2024-09-17T03:55:09.109930+0000","last_unstale":"2024-09-17T03:55:09.112617+0000","last_undegraded":"2024-09-17T03:55:09.112617+0000","last_fullsized":"2024-09-17T03:55:09.112617+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:42.335248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.095435+0000","last_change":"2024-09-17T03:55:09.095435+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.095435+0000","last_undegraded":"2024-09-17T03:55:09.095435+0000","last_fullsized":"2024-09-17T03:55:09.095435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424999+0000","last_change":"2024-09-17T03:55:09.102865+0000","last_active":"2024-09-17T03:55:11.424999+0000","last_peered":"2024-09-17T03:55:11.424999+0000","last_clean":"2024-09-17T03:55:11.424999+0000","last_became_active":"2024-09-17T03:55:09.100408+0000","last_became_peered":"2024-09-17T03:55:09.100408+0000","last_unstale":"2024-09-17T03:55:11.424999+0000","last_undegraded":"2024-09-17T03:55:11.424999+0000","last_fullsized":"2024-09-17T03:55:11.424999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:29.269455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085989+0000","last_change":"2024-09-17T03:55:09.085989+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085989+0000","last_undegraded":"2024-09-17T03:55:09.085989+0000","last_fullsized":"2024-09-17T03:55:09.085989+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109499+0000","last_change":"2024-09-17T03:55:09.106143+0000","last_active":"2024-09-17T03:55:09.109499+0000","last_peered":"2024-09-17T03:55:09.109499+0000","last_clean":"2024-09-17T03:55:09.109499+0000","last_became_active":"2024-09-17T03:55:09.105539+0000","last_became_peered":"2024-09-17T03:55:09.105539+0000","last_unstale":"2024-09-17T03:55:09.109499+0000","last_undegraded":"2024-09-17T03:55:09.109499+0000","last_fullsized":"2024-09-17T03:55:09.109499+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:18.027915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425055+0000","last_change":"2024-09-17T03:55:10.118502+0000","last_active":"2024-09-17T03:55:11.425055+0000","last_peered":"2024-09-17T03:55:11.425055+0000","last_clean":"2024-09-17T03:55:11.425055+0000","last_became_active":"2024-09-17T03:55:10.118276+0000","last_became_peered":"2024-09-17T03:55:10.118276+0000","last_unstale":"2024-09-17T03:55:11.425055+0000","last_undegraded":"2024-09-17T03:55:11.425055+0000","last_fullsized":"2024-09-17T03:55:11.425055+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:23.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425392+0000","last_change":"2024-09-17T03:55:09.099409+0000","last_active":"2024-09-17T03:55:11.425392+0000","last_peered":"2024-09-17T03:55:11.425392+0000","last_clean":"2024-09-17T03:55:11.425392+0000","last_became_active":"2024-09-17T03:55:09.097894+0000","last_became_peered":"2024-09-17T03:55:09.097894+0000","last_unstale":"2024-09-17T03:55:11.425392+0000","last_undegraded":"2024-09-17T03:55:11.425392+0000","last_fullsized":"2024-09-17T03:55:11.425392+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:30.592901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426005+0000","last_change":"2024-09-17T03:55:10.101315+0000","last_active":"2024-09-17T03:55:11.426005+0000","last_peered":"2024-09-17T03:55:11.426005+0000","last_clean":"2024-09-17T03:55:11.426005+0000","last_became_active":"2024-09-17T03:55:10.100936+0000","last_became_peered":"2024-09-17T03:55:10.100936+0000","last_unstale":"2024-09-17T03:55:11.426005+0000","last_undegraded":"2024-09-17T03:55:11.426005+0000","last_fullsized":"2024-09-17T03:55:11.426005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:51.665179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104091+0000","last_change":"2024-09-17T03:55:09.095974+0000","last_active":"2024-09-17T03:55:11.104091+0000","last_peered":"2024-09-17T03:55:11.104091+0000","last_clean":"2024-09-17T03:55:11.104091+0000","last_became_active":"2024-09-17T03:55:09.095632+0000","last_became_peered":"2024-09-17T03:55:09.095632+0000","last_unstale":"2024-09-17T03:55:11.104091+0000","last_undegraded":"2024-09-17T03:55:11.104091+0000","last_fullsized":"2024-09-17T03:55:11.104091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.401993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424900+0000","last_change":"2024-09-17T03:55:10.104348+0000","last_active":"2024-09-17T03:55:11.424900+0000","last_peered":"2024-09-17T03:55:11.424900+0000","last_clean":"2024-09-17T03:55:11.424900+0000","last_became_active":"2024-09-17T03:55:10.104019+0000","last_became_peered":"2024-09-17T03:55:10.104019+0000","last_unstale":"2024-09-17T03:55:11.424900+0000","last_undegraded":"2024-09-17T03:55:11.424900+0000","last_fullsized":"2024-09-17T03:55:11.424900+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.618440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.108346+0000","last_change":"2024-09-17T03:55:09.107009+0000","last_active":"2024-09-17T03:55:09.108346+0000","last_peered":"2024-09-17T03:55:09.108346+0000","last_clean":"2024-09-17T03:55:09.108346+0000","last_became_active":"2024-09-17T03:55:09.106706+0000","last_became_peered":"2024-09-17T03:55:09.106706+0000","last_unstale":"2024-09-17T03:55:09.108346+0000","last_undegraded":"2024-09-17T03:55:09.108346+0000","last_fullsized":"2024-09-17T03:55:09.108346+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:13.263489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424019+0000","last_change":"2024-09-17T03:55:10.101801+0000","last_active":"2024-09-17T03:55:11.424019+0000","last_peered":"2024-09-17T03:55:11.424019+0000","last_clean":"2024-09-17T03:55:11.424019+0000","last_became_active":"2024-09-17T03:55:10.101240+0000","last_became_peered":"2024-09-17T03:55:10.101240+0000","last_unstale":"2024-09-17T03:55:11.424019+0000","last_undegraded":"2024-09-17T03:55:11.424019+0000","last_fullsized":"2024-09-17T03:55:11.424019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:08.177233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109573+0000","last_change":"2024-09-17T03:55:09.105469+0000","last_active":"2024-09-17T03:55:09.109573+0000","last_peered":"2024-09-17T03:55:09.109573+0000","last_clean":"2024-09-17T03:55:09.109573+0000","last_became_active":"2024-09-17T03:55:09.105074+0000","last_became_peered":"2024-09-17T03:55:09.105074+0000","last_unstale":"2024-09-17T03:55:09.109573+0000","last_undegraded":"2024-09-17T03:55:09.109573+0000","last_fullsized":"2024-09-17T03:55:09.109573+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:37.786492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425201+0000","last_change":"2024-09-17T03:55:10.110424+0000","last_active":"2024-09-17T03:55:11.425201+0000","last_peered":"2024-09-17T03:55:11.425201+0000","last_clean":"2024-09-17T03:55:11.425201+0000","last_became_active":"2024-09-17T03:55:10.110057+0000","last_became_peered":"2024-09-17T03:55:10.110057+0000","last_unstale":"2024-09-17T03:55:11.425201+0000","last_undegraded":"2024-09-17T03:55:11.425201+0000","last_fullsized":"2024-09-17T03:55:11.425201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:40.073222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.111837+0000","last_change":"2024-09-17T03:55:09.109486+0000","last_active":"2024-09-17T03:55:09.111837+0000","last_peered":"2024-09-17T03:55:09.111837+0000","last_clean":"2024-09-17T03:55:09.111837+0000","last_became_active":"2024-09-17T03:55:09.108902+0000","last_became_peered":"2024-09-17T03:55:09.108902+0000","last_unstale":"2024-09-17T03:55:09.111837+0000","last_undegraded":"2024-09-17T03:55:09.111837+0000","last_fullsized":"2024-09-17T03:55:09.111837+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:45.582374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084094+0000","last_change":"2024-09-17T03:55:09.084094+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084094+0000","last_undegraded":"2024-09-17T03:55:09.084094+0000","last_fullsized":"2024-09-17T03:55:09.084094+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423957+0000","last_change":"2024-09-17T03:55:09.089727+0000","last_active":"2024-09-17T03:55:11.423957+0000","last_peered":"2024-09-17T03:55:11.423957+0000","last_clean":"2024-09-17T03:55:11.423957+0000","last_became_active":"2024-09-17T03:55:09.089416+0000","last_became_peered":"2024-09-17T03:55:09.089416+0000","last_unstale":"2024-09-17T03:55:11.423957+0000","last_undegraded":"2024-09-17T03:55:11.423957+0000","last_fullsized":"2024-09-17T03:55:11.423957+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:19.704845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426516+0000","last_change":"2024-09-17T03:55:10.106649+0000","last_active":"2024-09-17T03:55:11.426516+0000","last_peered":"2024-09-17T03:55:11.426516+0000","last_clean":"2024-09-17T03:55:11.426516+0000","last_became_active":"2024-09-17T03:55:10.106347+0000","last_became_peered":"2024-09-17T03:55:10.106347+0000","last_unstale":"2024-09-17T03:55:11.426516+0000","last_undegraded":"2024-09-17T03:55:11.426516+0000","last_fullsized":"2024-09-17T03:55:11.426516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:21.848148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.110091+0000","last_change":"2024-09-17T03:55:09.106794+0000","last_active":"2024-09-17T03:55:09.110091+0000","last_peered":"2024-09-17T03:55:09.110091+0000","last_clean":"2024-09-17T03:55:09.110091+0000","last_became_active":"2024-09-17T03:55:09.106461+0000","last_became_peered":"2024-09-17T03:55:09.106461+0000","last_unstale":"2024-09-17T03:55:09.110091+0000","last_undegraded":"2024-09-17T03:55:09.110091+0000","last_fullsized":"2024-09-17T03:55:09.110091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:22.706399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424964+0000","last_change":"2024-09-17T03:55:10.111117+0000","last_active":"2024-09-17T03:55:11.424964+0000","last_peered":"2024-09-17T03:55:11.424964+0000","last_clean":"2024-09-17T03:55:11.424964+0000","last_became_active":"2024-09-17T03:55:10.110769+0000","last_became_peered":"2024-09-17T03:55:10.110769+0000","last_unstale":"2024-09-17T03:55:11.424964+0000","last_undegraded":"2024-09-17T03:55:11.424964+0000","last_fullsized":"2024-09-17T03:55:11.424964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:34.220767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426576+0000","last_change":"2024-09-17T03:55:09.091992+0000","last_active":"2024-09-17T03:55:11.426576+0000","last_peered":"2024-09-17T03:55:11.426576+0000","last_clean":"2024-09-17T03:55:11.426576+0000","last_became_active":"2024-09-17T03:55:09.091655+0000","last_became_peered":"2024-09-17T03:55:09.091655+0000","last_unstale":"2024-09-17T03:55:11.426576+0000","last_undegraded":"2024-09-17T03:55:11.426576+0000","last_fullsized":"2024-09-17T03:55:11.426576+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:14.155619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424655+0000","last_change":"2024-09-17T03:55:10.125141+0000","last_active":"2024-09-17T03:55:11.424655+0000","last_peered":"2024-09-17T03:55:11.424655+0000","last_clean":"2024-09-17T03:55:11.424655+0000","last_became_active":"2024-09-17T03:55:10.124839+0000","last_became_peered":"2024-09-17T03:55:10.124839+0000","last_unstale":"2024-09-17T03:55:11.424655+0000","last_undegraded":"2024-09-17T03:55:11.424655+0000","last_fullsized":"2024-09-17T03:55:11.424655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:49.605272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424463+0000","last_change":"2024-09-17T03:55:09.092456+0000","last_active":"2024-09-17T03:55:11.424463+0000","last_peered":"2024-09-17T03:55:11.424463+0000","last_clean":"2024-09-17T03:55:11.424463+0000","last_became_active":"2024-09-17T03:55:09.092133+0000","last_became_peered":"2024-09-17T03:55:09.092133+0000","last_unstale":"2024-09-17T03:55:11.424463+0000","last_undegraded":"2024-09-17T03:55:11.424463+0000","last_fullsized":"2024-09-17T03:55:11.424463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:29.592608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425399+0000","last_change":"2024-09-17T03:55:10.105263+0000","last_active":"2024-09-17T03:55:11.425399+0000","last_peered":"2024-09-17T03:55:11.425399+0000","last_clean":"2024-09-17T03:55:11.425399+0000","last_became_active":"2024-09-17T03:55:10.104878+0000","last_became_peered":"2024-09-17T03:55:10.104878+0000","last_unstale":"2024-09-17T03:55:11.425399+0000","last_undegraded":"2024-09-17T03:55:11.425399+0000","last_fullsized":"2024-09-17T03:55:11.425399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:29.686662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.104632+0000","last_change":"2024-09-17T03:55:09.101334+0000","last_active":"2024-09-17T03:55:09.104632+0000","last_peered":"2024-09-17T03:55:09.104632+0000","last_clean":"2024-09-17T03:55:09.104632+0000","last_became_active":"2024-09-17T03:55:09.101016+0000","last_became_peered":"2024-09-17T03:55:09.101016+0000","last_unstale":"2024-09-17T03:55:09.104632+0000","last_undegraded":"2024-09-17T03:55:09.104632+0000","last_fullsized":"2024-09-17T03:55:09.104632+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:30.035885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425931+0000","last_change":"2024-09-17T03:55:10.097057+0000","last_active":"2024-09-17T03:55:11.425931+0000","last_peered":"2024-09-17T03:55:11.425931+0000","last_clean":"2024-09-17T03:55:11.425931+0000","last_became_active":"2024-09-17T03:55:10.096210+0000","last_became_peered":"2024-09-17T03:55:10.096210+0000","last_unstale":"2024-09-17T03:55:11.425931+0000","last_undegraded":"2024-09-17T03:55:11.425931+0000","last_fullsized":"2024-09-17T03:55:11.425931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:04.260938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104187+0000","last_change":"2024-09-17T03:55:09.099774+0000","last_active":"2024-09-17T03:55:11.104187+0000","last_peered":"2024-09-17T03:55:11.104187+0000","last_clean":"2024-09-17T03:55:11.104187+0000","last_became_active":"2024-09-17T03:55:09.099422+0000","last_became_peered":"2024-09-17T03:55:09.099422+0000","last_unstale":"2024-09-17T03:55:11.104187+0000","last_undegraded":"2024-09-17T03:55:11.104187+0000","last_fullsized":"2024-09-17T03:55:11.104187+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:56.088001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426331+0000","last_change":"2024-09-17T03:55:10.112375+0000","last_active":"2024-09-17T03:55:11.426331+0000","last_peered":"2024-09-17T03:55:11.426331+0000","last_clean":"2024-09-17T03:55:11.426331+0000","last_became_active":"2024-09-17T03:55:10.112033+0000","last_became_peered":"2024-09-17T03:55:10.112033+0000","last_unstale":"2024-09-17T03:55:11.426331+0000","last_undegraded":"2024-09-17T03:55:11.426331+0000","last_fullsized":"2024-09-17T03:55:11.426331+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426160+0000","last_change":"2024-09-17T03:55:09.103012+0000","last_active":"2024-09-17T03:55:11.426160+0000","last_peered":"2024-09-17T03:55:11.426160+0000","last_clean":"2024-09-17T03:55:11.426160+0000","last_became_active":"2024-09-17T03:55:09.102701+0000","last_became_peered":"2024-09-17T03:55:09.102701+0000","last_unstale":"2024-09-17T03:55:11.426160+0000","last_undegraded":"2024-09-17T03:55:11.426160+0000","last_fullsized":"2024-09-17T03:55:11.426160+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:02.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425468+0000","last_change":"2024-09-17T03:55:10.113187+0000","last_active":"2024-09-17T03:55:11.425468+0000","last_peered":"2024-09-17T03:55:11.425468+0000","last_clean":"2024-09-17T03:55:11.425468+0000","last_became_active":"2024-09-17T03:55:10.112812+0000","last_became_peered":"2024-09-17T03:55:10.112812+0000","last_unstale":"2024-09-17T03:55:11.425468+0000","last_undegraded":"2024-09-17T03:55:11.425468+0000","last_fullsized":"2024-09-17T03:55:11.425468+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:18.693960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112065+0000","last_change":"2024-09-17T03:55:09.108670+0000","last_active":"2024-09-17T03:55:09.112065+0000","last_peered":"2024-09-17T03:55:09.112065+0000","last_clean":"2024-09-17T03:55:09.112065+0000","last_became_active":"2024-09-17T03:55:09.106248+0000","last_became_peered":"2024-09-17T03:55:09.106248+0000","last_unstale":"2024-09-17T03:55:09.112065+0000","last_undegraded":"2024-09-17T03:55:09.112065+0000","last_fullsized":"2024-09-17T03:55:09.112065+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:54.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423930+0000","last_change":"2024-09-17T03:55:10.126012+0000","last_active":"2024-09-17T03:55:11.423930+0000","last_peered":"2024-09-17T03:55:11.423930+0000","last_clean":"2024-09-17T03:55:11.423930+0000","last_became_active":"2024-09-17T03:55:10.125592+0000","last_became_peered":"2024-09-17T03:55:10.125592+0000","last_unstale":"2024-09-17T03:55:11.423930+0000","last_undegraded":"2024-09-17T03:55:11.423930+0000","last_fullsized":"2024-09-17T03:55:11.423930+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:17.583329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424054+0000","last_change":"2024-09-17T03:55:09.096029+0000","last_active":"2024-09-17T03:55:11.424054+0000","last_peered":"2024-09-17T03:55:11.424054+0000","last_clean":"2024-09-17T03:55:11.424054+0000","last_became_active":"2024-09-17T03:55:09.091782+0000","last_became_peered":"2024-09-17T03:55:09.091782+0000","last_unstale":"2024-09-17T03:55:11.424054+0000","last_undegraded":"2024-09-17T03:55:11.424054+0000","last_fullsized":"2024-09-17T03:55:11.424054+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:17.583743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085271+0000","last_change":"2024-09-17T03:55:09.085271+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085271+0000","last_undegraded":"2024-09-17T03:55:09.085271+0000","last_fullsized":"2024-09-17T03:55:09.085271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104276+0000","last_change":"2024-09-17T03:55:09.096214+0000","last_active":"2024-09-17T03:55:11.104276+0000","last_peered":"2024-09-17T03:55:11.104276+0000","last_clean":"2024-09-17T03:55:11.104276+0000","last_became_active":"2024-09-17T03:55:09.095625+0000","last_became_peered":"2024-09-17T03:55:09.095625+0000","last_unstale":"2024-09-17T03:55:11.104276+0000","last_undegraded":"2024-09-17T03:55:11.104276+0000","last_fullsized":"2024-09-17T03:55:11.104276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:12.569897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":265,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425604+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:55:11.425604+0000","last_peered":"2024-09-17T03:55:11.425604+0000","last_clean":"2024-09-17T03:55:11.425604+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:55:11.425604+0000","last_undegraded":"2024-09-17T03:55:11.425604+0000","last_fullsized":"2024-09-17T03:55:11.425604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1157630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":567250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509511,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1327104,"data_stored":566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542217,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":1158738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:55:15.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:55:15.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 -- 172.21.15.129:0/3969760950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140ce170 msgr2=0x7f94140cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 --2- 172.21.15.129:0/3969760950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140ce170 0x7f94140cbbf0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f9414072360 tx=0x7f940802a560 comp rx=0 tx=0).stop 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 -- 172.21.15.129:0/3969760950 shutdown_connections 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 --2- 172.21.15.129:0/3969760950 >> v2:172.21.15.181:3300/0 conn(0x7f94140b2950 0x7f94140b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 --2- 172.21.15.129:0/3969760950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140ce170 0x7f94140cbbf0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 -- 172.21.15.129:0/3969760950 >> 172.21.15.129:0/3969760950 conn(0x7f941406d410 msgr2=0x7f94140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 -- 172.21.15.129:0/3969760950 shutdown_connections 2024-09-17T03:55:15.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.451+0000 7f941c000640 1 -- 172.21.15.129:0/3969760950 wait complete. 2024-09-17T03:55:15.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.453+0000 7f941c000640 1 Processor -- start 2024-09-17T03:55:15.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.454+0000 7f941c000640 1 -- start start 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 0x7f941409b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 0x7f941409a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9414070850 con 0x7f94140b39c0 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94140706d0 con 0x7f94140b2950 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f941c000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f941406f5e0 con 0x7f94140ce170 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f9418c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f9418c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53152/0 (socket says 172.21.15.129:53152) 2024-09-17T03:55:15.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.455+0000 7f9418c00640 1 -- 172.21.15.129:0/2978859359 learned_addr learned my addr 172.21.15.129:0/2978859359 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9419600640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 0x7f941409b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f941a000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 0x7f941409a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 -- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 msgr2=0x7f941409a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 0x7f941409a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 -- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 msgr2=0x7f941409b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 0x7f941409b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 -- 172.21.15.129:0/2978859359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941409b990 con 0x7f94140b39c0 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9419600640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 0x7f941409b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f941a000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 0x7f941409a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:15.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9418c00640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f940000d6d0 tx=0x7f940000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400017070 con 0x7f94140b39c0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9400005330 con 0x7f94140b39c0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9400009820 con 0x7f94140b39c0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941409d330 con 0x7f94140b39c0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.456+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94141f9380 con 0x7f94140b39c0 2024-09-17T03:55:15.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.457+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94141f8e10 con 0x7f94140b39c0 2024-09-17T03:55:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.459+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9400009bd0 con 0x7f94140b39c0 2024-09-17T03:55:15.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.460+0000 7f9411600640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 0x7f93f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.460+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f940009b140 con 0x7f94140b39c0 2024-09-17T03:55:15.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.460+0000 7f9419600640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 0x7f93f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.461+0000 7f9419600640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 0x7f93f407b0a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9408033000 tx=0x7f9408004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:15.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.464+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f941409b990 con 0x7f94140b39c0 2024-09-17T03:55:15.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.646+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f941406fab0 con 0x7f93f4078b90 2024-09-17T03:55:15.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.651+0000 7f9411600640 1 -- 172.21.15.129:0/2978859359 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250957 (secure 0 0 0) 0x7f941406fab0 con 0x7f93f4078b90 2024-09-17T03:55:15.655 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:15.656 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:55:15.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 msgr2=0x7f93f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 0x7f93f407b0a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9408033000 tx=0x7f9408004730 comp rx=0 tx=0).stop 2024-09-17T03:55:15.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 msgr2=0x7f941409cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f940000d6d0 tx=0x7f940000db90 comp rx=0 tx=0).stop 2024-09-17T03:55:15.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 shutdown_connections 2024-09-17T03:55:15.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93f4078b90 0x7f93f407b0a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94140ce170 0x7f941409a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94140b39c0 0x7f941409cdf0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 --2- 172.21.15.129:0/2978859359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94140b2950 0x7f941409b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.655+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 >> 172.21.15.129:0/2978859359 conn(0x7f941406d410 msgr2=0x7f94140cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:15.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.656+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 shutdown_connections 2024-09-17T03:55:15.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.656+0000 7f941c000640 1 -- 172.21.15.129:0/2978859359 wait complete. 2024-09-17T03:55:15.677 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-17T03:55:14.387295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":118,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302584,"kb_used_data":11428,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672136,"statfs":{"total":773094113280,"available":772784267264,"internally_reserved":0,"allocated":11702272,"data_stored":5658868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001744"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424841+0000","last_change":"2024-09-17T03:55:10.112479+0000","last_active":"2024-09-17T03:55:11.424841+0000","last_peered":"2024-09-17T03:55:11.424841+0000","last_clean":"2024-09-17T03:55:11.424841+0000","last_became_active":"2024-09-17T03:55:10.112184+0000","last_became_peered":"2024-09-17T03:55:10.112184+0000","last_unstale":"2024-09-17T03:55:11.424841+0000","last_undegraded":"2024-09-17T03:55:11.424841+0000","last_fullsized":"2024-09-17T03:55:11.424841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:12.328355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425890+0000","last_change":"2024-09-17T03:55:10.123473+0000","last_active":"2024-09-17T03:55:11.425890+0000","last_peered":"2024-09-17T03:55:11.425890+0000","last_clean":"2024-09-17T03:55:11.425890+0000","last_became_active":"2024-09-17T03:55:10.123217+0000","last_became_peered":"2024-09-17T03:55:10.123217+0000","last_unstale":"2024-09-17T03:55:11.425890+0000","last_undegraded":"2024-09-17T03:55:11.425890+0000","last_fullsized":"2024-09-17T03:55:11.425890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:50.947437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425736+0000","last_change":"2024-09-17T03:55:10.103667+0000","last_active":"2024-09-17T03:55:11.425736+0000","last_peered":"2024-09-17T03:55:11.425736+0000","last_clean":"2024-09-17T03:55:11.425736+0000","last_became_active":"2024-09-17T03:55:10.103336+0000","last_became_peered":"2024-09-17T03:55:10.103336+0000","last_unstale":"2024-09-17T03:55:11.425736+0000","last_undegraded":"2024-09-17T03:55:11.425736+0000","last_fullsized":"2024-09-17T03:55:11.425736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:33.517751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424369+0000","last_change":"2024-09-17T03:55:10.109342+0000","last_active":"2024-09-17T03:55:11.424369+0000","last_peered":"2024-09-17T03:55:11.424369+0000","last_clean":"2024-09-17T03:55:11.424369+0000","last_became_active":"2024-09-17T03:55:10.109013+0000","last_became_peered":"2024-09-17T03:55:10.109013+0000","last_unstale":"2024-09-17T03:55:11.424369+0000","last_undegraded":"2024-09-17T03:55:11.424369+0000","last_fullsized":"2024-09-17T03:55:11.424369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:01.880305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.093607+0000","last_change":"2024-09-17T03:55:09.093607+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.093607+0000","last_undegraded":"2024-09-17T03:55:09.093607+0000","last_fullsized":"2024-09-17T03:55:09.093607+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.090308+0000","last_change":"2024-09-17T03:55:09.090308+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.090308+0000","last_undegraded":"2024-09-17T03:55:09.090308+0000","last_fullsized":"2024-09-17T03:55:09.090308+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104179+0000","last_change":"2024-09-17T03:55:10.096583+0000","last_active":"2024-09-17T03:55:11.104179+0000","last_peered":"2024-09-17T03:55:11.104179+0000","last_clean":"2024-09-17T03:55:11.104179+0000","last_became_active":"2024-09-17T03:55:10.096289+0000","last_became_peered":"2024-09-17T03:55:10.096289+0000","last_unstale":"2024-09-17T03:55:11.104179+0000","last_undegraded":"2024-09-17T03:55:11.104179+0000","last_fullsized":"2024-09-17T03:55:11.104179+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424746+0000","last_change":"2024-09-17T03:55:10.116515+0000","last_active":"2024-09-17T03:55:11.424746+0000","last_peered":"2024-09-17T03:55:11.424746+0000","last_clean":"2024-09-17T03:55:11.424746+0000","last_became_active":"2024-09-17T03:55:10.116161+0000","last_became_peered":"2024-09-17T03:55:10.116161+0000","last_unstale":"2024-09-17T03:55:11.424746+0000","last_undegraded":"2024-09-17T03:55:11.424746+0000","last_fullsized":"2024-09-17T03:55:11.424746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.433598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424969+0000","last_change":"2024-09-17T03:55:10.113065+0000","last_active":"2024-09-17T03:55:11.424969+0000","last_peered":"2024-09-17T03:55:11.424969+0000","last_clean":"2024-09-17T03:55:11.424969+0000","last_became_active":"2024-09-17T03:55:10.111008+0000","last_became_peered":"2024-09-17T03:55:10.111008+0000","last_unstale":"2024-09-17T03:55:11.424969+0000","last_undegraded":"2024-09-17T03:55:11.424969+0000","last_fullsized":"2024-09-17T03:55:11.424969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:54.618141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424625+0000","last_change":"2024-09-17T03:55:10.105708+0000","last_active":"2024-09-17T03:55:11.424625+0000","last_peered":"2024-09-17T03:55:11.424625+0000","last_clean":"2024-09-17T03:55:11.424625+0000","last_became_active":"2024-09-17T03:55:10.105368+0000","last_became_peered":"2024-09-17T03:55:10.105368+0000","last_unstale":"2024-09-17T03:55:11.424625+0000","last_undegraded":"2024-09-17T03:55:11.424625+0000","last_fullsized":"2024-09-17T03:55:11.424625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:19.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425340+0000","last_change":"2024-09-17T03:55:10.107346+0000","last_active":"2024-09-17T03:55:11.425340+0000","last_peered":"2024-09-17T03:55:11.425340+0000","last_clean":"2024-09-17T03:55:11.425340+0000","last_became_active":"2024-09-17T03:55:10.106955+0000","last_became_peered":"2024-09-17T03:55:10.106955+0000","last_unstale":"2024-09-17T03:55:11.425340+0000","last_undegraded":"2024-09-17T03:55:11.425340+0000","last_fullsized":"2024-09-17T03:55:11.425340+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:29.542513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424410+0000","last_change":"2024-09-17T03:55:10.108582+0000","last_active":"2024-09-17T03:55:11.424410+0000","last_peered":"2024-09-17T03:55:11.424410+0000","last_clean":"2024-09-17T03:55:11.424410+0000","last_became_active":"2024-09-17T03:55:10.105982+0000","last_became_peered":"2024-09-17T03:55:10.105982+0000","last_unstale":"2024-09-17T03:55:11.424410+0000","last_undegraded":"2024-09-17T03:55:11.424410+0000","last_fullsized":"2024-09-17T03:55:11.424410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:31.031639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425062+0000","last_change":"2024-09-17T03:55:10.109256+0000","last_active":"2024-09-17T03:55:11.425062+0000","last_peered":"2024-09-17T03:55:11.425062+0000","last_clean":"2024-09-17T03:55:11.425062+0000","last_became_active":"2024-09-17T03:55:10.108832+0000","last_became_peered":"2024-09-17T03:55:10.108832+0000","last_unstale":"2024-09-17T03:55:11.425062+0000","last_undegraded":"2024-09-17T03:55:11.425062+0000","last_fullsized":"2024-09-17T03:55:11.425062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:16.522311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424801+0000","last_change":"2024-09-17T03:55:10.112919+0000","last_active":"2024-09-17T03:55:11.424801+0000","last_peered":"2024-09-17T03:55:11.424801+0000","last_clean":"2024-09-17T03:55:11.424801+0000","last_became_active":"2024-09-17T03:55:10.112583+0000","last_became_peered":"2024-09-17T03:55:10.112583+0000","last_unstale":"2024-09-17T03:55:11.424801+0000","last_undegraded":"2024-09-17T03:55:11.424801+0000","last_fullsized":"2024-09-17T03:55:11.424801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:27.443267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087537+0000","last_change":"2024-09-17T03:55:09.087537+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087537+0000","last_undegraded":"2024-09-17T03:55:09.087537+0000","last_fullsized":"2024-09-17T03:55:09.087537+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423770+0000","last_change":"2024-09-17T03:55:10.094106+0000","last_active":"2024-09-17T03:55:11.423770+0000","last_peered":"2024-09-17T03:55:11.423770+0000","last_clean":"2024-09-17T03:55:11.423770+0000","last_became_active":"2024-09-17T03:55:10.093763+0000","last_became_peered":"2024-09-17T03:55:10.093763+0000","last_unstale":"2024-09-17T03:55:11.423770+0000","last_undegraded":"2024-09-17T03:55:11.423770+0000","last_fullsized":"2024-09-17T03:55:11.423770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.835238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426449+0000","last_change":"2024-09-17T03:55:10.101103+0000","last_active":"2024-09-17T03:55:11.426449+0000","last_peered":"2024-09-17T03:55:11.426449+0000","last_clean":"2024-09-17T03:55:11.426449+0000","last_became_active":"2024-09-17T03:55:10.100775+0000","last_became_peered":"2024-09-17T03:55:10.100775+0000","last_unstale":"2024-09-17T03:55:11.426449+0000","last_undegraded":"2024-09-17T03:55:11.426449+0000","last_fullsized":"2024-09-17T03:55:11.426449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426439+0000","last_change":"2024-09-17T03:55:10.100108+0000","last_active":"2024-09-17T03:55:11.426439+0000","last_peered":"2024-09-17T03:55:11.426439+0000","last_clean":"2024-09-17T03:55:11.426439+0000","last_became_active":"2024-09-17T03:55:10.099786+0000","last_became_peered":"2024-09-17T03:55:10.099786+0000","last_unstale":"2024-09-17T03:55:11.426439+0000","last_undegraded":"2024-09-17T03:55:11.426439+0000","last_fullsized":"2024-09-17T03:55:11.426439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:00.641678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425612+0000","last_change":"2024-09-17T03:55:10.109163+0000","last_active":"2024-09-17T03:55:11.425612+0000","last_peered":"2024-09-17T03:55:11.425612+0000","last_clean":"2024-09-17T03:55:11.425612+0000","last_became_active":"2024-09-17T03:55:10.108810+0000","last_became_peered":"2024-09-17T03:55:10.108810+0000","last_unstale":"2024-09-17T03:55:11.425612+0000","last_undegraded":"2024-09-17T03:55:11.425612+0000","last_fullsized":"2024-09-17T03:55:11.425612+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:32.706728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426258+0000","last_change":"2024-09-17T03:55:10.102425+0000","last_active":"2024-09-17T03:55:11.426258+0000","last_peered":"2024-09-17T03:55:11.426258+0000","last_clean":"2024-09-17T03:55:11.426258+0000","last_became_active":"2024-09-17T03:55:10.102086+0000","last_became_peered":"2024-09-17T03:55:10.102086+0000","last_unstale":"2024-09-17T03:55:11.426258+0000","last_undegraded":"2024-09-17T03:55:11.426258+0000","last_fullsized":"2024-09-17T03:55:11.426258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426586+0000","last_change":"2024-09-17T03:55:10.100194+0000","last_active":"2024-09-17T03:55:11.426586+0000","last_peered":"2024-09-17T03:55:11.426586+0000","last_clean":"2024-09-17T03:55:11.426586+0000","last_became_active":"2024-09-17T03:55:10.099857+0000","last_became_peered":"2024-09-17T03:55:10.099857+0000","last_unstale":"2024-09-17T03:55:11.426586+0000","last_undegraded":"2024-09-17T03:55:11.426586+0000","last_fullsized":"2024-09-17T03:55:11.426586+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085847+0000","last_change":"2024-09-17T03:55:09.085847+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085847+0000","last_undegraded":"2024-09-17T03:55:09.085847+0000","last_fullsized":"2024-09-17T03:55:09.085847+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425299+0000","last_change":"2024-09-17T03:55:10.097618+0000","last_active":"2024-09-17T03:55:11.425299+0000","last_peered":"2024-09-17T03:55:11.425299+0000","last_clean":"2024-09-17T03:55:11.425299+0000","last_became_active":"2024-09-17T03:55:10.097328+0000","last_became_peered":"2024-09-17T03:55:10.097328+0000","last_unstale":"2024-09-17T03:55:11.425299+0000","last_undegraded":"2024-09-17T03:55:11.425299+0000","last_fullsized":"2024-09-17T03:55:11.425299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:23.303493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085796+0000","last_change":"2024-09-17T03:55:09.085796+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085796+0000","last_undegraded":"2024-09-17T03:55:09.085796+0000","last_fullsized":"2024-09-17T03:55:09.085796+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426752+0000","last_change":"2024-09-17T03:55:10.098678+0000","last_active":"2024-09-17T03:55:11.426752+0000","last_peered":"2024-09-17T03:55:11.426752+0000","last_clean":"2024-09-17T03:55:11.426752+0000","last_became_active":"2024-09-17T03:55:10.098318+0000","last_became_peered":"2024-09-17T03:55:10.098318+0000","last_unstale":"2024-09-17T03:55:11.426752+0000","last_undegraded":"2024-09-17T03:55:11.426752+0000","last_fullsized":"2024-09-17T03:55:11.426752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424684+0000","last_change":"2024-09-17T03:55:10.102840+0000","last_active":"2024-09-17T03:55:11.424684+0000","last_peered":"2024-09-17T03:55:11.424684+0000","last_clean":"2024-09-17T03:55:11.424684+0000","last_became_active":"2024-09-17T03:55:10.100850+0000","last_became_peered":"2024-09-17T03:55:10.100850+0000","last_unstale":"2024-09-17T03:55:11.424684+0000","last_undegraded":"2024-09-17T03:55:11.424684+0000","last_fullsized":"2024-09-17T03:55:11.424684+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:17.813222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.087802+0000","last_change":"2024-09-17T03:55:09.087802+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.087802+0000","last_undegraded":"2024-09-17T03:55:09.087802+0000","last_fullsized":"2024-09-17T03:55:09.087802+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089564+0000","last_change":"2024-09-17T03:55:09.089564+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089564+0000","last_undegraded":"2024-09-17T03:55:09.089564+0000","last_fullsized":"2024-09-17T03:55:09.089564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425371+0000","last_change":"2024-09-17T03:55:10.103912+0000","last_active":"2024-09-17T03:55:11.425371+0000","last_peered":"2024-09-17T03:55:11.425371+0000","last_clean":"2024-09-17T03:55:11.425371+0000","last_became_active":"2024-09-17T03:55:10.103569+0000","last_became_peered":"2024-09-17T03:55:10.103569+0000","last_unstale":"2024-09-17T03:55:11.425371+0000","last_undegraded":"2024-09-17T03:55:11.425371+0000","last_fullsized":"2024-09-17T03:55:11.425371+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:36.412867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104245+0000","last_change":"2024-09-17T03:55:10.111756+0000","last_active":"2024-09-17T03:55:11.104245+0000","last_peered":"2024-09-17T03:55:11.104245+0000","last_clean":"2024-09-17T03:55:11.104245+0000","last_became_active":"2024-09-17T03:55:10.111350+0000","last_became_peered":"2024-09-17T03:55:10.111350+0000","last_unstale":"2024-09-17T03:55:11.104245+0000","last_undegraded":"2024-09-17T03:55:11.104245+0000","last_fullsized":"2024-09-17T03:55:11.104245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423827+0000","last_change":"2024-09-17T03:55:10.110031+0000","last_active":"2024-09-17T03:55:11.423827+0000","last_peered":"2024-09-17T03:55:11.423827+0000","last_clean":"2024-09-17T03:55:11.423827+0000","last_became_active":"2024-09-17T03:55:10.106150+0000","last_became_peered":"2024-09-17T03:55:10.106150+0000","last_unstale":"2024-09-17T03:55:11.423827+0000","last_undegraded":"2024-09-17T03:55:11.423827+0000","last_fullsized":"2024-09-17T03:55:11.423827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:15.416205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085837+0000","last_change":"2024-09-17T03:55:09.085837+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085837+0000","last_undegraded":"2024-09-17T03:55:09.085837+0000","last_fullsized":"2024-09-17T03:55:09.085837+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":19,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104474+0000","last_change":"2024-09-17T03:55:09.098888+0000","last_active":"2024-09-17T03:55:11.104474+0000","last_peered":"2024-09-17T03:55:11.104474+0000","last_clean":"2024-09-17T03:55:11.104474+0000","last_became_active":"2024-09-17T03:55:09.098488+0000","last_became_peered":"2024-09-17T03:55:09.098488+0000","last_unstale":"2024-09-17T03:55:11.104474+0000","last_undegraded":"2024-09-17T03:55:11.104474+0000","last_fullsized":"2024-09-17T03:55:11.104474+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:43.529970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426303+0000","last_change":"2024-09-17T03:55:10.098038+0000","last_active":"2024-09-17T03:55:11.426303+0000","last_peered":"2024-09-17T03:55:11.426303+0000","last_clean":"2024-09-17T03:55:11.426303+0000","last_became_active":"2024-09-17T03:55:10.097706+0000","last_became_peered":"2024-09-17T03:55:10.097706+0000","last_unstale":"2024-09-17T03:55:11.426303+0000","last_undegraded":"2024-09-17T03:55:11.426303+0000","last_fullsized":"2024-09-17T03:55:11.426303+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:48.538101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424685+0000","last_change":"2024-09-17T03:55:09.102970+0000","last_active":"2024-09-17T03:55:11.424685+0000","last_peered":"2024-09-17T03:55:11.424685+0000","last_clean":"2024-09-17T03:55:11.424685+0000","last_became_active":"2024-09-17T03:55:09.102558+0000","last_became_peered":"2024-09-17T03:55:09.102558+0000","last_unstale":"2024-09-17T03:55:11.424685+0000","last_undegraded":"2024-09-17T03:55:11.424685+0000","last_fullsized":"2024-09-17T03:55:11.424685+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:26.840025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084952+0000","last_change":"2024-09-17T03:55:09.084952+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084952+0000","last_undegraded":"2024-09-17T03:55:09.084952+0000","last_fullsized":"2024-09-17T03:55:09.084952+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109001+0000","last_change":"2024-09-17T03:55:09.106007+0000","last_active":"2024-09-17T03:55:09.109001+0000","last_peered":"2024-09-17T03:55:09.109001+0000","last_clean":"2024-09-17T03:55:09.109001+0000","last_became_active":"2024-09-17T03:55:09.105510+0000","last_became_peered":"2024-09-17T03:55:09.105510+0000","last_unstale":"2024-09-17T03:55:09.109001+0000","last_undegraded":"2024-09-17T03:55:09.109001+0000","last_fullsized":"2024-09-17T03:55:09.109001+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:19.653445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425907+0000","last_change":"2024-09-17T03:55:10.111580+0000","last_active":"2024-09-17T03:55:11.425907+0000","last_peered":"2024-09-17T03:55:11.425907+0000","last_clean":"2024-09-17T03:55:11.425907+0000","last_became_active":"2024-09-17T03:55:10.111121+0000","last_became_peered":"2024-09-17T03:55:10.111121+0000","last_unstale":"2024-09-17T03:55:11.425907+0000","last_undegraded":"2024-09-17T03:55:11.425907+0000","last_fullsized":"2024-09-17T03:55:11.425907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112556+0000","last_change":"2024-09-17T03:55:09.109395+0000","last_active":"2024-09-17T03:55:09.112556+0000","last_peered":"2024-09-17T03:55:09.112556+0000","last_clean":"2024-09-17T03:55:09.112556+0000","last_became_active":"2024-09-17T03:55:09.107581+0000","last_became_peered":"2024-09-17T03:55:09.107581+0000","last_unstale":"2024-09-17T03:55:09.112556+0000","last_undegraded":"2024-09-17T03:55:09.112556+0000","last_fullsized":"2024-09-17T03:55:09.112556+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:51.503143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423981+0000","last_change":"2024-09-17T03:55:10.115350+0000","last_active":"2024-09-17T03:55:11.423981+0000","last_peered":"2024-09-17T03:55:11.423981+0000","last_clean":"2024-09-17T03:55:11.423981+0000","last_became_active":"2024-09-17T03:55:10.111606+0000","last_became_peered":"2024-09-17T03:55:10.111606+0000","last_unstale":"2024-09-17T03:55:11.423981+0000","last_undegraded":"2024-09-17T03:55:11.423981+0000","last_fullsized":"2024-09-17T03:55:11.423981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:26.916486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104020+0000","last_change":"2024-09-17T03:55:09.100552+0000","last_active":"2024-09-17T03:55:11.104020+0000","last_peered":"2024-09-17T03:55:11.104020+0000","last_clean":"2024-09-17T03:55:11.104020+0000","last_became_active":"2024-09-17T03:55:09.100230+0000","last_became_peered":"2024-09-17T03:55:09.100230+0000","last_unstale":"2024-09-17T03:55:11.104020+0000","last_undegraded":"2024-09-17T03:55:11.104020+0000","last_fullsized":"2024-09-17T03:55:11.104020+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:26.431547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425676+0000","last_change":"2024-09-17T03:55:10.125057+0000","last_active":"2024-09-17T03:55:11.425676+0000","last_peered":"2024-09-17T03:55:11.425676+0000","last_clean":"2024-09-17T03:55:11.425676+0000","last_became_active":"2024-09-17T03:55:10.124788+0000","last_became_peered":"2024-09-17T03:55:10.124788+0000","last_unstale":"2024-09-17T03:55:11.425676+0000","last_undegraded":"2024-09-17T03:55:11.425676+0000","last_fullsized":"2024-09-17T03:55:11.425676+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:49.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.103014+0000","last_change":"2024-09-17T03:55:09.098297+0000","last_active":"2024-09-17T03:55:09.103014+0000","last_peered":"2024-09-17T03:55:09.103014+0000","last_clean":"2024-09-17T03:55:09.103014+0000","last_became_active":"2024-09-17T03:55:09.097970+0000","last_became_peered":"2024-09-17T03:55:09.097970+0000","last_unstale":"2024-09-17T03:55:09.103014+0000","last_undegraded":"2024-09-17T03:55:09.103014+0000","last_fullsized":"2024-09-17T03:55:09.103014+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:08:34.560601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424050+0000","last_change":"2024-09-17T03:55:10.122441+0000","last_active":"2024-09-17T03:55:11.424050+0000","last_peered":"2024-09-17T03:55:11.424050+0000","last_clean":"2024-09-17T03:55:11.424050+0000","last_became_active":"2024-09-17T03:55:10.121881+0000","last_became_peered":"2024-09-17T03:55:10.121881+0000","last_unstale":"2024-09-17T03:55:11.424050+0000","last_undegraded":"2024-09-17T03:55:11.424050+0000","last_fullsized":"2024-09-17T03:55:11.424050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.313711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425520+0000","last_change":"2024-09-17T03:55:09.101386+0000","last_active":"2024-09-17T03:55:11.425520+0000","last_peered":"2024-09-17T03:55:11.425520+0000","last_clean":"2024-09-17T03:55:11.425520+0000","last_became_active":"2024-09-17T03:55:09.101041+0000","last_became_peered":"2024-09-17T03:55:09.101041+0000","last_unstale":"2024-09-17T03:55:11.425520+0000","last_undegraded":"2024-09-17T03:55:11.425520+0000","last_fullsized":"2024-09-17T03:55:11.425520+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:07.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104041+0000","last_change":"2024-09-17T03:55:10.097916+0000","last_active":"2024-09-17T03:55:11.104041+0000","last_peered":"2024-09-17T03:55:11.104041+0000","last_clean":"2024-09-17T03:55:11.104041+0000","last_became_active":"2024-09-17T03:55:10.097595+0000","last_became_peered":"2024-09-17T03:55:10.097595+0000","last_unstale":"2024-09-17T03:55:11.104041+0000","last_undegraded":"2024-09-17T03:55:11.104041+0000","last_fullsized":"2024-09-17T03:55:11.104041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423852+0000","last_change":"2024-09-17T03:55:09.093148+0000","last_active":"2024-09-17T03:55:11.423852+0000","last_peered":"2024-09-17T03:55:11.423852+0000","last_clean":"2024-09-17T03:55:11.423852+0000","last_became_active":"2024-09-17T03:55:09.092860+0000","last_became_peered":"2024-09-17T03:55:09.092860+0000","last_unstale":"2024-09-17T03:55:11.423852+0000","last_undegraded":"2024-09-17T03:55:11.423852+0000","last_fullsized":"2024-09-17T03:55:11.423852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:11.814702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104560+0000","last_change":"2024-09-17T03:55:10.102328+0000","last_active":"2024-09-17T03:55:11.104560+0000","last_peered":"2024-09-17T03:55:11.104560+0000","last_clean":"2024-09-17T03:55:11.104560+0000","last_became_active":"2024-09-17T03:55:10.101937+0000","last_became_peered":"2024-09-17T03:55:10.101937+0000","last_unstale":"2024-09-17T03:55:11.104560+0000","last_undegraded":"2024-09-17T03:55:11.104560+0000","last_fullsized":"2024-09-17T03:55:11.104560+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:53.259242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112624+0000","last_change":"2024-09-17T03:55:09.110979+0000","last_active":"2024-09-17T03:55:09.112624+0000","last_peered":"2024-09-17T03:55:09.112624+0000","last_clean":"2024-09-17T03:55:09.112624+0000","last_became_active":"2024-09-17T03:55:09.110640+0000","last_became_peered":"2024-09-17T03:55:09.110640+0000","last_unstale":"2024-09-17T03:55:09.112624+0000","last_undegraded":"2024-09-17T03:55:09.112624+0000","last_fullsized":"2024-09-17T03:55:09.112624+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:42.556858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426564+0000","last_change":"2024-09-17T03:55:10.105276+0000","last_active":"2024-09-17T03:55:11.426564+0000","last_peered":"2024-09-17T03:55:11.426564+0000","last_clean":"2024-09-17T03:55:11.426564+0000","last_became_active":"2024-09-17T03:55:10.104973+0000","last_became_peered":"2024-09-17T03:55:10.104973+0000","last_unstale":"2024-09-17T03:55:11.426564+0000","last_undegraded":"2024-09-17T03:55:11.426564+0000","last_fullsized":"2024-09-17T03:55:11.426564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:37.836511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.105305+0000","last_change":"2024-09-17T03:55:09.100586+0000","last_active":"2024-09-17T03:55:09.105305+0000","last_peered":"2024-09-17T03:55:09.105305+0000","last_clean":"2024-09-17T03:55:09.105305+0000","last_became_active":"2024-09-17T03:55:09.100303+0000","last_became_peered":"2024-09-17T03:55:09.100303+0000","last_unstale":"2024-09-17T03:55:09.105305+0000","last_undegraded":"2024-09-17T03:55:09.105305+0000","last_fullsized":"2024-09-17T03:55:09.105305+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:36.891586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.089096+0000","last_change":"2024-09-17T03:55:09.089096+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.089096+0000","last_undegraded":"2024-09-17T03:55:09.089096+0000","last_fullsized":"2024-09-17T03:55:09.089096+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424336+0000","last_change":"2024-09-17T03:55:09.101674+0000","last_active":"2024-09-17T03:55:11.424336+0000","last_peered":"2024-09-17T03:55:11.424336+0000","last_clean":"2024-09-17T03:55:11.424336+0000","last_became_active":"2024-09-17T03:55:09.101311+0000","last_became_peered":"2024-09-17T03:55:09.101311+0000","last_unstale":"2024-09-17T03:55:11.424336+0000","last_undegraded":"2024-09-17T03:55:11.424336+0000","last_fullsized":"2024-09-17T03:55:11.424336+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:43.324925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.105085+0000","last_change":"2024-09-17T03:55:10.100605+0000","last_active":"2024-09-17T03:55:11.105085+0000","last_peered":"2024-09-17T03:55:11.105085+0000","last_clean":"2024-09-17T03:55:11.105085+0000","last_became_active":"2024-09-17T03:55:10.100264+0000","last_became_peered":"2024-09-17T03:55:10.100264+0000","last_unstale":"2024-09-17T03:55:11.105085+0000","last_undegraded":"2024-09-17T03:55:11.105085+0000","last_fullsized":"2024-09-17T03:55:11.105085+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:31.420107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":21,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424538+0000","last_change":"2024-09-17T03:55:09.102173+0000","last_active":"2024-09-17T03:55:11.424538+0000","last_peered":"2024-09-17T03:55:11.424538+0000","last_clean":"2024-09-17T03:55:11.424538+0000","last_became_active":"2024-09-17T03:55:09.101451+0000","last_became_peered":"2024-09-17T03:55:09.101451+0000","last_unstale":"2024-09-17T03:55:11.424538+0000","last_undegraded":"2024-09-17T03:55:11.424538+0000","last_fullsized":"2024-09-17T03:55:11.424538+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:16.929007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085389+0000","last_change":"2024-09-17T03:55:09.085389+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085389+0000","last_undegraded":"2024-09-17T03:55:09.085389+0000","last_fullsized":"2024-09-17T03:55:09.085389+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.107231+0000","last_change":"2024-09-17T03:55:09.101519+0000","last_active":"2024-09-17T03:55:09.107231+0000","last_peered":"2024-09-17T03:55:09.107231+0000","last_clean":"2024-09-17T03:55:09.107231+0000","last_became_active":"2024-09-17T03:55:09.101211+0000","last_became_peered":"2024-09-17T03:55:09.101211+0000","last_unstale":"2024-09-17T03:55:09.107231+0000","last_undegraded":"2024-09-17T03:55:09.107231+0000","last_fullsized":"2024-09-17T03:55:09.107231+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:29.559882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.088070+0000","last_change":"2024-09-17T03:55:09.088070+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.088070+0000","last_undegraded":"2024-09-17T03:55:09.088070+0000","last_fullsized":"2024-09-17T03:55:09.088070+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424612+0000","last_change":"2024-09-17T03:55:09.103112+0000","last_active":"2024-09-17T03:55:11.424612+0000","last_peered":"2024-09-17T03:55:11.424612+0000","last_clean":"2024-09-17T03:55:11.424612+0000","last_became_active":"2024-09-17T03:55:09.102182+0000","last_became_peered":"2024-09-17T03:55:09.102182+0000","last_unstale":"2024-09-17T03:55:11.424612+0000","last_undegraded":"2024-09-17T03:55:11.424612+0000","last_fullsized":"2024-09-17T03:55:11.424612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:53:55.298078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084668+0000","last_change":"2024-09-17T03:55:09.084668+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084668+0000","last_undegraded":"2024-09-17T03:55:09.084668+0000","last_fullsized":"2024-09-17T03:55:09.084668+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112617+0000","last_change":"2024-09-17T03:55:09.110547+0000","last_active":"2024-09-17T03:55:09.112617+0000","last_peered":"2024-09-17T03:55:09.112617+0000","last_clean":"2024-09-17T03:55:09.112617+0000","last_became_active":"2024-09-17T03:55:09.109930+0000","last_became_peered":"2024-09-17T03:55:09.109930+0000","last_unstale":"2024-09-17T03:55:09.112617+0000","last_undegraded":"2024-09-17T03:55:09.112617+0000","last_fullsized":"2024-09-17T03:55:09.112617+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:42.335248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.095435+0000","last_change":"2024-09-17T03:55:09.095435+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.095435+0000","last_undegraded":"2024-09-17T03:55:09.095435+0000","last_fullsized":"2024-09-17T03:55:09.095435+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424999+0000","last_change":"2024-09-17T03:55:09.102865+0000","last_active":"2024-09-17T03:55:11.424999+0000","last_peered":"2024-09-17T03:55:11.424999+0000","last_clean":"2024-09-17T03:55:11.424999+0000","last_became_active":"2024-09-17T03:55:09.100408+0000","last_became_peered":"2024-09-17T03:55:09.100408+0000","last_unstale":"2024-09-17T03:55:11.424999+0000","last_undegraded":"2024-09-17T03:55:11.424999+0000","last_fullsized":"2024-09-17T03:55:11.424999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:29.269455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085989+0000","last_change":"2024-09-17T03:55:09.085989+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085989+0000","last_undegraded":"2024-09-17T03:55:09.085989+0000","last_fullsized":"2024-09-17T03:55:09.085989+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109499+0000","last_change":"2024-09-17T03:55:09.106143+0000","last_active":"2024-09-17T03:55:09.109499+0000","last_peered":"2024-09-17T03:55:09.109499+0000","last_clean":"2024-09-17T03:55:09.109499+0000","last_became_active":"2024-09-17T03:55:09.105539+0000","last_became_peered":"2024-09-17T03:55:09.105539+0000","last_unstale":"2024-09-17T03:55:09.109499+0000","last_undegraded":"2024-09-17T03:55:09.109499+0000","last_fullsized":"2024-09-17T03:55:09.109499+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:18.027915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425055+0000","last_change":"2024-09-17T03:55:10.118502+0000","last_active":"2024-09-17T03:55:11.425055+0000","last_peered":"2024-09-17T03:55:11.425055+0000","last_clean":"2024-09-17T03:55:11.425055+0000","last_became_active":"2024-09-17T03:55:10.118276+0000","last_became_peered":"2024-09-17T03:55:10.118276+0000","last_unstale":"2024-09-17T03:55:11.425055+0000","last_undegraded":"2024-09-17T03:55:11.425055+0000","last_fullsized":"2024-09-17T03:55:11.425055+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:23.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425392+0000","last_change":"2024-09-17T03:55:09.099409+0000","last_active":"2024-09-17T03:55:11.425392+0000","last_peered":"2024-09-17T03:55:11.425392+0000","last_clean":"2024-09-17T03:55:11.425392+0000","last_became_active":"2024-09-17T03:55:09.097894+0000","last_became_peered":"2024-09-17T03:55:09.097894+0000","last_unstale":"2024-09-17T03:55:11.425392+0000","last_undegraded":"2024-09-17T03:55:11.425392+0000","last_fullsized":"2024-09-17T03:55:11.425392+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:30.592901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426005+0000","last_change":"2024-09-17T03:55:10.101315+0000","last_active":"2024-09-17T03:55:11.426005+0000","last_peered":"2024-09-17T03:55:11.426005+0000","last_clean":"2024-09-17T03:55:11.426005+0000","last_became_active":"2024-09-17T03:55:10.100936+0000","last_became_peered":"2024-09-17T03:55:10.100936+0000","last_unstale":"2024-09-17T03:55:11.426005+0000","last_undegraded":"2024-09-17T03:55:11.426005+0000","last_fullsized":"2024-09-17T03:55:11.426005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:51.665179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104091+0000","last_change":"2024-09-17T03:55:09.095974+0000","last_active":"2024-09-17T03:55:11.104091+0000","last_peered":"2024-09-17T03:55:11.104091+0000","last_clean":"2024-09-17T03:55:11.104091+0000","last_became_active":"2024-09-17T03:55:09.095632+0000","last_became_peered":"2024-09-17T03:55:09.095632+0000","last_unstale":"2024-09-17T03:55:11.104091+0000","last_undegraded":"2024-09-17T03:55:11.104091+0000","last_fullsized":"2024-09-17T03:55:11.104091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.401993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424900+0000","last_change":"2024-09-17T03:55:10.104348+0000","last_active":"2024-09-17T03:55:11.424900+0000","last_peered":"2024-09-17T03:55:11.424900+0000","last_clean":"2024-09-17T03:55:11.424900+0000","last_became_active":"2024-09-17T03:55:10.104019+0000","last_became_peered":"2024-09-17T03:55:10.104019+0000","last_unstale":"2024-09-17T03:55:11.424900+0000","last_undegraded":"2024-09-17T03:55:11.424900+0000","last_fullsized":"2024-09-17T03:55:11.424900+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.618440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.108346+0000","last_change":"2024-09-17T03:55:09.107009+0000","last_active":"2024-09-17T03:55:09.108346+0000","last_peered":"2024-09-17T03:55:09.108346+0000","last_clean":"2024-09-17T03:55:09.108346+0000","last_became_active":"2024-09-17T03:55:09.106706+0000","last_became_peered":"2024-09-17T03:55:09.106706+0000","last_unstale":"2024-09-17T03:55:09.108346+0000","last_undegraded":"2024-09-17T03:55:09.108346+0000","last_fullsized":"2024-09-17T03:55:09.108346+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:13.263489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424019+0000","last_change":"2024-09-17T03:55:10.101801+0000","last_active":"2024-09-17T03:55:11.424019+0000","last_peered":"2024-09-17T03:55:11.424019+0000","last_clean":"2024-09-17T03:55:11.424019+0000","last_became_active":"2024-09-17T03:55:10.101240+0000","last_became_peered":"2024-09-17T03:55:10.101240+0000","last_unstale":"2024-09-17T03:55:11.424019+0000","last_undegraded":"2024-09-17T03:55:11.424019+0000","last_fullsized":"2024-09-17T03:55:11.424019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:08.177233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.109573+0000","last_change":"2024-09-17T03:55:09.105469+0000","last_active":"2024-09-17T03:55:09.109573+0000","last_peered":"2024-09-17T03:55:09.109573+0000","last_clean":"2024-09-17T03:55:09.109573+0000","last_became_active":"2024-09-17T03:55:09.105074+0000","last_became_peered":"2024-09-17T03:55:09.105074+0000","last_unstale":"2024-09-17T03:55:09.109573+0000","last_undegraded":"2024-09-17T03:55:09.109573+0000","last_fullsized":"2024-09-17T03:55:09.109573+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:37.786492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425201+0000","last_change":"2024-09-17T03:55:10.110424+0000","last_active":"2024-09-17T03:55:11.425201+0000","last_peered":"2024-09-17T03:55:11.425201+0000","last_clean":"2024-09-17T03:55:11.425201+0000","last_became_active":"2024-09-17T03:55:10.110057+0000","last_became_peered":"2024-09-17T03:55:10.110057+0000","last_unstale":"2024-09-17T03:55:11.425201+0000","last_undegraded":"2024-09-17T03:55:11.425201+0000","last_fullsized":"2024-09-17T03:55:11.425201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:40.073222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.111837+0000","last_change":"2024-09-17T03:55:09.109486+0000","last_active":"2024-09-17T03:55:09.111837+0000","last_peered":"2024-09-17T03:55:09.111837+0000","last_clean":"2024-09-17T03:55:09.111837+0000","last_became_active":"2024-09-17T03:55:09.108902+0000","last_became_peered":"2024-09-17T03:55:09.108902+0000","last_unstale":"2024-09-17T03:55:09.111837+0000","last_undegraded":"2024-09-17T03:55:09.111837+0000","last_fullsized":"2024-09-17T03:55:09.111837+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:45.582374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.084094+0000","last_change":"2024-09-17T03:55:09.084094+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.084094+0000","last_undegraded":"2024-09-17T03:55:09.084094+0000","last_fullsized":"2024-09-17T03:55:09.084094+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423957+0000","last_change":"2024-09-17T03:55:09.089727+0000","last_active":"2024-09-17T03:55:11.423957+0000","last_peered":"2024-09-17T03:55:11.423957+0000","last_clean":"2024-09-17T03:55:11.423957+0000","last_became_active":"2024-09-17T03:55:09.089416+0000","last_became_peered":"2024-09-17T03:55:09.089416+0000","last_unstale":"2024-09-17T03:55:11.423957+0000","last_undegraded":"2024-09-17T03:55:11.423957+0000","last_fullsized":"2024-09-17T03:55:11.423957+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:19.704845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426516+0000","last_change":"2024-09-17T03:55:10.106649+0000","last_active":"2024-09-17T03:55:11.426516+0000","last_peered":"2024-09-17T03:55:11.426516+0000","last_clean":"2024-09-17T03:55:11.426516+0000","last_became_active":"2024-09-17T03:55:10.106347+0000","last_became_peered":"2024-09-17T03:55:10.106347+0000","last_unstale":"2024-09-17T03:55:11.426516+0000","last_undegraded":"2024-09-17T03:55:11.426516+0000","last_fullsized":"2024-09-17T03:55:11.426516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:21.848148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.110091+0000","last_change":"2024-09-17T03:55:09.106794+0000","last_active":"2024-09-17T03:55:09.110091+0000","last_peered":"2024-09-17T03:55:09.110091+0000","last_clean":"2024-09-17T03:55:09.110091+0000","last_became_active":"2024-09-17T03:55:09.106461+0000","last_became_peered":"2024-09-17T03:55:09.106461+0000","last_unstale":"2024-09-17T03:55:09.110091+0000","last_undegraded":"2024-09-17T03:55:09.110091+0000","last_fullsized":"2024-09-17T03:55:09.110091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:22.706399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424964+0000","last_change":"2024-09-17T03:55:10.111117+0000","last_active":"2024-09-17T03:55:11.424964+0000","last_peered":"2024-09-17T03:55:11.424964+0000","last_clean":"2024-09-17T03:55:11.424964+0000","last_became_active":"2024-09-17T03:55:10.110769+0000","last_became_peered":"2024-09-17T03:55:10.110769+0000","last_unstale":"2024-09-17T03:55:11.424964+0000","last_undegraded":"2024-09-17T03:55:11.424964+0000","last_fullsized":"2024-09-17T03:55:11.424964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:34.220767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426576+0000","last_change":"2024-09-17T03:55:09.091992+0000","last_active":"2024-09-17T03:55:11.426576+0000","last_peered":"2024-09-17T03:55:11.426576+0000","last_clean":"2024-09-17T03:55:11.426576+0000","last_became_active":"2024-09-17T03:55:09.091655+0000","last_became_peered":"2024-09-17T03:55:09.091655+0000","last_unstale":"2024-09-17T03:55:11.426576+0000","last_undegraded":"2024-09-17T03:55:11.426576+0000","last_fullsized":"2024-09-17T03:55:11.426576+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:14.155619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424655+0000","last_change":"2024-09-17T03:55:10.125141+0000","last_active":"2024-09-17T03:55:11.424655+0000","last_peered":"2024-09-17T03:55:11.424655+0000","last_clean":"2024-09-17T03:55:11.424655+0000","last_became_active":"2024-09-17T03:55:10.124839+0000","last_became_peered":"2024-09-17T03:55:10.124839+0000","last_unstale":"2024-09-17T03:55:11.424655+0000","last_undegraded":"2024-09-17T03:55:11.424655+0000","last_fullsized":"2024-09-17T03:55:11.424655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:49.605272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424463+0000","last_change":"2024-09-17T03:55:09.092456+0000","last_active":"2024-09-17T03:55:11.424463+0000","last_peered":"2024-09-17T03:55:11.424463+0000","last_clean":"2024-09-17T03:55:11.424463+0000","last_became_active":"2024-09-17T03:55:09.092133+0000","last_became_peered":"2024-09-17T03:55:09.092133+0000","last_unstale":"2024-09-17T03:55:11.424463+0000","last_undegraded":"2024-09-17T03:55:11.424463+0000","last_fullsized":"2024-09-17T03:55:11.424463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:29.592608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425399+0000","last_change":"2024-09-17T03:55:10.105263+0000","last_active":"2024-09-17T03:55:11.425399+0000","last_peered":"2024-09-17T03:55:11.425399+0000","last_clean":"2024-09-17T03:55:11.425399+0000","last_became_active":"2024-09-17T03:55:10.104878+0000","last_became_peered":"2024-09-17T03:55:10.104878+0000","last_unstale":"2024-09-17T03:55:11.425399+0000","last_undegraded":"2024-09-17T03:55:11.425399+0000","last_fullsized":"2024-09-17T03:55:11.425399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:29.686662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.104632+0000","last_change":"2024-09-17T03:55:09.101334+0000","last_active":"2024-09-17T03:55:09.104632+0000","last_peered":"2024-09-17T03:55:09.104632+0000","last_clean":"2024-09-17T03:55:09.104632+0000","last_became_active":"2024-09-17T03:55:09.101016+0000","last_became_peered":"2024-09-17T03:55:09.101016+0000","last_unstale":"2024-09-17T03:55:09.104632+0000","last_undegraded":"2024-09-17T03:55:09.104632+0000","last_fullsized":"2024-09-17T03:55:09.104632+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:30.035885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425931+0000","last_change":"2024-09-17T03:55:10.097057+0000","last_active":"2024-09-17T03:55:11.425931+0000","last_peered":"2024-09-17T03:55:11.425931+0000","last_clean":"2024-09-17T03:55:11.425931+0000","last_became_active":"2024-09-17T03:55:10.096210+0000","last_became_peered":"2024-09-17T03:55:10.096210+0000","last_unstale":"2024-09-17T03:55:11.425931+0000","last_undegraded":"2024-09-17T03:55:11.425931+0000","last_fullsized":"2024-09-17T03:55:11.425931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:04.260938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104187+0000","last_change":"2024-09-17T03:55:09.099774+0000","last_active":"2024-09-17T03:55:11.104187+0000","last_peered":"2024-09-17T03:55:11.104187+0000","last_clean":"2024-09-17T03:55:11.104187+0000","last_became_active":"2024-09-17T03:55:09.099422+0000","last_became_peered":"2024-09-17T03:55:09.099422+0000","last_unstale":"2024-09-17T03:55:11.104187+0000","last_undegraded":"2024-09-17T03:55:11.104187+0000","last_fullsized":"2024-09-17T03:55:11.104187+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:56.088001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426331+0000","last_change":"2024-09-17T03:55:10.112375+0000","last_active":"2024-09-17T03:55:11.426331+0000","last_peered":"2024-09-17T03:55:11.426331+0000","last_clean":"2024-09-17T03:55:11.426331+0000","last_became_active":"2024-09-17T03:55:10.112033+0000","last_became_peered":"2024-09-17T03:55:10.112033+0000","last_unstale":"2024-09-17T03:55:11.426331+0000","last_undegraded":"2024-09-17T03:55:11.426331+0000","last_fullsized":"2024-09-17T03:55:11.426331+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426160+0000","last_change":"2024-09-17T03:55:09.103012+0000","last_active":"2024-09-17T03:55:11.426160+0000","last_peered":"2024-09-17T03:55:11.426160+0000","last_clean":"2024-09-17T03:55:11.426160+0000","last_became_active":"2024-09-17T03:55:09.102701+0000","last_became_peered":"2024-09-17T03:55:09.102701+0000","last_unstale":"2024-09-17T03:55:11.426160+0000","last_undegraded":"2024-09-17T03:55:11.426160+0000","last_fullsized":"2024-09-17T03:55:11.426160+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:02.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425468+0000","last_change":"2024-09-17T03:55:10.113187+0000","last_active":"2024-09-17T03:55:11.425468+0000","last_peered":"2024-09-17T03:55:11.425468+0000","last_clean":"2024-09-17T03:55:11.425468+0000","last_became_active":"2024-09-17T03:55:10.112812+0000","last_became_peered":"2024-09-17T03:55:10.112812+0000","last_unstale":"2024-09-17T03:55:11.425468+0000","last_undegraded":"2024-09-17T03:55:11.425468+0000","last_fullsized":"2024-09-17T03:55:11.425468+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:18.693960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-17T03:55:09.112065+0000","last_change":"2024-09-17T03:55:09.108670+0000","last_active":"2024-09-17T03:55:09.112065+0000","last_peered":"2024-09-17T03:55:09.112065+0000","last_clean":"2024-09-17T03:55:09.112065+0000","last_became_active":"2024-09-17T03:55:09.106248+0000","last_became_peered":"2024-09-17T03:55:09.106248+0000","last_unstale":"2024-09-17T03:55:09.112065+0000","last_undegraded":"2024-09-17T03:55:09.112065+0000","last_fullsized":"2024-09-17T03:55:09.112065+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:04:54.273948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423930+0000","last_change":"2024-09-17T03:55:10.126012+0000","last_active":"2024-09-17T03:55:11.423930+0000","last_peered":"2024-09-17T03:55:11.423930+0000","last_clean":"2024-09-17T03:55:11.423930+0000","last_became_active":"2024-09-17T03:55:10.125592+0000","last_became_peered":"2024-09-17T03:55:10.125592+0000","last_unstale":"2024-09-17T03:55:11.423930+0000","last_undegraded":"2024-09-17T03:55:11.423930+0000","last_fullsized":"2024-09-17T03:55:11.423930+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:17.583329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424054+0000","last_change":"2024-09-17T03:55:09.096029+0000","last_active":"2024-09-17T03:55:11.424054+0000","last_peered":"2024-09-17T03:55:11.424054+0000","last_clean":"2024-09-17T03:55:11.424054+0000","last_became_active":"2024-09-17T03:55:09.091782+0000","last_became_peered":"2024-09-17T03:55:09.091782+0000","last_unstale":"2024-09-17T03:55:11.424054+0000","last_undegraded":"2024-09-17T03:55:11.424054+0000","last_fullsized":"2024-09-17T03:55:11.424054+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:17.583743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":105,"state":"creating+peering","last_fresh":"2024-09-17T03:55:09.085271+0000","last_change":"2024-09-17T03:55:09.085271+0000","last_active":"2024-09-17T03:55:09.066966+0000","last_peered":"2024-09-17T03:55:09.066966+0000","last_clean":"2024-09-17T03:55:09.066966+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:55:09.085271+0000","last_undegraded":"2024-09-17T03:55:09.085271+0000","last_fullsized":"2024-09-17T03:55:09.085271+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104276+0000","last_change":"2024-09-17T03:55:09.096214+0000","last_active":"2024-09-17T03:55:11.104276+0000","last_peered":"2024-09-17T03:55:11.104276+0000","last_clean":"2024-09-17T03:55:11.104276+0000","last_became_active":"2024-09-17T03:55:09.095625+0000","last_became_peered":"2024-09-17T03:55:09.095625+0000","last_unstale":"2024-09-17T03:55:11.104276+0000","last_undegraded":"2024-09-17T03:55:11.104276+0000","last_fullsized":"2024-09-17T03:55:11.104276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:12.569897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":265,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425604+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:55:11.425604+0000","last_peered":"2024-09-17T03:55:11.425604+0000","last_clean":"2024-09-17T03:55:11.425604+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:55:11.425604+0000","last_undegraded":"2024-09-17T03:55:11.425604+0000","last_fullsized":"2024-09-17T03:55:11.425604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1157630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":567250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509511,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509513,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509513,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36608,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335232,"statfs":{"total":96636764160,"available":96599277568,"internally_reserved":0,"allocated":1286144,"data_stored":547296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1327104,"data_stored":566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542217,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":1158738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:PG 17.3b is not active+clean 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.093607+0000', 'last_change': '2024-09-17T03:55:09.093607+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.093607+0000', 'last_undegraded': '2024-09-17T03:55:09.093607+0000', 'last_fullsized': '2024-09-17T03:55:09.093607+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:PG 17.3a is not active+clean 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.090308+0000', 'last_change': '2024-09-17T03:55:09.090308+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.090308+0000', 'last_undegraded': '2024-09-17T03:55:09.090308+0000', 'last_fullsized': '2024-09-17T03:55:09.090308+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:PG 17.31 is not active+clean 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.087537+0000', 'last_change': '2024-09-17T03:55:09.087537+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.087537+0000', 'last_undegraded': '2024-09-17T03:55:09.087537+0000', 'last_fullsized': '2024-09-17T03:55:09.087537+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:PG 17.2a is not active+clean 2024-09-17T03:55:15.681 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085847+0000', 'last_change': '2024-09-17T03:55:09.085847+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085847+0000', 'last_undegraded': '2024-09-17T03:55:09.085847+0000', 'last_fullsized': '2024-09-17T03:55:09.085847+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.28 is not active+clean 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085796+0000', 'last_change': '2024-09-17T03:55:09.085796+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085796+0000', 'last_undegraded': '2024-09-17T03:55:09.085796+0000', 'last_fullsized': '2024-09-17T03:55:09.085796+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.25 is not active+clean 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.087802+0000', 'last_change': '2024-09-17T03:55:09.087802+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.087802+0000', 'last_undegraded': '2024-09-17T03:55:09.087802+0000', 'last_fullsized': '2024-09-17T03:55:09.087802+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.24 is not active+clean 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.089564+0000', 'last_change': '2024-09-17T03:55:09.089564+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.089564+0000', 'last_undegraded': '2024-09-17T03:55:09.089564+0000', 'last_fullsized': '2024-09-17T03:55:09.089564+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.20 is not active+clean 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.089096+0000', 'last_change': '2024-09-17T03:55:09.089096+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.089096+0000', 'last_undegraded': '2024-09-17T03:55:09.089096+0000', 'last_fullsized': '2024-09-17T03:55:09.089096+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085837+0000', 'last_change': '2024-09-17T03:55:09.085837+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085837+0000', 'last_undegraded': '2024-09-17T03:55:09.085837+0000', 'last_fullsized': '2024-09-17T03:55:09.085837+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.682 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.084952+0000', 'last_change': '2024-09-17T03:55:09.084952+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.084952+0000', 'last_undegraded': '2024-09-17T03:55:09.084952+0000', 'last_fullsized': '2024-09-17T03:55:09.084952+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:PG 17.14 is not active+clean 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.089096+0000', 'last_change': '2024-09-17T03:55:09.089096+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.089096+0000', 'last_undegraded': '2024-09-17T03:55:09.089096+0000', 'last_fullsized': '2024-09-17T03:55:09.089096+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:PG 17.12 is not active+clean 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085389+0000', 'last_change': '2024-09-17T03:55:09.085389+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085389+0000', 'last_undegraded': '2024-09-17T03:55:09.085389+0000', 'last_fullsized': '2024-09-17T03:55:09.085389+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:PG 17.13 is not active+clean 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.088070+0000', 'last_change': '2024-09-17T03:55:09.088070+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.088070+0000', 'last_undegraded': '2024-09-17T03:55:09.088070+0000', 'last_fullsized': '2024-09-17T03:55:09.088070+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:PG 17.11 is not active+clean 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.084668+0000', 'last_change': '2024-09-17T03:55:09.084668+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.084668+0000', 'last_undegraded': '2024-09-17T03:55:09.084668+0000', 'last_fullsized': '2024-09-17T03:55:09.084668+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.683 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.095435+0000', 'last_change': '2024-09-17T03:55:09.095435+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.095435+0000', 'last_undegraded': '2024-09-17T03:55:09.095435+0000', 'last_fullsized': '2024-09-17T03:55:09.095435+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085989+0000', 'last_change': '2024-09-17T03:55:09.085989+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085989+0000', 'last_undegraded': '2024-09-17T03:55:09.085989+0000', 'last_fullsized': '2024-09-17T03:55:09.085989+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:PG 17.9 is not active+clean 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.084094+0000', 'last_change': '2024-09-17T03:55:09.084094+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.084094+0000', 'last_undegraded': '2024-09-17T03:55:09.084094+0000', 'last_fullsized': '2024-09-17T03:55:09.084094+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:PG 17.10 is not active+clean 2024-09-17T03:55:15.684 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 105, 'state': 'creating+peering', 'last_fresh': '2024-09-17T03:55:09.085271+0000', 'last_change': '2024-09-17T03:55:09.085271+0000', 'last_active': '2024-09-17T03:55:09.066966+0000', 'last_peered': '2024-09-17T03:55:09.066966+0000', 'last_clean': '2024-09-17T03:55:09.066966+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T03:55:09.085271+0000', 'last_undegraded': '2024-09-17T03:55:09.085271+0000', 'last_fullsized': '2024-09-17T03:55:09.085271+0000', 'mapping_epoch': 105, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 105, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'last_clean_scrub_stamp': '2024-09-17T03:55:09.066966+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T03:55:15.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T03:55:15.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 -- 172.21.15.129:0/2838749179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e900cd390 msgr2=0x7f2e900af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 --2- 172.21.15.129:0/2838749179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e900cd390 0x7f2e900af7a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009830 tx=0x7f2e7802a510 comp rx=0 tx=0).stop 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 -- 172.21.15.129:0/2838749179 shutdown_connections 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 --2- 172.21.15.129:0/2838749179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e900b4180 0x7f2e900ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 --2- 172.21.15.129:0/2838749179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e900cd390 0x7f2e900af7a0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 -- 172.21.15.129:0/2838749179 >> 172.21.15.129:0/2838749179 conn(0x7f2e9006d410 msgr2=0x7f2e900b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 -- 172.21.15.129:0/2838749179 shutdown_connections 2024-09-17T03:55:15.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.936+0000 7f2e95200640 1 -- 172.21.15.129:0/2838749179 wait complete. 2024-09-17T03:55:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.938+0000 7f2e95200640 1 Processor -- start 2024-09-17T03:55:15.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.938+0000 7f2e95200640 1 -- start start 2024-09-17T03:55:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e900b4180 0x7f2e900ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e9015ce50 0x7f2e901576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9006f8a0 con 0x7f2e901e9ee0 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9006f720 con 0x7f2e900b4180 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e95200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2e9006fa20 con 0x7f2e9015ce50 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53156/0 (socket says 172.21.15.129:53156) 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 -- 172.21.15.129:0/1637166559 learned_addr learned my addr 172.21.15.129:0/1637166559 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e85400640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e9015ce50 0x7f2e901576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8de00640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e900b4180 0x7f2e900ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 -- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e9015ce50 msgr2=0x7f2e901576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e9015ce50 0x7f2e901576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 -- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e900b4180 msgr2=0x7f2e900ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e900b4180 0x7f2e900ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.940+0000 7f2e8e800640 1 -- 172.21.15.129:0/1637166559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e9009ae10 con 0x7f2e901e9ee0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e8e800640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800dac0 tx=0x7f2e8800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88017070 con 0x7f2e901e9ee0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e88003970 con 0x7f2e901e9ee0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e9009b0d0 con 0x7f2e901e9ee0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e88003da0 con 0x7f2e901e9ee0 2024-09-17T03:55:15.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.941+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e901f3770 con 0x7f2e901e9ee0 2024-09-17T03:55:15.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.942+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e901f3bb0 con 0x7f2e901e9ee0 2024-09-17T03:55:15.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.943+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2e88003f40 con 0x7f2e901e9ee0 2024-09-17T03:55:15.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.944+0000 7f2e86a00640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 0x7f2e6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:15.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.944+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f2e8809b0f0 con 0x7f2e901e9ee0 2024-09-17T03:55:15.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.945+0000 7f2e8de00640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 0x7f2e6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:15.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.945+0000 7f2e8de00640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 0x7f2e6c07b0a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2e80001720 tx=0x7f2e80007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:15.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:15.949+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2e88062ed0 con 0x7f2e901e9ee0 2024-09-17T03:55:16.055 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:55:16.055+0000 7f1d24e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:55:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.204+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f2e9006ff40 con 0x7f2e901e9ee0 2024-09-17T03:55:16.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.205+0000 7f2e86a00640 1 -- 172.21.15.129:0/1637166559 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1562 (secure 0 0 0) 0x7f2e88067900 con 0x7f2e901e9ee0 2024-09-17T03:55:16.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:16.205 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":367,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":79},{"state_name":"creating+peering","count":18}],"num_pgs":97,"num_pools":3,"num_objects":13,"data_bytes":592744,"bytes_used":309846016,"bytes_avail":772784267264,"bytes_total":773094113280,"inactive_pgs_ratio":0.18556700646877289,"write_bytes_sec":1194,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":80,"btime":"2024-09-17T03:55:13:284539+0000","id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":7005}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-17T03:54:56.382943+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T03:55:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 msgr2=0x7f2e6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 0x7f2e6c07b0a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2e80001720 tx=0x7f2e80007e90 comp rx=0 tx=0).stop 2024-09-17T03:55:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 msgr2=0x7f2e901f3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800dac0 tx=0x7f2e8800df80 comp rx=0 tx=0).stop 2024-09-17T03:55:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 shutdown_connections 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e6c078b90 0x7f2e6c07b0a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e901e9ee0 0x7f2e901f3f50 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e9015ce50 0x7f2e901576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 --2- 172.21.15.129:0/1637166559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e900b4180 0x7f2e900ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 >> 172.21.15.129:0/1637166559 conn(0x7f2e9006d410 msgr2=0x7f2e90072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 shutdown_connections 2024-09-17T03:55:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.207+0000 7f2e95200640 1 -- 172.21.15.129:0/1637166559 wait complete. 2024-09-17T03:55:16.232 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T03:55:16.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:55:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 -- 172.21.15.129:0/3165911316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 msgr2=0x7f2ee00b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 --2- 172.21.15.129:0/3165911316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee00b4140 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0009830 tx=0x7f2ed002a5c0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 -- 172.21.15.129:0/3165911316 shutdown_connections 2024-09-17T03:55:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 --2- 172.21.15.129:0/3165911316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee00b4140 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 -- 172.21.15.129:0/3165911316 >> 172.21.15.129:0/3165911316 conn(0x7f2ee006d410 msgr2=0x7f2ee00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 -- 172.21.15.129:0/3165911316 shutdown_connections 2024-09-17T03:55:16.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.454+0000 7f2ee7200640 1 -- 172.21.15.129:0/3165911316 wait complete. 2024-09-17T03:55:16.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.456+0000 7f2ee7200640 1 Processor -- start 2024-09-17T03:55:16.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.456+0000 7f2ee7200640 1 -- start start 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 0x7f2ee01568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ee00c74e0 0x7f2ee00c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee006f2e0 con 0x7f2ee00705a0 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee01612e0 con 0x7f2ee006eed0 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee006f460 con 0x7f2ee00c74e0 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53162/0 (socket says 172.21.15.129:53162) 2024-09-17T03:55:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 -- 172.21.15.129:0/4079827640 learned_addr learned my addr 172.21.15.129:0/4079827640 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2edfe00640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 0x7f2ee01568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ee5200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ee00c74e0 0x7f2ee00c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 -- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ee00c74e0 msgr2=0x7f2ee00c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ee00c74e0 0x7f2ee00c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 -- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 msgr2=0x7f2ee01568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 0x7f2ee01568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2ed7400640 1 -- 172.21.15.129:0/4079827640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee00c8250 con 0x7f2ee00705a0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.458+0000 7f2edfe00640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 0x7f2ee01568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2ed7400640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800c340 tx=0x7f2ec800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8017070 con 0x7f2ee00705a0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ec80035f0 con 0x7f2ee00705a0 2024-09-17T03:55:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee0155b30 con 0x7f2ee00705a0 2024-09-17T03:55:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8003a40 con 0x7f2ee00705a0 2024-09-17T03:55:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.459+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee01560f0 con 0x7f2ee00705a0 2024-09-17T03:55:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.460+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee00c7fc0 con 0x7f2ee00705a0 2024-09-17T03:55:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.461+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2ec801a070 con 0x7f2ee00705a0 2024-09-17T03:55:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.462+0000 7f2edcc00640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 0x7f2ebc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.463+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f2ec8099fe0 con 0x7f2ee00705a0 2024-09-17T03:55:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.463+0000 7f2edfe00640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 0x7f2ebc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:16.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.464+0000 7f2edfe00640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 0x7f2ebc07b0a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0034000 tx=0x7f2ed0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:16.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.467+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2ec8062450 con 0x7f2ee00705a0 2024-09-17T03:55:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.647+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ee009e150 con 0x7f2ebc078b90 2024-09-17T03:55:16.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.652+0000 7f2edcc00640 1 -- 172.21.15.129:0/4079827640 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7f2ed00035f0 con 0x7f2ebc078b90 2024-09-17T03:55:16.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:16.656 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:55:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 msgr2=0x7f2ebc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 0x7f2ebc07b0a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0034000 tx=0x7f2ed0004730 comp rx=0 tx=0).stop 2024-09-17T03:55:16.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 msgr2=0x7f2ee009dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800c340 tx=0x7f2ec800c800 comp rx=0 tx=0).stop 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 shutdown_connections 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ebc078b90 0x7f2ebc07b0a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2ee00c74e0 0x7f2ee00c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2ee00705a0 0x7f2ee009dc10 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 --2- 172.21.15.129:0/4079827640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2ee006eed0 0x7f2ee01568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 >> 172.21.15.129:0/4079827640 conn(0x7f2ee006d410 msgr2=0x7f2ee00c6a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 shutdown_connections 2024-09-17T03:55:16.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:16.656+0000 7f2ee7200640 1 -- 172.21.15.129:0/4079827640 wait complete. 2024-09-17T03:55:16.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:55:16.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:55:16.679 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-17T03:55:16.387993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302704,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672016,"statfs":{"total":773094113280,"available":772784144384,"internally_reserved":0,"allocated":11837440,"data_stored":5718724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.309996"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424841+0000","last_change":"2024-09-17T03:55:10.112479+0000","last_active":"2024-09-17T03:55:11.424841+0000","last_peered":"2024-09-17T03:55:11.424841+0000","last_clean":"2024-09-17T03:55:11.424841+0000","last_became_active":"2024-09-17T03:55:10.112184+0000","last_became_peered":"2024-09-17T03:55:10.112184+0000","last_unstale":"2024-09-17T03:55:11.424841+0000","last_undegraded":"2024-09-17T03:55:11.424841+0000","last_fullsized":"2024-09-17T03:55:11.424841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:12.328355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425890+0000","last_change":"2024-09-17T03:55:10.123473+0000","last_active":"2024-09-17T03:55:11.425890+0000","last_peered":"2024-09-17T03:55:11.425890+0000","last_clean":"2024-09-17T03:55:11.425890+0000","last_became_active":"2024-09-17T03:55:10.123217+0000","last_became_peered":"2024-09-17T03:55:10.123217+0000","last_unstale":"2024-09-17T03:55:11.425890+0000","last_undegraded":"2024-09-17T03:55:11.425890+0000","last_fullsized":"2024-09-17T03:55:11.425890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:50.947437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425736+0000","last_change":"2024-09-17T03:55:10.103667+0000","last_active":"2024-09-17T03:55:11.425736+0000","last_peered":"2024-09-17T03:55:11.425736+0000","last_clean":"2024-09-17T03:55:11.425736+0000","last_became_active":"2024-09-17T03:55:10.103336+0000","last_became_peered":"2024-09-17T03:55:10.103336+0000","last_unstale":"2024-09-17T03:55:11.425736+0000","last_undegraded":"2024-09-17T03:55:11.425736+0000","last_fullsized":"2024-09-17T03:55:11.425736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:33.517751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424369+0000","last_change":"2024-09-17T03:55:10.109342+0000","last_active":"2024-09-17T03:55:11.424369+0000","last_peered":"2024-09-17T03:55:11.424369+0000","last_clean":"2024-09-17T03:55:11.424369+0000","last_became_active":"2024-09-17T03:55:10.109013+0000","last_became_peered":"2024-09-17T03:55:10.109013+0000","last_unstale":"2024-09-17T03:55:11.424369+0000","last_undegraded":"2024-09-17T03:55:11.424369+0000","last_fullsized":"2024-09-17T03:55:11.424369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:01.880305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103153+0000","last_change":"2024-09-17T03:55:10.098689+0000","last_active":"2024-09-17T03:55:11.103153+0000","last_peered":"2024-09-17T03:55:11.103153+0000","last_clean":"2024-09-17T03:55:11.103153+0000","last_became_active":"2024-09-17T03:55:10.097160+0000","last_became_peered":"2024-09-17T03:55:10.097160+0000","last_unstale":"2024-09-17T03:55:11.103153+0000","last_undegraded":"2024-09-17T03:55:11.103153+0000","last_fullsized":"2024-09-17T03:55:11.103153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:59:16.574265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142768+0000","last_change":"2024-09-17T03:55:10.118739+0000","last_active":"2024-09-17T03:55:11.142768+0000","last_peered":"2024-09-17T03:55:11.142768+0000","last_clean":"2024-09-17T03:55:11.142768+0000","last_became_active":"2024-09-17T03:55:10.112849+0000","last_became_peered":"2024-09-17T03:55:10.112849+0000","last_unstale":"2024-09-17T03:55:11.142768+0000","last_undegraded":"2024-09-17T03:55:11.142768+0000","last_fullsized":"2024-09-17T03:55:11.142768+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104179+0000","last_change":"2024-09-17T03:55:10.096583+0000","last_active":"2024-09-17T03:55:11.104179+0000","last_peered":"2024-09-17T03:55:11.104179+0000","last_clean":"2024-09-17T03:55:11.104179+0000","last_became_active":"2024-09-17T03:55:10.096289+0000","last_became_peered":"2024-09-17T03:55:10.096289+0000","last_unstale":"2024-09-17T03:55:11.104179+0000","last_undegraded":"2024-09-17T03:55:11.104179+0000","last_fullsized":"2024-09-17T03:55:11.104179+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424746+0000","last_change":"2024-09-17T03:55:10.116515+0000","last_active":"2024-09-17T03:55:11.424746+0000","last_peered":"2024-09-17T03:55:11.424746+0000","last_clean":"2024-09-17T03:55:11.424746+0000","last_became_active":"2024-09-17T03:55:10.116161+0000","last_became_peered":"2024-09-17T03:55:10.116161+0000","last_unstale":"2024-09-17T03:55:11.424746+0000","last_undegraded":"2024-09-17T03:55:11.424746+0000","last_fullsized":"2024-09-17T03:55:11.424746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.433598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424969+0000","last_change":"2024-09-17T03:55:10.113065+0000","last_active":"2024-09-17T03:55:11.424969+0000","last_peered":"2024-09-17T03:55:11.424969+0000","last_clean":"2024-09-17T03:55:11.424969+0000","last_became_active":"2024-09-17T03:55:10.111008+0000","last_became_peered":"2024-09-17T03:55:10.111008+0000","last_unstale":"2024-09-17T03:55:11.424969+0000","last_undegraded":"2024-09-17T03:55:11.424969+0000","last_fullsized":"2024-09-17T03:55:11.424969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:54.618141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424625+0000","last_change":"2024-09-17T03:55:10.105708+0000","last_active":"2024-09-17T03:55:11.424625+0000","last_peered":"2024-09-17T03:55:11.424625+0000","last_clean":"2024-09-17T03:55:11.424625+0000","last_became_active":"2024-09-17T03:55:10.105368+0000","last_became_peered":"2024-09-17T03:55:10.105368+0000","last_unstale":"2024-09-17T03:55:11.424625+0000","last_undegraded":"2024-09-17T03:55:11.424625+0000","last_fullsized":"2024-09-17T03:55:11.424625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:19.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425340+0000","last_change":"2024-09-17T03:55:10.107346+0000","last_active":"2024-09-17T03:55:11.425340+0000","last_peered":"2024-09-17T03:55:11.425340+0000","last_clean":"2024-09-17T03:55:11.425340+0000","last_became_active":"2024-09-17T03:55:10.106955+0000","last_became_peered":"2024-09-17T03:55:10.106955+0000","last_unstale":"2024-09-17T03:55:11.425340+0000","last_undegraded":"2024-09-17T03:55:11.425340+0000","last_fullsized":"2024-09-17T03:55:11.425340+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:29.542513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424410+0000","last_change":"2024-09-17T03:55:10.108582+0000","last_active":"2024-09-17T03:55:11.424410+0000","last_peered":"2024-09-17T03:55:11.424410+0000","last_clean":"2024-09-17T03:55:11.424410+0000","last_became_active":"2024-09-17T03:55:10.105982+0000","last_became_peered":"2024-09-17T03:55:10.105982+0000","last_unstale":"2024-09-17T03:55:11.424410+0000","last_undegraded":"2024-09-17T03:55:11.424410+0000","last_fullsized":"2024-09-17T03:55:11.424410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:31.031639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425062+0000","last_change":"2024-09-17T03:55:10.109256+0000","last_active":"2024-09-17T03:55:11.425062+0000","last_peered":"2024-09-17T03:55:11.425062+0000","last_clean":"2024-09-17T03:55:11.425062+0000","last_became_active":"2024-09-17T03:55:10.108832+0000","last_became_peered":"2024-09-17T03:55:10.108832+0000","last_unstale":"2024-09-17T03:55:11.425062+0000","last_undegraded":"2024-09-17T03:55:11.425062+0000","last_fullsized":"2024-09-17T03:55:11.425062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:16.522311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424801+0000","last_change":"2024-09-17T03:55:10.112919+0000","last_active":"2024-09-17T03:55:11.424801+0000","last_peered":"2024-09-17T03:55:11.424801+0000","last_clean":"2024-09-17T03:55:11.424801+0000","last_became_active":"2024-09-17T03:55:10.112583+0000","last_became_peered":"2024-09-17T03:55:10.112583+0000","last_unstale":"2024-09-17T03:55:11.424801+0000","last_undegraded":"2024-09-17T03:55:11.424801+0000","last_fullsized":"2024-09-17T03:55:11.424801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:27.443267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142438+0000","last_change":"2024-09-17T03:55:10.112948+0000","last_active":"2024-09-17T03:55:11.142438+0000","last_peered":"2024-09-17T03:55:11.142438+0000","last_clean":"2024-09-17T03:55:11.142438+0000","last_became_active":"2024-09-17T03:55:10.112642+0000","last_became_peered":"2024-09-17T03:55:10.112642+0000","last_unstale":"2024-09-17T03:55:11.142438+0000","last_undegraded":"2024-09-17T03:55:11.142438+0000","last_fullsized":"2024-09-17T03:55:11.142438+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:40.241593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423770+0000","last_change":"2024-09-17T03:55:10.094106+0000","last_active":"2024-09-17T03:55:11.423770+0000","last_peered":"2024-09-17T03:55:11.423770+0000","last_clean":"2024-09-17T03:55:11.423770+0000","last_became_active":"2024-09-17T03:55:10.093763+0000","last_became_peered":"2024-09-17T03:55:10.093763+0000","last_unstale":"2024-09-17T03:55:11.423770+0000","last_undegraded":"2024-09-17T03:55:11.423770+0000","last_fullsized":"2024-09-17T03:55:11.423770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.835238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426449+0000","last_change":"2024-09-17T03:55:10.101103+0000","last_active":"2024-09-17T03:55:11.426449+0000","last_peered":"2024-09-17T03:55:11.426449+0000","last_clean":"2024-09-17T03:55:11.426449+0000","last_became_active":"2024-09-17T03:55:10.100775+0000","last_became_peered":"2024-09-17T03:55:10.100775+0000","last_unstale":"2024-09-17T03:55:11.426449+0000","last_undegraded":"2024-09-17T03:55:11.426449+0000","last_fullsized":"2024-09-17T03:55:11.426449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426439+0000","last_change":"2024-09-17T03:55:10.100108+0000","last_active":"2024-09-17T03:55:11.426439+0000","last_peered":"2024-09-17T03:55:11.426439+0000","last_clean":"2024-09-17T03:55:11.426439+0000","last_became_active":"2024-09-17T03:55:10.099786+0000","last_became_peered":"2024-09-17T03:55:10.099786+0000","last_unstale":"2024-09-17T03:55:11.426439+0000","last_undegraded":"2024-09-17T03:55:11.426439+0000","last_fullsized":"2024-09-17T03:55:11.426439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:00.641678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425612+0000","last_change":"2024-09-17T03:55:10.109163+0000","last_active":"2024-09-17T03:55:11.425612+0000","last_peered":"2024-09-17T03:55:11.425612+0000","last_clean":"2024-09-17T03:55:11.425612+0000","last_became_active":"2024-09-17T03:55:10.108810+0000","last_became_peered":"2024-09-17T03:55:10.108810+0000","last_unstale":"2024-09-17T03:55:11.425612+0000","last_undegraded":"2024-09-17T03:55:11.425612+0000","last_fullsized":"2024-09-17T03:55:11.425612+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:32.706728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426258+0000","last_change":"2024-09-17T03:55:10.102425+0000","last_active":"2024-09-17T03:55:11.426258+0000","last_peered":"2024-09-17T03:55:11.426258+0000","last_clean":"2024-09-17T03:55:11.426258+0000","last_became_active":"2024-09-17T03:55:10.102086+0000","last_became_peered":"2024-09-17T03:55:10.102086+0000","last_unstale":"2024-09-17T03:55:11.426258+0000","last_undegraded":"2024-09-17T03:55:11.426258+0000","last_fullsized":"2024-09-17T03:55:11.426258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426586+0000","last_change":"2024-09-17T03:55:10.100194+0000","last_active":"2024-09-17T03:55:11.426586+0000","last_peered":"2024-09-17T03:55:11.426586+0000","last_clean":"2024-09-17T03:55:11.426586+0000","last_became_active":"2024-09-17T03:55:10.099857+0000","last_became_peered":"2024-09-17T03:55:10.099857+0000","last_unstale":"2024-09-17T03:55:11.426586+0000","last_undegraded":"2024-09-17T03:55:11.426586+0000","last_fullsized":"2024-09-17T03:55:11.426586+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100819+0000","last_change":"2024-09-17T03:55:10.129746+0000","last_active":"2024-09-17T03:55:11.100819+0000","last_peered":"2024-09-17T03:55:11.100819+0000","last_clean":"2024-09-17T03:55:11.100819+0000","last_became_active":"2024-09-17T03:55:10.124450+0000","last_became_peered":"2024-09-17T03:55:10.124450+0000","last_unstale":"2024-09-17T03:55:11.100819+0000","last_undegraded":"2024-09-17T03:55:11.100819+0000","last_fullsized":"2024-09-17T03:55:11.100819+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425299+0000","last_change":"2024-09-17T03:55:10.097618+0000","last_active":"2024-09-17T03:55:11.425299+0000","last_peered":"2024-09-17T03:55:11.425299+0000","last_clean":"2024-09-17T03:55:11.425299+0000","last_became_active":"2024-09-17T03:55:10.097328+0000","last_became_peered":"2024-09-17T03:55:10.097328+0000","last_unstale":"2024-09-17T03:55:11.425299+0000","last_undegraded":"2024-09-17T03:55:11.425299+0000","last_fullsized":"2024-09-17T03:55:11.425299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:23.303493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100379+0000","last_change":"2024-09-17T03:55:10.107998+0000","last_active":"2024-09-17T03:55:11.100379+0000","last_peered":"2024-09-17T03:55:11.100379+0000","last_clean":"2024-09-17T03:55:11.100379+0000","last_became_active":"2024-09-17T03:55:10.107395+0000","last_became_peered":"2024-09-17T03:55:10.107395+0000","last_unstale":"2024-09-17T03:55:11.100379+0000","last_undegraded":"2024-09-17T03:55:11.100379+0000","last_fullsized":"2024-09-17T03:55:11.100379+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426752+0000","last_change":"2024-09-17T03:55:10.098678+0000","last_active":"2024-09-17T03:55:11.426752+0000","last_peered":"2024-09-17T03:55:11.426752+0000","last_clean":"2024-09-17T03:55:11.426752+0000","last_became_active":"2024-09-17T03:55:10.098318+0000","last_became_peered":"2024-09-17T03:55:10.098318+0000","last_unstale":"2024-09-17T03:55:11.426752+0000","last_undegraded":"2024-09-17T03:55:11.426752+0000","last_fullsized":"2024-09-17T03:55:11.426752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424684+0000","last_change":"2024-09-17T03:55:10.102840+0000","last_active":"2024-09-17T03:55:11.424684+0000","last_peered":"2024-09-17T03:55:11.424684+0000","last_clean":"2024-09-17T03:55:11.424684+0000","last_became_active":"2024-09-17T03:55:10.100850+0000","last_became_peered":"2024-09-17T03:55:10.100850+0000","last_unstale":"2024-09-17T03:55:11.424684+0000","last_undegraded":"2024-09-17T03:55:11.424684+0000","last_fullsized":"2024-09-17T03:55:11.424684+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:17.813222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142724+0000","last_change":"2024-09-17T03:55:10.122268+0000","last_active":"2024-09-17T03:55:11.142724+0000","last_peered":"2024-09-17T03:55:11.142724+0000","last_clean":"2024-09-17T03:55:11.142724+0000","last_became_active":"2024-09-17T03:55:10.121813+0000","last_became_peered":"2024-09-17T03:55:10.121813+0000","last_unstale":"2024-09-17T03:55:11.142724+0000","last_undegraded":"2024-09-17T03:55:11.142724+0000","last_fullsized":"2024-09-17T03:55:11.142724+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:14.202711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100713+0000","last_change":"2024-09-17T03:55:10.126728+0000","last_active":"2024-09-17T03:55:11.100713+0000","last_peered":"2024-09-17T03:55:11.100713+0000","last_clean":"2024-09-17T03:55:11.100713+0000","last_became_active":"2024-09-17T03:55:10.126437+0000","last_became_peered":"2024-09-17T03:55:10.126437+0000","last_unstale":"2024-09-17T03:55:11.100713+0000","last_undegraded":"2024-09-17T03:55:11.100713+0000","last_fullsized":"2024-09-17T03:55:11.100713+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425371+0000","last_change":"2024-09-17T03:55:10.103912+0000","last_active":"2024-09-17T03:55:11.425371+0000","last_peered":"2024-09-17T03:55:11.425371+0000","last_clean":"2024-09-17T03:55:11.425371+0000","last_became_active":"2024-09-17T03:55:10.103569+0000","last_became_peered":"2024-09-17T03:55:10.103569+0000","last_unstale":"2024-09-17T03:55:11.425371+0000","last_undegraded":"2024-09-17T03:55:11.425371+0000","last_fullsized":"2024-09-17T03:55:11.425371+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:36.412867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104245+0000","last_change":"2024-09-17T03:55:10.111756+0000","last_active":"2024-09-17T03:55:11.104245+0000","last_peered":"2024-09-17T03:55:11.104245+0000","last_clean":"2024-09-17T03:55:11.104245+0000","last_became_active":"2024-09-17T03:55:10.111350+0000","last_became_peered":"2024-09-17T03:55:10.111350+0000","last_unstale":"2024-09-17T03:55:11.104245+0000","last_undegraded":"2024-09-17T03:55:11.104245+0000","last_fullsized":"2024-09-17T03:55:11.104245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423827+0000","last_change":"2024-09-17T03:55:10.110031+0000","last_active":"2024-09-17T03:55:11.423827+0000","last_peered":"2024-09-17T03:55:11.423827+0000","last_clean":"2024-09-17T03:55:11.423827+0000","last_became_active":"2024-09-17T03:55:10.106150+0000","last_became_peered":"2024-09-17T03:55:10.106150+0000","last_unstale":"2024-09-17T03:55:11.423827+0000","last_undegraded":"2024-09-17T03:55:11.423827+0000","last_fullsized":"2024-09-17T03:55:11.423827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:15.416205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142425+0000","last_change":"2024-09-17T03:55:10.114672+0000","last_active":"2024-09-17T03:55:11.142425+0000","last_peered":"2024-09-17T03:55:11.142425+0000","last_clean":"2024-09-17T03:55:11.142425+0000","last_became_active":"2024-09-17T03:55:10.114304+0000","last_became_peered":"2024-09-17T03:55:10.114304+0000","last_unstale":"2024-09-17T03:55:11.142425+0000","last_undegraded":"2024-09-17T03:55:11.142425+0000","last_fullsized":"2024-09-17T03:55:11.142425+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:59.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142860+0000","last_change":"2024-09-17T03:55:10.118321+0000","last_active":"2024-09-17T03:55:11.142860+0000","last_peered":"2024-09-17T03:55:11.142860+0000","last_clean":"2024-09-17T03:55:11.142860+0000","last_became_active":"2024-09-17T03:55:10.118001+0000","last_became_peered":"2024-09-17T03:55:10.118001+0000","last_unstale":"2024-09-17T03:55:11.142860+0000","last_undegraded":"2024-09-17T03:55:11.142860+0000","last_fullsized":"2024-09-17T03:55:11.142860+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":19,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104474+0000","last_change":"2024-09-17T03:55:09.098888+0000","last_active":"2024-09-17T03:55:11.104474+0000","last_peered":"2024-09-17T03:55:11.104474+0000","last_clean":"2024-09-17T03:55:11.104474+0000","last_became_active":"2024-09-17T03:55:09.098488+0000","last_became_peered":"2024-09-17T03:55:09.098488+0000","last_unstale":"2024-09-17T03:55:11.104474+0000","last_undegraded":"2024-09-17T03:55:11.104474+0000","last_fullsized":"2024-09-17T03:55:11.104474+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:43.529970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426303+0000","last_change":"2024-09-17T03:55:10.098038+0000","last_active":"2024-09-17T03:55:11.426303+0000","last_peered":"2024-09-17T03:55:11.426303+0000","last_clean":"2024-09-17T03:55:11.426303+0000","last_became_active":"2024-09-17T03:55:10.097706+0000","last_became_peered":"2024-09-17T03:55:10.097706+0000","last_unstale":"2024-09-17T03:55:11.426303+0000","last_undegraded":"2024-09-17T03:55:11.426303+0000","last_fullsized":"2024-09-17T03:55:11.426303+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:48.538101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424685+0000","last_change":"2024-09-17T03:55:09.102970+0000","last_active":"2024-09-17T03:55:11.424685+0000","last_peered":"2024-09-17T03:55:11.424685+0000","last_clean":"2024-09-17T03:55:11.424685+0000","last_became_active":"2024-09-17T03:55:09.102558+0000","last_became_peered":"2024-09-17T03:55:09.102558+0000","last_unstale":"2024-09-17T03:55:11.424685+0000","last_undegraded":"2024-09-17T03:55:11.424685+0000","last_fullsized":"2024-09-17T03:55:11.424685+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:26.840025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142757+0000","last_change":"2024-09-17T03:55:10.126853+0000","last_active":"2024-09-17T03:55:11.142757+0000","last_peered":"2024-09-17T03:55:11.142757+0000","last_clean":"2024-09-17T03:55:11.142757+0000","last_became_active":"2024-09-17T03:55:10.123486+0000","last_became_peered":"2024-09-17T03:55:10.123486+0000","last_unstale":"2024-09-17T03:55:11.142757+0000","last_undegraded":"2024-09-17T03:55:11.142757+0000","last_fullsized":"2024-09-17T03:55:11.142757+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:31.237400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103305+0000","last_change":"2024-09-17T03:55:09.106007+0000","last_active":"2024-09-17T03:55:11.103305+0000","last_peered":"2024-09-17T03:55:11.103305+0000","last_clean":"2024-09-17T03:55:11.103305+0000","last_became_active":"2024-09-17T03:55:09.105510+0000","last_became_peered":"2024-09-17T03:55:09.105510+0000","last_unstale":"2024-09-17T03:55:11.103305+0000","last_undegraded":"2024-09-17T03:55:11.103305+0000","last_fullsized":"2024-09-17T03:55:11.103305+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:27:12.256566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425907+0000","last_change":"2024-09-17T03:55:10.111580+0000","last_active":"2024-09-17T03:55:11.425907+0000","last_peered":"2024-09-17T03:55:11.425907+0000","last_clean":"2024-09-17T03:55:11.425907+0000","last_became_active":"2024-09-17T03:55:10.111121+0000","last_became_peered":"2024-09-17T03:55:10.111121+0000","last_unstale":"2024-09-17T03:55:11.425907+0000","last_undegraded":"2024-09-17T03:55:11.425907+0000","last_fullsized":"2024-09-17T03:55:11.425907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103330+0000","last_change":"2024-09-17T03:55:09.109395+0000","last_active":"2024-09-17T03:55:11.103330+0000","last_peered":"2024-09-17T03:55:11.103330+0000","last_clean":"2024-09-17T03:55:11.103330+0000","last_became_active":"2024-09-17T03:55:09.107581+0000","last_became_peered":"2024-09-17T03:55:09.107581+0000","last_unstale":"2024-09-17T03:55:11.103330+0000","last_undegraded":"2024-09-17T03:55:11.103330+0000","last_fullsized":"2024-09-17T03:55:11.103330+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:55.628731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423981+0000","last_change":"2024-09-17T03:55:10.115350+0000","last_active":"2024-09-17T03:55:11.423981+0000","last_peered":"2024-09-17T03:55:11.423981+0000","last_clean":"2024-09-17T03:55:11.423981+0000","last_became_active":"2024-09-17T03:55:10.111606+0000","last_became_peered":"2024-09-17T03:55:10.111606+0000","last_unstale":"2024-09-17T03:55:11.423981+0000","last_undegraded":"2024-09-17T03:55:11.423981+0000","last_fullsized":"2024-09-17T03:55:11.423981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:26.916486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104020+0000","last_change":"2024-09-17T03:55:09.100552+0000","last_active":"2024-09-17T03:55:11.104020+0000","last_peered":"2024-09-17T03:55:11.104020+0000","last_clean":"2024-09-17T03:55:11.104020+0000","last_became_active":"2024-09-17T03:55:09.100230+0000","last_became_peered":"2024-09-17T03:55:09.100230+0000","last_unstale":"2024-09-17T03:55:11.104020+0000","last_undegraded":"2024-09-17T03:55:11.104020+0000","last_fullsized":"2024-09-17T03:55:11.104020+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:26.431547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425676+0000","last_change":"2024-09-17T03:55:10.125057+0000","last_active":"2024-09-17T03:55:11.425676+0000","last_peered":"2024-09-17T03:55:11.425676+0000","last_clean":"2024-09-17T03:55:11.425676+0000","last_became_active":"2024-09-17T03:55:10.124788+0000","last_became_peered":"2024-09-17T03:55:10.124788+0000","last_unstale":"2024-09-17T03:55:11.425676+0000","last_undegraded":"2024-09-17T03:55:11.425676+0000","last_fullsized":"2024-09-17T03:55:11.425676+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:49.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.101089+0000","last_change":"2024-09-17T03:55:09.098297+0000","last_active":"2024-09-17T03:55:11.101089+0000","last_peered":"2024-09-17T03:55:11.101089+0000","last_clean":"2024-09-17T03:55:11.101089+0000","last_became_active":"2024-09-17T03:55:09.097970+0000","last_became_peered":"2024-09-17T03:55:09.097970+0000","last_unstale":"2024-09-17T03:55:11.101089+0000","last_undegraded":"2024-09-17T03:55:11.101089+0000","last_fullsized":"2024-09-17T03:55:11.101089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:36.074034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424050+0000","last_change":"2024-09-17T03:55:10.122441+0000","last_active":"2024-09-17T03:55:11.424050+0000","last_peered":"2024-09-17T03:55:11.424050+0000","last_clean":"2024-09-17T03:55:11.424050+0000","last_became_active":"2024-09-17T03:55:10.121881+0000","last_became_peered":"2024-09-17T03:55:10.121881+0000","last_unstale":"2024-09-17T03:55:11.424050+0000","last_undegraded":"2024-09-17T03:55:11.424050+0000","last_fullsized":"2024-09-17T03:55:11.424050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.313711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425520+0000","last_change":"2024-09-17T03:55:09.101386+0000","last_active":"2024-09-17T03:55:11.425520+0000","last_peered":"2024-09-17T03:55:11.425520+0000","last_clean":"2024-09-17T03:55:11.425520+0000","last_became_active":"2024-09-17T03:55:09.101041+0000","last_became_peered":"2024-09-17T03:55:09.101041+0000","last_unstale":"2024-09-17T03:55:11.425520+0000","last_undegraded":"2024-09-17T03:55:11.425520+0000","last_fullsized":"2024-09-17T03:55:11.425520+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:07.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104041+0000","last_change":"2024-09-17T03:55:10.097916+0000","last_active":"2024-09-17T03:55:11.104041+0000","last_peered":"2024-09-17T03:55:11.104041+0000","last_clean":"2024-09-17T03:55:11.104041+0000","last_became_active":"2024-09-17T03:55:10.097595+0000","last_became_peered":"2024-09-17T03:55:10.097595+0000","last_unstale":"2024-09-17T03:55:11.104041+0000","last_undegraded":"2024-09-17T03:55:11.104041+0000","last_fullsized":"2024-09-17T03:55:11.104041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423852+0000","last_change":"2024-09-17T03:55:09.093148+0000","last_active":"2024-09-17T03:55:11.423852+0000","last_peered":"2024-09-17T03:55:11.423852+0000","last_clean":"2024-09-17T03:55:11.423852+0000","last_became_active":"2024-09-17T03:55:09.092860+0000","last_became_peered":"2024-09-17T03:55:09.092860+0000","last_unstale":"2024-09-17T03:55:11.423852+0000","last_undegraded":"2024-09-17T03:55:11.423852+0000","last_fullsized":"2024-09-17T03:55:11.423852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:11.814702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104560+0000","last_change":"2024-09-17T03:55:10.102328+0000","last_active":"2024-09-17T03:55:11.104560+0000","last_peered":"2024-09-17T03:55:11.104560+0000","last_clean":"2024-09-17T03:55:11.104560+0000","last_became_active":"2024-09-17T03:55:10.101937+0000","last_became_peered":"2024-09-17T03:55:10.101937+0000","last_unstale":"2024-09-17T03:55:11.104560+0000","last_undegraded":"2024-09-17T03:55:11.104560+0000","last_fullsized":"2024-09-17T03:55:11.104560+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:53.259242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102690+0000","last_change":"2024-09-17T03:55:09.110979+0000","last_active":"2024-09-17T03:55:11.102690+0000","last_peered":"2024-09-17T03:55:11.102690+0000","last_clean":"2024-09-17T03:55:11.102690+0000","last_became_active":"2024-09-17T03:55:09.110640+0000","last_became_peered":"2024-09-17T03:55:09.110640+0000","last_unstale":"2024-09-17T03:55:11.102690+0000","last_undegraded":"2024-09-17T03:55:11.102690+0000","last_fullsized":"2024-09-17T03:55:11.102690+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:28.520430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426564+0000","last_change":"2024-09-17T03:55:10.105276+0000","last_active":"2024-09-17T03:55:11.426564+0000","last_peered":"2024-09-17T03:55:11.426564+0000","last_clean":"2024-09-17T03:55:11.426564+0000","last_became_active":"2024-09-17T03:55:10.104973+0000","last_became_peered":"2024-09-17T03:55:10.104973+0000","last_unstale":"2024-09-17T03:55:11.426564+0000","last_undegraded":"2024-09-17T03:55:11.426564+0000","last_fullsized":"2024-09-17T03:55:11.426564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:37.836511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100549+0000","last_change":"2024-09-17T03:55:09.100586+0000","last_active":"2024-09-17T03:55:11.100549+0000","last_peered":"2024-09-17T03:55:11.100549+0000","last_clean":"2024-09-17T03:55:11.100549+0000","last_became_active":"2024-09-17T03:55:09.100303+0000","last_became_peered":"2024-09-17T03:55:09.100303+0000","last_unstale":"2024-09-17T03:55:11.100549+0000","last_undegraded":"2024-09-17T03:55:11.100549+0000","last_fullsized":"2024-09-17T03:55:11.100549+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:36.366891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100901+0000","last_change":"2024-09-17T03:55:10.127654+0000","last_active":"2024-09-17T03:55:11.100901+0000","last_peered":"2024-09-17T03:55:11.100901+0000","last_clean":"2024-09-17T03:55:11.100901+0000","last_became_active":"2024-09-17T03:55:10.115490+0000","last_became_peered":"2024-09-17T03:55:10.115490+0000","last_unstale":"2024-09-17T03:55:11.100901+0000","last_undegraded":"2024-09-17T03:55:11.100901+0000","last_fullsized":"2024-09-17T03:55:11.100901+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:50.875001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424336+0000","last_change":"2024-09-17T03:55:09.101674+0000","last_active":"2024-09-17T03:55:11.424336+0000","last_peered":"2024-09-17T03:55:11.424336+0000","last_clean":"2024-09-17T03:55:11.424336+0000","last_became_active":"2024-09-17T03:55:09.101311+0000","last_became_peered":"2024-09-17T03:55:09.101311+0000","last_unstale":"2024-09-17T03:55:11.424336+0000","last_undegraded":"2024-09-17T03:55:11.424336+0000","last_fullsized":"2024-09-17T03:55:11.424336+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:43.324925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.105085+0000","last_change":"2024-09-17T03:55:10.100605+0000","last_active":"2024-09-17T03:55:11.105085+0000","last_peered":"2024-09-17T03:55:11.105085+0000","last_clean":"2024-09-17T03:55:11.105085+0000","last_became_active":"2024-09-17T03:55:10.100264+0000","last_became_peered":"2024-09-17T03:55:10.100264+0000","last_unstale":"2024-09-17T03:55:11.105085+0000","last_undegraded":"2024-09-17T03:55:11.105085+0000","last_fullsized":"2024-09-17T03:55:11.105085+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:31.420107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":21,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424538+0000","last_change":"2024-09-17T03:55:09.102173+0000","last_active":"2024-09-17T03:55:11.424538+0000","last_peered":"2024-09-17T03:55:11.424538+0000","last_clean":"2024-09-17T03:55:11.424538+0000","last_became_active":"2024-09-17T03:55:09.101451+0000","last_became_peered":"2024-09-17T03:55:09.101451+0000","last_unstale":"2024-09-17T03:55:11.424538+0000","last_undegraded":"2024-09-17T03:55:11.424538+0000","last_fullsized":"2024-09-17T03:55:11.424538+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:16.929007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100938+0000","last_change":"2024-09-17T03:55:10.128915+0000","last_active":"2024-09-17T03:55:11.100938+0000","last_peered":"2024-09-17T03:55:11.100938+0000","last_clean":"2024-09-17T03:55:11.100938+0000","last_became_active":"2024-09-17T03:55:10.121387+0000","last_became_peered":"2024-09-17T03:55:10.121387+0000","last_unstale":"2024-09-17T03:55:11.100938+0000","last_undegraded":"2024-09-17T03:55:11.100938+0000","last_fullsized":"2024-09-17T03:55:11.100938+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142482+0000","last_change":"2024-09-17T03:55:09.101519+0000","last_active":"2024-09-17T03:55:11.142482+0000","last_peered":"2024-09-17T03:55:11.142482+0000","last_clean":"2024-09-17T03:55:11.142482+0000","last_became_active":"2024-09-17T03:55:09.101211+0000","last_became_peered":"2024-09-17T03:55:09.101211+0000","last_unstale":"2024-09-17T03:55:11.142482+0000","last_undegraded":"2024-09-17T03:55:11.142482+0000","last_fullsized":"2024-09-17T03:55:11.142482+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:58.858197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100665+0000","last_change":"2024-09-17T03:55:10.124544+0000","last_active":"2024-09-17T03:55:11.100665+0000","last_peered":"2024-09-17T03:55:11.100665+0000","last_clean":"2024-09-17T03:55:11.100665+0000","last_became_active":"2024-09-17T03:55:10.124209+0000","last_became_peered":"2024-09-17T03:55:10.124209+0000","last_unstale":"2024-09-17T03:55:11.100665+0000","last_undegraded":"2024-09-17T03:55:11.100665+0000","last_fullsized":"2024-09-17T03:55:11.100665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424612+0000","last_change":"2024-09-17T03:55:09.103112+0000","last_active":"2024-09-17T03:55:11.424612+0000","last_peered":"2024-09-17T03:55:11.424612+0000","last_clean":"2024-09-17T03:55:11.424612+0000","last_became_active":"2024-09-17T03:55:09.102182+0000","last_became_peered":"2024-09-17T03:55:09.102182+0000","last_unstale":"2024-09-17T03:55:11.424612+0000","last_undegraded":"2024-09-17T03:55:11.424612+0000","last_fullsized":"2024-09-17T03:55:11.424612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:53:55.298078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.101033+0000","last_change":"2024-09-17T03:55:10.120468+0000","last_active":"2024-09-17T03:55:11.101033+0000","last_peered":"2024-09-17T03:55:11.101033+0000","last_clean":"2024-09-17T03:55:11.101033+0000","last_became_active":"2024-09-17T03:55:10.119946+0000","last_became_peered":"2024-09-17T03:55:10.119946+0000","last_unstale":"2024-09-17T03:55:11.101033+0000","last_undegraded":"2024-09-17T03:55:11.101033+0000","last_fullsized":"2024-09-17T03:55:11.101033+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:48:16.242686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"106'8","reported_seq":23,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102570+0000","last_change":"2024-09-17T03:55:09.110547+0000","last_active":"2024-09-17T03:55:11.102570+0000","last_peered":"2024-09-17T03:55:11.102570+0000","last_clean":"2024-09-17T03:55:11.102570+0000","last_became_active":"2024-09-17T03:55:09.109930+0000","last_became_peered":"2024-09-17T03:55:09.109930+0000","last_unstale":"2024-09-17T03:55:11.102570+0000","last_undegraded":"2024-09-17T03:55:11.102570+0000","last_fullsized":"2024-09-17T03:55:11.102570+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.646247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102906+0000","last_change":"2024-09-17T03:55:10.108164+0000","last_active":"2024-09-17T03:55:11.102906+0000","last_peered":"2024-09-17T03:55:11.102906+0000","last_clean":"2024-09-17T03:55:11.102906+0000","last_became_active":"2024-09-17T03:55:10.106709+0000","last_became_peered":"2024-09-17T03:55:10.106709+0000","last_unstale":"2024-09-17T03:55:11.102906+0000","last_undegraded":"2024-09-17T03:55:11.102906+0000","last_fullsized":"2024-09-17T03:55:11.102906+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:34.367904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424999+0000","last_change":"2024-09-17T03:55:09.102865+0000","last_active":"2024-09-17T03:55:11.424999+0000","last_peered":"2024-09-17T03:55:11.424999+0000","last_clean":"2024-09-17T03:55:11.424999+0000","last_became_active":"2024-09-17T03:55:09.100408+0000","last_became_peered":"2024-09-17T03:55:09.100408+0000","last_unstale":"2024-09-17T03:55:11.424999+0000","last_undegraded":"2024-09-17T03:55:11.424999+0000","last_fullsized":"2024-09-17T03:55:11.424999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:29.269455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142886+0000","last_change":"2024-09-17T03:55:10.113442+0000","last_active":"2024-09-17T03:55:11.142886+0000","last_peered":"2024-09-17T03:55:11.142886+0000","last_clean":"2024-09-17T03:55:11.142886+0000","last_became_active":"2024-09-17T03:55:10.113158+0000","last_became_peered":"2024-09-17T03:55:10.113158+0000","last_unstale":"2024-09-17T03:55:11.142886+0000","last_undegraded":"2024-09-17T03:55:11.142886+0000","last_fullsized":"2024-09-17T03:55:11.142886+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103008+0000","last_change":"2024-09-17T03:55:09.106143+0000","last_active":"2024-09-17T03:55:11.103008+0000","last_peered":"2024-09-17T03:55:11.103008+0000","last_clean":"2024-09-17T03:55:11.103008+0000","last_became_active":"2024-09-17T03:55:09.105539+0000","last_became_peered":"2024-09-17T03:55:09.105539+0000","last_unstale":"2024-09-17T03:55:11.103008+0000","last_undegraded":"2024-09-17T03:55:11.103008+0000","last_fullsized":"2024-09-17T03:55:11.103008+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:29.106359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425055+0000","last_change":"2024-09-17T03:55:10.118502+0000","last_active":"2024-09-17T03:55:11.425055+0000","last_peered":"2024-09-17T03:55:11.425055+0000","last_clean":"2024-09-17T03:55:11.425055+0000","last_became_active":"2024-09-17T03:55:10.118276+0000","last_became_peered":"2024-09-17T03:55:10.118276+0000","last_unstale":"2024-09-17T03:55:11.425055+0000","last_undegraded":"2024-09-17T03:55:11.425055+0000","last_fullsized":"2024-09-17T03:55:11.425055+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:23.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425392+0000","last_change":"2024-09-17T03:55:09.099409+0000","last_active":"2024-09-17T03:55:11.425392+0000","last_peered":"2024-09-17T03:55:11.425392+0000","last_clean":"2024-09-17T03:55:11.425392+0000","last_became_active":"2024-09-17T03:55:09.097894+0000","last_became_peered":"2024-09-17T03:55:09.097894+0000","last_unstale":"2024-09-17T03:55:11.425392+0000","last_undegraded":"2024-09-17T03:55:11.425392+0000","last_fullsized":"2024-09-17T03:55:11.425392+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:30.592901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426005+0000","last_change":"2024-09-17T03:55:10.101315+0000","last_active":"2024-09-17T03:55:11.426005+0000","last_peered":"2024-09-17T03:55:11.426005+0000","last_clean":"2024-09-17T03:55:11.426005+0000","last_became_active":"2024-09-17T03:55:10.100936+0000","last_became_peered":"2024-09-17T03:55:10.100936+0000","last_unstale":"2024-09-17T03:55:11.426005+0000","last_undegraded":"2024-09-17T03:55:11.426005+0000","last_fullsized":"2024-09-17T03:55:11.426005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:51.665179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104091+0000","last_change":"2024-09-17T03:55:09.095974+0000","last_active":"2024-09-17T03:55:11.104091+0000","last_peered":"2024-09-17T03:55:11.104091+0000","last_clean":"2024-09-17T03:55:11.104091+0000","last_became_active":"2024-09-17T03:55:09.095632+0000","last_became_peered":"2024-09-17T03:55:09.095632+0000","last_unstale":"2024-09-17T03:55:11.104091+0000","last_undegraded":"2024-09-17T03:55:11.104091+0000","last_fullsized":"2024-09-17T03:55:11.104091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.401993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424900+0000","last_change":"2024-09-17T03:55:10.104348+0000","last_active":"2024-09-17T03:55:11.424900+0000","last_peered":"2024-09-17T03:55:11.424900+0000","last_clean":"2024-09-17T03:55:11.424900+0000","last_became_active":"2024-09-17T03:55:10.104019+0000","last_became_peered":"2024-09-17T03:55:10.104019+0000","last_unstale":"2024-09-17T03:55:11.424900+0000","last_undegraded":"2024-09-17T03:55:11.424900+0000","last_fullsized":"2024-09-17T03:55:11.424900+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.618440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142778+0000","last_change":"2024-09-17T03:55:09.107009+0000","last_active":"2024-09-17T03:55:11.142778+0000","last_peered":"2024-09-17T03:55:11.142778+0000","last_clean":"2024-09-17T03:55:11.142778+0000","last_became_active":"2024-09-17T03:55:09.106706+0000","last_became_peered":"2024-09-17T03:55:09.106706+0000","last_unstale":"2024-09-17T03:55:11.142778+0000","last_undegraded":"2024-09-17T03:55:11.142778+0000","last_fullsized":"2024-09-17T03:55:11.142778+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:35:46.653486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424019+0000","last_change":"2024-09-17T03:55:10.101801+0000","last_active":"2024-09-17T03:55:11.424019+0000","last_peered":"2024-09-17T03:55:11.424019+0000","last_clean":"2024-09-17T03:55:11.424019+0000","last_became_active":"2024-09-17T03:55:10.101240+0000","last_became_peered":"2024-09-17T03:55:10.101240+0000","last_unstale":"2024-09-17T03:55:11.424019+0000","last_undegraded":"2024-09-17T03:55:11.424019+0000","last_fullsized":"2024-09-17T03:55:11.424019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:08.177233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102580+0000","last_change":"2024-09-17T03:55:09.105469+0000","last_active":"2024-09-17T03:55:11.102580+0000","last_peered":"2024-09-17T03:55:11.102580+0000","last_clean":"2024-09-17T03:55:11.102580+0000","last_became_active":"2024-09-17T03:55:09.105074+0000","last_became_peered":"2024-09-17T03:55:09.105074+0000","last_unstale":"2024-09-17T03:55:11.102580+0000","last_undegraded":"2024-09-17T03:55:11.102580+0000","last_fullsized":"2024-09-17T03:55:11.102580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:30.326152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425201+0000","last_change":"2024-09-17T03:55:10.110424+0000","last_active":"2024-09-17T03:55:11.425201+0000","last_peered":"2024-09-17T03:55:11.425201+0000","last_clean":"2024-09-17T03:55:11.425201+0000","last_became_active":"2024-09-17T03:55:10.110057+0000","last_became_peered":"2024-09-17T03:55:10.110057+0000","last_unstale":"2024-09-17T03:55:11.425201+0000","last_undegraded":"2024-09-17T03:55:11.425201+0000","last_fullsized":"2024-09-17T03:55:11.425201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:40.073222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142451+0000","last_change":"2024-09-17T03:55:09.109486+0000","last_active":"2024-09-17T03:55:11.142451+0000","last_peered":"2024-09-17T03:55:11.142451+0000","last_clean":"2024-09-17T03:55:11.142451+0000","last_became_active":"2024-09-17T03:55:09.108902+0000","last_became_peered":"2024-09-17T03:55:09.108902+0000","last_unstale":"2024-09-17T03:55:11.142451+0000","last_undegraded":"2024-09-17T03:55:11.142451+0000","last_fullsized":"2024-09-17T03:55:11.142451+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:07:20.685338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100907+0000","last_change":"2024-09-17T03:55:10.123697+0000","last_active":"2024-09-17T03:55:11.100907+0000","last_peered":"2024-09-17T03:55:11.100907+0000","last_clean":"2024-09-17T03:55:11.100907+0000","last_became_active":"2024-09-17T03:55:10.123389+0000","last_became_peered":"2024-09-17T03:55:10.123389+0000","last_unstale":"2024-09-17T03:55:11.100907+0000","last_undegraded":"2024-09-17T03:55:11.100907+0000","last_fullsized":"2024-09-17T03:55:11.100907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:01.009413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423957+0000","last_change":"2024-09-17T03:55:09.089727+0000","last_active":"2024-09-17T03:55:11.423957+0000","last_peered":"2024-09-17T03:55:11.423957+0000","last_clean":"2024-09-17T03:55:11.423957+0000","last_became_active":"2024-09-17T03:55:09.089416+0000","last_became_peered":"2024-09-17T03:55:09.089416+0000","last_unstale":"2024-09-17T03:55:11.423957+0000","last_undegraded":"2024-09-17T03:55:11.423957+0000","last_fullsized":"2024-09-17T03:55:11.423957+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:19.704845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426516+0000","last_change":"2024-09-17T03:55:10.106649+0000","last_active":"2024-09-17T03:55:11.426516+0000","last_peered":"2024-09-17T03:55:11.426516+0000","last_clean":"2024-09-17T03:55:11.426516+0000","last_became_active":"2024-09-17T03:55:10.106347+0000","last_became_peered":"2024-09-17T03:55:10.106347+0000","last_unstale":"2024-09-17T03:55:11.426516+0000","last_undegraded":"2024-09-17T03:55:11.426516+0000","last_fullsized":"2024-09-17T03:55:11.426516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:21.848148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100730+0000","last_change":"2024-09-17T03:55:09.106794+0000","last_active":"2024-09-17T03:55:11.100730+0000","last_peered":"2024-09-17T03:55:11.100730+0000","last_clean":"2024-09-17T03:55:11.100730+0000","last_became_active":"2024-09-17T03:55:09.106461+0000","last_became_peered":"2024-09-17T03:55:09.106461+0000","last_unstale":"2024-09-17T03:55:11.100730+0000","last_undegraded":"2024-09-17T03:55:11.100730+0000","last_fullsized":"2024-09-17T03:55:11.100730+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:40.245287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424964+0000","last_change":"2024-09-17T03:55:10.111117+0000","last_active":"2024-09-17T03:55:11.424964+0000","last_peered":"2024-09-17T03:55:11.424964+0000","last_clean":"2024-09-17T03:55:11.424964+0000","last_became_active":"2024-09-17T03:55:10.110769+0000","last_became_peered":"2024-09-17T03:55:10.110769+0000","last_unstale":"2024-09-17T03:55:11.424964+0000","last_undegraded":"2024-09-17T03:55:11.424964+0000","last_fullsized":"2024-09-17T03:55:11.424964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:34.220767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426576+0000","last_change":"2024-09-17T03:55:09.091992+0000","last_active":"2024-09-17T03:55:11.426576+0000","last_peered":"2024-09-17T03:55:11.426576+0000","last_clean":"2024-09-17T03:55:11.426576+0000","last_became_active":"2024-09-17T03:55:09.091655+0000","last_became_peered":"2024-09-17T03:55:09.091655+0000","last_unstale":"2024-09-17T03:55:11.426576+0000","last_undegraded":"2024-09-17T03:55:11.426576+0000","last_fullsized":"2024-09-17T03:55:11.426576+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:14.155619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424655+0000","last_change":"2024-09-17T03:55:10.125141+0000","last_active":"2024-09-17T03:55:11.424655+0000","last_peered":"2024-09-17T03:55:11.424655+0000","last_clean":"2024-09-17T03:55:11.424655+0000","last_became_active":"2024-09-17T03:55:10.124839+0000","last_became_peered":"2024-09-17T03:55:10.124839+0000","last_unstale":"2024-09-17T03:55:11.424655+0000","last_undegraded":"2024-09-17T03:55:11.424655+0000","last_fullsized":"2024-09-17T03:55:11.424655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:49.605272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424463+0000","last_change":"2024-09-17T03:55:09.092456+0000","last_active":"2024-09-17T03:55:11.424463+0000","last_peered":"2024-09-17T03:55:11.424463+0000","last_clean":"2024-09-17T03:55:11.424463+0000","last_became_active":"2024-09-17T03:55:09.092133+0000","last_became_peered":"2024-09-17T03:55:09.092133+0000","last_unstale":"2024-09-17T03:55:11.424463+0000","last_undegraded":"2024-09-17T03:55:11.424463+0000","last_fullsized":"2024-09-17T03:55:11.424463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:29.592608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425399+0000","last_change":"2024-09-17T03:55:10.105263+0000","last_active":"2024-09-17T03:55:11.425399+0000","last_peered":"2024-09-17T03:55:11.425399+0000","last_clean":"2024-09-17T03:55:11.425399+0000","last_became_active":"2024-09-17T03:55:10.104878+0000","last_became_peered":"2024-09-17T03:55:10.104878+0000","last_unstale":"2024-09-17T03:55:11.425399+0000","last_undegraded":"2024-09-17T03:55:11.425399+0000","last_fullsized":"2024-09-17T03:55:11.425399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:29.686662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100964+0000","last_change":"2024-09-17T03:55:09.101334+0000","last_active":"2024-09-17T03:55:11.100964+0000","last_peered":"2024-09-17T03:55:11.100964+0000","last_clean":"2024-09-17T03:55:11.100964+0000","last_became_active":"2024-09-17T03:55:09.101016+0000","last_became_peered":"2024-09-17T03:55:09.101016+0000","last_unstale":"2024-09-17T03:55:11.100964+0000","last_undegraded":"2024-09-17T03:55:11.100964+0000","last_fullsized":"2024-09-17T03:55:11.100964+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:03.790620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425931+0000","last_change":"2024-09-17T03:55:10.097057+0000","last_active":"2024-09-17T03:55:11.425931+0000","last_peered":"2024-09-17T03:55:11.425931+0000","last_clean":"2024-09-17T03:55:11.425931+0000","last_became_active":"2024-09-17T03:55:10.096210+0000","last_became_peered":"2024-09-17T03:55:10.096210+0000","last_unstale":"2024-09-17T03:55:11.425931+0000","last_undegraded":"2024-09-17T03:55:11.425931+0000","last_fullsized":"2024-09-17T03:55:11.425931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:04.260938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104187+0000","last_change":"2024-09-17T03:55:09.099774+0000","last_active":"2024-09-17T03:55:11.104187+0000","last_peered":"2024-09-17T03:55:11.104187+0000","last_clean":"2024-09-17T03:55:11.104187+0000","last_became_active":"2024-09-17T03:55:09.099422+0000","last_became_peered":"2024-09-17T03:55:09.099422+0000","last_unstale":"2024-09-17T03:55:11.104187+0000","last_undegraded":"2024-09-17T03:55:11.104187+0000","last_fullsized":"2024-09-17T03:55:11.104187+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:56.088001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426331+0000","last_change":"2024-09-17T03:55:10.112375+0000","last_active":"2024-09-17T03:55:11.426331+0000","last_peered":"2024-09-17T03:55:11.426331+0000","last_clean":"2024-09-17T03:55:11.426331+0000","last_became_active":"2024-09-17T03:55:10.112033+0000","last_became_peered":"2024-09-17T03:55:10.112033+0000","last_unstale":"2024-09-17T03:55:11.426331+0000","last_undegraded":"2024-09-17T03:55:11.426331+0000","last_fullsized":"2024-09-17T03:55:11.426331+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426160+0000","last_change":"2024-09-17T03:55:09.103012+0000","last_active":"2024-09-17T03:55:11.426160+0000","last_peered":"2024-09-17T03:55:11.426160+0000","last_clean":"2024-09-17T03:55:11.426160+0000","last_became_active":"2024-09-17T03:55:09.102701+0000","last_became_peered":"2024-09-17T03:55:09.102701+0000","last_unstale":"2024-09-17T03:55:11.426160+0000","last_undegraded":"2024-09-17T03:55:11.426160+0000","last_fullsized":"2024-09-17T03:55:11.426160+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:02.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425468+0000","last_change":"2024-09-17T03:55:10.113187+0000","last_active":"2024-09-17T03:55:11.425468+0000","last_peered":"2024-09-17T03:55:11.425468+0000","last_clean":"2024-09-17T03:55:11.425468+0000","last_became_active":"2024-09-17T03:55:10.112812+0000","last_became_peered":"2024-09-17T03:55:10.112812+0000","last_unstale":"2024-09-17T03:55:11.425468+0000","last_undegraded":"2024-09-17T03:55:11.425468+0000","last_fullsized":"2024-09-17T03:55:11.425468+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:18.693960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102640+0000","last_change":"2024-09-17T03:55:09.108670+0000","last_active":"2024-09-17T03:55:11.102640+0000","last_peered":"2024-09-17T03:55:11.102640+0000","last_clean":"2024-09-17T03:55:11.102640+0000","last_became_active":"2024-09-17T03:55:09.106248+0000","last_became_peered":"2024-09-17T03:55:09.106248+0000","last_unstale":"2024-09-17T03:55:11.102640+0000","last_undegraded":"2024-09-17T03:55:11.102640+0000","last_fullsized":"2024-09-17T03:55:11.102640+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:02.604730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423930+0000","last_change":"2024-09-17T03:55:10.126012+0000","last_active":"2024-09-17T03:55:11.423930+0000","last_peered":"2024-09-17T03:55:11.423930+0000","last_clean":"2024-09-17T03:55:11.423930+0000","last_became_active":"2024-09-17T03:55:10.125592+0000","last_became_peered":"2024-09-17T03:55:10.125592+0000","last_unstale":"2024-09-17T03:55:11.423930+0000","last_undegraded":"2024-09-17T03:55:11.423930+0000","last_fullsized":"2024-09-17T03:55:11.423930+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:17.583329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424054+0000","last_change":"2024-09-17T03:55:09.096029+0000","last_active":"2024-09-17T03:55:11.424054+0000","last_peered":"2024-09-17T03:55:11.424054+0000","last_clean":"2024-09-17T03:55:11.424054+0000","last_became_active":"2024-09-17T03:55:09.091782+0000","last_became_peered":"2024-09-17T03:55:09.091782+0000","last_unstale":"2024-09-17T03:55:11.424054+0000","last_undegraded":"2024-09-17T03:55:11.424054+0000","last_fullsized":"2024-09-17T03:55:11.424054+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:17.583743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100467+0000","last_change":"2024-09-17T03:55:10.105398+0000","last_active":"2024-09-17T03:55:11.100467+0000","last_peered":"2024-09-17T03:55:11.100467+0000","last_clean":"2024-09-17T03:55:11.100467+0000","last_became_active":"2024-09-17T03:55:10.105037+0000","last_became_peered":"2024-09-17T03:55:10.105037+0000","last_unstale":"2024-09-17T03:55:11.100467+0000","last_undegraded":"2024-09-17T03:55:11.100467+0000","last_fullsized":"2024-09-17T03:55:11.100467+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104276+0000","last_change":"2024-09-17T03:55:09.096214+0000","last_active":"2024-09-17T03:55:11.104276+0000","last_peered":"2024-09-17T03:55:11.104276+0000","last_clean":"2024-09-17T03:55:11.104276+0000","last_became_active":"2024-09-17T03:55:09.095625+0000","last_became_peered":"2024-09-17T03:55:09.095625+0000","last_unstale":"2024-09-17T03:55:11.104276+0000","last_undegraded":"2024-09-17T03:55:11.104276+0000","last_fullsized":"2024-09-17T03:55:11.104276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:12.569897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":265,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425604+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:55:11.425604+0000","last_peered":"2024-09-17T03:55:11.425604+0000","last_clean":"2024-09-17T03:55:11.425604+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:55:11.425604+0000","last_undegraded":"2024-09-17T03:55:11.425604+0000","last_fullsized":"2024-09-17T03:55:11.425604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1157630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":567250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509512,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1339392,"data_stored":568328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":566770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542216,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1327104,"data_stored":566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542217,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":1158738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:55:19.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 -- 172.21.15.129:0/866409686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 msgr2=0x7fd6bc0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 --2- 172.21.15.129:0/866409686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0b4af0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009830 tx=0x7fd6a402a480 comp rx=0 tx=0).stop 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 -- 172.21.15.129:0/866409686 shutdown_connections 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 --2- 172.21.15.129:0/866409686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0b4af0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 --2- 172.21.15.129:0/866409686 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 0x7fd6bc160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.917+0000 7fd6bb200640 1 -- 172.21.15.129:0/866409686 >> 172.21.15.129:0/866409686 conn(0x7fd6bc06cfc0 msgr2=0x7fd6bc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.918+0000 7fd6bb200640 1 -- 172.21.15.129:0/866409686 shutdown_connections 2024-09-17T03:55:19.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.918+0000 7fd6bb200640 1 -- 172.21.15.129:0/866409686 wait complete. 2024-09-17T03:55:19.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.920+0000 7fd6bb200640 1 Processor -- start 2024-09-17T03:55:19.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.920+0000 7fd6bb200640 1 -- start start 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 0x7fd6bc072d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d270 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d0f0 con 0x7fd6bc0b46e0 2024-09-17T03:55:19.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.921+0000 7fd6bb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd6bc15d3f0 con 0x7fd6bc0cfb60 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53174/0 (socket says 172.21.15.129:53174) 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 -- 172.21.15.129:0/1355639915 learned_addr learned my addr 172.21.15.129:0/1355639915 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6b1400640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 0x7fd6bc072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6b9e00640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 -- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 msgr2=0x7fd6bc072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 0x7fd6bc072d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 -- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 msgr2=0x7fd6bc0cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 -- 172.21.15.129:0/1355639915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6bc1589b0 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.922+0000 7fd6ba800640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400cb40 tx=0x7fd6b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.923+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4016020 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.923+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd6b400ad30 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.923+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6bc15b8b0 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.923+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6b4012680 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.923+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6bc15b0b0 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.924+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6bc076fb0 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.925+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd6b4019020 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.926+0000 7fd6b2a00640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 0x7fd69807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:19.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.927+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fd6b409b230 con 0x7fd6bc0d02c0 2024-09-17T03:55:19.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.927+0000 7fd6b9e00640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 0x7fd69807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:19.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.928+0000 7fd6b9e00640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 0x7fd69807b0a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b4450 tx=0x7fd6ac005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:19.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:19.931+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd6bc076fb0 con 0x7fd6bc0d02c0 2024-09-17T03:55:20.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.110+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd6bc158c90 con 0x7fd698078b90 2024-09-17T03:55:20.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.115+0000 7fd6b2a00640 1 -- 172.21.15.129:0/1355639915 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252448 (secure 0 0 0) 0x7fd6bc158c90 con 0x7fd698078b90 2024-09-17T03:55:20.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:20.120 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:55:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.119+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 msgr2=0x7fd69807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.119+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 0x7fd69807b0a0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0b4450 tx=0x7fd6ac005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.119+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 msgr2=0x7fd6bc073270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.119+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd6b400cb40 tx=0x7fd6b400a060 comp rx=0 tx=0).stop 2024-09-17T03:55:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.119+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 shutdown_connections 2024-09-17T03:55:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd698078b90 0x7fd69807b0a0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6bc0d02c0 0x7fd6bc073270 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd6bc0cfb60 0x7fd6bc072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 --2- 172.21.15.129:0/1355639915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd6bc0b46e0 0x7fd6bc0cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 >> 172.21.15.129:0/1355639915 conn(0x7fd6bc06cfc0 msgr2=0x7fd6bc157a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 shutdown_connections 2024-09-17T03:55:20.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.120+0000 7fd6bb200640 1 -- 172.21.15.129:0/1355639915 wait complete. 2024-09-17T03:55:20.143 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-17T03:55:18.388578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":302704,"kb_used_data":11560,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754672016,"statfs":{"total":773094113280,"available":772784144384,"internally_reserved":0,"allocated":11837440,"data_stored":5718724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.308127"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424841+0000","last_change":"2024-09-17T03:55:10.112479+0000","last_active":"2024-09-17T03:55:11.424841+0000","last_peered":"2024-09-17T03:55:11.424841+0000","last_clean":"2024-09-17T03:55:11.424841+0000","last_became_active":"2024-09-17T03:55:10.112184+0000","last_became_peered":"2024-09-17T03:55:10.112184+0000","last_unstale":"2024-09-17T03:55:11.424841+0000","last_undegraded":"2024-09-17T03:55:11.424841+0000","last_fullsized":"2024-09-17T03:55:11.424841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:12.328355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425890+0000","last_change":"2024-09-17T03:55:10.123473+0000","last_active":"2024-09-17T03:55:11.425890+0000","last_peered":"2024-09-17T03:55:11.425890+0000","last_clean":"2024-09-17T03:55:11.425890+0000","last_became_active":"2024-09-17T03:55:10.123217+0000","last_became_peered":"2024-09-17T03:55:10.123217+0000","last_unstale":"2024-09-17T03:55:11.425890+0000","last_undegraded":"2024-09-17T03:55:11.425890+0000","last_fullsized":"2024-09-17T03:55:11.425890+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:50.947437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425736+0000","last_change":"2024-09-17T03:55:10.103667+0000","last_active":"2024-09-17T03:55:11.425736+0000","last_peered":"2024-09-17T03:55:11.425736+0000","last_clean":"2024-09-17T03:55:11.425736+0000","last_became_active":"2024-09-17T03:55:10.103336+0000","last_became_peered":"2024-09-17T03:55:10.103336+0000","last_unstale":"2024-09-17T03:55:11.425736+0000","last_undegraded":"2024-09-17T03:55:11.425736+0000","last_fullsized":"2024-09-17T03:55:11.425736+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:33.517751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424369+0000","last_change":"2024-09-17T03:55:10.109342+0000","last_active":"2024-09-17T03:55:11.424369+0000","last_peered":"2024-09-17T03:55:11.424369+0000","last_clean":"2024-09-17T03:55:11.424369+0000","last_became_active":"2024-09-17T03:55:10.109013+0000","last_became_peered":"2024-09-17T03:55:10.109013+0000","last_unstale":"2024-09-17T03:55:11.424369+0000","last_undegraded":"2024-09-17T03:55:11.424369+0000","last_fullsized":"2024-09-17T03:55:11.424369+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:01.880305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103153+0000","last_change":"2024-09-17T03:55:10.098689+0000","last_active":"2024-09-17T03:55:11.103153+0000","last_peered":"2024-09-17T03:55:11.103153+0000","last_clean":"2024-09-17T03:55:11.103153+0000","last_became_active":"2024-09-17T03:55:10.097160+0000","last_became_peered":"2024-09-17T03:55:10.097160+0000","last_unstale":"2024-09-17T03:55:11.103153+0000","last_undegraded":"2024-09-17T03:55:11.103153+0000","last_fullsized":"2024-09-17T03:55:11.103153+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:59:16.574265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142768+0000","last_change":"2024-09-17T03:55:10.118739+0000","last_active":"2024-09-17T03:55:11.142768+0000","last_peered":"2024-09-17T03:55:11.142768+0000","last_clean":"2024-09-17T03:55:11.142768+0000","last_became_active":"2024-09-17T03:55:10.112849+0000","last_became_peered":"2024-09-17T03:55:10.112849+0000","last_unstale":"2024-09-17T03:55:11.142768+0000","last_undegraded":"2024-09-17T03:55:11.142768+0000","last_fullsized":"2024-09-17T03:55:11.142768+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104179+0000","last_change":"2024-09-17T03:55:10.096583+0000","last_active":"2024-09-17T03:55:11.104179+0000","last_peered":"2024-09-17T03:55:11.104179+0000","last_clean":"2024-09-17T03:55:11.104179+0000","last_became_active":"2024-09-17T03:55:10.096289+0000","last_became_peered":"2024-09-17T03:55:10.096289+0000","last_unstale":"2024-09-17T03:55:11.104179+0000","last_undegraded":"2024-09-17T03:55:11.104179+0000","last_fullsized":"2024-09-17T03:55:11.104179+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424746+0000","last_change":"2024-09-17T03:55:10.116515+0000","last_active":"2024-09-17T03:55:11.424746+0000","last_peered":"2024-09-17T03:55:11.424746+0000","last_clean":"2024-09-17T03:55:11.424746+0000","last_became_active":"2024-09-17T03:55:10.116161+0000","last_became_peered":"2024-09-17T03:55:10.116161+0000","last_unstale":"2024-09-17T03:55:11.424746+0000","last_undegraded":"2024-09-17T03:55:11.424746+0000","last_fullsized":"2024-09-17T03:55:11.424746+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:08:35.433598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424969+0000","last_change":"2024-09-17T03:55:10.113065+0000","last_active":"2024-09-17T03:55:11.424969+0000","last_peered":"2024-09-17T03:55:11.424969+0000","last_clean":"2024-09-17T03:55:11.424969+0000","last_became_active":"2024-09-17T03:55:10.111008+0000","last_became_peered":"2024-09-17T03:55:10.111008+0000","last_unstale":"2024-09-17T03:55:11.424969+0000","last_undegraded":"2024-09-17T03:55:11.424969+0000","last_fullsized":"2024-09-17T03:55:11.424969+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:35:54.618141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424625+0000","last_change":"2024-09-17T03:55:10.105708+0000","last_active":"2024-09-17T03:55:11.424625+0000","last_peered":"2024-09-17T03:55:11.424625+0000","last_clean":"2024-09-17T03:55:11.424625+0000","last_became_active":"2024-09-17T03:55:10.105368+0000","last_became_peered":"2024-09-17T03:55:10.105368+0000","last_unstale":"2024-09-17T03:55:11.424625+0000","last_undegraded":"2024-09-17T03:55:11.424625+0000","last_fullsized":"2024-09-17T03:55:11.424625+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:19.059873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425340+0000","last_change":"2024-09-17T03:55:10.107346+0000","last_active":"2024-09-17T03:55:11.425340+0000","last_peered":"2024-09-17T03:55:11.425340+0000","last_clean":"2024-09-17T03:55:11.425340+0000","last_became_active":"2024-09-17T03:55:10.106955+0000","last_became_peered":"2024-09-17T03:55:10.106955+0000","last_unstale":"2024-09-17T03:55:11.425340+0000","last_undegraded":"2024-09-17T03:55:11.425340+0000","last_fullsized":"2024-09-17T03:55:11.425340+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:29.542513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424410+0000","last_change":"2024-09-17T03:55:10.108582+0000","last_active":"2024-09-17T03:55:11.424410+0000","last_peered":"2024-09-17T03:55:11.424410+0000","last_clean":"2024-09-17T03:55:11.424410+0000","last_became_active":"2024-09-17T03:55:10.105982+0000","last_became_peered":"2024-09-17T03:55:10.105982+0000","last_unstale":"2024-09-17T03:55:11.424410+0000","last_undegraded":"2024-09-17T03:55:11.424410+0000","last_fullsized":"2024-09-17T03:55:11.424410+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:31.031639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425062+0000","last_change":"2024-09-17T03:55:10.109256+0000","last_active":"2024-09-17T03:55:11.425062+0000","last_peered":"2024-09-17T03:55:11.425062+0000","last_clean":"2024-09-17T03:55:11.425062+0000","last_became_active":"2024-09-17T03:55:10.108832+0000","last_became_peered":"2024-09-17T03:55:10.108832+0000","last_unstale":"2024-09-17T03:55:11.425062+0000","last_undegraded":"2024-09-17T03:55:11.425062+0000","last_fullsized":"2024-09-17T03:55:11.425062+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:16.522311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424801+0000","last_change":"2024-09-17T03:55:10.112919+0000","last_active":"2024-09-17T03:55:11.424801+0000","last_peered":"2024-09-17T03:55:11.424801+0000","last_clean":"2024-09-17T03:55:11.424801+0000","last_became_active":"2024-09-17T03:55:10.112583+0000","last_became_peered":"2024-09-17T03:55:10.112583+0000","last_unstale":"2024-09-17T03:55:11.424801+0000","last_undegraded":"2024-09-17T03:55:11.424801+0000","last_fullsized":"2024-09-17T03:55:11.424801+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:27.443267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142438+0000","last_change":"2024-09-17T03:55:10.112948+0000","last_active":"2024-09-17T03:55:11.142438+0000","last_peered":"2024-09-17T03:55:11.142438+0000","last_clean":"2024-09-17T03:55:11.142438+0000","last_became_active":"2024-09-17T03:55:10.112642+0000","last_became_peered":"2024-09-17T03:55:10.112642+0000","last_unstale":"2024-09-17T03:55:11.142438+0000","last_undegraded":"2024-09-17T03:55:11.142438+0000","last_fullsized":"2024-09-17T03:55:11.142438+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:40.241593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423770+0000","last_change":"2024-09-17T03:55:10.094106+0000","last_active":"2024-09-17T03:55:11.423770+0000","last_peered":"2024-09-17T03:55:11.423770+0000","last_clean":"2024-09-17T03:55:11.423770+0000","last_became_active":"2024-09-17T03:55:10.093763+0000","last_became_peered":"2024-09-17T03:55:10.093763+0000","last_unstale":"2024-09-17T03:55:11.423770+0000","last_undegraded":"2024-09-17T03:55:11.423770+0000","last_fullsized":"2024-09-17T03:55:11.423770+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:25.835238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426449+0000","last_change":"2024-09-17T03:55:10.101103+0000","last_active":"2024-09-17T03:55:11.426449+0000","last_peered":"2024-09-17T03:55:11.426449+0000","last_clean":"2024-09-17T03:55:11.426449+0000","last_became_active":"2024-09-17T03:55:10.100775+0000","last_became_peered":"2024-09-17T03:55:10.100775+0000","last_unstale":"2024-09-17T03:55:11.426449+0000","last_undegraded":"2024-09-17T03:55:11.426449+0000","last_fullsized":"2024-09-17T03:55:11.426449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426439+0000","last_change":"2024-09-17T03:55:10.100108+0000","last_active":"2024-09-17T03:55:11.426439+0000","last_peered":"2024-09-17T03:55:11.426439+0000","last_clean":"2024-09-17T03:55:11.426439+0000","last_became_active":"2024-09-17T03:55:10.099786+0000","last_became_peered":"2024-09-17T03:55:10.099786+0000","last_unstale":"2024-09-17T03:55:11.426439+0000","last_undegraded":"2024-09-17T03:55:11.426439+0000","last_fullsized":"2024-09-17T03:55:11.426439+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:00.641678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425612+0000","last_change":"2024-09-17T03:55:10.109163+0000","last_active":"2024-09-17T03:55:11.425612+0000","last_peered":"2024-09-17T03:55:11.425612+0000","last_clean":"2024-09-17T03:55:11.425612+0000","last_became_active":"2024-09-17T03:55:10.108810+0000","last_became_peered":"2024-09-17T03:55:10.108810+0000","last_unstale":"2024-09-17T03:55:11.425612+0000","last_undegraded":"2024-09-17T03:55:11.425612+0000","last_fullsized":"2024-09-17T03:55:11.425612+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:32.706728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426258+0000","last_change":"2024-09-17T03:55:10.102425+0000","last_active":"2024-09-17T03:55:11.426258+0000","last_peered":"2024-09-17T03:55:11.426258+0000","last_clean":"2024-09-17T03:55:11.426258+0000","last_became_active":"2024-09-17T03:55:10.102086+0000","last_became_peered":"2024-09-17T03:55:10.102086+0000","last_unstale":"2024-09-17T03:55:11.426258+0000","last_undegraded":"2024-09-17T03:55:11.426258+0000","last_fullsized":"2024-09-17T03:55:11.426258+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426586+0000","last_change":"2024-09-17T03:55:10.100194+0000","last_active":"2024-09-17T03:55:11.426586+0000","last_peered":"2024-09-17T03:55:11.426586+0000","last_clean":"2024-09-17T03:55:11.426586+0000","last_became_active":"2024-09-17T03:55:10.099857+0000","last_became_peered":"2024-09-17T03:55:10.099857+0000","last_unstale":"2024-09-17T03:55:11.426586+0000","last_undegraded":"2024-09-17T03:55:11.426586+0000","last_fullsized":"2024-09-17T03:55:11.426586+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100819+0000","last_change":"2024-09-17T03:55:10.129746+0000","last_active":"2024-09-17T03:55:11.100819+0000","last_peered":"2024-09-17T03:55:11.100819+0000","last_clean":"2024-09-17T03:55:11.100819+0000","last_became_active":"2024-09-17T03:55:10.124450+0000","last_became_peered":"2024-09-17T03:55:10.124450+0000","last_unstale":"2024-09-17T03:55:11.100819+0000","last_undegraded":"2024-09-17T03:55:11.100819+0000","last_fullsized":"2024-09-17T03:55:11.100819+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.748141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425299+0000","last_change":"2024-09-17T03:55:10.097618+0000","last_active":"2024-09-17T03:55:11.425299+0000","last_peered":"2024-09-17T03:55:11.425299+0000","last_clean":"2024-09-17T03:55:11.425299+0000","last_became_active":"2024-09-17T03:55:10.097328+0000","last_became_peered":"2024-09-17T03:55:10.097328+0000","last_unstale":"2024-09-17T03:55:11.425299+0000","last_undegraded":"2024-09-17T03:55:11.425299+0000","last_fullsized":"2024-09-17T03:55:11.425299+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:23.303493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100379+0000","last_change":"2024-09-17T03:55:10.107998+0000","last_active":"2024-09-17T03:55:11.100379+0000","last_peered":"2024-09-17T03:55:11.100379+0000","last_clean":"2024-09-17T03:55:11.100379+0000","last_became_active":"2024-09-17T03:55:10.107395+0000","last_became_peered":"2024-09-17T03:55:10.107395+0000","last_unstale":"2024-09-17T03:55:11.100379+0000","last_undegraded":"2024-09-17T03:55:11.100379+0000","last_fullsized":"2024-09-17T03:55:11.100379+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426752+0000","last_change":"2024-09-17T03:55:10.098678+0000","last_active":"2024-09-17T03:55:11.426752+0000","last_peered":"2024-09-17T03:55:11.426752+0000","last_clean":"2024-09-17T03:55:11.426752+0000","last_became_active":"2024-09-17T03:55:10.098318+0000","last_became_peered":"2024-09-17T03:55:10.098318+0000","last_unstale":"2024-09-17T03:55:11.426752+0000","last_undegraded":"2024-09-17T03:55:11.426752+0000","last_fullsized":"2024-09-17T03:55:11.426752+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424684+0000","last_change":"2024-09-17T03:55:10.102840+0000","last_active":"2024-09-17T03:55:11.424684+0000","last_peered":"2024-09-17T03:55:11.424684+0000","last_clean":"2024-09-17T03:55:11.424684+0000","last_became_active":"2024-09-17T03:55:10.100850+0000","last_became_peered":"2024-09-17T03:55:10.100850+0000","last_unstale":"2024-09-17T03:55:11.424684+0000","last_undegraded":"2024-09-17T03:55:11.424684+0000","last_fullsized":"2024-09-17T03:55:11.424684+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:17.813222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142724+0000","last_change":"2024-09-17T03:55:10.122268+0000","last_active":"2024-09-17T03:55:11.142724+0000","last_peered":"2024-09-17T03:55:11.142724+0000","last_clean":"2024-09-17T03:55:11.142724+0000","last_became_active":"2024-09-17T03:55:10.121813+0000","last_became_peered":"2024-09-17T03:55:10.121813+0000","last_unstale":"2024-09-17T03:55:11.142724+0000","last_undegraded":"2024-09-17T03:55:11.142724+0000","last_fullsized":"2024-09-17T03:55:11.142724+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:14.202711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100713+0000","last_change":"2024-09-17T03:55:10.126728+0000","last_active":"2024-09-17T03:55:11.100713+0000","last_peered":"2024-09-17T03:55:11.100713+0000","last_clean":"2024-09-17T03:55:11.100713+0000","last_became_active":"2024-09-17T03:55:10.126437+0000","last_became_peered":"2024-09-17T03:55:10.126437+0000","last_unstale":"2024-09-17T03:55:11.100713+0000","last_undegraded":"2024-09-17T03:55:11.100713+0000","last_fullsized":"2024-09-17T03:55:11.100713+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:44.532100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425371+0000","last_change":"2024-09-17T03:55:10.103912+0000","last_active":"2024-09-17T03:55:11.425371+0000","last_peered":"2024-09-17T03:55:11.425371+0000","last_clean":"2024-09-17T03:55:11.425371+0000","last_became_active":"2024-09-17T03:55:10.103569+0000","last_became_peered":"2024-09-17T03:55:10.103569+0000","last_unstale":"2024-09-17T03:55:11.425371+0000","last_undegraded":"2024-09-17T03:55:11.425371+0000","last_fullsized":"2024-09-17T03:55:11.425371+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:36.412867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104245+0000","last_change":"2024-09-17T03:55:10.111756+0000","last_active":"2024-09-17T03:55:11.104245+0000","last_peered":"2024-09-17T03:55:11.104245+0000","last_clean":"2024-09-17T03:55:11.104245+0000","last_became_active":"2024-09-17T03:55:10.111350+0000","last_became_peered":"2024-09-17T03:55:10.111350+0000","last_unstale":"2024-09-17T03:55:11.104245+0000","last_undegraded":"2024-09-17T03:55:11.104245+0000","last_fullsized":"2024-09-17T03:55:11.104245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:26:45.161678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423827+0000","last_change":"2024-09-17T03:55:10.110031+0000","last_active":"2024-09-17T03:55:11.423827+0000","last_peered":"2024-09-17T03:55:11.423827+0000","last_clean":"2024-09-17T03:55:11.423827+0000","last_became_active":"2024-09-17T03:55:10.106150+0000","last_became_peered":"2024-09-17T03:55:10.106150+0000","last_unstale":"2024-09-17T03:55:11.423827+0000","last_undegraded":"2024-09-17T03:55:11.423827+0000","last_fullsized":"2024-09-17T03:55:11.423827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:23:15.416205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142425+0000","last_change":"2024-09-17T03:55:10.114672+0000","last_active":"2024-09-17T03:55:11.142425+0000","last_peered":"2024-09-17T03:55:11.142425+0000","last_clean":"2024-09-17T03:55:11.142425+0000","last_became_active":"2024-09-17T03:55:10.114304+0000","last_became_peered":"2024-09-17T03:55:10.114304+0000","last_unstale":"2024-09-17T03:55:11.142425+0000","last_undegraded":"2024-09-17T03:55:11.142425+0000","last_fullsized":"2024-09-17T03:55:11.142425+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:59.015404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142860+0000","last_change":"2024-09-17T03:55:10.118321+0000","last_active":"2024-09-17T03:55:11.142860+0000","last_peered":"2024-09-17T03:55:11.142860+0000","last_clean":"2024-09-17T03:55:11.142860+0000","last_became_active":"2024-09-17T03:55:10.118001+0000","last_became_peered":"2024-09-17T03:55:10.118001+0000","last_unstale":"2024-09-17T03:55:11.142860+0000","last_undegraded":"2024-09-17T03:55:11.142860+0000","last_fullsized":"2024-09-17T03:55:11.142860+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"106'4","reported_seq":19,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104474+0000","last_change":"2024-09-17T03:55:09.098888+0000","last_active":"2024-09-17T03:55:11.104474+0000","last_peered":"2024-09-17T03:55:11.104474+0000","last_clean":"2024-09-17T03:55:11.104474+0000","last_became_active":"2024-09-17T03:55:09.098488+0000","last_became_peered":"2024-09-17T03:55:09.098488+0000","last_unstale":"2024-09-17T03:55:11.104474+0000","last_undegraded":"2024-09-17T03:55:11.104474+0000","last_fullsized":"2024-09-17T03:55:11.104474+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:43.529970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426303+0000","last_change":"2024-09-17T03:55:10.098038+0000","last_active":"2024-09-17T03:55:11.426303+0000","last_peered":"2024-09-17T03:55:11.426303+0000","last_clean":"2024-09-17T03:55:11.426303+0000","last_became_active":"2024-09-17T03:55:10.097706+0000","last_became_peered":"2024-09-17T03:55:10.097706+0000","last_unstale":"2024-09-17T03:55:11.426303+0000","last_undegraded":"2024-09-17T03:55:11.426303+0000","last_fullsized":"2024-09-17T03:55:11.426303+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:35:48.538101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424685+0000","last_change":"2024-09-17T03:55:09.102970+0000","last_active":"2024-09-17T03:55:11.424685+0000","last_peered":"2024-09-17T03:55:11.424685+0000","last_clean":"2024-09-17T03:55:11.424685+0000","last_became_active":"2024-09-17T03:55:09.102558+0000","last_became_peered":"2024-09-17T03:55:09.102558+0000","last_unstale":"2024-09-17T03:55:11.424685+0000","last_undegraded":"2024-09-17T03:55:11.424685+0000","last_fullsized":"2024-09-17T03:55:11.424685+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:26.840025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142757+0000","last_change":"2024-09-17T03:55:10.126853+0000","last_active":"2024-09-17T03:55:11.142757+0000","last_peered":"2024-09-17T03:55:11.142757+0000","last_clean":"2024-09-17T03:55:11.142757+0000","last_became_active":"2024-09-17T03:55:10.123486+0000","last_became_peered":"2024-09-17T03:55:10.123486+0000","last_unstale":"2024-09-17T03:55:11.142757+0000","last_undegraded":"2024-09-17T03:55:11.142757+0000","last_fullsized":"2024-09-17T03:55:11.142757+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:31.237400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103305+0000","last_change":"2024-09-17T03:55:09.106007+0000","last_active":"2024-09-17T03:55:11.103305+0000","last_peered":"2024-09-17T03:55:11.103305+0000","last_clean":"2024-09-17T03:55:11.103305+0000","last_became_active":"2024-09-17T03:55:09.105510+0000","last_became_peered":"2024-09-17T03:55:09.105510+0000","last_unstale":"2024-09-17T03:55:11.103305+0000","last_undegraded":"2024-09-17T03:55:11.103305+0000","last_fullsized":"2024-09-17T03:55:11.103305+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:27:12.256566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425907+0000","last_change":"2024-09-17T03:55:10.111580+0000","last_active":"2024-09-17T03:55:11.425907+0000","last_peered":"2024-09-17T03:55:11.425907+0000","last_clean":"2024-09-17T03:55:11.425907+0000","last_became_active":"2024-09-17T03:55:10.111121+0000","last_became_peered":"2024-09-17T03:55:10.111121+0000","last_unstale":"2024-09-17T03:55:11.425907+0000","last_undegraded":"2024-09-17T03:55:11.425907+0000","last_fullsized":"2024-09-17T03:55:11.425907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103330+0000","last_change":"2024-09-17T03:55:09.109395+0000","last_active":"2024-09-17T03:55:11.103330+0000","last_peered":"2024-09-17T03:55:11.103330+0000","last_clean":"2024-09-17T03:55:11.103330+0000","last_became_active":"2024-09-17T03:55:09.107581+0000","last_became_peered":"2024-09-17T03:55:09.107581+0000","last_unstale":"2024-09-17T03:55:11.103330+0000","last_undegraded":"2024-09-17T03:55:11.103330+0000","last_fullsized":"2024-09-17T03:55:11.103330+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:55.628731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423981+0000","last_change":"2024-09-17T03:55:10.115350+0000","last_active":"2024-09-17T03:55:11.423981+0000","last_peered":"2024-09-17T03:55:11.423981+0000","last_clean":"2024-09-17T03:55:11.423981+0000","last_became_active":"2024-09-17T03:55:10.111606+0000","last_became_peered":"2024-09-17T03:55:10.111606+0000","last_unstale":"2024-09-17T03:55:11.423981+0000","last_undegraded":"2024-09-17T03:55:11.423981+0000","last_fullsized":"2024-09-17T03:55:11.423981+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:51:26.916486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104020+0000","last_change":"2024-09-17T03:55:09.100552+0000","last_active":"2024-09-17T03:55:11.104020+0000","last_peered":"2024-09-17T03:55:11.104020+0000","last_clean":"2024-09-17T03:55:11.104020+0000","last_became_active":"2024-09-17T03:55:09.100230+0000","last_became_peered":"2024-09-17T03:55:09.100230+0000","last_unstale":"2024-09-17T03:55:11.104020+0000","last_undegraded":"2024-09-17T03:55:11.104020+0000","last_fullsized":"2024-09-17T03:55:11.104020+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:26.431547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425676+0000","last_change":"2024-09-17T03:55:10.125057+0000","last_active":"2024-09-17T03:55:11.425676+0000","last_peered":"2024-09-17T03:55:11.425676+0000","last_clean":"2024-09-17T03:55:11.425676+0000","last_became_active":"2024-09-17T03:55:10.124788+0000","last_became_peered":"2024-09-17T03:55:10.124788+0000","last_unstale":"2024-09-17T03:55:11.425676+0000","last_undegraded":"2024-09-17T03:55:11.425676+0000","last_fullsized":"2024-09-17T03:55:11.425676+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:49.111695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.101089+0000","last_change":"2024-09-17T03:55:09.098297+0000","last_active":"2024-09-17T03:55:11.101089+0000","last_peered":"2024-09-17T03:55:11.101089+0000","last_clean":"2024-09-17T03:55:11.101089+0000","last_became_active":"2024-09-17T03:55:09.097970+0000","last_became_peered":"2024-09-17T03:55:09.097970+0000","last_unstale":"2024-09-17T03:55:11.101089+0000","last_undegraded":"2024-09-17T03:55:11.101089+0000","last_fullsized":"2024-09-17T03:55:11.101089+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:31:36.074034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424050+0000","last_change":"2024-09-17T03:55:10.122441+0000","last_active":"2024-09-17T03:55:11.424050+0000","last_peered":"2024-09-17T03:55:11.424050+0000","last_clean":"2024-09-17T03:55:11.424050+0000","last_became_active":"2024-09-17T03:55:10.121881+0000","last_became_peered":"2024-09-17T03:55:10.121881+0000","last_unstale":"2024-09-17T03:55:11.424050+0000","last_undegraded":"2024-09-17T03:55:11.424050+0000","last_fullsized":"2024-09-17T03:55:11.424050+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.313711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425520+0000","last_change":"2024-09-17T03:55:09.101386+0000","last_active":"2024-09-17T03:55:11.425520+0000","last_peered":"2024-09-17T03:55:11.425520+0000","last_clean":"2024-09-17T03:55:11.425520+0000","last_became_active":"2024-09-17T03:55:09.101041+0000","last_became_peered":"2024-09-17T03:55:09.101041+0000","last_unstale":"2024-09-17T03:55:11.425520+0000","last_undegraded":"2024-09-17T03:55:11.425520+0000","last_fullsized":"2024-09-17T03:55:11.425520+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:07.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104041+0000","last_change":"2024-09-17T03:55:10.097916+0000","last_active":"2024-09-17T03:55:11.104041+0000","last_peered":"2024-09-17T03:55:11.104041+0000","last_clean":"2024-09-17T03:55:11.104041+0000","last_became_active":"2024-09-17T03:55:10.097595+0000","last_became_peered":"2024-09-17T03:55:10.097595+0000","last_unstale":"2024-09-17T03:55:11.104041+0000","last_undegraded":"2024-09-17T03:55:11.104041+0000","last_fullsized":"2024-09-17T03:55:11.104041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.114279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423852+0000","last_change":"2024-09-17T03:55:09.093148+0000","last_active":"2024-09-17T03:55:11.423852+0000","last_peered":"2024-09-17T03:55:11.423852+0000","last_clean":"2024-09-17T03:55:11.423852+0000","last_became_active":"2024-09-17T03:55:09.092860+0000","last_became_peered":"2024-09-17T03:55:09.092860+0000","last_unstale":"2024-09-17T03:55:11.423852+0000","last_undegraded":"2024-09-17T03:55:11.423852+0000","last_fullsized":"2024-09-17T03:55:11.423852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:11.814702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104560+0000","last_change":"2024-09-17T03:55:10.102328+0000","last_active":"2024-09-17T03:55:11.104560+0000","last_peered":"2024-09-17T03:55:11.104560+0000","last_clean":"2024-09-17T03:55:11.104560+0000","last_became_active":"2024-09-17T03:55:10.101937+0000","last_became_peered":"2024-09-17T03:55:10.101937+0000","last_unstale":"2024-09-17T03:55:11.104560+0000","last_undegraded":"2024-09-17T03:55:11.104560+0000","last_fullsized":"2024-09-17T03:55:11.104560+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:53.259242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102690+0000","last_change":"2024-09-17T03:55:09.110979+0000","last_active":"2024-09-17T03:55:11.102690+0000","last_peered":"2024-09-17T03:55:11.102690+0000","last_clean":"2024-09-17T03:55:11.102690+0000","last_became_active":"2024-09-17T03:55:09.110640+0000","last_became_peered":"2024-09-17T03:55:09.110640+0000","last_unstale":"2024-09-17T03:55:11.102690+0000","last_undegraded":"2024-09-17T03:55:11.102690+0000","last_fullsized":"2024-09-17T03:55:11.102690+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:28.520430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426564+0000","last_change":"2024-09-17T03:55:10.105276+0000","last_active":"2024-09-17T03:55:11.426564+0000","last_peered":"2024-09-17T03:55:11.426564+0000","last_clean":"2024-09-17T03:55:11.426564+0000","last_became_active":"2024-09-17T03:55:10.104973+0000","last_became_peered":"2024-09-17T03:55:10.104973+0000","last_unstale":"2024-09-17T03:55:11.426564+0000","last_undegraded":"2024-09-17T03:55:11.426564+0000","last_fullsized":"2024-09-17T03:55:11.426564+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:37.836511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100549+0000","last_change":"2024-09-17T03:55:09.100586+0000","last_active":"2024-09-17T03:55:11.100549+0000","last_peered":"2024-09-17T03:55:11.100549+0000","last_clean":"2024-09-17T03:55:11.100549+0000","last_became_active":"2024-09-17T03:55:09.100303+0000","last_became_peered":"2024-09-17T03:55:09.100303+0000","last_unstale":"2024-09-17T03:55:11.100549+0000","last_undegraded":"2024-09-17T03:55:11.100549+0000","last_fullsized":"2024-09-17T03:55:11.100549+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:36.366891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100901+0000","last_change":"2024-09-17T03:55:10.127654+0000","last_active":"2024-09-17T03:55:11.100901+0000","last_peered":"2024-09-17T03:55:11.100901+0000","last_clean":"2024-09-17T03:55:11.100901+0000","last_became_active":"2024-09-17T03:55:10.115490+0000","last_became_peered":"2024-09-17T03:55:10.115490+0000","last_unstale":"2024-09-17T03:55:11.100901+0000","last_undegraded":"2024-09-17T03:55:11.100901+0000","last_fullsized":"2024-09-17T03:55:11.100901+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:50.875001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"106'3","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424336+0000","last_change":"2024-09-17T03:55:09.101674+0000","last_active":"2024-09-17T03:55:11.424336+0000","last_peered":"2024-09-17T03:55:11.424336+0000","last_clean":"2024-09-17T03:55:11.424336+0000","last_became_active":"2024-09-17T03:55:09.101311+0000","last_became_peered":"2024-09-17T03:55:09.101311+0000","last_unstale":"2024-09-17T03:55:11.424336+0000","last_undegraded":"2024-09-17T03:55:11.424336+0000","last_fullsized":"2024-09-17T03:55:11.424336+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:43.324925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.105085+0000","last_change":"2024-09-17T03:55:10.100605+0000","last_active":"2024-09-17T03:55:11.105085+0000","last_peered":"2024-09-17T03:55:11.105085+0000","last_clean":"2024-09-17T03:55:11.105085+0000","last_became_active":"2024-09-17T03:55:10.100264+0000","last_became_peered":"2024-09-17T03:55:10.100264+0000","last_unstale":"2024-09-17T03:55:11.105085+0000","last_undegraded":"2024-09-17T03:55:11.105085+0000","last_fullsized":"2024-09-17T03:55:11.105085+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:31.420107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"106'6","reported_seq":21,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424538+0000","last_change":"2024-09-17T03:55:09.102173+0000","last_active":"2024-09-17T03:55:11.424538+0000","last_peered":"2024-09-17T03:55:11.424538+0000","last_clean":"2024-09-17T03:55:11.424538+0000","last_became_active":"2024-09-17T03:55:09.101451+0000","last_became_peered":"2024-09-17T03:55:09.101451+0000","last_unstale":"2024-09-17T03:55:11.424538+0000","last_undegraded":"2024-09-17T03:55:11.424538+0000","last_fullsized":"2024-09-17T03:55:11.424538+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:16.929007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100938+0000","last_change":"2024-09-17T03:55:10.128915+0000","last_active":"2024-09-17T03:55:11.100938+0000","last_peered":"2024-09-17T03:55:11.100938+0000","last_clean":"2024-09-17T03:55:11.100938+0000","last_became_active":"2024-09-17T03:55:10.121387+0000","last_became_peered":"2024-09-17T03:55:10.121387+0000","last_unstale":"2024-09-17T03:55:11.100938+0000","last_undegraded":"2024-09-17T03:55:11.100938+0000","last_fullsized":"2024-09-17T03:55:11.100938+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:27:07.248320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142482+0000","last_change":"2024-09-17T03:55:09.101519+0000","last_active":"2024-09-17T03:55:11.142482+0000","last_peered":"2024-09-17T03:55:11.142482+0000","last_clean":"2024-09-17T03:55:11.142482+0000","last_became_active":"2024-09-17T03:55:09.101211+0000","last_became_peered":"2024-09-17T03:55:09.101211+0000","last_unstale":"2024-09-17T03:55:11.142482+0000","last_undegraded":"2024-09-17T03:55:11.142482+0000","last_fullsized":"2024-09-17T03:55:11.142482+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:58.858197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100665+0000","last_change":"2024-09-17T03:55:10.124544+0000","last_active":"2024-09-17T03:55:11.100665+0000","last_peered":"2024-09-17T03:55:11.100665+0000","last_clean":"2024-09-17T03:55:11.100665+0000","last_became_active":"2024-09-17T03:55:10.124209+0000","last_became_peered":"2024-09-17T03:55:10.124209+0000","last_unstale":"2024-09-17T03:55:11.100665+0000","last_undegraded":"2024-09-17T03:55:11.100665+0000","last_fullsized":"2024-09-17T03:55:11.100665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:35.341682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424612+0000","last_change":"2024-09-17T03:55:09.103112+0000","last_active":"2024-09-17T03:55:11.424612+0000","last_peered":"2024-09-17T03:55:11.424612+0000","last_clean":"2024-09-17T03:55:11.424612+0000","last_became_active":"2024-09-17T03:55:09.102182+0000","last_became_peered":"2024-09-17T03:55:09.102182+0000","last_unstale":"2024-09-17T03:55:11.424612+0000","last_undegraded":"2024-09-17T03:55:11.424612+0000","last_fullsized":"2024-09-17T03:55:11.424612+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:53:55.298078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.101033+0000","last_change":"2024-09-17T03:55:10.120468+0000","last_active":"2024-09-17T03:55:11.101033+0000","last_peered":"2024-09-17T03:55:11.101033+0000","last_clean":"2024-09-17T03:55:11.101033+0000","last_became_active":"2024-09-17T03:55:10.119946+0000","last_became_peered":"2024-09-17T03:55:10.119946+0000","last_unstale":"2024-09-17T03:55:11.101033+0000","last_undegraded":"2024-09-17T03:55:11.101033+0000","last_fullsized":"2024-09-17T03:55:11.101033+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:48:16.242686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"106'8","reported_seq":23,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102570+0000","last_change":"2024-09-17T03:55:09.110547+0000","last_active":"2024-09-17T03:55:11.102570+0000","last_peered":"2024-09-17T03:55:11.102570+0000","last_clean":"2024-09-17T03:55:11.102570+0000","last_became_active":"2024-09-17T03:55:09.109930+0000","last_became_peered":"2024-09-17T03:55:09.109930+0000","last_unstale":"2024-09-17T03:55:11.102570+0000","last_undegraded":"2024-09-17T03:55:11.102570+0000","last_fullsized":"2024-09-17T03:55:11.102570+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:45.646247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102906+0000","last_change":"2024-09-17T03:55:10.108164+0000","last_active":"2024-09-17T03:55:11.102906+0000","last_peered":"2024-09-17T03:55:11.102906+0000","last_clean":"2024-09-17T03:55:11.102906+0000","last_became_active":"2024-09-17T03:55:10.106709+0000","last_became_peered":"2024-09-17T03:55:10.106709+0000","last_unstale":"2024-09-17T03:55:11.102906+0000","last_undegraded":"2024-09-17T03:55:11.102906+0000","last_fullsized":"2024-09-17T03:55:11.102906+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:34.367904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424999+0000","last_change":"2024-09-17T03:55:09.102865+0000","last_active":"2024-09-17T03:55:11.424999+0000","last_peered":"2024-09-17T03:55:11.424999+0000","last_clean":"2024-09-17T03:55:11.424999+0000","last_became_active":"2024-09-17T03:55:09.100408+0000","last_became_peered":"2024-09-17T03:55:09.100408+0000","last_unstale":"2024-09-17T03:55:11.424999+0000","last_undegraded":"2024-09-17T03:55:11.424999+0000","last_fullsized":"2024-09-17T03:55:11.424999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:29.269455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142886+0000","last_change":"2024-09-17T03:55:10.113442+0000","last_active":"2024-09-17T03:55:11.142886+0000","last_peered":"2024-09-17T03:55:11.142886+0000","last_clean":"2024-09-17T03:55:11.142886+0000","last_became_active":"2024-09-17T03:55:10.113158+0000","last_became_peered":"2024-09-17T03:55:10.113158+0000","last_unstale":"2024-09-17T03:55:11.142886+0000","last_undegraded":"2024-09-17T03:55:11.142886+0000","last_fullsized":"2024-09-17T03:55:11.142886+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:29.231805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.103008+0000","last_change":"2024-09-17T03:55:09.106143+0000","last_active":"2024-09-17T03:55:11.103008+0000","last_peered":"2024-09-17T03:55:11.103008+0000","last_clean":"2024-09-17T03:55:11.103008+0000","last_became_active":"2024-09-17T03:55:09.105539+0000","last_became_peered":"2024-09-17T03:55:09.105539+0000","last_unstale":"2024-09-17T03:55:11.103008+0000","last_undegraded":"2024-09-17T03:55:11.103008+0000","last_fullsized":"2024-09-17T03:55:11.103008+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:29.106359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425055+0000","last_change":"2024-09-17T03:55:10.118502+0000","last_active":"2024-09-17T03:55:11.425055+0000","last_peered":"2024-09-17T03:55:11.425055+0000","last_clean":"2024-09-17T03:55:11.425055+0000","last_became_active":"2024-09-17T03:55:10.118276+0000","last_became_peered":"2024-09-17T03:55:10.118276+0000","last_unstale":"2024-09-17T03:55:11.425055+0000","last_undegraded":"2024-09-17T03:55:11.425055+0000","last_fullsized":"2024-09-17T03:55:11.425055+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:23.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"106'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425392+0000","last_change":"2024-09-17T03:55:09.099409+0000","last_active":"2024-09-17T03:55:11.425392+0000","last_peered":"2024-09-17T03:55:11.425392+0000","last_clean":"2024-09-17T03:55:11.425392+0000","last_became_active":"2024-09-17T03:55:09.097894+0000","last_became_peered":"2024-09-17T03:55:09.097894+0000","last_unstale":"2024-09-17T03:55:11.425392+0000","last_undegraded":"2024-09-17T03:55:11.425392+0000","last_fullsized":"2024-09-17T03:55:11.425392+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:30.592901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426005+0000","last_change":"2024-09-17T03:55:10.101315+0000","last_active":"2024-09-17T03:55:11.426005+0000","last_peered":"2024-09-17T03:55:11.426005+0000","last_clean":"2024-09-17T03:55:11.426005+0000","last_became_active":"2024-09-17T03:55:10.100936+0000","last_became_peered":"2024-09-17T03:55:10.100936+0000","last_unstale":"2024-09-17T03:55:11.426005+0000","last_undegraded":"2024-09-17T03:55:11.426005+0000","last_fullsized":"2024-09-17T03:55:11.426005+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:51.665179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104091+0000","last_change":"2024-09-17T03:55:09.095974+0000","last_active":"2024-09-17T03:55:11.104091+0000","last_peered":"2024-09-17T03:55:11.104091+0000","last_clean":"2024-09-17T03:55:11.104091+0000","last_became_active":"2024-09-17T03:55:09.095632+0000","last_became_peered":"2024-09-17T03:55:09.095632+0000","last_unstale":"2024-09-17T03:55:11.104091+0000","last_undegraded":"2024-09-17T03:55:11.104091+0000","last_fullsized":"2024-09-17T03:55:11.104091+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.401993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424900+0000","last_change":"2024-09-17T03:55:10.104348+0000","last_active":"2024-09-17T03:55:11.424900+0000","last_peered":"2024-09-17T03:55:11.424900+0000","last_clean":"2024-09-17T03:55:11.424900+0000","last_became_active":"2024-09-17T03:55:10.104019+0000","last_became_peered":"2024-09-17T03:55:10.104019+0000","last_unstale":"2024-09-17T03:55:11.424900+0000","last_undegraded":"2024-09-17T03:55:11.424900+0000","last_fullsized":"2024-09-17T03:55:11.424900+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:48.618440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142778+0000","last_change":"2024-09-17T03:55:09.107009+0000","last_active":"2024-09-17T03:55:11.142778+0000","last_peered":"2024-09-17T03:55:11.142778+0000","last_clean":"2024-09-17T03:55:11.142778+0000","last_became_active":"2024-09-17T03:55:09.106706+0000","last_became_peered":"2024-09-17T03:55:09.106706+0000","last_unstale":"2024-09-17T03:55:11.142778+0000","last_undegraded":"2024-09-17T03:55:11.142778+0000","last_fullsized":"2024-09-17T03:55:11.142778+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:35:46.653486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424019+0000","last_change":"2024-09-17T03:55:10.101801+0000","last_active":"2024-09-17T03:55:11.424019+0000","last_peered":"2024-09-17T03:55:11.424019+0000","last_clean":"2024-09-17T03:55:11.424019+0000","last_became_active":"2024-09-17T03:55:10.101240+0000","last_became_peered":"2024-09-17T03:55:10.101240+0000","last_unstale":"2024-09-17T03:55:11.424019+0000","last_undegraded":"2024-09-17T03:55:11.424019+0000","last_fullsized":"2024-09-17T03:55:11.424019+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:08.177233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102580+0000","last_change":"2024-09-17T03:55:09.105469+0000","last_active":"2024-09-17T03:55:11.102580+0000","last_peered":"2024-09-17T03:55:11.102580+0000","last_clean":"2024-09-17T03:55:11.102580+0000","last_became_active":"2024-09-17T03:55:09.105074+0000","last_became_peered":"2024-09-17T03:55:09.105074+0000","last_unstale":"2024-09-17T03:55:11.102580+0000","last_undegraded":"2024-09-17T03:55:11.102580+0000","last_fullsized":"2024-09-17T03:55:11.102580+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:30.326152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425201+0000","last_change":"2024-09-17T03:55:10.110424+0000","last_active":"2024-09-17T03:55:11.425201+0000","last_peered":"2024-09-17T03:55:11.425201+0000","last_clean":"2024-09-17T03:55:11.425201+0000","last_became_active":"2024-09-17T03:55:10.110057+0000","last_became_peered":"2024-09-17T03:55:10.110057+0000","last_unstale":"2024-09-17T03:55:11.425201+0000","last_undegraded":"2024-09-17T03:55:11.425201+0000","last_fullsized":"2024-09-17T03:55:11.425201+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:46:40.073222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.142451+0000","last_change":"2024-09-17T03:55:09.109486+0000","last_active":"2024-09-17T03:55:11.142451+0000","last_peered":"2024-09-17T03:55:11.142451+0000","last_clean":"2024-09-17T03:55:11.142451+0000","last_became_active":"2024-09-17T03:55:09.108902+0000","last_became_peered":"2024-09-17T03:55:09.108902+0000","last_unstale":"2024-09-17T03:55:11.142451+0000","last_undegraded":"2024-09-17T03:55:11.142451+0000","last_fullsized":"2024-09-17T03:55:11.142451+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:07:20.685338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100907+0000","last_change":"2024-09-17T03:55:10.123697+0000","last_active":"2024-09-17T03:55:11.100907+0000","last_peered":"2024-09-17T03:55:11.100907+0000","last_clean":"2024-09-17T03:55:11.100907+0000","last_became_active":"2024-09-17T03:55:10.123389+0000","last_became_peered":"2024-09-17T03:55:10.123389+0000","last_unstale":"2024-09-17T03:55:11.100907+0000","last_undegraded":"2024-09-17T03:55:11.100907+0000","last_fullsized":"2024-09-17T03:55:11.100907+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:50:01.009413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423957+0000","last_change":"2024-09-17T03:55:09.089727+0000","last_active":"2024-09-17T03:55:11.423957+0000","last_peered":"2024-09-17T03:55:11.423957+0000","last_clean":"2024-09-17T03:55:11.423957+0000","last_became_active":"2024-09-17T03:55:09.089416+0000","last_became_peered":"2024-09-17T03:55:09.089416+0000","last_unstale":"2024-09-17T03:55:11.423957+0000","last_undegraded":"2024-09-17T03:55:11.423957+0000","last_fullsized":"2024-09-17T03:55:11.423957+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:19.704845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426516+0000","last_change":"2024-09-17T03:55:10.106649+0000","last_active":"2024-09-17T03:55:11.426516+0000","last_peered":"2024-09-17T03:55:11.426516+0000","last_clean":"2024-09-17T03:55:11.426516+0000","last_became_active":"2024-09-17T03:55:10.106347+0000","last_became_peered":"2024-09-17T03:55:10.106347+0000","last_unstale":"2024-09-17T03:55:11.426516+0000","last_undegraded":"2024-09-17T03:55:11.426516+0000","last_fullsized":"2024-09-17T03:55:11.426516+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:47:21.848148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"106'2","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100730+0000","last_change":"2024-09-17T03:55:09.106794+0000","last_active":"2024-09-17T03:55:11.100730+0000","last_peered":"2024-09-17T03:55:11.100730+0000","last_clean":"2024-09-17T03:55:11.100730+0000","last_became_active":"2024-09-17T03:55:09.106461+0000","last_became_peered":"2024-09-17T03:55:09.106461+0000","last_unstale":"2024-09-17T03:55:11.100730+0000","last_undegraded":"2024-09-17T03:55:11.100730+0000","last_fullsized":"2024-09-17T03:55:11.100730+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:40.245287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424964+0000","last_change":"2024-09-17T03:55:10.111117+0000","last_active":"2024-09-17T03:55:11.424964+0000","last_peered":"2024-09-17T03:55:11.424964+0000","last_clean":"2024-09-17T03:55:11.424964+0000","last_became_active":"2024-09-17T03:55:10.110769+0000","last_became_peered":"2024-09-17T03:55:10.110769+0000","last_unstale":"2024-09-17T03:55:11.424964+0000","last_undegraded":"2024-09-17T03:55:11.424964+0000","last_fullsized":"2024-09-17T03:55:11.424964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:34.220767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"107'1","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426576+0000","last_change":"2024-09-17T03:55:09.091992+0000","last_active":"2024-09-17T03:55:11.426576+0000","last_peered":"2024-09-17T03:55:11.426576+0000","last_clean":"2024-09-17T03:55:11.426576+0000","last_became_active":"2024-09-17T03:55:09.091655+0000","last_became_peered":"2024-09-17T03:55:09.091655+0000","last_unstale":"2024-09-17T03:55:11.426576+0000","last_undegraded":"2024-09-17T03:55:11.426576+0000","last_fullsized":"2024-09-17T03:55:11.426576+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:14.155619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424655+0000","last_change":"2024-09-17T03:55:10.125141+0000","last_active":"2024-09-17T03:55:11.424655+0000","last_peered":"2024-09-17T03:55:11.424655+0000","last_clean":"2024-09-17T03:55:11.424655+0000","last_became_active":"2024-09-17T03:55:10.124839+0000","last_became_peered":"2024-09-17T03:55:10.124839+0000","last_unstale":"2024-09-17T03:55:11.424655+0000","last_undegraded":"2024-09-17T03:55:11.424655+0000","last_fullsized":"2024-09-17T03:55:11.424655+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:49.605272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424463+0000","last_change":"2024-09-17T03:55:09.092456+0000","last_active":"2024-09-17T03:55:11.424463+0000","last_peered":"2024-09-17T03:55:11.424463+0000","last_clean":"2024-09-17T03:55:11.424463+0000","last_became_active":"2024-09-17T03:55:09.092133+0000","last_became_peered":"2024-09-17T03:55:09.092133+0000","last_unstale":"2024-09-17T03:55:11.424463+0000","last_undegraded":"2024-09-17T03:55:11.424463+0000","last_fullsized":"2024-09-17T03:55:11.424463+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:37:29.592608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425399+0000","last_change":"2024-09-17T03:55:10.105263+0000","last_active":"2024-09-17T03:55:11.425399+0000","last_peered":"2024-09-17T03:55:11.425399+0000","last_clean":"2024-09-17T03:55:11.425399+0000","last_became_active":"2024-09-17T03:55:10.104878+0000","last_became_peered":"2024-09-17T03:55:10.104878+0000","last_unstale":"2024-09-17T03:55:11.425399+0000","last_undegraded":"2024-09-17T03:55:11.425399+0000","last_fullsized":"2024-09-17T03:55:11.425399+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:29.686662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100964+0000","last_change":"2024-09-17T03:55:09.101334+0000","last_active":"2024-09-17T03:55:11.100964+0000","last_peered":"2024-09-17T03:55:11.100964+0000","last_clean":"2024-09-17T03:55:11.100964+0000","last_became_active":"2024-09-17T03:55:09.101016+0000","last_became_peered":"2024-09-17T03:55:09.101016+0000","last_unstale":"2024-09-17T03:55:11.100964+0000","last_undegraded":"2024-09-17T03:55:11.100964+0000","last_fullsized":"2024-09-17T03:55:11.100964+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:03.790620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425931+0000","last_change":"2024-09-17T03:55:10.097057+0000","last_active":"2024-09-17T03:55:11.425931+0000","last_peered":"2024-09-17T03:55:11.425931+0000","last_clean":"2024-09-17T03:55:11.425931+0000","last_became_active":"2024-09-17T03:55:10.096210+0000","last_became_peered":"2024-09-17T03:55:10.096210+0000","last_unstale":"2024-09-17T03:55:11.425931+0000","last_undegraded":"2024-09-17T03:55:11.425931+0000","last_fullsized":"2024-09-17T03:55:11.425931+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:04.260938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104187+0000","last_change":"2024-09-17T03:55:09.099774+0000","last_active":"2024-09-17T03:55:11.104187+0000","last_peered":"2024-09-17T03:55:11.104187+0000","last_clean":"2024-09-17T03:55:11.104187+0000","last_became_active":"2024-09-17T03:55:09.099422+0000","last_became_peered":"2024-09-17T03:55:09.099422+0000","last_unstale":"2024-09-17T03:55:11.104187+0000","last_undegraded":"2024-09-17T03:55:11.104187+0000","last_fullsized":"2024-09-17T03:55:11.104187+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:55:56.088001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426331+0000","last_change":"2024-09-17T03:55:10.112375+0000","last_active":"2024-09-17T03:55:11.426331+0000","last_peered":"2024-09-17T03:55:11.426331+0000","last_clean":"2024-09-17T03:55:11.426331+0000","last_became_active":"2024-09-17T03:55:10.112033+0000","last_became_peered":"2024-09-17T03:55:10.112033+0000","last_unstale":"2024-09-17T03:55:11.426331+0000","last_undegraded":"2024-09-17T03:55:11.426331+0000","last_fullsized":"2024-09-17T03:55:11.426331+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:55:49.414906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.426160+0000","last_change":"2024-09-17T03:55:09.103012+0000","last_active":"2024-09-17T03:55:11.426160+0000","last_peered":"2024-09-17T03:55:11.426160+0000","last_clean":"2024-09-17T03:55:11.426160+0000","last_became_active":"2024-09-17T03:55:09.102701+0000","last_became_peered":"2024-09-17T03:55:09.102701+0000","last_unstale":"2024-09-17T03:55:11.426160+0000","last_undegraded":"2024-09-17T03:55:11.426160+0000","last_fullsized":"2024-09-17T03:55:11.426160+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:02.972775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425468+0000","last_change":"2024-09-17T03:55:10.113187+0000","last_active":"2024-09-17T03:55:11.425468+0000","last_peered":"2024-09-17T03:55:11.425468+0000","last_clean":"2024-09-17T03:55:11.425468+0000","last_became_active":"2024-09-17T03:55:10.112812+0000","last_became_peered":"2024-09-17T03:55:10.112812+0000","last_unstale":"2024-09-17T03:55:11.425468+0000","last_undegraded":"2024-09-17T03:55:11.425468+0000","last_fullsized":"2024-09-17T03:55:11.425468+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:18.693960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.102640+0000","last_change":"2024-09-17T03:55:09.108670+0000","last_active":"2024-09-17T03:55:11.102640+0000","last_peered":"2024-09-17T03:55:11.102640+0000","last_clean":"2024-09-17T03:55:11.102640+0000","last_became_active":"2024-09-17T03:55:09.106248+0000","last_became_peered":"2024-09-17T03:55:09.106248+0000","last_unstale":"2024-09-17T03:55:11.102640+0000","last_undegraded":"2024-09-17T03:55:11.102640+0000","last_fullsized":"2024-09-17T03:55:11.102640+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:02.604730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.423930+0000","last_change":"2024-09-17T03:55:10.126012+0000","last_active":"2024-09-17T03:55:11.423930+0000","last_peered":"2024-09-17T03:55:11.423930+0000","last_clean":"2024-09-17T03:55:11.423930+0000","last_became_active":"2024-09-17T03:55:10.125592+0000","last_became_peered":"2024-09-17T03:55:10.125592+0000","last_unstale":"2024-09-17T03:55:11.423930+0000","last_undegraded":"2024-09-17T03:55:11.423930+0000","last_fullsized":"2024-09-17T03:55:11.423930+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:17.583329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.424054+0000","last_change":"2024-09-17T03:55:09.096029+0000","last_active":"2024-09-17T03:55:11.424054+0000","last_peered":"2024-09-17T03:55:11.424054+0000","last_clean":"2024-09-17T03:55:11.424054+0000","last_became_active":"2024-09-17T03:55:09.091782+0000","last_became_peered":"2024-09-17T03:55:09.091782+0000","last_unstale":"2024-09-17T03:55:11.424054+0000","last_undegraded":"2024-09-17T03:55:11.424054+0000","last_fullsized":"2024-09-17T03:55:11.424054+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:17.583743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":13,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.100467+0000","last_change":"2024-09-17T03:55:10.105398+0000","last_active":"2024-09-17T03:55:11.100467+0000","last_peered":"2024-09-17T03:55:11.100467+0000","last_clean":"2024-09-17T03:55:11.100467+0000","last_became_active":"2024-09-17T03:55:10.105037+0000","last_became_peered":"2024-09-17T03:55:10.105037+0000","last_unstale":"2024-09-17T03:55:11.100467+0000","last_undegraded":"2024-09-17T03:55:11.100467+0000","last_fullsized":"2024-09-17T03:55:11.100467+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:09.066966+0000","last_clean_scrub_stamp":"2024-09-17T03:55:09.066966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:03.260613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.104276+0000","last_change":"2024-09-17T03:55:09.096214+0000","last_active":"2024-09-17T03:55:11.104276+0000","last_peered":"2024-09-17T03:55:11.104276+0000","last_clean":"2024-09-17T03:55:11.104276+0000","last_became_active":"2024-09-17T03:55:09.095625+0000","last_became_peered":"2024-09-17T03:55:09.095625+0000","last_unstale":"2024-09-17T03:55:11.104276+0000","last_undegraded":"2024-09-17T03:55:11.104276+0000","last_fullsized":"2024-09-17T03:55:11.104276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:55:08.064226+0000","last_clean_scrub_stamp":"2024-09-17T03:55:08.064226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:12.569897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":265,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-09-17T03:55:11.425604+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:55:11.425604+0000","last_peered":"2024-09-17T03:55:11.425604+0000","last_clean":"2024-09-17T03:55:11.425604+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:55:11.425604+0000","last_undegraded":"2024-09-17T03:55:11.425604+0000","last_fullsized":"2024-09-17T03:55:11.425604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509512,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37228,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334612,"statfs":{"total":96636764160,"available":96598642688,"internally_reserved":0,"allocated":1925120,"data_stored":1157630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":567250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509514,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40808,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331032,"statfs":{"total":96636764160,"available":96594976768,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509512,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36664,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335176,"statfs":{"total":96636764160,"available":96599220224,"internally_reserved":0,"allocated":1339392,"data_stored":568328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509514,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36648,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335192,"statfs":{"total":96636764160,"available":96599236608,"internally_reserved":0,"allocated":1323008,"data_stored":566646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509514,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36656,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335184,"statfs":{"total":96636764160,"available":96599228416,"internally_reserved":0,"allocated":1331200,"data_stored":566770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542217,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1327104,"data_stored":566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542218,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41400,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330440,"statfs":{"total":96636764160,"available":96594370560,"internally_reserved":0,"allocated":1937408,"data_stored":1158738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:55:20.147 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:55:20.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:55:20.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 -- 172.21.15.129:0/1919840442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f415b890 msgr2=0x7f79f406f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 --2- 172.21.15.129:0/1919840442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f415b890 0x7f79f406f810 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f79f40729b0 tx=0x7f79e402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 -- 172.21.15.129:0/1919840442 shutdown_connections 2024-09-17T03:55:20.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 --2- 172.21.15.129:0/1919840442 >> v2:172.21.15.181:3300/0 conn(0x7f79f40740f0 0x7f79f4074500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 --2- 172.21.15.129:0/1919840442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f415b890 0x7f79f406f810 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 -- 172.21.15.129:0/1919840442 >> 172.21.15.129:0/1919840442 conn(0x7f79f40d0120 msgr2=0x7f79f40d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:20.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 -- 172.21.15.129:0/1919840442 shutdown_connections 2024-09-17T03:55:20.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.410+0000 7f79f8a00640 1 -- 172.21.15.129:0/1919840442 wait complete. 2024-09-17T03:55:20.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.412+0000 7f79f8a00640 1 Processor -- start 2024-09-17T03:55:20.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.412+0000 7f79f8a00640 1 -- start start 2024-09-17T03:55:20.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79f40cf130 0x7f79f415e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f415b890 0x7f79f41f9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40b3300 con 0x7f79f40740f0 2024-09-17T03:55:20.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79f40b3180 con 0x7f79f415b890 2024-09-17T03:55:20.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40b3480 con 0x7f79f40cf130 2024-09-17T03:55:20.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53186/0 (socket says 172.21.15.129:53186) 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 -- 172.21.15.129:0/3164637205 learned_addr learned my addr 172.21.15.129:0/3164637205 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79e9e00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79f40cf130 0x7f79f415e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f3200640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f415b890 0x7f79f41f9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 -- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79f40cf130 msgr2=0x7f79f415e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79f40cf130 0x7f79f415e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 -- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f415b890 msgr2=0x7f79f41f9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f415b890 0x7f79f41f9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.414+0000 7f79f2800640 1 -- 172.21.15.129:0/3164637205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f415ec60 con 0x7f79f40740f0 2024-09-17T03:55:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79f2800640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f79e4034040 tx=0x7f79e4003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e403e030 con 0x7f79f40740f0 2024-09-17T03:55:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f79e4003d10 con 0x7f79f40740f0 2024-09-17T03:55:20.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e4035040 con 0x7f79f40740f0 2024-09-17T03:55:20.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f40c4440 con 0x7f79f40740f0 2024-09-17T03:55:20.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.415+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f40c4a00 con 0x7f79f40740f0 2024-09-17T03:55:20.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.416+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f40b3930 con 0x7f79f40740f0 2024-09-17T03:55:20.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.417+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f79e4041030 con 0x7f79f40740f0 2024-09-17T03:55:20.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.418+0000 7f79eb400640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 0x7f79d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.418+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f79e40bb520 con 0x7f79f40740f0 2024-09-17T03:55:20.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.419+0000 7f79e9e00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 0x7f79d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.420+0000 7f79e9e00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 0x7f79d007b0d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f79f406a0a0 tx=0x7f79dc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:20.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.423+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f79e40838c0 con 0x7f79f40740f0 2024-09-17T03:55:20.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.677+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f79f40c39f0 con 0x7f79f40740f0 2024-09-17T03:55:20.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.680+0000 7f79eb400640 1 -- 172.21.15.129:0/3164637205 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v33) ==== 190+0+147 (secure 0 0 0) 0x7f79e40831c0 con 0x7f79f40740f0 2024-09-17T03:55:20.680 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:55:20.680 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:55:20.681 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:55:20.681 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:55:20.681 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:55:20.681 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 msgr2=0x7f79d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 0x7f79d007b0d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f79f406a0a0 tx=0x7f79dc005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 msgr2=0x7f79f40c3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f79e4034040 tx=0x7f79e4003820 comp rx=0 tx=0).stop 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 shutdown_connections 2024-09-17T03:55:20.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79d0078bc0 0x7f79d007b0d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79f415b890 0x7f79f41f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79f40cf130 0x7f79f415e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 --2- 172.21.15.129:0/3164637205 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79f40740f0 0x7f79f40c3c10 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 >> 172.21.15.129:0/3164637205 conn(0x7f79f40d0120 msgr2=0x7f79f40afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 shutdown_connections 2024-09-17T03:55:20.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.683+0000 7f79f8a00640 1 -- 172.21.15.129:0/3164637205 wait complete. 2024-09-17T03:55:20.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 -- 172.21.15.129:0/4030271682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09780d03a0 msgr2=0x7f09780d07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 --2- 172.21.15.129:0/4030271682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09780d03a0 0x7f09780d07b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f097806fe80 tx=0x7f097002aa10 comp rx=0 tx=0).stop 2024-09-17T03:55:20.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 -- 172.21.15.129:0/4030271682 shutdown_connections 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 --2- 172.21.15.129:0/4030271682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09780d03a0 0x7f09780d07b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 --2- 172.21.15.129:0/4030271682 >> v2:172.21.15.181:3300/0 conn(0x7f0978160af0 0x7f09780b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 --2- 172.21.15.129:0/4030271682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780cf8a0 0x7f09780cd860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 -- 172.21.15.129:0/4030271682 >> 172.21.15.129:0/4030271682 conn(0x7f097806cfc0 msgr2=0x7f097806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 -- 172.21.15.129:0/4030271682 shutdown_connections 2024-09-17T03:55:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.927+0000 7f097ca00640 1 -- 172.21.15.129:0/4030271682 wait complete. 2024-09-17T03:55:20.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.929+0000 7f097ca00640 1 Processor -- start 2024-09-17T03:55:20.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.929+0000 7f097ca00640 1 -- start start 2024-09-17T03:55:20.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09780cf8a0 0x7f0978144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780d03a0 0x7f09781e6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f097815cef0 con 0x7f0978160af0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f097815cd70 con 0x7f09780cf8a0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f097ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f097815d070 con 0x7f09780d03a0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f0977200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f0977200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53204/0 (socket says 172.21.15.129:53204) 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f0976800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09780cf8a0 0x7f0978144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f0977200640 1 -- 172.21.15.129:0/3841219486 learned_addr learned my addr 172.21.15.129:0/3841219486 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.931+0000 7f096de00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780d03a0 0x7f09781e6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 -- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780d03a0 msgr2=0x7f09781e6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780d03a0 0x7f09781e6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 -- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09780cf8a0 msgr2=0x7f0978144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:20.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09780cf8a0 0x7f0978144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:20.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 -- 172.21.15.129:0/3841219486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09781eb0d0 con 0x7f0978160af0 2024-09-17T03:55:20.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f0977200640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0970009d60 tx=0x7f0970003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:20.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970033050 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0970004450 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970004860 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09781eb3b0 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.932+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09781e7490 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.933+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09781e76f0 con 0x7f0978160af0 2024-09-17T03:55:20.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.935+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0970036070 con 0x7f0978160af0 2024-09-17T03:55:20.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.936+0000 7f096f400640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 0x7f095407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:20.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.936+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f09700bc0e0 con 0x7f0978160af0 2024-09-17T03:55:20.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.936+0000 7f0976800640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 0x7f095407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:20.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.937+0000 7f0976800640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 0x7f095407b0d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0968005fd0 tx=0x7f096800a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:20.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:20.940+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0970083e30 con 0x7f0978160af0 2024-09-17T03:55:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.160+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f097815d590 con 0x7f0978160af0 2024-09-17T03:55:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.161+0000 7f096f400640 1 -- 172.21.15.129:0/3841219486 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5228 (secure 0 0 0) 0x7f0970088860 con 0x7f0978160af0 2024-09-17T03:55:21.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:21.161 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"btime":"2024-09-17T03:55:13:284539+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/4006656424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4006656424},{"type":"v1","addr":"172.21.15.129:6835","nonce":4006656424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/473985729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":473985729},{"type":"v1","addr":"172.21.15.181:6836","nonce":473985729}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/826515162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":826515162},{"type":"v1","addr":"172.21.15.181:6837","nonce":826515162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:12.783578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7005":{"gid":7005,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/153768962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":153768962},{"type":"v1","addr":"172.21.15.129:6834","nonce":153768962}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7005,"qdb_cluster":[7005]},"id":8}]} 2024-09-17T03:55:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 80 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.163+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 msgr2=0x7f095407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.163+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 0x7f095407b0d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0968005fd0 tx=0x7f096800a250 comp rx=0 tx=0).stop 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 msgr2=0x7f09781f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0970009d60 tx=0x7f0970003f10 comp rx=0 tx=0).stop 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 shutdown_connections 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0954078bc0 0x7f095407b0d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0978160af0 0x7f09781f5c70 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09780d03a0 0x7f09781e6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 --2- 172.21.15.129:0/3841219486 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09780cf8a0 0x7f0978144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 >> 172.21.15.129:0/3841219486 conn(0x7f097806cfc0 msgr2=0x7f097806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 shutdown_connections 2024-09-17T03:55:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.164+0000 7f097ca00640 1 -- 172.21.15.129:0/3841219486 wait complete. 2024-09-17T03:55:21.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 80, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:55:09.589645+0000', 'modified': '2024-09-17T03:55:12.783578+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7005}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7005': {'gid': 7005, 'name': 'c', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/153768962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 153768962}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 153768962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7005, 'qdb_cluster': [7005]} 2024-09-17T03:55:21.592 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:55:21.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:55:21.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.815+0000 7f7269400640 1 -- 172.21.15.129:0/1459727824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640cd1c0 msgr2=0x7f72641609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.815+0000 7f7269400640 1 --2- 172.21.15.129:0/1459727824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640cd1c0 0x7f72641609c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f725c005fd0 tx=0x7f725c02aa90 comp rx=0 tx=0).stop 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 -- 172.21.15.129:0/1459727824 shutdown_connections 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 --2- 172.21.15.129:0/1459727824 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640cd1c0 0x7f72641609c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 --2- 172.21.15.129:0/1459727824 >> v2:172.21.15.181:3300/0 conn(0x7f72640b4140 0x7f72640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 -- 172.21.15.129:0/1459727824 >> 172.21.15.129:0/1459727824 conn(0x7f726406d410 msgr2=0x7f72640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 -- 172.21.15.129:0/1459727824 shutdown_connections 2024-09-17T03:55:21.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.816+0000 7f7269400640 1 -- 172.21.15.129:0/1459727824 wait complete. 2024-09-17T03:55:21.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.818+0000 7f7269400640 1 Processor -- start 2024-09-17T03:55:21.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.818+0000 7f7269400640 1 -- start start 2024-09-17T03:55:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.819+0000 7f7269400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640705a0 0x7f72640c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7269400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72640b4140 0x7f72641f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7269400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7269400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7264160260 con 0x7f72640cd1c0 2024-09-17T03:55:21.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7269400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f72641600e0 con 0x7f72640b4140 2024-09-17T03:55:21.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7269400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7264160f00 con 0x7f72640705a0 2024-09-17T03:55:21.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53206/0 (socket says 172.21.15.129:53206) 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 -- 172.21.15.129:0/2704463846 learned_addr learned my addr 172.21.15.129:0/2704463846 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7262800640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640705a0 0x7f72640c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7259e00640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72640b4140 0x7f72641f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 -- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640705a0 msgr2=0x7f72640c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640705a0 0x7f72640c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 -- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72640b4140 msgr2=0x7f72641f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72640b4140 0x7f72641f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.820+0000 7f7263200640 1 -- 172.21.15.129:0/2704463846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72640cba20 con 0x7f72640cd1c0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f7263200640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f725c009bf0 tx=0x7f725c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f725b400640 1 -- 172.21.15.129:0/2704463846 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c044070 con 0x7f72640cd1c0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f725b400640 1 -- 172.21.15.129:0/2704463846 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f725c003e00 con 0x7f72640cd1c0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f725b400640 1 -- 172.21.15.129:0/2704463846 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c0044b0 con 0x7f72640cd1c0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72640cbd00 con 0x7f72640cd1c0 2024-09-17T03:55:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.821+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72640cc2c0 con 0x7f72640cd1c0 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.823+0000 7f725b400640 1 -- 172.21.15.129:0/2704463846 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f725c047070 con 0x7f72640cd1c0 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.824+0000 7f725b400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 0x7f724007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.824+0000 7f725b400640 1 -- 172.21.15.129:0/2704463846 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f725c0bbc40 con 0x7f72640cd1c0 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.825+0000 7f7262800640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 0x7f724007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.825+0000 7f72705d5740 1 Processor -- start 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.825+0000 7f72705d5740 1 -- start start 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f7262800640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 0x7f724007b0a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7254000f60 tx=0x7f7254009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f2545f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5626f24cf430 con 0x5626f217c3b0 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5626f24bf5d0 con 0x5626f21a5270 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.826+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5626f24aea00 con 0x5626f2184590 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53216/0 (socket says 172.21.15.129:53216) 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7262800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 -- 172.21.15.129:0/640411154 learned_addr learned my addr 172.21.15.129:0/640411154 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7263200640 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f2545f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 -- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 msgr2=0x5626f2545f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f2545f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 -- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 msgr2=0x5626f2575970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7259e00640 1 -- 172.21.15.129:0/640411154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626f2477520 con 0x5626f217c3b0 2024-09-17T03:55:21.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.827+0000 7f7262800640 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:55:21.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.828+0000 7f7259e00640 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f724c009b80 tx=0x7f724c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445bce3a3d12977b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.828+0000 7f7253000640 1 -- 172.21.15.129:0/640411154 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c004030 con 0x5626f217c3b0 2024-09-17T03:55:21.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.828+0000 7f7253000640 1 -- 172.21.15.129:0/640411154 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f724c0041d0 con 0x5626f217c3b0 2024-09-17T03:55:21.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.828+0000 7f7253000640 1 -- 172.21.15.129:0/640411154 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c0035f0 con 0x5626f217c3b0 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 -- 172.21.15.129:0/640411154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 msgr2=0x5626f2575eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f724c009b80 tx=0x7f724c011810 comp rx=0 tx=0).stop 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 -- 172.21.15.129:0/640411154 shutdown_connections 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f2545f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f217c3b0 0x5626f2575eb0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 --2- 172.21.15.129:0/640411154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 -- 172.21.15.129:0/640411154 >> 172.21.15.129:0/640411154 conn(0x5626f21993c0 msgr2=0x5626f254b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 -- 172.21.15.129:0/640411154 shutdown_connections 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.829+0000 7f72705d5740 1 -- 172.21.15.129:0/640411154 wait complete. 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.830+0000 7f72705d5740 1 -- start start 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.830+0000 7f72705d5740 1 Processor -- start 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2570450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f256eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5626f2455330 con 0x5626f21a5270 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5626f24b2810 con 0x5626f217c3b0 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.831+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5626f221f620 con 0x5626f2184590 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f72705d5740 10 client.0 _ll_register_callbacks cb 0x5626f245a420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53232/0 (socket says 172.21.15.129:53232) 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 -- 172.21.15.129:0/1764868876 learned_addr learned my addr 172.21.15.129:0/1764868876 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7262800640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2570450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7259e00640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f256eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 -- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 msgr2=0x5626f256eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f256eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 -- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 msgr2=0x5626f2570450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2570450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 -- 172.21.15.129:0/1764868876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626f2210850 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7263200640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f725c006010 tx=0x7f725c083230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953b80a09607b483 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.832+0000 7f7247400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c004d40 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f725c082bf0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724c0041d0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f725c0883c0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 10 client.7161 fetch_fsmap learned FSMap version 80 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) -- 0x5626f22876b0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f724c0041d0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 80) ==== 3502+0+0 (secure 0 0 0) 0x7f725c07f2e0 con 0x5626f21a5270 2024-09-17T03:55:21.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 10 client.7161 fetch_fsmap finished waiting for FSMap version 80 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 10 client.7161 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 20 client.7161 populate_metadata read hostname 'smithi129' 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2163e40 0x5626f25a3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 4 client.7161 mds_command: new command op to 7005 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.833+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 --> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626f24c1520 con 0x5626f2163e40 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.834+0000 7f7262800640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2163e40 0x5626f25a3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.834+0000 7f7247400640 10 client.7161 ms_handle_connect on v2:172.21.15.129:6832/153768962 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.834+0000 7f7262800640 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2163e40 0x5626f25a3230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8482984921d879b server_cookie=46f42b2cf4add163 in_seq=0 out_seq=0 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f7247400640 1 -- 172.21.15.129:0/1764868876 <== mds.0 v2:172.21.15.129:6832/153768962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5626f24c1520 con 0x5626f2163e40 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f7247400640 10 client.7161 handle_command_reply: tid=0 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 client.7161 shutdown 2024-09-17T03:55:21.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 msgr2=0x5626f256f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f725c006010 tx=0x7f725c083230 comp rx=0 tx=0).stop 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 shutdown_connections 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2163e40 0x5626f25a3230 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f21a5270 0x5626f256f0a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f2184590 0x5626f256eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f7247400640 0 client.7161 ms_handle_reset on v2:172.21.15.129:6832/153768962 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 --2- 172.21.15.129:0/1764868876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2570450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.835+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 >> 172.21.15.129:0/1764868876 conn(0x5626f21993c0 msgr2=0x5626f25471d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.836+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 shutdown_connections 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.836+0000 7f72705d5740 1 -- 172.21.15.129:0/1764868876 wait complete. 2024-09-17T03:55:21.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.836+0000 7f72705d5740 20 client.7161 trim_cache size 0 max 16384 2024-09-17T03:55:21.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.854+0000 7f72705d5740 1 Processor -- start 2024-09-17T03:55:21.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.855+0000 7f72705d5740 1 -- start start 2024-09-17T03:55:21.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f217c3b0 0x5626f259f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f72641f6b00 con 0x5626f2184590 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f72641f93e0 con 0x5626f21a5270 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f72641f64b0 con 0x5626f217c3b0 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53246/0 (socket says 172.21.15.129:53246) 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 -- 172.21.15.129:0/1893046017 learned_addr learned my addr 172.21.15.129:0/1893046017 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 -- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f217c3b0 msgr2=0x5626f259f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f217c3b0 0x5626f259f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 -- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 msgr2=0x5626f2575bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7263200640 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.856+0000 7f7259e00640 1 -- 172.21.15.129:0/1893046017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626f24aeb80 con 0x5626f2184590 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.857+0000 7f7259e00640 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f724c011d10 tx=0x7f724c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98478bc6a137a445 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.857+0000 7f7252600640 1 -- 172.21.15.129:0/1893046017 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c027070 con 0x5626f2184590 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.857+0000 7f7252600640 1 -- 172.21.15.129:0/1893046017 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f724c009df0 con 0x5626f2184590 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.857+0000 7f7252600640 1 -- 172.21.15.129:0/1893046017 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c0042d0 con 0x5626f2184590 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 -- 172.21.15.129:0/1893046017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 msgr2=0x5626f25756b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f724c011d10 tx=0x7f724c004730 comp rx=0 tx=0).stop 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 -- 172.21.15.129:0/1893046017 shutdown_connections 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f21a5270 0x5626f2575bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f25756b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 --2- 172.21.15.129:0/1893046017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f217c3b0 0x5626f259f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 -- 172.21.15.129:0/1893046017 >> 172.21.15.129:0/1893046017 conn(0x5626f21993c0 msgr2=0x5626f2593dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 -- 172.21.15.129:0/1893046017 shutdown_connections 2024-09-17T03:55:21.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.858+0000 7f72705d5740 1 -- 172.21.15.129:0/1893046017 wait complete. 2024-09-17T03:55:21.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.859+0000 7f72705d5740 1 -- start start 2024-09-17T03:55:21.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.859+0000 7f72705d5740 1 Processor -- start 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2548ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f21a5270 0x5626f2610b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5626f2572680 con 0x5626f2184590 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5626f2570990 con 0x5626f217c3b0 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5626f2571b20 con 0x5626f21a5270 2024-09-17T03:55:21.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f72705d5740 10 client.0 _ll_register_callbacks cb 0x7f725c083370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53260/0 (socket says 172.21.15.129:53260) 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 -- 172.21.15.129:0/2349559085 learned_addr learned my addr 172.21.15.129:0/2349559085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7263200640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f21a5270 0x5626f2610b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7262800640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2548ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 -- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f21a5270 msgr2=0x5626f2610b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f21a5270 0x5626f2610b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 -- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 msgr2=0x5626f2548ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2548ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.861+0000 7f7259e00640 1 -- 172.21.15.129:0/2349559085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626f256cb90 con 0x5626f2184590 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7259e00640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f724c01b000 tx=0x7f724c005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=730675678df3a4a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:21.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c027040 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f724c003db0 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f724c009df0 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=80) ==== 24+0+0 (secure 0 0 0) 0x7f724c0044f0 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f72705d5740 10 client.7167 fetch_fsmap learned FSMap version 80 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=80,monmap=2+}) -- 0x5626f2610310 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f724c009df0 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 80) ==== 3502+0+0 (secure 0 0 0) 0x7f724c018070 con 0x5626f2184590 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.862+0000 7f72705d5740 10 client.7167 fetch_fsmap finished waiting for FSMap version 80 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f72705d5740 10 client.7167 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f72705d5740 20 client.7167 populate_metadata read hostname 'smithi129' 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2575420 0x5626f2575830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:21.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f72705d5740 4 client.7167 mds_command: new command op to 7005 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 --> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] -- command(tid 0: {"prefix": "status"}) -- 0x5626f24bcfd0 con 0x5626f2575420 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f7262800640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2575420 0x5626f2575830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f7247400640 10 client.7167 ms_handle_connect on v2:172.21.15.129:6832/153768962 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.863+0000 7f7262800640 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2575420 0x5626f2575830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ce044127bccd74 server_cookie=e6322019b0ee1221 in_seq=0 out_seq=0 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f7247400640 1 -- 172.21.15.129:0/2349559085 <== mds.0 v2:172.21.15.129:6832/153768962 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5626f24bcfd0 con 0x5626f2575420 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f7247400640 10 client.7167 handle_command_reply: tid=0 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 client.7167 shutdown 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 msgr2=0x5626f2606aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f724c01b000 tx=0x7f724c005dc0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 shutdown_connections 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:6832/153768962,v1:172.21.15.129:6834/153768962] conn(0x5626f2575420 0x5626f2575830 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5626f21a5270 0x5626f2610b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5626f2184590 0x5626f2606aa0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f7247400640 0 client.7167 ms_handle_reset on v2:172.21.15.129:6832/153768962 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 --2- 172.21.15.129:0/2349559085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5626f217c3b0 0x5626f2548ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 >> 172.21.15.129:0/2349559085 conn(0x5626f21993c0 msgr2=0x5626f259f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 shutdown_connections 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 1 -- 172.21.15.129:0/2349559085 wait complete. 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.864+0000 7f72705d5740 20 client.7167 trim_cache size 0 max 16384 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:55:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 7005, 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.785487498, 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 80, 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 107, 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 107, 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.724981923000001 2024-09-17T03:55:21.871 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 msgr2=0x7f724007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 0x7f724007b0a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f7254000f60 tx=0x7f7254009040 comp rx=0 tx=0).stop 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 msgr2=0x7f72641f7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f725c009bf0 tx=0x7f725c003c10 comp rx=0 tx=0).stop 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 shutdown_connections 2024-09-17T03:55:21.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7240078b90 0x7f724007b0a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f72640cd1c0 0x7f72641f7830 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72640b4140 0x7f72641f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 --2- 172.21.15.129:0/2704463846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f72640705a0 0x7f72640c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 >> 172.21.15.129:0/2704463846 conn(0x7f726406d410 msgr2=0x7f726406f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 shutdown_connections 2024-09-17T03:55:21.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:21.865+0000 7f7269400640 1 -- 172.21.15.129:0/2704463846 wait complete. 2024-09-17T03:55:21.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.110+0000 7f96c9200640 1 -- 172.21.15.129:0/1493647481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b3b90 msgr2=0x7f96c40cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.110+0000 7f96c9200640 1 --2- 172.21.15.129:0/1493647481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b3b90 0x7f96c40cbbf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009860 tx=0x7f96ac02a510 comp rx=0 tx=0).stop 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.110+0000 7f96c9200640 1 -- 172.21.15.129:0/1493647481 shutdown_connections 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.110+0000 7f96c9200640 1 --2- 172.21.15.129:0/1493647481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b3b90 0x7f96c40cbbf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.110+0000 7f96c9200640 1 -- 172.21.15.129:0/1493647481 >> 172.21.15.129:0/1493647481 conn(0x7f96c406d410 msgr2=0x7f96c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.111+0000 7f96c9200640 1 -- 172.21.15.129:0/1493647481 shutdown_connections 2024-09-17T03:55:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.111+0000 7f96c9200640 1 -- 172.21.15.129:0/1493647481 wait complete. 2024-09-17T03:55:22.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.113+0000 7f96c9200640 1 Processor -- start 2024-09-17T03:55:22.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.113+0000 7f96c9200640 1 -- start start 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 0x7f96c4166e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96c40c9ca0 0x7f96c40ae710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96c406f760 con 0x7f96c40ce170 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f96c406f5e0 con 0x7f96c40c9ca0 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96c9200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f96c406f8e0 con 0x7f96c40b2950 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.114+0000 7f96b9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53270/0 (socket says 172.21.15.129:53270) 2024-09-17T03:55:22.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 -- 172.21.15.129:0/3421312630 learned_addr learned my addr 172.21.15.129:0/3421312630 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96c1e00640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 0x7f96c4166e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96c2800640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96c40c9ca0 0x7f96c40ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 -- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 msgr2=0x7f96c4166e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 0x7f96c4166e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 -- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96c40c9ca0 msgr2=0x7f96c40ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96c40c9ca0 0x7f96c40ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 -- 172.21.15.129:0/3421312630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96c415cb30 con 0x7f96c40ce170 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96c1e00640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 0x7f96c4166e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:22.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96b9400640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009980 tx=0x7f96ac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.115+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac005c90 con 0x7f96c40ce170 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.116+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96ac0318b0 con 0x7f96c40ce170 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.116+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96c4158530 con 0x7f96c40ce170 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.116+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ac031ce0 con 0x7f96c40ce170 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.116+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96c4158af0 con 0x7f96c40ce170 2024-09-17T03:55:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.117+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96c40cbbf0 con 0x7f96c40ce170 2024-09-17T03:55:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.118+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f96ac03b410 con 0x7f96c40ce170 2024-09-17T03:55:22.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.119+0000 7f96baa00640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 0x7f96a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:22.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.119+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f96ac0bc250 con 0x7f96c40ce170 2024-09-17T03:55:22.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.120+0000 7f96c1e00640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 0x7f96a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:22.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.120+0000 7f96c1e00640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 0x7f96a007b0a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f96b4000f60 tx=0x7f96b4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:22.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.124+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f96ac03f030 con 0x7f96c40ce170 2024-09-17T03:55:22.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.341+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f96c406a140 con 0x7f96c40ce170 2024-09-17T03:55:22.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.341+0000 7f96baa00640 1 -- 172.21.15.129:0/3421312630 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v80) ==== 53+0+73 (secure 0 0 0) 0x7f96ac084040 con 0x7f96c40ce170 2024-09-17T03:55:22.341 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 msgr2=0x7f96a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 0x7f96a007b0a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f96b4000f60 tx=0x7f96b4009040 comp rx=0 tx=0).stop 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 msgr2=0x7f96c40ae1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f96ac009980 tx=0x7f96ac003c10 comp rx=0 tx=0).stop 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 shutdown_connections 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96a0078b90 0x7f96a007b0a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96c40c9ca0 0x7f96c40ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96c40ce170 0x7f96c40ae1d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 --2- 172.21.15.129:0/3421312630 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96c40b2950 0x7f96c4166e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 >> 172.21.15.129:0/3421312630 conn(0x7f96c406d410 msgr2=0x7f96c4073a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 shutdown_connections 2024-09-17T03:55:22.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22.344+0000 7f96c9200640 1 -- 172.21.15.129:0/3421312630 wait complete. 2024-09-17T03:55:22.370 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:55:22.370 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:55:22.371 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:22.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.371 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.407 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42370sec preferred_lft 42370sec 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:55:22.408 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:55:22.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:55:22.410 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:22.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.410 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:55:22.503 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:55:22.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.507 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:55:22.543 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:55:22.543 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:55:22.543 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:55:22.543 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:55:22.543 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:55:22.543 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:55:22.544 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:55:22.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:55:22.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:55:22.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:55:22.544 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:55:22.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:55:22.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:22.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.740 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:55:22.775 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:55:22.775 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:55:22.776 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:22.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:55:22.872 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:55:22.872 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:22.907 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:22.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:55:22.907 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:22.983 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:55:22.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:22.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:23.178 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:23.179 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:23.179 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:23.199 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:23.200 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:23.200 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:23.200 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:55:23.200 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:55:23.229 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:55:23.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:55:23.231 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:55:23.231 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:55:23.231 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:55:23.231 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:55:23.231 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:55:23.234 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:55:23.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:23.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:23.291 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:55:23.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:55:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:23.389 INFO:teuthology.orchestra.run.smithi129.stdout:client6161 2024-09-17T03:55:23.390 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:23.390 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:23.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:23.426 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_1) 2024-09-17T03:55:23.507 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:23.507 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:23.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:23.583 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=dir_1/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T03:55:23.772 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T03:55:23.772 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T03:55:23.772 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.111588 s, 75.2 MB/s 2024-09-17T03:55:23.773 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:55:23.773 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:55:23.773 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T03:55:23.774 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:55:23.774 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_1/file_a").st_ino) 2024-09-17T03:55:23.774 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:55:23.873 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627777 2024-09-17T03:55:23.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:23.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:23.885 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_2) 2024-09-17T03:55:23.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:23.963 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_1/file_a dir_2/file_b) 2024-09-17T03:55:24.043 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:55:24.044 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:55:24.044 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T03:55:24.044 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:55:24.044 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_2/file_b").st_ino) 2024-09-17T03:55:24.044 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:55:24.183 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627777 2024-09-17T03:55:24.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:24.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:24.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 -- 172.21.15.129:0/1237402596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6730003ac0 msgr2=0x7f6730003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:24.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 --2- 172.21.15.129:0/1237402596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6730003ac0 0x7f6730003f50 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f673c0016c0 tx=0x7f6730005830 comp rx=0 tx=0).stop 2024-09-17T03:55:24.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 -- 172.21.15.129:0/1237402596 shutdown_connections 2024-09-17T03:55:24.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 --2- 172.21.15.129:0/1237402596 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6730003ac0 0x7f6730003f50 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 -- 172.21.15.129:0/1237402596 >> 172.21.15.129:0/1237402596 conn(0x7f674c06cfc0 msgr2=0x7f674c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:24.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 -- 172.21.15.129:0/1237402596 shutdown_connections 2024-09-17T03:55:24.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.457+0000 7f6750c00640 1 -- 172.21.15.129:0/1237402596 wait complete. 2024-09-17T03:55:24.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.459+0000 7f6750c00640 1 Processor -- start 2024-09-17T03:55:24.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.459+0000 7f6750c00640 1 -- start start 2024-09-17T03:55:24.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6730003ac0 0x7f674c1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:24.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674c0ad8d0 0x7f674c14cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0cf740 con 0x7f674c154b00 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f674c0cf5c0 con 0x7f6730003ac0 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6750c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f674c0cf8c0 con 0x7f674c0ad8d0 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59136/0 (socket says 172.21.15.129:59136) 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 -- 172.21.15.129:0/3558543675 learned_addr learned my addr 172.21.15.129:0/3558543675 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674b400640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6730003ac0 0x7f674c1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f6742a00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674c0ad8d0 0x7f674c14cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 -- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674c0ad8d0 msgr2=0x7f674c14cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674c0ad8d0 0x7f674c14cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 -- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6730003ac0 msgr2=0x7f674c1f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:24.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6730003ac0 0x7f674c1f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.461+0000 7f674be00640 1 -- 172.21.15.129:0/3558543675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c0b00f0 con 0x7f674c154b00 2024-09-17T03:55:24.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f674be00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f67300057e0 tx=0x7f6744010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:24.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f674400ee70 con 0x7f674c154b00 2024-09-17T03:55:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6744015070 con 0x7f674c154b00 2024-09-17T03:55:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744003970 con 0x7f674c154b00 2024-09-17T03:55:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c154020 con 0x7f674c154b00 2024-09-17T03:55:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.462+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1545e0 con 0x7f674c154b00 2024-09-17T03:55:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.463+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674c06a140 con 0x7f674c154b00 2024-09-17T03:55:24.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.465+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6744003b10 con 0x7f674c154b00 2024-09-17T03:55:24.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.466+0000 7f6743e00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 0x7f672807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:24.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.466+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f674409b4c0 con 0x7f674c154b00 2024-09-17T03:55:24.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.466+0000 7f674b400640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 0x7f672807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:24.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.467+0000 7f674b400640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 0x7f672807b0a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f673c032000 tx=0x7f673c001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:24.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.471+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f67440638f0 con 0x7f674c154b00 2024-09-17T03:55:24.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.688+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f674c151f80 con 0x7f674c154b00 2024-09-17T03:55:24.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.688+0000 7f6743e00640 1 -- 172.21.15.129:0/3558543675 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5228 (secure 0 0 0) 0x7f67440631f0 con 0x7f674c154b00 2024-09-17T03:55:24.688 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:24.689 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"btime":"2024-09-17T03:55:13:284539+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/4006656424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4006656424},{"type":"v1","addr":"172.21.15.129:6835","nonce":4006656424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/473985729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":473985729},{"type":"v1","addr":"172.21.15.181:6836","nonce":473985729}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/826515162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":826515162},{"type":"v1","addr":"172.21.15.181:6837","nonce":826515162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:12.783578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7005":{"gid":7005,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/153768962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":153768962},{"type":"v1","addr":"172.21.15.129:6834","nonce":153768962}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7005,"qdb_cluster":[7005]},"id":8}]} 2024-09-17T03:55:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 80 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 msgr2=0x7f672807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 0x7f672807b0a0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f673c032000 tx=0x7f673c001190 comp rx=0 tx=0).stop 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 msgr2=0x7f674c0d0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f67300057e0 tx=0x7f6744010e90 comp rx=0 tx=0).stop 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 shutdown_connections 2024-09-17T03:55:24.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6728078b90 0x7f672807b0a0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674c154b00 0x7f674c0d0260 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674c0ad8d0 0x7f674c14cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 --2- 172.21.15.129:0/3558543675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6730003ac0 0x7f674c1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 >> 172.21.15.129:0/3558543675 conn(0x7f674c06cfc0 msgr2=0x7f674c15f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 shutdown_connections 2024-09-17T03:55:24.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:24.691+0000 7f6750c00640 1 -- 172.21.15.129:0/3558543675 wait complete. 2024-09-17T03:55:24.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:55:24.970 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:55:24.970 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:55:24.970 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T03:55:25.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.113+0000 7f651b2ccf40 1 Processor -- start 2024-09-17T03:55:25.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.113+0000 7f651b2ccf40 1 -- start start 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0c53080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0ce0bf0 0x5582e0d72e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5582e0b58c60 con 0x5582e0cc9b70 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5582e0bb71d0 con 0x5582e0ce07e0 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5582e0c0a280 con 0x5582e0ce0bf0 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59146/0 (socket says 172.21.15.129:59146) 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 -- 172.21.15.129:0/1772942613 learned_addr learned my addr 172.21.15.129:0/1772942613 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514e00640 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0ce0bf0 0x5582e0d72e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6513a00640 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0c53080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 -- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0ce0bf0 msgr2=0x5582e0d72e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0ce0bf0 0x5582e0d72e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 -- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 msgr2=0x5582e0c53080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0c53080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.115+0000 7f6514400640 1 -- 172.21.15.129:0/1772942613 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e0cc2c20 con 0x5582e0cc9b70 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.116+0000 7f6514400640 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f650400b790 tx=0x7f650400bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad22494c97835d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.116+0000 7f6512600640 1 -- 172.21.15.129:0/1772942613 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650400ab40 con 0x5582e0cc9b70 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.116+0000 7f6512600640 1 -- 172.21.15.129:0/1772942613 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f650400ace0 con 0x5582e0cc9b70 2024-09-17T03:55:25.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.116+0000 7f6512600640 1 -- 172.21.15.129:0/1772942613 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504015680 con 0x5582e0cc9b70 2024-09-17T03:55:25.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 -- 172.21.15.129:0/1772942613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 msgr2=0x5582e0cca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f650400b790 tx=0x7f650400bc50 comp rx=0 tx=0).stop 2024-09-17T03:55:25.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 -- 172.21.15.129:0/1772942613 shutdown_connections 2024-09-17T03:55:25.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0ce0bf0 0x5582e0d72e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0c53080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/1772942613 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0cc9b70 0x5582e0cca0f0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 -- 172.21.15.129:0/1772942613 >> 172.21.15.129:0/1772942613 conn(0x5582e0b29ac0 msgr2=0x5582e0c520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 -- 172.21.15.129:0/1772942613 shutdown_connections 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.117+0000 7f651b2ccf40 1 -- 172.21.15.129:0/1772942613 wait complete. 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.118+0000 7f651b2ccf40 1 Processor -- start 2024-09-17T03:55:25.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.118+0000 7f651b2ccf40 1 -- start start 2024-09-17T03:55:25.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0cc9b70 0x5582e0cc72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0d68700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 0x5582e0d68c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5582e0cc67d0 con 0x5582e0ce0bf0 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5582e0cbdb80 con 0x5582e0ce07e0 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.119+0000 7f651b2ccf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5582e0cbdfa0 con 0x5582e0cc9b70 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 0x5582e0d68c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 0x5582e0d68c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59150/0 (socket says 172.21.15.129:59150) 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 -- 172.21.15.129:0/37603646 learned_addr learned my addr 172.21.15.129:0/37603646 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514400640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0cc9b70 0x5582e0cc72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6513a00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0d68700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 -- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0cc9b70 msgr2=0x5582e0cc72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5582e0cc9b70 0x5582e0cc72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 -- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 msgr2=0x5582e0d68700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5582e0ce07e0 0x5582e0d68700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 -- 172.21.15.129:0/37603646 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582e0c53420 con 0x5582e0ce0bf0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.120+0000 7f6514e00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 0x5582e0d68c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f650c025db0 tx=0x7f650c04d770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.121+0000 7f6511200640 1 -- 172.21.15.129:0/37603646 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c065030 con 0x5582e0ce0bf0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.121+0000 7f6511200640 1 -- 172.21.15.129:0/37603646 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f650c004110 con 0x5582e0ce0bf0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.121+0000 7f6511200640 1 -- 172.21.15.129:0/37603646 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c061450 con 0x5582e0ce0bf0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.121+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5582e0c53700 con 0x5582e0ce0bf0 2024-09-17T03:55:25.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.121+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582e0cc1ba0 con 0x5582e0ce0bf0 2024-09-17T03:55:25.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.123+0000 7f6511200640 1 -- 172.21.15.129:0/37603646 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f650c05e050 con 0x5582e0ce0bf0 2024-09-17T03:55:25.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.124+0000 7f6511200640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64f4077570 0x7f64f4079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.124+0000 7f6511200640 1 -- 172.21.15.129:0/37603646 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f650c0dd7e0 con 0x5582e0ce0bf0 2024-09-17T03:55:25.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.124+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5582e0cc1650 0x5582e0d678b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.124+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107) -- 0x5582e0cc77f0 con 0x5582e0cc1650 2024-09-17T03:55:25.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.124+0000 7f6514400640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64f4077570 0x7f64f4079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.125+0000 7f6513a00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5582e0cc1650 0x5582e0d678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.125+0000 7f6514400640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64f4077570 0x7f64f4079a80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6504000c00 tx=0x7f650400d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.125+0000 7f6513a00640 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5582e0cc1650 0x5582e0d678b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f6513a00640 1 -- 172.21.15.129:0/37603646 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv3 ondisk = 0) ==== 164+0+93 (crc 0 0 0) 0x7f65000116b0 con 0x5582e0cc1650 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5582e0cc1650 msgr2=0x5582e0d678b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5582e0cc1650 0x5582e0d678b0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64f4077570 msgr2=0x7f64f4079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f64f4077570 0x7f64f4079a80 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f6504000c00 tx=0x7f650400d040 comp rx=0 tx=0).stop 2024-09-17T03:55:25.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 msgr2=0x5582e0d68c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 --2- 172.21.15.129:0/37603646 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5582e0ce0bf0 0x5582e0d68c40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f650c025db0 tx=0x7f650c04d770 comp rx=0 tx=0).stop 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f6514e00640 1 -- 172.21.15.129:0/37603646 reap_dead start 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 shutdown_connections 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 >> 172.21.15.129:0/37603646 conn(0x5582e0b29ac0 msgr2=0x5582e0c52010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 shutdown_connections 2024-09-17T03:55:25.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.126+0000 7f651b2ccf40 1 -- 172.21.15.129:0/37603646 wait complete. 2024-09-17T03:55:25.138 INFO:teuthology.orchestra.run.smithi129.stdout:Wfile_adir_1 2024-09-17T03:55:25.138 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:55:25.333 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:55:25.333 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T03:55:25.333 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:55:25.333 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1099511627776, 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_a", 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 8 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "dir_1", 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 15 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 17, 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:55:25.334 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:55:25.335 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:25.354 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:25.355 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:25.390 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:25.390 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:25.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 -- 172.21.15.129:0/3013948570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 msgr2=0x7f7b2c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 --2- 172.21.15.129:0/3013948570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c0ccc80 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c072e00 tx=0x7f7b1802a560 comp rx=0 tx=0).stop 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 -- 172.21.15.129:0/3013948570 shutdown_connections 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 --2- 172.21.15.129:0/3013948570 >> v2:172.21.15.181:3300/0 conn(0x7f7b2c06f2e0 0x7f7b2c0cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 --2- 172.21.15.129:0/3013948570 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c0ccc80 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 --2- 172.21.15.129:0/3013948570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 0x7f7b2c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 -- 172.21.15.129:0/3013948570 >> 172.21.15.129:0/3013948570 conn(0x7f7b2c06d410 msgr2=0x7f7b2c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:25.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 -- 172.21.15.129:0/3013948570 shutdown_connections 2024-09-17T03:55:25.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.728+0000 7f7b34400640 1 -- 172.21.15.129:0/3013948570 wait complete. 2024-09-17T03:55:25.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.731+0000 7f7b34400640 1 Processor -- start 2024-09-17T03:55:25.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.731+0000 7f7b34400640 1 -- start start 2024-09-17T03:55:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 0x7f7b2c1570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 0x7f7b2c157600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c160b40 con 0x7f7b2c06eed0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1609c0 con 0x7f7b2c06f2e0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b34400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7b2c160cc0 con 0x7f7b2c0705a0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.732+0000 7f7b31a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59162/0 (socket says 172.21.15.129:59162) 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 -- 172.21.15.129:0/4042673619 learned_addr learned my addr 172.21.15.129:0/4042673619 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b32400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 0x7f7b2c157600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31000640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 0x7f7b2c1570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 -- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 msgr2=0x7f7b2c157600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 0x7f7b2c157600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 -- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 msgr2=0x7f7b2c1570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 0x7f7b2c1570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 -- 172.21.15.129:0/4042673619 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c15d1f0 con 0x7f7b2c06eed0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31000640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 0x7f7b2c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b31a00640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000d740 tx=0x7f7b2000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20017070 con 0x7f7b2c06eed0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b20009d20 con 0x7f7b2c06eed0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.733+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c15d4d0 con 0x7f7b2c06eed0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.734+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b200106c0 con 0x7f7b2c06eed0 2024-09-17T03:55:25.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.734+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c15da90 con 0x7f7b2c06eed0 2024-09-17T03:55:25.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.735+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b2c161220 con 0x7f7b2c06eed0 2024-09-17T03:55:25.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.736+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7b2001a070 con 0x7f7b2c06eed0 2024-09-17T03:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.737+0000 7f7b29600640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 0x7f7b0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.737+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f7b2009af10 con 0x7f7b2c06eed0 2024-09-17T03:55:25.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.738+0000 7f7b31000640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 0x7f7b0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:25.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.738+0000 7f7b31000640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 0x7f7b0c07b0a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7b18033000 tx=0x7f7b18004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:25.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.741+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7b20063350 con 0x7f7b2c06eed0 2024-09-17T03:55:25.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.937+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7b2c06a140 con 0x7f7b2c06eed0 2024-09-17T03:55:25.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.938+0000 7f7b29600640 1 -- 172.21.15.129:0/4042673619 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) ==== 80+0+0 (secure 0 0 0) 0x7f7b20062c50 con 0x7f7b2c06eed0 2024-09-17T03:55:25.938 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:55:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 msgr2=0x7f7b0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 0x7f7b0c07b0a0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7b18033000 tx=0x7f7b18004730 comp rx=0 tx=0).stop 2024-09-17T03:55:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 msgr2=0x7f7b2c15a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000d740 tx=0x7f7b2000dc00 comp rx=0 tx=0).stop 2024-09-17T03:55:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 shutdown_connections 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078b90 0x7f7b0c07b0a0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b2c0705a0 0x7f7b2c157600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b2c06f2e0 0x7f7b2c1570c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 --2- 172.21.15.129:0/4042673619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b2c06eed0 0x7f7b2c15a710 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 >> 172.21.15.129:0/4042673619 conn(0x7f7b2c06d410 msgr2=0x7f7b2c0ad600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 shutdown_connections 2024-09-17T03:55:25.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:25.940+0000 7f7b34400640 1 -- 172.21.15.129:0/4042673619 wait complete. 2024-09-17T03:55:25.962 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:55:25.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:25.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:26.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:55:26.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:26.050 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:55:26.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 -- 172.21.15.129:0/1962384859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700d0a50 msgr2=0x7f5c70162470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 --2- 172.21.15.129:0/1962384859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700d0a50 0x7f5c70162470 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f5c70071c60 tx=0x7f5c6802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 -- 172.21.15.129:0/1962384859 shutdown_connections 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 --2- 172.21.15.129:0/1962384859 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700d0a50 0x7f5c70162470 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 --2- 172.21.15.129:0/1962384859 >> v2:172.21.15.181:3300/0 conn(0x7f5c700ad8d0 0x7f5c7015e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 -- 172.21.15.129:0/1962384859 >> 172.21.15.129:0/1962384859 conn(0x7f5c7006cfc0 msgr2=0x7f5c7006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 -- 172.21.15.129:0/1962384859 shutdown_connections 2024-09-17T03:55:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.328+0000 7f5c75000640 1 -- 172.21.15.129:0/1962384859 wait complete. 2024-09-17T03:55:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.331+0000 7f5c75000640 1 Processor -- start 2024-09-17T03:55:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.331+0000 7f5c75000640 1 -- start start 2024-09-17T03:55:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 0x7f5c700afca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 0x7f5c700b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c70162b30 con 0x7f5c700ad8d0 2024-09-17T03:55:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5c701629b0 con 0x7f5c700cf910 2024-09-17T03:55:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c75000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5c70162cb0 con 0x7f5c70155380 2024-09-17T03:55:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:26.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59170/0 (socket says 172.21.15.129:59170) 2024-09-17T03:55:26.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.332+0000 7f5c6f400640 1 -- 172.21.15.129:0/213237895 learned_addr learned my addr 172.21.15.129:0/213237895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c66a00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 0x7f5c700afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6fe00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 0x7f5c700b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 -- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 msgr2=0x7f5c700b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 0x7f5c700b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 -- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 msgr2=0x7f5c700afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 0x7f5c700afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 -- 172.21.15.129:0/213237895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c70074fb0 con 0x7f5c700ad8d0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c66a00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 0x7f5c700afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6fe00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 0x7f5c700b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c6f400640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000e7c0 tx=0x7f5c6000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000ccc0 con 0x7f5c700ad8d0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c60010430 con 0x7f5c700ad8d0 2024-09-17T03:55:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c70155a60 con 0x7f5c700ad8d0 2024-09-17T03:55:26.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.333+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60010860 con 0x7f5c700ad8d0 2024-09-17T03:55:26.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.334+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c70156020 con 0x7f5c700ad8d0 2024-09-17T03:55:26.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.334+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c70163180 con 0x7f5c700ad8d0 2024-09-17T03:55:26.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.336+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5c60010a00 con 0x7f5c700ad8d0 2024-09-17T03:55:26.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.337+0000 7f5c67e00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 0x7f5c4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:26.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.337+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f5c6009b750 con 0x7f5c700ad8d0 2024-09-17T03:55:26.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.337+0000 7f5c66a00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 0x7f5c4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:26.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.338+0000 7f5c66a00640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 0x7f5c4c07b0a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5c700b4f40 tx=0x7f5c58009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:26.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.342+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5c60063b90 con 0x7f5c700ad8d0 2024-09-17T03:55:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.559+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c7006a140 con 0x7f5c700ad8d0 2024-09-17T03:55:26.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.560+0000 7f5c67e00640 1 -- 172.21.15.129:0/213237895 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5228 (secure 0 0 0) 0x7f5c60063490 con 0x7f5c700ad8d0 2024-09-17T03:55:26.560 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:26.560 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":80,"btime":"2024-09-17T03:55:13:284539+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/4006656424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":4006656424},{"type":"v1","addr":"172.21.15.129:6835","nonce":4006656424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6790,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/473985729","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":473985729},{"type":"v1","addr":"172.21.15.181:6836","nonce":473985729}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75},{"gid":6793,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/826515162","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":826515162},{"type":"v1","addr":"172.21.15.181:6837","nonce":826515162}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":75}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:12.783578+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7005},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7005":{"gid":7005,"name":"c","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/153768962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":153768962},{"type":"v1","addr":"172.21.15.129:6834","nonce":153768962}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7005,"qdb_cluster":[7005]},"id":8}]} 2024-09-17T03:55:26.560 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 80 2024-09-17T03:55:26.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 msgr2=0x7f5c4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:26.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 0x7f5c4c07b0a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5c700b4f40 tx=0x7f5c58009040 comp rx=0 tx=0).stop 2024-09-17T03:55:26.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 msgr2=0x7f5c70074a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000e7c0 tx=0x7f5c6000ec80 comp rx=0 tx=0).stop 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 shutdown_connections 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c4c078b90 0x7f5c4c07b0a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c70155380 0x7f5c700b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c700cf910 0x7f5c700afca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 --2- 172.21.15.129:0/213237895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c700ad8d0 0x7f5c70074a70 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 >> 172.21.15.129:0/213237895 conn(0x7f5c7006cfc0 msgr2=0x7f5c701620e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 shutdown_connections 2024-09-17T03:55:26.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:26.562+0000 7f5c75000640 1 -- 172.21.15.129:0/213237895 wait complete. 2024-09-17T03:55:26.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:55:26.830 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:55:26.833 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:55:26.833 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:55:26.833 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:26.834 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:55:26.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:26.834 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:55:26.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:26.835 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:55:26.835 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:55:26.859 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:55:26.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-17T03:55:26.860 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:55:26.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T03:55:26.861 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:55:26.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T03:55:26.864 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:55:26.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T03:55:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 -- 172.21.15.129:0/2208648959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 msgr2=0x7f8a580b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 --2- 172.21.15.129:0/2208648959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580b4140 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c005fd0 tx=0x7f8a4c02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 -- 172.21.15.129:0/2905393656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840b9af0 msgr2=0x7fac840d3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 --2- 172.21.15.129:0/2905393656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840b9af0 0x7fac840d3070 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fac840b7230 tx=0x7fac7002a560 comp rx=0 tx=0).stop 2024-09-17T03:55:27.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 -- 172.21.15.129:0/2208648959 shutdown_connections 2024-09-17T03:55:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 --2- 172.21.15.129:0/2208648959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806eed0 0x7f8a580ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 --2- 172.21.15.129:0/2208648959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580b4140 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7f8a60200640 1 -- 172.21.15.129:0/2208648959 >> 172.21.15.129:0/2208648959 conn(0x7f8a5806d410 msgr2=0x7f8a580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 -- 172.21.15.129:0/2905393656 shutdown_connections 2024-09-17T03:55:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 --2- 172.21.15.129:0/2905393656 >> v2:172.21.15.129:3300/0 conn(0x7fac840d4490 0x7fac840d48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 --2- 172.21.15.129:0/2905393656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840b9af0 0x7fac840d3070 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.137+0000 7fac89c00640 1 --2- 172.21.15.129:0/2905393656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840d1850 0x7fac840b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7fac89c00640 1 -- 172.21.15.129:0/2905393656 >> 172.21.15.129:0/2905393656 conn(0x7fac8408bff0 msgr2=0x7fac8409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f8a60200640 1 -- 172.21.15.129:0/2208648959 shutdown_connections 2024-09-17T03:55:27.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7fac89c00640 1 -- 172.21.15.129:0/2905393656 shutdown_connections 2024-09-17T03:55:27.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f8a60200640 1 -- 172.21.15.129:0/2208648959 wait complete. 2024-09-17T03:55:27.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7fac89c00640 1 -- 172.21.15.129:0/2905393656 wait complete. 2024-09-17T03:55:27.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 -- 172.21.15.129:0/190340711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c0d09d0 msgr2=0x7f484c15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 --2- 172.21.15.129:0/190340711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c0d09d0 0x7f484c15cbd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f4844005fd0 tx=0x7f484402a8b0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 -- 172.21.15.129:0/190340711 shutdown_connections 2024-09-17T03:55:27.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 --2- 172.21.15.129:0/190340711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c0d09d0 0x7f484c15cbd0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 -- 172.21.15.129:0/190340711 >> 172.21.15.129:0/190340711 conn(0x7f484c06cfc0 msgr2=0x7f484c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 -- 172.21.15.129:0/190340711 shutdown_connections 2024-09-17T03:55:27.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.138+0000 7f4851e00640 1 -- 172.21.15.129:0/190340711 wait complete. 2024-09-17T03:55:27.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.140+0000 7f8a60200640 1 Processor -- start 2024-09-17T03:55:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.140+0000 7fac89c00640 1 Processor -- start 2024-09-17T03:55:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.140+0000 7fac89c00640 1 -- start start 2024-09-17T03:55:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.140+0000 7f8a60200640 1 -- start start 2024-09-17T03:55:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.141+0000 7f4851e00640 1 Processor -- start 2024-09-17T03:55:27.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.141+0000 7f4851e00640 1 -- start start 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a5806eed0 0x7f8a58159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a58160200 con 0x7f8a580705a0 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8a581612e0 con 0x7f8a5806eed0 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a60200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8a580cd1c0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 0x7fac8415e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 0x7fac841618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 0x7fac840ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac84162a50 con 0x7fac840d4490 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac841628d0 con 0x7fac840d1850 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac89c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fac84162bd0 con 0x7fac840b9af0 2024-09-17T03:55:27.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a5ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac83400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 0x7fac841618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a5ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:52710/0 (socket says 172.21.15.129:52710) 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f8a5ce00640 1 -- 172.21.15.129:0/4180806850 learned_addr learned my addr 172.21.15.129:0/4180806850 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac83400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 0x7fac841618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:46256/0 (socket says 172.21.15.129:46256) 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac83400640 1 -- 172.21.15.129:0/3091999393 learned_addr learned my addr 172.21.15.129:0/3091999393 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:27.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7fac83e00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 0x7fac8415e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.142+0000 7f4851e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 0x7f484c0cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 0x7f484c15a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5d800640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a5806eed0 0x7f8a58159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f484c1f6850 0x7f484c1f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f484c15d4a0 con 0x7f484c160cf0 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f484c15d320 con 0x7f484c0cfb60 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f484c15d620 con 0x7f484c1f6850 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac89200640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 0x7fac840ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5e200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4850a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 0x7f484c0cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f484be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 0x7f484c15a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4850a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 0x7f484c0cb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:46260/0 (socket says 172.21.15.129:46260) 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4850a00640 1 -- 172.21.15.129:0/3090481337 learned_addr learned my addr 172.21.15.129:0/3090481337 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac83e00640 1 -- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 msgr2=0x7fac841618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac83e00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 0x7fac841618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac83e00640 1 -- 172.21.15.129:0/3091999393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 msgr2=0x7fac840ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac83e00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 0x7fac840ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac83e00640 1 -- 172.21.15.129:0/3091999393 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac841f47f0 con 0x7fac840b9af0 2024-09-17T03:55:27.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7fac89200640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 0x7fac840ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:27.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5ce00640 1 -- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a5806eed0 msgr2=0x7f8a58159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5ce00640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a5806eed0 0x7f8a58159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5ce00640 1 -- 172.21.15.129:0/4180806850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 msgr2=0x7f8a580ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5ce00640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f8a5ce00640 1 -- 172.21.15.129:0/4180806850 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a580aefd0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.143+0000 7f4851400640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f484c1f6850 0x7f484c1f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f8a5ce00640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8a440099b0 tx=0x7f8a44009e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7fac83e00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 0x7fac8415e920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac74009fd0 tx=0x7fac7400eae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 -- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f484c1f6850 msgr2=0x7f484c1f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f484c1f6850 0x7f484c1f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 -- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 msgr2=0x7f484c0cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 0x7f484c0cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 -- 172.21.15.129:0/3090481337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c1f71a0 con 0x7f484c160cf0 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44007c70 con 0x7f8a5806f2e0 2024-09-17T03:55:27.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a580af2b0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a4401e070 con 0x7f8a5806f2e0 2024-09-17T03:55:27.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74004220 con 0x7fac840b9af0 2024-09-17T03:55:27.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac740043c0 con 0x7fac840b9af0 2024-09-17T03:55:27.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7400ca70 con 0x7fac840b9af0 2024-09-17T03:55:27.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a44019650 con 0x7f8a5806f2e0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.144+0000 7f484be00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 0x7f484c15a270 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f483800c370 tx=0x7f483800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838011790 con 0x7f484c160cf0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4838017070 con 0x7f484c160cf0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4838011cc0 con 0x7f484c160cf0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484c0cbeb0 con 0x7f484c160cf0 2024-09-17T03:55:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484c15b010 con 0x7f484c160cf0 2024-09-17T03:55:27.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5815afb0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac841f4ad0 con 0x7fac840b9af0 2024-09-17T03:55:27.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.145+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac841f5090 con 0x7fac840b9af0 2024-09-17T03:55:27.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.146+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac840b3440 con 0x7fac840b9af0 2024-09-17T03:55:27.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.146+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a580aed40 con 0x7f8a5806f2e0 2024-09-17T03:55:27.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.147+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f484c0c7fe0 con 0x7f484c160cf0 2024-09-17T03:55:27.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.148+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8a440197f0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.149+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fac7403bd10 con 0x7fac840b9af0 2024-09-17T03:55:27.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.149+0000 7f8a55600640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 0x7f8a3807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.149+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f483801c460 con 0x7f484c160cf0 2024-09-17T03:55:27.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f8a4409f1f0 con 0x7f8a5806f2e0 2024-09-17T03:55:27.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7f4848c00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 0x7f482c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f483809ab20 con 0x7f484c160cf0 2024-09-17T03:55:27.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7f8a5d800640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 0x7f8a3807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7fac7be00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 0x7fac6407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7f4850a00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 0x7f482c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.150+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7fac7409b480 con 0x7fac840b9af0 2024-09-17T03:55:27.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.151+0000 7fac83400640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 0x7fac6407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.151+0000 7f8a5d800640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 0x7f8a3807b0a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c033000 tx=0x7f8a4c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.151+0000 7f4850a00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 0x7f482c07b0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4840007fd0 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.151+0000 7fac83400640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 0x7fac6407b0a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fac70009230 tx=0x7fac70004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.153+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fac740639a0 con 0x7fac840b9af0 2024-09-17T03:55:27.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.158+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8a44016070 con 0x7f8a5806f2e0 2024-09-17T03:55:27.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.159+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f484c0c7fe0 con 0x7f484c160cf0 2024-09-17T03:55:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 -- 172.21.15.129:0/4282498422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 msgr2=0x7f78500d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 --2- 172.21.15.129:0/4282498422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f78500d4540 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f78440098e0 tx=0x7f784402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 -- 172.21.15.129:0/4282498422 shutdown_connections 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 --2- 172.21.15.129:0/4282498422 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d4a80 0x7f78500d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 --2- 172.21.15.129:0/4282498422 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f78500d4540 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 -- 172.21.15.129:0/4282498422 >> 172.21.15.129:0/4282498422 conn(0x7f785009d8d0 msgr2=0x7f785008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 -- 172.21.15.129:0/4282498422 shutdown_connections 2024-09-17T03:55:27.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.201+0000 7f7856800640 1 -- 172.21.15.129:0/4282498422 wait complete. 2024-09-17T03:55:27.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.204+0000 7f7856800640 1 Processor -- start 2024-09-17T03:55:27.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.204+0000 7f7856800640 1 -- start start 2024-09-17T03:55:27.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f785014ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 0x7f78500a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78500d4a80 0x7f785015ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78500ba040 con 0x7f78500b3220 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f78500b9ec0 con 0x7f78500d4a80 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.205+0000 7f7856800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f78500d0ae0 con 0x7f78500d1500 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7855400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f785014ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7855400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f785014ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59236/0 (socket says 172.21.15.129:59236) 2024-09-17T03:55:27.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7855400640 1 -- 172.21.15.129:0/4005800504 learned_addr learned my addr 172.21.15.129:0/4005800504 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 0x7f78500a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7855e00640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78500d4a80 0x7f785015ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 -- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78500d4a80 msgr2=0x7f785015ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78500d4a80 0x7f785015ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 -- 172.21.15.129:0/4005800504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 msgr2=0x7f785014ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f785014ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.206+0000 7f7854a00640 1 -- 172.21.15.129:0/4005800504 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785014b1b0 con 0x7f78500d1500 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f7854a00640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 0x7f78500a1d70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f783c00d630 tx=0x7f783c00daf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c017070 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f783c004310 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785015d210 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f783c005260 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.207+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f785015d7d0 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.209+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78500b2b20 con 0x7f78500d1500 2024-09-17T03:55:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.210+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f783c005400 con 0x7f78500d1500 2024-09-17T03:55:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.212+0000 7f784d600640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 0x7f783007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.212+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(107..107 src has 1..107) ==== 5709+0+0 (secure 0 0 0) 0x7f783c09afd0 con 0x7f78500d1500 2024-09-17T03:55:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.212+0000 7f7855400640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 0x7f783007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.213+0000 7f7855400640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 0x7f783007b0a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7844000c00 tx=0x7f7844004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.216+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f783c020020 con 0x7f78500d1500 2024-09-17T03:55:27.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.428+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7fac840b4300 con 0x7fac840b9af0 2024-09-17T03:55:27.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.429+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7f484c15cbd0 con 0x7f484c160cf0 2024-09-17T03:55:27.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.429+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7f8a580ccc80 con 0x7f8a5806f2e0 2024-09-17T03:55:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.482+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7f785006a130 con 0x7f78500d1500 2024-09-17T03:55:27.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.483+0000 7fac7be00640 1 -- 172.21.15.129:0/3091999393 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6793 v81) ==== 95+0+0 (secure 0 0 0) 0x7fac740632a0 con 0x7fac840b9af0 2024-09-17T03:55:27.483 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6793 2024-09-17T03:55:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 msgr2=0x7fac6407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 0x7fac6407b0a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fac70009230 tx=0x7fac70004730 comp rx=0 tx=0).stop 2024-09-17T03:55:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 msgr2=0x7fac8415e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 0x7fac8415e920 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac74009fd0 tx=0x7fac7400eae0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 shutdown_connections 2024-09-17T03:55:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac64078b90 0x7fac6407b0a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac840d4490 0x7fac840ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac840d1850 0x7fac841618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 --2- 172.21.15.129:0/3091999393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac840b9af0 0x7fac8415e920 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.487+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 >> 172.21.15.129:0/3091999393 conn(0x7fac8408bff0 msgr2=0x7fac840b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.488+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 shutdown_connections 2024-09-17T03:55:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.488+0000 7fac89c00640 1 -- 172.21.15.129:0/3091999393 wait complete. 2024-09-17T03:55:27.494 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:55:27.494+0000 7f409c400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:55:27.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.496+0000 7f4848c00640 1 -- 172.21.15.129:0/3090481337 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6790 v82) ==== 95+0+0 (secure 0 0 0) 0x7f484c1f71a0 con 0x7f484c160cf0 2024-09-17T03:55:27.497 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6790 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.499+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 msgr2=0x7f482c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 0x7f482c07b0d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4840007fd0 tx=0x7f4840009040 comp rx=0 tx=0).stop 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 msgr2=0x7f484c15a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 0x7f484c15a270 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f483800c370 tx=0x7f483800c830 comp rx=0 tx=0).stop 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 shutdown_connections 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f482c078bc0 0x7f482c07b0d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f484c1f6850 0x7f484c1f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f484c160cf0 0x7f484c15a270 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 --2- 172.21.15.129:0/3090481337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f484c0cfb60 0x7f484c0cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 >> 172.21.15.129:0/3090481337 conn(0x7f484c06cfc0 msgr2=0x7f484c155620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 shutdown_connections 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.500+0000 7f4851e00640 1 -- 172.21.15.129:0/3090481337 wait complete. 2024-09-17T03:55:27.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.502+0000 7f784d600640 1 -- 172.21.15.129:0/4005800504 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7005 v82) ==== 95+0+0 (secure 0 0 0) 0x7f783c062d20 con 0x7f78500d1500 2024-09-17T03:55:27.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.502+0000 7f8a55600640 1 -- 172.21.15.129:0/4180806850 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6065 v82) ==== 95+0+0 (secure 0 0 0) 0x7f8a44067620 con 0x7f8a5806f2e0 2024-09-17T03:55:27.502 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 7005 2024-09-17T03:55:27.502 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6065 2024-09-17T03:55:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.504+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 msgr2=0x7f783007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 0x7f783007b0a0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7844000c00 tx=0x7f7844004730 comp rx=0 tx=0).stop 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 msgr2=0x7f78500a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 0x7f78500a1d70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f783c00d630 tx=0x7f783c00daf0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 msgr2=0x7f8a3807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 0x7f8a3807b0a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c033000 tx=0x7f8a4c004730 comp rx=0 tx=0).stop 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 msgr2=0x7f8a580c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8a440099b0 tx=0x7f8a44009e70 comp rx=0 tx=0).stop 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 shutdown_connections 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7830078b90 0x7f783007b0a0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78500d4a80 0x7f785015ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78500d1500 0x7f78500a1d70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 --2- 172.21.15.129:0/4005800504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78500b3220 0x7f785014ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 >> 172.21.15.129:0/4005800504 conn(0x7f785009d8d0 msgr2=0x7f78500b7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 shutdown_connections 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a38078b90 0x7f8a3807b0a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a580705a0 0x7f8a580ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a5806f2e0 0x7f8a580c9d30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 --2- 172.21.15.129:0/4180806850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a5806eed0 0x7f8a58159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 shutdown_connections 2024-09-17T03:55:27.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 >> 172.21.15.129:0/4180806850 conn(0x7f8a5806d410 msgr2=0x7f8a5815ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f7856800640 1 -- 172.21.15.129:0/4005800504 wait complete. 2024-09-17T03:55:27.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 shutdown_connections 2024-09-17T03:55:27.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.505+0000 7f8a60200640 1 -- 172.21.15.129:0/4180806850 wait complete. 2024-09-17T03:55:27.514 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:55:27.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:55:27.519 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:55:27.519 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:27.530 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:55:27.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:55:27.536 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:55:27.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:55:27.540 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:55:27.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:55:27.541 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:55:27.541 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:27.542 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:55:27.543 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:27.545 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:55:27.546 DEBUG:teuthology.parallel:result is None 2024-09-17T03:55:27.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:27.720 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:55:27.736 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:55:27.739 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:55:27.740 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:55:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 -- 172.21.15.129:0/1039934430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 msgr2=0x7faad80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 --2- 172.21.15.129:0/1039934430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad80ccc80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7faad80af320 tx=0x7faac402a560 comp rx=0 tx=0).stop 2024-09-17T03:55:27.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 -- 172.21.15.129:0/1039934430 shutdown_connections 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 --2- 172.21.15.129:0/1039934430 >> v2:172.21.15.181:3300/0 conn(0x7faad80cd1c0 0x7faad8070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 --2- 172.21.15.129:0/1039934430 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad80ccc80 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 --2- 172.21.15.129:0/1039934430 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad815da00 0x7faad80b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 -- 172.21.15.129:0/1039934430 >> 172.21.15.129:0/1039934430 conn(0x7faad806d410 msgr2=0x7faad80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 -- 172.21.15.129:0/1039934430 shutdown_connections 2024-09-17T03:55:27.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.776+0000 7faadde00640 1 -- 172.21.15.129:0/1039934430 wait complete. 2024-09-17T03:55:27.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.778+0000 7faadde00640 1 Processor -- start 2024-09-17T03:55:27.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.778+0000 7faadde00640 1 -- start start 2024-09-17T03:55:27.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 0x7faad815c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 0x7faad80c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faad80707c0 con 0x7faad80b46c0 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faad8070640 con 0x7faad815da00 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faadde00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faad8070940 con 0x7faad80cd1c0 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faad6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faad6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59284/0 (socket says 172.21.15.129:59284) 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faad6a00640 1 -- 172.21.15.129:0/4084852248 learned_addr learned my addr 172.21.15.129:0/4084852248 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:27.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faad7400640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 0x7faad80c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.780+0000 7faad6000640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 0x7faad815c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 -- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 msgr2=0x7faad815c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 0x7faad815c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 -- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 msgr2=0x7faad80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 0x7faad80c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:27.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 -- 172.21.15.129:0/4084852248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad8156ae0 con 0x7faad80b46c0 2024-09-17T03:55:27.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad7400640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 0x7faad80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:55:27.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6000640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 0x7faad815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:55:27.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faad6a00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7faac800d6d0 tx=0x7faac800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8017070 con 0x7faad80b46c0 2024-09-17T03:55:27.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faac8005330 con 0x7faad80b46c0 2024-09-17T03:55:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8009820 con 0x7faad80b46c0 2024-09-17T03:55:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad8156dc0 con 0x7faad80b46c0 2024-09-17T03:55:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.781+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad8157400 con 0x7faad80b46c0 2024-09-17T03:55:27.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.783+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad8070100 con 0x7faad80b46c0 2024-09-17T03:55:27.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.784+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faac8009bd0 con 0x7faad80b46c0 2024-09-17T03:55:27.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.785+0000 7faacf400640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 0x7faab807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:27.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.785+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7faac809afa0 con 0x7faad80b46c0 2024-09-17T03:55:27.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.785+0000 7faad6000640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 0x7faab807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:27.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.786+0000 7faad6000640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 0x7faab807b0a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faac4033000 tx=0x7faac4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:27.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:27.789+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faac8062dc0 con 0x7faad80b46c0 2024-09-17T03:55:28.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.007+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faad806a140 con 0x7faad80b46c0 2024-09-17T03:55:28.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.007+0000 7faacf400640 1 -- 172.21.15.129:0/4084852248 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+1912 (secure 0 0 0) 0x7faac80677f0 con 0x7faad80b46c0 2024-09-17T03:55:28.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:28.007 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":82,"btime":"2024-09-17T03:55:27:495471+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:27.495469+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-09-17T03:55:28.008 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 82 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.009+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 msgr2=0x7faab807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.009+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 0x7faab807b0a0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7faac4033000 tx=0x7faac4004730 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.009+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 msgr2=0x7faad8159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.009+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7faac800d6d0 tx=0x7faac800db90 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 shutdown_connections 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faab8078b90 0x7faab807b0a0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faad815da00 0x7faad80c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faad80cd1c0 0x7faad815c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 --2- 172.21.15.129:0/4084852248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faad80b46c0 0x7faad8159850 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:28.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 >> 172.21.15.129:0/4084852248 conn(0x7faad806d410 msgr2=0x7faad8074a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:28.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 shutdown_connections 2024-09-17T03:55:28.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:28.010+0000 7faadde00640 1 -- 172.21.15.129:0/4084852248 wait complete. 2024-09-17T03:55:28.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:55:09.589645+0000', 'modified': '2024-09-17T03:55:27.495469+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T03:55:28.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-09-17T03:55:29.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.670+0000 7feb93c00640 1 -- 172.21.15.129:0/1672125413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c0705a0 msgr2=0x7feb8c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.670+0000 7feb93c00640 1 --2- 172.21.15.129:0/1672125413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c0705a0 0x7feb8c0b4140 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7feb80009830 tx=0x7feb8002a510 comp rx=0 tx=0).stop 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 -- 172.21.15.129:0/1672125413 shutdown_connections 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 --2- 172.21.15.129:0/1672125413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c06eed0 0x7feb8c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 --2- 172.21.15.129:0/1672125413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c0705a0 0x7feb8c0b4140 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 -- 172.21.15.129:0/1672125413 >> 172.21.15.129:0/1672125413 conn(0x7feb8c06d410 msgr2=0x7feb8c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 -- 172.21.15.129:0/1672125413 shutdown_connections 2024-09-17T03:55:29.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.671+0000 7feb93c00640 1 -- 172.21.15.129:0/1672125413 wait complete. 2024-09-17T03:55:29.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.673+0000 7feb93c00640 1 Processor -- start 2024-09-17T03:55:29.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.673+0000 7feb93c00640 1 -- start start 2024-09-17T03:55:29.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.674+0000 7feb93c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:29.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.674+0000 7feb93c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 0x7feb8c0ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:29.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb93c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c0705a0 0x7feb8c0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:29.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb93c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb8c160200 con 0x7feb8c06eed0 2024-09-17T03:55:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb93c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feb8c1612e0 con 0x7feb8c06f2e0 2024-09-17T03:55:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb93c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7feb8c0cd1c0 con 0x7feb8c0705a0 2024-09-17T03:55:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59300/0 (socket says 172.21.15.129:59300) 2024-09-17T03:55:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 -- 172.21.15.129:0/2375735759 learned_addr learned my addr 172.21.15.129:0/2375735759 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c0705a0 0x7feb8c0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb8be00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 0x7feb8c0ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 -- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c0705a0 msgr2=0x7feb8c0cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c0705a0 0x7feb8c0cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 -- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 msgr2=0x7feb8c0ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 0x7feb8c0ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb91200640 1 -- 172.21.15.129:0/2375735759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8c0cb520 con 0x7feb8c06eed0 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.675+0000 7feb8be00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 0x7feb8c0ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb91200640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7feb80009950 tx=0x7feb80033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80005c90 con 0x7feb8c06eed0 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb80031e50 con 0x7feb8c06eed0 2024-09-17T03:55:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80005260 con 0x7feb8c06eed0 2024-09-17T03:55:29.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb8c0cb800 con 0x7feb8c06eed0 2024-09-17T03:55:29.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.676+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb8c0cbdc0 con 0x7feb8c06eed0 2024-09-17T03:55:29.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.677+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb8c0ccc80 con 0x7feb8c06eed0 2024-09-17T03:55:29.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.678+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feb80005400 con 0x7feb8c06eed0 2024-09-17T03:55:29.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.679+0000 7feb88c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 0x7feb6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:29.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.679+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7feb800bbf60 con 0x7feb8c06eed0 2024-09-17T03:55:29.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.680+0000 7feb8be00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 0x7feb6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:29.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.681+0000 7feb8be00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 0x7feb6c07b0a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feb8c070190 tx=0x7feb78005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:29.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.684+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feb8003f030 con 0x7feb8c06eed0 2024-09-17T03:55:29.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.900+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb8c06a140 con 0x7feb8c06eed0 2024-09-17T03:55:29.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.900+0000 7feb88c00640 1 -- 172.21.15.129:0/2375735759 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) ==== 94+0+5235 (secure 0 0 0) 0x7feb80084330 con 0x7feb8c06eed0 2024-09-17T03:55:29.901 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:29.901 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":85,"btime":"2024-09-17T03:55:29:511667+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:28.523695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-09-17T03:55:29.901 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 85 2024-09-17T03:55:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 msgr2=0x7feb6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 0x7feb6c07b0a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feb8c070190 tx=0x7feb78005f90 comp rx=0 tx=0).stop 2024-09-17T03:55:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 msgr2=0x7feb8c0c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7feb80009950 tx=0x7feb80033c20 comp rx=0 tx=0).stop 2024-09-17T03:55:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 shutdown_connections 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb6c078b90 0x7feb6c07b0a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb8c0705a0 0x7feb8c0cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb8c06f2e0 0x7feb8c0ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 --2- 172.21.15.129:0/2375735759 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb8c06eed0 0x7feb8c0c9ce0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 >> 172.21.15.129:0/2375735759 conn(0x7feb8c06d410 msgr2=0x7feb8c1609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 shutdown_connections 2024-09-17T03:55:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:29.903+0000 7feb93c00640 1 -- 172.21.15.129:0/2375735759 wait complete. 2024-09-17T03:55:30.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:55:09.589645+0000', 'modified': '2024-09-17T03:55:28.523695+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7257': {'gid': 7257, 'name': 'c', 'rank': 0, 'incarnation': 84, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.129:6835/1953968590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 1953968590}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1953968590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T03:55:30.329 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7257:up:reconnect 2024-09-17T03:55:31.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 -- 172.21.15.129:0/683104696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40ccc80 msgr2=0x7f62c40cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 --2- 172.21.15.129:0/683104696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40ccc80 0x7f62c40cd090 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f62c40c72d0 tx=0x7f62b002b1c0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 -- 172.21.15.129:0/683104696 shutdown_connections 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 --2- 172.21.15.129:0/683104696 >> v2:172.21.15.181:3300/0 conn(0x7f62c406f640 0x7f62c406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 --2- 172.21.15.129:0/683104696 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40ccc80 0x7f62c40cd090 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 -- 172.21.15.129:0/683104696 >> 172.21.15.129:0/683104696 conn(0x7f62c406d410 msgr2=0x7f62c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 -- 172.21.15.129:0/683104696 shutdown_connections 2024-09-17T03:55:31.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.566+0000 7f62cb200640 1 -- 172.21.15.129:0/683104696 wait complete. 2024-09-17T03:55:31.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.569+0000 7f62cb200640 1 Processor -- start 2024-09-17T03:55:31.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.569+0000 7f62cb200640 1 -- start start 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62c406f640 0x7f62c41f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62c40ccc80 0x7f62c40afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f62c40710e0 con 0x7f62c40af4c0 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f62c4070f60 con 0x7f62c406f640 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.570+0000 7f62cb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f62c406ff90 con 0x7f62c40ccc80 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59330/0 (socket says 172.21.15.129:59330) 2024-09-17T03:55:31.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 -- 172.21.15.129:0/292044064 learned_addr learned my addr 172.21.15.129:0/292044064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3e00640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62c406f640 0x7f62c41f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 -- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62c40ccc80 msgr2=0x7f62c40afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62c40ccc80 0x7f62c40afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 -- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62c406f640 msgr2=0x7f62c41f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62c406f640 0x7f62c41f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 -- 172.21.15.129:0/292044064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c41e4620 con 0x7f62c40af4c0 2024-09-17T03:55:31.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.571+0000 7f62c3400640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f62b0007440 tx=0x7f62b0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.572+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0042070 con 0x7f62c40af4c0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.572+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f62b0007c60 con 0x7f62c40af4c0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.572+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b0005c90 con 0x7f62c40af4c0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.572+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c41e4900 con 0x7f62c40af4c0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.572+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c41e4ec0 con 0x7f62c40af4c0 2024-09-17T03:55:31.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.573+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62c4070410 con 0x7f62c40af4c0 2024-09-17T03:55:31.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.574+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f62b0045070 con 0x7f62c40af4c0 2024-09-17T03:55:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.575+0000 7f62bbe00640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 0x7f62a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.575+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f62b00bafb0 con 0x7f62c40af4c0 2024-09-17T03:55:31.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.575+0000 7f62c3e00640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 0x7f62a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:31.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.576+0000 7f62c3e00640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 0x7f62a407b0a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f62b4007fd0 tx=0x7f62b4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.579+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f62b0083300 con 0x7f62c40af4c0 2024-09-17T03:55:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.799+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62c406a140 con 0x7f62c40af4c0 2024-09-17T03:55:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.799+0000 7f62bbe00640 1 -- 172.21.15.129:0/292044064 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f62b0082c00 con 0x7f62c40af4c0 2024-09-17T03:55:31.800 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:31.800 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:55:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:55:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 msgr2=0x7f62a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 0x7f62a407b0a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f62b4007fd0 tx=0x7f62b4009040 comp rx=0 tx=0).stop 2024-09-17T03:55:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 msgr2=0x7f62c415a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f62b0007440 tx=0x7f62b0003c10 comp rx=0 tx=0).stop 2024-09-17T03:55:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 shutdown_connections 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f62a4078b90 0x7f62a407b0a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f62c40ccc80 0x7f62c40afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f62c40af4c0 0x7f62c415a2b0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 --2- 172.21.15.129:0/292044064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f62c406f640 0x7f62c41f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 >> 172.21.15.129:0/292044064 conn(0x7f62c406d410 msgr2=0x7f62c40cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 shutdown_connections 2024-09-17T03:55:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:31.802+0000 7f62cb200640 1 -- 172.21.15.129:0/292044064 wait complete. 2024-09-17T03:55:32.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:55:09.589645+0000', 'modified': '2024-09-17T03:55:31.521116+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 111, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7257': {'gid': 7257, 'name': 'c', 'rank': 0, 'incarnation': 84, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.129:6835/1953968590', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 1953968590}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1953968590}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7257, 'qdb_cluster': [7257]} 2024-09-17T03:55:32.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:55:32.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:55:32.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 -- 172.21.15.129:0/4152517178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec8003c10 msgr2=0x7fdec8004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 --2- 172.21.15.129:0/4152517178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdec8003c10 0x7fdec8004060 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fded4009b50 tx=0x7fdec8005830 comp rx=0 tx=0).stop 2024-09-17T03:55:32.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee3400640 1 -- 172.21.15.129:0/4152517178 reap_dead start 2024-09-17T03:55:32.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 -- 172.21.15.129:0/4152517178 shutdown_connections 2024-09-17T03:55:32.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 -- 172.21.15.129:0/4152517178 >> 172.21.15.129:0/4152517178 conn(0x7fdee406d410 msgr2=0x7fdee40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 -- 172.21.15.129:0/4152517178 shutdown_connections 2024-09-17T03:55:32.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.450+0000 7fdee9a00640 1 -- 172.21.15.129:0/4152517178 wait complete. 2024-09-17T03:55:32.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.452+0000 7fdee9a00640 1 Processor -- start 2024-09-17T03:55:32.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.452+0000 7fdee9a00640 1 -- start start 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec8003c10 0x7fdee41f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 0x7fdee415cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdee40cbfa0 con 0x7fdee40ad8f0 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdee40cbe20 con 0x7fdec8003c10 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee9a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdee40cc120 con 0x7fdee40b16d0 2024-09-17T03:55:32.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59346/0 (socket says 172.21.15.129:59346) 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee3400640 1 -- 172.21.15.129:0/1962505616 learned_addr learned my addr 172.21.15.129:0/1962505616 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec8003c10 0x7fdee41f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.454+0000 7fdeda000640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 0x7fdee415cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 -- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 msgr2=0x7fdee415cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 0x7fdee415cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 -- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec8003c10 msgr2=0x7fdee41f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec8003c10 0x7fdee41f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 -- 172.21.15.129:0/1962505616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee41f5880 con 0x7fdee40ad8f0 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdeda000640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 0x7fdee415cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee3400640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fdee4160140 tx=0x7fdedc001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdedb400640 1 -- 172.21.15.129:0/1962505616 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc011870 con 0x7fdee40ad8f0 2024-09-17T03:55:32.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdedb400640 1 -- 172.21.15.129:0/1962505616 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdedc017070 con 0x7fdee40ad8f0 2024-09-17T03:55:32.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdedb400640 1 -- 172.21.15.129:0/1962505616 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdedc011da0 con 0x7fdee40ad8f0 2024-09-17T03:55:32.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee41f9680 con 0x7fdee40ad8f0 2024-09-17T03:55:32.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.455+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee41f9c40 con 0x7fdee40ad8f0 2024-09-17T03:55:32.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.459+0000 7fdedb400640 1 -- 172.21.15.129:0/1962505616 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdedc01c460 con 0x7fdee40ad8f0 2024-09-17T03:55:32.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.460+0000 7fdedb400640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 0x7fdec007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.460+0000 7fdef09ee740 1 Processor -- start 2024-09-17T03:55:32.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.460+0000 7fdedb400640 1 -- 172.21.15.129:0/1962505616 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fdedc09b550 con 0x7fdee40ad8f0 2024-09-17T03:55:32.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.460+0000 7fdef09ee740 1 -- start start 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.460+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 0x7fdec007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 0x7fdec007b0a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fded4035000 tx=0x7fded402d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c799fc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c799ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563c79956430 con 0x563c796033b0 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563c799465d0 con 0x563c7960b590 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.461+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563c79935a00 con 0x563c7962c270 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59354/0 (socket says 172.21.15.129:59354) 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c799fc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 -- 172.21.15.129:0/1616904435 learned_addr learned my addr 172.21.15.129:0/1616904435 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdee3400640 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c799ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 -- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 msgr2=0x563c799fc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c799fc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 -- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 msgr2=0x563c799ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c799ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 -- 172.21.15.129:0/1616904435 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c798fe520 con 0x563c796033b0 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.462+0000 7fdeda000640 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdecc006be0 tx=0x7fdecc006060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2948deca39127209 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.463+0000 7fded3400640 1 -- 172.21.15.129:0/1616904435 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc0039f0 con 0x563c796033b0 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.463+0000 7fded3400640 1 -- 172.21.15.129:0/1616904435 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdecc016030 con 0x563c796033b0 2024-09-17T03:55:32.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.463+0000 7fded3400640 1 -- 172.21.15.129:0/1616904435 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdecc003dd0 con 0x563c796033b0 2024-09-17T03:55:32.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.463+0000 7fdef09ee740 1 -- 172.21.15.129:0/1616904435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 msgr2=0x563c799fceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.463+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fdecc006be0 tx=0x7fdecc006060 comp rx=0 tx=0).stop 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 -- 172.21.15.129:0/1616904435 shutdown_connections 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c799ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799fceb0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1616904435 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c799fc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 -- 172.21.15.129:0/1616904435 >> 172.21.15.129:0/1616904435 conn(0x563c796203c0 msgr2=0x563c799d2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 -- 172.21.15.129:0/1616904435 shutdown_connections 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.464+0000 7fdef09ee740 1 -- 172.21.15.129:0/1616904435 wait complete. 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.465+0000 7fdef09ee740 1 -- start start 2024-09-17T03:55:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.465+0000 7fdef09ee740 1 Processor -- start 2024-09-17T03:55:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7960b590 0x563c79a0fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7962c270 0x563c79a109f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563c798dc330 con 0x563c796033b0 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563c79939810 con 0x563c7962c270 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563c796a6620 con 0x563c7960b590 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.466+0000 7fdef09ee740 10 client.0 _ll_register_callbacks cb 0x563c798e1420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59370/0 (socket says 172.21.15.129:59370) 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 -- 172.21.15.129:0/119276657 learned_addr learned my addr 172.21.15.129:0/119276657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee3400640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7962c270 0x563c79a109f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdeda000640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7960b590 0x563c79a0fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 -- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7960b590 msgr2=0x563c79a0fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7960b590 0x563c79a0fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 -- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7962c270 msgr2=0x563c79a109f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7962c270 0x563c79a109f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 -- 172.21.15.129:0/119276657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c79697850 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdee2a00640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fded4036040 tx=0x7fded40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d493bb390b85b404 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.467+0000 7fdec7e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4050070 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fded403ec50 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdecc016030 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fded405b050 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 10 client.7296 fetch_fsmap learned FSMap version 87 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) -- 0x563c7970e6b0 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdecc016030 con 0x563c796033b0 2024-09-17T03:55:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 87) ==== 3502+0+0 (secure 0 0 0) 0x7fded405e020 con 0x563c796033b0 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 10 client.7296 fetch_fsmap finished waiting for FSMap version 87 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 10 client.7296 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 20 client.7296 populate_metadata read hostname 'smithi129' 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c795eae40 0x563c799f4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 4 client.7296 mds_command: new command op to 7257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.468+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 --> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c79948520 con 0x563c795eae40 2024-09-17T03:55:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.469+0000 7fdeda000640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c795eae40 0x563c799f4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.469+0000 7fdeda000640 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c795eae40 0x563c799f4b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc005ad706bdc0f server_cookie=c7cb14faac739d56 in_seq=0 out_seq=0 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.469+0000 7fdec7e00640 10 client.7296 ms_handle_connect on v2:172.21.15.129:6834/1953968590 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdec7e00640 1 -- 172.21.15.129:0/119276657 <== mds.0 v2:172.21.15.129:6834/1953968590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x563c79948520 con 0x563c795eae40 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdec7e00640 10 client.7296 handle_command_reply: tid=0 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 client.7296 shutdown 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 msgr2=0x563c799f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fded4036040 tx=0x7fded40046c0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 shutdown_connections 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c795eae40 0x563c799f4b10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7962c270 0x563c79a109f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7960b590 0x563c79a0fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdec7e00640 0 client.7296 ms_handle_reset on v2:172.21.15.129:6834/1953968590 2024-09-17T03:55:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 --2- 172.21.15.129:0/119276657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c799f5fc0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 >> 172.21.15.129:0/119276657 conn(0x563c796203c0 msgr2=0x563c79a1ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.470+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 shutdown_connections 2024-09-17T03:55:32.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.471+0000 7fdef09ee740 1 -- 172.21.15.129:0/119276657 wait complete. 2024-09-17T03:55:32.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.471+0000 7fdef09ee740 20 client.7296 trim_cache size 0 max 16384 2024-09-17T03:55:32.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.489+0000 7fdef09ee740 1 Processor -- start 2024-09-17T03:55:32.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.490+0000 7fdef09ee740 1 -- start start 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a91d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a92270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4161900 con 0x563c796033b0 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4159540 con 0x563c7960b590 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdee41612b0 con 0x563c7962c270 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59384/0 (socket says 172.21.15.129:59384) 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 -- 172.21.15.129:0/42868669 learned_addr learned my addr 172.21.15.129:0/42868669 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 -- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 msgr2=0x563c79a92270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a92270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdeda000640 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a91d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.491+0000 7fdee2a00640 1 -- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 msgr2=0x563c79a91d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fdee2a00640 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a91d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fdee2a00640 1 -- 172.21.15.129:0/42868669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c799d1820 con 0x563c796033b0 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fdeda000640 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a91d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fdee2a00640 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fded4000c00 tx=0x7fded4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7adcc7a6fdfdf18b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fded2a00640 1 -- 172.21.15.129:0/42868669 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4050070 con 0x563c796033b0 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fded2a00640 1 -- 172.21.15.129:0/42868669 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded4044040 con 0x563c796033b0 2024-09-17T03:55:32.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.492+0000 7fded2a00640 1 -- 172.21.15.129:0/42868669 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4038040 con 0x563c796033b0 2024-09-17T03:55:32.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 -- 172.21.15.129:0/42868669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 msgr2=0x563c79a2a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fded4000c00 tx=0x7fded4005830 comp rx=0 tx=0).stop 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 -- 172.21.15.129:0/42868669 shutdown_connections 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a92270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a91d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 --2- 172.21.15.129:0/42868669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2a7f0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 -- 172.21.15.129:0/42868669 >> 172.21.15.129:0/42868669 conn(0x563c796203c0 msgr2=0x563c79a3acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 -- 172.21.15.129:0/42868669 shutdown_connections 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.493+0000 7fdef09ee740 1 -- 172.21.15.129:0/42868669 wait complete. 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.494+0000 7fdef09ee740 1 -- start start 2024-09-17T03:55:32.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.494+0000 7fdef09ee740 1 Processor -- start 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a97020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a2a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563c799f89c0 con 0x563c796033b0 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563c799f6f40 con 0x563c7960b590 2024-09-17T03:55:32.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563c799f29f0 con 0x563c7962c270 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.495+0000 7fdef09ee740 10 client.0 _ll_register_callbacks cb 0x7fded40099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59390/0 (socket says 172.21.15.129:59390) 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 -- 172.21.15.129:0/1971787918 learned_addr learned my addr 172.21.15.129:0/1971787918 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdeda000640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a97020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee3400640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a2a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 -- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 msgr2=0x563c79a2a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a2a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 -- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 msgr2=0x563c79a97020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a97020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 -- 172.21.15.129:0/1971787918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c799d3650 con 0x563c796033b0 2024-09-17T03:55:32.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdee2a00640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fded4009890 tx=0x7fded403eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91abe67d12a4b68 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdec7e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4063020 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.496+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fded4055d10 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fded4044040 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) ==== 24+0+0 (secure 0 0 0) 0x7fded406d050 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 10 client.7302 fetch_fsmap learned FSMap version 87 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) -- 0x563c79a2f1c0 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fded4044040 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 87) ==== 3502+0+0 (secure 0 0 0) 0x7fded4070020 con 0x563c796033b0 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 10 client.7302 fetch_fsmap finished waiting for FSMap version 87 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 10 client.7302 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:55:32.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 20 client.7302 populate_metadata read hostname 'smithi129' 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c79a28d40 0x563c79a29150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 4 client.7302 mds_command: new command op to 7257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 --> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] -- command(tid 0: {"prefix": "status"}) -- 0x563c799eea50 con 0x563c79a28d40 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.497+0000 7fdeda000640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c79a28d40 0x563c79a29150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdeda000640 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c79a28d40 0x563c79a29150 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d3ff8edf0e666e server_cookie=ae7c4b8f8a9b0626 in_seq=0 out_seq=0 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdec7e00640 10 client.7302 ms_handle_connect on v2:172.21.15.129:6834/1953968590 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdec7e00640 1 -- 172.21.15.129:0/1971787918 <== mds.0 v2:172.21.15.129:6834/1953968590 1 ==== command_reply(tid 0: 0 ) ==== 8+0+333 (crc 0 0 0) 0x563c799eea50 con 0x563c79a28d40 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdec7e00640 10 client.7302 handle_command_reply: tid=0 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 client.7302 shutdown 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 msgr2=0x563c79a2ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fded4009890 tx=0x7fded403eeb0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 shutdown_connections 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:6834/1953968590,v1:172.21.15.129:6835/1953968590] conn(0x563c79a28d40 0x563c79a29150 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x563c7962c270 0x563c79a2a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563c7960b590 0x563c79a97020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 --2- 172.21.15.129:0/1971787918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x563c796033b0 0x563c79a2ec80 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 >> 172.21.15.129:0/1971787918 conn(0x563c796203c0 msgr2=0x563c799d27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.498+0000 7fdec7e00640 0 client.7302 ms_handle_reset on v2:172.21.15.129:6834/1953968590 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 shutdown_connections 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdef09ee740 1 -- 172.21.15.129:0/1971787918 wait complete. 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdef09ee740 20 client.7302 trim_cache size 0 max 16384 2024-09-17T03:55:32.503 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 7257, 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 3.9853649409999998, 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 87, 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 111, 2024-09-17T03:55:32.504 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 111, 2024-09-17T03:55:32.505 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 4.7546199260000002 2024-09-17T03:55:32.505 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:55:32.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 msgr2=0x7fdec007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 0x7fdec007b0a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fded4035000 tx=0x7fded402d040 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 msgr2=0x7fdee40add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fdee4160140 tx=0x7fdedc001b80 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 shutdown_connections 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdec0078b90 0x7fdec007b0a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdee40ad8f0 0x7fdee40add00 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdee40b16d0 0x7fdee415cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 --2- 172.21.15.129:0/1962505616 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdec8003c10 0x7fdee41f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.499+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 >> 172.21.15.129:0/1962505616 conn(0x7fdee406d410 msgr2=0x7fdee40ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.500+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 shutdown_connections 2024-09-17T03:55:32.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.500+0000 7fdee9a00640 1 -- 172.21.15.129:0/1962505616 wait complete. 2024-09-17T03:55:32.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 -- 172.21.15.129:0/860961262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 msgr2=0x7fd600004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 --2- 172.21.15.129:0/860961262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd600004800 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fd6180ca870 tx=0x7fd600005830 comp rx=0 tx=0).stop 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61da00640 1 -- 172.21.15.129:0/860961262 reap_dead start 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 -- 172.21.15.129:0/860961262 shutdown_connections 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 -- 172.21.15.129:0/860961262 >> 172.21.15.129:0/860961262 conn(0x7fd61806d410 msgr2=0x7fd6180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 -- 172.21.15.129:0/860961262 shutdown_connections 2024-09-17T03:55:32.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.742+0000 7fd61fa00640 1 -- 172.21.15.129:0/860961262 wait complete. 2024-09-17T03:55:32.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.745+0000 7fd61fa00640 1 Processor -- start 2024-09-17T03:55:32.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.745+0000 7fd61fa00640 1 -- start start 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 0x7fd6181f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd61815b6b0 0x7fd61815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd618160260 con 0x7fd6000041e0 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd6181600e0 con 0x7fd61815b6b0 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.746+0000 7fd61fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd6181610d0 con 0x7fd618099920 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59410/0 (socket says 172.21.15.129:59410) 2024-09-17T03:55:32.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 -- 172.21.15.129:0/938176745 learned_addr learned my addr 172.21.15.129:0/938176745 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd617e00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 0x7fd6181f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61da00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd61815b6b0 0x7fd61815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 -- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 msgr2=0x7fd6181f8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 0x7fd6181f8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 -- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd61815b6b0 msgr2=0x7fd61815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd61815b6b0 0x7fd61815bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 -- 172.21.15.129:0/938176745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6181f9a80 con 0x7fd6000041e0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd617e00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 0x7fd6181f8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd61d000640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd60c004dc0 tx=0x7fd60c0017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.747+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c010890 con 0x7fd6000041e0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.748+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd60c016070 con 0x7fd6000041e0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.748+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd60c010da0 con 0x7fd6000041e0 2024-09-17T03:55:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.748+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6181f9d60 con 0x7fd6000041e0 2024-09-17T03:55:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.748+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6181f5fe0 con 0x7fd6000041e0 2024-09-17T03:55:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.749+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6180ccc80 con 0x7fd6000041e0 2024-09-17T03:55:32.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.750+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd60c01b460 con 0x7fd6000041e0 2024-09-17T03:55:32.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.751+0000 7fd614c00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 0x7fd5f807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:32.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.751+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fd60c09b7a0 con 0x7fd6000041e0 2024-09-17T03:55:32.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.751+0000 7fd617e00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 0x7fd5f807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:32.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.752+0000 7fd617e00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 0x7fd5f807b0a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd604035000 tx=0x7fd6040046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:32.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.755+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd60c063ae0 con 0x7fd6000041e0 2024-09-17T03:55:32.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.974+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd61806a140 con 0x7fd6000041e0 2024-09-17T03:55:32.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.975+0000 7fd614c00640 1 -- 172.21.15.129:0/938176745 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v87) ==== 53+0+73 (secure 0 0 0) 0x7fd60c0633e0 con 0x7fd6000041e0 2024-09-17T03:55:32.975 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.977+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 msgr2=0x7fd5f807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.977+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 0x7fd5f807b0a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd604035000 tx=0x7fd6040046c0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.977+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 msgr2=0x7fd6181f6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.977+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fd60c004dc0 tx=0x7fd60c0017a0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 shutdown_connections 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5f8078b90 0x7fd5f807b0a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd61815b6b0 0x7fd61815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd618099920 0x7fd6181f8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 --2- 172.21.15.129:0/938176745 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd6000041e0 0x7fd6181f6500 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 >> 172.21.15.129:0/938176745 conn(0x7fd61806d410 msgr2=0x7fd6180b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 shutdown_connections 2024-09-17T03:55:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:32.978+0000 7fd61fa00640 1 -- 172.21.15.129:0/938176745 wait complete. 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:55:33.003 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:33.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.004 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:55:33.039 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:55:33.040 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42359sec preferred_lft 42359sec 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:55:33.041 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:55:33.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.042 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:55:33.135 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:55:33.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.138 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:55:33.173 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:55:33.173 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:55:33.173 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:55:33.173 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:55:33.173 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:55:33.174 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:55:33.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:55:33.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:33.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.370 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:55:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:55:33.406 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:55:33.406 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:33.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:55:33.502 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:55:33.502 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:33.538 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:55:33.538 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:55:33.538 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:33.613 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:55:33.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:33.809 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:33.810 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:33.810 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:33.830 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:33.831 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:33.831 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:33.831 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:55:33.831 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:55:33.861 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:55:33.861 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:55:33.862 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:55:33.863 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:55:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:55:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:55:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:55:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:55:33.866 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:55:33.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:33.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:55:33.923 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:55:34.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:55:34.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:55:34.013 INFO:teuthology.orchestra.run.smithi129.stdout:client7045 2024-09-17T03:55:34.013 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:55:34.013 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:55:34.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:55:34.050 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/file_a) 2024-09-17T03:55:34.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:34.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 -- 172.21.15.129:0/3134236731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 msgr2=0x7fa06c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 --2- 172.21.15.129:0/3134236731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa06c004c10 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fa070001660 tx=0x7fa06c005830 comp rx=0 tx=0).stop 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa086800640 1 -- 172.21.15.129:0/3134236731 reap_dead start 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 -- 172.21.15.129:0/3134236731 shutdown_connections 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 -- 172.21.15.129:0/3134236731 >> 172.21.15.129:0/3134236731 conn(0x7fa08806d410 msgr2=0x7fa0880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 -- 172.21.15.129:0/3134236731 shutdown_connections 2024-09-17T03:55:34.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.394+0000 7fa08ce00640 1 -- 172.21.15.129:0/3134236731 wait complete. 2024-09-17T03:55:34.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.396+0000 7fa08ce00640 1 Processor -- start 2024-09-17T03:55:34.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.397+0000 7fa08ce00640 1 -- start start 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa08809d870 0x7fa088154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0881f44b0 0x7fa0880ad810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880b4a10 con 0x7fa06c004800 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0880b4890 con 0x7fa0881f44b0 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa08ce00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa088070510 con 0x7fa08809d870 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa085e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:34.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa085e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38282/0 (socket says 172.21.15.129:38282) 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.398+0000 7fa085e00640 1 -- 172.21.15.129:0/873401935 learned_addr learned my addr 172.21.15.129:0/873401935 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa086800640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0881f44b0 0x7fa0880ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa07d400640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa08809d870 0x7fa088154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 -- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa08809d870 msgr2=0x7fa088154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa08809d870 0x7fa088154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 -- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0881f44b0 msgr2=0x7fa0880ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0881f44b0 0x7fa0880ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 -- 172.21.15.129:0/873401935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0881f3a10 con 0x7fa06c004800 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa085e00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa078006680 tx=0x7fa0780066b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078011700 con 0x7fa06c004800 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa078016020 con 0x7fa06c004800 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08809de10 con 0x7fa06c004800 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.399+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa078011b40 con 0x7fa06c004800 2024-09-17T03:55:34.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.400+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0881f3f90 con 0x7fa06c004800 2024-09-17T03:55:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.401+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08806a760 con 0x7fa06c004800 2024-09-17T03:55:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.402+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa078011ce0 con 0x7fa06c004800 2024-09-17T03:55:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.403+0000 7fa07ea00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 0x7fa06407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.403+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fa07809b340 con 0x7fa06c004800 2024-09-17T03:55:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.404+0000 7fa07d400640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 0x7fa06407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:34.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.404+0000 7fa07d400640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 0x7fa06407b0a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa070032000 tx=0x7fa070001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:34.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.408+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa08806a760 con 0x7fa06c004800 2024-09-17T03:55:34.513 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:55:34.514+0000 7f611ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:55:34.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.626+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa088070a70 con 0x7fa06c004800 2024-09-17T03:55:34.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.627+0000 7fa07ea00640 1 -- 172.21.15.129:0/873401935 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7fa078020030 con 0x7fa06c004800 2024-09-17T03:55:34.627 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:34.627 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:55:34.628 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 msgr2=0x7fa06407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 0x7fa06407b0a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa070032000 tx=0x7fa070001190 comp rx=0 tx=0).stop 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 msgr2=0x7fa0881f7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa078006680 tx=0x7fa0780066b0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 shutdown_connections 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa064078b90 0x7fa06407b0a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0881f44b0 0x7fa0880ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa08809d870 0x7fa088154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 --2- 172.21.15.129:0/873401935 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa06c004800 0x7fa0881f7900 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:34.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 >> 172.21.15.129:0/873401935 conn(0x7fa08806d410 msgr2=0x7fa088155bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:34.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 shutdown_connections 2024-09-17T03:55:34.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:34.630+0000 7fa08ce00640 1 -- 172.21.15.129:0/873401935 wait complete. 2024-09-17T03:55:34.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T03:55:34.893 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":1}} 2024-09-17T03:55:34.962 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-09-17T03:55:34.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 -- 172.21.15.129:0/3877539524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe458070100 msgr2=0x7fe458070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 --2- 172.21.15.129:0/3877539524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe458070100 0x7fe458070510 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fe450005fd0 tx=0x7fe45002aa90 comp rx=0 tx=0).stop 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 -- 172.21.15.129:0/3877539524 shutdown_connections 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 --2- 172.21.15.129:0/3877539524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe458070100 0x7fe458070510 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 --2- 172.21.15.129:0/3877539524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 0x7fe4580cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 -- 172.21.15.129:0/3877539524 >> 172.21.15.129:0/3877539524 conn(0x7fe45806d410 msgr2=0x7fe4580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 -- 172.21.15.129:0/3877539524 shutdown_connections 2024-09-17T03:55:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.181+0000 7fe45ca00640 1 -- 172.21.15.129:0/3877539524 wait complete. 2024-09-17T03:55:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.184+0000 7fe45ca00640 1 Processor -- start 2024-09-17T03:55:35.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.184+0000 7fe45ca00640 1 -- start start 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe458070100 0x7fe45809ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 0x7fe4581f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe458070bd0 con 0x7fe45815da00 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe458070a50 con 0x7fe458070100 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe45ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe458070d50 con 0x7fe4580ccc80 2024-09-17T03:55:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.185+0000 7fe456800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38294/0 (socket says 172.21.15.129:38294) 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 -- 172.21.15.129:0/2338995841 learned_addr learned my addr 172.21.15.129:0/2338995841 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe455e00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe458070100 0x7fe45809ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe44d400640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 0x7fe4581f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 -- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 msgr2=0x7fe4581f6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 0x7fe4581f6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 -- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe458070100 msgr2=0x7fe45809ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe458070100 0x7fe45809ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 -- 172.21.15.129:0/2338995841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45809d410 con 0x7fe45815da00 2024-09-17T03:55:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.186+0000 7fe456800640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe450009bf0 tx=0x7fe450003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.187+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450044070 con 0x7fe45815da00 2024-09-17T03:55:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.187+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe450004250 con 0x7fe45815da00 2024-09-17T03:55:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.187+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450004660 con 0x7fe45815da00 2024-09-17T03:55:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.187+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4580c8f70 con 0x7fe45815da00 2024-09-17T03:55:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.187+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4580c9530 con 0x7fe45815da00 2024-09-17T03:55:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.188+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe458070510 con 0x7fe45815da00 2024-09-17T03:55:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.189+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe450047070 con 0x7fe45815da00 2024-09-17T03:55:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.190+0000 7fe44ea00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 0x7fe43407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:55:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.190+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fe4500bbea0 con 0x7fe45815da00 2024-09-17T03:55:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.190+0000 7fe455e00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 0x7fe43407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:55:35.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.191+0000 7fe455e00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 0x7fe43407b0a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4480015e0 tx=0x7fe448009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:55:35.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.194+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe450032030 con 0x7fe45815da00 2024-09-17T03:55:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.414+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe45806a140 con 0x7fe45815da00 2024-09-17T03:55:35.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.414+0000 7fe44ea00640 1 -- 172.21.15.129:0/2338995841 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7fe450084160 con 0x7fe45815da00 2024-09-17T03:55:35.414 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:55:35.414 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:55:35.415 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 msgr2=0x7fe43407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 0x7fe43407b0a0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe4480015e0 tx=0x7fe448009040 comp rx=0 tx=0).stop 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 msgr2=0x7fe4581f56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe450009bf0 tx=0x7fe450003ce0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 shutdown_connections 2024-09-17T03:55:35.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe434078b90 0x7fe43407b0a0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe45815da00 0x7fe4581f56a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4580ccc80 0x7fe4581f6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 --2- 172.21.15.129:0/2338995841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe458070100 0x7fe45809ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 >> 172.21.15.129:0/2338995841 conn(0x7fe45806d410 msgr2=0x7fe4580ceba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 shutdown_connections 2024-09-17T03:55:35.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:55:35.417+0000 7fe45ca00640 1 -- 172.21.15.129:0/2338995841 wait complete. 2024-09-17T03:55:35.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:55:35.678 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T03:55:35.678 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T03:55:46.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:55:46.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:56:05.679 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T03:56:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.830+0000 7fc8b1e0bf40 1 Processor -- start 2024-09-17T03:56:05.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.830+0000 7fc8b1e0bf40 1 -- start start 2024-09-17T03:56:05.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8900022f0 0x5619a8f0d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec69e0 0x5619a8ec6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619a8e15c60 con 0x5619a8ec7330 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5619a8e741d0 con 0x7fc8900022f0 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5619a8f9e550 con 0x5619a8ec69e0 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aa200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8900022f0 0x5619a8f0d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52782/0 (socket says 172.21.15.129:52782) 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 -- 172.21.15.129:0/2135211690 learned_addr learned my addr 172.21.15.129:0/2135211690 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec69e0 0x5619a8ec6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 -- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec69e0 msgr2=0x5619a8ec6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec69e0 0x5619a8ec6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 -- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8900022f0 msgr2=0x5619a8f0d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8900022f0 0x5619a8f0d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.832+0000 7fc8aac00640 1 -- 172.21.15.129:0/2135211690 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619a9028b90 con 0x5619a8ec7330 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.833+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x5619a8ecbd50 tx=0x7fc8a4010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a37598246d1073 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.833+0000 7fc8a3e00640 1 -- 172.21.15.129:0/2135211690 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40102f0 con 0x5619a8ec7330 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.833+0000 7fc8a3e00640 1 -- 172.21.15.129:0/2135211690 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8a4010e50 con 0x5619a8ec7330 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.833+0000 7fc8a3e00640 1 -- 172.21.15.129:0/2135211690 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a400bb90 con 0x5619a8ec7330 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2135211690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 msgr2=0x5619a9027840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x5619a8ecbd50 tx=0x7fc8a4010c00 comp rx=0 tx=0).stop 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2135211690 shutdown_connections 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5619a8ec7330 0x5619a9027840 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec69e0 0x5619a8ec6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2135211690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8900022f0 0x5619a8f0d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2135211690 >> 172.21.15.129:0/2135211690 conn(0x5619a8de6ac0 msgr2=0x5619a8f0ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2135211690 shutdown_connections 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.834+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2135211690 wait complete. 2024-09-17T03:56:05.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.835+0000 7fc8b1e0bf40 1 Processor -- start 2024-09-17T03:56:05.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.835+0000 7fc8b1e0bf40 1 -- start start 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619a8ec69e0 0x5619a9037dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec7330 0x5619a8f95fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5619a8f0e370 con 0x7fc8900022f0 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5619a902a530 con 0x5619a8ec69e0 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.836+0000 7fc8b1e0bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5619a8f0dc30 con 0x5619a8ec7330 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52796/0 (socket says 172.21.15.129:52796) 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 learned_addr learned my addr 172.21.15.129:0/2963442633 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619a8ec69e0 0x5619a9037dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec7330 0x5619a8f95fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec7330 msgr2=0x5619a8f95fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec7330 0x5619a8f95fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619a8ec69e0 msgr2=0x5619a9037dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5619a8ec69e0 0x5619a9037dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619a9034db0 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5619a8ec7330 0x5619a8f95fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.837+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x5619a8f9d700 tx=0x7fc8a400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.838+0000 7fc8a2a00640 1 -- 172.21.15.129:0/2963442633 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a4004020 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.838+0000 7fc8a2a00640 1 -- 172.21.15.129:0/2963442633 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8a40041c0 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.838+0000 7fc8a2a00640 1 -- 172.21.15.129:0/2963442633 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a401a970 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.838+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5619a9035070 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.838+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619a9035630 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.840+0000 7fc8a2a00640 1 -- 172.21.15.129:0/2963442633 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc8a401ab10 con 0x7fc8900022f0 2024-09-17T03:56:05.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.841+0000 7fc8a2a00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 0x7fc88c079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.841+0000 7fc8a2a00640 1 -- 172.21.15.129:0/2963442633 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fc8a409be60 con 0x7fc8900022f0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.841+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x5619a8f944b0 0x5619a8f948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.841+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.842+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 0x7fc88c079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.842+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x5619a8f944b0 0x5619a8f948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.842+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 0x7fc88c079ab0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b620 tx=0x7fc89c002fb0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.843+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x5619a8f944b0 0x5619a8f948c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.843+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(1 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89800f8e0 con 0x5619a8f944b0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.843+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fc89800d890 0x7fc89800c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.843+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.844+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fc89800d890 0x7fc89800c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.844+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fc89800d890 0x7fc89800c370 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.845+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(2 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.845+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fc89c023270 0x7fc89c06d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.845+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.845+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fc89c023270 0x7fc89c06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.845+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fc89c023270 0x7fc89c06d920 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.846+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(3 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x7fc89c023270 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.846+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fc89801f8d0 0x7fc89801e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.846+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.846+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fc89801f8d0 0x7fc89801e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.847+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fc89801f8d0 0x7fc89801e350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.847+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(4 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.847+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(5 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x5619a8f944b0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(6 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x7fc89c023270 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(7 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.848+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(8 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x5619a8f944b0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(9 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(10 [pgnls start_epoch 111 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x7fc89c023270 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.849+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.850+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(11 [pgnls start_epoch 111 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x7fc89c023270 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.850+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.850+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(12 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.850+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.851+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(13 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89801c040 con 0x5619a8f944b0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.851+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fc898022440 0x7fc898022850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.851+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.851+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fc898022440 0x7fc898022850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.852+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fc898022440 0x7fc898022850 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.852+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(14 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.852+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(15 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(16 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(17 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fc89c01fbf0 0x7fc89c07c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.853+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c01fbf0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.854+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fc89c01fbf0 0x7fc89c07c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.855+0000 7fc8a9800640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fc89c01fbf0 0x7fc89c07c400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.855+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(18 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c01fbf0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.855+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.855+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(19 [pgnls start_epoch 111 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c023270 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.855+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.856+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 5 ==== osd_op_reply(20 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.856+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 6 ==== osd_op_reply(21 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(22 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c023270 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(23 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.857+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.858+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(24 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.858+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fc89c008690 0x7fc89c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.858+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c008690 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.858+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fc89c008690 0x7fc89c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.859+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fc89c008690 0x7fc89c06c3a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.859+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(25 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89c008690 2024-09-17T03:56:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.859+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.860+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(26 [pgnls start_epoch 111 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.860+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c01fbf0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.860+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(27 [pgnls start_epoch 111 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c01fbf0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.860+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.861+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(28 [pgnls start_epoch 111 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.861+0000 7fc8aac00640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5619a8f0b5c0 0x7fc8a4062940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.861+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f0b5c0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.861+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5619a8f0b5c0 0x7fc8a4062940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.862+0000 7fc8aa200640 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5619a8f0b5c0 0x7fc8a4062940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.862+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(29 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x5619a8f0b5c0 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.862+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c008690 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.863+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(30 [pgnls start_epoch 111 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89c008690 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.863+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.863+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(31 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.863+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.864+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(32 [pgnls start_epoch 111 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.864+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.864+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(33 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.864+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(34 [pgnls start_epoch 111 out=49b] v107'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(35 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c023270 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(36 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.865+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.866+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(37 [pgnls start_epoch 111 out=49b] v107'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c023270 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.866+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.866+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(38 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.866+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c01fbf0 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.867+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(39 [pgnls start_epoch 111 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c01fbf0 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.867+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c01fbf0 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.868+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(40 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c01fbf0 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.868+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c008690 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.868+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(41 [pgnls start_epoch 111 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89c008690 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.868+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.869+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(42 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.869+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c01fbf0 2024-09-17T03:56:05.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.869+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(43 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c01fbf0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.869+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.870+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(44 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.870+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.870+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 7 ==== osd_op_reply(45 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.870+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.871+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 8 ==== osd_op_reply(46 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.871+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.871+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 9 ==== osd_op_reply(47 [pgnls start_epoch 111 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.871+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.872+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 10 ==== osd_op_reply(48 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.872+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.873+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 6 ==== osd_op_reply(49 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.873+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.873+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 7 ==== osd_op_reply(50 [pgnls start_epoch 111 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.873+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c023270 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.874+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(51 [pgnls start_epoch 111 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x7fc89c023270 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.874+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f944b0 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.875+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 <== osd.7 v2:172.21.15.181:6826/580738384 11 ==== osd_op_reply(52 [pgnls start_epoch 111 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc898030040 con 0x5619a8f944b0 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.875+0000 7fc8a9800640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.875+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(53 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.875+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.876+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 8 ==== osd_op_reply(54 [pgnls start_epoch 111 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.876+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.876+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 9 ==== osd_op_reply(55 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.876+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x5619a8f0b5c0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.876+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(56 [pgnls start_epoch 111 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x5619a8f0b5c0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.877+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.877+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 12 ==== osd_op_reply(57 [pgnls start_epoch 111 out=81b] v107'8 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.877+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.878+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 10 ==== osd_op_reply(58 [pgnls start_epoch 111 out=49b] v107'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.878+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c008690 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.878+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(59 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89c008690 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.878+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 13 ==== osd_op_reply(60 [pgnls start_epoch 111 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89800d890 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(61 [pgnls start_epoch 111 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc89c061670 con 0x7fc89800d890 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc898022440 2024-09-17T03:56:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 <== osd.4 v2:172.21.15.181:6802/2229307524 11 ==== osd_op_reply(62 [pgnls start_epoch 111 out=81b] v107'9 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7fc89c061670 con 0x7fc898022440 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.879+0000 7fc8aa200640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89c008690 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(63 [pgnls start_epoch 111 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89c008690 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 111 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e111) -- 0x5619a8f94e00 con 0x7fc89801f8d0 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 <== osd.0 v2:172.21.15.129:6824/1655624368 14 ==== osd_op_reply(64 [pgnls start_epoch 111 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fc8a402d070 con 0x7fc89801f8d0 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000000 2024-09-17T03:56:05.911 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000001 2024-09-17T03:56:05.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fc89801f8d0 msgr2=0x7fc89801e350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fc89801f8d0 0x7fc89801e350 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fc89800d890 msgr2=0x7fc89800c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fc89800d890 0x7fc89800c370 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5619a8f0b5c0 msgr2=0x7fc8a4062940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5619a8f0b5c0 0x7fc8a4062940 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fc89c023270 msgr2=0x7fc89c06d920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fc89c023270 0x7fc89c06d920 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fc898022440 msgr2=0x7fc898022850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fc898022440 0x7fc898022850 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fc89c01fbf0 msgr2=0x7fc89c07c400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fc89c01fbf0 0x7fc89c07c400 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fc89c008690 msgr2=0x7fc89c06c3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fc89c008690 0x7fc89c06c3a0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x5619a8f944b0 msgr2=0x5619a8f948c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x5619a8f944b0 0x5619a8f948c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 reap_dead start 2024-09-17T03:56:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.880+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 reap_dead start 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8aac00640 1 -- 172.21.15.129:0/2963442633 reap_dead start 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 msgr2=0x7fc88c079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 0x7fc88c079ab0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00b620 tx=0x7fc89c002fb0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 msgr2=0x5619a8f964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x5619a8f9d700 tx=0x7fc8a400bed0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 shutdown_connections 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc88c0775a0 0x7fc88c079ab0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 --2- 172.21.15.129:0/2963442633 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8900022f0 0x5619a8f964e0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 >> 172.21.15.129:0/2963442633 conn(0x5619a8de6ac0 msgr2=0x5619a8f0f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 shutdown_connections 2024-09-17T03:56:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:05.881+0000 7fc8b1e0bf40 1 -- 172.21.15.129:0/2963442633 wait complete. 2024-09-17T03:56:05.917 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-09-17T03:56:05.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:06.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 -- 172.21.15.129:0/1461100580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba80ccc80 msgr2=0x7f0ba80cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 --2- 172.21.15.129:0/1461100580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80cd090 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0005fd0 tx=0x7f0ba002aa10 comp rx=0 tx=0).stop 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 -- 172.21.15.129:0/1461100580 shutdown_connections 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 --2- 172.21.15.129:0/1461100580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80cd090 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 --2- 172.21.15.129:0/1461100580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba80cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 -- 172.21.15.129:0/1461100580 >> 172.21.15.129:0/1461100580 conn(0x7f0ba806d410 msgr2=0x7f0ba80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 -- 172.21.15.129:0/1461100580 shutdown_connections 2024-09-17T03:56:06.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.141+0000 7f0bae000640 1 -- 172.21.15.129:0/1461100580 wait complete. 2024-09-17T03:56:06.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.143+0000 7f0bae000640 1 Processor -- start 2024-09-17T03:56:06.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.143+0000 7f0bae000640 1 -- start start 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba815f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba8160ba0 con 0x7f0ba8099570 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba8160a20 con 0x7f0ba80ccc80 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0bae000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba8160d20 con 0x7f0ba80cf2b0 2024-09-17T03:56:06.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0ba7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0ba7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52818/0 (socket says 172.21.15.129:52818) 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0ba7400640 1 -- 172.21.15.129:0/1115343544 learned_addr learned my addr 172.21.15.129:0/1115343544 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0ba6a00640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.145+0000 7f0ba7e00640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 -- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 msgr2=0x7f0ba815f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba815f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 -- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ba80ccc80 msgr2=0x7f0ba80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 -- 172.21.15.129:0/1115343544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba81f47f0 con 0x7f0ba8099570 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7e00640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0ba7400640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c3a0 tx=0x7f0b9800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98017070 con 0x7f0ba8099570 2024-09-17T03:56:06.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b980053c0 con 0x7f0ba8099570 2024-09-17T03:56:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b98003850 con 0x7f0ba8099570 2024-09-17T03:56:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba809a470 con 0x7f0ba8099570 2024-09-17T03:56:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.146+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba809aa30 con 0x7f0ba8099570 2024-09-17T03:56:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.147+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba8161240 con 0x7f0ba8099570 2024-09-17T03:56:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.149+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0b9801a070 con 0x7f0ba8099570 2024-09-17T03:56:06.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.150+0000 7f0b9f400640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 0x7f0b8807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.150+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f0b9809a130 con 0x7f0ba8099570 2024-09-17T03:56:06.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.151+0000 7f0ba6a00640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 0x7f0b8807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.151+0000 7f0ba6a00640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 0x7f0b8807b0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0ba81f7f20 tx=0x7f0b94005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:06.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.155+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0b98061e20 con 0x7f0ba8099570 2024-09-17T03:56:06.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.373+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ba806a140 con 0x7f0ba8099570 2024-09-17T03:56:06.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.374+0000 7f0b9f400640 1 -- 172.21.15.129:0/1115343544 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f0b98066850 con 0x7f0ba8099570 2024-09-17T03:56:06.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:06.374 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 msgr2=0x7f0b8807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 0x7f0b8807b0d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0ba81f7f20 tx=0x7f0b94005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 msgr2=0x7f0ba81f42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0b9800c3a0 tx=0x7f0b9800c860 comp rx=0 tx=0).stop 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 shutdown_connections 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b88078bc0 0x7f0b8807b0d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ba80cf2b0 0x7f0ba815f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ba80ccc80 0x7f0ba80c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 --2- 172.21.15.129:0/1115343544 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ba8099570 0x7f0ba81f42b0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 >> 172.21.15.129:0/1115343544 conn(0x7f0ba806d410 msgr2=0x7f0ba815c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 shutdown_connections 2024-09-17T03:56:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.377+0000 7f0bae000640 1 -- 172.21.15.129:0/1115343544 wait complete. 2024-09-17T03:56:06.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:56:06.649 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T03:56:06.650 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T03:56:06.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:06.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 -- 172.21.15.129:0/1332465166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 msgr2=0x7feb700ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 --2- 172.21.15.129:0/1332465166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb700ccc80 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7feb5c009830 tx=0x7feb5c02a510 comp rx=0 tx=0).stop 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 -- 172.21.15.129:0/1332465166 shutdown_connections 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 --2- 172.21.15.129:0/1332465166 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb700ccc80 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 --2- 172.21.15.129:0/1332465166 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb700705a0 0x7feb700b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 -- 172.21.15.129:0/1332465166 >> 172.21.15.129:0/1332465166 conn(0x7feb7006d410 msgr2=0x7feb700b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 -- 172.21.15.129:0/1332465166 shutdown_connections 2024-09-17T03:56:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.869+0000 7feb77e00640 1 -- 172.21.15.129:0/1332465166 wait complete. 2024-09-17T03:56:06.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.871+0000 7feb77e00640 1 Processor -- start 2024-09-17T03:56:06.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.872+0000 7feb77e00640 1 -- start start 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb7006f2e0 0x7feb700742e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb700705a0 0x7feb701f8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb70160200 con 0x7feb7006eed0 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feb701612e0 con 0x7feb7006f2e0 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb77e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7feb700cd1c0 con 0x7feb700705a0 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb75400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb75400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52828/0 (socket says 172.21.15.129:52828) 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.873+0000 7feb75400640 1 -- 172.21.15.129:0/3822201303 learned_addr learned my addr 172.21.15.129:0/3822201303 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 -- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb700705a0 msgr2=0x7feb701f8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb700705a0 0x7feb701f8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 -- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb7006f2e0 msgr2=0x7feb700742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb7006f2e0 0x7feb700742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 -- 172.21.15.129:0/3822201303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb700b22b0 con 0x7feb7006eed0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb75400640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7feb6400ea20 tx=0x7feb6400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6400cde0 con 0x7feb7006eed0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb64011560 con 0x7feb7006eed0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb701f9fc0 con 0x7feb7006eed0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64011990 con 0x7feb7006eed0 2024-09-17T03:56:06.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.874+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb701f9ce0 con 0x7feb7006eed0 2024-09-17T03:56:06.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.876+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb700ccc80 con 0x7feb7006eed0 2024-09-17T03:56:06.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.877+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feb64011b30 con 0x7feb7006eed0 2024-09-17T03:56:06.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.878+0000 7feb6d600640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 0x7feb5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:06.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.878+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7feb6409ba80 con 0x7feb7006eed0 2024-09-17T03:56:06.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.878+0000 7feb74a00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 0x7feb5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:06.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.879+0000 7feb74a00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 0x7feb5007b0a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7feb5c034000 tx=0x7feb5c0046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:06.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:06.882+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feb6409de20 con 0x7feb7006eed0 2024-09-17T03:56:07.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.099+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb7006a140 con 0x7feb7006eed0 2024-09-17T03:56:07.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.100+0000 7feb6d600640 1 -- 172.21.15.129:0/3822201303 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7feb64016030 con 0x7feb7006eed0 2024-09-17T03:56:07.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:07.100 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:07.100 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:07.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 msgr2=0x7feb5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 0x7feb5007b0a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7feb5c034000 tx=0x7feb5c0046c0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 msgr2=0x7feb70157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7feb6400ea20 tx=0x7feb6400eee0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 shutdown_connections 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb50078b90 0x7feb5007b0a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb700705a0 0x7feb701f8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb7006f2e0 0x7feb700742e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 --2- 172.21.15.129:0/3822201303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb7006eed0 0x7feb70157560 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 >> 172.21.15.129:0/3822201303 conn(0x7feb7006d410 msgr2=0x7feb700ad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 shutdown_connections 2024-09-17T03:56:07.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.102+0000 7feb77e00640 1 -- 172.21.15.129:0/3822201303 wait complete. 2024-09-17T03:56:07.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:56:07.383 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:56:07.383 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:56:07.383 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T03:56:07.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.522+0000 7fc414126f40 1 Processor -- start 2024-09-17T03:56:07.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.523+0000 7fc414126f40 1 -- start start 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b902640 0x55727b98c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 0x55727b98f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55727b807c60 con 0x55727b973150 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55727b8661d0 con 0x55727b902640 2024-09-17T03:56:07.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc414126f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55727b98dcd0 con 0x55727b904610 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc40d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc40d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52840/0 (socket says 172.21.15.129:52840) 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc40d200640 1 -- 172.21.15.129:0/3610963284 learned_addr learned my addr 172.21.15.129:0/3610963284 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.524+0000 7fc40c800640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b902640 0x55727b98c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 -- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 msgr2=0x55727b98f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40be00640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 0x55727b98f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 0x55727b98f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 -- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b902640 msgr2=0x55727b98c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b902640 0x55727b98c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 -- 172.21.15.129:0/3610963284 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727b8f89e0 con 0x55727b973150 2024-09-17T03:56:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40be00640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 0x55727b98f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:07.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40d200640 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc4040015b0 tx=0x7fc404001a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1b195351cb1f35e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40aa00640 1 -- 172.21.15.129:0/3610963284 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404017070 con 0x55727b973150 2024-09-17T03:56:07.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40aa00640 1 -- 172.21.15.129:0/3610963284 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4040027a0 con 0x55727b973150 2024-09-17T03:56:07.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.525+0000 7fc40aa00640 1 -- 172.21.15.129:0/3610963284 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404002c30 con 0x55727b973150 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 -- 172.21.15.129:0/3610963284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 msgr2=0x55727b98f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fc4040015b0 tx=0x7fc404001a70 comp rx=0 tx=0).stop 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 -- 172.21.15.129:0/3610963284 shutdown_connections 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b973150 0x55727b98f790 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b904610 0x55727b98f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 --2- 172.21.15.129:0/3610963284 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b902640 0x55727b98c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 -- 172.21.15.129:0/3610963284 >> 172.21.15.129:0/3610963284 conn(0x55727b7d8ac0 msgr2=0x55727b8fb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:07.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 -- 172.21.15.129:0/3610963284 shutdown_connections 2024-09-17T03:56:07.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.526+0000 7fc414126f40 1 -- 172.21.15.129:0/3610963284 wait complete. 2024-09-17T03:56:07.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.527+0000 7fc414126f40 1 Processor -- start 2024-09-17T03:56:07.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.527+0000 7fc414126f40 1 -- start start 2024-09-17T03:56:07.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b902640 0x55727ba26550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 0x55727b973e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b973150 0x55727ba2a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55727b8fa9e0 con 0x55727b904610 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55727b8f8c60 con 0x55727b973150 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.528+0000 7fc414126f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55727b8b95c0 con 0x55727b902640 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 0x55727b973e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 0x55727b973e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52844/0 (socket says 172.21.15.129:52844) 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 -- 172.21.15.129:0/758093131 learned_addr learned my addr 172.21.15.129:0/758093131 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40d200640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b973150 0x55727ba2a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40c800640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b902640 0x55727ba26550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 -- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b902640 msgr2=0x55727ba26550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55727b902640 0x55727ba26550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 -- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b973150 msgr2=0x55727ba2a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727b973150 0x55727ba2a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.529+0000 7fc40be00640 1 -- 172.21.15.129:0/758093131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55727ba27480 con 0x55727b904610 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc40be00640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 0x55727b973e00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8025e30 tx=0x7fc3f804d7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc409600640 1 -- 172.21.15.129:0/758093131 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8056670 con 0x55727b904610 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc409600640 1 -- 172.21.15.129:0/758093131 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc3f8066070 con 0x55727b904610 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc409600640 1 -- 172.21.15.129:0/758093131 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8056b80 con 0x55727b904610 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55727ba26ec0 con 0x55727b904610 2024-09-17T03:56:07.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.530+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55727ba29bc0 con 0x55727b904610 2024-09-17T03:56:07.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.532+0000 7fc409600640 1 -- 172.21.15.129:0/758093131 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc3f806a410 con 0x55727b904610 2024-09-17T03:56:07.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.533+0000 7fc409600640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec0775a0 0x7fc3ec079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.533+0000 7fc409600640 1 -- 172.21.15.129:0/758093131 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7fc3f80ddff0 con 0x55727b904610 2024-09-17T03:56:07.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.533+0000 7fc414126f40 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55727b961860 0x55727b98f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.533+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111) -- 0x55727b965560 con 0x55727b961860 2024-09-17T03:56:07.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.533+0000 7fc40c800640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec0775a0 0x7fc3ec079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.534+0000 7fc40d200640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55727b961860 0x55727b98f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.534+0000 7fc40d200640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55727b961860 0x55727b98f760 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.534+0000 7fc40c800640 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec0775a0 0x7fc3ec079ab0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc40000c6c0 tx=0x7fc400004a20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc40d200640 1 -- 172.21.15.129:0/758093131 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (99) out=99b] v0'0 uv9 ondisk = 0) ==== 164+0+99 (crc 0 0 0) 0x7fc40400f070 con 0x55727b961860 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55727b961860 msgr2=0x55727b98f760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55727b961860 0x55727b98f760 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec0775a0 msgr2=0x7fc3ec079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec0775a0 0x7fc3ec079ab0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc40000c6c0 tx=0x7fc400004a20 comp rx=0 tx=0).stop 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 msgr2=0x55727b973e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 --2- 172.21.15.129:0/758093131 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55727b904610 0x55727b973e00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8025e30 tx=0x7fc3f804d7d0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc40d200640 1 -- 172.21.15.129:0/758093131 reap_dead start 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 shutdown_connections 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 >> 172.21.15.129:0/758093131 conn(0x55727b7d8ac0 msgr2=0x55727b98c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 shutdown_connections 2024-09-17T03:56:07.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.535+0000 7fc414126f40 1 -- 172.21.15.129:0/758093131 wait complete. 2024-09-17T03:56:07.549 INFO:teuthology.orchestra.run.smithi129.stdout:] 10000000001 stray0 2024-09-17T03:56:07.549 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1536, 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "10000000001", 2024-09-17T03:56:07.690 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 13 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 256, 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "stray0", 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 5 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 17, 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:56:07.691 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:07.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:07.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.925+0000 7f059ca00640 1 -- 172.21.15.129:0/2537667794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f057c004800 msgr2=0x7f057c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.925+0000 7f059ca00640 1 --2- 172.21.15.129:0/2537667794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f057c004800 0x7f057c004c10 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0584009a10 tx=0x7f057c005830 comp rx=0 tx=0).stop 2024-09-17T03:56:07.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.925+0000 7f0596800640 1 -- 172.21.15.129:0/2537667794 reap_dead start 2024-09-17T03:56:07.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.925+0000 7f059ca00640 1 -- 172.21.15.129:0/2537667794 shutdown_connections 2024-09-17T03:56:07.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.925+0000 7f059ca00640 1 -- 172.21.15.129:0/2537667794 >> 172.21.15.129:0/2537667794 conn(0x7f059806d410 msgr2=0x7f05980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:07.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.926+0000 7f059ca00640 1 -- 172.21.15.129:0/2537667794 shutdown_connections 2024-09-17T03:56:07.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.926+0000 7f059ca00640 1 -- 172.21.15.129:0/2537667794 wait complete. 2024-09-17T03:56:07.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.928+0000 7f059ca00640 1 Processor -- start 2024-09-17T03:56:07.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.928+0000 7f059ca00640 1 -- start start 2024-09-17T03:56:07.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 0x7f05980c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f059815b000 0x7f059815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05980ccb40 con 0x7f05980c9cf0 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05980cc9c0 con 0x7f057c004800 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.929+0000 7f059ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05980b3d70 con 0x7f059815b000 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52858/0 (socket says 172.21.15.129:52858) 2024-09-17T03:56:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 -- 172.21.15.129:0/2403634048 learned_addr learned my addr 172.21.15.129:0/2403634048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0595e00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 0x7f05980c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0595400640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f059815b000 0x7f059815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 -- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f059815b000 msgr2=0x7f059815ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f059815b000 0x7f059815ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 -- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 msgr2=0x7f05980c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 0x7f05980c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 -- 172.21.15.129:0/2403634048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05981f5df0 con 0x7f05980c9cf0 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0595e00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 0x7f05980c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.930+0000 7f0596800640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0598160420 tx=0x7f0590006430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.931+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059000f960 con 0x7f05980c9cf0 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.931+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0590018070 con 0x7f05980c9cf0 2024-09-17T03:56:07.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.931+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059000fd80 con 0x7f05980c9cf0 2024-09-17T03:56:07.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.931+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05981f5550 con 0x7f05980c9cf0 2024-09-17T03:56:07.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.931+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05981f5b10 con 0x7f05980c9cf0 2024-09-17T03:56:07.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.932+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05980706a0 con 0x7f05980c9cf0 2024-09-17T03:56:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.933+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f059001d460 con 0x7f05980c9cf0 2024-09-17T03:56:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.934+0000 7f058e000640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 0x7f057807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.934+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f059009b480 con 0x7f05980c9cf0 2024-09-17T03:56:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.934+0000 7f0595e00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 0x7f057807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.935+0000 7f0595e00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 0x7f057807b0a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0588006bf0 tx=0x7f0588006720 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:07.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:07.938+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f05900638e0 con 0x7f05980c9cf0 2024-09-17T03:56:08.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.154+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05981f52e0 con 0x7f05980c9cf0 2024-09-17T03:56:08.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.155+0000 7f058e000640 1 -- 172.21.15.129:0/2403634048 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f05900631e0 con 0x7f05980c9cf0 2024-09-17T03:56:08.155 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:08.156 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:08.156 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 msgr2=0x7f057807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 0x7f057807b0a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0588006bf0 tx=0x7f0588006720 comp rx=0 tx=0).stop 2024-09-17T03:56:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 msgr2=0x7f05980ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f0598160420 tx=0x7f0590006430 comp rx=0 tx=0).stop 2024-09-17T03:56:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 shutdown_connections 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0578078b90 0x7f057807b0a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05980c9cf0 0x7f05980ca100 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f059815b000 0x7f059815ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 --2- 172.21.15.129:0/2403634048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f057c004800 0x7f05980c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 >> 172.21.15.129:0/2403634048 conn(0x7f059806d410 msgr2=0x7f05980b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:08.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 shutdown_connections 2024-09-17T03:56:08.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.157+0000 7f059ca00640 1 -- 172.21.15.129:0/2403634048 wait complete. 2024-09-17T03:56:08.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T03:56:08.420 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-09-17T03:56:08.420 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-09-17T03:56:08.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:08.420 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv dir_2/file_b dir_2/file_c) 2024-09-17T03:56:08.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 -- 172.21.15.129:0/3650640284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf0099570 msgr2=0x7facf006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 --2- 172.21.15.129:0/3650640284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf0099570 0x7facf006eed0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7facf00b3450 tx=0x7face402a4d0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 -- 172.21.15.129:0/3650640284 shutdown_connections 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 --2- 172.21.15.129:0/3650640284 >> v2:172.21.15.181:3300/0 conn(0x7facf00cf9c0 0x7facf00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 --2- 172.21.15.129:0/3650640284 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf0099570 0x7facf006eed0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 -- 172.21.15.129:0/3650640284 >> 172.21.15.129:0/3650640284 conn(0x7facf006d410 msgr2=0x7facf00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 -- 172.21.15.129:0/3650640284 shutdown_connections 2024-09-17T03:56:08.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.698+0000 7facf8400640 1 -- 172.21.15.129:0/3650640284 wait complete. 2024-09-17T03:56:08.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.701+0000 7facf8400640 1 Processor -- start 2024-09-17T03:56:08.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.701+0000 7facf8400640 1 -- start start 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf006f410 0x7facf00c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf0099570 0x7facf00743a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7facf0160ba0 con 0x7facf00cf9c0 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7facf0160a20 con 0x7facf0099570 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf8400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7facf0160d20 con 0x7facf006f410 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.702+0000 7facf6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf5a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf006f410 0x7facf00c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf0099570 0x7facf00743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52874/0 (socket says 172.21.15.129:52874) 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 -- 172.21.15.129:0/3598984548 learned_addr learned my addr 172.21.15.129:0/3598984548 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 -- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf006f410 msgr2=0x7facf00c9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf006f410 0x7facf00c9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 -- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf0099570 msgr2=0x7facf00743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf0099570 0x7facf00743a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 -- 172.21.15.129:0/3598984548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf015d020 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.703+0000 7facf6400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7face800b9e0 tx=0x7face800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.704+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8019070 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.704+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7face800cbd0 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.704+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8014440 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.704+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf015d2c0 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.704+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf00748e0 con 0x7facf00cf9c0 2024-09-17T03:56:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.705+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf01612c0 con 0x7facf00cf9c0 2024-09-17T03:56:08.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.706+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7face801c070 con 0x7facf00cf9c0 2024-09-17T03:56:08.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.707+0000 7faced600640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:08.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.707+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7face809aa70 con 0x7facf00cf9c0 2024-09-17T03:56:08.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.707+0000 7facf5a00640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:08.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.708+0000 7facf5a00640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7face4033000 tx=0x7face4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:08.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.711+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7face8062760 con 0x7facf00cf9c0 2024-09-17T03:56:08.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.930+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facf006a140 con 0x7facf00cf9c0 2024-09-17T03:56:08.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.931+0000 7faced600640 1 -- 172.21.15.129:0/3598984548 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7face8067190 con 0x7facf00cf9c0 2024-09-17T03:56:08.931 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:08.931 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:08.931 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:08.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 msgr2=0x7facd007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7face4033000 tx=0x7face4004730 comp rx=0 tx=0).stop 2024-09-17T03:56:08.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 msgr2=0x7facf015ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7face800b9e0 tx=0x7face800bea0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 shutdown_connections 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf00cf9c0 0x7facf015ea40 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf0099570 0x7facf00743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 --2- 172.21.15.129:0/3598984548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf006f410 0x7facf00c9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 >> 172.21.15.129:0/3598984548 conn(0x7facf006d410 msgr2=0x7facf00d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 shutdown_connections 2024-09-17T03:56:08.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:08.933+0000 7facf8400640 1 -- 172.21.15.129:0/3598984548 wait complete. 2024-09-17T03:56:08.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T03:56:09.194 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T03:56:09.195 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T03:56:09.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:09.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 -- 172.21.15.129:0/55550834 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15da00 msgr2=0x7f132c0b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:09.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 --2- 172.21.15.129:0/55550834 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15da00 0x7f132c0b4180 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f132c0ce800 tx=0x7f132002a510 comp rx=0 tx=0).stop 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 -- 172.21.15.129:0/55550834 shutdown_connections 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 --2- 172.21.15.129:0/55550834 >> v2:172.21.15.181:3300/0 conn(0x7f132c0cd390 0x7f132c070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 --2- 172.21.15.129:0/55550834 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15da00 0x7f132c0b4180 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 -- 172.21.15.129:0/55550834 >> 172.21.15.129:0/55550834 conn(0x7f132c06d410 msgr2=0x7f132c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 -- 172.21.15.129:0/55550834 shutdown_connections 2024-09-17T03:56:09.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.414+0000 7f1333600640 1 -- 172.21.15.129:0/55550834 wait complete. 2024-09-17T03:56:09.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.416+0000 7f1333600640 1 Processor -- start 2024-09-17T03:56:09.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.416+0000 7f1333600640 1 -- start start 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 0x7f132c1f6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15a470 0x7f132c1ec0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0709d0 con 0x7f132c0b46c0 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f132c070850 con 0x7f132c15da00 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1333600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f132c070b50 con 0x7f132c15a470 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1330c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:09.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1330c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52882/0 (socket says 172.21.15.129:52882) 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1330c00640 1 -- 172.21.15.129:0/2291724732 learned_addr learned my addr 172.21.15.129:0/2291724732 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f132be00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 0x7f132c1f6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.418+0000 7f1331600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15a470 0x7f132c1ec0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 -- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15a470 msgr2=0x7f132c1ec0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15a470 0x7f132c1ec0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 -- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 msgr2=0x7f132c1f6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 0x7f132c1f6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 -- 172.21.15.129:0/2291724732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c1f6740 con 0x7f132c0b46c0 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f132be00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 0x7f132c1f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1330c00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f132002ab00 tx=0x7f1320033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:09.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320031d60 con 0x7f132c0b46c0 2024-09-17T03:56:09.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1320033e10 con 0x7f132c0b46c0 2024-09-17T03:56:09.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1320005260 con 0x7f132c0b46c0 2024-09-17T03:56:09.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c1f9750 con 0x7f132c0b46c0 2024-09-17T03:56:09.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.419+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1f5a00 con 0x7f132c0b46c0 2024-09-17T03:56:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.421+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c070100 con 0x7f132c0b46c0 2024-09-17T03:56:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.422+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1320005400 con 0x7f132c0b46c0 2024-09-17T03:56:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.423+0000 7f1328c00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 0x7f130c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.423+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f13200bbf30 con 0x7f132c0b46c0 2024-09-17T03:56:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.423+0000 7f132be00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 0x7f130c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:09.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.424+0000 7f132be00640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 0x7f130c07b0a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f132c0b1e00 tx=0x7f1318009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:09.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.428+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f132003f030 con 0x7f132c0b46c0 2024-09-17T03:56:09.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.646+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f132c06a140 con 0x7f132c0b46c0 2024-09-17T03:56:09.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.646+0000 7f1328c00640 1 -- 172.21.15.129:0/2291724732 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f1320083c30 con 0x7f132c0b46c0 2024-09-17T03:56:09.646 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:09.647 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:09.647 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 msgr2=0x7f130c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 0x7f130c07b0a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f132c0b1e00 tx=0x7f1318009040 comp rx=0 tx=0).stop 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 msgr2=0x7f132c1f8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f132002ab00 tx=0x7f1320033c20 comp rx=0 tx=0).stop 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 shutdown_connections 2024-09-17T03:56:09.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f130c078b90 0x7f130c07b0a0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f132c15a470 0x7f132c1ec0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f132c15da00 0x7f132c1f6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 --2- 172.21.15.129:0/2291724732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f132c0b46c0 0x7f132c1f8e50 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 >> 172.21.15.129:0/2291724732 conn(0x7f132c06d410 msgr2=0x7f132c0ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 shutdown_connections 2024-09-17T03:56:09.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:09.649+0000 7f1333600640 1 -- 172.21.15.129:0/2291724732 wait complete. 2024-09-17T03:56:09.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T03:56:09.911 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-09-17T03:56:09.911 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-09-17T03:56:09.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.132+0000 7f061d000640 1 -- 172.21.15.129:0/3553397126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc004d40 msgr2=0x7f05fc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.132+0000 7f061d000640 1 --2- 172.21.15.129:0/3553397126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc004d40 0x7f05fc005150 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f0610005fd0 tx=0x7f05fc005830 comp rx=0 tx=0).stop 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.132+0000 7f0616800640 1 -- 172.21.15.129:0/3553397126 reap_dead start 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.132+0000 7f061d000640 1 -- 172.21.15.129:0/3553397126 shutdown_connections 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.132+0000 7f061d000640 1 -- 172.21.15.129:0/3553397126 >> 172.21.15.129:0/3553397126 conn(0x7f061806d410 msgr2=0x7f06180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.133+0000 7f061d000640 1 -- 172.21.15.129:0/3553397126 shutdown_connections 2024-09-17T03:56:10.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.133+0000 7f061d000640 1 -- 172.21.15.129:0/3553397126 wait complete. 2024-09-17T03:56:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.135+0000 7f061d000640 1 Processor -- start 2024-09-17T03:56:10.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.135+0000 7f061d000640 1 -- start start 2024-09-17T03:56:10.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc004d40 0x7f06181567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06181f6430 0x7f061815f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06180cd3c0 con 0x7f06180ca8c0 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f06180cd240 con 0x7f06181f6430 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.136+0000 7f061d000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f06180cd540 con 0x7f05fc004d40 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52886/0 (socket says 172.21.15.129:52886) 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 -- 172.21.15.129:0/2696624836 learned_addr learned my addr 172.21.15.129:0/2696624836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0615e00640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc004d40 0x7f06181567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f060fe00640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06181f6430 0x7f061815f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 -- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc004d40 msgr2=0x7f06181567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc004d40 0x7f06181567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 -- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06181f6430 msgr2=0x7f061815f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06181f6430 0x7f061815f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 -- 172.21.15.129:0/2696624836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061815fe20 con 0x7f06180ca8c0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.137+0000 7f0616800640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f061002aea0 tx=0x7f0610033b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.138+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610042070 con 0x7f06180ca8c0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.138+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0610003970 con 0x7f06180ca8c0 2024-09-17T03:56:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.138+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0618155710 con 0x7f06180ca8c0 2024-09-17T03:56:10.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.138+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610038040 con 0x7f06180ca8c0 2024-09-17T03:56:10.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.138+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06181559f0 con 0x7f06180ca8c0 2024-09-17T03:56:10.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.139+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061806a140 con 0x7f06180ca8c0 2024-09-17T03:56:10.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.140+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0610004030 con 0x7f06180ca8c0 2024-09-17T03:56:10.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.141+0000 7f060e000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 0x7f05f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.141+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f06100bb200 con 0x7f06180ca8c0 2024-09-17T03:56:10.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.141+0000 7f0615e00640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 0x7f05f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.142+0000 7f0615e00640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 0x7f05f407b0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0608009d70 tx=0x7f0608005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.145+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0610082ef0 con 0x7f06180ca8c0 2024-09-17T03:56:10.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.363+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06180b29f0 con 0x7f06180ca8c0 2024-09-17T03:56:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.364+0000 7f060e000640 1 -- 172.21.15.129:0/2696624836 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f0610087920 con 0x7f06180ca8c0 2024-09-17T03:56:10.364 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:10.364 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:10.364 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 msgr2=0x7f05f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 0x7f05f407b0d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0608009d70 tx=0x7f0608005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 msgr2=0x7f06180cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f061002aea0 tx=0x7f0610033b80 comp rx=0 tx=0).stop 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 shutdown_connections 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05f4078bc0 0x7f05f407b0d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06180ca8c0 0x7f06180cacd0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06181f6430 0x7f061815f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 --2- 172.21.15.129:0/2696624836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc004d40 0x7f06181567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 >> 172.21.15.129:0/2696624836 conn(0x7f061806d410 msgr2=0x7f06180ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 shutdown_connections 2024-09-17T03:56:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.366+0000 7f061d000640 1 -- 172.21.15.129:0/2696624836 wait complete. 2024-09-17T03:56:10.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:56:10.636 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:56:10.636 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:56:10.637 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T03:56:10.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.781+0000 7f45fe303f40 1 Processor -- start 2024-09-17T03:56:10.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.781+0000 7f45fe303f40 1 -- start start 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x5585643d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x55856445b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5585642d7c60 con 0x5585643d2640 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5585643361d0 con 0x5585643d4610 2024-09-17T03:56:10.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.782+0000 7f45fe303f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55856445df50 con 0x558564443150 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52898/0 (socket says 172.21.15.129:52898) 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 -- 172.21.15.129:0/774400516 learned_addr learned my addr 172.21.15.129:0/774400516 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f6a00640 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x5585643d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7e00640 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x55856445b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 -- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 msgr2=0x55856445b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x55856445b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 -- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 msgr2=0x5585643d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x5585643d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.783+0000 7f45f7400640 1 -- 172.21.15.129:0/774400516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558564439000 con 0x5585643d2640 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45f7400640 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f45e8025e60 tx=0x7f45e8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71df25d01adc1b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45f5600640 1 -- 172.21.15.129:0/774400516 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e804d8c0 con 0x5585643d2640 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45f5600640 1 -- 172.21.15.129:0/774400516 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e804dd70 con 0x5585643d2640 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45f5600640 1 -- 172.21.15.129:0/774400516 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e80566d0 con 0x5585643d2640 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45fe303f40 1 -- 172.21.15.129:0/774400516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 msgr2=0x55856445c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.784+0000 7f45fe303f40 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f45e8025e60 tx=0x7f45e8002fb0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 -- 172.21.15.129:0/774400516 shutdown_connections 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x55856445b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x5585643d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 --2- 172.21.15.129:0/774400516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445c280 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 -- 172.21.15.129:0/774400516 >> 172.21.15.129:0/774400516 conn(0x5585642a8ac0 msgr2=0x558564441510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 -- 172.21.15.129:0/774400516 shutdown_connections 2024-09-17T03:56:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.785+0000 7f45fe303f40 1 -- 172.21.15.129:0/774400516 wait complete. 2024-09-17T03:56:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.786+0000 7f45fe303f40 1 Processor -- start 2024-09-17T03:56:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.786+0000 7f45fe303f40 1 -- start start 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x55856443f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x558564437c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55856445cce0 con 0x5585643d2640 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55856445e9e0 con 0x5585643d4610 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45fe303f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5585643d2300 con 0x558564443150 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.787+0000 7f45f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52900/0 (socket says 172.21.15.129:52900) 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 -- 172.21.15.129:0/3960232076 learned_addr learned my addr 172.21.15.129:0/3960232076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f6a00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x55856443f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7e00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x558564437c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 -- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 msgr2=0x558564437c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558564443150 0x558564437c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 -- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 msgr2=0x55856443f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x55856443f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 -- 172.21.15.129:0/3960232076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585644f6ca0 con 0x5585643d2640 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f6a00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5585643d4610 0x55856443f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.788+0000 7f45f7400640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445a040 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f45e8046560 tx=0x7f45e8046470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.789+0000 7f45efe00640 1 -- 172.21.15.129:0/3960232076 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e806f030 con 0x5585643d2640 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.789+0000 7f45efe00640 1 -- 172.21.15.129:0/3960232076 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45e8057750 con 0x5585643d2640 2024-09-17T03:56:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.789+0000 7f45efe00640 1 -- 172.21.15.129:0/3960232076 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e80579a0 con 0x5585643d2640 2024-09-17T03:56:10.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.789+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5585644f6f80 con 0x5585643d2640 2024-09-17T03:56:10.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.789+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558564437710 con 0x5585643d2640 2024-09-17T03:56:10.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.791+0000 7f45efe00640 1 -- 172.21.15.129:0/3960232076 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f45e8072070 con 0x5585643d2640 2024-09-17T03:56:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45efe00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45d80775a0 0x7f45d8079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45efe00640 1 -- 172.21.15.129:0/3960232076 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7f45e80de800 con 0x5585643d2640 2024-09-17T03:56:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45fe303f40 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5585644f6530 0x558564441510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e111) -- 0x558564441a50 con 0x5585644f6530 2024-09-17T03:56:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45f6a00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45d80775a0 0x7f45d8079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.792+0000 7f45f7e00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5585644f6530 0x558564441510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45f7e00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5585644f6530 0x558564441510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45f6a00640 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45d80775a0 0x7f45d8079ab0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e4001690 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45f7e00640 1 -- 172.21.15.129:0/3960232076 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv10 ondisk = 0) ==== 164+0+93 (crc 0 0 0) 0x7f45f0003040 con 0x5585644f6530 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5585644f6530 msgr2=0x558564441510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x5585644f6530 0x558564441510 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45d80775a0 msgr2=0x7f45d8079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45d80775a0 0x7f45d8079ab0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f45e4000c00 tx=0x7f45e4001690 comp rx=0 tx=0).stop 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 msgr2=0x55856445a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 --2- 172.21.15.129:0/3960232076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5585643d2640 0x55856445a040 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f45e8046560 tx=0x7f45e8046470 comp rx=0 tx=0).stop 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45f7e00640 1 -- 172.21.15.129:0/3960232076 reap_dead start 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 shutdown_connections 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.793+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 >> 172.21.15.129:0/3960232076 conn(0x5585642a8ac0 msgr2=0x5585643cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.794+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 shutdown_connections 2024-09-17T03:56:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:10.794+0000 7f45fe303f40 1 -- 172.21.15.129:0/3960232076 wait complete. 2024-09-17T03:56:10.804 INFO:teuthology.orchestra.run.smithi129.stdout:W�file_cdir_2# 2024-09-17T03:56:10.805 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:56:10.944 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1099511628278, 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_c", 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 17 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:10.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "dir_2", 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 35 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 17, 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:56:10.946 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:10.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:10.956 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir_2/file_c dir_2/file_d) 2024-09-17T03:56:10.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:10.995 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_c) 2024-09-17T03:56:11.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 -- 172.21.15.129:0/2495005074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf54004800 msgr2=0x7faf54004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 --2- 172.21.15.129:0/2495005074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf54004800 0x7faf54004c10 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7faf58009b20 tx=0x7faf54005830 comp rx=0 tx=0).stop 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf6e800640 1 -- 172.21.15.129:0/2495005074 reap_dead start 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 -- 172.21.15.129:0/2495005074 shutdown_connections 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 -- 172.21.15.129:0/2495005074 >> 172.21.15.129:0/2495005074 conn(0x7faf7006d410 msgr2=0x7faf700b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 -- 172.21.15.129:0/2495005074 shutdown_connections 2024-09-17T03:56:11.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.335+0000 7faf74a00640 1 -- 172.21.15.129:0/2495005074 wait complete. 2024-09-17T03:56:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.338+0000 7faf74a00640 1 Processor -- start 2024-09-17T03:56:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.338+0000 7faf74a00640 1 -- start start 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf54004800 0x7faf7015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf70099af0 0x7faf701f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faf700b3b90 con 0x7faf7009c970 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faf700708e0 con 0x7faf70099af0 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.339+0000 7faf74a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faf700b3d10 con 0x7faf54004800 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52918/0 (socket says 172.21.15.129:52918) 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 -- 172.21.15.129:0/658852441 learned_addr learned my addr 172.21.15.129:0/658852441 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf65400640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf70099af0 0x7faf701f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6de00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf54004800 0x7faf7015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 -- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf54004800 msgr2=0x7faf7015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf54004800 0x7faf7015b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 -- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf70099af0 msgr2=0x7faf701f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf70099af0 0x7faf701f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 -- 172.21.15.129:0/658852441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf701f5920 con 0x7faf7009c970 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf6e800640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7faf70072e20 tx=0x7faf6800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.340+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68003d80 con 0x7faf7009c970 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.341+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf68004b20 con 0x7faf7009c970 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.341+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf701f5c00 con 0x7faf7009c970 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.341+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68004030 con 0x7faf7009c970 2024-09-17T03:56:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.341+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf701f55f0 con 0x7faf7009c970 2024-09-17T03:56:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.342+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf700cbee0 con 0x7faf7009c970 2024-09-17T03:56:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.343+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faf680041d0 con 0x7faf7009c970 2024-09-17T03:56:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.345+0000 7faf66a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 0x7faf4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.345+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5795+0+0 (secure 0 0 0) 0x7faf6809bb30 con 0x7faf7009c970 2024-09-17T03:56:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.345+0000 7faf6de00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 0x7faf4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:11.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.346+0000 7faf6de00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 0x7faf4c07b0a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf60006b70 tx=0x7faf600067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:11.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.349+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faf680638b0 con 0x7faf7009c970 2024-09-17T03:56:11.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.566+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf7006a140 con 0x7faf7009c970 2024-09-17T03:56:11.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.566+0000 7faf66a00640 1 -- 172.21.15.129:0/658852441 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7faf680682e0 con 0x7faf7009c970 2024-09-17T03:56:11.566 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:11.566 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:11.567 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 msgr2=0x7faf4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 0x7faf4c07b0a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf60006b70 tx=0x7faf600067a0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 msgr2=0x7faf701e8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7faf70072e20 tx=0x7faf6800f790 comp rx=0 tx=0).stop 2024-09-17T03:56:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 shutdown_connections 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf4c078b90 0x7faf4c07b0a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf7009c970 0x7faf701e8d60 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf70099af0 0x7faf701f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 --2- 172.21.15.129:0/658852441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf54004800 0x7faf7015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 >> 172.21.15.129:0/658852441 conn(0x7faf7006d410 msgr2=0x7faf70073620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 shutdown_connections 2024-09-17T03:56:11.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:11.569+0000 7faf74a00640 1 -- 172.21.15.129:0/658852441 wait complete. 2024-09-17T03:56:11.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T03:56:11.834 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":1}} 2024-09-17T03:56:11.835 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-09-17T03:56:11.835 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 0, waiting (timeout=60)... 2024-09-17T03:56:16.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:16.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:56:16.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:56:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 -- 172.21.15.129:0/1074547510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 msgr2=0x7fca180cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 --2- 172.21.15.129:0/1074547510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca180cbbf0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fca18072360 tx=0x7fca0802a540 comp rx=0 tx=0).stop 2024-09-17T03:56:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 -- 172.21.15.129:0/1074547510 shutdown_connections 2024-09-17T03:56:17.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 --2- 172.21.15.129:0/1074547510 >> v2:172.21.15.181:3300/0 conn(0x7fca180b2950 0x7fca180b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 --2- 172.21.15.129:0/1074547510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca180cbbf0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 -- 172.21.15.129:0/1074547510 >> 172.21.15.129:0/1074547510 conn(0x7fca1806d410 msgr2=0x7fca180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 -- 172.21.15.129:0/1074547510 shutdown_connections 2024-09-17T03:56:17.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.071+0000 7fca1fc00640 1 -- 172.21.15.129:0/1074547510 wait complete. 2024-09-17T03:56:17.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.073+0000 7fca1fc00640 1 Processor -- start 2024-09-17T03:56:17.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.074+0000 7fca1fc00640 1 -- start start 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 0x7fca1815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca180b39c0 0x7fca181f6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca18070850 con 0x7fca180ce170 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca180706d0 con 0x7fca180b39c0 2024-09-17T03:56:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1fc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fca1806f5e0 con 0x7fca180b2950 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37002/0 (socket says 172.21.15.129:37002) 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.075+0000 7fca1dc00640 1 -- 172.21.15.129:0/3665696154 learned_addr learned my addr 172.21.15.129:0/3665696154 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca0fe00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca180b39c0 0x7fca181f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1d200640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 0x7fca1815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 -- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 msgr2=0x7fca1815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 0x7fca1815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 -- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca180b39c0 msgr2=0x7fca181f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca180b39c0 0x7fca181f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 -- 172.21.15.129:0/3665696154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1809b1a0 con 0x7fca180ce170 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1d200640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 0x7fca1815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1dc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c760 tx=0x7fca1000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10003da0 con 0x7fca180ce170 2024-09-17T03:56:17.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca1000ce10 con 0x7fca180ce170 2024-09-17T03:56:17.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1815c0e0 con 0x7fca180ce170 2024-09-17T03:56:17.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.076+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100052d0 con 0x7fca180ce170 2024-09-17T03:56:17.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.077+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca1809d3d0 con 0x7fca180ce170 2024-09-17T03:56:17.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.078+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1806f400 con 0x7fca180ce170 2024-09-17T03:56:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.079+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fca10005470 con 0x7fca180ce170 2024-09-17T03:56:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.080+0000 7fca15600640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 0x7fc9f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.080+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fca1009ab90 con 0x7fca180ce170 2024-09-17T03:56:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.080+0000 7fca1d200640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 0x7fc9f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.081+0000 7fca1d200640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 0x7fc9f407b0d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fca08034000 tx=0x7fca08004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:17.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.084+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fca1806f400 con 0x7fca180ce170 2024-09-17T03:56:17.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.300+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca1809d630 con 0x7fca180ce170 2024-09-17T03:56:17.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.301+0000 7fca15600640 1 -- 172.21.15.129:0/3665696154 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7fca180b3dd0 con 0x7fca180ce170 2024-09-17T03:56:17.301 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:17.301 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:17.301 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 msgr2=0x7fc9f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 0x7fc9f407b0d0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fca08034000 tx=0x7fca08004730 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 msgr2=0x7fca1815e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c760 tx=0x7fca1000cc20 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 shutdown_connections 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9f4078bc0 0x7fc9f407b0d0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca180ce170 0x7fca1815e660 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca180b39c0 0x7fca181f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 --2- 172.21.15.129:0/3665696154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca180b2950 0x7fca1815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 >> 172.21.15.129:0/3665696154 conn(0x7fca1806d410 msgr2=0x7fca180b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 shutdown_connections 2024-09-17T03:56:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.303+0000 7fca1fc00640 1 -- 172.21.15.129:0/3665696154 wait complete. 2024-09-17T03:56:17.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T03:56:17.572 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T03:56:17.573 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T03:56:17.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:17.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 -- 172.21.15.129:0/2688331506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6930160cf0 msgr2=0x7f69300d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 --2- 172.21.15.129:0/2688331506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6930160cf0 0x7f69300d02c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6918009830 tx=0x7f691802a480 comp rx=0 tx=0).stop 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 -- 172.21.15.129:0/2688331506 shutdown_connections 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 --2- 172.21.15.129:0/2688331506 >> v2:172.21.15.181:3300/0 conn(0x7f69300d0800 0x7f693015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 --2- 172.21.15.129:0/2688331506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6930160cf0 0x7f69300d02c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 -- 172.21.15.129:0/2688331506 >> 172.21.15.129:0/2688331506 conn(0x7f693006cfc0 msgr2=0x7f693006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 -- 172.21.15.129:0/2688331506 shutdown_connections 2024-09-17T03:56:17.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.797+0000 7f6935400640 1 -- 172.21.15.129:0/2688331506 wait complete. 2024-09-17T03:56:17.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.799+0000 7f6935400640 1 Processor -- start 2024-09-17T03:56:17.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.799+0000 7f6935400640 1 -- start start 2024-09-17T03:56:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 0x7f69301f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69300d0800 0x7f69300a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 0x7f69300a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f693015d4a0 con 0x7f6930160cf0 2024-09-17T03:56:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f693015d320 con 0x7f69300d0800 2024-09-17T03:56:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6935400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f693015d620 con 0x7f69300cfb60 2024-09-17T03:56:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6934a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 0x7f69300a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 0x7f69301f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6927400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69300d0800 0x7f69300a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6934a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 0x7f69300a9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37016/0 (socket says 172.21.15.129:37016) 2024-09-17T03:56:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f6934a00640 1 -- 172.21.15.129:0/3227910592 learned_addr learned my addr 172.21.15.129:0/3227910592 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 -- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69300d0800 msgr2=0x7f69300a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69300d0800 0x7f69300a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 -- 172.21.15.129:0/3227910592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 msgr2=0x7f69300a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:17.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 0x7f69300a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:17.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.801+0000 7f692fe00640 1 -- 172.21.15.129:0/3227910592 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69300aa460 con 0x7f69300cfb60 2024-09-17T03:56:17.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f692fe00640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 0x7f69301f6e40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f692000c700 tx=0x7f692000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:17.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920003b70 con 0x7f69300cfb60 2024-09-17T03:56:17.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6920003d10 con 0x7f69300cfb60 2024-09-17T03:56:17.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69300aa740 con 0x7f69300cfb60 2024-09-17T03:56:17.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692000aab0 con 0x7f69300cfb60 2024-09-17T03:56:17.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.802+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69300aad00 con 0x7f69300cfb60 2024-09-17T03:56:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.803+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69300b2950 con 0x7f69300cfb60 2024-09-17T03:56:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.805+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f692000ac50 con 0x7f69300cfb60 2024-09-17T03:56:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.805+0000 7f692cc00640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 0x7f690c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.806+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f692009a890 con 0x7f69300cfb60 2024-09-17T03:56:17.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.806+0000 7f6927400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 0x7f690c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:17.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.806+0000 7f6927400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 0x7f690c07b0a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6918034000 tx=0x7f6918004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:17.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:17.811+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f69300b2950 con 0x7f69300cfb60 2024-09-17T03:56:18.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.030+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f693015cbd0 con 0x7f69300cfb60 2024-09-17T03:56:18.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.031+0000 7f692cc00640 1 -- 172.21.15.129:0/3227910592 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f69300d0c10 con 0x7f69300cfb60 2024-09-17T03:56:18.031 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:18.031 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:18.032 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.033+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 msgr2=0x7f690c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.033+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 0x7f690c07b0a0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6918034000 tx=0x7f6918004730 comp rx=0 tx=0).stop 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 msgr2=0x7f69301f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 0x7f69301f6e40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f692000c700 tx=0x7f692000cbc0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 shutdown_connections 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f690c078b90 0x7f690c07b0a0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6930160cf0 0x7f69300a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69300d0800 0x7f69300a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 --2- 172.21.15.129:0/3227910592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69300cfb60 0x7f69301f6e40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 >> 172.21.15.129:0/3227910592 conn(0x7f693006cfc0 msgr2=0x7f693006f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:18.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 shutdown_connections 2024-09-17T03:56:18.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.034+0000 7f6935400640 1 -- 172.21.15.129:0/3227910592 wait complete. 2024-09-17T03:56:18.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T03:56:18.301 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_reintegrated":2}} 2024-09-17T03:56:18.302 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 2 } } 2024-09-17T03:56:18.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:18.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.522+0000 7efff4600640 1 -- 172.21.15.129:0/1937282603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0cd390 msgr2=0x7effec1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.522+0000 7efff4600640 1 --2- 172.21.15.129:0/1937282603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0cd390 0x7effec1609c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7effe4005fd0 tx=0x7effe402a940 comp rx=0 tx=0).stop 2024-09-17T03:56:18.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.523+0000 7efff4600640 1 -- 172.21.15.129:0/1937282603 shutdown_connections 2024-09-17T03:56:18.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.523+0000 7efff4600640 1 --2- 172.21.15.129:0/1937282603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0cd390 0x7effec1609c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.523+0000 7efff4600640 1 -- 172.21.15.129:0/1937282603 >> 172.21.15.129:0/1937282603 conn(0x7effec06d410 msgr2=0x7effec0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:18.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.523+0000 7efff4600640 1 -- 172.21.15.129:0/1937282603 shutdown_connections 2024-09-17T03:56:18.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.523+0000 7efff4600640 1 -- 172.21.15.129:0/1937282603 wait complete. 2024-09-17T03:56:18.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.525+0000 7efff4600640 1 Processor -- start 2024-09-17T03:56:18.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.525+0000 7efff4600640 1 -- start start 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 0x7effec1ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effec0ca8c0 0x7effec0c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7effec160f00 con 0x7effec0b4140 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7effec070d40 con 0x7effec0ca8c0 2024-09-17T03:56:18.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff4600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7effec161080 con 0x7effec0705a0 2024-09-17T03:56:18.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:18.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37038/0 (socket says 172.21.15.129:37038) 2024-09-17T03:56:18.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 -- 172.21.15.129:0/2385830931 learned_addr learned my addr 172.21.15.129:0/2385830931 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:18.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff2600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effec0ca8c0 0x7effec0c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:18.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1c00640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 0x7effec1ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 -- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 msgr2=0x7effec1ef6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 0x7effec1ef6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 -- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effec0ca8c0 msgr2=0x7effec0c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effec0ca8c0 0x7effec0c95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1200640 1 -- 172.21.15.129:0/2385830931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec0c82a0 con 0x7effec0b4140 2024-09-17T03:56:18.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.527+0000 7efff1c00640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 0x7effec1ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7efff1200640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7effd8001eb0 tx=0x7effd800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8017070 con 0x7effec0b4140 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effd80053c0 con 0x7effec0b4140 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd8003830 con 0x7effec0b4140 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effec0c8580 con 0x7effec0b4140 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.528+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effec0c8b40 con 0x7effec0b4140 2024-09-17T03:56:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.529+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effec154c70 con 0x7effec0b4140 2024-09-17T03:56:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.530+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7effd801a070 con 0x7effec0b4140 2024-09-17T03:56:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.531+0000 7effe9600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 0x7effcc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.531+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7effd809a0c0 con 0x7effec0b4140 2024-09-17T03:56:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.532+0000 7efff1c00640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 0x7effcc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.532+0000 7efff1c00640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 0x7effcc07b0a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7effe0000f60 tx=0x7effe0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:18.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.536+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7effec154c70 con 0x7effec0b4140 2024-09-17T03:56:18.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.756+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effec1609c0 con 0x7effec0b4140 2024-09-17T03:56:18.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.756+0000 7effe9600640 1 -- 172.21.15.129:0/2385830931 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7effd8061e20 con 0x7effec0b4140 2024-09-17T03:56:18.757 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:18.757 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:18.757 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:18.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 msgr2=0x7effcc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 0x7effcc07b0a0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7effe0000f60 tx=0x7effe0009040 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 msgr2=0x7effec0c9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7effd8001eb0 tx=0x7effd800c890 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 shutdown_connections 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effcc078b90 0x7effcc07b0a0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7effec0ca8c0 0x7effec0c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7effec0b4140 0x7effec0c9060 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 --2- 172.21.15.129:0/2385830931 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7effec0705a0 0x7effec1ef6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 >> 172.21.15.129:0/2385830931 conn(0x7effec06d410 msgr2=0x7effec0c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 shutdown_connections 2024-09-17T03:56:18.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:18.759+0000 7efff4600640 1 -- 172.21.15.129:0/2385830931 wait complete. 2024-09-17T03:56:18.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:56:19.028 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:56:19.028 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:56:19.029 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T03:56:19.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.169+0000 7f99ded20f40 1 Processor -- start 2024-09-17T03:56:19.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.169+0000 7f99ded20f40 1 -- start start 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x5591042772c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910428f7e0 0x559104315890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559104107c60 con 0x5591041ffaf0 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5591041661d0 con 0x559104278b70 2024-09-17T03:56:19.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99ded20f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5591041b8d10 con 0x55910428f7e0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99d7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99d7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37046/0 (socket says 172.21.15.129:37046) 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99d7400640 1 -- 172.21.15.129:0/3298423727 learned_addr learned my addr 172.21.15.129:0/3298423727 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99d7e00640 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910428f7e0 0x559104315890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.171+0000 7f99d6a00640 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x5591042772c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 -- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910428f7e0 msgr2=0x559104315890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910428f7e0 0x559104315890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 -- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 msgr2=0x5591042772c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x5591042772c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 -- 172.21.15.129:0/3298423727 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559104275320 con 0x5591041ffaf0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d7400640 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f99c8001e10 tx=0x7f99c800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4ecbe17f1d0e1a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d5600640 1 -- 172.21.15.129:0/3298423727 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8016070 con 0x5591041ffaf0 2024-09-17T03:56:19.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d5600640 1 -- 172.21.15.129:0/3298423727 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99c800d7c0 con 0x5591041ffaf0 2024-09-17T03:56:19.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.172+0000 7f99d5600640 1 -- 172.21.15.129:0/3298423727 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c800dc10 con 0x5591041ffaf0 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 -- 172.21.15.129:0/3298423727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 msgr2=0x5591042792b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f99c8001e10 tx=0x7f99c800b650 comp rx=0 tx=0).stop 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 -- 172.21.15.129:0/3298423727 shutdown_connections 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910428f7e0 0x559104315890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x5591042772c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 --2- 172.21.15.129:0/3298423727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5591041ffaf0 0x5591042792b0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 -- 172.21.15.129:0/3298423727 >> 172.21.15.129:0/3298423727 conn(0x5591040d8ac0 msgr2=0x559104277dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 -- 172.21.15.129:0/3298423727 shutdown_connections 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.173+0000 7f99ded20f40 1 -- 172.21.15.129:0/3298423727 wait complete. 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.174+0000 7f99ded20f40 1 Processor -- start 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.174+0000 7f99ded20f40 1 -- start start 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5591041ffaf0 0x559104282ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x55910432a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 0x559104325d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559104202630 con 0x55910428f7e0 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x559104202e00 con 0x559104278b70 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.175+0000 7f99ded20f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x559104283850 con 0x5591041ffaf0 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 0x559104325d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d6a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x55910432a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5591041ffaf0 0x559104282ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 0x559104325d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37052/0 (socket says 172.21.15.129:37052) 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 -- 172.21.15.129:0/1950588295 learned_addr learned my addr 172.21.15.129:0/1950588295 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 -- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5591041ffaf0 msgr2=0x559104282ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5591041ffaf0 0x559104282ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 -- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 msgr2=0x55910432a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x55910432a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d7e00640 1 -- 172.21.15.129:0/1950588295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55910432a9e0 con 0x55910428f7e0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.176+0000 7f99d6a00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559104278b70 0x55910432a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99d7e00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 0x559104325d80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a600 tx=0x7f99d004d6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99cfe00640 1 -- 172.21.15.129:0/1950588295 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d0065070 con 0x55910428f7e0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99cfe00640 1 -- 172.21.15.129:0/1950588295 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99d006a550 con 0x55910428f7e0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55910427ff70 con 0x55910428f7e0 2024-09-17T03:56:19.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99cfe00640 1 -- 172.21.15.129:0/1950588295 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d006a980 con 0x55910428f7e0 2024-09-17T03:56:19.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.177+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559104280530 con 0x55910428f7e0 2024-09-17T03:56:19.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.179+0000 7f99cfe00640 1 -- 172.21.15.129:0/1950588295 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f99d005e040 con 0x55910428f7e0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.180+0000 7f99cfe00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99b8077570 0x7f99b8079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.180+0000 7f99cfe00640 1 -- 172.21.15.129:0/1950588295 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f99d00de5f0 con 0x55910428f7e0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.180+0000 7f99ded20f40 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x559104283420 0x5591041b78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.180+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5591041b7e00 con 0x559104283420 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.181+0000 7f99d6a00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x559104283420 0x5591041b78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.181+0000 7f99d7400640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99b8077570 0x7f99b8079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.181+0000 7f99d6a00640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x559104283420 0x5591041b78c0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.181+0000 7f99d7400640 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99b8077570 0x7f99b8079a80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f99c8000c00 tx=0x7f99c8001080 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99d6a00640 1 -- 172.21.15.129:0/1950588295 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (93) out=93b] v0'0 uv11 ondisk = 0) ==== 164+0+93 (crc 0 0 0) 0x7f99c400cd20 con 0x559104283420 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x559104283420 msgr2=0x5591041b78c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x559104283420 0x5591041b78c0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99b8077570 msgr2=0x7f99b8079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99b8077570 0x7f99b8079a80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f99c8000c00 tx=0x7f99c8001080 comp rx=0 tx=0).stop 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 msgr2=0x559104325d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 --2- 172.21.15.129:0/1950588295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910428f7e0 0x559104325d80 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f99d000a600 tx=0x7f99d004d6a0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99d7e00640 1 -- 172.21.15.129:0/1950588295 reap_dead start 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 shutdown_connections 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 >> 172.21.15.129:0/1950588295 conn(0x5591040d8ac0 msgr2=0x5591042638b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 shutdown_connections 2024-09-17T03:56:19.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.182+0000 7f99ded20f40 1 -- 172.21.15.129:0/1950588295 wait complete. 2024-09-17T03:56:19.192 INFO:teuthology.orchestra.run.smithi129.stdout:W�file_ddir_2. 2024-09-17T03:56:19.192 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1099511628278, 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_d", 2024-09-17T03:56:19.334 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 27 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "dir_2", 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 46 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 17, 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:56:19.335 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:19.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:19.345 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_2/file_d) 2024-09-17T03:56:19.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 -- 172.21.15.129:0/3379030592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980b4140 msgr2=0x7f2b980ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 --2- 172.21.15.129:0/3379030592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980b4140 0x7f2b980ccc80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f2b84009830 tx=0x7f2b8402a520 comp rx=0 tx=0).stop 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 -- 172.21.15.129:0/3379030592 shutdown_connections 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 --2- 172.21.15.129:0/3379030592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980cd1c0 0x7f2b981609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 --2- 172.21.15.129:0/3379030592 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980b4140 0x7f2b980ccc80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 -- 172.21.15.129:0/3379030592 >> 172.21.15.129:0/3379030592 conn(0x7f2b9806d410 msgr2=0x7f2b980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 -- 172.21.15.129:0/3379030592 shutdown_connections 2024-09-17T03:56:19.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.644+0000 7f2ba0000640 1 -- 172.21.15.129:0/3379030592 wait complete. 2024-09-17T03:56:19.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.647+0000 7f2ba0000640 1 Processor -- start 2024-09-17T03:56:19.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.647+0000 7f2ba0000640 1 -- start start 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980b4140 0x7f2b981f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980cd1c0 0x7f2b9815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2b98160260 con 0x7f2b980705a0 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2b981600e0 con 0x7f2b980cd1c0 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.648+0000 7f2ba0000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2b98160f00 con 0x7f2b980b4140 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37064/0 (socket says 172.21.15.129:37064) 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 -- 172.21.15.129:0/2475216675 learned_addr learned my addr 172.21.15.129:0/2475216675 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9cc00640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980b4140 0x7f2b981f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9e000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980cd1c0 0x7f2b9815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 -- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980b4140 msgr2=0x7f2b981f5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980b4140 0x7f2b981f5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 -- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980cd1c0 msgr2=0x7f2b9815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980cd1c0 0x7f2b9815b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 -- 172.21.15.129:0/2475216675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b981f5940 con 0x7f2b980705a0 2024-09-17T03:56:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.649+0000 7f2b9d600640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c370 tx=0x7f2b8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.650+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c0052d0 con 0x7f2b980705a0 2024-09-17T03:56:19.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.650+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b8c007020 con 0x7f2b980705a0 2024-09-17T03:56:19.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.650+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9815c6f0 con 0x7f2b980705a0 2024-09-17T03:56:19.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.650+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c007470 con 0x7f2b980705a0 2024-09-17T03:56:19.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.650+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b980c3fb0 con 0x7f2b980705a0 2024-09-17T03:56:19.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.651+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b980ccc80 con 0x7f2b980705a0 2024-09-17T03:56:19.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.652+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2b8c0067d0 con 0x7f2b980705a0 2024-09-17T03:56:19.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.653+0000 7f2b95600640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 0x7f2b7807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:19.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.653+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f2b8c09a1f0 con 0x7f2b980705a0 2024-09-17T03:56:19.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.654+0000 7f2b9cc00640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 0x7f2b7807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.654+0000 7f2b9cc00640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 0x7f2b7807b0d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2b84034000 tx=0x7f2b84004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:19.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.658+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2b980cd5d0 con 0x7f2b980705a0 2024-09-17T03:56:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.875+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b9806a140 con 0x7f2b980705a0 2024-09-17T03:56:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.875+0000 7f2b95600640 1 -- 172.21.15.129:0/2475216675 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f2b8c061f40 con 0x7f2b980705a0 2024-09-17T03:56:19.876 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:19.876 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 msgr2=0x7f2b7807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 0x7f2b7807b0d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2b84034000 tx=0x7f2b84004730 comp rx=0 tx=0).stop 2024-09-17T03:56:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 msgr2=0x7f2b980c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00c370 tx=0x7f2b8c00c830 comp rx=0 tx=0).stop 2024-09-17T03:56:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 shutdown_connections 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b78078bc0 0x7f2b7807b0d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b980cd1c0 0x7f2b9815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b980b4140 0x7f2b981f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 --2- 172.21.15.129:0/2475216675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b980705a0 0x7f2b980c4a90 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 >> 172.21.15.129:0/2475216675 conn(0x7f2b9806d410 msgr2=0x7f2b980ad6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 shutdown_connections 2024-09-17T03:56:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:19.878+0000 7f2ba0000640 1 -- 172.21.15.129:0/2475216675 wait complete. 2024-09-17T03:56:19.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:56:20.146 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T03:56:20.146 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T03:56:20.146 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 1, waiting (timeout=60)... 2024-09-17T03:56:25.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:25.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 -- 172.21.15.129:0/2201346715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 msgr2=0x7fb6a40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 --2- 172.21.15.129:0/2201346715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a40b4140 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fb694005fd0 tx=0x7fb69402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 -- 172.21.15.129:0/2201346715 shutdown_connections 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 --2- 172.21.15.129:0/2201346715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a40cd390 0x7fb6a41609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 --2- 172.21.15.129:0/2201346715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a40b4140 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 -- 172.21.15.129:0/2201346715 >> 172.21.15.129:0/2201346715 conn(0x7fb6a406d410 msgr2=0x7fb6a40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 -- 172.21.15.129:0/2201346715 shutdown_connections 2024-09-17T03:56:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.378+0000 7fb6aa800640 1 -- 172.21.15.129:0/2201346715 wait complete. 2024-09-17T03:56:25.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.380+0000 7fb6aa800640 1 Processor -- start 2024-09-17T03:56:25.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.381+0000 7fb6aa800640 1 -- start start 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a406eed0 0x7fb6a40c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a4157340 0x7fb6a41595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4160f00 con 0x7fb6a40705a0 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a406f4f0 con 0x7fb6a406eed0 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6aa800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a4161080 con 0x7fb6a4157340 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb69aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb69aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59514/0 (socket says 172.21.15.129:59514) 2024-09-17T03:56:25.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb69aa00640 1 -- 172.21.15.129:0/4177704502 learned_addr learned my addr 172.21.15.129:0/4177704502 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.382+0000 7fb6a3400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a406eed0 0x7fb6a40c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb6a3e00640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a4157340 0x7fb6a41595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 -- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a4157340 msgr2=0x7fb6a41595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a4157340 0x7fb6a41595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 -- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a406eed0 msgr2=0x7fb6a40c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a406eed0 0x7fb6a40c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 -- 172.21.15.129:0/4177704502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a415bac0 con 0x7fb6a40705a0 2024-09-17T03:56:25.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69aa00640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c340 tx=0x7fb68c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:25.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0052d0 con 0x7fb6a40705a0 2024-09-17T03:56:25.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb68c007020 con 0x7fb6a40705a0 2024-09-17T03:56:25.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c007470 con 0x7fb6a40705a0 2024-09-17T03:56:25.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.383+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a415bd50 con 0x7fb6a40705a0 2024-09-17T03:56:25.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.384+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a415e6f0 con 0x7fb6a40705a0 2024-09-17T03:56:25.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.384+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a40cc3a0 con 0x7fb6a40705a0 2024-09-17T03:56:25.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.386+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb68c0067d0 con 0x7fb6a40705a0 2024-09-17T03:56:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.387+0000 7fb69be00640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 0x7fb68007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.387+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7fb68c09a300 con 0x7fb6a40705a0 2024-09-17T03:56:25.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.387+0000 7fb6a3400640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 0x7fb68007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:25.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.388+0000 7fb6a3400640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 0x7fb68007b0a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb694033000 tx=0x7fb694004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.391+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb6a40cc3a0 con 0x7fb6a40705a0 2024-09-17T03:56:25.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.611+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6a40b4140 con 0x7fb6a40705a0 2024-09-17T03:56:25.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.612+0000 7fb69be00640 1 -- 172.21.15.129:0/4177704502 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7fb68c09a0c0 con 0x7fb6a40705a0 2024-09-17T03:56:25.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:25.612 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:25.613 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 msgr2=0x7fb68007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 0x7fb68007b0a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fb694033000 tx=0x7fb694004730 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 msgr2=0x7fb6a4159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c340 tx=0x7fb68c00c800 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 shutdown_connections 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb680078b90 0x7fb68007b0a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6a4157340 0x7fb6a41595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6a40705a0 0x7fb6a4159080 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 --2- 172.21.15.129:0/4177704502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6a406eed0 0x7fb6a40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:25.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.614+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 >> 172.21.15.129:0/4177704502 conn(0x7fb6a406d410 msgr2=0x7fb6a415f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:25.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.615+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 shutdown_connections 2024-09-17T03:56:25.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:25.615+0000 7fb6aa800640 1 -- 172.21.15.129:0/4177704502 wait complete. 2024-09-17T03:56:25.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:56:25.883 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-09-17T03:56:25.884 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-09-17T03:56:25.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 -- 172.21.15.129:0/3463944381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 msgr2=0x7f131c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 --2- 172.21.15.129:0/3463944381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f131c004800 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f13200017d0 tx=0x7f131c005830 comp rx=0 tx=0).stop 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f1339800640 1 -- 172.21.15.129:0/3463944381 reap_dead start 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 -- 172.21.15.129:0/3463944381 shutdown_connections 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 -- 172.21.15.129:0/3463944381 >> 172.21.15.129:0/3463944381 conn(0x7f133406d410 msgr2=0x7f13340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 -- 172.21.15.129:0/3463944381 shutdown_connections 2024-09-17T03:56:26.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.104+0000 7f133b800640 1 -- 172.21.15.129:0/3463944381 wait complete. 2024-09-17T03:56:26.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.107+0000 7f133b800640 1 Processor -- start 2024-09-17T03:56:26.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.107+0000 7f133b800640 1 -- start start 2024-09-17T03:56:26.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13341471e0 0x7f13340c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13340c4fa0 0x7f13340b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f133415ae00 con 0x7f131c0041c0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1334071170 con 0x7f13340c4fa0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f133b800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f133415af80 con 0x7f13341471e0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f1338e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f1338e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59534/0 (socket says 172.21.15.129:59534) 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.108+0000 7f1338e00640 1 -- 172.21.15.129:0/46320599 learned_addr learned my addr 172.21.15.129:0/46320599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 -- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13341471e0 msgr2=0x7f13340c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13341471e0 0x7f13340c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 -- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13340c4fa0 msgr2=0x7f13340b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13340c4fa0 0x7f13340b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 -- 172.21.15.129:0/46320599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f133409ab40 con 0x7f131c0041c0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1338e00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f132800eaa0 tx=0x7f132800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132800dc40 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f132800dde0 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f133409ae20 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13280042b0 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.109+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1334146cc0 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.110+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133406f640 con 0x7f131c0041c0 2024-09-17T03:56:26.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.112+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f13280035f0 con 0x7f131c0041c0 2024-09-17T03:56:26.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.113+0000 7f1330c00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 0x7f131407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.113+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f132809b950 con 0x7f131c0041c0 2024-09-17T03:56:26.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.113+0000 7f1333e00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 0x7f131407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.114+0000 7f1333e00640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 0x7f131407b0a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1320032000 tx=0x7f13200012f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:26.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.117+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1328063cf0 con 0x7f131c0041c0 2024-09-17T03:56:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.334+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f133406a140 con 0x7f131c0041c0 2024-09-17T03:56:26.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.335+0000 7f1330c00640 1 -- 172.21.15.129:0/46320599 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f13280635f0 con 0x7f131c0041c0 2024-09-17T03:56:26.518 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:26.518 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:26.520 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:26.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 msgr2=0x7f131407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 0x7f131407b0a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1320032000 tx=0x7f13200012f0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 msgr2=0x7f133409a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f132800eaa0 tx=0x7f132800ef60 comp rx=0 tx=0).stop 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 shutdown_connections 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1314078b90 0x7f131407b0a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13340c4fa0 0x7f13340b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13341471e0 0x7f13340c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 --2- 172.21.15.129:0/46320599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f131c0041c0 0x7f133409a1d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 >> 172.21.15.129:0/46320599 conn(0x7f133406d410 msgr2=0x7f133415a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 shutdown_connections 2024-09-17T03:56:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.338+0000 7f133b800640 1 -- 172.21.15.129:0/46320599 wait complete. 2024-09-17T03:56:26.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-09-17T03:56:26.765 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":1}} 2024-09-17T03:56:26.765 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-09-17T03:56:26.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:26.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 -- 172.21.15.129:0/1805343366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 msgr2=0x7f901006fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 --2- 172.21.15.129:0/1805343366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f901006fa50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f901006f050 tx=0x7f900802aa80 comp rx=0 tx=0).stop 2024-09-17T03:56:26.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 -- 172.21.15.129:0/1805343366 shutdown_connections 2024-09-17T03:56:26.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 --2- 172.21.15.129:0/1805343366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f901006fa50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 --2- 172.21.15.129:0/1805343366 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 0x7f90100cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 -- 172.21.15.129:0/1805343366 >> 172.21.15.129:0/1805343366 conn(0x7f901006d410 msgr2=0x7f90100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:26.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 -- 172.21.15.129:0/1805343366 shutdown_connections 2024-09-17T03:56:26.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.987+0000 7f9018600640 1 -- 172.21.15.129:0/1805343366 wait complete. 2024-09-17T03:56:26.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.989+0000 7f9018600640 1 Processor -- start 2024-09-17T03:56:26.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.989+0000 7f9018600640 1 -- start start 2024-09-17T03:56:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f90100ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 0x7f90100c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90100710e0 con 0x7f90100af4c0 2024-09-17T03:56:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9010070f60 con 0x7f901006f640 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.990+0000 7f9018600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f901006ff90 con 0x7f90100ccc80 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59540/0 (socket says 172.21.15.129:59540) 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 -- 172.21.15.129:0/2063880359 learned_addr learned my addr 172.21.15.129:0/2063880359 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9016600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 0x7f90100c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015c00640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f90100ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 -- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 msgr2=0x7f90100c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 0x7f90100c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 -- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 msgr2=0x7f90100ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f90100ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.991+0000 7f9015200640 1 -- 172.21.15.129:0/2063880359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90100ae960 con 0x7f90100af4c0 2024-09-17T03:56:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f9015200640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00e790 tx=0x7f8ffc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc00eea0 con 0x7f90100af4c0 2024-09-17T03:56:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ffc00cdb0 con 0x7f90100af4c0 2024-09-17T03:56:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ffc010690 con 0x7f90100af4c0 2024-09-17T03:56:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90101f8f60 con 0x7f90100af4c0 2024-09-17T03:56:26.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.992+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90100aec40 con 0x7f90100af4c0 2024-09-17T03:56:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.993+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9010070430 con 0x7f90100af4c0 2024-09-17T03:56:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.994+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8ffc010830 con 0x7f90100af4c0 2024-09-17T03:56:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.995+0000 7f900d600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 0x7f8ff007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.995+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f8ffc09b810 con 0x7f90100af4c0 2024-09-17T03:56:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.996+0000 7f9015c00640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 0x7f8ff007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:26.997+0000 7f9015c00640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 0x7f8ff007b0a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9004001720 tx=0x7f9004009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:27.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.000+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8ffc0634d0 con 0x7f90100af4c0 2024-09-17T03:56:27.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.216+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f901006a140 con 0x7f90100af4c0 2024-09-17T03:56:27.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.217+0000 7f900d600640 1 -- 172.21.15.129:0/2063880359 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f8ffc067f00 con 0x7f90100af4c0 2024-09-17T03:56:27.217 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:27.217 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:27.218 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 msgr2=0x7f8ff007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 0x7f8ff007b0a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f9004001720 tx=0x7f9004009040 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 msgr2=0x7f90100adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00e790 tx=0x7f8ffc00ec50 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 shutdown_connections 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff0078b90 0x7f8ff007b0a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90100ccc80 0x7f90100c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90100af4c0 0x7f90100adb80 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 --2- 172.21.15.129:0/2063880359 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f901006f640 0x7f90100ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.219+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 >> 172.21.15.129:0/2063880359 conn(0x7f901006d410 msgr2=0x7f901015c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:27.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.220+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 shutdown_connections 2024-09-17T03:56:27.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.220+0000 7f9018600640 1 -- 172.21.15.129:0/2063880359 wait complete. 2024-09-17T03:56:27.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache 2024-09-17T03:56:27.479 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":3,"strays_enqueued":1,"strays_reintegrated":2,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0}} 2024-09-17T03:56:27.480 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 3, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 2, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 } } 2024-09-17T03:56:27.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:27.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 -- 172.21.15.129:0/2681144328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940b3440 msgr2=0x7f97940b9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 --2- 172.21.15.129:0/2681144328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940b3440 0x7f97940b9af0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f978c005fd0 tx=0x7f978c02a8b0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 -- 172.21.15.129:0/2681144328 shutdown_connections 2024-09-17T03:56:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 --2- 172.21.15.129:0/2681144328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940b3440 0x7f97940b9af0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 -- 172.21.15.129:0/2681144328 >> 172.21.15.129:0/2681144328 conn(0x7f979408bff0 msgr2=0x7f979409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 -- 172.21.15.129:0/2681144328 shutdown_connections 2024-09-17T03:56:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.698+0000 7f9793e00640 1 -- 172.21.15.129:0/2681144328 wait complete. 2024-09-17T03:56:27.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.700+0000 7f9793e00640 1 Processor -- start 2024-09-17T03:56:27.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.700+0000 7f9793e00640 1 -- start start 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 0x7f979409e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97940d4ba0 0x7f97940ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97940d3510 con 0x7f97940d1850 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97940d3390 con 0x7f97940b3440 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f97940cf7c0 con 0x7f97940d4ba0 2024-09-17T03:56:27.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59552/0 (socket says 172.21.15.129:59552) 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 -- 172.21.15.129:0/5360648 learned_addr learned my addr 172.21.15.129:0/5360648 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792a00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 0x7f979409e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9793400640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97940d4ba0 0x7f97940ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 -- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97940d4ba0 msgr2=0x7f97940ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97940d4ba0 0x7f97940ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 -- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 msgr2=0x7f979409e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 0x7f979409e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792000640 1 -- 172.21.15.129:0/5360648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9794152600 con 0x7f97940d1850 2024-09-17T03:56:27.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.702+0000 7f9792a00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 0x7f979409e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f9792000640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f978000c370 tx=0x7f978000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780017070 con 0x7f97940d1850 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97800035f0 con 0x7f97940d1850 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97940c9120 con 0x7f97940d1850 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9780003a40 con 0x7f97940d1850 2024-09-17T03:56:27.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.703+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97940c96e0 con 0x7f97940d1850 2024-09-17T03:56:27.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.704+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97940cfc70 con 0x7f97940d1850 2024-09-17T03:56:27.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.705+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f978001a070 con 0x7f97940d1850 2024-09-17T03:56:27.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.706+0000 7f978aa00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 0x7f977407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:27.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.706+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f978009a1a0 con 0x7f97940d1850 2024-09-17T03:56:27.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.707+0000 7f9792a00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 0x7f977407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:27.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.707+0000 7f9792a00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 0x7f977407b0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9784007fd0 tx=0x7f9784009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:27.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.711+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f97800624b0 con 0x7f97940d1850 2024-09-17T03:56:27.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.933+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97940b4300 con 0x7f97940d1850 2024-09-17T03:56:27.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.933+0000 7f978aa00640 1 -- 172.21.15.129:0/5360648 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f9780061db0 con 0x7f97940d1850 2024-09-17T03:56:27.934 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:27.934 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:27.934 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.935+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 msgr2=0x7f977407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.935+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 0x7f977407b0a0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9784007fd0 tx=0x7f9784009040 comp rx=0 tx=0).stop 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 msgr2=0x7f97941520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f978000c370 tx=0x7f978000c830 comp rx=0 tx=0).stop 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 shutdown_connections 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9774078b90 0x7f977407b0a0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97940d4ba0 0x7f97940ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97940d1850 0x7f97941520c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 --2- 172.21.15.129:0/5360648 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97940b3440 0x7f979409e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:27.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 >> 172.21.15.129:0/5360648 conn(0x7f979408bff0 msgr2=0x7f9794161d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:27.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 shutdown_connections 2024-09-17T03:56:27.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:27.936+0000 7f9793e00640 1 -- 172.21.15.129:0/5360648 wait complete. 2024-09-17T03:56:27.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue 2024-09-17T03:56:28.197 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":2,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":2,"pq_executed":1,"pq_item_in_journal":0}} 2024-09-17T03:56:28.197 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 2, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 2, "pq_item_in_journal": 0 } } 2024-09-17T03:56:28.198 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T03:56:28.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.335+0000 7fe20f861f40 1 Processor -- start 2024-09-17T03:56:28.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.335+0000 7fe20f861f40 1 -- start start 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbafab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbb86440 0x55eddbafb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eddba00c60 con 0x55eddbaf2910 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eddba5f1d0 con 0x55eddbb80360 2024-09-17T03:56:28.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe20f861f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55eddbb80770 con 0x55eddbb86440 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe209400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe209400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59578/0 (socket says 172.21.15.129:59578) 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.336+0000 7fe209400640 1 -- 172.21.15.129:0/2295384222 learned_addr learned my addr 172.21.15.129:0/2295384222 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe203e00640 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbb86440 0x55eddbafb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe208a00640 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbafab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 -- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbb86440 msgr2=0x55eddbafb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbb86440 0x55eddbafb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 -- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 msgr2=0x55eddbafab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbafab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 -- 172.21.15.129:0/2295384222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddbb87a80 con 0x55eddbaf2910 2024-09-17T03:56:28.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe209400640 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe204025c50 tx=0x7fe204003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1274ecfaab11047e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe202a00640 1 -- 172.21.15.129:0/2295384222 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204064030 con 0x55eddbaf2910 2024-09-17T03:56:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe202a00640 1 -- 172.21.15.129:0/2295384222 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe204004020 con 0x55eddbaf2910 2024-09-17T03:56:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.337+0000 7fe202a00640 1 -- 172.21.15.129:0/2295384222 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe204004230 con 0x55eddbaf2910 2024-09-17T03:56:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 -- 172.21.15.129:0/2295384222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 msgr2=0x55eddbaf2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fe204025c50 tx=0x7fe204003330 comp rx=0 tx=0).stop 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 -- 172.21.15.129:0/2295384222 shutdown_connections 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbaf2910 0x55eddbaf2d20 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbb86440 0x55eddbafb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2295384222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbafab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 -- 172.21.15.129:0/2295384222 >> 172.21.15.129:0/2295384222 conn(0x55eddb9d1ac0 msgr2=0x55eddbb8a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 -- 172.21.15.129:0/2295384222 shutdown_connections 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.338+0000 7fe20f861f40 1 -- 172.21.15.129:0/2295384222 wait complete. 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.339+0000 7fe20f861f40 1 Processor -- start 2024-09-17T03:56:28.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.339+0000 7fe20f861f40 1 -- start start 2024-09-17T03:56:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbaf2910 0x55eddbb5e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbb89590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55eddbb67c00 con 0x55eddbb86440 2024-09-17T03:56:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55eddbb856f0 con 0x55eddbb80360 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe20f861f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55eddbb5f0a0 con 0x55eddbaf2910 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59582/0 (socket says 172.21.15.129:59582) 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 learned_addr learned my addr 172.21.15.129:0/2730065995 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbb89590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbaf2910 0x55eddbb5e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbaf2910 msgr2=0x55eddbb5e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbaf2910 0x55eddbb5e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 msgr2=0x55eddbb89590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbb89590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eddbb6f7d0 con 0x55eddbb86440 2024-09-17T03:56:28.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55eddbaf2910 0x55eddbb5e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.341+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eddbb80360 0x55eddbb89590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe2040045b0 tx=0x7fe20404daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe201600640 1 -- 172.21.15.129:0/2730065995 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20404ddd0 con 0x55eddbb86440 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe201600640 1 -- 172.21.15.129:0/2730065995 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe204060560 con 0x55eddbb86440 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55eddbafa610 con 0x55eddbb86440 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe201600640 1 -- 172.21.15.129:0/2730065995 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2040609b0 con 0x55eddbb86440 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.342+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eddbc22c10 con 0x55eddbb86440 2024-09-17T03:56:28.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.344+0000 7fe201600640 1 -- 172.21.15.129:0/2730065995 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe204067070 con 0x55eddbb86440 2024-09-17T03:56:28.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.345+0000 7fe201600640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 0x7fe1e8079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.345+0000 7fe201600640 1 -- 172.21.15.129:0/2730065995 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7fe2040dfc00 con 0x55eddbb86440 2024-09-17T03:56:28.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.346+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x55eddbb6e420 0x55eddbb6e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.346+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.346+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 0x7fe1e8079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.346+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x55eddbb6e420 0x55eddbb6e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.346+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 0x7fe1e8079ab0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00d650 tx=0x7fe1fc004880 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.347+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x55eddbb6e420 0x55eddbb6e830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.347+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.347+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fe1f4010920 0x7fe1f400d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.347+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.348+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fe1f4010920 0x7fe1f400d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.348+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fe1f4010920 0x7fe1f400d380 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.348+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.349+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fe1fc017410 0x7fe1fc017820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.349+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.349+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fe1fc017410 0x7fe1fc017820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.349+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fe1fc017410 0x7fe1fc017820 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.350+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.350+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fe1f401e8b0 0x7fe1f401d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.350+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.350+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fe1f401e8b0 0x7fe1f401d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.350+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fe1f401e8b0 0x7fe1f401d330 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.351+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.351+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.351+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.351+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.352+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(6 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.352+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.352+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.352+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(8 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(10 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.353+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.354+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.354+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.354+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.354+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.355+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(13 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.355+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fe1f402d040 0x7fe1f4021420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.355+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.355+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fe1f402d040 0x7fe1f4021420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.356+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fe1f402d040 0x7fe1f4021420 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.356+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.356+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.356+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(15 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fe1fc028940 0x7fe1fc026410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.357+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.358+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fe1fc028940 0x7fe1fc026410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.359+0000 7fe203e00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fe1fc028940 0x7fe1fc026410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.359+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.359+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.359+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(19 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.359+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:20 17.32 17:4c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.360+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 5 ==== osd_op_reply(20 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.360+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:21 17.a 17:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.360+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 6 ==== osd_op_reply(21 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.360+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:22 17.2a 17:54000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.361+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(22 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.361+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 17.1a 17:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.361+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(23 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.361+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 17.3a 17:5c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.361+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(24 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.362+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fe1fc0153c0 0x7fe1fc0157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.362+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:25 17.6 17:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc0153c0 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.362+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fe1fc0153c0 0x7fe1fc0157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.363+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fe1fc0153c0 0x7fe1fc0157d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.363+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1fc0153c0 2024-09-17T03:56:28.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.363+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 17.26 17:64000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.363+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(26 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.363+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:27 17.16 17:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.364+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(27 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.364+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:28 17.36 17:6c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.365+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.365+0000 7fe209400640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fe2040bd040 0x7fe2040a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.365+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:29 17.e 17:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe2040bd040 2024-09-17T03:56:28.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.365+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fe2040bd040 0x7fe2040a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.365+0000 7fe208a00640 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fe2040bd040 0x7fe2040a68d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.366+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(29 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe2040bd040 2024-09-17T03:56:28.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.366+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:30 17.2e 17:74000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc0153c0 2024-09-17T03:56:28.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.366+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(30 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1fc0153c0 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.366+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:31 17.1e 17:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.367+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(31 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.367+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:32 17.3e 17:7c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.367+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(32 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.367+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:33 17.1 17:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(33 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:34 17.21 17:84000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(34 [pgnls start_epoch 113 out=49b] v107'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:35 17.11 17:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(35 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.368+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:36 17.31 17:8c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(36 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:37 17.9 17:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(37 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:38 17.29 17:94000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(38 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.369+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:39 17.19 17:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.370+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(39 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.370+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:40 17.39 17:9c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.370+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(40 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.370+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:41 17.5 17:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc0153c0 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.371+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(41 [pgnls start_epoch 113 out=49b] v107'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1fc0153c0 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.371+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:42 17.25 17:a4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.371+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(42 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.371+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:43 17.15 17:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.372+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(43 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.372+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:44 17.35 17:ac000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.372+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(44 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.372+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:45 17.d 17:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.373+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 7 ==== osd_op_reply(45 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.373+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:46 17.2d 17:b4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.373+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 8 ==== osd_op_reply(46 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.373+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:47 17.1d 17:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.374+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 9 ==== osd_op_reply(47 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.374+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:48 17.3d 17:bc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.374+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 10 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.374+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:49 17.3 17:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.375+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 6 ==== osd_op_reply(49 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.375+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:50 17.23 17:c4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.375+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 7 ==== osd_op_reply(50 [pgnls start_epoch 113 out=49b] v107'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.375+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:51 17.13 17:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc017410 2024-09-17T03:56:28.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.376+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(51 [pgnls start_epoch 113 out=49b] v107'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc017410 2024-09-17T03:56:28.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.376+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:52 17.33 17:cc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x55eddbb6e420 2024-09-17T03:56:28.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.376+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.7 v2:172.21.15.181:6826/580738384 11 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v107'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x55eddbb6e420 2024-09-17T03:56:28.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.376+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:53 17.b 17:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(53 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:54 17.2b 17:d4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 8 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:55 17.1b 17:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 9 ==== osd_op_reply(55 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.377+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:56 17.3b 17:dc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe2040bd040 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.378+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v107'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe2040bd040 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.378+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 17.7 17:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.378+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 12 ==== osd_op_reply(57 [pgnls start_epoch 113 out=49b] v113'12 uv12 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.378+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:58 17.27 17:e4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.379+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 10 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v107'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.379+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:59 17.17 17:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc0153c0 2024-09-17T03:56:28.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.379+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1fc0153c0 2024-09-17T03:56:28.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.379+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:60 17.37 17:ec000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f401e8b0 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.0 v2:172.21.15.129:6824/1655624368 13 ==== osd_op_reply(60 [pgnls start_epoch 113 out=49b] v107'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1f401e8b0 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:61 17.f 17:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f4010920 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v107'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f4010920 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:62 17.2f 17:f4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1f402d040 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 <== osd.4 v2:172.21.15.181:6802/2229307524 11 ==== osd_op_reply(62 [pgnls start_epoch 113 out=49b] v113'10 uv10 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1fc00ab10 con 0x7fe1f402d040 2024-09-17T03:56:28.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.380+0000 7fe208a00640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:63 17.1f 17:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc0153c0 2024-09-17T03:56:28.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.381+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v107'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe20400b040 con 0x7fe1fc0153c0 2024-09-17T03:56:28.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.381+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:64 17.3f 17:fc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55eddbafa160 con 0x7fe1fc028940 2024-09-17T03:56:28.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe203e00640 1 -- 172.21.15.129:0/2730065995 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v107'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fe1f400f5c0 con 0x7fe1fc028940 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fe1f401e8b0 msgr2=0x7fe1f401d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fe1f401e8b0 0x7fe1f401d330 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fe1f4010920 msgr2=0x7fe1f400d380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fe1f4010920 0x7fe1f400d380 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fe2040bd040 msgr2=0x7fe2040a68d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fe2040bd040 0x7fe2040a68d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fe1fc017410 msgr2=0x7fe1fc017820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fe1fc017410 0x7fe1fc017820 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fe1f402d040 msgr2=0x7fe1f4021420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fe1f402d040 0x7fe1f4021420 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fe1fc028940 msgr2=0x7fe1fc026410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fe1fc028940 0x7fe1fc026410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fe1fc0153c0 msgr2=0x7fe1fc0157d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fe1fc0153c0 0x7fe1fc0157d0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x55eddbb6e420 msgr2=0x55eddbb6e830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x55eddbb6e420 0x55eddbb6e830 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 reap_dead start 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 reap_dead start 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 reap_dead start 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe209400640 1 -- 172.21.15.129:0/2730065995 reap_dead start 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 msgr2=0x7fe1e8079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 0x7fe1e8079ab0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc00d650 tx=0x7fe1fc004880 comp rx=0 tx=0).stop 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 msgr2=0x55eddbb89ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe2040045b0 tx=0x7fe20404daa0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 shutdown_connections 2024-09-17T03:56:28.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1e80775a0 0x7fe1e8079ab0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 --2- 172.21.15.129:0/2730065995 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55eddbb86440 0x55eddbb89ad0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 >> 172.21.15.129:0/2730065995 conn(0x55eddb9d1ac0 msgr2=0x55eddbb85930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:28.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.382+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 shutdown_connections 2024-09-17T03:56:28.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.383+0000 7fe20f861f40 1 -- 172.21.15.129:0/2730065995 wait complete. 2024-09-17T03:56:28.393 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-09-17T03:56:28.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 -- 172.21.15.129:0/1292913637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 msgr2=0x7efd780cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 --2- 172.21.15.129:0/1292913637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780cd1c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7efd780d1370 tx=0x7efd7002aa10 comp rx=0 tx=0).stop 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 -- 172.21.15.129:0/1292913637 shutdown_connections 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 --2- 172.21.15.129:0/1292913637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780cd1c0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 --2- 172.21.15.129:0/1292913637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd7806eed0 0x7efd780ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 --2- 172.21.15.129:0/1292913637 >> v2:172.21.15.181:3300/0 conn(0x7efd780705a0 0x7efd780b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 -- 172.21.15.129:0/1292913637 >> 172.21.15.129:0/1292913637 conn(0x7efd7806d410 msgr2=0x7efd780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:28.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 -- 172.21.15.129:0/1292913637 shutdown_connections 2024-09-17T03:56:28.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.612+0000 7efd7f600640 1 -- 172.21.15.129:0/1292913637 wait complete. 2024-09-17T03:56:28.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.614+0000 7efd7f600640 1 Processor -- start 2024-09-17T03:56:28.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.614+0000 7efd7f600640 1 -- start start 2024-09-17T03:56:28.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7806eed0 0x7efd781f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd780705a0 0x7efd780c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efd78160b40 con 0x7efd7806f2e0 2024-09-17T03:56:28.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efd781609c0 con 0x7efd7806eed0 2024-09-17T03:56:28.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7f600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efd78160cc0 con 0x7efd780705a0 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59610/0 (socket says 172.21.15.129:59610) 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 -- 172.21.15.129:0/3698192978 learned_addr learned my addr 172.21.15.129:0/3698192978 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 -- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd780705a0 msgr2=0x7efd780c59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd780705a0 0x7efd780c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd7cc00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7806eed0 0x7efd781f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 -- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7806eed0 msgr2=0x7efd781f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.616+0000 7efd6fe00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7806eed0 0x7efd781f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd6fe00640 1 -- 172.21.15.129:0/3698192978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd781e6140 con 0x7efd7806f2e0 2024-09-17T03:56:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd6fe00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7efd6000d690 tx=0x7efd6000db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60017070 con 0x7efd7806f2e0 2024-09-17T03:56:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efd60005220 con 0x7efd7806f2e0 2024-09-17T03:56:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd781f9e30 con 0x7efd7806f2e0 2024-09-17T03:56:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600097c0 con 0x7efd7806f2e0 2024-09-17T03:56:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.617+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd781f9870 con 0x7efd7806f2e0 2024-09-17T03:56:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.618+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd78161220 con 0x7efd7806f2e0 2024-09-17T03:56:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.619+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efd60009960 con 0x7efd7806f2e0 2024-09-17T03:56:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.621+0000 7efd75600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 0x7efd5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.621+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7efd6009ae70 con 0x7efd7806f2e0 2024-09-17T03:56:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.621+0000 7efd7cc00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 0x7efd5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:28.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.622+0000 7efd7cc00640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 0x7efd5407b0a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efd68005d80 tx=0x7efd68005c70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:28.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.626+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efd60063210 con 0x7efd7806f2e0 2024-09-17T03:56:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.847+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd781f9320 con 0x7efd7806f2e0 2024-09-17T03:56:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.847+0000 7efd75600640 1 -- 172.21.15.129:0/3698192978 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7efd60062b10 con 0x7efd7806f2e0 2024-09-17T03:56:28.847 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:28.848 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:28.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 msgr2=0x7efd5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 0x7efd5407b0a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efd68005d80 tx=0x7efd68005c70 comp rx=0 tx=0).stop 2024-09-17T03:56:28.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 msgr2=0x7efd780ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:28.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7efd6000d690 tx=0x7efd6000db50 comp rx=0 tx=0).stop 2024-09-17T03:56:28.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 shutdown_connections 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efd54078b90 0x7efd5407b0a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efd780705a0 0x7efd780c59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efd7806f2e0 0x7efd780ae7e0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 --2- 172.21.15.129:0/3698192978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efd7806eed0 0x7efd781f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 >> 172.21.15.129:0/3698192978 conn(0x7efd7806d410 msgr2=0x7efd780cb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 shutdown_connections 2024-09-17T03:56:28.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:28.850+0000 7efd7f600640 1 -- 172.21.15.129:0/3698192978 wait complete. 2024-09-17T03:56:28.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:56:29.115 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":3}} 2024-09-17T03:56:29.115 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 3 } } 2024-09-17T03:56:29.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:29.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 -- 172.21.15.129:0/558526499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b3b90 msgr2=0x7f44040cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 --2- 172.21.15.129:0/558526499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b3b90 0x7f44040cbbf0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005fd0 tx=0x7f43f002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 -- 172.21.15.129:0/558526499 shutdown_connections 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 --2- 172.21.15.129:0/558526499 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040b2950 0x7f440406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 --2- 172.21.15.129:0/558526499 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b3b90 0x7f44040cbbf0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.334+0000 7f4408c00640 1 -- 172.21.15.129:0/558526499 >> 172.21.15.129:0/558526499 conn(0x7f440406d410 msgr2=0x7f44040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.335+0000 7f4408c00640 1 -- 172.21.15.129:0/558526499 shutdown_connections 2024-09-17T03:56:29.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.335+0000 7f4408c00640 1 -- 172.21.15.129:0/558526499 wait complete. 2024-09-17T03:56:29.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.337+0000 7f4408c00640 1 Processor -- start 2024-09-17T03:56:29.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.337+0000 7f4408c00640 1 -- start start 2024-09-17T03:56:29.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040ce170 0x7f44040c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44040c9600 0x7f44040c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f440406f410 con 0x7f44040b2950 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4404070910 con 0x7f44040c9600 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.338+0000 7f4408c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f440406f590 con 0x7f44040ce170 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:29.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59630/0 (socket says 172.21.15.129:59630) 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 -- 172.21.15.129:0/4074375785 learned_addr learned my addr 172.21.15.129:0/4074375785 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4402800640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44040c9600 0x7f44040c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401400640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040ce170 0x7f44040c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 -- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040ce170 msgr2=0x7f44040c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040ce170 0x7f44040c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 -- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44040c9600 msgr2=0x7f44040c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44040c9600 0x7f44040c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 -- 172.21.15.129:0/4074375785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440415c530 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.339+0000 7f4401e00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c3d0 tx=0x7f43f400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.340+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4017070 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.340+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43f40053c0 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.340+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4003850 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.340+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440415c810 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.340+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440415cdd0 con 0x7f44040b2950 2024-09-17T03:56:29.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.341+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440406fa40 con 0x7f44040b2950 2024-09-17T03:56:29.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.342+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f43f401a070 con 0x7f44040b2950 2024-09-17T03:56:29.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.343+0000 7f43fa000640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 0x7f43e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:29.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.343+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f43f409a0a0 con 0x7f44040b2950 2024-09-17T03:56:29.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.343+0000 7f4401400640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 0x7f43e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:29.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.344+0000 7f4401400640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 0x7f43e407b0d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f43f0033000 tx=0x7f43f0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:29.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.347+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f43f4061df0 con 0x7f44040b2950 2024-09-17T03:56:29.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.564+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f440406a140 con 0x7f44040b2950 2024-09-17T03:56:29.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.564+0000 7f43fa000640 1 -- 172.21.15.129:0/4074375785 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f43f4066820 con 0x7f44040b2950 2024-09-17T03:56:29.565 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:29.565 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 msgr2=0x7f43e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 0x7f43e407b0d0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f43f0033000 tx=0x7f43f0004730 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 msgr2=0x7f4404158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c3d0 tx=0x7f43f400c890 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 shutdown_connections 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43e4078bc0 0x7f43e407b0d0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44040c9600 0x7f44040c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f44040ce170 0x7f44040c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 --2- 172.21.15.129:0/4074375785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44040b2950 0x7f4404158640 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 >> 172.21.15.129:0/4074375785 conn(0x7f440406d410 msgr2=0x7f4404073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 shutdown_connections 2024-09-17T03:56:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:29.567+0000 7f4408c00640 1 -- 172.21.15.129:0/4074375785 wait complete. 2024-09-17T03:56:29.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:56:29.832 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-09-17T03:56:29.832 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-09-17T03:56:29.832 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:56:29.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:56:29.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:56:30.020 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:56:30.020 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:56:30.020 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:30.021 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:30.056 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:30.057 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:30.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:56:30.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 -- 172.21.15.129:0/3716489283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80b4140 msgr2=0x7f15b80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 --2- 172.21.15.129:0/3716489283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80b4140 0x7f15b80ccc80 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009830 tx=0x7f15a402a510 comp rx=0 tx=0).stop 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 -- 172.21.15.129:0/3716489283 shutdown_connections 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 --2- 172.21.15.129:0/3716489283 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 0x7f15b81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 --2- 172.21.15.129:0/3716489283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80b4140 0x7f15b80ccc80 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 -- 172.21.15.129:0/3716489283 >> 172.21.15.129:0/3716489283 conn(0x7f15b806d410 msgr2=0x7f15b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 -- 172.21.15.129:0/3716489283 shutdown_connections 2024-09-17T03:56:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.395+0000 7f15bf200640 1 -- 172.21.15.129:0/3716489283 wait complete. 2024-09-17T03:56:30.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.397+0000 7f15bf200640 1 Processor -- start 2024-09-17T03:56:30.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.398+0000 7f15bf200640 1 -- start start 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80b4140 0x7f15b80d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 0x7f15b80d0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f15b8160260 con 0x7f15b80705a0 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f15b81600e0 con 0x7f15b80b4140 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f15b8160f00 con 0x7f15b80cd1c0 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15b7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15b7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59656/0 (socket says 172.21.15.129:59656) 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15b7e00640 1 -- 172.21.15.129:0/2251291202 learned_addr learned my addr 172.21.15.129:0/2251291202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15bd200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 0x7f15b80d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.399+0000 7f15b7400640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80b4140 0x7f15b80d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 -- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 msgr2=0x7f15b80d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 0x7f15b80d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 -- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80b4140 msgr2=0x7f15b80d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80b4140 0x7f15b80d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 -- 172.21.15.129:0/2251291202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b81f6b20 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15b7e00640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f15a800cbc0 tx=0x7f15a800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8016020 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15a800ad90 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a8012680 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.400+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b81f5fa0 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.401+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81f6560 con 0x7f15b80705a0 2024-09-17T03:56:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.401+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b8076760 con 0x7f15b80705a0 2024-09-17T03:56:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.403+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f15a8019020 con 0x7f15b80705a0 2024-09-17T03:56:30.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.404+0000 7f15afe00640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 0x7f159807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:30.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.404+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f15a809b060 con 0x7f15b80705a0 2024-09-17T03:56:30.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.404+0000 7f15b7400640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 0x7f159807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:30.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.405+0000 7f15b7400640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 0x7f159807b0d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f15a4034000 tx=0x7f15a4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.408+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f15b8076760 con 0x7f15b80705a0 2024-09-17T03:56:30.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.604+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f15b80ccc80 con 0x7f15b80705a0 2024-09-17T03:56:30.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.604+0000 7f15afe00640 1 -- 172.21.15.129:0/2251291202 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v113) ==== 80+0+122 (secure 0 0 0) 0x7f15a8020030 con 0x7f15b80705a0 2024-09-17T03:56:30.604 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6834/153768962 2024-09-18T03:55:27.495282+0000 2024-09-17T03:56:30.604 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6832/153768962 2024-09-18T03:55:27.495282+0000 2024-09-17T03:56:30.605 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 msgr2=0x7f159807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 0x7f159807b0d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f15a4034000 tx=0x7f15a4004730 comp rx=0 tx=0).stop 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 msgr2=0x7f15b81f5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f15a800cbc0 tx=0x7f15a800a060 comp rx=0 tx=0).stop 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 shutdown_connections 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1598078bc0 0x7f159807b0d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15b80cd1c0 0x7f15b80d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15b80b4140 0x7f15b80d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 --2- 172.21.15.129:0/2251291202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15b80705a0 0x7f15b81f5420 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:30.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 >> 172.21.15.129:0/2251291202 conn(0x7f15b806d410 msgr2=0x7f15b80adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:30.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 shutdown_connections 2024-09-17T03:56:30.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:30.607+0000 7f15bf200640 1 -- 172.21.15.129:0/2251291202 wait complete. 2024-09-17T03:56:30.629 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:56:30.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:30.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:56:30.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:56:30.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:30.692 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:56:30.728 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:56:30.729 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:56:30.729 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:30.767 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:56:30.767 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:56:30.767 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:30.767 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:30.806 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:56:30.806 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:56:30.806 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:56:30.807 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:30.844 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:56:30.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:31.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 -- 172.21.15.129:0/1900359286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 msgr2=0x7f41240b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 --2- 172.21.15.129:0/1900359286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240b3480 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f410c009830 tx=0x7f410c02a510 comp rx=0 tx=0).stop 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 -- 172.21.15.129:0/1900359286 shutdown_connections 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 --2- 172.21.15.129:0/1900359286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240b3480 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 --2- 172.21.15.129:0/1900359286 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240ce170 0x7f41240cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 -- 172.21.15.129:0/1900359286 >> 172.21.15.129:0/1900359286 conn(0x7f412406d410 msgr2=0x7f41240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 -- 172.21.15.129:0/1900359286 shutdown_connections 2024-09-17T03:56:31.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.068+0000 7f412b800640 1 -- 172.21.15.129:0/1900359286 wait complete. 2024-09-17T03:56:31.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.070+0000 7f412b800640 1 Processor -- start 2024-09-17T03:56:31.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.071+0000 7f412b800640 1 -- start start 2024-09-17T03:56:31.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 0x7f41241e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 0x7f41241e4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f412406f5e0 con 0x7f41240b2950 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f41240708e0 con 0x7f41240ce170 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f412b800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f412406f760 con 0x7f41240b39c0 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f4128e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f4128e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59688/0 (socket says 172.21.15.129:59688) 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.072+0000 7f4128e00640 1 -- 172.21.15.129:0/2684941819 learned_addr learned my addr 172.21.15.129:0/2684941819 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4129800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 0x7f41241e4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f411be00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 0x7f41241e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 -- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 msgr2=0x7f41241e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 0x7f41241e43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 -- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 msgr2=0x7f41241e4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 0x7f41241e4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 -- 172.21.15.129:0/2684941819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41241e5720 con 0x7f41240b2950 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f411be00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 0x7f41241e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4129800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 0x7f41241e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4128e00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f411400e7c0 tx=0x7f411400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411400ccc0 con 0x7f41240b2950 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4114010430 con 0x7f41240b2950 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4114010860 con 0x7f41240b2950 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.073+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412409d280 con 0x7f41240b2950 2024-09-17T03:56:31.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.074+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412409da00 con 0x7f41240b2950 2024-09-17T03:56:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.075+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41240b3480 con 0x7f41240b2950 2024-09-17T03:56:31.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.076+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4114010a00 con 0x7f41240b2950 2024-09-17T03:56:31.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.077+0000 7f4121600640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 0x7f410007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.077+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f411409b990 con 0x7f41240b2950 2024-09-17T03:56:31.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.077+0000 7f411be00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 0x7f410007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.078+0000 7f411be00640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 0x7f410007b0a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f410c034000 tx=0x7f410c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:31.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.081+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4114063c20 con 0x7f41240b2950 2024-09-17T03:56:31.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.299+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f412406a140 con 0x7f41240b2950 2024-09-17T03:56:31.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.300+0000 7f4121600640 1 -- 172.21.15.129:0/2684941819 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f4114063520 con 0x7f41240b2950 2024-09-17T03:56:31.301 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:31.301 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:31.302 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:31.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 msgr2=0x7f410007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 0x7f410007b0a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f410c034000 tx=0x7f410c004730 comp rx=0 tx=0).stop 2024-09-17T03:56:31.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 msgr2=0x7f41240c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f411400e7c0 tx=0x7f411400ec80 comp rx=0 tx=0).stop 2024-09-17T03:56:31.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 shutdown_connections 2024-09-17T03:56:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4100078b90 0x7f410007b0a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41240ce170 0x7f41241e4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41240b39c0 0x7f41241e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 --2- 172.21.15.129:0/2684941819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41240b2950 0x7f41240c8e50 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 >> 172.21.15.129:0/2684941819 conn(0x7f412406d410 msgr2=0x7f41240741d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:31.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 shutdown_connections 2024-09-17T03:56:31.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.303+0000 7f412b800640 1 -- 172.21.15.129:0/2684941819 wait complete. 2024-09-17T03:56:31.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 -- 172.21.15.129:0/2005041189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 msgr2=0x7fefd8165bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 --2- 172.21.15.129:0/2005041189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd8165bd0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005fd0 tx=0x7fefd002bc80 comp rx=0 tx=0).stop 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 -- 172.21.15.129:0/2005041189 shutdown_connections 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 --2- 172.21.15.129:0/2005041189 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd8165bd0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 -- 172.21.15.129:0/2005041189 >> 172.21.15.129:0/2005041189 conn(0x7fefd806d410 msgr2=0x7fefd80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 -- 172.21.15.129:0/2005041189 shutdown_connections 2024-09-17T03:56:31.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.545+0000 7fefdd800640 1 -- 172.21.15.129:0/2005041189 wait complete. 2024-09-17T03:56:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.548+0000 7fefdd800640 1 Processor -- start 2024-09-17T03:56:31.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.548+0000 7fefdd800640 1 -- start start 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 0x7fefd80c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 0x7fefd8162b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fefd80710e0 con 0x7fefd81637b0 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8070f60 con 0x7fefd80ccc80 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefdd800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fefd8166110 con 0x7fefd80af4c0 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefd7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefd7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59694/0 (socket says 172.21.15.129:59694) 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.549+0000 7fefd7400640 1 -- 172.21.15.129:0/2399698880 learned_addr learned my addr 172.21.15.129:0/2399698880 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd6a00640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 0x7fefd80c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd6000640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 0x7fefd8162b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 -- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 msgr2=0x7fefd80c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 0x7fefd80c5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 -- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 msgr2=0x7fefd8162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 0x7fefd8162b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 -- 172.21.15.129:0/2399698880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd8156840 con 0x7fefd81637b0 2024-09-17T03:56:31.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd6a00640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 0x7fefd80c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:31.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd6000640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 0x7fefd8162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefd7400640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005fd0 tx=0x7fefd00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd0041020 con 0x7fefd81637b0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefd003d530 con 0x7fefd81637b0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd8156b20 con 0x7fefd81637b0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.550+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd003d960 con 0x7fefd81637b0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.551+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd80ae0e0 con 0x7fefd81637b0 2024-09-17T03:56:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.552+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd815ae00 con 0x7fefd81637b0 2024-09-17T03:56:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.553+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fefd0044020 con 0x7fefd81637b0 2024-09-17T03:56:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.554+0000 7fefcea00640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 0x7fefb807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.554+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7fefd00bc8e0 con 0x7fefd81637b0 2024-09-17T03:56:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.554+0000 7fefd6a00640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 0x7fefb807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:31.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.555+0000 7fefd6a00640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 0x7fefb807b0d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fefc8007fd0 tx=0x7fefc8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:31.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.558+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fefd004b030 con 0x7fefd81637b0 2024-09-17T03:56:31.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.778+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefd81fa3f0 con 0x7fefd81637b0 2024-09-17T03:56:31.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.778+0000 7fefcea00640 1 -- 172.21.15.129:0/2399698880 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7fefd0084c80 con 0x7fefd81637b0 2024-09-17T03:56:31.778 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:31.778 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 msgr2=0x7fefb807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 0x7fefb807b0d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fefc8007fd0 tx=0x7fefc8009040 comp rx=0 tx=0).stop 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 msgr2=0x7fefd81f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fefd0005fd0 tx=0x7fefd00042f0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 shutdown_connections 2024-09-17T03:56:31.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fefb8078bc0 0x7fefb807b0d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fefd81637b0 0x7fefd81f5490 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fefd80ccc80 0x7fefd8162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 --2- 172.21.15.129:0/2399698880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fefd80af4c0 0x7fefd80c5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 >> 172.21.15.129:0/2399698880 conn(0x7fefd806d410 msgr2=0x7fefd815c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 shutdown_connections 2024-09-17T03:56:31.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:31.781+0000 7fefdd800640 1 -- 172.21.15.129:0/2399698880 wait complete. 2024-09-17T03:56:31.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:32.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 -- 172.21.15.129:0/7262556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 msgr2=0x7fc364004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 --2- 172.21.15.129:0/7262556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc364004800 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fc3800ca870 tx=0x7fc364005830 comp rx=0 tx=0).stop 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc385000640 1 -- 172.21.15.129:0/7262556 reap_dead start 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 -- 172.21.15.129:0/7262556 shutdown_connections 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 -- 172.21.15.129:0/7262556 >> 172.21.15.129:0/7262556 conn(0x7fc38006d410 msgr2=0x7fc3800b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 -- 172.21.15.129:0/7262556 shutdown_connections 2024-09-17T03:56:32.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.026+0000 7fc387000640 1 -- 172.21.15.129:0/7262556 wait complete. 2024-09-17T03:56:32.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.029+0000 7fc387000640 1 Processor -- start 2024-09-17T03:56:32.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.029+0000 7fc387000640 1 -- start start 2024-09-17T03:56:32.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 0x7fc3800b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3801f5c20 0x7fc3801f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3800b4140 con 0x7fc364003fd0 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc38006f4f0 con 0x7fc3801f5c20 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc387000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3800b42c0 con 0x7fc3800b0ed0 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc37fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc37fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59722/0 (socket says 172.21.15.129:59722) 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.030+0000 7fc37fe00640 1 -- 172.21.15.129:0/308100097 learned_addr learned my addr 172.21.15.129:0/308100097 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc385000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3801f5c20 0x7fc3801f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37f400640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 0x7fc3800b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 -- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 msgr2=0x7fc3800b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 0x7fc3800b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 -- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3801f5c20 msgr2=0x7fc3801f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3801f5c20 0x7fc3801f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 -- 172.21.15.129:0/308100097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3800b1820 con 0x7fc364003fd0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37f400640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 0x7fc3800b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc37fe00640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc370011a80 tx=0x7fc370011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37000fd50 con 0x7fc364003fd0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc370016070 con 0x7fc364003fd0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370012690 con 0x7fc364003fd0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.031+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3800c74a0 con 0x7fc364003fd0 2024-09-17T03:56:32.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.032+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3800c7a40 con 0x7fc364003fd0 2024-09-17T03:56:32.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.033+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc38015f060 con 0x7fc364003fd0 2024-09-17T03:56:32.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.034+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc370018030 con 0x7fc364003fd0 2024-09-17T03:56:32.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.035+0000 7fc377e00640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 0x7fc36007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.035+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7fc37009b610 con 0x7fc364003fd0 2024-09-17T03:56:32.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.035+0000 7fc37f400640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 0x7fc36007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.036+0000 7fc37f400640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 0x7fc36007b0a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc36c033000 tx=0x7fc36c001aa0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.039+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc38015f060 con 0x7fc364003fd0 2024-09-17T03:56:32.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.225+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3800ccc80 con 0x7fc364003fd0 2024-09-17T03:56:32.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.226+0000 7fc377e00640 1 -- 172.21.15.129:0/308100097 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17018 (secure 0 0 0) 0x7fc3800b1820 con 0x7fc364003fd0 2024-09-17T03:56:32.226 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:32.226 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":113,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:17.647512+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:55:07.469103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-17T03:55:08.543736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.25","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/153768962":"2024-09-18T03:55:27.495282+0000","172.21.15.129:6832/153768962":"2024-09-18T03:55:27.495282+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:32.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 msgr2=0x7fc36007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 0x7fc36007b0a0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fc36c033000 tx=0x7fc36c001aa0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 msgr2=0x7fc3800cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fc370011a80 tx=0x7fc370011f40 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 shutdown_connections 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc360078b90 0x7fc36007b0a0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3801f5c20 0x7fc3801f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3800b0ed0 0x7fc3800b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 --2- 172.21.15.129:0/308100097 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc364003fd0 0x7fc3800cb120 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 >> 172.21.15.129:0/308100097 conn(0x7fc38006d410 msgr2=0x7fc380072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 shutdown_connections 2024-09-17T03:56:32.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.228+0000 7fc387000640 1 -- 172.21.15.129:0/308100097 wait complete. 2024-09-17T03:56:32.250 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:56:32.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 -- 172.21.15.129:0/3545999846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 msgr2=0x7f2e780ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 --2- 172.21.15.129:0/3545999846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780ccd20 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2e78072920 tx=0x7f2e7002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 -- 172.21.15.129:0/3545999846 shutdown_connections 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 --2- 172.21.15.129:0/3545999846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780ccd20 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 --2- 172.21.15.129:0/3545999846 >> v2:172.21.15.181:3300/0 conn(0x7f2e78070a50 0x7f2e780cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 -- 172.21.15.129:0/3545999846 >> 172.21.15.129:0/3545999846 conn(0x7f2e7806d410 msgr2=0x7f2e780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.470+0000 7f2e7d800640 1 -- 172.21.15.129:0/3545999846 shutdown_connections 2024-09-17T03:56:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.471+0000 7f2e7d800640 1 -- 172.21.15.129:0/3545999846 wait complete. 2024-09-17T03:56:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.473+0000 7f2e7d800640 1 Processor -- start 2024-09-17T03:56:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.473+0000 7f2e7d800640 1 -- start start 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e78070a50 0x7f2e780ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e780cc3e0 0x7f2e781530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2e780cc1c0 con 0x7f2e780cc910 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2e780cc040 con 0x7f2e780cc3e0 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.474+0000 7f2e7d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2e780af070 con 0x7f2e78070a50 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59740/0 (socket says 172.21.15.129:59740) 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 -- 172.21.15.129:0/2352623378 learned_addr learned my addr 172.21.15.129:0/2352623378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e76a00640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e78070a50 0x7f2e780ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e76000640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e780cc3e0 0x7f2e781530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 -- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e78070a50 msgr2=0x7f2e780ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e78070a50 0x7f2e780ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 -- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e780cc3e0 msgr2=0x7f2e781530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e780cc3e0 0x7f2e781530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 -- 172.21.15.129:0/2352623378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7815dfb0 con 0x7f2e780cc910 2024-09-17T03:56:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.475+0000 7f2e77400640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f2e70005fd0 tx=0x7f2e700316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.476+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70031990 con 0x7f2e780cc910 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.476+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e70031c20 con 0x7f2e780cc910 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.476+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7815e290 con 0x7f2e780cc910 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.476+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e700330d0 con 0x7f2e780cc910 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.476+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e780b0ec0 con 0x7f2e780cc910 2024-09-17T03:56:32.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.477+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e780c70f0 con 0x7f2e780cc910 2024-09-17T03:56:32.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.478+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2e70033270 con 0x7f2e780cc910 2024-09-17T03:56:32.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.479+0000 7f2e6ea00640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 0x7f2e5807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.479+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f2e700bc080 con 0x7f2e780cc910 2024-09-17T03:56:32.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.479+0000 7f2e76a00640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 0x7f2e5807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.480+0000 7f2e76a00640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 0x7f2e5807b0d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009b30 tx=0x7f2e68005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.484+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2e780c70f0 con 0x7f2e780cc910 2024-09-17T03:56:32.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.706+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e780c7720 con 0x7f2e780cc910 2024-09-17T03:56:32.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.708+0000 7f2e6ea00640 1 -- 172.21.15.129:0/2352623378 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f2e7003f030 con 0x7f2e780cc910 2024-09-17T03:56:32.708 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:32.708 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:32.708 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 msgr2=0x7f2e5807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 0x7f2e5807b0d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009b30 tx=0x7f2e68005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 msgr2=0x7f2e780c80b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f2e70005fd0 tx=0x7f2e700316d0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 shutdown_connections 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e58078bc0 0x7f2e5807b0d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e780cc910 0x7f2e780c80b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2e780cc3e0 0x7f2e781530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 --2- 172.21.15.129:0/2352623378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2e78070a50 0x7f2e780ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 >> 172.21.15.129:0/2352623378 conn(0x7f2e7806d410 msgr2=0x7f2e780caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:32.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 shutdown_connections 2024-09-17T03:56:32.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.711+0000 7f2e7d800640 1 -- 172.21.15.129:0/2352623378 wait complete. 2024-09-17T03:56:32.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:32.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 -- 172.21.15.129:0/4262759480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 msgr2=0x7f986c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 --2- 172.21.15.129:0/4262759480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c0b4140 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f985c005fd0 tx=0x7f985c02a940 comp rx=0 tx=0).stop 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 -- 172.21.15.129:0/4262759480 shutdown_connections 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 --2- 172.21.15.129:0/4262759480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c0b4140 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 -- 172.21.15.129:0/4262759480 >> 172.21.15.129:0/4262759480 conn(0x7f986c06d410 msgr2=0x7f986c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 -- 172.21.15.129:0/4262759480 shutdown_connections 2024-09-17T03:56:32.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.956+0000 7f9872600640 1 -- 172.21.15.129:0/4262759480 wait complete. 2024-09-17T03:56:32.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.958+0000 7f9872600640 1 Processor -- start 2024-09-17T03:56:32.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.959+0000 7f9872600640 1 -- start start 2024-09-17T03:56:32.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f986c06eed0 0x7f986c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 0x7f986c09cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f986c06f2e0 con 0x7f986c0705a0 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f986c1612e0 con 0x7f986c15be50 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f9872600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f986c06f460 con 0x7f986c06eed0 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f986aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f986aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59748/0 (socket says 172.21.15.129:59748) 2024-09-17T03:56:32.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.960+0000 7f986aa00640 1 -- 172.21.15.129:0/2128698474 learned_addr learned my addr 172.21.15.129:0/2128698474 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986b400640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f986c06eed0 0x7f986c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986be00640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 0x7f986c09cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 -- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f986c06eed0 msgr2=0x7f986c15b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f986c06eed0 0x7f986c15b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 -- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 msgr2=0x7f986c09cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 0x7f986c09cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 -- 172.21.15.129:0/2128698474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986c09c120 con 0x7f986c0705a0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986be00640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 0x7f986c09cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f986aa00640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f985800c340 tx=0x7f985800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858017070 con 0x7f986c0705a0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98580035f0 con 0x7f986c0705a0 2024-09-17T03:56:32.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858003a80 con 0x7f986c0705a0 2024-09-17T03:56:32.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986c1e74a0 con 0x7f986c0705a0 2024-09-17T03:56:32.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.961+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f986c1e7a60 con 0x7f986c0705a0 2024-09-17T03:56:32.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.962+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f986c0b4140 con 0x7f986c0705a0 2024-09-17T03:56:32.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.964+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f985801a070 con 0x7f986c0705a0 2024-09-17T03:56:32.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.965+0000 7f9863400640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 0x7f984c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:32.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.965+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f9858099fe0 con 0x7f986c0705a0 2024-09-17T03:56:32.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.965+0000 7f986b400640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 0x7f984c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:32.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.966+0000 7f986b400640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 0x7f984c07b0a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f985c034000 tx=0x7f985c004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:32.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:32.969+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9858062430 con 0x7f986c0705a0 2024-09-17T03:56:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.184+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f986c06a140 con 0x7f986c0705a0 2024-09-17T03:56:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.184+0000 7f9863400640 1 -- 172.21.15.129:0/2128698474 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+5239 (secure 0 0 0) 0x7f9858061d30 con 0x7f986c0705a0 2024-09-17T03:56:33.185 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:33.185 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":87,"btime":"2024-09-17T03:55:31:521125+0000","default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:55:09.589645+0000","modified":"2024-09-17T03:55:31.521116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":111,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7257":{"gid":7257,"name":"c","rank":0,"incarnation":84,"state":"up:active","state_seq":4,"addr":"172.21.15.129:6835/1953968590","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1953968590},{"type":"v1","addr":"172.21.15.129:6835","nonce":1953968590}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7257,"qdb_cluster":[7257]},"id":8}]} 2024-09-17T03:56:33.185 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 87 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 msgr2=0x7f984c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 0x7f984c07b0a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f985c034000 tx=0x7f985c004ab0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 msgr2=0x7f986c09c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f985800c340 tx=0x7f985800c800 comp rx=0 tx=0).stop 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 shutdown_connections 2024-09-17T03:56:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f984c078b90 0x7f984c07b0a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f986c15be50 0x7f986c09cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f986c0705a0 0x7f986c09c5d0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 --2- 172.21.15.129:0/2128698474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f986c06eed0 0x7f986c15b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 >> 172.21.15.129:0/2128698474 conn(0x7f986c06d410 msgr2=0x7f986c15e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 shutdown_connections 2024-09-17T03:56:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.187+0000 7f9872600640 1 -- 172.21.15.129:0/2128698474 wait complete. 2024-09-17T03:56:33.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 -- 172.21.15.129:0/1954706570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00b4140 msgr2=0x7f23f00ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 --2- 172.21.15.129:0/1954706570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00b4140 0x7f23f00ccc80 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f23dc005fd0 tx=0x7f23dc02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 -- 172.21.15.129:0/1954706570 shutdown_connections 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 --2- 172.21.15.129:0/1954706570 >> v2:172.21.15.181:3300/0 conn(0x7f23f00cd1c0 0x7f23f01609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 --2- 172.21.15.129:0/1954706570 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00b4140 0x7f23f00ccc80 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 -- 172.21.15.129:0/1954706570 >> 172.21.15.129:0/1954706570 conn(0x7f23f006d410 msgr2=0x7f23f00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 -- 172.21.15.129:0/1954706570 shutdown_connections 2024-09-17T03:56:33.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.433+0000 7f23f6e00640 1 -- 172.21.15.129:0/1954706570 wait complete. 2024-09-17T03:56:33.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.436+0000 7f23f6e00640 1 Processor -- start 2024-09-17T03:56:33.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.436+0000 7f23f6e00640 1 -- start start 2024-09-17T03:56:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00705a0 0x7f23f0155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00cd1c0 0x7f23f01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23f0160260 con 0x7f23f00b4140 2024-09-17T03:56:33.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f23f01600e0 con 0x7f23f00cd1c0 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.437+0000 7f23f6e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f23f0160f00 con 0x7f23f00705a0 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59754/0 (socket says 172.21.15.129:59754) 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 -- 172.21.15.129:0/617441153 learned_addr learned my addr 172.21.15.129:0/617441153 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23efe00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00705a0 0x7f23f0155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23f4e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00cd1c0 0x7f23f01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 -- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00705a0 msgr2=0x7f23f0155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00705a0 0x7f23f0155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 -- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00cd1c0 msgr2=0x7f23f01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00cd1c0 0x7f23f01593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 -- 172.21.15.129:0/617441153 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f015c8c0 con 0x7f23f00b4140 2024-09-17T03:56:33.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.438+0000 7f23ef400640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f23dc034040 tx=0x7f23dc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.439+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc03e030 con 0x7f23f00b4140 2024-09-17T03:56:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.439+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23dc0039f0 con 0x7f23f00b4140 2024-09-17T03:56:33.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.439+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f015d380 con 0x7f23f00b4140 2024-09-17T03:56:33.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.439+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc035040 con 0x7f23f00b4140 2024-09-17T03:56:33.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.439+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f015cb60 con 0x7f23f00b4140 2024-09-17T03:56:33.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.440+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f015d0f0 con 0x7f23f00b4140 2024-09-17T03:56:33.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.441+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f23dc041030 con 0x7f23f00b4140 2024-09-17T03:56:33.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.442+0000 7f23e7e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 0x7f23d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.442+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f23dc0bb1e0 con 0x7f23f00b4140 2024-09-17T03:56:33.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.443+0000 7f23efe00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 0x7f23d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.443+0000 7f23efe00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 0x7f23d007b0d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f23e00017e0 tx=0x7f23e0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:33.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.446+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f23dc082f30 con 0x7f23f00b4140 2024-09-17T03:56:33.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.630+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23f00ccc80 con 0x7f23f00b4140 2024-09-17T03:56:33.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.632+0000 7f23e7e00640 1 -- 172.21.15.129:0/617441153 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17018 (secure 0 0 0) 0x7f23dc087960 con 0x7f23f00b4140 2024-09-17T03:56:33.632 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:33.632 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":113,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:17.647512+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:55:07.469103+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-17T03:55:08.543736+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.25","mappings":[{"from":0,"to":2}]},{"pgid":"17.3f","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/153768962":"2024-09-18T03:55:27.495282+0000","172.21.15.129:6832/153768962":"2024-09-18T03:55:27.495282+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 msgr2=0x7f23d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 0x7f23d007b0d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f23e00017e0 tx=0x7f23e0009040 comp rx=0 tx=0).stop 2024-09-17T03:56:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 msgr2=0x7f23f0158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f23dc034040 tx=0x7f23dc005e80 comp rx=0 tx=0).stop 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 shutdown_connections 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23d0078bc0 0x7f23d007b0d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00cd1c0 0x7f23f01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23f00b4140 0x7f23f0158e70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 --2- 172.21.15.129:0/617441153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23f00705a0 0x7f23f0155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 >> 172.21.15.129:0/617441153 conn(0x7f23f006d410 msgr2=0x7f23f0072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 shutdown_connections 2024-09-17T03:56:33.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.634+0000 7f23f6e00640 1 -- 172.21.15.129:0/617441153 wait complete. 2024-09-17T03:56:33.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.883+0000 7f4df6200640 1 -- 172.21.15.129:0/3070096130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 msgr2=0x7f4dd8003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.883+0000 7f4df6200640 1 --2- 172.21.15.129:0/3070096130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4dd8003ee0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c470 tx=0x7f4dd8005830 comp rx=0 tx=0).stop 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.883+0000 7f4df5800640 1 -- 172.21.15.129:0/3070096130 reap_dead start 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.884+0000 7f4df6200640 1 -- 172.21.15.129:0/3070096130 shutdown_connections 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.884+0000 7f4df6200640 1 -- 172.21.15.129:0/3070096130 >> 172.21.15.129:0/3070096130 conn(0x7f4df006cfc0 msgr2=0x7f4df006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.884+0000 7f4df6200640 1 -- 172.21.15.129:0/3070096130 shutdown_connections 2024-09-17T03:56:33.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.884+0000 7f4df6200640 1 -- 172.21.15.129:0/3070096130 wait complete. 2024-09-17T03:56:33.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.886+0000 7f4df6200640 1 Processor -- start 2024-09-17T03:56:33.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.886+0000 7f4df6200640 1 -- start start 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df01f6cf0 0x7f4df01f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4df015df70 0x7f4df015e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00d0440 con 0x7f4dd8003a70 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00d02c0 con 0x7f4df01f6cf0 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.887+0000 7f4df6200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4df00d05c0 con 0x7f4df015df70 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44726/0 (socket says 172.21.15.129:44726) 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 -- 172.21.15.129:0/527456655 learned_addr learned my addr 172.21.15.129:0/527456655 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4defe00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df01f6cf0 0x7f4df01f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df5800640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4df015df70 0x7f4df015e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 -- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4df015df70 msgr2=0x7f4df015e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4df015df70 0x7f4df015e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 -- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df01f6cf0 msgr2=0x7f4df01f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df01f6cf0 0x7f4df01f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:33.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 -- 172.21.15.129:0/527456655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df015e8c0 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.888+0000 7f4df4e00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c4d0 tx=0x7f4de4006290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.889+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4042070 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.889+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4de40038e0 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.889+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df01f32e0 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.889+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4039070 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.889+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df01f3860 con 0x7f4dd8003a70 2024-09-17T03:56:33.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.890+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df015cbd0 con 0x7f4dd8003a70 2024-09-17T03:56:33.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.891+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4de4045070 con 0x7f4dd8003a70 2024-09-17T03:56:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.892+0000 7f4decc00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 0x7f4dd007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.892+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5853+0+0 (secure 0 0 0) 0x7f4de40ba7b0 con 0x7f4dd8003a70 2024-09-17T03:56:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.893+0000 7f4defe00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 0x7f4dd007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:33.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.894+0000 7f4defe00640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 0x7f4dd007b0a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4df00af8a0 tx=0x7f4ddc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:33.897+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4df015e8c0 con 0x7f4dd8003a70 2024-09-17T03:56:34.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.115+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f4df006a140 con 0x7f4dd8003a70 2024-09-17T03:56:34.115 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:56:34.116+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:56:34.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.118+0000 7f4decc00640 1 -- 172.21.15.129:0/527456655 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v88) ==== 161+0+0 (secure 0 0 0) 0x7f4df00d0c10 con 0x7f4dd8003a70 2024-09-17T03:56:34.118 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:56:34.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 msgr2=0x7f4dd007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:34.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 0x7f4dd007b0a0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4df00af8a0 tx=0x7f4ddc005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:34.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 msgr2=0x7f4df0158fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:34.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f4de400c4d0 tx=0x7f4de4006290 comp rx=0 tx=0).stop 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 shutdown_connections 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd0078b90 0x7f4dd007b0a0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4df015df70 0x7f4df015e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df01f6cf0 0x7f4df01f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 --2- 172.21.15.129:0/527456655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4dd8003a70 0x7f4df0158fd0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 >> 172.21.15.129:0/527456655 conn(0x7f4df006cfc0 msgr2=0x7f4df0075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:34.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.121+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 shutdown_connections 2024-09-17T03:56:34.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.122+0000 7f4df6200640 1 -- 172.21.15.129:0/527456655 wait complete. 2024-09-17T03:56:34.134 INFO:tasks.ceph.mds.c.smithi129.stderr: -7776> 2024-09-17T03:55:34.514+0000 7f611ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:56:34.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:56:34.260 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:56:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 -- 172.21.15.129:0/1581165971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d400cdf10 msgr2=0x7f5d400cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 --2- 172.21.15.129:0/1581165971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d400cdf10 0x7f5d400cd5e0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f5d30005fd0 tx=0x7f5d3002a940 comp rx=0 tx=0).stop 2024-09-17T03:56:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 -- 172.21.15.129:0/1581165971 shutdown_connections 2024-09-17T03:56:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 --2- 172.21.15.129:0/1581165971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d400cdf10 0x7f5d400cd5e0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 -- 172.21.15.129:0/1581165971 >> 172.21.15.129:0/1581165971 conn(0x7f5d4006cfc0 msgr2=0x7f5d4006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 -- 172.21.15.129:0/1581165971 shutdown_connections 2024-09-17T03:56:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.378+0000 7f5d45000640 1 -- 172.21.15.129:0/1581165971 wait complete. 2024-09-17T03:56:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.380+0000 7f5d45000640 1 Processor -- start 2024-09-17T03:56:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.380+0000 7f5d45000640 1 -- start start 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d400b4ee0 0x7f5d4014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d400cdf10 0x7f5d4014a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4015d630 con 0x7f5d4014a730 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d400b5500 con 0x7f5d400b4ee0 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d45000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4015d7b0 con 0x7f5d400cdf10 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d3fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44774/0 (socket says 172.21.15.129:44774) 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d3f400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d400b4ee0 0x7f5d4014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d3fe00640 1 -- 172.21.15.129:0/3017696303 learned_addr learned my addr 172.21.15.129:0/3017696303 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.382+0000 7f5d3ea00640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d400cdf10 0x7f5d4014a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 -- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d400cdf10 msgr2=0x7f5d4014a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d400cdf10 0x7f5d4014a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 -- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d400b4ee0 msgr2=0x7f5d4014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d400b4ee0 0x7f5d4014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 -- 172.21.15.129:0/3017696303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d401f5cf0 con 0x7f5d4014a730 2024-09-17T03:56:34.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d3fe00640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c400 tx=0x7f5d3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38011790 con 0x7f5d4014a730 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d38017070 con 0x7f5d4014a730 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38011c80 con 0x7f5d4014a730 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d40155c20 con 0x7f5d4014a730 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.383+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d401561e0 con 0x7f5d4014a730 2024-09-17T03:56:34.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.385+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d400cd5e0 con 0x7f5d4014a730 2024-09-17T03:56:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.386+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5d3801c460 con 0x7f5d4014a730 2024-09-17T03:56:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.387+0000 7f5d37400640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 0x7f5d2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:34.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.387+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5939+0+0 (secure 0 0 0) 0x7f5d3809a970 con 0x7f5d4014a730 2024-09-17T03:56:34.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.387+0000 7f5d3f400640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 0x7f5d2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:34.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.388+0000 7f5d3f400640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 0x7f5d2007b0a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5d30034000 tx=0x7f5d30004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:34.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.391+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5d380626e0 con 0x7f5d4014a730 2024-09-17T03:56:34.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:34.610+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5d4014f070 con 0x7f5d4014a730 2024-09-17T03:56:35.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.125+0000 7f5d37400640 1 -- 172.21.15.129:0/3017696303 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v89) ==== 104+0+0 (secure 0 0 0) 0x7f5d3809a0a0 con 0x7f5d4014a730 2024-09-17T03:56:35.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 msgr2=0x7f5d2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:35.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 0x7f5d2007b0a0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5d30034000 tx=0x7f5d30004ab0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 msgr2=0x7f5d401513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:35.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c400 tx=0x7f5d3800c8c0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 shutdown_connections 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d20078b90 0x7f5d2007b0a0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d4014a730 0x7f5d401513f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d400cdf10 0x7f5d4014a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 --2- 172.21.15.129:0/3017696303 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d400b4ee0 0x7f5d4014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 >> 172.21.15.129:0/3017696303 conn(0x7f5d4006cfc0 msgr2=0x7f5d401e6060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 shutdown_connections 2024-09-17T03:56:35.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.128+0000 7f5d45000640 1 -- 172.21.15.129:0/3017696303 wait complete. 2024-09-17T03:56:35.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 -- 172.21.15.129:0/1036055681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 msgr2=0x7f1aa80cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 --2- 172.21.15.129:0/1036055681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa80cbea0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0005fd0 tx=0x7f1aa002a820 comp rx=0 tx=0).stop 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 -- 172.21.15.129:0/1036055681 shutdown_connections 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 --2- 172.21.15.129:0/1036055681 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa80cbea0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 --2- 172.21.15.129:0/1036055681 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa80b3f70 0x7f1aa80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 -- 172.21.15.129:0/1036055681 >> 172.21.15.129:0/1036055681 conn(0x7f1aa80ccc40 msgr2=0x7f1aa80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:35.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 -- 172.21.15.129:0/1036055681 shutdown_connections 2024-09-17T03:56:35.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.377+0000 7f1aada00640 1 -- 172.21.15.129:0/1036055681 wait complete. 2024-09-17T03:56:35.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.380+0000 7f1aada00640 1 Processor -- start 2024-09-17T03:56:35.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.380+0000 7f1aada00640 1 -- start start 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa81612a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa8158410 0x7f1aa8166170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa80cc910 con 0x7f1aa8074580 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa80cc790 con 0x7f1aa80cba90 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.381+0000 7f1aada00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa8166f20 con 0x7f1aa8158410 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:35.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44782/0 (socket says 172.21.15.129:44782) 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 -- 172.21.15.129:0/2454120526 learned_addr learned my addr 172.21.15.129:0/2454120526 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa7400640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa8158410 0x7f1aa8166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1a9e000640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa81612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 -- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa8158410 msgr2=0x7f1aa8166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa8158410 0x7f1aa8166170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 -- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 msgr2=0x7f1aa81612a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa81612a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:35.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 -- 172.21.15.129:0/2454120526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa8159c00 con 0x7f1aa8074580 2024-09-17T03:56:35.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.382+0000 7f1aa6a00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800e790 tx=0x7f1a9800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:35.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.383+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9800eea0 con 0x7f1aa8074580 2024-09-17T03:56:35.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.383+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a9800cdb0 con 0x7f1aa8074580 2024-09-17T03:56:35.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.383+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98010690 con 0x7f1aa8074580 2024-09-17T03:56:35.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.383+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa8158c00 con 0x7f1aa8074580 2024-09-17T03:56:35.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.383+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa8159ea0 con 0x7f1aa8074580 2024-09-17T03:56:35.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.384+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa80b3f70 con 0x7f1aa8074580 2024-09-17T03:56:35.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.385+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1a98010830 con 0x7f1aa8074580 2024-09-17T03:56:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.386+0000 7f1a9f400640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 0x7f1a8407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.386+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 5939+0+0 (secure 0 0 0) 0x7f1a9809b6c0 con 0x7f1aa8074580 2024-09-17T03:56:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.387+0000 7f1a9e000640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 0x7f1a8407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.387+0000 7f1a9e000640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 0x7f1a8407b0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1a90007fd0 tx=0x7f1a90001390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:35.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.391+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1a98063440 con 0x7f1aa8074580 2024-09-17T03:56:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:35.575+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1aa8075020 con 0x7f1aa8074580 2024-09-17T03:56:36.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.128+0000 7f1a9f400640 1 -- 172.21.15.129:0/2454120526 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7f1a98067e70 con 0x7f1aa8074580 2024-09-17T03:56:36.128 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:56:36.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 msgr2=0x7f1a8407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:36.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 0x7f1a8407b0d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1a90007fd0 tx=0x7f1a90001390 comp rx=0 tx=0).stop 2024-09-17T03:56:36.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 msgr2=0x7f1aa815a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800e790 tx=0x7f1a9800ec50 comp rx=0 tx=0).stop 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 shutdown_connections 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1a84078bc0 0x7f1a8407b0d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1aa8158410 0x7f1aa8166170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1aa80cba90 0x7f1aa81612a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 --2- 172.21.15.129:0/2454120526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1aa8074580 0x7f1aa815a6a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.131+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 >> 172.21.15.129:0/2454120526 conn(0x7f1aa80ccc40 msgr2=0x7f1aa80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.132+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 shutdown_connections 2024-09-17T03:56:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.132+0000 7f1aada00640 1 -- 172.21.15.129:0/2454120526 wait complete. 2024-09-17T03:56:36.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 -- 172.21.15.129:0/2576051773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a780c66a0 msgr2=0x7f3a780c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/2576051773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a780c66a0 0x7f3a780c6ab0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f3a64005fd0 tx=0x7f3a6402a4b0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 -- 172.21.15.129:0/2576051773 shutdown_connections 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/2576051773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a780c66a0 0x7f3a780c6ab0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/2576051773 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a78144780 0x7f3a78072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 -- 172.21.15.129:0/2576051773 >> 172.21.15.129:0/2576051773 conn(0x7f3a780b1c50 msgr2=0x7f3a780b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 -- 172.21.15.129:0/2576051773 shutdown_connections 2024-09-17T03:56:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.384+0000 7f3a7cc00640 1 -- 172.21.15.129:0/2576051773 wait complete. 2024-09-17T03:56:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.386+0000 7f3a7cc00640 1 Processor -- start 2024-09-17T03:56:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.387+0000 7f3a7cc00640 1 -- start start 2024-09-17T03:56:36.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a780c66a0 0x7f3a78152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:36.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a780cc9f0 0x7f3a780742d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a780ce500 con 0x7f3a78144780 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a780ce380 con 0x7f3a780c66a0 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a7cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3a780ce680 con 0x7f3a780cc9f0 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44800/0 (socket says 172.21.15.129:44800) 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.388+0000 7f3a77e00640 1 -- 172.21.15.129:0/3520830253 learned_addr learned my addr 172.21.15.129:0/3520830253 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77400640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a780c66a0 0x7f3a78152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a76a00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a780cc9f0 0x7f3a780742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 -- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a780cc9f0 msgr2=0x7f3a780742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a780cc9f0 0x7f3a780742d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 -- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a780c66a0 msgr2=0x7f3a78152f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a780c66a0 0x7f3a78152f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 -- 172.21.15.129:0/3520830253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a78153480 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a77e00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000c700 tx=0x7f3a7000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.389+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70003b70 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.390+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a70003d10 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.390+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7000aab0 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.390+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a780b5e60 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.390+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a78156740 con 0x7f3a78144780 2024-09-17T03:56:36.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.391+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a780b58f0 con 0x7f3a78144780 2024-09-17T03:56:36.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.392+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a7000ac50 con 0x7f3a78144780 2024-09-17T03:56:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.393+0000 7f3a6f400640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 0x7f3a5807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.393+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5501+0+0 (secure 0 0 0) 0x7f3a7009a990 con 0x7f3a78144780 2024-09-17T03:56:36.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.393+0000 7f3a77400640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 0x7f3a5807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:36.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.394+0000 7f3a77400640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 0x7f3a5807b0a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009b30 tx=0x7f3a68005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:36.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.398+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a78153480 con 0x7f3a78144780 2024-09-17T03:56:36.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:36.586+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3a780b5b80 con 0x7f3a78144780 2024-09-17T03:56:37.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.131+0000 7f3a6f400640 1 -- 172.21.15.129:0/3520830253 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7f3a78144b90 con 0x7f3a78144780 2024-09-17T03:56:37.130 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:56:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.134+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 msgr2=0x7f3a5807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.134+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 0x7f3a5807b0a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3a68009b30 tx=0x7f3a68005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.134+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 msgr2=0x7f3a780c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.134+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000c700 tx=0x7f3a7000cbc0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 shutdown_connections 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a58078b90 0x7f3a5807b0a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a78144780 0x7f3a780c85b0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a780cc9f0 0x7f3a780742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 --2- 172.21.15.129:0/3520830253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a780c66a0 0x7f3a78152f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 >> 172.21.15.129:0/3520830253 conn(0x7f3a780b1c50 msgr2=0x7f3a78073490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 shutdown_connections 2024-09-17T03:56:37.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.135+0000 7f3a7cc00640 1 -- 172.21.15.129:0/3520830253 wait complete. 2024-09-17T03:56:37.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:56:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 -- 172.21.15.129:0/297061081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc3e0 msgr2=0x7f764c15f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 --2- 172.21.15.129:0/297061081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc3e0 0x7f764c15f060 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f764c0cc390 tx=0x7f763802a540 comp rx=0 tx=0).stop 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 -- 172.21.15.129:0/297061081 shutdown_connections 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 --2- 172.21.15.129:0/297061081 >> v2:172.21.15.181:3300/0 conn(0x7f764c0cc910 0x7f764c0ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 --2- 172.21.15.129:0/297061081 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc3e0 0x7f764c15f060 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 -- 172.21.15.129:0/297061081 >> 172.21.15.129:0/297061081 conn(0x7f764c06d410 msgr2=0x7f764c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 -- 172.21.15.129:0/297061081 shutdown_connections 2024-09-17T03:56:37.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.387+0000 7f7653600640 1 -- 172.21.15.129:0/297061081 wait complete. 2024-09-17T03:56:37.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.390+0000 7f7653600640 1 Processor -- start 2024-09-17T03:56:37.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.390+0000 7f7653600640 1 -- start start 2024-09-17T03:56:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 0x7f764c0c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 0x7f764c0b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f764c15f720 con 0x7f764c0cc910 2024-09-17T03:56:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f764c15f5a0 con 0x7f764c070a50 2024-09-17T03:56:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7653600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f764c0af070 con 0x7f764c0cc3e0 2024-09-17T03:56:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7651600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7651600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44824/0 (socket says 172.21.15.129:44824) 2024-09-17T03:56:37.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.391+0000 7f7651600640 1 -- 172.21.15.129:0/2238930496 learned_addr learned my addr 172.21.15.129:0/2238930496 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7650c00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 0x7f764c0c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f764be00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 0x7f764c0b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 -- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 msgr2=0x7f764c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 0x7f764c0b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 -- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 msgr2=0x7f764c0c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 0x7f764c0c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 -- 172.21.15.129:0/2238930496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f764c1f5b10 con 0x7f764c0cc910 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7650c00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 0x7f764c0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f764be00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 0x7f764c0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7651600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f764400cce0 tx=0x7f764400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764400f450 con 0x7f764c0cc910 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f764400f6e0 con 0x7f764c0cc910 2024-09-17T03:56:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764400fb30 con 0x7f764c0cc910 2024-09-17T03:56:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.392+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f764c1f5df0 con 0x7f764c0cc910 2024-09-17T03:56:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.393+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f764c1f63b0 con 0x7f764c0cc910 2024-09-17T03:56:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.394+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f764c15f060 con 0x7f764c0cc910 2024-09-17T03:56:37.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.395+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f764400fcd0 con 0x7f764c0cc910 2024-09-17T03:56:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.396+0000 7f7648c00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 0x7f762c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.396+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f764409b5b0 con 0x7f764c0cc910 2024-09-17T03:56:37.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.396+0000 7f7650c00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 0x7f762c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.397+0000 7f7650c00640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 0x7f762c07b0a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f764000a830 tx=0x7f7640008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:37.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.400+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7644014030 con 0x7f764c0cc910 2024-09-17T03:56:37.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.584+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f764c06a140 con 0x7f764c0cc910 2024-09-17T03:56:37.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.588+0000 7f7648c00640 1 -- 172.21.15.129:0/2238930496 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v49) ==== 71+0+0 (secure 0 0 0) 0x7f7644063c80 con 0x7f764c0cc910 2024-09-17T03:56:37.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 msgr2=0x7f762c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 0x7f762c07b0a0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f764000a830 tx=0x7f7640008040 comp rx=0 tx=0).stop 2024-09-17T03:56:37.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 msgr2=0x7f764c1f82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f764400cce0 tx=0x7f764400a620 comp rx=0 tx=0).stop 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 shutdown_connections 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f762c078b90 0x7f762c07b0a0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f764c0cc910 0x7f764c1f82a0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f764c0cc3e0 0x7f764c0b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 --2- 172.21.15.129:0/2238930496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f764c070a50 0x7f764c0c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 >> 172.21.15.129:0/2238930496 conn(0x7f764c06d410 msgr2=0x7f764c1629a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 shutdown_connections 2024-09-17T03:56:37.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.591+0000 7f7653600640 1 -- 172.21.15.129:0/2238930496 wait complete. 2024-09-17T03:56:37.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration' 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.837+0000 7f9f89600640 1 -- 172.21.15.129:0/2097686027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f840cfb60 msgr2=0x7f9f84160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.837+0000 7f9f89600640 1 --2- 172.21.15.129:0/2097686027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f840cfb60 0x7f9f84160cf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9f74005fd0 tx=0x7f9f7402a540 comp rx=0 tx=0).stop 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 -- 172.21.15.129:0/2097686027 shutdown_connections 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 --2- 172.21.15.129:0/2097686027 >> v2:172.21.15.181:3300/0 conn(0x7f9f840d09d0 0x7f9f840b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 --2- 172.21.15.129:0/2097686027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f840cfb60 0x7f9f84160cf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 -- 172.21.15.129:0/2097686027 >> 172.21.15.129:0/2097686027 conn(0x7f9f8406cfc0 msgr2=0x7f9f8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 -- 172.21.15.129:0/2097686027 shutdown_connections 2024-09-17T03:56:37.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.838+0000 7f9f89600640 1 -- 172.21.15.129:0/2097686027 wait complete. 2024-09-17T03:56:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.840+0000 7f9f89600640 1 Processor -- start 2024-09-17T03:56:37.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.840+0000 7f9f89600640 1 -- start start 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.841+0000 7f9f89600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f89600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f840cfb60 0x7f9f840cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f89600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841f36a0 0x7f9f840cb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f89600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8415d270 con 0x7f9f840b46e0 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f89600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8415d0f0 con 0x7f9f840cfb60 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f89600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8415d3f0 con 0x7f9f841f36a0 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44840/0 (socket says 172.21.15.129:44840) 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 -- 172.21.15.129:0/1539487358 learned_addr learned my addr 172.21.15.129:0/1539487358 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:37.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f88c00640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841f36a0 0x7f9f840cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f7b400640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f840cfb60 0x7f9f840cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 -- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841f36a0 msgr2=0x7f9f840cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841f36a0 0x7f9f840cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 -- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f840cfb60 msgr2=0x7f9f840cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f840cfb60 0x7f9f840cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:37.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.842+0000 7f9f83e00640 1 -- 172.21.15.129:0/1539487358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f840ca390 con 0x7f9f840b46e0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f83e00640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f9f74034040 tx=0x7f9f740038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7403e030 con 0x7f9f840b46e0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f840ca670 con 0x7f9f840b46e0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f74003d90 con 0x7f9f840b46e0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74035040 con 0x7f9f840b46e0 2024-09-17T03:56:37.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.843+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f840cac30 con 0x7f9f840b46e0 2024-09-17T03:56:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.844+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8415d8a0 con 0x7f9f840b46e0 2024-09-17T03:56:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.846+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9f74041030 con 0x7f9f840b46e0 2024-09-17T03:56:37.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.846+0000 7f9f80c00640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.847+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f9f740bb7b0 con 0x7f9f840b46e0 2024-09-17T03:56:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.847+0000 7f9f7b400640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:37.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.848+0000 7f9f7b400640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9f840ced90 tx=0x7f9f6c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:37.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:37.849+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9f74083df0 con 0x7f9f840b46e0 2024-09-17T03:56:38.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.101+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]} v 0) -- 0x7f9f84157c00 con 0x7f9f840b46e0 2024-09-17T03:56:38.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.143+0000 7f9f80c00640 1 -- 172.21.15.129:0/1539487358 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_hardlink_reintegration"]}]=0 v371) ==== 142+0+0 (secure 0 0 0) 0x7f9f740836f0 con 0x7f9f840b46e0 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 msgr2=0x7f9f6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9f840ced90 tx=0x7f9f6c009040 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 msgr2=0x7f9f841f6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f9f74034040 tx=0x7f9f740038a0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 shutdown_connections 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841f36a0 0x7f9f840cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f840cfb60 0x7f9f840cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 --2- 172.21.15.129:0/1539487358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840b46e0 0x7f9f841f6430 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.145+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 >> 172.21.15.129:0/1539487358 conn(0x7f9f8406cfc0 msgr2=0x7f9f8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.146+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 shutdown_connections 2024-09-17T03:56:38.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.146+0000 7f9f89600640 1 -- 172.21.15.129:0/1539487358 wait complete. 2024-09-17T03:56:38.167 INFO:tasks.cephfs_test_runner:That removal of primary dentry of hardlinked inode results ... ok 2024-09-17T03:56:38.167 INFO:tasks.cephfs_test_runner:Starting test: test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) Reproduce https://tracker.ceph.com/issues/53597 2024-09-17T03:56:38.167 INFO:tasks.cephfs_test_runner:test_migrate_unlinked_dir (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:56:38.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.392+0000 7f187be00640 1 -- 172.21.15.129:0/2877548910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c0d0a50 msgr2=0x7f187c15e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.392+0000 7f187be00640 1 --2- 172.21.15.129:0/2877548910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c0d0a50 0x7f187c15e880 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1864009860 tx=0x7f186402a530 comp rx=0 tx=0).stop 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.393+0000 7f187be00640 1 -- 172.21.15.129:0/2877548910 shutdown_connections 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.393+0000 7f187be00640 1 --2- 172.21.15.129:0/2877548910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c0d0a50 0x7f187c15e880 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.393+0000 7f187be00640 1 -- 172.21.15.129:0/2877548910 >> 172.21.15.129:0/2877548910 conn(0x7f187c06cfc0 msgr2=0x7f187c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.393+0000 7f187be00640 1 -- 172.21.15.129:0/2877548910 shutdown_connections 2024-09-17T03:56:38.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.393+0000 7f187be00640 1 -- 172.21.15.129:0/2877548910 wait complete. 2024-09-17T03:56:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.395+0000 7f187be00640 1 Processor -- start 2024-09-17T03:56:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.395+0000 7f187be00640 1 -- start start 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.396+0000 7f187be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f187c0ad8d0 0x7f187c0b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.396+0000 7f187be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.396+0000 7f187be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c072d30 0x7f187c15d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.396+0000 7f187be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f187c162f10 con 0x7f187c0cf910 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f187be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f187c162d90 con 0x7f187c0ad8d0 2024-09-17T03:56:38.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f187be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f187c163090 con 0x7f187c072d30 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44852/0 (socket says 172.21.15.129:44852) 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 -- 172.21.15.129:0/4259060114 learned_addr learned my addr 172.21.15.129:0/4259060114 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f187aa00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f187c0ad8d0 0x7f187c0b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f187b400640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c072d30 0x7f187c15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 -- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c072d30 msgr2=0x7f187c15d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c072d30 0x7f187c15d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 -- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f187c0ad8d0 msgr2=0x7f187c0b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f187c0ad8d0 0x7f187c0b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.397+0000 7f1872000640 1 -- 172.21.15.129:0/4259060114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187c14b100 con 0x7f187c0cf910 2024-09-17T03:56:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f1872000640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f186402ab20 tx=0x7f1864003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864005c90 con 0x7f187c0cf910 2024-09-17T03:56:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1864005020 con 0x7f187c0cf910 2024-09-17T03:56:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187c074db0 con 0x7f187c0cf910 2024-09-17T03:56:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1864005410 con 0x7f187c0cf910 2024-09-17T03:56:38.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.398+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f187c073b80 con 0x7f187c0cf910 2024-09-17T03:56:38.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.399+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f187c070f00 con 0x7f187c0cf910 2024-09-17T03:56:38.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.400+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f18640317c0 con 0x7f187c0cf910 2024-09-17T03:56:38.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.401+0000 7f1873400640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 0x7f185807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.401+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f18640bbae0 con 0x7f187c0cf910 2024-09-17T03:56:38.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.402+0000 7f187aa00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 0x7f185807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.402+0000 7f187aa00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 0x7f185807b0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f186c001120 tx=0x7f186c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:38.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.406+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f187c070f00 con 0x7f187c0cf910 2024-09-17T03:56:38.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.595+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f187c15e880 con 0x7f187c0cf910 2024-09-17T03:56:38.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.596+0000 7f1873400640 1 -- 172.21.15.129:0/4259060114 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+13222 (secure 0 0 0) 0x7f187c14b100 con 0x7f187c0cf910 2024-09-17T03:56:38.595 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:38.595 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":116,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:37.129323+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/1953968590":"2024-09-18T03:56:34.116962+0000","172.21.15.129:6834/1953968590":"2024-09-18T03:56:34.116962+0000","172.21.15.129:6834/153768962":"2024-09-18T03:55:27.495282+0000","172.21.15.129:6832/153768962":"2024-09-18T03:55:27.495282+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.598+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 msgr2=0x7f185807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.598+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 0x7f185807b0d0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f186c001120 tx=0x7f186c009040 comp rx=0 tx=0).stop 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 msgr2=0x7f187c14abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f186402ab20 tx=0x7f1864003c10 comp rx=0 tx=0).stop 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 shutdown_connections 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1858078bc0 0x7f185807b0d0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f187c072d30 0x7f187c15d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f187c0cf910 0x7f187c14abc0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 --2- 172.21.15.129:0/4259060114 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f187c0ad8d0 0x7f187c0b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 >> 172.21.15.129:0/4259060114 conn(0x7f187c06cfc0 msgr2=0x7f187c15a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:38.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 shutdown_connections 2024-09-17T03:56:38.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.599+0000 7f187be00640 1 -- 172.21.15.129:0/4259060114 wait complete. 2024-09-17T03:56:38.623 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:56:38.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:56:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 -- 172.21.15.129:0/623066971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c0ceb30 msgr2=0x7f3a8c0ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 --2- 172.21.15.129:0/623066971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c0ceb30 0x7f3a8c0ca9f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a84005fd0 tx=0x7f3a8402a820 comp rx=0 tx=0).stop 2024-09-17T03:56:38.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 -- 172.21.15.129:0/623066971 shutdown_connections 2024-09-17T03:56:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 --2- 172.21.15.129:0/623066971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c0ceb30 0x7f3a8c0ca9f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 --2- 172.21.15.129:0/623066971 >> v2:172.21.15.181:3300/0 conn(0x7f3a8c0cc170 0x7f3a8c0ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 -- 172.21.15.129:0/623066971 >> 172.21.15.129:0/623066971 conn(0x7f3a8c0ccc40 msgr2=0x7f3a8c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 -- 172.21.15.129:0/623066971 shutdown_connections 2024-09-17T03:56:38.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.848+0000 7f3a92e00640 1 -- 172.21.15.129:0/623066971 wait complete. 2024-09-17T03:56:38.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.850+0000 7f3a92e00640 1 Processor -- start 2024-09-17T03:56:38.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.851+0000 7f3a92e00640 1 -- start start 2024-09-17T03:56:38.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c070b20 0x7f3a8c14bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a8c0cc170 0x7f3a8c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c0cb2c0 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a8c0cb140 con 0x7f3a8c0cc170 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a92e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3a8c0cb440 con 0x7f3a8c070b20 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a90e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a90e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44858/0 (socket says 172.21.15.129:44858) 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.852+0000 7f3a90e00640 1 -- 172.21.15.129:0/2372373423 learned_addr learned my addr 172.21.15.129:0/2372373423 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a8be00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c070b20 0x7f3a8c14bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 -- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c070b20 msgr2=0x7f3a8c14bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c070b20 0x7f3a8c14bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 -- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a8c0cc170 msgr2=0x7f3a8c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:38.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a8c0cc170 0x7f3a8c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:38.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 -- 172.21.15.129:0/2372373423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a8c14c270 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a90e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3a84034040 tx=0x7f3a84032be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.853+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84041020 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.854+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a84004040 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.854+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a8c1f8b70 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.854+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84036040 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.854+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a8c1f92c0 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.855+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a8c0ce5f0 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.857+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a84044020 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.858+0000 7f3a83e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 0x7f3a6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:38.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.858+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f3a840bbe10 con 0x7f3a8c0ceb30 2024-09-17T03:56:38.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.859+0000 7f3a8be00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 0x7f3a6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:38.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.859+0000 7f3a8be00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 0x7f3a6c07b0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c001880 tx=0x7f3a7c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:38.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:38.862+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a8404b070 con 0x7f3a8c0ceb30 2024-09-17T03:56:39.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.043+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a8c1f8d70 con 0x7f3a8c0ceb30 2024-09-17T03:56:39.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.044+0000 7f3a83e00640 1 -- 172.21.15.129:0/2372373423 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) ==== 93+0+10 (secure 0 0 0) 0x7f3a840845d0 con 0x7f3a8c0ceb30 2024-09-17T03:56:39.043 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 msgr2=0x7f3a6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 0x7f3a6c07b0a0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c001880 tx=0x7f3a7c009040 comp rx=0 tx=0).stop 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 msgr2=0x7f3a8c073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f3a84034040 tx=0x7f3a84032be0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 shutdown_connections 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a6c078b90 0x7f3a6c07b0a0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a8c0ceb30 0x7f3a8c073c50 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a8c0cc170 0x7f3a8c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.046+0000 7f3a92e00640 1 --2- 172.21.15.129:0/2372373423 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a8c070b20 0x7f3a8c14bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.047+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 >> 172.21.15.129:0/2372373423 conn(0x7f3a8c0ccc40 msgr2=0x7f3a8c0ad380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:39.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.047+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 shutdown_connections 2024-09-17T03:56:39.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.047+0000 7f3a92e00640 1 -- 172.21.15.129:0/2372373423 wait complete. 2024-09-17T03:56:39.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-09-17T03:56:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 -- 172.21.15.129:0/1570011084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e8160cf0 msgr2=0x7fd7e80d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/1570011084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e8160cf0 0x7fd7e80d02c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009830 tx=0x7fd7d002a480 comp rx=0 tx=0).stop 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 -- 172.21.15.129:0/1570011084 shutdown_connections 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/1570011084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e80d0800 0x7fd7e815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/1570011084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e8160cf0 0x7fd7e80d02c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 -- 172.21.15.129:0/1570011084 >> 172.21.15.129:0/1570011084 conn(0x7fd7e806cfc0 msgr2=0x7fd7e806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 -- 172.21.15.129:0/1570011084 shutdown_connections 2024-09-17T03:56:39.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.294+0000 7fd7ecc00640 1 -- 172.21.15.129:0/1570011084 wait complete. 2024-09-17T03:56:39.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.297+0000 7fd7ecc00640 1 Processor -- start 2024-09-17T03:56:39.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.297+0000 7fd7ecc00640 1 -- start start 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7e80d0800 0x7fd7e81f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e8160cf0 0x7fd7e81f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e815d290 con 0x7fd7e80cfb60 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e815d110 con 0x7fd7e80d0800 2024-09-17T03:56:39.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7ecc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd7e815d410 con 0x7fd7e8160cf0 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.298+0000 7fd7e7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44878/0 (socket says 172.21.15.129:44878) 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 -- 172.21.15.129:0/2571755047 learned_addr learned my addr 172.21.15.129:0/2571755047 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7dea00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7e80d0800 0x7fd7e81f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7e00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e8160cf0 0x7fd7e81f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 -- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e8160cf0 msgr2=0x7fd7e81f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:39.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e8160cf0 0x7fd7e81f94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 -- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7e80d0800 msgr2=0x7fd7e81f8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7e80d0800 0x7fd7e81f8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 -- 172.21.15.129:0/2571755047 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e81385a0 con 0x7fd7e80cfb60 2024-09-17T03:56:39.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.299+0000 7fd7e7400640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800c3c0 tx=0x7fd7d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.300+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8017070 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.300+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7d80035f0 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.300+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8003a20 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.300+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e8138880 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.300+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e8138e40 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.301+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7e815d8d0 con 0x7fd7e80cfb60 2024-09-17T03:56:39.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.302+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd7d801a070 con 0x7fd7e80cfb60 2024-09-17T03:56:39.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.304+0000 7fd7dfe00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 0x7fd7c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:39.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.304+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7fd7d8099d40 con 0x7fd7e80cfb60 2024-09-17T03:56:39.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.304+0000 7fd7dea00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 0x7fd7c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:39.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.305+0000 7fd7dea00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 0x7fd7c407b0a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0034000 tx=0x7fd7d0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:39.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.308+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd7e80d0c10 con 0x7fd7e80cfb60 2024-09-17T03:56:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:39.558+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) -- 0x7fd7e806a140 con 0x7fd7e80cfb60 2024-09-17T03:56:40.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.151+0000 7fd7dfe00640 1 -- 172.21.15.129:0/2571755047 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v373) ==== 143+0+0 (secure 0 0 0) 0x7fd7d8061d30 con 0x7fd7e80cfb60 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.153+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 msgr2=0x7fd7c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.153+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 0x7fd7c407b0a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0034000 tx=0x7fd7d0004730 comp rx=0 tx=0).stop 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.153+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 msgr2=0x7fd7e80cb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.153+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800c3c0 tx=0x7fd7d800c880 comp rx=0 tx=0).stop 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 shutdown_connections 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd7c4078b90 0x7fd7c407b0a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7e8160cf0 0x7fd7e81f94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7e80d0800 0x7fd7e81f8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 --2- 172.21.15.129:0/2571755047 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7e80cfb60 0x7fd7e80cb3d0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 >> 172.21.15.129:0/2571755047 conn(0x7fd7e806cfc0 msgr2=0x7fd7e8155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:40.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 shutdown_connections 2024-09-17T03:56:40.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.154+0000 7fd7ecc00640 1 -- 172.21.15.129:0/2571755047 wait complete. 2024-09-17T03:56:40.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:56:40.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 -- 172.21.15.129:0/3289290360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 msgr2=0x7fc6640b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 --2- 172.21.15.129:0/3289290360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6640b4140 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fc664072080 tx=0x7fc65402a560 comp rx=0 tx=0).stop 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 -- 172.21.15.129:0/3289290360 shutdown_connections 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 --2- 172.21.15.129:0/3289290360 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 0x7fc6640cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 --2- 172.21.15.129:0/3289290360 >> v2:172.21.15.181:3300/0 conn(0x7fc66406eed0 0x7fc6640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 --2- 172.21.15.129:0/3289290360 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6640b4140 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 -- 172.21.15.129:0/3289290360 >> 172.21.15.129:0/3289290360 conn(0x7fc66406d410 msgr2=0x7fc6640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 -- 172.21.15.129:0/3289290360 shutdown_connections 2024-09-17T03:56:40.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.397+0000 7fc668c00640 1 -- 172.21.15.129:0/3289290360 wait complete. 2024-09-17T03:56:40.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.399+0000 7fc668c00640 1 Processor -- start 2024-09-17T03:56:40.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.399+0000 7fc668c00640 1 -- start start 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 0x7fc6641e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 0x7fc6641f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc664160b40 con 0x7fc6640705a0 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641609c0 con 0x7fc66406eed0 2024-09-17T03:56:40.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc668c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc664160cc0 con 0x7fc66406f2e0 2024-09-17T03:56:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc662800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc662800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44898/0 (socket says 172.21.15.129:44898) 2024-09-17T03:56:40.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.401+0000 7fc662800640 1 -- 172.21.15.129:0/2563135494 learned_addr learned my addr 172.21.15.129:0/2563135494 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc661e00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 0x7fc6641e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc661400640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 0x7fc6641f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 -- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 msgr2=0x7fc6641f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 0x7fc6641f92c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 -- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 msgr2=0x7fc6641e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 0x7fc6641e6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 -- 172.21.15.129:0/2563135494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6641e6ae0 con 0x7fc6640705a0 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc661e00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 0x7fc6641e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:40.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc662800640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00e990 tx=0x7fc65c00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c004d20 con 0x7fc6640705a0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc65c00ca60 con 0x7fc6640705a0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6641e6dc0 con 0x7fc6640705a0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c005020 con 0x7fc6640705a0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.402+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6641e7380 con 0x7fc6640705a0 2024-09-17T03:56:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.403+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6641e6850 con 0x7fc6640705a0 2024-09-17T03:56:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.405+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc65c0051c0 con 0x7fc6640705a0 2024-09-17T03:56:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.406+0000 7fc65a000640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 0x7fc64407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.406+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7fc65c09b370 con 0x7fc6640705a0 2024-09-17T03:56:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.406+0000 7fc661e00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 0x7fc64407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.407+0000 7fc661e00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 0x7fc64407b0a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc654033000 tx=0x7fc654004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:40.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.410+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc65c016030 con 0x7fc6640705a0 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.595+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fc664161250 con 0x7fc6640705a0 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.596+0000 7fc65a000640 1 -- 172.21.15.129:0/2563135494 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v49) ==== 59+0+2294 (secure 0 0 0) 0x7fc65c06c020 con 0x7fc6640705a0 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:56:40.595 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:56:40.596 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:56:40.597 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:56:40.598 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:56:40.599 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:56:40.599 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:56:40.599 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:56:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.598+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 msgr2=0x7fc64407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.598+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 0x7fc64407b0a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc654033000 tx=0x7fc654004730 comp rx=0 tx=0).stop 2024-09-17T03:56:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.598+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 msgr2=0x7fc6641f9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.598+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00e990 tx=0x7fc65c00ed90 comp rx=0 tx=0).stop 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 shutdown_connections 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc644078b90 0x7fc64407b0a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6640705a0 0x7fc6641f9800 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc66406f2e0 0x7fc6641f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 --2- 172.21.15.129:0/2563135494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc66406eed0 0x7fc6641e6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 >> 172.21.15.129:0/2563135494 conn(0x7fc66406d410 msgr2=0x7fc6640cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:40.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 shutdown_connections 2024-09-17T03:56:40.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.599+0000 7fc668c00640 1 -- 172.21.15.129:0/2563135494 wait complete. 2024-09-17T03:56:40.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.850+0000 7f38a6c00640 1 -- 172.21.15.129:0/774782630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3888004e00 msgr2=0x7f3888005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.850+0000 7f38a6c00640 1 --2- 172.21.15.129:0/774782630 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3888004e00 0x7f3888005290 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3898009ed0 tx=0x7f3888005830 comp rx=0 tx=0).stop 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.850+0000 7f38a6200640 1 -- 172.21.15.129:0/774782630 reap_dead start 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.850+0000 7f38a6c00640 1 -- 172.21.15.129:0/774782630 shutdown_connections 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.850+0000 7f38a6c00640 1 -- 172.21.15.129:0/774782630 >> 172.21.15.129:0/774782630 conn(0x7f38a00d00b0 msgr2=0x7f38a00d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.851+0000 7f38a6c00640 1 -- 172.21.15.129:0/774782630 shutdown_connections 2024-09-17T03:56:40.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.851+0000 7f38a6c00640 1 -- 172.21.15.129:0/774782630 wait complete. 2024-09-17T03:56:40.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.853+0000 7f38a6c00640 1 Processor -- start 2024-09-17T03:56:40.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.853+0000 7f38a6c00640 1 -- start start 2024-09-17T03:56:40.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3888004e00 0x7f38a009d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38a009b560 0x7f38a009b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38a0165b70 con 0x7f38a0153d70 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f38a01659f0 con 0x7f3888004e00 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.854+0000 7f38a6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f38a0165cf0 con 0x7f38a009b560 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44926/0 (socket says 172.21.15.129:44926) 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 -- 172.21.15.129:0/2038812460 learned_addr learned my addr 172.21.15.129:0/2038812460 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a5800640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3888004e00 0x7f38a009d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a6200640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38a009b560 0x7f38a009b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 -- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38a009b560 msgr2=0x7f38a009b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38a009b560 0x7f38a009b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 -- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3888004e00 msgr2=0x7f38a009d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3888004e00 0x7f38a009d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 -- 172.21.15.129:0/2038812460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a009beb0 con 0x7f38a0153d70 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.855+0000 7f38a4e00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f388c006610 tx=0x7f388c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.856+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c015070 con 0x7f38a0153d70 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.856+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f388c011900 con 0x7f38a0153d70 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.856+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388c011d50 con 0x7f38a0153d70 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.856+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a00ae930 con 0x7f38a0153d70 2024-09-17T03:56:40.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.856+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a00aeef0 con 0x7f38a0153d70 2024-09-17T03:56:40.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.857+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a00ae690 con 0x7f38a0153d70 2024-09-17T03:56:40.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.858+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f388c018070 con 0x7f38a0153d70 2024-09-17T03:56:40.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.859+0000 7f389d600640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 0x7f388007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:40.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.859+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f388c09b800 con 0x7f38a0153d70 2024-09-17T03:56:40.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.860+0000 7f38a5800640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 0x7f388007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:40.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.860+0000 7f38a5800640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 0x7f388007b0a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3894006fd0 tx=0x7f3894006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:40.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:40.863+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f38a00ae690 con 0x7f38a0153d70 2024-09-17T03:56:41.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.047+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f38a0072e10 con 0x7f38a0153d70 2024-09-17T03:56:41.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.051+0000 7f389d600640 1 -- 172.21.15.129:0/2038812460 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v50)=0 v50) ==== 122+0+0 (secure 0 0 0) 0x7f38a009beb0 con 0x7f38a0153d70 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.053+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 msgr2=0x7f388007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.053+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 0x7f388007b0a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3894006fd0 tx=0x7f3894006b10 comp rx=0 tx=0).stop 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.053+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 msgr2=0x7f38a0160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.053+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f388c006610 tx=0x7f388c0039c0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.053+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 shutdown_connections 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3880078b90 0x7f388007b0a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38a009b560 0x7f38a009b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38a0153d70 0x7f38a0160490 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 --2- 172.21.15.129:0/2038812460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3888004e00 0x7f38a009d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 >> 172.21.15.129:0/2038812460 conn(0x7f38a00d00b0 msgr2=0x7f38a015a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 shutdown_connections 2024-09-17T03:56:41.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.054+0000 7f38a6c00640 1 -- 172.21.15.129:0/2038812460 wait complete. 2024-09-17T03:56:41.074 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:56:41.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:56:41.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:56:41.248 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:56:41.248 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:41.248 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:41.284 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:56:41.284 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:56:41.284 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:41.321 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:56:41.321 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:41.321 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:41.359 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:56:41.359 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:56:41.359 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:41.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:56:41.397 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:56:41.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:41.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 -- 172.21.15.129:0/328443203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c15b090 msgr2=0x7f946c15f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 --2- 172.21.15.129:0/328443203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c15b090 0x7f946c15f670 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f945c005fd0 tx=0x7f945c02a940 comp rx=0 tx=0).stop 2024-09-17T03:56:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 -- 172.21.15.129:0/328443203 shutdown_connections 2024-09-17T03:56:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 --2- 172.21.15.129:0/328443203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c15b090 0x7f946c15f670 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 -- 172.21.15.129:0/328443203 >> 172.21.15.129:0/328443203 conn(0x7f946c06d410 msgr2=0x7f946c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 -- 172.21.15.129:0/328443203 shutdown_connections 2024-09-17T03:56:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.619+0000 7f9471e00640 1 -- 172.21.15.129:0/328443203 wait complete. 2024-09-17T03:56:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.621+0000 7f9471e00640 1 Processor -- start 2024-09-17T03:56:41.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.621+0000 7f9471e00640 1 -- start start 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946c0ce170 0x7f946c1f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 0x7f946c0ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f946c15b6d0 con 0x7f946c0ccbb0 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0b2b60 con 0x7f946c0ce170 2024-09-17T03:56:41.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f9471e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f946c15b850 con 0x7f946c15b090 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44930/0 (socket says 172.21.15.129:44930) 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 -- 172.21.15.129:0/4258761161 learned_addr learned my addr 172.21.15.129:0/4258761161 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946a000640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946c0ce170 0x7f946c1f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946b400640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 0x7f946c0ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 -- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 msgr2=0x7f946c0ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 0x7f946c0ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 -- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946c0ce170 msgr2=0x7f946c1f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946c0ce170 0x7f946c1f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946aa00640 1 -- 172.21.15.129:0/4258761161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c1f4e10 con 0x7f946c0ccbb0 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.623+0000 7f946b400640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 0x7f946c0ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f946aa00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f945c009cd0 tx=0x7f945c031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:41.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c03e070 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f945c004540 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c043460 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946c1f50f0 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.624+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946c1f56b0 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.625+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c15f670 con 0x7f946c0ccbb0 2024-09-17T03:56:41.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.626+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f945c043600 con 0x7f946c0ccbb0 2024-09-17T03:56:41.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.627+0000 7f9463400640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 0x7f944c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:41.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.627+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5029+0+0 (secure 0 0 0) 0x7f945c0bc080 con 0x7f946c0ccbb0 2024-09-17T03:56:41.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.628+0000 7f946a000640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 0x7f944c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:41.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.628+0000 7f946a000640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 0x7f944c07b0a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f946c0af310 tx=0x7f9458009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:41.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.631+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f945c04a070 con 0x7f946c0ccbb0 2024-09-17T03:56:41.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.848+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946c06a140 con 0x7f946c0ccbb0 2024-09-17T03:56:41.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.848+0000 7f9463400640 1 -- 172.21.15.129:0/4258761161 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+3923 (secure 0 0 0) 0x7f945c08be20 con 0x7f946c0ccbb0 2024-09-17T03:56:41.848 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:41.848 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":89,"btime":"2024-09-17T03:56:35:119718+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6218,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3398806618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":3398806618},{"type":"v1","addr":"172.21.15.181:6837","nonce":3398806618}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":6985,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/1644146155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1644146155},{"type":"v1","addr":"172.21.15.181:6835","nonce":1644146155}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7000,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1606391077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1606391077},{"type":"v1","addr":"172.21.15.129:6833","nonce":1606391077}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":83},{"gid":7539,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3178744386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3178744386},{"type":"v1","addr":"172.21.15.129:6835","nonce":3178744386}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":89}],"filesystems":[]} 2024-09-17T03:56:41.848 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 89 2024-09-17T03:56:41.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 msgr2=0x7f944c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 0x7f944c07b0a0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f946c0af310 tx=0x7f9458009040 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 msgr2=0x7f946c0997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f945c009cd0 tx=0x7f945c031e20 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 shutdown_connections 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f944c078b90 0x7f944c07b0a0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946c15b090 0x7f946c0ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946c0ce170 0x7f946c1f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 --2- 172.21.15.129:0/4258761161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f946c0ccbb0 0x7f946c0997f0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 >> 172.21.15.129:0/4258761161 conn(0x7f946c06d410 msgr2=0x7f946c15e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 shutdown_connections 2024-09-17T03:56:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:41.851+0000 7f9471e00640 1 -- 172.21.15.129:0/4258761161 wait complete. 2024-09-17T03:56:41.871 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:56:41.872 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:56:41.872 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:56:41.872 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:56:41.872 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:56:41.872 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:56:41.872 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:56:41.873 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:56:41.873 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:56:41.873 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T03:56:41.873 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:56:41.873 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:56:41.873 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:56:41.873 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T03:56:41.873 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:56:41.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:56:41.897 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:56:41.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:56:41.898 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:56:41.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:56:41.899 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:56:41.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:56:41.900 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:56:41.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:56:41.942 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:56:41.942 DEBUG:teuthology.parallel:result is None 2024-09-17T03:56:41.944 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:56:41.945 DEBUG:teuthology.parallel:result is None 2024-09-17T03:56:41.945 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:56:41.945 DEBUG:teuthology.parallel:result is None 2024-09-17T03:56:41.947 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:56:41.947 DEBUG:teuthology.parallel:result is None 2024-09-17T03:56:41.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:56:42.139 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:56:42.139 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:56:42.141 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:56:42.142 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 -- 172.21.15.129:0/1589737373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda34005150 msgr2=0x7fda3400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 --2- 172.21.15.129:0/1589737373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda34005150 0x7fda3400b9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fda44009ed0 tx=0x7fda34005830 comp rx=0 tx=0).stop 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda51a00640 1 -- 172.21.15.129:0/1589737373 reap_dead start 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 -- 172.21.15.129:0/1589737373 shutdown_connections 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 -- 172.21.15.129:0/1589737373 >> 172.21.15.129:0/1589737373 conn(0x7fda4c06cfc0 msgr2=0x7fda4c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 -- 172.21.15.129:0/1589737373 shutdown_connections 2024-09-17T03:56:42.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.185+0000 7fda52400640 1 -- 172.21.15.129:0/1589737373 wait complete. 2024-09-17T03:56:42.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.188+0000 7fda52400640 1 Processor -- start 2024-09-17T03:56:42.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.188+0000 7fda52400640 1 -- start start 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 0x7fda4c157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 0x7fda4c0af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0b4c70 con 0x7fda34005150 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c0b4af0 con 0x7fda4c144630 2024-09-17T03:56:42.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda52400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c0b4df0 con 0x7fda4c0b1cb0 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.189+0000 7fda51000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 0x7fda4c157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda51a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 0x7fda4c0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda51a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:41020/0 (socket says 172.21.15.129:41020) 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda51000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 0x7fda4c157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44984/0 (socket says 172.21.15.129:44984) 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda51a00640 1 -- 172.21.15.129:0/3001148735 learned_addr learned my addr 172.21.15.129:0/3001148735 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 -- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 msgr2=0x7fda4c0b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 -- 172.21.15.129:0/3001148735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 msgr2=0x7fda4c157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 0x7fda4c157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:42.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 -- 172.21.15.129:0/3001148735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0b2600 con 0x7fda4c144630 2024-09-17T03:56:42.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda51a00640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:56:42.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.190+0000 7fda4be00640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 0x7fda4c0af400 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fda380038b0 tx=0x7fda38003cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.193+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3801e070 con 0x7fda4c144630 2024-09-17T03:56:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.193+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda38019660 con 0x7fda4c144630 2024-09-17T03:56:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.193+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38019a90 con 0x7fda4c144630 2024-09-17T03:56:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.193+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c144dd0 con 0x7fda4c144630 2024-09-17T03:56:42.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.193+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c145390 con 0x7fda4c144630 2024-09-17T03:56:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.194+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c073690 con 0x7fda4c144630 2024-09-17T03:56:42.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.195+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fda38021070 con 0x7fda4c144630 2024-09-17T03:56:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.196+0000 7fda48c00640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 0x7fda2c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.196+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 5029+0+0 (secure 0 0 0) 0x7fda3809ec00 con 0x7fda4c144630 2024-09-17T03:56:42.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.197+0000 7fda51000640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 0x7fda2c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:42.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.197+0000 7fda51000640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 0x7fda2c07b0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0b4450 tx=0x7fda40005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:42.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.207+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fda4c073690 con 0x7fda4c144630 2024-09-17T03:56:42.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:42.402+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fda4c160cf0 con 0x7fda4c144630 2024-09-17T03:56:43.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.212+0000 7fda48c00640 1 -- 172.21.15.129:0/3001148735 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) ==== 97+0+0 (secure 0 0 0) 0x7fda4c0b2600 con 0x7fda4c144630 2024-09-17T03:56:43.212 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:56:43.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 msgr2=0x7fda2c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 0x7fda2c07b0d0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0b4450 tx=0x7fda40005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 msgr2=0x7fda4c0af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 0x7fda4c0af400 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fda380038b0 tx=0x7fda38003cb0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 shutdown_connections 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda2c078bc0 0x7fda2c07b0d0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda4c0b1cb0 0x7fda4c0b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda4c144630 0x7fda4c0af400 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 --2- 172.21.15.129:0/3001148735 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda34005150 0x7fda4c157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 >> 172.21.15.129:0/3001148735 conn(0x7fda4c06cfc0 msgr2=0x7fda4c072f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 shutdown_connections 2024-09-17T03:56:43.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.215+0000 7fda52400640 1 -- 172.21.15.129:0/3001148735 wait complete. 2024-09-17T03:56:43.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:56:43.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 -- 172.21.15.129:0/2194363881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 msgr2=0x7fe8500b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 --2- 172.21.15.129:0/2194363881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500b4140 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fe850072e00 tx=0x7fe84002a540 comp rx=0 tx=0).stop 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 -- 172.21.15.129:0/2194363881 shutdown_connections 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 --2- 172.21.15.129:0/2194363881 >> v2:172.21.15.181:3300/0 conn(0x7fe8500cd390 0x7fe8501609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 --2- 172.21.15.129:0/2194363881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500b4140 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 -- 172.21.15.129:0/2194363881 >> 172.21.15.129:0/2194363881 conn(0x7fe85006d410 msgr2=0x7fe8500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 -- 172.21.15.129:0/2194363881 shutdown_connections 2024-09-17T03:56:43.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.461+0000 7fe857a00640 1 -- 172.21.15.129:0/2194363881 wait complete. 2024-09-17T03:56:43.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.463+0000 7fe857a00640 1 Processor -- start 2024-09-17T03:56:43.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.463+0000 7fe857a00640 1 -- start start 2024-09-17T03:56:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 0x7fe8500cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 0x7fe8500b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe850160250 con 0x7fe8500705a0 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe85006f4f0 con 0x7fe85006eed0 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe857a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe850160f00 con 0x7fe8500cc600 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe84fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe84fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44996/0 (socket says 172.21.15.129:44996) 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.465+0000 7fe84fe00640 1 -- 172.21.15.129:0/3876074650 learned_addr learned my addr 172.21.15.129:0/3876074650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe855a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 0x7fe8500b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe855000640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 0x7fe8500cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 -- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 msgr2=0x7fe8500b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 0x7fe8500b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 -- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 msgr2=0x7fe8500cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 0x7fe8500cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 -- 172.21.15.129:0/3876074650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8501f9090 con 0x7fe8500705a0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe855a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 0x7fe8500b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe855000640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 0x7fe8500cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84fe00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00c340 tx=0x7fe83c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c017070 con 0x7fe8500705a0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe83c0035f0 con 0x7fe8500705a0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8501f9370 con 0x7fe8500705a0 2024-09-17T03:56:43.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c003a40 con 0x7fe8500705a0 2024-09-17T03:56:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.466+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8501f9930 con 0x7fe8500705a0 2024-09-17T03:56:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.467+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8501f9d10 con 0x7fe8500705a0 2024-09-17T03:56:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.469+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe83c01a070 con 0x7fe8500705a0 2024-09-17T03:56:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.469+0000 7fe84cc00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 0x7fe83007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.470+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 4857+0+0 (secure 0 0 0) 0x7fe83c099d50 con 0x7fe8500705a0 2024-09-17T03:56:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.470+0000 7fe855000640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 0x7fe83007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.471+0000 7fe855000640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 0x7fe83007b0a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe840034000 tx=0x7fe840004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:43.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.474+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe8501f9d10 con 0x7fe8500705a0 2024-09-17T03:56:43.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.726+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe8500b4140 con 0x7fe8500705a0 2024-09-17T03:56:43.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.727+0000 7fe84cc00640 1 -- 172.21.15.129:0/3876074650 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v33) ==== 80+0+5304 (secure 0 0 0) 0x7fe83c062440 con 0x7fe8500705a0 2024-09-17T03:56:43.726 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:43.726 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:56:43.726 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:56:43.726 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.727 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:56:43.728 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:56:43.729 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.730 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.731 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.733 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.735 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:56:43.736 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.737 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.738 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:56:43.740 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:56:43.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 msgr2=0x7fe83007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 0x7fe83007b0a0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe840034000 tx=0x7fe840004730 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 msgr2=0x7fe8500caf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00c340 tx=0x7fe83c00c800 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 shutdown_connections 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe830078b90 0x7fe83007b0a0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8500cc600 0x7fe8500b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8500705a0 0x7fe8500caf90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 --2- 172.21.15.129:0/3876074650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe85006eed0 0x7fe8500cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.729+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 >> 172.21.15.129:0/3876074650 conn(0x7fe85006d410 msgr2=0x7fe85006f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.730+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 shutdown_connections 2024-09-17T03:56:43.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.730+0000 7fe857a00640 1 -- 172.21.15.129:0/3876074650 wait complete. 2024-09-17T03:56:43.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 -- 172.21.15.129:0/2393312534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c160cf0 msgr2=0x7fcc6c0d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 --2- 172.21.15.129:0/2393312534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c0d02c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009830 tx=0x7fcc5402a4b0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 -- 172.21.15.129:0/2393312534 shutdown_connections 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 --2- 172.21.15.129:0/2393312534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c0d02c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 -- 172.21.15.129:0/2393312534 >> 172.21.15.129:0/2393312534 conn(0x7fcc6c06cfc0 msgr2=0x7fcc6c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 -- 172.21.15.129:0/2393312534 shutdown_connections 2024-09-17T03:56:43.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.971+0000 7fcc71e00640 1 -- 172.21.15.129:0/2393312534 wait complete. 2024-09-17T03:56:43.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.973+0000 7fcc71e00640 1 Processor -- start 2024-09-17T03:56:43.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.973+0000 7fcc71e00640 1 -- start start 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 0x7fcc6c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc6c0d0800 0x7fcc6c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c15d460 con 0x7fcc6c160cf0 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6c15d2e0 con 0x7fcc6c0cfb60 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcc6c15d5e0 con 0x7fcc6c0d0800 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60506/0 (socket says 172.21.15.129:60506) 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.975+0000 7fcc71400640 1 -- 172.21.15.129:0/1836680005 learned_addr learned my addr 172.21.15.129:0/1836680005 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc63e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc6c0d0800 0x7fcc6c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc70a00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 0x7fcc6c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 -- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc6c0d0800 msgr2=0x7fcc6c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc6c0d0800 0x7fcc6c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 -- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 msgr2=0x7fcc6c0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 0x7fcc6c0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 -- 172.21.15.129:0/1836680005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6c1f96d0 con 0x7fcc6c160cf0 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc70a00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 0x7fcc6c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71400640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400cbc0 tx=0x7fcc6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64016020 con 0x7fcc6c160cf0 2024-09-17T03:56:43.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc64012420 con 0x7fcc6c160cf0 2024-09-17T03:56:43.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64012870 con 0x7fcc6c160cf0 2024-09-17T03:56:43.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6c1f99b0 con 0x7fcc6c160cf0 2024-09-17T03:56:43.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.976+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc6c1f9f70 con 0x7fcc6c160cf0 2024-09-17T03:56:43.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.978+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6c1fa350 con 0x7fcc6c160cf0 2024-09-17T03:56:43.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.979+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcc64019020 con 0x7fcc6c160cf0 2024-09-17T03:56:43.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.980+0000 7fcc69600640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 0x7fcc4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:43.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.980+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 4857+0+0 (secure 0 0 0) 0x7fcc6409ae00 con 0x7fcc6c160cf0 2024-09-17T03:56:43.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.980+0000 7fcc70a00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 0x7fcc4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:43.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.981+0000 7fcc70a00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 0x7fcc4807b0a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:43.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:43.984+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcc6c1fa350 con 0x7fcc6c160cf0 2024-09-17T03:56:44.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.170+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fcc6c0d02c0 con 0x7fcc6c160cf0 2024-09-17T03:56:44.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.170+0000 7fcc69600640 1 -- 172.21.15.129:0/1836680005 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) ==== 84+0+65 (secure 0 0 0) 0x7fcc64020030 con 0x7fcc6c160cf0 2024-09-17T03:56:44.169 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:44.169 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:56:44.169 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:56:44.170 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:56:44.170 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:56:44.170 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:56:44.170 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 msgr2=0x7fcc4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 0x7fcc4807b0a0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c007fd0 tx=0x7fcc5c009040 comp rx=0 tx=0).stop 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 msgr2=0x7fcc6c15b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400cbc0 tx=0x7fcc6400a430 comp rx=0 tx=0).stop 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 shutdown_connections 2024-09-17T03:56:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc48078b90 0x7fcc4807b0a0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc6c160cf0 0x7fcc6c15b620 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc6c0d0800 0x7fcc6c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 --2- 172.21.15.129:0/1836680005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c0cfb60 0x7fcc6c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 >> 172.21.15.129:0/1836680005 conn(0x7fcc6c06cfc0 msgr2=0x7fcc6c1435b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 shutdown_connections 2024-09-17T03:56:44.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.173+0000 7fcc71e00640 1 -- 172.21.15.129:0/1836680005 wait complete. 2024-09-17T03:56:44.196 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:56:44.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:56:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.421+0000 7f2a9da00640 1 -- 172.21.15.129:0/3882425670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 msgr2=0x7f2a7c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.421+0000 7f2a9da00640 1 --2- 172.21.15.129:0/3882425670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a7c004c10 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000baf0 tx=0x7f2a7c005830 comp rx=0 tx=0).stop 2024-09-17T03:56:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.421+0000 7f2a97400640 1 -- 172.21.15.129:0/3882425670 reap_dead start 2024-09-17T03:56:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.421+0000 7f2a9da00640 1 -- 172.21.15.129:0/3882425670 shutdown_connections 2024-09-17T03:56:44.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.421+0000 7f2a9da00640 1 -- 172.21.15.129:0/3882425670 >> 172.21.15.129:0/3882425670 conn(0x7f2a980ccc40 msgr2=0x7f2a980cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.422+0000 7f2a9da00640 1 -- 172.21.15.129:0/3882425670 shutdown_connections 2024-09-17T03:56:44.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.422+0000 7f2a9da00640 1 -- 172.21.15.129:0/3882425670 wait complete. 2024-09-17T03:56:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.424+0000 7f2a9da00640 1 Processor -- start 2024-09-17T03:56:44.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.424+0000 7f2a9da00640 1 -- start start 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a981f63a0 0x7f2a981e4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a981f67b0 0x7f2a981e53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a980cc3e0 con 0x7f2a7c004800 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a980b4e20 con 0x7f2a981f63a0 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a9da00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2a980cc560 con 0x7f2a981f67b0 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:44.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60528/0 (socket says 172.21.15.129:60528) 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 -- 172.21.15.129:0/2697661367 learned_addr learned my addr 172.21.15.129:0/2697661367 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 -- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a981f67b0 msgr2=0x7f2a981e53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a981f67b0 0x7f2a981e53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 -- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a981f63a0 msgr2=0x7f2a981e4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a981f63a0 0x7f2a981e4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.426+0000 7f2a96a00640 1 -- 172.21.15.129:0/2697661367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a981e1920 con 0x7f2a7c004800 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a96a00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e570 tx=0x7f2a8800ea30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88015070 con 0x7f2a7c004800 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a880109a0 con 0x7f2a7c004800 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88010df0 con 0x7f2a7c004800 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a981f6f80 con 0x7f2a7c004800 2024-09-17T03:56:44.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.427+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9809ab00 con 0x7f2a7c004800 2024-09-17T03:56:44.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.428+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a980b3f70 con 0x7f2a7c004800 2024-09-17T03:56:44.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.430+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a88018070 con 0x7f2a7c004800 2024-09-17T03:56:44.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.430+0000 7f2a8f400640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 0x7f2a7407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:44.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.431+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(121..121 src has 1..121) ==== 4857+0+0 (secure 0 0 0) 0x7f2a8809adf0 con 0x7f2a7c004800 2024-09-17T03:56:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.431+0000 7f2a8e000640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 0x7f2a7407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:44.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.432+0000 7f2a8e000640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 0x7f2a7407b0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2a80036000 tx=0x7f2a80004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:44.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.435+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a981e1920 con 0x7f2a7c004800 2024-09-17T03:56:44.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:44.620+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2a98073030 con 0x7f2a7c004800 2024-09-17T03:56:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.184+0000 7f2a8f400640 1 -- 172.21.15.129:0/2697661367 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) ==== 138+0+0 (secure 0 0 0) 0x7f2a88063570 con 0x7f2a7c004800 2024-09-17T03:56:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.187+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 msgr2=0x7f2a7407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.187+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 0x7f2a7407b0d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f2a80036000 tx=0x7f2a80004730 comp rx=0 tx=0).stop 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.187+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 msgr2=0x7f2a981d4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.187+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800e570 tx=0x7f2a8800ea30 comp rx=0 tx=0).stop 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 shutdown_connections 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a74078bc0 0x7f2a7407b0d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a981f67b0 0x7f2a981e53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a981f63a0 0x7f2a981e4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 --2- 172.21.15.129:0/2697661367 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a7c004800 0x7f2a981d4240 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 >> 172.21.15.129:0/2697661367 conn(0x7f2a980ccc40 msgr2=0x7f2a980b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 shutdown_connections 2024-09-17T03:56:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.188+0000 7f2a9da00640 1 -- 172.21.15.129:0/2697661367 wait complete. 2024-09-17T03:56:45.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:56:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.444+0000 7fdaaae00640 1 -- 172.21.15.129:0/908022506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c003ca0 msgr2=0x7fda8c0040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.444+0000 7fdaaae00640 1 --2- 172.21.15.129:0/908022506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c003ca0 0x7fda8c0040f0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fda98006f90 tx=0x7fda8c005830 comp rx=0 tx=0).stop 2024-09-17T03:56:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.445+0000 7fdaaae00640 1 -- 172.21.15.129:0/908022506 shutdown_connections 2024-09-17T03:56:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.445+0000 7fdaaae00640 1 --2- 172.21.15.129:0/908022506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda8c003ca0 0x7fda8c0040f0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.445+0000 7fdaaae00640 1 -- 172.21.15.129:0/908022506 >> 172.21.15.129:0/908022506 conn(0x7fdaa40d0120 msgr2=0x7fdaa40d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:45.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.445+0000 7fdaaae00640 1 -- 172.21.15.129:0/908022506 shutdown_connections 2024-09-17T03:56:45.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.445+0000 7fdaaae00640 1 -- 172.21.15.129:0/908022506 wait complete. 2024-09-17T03:56:45.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.447+0000 7fdaaae00640 1 Processor -- start 2024-09-17T03:56:45.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.447+0000 7fdaaae00640 1 -- start start 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda8c003ca0 0x7fdaa4150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaa41f9150 0x7fdaa40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa40cf540 con 0x7fdaa41523f0 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4074920 con 0x7fda8c003ca0 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.448+0000 7fdaaae00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa40cf6c0 con 0x7fdaa41f9150 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60542/0 (socket says 172.21.15.129:60542) 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 -- 172.21.15.129:0/1507581794 learned_addr learned my addr 172.21.15.129:0/1507581794 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:45.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9a00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda8c003ca0 0x7fdaa4150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaaa400640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaa41f9150 0x7fdaa40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 -- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaa41f9150 msgr2=0x7fdaa40c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaa41f9150 0x7fdaa40c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 -- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda8c003ca0 msgr2=0x7fdaa4150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda8c003ca0 0x7fdaa4150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.449+0000 7fdaa9000640 1 -- 172.21.15.129:0/1507581794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa41f9560 con 0x7fdaa41523f0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaa9000640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fda90011a50 tx=0x7fda90011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9001f070 con 0x7fdaa41523f0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda90012450 con 0x7fdaa41523f0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90012880 con 0x7fdaa41523f0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa4154500 con 0x7fdaa41523f0 2024-09-17T03:56:45.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.450+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa40c8480 con 0x7fdaa41523f0 2024-09-17T03:56:45.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.451+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa40ca4f0 con 0x7fdaa41523f0 2024-09-17T03:56:45.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.452+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fda90022070 con 0x7fdaa41523f0 2024-09-17T03:56:45.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.454+0000 7fdaa1600640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 0x7fda8407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:45.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.454+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5227+0+0 (secure 0 0 0) 0x7fda9009be50 con 0x7fdaa41523f0 2024-09-17T03:56:45.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.454+0000 7fdaa9a00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 0x7fda8407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:45.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.455+0000 7fdaa9a00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 0x7fda8407b0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fda98033000 tx=0x7fdaa40750f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:45.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.457+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdaa40ca4f0 con 0x7fdaa41523f0 2024-09-17T03:56:45.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:45.641+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fdaa406f810 con 0x7fdaa41523f0 2024-09-17T03:56:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.187+0000 7fdaa1600640 1 -- 172.21.15.129:0/1507581794 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) ==== 170+0+0 (secure 0 0 0) 0x7fdaa41f9560 con 0x7fdaa41523f0 2024-09-17T03:56:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 msgr2=0x7fda8407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 0x7fda8407b0a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fda98033000 tx=0x7fdaa40750f0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 msgr2=0x7fdaa41f8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fda90011a50 tx=0x7fda90011f10 comp rx=0 tx=0).stop 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 shutdown_connections 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda84078b90 0x7fda8407b0a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdaa41f9150 0x7fdaa40c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdaa41523f0 0x7fdaa41f8c10 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 --2- 172.21.15.129:0/1507581794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda8c003ca0 0x7fdaa4150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.190+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 >> 172.21.15.129:0/1507581794 conn(0x7fdaa40d0120 msgr2=0x7fdaa415def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.191+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 shutdown_connections 2024-09-17T03:56:46.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.191+0000 7fdaaae00640 1 -- 172.21.15.129:0/1507581794 wait complete. 2024-09-17T03:56:46.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 -- 172.21.15.129:0/2823214300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b1c40 msgr2=0x7fc3680b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 --2- 172.21.15.129:0/2823214300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b1c40 0x7fc3680b2050 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fc350005fd0 tx=0x7fc35002a540 comp rx=0 tx=0).stop 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 -- 172.21.15.129:0/2823214300 shutdown_connections 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 --2- 172.21.15.129:0/2823214300 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b3010 0x7fc3680b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 --2- 172.21.15.129:0/2823214300 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b1c40 0x7fc3680b2050 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 -- 172.21.15.129:0/2823214300 >> 172.21.15.129:0/2823214300 conn(0x7fc3680ccc40 msgr2=0x7fc3680cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.459+0000 7fc36ca00640 1 -- 172.21.15.129:0/2823214300 shutdown_connections 2024-09-17T03:56:46.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.460+0000 7fc36ca00640 1 -- 172.21.15.129:0/2823214300 wait complete. 2024-09-17T03:56:46.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.462+0000 7fc36ca00640 1 Processor -- start 2024-09-17T03:56:46.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.462+0000 7fc36ca00640 1 -- start start 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b1c40 0x7fc3681e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b3010 0x7fc3681e9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3680b3ae0 con 0x7fc368074800 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc3680b3960 con 0x7fc3680b3010 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.463+0000 7fc36ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3680b3c60 con 0x7fc3680b1c40 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60548/0 (socket says 172.21.15.129:60548) 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 -- 172.21.15.129:0/3768197948 learned_addr learned my addr 172.21.15.129:0/3768197948 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:46.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc35d400640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b1c40 0x7fc3681e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc366800640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b3010 0x7fc3681e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 -- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b1c40 msgr2=0x7fc3681e9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b1c40 0x7fc3681e9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 -- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b3010 msgr2=0x7fc3681e9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b3010 0x7fc3681e9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 -- 172.21.15.129:0/3768197948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36809ddd0 con 0x7fc368074800 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.464+0000 7fc365e00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc35800c3d0 tx=0x7fc35800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.465+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358017070 con 0x7fc368074800 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.465+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc3580053c0 con 0x7fc368074800 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.465+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc358003850 con 0x7fc368074800 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.465+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36809acb0 con 0x7fc368074800 2024-09-17T03:56:46.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.465+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36809b270 con 0x7fc368074800 2024-09-17T03:56:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.466+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3680b4180 con 0x7fc368074800 2024-09-17T03:56:46.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.467+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc35801a070 con 0x7fc368074800 2024-09-17T03:56:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.468+0000 7fc35ea00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 0x7fc34407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.468+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 5609+0+0 (secure 0 0 0) 0x7fc358099fe0 con 0x7fc368074800 2024-09-17T03:56:46.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.469+0000 7fc35d400640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 0x7fc34407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:46.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.470+0000 7fc35d400640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 0x7fc34407b0a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc350033000 tx=0x7fc350004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:46.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.473+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc3680cbaf0 con 0x7fc368074800 2024-09-17T03:56:46.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:46.693+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc3680751a0 con 0x7fc368074800 2024-09-17T03:56:47.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:56:47.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:56:47.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.214+0000 7fc35ea00640 1 -- 172.21.15.129:0/3768197948 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v92) ==== 174+0+0 (secure 0 0 0) 0x7fc358061e40 con 0x7fc368074800 2024-09-17T03:56:47.216 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 18 and data pool 19 2024-09-17T03:56:47.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.219+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 msgr2=0x7fc34407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 0x7fc34407b0a0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc350033000 tx=0x7fc350004730 comp rx=0 tx=0).stop 2024-09-17T03:56:47.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 msgr2=0x7fc36814f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc35800c3d0 tx=0x7fc35800c890 comp rx=0 tx=0).stop 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 shutdown_connections 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc344078b90 0x7fc34407b0a0 secure :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc350033000 tx=0x7fc350004730 comp rx=0 tx=0).stop 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3680b3010 0x7fc3681e9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3680b1c40 0x7fc3681e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 --2- 172.21.15.129:0/3768197948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc368074800 0x7fc36814f5c0 secure :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fc35800c3d0 tx=0x7fc35800c890 comp rx=0 tx=0).stop 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.220+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 >> 172.21.15.129:0/3768197948 conn(0x7fc3680ccc40 msgr2=0x7fc3680cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:47.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.221+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 shutdown_connections 2024-09-17T03:56:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.222+0000 7fc36ca00640 1 -- 172.21.15.129:0/3768197948 wait complete. 2024-09-17T03:56:47.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 -- 172.21.15.129:0/821905586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2940ccc80 msgr2=0x7fc2940cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 --2- 172.21.15.129:0/821905586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2940ccc80 0x7fc2940cd090 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fc280005fd0 tx=0x7fc28002b230 comp rx=0 tx=0).stop 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 -- 172.21.15.129:0/821905586 shutdown_connections 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 --2- 172.21.15.129:0/821905586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc29406f640 0x7fc29406fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 --2- 172.21.15.129:0/821905586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2940ccc80 0x7fc2940cd090 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.477+0000 7fc29a600640 1 -- 172.21.15.129:0/821905586 >> 172.21.15.129:0/821905586 conn(0x7fc29406d410 msgr2=0x7fc2940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.478+0000 7fc29a600640 1 -- 172.21.15.129:0/821905586 shutdown_connections 2024-09-17T03:56:47.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.478+0000 7fc29a600640 1 -- 172.21.15.129:0/821905586 wait complete. 2024-09-17T03:56:47.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.480+0000 7fc29a600640 1 Processor -- start 2024-09-17T03:56:47.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.480+0000 7fc29a600640 1 -- start start 2024-09-17T03:56:47.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2940af4c0 0x7fc2940c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2940ccc80 0x7fc29409d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2940710e0 con 0x7fc29406f640 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc294070f60 con 0x7fc2940ccc80 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.481+0000 7fc29a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc29406ff90 con 0x7fc2940af4c0 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60580/0 (socket says 172.21.15.129:60580) 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 -- 172.21.15.129:0/1266576539 learned_addr learned my addr 172.21.15.129:0/1266576539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc292a00640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2940af4c0 0x7fc2940c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293e00640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2940ccc80 0x7fc29409d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 -- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2940af4c0 msgr2=0x7fc2940c65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2940af4c0 0x7fc2940c65c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 -- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2940ccc80 msgr2=0x7fc29409d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2940ccc80 0x7fc29409d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 -- 172.21.15.129:0/1266576539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2940c8450 con 0x7fc29406f640 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.482+0000 7fc293400640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c3a0 tx=0x7fc28400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.483+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2840052d0 con 0x7fc29406f640 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.483+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc284007020 con 0x7fc29406f640 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.483+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2940c7e90 con 0x7fc29406f640 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.483+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284007470 con 0x7fc29406f640 2024-09-17T03:56:47.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.483+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2940caef0 con 0x7fc29406f640 2024-09-17T03:56:47.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.484+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2940703c0 con 0x7fc29406f640 2024-09-17T03:56:47.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.485+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc2840067d0 con 0x7fc29406f640 2024-09-17T03:56:47.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.486+0000 7fc28b400640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 0x7fc27407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.486+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7fc28409a170 con 0x7fc29406f640 2024-09-17T03:56:47.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.486+0000 7fc292a00640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 0x7fc27407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.487+0000 7fc292a00640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 0x7fc27407b0a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc280033000 tx=0x7fc28002d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:47.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.491+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc2940cd5d0 con 0x7fc29406f640 2024-09-17T03:56:47.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.673+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc29406a140 con 0x7fc29406f640 2024-09-17T03:56:47.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.674+0000 7fc28b400640 1 -- 172.21.15.129:0/1266576539 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+16800 (secure 0 0 0) 0x7fc284061f60 con 0x7fc29406f640 2024-09-17T03:56:47.675 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:47.675 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":124,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:46.694491+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:56:44.632624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-17T03:56:45.653345+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 msgr2=0x7fc27407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 0x7fc27407b0a0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc280033000 tx=0x7fc28002d040 comp rx=0 tx=0).stop 2024-09-17T03:56:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 msgr2=0x7fc2940c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fc28400c3a0 tx=0x7fc28400c860 comp rx=0 tx=0).stop 2024-09-17T03:56:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 shutdown_connections 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc274078b90 0x7fc27407b0a0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2940ccc80 0x7fc29409d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2940af4c0 0x7fc2940c65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 --2- 172.21.15.129:0/1266576539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc29406f640 0x7fc2940c7320 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 >> 172.21.15.129:0/1266576539 conn(0x7fc29406d410 msgr2=0x7fc29415d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 shutdown_connections 2024-09-17T03:56:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.677+0000 7fc29a600640 1 -- 172.21.15.129:0/1266576539 wait complete. 2024-09-17T03:56:47.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 -- 172.21.15.129:0/339370669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc0d09d0 msgr2=0x7fdbcc15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 --2- 172.21.15.129:0/339370669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc0d09d0 0x7fdbcc15cbd0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4005fd0 tx=0x7fdbc402a840 comp rx=0 tx=0).stop 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 -- 172.21.15.129:0/339370669 shutdown_connections 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 --2- 172.21.15.129:0/339370669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc0d09d0 0x7fdbcc15cbd0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 -- 172.21.15.129:0/339370669 >> 172.21.15.129:0/339370669 conn(0x7fdbcc06cfc0 msgr2=0x7fdbcc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 -- 172.21.15.129:0/339370669 shutdown_connections 2024-09-17T03:56:47.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.919+0000 7fdbd2200640 1 -- 172.21.15.129:0/339370669 wait complete. 2024-09-17T03:56:47.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.922+0000 7fdbd2200640 1 Processor -- start 2024-09-17T03:56:47.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.922+0000 7fdbd2200640 1 -- start start 2024-09-17T03:56:47.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbcc0cfb60 0x7fdbcc0c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc15ba80 0x7fdbcc1f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc15d4a0 con 0x7fdbcc160cf0 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdbcc15d320 con 0x7fdbcc15ba80 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbd2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdbcc15d620 con 0x7fdbcc0cfb60 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.923+0000 7fdbcbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60596/0 (socket says 172.21.15.129:60596) 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 -- 172.21.15.129:0/2538493886 learned_addr learned my addr 172.21.15.129:0/2538493886 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbd0e00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbcc0cfb60 0x7fdbcc0c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbd1800640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc15ba80 0x7fdbcc1f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 -- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbcc0cfb60 msgr2=0x7fdbcc0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbcc0cfb60 0x7fdbcc0c7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 -- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc15ba80 msgr2=0x7fdbcc1f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc15ba80 0x7fdbcc1f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 -- 172.21.15.129:0/2538493886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbcc15a9e0 con 0x7fdbcc160cf0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbcbe00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8001220 tx=0x7fdbb800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.924+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8016020 con 0x7fdbcc160cf0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.925+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbcc15acc0 con 0x7fdbcc160cf0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.925+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbb8012920 con 0x7fdbcc160cf0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.925+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8012db0 con 0x7fdbcc160cf0 2024-09-17T03:56:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.925+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbcc15b280 con 0x7fdbcc160cf0 2024-09-17T03:56:47.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.926+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbcc0cf300 con 0x7fdbcc160cf0 2024-09-17T03:56:47.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.927+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdbb8019020 con 0x7fdbcc160cf0 2024-09-17T03:56:47.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.928+0000 7fdbc8c00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 0x7fdbac07b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:47.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.928+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(124..124 src has 1..124) ==== 5709+0+0 (secure 0 0 0) 0x7fdbb809ad80 con 0x7fdbcc160cf0 2024-09-17T03:56:47.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.928+0000 7fdbd0e00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 0x7fdbac07b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:47.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.929+0000 7fdbd0e00640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 0x7fdbac07b100 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc000f60 tx=0x7fdbbc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:47.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:47.932+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdbcc0cf300 con 0x7fdbcc160cf0 2024-09-17T03:56:48.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.117+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdbcc15a770 con 0x7fdbcc160cf0 2024-09-17T03:56:48.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.118+0000 7fdbc8c00640 1 -- 172.21.15.129:0/2538493886 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) ==== 74+0+16800 (secure 0 0 0) 0x7fdbb8020030 con 0x7fdbcc160cf0 2024-09-17T03:56:48.118 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:48.118 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":124,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:46.694491+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:56:44.632624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-17T03:56:45.653345+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 msgr2=0x7fdbac07b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 0x7fdbac07b100 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc000f60 tx=0x7fdbbc009040 comp rx=0 tx=0).stop 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 msgr2=0x7fdbcc1f44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8001220 tx=0x7fdbb800c940 comp rx=0 tx=0).stop 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 shutdown_connections 2024-09-17T03:56:48.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdbac078bf0 0x7fdbac07b100 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbcc15ba80 0x7fdbcc1f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbcc160cf0 0x7fdbcc1f44e0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 --2- 172.21.15.129:0/2538493886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbcc0cfb60 0x7fdbcc0c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 >> 172.21.15.129:0/2538493886 conn(0x7fdbcc06cfc0 msgr2=0x7fdbcc0b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 shutdown_connections 2024-09-17T03:56:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.121+0000 7fdbd2200640 1 -- 172.21.15.129:0/2538493886 wait complete. 2024-09-17T03:56:48.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:56:48.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 -- 172.21.15.129:0/2930257951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 msgr2=0x7f79480af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 --2- 172.21.15.129:0/2930257951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480af480 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f79480d1260 tx=0x7f793802a440 comp rx=0 tx=0).stop 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 -- 172.21.15.129:0/2930257951 shutdown_connections 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 --2- 172.21.15.129:0/2930257951 >> v2:172.21.15.181:3300/0 conn(0x7f79480b3010 0x7f79480b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 --2- 172.21.15.129:0/2930257951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480af480 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 -- 172.21.15.129:0/2930257951 >> 172.21.15.129:0/2930257951 conn(0x7f79480ccc40 msgr2=0x7f79480cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 -- 172.21.15.129:0/2930257951 shutdown_connections 2024-09-17T03:56:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.369+0000 7f794d600640 1 -- 172.21.15.129:0/2930257951 wait complete. 2024-09-17T03:56:48.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.371+0000 7f794d600640 1 Processor -- start 2024-09-17T03:56:48.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.371+0000 7f794d600640 1 -- start start 2024-09-17T03:56:48.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:48.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 0x7f794809ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:48.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79480b3010 0x7f794809d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:48.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79480b3ae0 con 0x7f7948074800 2024-09-17T03:56:48.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79480b3960 con 0x7f79480b1c40 2024-09-17T03:56:48.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f794d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f79480b3c60 con 0x7f79480b3010 2024-09-17T03:56:48.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:48.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60618/0 (socket says 172.21.15.129:60618) 2024-09-17T03:56:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 -- 172.21.15.129:0/4012045657 learned_addr learned my addr 172.21.15.129:0/4012045657 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f793e000640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 0x7f794809ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7947400640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79480b3010 0x7f794809d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 -- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79480b3010 msgr2=0x7f794809d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79480b3010 0x7f794809d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 -- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 msgr2=0x7f794809ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 0x7f794809ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.373+0000 7f7946a00640 1 -- 172.21.15.129:0/4012045657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79480cdd90 con 0x7f7948074800 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f793e000640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 0x7f794809ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f7946a00640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f7938000c00 tx=0x7f7938033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793803f020 con 0x7f7948074800 2024-09-17T03:56:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7938033e10 con 0x7f7948074800 2024-09-17T03:56:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7938037040 con 0x7f7948074800 2024-09-17T03:56:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79480ce070 con 0x7f7948074800 2024-09-17T03:56:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.374+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79480ce630 con 0x7f7948074800 2024-09-17T03:56:48.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.375+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79480b41c0 con 0x7f7948074800 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.376+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7938042020 con 0x7f7948074800 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.377+0000 7f793f400640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 0x7f792407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.377+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f79380baeb0 con 0x7f7948074800 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.378+0000 7f793e000640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 0x7f792407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.379+0000 7f793e000640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 0x7f792407b0a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7930007fd0 tx=0x7f7930009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.382+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7938049030 con 0x7f7948074800 2024-09-17T03:56:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:48.603+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f79480c3a90 con 0x7f7948074800 2024-09-17T03:56:49.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.226+0000 7f793f400640 1 -- 172.21.15.129:0/4012045657 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v95) ==== 118+0+0 (secure 0 0 0) 0x7f79380832e0 con 0x7f7948074800 2024-09-17T03:56:49.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 msgr2=0x7f792407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 0x7f792407b0a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7930007fd0 tx=0x7f7930009040 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 msgr2=0x7f79480cf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f7938000c00 tx=0x7f7938033c20 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 shutdown_connections 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7924078b90 0x7f792407b0a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79480b3010 0x7f794809d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79480b1c40 0x7f794809ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 --2- 172.21.15.129:0/4012045657 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7948074800 0x7f79480cf260 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 >> 172.21.15.129:0/4012045657 conn(0x7f79480ccc40 msgr2=0x7f79480cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 shutdown_connections 2024-09-17T03:56:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.229+0000 7f794d600640 1 -- 172.21.15.129:0/4012045657 wait complete. 2024-09-17T03:56:49.254 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:56:49.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 -- 172.21.15.129:0/575696809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 msgr2=0x7f27940af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 --2- 172.21.15.129:0/575696809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940af480 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f2784005fd0 tx=0x7f278402a5c0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 -- 172.21.15.129:0/575696809 shutdown_connections 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 --2- 172.21.15.129:0/575696809 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 0x7f27940b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 --2- 172.21.15.129:0/575696809 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940af480 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 -- 172.21.15.129:0/575696809 >> 172.21.15.129:0/575696809 conn(0x7f27940ccc40 msgr2=0x7f27940cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 -- 172.21.15.129:0/575696809 shutdown_connections 2024-09-17T03:56:49.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.479+0000 7f2799600640 1 -- 172.21.15.129:0/575696809 wait complete. 2024-09-17T03:56:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.482+0000 7f2799600640 1 Processor -- start 2024-09-17T03:56:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.482+0000 7f2799600640 1 -- start start 2024-09-17T03:56:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f279406eea0 0x7f27940cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:49.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 0x7f27940c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f27940b41a0 con 0x7f2794074800 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f27940b4020 con 0x7f279406eea0 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.483+0000 7f2799600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f27940b4320 con 0x7f27940b1c40 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f279406eea0 0x7f27940cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60632/0 (socket says 172.21.15.129:60632) 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 -- 172.21.15.129:0/633602774 learned_addr learned my addr 172.21.15.129:0/633602774 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2793400640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 0x7f27940c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 -- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 msgr2=0x7f27940c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 0x7f27940c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 -- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f279406eea0 msgr2=0x7f27940cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f279406eea0 0x7f27940cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 -- 172.21.15.129:0/633602774 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27941f8df0 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.484+0000 7f2792000640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f278000d650 tx=0x7f278000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.485+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780017070 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.485+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2780005330 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.485+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780009820 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.485+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27941f90d0 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.485+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27941f9690 con 0x7f2794074800 2024-09-17T03:56:49.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.486+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f279406fb50 con 0x7f2794074800 2024-09-17T03:56:49.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.487+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f27800099c0 con 0x7f2794074800 2024-09-17T03:56:49.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.489+0000 7f278aa00640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 0x7f277407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:49.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.489+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f278009adb0 con 0x7f2794074800 2024-09-17T03:56:49.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.489+0000 7f2792a00640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 0x7f277407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:49.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.490+0000 7f2792a00640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 0x7f277407b0a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2784033000 tx=0x7f2784004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:49.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.494+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f278009f070 con 0x7f2794074800 2024-09-17T03:56:49.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:49.711+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f27941f8b80 con 0x7f2794074800 2024-09-17T03:56:50.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.228+0000 7f278aa00640 1 -- 172.21.15.129:0/633602774 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v96) ==== 122+0+0 (secure 0 0 0) 0x7f2780062b00 con 0x7f2794074800 2024-09-17T03:56:50.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.230+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 msgr2=0x7f277407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 0x7f277407b0a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2784033000 tx=0x7f2784004730 comp rx=0 tx=0).stop 2024-09-17T03:56:50.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 msgr2=0x7f27940cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f278000d650 tx=0x7f278000db10 comp rx=0 tx=0).stop 2024-09-17T03:56:50.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 shutdown_connections 2024-09-17T03:56:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2774078b90 0x7f277407b0a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f27940b1c40 0x7f27940c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2794074800 0x7f27940cf6f0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 --2- 172.21.15.129:0/633602774 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f279406eea0 0x7f27940cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 >> 172.21.15.129:0/633602774 conn(0x7f27940ccc40 msgr2=0x7f279406f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:50.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 shutdown_connections 2024-09-17T03:56:50.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.231+0000 7f2799600640 1 -- 172.21.15.129:0/633602774 wait complete. 2024-09-17T03:56:50.258 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:56:50.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 -- 172.21.15.129:0/3052652999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7f0003c10 msgr2=0x7fb7f0004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 --2- 172.21.15.129:0/3052652999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7f0003c10 0x7fb7f0004060 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009b50 tx=0x7fb7f0005830 comp rx=0 tx=0).stop 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb80a800640 1 -- 172.21.15.129:0/3052652999 reap_dead start 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 -- 172.21.15.129:0/3052652999 shutdown_connections 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 -- 172.21.15.129:0/3052652999 >> 172.21.15.129:0/3052652999 conn(0x7fb80c06d410 msgr2=0x7fb80c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 -- 172.21.15.129:0/3052652999 shutdown_connections 2024-09-17T03:56:50.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.480+0000 7fb810c00640 1 -- 172.21.15.129:0/3052652999 wait complete. 2024-09-17T03:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.483+0000 7fb810c00640 1 Processor -- start 2024-09-17T03:56:50.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.483+0000 7fb810c00640 1 -- start start 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80c0c9950 0x7fb80c0744d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80c0c9d60 0x7fb80c074a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c06f2e0 con 0x7fb7f0003c10 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb80c161320 con 0x7fb80c0c9950 2024-09-17T03:56:50.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb810c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb80c06f460 con 0x7fb80c0c9d60 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.484+0000 7fb809e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60660/0 (socket says 172.21.15.129:60660) 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 -- 172.21.15.129:0/1748452988 learned_addr learned my addr 172.21.15.129:0/1748452988 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb801400640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80c0c9950 0x7fb80c0744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb80a800640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80c0c9d60 0x7fb80c074a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 -- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80c0c9d60 msgr2=0x7fb80c074a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80c0c9d60 0x7fb80c074a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 -- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80c0c9950 msgr2=0x7fb80c0744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80c0c9950 0x7fb80c0744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 -- 172.21.15.129:0/1748452988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80c0ca270 con 0x7fb7f0003c10 2024-09-17T03:56:50.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.485+0000 7fb809e00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009c70 tx=0x7fb7fc033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.486+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc038040 con 0x7fb7f0003c10 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.486+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7fc02b7e0 con 0x7fb7f0003c10 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.486+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc004280 con 0x7fb7f0003c10 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.486+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80c0cc8a0 con 0x7fb7f0003c10 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.486+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb80c1f9520 con 0x7fb7f0003c10 2024-09-17T03:56:50.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.487+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80c0cfae0 con 0x7fb7f0003c10 2024-09-17T03:56:50.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.488+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb7fc045020 con 0x7fb7f0003c10 2024-09-17T03:56:50.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.489+0000 7fb802a00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 0x7fb7e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.489+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7fb7fc0bb750 con 0x7fb7f0003c10 2024-09-17T03:56:50.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.490+0000 7fb801400640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 0x7fb7e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.490+0000 7fb801400640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 0x7fb7e807b0a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb80c073cc0 tx=0x7fb7f4006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:50.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.493+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb80c0cfae0 con 0x7fb7f0003c10 2024-09-17T03:56:50.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.712+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb80c06a140 con 0x7fb7f0003c10 2024-09-17T03:56:50.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.713+0000 7fb802a00640 1 -- 172.21.15.129:0/1748452988 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5228 (secure 0 0 0) 0x7fb7fc02b9b0 con 0x7fb7f0003c10 2024-09-17T03:56:50.714 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:50.714 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":96,"btime":"2024-09-17T03:56:50:227627+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1550025085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1550025085},{"type":"v1","addr":"172.21.15.181:6837","nonce":1550025085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":7309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3323106480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3323106480},{"type":"v1","addr":"172.21.15.181:6836","nonce":3323106480}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:49.713437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:56:50.715 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 96 2024-09-17T03:56:50.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 msgr2=0x7fb7e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 0x7fb7e807b0a0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb80c073cc0 tx=0x7fb7f4006b10 comp rx=0 tx=0).stop 2024-09-17T03:56:50.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 msgr2=0x7fb80c1f8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009c70 tx=0x7fb7fc033670 comp rx=0 tx=0).stop 2024-09-17T03:56:50.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 shutdown_connections 2024-09-17T03:56:50.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e8078b90 0x7fb7e807b0a0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80c0c9d60 0x7fb80c074a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80c0c9950 0x7fb80c0744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 --2- 172.21.15.129:0/1748452988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7f0003c10 0x7fb80c1f8910 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 >> 172.21.15.129:0/1748452988 conn(0x7fb80c06d410 msgr2=0x7fb80c06f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:50.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 shutdown_connections 2024-09-17T03:56:50.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.716+0000 7fb810c00640 1 -- 172.21.15.129:0/1748452988 wait complete. 2024-09-17T03:56:50.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 -- 172.21.15.129:0/495692944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc2c004d90 msgr2=0x7efc2c005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 --2- 172.21.15.129:0/495692944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc2c004d90 0x7efc2c005260 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7efc40005fd0 tx=0x7efc2c005830 comp rx=0 tx=0).stop 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc47e00640 1 -- 172.21.15.129:0/495692944 reap_dead start 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 -- 172.21.15.129:0/495692944 shutdown_connections 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 -- 172.21.15.129:0/495692944 >> 172.21.15.129:0/495692944 conn(0x7efc4806d410 msgr2=0x7efc480b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 -- 172.21.15.129:0/495692944 shutdown_connections 2024-09-17T03:56:50.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.965+0000 7efc4e400640 1 -- 172.21.15.129:0/495692944 wait complete. 2024-09-17T03:56:50.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.968+0000 7efc4e400640 1 Processor -- start 2024-09-17T03:56:50.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.968+0000 7efc4e400640 1 -- start start 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 0x7efc480ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc4815b120 0x7efc4815b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc480b4a10 con 0x7efc481f5590 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc480b4890 con 0x7efc480c8190 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.969+0000 7efc4e400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efc48070510 con 0x7efc4815b120 2024-09-17T03:56:50.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60688/0 (socket says 172.21.15.129:60688) 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 -- 172.21.15.129:0/273358855 learned_addr learned my addr 172.21.15.129:0/273358855 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc46a00640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc4815b120 0x7efc4815b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 0x7efc480ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 -- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc4815b120 msgr2=0x7efc4815b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc4815b120 0x7efc4815b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 -- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 msgr2=0x7efc480ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 0x7efc480ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 -- 172.21.15.129:0/273358855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4815ba70 con 0x7efc481f5590 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 0x7efc480ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:50.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.970+0000 7efc47e00640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7efc4002aea0 tx=0x7efc40033b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.971+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40042070 con 0x7efc481f5590 2024-09-17T03:56:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.971+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc40003970 con 0x7efc481f5590 2024-09-17T03:56:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.971+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc480cc750 con 0x7efc481f5590 2024-09-17T03:56:50.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.971+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc40038040 con 0x7efc481f5590 2024-09-17T03:56:50.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.971+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc481f64f0 con 0x7efc481f5590 2024-09-17T03:56:50.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.972+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc48156ac0 con 0x7efc481f5590 2024-09-17T03:56:50.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.973+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efc40004030 con 0x7efc481f5590 2024-09-17T03:56:50.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.974+0000 7efc3f400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 0x7efc2807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:50.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.974+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7efc400baff0 con 0x7efc481f5590 2024-09-17T03:56:50.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.975+0000 7efc47400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 0x7efc2807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.975+0000 7efc47400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 0x7efc2807b0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efc480993c0 tx=0x7efc38006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:50.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:50.979+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efc48156ac0 con 0x7efc481f5590 2024-09-17T03:56:51.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.164+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc48070a00 con 0x7efc481f5590 2024-09-17T03:56:51.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.165+0000 7efc3f400640 1 -- 172.21.15.129:0/273358855 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) ==== 74+0+16787 (secure 0 0 0) 0x7efc4815ba70 con 0x7efc481f5590 2024-09-17T03:56:51.167 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:51.167 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":125,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:56:48.210849+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:56:44.632624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-17T03:56:45.653345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:56:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 msgr2=0x7efc2807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 0x7efc2807b0d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7efc480993c0 tx=0x7efc38006b70 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 msgr2=0x7efc481f59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7efc4002aea0 tx=0x7efc40033b80 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 shutdown_connections 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc28078bc0 0x7efc2807b0d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc481f5590 0x7efc481f59a0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc4815b120 0x7efc4815b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 --2- 172.21.15.129:0/273358855 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc480c8190 0x7efc480ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 >> 172.21.15.129:0/273358855 conn(0x7efc4806d410 msgr2=0x7efc48099510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.168+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 shutdown_connections 2024-09-17T03:56:51.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.169+0000 7efc4e400640 1 -- 172.21.15.129:0/273358855 wait complete. 2024-09-17T03:56:51.192 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:56:51.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 -- 172.21.15.129:0/38586562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80cd390 msgr2=0x7fa1d81609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 --2- 172.21.15.129:0/38586562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80cd390 0x7fa1d81609c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0005fd0 tx=0x7fa1d002a940 comp rx=0 tx=0).stop 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 -- 172.21.15.129:0/38586562 shutdown_connections 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 --2- 172.21.15.129:0/38586562 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80cd390 0x7fa1d81609c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 -- 172.21.15.129:0/38586562 >> 172.21.15.129:0/38586562 conn(0x7fa1d806d410 msgr2=0x7fa1d80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:51.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 -- 172.21.15.129:0/38586562 shutdown_connections 2024-09-17T03:56:51.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.415+0000 7fa1df000640 1 -- 172.21.15.129:0/38586562 wait complete. 2024-09-17T03:56:51.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.417+0000 7fa1df000640 1 Processor -- start 2024-09-17T03:56:51.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.417+0000 7fa1df000640 1 -- start start 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 0x7fa1d8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 0x7fa1d81f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8160f00 con 0x7fa1d80b4140 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d8070d40 con 0x7fa1d80705a0 2024-09-17T03:56:51.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1df000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa1d8161080 con 0x7fa1d81f7340 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60702/0 (socket says 172.21.15.129:60702) 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 -- 172.21.15.129:0/3891156035 learned_addr learned my addr 172.21.15.129:0/3891156035 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1dd000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 0x7fa1d81f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7e00640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 0x7fa1d8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 -- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 msgr2=0x7fa1d81f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 0x7fa1d81f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 -- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 msgr2=0x7fa1d8157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 0x7fa1d8157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7400640 1 -- 172.21.15.129:0/3891156035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d809db00 con 0x7fa1d80b4140 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1dd000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 0x7fa1d81f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.419+0000 7fa1d7e00640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 0x7fa1d8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:51.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1d7400640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c390 tx=0x7fa1c400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4011790 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa1c4017070 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d809dde0 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4011cc0 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.420+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d81552e0 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.421+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d81609c0 con 0x7fa1d80b4140 2024-09-17T03:56:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.423+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa1c401c460 con 0x7fa1d80b4140 2024-09-17T03:56:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.424+0000 7fa1cfe00640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 0x7fa1b807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.424+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7fa1c409ab30 con 0x7fa1d80b4140 2024-09-17T03:56:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.425+0000 7fa1d7e00640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 0x7fa1b807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.425+0000 7fa1d7e00640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 0x7fa1b807b0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8000f60 tx=0x7fa1c8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:51.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.429+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa1c4062ef0 con 0x7fa1d80b4140 2024-09-17T03:56:51.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.609+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa1d81584d0 con 0x7fa1b8078bc0 2024-09-17T03:56:51.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.615+0000 7fa1cfe00640 1 -- 172.21.15.129:0/3891156035 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7fa1d81584d0 con 0x7fa1b8078bc0 2024-09-17T03:56:51.618 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:51.619 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 msgr2=0x7fa1b807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 0x7fa1b807b0d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8000f60 tx=0x7fa1c8009040 comp rx=0 tx=0).stop 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 msgr2=0x7fa1d8154da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fa1c400c390 tx=0x7fa1c400c850 comp rx=0 tx=0).stop 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 shutdown_connections 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa1b8078bc0 0x7fa1b807b0d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa1d81f7340 0x7fa1d81f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1d80b4140 0x7fa1d8154da0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 --2- 172.21.15.129:0/3891156035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa1d80705a0 0x7fa1d8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 >> 172.21.15.129:0/3891156035 conn(0x7fa1d806d410 msgr2=0x7fa1d80ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 shutdown_connections 2024-09-17T03:56:51.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.619+0000 7fa1df000640 1 -- 172.21.15.129:0/3891156035 wait complete. 2024-09-17T03:56:51.641 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-17T03:56:50.415522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304436,"kb_used_data":13316,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670284,"statfs":{"total":773094113280,"available":772782370816,"internally_reserved":0,"allocated":13635584,"data_stored":6542424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.229432"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525617+0000","last_change":"2024-09-17T03:56:47.222642+0000","last_active":"2024-09-17T03:56:48.525617+0000","last_peered":"2024-09-17T03:56:48.525617+0000","last_clean":"2024-09-17T03:56:48.525617+0000","last_became_active":"2024-09-17T03:56:47.221833+0000","last_became_peered":"2024-09-17T03:56:47.221833+0000","last_unstale":"2024-09-17T03:56:48.525617+0000","last_undegraded":"2024-09-17T03:56:48.525617+0000","last_fullsized":"2024-09-17T03:56:48.525617+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:01.568796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224457+0000","last_change":"2024-09-17T03:56:47.226856+0000","last_active":"2024-09-17T03:56:48.224457+0000","last_peered":"2024-09-17T03:56:48.224457+0000","last_clean":"2024-09-17T03:56:48.224457+0000","last_became_active":"2024-09-17T03:56:47.225956+0000","last_became_peered":"2024-09-17T03:56:47.225956+0000","last_unstale":"2024-09-17T03:56:48.224457+0000","last_undegraded":"2024-09-17T03:56:48.224457+0000","last_fullsized":"2024-09-17T03:56:48.224457+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:11.178671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235698+0000","last_change":"2024-09-17T03:56:47.228360+0000","last_active":"2024-09-17T03:56:47.235698+0000","last_peered":"2024-09-17T03:56:47.235698+0000","last_clean":"2024-09-17T03:56:47.235698+0000","last_became_active":"2024-09-17T03:56:47.224643+0000","last_became_peered":"2024-09-17T03:56:47.224643+0000","last_unstale":"2024-09-17T03:56:47.235698+0000","last_undegraded":"2024-09-17T03:56:47.235698+0000","last_fullsized":"2024-09-17T03:56:47.235698+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:52.533919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525343+0000","last_change":"2024-09-17T03:56:47.215839+0000","last_active":"2024-09-17T03:56:48.525343+0000","last_peered":"2024-09-17T03:56:48.525343+0000","last_clean":"2024-09-17T03:56:48.525343+0000","last_became_active":"2024-09-17T03:56:47.215544+0000","last_became_peered":"2024-09-17T03:56:47.215544+0000","last_unstale":"2024-09-17T03:56:48.525343+0000","last_undegraded":"2024-09-17T03:56:48.525343+0000","last_fullsized":"2024-09-17T03:56:48.525343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:14.487345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224877+0000","last_change":"2024-09-17T03:56:47.222637+0000","last_active":"2024-09-17T03:56:48.224877+0000","last_peered":"2024-09-17T03:56:48.224877+0000","last_clean":"2024-09-17T03:56:48.224877+0000","last_became_active":"2024-09-17T03:56:47.221854+0000","last_became_peered":"2024-09-17T03:56:47.221854+0000","last_unstale":"2024-09-17T03:56:48.224877+0000","last_undegraded":"2024-09-17T03:56:48.224877+0000","last_fullsized":"2024-09-17T03:56:48.224877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:08.063223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225664+0000","last_change":"2024-09-17T03:56:47.226307+0000","last_active":"2024-09-17T03:56:48.225664+0000","last_peered":"2024-09-17T03:56:48.225664+0000","last_clean":"2024-09-17T03:56:48.225664+0000","last_became_active":"2024-09-17T03:56:47.223792+0000","last_became_peered":"2024-09-17T03:56:47.223792+0000","last_unstale":"2024-09-17T03:56:48.225664+0000","last_undegraded":"2024-09-17T03:56:48.225664+0000","last_fullsized":"2024-09-17T03:56:48.225664+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:56.601035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225497+0000","last_change":"2024-09-17T03:56:47.228003+0000","last_active":"2024-09-17T03:56:48.225497+0000","last_peered":"2024-09-17T03:56:48.225497+0000","last_clean":"2024-09-17T03:56:48.225497+0000","last_became_active":"2024-09-17T03:56:47.227709+0000","last_became_peered":"2024-09-17T03:56:47.227709+0000","last_unstale":"2024-09-17T03:56:48.225497+0000","last_undegraded":"2024-09-17T03:56:48.225497+0000","last_fullsized":"2024-09-17T03:56:48.225497+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:55.407940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.227433+0000","last_change":"2024-09-17T03:56:47.219021+0000","last_active":"2024-09-17T03:56:47.227433+0000","last_peered":"2024-09-17T03:56:47.227433+0000","last_clean":"2024-09-17T03:56:47.227433+0000","last_became_active":"2024-09-17T03:56:47.218277+0000","last_became_peered":"2024-09-17T03:56:47.218277+0000","last_unstale":"2024-09-17T03:56:47.227433+0000","last_undegraded":"2024-09-17T03:56:47.227433+0000","last_fullsized":"2024-09-17T03:56:47.227433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:42.870804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235695+0000","last_change":"2024-09-17T03:56:47.225065+0000","last_active":"2024-09-17T03:56:47.235695+0000","last_peered":"2024-09-17T03:56:47.235695+0000","last_clean":"2024-09-17T03:56:47.235695+0000","last_became_active":"2024-09-17T03:56:47.224774+0000","last_became_peered":"2024-09-17T03:56:47.224774+0000","last_unstale":"2024-09-17T03:56:47.235695+0000","last_undegraded":"2024-09-17T03:56:47.235695+0000","last_fullsized":"2024-09-17T03:56:47.235695+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:02.952951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224700+0000","last_change":"2024-09-17T03:56:47.222081+0000","last_active":"2024-09-17T03:56:48.224700+0000","last_peered":"2024-09-17T03:56:48.224700+0000","last_clean":"2024-09-17T03:56:48.224700+0000","last_became_active":"2024-09-17T03:56:47.221779+0000","last_became_peered":"2024-09-17T03:56:47.221779+0000","last_unstale":"2024-09-17T03:56:48.224700+0000","last_undegraded":"2024-09-17T03:56:48.224700+0000","last_fullsized":"2024-09-17T03:56:48.224700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:52.307088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.233157+0000","last_change":"2024-09-17T03:56:47.228179+0000","last_active":"2024-09-17T03:56:47.233157+0000","last_peered":"2024-09-17T03:56:47.233157+0000","last_clean":"2024-09-17T03:56:47.233157+0000","last_became_active":"2024-09-17T03:56:47.227837+0000","last_became_peered":"2024-09-17T03:56:47.227837+0000","last_unstale":"2024-09-17T03:56:47.233157+0000","last_undegraded":"2024-09-17T03:56:47.233157+0000","last_fullsized":"2024-09-17T03:56:47.233157+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:06.592340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235731+0000","last_change":"2024-09-17T03:56:47.222943+0000","last_active":"2024-09-17T03:56:47.235731+0000","last_peered":"2024-09-17T03:56:47.235731+0000","last_clean":"2024-09-17T03:56:47.235731+0000","last_became_active":"2024-09-17T03:56:47.222620+0000","last_became_peered":"2024-09-17T03:56:47.222620+0000","last_unstale":"2024-09-17T03:56:47.235731+0000","last_undegraded":"2024-09-17T03:56:47.235731+0000","last_fullsized":"2024-09-17T03:56:47.235731+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:18:57.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525194+0000","last_change":"2024-09-17T03:56:47.217383+0000","last_active":"2024-09-17T03:56:48.525194+0000","last_peered":"2024-09-17T03:56:48.525194+0000","last_clean":"2024-09-17T03:56:48.525194+0000","last_became_active":"2024-09-17T03:56:47.215332+0000","last_became_peered":"2024-09-17T03:56:47.215332+0000","last_unstale":"2024-09-17T03:56:48.525194+0000","last_undegraded":"2024-09-17T03:56:48.525194+0000","last_fullsized":"2024-09-17T03:56:48.525194+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:18.365741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224479+0000","last_change":"2024-09-17T03:56:47.228101+0000","last_active":"2024-09-17T03:56:48.224479+0000","last_peered":"2024-09-17T03:56:48.224479+0000","last_clean":"2024-09-17T03:56:48.224479+0000","last_became_active":"2024-09-17T03:56:47.227750+0000","last_became_peered":"2024-09-17T03:56:47.227750+0000","last_unstale":"2024-09-17T03:56:48.224479+0000","last_undegraded":"2024-09-17T03:56:48.224479+0000","last_fullsized":"2024-09-17T03:56:48.224479+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:31.667211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525083+0000","last_change":"2024-09-17T03:56:47.220677+0000","last_active":"2024-09-17T03:56:48.525083+0000","last_peered":"2024-09-17T03:56:48.525083+0000","last_clean":"2024-09-17T03:56:48.525083+0000","last_became_active":"2024-09-17T03:56:47.220332+0000","last_became_peered":"2024-09-17T03:56:47.220332+0000","last_unstale":"2024-09-17T03:56:48.525083+0000","last_undegraded":"2024-09-17T03:56:48.525083+0000","last_fullsized":"2024-09-17T03:56:48.525083+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:58.489644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982294+0000","last_change":"2024-09-17T03:56:47.221379+0000","last_active":"2024-09-17T03:56:48.982294+0000","last_peered":"2024-09-17T03:56:48.982294+0000","last_clean":"2024-09-17T03:56:48.982294+0000","last_became_active":"2024-09-17T03:56:47.221019+0000","last_became_peered":"2024-09-17T03:56:47.221019+0000","last_unstale":"2024-09-17T03:56:48.982294+0000","last_undegraded":"2024-09-17T03:56:48.982294+0000","last_fullsized":"2024-09-17T03:56:48.982294+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:34.117078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225622+0000","last_change":"2024-09-17T03:56:47.229815+0000","last_active":"2024-09-17T03:56:48.225622+0000","last_peered":"2024-09-17T03:56:48.225622+0000","last_clean":"2024-09-17T03:56:48.225622+0000","last_became_active":"2024-09-17T03:56:47.229173+0000","last_became_peered":"2024-09-17T03:56:47.229173+0000","last_unstale":"2024-09-17T03:56:48.225622+0000","last_undegraded":"2024-09-17T03:56:48.225622+0000","last_fullsized":"2024-09-17T03:56:48.225622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.086311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983199+0000","last_change":"2024-09-17T03:56:47.224084+0000","last_active":"2024-09-17T03:56:48.983199+0000","last_peered":"2024-09-17T03:56:48.983199+0000","last_clean":"2024-09-17T03:56:48.983199+0000","last_became_active":"2024-09-17T03:56:47.223357+0000","last_became_peered":"2024-09-17T03:56:47.223357+0000","last_unstale":"2024-09-17T03:56:48.983199+0000","last_undegraded":"2024-09-17T03:56:48.983199+0000","last_fullsized":"2024-09-17T03:56:48.983199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:08.958502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.225896+0000","last_change":"2024-09-17T03:56:47.221033+0000","last_active":"2024-09-17T03:56:47.225896+0000","last_peered":"2024-09-17T03:56:47.225896+0000","last_clean":"2024-09-17T03:56:47.225896+0000","last_became_active":"2024-09-17T03:56:47.216921+0000","last_became_peered":"2024-09-17T03:56:47.216921+0000","last_unstale":"2024-09-17T03:56:47.225896+0000","last_undegraded":"2024-09-17T03:56:47.225896+0000","last_fullsized":"2024-09-17T03:56:47.225896+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:17.067518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983046+0000","last_change":"2024-09-17T03:56:47.228717+0000","last_active":"2024-09-17T03:56:48.983046+0000","last_peered":"2024-09-17T03:56:48.983046+0000","last_clean":"2024-09-17T03:56:48.983046+0000","last_became_active":"2024-09-17T03:56:47.228384+0000","last_became_peered":"2024-09-17T03:56:47.228384+0000","last_unstale":"2024-09-17T03:56:48.983046+0000","last_undegraded":"2024-09-17T03:56:48.983046+0000","last_fullsized":"2024-09-17T03:56:48.983046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:32.581668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224627+0000","last_change":"2024-09-17T03:56:47.222969+0000","last_active":"2024-09-17T03:56:48.224627+0000","last_peered":"2024-09-17T03:56:48.224627+0000","last_clean":"2024-09-17T03:56:48.224627+0000","last_became_active":"2024-09-17T03:56:47.222255+0000","last_became_peered":"2024-09-17T03:56:47.222255+0000","last_unstale":"2024-09-17T03:56:48.224627+0000","last_undegraded":"2024-09-17T03:56:48.224627+0000","last_fullsized":"2024-09-17T03:56:48.224627+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:35.099254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.236086+0000","last_change":"2024-09-17T03:56:47.231386+0000","last_active":"2024-09-17T03:56:47.236086+0000","last_peered":"2024-09-17T03:56:47.236086+0000","last_clean":"2024-09-17T03:56:47.236086+0000","last_became_active":"2024-09-17T03:56:47.231072+0000","last_became_peered":"2024-09-17T03:56:47.231072+0000","last_unstale":"2024-09-17T03:56:47.236086+0000","last_undegraded":"2024-09-17T03:56:47.236086+0000","last_fullsized":"2024-09-17T03:56:47.236086+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:13.491479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235663+0000","last_change":"2024-09-17T03:56:47.224215+0000","last_active":"2024-09-17T03:56:47.235663+0000","last_peered":"2024-09-17T03:56:47.235663+0000","last_clean":"2024-09-17T03:56:47.235663+0000","last_became_active":"2024-09-17T03:56:47.223097+0000","last_became_peered":"2024-09-17T03:56:47.223097+0000","last_unstale":"2024-09-17T03:56:47.235663+0000","last_undegraded":"2024-09-17T03:56:47.235663+0000","last_fullsized":"2024-09-17T03:56:47.235663+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:35.738186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.218567+0000","last_change":"2024-09-17T03:56:47.211424+0000","last_active":"2024-09-17T03:56:47.218567+0000","last_peered":"2024-09-17T03:56:47.218567+0000","last_clean":"2024-09-17T03:56:47.218567+0000","last_became_active":"2024-09-17T03:56:47.211022+0000","last_became_peered":"2024-09-17T03:56:47.211022+0000","last_unstale":"2024-09-17T03:56:47.218567+0000","last_undegraded":"2024-09-17T03:56:47.218567+0000","last_fullsized":"2024-09-17T03:56:47.218567+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:34.106940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225805+0000","last_change":"2024-09-17T03:56:47.230581+0000","last_active":"2024-09-17T03:56:48.225805+0000","last_peered":"2024-09-17T03:56:48.225805+0000","last_clean":"2024-09-17T03:56:48.225805+0000","last_became_active":"2024-09-17T03:56:47.230229+0000","last_became_peered":"2024-09-17T03:56:47.230229+0000","last_unstale":"2024-09-17T03:56:48.225805+0000","last_undegraded":"2024-09-17T03:56:48.225805+0000","last_fullsized":"2024-09-17T03:56:48.225805+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:26.439048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983202+0000","last_change":"2024-09-17T03:56:47.223680+0000","last_active":"2024-09-17T03:56:48.983202+0000","last_peered":"2024-09-17T03:56:48.983202+0000","last_clean":"2024-09-17T03:56:48.983202+0000","last_became_active":"2024-09-17T03:56:47.222950+0000","last_became_peered":"2024-09-17T03:56:47.222950+0000","last_unstale":"2024-09-17T03:56:48.983202+0000","last_undegraded":"2024-09-17T03:56:48.983202+0000","last_fullsized":"2024-09-17T03:56:48.983202+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:13.849851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525445+0000","last_change":"2024-09-17T03:56:47.220730+0000","last_active":"2024-09-17T03:56:48.525445+0000","last_peered":"2024-09-17T03:56:48.525445+0000","last_clean":"2024-09-17T03:56:48.525445+0000","last_became_active":"2024-09-17T03:56:47.219992+0000","last_became_peered":"2024-09-17T03:56:47.219992+0000","last_unstale":"2024-09-17T03:56:48.525445+0000","last_undegraded":"2024-09-17T03:56:48.525445+0000","last_fullsized":"2024-09-17T03:56:48.525445+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:41.911074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982789+0000","last_change":"2024-09-17T03:56:47.220610+0000","last_active":"2024-09-17T03:56:48.982789+0000","last_peered":"2024-09-17T03:56:48.982789+0000","last_clean":"2024-09-17T03:56:48.982789+0000","last_became_active":"2024-09-17T03:56:47.220327+0000","last_became_peered":"2024-09-17T03:56:47.220327+0000","last_unstale":"2024-09-17T03:56:48.982789+0000","last_undegraded":"2024-09-17T03:56:48.982789+0000","last_fullsized":"2024-09-17T03:56:48.982789+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:35.648336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525514+0000","last_change":"2024-09-17T03:56:47.222410+0000","last_active":"2024-09-17T03:56:48.525514+0000","last_peered":"2024-09-17T03:56:48.525514+0000","last_clean":"2024-09-17T03:56:48.525514+0000","last_became_active":"2024-09-17T03:56:47.222086+0000","last_became_peered":"2024-09-17T03:56:47.222086+0000","last_unstale":"2024-09-17T03:56:48.525514+0000","last_undegraded":"2024-09-17T03:56:48.525514+0000","last_fullsized":"2024-09-17T03:56:48.525514+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.217402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525553+0000","last_change":"2024-09-17T03:56:47.224892+0000","last_active":"2024-09-17T03:56:48.525553+0000","last_peered":"2024-09-17T03:56:48.525553+0000","last_clean":"2024-09-17T03:56:48.525553+0000","last_became_active":"2024-09-17T03:56:47.222943+0000","last_became_peered":"2024-09-17T03:56:47.222943+0000","last_unstale":"2024-09-17T03:56:48.525553+0000","last_undegraded":"2024-09-17T03:56:48.525553+0000","last_fullsized":"2024-09-17T03:56:48.525553+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:14.194488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223932+0000","last_change":"2024-09-17T03:56:47.233668+0000","last_active":"2024-09-17T03:56:48.223932+0000","last_peered":"2024-09-17T03:56:48.223932+0000","last_clean":"2024-09-17T03:56:48.223932+0000","last_became_active":"2024-09-17T03:56:47.232545+0000","last_became_peered":"2024-09-17T03:56:47.232545+0000","last_unstale":"2024-09-17T03:56:48.223932+0000","last_undegraded":"2024-09-17T03:56:48.223932+0000","last_fullsized":"2024-09-17T03:56:48.223932+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:34.117078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223634+0000","last_change":"2024-09-17T03:56:47.227702+0000","last_active":"2024-09-17T03:56:48.223634+0000","last_peered":"2024-09-17T03:56:48.223634+0000","last_clean":"2024-09-17T03:56:48.223634+0000","last_became_active":"2024-09-17T03:56:47.227410+0000","last_became_peered":"2024-09-17T03:56:47.227410+0000","last_unstale":"2024-09-17T03:56:48.223634+0000","last_undegraded":"2024-09-17T03:56:48.223634+0000","last_fullsized":"2024-09-17T03:56:48.223634+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:39.333054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983374+0000","last_change":"2024-09-17T03:56:47.220163+0000","last_active":"2024-09-17T03:56:48.983374+0000","last_peered":"2024-09-17T03:56:48.983374+0000","last_clean":"2024-09-17T03:56:48.983374+0000","last_became_active":"2024-09-17T03:56:47.219857+0000","last_became_peered":"2024-09-17T03:56:47.219857+0000","last_unstale":"2024-09-17T03:56:48.983374+0000","last_undegraded":"2024-09-17T03:56:48.983374+0000","last_fullsized":"2024-09-17T03:56:48.983374+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:08.063223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225000+0000","last_change":"2024-09-17T03:56:46.221818+0000","last_active":"2024-09-17T03:56:48.225000+0000","last_peered":"2024-09-17T03:56:48.225000+0000","last_clean":"2024-09-17T03:56:48.225000+0000","last_became_active":"2024-09-17T03:56:46.221498+0000","last_became_peered":"2024-09-17T03:56:46.221498+0000","last_unstale":"2024-09-17T03:56:48.225000+0000","last_undegraded":"2024-09-17T03:56:48.225000+0000","last_fullsized":"2024-09-17T03:56:48.225000+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:33.209440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.237834+0000","last_change":"2024-09-17T03:56:47.233289+0000","last_active":"2024-09-17T03:56:47.237834+0000","last_peered":"2024-09-17T03:56:47.237834+0000","last_clean":"2024-09-17T03:56:47.237834+0000","last_became_active":"2024-09-17T03:56:47.232977+0000","last_became_peered":"2024-09-17T03:56:47.232977+0000","last_unstale":"2024-09-17T03:56:47.237834+0000","last_undegraded":"2024-09-17T03:56:47.237834+0000","last_fullsized":"2024-09-17T03:56:47.237834+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:04.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983386+0000","last_change":"2024-09-17T03:56:46.201459+0000","last_active":"2024-09-17T03:56:48.983386+0000","last_peered":"2024-09-17T03:56:48.983386+0000","last_clean":"2024-09-17T03:56:48.983386+0000","last_became_active":"2024-09-17T03:56:46.201121+0000","last_became_peered":"2024-09-17T03:56:46.201121+0000","last_unstale":"2024-09-17T03:56:48.983386+0000","last_undegraded":"2024-09-17T03:56:48.983386+0000","last_fullsized":"2024-09-17T03:56:48.983386+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:48:16.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224505+0000","last_change":"2024-09-17T03:56:47.223264+0000","last_active":"2024-09-17T03:56:48.224505+0000","last_peered":"2024-09-17T03:56:48.224505+0000","last_clean":"2024-09-17T03:56:48.224505+0000","last_became_active":"2024-09-17T03:56:47.222113+0000","last_became_peered":"2024-09-17T03:56:47.222113+0000","last_unstale":"2024-09-17T03:56:48.224505+0000","last_undegraded":"2024-09-17T03:56:48.224505+0000","last_fullsized":"2024-09-17T03:56:48.224505+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:52.547094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.242302+0000","last_change":"2024-09-17T03:56:46.208768+0000","last_active":"2024-09-17T03:56:47.242302+0000","last_peered":"2024-09-17T03:56:47.242302+0000","last_clean":"2024-09-17T03:56:47.242302+0000","last_became_active":"2024-09-17T03:56:46.208293+0000","last_became_peered":"2024-09-17T03:56:46.208293+0000","last_unstale":"2024-09-17T03:56:47.242302+0000","last_undegraded":"2024-09-17T03:56:47.242302+0000","last_fullsized":"2024-09-17T03:56:47.242302+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.568339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224720+0000","last_change":"2024-09-17T03:56:47.235062+0000","last_active":"2024-09-17T03:56:48.224720+0000","last_peered":"2024-09-17T03:56:48.224720+0000","last_clean":"2024-09-17T03:56:48.224720+0000","last_became_active":"2024-09-17T03:56:47.232532+0000","last_became_peered":"2024-09-17T03:56:47.232532+0000","last_unstale":"2024-09-17T03:56:48.224720+0000","last_undegraded":"2024-09-17T03:56:48.224720+0000","last_fullsized":"2024-09-17T03:56:48.224720+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:26:58.839963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224800+0000","last_change":"2024-09-17T03:56:46.211517+0000","last_active":"2024-09-17T03:56:48.224800+0000","last_peered":"2024-09-17T03:56:48.224800+0000","last_clean":"2024-09-17T03:56:48.224800+0000","last_became_active":"2024-09-17T03:56:46.211115+0000","last_became_peered":"2024-09-17T03:56:46.211115+0000","last_unstale":"2024-09-17T03:56:48.224800+0000","last_undegraded":"2024-09-17T03:56:48.224800+0000","last_fullsized":"2024-09-17T03:56:48.224800+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:20.640316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983120+0000","last_change":"2024-09-17T03:56:47.223609+0000","last_active":"2024-09-17T03:56:48.983120+0000","last_peered":"2024-09-17T03:56:48.983120+0000","last_clean":"2024-09-17T03:56:48.983120+0000","last_became_active":"2024-09-17T03:56:47.223271+0000","last_became_peered":"2024-09-17T03:56:47.223271+0000","last_unstale":"2024-09-17T03:56:48.983120+0000","last_undegraded":"2024-09-17T03:56:48.983120+0000","last_fullsized":"2024-09-17T03:56:48.983120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:52.547094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224403+0000","last_change":"2024-09-17T03:56:46.215297+0000","last_active":"2024-09-17T03:56:48.224403+0000","last_peered":"2024-09-17T03:56:48.224403+0000","last_clean":"2024-09-17T03:56:48.224403+0000","last_became_active":"2024-09-17T03:56:46.214916+0000","last_became_peered":"2024-09-17T03:56:46.214916+0000","last_unstale":"2024-09-17T03:56:48.224403+0000","last_undegraded":"2024-09-17T03:56:48.224403+0000","last_fullsized":"2024-09-17T03:56:48.224403+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:39.368830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225548+0000","last_change":"2024-09-17T03:56:47.226911+0000","last_active":"2024-09-17T03:56:48.225548+0000","last_peered":"2024-09-17T03:56:48.225548+0000","last_clean":"2024-09-17T03:56:48.225548+0000","last_became_active":"2024-09-17T03:56:47.226567+0000","last_became_peered":"2024-09-17T03:56:47.226567+0000","last_unstale":"2024-09-17T03:56:48.225548+0000","last_undegraded":"2024-09-17T03:56:48.225548+0000","last_fullsized":"2024-09-17T03:56:48.225548+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:43:00.943053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983024+0000","last_change":"2024-09-17T03:56:46.217856+0000","last_active":"2024-09-17T03:56:48.983024+0000","last_peered":"2024-09-17T03:56:48.983024+0000","last_clean":"2024-09-17T03:56:48.983024+0000","last_became_active":"2024-09-17T03:56:46.217501+0000","last_became_peered":"2024-09-17T03:56:46.217501+0000","last_unstale":"2024-09-17T03:56:48.983024+0000","last_undegraded":"2024-09-17T03:56:48.983024+0000","last_fullsized":"2024-09-17T03:56:48.983024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:13.329585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.231576+0000","last_change":"2024-09-17T03:56:47.220276+0000","last_active":"2024-09-17T03:56:47.231576+0000","last_peered":"2024-09-17T03:56:47.231576+0000","last_clean":"2024-09-17T03:56:47.231576+0000","last_became_active":"2024-09-17T03:56:47.219915+0000","last_became_peered":"2024-09-17T03:56:47.219915+0000","last_unstale":"2024-09-17T03:56:47.231576+0000","last_undegraded":"2024-09-17T03:56:47.231576+0000","last_fullsized":"2024-09-17T03:56:47.231576+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:13.624271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223869+0000","last_change":"2024-09-17T03:56:46.213836+0000","last_active":"2024-09-17T03:56:48.223869+0000","last_peered":"2024-09-17T03:56:48.223869+0000","last_clean":"2024-09-17T03:56:48.223869+0000","last_became_active":"2024-09-17T03:56:46.213524+0000","last_became_peered":"2024-09-17T03:56:46.213524+0000","last_unstale":"2024-09-17T03:56:48.223869+0000","last_undegraded":"2024-09-17T03:56:48.223869+0000","last_fullsized":"2024-09-17T03:56:48.223869+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:21.705057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224333+0000","last_change":"2024-09-17T03:56:47.229184+0000","last_active":"2024-09-17T03:56:48.224333+0000","last_peered":"2024-09-17T03:56:48.224333+0000","last_clean":"2024-09-17T03:56:48.224333+0000","last_became_active":"2024-09-17T03:56:47.228902+0000","last_became_peered":"2024-09-17T03:56:47.228902+0000","last_unstale":"2024-09-17T03:56:48.224333+0000","last_undegraded":"2024-09-17T03:56:48.224333+0000","last_fullsized":"2024-09-17T03:56:48.224333+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:43.863945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224486+0000","last_change":"2024-09-17T03:56:46.216835+0000","last_active":"2024-09-17T03:56:48.224486+0000","last_peered":"2024-09-17T03:56:48.224486+0000","last_clean":"2024-09-17T03:56:48.224486+0000","last_became_active":"2024-09-17T03:56:46.214002+0000","last_became_peered":"2024-09-17T03:56:46.214002+0000","last_unstale":"2024-09-17T03:56:48.224486+0000","last_undegraded":"2024-09-17T03:56:48.224486+0000","last_fullsized":"2024-09-17T03:56:48.224486+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:25.523123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525511+0000","last_change":"2024-09-17T03:56:47.227059+0000","last_active":"2024-09-17T03:56:48.525511+0000","last_peered":"2024-09-17T03:56:48.525511+0000","last_clean":"2024-09-17T03:56:48.525511+0000","last_became_active":"2024-09-17T03:56:47.226583+0000","last_became_peered":"2024-09-17T03:56:47.226583+0000","last_unstale":"2024-09-17T03:56:48.525511+0000","last_undegraded":"2024-09-17T03:56:48.525511+0000","last_fullsized":"2024-09-17T03:56:48.525511+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:00.421206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983285+0000","last_change":"2024-09-17T03:56:46.208694+0000","last_active":"2024-09-17T03:56:48.983285+0000","last_peered":"2024-09-17T03:56:48.983285+0000","last_clean":"2024-09-17T03:56:48.983285+0000","last_became_active":"2024-09-17T03:56:46.207738+0000","last_became_peered":"2024-09-17T03:56:46.207738+0000","last_unstale":"2024-09-17T03:56:48.983285+0000","last_undegraded":"2024-09-17T03:56:48.983285+0000","last_fullsized":"2024-09-17T03:56:48.983285+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:45:08.814944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983131+0000","last_change":"2024-09-17T03:56:47.218716+0000","last_active":"2024-09-17T03:56:48.983131+0000","last_peered":"2024-09-17T03:56:48.983131+0000","last_clean":"2024-09-17T03:56:48.983131+0000","last_became_active":"2024-09-17T03:56:47.217488+0000","last_became_peered":"2024-09-17T03:56:47.217488+0000","last_unstale":"2024-09-17T03:56:48.983131+0000","last_undegraded":"2024-09-17T03:56:48.983131+0000","last_fullsized":"2024-09-17T03:56:48.983131+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:25.487664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983350+0000","last_change":"2024-09-17T03:56:46.211293+0000","last_active":"2024-09-17T03:56:48.983350+0000","last_peered":"2024-09-17T03:56:48.983350+0000","last_clean":"2024-09-17T03:56:48.983350+0000","last_became_active":"2024-09-17T03:56:46.210425+0000","last_became_peered":"2024-09-17T03:56:46.210425+0000","last_unstale":"2024-09-17T03:56:48.983350+0000","last_undegraded":"2024-09-17T03:56:48.983350+0000","last_fullsized":"2024-09-17T03:56:48.983350+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:28.941014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224722+0000","last_change":"2024-09-17T03:56:47.225731+0000","last_active":"2024-09-17T03:56:48.224722+0000","last_peered":"2024-09-17T03:56:48.224722+0000","last_clean":"2024-09-17T03:56:48.224722+0000","last_became_active":"2024-09-17T03:56:47.224955+0000","last_became_peered":"2024-09-17T03:56:47.224955+0000","last_unstale":"2024-09-17T03:56:48.224722+0000","last_undegraded":"2024-09-17T03:56:48.224722+0000","last_fullsized":"2024-09-17T03:56:48.224722+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:08.958502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982739+0000","last_change":"2024-09-17T03:56:46.203171+0000","last_active":"2024-09-17T03:56:48.982739+0000","last_peered":"2024-09-17T03:56:48.982739+0000","last_clean":"2024-09-17T03:56:48.982739+0000","last_became_active":"2024-09-17T03:56:46.202846+0000","last_became_peered":"2024-09-17T03:56:46.202846+0000","last_unstale":"2024-09-17T03:56:48.982739+0000","last_undegraded":"2024-09-17T03:56:48.982739+0000","last_fullsized":"2024-09-17T03:56:48.982739+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:54.802176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224974+0000","last_change":"2024-09-17T03:56:47.226840+0000","last_active":"2024-09-17T03:56:48.224974+0000","last_peered":"2024-09-17T03:56:48.224974+0000","last_clean":"2024-09-17T03:56:48.224974+0000","last_became_active":"2024-09-17T03:56:47.226351+0000","last_became_peered":"2024-09-17T03:56:47.226351+0000","last_unstale":"2024-09-17T03:56:48.224974+0000","last_undegraded":"2024-09-17T03:56:48.224974+0000","last_fullsized":"2024-09-17T03:56:48.224974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:42.612574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":19,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.245850+0000","last_change":"2024-09-17T03:56:46.217357+0000","last_active":"2024-09-17T03:56:47.245850+0000","last_peered":"2024-09-17T03:56:47.245850+0000","last_clean":"2024-09-17T03:56:47.245850+0000","last_became_active":"2024-09-17T03:56:46.216452+0000","last_became_peered":"2024-09-17T03:56:46.216452+0000","last_unstale":"2024-09-17T03:56:47.245850+0000","last_undegraded":"2024-09-17T03:56:47.245850+0000","last_fullsized":"2024-09-17T03:56:47.245850+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:36.690725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525429+0000","last_change":"2024-09-17T03:56:47.212314+0000","last_active":"2024-09-17T03:56:48.525429+0000","last_peered":"2024-09-17T03:56:48.525429+0000","last_clean":"2024-09-17T03:56:48.525429+0000","last_became_active":"2024-09-17T03:56:47.211998+0000","last_became_peered":"2024-09-17T03:56:47.211998+0000","last_unstale":"2024-09-17T03:56:48.525429+0000","last_undegraded":"2024-09-17T03:56:48.525429+0000","last_fullsized":"2024-09-17T03:56:48.525429+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:36.840465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525280+0000","last_change":"2024-09-17T03:56:46.216254+0000","last_active":"2024-09-17T03:56:48.525280+0000","last_peered":"2024-09-17T03:56:48.525280+0000","last_clean":"2024-09-17T03:56:48.525280+0000","last_became_active":"2024-09-17T03:56:46.215966+0000","last_became_peered":"2024-09-17T03:56:46.215966+0000","last_unstale":"2024-09-17T03:56:48.525280+0000","last_undegraded":"2024-09-17T03:56:48.525280+0000","last_fullsized":"2024-09-17T03:56:48.525280+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:47.037513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224373+0000","last_change":"2024-09-17T03:56:47.225951+0000","last_active":"2024-09-17T03:56:48.224373+0000","last_peered":"2024-09-17T03:56:48.224373+0000","last_clean":"2024-09-17T03:56:48.224373+0000","last_became_active":"2024-09-17T03:56:47.225644+0000","last_became_peered":"2024-09-17T03:56:47.225644+0000","last_unstale":"2024-09-17T03:56:48.224373+0000","last_undegraded":"2024-09-17T03:56:48.224373+0000","last_fullsized":"2024-09-17T03:56:48.224373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:27.107204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224457+0000","last_change":"2024-09-17T03:56:46.217401+0000","last_active":"2024-09-17T03:56:48.224457+0000","last_peered":"2024-09-17T03:56:48.224457+0000","last_clean":"2024-09-17T03:56:48.224457+0000","last_became_active":"2024-09-17T03:56:46.217060+0000","last_became_peered":"2024-09-17T03:56:46.217060+0000","last_unstale":"2024-09-17T03:56:48.224457+0000","last_undegraded":"2024-09-17T03:56:48.224457+0000","last_fullsized":"2024-09-17T03:56:48.224457+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:33:54.856357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235630+0000","last_change":"2024-09-17T03:56:47.223392+0000","last_active":"2024-09-17T03:56:47.235630+0000","last_peered":"2024-09-17T03:56:47.235630+0000","last_clean":"2024-09-17T03:56:47.235630+0000","last_became_active":"2024-09-17T03:56:47.222436+0000","last_became_peered":"2024-09-17T03:56:47.222436+0000","last_unstale":"2024-09-17T03:56:47.235630+0000","last_undegraded":"2024-09-17T03:56:47.235630+0000","last_fullsized":"2024-09-17T03:56:47.235630+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:13.940824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":21,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.246887+0000","last_change":"2024-09-17T03:56:46.212150+0000","last_active":"2024-09-17T03:56:47.246887+0000","last_peered":"2024-09-17T03:56:47.246887+0000","last_clean":"2024-09-17T03:56:47.246887+0000","last_became_active":"2024-09-17T03:56:46.211650+0000","last_became_peered":"2024-09-17T03:56:46.211650+0000","last_unstale":"2024-09-17T03:56:47.246887+0000","last_undegraded":"2024-09-17T03:56:47.246887+0000","last_fullsized":"2024-09-17T03:56:47.246887+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:37.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.237983+0000","last_change":"2024-09-17T03:56:47.233203+0000","last_active":"2024-09-17T03:56:47.237983+0000","last_peered":"2024-09-17T03:56:47.237983+0000","last_clean":"2024-09-17T03:56:47.237983+0000","last_became_active":"2024-09-17T03:56:47.232364+0000","last_became_peered":"2024-09-17T03:56:47.232364+0000","last_unstale":"2024-09-17T03:56:47.237983+0000","last_undegraded":"2024-09-17T03:56:47.237983+0000","last_fullsized":"2024-09-17T03:56:47.237983+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.701043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983401+0000","last_change":"2024-09-17T03:56:46.216394+0000","last_active":"2024-09-17T03:56:48.983401+0000","last_peered":"2024-09-17T03:56:48.983401+0000","last_clean":"2024-09-17T03:56:48.983401+0000","last_became_active":"2024-09-17T03:56:46.215959+0000","last_became_peered":"2024-09-17T03:56:46.215959+0000","last_unstale":"2024-09-17T03:56:48.983401+0000","last_undegraded":"2024-09-17T03:56:48.983401+0000","last_fullsized":"2024-09-17T03:56:48.983401+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:09.625968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983259+0000","last_change":"2024-09-17T03:56:47.214643+0000","last_active":"2024-09-17T03:56:48.983259+0000","last_peered":"2024-09-17T03:56:48.983259+0000","last_clean":"2024-09-17T03:56:48.983259+0000","last_became_active":"2024-09-17T03:56:47.214313+0000","last_became_peered":"2024-09-17T03:56:47.214313+0000","last_unstale":"2024-09-17T03:56:48.983259+0000","last_undegraded":"2024-09-17T03:56:48.983259+0000","last_fullsized":"2024-09-17T03:56:48.983259+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:47.555750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.242255+0000","last_change":"2024-09-17T03:56:46.200079+0000","last_active":"2024-09-17T03:56:47.242255+0000","last_peered":"2024-09-17T03:56:47.242255+0000","last_clean":"2024-09-17T03:56:47.242255+0000","last_became_active":"2024-09-17T03:56:46.199729+0000","last_became_peered":"2024-09-17T03:56:46.199729+0000","last_unstale":"2024-09-17T03:56:47.242255+0000","last_undegraded":"2024-09-17T03:56:47.242255+0000","last_fullsized":"2024-09-17T03:56:47.242255+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:29.047465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983132+0000","last_change":"2024-09-17T03:56:47.215220+0000","last_active":"2024-09-17T03:56:48.983132+0000","last_peered":"2024-09-17T03:56:48.983132+0000","last_clean":"2024-09-17T03:56:48.983132+0000","last_became_active":"2024-09-17T03:56:47.214916+0000","last_became_peered":"2024-09-17T03:56:47.214916+0000","last_unstale":"2024-09-17T03:56:48.983132+0000","last_undegraded":"2024-09-17T03:56:48.983132+0000","last_fullsized":"2024-09-17T03:56:48.983132+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:35.099254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225382+0000","last_change":"2024-09-17T03:56:46.208123+0000","last_active":"2024-09-17T03:56:48.225382+0000","last_peered":"2024-09-17T03:56:48.225382+0000","last_clean":"2024-09-17T03:56:48.225382+0000","last_became_active":"2024-09-17T03:56:46.207761+0000","last_became_peered":"2024-09-17T03:56:46.207761+0000","last_unstale":"2024-09-17T03:56:48.225382+0000","last_undegraded":"2024-09-17T03:56:48.225382+0000","last_fullsized":"2024-09-17T03:56:48.225382+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:45.606730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224837+0000","last_change":"2024-09-17T03:56:47.220866+0000","last_active":"2024-09-17T03:56:48.224837+0000","last_peered":"2024-09-17T03:56:48.224837+0000","last_clean":"2024-09-17T03:56:48.224837+0000","last_became_active":"2024-09-17T03:56:47.220579+0000","last_became_peered":"2024-09-17T03:56:47.220579+0000","last_unstale":"2024-09-17T03:56:48.224837+0000","last_undegraded":"2024-09-17T03:56:48.224837+0000","last_fullsized":"2024-09-17T03:56:48.224837+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:27.069770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982818+0000","last_change":"2024-09-17T03:56:46.201378+0000","last_active":"2024-09-17T03:56:48.982818+0000","last_peered":"2024-09-17T03:56:48.982818+0000","last_clean":"2024-09-17T03:56:48.982818+0000","last_became_active":"2024-09-17T03:56:46.201039+0000","last_became_peered":"2024-09-17T03:56:46.201039+0000","last_unstale":"2024-09-17T03:56:48.982818+0000","last_undegraded":"2024-09-17T03:56:48.982818+0000","last_fullsized":"2024-09-17T03:56:48.982818+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:52.630528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525143+0000","last_change":"2024-09-17T03:56:47.226448+0000","last_active":"2024-09-17T03:56:48.525143+0000","last_peered":"2024-09-17T03:56:48.525143+0000","last_clean":"2024-09-17T03:56:48.525143+0000","last_became_active":"2024-09-17T03:56:47.225356+0000","last_became_peered":"2024-09-17T03:56:47.225356+0000","last_unstale":"2024-09-17T03:56:48.525143+0000","last_undegraded":"2024-09-17T03:56:48.525143+0000","last_fullsized":"2024-09-17T03:56:48.525143+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:24.859619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982399+0000","last_change":"2024-09-17T03:56:46.213898+0000","last_active":"2024-09-17T03:56:48.982399+0000","last_peered":"2024-09-17T03:56:48.982399+0000","last_clean":"2024-09-17T03:56:48.982399+0000","last_became_active":"2024-09-17T03:56:46.213451+0000","last_became_peered":"2024-09-17T03:56:46.213451+0000","last_unstale":"2024-09-17T03:56:48.982399+0000","last_undegraded":"2024-09-17T03:56:48.982399+0000","last_fullsized":"2024-09-17T03:56:48.982399+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:59.144954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225534+0000","last_change":"2024-09-17T03:56:47.230686+0000","last_active":"2024-09-17T03:56:48.225534+0000","last_peered":"2024-09-17T03:56:48.225534+0000","last_clean":"2024-09-17T03:56:48.225534+0000","last_became_active":"2024-09-17T03:56:47.230375+0000","last_became_peered":"2024-09-17T03:56:47.230375+0000","last_unstale":"2024-09-17T03:56:48.225534+0000","last_undegraded":"2024-09-17T03:56:48.225534+0000","last_fullsized":"2024-09-17T03:56:48.225534+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:36.072940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983156+0000","last_change":"2024-09-17T03:56:46.210732+0000","last_active":"2024-09-17T03:56:48.983156+0000","last_peered":"2024-09-17T03:56:48.983156+0000","last_clean":"2024-09-17T03:56:48.983156+0000","last_became_active":"2024-09-17T03:56:46.210356+0000","last_became_peered":"2024-09-17T03:56:46.210356+0000","last_unstale":"2024-09-17T03:56:48.983156+0000","last_undegraded":"2024-09-17T03:56:48.983156+0000","last_fullsized":"2024-09-17T03:56:48.983156+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:08.915911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224758+0000","last_change":"2024-09-17T03:56:47.223116+0000","last_active":"2024-09-17T03:56:48.224758+0000","last_peered":"2024-09-17T03:56:48.224758+0000","last_clean":"2024-09-17T03:56:48.224758+0000","last_became_active":"2024-09-17T03:56:47.222835+0000","last_became_peered":"2024-09-17T03:56:47.222835+0000","last_unstale":"2024-09-17T03:56:48.224758+0000","last_undegraded":"2024-09-17T03:56:48.224758+0000","last_fullsized":"2024-09-17T03:56:48.224758+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:30.772127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982018+0000","last_change":"2024-09-17T03:56:46.206489+0000","last_active":"2024-09-17T03:56:48.982018+0000","last_peered":"2024-09-17T03:56:48.982018+0000","last_clean":"2024-09-17T03:56:48.982018+0000","last_became_active":"2024-09-17T03:56:46.206092+0000","last_became_peered":"2024-09-17T03:56:46.206092+0000","last_unstale":"2024-09-17T03:56:48.982018+0000","last_undegraded":"2024-09-17T03:56:48.982018+0000","last_fullsized":"2024-09-17T03:56:48.982018+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:33.209440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223790+0000","last_change":"2024-09-17T03:56:47.223917+0000","last_active":"2024-09-17T03:56:48.223790+0000","last_peered":"2024-09-17T03:56:48.223790+0000","last_clean":"2024-09-17T03:56:48.223790+0000","last_became_active":"2024-09-17T03:56:47.221771+0000","last_became_peered":"2024-09-17T03:56:47.221771+0000","last_unstale":"2024-09-17T03:56:48.223790+0000","last_undegraded":"2024-09-17T03:56:48.223790+0000","last_fullsized":"2024-09-17T03:56:48.223790+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:59.360058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.202199+0000","last_change":"2024-09-17T03:56:46.200485+0000","last_active":"2024-09-17T03:56:47.202199+0000","last_peered":"2024-09-17T03:56:47.202199+0000","last_clean":"2024-09-17T03:56:47.202199+0000","last_became_active":"2024-09-17T03:56:46.200119+0000","last_became_peered":"2024-09-17T03:56:46.200119+0000","last_unstale":"2024-09-17T03:56:47.202199+0000","last_undegraded":"2024-09-17T03:56:47.202199+0000","last_fullsized":"2024-09-17T03:56:47.202199+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:59.106619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225528+0000","last_change":"2024-09-17T03:56:47.222566+0000","last_active":"2024-09-17T03:56:48.225528+0000","last_peered":"2024-09-17T03:56:48.225528+0000","last_clean":"2024-09-17T03:56:48.225528+0000","last_became_active":"2024-09-17T03:56:47.222231+0000","last_became_peered":"2024-09-17T03:56:47.222231+0000","last_unstale":"2024-09-17T03:56:48.225528+0000","last_undegraded":"2024-09-17T03:56:48.225528+0000","last_fullsized":"2024-09-17T03:56:48.225528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:16.355625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525708+0000","last_change":"2024-09-17T03:56:46.214653+0000","last_active":"2024-09-17T03:56:48.525708+0000","last_peered":"2024-09-17T03:56:48.525708+0000","last_clean":"2024-09-17T03:56:48.525708+0000","last_became_active":"2024-09-17T03:56:46.213974+0000","last_became_peered":"2024-09-17T03:56:46.213974+0000","last_unstale":"2024-09-17T03:56:48.525708+0000","last_undegraded":"2024-09-17T03:56:48.525708+0000","last_fullsized":"2024-09-17T03:56:48.525708+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:34.003966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525551+0000","last_change":"2024-09-17T03:56:47.222960+0000","last_active":"2024-09-17T03:56:48.525551+0000","last_peered":"2024-09-17T03:56:48.525551+0000","last_clean":"2024-09-17T03:56:48.525551+0000","last_became_active":"2024-09-17T03:56:47.222329+0000","last_became_peered":"2024-09-17T03:56:47.222329+0000","last_unstale":"2024-09-17T03:56:48.525551+0000","last_undegraded":"2024-09-17T03:56:48.525551+0000","last_fullsized":"2024-09-17T03:56:48.525551+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:24.030688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224296+0000","last_change":"2024-09-17T03:56:46.218880+0000","last_active":"2024-09-17T03:56:48.224296+0000","last_peered":"2024-09-17T03:56:48.224296+0000","last_clean":"2024-09-17T03:56:48.224296+0000","last_became_active":"2024-09-17T03:56:46.218495+0000","last_became_peered":"2024-09-17T03:56:46.218495+0000","last_unstale":"2024-09-17T03:56:48.224296+0000","last_undegraded":"2024-09-17T03:56:48.224296+0000","last_fullsized":"2024-09-17T03:56:48.224296+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:46.475056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224744+0000","last_change":"2024-09-17T03:56:47.232278+0000","last_active":"2024-09-17T03:56:48.224744+0000","last_peered":"2024-09-17T03:56:48.224744+0000","last_clean":"2024-09-17T03:56:48.224744+0000","last_became_active":"2024-09-17T03:56:47.230804+0000","last_became_peered":"2024-09-17T03:56:47.230804+0000","last_unstale":"2024-09-17T03:56:48.224744+0000","last_undegraded":"2024-09-17T03:56:48.224744+0000","last_fullsized":"2024-09-17T03:56:48.224744+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:47.080381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224864+0000","last_change":"2024-09-17T03:56:46.215999+0000","last_active":"2024-09-17T03:56:48.224864+0000","last_peered":"2024-09-17T03:56:48.224864+0000","last_clean":"2024-09-17T03:56:48.224864+0000","last_became_active":"2024-09-17T03:56:46.213482+0000","last_became_peered":"2024-09-17T03:56:46.213482+0000","last_unstale":"2024-09-17T03:56:48.224864+0000","last_undegraded":"2024-09-17T03:56:48.224864+0000","last_fullsized":"2024-09-17T03:56:48.224864+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:40.369154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225612+0000","last_change":"2024-09-17T03:56:47.234940+0000","last_active":"2024-09-17T03:56:48.225612+0000","last_peered":"2024-09-17T03:56:48.225612+0000","last_clean":"2024-09-17T03:56:48.225612+0000","last_became_active":"2024-09-17T03:56:47.234637+0000","last_became_peered":"2024-09-17T03:56:47.234637+0000","last_unstale":"2024-09-17T03:56:48.225612+0000","last_undegraded":"2024-09-17T03:56:48.225612+0000","last_fullsized":"2024-09-17T03:56:48.225612+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:50:32.130851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.203438+0000","last_change":"2024-09-17T03:56:46.212907+0000","last_active":"2024-09-17T03:56:47.203438+0000","last_peered":"2024-09-17T03:56:47.203438+0000","last_clean":"2024-09-17T03:56:47.203438+0000","last_became_active":"2024-09-17T03:56:46.212481+0000","last_became_peered":"2024-09-17T03:56:46.212481+0000","last_unstale":"2024-09-17T03:56:47.203438+0000","last_undegraded":"2024-09-17T03:56:47.203438+0000","last_fullsized":"2024-09-17T03:56:47.203438+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:25.876841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224209+0000","last_change":"2024-09-17T03:56:47.226507+0000","last_active":"2024-09-17T03:56:48.224209+0000","last_peered":"2024-09-17T03:56:48.224209+0000","last_clean":"2024-09-17T03:56:48.224209+0000","last_became_active":"2024-09-17T03:56:47.226192+0000","last_became_peered":"2024-09-17T03:56:47.226192+0000","last_unstale":"2024-09-17T03:56:48.224209+0000","last_undegraded":"2024-09-17T03:56:48.224209+0000","last_fullsized":"2024-09-17T03:56:48.224209+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.351473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224495+0000","last_change":"2024-09-17T03:56:46.213326+0000","last_active":"2024-09-17T03:56:48.224495+0000","last_peered":"2024-09-17T03:56:48.224495+0000","last_clean":"2024-09-17T03:56:48.224495+0000","last_became_active":"2024-09-17T03:56:46.213002+0000","last_became_peered":"2024-09-17T03:56:46.213002+0000","last_unstale":"2024-09-17T03:56:48.224495+0000","last_undegraded":"2024-09-17T03:56:48.224495+0000","last_fullsized":"2024-09-17T03:56:48.224495+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:11.449898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982480+0000","last_change":"2024-09-17T03:56:47.207942+0000","last_active":"2024-09-17T03:56:48.982480+0000","last_peered":"2024-09-17T03:56:48.982480+0000","last_clean":"2024-09-17T03:56:48.982480+0000","last_became_active":"2024-09-17T03:56:47.207492+0000","last_became_peered":"2024-09-17T03:56:47.207492+0000","last_unstale":"2024-09-17T03:56:48.982480+0000","last_undegraded":"2024-09-17T03:56:48.982480+0000","last_fullsized":"2024-09-17T03:56:48.982480+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:54:50.291182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224609+0000","last_change":"2024-09-17T03:56:46.219807+0000","last_active":"2024-09-17T03:56:48.224609+0000","last_peered":"2024-09-17T03:56:48.224609+0000","last_clean":"2024-09-17T03:56:48.224609+0000","last_became_active":"2024-09-17T03:56:46.219426+0000","last_became_peered":"2024-09-17T03:56:46.219426+0000","last_unstale":"2024-09-17T03:56:48.224609+0000","last_undegraded":"2024-09-17T03:56:48.224609+0000","last_fullsized":"2024-09-17T03:56:48.224609+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:18:52.040688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982372+0000","last_change":"2024-09-17T03:56:47.234987+0000","last_active":"2024-09-17T03:56:48.982372+0000","last_peered":"2024-09-17T03:56:48.982372+0000","last_clean":"2024-09-17T03:56:48.982372+0000","last_became_active":"2024-09-17T03:56:47.234350+0000","last_became_peered":"2024-09-17T03:56:47.234350+0000","last_unstale":"2024-09-17T03:56:48.982372+0000","last_undegraded":"2024-09-17T03:56:48.982372+0000","last_fullsized":"2024-09-17T03:56:48.982372+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.351473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982293+0000","last_change":"2024-09-17T03:56:46.212792+0000","last_active":"2024-09-17T03:56:48.982293+0000","last_peered":"2024-09-17T03:56:48.982293+0000","last_clean":"2024-09-17T03:56:48.982293+0000","last_became_active":"2024-09-17T03:56:46.212378+0000","last_became_peered":"2024-09-17T03:56:46.212378+0000","last_unstale":"2024-09-17T03:56:48.982293+0000","last_undegraded":"2024-09-17T03:56:48.982293+0000","last_fullsized":"2024-09-17T03:56:48.982293+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:12.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224850+0000","last_change":"2024-09-17T03:56:47.214213+0000","last_active":"2024-09-17T03:56:48.224850+0000","last_peered":"2024-09-17T03:56:48.224850+0000","last_clean":"2024-09-17T03:56:48.224850+0000","last_became_active":"2024-09-17T03:56:47.213931+0000","last_became_peered":"2024-09-17T03:56:47.213931+0000","last_unstale":"2024-09-17T03:56:48.224850+0000","last_undegraded":"2024-09-17T03:56:48.224850+0000","last_fullsized":"2024-09-17T03:56:48.224850+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.616228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.202463+0000","last_change":"2024-09-17T03:56:46.206909+0000","last_active":"2024-09-17T03:56:47.202463+0000","last_peered":"2024-09-17T03:56:47.202463+0000","last_clean":"2024-09-17T03:56:47.202463+0000","last_became_active":"2024-09-17T03:56:46.206559+0000","last_became_peered":"2024-09-17T03:56:46.206559+0000","last_unstale":"2024-09-17T03:56:47.202463+0000","last_undegraded":"2024-09-17T03:56:47.202463+0000","last_fullsized":"2024-09-17T03:56:47.202463+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:11.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224485+0000","last_change":"2024-09-17T03:56:47.225125+0000","last_active":"2024-09-17T03:56:48.224485+0000","last_peered":"2024-09-17T03:56:48.224485+0000","last_clean":"2024-09-17T03:56:48.224485+0000","last_became_active":"2024-09-17T03:56:47.224755+0000","last_became_peered":"2024-09-17T03:56:47.224755+0000","last_unstale":"2024-09-17T03:56:48.224485+0000","last_undegraded":"2024-09-17T03:56:48.224485+0000","last_fullsized":"2024-09-17T03:56:48.224485+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:32.581668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224972+0000","last_change":"2024-09-17T03:56:46.221826+0000","last_active":"2024-09-17T03:56:48.224972+0000","last_peered":"2024-09-17T03:56:48.224972+0000","last_clean":"2024-09-17T03:56:48.224972+0000","last_became_active":"2024-09-17T03:56:46.221489+0000","last_became_peered":"2024-09-17T03:56:46.221489+0000","last_unstale":"2024-09-17T03:56:48.224972+0000","last_undegraded":"2024-09-17T03:56:48.224972+0000","last_fullsized":"2024-09-17T03:56:48.224972+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:12.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":300,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223326+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:56:48.223326+0000","last_peered":"2024-09-17T03:56:48.223326+0000","last_clean":"2024-09-17T03:56:48.223326+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:56:48.223326+0000","last_undegraded":"2024-09-17T03:56:48.223326+0000","last_fullsized":"2024-09-17T03:56:48.223326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509531,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2170880,"data_stored":1264589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":672083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509533,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1556480,"data_stored":672041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509531,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1548288,"data_stored":665831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509533,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":672587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509533,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":671971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542235,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":667283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41600,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330240,"statfs":{"total":96636764160,"available":96594165760,"internally_reserved":0,"allocated":2134016,"data_stored":1256039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:56:51.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:56:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 -- 172.21.15.129:0/2690288865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00b4290 msgr2=0x7f88d00cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 --2- 172.21.15.129:0/2690288865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00b4290 0x7f88d00cf390 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f88bc009830 tx=0x7f88bc02a510 comp rx=0 tx=0).stop 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 -- 172.21.15.129:0/2690288865 shutdown_connections 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 --2- 172.21.15.129:0/2690288865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00cf8d0 0x7f88d00ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 --2- 172.21.15.129:0/2690288865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00b4290 0x7f88d00cf390 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 -- 172.21.15.129:0/2690288865 >> 172.21.15.129:0/2690288865 conn(0x7f88d006d410 msgr2=0x7f88d00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 -- 172.21.15.129:0/2690288865 shutdown_connections 2024-09-17T03:56:51.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.902+0000 7f88d7400640 1 -- 172.21.15.129:0/2690288865 wait complete. 2024-09-17T03:56:51.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.905+0000 7f88d7400640 1 Processor -- start 2024-09-17T03:56:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.905+0000 7f88d7400640 1 -- start start 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00b4290 0x7f88d015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88d00cd370 0x7f88d015b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f88d015ed90 con 0x7f88d00cf8d0 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f88d015ec10 con 0x7f88d00cd370 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d7400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f88d00cae00 con 0x7f88d00b4290 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.906+0000 7f88d5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60716/0 (socket says 172.21.15.129:60716) 2024-09-17T03:56:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 -- 172.21.15.129:0/2403244445 learned_addr learned my addr 172.21.15.129:0/2403244445 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88cfe00640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88d00cd370 0x7f88d015b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d4a00640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00b4290 0x7f88d015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 -- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00b4290 msgr2=0x7f88d015ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00b4290 0x7f88d015ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 -- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88d00cd370 msgr2=0x7f88d015b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88d00cd370 0x7f88d015b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 -- 172.21.15.129:0/2403244445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d00c7890 con 0x7f88d00cf8d0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.907+0000 7f88d5400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f88c800e6a0 tx=0x7f88c800eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.908+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c800cab0 con 0x7f88d00cf8d0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.908+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f88c800cc50 con 0x7f88d00cf8d0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.908+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d00c7d40 con 0x7f88d00cf8d0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.908+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8005280 con 0x7f88d00cf8d0 2024-09-17T03:56:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.908+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d01f63f0 con 0x7f88d00cf8d0 2024-09-17T03:56:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.909+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d00c7600 con 0x7f88d00cf8d0 2024-09-17T03:56:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.910+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f88c8005420 con 0x7f88d00cf8d0 2024-09-17T03:56:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.911+0000 7f88ccc00640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 0x7f88b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.911+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f88c809b4e0 con 0x7f88d00cf8d0 2024-09-17T03:56:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.912+0000 7f88d4a00640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 0x7f88b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.912+0000 7f88d4a00640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 0x7f88b007b0a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88c0001770 tx=0x7f88c0001e70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:51.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:51.915+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f88c800ce20 con 0x7f88d00cf8d0 2024-09-17T03:56:52.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.096+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f88d006a140 con 0x7f88b0078b90 2024-09-17T03:56:52.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.101+0000 7f88ccc00640 1 -- 172.21.15.129:0/2403244445 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7f88d006a140 con 0x7f88b0078b90 2024-09-17T03:56:52.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 msgr2=0x7f88b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 0x7f88b007b0a0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f88c0001770 tx=0x7f88c0001e70 comp rx=0 tx=0).stop 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 msgr2=0x7f88d01f6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f88c800e6a0 tx=0x7f88c800eb60 comp rx=0 tx=0).stop 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 shutdown_connections 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f88b0078b90 0x7f88b007b0a0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88d00cf8d0 0x7f88d01f6910 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88d00cd370 0x7f88d015b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 --2- 172.21.15.129:0/2403244445 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88d00b4290 0x7f88d015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 >> 172.21.15.129:0/2403244445 conn(0x7f88d006d410 msgr2=0x7f88d00d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 shutdown_connections 2024-09-17T03:56:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.105+0000 7f88d7400640 1 -- 172.21.15.129:0/2403244445 wait complete. 2024-09-17T03:56:52.129 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-17T03:56:50.415522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":304436,"kb_used_data":13316,"kb_used_omap":12,"kb_used_meta":290931,"kb_avail":754670284,"statfs":{"total":773094113280,"available":772782370816,"internally_reserved":0,"allocated":13635584,"data_stored":6542424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":297913904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.229432"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525617+0000","last_change":"2024-09-17T03:56:47.222642+0000","last_active":"2024-09-17T03:56:48.525617+0000","last_peered":"2024-09-17T03:56:48.525617+0000","last_clean":"2024-09-17T03:56:48.525617+0000","last_became_active":"2024-09-17T03:56:47.221833+0000","last_became_peered":"2024-09-17T03:56:47.221833+0000","last_unstale":"2024-09-17T03:56:48.525617+0000","last_undegraded":"2024-09-17T03:56:48.525617+0000","last_fullsized":"2024-09-17T03:56:48.525617+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:01.568796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224457+0000","last_change":"2024-09-17T03:56:47.226856+0000","last_active":"2024-09-17T03:56:48.224457+0000","last_peered":"2024-09-17T03:56:48.224457+0000","last_clean":"2024-09-17T03:56:48.224457+0000","last_became_active":"2024-09-17T03:56:47.225956+0000","last_became_peered":"2024-09-17T03:56:47.225956+0000","last_unstale":"2024-09-17T03:56:48.224457+0000","last_undegraded":"2024-09-17T03:56:48.224457+0000","last_fullsized":"2024-09-17T03:56:48.224457+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:11.178671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235698+0000","last_change":"2024-09-17T03:56:47.228360+0000","last_active":"2024-09-17T03:56:47.235698+0000","last_peered":"2024-09-17T03:56:47.235698+0000","last_clean":"2024-09-17T03:56:47.235698+0000","last_became_active":"2024-09-17T03:56:47.224643+0000","last_became_peered":"2024-09-17T03:56:47.224643+0000","last_unstale":"2024-09-17T03:56:47.235698+0000","last_undegraded":"2024-09-17T03:56:47.235698+0000","last_fullsized":"2024-09-17T03:56:47.235698+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:52.533919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525343+0000","last_change":"2024-09-17T03:56:47.215839+0000","last_active":"2024-09-17T03:56:48.525343+0000","last_peered":"2024-09-17T03:56:48.525343+0000","last_clean":"2024-09-17T03:56:48.525343+0000","last_became_active":"2024-09-17T03:56:47.215544+0000","last_became_peered":"2024-09-17T03:56:47.215544+0000","last_unstale":"2024-09-17T03:56:48.525343+0000","last_undegraded":"2024-09-17T03:56:48.525343+0000","last_fullsized":"2024-09-17T03:56:48.525343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:14.487345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224877+0000","last_change":"2024-09-17T03:56:47.222637+0000","last_active":"2024-09-17T03:56:48.224877+0000","last_peered":"2024-09-17T03:56:48.224877+0000","last_clean":"2024-09-17T03:56:48.224877+0000","last_became_active":"2024-09-17T03:56:47.221854+0000","last_became_peered":"2024-09-17T03:56:47.221854+0000","last_unstale":"2024-09-17T03:56:48.224877+0000","last_undegraded":"2024-09-17T03:56:48.224877+0000","last_fullsized":"2024-09-17T03:56:48.224877+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:08.063223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225664+0000","last_change":"2024-09-17T03:56:47.226307+0000","last_active":"2024-09-17T03:56:48.225664+0000","last_peered":"2024-09-17T03:56:48.225664+0000","last_clean":"2024-09-17T03:56:48.225664+0000","last_became_active":"2024-09-17T03:56:47.223792+0000","last_became_peered":"2024-09-17T03:56:47.223792+0000","last_unstale":"2024-09-17T03:56:48.225664+0000","last_undegraded":"2024-09-17T03:56:48.225664+0000","last_fullsized":"2024-09-17T03:56:48.225664+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:56.601035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225497+0000","last_change":"2024-09-17T03:56:47.228003+0000","last_active":"2024-09-17T03:56:48.225497+0000","last_peered":"2024-09-17T03:56:48.225497+0000","last_clean":"2024-09-17T03:56:48.225497+0000","last_became_active":"2024-09-17T03:56:47.227709+0000","last_became_peered":"2024-09-17T03:56:47.227709+0000","last_unstale":"2024-09-17T03:56:48.225497+0000","last_undegraded":"2024-09-17T03:56:48.225497+0000","last_fullsized":"2024-09-17T03:56:48.225497+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:03:55.407940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.227433+0000","last_change":"2024-09-17T03:56:47.219021+0000","last_active":"2024-09-17T03:56:47.227433+0000","last_peered":"2024-09-17T03:56:47.227433+0000","last_clean":"2024-09-17T03:56:47.227433+0000","last_became_active":"2024-09-17T03:56:47.218277+0000","last_became_peered":"2024-09-17T03:56:47.218277+0000","last_unstale":"2024-09-17T03:56:47.227433+0000","last_undegraded":"2024-09-17T03:56:47.227433+0000","last_fullsized":"2024-09-17T03:56:47.227433+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:42.870804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235695+0000","last_change":"2024-09-17T03:56:47.225065+0000","last_active":"2024-09-17T03:56:47.235695+0000","last_peered":"2024-09-17T03:56:47.235695+0000","last_clean":"2024-09-17T03:56:47.235695+0000","last_became_active":"2024-09-17T03:56:47.224774+0000","last_became_peered":"2024-09-17T03:56:47.224774+0000","last_unstale":"2024-09-17T03:56:47.235695+0000","last_undegraded":"2024-09-17T03:56:47.235695+0000","last_fullsized":"2024-09-17T03:56:47.235695+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:02.952951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224700+0000","last_change":"2024-09-17T03:56:47.222081+0000","last_active":"2024-09-17T03:56:48.224700+0000","last_peered":"2024-09-17T03:56:48.224700+0000","last_clean":"2024-09-17T03:56:48.224700+0000","last_became_active":"2024-09-17T03:56:47.221779+0000","last_became_peered":"2024-09-17T03:56:47.221779+0000","last_unstale":"2024-09-17T03:56:48.224700+0000","last_undegraded":"2024-09-17T03:56:48.224700+0000","last_fullsized":"2024-09-17T03:56:48.224700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:52.307088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.233157+0000","last_change":"2024-09-17T03:56:47.228179+0000","last_active":"2024-09-17T03:56:47.233157+0000","last_peered":"2024-09-17T03:56:47.233157+0000","last_clean":"2024-09-17T03:56:47.233157+0000","last_became_active":"2024-09-17T03:56:47.227837+0000","last_became_peered":"2024-09-17T03:56:47.227837+0000","last_unstale":"2024-09-17T03:56:47.233157+0000","last_undegraded":"2024-09-17T03:56:47.233157+0000","last_fullsized":"2024-09-17T03:56:47.233157+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:06.592340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235731+0000","last_change":"2024-09-17T03:56:47.222943+0000","last_active":"2024-09-17T03:56:47.235731+0000","last_peered":"2024-09-17T03:56:47.235731+0000","last_clean":"2024-09-17T03:56:47.235731+0000","last_became_active":"2024-09-17T03:56:47.222620+0000","last_became_peered":"2024-09-17T03:56:47.222620+0000","last_unstale":"2024-09-17T03:56:47.235731+0000","last_undegraded":"2024-09-17T03:56:47.235731+0000","last_fullsized":"2024-09-17T03:56:47.235731+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:18:57.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525194+0000","last_change":"2024-09-17T03:56:47.217383+0000","last_active":"2024-09-17T03:56:48.525194+0000","last_peered":"2024-09-17T03:56:48.525194+0000","last_clean":"2024-09-17T03:56:48.525194+0000","last_became_active":"2024-09-17T03:56:47.215332+0000","last_became_peered":"2024-09-17T03:56:47.215332+0000","last_unstale":"2024-09-17T03:56:48.525194+0000","last_undegraded":"2024-09-17T03:56:48.525194+0000","last_fullsized":"2024-09-17T03:56:48.525194+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:04:18.365741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224479+0000","last_change":"2024-09-17T03:56:47.228101+0000","last_active":"2024-09-17T03:56:48.224479+0000","last_peered":"2024-09-17T03:56:48.224479+0000","last_clean":"2024-09-17T03:56:48.224479+0000","last_became_active":"2024-09-17T03:56:47.227750+0000","last_became_peered":"2024-09-17T03:56:47.227750+0000","last_unstale":"2024-09-17T03:56:48.224479+0000","last_undegraded":"2024-09-17T03:56:48.224479+0000","last_fullsized":"2024-09-17T03:56:48.224479+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:31.667211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525083+0000","last_change":"2024-09-17T03:56:47.220677+0000","last_active":"2024-09-17T03:56:48.525083+0000","last_peered":"2024-09-17T03:56:48.525083+0000","last_clean":"2024-09-17T03:56:48.525083+0000","last_became_active":"2024-09-17T03:56:47.220332+0000","last_became_peered":"2024-09-17T03:56:47.220332+0000","last_unstale":"2024-09-17T03:56:48.525083+0000","last_undegraded":"2024-09-17T03:56:48.525083+0000","last_fullsized":"2024-09-17T03:56:48.525083+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:58.489644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982294+0000","last_change":"2024-09-17T03:56:47.221379+0000","last_active":"2024-09-17T03:56:48.982294+0000","last_peered":"2024-09-17T03:56:48.982294+0000","last_clean":"2024-09-17T03:56:48.982294+0000","last_became_active":"2024-09-17T03:56:47.221019+0000","last_became_peered":"2024-09-17T03:56:47.221019+0000","last_unstale":"2024-09-17T03:56:48.982294+0000","last_undegraded":"2024-09-17T03:56:48.982294+0000","last_fullsized":"2024-09-17T03:56:48.982294+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:34.117078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225622+0000","last_change":"2024-09-17T03:56:47.229815+0000","last_active":"2024-09-17T03:56:48.225622+0000","last_peered":"2024-09-17T03:56:48.225622+0000","last_clean":"2024-09-17T03:56:48.225622+0000","last_became_active":"2024-09-17T03:56:47.229173+0000","last_became_peered":"2024-09-17T03:56:47.229173+0000","last_unstale":"2024-09-17T03:56:48.225622+0000","last_undegraded":"2024-09-17T03:56:48.225622+0000","last_fullsized":"2024-09-17T03:56:48.225622+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:40:55.086311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983199+0000","last_change":"2024-09-17T03:56:47.224084+0000","last_active":"2024-09-17T03:56:48.983199+0000","last_peered":"2024-09-17T03:56:48.983199+0000","last_clean":"2024-09-17T03:56:48.983199+0000","last_became_active":"2024-09-17T03:56:47.223357+0000","last_became_peered":"2024-09-17T03:56:47.223357+0000","last_unstale":"2024-09-17T03:56:48.983199+0000","last_undegraded":"2024-09-17T03:56:48.983199+0000","last_fullsized":"2024-09-17T03:56:48.983199+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:08.958502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.225896+0000","last_change":"2024-09-17T03:56:47.221033+0000","last_active":"2024-09-17T03:56:47.225896+0000","last_peered":"2024-09-17T03:56:47.225896+0000","last_clean":"2024-09-17T03:56:47.225896+0000","last_became_active":"2024-09-17T03:56:47.216921+0000","last_became_peered":"2024-09-17T03:56:47.216921+0000","last_unstale":"2024-09-17T03:56:47.225896+0000","last_undegraded":"2024-09-17T03:56:47.225896+0000","last_fullsized":"2024-09-17T03:56:47.225896+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:01:17.067518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983046+0000","last_change":"2024-09-17T03:56:47.228717+0000","last_active":"2024-09-17T03:56:48.983046+0000","last_peered":"2024-09-17T03:56:48.983046+0000","last_clean":"2024-09-17T03:56:48.983046+0000","last_became_active":"2024-09-17T03:56:47.228384+0000","last_became_peered":"2024-09-17T03:56:47.228384+0000","last_unstale":"2024-09-17T03:56:48.983046+0000","last_undegraded":"2024-09-17T03:56:48.983046+0000","last_fullsized":"2024-09-17T03:56:48.983046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:32.581668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224627+0000","last_change":"2024-09-17T03:56:47.222969+0000","last_active":"2024-09-17T03:56:48.224627+0000","last_peered":"2024-09-17T03:56:48.224627+0000","last_clean":"2024-09-17T03:56:48.224627+0000","last_became_active":"2024-09-17T03:56:47.222255+0000","last_became_peered":"2024-09-17T03:56:47.222255+0000","last_unstale":"2024-09-17T03:56:48.224627+0000","last_undegraded":"2024-09-17T03:56:48.224627+0000","last_fullsized":"2024-09-17T03:56:48.224627+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:35.099254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.236086+0000","last_change":"2024-09-17T03:56:47.231386+0000","last_active":"2024-09-17T03:56:47.236086+0000","last_peered":"2024-09-17T03:56:47.236086+0000","last_clean":"2024-09-17T03:56:47.236086+0000","last_became_active":"2024-09-17T03:56:47.231072+0000","last_became_peered":"2024-09-17T03:56:47.231072+0000","last_unstale":"2024-09-17T03:56:47.236086+0000","last_undegraded":"2024-09-17T03:56:47.236086+0000","last_fullsized":"2024-09-17T03:56:47.236086+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:13.491479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235663+0000","last_change":"2024-09-17T03:56:47.224215+0000","last_active":"2024-09-17T03:56:47.235663+0000","last_peered":"2024-09-17T03:56:47.235663+0000","last_clean":"2024-09-17T03:56:47.235663+0000","last_became_active":"2024-09-17T03:56:47.223097+0000","last_became_peered":"2024-09-17T03:56:47.223097+0000","last_unstale":"2024-09-17T03:56:47.235663+0000","last_undegraded":"2024-09-17T03:56:47.235663+0000","last_fullsized":"2024-09-17T03:56:47.235663+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:35.738186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.218567+0000","last_change":"2024-09-17T03:56:47.211424+0000","last_active":"2024-09-17T03:56:47.218567+0000","last_peered":"2024-09-17T03:56:47.218567+0000","last_clean":"2024-09-17T03:56:47.218567+0000","last_became_active":"2024-09-17T03:56:47.211022+0000","last_became_peered":"2024-09-17T03:56:47.211022+0000","last_unstale":"2024-09-17T03:56:47.218567+0000","last_undegraded":"2024-09-17T03:56:47.218567+0000","last_fullsized":"2024-09-17T03:56:47.218567+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:34.106940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225805+0000","last_change":"2024-09-17T03:56:47.230581+0000","last_active":"2024-09-17T03:56:48.225805+0000","last_peered":"2024-09-17T03:56:48.225805+0000","last_clean":"2024-09-17T03:56:48.225805+0000","last_became_active":"2024-09-17T03:56:47.230229+0000","last_became_peered":"2024-09-17T03:56:47.230229+0000","last_unstale":"2024-09-17T03:56:48.225805+0000","last_undegraded":"2024-09-17T03:56:48.225805+0000","last_fullsized":"2024-09-17T03:56:48.225805+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:26.439048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983202+0000","last_change":"2024-09-17T03:56:47.223680+0000","last_active":"2024-09-17T03:56:48.983202+0000","last_peered":"2024-09-17T03:56:48.983202+0000","last_clean":"2024-09-17T03:56:48.983202+0000","last_became_active":"2024-09-17T03:56:47.222950+0000","last_became_peered":"2024-09-17T03:56:47.222950+0000","last_unstale":"2024-09-17T03:56:48.983202+0000","last_undegraded":"2024-09-17T03:56:48.983202+0000","last_fullsized":"2024-09-17T03:56:48.983202+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:13.849851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525445+0000","last_change":"2024-09-17T03:56:47.220730+0000","last_active":"2024-09-17T03:56:48.525445+0000","last_peered":"2024-09-17T03:56:48.525445+0000","last_clean":"2024-09-17T03:56:48.525445+0000","last_became_active":"2024-09-17T03:56:47.219992+0000","last_became_peered":"2024-09-17T03:56:47.219992+0000","last_unstale":"2024-09-17T03:56:48.525445+0000","last_undegraded":"2024-09-17T03:56:48.525445+0000","last_fullsized":"2024-09-17T03:56:48.525445+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:41.911074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982789+0000","last_change":"2024-09-17T03:56:47.220610+0000","last_active":"2024-09-17T03:56:48.982789+0000","last_peered":"2024-09-17T03:56:48.982789+0000","last_clean":"2024-09-17T03:56:48.982789+0000","last_became_active":"2024-09-17T03:56:47.220327+0000","last_became_peered":"2024-09-17T03:56:47.220327+0000","last_unstale":"2024-09-17T03:56:48.982789+0000","last_undegraded":"2024-09-17T03:56:48.982789+0000","last_fullsized":"2024-09-17T03:56:48.982789+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:35.648336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525514+0000","last_change":"2024-09-17T03:56:47.222410+0000","last_active":"2024-09-17T03:56:48.525514+0000","last_peered":"2024-09-17T03:56:48.525514+0000","last_clean":"2024-09-17T03:56:48.525514+0000","last_became_active":"2024-09-17T03:56:47.222086+0000","last_became_peered":"2024-09-17T03:56:47.222086+0000","last_unstale":"2024-09-17T03:56:48.525514+0000","last_undegraded":"2024-09-17T03:56:48.525514+0000","last_fullsized":"2024-09-17T03:56:48.525514+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:47.217402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525553+0000","last_change":"2024-09-17T03:56:47.224892+0000","last_active":"2024-09-17T03:56:48.525553+0000","last_peered":"2024-09-17T03:56:48.525553+0000","last_clean":"2024-09-17T03:56:48.525553+0000","last_became_active":"2024-09-17T03:56:47.222943+0000","last_became_peered":"2024-09-17T03:56:47.222943+0000","last_unstale":"2024-09-17T03:56:48.525553+0000","last_undegraded":"2024-09-17T03:56:48.525553+0000","last_fullsized":"2024-09-17T03:56:48.525553+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:14.194488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223932+0000","last_change":"2024-09-17T03:56:47.233668+0000","last_active":"2024-09-17T03:56:48.223932+0000","last_peered":"2024-09-17T03:56:48.223932+0000","last_clean":"2024-09-17T03:56:48.223932+0000","last_became_active":"2024-09-17T03:56:47.232545+0000","last_became_peered":"2024-09-17T03:56:47.232545+0000","last_unstale":"2024-09-17T03:56:48.223932+0000","last_undegraded":"2024-09-17T03:56:48.223932+0000","last_fullsized":"2024-09-17T03:56:48.223932+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:34.117078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223634+0000","last_change":"2024-09-17T03:56:47.227702+0000","last_active":"2024-09-17T03:56:48.223634+0000","last_peered":"2024-09-17T03:56:48.223634+0000","last_clean":"2024-09-17T03:56:48.223634+0000","last_became_active":"2024-09-17T03:56:47.227410+0000","last_became_peered":"2024-09-17T03:56:47.227410+0000","last_unstale":"2024-09-17T03:56:48.223634+0000","last_undegraded":"2024-09-17T03:56:48.223634+0000","last_fullsized":"2024-09-17T03:56:48.223634+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:39.333054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983374+0000","last_change":"2024-09-17T03:56:47.220163+0000","last_active":"2024-09-17T03:56:48.983374+0000","last_peered":"2024-09-17T03:56:48.983374+0000","last_clean":"2024-09-17T03:56:48.983374+0000","last_became_active":"2024-09-17T03:56:47.219857+0000","last_became_peered":"2024-09-17T03:56:47.219857+0000","last_unstale":"2024-09-17T03:56:48.983374+0000","last_undegraded":"2024-09-17T03:56:48.983374+0000","last_fullsized":"2024-09-17T03:56:48.983374+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:08.063223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"124'4","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225000+0000","last_change":"2024-09-17T03:56:46.221818+0000","last_active":"2024-09-17T03:56:48.225000+0000","last_peered":"2024-09-17T03:56:48.225000+0000","last_clean":"2024-09-17T03:56:48.225000+0000","last_became_active":"2024-09-17T03:56:46.221498+0000","last_became_peered":"2024-09-17T03:56:46.221498+0000","last_unstale":"2024-09-17T03:56:48.225000+0000","last_undegraded":"2024-09-17T03:56:48.225000+0000","last_fullsized":"2024-09-17T03:56:48.225000+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:33.209440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.237834+0000","last_change":"2024-09-17T03:56:47.233289+0000","last_active":"2024-09-17T03:56:47.237834+0000","last_peered":"2024-09-17T03:56:47.237834+0000","last_clean":"2024-09-17T03:56:47.237834+0000","last_became_active":"2024-09-17T03:56:47.232977+0000","last_became_peered":"2024-09-17T03:56:47.232977+0000","last_unstale":"2024-09-17T03:56:47.237834+0000","last_undegraded":"2024-09-17T03:56:47.237834+0000","last_fullsized":"2024-09-17T03:56:47.237834+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:04.034200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983386+0000","last_change":"2024-09-17T03:56:46.201459+0000","last_active":"2024-09-17T03:56:48.983386+0000","last_peered":"2024-09-17T03:56:48.983386+0000","last_clean":"2024-09-17T03:56:48.983386+0000","last_became_active":"2024-09-17T03:56:46.201121+0000","last_became_peered":"2024-09-17T03:56:46.201121+0000","last_unstale":"2024-09-17T03:56:48.983386+0000","last_undegraded":"2024-09-17T03:56:48.983386+0000","last_fullsized":"2024-09-17T03:56:48.983386+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:48:16.181438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224505+0000","last_change":"2024-09-17T03:56:47.223264+0000","last_active":"2024-09-17T03:56:48.224505+0000","last_peered":"2024-09-17T03:56:48.224505+0000","last_clean":"2024-09-17T03:56:48.224505+0000","last_became_active":"2024-09-17T03:56:47.222113+0000","last_became_peered":"2024-09-17T03:56:47.222113+0000","last_unstale":"2024-09-17T03:56:48.224505+0000","last_undegraded":"2024-09-17T03:56:48.224505+0000","last_fullsized":"2024-09-17T03:56:48.224505+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:52.547094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"124'2","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.242302+0000","last_change":"2024-09-17T03:56:46.208768+0000","last_active":"2024-09-17T03:56:47.242302+0000","last_peered":"2024-09-17T03:56:47.242302+0000","last_clean":"2024-09-17T03:56:47.242302+0000","last_became_active":"2024-09-17T03:56:46.208293+0000","last_became_peered":"2024-09-17T03:56:46.208293+0000","last_unstale":"2024-09-17T03:56:47.242302+0000","last_undegraded":"2024-09-17T03:56:47.242302+0000","last_fullsized":"2024-09-17T03:56:47.242302+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:42.568339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224720+0000","last_change":"2024-09-17T03:56:47.235062+0000","last_active":"2024-09-17T03:56:48.224720+0000","last_peered":"2024-09-17T03:56:48.224720+0000","last_clean":"2024-09-17T03:56:48.224720+0000","last_became_active":"2024-09-17T03:56:47.232532+0000","last_became_peered":"2024-09-17T03:56:47.232532+0000","last_unstale":"2024-09-17T03:56:48.224720+0000","last_undegraded":"2024-09-17T03:56:48.224720+0000","last_fullsized":"2024-09-17T03:56:48.224720+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:26:58.839963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224800+0000","last_change":"2024-09-17T03:56:46.211517+0000","last_active":"2024-09-17T03:56:48.224800+0000","last_peered":"2024-09-17T03:56:48.224800+0000","last_clean":"2024-09-17T03:56:48.224800+0000","last_became_active":"2024-09-17T03:56:46.211115+0000","last_became_peered":"2024-09-17T03:56:46.211115+0000","last_unstale":"2024-09-17T03:56:48.224800+0000","last_undegraded":"2024-09-17T03:56:48.224800+0000","last_fullsized":"2024-09-17T03:56:48.224800+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:20.640316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983120+0000","last_change":"2024-09-17T03:56:47.223609+0000","last_active":"2024-09-17T03:56:48.983120+0000","last_peered":"2024-09-17T03:56:48.983120+0000","last_clean":"2024-09-17T03:56:48.983120+0000","last_became_active":"2024-09-17T03:56:47.223271+0000","last_became_peered":"2024-09-17T03:56:47.223271+0000","last_unstale":"2024-09-17T03:56:48.983120+0000","last_undegraded":"2024-09-17T03:56:48.983120+0000","last_fullsized":"2024-09-17T03:56:48.983120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:30:52.547094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224403+0000","last_change":"2024-09-17T03:56:46.215297+0000","last_active":"2024-09-17T03:56:48.224403+0000","last_peered":"2024-09-17T03:56:48.224403+0000","last_clean":"2024-09-17T03:56:48.224403+0000","last_became_active":"2024-09-17T03:56:46.214916+0000","last_became_peered":"2024-09-17T03:56:46.214916+0000","last_unstale":"2024-09-17T03:56:48.224403+0000","last_undegraded":"2024-09-17T03:56:48.224403+0000","last_fullsized":"2024-09-17T03:56:48.224403+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:39.368830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225548+0000","last_change":"2024-09-17T03:56:47.226911+0000","last_active":"2024-09-17T03:56:48.225548+0000","last_peered":"2024-09-17T03:56:48.225548+0000","last_clean":"2024-09-17T03:56:48.225548+0000","last_became_active":"2024-09-17T03:56:47.226567+0000","last_became_peered":"2024-09-17T03:56:47.226567+0000","last_unstale":"2024-09-17T03:56:48.225548+0000","last_undegraded":"2024-09-17T03:56:48.225548+0000","last_fullsized":"2024-09-17T03:56:48.225548+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:43:00.943053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983024+0000","last_change":"2024-09-17T03:56:46.217856+0000","last_active":"2024-09-17T03:56:48.983024+0000","last_peered":"2024-09-17T03:56:48.983024+0000","last_clean":"2024-09-17T03:56:48.983024+0000","last_became_active":"2024-09-17T03:56:46.217501+0000","last_became_peered":"2024-09-17T03:56:46.217501+0000","last_unstale":"2024-09-17T03:56:48.983024+0000","last_undegraded":"2024-09-17T03:56:48.983024+0000","last_fullsized":"2024-09-17T03:56:48.983024+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:13.329585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.231576+0000","last_change":"2024-09-17T03:56:47.220276+0000","last_active":"2024-09-17T03:56:47.231576+0000","last_peered":"2024-09-17T03:56:47.231576+0000","last_clean":"2024-09-17T03:56:47.231576+0000","last_became_active":"2024-09-17T03:56:47.219915+0000","last_became_peered":"2024-09-17T03:56:47.219915+0000","last_unstale":"2024-09-17T03:56:47.231576+0000","last_undegraded":"2024-09-17T03:56:47.231576+0000","last_fullsized":"2024-09-17T03:56:47.231576+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:13.624271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223869+0000","last_change":"2024-09-17T03:56:46.213836+0000","last_active":"2024-09-17T03:56:48.223869+0000","last_peered":"2024-09-17T03:56:48.223869+0000","last_clean":"2024-09-17T03:56:48.223869+0000","last_became_active":"2024-09-17T03:56:46.213524+0000","last_became_peered":"2024-09-17T03:56:46.213524+0000","last_unstale":"2024-09-17T03:56:48.223869+0000","last_undegraded":"2024-09-17T03:56:48.223869+0000","last_fullsized":"2024-09-17T03:56:48.223869+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:21.705057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224333+0000","last_change":"2024-09-17T03:56:47.229184+0000","last_active":"2024-09-17T03:56:48.224333+0000","last_peered":"2024-09-17T03:56:48.224333+0000","last_clean":"2024-09-17T03:56:48.224333+0000","last_became_active":"2024-09-17T03:56:47.228902+0000","last_became_peered":"2024-09-17T03:56:47.228902+0000","last_unstale":"2024-09-17T03:56:48.224333+0000","last_undegraded":"2024-09-17T03:56:48.224333+0000","last_fullsized":"2024-09-17T03:56:48.224333+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:43.863945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224486+0000","last_change":"2024-09-17T03:56:46.216835+0000","last_active":"2024-09-17T03:56:48.224486+0000","last_peered":"2024-09-17T03:56:48.224486+0000","last_clean":"2024-09-17T03:56:48.224486+0000","last_became_active":"2024-09-17T03:56:46.214002+0000","last_became_peered":"2024-09-17T03:56:46.214002+0000","last_unstale":"2024-09-17T03:56:48.224486+0000","last_undegraded":"2024-09-17T03:56:48.224486+0000","last_fullsized":"2024-09-17T03:56:48.224486+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:57:25.523123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525511+0000","last_change":"2024-09-17T03:56:47.227059+0000","last_active":"2024-09-17T03:56:48.525511+0000","last_peered":"2024-09-17T03:56:48.525511+0000","last_clean":"2024-09-17T03:56:48.525511+0000","last_became_active":"2024-09-17T03:56:47.226583+0000","last_became_peered":"2024-09-17T03:56:47.226583+0000","last_unstale":"2024-09-17T03:56:48.525511+0000","last_undegraded":"2024-09-17T03:56:48.525511+0000","last_fullsized":"2024-09-17T03:56:48.525511+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:00.421206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983285+0000","last_change":"2024-09-17T03:56:46.208694+0000","last_active":"2024-09-17T03:56:48.983285+0000","last_peered":"2024-09-17T03:56:48.983285+0000","last_clean":"2024-09-17T03:56:48.983285+0000","last_became_active":"2024-09-17T03:56:46.207738+0000","last_became_peered":"2024-09-17T03:56:46.207738+0000","last_unstale":"2024-09-17T03:56:48.983285+0000","last_undegraded":"2024-09-17T03:56:48.983285+0000","last_fullsized":"2024-09-17T03:56:48.983285+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:45:08.814944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983131+0000","last_change":"2024-09-17T03:56:47.218716+0000","last_active":"2024-09-17T03:56:48.983131+0000","last_peered":"2024-09-17T03:56:48.983131+0000","last_clean":"2024-09-17T03:56:48.983131+0000","last_became_active":"2024-09-17T03:56:47.217488+0000","last_became_peered":"2024-09-17T03:56:47.217488+0000","last_unstale":"2024-09-17T03:56:48.983131+0000","last_undegraded":"2024-09-17T03:56:48.983131+0000","last_fullsized":"2024-09-17T03:56:48.983131+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:17:25.487664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983350+0000","last_change":"2024-09-17T03:56:46.211293+0000","last_active":"2024-09-17T03:56:48.983350+0000","last_peered":"2024-09-17T03:56:48.983350+0000","last_clean":"2024-09-17T03:56:48.983350+0000","last_became_active":"2024-09-17T03:56:46.210425+0000","last_became_peered":"2024-09-17T03:56:46.210425+0000","last_unstale":"2024-09-17T03:56:48.983350+0000","last_undegraded":"2024-09-17T03:56:48.983350+0000","last_fullsized":"2024-09-17T03:56:48.983350+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:28.941014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224722+0000","last_change":"2024-09-17T03:56:47.225731+0000","last_active":"2024-09-17T03:56:48.224722+0000","last_peered":"2024-09-17T03:56:48.224722+0000","last_clean":"2024-09-17T03:56:48.224722+0000","last_became_active":"2024-09-17T03:56:47.224955+0000","last_became_peered":"2024-09-17T03:56:47.224955+0000","last_unstale":"2024-09-17T03:56:48.224722+0000","last_undegraded":"2024-09-17T03:56:48.224722+0000","last_fullsized":"2024-09-17T03:56:48.224722+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:08.958502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"124'3","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982739+0000","last_change":"2024-09-17T03:56:46.203171+0000","last_active":"2024-09-17T03:56:48.982739+0000","last_peered":"2024-09-17T03:56:48.982739+0000","last_clean":"2024-09-17T03:56:48.982739+0000","last_became_active":"2024-09-17T03:56:46.202846+0000","last_became_peered":"2024-09-17T03:56:46.202846+0000","last_unstale":"2024-09-17T03:56:48.982739+0000","last_undegraded":"2024-09-17T03:56:48.982739+0000","last_fullsized":"2024-09-17T03:56:48.982739+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:54.802176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224974+0000","last_change":"2024-09-17T03:56:47.226840+0000","last_active":"2024-09-17T03:56:48.224974+0000","last_peered":"2024-09-17T03:56:48.224974+0000","last_clean":"2024-09-17T03:56:48.224974+0000","last_became_active":"2024-09-17T03:56:47.226351+0000","last_became_peered":"2024-09-17T03:56:47.226351+0000","last_unstale":"2024-09-17T03:56:48.224974+0000","last_undegraded":"2024-09-17T03:56:48.224974+0000","last_fullsized":"2024-09-17T03:56:48.224974+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:42.612574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"124'6","reported_seq":19,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.245850+0000","last_change":"2024-09-17T03:56:46.217357+0000","last_active":"2024-09-17T03:56:47.245850+0000","last_peered":"2024-09-17T03:56:47.245850+0000","last_clean":"2024-09-17T03:56:47.245850+0000","last_became_active":"2024-09-17T03:56:46.216452+0000","last_became_peered":"2024-09-17T03:56:46.216452+0000","last_unstale":"2024-09-17T03:56:47.245850+0000","last_undegraded":"2024-09-17T03:56:47.245850+0000","last_fullsized":"2024-09-17T03:56:47.245850+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:36.690725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525429+0000","last_change":"2024-09-17T03:56:47.212314+0000","last_active":"2024-09-17T03:56:48.525429+0000","last_peered":"2024-09-17T03:56:48.525429+0000","last_clean":"2024-09-17T03:56:48.525429+0000","last_became_active":"2024-09-17T03:56:47.211998+0000","last_became_peered":"2024-09-17T03:56:47.211998+0000","last_unstale":"2024-09-17T03:56:48.525429+0000","last_undegraded":"2024-09-17T03:56:48.525429+0000","last_fullsized":"2024-09-17T03:56:48.525429+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:04:36.840465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525280+0000","last_change":"2024-09-17T03:56:46.216254+0000","last_active":"2024-09-17T03:56:48.525280+0000","last_peered":"2024-09-17T03:56:48.525280+0000","last_clean":"2024-09-17T03:56:48.525280+0000","last_became_active":"2024-09-17T03:56:46.215966+0000","last_became_peered":"2024-09-17T03:56:46.215966+0000","last_unstale":"2024-09-17T03:56:48.525280+0000","last_undegraded":"2024-09-17T03:56:48.525280+0000","last_fullsized":"2024-09-17T03:56:48.525280+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:47.037513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224373+0000","last_change":"2024-09-17T03:56:47.225951+0000","last_active":"2024-09-17T03:56:48.224373+0000","last_peered":"2024-09-17T03:56:48.224373+0000","last_clean":"2024-09-17T03:56:48.224373+0000","last_became_active":"2024-09-17T03:56:47.225644+0000","last_became_peered":"2024-09-17T03:56:47.225644+0000","last_unstale":"2024-09-17T03:56:48.224373+0000","last_undegraded":"2024-09-17T03:56:48.224373+0000","last_fullsized":"2024-09-17T03:56:48.224373+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:27.107204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224457+0000","last_change":"2024-09-17T03:56:46.217401+0000","last_active":"2024-09-17T03:56:48.224457+0000","last_peered":"2024-09-17T03:56:48.224457+0000","last_clean":"2024-09-17T03:56:48.224457+0000","last_became_active":"2024-09-17T03:56:46.217060+0000","last_became_peered":"2024-09-17T03:56:46.217060+0000","last_unstale":"2024-09-17T03:56:48.224457+0000","last_undegraded":"2024-09-17T03:56:48.224457+0000","last_fullsized":"2024-09-17T03:56:48.224457+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:33:54.856357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.235630+0000","last_change":"2024-09-17T03:56:47.223392+0000","last_active":"2024-09-17T03:56:47.235630+0000","last_peered":"2024-09-17T03:56:47.235630+0000","last_clean":"2024-09-17T03:56:47.235630+0000","last_became_active":"2024-09-17T03:56:47.222436+0000","last_became_peered":"2024-09-17T03:56:47.222436+0000","last_unstale":"2024-09-17T03:56:47.235630+0000","last_undegraded":"2024-09-17T03:56:47.235630+0000","last_fullsized":"2024-09-17T03:56:47.235630+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:13.940824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"124'8","reported_seq":21,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.246887+0000","last_change":"2024-09-17T03:56:46.212150+0000","last_active":"2024-09-17T03:56:47.246887+0000","last_peered":"2024-09-17T03:56:47.246887+0000","last_clean":"2024-09-17T03:56:47.246887+0000","last_became_active":"2024-09-17T03:56:46.211650+0000","last_became_peered":"2024-09-17T03:56:46.211650+0000","last_unstale":"2024-09-17T03:56:47.246887+0000","last_undegraded":"2024-09-17T03:56:47.246887+0000","last_fullsized":"2024-09-17T03:56:47.246887+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:37.049695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":10,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.237983+0000","last_change":"2024-09-17T03:56:47.233203+0000","last_active":"2024-09-17T03:56:47.237983+0000","last_peered":"2024-09-17T03:56:47.237983+0000","last_clean":"2024-09-17T03:56:47.237983+0000","last_became_active":"2024-09-17T03:56:47.232364+0000","last_became_peered":"2024-09-17T03:56:47.232364+0000","last_unstale":"2024-09-17T03:56:47.237983+0000","last_undegraded":"2024-09-17T03:56:47.237983+0000","last_fullsized":"2024-09-17T03:56:47.237983+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:50:54.701043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983401+0000","last_change":"2024-09-17T03:56:46.216394+0000","last_active":"2024-09-17T03:56:48.983401+0000","last_peered":"2024-09-17T03:56:48.983401+0000","last_clean":"2024-09-17T03:56:48.983401+0000","last_became_active":"2024-09-17T03:56:46.215959+0000","last_became_peered":"2024-09-17T03:56:46.215959+0000","last_unstale":"2024-09-17T03:56:48.983401+0000","last_undegraded":"2024-09-17T03:56:48.983401+0000","last_fullsized":"2024-09-17T03:56:48.983401+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:09.625968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983259+0000","last_change":"2024-09-17T03:56:47.214643+0000","last_active":"2024-09-17T03:56:48.983259+0000","last_peered":"2024-09-17T03:56:48.983259+0000","last_clean":"2024-09-17T03:56:48.983259+0000","last_became_active":"2024-09-17T03:56:47.214313+0000","last_became_peered":"2024-09-17T03:56:47.214313+0000","last_unstale":"2024-09-17T03:56:48.983259+0000","last_undegraded":"2024-09-17T03:56:48.983259+0000","last_fullsized":"2024-09-17T03:56:48.983259+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:52:47.555750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"124'2","reported_seq":15,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.242255+0000","last_change":"2024-09-17T03:56:46.200079+0000","last_active":"2024-09-17T03:56:47.242255+0000","last_peered":"2024-09-17T03:56:47.242255+0000","last_clean":"2024-09-17T03:56:47.242255+0000","last_became_active":"2024-09-17T03:56:46.199729+0000","last_became_peered":"2024-09-17T03:56:46.199729+0000","last_unstale":"2024-09-17T03:56:47.242255+0000","last_undegraded":"2024-09-17T03:56:47.242255+0000","last_fullsized":"2024-09-17T03:56:47.242255+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:04:29.047465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983132+0000","last_change":"2024-09-17T03:56:47.215220+0000","last_active":"2024-09-17T03:56:48.983132+0000","last_peered":"2024-09-17T03:56:48.983132+0000","last_clean":"2024-09-17T03:56:48.983132+0000","last_became_active":"2024-09-17T03:56:47.214916+0000","last_became_peered":"2024-09-17T03:56:47.214916+0000","last_unstale":"2024-09-17T03:56:48.983132+0000","last_undegraded":"2024-09-17T03:56:48.983132+0000","last_fullsized":"2024-09-17T03:56:48.983132+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:35.099254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225382+0000","last_change":"2024-09-17T03:56:46.208123+0000","last_active":"2024-09-17T03:56:48.225382+0000","last_peered":"2024-09-17T03:56:48.225382+0000","last_clean":"2024-09-17T03:56:48.225382+0000","last_became_active":"2024-09-17T03:56:46.207761+0000","last_became_peered":"2024-09-17T03:56:46.207761+0000","last_unstale":"2024-09-17T03:56:48.225382+0000","last_undegraded":"2024-09-17T03:56:48.225382+0000","last_fullsized":"2024-09-17T03:56:48.225382+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:45.606730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224837+0000","last_change":"2024-09-17T03:56:47.220866+0000","last_active":"2024-09-17T03:56:48.224837+0000","last_peered":"2024-09-17T03:56:48.224837+0000","last_clean":"2024-09-17T03:56:48.224837+0000","last_became_active":"2024-09-17T03:56:47.220579+0000","last_became_peered":"2024-09-17T03:56:47.220579+0000","last_unstale":"2024-09-17T03:56:48.224837+0000","last_undegraded":"2024-09-17T03:56:48.224837+0000","last_fullsized":"2024-09-17T03:56:48.224837+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:27.069770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982818+0000","last_change":"2024-09-17T03:56:46.201378+0000","last_active":"2024-09-17T03:56:48.982818+0000","last_peered":"2024-09-17T03:56:48.982818+0000","last_clean":"2024-09-17T03:56:48.982818+0000","last_became_active":"2024-09-17T03:56:46.201039+0000","last_became_peered":"2024-09-17T03:56:46.201039+0000","last_unstale":"2024-09-17T03:56:48.982818+0000","last_undegraded":"2024-09-17T03:56:48.982818+0000","last_fullsized":"2024-09-17T03:56:48.982818+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:52.630528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525143+0000","last_change":"2024-09-17T03:56:47.226448+0000","last_active":"2024-09-17T03:56:48.525143+0000","last_peered":"2024-09-17T03:56:48.525143+0000","last_clean":"2024-09-17T03:56:48.525143+0000","last_became_active":"2024-09-17T03:56:47.225356+0000","last_became_peered":"2024-09-17T03:56:47.225356+0000","last_unstale":"2024-09-17T03:56:48.525143+0000","last_undegraded":"2024-09-17T03:56:48.525143+0000","last_fullsized":"2024-09-17T03:56:48.525143+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:24.859619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982399+0000","last_change":"2024-09-17T03:56:46.213898+0000","last_active":"2024-09-17T03:56:48.982399+0000","last_peered":"2024-09-17T03:56:48.982399+0000","last_clean":"2024-09-17T03:56:48.982399+0000","last_became_active":"2024-09-17T03:56:46.213451+0000","last_became_peered":"2024-09-17T03:56:46.213451+0000","last_unstale":"2024-09-17T03:56:48.982399+0000","last_undegraded":"2024-09-17T03:56:48.982399+0000","last_fullsized":"2024-09-17T03:56:48.982399+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:59.144954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225534+0000","last_change":"2024-09-17T03:56:47.230686+0000","last_active":"2024-09-17T03:56:48.225534+0000","last_peered":"2024-09-17T03:56:48.225534+0000","last_clean":"2024-09-17T03:56:48.225534+0000","last_became_active":"2024-09-17T03:56:47.230375+0000","last_became_peered":"2024-09-17T03:56:47.230375+0000","last_unstale":"2024-09-17T03:56:48.225534+0000","last_undegraded":"2024-09-17T03:56:48.225534+0000","last_fullsized":"2024-09-17T03:56:48.225534+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:36.072940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.983156+0000","last_change":"2024-09-17T03:56:46.210732+0000","last_active":"2024-09-17T03:56:48.983156+0000","last_peered":"2024-09-17T03:56:48.983156+0000","last_clean":"2024-09-17T03:56:48.983156+0000","last_became_active":"2024-09-17T03:56:46.210356+0000","last_became_peered":"2024-09-17T03:56:46.210356+0000","last_unstale":"2024-09-17T03:56:48.983156+0000","last_undegraded":"2024-09-17T03:56:48.983156+0000","last_fullsized":"2024-09-17T03:56:48.983156+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:08.915911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224758+0000","last_change":"2024-09-17T03:56:47.223116+0000","last_active":"2024-09-17T03:56:48.224758+0000","last_peered":"2024-09-17T03:56:48.224758+0000","last_clean":"2024-09-17T03:56:48.224758+0000","last_became_active":"2024-09-17T03:56:47.222835+0000","last_became_peered":"2024-09-17T03:56:47.222835+0000","last_unstale":"2024-09-17T03:56:48.224758+0000","last_undegraded":"2024-09-17T03:56:48.224758+0000","last_fullsized":"2024-09-17T03:56:48.224758+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:30.772127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982018+0000","last_change":"2024-09-17T03:56:46.206489+0000","last_active":"2024-09-17T03:56:48.982018+0000","last_peered":"2024-09-17T03:56:48.982018+0000","last_clean":"2024-09-17T03:56:48.982018+0000","last_became_active":"2024-09-17T03:56:46.206092+0000","last_became_peered":"2024-09-17T03:56:46.206092+0000","last_unstale":"2024-09-17T03:56:48.982018+0000","last_undegraded":"2024-09-17T03:56:48.982018+0000","last_fullsized":"2024-09-17T03:56:48.982018+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:33.209440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223790+0000","last_change":"2024-09-17T03:56:47.223917+0000","last_active":"2024-09-17T03:56:48.223790+0000","last_peered":"2024-09-17T03:56:48.223790+0000","last_clean":"2024-09-17T03:56:48.223790+0000","last_became_active":"2024-09-17T03:56:47.221771+0000","last_became_peered":"2024-09-17T03:56:47.221771+0000","last_unstale":"2024-09-17T03:56:48.223790+0000","last_undegraded":"2024-09-17T03:56:48.223790+0000","last_fullsized":"2024-09-17T03:56:48.223790+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:26:59.360058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.202199+0000","last_change":"2024-09-17T03:56:46.200485+0000","last_active":"2024-09-17T03:56:47.202199+0000","last_peered":"2024-09-17T03:56:47.202199+0000","last_clean":"2024-09-17T03:56:47.202199+0000","last_became_active":"2024-09-17T03:56:46.200119+0000","last_became_peered":"2024-09-17T03:56:46.200119+0000","last_unstale":"2024-09-17T03:56:47.202199+0000","last_undegraded":"2024-09-17T03:56:47.202199+0000","last_fullsized":"2024-09-17T03:56:47.202199+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:59.106619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225528+0000","last_change":"2024-09-17T03:56:47.222566+0000","last_active":"2024-09-17T03:56:48.225528+0000","last_peered":"2024-09-17T03:56:48.225528+0000","last_clean":"2024-09-17T03:56:48.225528+0000","last_became_active":"2024-09-17T03:56:47.222231+0000","last_became_peered":"2024-09-17T03:56:47.222231+0000","last_unstale":"2024-09-17T03:56:48.225528+0000","last_undegraded":"2024-09-17T03:56:48.225528+0000","last_fullsized":"2024-09-17T03:56:48.225528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:16.355625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525708+0000","last_change":"2024-09-17T03:56:46.214653+0000","last_active":"2024-09-17T03:56:48.525708+0000","last_peered":"2024-09-17T03:56:48.525708+0000","last_clean":"2024-09-17T03:56:48.525708+0000","last_became_active":"2024-09-17T03:56:46.213974+0000","last_became_peered":"2024-09-17T03:56:46.213974+0000","last_unstale":"2024-09-17T03:56:48.525708+0000","last_undegraded":"2024-09-17T03:56:48.525708+0000","last_fullsized":"2024-09-17T03:56:48.525708+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:34.003966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.525551+0000","last_change":"2024-09-17T03:56:47.222960+0000","last_active":"2024-09-17T03:56:48.525551+0000","last_peered":"2024-09-17T03:56:48.525551+0000","last_clean":"2024-09-17T03:56:48.525551+0000","last_became_active":"2024-09-17T03:56:47.222329+0000","last_became_peered":"2024-09-17T03:56:47.222329+0000","last_unstale":"2024-09-17T03:56:48.525551+0000","last_undegraded":"2024-09-17T03:56:48.525551+0000","last_fullsized":"2024-09-17T03:56:48.525551+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:24.030688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"124'1","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224296+0000","last_change":"2024-09-17T03:56:46.218880+0000","last_active":"2024-09-17T03:56:48.224296+0000","last_peered":"2024-09-17T03:56:48.224296+0000","last_clean":"2024-09-17T03:56:48.224296+0000","last_became_active":"2024-09-17T03:56:46.218495+0000","last_became_peered":"2024-09-17T03:56:46.218495+0000","last_unstale":"2024-09-17T03:56:48.224296+0000","last_undegraded":"2024-09-17T03:56:48.224296+0000","last_fullsized":"2024-09-17T03:56:48.224296+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:46.475056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224744+0000","last_change":"2024-09-17T03:56:47.232278+0000","last_active":"2024-09-17T03:56:48.224744+0000","last_peered":"2024-09-17T03:56:48.224744+0000","last_clean":"2024-09-17T03:56:48.224744+0000","last_became_active":"2024-09-17T03:56:47.230804+0000","last_became_peered":"2024-09-17T03:56:47.230804+0000","last_unstale":"2024-09-17T03:56:48.224744+0000","last_undegraded":"2024-09-17T03:56:48.224744+0000","last_fullsized":"2024-09-17T03:56:48.224744+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:47.080381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224864+0000","last_change":"2024-09-17T03:56:46.215999+0000","last_active":"2024-09-17T03:56:48.224864+0000","last_peered":"2024-09-17T03:56:48.224864+0000","last_clean":"2024-09-17T03:56:48.224864+0000","last_became_active":"2024-09-17T03:56:46.213482+0000","last_became_peered":"2024-09-17T03:56:46.213482+0000","last_unstale":"2024-09-17T03:56:48.224864+0000","last_undegraded":"2024-09-17T03:56:48.224864+0000","last_fullsized":"2024-09-17T03:56:48.224864+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:40.369154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.225612+0000","last_change":"2024-09-17T03:56:47.234940+0000","last_active":"2024-09-17T03:56:48.225612+0000","last_peered":"2024-09-17T03:56:48.225612+0000","last_clean":"2024-09-17T03:56:48.225612+0000","last_became_active":"2024-09-17T03:56:47.234637+0000","last_became_peered":"2024-09-17T03:56:47.234637+0000","last_unstale":"2024-09-17T03:56:48.225612+0000","last_undegraded":"2024-09-17T03:56:48.225612+0000","last_fullsized":"2024-09-17T03:56:48.225612+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:50:32.130851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.203438+0000","last_change":"2024-09-17T03:56:46.212907+0000","last_active":"2024-09-17T03:56:47.203438+0000","last_peered":"2024-09-17T03:56:47.203438+0000","last_clean":"2024-09-17T03:56:47.203438+0000","last_became_active":"2024-09-17T03:56:46.212481+0000","last_became_peered":"2024-09-17T03:56:46.212481+0000","last_unstale":"2024-09-17T03:56:47.203438+0000","last_undegraded":"2024-09-17T03:56:47.203438+0000","last_fullsized":"2024-09-17T03:56:47.203438+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:25.876841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224209+0000","last_change":"2024-09-17T03:56:47.226507+0000","last_active":"2024-09-17T03:56:48.224209+0000","last_peered":"2024-09-17T03:56:48.224209+0000","last_clean":"2024-09-17T03:56:48.224209+0000","last_became_active":"2024-09-17T03:56:47.226192+0000","last_became_peered":"2024-09-17T03:56:47.226192+0000","last_unstale":"2024-09-17T03:56:48.224209+0000","last_undegraded":"2024-09-17T03:56:48.224209+0000","last_fullsized":"2024-09-17T03:56:48.224209+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.351473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224495+0000","last_change":"2024-09-17T03:56:46.213326+0000","last_active":"2024-09-17T03:56:48.224495+0000","last_peered":"2024-09-17T03:56:48.224495+0000","last_clean":"2024-09-17T03:56:48.224495+0000","last_became_active":"2024-09-17T03:56:46.213002+0000","last_became_peered":"2024-09-17T03:56:46.213002+0000","last_unstale":"2024-09-17T03:56:48.224495+0000","last_undegraded":"2024-09-17T03:56:48.224495+0000","last_fullsized":"2024-09-17T03:56:48.224495+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:11.449898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982480+0000","last_change":"2024-09-17T03:56:47.207942+0000","last_active":"2024-09-17T03:56:48.982480+0000","last_peered":"2024-09-17T03:56:48.982480+0000","last_clean":"2024-09-17T03:56:48.982480+0000","last_became_active":"2024-09-17T03:56:47.207492+0000","last_became_peered":"2024-09-17T03:56:47.207492+0000","last_unstale":"2024-09-17T03:56:48.982480+0000","last_undegraded":"2024-09-17T03:56:48.982480+0000","last_fullsized":"2024-09-17T03:56:48.982480+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:54:50.291182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224609+0000","last_change":"2024-09-17T03:56:46.219807+0000","last_active":"2024-09-17T03:56:48.224609+0000","last_peered":"2024-09-17T03:56:48.224609+0000","last_clean":"2024-09-17T03:56:48.224609+0000","last_became_active":"2024-09-17T03:56:46.219426+0000","last_became_peered":"2024-09-17T03:56:46.219426+0000","last_unstale":"2024-09-17T03:56:48.224609+0000","last_undegraded":"2024-09-17T03:56:48.224609+0000","last_fullsized":"2024-09-17T03:56:48.224609+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:18:52.040688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982372+0000","last_change":"2024-09-17T03:56:47.234987+0000","last_active":"2024-09-17T03:56:48.982372+0000","last_peered":"2024-09-17T03:56:48.982372+0000","last_clean":"2024-09-17T03:56:48.982372+0000","last_became_active":"2024-09-17T03:56:47.234350+0000","last_became_peered":"2024-09-17T03:56:47.234350+0000","last_unstale":"2024-09-17T03:56:48.982372+0000","last_undegraded":"2024-09-17T03:56:48.982372+0000","last_fullsized":"2024-09-17T03:56:48.982372+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:18:01.351473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.982293+0000","last_change":"2024-09-17T03:56:46.212792+0000","last_active":"2024-09-17T03:56:48.982293+0000","last_peered":"2024-09-17T03:56:48.982293+0000","last_clean":"2024-09-17T03:56:48.982293+0000","last_became_active":"2024-09-17T03:56:46.212378+0000","last_became_peered":"2024-09-17T03:56:46.212378+0000","last_unstale":"2024-09-17T03:56:48.982293+0000","last_undegraded":"2024-09-17T03:56:48.982293+0000","last_fullsized":"2024-09-17T03:56:48.982293+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:12.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224850+0000","last_change":"2024-09-17T03:56:47.214213+0000","last_active":"2024-09-17T03:56:48.224850+0000","last_peered":"2024-09-17T03:56:48.224850+0000","last_clean":"2024-09-17T03:56:48.224850+0000","last_became_active":"2024-09-17T03:56:47.213931+0000","last_became_peered":"2024-09-17T03:56:47.213931+0000","last_unstale":"2024-09-17T03:56:48.224850+0000","last_undegraded":"2024-09-17T03:56:48.224850+0000","last_fullsized":"2024-09-17T03:56:48.224850+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:46.616228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":13,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-17T03:56:47.202463+0000","last_change":"2024-09-17T03:56:46.206909+0000","last_active":"2024-09-17T03:56:47.202463+0000","last_peered":"2024-09-17T03:56:47.202463+0000","last_clean":"2024-09-17T03:56:47.202463+0000","last_became_active":"2024-09-17T03:56:46.206559+0000","last_became_peered":"2024-09-17T03:56:46.206559+0000","last_unstale":"2024-09-17T03:56:47.202463+0000","last_undegraded":"2024-09-17T03:56:47.202463+0000","last_fullsized":"2024-09-17T03:56:47.202463+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:11.688676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224485+0000","last_change":"2024-09-17T03:56:47.225125+0000","last_active":"2024-09-17T03:56:48.224485+0000","last_peered":"2024-09-17T03:56:48.224485+0000","last_clean":"2024-09-17T03:56:48.224485+0000","last_became_active":"2024-09-17T03:56:47.224755+0000","last_became_peered":"2024-09-17T03:56:47.224755+0000","last_unstale":"2024-09-17T03:56:48.224485+0000","last_undegraded":"2024-09-17T03:56:48.224485+0000","last_fullsized":"2024-09-17T03:56:48.224485+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:46.184680+0000","last_clean_scrub_stamp":"2024-09-17T03:56:46.184680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:32.581668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"124'2","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.224972+0000","last_change":"2024-09-17T03:56:46.221826+0000","last_active":"2024-09-17T03:56:48.224972+0000","last_peered":"2024-09-17T03:56:48.224972+0000","last_clean":"2024-09-17T03:56:48.224972+0000","last_became_active":"2024-09-17T03:56:46.221489+0000","last_became_peered":"2024-09-17T03:56:46.221489+0000","last_unstale":"2024-09-17T03:56:48.224972+0000","last_undegraded":"2024-09-17T03:56:48.224972+0000","last_fullsized":"2024-09-17T03:56:48.224972+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:56:45.175182+0000","last_clean_scrub_stamp":"2024-09-17T03:56:45.175182+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:12.902337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":300,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-17T03:56:48.223326+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:56:48.223326+0000","last_peered":"2024-09-17T03:56:48.223326+0000","last_clean":"2024-09-17T03:56:48.223326+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:56:48.223326+0000","last_undegraded":"2024-09-17T03:56:48.223326+0000","last_fullsized":"2024-09-17T03:56:48.223326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509531,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37468,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334372,"statfs":{"total":96636764160,"available":96598396928,"internally_reserved":0,"allocated":2170880,"data_stored":1264589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":672083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509533,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1556480,"data_stored":672041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509531,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36868,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334972,"statfs":{"total":96636764160,"available":96599011328,"internally_reserved":0,"allocated":1548288,"data_stored":665831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509533,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36872,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334968,"statfs":{"total":96636764160,"available":96599007232,"internally_reserved":0,"allocated":1560576,"data_stored":672587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509533,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":671971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542235,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1552384,"data_stored":667283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542236,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41600,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330240,"statfs":{"total":96636764160,"available":96594165760,"internally_reserved":0,"allocated":2134016,"data_stored":1256039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:56:52.133 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:56:52.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 -- 172.21.15.129:0/804039677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 msgr2=0x7f9238160d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 --2- 172.21.15.129:0/804039677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f9238160d50 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f922c005fd0 tx=0x7f922c02a540 comp rx=0 tx=0).stop 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 -- 172.21.15.129:0/804039677 shutdown_connections 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 --2- 172.21.15.129:0/804039677 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f923806ee70 0x7f923806f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 --2- 172.21.15.129:0/804039677 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f9238160d50 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 -- 172.21.15.129:0/804039677 >> 172.21.15.129:0/804039677 conn(0x7f92380d0120 msgr2=0x7f92380d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.396+0000 7f923e600640 1 -- 172.21.15.129:0/804039677 shutdown_connections 2024-09-17T03:56:52.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.397+0000 7f923e600640 1 -- 172.21.15.129:0/804039677 wait complete. 2024-09-17T03:56:52.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.399+0000 7f923e600640 1 Processor -- start 2024-09-17T03:56:52.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.399+0000 7f923e600640 1 -- start start 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 0x7f92380b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9238072ee0 0x7f92380c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92380cf450 con 0x7f9238156f20 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92380cf2d0 con 0x7f923806ee70 2024-09-17T03:56:52.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.400+0000 7f923e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f92380cf5d0 con 0x7f9238072ee0 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60734/0 (socket says 172.21.15.129:60734) 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 -- 172.21.15.129:0/3348117639 learned_addr learned my addr 172.21.15.129:0/3348117639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923d200640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 0x7f92380b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f9237e00640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9238072ee0 0x7f92380c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 -- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9238072ee0 msgr2=0x7f92380c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9238072ee0 0x7f92380c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 -- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 msgr2=0x7f92380b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 0x7f92380b1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 -- 172.21.15.129:0/3348117639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92380cbbb0 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923d200640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 0x7f92380b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f923dc00640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9230002230 tx=0x7f923000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.401+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230003b70 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.402+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9230003d10 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.402+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92380cc020 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.402+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9230005330 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.402+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92380caa50 con 0x7f9238156f20 2024-09-17T03:56:52.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.403+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9238073ca0 con 0x7f9238156f20 2024-09-17T03:56:52.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.404+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f92300056e0 con 0x7f9238156f20 2024-09-17T03:56:52.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.405+0000 7f9234c00640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 0x7f921807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.405+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f923009ada0 con 0x7f9238156f20 2024-09-17T03:56:52.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.406+0000 7f923d200640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 0x7f921807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.406+0000 7f923d200640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 0x7f921807b0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f922c033000 tx=0x7f922c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:52.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.410+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f923806f7c0 con 0x7f9238156f20 2024-09-17T03:56:52.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.666+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f9238072c20 con 0x7f9238156f20 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.670+0000 7f9234c00640 1 -- 172.21.15.129:0/3348117639 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v34) ==== 190+0+147 (secure 0 0 0) 0x7f9230063140 con 0x7f9238156f20 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:56:52.670 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 msgr2=0x7f921807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 0x7f921807b0d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f922c033000 tx=0x7f922c004730 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 msgr2=0x7f92380c5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9230002230 tx=0x7f923000cb90 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 shutdown_connections 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9218078bc0 0x7f921807b0d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9238156f20 0x7f92380c5a40 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9238072ee0 0x7f92380c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 --2- 172.21.15.129:0/3348117639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f923806ee70 0x7f92380b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.672+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 >> 172.21.15.129:0/3348117639 conn(0x7f92380d0120 msgr2=0x7f92380b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:52.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.673+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 shutdown_connections 2024-09-17T03:56:52.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.673+0000 7f923e600640 1 -- 172.21.15.129:0/3348117639 wait complete. 2024-09-17T03:56:52.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:52.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 -- 172.21.15.129:0/1201358852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00b46e0 msgr2=0x7f47f00b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1201358852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00b46e0 0x7f47f00b4af0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f47f01609e0 tx=0x7f47d802a4b0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 -- 172.21.15.129:0/1201358852 shutdown_connections 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1201358852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00b46e0 0x7f47f00b4af0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1201358852 >> v2:172.21.15.181:3300/0 conn(0x7f47f00cfb60 0x7f47f0160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 -- 172.21.15.129:0/1201358852 >> 172.21.15.129:0/1201358852 conn(0x7f47f006cfc0 msgr2=0x7f47f006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 -- 172.21.15.129:0/1201358852 shutdown_connections 2024-09-17T03:56:52.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.917+0000 7f47f4a00640 1 -- 172.21.15.129:0/1201358852 wait complete. 2024-09-17T03:56:52.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.919+0000 7f47f4a00640 1 Processor -- start 2024-09-17T03:56:52.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.920+0000 7f47f4a00640 1 -- start start 2024-09-17T03:56:52.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47f00b46e0 0x7f47f0157210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 0x7f47f00b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47f015d1f0 con 0x7f47f00cfb60 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47f015d070 con 0x7f47f00d02c0 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47f4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f47f015d370 con 0x7f47f00b46e0 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47e6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47e6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60752/0 (socket says 172.21.15.129:60752) 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.921+0000 7f47e6a00640 1 -- 172.21.15.129:0/1463960910 learned_addr learned my addr 172.21.15.129:0/1463960910 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47ef400640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47f00b46e0 0x7f47f0157210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47efe00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 0x7f47f00b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 -- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47f00b46e0 msgr2=0x7f47f0157210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47f00b46e0 0x7f47f0157210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 -- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 msgr2=0x7f47f00b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 0x7f47f00b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 -- 172.21.15.129:0/1463960910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f01e85a0 con 0x7f47f00cfb60 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47efe00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 0x7f47f00b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e6a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f47d8034040 tx=0x7f47d80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d803e030 con 0x7f47f00cfb60 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47d80047a0 con 0x7f47f00cfb60 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f01e8880 con 0x7f47f00cfb60 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.922+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8035040 con 0x7f47f00cfb60 2024-09-17T03:56:52.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.923+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f01e8e40 con 0x7f47f00cfb60 2024-09-17T03:56:52.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.924+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47f015d8a0 con 0x7f47f00cfb60 2024-09-17T03:56:52.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.925+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f47d8041030 con 0x7f47f00cfb60 2024-09-17T03:56:52.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.926+0000 7f47e7e00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 0x7f47cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:52.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.926+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f47d80bb700 con 0x7f47f00cfb60 2024-09-17T03:56:52.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.927+0000 7f47ef400640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 0x7f47cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:52.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.927+0000 7f47ef400640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 0x7f47cc07b0a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b4450 tx=0x7f47e0008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:52.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:52.931+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f47d8083aa0 con 0x7f47f00cfb60 2024-09-17T03:56:53.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.150+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47f006a140 con 0x7f47f00cfb60 2024-09-17T03:56:53.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.151+0000 7f47e7e00640 1 -- 172.21.15.129:0/1463960910 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5228 (secure 0 0 0) 0x7f47d80833a0 con 0x7f47f00cfb60 2024-09-17T03:56:53.151 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:53.151 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":96,"btime":"2024-09-17T03:56:50:227627+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1550025085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1550025085},{"type":"v1","addr":"172.21.15.181:6837","nonce":1550025085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":7309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3323106480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3323106480},{"type":"v1","addr":"172.21.15.181:6836","nonce":3323106480}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91},{"gid":7318,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:49.713437+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:56:53.151 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 96 2024-09-17T03:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 msgr2=0x7f47cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 0x7f47cc07b0a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f47f00b4450 tx=0x7f47e0008040 comp rx=0 tx=0).stop 2024-09-17T03:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 msgr2=0x7f47f01e9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f47d8034040 tx=0x7f47d80039e0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 shutdown_connections 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47cc078b90 0x7f47cc07b0a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47f00d02c0 0x7f47f00b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47f00cfb60 0x7f47f01e9640 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 --2- 172.21.15.129:0/1463960910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47f00b46e0 0x7f47f0157210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 >> 172.21.15.129:0/1463960910 conn(0x7f47f006cfc0 msgr2=0x7f47f0143910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 shutdown_connections 2024-09-17T03:56:53.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.153+0000 7f47f4a00640 1 -- 172.21.15.129:0/1463960910 wait complete. 2024-09-17T03:56:53.181 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:56:53.180+0000 7f809be00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:56:53.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 96, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:56:49.713437+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7638}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:56:53.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:56:53.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:56:53.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 -- 172.21.15.129:0/357907385 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 msgr2=0x7f3f2415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 --2- 172.21.15.129:0/357907385 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f2415f060 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3f10005fd0 tx=0x7f3f1002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 -- 172.21.15.129:0/357907385 shutdown_connections 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 --2- 172.21.15.129:0/357907385 >> v2:172.21.15.181:3300/0 conn(0x7f3f240cc910 0x7f3f240ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 --2- 172.21.15.129:0/357907385 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f2415f060 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 -- 172.21.15.129:0/357907385 >> 172.21.15.129:0/357907385 conn(0x7f3f2406d410 msgr2=0x7f3f240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 -- 172.21.15.129:0/357907385 shutdown_connections 2024-09-17T03:56:53.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.807+0000 7f3f2a000640 1 -- 172.21.15.129:0/357907385 wait complete. 2024-09-17T03:56:53.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.809+0000 7f3f2a000640 1 Processor -- start 2024-09-17T03:56:53.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.810+0000 7f3f2a000640 1 -- start start 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24070a50 0x7f3f24161c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f241f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f240af070 con 0x7f3f240cc910 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2415f7b0 con 0x7f3f24070a50 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f2a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f240af1f0 con 0x7f3f240cc3e0 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38538/0 (socket says 172.21.15.129:38538) 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f23e00640 1 -- 172.21.15.129:0/436586617 learned_addr learned my addr 172.21.15.129:0/436586617 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f22a00640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f241f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.811+0000 7f3f23400640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24070a50 0x7f3f24161c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 -- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 msgr2=0x7f3f241f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f241f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 -- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24070a50 msgr2=0x7f3f24161c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24070a50 0x7f3f24161c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 -- 172.21.15.129:0/436586617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f24074690 con 0x7f3f240cc910 2024-09-17T03:56:53.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f23e00640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00ea10 tx=0x7f3f1c00eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f1b400640 1 -- 172.21.15.129:0/436586617 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c00cde0 con 0x7f3f240cc910 2024-09-17T03:56:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f1b400640 1 -- 172.21.15.129:0/436586617 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f1c004c10 con 0x7f3f240cc910 2024-09-17T03:56:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.812+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f24074970 con 0x7f3f240cc910 2024-09-17T03:56:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.813+0000 7f3f1b400640 1 -- 172.21.15.129:0/436586617 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c005020 con 0x7f3f240cc910 2024-09-17T03:56:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.813+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2415a070 con 0x7f3f240cc910 2024-09-17T03:56:53.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.815+0000 7f3f1b400640 1 -- 172.21.15.129:0/436586617 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f1c0051c0 con 0x7f3f240cc910 2024-09-17T03:56:53.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.816+0000 7f3f1b400640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 0x7f3f0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.816+0000 7f3f1b400640 1 -- 172.21.15.129:0/436586617 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f3f1c09ba70 con 0x7f3f240cc910 2024-09-17T03:56:53.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.817+0000 7f3f23400640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 0x7f3f0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.817+0000 7f3f31019740 1 Processor -- start 2024-09-17T03:56:53.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.817+0000 7f3f31019740 1 -- start start 2024-09-17T03:56:53.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.817+0000 7f3f23400640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 0x7f3f0407b0a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007fd0 tx=0x7f3f14009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848ba970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x55988488af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x559884814430 con 0x5598844c13b0 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598848045d0 con 0x5598844c9590 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.818+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5598847f3a00 con 0x5598844ea270 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38540/0 (socket says 172.21.15.129:38540) 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 -- 172.21.15.129:0/528655344 learned_addr learned my addr 172.21.15.129:0/528655344 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f23400640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848ba970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f23e00640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x55988488af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 -- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 msgr2=0x5598848ba970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848ba970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 -- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 msgr2=0x55988488af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x55988488af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 -- 172.21.15.129:0/528655344 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598847bc520 con 0x5598844c13b0 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f23400640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848ba970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.819+0000 7f3f22a00640 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f3f10006010 tx=0x7f3f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02d95c880d64b7c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.820+0000 7f3f0f400640 1 -- 172.21.15.129:0/528655344 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10040070 con 0x5598844c13b0 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.820+0000 7f3f0f400640 1 -- 172.21.15.129:0/528655344 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f10004030 con 0x5598844c13b0 2024-09-17T03:56:53.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.820+0000 7f3f0f400640 1 -- 172.21.15.129:0/528655344 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10035040 con 0x5598844c13b0 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.820+0000 7f3f31019740 1 -- 172.21.15.129:0/528655344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 msgr2=0x5598848baeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.820+0000 7f3f31019740 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f3f10006010 tx=0x7f3f100046c0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 -- 172.21.15.129:0/528655344 shutdown_connections 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x55988488af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c13b0 0x5598848baeb0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 --2- 172.21.15.129:0/528655344 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848ba970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 -- 172.21.15.129:0/528655344 >> 172.21.15.129:0/528655344 conn(0x5598844de3c0 msgr2=0x559884890970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 -- 172.21.15.129:0/528655344 shutdown_connections 2024-09-17T03:56:53.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.821+0000 7f3f31019740 1 -- 172.21.15.129:0/528655344 wait complete. 2024-09-17T03:56:53.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.822+0000 7f3f31019740 1 -- start start 2024-09-17T03:56:53.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.822+0000 7f3f31019740 1 Processor -- start 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c13b0 0x5598848b6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55988479a330 con 0x5598844c9590 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598847f7810 con 0x5598844c13b0 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x559884564620 con 0x5598844ea270 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.823+0000 7f3f31019740 10 client.0 _ll_register_callbacks cb 0x55988479f420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38544/0 (socket says 172.21.15.129:38544) 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 -- 172.21.15.129:0/4294138791 learned_addr learned my addr 172.21.15.129:0/4294138791 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f23400640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c13b0 0x5598848b6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f23e00640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 -- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 msgr2=0x5598848cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 -- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c13b0 msgr2=0x5598848b6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c13b0 0x5598848b6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 -- 172.21.15.129:0/4294138791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559884555850 con 0x5598844c9590 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f22a00640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3f1002abc0 tx=0x7f3f1002ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c867780ee2b4d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.824+0000 7f3f03e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f10040040 con 0x5598844c9590 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f3f100035f0 con 0x5598844c9590 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f10004030 con 0x5598844c9590 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f3f10055050 con 0x5598844c9590 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 10 client.7752 fetch_fsmap learned FSMap version 96 2024-09-17T03:56:53.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x5598845cc6b0 con 0x5598844c9590 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f10004030 con 0x5598844c9590 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 96) ==== 3502+0+0 (secure 0 0 0) 0x7f3f10058020 con 0x5598844c9590 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 10 client.7752 fetch_fsmap finished waiting for FSMap version 96 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 10 client.7752 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 20 client.7752 populate_metadata read hostname 'smithi129' 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x5598844a8e40 0x5598848e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 4 client.7752 mds_command: new command op to 7638 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.825+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 --> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559884806520 con 0x5598844a8e40 2024-09-17T03:56:53.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.826+0000 7f3f23400640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x5598844a8e40 0x5598848e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.826+0000 7f3f03e00640 10 client.7752 ms_handle_connect on v2:172.21.15.129:6832/1495946 2024-09-17T03:56:53.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.826+0000 7f3f23400640 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x5598844a8e40 0x5598848e3ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=218008beb37ca83a server_cookie=2c2a05ed452eae1b in_seq=0 out_seq=0 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.827+0000 7f3f03e00640 1 -- 172.21.15.129:0/4294138791 <== mds.0 v2:172.21.15.129:6832/1495946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x559884806520 con 0x5598844a8e40 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.827+0000 7f3f03e00640 10 client.7752 handle_command_reply: tid=0 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.827+0000 7f3f31019740 1 client.7752 shutdown 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.827+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 msgr2=0x5598848b1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.827+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3f1002abc0 tx=0x7f3f1002ad50 comp rx=0 tx=0).stop 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 shutdown_connections 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x5598844a8e40 0x5598848e3ed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844ea270 0x5598848cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b1020 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 --2- 172.21.15.129:0/4294138791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c13b0 0x5598848b6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 >> 172.21.15.129:0/4294138791 conn(0x5598844de3c0 msgr2=0x5598848d8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f03e00640 0 client.7752 ms_handle_reset on v2:172.21.15.129:6832/1495946 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 shutdown_connections 2024-09-17T03:56:53.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 1 -- 172.21.15.129:0/4294138791 wait complete. 2024-09-17T03:56:53.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.828+0000 7f3f31019740 20 client.7752 trim_cache size 0 max 16384 2024-09-17T03:56:53.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.846+0000 7f3f31019740 1 Processor -- start 2024-09-17T03:56:53.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.847+0000 7f3f31019740 1 -- start start 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844de3c0 0x5598848d7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x5598848d84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f242011f0 con 0x5598844c9590 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f241f90f0 con 0x5598844de3c0 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f241f9740 con 0x5598844c13b0 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38558/0 (socket says 172.21.15.129:38558) 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.848+0000 7f3f23e00640 1 -- 172.21.15.129:0/1409783464 learned_addr learned my addr 172.21.15.129:0/1409783464 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f22a00640 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x5598848d84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23400640 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844de3c0 0x5598848d7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 -- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 msgr2=0x5598848d84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x5598848d84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 -- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844de3c0 msgr2=0x5598848d7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844de3c0 0x5598848d7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 -- 172.21.15.129:0/1409783464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598848d8b40 con 0x5598844c9590 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f23e00640 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0637e0 tx=0x7f3f1c0632a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249ed83c5e91de34 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f0ea00640 1 -- 172.21.15.129:0/1409783464 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c07d020 con 0x5598844c9590 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f0ea00640 1 -- 172.21.15.129:0/1409783464 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f1c068280 con 0x5598844c9590 2024-09-17T03:56:53.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.849+0000 7f3f0ea00640 1 -- 172.21.15.129:0/1409783464 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c062cb0 con 0x5598844c9590 2024-09-17T03:56:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 -- 172.21.15.129:0/1409783464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 msgr2=0x5598848b25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c0637e0 tx=0x7f3f1c0632a0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 -- 172.21.15.129:0/1409783464 shutdown_connections 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844c9590 0x5598848b25f0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x5598848d84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 --2- 172.21.15.129:0/1409783464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844de3c0 0x5598848d7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 -- 172.21.15.129:0/1409783464 >> 172.21.15.129:0/1409783464 conn(0x5598848b6830 msgr2=0x5598848b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 -- 172.21.15.129:0/1409783464 shutdown_connections 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.850+0000 7f3f31019740 1 -- 172.21.15.129:0/1409783464 wait complete. 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.851+0000 7f3f31019740 1 -- start start 2024-09-17T03:56:53.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.851+0000 7f3f31019740 1 Processor -- start 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x55988494ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x5598849584d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5598848e8870 con 0x5598844de3c0 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5598848b6c40 con 0x5598844c9590 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5598848b7240 con 0x5598844c13b0 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f31019740 10 client.0 _ll_register_callbacks cb 0x7f3f100056c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38564/0 (socket says 172.21.15.129:38564) 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 -- 172.21.15.129:0/3004552641 learned_addr learned my addr 172.21.15.129:0/3004552641 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:53.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23400640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x55988494ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f22a00640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x5598849584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 -- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 msgr2=0x55988494ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x55988494ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 -- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 msgr2=0x5598849584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x5598849584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23e00640 1 -- 172.21.15.129:0/3004552641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598848ce2e0 con 0x5598844de3c0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f23400640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x55988494ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.853+0000 7f3f22a00640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x5598849584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f23e00640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c063310 tx=0x7f3f1c062ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65af96fc3f064964 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c07d070 con 0x5598844de3c0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f3f1c05f3a0 con 0x5598844de3c0 2024-09-17T03:56:53.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f1c068280 con 0x5598844de3c0 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=96) ==== 24+0+0 (secure 0 0 0) 0x7f3f1c064ad0 con 0x5598844de3c0 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f31019740 10 client.7758 fetch_fsmap learned FSMap version 96 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=96,monmap=2+}) -- 0x5598848aca90 con 0x5598844de3c0 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f1c068280 con 0x5598844de3c0 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.854+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 96) ==== 3502+0+0 (secure 0 0 0) 0x7f3f1c020030 con 0x5598844de3c0 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 10 client.7758 fetch_fsmap finished waiting for FSMap version 96 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 10 client.7758 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 20 client.7758 populate_metadata read hostname 'smithi129' 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55988447e280 0x559884958f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 4 client.7758 mds_command: new command op to 7638 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 --> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] -- command(tid 0: {"prefix": "status"}) -- 0x559884801fd0 con 0x55988447e280 2024-09-17T03:56:53.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f23400640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55988447e280 0x559884958f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f23400640 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55988447e280 0x559884958f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559cad29ebe7ea8e server_cookie=e9a99f62d79b3bc9 in_seq=0 out_seq=0 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.855+0000 7f3f03e00640 10 client.7758 ms_handle_connect on v2:172.21.15.129:6832/1495946 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f03e00640 1 -- 172.21.15.129:0/3004552641 <== mds.0 v2:172.21.15.129:6832/1495946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x559884801fd0 con 0x55988447e280 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f03e00640 10 client.7758 handle_command_reply: tid=0 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 client.7758 shutdown 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 msgr2=0x559884958a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c063310 tx=0x7f3f1c062ce0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 shutdown_connections 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55988447e280 0x559884958f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5598844de3c0 0x559884958a10 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5598844c9590 0x5598849584d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 --2- 172.21.15.129:0/3004552641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5598844c13b0 0x55988494ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f03e00640 0 client.7758 ms_handle_reset on v2:172.21.15.129:6832/1495946 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 >> 172.21.15.129:0/3004552641 conn(0x5598848b6830 msgr2=0x55988488c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 shutdown_connections 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 1 -- 172.21.15.129:0/3004552641 wait complete. 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.856+0000 7f3f31019740 20 client.7758 trim_cache size 0 max 16384 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 7638, 2024-09-17T03:56:53.860 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.6448126370000002, 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 96, 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 125, 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 125, 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.710359124 2024-09-17T03:56:53.861 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 msgr2=0x7f3f0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 0x7f3f0407b0a0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3f14007fd0 tx=0x7f3f14009040 comp rx=0 tx=0).stop 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 msgr2=0x7f3f24159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00ea10 tx=0x7f3f1c00eed0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 shutdown_connections 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f04078b90 0x7f3f0407b0a0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f240cc910 0x7f3f24159230 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f240cc3e0 0x7f3f241f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 --2- 172.21.15.129:0/436586617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f24070a50 0x7f3f24161c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 >> 172.21.15.129:0/436586617 conn(0x7f3f2406d410 msgr2=0x7f3f240cb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 shutdown_connections 2024-09-17T03:56:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:53.857+0000 7f3f2a000640 1 -- 172.21.15.129:0/436586617 wait complete. 2024-09-17T03:56:53.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:56:54.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 -- 172.21.15.129:0/3812919403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 msgr2=0x7f254c0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 --2- 172.21.15.129:0/3812919403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c0b3480 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f254c06dd20 tx=0x7f253402a560 comp rx=0 tx=0).stop 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 -- 172.21.15.129:0/3812919403 shutdown_connections 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 --2- 172.21.15.129:0/3812919403 >> v2:172.21.15.181:3300/0 conn(0x7f254c0b39c0 0x7f254c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 --2- 172.21.15.129:0/3812919403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c0b3480 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 --2- 172.21.15.129:0/3812919403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 -- 172.21.15.129:0/3812919403 >> 172.21.15.129:0/3812919403 conn(0x7f254c06d410 msgr2=0x7f254c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 -- 172.21.15.129:0/3812919403 shutdown_connections 2024-09-17T03:56:54.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.102+0000 7f2551800640 1 -- 172.21.15.129:0/3812919403 wait complete. 2024-09-17T03:56:54.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.104+0000 7f2551800640 1 Processor -- start 2024-09-17T03:56:54.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.105+0000 7f2551800640 1 -- start start 2024-09-17T03:56:54.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:54.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f254c0b39c0 0x7f254c154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:54.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:54.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f254c070850 con 0x7f254c0b2950 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f254c0706d0 con 0x7f254c0b39c0 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f2551800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f254c06f410 con 0x7f254c0ce170 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f254aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f254aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38582/0 (socket says 172.21.15.129:38582) 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.106+0000 7f254aa00640 1 -- 172.21.15.129:0/2067309861 learned_addr learned my addr 172.21.15.129:0/2067309861 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254b400640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:54.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f2543e00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f254c0b39c0 0x7f254c154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:54.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 -- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 msgr2=0x7f254c155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:54.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 -- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f254c0b39c0 msgr2=0x7f254c154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f254c0b39c0 0x7f254c154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 -- 172.21.15.129:0/2067309861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c1571b0 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254b400640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f254aa00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e7c0 tx=0x7f253c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c00cc60 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f253c00ce20 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c157490 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.107+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c005280 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.108+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c159080 con 0x7f254c0b2950 2024-09-17T03:56:54.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.109+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c1592e0 con 0x7f254c0b2950 2024-09-17T03:56:54.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.111+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f253c005420 con 0x7f254c0b2950 2024-09-17T03:56:54.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.112+0000 7f2542a00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 0x7f252807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:54.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.112+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f253c09b950 con 0x7f254c0b2950 2024-09-17T03:56:54.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.113+0000 7f2543e00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 0x7f252807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:54.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.113+0000 7f2543e00640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 0x7f252807b0a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2534033000 tx=0x7f2534004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:54.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.115+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f253c063f00 con 0x7f254c0b2950 2024-09-17T03:56:54.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.342+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f254c06f8f0 con 0x7f254c0b2950 2024-09-17T03:56:54.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.343+0000 7f2542a00640 1 -- 172.21.15.129:0/2067309861 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v96) ==== 53+0+73 (secure 0 0 0) 0x7f253c063800 con 0x7f254c0b2950 2024-09-17T03:56:54.343 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.345+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 msgr2=0x7f252807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.345+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 0x7f252807b0a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2534033000 tx=0x7f2534004730 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.345+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 msgr2=0x7f254c1540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.345+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f253c00e7c0 tx=0x7f253c00ec80 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 shutdown_connections 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2528078b90 0x7f252807b0a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f254c0ce170 0x7f254c155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f254c0b39c0 0x7f254c154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 --2- 172.21.15.129:0/2067309861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f254c0b2950 0x7f254c1540e0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 >> 172.21.15.129:0/2067309861 conn(0x7f254c06d410 msgr2=0x7f254c072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:54.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 shutdown_connections 2024-09-17T03:56:54.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54.346+0000 7f2551800640 1 -- 172.21.15.129:0/2067309861 wait complete. 2024-09-17T03:56:54.371 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:56:54.372 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:56:54.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.373 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:56:54.408 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:56:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:56:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:56:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.409 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:56:54.411 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:56:54.412 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42278sec preferred_lft 42278sec 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:56:54.413 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:56:54.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.414 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:56:54.502 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:56:54.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.505 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:56:54.541 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:56:54.542 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:56:54.542 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:56:54.542 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:56:54.542 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:56:54.542 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:56:54.542 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:56:54.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:56:54.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:56:54.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:56:54.543 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:56:54.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:56:54.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:56:54.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.739 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:56:54.774 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:56:54.774 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:56:54.775 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:56:54.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.775 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:56:54.870 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:56:54.871 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:56:54.906 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:56:54.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:56:54.907 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:56:54.981 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:56:54.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:54.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:56:55.176 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:56:55.176 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:56:55.176 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:56:55.197 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:56:55.197 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:56:55.197 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:56:55.197 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:56:55.197 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:56:55.228 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:56:55.228 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:56:55.228 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:56:55.228 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:56:55.229 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:56:55.229 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:56:55.229 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:56:55.229 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:56:55.230 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:56:55.230 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:56:55.231 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:56:55.231 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:56:55.231 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:56:55.233 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:56:55.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:56:55.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:56:55.291 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:56:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:56:55.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:56:55.387 INFO:teuthology.orchestra.run.smithi129.stdout:client7435 2024-09-17T03:56:55.387 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:56:55.387 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:56:55.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-09-17T03:56:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 -- 172.21.15.129:0/3048183725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 msgr2=0x7fad480b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 --2- 172.21.15.129:0/3048183725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad480b2050 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fad34009830 tx=0x7fad3402a500 comp rx=0 tx=0).stop 2024-09-17T03:56:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 -- 172.21.15.129:0/3048183725 shutdown_connections 2024-09-17T03:56:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 --2- 172.21.15.129:0/3048183725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad480b2050 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 --2- 172.21.15.129:0/3048183725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad48074800 0x7fad480af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.686+0000 7fad4ee00640 1 -- 172.21.15.129:0/3048183725 >> 172.21.15.129:0/3048183725 conn(0x7fad480ccc40 msgr2=0x7fad480cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.687+0000 7fad4ee00640 1 -- 172.21.15.129:0/3048183725 shutdown_connections 2024-09-17T03:56:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.687+0000 7fad4ee00640 1 -- 172.21.15.129:0/3048183725 wait complete. 2024-09-17T03:56:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.689+0000 7fad4ee00640 1 Processor -- start 2024-09-17T03:56:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.689+0000 7fad4ee00640 1 -- start start 2024-09-17T03:56:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 0x7fad4814f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad48074800 0x7fad4809aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad480b41a0 con 0x7fad480b1c40 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fad480b4020 con 0x7fad48074800 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.690+0000 7fad4ee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fad480b4320 con 0x7fad4806eea0 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38608/0 (socket says 172.21.15.129:38608) 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 -- 172.21.15.129:0/2848901236 learned_addr learned my addr 172.21.15.129:0/2848901236 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad47e00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 0x7fad4814f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad47400640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad48074800 0x7fad4809aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 -- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 msgr2=0x7fad4814f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 0x7fad4814f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 -- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad48074800 msgr2=0x7fad4809aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad48074800 0x7fad4809aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad4ce00640 1 -- 172.21.15.129:0/2848901236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4809b520 con 0x7fad480b1c40 2024-09-17T03:56:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.691+0000 7fad47e00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 0x7fad4814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:56:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad4ce00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c730 tx=0x7fad4000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad40003b70 con 0x7fad480b1c40 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad40003d10 con 0x7fad480b1c40 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4809b800 con 0x7fad480b1c40 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad400052d0 con 0x7fad480b1c40 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.692+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4809bdc0 con 0x7fad480b1c40 2024-09-17T03:56:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.693+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4806fb30 con 0x7fad480b1c40 2024-09-17T03:56:55.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.694+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fad40005470 con 0x7fad480b1c40 2024-09-17T03:56:55.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.696+0000 7fad3fe00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 0x7fad2807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:55.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.696+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7fad4009ab50 con 0x7fad480b1c40 2024-09-17T03:56:55.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.697+0000 7fad47e00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 0x7fad2807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:55.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.697+0000 7fad47e00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 0x7fad2807b0a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fad38009b80 tx=0x7fad38005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:55.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.701+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fad480cbaf0 con 0x7fad480b1c40 2024-09-17T03:56:55.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:55.923+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7fad4809b2b0 con 0x7fad480b1c40 2024-09-17T03:56:56.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.262+0000 7fad3fe00640 1 -- 172.21.15.129:0/2848901236 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v97) ==== 105+0+0 (secure 0 0 0) 0x7fad40062f80 con 0x7fad480b1c40 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.264+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 msgr2=0x7fad2807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.264+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 0x7fad2807b0a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fad38009b80 tx=0x7fad38005f90 comp rx=0 tx=0).stop 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.264+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 msgr2=0x7fad4809d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.264+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c730 tx=0x7fad4000cb30 comp rx=0 tx=0).stop 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 shutdown_connections 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad28078b90 0x7fad2807b0a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad480b1c40 0x7fad4809d000 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad48074800 0x7fad4809aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 --2- 172.21.15.129:0/2848901236 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad4806eea0 0x7fad4814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 >> 172.21.15.129:0/2848901236 conn(0x7fad480ccc40 msgr2=0x7fad480cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 shutdown_connections 2024-09-17T03:56:56.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.265+0000 7fad4ee00640 1 -- 172.21.15.129:0/2848901236 wait complete. 2024-09-17T03:56:56.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:56:56.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 -- 172.21.15.129:0/1690031603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 msgr2=0x7f946015ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 --2- 172.21.15.129:0/1690031603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f946015ae00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f945400a940 tx=0x7f945402b8f0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 -- 172.21.15.129:0/1690031603 shutdown_connections 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 --2- 172.21.15.129:0/1690031603 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f946015ae00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 -- 172.21.15.129:0/1690031603 >> 172.21.15.129:0/1690031603 conn(0x7f946006d410 msgr2=0x7f94600b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 -- 172.21.15.129:0/1690031603 shutdown_connections 2024-09-17T03:56:56.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.514+0000 7f9468800640 1 -- 172.21.15.129:0/1690031603 wait complete. 2024-09-17T03:56:56.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.517+0000 7f9468800640 1 Processor -- start 2024-09-17T03:56:56.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.517+0000 7f9468800640 1 -- start start 2024-09-17T03:56:56.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:56.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946015b340 0x7f946009dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:56.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946015a1b0 0x7f94600ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f946006fd50 con 0x7f94600af4c0 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9460071170 con 0x7f946015a1b0 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9468800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f946006fed0 con 0x7f946015b340 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9465e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9465e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38612/0 (socket says 172.21.15.129:38612) 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.518+0000 7f9465e00640 1 -- 172.21.15.129:0/1273339568 learned_addr learned my addr 172.21.15.129:0/1273339568 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9466800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946015a1b0 0x7f94600ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:56.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465400640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946015b340 0x7f946009dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 -- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946015b340 msgr2=0x7f946009dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946015b340 0x7f946009dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 -- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946015a1b0 msgr2=0x7f94600ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946015a1b0 0x7f94600ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 -- 172.21.15.129:0/1273339568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94601f63f0 con 0x7f94600af4c0 2024-09-17T03:56:56.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f9465e00640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f945402bee0 tx=0x7f94540042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.519+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454005c90 con 0x7f94600af4c0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.520+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f945402e070 con 0x7f94600af4c0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.520+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94601f8e00 con 0x7f94600af4c0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.520+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454005570 con 0x7f94600af4c0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.520+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94601f98e0 con 0x7f94600af4c0 2024-09-17T03:56:56.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.521+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9460070350 con 0x7f94600af4c0 2024-09-17T03:56:56.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.522+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f94540047a0 con 0x7f94600af4c0 2024-09-17T03:56:56.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.523+0000 7f945a000640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 0x7f944407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:56:56.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.523+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f94540bc5e0 con 0x7f94600af4c0 2024-09-17T03:56:56.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.523+0000 7f9465400640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 0x7f944407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:56:56.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.524+0000 7f9465400640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 0x7f944407b0a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f946015aa30 tx=0x7f9450009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:56:56.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.527+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9454084330 con 0x7f94600af4c0 2024-09-17T03:56:56.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.746+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f946006a140 con 0x7f94600af4c0 2024-09-17T03:56:56.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.747+0000 7f945a000640 1 -- 172.21.15.129:0/1273339568 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+5244 (secure 0 0 0) 0x7f9454088d60 con 0x7f94600af4c0 2024-09-17T03:56:56.747 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:56:56.747 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":98,"btime":"2024-09-17T03:56:56:266393+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1550025085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1550025085},{"type":"v1","addr":"172.21.15.181:6837","nonce":1550025085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":7309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3323106480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3323106480},{"type":"v1","addr":"172.21.15.181:6836","nonce":3323106480}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:56.266363+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:creating","state_seq":1,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:56:56.747 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 98 2024-09-17T03:56:56.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 msgr2=0x7f944407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 0x7f944407b0a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f946015aa30 tx=0x7f9450009040 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 msgr2=0x7f94600c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f945402bee0 tx=0x7f94540042f0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 shutdown_connections 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9444078b90 0x7f944407b0a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f946015a1b0 0x7f94600ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f946015b340 0x7f946009dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 --2- 172.21.15.129:0/1273339568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94600af4c0 0x7f94600c6b30 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 >> 172.21.15.129:0/1273339568 conn(0x7f946006d410 msgr2=0x7f94600cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 shutdown_connections 2024-09-17T03:56:56.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:56:56.749+0000 7f9468800640 1 -- 172.21.15.129:0/1273339568 wait complete. 2024-09-17T03:56:56.772 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-09-17T03:56:58.181 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T03:56:58.181+0000 7f809be00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-17T03:57:01.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:02.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 -- 172.21.15.129:0/343222336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4214072d30 msgr2=0x7f421415ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 --2- 172.21.15.129:0/343222336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4214072d30 0x7f421415ed50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f420c009ca0 tx=0x7f420c02aab0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 -- 172.21.15.129:0/343222336 shutdown_connections 2024-09-17T03:57:02.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 --2- 172.21.15.129:0/343222336 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4214072d30 0x7f421415ed50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 --2- 172.21.15.129:0/343222336 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42140d0bc0 0x7f421415e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 -- 172.21.15.129:0/343222336 >> 172.21.15.129:0/343222336 conn(0x7f421406cfc0 msgr2=0x7f421406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:02.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 -- 172.21.15.129:0/343222336 shutdown_connections 2024-09-17T03:57:02.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.007+0000 7f4213200640 1 -- 172.21.15.129:0/343222336 wait complete. 2024-09-17T03:57:02.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.009+0000 7f4213200640 1 Processor -- start 2024-09-17T03:57:02.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.009+0000 7f4213200640 1 -- start start 2024-09-17T03:57:02.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4214072d30 0x7f421414acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42140d0bc0 0x7f4214151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4214165910 con 0x7f421415bb00 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4214165790 con 0x7f42140d0bc0 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4213200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4214165a90 con 0x7f4214072d30 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4212800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4212800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38614/0 (socket says 172.21.15.129:38614) 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4212800640 1 -- 172.21.15.129:0/2671856252 learned_addr learned my addr 172.21.15.129:0/2671856252 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4211e00640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4214072d30 0x7f421414acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.011+0000 7f4209400640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42140d0bc0 0x7f4214151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 -- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4214072d30 msgr2=0x7f421414acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4214072d30 0x7f421414acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 -- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42140d0bc0 msgr2=0x7f4214151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42140d0bc0 0x7f4214151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 -- 172.21.15.129:0/2671856252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421415a640 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4212800640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f420c009ca0 tx=0x7f420c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c044070 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f420c004040 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c004430 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.012+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4214151310 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.013+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42141518d0 con 0x7f421415bb00 2024-09-17T03:57:02.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.014+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42140d04b0 con 0x7f421415bb00 2024-09-17T03:57:02.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.015+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f420c047070 con 0x7f421415bb00 2024-09-17T03:57:02.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.016+0000 7f420aa00640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 0x7f41f007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.016+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f420c0bbb30 con 0x7f421415bb00 2024-09-17T03:57:02.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.016+0000 7f4211e00640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 0x7f41f007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.017+0000 7f4211e00640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 0x7f41f007b0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4204001720 tx=0x7f4204009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:02.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.021+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f420c032030 con 0x7f421415bb00 2024-09-17T03:57:02.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.238+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f421415a3d0 con 0x7f421415bb00 2024-09-17T03:57:02.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.239+0000 7f420aa00640 1 -- 172.21.15.129:0/2671856252 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5247 (secure 0 0 0) 0x7f420c083910 con 0x7f421415bb00 2024-09-17T03:57:02.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:02.239 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":99,"btime":"2024-09-17T03:56:57:268866+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1550025085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1550025085},{"type":"v1","addr":"172.21.15.181:6837","nonce":1550025085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":7309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3323106480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3323106480},{"type":"v1","addr":"172.21.15.181:6836","nonce":3323106480}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:57.268858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7318,7638]},"id":9}]} 2024-09-17T03:57:02.239 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 99 2024-09-17T03:57:02.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 msgr2=0x7f41f007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 0x7f41f007b0d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4204001720 tx=0x7f4204009040 comp rx=0 tx=0).stop 2024-09-17T03:57:02.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 msgr2=0x7f421415bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f420c009ca0 tx=0x7f420c003ce0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 shutdown_connections 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f41f0078bc0 0x7f41f007b0d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f421415bb00 0x7f421415bf40 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42140d0bc0 0x7f4214151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 --2- 172.21.15.129:0/2671856252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4214072d30 0x7f421414acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 >> 172.21.15.129:0/2671856252 conn(0x7f421406cfc0 msgr2=0x7f42140cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 shutdown_connections 2024-09-17T03:57:02.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.241+0000 7f4213200640 1 -- 172.21.15.129:0/2671856252 wait complete. 2024-09-17T03:57:02.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.490+0000 7f20ae000640 1 -- 172.21.15.129:0/1583223141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 msgr2=0x7f20a80cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.490+0000 7f20ae000640 1 --2- 172.21.15.129:0/1583223141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80cf390 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f20a80c3d50 tx=0x7f209002a530 comp rx=0 tx=0).stop 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 -- 172.21.15.129:0/1583223141 shutdown_connections 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 --2- 172.21.15.129:0/1583223141 >> v2:172.21.15.181:3300/0 conn(0x7f20a80cf8d0 0x7f20a80ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 --2- 172.21.15.129:0/1583223141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80cf390 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 -- 172.21.15.129:0/1583223141 >> 172.21.15.129:0/1583223141 conn(0x7f20a806d410 msgr2=0x7f20a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 -- 172.21.15.129:0/1583223141 shutdown_connections 2024-09-17T03:57:02.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.491+0000 7f20ae000640 1 -- 172.21.15.129:0/1583223141 wait complete. 2024-09-17T03:57:02.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.493+0000 7f20ae000640 1 Processor -- start 2024-09-17T03:57:02.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.493+0000 7f20ae000640 1 -- start start 2024-09-17T03:57:02.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20a80cd370 0x7f20a8159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a80cf8d0 0x7f20a8073ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20a815ed90 con 0x7f20a80b4290 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20a815ec10 con 0x7f20a80cd370 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.494+0000 7f20ae000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f20a80cae00 con 0x7f20a80cf8d0 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38636/0 (socket says 172.21.15.129:38636) 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 -- 172.21.15.129:0/489320667 learned_addr learned my addr 172.21.15.129:0/489320667 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f209fe00640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20a80cd370 0x7f20a8159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7e00640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a80cf8d0 0x7f20a8073ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 -- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a80cf8d0 msgr2=0x7f20a8073ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a80cf8d0 0x7f20a8073ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 -- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20a80cd370 msgr2=0x7f20a8159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20a80cd370 0x7f20a8159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 -- 172.21.15.129:0/489320667 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a815b590 con 0x7f20a80b4290 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.495+0000 7f20a7400640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f209800c730 tx=0x7f209800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.496+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098016020 con 0x7f20a80b4290 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.496+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f209800a950 con 0x7f20a80b4290 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.496+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209800ada0 con 0x7f20a80b4290 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.496+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a809e010 con 0x7f20a80b4290 2024-09-17T03:57:02.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.496+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a80c9e20 con 0x7f20a80b4290 2024-09-17T03:57:02.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.497+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a8050ff0 con 0x7f20a80b4290 2024-09-17T03:57:02.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.498+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2098019020 con 0x7f20a80b4290 2024-09-17T03:57:02.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.499+0000 7f209f400640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 0x7f208407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:02.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.499+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f209809a7c0 con 0x7f20a80b4290 2024-09-17T03:57:02.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.500+0000 7f209fe00640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 0x7f208407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:02.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.500+0000 7f209fe00640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 0x7f208407b0a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2090034000 tx=0x7f2090004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:02.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.504+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f20a8050ff0 con 0x7f20a80b4290 2024-09-17T03:57:02.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.720+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20a80702e0 con 0x7f20a80b4290 2024-09-17T03:57:02.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.721+0000 7f209f400640 1 -- 172.21.15.129:0/489320667 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5247 (secure 0 0 0) 0x7f20a80702e0 con 0x7f20a80b4290 2024-09-17T03:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:02.721 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":99,"btime":"2024-09-17T03:56:57:268866+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6479,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1550025085","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1550025085},{"type":"v1","addr":"172.21.15.181:6837","nonce":1550025085}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":90},{"gid":7309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3323106480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3323106480},{"type":"v1","addr":"172.21.15.181:6836","nonce":3323106480}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:57.268858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7318,7638]},"id":9}]} 2024-09-17T03:57:02.722 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 99 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 msgr2=0x7f208407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 0x7f208407b0a0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f2090034000 tx=0x7f2090004730 comp rx=0 tx=0).stop 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 msgr2=0x7f20a80c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f209800c730 tx=0x7f209800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 shutdown_connections 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2084078b90 0x7f208407b0a0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a80cf8d0 0x7f20a8073ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20a80cd370 0x7f20a8159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 --2- 172.21.15.129:0/489320667 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a80b4290 0x7f20a80c7300 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.723+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 >> 172.21.15.129:0/489320667 conn(0x7f20a806d410 msgr2=0x7f20a806eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:02.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.724+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 shutdown_connections 2024-09-17T03:57:02.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.724+0000 7f20ae000640 1 -- 172.21.15.129:0/489320667 wait complete. 2024-09-17T03:57:02.747 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are c and a 2024-09-17T03:57:02.747 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:57:02.748 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:57:02.772 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:57:02.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 -- 172.21.15.129:0/264206256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 msgr2=0x7f4a240cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 --2- 172.21.15.129:0/264206256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a240cd860 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009830 tx=0x7f4a1402a510 comp rx=0 tx=0).stop 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 -- 172.21.15.129:0/264206256 shutdown_connections 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 --2- 172.21.15.129:0/264206256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a24160af0 0x7f4a240b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 --2- 172.21.15.129:0/264206256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a240cd860 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 -- 172.21.15.129:0/264206256 >> 172.21.15.129:0/264206256 conn(0x7f4a2406cfc0 msgr2=0x7f4a2406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 -- 172.21.15.129:0/264206256 shutdown_connections 2024-09-17T03:57:02.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.995+0000 7f4a28a00640 1 -- 172.21.15.129:0/264206256 wait complete. 2024-09-17T03:57:02.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.998+0000 7f4a28a00640 1 Processor -- start 2024-09-17T03:57:02.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.998+0000 7f4a28a00640 1 -- start start 2024-09-17T03:57:03.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 0x7f4a24155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a24160af0 0x7f4a24155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2415cf70 con 0x7f4a240cf8a0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2415cdf0 con 0x7f4a24160af0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a28a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2415d0f0 con 0x7f4a240d03a0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38658/0 (socket says 172.21.15.129:38658) 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:02.999+0000 7f4a23400640 1 -- 172.21.15.129:0/934623778 learned_addr learned my addr 172.21.15.129:0/934623778 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a1aa00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 0x7f4a24155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23e00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a24160af0 0x7f4a24155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 -- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 msgr2=0x7f4a24155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 0x7f4a24155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 -- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a24160af0 msgr2=0x7f4a24155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a24160af0 0x7f4a24155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 -- 172.21.15.129:0/934623778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a241f5df0 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a1aa00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 0x7f4a24155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a23400640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009950 tx=0x7f4a14033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14031d60 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a14033e10 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a241ea150 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.000+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a14005260 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.001+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a241ea710 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.002+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a240cece0 con 0x7f4a240cf8a0 2024-09-17T03:57:03.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.003+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4a14005400 con 0x7f4a240cf8a0 2024-09-17T03:57:03.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.004+0000 7f4a1be00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 0x7f4a0007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.004+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(125..125 src has 1..125) ==== 5709+0+0 (secure 0 0 0) 0x7f4a140bbee0 con 0x7f4a240cf8a0 2024-09-17T03:57:03.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.004+0000 7f4a1aa00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 0x7f4a0007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.005+0000 7f4a1aa00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 0x7f4a0007b0d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4a2415e600 tx=0x7f4a0c005f10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:03.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.008+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4a240cece0 con 0x7f4a240cf8a0 2024-09-17T03:57:03.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.265+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7f4a2415d5a0 con 0x7f4a240cf8a0 2024-09-17T03:57:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.485+0000 7f4a1be00640 1 -- 172.21.15.129:0/934623778 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6479 v100) ==== 95+0+0 (secure 0 0 0) 0x7f4a1403f030 con 0x7f4a240cf8a0 2024-09-17T03:57:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6479 2024-09-17T03:57:03.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 msgr2=0x7f4a0007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 0x7f4a0007b0d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4a2415e600 tx=0x7f4a0c005f10 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 msgr2=0x7f4a2415bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009950 tx=0x7f4a14033c20 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 shutdown_connections 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a00078bc0 0x7f4a0007b0d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a24160af0 0x7f4a24155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a240d03a0 0x7f4a24155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 --2- 172.21.15.129:0/934623778 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a240cf8a0 0x7f4a2415bbd0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.487+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 >> 172.21.15.129:0/934623778 conn(0x7f4a2406cfc0 msgr2=0x7f4a2415e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.488+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 shutdown_connections 2024-09-17T03:57:03.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.488+0000 7f4a28a00640 1 -- 172.21.15.129:0/934623778 wait complete. 2024-09-17T03:57:03.517 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:57:03.517 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:57:03.542 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:57:03.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 -- 172.21.15.129:0/2493176094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00b3b90 msgr2=0x7fbfa00cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 --2- 172.21.15.129:0/2493176094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00b3b90 0x7fbfa00cbbf0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009830 tx=0x7fbf8802a510 comp rx=0 tx=0).stop 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 -- 172.21.15.129:0/2493176094 shutdown_connections 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 --2- 172.21.15.129:0/2493176094 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa00b2950 0x7fbfa006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 --2- 172.21.15.129:0/2493176094 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00b3b90 0x7fbfa00cbbf0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.762+0000 7fbfa6800640 1 -- 172.21.15.129:0/2493176094 >> 172.21.15.129:0/2493176094 conn(0x7fbfa006d410 msgr2=0x7fbfa00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.763+0000 7fbfa6800640 1 -- 172.21.15.129:0/2493176094 shutdown_connections 2024-09-17T03:57:03.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.763+0000 7fbfa6800640 1 -- 172.21.15.129:0/2493176094 wait complete. 2024-09-17T03:57:03.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.765+0000 7fbfa6800640 1 Processor -- start 2024-09-17T03:57:03.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.765+0000 7fbfa6800640 1 -- start start 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 0x7fbfa00b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 0x7fbfa015e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa006f410 con 0x7fbfa00ce170 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa0070910 con 0x7fbfa00b2950 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.766+0000 7fbfa6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa006f590 con 0x7fbfa0157120 2024-09-17T03:57:03.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38676/0 (socket says 172.21.15.129:38676) 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 -- 172.21.15.129:0/611203490 learned_addr learned my addr 172.21.15.129:0/611203490 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf9fe00640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 0x7fbfa015e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf9f400640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 0x7fbfa00b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 -- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 msgr2=0x7fbfa015e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 0x7fbfa015e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 -- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 msgr2=0x7fbfa00b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 0x7fbfa00b1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 -- 172.21.15.129:0/611203490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa0158c30 con 0x7fbfa00ce170 2024-09-17T03:57:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf9fe00640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 0x7fbfa015e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf9f400640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 0x7fbfa00b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.767+0000 7fbf97e00640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbf8802ab00 tx=0x7fbf88033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.768+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88005c90 con 0x7fbfa00ce170 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.768+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf88031e50 con 0x7fbfa00ce170 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.768+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88005260 con 0x7fbfa00ce170 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.768+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa01580b0 con 0x7fbfa00ce170 2024-09-17T03:57:03.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.768+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa0158670 con 0x7fbfa00ce170 2024-09-17T03:57:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.769+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa00cc850 con 0x7fbfa00ce170 2024-09-17T03:57:03.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.770+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbf88005400 con 0x7fbfa00ce170 2024-09-17T03:57:03.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.771+0000 7fbf97400640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 0x7fbf7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:03.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.771+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 5709+0+0 (secure 0 0 0) 0x7fbf880bbe60 con 0x7fbfa00ce170 2024-09-17T03:57:03.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.772+0000 7fbf9f400640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 0x7fbf7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:03.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.773+0000 7fbf9f400640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 0x7fbf7c07b0a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbf90001220 tx=0x7fbf90009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:03.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:03.776+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbfa00cc850 con 0x7fbfa00ce170 2024-09-17T03:57:04.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.031+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fbfa00c3ab0 con 0x7fbfa00ce170 2024-09-17T03:57:04.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.505+0000 7fbf97400640 1 -- 172.21.15.129:0/611203490 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7309 v101) ==== 95+0+0 (secure 0 0 0) 0x7fbfa00c3ab0 con 0x7fbfa00ce170 2024-09-17T03:57:04.506 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 7309 2024-09-17T03:57:04.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 msgr2=0x7fbf7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:04.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 0x7fbf7c07b0a0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbf90001220 tx=0x7fbf90009040 comp rx=0 tx=0).stop 2024-09-17T03:57:04.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 msgr2=0x7fbfa01569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fbf8802ab00 tx=0x7fbf88033c20 comp rx=0 tx=0).stop 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 shutdown_connections 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf7c078b90 0x7fbf7c07b0a0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbfa0157120 0x7fbfa015e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbfa00ce170 0x7fbfa01569e0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 --2- 172.21.15.129:0/611203490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbfa00b2950 0x7fbfa00b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 >> 172.21.15.129:0/611203490 conn(0x7fbfa006d410 msgr2=0x7fbfa00ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 shutdown_connections 2024-09-17T03:57:04.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.508+0000 7fbfa6800640 1 -- 172.21.15.129:0/611203490 wait complete. 2024-09-17T03:57:04.534 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c ' 2024-09-17T03:57:04.534 DEBUG:teuthology.orchestra.run.smithi129:> mkdir pin 2024-09-17T03:57:04.534 DEBUG:teuthology.orchestra.run.smithi129:> touch pin/placeholder 2024-09-17T03:57:04.534 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:57:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:57:04.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:57:04.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:57:04.634 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 pin) 2024-09-17T03:57:04.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:04.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 -- 172.21.15.129:0/4189783977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b8068d70 msgr2=0x7f65b80b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 --2- 172.21.15.129:0/4189783977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b8068d70 0x7f65b80b3960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f65a8009be0 tx=0x7f65a802a840 comp rx=0 tx=0).stop 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 -- 172.21.15.129:0/4189783977 shutdown_connections 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 --2- 172.21.15.129:0/4189783977 >> v2:172.21.15.181:3300/0 conn(0x7f65b815c950 0x7f65b8071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 --2- 172.21.15.129:0/4189783977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b8068d70 0x7f65b80b3960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.934+0000 7f65bd200640 1 -- 172.21.15.129:0/4189783977 >> 172.21.15.129:0/4189783977 conn(0x7f65b80d0a70 msgr2=0x7f65b80d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.935+0000 7f65bd200640 1 -- 172.21.15.129:0/4189783977 shutdown_connections 2024-09-17T03:57:04.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.935+0000 7f65bd200640 1 -- 172.21.15.129:0/4189783977 wait complete. 2024-09-17T03:57:04.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.937+0000 7f65bd200640 1 Processor -- start 2024-09-17T03:57:04.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.937+0000 7f65bd200640 1 -- start start 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65b8068d70 0x7f65b80a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b80719d0 0x7f65b81f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65b80cce20 con 0x7f65b815c950 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f65b80ccca0 con 0x7f65b8068d70 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.938+0000 7f65bd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f65b80ccfa0 con 0x7f65b80719d0 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44132/0 (socket says 172.21.15.129:44132) 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 -- 172.21.15.129:0/1148029416 learned_addr learned my addr 172.21.15.129:0/1148029416 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b5e00640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65b8068d70 0x7f65b80a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:04.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b5400640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b80719d0 0x7f65b81f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 -- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b80719d0 msgr2=0x7f65b81f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b80719d0 0x7f65b81f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 -- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65b8068d70 msgr2=0x7f65b80a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65b8068d70 0x7f65b80a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 -- 172.21.15.129:0/1148029416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65b80c4640 con 0x7f65b815c950 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.939+0000 7f65b6800640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c3c0 tx=0x7f65b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.940+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0017070 con 0x7f65b815c950 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.940+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65b00053c0 con 0x7f65b815c950 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.940+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0003850 con 0x7f65b815c950 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.940+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65b8071710 con 0x7f65b815c950 2024-09-17T03:57:04.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.940+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65b80c4b90 con 0x7f65b815c950 2024-09-17T03:57:04.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.941+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65b80b3960 con 0x7f65b815c950 2024-09-17T03:57:04.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.942+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f65b001a070 con 0x7f65b815c950 2024-09-17T03:57:04.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.943+0000 7f65ae000640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 0x7f659807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:04.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.943+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f65b0099fe0 con 0x7f65b815c950 2024-09-17T03:57:04.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.944+0000 7f65b5e00640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 0x7f659807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:04.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.944+0000 7f65b5e00640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 0x7f659807b0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f65a8034000 tx=0x7f65a8004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:04.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:04.947+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f65b0061d30 con 0x7f65b815c950 2024-09-17T03:57:05.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.164+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65b80b2120 con 0x7f65b815c950 2024-09-17T03:57:05.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.165+0000 7f65ae000640 1 -- 172.21.15.129:0/1148029416 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+3593 (secure 0 0 0) 0x7f65b0066760 con 0x7f65b815c950 2024-09-17T03:57:05.165 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:05.165 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":101,"btime":"2024-09-17T03:57:04:496084+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:57.268858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7318,7638]},"id":9}]} 2024-09-17T03:57:05.165 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 101 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 msgr2=0x7f659807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 0x7f659807b0d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f65a8034000 tx=0x7f65a8004ab0 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 msgr2=0x7f65b8157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c3c0 tx=0x7f65b000c880 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 shutdown_connections 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6598078bc0 0x7f659807b0d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65b815c950 0x7f65b8157ab0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65b80719d0 0x7f65b81f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 --2- 172.21.15.129:0/1148029416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65b8068d70 0x7f65b80a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 >> 172.21.15.129:0/1148029416 conn(0x7f65b80d0a70 msgr2=0x7f65b81591e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:05.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 shutdown_connections 2024-09-17T03:57:05.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:05.167+0000 7f65bd200640 1 -- 172.21.15.129:0/1148029416 wait complete. 2024-09-17T03:57:05.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:57:05.430 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:57:05.430 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:57:05.431 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-09-17T03:57:07.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:07.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 -- 172.21.15.129:0/980605082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c15b340 msgr2=0x7f6c7c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 --2- 172.21.15.129:0/980605082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0ccc80 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c06f050 tx=0x7f6c6802a5c0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 -- 172.21.15.129:0/980605082 shutdown_connections 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 --2- 172.21.15.129:0/980605082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0ccc80 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 --2- 172.21.15.129:0/980605082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 0x7f6c7c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 -- 172.21.15.129:0/980605082 >> 172.21.15.129:0/980605082 conn(0x7f6c7c06d410 msgr2=0x7f6c7c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 -- 172.21.15.129:0/980605082 shutdown_connections 2024-09-17T03:57:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.662+0000 7f6c81200640 1 -- 172.21.15.129:0/980605082 wait complete. 2024-09-17T03:57:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.665+0000 7f6c81200640 1 Processor -- start 2024-09-17T03:57:07.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.665+0000 7f6c81200640 1 -- start start 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 0x7f6c7c0c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c0710e0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c070f60 con 0x7f6c7c15b340 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c81200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c7c15ed50 con 0x7f6c7c0af4c0 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c79400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c79400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44144/0 (socket says 172.21.15.129:44144) 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c79400640 1 -- 172.21.15.129:0/1329374914 learned_addr learned my addr 172.21.15.129:0/1329374914 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.666+0000 7f6c79e00640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 0x7f6c7c0c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c7a800640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 -- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 msgr2=0x7f6c7c0c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:07.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 0x7f6c7c0c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 -- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c7c15b340 msgr2=0x7f6c7c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:07.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 -- 172.21.15.129:0/1329374914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c15a3c0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.667+0000 7f6c79400640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6c68034040 tx=0x7f6c68003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.668+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6803e030 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.668+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c68003e00 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.668+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7c0c7040 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.668+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c68035040 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.668+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7c0c7600 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.669+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c7c15f1d0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.670+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c68041030 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.672+0000 7f6c72000640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 0x7f6c5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:07.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.672+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f6c680bb6a0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.673+0000 7f6c79e00640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 0x7f6c5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:07.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.673+0000 7f6c79e00640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 0x7f6c5c07b0a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005fd0 tx=0x7f6c6c00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:07.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.676+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c680833f0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.897+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c7c0c6dd0 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.898+0000 7f6c72000640 1 -- 172.21.15.129:0/1329374914 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+3593 (secure 0 0 0) 0x7f6c68087e20 con 0x7f6c7c0cd1c0 2024-09-17T03:57:07.898 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:07.898 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":101,"btime":"2024-09-17T03:57:04:496084+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:57.268858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7318,7638]},"id":9}]} 2024-09-17T03:57:07.898 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 101 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.900+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 msgr2=0x7f6c5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.900+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 0x7f6c5c07b0a0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c005fd0 tx=0x7f6c6c00b2e0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.900+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 msgr2=0x7f6c7c0c8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.900+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6c68034040 tx=0x7f6c68003910 comp rx=0 tx=0).stop 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 shutdown_connections 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c5c078b90 0x7f6c5c07b0a0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c7c15b340 0x7f6c7c0c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c7c0cd1c0 0x7f6c7c0c8b50 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 --2- 172.21.15.129:0/1329374914 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c7c0af4c0 0x7f6c7c0c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:07.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 >> 172.21.15.129:0/1329374914 conn(0x7f6c7c06d410 msgr2=0x7f6c7c073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:07.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 shutdown_connections 2024-09-17T03:57:07.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:07.901+0000 7f6c81200640 1 -- 172.21.15.129:0/1329374914 wait complete. 2024-09-17T03:57:07.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:57:08.162 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:57:08.163 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:57:08.163 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-09-17T03:57:10.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 -- 172.21.15.129:0/1463459622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980af4c0 msgr2=0x7f3f9815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 --2- 172.21.15.129:0/1463459622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980af4c0 0x7f3f9815ae00 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3f9806f050 tx=0x7f3f8c02a520 comp rx=0 tx=0).stop 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 -- 172.21.15.129:0/1463459622 shutdown_connections 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 --2- 172.21.15.129:0/1463459622 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 0x7f3f980ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 --2- 172.21.15.129:0/1463459622 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980af4c0 0x7f3f9815ae00 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 -- 172.21.15.129:0/1463459622 >> 172.21.15.129:0/1463459622 conn(0x7f3f9806d410 msgr2=0x7f3f980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 -- 172.21.15.129:0/1463459622 shutdown_connections 2024-09-17T03:57:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.396+0000 7f3fa0000640 1 -- 172.21.15.129:0/1463459622 wait complete. 2024-09-17T03:57:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.398+0000 7f3fa0000640 1 Processor -- start 2024-09-17T03:57:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.399+0000 7f3fa0000640 1 -- start start 2024-09-17T03:57:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 0x7f3f9814c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 0x7f3f981f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f980710e0 con 0x7f3f980cd1c0 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f98070f60 con 0x7f3f980af4c0 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3fa0000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f9815ed50 con 0x7f3f9815b340 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3f9cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3f9cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44164/0 (socket says 172.21.15.129:44164) 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.400+0000 7f3f9cc00640 1 -- 172.21.15.129:0/3628218491 learned_addr learned my addr 172.21.15.129:0/3628218491 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9e000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 0x7f3f981f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9d600640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 0x7f3f9814c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 -- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 msgr2=0x7f3f981f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 0x7f3f981f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 -- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 msgr2=0x7f3f9814c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 0x7f3f9814c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 -- 172.21.15.129:0/3628218491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f980c68d0 con 0x7f3f980cd1c0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9d600640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 0x7f3f9814c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f9cc00640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b440 tx=0x7f3f8400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84015020 con 0x7f3f980cd1c0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f84011980 con 0x7f3f980cd1c0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84011dd0 con 0x7f3f980cd1c0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.401+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f981f6a70 con 0x7f3f980cd1c0 2024-09-17T03:57:10.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.402+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f981f9760 con 0x7f3f980cd1c0 2024-09-17T03:57:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.403+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f981f67e0 con 0x7f3f980cd1c0 2024-09-17T03:57:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.404+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f84018020 con 0x7f3f980cd1c0 2024-09-17T03:57:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.405+0000 7f3f95600640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 0x7f3f7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.405+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f3f8409ae60 con 0x7f3f980cd1c0 2024-09-17T03:57:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.405+0000 7f3f9d600640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 0x7f3f7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:10.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.406+0000 7f3f9d600640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 0x7f3f7807b0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c033000 tx=0x7f3f8c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.409+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3f8401f020 con 0x7f3f980cd1c0 2024-09-17T03:57:10.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.631+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f981f9550 con 0x7f3f980cd1c0 2024-09-17T03:57:10.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.631+0000 7f3f95600640 1 -- 172.21.15.129:0/3628218491 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) ==== 95+0+3593 (secure 0 0 0) 0x7f3f8400d040 con 0x7f3f980cd1c0 2024-09-17T03:57:10.632 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:10.632 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":101,"btime":"2024-09-17T03:57:04:496084+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:56:57.268858+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7318,7638]},"id":9}]} 2024-09-17T03:57:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 101 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 msgr2=0x7f3f7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 0x7f3f7807b0a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c033000 tx=0x7f3f8c004730 comp rx=0 tx=0).stop 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 msgr2=0x7f3f981f9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400b440 tx=0x7f3f8400b900 comp rx=0 tx=0).stop 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 shutdown_connections 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f78078b90 0x7f3f7807b0a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f9815b340 0x7f3f981f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f980cd1c0 0x7f3f981f9f60 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 --2- 172.21.15.129:0/3628218491 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f980af4c0 0x7f3f9814c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 >> 172.21.15.129:0/3628218491 conn(0x7f3f9806d410 msgr2=0x7f3f980ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 shutdown_connections 2024-09-17T03:57:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:10.634+0000 7f3fa0000640 1 -- 172.21.15.129:0/3628218491 wait complete. 2024-09-17T03:57:10.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:57:10.896 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:57:10.896 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:57:10.897 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/pin', 1)] 2024-09-17T03:57:12.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:13.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 -- 172.21.15.129:0/2050290796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03200705a0 msgr2=0x7f03200b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:13.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 --2- 172.21.15.129:0/2050290796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03200705a0 0x7f03200b4140 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f0310009830 tx=0x7f031002a510 comp rx=0 tx=0).stop 2024-09-17T03:57:13.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 -- 172.21.15.129:0/2050290796 shutdown_connections 2024-09-17T03:57:13.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 --2- 172.21.15.129:0/2050290796 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f032006eed0 0x7f03200ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 --2- 172.21.15.129:0/2050290796 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03200705a0 0x7f03200b4140 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 -- 172.21.15.129:0/2050290796 >> 172.21.15.129:0/2050290796 conn(0x7f032006d410 msgr2=0x7f03200b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:13.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 -- 172.21.15.129:0/2050290796 shutdown_connections 2024-09-17T03:57:13.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.135+0000 7f0326400640 1 -- 172.21.15.129:0/2050290796 wait complete. 2024-09-17T03:57:13.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.138+0000 7f0326400640 1 Processor -- start 2024-09-17T03:57:13.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.138+0000 7f0326400640 1 -- start start 2024-09-17T03:57:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f032006f2e0 0x7f03201f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03200705a0 0x7f03200c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0320160200 con 0x7f032006eed0 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f03201612e0 con 0x7f032006f2e0 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f0326400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f03200cd1c0 con 0x7f03200705a0 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f031f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f031f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44190/0 (socket says 172.21.15.129:44190) 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.139+0000 7f031f400640 1 -- 172.21.15.129:0/463542695 learned_addr learned my addr 172.21.15.129:0/463542695 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:13.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 -- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03200705a0 msgr2=0x7f03200c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03200705a0 0x7f03200c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 -- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f032006f2e0 msgr2=0x7f03201f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f032006f2e0 0x7f03201f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 -- 172.21.15.129:0/463542695 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03201f6a80 con 0x7f032006eed0 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f031f400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f0310009950 tx=0x7f0310033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310005c90 con 0x7f032006eed0 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0310031e50 con 0x7f032006eed0 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03201f6d60 con 0x7f032006eed0 2024-09-17T03:57:13.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.140+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310005260 con 0x7f032006eed0 2024-09-17T03:57:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.141+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03200c87d0 con 0x7f032006eed0 2024-09-17T03:57:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.142+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f032006a140 con 0x7f032006eed0 2024-09-17T03:57:13.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.143+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0310005400 con 0x7f032006eed0 2024-09-17T03:57:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.144+0000 7f0317400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 0x7f030007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.144+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f03100bbe90 con 0x7f032006eed0 2024-09-17T03:57:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.144+0000 7f031ea00640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 0x7f030007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:13.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.145+0000 7f031ea00640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 0x7f030007b0a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f030c0058c0 tx=0x7f030c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:13.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.148+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f031003f030 con 0x7f032006eed0 2024-09-17T03:57:13.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.368+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03200af260 con 0x7f032006eed0 2024-09-17T03:57:13.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.368+0000 7f0317400640 1 -- 172.21.15.129:0/463542695 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+3595 (secure 0 0 0) 0x7f03100bd090 con 0x7f032006eed0 2024-09-17T03:57:13.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:13.369 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":102,"btime":"2024-09-17T03:57:12:884041+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:12.151042+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638,7318]},"id":9}]} 2024-09-17T03:57:13.370 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 102 2024-09-17T03:57:13.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 msgr2=0x7f030007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:13.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 0x7f030007b0a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f030c0058c0 tx=0x7f030c005850 comp rx=0 tx=0).stop 2024-09-17T03:57:13.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 msgr2=0x7f03201e6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:13.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f0310009950 tx=0x7f0310033c20 comp rx=0 tx=0).stop 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 shutdown_connections 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0300078b90 0x7f030007b0a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03200705a0 0x7f03200c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f032006f2e0 0x7f03201f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 --2- 172.21.15.129:0/463542695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f032006eed0 0x7f03201e6170 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 >> 172.21.15.129:0/463542695 conn(0x7f032006d410 msgr2=0x7f03200cae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 shutdown_connections 2024-09-17T03:57:13.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:13.371+0000 7f0326400640 1 -- 172.21.15.129:0/463542695 wait complete. 2024-09-17T03:57:13.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:57:13.633 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"8","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/pin","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-09-17T03:57:13.634 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "8" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/pin", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:57:13.634 INFO:tasks.cephfs.cephfs_test_case:[('/pin', 1)] =?= [('/pin', 1)] 2024-09-17T03:57:13.634 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:57:13.634 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> import fcntl 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> import sys 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> import time 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:57:13.670 DEBUG:teuthology.orchestra.run.smithi129:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> os.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked") 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> fd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY) 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> while True: 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> print("open_dir_background: keeping dir open", file=sys.stderr) 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> if os.read(0, 4096) == b"": 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> break 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> except BlockingIOError: 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> pass 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> time.sleep(2) 2024-09-17T03:57:13.671 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:57:13.715 DEBUG:teuthology.orchestra.run.smithi129:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-09-17T03:57:13.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T03:57:13.752 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked': No such file or directory 2024-09-17T03:57:13.813 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:14.754 DEBUG:teuthology.orchestra.run.smithi129:> stat /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-09-17T03:57:14.792 INFO:teuthology.orchestra.run.smithi129.stdout: File: /home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked 2024-09-17T03:57:14.792 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-09-17T03:57:14.792 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 29h/41d Inode: 2199023255552 Links: 2 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0755/drwxr-xr-x) Uid: ( 1000/ ubuntu) Gid: ( 1333/ ubuntu) 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout:Context: unconfined_u:object_r:unlabeled_t:s0 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-17 03:57:13.810437172 +0000 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-17 03:57:13.810437172 +0000 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-17 03:57:13.810437172 +0000 2024-09-17T03:57:14.793 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-17 03:57:13.810437172 +0000 2024-09-17T03:57:14.794 INFO:tasks.cephfs.mount:File pin/to-be-unlinked became visible from 0 after 1s 2024-09-17T03:57:14.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:57:14.794 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir pin/to-be-unlinked) 2024-09-17T03:57:14.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache num_strays 2024-09-17T03:57:15.150 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":1}} 2024-09-17T03:57:15.150 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-09-17T03:57:15.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:57:15.389 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T03:57:15.389 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T03:57:15.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:57:15.628 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T03:57:15.629 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T03:57:15.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-17T03:57:15.815 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:15.882 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:57:15.883 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:57:15.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json cache drop 2024-09-17T03:57:16.127 INFO:teuthology.orchestra.run.smithi129.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":378,"bytes":66666}},"duration":0.0067862729999999998} 2024-09-17T03:57:16.128 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 66666, "items": 378 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.006786273, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-09-17T03:57:16.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-09-17T03:57:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 -- 172.21.15.129:0/3723140243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e4074800 msgr2=0x7f46e40af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 --2- 172.21.15.129:0/3723140243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e4074800 0x7f46e40af480 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46d4009830 tx=0x7f46d402a520 comp rx=0 tx=0).stop 2024-09-17T03:57:16.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 -- 172.21.15.129:0/3723140243 shutdown_connections 2024-09-17T03:57:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 --2- 172.21.15.129:0/3723140243 >> v2:172.21.15.181:3300/0 conn(0x7f46e40b1c40 0x7f46e40b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 --2- 172.21.15.129:0/3723140243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e4074800 0x7f46e40af480 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 -- 172.21.15.129:0/3723140243 >> 172.21.15.129:0/3723140243 conn(0x7f46e40ccc40 msgr2=0x7f46e40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 -- 172.21.15.129:0/3723140243 shutdown_connections 2024-09-17T03:57:16.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.352+0000 7f46e9000640 1 -- 172.21.15.129:0/3723140243 wait complete. 2024-09-17T03:57:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.354+0000 7f46e9000640 1 Processor -- start 2024-09-17T03:57:16.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.354+0000 7f46e9000640 1 -- start start 2024-09-17T03:57:16.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e406eea0 0x7f46e40c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46e40b1c40 0x7f46e409cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40b41a0 con 0x7f46e4074800 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46e40b4020 con 0x7f46e40b1c40 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e9000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f46e40b4320 con 0x7f46e406eea0 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:16.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45092/0 (socket says 172.21.15.129:45092) 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 -- 172.21.15.129:0/3484277922 learned_addr learned my addr 172.21.15.129:0/3484277922 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e2800640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46e40b1c40 0x7f46e409cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1e00640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e406eea0 0x7f46e40c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 -- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e406eea0 msgr2=0x7f46e40c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e406eea0 0x7f46e40c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 -- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46e40b1c40 msgr2=0x7f46e409cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:16.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46e40b1c40 0x7f46e409cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.356+0000 7f46e1400640 1 -- 172.21.15.129:0/3484277922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e41f3d50 con 0x7f46e4074800 2024-09-17T03:57:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46e1400640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f46d000e9c0 tx=0x7f46d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d000ccc0 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46d0004020 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e41f4030 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d000ce60 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.357+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e41f45f0 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.358+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e406fb30 con 0x7f46e4074800 2024-09-17T03:57:16.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.359+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f46d00035f0 con 0x7f46e4074800 2024-09-17T03:57:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.361+0000 7f46da000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 0x7f46c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.361+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(127..127 src has 1..127) ==== 5709+0+0 (secure 0 0 0) 0x7f46d009b6a0 con 0x7f46e4074800 2024-09-17T03:57:16.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.361+0000 7f46e1e00640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 0x7f46c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:16.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.362+0000 7f46e1e00640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 0x7f46c407b0d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f46d4034000 tx=0x7f46d4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.365+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f46d0063400 con 0x7f46e4074800 2024-09-17T03:57:16.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.586+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f46e4070b60 con 0x7f46e4074800 2024-09-17T03:57:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.903+0000 7f46da000640 1 -- 172.21.15.129:0/3484277922 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v103) ==== 105+0+0 (secure 0 0 0) 0x7f46d0067e30 con 0x7f46e4074800 2024-09-17T03:57:16.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 msgr2=0x7f46c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 0x7f46c407b0d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f46d4034000 tx=0x7f46d4004730 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 msgr2=0x7f46e415a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f46d000e9c0 tx=0x7f46d000ee80 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 shutdown_connections 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46c4078bc0 0x7f46c407b0d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46e40b1c40 0x7f46e409cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46e4074800 0x7f46e415a6e0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 --2- 172.21.15.129:0/3484277922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46e406eea0 0x7f46e40c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.906+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 >> 172.21.15.129:0/3484277922 conn(0x7f46e40ccc40 msgr2=0x7f46e415c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.907+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 shutdown_connections 2024-09-17T03:57:16.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:16.907+0000 7f46e9000640 1 -- 172.21.15.129:0/3484277922 wait complete. 2024-09-17T03:57:16.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:17.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:57:17.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.159+0000 7f9b05600640 1 -- 172.21.15.129:0/332141179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b000705a0 msgr2=0x7f9b000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.159+0000 7f9b05600640 1 --2- 172.21.15.129:0/332141179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b000705a0 0x7f9b000b4140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9af0005fd0 tx=0x7f9af002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 -- 172.21.15.129:0/332141179 shutdown_connections 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 --2- 172.21.15.129:0/332141179 >> v2:172.21.15.181:3300/0 conn(0x7f9b0006eed0 0x7f9b000ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 --2- 172.21.15.129:0/332141179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b000705a0 0x7f9b000b4140 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 -- 172.21.15.129:0/332141179 >> 172.21.15.129:0/332141179 conn(0x7f9b0006d410 msgr2=0x7f9b000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 -- 172.21.15.129:0/332141179 shutdown_connections 2024-09-17T03:57:17.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.160+0000 7f9b05600640 1 -- 172.21.15.129:0/332141179 wait complete. 2024-09-17T03:57:17.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.162+0000 7f9b05600640 1 Processor -- start 2024-09-17T03:57:17.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.162+0000 7f9b05600640 1 -- start start 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0006eed0 0x7f9b000c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b0006f2e0 0x7f9b000c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b00160200 con 0x7f9b000705a0 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b001612e0 con 0x7f9b0006eed0 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9b05600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9b000cd1c0 con 0x7f9b0006f2e0 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9aff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9aff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45106/0 (socket says 172.21.15.129:45106) 2024-09-17T03:57:17.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9aff400640 1 -- 172.21.15.129:0/2231023492 learned_addr learned my addr 172.21.15.129:0/2231023492 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9afea00640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0006eed0 0x7f9b000c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.164+0000 7f9afe000640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b0006f2e0 0x7f9b000c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 -- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b0006f2e0 msgr2=0x7f9b000c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b0006f2e0 0x7f9b000c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 -- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0006eed0 msgr2=0x7f9b000c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0006eed0 0x7f9b000c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 -- 172.21.15.129:0/2231023492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b000c9e10 con 0x7f9b000705a0 2024-09-17T03:57:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9aff400640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9af8002230 tx=0x7f9af800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8003b70 con 0x7f9b000705a0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.165+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9af8003d10 con 0x7f9b000705a0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.166+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b000c8830 con 0x7f9b000705a0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.166+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af8005330 con 0x7f9b000705a0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.166+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b000c8df0 con 0x7f9b000705a0 2024-09-17T03:57:17.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.167+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b000c85a0 con 0x7f9b000705a0 2024-09-17T03:57:17.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.169+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9af80056e0 con 0x7f9b000705a0 2024-09-17T03:57:17.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.170+0000 7f9af6a00640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 0x7f9ae007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:17.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.171+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 5767+0+0 (secure 0 0 0) 0x7f9af809acd0 con 0x7f9b000705a0 2024-09-17T03:57:17.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.171+0000 7f9afea00640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 0x7f9ae007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:17.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.172+0000 7f9afea00640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 0x7f9ae007b0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9af0033000 tx=0x7f9af0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:17.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.175+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9af80629e0 con 0x7f9b000705a0 2024-09-17T03:57:17.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.401+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b000b4100 con 0x7f9b000705a0 2024-09-17T03:57:17.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.401+0000 7f9af6a00640 1 -- 172.21.15.129:0/2231023492 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3592 (secure 0 0 0) 0x7f9b000b4100 con 0x7f9b000705a0 2024-09-17T03:57:17.402 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:17.402 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":104,"btime":"2024-09-17T03:57:16:908874+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:16.908871+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:17.402 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 104 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 msgr2=0x7f9ae007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 0x7f9ae007b0d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9af0033000 tx=0x7f9af0004730 comp rx=0 tx=0).stop 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 msgr2=0x7f9b0014bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f9af8002230 tx=0x7f9af800cb90 comp rx=0 tx=0).stop 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 shutdown_connections 2024-09-17T03:57:17.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ae0078bc0 0x7f9ae007b0d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b000705a0 0x7f9b0014bec0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b0006f2e0 0x7f9b000c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 --2- 172.21.15.129:0/2231023492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b0006eed0 0x7f9b000c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 >> 172.21.15.129:0/2231023492 conn(0x7f9b0006d410 msgr2=0x7f9b000cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 shutdown_connections 2024-09-17T03:57:17.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:17.404+0000 7f9b05600640 1 -- 172.21.15.129:0/2231023492 wait complete. 2024-09-17T03:57:17.628 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:16.908871+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:17.629 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:17.817 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:18.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.858+0000 7fd5d8e00640 1 -- 172.21.15.129:0/3560425506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d4099570 msgr2=0x7fd5d406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.858+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/3560425506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d4099570 0x7fd5d406eed0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4005fd0 tx=0x7fd5c402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.858+0000 7fd5d8e00640 1 -- 172.21.15.129:0/3560425506 shutdown_connections 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.858+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/3560425506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d40cf9c0 0x7fd5d40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.858+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/3560425506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d4099570 0x7fd5d406eed0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.859+0000 7fd5d8e00640 1 -- 172.21.15.129:0/3560425506 >> 172.21.15.129:0/3560425506 conn(0x7fd5d406d410 msgr2=0x7fd5d40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.859+0000 7fd5d8e00640 1 -- 172.21.15.129:0/3560425506 shutdown_connections 2024-09-17T03:57:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.859+0000 7fd5d8e00640 1 -- 172.21.15.129:0/3560425506 wait complete. 2024-09-17T03:57:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.861+0000 7fd5d8e00640 1 Processor -- start 2024-09-17T03:57:18.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.861+0000 7fd5d8e00640 1 -- start start 2024-09-17T03:57:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.862+0000 7fd5d8e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d406f410 0x7fd5d41e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d8e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d40cf9c0 0x7fd5d41f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d8e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4160ba0 con 0x7fd5d4099570 2024-09-17T03:57:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d8e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4160a20 con 0x7fd5d40cf9c0 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d8e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd5d4160d20 con 0x7fd5d406f410 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45112/0 (socket says 172.21.15.129:45112) 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 -- 172.21.15.129:0/742445748 learned_addr learned my addr 172.21.15.129:0/742445748 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d2800640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d40cf9c0 0x7fd5d41f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d406f410 0x7fd5d41e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 -- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d406f410 msgr2=0x7fd5d41e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d406f410 0x7fd5d41e60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 -- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d40cf9c0 msgr2=0x7fd5d41f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d40cf9c0 0x7fd5d41f58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.863+0000 7fd5d1400640 1 -- 172.21.15.129:0/742445748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d40cb5b0 con 0x7fd5d4099570 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5d1400640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c400 tx=0x7fd5c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0017070 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd5c00053c0 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c0003850 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d40cb890 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.864+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d40cbe50 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.865+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d41611d0 con 0x7fd5d4099570 2024-09-17T03:57:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.866+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd5c001a070 con 0x7fd5d4099570 2024-09-17T03:57:18.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.868+0000 7fd5ca000640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 0x7fd5b407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:18.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.868+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fd5c0099ff0 con 0x7fd5d4099570 2024-09-17T03:57:18.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.868+0000 7fd5d1e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 0x7fd5b407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:18.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.869+0000 7fd5d1e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 0x7fd5b407b0a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4033000 tx=0x7fd5c4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:18.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:18.873+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd5c0061df0 con 0x7fd5d4099570 2024-09-17T03:57:19.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.093+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5d406a140 con 0x7fd5d4099570 2024-09-17T03:57:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.094+0000 7fd5ca000640 1 -- 172.21.15.129:0/742445748 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3592 (secure 0 0 0) 0x7fd5c0066820 con 0x7fd5d4099570 2024-09-17T03:57:19.094 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:19.094 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":104,"btime":"2024-09-17T03:57:16:908874+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:16.908871+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:19.094 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 104 2024-09-17T03:57:19.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 msgr2=0x7fd5b407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:19.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 0x7fd5b407b0a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4033000 tx=0x7fd5c4004730 comp rx=0 tx=0).stop 2024-09-17T03:57:19.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 msgr2=0x7fd5d41f5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:19.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fd5c000c400 tx=0x7fd5c000c8c0 comp rx=0 tx=0).stop 2024-09-17T03:57:19.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 shutdown_connections 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd5b4078b90 0x7fd5b407b0a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d40cf9c0 0x7fd5d41f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5d4099570 0x7fd5d41f5390 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 --2- 172.21.15.129:0/742445748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5d406f410 0x7fd5d41e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 >> 172.21.15.129:0/742445748 conn(0x7fd5d406d410 msgr2=0x7fd5d40d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.096+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 shutdown_connections 2024-09-17T03:57:19.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:19.097+0000 7fd5d8e00640 1 -- 172.21.15.129:0/742445748 wait complete. 2024-09-17T03:57:19.321 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:16.908871+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:19.321 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:20.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:20.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 -- 172.21.15.129:0/578354927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3006ecb0 msgr2=0x7f0e3006f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:20.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 --2- 172.21.15.129:0/578354927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3006ecb0 0x7f0e3006f0c0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f0e3006fcc0 tx=0x7f0e2802a980 comp rx=0 tx=0).stop 2024-09-17T03:57:20.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 -- 172.21.15.129:0/578354927 shutdown_connections 2024-09-17T03:57:20.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 --2- 172.21.15.129:0/578354927 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3006ecb0 0x7f0e3006f0c0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 --2- 172.21.15.129:0/578354927 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e300d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 --2- 172.21.15.129:0/578354927 >> v2:172.21.15.181:3300/0 conn(0x7f0e30145e60 0x7f0e300d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 -- 172.21.15.129:0/578354927 >> 172.21.15.129:0/578354927 conn(0x7f0e3006cfc0 msgr2=0x7f0e3006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:20.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 -- 172.21.15.129:0/578354927 shutdown_connections 2024-09-17T03:57:20.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.552+0000 7f0e2f200640 1 -- 172.21.15.129:0/578354927 wait complete. 2024-09-17T03:57:20.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.554+0000 7f0e2f200640 1 Processor -- start 2024-09-17T03:57:20.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.554+0000 7f0e2f200640 1 -- start start 2024-09-17T03:57:20.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3006ecb0 0x7f0e301f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:20.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e3013ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300cef20 con 0x7f0e30145e60 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300ceda0 con 0x7f0e3006ecb0 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.555+0000 7f0e2f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0e3006f600 con 0x7f0e301458a0 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45138/0 (socket says 172.21.15.129:45138) 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 -- 172.21.15.129:0/2439650374 learned_addr learned my addr 172.21.15.129:0/2439650374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2d400640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e3013ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3006ecb0 0x7f0e301f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 -- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 msgr2=0x7f0e3013ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:20.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e3013ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 -- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3006ecb0 msgr2=0x7f0e301f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3006ecb0 0x7f0e301f7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2e800640 1 -- 172.21.15.129:0/2439650374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e301f7990 con 0x7f0e30145e60 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.556+0000 7f0e2d400640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e3013ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e2e800640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009d60 tx=0x7f0e28003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28034040 con 0x7f0e30145e60 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e28004540 con 0x7f0e30145e60 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e28045480 con 0x7f0e30145e60 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e300c4fa0 con 0x7f0e30145e60 2024-09-17T03:57:20.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.557+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e300ae870 con 0x7f0e30145e60 2024-09-17T03:57:20.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.558+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3006f0c0 con 0x7f0e30145e60 2024-09-17T03:57:20.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.559+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0e28037020 con 0x7f0e30145e60 2024-09-17T03:57:20.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.560+0000 7f0e26000640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 0x7f0e1007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:20.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.560+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f0e280bc5c0 con 0x7f0e30145e60 2024-09-17T03:57:20.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.561+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 0x7f0e1007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:20.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.562+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 0x7f0e1007b0d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bad0 tx=0x7f0e2000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:20.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.565+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0e301f7990 con 0x7f0e30145e60 2024-09-17T03:57:20.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.782+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e3006a140 con 0x7f0e30145e60 2024-09-17T03:57:20.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.782+0000 7f0e26000640 1 -- 172.21.15.129:0/2439650374 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3592 (secure 0 0 0) 0x7f0e28084a60 con 0x7f0e30145e60 2024-09-17T03:57:20.783 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:20.783 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":104,"btime":"2024-09-17T03:57:16:908874+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:16.908871+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:20.783 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 104 2024-09-17T03:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 msgr2=0x7f0e1007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:20.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 0x7f0e1007b0d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000bad0 tx=0x7f0e2000b2e0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 msgr2=0x7f0e300b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009d60 tx=0x7f0e28003f10 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 shutdown_connections 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e10078bc0 0x7f0e1007b0d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e30145e60 0x7f0e300b1740 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e301458a0 0x7f0e3013ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 --2- 172.21.15.129:0/2439650374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3006ecb0 0x7f0e301f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 >> 172.21.15.129:0/2439650374 conn(0x7f0e3006cfc0 msgr2=0x7f0e300ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 shutdown_connections 2024-09-17T03:57:20.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:20.785+0000 7f0e2f200640 1 -- 172.21.15.129:0/2439650374 wait complete. 2024-09-17T03:57:21.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:16.908871+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:21.015 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:21.821 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:22.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.246+0000 7f560e600640 1 -- 172.21.15.129:0/3437530632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 msgr2=0x7f56080b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.246+0000 7f560e600640 1 --2- 172.21.15.129:0/3437530632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f56080b4140 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f5608072e00 tx=0x7f55f802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 -- 172.21.15.129:0/3437530632 shutdown_connections 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 --2- 172.21.15.129:0/3437530632 >> v2:172.21.15.181:3300/0 conn(0x7f56080cd390 0x7f56081609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 --2- 172.21.15.129:0/3437530632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f56080b4140 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 -- 172.21.15.129:0/3437530632 >> 172.21.15.129:0/3437530632 conn(0x7f560806d410 msgr2=0x7f56080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:22.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 -- 172.21.15.129:0/3437530632 shutdown_connections 2024-09-17T03:57:22.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.247+0000 7f560e600640 1 -- 172.21.15.129:0/3437530632 wait complete. 2024-09-17T03:57:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.249+0000 7f560e600640 1 Processor -- start 2024-09-17T03:57:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.249+0000 7f560e600640 1 -- start start 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f560806eed0 0x7f56080c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56081f3430 0x7f56080c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5608160250 con 0x7f56080705a0 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f560806f4f0 con 0x7f56081f3430 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.250+0000 7f560e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5608160f00 con 0x7f560806eed0 2024-09-17T03:57:22.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45162/0 (socket says 172.21.15.129:45162) 2024-09-17T03:57:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 -- 172.21.15.129:0/1032596536 learned_addr learned my addr 172.21.15.129:0/1032596536 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f5607400640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f560806eed0 0x7f56080c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f5607e00640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56081f3430 0x7f56080c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 -- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f560806eed0 msgr2=0x7f56080c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f560806eed0 0x7f56080c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 -- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56081f3430 msgr2=0x7f56080c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56081f3430 0x7f56080c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 -- 172.21.15.129:0/1032596536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56081f3fa0 con 0x7f56080705a0 2024-09-17T03:57:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.251+0000 7f55fea00640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c370 tx=0x7f55f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.252+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0017070 con 0x7f56080705a0 2024-09-17T03:57:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.252+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55f00053c0 con 0x7f56080705a0 2024-09-17T03:57:22.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.252+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f0003850 con 0x7f56080705a0 2024-09-17T03:57:22.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.252+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56081f4280 con 0x7f56080705a0 2024-09-17T03:57:22.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.252+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56081576a0 con 0x7f56080705a0 2024-09-17T03:57:22.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.253+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56081f3d10 con 0x7f56080705a0 2024-09-17T03:57:22.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.254+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f55f001a070 con 0x7f56080705a0 2024-09-17T03:57:22.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.255+0000 7f55ffe00640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 0x7f55e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.255+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f55f009a110 con 0x7f56080705a0 2024-09-17T03:57:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.256+0000 7f5607400640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 0x7f55e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:22.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.256+0000 7f5607400640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 0x7f55e407b0a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f55f8033000 tx=0x7f55f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:22.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.260+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f55f0061e20 con 0x7f56080705a0 2024-09-17T03:57:22.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.478+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56081f7980 con 0x7f56080705a0 2024-09-17T03:57:22.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.478+0000 7f55ffe00640 1 -- 172.21.15.129:0/1032596536 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+3592 (secure 0 0 0) 0x7f55f0066850 con 0x7f56080705a0 2024-09-17T03:57:22.479 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:22.479 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":104,"btime":"2024-09-17T03:57:16:908874+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:16.908871+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:22.479 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 104 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.480+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 msgr2=0x7f55e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 0x7f55e407b0a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f55f8033000 tx=0x7f55f8004730 comp rx=0 tx=0).stop 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 msgr2=0x7f5608157bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f55f000c370 tx=0x7f55f000c830 comp rx=0 tx=0).stop 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 shutdown_connections 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55e4078b90 0x7f55e407b0a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56081f3430 0x7f56080c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56080705a0 0x7f5608157bc0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 --2- 172.21.15.129:0/1032596536 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f560806eed0 0x7f56080c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 >> 172.21.15.129:0/1032596536 conn(0x7f560806d410 msgr2=0x7f56080aed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 shutdown_connections 2024-09-17T03:57:22.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:22.481+0000 7f560e600640 1 -- 172.21.15.129:0/1032596536 wait complete. 2024-09-17T03:57:22.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:16.908871+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:22.705 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:23.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:23.823 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:23.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 -- 172.21.15.129:0/3879533880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc004800 msgr2=0x7f54fc004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 --2- 172.21.15.129:0/3879533880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54fc004800 0x7f54fc004c70 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5508001830 tx=0x7f54fc005830 comp rx=0 tx=0).stop 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551ce00640 1 -- 172.21.15.129:0/3879533880 reap_dead start 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 -- 172.21.15.129:0/3879533880 shutdown_connections 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 -- 172.21.15.129:0/3879533880 >> 172.21.15.129:0/3879533880 conn(0x7f551806cfc0 msgr2=0x7f551806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 -- 172.21.15.129:0/3879533880 shutdown_connections 2024-09-17T03:57:23.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.931+0000 7f551d800640 1 -- 172.21.15.129:0/3879533880 wait complete. 2024-09-17T03:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.933+0000 7f551d800640 1 Processor -- start 2024-09-17T03:57:23.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.934+0000 7f551d800640 1 -- start start 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54fc004800 0x7f55181f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55181443b0 0x7f55180c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f551815cbd0 con 0x7f55181447c0 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55180d0be0 con 0x7f55181443b0 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f551815cd50 con 0x7f54fc004800 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:23.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54676/0 (socket says 172.21.15.129:54676) 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f551ce00640 1 -- 172.21.15.129:0/206614350 learned_addr learned my addr 172.21.15.129:0/206614350 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f5517400640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55181443b0 0x7f55180c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.935+0000 7f5517e00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54fc004800 0x7f55181f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 -- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54fc004800 msgr2=0x7f55181f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54fc004800 0x7f55181f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 -- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55181443b0 msgr2=0x7f55180c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55181443b0 0x7f55180c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 -- 172.21.15.129:0/206614350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55180c5f70 con 0x7f55181447c0 2024-09-17T03:57:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f551ce00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5510005fd0 tx=0x7f551000f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.936+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510004b20 con 0x7f55181447c0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.937+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5510004cc0 con 0x7f55181447c0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.937+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510003bd0 con 0x7f55181447c0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.937+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55180c51b0 con 0x7f55181447c0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.937+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55180c5770 con 0x7f55181447c0 2024-09-17T03:57:23.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.938+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5518144ce0 con 0x7f55181447c0 2024-09-17T03:57:23.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.939+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5510004030 con 0x7f55181447c0 2024-09-17T03:57:23.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.940+0000 7f550fe00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 0x7f54f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:23.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.940+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f551009b130 con 0x7f55181447c0 2024-09-17T03:57:23.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.941+0000 7f5517e00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 0x7f54f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:23.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.941+0000 7f5517e00640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 0x7f54f407b0d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f55180b4450 tx=0x7f5500001040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:23.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:23.945+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5510062e40 con 0x7f55181447c0 2024-09-17T03:57:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.161+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55180b46e0 con 0x7f55181447c0 2024-09-17T03:57:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.162+0000 7f550fe00640 1 -- 172.21.15.129:0/206614350 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f5510067870 con 0x7f55181447c0 2024-09-17T03:57:24.161 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:24.161 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:24.161 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 msgr2=0x7f54f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 0x7f54f407b0d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f55180b4450 tx=0x7f5500001040 comp rx=0 tx=0).stop 2024-09-17T03:57:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 msgr2=0x7f55180c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5510005fd0 tx=0x7f551000f3c0 comp rx=0 tx=0).stop 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 shutdown_connections 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078bc0 0x7f54f407b0d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55181447c0 0x7f55180c70f0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55181443b0 0x7f55180c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 --2- 172.21.15.129:0/206614350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54fc004800 0x7f55181f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 >> 172.21.15.129:0/206614350 conn(0x7f551806cfc0 msgr2=0x7f551815f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 shutdown_connections 2024-09-17T03:57:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:24.164+0000 7f551d800640 1 -- 172.21.15.129:0/206614350 wait complete. 2024-09-17T03:57:24.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:24.390 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:25.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 -- 172.21.15.129:0/1100810332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc0705a0 msgr2=0x7f26fc0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 --2- 172.21.15.129:0/1100810332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc0705a0 0x7f26fc0b4140 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f26ec005fd0 tx=0x7f26ec02a560 comp rx=0 tx=0).stop 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 -- 172.21.15.129:0/1100810332 shutdown_connections 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 --2- 172.21.15.129:0/1100810332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc06eed0 0x7f26fc0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 --2- 172.21.15.129:0/1100810332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc0705a0 0x7f26fc0b4140 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 -- 172.21.15.129:0/1100810332 >> 172.21.15.129:0/1100810332 conn(0x7f26fc06d410 msgr2=0x7f26fc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:25.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 -- 172.21.15.129:0/1100810332 shutdown_connections 2024-09-17T03:57:25.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.617+0000 7f2700c00640 1 -- 172.21.15.129:0/1100810332 wait complete. 2024-09-17T03:57:25.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.620+0000 7f2700c00640 1 Processor -- start 2024-09-17T03:57:25.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.620+0000 7f2700c00640 1 -- start start 2024-09-17T03:57:25.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:25.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 0x7f26fc1e7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:25.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc0705a0 0x7f26fc1e80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:25.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc160200 con 0x7f26fc06eed0 2024-09-17T03:57:25.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1612e0 con 0x7f26fc06f2e0 2024-09-17T03:57:25.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f2700c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0cd1c0 con 0x7f26fc0705a0 2024-09-17T03:57:25.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f26f9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f26f9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54694/0 (socket says 172.21.15.129:54694) 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.621+0000 7f26f9e00640 1 -- 172.21.15.129:0/3931776649 learned_addr learned my addr 172.21.15.129:0/3931776649 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9400640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 0x7f26fc1e7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26fa800640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc0705a0 0x7f26fc1e80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 -- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc0705a0 msgr2=0x7f26fc1e80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc0705a0 0x7f26fc1e80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 -- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 msgr2=0x7f26fc1e7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:25.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 0x7f26fc1e7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 -- 172.21.15.129:0/3931776649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc15b3f0 con 0x7f26fc06eed0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9400640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 0x7f26fc1e7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f9e00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f26ec034040 tx=0x7f26ec0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec03e030 con 0x7f26fc06eed0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26ec0047a0 con 0x7f26fc06eed0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.622+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc1e6ad0 con 0x7f26fc06eed0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.623+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec035040 con 0x7f26fc06eed0 2024-09-17T03:57:25.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.623+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc1e7090 con 0x7f26fc06eed0 2024-09-17T03:57:25.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.624+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26fc0ccc80 con 0x7f26fc06eed0 2024-09-17T03:57:25.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.625+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f26ec041030 con 0x7f26fc06eed0 2024-09-17T03:57:25.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.626+0000 7f26f2000640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 0x7f26dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:25.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.626+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f26ec0bb660 con 0x7f26fc06eed0 2024-09-17T03:57:25.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.626+0000 7f26f9400640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 0x7f26dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:25.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.627+0000 7f26f9400640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 0x7f26dc07b0a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f26fc15eb30 tx=0x7f26e8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:25.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.630+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f26ec083a50 con 0x7f26fc06eed0 2024-09-17T03:57:25.825 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.848+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26fc06a140 con 0x7f26fc06eed0 2024-09-17T03:57:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.849+0000 7f26f2000640 1 -- 172.21.15.129:0/3931776649 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f26ec083350 con 0x7f26fc06eed0 2024-09-17T03:57:25.848 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:25.848 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:25.849 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:25.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 msgr2=0x7f26dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 0x7f26dc07b0a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f26fc15eb30 tx=0x7f26e8008040 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 msgr2=0x7f26fc0c9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f26ec034040 tx=0x7f26ec0039e0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 shutdown_connections 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26dc078b90 0x7f26dc07b0a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26fc0705a0 0x7f26fc1e80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26fc06f2e0 0x7f26fc1e7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 --2- 172.21.15.129:0/3931776649 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26fc06eed0 0x7f26fc0c9e10 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 >> 172.21.15.129:0/3931776649 conn(0x7f26fc06d410 msgr2=0x7f26fc0d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:25.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.851+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 shutdown_connections 2024-09-17T03:57:25.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:25.852+0000 7f2700c00640 1 -- 172.21.15.129:0/3931776649 wait complete. 2024-09-17T03:57:26.075 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:26.075 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:27.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:27.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 -- 172.21.15.129:0/4127600268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf801524c0 msgr2=0x7faf800cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:27.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 --2- 172.21.15.129:0/4127600268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf801524c0 0x7faf800cf5c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7faf8015f1d0 tx=0x7faf7002a530 comp rx=0 tx=0).stop 2024-09-17T03:57:27.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 -- 172.21.15.129:0/4127600268 shutdown_connections 2024-09-17T03:57:27.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 --2- 172.21.15.129:0/4127600268 >> v2:172.21.15.181:3300/0 conn(0x7faf801528d0 0x7faf80072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 --2- 172.21.15.129:0/4127600268 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf801524c0 0x7faf800cf5c0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.313+0000 7faf84c00640 1 -- 172.21.15.129:0/4127600268 >> 172.21.15.129:0/4127600268 conn(0x7faf8006cfc0 msgr2=0x7faf8006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:27.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.314+0000 7faf84c00640 1 -- 172.21.15.129:0/4127600268 shutdown_connections 2024-09-17T03:57:27.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.314+0000 7faf84c00640 1 -- 172.21.15.129:0/4127600268 wait complete. 2024-09-17T03:57:27.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.316+0000 7faf84c00640 1 Processor -- start 2024-09-17T03:57:27.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.316+0000 7faf84c00640 1 -- start start 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 0x7faf801e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 0x7faf801e7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faf800d02c0 con 0x7faf800cfb00 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faf800737d0 con 0x7faf801528d0 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf84c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faf800d0440 con 0x7faf801524c0 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.317+0000 7faf7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54710/0 (socket says 172.21.15.129:54710) 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 -- 172.21.15.129:0/2277117938 learned_addr learned my addr 172.21.15.129:0/2277117938 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 -- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 msgr2=0x7faf801e7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:27.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7ea00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 0x7faf801e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7fe00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 0x7faf801e7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 0x7faf801e7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 -- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 msgr2=0x7faf801e7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 0x7faf801e7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 -- 172.21.15.129:0/2277117938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf801e80c0 con 0x7faf800cfb00 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7ea00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 0x7faf801e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7fe00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 0x7faf801e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.318+0000 7faf7f400640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7faf70009920 tx=0x7faf70033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.319+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70031d60 con 0x7faf800cfb00 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.319+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faf70033e10 con 0x7faf800cfb00 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.319+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf70005260 con 0x7faf800cfb00 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.319+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf8015a500 con 0x7faf800cfb00 2024-09-17T03:57:27.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.319+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf80145f20 con 0x7faf800cfb00 2024-09-17T03:57:27.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.320+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf8015b1b0 con 0x7faf800cfb00 2024-09-17T03:57:27.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.321+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faf70005400 con 0x7faf800cfb00 2024-09-17T03:57:27.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.323+0000 7faf77400640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 0x7faf6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:27.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.323+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7faf700bbeb0 con 0x7faf800cfb00 2024-09-17T03:57:27.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.323+0000 7faf7ea00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 0x7faf6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:27.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.324+0000 7faf7ea00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 0x7faf6007b0a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faf801f4d80 tx=0x7faf6c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:27.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.327+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faf7003f030 con 0x7faf800cfb00 2024-09-17T03:57:27.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.545+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf8015a6d0 con 0x7faf800cfb00 2024-09-17T03:57:27.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.546+0000 7faf77400640 1 -- 172.21.15.129:0/2277117938 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7faf8015a6d0 con 0x7faf800cfb00 2024-09-17T03:57:27.545 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:27.545 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:27.546 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 msgr2=0x7faf6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 0x7faf6007b0a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faf801f4d80 tx=0x7faf6c009040 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 msgr2=0x7faf80145620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7faf70009920 tx=0x7faf70033c20 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 shutdown_connections 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faf60078b90 0x7faf6007b0a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf801528d0 0x7faf801e7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf801524c0 0x7faf801e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 --2- 172.21.15.129:0/2277117938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf800cfb00 0x7faf80145620 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 >> 172.21.15.129:0/2277117938 conn(0x7faf8006cfc0 msgr2=0x7faf8015bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:27.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.548+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 shutdown_connections 2024-09-17T03:57:27.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:27.549+0000 7faf84c00640 1 -- 172.21.15.129:0/2277117938 wait complete. 2024-09-17T03:57:27.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:27.780 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:27.827 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:28.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:29.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 -- 172.21.15.129:0/3486955932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 msgr2=0x7f1850005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:29.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 --2- 172.21.15.129:0/3486955932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f1850005150 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f1864005fd0 tx=0x7f1850005830 comp rx=0 tx=0).stop 2024-09-17T03:57:29.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f186be00640 1 -- 172.21.15.129:0/3486955932 reap_dead start 2024-09-17T03:57:29.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 -- 172.21.15.129:0/3486955932 shutdown_connections 2024-09-17T03:57:29.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 -- 172.21.15.129:0/3486955932 >> 172.21.15.129:0/3486955932 conn(0x7f186c06d410 msgr2=0x7f186c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:29.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 -- 172.21.15.129:0/3486955932 shutdown_connections 2024-09-17T03:57:29.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.016+0000 7f1872600640 1 -- 172.21.15.129:0/3486955932 wait complete. 2024-09-17T03:57:29.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.018+0000 7f1872600640 1 Processor -- start 2024-09-17T03:57:29.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.018+0000 7f1872600640 1 -- start start 2024-09-17T03:57:29.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f186c0c9480 0x7f186c1f2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 0x7f186c0b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f186c0b4140 con 0x7f1850004d40 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f186c06f4f0 con 0x7f186c0b0ed0 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f1872600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f186c0b42c0 con 0x7f186c0c9480 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54724/0 (socket says 172.21.15.129:54724) 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 -- 172.21.15.129:0/3079276377 learned_addr learned my addr 172.21.15.129:0/3079276377 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 -- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f186c0c9480 msgr2=0x7f186c1f2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f186c0c9480 0x7f186c1f2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186be00640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 0x7f186c0b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:29.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 -- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 msgr2=0x7f186c0b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:29.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 0x7f186c0b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186b400640 1 -- 172.21.15.129:0/3079276377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186c0b1820 con 0x7f1850004d40 2024-09-17T03:57:29.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.020+0000 7f186be00640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 0x7f186c0b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:29.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f186b400640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f185c011b10 tx=0x7f185c009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c01e070 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f185c004050 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c00fea0 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186c0c4f50 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.021+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186c0c5510 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.022+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186c0ad050 con 0x7f1850004d40 2024-09-17T03:57:29.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.023+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f185c021070 con 0x7f1850004d40 2024-09-17T03:57:29.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.024+0000 7f1863400640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 0x7f184c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:29.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.025+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f185c09b760 con 0x7f1850004d40 2024-09-17T03:57:29.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.025+0000 7f186aa00640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 0x7f184c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:29.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.026+0000 7f186aa00640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 0x7f184c07b0d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f186c071960 tx=0x7f1858000f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:29.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.029+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f186c0ad050 con 0x7f1850004d40 2024-09-17T03:57:29.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.247+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f186c0ccc80 con 0x7f1850004d40 2024-09-17T03:57:29.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.248+0000 7f1863400640 1 -- 172.21.15.129:0/3079276377 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f186c0b1820 con 0x7f1850004d40 2024-09-17T03:57:29.247 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:29.247 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:29.248 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:29.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.250+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 msgr2=0x7f184c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:29.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.250+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 0x7f184c07b0d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f186c071960 tx=0x7f1858000f30 comp rx=0 tx=0).stop 2024-09-17T03:57:29.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.250+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 msgr2=0x7f186c0c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:29.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.250+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f185c011b10 tx=0x7f185c009e90 comp rx=0 tx=0).stop 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 shutdown_connections 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f184c078bc0 0x7f184c07b0d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f186c0b0ed0 0x7f186c0b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f186c0c9480 0x7f186c1f2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 --2- 172.21.15.129:0/3079276377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1850004d40 0x7f186c0c6460 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 >> 172.21.15.129:0/3079276377 conn(0x7f186c06d410 msgr2=0x7f186c0d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 shutdown_connections 2024-09-17T03:57:29.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:29.251+0000 7f1872600640 1 -- 172.21.15.129:0/3079276377 wait complete. 2024-09-17T03:57:29.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:29.476 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:29.828 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:30.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:30.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 -- 172.21.15.129:0/3193513447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c072f40 msgr2=0x7efe0c073350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:30.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 --2- 172.21.15.129:0/3193513447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c072f40 0x7efe0c073350 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7efe0c073a10 tx=0x7efe0402a920 comp rx=0 tx=0).stop 2024-09-17T03:57:30.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 -- 172.21.15.129:0/3193513447 shutdown_connections 2024-09-17T03:57:30.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 --2- 172.21.15.129:0/3193513447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c072f40 0x7efe0c073350 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 -- 172.21.15.129:0/3193513447 >> 172.21.15.129:0/3193513447 conn(0x7efe0c06cfc0 msgr2=0x7efe0c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:30.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 -- 172.21.15.129:0/3193513447 shutdown_connections 2024-09-17T03:57:30.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.706+0000 7efe11c00640 1 -- 172.21.15.129:0/3193513447 wait complete. 2024-09-17T03:57:30.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.708+0000 7efe11c00640 1 Processor -- start 2024-09-17T03:57:30.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.708+0000 7efe11c00640 1 -- start start 2024-09-17T03:57:30.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe0c072f40 0x7efe0c156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:30.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe0c1528d0 0x7efe0c14ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c0d0440 con 0x7efe0c1524c0 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c0d02c0 con 0x7efe0c072f40 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efe0c0d05c0 con 0x7efe0c1528d0 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe0b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe0b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54754/0 (socket says 172.21.15.129:54754) 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe0b400640 1 -- 172.21.15.129:0/4158310631 learned_addr learned my addr 172.21.15.129:0/4158310631 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe11200640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe0c1528d0 0x7efe0c14ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.710+0000 7efe0be00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe0c072f40 0x7efe0c156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 -- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe0c1528d0 msgr2=0x7efe0c14ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe0c1528d0 0x7efe0c14ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 -- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe0c072f40 msgr2=0x7efe0c156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:30.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe0c072f40 0x7efe0c156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 -- 172.21.15.129:0/4158310631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c0b49f0 con 0x7efe0c1524c0 2024-09-17T03:57:30.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe0b400640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c400 tx=0x7efdf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:30.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8017070 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efdf80053c0 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c0b4cd0 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8003850 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.711+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c1f51f0 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.712+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0c0d0aa0 con 0x7efe0c1524c0 2024-09-17T03:57:30.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.714+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efdf801a070 con 0x7efe0c1524c0 2024-09-17T03:57:30.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.714+0000 7efe03e00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 0x7efdec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:30.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.715+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7efdf809a170 con 0x7efe0c1524c0 2024-09-17T03:57:30.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.715+0000 7efe0be00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 0x7efdec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:30.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.715+0000 7efe0be00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 0x7efdec07b0a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000f60 tx=0x7efdfc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:30.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.719+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efdf8061e80 con 0x7efe0c1524c0 2024-09-17T03:57:30.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.939+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe0c0b4780 con 0x7efe0c1524c0 2024-09-17T03:57:30.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.940+0000 7efe03e00640 1 -- 172.21.15.129:0/4158310631 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7efdf80668b0 con 0x7efe0c1524c0 2024-09-17T03:57:30.939 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:30.939 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:30.939 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 msgr2=0x7efdec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 0x7efdec07b0a0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7efdfc000f60 tx=0x7efdfc009040 comp rx=0 tx=0).stop 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 msgr2=0x7efe0c14c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c400 tx=0x7efdf800c8c0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 shutdown_connections 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efdec078b90 0x7efdec07b0a0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe0c1528d0 0x7efe0c14ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe0c1524c0 0x7efe0c14c7b0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 --2- 172.21.15.129:0/4158310631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe0c072f40 0x7efe0c156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:30.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 >> 172.21.15.129:0/4158310631 conn(0x7efe0c06cfc0 msgr2=0x7efe0c15bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:30.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 shutdown_connections 2024-09-17T03:57:30.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:30.943+0000 7efe11c00640 1 -- 172.21.15.129:0/4158310631 wait complete. 2024-09-17T03:57:31.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:31.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:31.830 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:32.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:32.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 -- 172.21.15.129:0/969180563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240d0050 msgr2=0x7f59240d14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:32.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 --2- 172.21.15.129:0/969180563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240d0050 0x7f59240d14d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f591c005fd0 tx=0x7f591c02a8b0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 -- 172.21.15.129:0/969180563 shutdown_connections 2024-09-17T03:57:32.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 --2- 172.21.15.129:0/969180563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240d0050 0x7f59240d14d0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 --2- 172.21.15.129:0/969180563 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 -- 172.21.15.129:0/969180563 >> 172.21.15.129:0/969180563 conn(0x7f592408bff0 msgr2=0x7f592409d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:32.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 -- 172.21.15.129:0/969180563 shutdown_connections 2024-09-17T03:57:32.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.399+0000 7f5929200640 1 -- 172.21.15.129:0/969180563 wait complete. 2024-09-17T03:57:32.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.401+0000 7f5929200640 1 Processor -- start 2024-09-17T03:57:32.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.401+0000 7f5929200640 1 -- start start 2024-09-17T03:57:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 0x7f5924160ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f59240b1eb0 con 0x7f59240b3430 2024-09-17T03:57:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f59240b1d30 con 0x7f59240d0050 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5929200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f59240b2030 con 0x7f59240b9be0 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5923400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5923400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:54774/0 (socket says 172.21.15.129:54774) 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f5923400640 1 -- 172.21.15.129:0/2179106374 learned_addr learned my addr 172.21.15.129:0/2179106374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.403+0000 7f591aa00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 -- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 msgr2=0x7f59240a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923e00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 0x7f5924160ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 -- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 msgr2=0x7f5924160ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 0x7f5924160ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 -- 172.21.15.129:0/2179106374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59240a1c00 con 0x7f59240b3430 2024-09-17T03:57:32.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f591aa00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923e00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 0x7f5924160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5923400640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f591400d6d0 tx=0x7f591400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914017070 con 0x7f59240b3430 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5914004310 con 0x7f59240b3430 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592409ee30 con 0x7f59240b3430 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5914009a80 con 0x7f59240b3430 2024-09-17T03:57:32.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.404+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59240cd580 con 0x7f59240b3430 2024-09-17T03:57:32.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.405+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59240b2550 con 0x7f59240b3430 2024-09-17T03:57:32.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.407+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f59140035f0 con 0x7f59240b3430 2024-09-17T03:57:32.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.407+0000 7f591be00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 0x7f590007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:32.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.408+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f591409b0e0 con 0x7f59240b3430 2024-09-17T03:57:32.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.408+0000 7f591aa00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 0x7f590007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:32.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.409+0000 7f591aa00640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 0x7f590007b0d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f59241618c0 tx=0x7f590c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:32.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.412+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5914063450 con 0x7f59240b3430 2024-09-17T03:57:32.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.631+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f592406a130 con 0x7f59240b3430 2024-09-17T03:57:32.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.632+0000 7f591be00640 1 -- 172.21.15.129:0/2179106374 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f5914062d50 con 0x7f59240b3430 2024-09-17T03:57:32.631 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:32.631 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:32.631 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 msgr2=0x7f590007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 0x7f590007b0d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f59241618c0 tx=0x7f590c009040 comp rx=0 tx=0).stop 2024-09-17T03:57:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 msgr2=0x7f59240cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f591400d6d0 tx=0x7f591400db90 comp rx=0 tx=0).stop 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 shutdown_connections 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5900078bc0 0x7f590007b0d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f59240d0050 0x7f5924160ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f59240b9be0 0x7f59240a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 --2- 172.21.15.129:0/2179106374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f59240b3430 0x7f59240cdd80 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 >> 172.21.15.129:0/2179106374 conn(0x7f592408bff0 msgr2=0x7f592406f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 shutdown_connections 2024-09-17T03:57:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:32.634+0000 7f5929200640 1 -- 172.21.15.129:0/2179106374 wait complete. 2024-09-17T03:57:32.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:32.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:33.832 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:33.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3284270298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0cfb60 msgr2=0x7fe89c160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3284270298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0cfb60 0x7fe89c160cf0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fe89c1609e0 tx=0x7fe88c02a4b0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3284270298 shutdown_connections 2024-09-17T03:57:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3284270298 >> v2:172.21.15.181:3300/0 conn(0x7fe89c0b46e0 0x7fe89c0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3284270298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0cfb60 0x7fe89c160cf0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3284270298 >> 172.21.15.129:0/3284270298 conn(0x7fe89c06cfc0 msgr2=0x7fe89c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3284270298 shutdown_connections 2024-09-17T03:57:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.095+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3284270298 wait complete. 2024-09-17T03:57:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.097+0000 7fe8a0a00640 1 Processor -- start 2024-09-17T03:57:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.097+0000 7fe8a0a00640 1 -- start start 2024-09-17T03:57:34.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:34.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe89c0cfb60 0x7fe89c14f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:34.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 0x7fe89c1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c15d1d0 con 0x7fe89c0b46e0 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe89c15d050 con 0x7fe89c0cfb60 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe8a0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe89c15d350 con 0x7fe89c0d02c0 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35004/0 (socket says 172.21.15.129:35004) 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 -- 172.21.15.129:0/3020716084 learned_addr learned my addr 172.21.15.129:0/3020716084 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe891e00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe89c0cfb60 0x7fe89c14f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89b200640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 0x7fe89c1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 -- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 msgr2=0x7fe89c1f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 0x7fe89c1f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 -- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe89c0cfb60 msgr2=0x7fe89c14f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.099+0000 7fe89a800640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe89c0cfb60 0x7fe89c14f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe89a800640 1 -- 172.21.15.129:0/3020716084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89c14f550 con 0x7fe89c0b46e0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe89b200640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 0x7fe89c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe89a800640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fe88c034040 tx=0x7fe88c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c03e030 con 0x7fe89c0b46e0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe88c0047a0 con 0x7fe89c0b46e0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c035040 con 0x7fe89c0b46e0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89c1e7050 con 0x7fe89c0b46e0 2024-09-17T03:57:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.100+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89c14db80 con 0x7fe89c0b46e0 2024-09-17T03:57:34.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.101+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe89c15d800 con 0x7fe89c0b46e0 2024-09-17T03:57:34.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.102+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe88c041030 con 0x7fe89c0b46e0 2024-09-17T03:57:34.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.104+0000 7fe893400640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 0x7fe87807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:34.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.104+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fe88c0bb600 con 0x7fe89c0b46e0 2024-09-17T03:57:34.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.104+0000 7fe891e00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 0x7fe87807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.105+0000 7fe891e00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 0x7fe87807b0a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe884005fd0 tx=0x7fe884005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:34.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.109+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe88c083960 con 0x7fe89c0b46e0 2024-09-17T03:57:34.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.330+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe89c06a140 con 0x7fe89c0b46e0 2024-09-17T03:57:34.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.331+0000 7fe893400640 1 -- 172.21.15.129:0/3020716084 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7fe88c083260 con 0x7fe89c0b46e0 2024-09-17T03:57:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:34.331 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:34.331 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:34.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 msgr2=0x7fe87807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:34.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 0x7fe87807b0a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fe884005fd0 tx=0x7fe884005850 comp rx=0 tx=0).stop 2024-09-17T03:57:34.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 msgr2=0x7fe89c1f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:34.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fe88c034040 tx=0x7fe88c0039e0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 shutdown_connections 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe878078b90 0x7fe87807b0a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe89c0d02c0 0x7fe89c1f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe89c0cfb60 0x7fe89c14f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 --2- 172.21.15.129:0/3020716084 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe89c0b46e0 0x7fe89c1f66a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 >> 172.21.15.129:0/3020716084 conn(0x7fe89c06cfc0 msgr2=0x7fe89c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 shutdown_connections 2024-09-17T03:57:34.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:34.333+0000 7fe8a0a00640 1 -- 172.21.15.129:0/3020716084 wait complete. 2024-09-17T03:57:34.560 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:34.561 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:35.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 -- 172.21.15.129:0/3086205345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a80705a0 msgr2=0x7fe0a80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3086205345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a80705a0 0x7fe0a80b4140 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fe098005fd0 tx=0x7fe09802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 -- 172.21.15.129:0/3086205345 shutdown_connections 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3086205345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a80cd390 0x7fe0a81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3086205345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a80705a0 0x7fe0a80b4140 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.784+0000 7fe0ad200640 1 -- 172.21.15.129:0/3086205345 >> 172.21.15.129:0/3086205345 conn(0x7fe0a806d410 msgr2=0x7fe0a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:35.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.785+0000 7fe0ad200640 1 -- 172.21.15.129:0/3086205345 shutdown_connections 2024-09-17T03:57:35.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.785+0000 7fe0ad200640 1 -- 172.21.15.129:0/3086205345 wait complete. 2024-09-17T03:57:35.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.787+0000 7fe0ad200640 1 Processor -- start 2024-09-17T03:57:35.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.787+0000 7fe0ad200640 1 -- start start 2024-09-17T03:57:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0a80705a0 0x7fe0a8159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a8157340 0x7fe0a81595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a8160f00 con 0x7fe0a806eed0 2024-09-17T03:57:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a806f4f0 con 0x7fe0a80705a0 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0ad200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a8161080 con 0x7fe0a8157340 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.788+0000 7fe0a5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35024/0 (socket says 172.21.15.129:35024) 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 -- 172.21.15.129:0/3296659584 learned_addr learned my addr 172.21.15.129:0/3296659584 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5400640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0a80705a0 0x7fe0a8159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a6800640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a8157340 0x7fe0a81595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 -- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a8157340 msgr2=0x7fe0a81595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a8157340 0x7fe0a81595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 -- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0a80705a0 msgr2=0x7fe0a8159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:35.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0a80705a0 0x7fe0a8159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:35.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 -- 172.21.15.129:0/3296659584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a815bac0 con 0x7fe0a806eed0 2024-09-17T03:57:35.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe0a5e00640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe098034040 tx=0x7fe0980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.789+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09803e030 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.790+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe098003ca0 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.790+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a815bd50 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.790+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098035040 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.790+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a815e6f0 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.791+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a80b4140 con 0x7fe0a806eed0 2024-09-17T03:57:35.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.792+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe098041030 con 0x7fe0a806eed0 2024-09-17T03:57:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.793+0000 7fe09e000640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 0x7fe08807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.793+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fe0980bb340 con 0x7fe0a806eed0 2024-09-17T03:57:35.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.793+0000 7fe0a5400640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 0x7fe08807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:35.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.794+0000 7fe0a5400640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 0x7fe08807b0a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0a815b560 tx=0x7fe094009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:35.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:35.797+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe0980830e0 con 0x7fe0a806eed0 2024-09-17T03:57:35.834 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:36.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.018+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0a806a140 con 0x7fe0a806eed0 2024-09-17T03:57:36.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.018+0000 7fe09e000640 1 -- 172.21.15.129:0/3296659584 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7fe098087b10 con 0x7fe0a806eed0 2024-09-17T03:57:36.019 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:36.019 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:36.019 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 msgr2=0x7fe08807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 0x7fe08807b0a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fe0a815b560 tx=0x7fe094009040 comp rx=0 tx=0).stop 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 msgr2=0x7fe0a80c5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe098034040 tx=0x7fe0980038a0 comp rx=0 tx=0).stop 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 shutdown_connections 2024-09-17T03:57:36.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe088078b90 0x7fe08807b0a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0a8157340 0x7fe0a81595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0a80705a0 0x7fe0a8159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 --2- 172.21.15.129:0/3296659584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0a806eed0 0x7fe0a80c5b20 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 >> 172.21.15.129:0/3296659584 conn(0x7fe0a806d410 msgr2=0x7fe0a815f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 shutdown_connections 2024-09-17T03:57:36.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:36.021+0000 7fe0ad200640 1 -- 172.21.15.129:0/3296659584 wait complete. 2024-09-17T03:57:36.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:36.249 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:37.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:37.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 -- 172.21.15.129:0/2196343620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc0d0a50 msgr2=0x7f58fc160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:37.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 --2- 172.21.15.129:0/2196343620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc0d0a50 0x7f58fc160210 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f58fc1463d0 tx=0x7f58e802a4a0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 -- 172.21.15.129:0/2196343620 shutdown_connections 2024-09-17T03:57:37.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 --2- 172.21.15.129:0/2196343620 >> v2:172.21.15.181:3300/0 conn(0x7f58fc1458a0 0x7f58fc160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 --2- 172.21.15.129:0/2196343620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc0d0a50 0x7f58fc160210 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 -- 172.21.15.129:0/2196343620 >> 172.21.15.129:0/2196343620 conn(0x7f58fc06cfc0 msgr2=0x7f58fc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:37.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 -- 172.21.15.129:0/2196343620 shutdown_connections 2024-09-17T03:57:37.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.480+0000 7f5902a00640 1 -- 172.21.15.129:0/2196343620 wait complete. 2024-09-17T03:57:37.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.482+0000 7f5902a00640 1 Processor -- start 2024-09-17T03:57:37.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.482+0000 7f5902a00640 1 -- start start 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 0x7f58fc0aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 0x7f58fc0ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc0cef20 con 0x7f58fc1458a0 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc0ceda0 con 0x7f58fc0ab0a0 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc160c90 con 0x7f58fc145e60 2024-09-17T03:57:37.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5901600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5901600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35038/0 (socket says 172.21.15.129:35038) 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5901600640 1 -- 172.21.15.129:0/2903115217 learned_addr learned my addr 172.21.15.129:0/2903115217 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5901600640 1 -- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 msgr2=0x7f58fc0aab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5900c00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 0x7f58fc0aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.484+0000 7f5902000640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 0x7f58fc0ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5901600640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 0x7f58fc0aab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5901600640 1 -- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 msgr2=0x7f58fc0ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5901600640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 0x7f58fc0ab4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5901600640 1 -- 172.21.15.129:0/2903115217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc0ad550 con 0x7f58fc1458a0 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5900c00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 0x7f58fc0aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5902000640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 0x7f58fc0ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:37.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5901600640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f58f000cb70 tx=0x7f58f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0016020 con 0x7f58fc1458a0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f58f000ae50 con 0x7f58fc1458a0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc0ae2e0 con 0x7f58fc1458a0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0012680 con 0x7f58fc1458a0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.485+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc0ae8a0 con 0x7f58fc1458a0 2024-09-17T03:57:37.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.486+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58fc160210 con 0x7f58fc1458a0 2024-09-17T03:57:37.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.487+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f58f0019020 con 0x7f58fc1458a0 2024-09-17T03:57:37.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.488+0000 7f58f9600640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 0x7f58dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:37.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.488+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f58f009b410 con 0x7f58fc1458a0 2024-09-17T03:57:37.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.489+0000 7f5900c00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 0x7f58dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:37.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.490+0000 7f5900c00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 0x7f58dc07b0d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f58e8034000 tx=0x7f58e8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:37.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.493+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f58f0020020 con 0x7f58fc1458a0 2024-09-17T03:57:37.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.710+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58fc06a140 con 0x7f58fc1458a0 2024-09-17T03:57:37.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.710+0000 7f58f9600640 1 -- 172.21.15.129:0/2903115217 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f58f00631b0 con 0x7f58fc1458a0 2024-09-17T03:57:37.711 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:37.711 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:37.712 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 msgr2=0x7f58dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 0x7f58dc07b0d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f58e8034000 tx=0x7f58e8004730 comp rx=0 tx=0).stop 2024-09-17T03:57:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 msgr2=0x7f58fc0cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f58f000cb70 tx=0x7f58f000a430 comp rx=0 tx=0).stop 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 shutdown_connections 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f58dc078bc0 0x7f58dc07b0d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f58fc0ab0a0 0x7f58fc0ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f58fc145e60 0x7f58fc0aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 --2- 172.21.15.129:0/2903115217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58fc1458a0 0x7f58fc0cbd70 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:37.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 >> 172.21.15.129:0/2903115217 conn(0x7f58fc06cfc0 msgr2=0x7f58fc143740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 shutdown_connections 2024-09-17T03:57:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:37.713+0000 7f5902a00640 1 -- 172.21.15.129:0/2903115217 wait complete. 2024-09-17T03:57:37.835 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:37.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:37.940 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:38.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:39.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 -- 172.21.15.129:0/3802795606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 msgr2=0x7f0508004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:39.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 --2- 172.21.15.129:0/3802795606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f0508004c10 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f05100016c0 tx=0x7f0508005830 comp rx=0 tx=0).stop 2024-09-17T03:57:39.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 -- 172.21.15.129:0/3802795606 shutdown_connections 2024-09-17T03:57:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 --2- 172.21.15.129:0/3802795606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f0508004c10 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 -- 172.21.15.129:0/3802795606 >> 172.21.15.129:0/3802795606 conn(0x7f052406cfc0 msgr2=0x7f052406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 -- 172.21.15.129:0/3802795606 shutdown_connections 2024-09-17T03:57:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.173+0000 7f0529c00640 1 -- 172.21.15.129:0/3802795606 wait complete. 2024-09-17T03:57:39.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.175+0000 7f0529c00640 1 Processor -- start 2024-09-17T03:57:39.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.175+0000 7f0529c00640 1 -- start start 2024-09-17T03:57:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f052415ddf0 0x7f05240b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05240b2880 0x7f0524148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f052415ec10 con 0x7f0508004800 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f052415ea90 con 0x7f05240b2880 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05240d0340 con 0x7f052415ddf0 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35050/0 (socket says 172.21.15.129:35050) 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 -- 172.21.15.129:0/1673763356 learned_addr learned my addr 172.21.15.129:0/1673763356 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0529200640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05240b2880 0x7f0524148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523400640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f052415ddf0 0x7f05240b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 -- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f052415ddf0 msgr2=0x7f05240b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f052415ddf0 0x7f05240b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 -- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05240b2880 msgr2=0x7f0524148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05240b2880 0x7f0524148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.177+0000 7f0523e00640 1 -- 172.21.15.129:0/1673763356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0524148f30 con 0x7f0508004800 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f0523e00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0514010b70 tx=0x7f051400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514004d20 con 0x7f0508004800 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0514010e70 con 0x7f0508004800 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514003bd0 con 0x7f0508004800 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0524149210 con 0x7f0508004800 2024-09-17T03:57:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.178+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f052414be00 con 0x7f0508004800 2024-09-17T03:57:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.179+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05240d08a0 con 0x7f0508004800 2024-09-17T03:57:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.180+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0514004050 con 0x7f0508004800 2024-09-17T03:57:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.181+0000 7f051be00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 0x7f050407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:39.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.181+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f051409baa0 con 0x7f0508004800 2024-09-17T03:57:39.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.182+0000 7f0523400640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 0x7f050407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:39.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.183+0000 7f0523400640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 0x7f050407b0a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0510032000 tx=0x7f0510001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:39.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.186+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0514063e90 con 0x7f0508004800 2024-09-17T03:57:39.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.406+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f052406a140 con 0x7f0508004800 2024-09-17T03:57:39.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.406+0000 7f051be00640 1 -- 172.21.15.129:0/1673763356 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f0514063790 con 0x7f0508004800 2024-09-17T03:57:39.407 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:39.407 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:39.408 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:39.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.408+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 msgr2=0x7f050407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.408+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 0x7f050407b0a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f0510032000 tx=0x7f0510001190 comp rx=0 tx=0).stop 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.408+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 msgr2=0x7f052414b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.408+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f0514010b70 tx=0x7f051400e720 comp rx=0 tx=0).stop 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 shutdown_connections 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0504078b90 0x7f050407b0a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05240b2880 0x7f0524148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f052415ddf0 0x7f05240b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 --2- 172.21.15.129:0/1673763356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0508004800 0x7f052414b8c0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:39.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 >> 172.21.15.129:0/1673763356 conn(0x7f052406cfc0 msgr2=0x7f052406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:39.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 shutdown_connections 2024-09-17T03:57:39.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:39.409+0000 7f0529c00640 1 -- 172.21.15.129:0/1673763356 wait complete. 2024-09-17T03:57:39.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:39.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:39.835 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:40.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:40.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 -- 172.21.15.129:0/985577519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f529c004800 msgr2=0x7f529c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:40.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 --2- 172.21.15.129:0/985577519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f529c004800 0x7f529c004c10 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009b50 tx=0x7f529c005830 comp rx=0 tx=0).stop 2024-09-17T03:57:40.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 -- 172.21.15.129:0/985577519 shutdown_connections 2024-09-17T03:57:40.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 --2- 172.21.15.129:0/985577519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f529c004800 0x7f529c004c10 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:40.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 -- 172.21.15.129:0/985577519 >> 172.21.15.129:0/985577519 conn(0x7f52b806d410 msgr2=0x7f52b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:40.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 -- 172.21.15.129:0/985577519 shutdown_connections 2024-09-17T03:57:40.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.868+0000 7f52be200640 1 -- 172.21.15.129:0/985577519 wait complete. 2024-09-17T03:57:40.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.870+0000 7f52be200640 1 Processor -- start 2024-09-17T03:57:40.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.870+0000 7f52be200640 1 -- start start 2024-09-17T03:57:40.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f529c004800 0x7f52b81f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:40.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:40.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 0x7f52b81e70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:40.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f52b80cd420 con 0x7f52b81e8e20 2024-09-17T03:57:40.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f52b80cd2a0 con 0x7f529c004800 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52be200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f52b80cd5a0 con 0x7f52b8158de0 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35066/0 (socket says 172.21.15.129:35066) 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 -- 172.21.15.129:0/983755074 learned_addr learned my addr 172.21.15.129:0/983755074 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b7400640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f529c004800 0x7f52b81f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b7e00640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 0x7f52b81e70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 -- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 msgr2=0x7f52b81e70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 0x7f52b81e70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 -- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f529c004800 msgr2=0x7f52b81f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f529c004800 0x7f52b81f7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b6a00640 1 -- 172.21.15.129:0/983755074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b81f4ee0 con 0x7f52b81e8e20 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.872+0000 7f52b7e00640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 0x7f52b81e70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52b6a00640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009a70 tx=0x7f52a40336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a4038040 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52a4041070 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b81e9550 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a403c430 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.873+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b81e68a0 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.874+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b8070210 con 0x7f52b81e8e20 2024-09-17T03:57:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.876+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52a403c5d0 con 0x7f52b81e8e20 2024-09-17T03:57:40.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.876+0000 7f52af400640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 0x7f529807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:40.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.877+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f52a40bb870 con 0x7f52b81e8e20 2024-09-17T03:57:40.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.877+0000 7f52b7400640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 0x7f529807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:40.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.878+0000 7f52b7400640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 0x7f529807b0a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001be0 tx=0x7f52a80019e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:40.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:40.881+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f52a4083580 con 0x7f52b81e8e20 2024-09-17T03:57:41.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.099+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52b81e92e0 con 0x7f52b81e8e20 2024-09-17T03:57:41.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.100+0000 7f52af400640 1 -- 172.21.15.129:0/983755074 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f52a4087fb0 con 0x7f52b81e8e20 2024-09-17T03:57:41.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:41.100 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:41.101 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:41.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 msgr2=0x7f529807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 0x7f529807b0a0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001be0 tx=0x7f52a80019e0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 msgr2=0x7f52b81f49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009a70 tx=0x7f52a40336f0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 shutdown_connections 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5298078b90 0x7f529807b0a0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f52b8158de0 0x7f52b81e70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52b81e8e20 0x7f52b81f49a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 --2- 172.21.15.129:0/983755074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f529c004800 0x7f52b81f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 >> 172.21.15.129:0/983755074 conn(0x7f52b806d410 msgr2=0x7f52b8159c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 shutdown_connections 2024-09-17T03:57:41.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:41.102+0000 7f52be200640 1 -- 172.21.15.129:0/983755074 wait complete. 2024-09-17T03:57:41.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:41.329 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:41.837 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:42.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 -- 172.21.15.129:0/489690196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13300af4c0 msgr2=0x7f133015ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 --2- 172.21.15.129:0/489690196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13300af4c0 0x7f133015ae00 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f132000a940 tx=0x7f132002b8f0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 -- 172.21.15.129:0/489690196 shutdown_connections 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 --2- 172.21.15.129:0/489690196 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13300af4c0 0x7f133015ae00 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 -- 172.21.15.129:0/489690196 >> 172.21.15.129:0/489690196 conn(0x7f133006d410 msgr2=0x7f13300b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 -- 172.21.15.129:0/489690196 shutdown_connections 2024-09-17T03:57:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.559+0000 7f1335800640 1 -- 172.21.15.129:0/489690196 wait complete. 2024-09-17T03:57:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.561+0000 7f1335800640 1 Processor -- start 2024-09-17T03:57:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.562+0000 7f1335800640 1 -- start start 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13300af4c0 0x7f13301f9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 0x7f13301e88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f133006fd50 con 0x7f133015b340 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1330071170 con 0x7f13300ca7a0 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f1335800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f133006fed0 con 0x7f13300af4c0 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f132e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f132e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35074/0 (socket says 172.21.15.129:35074) 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.563+0000 7f132e000640 1 -- 172.21.15.129:0/4262260245 learned_addr learned my addr 172.21.15.129:0/4262260245 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132ea00640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13300af4c0 0x7f13301f9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132f400640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 0x7f13301e88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 -- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13300af4c0 msgr2=0x7f13301f9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13300af4c0 0x7f13301f9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 -- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 msgr2=0x7f13301e88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 0x7f13301e88c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 -- 172.21.15.129:0/4262260245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13300ba9b0 con 0x7f133015b340 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132f400640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 0x7f13301e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:42.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f132e000640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c400 tx=0x7f131c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c011790 con 0x7f133015b340 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f131c017070 con 0x7f133015b340 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13300bac90 con 0x7f133015b340 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.564+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c011c60 con 0x7f133015b340 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.565+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13300bb250 con 0x7f133015b340 2024-09-17T03:57:42.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.566+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1330070350 con 0x7f133015b340 2024-09-17T03:57:42.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.567+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f131c01c460 con 0x7f133015b340 2024-09-17T03:57:42.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.568+0000 7f1326a00640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 0x7f131007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:42.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.568+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f131c09ab80 con 0x7f133015b340 2024-09-17T03:57:42.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.568+0000 7f132ea00640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 0x7f131007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:42.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.569+0000 7f132ea00640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 0x7f131007b0a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1320036000 tx=0x7f1320004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:42.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.572+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f131c062f70 con 0x7f133015b340 2024-09-17T03:57:42.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.791+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f133006a140 con 0x7f133015b340 2024-09-17T03:57:42.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.791+0000 7f1326a00640 1 -- 172.21.15.129:0/4262260245 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f131c062870 con 0x7f133015b340 2024-09-17T03:57:42.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:42.792 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:42.792 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:42.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 msgr2=0x7f131007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:42.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 0x7f131007b0a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1320036000 tx=0x7f1320004730 comp rx=0 tx=0).stop 2024-09-17T03:57:42.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 msgr2=0x7f133015a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:42.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f131c00c400 tx=0x7f131c00c8c0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 shutdown_connections 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1310078b90 0x7f131007b0a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13300ca7a0 0x7f13301e88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f133015b340 0x7f133015a2b0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 --2- 172.21.15.129:0/4262260245 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13300af4c0 0x7f13301f9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 >> 172.21.15.129:0/4262260245 conn(0x7f133006d410 msgr2=0x7f13300b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 shutdown_connections 2024-09-17T03:57:42.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:42.794+0000 7f1335800640 1 -- 172.21.15.129:0/4262260245 wait complete. 2024-09-17T03:57:43.019 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:43.019 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:43.839 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:44.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.243+0000 7ff90d200640 1 -- 172.21.15.129:0/2052553006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8ec004800 msgr2=0x7ff8ec004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.243+0000 7ff90d200640 1 --2- 172.21.15.129:0/2052553006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8ec004800 0x7ff8ec004c10 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff908072c70 tx=0x7ff8ec005830 comp rx=0 tx=0).stop 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.243+0000 7ff906800640 1 -- 172.21.15.129:0/2052553006 reap_dead start 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.243+0000 7ff90d200640 1 -- 172.21.15.129:0/2052553006 shutdown_connections 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.243+0000 7ff90d200640 1 -- 172.21.15.129:0/2052553006 >> 172.21.15.129:0/2052553006 conn(0x7ff90806d410 msgr2=0x7ff9080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.244+0000 7ff90d200640 1 -- 172.21.15.129:0/2052553006 shutdown_connections 2024-09-17T03:57:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.244+0000 7ff90d200640 1 -- 172.21.15.129:0/2052553006 wait complete. 2024-09-17T03:57:44.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.246+0000 7ff90d200640 1 Processor -- start 2024-09-17T03:57:44.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.246+0000 7ff90d200640 1 -- start start 2024-09-17T03:57:44.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8ec004800 0x7ff908159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:44.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:44.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff90815c3c0 0x7ff90815c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:44.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff908160220 con 0x7ff908157ae0 2024-09-17T03:57:44.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff9080cd5a0 con 0x7ff8ec004800 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.247+0000 7ff90d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff90806f2e0 con 0x7ff90815c3c0 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35756/0 (socket says 172.21.15.129:35756) 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 -- 172.21.15.129:0/234838992 learned_addr learned my addr 172.21.15.129:0/234838992 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff905e00640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8ec004800 0x7ff908159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff906800640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff90815c3c0 0x7ff90815c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 -- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff90815c3c0 msgr2=0x7ff90815c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff90815c3c0 0x7ff90815c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 -- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8ec004800 msgr2=0x7ff908159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8ec004800 0x7ff908159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 -- 172.21.15.129:0/234838992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9081585a0 con 0x7ff908157ae0 2024-09-17T03:57:44.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.248+0000 7ff8fd400640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c3d0 tx=0x7ff8f0006d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:44.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.249+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0031040 con 0x7ff908157ae0 2024-09-17T03:57:44.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.249+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8f0003610 con 0x7ff908157ae0 2024-09-17T03:57:44.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.249+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff908158880 con 0x7ff908157ae0 2024-09-17T03:57:44.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.249+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f0003a60 con 0x7ff908157ae0 2024-09-17T03:57:44.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.249+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9080c7a60 con 0x7ff908157ae0 2024-09-17T03:57:44.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.250+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff908158310 con 0x7ff908157ae0 2024-09-17T03:57:44.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.251+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff8f0034070 con 0x7ff908157ae0 2024-09-17T03:57:44.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.253+0000 7ff8fea00640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 0x7ff8e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:44.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.253+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7ff8f00badd0 con 0x7ff908157ae0 2024-09-17T03:57:44.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.253+0000 7ff905e00640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 0x7ff8e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:44.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.254+0000 7ff905e00640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 0x7ff8e407b0a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009c50 tx=0x7ff8f8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:44.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.258+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff8f0083130 con 0x7ff908157ae0 2024-09-17T03:57:44.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.475+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9080b4140 con 0x7ff908157ae0 2024-09-17T03:57:44.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.475+0000 7ff8fea00640 1 -- 172.21.15.129:0/234838992 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7ff8f0082a30 con 0x7ff908157ae0 2024-09-17T03:57:44.476 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:44.476 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:44.476 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 msgr2=0x7ff8e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 0x7ff8e407b0a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff8f8009c50 tx=0x7ff8f8005f90 comp rx=0 tx=0).stop 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 msgr2=0x7ff9080c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7ff8f000c3d0 tx=0x7ff8f0006d00 comp rx=0 tx=0).stop 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 shutdown_connections 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff8e4078b90 0x7ff8e407b0a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff90815c3c0 0x7ff90815c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff908157ae0 0x7ff9080c7f80 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 --2- 172.21.15.129:0/234838992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8ec004800 0x7ff908159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 >> 172.21.15.129:0/234838992 conn(0x7ff90806d410 msgr2=0x7ff90815eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 shutdown_connections 2024-09-17T03:57:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:44.478+0000 7ff90d200640 1 -- 172.21.15.129:0/234838992 wait complete. 2024-09-17T03:57:44.703 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:44.704 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:45.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:45.839 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:45.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 -- 172.21.15.129:0/1999327911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f88004800 msgr2=0x7f3f88004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:45.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1999327911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f88004800 0x7f3f88004c10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c009b50 tx=0x7f3f88005830 comp rx=0 tx=0).stop 2024-09-17T03:57:45.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa3400640 1 -- 172.21.15.129:0/1999327911 reap_dead start 2024-09-17T03:57:45.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 -- 172.21.15.129:0/1999327911 shutdown_connections 2024-09-17T03:57:45.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 -- 172.21.15.129:0/1999327911 >> 172.21.15.129:0/1999327911 conn(0x7f3fa406d410 msgr2=0x7f3fa40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:45.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 -- 172.21.15.129:0/1999327911 shutdown_connections 2024-09-17T03:57:45.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.939+0000 7f3fa9800640 1 -- 172.21.15.129:0/1999327911 wait complete. 2024-09-17T03:57:45.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.942+0000 7f3fa9800640 1 Processor -- start 2024-09-17T03:57:45.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.942+0000 7f3fa9800640 1 -- start start 2024-09-17T03:57:45.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f88004800 0x7f3fa41f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:45.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 0x7f3fa40cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:45.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40b4300 con 0x7f3fa40cefc0 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa40b4180 con 0x7f3fa4153e60 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3fa40b4480 con 0x7f3f88004800 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35768/0 (socket says 172.21.15.129:35768) 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.943+0000 7f3fa3400640 1 -- 172.21.15.129:0/1736507890 learned_addr learned my addr 172.21.15.129:0/1736507890 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3f9a000640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 0x7f3fa40cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa2a00640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f88004800 0x7f3fa41f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 -- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f88004800 msgr2=0x7f3fa41f76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f88004800 0x7f3fa41f76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 -- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 msgr2=0x7f3fa40cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 0x7f3fa40cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:45.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 -- 172.21.15.129:0/1736507890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa40cf3d0 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3f9a000640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 0x7f3fa40cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa3400640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40cddd0 tx=0x7f3f9c006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c00fa80 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f9c019070 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa41548e0 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9c011430 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.944+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41f6eb0 con 0x7f3fa40cefc0 2024-09-17T03:57:45.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.945+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4154e00 con 0x7f3fa40cefc0 2024-09-17T03:57:45.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.947+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f9c0115d0 con 0x7f3fa40cefc0 2024-09-17T03:57:45.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.948+0000 7f3f9b400640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 0x7f3f8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:45.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.948+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f3f9c09b2e0 con 0x7f3fa40cefc0 2024-09-17T03:57:45.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.948+0000 7f3fa2a00640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 0x7f3f8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:45.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.949+0000 7f3fa2a00640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 0x7f3f8007b0a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3f94005eb0 tx=0x7f3f94005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:45.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:45.952+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3fa4154e00 con 0x7f3fa40cefc0 2024-09-17T03:57:46.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.174+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fa40b29f0 con 0x7f3fa40cefc0 2024-09-17T03:57:46.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.174+0000 7f3f9b400640 1 -- 172.21.15.129:0/1736507890 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f3fa40cf3d0 con 0x7f3fa40cefc0 2024-09-17T03:57:46.174 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:46.175 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:46.175 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 msgr2=0x7f3f8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 0x7f3f8007b0a0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3f94005eb0 tx=0x7f3f94005f90 comp rx=0 tx=0).stop 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 msgr2=0x7f3fa4157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f3fa40cddd0 tx=0x7f3f9c006720 comp rx=0 tx=0).stop 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 shutdown_connections 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f80078b90 0x7f3f8007b0a0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fa40cefc0 0x7f3fa4157420 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fa4153e60 0x7f3fa40cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:46.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 --2- 172.21.15.129:0/1736507890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f88004800 0x7f3fa41f76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 >> 172.21.15.129:0/1736507890 conn(0x7f3fa406d410 msgr2=0x7f3fa40b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 shutdown_connections 2024-09-17T03:57:46.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:46.177+0000 7f3fa9800640 1 -- 172.21.15.129:0/1736507890 wait complete. 2024-09-17T03:57:46.401 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:46.401 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:47.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:57:47.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:57:47.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:47.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 -- 172.21.15.129:0/497503502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e8070a50 msgr2=0x7f11e80cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:47.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 --2- 172.21.15.129:0/497503502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e8070a50 0x7f11e80cbb00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f11e80c3d50 tx=0x7f11d802a540 comp rx=0 tx=0).stop 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 -- 172.21.15.129:0/497503502 shutdown_connections 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 --2- 172.21.15.129:0/497503502 >> v2:172.21.15.181:3300/0 conn(0x7f11e80cc910 0x7f11e80ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 --2- 172.21.15.129:0/497503502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e8070a50 0x7f11e80cbb00 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 -- 172.21.15.129:0/497503502 >> 172.21.15.129:0/497503502 conn(0x7f11e806d410 msgr2=0x7f11e80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 -- 172.21.15.129:0/497503502 shutdown_connections 2024-09-17T03:57:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.631+0000 7f11ee400640 1 -- 172.21.15.129:0/497503502 wait complete. 2024-09-17T03:57:47.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.633+0000 7f11ee400640 1 Processor -- start 2024-09-17T03:57:47.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.634+0000 7f11ee400640 1 -- start start 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11e8070a50 0x7f11e81593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 0x7f11e815a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f11e80cc1c0 con 0x7f11e80cc910 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11e80cc040 con 0x7f11e8070a50 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11ee400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f11e80af070 con 0x7f11e80cc3e0 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11e7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35786/0 (socket says 172.21.15.129:35786) 2024-09-17T03:57:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11e7e00640 1 -- 172.21.15.129:0/1358562801 learned_addr learned my addr 172.21.15.129:0/1358562801 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.635+0000 7f11e7400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11e8070a50 0x7f11e81593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11dea00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 0x7f11e815a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 -- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 msgr2=0x7f11e815a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 0x7f11e815a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 -- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11e8070a50 msgr2=0x7f11e81593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11e8070a50 0x7f11e81593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 -- 172.21.15.129:0/1358562801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8159900 con 0x7f11e80cc910 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11dea00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 0x7f11e815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:47.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11e7e00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f11e000d9a0 tx=0x7f11e000de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0017070 con 0x7f11e80cc910 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11e000bda0 con 0x7f11e80cc910 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e80b0c20 con 0x7f11e80cc910 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.636+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00106c0 con 0x7f11e80cc910 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.637+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e80b11e0 con 0x7f11e80cc910 2024-09-17T03:57:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.638+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e80cbb00 con 0x7f11e80cc910 2024-09-17T03:57:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.639+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f11e0010860 con 0x7f11e80cc910 2024-09-17T03:57:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.640+0000 7f11dfe00640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 0x7f11c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.640+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f11e009b1f0 con 0x7f11e80cc910 2024-09-17T03:57:47.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.640+0000 7f11e7400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 0x7f11c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:47.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.641+0000 7f11e7400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 0x7f11c407b0a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f11d8034000 tx=0x7f11d8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:47.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.644+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11e8159900 con 0x7f11e80cc910 2024-09-17T03:57:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:47.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.864+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11e806a140 con 0x7f11e80cc910 2024-09-17T03:57:47.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.865+0000 7f11dfe00640 1 -- 172.21.15.129:0/1358562801 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f11e0063020 con 0x7f11e80cc910 2024-09-17T03:57:47.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:47.865 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:47.865 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:47.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 msgr2=0x7f11c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:47.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 0x7f11c407b0a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f11d8034000 tx=0x7f11d8004730 comp rx=0 tx=0).stop 2024-09-17T03:57:47.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 msgr2=0x7f11e815a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:47.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f11e000d9a0 tx=0x7f11e000de60 comp rx=0 tx=0).stop 2024-09-17T03:57:47.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 shutdown_connections 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f11c4078b90 0x7f11c407b0a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.867+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11e80cc910 0x7f11e815a960 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.868+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11e80cc3e0 0x7f11e815a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.868+0000 7f11ee400640 1 --2- 172.21.15.129:0/1358562801 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11e8070a50 0x7f11e81593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.868+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 >> 172.21.15.129:0/1358562801 conn(0x7f11e806d410 msgr2=0x7f11e80ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.868+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 shutdown_connections 2024-09-17T03:57:47.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:47.868+0000 7f11ee400640 1 -- 172.21.15.129:0/1358562801 wait complete. 2024-09-17T03:57:48.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:48.093 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:49.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:49.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 -- 172.21.15.129:0/585669653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00cf8a0 msgr2=0x7fedf00cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 --2- 172.21.15.129:0/585669653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00cf8a0 0x7fedf00cd860 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fede0005fd0 tx=0x7fede002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 -- 172.21.15.129:0/585669653 shutdown_connections 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 --2- 172.21.15.129:0/585669653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf0160af0 0x7fedf00b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 --2- 172.21.15.129:0/585669653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00cf8a0 0x7fedf00cd860 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 -- 172.21.15.129:0/585669653 >> 172.21.15.129:0/585669653 conn(0x7fedf006cfc0 msgr2=0x7fedf006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 -- 172.21.15.129:0/585669653 shutdown_connections 2024-09-17T03:57:49.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.326+0000 7fedf5e00640 1 -- 172.21.15.129:0/585669653 wait complete. 2024-09-17T03:57:49.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.328+0000 7fedf5e00640 1 Processor -- start 2024-09-17T03:57:49.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.329+0000 7fedf5e00640 1 -- start start 2024-09-17T03:57:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf00cf8a0 0x7fedf01e8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00d03a0 0x7fedf01f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fedf015cf70 con 0x7fedf0160af0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fedf015cdf0 con 0x7fedf00d03a0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fedf015d0f0 con 0x7fedf00cf8a0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35802/0 (socket says 172.21.15.129:35802) 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf5400640 1 -- 172.21.15.129:0/68824962 learned_addr learned my addr 172.21.15.129:0/68824962 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.330+0000 7fedf4a00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf00cf8a0 0x7fedf01e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fede7e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00d03a0 0x7fedf01f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 -- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf00cf8a0 msgr2=0x7fedf01e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf00cf8a0 0x7fedf01e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 -- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00d03a0 msgr2=0x7fedf01f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00d03a0 0x7fedf01f2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 -- 172.21.15.129:0/68824962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf01e74c0 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5400640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fede800c6b0 tx=0x7fede800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8003b70 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fede8003d30 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8005330 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.331+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf01e77a0 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.332+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf01e7d60 con 0x7fedf0160af0 2024-09-17T03:57:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.333+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf015d5a0 con 0x7fedf0160af0 2024-09-17T03:57:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.334+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fede80054d0 con 0x7fedf0160af0 2024-09-17T03:57:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.335+0000 7feded600640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 0x7fedcc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.336+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fede809aa40 con 0x7fedf0160af0 2024-09-17T03:57:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.336+0000 7fedf4a00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 0x7fedcc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.337+0000 7fedf4a00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 0x7fedcc07b0a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fede0033000 tx=0x7fede0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:49.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.340+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fede8062750 con 0x7fedf0160af0 2024-09-17T03:57:49.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.560+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fedf01e7250 con 0x7fedf0160af0 2024-09-17T03:57:49.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.560+0000 7feded600640 1 -- 172.21.15.129:0/68824962 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7fede8067180 con 0x7fedf0160af0 2024-09-17T03:57:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:49.560 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:49.561 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 msgr2=0x7fedcc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 0x7fedcc07b0a0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fede0033000 tx=0x7fede0004730 comp rx=0 tx=0).stop 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 msgr2=0x7fedf00c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fede800c6b0 tx=0x7fede800cb70 comp rx=0 tx=0).stop 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 shutdown_connections 2024-09-17T03:57:49.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fedcc078b90 0x7fedcc07b0a0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fedf0160af0 0x7fedf00c9610 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fedf00d03a0 0x7fedf01f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 --2- 172.21.15.129:0/68824962 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fedf00cf8a0 0x7fedf01e8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 >> 172.21.15.129:0/68824962 conn(0x7fedf006cfc0 msgr2=0x7fedf00cc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 shutdown_connections 2024-09-17T03:57:49.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:49.563+0000 7fedf5e00640 1 -- 172.21.15.129:0/68824962 wait complete. 2024-09-17T03:57:49.787 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:49.787 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:49.843 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:50.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:51.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 -- 172.21.15.129:0/3446962293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f9c003a00 msgr2=0x7f2f9c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3446962293 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f9c003a00 0x7f2f9c003e70 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009b50 tx=0x7f2f9c005830 comp rx=0 tx=0).stop 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fb7e00640 1 -- 172.21.15.129:0/3446962293 reap_dead start 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 -- 172.21.15.129:0/3446962293 shutdown_connections 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 -- 172.21.15.129:0/3446962293 >> 172.21.15.129:0/3446962293 conn(0x7f2fb806d410 msgr2=0x7f2fb80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 -- 172.21.15.129:0/3446962293 shutdown_connections 2024-09-17T03:57:51.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.020+0000 7f2fbe400640 1 -- 172.21.15.129:0/3446962293 wait complete. 2024-09-17T03:57:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.022+0000 7f2fbe400640 1 Processor -- start 2024-09-17T03:57:51.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.023+0000 7f2fbe400640 1 -- start start 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f9c003a00 0x7f2fb81f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fb815cae0 0x7f2fb815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80cb150 con 0x7f2fb815b2d0 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb80cafd0 con 0x7f2fb815cae0 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fbe400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb81f6df0 con 0x7f2f9c003a00 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fb6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fb6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35826/0 (socket says 172.21.15.129:35826) 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fb6a00640 1 -- 172.21.15.129:0/3617945126 learned_addr learned my addr 172.21.15.129:0/3617945126 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fb7400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f9c003a00 0x7f2fb81f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.024+0000 7f2fb7e00640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fb815cae0 0x7f2fb815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 -- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f9c003a00 msgr2=0x7f2fb81f6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f9c003a00 0x7f2fb81f6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 -- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fb815cae0 msgr2=0x7f2fb815cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fb815cae0 0x7f2fb815cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 -- 172.21.15.129:0/3617945126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb815d430 con 0x7f2fb815b2d0 2024-09-17T03:57:51.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fb6a00640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4011a90 tx=0x7f2fa4011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa401e070 con 0x7f2fb815b2d0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2fa400fe70 con 0x7f2fb815b2d0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fa40042b0 con 0x7f2fb815b2d0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.025+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb815f260 con 0x7f2fb815b2d0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.026+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb80741c0 con 0x7f2fb815b2d0 2024-09-17T03:57:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.027+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fb80b4290 con 0x7f2fb815b2d0 2024-09-17T03:57:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.028+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2fa4021070 con 0x7f2fb815b2d0 2024-09-17T03:57:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.029+0000 7f2faf400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 0x7f2f9807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.029+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f2fa409b810 con 0x7f2fb815b2d0 2024-09-17T03:57:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.030+0000 7f2fb7400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 0x7f2f9807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.030+0000 7f2fb7400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 0x7f2f9807b0a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8035000 tx=0x7f2fa802d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:51.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.033+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2fb815d430 con 0x7f2fb815b2d0 2024-09-17T03:57:51.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.259+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fb806a140 con 0x7f2fb815b2d0 2024-09-17T03:57:51.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.259+0000 7f2faf400640 1 -- 172.21.15.129:0/3617945126 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f2fa40635b0 con 0x7f2fb815b2d0 2024-09-17T03:57:51.259 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:51.260 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:51.260 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 msgr2=0x7f2f9807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 0x7f2f9807b0a0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8035000 tx=0x7f2fa802d040 comp rx=0 tx=0).stop 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 msgr2=0x7f2fb8099110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f2fa4011a90 tx=0x7f2fa4011f50 comp rx=0 tx=0).stop 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 shutdown_connections 2024-09-17T03:57:51.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2f98078b90 0x7f2f9807b0a0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2fb815cae0 0x7f2fb815cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2fb815b2d0 0x7f2fb8099110 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 --2- 172.21.15.129:0/3617945126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f9c003a00 0x7f2fb81f6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 >> 172.21.15.129:0/3617945126 conn(0x7f2fb806d410 msgr2=0x7f2fb80700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 shutdown_connections 2024-09-17T03:57:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:51.262+0000 7f2fbe400640 1 -- 172.21.15.129:0/3617945126 wait complete. 2024-09-17T03:57:51.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:51.490 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:52.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:52.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 -- 172.21.15.129:0/3570181823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a50070a50 msgr2=0x7f3a500cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:52.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 --2- 172.21.15.129:0/3570181823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a50070a50 0x7f3a500cbb00 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f3a44009830 tx=0x7f3a4402a5c0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 -- 172.21.15.129:0/3570181823 shutdown_connections 2024-09-17T03:57:52.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 --2- 172.21.15.129:0/3570181823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a50070a50 0x7f3a500cbb00 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 -- 172.21.15.129:0/3570181823 >> 172.21.15.129:0/3570181823 conn(0x7f3a5006d410 msgr2=0x7f3a500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:52.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 -- 172.21.15.129:0/3570181823 shutdown_connections 2024-09-17T03:57:52.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.722+0000 7f3a57e00640 1 -- 172.21.15.129:0/3570181823 wait complete. 2024-09-17T03:57:52.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.724+0000 7f3a57e00640 1 Processor -- start 2024-09-17T03:57:52.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.724+0000 7f3a57e00640 1 -- start start 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a50070a50 0x7f3a50159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a500cc910 0x7f3a5015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3a500af1f0 con 0x7f3a500cc3e0 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3a500af070 con 0x7f3a50070a50 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a57e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3a500af370 con 0x7f3a500cc910 2024-09-17T03:57:52.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:52.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35854/0 (socket says 172.21.15.129:35854) 2024-09-17T03:57:52.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a54a00640 1 -- 172.21.15.129:0/1948505441 learned_addr learned my addr 172.21.15.129:0/1948505441 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a55e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a500cc910 0x7f3a5015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.726+0000 7f3a55400640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a50070a50 0x7f3a50159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 -- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a500cc910 msgr2=0x7f3a5015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a500cc910 0x7f3a5015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 -- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a50070a50 msgr2=0x7f3a50159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a50070a50 0x7f3a50159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 -- 172.21.15.129:0/1948505441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a501f5680 con 0x7f3a500cc3e0 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a54a00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c3a0 tx=0x7f3a3c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:52.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c017070 con 0x7f3a500cc3e0 2024-09-17T03:57:52.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a3c0035f0 con 0x7f3a500cc3e0 2024-09-17T03:57:52.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c003a40 con 0x7f3a500cc3e0 2024-09-17T03:57:52.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a501f9460 con 0x7f3a500cc3e0 2024-09-17T03:57:52.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.727+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a501f9a20 con 0x7f3a500cc3e0 2024-09-17T03:57:52.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.728+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a500af840 con 0x7f3a500cc3e0 2024-09-17T03:57:52.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.730+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3a3c01a070 con 0x7f3a500cc3e0 2024-09-17T03:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.731+0000 7f3a4d600640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 0x7f3a3007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.731+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f3a3c099fe0 con 0x7f3a500cc3e0 2024-09-17T03:57:52.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.731+0000 7f3a55400640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 0x7f3a3007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:52.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.732+0000 7f3a55400640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 0x7f3a3007b0a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3a44034000 tx=0x7f3a44004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:52.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.736+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3a3c0623f0 con 0x7f3a500cc3e0 2024-09-17T03:57:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.955+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a5006a140 con 0x7f3a500cc3e0 2024-09-17T03:57:52.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.955+0000 7f3a4d600640 1 -- 172.21.15.129:0/1948505441 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+3593 (secure 0 0 0) 0x7f3a3c061cf0 con 0x7f3a500cc3e0 2024-09-17T03:57:52.955 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:52.956 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":105,"btime":"2024-09-17T03:57:22:935261+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:22.152671+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":7638,"mds_1":7318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7318":{"gid":7318,"name":"a","rank":1,"incarnation":98,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6835/3555442046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3555442046},{"type":"v1","addr":"172.21.15.129:6835","nonce":3555442046}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 105 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 msgr2=0x7f3a3007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 0x7f3a3007b0a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3a44034000 tx=0x7f3a44004730 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 msgr2=0x7f3a5015b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00c3a0 tx=0x7f3a3c00c860 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 shutdown_connections 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a30078b90 0x7f3a3007b0a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3a500cc910 0x7f3a5015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3a500cc3e0 0x7f3a5015b6c0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.957+0000 7f3a57e00640 1 --2- 172.21.15.129:0/1948505441 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3a50070a50 0x7f3a50159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:52.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.958+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 >> 172.21.15.129:0/1948505441 conn(0x7f3a5006d410 msgr2=0x7f3a500cb5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.958+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 shutdown_connections 2024-09-17T03:57:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:52.958+0000 7f3a57e00640 1 -- 172.21.15.129:0/1948505441 wait complete. 2024-09-17T03:57:53.185 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:57:53.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:22.152671+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 7638, 'mds_1': 7318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7318': {'gid': 7318, 'name': 'a', 'rank': 1, 'incarnation': 98, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6835/3555442046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3555442046}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3555442046}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:53.186 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7318:up:stopping 2024-09-17T03:57:53.848 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:54.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:54.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 -- 172.21.15.129:0/748099763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940b4140 msgr2=0x7fa3940ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 --2- 172.21.15.129:0/748099763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940b4140 0x7fa3940ccc80 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa380005fd0 tx=0x7fa38002a670 comp rx=0 tx=0).stop 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 -- 172.21.15.129:0/748099763 shutdown_connections 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 --2- 172.21.15.129:0/748099763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940b4140 0x7fa3940ccc80 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 -- 172.21.15.129:0/748099763 >> 172.21.15.129:0/748099763 conn(0x7fa39406d410 msgr2=0x7fa3940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 -- 172.21.15.129:0/748099763 shutdown_connections 2024-09-17T03:57:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.408+0000 7fa39aa00640 1 -- 172.21.15.129:0/748099763 wait complete. 2024-09-17T03:57:54.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.410+0000 7fa39aa00640 1 Processor -- start 2024-09-17T03:57:54.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.411+0000 7fa39aa00640 1 -- start start 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3940705a0 0x7fa3940c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3940b4140 0x7fa3940c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa394160260 con 0x7fa3940cd1c0 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa3941600e0 con 0x7fa3940b4140 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa39aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa3941610d0 con 0x7fa3940705a0 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa398a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa398a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60000/0 (socket says 172.21.15.129:60000) 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.412+0000 7fa398a00640 1 -- 172.21.15.129:0/1358871421 learned_addr learned my addr 172.21.15.129:0/1358871421 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa393400640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3940b4140 0x7fa3940c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa393e00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3940705a0 0x7fa3940c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 -- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3940705a0 msgr2=0x7fa3940c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3940705a0 0x7fa3940c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 -- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3940b4140 msgr2=0x7fa3940c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3940b4140 0x7fa3940c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 -- 172.21.15.129:0/1358871421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3941f6600 con 0x7fa3940cd1c0 2024-09-17T03:57:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa398a00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c6b0 tx=0x7fa38c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c003b70 con 0x7fa3940cd1c0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.413+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa38c003d10 con 0x7fa3940cd1c0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.414+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3941f68e0 con 0x7fa3940cd1c0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.414+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38c005330 con 0x7fa3940cd1c0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.414+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3941f7180 con 0x7fa3940cd1c0 2024-09-17T03:57:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.415+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3941f6370 con 0x7fa3940cd1c0 2024-09-17T03:57:54.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.416+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa38c0054d0 con 0x7fa3940cd1c0 2024-09-17T03:57:54.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.417+0000 7fa38be00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 0x7fa37407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:54.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.417+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fa38c09aa70 con 0x7fa3940cd1c0 2024-09-17T03:57:54.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.417+0000 7fa393e00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 0x7fa37407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:54.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.418+0000 7fa393e00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 0x7fa37407b0a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa384000f60 tx=0x7fa384009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:54.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.421+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa38c062790 con 0x7fa3940cd1c0 2024-09-17T03:57:54.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.640+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3941f6f70 con 0x7fa3940cd1c0 2024-09-17T03:57:54.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.641+0000 7fa38be00640 1 -- 172.21.15.129:0/1358871421 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3577 (secure 0 0 0) 0x7fa38c0671c0 con 0x7fa3940cd1c0 2024-09-17T03:57:54.641 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:54.641 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"btime":"2024-09-17T03:57:54:058118+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:52.153828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:54.642 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 107 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.643+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 msgr2=0x7fa37407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.643+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 0x7fa37407b0a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fa384000f60 tx=0x7fa384009040 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 msgr2=0x7fa3941f5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa38c00c6b0 tx=0x7fa38c00cb70 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 shutdown_connections 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa374078b90 0x7fa37407b0a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3940cd1c0 0x7fa3941f5560 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3940b4140 0x7fa3940c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 --2- 172.21.15.129:0/1358871421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3940705a0 0x7fa3940c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:54.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 >> 172.21.15.129:0/1358871421 conn(0x7fa39406d410 msgr2=0x7fa394073f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:54.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 shutdown_connections 2024-09-17T03:57:54.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:54.644+0000 7fa39aa00640 1 -- 172.21.15.129:0/1358871421 wait complete. 2024-09-17T03:57:54.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:56:46.694446+0000', 'modified': '2024-09-17T03:57:52.153828+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7638}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_7638': {'gid': 7638, 'name': 'c', 'rank': 0, 'incarnation': 92, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6834/1495946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1495946}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1495946}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7638, 'qdb_cluster': [7638]} 2024-09-17T03:57:54.870 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:57:54.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T03:57:55.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 -- 172.21.15.129:0/3028657888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1d4003a70 msgr2=0x7fd1d4003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 --2- 172.21.15.129:0/3028657888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1d4003a70 0x7fd1d4003ee0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009b20 tx=0x7fd1d4005830 comp rx=0 tx=0).stop 2024-09-17T03:57:55.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f5800640 1 -- 172.21.15.129:0/3028657888 reap_dead start 2024-09-17T03:57:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 -- 172.21.15.129:0/3028657888 shutdown_connections 2024-09-17T03:57:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 -- 172.21.15.129:0/3028657888 >> 172.21.15.129:0/3028657888 conn(0x7fd1f006cfc0 msgr2=0x7fd1f006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 -- 172.21.15.129:0/3028657888 shutdown_connections 2024-09-17T03:57:55.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.095+0000 7fd1f6200640 1 -- 172.21.15.129:0/3028657888 wait complete. 2024-09-17T03:57:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.097+0000 7fd1f6200640 1 Processor -- start 2024-09-17T03:57:55.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.097+0000 7fd1f6200640 1 -- start start 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1d4003a70 0x7fd1f00af5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1f015bba0 0x7fd1f00a9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f00b4d00 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f00d0c70 con 0x7fd1f015bba0 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f6200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd1f00b4e80 con 0x7fd1d4003a70 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60018/0 (socket says 172.21.15.129:60018) 2024-09-17T03:57:55.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1efe00640 1 -- 172.21.15.129:0/1508165632 learned_addr learned my addr 172.21.15.129:0/1508165632 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:55.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f5800640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1f015bba0 0x7fd1f00a9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.099+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1d4003a70 0x7fd1f00af5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 -- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1d4003a70 msgr2=0x7fd1f00af5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1d4003a70 0x7fd1f00af5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 -- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1f015bba0 msgr2=0x7fd1f00a9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1f015bba0 0x7fd1f00a9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 -- 172.21.15.129:0/1508165632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f015bfb0 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1efe00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00e670 tx=0x7fd1dc00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1ecc00640 1 -- 172.21.15.129:0/1508165632 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc015020 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1ecc00640 1 -- 172.21.15.129:0/1508165632 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1dc0109a0 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f01f4c60 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1ecc00640 1 -- 172.21.15.129:0/1508165632 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc010df0 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.100+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f01f50a0 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.103+0000 7fd1ecc00640 1 -- 172.21.15.129:0/1508165632 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd1dc018020 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.104+0000 7fd1ecc00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 0x7fd1d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.104+0000 7fd1ecc00640 1 -- 172.21.15.129:0/1508165632 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fd1dc09b4c0 con 0x7fd1f01f8cb0 2024-09-17T03:57:55.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.104+0000 7fd1fe7ff740 1 Processor -- start 2024-09-17T03:57:55.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.104+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 0x7fd1d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.104+0000 7fd1fe7ff740 1 -- start start 2024-09-17T03:57:55.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.105+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 0x7fd1d007b0a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0035000 tx=0x7fd1e002d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 0x55794d7130a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d6e3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55794d66c430 con 0x55794d342270 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55794d65c5d0 con 0x55794d3193b0 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55794d64ba00 con 0x55794d321590 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60030/0 (socket says 172.21.15.129:60030) 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2756053428 learned_addr learned my addr 172.21.15.129:0/2756053428 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d6e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 0x55794d7130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 msgr2=0x55794d6e3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d6e3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 msgr2=0x55794d7130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 0x55794d7130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2756053428 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794d614520 con 0x55794d342270 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d6e3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.106+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 0x55794d7130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:55.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.107+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002b4e0 tx=0x7fd1d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588ec3af2768c0ab server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.107+0000 7fd1e5600640 1 -- 172.21.15.129:0/2756053428 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0038040 con 0x55794d342270 2024-09-17T03:57:55.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.107+0000 7fd1e5600640 1 -- 172.21.15.129:0/2756053428 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1e0044070 con 0x55794d342270 2024-09-17T03:57:55.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.107+0000 7fd1e5600640 1 -- 172.21.15.129:0/2756053428 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0005590 con 0x55794d342270 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2756053428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 msgr2=0x55794d712b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002b4e0 tx=0x7fd1d4005830 comp rx=0 tx=0).stop 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2756053428 shutdown_connections 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d6e3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d3193b0 0x55794d7130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2756053428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d342270 0x55794d712b60 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2756053428 >> 172.21.15.129:0/2756053428 conn(0x55794d3363c0 msgr2=0x55794d6e8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2756053428 shutdown_connections 2024-09-17T03:57:55.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.108+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2756053428 wait complete. 2024-09-17T03:57:55.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.109+0000 7fd1fe7ff740 1 -- start start 2024-09-17T03:57:55.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.109+0000 7fd1fe7ff740 1 Processor -- start 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d7277e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d72ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55794d5f2330 con 0x55794d3193b0 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55794d64f810 con 0x55794d342270 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55794d3bc620 con 0x55794d321590 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1fe7ff740 10 client.0 _ll_register_callbacks cb 0x55794d5f7420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60034/0 (socket says 172.21.15.129:60034) 2024-09-17T03:57:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2922683703 learned_addr learned my addr 172.21.15.129:0/2922683703 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d7277e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d72ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 msgr2=0x55794d7277e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d7277e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 msgr2=0x55794d72ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d72ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 -- 172.21.15.129:0/2922683703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794d3ad850 con 0x55794d3193b0 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d7277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d72ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009c10 tx=0x7fd1e00056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bb0286fa5b2e8f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.111+0000 7fd1da000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:57:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0056070 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fd1e00049a0 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1e0044070 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fd1e0061050 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 10 client.8010 fetch_fsmap learned FSMap version 107 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x55794d4246b0 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1e0044070 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 107) ==== 2300+0+0 (secure 0 0 0) 0x7fd1e0064020 con 0x55794d3193b0 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 10 client.8010 fetch_fsmap finished waiting for FSMap version 107 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 10 client.8010 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 20 client.8010 populate_metadata read hostname 'smithi129' 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d300e40 0x55794d70be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 4 client.8010 mds_command: new command op to 7638 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:57:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.112+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 --> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55794d65e520 con 0x55794d300e40 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.113+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d300e40 0x55794d70be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.113+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d300e40 0x55794d70be20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c3be5f4b004d78 server_cookie=fbb90c99f4f446db in_seq=0 out_seq=0 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.113+0000 7fd1da000640 10 client.8010 ms_handle_connect on v2:172.21.15.129:6832/1495946 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1da000640 1 -- 172.21.15.129:0/2922683703 <== mds.0 v2:172.21.15.129:6832/1495946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55794d65e520 con 0x55794d300e40 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1da000640 10 client.8010 handle_command_reply: tid=0 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 client.8010 shutdown 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 msgr2=0x55794d740d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0009c10 tx=0x7fd1e00056f0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 shutdown_connections 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d300e40 0x55794d70be20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d72ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d321590 0x55794d7277e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1da000640 0 client.8010 ms_handle_reset on v2:172.21.15.129:6832/1495946 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2922683703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d3193b0 0x55794d740d80 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 >> 172.21.15.129:0/2922683703 conn(0x55794d3363c0 msgr2=0x55794d70a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 shutdown_connections 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.114+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2922683703 wait complete. 2024-09-17T03:57:55.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.115+0000 7fd1fe7ff740 20 client.8010 trim_cache size 0 max 16384 2024-09-17T03:57:55.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.133+0000 7fd1fe7ff740 1 Processor -- start 2024-09-17T03:57:55.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.133+0000 7fd1fe7ff740 1 -- start start 2024-09-17T03:57:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d70cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d708ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0201400 con 0x55794d321590 2024-09-17T03:57:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f0200630 con 0x55794d342270 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55794d70e6d0 con 0x55794d3193b0 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60040/0 (socket says 172.21.15.129:60040) 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 -- 172.21.15.129:0/3367849558 learned_addr learned my addr 172.21.15.129:0/3367849558 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d70cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1f5800640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d708ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 -- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 msgr2=0x55794d70cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d70cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 -- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 msgr2=0x55794d708ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d708ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1efe00640 1 -- 172.21.15.129:0/3367849558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55794d729510 con 0x55794d321590 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.135+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d70cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.136+0000 7fd1f5800640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d708ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:55.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.136+0000 7fd1efe00640 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc099910 tx=0x7fd1dc010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75744a3875c3a9a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.136+0000 7fd1e4c00640 1 -- 172.21.15.129:0/3367849558 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc067c00 con 0x55794d321590 2024-09-17T03:57:55.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.136+0000 7fd1e4c00640 1 -- 172.21.15.129:0/3367849558 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1dc05f390 con 0x55794d321590 2024-09-17T03:57:55.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.136+0000 7fd1e4c00640 1 -- 172.21.15.129:0/3367849558 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc062380 con 0x55794d321590 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/3367849558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 msgr2=0x55794d7131e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc099910 tx=0x7fd1dc010f90 comp rx=0 tx=0).stop 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/3367849558 shutdown_connections 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d708ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d7131e0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/3367849558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d70cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/3367849558 >> 172.21.15.129:0/3367849558 conn(0x55794d3363c0 msgr2=0x55794d70fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/3367849558 shutdown_connections 2024-09-17T03:57:55.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.137+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/3367849558 wait complete. 2024-09-17T03:57:55.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.138+0000 7fd1fe7ff740 1 -- start start 2024-09-17T03:57:55.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.138+0000 7fd1fe7ff740 1 Processor -- start 2024-09-17T03:57:55.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.139+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d7a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.139+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.139+0000 7fd1fe7ff740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d6e4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55794d729960 con 0x55794d321590 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55794d5f2330 con 0x55794d342270 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1fe7ff740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55794d6e7a50 con 0x55794d3193b0 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1fe7ff740 10 client.0 _ll_register_callbacks cb 0x7fd1e00099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60044/0 (socket says 172.21.15.129:60044) 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 -- 172.21.15.129:0/2287782411 learned_addr learned my addr 172.21.15.129:0/2287782411 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d7a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d6e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 -- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 msgr2=0x55794d7a3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d7a3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 -- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 msgr2=0x55794d6e4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d6e4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 -- 172.21.15.129:0/2287782411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55794d70f000 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1f5800640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d6e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1efe00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc09b010 tx=0x7fd1dc06cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3552fb4ee21775 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.140+0000 7fd1da000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc062580 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fd1dc05ea70 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1dc05f390 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) ==== 24+0+0 (secure 0 0 0) 0x7fd1dc082050 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 10 client.8016 fetch_fsmap learned FSMap version 107 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) -- 0x55794d7ae400 con 0x55794d321590 2024-09-17T03:57:55.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1dc05f390 con 0x55794d321590 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 107) ==== 2300+0+0 (secure 0 0 0) 0x7fd1dc05ec70 con 0x55794d321590 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 10 client.8016 fetch_fsmap finished waiting for FSMap version 107 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 10 client.8016 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 20 client.8016 populate_metadata read hostname 'smithi129' 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d7ad380 0x55794d7ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 4 client.8016 mds_command: new command op to 7638 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.141+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 --> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] -- command(tid 0: {"prefix": "status"}) -- 0x55794d659fd0 con 0x55794d7ad380 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d7ad380 0x55794d7ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1da000640 10 client.8016 ms_handle_connect on v2:172.21.15.129:6832/1495946 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1f4e00640 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d7ad380 0x55794d7ad790 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41ef79281a39c7cb server_cookie=88f5d27d61c6e59 in_seq=0 out_seq=0 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1da000640 1 -- 172.21.15.129:0/2287782411 <== mds.0 v2:172.21.15.129:6832/1495946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x55794d659fd0 con 0x55794d7ad380 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1da000640 10 client.8016 handle_command_reply: tid=0 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1fe7ff740 1 client.8016 shutdown 2024-09-17T03:57:55.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 msgr2=0x55794d6e3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.142+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc09b010 tx=0x7fd1dc06cc90 comp rx=0 tx=0).stop 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 shutdown_connections 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:6832/1495946,v1:172.21.15.129:6834/1495946] conn(0x55794d7ad380 0x55794d7ad790 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55794d342270 0x55794d6e4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55794d321590 0x55794d6e3e40 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1da000640 0 client.8016 ms_handle_reset on v2:172.21.15.129:6832/1495946 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 --2- 172.21.15.129:0/2287782411 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55794d3193b0 0x55794d7a3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 >> 172.21.15.129:0/2287782411 conn(0x55794d3363c0 msgr2=0x55794d70fe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 shutdown_connections 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 1 -- 172.21.15.129:0/2287782411 wait complete. 2024-09-17T03:57:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1fe7ff740 20 client.8016 trim_cache size 0 max 16384 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 7638, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 67.931557596000005, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 106, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 129, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 129, 2024-09-17T03:57:55.148 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 72.997105085000001 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 msgr2=0x7fd1d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 0x7fd1d007b0a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0035000 tx=0x7fd1e002d040 comp rx=0 tx=0).stop 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 msgr2=0x7fd1f015b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.143+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00e670 tx=0x7fd1dc00ea70 comp rx=0 tx=0).stop 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 shutdown_connections 2024-09-17T03:57:55.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd1d0078b90 0x7fd1d007b0a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1f015bba0 0x7fd1f00a9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1f01f8cb0 0x7fd1f015b660 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 --2- 172.21.15.129:0/1508165632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1d4003a70 0x7fd1f00af5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 >> 172.21.15.129:0/1508165632 conn(0x7fd1f006cfc0 msgr2=0x7fd1f00cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 shutdown_connections 2024-09-17T03:57:55.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55.144+0000 7fd1f6200640 1 -- 172.21.15.129:0/1508165632 wait complete. 2024-09-17T03:57:55.167 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:57:55.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:57:55.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:57:55.354 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:57:55.354 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:57:55.355 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport sys\nimport time\nimport os\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nos.mkdir("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked")\nfd = os.open("/home/ubuntu/cephtest/mnt.0/pin/to-be-unlinked", os.O_RDONLY)\nwhile True:\n print("open_dir_background: keeping dir open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi129') 2024-09-17T03:57:55.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-09-17T03:57:55.850 INFO:teuthology.orchestra.run.smithi129.stderr:open_dir_background: keeping dir open 2024-09-17T03:57:55.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:57:55.863 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:57:55.863 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:55.941 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:57:55.941 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:56.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.281+0000 7f1df3e00640 1 -- 172.21.15.129:0/2676012904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec0705a0 msgr2=0x7f1dec0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.281+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2676012904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec0705a0 0x7f1dec0b4140 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1dec072080 tx=0x7f1de002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 -- 172.21.15.129:0/2676012904 shutdown_connections 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2676012904 >> v2:172.21.15.181:3300/0 conn(0x7f1dec06eed0 0x7f1dec0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2676012904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec0705a0 0x7f1dec0b4140 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 -- 172.21.15.129:0/2676012904 >> 172.21.15.129:0/2676012904 conn(0x7f1dec06d410 msgr2=0x7f1dec0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 -- 172.21.15.129:0/2676012904 shutdown_connections 2024-09-17T03:57:56.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.282+0000 7f1df3e00640 1 -- 172.21.15.129:0/2676012904 wait complete. 2024-09-17T03:57:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.284+0000 7f1df3e00640 1 Processor -- start 2024-09-17T03:57:56.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.284+0000 7f1df3e00640 1 -- start start 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 0x7f1dec15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 0x7f1dec0b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec161250 con 0x7f1dec06eed0 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1dec1610d0 con 0x7f1dec06f2e0 2024-09-17T03:57:56.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.285+0000 7f1df3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1dec1613d0 con 0x7f1dec0705a0 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60072/0 (socket says 172.21.15.129:60072) 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 -- 172.21.15.129:0/2701900697 learned_addr learned my addr 172.21.15.129:0/2701900697 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 0x7f1dec0b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df0a00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 0x7f1dec15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 -- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 msgr2=0x7f1dec0b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 0x7f1dec0b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 -- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 msgr2=0x7f1dec15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 0x7f1dec15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 -- 172.21.15.129:0/2701900697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dec0c7ea0 con 0x7f1dec06eed0 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df0a00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 0x7f1dec15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 0x7f1dec0b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:56.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.286+0000 7f1df1400640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f1de0000c00 tx=0x7f1de0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.287+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de003e030 con 0x7f1dec06eed0 2024-09-17T03:57:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.287+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1de0003d10 con 0x7f1dec06eed0 2024-09-17T03:57:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.287+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de0035040 con 0x7f1dec06eed0 2024-09-17T03:57:56.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.287+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dec1f8070 con 0x7f1dec06eed0 2024-09-17T03:57:56.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.287+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dec1f7ab0 con 0x7f1dec06eed0 2024-09-17T03:57:56.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.288+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dec0c8070 con 0x7f1dec06eed0 2024-09-17T03:57:56.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.289+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1de0041030 con 0x7f1dec06eed0 2024-09-17T03:57:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.290+0000 7f1de9600640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 0x7f1dcc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.290+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f1de00bb620 con 0x7f1dec06eed0 2024-09-17T03:57:56.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.290+0000 7f1df0a00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 0x7f1dcc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.291+0000 7f1df0a00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 0x7f1dcc07b0d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1dec1f70c0 tx=0x7f1dd8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:56.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.294+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1de0083900 con 0x7f1dec06eed0 2024-09-17T03:57:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.491+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1dec0cd5a0 con 0x7f1dec06eed0 2024-09-17T03:57:56.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.491+0000 7f1de9600640 1 -- 172.21.15.129:0/2701900697 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v129) ==== 80+0+0 (secure 0 0 0) 0x7f1de0083200 con 0x7f1dec06eed0 2024-09-17T03:57:56.492 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 msgr2=0x7f1dcc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 0x7f1dcc07b0d0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1dec1f70c0 tx=0x7f1dd8005f90 comp rx=0 tx=0).stop 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 msgr2=0x7f1dec0c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f1de0000c00 tx=0x7f1de0003820 comp rx=0 tx=0).stop 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 shutdown_connections 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1dcc078bc0 0x7f1dcc07b0d0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1dec0705a0 0x7f1dec0b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1dec06f2e0 0x7f1dec15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 --2- 172.21.15.129:0/2701900697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1dec06eed0 0x7f1dec0c66e0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 >> 172.21.15.129:0/2701900697 conn(0x7f1dec06d410 msgr2=0x7f1dec0ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 shutdown_connections 2024-09-17T03:57:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.494+0000 7f1df3e00640 1 -- 172.21.15.129:0/2701900697 wait complete. 2024-09-17T03:57:56.516 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:57:56.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:57:56.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:57:56.599 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:57:56.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:57:56.599 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:57:56.652 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:57:56.653 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:57:56.653 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:56.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:57:56.690 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:57:56.690 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:57:56.690 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:56.726 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:57:56.726 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:57:56.726 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:57:56.726 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:57:56.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:57:56.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:56.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 -- 172.21.15.129:0/2220805535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 msgr2=0x7fa4640cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 --2- 172.21.15.129:0/2220805535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640cf580 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fa46406dd50 tx=0x7fa45c02a8b0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 -- 172.21.15.129:0/2220805535 shutdown_connections 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 --2- 172.21.15.129:0/2220805535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640cf580 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 --2- 172.21.15.129:0/2220805535 >> v2:172.21.15.181:3300/0 conn(0x7fa4640d16a0 0x7fa4640cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 -- 172.21.15.129:0/2220805535 >> 172.21.15.129:0/2220805535 conn(0x7fa46408bff0 msgr2=0x7fa46409d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 -- 172.21.15.129:0/2220805535 shutdown_connections 2024-09-17T03:57:56.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.989+0000 7fa469e00640 1 -- 172.21.15.129:0/2220805535 wait complete. 2024-09-17T03:57:56.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.991+0000 7fa469e00640 1 Processor -- start 2024-09-17T03:57:56.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.991+0000 7fa469e00640 1 -- start start 2024-09-17T03:57:56.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4640b9be0 0x7fa4641f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa4641f97f0 0x7fa4640b6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa4640cfac0 con 0x7fa4640b23e0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa4640b3670 con 0x7fa4640b9be0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa4640cfc40 con 0x7fa4641f97f0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60080/0 (socket says 172.21.15.129:60080) 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 -- 172.21.15.129:0/1158292926 learned_addr learned my addr 172.21.15.129:0/1158292926 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa469400640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa4641f97f0 0x7fa4640b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa45be00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4640b9be0 0x7fa4641f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 -- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa4641f97f0 msgr2=0x7fa4640b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa4641f97f0 0x7fa4640b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 -- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4640b9be0 msgr2=0x7fa4641f92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:56.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4640b9be0 0x7fa4641f92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:56.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.993+0000 7fa468a00640 1 -- 172.21.15.129:0/1158292926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4641f65b0 con 0x7fa4640b23e0 2024-09-17T03:57:56.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa468a00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c3c0 tx=0x7fa45400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454017070 con 0x7fa4640b23e0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4540035f0 con 0x7fa4640b23e0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4641f57f0 con 0x7fa4640b23e0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa454003a20 con 0x7fa4640b23e0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.994+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4641f5db0 con 0x7fa4640b23e0 2024-09-17T03:57:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.995+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4640d0120 con 0x7fa4640b23e0 2024-09-17T03:57:56.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.996+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa45401a070 con 0x7fa4640b23e0 2024-09-17T03:57:56.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.997+0000 7fa461600640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 0x7fa44007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:56.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.997+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fa454099fe0 con 0x7fa4640b23e0 2024-09-17T03:57:56.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.998+0000 7fa45be00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 0x7fa44007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:56.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:56.998+0000 7fa45be00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 0x7fa44007b0d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa4640cada0 tx=0x7fa44c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:57.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.002+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa4640b3bd0 con 0x7fa4640b23e0 2024-09-17T03:57:57.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.220+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4640d5560 con 0x7fa4640b23e0 2024-09-17T03:57:57.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.220+0000 7fa461600640 1 -- 172.21.15.129:0/1158292926 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3577 (secure 0 0 0) 0x7fa454062410 con 0x7fa4640b23e0 2024-09-17T03:57:57.220 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:57.221 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"btime":"2024-09-17T03:57:54:058118+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:52.153828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:57.221 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 107 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.222+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 msgr2=0x7fa44007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.222+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 0x7fa44007b0d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa4640cada0 tx=0x7fa44c005f90 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.222+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 msgr2=0x7fa4640c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.222+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7fa45400c3c0 tx=0x7fa45400c880 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 shutdown_connections 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa440078bc0 0x7fa44007b0d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa4641f97f0 0x7fa4640b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4640b9be0 0x7fa4641f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 --2- 172.21.15.129:0/1158292926 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4640b23e0 0x7fa4640c9a60 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 >> 172.21.15.129:0/1158292926 conn(0x7fa46408bff0 msgr2=0x7fa4640d4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 shutdown_connections 2024-09-17T03:57:57.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.223+0000 7fa469e00640 1 -- 172.21.15.129:0/1158292926 wait complete. 2024-09-17T03:57:57.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 -- 172.21.15.129:0/3878179000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 msgr2=0x7f9c68005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3878179000 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c68005150 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f9c78005fd0 tx=0x7f9c68005830 comp rx=0 tx=0).stop 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86200640 1 -- 172.21.15.129:0/3878179000 reap_dead start 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 -- 172.21.15.129:0/3878179000 shutdown_connections 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 -- 172.21.15.129:0/3878179000 >> 172.21.15.129:0/3878179000 conn(0x7f9c8006cfc0 msgr2=0x7f9c8006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 -- 172.21.15.129:0/3878179000 shutdown_connections 2024-09-17T03:57:57.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.465+0000 7f9c86c00640 1 -- 172.21.15.129:0/3878179000 wait complete. 2024-09-17T03:57:57.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.468+0000 7f9c86c00640 1 Processor -- start 2024-09-17T03:57:57.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.468+0000 7f9c86c00640 1 -- start start 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c80154370 0x7f9c80157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 0x7f9c8015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9c80073380 con 0x7f9c68004d40 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9c80073200 con 0x7f9c8015aa40 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c86c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9c80073500 con 0x7f9c80154370 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.469+0000 7f9c85800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60092/0 (socket says 172.21.15.129:60092) 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 -- 172.21.15.129:0/3726585531 learned_addr learned my addr 172.21.15.129:0/3726585531 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c86200640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 0x7f9c8015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c84e00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c80154370 0x7f9c80157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 -- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c80154370 msgr2=0x7f9c80157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c80154370 0x7f9c80157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 -- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 msgr2=0x7f9c8015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 0x7f9c8015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 -- 172.21.15.129:0/3726585531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8015b390 con 0x7f9c68004d40 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c86200640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 0x7f9c8015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c85800640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9c74011b10 tx=0x7f9c74009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.470+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7400fce0 con 0x7f9c68004d40 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.471+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9c7400fe80 con 0x7f9c68004d40 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.471+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74004280 con 0x7f9c68004d40 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.471+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c801f4f60 con 0x7f9c68004d40 2024-09-17T03:57:57.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.471+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c801f5500 con 0x7f9c68004d40 2024-09-17T03:57:57.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.472+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c800cf5c0 con 0x7f9c68004d40 2024-09-17T03:57:57.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.473+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9c740035f0 con 0x7f9c68004d40 2024-09-17T03:57:57.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.474+0000 7f9c7d600640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 0x7f9c6007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.474+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f9c7409b8a0 con 0x7f9c68004d40 2024-09-17T03:57:57.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.474+0000 7f9c84e00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 0x7f9c6007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.475+0000 7f9c84e00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 0x7f9c6007b0d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000fd0 tx=0x7f9c6c0067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:57.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.478+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9c8015b390 con 0x7f9c68004d40 2024-09-17T03:57:57.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.695+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c8006a140 con 0x7f9c68004d40 2024-09-17T03:57:57.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.696+0000 7f9c7d600640 1 -- 172.21.15.129:0/3726585531 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3577 (secure 0 0 0) 0x7f9c7409b090 con 0x7f9c68004d40 2024-09-17T03:57:57.696 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:57.696 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"btime":"2024-09-17T03:57:54:058118+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:52.153828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:57.696 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 107 2024-09-17T03:57:57.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 msgr2=0x7f9c6007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 0x7f9c6007b0d0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c000fd0 tx=0x7f9c6c0067a0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 msgr2=0x7f9c800b4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f9c74011b10 tx=0x7f9c74009f10 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 shutdown_connections 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9c60078bc0 0x7f9c6007b0d0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c8015aa40 0x7f9c8015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9c80154370 0x7f9c80157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 --2- 172.21.15.129:0/3726585531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9c68004d40 0x7f9c800b4700 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 >> 172.21.15.129:0/3726585531 conn(0x7f9c8006cfc0 msgr2=0x7f9c80158550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 shutdown_connections 2024-09-17T03:57:57.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.698+0000 7f9c86c00640 1 -- 172.21.15.129:0/3726585531 wait complete. 2024-09-17T03:57:57.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:57:57.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 -- 172.21.15.129:0/2288369739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3d0004040 msgr2=0x7fb3d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/2288369739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3d0004040 0x7fb3d0004800 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009b20 tx=0x7fb3d0005830 comp rx=0 tx=0).stop 2024-09-17T03:57:57.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3eb400640 1 -- 172.21.15.129:0/2288369739 reap_dead start 2024-09-17T03:57:57.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 -- 172.21.15.129:0/2288369739 shutdown_connections 2024-09-17T03:57:57.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 -- 172.21.15.129:0/2288369739 >> 172.21.15.129:0/2288369739 conn(0x7fb3ec09d8d0 msgr2=0x7fb3ec08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:57.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 -- 172.21.15.129:0/2288369739 shutdown_connections 2024-09-17T03:57:57.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.945+0000 7fb3ebe00640 1 -- 172.21.15.129:0/2288369739 wait complete. 2024-09-17T03:57:57.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.947+0000 7fb3ebe00640 1 Processor -- start 2024-09-17T03:57:57.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.947+0000 7fb3ebe00640 1 -- start start 2024-09-17T03:57:57.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 0x7fb3ec0cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 0x7fb3ec161e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec0d0ae0 con 0x7fb3ec14daf0 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec0ba0d0 con 0x7fb3ec14df00 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3ebe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ec0d0c60 con 0x7fb3d0004040 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60108/0 (socket says 172.21.15.129:60108) 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 -- 172.21.15.129:0/3064992715 learned_addr learned my addr 172.21.15.129:0/3064992715 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3eaa00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 0x7fb3ec0cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3eb400640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 0x7fb3ec161e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 -- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 msgr2=0x7fb3ec0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 0x7fb3ec0cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 -- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 msgr2=0x7fb3ec161e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 0x7fb3ec161e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:57.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3e2000640 1 -- 172.21.15.129:0/3064992715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec0ce330 con 0x7fb3ec14daf0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3eaa00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 0x7fb3ec0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.949+0000 7fb3eb400640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 0x7fb3ec161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3e2000640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009a70 tx=0x7fb3d4033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4038040 con 0x7fb3ec14daf0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb3d4033860 con 0x7fb3ec14daf0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ec0cb2a0 con 0x7fb3ec14daf0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4033cb0 con 0x7fb3ec14daf0 2024-09-17T03:57:57.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.950+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ec0c9740 con 0x7fb3ec14daf0 2024-09-17T03:57:57.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.951+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ec0ce5a0 con 0x7fb3ec14daf0 2024-09-17T03:57:57.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.952+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb3d4043020 con 0x7fb3ec14daf0 2024-09-17T03:57:57.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.953+0000 7fb3e3400640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 0x7fb3c807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:57.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.953+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fb3d40bb320 con 0x7fb3ec14daf0 2024-09-17T03:57:57.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.954+0000 7fb3eaa00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 0x7fb3c807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:57.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.954+0000 7fb3eaa00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 0x7fb3c807b0d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec06f4c0 tx=0x7fb3dc006af0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:57.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:57.958+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb3d4083680 con 0x7fb3ec14daf0 2024-09-17T03:57:58.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.142+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3ec0ceee0 con 0x7fb3ec14daf0 2024-09-17T03:57:58.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.143+0000 7fb3e3400640 1 -- 172.21.15.129:0/3064992715 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+16829 (secure 0 0 0) 0x7fb3d4082f80 con 0x7fb3ec14daf0 2024-09-17T03:57:58.144 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:58.145 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":129,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:57:17.916297+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:56:44.632624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-17T03:56:45.653345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.33","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 msgr2=0x7fb3c807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 0x7fb3c807b0d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec06f4c0 tx=0x7fb3dc006af0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 msgr2=0x7fb3ec161920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009a70 tx=0x7fb3d4033670 comp rx=0 tx=0).stop 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 shutdown_connections 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb3c8078bc0 0x7fb3c807b0d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb3ec14df00 0x7fb3ec161e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3ec14daf0 0x7fb3ec161920 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 --2- 172.21.15.129:0/3064992715 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3d0004040 0x7fb3ec0cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 >> 172.21.15.129:0/3064992715 conn(0x7fb3ec09d8d0 msgr2=0x7fb3ec0b8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:58.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 shutdown_connections 2024-09-17T03:57:58.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.146+0000 7fb3ebe00640 1 -- 172.21.15.129:0/3064992715 wait complete. 2024-09-17T03:57:58.168 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:57:58.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:58.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 -- 172.21.15.129:0/1729516739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44900705a0 msgr2=0x7f44900b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 --2- 172.21.15.129:0/1729516739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44900705a0 0x7f44900b4140 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f4490072e00 tx=0x7f448402a540 comp rx=0 tx=0).stop 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 -- 172.21.15.129:0/1729516739 shutdown_connections 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 --2- 172.21.15.129:0/1729516739 >> v2:172.21.15.181:3300/0 conn(0x7f44900cd390 0x7f44901609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 --2- 172.21.15.129:0/1729516739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f44900705a0 0x7f44900b4140 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 -- 172.21.15.129:0/1729516739 >> 172.21.15.129:0/1729516739 conn(0x7f449006d410 msgr2=0x7f44900b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 -- 172.21.15.129:0/1729516739 shutdown_connections 2024-09-17T03:57:58.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.389+0000 7f4497400640 1 -- 172.21.15.129:0/1729516739 wait complete. 2024-09-17T03:57:58.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.391+0000 7f4497400640 1 Processor -- start 2024-09-17T03:57:58.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.392+0000 7f4497400640 1 -- start start 2024-09-17T03:57:58.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 0x7f449015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f449009bee0 0x7f449009c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4490160250 con 0x7f449006eed0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f449006f4f0 con 0x7f44900705a0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4497400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4490160f00 con 0x7f449009bee0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4494a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4494a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60118/0 (socket says 172.21.15.129:60118) 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.393+0000 7f4494a00640 1 -- 172.21.15.129:0/1126486037 learned_addr learned my addr 172.21.15.129:0/1126486037 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 -- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f449009bee0 msgr2=0x7f449009c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f449009bee0 0x7f449009c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 -- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 msgr2=0x7f449015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f448fe00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 0x7f449015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 0x7f449015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 -- 172.21.15.129:0/1126486037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449009ad80 con 0x7f449006eed0 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f448fe00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 0x7f449015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:58.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4494a00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f4484009780 tx=0x7f4484033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484031d60 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4484033e10 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484005260 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449009b060 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.394+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44900c9b20 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.395+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44900c8370 con 0x7f449006eed0 2024-09-17T03:57:58.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.397+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4484005400 con 0x7f449006eed0 2024-09-17T03:57:58.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.397+0000 7f448cc00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 0x7f447007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.398+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f44840bc0c0 con 0x7f449006eed0 2024-09-17T03:57:58.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.398+0000 7f448fe00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 0x7f447007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.399+0000 7f448fe00640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 0x7f447007b0a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4490074790 tx=0x7f447c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:58.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.402+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f448403f030 con 0x7f449006eed0 2024-09-17T03:57:58.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.619+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44900b4140 con 0x7f449006eed0 2024-09-17T03:57:58.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.620+0000 7f448cc00640 1 -- 172.21.15.129:0/1126486037 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3577 (secure 0 0 0) 0x7f4484084420 con 0x7f449006eed0 2024-09-17T03:57:58.620 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:58.620 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"btime":"2024-09-17T03:57:54:058118+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:52.153828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:58.620 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 107 2024-09-17T03:57:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 msgr2=0x7f447007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 0x7f447007b0a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4490074790 tx=0x7f447c009040 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 msgr2=0x7f44900ca080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f4484009780 tx=0x7f4484033c20 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 shutdown_connections 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4470078b90 0x7f447007b0a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f449009bee0 0x7f449009c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f44900705a0 0x7f449015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 --2- 172.21.15.129:0/1126486037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f449006eed0 0x7f44900ca080 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 >> 172.21.15.129:0/1126486037 conn(0x7f449006d410 msgr2=0x7f44900ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.622+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 shutdown_connections 2024-09-17T03:57:58.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.623+0000 7f4497400640 1 -- 172.21.15.129:0/1126486037 wait complete. 2024-09-17T03:57:58.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:57:58.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 -- 172.21.15.129:0/1212212672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf10068d70 msgr2=0x7fdf100b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 --2- 172.21.15.129:0/1212212672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf10068d70 0x7fdf100b3960 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fdf100a2130 tx=0x7fdf0002a4a0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 -- 172.21.15.129:0/1212212672 shutdown_connections 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 --2- 172.21.15.129:0/1212212672 >> v2:172.21.15.181:3300/0 conn(0x7fdf10074730 0x7fdf100cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 --2- 172.21.15.129:0/1212212672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf10068d70 0x7fdf100b3960 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 -- 172.21.15.129:0/1212212672 >> 172.21.15.129:0/1212212672 conn(0x7fdf100d0a70 msgr2=0x7fdf100d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 -- 172.21.15.129:0/1212212672 shutdown_connections 2024-09-17T03:57:58.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.866+0000 7fdf14e00640 1 -- 172.21.15.129:0/1212212672 wait complete. 2024-09-17T03:57:58.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.868+0000 7fdf14e00640 1 Processor -- start 2024-09-17T03:57:58.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.868+0000 7fdf14e00640 1 -- start start 2024-09-17T03:57:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf10068d70 0x7fdf100a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 0x7fdf100ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100cce60 con 0x7fdf1015c950 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf100ccce0 con 0x7fdf10074730 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf14e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdf100ccfe0 con 0x7fdf10068d70 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf0e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf0e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60136/0 (socket says 172.21.15.129:60136) 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.870+0000 7fdf0e800640 1 -- 172.21.15.129:0/2805096571 learned_addr learned my addr 172.21.15.129:0/2805096571 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0de00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf10068d70 0x7fdf100a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf05400640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 0x7fdf100ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 -- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf10068d70 msgr2=0x7fdf100a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf10068d70 0x7fdf100a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 -- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 msgr2=0x7fdf100ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:58.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 0x7fdf100ca480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 -- 172.21.15.129:0/2805096571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf101f4450 con 0x7fdf1015c950 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf05400640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 0x7fdf100ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf0e800640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800ea70 tx=0x7fdf0800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800ce40 con 0x7fdf1015c950 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf08004c10 con 0x7fdf1015c950 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf101f4730 con 0x7fdf1015c950 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf08005020 con 0x7fdf1015c950 2024-09-17T03:57:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.871+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf101f65c0 con 0x7fdf1015c950 2024-09-17T03:57:58.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.872+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf101f4c50 con 0x7fdf1015c950 2024-09-17T03:57:58.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.874+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdf080051c0 con 0x7fdf1015c950 2024-09-17T03:57:58.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.874+0000 7fdf06a00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 0x7fdeec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:58.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.875+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fdf0809bcc0 con 0x7fdf1015c950 2024-09-17T03:57:58.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.875+0000 7fdf0de00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 0x7fdeec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:58.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.876+0000 7fdf0de00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 0x7fdeec07b0d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdf00034000 tx=0x7fdf00004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:58.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:58.879+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdf101f4c50 con 0x7fdf1015c950 2024-09-17T03:57:59.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.100+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf101f41e0 con 0x7fdf1015c950 2024-09-17T03:57:59.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.100+0000 7fdf06a00640 1 -- 172.21.15.129:0/2805096571 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+3577 (secure 0 0 0) 0x7fdf080a2080 con 0x7fdf1015c950 2024-09-17T03:57:59.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:59.101 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":107,"btime":"2024-09-17T03:57:54:058118+0000","default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:56:46.694446+0000","modified":"2024-09-17T03:57:52.153828+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7638},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_7638":{"gid":7638,"name":"c","rank":0,"incarnation":92,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6834/1495946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1495946},{"type":"v1","addr":"172.21.15.129:6834","nonce":1495946}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7638,"qdb_cluster":[7638]},"id":9}]} 2024-09-17T03:57:59.101 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 107 2024-09-17T03:57:59.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 msgr2=0x7fdeec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 0x7fdeec07b0d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fdf00034000 tx=0x7fdf00004730 comp rx=0 tx=0).stop 2024-09-17T03:57:59.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 msgr2=0x7fdf10155970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fdf0800ea70 tx=0x7fdf0800ef30 comp rx=0 tx=0).stop 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 shutdown_connections 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdeec078bc0 0x7fdeec07b0d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf1015c950 0x7fdf10155970 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf10074730 0x7fdf100ca480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 --2- 172.21.15.129:0/2805096571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf10068d70 0x7fdf100a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 >> 172.21.15.129:0/2805096571 conn(0x7fdf100d0a70 msgr2=0x7fdf10159c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 shutdown_connections 2024-09-17T03:57:59.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.103+0000 7fdf14e00640 1 -- 172.21.15.129:0/2805096571 wait complete. 2024-09-17T03:57:59.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 -- 172.21.15.129:0/2679310713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 msgr2=0x7f0ec80b9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/2679310713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80b9af0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0005fd0 tx=0x7f0ec002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 -- 172.21.15.129:0/2679310713 shutdown_connections 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/2679310713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80b9af0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 -- 172.21.15.129:0/2679310713 >> 172.21.15.129:0/2679310713 conn(0x7f0ec808bff0 msgr2=0x7f0ec809d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 -- 172.21.15.129:0/2679310713 shutdown_connections 2024-09-17T03:57:59.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.353+0000 7f0ec7e00640 1 -- 172.21.15.129:0/2679310713 wait complete. 2024-09-17T03:57:59.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.355+0000 7f0ec7e00640 1 Processor -- start 2024-09-17T03:57:59.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.355+0000 7f0ec7e00640 1 -- start start 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 0x7f0ec809e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec80d4ba0 0x7f0ec81517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec80d3510 con 0x7f0ec80b3440 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec80d3390 con 0x7f0ec80d4ba0 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec80cf7c0 con 0x7f0ec80d1850 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60168/0 (socket says 172.21.15.129:60168) 2024-09-17T03:57:59.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 -- 172.21.15.129:0/872237033 learned_addr learned my addr 172.21.15.129:0/872237033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec7400640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec80d4ba0 0x7f0ec81517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ebfe00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 0x7f0ec809e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 -- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 msgr2=0x7f0ec809e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 0x7f0ec809e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 -- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec80d4ba0 msgr2=0x7f0ec81517e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec80d4ba0 0x7f0ec81517e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ec6a00640 1 -- 172.21.15.129:0/872237033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec8152020 con 0x7f0ec80b3440 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.357+0000 7f0ebfe00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 0x7f0ec809e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ec6a00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c3a0 tx=0x7f0eb800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8017070 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0eb80053c0 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8003850 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec8151d20 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.358+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec81522c0 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.359+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec80cfc70 con 0x7f0ec80b3440 2024-09-17T03:57:59.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.360+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0eb801a070 con 0x7f0ec80b3440 2024-09-17T03:57:59.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.361+0000 7f0ebea00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 0x7f0ea407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.361+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7f0eb809a1c0 con 0x7f0ec80b3440 2024-09-17T03:57:59.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.362+0000 7f0ebfe00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 0x7f0ea407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.362+0000 7f0ebfe00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 0x7f0ea407b0d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00046b0 tx=0x7f0eb0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:59.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.366+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0eb8062520 con 0x7f0ec80b3440 2024-09-17T03:57:59.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.553+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ec809fcf0 con 0x7f0ec80b3440 2024-09-17T03:57:59.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.554+0000 7f0ebea00640 1 -- 172.21.15.129:0/872237033 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) ==== 74+0+16829 (secure 0 0 0) 0x7f0eb8061e20 con 0x7f0ec80b3440 2024-09-17T03:57:59.554 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:57:59.555 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":129,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:57:17.916297+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:56:44.632624+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-17T03:56:45.653345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.21","mappings":[{"from":2,"to":0}]},{"pgid":"19.33","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:57:59.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 msgr2=0x7f0ea407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 0x7f0ea407b0d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00046b0 tx=0x7f0eb0009040 comp rx=0 tx=0).stop 2024-09-17T03:57:59.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 msgr2=0x7f0ec80ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c3a0 tx=0x7f0eb800c860 comp rx=0 tx=0).stop 2024-09-17T03:57:59.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 shutdown_connections 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ea4078bc0 0x7f0ea407b0d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec80d4ba0 0x7f0ec81517e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ec80d1850 0x7f0ec809e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 --2- 172.21.15.129:0/872237033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ec80b3440 0x7f0ec80ce610 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 >> 172.21.15.129:0/872237033 conn(0x7f0ec808bff0 msgr2=0x7f0ec8161d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 shutdown_connections 2024-09-17T03:57:59.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.557+0000 7f0ec7e00640 1 -- 172.21.15.129:0/872237033 wait complete. 2024-09-17T03:57:59.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.801+0000 7fb6d7800640 1 -- 172.21.15.129:0/2190520466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00cd1c0 msgr2=0x7fb6d01609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.801+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2190520466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d01609c0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8005fd0 tx=0x7fb6c802aa90 comp rx=0 tx=0).stop 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 -- 172.21.15.129:0/2190520466 shutdown_connections 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2190520466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d01609c0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2190520466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00b4140 0x7fb6d00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 -- 172.21.15.129:0/2190520466 >> 172.21.15.129:0/2190520466 conn(0x7fb6d006d410 msgr2=0x7fb6d00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 -- 172.21.15.129:0/2190520466 shutdown_connections 2024-09-17T03:57:59.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.802+0000 7fb6d7800640 1 -- 172.21.15.129:0/2190520466 wait complete. 2024-09-17T03:57:59.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.804+0000 7fb6d7800640 1 Processor -- start 2024-09-17T03:57:59.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.804+0000 7fb6d7800640 1 -- start start 2024-09-17T03:57:59.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00705a0 0x7fb6d00c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d00c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d0160260 con 0x7fb6d00b4140 2024-09-17T03:57:59.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d01600e0 con 0x7fb6d00cd1c0 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.805+0000 7fb6d7800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d0160f00 con 0x7fb6d00705a0 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60190/0 (socket says 172.21.15.129:60190) 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 -- 172.21.15.129:0/2523477515 learned_addr learned my addr 172.21.15.129:0/2523477515 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6d4e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00705a0 0x7fb6d00c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6d5800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d00c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 -- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00705a0 msgr2=0x7fb6d00c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00705a0 0x7fb6d00c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 -- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6d00cd1c0 msgr2=0x7fb6d00c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d00c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 -- 172.21.15.129:0/2523477515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d00cada0 con 0x7fb6d00b4140 2024-09-17T03:57:59.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.806+0000 7fb6c7e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800d650 tx=0x7fb6b800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:59.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.807+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8017070 con 0x7fb6d00b4140 2024-09-17T03:57:59.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.807+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6b8005310 con 0x7fb6d00b4140 2024-09-17T03:57:59.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.807+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d00cb080 con 0x7fb6d00b4140 2024-09-17T03:57:59.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.807+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6b8009820 con 0x7fb6d00b4140 2024-09-17T03:57:59.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.807+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d00cb640 con 0x7fb6d00b4140 2024-09-17T03:57:59.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.808+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d00ccc80 con 0x7fb6d00b4140 2024-09-17T03:57:59.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.809+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb6b80099c0 con 0x7fb6d00b4140 2024-09-17T03:57:59.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.811+0000 7fb6cd600640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 0x7fb6ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:57:59.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.811+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 5767+0+0 (secure 0 0 0) 0x7fb6b809ae80 con 0x7fb6d00b4140 2024-09-17T03:57:59.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.811+0000 7fb6d4e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 0x7fb6ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:57:59.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.812+0000 7fb6d4e00640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 0x7fb6ac07b0a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:57:59.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:57:59.815+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb6b8063270 con 0x7fb6d00b4140 2024-09-17T03:58:00.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.031+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb6d006a140 con 0x7fb6d00b4140 2024-09-17T03:58:00.032 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:58:00.032+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:58:00.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.034+0000 7fb6cd600640 1 -- 172.21.15.129:0/2523477515 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v108) ==== 161+0+0 (secure 0 0 0) 0x7fb6b8062b70 con 0x7fb6d00b4140 2024-09-17T03:58:00.034 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.036+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 msgr2=0x7fb6ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 0x7fb6ac07b0a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0007fd0 tx=0x7fb6c0009040 comp rx=0 tx=0).stop 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 msgr2=0x7fb6d00c7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fb6b800d650 tx=0x7fb6b800db10 comp rx=0 tx=0).stop 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 shutdown_connections 2024-09-17T03:58:00.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6ac078b90 0x7fb6ac07b0a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6d00cd1c0 0x7fb6d00c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb6d00b4140 0x7fb6d00c7860 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 --2- 172.21.15.129:0/2523477515 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb6d00705a0 0x7fb6d00c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 >> 172.21.15.129:0/2523477515 conn(0x7fb6d006d410 msgr2=0x7fb6d015e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 shutdown_connections 2024-09-17T03:58:00.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.037+0000 7fb6d7800640 1 -- 172.21.15.129:0/2523477515 wait complete. 2024-09-17T03:58:00.046 INFO:tasks.ceph.mds.c.smithi129.stderr: -9857> 2024-09-17T03:56:53.180+0000 7f809be00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:58:00.049 INFO:tasks.ceph.mds.c.smithi129.stderr: -8136> 2024-09-17T03:56:58.181+0000 7f809be00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-09-17T03:58:00.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:58:00.175 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 -- 172.21.15.129:0/3268198421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 msgr2=0x7f648c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 --2- 172.21.15.129:0/3268198421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c0cd5e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f647c009be0 tx=0x7f647c02a920 comp rx=0 tx=0).stop 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 -- 172.21.15.129:0/3268198421 shutdown_connections 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 --2- 172.21.15.129:0/3268198421 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c0cd5e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 -- 172.21.15.129:0/3268198421 >> 172.21.15.129:0/3268198421 conn(0x7f648c06cfc0 msgr2=0x7f648c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 -- 172.21.15.129:0/3268198421 shutdown_connections 2024-09-17T03:58:00.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.289+0000 7f6491000640 1 -- 172.21.15.129:0/3268198421 wait complete. 2024-09-17T03:58:00.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.292+0000 7f6491000640 1 Processor -- start 2024-09-17T03:58:00.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.292+0000 7f6491000640 1 -- start start 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 0x7f648c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f648c15d630 con 0x7f648c0b4ee0 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f648c0b5540 con 0x7f648c1492e0 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f6491000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f648c15d7b0 con 0x7f648c0cdf10 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f648b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:00.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f648b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60222/0 (socket says 172.21.15.129:60222) 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.293+0000 7f648b400640 1 -- 172.21.15.129:0/1353318169 learned_addr learned my addr 172.21.15.129:0/1353318169 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6482a00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648be00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 0x7f648c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 -- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 msgr2=0x7f648c155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 -- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 msgr2=0x7f648c1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 0x7f648c1560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 -- 172.21.15.129:0/1353318169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648c15c030 con 0x7f648c0b4ee0 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6482a00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648be00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 0x7f648c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f648b400640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f647c000f80 tx=0x7f647c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c030e60 con 0x7f648c0b4ee0 2024-09-17T03:58:00.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f647c03e070 con 0x7f648c0b4ee0 2024-09-17T03:58:00.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c032330 con 0x7f648c0b4ee0 2024-09-17T03:58:00.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.294+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f648c14a850 con 0x7f648c0b4ee0 2024-09-17T03:58:00.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.295+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f648c1f3b60 con 0x7f648c0b4ee0 2024-09-17T03:58:00.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.296+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648c1f3dc0 con 0x7f648c0b4ee0 2024-09-17T03:58:00.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.297+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f647c0324d0 con 0x7f648c0b4ee0 2024-09-17T03:58:00.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.298+0000 7f6483e00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 0x7f646807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:00.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.298+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5853+0+0 (secure 0 0 0) 0x7f647c0bc5b0 con 0x7f648c0b4ee0 2024-09-17T03:58:00.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.298+0000 7f6482a00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 0x7f646807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:00.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.299+0000 7f6482a00640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 0x7f646807b0a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f648c1506c0 tx=0x7f6474009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:00.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.302+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f647c02d070 con 0x7f648c0b4ee0 2024-09-17T03:58:00.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:00.522+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f648c0cd5e0 con 0x7f648c0b4ee0 2024-09-17T03:58:01.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.101+0000 7f6483e00640 1 -- 172.21.15.129:0/1353318169 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v109) ==== 104+0+0 (secure 0 0 0) 0x7f647c084950 con 0x7f648c0b4ee0 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 msgr2=0x7f646807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 0x7f646807b0a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f648c1506c0 tx=0x7f6474009040 comp rx=0 tx=0).stop 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 msgr2=0x7f648c148a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f647c000f80 tx=0x7f647c003c20 comp rx=0 tx=0).stop 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 shutdown_connections 2024-09-17T03:58:01.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6468078b90 0x7f646807b0a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f648c1492e0 0x7f648c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f648c0cdf10 0x7f648c155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 --2- 172.21.15.129:0/1353318169 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f648c0b4ee0 0x7f648c148a30 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 >> 172.21.15.129:0/1353318169 conn(0x7f648c06cfc0 msgr2=0x7f648c15e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 shutdown_connections 2024-09-17T03:58:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.104+0000 7f6491000640 1 -- 172.21.15.129:0/1353318169 wait complete. 2024-09-17T03:58:01.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 -- 172.21.15.129:0/267321939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00cd100 msgr2=0x7f29b00c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 --2- 172.21.15.129:0/267321939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00cd100 0x7f29b00c6ab0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f29b0072710 tx=0x7f29a802a9a0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 -- 172.21.15.129:0/267321939 shutdown_connections 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 --2- 172.21.15.129:0/267321939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00cd100 0x7f29b00c6ab0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 --2- 172.21.15.129:0/267321939 >> v2:172.21.15.181:3300/0 conn(0x7f29b0144780 0x7f29b0072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.351+0000 7f29afe00640 1 -- 172.21.15.129:0/267321939 >> 172.21.15.129:0/267321939 conn(0x7f29b00b1c50 msgr2=0x7f29b00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:01.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.352+0000 7f29afe00640 1 -- 172.21.15.129:0/267321939 shutdown_connections 2024-09-17T03:58:01.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.352+0000 7f29afe00640 1 -- 172.21.15.129:0/267321939 wait complete. 2024-09-17T03:58:01.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.354+0000 7f29afe00640 1 Processor -- start 2024-09-17T03:58:01.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.354+0000 7f29afe00640 1 -- start start 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 0x7f29b01e6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29b013f860 0x7f29b013fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29b00ce500 con 0x7f29b00c66a0 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29b00ce380 con 0x7f29b0144780 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.355+0000 7f29afe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f29b00ce680 con 0x7f29b013f860 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60240/0 (socket says 172.21.15.129:60240) 2024-09-17T03:58:01.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 -- 172.21.15.129:0/2822360642 learned_addr learned my addr 172.21.15.129:0/2822360642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29a6000640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 0x7f29b01e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29af400640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29b013f860 0x7f29b013fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 -- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29b013f860 msgr2=0x7f29b013fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29b013f860 0x7f29b013fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 -- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 msgr2=0x7f29b01e6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 0x7f29b01e6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 -- 172.21.15.129:0/2822360642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29b00b3230 con 0x7f29b00c66a0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29a6000640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 0x7f29b01e6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.356+0000 7f29aea00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e7c0 tx=0x7f29a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.357+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a000ccc0 con 0x7f29b00c66a0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.357+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29a000ce80 con 0x7f29b00c66a0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.357+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29b00b3510 con 0x7f29b00c66a0 2024-09-17T03:58:01.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.357+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0005280 con 0x7f29b00c66a0 2024-09-17T03:58:01.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.357+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29b00b3ad0 con 0x7f29b00c66a0 2024-09-17T03:58:01.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.358+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29b00ceba0 con 0x7f29b00c66a0 2024-09-17T03:58:01.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.359+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f29a0005420 con 0x7f29b00c66a0 2024-09-17T03:58:01.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.360+0000 7f29a7400640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 0x7f298c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:01.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.360+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(130..130 src has 1..130) ==== 5853+0+0 (secure 0 0 0) 0x7f29a009b990 con 0x7f29b00c66a0 2024-09-17T03:58:01.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.361+0000 7f29a6000640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 0x7f298c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:01.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.361+0000 7f29a6000640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 0x7f298c07b0a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f29b00b4e70 tx=0x7f2998009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:01.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.365+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f29a0063c20 con 0x7f29b00c66a0 2024-09-17T03:58:01.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:01.550+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f29b015ca50 con 0x7f29b00c66a0 2024-09-17T03:58:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.109+0000 7f29a7400640 1 -- 172.21.15.129:0/2822360642 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v131) ==== 181+0+0 (secure 0 0 0) 0x7f29a009b030 con 0x7f29b00c66a0 2024-09-17T03:58:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:58:02.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.112+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 msgr2=0x7f298c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:02.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.112+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 0x7f298c07b0a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f29b00b4e70 tx=0x7f2998009040 comp rx=0 tx=0).stop 2024-09-17T03:58:02.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.112+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 msgr2=0x7f29b01f6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.112+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f29a000e7c0 tx=0x7f29a000ec80 comp rx=0 tx=0).stop 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.112+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 shutdown_connections 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f298c078b90 0x7f298c07b0a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29b013f860 0x7f29b013fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29b0144780 0x7f29b01e6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 --2- 172.21.15.129:0/2822360642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29b00c66a0 0x7f29b01f6010 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 >> 172.21.15.129:0/2822360642 conn(0x7f29b00b1c50 msgr2=0x7f29b00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 shutdown_connections 2024-09-17T03:58:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.113+0000 7f29afe00640 1 -- 172.21.15.129:0/2822360642 wait complete. 2024-09-17T03:58:02.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:58:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.363+0000 7f2cbca00640 1 -- 172.21.15.129:0/2572217769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb8151d40 msgr2=0x7f2cb8152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.363+0000 7f2cbca00640 1 --2- 172.21.15.129:0/2572217769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb8151d40 0x7f2cb8152150 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0005fd0 tx=0x7f2cb002a9f0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 -- 172.21.15.129:0/2572217769 shutdown_connections 2024-09-17T03:58:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 --2- 172.21.15.129:0/2572217769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb8151d40 0x7f2cb8152150 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 --2- 172.21.15.129:0/2572217769 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb80cfbc0 0x7f2cb80cffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 -- 172.21.15.129:0/2572217769 >> 172.21.15.129:0/2572217769 conn(0x7f2cb80cd570 msgr2=0x7f2cb80cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:02.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 -- 172.21.15.129:0/2572217769 shutdown_connections 2024-09-17T03:58:02.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.364+0000 7f2cbca00640 1 -- 172.21.15.129:0/2572217769 wait complete. 2024-09-17T03:58:02.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.366+0000 7f2cbca00640 1 Processor -- start 2024-09-17T03:58:02.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.366+0000 7f2cbca00640 1 -- start start 2024-09-17T03:58:02.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.367+0000 7f2cbca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:02.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.367+0000 7f2cbca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb8151d40 0x7f2cb8153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:02.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cbca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb815c0c0 0x7f2cb814ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:02.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cbca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb8152810 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cbca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb8152690 con 0x7f2cb815c0c0 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cbca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb8152990 con 0x7f2cb8151d40 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60254/0 (socket says 172.21.15.129:60254) 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 -- 172.21.15.129:0/1218487844 learned_addr learned my addr 172.21.15.129:0/1218487844 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb6800640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb815c0c0 0x7f2cb814ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cafe00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb8151d40 0x7f2cb8153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 -- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb8151d40 msgr2=0x7f2cb8153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb8151d40 0x7f2cb8153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 -- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb815c0c0 msgr2=0x7f2cb814ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb815c0c0 0x7f2cb814ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.368+0000 7f2cb5e00640 1 -- 172.21.15.129:0/1218487844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb81f5a80 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cb5e00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800b410 tx=0x7f2ca800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8015020 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ca8011940 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb81f5f00 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8011d90 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.369+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb81f64c0 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.370+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb80cffd0 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.371+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2ca8018020 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.372+0000 7f2cae000640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 0x7f2c9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:02.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.372+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 5415+0+0 (secure 0 0 0) 0x7f2ca809aa30 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.373+0000 7f2cafe00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 0x7f2c9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:02.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.374+0000 7f2cafe00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 0x7f2c9407b0a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0006fd0 tx=0x7f2ca0008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:02.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.377+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2ca801f020 con 0x7f2cb80cfbc0 2024-09-17T03:58:02.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:02.560+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2cb8152360 con 0x7f2cb80cfbc0 2024-09-17T03:58:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.115+0000 7f2cae000640 1 -- 172.21.15.129:0/1218487844 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v132) ==== 169+0+0 (secure 0 0 0) 0x7f2ca8063020 con 0x7f2cb80cfbc0 2024-09-17T03:58:03.116 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:58:03.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.122+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 msgr2=0x7f2c9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.122+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 0x7f2c9407b0a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0006fd0 tx=0x7f2ca0008040 comp rx=0 tx=0).stop 2024-09-17T03:58:03.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.123+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 msgr2=0x7f2cb80b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.123+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f2ca800b410 tx=0x7f2ca800b8d0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 shutdown_connections 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2c94078b90 0x7f2c9407b0a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2cb815c0c0 0x7f2cb814ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2cb8151d40 0x7f2cb8153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 --2- 172.21.15.129:0/1218487844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2cb80cfbc0 0x7f2cb80b4310 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.124+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 >> 172.21.15.129:0/1218487844 conn(0x7f2cb80cd570 msgr2=0x7f2cb80b62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:03.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.125+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 shutdown_connections 2024-09-17T03:58:03.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.126+0000 7f2cbca00640 1 -- 172.21.15.129:0/1218487844 wait complete. 2024-09-17T03:58:03.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:58:03.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 -- 172.21.15.129:0/109574476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80340705a0 msgr2=0x7f80340b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 --2- 172.21.15.129:0/109574476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80340705a0 0x7f80340b4140 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f80340d1370 tx=0x7f802402a530 comp rx=0 tx=0).stop 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 -- 172.21.15.129:0/109574476 shutdown_connections 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 --2- 172.21.15.129:0/109574476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340cd390 0x7f80341609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 --2- 172.21.15.129:0/109574476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f80340705a0 0x7f80340b4140 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 -- 172.21.15.129:0/109574476 >> 172.21.15.129:0/109574476 conn(0x7f803406d410 msgr2=0x7f80340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 -- 172.21.15.129:0/109574476 shutdown_connections 2024-09-17T03:58:03.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.381+0000 7f803a600640 1 -- 172.21.15.129:0/109574476 wait complete. 2024-09-17T03:58:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.384+0000 7f803a600640 1 Processor -- start 2024-09-17T03:58:03.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.384+0000 7f803a600640 1 -- start start 2024-09-17T03:58:03.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f803406eed0 0x7f80340c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340705a0 0x7f80340c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8034160f00 con 0x7f8034159200 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f803406f4f0 con 0x7f803406eed0 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f803a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8034161080 con 0x7f80340705a0 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f8033e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f8033e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60280/0 (socket says 172.21.15.129:60280) 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.385+0000 7f8033e00640 1 -- 172.21.15.129:0/948938747 learned_addr learned my addr 172.21.15.129:0/948938747 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033400640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f803406eed0 0x7f80340c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f802be00640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340705a0 0x7f80340c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 -- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340705a0 msgr2=0x7f80340c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340705a0 0x7f80340c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 -- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f803406eed0 msgr2=0x7f80340c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f803406eed0 0x7f80340c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 -- 172.21.15.129:0/948938747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803415bd40 con 0x7f8034159200 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f8033e00640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f802c00e9c0 tx=0x7f802c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c004d20 con 0x7f8034159200 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.386+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f802c00cb20 con 0x7f8034159200 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.387+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f802c005020 con 0x7f8034159200 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.387+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803415c1e0 con 0x7f8034159200 2024-09-17T03:58:03.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.387+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80340c73c0 con 0x7f8034159200 2024-09-17T03:58:03.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.388+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80340b4140 con 0x7f8034159200 2024-09-17T03:58:03.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.389+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f802c0051c0 con 0x7f8034159200 2024-09-17T03:58:03.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.390+0000 7f802b400640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 0x7f801007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.391+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f802c09b230 con 0x7f8034159200 2024-09-17T03:58:03.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.391+0000 7f8033400640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 0x7f801007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.391+0000 7f8033400640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 0x7f801007b0d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8024033000 tx=0x7f8024004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.395+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f802c016030 con 0x7f8034159200 2024-09-17T03:58:03.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.580+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f803406a140 con 0x7f8034159200 2024-09-17T03:58:03.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.585+0000 7f802b400640 1 -- 172.21.15.129:0/948938747 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v51) ==== 71+0+0 (secure 0 0 0) 0x7f802c063310 con 0x7f8034159200 2024-09-17T03:58:03.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.587+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 msgr2=0x7f801007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.587+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 0x7f801007b0d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8024033000 tx=0x7f8024004730 comp rx=0 tx=0).stop 2024-09-17T03:58:03.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.587+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 msgr2=0x7f80340c87d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.587+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f802c00e9c0 tx=0x7f802c00edc0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 shutdown_connections 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8010078bc0 0x7f801007b0d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8034159200 0x7f80340c87d0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80340705a0 0x7f80340c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 --2- 172.21.15.129:0/948938747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f803406eed0 0x7f80340c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 >> 172.21.15.129:0/948938747 conn(0x7f803406d410 msgr2=0x7f80340ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 shutdown_connections 2024-09-17T03:58:03.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.588+0000 7f803a600640 1 -- 172.21.15.129:0/948938747 wait complete. 2024-09-17T03:58:03.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir' 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 -- 172.21.15.129:0/3842407998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 msgr2=0x7fb2e00cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/3842407998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cb7c0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00cc7d0 tx=0x7fb2cc02a560 comp rx=0 tx=0).stop 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 -- 172.21.15.129:0/3842407998 shutdown_connections 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/3842407998 >> v2:172.21.15.181:3300/0 conn(0x7fb2e015c730 0x7fb2e0072c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/3842407998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cb7c0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/3842407998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e00cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 -- 172.21.15.129:0/3842407998 >> 172.21.15.129:0/3842407998 conn(0x7fb2e006ee70 msgr2=0x7fb2e006f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:03.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.832+0000 7fb2dfe00640 1 -- 172.21.15.129:0/3842407998 shutdown_connections 2024-09-17T03:58:03.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.833+0000 7fb2dfe00640 1 -- 172.21.15.129:0/3842407998 wait complete. 2024-09-17T03:58:03.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.835+0000 7fb2dfe00640 1 Processor -- start 2024-09-17T03:58:03.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.835+0000 7fb2dfe00640 1 -- start start 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e01e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00cf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e00b3450 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e00b32d0 con 0x7fb2e015c730 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.836+0000 7fb2dfe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e00b35d0 con 0x7fb2e00b5fa0 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52086/0 (socket says 172.21.15.129:52086) 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 -- 172.21.15.129:0/2719355803 learned_addr learned my addr 172.21.15.129:0/2719355803 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2dea00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e01e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2df400640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00cf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 -- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 msgr2=0x7fb2e01e8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e01e8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 -- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 msgr2=0x7fb2e00cf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00cf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 -- 172.21.15.129:0/2719355803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e01f5c20 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.837+0000 7fb2de000640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0099c0 tx=0x7fb2cc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.838+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc03e030 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.838+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2cc003ca0 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.838+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc035040 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.838+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e01f5f00 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.838+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e01f64c0 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.839+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e00b3a60 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.841+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb2cc041030 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.842+0000 7fb2d6a00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 0x7fb2c007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:03.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.842+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7fb2cc0baf00 con 0x7fb2e00cc3c0 2024-09-17T03:58:03.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.843+0000 7fb2dea00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 0x7fb2c007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:03.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.844+0000 7fb2dea00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 0x7fb2c007b0d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009ab0 tx=0x7fb2d0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:03.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:03.847+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb2cc0830f0 con 0x7fb2e00cc3c0 2024-09-17T03:58:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.097+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]} v 0) -- 0x7fb2e006fa20 con 0x7fb2e00cc3c0 2024-09-17T03:58:04.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.117+0000 7fb2d6a00640 1 -- 172.21.15.129:0/2719355803 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migrate_unlinked_dir"]}]=0 v446) ==== 140+0+0 (secure 0 0 0) 0x7fb2cc087b20 con 0x7fb2e00cc3c0 2024-09-17T03:58:04.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 msgr2=0x7fb2c007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 0x7fb2c007b0d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009ab0 tx=0x7fb2d0005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:04.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 msgr2=0x7fb2e00cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0099c0 tx=0x7fb2cc0038a0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 shutdown_connections 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2c0078bc0 0x7fb2c007b0d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00cf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cf380 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/2719355803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e01e8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 >> 172.21.15.129:0/2719355803 conn(0x7fb2e006ee70 msgr2=0x7fb2e00b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 shutdown_connections 2024-09-17T03:58:04.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.120+0000 7fb2dfe00640 1 -- 172.21.15.129:0/2719355803 wait complete. 2024-09-17T03:58:04.146 INFO:tasks.cephfs_test_runner:Reproduce https://tracker.ceph.com/issues/53597 ... ok 2024-09-17T03:58:04.147 INFO:tasks.cephfs_test_runner:Starting test: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, any non-purgeable strays 2024-09-17T03:58:04.147 INFO:tasks.cephfs_test_runner:test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:58:04.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 -- 172.21.15.129:0/24953912 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 msgr2=0x7f36e00ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 --2- 172.21.15.129:0/24953912 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e00ccc80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f36c8005fd0 tx=0x7f36c802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 -- 172.21.15.129:0/24953912 shutdown_connections 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 --2- 172.21.15.129:0/24953912 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e00ccc80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 --2- 172.21.15.129:0/24953912 >> v2:172.21.15.181:3300/0 conn(0x7f36e00705a0 0x7f36e00b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.368+0000 7f36e6400640 1 -- 172.21.15.129:0/24953912 >> 172.21.15.129:0/24953912 conn(0x7f36e006d410 msgr2=0x7f36e00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:04.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.369+0000 7f36e6400640 1 -- 172.21.15.129:0/24953912 shutdown_connections 2024-09-17T03:58:04.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.369+0000 7f36e6400640 1 -- 172.21.15.129:0/24953912 wait complete. 2024-09-17T03:58:04.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.371+0000 7f36e6400640 1 Processor -- start 2024-09-17T03:58:04.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.371+0000 7f36e6400640 1 -- start start 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e00705a0 0x7f36e00c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36e0160200 con 0x7f36e006f2e0 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36e01612e0 con 0x7f36e00705a0 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.372+0000 7f36e6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36e00cd1c0 con 0x7f36e006eed0 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52092/0 (socket says 172.21.15.129:52092) 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36df400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 -- 172.21.15.129:0/1207450501 learned_addr learned my addr 172.21.15.129:0/1207450501 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36dfe00640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e00705a0 0x7f36e00c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 -- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 msgr2=0x7f36e015c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e015c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 -- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e00705a0 msgr2=0x7f36e00c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e00705a0 0x7f36e00c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 -- 172.21.15.129:0/1207450501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e0157460 con 0x7f36e006f2e0 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.373+0000 7f36d6a00640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f36c8034040 tx=0x7f36c80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.374+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c803e030 con 0x7f36e006f2e0 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.374+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36c8003ae0 con 0x7f36e006f2e0 2024-09-17T03:58:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.374+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e015c960 con 0x7f36e006f2e0 2024-09-17T03:58:04.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.374+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36c8035040 con 0x7f36e006f2e0 2024-09-17T03:58:04.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.374+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e00c9e10 con 0x7f36e006f2e0 2024-09-17T03:58:04.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.375+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e00ccc80 con 0x7f36e006f2e0 2024-09-17T03:58:04.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.376+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f36c8041030 con 0x7f36e006f2e0 2024-09-17T03:58:04.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.377+0000 7f36d7e00640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 0x7f36bc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.377+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f36c80bb330 con 0x7f36e006f2e0 2024-09-17T03:58:04.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.378+0000 7f36df400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 0x7f36bc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.378+0000 7f36df400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 0x7f36bc07b0d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f36d00095f0 tx=0x7f36d0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:04.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.382+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f36c8083380 con 0x7f36e006f2e0 2024-09-17T03:58:04.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.568+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36e006a140 con 0x7f36e006f2e0 2024-09-17T03:58:04.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.569+0000 7f36d7e00640 1 -- 172.21.15.129:0/1207450501 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v132) ==== 74+0+13086 (secure 0 0 0) 0x7f36c8087db0 con 0x7f36e006f2e0 2024-09-17T03:58:04.569 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:04.569 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":132,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:03.110485+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/1495946":"2024-09-18T03:58:00.033355+0000","172.21.15.129:6832/1495946":"2024-09-18T03:58:00.033355+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:58:04.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 msgr2=0x7f36bc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 0x7f36bc07b0d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f36d00095f0 tx=0x7f36d0005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 msgr2=0x7f36e0156a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f36c8034040 tx=0x7f36c80035f0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 shutdown_connections 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36bc078bc0 0x7f36bc07b0d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36e00705a0 0x7f36e00c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36e006f2e0 0x7f36e0156a40 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 --2- 172.21.15.129:0/1207450501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36e006eed0 0x7f36e015c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 >> 172.21.15.129:0/1207450501 conn(0x7f36e006d410 msgr2=0x7f36e00c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 shutdown_connections 2024-09-17T03:58:04.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.571+0000 7f36e6400640 1 -- 172.21.15.129:0/1207450501 wait complete. 2024-09-17T03:58:04.593 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:58:04.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 -- 172.21.15.129:0/1079672887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 msgr2=0x7f80940b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 --2- 172.21.15.129:0/1079672887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f80940b49a0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f8080005fd0 tx=0x7f808002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 -- 172.21.15.129:0/1079672887 shutdown_connections 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 --2- 172.21.15.129:0/1079672887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f80940b49a0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 -- 172.21.15.129:0/1079672887 >> 172.21.15.129:0/1079672887 conn(0x7f80940ccc40 msgr2=0x7f80940cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 -- 172.21.15.129:0/1079672887 shutdown_connections 2024-09-17T03:58:04.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.816+0000 7f809c000640 1 -- 172.21.15.129:0/1079672887 wait complete. 2024-09-17T03:58:04.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.818+0000 7f809c000640 1 Processor -- start 2024-09-17T03:58:04.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.819+0000 7f809c000640 1 -- start start 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80940cba90 0x7f80940c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 0x7f80940ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8094167f30 con 0x7f8094074580 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f80940cc9b0 con 0x7f80940cba90 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f809c000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f80941680b0 con 0x7f80940ca2d0 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f8099600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f8099600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52118/0 (socket says 172.21.15.129:52118) 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.820+0000 7f8099600640 1 -- 172.21.15.129:0/2438800738 learned_addr learned my addr 172.21.15.129:0/2438800738 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8098c00640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80940cba90 0x7f80940c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f809a000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 0x7f80940ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 -- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 msgr2=0x7f80940ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 0x7f80940ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 -- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80940cba90 msgr2=0x7f80940c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80940cba90 0x7f80940c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 -- 172.21.15.129:0/2438800738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809409b2f0 con 0x7f8094074580 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f809a000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 0x7f80940ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:04.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8099600640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f808800c430 tx=0x7f808800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088017070 con 0x7f8094074580 2024-09-17T03:58:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80880053c0 con 0x7f8094074580 2024-09-17T03:58:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088003850 con 0x7f8094074580 2024-09-17T03:58:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.821+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f809409ccc0 con 0x7f8094074580 2024-09-17T03:58:04.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.822+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f809409b5d0 con 0x7f8094074580 2024-09-17T03:58:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.823+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f809409cab0 con 0x7f8094074580 2024-09-17T03:58:04.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.824+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f808801a070 con 0x7f8094074580 2024-09-17T03:58:04.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.825+0000 7f8091600640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 0x7f807407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:04.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.825+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f8088099d30 con 0x7f8094074580 2024-09-17T03:58:04.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.826+0000 7f8098c00640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 0x7f807407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:04.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.826+0000 7f8098c00640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 0x7f807407b0a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8080033000 tx=0x7f8080004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:04.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:04.830+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8088061e10 con 0x7f8094074580 2024-09-17T03:58:05.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.011+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f80940b3f70 con 0x7f8094074580 2024-09-17T03:58:05.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.012+0000 7f8091600640 1 -- 172.21.15.129:0/2438800738 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v132) ==== 93+0+10 (secure 0 0 0) 0x7f8088066840 con 0x7f8094074580 2024-09-17T03:58:05.012 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 msgr2=0x7f807407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 0x7f807407b0a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f8080033000 tx=0x7f8080004730 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 msgr2=0x7f809409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f808800c430 tx=0x7f808800c8f0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 shutdown_connections 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8074078b90 0x7f807407b0a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f80940ca2d0 0x7f80940ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80940cba90 0x7f80940c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 --2- 172.21.15.129:0/2438800738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8094074580 0x7f809409aa60 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.014+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 >> 172.21.15.129:0/2438800738 conn(0x7f80940ccc40 msgr2=0x7f80940b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:05.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.015+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 shutdown_connections 2024-09-17T03:58:05.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.015+0000 7f809c000640 1 -- 172.21.15.129:0/2438800738 wait complete. 2024-09-17T03:58:05.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-09-17T03:58:05.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 -- 172.21.15.129:0/2475733046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380b46e0 msgr2=0x7fc3380b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 --2- 172.21.15.129:0/2475733046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380b46e0 0x7fc3380b4af0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc328005fd0 tx=0x7fc32802a540 comp rx=0 tx=0).stop 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 -- 172.21.15.129:0/2475733046 shutdown_connections 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 --2- 172.21.15.129:0/2475733046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380b46e0 0x7fc3380b4af0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 --2- 172.21.15.129:0/2475733046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380cfb60 0x7fc338160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 -- 172.21.15.129:0/2475733046 >> 172.21.15.129:0/2475733046 conn(0x7fc33806cfc0 msgr2=0x7fc33806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 -- 172.21.15.129:0/2475733046 shutdown_connections 2024-09-17T03:58:05.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.259+0000 7fc33f200640 1 -- 172.21.15.129:0/2475733046 wait complete. 2024-09-17T03:58:05.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.262+0000 7fc33f200640 1 Processor -- start 2024-09-17T03:58:05.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.262+0000 7fc33f200640 1 -- start start 2024-09-17T03:58:05.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 0x7fc3381429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 0x7fc3381f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc33815d270 con 0x7fc3380d02c0 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc33815d0f0 con 0x7fc3380cfb60 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc33815d3f0 con 0x7fc3380b46e0 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52126/0 (socket says 172.21.15.129:52126) 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.263+0000 7fc33e800640 1 -- 172.21.15.129:0/1293824038 learned_addr learned my addr 172.21.15.129:0/1293824038 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33de00640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 0x7fc3381429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33d400640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 0x7fc3381f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 -- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 msgr2=0x7fc3381429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:05.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 0x7fc3381429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 -- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 msgr2=0x7fc3381f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 0x7fc3381f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 -- 172.21.15.129:0/1293824038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3380c5b50 con 0x7fc3380d02c0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33d400640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 0x7fc3381f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33de00640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 0x7fc3381429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33e800640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c760 tx=0x7fc33400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334003b70 con 0x7fc3380d02c0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc334003d10 con 0x7fc3380d02c0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3380c5e30 con 0x7fc3380d02c0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.264+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334005330 con 0x7fc3380d02c0 2024-09-17T03:58:05.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.265+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3381f7230 con 0x7fc3380d02c0 2024-09-17T03:58:05.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.266+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33815d8a0 con 0x7fc3380d02c0 2024-09-17T03:58:05.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.267+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc3340054d0 con 0x7fc3380d02c0 2024-09-17T03:58:05.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.268+0000 7fc332000640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 0x7fc31c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:05.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.268+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7fc33409a840 con 0x7fc3380d02c0 2024-09-17T03:58:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.269+0000 7fc33de00640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 0x7fc31c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:05.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.269+0000 7fc33de00640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 0x7fc31c07b0d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc32c005fd0 tx=0x7fc32c00f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:05.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.273+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc334062ee0 con 0x7fc3380d02c0 2024-09-17T03:58:05.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:05.523+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) -- 0x7fc33806a140 con 0x7fc3380d02c0 2024-09-17T03:58:06.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.125+0000 7fc332000640 1 -- 172.21.15.129:0/1293824038 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v448) ==== 144+0+0 (secure 0 0 0) 0x7fc3340627e0 con 0x7fc3380d02c0 2024-09-17T03:58:06.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 msgr2=0x7fc31c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 0x7fc31c07b0d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc32c005fd0 tx=0x7fc32c00f360 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 msgr2=0x7fc3381e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c760 tx=0x7fc33400cb60 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 shutdown_connections 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc31c078bc0 0x7fc31c07b0d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3380d02c0 0x7fc3381e8960 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3380cfb60 0x7fc3381f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 --2- 172.21.15.129:0/1293824038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3380b46e0 0x7fc3381429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 >> 172.21.15.129:0/1293824038 conn(0x7fc33806cfc0 msgr2=0x7fc3380cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:06.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 shutdown_connections 2024-09-17T03:58:06.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.128+0000 7fc33f200640 1 -- 172.21.15.129:0/1293824038 wait complete. 2024-09-17T03:58:06.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:58:06.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 -- 172.21.15.129:0/3512029117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ddc003a50 msgr2=0x7f4ddc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 --2- 172.21.15.129:0/3512029117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ddc003a50 0x7f4ddc003ec0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c4a0 tx=0x7f4ddc005830 comp rx=0 tx=0).stop 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 -- 172.21.15.129:0/3512029117 shutdown_connections 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 --2- 172.21.15.129:0/3512029117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ddc003a50 0x7f4ddc003ec0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 -- 172.21.15.129:0/3512029117 >> 172.21.15.129:0/3512029117 conn(0x7f4df806d410 msgr2=0x7f4df80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 -- 172.21.15.129:0/3512029117 shutdown_connections 2024-09-17T03:58:06.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.378+0000 7f4dff000640 1 -- 172.21.15.129:0/3512029117 wait complete. 2024-09-17T03:58:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.380+0000 7f4dff000640 1 Processor -- start 2024-09-17T03:58:06.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.380+0000 7f4dff000640 1 -- start start 2024-09-17T03:58:06.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 0x7f4df81f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 0x7f4df80b0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4df8160260 con 0x7f4df81eba40 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4df81600e0 con 0x7f4df80c7320 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dff000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4df80ccc80 con 0x7f4ddc003a50 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52144/0 (socket says 172.21.15.129:52144) 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 -- 172.21.15.129:0/2571535452 learned_addr learned my addr 172.21.15.129:0/2571535452 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7e00640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 0x7f4df81f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4dfd000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 0x7f4df80b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 -- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 msgr2=0x7f4df81f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 0x7f4df81f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 -- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 msgr2=0x7f4df80b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.382+0000 7f4df7400640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 0x7f4df80b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4df7400640 1 -- 172.21.15.129:0/2571535452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df80c9280 con 0x7f4df81eba40 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4df7e00640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 0x7f4df81f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4dfd000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 0x7f4df80b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4df7400640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f4de4011a10 tx=0x7f4de4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4004460 con 0x7f4df81eba40 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4de400f9a0 con 0x7f4df81eba40 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de400fdf0 con 0x7f4df81eba40 2024-09-17T03:58:06.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df80c9560 con 0x7f4df81eba40 2024-09-17T03:58:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.383+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df80c9b20 con 0x7f4df81eba40 2024-09-17T03:58:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.384+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df806a140 con 0x7f4df81eba40 2024-09-17T03:58:06.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.386+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4de4005020 con 0x7f4df81eba40 2024-09-17T03:58:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.386+0000 7f4defe00640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 0x7f4dd807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.387+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f4de409b4e0 con 0x7f4df81eba40 2024-09-17T03:58:06.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.387+0000 7f4df7e00640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 0x7f4dd807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.388+0000 7f4df7e00640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 0x7f4dd807b0d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4de8033000 tx=0x7f4de8001c30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:06.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.391+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4de4063d40 con 0x7f4df81eba40 2024-09-17T03:58:06.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.575+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f4df80af7f0 con 0x7f4df81eba40 2024-09-17T03:58:06.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.575+0000 7f4defe00640 1 -- 172.21.15.129:0/2571535452 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v51) ==== 59+0+2294 (secure 0 0 0) 0x7f4de4063640 con 0x7f4df81eba40 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:58:06.576 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:58:06.577 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:58:06.578 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:58:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 msgr2=0x7f4dd807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 0x7f4dd807b0d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4de8033000 tx=0x7f4de8001c30 comp rx=0 tx=0).stop 2024-09-17T03:58:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 msgr2=0x7f4df80ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f4de4011a10 tx=0x7f4de4011ed0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 shutdown_connections 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dd8078bc0 0x7f4dd807b0d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4df80c7320 0x7f4df80b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4df81eba40 0x7f4df80ca040 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 --2- 172.21.15.129:0/2571535452 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ddc003a50 0x7f4df81f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 >> 172.21.15.129:0/2571535452 conn(0x7f4df806d410 msgr2=0x7f4df81e81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 shutdown_connections 2024-09-17T03:58:06.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.578+0000 7f4dff000640 1 -- 172.21.15.129:0/2571535452 wait complete. 2024-09-17T03:58:06.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e31400640 1 -- 172.21.15.129:0/339250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e100039e0 msgr2=0x7f9e10003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e31400640 1 --2- 172.21.15.129:0/339250270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e100039e0 0x7f9e10003e10 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009b50 tx=0x7f9e10005830 comp rx=0 tx=0).stop 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e2b200640 1 -- 172.21.15.129:0/339250270 reap_dead start 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e31400640 1 -- 172.21.15.129:0/339250270 shutdown_connections 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e31400640 1 -- 172.21.15.129:0/339250270 >> 172.21.15.129:0/339250270 conn(0x7f9e2c0ccc40 msgr2=0x7f9e2c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.824+0000 7f9e31400640 1 -- 172.21.15.129:0/339250270 shutdown_connections 2024-09-17T03:58:06.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.825+0000 7f9e31400640 1 -- 172.21.15.129:0/339250270 wait complete. 2024-09-17T03:58:06.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.827+0000 7f9e31400640 1 Processor -- start 2024-09-17T03:58:06.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.827+0000 7f9e31400640 1 -- start start 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e100039e0 0x7f9e2c15c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e2c0cea10 0x7f9e2c15e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c0af810 con 0x7f9e2c0cef30 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c0af690 con 0x7f9e100039e0 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.828+0000 7f9e31400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2c0af990 con 0x7f9e2c0cea10 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52158/0 (socket says 172.21.15.129:52158) 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 -- 172.21.15.129:0/25874591 learned_addr learned my addr 172.21.15.129:0/25874591 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e2a800640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e100039e0 0x7f9e2c15c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e2b200640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e2c0cea10 0x7f9e2c15e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 -- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e2c0cea10 msgr2=0x7f9e2c15e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e2c0cea10 0x7f9e2c15e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 -- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e100039e0 msgr2=0x7f9e2c15c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e100039e0 0x7f9e2c15c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 -- 172.21.15.129:0/25874591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c0cf9f0 con 0x7f9e2c0cef30 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.829+0000 7f9e29e00640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f9e18004de0 tx=0x7f9e18001790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.830+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18010890 con 0x7f9e2c0cef30 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.830+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e18016070 con 0x7f9e2c0cef30 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.830+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18010cf0 con 0x7f9e2c0cef30 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.830+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2c15bd20 con 0x7f9e2c0cef30 2024-09-17T03:58:06.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.830+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e2c15c2e0 con 0x7f9e2c0cef30 2024-09-17T03:58:06.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.831+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e2c0b3420 con 0x7f9e2c0cef30 2024-09-17T03:58:06.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.832+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e1801b460 con 0x7f9e2c0cef30 2024-09-17T03:58:06.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.833+0000 7f9e22a00640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:06.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.833+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f9e1809b2d0 con 0x7f9e2c0cef30 2024-09-17T03:58:06.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.834+0000 7f9e2a800640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:06.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.834+0000 7f9e2a800640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c035000 tx=0x7f9e1c02d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:06.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:06.837+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e18063360 con 0x7f9e2c0cef30 2024-09-17T03:58:07.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.022+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f9e2c0cf780 con 0x7f9e2c0cef30 2024-09-17T03:58:07.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.027+0000 7f9e22a00640 1 -- 172.21.15.129:0/25874591 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) ==== 122+0+0 (secure 0 0 0) 0x7f9e18067d90 con 0x7f9e2c0cef30 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 msgr2=0x7f9e0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c035000 tx=0x7f9e1c02d040 comp rx=0 tx=0).stop 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 msgr2=0x7f9e2c0ce4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f9e18004de0 tx=0x7f9e18001790 comp rx=0 tx=0).stop 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 shutdown_connections 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e2c0cea10 0x7f9e2c15e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e2c0cef30 0x7f9e2c0ce4d0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 --2- 172.21.15.129:0/25874591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e100039e0 0x7f9e2c15c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 >> 172.21.15.129:0/25874591 conn(0x7f9e2c0ccc40 msgr2=0x7f9e2c0affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:07.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 shutdown_connections 2024-09-17T03:58:07.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.030+0000 7f9e31400640 1 -- 172.21.15.129:0/25874591 wait complete. 2024-09-17T03:58:07.051 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:58:07.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:58:07.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:58:07.221 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:58:07.221 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:07.221 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:07.257 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:58:07.257 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:58:07.257 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:07.294 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:58:07.294 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:07.294 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:07.331 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:58:07.332 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:58:07.332 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:07.370 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:58:07.370 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:58:07.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 -- 172.21.15.129:0/110189753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980b4140 msgr2=0x7f6f980ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 --2- 172.21.15.129:0/110189753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980b4140 0x7f6f980ccc80 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f6f80005fd0 tx=0x7f6f8002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 -- 172.21.15.129:0/110189753 shutdown_connections 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 --2- 172.21.15.129:0/110189753 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980cd1c0 0x7f6f981609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 --2- 172.21.15.129:0/110189753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980b4140 0x7f6f980ccc80 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 -- 172.21.15.129:0/110189753 >> 172.21.15.129:0/110189753 conn(0x7f6f9806d410 msgr2=0x7f6f980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 -- 172.21.15.129:0/110189753 shutdown_connections 2024-09-17T03:58:07.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.591+0000 7f6f9d400640 1 -- 172.21.15.129:0/110189753 wait complete. 2024-09-17T03:58:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.594+0000 7f6f9d400640 1 Processor -- start 2024-09-17T03:58:07.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.594+0000 7f6f9d400640 1 -- start start 2024-09-17T03:58:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 0x7f6f98159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:07.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f980b4140 0x7f6f980cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6f98160260 con 0x7f6f980cd1c0 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6f981600e0 con 0x7f6f980b4140 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f9d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6f98160f00 con 0x7f6f980705a0 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f97200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f97200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52178/0 (socket says 172.21.15.129:52178) 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.595+0000 7f6f97200640 1 -- 172.21.15.129:0/4284846083 learned_addr learned my addr 172.21.15.129:0/4284846083 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f8fe00640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f980b4140 0x7f6f980cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f96800640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 0x7f6f98159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 -- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 msgr2=0x7f6f98159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 0x7f6f98159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 -- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f980b4140 msgr2=0x7f6f980cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f980b4140 0x7f6f980cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 -- 172.21.15.129:0/4284846083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f980cc8a0 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f96800640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 0x7f6f98159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f97200640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000ea10 tx=0x7f6f9000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.596+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9000cde0 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.597+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6f90004c10 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.597+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f90005020 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.597+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f9815c300 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.597+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f981fa6c0 con 0x7f6f980cd1c0 2024-09-17T03:58:07.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.598+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f98160c10 con 0x7f6f980cd1c0 2024-09-17T03:58:07.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.599+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6f900051c0 con 0x7f6f980cd1c0 2024-09-17T03:58:07.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.600+0000 7f6f8ea00640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 0x7f6f7407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:07.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.601+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(132..132 src has 1..132) ==== 4943+0+0 (secure 0 0 0) 0x7f6f9009b7a0 con 0x7f6f980cd1c0 2024-09-17T03:58:07.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.601+0000 7f6f96800640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 0x7f6f7407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:07.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.601+0000 7f6f96800640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 0x7f6f7407b0d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007fd0 tx=0x7f6f88009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:07.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.605+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6f98160c10 con 0x7f6f980cd1c0 2024-09-17T03:58:07.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.823+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f980ccc80 con 0x7f6f980cd1c0 2024-09-17T03:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.824+0000 7f6f8ea00640 1 -- 172.21.15.129:0/4284846083 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+2270 (secure 0 0 0) 0x7f6f90063e40 con 0x7f6f980cd1c0 2024-09-17T03:58:07.824 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:07.824 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":109,"btime":"2024-09-17T03:58:01:096802+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7989,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2977701965","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2977701965},{"type":"v1","addr":"172.21.15.129:6835","nonce":2977701965}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107},{"gid":8088,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3917813771","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3917813771},{"type":"v1","addr":"172.21.15.129:6834","nonce":3917813771}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":109}],"filesystems":[]} 2024-09-17T03:58:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 109 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 msgr2=0x7f6f7407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 0x7f6f7407b0d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6f88007fd0 tx=0x7f6f88009040 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 msgr2=0x7f6f981f9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000ea10 tx=0x7f6f9000eed0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 shutdown_connections 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6f74078bc0 0x7f6f7407b0d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6f980cd1c0 0x7f6f981f9820 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f980b4140 0x7f6f980cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 --2- 172.21.15.129:0/4284846083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6f980705a0 0x7f6f98159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 >> 172.21.15.129:0/4284846083 conn(0x7f6f9806d410 msgr2=0x7f6f980b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:07.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 shutdown_connections 2024-09-17T03:58:07.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:07.826+0000 7f6f9d400640 1 -- 172.21.15.129:0/4284846083 wait complete. 2024-09-17T03:58:07.849 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:58:07.849 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:58:07.849 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:58:07.849 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:58:07.850 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:58:07.850 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:58:07.850 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:58:07.850 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:58:07.850 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:58:07.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:58:07.851 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:58:07.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:58:07.855 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:58:07.855 DEBUG:teuthology.parallel:result is None 2024-09-17T03:58:07.857 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:58:07.857 DEBUG:teuthology.parallel:result is None 2024-09-17T03:58:07.873 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:58:07.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:58:07.875 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:58:07.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:58:07.920 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:58:07.921 DEBUG:teuthology.parallel:result is None 2024-09-17T03:58:07.924 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:58:07.924 DEBUG:teuthology.parallel:result is None 2024-09-17T03:58:07.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:58:08.056 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:58:08.056 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:58:08.119 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:58:08.119 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:58:08.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 -- 172.21.15.129:0/4025606381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 msgr2=0x7f60340cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 --2- 172.21.15.129:0/4025606381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60340cd090 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f602c005fd0 tx=0x7f602c02aa10 comp rx=0 tx=0).stop 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 -- 172.21.15.129:0/4025606381 shutdown_connections 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 --2- 172.21.15.129:0/4025606381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60340cd090 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 --2- 172.21.15.129:0/4025606381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60340cf2b0 0x7f60340cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 -- 172.21.15.129:0/4025606381 >> 172.21.15.129:0/4025606381 conn(0x7f603406d410 msgr2=0x7f60340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 -- 172.21.15.129:0/4025606381 shutdown_connections 2024-09-17T03:58:08.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.161+0000 7f603c000640 1 -- 172.21.15.129:0/4025606381 wait complete. 2024-09-17T03:58:08.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.164+0000 7f603c000640 1 Processor -- start 2024-09-17T03:58:08.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.164+0000 7f603c000640 1 -- start start 2024-09-17T03:58:08.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 0x7f603415c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60341f9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60340cf2b0 0x7f60341f9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6034160ba0 con 0x7f60340ccc80 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6034160a20 con 0x7f60340cf2b0 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f603c000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6034160d20 con 0x7f6034099570 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.165+0000 7f6038c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60341f9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6038c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60341f9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52240/0 (socket says 172.21.15.129:52240) 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6038c00640 1 -- 172.21.15.129:0/2746783460 learned_addr learned my addr 172.21.15.129:0/2746783460 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 0x7f603415c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f603a000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60340cf2b0 0x7f60341f9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 -- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60340cf2b0 msgr2=0x7f60341f9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60340cf2b0 0x7f60341f9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 -- 172.21.15.129:0/2746783460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 msgr2=0x7f60341f9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60341f9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 -- 172.21.15.129:0/2746783460 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603415f950 con 0x7f6034099570 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.166+0000 7f6039600640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 0x7f603415c360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f602800c3c0 tx=0x7f602800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.167+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6028017070 con 0x7f6034099570 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.167+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60280053e0 con 0x7f6034099570 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.167+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603415f0b0 con 0x7f6034099570 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.167+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6028003850 con 0x7f6034099570 2024-09-17T03:58:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.167+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f603415f390 con 0x7f6034099570 2024-09-17T03:58:08.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.168+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6034161240 con 0x7f6034099570 2024-09-17T03:58:08.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.169+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f602801a070 con 0x7f6034099570 2024-09-17T03:58:08.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.170+0000 7f6031600640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 0x7f601407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:08.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.170+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(134..134 src has 1..134) ==== 4943+0+0 (secure 0 0 0) 0x7f6028099cf0 con 0x7f6034099570 2024-09-17T03:58:08.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.171+0000 7f6038c00640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 0x7f601407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:08.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.171+0000 7f6038c00640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 0x7f601407b0d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6034074850 tx=0x7f6020005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:08.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.176+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6028061d50 con 0x7f6034099570 2024-09-17T03:58:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:08.369+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f603406a140 con 0x7f6034099570 2024-09-17T03:58:09.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.157+0000 7f6031600640 1 -- 172.21.15.129:0/2746783460 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) ==== 97+0+0 (secure 0 0 0) 0x7f6028066780 con 0x7f6034099570 2024-09-17T03:58:09.157 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 msgr2=0x7f601407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 0x7f601407b0d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6034074850 tx=0x7f6020005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 msgr2=0x7f603415c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 0x7f603415c360 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f602800c3c0 tx=0x7f602800c880 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 shutdown_connections 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6014078bc0 0x7f601407b0d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60340cf2b0 0x7f60341f9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60340ccc80 0x7f60341f9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 --2- 172.21.15.129:0/2746783460 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6034099570 0x7f603415c360 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 >> 172.21.15.129:0/2746783460 conn(0x7f603406d410 msgr2=0x7f603406eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 shutdown_connections 2024-09-17T03:58:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.160+0000 7f603c000640 1 -- 172.21.15.129:0/2746783460 wait complete. 2024-09-17T03:58:09.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:58:09.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 -- 172.21.15.129:0/3823627096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940d0800 msgr2=0x7f119415cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 --2- 172.21.15.129:0/3823627096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940d0800 0x7f119415cbd0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f118c005fd0 tx=0x7f118c02a820 comp rx=0 tx=0).stop 2024-09-17T03:58:09.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 -- 172.21.15.129:0/3823627096 shutdown_connections 2024-09-17T03:58:09.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 --2- 172.21.15.129:0/3823627096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940d0800 0x7f119415cbd0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 --2- 172.21.15.129:0/3823627096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1194160cf0 0x7f11940d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 -- 172.21.15.129:0/3823627096 >> 172.21.15.129:0/3823627096 conn(0x7f119406cfc0 msgr2=0x7f119406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:09.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 -- 172.21.15.129:0/3823627096 shutdown_connections 2024-09-17T03:58:09.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.403+0000 7f119ac00640 1 -- 172.21.15.129:0/3823627096 wait complete. 2024-09-17T03:58:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.405+0000 7f119ac00640 1 Processor -- start 2024-09-17T03:58:09.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.406+0000 7f119ac00640 1 -- start start 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 0x7f11940acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 0x7f119415b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f119415d4a0 con 0x7f11940cfb60 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f119415d320 con 0x7f1194160cf0 2024-09-17T03:58:09.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f119415d620 con 0x7f11940d0800 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52244/0 (socket says 172.21.15.129:52244) 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 -- 172.21.15.129:0/4230103714 learned_addr learned my addr 172.21.15.129:0/4230103714 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1198e00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 0x7f11940acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119a200640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 0x7f119415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 -- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 msgr2=0x7f11940acbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 0x7f11940acbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 -- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 msgr2=0x7f119415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 0x7f119415b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 -- 172.21.15.129:0/4230103714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11940ad100 con 0x7f11940cfb60 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1198e00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 0x7f11940acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119a200640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 0x7f119415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1199800640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f118800c3c0 tx=0x7f118800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188017070 con 0x7f11940cfb60 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f11880035f0 con 0x7f11940cfb60 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119415a470 con 0x7f11940cfb60 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1188003a20 con 0x7f11940cfb60 2024-09-17T03:58:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.408+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119415aa30 con 0x7f11940cfb60 2024-09-17T03:58:09.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.410+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119415cbd0 con 0x7f11940cfb60 2024-09-17T03:58:09.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.411+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f118801a070 con 0x7f11940cfb60 2024-09-17T03:58:09.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.412+0000 7f1191600640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 0x7f117407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.412+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4857+0+0 (secure 0 0 0) 0x7f1188099cd0 con 0x7f11940cfb60 2024-09-17T03:58:09.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.412+0000 7f1198e00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 0x7f117407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.413+0000 7f1198e00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 0x7f117407b0d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f11940b2a70 tx=0x7f1180005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:09.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.418+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11940d0c10 con 0x7f11940cfb60 2024-09-17T03:58:09.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.664+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f11940b4980 con 0x7f11940cfb60 2024-09-17T03:58:09.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.665+0000 7f1191600640 1 -- 172.21.15.129:0/4230103714 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v35) ==== 80+0+5304 (secure 0 0 0) 0x7f1188062450 con 0x7f11940cfb60 2024-09-17T03:58:09.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:58:09.667 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:58:09.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.669 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.671 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:58:09.675 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:58:09.678 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.679 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:09.680 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:58:09.680 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 msgr2=0x7f117407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 0x7f117407b0d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f11940b2a70 tx=0x7f1180005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 msgr2=0x7f1194144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f118800c3c0 tx=0x7f118800c880 comp rx=0 tx=0).stop 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 shutdown_connections 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1174078bc0 0x7f117407b0d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1194160cf0 0x7f119415b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11940d0800 0x7f11940acbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 --2- 172.21.15.129:0/4230103714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11940cfb60 0x7f1194144e80 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 >> 172.21.15.129:0/4230103714 conn(0x7f119406cfc0 msgr2=0x7f1194156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 shutdown_connections 2024-09-17T03:58:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.669+0000 7f119ac00640 1 -- 172.21.15.129:0/4230103714 wait complete. 2024-09-17T03:58:09.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:58:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 -- 172.21.15.129:0/395666867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a4003a50 msgr2=0x7f81a4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 --2- 172.21.15.129:0/395666867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a4003a50 0x7f81a4003ec0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f81b000c7a0 tx=0x7f81a4005830 comp rx=0 tx=0).stop 2024-09-17T03:58:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 -- 172.21.15.129:0/395666867 shutdown_connections 2024-09-17T03:58:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 --2- 172.21.15.129:0/395666867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81a4003a50 0x7f81a4003ec0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 -- 172.21.15.129:0/395666867 >> 172.21.15.129:0/395666867 conn(0x7f81c009d930 msgr2=0x7f81c008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 -- 172.21.15.129:0/395666867 shutdown_connections 2024-09-17T03:58:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.915+0000 7f81c6800640 1 -- 172.21.15.129:0/395666867 wait complete. 2024-09-17T03:58:09.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.917+0000 7f81c6800640 1 Processor -- start 2024-09-17T03:58:09.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.917+0000 7f81c6800640 1 -- start start 2024-09-17T03:58:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 0x7f81c0163890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81c00b8bd0 0x7f81c00b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81c00d1820 con 0x7f81c00a0490 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f81c00ba230 con 0x7f81a4003a50 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81c6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f81c00d19a0 con 0x7f81c00b8bd0 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81bfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52264/0 (socket says 172.21.15.129:52264) 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81bfe00640 1 -- 172.21.15.129:0/2013797470 learned_addr learned my addr 172.21.15.129:0/2013797470 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81b6a00640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81c00b8bd0 0x7f81c00b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.919+0000 7f81bf400640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 0x7f81c0163890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 -- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81c00b8bd0 msgr2=0x7f81c00b8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81c00b8bd0 0x7f81c00b8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 -- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 msgr2=0x7f81c0163890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 0x7f81c0163890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 -- 172.21.15.129:0/2013797470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c00b9520 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bf400640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 0x7f81c0163890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81bfe00640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f81c008e5a0 tx=0x7f81b8011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004440 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81b8016070 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c0166520 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b800fbe0 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.920+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c0166ae0 con 0x7f81c00a0490 2024-09-17T03:58:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.921+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c00cf5a0 con 0x7f81c00a0490 2024-09-17T03:58:09.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.923+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f81b80035f0 con 0x7f81c00a0490 2024-09-17T03:58:09.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.923+0000 7f81b7e00640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 0x7f819c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:09.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.924+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4857+0+0 (secure 0 0 0) 0x7f81b809b1d0 con 0x7f81c00a0490 2024-09-17T03:58:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.924+0000 7f81bf400640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 0x7f819c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:09.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.925+0000 7f81bf400640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 0x7f819c07b0a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f81b0033000 tx=0x7f81c00d2a70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:09.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:09.928+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f81c00b9520 con 0x7f81c00a0490 2024-09-17T03:58:10.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.112+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f81c01662b0 con 0x7f81c00a0490 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.113+0000 7f81b7e00640 1 -- 172.21.15.129:0/2013797470 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) ==== 84+0+65 (secure 0 0 0) 0x7f81b8063960 con 0x7f81c00a0490 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:58:10.112 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:58:10.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 msgr2=0x7f819c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:10.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 0x7f819c07b0a0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f81b0033000 tx=0x7f81c00d2a70 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 msgr2=0x7f81c00a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f81c008e5a0 tx=0x7f81b8011e10 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 shutdown_connections 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f819c078b90 0x7f819c07b0a0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81c00a0490 0x7f81c00a08a0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81c00b8bd0 0x7f81c00b8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 --2- 172.21.15.129:0/2013797470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81a4003a50 0x7f81c0163890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 >> 172.21.15.129:0/2013797470 conn(0x7f81c009d930 msgr2=0x7f81c006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 shutdown_connections 2024-09-17T03:58:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.115+0000 7f81c6800640 1 -- 172.21.15.129:0/2013797470 wait complete. 2024-09-17T03:58:10.141 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:58:10.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:58:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 -- 172.21.15.129:0/583913045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb644004800 msgr2=0x7fb644004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:10.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 --2- 172.21.15.129:0/583913045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb644004800 0x7fb644004c10 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb65c06ea40 tx=0x7fb644005830 comp rx=0 tx=0).stop 2024-09-17T03:58:10.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb662000640 1 -- 172.21.15.129:0/583913045 reap_dead start 2024-09-17T03:58:10.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 -- 172.21.15.129:0/583913045 shutdown_connections 2024-09-17T03:58:10.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 -- 172.21.15.129:0/583913045 >> 172.21.15.129:0/583913045 conn(0x7fb65c0ccc40 msgr2=0x7fb65c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:10.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 -- 172.21.15.129:0/583913045 shutdown_connections 2024-09-17T03:58:10.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.369+0000 7fb664000640 1 -- 172.21.15.129:0/583913045 wait complete. 2024-09-17T03:58:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.371+0000 7fb664000640 1 Processor -- start 2024-09-17T03:58:10.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.371+0000 7fb664000640 1 -- start start 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb644004800 0x7fb65c1f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb65c1e6f90 0x7fb65c160e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c0b1dc0 con 0x7fb65c1f90b0 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb65c0b1c40 con 0x7fb644004800 2024-09-17T03:58:10.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb664000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb65c0b1f40 con 0x7fb65c1e6f90 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52282/0 (socket says 172.21.15.129:52282) 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 -- 172.21.15.129:0/4128823854 learned_addr learned my addr 172.21.15.129:0/4128823854 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb661600640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb644004800 0x7fb65c1f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb662000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb65c1e6f90 0x7fb65c160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 -- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb65c1e6f90 msgr2=0x7fb65c160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb65c1e6f90 0x7fb65c160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 -- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb644004800 msgr2=0x7fb65c1f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb644004800 0x7fb65c1f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.373+0000 7fb660c00640 1 -- 172.21.15.129:0/4128823854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65c1606c0 con 0x7fb65c1f90b0 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb660c00640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fb648036040 tx=0x7fb648034680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648038040 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb648034870 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648034c80 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65c1f6380 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.374+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65c1f6940 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.375+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb65c0ca810 con 0x7fb65c1f90b0 2024-09-17T03:58:10.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.377+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb648045020 con 0x7fb65c1f90b0 2024-09-17T03:58:10.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.377+0000 7fb659600640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 0x7fb63c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:10.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.378+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(135..135 src has 1..135) ==== 4857+0+0 (secure 0 0 0) 0x7fb6480bb240 con 0x7fb65c1f90b0 2024-09-17T03:58:10.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.378+0000 7fb661600640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 0x7fb63c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:10.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.379+0000 7fb661600640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 0x7fb63c07b0d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb650000f60 tx=0x7fb650006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.382+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb64804c020 con 0x7fb65c1f90b0 2024-09-17T03:58:10.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:10.564+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb65c0cc9b0 con 0x7fb65c1f90b0 2024-09-17T03:58:11.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.157+0000 7fb659600640 1 -- 172.21.15.129:0/4128823854 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) ==== 138+0+0 (secure 0 0 0) 0x7fb648034e20 con 0x7fb65c1f90b0 2024-09-17T03:58:11.156 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 msgr2=0x7fb63c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 0x7fb63c07b0d0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb650000f60 tx=0x7fb650006040 comp rx=0 tx=0).stop 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 msgr2=0x7fb65c1f3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fb648036040 tx=0x7fb648034680 comp rx=0 tx=0).stop 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 shutdown_connections 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb63c078bc0 0x7fb63c07b0d0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb65c1e6f90 0x7fb65c160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb65c1f90b0 0x7fb65c1f3050 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 --2- 172.21.15.129:0/4128823854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb644004800 0x7fb65c1f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 >> 172.21.15.129:0/4128823854 conn(0x7fb65c0ccc40 msgr2=0x7fb65c0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:11.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 shutdown_connections 2024-09-17T03:58:11.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.160+0000 7fb664000640 1 -- 172.21.15.129:0/4128823854 wait complete. 2024-09-17T03:58:11.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.419+0000 7fafcd200640 1 -- 172.21.15.129:0/1239089584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 msgr2=0x7fafc8160d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.419+0000 7fafcd200640 1 --2- 172.21.15.129:0/1239089584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8160d50 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7fafc80d0010 tx=0x7fafb802a4a0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 -- 172.21.15.129:0/1239089584 shutdown_connections 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 --2- 172.21.15.129:0/1239089584 >> v2:172.21.15.181:3300/0 conn(0x7fafc806ee70 0x7fafc806f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 --2- 172.21.15.129:0/1239089584 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8160d50 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 -- 172.21.15.129:0/1239089584 >> 172.21.15.129:0/1239089584 conn(0x7fafc80d0120 msgr2=0x7fafc80d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:11.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 -- 172.21.15.129:0/1239089584 shutdown_connections 2024-09-17T03:58:11.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.420+0000 7fafcd200640 1 -- 172.21.15.129:0/1239089584 wait complete. 2024-09-17T03:58:11.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.422+0000 7fafcd200640 1 Processor -- start 2024-09-17T03:58:11.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.422+0000 7fafcd200640 1 -- start start 2024-09-17T03:58:11.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc806ee70 0x7fafc80c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafc8072ee0 0x7fafc8155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafc80cf450 con 0x7fafc8156f20 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafc80cf2d0 con 0x7fafc806ee70 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.423+0000 7fafcd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fafc80cf5d0 con 0x7fafc8072ee0 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52306/0 (socket says 172.21.15.129:52306) 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 -- 172.21.15.129:0/2223013118 learned_addr learned my addr 172.21.15.129:0/2223013118 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:11.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7400640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc806ee70 0x7fafc80c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:11.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc6a00640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafc8072ee0 0x7fafc8155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 -- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafc8072ee0 msgr2=0x7fafc8155f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafc8072ee0 0x7fafc8155f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 -- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc806ee70 msgr2=0x7fafc80c6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc806ee70 0x7fafc80c6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 -- 172.21.15.129:0/2223013118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc81f3e20 con 0x7fafc8156f20 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.424+0000 7fafc7e00640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fafc000cb10 tx=0x7fafc000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.425+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0016020 con 0x7fafc8156f20 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.425+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc000ad60 con 0x7fafc8156f20 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.425+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc80c5fd0 con 0x7fafc8156f20 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.425+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0012680 con 0x7fafc8156f20 2024-09-17T03:58:11.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.425+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc80c6590 con 0x7fafc8156f20 2024-09-17T03:58:11.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.426+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc8073bc0 con 0x7fafc8156f20 2024-09-17T03:58:11.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.427+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fafc0019020 con 0x7fafc8156f20 2024-09-17T03:58:11.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.428+0000 7fafbf400640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 0x7fafa807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:11.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.428+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(136..136 src has 1..136) ==== 5227+0+0 (secure 0 0 0) 0x7fafc009b090 con 0x7fafc8156f20 2024-09-17T03:58:11.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.429+0000 7fafc7400640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 0x7fafa807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:11.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.429+0000 7fafc7400640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 0x7fafa807b0a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fafb8034000 tx=0x7fafb8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:11.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.433+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fafc0020020 con 0x7fafc8156f20 2024-09-17T03:58:11.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:11.615+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fafc8074000 con 0x7fafc8156f20 2024-09-17T03:58:12.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.160+0000 7fafbf400640 1 -- 172.21.15.129:0/2223013118 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) ==== 170+0+0 (secure 0 0 0) 0x7fafc806f7c0 con 0x7fafc8156f20 2024-09-17T03:58:12.159 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.163+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 msgr2=0x7fafa807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.163+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 0x7fafa807b0a0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fafb8034000 tx=0x7fafb8004730 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.163+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 msgr2=0x7fafc8157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.163+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fafc000cb10 tx=0x7fafc000a060 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 shutdown_connections 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafa8078b90 0x7fafa807b0a0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafc8156f20 0x7fafc8157480 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafc8072ee0 0x7fafc8155f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 --2- 172.21.15.129:0/2223013118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafc806ee70 0x7fafc80c6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 >> 172.21.15.129:0/2223013118 conn(0x7fafc80d0120 msgr2=0x7fafc815f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:12.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 shutdown_connections 2024-09-17T03:58:12.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.164+0000 7fafcd200640 1 -- 172.21.15.129:0/2223013118 wait complete. 2024-09-17T03:58:12.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:58:12.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 -- 172.21.15.129:0/2808309101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa640cb080 msgr2=0x7faa64072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:12.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 --2- 172.21.15.129:0/2808309101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa640cb080 0x7faa64072e10 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7faa50009830 tx=0x7faa5002a510 comp rx=0 tx=0).stop 2024-09-17T03:58:12.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 -- 172.21.15.129:0/2808309101 shutdown_connections 2024-09-17T03:58:12.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 --2- 172.21.15.129:0/2808309101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa64073350 0x7faa640ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 --2- 172.21.15.129:0/2808309101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa640cb080 0x7faa64072e10 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 -- 172.21.15.129:0/2808309101 >> 172.21.15.129:0/2808309101 conn(0x7faa640d00b0 msgr2=0x7faa640d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 -- 172.21.15.129:0/2808309101 shutdown_connections 2024-09-17T03:58:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.425+0000 7faa69c00640 1 -- 172.21.15.129:0/2808309101 wait complete. 2024-09-17T03:58:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.427+0000 7faa69c00640 1 Processor -- start 2024-09-17T03:58:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.427+0000 7faa69c00640 1 -- start start 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.428+0000 7faa69c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.428+0000 7faa69c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 0x7faa64158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.428+0000 7faa69c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 0x7faa640c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.428+0000 7faa69c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa640ced50 con 0x7faa64073350 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa69c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa640cebd0 con 0x7faa640cb080 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa69c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faa640ceed0 con 0x7faa6415c7b0 2024-09-17T03:58:12.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52320/0 (socket says 172.21.15.129:52320) 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 -- 172.21.15.129:0/1755639064 learned_addr learned my addr 172.21.15.129:0/1755639064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 -- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 msgr2=0x7faa640c99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa69200640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 0x7faa640c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 0x7faa640c99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 -- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 msgr2=0x7faa64158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63400640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 0x7faa64158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 0x7faa64158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63e00640 1 -- 172.21.15.129:0/1755639064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa640c8f10 con 0x7faa64073350 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa69200640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 0x7faa640c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.429+0000 7faa63400640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 0x7faa64158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa63e00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7faa5400c3f0 tx=0x7faa5400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:12.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54017070 con 0x7faa64073350 2024-09-17T03:58:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa540035f0 con 0x7faa64073350 2024-09-17T03:58:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa54003a40 con 0x7faa64073350 2024-09-17T03:58:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa6414bee0 con 0x7faa64073350 2024-09-17T03:58:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.430+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6414c6d0 con 0x7faa64073350 2024-09-17T03:58:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.431+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa640d0f50 con 0x7faa64073350 2024-09-17T03:58:12.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.433+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faa5401a070 con 0x7faa64073350 2024-09-17T03:58:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.433+0000 7faa5be00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 0x7faa4407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.434+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(137..137 src has 1..137) ==== 5609+0+0 (secure 0 0 0) 0x7faa5409a400 con 0x7faa64073350 2024-09-17T03:58:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.434+0000 7faa63400640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 0x7faa4407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:12.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.435+0000 7faa63400640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 0x7faa4407b0d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7faa50034000 tx=0x7faa50004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.437+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faa54021020 con 0x7faa64073350 2024-09-17T03:58:12.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:12.656+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7faa64160ec0 con 0x7faa64073350 2024-09-17T03:58:13.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.172+0000 7faa5be00640 1 -- 172.21.15.129:0/1755639064 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v112) ==== 174+0+0 (secure 0 0 0) 0x7faa54062980 con 0x7faa64073350 2024-09-17T03:58:13.171 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 20 and data pool 21 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.177+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 msgr2=0x7faa4407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.177+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 0x7faa4407b0d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7faa50034000 tx=0x7faa50004730 comp rx=0 tx=0).stop 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.177+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 msgr2=0x7faa6414b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.177+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7faa5400c3f0 tx=0x7faa5400c8b0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 shutdown_connections 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa44078bc0 0x7faa4407b0d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa6415c7b0 0x7faa640c99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa640cb080 0x7faa64158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 --2- 172.21.15.129:0/1755639064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa64073350 0x7faa6414b9a0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 >> 172.21.15.129:0/1755639064 conn(0x7faa640d00b0 msgr2=0x7faa640acfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:13.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 shutdown_connections 2024-09-17T03:58:13.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.178+0000 7faa69c00640 1 -- 172.21.15.129:0/1755639064 wait complete. 2024-09-17T03:58:13.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:58:13.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 -- 172.21.15.129:0/2380808858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b806f410 msgr2=0x7f16b80cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 --2- 172.21.15.129:0/2380808858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b806f410 0x7f16b80cf2b0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f16b806eaf0 tx=0x7f16a402a560 comp rx=0 tx=0).stop 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 -- 172.21.15.129:0/2380808858 shutdown_connections 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 --2- 172.21.15.129:0/2380808858 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b80cf7f0 0x7f16b80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 --2- 172.21.15.129:0/2380808858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b806f410 0x7f16b80cf2b0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 --2- 172.21.15.129:0/2380808858 >> v2:172.21.15.181:3300/0 conn(0x7f16b8099570 0x7f16b806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 -- 172.21.15.129:0/2380808858 >> 172.21.15.129:0/2380808858 conn(0x7f16b806d410 msgr2=0x7f16b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 -- 172.21.15.129:0/2380808858 shutdown_connections 2024-09-17T03:58:13.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.450+0000 7f16bde00640 1 -- 172.21.15.129:0/2380808858 wait complete. 2024-09-17T03:58:13.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.452+0000 7f16bde00640 1 Processor -- start 2024-09-17T03:58:13.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.452+0000 7f16bde00640 1 -- start start 2024-09-17T03:58:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b806f410 0x7f16b81f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b8099570 0x7f16b80c9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8160990 con 0x7f16b80cf7f0 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16b8160810 con 0x7f16b8099570 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16bde00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f16b8160b10 con 0x7f16b806f410 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52340/0 (socket says 172.21.15.129:52340) 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16b7400640 1 -- 172.21.15.129:0/4224235937 learned_addr learned my addr 172.21.15.129:0/4224235937 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16b6a00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b806f410 0x7f16b81f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.454+0000 7f16b6000640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b8099570 0x7f16b80c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 -- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b806f410 msgr2=0x7f16b81f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b806f410 0x7f16b81f4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 -- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b8099570 msgr2=0x7f16b80c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b8099570 0x7f16b80c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 -- 172.21.15.129:0/4224235937 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b809d600 con 0x7f16b80cf7f0 2024-09-17T03:58:13.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16b7400640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f16b000e9c0 tx=0x7f16b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0004d20 con 0x7f16b80cf7f0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16b000cac0 con 0x7f16b80cf7f0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0005020 con 0x7f16b80cf7f0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b809d8e0 con 0x7f16b80cf7f0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.455+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b81f4cc0 con 0x7f16b80cf7f0 2024-09-17T03:58:13.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.456+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b806a140 con 0x7f16b80cf7f0 2024-09-17T03:58:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.458+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16b00051c0 con 0x7f16b80cf7f0 2024-09-17T03:58:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.459+0000 7f16af400640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 0x7f169807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.459+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7f16b009b520 con 0x7f16b80cf7f0 2024-09-17T03:58:13.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.459+0000 7f16b6a00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 0x7f169807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.460+0000 7f16b6a00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 0x7f169807b0a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f16a800bb50 tx=0x7f16a800b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:13.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.463+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16b0016030 con 0x7f16b80cf7f0 2024-09-17T03:58:13.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.648+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f16b815b950 con 0x7f16b80cf7f0 2024-09-17T03:58:13.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.649+0000 7f16af400640 1 -- 172.21.15.129:0/4224235937 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16800 (secure 0 0 0) 0x7f16b0063270 con 0x7f16b80cf7f0 2024-09-17T03:58:13.649 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:13.649 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":138,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:12.656968+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:58:10.577224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-17T03:58:11.627337+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:58:13.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 msgr2=0x7f169807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 0x7f169807b0a0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f16a800bb50 tx=0x7f16a800b2e0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 msgr2=0x7f16b815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f16b000e9c0 tx=0x7f16b000ee80 comp rx=0 tx=0).stop 2024-09-17T03:58:13.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 shutdown_connections 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1698078b90 0x7f169807b0a0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16b80cf7f0 0x7f16b815f050 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16b8099570 0x7f16b80c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 --2- 172.21.15.129:0/4224235937 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16b806f410 0x7f16b81f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 >> 172.21.15.129:0/4224235937 conn(0x7f16b806d410 msgr2=0x7f16b815c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 shutdown_connections 2024-09-17T03:58:13.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.652+0000 7f16bde00640 1 -- 172.21.15.129:0/4224235937 wait complete. 2024-09-17T03:58:13.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:58:13.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 -- 172.21.15.129:0/187516494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc92c003a00 msgr2=0x7fc92c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 --2- 172.21.15.129:0/187516494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc92c003a00 0x7fc92c003e70 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fc938009b50 tx=0x7fc92c005830 comp rx=0 tx=0).stop 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc947400640 1 -- 172.21.15.129:0/187516494 reap_dead start 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 -- 172.21.15.129:0/187516494 shutdown_connections 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 -- 172.21.15.129:0/187516494 >> 172.21.15.129:0/187516494 conn(0x7fc94806d410 msgr2=0x7fc9480b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 -- 172.21.15.129:0/187516494 shutdown_connections 2024-09-17T03:58:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.901+0000 7fc94de00640 1 -- 172.21.15.129:0/187516494 wait complete. 2024-09-17T03:58:13.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.903+0000 7fc94de00640 1 Processor -- start 2024-09-17T03:58:13.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.904+0000 7fc94de00640 1 -- start start 2024-09-17T03:58:13.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 0x7fc9481f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9481f2c90 0x7fc9481588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9480b3b90 con 0x7fc948158e30 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9480708e0 con 0x7fc92c003a00 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc94de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9480b3d10 con 0x7fc9481f2c90 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc947400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc947400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57320/0 (socket says 172.21.15.129:57320) 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc947400640 1 -- 172.21.15.129:0/1466066632 learned_addr learned my addr 172.21.15.129:0/1466066632 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc946a00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 0x7fc9481f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.905+0000 7fc946000640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9481f2c90 0x7fc9481588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 -- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9481f2c90 msgr2=0x7fc9481588f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9481f2c90 0x7fc9481588f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 -- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 msgr2=0x7fc9481f3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:13.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 0x7fc9481f3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:13.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 -- 172.21.15.129:0/1466066632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc948159240 con 0x7fc948158e30 2024-09-17T03:58:13.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc946a00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 0x7fc9481f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc947400640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc940005fd0 tx=0x7fc940010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940003970 con 0x7fc948158e30 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc940015070 con 0x7fc948158e30 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc948159a10 con 0x7fc948158e30 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.906+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc940004030 con 0x7fc948158e30 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.907+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc948159fd0 con 0x7fc948158e30 2024-09-17T03:58:13.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.908+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc948159780 con 0x7fc948158e30 2024-09-17T03:58:13.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.909+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9400041d0 con 0x7fc948158e30 2024-09-17T03:58:13.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.910+0000 7fc93f400640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 0x7fc92807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:13.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.910+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(138..138 src has 1..138) ==== 5709+0+0 (secure 0 0 0) 0x7fc94009b820 con 0x7fc948158e30 2024-09-17T03:58:13.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.910+0000 7fc946a00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 0x7fc92807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:13.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.911+0000 7fc946a00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 0x7fc92807b0d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc938035000 tx=0x7fc93802d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:13.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:13.914+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc948159240 con 0x7fc948158e30 2024-09-17T03:58:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.097+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9481f43e0 con 0x7fc948158e30 2024-09-17T03:58:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.098+0000 7fc93f400640 1 -- 172.21.15.129:0/1466066632 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) ==== 74+0+16800 (secure 0 0 0) 0x7fc940063570 con 0x7fc948158e30 2024-09-17T03:58:14.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:14.098 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":138,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:12.656968+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:58:10.577224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-17T03:58:11.627337+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:58:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 msgr2=0x7fc92807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 0x7fc92807b0d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fc938035000 tx=0x7fc93802d040 comp rx=0 tx=0).stop 2024-09-17T03:58:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 msgr2=0x7fc9481f6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fc940005fd0 tx=0x7fc940010f40 comp rx=0 tx=0).stop 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 shutdown_connections 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc928078bc0 0x7fc92807b0d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc948158e30 0x7fc9481f6a10 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9481f2c90 0x7fc9481588f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 --2- 172.21.15.129:0/1466066632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc92c003a00 0x7fc9481f3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 >> 172.21.15.129:0/1466066632 conn(0x7fc94806d410 msgr2=0x7fc9480af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 shutdown_connections 2024-09-17T03:58:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.101+0000 7fc94de00640 1 -- 172.21.15.129:0/1466066632 wait complete. 2024-09-17T03:58:14.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:58:14.153 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T03:58:14.154+0000 7f0ad4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:58:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 -- 172.21.15.129:0/2519611437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56580b4020 msgr2=0x7f5658167600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 --2- 172.21.15.129:0/2519611437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56580b4020 0x7f5658167600 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f5650005fd0 tx=0x7f565002a820 comp rx=0 tx=0).stop 2024-09-17T03:58:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 -- 172.21.15.129:0/2519611437 shutdown_connections 2024-09-17T03:58:14.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 --2- 172.21.15.129:0/2519611437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56580b4020 0x7f5658167600 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 -- 172.21.15.129:0/2519611437 >> 172.21.15.129:0/2519611437 conn(0x7f56580ccc40 msgr2=0x7f56580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:14.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 -- 172.21.15.129:0/2519611437 shutdown_connections 2024-09-17T03:58:14.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.353+0000 7f565f800640 1 -- 172.21.15.129:0/2519611437 wait complete. 2024-09-17T03:58:14.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.355+0000 7f565f800640 1 Processor -- start 2024-09-17T03:58:14.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.355+0000 7f565f800640 1 -- start start 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56580b1c40 0x7f56580cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56580b4020 0x7f56580ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5658167cc0 con 0x7f5658074800 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5658167b40 con 0x7f56580b1c40 2024-09-17T03:58:14.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5658167e40 con 0x7f56580b4020 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57334/0 (socket says 172.21.15.129:57334) 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565ce00640 1 -- 172.21.15.129:0/2081353299 learned_addr learned my addr 172.21.15.129:0/2081353299 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f5657e00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56580b1c40 0x7f56580cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.357+0000 7f565d800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56580b4020 0x7f56580ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 -- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56580b4020 msgr2=0x7f56580ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56580b4020 0x7f56580ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 -- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56580b1c40 msgr2=0x7f56580cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56580b1c40 0x7f56580cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 -- 172.21.15.129:0/2081353299 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56581548b0 con 0x7f5658074800 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565ce00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f564800b3f0 tx=0x7f564800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648015020 con 0x7f5658074800 2024-09-17T03:58:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5648011920 con 0x7f5658074800 2024-09-17T03:58:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5648011d70 con 0x7f5658074800 2024-09-17T03:58:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5658154b70 con 0x7f5658074800 2024-09-17T03:58:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.358+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5658150bd0 con 0x7f5658074800 2024-09-17T03:58:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.359+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56580af480 con 0x7f5658074800 2024-09-17T03:58:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.361+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5648018020 con 0x7f5658074800 2024-09-17T03:58:14.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.362+0000 7f5654c00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 0x7f563807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:14.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.362+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5709+0+0 (secure 0 0 0) 0x7f564809a8a0 con 0x7f5658074800 2024-09-17T03:58:14.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.362+0000 7f5657e00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 0x7f563807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:14.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.363+0000 7f5657e00640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 0x7f563807b0a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5644000f60 tx=0x7f5644009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:14.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.366+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f564801f030 con 0x7f5658074800 2024-09-17T03:58:14.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:14.584+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f56580b16c0 con 0x7f5658074800 2024-09-17T03:58:15.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.203+0000 7f5654c00640 1 -- 172.21.15.129:0/2081353299 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v114) ==== 118+0+0 (secure 0 0 0) 0x7f5648062690 con 0x7f5658074800 2024-09-17T03:58:15.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 msgr2=0x7f563807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:15.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 0x7f563807b0a0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5644000f60 tx=0x7f5644009040 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 msgr2=0x7f5658147cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f564800b3f0 tx=0x7f564800b8b0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 shutdown_connections 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5638078b90 0x7f563807b0a0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56580b4020 0x7f56580ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56580b1c40 0x7f56580cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 --2- 172.21.15.129:0/2081353299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5658074800 0x7f5658147cf0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 >> 172.21.15.129:0/2081353299 conn(0x7f56580ccc40 msgr2=0x7f56580d0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 shutdown_connections 2024-09-17T03:58:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.206+0000 7f565f800640 1 -- 172.21.15.129:0/2081353299 wait complete. 2024-09-17T03:58:15.227 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:58:15.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.493+0000 7f26f8a00640 1 -- 172.21.15.129:0/969940668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 msgr2=0x7f26f40cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.493+0000 7f26f8a00640 1 --2- 172.21.15.129:0/969940668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f40cb080 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009830 tx=0x7f26e402a520 comp rx=0 tx=0).stop 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.493+0000 7f26f8a00640 1 -- 172.21.15.129:0/969940668 shutdown_connections 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.493+0000 7f26f8a00640 1 --2- 172.21.15.129:0/969940668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f40cb080 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.493+0000 7f26f8a00640 1 -- 172.21.15.129:0/969940668 >> 172.21.15.129:0/969940668 conn(0x7f26f40d00b0 msgr2=0x7f26f40d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.494+0000 7f26f8a00640 1 -- 172.21.15.129:0/969940668 shutdown_connections 2024-09-17T03:58:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.494+0000 7f26f8a00640 1 -- 172.21.15.129:0/969940668 wait complete. 2024-09-17T03:58:15.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.496+0000 7f26f8a00640 1 Processor -- start 2024-09-17T03:58:15.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.496+0000 7f26f8a00640 1 -- start start 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26f40704c0 0x7f26f409b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f4151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f26f40ced10 con 0x7f26f40700b0 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f26f40ceb90 con 0x7f26f415c7b0 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.497+0000 7f26f8a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f26f40cee90 con 0x7f26f40704c0 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57340/0 (socket says 172.21.15.129:57340) 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 -- 172.21.15.129:0/2573974842 learned_addr learned my addr 172.21.15.129:0/2573974842 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3e00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f4151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f2a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26f40704c0 0x7f26f409b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 -- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26f40704c0 msgr2=0x7f26f409b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26f40704c0 0x7f26f409b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 -- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 msgr2=0x7f26f4151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f4151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 -- 172.21.15.129:0/2573974842 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f409ceb0 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.498+0000 7f26f3400640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f26e402ab10 tx=0x7f26e4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.499+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4005c90 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.499+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f409d190 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.499+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26e4031e50 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.499+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4005260 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.499+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f409d750 con 0x7f26f40700b0 2024-09-17T03:58:15.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.500+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f40cb080 con 0x7f26f40700b0 2024-09-17T03:58:15.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.501+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f26e4005400 con 0x7f26f40700b0 2024-09-17T03:58:15.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.502+0000 7f26eb400640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 0x7f26d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:15.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.502+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5709+0+0 (secure 0 0 0) 0x7f26e40bbe40 con 0x7f26f40700b0 2024-09-17T03:58:15.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.503+0000 7f26f2a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 0x7f26d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:15.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.503+0000 7f26f2a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 0x7f26d407b0a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f26f4160190 tx=0x7f26e0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:15.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.507+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f26e403f030 con 0x7f26f40700b0 2024-09-17T03:58:15.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:15.724+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f26f40d0f50 con 0x7f26f40700b0 2024-09-17T03:58:16.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.205+0000 7f26eb400640 1 -- 172.21.15.129:0/2573974842 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v115) ==== 122+0+0 (secure 0 0 0) 0x7f26e40841f0 con 0x7f26f40700b0 2024-09-17T03:58:16.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 msgr2=0x7f26d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 0x7f26d407b0a0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f26f4160190 tx=0x7f26e0009040 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 msgr2=0x7f26f409abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f26e402ab10 tx=0x7f26e4033c20 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 shutdown_connections 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f26d4078b90 0x7f26d407b0a0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f26f415c7b0 0x7f26f4151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f26f40704c0 0x7f26f409b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 --2- 172.21.15.129:0/2573974842 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f26f40700b0 0x7f26f409abb0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 >> 172.21.15.129:0/2573974842 conn(0x7f26f40d00b0 msgr2=0x7f26f40b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 shutdown_connections 2024-09-17T03:58:16.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.208+0000 7f26f8a00640 1 -- 172.21.15.129:0/2573974842 wait complete. 2024-09-17T03:58:16.232 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:58:16.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:16.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 -- 172.21.15.129:0/1187873442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 msgr2=0x7f89b40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 --2- 172.21.15.129:0/1187873442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b40b4140 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f89b4072080 tx=0x7f89a402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 -- 172.21.15.129:0/1187873442 shutdown_connections 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 --2- 172.21.15.129:0/1187873442 >> v2:172.21.15.181:3300/0 conn(0x7f89b406eed0 0x7f89b40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 --2- 172.21.15.129:0/1187873442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b40b4140 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 -- 172.21.15.129:0/1187873442 >> 172.21.15.129:0/1187873442 conn(0x7f89b406d410 msgr2=0x7f89b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 -- 172.21.15.129:0/1187873442 shutdown_connections 2024-09-17T03:58:16.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.453+0000 7f89bb200640 1 -- 172.21.15.129:0/1187873442 wait complete. 2024-09-17T03:58:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.455+0000 7f89bb200640 1 Processor -- start 2024-09-17T03:58:16.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.455+0000 7f89bb200640 1 -- start start 2024-09-17T03:58:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 0x7f89b40c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89b406f2e0 0x7f89b415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f89b4161250 con 0x7f89b40705a0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f89b41610d0 con 0x7f89b406eed0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89bb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f89b41613d0 con 0x7f89b406f2e0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89b9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89b9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57358/0 (socket says 172.21.15.129:57358) 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89b9200640 1 -- 172.21.15.129:0/3783763653 learned_addr learned my addr 172.21.15.129:0/3783763653 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89b3e00640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 0x7f89b40c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.457+0000 7f89b3400640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89b406f2e0 0x7f89b415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 -- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89b406f2e0 msgr2=0x7f89b415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89b406f2e0 0x7f89b415a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 -- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 msgr2=0x7f89b40c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 0x7f89b40c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 -- 172.21.15.129:0/3783763653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b414e030 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b3e00640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 0x7f89b40c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89b9200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c730 tx=0x7f89ac00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac003b70 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89ac003d10 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b414e310 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89ac005330 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.458+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b414e8d0 con 0x7f89b40705a0 2024-09-17T03:58:16.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.459+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89b40cd5a0 con 0x7f89b40705a0 2024-09-17T03:58:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.461+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f89ac0056e0 con 0x7f89b40705a0 2024-09-17T03:58:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.461+0000 7f89abe00640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 0x7f899407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.462+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5709+0+0 (secure 0 0 0) 0x7f89ac09ad90 con 0x7f89b40705a0 2024-09-17T03:58:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.462+0000 7f89b3e00640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 0x7f899407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.463+0000 7f89b3e00640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 0x7f899407b0a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f89a4033000 tx=0x7f89a4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:16.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.466+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f89ac063130 con 0x7f89b40705a0 2024-09-17T03:58:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.687+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89b40af7f0 con 0x7f89b40705a0 2024-09-17T03:58:16.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.687+0000 7f89abe00640 1 -- 172.21.15.129:0/3783763653 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5239 (secure 0 0 0) 0x7f89ac062a30 con 0x7f89b40705a0 2024-09-17T03:58:16.687 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:16.687 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":115,"btime":"2024-09-17T03:58:16:203956+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3687707903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3687707903},{"type":"v1","addr":"172.21.15.181:6837","nonce":3687707903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/872498647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":872498647},{"type":"v1","addr":"172.21.15.181:6836","nonce":872498647}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":8178,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:15.725002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 115 2024-09-17T03:58:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 msgr2=0x7f899407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 0x7f899407b0a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f89a4033000 tx=0x7f89a4004730 comp rx=0 tx=0).stop 2024-09-17T03:58:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 msgr2=0x7f89b415aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f89ac00c730 tx=0x7f89ac00cbf0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 shutdown_connections 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8994078b90 0x7f899407b0a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f89b40705a0 0x7f89b415aa00 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f89b406f2e0 0x7f89b415a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 --2- 172.21.15.129:0/3783763653 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89b406eed0 0x7f89b40c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 >> 172.21.15.129:0/3783763653 conn(0x7f89b406d410 msgr2=0x7f89b40b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 shutdown_connections 2024-09-17T03:58:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.690+0000 7f89bb200640 1 -- 172.21.15.129:0/3783763653 wait complete. 2024-09-17T03:58:16.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 -- 172.21.15.129:0/2088991033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7080cd390 msgr2=0x7ff7081609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 --2- 172.21.15.129:0/2088991033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7080cd390 0x7ff7081609c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff700009be0 tx=0x7ff70002a8b0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 -- 172.21.15.129:0/2088991033 shutdown_connections 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 --2- 172.21.15.129:0/2088991033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff7080cd390 0x7ff7081609c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 -- 172.21.15.129:0/2088991033 >> 172.21.15.129:0/2088991033 conn(0x7ff70806d410 msgr2=0x7ff7080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 -- 172.21.15.129:0/2088991033 shutdown_connections 2024-09-17T03:58:16.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.936+0000 7ff70d200640 1 -- 172.21.15.129:0/2088991033 wait complete. 2024-09-17T03:58:16.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.939+0000 7ff70d200640 1 Processor -- start 2024-09-17T03:58:16.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.939+0000 7ff70d200640 1 -- start start 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7080705a0 0x7ff7081f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff70815a9a0 0x7ff708155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff708160f00 con 0x7ff7080b4140 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff708070d60 con 0x7ff7080705a0 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff70d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff708161080 con 0x7ff70815a9a0 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.940+0000 7ff705400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57374/0 (socket says 172.21.15.129:57374) 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 -- 172.21.15.129:0/68508854 learned_addr learned my addr 172.21.15.129:0/68508854 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff706800640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff70815a9a0 0x7ff708155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705e00640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7080705a0 0x7ff7081f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 -- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff70815a9a0 msgr2=0x7ff708155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff70815a9a0 0x7ff708155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 -- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7080705a0 msgr2=0x7ff7081f54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7080705a0 0x7ff7081f54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 -- 172.21.15.129:0/68508854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff70815b0e0 con 0x7ff7080b4140 2024-09-17T03:58:16.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff705400640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c3c0 tx=0x7ff6f4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.941+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4017070 con 0x7ff7080b4140 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.942+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff6f40053e0 con 0x7ff7080b4140 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.942+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff70815b3c0 con 0x7ff7080b4140 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.942+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4003850 con 0x7ff7080b4140 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.942+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff70815b980 con 0x7ff7080b4140 2024-09-17T03:58:16.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.943+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7080aec90 con 0x7ff7080b4140 2024-09-17T03:58:16.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.944+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff6f401a070 con 0x7ff7080b4140 2024-09-17T03:58:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.945+0000 7ff6fe000640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 0x7ff6e807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.945+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(139..139 src has 1..139) ==== 5709+0+0 (secure 0 0 0) 0x7ff6f409a100 con 0x7ff7080b4140 2024-09-17T03:58:16.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.946+0000 7ff705e00640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 0x7ff6e807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:16.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.946+0000 7ff705e00640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 0x7ff6e807b0d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8004600 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:16.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:16.949+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff7080aec90 con 0x7ff7080b4140 2024-09-17T03:58:17.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.134+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7081609c0 con 0x7ff7080b4140 2024-09-17T03:58:17.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.136+0000 7ff6fe000640 1 -- 172.21.15.129:0/68508854 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) ==== 74+0+16787 (secure 0 0 0) 0x7ff6f4061f10 con 0x7ff7080b4140 2024-09-17T03:58:17.135 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:17.135 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":139,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:14.186183+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:58:10.577224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-17T03:58:11.627337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:58:17.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 msgr2=0x7ff6e807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 0x7ff6e807b0d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000c00 tx=0x7ff6f8004600 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 msgr2=0x7ff70815a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c3c0 tx=0x7ff6f4001e70 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 shutdown_connections 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff6e8078bc0 0x7ff6e807b0d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff70815a9a0 0x7ff708155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff7080b4140 0x7ff70815a460 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 --2- 172.21.15.129:0/68508854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff7080705a0 0x7ff7081f54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 >> 172.21.15.129:0/68508854 conn(0x7ff70806d410 msgr2=0x7ff708072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 shutdown_connections 2024-09-17T03:58:17.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.138+0000 7ff70d200640 1 -- 172.21.15.129:0/68508854 wait complete. 2024-09-17T03:58:17.160 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:58:17.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:58:17.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:58:17.332 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 -- 172.21.15.129:0/555337516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 msgr2=0x7f14280d1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 --2- 172.21.15.129:0/555337516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f14280d1820 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f14280d2c80 tx=0x7f141802a530 comp rx=0 tx=0).stop 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 -- 172.21.15.129:0/555337516 shutdown_connections 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 --2- 172.21.15.129:0/555337516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 0x7f14280cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 --2- 172.21.15.129:0/555337516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f14280d1820 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 -- 172.21.15.129:0/555337516 >> 172.21.15.129:0/555337516 conn(0x7f142809d930 msgr2=0x7f142808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 -- 172.21.15.129:0/555337516 shutdown_connections 2024-09-17T03:58:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.387+0000 7f142d400640 1 -- 172.21.15.129:0/555337516 wait complete. 2024-09-17T03:58:17.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.390+0000 7f142d400640 1 Processor -- start 2024-09-17T03:58:17.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.390+0000 7f142d400640 1 -- start start 2024-09-17T03:58:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 0x7f14280b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 0x7f14280b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14280cfae0 con 0x7f14280b32f0 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14280d1f70 con 0x7f14281627a0 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f142d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14280cfc60 con 0x7f14280cf190 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.391+0000 7f1426800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57390/0 (socket says 172.21.15.129:57390) 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 -- 172.21.15.129:0/354596478 learned_addr learned my addr 172.21.15.129:0/354596478 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1425e00640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 0x7f14280b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1427200640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 0x7f14280b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 -- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 msgr2=0x7f14280b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 0x7f14280b8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 -- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 msgr2=0x7f14280b8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 0x7f14280b8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 -- 172.21.15.129:0/354596478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142815e7c0 con 0x7f14280b32f0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1427200640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 0x7f14280b8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f1426800640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1418000c00 tx=0x7f14180038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.392+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141803e030 con 0x7f14280b32f0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.393+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1418003d90 con 0x7f14280b32f0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.393+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f142809e340 con 0x7f14280b32f0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.393+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418035040 con 0x7f14280b32f0 2024-09-17T03:58:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.393+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f142809e900 con 0x7f14280b32f0 2024-09-17T03:58:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.394+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14281634d0 con 0x7f14280b32f0 2024-09-17T03:58:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.395+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1418041030 con 0x7f14280b32f0 2024-09-17T03:58:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.396+0000 7f141ea00640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 0x7f140807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.396+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5738+0+0 (secure 0 0 0) 0x7f14180bb530 con 0x7f14280b32f0 2024-09-17T03:58:17.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.396+0000 7f1425e00640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 0x7f140807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.397+0000 7f1425e00640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 0x7f140807b0a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f14280d08f0 tx=0x7f1414005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:17.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.400+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1418083260 con 0x7f14280b32f0 2024-09-17T03:58:17.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.582+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14280d1320 con 0x7f1408078b90 2024-09-17T03:58:17.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.587+0000 7f141ea00640 1 -- 172.21.15.129:0/354596478 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7f14280d1320 con 0x7f1408078b90 2024-09-17T03:58:17.589 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:17.591 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:58:17.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 msgr2=0x7f140807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 0x7f140807b0a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f14280d08f0 tx=0x7f1414005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:17.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 msgr2=0x7f142815e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f1418000c00 tx=0x7f14180038a0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 shutdown_connections 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1408078b90 0x7f140807b0a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14281627a0 0x7f14280b8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14280cf190 0x7f14280b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 --2- 172.21.15.129:0/354596478 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14280b32f0 0x7f142815e280 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 >> 172.21.15.129:0/354596478 conn(0x7f142809d930 msgr2=0x7f142808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 shutdown_connections 2024-09-17T03:58:17.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.591+0000 7f142d400640 1 -- 172.21.15.129:0/354596478 wait complete. 2024-09-17T03:58:17.613 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-17T03:58:17.219112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309916,"kb_used_data":14624,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664804,"statfs":{"total":773094113280,"available":772776759296,"internally_reserved":0,"allocated":14974976,"data_stored":7159522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.055109"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195048+0000","last_change":"2024-09-17T03:58:13.210124+0000","last_active":"2024-09-17T03:58:14.195048+0000","last_peered":"2024-09-17T03:58:14.195048+0000","last_clean":"2024-09-17T03:58:14.195048+0000","last_became_active":"2024-09-17T03:58:13.209826+0000","last_became_peered":"2024-09-17T03:58:13.209826+0000","last_unstale":"2024-09-17T03:58:14.195048+0000","last_undegraded":"2024-09-17T03:58:14.195048+0000","last_fullsized":"2024-09-17T03:58:14.195048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.883085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203371+0000","last_change":"2024-09-17T03:58:13.197182+0000","last_active":"2024-09-17T03:58:13.203371+0000","last_peered":"2024-09-17T03:58:13.203371+0000","last_clean":"2024-09-17T03:58:13.203371+0000","last_became_active":"2024-09-17T03:58:13.196886+0000","last_became_peered":"2024-09-17T03:58:13.196886+0000","last_unstale":"2024-09-17T03:58:13.203371+0000","last_undegraded":"2024-09-17T03:58:13.203371+0000","last_fullsized":"2024-09-17T03:58:13.203371+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:15.551727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202868+0000","last_change":"2024-09-17T03:58:13.198559+0000","last_active":"2024-09-17T03:58:13.202868+0000","last_peered":"2024-09-17T03:58:13.202868+0000","last_clean":"2024-09-17T03:58:13.202868+0000","last_became_active":"2024-09-17T03:58:13.197992+0000","last_became_peered":"2024-09-17T03:58:13.197992+0000","last_unstale":"2024-09-17T03:58:13.202868+0000","last_undegraded":"2024-09-17T03:58:13.202868+0000","last_fullsized":"2024-09-17T03:58:13.202868+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:42:01.710483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195216+0000","last_change":"2024-09-17T03:58:13.199007+0000","last_active":"2024-09-17T03:58:14.195216+0000","last_peered":"2024-09-17T03:58:14.195216+0000","last_clean":"2024-09-17T03:58:14.195216+0000","last_became_active":"2024-09-17T03:58:13.198682+0000","last_became_peered":"2024-09-17T03:58:13.198682+0000","last_unstale":"2024-09-17T03:58:14.195216+0000","last_undegraded":"2024-09-17T03:58:14.195216+0000","last_fullsized":"2024-09-17T03:58:14.195216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:53.789318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193897+0000","last_change":"2024-09-17T03:58:13.210650+0000","last_active":"2024-09-17T03:58:14.193897+0000","last_peered":"2024-09-17T03:58:14.193897+0000","last_clean":"2024-09-17T03:58:14.193897+0000","last_became_active":"2024-09-17T03:58:13.210304+0000","last_became_peered":"2024-09-17T03:58:13.210304+0000","last_unstale":"2024-09-17T03:58:14.193897+0000","last_undegraded":"2024-09-17T03:58:14.193897+0000","last_fullsized":"2024-09-17T03:58:14.193897+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:27.323807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193911+0000","last_change":"2024-09-17T03:58:13.205950+0000","last_active":"2024-09-17T03:58:14.193911+0000","last_peered":"2024-09-17T03:58:14.193911+0000","last_clean":"2024-09-17T03:58:14.193911+0000","last_became_active":"2024-09-17T03:58:13.205112+0000","last_became_peered":"2024-09-17T03:58:13.205112+0000","last_unstale":"2024-09-17T03:58:14.193911+0000","last_undegraded":"2024-09-17T03:58:14.193911+0000","last_fullsized":"2024-09-17T03:58:14.193911+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:05.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213829+0000","last_change":"2024-09-17T03:58:13.208974+0000","last_active":"2024-09-17T03:58:13.213829+0000","last_peered":"2024-09-17T03:58:13.213829+0000","last_clean":"2024-09-17T03:58:13.213829+0000","last_became_active":"2024-09-17T03:58:13.208167+0000","last_became_peered":"2024-09-17T03:58:13.208167+0000","last_unstale":"2024-09-17T03:58:13.213829+0000","last_undegraded":"2024-09-17T03:58:13.213829+0000","last_fullsized":"2024-09-17T03:58:13.213829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:12.378776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193840+0000","last_change":"2024-09-17T03:58:13.191928+0000","last_active":"2024-09-17T03:58:14.193840+0000","last_peered":"2024-09-17T03:58:14.193840+0000","last_clean":"2024-09-17T03:58:14.193840+0000","last_became_active":"2024-09-17T03:58:13.191555+0000","last_became_peered":"2024-09-17T03:58:13.191555+0000","last_unstale":"2024-09-17T03:58:14.193840+0000","last_undegraded":"2024-09-17T03:58:14.193840+0000","last_fullsized":"2024-09-17T03:58:14.193840+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:51:10.629162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.197765+0000","last_change":"2024-09-17T03:58:13.194603+0000","last_active":"2024-09-17T03:58:13.197765+0000","last_peered":"2024-09-17T03:58:13.197765+0000","last_clean":"2024-09-17T03:58:13.197765+0000","last_became_active":"2024-09-17T03:58:13.194324+0000","last_became_peered":"2024-09-17T03:58:13.194324+0000","last_unstale":"2024-09-17T03:58:13.197765+0000","last_undegraded":"2024-09-17T03:58:13.197765+0000","last_fullsized":"2024-09-17T03:58:13.197765+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.566895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.225062+0000","last_change":"2024-09-17T03:58:13.218976+0000","last_active":"2024-09-17T03:58:13.225062+0000","last_peered":"2024-09-17T03:58:13.225062+0000","last_clean":"2024-09-17T03:58:13.225062+0000","last_became_active":"2024-09-17T03:58:13.218566+0000","last_became_peered":"2024-09-17T03:58:13.218566+0000","last_unstale":"2024-09-17T03:58:13.225062+0000","last_undegraded":"2024-09-17T03:58:13.225062+0000","last_fullsized":"2024-09-17T03:58:13.225062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:29.157380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194344+0000","last_change":"2024-09-17T03:58:13.197033+0000","last_active":"2024-09-17T03:58:14.194344+0000","last_peered":"2024-09-17T03:58:14.194344+0000","last_clean":"2024-09-17T03:58:14.194344+0000","last_became_active":"2024-09-17T03:58:13.196347+0000","last_became_peered":"2024-09-17T03:58:13.196347+0000","last_unstale":"2024-09-17T03:58:14.194344+0000","last_undegraded":"2024-09-17T03:58:14.194344+0000","last_fullsized":"2024-09-17T03:58:14.194344+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:57.150602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218179+0000","last_change":"2024-09-17T03:58:13.213175+0000","last_active":"2024-09-17T03:58:13.218179+0000","last_peered":"2024-09-17T03:58:13.218179+0000","last_clean":"2024-09-17T03:58:13.218179+0000","last_became_active":"2024-09-17T03:58:13.211551+0000","last_became_peered":"2024-09-17T03:58:13.211551+0000","last_unstale":"2024-09-17T03:58:13.218179+0000","last_undegraded":"2024-09-17T03:58:13.218179+0000","last_fullsized":"2024-09-17T03:58:13.218179+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:00.545295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194118+0000","last_change":"2024-09-17T03:58:13.209833+0000","last_active":"2024-09-17T03:58:14.194118+0000","last_peered":"2024-09-17T03:58:14.194118+0000","last_clean":"2024-09-17T03:58:14.194118+0000","last_became_active":"2024-09-17T03:58:13.209524+0000","last_became_peered":"2024-09-17T03:58:13.209524+0000","last_unstale":"2024-09-17T03:58:14.194118+0000","last_undegraded":"2024-09-17T03:58:14.194118+0000","last_fullsized":"2024-09-17T03:58:14.194118+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.519428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194749+0000","last_change":"2024-09-17T03:58:13.207886+0000","last_active":"2024-09-17T03:58:14.194749+0000","last_peered":"2024-09-17T03:58:14.194749+0000","last_clean":"2024-09-17T03:58:14.194749+0000","last_became_active":"2024-09-17T03:58:13.207566+0000","last_became_peered":"2024-09-17T03:58:13.207566+0000","last_unstale":"2024-09-17T03:58:14.194749+0000","last_undegraded":"2024-09-17T03:58:14.194749+0000","last_fullsized":"2024-09-17T03:58:14.194749+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:44.777400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221333+0000","last_change":"2024-09-17T03:58:13.214462+0000","last_active":"2024-09-17T03:58:13.221333+0000","last_peered":"2024-09-17T03:58:13.221333+0000","last_clean":"2024-09-17T03:58:13.221333+0000","last_became_active":"2024-09-17T03:58:13.213558+0000","last_became_peered":"2024-09-17T03:58:13.213558+0000","last_unstale":"2024-09-17T03:58:13.221333+0000","last_undegraded":"2024-09-17T03:58:13.221333+0000","last_fullsized":"2024-09-17T03:58:13.221333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:17.721542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206889+0000","last_change":"2024-09-17T03:58:13.196823+0000","last_active":"2024-09-17T03:58:13.206889+0000","last_peered":"2024-09-17T03:58:13.206889+0000","last_clean":"2024-09-17T03:58:13.206889+0000","last_became_active":"2024-09-17T03:58:13.196229+0000","last_became_peered":"2024-09-17T03:58:13.196229+0000","last_unstale":"2024-09-17T03:58:13.206889+0000","last_undegraded":"2024-09-17T03:58:13.206889+0000","last_fullsized":"2024-09-17T03:58:13.206889+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:33.104699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221526+0000","last_change":"2024-09-17T03:58:13.216483+0000","last_active":"2024-09-17T03:58:13.221526+0000","last_peered":"2024-09-17T03:58:13.221526+0000","last_clean":"2024-09-17T03:58:13.221526+0000","last_became_active":"2024-09-17T03:58:13.215470+0000","last_became_peered":"2024-09-17T03:58:13.215470+0000","last_unstale":"2024-09-17T03:58:13.221526+0000","last_undegraded":"2024-09-17T03:58:13.221526+0000","last_fullsized":"2024-09-17T03:58:13.221526+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:45.587681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202663+0000","last_change":"2024-09-17T03:58:13.197076+0000","last_active":"2024-09-17T03:58:13.202663+0000","last_peered":"2024-09-17T03:58:13.202663+0000","last_clean":"2024-09-17T03:58:13.202663+0000","last_became_active":"2024-09-17T03:58:13.196769+0000","last_became_peered":"2024-09-17T03:58:13.196769+0000","last_unstale":"2024-09-17T03:58:13.202663+0000","last_undegraded":"2024-09-17T03:58:13.202663+0000","last_fullsized":"2024-09-17T03:58:13.202663+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.630755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194628+0000","last_change":"2024-09-17T03:58:13.201472+0000","last_active":"2024-09-17T03:58:14.194628+0000","last_peered":"2024-09-17T03:58:14.194628+0000","last_clean":"2024-09-17T03:58:14.194628+0000","last_became_active":"2024-09-17T03:58:13.201148+0000","last_became_peered":"2024-09-17T03:58:13.201148+0000","last_unstale":"2024-09-17T03:58:14.194628+0000","last_undegraded":"2024-09-17T03:58:14.194628+0000","last_fullsized":"2024-09-17T03:58:14.194628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:31.684429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.204183+0000","last_change":"2024-09-17T03:58:13.200828+0000","last_active":"2024-09-17T03:58:13.204183+0000","last_peered":"2024-09-17T03:58:13.204183+0000","last_clean":"2024-09-17T03:58:13.204183+0000","last_became_active":"2024-09-17T03:58:13.200506+0000","last_became_peered":"2024-09-17T03:58:13.200506+0000","last_unstale":"2024-09-17T03:58:13.204183+0000","last_undegraded":"2024-09-17T03:58:13.204183+0000","last_fullsized":"2024-09-17T03:58:13.204183+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:27.960576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.225031+0000","last_change":"2024-09-17T03:58:13.218497+0000","last_active":"2024-09-17T03:58:13.225031+0000","last_peered":"2024-09-17T03:58:13.225031+0000","last_clean":"2024-09-17T03:58:13.225031+0000","last_became_active":"2024-09-17T03:58:13.218096+0000","last_became_peered":"2024-09-17T03:58:13.218096+0000","last_unstale":"2024-09-17T03:58:13.225031+0000","last_undegraded":"2024-09-17T03:58:13.225031+0000","last_fullsized":"2024-09-17T03:58:13.225031+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:51.038577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194582+0000","last_change":"2024-09-17T03:58:13.196674+0000","last_active":"2024-09-17T03:58:14.194582+0000","last_peered":"2024-09-17T03:58:14.194582+0000","last_clean":"2024-09-17T03:58:14.194582+0000","last_became_active":"2024-09-17T03:58:13.196343+0000","last_became_peered":"2024-09-17T03:58:13.196343+0000","last_unstale":"2024-09-17T03:58:14.194582+0000","last_undegraded":"2024-09-17T03:58:14.194582+0000","last_fullsized":"2024-09-17T03:58:14.194582+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:28.637873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194242+0000","last_change":"2024-09-17T03:58:13.200728+0000","last_active":"2024-09-17T03:58:14.194242+0000","last_peered":"2024-09-17T03:58:14.194242+0000","last_clean":"2024-09-17T03:58:14.194242+0000","last_became_active":"2024-09-17T03:58:13.200428+0000","last_became_peered":"2024-09-17T03:58:13.200428+0000","last_unstale":"2024-09-17T03:58:14.194242+0000","last_undegraded":"2024-09-17T03:58:14.194242+0000","last_fullsized":"2024-09-17T03:58:14.194242+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:43.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221441+0000","last_change":"2024-09-17T03:58:13.215991+0000","last_active":"2024-09-17T03:58:13.221441+0000","last_peered":"2024-09-17T03:58:13.221441+0000","last_clean":"2024-09-17T03:58:13.221441+0000","last_became_active":"2024-09-17T03:58:13.215679+0000","last_became_peered":"2024-09-17T03:58:13.215679+0000","last_unstale":"2024-09-17T03:58:13.221441+0000","last_undegraded":"2024-09-17T03:58:13.221441+0000","last_fullsized":"2024-09-17T03:58:13.221441+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:14.450014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211019+0000","last_change":"2024-09-17T03:58:13.205986+0000","last_active":"2024-09-17T03:58:13.211019+0000","last_peered":"2024-09-17T03:58:13.211019+0000","last_clean":"2024-09-17T03:58:13.211019+0000","last_became_active":"2024-09-17T03:58:13.205644+0000","last_became_peered":"2024-09-17T03:58:13.205644+0000","last_unstale":"2024-09-17T03:58:13.211019+0000","last_undegraded":"2024-09-17T03:58:13.211019+0000","last_fullsized":"2024-09-17T03:58:13.211019+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:12.588562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194669+0000","last_change":"2024-09-17T03:58:13.202774+0000","last_active":"2024-09-17T03:58:14.194669+0000","last_peered":"2024-09-17T03:58:14.194669+0000","last_clean":"2024-09-17T03:58:14.194669+0000","last_became_active":"2024-09-17T03:58:13.201333+0000","last_became_peered":"2024-09-17T03:58:13.201333+0000","last_unstale":"2024-09-17T03:58:14.194669+0000","last_undegraded":"2024-09-17T03:58:14.194669+0000","last_fullsized":"2024-09-17T03:58:14.194669+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:20.148226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211663+0000","last_change":"2024-09-17T03:58:13.202837+0000","last_active":"2024-09-17T03:58:13.211663+0000","last_peered":"2024-09-17T03:58:13.211663+0000","last_clean":"2024-09-17T03:58:13.211663+0000","last_became_active":"2024-09-17T03:58:13.202450+0000","last_became_peered":"2024-09-17T03:58:13.202450+0000","last_unstale":"2024-09-17T03:58:13.211663+0000","last_undegraded":"2024-09-17T03:58:13.211663+0000","last_fullsized":"2024-09-17T03:58:13.211663+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:48.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213937+0000","last_change":"2024-09-17T03:58:13.208000+0000","last_active":"2024-09-17T03:58:13.213937+0000","last_peered":"2024-09-17T03:58:13.213937+0000","last_clean":"2024-09-17T03:58:13.213937+0000","last_became_active":"2024-09-17T03:58:13.206625+0000","last_became_peered":"2024-09-17T03:58:13.206625+0000","last_unstale":"2024-09-17T03:58:13.213937+0000","last_undegraded":"2024-09-17T03:58:13.213937+0000","last_fullsized":"2024-09-17T03:58:13.213937+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:25.907710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193976+0000","last_change":"2024-09-17T03:58:13.203722+0000","last_active":"2024-09-17T03:58:14.193976+0000","last_peered":"2024-09-17T03:58:14.193976+0000","last_clean":"2024-09-17T03:58:14.193976+0000","last_became_active":"2024-09-17T03:58:13.200741+0000","last_became_peered":"2024-09-17T03:58:13.200741+0000","last_unstale":"2024-09-17T03:58:14.193976+0000","last_undegraded":"2024-09-17T03:58:14.193976+0000","last_fullsized":"2024-09-17T03:58:14.193976+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:40.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194834+0000","last_change":"2024-09-17T03:58:13.202191+0000","last_active":"2024-09-17T03:58:14.194834+0000","last_peered":"2024-09-17T03:58:14.194834+0000","last_clean":"2024-09-17T03:58:14.194834+0000","last_became_active":"2024-09-17T03:58:13.201859+0000","last_became_peered":"2024-09-17T03:58:13.201859+0000","last_unstale":"2024-09-17T03:58:14.194834+0000","last_undegraded":"2024-09-17T03:58:14.194834+0000","last_fullsized":"2024-09-17T03:58:14.194834+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:48.883599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206569+0000","last_change":"2024-09-17T03:58:13.202176+0000","last_active":"2024-09-17T03:58:13.206569+0000","last_peered":"2024-09-17T03:58:13.206569+0000","last_clean":"2024-09-17T03:58:13.206569+0000","last_became_active":"2024-09-17T03:58:13.201890+0000","last_became_peered":"2024-09-17T03:58:13.201890+0000","last_unstale":"2024-09-17T03:58:13.206569+0000","last_undegraded":"2024-09-17T03:58:13.206569+0000","last_fullsized":"2024-09-17T03:58:13.206569+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:04.144165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206853+0000","last_change":"2024-09-17T03:58:13.194115+0000","last_active":"2024-09-17T03:58:13.206853+0000","last_peered":"2024-09-17T03:58:13.206853+0000","last_clean":"2024-09-17T03:58:13.206853+0000","last_became_active":"2024-09-17T03:58:13.193359+0000","last_became_peered":"2024-09-17T03:58:13.193359+0000","last_unstale":"2024-09-17T03:58:13.206853+0000","last_undegraded":"2024-09-17T03:58:13.206853+0000","last_fullsized":"2024-09-17T03:58:13.206853+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:03.425741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221384+0000","last_change":"2024-09-17T03:58:13.214802+0000","last_active":"2024-09-17T03:58:13.221384+0000","last_peered":"2024-09-17T03:58:13.221384+0000","last_clean":"2024-09-17T03:58:13.221384+0000","last_became_active":"2024-09-17T03:58:13.213477+0000","last_became_peered":"2024-09-17T03:58:13.213477+0000","last_unstale":"2024-09-17T03:58:13.221384+0000","last_undegraded":"2024-09-17T03:58:13.221384+0000","last_fullsized":"2024-09-17T03:58:13.221384+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.520646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217520+0000","last_change":"2024-09-17T03:58:12.191412+0000","last_active":"2024-09-17T03:58:13.217520+0000","last_peered":"2024-09-17T03:58:13.217520+0000","last_clean":"2024-09-17T03:58:13.217520+0000","last_became_active":"2024-09-17T03:58:12.191081+0000","last_became_peered":"2024-09-17T03:58:12.191081+0000","last_unstale":"2024-09-17T03:58:13.217520+0000","last_undegraded":"2024-09-17T03:58:13.217520+0000","last_fullsized":"2024-09-17T03:58:13.217520+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:52.781413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.196046+0000","last_change":"2024-09-17T03:58:13.193155+0000","last_active":"2024-09-17T03:58:13.196046+0000","last_peered":"2024-09-17T03:58:13.196046+0000","last_clean":"2024-09-17T03:58:13.196046+0000","last_became_active":"2024-09-17T03:58:13.192825+0000","last_became_peered":"2024-09-17T03:58:13.192825+0000","last_unstale":"2024-09-17T03:58:13.196046+0000","last_undegraded":"2024-09-17T03:58:13.196046+0000","last_fullsized":"2024-09-17T03:58:13.196046+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:26.594749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.208724+0000","last_change":"2024-09-17T03:58:12.189882+0000","last_active":"2024-09-17T03:58:13.208724+0000","last_peered":"2024-09-17T03:58:13.208724+0000","last_clean":"2024-09-17T03:58:13.208724+0000","last_became_active":"2024-09-17T03:58:12.189556+0000","last_became_peered":"2024-09-17T03:58:12.189556+0000","last_unstale":"2024-09-17T03:58:13.208724+0000","last_undegraded":"2024-09-17T03:58:13.208724+0000","last_fullsized":"2024-09-17T03:58:13.208724+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:29.533857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194271+0000","last_change":"2024-09-17T03:58:13.200790+0000","last_active":"2024-09-17T03:58:14.194271+0000","last_peered":"2024-09-17T03:58:14.194271+0000","last_clean":"2024-09-17T03:58:14.194271+0000","last_became_active":"2024-09-17T03:58:13.199709+0000","last_became_peered":"2024-09-17T03:58:13.199709+0000","last_unstale":"2024-09-17T03:58:14.194271+0000","last_undegraded":"2024-09-17T03:58:14.194271+0000","last_fullsized":"2024-09-17T03:58:14.194271+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:06.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194510+0000","last_change":"2024-09-17T03:58:12.182338+0000","last_active":"2024-09-17T03:58:14.194510+0000","last_peered":"2024-09-17T03:58:14.194510+0000","last_clean":"2024-09-17T03:58:14.194510+0000","last_became_active":"2024-09-17T03:58:12.182023+0000","last_became_peered":"2024-09-17T03:58:12.182023+0000","last_unstale":"2024-09-17T03:58:14.194510+0000","last_undegraded":"2024-09-17T03:58:14.194510+0000","last_fullsized":"2024-09-17T03:58:14.194510+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:42.424799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217966+0000","last_change":"2024-09-17T03:58:13.212687+0000","last_active":"2024-09-17T03:58:13.217966+0000","last_peered":"2024-09-17T03:58:13.217966+0000","last_clean":"2024-09-17T03:58:13.217966+0000","last_became_active":"2024-09-17T03:58:13.211587+0000","last_became_peered":"2024-09-17T03:58:13.211587+0000","last_unstale":"2024-09-17T03:58:13.217966+0000","last_undegraded":"2024-09-17T03:58:13.217966+0000","last_fullsized":"2024-09-17T03:58:13.217966+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:35.560553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195184+0000","last_change":"2024-09-17T03:58:12.177610+0000","last_active":"2024-09-17T03:58:14.195184+0000","last_peered":"2024-09-17T03:58:14.195184+0000","last_clean":"2024-09-17T03:58:14.195184+0000","last_became_active":"2024-09-17T03:58:12.177288+0000","last_became_peered":"2024-09-17T03:58:12.177288+0000","last_unstale":"2024-09-17T03:58:14.195184+0000","last_undegraded":"2024-09-17T03:58:14.195184+0000","last_fullsized":"2024-09-17T03:58:14.195184+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:51:09.621257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.208294+0000","last_change":"2024-09-17T03:58:13.205019+0000","last_active":"2024-09-17T03:58:13.208294+0000","last_peered":"2024-09-17T03:58:13.208294+0000","last_clean":"2024-09-17T03:58:13.208294+0000","last_became_active":"2024-09-17T03:58:13.201214+0000","last_became_peered":"2024-09-17T03:58:13.201214+0000","last_unstale":"2024-09-17T03:58:13.208294+0000","last_undegraded":"2024-09-17T03:58:13.208294+0000","last_fullsized":"2024-09-17T03:58:13.208294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:58.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211601+0000","last_change":"2024-09-17T03:58:12.190947+0000","last_active":"2024-09-17T03:58:13.211601+0000","last_peered":"2024-09-17T03:58:13.211601+0000","last_clean":"2024-09-17T03:58:13.211601+0000","last_became_active":"2024-09-17T03:58:12.189957+0000","last_became_peered":"2024-09-17T03:58:12.189957+0000","last_unstale":"2024-09-17T03:58:13.211601+0000","last_undegraded":"2024-09-17T03:58:13.211601+0000","last_fullsized":"2024-09-17T03:58:13.211601+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:12.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195391+0000","last_change":"2024-09-17T03:58:13.200333+0000","last_active":"2024-09-17T03:58:14.195391+0000","last_peered":"2024-09-17T03:58:14.195391+0000","last_clean":"2024-09-17T03:58:14.195391+0000","last_became_active":"2024-09-17T03:58:13.199867+0000","last_became_peered":"2024-09-17T03:58:13.199867+0000","last_unstale":"2024-09-17T03:58:14.195391+0000","last_undegraded":"2024-09-17T03:58:14.195391+0000","last_fullsized":"2024-09-17T03:58:14.195391+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:19.472713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193481+0000","last_change":"2024-09-17T03:58:12.190669+0000","last_active":"2024-09-17T03:58:14.193481+0000","last_peered":"2024-09-17T03:58:14.193481+0000","last_clean":"2024-09-17T03:58:14.193481+0000","last_became_active":"2024-09-17T03:58:12.189905+0000","last_became_peered":"2024-09-17T03:58:12.189905+0000","last_unstale":"2024-09-17T03:58:14.193481+0000","last_undegraded":"2024-09-17T03:58:14.193481+0000","last_fullsized":"2024-09-17T03:58:14.193481+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:39.303511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194960+0000","last_change":"2024-09-17T03:58:13.186201+0000","last_active":"2024-09-17T03:58:13.194960+0000","last_peered":"2024-09-17T03:58:13.194960+0000","last_clean":"2024-09-17T03:58:13.194960+0000","last_became_active":"2024-09-17T03:58:13.185914+0000","last_became_peered":"2024-09-17T03:58:13.185914+0000","last_unstale":"2024-09-17T03:58:13.194960+0000","last_undegraded":"2024-09-17T03:58:13.194960+0000","last_fullsized":"2024-09-17T03:58:13.194960+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:55.317170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.189775+0000","last_change":"2024-09-17T03:58:12.182011+0000","last_active":"2024-09-17T03:58:13.189775+0000","last_peered":"2024-09-17T03:58:13.189775+0000","last_clean":"2024-09-17T03:58:13.189775+0000","last_became_active":"2024-09-17T03:58:12.180843+0000","last_became_peered":"2024-09-17T03:58:12.180843+0000","last_unstale":"2024-09-17T03:58:13.189775+0000","last_undegraded":"2024-09-17T03:58:13.189775+0000","last_fullsized":"2024-09-17T03:58:13.189775+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:46.265281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193571+0000","last_change":"2024-09-17T03:58:13.193865+0000","last_active":"2024-09-17T03:58:14.193571+0000","last_peered":"2024-09-17T03:58:14.193571+0000","last_clean":"2024-09-17T03:58:14.193571+0000","last_became_active":"2024-09-17T03:58:13.192337+0000","last_became_peered":"2024-09-17T03:58:13.192337+0000","last_unstale":"2024-09-17T03:58:14.193571+0000","last_undegraded":"2024-09-17T03:58:14.193571+0000","last_fullsized":"2024-09-17T03:58:14.193571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:09.119456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193245+0000","last_change":"2024-09-17T03:58:12.187713+0000","last_active":"2024-09-17T03:58:14.193245+0000","last_peered":"2024-09-17T03:58:14.193245+0000","last_clean":"2024-09-17T03:58:14.193245+0000","last_became_active":"2024-09-17T03:58:12.187296+0000","last_became_peered":"2024-09-17T03:58:12.187296+0000","last_unstale":"2024-09-17T03:58:14.193245+0000","last_undegraded":"2024-09-17T03:58:14.193245+0000","last_fullsized":"2024-09-17T03:58:14.193245+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:32.836934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194111+0000","last_change":"2024-09-17T03:58:13.203028+0000","last_active":"2024-09-17T03:58:14.194111+0000","last_peered":"2024-09-17T03:58:14.194111+0000","last_clean":"2024-09-17T03:58:14.194111+0000","last_became_active":"2024-09-17T03:58:13.199385+0000","last_became_peered":"2024-09-17T03:58:13.199385+0000","last_unstale":"2024-09-17T03:58:14.194111+0000","last_undegraded":"2024-09-17T03:58:14.194111+0000","last_fullsized":"2024-09-17T03:58:14.194111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:58.554002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.177442+0000","last_change":"2024-09-17T03:58:12.181916+0000","last_active":"2024-09-17T03:58:13.177442+0000","last_peered":"2024-09-17T03:58:13.177442+0000","last_clean":"2024-09-17T03:58:13.177442+0000","last_became_active":"2024-09-17T03:58:12.181608+0000","last_became_peered":"2024-09-17T03:58:12.181608+0000","last_unstale":"2024-09-17T03:58:13.177442+0000","last_undegraded":"2024-09-17T03:58:13.177442+0000","last_fullsized":"2024-09-17T03:58:13.177442+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:36.142435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221504+0000","last_change":"2024-09-17T03:58:13.216490+0000","last_active":"2024-09-17T03:58:13.221504+0000","last_peered":"2024-09-17T03:58:13.221504+0000","last_clean":"2024-09-17T03:58:13.221504+0000","last_became_active":"2024-09-17T03:58:13.215403+0000","last_became_peered":"2024-09-17T03:58:13.215403+0000","last_unstale":"2024-09-17T03:58:13.221504+0000","last_undegraded":"2024-09-17T03:58:13.221504+0000","last_fullsized":"2024-09-17T03:58:13.221504+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:47.683195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203799+0000","last_change":"2024-09-17T03:58:12.189856+0000","last_active":"2024-09-17T03:58:13.203799+0000","last_peered":"2024-09-17T03:58:13.203799+0000","last_clean":"2024-09-17T03:58:13.203799+0000","last_became_active":"2024-09-17T03:58:12.189459+0000","last_became_peered":"2024-09-17T03:58:12.189459+0000","last_unstale":"2024-09-17T03:58:13.203799+0000","last_undegraded":"2024-09-17T03:58:13.203799+0000","last_fullsized":"2024-09-17T03:58:13.203799+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:07.684708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.212020+0000","last_change":"2024-09-17T03:58:13.209041+0000","last_active":"2024-09-17T03:58:13.212020+0000","last_peered":"2024-09-17T03:58:13.212020+0000","last_clean":"2024-09-17T03:58:13.212020+0000","last_became_active":"2024-09-17T03:58:13.208747+0000","last_became_peered":"2024-09-17T03:58:13.208747+0000","last_unstale":"2024-09-17T03:58:13.212020+0000","last_undegraded":"2024-09-17T03:58:13.212020+0000","last_fullsized":"2024-09-17T03:58:13.212020+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:00.217845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":19,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217870+0000","last_change":"2024-09-17T03:58:12.187293+0000","last_active":"2024-09-17T03:58:13.217870+0000","last_peered":"2024-09-17T03:58:13.217870+0000","last_clean":"2024-09-17T03:58:13.217870+0000","last_became_active":"2024-09-17T03:58:12.186432+0000","last_became_peered":"2024-09-17T03:58:12.186432+0000","last_unstale":"2024-09-17T03:58:13.217870+0000","last_undegraded":"2024-09-17T03:58:13.217870+0000","last_fullsized":"2024-09-17T03:58:13.217870+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:41.484361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.215831+0000","last_change":"2024-09-17T03:58:13.207804+0000","last_active":"2024-09-17T03:58:13.215831+0000","last_peered":"2024-09-17T03:58:13.215831+0000","last_clean":"2024-09-17T03:58:13.215831+0000","last_became_active":"2024-09-17T03:58:13.207095+0000","last_became_peered":"2024-09-17T03:58:13.207095+0000","last_unstale":"2024-09-17T03:58:13.215831+0000","last_undegraded":"2024-09-17T03:58:13.215831+0000","last_fullsized":"2024-09-17T03:58:13.215831+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:48.837854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211112+0000","last_change":"2024-09-17T03:58:12.183495+0000","last_active":"2024-09-17T03:58:13.211112+0000","last_peered":"2024-09-17T03:58:13.211112+0000","last_clean":"2024-09-17T03:58:13.211112+0000","last_became_active":"2024-09-17T03:58:12.183185+0000","last_became_peered":"2024-09-17T03:58:12.183185+0000","last_unstale":"2024-09-17T03:58:13.211112+0000","last_undegraded":"2024-09-17T03:58:13.211112+0000","last_fullsized":"2024-09-17T03:58:13.211112+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:28.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203427+0000","last_change":"2024-09-17T03:58:13.196869+0000","last_active":"2024-09-17T03:58:13.203427+0000","last_peered":"2024-09-17T03:58:13.203427+0000","last_clean":"2024-09-17T03:58:13.203427+0000","last_became_active":"2024-09-17T03:58:13.194671+0000","last_became_peered":"2024-09-17T03:58:13.194671+0000","last_unstale":"2024-09-17T03:58:13.203427+0000","last_undegraded":"2024-09-17T03:58:13.203427+0000","last_fullsized":"2024-09-17T03:58:13.203427+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:41.493738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.193682+0000","last_change":"2024-09-17T03:58:12.181431+0000","last_active":"2024-09-17T03:58:13.193682+0000","last_peered":"2024-09-17T03:58:13.193682+0000","last_clean":"2024-09-17T03:58:13.193682+0000","last_became_active":"2024-09-17T03:58:12.181091+0000","last_became_peered":"2024-09-17T03:58:12.181091+0000","last_unstale":"2024-09-17T03:58:13.193682+0000","last_undegraded":"2024-09-17T03:58:13.193682+0000","last_fullsized":"2024-09-17T03:58:13.193682+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:02.535381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193852+0000","last_change":"2024-09-17T03:58:13.208519+0000","last_active":"2024-09-17T03:58:14.193852+0000","last_peered":"2024-09-17T03:58:14.193852+0000","last_clean":"2024-09-17T03:58:14.193852+0000","last_became_active":"2024-09-17T03:58:13.207999+0000","last_became_peered":"2024-09-17T03:58:13.207999+0000","last_unstale":"2024-09-17T03:58:14.193852+0000","last_undegraded":"2024-09-17T03:58:14.193852+0000","last_fullsized":"2024-09-17T03:58:14.193852+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:18.592514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193756+0000","last_change":"2024-09-17T03:58:12.178728+0000","last_active":"2024-09-17T03:58:14.193756+0000","last_peered":"2024-09-17T03:58:14.193756+0000","last_clean":"2024-09-17T03:58:14.193756+0000","last_became_active":"2024-09-17T03:58:12.178119+0000","last_became_peered":"2024-09-17T03:58:12.178119+0000","last_unstale":"2024-09-17T03:58:14.193756+0000","last_undegraded":"2024-09-17T03:58:14.193756+0000","last_fullsized":"2024-09-17T03:58:14.193756+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:56:14.350160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210226+0000","last_change":"2024-09-17T03:58:13.200404+0000","last_active":"2024-09-17T03:58:13.210226+0000","last_peered":"2024-09-17T03:58:13.210226+0000","last_clean":"2024-09-17T03:58:13.210226+0000","last_became_active":"2024-09-17T03:58:13.200049+0000","last_became_peered":"2024-09-17T03:58:13.200049+0000","last_unstale":"2024-09-17T03:58:13.210226+0000","last_undegraded":"2024-09-17T03:58:13.210226+0000","last_fullsized":"2024-09-17T03:58:13.210226+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:21.313582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":21,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218155+0000","last_change":"2024-09-17T03:58:12.188462+0000","last_active":"2024-09-17T03:58:13.218155+0000","last_peered":"2024-09-17T03:58:13.218155+0000","last_clean":"2024-09-17T03:58:13.218155+0000","last_became_active":"2024-09-17T03:58:12.188155+0000","last_became_peered":"2024-09-17T03:58:12.188155+0000","last_unstale":"2024-09-17T03:58:13.218155+0000","last_undegraded":"2024-09-17T03:58:13.218155+0000","last_fullsized":"2024-09-17T03:58:13.218155+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:22.173349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221298+0000","last_change":"2024-09-17T03:58:13.214245+0000","last_active":"2024-09-17T03:58:13.221298+0000","last_peered":"2024-09-17T03:58:13.221298+0000","last_clean":"2024-09-17T03:58:13.221298+0000","last_became_active":"2024-09-17T03:58:13.212075+0000","last_became_peered":"2024-09-17T03:58:13.212075+0000","last_unstale":"2024-09-17T03:58:13.221298+0000","last_undegraded":"2024-09-17T03:58:13.221298+0000","last_fullsized":"2024-09-17T03:58:13.221298+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:00.392907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193650+0000","last_change":"2024-09-17T03:58:12.184655+0000","last_active":"2024-09-17T03:58:14.193650+0000","last_peered":"2024-09-17T03:58:14.193650+0000","last_clean":"2024-09-17T03:58:14.193650+0000","last_became_active":"2024-09-17T03:58:12.184308+0000","last_became_peered":"2024-09-17T03:58:12.184308+0000","last_unstale":"2024-09-17T03:58:14.193650+0000","last_undegraded":"2024-09-17T03:58:14.193650+0000","last_fullsized":"2024-09-17T03:58:14.193650+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:38.876264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214452+0000","last_change":"2024-09-17T03:58:13.212198+0000","last_active":"2024-09-17T03:58:13.214452+0000","last_peered":"2024-09-17T03:58:13.214452+0000","last_clean":"2024-09-17T03:58:13.214452+0000","last_became_active":"2024-09-17T03:58:13.211866+0000","last_became_peered":"2024-09-17T03:58:13.211866+0000","last_unstale":"2024-09-17T03:58:13.214452+0000","last_undegraded":"2024-09-17T03:58:13.214452+0000","last_fullsized":"2024-09-17T03:58:13.214452+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:14.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.204760+0000","last_change":"2024-09-17T03:58:12.185808+0000","last_active":"2024-09-17T03:58:13.204760+0000","last_peered":"2024-09-17T03:58:13.204760+0000","last_clean":"2024-09-17T03:58:13.204760+0000","last_became_active":"2024-09-17T03:58:12.185447+0000","last_became_peered":"2024-09-17T03:58:12.185447+0000","last_unstale":"2024-09-17T03:58:13.204760+0000","last_undegraded":"2024-09-17T03:58:13.204760+0000","last_fullsized":"2024-09-17T03:58:13.204760+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:16.600004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210561+0000","last_change":"2024-09-17T03:58:13.200760+0000","last_active":"2024-09-17T03:58:13.210561+0000","last_peered":"2024-09-17T03:58:13.210561+0000","last_clean":"2024-09-17T03:58:13.210561+0000","last_became_active":"2024-09-17T03:58:13.200447+0000","last_became_peered":"2024-09-17T03:58:13.200447+0000","last_unstale":"2024-09-17T03:58:13.210561+0000","last_undegraded":"2024-09-17T03:58:13.210561+0000","last_fullsized":"2024-09-17T03:58:13.210561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:57:05.971575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.212750+0000","last_change":"2024-09-17T03:58:12.190248+0000","last_active":"2024-09-17T03:58:13.212750+0000","last_peered":"2024-09-17T03:58:13.212750+0000","last_clean":"2024-09-17T03:58:13.212750+0000","last_became_active":"2024-09-17T03:58:12.189174+0000","last_became_peered":"2024-09-17T03:58:12.189174+0000","last_unstale":"2024-09-17T03:58:13.212750+0000","last_undegraded":"2024-09-17T03:58:13.212750+0000","last_fullsized":"2024-09-17T03:58:13.212750+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:49.176315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194391+0000","last_change":"2024-09-17T03:58:13.203054+0000","last_active":"2024-09-17T03:58:14.194391+0000","last_peered":"2024-09-17T03:58:14.194391+0000","last_clean":"2024-09-17T03:58:14.194391+0000","last_became_active":"2024-09-17T03:58:13.202746+0000","last_became_peered":"2024-09-17T03:58:13.202746+0000","last_unstale":"2024-09-17T03:58:14.194391+0000","last_undegraded":"2024-09-17T03:58:14.194391+0000","last_fullsized":"2024-09-17T03:58:14.194391+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:17.180677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.176283+0000","last_change":"2024-09-17T03:58:12.185416+0000","last_active":"2024-09-17T03:58:13.176283+0000","last_peered":"2024-09-17T03:58:13.176283+0000","last_clean":"2024-09-17T03:58:13.176283+0000","last_became_active":"2024-09-17T03:58:12.185123+0000","last_became_peered":"2024-09-17T03:58:12.185123+0000","last_unstale":"2024-09-17T03:58:13.176283+0000","last_undegraded":"2024-09-17T03:58:13.176283+0000","last_fullsized":"2024-09-17T03:58:13.176283+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:09.325907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195014+0000","last_change":"2024-09-17T03:58:13.202130+0000","last_active":"2024-09-17T03:58:14.195014+0000","last_peered":"2024-09-17T03:58:14.195014+0000","last_clean":"2024-09-17T03:58:14.195014+0000","last_became_active":"2024-09-17T03:58:13.201756+0000","last_became_peered":"2024-09-17T03:58:13.201756+0000","last_unstale":"2024-09-17T03:58:14.195014+0000","last_undegraded":"2024-09-17T03:58:14.195014+0000","last_fullsized":"2024-09-17T03:58:14.195014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:58.838185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.205731+0000","last_change":"2024-09-17T03:58:12.185147+0000","last_active":"2024-09-17T03:58:13.205731+0000","last_peered":"2024-09-17T03:58:13.205731+0000","last_clean":"2024-09-17T03:58:13.205731+0000","last_became_active":"2024-09-17T03:58:12.184785+0000","last_became_peered":"2024-09-17T03:58:12.184785+0000","last_unstale":"2024-09-17T03:58:13.205731+0000","last_undegraded":"2024-09-17T03:58:13.205731+0000","last_fullsized":"2024-09-17T03:58:13.205731+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:20.814706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214131+0000","last_change":"2024-09-17T03:58:13.203540+0000","last_active":"2024-09-17T03:58:13.214131+0000","last_peered":"2024-09-17T03:58:13.214131+0000","last_clean":"2024-09-17T03:58:13.214131+0000","last_became_active":"2024-09-17T03:58:13.200892+0000","last_became_peered":"2024-09-17T03:58:13.200892+0000","last_unstale":"2024-09-17T03:58:13.214131+0000","last_undegraded":"2024-09-17T03:58:13.214131+0000","last_fullsized":"2024-09-17T03:58:13.214131+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:17.706429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.196607+0000","last_change":"2024-09-17T03:58:12.191693+0000","last_active":"2024-09-17T03:58:13.196607+0000","last_peered":"2024-09-17T03:58:13.196607+0000","last_clean":"2024-09-17T03:58:13.196607+0000","last_became_active":"2024-09-17T03:58:12.190898+0000","last_became_peered":"2024-09-17T03:58:12.190898+0000","last_unstale":"2024-09-17T03:58:13.196607+0000","last_undegraded":"2024-09-17T03:58:13.196607+0000","last_fullsized":"2024-09-17T03:58:13.196607+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:43.769495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193347+0000","last_change":"2024-09-17T03:58:13.198845+0000","last_active":"2024-09-17T03:58:14.193347+0000","last_peered":"2024-09-17T03:58:14.193347+0000","last_clean":"2024-09-17T03:58:14.193347+0000","last_became_active":"2024-09-17T03:58:13.198516+0000","last_became_peered":"2024-09-17T03:58:13.198516+0000","last_unstale":"2024-09-17T03:58:14.193347+0000","last_undegraded":"2024-09-17T03:58:14.193347+0000","last_fullsized":"2024-09-17T03:58:14.193347+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:22.267943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202927+0000","last_change":"2024-09-17T03:58:12.192937+0000","last_active":"2024-09-17T03:58:13.202927+0000","last_peered":"2024-09-17T03:58:13.202927+0000","last_clean":"2024-09-17T03:58:13.202927+0000","last_became_active":"2024-09-17T03:58:12.192315+0000","last_became_peered":"2024-09-17T03:58:12.192315+0000","last_unstale":"2024-09-17T03:58:13.202927+0000","last_undegraded":"2024-09-17T03:58:13.202927+0000","last_fullsized":"2024-09-17T03:58:13.202927+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:57.830280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213802+0000","last_change":"2024-09-17T03:58:13.208665+0000","last_active":"2024-09-17T03:58:13.213802+0000","last_peered":"2024-09-17T03:58:13.213802+0000","last_clean":"2024-09-17T03:58:13.213802+0000","last_became_active":"2024-09-17T03:58:13.207794+0000","last_became_peered":"2024-09-17T03:58:13.207794+0000","last_unstale":"2024-09-17T03:58:13.213802+0000","last_undegraded":"2024-09-17T03:58:13.213802+0000","last_fullsized":"2024-09-17T03:58:13.213802+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:20.656450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194661+0000","last_change":"2024-09-17T03:58:12.184241+0000","last_active":"2024-09-17T03:58:13.194661+0000","last_peered":"2024-09-17T03:58:13.194661+0000","last_clean":"2024-09-17T03:58:13.194661+0000","last_became_active":"2024-09-17T03:58:12.183861+0000","last_became_peered":"2024-09-17T03:58:12.183861+0000","last_unstale":"2024-09-17T03:58:13.194661+0000","last_undegraded":"2024-09-17T03:58:13.194661+0000","last_fullsized":"2024-09-17T03:58:13.194661+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:27.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194182+0000","last_change":"2024-09-17T03:58:13.210907+0000","last_active":"2024-09-17T03:58:14.194182+0000","last_peered":"2024-09-17T03:58:14.194182+0000","last_clean":"2024-09-17T03:58:14.194182+0000","last_became_active":"2024-09-17T03:58:13.210573+0000","last_became_peered":"2024-09-17T03:58:13.210573+0000","last_unstale":"2024-09-17T03:58:14.194182+0000","last_undegraded":"2024-09-17T03:58:14.194182+0000","last_fullsized":"2024-09-17T03:58:14.194182+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:01.071588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.216971+0000","last_change":"2024-09-17T03:58:12.191164+0000","last_active":"2024-09-17T03:58:13.216971+0000","last_peered":"2024-09-17T03:58:13.216971+0000","last_clean":"2024-09-17T03:58:13.216971+0000","last_became_active":"2024-09-17T03:58:12.190803+0000","last_became_peered":"2024-09-17T03:58:12.190803+0000","last_unstale":"2024-09-17T03:58:13.216971+0000","last_undegraded":"2024-09-17T03:58:13.216971+0000","last_fullsized":"2024-09-17T03:58:13.216971+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:18.464808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210281+0000","last_change":"2024-09-17T03:58:13.200728+0000","last_active":"2024-09-17T03:58:13.210281+0000","last_peered":"2024-09-17T03:58:13.210281+0000","last_clean":"2024-09-17T03:58:13.210281+0000","last_became_active":"2024-09-17T03:58:13.200366+0000","last_became_peered":"2024-09-17T03:58:13.200366+0000","last_unstale":"2024-09-17T03:58:13.210281+0000","last_undegraded":"2024-09-17T03:58:13.210281+0000","last_fullsized":"2024-09-17T03:58:13.210281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:43.282925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194564+0000","last_change":"2024-09-17T03:58:12.185932+0000","last_active":"2024-09-17T03:58:14.194564+0000","last_peered":"2024-09-17T03:58:14.194564+0000","last_clean":"2024-09-17T03:58:14.194564+0000","last_became_active":"2024-09-17T03:58:12.183505+0000","last_became_peered":"2024-09-17T03:58:12.183505+0000","last_unstale":"2024-09-17T03:58:14.194564+0000","last_undegraded":"2024-09-17T03:58:14.194564+0000","last_fullsized":"2024-09-17T03:58:14.194564+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:00.381361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203348+0000","last_change":"2024-09-17T03:58:13.199488+0000","last_active":"2024-09-17T03:58:13.203348+0000","last_peered":"2024-09-17T03:58:13.203348+0000","last_clean":"2024-09-17T03:58:13.203348+0000","last_became_active":"2024-09-17T03:58:13.199176+0000","last_became_peered":"2024-09-17T03:58:13.199176+0000","last_unstale":"2024-09-17T03:58:13.203348+0000","last_undegraded":"2024-09-17T03:58:13.203348+0000","last_fullsized":"2024-09-17T03:58:13.203348+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:50.942453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194630+0000","last_change":"2024-09-17T03:58:12.187122+0000","last_active":"2024-09-17T03:58:14.194630+0000","last_peered":"2024-09-17T03:58:14.194630+0000","last_clean":"2024-09-17T03:58:14.194630+0000","last_became_active":"2024-09-17T03:58:12.186019+0000","last_became_peered":"2024-09-17T03:58:12.186019+0000","last_unstale":"2024-09-17T03:58:14.194630+0000","last_undegraded":"2024-09-17T03:58:14.194630+0000","last_fullsized":"2024-09-17T03:58:14.194630+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:25.273252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194517+0000","last_change":"2024-09-17T03:58:13.197369+0000","last_active":"2024-09-17T03:58:14.194517+0000","last_peered":"2024-09-17T03:58:14.194517+0000","last_clean":"2024-09-17T03:58:14.194517+0000","last_became_active":"2024-09-17T03:58:13.196755+0000","last_became_peered":"2024-09-17T03:58:13.196755+0000","last_unstale":"2024-09-17T03:58:14.194517+0000","last_undegraded":"2024-09-17T03:58:14.194517+0000","last_fullsized":"2024-09-17T03:58:14.194517+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:08.405776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194289+0000","last_change":"2024-09-17T03:58:12.185096+0000","last_active":"2024-09-17T03:58:14.194289+0000","last_peered":"2024-09-17T03:58:14.194289+0000","last_clean":"2024-09-17T03:58:14.194289+0000","last_became_active":"2024-09-17T03:58:12.184809+0000","last_became_peered":"2024-09-17T03:58:12.184809+0000","last_unstale":"2024-09-17T03:58:14.194289+0000","last_undegraded":"2024-09-17T03:58:14.194289+0000","last_fullsized":"2024-09-17T03:58:14.194289+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:21:45.665906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206446+0000","last_change":"2024-09-17T03:58:13.202016+0000","last_active":"2024-09-17T03:58:13.206446+0000","last_peered":"2024-09-17T03:58:13.206446+0000","last_clean":"2024-09-17T03:58:13.206446+0000","last_became_active":"2024-09-17T03:58:13.201551+0000","last_became_peered":"2024-09-17T03:58:13.201551+0000","last_unstale":"2024-09-17T03:58:13.206446+0000","last_undegraded":"2024-09-17T03:58:13.206446+0000","last_fullsized":"2024-09-17T03:58:13.206446+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:54.860771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.188639+0000","last_change":"2024-09-17T03:58:12.184231+0000","last_active":"2024-09-17T03:58:13.188639+0000","last_peered":"2024-09-17T03:58:13.188639+0000","last_clean":"2024-09-17T03:58:13.188639+0000","last_became_active":"2024-09-17T03:58:12.183961+0000","last_became_peered":"2024-09-17T03:58:12.183961+0000","last_unstale":"2024-09-17T03:58:13.188639+0000","last_undegraded":"2024-09-17T03:58:13.188639+0000","last_fullsized":"2024-09-17T03:58:13.188639+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:42.110947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194294+0000","last_change":"2024-09-17T03:58:13.201893+0000","last_active":"2024-09-17T03:58:14.194294+0000","last_peered":"2024-09-17T03:58:14.194294+0000","last_clean":"2024-09-17T03:58:14.194294+0000","last_became_active":"2024-09-17T03:58:13.201569+0000","last_became_peered":"2024-09-17T03:58:13.201569+0000","last_unstale":"2024-09-17T03:58:14.194294+0000","last_undegraded":"2024-09-17T03:58:14.194294+0000","last_fullsized":"2024-09-17T03:58:14.194294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:34.930836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.201465+0000","last_change":"2024-09-17T03:58:12.190851+0000","last_active":"2024-09-17T03:58:13.201465+0000","last_peered":"2024-09-17T03:58:13.201465+0000","last_clean":"2024-09-17T03:58:13.201465+0000","last_became_active":"2024-09-17T03:58:12.190544+0000","last_became_peered":"2024-09-17T03:58:12.190544+0000","last_unstale":"2024-09-17T03:58:13.201465+0000","last_undegraded":"2024-09-17T03:58:13.201465+0000","last_fullsized":"2024-09-17T03:58:13.201465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:47.875694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218273+0000","last_change":"2024-09-17T03:58:13.213318+0000","last_active":"2024-09-17T03:58:13.218273+0000","last_peered":"2024-09-17T03:58:13.218273+0000","last_clean":"2024-09-17T03:58:13.218273+0000","last_became_active":"2024-09-17T03:58:13.212980+0000","last_became_peered":"2024-09-17T03:58:13.212980+0000","last_unstale":"2024-09-17T03:58:13.218273+0000","last_undegraded":"2024-09-17T03:58:13.218273+0000","last_fullsized":"2024-09-17T03:58:13.218273+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:55.966116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194622+0000","last_change":"2024-09-17T03:58:12.182076+0000","last_active":"2024-09-17T03:58:13.194622+0000","last_peered":"2024-09-17T03:58:13.194622+0000","last_clean":"2024-09-17T03:58:13.194622+0000","last_became_active":"2024-09-17T03:58:12.181735+0000","last_became_peered":"2024-09-17T03:58:12.181735+0000","last_unstale":"2024-09-17T03:58:13.194622+0000","last_undegraded":"2024-09-17T03:58:13.194622+0000","last_fullsized":"2024-09-17T03:58:13.194622+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:40.387720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193843+0000","last_change":"2024-09-17T03:58:13.197885+0000","last_active":"2024-09-17T03:58:14.193843+0000","last_peered":"2024-09-17T03:58:14.193843+0000","last_clean":"2024-09-17T03:58:14.193843+0000","last_became_active":"2024-09-17T03:58:13.197577+0000","last_became_peered":"2024-09-17T03:58:13.197577+0000","last_unstale":"2024-09-17T03:58:14.193843+0000","last_undegraded":"2024-09-17T03:58:14.193843+0000","last_fullsized":"2024-09-17T03:58:14.193843+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:25.332392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.175736+0000","last_change":"2024-09-17T03:58:12.188614+0000","last_active":"2024-09-17T03:58:13.175736+0000","last_peered":"2024-09-17T03:58:13.175736+0000","last_clean":"2024-09-17T03:58:13.175736+0000","last_became_active":"2024-09-17T03:58:12.188308+0000","last_became_peered":"2024-09-17T03:58:12.188308+0000","last_unstale":"2024-09-17T03:58:13.175736+0000","last_undegraded":"2024-09-17T03:58:13.175736+0000","last_fullsized":"2024-09-17T03:58:13.175736+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.875180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213963+0000","last_change":"2024-09-17T03:58:13.208342+0000","last_active":"2024-09-17T03:58:13.213963+0000","last_peered":"2024-09-17T03:58:13.213963+0000","last_clean":"2024-09-17T03:58:13.213963+0000","last_became_active":"2024-09-17T03:58:13.207033+0000","last_became_peered":"2024-09-17T03:58:13.207033+0000","last_unstale":"2024-09-17T03:58:13.213963+0000","last_undegraded":"2024-09-17T03:58:13.213963+0000","last_fullsized":"2024-09-17T03:58:13.213963+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:18:19.772741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214358+0000","last_change":"2024-09-17T03:58:12.184834+0000","last_active":"2024-09-17T03:58:13.214358+0000","last_peered":"2024-09-17T03:58:13.214358+0000","last_clean":"2024-09-17T03:58:13.214358+0000","last_became_active":"2024-09-17T03:58:12.184495+0000","last_became_peered":"2024-09-17T03:58:12.184495+0000","last_unstale":"2024-09-17T03:58:13.214358+0000","last_undegraded":"2024-09-17T03:58:13.214358+0000","last_fullsized":"2024-09-17T03:58:13.214358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:01.983852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":325,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.201336+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:58:13.201336+0000","last_peered":"2024-09-17T03:58:13.201336+0000","last_clean":"2024-09-17T03:58:13.201336+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:58:13.201336+0000","last_undegraded":"2024-09-17T03:58:13.201336+0000","last_fullsized":"2024-09-17T03:58:13.201336+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509548,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37620,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334220,"statfs":{"total":96636764160,"available":96598241280,"internally_reserved":0,"allocated":2322432,"data_stored":1338896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1724416,"data_stored":747898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509550,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1720320,"data_stored":747876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":1724416,"data_stored":747966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509550,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1736704,"data_stored":749558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509550,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1736704,"data_stored":754758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41180,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330660,"statfs":{"total":96636764160,"available":96594595840,"internally_reserved":0,"allocated":1708032,"data_stored":741101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41756,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330084,"statfs":{"total":96636764160,"available":96594006016,"internally_reserved":0,"allocated":2301952,"data_stored":1331469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:58:17.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:58:17.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 -- 172.21.15.129:0/2920861152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f746c003a00 msgr2=0x7f746c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 --2- 172.21.15.129:0/2920861152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f746c003a00 0x7f746c003e70 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7478009b50 tx=0x7f746c005830 comp rx=0 tx=0).stop 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f7486800640 1 -- 172.21.15.129:0/2920861152 reap_dead start 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 -- 172.21.15.129:0/2920861152 shutdown_connections 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 -- 172.21.15.129:0/2920861152 >> 172.21.15.129:0/2920861152 conn(0x7f748806d410 msgr2=0x7f74880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 -- 172.21.15.129:0/2920861152 shutdown_connections 2024-09-17T03:58:17.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.877+0000 7f748d200640 1 -- 172.21.15.129:0/2920861152 wait complete. 2024-09-17T03:58:17.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.879+0000 7f748d200640 1 Processor -- start 2024-09-17T03:58:17.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.880+0000 7f748d200640 1 -- start start 2024-09-17T03:58:17.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f746c003a00 0x7f74881f8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74881f6130 0x7f748815a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f74880b3480 con 0x7f74880afd50 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f74880708e0 con 0x7f746c003a00 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f748d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f74880b3600 con 0x7f74881f6130 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f7486800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f7486800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57412/0 (socket says 172.21.15.129:57412) 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.881+0000 7f7486800640 1 -- 172.21.15.129:0/2051118496 learned_addr learned my addr 172.21.15.129:0/2051118496 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7485e00640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f746c003a00 0x7f74881f8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7485400640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74881f6130 0x7f748815a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 -- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74881f6130 msgr2=0x7f748815a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74881f6130 0x7f748815a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 -- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f746c003a00 msgr2=0x7f74881f8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f746c003a00 0x7f74881f8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 -- 172.21.15.129:0/2051118496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74881f56b0 con 0x7f74880afd50 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f7486800640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f74880ca840 tx=0x7f7480006710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748000ea10 con 0x7f74880afd50 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7480018070 con 0x7f74880afd50 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.882+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74881f9230 con 0x7f74880afd50 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.883+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7480011460 con 0x7f74880afd50 2024-09-17T03:58:17.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.883+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74881f9a20 con 0x7f74880afd50 2024-09-17T03:58:17.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.884+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74881f94b0 con 0x7f74880afd50 2024-09-17T03:58:17.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.885+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7480011600 con 0x7f74880afd50 2024-09-17T03:58:17.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.886+0000 7f747e000640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 0x7f746807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:17.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.886+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(140..140 src has 1..140) ==== 5738+0+0 (secure 0 0 0) 0x7f748009b4a0 con 0x7f74880afd50 2024-09-17T03:58:17.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.887+0000 7f7485e00640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 0x7f746807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:17.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.887+0000 7f7485e00640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 0x7f746807b0a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7478035000 tx=0x7f747802d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:17.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:17.890+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f74881f56b0 con 0x7f74880afd50 2024-09-17T03:58:18.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.071+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74880cbbf0 con 0x7f7468078b90 2024-09-17T03:58:18.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.076+0000 7f747e000640 1 -- 172.21.15.129:0/2051118496 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252472 (secure 0 0 0) 0x7f74880cbbf0 con 0x7f7468078b90 2024-09-17T03:58:18.078 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:18.079 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 msgr2=0x7f746807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 0x7f746807b0a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f7478035000 tx=0x7f747802d040 comp rx=0 tx=0).stop 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 msgr2=0x7f74880adbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f74880ca840 tx=0x7f7480006710 comp rx=0 tx=0).stop 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 shutdown_connections 2024-09-17T03:58:18.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7468078b90 0x7f746807b0a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74880afd50 0x7f74880adbd0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74881f6130 0x7f748815a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 --2- 172.21.15.129:0/2051118496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f746c003a00 0x7f74881f8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 >> 172.21.15.129:0/2051118496 conn(0x7f748806d410 msgr2=0x7f74880ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 shutdown_connections 2024-09-17T03:58:18.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.080+0000 7f748d200640 1 -- 172.21.15.129:0/2051118496 wait complete. 2024-09-17T03:58:18.102 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":400,"stamp":"2024-09-17T03:58:17.219112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":309916,"kb_used_data":14624,"kb_used_omap":12,"kb_used_meta":295091,"kb_avail":754664804,"statfs":{"total":773094113280,"available":772776759296,"internally_reserved":0,"allocated":14974976,"data_stored":7159522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":302173744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.055109"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195048+0000","last_change":"2024-09-17T03:58:13.210124+0000","last_active":"2024-09-17T03:58:14.195048+0000","last_peered":"2024-09-17T03:58:14.195048+0000","last_clean":"2024-09-17T03:58:14.195048+0000","last_became_active":"2024-09-17T03:58:13.209826+0000","last_became_peered":"2024-09-17T03:58:13.209826+0000","last_unstale":"2024-09-17T03:58:14.195048+0000","last_undegraded":"2024-09-17T03:58:14.195048+0000","last_fullsized":"2024-09-17T03:58:14.195048+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:17.883085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203371+0000","last_change":"2024-09-17T03:58:13.197182+0000","last_active":"2024-09-17T03:58:13.203371+0000","last_peered":"2024-09-17T03:58:13.203371+0000","last_clean":"2024-09-17T03:58:13.203371+0000","last_became_active":"2024-09-17T03:58:13.196886+0000","last_became_peered":"2024-09-17T03:58:13.196886+0000","last_unstale":"2024-09-17T03:58:13.203371+0000","last_undegraded":"2024-09-17T03:58:13.203371+0000","last_fullsized":"2024-09-17T03:58:13.203371+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:15.551727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202868+0000","last_change":"2024-09-17T03:58:13.198559+0000","last_active":"2024-09-17T03:58:13.202868+0000","last_peered":"2024-09-17T03:58:13.202868+0000","last_clean":"2024-09-17T03:58:13.202868+0000","last_became_active":"2024-09-17T03:58:13.197992+0000","last_became_peered":"2024-09-17T03:58:13.197992+0000","last_unstale":"2024-09-17T03:58:13.202868+0000","last_undegraded":"2024-09-17T03:58:13.202868+0000","last_fullsized":"2024-09-17T03:58:13.202868+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:42:01.710483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195216+0000","last_change":"2024-09-17T03:58:13.199007+0000","last_active":"2024-09-17T03:58:14.195216+0000","last_peered":"2024-09-17T03:58:14.195216+0000","last_clean":"2024-09-17T03:58:14.195216+0000","last_became_active":"2024-09-17T03:58:13.198682+0000","last_became_peered":"2024-09-17T03:58:13.198682+0000","last_unstale":"2024-09-17T03:58:14.195216+0000","last_undegraded":"2024-09-17T03:58:14.195216+0000","last_fullsized":"2024-09-17T03:58:14.195216+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:53.789318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193897+0000","last_change":"2024-09-17T03:58:13.210650+0000","last_active":"2024-09-17T03:58:14.193897+0000","last_peered":"2024-09-17T03:58:14.193897+0000","last_clean":"2024-09-17T03:58:14.193897+0000","last_became_active":"2024-09-17T03:58:13.210304+0000","last_became_peered":"2024-09-17T03:58:13.210304+0000","last_unstale":"2024-09-17T03:58:14.193897+0000","last_undegraded":"2024-09-17T03:58:14.193897+0000","last_fullsized":"2024-09-17T03:58:14.193897+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:19:27.323807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193911+0000","last_change":"2024-09-17T03:58:13.205950+0000","last_active":"2024-09-17T03:58:14.193911+0000","last_peered":"2024-09-17T03:58:14.193911+0000","last_clean":"2024-09-17T03:58:14.193911+0000","last_became_active":"2024-09-17T03:58:13.205112+0000","last_became_peered":"2024-09-17T03:58:13.205112+0000","last_unstale":"2024-09-17T03:58:14.193911+0000","last_undegraded":"2024-09-17T03:58:14.193911+0000","last_fullsized":"2024-09-17T03:58:14.193911+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:05.034977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213829+0000","last_change":"2024-09-17T03:58:13.208974+0000","last_active":"2024-09-17T03:58:13.213829+0000","last_peered":"2024-09-17T03:58:13.213829+0000","last_clean":"2024-09-17T03:58:13.213829+0000","last_became_active":"2024-09-17T03:58:13.208167+0000","last_became_peered":"2024-09-17T03:58:13.208167+0000","last_unstale":"2024-09-17T03:58:13.213829+0000","last_undegraded":"2024-09-17T03:58:13.213829+0000","last_fullsized":"2024-09-17T03:58:13.213829+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:12.378776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193840+0000","last_change":"2024-09-17T03:58:13.191928+0000","last_active":"2024-09-17T03:58:14.193840+0000","last_peered":"2024-09-17T03:58:14.193840+0000","last_clean":"2024-09-17T03:58:14.193840+0000","last_became_active":"2024-09-17T03:58:13.191555+0000","last_became_peered":"2024-09-17T03:58:13.191555+0000","last_unstale":"2024-09-17T03:58:14.193840+0000","last_undegraded":"2024-09-17T03:58:14.193840+0000","last_fullsized":"2024-09-17T03:58:14.193840+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:51:10.629162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.197765+0000","last_change":"2024-09-17T03:58:13.194603+0000","last_active":"2024-09-17T03:58:13.197765+0000","last_peered":"2024-09-17T03:58:13.197765+0000","last_clean":"2024-09-17T03:58:13.197765+0000","last_became_active":"2024-09-17T03:58:13.194324+0000","last_became_peered":"2024-09-17T03:58:13.194324+0000","last_unstale":"2024-09-17T03:58:13.197765+0000","last_undegraded":"2024-09-17T03:58:13.197765+0000","last_fullsized":"2024-09-17T03:58:13.197765+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:40.566895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.225062+0000","last_change":"2024-09-17T03:58:13.218976+0000","last_active":"2024-09-17T03:58:13.225062+0000","last_peered":"2024-09-17T03:58:13.225062+0000","last_clean":"2024-09-17T03:58:13.225062+0000","last_became_active":"2024-09-17T03:58:13.218566+0000","last_became_peered":"2024-09-17T03:58:13.218566+0000","last_unstale":"2024-09-17T03:58:13.225062+0000","last_undegraded":"2024-09-17T03:58:13.225062+0000","last_fullsized":"2024-09-17T03:58:13.225062+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:29.157380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194344+0000","last_change":"2024-09-17T03:58:13.197033+0000","last_active":"2024-09-17T03:58:14.194344+0000","last_peered":"2024-09-17T03:58:14.194344+0000","last_clean":"2024-09-17T03:58:14.194344+0000","last_became_active":"2024-09-17T03:58:13.196347+0000","last_became_peered":"2024-09-17T03:58:13.196347+0000","last_unstale":"2024-09-17T03:58:14.194344+0000","last_undegraded":"2024-09-17T03:58:14.194344+0000","last_fullsized":"2024-09-17T03:58:14.194344+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:57.150602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218179+0000","last_change":"2024-09-17T03:58:13.213175+0000","last_active":"2024-09-17T03:58:13.218179+0000","last_peered":"2024-09-17T03:58:13.218179+0000","last_clean":"2024-09-17T03:58:13.218179+0000","last_became_active":"2024-09-17T03:58:13.211551+0000","last_became_peered":"2024-09-17T03:58:13.211551+0000","last_unstale":"2024-09-17T03:58:13.218179+0000","last_undegraded":"2024-09-17T03:58:13.218179+0000","last_fullsized":"2024-09-17T03:58:13.218179+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:00.545295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194118+0000","last_change":"2024-09-17T03:58:13.209833+0000","last_active":"2024-09-17T03:58:14.194118+0000","last_peered":"2024-09-17T03:58:14.194118+0000","last_clean":"2024-09-17T03:58:14.194118+0000","last_became_active":"2024-09-17T03:58:13.209524+0000","last_became_peered":"2024-09-17T03:58:13.209524+0000","last_unstale":"2024-09-17T03:58:14.194118+0000","last_undegraded":"2024-09-17T03:58:14.194118+0000","last_fullsized":"2024-09-17T03:58:14.194118+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:18.519428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194749+0000","last_change":"2024-09-17T03:58:13.207886+0000","last_active":"2024-09-17T03:58:14.194749+0000","last_peered":"2024-09-17T03:58:14.194749+0000","last_clean":"2024-09-17T03:58:14.194749+0000","last_became_active":"2024-09-17T03:58:13.207566+0000","last_became_peered":"2024-09-17T03:58:13.207566+0000","last_unstale":"2024-09-17T03:58:14.194749+0000","last_undegraded":"2024-09-17T03:58:14.194749+0000","last_fullsized":"2024-09-17T03:58:14.194749+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:44.777400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221333+0000","last_change":"2024-09-17T03:58:13.214462+0000","last_active":"2024-09-17T03:58:13.221333+0000","last_peered":"2024-09-17T03:58:13.221333+0000","last_clean":"2024-09-17T03:58:13.221333+0000","last_became_active":"2024-09-17T03:58:13.213558+0000","last_became_peered":"2024-09-17T03:58:13.213558+0000","last_unstale":"2024-09-17T03:58:13.221333+0000","last_undegraded":"2024-09-17T03:58:13.221333+0000","last_fullsized":"2024-09-17T03:58:13.221333+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:17.721542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206889+0000","last_change":"2024-09-17T03:58:13.196823+0000","last_active":"2024-09-17T03:58:13.206889+0000","last_peered":"2024-09-17T03:58:13.206889+0000","last_clean":"2024-09-17T03:58:13.206889+0000","last_became_active":"2024-09-17T03:58:13.196229+0000","last_became_peered":"2024-09-17T03:58:13.196229+0000","last_unstale":"2024-09-17T03:58:13.206889+0000","last_undegraded":"2024-09-17T03:58:13.206889+0000","last_fullsized":"2024-09-17T03:58:13.206889+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:33.104699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221526+0000","last_change":"2024-09-17T03:58:13.216483+0000","last_active":"2024-09-17T03:58:13.221526+0000","last_peered":"2024-09-17T03:58:13.221526+0000","last_clean":"2024-09-17T03:58:13.221526+0000","last_became_active":"2024-09-17T03:58:13.215470+0000","last_became_peered":"2024-09-17T03:58:13.215470+0000","last_unstale":"2024-09-17T03:58:13.221526+0000","last_undegraded":"2024-09-17T03:58:13.221526+0000","last_fullsized":"2024-09-17T03:58:13.221526+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:45.587681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202663+0000","last_change":"2024-09-17T03:58:13.197076+0000","last_active":"2024-09-17T03:58:13.202663+0000","last_peered":"2024-09-17T03:58:13.202663+0000","last_clean":"2024-09-17T03:58:13.202663+0000","last_became_active":"2024-09-17T03:58:13.196769+0000","last_became_peered":"2024-09-17T03:58:13.196769+0000","last_unstale":"2024-09-17T03:58:13.202663+0000","last_undegraded":"2024-09-17T03:58:13.202663+0000","last_fullsized":"2024-09-17T03:58:13.202663+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:44:28.630755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194628+0000","last_change":"2024-09-17T03:58:13.201472+0000","last_active":"2024-09-17T03:58:14.194628+0000","last_peered":"2024-09-17T03:58:14.194628+0000","last_clean":"2024-09-17T03:58:14.194628+0000","last_became_active":"2024-09-17T03:58:13.201148+0000","last_became_peered":"2024-09-17T03:58:13.201148+0000","last_unstale":"2024-09-17T03:58:14.194628+0000","last_undegraded":"2024-09-17T03:58:14.194628+0000","last_fullsized":"2024-09-17T03:58:14.194628+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:31.684429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.204183+0000","last_change":"2024-09-17T03:58:13.200828+0000","last_active":"2024-09-17T03:58:13.204183+0000","last_peered":"2024-09-17T03:58:13.204183+0000","last_clean":"2024-09-17T03:58:13.204183+0000","last_became_active":"2024-09-17T03:58:13.200506+0000","last_became_peered":"2024-09-17T03:58:13.200506+0000","last_unstale":"2024-09-17T03:58:13.204183+0000","last_undegraded":"2024-09-17T03:58:13.204183+0000","last_fullsized":"2024-09-17T03:58:13.204183+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:27.960576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.225031+0000","last_change":"2024-09-17T03:58:13.218497+0000","last_active":"2024-09-17T03:58:13.225031+0000","last_peered":"2024-09-17T03:58:13.225031+0000","last_clean":"2024-09-17T03:58:13.225031+0000","last_became_active":"2024-09-17T03:58:13.218096+0000","last_became_peered":"2024-09-17T03:58:13.218096+0000","last_unstale":"2024-09-17T03:58:13.225031+0000","last_undegraded":"2024-09-17T03:58:13.225031+0000","last_fullsized":"2024-09-17T03:58:13.225031+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:51.038577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194582+0000","last_change":"2024-09-17T03:58:13.196674+0000","last_active":"2024-09-17T03:58:14.194582+0000","last_peered":"2024-09-17T03:58:14.194582+0000","last_clean":"2024-09-17T03:58:14.194582+0000","last_became_active":"2024-09-17T03:58:13.196343+0000","last_became_peered":"2024-09-17T03:58:13.196343+0000","last_unstale":"2024-09-17T03:58:14.194582+0000","last_undegraded":"2024-09-17T03:58:14.194582+0000","last_fullsized":"2024-09-17T03:58:14.194582+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:28.637873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194242+0000","last_change":"2024-09-17T03:58:13.200728+0000","last_active":"2024-09-17T03:58:14.194242+0000","last_peered":"2024-09-17T03:58:14.194242+0000","last_clean":"2024-09-17T03:58:14.194242+0000","last_became_active":"2024-09-17T03:58:13.200428+0000","last_became_peered":"2024-09-17T03:58:13.200428+0000","last_unstale":"2024-09-17T03:58:14.194242+0000","last_undegraded":"2024-09-17T03:58:14.194242+0000","last_fullsized":"2024-09-17T03:58:14.194242+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:43.294681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221441+0000","last_change":"2024-09-17T03:58:13.215991+0000","last_active":"2024-09-17T03:58:13.221441+0000","last_peered":"2024-09-17T03:58:13.221441+0000","last_clean":"2024-09-17T03:58:13.221441+0000","last_became_active":"2024-09-17T03:58:13.215679+0000","last_became_peered":"2024-09-17T03:58:13.215679+0000","last_unstale":"2024-09-17T03:58:13.221441+0000","last_undegraded":"2024-09-17T03:58:13.221441+0000","last_fullsized":"2024-09-17T03:58:13.221441+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:14.450014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211019+0000","last_change":"2024-09-17T03:58:13.205986+0000","last_active":"2024-09-17T03:58:13.211019+0000","last_peered":"2024-09-17T03:58:13.211019+0000","last_clean":"2024-09-17T03:58:13.211019+0000","last_became_active":"2024-09-17T03:58:13.205644+0000","last_became_peered":"2024-09-17T03:58:13.205644+0000","last_unstale":"2024-09-17T03:58:13.211019+0000","last_undegraded":"2024-09-17T03:58:13.211019+0000","last_fullsized":"2024-09-17T03:58:13.211019+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:12.588562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194669+0000","last_change":"2024-09-17T03:58:13.202774+0000","last_active":"2024-09-17T03:58:14.194669+0000","last_peered":"2024-09-17T03:58:14.194669+0000","last_clean":"2024-09-17T03:58:14.194669+0000","last_became_active":"2024-09-17T03:58:13.201333+0000","last_became_peered":"2024-09-17T03:58:13.201333+0000","last_unstale":"2024-09-17T03:58:14.194669+0000","last_undegraded":"2024-09-17T03:58:14.194669+0000","last_fullsized":"2024-09-17T03:58:14.194669+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:20.148226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211663+0000","last_change":"2024-09-17T03:58:13.202837+0000","last_active":"2024-09-17T03:58:13.211663+0000","last_peered":"2024-09-17T03:58:13.211663+0000","last_clean":"2024-09-17T03:58:13.211663+0000","last_became_active":"2024-09-17T03:58:13.202450+0000","last_became_peered":"2024-09-17T03:58:13.202450+0000","last_unstale":"2024-09-17T03:58:13.211663+0000","last_undegraded":"2024-09-17T03:58:13.211663+0000","last_fullsized":"2024-09-17T03:58:13.211663+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:48.686888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213937+0000","last_change":"2024-09-17T03:58:13.208000+0000","last_active":"2024-09-17T03:58:13.213937+0000","last_peered":"2024-09-17T03:58:13.213937+0000","last_clean":"2024-09-17T03:58:13.213937+0000","last_became_active":"2024-09-17T03:58:13.206625+0000","last_became_peered":"2024-09-17T03:58:13.206625+0000","last_unstale":"2024-09-17T03:58:13.213937+0000","last_undegraded":"2024-09-17T03:58:13.213937+0000","last_fullsized":"2024-09-17T03:58:13.213937+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:25.907710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193976+0000","last_change":"2024-09-17T03:58:13.203722+0000","last_active":"2024-09-17T03:58:14.193976+0000","last_peered":"2024-09-17T03:58:14.193976+0000","last_clean":"2024-09-17T03:58:14.193976+0000","last_became_active":"2024-09-17T03:58:13.200741+0000","last_became_peered":"2024-09-17T03:58:13.200741+0000","last_unstale":"2024-09-17T03:58:14.193976+0000","last_undegraded":"2024-09-17T03:58:14.193976+0000","last_fullsized":"2024-09-17T03:58:14.193976+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:22:40.282731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194834+0000","last_change":"2024-09-17T03:58:13.202191+0000","last_active":"2024-09-17T03:58:14.194834+0000","last_peered":"2024-09-17T03:58:14.194834+0000","last_clean":"2024-09-17T03:58:14.194834+0000","last_became_active":"2024-09-17T03:58:13.201859+0000","last_became_peered":"2024-09-17T03:58:13.201859+0000","last_unstale":"2024-09-17T03:58:14.194834+0000","last_undegraded":"2024-09-17T03:58:14.194834+0000","last_fullsized":"2024-09-17T03:58:14.194834+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:48.883599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206569+0000","last_change":"2024-09-17T03:58:13.202176+0000","last_active":"2024-09-17T03:58:13.206569+0000","last_peered":"2024-09-17T03:58:13.206569+0000","last_clean":"2024-09-17T03:58:13.206569+0000","last_became_active":"2024-09-17T03:58:13.201890+0000","last_became_peered":"2024-09-17T03:58:13.201890+0000","last_unstale":"2024-09-17T03:58:13.206569+0000","last_undegraded":"2024-09-17T03:58:13.206569+0000","last_fullsized":"2024-09-17T03:58:13.206569+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:04.144165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206853+0000","last_change":"2024-09-17T03:58:13.194115+0000","last_active":"2024-09-17T03:58:13.206853+0000","last_peered":"2024-09-17T03:58:13.206853+0000","last_clean":"2024-09-17T03:58:13.206853+0000","last_became_active":"2024-09-17T03:58:13.193359+0000","last_became_peered":"2024-09-17T03:58:13.193359+0000","last_unstale":"2024-09-17T03:58:13.206853+0000","last_undegraded":"2024-09-17T03:58:13.206853+0000","last_fullsized":"2024-09-17T03:58:13.206853+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:03.425741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221384+0000","last_change":"2024-09-17T03:58:13.214802+0000","last_active":"2024-09-17T03:58:13.221384+0000","last_peered":"2024-09-17T03:58:13.221384+0000","last_clean":"2024-09-17T03:58:13.221384+0000","last_became_active":"2024-09-17T03:58:13.213477+0000","last_became_peered":"2024-09-17T03:58:13.213477+0000","last_unstale":"2024-09-17T03:58:13.221384+0000","last_undegraded":"2024-09-17T03:58:13.221384+0000","last_fullsized":"2024-09-17T03:58:13.221384+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:15.520646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"138'4","reported_seq":17,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217520+0000","last_change":"2024-09-17T03:58:12.191412+0000","last_active":"2024-09-17T03:58:13.217520+0000","last_peered":"2024-09-17T03:58:13.217520+0000","last_clean":"2024-09-17T03:58:13.217520+0000","last_became_active":"2024-09-17T03:58:12.191081+0000","last_became_peered":"2024-09-17T03:58:12.191081+0000","last_unstale":"2024-09-17T03:58:13.217520+0000","last_undegraded":"2024-09-17T03:58:13.217520+0000","last_fullsized":"2024-09-17T03:58:13.217520+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:52.781413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.196046+0000","last_change":"2024-09-17T03:58:13.193155+0000","last_active":"2024-09-17T03:58:13.196046+0000","last_peered":"2024-09-17T03:58:13.196046+0000","last_clean":"2024-09-17T03:58:13.196046+0000","last_became_active":"2024-09-17T03:58:13.192825+0000","last_became_peered":"2024-09-17T03:58:13.192825+0000","last_unstale":"2024-09-17T03:58:13.196046+0000","last_undegraded":"2024-09-17T03:58:13.196046+0000","last_fullsized":"2024-09-17T03:58:13.196046+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:26.594749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.208724+0000","last_change":"2024-09-17T03:58:12.189882+0000","last_active":"2024-09-17T03:58:13.208724+0000","last_peered":"2024-09-17T03:58:13.208724+0000","last_clean":"2024-09-17T03:58:13.208724+0000","last_became_active":"2024-09-17T03:58:12.189556+0000","last_became_peered":"2024-09-17T03:58:12.189556+0000","last_unstale":"2024-09-17T03:58:13.208724+0000","last_undegraded":"2024-09-17T03:58:13.208724+0000","last_fullsized":"2024-09-17T03:58:13.208724+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:29.533857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194271+0000","last_change":"2024-09-17T03:58:13.200790+0000","last_active":"2024-09-17T03:58:14.194271+0000","last_peered":"2024-09-17T03:58:14.194271+0000","last_clean":"2024-09-17T03:58:14.194271+0000","last_became_active":"2024-09-17T03:58:13.199709+0000","last_became_peered":"2024-09-17T03:58:13.199709+0000","last_unstale":"2024-09-17T03:58:14.194271+0000","last_undegraded":"2024-09-17T03:58:14.194271+0000","last_fullsized":"2024-09-17T03:58:14.194271+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:06.223848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194510+0000","last_change":"2024-09-17T03:58:12.182338+0000","last_active":"2024-09-17T03:58:14.194510+0000","last_peered":"2024-09-17T03:58:14.194510+0000","last_clean":"2024-09-17T03:58:14.194510+0000","last_became_active":"2024-09-17T03:58:12.182023+0000","last_became_peered":"2024-09-17T03:58:12.182023+0000","last_unstale":"2024-09-17T03:58:14.194510+0000","last_undegraded":"2024-09-17T03:58:14.194510+0000","last_fullsized":"2024-09-17T03:58:14.194510+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:42.424799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217966+0000","last_change":"2024-09-17T03:58:13.212687+0000","last_active":"2024-09-17T03:58:13.217966+0000","last_peered":"2024-09-17T03:58:13.217966+0000","last_clean":"2024-09-17T03:58:13.217966+0000","last_became_active":"2024-09-17T03:58:13.211587+0000","last_became_peered":"2024-09-17T03:58:13.211587+0000","last_unstale":"2024-09-17T03:58:13.217966+0000","last_undegraded":"2024-09-17T03:58:13.217966+0000","last_fullsized":"2024-09-17T03:58:13.217966+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:35.560553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"138'2","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195184+0000","last_change":"2024-09-17T03:58:12.177610+0000","last_active":"2024-09-17T03:58:14.195184+0000","last_peered":"2024-09-17T03:58:14.195184+0000","last_clean":"2024-09-17T03:58:14.195184+0000","last_became_active":"2024-09-17T03:58:12.177288+0000","last_became_peered":"2024-09-17T03:58:12.177288+0000","last_unstale":"2024-09-17T03:58:14.195184+0000","last_undegraded":"2024-09-17T03:58:14.195184+0000","last_fullsized":"2024-09-17T03:58:14.195184+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:51:09.621257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.208294+0000","last_change":"2024-09-17T03:58:13.205019+0000","last_active":"2024-09-17T03:58:13.208294+0000","last_peered":"2024-09-17T03:58:13.208294+0000","last_clean":"2024-09-17T03:58:13.208294+0000","last_became_active":"2024-09-17T03:58:13.201214+0000","last_became_peered":"2024-09-17T03:58:13.201214+0000","last_unstale":"2024-09-17T03:58:13.208294+0000","last_undegraded":"2024-09-17T03:58:13.208294+0000","last_fullsized":"2024-09-17T03:58:13.208294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:58.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211601+0000","last_change":"2024-09-17T03:58:12.190947+0000","last_active":"2024-09-17T03:58:13.211601+0000","last_peered":"2024-09-17T03:58:13.211601+0000","last_clean":"2024-09-17T03:58:13.211601+0000","last_became_active":"2024-09-17T03:58:12.189957+0000","last_became_peered":"2024-09-17T03:58:12.189957+0000","last_unstale":"2024-09-17T03:58:13.211601+0000","last_undegraded":"2024-09-17T03:58:13.211601+0000","last_fullsized":"2024-09-17T03:58:13.211601+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:12.520179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195391+0000","last_change":"2024-09-17T03:58:13.200333+0000","last_active":"2024-09-17T03:58:14.195391+0000","last_peered":"2024-09-17T03:58:14.195391+0000","last_clean":"2024-09-17T03:58:14.195391+0000","last_became_active":"2024-09-17T03:58:13.199867+0000","last_became_peered":"2024-09-17T03:58:13.199867+0000","last_unstale":"2024-09-17T03:58:14.195391+0000","last_undegraded":"2024-09-17T03:58:14.195391+0000","last_fullsized":"2024-09-17T03:58:14.195391+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:19.472713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193481+0000","last_change":"2024-09-17T03:58:12.190669+0000","last_active":"2024-09-17T03:58:14.193481+0000","last_peered":"2024-09-17T03:58:14.193481+0000","last_clean":"2024-09-17T03:58:14.193481+0000","last_became_active":"2024-09-17T03:58:12.189905+0000","last_became_peered":"2024-09-17T03:58:12.189905+0000","last_unstale":"2024-09-17T03:58:14.193481+0000","last_undegraded":"2024-09-17T03:58:14.193481+0000","last_fullsized":"2024-09-17T03:58:14.193481+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:39.303511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194960+0000","last_change":"2024-09-17T03:58:13.186201+0000","last_active":"2024-09-17T03:58:13.194960+0000","last_peered":"2024-09-17T03:58:13.194960+0000","last_clean":"2024-09-17T03:58:13.194960+0000","last_became_active":"2024-09-17T03:58:13.185914+0000","last_became_peered":"2024-09-17T03:58:13.185914+0000","last_unstale":"2024-09-17T03:58:13.194960+0000","last_undegraded":"2024-09-17T03:58:13.194960+0000","last_fullsized":"2024-09-17T03:58:13.194960+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:55.317170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.189775+0000","last_change":"2024-09-17T03:58:12.182011+0000","last_active":"2024-09-17T03:58:13.189775+0000","last_peered":"2024-09-17T03:58:13.189775+0000","last_clean":"2024-09-17T03:58:13.189775+0000","last_became_active":"2024-09-17T03:58:12.180843+0000","last_became_peered":"2024-09-17T03:58:12.180843+0000","last_unstale":"2024-09-17T03:58:13.189775+0000","last_undegraded":"2024-09-17T03:58:13.189775+0000","last_fullsized":"2024-09-17T03:58:13.189775+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:46.265281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193571+0000","last_change":"2024-09-17T03:58:13.193865+0000","last_active":"2024-09-17T03:58:14.193571+0000","last_peered":"2024-09-17T03:58:14.193571+0000","last_clean":"2024-09-17T03:58:14.193571+0000","last_became_active":"2024-09-17T03:58:13.192337+0000","last_became_peered":"2024-09-17T03:58:13.192337+0000","last_unstale":"2024-09-17T03:58:14.193571+0000","last_undegraded":"2024-09-17T03:58:14.193571+0000","last_fullsized":"2024-09-17T03:58:14.193571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:09.119456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193245+0000","last_change":"2024-09-17T03:58:12.187713+0000","last_active":"2024-09-17T03:58:14.193245+0000","last_peered":"2024-09-17T03:58:14.193245+0000","last_clean":"2024-09-17T03:58:14.193245+0000","last_became_active":"2024-09-17T03:58:12.187296+0000","last_became_peered":"2024-09-17T03:58:12.187296+0000","last_unstale":"2024-09-17T03:58:14.193245+0000","last_undegraded":"2024-09-17T03:58:14.193245+0000","last_fullsized":"2024-09-17T03:58:14.193245+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:32.836934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194111+0000","last_change":"2024-09-17T03:58:13.203028+0000","last_active":"2024-09-17T03:58:14.194111+0000","last_peered":"2024-09-17T03:58:14.194111+0000","last_clean":"2024-09-17T03:58:14.194111+0000","last_became_active":"2024-09-17T03:58:13.199385+0000","last_became_peered":"2024-09-17T03:58:13.199385+0000","last_unstale":"2024-09-17T03:58:14.194111+0000","last_undegraded":"2024-09-17T03:58:14.194111+0000","last_fullsized":"2024-09-17T03:58:14.194111+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:58.554002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.177442+0000","last_change":"2024-09-17T03:58:12.181916+0000","last_active":"2024-09-17T03:58:13.177442+0000","last_peered":"2024-09-17T03:58:13.177442+0000","last_clean":"2024-09-17T03:58:13.177442+0000","last_became_active":"2024-09-17T03:58:12.181608+0000","last_became_peered":"2024-09-17T03:58:12.181608+0000","last_unstale":"2024-09-17T03:58:13.177442+0000","last_undegraded":"2024-09-17T03:58:13.177442+0000","last_fullsized":"2024-09-17T03:58:13.177442+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:36.142435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221504+0000","last_change":"2024-09-17T03:58:13.216490+0000","last_active":"2024-09-17T03:58:13.221504+0000","last_peered":"2024-09-17T03:58:13.221504+0000","last_clean":"2024-09-17T03:58:13.221504+0000","last_became_active":"2024-09-17T03:58:13.215403+0000","last_became_peered":"2024-09-17T03:58:13.215403+0000","last_unstale":"2024-09-17T03:58:13.221504+0000","last_undegraded":"2024-09-17T03:58:13.221504+0000","last_fullsized":"2024-09-17T03:58:13.221504+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:47.683195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203799+0000","last_change":"2024-09-17T03:58:12.189856+0000","last_active":"2024-09-17T03:58:13.203799+0000","last_peered":"2024-09-17T03:58:13.203799+0000","last_clean":"2024-09-17T03:58:13.203799+0000","last_became_active":"2024-09-17T03:58:12.189459+0000","last_became_peered":"2024-09-17T03:58:12.189459+0000","last_unstale":"2024-09-17T03:58:13.203799+0000","last_undegraded":"2024-09-17T03:58:13.203799+0000","last_fullsized":"2024-09-17T03:58:13.203799+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:07.684708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.212020+0000","last_change":"2024-09-17T03:58:13.209041+0000","last_active":"2024-09-17T03:58:13.212020+0000","last_peered":"2024-09-17T03:58:13.212020+0000","last_clean":"2024-09-17T03:58:13.212020+0000","last_became_active":"2024-09-17T03:58:13.208747+0000","last_became_peered":"2024-09-17T03:58:13.208747+0000","last_unstale":"2024-09-17T03:58:13.212020+0000","last_undegraded":"2024-09-17T03:58:13.212020+0000","last_fullsized":"2024-09-17T03:58:13.212020+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:12:00.217845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"138'6","reported_seq":19,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.217870+0000","last_change":"2024-09-17T03:58:12.187293+0000","last_active":"2024-09-17T03:58:13.217870+0000","last_peered":"2024-09-17T03:58:13.217870+0000","last_clean":"2024-09-17T03:58:13.217870+0000","last_became_active":"2024-09-17T03:58:12.186432+0000","last_became_peered":"2024-09-17T03:58:12.186432+0000","last_unstale":"2024-09-17T03:58:13.217870+0000","last_undegraded":"2024-09-17T03:58:13.217870+0000","last_fullsized":"2024-09-17T03:58:13.217870+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:41.484361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.215831+0000","last_change":"2024-09-17T03:58:13.207804+0000","last_active":"2024-09-17T03:58:13.215831+0000","last_peered":"2024-09-17T03:58:13.215831+0000","last_clean":"2024-09-17T03:58:13.215831+0000","last_became_active":"2024-09-17T03:58:13.207095+0000","last_became_peered":"2024-09-17T03:58:13.207095+0000","last_unstale":"2024-09-17T03:58:13.215831+0000","last_undegraded":"2024-09-17T03:58:13.215831+0000","last_fullsized":"2024-09-17T03:58:13.215831+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:48.837854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.211112+0000","last_change":"2024-09-17T03:58:12.183495+0000","last_active":"2024-09-17T03:58:13.211112+0000","last_peered":"2024-09-17T03:58:13.211112+0000","last_clean":"2024-09-17T03:58:13.211112+0000","last_became_active":"2024-09-17T03:58:12.183185+0000","last_became_peered":"2024-09-17T03:58:12.183185+0000","last_unstale":"2024-09-17T03:58:13.211112+0000","last_undegraded":"2024-09-17T03:58:13.211112+0000","last_fullsized":"2024-09-17T03:58:13.211112+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:25:28.001445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203427+0000","last_change":"2024-09-17T03:58:13.196869+0000","last_active":"2024-09-17T03:58:13.203427+0000","last_peered":"2024-09-17T03:58:13.203427+0000","last_clean":"2024-09-17T03:58:13.203427+0000","last_became_active":"2024-09-17T03:58:13.194671+0000","last_became_peered":"2024-09-17T03:58:13.194671+0000","last_unstale":"2024-09-17T03:58:13.203427+0000","last_undegraded":"2024-09-17T03:58:13.203427+0000","last_fullsized":"2024-09-17T03:58:13.203427+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:41.493738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.193682+0000","last_change":"2024-09-17T03:58:12.181431+0000","last_active":"2024-09-17T03:58:13.193682+0000","last_peered":"2024-09-17T03:58:13.193682+0000","last_clean":"2024-09-17T03:58:13.193682+0000","last_became_active":"2024-09-17T03:58:12.181091+0000","last_became_peered":"2024-09-17T03:58:12.181091+0000","last_unstale":"2024-09-17T03:58:13.193682+0000","last_undegraded":"2024-09-17T03:58:13.193682+0000","last_fullsized":"2024-09-17T03:58:13.193682+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:58:02.535381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193852+0000","last_change":"2024-09-17T03:58:13.208519+0000","last_active":"2024-09-17T03:58:14.193852+0000","last_peered":"2024-09-17T03:58:14.193852+0000","last_clean":"2024-09-17T03:58:14.193852+0000","last_became_active":"2024-09-17T03:58:13.207999+0000","last_became_peered":"2024-09-17T03:58:13.207999+0000","last_unstale":"2024-09-17T03:58:14.193852+0000","last_undegraded":"2024-09-17T03:58:14.193852+0000","last_fullsized":"2024-09-17T03:58:14.193852+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:18.592514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193756+0000","last_change":"2024-09-17T03:58:12.178728+0000","last_active":"2024-09-17T03:58:14.193756+0000","last_peered":"2024-09-17T03:58:14.193756+0000","last_clean":"2024-09-17T03:58:14.193756+0000","last_became_active":"2024-09-17T03:58:12.178119+0000","last_became_peered":"2024-09-17T03:58:12.178119+0000","last_unstale":"2024-09-17T03:58:14.193756+0000","last_undegraded":"2024-09-17T03:58:14.193756+0000","last_fullsized":"2024-09-17T03:58:14.193756+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:56:14.350160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210226+0000","last_change":"2024-09-17T03:58:13.200404+0000","last_active":"2024-09-17T03:58:13.210226+0000","last_peered":"2024-09-17T03:58:13.210226+0000","last_clean":"2024-09-17T03:58:13.210226+0000","last_became_active":"2024-09-17T03:58:13.200049+0000","last_became_peered":"2024-09-17T03:58:13.200049+0000","last_unstale":"2024-09-17T03:58:13.210226+0000","last_undegraded":"2024-09-17T03:58:13.210226+0000","last_fullsized":"2024-09-17T03:58:13.210226+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:21.313582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"138'8","reported_seq":21,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218155+0000","last_change":"2024-09-17T03:58:12.188462+0000","last_active":"2024-09-17T03:58:13.218155+0000","last_peered":"2024-09-17T03:58:13.218155+0000","last_clean":"2024-09-17T03:58:13.218155+0000","last_became_active":"2024-09-17T03:58:12.188155+0000","last_became_peered":"2024-09-17T03:58:12.188155+0000","last_unstale":"2024-09-17T03:58:13.218155+0000","last_undegraded":"2024-09-17T03:58:13.218155+0000","last_fullsized":"2024-09-17T03:58:13.218155+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:22.173349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.221298+0000","last_change":"2024-09-17T03:58:13.214245+0000","last_active":"2024-09-17T03:58:13.221298+0000","last_peered":"2024-09-17T03:58:13.221298+0000","last_clean":"2024-09-17T03:58:13.221298+0000","last_became_active":"2024-09-17T03:58:13.212075+0000","last_became_peered":"2024-09-17T03:58:13.212075+0000","last_unstale":"2024-09-17T03:58:13.221298+0000","last_undegraded":"2024-09-17T03:58:13.221298+0000","last_fullsized":"2024-09-17T03:58:13.221298+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:32:00.392907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193650+0000","last_change":"2024-09-17T03:58:12.184655+0000","last_active":"2024-09-17T03:58:14.193650+0000","last_peered":"2024-09-17T03:58:14.193650+0000","last_clean":"2024-09-17T03:58:14.193650+0000","last_became_active":"2024-09-17T03:58:12.184308+0000","last_became_peered":"2024-09-17T03:58:12.184308+0000","last_unstale":"2024-09-17T03:58:14.193650+0000","last_undegraded":"2024-09-17T03:58:14.193650+0000","last_fullsized":"2024-09-17T03:58:14.193650+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:06:38.876264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214452+0000","last_change":"2024-09-17T03:58:13.212198+0000","last_active":"2024-09-17T03:58:13.214452+0000","last_peered":"2024-09-17T03:58:13.214452+0000","last_clean":"2024-09-17T03:58:13.214452+0000","last_became_active":"2024-09-17T03:58:13.211866+0000","last_became_peered":"2024-09-17T03:58:13.211866+0000","last_unstale":"2024-09-17T03:58:13.214452+0000","last_undegraded":"2024-09-17T03:58:13.214452+0000","last_fullsized":"2024-09-17T03:58:13.214452+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:14.916036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.204760+0000","last_change":"2024-09-17T03:58:12.185808+0000","last_active":"2024-09-17T03:58:13.204760+0000","last_peered":"2024-09-17T03:58:13.204760+0000","last_clean":"2024-09-17T03:58:13.204760+0000","last_became_active":"2024-09-17T03:58:12.185447+0000","last_became_peered":"2024-09-17T03:58:12.185447+0000","last_unstale":"2024-09-17T03:58:13.204760+0000","last_undegraded":"2024-09-17T03:58:13.204760+0000","last_fullsized":"2024-09-17T03:58:13.204760+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:16.600004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210561+0000","last_change":"2024-09-17T03:58:13.200760+0000","last_active":"2024-09-17T03:58:13.210561+0000","last_peered":"2024-09-17T03:58:13.210561+0000","last_clean":"2024-09-17T03:58:13.210561+0000","last_became_active":"2024-09-17T03:58:13.200447+0000","last_became_peered":"2024-09-17T03:58:13.200447+0000","last_unstale":"2024-09-17T03:58:13.210561+0000","last_undegraded":"2024-09-17T03:58:13.210561+0000","last_fullsized":"2024-09-17T03:58:13.210561+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:57:05.971575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"138'1","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.212750+0000","last_change":"2024-09-17T03:58:12.190248+0000","last_active":"2024-09-17T03:58:13.212750+0000","last_peered":"2024-09-17T03:58:13.212750+0000","last_clean":"2024-09-17T03:58:13.212750+0000","last_became_active":"2024-09-17T03:58:12.189174+0000","last_became_peered":"2024-09-17T03:58:12.189174+0000","last_unstale":"2024-09-17T03:58:13.212750+0000","last_undegraded":"2024-09-17T03:58:13.212750+0000","last_fullsized":"2024-09-17T03:58:13.212750+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:49.176315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194391+0000","last_change":"2024-09-17T03:58:13.203054+0000","last_active":"2024-09-17T03:58:14.194391+0000","last_peered":"2024-09-17T03:58:14.194391+0000","last_clean":"2024-09-17T03:58:14.194391+0000","last_became_active":"2024-09-17T03:58:13.202746+0000","last_became_peered":"2024-09-17T03:58:13.202746+0000","last_unstale":"2024-09-17T03:58:14.194391+0000","last_undegraded":"2024-09-17T03:58:14.194391+0000","last_fullsized":"2024-09-17T03:58:14.194391+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:17.180677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.176283+0000","last_change":"2024-09-17T03:58:12.185416+0000","last_active":"2024-09-17T03:58:13.176283+0000","last_peered":"2024-09-17T03:58:13.176283+0000","last_clean":"2024-09-17T03:58:13.176283+0000","last_became_active":"2024-09-17T03:58:12.185123+0000","last_became_peered":"2024-09-17T03:58:12.185123+0000","last_unstale":"2024-09-17T03:58:13.176283+0000","last_undegraded":"2024-09-17T03:58:13.176283+0000","last_fullsized":"2024-09-17T03:58:13.176283+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:09.325907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.195014+0000","last_change":"2024-09-17T03:58:13.202130+0000","last_active":"2024-09-17T03:58:14.195014+0000","last_peered":"2024-09-17T03:58:14.195014+0000","last_clean":"2024-09-17T03:58:14.195014+0000","last_became_active":"2024-09-17T03:58:13.201756+0000","last_became_peered":"2024-09-17T03:58:13.201756+0000","last_unstale":"2024-09-17T03:58:14.195014+0000","last_undegraded":"2024-09-17T03:58:14.195014+0000","last_fullsized":"2024-09-17T03:58:14.195014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:58.838185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.205731+0000","last_change":"2024-09-17T03:58:12.185147+0000","last_active":"2024-09-17T03:58:13.205731+0000","last_peered":"2024-09-17T03:58:13.205731+0000","last_clean":"2024-09-17T03:58:13.205731+0000","last_became_active":"2024-09-17T03:58:12.184785+0000","last_became_peered":"2024-09-17T03:58:12.184785+0000","last_unstale":"2024-09-17T03:58:13.205731+0000","last_undegraded":"2024-09-17T03:58:13.205731+0000","last_fullsized":"2024-09-17T03:58:13.205731+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:31:20.814706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214131+0000","last_change":"2024-09-17T03:58:13.203540+0000","last_active":"2024-09-17T03:58:13.214131+0000","last_peered":"2024-09-17T03:58:13.214131+0000","last_clean":"2024-09-17T03:58:13.214131+0000","last_became_active":"2024-09-17T03:58:13.200892+0000","last_became_peered":"2024-09-17T03:58:13.200892+0000","last_unstale":"2024-09-17T03:58:13.214131+0000","last_undegraded":"2024-09-17T03:58:13.214131+0000","last_fullsized":"2024-09-17T03:58:13.214131+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:17.706429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.196607+0000","last_change":"2024-09-17T03:58:12.191693+0000","last_active":"2024-09-17T03:58:13.196607+0000","last_peered":"2024-09-17T03:58:13.196607+0000","last_clean":"2024-09-17T03:58:13.196607+0000","last_became_active":"2024-09-17T03:58:12.190898+0000","last_became_peered":"2024-09-17T03:58:12.190898+0000","last_unstale":"2024-09-17T03:58:13.196607+0000","last_undegraded":"2024-09-17T03:58:13.196607+0000","last_fullsized":"2024-09-17T03:58:13.196607+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:28:43.769495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193347+0000","last_change":"2024-09-17T03:58:13.198845+0000","last_active":"2024-09-17T03:58:14.193347+0000","last_peered":"2024-09-17T03:58:14.193347+0000","last_clean":"2024-09-17T03:58:14.193347+0000","last_became_active":"2024-09-17T03:58:13.198516+0000","last_became_peered":"2024-09-17T03:58:13.198516+0000","last_unstale":"2024-09-17T03:58:14.193347+0000","last_undegraded":"2024-09-17T03:58:14.193347+0000","last_fullsized":"2024-09-17T03:58:14.193347+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:22.267943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.202927+0000","last_change":"2024-09-17T03:58:12.192937+0000","last_active":"2024-09-17T03:58:13.202927+0000","last_peered":"2024-09-17T03:58:13.202927+0000","last_clean":"2024-09-17T03:58:13.202927+0000","last_became_active":"2024-09-17T03:58:12.192315+0000","last_became_peered":"2024-09-17T03:58:12.192315+0000","last_unstale":"2024-09-17T03:58:13.202927+0000","last_undegraded":"2024-09-17T03:58:13.202927+0000","last_fullsized":"2024-09-17T03:58:13.202927+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:57.830280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213802+0000","last_change":"2024-09-17T03:58:13.208665+0000","last_active":"2024-09-17T03:58:13.213802+0000","last_peered":"2024-09-17T03:58:13.213802+0000","last_clean":"2024-09-17T03:58:13.213802+0000","last_became_active":"2024-09-17T03:58:13.207794+0000","last_became_peered":"2024-09-17T03:58:13.207794+0000","last_unstale":"2024-09-17T03:58:13.213802+0000","last_undegraded":"2024-09-17T03:58:13.213802+0000","last_fullsized":"2024-09-17T03:58:13.213802+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:20.656450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194661+0000","last_change":"2024-09-17T03:58:12.184241+0000","last_active":"2024-09-17T03:58:13.194661+0000","last_peered":"2024-09-17T03:58:13.194661+0000","last_clean":"2024-09-17T03:58:13.194661+0000","last_became_active":"2024-09-17T03:58:12.183861+0000","last_became_peered":"2024-09-17T03:58:12.183861+0000","last_unstale":"2024-09-17T03:58:13.194661+0000","last_undegraded":"2024-09-17T03:58:13.194661+0000","last_fullsized":"2024-09-17T03:58:13.194661+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:27.527226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194182+0000","last_change":"2024-09-17T03:58:13.210907+0000","last_active":"2024-09-17T03:58:14.194182+0000","last_peered":"2024-09-17T03:58:14.194182+0000","last_clean":"2024-09-17T03:58:14.194182+0000","last_became_active":"2024-09-17T03:58:13.210573+0000","last_became_peered":"2024-09-17T03:58:13.210573+0000","last_unstale":"2024-09-17T03:58:14.194182+0000","last_undegraded":"2024-09-17T03:58:14.194182+0000","last_fullsized":"2024-09-17T03:58:14.194182+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:59:01.071588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"138'2","reported_seq":15,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.216971+0000","last_change":"2024-09-17T03:58:12.191164+0000","last_active":"2024-09-17T03:58:13.216971+0000","last_peered":"2024-09-17T03:58:13.216971+0000","last_clean":"2024-09-17T03:58:13.216971+0000","last_became_active":"2024-09-17T03:58:12.190803+0000","last_became_peered":"2024-09-17T03:58:12.190803+0000","last_unstale":"2024-09-17T03:58:13.216971+0000","last_undegraded":"2024-09-17T03:58:13.216971+0000","last_fullsized":"2024-09-17T03:58:13.216971+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:18.464808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.210281+0000","last_change":"2024-09-17T03:58:13.200728+0000","last_active":"2024-09-17T03:58:13.210281+0000","last_peered":"2024-09-17T03:58:13.210281+0000","last_clean":"2024-09-17T03:58:13.210281+0000","last_became_active":"2024-09-17T03:58:13.200366+0000","last_became_peered":"2024-09-17T03:58:13.200366+0000","last_unstale":"2024-09-17T03:58:13.210281+0000","last_undegraded":"2024-09-17T03:58:13.210281+0000","last_fullsized":"2024-09-17T03:58:13.210281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:20:43.282925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"138'1","reported_seq":16,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194564+0000","last_change":"2024-09-17T03:58:12.185932+0000","last_active":"2024-09-17T03:58:14.194564+0000","last_peered":"2024-09-17T03:58:14.194564+0000","last_clean":"2024-09-17T03:58:14.194564+0000","last_became_active":"2024-09-17T03:58:12.183505+0000","last_became_peered":"2024-09-17T03:58:12.183505+0000","last_unstale":"2024-09-17T03:58:14.194564+0000","last_undegraded":"2024-09-17T03:58:14.194564+0000","last_fullsized":"2024-09-17T03:58:14.194564+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:00.381361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.203348+0000","last_change":"2024-09-17T03:58:13.199488+0000","last_active":"2024-09-17T03:58:13.203348+0000","last_peered":"2024-09-17T03:58:13.203348+0000","last_clean":"2024-09-17T03:58:13.203348+0000","last_became_active":"2024-09-17T03:58:13.199176+0000","last_became_peered":"2024-09-17T03:58:13.199176+0000","last_unstale":"2024-09-17T03:58:13.203348+0000","last_undegraded":"2024-09-17T03:58:13.203348+0000","last_fullsized":"2024-09-17T03:58:13.203348+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:24:50.942453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194630+0000","last_change":"2024-09-17T03:58:12.187122+0000","last_active":"2024-09-17T03:58:14.194630+0000","last_peered":"2024-09-17T03:58:14.194630+0000","last_clean":"2024-09-17T03:58:14.194630+0000","last_became_active":"2024-09-17T03:58:12.186019+0000","last_became_peered":"2024-09-17T03:58:12.186019+0000","last_unstale":"2024-09-17T03:58:14.194630+0000","last_undegraded":"2024-09-17T03:58:14.194630+0000","last_fullsized":"2024-09-17T03:58:14.194630+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:25.273252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194517+0000","last_change":"2024-09-17T03:58:13.197369+0000","last_active":"2024-09-17T03:58:14.194517+0000","last_peered":"2024-09-17T03:58:14.194517+0000","last_clean":"2024-09-17T03:58:14.194517+0000","last_became_active":"2024-09-17T03:58:13.196755+0000","last_became_peered":"2024-09-17T03:58:13.196755+0000","last_unstale":"2024-09-17T03:58:14.194517+0000","last_undegraded":"2024-09-17T03:58:14.194517+0000","last_fullsized":"2024-09-17T03:58:14.194517+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:08.405776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194289+0000","last_change":"2024-09-17T03:58:12.185096+0000","last_active":"2024-09-17T03:58:14.194289+0000","last_peered":"2024-09-17T03:58:14.194289+0000","last_clean":"2024-09-17T03:58:14.194289+0000","last_became_active":"2024-09-17T03:58:12.184809+0000","last_became_peered":"2024-09-17T03:58:12.184809+0000","last_unstale":"2024-09-17T03:58:14.194289+0000","last_undegraded":"2024-09-17T03:58:14.194289+0000","last_fullsized":"2024-09-17T03:58:14.194289+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:21:45.665906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.206446+0000","last_change":"2024-09-17T03:58:13.202016+0000","last_active":"2024-09-17T03:58:13.206446+0000","last_peered":"2024-09-17T03:58:13.206446+0000","last_clean":"2024-09-17T03:58:13.206446+0000","last_became_active":"2024-09-17T03:58:13.201551+0000","last_became_peered":"2024-09-17T03:58:13.201551+0000","last_unstale":"2024-09-17T03:58:13.206446+0000","last_undegraded":"2024-09-17T03:58:13.206446+0000","last_fullsized":"2024-09-17T03:58:13.206446+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:54.860771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.188639+0000","last_change":"2024-09-17T03:58:12.184231+0000","last_active":"2024-09-17T03:58:13.188639+0000","last_peered":"2024-09-17T03:58:13.188639+0000","last_clean":"2024-09-17T03:58:13.188639+0000","last_became_active":"2024-09-17T03:58:12.183961+0000","last_became_peered":"2024-09-17T03:58:12.183961+0000","last_unstale":"2024-09-17T03:58:13.188639+0000","last_undegraded":"2024-09-17T03:58:13.188639+0000","last_fullsized":"2024-09-17T03:58:13.188639+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:07:42.110947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.194294+0000","last_change":"2024-09-17T03:58:13.201893+0000","last_active":"2024-09-17T03:58:14.194294+0000","last_peered":"2024-09-17T03:58:14.194294+0000","last_clean":"2024-09-17T03:58:14.194294+0000","last_became_active":"2024-09-17T03:58:13.201569+0000","last_became_peered":"2024-09-17T03:58:13.201569+0000","last_unstale":"2024-09-17T03:58:14.194294+0000","last_undegraded":"2024-09-17T03:58:14.194294+0000","last_fullsized":"2024-09-17T03:58:14.194294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:34.930836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.201465+0000","last_change":"2024-09-17T03:58:12.190851+0000","last_active":"2024-09-17T03:58:13.201465+0000","last_peered":"2024-09-17T03:58:13.201465+0000","last_clean":"2024-09-17T03:58:13.201465+0000","last_became_active":"2024-09-17T03:58:12.190544+0000","last_became_peered":"2024-09-17T03:58:12.190544+0000","last_unstale":"2024-09-17T03:58:13.201465+0000","last_undegraded":"2024-09-17T03:58:13.201465+0000","last_fullsized":"2024-09-17T03:58:13.201465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:27:47.875694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.218273+0000","last_change":"2024-09-17T03:58:13.213318+0000","last_active":"2024-09-17T03:58:13.218273+0000","last_peered":"2024-09-17T03:58:13.218273+0000","last_clean":"2024-09-17T03:58:13.218273+0000","last_became_active":"2024-09-17T03:58:13.212980+0000","last_became_peered":"2024-09-17T03:58:13.212980+0000","last_unstale":"2024-09-17T03:58:13.218273+0000","last_undegraded":"2024-09-17T03:58:13.218273+0000","last_fullsized":"2024-09-17T03:58:13.218273+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:55.966116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.194622+0000","last_change":"2024-09-17T03:58:12.182076+0000","last_active":"2024-09-17T03:58:13.194622+0000","last_peered":"2024-09-17T03:58:13.194622+0000","last_clean":"2024-09-17T03:58:13.194622+0000","last_became_active":"2024-09-17T03:58:12.181735+0000","last_became_peered":"2024-09-17T03:58:12.181735+0000","last_unstale":"2024-09-17T03:58:13.194622+0000","last_undegraded":"2024-09-17T03:58:13.194622+0000","last_fullsized":"2024-09-17T03:58:13.194622+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:40.387720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-17T03:58:14.193843+0000","last_change":"2024-09-17T03:58:13.197885+0000","last_active":"2024-09-17T03:58:14.193843+0000","last_peered":"2024-09-17T03:58:14.193843+0000","last_clean":"2024-09-17T03:58:14.193843+0000","last_became_active":"2024-09-17T03:58:13.197577+0000","last_became_peered":"2024-09-17T03:58:13.197577+0000","last_unstale":"2024-09-17T03:58:14.193843+0000","last_undegraded":"2024-09-17T03:58:14.193843+0000","last_fullsized":"2024-09-17T03:58:14.193843+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:25.332392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.175736+0000","last_change":"2024-09-17T03:58:12.188614+0000","last_active":"2024-09-17T03:58:13.175736+0000","last_peered":"2024-09-17T03:58:13.175736+0000","last_clean":"2024-09-17T03:58:13.175736+0000","last_became_active":"2024-09-17T03:58:12.188308+0000","last_became_peered":"2024-09-17T03:58:12.188308+0000","last_unstale":"2024-09-17T03:58:13.175736+0000","last_undegraded":"2024-09-17T03:58:13.175736+0000","last_fullsized":"2024-09-17T03:58:13.175736+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:36:16.875180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.213963+0000","last_change":"2024-09-17T03:58:13.208342+0000","last_active":"2024-09-17T03:58:13.213963+0000","last_peered":"2024-09-17T03:58:13.213963+0000","last_clean":"2024-09-17T03:58:13.213963+0000","last_became_active":"2024-09-17T03:58:13.207033+0000","last_became_peered":"2024-09-17T03:58:13.207033+0000","last_unstale":"2024-09-17T03:58:13.213963+0000","last_undegraded":"2024-09-17T03:58:13.213963+0000","last_fullsized":"2024-09-17T03:58:13.213963+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:12.157014+0000","last_clean_scrub_stamp":"2024-09-17T03:58:12.157014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:18:19.772741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"138'3","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.214358+0000","last_change":"2024-09-17T03:58:12.184834+0000","last_active":"2024-09-17T03:58:13.214358+0000","last_peered":"2024-09-17T03:58:13.214358+0000","last_clean":"2024-09-17T03:58:13.214358+0000","last_became_active":"2024-09-17T03:58:12.184495+0000","last_became_peered":"2024-09-17T03:58:12.184495+0000","last_unstale":"2024-09-17T03:58:13.214358+0000","last_undegraded":"2024-09-17T03:58:13.214358+0000","last_fullsized":"2024-09-17T03:58:13.214358+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:58:11.149109+0000","last_clean_scrub_stamp":"2024-09-17T03:58:11.149109+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:47:01.983852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":325,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-17T03:58:13.201336+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:58:13.201336+0000","last_peered":"2024-09-17T03:58:13.201336+0000","last_clean":"2024-09-17T03:58:13.201336+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:58:13.201336+0000","last_undegraded":"2024-09-17T03:58:13.201336+0000","last_fullsized":"2024-09-17T03:58:13.201336+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509548,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37620,"kb_used_data":2268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334220,"statfs":{"total":96636764160,"available":96598241280,"internally_reserved":0,"allocated":2322432,"data_stored":1338896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37032,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334808,"statfs":{"total":96636764160,"available":96598843392,"internally_reserved":0,"allocated":1724416,"data_stored":747898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509550,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41192,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330648,"statfs":{"total":96636764160,"available":96594583552,"internally_reserved":0,"allocated":1720320,"data_stored":747876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509548,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37036,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334804,"statfs":{"total":96636764160,"available":96598839296,"internally_reserved":0,"allocated":1724416,"data_stored":747966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509550,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37048,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334792,"statfs":{"total":96636764160,"available":96598827008,"internally_reserved":0,"allocated":1736704,"data_stored":749558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509550,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37052,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334788,"statfs":{"total":96636764160,"available":96598822912,"internally_reserved":0,"allocated":1736704,"data_stored":754758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41180,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330660,"statfs":{"total":96636764160,"available":96594595840,"internally_reserved":0,"allocated":1708032,"data_stored":741101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41756,"kb_used_data":2248,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330084,"statfs":{"total":96636764160,"available":96594006016,"internally_reserved":0,"allocated":2301952,"data_stored":1331469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:58:18.106 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:58:18.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 -- 172.21.15.129:0/1390938124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764156f20 msgr2=0x7fd764160d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 --2- 172.21.15.129:0/1390938124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764156f20 0x7fd764160d50 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fd7640d0010 tx=0x7fd75402a4a0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 -- 172.21.15.129:0/1390938124 shutdown_connections 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 --2- 172.21.15.129:0/1390938124 >> v2:172.21.15.181:3300/0 conn(0x7fd76406ee70 0x7fd76406f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 --2- 172.21.15.129:0/1390938124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764156f20 0x7fd764160d50 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 -- 172.21.15.129:0/1390938124 >> 172.21.15.129:0/1390938124 conn(0x7fd7640d0120 msgr2=0x7fd7640d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 -- 172.21.15.129:0/1390938124 shutdown_connections 2024-09-17T03:58:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.372+0000 7fd769200640 1 -- 172.21.15.129:0/1390938124 wait complete. 2024-09-17T03:58:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.375+0000 7fd769200640 1 Processor -- start 2024-09-17T03:58:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.375+0000 7fd769200640 1 -- start start 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76406ee70 0x7fd764154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 0x7fd7640aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd7640cf450 con 0x7fd764072ee0 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd7640cf2d0 con 0x7fd764156f20 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd769200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd7640cf5d0 con 0x7fd76406ee70 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd762a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd762a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57436/0 (socket says 172.21.15.129:57436) 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.376+0000 7fd762a00640 1 -- 172.21.15.129:0/2204856971 learned_addr learned my addr 172.21.15.129:0/2204856971 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd763400640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76406ee70 0x7fd764154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd763e00640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 0x7fd7640aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 -- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76406ee70 msgr2=0x7fd764154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76406ee70 0x7fd764154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 -- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 msgr2=0x7fd7640aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 0x7fd7640aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 -- 172.21.15.129:0/2204856971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76415c0a0 con 0x7fd764072ee0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd763e00640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 0x7fd7640aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd762a00640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd75000d6d0 tx=0x7fd75000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750017070 con 0x7fd764072ee0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd750004310 con 0x7fd764072ee0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76415b130 con 0x7fd764072ee0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.377+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd750009a80 con 0x7fd764072ee0 2024-09-17T03:58:18.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.378+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd76415b6f0 con 0x7fd764072ee0 2024-09-17T03:58:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.379+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd76415c310 con 0x7fd764072ee0 2024-09-17T03:58:18.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.380+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd750009c20 con 0x7fd764072ee0 2024-09-17T03:58:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.381+0000 7fd75b400640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 0x7fd74407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.381+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7fd75009ad50 con 0x7fd764072ee0 2024-09-17T03:58:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.381+0000 7fd763400640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 0x7fd74407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.382+0000 7fd763400640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 0x7fd74407b0a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd754034000 tx=0x7fd754004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:18.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.385+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd7500631f0 con 0x7fd764072ee0 2024-09-17T03:58:18.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.637+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fd764073bc0 con 0x7fd764072ee0 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.641+0000 7fd75b400640 1 -- 172.21.15.129:0/2204856971 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v36) ==== 190+0+147 (secure 0 0 0) 0x7fd750062af0 con 0x7fd764072ee0 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:58:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:58:18.641 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 msgr2=0x7fd74407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 0x7fd74407b0a0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd754034000 tx=0x7fd754004730 comp rx=0 tx=0).stop 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 msgr2=0x7fd7640aa400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd75000d6d0 tx=0x7fd75000db90 comp rx=0 tx=0).stop 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 shutdown_connections 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd744078b90 0x7fd74407b0a0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd764156f20 0x7fd7640aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd764072ee0 0x7fd7640aa400 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 --2- 172.21.15.129:0/2204856971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76406ee70 0x7fd764154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.643+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 >> 172.21.15.129:0/2204856971 conn(0x7fd7640d0120 msgr2=0x7fd7640d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.644+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 shutdown_connections 2024-09-17T03:58:18.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.644+0000 7fd769200640 1 -- 172.21.15.129:0/2204856971 wait complete. 2024-09-17T03:58:18.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 -- 172.21.15.129:0/61266331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 msgr2=0x7f71280cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 --2- 172.21.15.129:0/61266331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71280cd090 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f71280ce800 tx=0x7f711402a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 -- 172.21.15.129:0/61266331 shutdown_connections 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 --2- 172.21.15.129:0/61266331 >> v2:172.21.15.181:3300/0 conn(0x7f7128070100 0x7f7128070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 --2- 172.21.15.129:0/61266331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71280cd090 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 -- 172.21.15.129:0/61266331 >> 172.21.15.129:0/61266331 conn(0x7f712806d410 msgr2=0x7f71280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.890+0000 7f712fa00640 1 -- 172.21.15.129:0/61266331 shutdown_connections 2024-09-17T03:58:18.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.891+0000 7f712fa00640 1 -- 172.21.15.129:0/61266331 wait complete. 2024-09-17T03:58:18.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.893+0000 7f712fa00640 1 Processor -- start 2024-09-17T03:58:18.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.893+0000 7f712fa00640 1 -- start start 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.894+0000 7f712fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7128070100 0x7f712809dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.894+0000 7f712fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.894+0000 7f712fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f712815da00 0x7f71281f8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f712fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71280b4a10 con 0x7f71280ccc80 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f712fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71280b4890 con 0x7f7128070100 2024-09-17T03:58:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f712fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7128070a50 con 0x7f712815da00 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57460/0 (socket says 172.21.15.129:57460) 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 -- 172.21.15.129:0/2189052992 learned_addr learned my addr 172.21.15.129:0/2189052992 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f712d000640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7128070100 0x7f712809dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f712da00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f712815da00 0x7f71281f8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 -- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f712815da00 msgr2=0x7f71281f8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f712815da00 0x7f71281f8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 -- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7128070100 msgr2=0x7f712809dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7128070100 0x7f712809dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.895+0000 7f7127e00640 1 -- 172.21.15.129:0/2189052992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71280cb9a0 con 0x7f71280ccc80 2024-09-17T03:58:18.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f7127e00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7114034040 tx=0x7f7114003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f711403e030 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7114003d10 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71280c8db0 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7114035040 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.896+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71280c9370 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.897+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7128070510 con 0x7f71280ccc80 2024-09-17T03:58:18.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.898+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7114041030 con 0x7f71280ccc80 2024-09-17T03:58:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.899+0000 7f7124c00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 0x7f710807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.899+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f71140bb440 con 0x7f71280ccc80 2024-09-17T03:58:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.900+0000 7f712d000640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 0x7f710807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.901+0000 7f712d000640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 0x7f710807b0a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f711c001720 tx=0x7f711c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:18.904+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7114083200 con 0x7f71280ccc80 2024-09-17T03:58:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.121+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71280cb730 con 0x7f71280ccc80 2024-09-17T03:58:19.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.121+0000 7f7124c00640 1 -- 172.21.15.129:0/2189052992 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5239 (secure 0 0 0) 0x7f7114087c30 con 0x7f71280ccc80 2024-09-17T03:58:19.121 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:19.121 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":115,"btime":"2024-09-17T03:58:16:203956+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3687707903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3687707903},{"type":"v1","addr":"172.21.15.181:6837","nonce":3687707903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/872498647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":872498647},{"type":"v1","addr":"172.21.15.181:6836","nonce":872498647}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":8178,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:15.725002+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:19.121 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 115 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 msgr2=0x7f710807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 0x7f710807b0a0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f711c001720 tx=0x7f711c009040 comp rx=0 tx=0).stop 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 msgr2=0x7f71281ec1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f7114034040 tx=0x7f7114003820 comp rx=0 tx=0).stop 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 shutdown_connections 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7108078b90 0x7f710807b0a0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f712815da00 0x7f71281f8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71280ccc80 0x7f71281ec1b0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 --2- 172.21.15.129:0/2189052992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7128070100 0x7f712809dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 >> 172.21.15.129:0/2189052992 conn(0x7f712806d410 msgr2=0x7f71280cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 shutdown_connections 2024-09-17T03:58:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.124+0000 7f712fa00640 1 -- 172.21.15.129:0/2189052992 wait complete. 2024-09-17T03:58:19.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 115, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:15.725002+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8181}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:58:19.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:58:19.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T03:58:19.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.776+0000 7f704ec00640 1 -- 172.21.15.129:0/1384716562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70480b9b40 msgr2=0x7f70480d2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.776+0000 7f704ec00640 1 --2- 172.21.15.129:0/1384716562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70480b9b40 0x7f70480d2a70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f70480c7eb0 tx=0x7f703002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 -- 172.21.15.129:0/1384716562 shutdown_connections 2024-09-17T03:58:19.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 --2- 172.21.15.129:0/1384716562 >> v2:172.21.15.181:3300/0 conn(0x7f70480b18b0 0x7f70480b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 --2- 172.21.15.129:0/1384716562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70480b9b40 0x7f70480d2a70 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 -- 172.21.15.129:0/1384716562 >> 172.21.15.129:0/1384716562 conn(0x7f70480d4610 msgr2=0x7f70480d4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 -- 172.21.15.129:0/1384716562 shutdown_connections 2024-09-17T03:58:19.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.777+0000 7f704ec00640 1 -- 172.21.15.129:0/1384716562 wait complete. 2024-09-17T03:58:19.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.779+0000 7f704ec00640 1 Processor -- start 2024-09-17T03:58:19.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.779+0000 7f704ec00640 1 -- start start 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70480b18b0 0x7f704815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70480b9b40 0x7f70480b7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70480b2200 con 0x7f7048161d40 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70480d31c0 con 0x7f70480b9b40 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70480b2380 con 0x7f70480b18b0 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57476/0 (socket says 172.21.15.129:57476) 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 -- 172.21.15.129:0/2578659923 learned_addr learned my addr 172.21.15.129:0/2578659923 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:19.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f7047e00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70480b18b0 0x7f704815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f7047400640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70480b9b40 0x7f70480b7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 -- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70480b18b0 msgr2=0x7f704815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70480b18b0 0x7f704815b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 -- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70480b9b40 msgr2=0x7f70480b7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70480b9b40 0x7f70480b7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.781+0000 7f704cc00640 1 -- 172.21.15.129:0/2578659923 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70481f89a0 con 0x7f7048161d40 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f704cc00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f704000e990 tx=0x7f704000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f703fe00640 1 -- 172.21.15.129:0/2578659923 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040004d20 con 0x7f7048161d40 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f703fe00640 1 -- 172.21.15.129:0/2578659923 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f704000caa0 con 0x7f7048161d40 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f703fe00640 1 -- 172.21.15.129:0/2578659923 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040005020 con 0x7f7048161d40 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70481f8c80 con 0x7f7048161d40 2024-09-17T03:58:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.782+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70481f9240 con 0x7f7048161d40 2024-09-17T03:58:19.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.785+0000 7f703fe00640 1 -- 172.21.15.129:0/2578659923 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70400051c0 con 0x7f7048161d40 2024-09-17T03:58:19.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.786+0000 7f703fe00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 0x7f702407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.786+0000 7f703fe00640 1 -- 172.21.15.129:0/2578659923 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f704009b670 con 0x7f7048161d40 2024-09-17T03:58:19.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.787+0000 7f7047e00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 0x7f702407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.787+0000 7f7055c11740 1 Processor -- start 2024-09-17T03:58:19.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.787+0000 7f7055c11740 1 -- start start 2024-09-17T03:58:19.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.787+0000 7f7047e00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 0x7f702407b0a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f70480d3700 tx=0x7f7038009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4ec6270 0x5567f5296970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f5266f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567f51f0430 con 0x5567f4e9d3b0 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567f51e05d0 con 0x5567f4ea5590 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.788+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5567f51cfa00 con 0x5567f4ec6270 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57488/0 (socket says 172.21.15.129:57488) 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 -- 172.21.15.129:0/1364030476 learned_addr learned my addr 172.21.15.129:0/1364030476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:19.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047e00640 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4ec6270 0x5567f5296970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f704cc00640 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f5266f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 -- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4ec6270 msgr2=0x5567f5296970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4ec6270 0x5567f5296970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 -- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 msgr2=0x5567f5266f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f5266f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 -- 172.21.15.129:0/1364030476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f5198520 con 0x5567f4e9d3b0 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.789+0000 7f7047400640 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f7030034040 tx=0x7f70300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77723cd1fc26cd78 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.790+0000 7f7037e00640 1 -- 172.21.15.129:0/1364030476 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030040070 con 0x5567f4e9d3b0 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.790+0000 7f7037e00640 1 -- 172.21.15.129:0/1364030476 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7030004330 con 0x5567f4e9d3b0 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.790+0000 7f7037e00640 1 -- 172.21.15.129:0/1364030476 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030035040 con 0x5567f4e9d3b0 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.790+0000 7f7055c11740 1 -- 172.21.15.129:0/1364030476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 msgr2=0x5567f5296eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.790+0000 7f7055c11740 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f7030034040 tx=0x7f70300046c0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 -- 172.21.15.129:0/1364030476 shutdown_connections 2024-09-17T03:58:19.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f5266f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4e9d3b0 0x5567f5296eb0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 --2- 172.21.15.129:0/1364030476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4ec6270 0x5567f5296970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 -- 172.21.15.129:0/1364030476 >> 172.21.15.129:0/1364030476 conn(0x5567f4eba3c0 msgr2=0x5567f526c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 -- 172.21.15.129:0/1364030476 shutdown_connections 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.791+0000 7f7055c11740 1 -- 172.21.15.129:0/1364030476 wait complete. 2024-09-17T03:58:19.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.792+0000 7f7055c11740 1 -- start start 2024-09-17T03:58:19.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.792+0000 7f7055c11740 1 Processor -- start 2024-09-17T03:58:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.793+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52904b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.793+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f52ab1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7055c11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7055c11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567f5176330 con 0x5567f4ec6270 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567f51d3810 con 0x5567f4ea5590 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5567f4f40620 con 0x5567f4e9d3b0 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7055c11740 10 client.0 _ll_register_callbacks cb 0x5567f517b420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57492/0 (socket says 172.21.15.129:57492) 2024-09-17T03:58:19.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 -- 172.21.15.129:0/3700866013 learned_addr learned my addr 172.21.15.129:0/3700866013 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7047400640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f52ab1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7047e00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52904b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 -- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 msgr2=0x5567f52904b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52904b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 -- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 msgr2=0x5567f52ab1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f52ab1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 -- 172.21.15.129:0/3700866013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f4f31850 con 0x5567f4ec6270 2024-09-17T03:58:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f704cc00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f7040099ae0 tx=0x7f7040062b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1e960a3d9f7e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.794+0000 7f7034c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:58:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704001f040 con 0x5567f4ec6270 2024-09-17T03:58:19.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f704004e820 con 0x5567f4ec6270 2024-09-17T03:58:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7030004330 con 0x5567f4ec6270 2024-09-17T03:58:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f70400633a0 con 0x5567f4ec6270 2024-09-17T03:58:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 10 client.8313 fetch_fsmap learned FSMap version 115 2024-09-17T03:58:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x5567f4fa86b0 con 0x5567f4ec6270 2024-09-17T03:58:19.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7030004330 con 0x5567f4ec6270 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f704004ace0 con 0x5567f4ec6270 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 10 client.8313 fetch_fsmap finished waiting for FSMap version 115 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 10 client.8313 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 20 client.8313 populate_metadata read hostname 'smithi129' 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f4e84e40 0x5567f52c1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 4 client.8313 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.795+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 --> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567f51e2520 con 0x5567f4e84e40 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.796+0000 7f7047e00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f4e84e40 0x5567f52c1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.796+0000 7f7034c00640 10 client.8313 ms_handle_connect on v2:172.21.15.129:6833/431415571 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.796+0000 7f7047e00640 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f4e84e40 0x5567f52c1070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27174973b7520148 server_cookie=50993fcd37f3b187 in_seq=0 out_seq=0 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7034c00640 1 -- 172.21.15.129:0/3700866013 <== mds.0 v2:172.21.15.129:6833/431415571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5567f51e2520 con 0x5567f4e84e40 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7034c00640 10 client.8313 handle_command_reply: tid=0 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 client.8313 shutdown 2024-09-17T03:58:19.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 msgr2=0x5567f52c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f7040099ae0 tx=0x7f7040062b80 comp rx=0 tx=0).stop 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 shutdown_connections 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f4e84e40 0x5567f52c1070 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ec6270 0x5567f52c4480 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ea5590 0x5567f52ab1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7034c00640 0 client.8313 ms_handle_reset on v2:172.21.15.129:6833/431415571 2024-09-17T03:58:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 --2- 172.21.15.129:0/3700866013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52904b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.797+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 >> 172.21.15.129:0/3700866013 conn(0x5567f4eba3c0 msgr2=0x5567f52b1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.798+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 shutdown_connections 2024-09-17T03:58:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.798+0000 7f7055c11740 1 -- 172.21.15.129:0/3700866013 wait complete. 2024-09-17T03:58:19.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.798+0000 7f7055c11740 20 client.8313 trim_cache size 0 max 16384 2024-09-17T03:58:19.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.816+0000 7f7055c11740 1 Processor -- start 2024-09-17T03:58:19.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.816+0000 7f7055c11740 1 -- start start 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52932c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f5294220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70482010e0 con 0x5567f4ea5590 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70481f5cc0 con 0x5567f4ec6270 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70481f6310 con 0x5567f4e9d3b0 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57500/0 (socket says 172.21.15.129:57500) 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 -- 172.21.15.129:0/2332702325 learned_addr learned my addr 172.21.15.129:0/2332702325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047e00640 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52932c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f704cc00640 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f5294220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 -- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 msgr2=0x5567f52932c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52932c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 -- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 msgr2=0x5567f5294220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f5294220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.818+0000 7f7047400640 1 -- 172.21.15.129:0/2332702325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567f5262350 con 0x5567f4ea5590 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.819+0000 7f7047400640 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f703002abe0 tx=0x7f703002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adea1ef1938545e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.819+0000 7f7037400640 1 -- 172.21.15.129:0/2332702325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030035050 con 0x5567f4ea5590 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.819+0000 7f7037400640 1 -- 172.21.15.129:0/2332702325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70300047a0 con 0x5567f4ea5590 2024-09-17T03:58:19.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.819+0000 7f7037400640 1 -- 172.21.15.129:0/2332702325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030040040 con 0x5567f4ea5590 2024-09-17T03:58:19.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 -- 172.21.15.129:0/2332702325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 msgr2=0x5567f5293800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f703002abe0 tx=0x7f703002a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 -- 172.21.15.129:0/2332702325 shutdown_connections 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f5294220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5293800 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 --2- 172.21.15.129:0/2332702325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f52932c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 -- 172.21.15.129:0/2332702325 >> 172.21.15.129:0/2332702325 conn(0x5567f4eba3c0 msgr2=0x5567f5267c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 -- 172.21.15.129:0/2332702325 shutdown_connections 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.820+0000 7f7055c11740 1 -- 172.21.15.129:0/2332702325 wait complete. 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.821+0000 7f7055c11740 1 -- start start 2024-09-17T03:58:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.821+0000 7f7055c11740 1 Processor -- start 2024-09-17T03:58:19.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.822+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f53286e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.822+0000 7f7055c11740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7055c11740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f52956c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7055c11740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5567f52a9740 con 0x5567f4ea5590 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5567f5292800 con 0x5567f4ec6270 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7055c11740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5567f52a8c70 con 0x5567f4e9d3b0 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7055c11740 10 client.0 _ll_register_callbacks cb 0x7f7040063200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57510/0 (socket says 172.21.15.129:57510) 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 -- 172.21.15.129:0/2766503553 learned_addr learned my addr 172.21.15.129:0/2766503553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047e00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f53286e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f704cc00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f52956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 -- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 msgr2=0x5567f53286e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f53286e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 -- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 msgr2=0x5567f52956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f52956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 -- 172.21.15.129:0/2766503553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567f5328c20 con 0x5567f4ea5590 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047e00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f53286e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f704cc00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f52956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7047400640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f7030009850 tx=0x7f7030004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71ac5c6a0f58e7b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.823+0000 7f7034c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7030035040 con 0x5567f4ea5590 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f7030004f60 con 0x5567f4ea5590 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70300047a0 con 0x5567f4ea5590 2024-09-17T03:58:19.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=115) ==== 24+0+0 (secure 0 0 0) 0x7f7030060050 con 0x5567f4ea5590 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 10 client.8319 fetch_fsmap learned FSMap version 115 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=115,monmap=2+}) -- 0x5567f5332520 con 0x5567f4ea5590 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70300047a0 con 0x5567f4ea5590 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 115) ==== 3502+0+0 (secure 0 0 0) 0x7f7030063020 con 0x5567f4ea5590 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 10 client.8319 fetch_fsmap finished waiting for FSMap version 115 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 10 client.8319 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 20 client.8319 populate_metadata read hostname 'smithi129' 2024-09-17T03:58:19.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f52968b0 0x5567f5296cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 4 client.8319 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:58:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.824+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 --> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] -- command(tid 0: {"prefix": "status"}) -- 0x5567f51ddfd0 con 0x5567f52968b0 2024-09-17T03:58:19.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7047e00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f52968b0 0x5567f5296cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7047e00640 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f52968b0 0x5567f5296cc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e08c7faace0a69c server_cookie=c2a5c23c1652317e in_seq=0 out_seq=0 2024-09-17T03:58:19.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7034c00640 10 client.8319 ms_handle_connect on v2:172.21.15.129:6833/431415571 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7034c00640 1 -- 172.21.15.129:0/2766503553 <== mds.0 v2:172.21.15.129:6833/431415571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5567f51ddfd0 con 0x5567f52968b0 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7034c00640 10 client.8319 handle_command_reply: tid=0 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.825+0000 7f7055c11740 1 client.8319 shutdown 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 msgr2=0x5567f5331cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f7030009850 tx=0x7f7030004b60 comp rx=0 tx=0).stop 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 shutdown_connections 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5567f52968b0 0x5567f5296cc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5567f4ec6270 0x5567f52956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5567f4ea5590 0x5567f5331cf0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 --2- 172.21.15.129:0/2766503553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5567f4e9d3b0 0x5567f53286e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7034c00640 0 client.8319 ms_handle_reset on v2:172.21.15.129:6833/431415571 2024-09-17T03:58:19.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 >> 172.21.15.129:0/2766503553 conn(0x5567f4eba3c0 msgr2=0x5567f5291040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 shutdown_connections 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 1 -- 172.21.15.129:0/2766503553 wait complete. 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.826+0000 7f7055c11740 20 client.8319 trim_cache size 0 max 16384 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 8181, 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:58:19.830 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.6524769570000002, 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 115, 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 141, 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 139, 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.700223803 2024-09-17T03:58:19.831 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 msgr2=0x7f702407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 0x7f702407b0a0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f70480d3700 tx=0x7f7038009040 comp rx=0 tx=0).stop 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 msgr2=0x7f70480b81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f704000e990 tx=0x7f704000ee50 comp rx=0 tx=0).stop 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 shutdown_connections 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7024078b90 0x7f702407b0a0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7048161d40 0x7f70480b81f0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70480b9b40 0x7f70480b7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 --2- 172.21.15.129:0/2578659923 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70480b18b0 0x7f704815b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 >> 172.21.15.129:0/2578659923 conn(0x7f70480d4610 msgr2=0x7f704815ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 shutdown_connections 2024-09-17T03:58:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:19.827+0000 7f704ec00640 1 -- 172.21.15.129:0/2578659923 wait complete. 2024-09-17T03:58:19.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 -- 172.21.15.129:0/2261902530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80d0a50 msgr2=0x7faac8160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 --2- 172.21.15.129:0/2261902530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80d0a50 0x7faac8160210 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7faab4005fd0 tx=0x7faab402a540 comp rx=0 tx=0).stop 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 -- 172.21.15.129:0/2261902530 shutdown_connections 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 --2- 172.21.15.129:0/2261902530 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac81458a0 0x7faac8160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 --2- 172.21.15.129:0/2261902530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80d0a50 0x7faac8160210 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 -- 172.21.15.129:0/2261902530 >> 172.21.15.129:0/2261902530 conn(0x7faac806cfc0 msgr2=0x7faac806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 -- 172.21.15.129:0/2261902530 shutdown_connections 2024-09-17T03:58:20.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.074+0000 7faacf000640 1 -- 172.21.15.129:0/2261902530 wait complete. 2024-09-17T03:58:20.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.076+0000 7faacf000640 1 Processor -- start 2024-09-17T03:58:20.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.077+0000 7faacf000640 1 -- start start 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 0x7faac815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 0x7faac80cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faac80cef20 con 0x7faac80c82a0 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faac80ceda0 con 0x7faac81458a0 2024-09-17T03:58:20.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faacf000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faac8160c90 con 0x7faac8145e60 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faace600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faace600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57516/0 (socket says 172.21.15.129:57516) 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.078+0000 7faace600640 1 -- 172.21.15.129:0/1332150640 learned_addr learned my addr 172.21.15.129:0/1332150640 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacd200640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 0x7faac80cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacdc00640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 0x7faac815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 -- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 msgr2=0x7faac80cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 0x7faac80cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 -- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 msgr2=0x7faac815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:20.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 0x7faac815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 -- 172.21.15.129:0/1332150640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac80c7200 con 0x7faac80c82a0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacd200640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 0x7faac80cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacdc00640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 0x7faac815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faace600640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7faac000ea10 tx=0x7faac000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac000cde0 con 0x7faac80c82a0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faac0004c10 con 0x7faac80c82a0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac80c74e0 con 0x7faac80c82a0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0005020 con 0x7faac80c82a0 2024-09-17T03:58:20.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.079+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac80c7aa0 con 0x7faac80c82a0 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.080+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac8160210 con 0x7faac80c82a0 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.082+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faac00051c0 con 0x7faac80c82a0 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.082+0000 7faac5600640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 0x7faaa807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.083+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7faac009bca0 con 0x7faac80c82a0 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.083+0000 7faacdc00640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 0x7faaa807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.084+0000 7faacdc00640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 0x7faaa807b0a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faabc0016a0 tx=0x7faabc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.087+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faac0064020 con 0x7faac80c82a0 2024-09-17T03:58:20.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.307+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faac806a140 con 0x7faac80c82a0 2024-09-17T03:58:20.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.308+0000 7faac5600640 1 -- 172.21.15.129:0/1332150640 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v115) ==== 53+0+73 (secure 0 0 0) 0x7faac0063920 con 0x7faac80c82a0 2024-09-17T03:58:20.307 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:58:20.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 msgr2=0x7faaa807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:20.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 0x7faaa807b0a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7faabc0016a0 tx=0x7faabc009040 comp rx=0 tx=0).stop 2024-09-17T03:58:20.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 msgr2=0x7faac80c86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7faac000ea10 tx=0x7faac000eed0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 shutdown_connections 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faaa8078b90 0x7faaa807b0a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faac80c82a0 0x7faac80c86b0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faac8145e60 0x7faac80cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 --2- 172.21.15.129:0/1332150640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faac81458a0 0x7faac815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 >> 172.21.15.129:0/1332150640 conn(0x7faac806cfc0 msgr2=0x7faac80b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 shutdown_connections 2024-09-17T03:58:20.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20.310+0000 7faacf000640 1 -- 172.21.15.129:0/1332150640 wait complete. 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:58:20.332 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:20.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.333 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:58:20.368 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:58:20.369 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42192sec preferred_lft 42192sec 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:58:20.370 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:20.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.372 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:58:20.462 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:58:20.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.465 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:58:20.501 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:58:20.501 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:58:20.501 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:58:20.501 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:58:20.501 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:58:20.502 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:58:20.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:58:20.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:58:20.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.698 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:58:20.733 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:58:20.733 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:58:20.734 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:20.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:58:20.831 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:58:20.831 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:20.867 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:20.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:58:20.867 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:20.943 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:58:20.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:20.943 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:21.138 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:21.138 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:21.138 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:21.158 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:21.158 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:21.158 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:21.158 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:58:21.158 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:58:21.188 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:21.188 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:21.188 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:21.189 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:58:21.189 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:58:21.189 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:58:21.189 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:58:21.189 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:58:21.190 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:58:21.190 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:58:21.190 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:58:21.190 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:58:21.190 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:58:21.193 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:58:21.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:21.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:21.253 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:58:21.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:58:21.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:58:21.351 INFO:teuthology.orchestra.run.smithi129.stdout:client7849 2024-09-17T03:58:21.352 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:21.352 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:21.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 -- 172.21.15.129:0/9626610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c06eea0 msgr2=0x7f308c06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 --2- 172.21.15.129:0/9626610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c06eea0 0x7f308c06f2b0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f308c069b60 tx=0x7f308802aa30 comp rx=0 tx=0).stop 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 -- 172.21.15.129:0/9626610 shutdown_connections 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 --2- 172.21.15.129:0/9626610 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c06eea0 0x7f308c06f2b0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 --2- 172.21.15.129:0/9626610 >> v2:172.21.15.181:3300/0 conn(0x7f308c0b1c40 0x7f308c0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 --2- 172.21.15.129:0/9626610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c074800 0x7f308c0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 -- 172.21.15.129:0/9626610 >> 172.21.15.129:0/9626610 conn(0x7f308c0ccc40 msgr2=0x7f308c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 -- 172.21.15.129:0/9626610 shutdown_connections 2024-09-17T03:58:21.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.652+0000 7f3094800640 1 -- 172.21.15.129:0/9626610 wait complete. 2024-09-17T03:58:21.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.655+0000 7f3094800640 1 Processor -- start 2024-09-17T03:58:21.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.655+0000 7f3094800640 1 -- start start 2024-09-17T03:58:21.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 0x7f308c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:21.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:21.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f308c0b1c40 0x7f308c0ce2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:21.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f308c06f970 con 0x7f308c074800 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f308c06f7f0 con 0x7f308c0b1c40 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3094800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f308c06faf0 con 0x7f308c06eea0 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.656+0000 7f3091400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57538/0 (socket says 172.21.15.129:57538) 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 -- 172.21.15.129:0/2661774390 learned_addr learned my addr 172.21.15.129:0/2661774390 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3092800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f308c0b1c40 0x7f308c0ce2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091e00640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 0x7f308c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 -- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 msgr2=0x7f308c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 0x7f308c0cf780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 -- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f308c0b1c40 msgr2=0x7f308c0ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f308c0b1c40 0x7f308c0ce2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:21.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 -- 172.21.15.129:0/2661774390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308c1f5840 con 0x7f308c074800 2024-09-17T03:58:21.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091e00640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 0x7f308c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:21.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.657+0000 7f3091400640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f307c00e790 tx=0x7f307c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.658+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c00cac0 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.658+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f307c00cc60 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.658+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f308c1f5b20 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.658+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c005280 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.658+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f308c15d500 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.659+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f308c0af480 con 0x7f308c074800 2024-09-17T03:58:21.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.660+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f307c005420 con 0x7f308c074800 2024-09-17T03:58:21.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.661+0000 7f3086000640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 0x7f307007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:21.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.661+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f307c09b650 con 0x7f308c074800 2024-09-17T03:58:21.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.661+0000 7f3091e00640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 0x7f307007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:21.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.662+0000 7f3091e00640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 0x7f307007b0a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3080009b80 tx=0x7f3080005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:21.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.665+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f307c0639d0 con 0x7f308c074800 2024-09-17T03:58:21.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:21.885+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7f308c0cdf40 con 0x7f308c074800 2024-09-17T03:58:22.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.244+0000 7f3086000640 1 -- 172.21.15.129:0/2661774390 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v116) ==== 105+0+0 (secure 0 0 0) 0x7f307c0632d0 con 0x7f308c074800 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 msgr2=0x7f307007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 0x7f307007b0a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f3080009b80 tx=0x7f3080005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 msgr2=0x7f308c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f307c00e790 tx=0x7f307c00ec50 comp rx=0 tx=0).stop 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 shutdown_connections 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3070078b90 0x7f307007b0a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f308c0b1c40 0x7f308c0ce2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f308c074800 0x7f308c15a090 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 --2- 172.21.15.129:0/2661774390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f308c06eea0 0x7f308c0cf780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 >> 172.21.15.129:0/2661774390 conn(0x7f308c0ccc40 msgr2=0x7f308c15c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 shutdown_connections 2024-09-17T03:58:22.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.247+0000 7f3094800640 1 -- 172.21.15.129:0/2661774390 wait complete. 2024-09-17T03:58:22.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:22.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 -- 172.21.15.129:0/2428593568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c0cd390 msgr2=0x7ff37c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 --2- 172.21.15.129:0/2428593568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c0cd390 0x7ff37c1609c0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ff37c0d1370 tx=0x7ff37402aa30 comp rx=0 tx=0).stop 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 -- 172.21.15.129:0/2428593568 shutdown_connections 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 --2- 172.21.15.129:0/2428593568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c0cd390 0x7ff37c1609c0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 --2- 172.21.15.129:0/2428593568 >> v2:172.21.15.181:3300/0 conn(0x7ff37c0705a0 0x7ff37c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 -- 172.21.15.129:0/2428593568 >> 172.21.15.129:0/2428593568 conn(0x7ff37c06d410 msgr2=0x7ff37c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 -- 172.21.15.129:0/2428593568 shutdown_connections 2024-09-17T03:58:22.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.498+0000 7ff382200640 1 -- 172.21.15.129:0/2428593568 wait complete. 2024-09-17T03:58:22.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.500+0000 7ff382200640 1 Processor -- start 2024-09-17T03:58:22.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.500+0000 7ff382200640 1 -- start start 2024-09-17T03:58:22.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:22.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff37c0705a0 0x7ff37c1e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:22.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff37c1573a0 0x7ff37c15a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:22.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c160250 con 0x7ff37c06eed0 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c06f4f0 con 0x7ff37c0705a0 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff382200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff37c160f00 con 0x7ff37c1573a0 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57550/0 (socket says 172.21.15.129:57550) 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 -- 172.21.15.129:0/301051615 learned_addr learned my addr 172.21.15.129:0/301051615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 -- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff37c1573a0 msgr2=0x7ff37c15a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff37c1573a0 0x7ff37c15a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 -- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff37c0705a0 msgr2=0x7ff37c1e6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff37c0705a0 0x7ff37c1e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.502+0000 7ff37b400640 1 -- 172.21.15.129:0/301051615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c0bb770 con 0x7ff37c06eed0 2024-09-17T03:58:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff37b400640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00d6d0 tx=0x7ff36c00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:22.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c017070 con 0x7ff37c06eed0 2024-09-17T03:58:22.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff36c005330 con 0x7ff37c06eed0 2024-09-17T03:58:22.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c1e6770 con 0x7ff37c06eed0 2024-09-17T03:58:22.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c009820 con 0x7ff37c06eed0 2024-09-17T03:58:22.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.503+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c0bf850 con 0x7ff37c06eed0 2024-09-17T03:58:22.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.504+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff37c0b4140 con 0x7ff37c06eed0 2024-09-17T03:58:22.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.505+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff36c009bd0 con 0x7ff37c06eed0 2024-09-17T03:58:22.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.506+0000 7ff373e00640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 0x7ff35807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:22.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.506+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7ff36c09b140 con 0x7ff37c06eed0 2024-09-17T03:58:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.507+0000 7ff372a00640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 0x7ff35807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:22.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.508+0000 7ff372a00640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 0x7ff35807b0a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff364006fd0 tx=0x7ff364006bc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:22.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.511+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff36c0634c0 con 0x7ff37c06eed0 2024-09-17T03:58:22.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.730+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff37c06a140 con 0x7ff37c06eed0 2024-09-17T03:58:22.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.731+0000 7ff373e00640 1 -- 172.21.15.129:0/301051615 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+5255 (secure 0 0 0) 0x7ff36c062dc0 con 0x7ff37c06eed0 2024-09-17T03:58:22.731 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:22.731 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":117,"btime":"2024-09-17T03:58:22:248636+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3687707903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3687707903},{"type":"v1","addr":"172.21.15.181:6837","nonce":3687707903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/872498647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":872498647},{"type":"v1","addr":"172.21.15.181:6836","nonce":872498647}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:22.248566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:creating","state_seq":1,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:22.731 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 117 2024-09-17T03:58:22.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 msgr2=0x7ff35807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 0x7ff35807b0a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7ff364006fd0 tx=0x7ff364006bc0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 msgr2=0x7ff37c1e6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00d6d0 tx=0x7ff36c00db90 comp rx=0 tx=0).stop 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 shutdown_connections 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff358078b90 0x7ff35807b0a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff37c1573a0 0x7ff37c15a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff37c0705a0 0x7ff37c1e6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 --2- 172.21.15.129:0/301051615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff37c06eed0 0x7ff37c1e6120 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 >> 172.21.15.129:0/301051615 conn(0x7ff37c06d410 msgr2=0x7ff37c06fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 shutdown_connections 2024-09-17T03:58:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:22.733+0000 7ff382200640 1 -- 172.21.15.129:0/301051615 wait complete. 2024-09-17T03:58:22.760 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-09-17T03:58:27.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:27.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.990+0000 7f46b2000640 1 -- 172.21.15.129:0/43274289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac0d09d0 msgr2=0x7f46ac15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:27.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 --2- 172.21.15.129:0/43274289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac0d09d0 0x7f46ac15cbd0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f46a4005fd0 tx=0x7f46a402aaa0 comp rx=0 tx=0).stop 2024-09-17T03:58:27.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 -- 172.21.15.129:0/43274289 shutdown_connections 2024-09-17T03:58:27.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 --2- 172.21.15.129:0/43274289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac0d09d0 0x7f46ac15cbd0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:27.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 -- 172.21.15.129:0/43274289 >> 172.21.15.129:0/43274289 conn(0x7f46ac06cfc0 msgr2=0x7f46ac06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:27.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 -- 172.21.15.129:0/43274289 shutdown_connections 2024-09-17T03:58:27.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.991+0000 7f46b2000640 1 -- 172.21.15.129:0/43274289 wait complete. 2024-09-17T03:58:27.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.993+0000 7f46b2000640 1 Processor -- start 2024-09-17T03:58:27.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.993+0000 7f46b2000640 1 -- start start 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46ac0cfb60 0x7f46ac0c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ac15b930 0x7f46ac0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15d4a0 con 0x7f46ac160cf0 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46ac15d320 con 0x7f46ac15b930 2024-09-17T03:58:27.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f46ac15d620 con 0x7f46ac0cfb60 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57740/0 (socket says 172.21.15.129:57740) 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46abe00640 1 -- 172.21.15.129:0/1178912795 learned_addr learned my addr 172.21.15.129:0/1178912795 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b1600640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ac15b930 0x7f46ac0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46b0c00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46ac0cfb60 0x7f46ac0c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46abe00640 1 -- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46ac0cfb60 msgr2=0x7f46ac0c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.995+0000 7f46abe00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46ac0cfb60 0x7f46ac0c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46abe00640 1 -- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ac15b930 msgr2=0x7f46ac0c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46abe00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ac15b930 0x7f46ac0c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46abe00640 1 -- 172.21.15.129:0/1178912795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46ac15bd40 con 0x7f46ac160cf0 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46abe00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f469800c370 tx=0x7f469800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:27.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698017070 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46980035f0 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46ac0cbab0 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698003a40 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.996+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46ac0cc070 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.997+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46ac0cb820 con 0x7f46ac160cf0 2024-09-17T03:58:27.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.998+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f469801a070 con 0x7f46ac160cf0 2024-09-17T03:58:27.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:27.999+0000 7f46a8c00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 0x7f468c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.000+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f4698099fb0 con 0x7f46ac160cf0 2024-09-17T03:58:28.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.000+0000 7f46b0c00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 0x7f468c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.001+0000 7f46b0c00640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 0x7f468c07b0a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f469c000f60 tx=0x7f469c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:28.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.004+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f46ac15bd40 con 0x7f46ac160cf0 2024-09-17T03:58:28.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.224+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46ac15cbd0 con 0x7f46ac160cf0 2024-09-17T03:58:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.225+0000 7f46a8c00640 1 -- 172.21.15.129:0/1178912795 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5258 (secure 0 0 0) 0x7f4698061d70 con 0x7f46ac160cf0 2024-09-17T03:58:28.225 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:28.225 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":118,"btime":"2024-09-17T03:58:23:252508+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3687707903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3687707903},{"type":"v1","addr":"172.21.15.181:6837","nonce":3687707903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/872498647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":872498647},{"type":"v1","addr":"172.21.15.181:6836","nonce":872498647}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:23.252501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8178,8181]},"id":10}]} 2024-09-17T03:58:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 118 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 msgr2=0x7f468c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 0x7f468c07b0a0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f469c000f60 tx=0x7f469c009040 comp rx=0 tx=0).stop 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 msgr2=0x7f46ac15b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f469800c370 tx=0x7f469800c830 comp rx=0 tx=0).stop 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 shutdown_connections 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f468c078b90 0x7f468c07b0a0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46ac15b930 0x7f46ac0c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46ac160cf0 0x7f46ac15b3f0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 --2- 172.21.15.129:0/1178912795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46ac0cfb60 0x7f46ac0c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.227+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 >> 172.21.15.129:0/1178912795 conn(0x7f46ac06cfc0 msgr2=0x7f46ac153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:28.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.228+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 shutdown_connections 2024-09-17T03:58:28.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.228+0000 7f46b2000640 1 -- 172.21.15.129:0/1178912795 wait complete. 2024-09-17T03:58:28.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 -- 172.21.15.129:0/156055797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc3e0 msgr2=0x7f04c815f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 --2- 172.21.15.129:0/156055797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc3e0 0x7f04c815f060 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f04c80cc390 tx=0x7f04b402a420 comp rx=0 tx=0).stop 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 -- 172.21.15.129:0/156055797 shutdown_connections 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 --2- 172.21.15.129:0/156055797 >> v2:172.21.15.181:3300/0 conn(0x7f04c80cc910 0x7f04c80ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 --2- 172.21.15.129:0/156055797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc3e0 0x7f04c815f060 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 -- 172.21.15.129:0/156055797 >> 172.21.15.129:0/156055797 conn(0x7f04c806d410 msgr2=0x7f04c80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.470+0000 7f04ce800640 1 -- 172.21.15.129:0/156055797 shutdown_connections 2024-09-17T03:58:28.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.471+0000 7f04ce800640 1 -- 172.21.15.129:0/156055797 wait complete. 2024-09-17T03:58:28.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.473+0000 7f04ce800640 1 Processor -- start 2024-09-17T03:58:28.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.473+0000 7f04ce800640 1 -- start start 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04c8070a50 0x7f04c8153160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04c80cc3e0 0x7f04c80b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04c815f720 con 0x7f04c80cc910 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f04c815f5a0 con 0x7f04c8070a50 2024-09-17T03:58:28.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.474+0000 7f04ce800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f04c80af070 con 0x7f04c80cc3e0 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57746/0 (socket says 172.21.15.129:57746) 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 -- 172.21.15.129:0/680917782 learned_addr learned my addr 172.21.15.129:0/680917782 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7400640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04c8070a50 0x7f04c8153160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c6a00640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04c80cc3e0 0x7f04c80b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 -- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04c80cc3e0 msgr2=0x7f04c80b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04c80cc3e0 0x7f04c80b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 -- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04c8070a50 msgr2=0x7f04c8153160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04c8070a50 0x7f04c8153160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 -- 172.21.15.129:0/680917782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c81536a0 con 0x7f04c80cc910 2024-09-17T03:58:28.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.475+0000 7f04c7e00640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e990 tx=0x7f04c000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:28.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.476+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0004d20 con 0x7f04c80cc910 2024-09-17T03:58:28.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.476+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04c000caa0 con 0x7f04c80cc910 2024-09-17T03:58:28.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.476+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0005020 con 0x7f04c80cc910 2024-09-17T03:58:28.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.476+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c80ca0e0 con 0x7f04c80cc910 2024-09-17T03:58:28.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.476+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c8072cc0 con 0x7f04c80cc910 2024-09-17T03:58:28.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.477+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c815f060 con 0x7f04c80cc910 2024-09-17T03:58:28.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.478+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f04c00051c0 con 0x7f04c80cc910 2024-09-17T03:58:28.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.479+0000 7f04bf400640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 0x7f04a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.479+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f04c009b600 con 0x7f04c80cc910 2024-09-17T03:58:28.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.479+0000 7f04c7400640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 0x7f04a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.480+0000 7f04c7400640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 0x7f04a807b0a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f04b8001770 tx=0x7f04b8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:28.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.483+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f04c0016030 con 0x7f04c80cc910 2024-09-17T03:58:28.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.703+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04c806a140 con 0x7f04c80cc910 2024-09-17T03:58:28.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.704+0000 7f04bf400640 1 -- 172.21.15.129:0/680917782 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) ==== 95+0+5258 (secure 0 0 0) 0x7f04c0063aa0 con 0x7f04c80cc910 2024-09-17T03:58:28.704 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:28.704 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":118,"btime":"2024-09-17T03:58:23:252508+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6830,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3687707903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3687707903},{"type":"v1","addr":"172.21.15.181:6837","nonce":3687707903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110},{"gid":7729,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/872498647","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":872498647},{"type":"v1","addr":"172.21.15.181:6836","nonce":872498647}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":110}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:23.252501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8178,8181]},"id":10}]} 2024-09-17T03:58:28.705 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 118 2024-09-17T03:58:28.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 msgr2=0x7f04a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 0x7f04a807b0a0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f04b8001770 tx=0x7f04b8009040 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 msgr2=0x7f04c80b11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f04c000e990 tx=0x7f04c000ee50 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 shutdown_connections 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04a8078b90 0x7f04a807b0a0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04c80cc910 0x7f04c80b11e0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04c80cc3e0 0x7f04c80b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 --2- 172.21.15.129:0/680917782 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04c8070a50 0x7f04c8153160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.706+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 >> 172.21.15.129:0/680917782 conn(0x7f04c806d410 msgr2=0x7f04c80ca7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.707+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 shutdown_connections 2024-09-17T03:58:28.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.707+0000 7f04ce800640 1 -- 172.21.15.129:0/680917782 wait complete. 2024-09-17T03:58:28.729 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-09-17T03:58:28.729 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T03:58:28.730 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:58:28.754 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T03:58:28.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T03:58:28.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 -- 172.21.15.129:0/1122232171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d8003a00 msgr2=0x7f10d8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 --2- 172.21.15.129:0/1122232171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d8003a00 0x7f10d8003e70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f10f4072c70 tx=0x7f10d8005830 comp rx=0 tx=0).stop 2024-09-17T03:58:28.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10f3e00640 1 -- 172.21.15.129:0/1122232171 reap_dead start 2024-09-17T03:58:28.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 -- 172.21.15.129:0/1122232171 shutdown_connections 2024-09-17T03:58:28.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 -- 172.21.15.129:0/1122232171 >> 172.21.15.129:0/1122232171 conn(0x7f10f406d410 msgr2=0x7f10f40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:28.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 -- 172.21.15.129:0/1122232171 shutdown_connections 2024-09-17T03:58:28.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.978+0000 7f10fa400640 1 -- 172.21.15.129:0/1122232171 wait complete. 2024-09-17T03:58:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.980+0000 7f10fa400640 1 Processor -- start 2024-09-17T03:58:28.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.980+0000 7f10fa400640 1 -- start start 2024-09-17T03:58:28.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8003a00 0x7f10f4157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10f415da60 0x7f10f41f6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10f4160260 con 0x7f10f40c6cf0 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10f41600e0 con 0x7f10f415da60 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10fa400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10f41609c0 con 0x7f10d8003a00 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57774/0 (socket says 172.21.15.129:57774) 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 -- 172.21.15.129:0/1218937742 learned_addr learned my addr 172.21.15.129:0/1218937742 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10f3400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8003a00 0x7f10f4157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10f3e00640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10f415da60 0x7f10f41f6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 -- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8003a00 msgr2=0x7f10f4157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8003a00 0x7f10f4157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 -- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10f415da60 msgr2=0x7f10f41f6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.982+0000 7f10eaa00640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10f415da60 0x7f10f41f6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:28.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10eaa00640 1 -- 172.21.15.129:0/1218937742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f41598f0 con 0x7f10f40c6cf0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10eaa00640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f10dc011820 tx=0x7f10dc011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc004460 con 0x7f10f40c6cf0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10dc016070 con 0x7f10f40c6cf0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f4158850 con 0x7f10f40c6cf0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc00fb80 con 0x7f10f40c6cf0 2024-09-17T03:58:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.983+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f4158e10 con 0x7f10f40c6cf0 2024-09-17T03:58:28.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.984+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f40ccc80 con 0x7f10f40c6cf0 2024-09-17T03:58:28.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.986+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10dc018030 con 0x7f10f40c6cf0 2024-09-17T03:58:28.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.987+0000 7f10ebe00640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 0x7f10d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:28.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.987+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(141..141 src has 1..141) ==== 5738+0+0 (secure 0 0 0) 0x7f10dc09b4f0 con 0x7f10f40c6cf0 2024-09-17T03:58:28.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.987+0000 7f10f3400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 0x7f10d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:28.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.988+0000 7f10f3400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 0x7f10d007b0d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f10f4072c70 tx=0x7f10e4038040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:28.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:28.991+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10dc063220 con 0x7f10f40c6cf0 2024-09-17T03:58:29.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.253+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7f10f406a140 con 0x7f10f40c6cf0 2024-09-17T03:58:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.283+0000 7f10ebe00640 1 -- 172.21.15.129:0/1218937742 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 7729 v119) ==== 95+0+0 (secure 0 0 0) 0x7f10dc067c50 con 0x7f10f40c6cf0 2024-09-17T03:58:29.283 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 7729 2024-09-17T03:58:29.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 msgr2=0x7f10d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:29.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 0x7f10d007b0d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f10f4072c70 tx=0x7f10e4038040 comp rx=0 tx=0).stop 2024-09-17T03:58:29.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 msgr2=0x7f10f415d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f10dc011820 tx=0x7f10dc011ce0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 shutdown_connections 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10d0078bc0 0x7f10d007b0d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10f415da60 0x7f10f41f6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10f40c6cf0 0x7f10f415d520 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 --2- 172.21.15.129:0/1218937742 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8003a00 0x7f10f4157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 >> 172.21.15.129:0/1218937742 conn(0x7f10f406d410 msgr2=0x7f10f40ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 shutdown_connections 2024-09-17T03:58:29.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.286+0000 7f10fa400640 1 -- 172.21.15.129:0/1218937742 wait complete. 2024-09-17T03:58:29.311 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T03:58:29.312 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:58:29.335 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T03:58:29.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 -- 172.21.15.129:0/4090155965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c0ccc80 msgr2=0x7f519c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 --2- 172.21.15.129:0/4090155965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c0ccc80 0x7f519c0cd090 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5194005fd0 tx=0x7f519402a820 comp rx=0 tx=0).stop 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 -- 172.21.15.129:0/4090155965 shutdown_connections 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 --2- 172.21.15.129:0/4090155965 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c0ccc80 0x7f519c0cd090 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 --2- 172.21.15.129:0/4090155965 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0cf2b0 0x7f519c0cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.560+0000 7f51a2800640 1 -- 172.21.15.129:0/4090155965 >> 172.21.15.129:0/4090155965 conn(0x7f519c06d410 msgr2=0x7f519c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.561+0000 7f51a2800640 1 -- 172.21.15.129:0/4090155965 shutdown_connections 2024-09-17T03:58:29.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.561+0000 7f51a2800640 1 -- 172.21.15.129:0/4090155965 wait complete. 2024-09-17T03:58:29.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.563+0000 7f51a2800640 1 Processor -- start 2024-09-17T03:58:29.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.563+0000 7f51a2800640 1 -- start start 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0ccc80 0x7f519c09d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f519c0cf2b0 0x7f519c09dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f519c160ba0 con 0x7f519c099570 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f519c160a20 con 0x7f519c0cf2b0 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f51a2800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f519c160d20 con 0x7f519c0ccc80 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.564+0000 7f519b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57800/0 (socket says 172.21.15.129:57800) 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 -- 172.21.15.129:0/918440576 learned_addr learned my addr 172.21.15.129:0/918440576 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:29.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519aa00640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0ccc80 0x7f519c09d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519be00640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f519c0cf2b0 0x7f519c09dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 -- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0ccc80 msgr2=0x7f519c09d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0ccc80 0x7f519c09d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 -- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f519c0cf2b0 msgr2=0x7f519c09dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f519c0cf2b0 0x7f519c09dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:29.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 -- 172.21.15.129:0/918440576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c1489d0 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.565+0000 7f519b400640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f518c00c3c0 tx=0x7f518c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.566+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c017070 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.566+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f518c0035f0 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.566+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c003a80 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.566+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c1469e0 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.566+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c146fa0 con 0x7f519c099570 2024-09-17T03:58:29.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.567+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f519c148ba0 con 0x7f519c099570 2024-09-17T03:58:29.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.568+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f518c01a070 con 0x7f519c099570 2024-09-17T03:58:29.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.569+0000 7f5193400640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 0x7f517c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:29.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.569+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(142..142 src has 1..142) ==== 5738+0+0 (secure 0 0 0) 0x7f518c099fd0 con 0x7f519c099570 2024-09-17T03:58:29.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.570+0000 7f519aa00640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 0x7f517c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:29.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.570+0000 7f519aa00640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 0x7f517c07b0a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f519c1f6a60 tx=0x7f518800b390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:29.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.574+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f518c061d90 con 0x7f519c099570 2024-09-17T03:58:29.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:29.829+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7f519c161240 con 0x7f519c099570 2024-09-17T03:58:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.285+0000 7f5193400640 1 -- 172.21.15.129:0/918440576 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6830 v120) ==== 95+0+0 (secure 0 0 0) 0x7f518c0667c0 con 0x7f519c099570 2024-09-17T03:58:30.285 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 6830 2024-09-17T03:58:30.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 msgr2=0x7f517c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 0x7f517c07b0a0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f519c1f6a60 tx=0x7f518800b390 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 msgr2=0x7f519c09d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f518c00c3c0 tx=0x7f518c00c880 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 shutdown_connections 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f517c078b90 0x7f517c07b0a0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f519c0cf2b0 0x7f519c09dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f519c0ccc80 0x7f519c09d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 --2- 172.21.15.129:0/918440576 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f519c099570 0x7f519c09d210 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 >> 172.21.15.129:0/918440576 conn(0x7f519c06d410 msgr2=0x7f519c0b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 shutdown_connections 2024-09-17T03:58:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.288+0000 7f51a2800640 1 -- 172.21.15.129:0/918440576 wait complete. 2024-09-17T03:58:30.316 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c ' 2024-09-17T03:58:30.317 DEBUG:teuthology.orchestra.run.smithi129:> mkdir dir_1 dir_2 2024-09-17T03:58:30.317 DEBUG:teuthology.orchestra.run.smithi129:> touch dir_1/original 2024-09-17T03:58:30.317 DEBUG:teuthology.orchestra.run.smithi129:> ln dir_1/original dir_2/linkto 2024-09-17T03:58:30.317 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:58:30.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:58:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:58:30.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:30.430 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 dir_1) 2024-09-17T03:58:30.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:30.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 -- 172.21.15.129:0/843454827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 msgr2=0x7fe6141609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 --2- 172.21.15.129:0/843454827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141609c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe614072080 tx=0x7fe60c02aa30 comp rx=0 tx=0).stop 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 -- 172.21.15.129:0/843454827 shutdown_connections 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 --2- 172.21.15.129:0/843454827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141609c0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 --2- 172.21.15.129:0/843454827 >> v2:172.21.15.181:3300/0 conn(0x7fe6140b4140 0x7fe6140ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 -- 172.21.15.129:0/843454827 >> 172.21.15.129:0/843454827 conn(0x7fe61406d410 msgr2=0x7fe6140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 -- 172.21.15.129:0/843454827 shutdown_connections 2024-09-17T03:58:30.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.731+0000 7fe61b600640 1 -- 172.21.15.129:0/843454827 wait complete. 2024-09-17T03:58:30.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.733+0000 7fe61b600640 1 Processor -- start 2024-09-17T03:58:30.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.733+0000 7fe61b600640 1 -- start start 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6140705a0 0x7fe61415e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6140b4140 0x7fe614157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe614160260 con 0x7fe6140cd1c0 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe6141600e0 con 0x7fe6140705a0 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe61b600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe614160f00 con 0x7fe6140b4140 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe619600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe619600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57810/0 (socket says 172.21.15.129:57810) 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe619600640 1 -- 172.21.15.129:0/3198509011 learned_addr learned my addr 172.21.15.129:0/3198509011 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe60be00640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6140b4140 0x7fe614157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:30.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.735+0000 7fe618c00640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6140705a0 0x7fe61415e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 -- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6140b4140 msgr2=0x7fe614157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6140b4140 0x7fe614157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 -- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6140705a0 msgr2=0x7fe61415e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6140705a0 0x7fe61415e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 -- 172.21.15.129:0/3198509011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6141f9990 con 0x7fe6140cd1c0 2024-09-17T03:58:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe619600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fe60c02b020 tx=0x7fe60c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c033050 con 0x7fe6140cd1c0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe60c004450 con 0x7fe6140cd1c0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6141f9c70 con 0x7fe6140cd1c0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c038450 con 0x7fe6140cd1c0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.736+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6141fa230 con 0x7fe6140cd1c0 2024-09-17T03:58:30.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.737+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6140ccc80 con 0x7fe6140cd1c0 2024-09-17T03:58:30.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.739+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe60c036070 con 0x7fe6140cd1c0 2024-09-17T03:58:30.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.739+0000 7fe611600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 0x7fe5f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:30.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.740+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fe60c0bc240 con 0x7fe6140cd1c0 2024-09-17T03:58:30.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.740+0000 7fe618c00640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 0x7fe5f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:30.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.741+0000 7fe618c00640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 0x7fe5f007b0a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:30.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.744+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe6140cd5d0 con 0x7fe6140cd1c0 2024-09-17T03:58:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.960+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe61406a140 con 0x7fe6140cd1c0 2024-09-17T03:58:30.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.961+0000 7fe611600640 1 -- 172.21.15.129:0/3198509011 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+3604 (secure 0 0 0) 0x7fe60c083f70 con 0x7fe6140cd1c0 2024-09-17T03:58:30.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:30.962 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":120,"btime":"2024-09-17T03:58:30:284303+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:23.252501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8178,8181]},"id":10}]} 2024-09-17T03:58:30.962 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 120 2024-09-17T03:58:30.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 msgr2=0x7fe5f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 0x7fe5f007b0a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fe604006fd0 tx=0x7fe604009060 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 msgr2=0x7fe6141e7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fe60c02b020 tx=0x7fe60c003f10 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 shutdown_connections 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe5f0078b90 0x7fe5f007b0a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6140cd1c0 0x7fe6141e7720 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6140b4140 0x7fe614157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 --2- 172.21.15.129:0/3198509011 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6140705a0 0x7fe61415e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 >> 172.21.15.129:0/3198509011 conn(0x7fe61406d410 msgr2=0x7fe6140ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:30.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 shutdown_connections 2024-09-17T03:58:30.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:30.963+0000 7fe61b600640 1 -- 172.21.15.129:0/3198509011 wait complete. 2024-09-17T03:58:30.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T03:58:31.227 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:58:31.227 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:58:31.228 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-09-17T03:58:33.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:33.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 -- 172.21.15.129:0/2890661916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1458a0 msgr2=0x7fda5c160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:33.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 --2- 172.21.15.129:0/2890661916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1458a0 0x7fda5c160750 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0b3980 tx=0x7fda5402aa30 comp rx=0 tx=0).stop 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 -- 172.21.15.129:0/2890661916 shutdown_connections 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 --2- 172.21.15.129:0/2890661916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1458a0 0x7fda5c160750 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 --2- 172.21.15.129:0/2890661916 >> v2:172.21.15.181:3300/0 conn(0x7fda5c0d0a50 0x7fda5c160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 -- 172.21.15.129:0/2890661916 >> 172.21.15.129:0/2890661916 conn(0x7fda5c06cfc0 msgr2=0x7fda5c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 -- 172.21.15.129:0/2890661916 shutdown_connections 2024-09-17T03:58:33.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.461+0000 7fda61400640 1 -- 172.21.15.129:0/2890661916 wait complete. 2024-09-17T03:58:33.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.463+0000 7fda61400640 1 Processor -- start 2024-09-17T03:58:33.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.463+0000 7fda61400640 1 -- start start 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda5c1458a0 0x7fda5c139290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda5c145e60 0x7fda5c15f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0cef20 con 0x7fda5c1f89b0 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fda5c0ceda0 con 0x7fda5c1458a0 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda61400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fda5c160c90 con 0x7fda5c145e60 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:33.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57828/0 (socket says 172.21.15.129:57828) 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 -- 172.21.15.129:0/2853636856 learned_addr learned my addr 172.21.15.129:0/2853636856 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda5be00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda5c1458a0 0x7fda5c139290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda53400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda5c145e60 0x7fda5c15f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 -- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda5c145e60 msgr2=0x7fda5c15f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda5c145e60 0x7fda5c15f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 -- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda5c1458a0 msgr2=0x7fda5c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda5c1458a0 0x7fda5c139290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.465+0000 7fda60a00640 1 -- 172.21.15.129:0/2853636856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5c1f9300 con 0x7fda5c1f89b0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda60a00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fda54009d60 tx=0x7fda54003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54033050 con 0x7fda5c1f89b0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fda54004550 con 0x7fda5c1f89b0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54038450 con 0x7fda5c1f89b0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5c0cbfc0 con 0x7fda5c1f89b0 2024-09-17T03:58:33.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.466+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5c0cc2a0 con 0x7fda5c1f89b0 2024-09-17T03:58:33.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.467+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda5c160210 con 0x7fda5c1f89b0 2024-09-17T03:58:33.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.469+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fda54036070 con 0x7fda5c1f89b0 2024-09-17T03:58:33.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.469+0000 7fda58c00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 0x7fda3807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:33.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.470+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fda540bc490 con 0x7fda5c1f89b0 2024-09-17T03:58:33.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.470+0000 7fda5be00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 0x7fda3807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:33.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.471+0000 7fda5be00640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 0x7fda3807b0a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fda4c007fd0 tx=0x7fda4c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:33.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.474+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fda5c1f9300 con 0x7fda5c1f89b0 2024-09-17T03:58:33.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.691+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda5c0cbd50 con 0x7fda5c1f89b0 2024-09-17T03:58:33.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.692+0000 7fda58c00640 1 -- 172.21.15.129:0/2853636856 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+3604 (secure 0 0 0) 0x7fda540848b0 con 0x7fda5c1f89b0 2024-09-17T03:58:33.692 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":120,"btime":"2024-09-17T03:58:30:284303+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:23.252501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8178,8181]},"id":10}]} 2024-09-17T03:58:33.692 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 120 2024-09-17T03:58:33.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 msgr2=0x7fda3807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 0x7fda3807b0a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fda4c007fd0 tx=0x7fda4c009040 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 msgr2=0x7fda5c1f8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fda54009d60 tx=0x7fda54003f10 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 shutdown_connections 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fda38078b90 0x7fda3807b0a0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fda5c1f89b0 0x7fda5c1f8dc0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fda5c145e60 0x7fda5c15f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 --2- 172.21.15.129:0/2853636856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda5c1458a0 0x7fda5c139290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 >> 172.21.15.129:0/2853636856 conn(0x7fda5c06cfc0 msgr2=0x7fda5c0b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 shutdown_connections 2024-09-17T03:58:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:33.694+0000 7fda61400640 1 -- 172.21.15.129:0/2853636856 wait complete. 2024-09-17T03:58:33.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T03:58:33.955 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:58:33.955 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:58:33.956 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-09-17T03:58:35.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 -- 172.21.15.129:0/1813483340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 msgr2=0x7f3c440cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 --2- 172.21.15.129:0/1813483340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cee20 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f3c30009830 tx=0x7f3c3002a510 comp rx=0 tx=0).stop 2024-09-17T03:58:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 -- 172.21.15.129:0/1813483340 shutdown_connections 2024-09-17T03:58:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 --2- 172.21.15.129:0/1813483340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cee20 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 --2- 172.21.15.129:0/1813483340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c4415a220 0x7f3c4415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 -- 172.21.15.129:0/1813483340 >> 172.21.15.129:0/1813483340 conn(0x7f3c44069150 msgr2=0x7f3c44069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:36.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 -- 172.21.15.129:0/1813483340 shutdown_connections 2024-09-17T03:58:36.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.190+0000 7f3c48c00640 1 -- 172.21.15.129:0/1813483340 wait complete. 2024-09-17T03:58:36.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.192+0000 7f3c48c00640 1 Processor -- start 2024-09-17T03:58:36.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.192+0000 7f3c48c00640 1 -- start start 2024-09-17T03:58:36.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c4415a220 0x7f3c440b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:36.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:36.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c441606c0 0x7f3c440b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c440cf6f0 con 0x7f3c441602b0 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3c440cf570 con 0x7f3c4415a220 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c48c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3c440d2200 con 0x7f3c441606c0 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39912/0 (socket says 172.21.15.129:39912) 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 -- 172.21.15.129:0/2729920506 learned_addr learned my addr 172.21.15.129:0/2729920506 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c43400640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c4415a220 0x7f3c440b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:36.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c43e00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c441606c0 0x7f3c440b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 -- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c441606c0 msgr2=0x7f3c440b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c441606c0 0x7f3c440b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 -- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c4415a220 msgr2=0x7f3c440b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c4415a220 0x7f3c440b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.194+0000 7f3c42a00640 1 -- 172.21.15.129:0/2729920506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c440b1530 con 0x7f3c441602b0 2024-09-17T03:58:36.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c42a00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3c3002ab00 tx=0x7f3c30033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30005c90 con 0x7f3c441602b0 2024-09-17T03:58:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c30031e50 con 0x7f3c441602b0 2024-09-17T03:58:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c440b7270 con 0x7f3c441602b0 2024-09-17T03:58:36.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30005260 con 0x7f3c441602b0 2024-09-17T03:58:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.195+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c441f35c0 con 0x7f3c441602b0 2024-09-17T03:58:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.196+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c440cbf70 con 0x7f3c441602b0 2024-09-17T03:58:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.197+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3c30005400 con 0x7f3c441602b0 2024-09-17T03:58:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.198+0000 7f3c3b400640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 0x7f3c2407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:36.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.199+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f3c300bbf50 con 0x7f3c441602b0 2024-09-17T03:58:36.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.199+0000 7f3c43400640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 0x7f3c2407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:36.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.200+0000 7f3c43400640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 0x7f3c2407b0d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ac0 tx=0x7f3c34005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:36.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.203+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3c3003f030 con 0x7f3c441602b0 2024-09-17T03:58:36.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.421+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c440d09e0 con 0x7f3c441602b0 2024-09-17T03:58:36.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.422+0000 7f3c3b400640 1 -- 172.21.15.129:0/2729920506 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) ==== 95+0+3604 (secure 0 0 0) 0x7f3c30084360 con 0x7f3c441602b0 2024-09-17T03:58:36.422 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:36.422 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":120,"btime":"2024-09-17T03:58:30:284303+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:23.252501+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8178,8181]},"id":10}]} 2024-09-17T03:58:36.422 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 120 2024-09-17T03:58:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 msgr2=0x7f3c2407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 0x7f3c2407b0d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f3c34009ac0 tx=0x7f3c34005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 msgr2=0x7f3c440cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f3c3002ab00 tx=0x7f3c30033c20 comp rx=0 tx=0).stop 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 shutdown_connections 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c24078bc0 0x7f3c2407b0d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c441606c0 0x7f3c440b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c441602b0 0x7f3c440cb510 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 --2- 172.21.15.129:0/2729920506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c4415a220 0x7f3c440b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 >> 172.21.15.129:0/2729920506 conn(0x7f3c44069150 msgr2=0x7f3c4415dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 shutdown_connections 2024-09-17T03:58:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:36.424+0000 7f3c48c00640 1 -- 172.21.15.129:0/2729920506 wait complete. 2024-09-17T03:58:36.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T03:58:36.687 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-09-17T03:58:36.688 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:58:36.688 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_1', 1)] 2024-09-17T03:58:38.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 -- 172.21.15.129:0/1685665254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc3e0 msgr2=0x7f39cc15f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 --2- 172.21.15.129:0/1685665254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc15f060 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f39b8009830 tx=0x7f39b802a510 comp rx=0 tx=0).stop 2024-09-17T03:58:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 -- 172.21.15.129:0/1685665254 shutdown_connections 2024-09-17T03:58:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 --2- 172.21.15.129:0/1685665254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc15f060 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 -- 172.21.15.129:0/1685665254 >> 172.21.15.129:0/1685665254 conn(0x7f39cc06d410 msgr2=0x7f39cc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 -- 172.21.15.129:0/1685665254 shutdown_connections 2024-09-17T03:58:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.921+0000 7f39d2000640 1 -- 172.21.15.129:0/1685665254 wait complete. 2024-09-17T03:58:38.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.923+0000 7f39d2000640 1 Processor -- start 2024-09-17T03:58:38.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.923+0000 7f39d2000640 1 -- start start 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 0x7f39cc1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc0af1f0 con 0x7f39cc0cc910 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f39cc0af070 con 0x7f39cc0cc3e0 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39d2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f39cc0af370 con 0x7f39cc070a50 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39936/0 (socket says 172.21.15.129:39936) 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39cbe00640 1 -- 172.21.15.129:0/3965189072 learned_addr learned my addr 172.21.15.129:0/3965189072 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39caa00640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.925+0000 7f39cb400640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 0x7f39cc1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:38.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 -- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 msgr2=0x7f39cc1f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 0x7f39cc1f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 -- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39cc0cc3e0 msgr2=0x7f39cc153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 -- 172.21.15.129:0/3965189072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39cc0ca2f0 con 0x7f39cc0cc910 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cb400640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 0x7f39cc1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39cbe00640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b9e0 tx=0x7f39c400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c4003b70 con 0x7f39cc0cc910 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39c4003e00 con 0x7f39cc0cc910 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39cc0ca010 con 0x7f39cc0cc910 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c400cb80 con 0x7f39cc0cc910 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.926+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39cc1f6070 con 0x7f39cc0cc910 2024-09-17T03:58:38.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.928+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39cc0c9b30 con 0x7f39cc0cc910 2024-09-17T03:58:38.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.929+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39c4004000 con 0x7f39cc0cc910 2024-09-17T03:58:38.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.930+0000 7f39c3400640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 0x7f39ac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:38.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.930+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f39c409ac10 con 0x7f39cc0cc910 2024-09-17T03:58:38.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.930+0000 7f39cb400640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 0x7f39ac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:38.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.931+0000 7f39cb400640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 0x7f39ac07b0d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f39bc000f60 tx=0x7f39bc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:38.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:38.935+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f39cc0c9b30 con 0x7f39cc0cc910 2024-09-17T03:58:39.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.150+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39cc0af810 con 0x7f39cc0cc910 2024-09-17T03:58:39.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.151+0000 7f39c3400640 1 -- 172.21.15.129:0/3965189072 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3605 (secure 0 0 0) 0x7f39c4062940 con 0x7f39cc0cc910 2024-09-17T03:58:39.151 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:39.151 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":121,"btime":"2024-09-17T03:58:38:521272+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:38.130092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181,8178]},"id":10}]} 2024-09-17T03:58:39.152 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 121 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 msgr2=0x7f39ac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 0x7f39ac07b0d0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f39bc000f60 tx=0x7f39bc009040 comp rx=0 tx=0).stop 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 msgr2=0x7f39cc1f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f39c400b9e0 tx=0x7f39c400bea0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 shutdown_connections 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f39ac078bc0 0x7f39ac07b0d0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39cc0cc910 0x7f39cc1f6e40 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39cc0cc3e0 0x7f39cc153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 --2- 172.21.15.129:0/3965189072 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39cc070a50 0x7f39cc1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 >> 172.21.15.129:0/3965189072 conn(0x7f39cc06d410 msgr2=0x7f39cc1629a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:39.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.153+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 shutdown_connections 2024-09-17T03:58:39.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.154+0000 7f39d2000640 1 -- 172.21.15.129:0/3965189072 wait complete. 2024-09-17T03:58:39.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T03:58:39.415 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"14","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-09-17T03:58:39.416 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/dir_1", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:58:39.416 INFO:tasks.cephfs.cephfs_test_case:[('/dir_1', 1)] =?= [('/dir_1', 1)] 2024-09-17T03:58:39.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:39.417 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 dir_2) 2024-09-17T03:58:39.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 -- 172.21.15.129:0/2801321823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c10162d00 msgr2=0x7f8c100cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 --2- 172.21.15.129:0/2801321823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c10162d00 0x7f8c100cf390 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8005fd0 tx=0x7f8bf802a5d0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 -- 172.21.15.129:0/2801321823 shutdown_connections 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 --2- 172.21.15.129:0/2801321823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c10162d00 0x7f8c100cf390 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 --2- 172.21.15.129:0/2801321823 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c100cd370 0x7f8c100b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 -- 172.21.15.129:0/2801321823 >> 172.21.15.129:0/2801321823 conn(0x7f8c1006d410 msgr2=0x7f8c100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 -- 172.21.15.129:0/2801321823 shutdown_connections 2024-09-17T03:58:39.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.714+0000 7f8c15600640 1 -- 172.21.15.129:0/2801321823 wait complete. 2024-09-17T03:58:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.717+0000 7f8c15600640 1 Processor -- start 2024-09-17T03:58:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.717+0000 7f8c15600640 1 -- start start 2024-09-17T03:58:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c10162d00 0x7f8c1015c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 0x7f8c10160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:39.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c100cb150 con 0x7f8c100cd370 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8c100cafd0 con 0x7f8c10162d00 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c15600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8c1015dce0 con 0x7f8c10163110 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c0ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c0ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39954/0 (socket says 172.21.15.129:39954) 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c0ea00640 1 -- 172.21.15.129:0/2852361440 learned_addr learned my addr 172.21.15.129:0/2852361440 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.718+0000 7f8c06000640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c10162d00 0x7f8c1015c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0f400640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 0x7f8c10160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 -- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 msgr2=0x7f8c10160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 0x7f8c10160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 -- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c10162d00 msgr2=0x7f8c1015c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c10162d00 0x7f8c1015c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 -- 172.21.15.129:0/2852361440 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c101f6df0 con 0x7f8c100cd370 2024-09-17T03:58:39.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0f400640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 0x7f8c10160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c0ea00640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000d7a0 tx=0x7f8c0000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00017070 con 0x7f8c100cd370 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c00009d20 con 0x7f8c100cd370 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c101f6030 con 0x7f8c100cd370 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.719+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c00004280 con 0x7f8c100cd370 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.720+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c101f65f0 con 0x7f8c100cd370 2024-09-17T03:58:39.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.721+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c10050f60 con 0x7f8c100cd370 2024-09-17T03:58:39.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.722+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8c00005020 con 0x7f8c100cd370 2024-09-17T03:58:39.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.723+0000 7f8c07400640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:39.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.723+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f8c0009b040 con 0x7f8c100cd370 2024-09-17T03:58:39.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.723+0000 7f8c06000640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:39.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.724+0000 7f8c06000640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:39.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.727+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8c10050f60 con 0x7f8c100cd370 2024-09-17T03:58:39.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.946+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8c100cf390 con 0x7f8c100cd370 2024-09-17T03:58:39.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.947+0000 7f8c07400640 1 -- 172.21.15.129:0/2852361440 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3605 (secure 0 0 0) 0x7f8c100cf390 con 0x7f8c100cd370 2024-09-17T03:58:39.947 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:39.947 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":121,"btime":"2024-09-17T03:58:38:521272+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:38.130092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181,8178]},"id":10}]} 2024-09-17T03:58:39.948 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 121 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.949+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 msgr2=0x7f8bec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.949+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8033000 tx=0x7f8bf8004730 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.949+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 msgr2=0x7f8c1015d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.949+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000d7a0 tx=0x7f8c0000dc60 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 shutdown_connections 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c10163110 0x7f8c10160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c10162d00 0x7f8c1015c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 --2- 172.21.15.129:0/2852361440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c100cd370 0x7f8c1015d590 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:39.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 >> 172.21.15.129:0/2852361440 conn(0x7f8c1006d410 msgr2=0x7f8c100b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:39.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 shutdown_connections 2024-09-17T03:58:39.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:39.950+0000 7f8c15600640 1 -- 172.21.15.129:0/2852361440 wait complete. 2024-09-17T03:58:39.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:58:40.215 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-09-17T03:58:40.215 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-09-17T03:58:40.216 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-09-17T03:58:42.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:42.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 -- 172.21.15.129:0/3793793983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0b4140 msgr2=0x7f8cac0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 --2- 172.21.15.129:0/3793793983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0b4140 0x7f8cac0ccc80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f8c98005fd0 tx=0x7f8c9802a670 comp rx=0 tx=0).stop 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 -- 172.21.15.129:0/3793793983 shutdown_connections 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 --2- 172.21.15.129:0/3793793983 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0b4140 0x7f8cac0ccc80 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 -- 172.21.15.129:0/3793793983 >> 172.21.15.129:0/3793793983 conn(0x7f8cac06d410 msgr2=0x7f8cac0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 -- 172.21.15.129:0/3793793983 shutdown_connections 2024-09-17T03:58:42.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.449+0000 7f8cb1400640 1 -- 172.21.15.129:0/3793793983 wait complete. 2024-09-17T03:58:42.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.451+0000 7f8cb1400640 1 Processor -- start 2024-09-17T03:58:42.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.452+0000 7f8cb1400640 1 -- start start 2024-09-17T03:58:42.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cac0b4140 0x7f8cac15a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cac0cd1c0 0x7f8cac15a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac160260 con 0x7f8cac0705a0 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac1600e0 con 0x7f8cac0cd1c0 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8cb1400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8cac1610d0 con 0x7f8cac0b4140 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8caa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8caa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39966/0 (socket says 172.21.15.129:39966) 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.453+0000 7f8caa800640 1 -- 172.21.15.129:0/1764052243 learned_addr learned my addr 172.21.15.129:0/1764052243 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8ca9e00640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cac0b4140 0x7f8cac15a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8cab200640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cac0cd1c0 0x7f8cac15a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:42.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 -- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cac0b4140 msgr2=0x7f8cac15a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cac0b4140 0x7f8cac15a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 -- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cac0cd1c0 msgr2=0x7f8cac15a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cac0cd1c0 0x7f8cac15a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 -- 172.21.15.129:0/1764052243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cac0cc450 con 0x7f8cac0705a0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8caa800640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c780 tx=0x7f8c9c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c003b70 con 0x7f8cac0705a0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8c9c003d10 con 0x7f8cac0705a0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cac0c66d0 con 0x7f8cac0705a0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c00aab0 con 0x7f8cac0705a0 2024-09-17T03:58:42.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.454+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cac0c6c90 con 0x7f8cac0705a0 2024-09-17T03:58:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.456+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cac0b2890 con 0x7f8cac0705a0 2024-09-17T03:58:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.457+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8c9c00ac50 con 0x7f8cac0705a0 2024-09-17T03:58:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.458+0000 7f8ca2a00640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 0x7f8c8c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.458+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f8c9c09ac50 con 0x7f8cac0705a0 2024-09-17T03:58:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.458+0000 7f8ca9e00640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 0x7f8c8c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:42.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.459+0000 7f8ca9e00640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 0x7f8c8c07b0d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8c98033000 tx=0x7f8c98004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:42.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.462+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8c9c062fd0 con 0x7f8cac0705a0 2024-09-17T03:58:42.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.683+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8cac0ccc80 con 0x7f8cac0705a0 2024-09-17T03:58:42.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.683+0000 7f8ca2a00640 1 -- 172.21.15.129:0/1764052243 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3605 (secure 0 0 0) 0x7f8c9c0628d0 con 0x7f8cac0705a0 2024-09-17T03:58:42.684 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:42.684 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":121,"btime":"2024-09-17T03:58:38:521272+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:38.130092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181,8178]},"id":10}]} 2024-09-17T03:58:42.684 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 121 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.685+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 msgr2=0x7f8c8c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.685+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 0x7f8c8c07b0d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f8c98033000 tx=0x7f8c98004730 comp rx=0 tx=0).stop 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.685+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 msgr2=0x7f8cac1f5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.685+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c00c780 tx=0x7f8c9c00cc40 comp rx=0 tx=0).stop 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 shutdown_connections 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8c8c078bc0 0x7f8c8c07b0d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cac0cd1c0 0x7f8cac15a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cac0b4140 0x7f8cac15a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 --2- 172.21.15.129:0/1764052243 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cac0705a0 0x7f8cac1f5960 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 >> 172.21.15.129:0/1764052243 conn(0x7f8cac06d410 msgr2=0x7f8cac0b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 shutdown_connections 2024-09-17T03:58:42.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:42.686+0000 7f8cb1400640 1 -- 172.21.15.129:0/1764052243 wait complete. 2024-09-17T03:58:42.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:58:42.945 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"14","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-09-17T03:58:42.946 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "14" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-09-17T03:58:42.946 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-09-17T03:58:44.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 -- 172.21.15.129:0/2189604820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c099570 msgr2=0x7fb74c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 --2- 172.21.15.129:0/2189604820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c099570 0x7fb74c06eed0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fb74c0af270 tx=0x7fb74002a560 comp rx=0 tx=0).stop 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 -- 172.21.15.129:0/2189604820 shutdown_connections 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 --2- 172.21.15.129:0/2189604820 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c0cf7f0 0x7fb74c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 --2- 172.21.15.129:0/2189604820 >> v2:172.21.15.181:3300/0 conn(0x7fb74c06f410 0x7fb74c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 --2- 172.21.15.129:0/2189604820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c099570 0x7fb74c06eed0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 -- 172.21.15.129:0/2189604820 >> 172.21.15.129:0/2189604820 conn(0x7fb74c06d410 msgr2=0x7fb74c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 -- 172.21.15.129:0/2189604820 shutdown_connections 2024-09-17T03:58:45.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.175+0000 7fb754800640 1 -- 172.21.15.129:0/2189604820 wait complete. 2024-09-17T03:58:45.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.178+0000 7fb754800640 1 Processor -- start 2024-09-17T03:58:45.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.178+0000 7fb754800640 1 -- start start 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb74c06f410 0x7fb74c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c099570 0x7fb74c15ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c160990 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb74c160810 con 0x7fb74c06f410 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb754800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c160b10 con 0x7fb74c099570 2024-09-17T03:58:45.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.179+0000 7fb752800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43388/0 (socket says 172.21.15.129:43388) 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 -- 172.21.15.129:0/2066450087 learned_addr learned my addr 172.21.15.129:0/2066450087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb751400640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c099570 0x7fb74c15ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb751e00640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb74c06f410 0x7fb74c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 -- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c099570 msgr2=0x7fb74c15ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c099570 0x7fb74c15ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 -- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb74c06f410 msgr2=0x7fb74c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb74c06f410 0x7fb74c15e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 -- 172.21.15.129:0/2066450087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c0745f0 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb752800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fb74800e990 tx=0x7fb74800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb748004d20 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb74800caa0 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.180+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74c073850 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.181+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb748005020 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.181+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74c073e10 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.182+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb74c06a140 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.183+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb7480051c0 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.184+0000 7fb746000640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 0x7fb73007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.184+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fb74809b650 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.184+0000 7fb751e00640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 0x7fb73007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.185+0000 7fb751e00640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 0x7fb73007b0a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb740033000 tx=0x7fb740004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.188+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb748016030 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.413+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb74c0b1a50 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.414+0000 7fb746000640 1 -- 172.21.15.129:0/2066450087 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3605 (secure 0 0 0) 0x7fb7480639d0 con 0x7fb74c0cf7f0 2024-09-17T03:58:45.414 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:45.414 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":121,"btime":"2024-09-17T03:58:38:521272+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:38.130092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181,8178]},"id":10}]} 2024-09-17T03:58:45.414 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 121 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 msgr2=0x7fb73007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 0x7fb73007b0a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb740033000 tx=0x7fb740004730 comp rx=0 tx=0).stop 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 msgr2=0x7fb74c1f95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7fb74800e990 tx=0x7fb74800ee50 comp rx=0 tx=0).stop 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 shutdown_connections 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb730078b90 0x7fb73007b0a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb74c0cf7f0 0x7fb74c1f95f0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb74c099570 0x7fb74c15ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 --2- 172.21.15.129:0/2066450087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb74c06f410 0x7fb74c15e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:45.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 >> 172.21.15.129:0/2066450087 conn(0x7fb74c06d410 msgr2=0x7fb74c15c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:45.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 shutdown_connections 2024-09-17T03:58:45.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:45.416+0000 7fb754800640 1 -- 172.21.15.129:0/2066450087 wait complete. 2024-09-17T03:58:45.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:58:45.680 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}}] 2024-09-17T03:58:45.681 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-09-17T03:58:45.681 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/dir_2', 0)] 2024-09-17T03:58:47.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:58:47.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:58:47.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.909+0000 7efe9e600640 1 -- 172.21.15.129:0/970343857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe980705a0 msgr2=0x7efe980b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.909+0000 7efe9e600640 1 --2- 172.21.15.129:0/970343857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe980705a0 0x7efe980b4140 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7efe88005fd0 tx=0x7efe8802a8b0 comp rx=0 tx=0).stop 2024-09-17T03:58:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 -- 172.21.15.129:0/970343857 shutdown_connections 2024-09-17T03:58:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 --2- 172.21.15.129:0/970343857 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 0x7efe980ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:47.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 --2- 172.21.15.129:0/970343857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe980705a0 0x7efe980b4140 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:47.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 -- 172.21.15.129:0/970343857 >> 172.21.15.129:0/970343857 conn(0x7efe9806d410 msgr2=0x7efe980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:47.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 -- 172.21.15.129:0/970343857 shutdown_connections 2024-09-17T03:58:47.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.910+0000 7efe9e600640 1 -- 172.21.15.129:0/970343857 wait complete. 2024-09-17T03:58:47.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.912+0000 7efe9e600640 1 Processor -- start 2024-09-17T03:58:47.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.912+0000 7efe9e600640 1 -- start start 2024-09-17T03:58:47.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.913+0000 7efe9e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 0x7efe981f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:47.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.913+0000 7efe9e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 0x7efe980c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe9e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe9e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe98160200 con 0x7efe980705a0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe9e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efe981612e0 con 0x7efe9806f2e0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe9e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efe980cd1c0 con 0x7efe9806eed0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43390/0 (socket says 172.21.15.129:43390) 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 -- 172.21.15.129:0/512934736 learned_addr learned my addr 172.21.15.129:0/512934736 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe96a00640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 0x7efe980c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97400640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 0x7efe981f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 -- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 msgr2=0x7efe981f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 0x7efe981f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 -- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 msgr2=0x7efe980c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 0x7efe980c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe97e00640 1 -- 172.21.15.129:0/512934736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe981f7a20 con 0x7efe980705a0 2024-09-17T03:58:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.914+0000 7efe96a00640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 0x7efe980c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:58:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe97e00640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7efe9000c3c0 tx=0x7efe9000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90011790 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe90017070 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90011ca0 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe981f7740 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.915+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe981f3930 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.916+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe981f74b0 con 0x7efe980705a0 2024-09-17T03:58:47.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.917+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efe9001c460 con 0x7efe980705a0 2024-09-17T03:58:47.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.918+0000 7efe8f400640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:47.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.918+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7efe9009a970 con 0x7efe980705a0 2024-09-17T03:58:47.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.919+0000 7efe97400640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:47.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.920+0000 7efe97400640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efe88034000 tx=0x7efe88004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:47.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:47.923+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efe900626a0 con 0x7efe980705a0 2024-09-17T03:58:48.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.138+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe981f3470 con 0x7efe980705a0 2024-09-17T03:58:48.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.139+0000 7efe8f400640 1 -- 172.21.15.129:0/512934736 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+3605 (secure 0 0 0) 0x7efe981f3470 con 0x7efe980705a0 2024-09-17T03:58:48.139 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:48.139 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":121,"btime":"2024-09-17T03:58:38:521272+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:38.130092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181,8178]},"id":10}]} 2024-09-17T03:58:48.139 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 121 2024-09-17T03:58:48.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 msgr2=0x7efe7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7efe88034000 tx=0x7efe88004ab0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 msgr2=0x7efe981f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7efe9000c3c0 tx=0x7efe9000c880 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 shutdown_connections 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe980705a0 0x7efe981f6ac0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9806f2e0 0x7efe980c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 --2- 172.21.15.129:0/512934736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9806eed0 0x7efe981f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 >> 172.21.15.129:0/512934736 conn(0x7efe9806d410 msgr2=0x7efe981609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 shutdown_connections 2024-09-17T03:58:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.141+0000 7efe9e600640 1 -- 172.21.15.129:0/512934736 wait complete. 2024-09-17T03:58:48.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-09-17T03:58:48.404 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"16","version":"16","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/dir_1","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/dir_2","dirfrag":"0x10000000001","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-09-17T03:58:48.405 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "16", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "16" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/dir_1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x10000000001", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/dir_2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "3", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T03:58:48.405 INFO:tasks.cephfs.cephfs_test_case:[('/dir_2', 0)] =?= [('/dir_2', 0)] 2024-09-17T03:58:48.406 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:48.406 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:48.441 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:48.442 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:48.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:58:48.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 -- 172.21.15.129:0/1267270344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c003a50 msgr2=0x7f259c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 --2- 172.21.15.129:0/1267270344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c003a50 0x7f259c003ec0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009b50 tx=0x7f259c005830 comp rx=0 tx=0).stop 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 -- 172.21.15.129:0/1267270344 shutdown_connections 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 --2- 172.21.15.129:0/1267270344 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f259c003a50 0x7f259c003ec0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 -- 172.21.15.129:0/1267270344 >> 172.21.15.129:0/1267270344 conn(0x7f25b806d410 msgr2=0x7f25b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 -- 172.21.15.129:0/1267270344 shutdown_connections 2024-09-17T03:58:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.780+0000 7f25bd200640 1 -- 172.21.15.129:0/1267270344 wait complete. 2024-09-17T03:58:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.782+0000 7f25bd200640 1 Processor -- start 2024-09-17T03:58:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.783+0000 7f25bd200640 1 -- start start 2024-09-17T03:58:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f259c003a50 0x7f25b81f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 0x7f25b80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25b80b4140 con 0x7f25b80b12e0 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25b806f4f0 con 0x7f25b8159140 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25bd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f25b80b42c0 con 0x7f259c003a50 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25b6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43420/0 (socket says 172.21.15.129:43420) 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.784+0000 7f25b6800640 1 -- 172.21.15.129:0/161318874 learned_addr learned my addr 172.21.15.129:0/161318874 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b5e00640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f259c003a50 0x7f25b81f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25ad400640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 0x7f25b80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 -- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f259c003a50 msgr2=0x7f25b81f4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f259c003a50 0x7f25b81f4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 -- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 msgr2=0x7f25b80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 0x7f25b80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 -- 172.21.15.129:0/161318874 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b815bff0 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25ad400640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 0x7f25b80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25b6800640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f25b80c72f0 tx=0x7f25b000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b000cc20 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25b0017070 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b00116c0 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b80b16f0 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.785+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b80c85c0 con 0x7f25b80b12e0 2024-09-17T03:58:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.787+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b815f250 con 0x7f25b80b12e0 2024-09-17T03:58:48.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.788+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f25b0011860 con 0x7f25b80b12e0 2024-09-17T03:58:48.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.789+0000 7f25aea00640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 0x7f259407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:48.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.789+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f25b009b310 con 0x7f25b80b12e0 2024-09-17T03:58:48.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.789+0000 7f25b5e00640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 0x7f259407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:48.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.790+0000 7f25b5e00640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 0x7f259407b0a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f25a8035000 tx=0x7f25a80046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:48.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.794+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f25b80b16f0 con 0x7f25b80b12e0 2024-09-17T03:58:48.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.988+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f25b80ccc80 con 0x7f25b80b12e0 2024-09-17T03:58:48.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.989+0000 7f25aea00640 1 -- 172.21.15.129:0/161318874 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) ==== 80+0+0 (secure 0 0 0) 0x7f25b0063690 con 0x7f25b80b12e0 2024-09-17T03:58:48.989 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 msgr2=0x7f259407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 0x7f259407b0a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f25a8035000 tx=0x7f25a80046c0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 msgr2=0x7f25b80c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f25b80c72f0 tx=0x7f25b000ed10 comp rx=0 tx=0).stop 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 shutdown_connections 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2594078b90 0x7f259407b0a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25b80b12e0 0x7f25b80c7320 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25b8159140 0x7f25b80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 --2- 172.21.15.129:0/161318874 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f259c003a50 0x7f25b81f4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 >> 172.21.15.129:0/161318874 conn(0x7f25b806d410 msgr2=0x7f25b80ae3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 shutdown_connections 2024-09-17T03:58:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:48.991+0000 7f25bd200640 1 -- 172.21.15.129:0/161318874 wait complete. 2024-09-17T03:58:49.014 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:58:49.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:49.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:49.118 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:58:49.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:49.118 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:58:49.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T03:58:49.450 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T03:58:49.450 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:58:49.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json cache drop 2024-09-17T03:58:49.693 INFO:teuthology.orchestra.run.smithi129.stdout:{"client_recall":{"return_code":0,"message":"(0) Success","recalled":0},"flush_journal":{"return_code":0,"message":""},"trim_cache":{"trimmed":0},"cache":{"pool":{"items":293,"bytes":60314}},"duration":0.0069483920000000003} 2024-09-17T03:58:49.693 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cache": { "pool": { "bytes": 60314, "items": 293 } }, "client_recall": { "message": "(0) Success", "recalled": 0, "return_code": 0 }, "duration": 0.006948392, "flush_journal": { "message": "", "return_code": 0 }, "trim_cache": { "trimmed": 0 } } 2024-09-17T03:58:49.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 -- 172.21.15.129:0/803523136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c0004d40 msgr2=0x7f97c0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 --2- 172.21.15.129:0/803523136 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97c0004d40 0x7f97c0005150 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f97d4002230 tx=0x7f97c0005830 comp rx=0 tx=0).stop 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97db400640 1 -- 172.21.15.129:0/803523136 reap_dead start 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 -- 172.21.15.129:0/803523136 shutdown_connections 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 -- 172.21.15.129:0/803523136 >> 172.21.15.129:0/803523136 conn(0x7f97dc06d410 msgr2=0x7f97dc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 -- 172.21.15.129:0/803523136 shutdown_connections 2024-09-17T03:58:49.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.915+0000 7f97e1a00640 1 -- 172.21.15.129:0/803523136 wait complete. 2024-09-17T03:58:49.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.917+0000 7f97e1a00640 1 Processor -- start 2024-09-17T03:58:49.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.918+0000 7f97e1a00640 1 -- start start 2024-09-17T03:58:49.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c0004d40 0x7f97dc0c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:49.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97dc0c3280 0x7f97dc1f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc1612e0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97dc0cd1c0 con 0x7f97c0004d40 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97e1a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f97dc0c4e90 con 0x7f97dc0c3280 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97d2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97d2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43440/0 (socket says 172.21.15.129:43440) 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97d2000640 1 -- 172.21.15.129:0/2577329837 learned_addr learned my addr 172.21.15.129:0/2577329837 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.919+0000 7f97daa00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c0004d40 0x7f97dc0c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97db400640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97dc0c3280 0x7f97dc1f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 -- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97dc0c3280 msgr2=0x7f97dc1f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97dc0c3280 0x7f97dc1f8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:49.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 -- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c0004d40 msgr2=0x7f97dc0c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c0004d40 0x7f97dc0c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 -- 172.21.15.129:0/2577329837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97dc0c7700 con 0x7f97dc0c2e70 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d2000640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f97c4011ae0 tx=0x7f97c4011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4004460 con 0x7f97dc0c2e70 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97c400f9a0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97dc0c79e0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.920+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c400fdf0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.921+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97dc0c7fa0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.922+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97dc0ccc80 con 0x7f97dc0c2e70 2024-09-17T03:58:49.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.923+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f97c4020020 con 0x7f97dc0c2e70 2024-09-17T03:58:49.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.924+0000 7f97d3400640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 0x7f97b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:49.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.924+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f97c409b6c0 con 0x7f97dc0c2e70 2024-09-17T03:58:49.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.924+0000 7f97daa00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 0x7f97b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:49.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.925+0000 7f97daa00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 0x7f97b807b0a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009c50 tx=0x7f97cc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:49.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:49.928+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f97c4063ae0 con 0x7f97dc0c2e70 2024-09-17T03:58:50.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.150+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f97dc06a140 con 0x7f97dc0c2e70 2024-09-17T03:58:50.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.150+0000 7f97d3400640 1 -- 172.21.15.129:0/2577329837 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v121) ==== 53+0+73 (secure 0 0 0) 0x7f97c40633e0 con 0x7f97dc0c2e70 2024-09-17T03:58:50.150 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 msgr2=0x7f97b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 0x7f97b807b0a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009c50 tx=0x7f97cc005f90 comp rx=0 tx=0).stop 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 msgr2=0x7f97dc1f89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f97c4011ae0 tx=0x7f97c4011ee0 comp rx=0 tx=0).stop 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 shutdown_connections 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97b8078b90 0x7f97b807b0a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.152+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97dc0c3280 0x7f97dc1f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:50.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.153+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97dc0c2e70 0x7f97dc1f89d0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:50.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.153+0000 7f97e1a00640 1 --2- 172.21.15.129:0/2577329837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97c0004d40 0x7f97dc0c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:50.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.153+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 >> 172.21.15.129:0/2577329837 conn(0x7f97dc06d410 msgr2=0x7f97dc0b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:50.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.153+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 shutdown_connections 2024-09-17T03:58:50.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50.153+0000 7f97e1a00640 1 -- 172.21.15.129:0/2577329837 wait complete. 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:58:50.178 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:58:50.179 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:50.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.179 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:58:50.216 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.298 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42162sec preferred_lft 42162sec 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:58:50.299 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:58:50.300 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:58:50.301 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:58:50.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.301 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:58:50.354 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:58:50.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.357 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:58:50.393 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:58:50.393 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:58:50.393 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:58:50.394 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:58:50.394 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:58:50.394 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:58:50.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:58:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:58:50.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.592 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:58:50.627 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:58:50.627 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:58:50.628 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:58:50.724 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:58:50.724 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:50.759 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:58:50.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:58:50.760 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:50.835 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:58:50.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:50.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:51.032 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:51.032 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:51.032 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:58:51.052 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:58:51.084 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:58:51.085 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:58:51.085 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:58:51.086 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:58:51.086 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:58:51.086 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:58:51.086 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:58:51.086 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:58:51.089 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:58:51.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:51.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:51.146 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:58:51.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:58:51.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:58:51.237 INFO:teuthology.orchestra.run.smithi129.stdout:client8445 2024-09-17T03:58:51.237 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:51.237 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:51.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:51.273 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f dir_1/original) 2024-09-17T03:58:51.352 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:51.352 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:51.427 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:58:51.427 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:58:51.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:58:51.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 -- 172.21.15.129:0/956994837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540cd1c0 msgr2=0x7f4c541609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 --2- 172.21.15.129:0/956994837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540cd1c0 0x7f4c541609c0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f4c54072080 tx=0x7f4c4c02a5f0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 -- 172.21.15.129:0/956994837 shutdown_connections 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 --2- 172.21.15.129:0/956994837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540cd1c0 0x7f4c541609c0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 --2- 172.21.15.129:0/956994837 >> v2:172.21.15.181:3300/0 conn(0x7f4c540b4140 0x7f4c540ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 -- 172.21.15.129:0/956994837 >> 172.21.15.129:0/956994837 conn(0x7f4c5406d410 msgr2=0x7f4c540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 -- 172.21.15.129:0/956994837 shutdown_connections 2024-09-17T03:58:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.764+0000 7f4c5c600640 1 -- 172.21.15.129:0/956994837 wait complete. 2024-09-17T03:58:51.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.766+0000 7f4c5c600640 1 Processor -- start 2024-09-17T03:58:51.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.767+0000 7f4c5c600640 1 -- start start 2024-09-17T03:58:51.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c540b4140 0x7f4c541f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c540cd1c0 0x7f4c540cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c54160260 con 0x7f4c540705a0 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4c541600e0 con 0x7f4c540cd1c0 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c5c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4c54160f00 con 0x7f4c540b4140 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c59c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c59c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43446/0 (socket says 172.21.15.129:43446) 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.768+0000 7f4c59c00640 1 -- 172.21.15.129:0/1228132586 learned_addr learned my addr 172.21.15.129:0/1228132586 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59200640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c540b4140 0x7f4c541f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c5a600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c540cd1c0 0x7f4c540cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:51.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 -- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c540b4140 msgr2=0x7f4c541f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:51.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c540b4140 0x7f4c541f4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 -- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c540cd1c0 msgr2=0x7f4c540cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:51.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c540cd1c0 0x7f4c540cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 -- 172.21.15.129:0/1228132586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c540c4210 con 0x7f4c540705a0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c59c00640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c730 tx=0x7f4c4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48003b70 con 0x7f4c540705a0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c48003d30 con 0x7f4c540705a0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c4800aab0 con 0x7f4c540705a0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c5415bd50 con 0x7f4c540705a0 2024-09-17T03:58:51.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.769+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c5415c310 con 0x7f4c540705a0 2024-09-17T03:58:51.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.770+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c540ccc80 con 0x7f4c540705a0 2024-09-17T03:58:51.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.772+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4c4800ac50 con 0x7f4c540705a0 2024-09-17T03:58:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.773+0000 7f4c51600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 0x7f4c3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.773+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f4c4809ac80 con 0x7f4c540705a0 2024-09-17T03:58:51.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.773+0000 7f4c59200640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 0x7f4c3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:51.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.774+0000 7f4c59200640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 0x7f4c3407b0a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4c40005fd0 tx=0x7f4c40005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:51.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.777+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4c540cd5d0 con 0x7f4c540705a0 2024-09-17T03:58:51.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.973+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c5406a140 con 0x7f4c540705a0 2024-09-17T03:58:51.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.973+0000 7f4c51600640 1 -- 172.21.15.129:0/1228132586 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v143) ==== 80+0+0 (secure 0 0 0) 0x7f4c48063000 con 0x7f4c540705a0 2024-09-17T03:58:51.973 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:58:51.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 msgr2=0x7f4c3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:51.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 0x7f4c3407b0a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4c40005fd0 tx=0x7f4c40005850 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 msgr2=0x7f4c540c3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f4c4800c730 tx=0x7f4c4800cbf0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 shutdown_connections 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c34078b90 0x7f4c3407b0a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c540cd1c0 0x7f4c540cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c540b4140 0x7f4c541f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 --2- 172.21.15.129:0/1228132586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c540705a0 0x7f4c540c3cd0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 >> 172.21.15.129:0/1228132586 conn(0x7f4c5406d410 msgr2=0x7f4c540ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 shutdown_connections 2024-09-17T03:58:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:51.976+0000 7f4c5c600640 1 -- 172.21.15.129:0/1228132586 wait complete. 2024-09-17T03:58:51.998 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:58:51.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:51.999 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:58:52.064 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:58:52.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:58:52.065 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:58:52.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:58:52.380 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T03:58:52.380 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T03:58:52.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-09-17T03:58:52.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 -- 172.21.15.129:0/74125997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb8074800 msgr2=0x7f4bb80af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 --2- 172.21.15.129:0/74125997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb8074800 0x7f4bb80af480 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009830 tx=0x7f4ba802a520 comp rx=0 tx=0).stop 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 -- 172.21.15.129:0/74125997 shutdown_connections 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 --2- 172.21.15.129:0/74125997 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb80b3010 0x7f4bb80b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 --2- 172.21.15.129:0/74125997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb8074800 0x7f4bb80af480 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 -- 172.21.15.129:0/74125997 >> 172.21.15.129:0/74125997 conn(0x7f4bb80ccc40 msgr2=0x7f4bb80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 -- 172.21.15.129:0/74125997 shutdown_connections 2024-09-17T03:58:52.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.607+0000 7f4bbe600640 1 -- 172.21.15.129:0/74125997 wait complete. 2024-09-17T03:58:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.609+0000 7f4bbe600640 1 Processor -- start 2024-09-17T03:58:52.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.610+0000 7f4bbe600640 1 -- start start 2024-09-17T03:58:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb8074800 0x7f4bb8155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb80b3010 0x7f4bb80ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:52.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb80b3ae0 con 0x7f4bb80b1c40 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb80b3960 con 0x7f4bb80b3010 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bbe600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb80b3c60 con 0x7f4bb8074800 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4baea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4baea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43450/0 (socket says 172.21.15.129:43450) 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4baea00640 1 -- 172.21.15.129:0/194578662 learned_addr learned my addr 172.21.15.129:0/194578662 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bb7400640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb8074800 0x7f4bb8155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:52.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.611+0000 7f4bb7e00640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb80b3010 0x7f4bb80ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 -- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb8074800 msgr2=0x7f4bb8155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb8074800 0x7f4bb8155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 -- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb80b3010 msgr2=0x7f4bb80ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb80b3010 0x7f4bb80ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:52.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 -- 172.21.15.129:0/194578662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb815ba40 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4baea00640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c730 tx=0x7f4ba000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000ce60 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ba0006890 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0006d20 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.612+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb809d770 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.613+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb80c90f0 con 0x7f4bb80b1c40 2024-09-17T03:58:52.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.614+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb809d4e0 con 0x7f4bb80b1c40 2024-09-17T03:58:52.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.615+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4ba0007030 con 0x7f4bb80b1c40 2024-09-17T03:58:52.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.616+0000 7f4bafe00640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 0x7f4b9407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:52.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.617+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f4ba009a5c0 con 0x7f4bb80b1c40 2024-09-17T03:58:52.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.617+0000 7f4bb7400640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 0x7f4b9407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:52.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.618+0000 7f4bb7400640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 0x7f4b9407b0d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8034000 tx=0x7f4ba8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:52.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.621+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4bb80cbaf0 con 0x7f4bb80b1c40 2024-09-17T03:58:52.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:52.840+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f4bb80b4180 con 0x7f4bb80b1c40 2024-09-17T03:58:53.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.584+0000 7f4bafe00640 1 -- 172.21.15.129:0/194578662 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v122) ==== 105+0+0 (secure 0 0 0) 0x7f4ba0062a60 con 0x7f4bb80b1c40 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.587+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 msgr2=0x7f4b9407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.587+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 0x7f4b9407b0d0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8034000 tx=0x7f4ba8004730 comp rx=0 tx=0).stop 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.587+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 msgr2=0x7f4bb80cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.587+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000c730 tx=0x7f4ba000cb30 comp rx=0 tx=0).stop 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 shutdown_connections 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b94078bc0 0x7f4b9407b0d0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4bb80b3010 0x7f4bb80ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4bb80b1c40 0x7f4bb80cde70 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 --2- 172.21.15.129:0/194578662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4bb8074800 0x7f4bb8155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 >> 172.21.15.129:0/194578662 conn(0x7f4bb80ccc40 msgr2=0x7f4bb80ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:53.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 shutdown_connections 2024-09-17T03:58:53.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.588+0000 7f4bbe600640 1 -- 172.21.15.129:0/194578662 wait complete. 2024-09-17T03:58:53.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:53.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 -- 172.21.15.129:0/2845809420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f930c004800 msgr2=0x7f930c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:53.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 --2- 172.21.15.129:0/2845809420 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f930c004800 0x7f930c004c10 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f9310009b20 tx=0x7f930c005830 comp rx=0 tx=0).stop 2024-09-17T03:58:53.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932bc00640 1 -- 172.21.15.129:0/2845809420 reap_dead start 2024-09-17T03:58:53.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 -- 172.21.15.129:0/2845809420 shutdown_connections 2024-09-17T03:58:53.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 -- 172.21.15.129:0/2845809420 >> 172.21.15.129:0/2845809420 conn(0x7f932409ae80 msgr2=0x7f9324074580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:53.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 -- 172.21.15.129:0/2845809420 shutdown_connections 2024-09-17T03:58:53.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.837+0000 7f932a600640 1 -- 172.21.15.129:0/2845809420 wait complete. 2024-09-17T03:58:53.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.839+0000 7f932a600640 1 Processor -- start 2024-09-17T03:58:53.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.839+0000 7f932a600640 1 -- start start 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c004800 0x7f93240cafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f932415d190 0x7f932415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93240b4080 con 0x7f9324153150 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f93240b3f00 con 0x7f930c004800 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f93240b4200 con 0x7f932415d190 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53336/0 (socket says 172.21.15.129:53336) 2024-09-17T03:58:53.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 -- 172.21.15.129:0/796030974 learned_addr learned my addr 172.21.15.129:0/796030974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9329200640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c004800 0x7f93240cafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f932bc00640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f932415d190 0x7f932415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 -- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f932415d190 msgr2=0x7f932415d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f932415d190 0x7f932415d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 -- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c004800 msgr2=0x7f93240cafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c004800 0x7f93240cafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.841+0000 7f9323e00640 1 -- 172.21.15.129:0/796030974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93240c55d0 con 0x7f9324153150 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f9323e00640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9310009c40 tx=0x7f9310033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9310043070 con 0x7f9324153150 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9310004120 con 0x7f9324153150 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93240c58b0 con 0x7f9324153150 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9310038040 con 0x7f9324153150 2024-09-17T03:58:53.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.842+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93240ca7b0 con 0x7f9324153150 2024-09-17T03:58:53.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.843+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93240d0960 con 0x7f9324153150 2024-09-17T03:58:53.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.844+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9310046070 con 0x7f9324153150 2024-09-17T03:58:53.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.845+0000 7f9320c00640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 0x7f930407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:53.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.845+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f93100bb370 con 0x7f9324153150 2024-09-17T03:58:53.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.846+0000 7f9329200640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 0x7f930407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:53.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.847+0000 7f9329200640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 0x7f930407b0a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f93180017e0 tx=0x7f9318001110 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:53.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:53.850+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f93100830a0 con 0x7f9324153150 2024-09-17T03:58:54.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.067+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f932406a140 con 0x7f9324153150 2024-09-17T03:58:54.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.067+0000 7f9320c00640 1 -- 172.21.15.129:0/796030974 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f9310087ad0 con 0x7f9324153150 2024-09-17T03:58:54.067 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:54.067 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:54.068 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.069+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 msgr2=0x7f930407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.069+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 0x7f930407b0a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f93180017e0 tx=0x7f9318001110 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.069+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 msgr2=0x7f932415f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.069+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f9310009c40 tx=0x7f9310033670 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.069+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 shutdown_connections 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9304078b90 0x7f930407b0a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f932415d190 0x7f932415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9324153150 0x7f932415f740 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 --2- 172.21.15.129:0/796030974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f930c004800 0x7f93240cafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 >> 172.21.15.129:0/796030974 conn(0x7f932409ae80 msgr2=0x7f93240b3370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:54.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 shutdown_connections 2024-09-17T03:58:54.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:54.070+0000 7f932a600640 1 -- 172.21.15.129:0/796030974 wait complete. 2024-09-17T03:58:54.293 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:58:54.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:58:55.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 -- 172.21.15.129:0/4286729606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 msgr2=0x7f08f80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 --2- 172.21.15.129:0/4286729606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f80ccc80 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f08f8072e00 tx=0x7f08e402a580 comp rx=0 tx=0).stop 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 -- 172.21.15.129:0/4286729606 shutdown_connections 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 --2- 172.21.15.129:0/4286729606 >> v2:172.21.15.181:3300/0 conn(0x7f08f80cd1c0 0x7f08f81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 --2- 172.21.15.129:0/4286729606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f80ccc80 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.524+0000 7f08fd400640 1 -- 172.21.15.129:0/4286729606 >> 172.21.15.129:0/4286729606 conn(0x7f08f806d410 msgr2=0x7f08f80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:55.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.525+0000 7f08fd400640 1 -- 172.21.15.129:0/4286729606 shutdown_connections 2024-09-17T03:58:55.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.525+0000 7f08fd400640 1 -- 172.21.15.129:0/4286729606 wait complete. 2024-09-17T03:58:55.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.527+0000 7f08fd400640 1 Processor -- start 2024-09-17T03:58:55.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.527+0000 7f08fd400640 1 -- start start 2024-09-17T03:58:55.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f80705a0 0x7f08f80c6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 0x7f08f80b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08f8160260 con 0x7f08f80b4140 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81600e0 con 0x7f08f80705a0 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08fd400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f08f8160f00 con 0x7f08f80cd1c0 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08f5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.528+0000 7f08f5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53354/0 (socket says 172.21.15.129:53354) 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 -- 172.21.15.129:0/705168725 learned_addr learned my addr 172.21.15.129:0/705168725 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f6800640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f80705a0 0x7f08f80c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f7200640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 0x7f08f80b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 -- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 msgr2=0x7f08f80b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 0x7f08f80b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 -- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f80705a0 msgr2=0x7f08f80c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f80705a0 0x7f08f80c6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 -- 172.21.15.129:0/705168725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f80c5370 con 0x7f08f80b4140 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f7200640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 0x7f08f80b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08f5e00640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f08e4000c00 tx=0x7f08e40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.529+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e403e030 con 0x7f08f80b4140 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.530+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08e40047a0 con 0x7f08f80b4140 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.530+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f80c5650 con 0x7f08f80b4140 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.530+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4035040 con 0x7f08f80b4140 2024-09-17T03:58:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.530+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f80c5c10 con 0x7f08f80b4140 2024-09-17T03:58:55.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.531+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f80ccc80 con 0x7f08f80b4140 2024-09-17T03:58:55.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.532+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f08e4041030 con 0x7f08f80b4140 2024-09-17T03:58:55.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.533+0000 7f08eea00640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 0x7f08d807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:55.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.533+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f08e40bb650 con 0x7f08f80b4140 2024-09-17T03:58:55.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.533+0000 7f08f6800640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 0x7f08d807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.534+0000 7f08f6800640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 0x7f08d807b0a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f08e8006fd0 tx=0x7f08e8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:55.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.537+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f08f80cd5d0 con 0x7f08f80b4140 2024-09-17T03:58:55.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.755+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08f806a140 con 0x7f08f80b4140 2024-09-17T03:58:55.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.755+0000 7f08eea00640 1 -- 172.21.15.129:0/705168725 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f08e4083a60 con 0x7f08f80b4140 2024-09-17T03:58:55.756 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:55.756 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:55.756 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:58:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 msgr2=0x7f08d807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 0x7f08d807b0a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f08e8006fd0 tx=0x7f08e8008040 comp rx=0 tx=0).stop 2024-09-17T03:58:55.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 msgr2=0x7f08f815aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f08e4000c00 tx=0x7f08e40039e0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 shutdown_connections 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08d8078b90 0x7f08d807b0a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08f80cd1c0 0x7f08f80b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08f80b4140 0x7f08f815aa20 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 --2- 172.21.15.129:0/705168725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08f80705a0 0x7f08f80c6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 >> 172.21.15.129:0/705168725 conn(0x7f08f806d410 msgr2=0x7f08f80ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 shutdown_connections 2024-09-17T03:58:55.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:55.758+0000 7f08fd400640 1 -- 172.21.15.129:0/705168725 wait complete. 2024-09-17T03:58:55.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:58:55.986 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:58:56.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 -- 172.21.15.129:0/3602858643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800cfb60 msgr2=0x7fa780160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 --2- 172.21.15.129:0/3602858643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800cfb60 0x7fa780160cf0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fa770009830 tx=0x7fa77002a480 comp rx=0 tx=0).stop 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 -- 172.21.15.129:0/3602858643 shutdown_connections 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 --2- 172.21.15.129:0/3602858643 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800b46e0 0x7fa7800b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 --2- 172.21.15.129:0/3602858643 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800cfb60 0x7fa780160cf0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 -- 172.21.15.129:0/3602858643 >> 172.21.15.129:0/3602858643 conn(0x7fa78006cfc0 msgr2=0x7fa78006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:57.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 -- 172.21.15.129:0/3602858643 shutdown_connections 2024-09-17T03:58:57.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.211+0000 7fa785c00640 1 -- 172.21.15.129:0/3602858643 wait complete. 2024-09-17T03:58:57.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.213+0000 7fa785c00640 1 Processor -- start 2024-09-17T03:58:57.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.213+0000 7fa785c00640 1 -- start start 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.214+0000 7fa785c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7800b46e0 0x7fa780159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800cfb60 0x7fa780144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa78015d270 con 0x7fa7800d02c0 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa78015d0f0 con 0x7fa7800b46e0 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa78015d3f0 con 0x7fa7800cfb60 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53372/0 (socket says 172.21.15.129:53372) 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 -- 172.21.15.129:0/3125698207 learned_addr learned my addr 172.21.15.129:0/3125698207 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:57.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa777400640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800cfb60 0x7fa780144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa77fe00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7800b46e0 0x7fa780159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 -- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800cfb60 msgr2=0x7fa780144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800cfb60 0x7fa780144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 -- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7800b46e0 msgr2=0x7fa780159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7800b46e0 0x7fa780159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.215+0000 7fa785200640 1 -- 172.21.15.129:0/3125698207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78015a6e0 con 0x7fa7800d02c0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa785200640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c760 tx=0x7fa77800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa778003da0 con 0x7fa7800d02c0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa77800ce10 con 0x7fa7800d02c0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78015a9c0 con 0x7fa7800d02c0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7780052d0 con 0x7fa7800d02c0 2024-09-17T03:58:57.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.216+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78015af80 con 0x7fa7800d02c0 2024-09-17T03:58:57.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.217+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78015d8a0 con 0x7fa7800d02c0 2024-09-17T03:58:57.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.218+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa778005470 con 0x7fa7800d02c0 2024-09-17T03:58:57.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.219+0000 7fa77cc00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 0x7fa75c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:57.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.220+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fa77809aa80 con 0x7fa7800d02c0 2024-09-17T03:58:57.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.220+0000 7fa77fe00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 0x7fa75c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:57.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.221+0000 7fa77fe00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 0x7fa75c07b0a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa770034000 tx=0x7fa770004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:57.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.224+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa77809ce20 con 0x7fa7800d02c0 2024-09-17T03:58:57.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.441+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa78015a470 con 0x7fa7800d02c0 2024-09-17T03:58:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.442+0000 7fa77cc00640 1 -- 172.21.15.129:0/3125698207 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fa7780627b0 con 0x7fa7800d02c0 2024-09-17T03:58:57.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:57.441 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:57.442 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:58:57.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 msgr2=0x7fa75c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 0x7fa75c07b0a0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa770034000 tx=0x7fa770004730 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 msgr2=0x7fa78015b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fa77800c760 tx=0x7fa77800cc20 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 shutdown_connections 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa75c078b90 0x7fa75c07b0a0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7800d02c0 0x7fa78015b780 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7800cfb60 0x7fa780144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 --2- 172.21.15.129:0/3125698207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7800b46e0 0x7fa780159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.444+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 >> 172.21.15.129:0/3125698207 conn(0x7fa78006cfc0 msgr2=0x7fa78006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.445+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 shutdown_connections 2024-09-17T03:58:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:57.445+0000 7fa785c00640 1 -- 172.21.15.129:0/3125698207 wait complete. 2024-09-17T03:58:57.669 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:58:57.669 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:58:58.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:58:58.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 -- 172.21.15.129:0/867537883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 msgr2=0x7fc80c005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 --2- 172.21.15.129:0/867537883 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc80c005290 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc828071eb0 tx=0x7fc80c005830 comp rx=0 tx=0).stop 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827400640 1 -- 172.21.15.129:0/867537883 reap_dead start 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 -- 172.21.15.129:0/867537883 shutdown_connections 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 -- 172.21.15.129:0/867537883 >> 172.21.15.129:0/867537883 conn(0x7fc828069330 msgr2=0x7fc828071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 -- 172.21.15.129:0/867537883 shutdown_connections 2024-09-17T03:58:58.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.903+0000 7fc827e00640 1 -- 172.21.15.129:0/867537883 wait complete. 2024-09-17T03:58:58.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.906+0000 7fc827e00640 1 Processor -- start 2024-09-17T03:58:58.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.906+0000 7fc827e00640 1 -- start start 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8280cbb90 0x7fc82815b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 0x7fc8280c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc82815dd00 con 0x7fc80c004e00 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc82815db80 con 0x7fc8280c9430 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc827e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc8280d0e10 con 0x7fc8280cbb90 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc826a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc826a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53400/0 (socket says 172.21.15.129:53400) 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.907+0000 7fc826a00640 1 -- 172.21.15.129:0/4184671063 learned_addr learned my addr 172.21.15.129:0/4184671063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:58:58.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc81e000640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8280cbb90 0x7fc82815b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc827400640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 0x7fc8280c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 -- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8280cbb90 msgr2=0x7fc82815b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8280cbb90 0x7fc82815b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 -- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 msgr2=0x7fc8280c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 0x7fc8280c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 -- 172.21.15.129:0/4184671063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8281f5600 con 0x7fc80c004e00 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc827400640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 0x7fc8280c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc826a00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc81800d8a0 tx=0x7fc81800dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8180062e0 con 0x7fc80c004e00 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc818015020 con 0x7fc80c004e00 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8280cc330 con 0x7fc80c004e00 2024-09-17T03:58:58.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.908+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81800ba30 con 0x7fc80c004e00 2024-09-17T03:58:58.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.909+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8281f58c0 con 0x7fc80c004e00 2024-09-17T03:58:58.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.910+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8280d0700 con 0x7fc80c004e00 2024-09-17T03:58:58.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.911+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc81800bbd0 con 0x7fc80c004e00 2024-09-17T03:58:58.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.912+0000 7fc81f400640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 0x7fc80407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:58:58.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.912+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fc81809b4e0 con 0x7fc80c004e00 2024-09-17T03:58:58.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.912+0000 7fc81e000640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 0x7fc80407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:58:58.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.913+0000 7fc81e000640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 0x7fc80407b0a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc8280d1e20 tx=0x7fc810006d20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:58:58.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:58.916+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc81801f020 con 0x7fc80c004e00 2024-09-17T03:58:59.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.137+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc82806a130 con 0x7fc80c004e00 2024-09-17T03:58:59.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.138+0000 7fc81f400640 1 -- 172.21.15.129:0/4184671063 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fc818063860 con 0x7fc80c004e00 2024-09-17T03:58:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:58:59.137 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:58:59.137 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:58:59.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 msgr2=0x7fc80407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:59.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 0x7fc80407b0a0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc8280d1e20 tx=0x7fc810006d20 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 msgr2=0x7fc8281f60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc81800d8a0 tx=0x7fc81800dd60 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 shutdown_connections 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc804078b90 0x7fc80407b0a0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8280c9430 0x7fc8280c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8280cbb90 0x7fc82815b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 --2- 172.21.15.129:0/4184671063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc80c004e00 0x7fc8281f60c0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 >> 172.21.15.129:0/4184671063 conn(0x7fc828069330 msgr2=0x7fc82806f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 shutdown_connections 2024-09-17T03:58:59.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:58:59.140+0000 7fc827e00640 1 -- 172.21.15.129:0/4184671063 wait complete. 2024-09-17T03:58:59.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:58:59.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:00.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:00.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 -- 172.21.15.129:0/3573543186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0734004800 msgr2=0x7f0734004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 --2- 172.21.15.129:0/3573543186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0734004800 0x7f0734004c10 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f07380016f0 tx=0x7f0734005830 comp rx=0 tx=0).stop 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f074e800640 1 -- 172.21.15.129:0/3573543186 reap_dead start 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 -- 172.21.15.129:0/3573543186 shutdown_connections 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 -- 172.21.15.129:0/3573543186 >> 172.21.15.129:0/3573543186 conn(0x7f075006d410 msgr2=0x7f07500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 -- 172.21.15.129:0/3573543186 shutdown_connections 2024-09-17T03:59:00.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.595+0000 7f0755000640 1 -- 172.21.15.129:0/3573543186 wait complete. 2024-09-17T03:59:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.597+0000 7f0755000640 1 Processor -- start 2024-09-17T03:59:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.598+0000 7f0755000640 1 -- start start 2024-09-17T03:59:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734004800 0x7f07500c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:00.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07500c8010 0x7f07500d0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f07500cfaa0 con 0x7f07501542e0 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f075015ee20 con 0x7f0734004800 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0755000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f07500cfc20 con 0x7f07500c8010 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f074e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f074e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53410/0 (socket says 172.21.15.129:53410) 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f074e800640 1 -- 172.21.15.129:0/2461356214 learned_addr learned my addr 172.21.15.129:0/2461356214 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.599+0000 7f0745400640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07500c8010 0x7f07500d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074de00640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734004800 0x7f07500c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 -- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07500c8010 msgr2=0x7f07500d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07500c8010 0x7f07500d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 -- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734004800 msgr2=0x7f07500c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734004800 0x7f07500c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 -- 172.21.15.129:0/2461356214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07501f99c0 con 0x7f07501542e0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f074e800640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0748005fd0 tx=0x7f0748011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748004b20 con 0x7f07501542e0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0748004cc0 con 0x7f07501542e0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748005280 con 0x7f07501542e0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.600+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07500c7530 con 0x7f07501542e0 2024-09-17T03:59:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.601+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07501546f0 con 0x7f07501542e0 2024-09-17T03:59:00.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.602+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07500c7320 con 0x7f07501542e0 2024-09-17T03:59:00.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.603+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0748005420 con 0x7f07501542e0 2024-09-17T03:59:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.604+0000 7f0746a00640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 0x7f072c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.604+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f074809be60 con 0x7f07501542e0 2024-09-17T03:59:00.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.604+0000 7f074de00640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 0x7f072c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:00.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.605+0000 7f074de00640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 0x7f072c07b0d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0740006c10 tx=0x7f07400067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:00.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.608+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0748020060 con 0x7f07501542e0 2024-09-17T03:59:00.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.826+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f075006a140 con 0x7f07501542e0 2024-09-17T03:59:00.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.827+0000 7f0746a00640 1 -- 172.21.15.129:0/2461356214 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f0748063b90 con 0x7f07501542e0 2024-09-17T03:59:00.827 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:00.827 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:00.827 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 msgr2=0x7f072c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 0x7f072c07b0d0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0740006c10 tx=0x7f07400067a0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 msgr2=0x7f07501f8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f0748005fd0 tx=0x7f0748011ee0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 shutdown_connections 2024-09-17T03:59:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f072c078bc0 0x7f072c07b0d0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f07501542e0 0x7f07501f8c40 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f07500c8010 0x7f07500d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 --2- 172.21.15.129:0/2461356214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0734004800 0x7f07500c69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 >> 172.21.15.129:0/2461356214 conn(0x7f075006d410 msgr2=0x7f07500d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 shutdown_connections 2024-09-17T03:59:00.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:00.829+0000 7f0755000640 1 -- 172.21.15.129:0/2461356214 wait complete. 2024-09-17T03:59:01.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:01.054 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:02.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:02.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 -- 172.21.15.129:0/3617012536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80b4140 msgr2=0x7fddc80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:02.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 --2- 172.21.15.129:0/3617012536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80b4140 0x7fddc80ccc80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fddb4005fd0 tx=0x7fddb402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 -- 172.21.15.129:0/3617012536 shutdown_connections 2024-09-17T03:59:02.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 --2- 172.21.15.129:0/3617012536 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 0x7fddc81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 --2- 172.21.15.129:0/3617012536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80b4140 0x7fddc80ccc80 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 -- 172.21.15.129:0/3617012536 >> 172.21.15.129:0/3617012536 conn(0x7fddc806d410 msgr2=0x7fddc80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:02.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 -- 172.21.15.129:0/3617012536 shutdown_connections 2024-09-17T03:59:02.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.281+0000 7fddcf200640 1 -- 172.21.15.129:0/3617012536 wait complete. 2024-09-17T03:59:02.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.283+0000 7fddcf200640 1 Processor -- start 2024-09-17T03:59:02.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.283+0000 7fddcf200640 1 -- start start 2024-09-17T03:59:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc80b4140 0x7fddc81f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 0x7fddc815b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8160260 con 0x7fddc80705a0 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81600e0 con 0x7fddc80b4140 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fddc8160f00 con 0x7fddc80cd1c0 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53416/0 (socket says 172.21.15.129:53416) 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 -- 172.21.15.129:0/1198444195 learned_addr learned my addr 172.21.15.129:0/1198444195 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7400640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc80b4140 0x7fddc81f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddcd200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 0x7fddc815b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 -- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 msgr2=0x7fddc815b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 0x7fddc815b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 -- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc80b4140 msgr2=0x7fddc81f5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc80b4140 0x7fddc81f5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.285+0000 7fddc7e00640 1 -- 172.21.15.129:0/1198444195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc81f5940 con 0x7fddc80705a0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddc7e00640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fddb800c370 tx=0x7fddb800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb80052d0 con 0x7fddc80705a0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fddb8007020 con 0x7fddc80705a0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8007470 con 0x7fddc80705a0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc815c6f0 con 0x7fddc80705a0 2024-09-17T03:59:02.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.286+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc80c3fb0 con 0x7fddc80705a0 2024-09-17T03:59:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.287+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc80ccc80 con 0x7fddc80705a0 2024-09-17T03:59:02.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.289+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fddb80067d0 con 0x7fddc80705a0 2024-09-17T03:59:02.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.290+0000 7fddbfe00640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 0x7fdda807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:02.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.290+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fddb809a190 con 0x7fddc80705a0 2024-09-17T03:59:02.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.291+0000 7fddc7400640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 0x7fdda807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:02.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.291+0000 7fddc7400640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 0x7fdda807b0d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fddb4033000 tx=0x7fddb4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:02.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.295+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fddc80cd5d0 con 0x7fddc80705a0 2024-09-17T03:59:02.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.513+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddc806a140 con 0x7fddc80705a0 2024-09-17T03:59:02.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.514+0000 7fddbfe00640 1 -- 172.21.15.129:0/1198444195 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fddb8062640 con 0x7fddc80705a0 2024-09-17T03:59:02.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:02.513 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:02.514 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:02.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 msgr2=0x7fdda807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:02.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 0x7fdda807b0d0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fddb4033000 tx=0x7fddb4004730 comp rx=0 tx=0).stop 2024-09-17T03:59:02.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 msgr2=0x7fddc80c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fddb800c370 tx=0x7fddb800c830 comp rx=0 tx=0).stop 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 shutdown_connections 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdda8078bc0 0x7fdda807b0d0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fddc80cd1c0 0x7fddc815b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fddc80b4140 0x7fddc81f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 --2- 172.21.15.129:0/1198444195 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fddc80705a0 0x7fddc80c4a90 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 >> 172.21.15.129:0/1198444195 conn(0x7fddc806d410 msgr2=0x7fddc80ad6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 shutdown_connections 2024-09-17T03:59:02.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:02.516+0000 7fddcf200640 1 -- 172.21.15.129:0/1198444195 wait complete. 2024-09-17T03:59:02.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:02.740 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:03.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:03.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 -- 172.21.15.129:0/13177909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 msgr2=0x7f95a40d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 --2- 172.21.15.129:0/13177909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a40d4540 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f95940098e0 tx=0x7f959402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 -- 172.21.15.129:0/13177909 shutdown_connections 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 --2- 172.21.15.129:0/13177909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a40d4540 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 -- 172.21.15.129:0/13177909 >> 172.21.15.129:0/13177909 conn(0x7f95a409d8d0 msgr2=0x7f95a408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 -- 172.21.15.129:0/13177909 shutdown_connections 2024-09-17T03:59:03.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.972+0000 7f95aa600640 1 -- 172.21.15.129:0/13177909 wait complete. 2024-09-17T03:59:03.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.974+0000 7f95aa600640 1 Processor -- start 2024-09-17T03:59:03.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.974+0000 7f95aa600640 1 -- start start 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a40d16d0 0x7f95a4151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f95a40d4a80 0x7f95a41523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f95a40d0070 con 0x7f95a40b3220 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f95a40cfef0 con 0x7f95a40d16d0 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95aa600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f95a40b9ec0 con 0x7f95a40d4a80 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47392/0 (socket says 172.21.15.129:47392) 2024-09-17T03:59:03.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 -- 172.21.15.129:0/1533427107 learned_addr learned my addr 172.21.15.129:0/1533427107 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:03.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 -- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f95a40d4a80 msgr2=0x7f95a41523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f95a40d4a80 0x7f95a41523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 -- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a40d16d0 msgr2=0x7f95a4151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a40d16d0 0x7f95a4151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.976+0000 7f95a9200640 1 -- 172.21.15.129:0/1533427107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a40ce6a0 con 0x7f95a40b3220 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95a9200640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f9594009a00 tx=0x7f9594005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959403f070 con 0x7f95a40b3220 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9594004050 con 0x7f95a40b3220 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95940044a0 con 0x7f95a40b3220 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a40b4fc0 con 0x7f95a40b3220 2024-09-17T03:59:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.977+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a40b5580 con 0x7f95a40b3220 2024-09-17T03:59:03.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.978+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a40d0c50 con 0x7f95a40b3220 2024-09-17T03:59:03.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.979+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9594042070 con 0x7f95a40b3220 2024-09-17T03:59:03.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.980+0000 7f95a1600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 0x7f958007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:03.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.980+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f95940bbb90 con 0x7f95a40b3220 2024-09-17T03:59:03.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.981+0000 7f959be00640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 0x7f958007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:03.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.982+0000 7f959be00640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 0x7f958007b0d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f958c000f60 tx=0x7f958c0079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:03.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:03.985+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9594083fa0 con 0x7f95a40b3220 2024-09-17T03:59:04.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.205+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95a40ceae0 con 0x7f95a40b3220 2024-09-17T03:59:04.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.205+0000 7f95a1600640 1 -- 172.21.15.129:0/1533427107 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f95940838a0 con 0x7f95a40b3220 2024-09-17T03:59:04.204 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:04.205 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:04.205 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:04.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 msgr2=0x7f958007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:04.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 0x7f958007b0d0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f958c000f60 tx=0x7f958c0079b0 comp rx=0 tx=0).stop 2024-09-17T03:59:04.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 msgr2=0x7f95a4151340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f9594009a00 tx=0x7f9594005040 comp rx=0 tx=0).stop 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 shutdown_connections 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9580078bc0 0x7f958007b0d0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f95a40d4a80 0x7f95a41523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95a40d16d0 0x7f95a4151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 --2- 172.21.15.129:0/1533427107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95a40b3220 0x7f95a4151340 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 >> 172.21.15.129:0/1533427107 conn(0x7f95a409d8d0 msgr2=0x7f95a40b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 shutdown_connections 2024-09-17T03:59:04.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:04.208+0000 7f95aa600640 1 -- 172.21.15.129:0/1533427107 wait complete. 2024-09-17T03:59:04.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:04.435 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:05.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:05.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 -- 172.21.15.129:0/3609255438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540cd1c0 msgr2=0x7ff0541609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 --2- 172.21.15.129:0/3609255438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540cd1c0 0x7ff0541609c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7ff054072080 tx=0x7ff04c02aa10 comp rx=0 tx=0).stop 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 -- 172.21.15.129:0/3609255438 shutdown_connections 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 --2- 172.21.15.129:0/3609255438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540cd1c0 0x7ff0541609c0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 --2- 172.21.15.129:0/3609255438 >> v2:172.21.15.181:3300/0 conn(0x7ff0540b4140 0x7ff0540ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 -- 172.21.15.129:0/3609255438 >> 172.21.15.129:0/3609255438 conn(0x7ff05406d410 msgr2=0x7ff0540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 -- 172.21.15.129:0/3609255438 shutdown_connections 2024-09-17T03:59:05.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.659+0000 7ff05a000640 1 -- 172.21.15.129:0/3609255438 wait complete. 2024-09-17T03:59:05.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.661+0000 7ff05a000640 1 Processor -- start 2024-09-17T03:59:05.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.662+0000 7ff05a000640 1 -- start start 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 0x7ff05415e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 0x7ff05415c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff054160260 con 0x7ff0540b4140 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff0541600e0 con 0x7ff0540705a0 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff05a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff054160f00 con 0x7ff0540cd1c0 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff052a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff052a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47406/0 (socket says 172.21.15.129:47406) 2024-09-17T03:59:05.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.663+0000 7ff052a00640 1 -- 172.21.15.129:0/2296354088 learned_addr learned my addr 172.21.15.129:0/2296354088 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff053e00640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 0x7ff05415c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff053400640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 0x7ff05415e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 -- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 msgr2=0x7ff05415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 0x7ff05415c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 -- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 msgr2=0x7ff05415e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 0x7ff05415e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 -- 172.21.15.129:0/2296354088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05415ecc0 con 0x7ff0540b4140 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff053e00640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 0x7ff05415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff053400640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 0x7ff05415e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:05.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff052a00640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c9a0 tx=0x7ff04000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:05.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040007bb0 con 0x7ff0540b4140 2024-09-17T03:59:05.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff040007e40 con 0x7ff0540b4140 2024-09-17T03:59:05.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04000f6b0 con 0x7ff0540b4140 2024-09-17T03:59:05.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.664+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05415ba80 con 0x7ff0540b4140 2024-09-17T03:59:05.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.665+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05415bd20 con 0x7ff0540b4140 2024-09-17T03:59:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.666+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0540ccc80 con 0x7ff0540b4140 2024-09-17T03:59:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.667+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff04000f850 con 0x7ff0540b4140 2024-09-17T03:59:05.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.668+0000 7ff04b400640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 0x7ff03407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:05.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.668+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7ff04009b480 con 0x7ff0540b4140 2024-09-17T03:59:05.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.668+0000 7ff053400640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 0x7ff03407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:05.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.669+0000 7ff053400640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 0x7ff03407b0d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff044006fd0 tx=0x7ff044008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:05.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.672+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff040014030 con 0x7ff0540b4140 2024-09-17T03:59:05.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.889+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff05406a140 con 0x7ff0540b4140 2024-09-17T03:59:05.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.889+0000 7ff04b400640 1 -- 172.21.15.129:0/2296354088 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7ff040063920 con 0x7ff0540b4140 2024-09-17T03:59:05.889 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:05.889 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:05.889 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 msgr2=0x7ff03407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 0x7ff03407b0d0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7ff044006fd0 tx=0x7ff044008040 comp rx=0 tx=0).stop 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 msgr2=0x7ff054157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c9a0 tx=0x7ff04000ce60 comp rx=0 tx=0).stop 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 shutdown_connections 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff034078bc0 0x7ff03407b0d0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff0540cd1c0 0x7ff05415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff0540b4140 0x7ff054157ba0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 --2- 172.21.15.129:0/2296354088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff0540705a0 0x7ff05415e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 >> 172.21.15.129:0/2296354088 conn(0x7ff05406d410 msgr2=0x7ff0540adb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 shutdown_connections 2024-09-17T03:59:05.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:05.892+0000 7ff05a000640 1 -- 172.21.15.129:0/2296354088 wait complete. 2024-09-17T03:59:06.116 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:06.116 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:07.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:07.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 -- 172.21.15.129:0/1333112085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 msgr2=0x7fd10c0b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 --2- 172.21.15.129:0/1333112085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0b4290 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0b2630 tx=0x7fd0fc02a560 comp rx=0 tx=0).stop 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 -- 172.21.15.129:0/1333112085 shutdown_connections 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 --2- 172.21.15.129:0/1333112085 >> v2:172.21.15.181:3300/0 conn(0x7fd10c163110 0x7fd10c0cf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 --2- 172.21.15.129:0/1333112085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c162d00 0x7fd10c0cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 --2- 172.21.15.129:0/1333112085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0b4290 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 -- 172.21.15.129:0/1333112085 >> 172.21.15.129:0/1333112085 conn(0x7fd10c06d410 msgr2=0x7fd10c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 -- 172.21.15.129:0/1333112085 shutdown_connections 2024-09-17T03:59:07.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.347+0000 7fd111400640 1 -- 172.21.15.129:0/1333112085 wait complete. 2024-09-17T03:59:07.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.349+0000 7fd111400640 1 Processor -- start 2024-09-17T03:59:07.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.350+0000 7fd111400640 1 -- start start 2024-09-17T03:59:07.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:07.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd10c162d00 0x7fd10c158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:07.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c163110 0x7fd10c073f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c0caa40 con 0x7fd10c0cd370 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c0ca8c0 con 0x7fd10c162d00 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd111400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd10c0cabc0 con 0x7fd10c163110 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd10a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd10a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47422/0 (socket says 172.21.15.129:47422) 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd10a800640 1 -- 172.21.15.129:0/3668051597 learned_addr learned my addr 172.21.15.129:0/3668051597 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd109e00640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd10c162d00 0x7fd10c158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.351+0000 7fd10b200640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c163110 0x7fd10c073f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 -- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c163110 msgr2=0x7fd10c073f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c163110 0x7fd10c073f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 -- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd10c162d00 msgr2=0x7fd10c158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd10c162d00 0x7fd10c158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 -- 172.21.15.129:0/3668051597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c0c6440 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd10a800640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc034040 tx=0x7fd0fc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc03e030 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0fc003ca0 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0fc035040 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.352+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10c1f93a0 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.353+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10c1f9960 con 0x7fd10c0cd370 2024-09-17T03:59:07.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.354+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10c0cb0e0 con 0x7fd10c0cd370 2024-09-17T03:59:07.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.355+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0fc041030 con 0x7fd10c0cd370 2024-09-17T03:59:07.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.356+0000 7fd102a00640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 0x7fd0ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:07.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.356+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fd0fc0bb2d0 con 0x7fd10c0cd370 2024-09-17T03:59:07.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.356+0000 7fd109e00640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 0x7fd0ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:07.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.357+0000 7fd109e00640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 0x7fd0ec07b0a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0c6e70 tx=0x7fd0f8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:07.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.361+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0fc083130 con 0x7fd10c0cd370 2024-09-17T03:59:07.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.578+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd10c06a140 con 0x7fd10c0cd370 2024-09-17T03:59:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.579+0000 7fd102a00640 1 -- 172.21.15.129:0/3668051597 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fd0fc087b60 con 0x7fd10c0cd370 2024-09-17T03:59:07.579 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:07.579 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:07.579 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:07.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 msgr2=0x7fd0ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 0x7fd0ec07b0a0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fd10c0c6e70 tx=0x7fd0f8005f90 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 msgr2=0x7fd10c0c9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc034040 tx=0x7fd0fc0038a0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 shutdown_connections 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0ec078b90 0x7fd0ec07b0a0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd10c163110 0x7fd10c073f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd10c162d00 0x7fd10c158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 --2- 172.21.15.129:0/3668051597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd10c0cd370 0x7fd10c0c9ea0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.581+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 >> 172.21.15.129:0/3668051597 conn(0x7fd10c06d410 msgr2=0x7fd10c0ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.582+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 shutdown_connections 2024-09-17T03:59:07.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:07.582+0000 7fd111400640 1 -- 172.21.15.129:0/3668051597 wait complete. 2024-09-17T03:59:07.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:07.808 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:08.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.040+0000 7f9cf9c00640 1 -- 172.21.15.129:0/781898620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 msgr2=0x7f9cd8004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.040+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/781898620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cd8004080 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009b50 tx=0x7f9cd8005830 comp rx=0 tx=0).stop 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.040+0000 7f9cf9c00640 1 -- 172.21.15.129:0/781898620 shutdown_connections 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.040+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/781898620 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cd8004080 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.040+0000 7f9cf9c00640 1 -- 172.21.15.129:0/781898620 >> 172.21.15.129:0/781898620 conn(0x7f9cf406d410 msgr2=0x7f9cf40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:09.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.041+0000 7f9cf9c00640 1 -- 172.21.15.129:0/781898620 shutdown_connections 2024-09-17T03:59:09.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.041+0000 7f9cf9c00640 1 -- 172.21.15.129:0/781898620 wait complete. 2024-09-17T03:59:09.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.043+0000 7f9cf9c00640 1 Processor -- start 2024-09-17T03:59:09.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.043+0000 7f9cf9c00640 1 -- start start 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 0x7f9cf41f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 0x7f9cf41f4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf40cc930 con 0x7f9cd8003c30 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf40cc7b0 con 0x7f9cf40bf4c0 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf9c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9cf40ccab0 con 0x7f9cf40bf8d0 2024-09-17T03:59:09.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47448/0 (socket says 172.21.15.129:47448) 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 -- 172.21.15.129:0/4007382006 learned_addr learned my addr 172.21.15.129:0/4007382006 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf3400640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 0x7f9cf41f4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 -- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 msgr2=0x7f9cf41f4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cebe00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 0x7f9cf41f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 0x7f9cf41f4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 -- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 msgr2=0x7f9cf41f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 0x7f9cf41f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf2a00640 1 -- 172.21.15.129:0/4007382006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf40adb90 con 0x7f9cd8003c30 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cebe00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 0x7f9cf41f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.045+0000 7f9cf3400640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 0x7f9cf41f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9cf2a00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4036040 tx=0x7f9ce40336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4038040 con 0x7f9cd8003c30 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ce4041070 con 0x7f9cd8003c30 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf40ade70 con 0x7f9cd8003c30 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce403c430 con 0x7f9cd8003c30 2024-09-17T03:59:09.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.046+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf40ae430 con 0x7f9cd8003c30 2024-09-17T03:59:09.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.047+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf40b36e0 con 0x7f9cd8003c30 2024-09-17T03:59:09.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.048+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9ce403c5d0 con 0x7f9cd8003c30 2024-09-17T03:59:09.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.049+0000 7f9ceaa00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 0x7f9cd007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:09.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.049+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f9ce40bb810 con 0x7f9cd8003c30 2024-09-17T03:59:09.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.050+0000 7f9cebe00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 0x7f9cd007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:09.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.051+0000 7f9cebe00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 0x7f9cd007b0d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40c6760 tx=0x7f9cdc006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:09.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.054+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9ce4083c20 con 0x7f9cd8003c30 2024-09-17T03:59:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.278+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cf406a140 con 0x7f9cd8003c30 2024-09-17T03:59:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.279+0000 7f9ceaa00640 1 -- 172.21.15.129:0/4007382006 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f9ce4083520 con 0x7f9cd8003c30 2024-09-17T03:59:09.279 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:09.279 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 msgr2=0x7f9cd007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 0x7f9cd007b0d0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9cf40c6760 tx=0x7f9cdc006b10 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 msgr2=0x7f9cf40c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4036040 tx=0x7f9ce40336f0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 shutdown_connections 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9cd0078bc0 0x7f9cd007b0d0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9cf40bf8d0 0x7f9cf41f4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9cf40bf4c0 0x7f9cf41f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 --2- 172.21.15.129:0/4007382006 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9cd8003c30 0x7f9cf40c9c50 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 >> 172.21.15.129:0/4007382006 conn(0x7f9cf406d410 msgr2=0x7f9cf40ced50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 shutdown_connections 2024-09-17T03:59:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:09.281+0000 7f9cf9c00640 1 -- 172.21.15.129:0/4007382006 wait complete. 2024-09-17T03:59:09.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:09.509 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:10.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 -- 172.21.15.129:0/1851469884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 msgr2=0x7f47b00b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 --2- 172.21.15.129:0/1851469884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b00b33f0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f47b006eb20 tx=0x7f479c02a680 comp rx=0 tx=0).stop 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 -- 172.21.15.129:0/1851469884 shutdown_connections 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 --2- 172.21.15.129:0/1851469884 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b00b33f0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 --2- 172.21.15.129:0/1851469884 >> v2:172.21.15.181:3300/0 conn(0x7f47b00706a0 0x7f47b00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 -- 172.21.15.129:0/1851469884 >> 172.21.15.129:0/1851469884 conn(0x7f47b006d410 msgr2=0x7f47b00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:10.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.740+0000 7f47b5a00640 1 -- 172.21.15.129:0/1851469884 shutdown_connections 2024-09-17T03:59:10.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.741+0000 7f47b5a00640 1 -- 172.21.15.129:0/1851469884 wait complete. 2024-09-17T03:59:10.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.743+0000 7f47b5a00640 1 Processor -- start 2024-09-17T03:59:10.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.743+0000 7f47b5a00640 1 -- start start 2024-09-17T03:59:10.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 0x7f47b015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 0x7f47b00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47b015fa70 con 0x7f47b00cc260 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47b00cf390 con 0x7f47b00ae190 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47b5a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f47b015fbf0 con 0x7f47b00b3930 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47ae000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47ae000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47460/0 (socket says 172.21.15.129:47460) 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.744+0000 7f47ae000640 1 -- 172.21.15.129:0/3651986784 learned_addr learned my addr 172.21.15.129:0/3651986784 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 -- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 msgr2=0x7f47b015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47af400640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 0x7f47b00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47aea00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 0x7f47b015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 0x7f47b015d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 -- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 msgr2=0x7f47b00ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 0x7f47b00ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 -- 172.21.15.129:0/3651986784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b015bed0 con 0x7f47b00cc260 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47af400640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 0x7f47b00ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47aea00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 0x7f47b015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47ae000640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f479c0099c0 tx=0x7f479c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.745+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c03e030 con 0x7f47b00cc260 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.746+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f479c0047a0 con 0x7f47b00cc260 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.746+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c035040 con 0x7f47b00cc260 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.746+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b00c9d30 con 0x7f47b00cc260 2024-09-17T03:59:10.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.746+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b015db50 con 0x7f47b00cc260 2024-09-17T03:59:10.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.747+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47b00ada60 con 0x7f47b00cc260 2024-09-17T03:59:10.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.748+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f479c041030 con 0x7f47b00cc260 2024-09-17T03:59:10.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.749+0000 7f47a6a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 0x7f479007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:10.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.749+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f479c0bb550 con 0x7f47b00cc260 2024-09-17T03:59:10.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.750+0000 7f47aea00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 0x7f479007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:10.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.750+0000 7f47aea00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 0x7f479007b0d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009aa0 tx=0x7f47a0005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:10.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.753+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f479c083960 con 0x7f47b00cc260 2024-09-17T03:59:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.975+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47b0160110 con 0x7f47b00cc260 2024-09-17T03:59:10.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.976+0000 7f47a6a00640 1 -- 172.21.15.129:0/3651986784 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f479c083260 con 0x7f47b00cc260 2024-09-17T03:59:10.976 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:10.976 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:10.976 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:10.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 msgr2=0x7f479007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 0x7f479007b0d0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f47a0009aa0 tx=0x7f47a0005db0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 msgr2=0x7f47b0159100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f479c0099c0 tx=0x7f479c0039e0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 shutdown_connections 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4790078bc0 0x7f479007b0d0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47b00ae190 0x7f47b00ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47b00cc260 0x7f47b0159100 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 --2- 172.21.15.129:0/3651986784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47b00b3930 0x7f47b015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 >> 172.21.15.129:0/3651986784 conn(0x7f47b006d410 msgr2=0x7f47b0075e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 shutdown_connections 2024-09-17T03:59:10.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:10.978+0000 7f47b5a00640 1 -- 172.21.15.129:0/3651986784 wait complete. 2024-09-17T03:59:11.203 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:11.203 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:12.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 -- 172.21.15.129:0/1730150484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e880705a0 msgr2=0x7f5e880b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 --2- 172.21.15.129:0/1730150484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e880705a0 0x7f5e880b4140 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f5e88072080 tx=0x7f5e7802a560 comp rx=0 tx=0).stop 2024-09-17T03:59:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 -- 172.21.15.129:0/1730150484 shutdown_connections 2024-09-17T03:59:12.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 --2- 172.21.15.129:0/1730150484 >> v2:172.21.15.181:3300/0 conn(0x7f5e8806eed0 0x7f5e880ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 --2- 172.21.15.129:0/1730150484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e880705a0 0x7f5e880b4140 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 -- 172.21.15.129:0/1730150484 >> 172.21.15.129:0/1730150484 conn(0x7f5e8806d410 msgr2=0x7f5e880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 -- 172.21.15.129:0/1730150484 shutdown_connections 2024-09-17T03:59:12.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.435+0000 7f5e8e600640 1 -- 172.21.15.129:0/1730150484 wait complete. 2024-09-17T03:59:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.437+0000 7f5e8e600640 1 Processor -- start 2024-09-17T03:59:12.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.437+0000 7f5e8e600640 1 -- start start 2024-09-17T03:59:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e8806eed0 0x7f5e881e8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:12.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 0x7f5e880c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e88161250 con 0x7f5e8806f2e0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881610d0 con 0x7f5e880705a0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e8e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5e881613d0 con 0x7f5e8806eed0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47490/0 (socket says 172.21.15.129:47490) 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 -- 172.21.15.129:0/709181071 learned_addr learned my addr 172.21.15.129:0/709181071 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e87400640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e8806eed0 0x7f5e881e8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e87e00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 0x7f5e880c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 -- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e8806eed0 msgr2=0x7f5e881e8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e8806eed0 0x7f5e881e8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 -- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 msgr2=0x7f5e880c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 0x7f5e880c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e7ea00640 1 -- 172.21.15.129:0/709181071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e880c6810 con 0x7f5e8806f2e0 2024-09-17T03:59:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.439+0000 7f5e87e00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 0x7f5e880c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:12.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e7ea00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000e9b0 tx=0x7f5e7000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70004d20 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e7000cb20 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8815ec00 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70005020 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.440+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e880c7810 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.441+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e880cd5a0 con 0x7f5e8806f2e0 2024-09-17T03:59:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.442+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5e700051c0 con 0x7f5e8806f2e0 2024-09-17T03:59:12.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.443+0000 7f5e7fe00640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 0x7f5e6407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:12.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.444+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f5e7009b720 con 0x7f5e8806f2e0 2024-09-17T03:59:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.444+0000 7f5e87400640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 0x7f5e6407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:12.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.445+0000 7f5e87400640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 0x7f5e6407b0d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5e78033000 tx=0x7f5e78004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.448+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5e70016030 con 0x7f5e8806f2e0 2024-09-17T03:59:12.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.669+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e8806a140 con 0x7f5e8806f2e0 2024-09-17T03:59:12.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.669+0000 7f5e7fe00640 1 -- 172.21.15.129:0/709181071 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f5e70063450 con 0x7f5e8806f2e0 2024-09-17T03:59:12.669 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:12.669 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:12.670 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 msgr2=0x7f5e6407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 0x7f5e6407b0d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f5e78033000 tx=0x7f5e78004730 comp rx=0 tx=0).stop 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 msgr2=0x7f5e880c5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f5e7000e9b0 tx=0x7f5e7000ee70 comp rx=0 tx=0).stop 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 shutdown_connections 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e64078bc0 0x7f5e6407b0d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e880705a0 0x7f5e880c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e8806f2e0 0x7f5e880c5d90 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 --2- 172.21.15.129:0/709181071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e8806eed0 0x7f5e881e8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:12.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 >> 172.21.15.129:0/709181071 conn(0x7f5e8806d410 msgr2=0x7f5e880b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:12.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 shutdown_connections 2024-09-17T03:59:12.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:12.672+0000 7f5e8e600640 1 -- 172.21.15.129:0/709181071 wait complete. 2024-09-17T03:59:12.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:12.895 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:13.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.126+0000 7f7f60200640 1 -- 172.21.15.129:0/1199588514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580cf8d0 msgr2=0x7f7f580ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.126+0000 7f7f60200640 1 --2- 172.21.15.129:0/1199588514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580cf8d0 0x7f7f580ca8c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f7f50005fd0 tx=0x7f7f5002aaa0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.126+0000 7f7f60200640 1 -- 172.21.15.129:0/1199588514 shutdown_connections 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.126+0000 7f7f60200640 1 --2- 172.21.15.129:0/1199588514 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580cf8d0 0x7f7f580ca8c0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.126+0000 7f7f60200640 1 --2- 172.21.15.129:0/1199588514 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580b4290 0x7f7f580cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.127+0000 7f7f60200640 1 -- 172.21.15.129:0/1199588514 >> 172.21.15.129:0/1199588514 conn(0x7f7f5806d410 msgr2=0x7f7f580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.127+0000 7f7f60200640 1 -- 172.21.15.129:0/1199588514 shutdown_connections 2024-09-17T03:59:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.127+0000 7f7f60200640 1 -- 172.21.15.129:0/1199588514 wait complete. 2024-09-17T03:59:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.129+0000 7f7f60200640 1 Processor -- start 2024-09-17T03:59:14.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.129+0000 7f7f60200640 1 -- start start 2024-09-17T03:59:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580cd370 0x7f7f581570e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580cf8d0 0x7f7f5815ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5815ed90 con 0x7f7f580b4290 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5815ec10 con 0x7f7f580cd370 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f60200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7f580cae00 con 0x7f7f580cf8d0 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.130+0000 7f7f5d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42046/0 (socket says 172.21.15.129:42046) 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 -- 172.21.15.129:0/190694818 learned_addr learned my addr 172.21.15.129:0/190694818 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5e200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580cf8d0 0x7f7f5815ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5ce00640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580cd370 0x7f7f581570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 -- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580cf8d0 msgr2=0x7f7f5815ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580cf8d0 0x7f7f5815ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 -- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580cd370 msgr2=0x7f7f581570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580cd370 0x7f7f581570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 -- 172.21.15.129:0/190694818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5809db80 con 0x7f7f580b4290 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.131+0000 7f7f5d800640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00c3a0 tx=0x7f7f4c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.132+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017070 con 0x7f7f580b4290 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.132+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f4c0053c0 con 0x7f7f580b4290 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.132+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c003850 con 0x7f7f580b4290 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.132+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5809e100 con 0x7f7f580b4290 2024-09-17T03:59:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.132+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f5809de20 con 0x7f7f580b4290 2024-09-17T03:59:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.133+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5815a730 con 0x7f7f580b4290 2024-09-17T03:59:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.134+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7f4c01a070 con 0x7f7f580b4290 2024-09-17T03:59:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.136+0000 7f7f55600640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 0x7f7f3807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.136+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f7f4c09a130 con 0x7f7f580b4290 2024-09-17T03:59:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.136+0000 7f7f5ce00640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 0x7f7f3807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.137+0000 7f7f5ce00640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 0x7f7f3807b0a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7f5815a520 tx=0x7f7f4400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:14.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.140+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7f580cfce0 con 0x7f7f580b4290 2024-09-17T03:59:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.358+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f580cf390 con 0x7f7f580b4290 2024-09-17T03:59:14.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.359+0000 7f7f55600640 1 -- 172.21.15.129:0/190694818 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f7f4c061e60 con 0x7f7f580b4290 2024-09-17T03:59:14.359 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:14.359 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:14.359 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:14.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 msgr2=0x7f7f3807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 0x7f7f3807b0a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7f5815a520 tx=0x7f7f4400b2e0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 msgr2=0x7f7f58158490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c00c3a0 tx=0x7f7f4c00c860 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 shutdown_connections 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f38078b90 0x7f7f3807b0a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f580cf8d0 0x7f7f5815ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580cd370 0x7f7f581570e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 --2- 172.21.15.129:0/190694818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f580b4290 0x7f7f58158490 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 >> 172.21.15.129:0/190694818 conn(0x7f7f5806d410 msgr2=0x7f7f580b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 shutdown_connections 2024-09-17T03:59:14.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:14.361+0000 7f7f60200640 1 -- 172.21.15.129:0/190694818 wait complete. 2024-09-17T03:59:14.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:14.585 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:15.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:15.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.820+0000 7fbb42a00640 1 -- 172.21.15.129:0/4129464647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb24003ac0 msgr2=0x7fbb24003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:15.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.820+0000 7fbb42a00640 1 --2- 172.21.15.129:0/4129464647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb24003ac0 0x7fbb24003f30 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fbb300016c0 tx=0x7fbb24005830 comp rx=0 tx=0).stop 2024-09-17T03:59:15.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.820+0000 7fbb42a00640 1 -- 172.21.15.129:0/4129464647 shutdown_connections 2024-09-17T03:59:15.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.820+0000 7fbb42a00640 1 --2- 172.21.15.129:0/4129464647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb24003ac0 0x7fbb24003f30 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:15.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.820+0000 7fbb42a00640 1 -- 172.21.15.129:0/4129464647 >> 172.21.15.129:0/4129464647 conn(0x7fbb3c06cfc0 msgr2=0x7fbb3c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:15.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.821+0000 7fbb42a00640 1 -- 172.21.15.129:0/4129464647 shutdown_connections 2024-09-17T03:59:15.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.821+0000 7fbb42a00640 1 -- 172.21.15.129:0/4129464647 wait complete. 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.823+0000 7fbb42a00640 1 Processor -- start 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.823+0000 7fbb42a00640 1 -- start start 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb24003ac0 0x7fbb3c1f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb3c152df0 0x7fbb3c153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0cf950 con 0x7fbb3c15de50 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3c0cf7d0 con 0x7fbb3c152df0 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.824+0000 7fbb42a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c0cfad0 con 0x7fbb24003ac0 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42064/0 (socket says 172.21.15.129:42064) 2024-09-17T03:59:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 -- 172.21.15.129:0/1527442127 learned_addr learned my addr 172.21.15.129:0/1527442127 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb41600640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb24003ac0 0x7fbb3c1f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb40c00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb3c152df0 0x7fbb3c153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 -- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb24003ac0 msgr2=0x7fbb3c1f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb24003ac0 0x7fbb3c1f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 -- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb3c152df0 msgr2=0x7fbb3c153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb3c152df0 0x7fbb3c153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 -- 172.21.15.129:0/1527442127 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c15e7a0 con 0x7fbb3c15de50 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.825+0000 7fbb42000640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fbb34001330 tx=0x7fbb3400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.826+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34010890 con 0x7fbb3c15de50 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.826+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb34017070 con 0x7fbb3c15de50 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.826+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3c153910 con 0x7fbb3c15de50 2024-09-17T03:59:15.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.826+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb34010dc0 con 0x7fbb3c15de50 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.826+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3c155240 con 0x7fbb3c15de50 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.827+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3c072f40 con 0x7fbb3c15de50 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.828+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbb3401c460 con 0x7fbb3c15de50 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.829+0000 7fbb39600640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 0x7fbb2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.829+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fbb3409b0d0 con 0x7fbb3c15de50 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.830+0000 7fbb41600640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 0x7fbb2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.831+0000 7fbb41600640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 0x7fbb2007b0a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fbb30032000 tx=0x7fbb3002c040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:15.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:15.833+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbb3c15e7a0 con 0x7fbb3c15de50 2024-09-17T03:59:16.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.051+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb3c06a140 con 0x7fbb3c15de50 2024-09-17T03:59:16.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.052+0000 7fbb39600640 1 -- 172.21.15.129:0/1527442127 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fbb34063450 con 0x7fbb3c15de50 2024-09-17T03:59:16.052 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:16.052 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:16.052 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:16.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 msgr2=0x7fbb2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:16.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 0x7fbb2007b0a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fbb30032000 tx=0x7fbb3002c040 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 msgr2=0x7fbb3c15e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fbb34001330 tx=0x7fbb3400ea40 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 shutdown_connections 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb20078b90 0x7fbb2007b0a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb3c15de50 0x7fbb3c15e260 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb3c152df0 0x7fbb3c153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 --2- 172.21.15.129:0/1527442127 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb24003ac0 0x7fbb3c1f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 >> 172.21.15.129:0/1527442127 conn(0x7fbb3c06cfc0 msgr2=0x7fbb3c151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 shutdown_connections 2024-09-17T03:59:16.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:16.054+0000 7fbb42a00640 1 -- 172.21.15.129:0/1527442127 wait complete. 2024-09-17T03:59:16.279 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:16.279 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:17.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.511+0000 7f05fb200640 1 -- 172.21.15.129:0/4227439013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc1524c0 msgr2=0x7f05fc0cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.511+0000 7f05fb200640 1 --2- 172.21.15.129:0/4227439013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc1524c0 0x7f05fc0cf5c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009860 tx=0x7f05ec02a510 comp rx=0 tx=0).stop 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.511+0000 7f05fb200640 1 -- 172.21.15.129:0/4227439013 shutdown_connections 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.511+0000 7f05fb200640 1 --2- 172.21.15.129:0/4227439013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc1524c0 0x7f05fc0cf5c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.511+0000 7f05fb200640 1 -- 172.21.15.129:0/4227439013 >> 172.21.15.129:0/4227439013 conn(0x7f05fc06cfc0 msgr2=0x7f05fc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.512+0000 7f05fb200640 1 -- 172.21.15.129:0/4227439013 shutdown_connections 2024-09-17T03:59:17.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.512+0000 7f05fb200640 1 -- 172.21.15.129:0/4227439013 wait complete. 2024-09-17T03:59:17.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.514+0000 7f05fb200640 1 Processor -- start 2024-09-17T03:59:17.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.514+0000 7f05fb200640 1 -- start start 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 0x7f05fc1426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05fc1524c0 0x7f05fc155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc0d0650 con 0x7f05fc1528d0 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc0d04d0 con 0x7f05fc1524c0 2024-09-17T03:59:17.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.515+0000 7f05fb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05fc0d07d0 con 0x7f05fc0cfb00 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42076/0 (socket says 172.21.15.129:42076) 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 -- 172.21.15.129:0/3028927751 learned_addr learned my addr 172.21.15.129:0/3028927751 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05f3e00640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05fc1524c0 0x7f05fc155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05f9e00640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 0x7f05fc1426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 -- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 msgr2=0x7f05fc1426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 0x7f05fc1426f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 -- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05fc1524c0 msgr2=0x7f05fc155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05fc1524c0 0x7f05fc155290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 -- 172.21.15.129:0/3028927751 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc1f9a60 con 0x7f05fc1528d0 2024-09-17T03:59:17.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05f9e00640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 0x7f05fc1426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.516+0000 7f05fa800640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f05f400ca60 tx=0x7f05f400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.517+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4017070 con 0x7f05fc1528d0 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.517+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05f4005070 con 0x7f05fc1528d0 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.517+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05fc1547b0 con 0x7f05fc1528d0 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.517+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f40054c0 con 0x7f05fc1528d0 2024-09-17T03:59:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.517+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05fc154d70 con 0x7f05fc1528d0 2024-09-17T03:59:17.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.518+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05fc0cf5c0 con 0x7f05fc1528d0 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.519+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f05f401a070 con 0x7f05fc1528d0 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.520+0000 7f05f2000640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 0x7f05d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.520+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f05f409a610 con 0x7f05fc1528d0 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.520+0000 7f05f9e00640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 0x7f05d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.521+0000 7f05f9e00640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 0x7f05d807b0d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f05ec034000 tx=0x7f05ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:17.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:59:17.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.524+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f05f4062990 con 0x7f05fc1528d0 2024-09-17T03:59:17.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.748+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05fc06a140 con 0x7f05fc1528d0 2024-09-17T03:59:17.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.749+0000 7f05f2000640 1 -- 172.21.15.129:0/3028927751 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f05f4062290 con 0x7f05fc1528d0 2024-09-17T03:59:17.749 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:17.750 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:17.750 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.751+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 msgr2=0x7f05d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.751+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 0x7f05d807b0d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f05ec034000 tx=0x7f05ec004730 comp rx=0 tx=0).stop 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.751+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 msgr2=0x7f05fc1557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.751+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f05f400ca60 tx=0x7f05f400cf20 comp rx=0 tx=0).stop 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 shutdown_connections 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05d8078bc0 0x7f05d807b0d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05fc1528d0 0x7f05fc1557d0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05fc1524c0 0x7f05fc155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 --2- 172.21.15.129:0/3028927751 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05fc0cfb00 0x7f05fc1426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:17.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 >> 172.21.15.129:0/3028927751 conn(0x7f05fc06cfc0 msgr2=0x7f05fc159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:17.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 shutdown_connections 2024-09-17T03:59:17.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:17.752+0000 7f05fb200640 1 -- 172.21.15.129:0/3028927751 wait complete. 2024-09-17T03:59:17.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:17.979 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:18.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 -- 172.21.15.129:0/4267378343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240706a0 msgr2=0x7f7c240cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 --2- 172.21.15.129:0/4267378343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240706a0 0x7f7c240cc7b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7c24072e00 tx=0x7f7c1802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 -- 172.21.15.129:0/4267378343 shutdown_connections 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 --2- 172.21.15.129:0/4267378343 >> v2:172.21.15.181:3300/0 conn(0x7f7c240b3b00 0x7f7c240ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 --2- 172.21.15.129:0/4267378343 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240706a0 0x7f7c240cc7b0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 -- 172.21.15.129:0/4267378343 >> 172.21.15.129:0/4267378343 conn(0x7f7c2406d410 msgr2=0x7f7c240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:19.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 -- 172.21.15.129:0/4267378343 shutdown_connections 2024-09-17T03:59:19.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.211+0000 7f7c2b400640 1 -- 172.21.15.129:0/4267378343 wait complete. 2024-09-17T03:59:19.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.213+0000 7f7c2b400640 1 Processor -- start 2024-09-17T03:59:19.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.214+0000 7f7c2b400640 1 -- start start 2024-09-17T03:59:19.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:19.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c241f2960 0x7f7c241f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 0x7f7c241f90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240ceb90 con 0x7f7c240cc260 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c240cea10 con 0x7f7c241f2960 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c2b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7c240ced10 con 0x7f7c241f8ce0 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c28a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c28a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42100/0 (socket says 172.21.15.129:42100) 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.215+0000 7f7c28a00640 1 -- 172.21.15.129:0/1553399800 learned_addr learned my addr 172.21.15.129:0/1553399800 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c29400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 0x7f7c241f90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c23e00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c241f2960 0x7f7c241f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 -- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 msgr2=0x7f7c241f90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 0x7f7c241f90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 -- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c241f2960 msgr2=0x7f7c241f87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c241f2960 0x7f7c241f87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 -- 172.21.15.129:0/1553399800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2415e560 con 0x7f7c240cc260 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c29400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 0x7f7c241f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c28a00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7c18034040 tx=0x7f7c18003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c1803e030 con 0x7f7c240cc260 2024-09-17T03:59:19.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c18003e00 con 0x7f7c240cc260 2024-09-17T03:59:19.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c18035040 con 0x7f7c240cc260 2024-09-17T03:59:19.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.216+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c2415e840 con 0x7f7c240cc260 2024-09-17T03:59:19.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.217+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2415ee00 con 0x7f7c240cc260 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.218+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c240cf230 con 0x7f7c240cc260 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.219+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7c18041030 con 0x7f7c240cc260 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.220+0000 7f7c20c00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 0x7f7c0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.220+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f7c180bb5f0 con 0x7f7c240cc260 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.220+0000 7f7c23e00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 0x7f7c0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.221+0000 7f7c23e00640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 0x7f7c0407b0a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7c10006fd0 tx=0x7f7c10008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:19.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.224+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c18083970 con 0x7f7c240cc260 2024-09-17T03:59:19.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.444+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c2406a140 con 0x7f7c240cc260 2024-09-17T03:59:19.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.444+0000 7f7c20c00640 1 -- 172.21.15.129:0/1553399800 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f7c18083270 con 0x7f7c240cc260 2024-09-17T03:59:19.445 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:19.445 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:19.446 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 msgr2=0x7f7c0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 0x7f7c0407b0a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7c10006fd0 tx=0x7f7c10008040 comp rx=0 tx=0).stop 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 msgr2=0x7f7c2415dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7c18034040 tx=0x7f7c18003910 comp rx=0 tx=0).stop 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 shutdown_connections 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c04078b90 0x7f7c0407b0a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c241f8ce0 0x7f7c241f90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c241f2960 0x7f7c241f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 --2- 172.21.15.129:0/1553399800 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c240cc260 0x7f7c2415dd10 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:19.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 >> 172.21.15.129:0/1553399800 conn(0x7f7c2406d410 msgr2=0x7f7c2406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:19.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 shutdown_connections 2024-09-17T03:59:19.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:19.447+0000 7f7c2b400640 1 -- 172.21.15.129:0/1553399800 wait complete. 2024-09-17T03:59:19.670 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:19.671 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:20.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:20.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 -- 172.21.15.129:0/1769232182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000ce170 msgr2=0x7f9a000cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:20.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 --2- 172.21.15.129:0/1769232182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000ce170 0x7f9a000cbbf0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f99f4005fd0 tx=0x7f99f402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 -- 172.21.15.129:0/1769232182 shutdown_connections 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 --2- 172.21.15.129:0/1769232182 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a000b3b90 0x7f9a0006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 --2- 172.21.15.129:0/1769232182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000ce170 0x7f9a000cbbf0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 -- 172.21.15.129:0/1769232182 >> 172.21.15.129:0/1769232182 conn(0x7f9a0006d410 msgr2=0x7f9a000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 -- 172.21.15.129:0/1769232182 shutdown_connections 2024-09-17T03:59:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.902+0000 7f9a08400640 1 -- 172.21.15.129:0/1769232182 wait complete. 2024-09-17T03:59:20.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.904+0000 7f9a08400640 1 Processor -- start 2024-09-17T03:59:20.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.904+0000 7f9a08400640 1 -- start start 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a000ce170 0x7f9a000ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a0015e9d0 0x7f9a000ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0006f410 con 0x7f9a000b2950 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00070910 con 0x7f9a0015e9d0 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a08400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9a0006f590 con 0x7f9a000ce170 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a05a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a05a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42116/0 (socket says 172.21.15.129:42116) 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.906+0000 7f9a05a00640 1 -- 172.21.15.129:0/1350110968 learned_addr learned my addr 172.21.15.129:0/1350110968 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 -- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a000ce170 msgr2=0x7f9a000ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a000ce170 0x7f9a000ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:20.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 -- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a0015e9d0 msgr2=0x7f9a000ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a0015e9d0 0x7f9a000ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 -- 172.21.15.129:0/1350110968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a001e65a0 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a05a00640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034040 tx=0x7f99f40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f403e030 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99f4003ca0 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a001f5a00 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4035040 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.907+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a001f5fc0 con 0x7f9a000b2950 2024-09-17T03:59:20.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.909+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a0006fa40 con 0x7f9a000b2950 2024-09-17T03:59:20.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.910+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f99f4041030 con 0x7f9a000b2950 2024-09-17T03:59:20.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.911+0000 7f99fd600640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 0x7f99e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:20.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.911+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f99f40bb340 con 0x7f9a000b2950 2024-09-17T03:59:20.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.911+0000 7f9a05000640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 0x7f99e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:20.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.912+0000 7f9a05000640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 0x7f99e007b0a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000f60 tx=0x7f99ec0079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:20.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:20.915+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f99f4083760 con 0x7f9a000b2950 2024-09-17T03:59:21.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.130+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a0006a140 con 0x7f9a000b2950 2024-09-17T03:59:21.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.131+0000 7f99fd600640 1 -- 172.21.15.129:0/1350110968 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7f99f4083060 con 0x7f9a000b2950 2024-09-17T03:59:21.131 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:21.131 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:21.132 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:21.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 msgr2=0x7f99e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 0x7f99e007b0a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000f60 tx=0x7f99ec0079b0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 msgr2=0x7f9a001e5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f99f4034040 tx=0x7f99f40038a0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 shutdown_connections 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e0078b90 0x7f99e007b0a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a0015e9d0 0x7f9a000ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a000ce170 0x7f9a000ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 --2- 172.21.15.129:0/1350110968 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a000b2950 0x7f9a001e5cc0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.133+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 >> 172.21.15.129:0/1350110968 conn(0x7f9a0006d410 msgr2=0x7f9a000ca020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.134+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 shutdown_connections 2024-09-17T03:59:21.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:21.134+0000 7f9a08400640 1 -- 172.21.15.129:0/1350110968 wait complete. 2024-09-17T03:59:21.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:21.358 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:22.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:22.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 -- 172.21.15.129:0/1047874372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 msgr2=0x7fae2c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:22.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 --2- 172.21.15.129:0/1047874372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c1609c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fae24005fd0 tx=0x7fae2402aa90 comp rx=0 tx=0).stop 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 -- 172.21.15.129:0/1047874372 shutdown_connections 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 --2- 172.21.15.129:0/1047874372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c1609c0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 --2- 172.21.15.129:0/1047874372 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0b4140 0x7fae2c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 -- 172.21.15.129:0/1047874372 >> 172.21.15.129:0/1047874372 conn(0x7fae2c06d410 msgr2=0x7fae2c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 -- 172.21.15.129:0/1047874372 shutdown_connections 2024-09-17T03:59:22.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.584+0000 7fae33800640 1 -- 172.21.15.129:0/1047874372 wait complete. 2024-09-17T03:59:22.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.586+0000 7fae33800640 1 Processor -- start 2024-09-17T03:59:22.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.587+0000 7fae33800640 1 -- start start 2024-09-17T03:59:22.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0705a0 0x7fae2c15c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:22.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae2c0b4140 0x7fae2c0c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:22.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:22.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c160260 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae2c1600e0 con 0x7fae2c0b4140 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae33800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fae2c160f00 con 0x7fae2c0705a0 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae31800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae31800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:42132/0 (socket says 172.21.15.129:42132) 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.588+0000 7fae31800640 1 -- 172.21.15.129:0/3644175598 learned_addr learned my addr 172.21.15.129:0/3644175598 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae23e00640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae2c0b4140 0x7fae2c0c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae30e00640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0705a0 0x7fae2c15c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 -- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0705a0 msgr2=0x7fae2c15c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0705a0 0x7fae2c15c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 -- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae2c0b4140 msgr2=0x7fae2c0c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:22.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae2c0b4140 0x7fae2c0c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 -- 172.21.15.129:0/3644175598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c0c6e90 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae31800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fae24009d90 tx=0x7fae24003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24044070 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae24003e00 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2c1596d0 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.589+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae240044b0 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.590+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2c15be00 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.591+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2c0c8280 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.592+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fae24047070 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.593+0000 7fae29600640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 0x7fae0807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:22.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.593+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fae240bbd10 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.593+0000 7fae30e00640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 0x7fae0807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:22.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.594+0000 7fae30e00640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 0x7fae0807b0a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0015e0 tx=0x7fae1c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:22.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.597+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fae24032030 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.816+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae2c0adbd0 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.817+0000 7fae29600640 1 -- 172.21.15.129:0/3644175598 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) ==== 95+0+3603 (secure 0 0 0) 0x7fae240098b0 con 0x7fae2c0cd1c0 2024-09-17T03:59:22.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:22.817 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":123,"btime":"2024-09-17T03:58:53:588990+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:58:53.588984+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":8181,"mds_1":8178},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8178":{"gid":8178,"name":"c","rank":1,"incarnation":117,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6834/1563879536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1563879536},{"type":"v1","addr":"172.21.15.129:6834","nonce":1563879536}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:22.818 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 123 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 msgr2=0x7fae0807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 0x7fae0807b0a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fae1c0015e0 tx=0x7fae1c009040 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 msgr2=0x7fae2c0c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fae24009d90 tx=0x7fae24003c10 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 shutdown_connections 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078b90 0x7fae0807b0a0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.819+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae2c0cd1c0 0x7fae2c0c6950 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.820+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae2c0b4140 0x7fae2c0c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.820+0000 7fae33800640 1 --2- 172.21.15.129:0/3644175598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae2c0705a0 0x7fae2c15c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:22.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.820+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 >> 172.21.15.129:0/3644175598 conn(0x7fae2c06d410 msgr2=0x7fae2c0b1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:22.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.820+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 shutdown_connections 2024-09-17T03:59:22.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:22.820+0000 7fae33800640 1 -- 172.21.15.129:0/3644175598 wait complete. 2024-09-17T03:59:23.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:58:53.588984+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 8181, 'mds_1': 8178}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8178': {'gid': 8178, 'name': 'c', 'rank': 1, 'incarnation': 117, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6834/1563879536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1563879536}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 1563879536}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:23.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8178:up:stopping 2024-09-17T03:59:23.834 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:59:24.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:24.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 -- 172.21.15.129:0/62669341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67540ccc80 msgr2=0x7f67540cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 --2- 172.21.15.129:0/62669341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67540ccc80 0x7f67540cd090 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f67540ce800 tx=0x7f674002a540 comp rx=0 tx=0).stop 2024-09-17T03:59:24.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 -- 172.21.15.129:0/62669341 shutdown_connections 2024-09-17T03:59:24.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 --2- 172.21.15.129:0/62669341 >> v2:172.21.15.181:3300/0 conn(0x7f6754070100 0x7f6754070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 --2- 172.21.15.129:0/62669341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67540ccc80 0x7f67540cd090 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 -- 172.21.15.129:0/62669341 >> 172.21.15.129:0/62669341 conn(0x7f675406d410 msgr2=0x7f67540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 -- 172.21.15.129:0/62669341 shutdown_connections 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.268+0000 7f675a400640 1 -- 172.21.15.129:0/62669341 wait complete. 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.271+0000 7f675a400640 1 Processor -- start 2024-09-17T03:59:24.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.271+0000 7f675a400640 1 -- start start 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67540ccc80 0x7f67541ec2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 0x7f67540cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6754070a50 con 0x7f6754070100 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f675415d160 con 0x7f675415da00 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f675a400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6754070bd0 con 0x7f67540ccc80 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f6753400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f6753400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52564/0 (socket says 172.21.15.129:52564) 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.272+0000 7f6753400640 1 -- 172.21.15.129:0/2285832012 learned_addr learned my addr 172.21.15.129:0/2285832012 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6752a00640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67540ccc80 0x7f67541ec2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753e00640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 0x7f67540cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 -- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67540ccc80 msgr2=0x7f67541ec2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67540ccc80 0x7f67541ec2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 -- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 msgr2=0x7f67540cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 0x7f67540cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 -- 172.21.15.129:0/2285832012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67541f6190 con 0x7f6754070100 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753e00640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 0x7f67540cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f6753400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f674400b3f0 tx=0x7f674400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744015020 con 0x7f6754070100 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6744011920 con 0x7f6754070100 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67541f6470 con 0x7f6754070100 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.273+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744011d70 con 0x7f6754070100 2024-09-17T03:59:24.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.274+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67540b0920 con 0x7f6754070100 2024-09-17T03:59:24.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.275+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6754070510 con 0x7f6754070100 2024-09-17T03:59:24.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.276+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6744018020 con 0x7f6754070100 2024-09-17T03:59:24.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.277+0000 7f674b400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 0x7f673407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.277+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f674409a940 con 0x7f6754070100 2024-09-17T03:59:24.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.277+0000 7f6752a00640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 0x7f673407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.278+0000 7f6752a00640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 0x7f673407b0a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6740033000 tx=0x7f6740004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.281+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f674401f030 con 0x7f6754070100 2024-09-17T03:59:24.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.497+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f675406a140 con 0x7f6754070100 2024-09-17T03:59:24.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.497+0000 7f674b400640 1 -- 172.21.15.129:0/2285832012 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+2758 (secure 0 0 0) 0x7f6744062e80 con 0x7f6754070100 2024-09-17T03:59:24.497 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:24.498 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":124,"btime":"2024-09-17T03:59:23:692295+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:59:23.127760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:24.498 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 124 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.499+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 msgr2=0x7f673407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 0x7f673407b0a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f6740033000 tx=0x7f6740004730 comp rx=0 tx=0).stop 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 msgr2=0x7f67541f99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f674400b3f0 tx=0x7f674400b8b0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 shutdown_connections 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6734078b90 0x7f673407b0a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f675415da00 0x7f67540cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67540ccc80 0x7f67541ec2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 --2- 172.21.15.129:0/2285832012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6754070100 0x7f67541f99f0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 >> 172.21.15.129:0/2285832012 conn(0x7f675406d410 msgr2=0x7f67540ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 shutdown_connections 2024-09-17T03:59:24.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.500+0000 7f675a400640 1 -- 172.21.15.129:0/2285832012 wait complete. 2024-09-17T03:59:24.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 124, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:58:12.656922+0000', 'modified': '2024-09-17T03:59:23.127760+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8181}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_8181': {'gid': 8181, 'name': 'a', 'rank': 0, 'incarnation': 112, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/431415571', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 431415571}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 431415571}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8181, 'qdb_cluster': [8181]} 2024-09-17T03:59:24.724 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:59:24.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 -- 172.21.15.129:0/2912918482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd354005e50 msgr2=0x7fd354006260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 --2- 172.21.15.129:0/2912918482 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd354005e50 0x7fd354006260 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd370074820 tx=0x7fd354006940 comp rx=0 tx=0).stop 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36e800640 1 -- 172.21.15.129:0/2912918482 reap_dead start 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 -- 172.21.15.129:0/2912918482 shutdown_connections 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 -- 172.21.15.129:0/2912918482 >> 172.21.15.129:0/2912918482 conn(0x7fd37006cfc0 msgr2=0x7fd37006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 -- 172.21.15.129:0/2912918482 shutdown_connections 2024-09-17T03:59:24.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.947+0000 7fd36f200640 1 -- 172.21.15.129:0/2912918482 wait complete. 2024-09-17T03:59:24.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.949+0000 7fd36f200640 1 Processor -- start 2024-09-17T03:59:24.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.949+0000 7fd36f200640 1 -- start start 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 0x7fd3700b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 0x7fd370145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3700d0840 con 0x7fd370154cf0 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd3700d06c0 con 0x7fd354005e50 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd3700710c0 con 0x7fd370146640 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52586/0 (socket says 172.21.15.129:52586) 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 -- 172.21.15.129:0/2013725014 learned_addr learned my addr 172.21.15.129:0/2013725014 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36de00640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 0x7fd3700b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36d400640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 0x7fd370145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 -- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 msgr2=0x7fd370145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 0x7fd370145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 -- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 msgr2=0x7fd3700b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 0x7fd3700b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36e800640 1 -- 172.21.15.129:0/2013725014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd370155640 con 0x7fd370154cf0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36d400640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 0x7fd370145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.951+0000 7fd36de00640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 0x7fd3700b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd36e800640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd36802b660 tx=0x7fd3680049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd366000640 1 -- 172.21.15.129:0/2013725014 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36800c3a0 con 0x7fd370154cf0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd366000640 1 -- 172.21.15.129:0/2013725014 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd368001a00 con 0x7fd370154cf0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3701515e0 con 0x7fd370154cf0 2024-09-17T03:59:24.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd366000640 1 -- 172.21.15.129:0/2013725014 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36802b970 con 0x7fd370154cf0 2024-09-17T03:59:24.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.952+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd370156d20 con 0x7fd370154cf0 2024-09-17T03:59:24.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.954+0000 7fd366000640 1 -- 172.21.15.129:0/2013725014 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd36803f030 con 0x7fd370154cf0 2024-09-17T03:59:24.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.955+0000 7fd366000640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 0x7fd35007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.955+0000 7fd366000640 1 -- 172.21.15.129:0/2013725014 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fd3680ba7a0 con 0x7fd370154cf0 2024-09-17T03:59:24.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.956+0000 7fd37c440740 1 Processor -- start 2024-09-17T03:59:24.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.956+0000 7fd36de00640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 0x7fd35007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.956+0000 7fd37c440740 1 -- start start 2024-09-17T03:59:24.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd36de00640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 0x7fd35007b0a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd360009630 tx=0x7fd36000b040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e8190150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8190690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584e8119430 con 0x5584e7def270 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5584e81095d0 con 0x5584e7dce590 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.957+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5584e80f8a00 con 0x5584e7dc63b0 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52598/0 (socket says 172.21.15.129:52598) 2024-09-17T03:59:24.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 -- 172.21.15.129:0/91254776 learned_addr learned my addr 172.21.15.129:0/91254776 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36d400640 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e8190150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36e800640 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8190690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 -- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 msgr2=0x5584e8190150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e8190150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 -- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 msgr2=0x5584e8190690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8190690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 -- 172.21.15.129:0/91254776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e80c1520 con 0x5584e7def270 2024-09-17T03:59:24.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.958+0000 7fd36de00640 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd360013ef0 tx=0x7fd360016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ebf3485bb527f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd35aa00640 1 -- 172.21.15.129:0/91254776 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360028070 con 0x5584e7def270 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd35aa00640 1 -- 172.21.15.129:0/91254776 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd36001c030 con 0x5584e7def270 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd35aa00640 1 -- 172.21.15.129:0/91254776 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360023840 con 0x5584e7def270 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 -- 172.21.15.129:0/91254776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 msgr2=0x5584e81dac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd360013ef0 tx=0x7fd360016000 comp rx=0 tx=0).stop 2024-09-17T03:59:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 -- 172.21.15.129:0/91254776 shutdown_connections 2024-09-17T03:59:24.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8190690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e8190150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 --2- 172.21.15.129:0/91254776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7def270 0x5584e81dac10 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.959+0000 7fd37c440740 1 -- 172.21.15.129:0/91254776 >> 172.21.15.129:0/91254776 conn(0x5584e7de33c0 msgr2=0x5584e8195b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.960+0000 7fd37c440740 1 -- 172.21.15.129:0/91254776 shutdown_connections 2024-09-17T03:59:24.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.960+0000 7fd37c440740 1 -- 172.21.15.129:0/91254776 wait complete. 2024-09-17T03:59:24.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.961+0000 7fd37c440740 1 -- start start 2024-09-17T03:59:24.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.961+0000 7fd37c440740 1 Processor -- start 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e81ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7def270 0x5584e81d9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584e809f330 con 0x5584e7dce590 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5584e80fc810 con 0x5584e7def270 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5584e7e69620 con 0x5584e7dc63b0 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.962+0000 7fd37c440740 10 client.0 _ll_register_callbacks cb 0x5584e80a4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52602/0 (socket says 172.21.15.129:52602) 2024-09-17T03:59:24.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 -- 172.21.15.129:0/1381452559 learned_addr learned my addr 172.21.15.129:0/1381452559 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36e800640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7def270 0x5584e81d9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36de00640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e81ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 -- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 msgr2=0x5584e81ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e81ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 -- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7def270 msgr2=0x5584e81d9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7def270 0x5584e81d9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 -- 172.21.15.129:0/1381452559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e7e5a850 con 0x5584e7dce590 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd36d400640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0107f0 tx=0x7fd35c010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc9bf1daf1f0edc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.963+0000 7fd34f400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c004460 con 0x5584e7dce590 2024-09-17T03:59:24.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fd35c01ead0 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd36001c030 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) ==== 24+0+0 (secure 0 0 0) 0x7fd35c01ecd0 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 10 client.8598 fetch_fsmap learned FSMap version 125 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=2+}) -- 0x5584e7ed16b0 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd36001c030 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 125) ==== 2300+0+0 (secure 0 0 0) 0x7fd35c025050 con 0x5584e7dce590 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 10 client.8598 fetch_fsmap finished waiting for FSMap version 125 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 10 client.8598 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 20 client.8598 populate_metadata read hostname 'smithi129' 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e7dade40 0x5584e81d58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 4 client.8598 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.964+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 --> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584e810b520 con 0x5584e7dade40 2024-09-17T03:59:24.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.965+0000 7fd36de00640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e7dade40 0x5584e81d58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.965+0000 7fd36de00640 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e7dade40 0x5584e81d58b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec8ced167e12d0c8 server_cookie=6c8ff69c0cd0d4b2 in_seq=0 out_seq=0 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.965+0000 7fd34f400640 10 client.8598 ms_handle_connect on v2:172.21.15.129:6833/431415571 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd34f400640 1 -- 172.21.15.129:0/1381452559 <== mds.0 v2:172.21.15.129:6833/431415571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5584e810b520 con 0x5584e7dade40 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd34f400640 10 client.8598 handle_command_reply: tid=0 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 client.8598 shutdown 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 msgr2=0x5584e81dd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0107f0 tx=0x7fd35c010cb0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 shutdown_connections 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e7dade40 0x5584e81d58b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7def270 0x5584e81d9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81dd080 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 --2- 172.21.15.129:0/1381452559 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7dc63b0 0x5584e81ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 >> 172.21.15.129:0/1381452559 conn(0x5584e7de33c0 msgr2=0x5584e81bc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd34f400640 0 client.8598 ms_handle_reset on v2:172.21.15.129:6833/431415571 2024-09-17T03:59:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 shutdown_connections 2024-09-17T03:59:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.966+0000 7fd37c440740 1 -- 172.21.15.129:0/1381452559 wait complete. 2024-09-17T03:59:24.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.967+0000 7fd37c440740 20 client.8598 trim_cache size 0 max 16384 2024-09-17T03:59:24.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.985+0000 7fd37c440740 1 Processor -- start 2024-09-17T03:59:24.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.986+0000 7fd37c440740 1 -- start start 2024-09-17T03:59:24.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8195490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81959d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd3701fd430 con 0x5584e7dc63b0 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd370152b60 con 0x5584e7dce590 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5584e81dc6e0 con 0x5584e7def270 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd36de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd36de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52606/0 (socket says 172.21.15.129:52606) 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd36de00640 1 -- 172.21.15.129:0/873416428 learned_addr learned my addr 172.21.15.129:0/873416428 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd36e800640 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.987+0000 7fd36d400640 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 -- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 msgr2=0x5584e81959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 -- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 msgr2=0x5584e8195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8195490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 -- 172.21.15.129:0/873416428 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584e8196cc0 con 0x5584e7dc63b0 2024-09-17T03:59:24.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd36de00640 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c000 tx=0x7fd360004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27694a4245a9dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd35a000640 1 -- 172.21.15.129:0/873416428 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360034070 con 0x5584e7dc63b0 2024-09-17T03:59:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd35a000640 1 -- 172.21.15.129:0/873416428 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd36001c040 con 0x5584e7dc63b0 2024-09-17T03:59:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.988+0000 7fd35a000640 1 -- 172.21.15.129:0/873416428 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd360028040 con 0x5584e7dc63b0 2024-09-17T03:59:24.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 -- 172.21.15.129:0/873416428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 msgr2=0x5584e82024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd36000c000 tx=0x7fd360004730 comp rx=0 tx=0).stop 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 -- 172.21.15.129:0/873416428 shutdown_connections 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dce590 0x5584e8195490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 --2- 172.21.15.129:0/873416428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dc63b0 0x5584e82024b0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 -- 172.21.15.129:0/873416428 >> 172.21.15.129:0/873416428 conn(0x5584e7de33c0 msgr2=0x5584e81bf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 -- 172.21.15.129:0/873416428 shutdown_connections 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.989+0000 7fd37c440740 1 -- 172.21.15.129:0/873416428 wait complete. 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.990+0000 7fd37c440740 1 -- start start 2024-09-17T03:59:24.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.990+0000 7fd37c440740 1 Processor -- start 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.991+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dc63b0 0x5584e8255cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.991+0000 7fd37c440740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd37c440740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81bf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd37c440740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5584e81dd760 con 0x5584e7dce590 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5584e81961d0 con 0x5584e7dc63b0 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd37c440740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5584e81ddcc0 con 0x5584e7def270 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd37c440740 10 client.0 _ll_register_callbacks cb 0x7fd35c010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:59:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52608/0 (socket says 172.21.15.129:52608) 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 -- 172.21.15.129:0/4199926349 learned_addr learned my addr 172.21.15.129:0/4199926349 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36de00640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dc63b0 0x5584e8255cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36e800640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81bf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 -- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 msgr2=0x5584e81bf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81bf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 -- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dc63b0 msgr2=0x5584e8255cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dc63b0 0x5584e8255cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 -- 172.21.15.129:0/4199926349 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584e81dda00 con 0x5584e7dce590 2024-09-17T03:59:24.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36e800640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81bf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:24.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd36d400640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd35c004690 tx=0x7fd35c00fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf08634f23449a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:24.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.992+0000 7fd34f400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:59:24.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35c02c060 con 0x5584e7dce590 2024-09-17T03:59:24.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fd35c005440 con 0x5584e7dce590 2024-09-17T03:59:24.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd36001c040 con 0x5584e7dce590 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=125) ==== 24+0+0 (secure 0 0 0) 0x7fd35c032050 con 0x5584e7dce590 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 10 client.8604 fetch_fsmap learned FSMap version 125 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=125,monmap=2+}) -- 0x5584e825fff0 con 0x5584e7dce590 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd36001c040 con 0x5584e7dce590 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 125) ==== 2300+0+0 (secure 0 0 0) 0x7fd35c035020 con 0x5584e7dce590 2024-09-17T03:59:24.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 10 client.8604 fetch_fsmap finished waiting for FSMap version 125 2024-09-17T03:59:24.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 10 client.8604 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T03:59:24.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 20 client.8604 populate_metadata read hostname 'smithi129' 2024-09-17T03:59:24.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.993+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e825f1a0 0x5584e825f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:24.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd37c440740 4 client.8604 mds_command: new command op to 8181 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:59:24.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 --> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] -- command(tid 0: {"prefix": "status"}) -- 0x5584e8106fd0 con 0x5584e825f1a0 2024-09-17T03:59:24.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd36de00640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e825f1a0 0x5584e825f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:24.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd36de00640 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e825f1a0 0x5584e825f5b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233ddb62d725796b server_cookie=cd0fcbb75bd32c60 in_seq=0 out_seq=0 2024-09-17T03:59:24.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd34f400640 10 client.8604 ms_handle_connect on v2:172.21.15.129:6833/431415571 2024-09-17T03:59:25.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd34f400640 1 -- 172.21.15.129:0/4199926349 <== mds.0 v2:172.21.15.129:6833/431415571 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x5584e8106fd0 con 0x5584e825f1a0 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.994+0000 7fd34f400640 10 client.8604 handle_command_reply: tid=0 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 client.8604 shutdown 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 msgr2=0x5584e81bf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fd35c004690 tx=0x7fd35c00fde0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 shutdown_connections 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:6833/431415571,v1:172.21.15.129:6835/431415571] conn(0x5584e825f1a0 0x5584e825f5b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5584e7def270 0x5584e81bf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5584e7dce590 0x5584e81bf2b0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 --2- 172.21.15.129:0/4199926349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5584e7dc63b0 0x5584e8255cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 >> 172.21.15.129:0/4199926349 conn(0x5584e7de33c0 msgr2=0x5584e8195490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:25.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd34f400640 0 client.8604 ms_handle_reset on v2:172.21.15.129:6833/431415571 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 shutdown_connections 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 1 -- 172.21.15.129:0/4199926349 wait complete. 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.995+0000 7fd37c440740 20 client.8604 trim_cache size 0 max 16384 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 8181, 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:59:25.003 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 71.822606289000007, 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 124, 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 143, 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 143, 2024-09-17T03:59:25.004 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 76.870352742999998 2024-09-17T03:59:25.005 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:25.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 msgr2=0x7fd35007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 0x7fd35007b0a0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd360009630 tx=0x7fd36000b040 comp rx=0 tx=0).stop 2024-09-17T03:59:25.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 msgr2=0x7fd370155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd36802b660 tx=0x7fd3680049a0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 shutdown_connections 2024-09-17T03:59:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd350078b90 0x7fd35007b0a0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd370154cf0 0x7fd370155100 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd370146640 0x7fd370145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 --2- 172.21.15.129:0/2013725014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd354005e50 0x7fd3700b1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 >> 172.21.15.129:0/2013725014 conn(0x7fd37006cfc0 msgr2=0x7fd3700cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:25.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 shutdown_connections 2024-09-17T03:59:25.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:24.996+0000 7fd36f200640 1 -- 172.21.15.129:0/2013725014 wait complete. 2024-09-17T03:59:25.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:59:25.266 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T03:59:25.266 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T03:59:25.267 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:59:25.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:25.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:25.467 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:59:25.467 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:25.467 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:25.467 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:25.503 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:59:25.503 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:25.504 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:25.504 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:25.541 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:25.541 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:25.541 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:25.541 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:25.580 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:59:25.580 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:25.580 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:25.580 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:25.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:25.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:25.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 -- 172.21.15.129:0/3595280338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc720004800 msgr2=0x7fc720004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 --2- 172.21.15.129:0/3595280338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc720004800 0x7fc720004c10 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc728001800 tx=0x7fc720005830 comp rx=0 tx=0).stop 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 -- 172.21.15.129:0/3595280338 shutdown_connections 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 --2- 172.21.15.129:0/3595280338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc720004800 0x7fc720004c10 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 -- 172.21.15.129:0/3595280338 >> 172.21.15.129:0/3595280338 conn(0x7fc73c06d410 msgr2=0x7fc73c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 -- 172.21.15.129:0/3595280338 shutdown_connections 2024-09-17T03:59:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.841+0000 7fc741400640 1 -- 172.21.15.129:0/3595280338 wait complete. 2024-09-17T03:59:25.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.844+0000 7fc741400640 1 Processor -- start 2024-09-17T03:59:25.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.844+0000 7fc741400640 1 -- start start 2024-09-17T03:59:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 0x7fc73c15d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc73c1f95a0 0x7fc73c0c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c160260 con 0x7fc73c0c9710 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc73c1600e0 con 0x7fc720004800 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc741400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc73c06eed0 con 0x7fc73c1f95a0 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc73b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc73b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52636/0 (socket says 172.21.15.129:52636) 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.845+0000 7fc73b200640 1 -- 172.21.15.129:0/281568151 learned_addr learned my addr 172.21.15.129:0/281568151 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73a800640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 0x7fc73c15d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc739e00640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc73c1f95a0 0x7fc73c0c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 -- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc73c1f95a0 msgr2=0x7fc73c0c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc73c1f95a0 0x7fc73c0c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 -- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 msgr2=0x7fc73c15d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 0x7fc73c15d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 -- 172.21.15.129:0/281568151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73c0c9e50 con 0x7fc73c0c9710 2024-09-17T03:59:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73a800640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 0x7fc73c15d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc73b200640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0cdaf0 tx=0x7fc73400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734003d80 con 0x7fc73c0c9710 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc734004b20 con 0x7fc73c0c9710 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73c1f8ac0 con 0x7fc73c0c9710 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734004030 con 0x7fc73c0c9710 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.846+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc73c1f9080 con 0x7fc73c0c9710 2024-09-17T03:59:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.847+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc73c0c9ba0 con 0x7fc73c0c9710 2024-09-17T03:59:25.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.849+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc7340041d0 con 0x7fc73c0c9710 2024-09-17T03:59:25.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.850+0000 7fc732a00640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 0x7fc71c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:25.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.850+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fc73409bd70 con 0x7fc73c0c9710 2024-09-17T03:59:25.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.850+0000 7fc73a800640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 0x7fc71c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:25.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.851+0000 7fc73a800640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 0x7fc71c07b0a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc72c001cd0 tx=0x7fc72c001c00 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:25.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:25.854+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc7340640f0 con 0x7fc73c0c9710 2024-09-17T03:59:26.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.074+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc73c0ccc80 con 0x7fc73c0c9710 2024-09-17T03:59:26.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.075+0000 7fc732a00640 1 -- 172.21.15.129:0/281568151 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+3586 (secure 0 0 0) 0x7fc7340639f0 con 0x7fc73c0c9710 2024-09-17T03:59:26.075 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:26.075 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":125,"btime":"2024-09-17T03:59:24:701718+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2756681375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2756681375},{"type":"v1","addr":"172.21.15.129:6834","nonce":2756681375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:59:23.127760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 125 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 msgr2=0x7fc71c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 0x7fc71c07b0a0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc72c001cd0 tx=0x7fc72c001c00 comp rx=0 tx=0).stop 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 msgr2=0x7fc73c1e9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fc73c0cdaf0 tx=0x7fc73400f790 comp rx=0 tx=0).stop 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 shutdown_connections 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc71c078b90 0x7fc71c07b0a0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc73c0c9710 0x7fc73c1e9cd0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc73c1f95a0 0x7fc73c0c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 --2- 172.21.15.129:0/281568151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc720004800 0x7fc73c15d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 >> 172.21.15.129:0/281568151 conn(0x7fc73c06d410 msgr2=0x7fc73c0ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:26.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 shutdown_connections 2024-09-17T03:59:26.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.078+0000 7fc741400640 1 -- 172.21.15.129:0/281568151 wait complete. 2024-09-17T03:59:26.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.324+0000 7f3d96800640 1 -- 172.21.15.129:0/913563382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d74003a00 msgr2=0x7f3d74003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.324+0000 7f3d96800640 1 --2- 172.21.15.129:0/913563382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d74003a00 0x7f3d74003e70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009b50 tx=0x7f3d74005830 comp rx=0 tx=0).stop 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.324+0000 7f3d8fe00640 1 -- 172.21.15.129:0/913563382 reap_dead start 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.325+0000 7f3d96800640 1 -- 172.21.15.129:0/913563382 shutdown_connections 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.325+0000 7f3d96800640 1 -- 172.21.15.129:0/913563382 >> 172.21.15.129:0/913563382 conn(0x7f3d9006d410 msgr2=0x7f3d900b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.325+0000 7f3d96800640 1 -- 172.21.15.129:0/913563382 shutdown_connections 2024-09-17T03:59:26.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.325+0000 7f3d96800640 1 -- 172.21.15.129:0/913563382 wait complete. 2024-09-17T03:59:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.327+0000 7f3d96800640 1 Processor -- start 2024-09-17T03:59:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.327+0000 7f3d96800640 1 -- start start 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74003a00 0x7f3d900c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d900c9b90 0x7f3d90159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d901612e0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d900cd1c0 con 0x7f3d74003a00 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.328+0000 7f3d96800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d9015b190 con 0x7f3d900c9b90 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52658/0 (socket says 172.21.15.129:52658) 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 -- 172.21.15.129:0/98013295 learned_addr learned my addr 172.21.15.129:0/98013295 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8f400640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74003a00 0x7f3d900c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8fe00640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d900c9b90 0x7f3d90159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 -- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d900c9b90 msgr2=0x7f3d90159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d900c9b90 0x7f3d90159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 -- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74003a00 msgr2=0x7f3d900c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74003a00 0x7f3d900c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 -- 172.21.15.129:0/98013295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d900c9fa0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.329+0000 7f3d8ea00640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c010a10 tx=0x7f3d7c010ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.330+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c00eda0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.330+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d7c012450 con 0x7f3d9015a9b0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.330+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c012880 con 0x7f3d9015a9b0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.330+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d900c70e0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.330+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d9015b6c0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.331+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d900ccc80 con 0x7f3d9015a9b0 2024-09-17T03:59:26.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.332+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d7c012a20 con 0x7f3d9015a9b0 2024-09-17T03:59:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.333+0000 7f3d87400640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 0x7f3d7007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.333+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f3d7c09be80 con 0x7f3d9015a9b0 2024-09-17T03:59:26.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.334+0000 7f3d8f400640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 0x7f3d7007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.334+0000 7f3d8f400640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 0x7f3d7007b0a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009b50 tx=0x7f3d8002d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:26.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.336+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d900c9fa0 con 0x7f3d9015a9b0 2024-09-17T03:59:26.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.555+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d9006a140 con 0x7f3d9015a9b0 2024-09-17T03:59:26.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.556+0000 7f3d87400640 1 -- 172.21.15.129:0/98013295 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+3586 (secure 0 0 0) 0x7f3d7c063d50 con 0x7f3d9015a9b0 2024-09-17T03:59:26.556 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:26.556 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":125,"btime":"2024-09-17T03:59:24:701718+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2756681375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2756681375},{"type":"v1","addr":"172.21.15.129:6834","nonce":2756681375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:59:23.127760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:26.557 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 125 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 msgr2=0x7f3d7007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 0x7f3d7007b0a0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009b50 tx=0x7f3d8002d040 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 msgr2=0x7f3d900c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c010a10 tx=0x7f3d7c010ed0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 shutdown_connections 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d70078b90 0x7f3d7007b0a0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d900c9b90 0x7f3d90159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9015a9b0 0x7f3d900c9650 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 --2- 172.21.15.129:0/98013295 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74003a00 0x7f3d900c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.558+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 >> 172.21.15.129:0/98013295 conn(0x7f3d9006d410 msgr2=0x7f3d901609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.559+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 shutdown_connections 2024-09-17T03:59:26.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.559+0000 7f3d96800640 1 -- 172.21.15.129:0/98013295 wait complete. 2024-09-17T03:59:26.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:26.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.805+0000 7f8a89e00640 1 -- 172.21.15.129:0/1060885153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 msgr2=0x7f8a840b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.805+0000 7f8a89e00640 1 --2- 172.21.15.129:0/1060885153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a840b4140 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009830 tx=0x7f8a7402a4a0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 -- 172.21.15.129:0/1060885153 shutdown_connections 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 --2- 172.21.15.129:0/1060885153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406eed0 0x7f8a840ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 --2- 172.21.15.129:0/1060885153 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a840b4140 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 -- 172.21.15.129:0/1060885153 >> 172.21.15.129:0/1060885153 conn(0x7f8a8406d410 msgr2=0x7f8a840b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 -- 172.21.15.129:0/1060885153 shutdown_connections 2024-09-17T03:59:26.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.806+0000 7f8a89e00640 1 -- 172.21.15.129:0/1060885153 wait complete. 2024-09-17T03:59:26.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.808+0000 7f8a89e00640 1 Processor -- start 2024-09-17T03:59:26.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.808+0000 7f8a89e00640 1 -- start start 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.809+0000 7f8a89e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.809+0000 7f8a89e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 0x7f8a8409d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.809+0000 7f8a89e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a841f9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a89e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84160200 con 0x7f8a8406eed0 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a89e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841612e0 con 0x7f8a840705a0 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a89e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8a840cd1c0 con 0x7f8a8406f2e0 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52674/0 (socket says 172.21.15.129:52674) 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 -- 172.21.15.129:0/2863200171 learned_addr learned my addr 172.21.15.129:0/2863200171 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:26.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a7a000640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 0x7f8a8409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a83400640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a841f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 -- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 msgr2=0x7f8a8409d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 0x7f8a8409d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 -- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 msgr2=0x7f8a841f9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a841f9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a82a00640 1 -- 172.21.15.129:0/2863200171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a8415a3d0 con 0x7f8a8406eed0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a7a000640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 0x7f8a8409d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.810+0000 7f8a83400640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a841f9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a82a00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009950 tx=0x7f8a74003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a74005c90 con 0x7f8a8406eed0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8a74005020 con 0x7f8a8406eed0 2024-09-17T03:59:26.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a74005450 con 0x7f8a8406eed0 2024-09-17T03:59:26.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8415c560 con 0x7f8a8406eed0 2024-09-17T03:59:26.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.811+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a84099860 con 0x7f8a8406eed0 2024-09-17T03:59:26.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.812+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a840ccc80 con 0x7f8a8406eed0 2024-09-17T03:59:26.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.813+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8a740317c0 con 0x7f8a8406eed0 2024-09-17T03:59:26.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.814+0000 7f8a7be00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 0x7f8a6007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:26.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.814+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f8a740bbd70 con 0x7f8a8406eed0 2024-09-17T03:59:26.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.815+0000 7f8a7a000640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 0x7f8a6007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:26.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.816+0000 7f8a7a000640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 0x7f8a6007b0d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8a840ca840 tx=0x7f8a6c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:26.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:26.819+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8a7403f030 con 0x7f8a8406eed0 2024-09-17T03:59:27.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.003+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a8406a140 con 0x7f8a8406eed0 2024-09-17T03:59:27.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.004+0000 7f8a7be00640 1 -- 172.21.15.129:0/2863200171 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+16836 (secure 0 0 0) 0x7f8a74084210 con 0x7f8a8406eed0 2024-09-17T03:59:27.008 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:27.008 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":143,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:29.831098+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:58:10.577224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-17T03:58:11.627337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.4","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 msgr2=0x7f8a6007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 0x7f8a6007b0d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8a840ca840 tx=0x7f8a6c005f90 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 msgr2=0x7f8a84159e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8a74009950 tx=0x7f8a74003c10 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 shutdown_connections 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8a60078bc0 0x7f8a6007b0d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a840705a0 0x7f8a841f9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8a8406f2e0 0x7f8a8409d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 --2- 172.21.15.129:0/2863200171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8a8406eed0 0x7f8a84159e90 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 >> 172.21.15.129:0/2863200171 conn(0x7f8a8406d410 msgr2=0x7f8a840ad3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:27.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 shutdown_connections 2024-09-17T03:59:27.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.007+0000 7f8a89e00640 1 -- 172.21.15.129:0/2863200171 wait complete. 2024-09-17T03:59:27.031 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:59:27.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:27.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 -- 172.21.15.129:0/2540354413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100b4140 msgr2=0x7f15100ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 --2- 172.21.15.129:0/2540354413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100b4140 0x7f15100ccc80 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f1510072e00 tx=0x7f14f802a530 comp rx=0 tx=0).stop 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 -- 172.21.15.129:0/2540354413 shutdown_connections 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 --2- 172.21.15.129:0/2540354413 >> v2:172.21.15.181:3300/0 conn(0x7f15100cd1c0 0x7f15101609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 --2- 172.21.15.129:0/2540354413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100b4140 0x7f15100ccc80 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 -- 172.21.15.129:0/2540354413 >> 172.21.15.129:0/2540354413 conn(0x7f151006d410 msgr2=0x7f15100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 -- 172.21.15.129:0/2540354413 shutdown_connections 2024-09-17T03:59:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.252+0000 7f1515a00640 1 -- 172.21.15.129:0/2540354413 wait complete. 2024-09-17T03:59:27.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.254+0000 7f1515a00640 1 Processor -- start 2024-09-17T03:59:27.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.255+0000 7f1515a00640 1 -- start start 2024-09-17T03:59:27.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15100b4140 0x7f15101f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 0x7f15100c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1510160260 con 0x7f15100705a0 2024-09-17T03:59:27.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f15101600e0 con 0x7f15100b4140 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f1515a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1510160f00 con 0x7f15100cd1c0 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f150ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f150ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52692/0 (socket says 172.21.15.129:52692) 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.256+0000 7f150ea00640 1 -- 172.21.15.129:0/295379893 learned_addr learned my addr 172.21.15.129:0/295379893 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1506000640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15100b4140 0x7f15101f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150f400640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 0x7f15100c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 -- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 msgr2=0x7f15100c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 0x7f15100c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 -- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15100b4140 msgr2=0x7f15101f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15100b4140 0x7f15101f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 -- 172.21.15.129:0/295379893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15101fa3a0 con 0x7f15100705a0 2024-09-17T03:59:27.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150f400640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 0x7f15100c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f150ea00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f150000c3f0 tx=0x7f150000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500017070 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f15000035f0 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1500003a40 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15100d0d90 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.257+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15100c69f0 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.258+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15100ccc80 con 0x7f15100705a0 2024-09-17T03:59:27.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.260+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f150001a070 con 0x7f15100705a0 2024-09-17T03:59:27.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.260+0000 7f1507400640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 0x7f14ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.261+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f1500099ff0 con 0x7f15100705a0 2024-09-17T03:59:27.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.261+0000 7f1506000640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 0x7f14ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.262+0000 7f1506000640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 0x7f14ec07b0a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f14f8034000 tx=0x7f14f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:27.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.265+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f15100cd5d0 con 0x7f15100705a0 2024-09-17T03:59:27.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.485+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f151006a140 con 0x7f15100705a0 2024-09-17T03:59:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.486+0000 7f1507400640 1 -- 172.21.15.129:0/295379893 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+3586 (secure 0 0 0) 0x7f1500061df0 con 0x7f15100705a0 2024-09-17T03:59:27.486 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:27.486 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":125,"btime":"2024-09-17T03:59:24:701718+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2756681375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2756681375},{"type":"v1","addr":"172.21.15.129:6834","nonce":2756681375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:59:23.127760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 125 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.488+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 msgr2=0x7f14ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.488+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 0x7f14ec07b0a0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f14f8034000 tx=0x7f14f8004730 comp rx=0 tx=0).stop 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 msgr2=0x7f15100c3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f150000c3f0 tx=0x7f150000c8b0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 shutdown_connections 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14ec078b90 0x7f14ec07b0a0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f15100cd1c0 0x7f15100c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15100b4140 0x7f15101f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 --2- 172.21.15.129:0/295379893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f15100705a0 0x7f15100c3d40 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 >> 172.21.15.129:0/295379893 conn(0x7f151006d410 msgr2=0x7f15100b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 shutdown_connections 2024-09-17T03:59:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.489+0000 7f1515a00640 1 -- 172.21.15.129:0/295379893 wait complete. 2024-09-17T03:59:27.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 -- 172.21.15.129:0/2387015991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 msgr2=0x7fd1700b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 --2- 172.21.15.129:0/2387015991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700b33e0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fd1700cedf0 tx=0x7fd16402a4a0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 -- 172.21.15.129:0/2387015991 shutdown_connections 2024-09-17T03:59:27.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 --2- 172.21.15.129:0/2387015991 >> v2:172.21.15.181:3300/0 conn(0x7fd1700b6010 0x7fd1700b6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 --2- 172.21.15.129:0/2387015991 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700b33e0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 -- 172.21.15.129:0/2387015991 >> 172.21.15.129:0/2387015991 conn(0x7fd17009d890 msgr2=0x7fd1700d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:27.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 -- 172.21.15.129:0/2387015991 shutdown_connections 2024-09-17T03:59:27.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.735+0000 7fd176800640 1 -- 172.21.15.129:0/2387015991 wait complete. 2024-09-17T03:59:27.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.737+0000 7fd176800640 1 Processor -- start 2024-09-17T03:59:27.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.737+0000 7fd176800640 1 -- start start 2024-09-17T03:59:27.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd17006dea0 0x7fd170151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1700b6010 0x7fd1700ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1700d30e0 con 0x7fd17006fae0 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd1700b6f50 con 0x7fd17006dea0 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd176800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd1700d3260 con 0x7fd1700b6010 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52708/0 (socket says 172.21.15.129:52708) 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 -- 172.21.15.129:0/3120381560 learned_addr learned my addr 172.21.15.129:0/3120381560 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd175400640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd17006dea0 0x7fd170151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd175e00640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1700b6010 0x7fd1700ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 -- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1700b6010 msgr2=0x7fd1700ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1700b6010 0x7fd1700ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 -- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd17006dea0 msgr2=0x7fd170151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd17006dea0 0x7fd170151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.739+0000 7fd174a00640 1 -- 172.21.15.129:0/3120381560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17015ffe0 con 0x7fd17006fae0 2024-09-17T03:59:27.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd174a00640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e790 tx=0x7fd15c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00eea0 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd15c00cdb0 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c010690 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd170160260 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.740+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1700c9870 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.741+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1700d2f00 con 0x7fd17006fae0 2024-09-17T03:59:27.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.743+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd15c010830 con 0x7fd17006fae0 2024-09-17T03:59:27.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.743+0000 7fd16d600640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 0x7fd15007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:27.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.744+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7fd15c09b630 con 0x7fd17006fae0 2024-09-17T03:59:27.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.744+0000 7fd175400640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 0x7fd15007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:27.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.745+0000 7fd175400640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 0x7fd15007b0d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd164034000 tx=0x7fd164004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:27.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.748+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd1700d2f00 con 0x7fd17006fae0 2024-09-17T03:59:27.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.967+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd17006ec40 con 0x7fd17006fae0 2024-09-17T03:59:27.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.968+0000 7fd16d600640 1 -- 172.21.15.129:0/3120381560 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) ==== 95+0+3586 (secure 0 0 0) 0x7fd17015ffe0 con 0x7fd17006fae0 2024-09-17T03:59:27.968 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:27.968 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":125,"btime":"2024-09-17T03:59:24:701718+0000","default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2756681375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2756681375},{"type":"v1","addr":"172.21.15.129:6834","nonce":2756681375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:58:12.656922+0000","modified":"2024-09-17T03:59:23.127760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8181},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_8181":{"gid":8181,"name":"a","rank":0,"incarnation":112,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/431415571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":431415571},{"type":"v1","addr":"172.21.15.129:6835","nonce":431415571}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8181,"qdb_cluster":[8181]},"id":10}]} 2024-09-17T03:59:27.968 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 125 2024-09-17T03:59:27.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 msgr2=0x7fd15007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 0x7fd15007b0d0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd164034000 tx=0x7fd164004730 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 msgr2=0x7fd1700c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00e790 tx=0x7fd15c00ec50 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 shutdown_connections 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd150078bc0 0x7fd15007b0d0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1700b6010 0x7fd1700ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd17006fae0 0x7fd1700c9d90 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 --2- 172.21.15.129:0/3120381560 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd17006dea0 0x7fd170151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.970+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 >> 172.21.15.129:0/3120381560 conn(0x7fd17009d890 msgr2=0x7fd1700b1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.971+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 shutdown_connections 2024-09-17T03:59:27.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:27.971+0000 7fd176800640 1 -- 172.21.15.129:0/3120381560 wait complete. 2024-09-17T03:59:27.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:28.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 -- 172.21.15.129:0/3734751927 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 msgr2=0x7f780c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 --2- 172.21.15.129:0/3734751927 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0b4af0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f77f4005fd0 tx=0x7f77f402a540 comp rx=0 tx=0).stop 2024-09-17T03:59:28.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 -- 172.21.15.129:0/3734751927 shutdown_connections 2024-09-17T03:59:28.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 --2- 172.21.15.129:0/3734751927 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0b4af0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 --2- 172.21.15.129:0/3734751927 >> v2:172.21.15.181:3300/0 conn(0x7f780c0cfb60 0x7f780c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.216+0000 7f7812000640 1 -- 172.21.15.129:0/3734751927 >> 172.21.15.129:0/3734751927 conn(0x7f780c06cfc0 msgr2=0x7f780c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:28.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.217+0000 7f7812000640 1 -- 172.21.15.129:0/3734751927 shutdown_connections 2024-09-17T03:59:28.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.217+0000 7f7812000640 1 -- 172.21.15.129:0/3734751927 wait complete. 2024-09-17T03:59:28.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.219+0000 7f7812000640 1 Processor -- start 2024-09-17T03:59:28.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.219+0000 7f7812000640 1 -- start start 2024-09-17T03:59:28.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f780c0cfb60 0x7f780c155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 0x7f780c15e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f780c15d270 con 0x7f780c0d02c0 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f780c15d0f0 con 0x7f780c0cfb60 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.220+0000 7f7812000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f780c15d3f0 con 0x7f780c0b46e0 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7811600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 0x7f780c15e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:38500/0 (socket says 172.21.15.129:38500) 2024-09-17T03:59:28.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 -- 172.21.15.129:0/3882001198 learned_addr learned my addr 172.21.15.129:0/3882001198 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7803e00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f780c0cfb60 0x7f780c155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 -- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f780c0cfb60 msgr2=0x7f780c155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f780c0cfb60 0x7f780c155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 -- 172.21.15.129:0/3882001198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 msgr2=0x7f780c15e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 0x7f780c15e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7810c00640 1 -- 172.21.15.129:0/3882001198 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c1e9360 con 0x7f780c0b46e0 2024-09-17T03:59:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.221+0000 7f7811600640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 0x7f780c15e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T03:59:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7810c00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00e790 tx=0x7f77fc00ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc00eea0 con 0x7f780c0b46e0 2024-09-17T03:59:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77fc00ccc0 con 0x7f780c0b46e0 2024-09-17T03:59:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc004280 con 0x7f780c0b46e0 2024-09-17T03:59:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780c1e82c0 con 0x7f780c0b46e0 2024-09-17T03:59:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.222+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f780c1e8880 con 0x7f780c0b46e0 2024-09-17T03:59:28.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.223+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780c15d8a0 con 0x7f780c0b46e0 2024-09-17T03:59:28.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.225+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f77fc005030 con 0x7f780c0b46e0 2024-09-17T03:59:28.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.226+0000 7f7809600640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 0x7f77e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.226+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f77fc09b900 con 0x7f780c0b46e0 2024-09-17T03:59:28.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.226+0000 7f7803e00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 0x7f77e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.227+0000 7f7803e00640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 0x7f77e807b0a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f77f4033000 tx=0x7f77f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:28.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.229+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f77fc0636b0 con 0x7f780c0b46e0 2024-09-17T03:59:28.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.413+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f780c06a140 con 0x7f780c0b46e0 2024-09-17T03:59:28.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.415+0000 7f7809600640 1 -- 172.21.15.129:0/3882001198 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v143) ==== 74+0+16836 (secure 0 0 0) 0x7f77fc0680e0 con 0x7f780c0b46e0 2024-09-17T03:59:28.416 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:28.416 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":143,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:58:29.831098+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:58:10.577224+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-17T03:58:11.627337+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.4","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 msgr2=0x7f77e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 0x7f77e807b0a0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f77f4033000 tx=0x7f77f4004730 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 msgr2=0x7f780c0c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00e790 tx=0x7f77fc00ec50 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 shutdown_connections 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f77e8078b90 0x7f77e807b0a0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f780c0d02c0 0x7f780c15e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f780c0cfb60 0x7f780c155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 --2- 172.21.15.129:0/3882001198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f780c0b46e0 0x7f780c0c4fa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.417+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 >> 172.21.15.129:0/3882001198 conn(0x7f780c06cfc0 msgr2=0x7f780c0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:28.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.418+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 shutdown_connections 2024-09-17T03:59:28.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.418+0000 7f7812000640 1 -- 172.21.15.129:0/3882001198 wait complete. 2024-09-17T03:59:28.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T03:59:28.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 -- 172.21.15.129:0/641798045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc3e0 msgr2=0x7f90e415f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 --2- 172.21.15.129:0/641798045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc3e0 0x7f90e415f060 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f90cc005fd0 tx=0x7f90cc02a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 -- 172.21.15.129:0/641798045 shutdown_connections 2024-09-17T03:59:28.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 --2- 172.21.15.129:0/641798045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc3e0 0x7f90e415f060 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 --2- 172.21.15.129:0/641798045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e4070a50 0x7f90e40cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 -- 172.21.15.129:0/641798045 >> 172.21.15.129:0/641798045 conn(0x7f90e406d410 msgr2=0x7f90e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:28.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 -- 172.21.15.129:0/641798045 shutdown_connections 2024-09-17T03:59:28.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.665+0000 7f90e8e00640 1 -- 172.21.15.129:0/641798045 wait complete. 2024-09-17T03:59:28.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.667+0000 7f90e8e00640 1 Processor -- start 2024-09-17T03:59:28.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.667+0000 7f90e8e00640 1 -- start start 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e40cc3e0 0x7f90e41fa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 0x7f90e40b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90e415f7b0 con 0x7f90e4070a50 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f90e40af070 con 0x7f90e40cc910 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e8e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f90e415abe0 con 0x7f90e40cc3e0 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52728/0 (socket says 172.21.15.129:52728) 2024-09-17T03:59:28.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 -- 172.21.15.129:0/2239257439 learned_addr learned my addr 172.21.15.129:0/2239257439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90d9400640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e40cc3e0 0x7f90e41fa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e2800640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 0x7f90e40b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 -- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e40cc3e0 msgr2=0x7f90e41fa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e40cc3e0 0x7f90e41fa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 -- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 msgr2=0x7f90e40b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 0x7f90e40b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e1e00640 1 -- 172.21.15.129:0/2239257439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e41f62e0 con 0x7f90e4070a50 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.669+0000 7f90e2800640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 0x7f90e40b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90e1e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f90d400ea20 tx=0x7f90d400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d400ce40 con 0x7f90e4070a50 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90d4004020 con 0x7f90e4070a50 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e4160ac0 con 0x7f90e4070a50 2024-09-17T03:59:28.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4004470 con 0x7f90e4070a50 2024-09-17T03:59:28.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.670+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e415a920 con 0x7f90e4070a50 2024-09-17T03:59:28.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.671+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e41f60f0 con 0x7f90e4070a50 2024-09-17T03:59:28.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.672+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f90d40035f0 con 0x7f90e4070a50 2024-09-17T03:59:28.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.673+0000 7f90daa00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 0x7f90c007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:28.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.674+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(143..143 src has 1..143) ==== 5738+0+0 (secure 0 0 0) 0x7f90d409b550 con 0x7f90e4070a50 2024-09-17T03:59:28.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.674+0000 7f90d9400640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 0x7f90c007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:28.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.675+0000 7f90d9400640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 0x7f90c007b0d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f90cc033000 tx=0x7f90cc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:28.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.678+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f90d40639f0 con 0x7f90e4070a50 2024-09-17T03:59:28.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.902+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f90e40cbb00 con 0x7f90e4070a50 2024-09-17T03:59:28.903 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T03:59:28.903+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T03:59:28.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.905+0000 7f90daa00640 1 -- 172.21.15.129:0/2239257439 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v126) ==== 161+0+0 (secure 0 0 0) 0x7f90d40632f0 con 0x7f90e4070a50 2024-09-17T03:59:28.905 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T03:59:28.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 msgr2=0x7f90c007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 0x7f90c007b0d0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f90cc033000 tx=0x7f90cc004730 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 msgr2=0x7f90e409daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f90d400ea20 tx=0x7f90d400eee0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 shutdown_connections 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90c0078bc0 0x7f90c007b0d0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90e40cc910 0x7f90e40b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90e40cc3e0 0x7f90e41fa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 --2- 172.21.15.129:0/2239257439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90e4070a50 0x7f90e409daf0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 >> 172.21.15.129:0/2239257439 conn(0x7f90e406d410 msgr2=0x7f90e40caac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 shutdown_connections 2024-09-17T03:59:28.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:28.908+0000 7f90e8e00640 1 -- 172.21.15.129:0/2239257439 wait complete. 2024-09-17T03:59:28.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T03:59:29.049 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 -- 172.21.15.129:0/2497252837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b1c40 msgr2=0x7f86f40b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 --2- 172.21.15.129:0/2497252837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b1c40 0x7f86f40b2050 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f86e0005fd0 tx=0x7f86e002a540 comp rx=0 tx=0).stop 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 -- 172.21.15.129:0/2497252837 shutdown_connections 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 --2- 172.21.15.129:0/2497252837 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b3010 0x7f86f40b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 --2- 172.21.15.129:0/2497252837 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b1c40 0x7f86f40b2050 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.168+0000 7f86f9800640 1 -- 172.21.15.129:0/2497252837 >> 172.21.15.129:0/2497252837 conn(0x7f86f40ccc40 msgr2=0x7f86f40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.169+0000 7f86f9800640 1 -- 172.21.15.129:0/2497252837 shutdown_connections 2024-09-17T03:59:29.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.169+0000 7f86f9800640 1 -- 172.21.15.129:0/2497252837 wait complete. 2024-09-17T03:59:29.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.171+0000 7f86f9800640 1 Processor -- start 2024-09-17T03:59:29.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.171+0000 7f86f9800640 1 -- start start 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86f4074800 0x7f86f41534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b1c40 0x7f86f40c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f86f40b3ae0 con 0x7f86f40b3010 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f86f40b3960 con 0x7f86f4074800 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.172+0000 7f86f9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f86f40b3c60 con 0x7f86f40b1c40 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:29.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52746/0 (socket says 172.21.15.129:52746) 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 -- 172.21.15.129:0/4231002054 learned_addr learned my addr 172.21.15.129:0/4231002054 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f2000640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b1c40 0x7f86f40c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f2a00640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86f4074800 0x7f86f41534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 -- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b1c40 msgr2=0x7f86f40c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b1c40 0x7f86f40c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 -- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86f4074800 msgr2=0x7f86f41534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86f4074800 0x7f86f41534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 -- 172.21.15.129:0/4231002054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f415bcb0 con 0x7f86f40b3010 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.173+0000 7f86f3400640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e960 tx=0x7f86ec00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.174+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec004d20 con 0x7f86f40b3010 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.174+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86ec00cb20 con 0x7f86f40b3010 2024-09-17T03:59:29.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.174+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec005020 con 0x7f86f40b3010 2024-09-17T03:59:29.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.174+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f4153a10 con 0x7f86f40b3010 2024-09-17T03:59:29.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.174+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f40c46d0 con 0x7f86f40b3010 2024-09-17T03:59:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.175+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f40c4160 con 0x7f86f40b3010 2024-09-17T03:59:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.176+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f86ec0051c0 con 0x7f86f40b3010 2024-09-17T03:59:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.177+0000 7f86eaa00640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 0x7f86d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.177+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 5824+0+0 (secure 0 0 0) 0x7f86ec09b6d0 con 0x7f86f40b3010 2024-09-17T03:59:29.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.178+0000 7f86f2a00640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 0x7f86d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:29.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.178+0000 7f86f2a00640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 0x7f86d407b0a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f86e4007fd0 tx=0x7f86e4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:29.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.183+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f86ec016030 con 0x7f86f40b3010 2024-09-17T03:59:29.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.402+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f86f40b4180 con 0x7f86f40b3010 2024-09-17T03:59:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.907+0000 7f86eaa00640 1 -- 172.21.15.129:0/4231002054 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v127) ==== 104+0+0 (secure 0 0 0) 0x7f86ec063b40 con 0x7f86f40b3010 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 msgr2=0x7f86d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 0x7f86d407b0a0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f86e4007fd0 tx=0x7f86e4009040 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 msgr2=0x7f86f40c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e960 tx=0x7f86ec00ee20 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 shutdown_connections 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86d4078b90 0x7f86d407b0a0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86f40b3010 0x7f86f40c5ef0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86f40b1c40 0x7f86f40c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 --2- 172.21.15.129:0/4231002054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86f4074800 0x7f86f41534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.909+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 >> 172.21.15.129:0/4231002054 conn(0x7f86f40ccc40 msgr2=0x7f86f40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.910+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 shutdown_connections 2024-09-17T03:59:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:29.910+0000 7f86f9800640 1 -- 172.21.15.129:0/4231002054 wait complete. 2024-09-17T03:59:29.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T03:59:30.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 -- 172.21.15.129:0/3024276858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 msgr2=0x7f2108158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 --2- 172.21.15.129:0/3024276858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f2108158fb0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f2108159670 tx=0x7f20f802a670 comp rx=0 tx=0).stop 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 -- 172.21.15.129:0/3024276858 shutdown_connections 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 --2- 172.21.15.129:0/3024276858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f2108158fb0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 -- 172.21.15.129:0/3024276858 >> 172.21.15.129:0/3024276858 conn(0x7f21080b1c50 msgr2=0x7f21080b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 -- 172.21.15.129:0/3024276858 shutdown_connections 2024-09-17T03:59:30.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.159+0000 7f2107200640 1 -- 172.21.15.129:0/3024276858 wait complete. 2024-09-17T03:59:30.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.162+0000 7f2107200640 1 Processor -- start 2024-09-17T03:59:30.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.162+0000 7f2107200640 1 -- start start 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 0x7f21081f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21080c9150 0x7f21080c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2108144c00 con 0x7f210815fcd0 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2108144a80 con 0x7f21080cb9e0 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f2107200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2108144d80 con 0x7f21080c9150 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f20fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f20fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52766/0 (socket says 172.21.15.129:52766) 2024-09-17T03:59:30.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.163+0000 7f20fd400640 1 -- 172.21.15.129:0/2973006879 learned_addr learned my addr 172.21.15.129:0/2973006879 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f2105e00640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 0x7f21081f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f2106800640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21080c9150 0x7f21080c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 -- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21080c9150 msgr2=0x7f21080c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21080c9150 0x7f21080c9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 -- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 msgr2=0x7f21081f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 0x7f21081f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 -- 172.21.15.129:0/2973006879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21080c84b0 con 0x7f210815fcd0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f2105e00640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 0x7f21081f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fd400640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f20f000c3c0 tx=0x7f20f000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f0017070 con 0x7f210815fcd0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20f00035f0 con 0x7f210815fcd0 2024-09-17T03:59:30.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f0003a40 con 0x7f210815fcd0 2024-09-17T03:59:30.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.164+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21080c9aa0 con 0x7f210815fcd0 2024-09-17T03:59:30.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.165+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21080b2f50 con 0x7f210815fcd0 2024-09-17T03:59:30.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.166+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2108158fb0 con 0x7f210815fcd0 2024-09-17T03:59:30.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.167+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f20f001a070 con 0x7f210815fcd0 2024-09-17T03:59:30.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.168+0000 7f20fea00640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 0x7f20e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:30.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.168+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(144..144 src has 1..144) ==== 5824+0+0 (secure 0 0 0) 0x7f20f009a070 con 0x7f210815fcd0 2024-09-17T03:59:30.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.168+0000 7f2105e00640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 0x7f20e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:30.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.169+0000 7f2105e00640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 0x7f20e407b0a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f20f8033000 tx=0x7f20f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.172+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f21080c9aa0 con 0x7f210815fcd0 2024-09-17T03:59:30.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.356+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f210806f5c0 con 0x7f210815fcd0 2024-09-17T03:59:30.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.913+0000 7f20fea00640 1 -- 172.21.15.129:0/2973006879 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v145) ==== 181+0+0 (secure 0 0 0) 0x7f21080c84b0 con 0x7f210815fcd0 2024-09-17T03:59:30.914 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T03:59:30.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.916+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 msgr2=0x7f20e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:30.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.916+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 0x7f20e407b0a0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f20f8033000 tx=0x7f20f8004730 comp rx=0 tx=0).stop 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.916+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 msgr2=0x7f21080c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.916+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f20f000c3c0 tx=0x7f20f000c880 comp rx=0 tx=0).stop 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 shutdown_connections 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f20e4078b90 0x7f20e407b0a0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21080c9150 0x7f21080c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f210815fcd0 0x7f21080c7f70 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 --2- 172.21.15.129:0/2973006879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21080cb9e0 0x7f21081f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:30.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.917+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 >> 172.21.15.129:0/2973006879 conn(0x7f21080b1c50 msgr2=0x7f210815d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:30.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.918+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 shutdown_connections 2024-09-17T03:59:30.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:30.918+0000 7f2107200640 1 -- 172.21.15.129:0/2973006879 wait complete. 2024-09-17T03:59:30.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T03:59:31.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 -- 172.21.15.129:0/2581883476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f800b49a0 msgr2=0x7f4f800cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 --2- 172.21.15.129:0/2581883476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f800b49a0 0x7f4f800cc1a0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f4f78005fd0 tx=0x7f4f7802a820 comp rx=0 tx=0).stop 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 -- 172.21.15.129:0/2581883476 shutdown_connections 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 --2- 172.21.15.129:0/2581883476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f800b49a0 0x7f4f800cc1a0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 -- 172.21.15.129:0/2581883476 >> 172.21.15.129:0/2581883476 conn(0x7f4f800ccc40 msgr2=0x7f4f800cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 -- 172.21.15.129:0/2581883476 shutdown_connections 2024-09-17T03:59:31.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.169+0000 7f4f85a00640 1 -- 172.21.15.129:0/2581883476 wait complete. 2024-09-17T03:59:31.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.171+0000 7f4f85a00640 1 Processor -- start 2024-09-17T03:59:31.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.172+0000 7f4f85a00640 1 -- start start 2024-09-17T03:59:31.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 0x7f4f8014b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:31.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 0x7f4f801572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80167f30 con 0x7f4f80159b00 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f800cca00 con 0x7f4f800b49a0 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f85a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4f801680b0 con 0x7f4f80074580 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f7f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52780/0 (socket says 172.21.15.129:52780) 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.173+0000 7f4f7f400640 1 -- 172.21.15.129:0/1498041139 learned_addr learned my addr 172.21.15.129:0/1498041139 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7e000640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 0x7f4f801572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7ea00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 0x7f4f8014b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 -- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 msgr2=0x7f4f8014b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 0x7f4f8014b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 -- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 msgr2=0x7f4f801572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 0x7f4f801572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 -- 172.21.15.129:0/1498041139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8015b7e0 con 0x7f4f80159b00 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7e000640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 0x7f4f801572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7ea00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 0x7f4f8014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f7f400640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009cd0 tx=0x7f4f78032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78037040 con 0x7f4f80159b00 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f78042070 con 0x7f4f80159b00 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78004280 con 0x7f4f80159b00 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.174+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8015a0a0 con 0x7f4f80159b00 2024-09-17T03:59:31.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.175+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8015bd60 con 0x7f4f80159b00 2024-09-17T03:59:31.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.176+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f800b3f70 con 0x7f4f80159b00 2024-09-17T03:59:31.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.177+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4f78004420 con 0x7f4f80159b00 2024-09-17T03:59:31.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.178+0000 7f4f76a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 0x7f4f6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:31.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.178+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(145..145 src has 1..145) ==== 5386+0+0 (secure 0 0 0) 0x7f4f780bbc10 con 0x7f4f80159b00 2024-09-17T03:59:31.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.179+0000 7f4f7ea00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 0x7f4f6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:31.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.179+0000 7f4f7ea00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 0x7f4f6007b0a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000f60 tx=0x7f4f70009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:31.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.182+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4f780840f0 con 0x7f4f80159b00 2024-09-17T03:59:31.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.366+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4f800ca740 con 0x7f4f80159b00 2024-09-17T03:59:31.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.922+0000 7f4f76a00640 1 -- 172.21.15.129:0/1498041139 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v146) ==== 169+0+0 (secure 0 0 0) 0x7f4f780839f0 con 0x7f4f80159b00 2024-09-17T03:59:31.922 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T03:59:31.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 msgr2=0x7f4f6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:31.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 0x7f4f6007b0a0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000f60 tx=0x7f4f70009040 comp rx=0 tx=0).stop 2024-09-17T03:59:31.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 msgr2=0x7f4f801577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:31.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009cd0 tx=0x7f4f78032d10 comp rx=0 tx=0).stop 2024-09-17T03:59:31.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 shutdown_connections 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f60078b90 0x7f4f6007b0a0 secure :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000f60 tx=0x7f4f70009040 comp rx=0 tx=0).stop 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f80159b00 0x7f4f801577e0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f800b49a0 0x7f4f801572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 --2- 172.21.15.129:0/1498041139 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f80074580 0x7f4f8014b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 >> 172.21.15.129:0/1498041139 conn(0x7f4f800ccc40 msgr2=0x7f4f8015c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 shutdown_connections 2024-09-17T03:59:31.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:31.926+0000 7f4f85a00640 1 -- 172.21.15.129:0/1498041139 wait complete. 2024-09-17T03:59:31.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 -- 172.21.15.129:0/3249602212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10e4004800 msgr2=0x7f10e4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 --2- 172.21.15.129:0/3249602212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10e4004800 0x7f10e4004c10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f10e8001690 tx=0x7f10e4005830 comp rx=0 tx=0).stop 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 -- 172.21.15.129:0/3249602212 shutdown_connections 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 --2- 172.21.15.129:0/3249602212 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10e4004800 0x7f10e4004c10 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 -- 172.21.15.129:0/3249602212 >> 172.21.15.129:0/3249602212 conn(0x7f110006d410 msgr2=0x7f11000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 -- 172.21.15.129:0/3249602212 shutdown_connections 2024-09-17T03:59:32.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.175+0000 7f1105e00640 1 -- 172.21.15.129:0/3249602212 wait complete. 2024-09-17T03:59:32.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.178+0000 7f1105e00640 1 Processor -- start 2024-09-17T03:59:32.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.178+0000 7f1105e00640 1 -- start start 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e4004800 0x7f110009a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11001f5f10 0x7f11001f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1100160260 con 0x7f1100159d00 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11001600e0 con 0x7f10e4004800 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f1105e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f110006eed0 con 0x7f11001f5f10 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.179+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52788/0 (socket says 172.21.15.129:52788) 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 -- 172.21.15.129:0/771183058 learned_addr learned my addr 172.21.15.129:0/771183058 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10ff400640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11001f5f10 0x7f11001f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10fea00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e4004800 0x7f110009a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 -- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11001f5f10 msgr2=0x7f11001f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11001f5f10 0x7f11001f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 -- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e4004800 msgr2=0x7f110009a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e4004800 0x7f110009a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 -- 172.21.15.129:0/771183058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110015cfe0 con 0x7f1100159d00 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f10e802a8d0 tx=0x7f10e80063a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.180+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8003a00 con 0x7f1100159d00 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.181+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10e8039070 con 0x7f1100159d00 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.181+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f110015ca20 con 0x7f1100159d00 2024-09-17T03:59:32.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.181+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8033420 con 0x7f1100159d00 2024-09-17T03:59:32.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.181+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f110009b490 con 0x7f1100159d00 2024-09-17T03:59:32.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.182+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f110015c790 con 0x7f1100159d00 2024-09-17T03:59:32.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.183+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10e8044020 con 0x7f1100159d00 2024-09-17T03:59:32.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.184+0000 7f10f7400640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 0x7f10dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.184+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f10e80ba390 con 0x7f1100159d00 2024-09-17T03:59:32.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.185+0000 7f10fea00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 0x7f10dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.185+0000 7f10fea00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 0x7f10dc07b0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000fb0 tx=0x7f10f00067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:32.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.189+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10e8082b60 con 0x7f1100159d00 2024-09-17T03:59:32.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.373+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f11000c70c0 con 0x7f1100159d00 2024-09-17T03:59:32.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.378+0000 7f10f7400640 1 -- 172.21.15.129:0/771183058 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v53) ==== 71+0+0 (secure 0 0 0) 0x7f11000c70c0 con 0x7f1100159d00 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.380+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 msgr2=0x7f10dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.380+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 0x7f10dc07b0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f10f0000fb0 tx=0x7f10f00067a0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 msgr2=0x7f110009a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f10e802a8d0 tx=0x7f10e80063a0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 shutdown_connections 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078bc0 0x7f10dc07b0d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11001f5f10 0x7f11001f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1100159d00 0x7f110009a710 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 --2- 172.21.15.129:0/771183058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e4004800 0x7f110009a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 >> 172.21.15.129:0/771183058 conn(0x7f110006d410 msgr2=0x7f11000b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:32.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 shutdown_connections 2024-09-17T03:59:32.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.381+0000 7f1105e00640 1 -- 172.21.15.129:0/771183058 wait complete. 2024-09-17T03:59:32.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown' 2024-09-17T03:59:32.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 -- 172.21.15.129:0/3204496843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 msgr2=0x7f04440cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 --2- 172.21.15.129:0/3204496843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04440cd290 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f0434005fd0 tx=0x7f043402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 -- 172.21.15.129:0/3204496843 shutdown_connections 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 --2- 172.21.15.129:0/3204496843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f04440cc3c0 0x7f04440cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 --2- 172.21.15.129:0/3204496843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04440cd290 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 -- 172.21.15.129:0/3204496843 >> 172.21.15.129:0/3204496843 conn(0x7f044406ee70 msgr2=0x7f044406f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 -- 172.21.15.129:0/3204496843 shutdown_connections 2024-09-17T03:59:32.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.628+0000 7f0448c00640 1 -- 172.21.15.129:0/3204496843 wait complete. 2024-09-17T03:59:32.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.630+0000 7f0448c00640 1 Processor -- start 2024-09-17T03:59:32.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.630+0000 7f0448c00640 1 -- start start 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04441f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f044415c730 0x7f0444156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f04440b3650 con 0x7f04440cc3c0 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f04440b34d0 con 0x7f04440b5fa0 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0448c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f04440b37d0 con 0x7f044415c730 2024-09-17T03:59:32.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f043be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f043be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52792/0 (socket says 172.21.15.129:52792) 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f043be00640 1 -- 172.21.15.129:0/1085682955 learned_addr learned my addr 172.21.15.129:0/1085682955 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0443400640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04441f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.632+0000 7f0443e00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f044415c730 0x7f0444156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 -- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f044415c730 msgr2=0x7f0444156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f044415c730 0x7f0444156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 -- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 msgr2=0x7f04441f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04441f75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 -- 172.21.15.129:0/1085682955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04441f7af0 con 0x7f04440cc3c0 2024-09-17T03:59:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043be00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f042c00e7c0 tx=0x7f042c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:32.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c00ccc0 con 0x7f04440cc3c0 2024-09-17T03:59:32.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f042c010430 con 0x7f04440cc3c0 2024-09-17T03:59:32.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.633+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04441f3ab0 con 0x7f04440cc3c0 2024-09-17T03:59:32.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.634+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c010860 con 0x7f04440cc3c0 2024-09-17T03:59:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.634+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04441f4070 con 0x7f04440cc3c0 2024-09-17T03:59:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.635+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04440b3cf0 con 0x7f04440cc3c0 2024-09-17T03:59:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.636+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f042c010a00 con 0x7f04440cc3c0 2024-09-17T03:59:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.637+0000 7f043b400640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 0x7f042007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.637+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f042c09b510 con 0x7f04440cc3c0 2024-09-17T03:59:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.638+0000 7f0443400640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 0x7f042007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:32.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.638+0000 7f0443400640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 0x7f042007b0a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0434033000 tx=0x7f0434004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:32.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.641+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f04441f7af0 con 0x7f04440cc3c0 2024-09-17T03:59:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.899+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]} v 0) -- 0x7f044406fa20 con 0x7f04440cc3c0 2024-09-17T03:59:32.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.924+0000 7f043b400640 1 -- 172.21.15.129:0/1085682955 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_migration_on_shutdown"]}]=0 v520) ==== 141+0+0 (secure 0 0 0) 0x7f042c0634e0 con 0x7f04440cc3c0 2024-09-17T03:59:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 msgr2=0x7f042007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 0x7f042007b0a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0434033000 tx=0x7f0434004730 comp rx=0 tx=0).stop 2024-09-17T03:59:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 msgr2=0x7f04441f62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f042c00e7c0 tx=0x7f042c00ec80 comp rx=0 tx=0).stop 2024-09-17T03:59:32.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 shutdown_connections 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0420078b90 0x7f042007b0a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f044415c730 0x7f0444156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f04440cc3c0 0x7f04441f62f0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 --2- 172.21.15.129:0/1085682955 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440b5fa0 0x7f04441f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 >> 172.21.15.129:0/1085682955 conn(0x7f044406ee70 msgr2=0x7f04440b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 shutdown_connections 2024-09-17T03:59:32.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:32.927+0000 7f0448c00640 1 -- 172.21.15.129:0/1085682955 wait complete. 2024-09-17T03:59:32.957 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, any non-purgeable strays ... ok 2024-09-17T03:59:32.957 INFO:tasks.cephfs_test_runner:Starting test: test_mv_cleanup (tasks.cephfs.test_strays.TestStrays) That when doing a rename from A to B, and B has no hardlinks, 2024-09-17T03:59:32.958 INFO:tasks.cephfs_test_runner:test_mv_cleanup (tasks.cephfs.test_strays.TestStrays) 2024-09-17T03:59:32.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 -- 172.21.15.129:0/714209341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99e8003c80 msgr2=0x7f99e80040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 --2- 172.21.15.129:0/714209341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f99e8003c80 0x7f99e80040d0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f99f4006fb0 tx=0x7f99e8005830 comp rx=0 tx=0).stop 2024-09-17T03:59:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a09600640 1 -- 172.21.15.129:0/714209341 reap_dead start 2024-09-17T03:59:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 -- 172.21.15.129:0/714209341 shutdown_connections 2024-09-17T03:59:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 -- 172.21.15.129:0/714209341 >> 172.21.15.129:0/714209341 conn(0x7f9a0406cfc0 msgr2=0x7f9a0406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 -- 172.21.15.129:0/714209341 shutdown_connections 2024-09-17T03:59:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.180+0000 7f9a0a000640 1 -- 172.21.15.129:0/714209341 wait complete. 2024-09-17T03:59:33.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.182+0000 7f9a0a000640 1 Processor -- start 2024-09-17T03:59:33.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.182+0000 7f9a0a000640 1 -- start start 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99e8003c80 0x7f9a040ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a040ad1a0 0x7f9a040b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a040cef20 con 0x7f9a041f5de0 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9a040ceda0 con 0x7f9a040ad1a0 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.183+0000 7f9a0a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9a040d0a50 con 0x7f99e8003c80 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52810/0 (socket says 172.21.15.129:52810) 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 -- 172.21.15.129:0/951293918 learned_addr learned my addr 172.21.15.129:0/951293918 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a08c00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99e8003c80 0x7f9a040ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a09600640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a040ad1a0 0x7f9a040b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 -- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99e8003c80 msgr2=0x7f9a040ab4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99e8003c80 0x7f9a040ab4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 -- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a040ad1a0 msgr2=0x7f9a040b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a040ad1a0 0x7f9a040b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.184+0000 7f9a03e00640 1 -- 172.21.15.129:0/951293918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a040ad5b0 con 0x7f9a041f5de0 2024-09-17T03:59:33.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a03e00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011ab0 tx=0x7f99f0011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:33.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f001f070 con 0x7f9a041f5de0 2024-09-17T03:59:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f99f000fe40 con 0x7f9a041f5de0 2024-09-17T03:59:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f00126b0 con 0x7f9a041f5de0 2024-09-17T03:59:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a040abe80 con 0x7f9a041f5de0 2024-09-17T03:59:33.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.185+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a040b00a0 con 0x7f9a041f5de0 2024-09-17T03:59:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.186+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a040abbf0 con 0x7f9a041f5de0 2024-09-17T03:59:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.187+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f99f0022070 con 0x7f9a041f5de0 2024-09-17T03:59:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.188+0000 7f9a00c00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 0x7f99e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.189+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f99f009b590 con 0x7f9a041f5de0 2024-09-17T03:59:33.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.189+0000 7f9a08c00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 0x7f99e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.190+0000 7f9a08c00640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 0x7f99e407b0a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f99f4033000 tx=0x7f99f402d000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:33.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.193+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9a040ad5b0 con 0x7f9a041f5de0 2024-09-17T03:59:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.388+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9a040d0340 con 0x7f9a041f5de0 2024-09-17T03:59:33.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.388+0000 7f9a00c00640 1 -- 172.21.15.129:0/951293918 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v146) ==== 74+0+13090 (secure 0 0 0) 0x7f99f00635e0 con 0x7f9a041f5de0 2024-09-17T03:59:33.389 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:33.389 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":146,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:31.915354+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/431415571":"2024-09-18T03:59:28.904199+0000","172.21.15.129:6833/431415571":"2024-09-18T03:59:28.904199+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 msgr2=0x7f99e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 0x7f99e407b0a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f99f4033000 tx=0x7f99f402d000 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 msgr2=0x7f9a040acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f99f0011ab0 tx=0x7f99f0011f70 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 shutdown_connections 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f99e4078b90 0x7f99e407b0a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a040ad1a0 0x7f9a040b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a041f5de0 0x7f9a040acc60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 --2- 172.21.15.129:0/951293918 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99e8003c80 0x7f9a040ab4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 >> 172.21.15.129:0/951293918 conn(0x7f9a0406cfc0 msgr2=0x7f9a04143960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 shutdown_connections 2024-09-17T03:59:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.391+0000 7f9a0a000640 1 -- 172.21.15.129:0/951293918 wait complete. 2024-09-17T03:59:33.416 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T03:59:33.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 -- 172.21.15.129:0/2027211098 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b080cdd90 msgr2=0x7f5b08070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 --2- 172.21.15.129:0/2027211098 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b080cdd90 0x7f5b08070290 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009830 tx=0x7f5af802a480 comp rx=0 tx=0).stop 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 -- 172.21.15.129:0/2027211098 shutdown_connections 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 --2- 172.21.15.129:0/2027211098 >> v2:172.21.15.181:3300/0 conn(0x7f5b080b2350 0x7f5b080b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 --2- 172.21.15.129:0/2027211098 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b080cdd90 0x7f5b08070290 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 -- 172.21.15.129:0/2027211098 >> 172.21.15.129:0/2027211098 conn(0x7f5b080ccc40 msgr2=0x7f5b080cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 -- 172.21.15.129:0/2027211098 shutdown_connections 2024-09-17T03:59:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.639+0000 7f5b0f000640 1 -- 172.21.15.129:0/2027211098 wait complete. 2024-09-17T03:59:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.641+0000 7f5b0f000640 1 Processor -- start 2024-09-17T03:59:33.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.642+0000 7f5b0f000640 1 -- start start 2024-09-17T03:59:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b0815ee00 0x7f5b0814d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b081f9060 0x7f5b081f9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080b3550 con 0x7f5b080cdd90 2024-09-17T03:59:33.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b080711e0 con 0x7f5b0815ee00 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b0f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5b080b36d0 con 0x7f5b081f9060 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b07e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52820/0 (socket says 172.21.15.129:52820) 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.643+0000 7f5b07e00640 1 -- 172.21.15.129:0/1181069922 learned_addr learned my addr 172.21.15.129:0/1181069922 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b0d000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b081f9060 0x7f5b081f9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5aff400640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b0815ee00 0x7f5b0814d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 -- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b081f9060 msgr2=0x7f5b081f9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b081f9060 0x7f5b081f9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 -- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b0815ee00 msgr2=0x7f5b0814d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b0815ee00 0x7f5b0814d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 -- 172.21.15.129:0/1181069922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b081f99b0 con 0x7f5b080cdd90 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b07e00640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009950 tx=0x7f5af8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.644+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8031d60 con 0x7f5b080cdd90 2024-09-17T03:59:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.645+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5af8033e10 con 0x7f5b080cdd90 2024-09-17T03:59:33.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.645+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b080c7dd0 con 0x7f5b080cdd90 2024-09-17T03:59:33.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.645+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8005260 con 0x7f5b080cdd90 2024-09-17T03:59:33.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.645+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0809b1b0 con 0x7f5b080cdd90 2024-09-17T03:59:33.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.646+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b080b3010 con 0x7f5b080cdd90 2024-09-17T03:59:33.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.647+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5af8005400 con 0x7f5b080cdd90 2024-09-17T03:59:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.648+0000 7f5b04c00640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 0x7f5ae407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.648+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f5af80bbba0 con 0x7f5b080cdd90 2024-09-17T03:59:33.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.648+0000 7f5aff400640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 0x7f5ae407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:33.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.649+0000 7f5aff400640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 0x7f5ae407b0a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5b080703b0 tx=0x7f5af0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:33.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.653+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5b081f99b0 con 0x7f5b080cdd90 2024-09-17T03:59:33.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.839+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5b080adf00 con 0x7f5b080cdd90 2024-09-17T03:59:33.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.839+0000 7f5b04c00640 1 -- 172.21.15.129:0/1181069922 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v146) ==== 93+0+10 (secure 0 0 0) 0x7f5af803f030 con 0x7f5b080cdd90 2024-09-17T03:59:33.839 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T03:59:33.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 msgr2=0x7f5ae407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 0x7f5ae407b0a0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5b080703b0 tx=0x7f5af0009040 comp rx=0 tx=0).stop 2024-09-17T03:59:33.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 msgr2=0x7f5b0809d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009950 tx=0x7f5af8033c20 comp rx=0 tx=0).stop 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 shutdown_connections 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae4078b90 0x7f5ae407b0a0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b081f9060 0x7f5b081f9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b0815ee00 0x7f5b0814d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 --2- 172.21.15.129:0/1181069922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b080cdd90 0x7f5b0809d7d0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 >> 172.21.15.129:0/1181069922 conn(0x7f5b080ccc40 msgr2=0x7f5b0815da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 shutdown_connections 2024-09-17T03:59:33.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:33.842+0000 7f5b0f000640 1 -- 172.21.15.129:0/1181069922 wait complete. 2024-09-17T03:59:33.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup' 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 -- 172.21.15.129:0/3804792757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 msgr2=0x7fcb440b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 --2- 172.21.15.129:0/3804792757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440b4af0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7fcb441609e0 tx=0x7fcb2c02a4a0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 -- 172.21.15.129:0/3804792757 shutdown_connections 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 --2- 172.21.15.129:0/3804792757 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440b4af0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 --2- 172.21.15.129:0/3804792757 >> v2:172.21.15.181:3300/0 conn(0x7fcb440cfb60 0x7fcb44160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 -- 172.21.15.129:0/3804792757 >> 172.21.15.129:0/3804792757 conn(0x7fcb4406cfc0 msgr2=0x7fcb4406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:34.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 -- 172.21.15.129:0/3804792757 shutdown_connections 2024-09-17T03:59:34.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.087+0000 7fcb49200640 1 -- 172.21.15.129:0/3804792757 wait complete. 2024-09-17T03:59:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.090+0000 7fcb49200640 1 Processor -- start 2024-09-17T03:59:34.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.090+0000 7fcb49200640 1 -- start start 2024-09-17T03:59:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:34.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 0x7fcb440aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 0x7fcb440aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4415d1d0 con 0x7fcb440b46e0 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4415d050 con 0x7fcb440cfb60 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb49200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4415d350 con 0x7fcb440d02c0 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb43400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:34.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb43400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55780/0 (socket says 172.21.15.129:55780) 2024-09-17T03:59:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.091+0000 7fcb43400640 1 -- 172.21.15.129:0/4232797439 learned_addr learned my addr 172.21.15.129:0/4232797439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb3aa00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 0x7fcb440aa7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 -- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 msgr2=0x7fcb440aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43e00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 0x7fcb440aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:34.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 0x7fcb440aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 -- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 msgr2=0x7fcb440aa7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 0x7fcb440aa7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 -- 172.21.15.129:0/4232797439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb440a9e10 con 0x7fcb440b46e0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb3aa00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 0x7fcb440aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43e00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 0x7fcb440aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb43400640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400d6d0 tx=0x7fcb3400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34017070 con 0x7fcb440b46e0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcb34004310 con 0x7fcb440b46e0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb34009a80 con 0x7fcb440b46e0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb440aa0f0 con 0x7fcb440b46e0 2024-09-17T03:59:34.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.092+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb441588a0 con 0x7fcb440b46e0 2024-09-17T03:59:34.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.094+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb4415d800 con 0x7fcb440b46e0 2024-09-17T03:59:34.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.095+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcb340035f0 con 0x7fcb440b46e0 2024-09-17T03:59:34.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.096+0000 7fcb3be00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 0x7fcb2007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:34.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.096+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7fcb3409abf0 con 0x7fcb440b46e0 2024-09-17T03:59:34.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.097+0000 7fcb3aa00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 0x7fcb2007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:34.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.097+0000 7fcb3aa00640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 0x7fcb2007b0d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c034000 tx=0x7fcb2c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.100+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcb34063450 con 0x7fcb440b46e0 2024-09-17T03:59:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.352+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]} v 0) -- 0x7fcb4406a140 con 0x7fcb440b46e0 2024-09-17T03:59:34.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.934+0000 7fcb3be00640 1 -- 172.21.15.129:0/4232797439 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]}]=0 v522) ==== 133+0+0 (secure 0 0 0) 0x7fcb34062d50 con 0x7fcb440b46e0 2024-09-17T03:59:34.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 msgr2=0x7fcb2007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:34.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 0x7fcb2007b0d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c034000 tx=0x7fcb2c004730 comp rx=0 tx=0).stop 2024-09-17T03:59:34.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 msgr2=0x7fcb440a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400d6d0 tx=0x7fcb3400db90 comp rx=0 tx=0).stop 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 shutdown_connections 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcb20078bc0 0x7fcb2007b0d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcb440d02c0 0x7fcb440aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb440cfb60 0x7fcb440aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 --2- 172.21.15.129:0/4232797439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcb440b46e0 0x7fcb440a94c0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 >> 172.21.15.129:0/4232797439 conn(0x7fcb4406cfc0 msgr2=0x7fcb4415f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 shutdown_connections 2024-09-17T03:59:34.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:34.937+0000 7fcb49200640 1 -- 172.21.15.129:0/4232797439 wait complete. 2024-09-17T03:59:34.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 -- 172.21.15.129:0/4038320345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 msgr2=0x7fd4480ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 --2- 172.21.15.129:0/4038320345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd4480ccc80 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fd448072e00 tx=0x7fd43402a560 comp rx=0 tx=0).stop 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 -- 172.21.15.129:0/4038320345 shutdown_connections 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 --2- 172.21.15.129:0/4038320345 >> v2:172.21.15.181:3300/0 conn(0x7fd4480cd1c0 0x7fd4481609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 --2- 172.21.15.129:0/4038320345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd4480ccc80 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.181+0000 7fd44f800640 1 -- 172.21.15.129:0/4038320345 >> 172.21.15.129:0/4038320345 conn(0x7fd44806d410 msgr2=0x7fd4480b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.182+0000 7fd44f800640 1 -- 172.21.15.129:0/4038320345 shutdown_connections 2024-09-17T03:59:35.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.182+0000 7fd44f800640 1 -- 172.21.15.129:0/4038320345 wait complete. 2024-09-17T03:59:35.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.184+0000 7fd44f800640 1 Processor -- start 2024-09-17T03:59:35.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.184+0000 7fd44f800640 1 -- start start 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4480705a0 0x7fd4481f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4480cd1c0 0x7fd4480c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd448160260 con 0x7fd4480b4140 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4481600e0 con 0x7fd4480cd1c0 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.185+0000 7fd44f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd448160f00 con 0x7fd4480705a0 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55806/0 (socket says 172.21.15.129:55806) 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 -- 172.21.15.129:0/2456005698 learned_addr learned my addr 172.21.15.129:0/2456005698 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd44d800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4480cd1c0 0x7fd4480c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd44ce00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4480705a0 0x7fd4481f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 -- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4480705a0 msgr2=0x7fd4481f58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4480705a0 0x7fd4481f58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 -- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4480cd1c0 msgr2=0x7fd4480c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4480cd1c0 0x7fd4480c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 -- 172.21.15.129:0/2456005698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd448159aa0 con 0x7fd4480b4140 2024-09-17T03:59:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.186+0000 7fd447e00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd434034040 tx=0x7fd434003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.187+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43403e030 con 0x7fd4480b4140 2024-09-17T03:59:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.187+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd434003e00 con 0x7fd4480b4140 2024-09-17T03:59:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.187+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd434035040 con 0x7fd4480b4140 2024-09-17T03:59:35.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.187+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4481f5e20 con 0x7fd4480b4140 2024-09-17T03:59:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.187+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4480cc2a0 con 0x7fd4480b4140 2024-09-17T03:59:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.188+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4480ccc80 con 0x7fd4480b4140 2024-09-17T03:59:35.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.189+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd434041030 con 0x7fd4480b4140 2024-09-17T03:59:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.190+0000 7fd444c00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 0x7fd42807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.190+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7fd4340bb1c0 con 0x7fd4480b4140 2024-09-17T03:59:35.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.191+0000 7fd44ce00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 0x7fd42807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.192+0000 7fd44ce00640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 0x7fd42807b0a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd43c006fd0 tx=0x7fd43c009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:35.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.194+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd448159aa0 con 0x7fd4480b4140 2024-09-17T03:59:35.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.378+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fd44806a140 con 0x7fd4480b4140 2024-09-17T03:59:35.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.379+0000 7fd444c00640 1 -- 172.21.15.129:0/2456005698 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v53) ==== 59+0+2294 (secure 0 0 0) 0x7fd4480cd5d0 con 0x7fd4480b4140 2024-09-17T03:59:35.380 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T03:59:35.380 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T03:59:35.380 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T03:59:35.380 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T03:59:35.380 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T03:59:35.381 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T03:59:35.381 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T03:59:35.381 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T03:59:35.381 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T03:59:35.381 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T03:59:35.382 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T03:59:35.383 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T03:59:35.384 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T03:59:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 msgr2=0x7fd42807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 0x7fd42807b0a0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd43c006fd0 tx=0x7fd43c009060 comp rx=0 tx=0).stop 2024-09-17T03:59:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 msgr2=0x7fd44815c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fd434034040 tx=0x7fd434003910 comp rx=0 tx=0).stop 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 shutdown_connections 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd428078b90 0x7fd42807b0a0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4480cd1c0 0x7fd4480c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4480b4140 0x7fd44815c0d0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 --2- 172.21.15.129:0/2456005698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4480705a0 0x7fd4481f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.381+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 >> 172.21.15.129:0/2456005698 conn(0x7fd44806d410 msgr2=0x7fd4480b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:35.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.382+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 shutdown_connections 2024-09-17T03:59:35.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.382+0000 7fd44f800640 1 -- 172.21.15.129:0/2456005698 wait complete. 2024-09-17T03:59:35.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 -- 172.21.15.129:0/4116830374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 msgr2=0x7f580c0ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 --2- 172.21.15.129:0/4116830374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0ce910 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57f4009860 tx=0x7f57f402a5a0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 -- 172.21.15.129:0/4116830374 shutdown_connections 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 --2- 172.21.15.129:0/4116830374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0ce910 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 -- 172.21.15.129:0/4116830374 >> 172.21.15.129:0/4116830374 conn(0x7f580c0ccc40 msgr2=0x7f580c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 -- 172.21.15.129:0/4116830374 shutdown_connections 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.630+0000 7f5810e00640 1 -- 172.21.15.129:0/4116830374 wait complete. 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.632+0000 7f5810e00640 1 Processor -- start 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.632+0000 7f5810e00640 1 -- start start 2024-09-17T03:59:35.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f580c0b4220 0x7f580c14b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f580c0caec0 con 0x7f580c0b2650 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f580c0cad40 con 0x7f580c0b4220 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5810e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f580c0cb040 con 0x7f580c0ce500 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5809e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5809e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55816/0 (socket says 172.21.15.129:55816) 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5809e00640 1 -- 172.21.15.129:0/1888350849 learned_addr learned my addr 172.21.15.129:0/1888350849 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f580a800640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5803e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f580c0b4220 0x7f580c14b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5809e00640 1 -- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 msgr2=0x7f580c0c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.634+0000 7f5809e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5809e00640 1 -- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f580c0b4220 msgr2=0x7f580c14b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5809e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f580c0b4220 0x7f580c14b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5809e00640 1 -- 172.21.15.129:0/1888350849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580c09c1b0 con 0x7f580c0b2650 2024-09-17T03:59:35.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5809e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c400 tx=0x7f57fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc011850 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580c09c490 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57fc017070 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57fc011da0 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.635+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580c09ca50 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.636+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f580c09bf30 con 0x7f580c0b2650 2024-09-17T03:59:35.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.638+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f57fc01c460 con 0x7f580c0b2650 2024-09-17T03:59:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.639+0000 7f5802000640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 0x7f57e807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.639+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f57fc09a7b0 con 0x7f580c0b2650 2024-09-17T03:59:35.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.640+0000 7f5803e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 0x7f57e807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:35.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.640+0000 7f5803e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 0x7f57e807b0d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f57f4034000 tx=0x7f57f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:35.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.644+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f580c09bf30 con 0x7f580c0b2650 2024-09-17T03:59:35.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.825+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f580c0ce910 con 0x7f580c0b2650 2024-09-17T03:59:35.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.830+0000 7f5802000640 1 -- 172.21.15.129:0/1888350849 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v54)=0 v54) ==== 122+0+0 (secure 0 0 0) 0x7f57fc062780 con 0x7f580c0b2650 2024-09-17T03:59:35.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.832+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 msgr2=0x7f57e807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.832+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 0x7f57e807b0d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f57f4034000 tx=0x7f57f4004730 comp rx=0 tx=0).stop 2024-09-17T03:59:35.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.832+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 msgr2=0x7f580c14aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.832+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f57fc00c400 tx=0x7f57fc00c8c0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 shutdown_connections 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e8078bc0 0x7f57e807b0d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f580c0ce500 0x7f580c0c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f580c0b4220 0x7f580c14b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 --2- 172.21.15.129:0/1888350849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f580c0b2650 0x7f580c14aa30 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 >> 172.21.15.129:0/1888350849 conn(0x7f580c0ccc40 msgr2=0x7f580c0ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 shutdown_connections 2024-09-17T03:59:35.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:35.833+0000 7f5810e00640 1 -- 172.21.15.129:0/1888350849 wait complete. 2024-09-17T03:59:35.856 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:59:35.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:35.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:36.037 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:59:36.040 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:36.040 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:36.076 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:59:36.076 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:36.076 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:36.113 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:36.113 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:36.113 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:36.151 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:59:36.151 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:36.151 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:36.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:36.190 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T03:59:36.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:36.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 -- 172.21.15.129:0/1762933247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d780705a0 msgr2=0x7f5d780b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/1762933247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d780705a0 0x7f5d780b4140 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f5d78072e00 tx=0x7f5d6802a5f0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 -- 172.21.15.129:0/1762933247 shutdown_connections 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/1762933247 >> v2:172.21.15.181:3300/0 conn(0x7f5d780cd390 0x7f5d781609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/1762933247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d780705a0 0x7f5d780b4140 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 -- 172.21.15.129:0/1762933247 >> 172.21.15.129:0/1762933247 conn(0x7f5d7806d410 msgr2=0x7f5d780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 -- 172.21.15.129:0/1762933247 shutdown_connections 2024-09-17T03:59:36.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.411+0000 7f5d7fa00640 1 -- 172.21.15.129:0/1762933247 wait complete. 2024-09-17T03:59:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.413+0000 7f5d7fa00640 1 Processor -- start 2024-09-17T03:59:36.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.414+0000 7f5d7fa00640 1 -- start start 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d7806eed0 0x7f5d780c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d780705a0 0x7f5d781f7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5d78160250 con 0x7f5d781f7940 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7806f4f0 con 0x7f5d780705a0 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5d78160f00 con 0x7f5d7806eed0 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55830/0 (socket says 172.21.15.129:55830) 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7da00640 1 -- 172.21.15.129:0/708120893 learned_addr learned my addr 172.21.15.129:0/708120893 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d6fe00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d780705a0 0x7f5d781f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.415+0000 7f5d7d000640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d7806eed0 0x7f5d780c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 -- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d7806eed0 msgr2=0x7f5d780c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d7806eed0 0x7f5d780c9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 -- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d780705a0 msgr2=0x7f5d781f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d780705a0 0x7f5d781f7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 -- 172.21.15.129:0/708120893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d780ca110 con 0x7f5d781f7940 2024-09-17T03:59:36.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7da00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000ea10 tx=0x7f5d7000eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7000cde0 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5d70004c10 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d781569a0 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70005020 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.416+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d78156f60 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.417+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d780b4140 con 0x7f5d781f7940 2024-09-17T03:59:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.419+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5d700051c0 con 0x7f5d781f7940 2024-09-17T03:59:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.420+0000 7f5d75600640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 0x7f5d5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.420+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f5d7009b740 con 0x7f5d781f7940 2024-09-17T03:59:36.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.420+0000 7f5d7d000640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 0x7f5d5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.421+0000 7f5d7d000640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 0x7f5d5407b0a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5d68033000 tx=0x7f5d68004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:36.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.424+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5d70063820 con 0x7f5d781f7940 2024-09-17T03:59:36.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.644+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d7806a140 con 0x7f5d781f7940 2024-09-17T03:59:36.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.645+0000 7f5d75600640 1 -- 172.21.15.129:0/708120893 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+2270 (secure 0 0 0) 0x7f5d70068250 con 0x7f5d781f7940 2024-09-17T03:59:36.645 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:36.645 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":127,"btime":"2024-09-17T03:59:29:906499+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8580,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2756681375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2756681375},{"type":"v1","addr":"172.21.15.129:6834","nonce":2756681375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":125},{"gid":8652,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3865619686","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3865619686},{"type":"v1","addr":"172.21.15.129:6835","nonce":3865619686}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[]} 2024-09-17T03:59:36.645 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 127 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 msgr2=0x7f5d5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 0x7f5d5407b0a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5d68033000 tx=0x7f5d68004730 comp rx=0 tx=0).stop 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 msgr2=0x7f5d7809d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000ea10 tx=0x7f5d7000eed0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 shutdown_connections 2024-09-17T03:59:36.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5d54078b90 0x7f5d5407b0a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5d781f7940 0x7f5d7809d3a0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5d780705a0 0x7f5d781f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 --2- 172.21.15.129:0/708120893 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5d7806eed0 0x7f5d780c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 >> 172.21.15.129:0/708120893 conn(0x7f5d7806d410 msgr2=0x7f5d7806fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 shutdown_connections 2024-09-17T03:59:36.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.647+0000 7f5d7fa00640 1 -- 172.21.15.129:0/708120893 wait complete. 2024-09-17T03:59:36.672 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T03:59:36.672 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T03:59:36.672 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T03:59:36.672 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:59:36.673 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T03:59:36.673 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T03:59:36.673 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T03:59:36.673 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T03:59:36.673 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T03:59:36.673 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T03:59:36.674 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T03:59:36.674 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T03:59:36.678 INFO:tasks.ceph.mds.b:Started 2024-09-17T03:59:36.678 DEBUG:teuthology.parallel:result is None 2024-09-17T03:59:36.680 INFO:tasks.ceph.mds.d:Started 2024-09-17T03:59:36.680 DEBUG:teuthology.parallel:result is None 2024-09-17T03:59:36.695 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T03:59:36.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T03:59:36.699 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T03:59:36.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T03:59:36.743 INFO:tasks.ceph.mds.c:Started 2024-09-17T03:59:36.743 DEBUG:teuthology.parallel:result is None 2024-09-17T03:59:36.746 INFO:tasks.ceph.mds.a:Started 2024-09-17T03:59:36.746 DEBUG:teuthology.parallel:result is None 2024-09-17T03:59:36.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T03:59:36.873 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T03:59:36.873 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T03:59:36.945 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T03:59:36.945 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T03:59:36.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 -- 172.21.15.129:0/4225577639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f5815a220 msgr2=0x7f7f5815d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 --2- 172.21.15.129:0/4225577639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f5815a220 0x7f7f5815d660 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009830 tx=0x7f7f4802a4b0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 -- 172.21.15.129:0/4225577639 shutdown_connections 2024-09-17T03:59:36.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 --2- 172.21.15.129:0/4225577639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f5815a220 0x7f7f5815d660 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 -- 172.21.15.129:0/4225577639 >> 172.21.15.129:0/4225577639 conn(0x7f7f58069150 msgr2=0x7f7f58069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:36.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 -- 172.21.15.129:0/4225577639 shutdown_connections 2024-09-17T03:59:36.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.982+0000 7f7f57e00640 1 -- 172.21.15.129:0/4225577639 wait complete. 2024-09-17T03:59:36.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.984+0000 7f7f57e00640 1 Processor -- start 2024-09-17T03:59:36.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.984+0000 7f7f57e00640 1 -- start start 2024-09-17T03:59:36.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f581602b0 0x7f7f580b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 0x7f7f581583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7f580d0660 con 0x7f7f5815a220 2024-09-17T03:59:36.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581608d0 con 0x7f7f580b7c90 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7f580d07e0 con 0x7f7f581602b0 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f56a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f56a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55872/0 (socket says 172.21.15.129:55872) 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f56a00640 1 -- 172.21.15.129:0/664558013 learned_addr learned my addr 172.21.15.129:0/664558013 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f4e000640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f581602b0 0x7f7f580b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.986+0000 7f7f57400640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 0x7f7f581583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 -- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f581602b0 msgr2=0x7f7f580b7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f581602b0 0x7f7f580b7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 -- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 msgr2=0x7f7f581583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 0x7f7f581583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 -- 172.21.15.129:0/664558013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f580b4cf0 con 0x7f7f5815a220 2024-09-17T03:59:36.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f57400640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 0x7f7f581583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f56a00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009950 tx=0x7f7f48003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48005c90 con 0x7f7f5815a220 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f580b5240 con 0x7f7f5815a220 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7f48005020 con 0x7f7f5815a220 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48005450 con 0x7f7f5815a220 2024-09-17T03:59:36.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.987+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581f6410 con 0x7f7f5815a220 2024-09-17T03:59:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.988+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5815d660 con 0x7f7f5815a220 2024-09-17T03:59:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.990+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7f480317c0 con 0x7f7f5815a220 2024-09-17T03:59:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.990+0000 7f7f4f400640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 0x7f7f3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:36.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.991+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 4943+0+0 (secure 0 0 0) 0x7f7f480bbb30 con 0x7f7f5815a220 2024-09-17T03:59:36.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.991+0000 7f7f4e000640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 0x7f7f3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:36.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.992+0000 7f7f4e000640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 0x7f7f3407b0a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7f40001220 tx=0x7f7f40009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:36.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:36.995+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7f4803f030 con 0x7f7f5815a220 2024-09-17T03:59:37.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.190+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7f5815c090 con 0x7f7f5815a220 2024-09-17T03:59:37.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.960+0000 7f7f4f400640 1 -- 172.21.15.129:0/664558013 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v148) ==== 97+0+0 (secure 0 0 0) 0x7f7f48084260 con 0x7f7f5815a220 2024-09-17T03:59:37.961 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 msgr2=0x7f7f3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 0x7f7f3407b0a0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7f40001220 tx=0x7f7f40009040 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 msgr2=0x7f7f580b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009950 tx=0x7f7f48003c10 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 shutdown_connections 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7f34078b90 0x7f7f3407b0a0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7f580b7c90 0x7f7f581583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7f581602b0 0x7f7f580b7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 --2- 172.21.15.129:0/664558013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7f5815a220 0x7f7f580b47b0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.964+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 >> 172.21.15.129:0/664558013 conn(0x7f7f58069150 msgr2=0x7f7f580cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:37.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.965+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 shutdown_connections 2024-09-17T03:59:37.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:37.965+0000 7f7f57e00640 1 -- 172.21.15.129:0/664558013 wait complete. 2024-09-17T03:59:37.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T03:59:38.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 -- 172.21.15.129:0/3732827635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0ce170 msgr2=0x7f70fc0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 --2- 172.21.15.129:0/3732827635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0ce170 0x7f70fc0cbbf0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f70fc072360 tx=0x7f70f002a560 comp rx=0 tx=0).stop 2024-09-17T03:59:38.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 -- 172.21.15.129:0/3732827635 shutdown_connections 2024-09-17T03:59:38.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 --2- 172.21.15.129:0/3732827635 >> v2:172.21.15.181:3300/0 conn(0x7f70fc0b2950 0x7f70fc0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 --2- 172.21.15.129:0/3732827635 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0ce170 0x7f70fc0cbbf0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.209+0000 7f7103800640 1 -- 172.21.15.129:0/3732827635 >> 172.21.15.129:0/3732827635 conn(0x7f70fc06d410 msgr2=0x7f70fc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:38.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.210+0000 7f7103800640 1 -- 172.21.15.129:0/3732827635 shutdown_connections 2024-09-17T03:59:38.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.210+0000 7f7103800640 1 -- 172.21.15.129:0/3732827635 wait complete. 2024-09-17T03:59:38.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.212+0000 7f7103800640 1 Processor -- start 2024-09-17T03:59:38.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.212+0000 7f7103800640 1 -- start start 2024-09-17T03:59:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 0x7f70fc1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70fc0ce170 0x7f70fc1f9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc070890 con 0x7f70fc0b2950 2024-09-17T03:59:38.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc070710 con 0x7f70fc0b39c0 2024-09-17T03:59:38.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.213+0000 7f7103800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70fc06f5e0 con 0x7f70fc0ce170 2024-09-17T03:59:38.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55902/0 (socket says 172.21.15.129:55902) 2024-09-17T03:59:38.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 -- 172.21.15.129:0/1236699167 learned_addr learned my addr 172.21.15.129:0/1236699167 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7101800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70fc0ce170 0x7f70fc1f9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f70fbe00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 0x7f70fc1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 -- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70fc0ce170 msgr2=0x7f70fc1f9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70fc0ce170 0x7f70fc1f9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 -- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 msgr2=0x7f70fc1f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 0x7f70fc1f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f7100e00640 1 -- 172.21.15.129:0/1236699167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc15cf20 con 0x7f70fc0b2950 2024-09-17T03:59:38.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.214+0000 7f70fbe00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 0x7f70fc1f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:38.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f7100e00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f70f00099c0 tx=0x7f70f00039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:38.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f003e030 con 0x7f70fc0b2950 2024-09-17T03:59:38.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70f00047a0 con 0x7f70fc0b2950 2024-09-17T03:59:38.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f0035040 con 0x7f70fc0b2950 2024-09-17T03:59:38.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70fc15d1c0 con 0x7f70fc0b2950 2024-09-17T03:59:38.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.215+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70fc15d780 con 0x7f70fc0b2950 2024-09-17T03:59:38.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.216+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70fc0c3b80 con 0x7f70fc0b2950 2024-09-17T03:59:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.217+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70f0041030 con 0x7f70fc0b2950 2024-09-17T03:59:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.218+0000 7f70f8c00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 0x7f70dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.218+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 4857+0+0 (secure 0 0 0) 0x7f70f00bb2b0 con 0x7f70fc0b2950 2024-09-17T03:59:38.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.219+0000 7f70fbe00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 0x7f70dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.219+0000 7f70fbe00640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 0x7f70dc07b0a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0cd890 tx=0x7f70e8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:38.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.223+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f70fc0c3b80 con 0x7f70fc0b2950 2024-09-17T03:59:38.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.471+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f70fc0b3480 con 0x7f70fc0b2950 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.472+0000 7f70f8c00640 1 -- 172.21.15.129:0/1236699167 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v36) ==== 80+0+5304 (secure 0 0 0) 0x7f70fc0b3dd0 con 0x7f70fc0b2950 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T03:59:38.472 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T03:59:38.475 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.482 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:59:38.483 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T03:59:38.485 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T03:59:38.486 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T03:59:38.487 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 msgr2=0x7f70dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 0x7f70dc07b0a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0cd890 tx=0x7f70e8008040 comp rx=0 tx=0).stop 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 msgr2=0x7f70fc0b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f70f00099c0 tx=0x7f70f00039e0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 shutdown_connections 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70dc078b90 0x7f70dc07b0a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70fc0ce170 0x7f70fc1f9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70fc0b39c0 0x7f70fc1f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 --2- 172.21.15.129:0/1236699167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70fc0b2950 0x7f70fc0b1aa0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 >> 172.21.15.129:0/1236699167 conn(0x7f70fc06d410 msgr2=0x7f70fc0cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:38.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 shutdown_connections 2024-09-17T03:59:38.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.474+0000 7f7103800640 1 -- 172.21.15.129:0/1236699167 wait complete. 2024-09-17T03:59:38.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T03:59:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 -- 172.21.15.129:0/3061953639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 msgr2=0x7fc3d40b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 --2- 172.21.15.129:0/3061953639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40b4af0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40cc9e0 tx=0x7fc3bc02a4d0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 -- 172.21.15.129:0/3061953639 shutdown_connections 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 --2- 172.21.15.129:0/3061953639 >> v2:172.21.15.181:3300/0 conn(0x7fc3d40d02c0 0x7fc3d40d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 --2- 172.21.15.129:0/3061953639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40b4af0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 --2- 172.21.15.129:0/3061953639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 0x7fc3d4160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 -- 172.21.15.129:0/3061953639 >> 172.21.15.129:0/3061953639 conn(0x7fc3d406cfc0 msgr2=0x7fc3d406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 -- 172.21.15.129:0/3061953639 shutdown_connections 2024-09-17T03:59:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.719+0000 7fc3da200640 1 -- 172.21.15.129:0/3061953639 wait complete. 2024-09-17T03:59:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.721+0000 7fc3da200640 1 Processor -- start 2024-09-17T03:59:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.721+0000 7fc3da200640 1 -- start start 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 0x7fc3d413b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3d40d02c0 0x7fc3d4074250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d415d040 con 0x7fc3d40b46e0 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d415cec0 con 0x7fc3d40d02c0 2024-09-17T03:59:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3da200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d415d1c0 con 0x7fc3d40cfb60 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3d8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3d8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55924/0 (socket says 172.21.15.129:55924) 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3d8e00640 1 -- 172.21.15.129:0/1403621518 learned_addr learned my addr 172.21.15.129:0/1403621518 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3cbe00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 0x7fc3d413b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.723+0000 7fc3d9800640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3d40d02c0 0x7fc3d4074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 -- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 msgr2=0x7fc3d413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 0x7fc3d413b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 -- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3d40d02c0 msgr2=0x7fc3d4074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3d40d02c0 0x7fc3d4074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 -- 172.21.15.129:0/1403621518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d413bdb0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d8e00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400e6f0 tx=0x7fc3c400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4004230 con 0x7fc3d40b46e0 2024-09-17T03:59:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc3c4004410 con 0x7fc3d40b46e0 2024-09-17T03:59:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d4074790 con 0x7fc3d40b46e0 2024-09-17T03:59:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c400cb10 con 0x7fc3d40b46e0 2024-09-17T03:59:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.724+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d41e8630 con 0x7fc3d40b46e0 2024-09-17T03:59:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.725+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3d415d6e0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.727+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc3c400ccb0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.728+0000 7fc3d1600640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 0x7fc3b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.728+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 4857+0+0 (secure 0 0 0) 0x7fc3c409b3e0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.729+0000 7fc3cbe00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 0x7fc3b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.729+0000 7fc3cbe00640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 0x7fc3b007b0a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc033000 tx=0x7fc3bc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.733+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc3c4063ad0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.917+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc3d406a140 con 0x7fc3d40b46e0 2024-09-17T03:59:38.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.918+0000 7fc3d1600640 1 -- 172.21.15.129:0/1403621518 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v148) ==== 84+0+65 (secure 0 0 0) 0x7fc3c40633d0 con 0x7fc3d40b46e0 2024-09-17T03:59:38.918 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:38.918 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T03:59:38.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:38.919 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T03:59:38.919 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T03:59:38.919 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:38.919 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 msgr2=0x7fc3b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 0x7fc3b007b0a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc033000 tx=0x7fc3bc004730 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 msgr2=0x7fc3d40c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fc3c400e6f0 tx=0x7fc3c400ebb0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 shutdown_connections 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3b0078b90 0x7fc3b007b0a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3d40d02c0 0x7fc3d4074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc3d40cfb60 0x7fc3d413b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 --2- 172.21.15.129:0/1403621518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc3d40b46e0 0x7fc3d40c93b0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.920+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 >> 172.21.15.129:0/1403621518 conn(0x7fc3d406cfc0 msgr2=0x7fc3d415a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.921+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 shutdown_connections 2024-09-17T03:59:38.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:38.921+0000 7fc3da200640 1 -- 172.21.15.129:0/1403621518 wait complete. 2024-09-17T03:59:38.947 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T03:59:38.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T03:59:39.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 -- 172.21.15.129:0/378714226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b3f70 msgr2=0x7f30dc0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 --2- 172.21.15.129:0/378714226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b3f70 0x7f30dc0b49a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f30dc15ce40 tx=0x7f30cc02a4d0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 -- 172.21.15.129:0/378714226 shutdown_connections 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 --2- 172.21.15.129:0/378714226 >> v2:172.21.15.181:3300/0 conn(0x7f30dc0b2360 0x7f30dc06eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 --2- 172.21.15.129:0/378714226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 0x7f30dc0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 --2- 172.21.15.129:0/378714226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b3f70 0x7f30dc0b49a0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 -- 172.21.15.129:0/378714226 >> 172.21.15.129:0/378714226 conn(0x7f30dc0ccc40 msgr2=0x7f30dc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 -- 172.21.15.129:0/378714226 shutdown_connections 2024-09-17T03:59:39.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.173+0000 7f30e3c00640 1 -- 172.21.15.129:0/378714226 wait complete. 2024-09-17T03:59:39.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.176+0000 7f30e3c00640 1 Processor -- start 2024-09-17T03:59:39.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.176+0000 7f30e3c00640 1 -- start start 2024-09-17T03:59:39.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 0x7f30dc1e2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:39.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30dc1f5560 0x7f30dc1e1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc06fad0 con 0x7f30dc0b2360 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc06f950 con 0x7f30dc1f5560 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30e3c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f30dc06fc50 con 0x7f30dc074580 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30d3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30d3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55948/0 (socket says 172.21.15.129:55948) 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.177+0000 7f30d3e00640 1 -- 172.21.15.129:0/2005112885 learned_addr learned my addr 172.21.15.129:0/2005112885 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30e1200640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 0x7f30dc1e2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:39.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30e1c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30dc1f5560 0x7f30dc1e1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 -- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 msgr2=0x7f30dc1e2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 0x7f30dc1e2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 -- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30dc1f5560 msgr2=0x7f30dc1e1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30dc1f5560 0x7f30dc1e1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 -- 172.21.15.129:0/2005112885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc1f6060 con 0x7f30dc0b2360 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d3e00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f30c400d650 tx=0x7f30c400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.178+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4017070 con 0x7f30dc0b2360 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.179+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f30c4004310 con 0x7f30dc0b2360 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.179+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4005280 con 0x7f30dc0b2360 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.179+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc1f6340 con 0x7f30dc0b2360 2024-09-17T03:59:39.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.179+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc09c4e0 con 0x7f30dc0b2360 2024-09-17T03:59:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.180+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30dc0b3f70 con 0x7f30dc0b2360 2024-09-17T03:59:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.181+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f30c4005420 con 0x7f30dc0b2360 2024-09-17T03:59:39.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.182+0000 7f30d9600640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 0x7f30b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:39.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.182+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(148..148 src has 1..148) ==== 4857+0+0 (secure 0 0 0) 0x7f30c409ad30 con 0x7f30dc0b2360 2024-09-17T03:59:39.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.182+0000 7f30e1200640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 0x7f30b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:39.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.183+0000 7f30e1200640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 0x7f30b807b0a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f30cc033000 tx=0x7f30cc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:39.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.187+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f30c4020020 con 0x7f30dc0b2360 2024-09-17T03:59:39.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.370+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f30dc0733a0 con 0x7f30dc0b2360 2024-09-17T03:59:39.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.970+0000 7f30d9600640 1 -- 172.21.15.129:0/2005112885 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v149) ==== 138+0+0 (secure 0 0 0) 0x7f30c4062dd0 con 0x7f30dc0b2360 2024-09-17T03:59:39.971 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T03:59:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 msgr2=0x7f30b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 0x7f30b807b0a0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f30cc033000 tx=0x7f30cc004730 comp rx=0 tx=0).stop 2024-09-17T03:59:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 msgr2=0x7f30dc09ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f30c400d650 tx=0x7f30c400db10 comp rx=0 tx=0).stop 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 shutdown_connections 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f30b8078b90 0x7f30b807b0a0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30dc1f5560 0x7f30dc1e1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30dc0b2360 0x7f30dc09ca00 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 --2- 172.21.15.129:0/2005112885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30dc074580 0x7f30dc1e2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.975+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 >> 172.21.15.129:0/2005112885 conn(0x7f30dc0ccc40 msgr2=0x7f30dc073ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.976+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 shutdown_connections 2024-09-17T03:59:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:39.976+0000 7f30e3c00640 1 -- 172.21.15.129:0/2005112885 wait complete. 2024-09-17T03:59:40.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 -- 172.21.15.129:0/3594170713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a80cf2d0 msgr2=0x7ff8a8167530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 --2- 172.21.15.129:0/3594170713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8167530 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0005fd0 tx=0x7ff8a002aa10 comp rx=0 tx=0).stop 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 -- 172.21.15.129:0/3594170713 shutdown_connections 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 --2- 172.21.15.129:0/3594170713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8167530 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 -- 172.21.15.129:0/3594170713 >> 172.21.15.129:0/3594170713 conn(0x7ff8a80d0120 msgr2=0x7ff8a80d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 -- 172.21.15.129:0/3594170713 shutdown_connections 2024-09-17T03:59:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.234+0000 7ff8a7200640 1 -- 172.21.15.129:0/3594170713 wait complete. 2024-09-17T03:59:40.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.237+0000 7ff8a7200640 1 Processor -- start 2024-09-17T03:59:40.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.237+0000 7ff8a7200640 1 -- start start 2024-09-17T03:59:40.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff8a806ee70 0x7ff8a8154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:40.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a8167e00 con 0x7ff8a8156f20 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a8167c80 con 0x7ff8a80cf2d0 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a8167f80 con 0x7ff8a806ee70 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.238+0000 7ff8a6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55962/0 (socket says 172.21.15.129:55962) 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 -- 172.21.15.129:0/710988985 learned_addr learned my addr 172.21.15.129:0/710988985 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff89d400640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:40.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a5e00640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff8a806ee70 0x7ff8a8154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 -- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff8a806ee70 msgr2=0x7ff8a8154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff8a806ee70 0x7ff8a8154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 -- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8a80cf2d0 msgr2=0x7ff8a8072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8072b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 -- 172.21.15.129:0/710988985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a8155070 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.239+0000 7ff8a6800640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006010 tx=0x7ff8a0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.240+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0046070 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.240+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff8a00045c0 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.240+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0033070 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.240+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a8166c10 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.240+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a80b55e0 con 0x7ff8a8156f20 2024-09-17T03:59:40.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.241+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a80af6a0 con 0x7ff8a8156f20 2024-09-17T03:59:40.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.243+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff8a0049070 con 0x7ff8a8156f20 2024-09-17T03:59:40.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.244+0000 7ff89ea00640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 0x7ff88407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:40.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.244+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 5227+0+0 (secure 0 0 0) 0x7ff8a00bbea0 con 0x7ff8a8156f20 2024-09-17T03:59:40.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.244+0000 7ff8a5e00640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 0x7ff88407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:40.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.245+0000 7ff8a5e00640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 0x7ff88407b0a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80afb70 tx=0x7ff898009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:40.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.247+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff8a003e070 con 0x7ff8a8156f20 2024-09-17T03:59:40.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.431+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff8a80afe60 con 0x7ff8a8156f20 2024-09-17T03:59:40.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.977+0000 7ff89ea00640 1 -- 172.21.15.129:0/710988985 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v150) ==== 170+0+0 (secure 0 0 0) 0x7ff8a0083f70 con 0x7ff8a8156f20 2024-09-17T03:59:40.978 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T03:59:40.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.980+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 msgr2=0x7ff88407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:40.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.980+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 0x7ff88407b0a0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80afb70 tx=0x7ff898009040 comp rx=0 tx=0).stop 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.980+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 msgr2=0x7ff8a80730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.980+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0006010 tx=0x7ff8a0003f10 comp rx=0 tx=0).stop 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 shutdown_connections 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff884078b90 0x7ff88407b0a0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff8a8156f20 0x7ff8a80730d0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8a80cf2d0 0x7ff8a8072b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 --2- 172.21.15.129:0/710988985 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff8a806ee70 0x7ff8a8154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:40.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 >> 172.21.15.129:0/710988985 conn(0x7ff8a80d0120 msgr2=0x7ff8a806f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:40.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.981+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 shutdown_connections 2024-09-17T03:59:40.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:40.982+0000 7ff8a7200640 1 -- 172.21.15.129:0/710988985 wait complete. 2024-09-17T03:59:41.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 -- 172.21.15.129:0/521625650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a40051c0 msgr2=0x7f20a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 --2- 172.21.15.129:0/521625650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a40051c0 0x7f20a400b9c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f20c00cc370 tx=0x7f20a4005830 comp rx=0 tx=0).stop 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 -- 172.21.15.129:0/521625650 shutdown_connections 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 --2- 172.21.15.129:0/521625650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20a40051c0 0x7f20a400b9c0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 -- 172.21.15.129:0/521625650 >> 172.21.15.129:0/521625650 conn(0x7f20c00ccc40 msgr2=0x7f20c00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 -- 172.21.15.129:0/521625650 shutdown_connections 2024-09-17T03:59:41.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.247+0000 7f20c7200640 1 -- 172.21.15.129:0/521625650 wait complete. 2024-09-17T03:59:41.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.249+0000 7f20c7200640 1 Processor -- start 2024-09-17T03:59:41.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.249+0000 7f20c7200640 1 -- start start 2024-09-17T03:59:41.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a40051c0 0x7f20c009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:41.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c01565f0 0x7f20c0154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20c00b23e0 con 0x7f20c01561e0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20c00b2260 con 0x7f20c01565f0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f20c00b2560 con 0x7f20a40051c0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55976/0 (socket says 172.21.15.129:55976) 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 -- 172.21.15.129:0/3768767504 learned_addr learned my addr 172.21.15.129:0/3768767504 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c5200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c01565f0 0x7f20c0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a40051c0 0x7f20c009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 -- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a40051c0 msgr2=0x7f20c009cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a40051c0 0x7f20c009cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:41.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 -- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c01565f0 msgr2=0x7f20c0154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c01565f0 0x7f20c0154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 -- 172.21.15.129:0/3768767504 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c0156dc0 con 0x7f20c01561e0 2024-09-17T03:59:41.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20b7400640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f20a8004cc0 tx=0x7f20a8001730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8015020 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20a8011980 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c01570a0 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8011dd0 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.252+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c015e180 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.253+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c0156b30 con 0x7f20c01561e0 2024-09-17T03:59:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.255+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f20a8018020 con 0x7f20c01561e0 2024-09-17T03:59:41.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.256+0000 7f20bcc00640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 0x7f209c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:41.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.256+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5609+0+0 (secure 0 0 0) 0x7f20a809b530 con 0x7f20c01561e0 2024-09-17T03:59:41.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.257+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 0x7f209c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:41.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.257+0000 7f20bfe00640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 0x7f209c07b0d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009ad0 tx=0x7f20b0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:41.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.261+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f20a801f020 con 0x7f20c01561e0 2024-09-17T03:59:41.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.480+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f20c00af800 con 0x7f20c01561e0 2024-09-17T03:59:41.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:41.997+0000 7f20bcc00640 1 -- 172.21.15.129:0/3768767504 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v131) ==== 174+0+0 (secure 0 0 0) 0x7f20a8063400 con 0x7f20c01561e0 2024-09-17T03:59:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 22 and data pool 23 2024-09-17T03:59:42.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.004+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 msgr2=0x7f209c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.004+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 0x7f209c07b0d0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009ad0 tx=0x7f20b0005f90 comp rx=0 tx=0).stop 2024-09-17T03:59:42.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.004+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 msgr2=0x7f20c01540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.004+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f20a8004cc0 tx=0x7f20a8001730 comp rx=0 tx=0).stop 2024-09-17T03:59:42.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 shutdown_connections 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f209c078bc0 0x7f209c07b0d0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20c01565f0 0x7f20c0154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20c01561e0 0x7f20c01540d0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 --2- 172.21.15.129:0/3768767504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20a40051c0 0x7f20c009cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 >> 172.21.15.129:0/3768767504 conn(0x7f20c00ccc40 msgr2=0x7f20c015c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 shutdown_connections 2024-09-17T03:59:42.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.009+0000 7f20c7200640 1 -- 172.21.15.129:0/3768767504 wait complete. 2024-09-17T03:59:42.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 -- 172.21.15.129:0/4201410765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 msgr2=0x7f2be00cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/4201410765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be00cf5c0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0005fd0 tx=0x7f2bd002a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 -- 172.21.15.129:0/4201410765 shutdown_connections 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/4201410765 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 0x7f2be015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/4201410765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be00cf5c0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 -- 172.21.15.129:0/4201410765 >> 172.21.15.129:0/4201410765 conn(0x7f2be006cfc0 msgr2=0x7f2be006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 -- 172.21.15.129:0/4201410765 shutdown_connections 2024-09-17T03:59:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.262+0000 7f2bdfe00640 1 -- 172.21.15.129:0/4201410765 wait complete. 2024-09-17T03:59:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.265+0000 7f2bdfe00640 1 Processor -- start 2024-09-17T03:59:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.265+0000 7f2bdfe00640 1 -- start start 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 0x7f2be00c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be015c1e0 0x7f2be0142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2be00d02c0 con 0x7f2be01524c0 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2be0152b10 con 0x7f2be015c1e0 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bdfe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2be00d0440 con 0x7f2be00cfb00 2024-09-17T03:59:42.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bd6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bd6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56000/0 (socket says 172.21.15.129:56000) 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.266+0000 7f2bd6000640 1 -- 172.21.15.129:0/489355911 learned_addr learned my addr 172.21.15.129:0/489355911 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bdf400640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be015c1e0 0x7f2be0142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bdea00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 0x7f2be00c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 -- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 msgr2=0x7f2be00c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 0x7f2be00c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 -- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be015c1e0 msgr2=0x7f2be0142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be015c1e0 0x7f2be0142ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 -- 172.21.15.129:0/489355911 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be0150d70 con 0x7f2be01524c0 2024-09-17T03:59:42.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd6000640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800e790 tx=0x7f2bc800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.267+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc800eea0 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.268+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bc800cdb0 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.268+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be0157450 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.268+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8010690 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.268+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be0157a10 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.269+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be00d08f0 con 0x7f2be01524c0 2024-09-17T03:59:42.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.270+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2bc8010830 con 0x7f2be01524c0 2024-09-17T03:59:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.271+0000 7f2bd7400640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 0x7f2bbc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.271+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5709+0+0 (secure 0 0 0) 0x7f2bc809b5d0 con 0x7f2be01524c0 2024-09-17T03:59:42.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.271+0000 7f2bdea00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 0x7f2bbc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.272+0000 7f2bdea00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 0x7f2bbc07b0d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0033000 tx=0x7f2bd0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:42.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.276+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2bc8063440 con 0x7f2be01524c0 2024-09-17T03:59:42.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.459+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2be006a140 con 0x7f2be01524c0 2024-09-17T03:59:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.460+0000 7f2bd7400640 1 -- 172.21.15.129:0/489355911 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16792 (secure 0 0 0) 0x7f2bc8067e70 con 0x7f2be01524c0 2024-09-17T03:59:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:42.461 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":151,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:41.482225+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:59:39.383572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-17T03:59:40.444542+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 msgr2=0x7f2bbc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 0x7f2bbc07b0d0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0033000 tx=0x7f2bd0004730 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 msgr2=0x7f2be0142980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800e790 tx=0x7f2bc800ec50 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 shutdown_connections 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bbc078bc0 0x7f2bbc07b0d0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be015c1e0 0x7f2be0142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be01524c0 0x7f2be0142980 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/489355911 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be00cfb00 0x7f2be00c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.462+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 >> 172.21.15.129:0/489355911 conn(0x7f2be006cfc0 msgr2=0x7f2be006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:42.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.463+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 shutdown_connections 2024-09-17T03:59:42.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.463+0000 7f2bdfe00640 1 -- 172.21.15.129:0/489355911 wait complete. 2024-09-17T03:59:42.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 -- 172.21.15.129:0/3816848229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18000b3b90 msgr2=0x7f18000cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 --2- 172.21.15.129:0/3816848229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18000b3b90 0x7f18000cbbf0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009860 tx=0x7f17e802a5a0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 -- 172.21.15.129:0/3816848229 shutdown_connections 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 --2- 172.21.15.129:0/3816848229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18000b3b90 0x7f18000cbbf0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 -- 172.21.15.129:0/3816848229 >> 172.21.15.129:0/3816848229 conn(0x7f180006d410 msgr2=0x7f18000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:42.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 -- 172.21.15.129:0/3816848229 shutdown_connections 2024-09-17T03:59:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.710+0000 7f1806e00640 1 -- 172.21.15.129:0/3816848229 wait complete. 2024-09-17T03:59:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.712+0000 7f1806e00640 1 Processor -- start 2024-09-17T03:59:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.712+0000 7f1806e00640 1 -- start start 2024-09-17T03:59:42.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18000b2950 0x7f1800159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1800156840 0x7f18001f2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f180006f760 con 0x7f18000ce170 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f180006f5e0 con 0x7f18000b2950 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1806e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f180006f8e0 con 0x7f1800156840 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56016/0 (socket says 172.21.15.129:56016) 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 -- 172.21.15.129:0/1714216776 learned_addr learned my addr 172.21.15.129:0/1714216776 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f1804e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1800156840 0x7f18001f2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17ffe00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18000b2950 0x7f1800159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 -- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1800156840 msgr2=0x7f18001f2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1800156840 0x7f18001f2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 -- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18000b2950 msgr2=0x7f1800159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18000b2950 0x7f1800159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.714+0000 7f17f7400640 1 -- 172.21.15.129:0/1714216776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180015e870 con 0x7f18000ce170 2024-09-17T03:59:42.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f17f7400640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f17e802ab90 tx=0x7f17e8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005c90 con 0x7f18000ce170 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17e8031e50 con 0x7f18000ce170 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180015dab0 con 0x7f18000ce170 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005260 con 0x7f18000ce170 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.715+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180015e070 con 0x7f18000ce170 2024-09-17T03:59:42.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.716+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1800156d70 con 0x7f18000ce170 2024-09-17T03:59:42.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.717+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f17e8005400 con 0x7f18000ce170 2024-09-17T03:59:42.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.718+0000 7f17fcc00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 0x7f17dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:42.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.718+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 5709+0+0 (secure 0 0 0) 0x7f17e80bbca0 con 0x7f18000ce170 2024-09-17T03:59:42.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.719+0000 7f17ffe00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 0x7f17dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:42.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.720+0000 7f17ffe00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 0x7f17dc07b0d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f17f0001120 tx=0x7f17f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:42.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.723+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f17e803f030 con 0x7f18000ce170 2024-09-17T03:59:42.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.906+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18000cbbf0 con 0x7f18000ce170 2024-09-17T03:59:42.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.907+0000 7f17fcc00640 1 -- 172.21.15.129:0/1714216776 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v151) ==== 74+0+16792 (secure 0 0 0) 0x7f17e8083a80 con 0x7f18000ce170 2024-09-17T03:59:42.906 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:42.907 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":151,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:41.482225+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:59:39.383572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-17T03:59:40.444542+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.909+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 msgr2=0x7f17dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.909+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 0x7f17dc07b0d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f17f0001120 tx=0x7f17f0009040 comp rx=0 tx=0).stop 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.909+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 msgr2=0x7f18001f21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.909+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f17e802ab90 tx=0x7f17e8033c20 comp rx=0 tx=0).stop 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 shutdown_connections 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17dc078bc0 0x7f17dc07b0d0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1800156840 0x7f18001f2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18000ce170 0x7f18001f21c0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 --2- 172.21.15.129:0/1714216776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18000b2950 0x7f1800159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 >> 172.21.15.129:0/1714216776 conn(0x7f180006d410 msgr2=0x7f18000c9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 shutdown_connections 2024-09-17T03:59:42.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:42.910+0000 7f1806e00640 1 -- 172.21.15.129:0/1714216776 wait complete. 2024-09-17T03:59:42.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 -- 172.21.15.129:0/1156170954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e6c003a70 msgr2=0x7f9e6c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 --2- 172.21.15.129:0/1156170954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e6c003a70 0x7f9e6c003ee0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009b50 tx=0x7f9e6c005830 comp rx=0 tx=0).stop 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8a800640 1 -- 172.21.15.129:0/1156170954 reap_dead start 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 -- 172.21.15.129:0/1156170954 shutdown_connections 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 -- 172.21.15.129:0/1156170954 >> 172.21.15.129:0/1156170954 conn(0x7f9e840d00b0 msgr2=0x7f9e840d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 -- 172.21.15.129:0/1156170954 shutdown_connections 2024-09-17T03:59:43.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.165+0000 7f9e8b200640 1 -- 172.21.15.129:0/1156170954 wait complete. 2024-09-17T03:59:43.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.168+0000 7f9e8b200640 1 Processor -- start 2024-09-17T03:59:43.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.168+0000 7f9e8b200640 1 -- start start 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e6c003a70 0x7f9e840c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8415cce0 0x7f9e8414f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e840733a0 con 0x7f9e840b3db0 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e84073220 con 0x7f9e8415cce0 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.169+0000 7f9e8b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e84073520 con 0x7f9e6c003a70 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:43.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56034/0 (socket says 172.21.15.129:56034) 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 -- 172.21.15.129:0/975180095 learned_addr learned my addr 172.21.15.129:0/975180095 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e89400640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8415cce0 0x7f9e8414f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e89e00640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e6c003a70 0x7f9e840c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 -- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e6c003a70 msgr2=0x7f9e840c9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e6c003a70 0x7f9e840c9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 -- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8415cce0 msgr2=0x7f9e8414f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8415cce0 0x7f9e8414f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 -- 172.21.15.129:0/975180095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8415d6f0 con 0x7f9e840b3db0 2024-09-17T03:59:43.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.170+0000 7f9e8a800640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f9e841601c0 tx=0x7f9e80006660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.171+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8000fa80 con 0x7f9e840b3db0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.171+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e8000fc40 con 0x7f9e840b3db0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.171+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e80011430 con 0x7f9e840b3db0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.171+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e840c9ea0 con 0x7f9e840b3db0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.171+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e8415d990 con 0x7f9e840b3db0 2024-09-17T03:59:43.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.172+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e840cb080 con 0x7f9e840b3db0 2024-09-17T03:59:43.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.173+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e800115d0 con 0x7f9e840b3db0 2024-09-17T03:59:43.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.175+0000 7f9e7e000640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:43.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.175+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f9e8009b4e0 con 0x7f9e840b3db0 2024-09-17T03:59:43.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.175+0000 7f9e89e00640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:43.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.176+0000 7f9e89e00640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9e78035000 tx=0x7f9e7802d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:43.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.179+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e800639a0 con 0x7f9e840b3db0 2024-09-17T03:59:43.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:43.398+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9e840cf750 con 0x7f9e840b3db0 2024-09-17T03:59:44.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.008+0000 7f9e7e000640 1 -- 172.21.15.129:0/975180095 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v134) ==== 118+0+0 (secure 0 0 0) 0x7f9e800632a0 con 0x7f9e840b3db0 2024-09-17T03:59:44.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 msgr2=0x7f9e6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:44.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f9e78035000 tx=0x7f9e7802d040 comp rx=0 tx=0).stop 2024-09-17T03:59:44.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 msgr2=0x7f9e841f7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:44.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f9e841601c0 tx=0x7f9e80006660 comp rx=0 tx=0).stop 2024-09-17T03:59:44.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 shutdown_connections 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e840b3db0 0x7f9e841f7110 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8415cce0 0x7f9e8414f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 --2- 172.21.15.129:0/975180095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e6c003a70 0x7f9e840c9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 >> 172.21.15.129:0/975180095 conn(0x7f9e840d00b0 msgr2=0x7f9e84160a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 shutdown_connections 2024-09-17T03:59:44.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.011+0000 7f9e8b200640 1 -- 172.21.15.129:0/975180095 wait complete. 2024-09-17T03:59:44.037 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T03:59:44.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T03:59:44.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 -- 172.21.15.129:0/1969365895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4d0003a70 msgr2=0x7fd4d0003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/1969365895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4d0003a70 0x7fd4d0003ee0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00c290 tx=0x7fd4d0005830 comp rx=0 tx=0).stop 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4ebe00640 1 -- 172.21.15.129:0/1969365895 reap_dead start 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 -- 172.21.15.129:0/1969365895 shutdown_connections 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 -- 172.21.15.129:0/1969365895 >> 172.21.15.129:0/1969365895 conn(0x7fd4ec0d00b0 msgr2=0x7fd4ec0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 -- 172.21.15.129:0/1969365895 shutdown_connections 2024-09-17T03:59:44.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.261+0000 7fd4f0e00640 1 -- 172.21.15.129:0/1969365895 wait complete. 2024-09-17T03:59:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.264+0000 7fd4f0e00640 1 Processor -- start 2024-09-17T03:59:44.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.264+0000 7fd4f0e00640 1 -- start start 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 0x7fd4ec151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 0x7fd4ec1f8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec0731a0 con 0x7fd4ec1f9190 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4ec073020 con 0x7fd4d0003a70 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4f0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd4ec070900 con 0x7fd4ec15f5d0 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41256/0 (socket says 172.21.15.129:41256) 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.265+0000 7fd4ebe00640 1 -- 172.21.15.129:0/2887439970 learned_addr learned my addr 172.21.15.129:0/2887439970 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4eaa00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 0x7fd4ec1f8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:44.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 -- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 msgr2=0x7fd4ec1f8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4eb400640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 0x7fd4ec151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 0x7fd4ec1f8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 -- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 msgr2=0x7fd4ec151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 0x7fd4ec151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 -- 172.21.15.129:0/2887439970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4ec1f95a0 con 0x7fd4ec1f9190 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4eaa00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 0x7fd4ec1f8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4eb400640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 0x7fd4ec151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4ebe00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec1601c0 tx=0x7fd4e400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4017070 con 0x7fd4ec1f9190 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4e4015070 con 0x7fd4ec1f9190 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4ec0b4490 con 0x7fd4ec1f9190 2024-09-17T03:59:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.266+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4e4010ea0 con 0x7fd4ec1f9190 2024-09-17T03:59:44.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.267+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4ec0b4a50 con 0x7fd4ec1f9190 2024-09-17T03:59:44.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.268+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4ec0b4200 con 0x7fd4ec1f9190 2024-09-17T03:59:44.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.269+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd4e4004030 con 0x7fd4ec1f9190 2024-09-17T03:59:44.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.270+0000 7fd4e3400640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 0x7fd4cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:44.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.270+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fd4e409b640 con 0x7fd4ec1f9190 2024-09-17T03:59:44.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.270+0000 7fd4eb400640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 0x7fd4cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:44.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.271+0000 7fd4eb400640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 0x7fd4cc07b0d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc033000 tx=0x7fd4dc02d000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:44.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.274+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4ec1f95a0 con 0x7fd4ec1f9190 2024-09-17T03:59:44.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:44.493+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd4ec0d0f50 con 0x7fd4ec1f9190 2024-09-17T03:59:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.017+0000 7fd4e3400640 1 -- 172.21.15.129:0/2887439970 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v135) ==== 122+0+0 (secure 0 0 0) 0x7fd4e4063a70 con 0x7fd4ec1f9190 2024-09-17T03:59:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 msgr2=0x7fd4cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 0x7fd4cc07b0d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc033000 tx=0x7fd4dc02d000 comp rx=0 tx=0).stop 2024-09-17T03:59:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 msgr2=0x7fd4ec1f5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec1601c0 tx=0x7fd4e400e720 comp rx=0 tx=0).stop 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 shutdown_connections 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4cc078bc0 0x7fd4cc07b0d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4ec1f9190 0x7fd4ec1f5630 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4ec15f5d0 0x7fd4ec1f8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 --2- 172.21.15.129:0/2887439970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4d0003a70 0x7fd4ec151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 >> 172.21.15.129:0/2887439970 conn(0x7fd4ec0d00b0 msgr2=0x7fd4ec09dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 shutdown_connections 2024-09-17T03:59:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.020+0000 7fd4f0e00640 1 -- 172.21.15.129:0/2887439970 wait complete. 2024-09-17T03:59:45.042 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T03:59:45.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:45.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 -- 172.21.15.129:0/2704321598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600b46e0 msgr2=0x7f83600b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 --2- 172.21.15.129:0/2704321598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600b46e0 0x7f83600b4af0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f834c009830 tx=0x7f834c02a480 comp rx=0 tx=0).stop 2024-09-17T03:59:45.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 -- 172.21.15.129:0/2704321598 shutdown_connections 2024-09-17T03:59:45.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 --2- 172.21.15.129:0/2704321598 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600b46e0 0x7f83600b4af0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 --2- 172.21.15.129:0/2704321598 >> v2:172.21.15.181:3300/0 conn(0x7f83600cfb60 0x7f8360160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 -- 172.21.15.129:0/2704321598 >> 172.21.15.129:0/2704321598 conn(0x7f836006cfc0 msgr2=0x7f836006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:45.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 -- 172.21.15.129:0/2704321598 shutdown_connections 2024-09-17T03:59:45.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.264+0000 7f835fe00640 1 -- 172.21.15.129:0/2704321598 wait complete. 2024-09-17T03:59:45.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.266+0000 7f835fe00640 1 Processor -- start 2024-09-17T03:59:45.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.266+0000 7f835fe00640 1 -- start start 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83600b46e0 0x7f83600c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600d02c0 0x7f83601f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f836015d270 con 0x7f83600cfb60 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f836015d0f0 con 0x7f83600b46e0 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835fe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f836015d3f0 con 0x7f83600d02c0 2024-09-17T03:59:45.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41260/0 (socket says 172.21.15.129:41260) 2024-09-17T03:59:45.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835e000640 1 -- 172.21.15.129:0/662527183 learned_addr learned my addr 172.21.15.129:0/662527183 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:45.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835f400640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600d02c0 0x7f83601f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835ea00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83600b46e0 0x7f83600c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835e000640 1 -- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600d02c0 msgr2=0x7f83601f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.268+0000 7f835e000640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600d02c0 0x7f83601f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835e000640 1 -- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83600b46e0 msgr2=0x7f83600c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835e000640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83600b46e0 0x7f83600c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835e000640 1 -- 172.21.15.129:0/662527183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8360152f50 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835e000640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f834c02aa70 tx=0x7f834c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c005c90 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f834c031e50 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8360153230 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834c005260 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.269+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83601f6820 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.271+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f836015d8a0 con 0x7f83600cfb60 2024-09-17T03:59:45.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.272+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f834c005400 con 0x7f83600cfb60 2024-09-17T03:59:45.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.272+0000 7f8356a00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 0x7f834007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.273+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f834c0bbd50 con 0x7f83600cfb60 2024-09-17T03:59:45.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.273+0000 7f835ea00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 0x7f834007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.274+0000 7f835ea00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 0x7f834007b0a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f83600b4450 tx=0x7f8350005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:45.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.277+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f834c03f030 con 0x7f83600cfb60 2024-09-17T03:59:45.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.492+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f836006a140 con 0x7f83600cfb60 2024-09-17T03:59:45.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.493+0000 7f8356a00640 1 -- 172.21.15.129:0/662527183 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f834c083b40 con 0x7f83600cfb60 2024-09-17T03:59:45.493 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:45.493 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:45.493 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 msgr2=0x7f834007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 0x7f834007b0a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f83600b4450 tx=0x7f8350005f90 comp rx=0 tx=0).stop 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 msgr2=0x7f83600c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f834c02aa70 tx=0x7f834c033c20 comp rx=0 tx=0).stop 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 shutdown_connections 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8340078b90 0x7f834007b0a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83600d02c0 0x7f83601f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f83600cfb60 0x7f83600c8900 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 --2- 172.21.15.129:0/662527183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83600b46e0 0x7f83600c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 >> 172.21.15.129:0/662527183 conn(0x7f836006cfc0 msgr2=0x7f836015ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 shutdown_connections 2024-09-17T03:59:45.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.496+0000 7f835fe00640 1 -- 172.21.15.129:0/662527183 wait complete. 2024-09-17T03:59:45.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:45.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 -- 172.21.15.129:0/13892001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 msgr2=0x7fba18072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 --2- 172.21.15.129:0/13892001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba18072f40 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fba180727e0 tx=0x7fba1002aac0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 -- 172.21.15.129:0/13892001 shutdown_connections 2024-09-17T03:59:45.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 --2- 172.21.15.129:0/13892001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba18072f40 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 --2- 172.21.15.129:0/13892001 >> v2:172.21.15.181:3300/0 conn(0x7fba181524c0 0x7fba180cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 -- 172.21.15.129:0/13892001 >> 172.21.15.129:0/13892001 conn(0x7fba1806cfc0 msgr2=0x7fba1806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:45.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 -- 172.21.15.129:0/13892001 shutdown_connections 2024-09-17T03:59:45.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.737+0000 7fba1d800640 1 -- 172.21.15.129:0/13892001 wait complete. 2024-09-17T03:59:45.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.739+0000 7fba1d800640 1 Processor -- start 2024-09-17T03:59:45.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.739+0000 7fba1d800640 1 -- start start 2024-09-17T03:59:45.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 0x7fba181f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 0x7fba180aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fba180d02c0 con 0x7fba181528d0 2024-09-17T03:59:45.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fba180737e0 con 0x7fba180cfb00 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fba180d0440 con 0x7fba181524c0 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41274/0 (socket says 172.21.15.129:41274) 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 -- 172.21.15.129:0/147466711 learned_addr learned my addr 172.21.15.129:0/147466711 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba17e00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 0x7fba181f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba17400640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 0x7fba180aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 -- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 msgr2=0x7fba180aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 0x7fba180aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 -- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 msgr2=0x7fba181f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 0x7fba181f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.741+0000 7fba1ce00640 1 -- 172.21.15.129:0/147466711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1814cf60 con 0x7fba181528d0 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba17400640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 0x7fba180aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:45.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba17e00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 0x7fba181f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba1ce00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fba1002b0b0 tx=0x7fba10003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10004250 con 0x7fba181528d0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba100043f0 con 0x7fba181528d0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1814bec0 con 0x7fba181528d0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10004820 con 0x7fba181528d0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.742+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1814c480 con 0x7fba181528d0 2024-09-17T03:59:45.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.743+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba180d0960 con 0x7fba181528d0 2024-09-17T03:59:45.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.745+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fba10049440 con 0x7fba181528d0 2024-09-17T03:59:45.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.746+0000 7fba0fe00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 0x7fb9f807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:45.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.746+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fba100bbb40 con 0x7fba181528d0 2024-09-17T03:59:45.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.746+0000 7fba17e00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 0x7fb9f807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:45.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.747+0000 7fba17e00640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 0x7fb9f807b0d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fba08005fd0 tx=0x7fba0800f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:45.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.750+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fba10032030 con 0x7fba181528d0 2024-09-17T03:59:45.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.931+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fba1806a140 con 0x7fba181528d0 2024-09-17T03:59:45.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.933+0000 7fba0fe00640 1 -- 172.21.15.129:0/147466711 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16779 (secure 0 0 0) 0x7fba10083f70 con 0x7fba181528d0 2024-09-17T03:59:45.932 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:45.932 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":152,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:42.980414+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:59:39.383572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-17T03:59:40.444542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 msgr2=0x7fb9f807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 0x7fb9f807b0d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fba08005fd0 tx=0x7fba0800f360 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 msgr2=0x7fba180ab420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fba1002b0b0 tx=0x7fba10003f10 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 shutdown_connections 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9f8078bc0 0x7fb9f807b0d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba181528d0 0x7fba180ab420 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba181524c0 0x7fba180aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 --2- 172.21.15.129:0/147466711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba180cfb00 0x7fba181f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:45.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.935+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 >> 172.21.15.129:0/147466711 conn(0x7fba1806cfc0 msgr2=0x7fba180b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:45.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.936+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 shutdown_connections 2024-09-17T03:59:45.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:45.936+0000 7fba1d800640 1 -- 172.21.15.129:0/147466711 wait complete. 2024-09-17T03:59:45.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T03:59:45.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:59:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 -- 172.21.15.129:0/1415913719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7c004d40 msgr2=0x7f4a7c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 --2- 172.21.15.129:0/1415913719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a7c004d40 0x7f4a7c0051f0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009b20 tx=0x7f4a7c005830 comp rx=0 tx=0).stop 2024-09-17T03:59:46.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9d400640 1 -- 172.21.15.129:0/1415913719 reap_dead start 2024-09-17T03:59:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 -- 172.21.15.129:0/1415913719 shutdown_connections 2024-09-17T03:59:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 -- 172.21.15.129:0/1415913719 >> 172.21.15.129:0/1415913719 conn(0x7f4a9808bff0 msgr2=0x7f4a9809d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 -- 172.21.15.129:0/1415913719 shutdown_connections 2024-09-17T03:59:46.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.182+0000 7f4a9de00640 1 -- 172.21.15.129:0/1415913719 wait complete. 2024-09-17T03:59:46.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.184+0000 7f4a9de00640 1 Processor -- start 2024-09-17T03:59:46.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.184+0000 7f4a9de00640 1 -- start start 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a980cdd90 0x7f4a9809ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a980cf170 0x7f4a980cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a980b9e80 con 0x7f4a9809e2b0 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4a980b9d00 con 0x7f4a980cf170 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4a980d3070 con 0x7f4a980cdd90 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41284/0 (socket says 172.21.15.129:41284) 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 -- 172.21.15.129:0/717586171 learned_addr learned my addr 172.21.15.129:0/717586171 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a97e00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a980cdd90 0x7f4a9809ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9d400640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a980cf170 0x7f4a980cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 -- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a980cdd90 msgr2=0x7f4a9809ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a980cdd90 0x7f4a9809ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 -- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a980cf170 msgr2=0x7f4a980cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a980cf170 0x7f4a980cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.186+0000 7f4a9ca00640 1 -- 172.21.15.129:0/717586171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a980cfac0 con 0x7f4a9809e2b0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a9ca00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4a88006290 tx=0x7f4a88006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88015070 con 0x7f4a9809e2b0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4a8800fb30 con 0x7f4a9809e2b0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a980b9490 con 0x7f4a9809e2b0 2024-09-17T03:59:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88011430 con 0x7f4a9809e2b0 2024-09-17T03:59:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.187+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a980a1860 con 0x7f4a9809e2b0 2024-09-17T03:59:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.188+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a980d4490 con 0x7f4a9809e2b0 2024-09-17T03:59:46.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.189+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4a880117e0 con 0x7f4a9809e2b0 2024-09-17T03:59:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.190+0000 7f4a94c00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 0x7f4a7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.190+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f4a8809bf10 con 0x7f4a9809e2b0 2024-09-17T03:59:46.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.191+0000 7f4a97e00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 0x7f4a7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.192+0000 7f4a97e00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 0x7f4a7807b0a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4a84006fd0 tx=0x7f4a84006f20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:46.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.195+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4a980cfac0 con 0x7f4a9809e2b0 2024-09-17T03:59:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.375+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4a9806a130 con 0x7f4a78078b90 2024-09-17T03:59:46.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.380+0000 7f4a94c00640 1 -- 172.21.15.129:0/717586171 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246434 (secure 0 0 0) 0x7f4a9806a130 con 0x7f4a78078b90 2024-09-17T03:59:46.382 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:46.384 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 msgr2=0x7f4a7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 0x7f4a7807b0a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4a84006fd0 tx=0x7f4a84006f20 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 msgr2=0x7f4a980b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f4a88006290 tx=0x7f4a88006750 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 shutdown_connections 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a78078b90 0x7f4a7807b0a0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a980cf170 0x7f4a980cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a980cdd90 0x7f4a9809ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 --2- 172.21.15.129:0/717586171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a9809e2b0 0x7f4a980b8c40 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.384+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 >> 172.21.15.129:0/717586171 conn(0x7f4a9808bff0 msgr2=0x7f4a98161b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:46.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.385+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 shutdown_connections 2024-09-17T03:59:46.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.385+0000 7f4a9de00640 1 -- 172.21.15.129:0/717586171 wait complete. 2024-09-17T03:59:46.405 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-17T03:59:44.467532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592654,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":129,"num_write_kb":1634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":116,"acting":116,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":5,"kb":754974720,"kb_used":315356,"kb_used_data":15916,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754659364,"statfs":{"total":773094113280,"available":772771188736,"internally_reserved":0,"allocated":16297984,"data_stored":7758905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2286,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.493087"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992469+0000","last_change":"2024-09-17T03:59:42.013350+0000","last_active":"2024-09-17T03:59:42.992469+0000","last_peered":"2024-09-17T03:59:42.992469+0000","last_clean":"2024-09-17T03:59:42.992469+0000","last_became_active":"2024-09-17T03:59:42.013032+0000","last_became_peered":"2024-09-17T03:59:42.013032+0000","last_unstale":"2024-09-17T03:59:42.992469+0000","last_undegraded":"2024-09-17T03:59:42.992469+0000","last_fullsized":"2024-09-17T03:59:42.992469+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:35.520497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991172+0000","last_change":"2024-09-17T03:59:42.020361+0000","last_active":"2024-09-17T03:59:42.991172+0000","last_peered":"2024-09-17T03:59:42.991172+0000","last_clean":"2024-09-17T03:59:42.991172+0000","last_became_active":"2024-09-17T03:59:42.018371+0000","last_became_peered":"2024-09-17T03:59:42.018371+0000","last_unstale":"2024-09-17T03:59:42.991172+0000","last_undegraded":"2024-09-17T03:59:42.991172+0000","last_fullsized":"2024-09-17T03:59:42.991172+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:33.308068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021220+0000","last_change":"2024-09-17T03:59:41.993503+0000","last_active":"2024-09-17T03:59:43.021220+0000","last_peered":"2024-09-17T03:59:43.021220+0000","last_clean":"2024-09-17T03:59:43.021220+0000","last_became_active":"2024-09-17T03:59:41.993178+0000","last_became_peered":"2024-09-17T03:59:41.993178+0000","last_unstale":"2024-09-17T03:59:43.021220+0000","last_undegraded":"2024-09-17T03:59:43.021220+0000","last_fullsized":"2024-09-17T03:59:43.021220+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:14.402939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991846+0000","last_change":"2024-09-17T03:59:42.010738+0000","last_active":"2024-09-17T03:59:42.991846+0000","last_peered":"2024-09-17T03:59:42.991846+0000","last_clean":"2024-09-17T03:59:42.991846+0000","last_became_active":"2024-09-17T03:59:42.010258+0000","last_became_peered":"2024-09-17T03:59:42.010258+0000","last_unstale":"2024-09-17T03:59:42.991846+0000","last_undegraded":"2024-09-17T03:59:42.991846+0000","last_fullsized":"2024-09-17T03:59:42.991846+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:06.976385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986922+0000","last_change":"2024-09-17T03:59:42.015045+0000","last_active":"2024-09-17T03:59:42.986922+0000","last_peered":"2024-09-17T03:59:42.986922+0000","last_clean":"2024-09-17T03:59:42.986922+0000","last_became_active":"2024-09-17T03:59:42.013752+0000","last_became_peered":"2024-09-17T03:59:42.013752+0000","last_unstale":"2024-09-17T03:59:42.986922+0000","last_undegraded":"2024-09-17T03:59:42.986922+0000","last_fullsized":"2024-09-17T03:59:42.986922+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:41.334124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990490+0000","last_change":"2024-09-17T03:59:42.008990+0000","last_active":"2024-09-17T03:59:42.990490+0000","last_peered":"2024-09-17T03:59:42.990490+0000","last_clean":"2024-09-17T03:59:42.990490+0000","last_became_active":"2024-09-17T03:59:42.008633+0000","last_became_peered":"2024-09-17T03:59:42.008633+0000","last_unstale":"2024-09-17T03:59:42.990490+0000","last_undegraded":"2024-09-17T03:59:42.990490+0000","last_fullsized":"2024-09-17T03:59:42.990490+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:51:50.379861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991636+0000","last_change":"2024-09-17T03:59:42.018251+0000","last_active":"2024-09-17T03:59:42.991636+0000","last_peered":"2024-09-17T03:59:42.991636+0000","last_clean":"2024-09-17T03:59:42.991636+0000","last_became_active":"2024-09-17T03:59:42.017826+0000","last_became_peered":"2024-09-17T03:59:42.017826+0000","last_unstale":"2024-09-17T03:59:42.991636+0000","last_undegraded":"2024-09-17T03:59:42.991636+0000","last_fullsized":"2024-09-17T03:59:42.991636+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:38.482018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991375+0000","last_change":"2024-09-17T03:59:42.014651+0000","last_active":"2024-09-17T03:59:42.991375+0000","last_peered":"2024-09-17T03:59:42.991375+0000","last_clean":"2024-09-17T03:59:42.991375+0000","last_became_active":"2024-09-17T03:59:42.014349+0000","last_became_peered":"2024-09-17T03:59:42.014349+0000","last_unstale":"2024-09-17T03:59:42.991375+0000","last_undegraded":"2024-09-17T03:59:42.991375+0000","last_fullsized":"2024-09-17T03:59:42.991375+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:18.715648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991226+0000","last_change":"2024-09-17T03:59:42.021111+0000","last_active":"2024-09-17T03:59:42.991226+0000","last_peered":"2024-09-17T03:59:42.991226+0000","last_clean":"2024-09-17T03:59:42.991226+0000","last_became_active":"2024-09-17T03:59:42.020548+0000","last_became_peered":"2024-09-17T03:59:42.020548+0000","last_unstale":"2024-09-17T03:59:42.991226+0000","last_undegraded":"2024-09-17T03:59:42.991226+0000","last_fullsized":"2024-09-17T03:59:42.991226+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:01.488453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990806+0000","last_change":"2024-09-17T03:59:42.019523+0000","last_active":"2024-09-17T03:59:42.990806+0000","last_peered":"2024-09-17T03:59:42.990806+0000","last_clean":"2024-09-17T03:59:42.990806+0000","last_became_active":"2024-09-17T03:59:42.019207+0000","last_became_peered":"2024-09-17T03:59:42.019207+0000","last_unstale":"2024-09-17T03:59:42.990806+0000","last_undegraded":"2024-09-17T03:59:42.990806+0000","last_fullsized":"2024-09-17T03:59:42.990806+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:30:18.195138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021283+0000","last_change":"2024-09-17T03:59:41.992472+0000","last_active":"2024-09-17T03:59:43.021283+0000","last_peered":"2024-09-17T03:59:43.021283+0000","last_clean":"2024-09-17T03:59:43.021283+0000","last_became_active":"2024-09-17T03:59:41.991865+0000","last_became_peered":"2024-09-17T03:59:41.991865+0000","last_unstale":"2024-09-17T03:59:43.021283+0000","last_undegraded":"2024-09-17T03:59:43.021283+0000","last_fullsized":"2024-09-17T03:59:43.021283+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021057+0000","last_change":"2024-09-17T03:59:42.004646+0000","last_active":"2024-09-17T03:59:43.021057+0000","last_peered":"2024-09-17T03:59:43.021057+0000","last_clean":"2024-09-17T03:59:43.021057+0000","last_became_active":"2024-09-17T03:59:42.004168+0000","last_became_peered":"2024-09-17T03:59:42.004168+0000","last_unstale":"2024-09-17T03:59:43.021057+0000","last_undegraded":"2024-09-17T03:59:43.021057+0000","last_fullsized":"2024-09-17T03:59:43.021057+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:46.536800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991248+0000","last_change":"2024-09-17T03:59:42.008494+0000","last_active":"2024-09-17T03:59:42.991248+0000","last_peered":"2024-09-17T03:59:42.991248+0000","last_clean":"2024-09-17T03:59:42.991248+0000","last_became_active":"2024-09-17T03:59:42.008178+0000","last_became_peered":"2024-09-17T03:59:42.008178+0000","last_unstale":"2024-09-17T03:59:42.991248+0000","last_undegraded":"2024-09-17T03:59:42.991248+0000","last_fullsized":"2024-09-17T03:59:42.991248+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:52.965716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986809+0000","last_change":"2024-09-17T03:59:42.012686+0000","last_active":"2024-09-17T03:59:42.986809+0000","last_peered":"2024-09-17T03:59:42.986809+0000","last_clean":"2024-09-17T03:59:42.986809+0000","last_became_active":"2024-09-17T03:59:42.008671+0000","last_became_peered":"2024-09-17T03:59:42.008671+0000","last_unstale":"2024-09-17T03:59:42.986809+0000","last_undegraded":"2024-09-17T03:59:42.986809+0000","last_fullsized":"2024-09-17T03:59:42.986809+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:40.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021809+0000","last_change":"2024-09-17T03:59:42.001739+0000","last_active":"2024-09-17T03:59:43.021809+0000","last_peered":"2024-09-17T03:59:43.021809+0000","last_clean":"2024-09-17T03:59:43.021809+0000","last_became_active":"2024-09-17T03:59:42.000430+0000","last_became_peered":"2024-09-17T03:59:42.000430+0000","last_unstale":"2024-09-17T03:59:43.021809+0000","last_undegraded":"2024-09-17T03:59:43.021809+0000","last_fullsized":"2024-09-17T03:59:43.021809+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:50.036438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990919+0000","last_change":"2024-09-17T03:59:42.009710+0000","last_active":"2024-09-17T03:59:42.990919+0000","last_peered":"2024-09-17T03:59:42.990919+0000","last_clean":"2024-09-17T03:59:42.990919+0000","last_became_active":"2024-09-17T03:59:42.008990+0000","last_became_peered":"2024-09-17T03:59:42.008990+0000","last_unstale":"2024-09-17T03:59:42.990919+0000","last_undegraded":"2024-09-17T03:59:42.990919+0000","last_fullsized":"2024-09-17T03:59:42.990919+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:59.836793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990584+0000","last_change":"2024-09-17T03:59:42.012956+0000","last_active":"2024-09-17T03:59:42.990584+0000","last_peered":"2024-09-17T03:59:42.990584+0000","last_clean":"2024-09-17T03:59:42.990584+0000","last_became_active":"2024-09-17T03:59:42.012463+0000","last_became_peered":"2024-09-17T03:59:42.012463+0000","last_unstale":"2024-09-17T03:59:42.990584+0000","last_undegraded":"2024-09-17T03:59:42.990584+0000","last_fullsized":"2024-09-17T03:59:42.990584+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:02.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021665+0000","last_change":"2024-09-17T03:59:41.994325+0000","last_active":"2024-09-17T03:59:43.021665+0000","last_peered":"2024-09-17T03:59:43.021665+0000","last_clean":"2024-09-17T03:59:43.021665+0000","last_became_active":"2024-09-17T03:59:41.993997+0000","last_became_peered":"2024-09-17T03:59:41.993997+0000","last_unstale":"2024-09-17T03:59:43.021665+0000","last_undegraded":"2024-09-17T03:59:43.021665+0000","last_fullsized":"2024-09-17T03:59:43.021665+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:44.653789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987717+0000","last_change":"2024-09-17T03:59:42.004001+0000","last_active":"2024-09-17T03:59:42.987717+0000","last_peered":"2024-09-17T03:59:42.987717+0000","last_clean":"2024-09-17T03:59:42.987717+0000","last_became_active":"2024-09-17T03:59:42.003335+0000","last_became_peered":"2024-09-17T03:59:42.003335+0000","last_unstale":"2024-09-17T03:59:42.987717+0000","last_undegraded":"2024-09-17T03:59:42.987717+0000","last_fullsized":"2024-09-17T03:59:42.987717+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:28.644516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021331+0000","last_change":"2024-09-17T03:59:42.002284+0000","last_active":"2024-09-17T03:59:43.021331+0000","last_peered":"2024-09-17T03:59:43.021331+0000","last_clean":"2024-09-17T03:59:43.021331+0000","last_became_active":"2024-09-17T03:59:42.001961+0000","last_became_peered":"2024-09-17T03:59:42.001961+0000","last_unstale":"2024-09-17T03:59:43.021331+0000","last_undegraded":"2024-09-17T03:59:43.021331+0000","last_fullsized":"2024-09-17T03:59:43.021331+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:25.243998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021258+0000","last_change":"2024-09-17T03:59:42.006997+0000","last_active":"2024-09-17T03:59:43.021258+0000","last_peered":"2024-09-17T03:59:43.021258+0000","last_clean":"2024-09-17T03:59:43.021258+0000","last_became_active":"2024-09-17T03:59:42.003755+0000","last_became_peered":"2024-09-17T03:59:42.003755+0000","last_unstale":"2024-09-17T03:59:43.021258+0000","last_undegraded":"2024-09-17T03:59:43.021258+0000","last_fullsized":"2024-09-17T03:59:43.021258+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:54.446834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024288+0000","last_change":"2024-09-17T03:59:42.003851+0000","last_active":"2024-09-17T03:59:43.024288+0000","last_peered":"2024-09-17T03:59:43.024288+0000","last_clean":"2024-09-17T03:59:43.024288+0000","last_became_active":"2024-09-17T03:59:42.003547+0000","last_became_peered":"2024-09-17T03:59:42.003547+0000","last_unstale":"2024-09-17T03:59:43.024288+0000","last_undegraded":"2024-09-17T03:59:43.024288+0000","last_fullsized":"2024-09-17T03:59:43.024288+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:07.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024306+0000","last_change":"2024-09-17T03:59:42.009686+0000","last_active":"2024-09-17T03:59:43.024306+0000","last_peered":"2024-09-17T03:59:43.024306+0000","last_clean":"2024-09-17T03:59:43.024306+0000","last_became_active":"2024-09-17T03:59:42.009343+0000","last_became_peered":"2024-09-17T03:59:42.009343+0000","last_unstale":"2024-09-17T03:59:43.024306+0000","last_undegraded":"2024-09-17T03:59:43.024306+0000","last_fullsized":"2024-09-17T03:59:43.024306+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:06.427384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987590+0000","last_change":"2024-09-17T03:59:42.006585+0000","last_active":"2024-09-17T03:59:42.987590+0000","last_peered":"2024-09-17T03:59:42.987590+0000","last_clean":"2024-09-17T03:59:42.987590+0000","last_became_active":"2024-09-17T03:59:42.003707+0000","last_became_peered":"2024-09-17T03:59:42.003707+0000","last_unstale":"2024-09-17T03:59:42.987590+0000","last_undegraded":"2024-09-17T03:59:42.987590+0000","last_fullsized":"2024-09-17T03:59:42.987590+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:16.138671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"151'4","reported_seq":19,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992413+0000","last_change":"2024-09-17T03:59:41.003426+0000","last_active":"2024-09-17T03:59:42.992413+0000","last_peered":"2024-09-17T03:59:42.992413+0000","last_clean":"2024-09-17T03:59:42.992413+0000","last_became_active":"2024-09-17T03:59:41.003110+0000","last_became_peered":"2024-09-17T03:59:41.003110+0000","last_unstale":"2024-09-17T03:59:42.992413+0000","last_undegraded":"2024-09-17T03:59:42.992413+0000","last_fullsized":"2024-09-17T03:59:42.992413+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:02.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024778+0000","last_change":"2024-09-17T03:59:41.990740+0000","last_active":"2024-09-17T03:59:43.024778+0000","last_peered":"2024-09-17T03:59:43.024778+0000","last_clean":"2024-09-17T03:59:43.024778+0000","last_became_active":"2024-09-17T03:59:41.990408+0000","last_became_peered":"2024-09-17T03:59:41.990408+0000","last_unstale":"2024-09-17T03:59:43.024778+0000","last_undegraded":"2024-09-17T03:59:43.024778+0000","last_fullsized":"2024-09-17T03:59:43.024778+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:42.978938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024576+0000","last_change":"2024-09-17T03:59:41.003619+0000","last_active":"2024-09-17T03:59:43.024576+0000","last_peered":"2024-09-17T03:59:43.024576+0000","last_clean":"2024-09-17T03:59:43.024576+0000","last_became_active":"2024-09-17T03:59:41.003234+0000","last_became_peered":"2024-09-17T03:59:41.003234+0000","last_unstale":"2024-09-17T03:59:43.024576+0000","last_undegraded":"2024-09-17T03:59:43.024576+0000","last_fullsized":"2024-09-17T03:59:43.024576+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:39.325179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021572+0000","last_change":"2024-09-17T03:59:42.017255+0000","last_active":"2024-09-17T03:59:43.021572+0000","last_peered":"2024-09-17T03:59:43.021572+0000","last_clean":"2024-09-17T03:59:43.021572+0000","last_became_active":"2024-09-17T03:59:42.015864+0000","last_became_peered":"2024-09-17T03:59:42.015864+0000","last_unstale":"2024-09-17T03:59:43.021572+0000","last_undegraded":"2024-09-17T03:59:43.021572+0000","last_fullsized":"2024-09-17T03:59:43.021572+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:56.580400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987215+0000","last_change":"2024-09-17T03:59:42.013270+0000","last_active":"2024-09-17T03:59:42.987215+0000","last_peered":"2024-09-17T03:59:42.987215+0000","last_clean":"2024-09-17T03:59:42.987215+0000","last_became_active":"2024-09-17T03:59:42.011974+0000","last_became_peered":"2024-09-17T03:59:42.011974+0000","last_unstale":"2024-09-17T03:59:42.987215+0000","last_undegraded":"2024-09-17T03:59:42.987215+0000","last_fullsized":"2024-09-17T03:59:42.987215+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:09.216064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021468+0000","last_change":"2024-09-17T03:59:41.006563+0000","last_active":"2024-09-17T03:59:43.021468+0000","last_peered":"2024-09-17T03:59:43.021468+0000","last_clean":"2024-09-17T03:59:43.021468+0000","last_became_active":"2024-09-17T03:59:41.006258+0000","last_became_peered":"2024-09-17T03:59:41.006258+0000","last_unstale":"2024-09-17T03:59:43.021468+0000","last_undegraded":"2024-09-17T03:59:43.021468+0000","last_fullsized":"2024-09-17T03:59:43.021468+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:47.961188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990679+0000","last_change":"2024-09-17T03:59:41.004946+0000","last_active":"2024-09-17T03:59:42.990679+0000","last_peered":"2024-09-17T03:59:42.990679+0000","last_clean":"2024-09-17T03:59:42.990679+0000","last_became_active":"2024-09-17T03:59:41.004590+0000","last_became_peered":"2024-09-17T03:59:41.004590+0000","last_unstale":"2024-09-17T03:59:42.990679+0000","last_undegraded":"2024-09-17T03:59:42.990679+0000","last_fullsized":"2024-09-17T03:59:42.990679+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:49.635573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986942+0000","last_change":"2024-09-17T03:59:40.999099+0000","last_active":"2024-09-17T03:59:42.986942+0000","last_peered":"2024-09-17T03:59:42.986942+0000","last_clean":"2024-09-17T03:59:42.986942+0000","last_became_active":"2024-09-17T03:59:40.998797+0000","last_became_peered":"2024-09-17T03:59:40.998797+0000","last_unstale":"2024-09-17T03:59:42.986942+0000","last_undegraded":"2024-09-17T03:59:42.986942+0000","last_fullsized":"2024-09-17T03:59:42.986942+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:08.365322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024719+0000","last_change":"2024-09-17T03:59:41.993617+0000","last_active":"2024-09-17T03:59:43.024719+0000","last_peered":"2024-09-17T03:59:43.024719+0000","last_clean":"2024-09-17T03:59:43.024719+0000","last_became_active":"2024-09-17T03:59:41.993273+0000","last_became_peered":"2024-09-17T03:59:41.993273+0000","last_unstale":"2024-09-17T03:59:43.024719+0000","last_undegraded":"2024-09-17T03:59:43.024719+0000","last_fullsized":"2024-09-17T03:59:43.024719+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:43.015885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"151'6","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024326+0000","last_change":"2024-09-17T03:59:40.996955+0000","last_active":"2024-09-17T03:59:43.024326+0000","last_peered":"2024-09-17T03:59:43.024326+0000","last_clean":"2024-09-17T03:59:43.024326+0000","last_became_active":"2024-09-17T03:59:40.995219+0000","last_became_peered":"2024-09-17T03:59:40.995219+0000","last_unstale":"2024-09-17T03:59:43.024326+0000","last_undegraded":"2024-09-17T03:59:43.024326+0000","last_fullsized":"2024-09-17T03:59:43.024326+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:17.857549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987711+0000","last_change":"2024-09-17T03:59:42.016664+0000","last_active":"2024-09-17T03:59:42.987711+0000","last_peered":"2024-09-17T03:59:42.987711+0000","last_clean":"2024-09-17T03:59:42.987711+0000","last_became_active":"2024-09-17T03:59:42.016336+0000","last_became_peered":"2024-09-17T03:59:42.016336+0000","last_unstale":"2024-09-17T03:59:42.987711+0000","last_undegraded":"2024-09-17T03:59:42.987711+0000","last_fullsized":"2024-09-17T03:59:42.987711+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:03.621618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021046+0000","last_change":"2024-09-17T03:59:42.004585+0000","last_active":"2024-09-17T03:59:43.021046+0000","last_peered":"2024-09-17T03:59:43.021046+0000","last_clean":"2024-09-17T03:59:43.021046+0000","last_became_active":"2024-09-17T03:59:42.004209+0000","last_became_peered":"2024-09-17T03:59:42.004209+0000","last_unstale":"2024-09-17T03:59:43.021046+0000","last_undegraded":"2024-09-17T03:59:43.021046+0000","last_fullsized":"2024-09-17T03:59:43.021046+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:36.344692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"151'8","reported_seq":23,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024132+0000","last_change":"2024-09-17T03:59:41.002551+0000","last_active":"2024-09-17T03:59:43.024132+0000","last_peered":"2024-09-17T03:59:43.024132+0000","last_clean":"2024-09-17T03:59:43.024132+0000","last_became_active":"2024-09-17T03:59:41.001997+0000","last_became_peered":"2024-09-17T03:59:41.001997+0000","last_unstale":"2024-09-17T03:59:43.024132+0000","last_undegraded":"2024-09-17T03:59:43.024132+0000","last_fullsized":"2024-09-17T03:59:43.024132+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:43.402112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.f","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021602+0000","last_change":"2024-09-17T03:59:41.003812+0000","last_active":"2024-09-17T03:59:43.021602+0000","last_peered":"2024-09-17T03:59:43.021602+0000","last_clean":"2024-09-17T03:59:43.021602+0000","last_became_active":"2024-09-17T03:59:41.003482+0000","last_became_peered":"2024-09-17T03:59:41.003482+0000","last_unstale":"2024-09-17T03:59:43.021602+0000","last_undegraded":"2024-09-17T03:59:43.021602+0000","last_fullsized":"2024-09-17T03:59:43.021602+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:58.354724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.e","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021353+0000","last_change":"2024-09-17T03:59:41.001159+0000","last_active":"2024-09-17T03:59:43.021353+0000","last_peered":"2024-09-17T03:59:43.021353+0000","last_clean":"2024-09-17T03:59:43.021353+0000","last_became_active":"2024-09-17T03:59:40.998274+0000","last_became_peered":"2024-09-17T03:59:40.998274+0000","last_unstale":"2024-09-17T03:59:43.021353+0000","last_undegraded":"2024-09-17T03:59:43.021353+0000","last_fullsized":"2024-09-17T03:59:43.021353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:59.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.d","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992254+0000","last_change":"2024-09-17T03:59:41.008320+0000","last_active":"2024-09-17T03:59:42.992254+0000","last_peered":"2024-09-17T03:59:42.992254+0000","last_clean":"2024-09-17T03:59:42.992254+0000","last_became_active":"2024-09-17T03:59:41.007982+0000","last_became_peered":"2024-09-17T03:59:41.007982+0000","last_unstale":"2024-09-17T03:59:42.992254+0000","last_undegraded":"2024-09-17T03:59:42.992254+0000","last_fullsized":"2024-09-17T03:59:42.992254+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:27.259566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987141+0000","last_change":"2024-09-17T03:59:42.018636+0000","last_active":"2024-09-17T03:59:42.987141+0000","last_peered":"2024-09-17T03:59:42.987141+0000","last_clean":"2024-09-17T03:59:42.987141+0000","last_became_active":"2024-09-17T03:59:42.017515+0000","last_became_peered":"2024-09-17T03:59:42.017515+0000","last_unstale":"2024-09-17T03:59:42.987141+0000","last_undegraded":"2024-09-17T03:59:42.987141+0000","last_fullsized":"2024-09-17T03:59:42.987141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:12.460346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992156+0000","last_change":"2024-09-17T03:59:41.004362+0000","last_active":"2024-09-17T03:59:42.992156+0000","last_peered":"2024-09-17T03:59:42.992156+0000","last_clean":"2024-09-17T03:59:42.992156+0000","last_became_active":"2024-09-17T03:59:41.003967+0000","last_became_peered":"2024-09-17T03:59:41.003967+0000","last_unstale":"2024-09-17T03:59:42.992156+0000","last_undegraded":"2024-09-17T03:59:42.992156+0000","last_fullsized":"2024-09-17T03:59:42.992156+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:07.652544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021849+0000","last_change":"2024-09-17T03:59:41.992367+0000","last_active":"2024-09-17T03:59:43.021849+0000","last_peered":"2024-09-17T03:59:43.021849+0000","last_clean":"2024-09-17T03:59:43.021849+0000","last_became_active":"2024-09-17T03:59:41.991905+0000","last_became_peered":"2024-09-17T03:59:41.991905+0000","last_unstale":"2024-09-17T03:59:43.021849+0000","last_undegraded":"2024-09-17T03:59:43.021849+0000","last_fullsized":"2024-09-17T03:59:43.021849+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:18.333814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987408+0000","last_change":"2024-09-17T03:59:42.007773+0000","last_active":"2024-09-17T03:59:42.987408+0000","last_peered":"2024-09-17T03:59:42.987408+0000","last_clean":"2024-09-17T03:59:42.987408+0000","last_became_active":"2024-09-17T03:59:42.007480+0000","last_became_peered":"2024-09-17T03:59:42.007480+0000","last_unstale":"2024-09-17T03:59:42.987408+0000","last_undegraded":"2024-09-17T03:59:42.987408+0000","last_fullsized":"2024-09-17T03:59:42.987408+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:20.412155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991757+0000","last_change":"2024-09-17T03:59:41.002988+0000","last_active":"2024-09-17T03:59:42.991757+0000","last_peered":"2024-09-17T03:59:42.991757+0000","last_clean":"2024-09-17T03:59:42.991757+0000","last_became_active":"2024-09-17T03:59:41.002652+0000","last_became_peered":"2024-09-17T03:59:41.002652+0000","last_unstale":"2024-09-17T03:59:42.991757+0000","last_undegraded":"2024-09-17T03:59:42.991757+0000","last_fullsized":"2024-09-17T03:59:42.991757+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:59.847457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990998+0000","last_change":"2024-09-17T03:59:42.014948+0000","last_active":"2024-09-17T03:59:42.990998+0000","last_peered":"2024-09-17T03:59:42.990998+0000","last_clean":"2024-09-17T03:59:42.990998+0000","last_became_active":"2024-09-17T03:59:42.014492+0000","last_became_peered":"2024-09-17T03:59:42.014492+0000","last_unstale":"2024-09-17T03:59:42.990998+0000","last_undegraded":"2024-09-17T03:59:42.990998+0000","last_fullsized":"2024-09-17T03:59:42.990998+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:26.810565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991158+0000","last_change":"2024-09-17T03:59:41.008494+0000","last_active":"2024-09-17T03:59:42.991158+0000","last_peered":"2024-09-17T03:59:42.991158+0000","last_clean":"2024-09-17T03:59:42.991158+0000","last_became_active":"2024-09-17T03:59:41.008146+0000","last_became_peered":"2024-09-17T03:59:41.008146+0000","last_unstale":"2024-09-17T03:59:42.991158+0000","last_undegraded":"2024-09-17T03:59:42.991158+0000","last_fullsized":"2024-09-17T03:59:42.991158+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:38.735689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987765+0000","last_change":"2024-09-17T03:59:42.017076+0000","last_active":"2024-09-17T03:59:42.987765+0000","last_peered":"2024-09-17T03:59:42.987765+0000","last_clean":"2024-09-17T03:59:42.987765+0000","last_became_active":"2024-09-17T03:59:42.015449+0000","last_became_peered":"2024-09-17T03:59:42.015449+0000","last_unstale":"2024-09-17T03:59:42.987765+0000","last_undegraded":"2024-09-17T03:59:42.987765+0000","last_fullsized":"2024-09-17T03:59:42.987765+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:40.898210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"152'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.023389+0000","last_change":"2024-09-17T03:59:40.994555+0000","last_active":"2024-09-17T03:59:43.023389+0000","last_peered":"2024-09-17T03:59:43.023389+0000","last_clean":"2024-09-17T03:59:43.023389+0000","last_became_active":"2024-09-17T03:59:40.994217+0000","last_became_peered":"2024-09-17T03:59:40.994217+0000","last_unstale":"2024-09-17T03:59:43.023389+0000","last_undegraded":"2024-09-17T03:59:43.023389+0000","last_fullsized":"2024-09-17T03:59:43.023389+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:45.696048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991366+0000","last_change":"2024-09-17T03:59:40.998993+0000","last_active":"2024-09-17T03:59:42.991366+0000","last_peered":"2024-09-17T03:59:42.991366+0000","last_clean":"2024-09-17T03:59:42.991366+0000","last_became_active":"2024-09-17T03:59:40.998688+0000","last_became_peered":"2024-09-17T03:59:40.998688+0000","last_unstale":"2024-09-17T03:59:42.991366+0000","last_undegraded":"2024-09-17T03:59:42.991366+0000","last_fullsized":"2024-09-17T03:59:42.991366+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:48.469413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024460+0000","last_change":"2024-09-17T03:59:40.993985+0000","last_active":"2024-09-17T03:59:43.024460+0000","last_peered":"2024-09-17T03:59:43.024460+0000","last_clean":"2024-09-17T03:59:43.024460+0000","last_became_active":"2024-09-17T03:59:40.993478+0000","last_became_peered":"2024-09-17T03:59:40.993478+0000","last_unstale":"2024-09-17T03:59:43.024460+0000","last_undegraded":"2024-09-17T03:59:43.024460+0000","last_fullsized":"2024-09-17T03:59:43.024460+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:48.777659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024494+0000","last_change":"2024-09-17T03:59:42.008925+0000","last_active":"2024-09-17T03:59:43.024494+0000","last_peered":"2024-09-17T03:59:43.024494+0000","last_clean":"2024-09-17T03:59:43.024494+0000","last_became_active":"2024-09-17T03:59:42.008558+0000","last_became_peered":"2024-09-17T03:59:42.008558+0000","last_unstale":"2024-09-17T03:59:43.024494+0000","last_undegraded":"2024-09-17T03:59:43.024494+0000","last_fullsized":"2024-09-17T03:59:43.024494+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:32.847843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:39.969976+0000","last_change":"2024-09-17T03:59:39.969976+0000","last_active":"2024-09-17T03:59:39.969976+0000","last_peered":"2024-09-17T03:59:39.969976+0000","last_clean":"2024-09-17T03:59:39.969976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:39.969976+0000","last_undegraded":"2024-09-17T03:59:39.969976+0000","last_fullsized":"2024-09-17T03:59:39.969976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986816+0000","last_change":"2024-09-17T03:59:42.009605+0000","last_active":"2024-09-17T03:59:42.986816+0000","last_peered":"2024-09-17T03:59:42.986816+0000","last_clean":"2024-09-17T03:59:42.986816+0000","last_became_active":"2024-09-17T03:59:42.008457+0000","last_became_peered":"2024-09-17T03:59:42.008457+0000","last_unstale":"2024-09-17T03:59:42.986816+0000","last_undegraded":"2024-09-17T03:59:42.986816+0000","last_fullsized":"2024-09-17T03:59:42.986816+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:32.937097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990858+0000","last_change":"2024-09-17T03:59:41.001408+0000","last_active":"2024-09-17T03:59:42.990858+0000","last_peered":"2024-09-17T03:59:42.990858+0000","last_clean":"2024-09-17T03:59:42.990858+0000","last_became_active":"2024-09-17T03:59:41.000952+0000","last_became_peered":"2024-09-17T03:59:41.000952+0000","last_unstale":"2024-09-17T03:59:42.990858+0000","last_undegraded":"2024-09-17T03:59:42.990858+0000","last_fullsized":"2024-09-17T03:59:42.990858+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:10.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021108+0000","last_change":"2024-09-17T03:59:42.005720+0000","last_active":"2024-09-17T03:59:43.021108+0000","last_peered":"2024-09-17T03:59:43.021108+0000","last_clean":"2024-09-17T03:59:43.021108+0000","last_became_active":"2024-09-17T03:59:42.005032+0000","last_became_peered":"2024-09-17T03:59:42.005032+0000","last_unstale":"2024-09-17T03:59:43.021108+0000","last_undegraded":"2024-09-17T03:59:43.021108+0000","last_fullsized":"2024-09-17T03:59:43.021108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:58.113705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024130+0000","last_change":"2024-09-17T03:59:41.007461+0000","last_active":"2024-09-17T03:59:43.024130+0000","last_peered":"2024-09-17T03:59:43.024130+0000","last_clean":"2024-09-17T03:59:43.024130+0000","last_became_active":"2024-09-17T03:59:41.007095+0000","last_became_peered":"2024-09-17T03:59:41.007095+0000","last_unstale":"2024-09-17T03:59:43.024130+0000","last_undegraded":"2024-09-17T03:59:43.024130+0000","last_fullsized":"2024-09-17T03:59:43.024130+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.287988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-17T03:59:40.972271+0000","last_change":"2024-09-17T03:59:40.972271+0000","last_active":"2024-09-17T03:59:40.972271+0000","last_peered":"2024-09-17T03:59:40.972271+0000","last_clean":"2024-09-17T03:59:40.972271+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T03:59:40.972271+0000","last_undegraded":"2024-09-17T03:59:40.972271+0000","last_fullsized":"2024-09-17T03:59:40.972271+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024744+0000","last_change":"2024-09-17T03:59:41.002325+0000","last_active":"2024-09-17T03:59:43.024744+0000","last_peered":"2024-09-17T03:59:43.024744+0000","last_clean":"2024-09-17T03:59:43.024744+0000","last_became_active":"2024-09-17T03:59:41.001931+0000","last_became_peered":"2024-09-17T03:59:41.001931+0000","last_unstale":"2024-09-17T03:59:43.024744+0000","last_undegraded":"2024-09-17T03:59:43.024744+0000","last_fullsized":"2024-09-17T03:59:43.024744+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:22:45.390790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":353,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991235+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:59:42.991235+0000","last_peered":"2024-09-17T03:59:42.991235+0000","last_clean":"2024-09-17T03:59:42.991235+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:59:42.991235+0000","last_undegraded":"2024-09-17T03:59:42.991235+0000","last_fullsized":"2024-09-17T03:59:42.991235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":76,"acting":76,"num_store_stats":5},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2286,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25,"ondisk_log_size":25,"up":38,"acting":38,"num_store_stats":5},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509566,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2519040,"data_stored":1423848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509569,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":1908736,"data_stored":831730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41380,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330460,"statfs":{"total":96636764160,"available":96594391040,"internally_reserved":0,"allocated":1916928,"data_stored":832808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509565,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":805959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37160,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334680,"statfs":{"total":96636764160,"available":96598712320,"internally_reserved":0,"allocated":1851392,"data_stored":805959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":37156,"kb_used_data":1808,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334684,"statfs":{"total":96636764160,"available":96598716416,"internally_reserved":0,"allocated":1851392,"data_stored":805959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542270,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":1904640,"data_stored":831148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542271,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2494464,"data_stored":1421494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:59:46.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 -- 172.21.15.129:0/3117864845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 msgr2=0x7f43001609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 --2- 172.21.15.129:0/3117864845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001609c0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f42f8005fd0 tx=0x7f42f802aa90 comp rx=0 tx=0).stop 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 -- 172.21.15.129:0/3117864845 shutdown_connections 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 --2- 172.21.15.129:0/3117864845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001609c0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 --2- 172.21.15.129:0/3117864845 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000b4140 0x7f43000ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 -- 172.21.15.129:0/3117864845 >> 172.21.15.129:0/3117864845 conn(0x7f430006d410 msgr2=0x7f43000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 -- 172.21.15.129:0/3117864845 shutdown_connections 2024-09-17T03:59:46.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.665+0000 7f4307200640 1 -- 172.21.15.129:0/3117864845 wait complete. 2024-09-17T03:59:46.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.667+0000 7f4307200640 1 Processor -- start 2024-09-17T03:59:46.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.668+0000 7f4307200640 1 -- start start 2024-09-17T03:59:46.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000705a0 0x7f430015e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43000b4140 0x7f43001e6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4300160260 con 0x7f43000cd1c0 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43001600e0 con 0x7f43000b4140 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4307200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4300160f00 con 0x7f43000705a0 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4305200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4305200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41304/0 (socket says 172.21.15.129:41304) 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f4305200640 1 -- 172.21.15.129:0/1598524650 learned_addr learned my addr 172.21.15.129:0/1598524650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f42ffe00640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000705a0 0x7f430015e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.669+0000 7f42ff400640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43000b4140 0x7f43001e6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 -- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000705a0 msgr2=0x7f430015e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000705a0 0x7f430015e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 -- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43000b4140 msgr2=0x7f43001e6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43000b4140 0x7f43001e6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 -- 172.21.15.129:0/1598524650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f430015eb20 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4305200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009d90 tx=0x7f42f8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8044070 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42f8004250 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f8004660 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.670+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43001e8dc0 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.671+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f430015e590 con 0x7f43000cd1c0 2024-09-17T03:59:46.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.672+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43000b2170 con 0x7f43000cd1c0 2024-09-17T03:59:46.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.673+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f42f8047070 con 0x7f43000cd1c0 2024-09-17T03:59:46.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.674+0000 7f42f7e00640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 0x7f42e007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:46.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.674+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f42f80bbcb0 con 0x7f43000cd1c0 2024-09-17T03:59:46.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.674+0000 7f42ffe00640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 0x7f42e007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:46.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.675+0000 7f42ffe00640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 0x7f42e007b0d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f42f0001720 tx=0x7f42f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:46.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.678+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f43001e8dc0 con 0x7f43000cd1c0 2024-09-17T03:59:46.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.857+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43000ccc80 con 0x7f42e0078bc0 2024-09-17T03:59:46.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.862+0000 7f42f7e00640 1 -- 172.21.15.129:0/1598524650 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252474 (secure 0 0 0) 0x7f43000ccc80 con 0x7f42e0078bc0 2024-09-17T03:59:46.864 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 msgr2=0x7f42e007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 0x7f42e007b0d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f42f0001720 tx=0x7f42f0009040 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 msgr2=0x7f43001e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f42f8009d90 tx=0x7f42f8003ce0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 shutdown_connections 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e0078bc0 0x7f42e007b0d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43000cd1c0 0x7f43001e6d50 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.866+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43000b4140 0x7f43001e6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.867+0000 7f4307200640 1 --2- 172.21.15.129:0/1598524650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43000705a0 0x7f430015e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.867+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 >> 172.21.15.129:0/1598524650 conn(0x7f430006d410 msgr2=0x7f4300072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:46.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.867+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 shutdown_connections 2024-09-17T03:59:46.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:46.867+0000 7f4307200640 1 -- 172.21.15.129:0/1598524650 wait complete. 2024-09-17T03:59:46.889 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-17T03:59:46.468199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315504,"kb_used_data":16072,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754659216,"statfs":{"total":773094113280,"available":772771037184,"internally_reserved":0,"allocated":16457728,"data_stored":7834564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001595"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992469+0000","last_change":"2024-09-17T03:59:42.013350+0000","last_active":"2024-09-17T03:59:42.992469+0000","last_peered":"2024-09-17T03:59:42.992469+0000","last_clean":"2024-09-17T03:59:42.992469+0000","last_became_active":"2024-09-17T03:59:42.013032+0000","last_became_peered":"2024-09-17T03:59:42.013032+0000","last_unstale":"2024-09-17T03:59:42.992469+0000","last_undegraded":"2024-09-17T03:59:42.992469+0000","last_fullsized":"2024-09-17T03:59:42.992469+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:35.520497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991172+0000","last_change":"2024-09-17T03:59:42.020361+0000","last_active":"2024-09-17T03:59:42.991172+0000","last_peered":"2024-09-17T03:59:42.991172+0000","last_clean":"2024-09-17T03:59:42.991172+0000","last_became_active":"2024-09-17T03:59:42.018371+0000","last_became_peered":"2024-09-17T03:59:42.018371+0000","last_unstale":"2024-09-17T03:59:42.991172+0000","last_undegraded":"2024-09-17T03:59:42.991172+0000","last_fullsized":"2024-09-17T03:59:42.991172+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:33.308068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021220+0000","last_change":"2024-09-17T03:59:41.993503+0000","last_active":"2024-09-17T03:59:43.021220+0000","last_peered":"2024-09-17T03:59:43.021220+0000","last_clean":"2024-09-17T03:59:43.021220+0000","last_became_active":"2024-09-17T03:59:41.993178+0000","last_became_peered":"2024-09-17T03:59:41.993178+0000","last_unstale":"2024-09-17T03:59:43.021220+0000","last_undegraded":"2024-09-17T03:59:43.021220+0000","last_fullsized":"2024-09-17T03:59:43.021220+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:14.402939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084092+0000","last_change":"2024-09-17T03:59:42.003075+0000","last_active":"2024-09-17T03:59:43.084092+0000","last_peered":"2024-09-17T03:59:43.084092+0000","last_clean":"2024-09-17T03:59:43.084092+0000","last_became_active":"2024-09-17T03:59:42.002702+0000","last_became_peered":"2024-09-17T03:59:42.002702+0000","last_unstale":"2024-09-17T03:59:43.084092+0000","last_undegraded":"2024-09-17T03:59:43.084092+0000","last_fullsized":"2024-09-17T03:59:43.084092+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:07.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991846+0000","last_change":"2024-09-17T03:59:42.010738+0000","last_active":"2024-09-17T03:59:42.991846+0000","last_peered":"2024-09-17T03:59:42.991846+0000","last_clean":"2024-09-17T03:59:42.991846+0000","last_became_active":"2024-09-17T03:59:42.010258+0000","last_became_peered":"2024-09-17T03:59:42.010258+0000","last_unstale":"2024-09-17T03:59:42.991846+0000","last_undegraded":"2024-09-17T03:59:42.991846+0000","last_fullsized":"2024-09-17T03:59:42.991846+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:06.976385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271053+0000","last_change":"2024-09-17T03:59:42.006924+0000","last_active":"2024-09-17T03:59:43.271053+0000","last_peered":"2024-09-17T03:59:43.271053+0000","last_clean":"2024-09-17T03:59:43.271053+0000","last_became_active":"2024-09-17T03:59:42.006603+0000","last_became_peered":"2024-09-17T03:59:42.006603+0000","last_unstale":"2024-09-17T03:59:43.271053+0000","last_undegraded":"2024-09-17T03:59:43.271053+0000","last_fullsized":"2024-09-17T03:59:43.271053+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:45.995935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986922+0000","last_change":"2024-09-17T03:59:42.015045+0000","last_active":"2024-09-17T03:59:42.986922+0000","last_peered":"2024-09-17T03:59:42.986922+0000","last_clean":"2024-09-17T03:59:42.986922+0000","last_became_active":"2024-09-17T03:59:42.013752+0000","last_became_peered":"2024-09-17T03:59:42.013752+0000","last_unstale":"2024-09-17T03:59:42.986922+0000","last_undegraded":"2024-09-17T03:59:42.986922+0000","last_fullsized":"2024-09-17T03:59:42.986922+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:41.334124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990490+0000","last_change":"2024-09-17T03:59:42.008990+0000","last_active":"2024-09-17T03:59:42.990490+0000","last_peered":"2024-09-17T03:59:42.990490+0000","last_clean":"2024-09-17T03:59:42.990490+0000","last_became_active":"2024-09-17T03:59:42.008633+0000","last_became_peered":"2024-09-17T03:59:42.008633+0000","last_unstale":"2024-09-17T03:59:42.990490+0000","last_undegraded":"2024-09-17T03:59:42.990490+0000","last_fullsized":"2024-09-17T03:59:42.990490+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:51:50.379861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991636+0000","last_change":"2024-09-17T03:59:42.018251+0000","last_active":"2024-09-17T03:59:42.991636+0000","last_peered":"2024-09-17T03:59:42.991636+0000","last_clean":"2024-09-17T03:59:42.991636+0000","last_became_active":"2024-09-17T03:59:42.017826+0000","last_became_peered":"2024-09-17T03:59:42.017826+0000","last_unstale":"2024-09-17T03:59:42.991636+0000","last_undegraded":"2024-09-17T03:59:42.991636+0000","last_fullsized":"2024-09-17T03:59:42.991636+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:50:38.482018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991375+0000","last_change":"2024-09-17T03:59:42.014651+0000","last_active":"2024-09-17T03:59:42.991375+0000","last_peered":"2024-09-17T03:59:42.991375+0000","last_clean":"2024-09-17T03:59:42.991375+0000","last_became_active":"2024-09-17T03:59:42.014349+0000","last_became_peered":"2024-09-17T03:59:42.014349+0000","last_unstale":"2024-09-17T03:59:42.991375+0000","last_undegraded":"2024-09-17T03:59:42.991375+0000","last_fullsized":"2024-09-17T03:59:42.991375+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:18.715648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991226+0000","last_change":"2024-09-17T03:59:42.021111+0000","last_active":"2024-09-17T03:59:42.991226+0000","last_peered":"2024-09-17T03:59:42.991226+0000","last_clean":"2024-09-17T03:59:42.991226+0000","last_became_active":"2024-09-17T03:59:42.020548+0000","last_became_peered":"2024-09-17T03:59:42.020548+0000","last_unstale":"2024-09-17T03:59:42.991226+0000","last_undegraded":"2024-09-17T03:59:42.991226+0000","last_fullsized":"2024-09-17T03:59:42.991226+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:49:01.488453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990806+0000","last_change":"2024-09-17T03:59:42.019523+0000","last_active":"2024-09-17T03:59:42.990806+0000","last_peered":"2024-09-17T03:59:42.990806+0000","last_clean":"2024-09-17T03:59:42.990806+0000","last_became_active":"2024-09-17T03:59:42.019207+0000","last_became_peered":"2024-09-17T03:59:42.019207+0000","last_unstale":"2024-09-17T03:59:42.990806+0000","last_undegraded":"2024-09-17T03:59:42.990806+0000","last_fullsized":"2024-09-17T03:59:42.990806+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:30:18.195138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021283+0000","last_change":"2024-09-17T03:59:41.992472+0000","last_active":"2024-09-17T03:59:43.021283+0000","last_peered":"2024-09-17T03:59:43.021283+0000","last_clean":"2024-09-17T03:59:43.021283+0000","last_became_active":"2024-09-17T03:59:41.991865+0000","last_became_peered":"2024-09-17T03:59:41.991865+0000","last_unstale":"2024-09-17T03:59:43.021283+0000","last_undegraded":"2024-09-17T03:59:43.021283+0000","last_fullsized":"2024-09-17T03:59:43.021283+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:32.979332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021057+0000","last_change":"2024-09-17T03:59:42.004646+0000","last_active":"2024-09-17T03:59:43.021057+0000","last_peered":"2024-09-17T03:59:43.021057+0000","last_clean":"2024-09-17T03:59:43.021057+0000","last_became_active":"2024-09-17T03:59:42.004168+0000","last_became_peered":"2024-09-17T03:59:42.004168+0000","last_unstale":"2024-09-17T03:59:43.021057+0000","last_undegraded":"2024-09-17T03:59:43.021057+0000","last_fullsized":"2024-09-17T03:59:43.021057+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:46.536800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991248+0000","last_change":"2024-09-17T03:59:42.008494+0000","last_active":"2024-09-17T03:59:42.991248+0000","last_peered":"2024-09-17T03:59:42.991248+0000","last_clean":"2024-09-17T03:59:42.991248+0000","last_became_active":"2024-09-17T03:59:42.008178+0000","last_became_peered":"2024-09-17T03:59:42.008178+0000","last_unstale":"2024-09-17T03:59:42.991248+0000","last_undegraded":"2024-09-17T03:59:42.991248+0000","last_fullsized":"2024-09-17T03:59:42.991248+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:52.965716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986809+0000","last_change":"2024-09-17T03:59:42.012686+0000","last_active":"2024-09-17T03:59:42.986809+0000","last_peered":"2024-09-17T03:59:42.986809+0000","last_clean":"2024-09-17T03:59:42.986809+0000","last_became_active":"2024-09-17T03:59:42.008671+0000","last_became_peered":"2024-09-17T03:59:42.008671+0000","last_unstale":"2024-09-17T03:59:42.986809+0000","last_undegraded":"2024-09-17T03:59:42.986809+0000","last_fullsized":"2024-09-17T03:59:42.986809+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:40.292330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271949+0000","last_change":"2024-09-17T03:59:42.009154+0000","last_active":"2024-09-17T03:59:43.271949+0000","last_peered":"2024-09-17T03:59:43.271949+0000","last_clean":"2024-09-17T03:59:43.271949+0000","last_became_active":"2024-09-17T03:59:42.008841+0000","last_became_peered":"2024-09-17T03:59:42.008841+0000","last_unstale":"2024-09-17T03:59:43.271949+0000","last_undegraded":"2024-09-17T03:59:43.271949+0000","last_fullsized":"2024-09-17T03:59:43.271949+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:39.149070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271825+0000","last_change":"2024-09-17T03:59:42.011497+0000","last_active":"2024-09-17T03:59:43.271825+0000","last_peered":"2024-09-17T03:59:43.271825+0000","last_clean":"2024-09-17T03:59:43.271825+0000","last_became_active":"2024-09-17T03:59:42.011056+0000","last_became_peered":"2024-09-17T03:59:42.011056+0000","last_unstale":"2024-09-17T03:59:43.271825+0000","last_undegraded":"2024-09-17T03:59:43.271825+0000","last_fullsized":"2024-09-17T03:59:43.271825+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:43.644925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271596+0000","last_change":"2024-09-17T03:59:42.006358+0000","last_active":"2024-09-17T03:59:43.271596+0000","last_peered":"2024-09-17T03:59:43.271596+0000","last_clean":"2024-09-17T03:59:43.271596+0000","last_became_active":"2024-09-17T03:59:42.005699+0000","last_became_peered":"2024-09-17T03:59:42.005699+0000","last_unstale":"2024-09-17T03:59:43.271596+0000","last_undegraded":"2024-09-17T03:59:43.271596+0000","last_fullsized":"2024-09-17T03:59:43.271596+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:59.357019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021809+0000","last_change":"2024-09-17T03:59:42.001739+0000","last_active":"2024-09-17T03:59:43.021809+0000","last_peered":"2024-09-17T03:59:43.021809+0000","last_clean":"2024-09-17T03:59:43.021809+0000","last_became_active":"2024-09-17T03:59:42.000430+0000","last_became_peered":"2024-09-17T03:59:42.000430+0000","last_unstale":"2024-09-17T03:59:43.021809+0000","last_undegraded":"2024-09-17T03:59:43.021809+0000","last_fullsized":"2024-09-17T03:59:43.021809+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:50.036438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271209+0000","last_change":"2024-09-17T03:59:42.012643+0000","last_active":"2024-09-17T03:59:43.271209+0000","last_peered":"2024-09-17T03:59:43.271209+0000","last_clean":"2024-09-17T03:59:43.271209+0000","last_became_active":"2024-09-17T03:59:42.012321+0000","last_became_peered":"2024-09-17T03:59:42.012321+0000","last_unstale":"2024-09-17T03:59:43.271209+0000","last_undegraded":"2024-09-17T03:59:43.271209+0000","last_fullsized":"2024-09-17T03:59:43.271209+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:00.499687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990919+0000","last_change":"2024-09-17T03:59:42.009710+0000","last_active":"2024-09-17T03:59:42.990919+0000","last_peered":"2024-09-17T03:59:42.990919+0000","last_clean":"2024-09-17T03:59:42.990919+0000","last_became_active":"2024-09-17T03:59:42.008990+0000","last_became_peered":"2024-09-17T03:59:42.008990+0000","last_unstale":"2024-09-17T03:59:42.990919+0000","last_undegraded":"2024-09-17T03:59:42.990919+0000","last_fullsized":"2024-09-17T03:59:42.990919+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:59.836793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.270860+0000","last_change":"2024-09-17T03:59:42.008677+0000","last_active":"2024-09-17T03:59:43.270860+0000","last_peered":"2024-09-17T03:59:43.270860+0000","last_clean":"2024-09-17T03:59:43.270860+0000","last_became_active":"2024-09-17T03:59:42.007983+0000","last_became_peered":"2024-09-17T03:59:42.007983+0000","last_unstale":"2024-09-17T03:59:43.270860+0000","last_undegraded":"2024-09-17T03:59:43.270860+0000","last_fullsized":"2024-09-17T03:59:43.270860+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:35.039106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990584+0000","last_change":"2024-09-17T03:59:42.012956+0000","last_active":"2024-09-17T03:59:42.990584+0000","last_peered":"2024-09-17T03:59:42.990584+0000","last_clean":"2024-09-17T03:59:42.990584+0000","last_became_active":"2024-09-17T03:59:42.012463+0000","last_became_peered":"2024-09-17T03:59:42.012463+0000","last_unstale":"2024-09-17T03:59:42.990584+0000","last_undegraded":"2024-09-17T03:59:42.990584+0000","last_fullsized":"2024-09-17T03:59:42.990584+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:02.089444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021665+0000","last_change":"2024-09-17T03:59:41.994325+0000","last_active":"2024-09-17T03:59:43.021665+0000","last_peered":"2024-09-17T03:59:43.021665+0000","last_clean":"2024-09-17T03:59:43.021665+0000","last_became_active":"2024-09-17T03:59:41.993997+0000","last_became_peered":"2024-09-17T03:59:41.993997+0000","last_unstale":"2024-09-17T03:59:43.021665+0000","last_undegraded":"2024-09-17T03:59:43.021665+0000","last_fullsized":"2024-09-17T03:59:43.021665+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:44.653789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987717+0000","last_change":"2024-09-17T03:59:42.004001+0000","last_active":"2024-09-17T03:59:42.987717+0000","last_peered":"2024-09-17T03:59:42.987717+0000","last_clean":"2024-09-17T03:59:42.987717+0000","last_became_active":"2024-09-17T03:59:42.003335+0000","last_became_peered":"2024-09-17T03:59:42.003335+0000","last_unstale":"2024-09-17T03:59:42.987717+0000","last_undegraded":"2024-09-17T03:59:42.987717+0000","last_fullsized":"2024-09-17T03:59:42.987717+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:28.644516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021331+0000","last_change":"2024-09-17T03:59:42.002284+0000","last_active":"2024-09-17T03:59:43.021331+0000","last_peered":"2024-09-17T03:59:43.021331+0000","last_clean":"2024-09-17T03:59:43.021331+0000","last_became_active":"2024-09-17T03:59:42.001961+0000","last_became_peered":"2024-09-17T03:59:42.001961+0000","last_unstale":"2024-09-17T03:59:43.021331+0000","last_undegraded":"2024-09-17T03:59:43.021331+0000","last_fullsized":"2024-09-17T03:59:43.021331+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:25.243998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084183+0000","last_change":"2024-09-17T03:59:42.010531+0000","last_active":"2024-09-17T03:59:43.084183+0000","last_peered":"2024-09-17T03:59:43.084183+0000","last_clean":"2024-09-17T03:59:43.084183+0000","last_became_active":"2024-09-17T03:59:42.010194+0000","last_became_peered":"2024-09-17T03:59:42.010194+0000","last_unstale":"2024-09-17T03:59:43.084183+0000","last_undegraded":"2024-09-17T03:59:43.084183+0000","last_fullsized":"2024-09-17T03:59:43.084183+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:06.427384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021258+0000","last_change":"2024-09-17T03:59:42.006997+0000","last_active":"2024-09-17T03:59:43.021258+0000","last_peered":"2024-09-17T03:59:43.021258+0000","last_clean":"2024-09-17T03:59:43.021258+0000","last_became_active":"2024-09-17T03:59:42.003755+0000","last_became_peered":"2024-09-17T03:59:42.003755+0000","last_unstale":"2024-09-17T03:59:43.021258+0000","last_undegraded":"2024-09-17T03:59:43.021258+0000","last_fullsized":"2024-09-17T03:59:43.021258+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:54.446834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024288+0000","last_change":"2024-09-17T03:59:42.003851+0000","last_active":"2024-09-17T03:59:43.024288+0000","last_peered":"2024-09-17T03:59:43.024288+0000","last_clean":"2024-09-17T03:59:43.024288+0000","last_became_active":"2024-09-17T03:59:42.003547+0000","last_became_peered":"2024-09-17T03:59:42.003547+0000","last_unstale":"2024-09-17T03:59:43.024288+0000","last_undegraded":"2024-09-17T03:59:43.024288+0000","last_fullsized":"2024-09-17T03:59:43.024288+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:07.088095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024306+0000","last_change":"2024-09-17T03:59:42.009686+0000","last_active":"2024-09-17T03:59:43.024306+0000","last_peered":"2024-09-17T03:59:43.024306+0000","last_clean":"2024-09-17T03:59:43.024306+0000","last_became_active":"2024-09-17T03:59:42.009343+0000","last_became_peered":"2024-09-17T03:59:42.009343+0000","last_unstale":"2024-09-17T03:59:43.024306+0000","last_undegraded":"2024-09-17T03:59:43.024306+0000","last_fullsized":"2024-09-17T03:59:43.024306+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:06.427384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.270763+0000","last_change":"2024-09-17T03:59:42.010026+0000","last_active":"2024-09-17T03:59:43.270763+0000","last_peered":"2024-09-17T03:59:43.270763+0000","last_clean":"2024-09-17T03:59:43.270763+0000","last_became_active":"2024-09-17T03:59:42.006964+0000","last_became_peered":"2024-09-17T03:59:42.006964+0000","last_unstale":"2024-09-17T03:59:43.270763+0000","last_undegraded":"2024-09-17T03:59:43.270763+0000","last_fullsized":"2024-09-17T03:59:43.270763+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:47.508365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987590+0000","last_change":"2024-09-17T03:59:42.006585+0000","last_active":"2024-09-17T03:59:42.987590+0000","last_peered":"2024-09-17T03:59:42.987590+0000","last_clean":"2024-09-17T03:59:42.987590+0000","last_became_active":"2024-09-17T03:59:42.003707+0000","last_became_peered":"2024-09-17T03:59:42.003707+0000","last_unstale":"2024-09-17T03:59:42.987590+0000","last_undegraded":"2024-09-17T03:59:42.987590+0000","last_fullsized":"2024-09-17T03:59:42.987590+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:16.138671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"151'4","reported_seq":19,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992413+0000","last_change":"2024-09-17T03:59:41.003426+0000","last_active":"2024-09-17T03:59:42.992413+0000","last_peered":"2024-09-17T03:59:42.992413+0000","last_clean":"2024-09-17T03:59:42.992413+0000","last_became_active":"2024-09-17T03:59:41.003110+0000","last_became_peered":"2024-09-17T03:59:41.003110+0000","last_unstale":"2024-09-17T03:59:42.992413+0000","last_undegraded":"2024-09-17T03:59:42.992413+0000","last_fullsized":"2024-09-17T03:59:42.992413+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:02.485795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024778+0000","last_change":"2024-09-17T03:59:41.990740+0000","last_active":"2024-09-17T03:59:43.024778+0000","last_peered":"2024-09-17T03:59:43.024778+0000","last_clean":"2024-09-17T03:59:43.024778+0000","last_became_active":"2024-09-17T03:59:41.990408+0000","last_became_peered":"2024-09-17T03:59:41.990408+0000","last_unstale":"2024-09-17T03:59:43.024778+0000","last_undegraded":"2024-09-17T03:59:43.024778+0000","last_fullsized":"2024-09-17T03:59:43.024778+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:13:42.978938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024576+0000","last_change":"2024-09-17T03:59:41.003619+0000","last_active":"2024-09-17T03:59:43.024576+0000","last_peered":"2024-09-17T03:59:43.024576+0000","last_clean":"2024-09-17T03:59:43.024576+0000","last_became_active":"2024-09-17T03:59:41.003234+0000","last_became_peered":"2024-09-17T03:59:41.003234+0000","last_unstale":"2024-09-17T03:59:43.024576+0000","last_undegraded":"2024-09-17T03:59:43.024576+0000","last_fullsized":"2024-09-17T03:59:43.024576+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:39.325179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271359+0000","last_change":"2024-09-17T03:59:42.004921+0000","last_active":"2024-09-17T03:59:43.271359+0000","last_peered":"2024-09-17T03:59:43.271359+0000","last_clean":"2024-09-17T03:59:43.271359+0000","last_became_active":"2024-09-17T03:59:42.004571+0000","last_became_peered":"2024-09-17T03:59:42.004571+0000","last_unstale":"2024-09-17T03:59:43.271359+0000","last_undegraded":"2024-09-17T03:59:43.271359+0000","last_fullsized":"2024-09-17T03:59:43.271359+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:48.963483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084218+0000","last_change":"2024-09-17T03:59:41.008088+0000","last_active":"2024-09-17T03:59:43.084218+0000","last_peered":"2024-09-17T03:59:43.084218+0000","last_clean":"2024-09-17T03:59:43.084218+0000","last_became_active":"2024-09-17T03:59:41.007757+0000","last_became_peered":"2024-09-17T03:59:41.007757+0000","last_unstale":"2024-09-17T03:59:43.084218+0000","last_undegraded":"2024-09-17T03:59:43.084218+0000","last_fullsized":"2024-09-17T03:59:43.084218+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:25:41.774407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021572+0000","last_change":"2024-09-17T03:59:42.017255+0000","last_active":"2024-09-17T03:59:43.021572+0000","last_peered":"2024-09-17T03:59:43.021572+0000","last_clean":"2024-09-17T03:59:43.021572+0000","last_became_active":"2024-09-17T03:59:42.015864+0000","last_became_peered":"2024-09-17T03:59:42.015864+0000","last_unstale":"2024-09-17T03:59:43.021572+0000","last_undegraded":"2024-09-17T03:59:43.021572+0000","last_fullsized":"2024-09-17T03:59:43.021572+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:56.580400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084332+0000","last_change":"2024-09-17T03:59:41.002603+0000","last_active":"2024-09-17T03:59:43.084332+0000","last_peered":"2024-09-17T03:59:43.084332+0000","last_clean":"2024-09-17T03:59:43.084332+0000","last_became_active":"2024-09-17T03:59:41.002261+0000","last_became_peered":"2024-09-17T03:59:41.002261+0000","last_unstale":"2024-09-17T03:59:43.084332+0000","last_undegraded":"2024-09-17T03:59:43.084332+0000","last_fullsized":"2024-09-17T03:59:43.084332+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:53.383730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987215+0000","last_change":"2024-09-17T03:59:42.013270+0000","last_active":"2024-09-17T03:59:42.987215+0000","last_peered":"2024-09-17T03:59:42.987215+0000","last_clean":"2024-09-17T03:59:42.987215+0000","last_became_active":"2024-09-17T03:59:42.011974+0000","last_became_peered":"2024-09-17T03:59:42.011974+0000","last_unstale":"2024-09-17T03:59:42.987215+0000","last_undegraded":"2024-09-17T03:59:42.987215+0000","last_fullsized":"2024-09-17T03:59:42.987215+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:09.216064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021468+0000","last_change":"2024-09-17T03:59:41.006563+0000","last_active":"2024-09-17T03:59:43.021468+0000","last_peered":"2024-09-17T03:59:43.021468+0000","last_clean":"2024-09-17T03:59:43.021468+0000","last_became_active":"2024-09-17T03:59:41.006258+0000","last_became_peered":"2024-09-17T03:59:41.006258+0000","last_unstale":"2024-09-17T03:59:43.021468+0000","last_undegraded":"2024-09-17T03:59:43.021468+0000","last_fullsized":"2024-09-17T03:59:43.021468+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:19:47.961188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083856+0000","last_change":"2024-09-17T03:59:42.017999+0000","last_active":"2024-09-17T03:59:43.083856+0000","last_peered":"2024-09-17T03:59:43.083856+0000","last_clean":"2024-09-17T03:59:43.083856+0000","last_became_active":"2024-09-17T03:59:42.016073+0000","last_became_peered":"2024-09-17T03:59:42.016073+0000","last_unstale":"2024-09-17T03:59:43.083856+0000","last_undegraded":"2024-09-17T03:59:43.083856+0000","last_fullsized":"2024-09-17T03:59:43.083856+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:57.028846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084017+0000","last_change":"2024-09-17T03:59:41.005781+0000","last_active":"2024-09-17T03:59:43.084017+0000","last_peered":"2024-09-17T03:59:43.084017+0000","last_clean":"2024-09-17T03:59:43.084017+0000","last_became_active":"2024-09-17T03:59:41.005422+0000","last_became_peered":"2024-09-17T03:59:41.005422+0000","last_unstale":"2024-09-17T03:59:43.084017+0000","last_undegraded":"2024-09-17T03:59:43.084017+0000","last_fullsized":"2024-09-17T03:59:43.084017+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:29.927570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083789+0000","last_change":"2024-09-17T03:59:42.014694+0000","last_active":"2024-09-17T03:59:43.083789+0000","last_peered":"2024-09-17T03:59:43.083789+0000","last_clean":"2024-09-17T03:59:43.083789+0000","last_became_active":"2024-09-17T03:59:42.009714+0000","last_became_peered":"2024-09-17T03:59:42.009714+0000","last_unstale":"2024-09-17T03:59:43.083789+0000","last_undegraded":"2024-09-17T03:59:43.083789+0000","last_fullsized":"2024-09-17T03:59:43.083789+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:57.187341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990679+0000","last_change":"2024-09-17T03:59:41.004946+0000","last_active":"2024-09-17T03:59:42.990679+0000","last_peered":"2024-09-17T03:59:42.990679+0000","last_clean":"2024-09-17T03:59:42.990679+0000","last_became_active":"2024-09-17T03:59:41.004590+0000","last_became_peered":"2024-09-17T03:59:41.004590+0000","last_unstale":"2024-09-17T03:59:42.990679+0000","last_undegraded":"2024-09-17T03:59:42.990679+0000","last_fullsized":"2024-09-17T03:59:42.990679+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:49.635573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.270936+0000","last_change":"2024-09-17T03:59:42.009259+0000","last_active":"2024-09-17T03:59:43.270936+0000","last_peered":"2024-09-17T03:59:43.270936+0000","last_clean":"2024-09-17T03:59:43.270936+0000","last_became_active":"2024-09-17T03:59:42.008393+0000","last_became_peered":"2024-09-17T03:59:42.008393+0000","last_unstale":"2024-09-17T03:59:43.270936+0000","last_undegraded":"2024-09-17T03:59:43.270936+0000","last_fullsized":"2024-09-17T03:59:43.270936+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:35:25.965860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986942+0000","last_change":"2024-09-17T03:59:40.999099+0000","last_active":"2024-09-17T03:59:42.986942+0000","last_peered":"2024-09-17T03:59:42.986942+0000","last_clean":"2024-09-17T03:59:42.986942+0000","last_became_active":"2024-09-17T03:59:40.998797+0000","last_became_peered":"2024-09-17T03:59:40.998797+0000","last_unstale":"2024-09-17T03:59:42.986942+0000","last_undegraded":"2024-09-17T03:59:42.986942+0000","last_fullsized":"2024-09-17T03:59:42.986942+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:06:08.365322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271973+0000","last_change":"2024-09-17T03:59:42.011725+0000","last_active":"2024-09-17T03:59:43.271973+0000","last_peered":"2024-09-17T03:59:43.271973+0000","last_clean":"2024-09-17T03:59:43.271973+0000","last_became_active":"2024-09-17T03:59:42.011450+0000","last_became_peered":"2024-09-17T03:59:42.011450+0000","last_unstale":"2024-09-17T03:59:43.271973+0000","last_undegraded":"2024-09-17T03:59:43.271973+0000","last_fullsized":"2024-09-17T03:59:43.271973+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:08.108019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271903+0000","last_change":"2024-09-17T03:59:40.999472+0000","last_active":"2024-09-17T03:59:43.271903+0000","last_peered":"2024-09-17T03:59:43.271903+0000","last_clean":"2024-09-17T03:59:43.271903+0000","last_became_active":"2024-09-17T03:59:40.998455+0000","last_became_peered":"2024-09-17T03:59:40.998455+0000","last_unstale":"2024-09-17T03:59:43.271903+0000","last_undegraded":"2024-09-17T03:59:43.271903+0000","last_fullsized":"2024-09-17T03:59:43.271903+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:33:36.397871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271530+0000","last_change":"2024-09-17T03:59:42.008066+0000","last_active":"2024-09-17T03:59:43.271530+0000","last_peered":"2024-09-17T03:59:43.271530+0000","last_clean":"2024-09-17T03:59:43.271530+0000","last_became_active":"2024-09-17T03:59:42.007465+0000","last_became_peered":"2024-09-17T03:59:42.007465+0000","last_unstale":"2024-09-17T03:59:43.271530+0000","last_undegraded":"2024-09-17T03:59:43.271530+0000","last_fullsized":"2024-09-17T03:59:43.271530+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:00.694761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"151'3","reported_seq":18,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271712+0000","last_change":"2024-09-17T03:59:40.994599+0000","last_active":"2024-09-17T03:59:43.271712+0000","last_peered":"2024-09-17T03:59:43.271712+0000","last_clean":"2024-09-17T03:59:43.271712+0000","last_became_active":"2024-09-17T03:59:40.994217+0000","last_became_peered":"2024-09-17T03:59:40.994217+0000","last_unstale":"2024-09-17T03:59:43.271712+0000","last_undegraded":"2024-09-17T03:59:43.271712+0000","last_fullsized":"2024-09-17T03:59:43.271712+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:01.848520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024719+0000","last_change":"2024-09-17T03:59:41.993617+0000","last_active":"2024-09-17T03:59:43.024719+0000","last_peered":"2024-09-17T03:59:43.024719+0000","last_clean":"2024-09-17T03:59:43.024719+0000","last_became_active":"2024-09-17T03:59:41.993273+0000","last_became_peered":"2024-09-17T03:59:41.993273+0000","last_unstale":"2024-09-17T03:59:43.024719+0000","last_undegraded":"2024-09-17T03:59:43.024719+0000","last_fullsized":"2024-09-17T03:59:43.024719+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:43.015885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"151'6","reported_seq":21,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024326+0000","last_change":"2024-09-17T03:59:40.996955+0000","last_active":"2024-09-17T03:59:43.024326+0000","last_peered":"2024-09-17T03:59:43.024326+0000","last_clean":"2024-09-17T03:59:43.024326+0000","last_became_active":"2024-09-17T03:59:40.995219+0000","last_became_peered":"2024-09-17T03:59:40.995219+0000","last_unstale":"2024-09-17T03:59:43.024326+0000","last_undegraded":"2024-09-17T03:59:43.024326+0000","last_fullsized":"2024-09-17T03:59:43.024326+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:17.857549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271165+0000","last_change":"2024-09-17T03:59:42.012807+0000","last_active":"2024-09-17T03:59:43.271165+0000","last_peered":"2024-09-17T03:59:43.271165+0000","last_clean":"2024-09-17T03:59:43.271165+0000","last_became_active":"2024-09-17T03:59:42.012500+0000","last_became_peered":"2024-09-17T03:59:42.012500+0000","last_unstale":"2024-09-17T03:59:43.271165+0000","last_undegraded":"2024-09-17T03:59:43.271165+0000","last_fullsized":"2024-09-17T03:59:43.271165+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:26.455958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083985+0000","last_change":"2024-09-17T03:59:41.000362+0000","last_active":"2024-09-17T03:59:43.083985+0000","last_peered":"2024-09-17T03:59:43.083985+0000","last_clean":"2024-09-17T03:59:43.083985+0000","last_became_active":"2024-09-17T03:59:41.000047+0000","last_became_peered":"2024-09-17T03:59:41.000047+0000","last_unstale":"2024-09-17T03:59:43.083985+0000","last_undegraded":"2024-09-17T03:59:43.083985+0000","last_fullsized":"2024-09-17T03:59:43.083985+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:17:09.114474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987711+0000","last_change":"2024-09-17T03:59:42.016664+0000","last_active":"2024-09-17T03:59:42.987711+0000","last_peered":"2024-09-17T03:59:42.987711+0000","last_clean":"2024-09-17T03:59:42.987711+0000","last_became_active":"2024-09-17T03:59:42.016336+0000","last_became_peered":"2024-09-17T03:59:42.016336+0000","last_unstale":"2024-09-17T03:59:42.987711+0000","last_undegraded":"2024-09-17T03:59:42.987711+0000","last_fullsized":"2024-09-17T03:59:42.987711+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:03.621618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271541+0000","last_change":"2024-09-17T03:59:40.989494+0000","last_active":"2024-09-17T03:59:43.271541+0000","last_peered":"2024-09-17T03:59:43.271541+0000","last_clean":"2024-09-17T03:59:43.271541+0000","last_became_active":"2024-09-17T03:59:40.986135+0000","last_became_peered":"2024-09-17T03:59:40.986135+0000","last_unstale":"2024-09-17T03:59:43.271541+0000","last_undegraded":"2024-09-17T03:59:43.271541+0000","last_fullsized":"2024-09-17T03:59:43.271541+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:36.941302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083801+0000","last_change":"2024-09-17T03:59:42.013768+0000","last_active":"2024-09-17T03:59:43.083801+0000","last_peered":"2024-09-17T03:59:43.083801+0000","last_clean":"2024-09-17T03:59:43.083801+0000","last_became_active":"2024-09-17T03:59:42.013476+0000","last_became_peered":"2024-09-17T03:59:42.013476+0000","last_unstale":"2024-09-17T03:59:43.083801+0000","last_undegraded":"2024-09-17T03:59:43.083801+0000","last_fullsized":"2024-09-17T03:59:43.083801+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:55.224415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083928+0000","last_change":"2024-09-17T03:59:41.007906+0000","last_active":"2024-09-17T03:59:43.083928+0000","last_peered":"2024-09-17T03:59:43.083928+0000","last_clean":"2024-09-17T03:59:43.083928+0000","last_became_active":"2024-09-17T03:59:41.007579+0000","last_became_peered":"2024-09-17T03:59:41.007579+0000","last_unstale":"2024-09-17T03:59:43.083928+0000","last_undegraded":"2024-09-17T03:59:43.083928+0000","last_fullsized":"2024-09-17T03:59:43.083928+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:21.997431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021046+0000","last_change":"2024-09-17T03:59:42.004585+0000","last_active":"2024-09-17T03:59:43.021046+0000","last_peered":"2024-09-17T03:59:43.021046+0000","last_clean":"2024-09-17T03:59:43.021046+0000","last_became_active":"2024-09-17T03:59:42.004209+0000","last_became_peered":"2024-09-17T03:59:42.004209+0000","last_unstale":"2024-09-17T03:59:43.021046+0000","last_undegraded":"2024-09-17T03:59:43.021046+0000","last_fullsized":"2024-09-17T03:59:43.021046+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:36.344692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"151'8","reported_seq":23,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024132+0000","last_change":"2024-09-17T03:59:41.002551+0000","last_active":"2024-09-17T03:59:43.024132+0000","last_peered":"2024-09-17T03:59:43.024132+0000","last_clean":"2024-09-17T03:59:43.024132+0000","last_became_active":"2024-09-17T03:59:41.001997+0000","last_became_peered":"2024-09-17T03:59:41.001997+0000","last_unstale":"2024-09-17T03:59:43.024132+0000","last_undegraded":"2024-09-17T03:59:43.024132+0000","last_fullsized":"2024-09-17T03:59:43.024132+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:11:43.402112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084442+0000","last_change":"2024-09-17T03:59:42.012326+0000","last_active":"2024-09-17T03:59:43.084442+0000","last_peered":"2024-09-17T03:59:43.084442+0000","last_clean":"2024-09-17T03:59:43.084442+0000","last_became_active":"2024-09-17T03:59:42.011872+0000","last_became_peered":"2024-09-17T03:59:42.011872+0000","last_unstale":"2024-09-17T03:59:43.084442+0000","last_undegraded":"2024-09-17T03:59:43.084442+0000","last_fullsized":"2024-09-17T03:59:43.084442+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:34:43.015885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021602+0000","last_change":"2024-09-17T03:59:41.003812+0000","last_active":"2024-09-17T03:59:43.021602+0000","last_peered":"2024-09-17T03:59:43.021602+0000","last_clean":"2024-09-17T03:59:43.021602+0000","last_became_active":"2024-09-17T03:59:41.003482+0000","last_became_peered":"2024-09-17T03:59:41.003482+0000","last_unstale":"2024-09-17T03:59:43.021602+0000","last_undegraded":"2024-09-17T03:59:43.021602+0000","last_fullsized":"2024-09-17T03:59:43.021602+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:58.354724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.272193+0000","last_change":"2024-09-17T03:59:41.999885+0000","last_active":"2024-09-17T03:59:43.272193+0000","last_peered":"2024-09-17T03:59:43.272193+0000","last_clean":"2024-09-17T03:59:43.272193+0000","last_became_active":"2024-09-17T03:59:41.999564+0000","last_became_peered":"2024-09-17T03:59:41.999564+0000","last_unstale":"2024-09-17T03:59:43.272193+0000","last_undegraded":"2024-09-17T03:59:43.272193+0000","last_fullsized":"2024-09-17T03:59:43.272193+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:50.082778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021353+0000","last_change":"2024-09-17T03:59:41.001159+0000","last_active":"2024-09-17T03:59:43.021353+0000","last_peered":"2024-09-17T03:59:43.021353+0000","last_clean":"2024-09-17T03:59:43.021353+0000","last_became_active":"2024-09-17T03:59:40.998274+0000","last_became_peered":"2024-09-17T03:59:40.998274+0000","last_unstale":"2024-09-17T03:59:43.021353+0000","last_undegraded":"2024-09-17T03:59:43.021353+0000","last_fullsized":"2024-09-17T03:59:43.021353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:59.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084335+0000","last_change":"2024-09-17T03:59:42.002910+0000","last_active":"2024-09-17T03:59:43.084335+0000","last_peered":"2024-09-17T03:59:43.084335+0000","last_clean":"2024-09-17T03:59:43.084335+0000","last_became_active":"2024-09-17T03:59:42.002255+0000","last_became_peered":"2024-09-17T03:59:42.002255+0000","last_unstale":"2024-09-17T03:59:43.084335+0000","last_undegraded":"2024-09-17T03:59:43.084335+0000","last_fullsized":"2024-09-17T03:59:43.084335+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:32.847843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"151'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992254+0000","last_change":"2024-09-17T03:59:41.008320+0000","last_active":"2024-09-17T03:59:42.992254+0000","last_peered":"2024-09-17T03:59:42.992254+0000","last_clean":"2024-09-17T03:59:42.992254+0000","last_became_active":"2024-09-17T03:59:41.007982+0000","last_became_peered":"2024-09-17T03:59:41.007982+0000","last_unstale":"2024-09-17T03:59:42.992254+0000","last_undegraded":"2024-09-17T03:59:42.992254+0000","last_fullsized":"2024-09-17T03:59:42.992254+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:27.259566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987141+0000","last_change":"2024-09-17T03:59:42.018636+0000","last_active":"2024-09-17T03:59:42.987141+0000","last_peered":"2024-09-17T03:59:42.987141+0000","last_clean":"2024-09-17T03:59:42.987141+0000","last_became_active":"2024-09-17T03:59:42.017515+0000","last_became_peered":"2024-09-17T03:59:42.017515+0000","last_unstale":"2024-09-17T03:59:42.987141+0000","last_undegraded":"2024-09-17T03:59:42.987141+0000","last_fullsized":"2024-09-17T03:59:42.987141+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:12.460346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.992156+0000","last_change":"2024-09-17T03:59:41.004362+0000","last_active":"2024-09-17T03:59:42.992156+0000","last_peered":"2024-09-17T03:59:42.992156+0000","last_clean":"2024-09-17T03:59:42.992156+0000","last_became_active":"2024-09-17T03:59:41.003967+0000","last_became_peered":"2024-09-17T03:59:41.003967+0000","last_unstale":"2024-09-17T03:59:42.992156+0000","last_undegraded":"2024-09-17T03:59:42.992156+0000","last_fullsized":"2024-09-17T03:59:42.992156+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:07.652544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021849+0000","last_change":"2024-09-17T03:59:41.992367+0000","last_active":"2024-09-17T03:59:43.021849+0000","last_peered":"2024-09-17T03:59:43.021849+0000","last_clean":"2024-09-17T03:59:43.021849+0000","last_became_active":"2024-09-17T03:59:41.991905+0000","last_became_peered":"2024-09-17T03:59:41.991905+0000","last_unstale":"2024-09-17T03:59:43.021849+0000","last_undegraded":"2024-09-17T03:59:43.021849+0000","last_fullsized":"2024-09-17T03:59:43.021849+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:47:18.333814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.084024+0000","last_change":"2024-09-17T03:59:41.007749+0000","last_active":"2024-09-17T03:59:43.084024+0000","last_peered":"2024-09-17T03:59:43.084024+0000","last_clean":"2024-09-17T03:59:43.084024+0000","last_became_active":"2024-09-17T03:59:41.007457+0000","last_became_peered":"2024-09-17T03:59:41.007457+0000","last_unstale":"2024-09-17T03:59:43.084024+0000","last_undegraded":"2024-09-17T03:59:43.084024+0000","last_fullsized":"2024-09-17T03:59:43.084024+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:31.191537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987408+0000","last_change":"2024-09-17T03:59:42.007773+0000","last_active":"2024-09-17T03:59:42.987408+0000","last_peered":"2024-09-17T03:59:42.987408+0000","last_clean":"2024-09-17T03:59:42.987408+0000","last_became_active":"2024-09-17T03:59:42.007480+0000","last_became_peered":"2024-09-17T03:59:42.007480+0000","last_unstale":"2024-09-17T03:59:42.987408+0000","last_undegraded":"2024-09-17T03:59:42.987408+0000","last_fullsized":"2024-09-17T03:59:42.987408+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:20.412155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991757+0000","last_change":"2024-09-17T03:59:41.002988+0000","last_active":"2024-09-17T03:59:42.991757+0000","last_peered":"2024-09-17T03:59:42.991757+0000","last_clean":"2024-09-17T03:59:42.991757+0000","last_became_active":"2024-09-17T03:59:41.002652+0000","last_became_peered":"2024-09-17T03:59:41.002652+0000","last_unstale":"2024-09-17T03:59:42.991757+0000","last_undegraded":"2024-09-17T03:59:42.991757+0000","last_fullsized":"2024-09-17T03:59:42.991757+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:59.847457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990998+0000","last_change":"2024-09-17T03:59:42.014948+0000","last_active":"2024-09-17T03:59:42.990998+0000","last_peered":"2024-09-17T03:59:42.990998+0000","last_clean":"2024-09-17T03:59:42.990998+0000","last_became_active":"2024-09-17T03:59:42.014492+0000","last_became_peered":"2024-09-17T03:59:42.014492+0000","last_unstale":"2024-09-17T03:59:42.990998+0000","last_undegraded":"2024-09-17T03:59:42.990998+0000","last_fullsized":"2024-09-17T03:59:42.990998+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:26.810565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083783+0000","last_change":"2024-09-17T03:59:41.003821+0000","last_active":"2024-09-17T03:59:43.083783+0000","last_peered":"2024-09-17T03:59:43.083783+0000","last_clean":"2024-09-17T03:59:43.083783+0000","last_became_active":"2024-09-17T03:59:41.003508+0000","last_became_peered":"2024-09-17T03:59:41.003508+0000","last_unstale":"2024-09-17T03:59:43.083783+0000","last_undegraded":"2024-09-17T03:59:43.083783+0000","last_fullsized":"2024-09-17T03:59:43.083783+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:18:00.716135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271077+0000","last_change":"2024-09-17T03:59:42.011799+0000","last_active":"2024-09-17T03:59:43.271077+0000","last_peered":"2024-09-17T03:59:43.271077+0000","last_clean":"2024-09-17T03:59:43.271077+0000","last_became_active":"2024-09-17T03:59:42.011506+0000","last_became_peered":"2024-09-17T03:59:42.011506+0000","last_unstale":"2024-09-17T03:59:43.271077+0000","last_undegraded":"2024-09-17T03:59:43.271077+0000","last_fullsized":"2024-09-17T03:59:43.271077+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:02:37.221034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991158+0000","last_change":"2024-09-17T03:59:41.008494+0000","last_active":"2024-09-17T03:59:42.991158+0000","last_peered":"2024-09-17T03:59:42.991158+0000","last_clean":"2024-09-17T03:59:42.991158+0000","last_became_active":"2024-09-17T03:59:41.008146+0000","last_became_peered":"2024-09-17T03:59:41.008146+0000","last_unstale":"2024-09-17T03:59:42.991158+0000","last_undegraded":"2024-09-17T03:59:42.991158+0000","last_fullsized":"2024-09-17T03:59:42.991158+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:09:38.735689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.272064+0000","last_change":"2024-09-17T03:59:42.003583+0000","last_active":"2024-09-17T03:59:43.272064+0000","last_peered":"2024-09-17T03:59:43.272064+0000","last_clean":"2024-09-17T03:59:43.272064+0000","last_became_active":"2024-09-17T03:59:42.002888+0000","last_became_peered":"2024-09-17T03:59:42.002888+0000","last_unstale":"2024-09-17T03:59:43.272064+0000","last_undegraded":"2024-09-17T03:59:43.272064+0000","last_fullsized":"2024-09-17T03:59:43.272064+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:05.990452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"151'2","reported_seq":17,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.272153+0000","last_change":"2024-09-17T03:59:40.998679+0000","last_active":"2024-09-17T03:59:43.272153+0000","last_peered":"2024-09-17T03:59:43.272153+0000","last_clean":"2024-09-17T03:59:43.272153+0000","last_became_active":"2024-09-17T03:59:40.997799+0000","last_became_peered":"2024-09-17T03:59:40.997799+0000","last_unstale":"2024-09-17T03:59:43.272153+0000","last_undegraded":"2024-09-17T03:59:43.272153+0000","last_fullsized":"2024-09-17T03:59:43.272153+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:06.085800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.987765+0000","last_change":"2024-09-17T03:59:42.017076+0000","last_active":"2024-09-17T03:59:42.987765+0000","last_peered":"2024-09-17T03:59:42.987765+0000","last_clean":"2024-09-17T03:59:42.987765+0000","last_became_active":"2024-09-17T03:59:42.015449+0000","last_became_peered":"2024-09-17T03:59:42.015449+0000","last_unstale":"2024-09-17T03:59:42.987765+0000","last_undegraded":"2024-09-17T03:59:42.987765+0000","last_fullsized":"2024-09-17T03:59:42.987765+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:27:40.898210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"152'1","reported_seq":16,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.023389+0000","last_change":"2024-09-17T03:59:40.994555+0000","last_active":"2024-09-17T03:59:43.023389+0000","last_peered":"2024-09-17T03:59:43.023389+0000","last_clean":"2024-09-17T03:59:43.023389+0000","last_became_active":"2024-09-17T03:59:40.994217+0000","last_became_peered":"2024-09-17T03:59:40.994217+0000","last_unstale":"2024-09-17T03:59:43.023389+0000","last_undegraded":"2024-09-17T03:59:43.023389+0000","last_fullsized":"2024-09-17T03:59:43.023389+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:45.696048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271573+0000","last_change":"2024-09-17T03:59:42.005808+0000","last_active":"2024-09-17T03:59:43.271573+0000","last_peered":"2024-09-17T03:59:43.271573+0000","last_clean":"2024-09-17T03:59:43.271573+0000","last_became_active":"2024-09-17T03:59:42.005350+0000","last_became_peered":"2024-09-17T03:59:42.005350+0000","last_unstale":"2024-09-17T03:59:43.271573+0000","last_undegraded":"2024-09-17T03:59:43.271573+0000","last_fullsized":"2024-09-17T03:59:43.271573+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:37:46.698343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991366+0000","last_change":"2024-09-17T03:59:40.998993+0000","last_active":"2024-09-17T03:59:42.991366+0000","last_peered":"2024-09-17T03:59:42.991366+0000","last_clean":"2024-09-17T03:59:42.991366+0000","last_became_active":"2024-09-17T03:59:40.998688+0000","last_became_peered":"2024-09-17T03:59:40.998688+0000","last_unstale":"2024-09-17T03:59:42.991366+0000","last_undegraded":"2024-09-17T03:59:42.991366+0000","last_fullsized":"2024-09-17T03:59:42.991366+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:37:48.469413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271736+0000","last_change":"2024-09-17T03:59:42.003622+0000","last_active":"2024-09-17T03:59:43.271736+0000","last_peered":"2024-09-17T03:59:43.271736+0000","last_clean":"2024-09-17T03:59:43.271736+0000","last_became_active":"2024-09-17T03:59:42.003322+0000","last_became_peered":"2024-09-17T03:59:42.003322+0000","last_unstale":"2024-09-17T03:59:43.271736+0000","last_undegraded":"2024-09-17T03:59:43.271736+0000","last_fullsized":"2024-09-17T03:59:43.271736+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:27.510624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271649+0000","last_change":"2024-09-17T03:59:40.999837+0000","last_active":"2024-09-17T03:59:43.271649+0000","last_peered":"2024-09-17T03:59:43.271649+0000","last_clean":"2024-09-17T03:59:43.271649+0000","last_became_active":"2024-09-17T03:59:40.998983+0000","last_became_peered":"2024-09-17T03:59:40.998983+0000","last_unstale":"2024-09-17T03:59:43.271649+0000","last_undegraded":"2024-09-17T03:59:43.271649+0000","last_fullsized":"2024-09-17T03:59:43.271649+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:06.791091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271423+0000","last_change":"2024-09-17T03:59:42.010691+0000","last_active":"2024-09-17T03:59:43.271423+0000","last_peered":"2024-09-17T03:59:43.271423+0000","last_clean":"2024-09-17T03:59:43.271423+0000","last_became_active":"2024-09-17T03:59:42.007357+0000","last_became_peered":"2024-09-17T03:59:42.007357+0000","last_unstale":"2024-09-17T03:59:43.271423+0000","last_undegraded":"2024-09-17T03:59:43.271423+0000","last_fullsized":"2024-09-17T03:59:43.271423+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:17.740214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024460+0000","last_change":"2024-09-17T03:59:40.993985+0000","last_active":"2024-09-17T03:59:43.024460+0000","last_peered":"2024-09-17T03:59:43.024460+0000","last_clean":"2024-09-17T03:59:43.024460+0000","last_became_active":"2024-09-17T03:59:40.993478+0000","last_became_peered":"2024-09-17T03:59:40.993478+0000","last_unstale":"2024-09-17T03:59:43.024460+0000","last_undegraded":"2024-09-17T03:59:43.024460+0000","last_fullsized":"2024-09-17T03:59:43.024460+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:48.777659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024494+0000","last_change":"2024-09-17T03:59:42.008925+0000","last_active":"2024-09-17T03:59:43.024494+0000","last_peered":"2024-09-17T03:59:43.024494+0000","last_clean":"2024-09-17T03:59:43.024494+0000","last_became_active":"2024-09-17T03:59:42.008558+0000","last_became_peered":"2024-09-17T03:59:42.008558+0000","last_unstale":"2024-09-17T03:59:43.024494+0000","last_undegraded":"2024-09-17T03:59:43.024494+0000","last_fullsized":"2024-09-17T03:59:43.024494+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:32.847843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.083946+0000","last_change":"2024-09-17T03:59:41.005801+0000","last_active":"2024-09-17T03:59:43.083946+0000","last_peered":"2024-09-17T03:59:43.083946+0000","last_clean":"2024-09-17T03:59:43.083946+0000","last_became_active":"2024-09-17T03:59:41.005319+0000","last_became_peered":"2024-09-17T03:59:41.005319+0000","last_unstale":"2024-09-17T03:59:43.083946+0000","last_undegraded":"2024-09-17T03:59:43.083946+0000","last_fullsized":"2024-09-17T03:59:43.083946+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:20.857340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.986816+0000","last_change":"2024-09-17T03:59:42.009605+0000","last_active":"2024-09-17T03:59:42.986816+0000","last_peered":"2024-09-17T03:59:42.986816+0000","last_clean":"2024-09-17T03:59:42.986816+0000","last_became_active":"2024-09-17T03:59:42.008457+0000","last_became_peered":"2024-09-17T03:59:42.008457+0000","last_unstale":"2024-09-17T03:59:42.986816+0000","last_undegraded":"2024-09-17T03:59:42.986816+0000","last_fullsized":"2024-09-17T03:59:42.986816+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:32.937097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.990858+0000","last_change":"2024-09-17T03:59:41.001408+0000","last_active":"2024-09-17T03:59:42.990858+0000","last_peered":"2024-09-17T03:59:42.990858+0000","last_clean":"2024-09-17T03:59:42.990858+0000","last_became_active":"2024-09-17T03:59:41.000952+0000","last_became_peered":"2024-09-17T03:59:41.000952+0000","last_unstale":"2024-09-17T03:59:42.990858+0000","last_undegraded":"2024-09-17T03:59:42.990858+0000","last_fullsized":"2024-09-17T03:59:42.990858+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:10.931815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.021108+0000","last_change":"2024-09-17T03:59:42.005720+0000","last_active":"2024-09-17T03:59:43.021108+0000","last_peered":"2024-09-17T03:59:43.021108+0000","last_clean":"2024-09-17T03:59:43.021108+0000","last_became_active":"2024-09-17T03:59:42.005032+0000","last_became_peered":"2024-09-17T03:59:42.005032+0000","last_unstale":"2024-09-17T03:59:43.021108+0000","last_undegraded":"2024-09-17T03:59:43.021108+0000","last_fullsized":"2024-09-17T03:59:43.021108+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:14:58.113705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024130+0000","last_change":"2024-09-17T03:59:41.007461+0000","last_active":"2024-09-17T03:59:43.024130+0000","last_peered":"2024-09-17T03:59:43.024130+0000","last_clean":"2024-09-17T03:59:43.024130+0000","last_became_active":"2024-09-17T03:59:41.007095+0000","last_became_peered":"2024-09-17T03:59:41.007095+0000","last_unstale":"2024-09-17T03:59:43.024130+0000","last_undegraded":"2024-09-17T03:59:43.024130+0000","last_fullsized":"2024-09-17T03:59:43.024130+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:13:54.287988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":13,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.271987+0000","last_change":"2024-09-17T03:59:42.003955+0000","last_active":"2024-09-17T03:59:43.271987+0000","last_peered":"2024-09-17T03:59:43.271987+0000","last_clean":"2024-09-17T03:59:43.271987+0000","last_became_active":"2024-09-17T03:59:42.003279+0000","last_became_peered":"2024-09-17T03:59:42.003279+0000","last_unstale":"2024-09-17T03:59:43.271987+0000","last_undegraded":"2024-09-17T03:59:43.271987+0000","last_fullsized":"2024-09-17T03:59:43.271987+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:40.972271+0000","last_clean_scrub_stamp":"2024-09-17T03:59:40.972271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:05.014789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":15,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:43.024744+0000","last_change":"2024-09-17T03:59:41.002325+0000","last_active":"2024-09-17T03:59:43.024744+0000","last_peered":"2024-09-17T03:59:43.024744+0000","last_clean":"2024-09-17T03:59:43.024744+0000","last_became_active":"2024-09-17T03:59:41.001931+0000","last_became_peered":"2024-09-17T03:59:41.001931+0000","last_unstale":"2024-09-17T03:59:43.024744+0000","last_undegraded":"2024-09-17T03:59:43.024744+0000","last_fullsized":"2024-09-17T03:59:43.024744+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":149,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:59:39.969976+0000","last_clean_scrub_stamp":"2024-09-17T03:59:39.969976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:22:45.390790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":353,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-17T03:59:42.991235+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T03:59:42.991235+0000","last_peered":"2024-09-17T03:59:42.991235+0000","last_clean":"2024-09-17T03:59:42.991235+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T03:59:42.991235+0000","last_undegraded":"2024-09-17T03:59:42.991235+0000","last_fullsized":"2024-09-17T03:59:42.991235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509566,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37812,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334028,"statfs":{"total":96636764160,"available":96598044672,"internally_reserved":0,"allocated":2519040,"data_stored":1423848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509569,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":1908736,"data_stored":831730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509568,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41380,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330460,"statfs":{"total":96636764160,"available":96594391040,"internally_reserved":0,"allocated":1916928,"data_stored":832808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509566,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1904640,"data_stored":831160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509568,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1904640,"data_stored":831216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509568,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37208,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334632,"statfs":{"total":96636764160,"available":96598663168,"internally_reserved":0,"allocated":1904640,"data_stored":831160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542270,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41372,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330468,"statfs":{"total":96636764160,"available":96594399232,"internally_reserved":0,"allocated":1904640,"data_stored":831148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542271,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2436,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2494464,"data_stored":1421494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T03:59:46.896 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T03:59:46.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T03:59:46.955 INFO:tasks.ceph.mds.d.smithi181.stderr:2024-09-17T03:59:46.955+0000 7f8b85800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T03:59:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 -- 172.21.15.129:0/350869040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32380b3c90 msgr2=0x7f323806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 --2- 172.21.15.129:0/350869040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32380b3c90 0x7f323806eee0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f32380712a0 tx=0x7f323002aa30 comp rx=0 tx=0).stop 2024-09-17T03:59:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 -- 172.21.15.129:0/350869040 shutdown_connections 2024-09-17T03:59:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 --2- 172.21.15.129:0/350869040 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32380b3c90 0x7f323806eee0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 --2- 172.21.15.129:0/350869040 >> v2:172.21.15.181:3300/0 conn(0x7f323815d840 0x7f32380b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 --2- 172.21.15.129:0/350869040 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 0x7f32380703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 -- 172.21.15.129:0/350869040 >> 172.21.15.129:0/350869040 conn(0x7f323806d7a0 msgr2=0x7f32380b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:47.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 -- 172.21.15.129:0/350869040 shutdown_connections 2024-09-17T03:59:47.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.159+0000 7f323e200640 1 -- 172.21.15.129:0/350869040 wait complete. 2024-09-17T03:59:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.161+0000 7f323e200640 1 Processor -- start 2024-09-17T03:59:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.161+0000 7f323e200640 1 -- start start 2024-09-17T03:59:47.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32380b3c90 0x7f3238148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 0x7f323814b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32380b2b30 con 0x7f323815d840 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f32380ad550 con 0x7f32380b3c90 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f323e200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f32380b2cb0 con 0x7f32380b4d70 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41324/0 (socket says 172.21.15.129:41324) 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 -- 172.21.15.129:0/562332938 learned_addr learned my addr 172.21.15.129:0/562332938 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237400640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32380b3c90 0x7f3238148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f322ea00640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 0x7f323814b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 -- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 msgr2=0x7f323814b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 0x7f323814b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 -- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32380b3c90 msgr2=0x7f3238148b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32380b3c90 0x7f3238148b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.163+0000 7f3237e00640 1 -- 172.21.15.129:0/562332938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3238149050 con 0x7f323815d840 2024-09-17T03:59:47.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f3237e00640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3230005fd0 tx=0x7f3230003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:47.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230033050 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3230004450 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3238149420 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230004840 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.164+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32381499e0 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.165+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32380703b0 con 0x7f323815d840 2024-09-17T03:59:47.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.166+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3230036070 con 0x7f323815d840 2024-09-17T03:59:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.167+0000 7f322fe00640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 0x7f321407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.167+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f32300bc2b0 con 0x7f323815d840 2024-09-17T03:59:47.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.168+0000 7f3237400640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 0x7f321407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.168+0000 7f3237400640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 0x7f321407b0d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f322800daa0 tx=0x7f3228005d70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:47.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.172+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3230084650 con 0x7f323815d840 2024-09-17T03:59:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.428+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f323815c710 con 0x7f323815d840 2024-09-17T03:59:47.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.431+0000 7f322fe00640 1 -- 172.21.15.129:0/562332938 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v37) ==== 190+0+147 (secure 0 0 0) 0x7f3230083f50 con 0x7f323815d840 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T03:59:47.431 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T03:59:47.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 msgr2=0x7f321407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 0x7f321407b0d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f322800daa0 tx=0x7f3228005d70 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 msgr2=0x7f323809c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f3230005fd0 tx=0x7f3230003f10 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 shutdown_connections 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3214078bc0 0x7f321407b0d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f323815d840 0x7f323809c4b0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32380b4d70 0x7f323814b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 --2- 172.21.15.129:0/562332938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32380b3c90 0x7f3238148b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 >> 172.21.15.129:0/562332938 conn(0x7f323806d7a0 msgr2=0x7f32380b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 shutdown_connections 2024-09-17T03:59:47.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.434+0000 7f323e200640 1 -- 172.21.15.129:0/562332938 wait complete. 2024-09-17T03:59:47.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:47.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:59:47.620 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 -- 172.21.15.129:0/3332425403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d4dd0 msgr2=0x7f1fe00cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 --2- 172.21.15.129:0/3332425403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d4dd0 0x7f1fe00cf040 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc005fd0 tx=0x7f1fdc02a8b0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 -- 172.21.15.129:0/3332425403 shutdown_connections 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 --2- 172.21.15.129:0/3332425403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d4dd0 0x7f1fe00cf040 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 --2- 172.21.15.129:0/3332425403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 0x7f1fe006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 -- 172.21.15.129:0/3332425403 >> 172.21.15.129:0/3332425403 conn(0x7f1fe009da40 msgr2=0x7f1fe008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 -- 172.21.15.129:0/3332425403 shutdown_connections 2024-09-17T03:59:47.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.680+0000 7f1fe8800640 1 -- 172.21.15.129:0/3332425403 wait complete. 2024-09-17T03:59:47.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.682+0000 7f1fe8800640 1 Processor -- start 2024-09-17T03:59:47.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.683+0000 7f1fe8800640 1 -- start start 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe00b97d0 0x7f1fe01f5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 0x7f1fe015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00cfb30 con 0x7f1fe00d46c0 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00cf9b0 con 0x7f1fe00b97d0 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe8800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe00cfcb0 con 0x7f1fe00d0c20 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41334/0 (socket says 172.21.15.129:41334) 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe6800640 1 -- 172.21.15.129:0/2127401709 learned_addr learned my addr 172.21.15.129:0/2127401709 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.684+0000 7f1fe5400640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 0x7f1fe015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe5e00640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe00b97d0 0x7f1fe01f5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 -- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 msgr2=0x7f1fe015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 0x7f1fe015eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 -- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe00b97d0 msgr2=0x7f1fe01f5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe00b97d0 0x7f1fe01f5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 -- 172.21.15.129:0/2127401709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe00a1ed0 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fe6800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc006010 tx=0x7f1fdc031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.685+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc003e00 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.686+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fdc031a80 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.686+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe00d5010 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.686+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0330d0 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.686+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe009ec10 con 0x7f1fe00d46c0 2024-09-17T03:59:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.687+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe009ede0 con 0x7f1fe00d46c0 2024-09-17T03:59:47.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.688+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1fdc033270 con 0x7f1fe00d46c0 2024-09-17T03:59:47.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.689+0000 7f1fda000640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 0x7f1fc407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:47.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.689+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f1fdc0bc160 con 0x7f1fe00d46c0 2024-09-17T03:59:47.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.689+0000 7f1fe5e00640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 0x7f1fc407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:47.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.690+0000 7f1fe5e00640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 0x7f1fc407b0a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00d5750 tx=0x7f1fd4005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:47.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.693+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1fe009ede0 con 0x7f1fe00d46c0 2024-09-17T03:59:47.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.912+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1fe00d0160 con 0x7f1fe00d46c0 2024-09-17T03:59:47.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.913+0000 7f1fda000640 1 -- 172.21.15.129:0/2127401709 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f1fe00d5010 con 0x7f1fe00d46c0 2024-09-17T03:59:47.912 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:47.912 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:47.912 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 msgr2=0x7f1fc407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 0x7f1fc407b0a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00d5750 tx=0x7f1fd4005f30 comp rx=0 tx=0).stop 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 msgr2=0x7f1fe00d4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc006010 tx=0x7f1fdc031650 comp rx=0 tx=0).stop 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 shutdown_connections 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fc4078b90 0x7f1fc407b0a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1fe00d46c0 0x7f1fe00d4ad0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe00d0c20 0x7f1fe015eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.915+0000 7f1fe8800640 1 --2- 172.21.15.129:0/2127401709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe00b97d0 0x7f1fe01f5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.916+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 >> 172.21.15.129:0/2127401709 conn(0x7f1fe009da40 msgr2=0x7f1fe00b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.916+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 shutdown_connections 2024-09-17T03:59:47.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:47.916+0000 7f1fe8800640 1 -- 172.21.15.129:0/2127401709 wait complete. 2024-09-17T03:59:48.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 135, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T03:59:41.482169+0000', 'modified': '2024-09-17T03:59:44.493748+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8185}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8185': {'gid': 8185, 'name': 'd', 'rank': 0, 'incarnation': 131, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2593020253', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2593020253}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2593020253}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8185, 'qdb_cluster': [8185]} 2024-09-17T03:59:48.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T03:59:48.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.565+0000 7f391ac00640 1 -- 172.21.15.129:0/1850376402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140cd390 msgr2=0x7f39141609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.565+0000 7f391ac00640 1 --2- 172.21.15.129:0/1850376402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140cd390 0x7f39141609c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f390c005fd0 tx=0x7f390c02aaa0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 -- 172.21.15.129:0/1850376402 shutdown_connections 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 --2- 172.21.15.129:0/1850376402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140cd390 0x7f39141609c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 --2- 172.21.15.129:0/1850376402 >> v2:172.21.15.181:3300/0 conn(0x7f39140705a0 0x7f39140b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 -- 172.21.15.129:0/1850376402 >> 172.21.15.129:0/1850376402 conn(0x7f391406d410 msgr2=0x7f39140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 -- 172.21.15.129:0/1850376402 shutdown_connections 2024-09-17T03:59:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.566+0000 7f391ac00640 1 -- 172.21.15.129:0/1850376402 wait complete. 2024-09-17T03:59:48.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.568+0000 7f391ac00640 1 Processor -- start 2024-09-17T03:59:48.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.568+0000 7f391ac00640 1 -- start start 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 0x7f39140c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39140adf90 0x7f39140ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3914160f00 con 0x7f391406eed0 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f391406f4f0 con 0x7f39140adf90 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f391ac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3914161080 con 0x7f39140705a0 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41336/0 (socket says 172.21.15.129:41336) 2024-09-17T03:59:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 -- 172.21.15.129:0/2505986549 learned_addr learned my addr 172.21.15.129:0/2505986549 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3918c00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39140adf90 0x7f39140ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f390b400640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 0x7f39140c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 -- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 msgr2=0x7f39140c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 0x7f39140c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 -- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39140adf90 msgr2=0x7f39140ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39140adf90 0x7f39140ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f3913e00640 1 -- 172.21.15.129:0/2505986549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39141fa010 con 0x7f391406eed0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.570+0000 7f390b400640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 0x7f39140c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f3913e00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f390400e790 tx=0x7f390400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f3910c00640 1 -- 172.21.15.129:0/2505986549 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390400eea0 con 0x7f391406eed0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39140ae8e0 con 0x7f391406eed0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f3910c00640 1 -- 172.21.15.129:0/2505986549 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f390400cdb0 con 0x7f391406eed0 2024-09-17T03:59:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f3910c00640 1 -- 172.21.15.129:0/2505986549 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904010690 con 0x7f391406eed0 2024-09-17T03:59:48.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.571+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39140b1be0 con 0x7f391406eed0 2024-09-17T03:59:48.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.574+0000 7f3910c00640 1 -- 172.21.15.129:0/2505986549 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3904010830 con 0x7f391406eed0 2024-09-17T03:59:48.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.575+0000 7f3910c00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.575+0000 7f3910c00640 1 -- 172.21.15.129:0/2505986549 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f390409b770 con 0x7f391406eed0 2024-09-17T03:59:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.576+0000 7f3921c66740 1 Processor -- start 2024-09-17T03:59:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.576+0000 7f3921c66740 1 -- start start 2024-09-17T03:59:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.576+0000 7f390b400640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.576+0000 7f390b400640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 0x7f38f007b0d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f391415f5d0 tx=0x7f38fc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b3eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 0x560bd7b0ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560bd7a98430 con 0x560bd776e270 2024-09-17T03:59:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560bd7a885d0 con 0x560bd77453b0 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x560bd7a77a00 con 0x560bd774d590 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3913e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3913e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41352/0 (socket says 172.21.15.129:41352) 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.577+0000 7f3913e00640 1 -- 172.21.15.129:0/3147871082 learned_addr learned my addr 172.21.15.129:0/3147871082 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 -- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 msgr2=0x560bd7b0ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3918c00640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 0x560bd7b0ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f390b400640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b3eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 0x560bd7b0ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 -- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 msgr2=0x560bd7b3eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b3eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 -- 172.21.15.129:0/3147871082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd7a40520 con 0x560bd776e270 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3918c00640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 0x560bd7b0ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f390b400640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b3eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3913e00640 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f39040631d0 tx=0x7f3904062c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f001792aa1718e1b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3903e00640 1 -- 172.21.15.129:0/3147871082 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904067f00 con 0x560bd776e270 2024-09-17T03:59:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3903e00640 1 -- 172.21.15.129:0/3147871082 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f390406cdb0 con 0x560bd776e270 2024-09-17T03:59:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.578+0000 7f3903e00640 1 -- 172.21.15.129:0/3147871082 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3904062970 con 0x560bd776e270 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 -- 172.21.15.129:0/3147871082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 msgr2=0x560bd7b3e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f39040631d0 tx=0x7f3904062c40 comp rx=0 tx=0).stop 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 -- 172.21.15.129:0/3147871082 shutdown_connections 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd774d590 0x560bd7b0ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b3eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 --2- 172.21.15.129:0/3147871082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd776e270 0x560bd7b3e970 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 -- 172.21.15.129:0/3147871082 >> 172.21.15.129:0/3147871082 conn(0x560bd77623c0 msgr2=0x560bd7b14970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 -- 172.21.15.129:0/3147871082 shutdown_connections 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.579+0000 7f3921c66740 1 -- 172.21.15.129:0/3147871082 wait complete. 2024-09-17T03:59:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.580+0000 7f3921c66740 1 -- start start 2024-09-17T03:59:48.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.580+0000 7f3921c66740 1 Processor -- start 2024-09-17T03:59:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b6c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7b68f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560bd7a1e330 con 0x560bd774d590 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560bd7a7b810 con 0x560bd776e270 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x560bd77e8620 con 0x560bd77453b0 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3921c66740 10 client.0 _ll_register_callbacks cb 0x560bd7a23420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41368/0 (socket says 172.21.15.129:41368) 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 -- 172.21.15.129:0/114574062 learned_addr learned my addr 172.21.15.129:0/114574062 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3913e00640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b6c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f3918c00640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7b68f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 -- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 msgr2=0x560bd7b6c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b6c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 -- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 msgr2=0x560bd7b68f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.582+0000 7f390b400640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7b68f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f390b400640 1 -- 172.21.15.129:0/114574062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd77d9850 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3918c00640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7b68f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f390b400640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f38fc001e40 tx=0x7f38fc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a43c723467f175b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3900c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc003b00 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f38fc003ca0 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390406cdb0 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f38fc022430 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3921c66740 10 client.8865 fetch_fsmap learned FSMap version 135 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.583+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) -- 0x560bd78506b0 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f390406cdb0 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 135) ==== 3502+0+0 (secure 0 0 0) 0x7f38fc0225d0 con 0x560bd774d590 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 10 client.8865 fetch_fsmap finished waiting for FSMap version 135 2024-09-17T03:59:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 10 client.8865 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T03:59:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 20 client.8865 populate_metadata read hostname 'smithi129' 2024-09-17T03:59:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd772ce40 0x560bd7b25bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 4 client.8865 mds_command: new command op to 8185 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T03:59:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 --> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bd7a8a520 con 0x560bd772ce40 2024-09-17T03:59:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.584+0000 7f3913e00640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd772ce40 0x560bd7b25bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.585+0000 7f3900c00640 10 client.8865 ms_handle_connect on v2:172.21.15.181:6835/2593020253 2024-09-17T03:59:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.585+0000 7f3913e00640 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd772ce40 0x560bd7b25bc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99976067dc197f27 server_cookie=413d9750603cf6a4 in_seq=0 out_seq=0 2024-09-17T03:59:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.586+0000 7f3900c00640 1 -- 172.21.15.129:0/114574062 <== mds.0 v2:172.21.15.181:6835/2593020253 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x560bd7a8a520 con 0x560bd772ce40 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.586+0000 7f3900c00640 10 client.8865 handle_command_reply: tid=0 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.586+0000 7f3921c66740 1 client.8865 shutdown 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 msgr2=0x560bd7b6c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f38fc001e40 tx=0x7f38fc000bc0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 shutdown_connections 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd772ce40 0x560bd7b25bc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7b68f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b6c800 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 --2- 172.21.15.129:0/114574062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b6c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3900c00640 0 client.8865 ms_handle_reset on v2:172.21.15.181:6835/2593020253 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 >> 172.21.15.129:0/114574062 conn(0x560bd77623c0 msgr2=0x560bd7b4ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 shutdown_connections 2024-09-17T03:59:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 1 -- 172.21.15.129:0/114574062 wait complete. 2024-09-17T03:59:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.587+0000 7f3921c66740 20 client.8865 trim_cache size 0 max 16384 2024-09-17T03:59:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.606+0000 7f3921c66740 1 Processor -- start 2024-09-17T03:59:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.606+0000 7f3921c66740 1 -- start start 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b0fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd776e270 0x560bd7b38a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3914201450 con 0x560bd774d590 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f391415d9d0 con 0x560bd77453b0 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f391415e020 con 0x560bd776e270 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.607+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41372/0 (socket says 172.21.15.129:41372) 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 -- 172.21.15.129:0/595064273 learned_addr learned my addr 172.21.15.129:0/595064273 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f3918c00640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd776e270 0x560bd7b38a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f3913e00640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b0fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 -- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd776e270 msgr2=0x560bd7b38a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd776e270 0x560bd7b38a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 -- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 msgr2=0x560bd7b0fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b0fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 -- 172.21.15.129:0/595064273 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bd7b15650 con 0x560bd774d590 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f3913e00640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b0fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f390b400640 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e600 tx=0x7f38fc00e630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2077c108aac051 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.608+0000 7f3903400640 1 -- 172.21.15.129:0/595064273 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0225d0 con 0x560bd774d590 2024-09-17T03:59:48.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3903400640 1 -- 172.21.15.129:0/595064273 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38fc022770 con 0x560bd774d590 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3903400640 1 -- 172.21.15.129:0/595064273 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc003b00 con 0x560bd774d590 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 -- 172.21.15.129:0/595064273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 msgr2=0x560bd7b101b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00e600 tx=0x7f38fc00e630 comp rx=0 tx=0).stop 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 -- 172.21.15.129:0/595064273 shutdown_connections 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd776e270 0x560bd7b38a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b101b0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 --2- 172.21.15.129:0/595064273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd77453b0 0x560bd7b0fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 -- 172.21.15.129:0/595064273 >> 172.21.15.129:0/595064273 conn(0x560bd77623c0 msgr2=0x560bd7b68c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 -- 172.21.15.129:0/595064273 shutdown_connections 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.609+0000 7f3921c66740 1 -- 172.21.15.129:0/595064273 wait complete. 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.610+0000 7f3921c66740 1 -- start start 2024-09-17T03:59:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.611+0000 7f3921c66740 1 Processor -- start 2024-09-17T03:59:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b34700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7bd7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560bd7b69b20 con 0x560bd774d590 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560bd7b51d60 con 0x560bd776e270 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x560bd7b51090 con 0x560bd77453b0 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f3921c66740 10 client.0 _ll_register_callbacks cb 0x7f38fc005300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f390b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41384/0 (socket says 172.21.15.129:41384) 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.612+0000 7f390b400640 1 -- 172.21.15.129:0/3180892527 learned_addr learned my addr 172.21.15.129:0/3180892527 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3913e00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b34700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3918c00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7bd7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 -- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 msgr2=0x560bd7b34700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b34700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 -- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 msgr2=0x560bd7bd7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7bd7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 -- 172.21.15.129:0/3180892527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bd7b3a740 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3913e00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3918c00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7bd7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f390b400640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005040 tx=0x7f38fc003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a528d1b3a12eda3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3900c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc03a070 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f38fc02c840 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.613+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38fc022770 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=135) ==== 24+0+0 (secure 0 0 0) 0x7f38fc045050 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 10 client.8871 fetch_fsmap learned FSMap version 135 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=135,monmap=2+}) -- 0x560bd7b7bc10 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38fc022770 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 135) ==== 3502+0+0 (secure 0 0 0) 0x7f38fc048020 con 0x560bd774d590 2024-09-17T03:59:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 10 client.8871 fetch_fsmap finished waiting for FSMap version 135 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 10 client.8871 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 20 client.8871 populate_metadata read hostname 'smithi129' 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd7b6c250 0x560bd7b50560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 4 client.8871 mds_command: new command op to 8185 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 --> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] -- command(tid 0: {"prefix": "status"}) -- 0x560bd7a85fd0 con 0x560bd7b6c250 2024-09-17T03:59:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.614+0000 7f3913e00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd7b6c250 0x560bd7b50560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.615+0000 7f3913e00640 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd7b6c250 0x560bd7b50560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16ff6343d59ed6e6 server_cookie=bca2c26dd090605 in_seq=0 out_seq=0 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.615+0000 7f3900c00640 10 client.8871 ms_handle_connect on v2:172.21.15.181:6835/2593020253 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.615+0000 7f3900c00640 1 -- 172.21.15.129:0/3180892527 <== mds.0 v2:172.21.15.181:6835/2593020253 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560bd7a85fd0 con 0x560bd7b6c250 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.615+0000 7f3900c00640 10 client.8871 handle_command_reply: tid=0 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.615+0000 7f3921c66740 1 client.8871 shutdown 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 msgr2=0x560bd7b34c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f38fc005040 tx=0x7f38fc003d90 comp rx=0 tx=0).stop 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 shutdown_connections 2024-09-17T03:59:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:6835/2593020253,v1:172.21.15.181:6837/2593020253] conn(0x560bd7b6c250 0x560bd7b50560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560bd776e270 0x560bd7bd7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560bd774d590 0x560bd7b34c40 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3900c00640 0 client.8871 ms_handle_reset on v2:172.21.15.181:6835/2593020253 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 --2- 172.21.15.129:0/3180892527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560bd77453b0 0x560bd7b34700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 >> 172.21.15.129:0/3180892527 conn(0x560bd77623c0 msgr2=0x560bd7b6cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 shutdown_connections 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 1 -- 172.21.15.129:0/3180892527 wait complete. 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f3921c66740 20 client.8871 trim_cache size 0 max 16384 2024-09-17T03:59:48.619 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 8185, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.616436363, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 135, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 152, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 152, 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.737640520999999 2024-09-17T03:59:48.620 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:48.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 msgr2=0x7f38f007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 0x7f38f007b0d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f391415f5d0 tx=0x7f38fc009040 comp rx=0 tx=0).stop 2024-09-17T03:59:48.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 msgr2=0x7f39141f9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.616+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f390400e790 tx=0x7f390400ec50 comp rx=0 tx=0).stop 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 shutdown_connections 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38f0078bc0 0x7f38f007b0d0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39140adf90 0x7f39140ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39140705a0 0x7f39140c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 --2- 172.21.15.129:0/2505986549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f391406eed0 0x7f39141f9ad0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 >> 172.21.15.129:0/2505986549 conn(0x7f391406d410 msgr2=0x7f39140746e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 shutdown_connections 2024-09-17T03:59:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.617+0000 7f391ac00640 1 -- 172.21.15.129:0/2505986549 wait complete. 2024-09-17T03:59:48.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T03:59:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 -- 172.21.15.129:0/2709967721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a81458a0 msgr2=0x7fe8a8160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/2709967721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a81458a0 0x7fe8a8160750 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0009830 tx=0x7fe8a002a480 comp rx=0 tx=0).stop 2024-09-17T03:59:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 -- 172.21.15.129:0/2709967721 shutdown_connections 2024-09-17T03:59:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/2709967721 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a81458a0 0x7fe8a8160750 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/2709967721 >> v2:172.21.15.181:3300/0 conn(0x7fe8a80d0a50 0x7fe8a8160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 -- 172.21.15.129:0/2709967721 >> 172.21.15.129:0/2709967721 conn(0x7fe8a806cfc0 msgr2=0x7fe8a806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 -- 172.21.15.129:0/2709967721 shutdown_connections 2024-09-17T03:59:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.859+0000 7fe8a7e00640 1 -- 172.21.15.129:0/2709967721 wait complete. 2024-09-17T03:59:48.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.861+0000 7fe8a7e00640 1 Processor -- start 2024-09-17T03:59:48.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.861+0000 7fe8a7e00640 1 -- start start 2024-09-17T03:59:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 0x7fe8a8141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a8145e60 0x7fe8a81e8610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a80cef20 con 0x7fe8a8074970 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a80ceda0 con 0x7fe8a81458a0 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a8160c90 con 0x7fe8a8145e60 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41400/0 (socket says 172.21.15.129:41400) 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7400640 1 -- 172.21.15.129:0/4122099886 learned_addr learned my addr 172.21.15.129:0/4122099886 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a6000640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a8145e60 0x7fe8a81e8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a6a00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 0x7fe8a8141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7400640 1 -- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a8145e60 msgr2=0x7fe8a81e8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.863+0000 7fe8a7400640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a8145e60 0x7fe8a81e8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7400640 1 -- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 msgr2=0x7fe8a8141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7400640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 0x7fe8a8141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7400640 1 -- 172.21.15.129:0/4122099886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a81f88e0 con 0x7fe8a8074970 2024-09-17T03:59:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a6a00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 0x7fe8a8141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:48.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7400640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe8a002aa70 tx=0x7fe8a0033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0031b50 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8a0031de0 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a81f8bc0 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a0005280 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.864+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a81f9180 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.865+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a8160210 con 0x7fe8a8074970 2024-09-17T03:59:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.866+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe8a0005420 con 0x7fe8a8074970 2024-09-17T03:59:48.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.867+0000 7fe89ea00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 0x7fe88807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:48.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.868+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fe8a00bc0e0 con 0x7fe8a8074970 2024-09-17T03:59:48.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.868+0000 7fe8a6a00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 0x7fe88807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:48.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.868+0000 7fe8a6a00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 0x7fe88807b0a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe8980015a0 tx=0x7fe898009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:48.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:48.872+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe8a003f030 con 0x7fe8a8074970 2024-09-17T03:59:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.095+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe8a806a140 con 0x7fe8a8074970 2024-09-17T03:59:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.096+0000 7fe89ea00640 1 -- 172.21.15.129:0/4122099886 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v135) ==== 53+0+73 (secure 0 0 0) 0x7fe8a0084510 con 0x7fe8a8074970 2024-09-17T03:59:49.095 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 msgr2=0x7fe88807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 0x7fe88807b0a0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe8980015a0 tx=0x7fe898009040 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 msgr2=0x7fe8a81e8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7fe8a002aa70 tx=0x7fe8a0033ba0 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 shutdown_connections 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe888078b90 0x7fe88807b0a0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe8a8074970 0x7fe8a81e8b50 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe8a8145e60 0x7fe8a81e8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 --2- 172.21.15.129:0/4122099886 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8a81458a0 0x7fe8a8141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 >> 172.21.15.129:0/4122099886 conn(0x7fe8a806cfc0 msgr2=0x7fe8a81e5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 shutdown_connections 2024-09-17T03:59:49.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49.098+0000 7fe8a7e00640 1 -- 172.21.15.129:0/4122099886 wait complete. 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T03:59:49.123 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:59:49.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.124 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T03:59:49.159 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:59:49.160 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42103sec preferred_lft 42103sec 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.161 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T03:59:49.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.163 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T03:59:49.253 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T03:59:49.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.256 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T03:59:49.292 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T03:59:49.293 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T03:59:49.293 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T03:59:49.293 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T03:59:49.293 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T03:59:49.293 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T03:59:49.293 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T03:59:49.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T03:59:49.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T03:59:49.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T03:59:49.294 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T03:59:49.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T03:59:49.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:59:49.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.490 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T03:59:49.525 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T03:59:49.525 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T03:59:49.526 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:59:49.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T03:59:49.622 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:59:49.622 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T03:59:49.658 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T03:59:49.658 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T03:59:49.658 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:59:49.732 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T03:59:49.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:59:49.925 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:59:49.926 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:59:49.926 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:59:49.946 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:59:49.946 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:59:49.946 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:59:49.946 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:59:49.946 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:59:49.976 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T03:59:49.976 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T03:59:49.977 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T03:59:49.978 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T03:59:49.978 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T03:59:49.978 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T03:59:49.978 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T03:59:49.978 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T03:59:49.981 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T03:59:49.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:49.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T03:59:50.041 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T03:59:50.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T03:59:50.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:59:50.143 INFO:teuthology.orchestra.run.smithi129.stdout:client8880 2024-09-17T03:59:50.144 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:50.144 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:50.179 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:50.179 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:50.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:50.255 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file_a bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T03:59:50.427 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T03:59:50.427 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T03:59:50.428 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0883146 s, 95.0 MB/s 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_a").st_ino) 2024-09-17T03:59:50.429 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:59:50.528 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627776 2024-09-17T03:59:50.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:59:50.540 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:50.540 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:50.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:50.618 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file_b bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T03:59:50.818 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T03:59:50.818 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T03:59:50.818 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.123099 s, 68.1 MB/s 2024-09-17T03:59:50.819 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T03:59:50.820 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T03:59:50.820 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T03:59:50.820 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T03:59:50.820 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_b").st_ino) 2024-09-17T03:59:50.820 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T03:59:50.918 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627777 2024-09-17T03:59:50.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T03:59:50.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 -- 172.21.15.129:0/1398899733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9710003c30 msgr2=0x7f9710004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 --2- 172.21.15.129:0/1398899733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9710003c30 0x7f9710004080 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f971c00c4a0 tx=0x7f9710005830 comp rx=0 tx=0).stop 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 -- 172.21.15.129:0/1398899733 shutdown_connections 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 --2- 172.21.15.129:0/1398899733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9710003c30 0x7f9710004080 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 -- 172.21.15.129:0/1398899733 >> 172.21.15.129:0/1398899733 conn(0x7f972806d410 msgr2=0x7f97280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:51.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 -- 172.21.15.129:0/1398899733 shutdown_connections 2024-09-17T03:59:51.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.192+0000 7f9730200640 1 -- 172.21.15.129:0/1398899733 wait complete. 2024-09-17T03:59:51.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.195+0000 7f9730200640 1 Processor -- start 2024-09-17T03:59:51.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.195+0000 7f9730200640 1 -- start start 2024-09-17T03:59:51.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 0x7f9728154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 0x7f97280b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97281600e0 con 0x7f97280cc610 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97280b51d0 con 0x7f97281f9630 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f9730200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9728160260 con 0x7f9710003c30 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f972e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f972e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41422/0 (socket says 172.21.15.129:41422) 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.196+0000 7f972e200640 1 -- 172.21.15.129:0/3416831069 learned_addr learned my addr 172.21.15.129:0/3416831069 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972d800640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 0x7f9728154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972ce00640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 0x7f97280b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 -- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 msgr2=0x7f9728154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 0x7f9728154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 -- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 msgr2=0x7f97280b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 0x7f97280b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 -- 172.21.15.129:0/3416831069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9728155130 con 0x7f97280cc610 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972ce00640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 0x7f97280b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972d800640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 0x7f9728154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f972e200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9728072640 tx=0x7f9720001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720011870 con 0x7f97280cc610 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9720017070 con 0x7f97280cc610 2024-09-17T03:59:51.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97281f8b50 con 0x7f97280cc610 2024-09-17T03:59:51.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720011d60 con 0x7f97280cc610 2024-09-17T03:59:51.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.197+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97281f9110 con 0x7f97280cc610 2024-09-17T03:59:51.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.199+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97280ccf90 con 0x7f97280cc610 2024-09-17T03:59:51.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.200+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f972001c460 con 0x7f97280cc610 2024-09-17T03:59:51.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.201+0000 7f9725600640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 0x7f970807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.201+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f972009b5b0 con 0x7f97280cc610 2024-09-17T03:59:51.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.201+0000 7f972d800640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 0x7f970807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.202+0000 7f972d800640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 0x7f970807b0a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f971c00c470 tx=0x7f971c02d070 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.205+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f97200639e0 con 0x7f97280cc610 2024-09-17T03:59:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.424+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f972806a140 con 0x7f97280cc610 2024-09-17T03:59:51.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.424+0000 7f9725600640 1 -- 172.21.15.129:0/3416831069 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f97200632e0 con 0x7f97280cc610 2024-09-17T03:59:51.423 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:51.423 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:51.424 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.426+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 msgr2=0x7f970807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.426+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 0x7f970807b0a0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f971c00c470 tx=0x7f971c02d070 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.426+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 msgr2=0x7f97280b1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.426+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f9728072640 tx=0x7f9720001b80 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 shutdown_connections 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9708078b90 0x7f970807b0a0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97280cc610 0x7f97280b1920 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97281f9630 0x7f97280b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 --2- 172.21.15.129:0/3416831069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9710003c30 0x7f9728154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 >> 172.21.15.129:0/3416831069 conn(0x7f972806d410 msgr2=0x7f9728072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:51.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 shutdown_connections 2024-09-17T03:59:51.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.427+0000 7f9730200640 1 -- 172.21.15.129:0/3416831069 wait complete. 2024-09-17T03:59:51.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-09-17T03:59:51.703 INFO:teuthology.orchestra.run.smithi181.stdout:{"message":"","return_code":0} 2024-09-17T03:59:51.703 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:59:51.703 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000000.00000000 parent 2024-09-17T03:59:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.846+0000 7f0037729f40 1 Processor -- start 2024-09-17T03:59:51.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.846+0000 7f0037729f40 1 -- start start 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feae85640 0x562feae7b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feaef6150 0x562feae84370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562feade91d0 con 0x562feae87610 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562feadf5f90 con 0x562feaef6150 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0037729f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x562fead8ac60 con 0x562feae85640 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41424/0 (socket says 172.21.15.129:41424) 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 -- 172.21.15.129:0/2858156122 learned_addr learned my addr 172.21.15.129:0/2858156122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0030800640 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feae85640 0x562feae7b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f0031200640 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feaef6150 0x562feae84370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 -- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feae85640 msgr2=0x562feae7b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feae85640 0x562feae7b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 -- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feaef6150 msgr2=0x562feae84370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feaef6150 0x562feae84370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.848+0000 7f002fe00640 1 -- 172.21.15.129:0/2858156122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562feaf0f450 con 0x562feae87610 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f002fe00640 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d70 tx=0x7f001c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670634c1f4cc3800 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f002ea00640 1 -- 172.21.15.129:0/2858156122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c00bd60 con 0x562feae87610 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f002ea00640 1 -- 172.21.15.129:0/2858156122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f001c00f400 con 0x562feae87610 2024-09-17T03:59:51.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f002ea00640 1 -- 172.21.15.129:0/2858156122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f001c00f850 con 0x562feae87610 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f0037729f40 1 -- 172.21.15.129:0/2858156122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 msgr2=0x562feaf9e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f0037729f40 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f001c001d70 tx=0x7f001c00b5b0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.849+0000 7f0037729f40 1 -- 172.21.15.129:0/2858156122 shutdown_connections 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feaef6150 0x562feae84370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae87610 0x562feaf9e660 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 --2- 172.21.15.129:0/2858156122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feae85640 0x562feae7b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 -- 172.21.15.129:0/2858156122 >> 172.21.15.129:0/2858156122 conn(0x562fead5bac0 msgr2=0x562feaef4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 -- 172.21.15.129:0/2858156122 shutdown_connections 2024-09-17T03:59:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.850+0000 7f0037729f40 1 -- 172.21.15.129:0/2858156122 wait complete. 2024-09-17T03:59:51.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.851+0000 7f0037729f40 1 Processor -- start 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.851+0000 7f0037729f40 1 -- start start 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 0x562feaef4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feae87610 0x562feaef6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feaef6150 0x562feaef7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x562feaf0f780 con 0x562feae85640 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x562feaef1150 con 0x562feae87610 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0037729f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x562feaef2600 con 0x562feaef6150 2024-09-17T03:59:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0030800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 0x562feaef4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0030800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 0x562feaef4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41438/0 (socket says 172.21.15.129:41438) 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0030800640 1 -- 172.21.15.129:0/2535535519 learned_addr learned my addr 172.21.15.129:0/2535535519 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f002fe00640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feae87610 0x562feaef6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.852+0000 7f0031200640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feaef6150 0x562feaef7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 -- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feaef6150 msgr2=0x562feaef7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x562feaef6150 0x562feaef7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 -- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feae87610 msgr2=0x562feaef6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562feae87610 0x562feaef6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 -- 172.21.15.129:0/2535535519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562feaef2850 con 0x562feae85640 2024-09-17T03:59:51.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0030800640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 0x562feaef4940 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f00240464a0 tx=0x7f002404eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f002d600640 1 -- 172.21.15.129:0/2535535519 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024066030 con 0x562feae85640 2024-09-17T03:59:51.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f002d600640 1 -- 172.21.15.129:0/2535535519 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0024056470 con 0x562feae85640 2024-09-17T03:59:51.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562feaef2b30 con 0x562feae85640 2024-09-17T03:59:51.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.853+0000 7f002d600640 1 -- 172.21.15.129:0/2535535519 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240566a0 con 0x562feae85640 2024-09-17T03:59:51.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.854+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562feaeed980 con 0x562feae85640 2024-09-17T03:59:51.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.856+0000 7f002d600640 1 -- 172.21.15.129:0/2535535519 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0024069070 con 0x562feae85640 2024-09-17T03:59:51.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f002d600640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010077570 0x7f0010079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f002d600640 1 -- 172.21.15.129:0/2535535519 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f00240dd760 con 0x562feae85640 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f002fe00640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010077570 0x7f0010079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f0037729f40 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x562feaeed310 0x562feaeeecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 23.30 23:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152) -- 0x562feaef4e80 con 0x562feaeed310 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.857+0000 7f0031200640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x562feaeed310 0x562feaeeecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.858+0000 7f002fe00640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010077570 0x7f0010079a80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x562feaf0d7a0 tx=0x7f001c00e000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.858+0000 7f0031200640 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x562feaeed310 0x562feaeeecc0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.858+0000 7f0031200640 1 -- 172.21.15.129:0/2535535519 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (62) out=62b] v0'0 uv3 ondisk = 0) ==== 164+0+62 (crc 0 0 0) 0x7f0028006820 con 0x562feaeed310 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.858+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x562feaeed310 msgr2=0x562feaeeecc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x562feaeed310 0x562feaeeecc0 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010077570 msgr2=0x7f0010079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010077570 0x7f0010079a80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x562feaf0d7a0 tx=0x7f001c00e000 comp rx=0 tx=0).stop 2024-09-17T03:59:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 msgr2=0x562feaef4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 --2- 172.21.15.129:0/2535535519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x562feae85640 0x562feaef4940 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f00240464a0 tx=0x7f002404eca0 comp rx=0 tx=0).stop 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0031200640 1 -- 172.21.15.129:0/2535535519 reap_dead start 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 shutdown_connections 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 >> 172.21.15.129:0/2535535519 conn(0x562fead5bac0 msgr2=0x562feae85a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 shutdown_connections 2024-09-17T03:59:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:51.859+0000 7f0037729f40 1 -- 172.21.15.129:0/2535535519 wait complete. 2024-09-17T03:59:51.870 INFO:teuthology.orchestra.run.smithi129.stdout:8file_a 2024-09-17T03:59:51.871 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:59:52.010 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:52.010 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627776, 2024-09-17T03:59:52.010 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:59:52.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_a", 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 6 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 23, 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:52.011 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:52.022 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T03:59:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.165+0000 7fec0ede3f40 1 Processor -- start 2024-09-17T03:59:52.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.165+0000 7fec0ede3f40 1 -- start start 2024-09-17T03:59:52.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e3245f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e2aa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55745e1b2c60 con 0x55745e2a8110 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55745e2111d0 con 0x55745e2a8520 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.166+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55745e264680 con 0x55745e337fa0 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41462/0 (socket says 172.21.15.129:41462) 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 -- 172.21.15.129:0/3206629455 learned_addr learned my addr 172.21.15.129:0/3206629455 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec07600640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e3245f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08a00640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e2aa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 -- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 msgr2=0x55745e2aa6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e2aa6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 -- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 msgr2=0x55745e3245f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e3245f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 -- 172.21.15.129:0/3206629455 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745e291080 con 0x55745e2a8110 2024-09-17T03:59:52.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08a00640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e2aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec07600640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e3245f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.167+0000 7fec08000640 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7febf8025de0 tx=0x7febf8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1d6f37ba6e9d5d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec06200640 1 -- 172.21.15.129:0/3206629455 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf804dcd0 con 0x55745e2a8110 2024-09-17T03:59:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec06200640 1 -- 172.21.15.129:0/3206629455 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febf8056470 con 0x55745e2a8110 2024-09-17T03:59:52.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec06200640 1 -- 172.21.15.129:0/3206629455 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8056880 con 0x55745e2a8110 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 -- 172.21.15.129:0/3206629455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 msgr2=0x55745e290b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7febf8025de0 tx=0x7febf8002fb0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 -- 172.21.15.129:0/3206629455 shutdown_connections 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e2aa6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e3245f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/3206629455 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e290b40 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 -- 172.21.15.129:0/3206629455 >> 172.21.15.129:0/3206629455 conn(0x55745e183ac0 msgr2=0x55745e330730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 -- 172.21.15.129:0/3206629455 shutdown_connections 2024-09-17T03:59:52.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.168+0000 7fec0ede3f40 1 -- 172.21.15.129:0/3206629455 wait complete. 2024-09-17T03:59:52.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.169+0000 7fec0ede3f40 1 Processor -- start 2024-09-17T03:59:52.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.169+0000 7fec0ede3f40 1 -- start start 2024-09-17T03:59:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e323650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e33be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e33c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55745e2a3510 con 0x55745e2a8110 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55745e32ea60 con 0x55745e2a8520 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec0ede3f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55745e33ee50 con 0x55745e337fa0 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec08000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e323650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec08000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e323650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41470/0 (socket says 172.21.15.129:41470) 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec08000640 1 -- 172.21.15.129:0/1250040885 learned_addr learned my addr 172.21.15.129:0/1250040885 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec08a00640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e33c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.171+0000 7fec07600640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e33be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 -- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 msgr2=0x55745e33c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55745e337fa0 0x55745e33c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 -- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 msgr2=0x55745e33be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55745e2a8520 0x55745e33be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 -- 172.21.15.129:0/1250040885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55745e33b130 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec08000640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e323650 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7febf80464e0 tx=0x7febf80041d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec04e00640 1 -- 172.21.15.129:0/1250040885 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf806f020 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec04e00640 1 -- 172.21.15.129:0/1250040885 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7febf80577b0 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec04e00640 1 -- 172.21.15.129:0/1250040885 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf8057c00 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55745e33b410 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.172+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55745e264f00 con 0x55745e2a8110 2024-09-17T03:59:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.175+0000 7fec04e00640 1 -- 172.21.15.129:0/1250040885 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7febf8072020 con 0x55745e2a8110 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.175+0000 7fec04e00640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febe80775a0 0x7febe8079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.176+0000 7fec04e00640 1 -- 172.21.15.129:0/1250040885 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7febf80df470 con 0x55745e2a8110 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.176+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55745e336a30 0x55745e3ddc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.176+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 23.7 23:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152) -- 0x55745e2939e0 con 0x55745e336a30 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.176+0000 7fec07600640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febe80775a0 0x7febe8079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.176+0000 7fec08a00640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55745e336a30 0x55745e3ddc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec07600640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febe80775a0 0x7febe8079ab0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x55745e2a97e0 tx=0x7febf4003760 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec08a00640 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55745e336a30 0x55745e3ddc90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec08a00640 1 -- 172.21.15.129:0/1250040885 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (62) out=62b] v0'0 uv2 ondisk = 0) ==== 164+0+62 (crc 0 0 0) 0x7fec0000b040 con 0x55745e336a30 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55745e336a30 msgr2=0x55745e3ddc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55745e336a30 0x55745e3ddc90 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febe80775a0 msgr2=0x7febe8079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7febe80775a0 0x7febe8079ab0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x55745e2a97e0 tx=0x7febf4003760 comp rx=0 tx=0).stop 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 msgr2=0x55745e323650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 --2- 172.21.15.129:0/1250040885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55745e2a8110 0x55745e323650 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7febf80464e0 tx=0x7febf80041d0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec08a00640 1 -- 172.21.15.129:0/1250040885 reap_dead start 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 shutdown_connections 2024-09-17T03:59:52.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.177+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 >> 172.21.15.129:0/1250040885 conn(0x55745e183ac0 msgr2=0x55745e2aa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:52.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.178+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 shutdown_connections 2024-09-17T03:59:52.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.178+0000 7fec0ede3f40 1 -- 172.21.15.129:0/1250040885 wait complete. 2024-09-17T03:59:52.191 INFO:teuthology.orchestra.run.smithi129.stdout:8file_b  2024-09-17T03:59:52.191 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:59:52.332 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:52.332 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T03:59:52.332 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_b", 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 12 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 23, 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:52.333 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:52.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:52.344 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv file_a file_b) 2024-09-17T03:59:52.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:52.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 -- 172.21.15.129:0/3433667201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35540cc910 msgr2=0x7f35540ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 --2- 172.21.15.129:0/3433667201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35540cc910 0x7f35540ccd20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f35540cc390 tx=0x7f354c02a520 comp rx=0 tx=0).stop 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 -- 172.21.15.129:0/3433667201 shutdown_connections 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 --2- 172.21.15.129:0/3433667201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f35540cc910 0x7f35540ccd20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 --2- 172.21.15.129:0/3433667201 >> v2:172.21.15.129:3300/0 conn(0x7f35540cc3e0 0x7f355415f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 -- 172.21.15.129:0/3433667201 >> 172.21.15.129:0/3433667201 conn(0x7f355406d410 msgr2=0x7f35540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 -- 172.21.15.129:0/3433667201 shutdown_connections 2024-09-17T03:59:52.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.646+0000 7f3558a00640 1 -- 172.21.15.129:0/3433667201 wait complete. 2024-09-17T03:59:52.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.649+0000 7f3558a00640 1 Processor -- start 2024-09-17T03:59:52.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.649+0000 7f3558a00640 1 -- start start 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3554070a50 0x7f35541f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35540cc910 0x7f355415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f355415f720 con 0x7f35540cc3e0 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f355415f5a0 con 0x7f3554070a50 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3558a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f35540af070 con 0x7f35540cc910 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3549400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3549400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41490/0 (socket says 172.21.15.129:41490) 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3549400640 1 -- 172.21.15.129:0/2980799832 learned_addr learned my addr 172.21.15.129:0/2980799832 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.650+0000 7f3551e00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3554070a50 0x7f35541f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3552800640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35540cc910 0x7f355415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 -- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35540cc910 msgr2=0x7f355415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35540cc910 0x7f355415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 -- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3554070a50 msgr2=0x7f35541f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3554070a50 0x7f35541f2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 -- 172.21.15.129:0/2980799832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3554074a60 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f3549400640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f353c00e790 tx=0x7f353c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.651+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c00eea0 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.652+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3554074d40 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.652+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f353c00cdb0 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.652+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353c010690 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.652+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35541f5960 con 0x7f35540cc3e0 2024-09-17T03:59:52.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.653+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355415f060 con 0x7f35540cc3e0 2024-09-17T03:59:52.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.654+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f353c010830 con 0x7f35540cc3e0 2024-09-17T03:59:52.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.656+0000 7f354aa00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 0x7f353007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:52.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.656+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f353c09b770 con 0x7f35540cc3e0 2024-09-17T03:59:52.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.656+0000 7f3551e00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 0x7f353007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:52.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.657+0000 7f3551e00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 0x7f353007b0a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f35440017a0 tx=0x7f3544001730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:52.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.660+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f353c0634d0 con 0x7f35540cc3e0 2024-09-17T03:59:52.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.879+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f355406a140 con 0x7f35540cc3e0 2024-09-17T03:59:52.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.879+0000 7f354aa00640 1 -- 172.21.15.129:0/2980799832 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f353c067f00 con 0x7f35540cc3e0 2024-09-17T03:59:52.879 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:52.879 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:52.880 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:52.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 msgr2=0x7f353007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 0x7f353007b0a0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f35440017a0 tx=0x7f3544001730 comp rx=0 tx=0).stop 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 msgr2=0x7f355415e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f353c00e790 tx=0x7f353c00ec50 comp rx=0 tx=0).stop 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 shutdown_connections 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3530078b90 0x7f353007b0a0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35540cc910 0x7f355415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35540cc3e0 0x7f355415e3f0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 --2- 172.21.15.129:0/2980799832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3554070a50 0x7f35541f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:52.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 >> 172.21.15.129:0/2980799832 conn(0x7f355406d410 msgr2=0x7f35540d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:52.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 shutdown_connections 2024-09-17T03:59:52.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:52.882+0000 7f3558a00640 1 -- 172.21.15.129:0/2980799832 wait complete. 2024-09-17T03:59:52.905 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-09-17T03:59:53.148 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T03:59:53.148 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T03:59:53.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:53.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 -- 172.21.15.129:0/1535472557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 msgr2=0x7fbdf815f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 --2- 172.21.15.129:0/1535472557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf815f060 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009830 tx=0x7fbde402a520 comp rx=0 tx=0).stop 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 -- 172.21.15.129:0/1535472557 shutdown_connections 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 --2- 172.21.15.129:0/1535472557 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 0x7fbdf80ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 --2- 172.21.15.129:0/1535472557 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf815f060 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 -- 172.21.15.129:0/1535472557 >> 172.21.15.129:0/1535472557 conn(0x7fbdf806d410 msgr2=0x7fbdf80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 -- 172.21.15.129:0/1535472557 shutdown_connections 2024-09-17T03:59:53.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.371+0000 7fbe00400640 1 -- 172.21.15.129:0/1535472557 wait complete. 2024-09-17T03:59:53.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.373+0000 7fbe00400640 1 Processor -- start 2024-09-17T03:59:53.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.374+0000 7fbe00400640 1 -- start start 2024-09-17T03:59:53.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 0x7fbdf80b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 0x7fbdf8159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf815f720 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf815f5a0 con 0x7fbdf8070a50 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbe00400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbdf80af070 con 0x7fbdf80cc910 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbdfd000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbdfd000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41496/0 (socket says 172.21.15.129:41496) 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbdfd000640 1 -- 172.21.15.129:0/1856240351 learned_addr learned my addr 172.21.15.129:0/1856240351 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.375+0000 7fbdfe400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 0x7fbdf8159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfda00640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 0x7fbdf80b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:53.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 -- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 msgr2=0x7fbdf8159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 0x7fbdf8159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 -- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 msgr2=0x7fbdf80b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 0x7fbdf80b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 -- 172.21.15.129:0/1856240351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf815d520 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfda00640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 0x7fbdf80b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:53.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdfd000640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009950 tx=0x7fbde4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4031d60 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbde4033e10 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf815d800 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4033310 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.376+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf815a3a0 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.377+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf80b06f0 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.379+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbde40334b0 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.380+0000 7fbdf5600640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 0x7fbdd807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:53.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.380+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fbde40bc150 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.380+0000 7fbdfda00640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 0x7fbdd807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:53.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.381+0000 7fbdfda00640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 0x7fbdd807b0a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0015a0 tx=0x7fbdec009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:53.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.384+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbde403f030 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.602+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdf8051000 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.603+0000 7fbdf5600640 1 -- 172.21.15.129:0/1856240351 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7fbde4087490 con 0x7fbdf80cc3e0 2024-09-17T03:59:53.603 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:53.603 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:53.603 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 msgr2=0x7fbdd807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 0x7fbdd807b0a0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbdec0015a0 tx=0x7fbdec009040 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 msgr2=0x7fbdf80c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009950 tx=0x7fbde4033c20 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 shutdown_connections 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbdd8078b90 0x7fbdd807b0a0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbdf80cc910 0x7fbdf8159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbdf80cc3e0 0x7fbdf80c4ad0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 --2- 172.21.15.129:0/1856240351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbdf8070a50 0x7fbdf80b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.605+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 >> 172.21.15.129:0/1856240351 conn(0x7fbdf806d410 msgr2=0x7fbdf80ccd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:53.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.606+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 shutdown_connections 2024-09-17T03:59:53.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:53.606+0000 7fbe00400640 1 -- 172.21.15.129:0/1856240351 wait complete. 2024-09-17T03:59:53.628 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T03:59:53.867 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-09-17T03:59:53.867 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-09-17T03:59:53.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 -- 172.21.15.129:0/909889674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d16d0 msgr2=0x7f66e0169710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 --2- 172.21.15.129:0/909889674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d16d0 0x7f66e0169710 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f66e00b7180 tx=0x7f66d802aac0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 -- 172.21.15.129:0/909889674 shutdown_connections 2024-09-17T03:59:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 --2- 172.21.15.129:0/909889674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d16d0 0x7f66e0169710 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 --2- 172.21.15.129:0/909889674 >> v2:172.21.15.181:3300/0 conn(0x7f66e00b3220 0x7f66e00d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 -- 172.21.15.129:0/909889674 >> 172.21.15.129:0/909889674 conn(0x7f66e009d8d0 msgr2=0x7f66e008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:54.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 -- 172.21.15.129:0/909889674 shutdown_connections 2024-09-17T03:59:54.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.089+0000 7f66e6400640 1 -- 172.21.15.129:0/909889674 wait complete. 2024-09-17T03:59:54.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.091+0000 7f66e6400640 1 Processor -- start 2024-09-17T03:59:54.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.091+0000 7f66e6400640 1 -- start start 2024-09-17T03:59:54.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 0x7f66e00b6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 0x7f66e0150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66e00d0070 con 0x7f66e00d4a80 2024-09-17T03:59:54.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f66e00cfef0 con 0x7f66e00b3220 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f66e00b9ec0 con 0x7f66e00d16d0 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48632/0 (socket says 172.21.15.129:48632) 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 -- 172.21.15.129:0/4237082316 learned_addr learned my addr 172.21.15.129:0/4237082316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 -- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 msgr2=0x7f66e0150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5000640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 0x7f66e00b6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66d7e00640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 0x7f66e0150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 0x7f66e0150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.093+0000 7f66e5a00640 1 -- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 msgr2=0x7f66e00b6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e5a00640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 0x7f66e00b6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e5a00640 1 -- 172.21.15.129:0/4237082316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66e0152780 con 0x7f66e00d4a80 2024-09-17T03:59:54.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e5000640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 0x7f66e00b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66d7e00640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 0x7f66e0150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e5a00640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f66d802b0b0 tx=0x7f66d8003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8003e50 con 0x7f66e00d4a80 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f66d8004250 con 0x7f66e00d4a80 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66e01e73e0 con 0x7f66e00d4a80 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66d8004680 con 0x7f66e00d4a80 2024-09-17T03:59:54.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.094+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66e0152f90 con 0x7f66e00d4a80 2024-09-17T03:59:54.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.095+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66e00d0c50 con 0x7f66e00d4a80 2024-09-17T03:59:54.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.097+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f66d8049430 con 0x7f66e00d4a80 2024-09-17T03:59:54.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.097+0000 7f66dd600640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 0x7f66bc07b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.098+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f66d80bbb50 con 0x7f66e00d4a80 2024-09-17T03:59:54.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.098+0000 7f66e5000640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 0x7f66bc07b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.099+0000 7f66e5000640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 0x7f66bc07b270 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009aa0 tx=0x7f66d0005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:54.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.102+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f66d8032030 con 0x7f66e00d4a80 2024-09-17T03:59:54.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.319+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66e006a130 con 0x7f66e00d4a80 2024-09-17T03:59:54.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.319+0000 7f66dd600640 1 -- 172.21.15.129:0/4237082316 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f66d8004820 con 0x7f66e00d4a80 2024-09-17T03:59:54.320 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:54.320 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:54.320 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:54.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 msgr2=0x7f66bc07b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 0x7f66bc07b270 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f66d0009aa0 tx=0x7f66d0005f30 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 msgr2=0x7f66e0151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f66d802b0b0 tx=0x7f66d8003ce0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 shutdown_connections 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f66bc078d60 0x7f66bc07b270 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66e00d4a80 0x7f66e0151280 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f66e00d16d0 0x7f66e0150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 --2- 172.21.15.129:0/4237082316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f66e00b3220 0x7f66e00b6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.322+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 >> 172.21.15.129:0/4237082316 conn(0x7f66e009d8d0 msgr2=0x7f66e0075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:54.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.323+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 shutdown_connections 2024-09-17T03:59:54.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.323+0000 7f66e6400640 1 -- 172.21.15.129:0/4237082316 wait complete. 2024-09-17T03:59:54.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump purge_queue pq_executed 2024-09-17T03:59:54.587 INFO:teuthology.orchestra.run.smithi181.stdout:{"purge_queue":{"pq_executed":1}} 2024-09-17T03:59:54.587 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-09-17T03:59:54.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 -- 172.21.15.129:0/1454426155 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815b8b0 msgr2=0x7f3d9815f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 --2- 172.21.15.129:0/1454426155 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815b8b0 0x7f3d9815f2e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009860 tx=0x7f3d8802a510 comp rx=0 tx=0).stop 2024-09-17T03:59:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 -- 172.21.15.129:0/1454426155 shutdown_connections 2024-09-17T03:59:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 --2- 172.21.15.129:0/1454426155 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815b8b0 0x7f3d9815f2e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 -- 172.21.15.129:0/1454426155 >> 172.21.15.129:0/1454426155 conn(0x7f3d980d0a70 msgr2=0x7f3d980d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 -- 172.21.15.129:0/1454426155 shutdown_connections 2024-09-17T03:59:54.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.808+0000 7f3d9f000640 1 -- 172.21.15.129:0/1454426155 wait complete. 2024-09-17T03:59:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.810+0000 7f3d9f000640 1 Processor -- start 2024-09-17T03:59:54.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.810+0000 7f3d9f000640 1 -- start start 2024-09-17T03:59:54.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d9815b8b0 0x7f3d980befe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815f820 0x7f3d980afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d980c4140 con 0x7f3d9815fc30 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d98071fb0 con 0x7f3d9815b8b0 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d980c42c0 con 0x7f3d9815f820 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48636/0 (socket says 172.21.15.129:48636) 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d9d000640 1 -- 172.21.15.129:0/2461015816 learned_addr learned my addr 172.21.15.129:0/2461015816 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d97e00640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d9815b8b0 0x7f3d980befe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.812+0000 7f3d8f400640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815f820 0x7f3d980afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 -- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815f820 msgr2=0x7f3d980afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815f820 0x7f3d980afc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 -- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d9815b8b0 msgr2=0x7f3d980befe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:54.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d9815b8b0 0x7f3d980befe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:54.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 -- 172.21.15.129:0/2461015816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d980b0160 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9d000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3d90002230 tx=0x7f3d9000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90016020 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d9000ad40 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d980c8270 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90012680 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.813+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d981f7e10 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.814+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d981f7920 con 0x7f3d9815fc30 2024-09-17T03:59:54.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.816+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d90019020 con 0x7f3d9815fc30 2024-09-17T03:59:54.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.816+0000 7f3d94c00640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 0x7f3d7407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:54.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.817+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f3d9009b260 con 0x7f3d9815fc30 2024-09-17T03:59:54.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.817+0000 7f3d97e00640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 0x7f3d7407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:54.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.818+0000 7f3d97e00640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 0x7f3d7407b0a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3d88034000 tx=0x7f3d88004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:54.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:54.821+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d90020020 con 0x7f3d9815fc30 2024-09-17T03:59:55.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.045+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d980d16d0 con 0x7f3d9815fc30 2024-09-17T03:59:55.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.046+0000 7f3d94c00640 1 -- 172.21.15.129:0/2461015816 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f3d90062fd0 con 0x7f3d9815fc30 2024-09-17T03:59:55.046 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:55.047 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:55.047 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 msgr2=0x7f3d7407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 0x7f3d7407b0a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3d88034000 tx=0x7f3d88004730 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 msgr2=0x7f3d980b18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f3d90002230 tx=0x7f3d9000cf50 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 shutdown_connections 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d74078b90 0x7f3d7407b0a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d9815fc30 0x7f3d980b18d0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d9815f820 0x7f3d980afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 --2- 172.21.15.129:0/2461015816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d9815b8b0 0x7f3d980befe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 >> 172.21.15.129:0/2461015816 conn(0x7f3d980d0a70 msgr2=0x7f3d980740a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.049+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 shutdown_connections 2024-09-17T03:59:55.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.050+0000 7f3d9f000640 1 -- 172.21.15.129:0/2461015816 wait complete. 2024-09-17T03:59:55.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache 2024-09-17T03:59:55.313 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1,"strays_enqueued":1,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0}} 2024-09-17T03:59:55.313 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1, "strays_enqueued": 1, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 } } 2024-09-17T03:59:55.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:55.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 -- 172.21.15.129:0/929669340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0b3220 msgr2=0x7f544c0d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 --2- 172.21.15.129:0/929669340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0b3220 0x7f544c0d4540 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f543c009820 tx=0x7f543c02a5a0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 -- 172.21.15.129:0/929669340 shutdown_connections 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 --2- 172.21.15.129:0/929669340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0b3220 0x7f544c0d4540 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 -- 172.21.15.129:0/929669340 >> 172.21.15.129:0/929669340 conn(0x7f544c09d8d0 msgr2=0x7f544c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 -- 172.21.15.129:0/929669340 shutdown_connections 2024-09-17T03:59:55.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.533+0000 7f544b200640 1 -- 172.21.15.129:0/929669340 wait complete. 2024-09-17T03:59:55.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.535+0000 7f544b200640 1 Processor -- start 2024-09-17T03:59:55.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.536+0000 7f544b200640 1 -- start start 2024-09-17T03:59:55.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f544c0b3220 0x7f544c155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:55.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0d4a80 0x7f544c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f544c0d0070 con 0x7f544c0d16d0 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f544c0cfef0 con 0x7f544c0d4a80 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f544c0b9ec0 con 0x7f544c0b3220 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f5441400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f5441400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48648/0 (socket says 172.21.15.129:48648) 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f5441400640 1 -- 172.21.15.129:0/1303671579 learned_addr learned my addr 172.21.15.129:0/1303671579 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f5449e00640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f544c0b3220 0x7f544c155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.537+0000 7f544a800640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0d4a80 0x7f544c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 -- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f544c0b3220 msgr2=0x7f544c155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f544c0b3220 0x7f544c155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 -- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0d4a80 msgr2=0x7f544c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0d4a80 0x7f544c0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 -- 172.21.15.129:0/1303671579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c156190 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5441400640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f543400c430 tx=0x7f543400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434017070 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1f92d0 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54340053c0 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5434003850 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.538+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f544c1f9890 con 0x7f544c0d16d0 2024-09-17T03:59:55.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.540+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544c1f9040 con 0x7f544c0d16d0 2024-09-17T03:59:55.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.541+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f543401a070 con 0x7f544c0d16d0 2024-09-17T03:59:55.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.542+0000 7f5442a00640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 0x7f542807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:55.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.543+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f5434099fe0 con 0x7f544c0d16d0 2024-09-17T03:59:55.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.543+0000 7f5449e00640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 0x7f542807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:55.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.543+0000 7f5449e00640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 0x7f542807b0d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f543c034000 tx=0x7f543c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:55.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.547+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5434061e30 con 0x7f544c0d16d0 2024-09-17T03:59:55.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.763+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f544c0d0c50 con 0x7f544c0d16d0 2024-09-17T03:59:55.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.764+0000 7f5442a00640 1 -- 172.21.15.129:0/1303671579 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f5434066860 con 0x7f544c0d16d0 2024-09-17T03:59:55.764 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:55.764 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:55.764 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:55.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 msgr2=0x7f542807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 0x7f542807b0d0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f543c034000 tx=0x7f543c004730 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 msgr2=0x7f544c09e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f543400c430 tx=0x7f543400c8f0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 shutdown_connections 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5428078bc0 0x7f542807b0d0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f544c0d4a80 0x7f544c0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f544c0d16d0 0x7f544c09e240 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 --2- 172.21.15.129:0/1303671579 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f544c0b3220 0x7f544c155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 >> 172.21.15.129:0/1303671579 conn(0x7f544c09d8d0 msgr2=0x7f544c0d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 shutdown_connections 2024-09-17T03:59:55.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:55.766+0000 7f544b200640 1 -- 172.21.15.129:0/1303671579 wait complete. 2024-09-17T03:59:55.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump purge_queue 2024-09-17T03:59:56.026 INFO:teuthology.orchestra.run.smithi181.stdout:{"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":5,"pq_executing":0,"pq_executing_high_water":1,"pq_executed_ops":5,"pq_executed":1,"pq_item_in_journal":0}} 2024-09-17T03:59:56.026 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1, "pq_executed_ops": 5, "pq_executing": 0, "pq_executing_high_water": 1, "pq_executing_ops": 0, "pq_executing_ops_high_water": 5, "pq_item_in_journal": 0 } } 2024-09-17T03:59:56.027 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T03:59:56.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.165+0000 7fa4de5d3f40 1 Processor -- start 2024-09-17T03:59:56.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.165+0000 7fa4de5d3f40 1 -- start start 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.166+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9740a5c0 0x55cf9749d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.166+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.166+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9749d140 0x55cf97490fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf97314c60 con 0x55cf97492b70 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cf973731d0 con 0x55cf9740a5c0 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cf974934c0 con 0x55cf9749d140 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48668/0 (socket says 172.21.15.129:48668) 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 -- 172.21.15.129:0/761322714 learned_addr learned my addr 172.21.15.129:0/761322714 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4d8200640 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9749d140 0x55cf97490fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4d7800640 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9740a5c0 0x55cf9749d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 -- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9749d140 msgr2=0x55cf97490fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:56.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9749d140 0x55cf97490fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 -- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9740a5c0 msgr2=0x55cf9749d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9740a5c0 0x55cf9749d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.167+0000 7fa4cee00640 1 -- 172.21.15.129:0/761322714 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf9740cba0 con 0x55cf97492b70 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4cee00640 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001d50 tx=0x7fa4c000b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7e149a4cff7e96 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4d6400640 1 -- 172.21.15.129:0/761322714 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000bd80 con 0x55cf97492b70 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4d6400640 1 -- 172.21.15.129:0/761322714 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4c000f400 con 0x55cf97492b70 2024-09-17T03:59:56.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4d6400640 1 -- 172.21.15.129:0/761322714 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c000f850 con 0x55cf97492b70 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/761322714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 msgr2=0x55cf97496280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0001d50 tx=0x7fa4c000b5b0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/761322714 shutdown_connections 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9749d140 0x55cf97490fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97496280 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/761322714 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9740a5c0 0x55cf9749d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/761322714 >> 172.21.15.129:0/761322714 conn(0x55cf972e5ac0 msgr2=0x55cf97495980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:56.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.168+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/761322714 shutdown_connections 2024-09-17T03:59:56.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.169+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/761322714 wait complete. 2024-09-17T03:59:56.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.170+0000 7fa4de5d3f40 1 Processor -- start 2024-09-17T03:59:56.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.170+0000 7fa4de5d3f40 1 -- start start 2024-09-17T03:59:56.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9740a5c0 0x55cf97488d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9749d140 0x55cf97490c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cf97492590 con 0x55cf97492b70 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cf9740d000 con 0x55cf9749d140 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4de5d3f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cf97499d40 con 0x55cf9740a5c0 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4cee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4cee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48684/0 (socket says 172.21.15.129:48684) 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 learned_addr learned my addr 172.21.15.129:0/632881887 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.171+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9740a5c0 0x55cf97488d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9749d140 0x55cf97490c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9740a5c0 msgr2=0x55cf97488d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cf9740a5c0 0x55cf97488d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9749d140 msgr2=0x55cf97490c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cf9749d140 0x55cf97490c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf9748f270 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x55cf9740c830 tx=0x7fa4c000ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4d5000640 1 -- 172.21.15.129:0/632881887 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c001b430 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.172+0000 7fa4d5000640 1 -- 172.21.15.129:0/632881887 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa4c0020020 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.173+0000 7fa4d5000640 1 -- 172.21.15.129:0/632881887 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c001b700 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.173+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cf97491800 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.173+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf9748ea70 con 0x55cf97492b70 2024-09-17T03:59:56.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.175+0000 7fa4d5000640 1 -- 172.21.15.129:0/632881887 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa4c0017070 con 0x55cf97492b70 2024-09-17T03:59:56.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4d5000640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa4b40775a0 0x7fa4b4079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4d5000640 1 -- 172.21.15.129:0/632881887 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fa4c009ab40 con 0x55cf97492b70 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa4b40775a0 0x7fa4b4079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55cf97499090 0x55cf974994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 23.0 23:00000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.176+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55cf97499090 0x55cf974994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.177+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa4b40775a0 0x7fa4b4079ab0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800c740 tx=0x7fa4c8004880 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.177+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55cf97499090 0x55cf974994a0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.177+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.177+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fa4d0022d70 0x7fa4d0061aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.177+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:2 23.20 23:04000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.178+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fa4d0022d70 0x7fa4d0061aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.178+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fa4d0022d70 0x7fa4d0061aa0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.178+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(2 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d0022d70 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.179+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fa4c8008950 0x7fa4c800b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.179+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:3 23.10 23:08000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.179+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fa4c8008950 0x7fa4c800b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.179+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fa4c8008950 0x7fa4c800b440 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(3 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:4 23.30 23:0c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(4 [pgnls start_epoch 152 out=81b] v152'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fa4d001f6f0 0x7fa4d0055a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:5 23.8 23:10000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d001f6f0 2024-09-17T03:59:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.180+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fa4d001f6f0 0x7fa4d0055a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.181+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fa4d001f6f0 0x7fa4d0055a90 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.182+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(5 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4d001f6f0 2024-09-17T03:59:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.182+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa4d000c2b0 0x7fa4d0072950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.182+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:6 23.28 23:14000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.182+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa4d000c2b0 0x7fa4d0072950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.183+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa4d000c2b0 0x7fa4d0072950 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.184+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(6 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d000c2b0 2024-09-17T03:59:56.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.184+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:7 23.18 23:18000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.184+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(7 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.184+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:8 23.38 23:1c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.185+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(8 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d000c2b0 2024-09-17T03:59:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.185+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fa4c802b040 0x7fa4c802b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.185+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:9 23.4 23:20000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.185+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fa4c802b040 0x7fa4c802b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.186+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fa4c802b040 0x7fa4c802b960 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.186+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(9 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.186+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:10 23.24 23:24000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.186+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(10 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.186+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:11 23.14 23:28000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(11 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d0022d70 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 23.34 23:2c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(12 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d000c2b0 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:13 23.c 23:30000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(13 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4cee00640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fa4c0062c10 0x7fa4c0061680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.187+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:14 23.2c 23:34000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.188+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fa4c0062c10 0x7fa4c0061680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.188+0000 7fa4d8200640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fa4c0062c10 0x7fa4c0061680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.189+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(14 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.189+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:15 23.1c 23:38000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.189+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(15 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.189+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:16 23.3c 23:3c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(16 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:17 23.2 23:40000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(17 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c800a8a0 con 0x7fa4d0022d70 2024-09-17T03:59:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fa4c8016920 0x7fa4c802a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:18 23.22 23:44000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8016920 2024-09-17T03:59:56.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.190+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fa4c8016920 0x7fa4c802a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.191+0000 7fa4d7800640 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fa4c8016920 0x7fa4c802a3e0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.191+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(18 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4c8016920 2024-09-17T03:59:56.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.191+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:19 23.12 23:48000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.192+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(19 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.192+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:20 23.32 23:4c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.192+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(20 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.192+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:21 23.a 23:50000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.193+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(21 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.193+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:22 23.2a 23:54000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.193+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(22 [pgnls start_epoch 152 out=81b] v152'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.193+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 23.1a 23:58000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(23 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:24 23.3a 23:5c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(24 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:25 23.6 23:60000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(25 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.194+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 23.26 23:64000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(26 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:27 23.16 23:68000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(27 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:28 23.36 23:6c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(28 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.195+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:29 23.e 23:70000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(29 [pgnls start_epoch 152 out=49b] v152'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:30 23.2e 23:74000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(30 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:31 23.1e 23:78000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(31 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.196+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:32 23.3e 23:7c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.197+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(32 [pgnls start_epoch 152 out=49b] v152'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.197+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:33 23.1 23:80000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.197+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(33 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.197+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:34 23.21 23:84000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8016920 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.198+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(34 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4c8016920 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.198+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:35 23.11 23:88000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.198+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(35 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.198+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:36 23.31 23:8c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d001f6f0 2024-09-17T03:59:56.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.199+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(36 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4d001f6f0 2024-09-17T03:59:56.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.199+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:37 23.9 23:90000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.199+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(37 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.199+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:38 23.29 23:94000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(38 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:39 23.19 23:98000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(39 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:40 23.39 23:9c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(40 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.200+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:41 23.5 23:a0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.201+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(41 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.201+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:42 23.25 23:a4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.201+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(42 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.201+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:43 23.15 23:a8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8016920 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.202+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(43 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4c8016920 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.202+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:44 23.35 23:ac000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.202+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 7 ==== osd_op_reply(44 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.202+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:45 23.d 23:b0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(45 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:46 23.2d 23:b4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(46 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:47 23.1d 23:b8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(47 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.203+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:48 23.3d 23:bc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.204+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 8 ==== osd_op_reply(48 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.204+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:49 23.3 23:c0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8016920 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.204+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(49 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4c8016920 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.204+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:50 23.23 23:c4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.205+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 9 ==== osd_op_reply(50 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.205+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:51 23.13 23:c8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.205+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(51 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.205+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:52 23.33 23:cc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.206+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 10 ==== osd_op_reply(52 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.206+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:53 23.b 23:d0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.206+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(53 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.206+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:54 23.2b 23:d4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.207+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(54 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.207+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:55 23.1b 23:d8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8008950 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.207+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(55 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c8008950 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.207+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:56 23.3b 23:dc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.208+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 8 ==== osd_op_reply(56 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.208+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 23.7 23:e0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x55cf97499090 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.208+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(57 [pgnls start_epoch 152 out=49b] v152'3 uv3 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x55cf97499090 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.208+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:58 23.27 23:e4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c0062c10 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.209+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.5 v2:172.21.15.181:6818/3094080342 11 ==== osd_op_reply(58 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4c0062c10 2024-09-17T03:59:56.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.209+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:59 23.17 23:e8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.209+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(59 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.209+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:60 23.37 23:ec000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d000c2b0 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.6 v2:172.21.15.181:6810/4136500368 9 ==== osd_op_reply(60 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d000c2b0 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:61 23.f 23:f0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d0022d70 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.3 v2:172.21.15.129:6808/2225231477 10 ==== osd_op_reply(61 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4d0022d70 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:62 23.2f 23:f4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c802b040 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 <== osd.2 v2:172.21.15.129:6809/283507149 9 ==== osd_op_reply(62 [pgnls start_epoch 152 out=49b] v152'2 uv2 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c005ec20 con 0x7fa4c802b040 2024-09-17T03:59:56.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.210+0000 7fa4cee00640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:63 23.1f 23:f8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4c8016920 2024-09-17T03:59:56.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.211+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(63 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4c8039040 con 0x7fa4c8016920 2024-09-17T03:59:56.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.211+0000 7fa4d7800640 1 -- 172.21.15.129:0/632881887 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:64 23.3f 23:fc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55cf973c4910 con 0x7fa4d001f6f0 2024-09-17T03:59:56.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(64 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fa4d000b040 con 0x7fa4d001f6f0 2024-09-17T03:59:56.213 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000000 2024-09-17T03:59:56.213 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000001 2024-09-17T03:59:56.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55cf97499090 msgr2=0x55cf974994a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55cf97499090 0x55cf974994a0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fa4c8008950 msgr2=0x7fa4c800b440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fa4c8008950 0x7fa4c800b440 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fa4c802b040 msgr2=0x7fa4c802b960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fa4c802b040 0x7fa4c802b960 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fa4d0022d70 msgr2=0x7fa4d0061aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fa4d0022d70 0x7fa4d0061aa0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 reap_dead start 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fa4c8016920 msgr2=0x7fa4c802a3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fa4c8016920 0x7fa4c802a3e0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fa4c0062c10 msgr2=0x7fa4c0061680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fa4c0062c10 0x7fa4c0061680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa4d000c2b0 msgr2=0x7fa4d0072950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa4d000c2b0 0x7fa4d0072950 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fa4d001f6f0 msgr2=0x7fa4d0055a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fa4d001f6f0 0x7fa4d0055a90 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa4b40775a0 msgr2=0x7fa4b4079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa4b40775a0 0x7fa4b4079ab0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa4c800c740 tx=0x7fa4c8004880 comp rx=0 tx=0).stop 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 msgr2=0x55cf97490710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4d8200640 1 -- 172.21.15.129:0/632881887 reap_dead start 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x55cf9740c830 tx=0x7fa4c000ff50 comp rx=0 tx=0).stop 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 shutdown_connections 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 --2- 172.21.15.129:0/632881887 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cf97492b70 0x55cf97490710 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 >> 172.21.15.129:0/632881887 conn(0x55cf972e5ac0 msgr2=0x55cf97495980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:56.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 shutdown_connections 2024-09-17T03:59:56.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.212+0000 7fa4de5d3f40 1 -- 172.21.15.129:0/632881887 wait complete. 2024-09-17T03:59:56.227 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 are absent 2024-09-17T03:59:56.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 -- 172.21.15.129:0/2674572786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00d4c50 msgr2=0x7f55f00ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2674572786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00d4c50 0x7f55f00ceee0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009830 tx=0x7f55dc02a510 comp rx=0 tx=0).stop 2024-09-17T03:59:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 -- 172.21.15.129:0/2674572786 shutdown_connections 2024-09-17T03:59:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2674572786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f00cf420 0x7f55f00cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2674572786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00d4c50 0x7f55f00ceee0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 -- 172.21.15.129:0/2674572786 >> 172.21.15.129:0/2674572786 conn(0x7f55f009d8d0 msgr2=0x7f55f008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:56.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 -- 172.21.15.129:0/2674572786 shutdown_connections 2024-09-17T03:59:56.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.449+0000 7f55f6c00640 1 -- 172.21.15.129:0/2674572786 wait complete. 2024-09-17T03:59:56.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.451+0000 7f55f6c00640 1 Processor -- start 2024-09-17T03:59:56.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.451+0000 7f55f6c00640 1 -- start start 2024-09-17T03:59:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55f00cf420 0x7f55f01f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f01f5950 0x7f55f00d4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55f00d0ae0 con 0x7f55f00b3220 2024-09-17T03:59:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55f00ba0d0 con 0x7f55f00cf420 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f55f00d0c60 con 0x7f55f01f5950 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48714/0 (socket says 172.21.15.129:48714) 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f5800640 1 -- 172.21.15.129:0/2552041684 learned_addr learned my addr 172.21.15.129:0/2552041684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f6200640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f01f5950 0x7f55f00d4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.453+0000 7f55f4e00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55f00cf420 0x7f55f01f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 -- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f01f5950 msgr2=0x7f55f00d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f01f5950 0x7f55f00d4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 -- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55f00cf420 msgr2=0x7f55f01f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55f00cf420 0x7f55f01f5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 -- 172.21.15.129:0/2552041684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55f01f61d0 con 0x7f55f00b3220 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f5800640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f55e400c400 tx=0x7f55e400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4017070 con 0x7f55f00b3220 2024-09-17T03:59:56.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55e40035f0 con 0x7f55f00b3220 2024-09-17T03:59:56.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55f01f64e0 con 0x7f55f00b3220 2024-09-17T03:59:56.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4003a40 con 0x7f55f00b3220 2024-09-17T03:59:56.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.454+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55f01f6aa0 con 0x7f55f00b3220 2024-09-17T03:59:56.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.456+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55f00cf830 con 0x7f55f00b3220 2024-09-17T03:59:56.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.457+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f55e401a070 con 0x7f55f00b3220 2024-09-17T03:59:56.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.458+0000 7f55ed600640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 0x7f55d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:56.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.458+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f55e409a340 con 0x7f55f00b3220 2024-09-17T03:59:56.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.459+0000 7f55f4e00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 0x7f55d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:56.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.459+0000 7f55f4e00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 0x7f55d007b0d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f55dc034000 tx=0x7f55dc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:56.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.461+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f55e4021020 con 0x7f55f00b3220 2024-09-17T03:59:56.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.679+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55f006a130 con 0x7f55f00b3220 2024-09-17T03:59:56.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.680+0000 7f55ed600640 1 -- 172.21.15.129:0/2552041684 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f55e4062110 con 0x7f55f00b3220 2024-09-17T03:59:56.680 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:56.680 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:56.680 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 msgr2=0x7f55d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 0x7f55d007b0d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f55dc034000 tx=0x7f55dc004730 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 msgr2=0x7f55f0151c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f55e400c400 tx=0x7f55e400c8c0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 shutdown_connections 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f55d0078bc0 0x7f55d007b0d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55f01f5950 0x7f55f00d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55f00cf420 0x7f55f01f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 --2- 172.21.15.129:0/2552041684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55f00b3220 0x7f55f0151c80 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:56.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 >> 172.21.15.129:0/2552041684 conn(0x7f55f009d8d0 msgr2=0x7f55f006e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:56.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 shutdown_connections 2024-09-17T03:59:56.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:56.682+0000 7f55f6c00640 1 -- 172.21.15.129:0/2552041684 wait complete. 2024-09-17T03:59:56.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-09-17T03:59:56.952 INFO:teuthology.orchestra.run.smithi181.stdout:{"message":"","return_code":0} 2024-09-17T03:59:56.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T03:59:56.953 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000000.00000000 parent 2024-09-17T03:59:57.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.097+0000 7fa335678f40 1 Processor -- start 2024-09-17T03:59:57.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.097+0000 7fa335678f40 1 -- start start 2024-09-17T03:59:57.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa535ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa4a77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a4fa3adc60 con 0x55a4fa519150 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a4fa40c1d0 con 0x55a4fa4a8640 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.098+0000 7fa335678f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55a4fa533cd0 con 0x55a4fa4aa610 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48724/0 (socket says 172.21.15.129:48724) 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 -- 172.21.15.129:0/2590947346 learned_addr learned my addr 172.21.15.129:0/2590947346 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32de00640 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa4a77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32e800640 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa535ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 -- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 msgr2=0x55a4fa4a77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa4a77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 -- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 msgr2=0x55a4fa535ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa535ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.099+0000 7fa32f200640 1 -- 172.21.15.129:0/2590947346 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4fa532f30 con 0x55a4fa519150 2024-09-17T03:59:57.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa32f200640 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fa328046530 tx=0x7fa328003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f88968fa287a809 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa32ca00640 1 -- 172.21.15.129:0/2590947346 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328065030 con 0x55a4fa519150 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa32ca00640 1 -- 172.21.15.129:0/2590947346 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa32804d840 con 0x55a4fa519150 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa32ca00640 1 -- 172.21.15.129:0/2590947346 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328004260 con 0x55a4fa519150 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa335678f40 1 -- 172.21.15.129:0/2590947346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 msgr2=0x55a4fa4a7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.100+0000 7fa335678f40 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fa328046530 tx=0x7fa328003330 comp rx=0 tx=0).stop 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 -- 172.21.15.129:0/2590947346 shutdown_connections 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa4a7d10 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa4a77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 --2- 172.21.15.129:0/2590947346 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa535ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 -- 172.21.15.129:0/2590947346 >> 172.21.15.129:0/2590947346 conn(0x55a4fa37eac0 msgr2=0x55a4fa532020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:57.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 -- 172.21.15.129:0/2590947346 shutdown_connections 2024-09-17T03:59:57.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.101+0000 7fa335678f40 1 -- 172.21.15.129:0/2590947346 wait complete. 2024-09-17T03:59:57.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.102+0000 7fa335678f40 1 Processor -- start 2024-09-17T03:59:57.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.102+0000 7fa335678f40 1 -- start start 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa5d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa50fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa535ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a4fa4a1750 con 0x55a4fa519150 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a4fa5301f0 con 0x55a4fa4a8640 2024-09-17T03:59:57.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.103+0000 7fa335678f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55a4fa4a1470 con 0x55a4fa4aa610 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa535ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa535ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48736/0 (socket says 172.21.15.129:48736) 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 -- 172.21.15.129:0/1783735141 learned_addr learned my addr 172.21.15.129:0/1783735141 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32e800640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa5d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32de00640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa50fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 -- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 msgr2=0x55a4fa50fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a4fa4aa610 0x55a4fa50fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 -- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 msgr2=0x55a4fa5d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a4fa4a8640 0x55a4fa5d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 -- 172.21.15.129:0/1783735141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4fa5326b0 con 0x55a4fa519150 2024-09-17T03:59:57.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.104+0000 7fa32f200640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa535ac0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fa32800a600 tx=0x7fa32804d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.105+0000 7fa327400640 1 -- 172.21.15.129:0/1783735141 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32804dde0 con 0x55a4fa519150 2024-09-17T03:59:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.105+0000 7fa327400640 1 -- 172.21.15.129:0/1783735141 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa328061570 con 0x55a4fa519150 2024-09-17T03:59:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.105+0000 7fa327400640 1 -- 172.21.15.129:0/1783735141 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa328061980 con 0x55a4fa519150 2024-09-17T03:59:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.105+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a4fa5cf910 con 0x55a4fa519150 2024-09-17T03:59:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.105+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4fa510320 con 0x55a4fa519150 2024-09-17T03:59:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.107+0000 7fa327400640 1 -- 172.21.15.129:0/1783735141 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa328068070 con 0x55a4fa519150 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.107+0000 7fa327400640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa310077570 0x7fa310079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.108+0000 7fa327400640 1 -- 172.21.15.129:0/1783735141 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fa3280df840 con 0x55a4fa519150 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.108+0000 7fa335678f40 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55a4fa5cc280 0x55a4fa5cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.108+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 23.30 23:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e152) -- 0x55a4fa5ccbd0 con 0x55a4fa5cc280 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.108+0000 7fa32e800640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa310077570 0x7fa310079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.108+0000 7fa32de00640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55a4fa5cc280 0x55a4fa5cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa32e800640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa310077570 0x7fa310079a80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa320011560 tx=0x7fa320002ad0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa32de00640 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55a4fa5cc280 0x55a4fa5cc690 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa32de00640 1 -- 172.21.15.129:0/1783735141 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (62) out=62b] v0'0 uv4 ondisk = 0) ==== 164+0+62 (crc 0 0 0) 0x7fa31c00b8b0 con 0x55a4fa5cc280 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55a4fa5cc280 msgr2=0x55a4fa5cc690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55a4fa5cc280 0x55a4fa5cc690 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa310077570 msgr2=0x7fa310079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa310077570 0x7fa310079a80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fa320011560 tx=0x7fa320002ad0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 msgr2=0x55a4fa535ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 --2- 172.21.15.129:0/1783735141 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a4fa519150 0x55a4fa535ac0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7fa32800a600 tx=0x7fa32804d840 comp rx=0 tx=0).stop 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa32f200640 1 -- 172.21.15.129:0/1783735141 reap_dead start 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 shutdown_connections 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.109+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 >> 172.21.15.129:0/1783735141 conn(0x55a4fa37eac0 msgr2=0x55a4fa45fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.110+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 shutdown_connections 2024-09-17T03:59:57.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.110+0000 7fa335678f40 1 -- 172.21.15.129:0/1783735141 wait complete. 2024-09-17T03:59:57.120 INFO:teuthology.orchestra.run.smithi129.stdout:8file_b 2024-09-17T03:59:57.120 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627776, 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "file_b", 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 14 2024-09-17T03:59:57.261 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T03:59:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T03:59:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 23, 2024-09-17T03:59:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T03:59:57.262 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T03:59:57.262 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:57.273 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T03:59:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.411+0000 7f0af8b15f40 1 Processor -- start 2024-09-17T03:59:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.411+0000 7f0af8b15f40 1 -- start start 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e03f4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e043f5c0 0x55d2e04c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d2e03a21d0 con 0x55d2e04c2280 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d2e03aef90 con 0x55d2e043f5c0 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d2e0343c60 con 0x55d2e04395c0 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48748/0 (socket says 172.21.15.129:48748) 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 -- 172.21.15.129:0/1393253249 learned_addr learned my addr 172.21.15.129:0/1393253249 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af1c00640 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e03f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af1200640 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e043f5c0 0x55d2e04c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 -- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 msgr2=0x55d2e03f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e03f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 -- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e043f5c0 msgr2=0x55d2e04c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e043f5c0 0x55d2e04c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.413+0000 7f0af2600640 1 -- 172.21.15.129:0/1393253249 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2e04c55e0 con 0x55d2e04c2280 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0af2600640 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0aec057040 tx=0x7f0aec003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaba10fa107c992c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0aeb400640 1 -- 172.21.15.129:0/1393253249 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec065030 con 0x55d2e04c2280 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0aeb400640 1 -- 172.21.15.129:0/1393253249 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0aec004020 con 0x55d2e04c2280 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0aeb400640 1 -- 172.21.15.129:0/1393253249 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec04db00 con 0x55d2e04c2280 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0af8b15f40 1 -- 172.21.15.129:0/1393253249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 msgr2=0x55d2e04c31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.414+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f0aec057040 tx=0x7f0aec003330 comp rx=0 tx=0).stop 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 -- 172.21.15.129:0/1393253249 shutdown_connections 2024-09-17T03:59:57.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e04c2280 0x55d2e04c31e0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e043f5c0 0x55d2e04c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/1393253249 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e03f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 -- 172.21.15.129:0/1393253249 >> 172.21.15.129:0/1393253249 conn(0x55d2e0314ac0 msgr2=0x55d2e043cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 -- 172.21.15.129:0/1393253249 shutdown_connections 2024-09-17T03:59:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.415+0000 7f0af8b15f40 1 -- 172.21.15.129:0/1393253249 wait complete. 2024-09-17T03:59:57.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.416+0000 7f0af8b15f40 1 Processor -- start 2024-09-17T03:59:57.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.416+0000 7f0af8b15f40 1 -- start start 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e0565610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e04c2280 0x55d2e03f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d2e04c5850 con 0x55d2e043f5c0 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d2e043cb60 con 0x55d2e04c2280 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.417+0000 7f0af8b15f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d2e0564af0 con 0x55d2e04395c0 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48754/0 (socket says 172.21.15.129:48754) 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 learned_addr learned my addr 172.21.15.129:0/3336907495 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 msgr2=0x55d2e0565610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e0565610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e04c2280 0x55d2e03f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e0565610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e04c2280 msgr2=0x55d2e03f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e04c2280 0x55d2e03f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2e0562700 con 0x55d2e043f5c0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d2e04c2280 0x55d2e03f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d2e04395c0 0x55d2e0565610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.418+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000ed80 tx=0x7f0ae000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.419+0000 7f0aea000640 1 -- 172.21.15.129:0/3336907495 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0015020 con 0x55d2e043f5c0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.419+0000 7f0aea000640 1 -- 172.21.15.129:0/3336907495 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ae0011450 con 0x55d2e043f5c0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.419+0000 7f0aea000640 1 -- 172.21.15.129:0/3336907495 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0011880 con 0x55d2e043f5c0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.419+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d2e05629e0 con 0x55d2e043f5c0 2024-09-17T03:59:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.419+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2e0562fa0 con 0x55d2e043f5c0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.421+0000 7f0aea000640 1 -- 172.21.15.129:0/3336907495 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0ae0018020 con 0x55d2e043f5c0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0aea000640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 0x7f0ad4079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0aea000640 1 -- 172.21.15.129:0/3336907495 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f0ae0099c40 con 0x55d2e043f5c0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55d2e0563cc0 0x55d2e05640d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 23.0 23:00000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x55d2e0563cc0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 0x7f0ad4079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.422+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55d2e0563cc0 0x55d2e05640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.423+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 0x7f0ad4079a80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400f540 tx=0x7f0ae4002ba0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.423+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55d2e0563cc0 0x55d2e05640d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.423+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec05e040 con 0x55d2e0563cc0 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.423+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0aec061070 0x7f0aec055410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.423+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:2 23.20 23:04000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec061070 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.424+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0aec061070 0x7f0aec055410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.424+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0aec061070 0x7f0aec055410 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.425+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(2 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.425+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0ae4010bd0 0x7f0ae401c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.425+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:3 23.10 23:08000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae4010bd0 2024-09-17T03:59:57.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.425+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0ae4010bd0 0x7f0ae401c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.425+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0ae4010bd0 0x7f0ae401c920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.426+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(3 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.426+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:4 23.30 23:0c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x55d2e0563cc0 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.426+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(4 [pgnls start_epoch 152 out=81b] v152'4 uv4 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f0aec05e040 con 0x55d2e0563cc0 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.426+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 0x7f0aec06c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.426+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:5 23.8 23:10000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec022dd0 2024-09-17T03:59:57.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.427+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 0x7f0aec06c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.427+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 0x7f0aec06c450 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.428+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(5 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0aec022dd0 2024-09-17T03:59:57.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.428+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 0x7f0aec003920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.428+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:6 23.28 23:14000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec01f750 2024-09-17T03:59:57.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.428+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 0x7f0aec003920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.428+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 0x7f0aec003920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(6 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:7 23.18 23:18000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae4010bd0 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(7 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:8 23.38 23:1c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec01f750 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(8 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.429+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f0ae402c3a0 0x7f0ae402c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.430+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:9 23.4 23:20000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae402c3a0 2024-09-17T03:59:57.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.430+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f0ae402c3a0 0x7f0ae402c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.430+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f0ae402c3a0 0x7f0ae402c7b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(9 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:10 23.24 23:24000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae4010bd0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(10 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:11 23.14 23:28000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec061070 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(11 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.431+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 23.34 23:2c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec01f750 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(12 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:13 23.c 23:30000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae4010bd0 2024-09-17T03:59:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(13 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af1200640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0ae004a040 0x7f0ae0060770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:14 23.2c 23:34000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae004a040 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.432+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0ae004a040 0x7f0ae0060770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.433+0000 7f0af2600640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0ae004a040 0x7f0ae0060770 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.433+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(14 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.433+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:15 23.1c 23:38000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae402c3a0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(15 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:16 23.3c 23:3c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae4010bd0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(16 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:17 23.2 23:40000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec061070 2024-09-17T03:59:57.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(17 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0ae402f930 0x7f0ae401b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:57.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.434+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:18 23.22 23:44000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae402f930 2024-09-17T03:59:57.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.435+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0ae402f930 0x7f0ae401b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:57.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.435+0000 7f0af1c00640 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0ae402f930 0x7f0ae401b3a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.436+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(18 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0ae402f930 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.436+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:19 23.12 23:48000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0aec061070 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.436+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(19 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.436+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:20 23.32 23:4c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x55d2e056ed70 con 0x7f0ae004a040 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(20 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:21 23.a 23:50000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(21 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:22 23.2a 23:54000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(22 [pgnls start_epoch 152 out=81b] v152'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.437+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 23.1a 23:58000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.438+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(23 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.438+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:24 23.3a 23:5c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.438+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(24 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.438+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:25 23.6 23:60000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(25 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 23.26 23:64000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(26 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:27 23.16 23:68000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(27 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.439+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:28 23.36 23:6c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.440+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(28 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.440+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:29 23.e 23:70000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae4010bd0 2024-09-17T03:59:57.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.440+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(29 [pgnls start_epoch 152 out=49b] v152'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.440+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:30 23.2e 23:74000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.440+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(30 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.441+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:31 23.1e 23:78000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.441+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(31 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.441+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:32 23.3e 23:7c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.441+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(32 [pgnls start_epoch 152 out=49b] v152'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.441+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:33 23.1 23:80000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(33 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:34 23.21 23:84000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402f930 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(34 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0ae402f930 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:35 23.11 23:88000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(35 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.442+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:36 23.31 23:8c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec022dd0 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.443+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(36 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0aec022dd0 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.443+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:37 23.9 23:90000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.443+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(37 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.443+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:38 23.29 23:94000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(38 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:39 23.19 23:98000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(39 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:40 23.39 23:9c000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(40 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:41 23.5 23:a0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(41 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.444+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:42 23.25 23:a4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.445+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(42 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.445+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:43 23.15 23:a8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402f930 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.445+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(43 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0ae402f930 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.445+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:44 23.35 23:ac000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 7 ==== osd_op_reply(44 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:45 23.d 23:b0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(45 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:46 23.2d 23:b4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(46 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.446+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:47 23.1d 23:b8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.447+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(47 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.447+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:48 23.3d 23:bc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.447+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 8 ==== osd_op_reply(48 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.447+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:49 23.3 23:c0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402f930 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.448+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(49 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0ae402f930 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.448+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:50 23.23 23:c4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.448+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 9 ==== osd_op_reply(50 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.448+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:51 23.13 23:c8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.449+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(51 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.449+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:52 23.33 23:cc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 10 ==== osd_op_reply(52 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:53 23.b 23:d0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(53 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:54 23.2b 23:d4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(54 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.450+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:55 23.1b 23:d8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae4010bd0 2024-09-17T03:59:57.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.451+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(55 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae4010bd0 2024-09-17T03:59:57.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.451+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:56 23.3b 23:dc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.451+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 8 ==== osd_op_reply(56 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.451+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 23.7 23:e0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x55d2e0563cc0 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(57 [pgnls start_epoch 152 out=49b] v152'3 uv3 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x55d2e0563cc0 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:58 23.27 23:e4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae004a040 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.5 v2:172.21.15.181:6818/3094080342 11 ==== osd_op_reply(58 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0ae004a040 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:59 23.17 23:e8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(59 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.452+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:60 23.37 23:ec000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec01f750 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.453+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.6 v2:172.21.15.181:6810/4136500368 9 ==== osd_op_reply(60 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec01f750 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.453+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:61 23.f 23:f0000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec061070 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.453+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.3 v2:172.21.15.129:6808/2225231477 10 ==== osd_op_reply(61 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0aec061070 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.453+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:62 23.2f 23:f4000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402c3a0 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.454+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 <== osd.2 v2:172.21.15.129:6809/283507149 9 ==== osd_op_reply(62 [pgnls start_epoch 152 out=49b] v152'2 uv2 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae0021070 con 0x7f0ae402c3a0 2024-09-17T03:59:57.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.454+0000 7f0af1200640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:63 23.1f 23:f8000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0ae402f930 2024-09-17T03:59:57.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.454+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(63 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0ae4010490 con 0x7f0ae402f930 2024-09-17T03:59:57.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.454+0000 7f0af1c00640 1 -- 172.21.15.129:0/3336907495 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:64 23.3f 23:fc000000::::head [pgnls start_epoch 152 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e152) -- 0x7f0aec087040 con 0x7f0aec022dd0 2024-09-17T03:59:57.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(64 [pgnls start_epoch 152 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0aec077040 con 0x7f0aec022dd0 2024-09-17T03:59:57.456 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000000 2024-09-17T03:59:57.456 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000001 2024-09-17T03:59:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55d2e0563cc0 msgr2=0x55d2e05640d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55d2e0563cc0 0x55d2e05640d0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0ae4010bd0 msgr2=0x7f0ae401c920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0ae4010bd0 0x7f0ae401c920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f0ae402c3a0 msgr2=0x7f0ae402c7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f0ae402c3a0 0x7f0ae402c7b0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0aec061070 msgr2=0x7f0aec055410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0aec061070 0x7f0aec055410 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0ae402f930 msgr2=0x7f0ae401b3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0ae402f930 0x7f0ae401b3a0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0ae004a040 msgr2=0x7f0ae0060770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0ae004a040 0x7f0ae0060770 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 msgr2=0x7f0aec003920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 reap_dead start 2024-09-17T03:59:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 0x7f0aec003920 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af2600640 1 -- 172.21.15.129:0/3336907495 reap_dead start 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 msgr2=0x7f0aec06c450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 0x7f0aec06c450 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 msgr2=0x7f0ad4079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 0x7f0ad4079a80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400f540 tx=0x7f0ae4002ba0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 msgr2=0x55d2e04c1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f0ae000ed80 tx=0x7f0ae000c620 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 shutdown_connections 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0aec022dd0 0x7f0aec06c450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0ad4077570 0x7f0ad4079a80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0aec01f750 0x7f0aec003920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 --2- 172.21.15.129:0/3336907495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d2e043f5c0 0x55d2e04c1830 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:57.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 >> 172.21.15.129:0/3336907495 conn(0x55d2e0314ac0 msgr2=0x55d2e04c06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:57.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 shutdown_connections 2024-09-17T03:59:57.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:57.455+0000 7f0af8b15f40 1 -- 172.21.15.129:0/3336907495 wait complete. 2024-09-17T03:59:57.468 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2024-09-17T03:59:57.468 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T03:59:57.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:57.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T03:59:57.642 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T03:59:57.642 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:57.642 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:57.642 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:57.678 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:57.678 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:57.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 -- 172.21.15.129:0/1608902546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec0705a0 msgr2=0x7f05ec0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 --2- 172.21.15.129:0/1608902546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec0705a0 0x7f05ec0b4140 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f05dc009830 tx=0x7f05dc02a5c0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 -- 172.21.15.129:0/1608902546 shutdown_connections 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 --2- 172.21.15.129:0/1608902546 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec0705a0 0x7f05ec0b4140 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 -- 172.21.15.129:0/1608902546 >> 172.21.15.129:0/1608902546 conn(0x7f05ec06d410 msgr2=0x7f05ec0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 -- 172.21.15.129:0/1608902546 shutdown_connections 2024-09-17T03:59:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.018+0000 7f05f0c00640 1 -- 172.21.15.129:0/1608902546 wait complete. 2024-09-17T03:59:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.020+0000 7f05f0c00640 1 Processor -- start 2024-09-17T03:59:58.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.020+0000 7f05f0c00640 1 -- start start 2024-09-17T03:59:58.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 0x7f05ec1f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 0x7f05ec1f92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec06f2e0 con 0x7f05ec06eed0 2024-09-17T03:59:58.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05ec1612e0 con 0x7f05ec0705a0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.021+0000 7f05f0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05ec06f460 con 0x7f05ec1571d0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48760/0 (socket says 172.21.15.129:48760) 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 -- 172.21.15.129:0/203336746 learned_addr learned my addr 172.21.15.129:0/203336746 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 -- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 msgr2=0x7f05ec1f92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05ea800640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 0x7f05ec1f92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9400640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 0x7f05ec1f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 0x7f05ec1f92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 -- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 msgr2=0x7f05ec1f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 0x7f05ec1f8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 -- 172.21.15.129:0/203336746 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ec15bab0 con 0x7f05ec06eed0 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05ea800640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 0x7f05ec1f92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:58.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9400640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 0x7f05ec1f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.022+0000 7f05e9e00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000c00 tx=0x7f05dc033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.023+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc031b50 con 0x7f05ec06eed0 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.023+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f05dc031de0 con 0x7f05ec06eed0 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.023+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ec15bd90 con 0x7f05ec06eed0 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.023+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc0332d0 con 0x7f05ec06eed0 2024-09-17T03:59:58.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.023+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ec15d8f0 con 0x7f05ec06eed0 2024-09-17T03:59:58.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.024+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ec15db50 con 0x7f05ec06eed0 2024-09-17T03:59:58.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.025+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f05dc033470 con 0x7f05ec06eed0 2024-09-17T03:59:58.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.026+0000 7f05e2000640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 0x7f05cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.026+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f05dc0bc0c0 con 0x7f05ec06eed0 2024-09-17T03:59:58.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.026+0000 7f05e9400640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 0x7f05cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.027+0000 7f05e9400640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 0x7f05cc07b0d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f05ec1f8d10 tx=0x7f05d8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:58.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.030+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f05dc03f030 con 0x7f05ec06eed0 2024-09-17T03:59:58.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.226+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f05ec0b4140 con 0x7f05ec06eed0 2024-09-17T03:59:58.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.227+0000 7f05e2000640 1 -- 172.21.15.129:0/203336746 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v152) ==== 80+0+0 (secure 0 0 0) 0x7f05dc084460 con 0x7f05ec06eed0 2024-09-17T03:59:58.227 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 msgr2=0x7f05cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 0x7f05cc07b0d0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f05ec1f8d10 tx=0x7f05d8009040 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 msgr2=0x7f05ec0c5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f05dc000c00 tx=0x7f05dc033ba0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 shutdown_connections 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f05cc078bc0 0x7f05cc07b0d0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05ec1571d0 0x7f05ec1f92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05ec0705a0 0x7f05ec1f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 --2- 172.21.15.129:0/203336746 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05ec06eed0 0x7f05ec0c5b20 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 >> 172.21.15.129:0/203336746 conn(0x7f05ec06d410 msgr2=0x7f05ec073b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 shutdown_connections 2024-09-17T03:59:58.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.229+0000 7f05f0c00640 1 -- 172.21.15.129:0/203336746 wait complete. 2024-09-17T03:59:58.255 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T03:59:58.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:58.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T03:59:58.357 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T03:59:58.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T03:59:58.357 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T03:59:58.401 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:58.401 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:58.401 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:58.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:58.437 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:58.437 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T03:59:58.438 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:58.477 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T03:59:58.477 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T03:59:58.477 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T03:59:58.478 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T03:59:58.513 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T03:59:58.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 -- 172.21.15.129:0/3797998335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c0ccc80 msgr2=0x7f1e4c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 --2- 172.21.15.129:0/3797998335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0cd090 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0ce800 tx=0x7f1e3802a540 comp rx=0 tx=0).stop 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 -- 172.21.15.129:0/3797998335 shutdown_connections 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 --2- 172.21.15.129:0/3797998335 >> v2:172.21.15.181:3300/0 conn(0x7f1e4c070100 0x7f1e4c070510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 --2- 172.21.15.129:0/3797998335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0cd090 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 -- 172.21.15.129:0/3797998335 >> 172.21.15.129:0/3797998335 conn(0x7f1e4c06d410 msgr2=0x7f1e4c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.737+0000 7f1e53800640 1 -- 172.21.15.129:0/3797998335 shutdown_connections 2024-09-17T03:59:58.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.738+0000 7f1e53800640 1 -- 172.21.15.129:0/3797998335 wait complete. 2024-09-17T03:59:58.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.740+0000 7f1e53800640 1 Processor -- start 2024-09-17T03:59:58.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.740+0000 7f1e53800640 1 -- start start 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1e4c070100 0x7f1e4c0b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c070a50 con 0x7f1e4c15da00 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4c15d160 con 0x7f1e4c0ccc80 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e53800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4c070bd0 con 0x7f1e4c070100 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.741+0000 7f1e51800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48786/0 (socket says 172.21.15.129:48786) 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 -- 172.21.15.129:0/2796487997 learned_addr learned my addr 172.21.15.129:0/2796487997 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e4be00640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e50e00640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1e4c070100 0x7f1e4c0b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 -- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1e4c070100 msgr2=0x7f1e4c0b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1e4c070100 0x7f1e4c0b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 -- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c0ccc80 msgr2=0x7f1e4c0b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 -- 172.21.15.129:0/2796487997 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e4c1590b0 con 0x7f1e4c15da00 2024-09-17T03:59:58.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.742+0000 7f1e51800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400e990 tx=0x7f1e4400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:58.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.743+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44004d20 con 0x7f1e4c15da00 2024-09-17T03:59:58.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.743+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e4400ca40 con 0x7f1e4c15da00 2024-09-17T03:59:58.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.743+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e44005020 con 0x7f1e4c15da00 2024-09-17T03:59:58.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.743+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e4c158360 con 0x7f1e4c15da00 2024-09-17T03:59:58.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.743+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e4c158af0 con 0x7f1e4c15da00 2024-09-17T03:59:58.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.744+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4c070510 con 0x7f1e4c15da00 2024-09-17T03:59:58.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.745+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1e440051c0 con 0x7f1e4c15da00 2024-09-17T03:59:58.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.746+0000 7f1e48c00640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 0x7f1e2c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:58.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.746+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f1e4409b610 con 0x7f1e4c15da00 2024-09-17T03:59:58.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.747+0000 7f1e50e00640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 0x7f1e2c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:58.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.747+0000 7f1e50e00640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 0x7f1e2c07b0a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1e40006fd0 tx=0x7f1e40009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:58.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.750+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1e44016030 con 0x7f1e4c15da00 2024-09-17T03:59:58.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.969+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e4c06a140 con 0x7f1e4c15da00 2024-09-17T03:59:58.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.969+0000 7f1e48c00640 1 -- 172.21.15.129:0/2796487997 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f1e44063360 con 0x7f1e4c15da00 2024-09-17T03:59:58.970 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:58.970 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:58.970 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:58.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 msgr2=0x7f1e2c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 0x7f1e2c07b0a0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f1e40006fd0 tx=0x7f1e40009060 comp rx=0 tx=0).stop 2024-09-17T03:59:58.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 msgr2=0x7f1e4c1f5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:58.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f1e4400e990 tx=0x7f1e4400ee50 comp rx=0 tx=0).stop 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 shutdown_connections 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e2c078b90 0x7f1e2c07b0a0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e4c15da00 0x7f1e4c1f5ef0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1e4c0ccc80 0x7f1e4c0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 --2- 172.21.15.129:0/2796487997 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1e4c070100 0x7f1e4c0b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 >> 172.21.15.129:0/2796487997 conn(0x7f1e4c06d410 msgr2=0x7f1e4c0b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 shutdown_connections 2024-09-17T03:59:58.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:58.972+0000 7f1e53800640 1 -- 172.21.15.129:0/2796487997 wait complete. 2024-09-17T03:59:59.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 -- 172.21.15.129:0/4175881144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0cd390 msgr2=0x7f444c06f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 --2- 172.21.15.129:0/4175881144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0cd390 0x7f444c06f640 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f444c071da0 tx=0x7f444402aac0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 -- 172.21.15.129:0/4175881144 shutdown_connections 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 --2- 172.21.15.129:0/4175881144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0cd390 0x7f444c06f640 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 --2- 172.21.15.129:0/4175881144 >> v2:172.21.15.181:3300/0 conn(0x7f444c0af4c0 0x7f444c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 -- 172.21.15.129:0/4175881144 >> 172.21.15.129:0/4175881144 conn(0x7f444c06d410 msgr2=0x7f444c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 -- 172.21.15.129:0/4175881144 shutdown_connections 2024-09-17T03:59:59.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.220+0000 7f4452200640 1 -- 172.21.15.129:0/4175881144 wait complete. 2024-09-17T03:59:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.222+0000 7f4452200640 1 Processor -- start 2024-09-17T03:59:59.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.223+0000 7f4452200640 1 -- start start 2024-09-17T03:59:59.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 0x7f444c0c4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 0x7f444c158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f444c06fb80 con 0x7f444c0b1770 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f444c0711c0 con 0x7f444c0af4c0 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f4452200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f444c06fd00 con 0x7f444c15b340 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f444be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f444be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48818/0 (socket says 172.21.15.129:48818) 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.224+0000 7f444be00640 1 -- 172.21.15.129:0/510806167 learned_addr learned my addr 172.21.15.129:0/510806167 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:59.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444b400640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 0x7f444c0c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 -- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 msgr2=0x7f444c158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444aa00640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 0x7f444c158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 0x7f444c158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 -- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 msgr2=0x7f444c0c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 0x7f444c0c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 -- 172.21.15.129:0/510806167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c0b0870 con 0x7f444c0b1770 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444aa00640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 0x7f444c158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444b400640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 0x7f444c0c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f444be00640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f4444009d60 tx=0x7f4444003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.225+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4444044070 con 0x7f444c0b1770 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.226+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4444003e50 con 0x7f444c0b1770 2024-09-17T03:59:59.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.226+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44440044b0 con 0x7f444c0b1770 2024-09-17T03:59:59.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.226+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444c0b0b50 con 0x7f444c0b1770 2024-09-17T03:59:59.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.226+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444c0c3d10 con 0x7f444c0b1770 2024-09-17T03:59:59.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.227+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f444c0c3f70 con 0x7f444c0b1770 2024-09-17T03:59:59.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.228+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4444004650 con 0x7f444c0b1770 2024-09-17T03:59:59.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.229+0000 7f4443400640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 0x7f442c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.229+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f44440bb3a0 con 0x7f444c0b1770 2024-09-17T03:59:59.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.229+0000 7f444b400640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 0x7f442c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.230+0000 7f444b400640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 0x7f442c07b0a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f443c009b80 tx=0x7f443c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:59.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.234+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4444032030 con 0x7f444c0b1770 2024-09-17T03:59:59.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.451+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f444c15ae00 con 0x7f444c0b1770 2024-09-17T03:59:59.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.452+0000 7f4443400640 1 -- 172.21.15.129:0/510806167 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7f44440837d0 con 0x7f444c0b1770 2024-09-17T03:59:59.452 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:59.452 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T03:59:59.452 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T03:59:59.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 msgr2=0x7f442c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 0x7f442c07b0a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f443c009b80 tx=0x7f443c005f90 comp rx=0 tx=0).stop 2024-09-17T03:59:59.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 msgr2=0x7f444c158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f4444009d60 tx=0x7f4444003ce0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 shutdown_connections 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f442c078b90 0x7f442c07b0a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f444c0b1770 0x7f444c158f40 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f444c15b340 0x7f444c158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 --2- 172.21.15.129:0/510806167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f444c0af4c0 0x7f444c0c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 >> 172.21.15.129:0/510806167 conn(0x7f444c06d410 msgr2=0x7f444c15c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 shutdown_connections 2024-09-17T03:59:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.454+0000 7f4452200640 1 -- 172.21.15.129:0/510806167 wait complete. 2024-09-17T03:59:59.479 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T03:59:59.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 -- 172.21.15.129:0/3157814903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f566415da00 msgr2=0x7f56640b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 --2- 172.21.15.129:0/3157814903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f566415da00 0x7f56640b4180 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f5654005fd0 tx=0x7f565402a5d0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 -- 172.21.15.129:0/3157814903 shutdown_connections 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 --2- 172.21.15.129:0/3157814903 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f56640b46c0 0x7f56640ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 --2- 172.21.15.129:0/3157814903 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f566415da00 0x7f56640b4180 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 -- 172.21.15.129:0/3157814903 >> 172.21.15.129:0/3157814903 conn(0x7f566406d410 msgr2=0x7f56640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 -- 172.21.15.129:0/3157814903 shutdown_connections 2024-09-17T03:59:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.699+0000 7f566a600640 1 -- 172.21.15.129:0/3157814903 wait complete. 2024-09-17T03:59:59.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.701+0000 7f566a600640 1 Processor -- start 2024-09-17T03:59:59.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.702+0000 7f566a600640 1 -- start start 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56640b46c0 0x7f56641588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 0x7f5664157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5664070990 con 0x7f56640cd1c0 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5664070810 con 0x7f56640b46c0 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f566a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5664070b10 con 0x7f566415da00 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f565aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f565aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48832/0 (socket says 172.21.15.129:48832) 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.703+0000 7f565aa00640 1 -- 172.21.15.129:0/1985037992 learned_addr learned my addr 172.21.15.129:0/1985037992 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T03:59:59.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f5663400640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56640b46c0 0x7f56641588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f5663e00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 0x7f5664157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 -- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 msgr2=0x7f5664157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 0x7f5664157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 -- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56640b46c0 msgr2=0x7f56641588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56640b46c0 0x7f56641588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 -- 172.21.15.129:0/1985037992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5664156090 con 0x7f56640cd1c0 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f5663e00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 0x7f5664157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565aa00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f564c00d770 tx=0x7f564c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c017070 con 0x7f56640cd1c0 2024-09-17T03:59:59.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f564c009d20 con 0x7f56640cd1c0 2024-09-17T03:59:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.704+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c0106c0 con 0x7f56640cd1c0 2024-09-17T03:59:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.705+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5664156370 con 0x7f56640cd1c0 2024-09-17T03:59:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.705+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641595c0 con 0x7f56640cd1c0 2024-09-17T03:59:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.706+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566406a140 con 0x7f56640cd1c0 2024-09-17T03:59:59.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.707+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f564c01a070 con 0x7f56640cd1c0 2024-09-17T03:59:59.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.708+0000 7f565be00640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 0x7f564007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T03:59:59.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.708+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f564c09ad30 con 0x7f56640cd1c0 2024-09-17T03:59:59.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.708+0000 7f5663400640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 0x7f564007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T03:59:59.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.709+0000 7f5663400640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 0x7f564007b0d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5654033000 tx=0x7f5654004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T03:59:59.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.712+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f564c062ba0 con 0x7f56640cd1c0 2024-09-17T03:59:59.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.896+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56640ad4c0 con 0x7f56640cd1c0 2024-09-17T03:59:59.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.897+0000 7f565be00640 1 -- 172.21.15.129:0/1985037992 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16779 (secure 0 0 0) 0x7f564c0675d0 con 0x7f56640cd1c0 2024-09-17T03:59:59.897 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T03:59:59.898 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":152,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:42.980414+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:59:39.383572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-17T03:59:40.444542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T03:59:59.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 msgr2=0x7f564007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 0x7f564007b0d0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f5654033000 tx=0x7f5654004730 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 msgr2=0x7f5664158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f564c00d770 tx=0x7f564c00dc30 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 shutdown_connections 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5640078bc0 0x7f564007b0d0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f566415da00 0x7f5664157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56640cd1c0 0x7f5664158e10 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 --2- 172.21.15.129:0/1985037992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f56640b46c0 0x7f56641588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 >> 172.21.15.129:0/1985037992 conn(0x7f566406d410 msgr2=0x7f5664099610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 shutdown_connections 2024-09-17T03:59:59.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T03:59:59.899+0000 7f566a600640 1 -- 172.21.15.129:0/1985037992 wait complete. 2024-09-17T03:59:59.923 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T03:59:59.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 -- 172.21.15.129:0/3530051435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4b0004800 msgr2=0x7ff4b0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/3530051435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4b0004800 0x7ff4b0004c10 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400c4a0 tx=0x7ff4b0005830 comp rx=0 tx=0).stop 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 -- 172.21.15.129:0/3530051435 shutdown_connections 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/3530051435 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4b0004800 0x7ff4b0004c10 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 -- 172.21.15.129:0/3530051435 >> 172.21.15.129:0/3530051435 conn(0x7ff4cc08bff0 msgr2=0x7ff4cc09d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 -- 172.21.15.129:0/3530051435 shutdown_connections 2024-09-17T04:00:00.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.147+0000 7ff4d0c00640 1 -- 172.21.15.129:0/3530051435 wait complete. 2024-09-17T04:00:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.149+0000 7ff4d0c00640 1 Processor -- start 2024-09-17T04:00:00.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.149+0000 7ff4d0c00640 1 -- start start 2024-09-17T04:00:00.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4b0004800 0x7ff4cc0ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4cc14d7b0 0x7ff4cc14dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc0d0f90 con 0x7ff4cc14e100 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc0b3690 con 0x7ff4cc14d7b0 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4d0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc0d1110 con 0x7ff4b0004800 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48858/0 (socket says 172.21.15.129:48858) 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4cbe00640 1 -- 172.21.15.129:0/1902226222 learned_addr learned my addr 172.21.15.129:0/1902226222 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4cb400640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4b0004800 0x7ff4cc0ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.151+0000 7ff4c2a00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4cc14d7b0 0x7ff4cc14dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 -- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4b0004800 msgr2=0x7ff4cc0ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4b0004800 0x7ff4cc0ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 -- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4cc14d7b0 msgr2=0x7ff4cc14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4cc14d7b0 0x7ff4cc14dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 -- 172.21.15.129:0/1902226222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc1f6300 con 0x7ff4cc14e100 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4cbe00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4005fd0 tx=0x7ff4c40099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004b20 con 0x7ff4cc14e100 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4c4004db0 con 0x7ff4cc14e100 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4011970 con 0x7ff4cc14e100 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc0cbb90 con 0x7ff4cc14e100 2024-09-17T04:00:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.152+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc0b7590 con 0x7ff4cc14e100 2024-09-17T04:00:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.153+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4cc1f97d0 con 0x7ff4cc14e100 2024-09-17T04:00:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.154+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff4c401b020 con 0x7ff4cc14e100 2024-09-17T04:00:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.155+0000 7ff4c3e00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 0x7ff4a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.156+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7ff4c409afc0 con 0x7ff4cc14e100 2024-09-17T04:00:00.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.156+0000 7ff4cb400640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 0x7ff4a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.156+0000 7ff4cb400640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 0x7ff4a807b0a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:00.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.160+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff4c4063400 con 0x7ff4cc14e100 2024-09-17T04:00:00.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.380+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4cc0d1670 con 0x7ff4cc14e100 2024-09-17T04:00:00.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.381+0000 7ff4c3e00640 1 -- 172.21.15.129:0/1902226222 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7ff4c4062d00 con 0x7ff4cc14e100 2024-09-17T04:00:00.381 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:00.381 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T04:00:00.381 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 msgr2=0x7ff4a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 0x7ff4a807b0a0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc006b70 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 msgr2=0x7ff4cc0b7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4005fd0 tx=0x7ff4c40099c0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 shutdown_connections 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.383+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4a8078b90 0x7ff4a807b0a0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4cc14e100 0x7ff4cc0b7ab0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4cc14d7b0 0x7ff4cc14dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 --2- 172.21.15.129:0/1902226222 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4b0004800 0x7ff4cc0ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 >> 172.21.15.129:0/1902226222 conn(0x7ff4cc08bff0 msgr2=0x7ff4cc0b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:00.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 shutdown_connections 2024-09-17T04:00:00.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.384+0000 7ff4d0c00640 1 -- 172.21.15.129:0/1902226222 wait complete. 2024-09-17T04:00:00.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 -- 172.21.15.129:0/2470366980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 msgr2=0x7fb5280d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 --2- 172.21.15.129:0/2470366980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280d0fc0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fb5280d5040 tx=0x7fb51002a540 comp rx=0 tx=0).stop 2024-09-17T04:00:00.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 -- 172.21.15.129:0/2470366980 shutdown_connections 2024-09-17T04:00:00.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 --2- 172.21.15.129:0/2470366980 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280d1500 0x7fb5280d1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 --2- 172.21.15.129:0/2470366980 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280d0fc0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 --2- 172.21.15.129:0/2470366980 >> v2:172.21.15.181:3300/0 conn(0x7fb5280b3220 0x7fb5280d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 -- 172.21.15.129:0/2470366980 >> 172.21.15.129:0/2470366980 conn(0x7fb52809d8d0 msgr2=0x7fb52808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:00.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 -- 172.21.15.129:0/2470366980 shutdown_connections 2024-09-17T04:00:00.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.633+0000 7fb52de00640 1 -- 172.21.15.129:0/2470366980 wait complete. 2024-09-17T04:00:00.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.635+0000 7fb52de00640 1 Processor -- start 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.635+0000 7fb52de00640 1 -- start start 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280b3220 0x7fb52814e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 0x7fb5280c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280ba040 con 0x7fb5280d4a80 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb5280b9ec0 con 0x7fb5280d1500 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb5280d0ae0 con 0x7fb5280b3220 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48874/0 (socket says 172.21.15.129:48874) 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52d400640 1 -- 172.21.15.129:0/3379353176 learned_addr learned my addr 172.21.15.129:0/3379353176 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb52ca00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280b3220 0x7fb52814e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.637+0000 7fb51fe00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 0x7fb5280c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 -- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280b3220 msgr2=0x7fb52814e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280b3220 0x7fb52814e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 -- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 msgr2=0x7fb5280c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 0x7fb5280c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 -- 172.21.15.129:0/3379353176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5280c9c20 con 0x7fb5280d4a80 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb51fe00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 0x7fb5280c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52d400640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fb52000da10 tx=0x7fb52000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520017070 con 0x7fb5280d4a80 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb520003970 con 0x7fb5280d4a80 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52000be60 con 0x7fb5280d4a80 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52815dc80 con 0x7fb5280d4a80 2024-09-17T04:00:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.638+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5281603a0 con 0x7fb5280d4a80 2024-09-17T04:00:00.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.639+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5281601d0 con 0x7fb5280d4a80 2024-09-17T04:00:00.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.641+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb520010460 con 0x7fb5280d4a80 2024-09-17T04:00:00.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.642+0000 7fb525600640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 0x7fb50407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:00.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.642+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7fb52009b220 con 0x7fb5280d4a80 2024-09-17T04:00:00.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.643+0000 7fb52ca00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 0x7fb50407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:00.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.643+0000 7fb52ca00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 0x7fb50407b0d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb518005fd0 tx=0x7fb518005da0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:00.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.647+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb5200635c0 con 0x7fb5280d4a80 2024-09-17T04:00:00.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.864+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb52815e1a0 con 0x7fb5280d4a80 2024-09-17T04:00:00.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.864+0000 7fb525600640 1 -- 172.21.15.129:0/3379353176 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5245 (secure 0 0 0) 0x7fb52815e1a0 con 0x7fb5280d4a80 2024-09-17T04:00:00.864 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:00.865 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":135,"btime":"2024-09-17T03:59:45:009246+0000","default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T03:59:41.482169+0000","modified":"2024-09-17T03:59:44.493748+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8185":{"gid":8185,"name":"d","rank":0,"incarnation":131,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2593020253","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2593020253},{"type":"v1","addr":"172.21.15.181:6837","nonce":2593020253}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8185,"qdb_cluster":[8185]},"id":11}]} 2024-09-17T04:00:00.865 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 135 2024-09-17T04:00:00.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 msgr2=0x7fb50407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 0x7fb50407b0d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb518005fd0 tx=0x7fb518005da0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 msgr2=0x7fb5280c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fb52000da10 tx=0x7fb52000ded0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 shutdown_connections 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb504078bc0 0x7fb50407b0d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb5280d4a80 0x7fb5280c96e0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5280d1500 0x7fb5280c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 --2- 172.21.15.129:0/3379353176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb5280b3220 0x7fb52814e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 >> 172.21.15.129:0/3379353176 conn(0x7fb52809d8d0 msgr2=0x7fb5280b9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 shutdown_connections 2024-09-17T04:00:00.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:00.867+0000 7fb52de00640 1 -- 172.21.15.129:0/3379353176 wait complete. 2024-09-17T04:00:00.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.114+0000 7ff3ab000640 1 -- 172.21.15.129:0/2606735250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 msgr2=0x7ff3a40b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.114+0000 7ff3ab000640 1 --2- 172.21.15.129:0/2606735250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a40b3440 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff398009860 tx=0x7ff39802a5a0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.114+0000 7ff3ab000640 1 -- 172.21.15.129:0/2606735250 shutdown_connections 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.114+0000 7ff3ab000640 1 --2- 172.21.15.129:0/2606735250 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a40b3440 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.114+0000 7ff3ab000640 1 -- 172.21.15.129:0/2606735250 >> 172.21.15.129:0/2606735250 conn(0x7ff3a408bff0 msgr2=0x7ff3a409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.115+0000 7ff3ab000640 1 -- 172.21.15.129:0/2606735250 shutdown_connections 2024-09-17T04:00:01.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.115+0000 7ff3ab000640 1 -- 172.21.15.129:0/2606735250 wait complete. 2024-09-17T04:00:01.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.117+0000 7ff3ab000640 1 Processor -- start 2024-09-17T04:00:01.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.117+0000 7ff3ab000640 1 -- start start 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a41f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a40d4ba0 0x7ff3a41f9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a40d34f0 con 0x7ff3a40b9af0 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a40d3370 con 0x7ff3a40d4ba0 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.118+0000 7ff3ab000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff3a40cf7c0 con 0x7ff3a40d1850 2024-09-17T04:00:01.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48882/0 (socket says 172.21.15.129:48882) 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 -- 172.21.15.129:0/829222091 learned_addr learned my addr 172.21.15.129:0/829222091 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9200640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a41f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3aa600640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a40d4ba0 0x7ff3a41f9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 -- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 msgr2=0x7ff3a41f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a41f8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 -- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a40d4ba0 msgr2=0x7ff3a41f9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a40d4ba0 0x7ff3a41f9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.119+0000 7ff3a9c00640 1 -- 172.21.15.129:0/829222091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a40ccb60 con 0x7ff3a40b9af0 2024-09-17T04:00:01.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3a9c00640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7ff398009980 tx=0x7ff398033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:01.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398005c90 con 0x7ff3a40b9af0 2024-09-17T04:00:01.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff398031e50 con 0x7ff3a40b9af0 2024-09-17T04:00:01.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a40cce20 con 0x7ff3a40b9af0 2024-09-17T04:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff398005260 con 0x7ff3a40b9af0 2024-09-17T04:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.120+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a40cd3e0 con 0x7ff3a40b9af0 2024-09-17T04:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.121+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a40cfcb0 con 0x7ff3a40b9af0 2024-09-17T04:00:01.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.122+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff398005400 con 0x7ff3a40b9af0 2024-09-17T04:00:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.123+0000 7ff3a1600640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 0x7ff38407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.123+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7ff3980bbd80 con 0x7ff3a40b9af0 2024-09-17T04:00:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.124+0000 7ff3a9200640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 0x7ff38407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.124+0000 7ff3a9200640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 0x7ff38407b0d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4155280 tx=0x7ff390009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:01.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.128+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff39803f030 con 0x7ff3a40b9af0 2024-09-17T04:00:01.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.312+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3a40b4300 con 0x7ff3a40b9af0 2024-09-17T04:00:01.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.313+0000 7ff3a1600640 1 -- 172.21.15.129:0/829222091 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) ==== 74+0+16779 (secure 0 0 0) 0x7ff398083ad0 con 0x7ff3a40b9af0 2024-09-17T04:00:01.313 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:01.314 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":152,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T03:59:42.980414+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-17T03:59:39.383572+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"151","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-17T03:59:40.444542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:01.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 msgr2=0x7ff38407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 0x7ff38407b0d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4155280 tx=0x7ff390009040 comp rx=0 tx=0).stop 2024-09-17T04:00:01.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 msgr2=0x7ff3a41f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7ff398009980 tx=0x7ff398033c20 comp rx=0 tx=0).stop 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 shutdown_connections 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff384078bc0 0x7ff38407b0d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3a40d4ba0 0x7ff3a41f9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3a40d1850 0x7ff3a41f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 --2- 172.21.15.129:0/829222091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3a40b9af0 0x7ff3a41f5c30 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 >> 172.21.15.129:0/829222091 conn(0x7ff3a408bff0 msgr2=0x7ff3a40d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 shutdown_connections 2024-09-17T04:00:01.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.316+0000 7ff3ab000640 1 -- 172.21.15.129:0/829222091 wait complete. 2024-09-17T04:00:01.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 -- 172.21.15.129:0/1448899705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4180004d40 msgr2=0x7f4180005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 --2- 172.21.15.129:0/1448899705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4180004d40 0x7f4180005150 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f419c070550 tx=0x7f4180005830 comp rx=0 tx=0).stop 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a1000640 1 -- 172.21.15.129:0/1448899705 reap_dead start 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 -- 172.21.15.129:0/1448899705 shutdown_connections 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 -- 172.21.15.129:0/1448899705 >> 172.21.15.129:0/1448899705 conn(0x7f419c06d410 msgr2=0x7f419c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 -- 172.21.15.129:0/1448899705 shutdown_connections 2024-09-17T04:00:01.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.565+0000 7f41a3000640 1 -- 172.21.15.129:0/1448899705 wait complete. 2024-09-17T04:00:01.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.567+0000 7f41a3000640 1 Processor -- start 2024-09-17T04:00:01.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.568+0000 7f41a3000640 1 -- start start 2024-09-17T04:00:01.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c153370 0x7f419c153780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 0x7f419c1f92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f419c160260 con 0x7f419c0cb970 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1600e0 con 0x7f419c153370 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f41a3000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f419c1609c0 con 0x7f419c153cc0 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f419be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f419be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48898/0 (socket says 172.21.15.129:48898) 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.569+0000 7f419be00640 1 -- 172.21.15.129:0/3440149424 learned_addr learned my addr 172.21.15.129:0/3440149424 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f4193400640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c153370 0x7f419c153780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f41a1000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 0x7f419c1f92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 -- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 msgr2=0x7f419c1f92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 0x7f419c1f92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 -- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c153370 msgr2=0x7f419c153780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c153370 0x7f419c153780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 -- 172.21.15.129:0/3440149424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c15a220 con 0x7f419c0cb970 2024-09-17T04:00:01.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f41a1000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 0x7f419c1f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f419be00640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f418c006630 tx=0x7f418c003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c015070 con 0x7f419c0cb970 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f418c00f920 con 0x7f419c0cb970 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c00fd70 con 0x7f419c0cb970 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c15a500 con 0x7f419c0cb970 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.570+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c0cbd80 con 0x7f419c0cb970 2024-09-17T04:00:01.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.571+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419c0ccc80 con 0x7f419c0cb970 2024-09-17T04:00:01.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.572+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f418c011430 con 0x7f419c0cb970 2024-09-17T04:00:01.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.573+0000 7f4198c00640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 0x7f417807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:01.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.574+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(152..152 src has 1..152) ==== 5709+0+0 (secure 0 0 0) 0x7f418c09bae0 con 0x7f419c0cb970 2024-09-17T04:00:01.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.574+0000 7f4193400640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 0x7f417807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:01.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.575+0000 7f4193400640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 0x7f417807b0a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f419c06dda0 tx=0x7f4184005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:01.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.578+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f419c0cd5d0 con 0x7f419c0cb970 2024-09-17T04:00:01.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.795+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f419c06a140 con 0x7f419c0cb970 2024-09-17T04:00:01.795 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:00:01.795+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:00:01.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.798+0000 7f4198c00640 1 -- 172.21.15.129:0/3440149424 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v136) ==== 161+0+0 (secure 0 0 0) 0x7f418c063f10 con 0x7f419c0cb970 2024-09-17T04:00:01.798 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 msgr2=0x7f417807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 0x7f417807b0a0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f419c06dda0 tx=0x7f4184005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 msgr2=0x7f419c1f5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f418c006630 tx=0x7f418c003aa0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 shutdown_connections 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4178078b90 0x7f417807b0a0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f419c153cc0 0x7f419c1f92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f419c153370 0x7f419c153780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 --2- 172.21.15.129:0/3440149424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f419c0cb970 0x7f419c1f5440 secure :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f418c006630 tx=0x7f418c003aa0 comp rx=0 tx=0).stop 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.800+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 >> 172.21.15.129:0/3440149424 conn(0x7f419c06d410 msgr2=0x7f419c0ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.801+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 shutdown_connections 2024-09-17T04:00:01.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:01.801+0000 7f41a3000640 1 -- 172.21.15.129:0/3440149424 wait complete. 2024-09-17T04:00:01.814 INFO:tasks.ceph.mds.d.smithi181.stderr: -6426> 2024-09-17T03:59:46.955+0000 7f8b85800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:00:01.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:00:01.940 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:00:02.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 -- 172.21.15.129:0/1852142894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 msgr2=0x7f789c0d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:02.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 --2- 172.21.15.129:0/1852142894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c0d07e0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f7884009900 tx=0x7f788402a610 comp rx=0 tx=0).stop 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 -- 172.21.15.129:0/1852142894 shutdown_connections 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 --2- 172.21.15.129:0/1852142894 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c15b4c0 0x7f789c15b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 --2- 172.21.15.129:0/1852142894 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c0d07e0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 -- 172.21.15.129:0/1852142894 >> 172.21.15.129:0/1852142894 conn(0x7f789c06cfc0 msgr2=0x7f789c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 -- 172.21.15.129:0/1852142894 shutdown_connections 2024-09-17T04:00:02.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.047+0000 7f78a0a00640 1 -- 172.21.15.129:0/1852142894 wait complete. 2024-09-17T04:00:02.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.049+0000 7f78a0a00640 1 Processor -- start 2024-09-17T04:00:02.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.050+0000 7f78a0a00640 1 -- start start 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c073520 0x7f789c153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789c15b4c0 0x7f789c15e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0b4280 con 0x7f789c0d03d0 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f789c0b4100 con 0x7f789c15b4c0 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f78a0a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f789c0b4400 con 0x7f789c073520 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f7892a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:02.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f7892a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48912/0 (socket says 172.21.15.129:48912) 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f7892a00640 1 -- 172.21.15.129:0/3947142469 learned_addr learned my addr 172.21.15.129:0/3947142469 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f789b400640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c073520 0x7f789c153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.051+0000 7f789be00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789c15b4c0 0x7f789c15e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 -- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c073520 msgr2=0x7f789c153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c073520 0x7f789c153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 -- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789c15b4c0 msgr2=0x7f789c15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789c15b4c0 0x7f789c15e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 -- 172.21.15.129:0/3947142469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789c154750 con 0x7f789c0d03d0 2024-09-17T04:00:02.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7892a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7884000c00 tx=0x7f78840038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:02.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788403e030 con 0x7f789c0d03d0 2024-09-17T04:00:02.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7884003ca0 con 0x7f789c0d03d0 2024-09-17T04:00:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.052+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789c1f6080 con 0x7f789c0d03d0 2024-09-17T04:00:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.053+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7884035040 con 0x7f789c0d03d0 2024-09-17T04:00:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.053+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f789c1f6640 con 0x7f789c0d03d0 2024-09-17T04:00:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.054+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f789c0b4990 con 0x7f789c0d03d0 2024-09-17T04:00:02.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.055+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7884041030 con 0x7f789c0d03d0 2024-09-17T04:00:02.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.056+0000 7f7893e00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 0x7f787807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:02.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.056+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5795+0+0 (secure 0 0 0) 0x7f78840bb280 con 0x7f789c0d03d0 2024-09-17T04:00:02.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.057+0000 7f789b400640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 0x7f787807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:02.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.057+0000 7f789b400640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 0x7f787807b0d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f788c000f60 tx=0x7f788c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.061+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7884083000 con 0x7f789c0d03d0 2024-09-17T04:00:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.283+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f789c1549e0 con 0x7f789c0d03d0 2024-09-17T04:00:02.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.799+0000 7f7893e00640 1 -- 172.21.15.129:0/3947142469 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v137) ==== 104+0+0 (secure 0 0 0) 0x7f7884087a30 con 0x7f789c0d03d0 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 msgr2=0x7f787807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 0x7f787807b0d0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f788c000f60 tx=0x7f788c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 msgr2=0x7f789c15e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7884000c00 tx=0x7f78840038a0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 shutdown_connections 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7878078bc0 0x7f787807b0d0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f789c15b4c0 0x7f789c15e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f789c0d03d0 0x7f789c15e170 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 --2- 172.21.15.129:0/3947142469 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f789c073520 0x7f789c153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:02.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.802+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 >> 172.21.15.129:0/3947142469 conn(0x7f789c06cfc0 msgr2=0x7f789c158da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:02.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.803+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 shutdown_connections 2024-09-17T04:00:02.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:02.803+0000 7f78a0a00640 1 -- 172.21.15.129:0/3947142469 wait complete. 2024-09-17T04:00:02.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 -- 172.21.15.129:0/1969640858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 msgr2=0x7f153400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 --2- 172.21.15.129:0/1969640858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f153400b9c0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f1548001dc0 tx=0x7f1534005830 comp rx=0 tx=0).stop 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 -- 172.21.15.129:0/1969640858 shutdown_connections 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 --2- 172.21.15.129:0/1969640858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f153400b9c0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 -- 172.21.15.129:0/1969640858 >> 172.21.15.129:0/1969640858 conn(0x7f15500ccc40 msgr2=0x7f15500cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.058+0000 7f1555800640 1 -- 172.21.15.129:0/1969640858 shutdown_connections 2024-09-17T04:00:03.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.059+0000 7f1555800640 1 -- 172.21.15.129:0/1969640858 wait complete. 2024-09-17T04:00:03.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.061+0000 7f1555800640 1 Processor -- start 2024-09-17T04:00:03.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.061+0000 7f1555800640 1 -- start start 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f155009d370 0x7f15501f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155009d780 0x7f15501f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f15500b4d90 con 0x7f1534005350 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f15500b4c10 con 0x7f155009d780 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f1555800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f15500b1e20 con 0x7f155009d370 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.062+0000 7f154ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:03.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48928/0 (socket says 172.21.15.129:48928) 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 -- 172.21.15.129:0/3758858908 learned_addr learned my addr 172.21.15.129:0/3758858908 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 -- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f155009d370 msgr2=0x7f15501f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f155009d370 0x7f15501f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154f400640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155009d780 0x7f15501f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 -- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155009d780 msgr2=0x7f15501f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155009d780 0x7f15501f8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 -- 172.21.15.129:0/3758858908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15501e8210 con 0x7f1534005350 2024-09-17T04:00:03.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f154ea00640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1540011ab0 tx=0x7f1540011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154001e070 con 0x7f1534005350 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f154000fe70 con 0x7f1534005350 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.063+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15400042b0 con 0x7f1534005350 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.064+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155015be50 con 0x7f1534005350 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.064+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155015c5b0 con 0x7f1534005350 2024-09-17T04:00:03.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.065+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15500b2380 con 0x7f1534005350 2024-09-17T04:00:03.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.066+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1540021070 con 0x7f1534005350 2024-09-17T04:00:03.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.067+0000 7f1546a00640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 0x7f153007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:03.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.067+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(153..153 src has 1..153) ==== 5795+0+0 (secure 0 0 0) 0x7f154009b880 con 0x7f1534005350 2024-09-17T04:00:03.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.067+0000 7f154e000640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 0x7f153007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:03.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.068+0000 7f154e000640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 0x7f153007b0a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f155015cdd0 tx=0x7f153c000ed0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:03.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.071+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1540063570 con 0x7f1534005350 2024-09-17T04:00:03.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:03.253+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f15500b10e0 con 0x7f1534005350 2024-09-17T04:00:04.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.092+0000 7f1546a00640 1 -- 172.21.15.129:0/3758858908 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v154) ==== 181+0+0 (secure 0 0 0) 0x7f1540067fa0 con 0x7f1534005350 2024-09-17T04:00:04.093 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:00:04.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.096+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 msgr2=0x7f153007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:04.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.096+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 0x7f153007b0a0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f155015cdd0 tx=0x7f153c000ed0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.096+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 msgr2=0x7f155015b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:04.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.096+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f1540011ab0 tx=0x7f1540011f70 comp rx=0 tx=0).stop 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 shutdown_connections 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1530078b90 0x7f153007b0a0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155009d780 0x7f15501f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f155009d370 0x7f15501f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 --2- 172.21.15.129:0/3758858908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1534005350 0x7f155015b910 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 >> 172.21.15.129:0/3758858908 conn(0x7f15500ccc40 msgr2=0x7f15500afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 shutdown_connections 2024-09-17T04:00:04.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.097+0000 7f1555800640 1 -- 172.21.15.129:0/3758858908 wait complete. 2024-09-17T04:00:04.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:00:04.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 -- 172.21.15.129:0/692776961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0003f90 msgr2=0x7efff0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:04.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 --2- 172.21.15.129:0/692776961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efff0003f90 0x7efff0004800 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7efff8009b20 tx=0x7efff0005830 comp rx=0 tx=0).stop 2024-09-17T04:00:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0010a00640 1 -- 172.21.15.129:0/692776961 reap_dead start 2024-09-17T04:00:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 -- 172.21.15.129:0/692776961 shutdown_connections 2024-09-17T04:00:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 -- 172.21.15.129:0/692776961 >> 172.21.15.129:0/692776961 conn(0x7f000c0ccc40 msgr2=0x7f000c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 -- 172.21.15.129:0/692776961 shutdown_connections 2024-09-17T04:00:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.351+0000 7f0012a00640 1 -- 172.21.15.129:0/692776961 wait complete. 2024-09-17T04:00:04.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.353+0000 7f0012a00640 1 Processor -- start 2024-09-17T04:00:04.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.353+0000 7f0012a00640 1 -- start start 2024-09-17T04:00:04.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 0x7f000c1f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:04.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 0x7f000c0c8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0b4b20 con 0x7f000c09aaf0 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0b49a0 con 0x7f000c14e280 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0012a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f000c0b4ca0 con 0x7efff0003f90 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52524/0 (socket says 172.21.15.129:52524) 2024-09-17T04:00:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 -- 172.21.15.129:0/6081215 learned_addr learned my addr 172.21.15.129:0/6081215 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:04.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000be00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 0x7f000c1f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:04.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0010a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 0x7f000c0c8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:04.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 -- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 msgr2=0x7f000c1f84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:04.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 0x7f000c1f84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 -- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 msgr2=0x7f000c0c8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 0x7f000c0c8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000b400640 1 -- 172.21.15.129:0/6081215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c09cc50 con 0x7f000c09aaf0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f000be00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 0x7f000c1f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.355+0000 7f0010a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 0x7f000c0c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f000b400640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7efff8009a70 tx=0x7efff8033650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8038040 con 0x7f000c09aaf0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efff8041070 con 0x7f000c09aaf0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff8033d70 con 0x7f000c09aaf0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000c1f8e90 con 0x7f000c09aaf0 2024-09-17T04:00:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.356+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f000c0c8060 con 0x7f000c09aaf0 2024-09-17T04:00:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.357+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f000c0b3f70 con 0x7f000c09aaf0 2024-09-17T04:00:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.358+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efff802b870 con 0x7f000c09aaf0 2024-09-17T04:00:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.359+0000 7f0003e00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 0x7effec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.360+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(154..154 src has 1..154) ==== 5357+0+0 (secure 0 0 0) 0x7efff80bb590 con 0x7f000c09aaf0 2024-09-17T04:00:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.360+0000 7f000be00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 0x7effec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.361+0000 7f000be00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 0x7effec07b0a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efffc0011a0 tx=0x7efffc006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:04.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.364+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efff8083bb0 con 0x7f000c09aaf0 2024-09-17T04:00:04.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:04.547+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f000c1f8c20 con 0x7f000c09aaf0 2024-09-17T04:00:05.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.094+0000 7f0003e00640 1 -- 172.21.15.129:0/6081215 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v155) ==== 169+0+0 (secure 0 0 0) 0x7efff80834b0 con 0x7f000c09aaf0 2024-09-17T04:00:05.094 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:00:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.097+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 msgr2=0x7effec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.097+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 0x7effec07b0a0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7efffc0011a0 tx=0x7efffc006b10 comp rx=0 tx=0).stop 2024-09-17T04:00:05.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.097+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 msgr2=0x7f000c09c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.097+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7efff8009a70 tx=0x7efff8033650 comp rx=0 tx=0).stop 2024-09-17T04:00:05.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 shutdown_connections 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7effec078b90 0x7effec07b0a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f000c14e280 0x7f000c0c8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f000c09aaf0 0x7f000c09c710 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 --2- 172.21.15.129:0/6081215 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efff0003f90 0x7f000c1f84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 >> 172.21.15.129:0/6081215 conn(0x7f000c0ccc40 msgr2=0x7f000c0cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.098+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 shutdown_connections 2024-09-17T04:00:05.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.099+0000 7f0012a00640 1 -- 172.21.15.129:0/6081215 wait complete. 2024-09-17T04:00:05.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:00:05.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 -- 172.21.15.129:0/72239787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf41602b0 msgr2=0x7f2bf40cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 --2- 172.21.15.129:0/72239787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf41602b0 0x7f2bf40cee20 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40ce910 tx=0x7f2bdc02a560 comp rx=0 tx=0).stop 2024-09-17T04:00:05.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 -- 172.21.15.129:0/72239787 shutdown_connections 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 --2- 172.21.15.129:0/72239787 >> v2:172.21.15.181:3300/0 conn(0x7f2bf41606c0 0x7f2bf40cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 --2- 172.21.15.129:0/72239787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf41602b0 0x7f2bf40cee20 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 --2- 172.21.15.129:0/72239787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf415a220 0x7f2bf415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 -- 172.21.15.129:0/72239787 >> 172.21.15.129:0/72239787 conn(0x7f2bf4069150 msgr2=0x7f2bf4069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 -- 172.21.15.129:0/72239787 shutdown_connections 2024-09-17T04:00:05.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.347+0000 7f2bfa400640 1 -- 172.21.15.129:0/72239787 wait complete. 2024-09-17T04:00:05.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.349+0000 7f2bfa400640 1 Processor -- start 2024-09-17T04:00:05.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.349+0000 7f2bfa400640 1 -- start start 2024-09-17T04:00:05.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 0x7f2bf40b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf41606c0 0x7f2bf4158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf40d00d0 con 0x7f2bf415a220 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2bf40cff50 con 0x7f2bf41606c0 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bfa400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2bf40d0250 con 0x7f2bf41602b0 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52540/0 (socket says 172.21.15.129:52540) 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 -- 172.21.15.129:0/43432705 learned_addr learned my addr 172.21.15.129:0/43432705 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bebe00640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 0x7f2bf40b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9a00640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf41606c0 0x7f2bf4158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 -- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 msgr2=0x7f2bf40b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 0x7f2bf40b6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 -- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf41606c0 msgr2=0x7f2bf4158500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf41606c0 0x7f2bf4158500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bf9000640 1 -- 172.21.15.129:0/43432705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bf41f7160 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.351+0000 7f2bebe00640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 0x7f2bf40b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bf9000640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2be400e7c0 tx=0x7f2be400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be400ccc0 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2be400ce60 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bf41f7440 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4005280 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.352+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bf41f7a00 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.353+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bf41f7de0 con 0x7f2bf415a220 2024-09-17T04:00:05.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.354+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2be4005420 con 0x7f2bf415a220 2024-09-17T04:00:05.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.355+0000 7f2bf1600640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 0x7f2bd007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.355+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f2be409b4a0 con 0x7f2bf415a220 2024-09-17T04:00:05.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.356+0000 7f2bebe00640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 0x7f2bd007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.357+0000 7f2bebe00640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 0x7f2bd007b0a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc033000 tx=0x7f2bdc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:05.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.360+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2bf41f7de0 con 0x7f2bf415a220 2024-09-17T04:00:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.540+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f2bf40d0fa0 con 0x7f2bf415a220 2024-09-17T04:00:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.544+0000 7f2bf1600640 1 -- 172.21.15.129:0/43432705 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v55) ==== 71+0+0 (secure 0 0 0) 0x7f2be4063b40 con 0x7f2bf415a220 2024-09-17T04:00:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 msgr2=0x7f2bd007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 0x7f2bd007b0a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc033000 tx=0x7f2bdc004730 comp rx=0 tx=0).stop 2024-09-17T04:00:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 msgr2=0x7f2bf414f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2be400e7c0 tx=0x7f2be400ec80 comp rx=0 tx=0).stop 2024-09-17T04:00:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 shutdown_connections 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bd0078b90 0x7f2bd007b0a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bf41606c0 0x7f2bf4158500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bf41602b0 0x7f2bf40b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 --2- 172.21.15.129:0/43432705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bf415a220 0x7f2bf414f3a0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 >> 172.21.15.129:0/43432705 conn(0x7f2bf4069150 msgr2=0x7f2bf415c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 shutdown_connections 2024-09-17T04:00:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.547+0000 7f2bfa400640 1 -- 172.21.15.129:0/43432705 wait complete. 2024-09-17T04:00:05.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup' 2024-09-17T04:00:05.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 -- 172.21.15.129:0/882955453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80cfb50 msgr2=0x7f71d80cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 --2- 172.21.15.129:0/882955453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80cfb50 0x7f71d80cd3c0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f71d806dcc0 tx=0x7f71c402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 -- 172.21.15.129:0/882955453 shutdown_connections 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 --2- 172.21.15.129:0/882955453 >> v2:172.21.15.181:3300/0 conn(0x7f71d806ecb0 0x7f71d806f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 --2- 172.21.15.129:0/882955453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80cfb50 0x7f71d80cd3c0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 -- 172.21.15.129:0/882955453 >> 172.21.15.129:0/882955453 conn(0x7f71d806cfc0 msgr2=0x7f71d806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 -- 172.21.15.129:0/882955453 shutdown_connections 2024-09-17T04:00:05.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.791+0000 7f71dde00640 1 -- 172.21.15.129:0/882955453 wait complete. 2024-09-17T04:00:05.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.793+0000 7f71dde00640 1 Processor -- start 2024-09-17T04:00:05.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.793+0000 7f71dde00640 1 -- start start 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71d806ecb0 0x7f71d80ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d80cfb50 0x7f71d8141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f71d80d0730 con 0x7f71d80736a0 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f71d80d05b0 con 0x7f71d80cfb50 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dde00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f71d80d08b0 con 0x7f71d806ecb0 2024-09-17T04:00:05.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52550/0 (socket says 172.21.15.129:52550) 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 -- 172.21.15.129:0/2178181043 learned_addr learned my addr 172.21.15.129:0/2178181043 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dd400640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d80cfb50 0x7f71d8141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71dca00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71d806ecb0 0x7f71d80ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 -- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71d806ecb0 msgr2=0x7f71d80ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71d806ecb0 0x7f71d80ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 -- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d80cfb50 msgr2=0x7f71d8141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d80cfb50 0x7f71d8141b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.795+0000 7f71d7e00640 1 -- 172.21.15.129:0/2178181043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d8142330 con 0x7f71d80736a0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71d7e00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f71c40099c0 tx=0x7f71c40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c403e030 con 0x7f71d80736a0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f71c4003db0 con 0x7f71d80736a0 2024-09-17T04:00:05.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4035040 con 0x7f71d80736a0 2024-09-17T04:00:05.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d815cff0 con 0x7f71d80736a0 2024-09-17T04:00:05.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.796+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d815d5b0 con 0x7f71d80736a0 2024-09-17T04:00:05.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.797+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d81557d0 con 0x7f71d80736a0 2024-09-17T04:00:05.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.799+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f71c4041030 con 0x7f71d80736a0 2024-09-17T04:00:05.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.799+0000 7f71d4c00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 0x7f71b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:05.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.800+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f71c40bb440 con 0x7f71d80736a0 2024-09-17T04:00:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.800+0000 7f71dca00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 0x7f71b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.801+0000 7f71dca00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 0x7f71b807b0a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f71c80015e0 tx=0x7f71c8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:05.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:05.804+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f71d81557d0 con 0x7f71d80736a0 2024-09-17T04:00:06.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.069+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]} v 0) -- 0x7f71d81420c0 con 0x7f71d80736a0 2024-09-17T04:00:06.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.108+0000 7f71d4c00640 1 -- 172.21.15.129:0/2178181043 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_cleanup"]}]=0 v550) ==== 130+0+0 (secure 0 0 0) 0x7f71c4083c10 con 0x7f71d80736a0 2024-09-17T04:00:06.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 msgr2=0x7f71b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 0x7f71b807b0a0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f71c80015e0 tx=0x7f71c8009040 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 msgr2=0x7f71d81415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f71c40099c0 tx=0x7f71c40038a0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 shutdown_connections 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f71b8078b90 0x7f71b807b0a0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71d80cfb50 0x7f71d8141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71d80736a0 0x7f71d81415e0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 --2- 172.21.15.129:0/2178181043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f71d806ecb0 0x7f71d80ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 >> 172.21.15.129:0/2178181043 conn(0x7f71d806cfc0 msgr2=0x7f71d81550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 shutdown_connections 2024-09-17T04:00:06.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.110+0000 7f71dde00640 1 -- 172.21.15.129:0/2178181043 wait complete. 2024-09-17T04:00:06.132 INFO:tasks.cephfs_test_runner:That when doing a rename from A to B, and B has no hardlinks, ... ok 2024-09-17T04:00:06.132 INFO:tasks.cephfs_test_runner:Starting test: test_mv_hardlink_cleanup (tasks.cephfs.test_strays.TestStrays) That when doing a rename from A to B, and B has hardlinks, 2024-09-17T04:00:06.133 INFO:tasks.cephfs_test_runner:test_mv_hardlink_cleanup (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:00:06.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 -- 172.21.15.129:0/717539944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c160470 msgr2=0x7f520c160f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 --2- 172.21.15.129:0/717539944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c160470 0x7f520c160f40 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f520c06f140 tx=0x7f520002a4b0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 -- 172.21.15.129:0/717539944 shutdown_connections 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 --2- 172.21.15.129:0/717539944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0cf390 0x7f520c0cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 --2- 172.21.15.129:0/717539944 >> v2:172.21.15.181:3300/0 conn(0x7f520c0ccc80 0x7f520c0cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 --2- 172.21.15.129:0/717539944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c160470 0x7f520c160f40 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 -- 172.21.15.129:0/717539944 >> 172.21.15.129:0/717539944 conn(0x7f520c06d410 msgr2=0x7f520c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 -- 172.21.15.129:0/717539944 shutdown_connections 2024-09-17T04:00:06.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.353+0000 7f5213400640 1 -- 172.21.15.129:0/717539944 wait complete. 2024-09-17T04:00:06.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.356+0000 7f5213400640 1 Processor -- start 2024-09-17T04:00:06.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.356+0000 7f5213400640 1 -- start start 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccc80 0x7f520c15f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c160470 0x7f520c1ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f520c0ce500 con 0x7f520c0cf390 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f520c0ce380 con 0x7f520c160470 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f5213400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f520c0ce680 con 0x7f520c0ccc80 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.357+0000 7f520be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52570/0 (socket says 172.21.15.129:52570) 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 -- 172.21.15.129:0/1661031791 learned_addr learned my addr 172.21.15.129:0/1661031791 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f5210a00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccc80 0x7f520c15f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f5211400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c160470 0x7f520c1ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 -- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccc80 msgr2=0x7f520c15f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccc80 0x7f520c15f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 -- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c160470 msgr2=0x7f520c1ea6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c160470 0x7f520c1ea6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 -- 172.21.15.129:0/1661031791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c15f540 con 0x7f520c0cf390 2024-09-17T04:00:06.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f520be00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f51f800b410 tx=0x7f51f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.358+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8015020 con 0x7f520c0cf390 2024-09-17T04:00:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.359+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51f8011920 con 0x7f520c0cf390 2024-09-17T04:00:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.359+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8011d70 con 0x7f520c0cf390 2024-09-17T04:00:06.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.359+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c15fa50 con 0x7f520c0cf390 2024-09-17T04:00:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.359+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c15e470 con 0x7f520c0cf390 2024-09-17T04:00:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.360+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c15f7c0 con 0x7f520c0cf390 2024-09-17T04:00:06.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.361+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f51f8018020 con 0x7f520c0cf390 2024-09-17T04:00:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.362+0000 7f5208c00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 0x7f51ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.363+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f51f809a7e0 con 0x7f520c0cf390 2024-09-17T04:00:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.363+0000 7f5210a00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 0x7f51ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.364+0000 7f5210a00640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 0x7f51ec07b0d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5200033000 tx=0x7f5200004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:06.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.367+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f51f801f020 con 0x7f520c0cf390 2024-09-17T04:00:06.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.553+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f520c0cf7a0 con 0x7f520c0cf390 2024-09-17T04:00:06.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.554+0000 7f5208c00640 1 -- 172.21.15.129:0/1661031791 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) ==== 74+0+13092 (secure 0 0 0) 0x7f51f80628c0 con 0x7f520c0cf390 2024-09-17T04:00:06.554 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:06.554 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":155,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:05.093398+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/2593020253":"2024-09-18T04:00:01.796792+0000","172.21.15.181:6835/2593020253":"2024-09-18T04:00:01.796792+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 msgr2=0x7f51ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 0x7f51ec07b0d0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5200033000 tx=0x7f5200004730 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 msgr2=0x7f520c155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f51f800b410 tx=0x7f51f800b8d0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 shutdown_connections 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51ec078bc0 0x7f51ec07b0d0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f520c160470 0x7f520c1ea6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f520c0cf390 0x7f520c155b80 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 --2- 172.21.15.129:0/1661031791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f520c0ccc80 0x7f520c15f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 >> 172.21.15.129:0/1661031791 conn(0x7f520c06d410 msgr2=0x7f520c0742d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 shutdown_connections 2024-09-17T04:00:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.556+0000 7f5213400640 1 -- 172.21.15.129:0/1661031791 wait complete. 2024-09-17T04:00:06.579 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:00:06.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 -- 172.21.15.129:0/4033223548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c004800 msgr2=0x7f5e1c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 --2- 172.21.15.129:0/4033223548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e1c004800 0x7f5e1c004c10 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009b80 tx=0x7f5e1c005830 comp rx=0 tx=0).stop 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e37e00640 1 -- 172.21.15.129:0/4033223548 reap_dead start 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 -- 172.21.15.129:0/4033223548 shutdown_connections 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 -- 172.21.15.129:0/4033223548 >> 172.21.15.129:0/4033223548 conn(0x7f5e3806cde0 msgr2=0x7f5e3806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 -- 172.21.15.129:0/4033223548 shutdown_connections 2024-09-17T04:00:06.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.807+0000 7f5e3d200640 1 -- 172.21.15.129:0/4033223548 wait complete. 2024-09-17T04:00:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.810+0000 7f5e3d200640 1 Processor -- start 2024-09-17T04:00:06.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.810+0000 7f5e3d200640 1 -- start start 2024-09-17T04:00:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c004800 0x7f5e38156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 0x7f5e381f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5e380cfcc0 con 0x7f5e380b1190 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5e380cfb40 con 0x7f5e1c004800 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e3d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5e380cfe40 con 0x7f5e381f5560 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e2ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e2ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52588/0 (socket says 172.21.15.129:52588) 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.811+0000 7f5e2ea00640 1 -- 172.21.15.129:0/3311761067 learned_addr learned my addr 172.21.15.129:0/3311761067 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e37400640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c004800 0x7f5e38156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e37e00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 0x7f5e381f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 -- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 msgr2=0x7f5e381f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 0x7f5e381f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 -- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c004800 msgr2=0x7f5e38156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c004800 0x7f5e38156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 -- 172.21.15.129:0/3311761067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e380b32f0 con 0x7f5e380b1190 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e37e00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 0x7f5e381f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2ea00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009ca0 tx=0x7f5e20033b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.812+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20038040 con 0x7f5e380b1190 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.813+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e20033ca0 con 0x7f5e380b1190 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.813+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e380b35d0 con 0x7f5e380b1190 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.813+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20004280 con 0x7f5e380b1190 2024-09-17T04:00:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.813+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e380b3b90 con 0x7f5e380b1190 2024-09-17T04:00:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.814+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e38074580 con 0x7f5e380b1190 2024-09-17T04:00:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.815+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5e20045020 con 0x7f5e380b1190 2024-09-17T04:00:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.816+0000 7f5e2fe00640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 0x7f5e1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.816+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f5e200bb880 con 0x7f5e380b1190 2024-09-17T04:00:06.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.817+0000 7f5e37400640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 0x7f5e1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:06.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.817+0000 7f5e37400640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 0x7f5e1407b0a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5e28006b50 tx=0x7f5e280067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:06.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:06.820+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5e200838d0 con 0x7f5e380b1190 2024-09-17T04:00:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.002+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e3806a140 con 0x7f5e380b1190 2024-09-17T04:00:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.002+0000 7f5e2fe00640 1 -- 172.21.15.129:0/3311761067 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) ==== 93+0+10 (secure 0 0 0) 0x7f5e20088300 con 0x7f5e380b1190 2024-09-17T04:00:07.002 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 msgr2=0x7f5e1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 0x7f5e1407b0a0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5e28006b50 tx=0x7f5e280067a0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 msgr2=0x7f5e381568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f5e20009ca0 tx=0x7f5e20033b30 comp rx=0 tx=0).stop 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 shutdown_connections 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5e14078b90 0x7f5e1407b0a0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5e381f5560 0x7f5e381f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5e380b1190 0x7f5e381568d0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 --2- 172.21.15.129:0/3311761067 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5e1c004800 0x7f5e38156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 >> 172.21.15.129:0/3311761067 conn(0x7f5e3806cde0 msgr2=0x7f5e3815f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:07.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 shutdown_connections 2024-09-17T04:00:07.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.005+0000 7f5e3d200640 1 -- 172.21.15.129:0/3311761067 wait complete. 2024-09-17T04:00:07.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup' 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.254+0000 7f2180600640 1 -- 172.21.15.129:0/4214145653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 msgr2=0x7f216000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.254+0000 7f2180600640 1 --2- 172.21.15.129:0/4214145653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f216000b9c0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f2178070550 tx=0x7f2160005830 comp rx=0 tx=0).stop 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.254+0000 7f217e600640 1 -- 172.21.15.129:0/4214145653 reap_dead start 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.255+0000 7f2180600640 1 -- 172.21.15.129:0/4214145653 shutdown_connections 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.255+0000 7f2180600640 1 -- 172.21.15.129:0/4214145653 >> 172.21.15.129:0/4214145653 conn(0x7f217806d410 msgr2=0x7f21780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.255+0000 7f2180600640 1 -- 172.21.15.129:0/4214145653 shutdown_connections 2024-09-17T04:00:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.255+0000 7f2180600640 1 -- 172.21.15.129:0/4214145653 wait complete. 2024-09-17T04:00:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.257+0000 7f2180600640 1 Processor -- start 2024-09-17T04:00:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.257+0000 7f2180600640 1 -- start start 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21780adb60 0x7f21781f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21781f31d0 0x7f21780c3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21780cd510 con 0x7f2160005150 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f21780cd390 con 0x7f21780adb60 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.258+0000 7f2180600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f217806f2e0 con 0x7f21781f31d0 2024-09-17T04:00:07.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52610/0 (socket says 172.21.15.129:52610) 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 -- 172.21.15.129:0/1262758426 learned_addr learned my addr 172.21.15.129:0/1262758426 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217d200640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21780adb60 0x7f21781f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 -- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21781f31d0 msgr2=0x7f21780c3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21781f31d0 0x7f21780c3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 -- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21780adb60 msgr2=0x7f21781f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21780adb60 0x7f21781f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 -- 172.21.15.129:0/1262758426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21780c8410 con 0x7f2160005150 2024-09-17T04:00:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.259+0000 7f217dc00640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f216c011a60 tx=0x7f216c011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.260+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c00fc40 con 0x7f2160005150 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.260+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f216c00fde0 con 0x7f2160005150 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.260+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c004280 con 0x7f2160005150 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.260+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21780c86f0 con 0x7f2160005150 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.260+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21780c8cb0 con 0x7f2160005150 2024-09-17T04:00:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.261+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21780b4140 con 0x7f2160005150 2024-09-17T04:00:07.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.262+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f216c0035f0 con 0x7f2160005150 2024-09-17T04:00:07.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.263+0000 7f2175600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 0x7f215807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:07.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.263+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f216c09b5b0 con 0x7f2160005150 2024-09-17T04:00:07.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.264+0000 7f217d200640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 0x7f215807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:07.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.264+0000 7f217d200640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 0x7f215807b0a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2164000fd0 tx=0x7f2164001510 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:07.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.268+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f216c063c50 con 0x7f2160005150 2024-09-17T04:00:07.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:07.523+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]} v 0) -- 0x7f217806a140 con 0x7f2160005150 2024-09-17T04:00:08.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.116+0000 7f2175600640 1 -- 172.21.15.129:0/1262758426 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]}]=0 v552) ==== 142+0+0 (secure 0 0 0) 0x7f216c063550 con 0x7f2160005150 2024-09-17T04:00:08.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 msgr2=0x7f215807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 0x7f215807b0a0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f2164000fd0 tx=0x7f2164001510 comp rx=0 tx=0).stop 2024-09-17T04:00:08.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 msgr2=0x7f21780ad620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f216c011a60 tx=0x7f216c011f20 comp rx=0 tx=0).stop 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 shutdown_connections 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2158078b90 0x7f215807b0a0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21781f31d0 0x7f21780c3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21780adb60 0x7f21781f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 --2- 172.21.15.129:0/1262758426 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2160005150 0x7f21780ad620 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 >> 172.21.15.129:0/1262758426 conn(0x7f217806d410 msgr2=0x7f217815f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 shutdown_connections 2024-09-17T04:00:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.119+0000 7f2180600640 1 -- 172.21.15.129:0/1262758426 wait complete. 2024-09-17T04:00:08.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:00:08.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 -- 172.21.15.129:0/2490405827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 msgr2=0x7f05380ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 --2- 172.21.15.129:0/2490405827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ccc80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0524009830 tx=0x7f052402a510 comp rx=0 tx=0).stop 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 -- 172.21.15.129:0/2490405827 shutdown_connections 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 --2- 172.21.15.129:0/2490405827 >> v2:172.21.15.181:3300/0 conn(0x7f05380cd1c0 0x7f05381609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 --2- 172.21.15.129:0/2490405827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ccc80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 -- 172.21.15.129:0/2490405827 >> 172.21.15.129:0/2490405827 conn(0x7f053806d410 msgr2=0x7f05380b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 -- 172.21.15.129:0/2490405827 shutdown_connections 2024-09-17T04:00:08.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.364+0000 7f053d800640 1 -- 172.21.15.129:0/2490405827 wait complete. 2024-09-17T04:00:08.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.366+0000 7f053d800640 1 Processor -- start 2024-09-17T04:00:08.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.366+0000 7f053d800640 1 -- start start 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05380705a0 0x7f05380c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0538160260 con 0x7f05380cd1c0 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05381600e0 con 0x7f05380705a0 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f053d800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0538160f00 con 0x7f05380b4140 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f0537400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f0537400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52620/0 (socket says 172.21.15.129:52620) 2024-09-17T04:00:08.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f0537400640 1 -- 172.21.15.129:0/1906228713 learned_addr learned my addr 172.21.15.129:0/1906228713 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f0536000640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.368+0000 7f0536a00640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05380705a0 0x7f05380c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 -- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 msgr2=0x7f05380ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 -- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05380705a0 msgr2=0x7f05380c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05380705a0 0x7f05380c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 -- 172.21.15.129:0/1906228713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05380c3c70 con 0x7f05380cd1c0 2024-09-17T04:00:08.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f0537400640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f053000c370 tx=0x7f053000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053001b070 con 0x7f05380cd1c0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05380c5650 con 0x7f05380cd1c0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f053000b570 con 0x7f05380cd1c0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.369+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053000b9c0 con 0x7f05380cd1c0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.370+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05380c3f50 con 0x7f05380cd1c0 2024-09-17T04:00:08.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.371+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05380ccc80 con 0x7f05380cd1c0 2024-09-17T04:00:08.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.372+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f053001e070 con 0x7f05380cd1c0 2024-09-17T04:00:08.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.373+0000 7f052ea00640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 0x7f051807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.373+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f053009b130 con 0x7f05380cd1c0 2024-09-17T04:00:08.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.373+0000 7f0536a00640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 0x7f051807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.374+0000 7f0536a00640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 0x7f051807b0d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0528000f60 tx=0x7f0528009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:08.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.377+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0530063100 con 0x7f05380cd1c0 2024-09-17T04:00:08.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.562+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f053806a140 con 0x7f05380cd1c0 2024-09-17T04:00:08.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.562+0000 7f052ea00640 1 -- 172.21.15.129:0/1906228713 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v55) ==== 59+0+2294 (secure 0 0 0) 0x7f0530067b30 con 0x7f05380cd1c0 2024-09-17T04:00:08.562 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:00:08.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:00:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:00:08.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:00:08.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:00:08.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:00:08.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 msgr2=0x7f051807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 0x7f051807b0d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0528000f60 tx=0x7f0528009040 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 msgr2=0x7f05380cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f053000c370 tx=0x7f053000c830 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 shutdown_connections 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0518078bc0 0x7f051807b0d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05380cd1c0 0x7f05380cae00 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05380b4140 0x7f05380ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 --2- 172.21.15.129:0/1906228713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05380705a0 0x7f05380c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 >> 172.21.15.129:0/1906228713 conn(0x7f053806d410 msgr2=0x7f0538074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.565+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 shutdown_connections 2024-09-17T04:00:08.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.566+0000 7f053d800640 1 -- 172.21.15.129:0/1906228713 wait complete. 2024-09-17T04:00:08.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 -- 172.21.15.129:0/277950765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 msgr2=0x7f673c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 --2- 172.21.15.129:0/277950765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c0b3420 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f673c071e10 tx=0x7f673402aa30 comp rx=0 tx=0).stop 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 -- 172.21.15.129:0/277950765 shutdown_connections 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 --2- 172.21.15.129:0/277950765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c0b3420 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 --2- 172.21.15.129:0/277950765 >> v2:172.21.15.181:3300/0 conn(0x7f673c074800 0x7f673c0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.814+0000 7f6742c00640 1 -- 172.21.15.129:0/277950765 >> 172.21.15.129:0/277950765 conn(0x7f673c0ccc40 msgr2=0x7f673c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.815+0000 7f6742c00640 1 -- 172.21.15.129:0/277950765 shutdown_connections 2024-09-17T04:00:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.815+0000 7f6742c00640 1 -- 172.21.15.129:0/277950765 wait complete. 2024-09-17T04:00:08.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.817+0000 7f6742c00640 1 Processor -- start 2024-09-17T04:00:08.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.817+0000 7f6742c00640 1 -- start start 2024-09-17T04:00:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f673c074800 0x7f673c09c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 0x7f673c0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f673c0b3ae0 con 0x7f673c0b3010 2024-09-17T04:00:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f673c0b3960 con 0x7f673c074800 2024-09-17T04:00:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6742c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f673c0b3c60 con 0x7f673c0b1c40 2024-09-17T04:00:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.818+0000 7f6740c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52640/0 (socket says 172.21.15.129:52640) 2024-09-17T04:00:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 -- 172.21.15.129:0/3787530783 learned_addr learned my addr 172.21.15.129:0/3787530783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f673be00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f673c074800 0x7f673c09c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f673b400640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 0x7f673c0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 -- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 msgr2=0x7f673c0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 0x7f673c0c96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 -- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f673c074800 msgr2=0x7f673c09c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f673c074800 0x7f673c09c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 -- 172.21.15.129:0/3787530783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673c0ce300 con 0x7f673c0b3010 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f673b400640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 0x7f673c0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6740c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f673402b020 tx=0x7f6734003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.819+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734033050 con 0x7f673c0b3010 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.820+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6734004450 con 0x7f673c0b3010 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.820+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f673c0ce5e0 con 0x7f673c0b3010 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.820+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734038450 con 0x7f673c0b3010 2024-09-17T04:00:08.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.820+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f673c0ceba0 con 0x7f673c0b3010 2024-09-17T04:00:08.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.821+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673c0d0dd0 con 0x7f673c0b3010 2024-09-17T04:00:08.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.822+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6734036070 con 0x7f673c0b3010 2024-09-17T04:00:08.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.823+0000 7f6733e00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 0x7f671c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:08.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.823+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f67340bbf80 con 0x7f673c0b3010 2024-09-17T04:00:08.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.823+0000 7f673be00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 0x7f671c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:08.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.824+0000 7f673be00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 0x7f671c07b0d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f672c005fd0 tx=0x7f672c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:08.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:08.827+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f673c0d0dd0 con 0x7f673c0b3010 2024-09-17T04:00:09.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.014+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f673c0b41e0 con 0x7f673c0b3010 2024-09-17T04:00:09.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.019+0000 7f6733e00640 1 -- 172.21.15.129:0/3787530783 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v56)=0 v56) ==== 122+0+0 (secure 0 0 0) 0x7f673c0cbaf0 con 0x7f673c0b3010 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 msgr2=0x7f671c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 0x7f671c07b0d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f672c005fd0 tx=0x7f672c005850 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 msgr2=0x7f673c1e9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f673402b020 tx=0x7f6734003f10 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 shutdown_connections 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f671c078bc0 0x7f671c07b0d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f673c0b3010 0x7f673c1e9420 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f673c0b1c40 0x7f673c0c96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 --2- 172.21.15.129:0/3787530783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f673c074800 0x7f673c09c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 >> 172.21.15.129:0/3787530783 conn(0x7f673c0ccc40 msgr2=0x7f673c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.021+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 shutdown_connections 2024-09-17T04:00:09.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.022+0000 7f6742c00640 1 -- 172.21.15.129:0/3787530783 wait complete. 2024-09-17T04:00:09.044 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:00:09.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:09.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:09.231 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:00:09.231 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:09.231 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:09.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:00:09.267 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:09.267 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:09.304 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:09.305 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:09.305 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:09.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:00:09.342 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:09.342 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:09.381 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:09.381 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:00:09.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 -- 172.21.15.129:0/4271606372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940b46c0 msgr2=0x7f64940ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 --2- 172.21.15.129:0/4271606372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940b46c0 0x7f64940ccc80 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f647c009830 tx=0x7f647c02a510 comp rx=0 tx=0).stop 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 -- 172.21.15.129:0/4271606372 shutdown_connections 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 --2- 172.21.15.129:0/4271606372 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940b46c0 0x7f64940ccc80 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 --2- 172.21.15.129:0/4271606372 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f649415da00 0x7f64940b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 -- 172.21.15.129:0/4271606372 >> 172.21.15.129:0/4271606372 conn(0x7f649406d410 msgr2=0x7f64940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 -- 172.21.15.129:0/4271606372 shutdown_connections 2024-09-17T04:00:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.602+0000 7f6498a00640 1 -- 172.21.15.129:0/4271606372 wait complete. 2024-09-17T04:00:09.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.604+0000 7f6498a00640 1 Processor -- start 2024-09-17T04:00:09.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.605+0000 7f6498a00640 1 -- start start 2024-09-17T04:00:09.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64940b46c0 0x7f6494154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f649415da00 0x7f64941f6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6494070990 con 0x7f64940cd1c0 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6494070810 con 0x7f649415da00 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6498a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6494070b10 con 0x7f64940b46c0 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6489400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6489400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52650/0 (socket says 172.21.15.129:52650) 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.606+0000 7f6489400640 1 -- 172.21.15.129:0/1441685516 learned_addr learned my addr 172.21.15.129:0/1441685516 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6491e00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64940b46c0 0x7f6494154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6492800640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f649415da00 0x7f64941f6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 -- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64940b46c0 msgr2=0x7f6494154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64940b46c0 0x7f6494154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 -- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f649415da00 msgr2=0x7f64941f6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f649415da00 0x7f64941f6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 -- 172.21.15.129:0/1441685516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6494155410 con 0x7f64940cd1c0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.607+0000 7f6489400640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f647c009950 tx=0x7f647c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.608+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c031d60 con 0x7f64940cd1c0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.608+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f647c033e10 con 0x7f64940cd1c0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.608+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64941f10d0 con 0x7f64940cd1c0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.608+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c033310 con 0x7f64940cd1c0 2024-09-17T04:00:09.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.608+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6494155b70 con 0x7f64940cd1c0 2024-09-17T04:00:09.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.609+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64941555e0 con 0x7f64940cd1c0 2024-09-17T04:00:09.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.610+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f647c0334b0 con 0x7f64940cd1c0 2024-09-17T04:00:09.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.611+0000 7f648aa00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 0x7f647007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:09.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.611+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7f647c0bbf30 con 0x7f64940cd1c0 2024-09-17T04:00:09.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.612+0000 7f6491e00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 0x7f647007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:09.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.612+0000 7f6491e00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 0x7f647007b0d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6484009b30 tx=0x7f6484005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:09.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.616+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f647c03f030 con 0x7f64940cd1c0 2024-09-17T04:00:09.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.834+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f649406a140 con 0x7f64940cd1c0 2024-09-17T04:00:09.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.835+0000 7f648aa00640 1 -- 172.21.15.129:0/1441685516 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+3925 (secure 0 0 0) 0x7f647c0845d0 con 0x7f64940cd1c0 2024-09-17T04:00:09.835 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:09.835 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":137,"btime":"2024-09-17T04:00:02:799215+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7181,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/2799639853","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2799639853},{"type":"v1","addr":"172.21.15.129:6834","nonce":2799639853}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":129},{"gid":7184,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3058176883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3058176883},{"type":"v1","addr":"172.21.15.129:6835","nonce":3058176883}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":130},{"gid":7373,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/258986381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":258986381},{"type":"v1","addr":"172.21.15.181:6837","nonce":258986381}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137},{"gid":8182,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3137138864","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3137138864},{"type":"v1","addr":"172.21.15.181:6836","nonce":3137138864}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":128}],"filesystems":[]} 2024-09-17T04:00:09.836 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 137 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.837+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 msgr2=0x7f647007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.837+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 0x7f647007b0d0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f6484009b30 tx=0x7f6484005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.837+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 msgr2=0x7f64941f0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.837+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f647c009950 tx=0x7f647c033c20 comp rx=0 tx=0).stop 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.837+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 shutdown_connections 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6470078bc0 0x7f647007b0d0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f649415da00 0x7f64941f6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f64940cd1c0 0x7f64941f0b90 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 --2- 172.21.15.129:0/1441685516 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f64940b46c0 0x7f6494154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 >> 172.21.15.129:0/1441685516 conn(0x7f649406d410 msgr2=0x7f64940b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 shutdown_connections 2024-09-17T04:00:09.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:09.838+0000 7f6498a00640 1 -- 172.21.15.129:0/1441685516 wait complete. 2024-09-17T04:00:09.861 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:00:09.862 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:00:09.862 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:00:09.862 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:09.862 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:00:09.862 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:00:09.863 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:00:09.863 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:09.863 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:00:09.863 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:00:09.863 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:00:09.863 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:09.864 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:00:09.864 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:00:09.864 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:00:09.864 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:09.887 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:00:09.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:00:09.888 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:00:09.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:00:09.889 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:00:09.890 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:00:09.890 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:00:09.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:00:09.932 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:00:09.933 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:09.934 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:00:09.934 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:09.936 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:00:09.936 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:09.937 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:00:09.937 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:09.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:00:10.131 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:00:10.132 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:00:10.135 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:00:10.137 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:00:10.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 -- 172.21.15.129:0/3610861038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 msgr2=0x7ff25c15ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 --2- 172.21.15.129:0/3610861038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c15ed50 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7ff254005fd0 tx=0x7ff25402bbc0 comp rx=0 tx=0).stop 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 -- 172.21.15.129:0/3610861038 shutdown_connections 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 --2- 172.21.15.129:0/3610861038 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c15ed50 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 --2- 172.21.15.129:0/3610861038 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c070f60 0x7ff25c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 -- 172.21.15.129:0/3610861038 >> 172.21.15.129:0/3610861038 conn(0x7ff25c06d410 msgr2=0x7ff25c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 -- 172.21.15.129:0/3610861038 shutdown_connections 2024-09-17T04:00:10.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.171+0000 7ff264000640 1 -- 172.21.15.129:0/3610861038 wait complete. 2024-09-17T04:00:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.173+0000 7ff264000640 1 Processor -- start 2024-09-17T04:00:10.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.174+0000 7ff264000640 1 -- start start 2024-09-17T04:00:10.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 0x7ff25c0d0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c0af4c0 0x7ff25c09c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c15f410 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff25c15f290 con 0x7ff25c070f60 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff264000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff25c15ae00 con 0x7ff25c0af4c0 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff262000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff262000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52706/0 (socket says 172.21.15.129:52706) 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff262000640 1 -- 172.21.15.129:0/4288655483 learned_addr learned my addr 172.21.15.129:0/4288655483 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff261600640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 0x7ff25c0d0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.175+0000 7ff260c00640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c0af4c0 0x7ff25c09c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:10.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 -- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c0af4c0 msgr2=0x7ff25c09c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c0af4c0 0x7ff25c09c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 -- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 msgr2=0x7ff25c0d0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 0x7ff25c0d0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 -- 172.21.15.129:0/4288655483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25c0cba10 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff261600640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 0x7ff25c0d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff262000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7ff254006010 tx=0x7ff25402d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254004250 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff254043070 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25c0ca970 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.176+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff254004740 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.177+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25c0caf30 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.178+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff25c15b2f0 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.179+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff254048460 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.180+0000 7ff259600640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 0x7ff23c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:10.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.180+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(155..155 src has 1..155) ==== 4943+0+0 (secure 0 0 0) 0x7ff2540bbda0 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.181+0000 7ff261600640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 0x7ff23c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:10.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.181+0000 7ff261600640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 0x7ff23c07b0a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff2500095f0 tx=0x7ff250005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:10.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.184+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff25404d020 con 0x7ff25c0cd1c0 2024-09-17T04:00:10.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:10.378+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff25c15d640 con 0x7ff25c0cd1c0 2024-09-17T04:00:11.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.139+0000 7ff259600640 1 -- 172.21.15.129:0/4288655483 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v157) ==== 97+0+0 (secure 0 0 0) 0x7ff254083df0 con 0x7ff25c0cd1c0 2024-09-17T04:00:11.139 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 msgr2=0x7ff23c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 0x7ff23c07b0a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff2500095f0 tx=0x7ff250005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 msgr2=0x7ff25c09ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7ff254006010 tx=0x7ff25402d950 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 shutdown_connections 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff23c078b90 0x7ff23c07b0a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff25c0cd1c0 0x7ff25c09ccb0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff25c0af4c0 0x7ff25c09c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 --2- 172.21.15.129:0/4288655483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff25c070f60 0x7ff25c0d0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.143+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 >> 172.21.15.129:0/4288655483 conn(0x7ff25c06d410 msgr2=0x7ff25c073a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:11.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.144+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 shutdown_connections 2024-09-17T04:00:11.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.144+0000 7ff264000640 1 -- 172.21.15.129:0/4288655483 wait complete. 2024-09-17T04:00:11.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:00:11.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 -- 172.21.15.129:0/55081325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f732c005280 msgr2=0x7f732c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 --2- 172.21.15.129:0/55081325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f732c005280 0x7f732c00b9c0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f7340005fd0 tx=0x7f732c005830 comp rx=0 tx=0).stop 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 -- 172.21.15.129:0/55081325 shutdown_connections 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 --2- 172.21.15.129:0/55081325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f732c005280 0x7f732c00b9c0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 -- 172.21.15.129:0/55081325 >> 172.21.15.129:0/55081325 conn(0x7f734806d410 msgr2=0x7f73480b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 -- 172.21.15.129:0/55081325 shutdown_connections 2024-09-17T04:00:11.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.390+0000 7f734ec00640 1 -- 172.21.15.129:0/55081325 wait complete. 2024-09-17T04:00:11.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.393+0000 7f734ec00640 1 Processor -- start 2024-09-17T04:00:11.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.393+0000 7f734ec00640 1 -- start start 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 0x7f734815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 0x7f73481e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f73480cc930 con 0x7f73480c9ef0 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f73480cc7b0 con 0x7f732c005280 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f734ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f73480ccab0 con 0x7f73481e7390 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f7347400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.394+0000 7f7347400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52720/0 (socket says 172.21.15.129:52720) 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 -- 172.21.15.129:0/1075213178 learned_addr learned my addr 172.21.15.129:0/1075213178 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347e00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 0x7f734815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f734cc00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 0x7f73481e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 -- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 msgr2=0x7f73481e77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 0x7f73481e77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 -- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 msgr2=0x7f734815b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 0x7f734815b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 -- 172.21.15.129:0/1075213178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73480c4ec0 con 0x7f73480c9ef0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347e00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 0x7f734815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f734cc00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 0x7f73481e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f7347400640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f7334006630 tx=0x7f73340039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334015070 con 0x7f73480c9ef0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7334011900 con 0x7f73480c9ef0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73481e5f30 con 0x7f73480c9ef0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.395+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7334011d30 con 0x7f73480c9ef0 2024-09-17T04:00:11.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.396+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73480c7450 con 0x7f73480c9ef0 2024-09-17T04:00:11.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.397+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73481e5ca0 con 0x7f73480c9ef0 2024-09-17T04:00:11.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.398+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7334018070 con 0x7f73480c9ef0 2024-09-17T04:00:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.399+0000 7f733fe00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 0x7f732807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.399+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 4857+0+0 (secure 0 0 0) 0x7f733409b7d0 con 0x7f73480c9ef0 2024-09-17T04:00:11.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.400+0000 7f7347e00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 0x7f732807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.400+0000 7f7347e00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 0x7f732807b0d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7338006420 tx=0x7f7338006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:11.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.404+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7334063f60 con 0x7f73480c9ef0 2024-09-17T04:00:11.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.655+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f73480c8cd0 con 0x7f73480c9ef0 2024-09-17T04:00:11.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.656+0000 7f733fe00640 1 -- 172.21.15.129:0/1075213178 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) ==== 80+0+5304 (secure 0 0 0) 0x7f7334063860 con 0x7f73480c9ef0 2024-09-17T04:00:11.656 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:11.656 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:00:11.656 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:00:11.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:00:11.657 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.658 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.659 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:00:11.660 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.662 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:00:11.664 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:11.665 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.666 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.667 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:11.668 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.669 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:00:11.670 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:00:11.671 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 msgr2=0x7f732807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 0x7f732807b0d0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7338006420 tx=0x7f7338006b10 comp rx=0 tx=0).stop 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 msgr2=0x7f73481e6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f7334006630 tx=0x7f73340039c0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 shutdown_connections 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.658+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7328078bc0 0x7f732807b0d0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f73481e7390 0x7f73481e77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73480c9ef0 0x7f73481e6e50 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 --2- 172.21.15.129:0/1075213178 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f732c005280 0x7f734815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 >> 172.21.15.129:0/1075213178 conn(0x7f734806d410 msgr2=0x7f73480af060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:11.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 shutdown_connections 2024-09-17T04:00:11.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.659+0000 7f734ec00640 1 -- 172.21.15.129:0/1075213178 wait complete. 2024-09-17T04:00:11.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 -- 172.21.15.129:0/3165604512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b04162d00 msgr2=0x7f5b040cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 --2- 172.21.15.129:0/3165604512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b04162d00 0x7f5b040cf390 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5aec005fd0 tx=0x7f5aec02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 -- 172.21.15.129:0/3165604512 shutdown_connections 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 --2- 172.21.15.129:0/3165604512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b04162d00 0x7f5b040cf390 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 --2- 172.21.15.129:0/3165604512 >> v2:172.21.15.181:3300/0 conn(0x7f5b040cd370 0x7f5b040b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.906+0000 7f5b0b400640 1 -- 172.21.15.129:0/3165604512 >> 172.21.15.129:0/3165604512 conn(0x7f5b0406d410 msgr2=0x7f5b040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.907+0000 7f5b0b400640 1 -- 172.21.15.129:0/3165604512 shutdown_connections 2024-09-17T04:00:11.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.907+0000 7f5b0b400640 1 -- 172.21.15.129:0/3165604512 wait complete. 2024-09-17T04:00:11.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.909+0000 7f5b0b400640 1 Processor -- start 2024-09-17T04:00:11.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.909+0000 7f5b0b400640 1 -- start start 2024-09-17T04:00:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b04162d00 0x7f5b040c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b04163110 0x7f5b040b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b040cb150 con 0x7f5b04163110 2024-09-17T04:00:11.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b040cafd0 con 0x7f5b04162d00 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.910+0000 7f5b0b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5b040ca360 con 0x7f5b040cd370 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b09400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b04163110 0x7f5b040b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5afbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b04162d00 0x7f5b040c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:48142/0 (socket says 172.21.15.129:48142) 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 -- 172.21.15.129:0/2500845773 learned_addr learned my addr 172.21.15.129:0/2500845773 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 -- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b04162d00 msgr2=0x7f5b040c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b04162d00 0x7f5b040c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 -- 172.21.15.129:0/2500845773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b04163110 msgr2=0x7f5b040b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b04163110 0x7f5b040b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.911+0000 7f5b08a00640 1 -- 172.21.15.129:0/2500845773 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0415cc70 con 0x7f5b040cd370 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b08a00640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5af400e9f0 tx=0x7f5af400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:11.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af400cd20 con 0x7f5b040cd370 2024-09-17T04:00:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5af4004020 con 0x7f5b040cd370 2024-09-17T04:00:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4004470 con 0x7f5b040cd370 2024-09-17T04:00:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b0415cf50 con 0x7f5b040cd370 2024-09-17T04:00:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.912+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b0415d510 con 0x7f5b040cd370 2024-09-17T04:00:11.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.913+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b040cfc40 con 0x7f5b040cd370 2024-09-17T04:00:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.915+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5af4005030 con 0x7f5b040cd370 2024-09-17T04:00:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.916+0000 7f5b01600640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 0x7f5ae007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.916+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(159..159 src has 1..159) ==== 4857+0+0 (secure 0 0 0) 0x7f5af409b5c0 con 0x7f5b040cd370 2024-09-17T04:00:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.916+0000 7f5afbe00640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 0x7f5ae007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:11.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.917+0000 7f5afbe00640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 0x7f5ae007b0d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5aec033000 tx=0x7f5aec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:11.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:11.920+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5af4063760 con 0x7f5b040cd370 2024-09-17T04:00:12.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.102+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f5b0406a140 con 0x7f5b040cd370 2024-09-17T04:00:12.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.103+0000 7f5b01600640 1 -- 172.21.15.129:0/2500845773 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v159) ==== 84+0+65 (secure 0 0 0) 0x7f5af4068190 con 0x7f5b040cd370 2024-09-17T04:00:12.103 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:12.103 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:00:12.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:12.104 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:00:12.104 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:00:12.104 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:12.104 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:00:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 msgr2=0x7f5ae007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 0x7f5ae007b0d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5aec033000 tx=0x7f5aec004730 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 msgr2=0x7f5b04099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5af400e9f0 tx=0x7f5af400eeb0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 shutdown_connections 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ae0078bc0 0x7f5ae007b0d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b04163110 0x7f5b040b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b04162d00 0x7f5b040c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 --2- 172.21.15.129:0/2500845773 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b040cd370 0x7f5b04099ab0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 >> 172.21.15.129:0/2500845773 conn(0x7f5b0406d410 msgr2=0x7f5b040ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:12.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 shutdown_connections 2024-09-17T04:00:12.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.105+0000 7f5b0b400640 1 -- 172.21.15.129:0/2500845773 wait complete. 2024-09-17T04:00:12.127 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:00:12.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 -- 172.21.15.129:0/129875740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 msgr2=0x7f36480cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 --2- 172.21.15.129:0/129875740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36480cbea0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3640005fd0 tx=0x7f364002aa10 comp rx=0 tx=0).stop 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 -- 172.21.15.129:0/129875740 shutdown_connections 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 --2- 172.21.15.129:0/129875740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36480cbea0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 --2- 172.21.15.129:0/129875740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648074580 0x7f36480b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.354+0000 7f364cc00640 1 -- 172.21.15.129:0/129875740 >> 172.21.15.129:0/129875740 conn(0x7f36480ccc40 msgr2=0x7f36480cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.355+0000 7f364cc00640 1 -- 172.21.15.129:0/129875740 shutdown_connections 2024-09-17T04:00:12.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.355+0000 7f364cc00640 1 -- 172.21.15.129:0/129875740 wait complete. 2024-09-17T04:00:12.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.357+0000 7f364cc00640 1 Processor -- start 2024-09-17T04:00:12.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.357+0000 7f364cc00640 1 -- start start 2024-09-17T04:00:12.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3648074580 0x7f3648157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 0x7f36481f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36480cc930 con 0x7f36480cba90 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36480cc7b0 con 0x7f3648074580 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f364cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3648166f20 con 0x7f3648158770 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.358+0000 7f363fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52752/0 (socket says 172.21.15.129:52752) 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 -- 172.21.15.129:0/3753629277 learned_addr learned my addr 172.21.15.129:0/3753629277 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:12.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f3645e00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3648074580 0x7f3648157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f3646800640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 0x7f36481f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 -- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 msgr2=0x7f36481f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 0x7f36481f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 -- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3648074580 msgr2=0x7f3648157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3648074580 0x7f3648157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 -- 172.21.15.129:0/3753629277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36480cf0c0 con 0x7f36480cba90 2024-09-17T04:00:12.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f3646800640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 0x7f36481f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:12.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363fe00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f363000d700 tx=0x7f363000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:12.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.359+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630017070 con 0x7f36480cba90 2024-09-17T04:00:12.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.360+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3630009b10 con 0x7f36480cba90 2024-09-17T04:00:12.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.360+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36480cede0 con 0x7f36480cba90 2024-09-17T04:00:12.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.360+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3630010460 con 0x7f36480cba90 2024-09-17T04:00:12.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.360+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36481f8b50 con 0x7f36480cba90 2024-09-17T04:00:12.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.361+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36480b3f70 con 0x7f36480cba90 2024-09-17T04:00:12.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.362+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f363001a070 con 0x7f36480cba90 2024-09-17T04:00:12.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.363+0000 7f363e000640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 0x7f362407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:12.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.363+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(159..159 src has 1..159) ==== 4857+0+0 (secure 0 0 0) 0x7f363009ab10 con 0x7f36480cba90 2024-09-17T04:00:12.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.364+0000 7f3645e00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 0x7f362407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:12.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.364+0000 7f3645e00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 0x7f362407b0d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f36380015e0 tx=0x7f3638009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.367+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3630063290 con 0x7f36480cba90 2024-09-17T04:00:12.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:12.550+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f36480b1fc0 con 0x7f36480cba90 2024-09-17T04:00:13.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.148+0000 7f363e000640 1 -- 172.21.15.129:0/3753629277 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v160) ==== 138+0+0 (secure 0 0 0) 0x7f3630062b90 con 0x7f36480cba90 2024-09-17T04:00:13.149 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:00:13.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 msgr2=0x7f362407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 0x7f362407b0d0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f36380015e0 tx=0x7f3638009040 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 msgr2=0x7f36481f9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f363000d700 tx=0x7f363000dbc0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 shutdown_connections 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3624078bc0 0x7f362407b0d0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3648158770 0x7f36481f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36480cba90 0x7f36481f9350 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.151+0000 7f364cc00640 1 --2- 172.21.15.129:0/3753629277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3648074580 0x7f3648157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.152+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 >> 172.21.15.129:0/3753629277 conn(0x7f36480ccc40 msgr2=0x7f36480afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.152+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 shutdown_connections 2024-09-17T04:00:13.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.152+0000 7f364cc00640 1 -- 172.21.15.129:0/3753629277 wait complete. 2024-09-17T04:00:13.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:00:13.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 -- 172.21.15.129:0/582480097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce406f280 msgr2=0x7efce40b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 --2- 172.21.15.129:0/582480097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce406f280 0x7efce40b3180 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009830 tx=0x7efcd002a420 comp rx=0 tx=0).stop 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 -- 172.21.15.129:0/582480097 shutdown_connections 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 --2- 172.21.15.129:0/582480097 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce406f280 0x7efce40b3180 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 --2- 172.21.15.129:0/582480097 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce40cc600 0x7efce415b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 -- 172.21.15.129:0/582480097 >> 172.21.15.129:0/582480097 conn(0x7efce406d7a0 msgr2=0x7efce40b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 -- 172.21.15.129:0/582480097 shutdown_connections 2024-09-17T04:00:13.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.409+0000 7efce8c00640 1 -- 172.21.15.129:0/582480097 wait complete. 2024-09-17T04:00:13.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.411+0000 7efce8c00640 1 Processor -- start 2024-09-17T04:00:13.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.412+0000 7efce8c00640 1 -- start start 2024-09-17T04:00:13.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce406f280 0x7efce4151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:13.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce4074750 0x7efce409db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efce40af2d0 con 0x7efce40cc600 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efce40af150 con 0x7efce4074750 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce8c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efce40af450 con 0x7efce406f280 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52754/0 (socket says 172.21.15.129:52754) 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.413+0000 7efce2800640 1 -- 172.21.15.129:0/430115799 learned_addr learned my addr 172.21.15.129:0/430115799 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce1400640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce4074750 0x7efce409db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce1e00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce406f280 0x7efce4151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:13.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 -- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce406f280 msgr2=0x7efce4151650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:13.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce406f280 0x7efce4151650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 -- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce4074750 msgr2=0x7efce409db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce4074750 0x7efce409db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 -- 172.21.15.129:0/430115799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce40b1540 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efce2800640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00cb70 tx=0x7efcdc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc016020 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.414+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efcdc00ae30 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.415+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efce41ea300 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.415+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcdc012660 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.415+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efce41ea8c0 con 0x7efce40cc600 2024-09-17T04:00:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.416+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efce4076830 con 0x7efce40cc600 2024-09-17T04:00:13.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.417+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efcdc019020 con 0x7efce40cc600 2024-09-17T04:00:13.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.418+0000 7efcda000640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 0x7efcc407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:13.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.418+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(160..160 src has 1..160) ==== 5227+0+0 (secure 0 0 0) 0x7efcdc09b440 con 0x7efce40cc600 2024-09-17T04:00:13.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.419+0000 7efce1e00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 0x7efcc407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:13.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.419+0000 7efce1e00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 0x7efcc407b0d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcd40095f0 tx=0x7efcd4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:13.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.422+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efce4076830 con 0x7efce40cc600 2024-09-17T04:00:13.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:13.605+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efce41ea090 con 0x7efce40cc600 2024-09-17T04:00:14.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.151+0000 7efcda000640 1 -- 172.21.15.129:0/430115799 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v161) ==== 170+0+0 (secure 0 0 0) 0x7efcdc020070 con 0x7efce40cc600 2024-09-17T04:00:14.152 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 msgr2=0x7efcc407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 0x7efcc407b0d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7efcd40095f0 tx=0x7efcd4005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 msgr2=0x7efce40b08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7efcdc00cb70 tx=0x7efcdc00a430 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 shutdown_connections 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efcc4078bc0 0x7efcc407b0d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efce40cc600 0x7efce40b08c0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efce4074750 0x7efce409db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 --2- 172.21.15.129:0/430115799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efce406f280 0x7efce4151650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.155+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 >> 172.21.15.129:0/430115799 conn(0x7efce406d7a0 msgr2=0x7efce406f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.156+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 shutdown_connections 2024-09-17T04:00:14.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.156+0000 7efce8c00640 1 -- 172.21.15.129:0/430115799 wait complete. 2024-09-17T04:00:14.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:00:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 -- 172.21.15.129:0/2363728967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700b4220 msgr2=0x7f03700b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 --2- 172.21.15.129:0/2363728967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700b4220 0x7f03700b4630 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f0368005fd0 tx=0x7f036802a8d0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 -- 172.21.15.129:0/2363728967 shutdown_connections 2024-09-17T04:00:14.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 --2- 172.21.15.129:0/2363728967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700b4220 0x7f03700b4630 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 -- 172.21.15.129:0/2363728967 >> 172.21.15.129:0/2363728967 conn(0x7f03700ccc40 msgr2=0x7f03700cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:14.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 -- 172.21.15.129:0/2363728967 shutdown_connections 2024-09-17T04:00:14.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.421+0000 7f0375600640 1 -- 172.21.15.129:0/2363728967 wait complete. 2024-09-17T04:00:14.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.423+0000 7f0375600640 1 Processor -- start 2024-09-17T04:00:14.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.423+0000 7f0375600640 1 -- start start 2024-09-17T04:00:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 0x7f03700c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:14.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 0x7f03700d0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03700cad80 con 0x7f03700b4220 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f03700cac00 con 0x7f03700ce500 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f0375600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f03700caf00 con 0x7f03700b2650 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35288/0 (socket says 172.21.15.129:35288) 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 -- 172.21.15.129:0/1166313580 learned_addr learned my addr 172.21.15.129:0/1166313580 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036f400640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 0x7f03700d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 -- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 msgr2=0x7f03700c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036ea00640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 0x7f03700c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 0x7f03700c3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 -- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 msgr2=0x7f03700d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 0x7f03700d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036e000640 1 -- 172.21.15.129:0/1166313580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03700c41f0 con 0x7f03700b4220 2024-09-17T04:00:14.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036f400640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 0x7f03700d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.425+0000 7f036ea00640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 0x7f03700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f036e000640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f035c00c400 tx=0x7f035c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c017070 con 0x7f03700b4220 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f035c0053c0 con 0x7f03700b4220 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03700c4700 con 0x7f03700b4220 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035c003850 con 0x7f03700b4220 2024-09-17T04:00:14.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.426+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03701f5ea0 con 0x7f03700b4220 2024-09-17T04:00:14.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.427+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03700b4630 con 0x7f03700b4220 2024-09-17T04:00:14.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.429+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f035c01a070 con 0x7f03700b4220 2024-09-17T04:00:14.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.430+0000 7f0366a00640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 0x7f035007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:14.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.430+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(161..161 src has 1..161) ==== 5609+0+0 (secure 0 0 0) 0x7f035c09a390 con 0x7f03700b4220 2024-09-17T04:00:14.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.430+0000 7f036ea00640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 0x7f035007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:14.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.431+0000 7f036ea00640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 0x7f035007b0a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0360001860 tx=0x7f0360009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:14.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.433+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f035c021020 con 0x7f03700b4220 2024-09-17T04:00:14.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:14.650+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f03700c4490 con 0x7f03700b4220 2024-09-17T04:00:15.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.165+0000 7f0366a00640 1 -- 172.21.15.129:0/1166313580 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v140) ==== 174+0+0 (secure 0 0 0) 0x7f03700b4b70 con 0x7f03700b4220 2024-09-17T04:00:15.165 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 24 and data pool 25 2024-09-17T04:00:15.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 msgr2=0x7f035007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 0x7f035007b0a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0360001860 tx=0x7f0360009040 comp rx=0 tx=0).stop 2024-09-17T04:00:15.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 msgr2=0x7f03700d01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f035c00c400 tx=0x7f035c00c8c0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 shutdown_connections 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0350078b90 0x7f035007b0a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700ce500 0x7f03700d0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03700b4220 0x7f03700d01c0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 --2- 172.21.15.129:0/1166313580 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03700b2650 0x7f03700c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.168+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 >> 172.21.15.129:0/1166313580 conn(0x7f03700ccc40 msgr2=0x7f03700737d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:15.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.169+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 shutdown_connections 2024-09-17T04:00:15.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.170+0000 7f0375600640 1 -- 172.21.15.129:0/1166313580 wait complete. 2024-09-17T04:00:15.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 -- 172.21.15.129:0/3628295348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 msgr2=0x7f60a4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 --2- 172.21.15.129:0/3628295348 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60a4004c10 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0017a0 tx=0x7f60a4005830 comp rx=0 tx=0).stop 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60bf400640 1 -- 172.21.15.129:0/3628295348 reap_dead start 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 -- 172.21.15.129:0/3628295348 shutdown_connections 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 -- 172.21.15.129:0/3628295348 >> 172.21.15.129:0/3628295348 conn(0x7f60c006d410 msgr2=0x7f60c00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 -- 172.21.15.129:0/3628295348 shutdown_connections 2024-09-17T04:00:15.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.440+0000 7f60c5600640 1 -- 172.21.15.129:0/3628295348 wait complete. 2024-09-17T04:00:15.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.443+0000 7f60c5600640 1 Processor -- start 2024-09-17T04:00:15.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.443+0000 7f60c5600640 1 -- start start 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60c01f4c00 0x7f60c014ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60c00d01c0 0x7f60c00d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f60c00cd390 con 0x7f60a4004800 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f60c0070d10 con 0x7f60c00d01c0 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.444+0000 7f60c5600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f60c00cd510 con 0x7f60c01f4c00 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35300/0 (socket says 172.21.15.129:35300) 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 -- 172.21.15.129:0/2528585204 learned_addr learned my addr 172.21.15.129:0/2528585204 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60be000640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60c01f4c00 0x7f60c014ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bf400640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60c00d01c0 0x7f60c00d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 -- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60c01f4c00 msgr2=0x7f60c014ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60c01f4c00 0x7f60c014ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 -- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60c00d01c0 msgr2=0x7f60c00d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60c00d01c0 0x7f60c00d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 -- 172.21.15.129:0/2528585204 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c00d0b10 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.445+0000 7f60bea00640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f60b0001220 tx=0x7f60b00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.446+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0015070 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.446+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60b0011920 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.446+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c01e8ef0 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.446+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b0011d70 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.446+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c01e94b0 con 0x7f60a4004800 2024-09-17T04:00:15.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.447+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c00c73f0 con 0x7f60a4004800 2024-09-17T04:00:15.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.448+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f60b00103c0 con 0x7f60a4004800 2024-09-17T04:00:15.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.449+0000 7f60b6a00640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 0x7f60a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.449+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5709+0+0 (secure 0 0 0) 0x7f60b009bab0 con 0x7f60a4004800 2024-09-17T04:00:15.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.450+0000 7f60be000640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 0x7f60a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.450+0000 7f60be000640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 0x7f60a007b0a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f60ac032000 tx=0x7f60ac001360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:15.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.453+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f60c00d0b10 con 0x7f60a4004800 2024-09-17T04:00:15.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.636+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60c006a140 con 0x7f60a4004800 2024-09-17T04:00:15.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.637+0000 7f60b6a00640 1 -- 172.21.15.129:0/2528585204 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) ==== 74+0+16800 (secure 0 0 0) 0x7f60b00638a0 con 0x7f60a4004800 2024-09-17T04:00:15.637 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:15.637 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":162,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:14.652422+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:12.563914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:13.619002+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 msgr2=0x7f60a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 0x7f60a007b0a0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f60ac032000 tx=0x7f60ac001360 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 msgr2=0x7f60c01e8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f60b0001220 tx=0x7f60b00039c0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 shutdown_connections 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60a0078b90 0x7f60a007b0a0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60c00d01c0 0x7f60c00d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60c01f4c00 0x7f60c014ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 --2- 172.21.15.129:0/2528585204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60a4004800 0x7f60c01e8870 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 >> 172.21.15.129:0/2528585204 conn(0x7f60c006d410 msgr2=0x7f60c00ad660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 shutdown_connections 2024-09-17T04:00:15.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.639+0000 7f60c5600640 1 -- 172.21.15.129:0/2528585204 wait complete. 2024-09-17T04:00:15.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:15.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 -- 172.21.15.129:0/2498575353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b640705a0 msgr2=0x7f4b640b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2498575353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b640705a0 0x7f4b640b4140 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f4b58005fd0 tx=0x7f4b5802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 -- 172.21.15.129:0/2498575353 shutdown_connections 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2498575353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b640cd390 0x7f4b641609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2498575353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b640705a0 0x7f4b640b4140 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 -- 172.21.15.129:0/2498575353 >> 172.21.15.129:0/2498575353 conn(0x7f4b6406d410 msgr2=0x7f4b640b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 -- 172.21.15.129:0/2498575353 shutdown_connections 2024-09-17T04:00:15.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.883+0000 7f4b6be00640 1 -- 172.21.15.129:0/2498575353 wait complete. 2024-09-17T04:00:15.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.885+0000 7f4b6be00640 1 Processor -- start 2024-09-17T04:00:15.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.886+0000 7f4b6be00640 1 -- start start 2024-09-17T04:00:15.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b640705a0 0x7f4b641e8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b641e9d40 0x7f4b641f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b64160f00 con 0x7f4b6406eed0 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6406f4f0 con 0x7f4b640705a0 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b6be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4b64161080 con 0x7f4b641e9d40 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b69400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b69400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35320/0 (socket says 172.21.15.129:35320) 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.887+0000 7f4b69400640 1 -- 172.21.15.129:0/2883848556 learned_addr learned my addr 172.21.15.129:0/2883848556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69e00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b641e9d40 0x7f4b641f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b68a00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b640705a0 0x7f4b641e8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 -- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b641e9d40 msgr2=0x7f4b641f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b641e9d40 0x7f4b641f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 -- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b640705a0 msgr2=0x7f4b641e8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b640705a0 0x7f4b641e8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 -- 172.21.15.129:0/2883848556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b641e7360 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b69400640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b580038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5803e030 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b58003ca0 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b641e75d0 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.888+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58035040 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.889+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b641e94e0 con 0x7f4b6406eed0 2024-09-17T04:00:15.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.890+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b641f5740 con 0x7f4b6406eed0 2024-09-17T04:00:15.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.891+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4b58041030 con 0x7f4b6406eed0 2024-09-17T04:00:15.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.892+0000 7f4b61600640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 0x7f4b4407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:15.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.892+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(162..162 src has 1..162) ==== 5709+0+0 (secure 0 0 0) 0x7f4b580bb310 con 0x7f4b6406eed0 2024-09-17T04:00:15.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.893+0000 7f4b68a00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 0x7f4b4407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:15.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.893+0000 7f4b68a00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 0x7f4b4407b0a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4b50007fd0 tx=0x7f4b50009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:15.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:15.896+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4b58083740 con 0x7f4b6406eed0 2024-09-17T04:00:16.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.079+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b640b4140 con 0x7f4b6406eed0 2024-09-17T04:00:16.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.080+0000 7f4b61600640 1 -- 172.21.15.129:0/2883848556 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v162) ==== 74+0+16800 (secure 0 0 0) 0x7f4b58083040 con 0x7f4b6406eed0 2024-09-17T04:00:16.081 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:16.081 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":162,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:14.652422+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:12.563914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:13.619002+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 msgr2=0x7f4b4407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 0x7f4b4407b0a0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f4b50007fd0 tx=0x7f4b50009040 comp rx=0 tx=0).stop 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 msgr2=0x7f4b64099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f4b58000c00 tx=0x7f4b580038a0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 shutdown_connections 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b44078b90 0x7f4b4407b0a0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b641e9d40 0x7f4b641f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b640705a0 0x7f4b641e8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 --2- 172.21.15.129:0/2883848556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b6406eed0 0x7f4b64099ab0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 >> 172.21.15.129:0/2883848556 conn(0x7f4b6406d410 msgr2=0x7f4b640ad3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:16.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 shutdown_connections 2024-09-17T04:00:16.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.083+0000 7f4b6be00640 1 -- 172.21.15.129:0/2883848556 wait complete. 2024-09-17T04:00:16.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.331+0000 7fc734000640 1 -- 172.21.15.129:0/486526916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc714004800 msgr2=0x7fc714004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.331+0000 7fc734000640 1 --2- 172.21.15.129:0/486526916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc714004800 0x7fc714004c10 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fc71800c470 tx=0x7fc714005830 comp rx=0 tx=0).stop 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.331+0000 7fc734000640 1 -- 172.21.15.129:0/486526916 shutdown_connections 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.331+0000 7fc734000640 1 --2- 172.21.15.129:0/486526916 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc714004800 0x7fc714004c10 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.331+0000 7fc734000640 1 -- 172.21.15.129:0/486526916 >> 172.21.15.129:0/486526916 conn(0x7fc72c0ccc40 msgr2=0x7fc72c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.332+0000 7fc734000640 1 -- 172.21.15.129:0/486526916 shutdown_connections 2024-09-17T04:00:16.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.332+0000 7fc734000640 1 -- 172.21.15.129:0/486526916 wait complete. 2024-09-17T04:00:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.334+0000 7fc734000640 1 Processor -- start 2024-09-17T04:00:16.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.334+0000 7fc734000640 1 -- start start 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc714004800 0x7fc72c1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc72c157240 0x7fc72c15ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c0af600 con 0x7fc72c1531f0 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c0af480 con 0x7fc714004800 2024-09-17T04:00:16.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.335+0000 7fc734000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc72c0af780 con 0x7fc72c157240 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35338/0 (socket says 172.21.15.129:35338) 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 -- 172.21.15.129:0/4235855556 learned_addr learned my addr 172.21.15.129:0/4235855556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc731600640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc714004800 0x7fc72c1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc730c00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc72c157240 0x7fc72c15ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 -- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc72c157240 msgr2=0x7fc72c15ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc72c157240 0x7fc72c15ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 -- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc714004800 msgr2=0x7fc72c1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc714004800 0x7fc72c1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 -- 172.21.15.129:0/4235855556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c1548c0 con 0x7fc72c1531f0 2024-09-17T04:00:16.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.336+0000 7fc732000640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc724005fd0 tx=0x7fc72400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.337+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724003d80 con 0x7fc72c1531f0 2024-09-17T04:00:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.337+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc724004b20 con 0x7fc72c1531f0 2024-09-17T04:00:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.337+0000 7fc734000640 1 -- 172.21.15.129:0/4235855556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72c09aa60 con 0x7fc72c1531f0 2024-09-17T04:00:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.337+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724004030 con 0x7fc72c1531f0 2024-09-17T04:00:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.337+0000 7fc734000640 1 -- 172.21.15.129:0/4235855556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72c0cf690 con 0x7fc72c1531f0 2024-09-17T04:00:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.338+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc72c0b21c0 con 0x7fc72c1531f0 2024-09-17T04:00:16.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.339+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc7240041d0 con 0x7fc72c1531f0 2024-09-17T04:00:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.340+0000 7fc729600640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 0x7fc70c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.340+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fc72409bca0 con 0x7fc72c1531f0 2024-09-17T04:00:16.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.341+0000 7fc731600640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 0x7fc70c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:16.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.341+0000 7fc731600640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 0x7fc70c07b0d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc72c052910 tx=0x7fc720006ef0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:16.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.344+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc7240640e0 con 0x7fc72c1531f0 2024-09-17T04:00:16.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:16.562+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc72c050ff0 con 0x7fc72c1531f0 2024-09-17T04:00:17.150 INFO:tasks.ceph.mds.d.smithi181.stderr:2024-09-17T04:00:17.151+0000 7f3033400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:00:17.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.196+0000 7fc729600640 1 -- 172.21.15.129:0/4235855556 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v143) ==== 118+0+0 (secure 0 0 0) 0x7fc7240639e0 con 0x7fc72c1531f0 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.198+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 msgr2=0x7fc70c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.198+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 0x7fc70c07b0d0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fc72c052910 tx=0x7fc720006ef0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.198+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 msgr2=0x7fc72c15b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.198+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc724005fd0 tx=0x7fc72400f790 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 shutdown_connections 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc70c078bc0 0x7fc70c07b0d0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc72c1531f0 0x7fc72c15b340 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc72c157240 0x7fc72c15ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 --2- 172.21.15.129:0/4235855556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc714004800 0x7fc72c1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 >> 172.21.15.129:0/4235855556 conn(0x7fc72c0ccc40 msgr2=0x7fc72c0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:17.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 shutdown_connections 2024-09-17T04:00:17.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.199+0000 7fc71ea00640 1 -- 172.21.15.129:0/4235855556 wait complete. 2024-09-17T04:00:17.226 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:00:17.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:00:17.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 -- 172.21.15.129:0/3650316665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 msgr2=0x7fc9a0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:17.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/3650316665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9a0004c10 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc09dc20 tx=0x7fc9a0005830 comp rx=0 tx=0).stop 2024-09-17T04:00:17.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9ba800640 1 -- 172.21.15.129:0/3650316665 reap_dead start 2024-09-17T04:00:17.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 -- 172.21.15.129:0/3650316665 shutdown_connections 2024-09-17T04:00:17.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 -- 172.21.15.129:0/3650316665 >> 172.21.15.129:0/3650316665 conn(0x7fc9bc0ccc40 msgr2=0x7fc9bc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:17.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 -- 172.21.15.129:0/3650316665 shutdown_connections 2024-09-17T04:00:17.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.484+0000 7fc9c0c00640 1 -- 172.21.15.129:0/3650316665 wait complete. 2024-09-17T04:00:17.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.486+0000 7fc9c0c00640 1 Processor -- start 2024-09-17T04:00:17.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.486+0000 7fc9c0c00640 1 -- start start 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 0x7fc9bc15f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0adfa0 0x7fc9bc0ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc0b21d0 con 0x7fc9a0004800 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9bc0b2050 con 0x7fc9bc0adfa0 2024-09-17T04:00:17.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.488+0000 7fc9c0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9bc0b2350 con 0x7fc9bc15f120 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35358/0 (socket says 172.21.15.129:35358) 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 -- 172.21.15.129:0/1559892599 learned_addr learned my addr 172.21.15.129:0/1559892599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9400640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 0x7fc9bc15f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9ba800640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0adfa0 0x7fc9bc0ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 -- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 msgr2=0x7fc9bc15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 0x7fc9bc15f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 -- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0adfa0 msgr2=0x7fc9bc0ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0adfa0 0x7fc9bc0ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 -- 172.21.15.129:0/1559892599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9bc1f33e0 con 0x7fc9a0004800 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9400640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 0x7fc9bc15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:17.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.489+0000 7fc9b9e00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac010e30 tx=0x7fc9ac00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:17.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.490+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac017070 con 0x7fc9a0004800 2024-09-17T04:00:17.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.490+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9ac015070 con 0x7fc9a0004800 2024-09-17T04:00:17.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.490+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ac011be0 con 0x7fc9a0004800 2024-09-17T04:00:17.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.490+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9bc0ae8f0 con 0x7fc9a0004800 2024-09-17T04:00:17.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.490+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9bc1f5e40 con 0x7fc9a0004800 2024-09-17T04:00:17.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.491+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9bc0b4230 con 0x7fc9a0004800 2024-09-17T04:00:17.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.492+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9ac01c460 con 0x7fc9a0004800 2024-09-17T04:00:17.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.493+0000 7fc9b2000640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:17.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.493+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fc9ac09bee0 con 0x7fc9a0004800 2024-09-17T04:00:17.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.494+0000 7fc9b9400640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.494+0000 7fc9b9400640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8035000 tx=0x7fc9a802d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:17.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.497+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9bc0ae8f0 con 0x7fc9a0004800 2024-09-17T04:00:17.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:17.715+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fc9bc06fe00 con 0x7fc9a0004800 2024-09-17T04:00:17.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:00:17.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:00:18.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.206+0000 7fc9b2000640 1 -- 172.21.15.129:0/1559892599 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v144) ==== 122+0+0 (secure 0 0 0) 0x7fc9bc1f33e0 con 0x7fc9a0004800 2024-09-17T04:00:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 msgr2=0x7fc99c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8035000 tx=0x7fc9a802d040 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 msgr2=0x7fc9bc1f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac010e30 tx=0x7fc9ac00e720 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 shutdown_connections 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0adfa0 0x7fc9bc0ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9bc15f120 0x7fc9bc15f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 --2- 172.21.15.129:0/1559892599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9a0004800 0x7fc9bc1f2ea0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 >> 172.21.15.129:0/1559892599 conn(0x7fc9bc0ccc40 msgr2=0x7fc9bc1536d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 shutdown_connections 2024-09-17T04:00:18.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.209+0000 7fc9c0c00640 1 -- 172.21.15.129:0/1559892599 wait complete. 2024-09-17T04:00:18.235 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:00:18.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 -- 172.21.15.129:0/3215310257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100cf1d0 msgr2=0x7f76100cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 --2- 172.21.15.129:0/3215310257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100cf1d0 0x7f76100cf5e0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f7608005fd0 tx=0x7f760802a580 comp rx=0 tx=0).stop 2024-09-17T04:00:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 -- 172.21.15.129:0/3215310257 shutdown_connections 2024-09-17T04:00:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 --2- 172.21.15.129:0/3215310257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100cf1d0 0x7f76100cf5e0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 --2- 172.21.15.129:0/3215310257 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f76100ccc80 0x7f76100cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 --2- 172.21.15.129:0/3215310257 >> v2:172.21.15.181:3300/0 conn(0x7f7610099570 0x7f761006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 -- 172.21.15.129:0/3215310257 >> 172.21.15.129:0/3215310257 conn(0x7f761006d410 msgr2=0x7f76100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:18.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 -- 172.21.15.129:0/3215310257 shutdown_connections 2024-09-17T04:00:18.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.453+0000 7f7614e00640 1 -- 172.21.15.129:0/3215310257 wait complete. 2024-09-17T04:00:18.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.455+0000 7f7614e00640 1 Processor -- start 2024-09-17T04:00:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.456+0000 7f7614e00640 1 -- start start 2024-09-17T04:00:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 0x7f761015eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 0x7f761015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7610160990 con 0x7f76100ccc80 2024-09-17T04:00:18.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7610160810 con 0x7f76100cf1d0 2024-09-17T04:00:18.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f7614e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7610160b10 con 0x7f7610099570 2024-09-17T04:00:18.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f760d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f760d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35370/0 (socket says 172.21.15.129:35370) 2024-09-17T04:00:18.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.457+0000 7f760d400640 1 -- 172.21.15.129:0/1962441176 learned_addr learned my addr 172.21.15.129:0/1962441176 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:18.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760de00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 0x7f761015eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760e800640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 0x7f761015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 -- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 msgr2=0x7f761015eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 0x7f761015eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 -- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 msgr2=0x7f761015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 0x7f761015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 -- 172.21.15.129:0/1962441176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76100cc900 con 0x7f76100ccc80 2024-09-17T04:00:18.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760de00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 0x7f761015eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760e800640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 0x7f761015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f760d400640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00d680 tx=0x7f75fc00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc017070 con 0x7f76100ccc80 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75fc005310 con 0x7f76100ccc80 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.458+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc009820 con 0x7f76100ccc80 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.459+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761015d310 con 0x7f76100ccc80 2024-09-17T04:00:18.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.459+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76100cad20 con 0x7f76100ccc80 2024-09-17T04:00:18.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.460+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7610160f70 con 0x7f76100ccc80 2024-09-17T04:00:18.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.461+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f75fc0099c0 con 0x7f76100ccc80 2024-09-17T04:00:18.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.462+0000 7f7606000640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 0x7f75f007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.462+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f75fc09ae00 con 0x7f76100ccc80 2024-09-17T04:00:18.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.463+0000 7f760de00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 0x7f75f007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.464+0000 7f760de00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 0x7f75f007b0d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7600005fd0 tx=0x7f76000098d0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:18.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.467+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75fc0631a0 con 0x7f76100ccc80 2024-09-17T04:00:18.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.685+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f761006a140 con 0x7f76100ccc80 2024-09-17T04:00:18.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.685+0000 7f7606000640 1 -- 172.21.15.129:0/1962441176 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f75fc062aa0 con 0x7f76100ccc80 2024-09-17T04:00:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:18.686 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:18.686 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 msgr2=0x7f75f007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 0x7f75f007b0d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f7600005fd0 tx=0x7f76000098d0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 msgr2=0x7f761015f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00d680 tx=0x7f75fc00db40 comp rx=0 tx=0).stop 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 shutdown_connections 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75f0078bc0 0x7f75f007b0d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f76100cf1d0 0x7f761015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f76100ccc80 0x7f761015f070 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 --2- 172.21.15.129:0/1962441176 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7610099570 0x7f761015eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 >> 172.21.15.129:0/1962441176 conn(0x7f761006d410 msgr2=0x7f76100b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:18.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 shutdown_connections 2024-09-17T04:00:18.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.688+0000 7f7614e00640 1 -- 172.21.15.129:0/1962441176 wait complete. 2024-09-17T04:00:18.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:18.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 -- 172.21.15.129:0/2977977099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b40705a0 msgr2=0x7f79b40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 --2- 172.21.15.129:0/2977977099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b40705a0 0x7f79b40b4140 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f79b4072080 tx=0x7f79a402a560 comp rx=0 tx=0).stop 2024-09-17T04:00:18.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 -- 172.21.15.129:0/2977977099 shutdown_connections 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 --2- 172.21.15.129:0/2977977099 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b406f2e0 0x7f79b40cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 --2- 172.21.15.129:0/2977977099 >> v2:172.21.15.181:3300/0 conn(0x7f79b406eed0 0x7f79b40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 --2- 172.21.15.129:0/2977977099 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b40705a0 0x7f79b40b4140 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 -- 172.21.15.129:0/2977977099 >> 172.21.15.129:0/2977977099 conn(0x7f79b406d410 msgr2=0x7f79b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 -- 172.21.15.129:0/2977977099 shutdown_connections 2024-09-17T04:00:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.935+0000 7f79bac00640 1 -- 172.21.15.129:0/2977977099 wait complete. 2024-09-17T04:00:18.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.937+0000 7f79bac00640 1 Processor -- start 2024-09-17T04:00:18.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.937+0000 7f79bac00640 1 -- start start 2024-09-17T04:00:18.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79b406eed0 0x7f79b40c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 0x7f79b41583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4160b40 con 0x7f79b406f2e0 2024-09-17T04:00:18.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79b41609c0 con 0x7f79b406eed0 2024-09-17T04:00:18.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79bac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f79b4160cc0 con 0x7f79b40705a0 2024-09-17T04:00:18.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35394/0 (socket says 172.21.15.129:35394) 2024-09-17T04:00:18.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 -- 172.21.15.129:0/1259843111 learned_addr learned my addr 172.21.15.129:0/1259843111 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79b3e00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79b406eed0 0x7f79b40c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79b8c00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 0x7f79b41583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 -- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 msgr2=0x7f79b41583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 0x7f79b41583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 -- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79b406eed0 msgr2=0x7f79b40c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.939+0000 7f79abe00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79b406eed0 0x7f79b40c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79abe00640 1 -- 172.21.15.129:0/1259843111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b41f4030 con 0x7f79b406f2e0 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79b8c00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 0x7f79b41583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79abe00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f799c00d690 tx=0x7f799c00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c017070 con 0x7f79b406f2e0 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f799c009ab0 con 0x7f79b406f2e0 2024-09-17T04:00:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799c010460 con 0x7f79b406f2e0 2024-09-17T04:00:18.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b41f4310 con 0x7f79b406f2e0 2024-09-17T04:00:18.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.940+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b41f48d0 con 0x7f79b406f2e0 2024-09-17T04:00:18.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.941+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b41f3da0 con 0x7f79b406f2e0 2024-09-17T04:00:18.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.943+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f799c01a070 con 0x7f79b406f2e0 2024-09-17T04:00:18.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.944+0000 7f79b0c00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 0x7f799007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:18.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.944+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f799c09ad60 con 0x7f79b406f2e0 2024-09-17T04:00:18.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.945+0000 7f79b3e00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 0x7f799007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:18.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.945+0000 7f79b3e00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 0x7f799007b0a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f79a4033000 tx=0x7f79a4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:18.949+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f799c062ab0 con 0x7f79b406f2e0 2024-09-17T04:00:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.132+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f79b4161250 con 0x7f79b406f2e0 2024-09-17T04:00:19.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.133+0000 7f79b0c00640 1 -- 172.21.15.129:0/1259843111 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+16787 (secure 0 0 0) 0x7f799c0674e0 con 0x7f79b406f2e0 2024-09-17T04:00:19.133 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:19.133 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":163,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:16.178377+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:12.563914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:13.619002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 msgr2=0x7f799007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 0x7f799007b0a0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f79a4033000 tx=0x7f79a4004730 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 msgr2=0x7f79b40c9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f799c00d690 tx=0x7f799c00db50 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 shutdown_connections 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7990078b90 0x7f799007b0a0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f79b40705a0 0x7f79b41583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79b406f2e0 0x7f79b40c9ee0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 --2- 172.21.15.129:0/1259843111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79b406eed0 0x7f79b40c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.135+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 >> 172.21.15.129:0/1259843111 conn(0x7f79b406d410 msgr2=0x7f79b40cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:19.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.136+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 shutdown_connections 2024-09-17T04:00:19.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.136+0000 7f79bac00640 1 -- 172.21.15.129:0/1259843111 wait complete. 2024-09-17T04:00:19.161 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:00:19.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:19.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 -- 172.21.15.129:0/2355988249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4015ef20 msgr2=0x7f2d400d09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 --2- 172.21.15.129:0/2355988249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4015ef20 0x7f2d400d09f0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f2d38005fd0 tx=0x7f2d3802b980 comp rx=0 tx=0).stop 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 -- 172.21.15.129:0/2355988249 shutdown_connections 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 --2- 172.21.15.129:0/2355988249 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d4015ef20 0x7f2d400d09f0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 --2- 172.21.15.129:0/2355988249 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d400d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 -- 172.21.15.129:0/2355988249 >> 172.21.15.129:0/2355988249 conn(0x7f2d4006cfc0 msgr2=0x7f2d4006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 -- 172.21.15.129:0/2355988249 shutdown_connections 2024-09-17T04:00:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.383+0000 7f2d44c00640 1 -- 172.21.15.129:0/2355988249 wait complete. 2024-09-17T04:00:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.385+0000 7f2d44c00640 1 Processor -- start 2024-09-17T04:00:19.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.385+0000 7f2d44c00640 1 -- start start 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d4015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4015ef20 0x7f2d4015d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d400ce310 con 0x7f2d40072d30 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2d400ce190 con 0x7f2d4015ef20 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d44c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2d400ce490 con 0x7f2d4015bb00 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d3f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d3f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35414/0 (socket says 172.21.15.129:35414) 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d3f400640 1 -- 172.21.15.129:0/3139935767 learned_addr learned my addr 172.21.15.129:0/3139935767 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d3fe00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4015ef20 0x7f2d4015d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.387+0000 7f2d3ea00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d4015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 -- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 msgr2=0x7f2d4015c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d4015c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 -- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4015ef20 msgr2=0x7f2d4015d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4015ef20 0x7f2d4015d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 -- 172.21.15.129:0/3139935767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d401f5600 con 0x7f2d40072d30 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3ea00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d4015c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d3f400640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000e790 tx=0x7f2d3000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3000eea0 con 0x7f2d40072d30 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d3000cdb0 con 0x7f2d40072d30 2024-09-17T04:00:19.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d40159630 con 0x7f2d40072d30 2024-09-17T04:00:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30010690 con 0x7f2d40072d30 2024-09-17T04:00:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.388+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d401f58a0 con 0x7f2d40072d30 2024-09-17T04:00:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.389+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d400ce9b0 con 0x7f2d40072d30 2024-09-17T04:00:19.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.391+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2d30010830 con 0x7f2d40072d30 2024-09-17T04:00:19.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.391+0000 7f2d37400640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.392+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f2d3009b6a0 con 0x7f2d40072d30 2024-09-17T04:00:19.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.392+0000 7f2d3ea00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.392+0000 7f2d3ea00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c007fd0 tx=0x7f2d2c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:19.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.396+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2d30063520 con 0x7f2d40072d30 2024-09-17T04:00:19.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.576+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d401f5450 con 0x7f2d20078b90 2024-09-17T04:00:19.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.581+0000 7f2d37400640 1 -- 172.21.15.129:0/3139935767 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248843 (secure 0 0 0) 0x7f2d401f5450 con 0x7f2d20078b90 2024-09-17T04:00:19.584 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:19.586 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 msgr2=0x7f2d2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c007fd0 tx=0x7f2d2c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 msgr2=0x7f2d400aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000e790 tx=0x7f2d3000ec50 comp rx=0 tx=0).stop 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 shutdown_connections 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d4015ef20 0x7f2d4015d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d4015bb00 0x7f2d4015c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 --2- 172.21.15.129:0/3139935767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d40072d30 0x7f2d400aed50 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 >> 172.21.15.129:0/3139935767 conn(0x7f2d4006cfc0 msgr2=0x7f2d400cf580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 shutdown_connections 2024-09-17T04:00:19.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.585+0000 7f2d44c00640 1 -- 172.21.15.129:0/3139935767 wait complete. 2024-09-17T04:00:19.607 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-17T04:00:18.476783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":105,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324652,"kb_used_data":16904,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754650068,"statfs":{"total":773094113280,"available":772761669632,"internally_reserved":0,"allocated":17309696,"data_stored":8212966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.325327"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167060+0000","last_change":"2024-09-17T04:00:14.167060+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167060+0000","last_undegraded":"2024-09-17T04:00:14.167060+0000","last_fullsized":"2024-09-17T04:00:14.167060+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977616+0000","last_change":"2024-09-17T04:00:15.198110+0000","last_active":"2024-09-17T04:00:17.977616+0000","last_peered":"2024-09-17T04:00:17.977616+0000","last_clean":"2024-09-17T04:00:17.977616+0000","last_became_active":"2024-09-17T04:00:15.197746+0000","last_became_peered":"2024-09-17T04:00:15.197746+0000","last_unstale":"2024-09-17T04:00:17.977616+0000","last_undegraded":"2024-09-17T04:00:17.977616+0000","last_fullsized":"2024-09-17T04:00:17.977616+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.169862+0000","last_change":"2024-09-17T04:00:14.169862+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.169862+0000","last_undegraded":"2024-09-17T04:00:14.169862+0000","last_fullsized":"2024-09-17T04:00:14.169862+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977850+0000","last_change":"2024-09-17T04:00:15.198044+0000","last_active":"2024-09-17T04:00:17.977850+0000","last_peered":"2024-09-17T04:00:17.977850+0000","last_clean":"2024-09-17T04:00:17.977850+0000","last_became_active":"2024-09-17T04:00:15.197647+0000","last_became_peered":"2024-09-17T04:00:15.197647+0000","last_unstale":"2024-09-17T04:00:17.977850+0000","last_undegraded":"2024-09-17T04:00:17.977850+0000","last_fullsized":"2024-09-17T04:00:17.977850+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.114514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.173655+0000","last_change":"2024-09-17T04:00:14.173655+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.173655+0000","last_undegraded":"2024-09-17T04:00:14.173655+0000","last_fullsized":"2024-09-17T04:00:14.173655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163297+0000","last_change":"2024-09-17T04:00:14.163297+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163297+0000","last_undegraded":"2024-09-17T04:00:14.163297+0000","last_fullsized":"2024-09-17T04:00:14.163297+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158538+0000","last_change":"2024-09-17T04:00:14.158538+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158538+0000","last_undegraded":"2024-09-17T04:00:14.158538+0000","last_fullsized":"2024-09-17T04:00:14.158538+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163875+0000","last_change":"2024-09-17T04:00:14.163875+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163875+0000","last_undegraded":"2024-09-17T04:00:14.163875+0000","last_fullsized":"2024-09-17T04:00:14.163875+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161997+0000","last_change":"2024-09-17T04:00:14.161997+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161997+0000","last_undegraded":"2024-09-17T04:00:14.161997+0000","last_fullsized":"2024-09-17T04:00:14.161997+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185181+0000","last_change":"2024-09-17T04:00:15.187116+0000","last_active":"2024-09-17T04:00:16.185181+0000","last_peered":"2024-09-17T04:00:16.185181+0000","last_clean":"2024-09-17T04:00:16.185181+0000","last_became_active":"2024-09-17T04:00:15.186491+0000","last_became_peered":"2024-09-17T04:00:15.186491+0000","last_unstale":"2024-09-17T04:00:16.185181+0000","last_undegraded":"2024-09-17T04:00:16.185181+0000","last_fullsized":"2024-09-17T04:00:16.185181+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:20.978854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.159485+0000","last_change":"2024-09-17T04:00:14.159485+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.159485+0000","last_undegraded":"2024-09-17T04:00:14.159485+0000","last_fullsized":"2024-09-17T04:00:14.159485+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977913+0000","last_change":"2024-09-17T04:00:15.202361+0000","last_active":"2024-09-17T04:00:17.977913+0000","last_peered":"2024-09-17T04:00:17.977913+0000","last_clean":"2024-09-17T04:00:17.977913+0000","last_became_active":"2024-09-17T04:00:15.202042+0000","last_became_peered":"2024-09-17T04:00:15.202042+0000","last_unstale":"2024-09-17T04:00:17.977913+0000","last_undegraded":"2024-09-17T04:00:17.977913+0000","last_fullsized":"2024-09-17T04:00:17.977913+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:31.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977369+0000","last_change":"2024-09-17T04:00:15.205859+0000","last_active":"2024-09-17T04:00:17.977369+0000","last_peered":"2024-09-17T04:00:17.977369+0000","last_clean":"2024-09-17T04:00:17.977369+0000","last_became_active":"2024-09-17T04:00:15.205527+0000","last_became_peered":"2024-09-17T04:00:15.205527+0000","last_unstale":"2024-09-17T04:00:17.977369+0000","last_undegraded":"2024-09-17T04:00:17.977369+0000","last_fullsized":"2024-09-17T04:00:17.977369+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:02:13.228284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160861+0000","last_change":"2024-09-17T04:00:14.160861+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160861+0000","last_undegraded":"2024-09-17T04:00:14.160861+0000","last_fullsized":"2024-09-17T04:00:14.160861+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977556+0000","last_change":"2024-09-17T04:00:15.195580+0000","last_active":"2024-09-17T04:00:17.977556+0000","last_peered":"2024-09-17T04:00:17.977556+0000","last_clean":"2024-09-17T04:00:17.977556+0000","last_became_active":"2024-09-17T04:00:15.194025+0000","last_became_peered":"2024-09-17T04:00:15.194025+0000","last_unstale":"2024-09-17T04:00:17.977556+0000","last_undegraded":"2024-09-17T04:00:17.977556+0000","last_fullsized":"2024-09-17T04:00:17.977556+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:34.997014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163519+0000","last_change":"2024-09-17T04:00:14.163519+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163519+0000","last_undegraded":"2024-09-17T04:00:14.163519+0000","last_fullsized":"2024-09-17T04:00:14.163519+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.170293+0000","last_change":"2024-09-17T04:00:14.170293+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.170293+0000","last_undegraded":"2024-09-17T04:00:14.170293+0000","last_fullsized":"2024-09-17T04:00:14.170293+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166156+0000","last_change":"2024-09-17T04:00:14.166156+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166156+0000","last_undegraded":"2024-09-17T04:00:14.166156+0000","last_fullsized":"2024-09-17T04:00:14.166156+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184909+0000","last_change":"2024-09-17T04:00:15.185185+0000","last_active":"2024-09-17T04:00:16.184909+0000","last_peered":"2024-09-17T04:00:16.184909+0000","last_clean":"2024-09-17T04:00:16.184909+0000","last_became_active":"2024-09-17T04:00:15.182697+0000","last_became_peered":"2024-09-17T04:00:15.182697+0000","last_unstale":"2024-09-17T04:00:16.184909+0000","last_undegraded":"2024-09-17T04:00:16.184909+0000","last_fullsized":"2024-09-17T04:00:16.184909+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:06.451276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171386+0000","last_change":"2024-09-17T04:00:14.171386+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171386+0000","last_undegraded":"2024-09-17T04:00:14.171386+0000","last_fullsized":"2024-09-17T04:00:14.171386+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171738+0000","last_change":"2024-09-17T04:00:14.171738+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171738+0000","last_undegraded":"2024-09-17T04:00:14.171738+0000","last_fullsized":"2024-09-17T04:00:14.171738+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184888+0000","last_change":"2024-09-17T04:00:15.187015+0000","last_active":"2024-09-17T04:00:16.184888+0000","last_peered":"2024-09-17T04:00:16.184888+0000","last_clean":"2024-09-17T04:00:16.184888+0000","last_became_active":"2024-09-17T04:00:15.186574+0000","last_became_peered":"2024-09-17T04:00:15.186574+0000","last_unstale":"2024-09-17T04:00:16.184888+0000","last_undegraded":"2024-09-17T04:00:16.184888+0000","last_fullsized":"2024-09-17T04:00:16.184888+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:05.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167448+0000","last_change":"2024-09-17T04:00:14.167448+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167448+0000","last_undegraded":"2024-09-17T04:00:14.167448+0000","last_fullsized":"2024-09-17T04:00:14.167448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161007+0000","last_change":"2024-09-17T04:00:14.161007+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161007+0000","last_undegraded":"2024-09-17T04:00:14.161007+0000","last_fullsized":"2024-09-17T04:00:14.161007+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977962+0000","last_change":"2024-09-17T04:00:15.196465+0000","last_active":"2024-09-17T04:00:17.977962+0000","last_peered":"2024-09-17T04:00:17.977962+0000","last_clean":"2024-09-17T04:00:17.977962+0000","last_became_active":"2024-09-17T04:00:15.195761+0000","last_became_peered":"2024-09-17T04:00:15.195761+0000","last_unstale":"2024-09-17T04:00:17.977962+0000","last_undegraded":"2024-09-17T04:00:17.977962+0000","last_fullsized":"2024-09-17T04:00:17.977962+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:29.557541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157433+0000","last_change":"2024-09-17T04:00:14.157433+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157433+0000","last_undegraded":"2024-09-17T04:00:14.157433+0000","last_fullsized":"2024-09-17T04:00:14.157433+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.169429+0000","last_change":"2024-09-17T04:00:14.169429+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.169429+0000","last_undegraded":"2024-09-17T04:00:14.169429+0000","last_fullsized":"2024-09-17T04:00:14.169429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.170364+0000","last_change":"2024-09-17T04:00:14.170364+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.170364+0000","last_undegraded":"2024-09-17T04:00:14.170364+0000","last_fullsized":"2024-09-17T04:00:14.170364+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977568+0000","last_change":"2024-09-17T04:00:15.203508+0000","last_active":"2024-09-17T04:00:17.977568+0000","last_peered":"2024-09-17T04:00:17.977568+0000","last_clean":"2024-09-17T04:00:17.977568+0000","last_became_active":"2024-09-17T04:00:15.203187+0000","last_became_peered":"2024-09-17T04:00:15.203187+0000","last_unstale":"2024-09-17T04:00:17.977568+0000","last_undegraded":"2024-09-17T04:00:17.977568+0000","last_fullsized":"2024-09-17T04:00:17.977568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:55.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160314+0000","last_change":"2024-09-17T04:00:14.160314+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160314+0000","last_undegraded":"2024-09-17T04:00:14.160314+0000","last_fullsized":"2024-09-17T04:00:14.160314+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166972+0000","last_change":"2024-09-17T04:00:14.166972+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166972+0000","last_undegraded":"2024-09-17T04:00:14.166972+0000","last_fullsized":"2024-09-17T04:00:14.166972+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171353+0000","last_change":"2024-09-17T04:00:14.171353+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171353+0000","last_undegraded":"2024-09-17T04:00:14.171353+0000","last_fullsized":"2024-09-17T04:00:14.171353+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161968+0000","last_change":"2024-09-17T04:00:14.161968+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161968+0000","last_undegraded":"2024-09-17T04:00:14.161968+0000","last_fullsized":"2024-09-17T04:00:14.161968+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186077+0000","last_change":"2024-09-17T04:00:14.184544+0000","last_active":"2024-09-17T04:00:14.186077+0000","last_peered":"2024-09-17T04:00:14.186077+0000","last_clean":"2024-09-17T04:00:14.186077+0000","last_became_active":"2024-09-17T04:00:14.184254+0000","last_became_peered":"2024-09-17T04:00:14.184254+0000","last_unstale":"2024-09-17T04:00:14.186077+0000","last_undegraded":"2024-09-17T04:00:14.186077+0000","last_fullsized":"2024-09-17T04:00:14.186077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:05.130672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186087+0000","last_change":"2024-09-17T04:00:15.203100+0000","last_active":"2024-09-17T04:00:16.186087+0000","last_peered":"2024-09-17T04:00:16.186087+0000","last_clean":"2024-09-17T04:00:16.186087+0000","last_became_active":"2024-09-17T04:00:15.202011+0000","last_became_peered":"2024-09-17T04:00:15.202011+0000","last_unstale":"2024-09-17T04:00:16.186087+0000","last_undegraded":"2024-09-17T04:00:16.186087+0000","last_fullsized":"2024-09-17T04:00:16.186087+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:07.578261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.184031+0000","last_change":"2024-09-17T04:00:14.178919+0000","last_active":"2024-09-17T04:00:14.184031+0000","last_peered":"2024-09-17T04:00:14.184031+0000","last_clean":"2024-09-17T04:00:14.184031+0000","last_became_active":"2024-09-17T04:00:14.177710+0000","last_became_peered":"2024-09-17T04:00:14.177710+0000","last_unstale":"2024-09-17T04:00:14.184031+0000","last_undegraded":"2024-09-17T04:00:14.184031+0000","last_fullsized":"2024-09-17T04:00:14.184031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:27.461533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.159468+0000","last_change":"2024-09-17T04:00:14.159468+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.159468+0000","last_undegraded":"2024-09-17T04:00:14.159468+0000","last_fullsized":"2024-09-17T04:00:14.159468+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.182124+0000","last_change":"2024-09-17T04:00:14.173566+0000","last_active":"2024-09-17T04:00:14.182124+0000","last_peered":"2024-09-17T04:00:14.182124+0000","last_clean":"2024-09-17T04:00:14.182124+0000","last_became_active":"2024-09-17T04:00:14.173201+0000","last_became_peered":"2024-09-17T04:00:14.173201+0000","last_unstale":"2024-09-17T04:00:14.182124+0000","last_undegraded":"2024-09-17T04:00:14.182124+0000","last_fullsized":"2024-09-17T04:00:14.182124+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:45:33.665800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160725+0000","last_change":"2024-09-17T04:00:14.160725+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160725+0000","last_undegraded":"2024-09-17T04:00:14.160725+0000","last_fullsized":"2024-09-17T04:00:14.160725+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977943+0000","last_change":"2024-09-17T04:00:14.180751+0000","last_active":"2024-09-17T04:00:17.977943+0000","last_peered":"2024-09-17T04:00:17.977943+0000","last_clean":"2024-09-17T04:00:17.977943+0000","last_became_active":"2024-09-17T04:00:14.180400+0000","last_became_peered":"2024-09-17T04:00:14.180400+0000","last_unstale":"2024-09-17T04:00:17.977943+0000","last_undegraded":"2024-09-17T04:00:17.977943+0000","last_fullsized":"2024-09-17T04:00:17.977943+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:14.945761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185444+0000","last_change":"2024-09-17T04:00:15.183077+0000","last_active":"2024-09-17T04:00:16.185444+0000","last_peered":"2024-09-17T04:00:16.185444+0000","last_clean":"2024-09-17T04:00:16.185444+0000","last_became_active":"2024-09-17T04:00:15.182579+0000","last_became_peered":"2024-09-17T04:00:15.182579+0000","last_unstale":"2024-09-17T04:00:16.185444+0000","last_undegraded":"2024-09-17T04:00:16.185444+0000","last_fullsized":"2024-09-17T04:00:16.185444+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:32.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185456+0000","last_change":"2024-09-17T04:00:14.178007+0000","last_active":"2024-09-17T04:00:14.185456+0000","last_peered":"2024-09-17T04:00:14.185456+0000","last_clean":"2024-09-17T04:00:14.185456+0000","last_became_active":"2024-09-17T04:00:14.177397+0000","last_became_peered":"2024-09-17T04:00:14.177397+0000","last_unstale":"2024-09-17T04:00:14.185456+0000","last_undegraded":"2024-09-17T04:00:14.185456+0000","last_fullsized":"2024-09-17T04:00:14.185456+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:41.553403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161193+0000","last_change":"2024-09-17T04:00:14.161193+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161193+0000","last_undegraded":"2024-09-17T04:00:14.161193+0000","last_fullsized":"2024-09-17T04:00:14.161193+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185788+0000","last_change":"2024-09-17T04:00:14.181701+0000","last_active":"2024-09-17T04:00:14.185788+0000","last_peered":"2024-09-17T04:00:14.185788+0000","last_clean":"2024-09-17T04:00:14.185788+0000","last_became_active":"2024-09-17T04:00:14.180617+0000","last_became_peered":"2024-09-17T04:00:14.180617+0000","last_unstale":"2024-09-17T04:00:14.185788+0000","last_undegraded":"2024-09-17T04:00:14.185788+0000","last_fullsized":"2024-09-17T04:00:14.185788+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:31.039000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157272+0000","last_change":"2024-09-17T04:00:14.157272+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157272+0000","last_undegraded":"2024-09-17T04:00:14.157272+0000","last_fullsized":"2024-09-17T04:00:14.157272+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.184030+0000","last_change":"2024-09-17T04:00:14.180760+0000","last_active":"2024-09-17T04:00:14.184030+0000","last_peered":"2024-09-17T04:00:14.184030+0000","last_clean":"2024-09-17T04:00:14.184030+0000","last_became_active":"2024-09-17T04:00:14.180063+0000","last_became_peered":"2024-09-17T04:00:14.180063+0000","last_unstale":"2024-09-17T04:00:14.184030+0000","last_undegraded":"2024-09-17T04:00:14.184030+0000","last_fullsized":"2024-09-17T04:00:14.184030+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:01.751763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166153+0000","last_change":"2024-09-17T04:00:14.166153+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166153+0000","last_undegraded":"2024-09-17T04:00:14.166153+0000","last_fullsized":"2024-09-17T04:00:14.166153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.178035+0000","last_change":"2024-09-17T04:00:14.170243+0000","last_active":"2024-09-17T04:00:14.178035+0000","last_peered":"2024-09-17T04:00:14.178035+0000","last_clean":"2024-09-17T04:00:14.178035+0000","last_became_active":"2024-09-17T04:00:14.169878+0000","last_became_peered":"2024-09-17T04:00:14.169878+0000","last_unstale":"2024-09-17T04:00:14.178035+0000","last_undegraded":"2024-09-17T04:00:14.178035+0000","last_fullsized":"2024-09-17T04:00:14.178035+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:00.433111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162135+0000","last_change":"2024-09-17T04:00:14.162135+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162135+0000","last_undegraded":"2024-09-17T04:00:14.162135+0000","last_fullsized":"2024-09-17T04:00:14.162135+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181836+0000","last_change":"2024-09-17T04:00:14.178548+0000","last_active":"2024-09-17T04:00:14.181836+0000","last_peered":"2024-09-17T04:00:14.181836+0000","last_clean":"2024-09-17T04:00:14.181836+0000","last_became_active":"2024-09-17T04:00:14.178173+0000","last_became_peered":"2024-09-17T04:00:14.178173+0000","last_unstale":"2024-09-17T04:00:14.181836+0000","last_undegraded":"2024-09-17T04:00:14.181836+0000","last_fullsized":"2024-09-17T04:00:14.181836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:42.480246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157748+0000","last_change":"2024-09-17T04:00:14.157748+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157748+0000","last_undegraded":"2024-09-17T04:00:14.157748+0000","last_fullsized":"2024-09-17T04:00:14.157748+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186156+0000","last_change":"2024-09-17T04:00:14.184902+0000","last_active":"2024-09-17T04:00:14.186156+0000","last_peered":"2024-09-17T04:00:14.186156+0000","last_clean":"2024-09-17T04:00:14.186156+0000","last_became_active":"2024-09-17T04:00:14.184624+0000","last_became_peered":"2024-09-17T04:00:14.184624+0000","last_unstale":"2024-09-17T04:00:14.186156+0000","last_undegraded":"2024-09-17T04:00:14.186156+0000","last_fullsized":"2024-09-17T04:00:14.186156+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:59.180634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185090+0000","last_change":"2024-09-17T04:00:15.180729+0000","last_active":"2024-09-17T04:00:16.185090+0000","last_peered":"2024-09-17T04:00:16.185090+0000","last_clean":"2024-09-17T04:00:16.185090+0000","last_became_active":"2024-09-17T04:00:15.180358+0000","last_became_peered":"2024-09-17T04:00:15.180358+0000","last_unstale":"2024-09-17T04:00:16.185090+0000","last_undegraded":"2024-09-17T04:00:16.185090+0000","last_fullsized":"2024-09-17T04:00:16.185090+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:55.762158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181150+0000","last_change":"2024-09-17T04:00:14.177482+0000","last_active":"2024-09-17T04:00:14.181150+0000","last_peered":"2024-09-17T04:00:14.181150+0000","last_clean":"2024-09-17T04:00:14.181150+0000","last_became_active":"2024-09-17T04:00:14.177185+0000","last_became_peered":"2024-09-17T04:00:14.177185+0000","last_unstale":"2024-09-17T04:00:14.181150+0000","last_undegraded":"2024-09-17T04:00:14.181150+0000","last_fullsized":"2024-09-17T04:00:14.181150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:32.018406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162766+0000","last_change":"2024-09-17T04:00:14.162766+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162766+0000","last_undegraded":"2024-09-17T04:00:14.162766+0000","last_fullsized":"2024-09-17T04:00:14.162766+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183960+0000","last_change":"2024-09-17T04:00:14.180486+0000","last_active":"2024-09-17T04:00:14.183960+0000","last_peered":"2024-09-17T04:00:14.183960+0000","last_clean":"2024-09-17T04:00:14.183960+0000","last_became_active":"2024-09-17T04:00:14.180077+0000","last_became_peered":"2024-09-17T04:00:14.180077+0000","last_unstale":"2024-09-17T04:00:14.183960+0000","last_undegraded":"2024-09-17T04:00:14.183960+0000","last_fullsized":"2024-09-17T04:00:14.183960+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:25.031267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158293+0000","last_change":"2024-09-17T04:00:14.158293+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158293+0000","last_undegraded":"2024-09-17T04:00:14.158293+0000","last_fullsized":"2024-09-17T04:00:14.158293+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185646+0000","last_change":"2024-09-17T04:00:14.179129+0000","last_active":"2024-09-17T04:00:16.185646+0000","last_peered":"2024-09-17T04:00:16.185646+0000","last_clean":"2024-09-17T04:00:16.185646+0000","last_became_active":"2024-09-17T04:00:14.177651+0000","last_became_peered":"2024-09-17T04:00:14.177651+0000","last_unstale":"2024-09-17T04:00:16.185646+0000","last_undegraded":"2024-09-17T04:00:16.185646+0000","last_fullsized":"2024-09-17T04:00:16.185646+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:50.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158568+0000","last_change":"2024-09-17T04:00:14.158568+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158568+0000","last_undegraded":"2024-09-17T04:00:14.158568+0000","last_fullsized":"2024-09-17T04:00:14.158568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181930+0000","last_change":"2024-09-17T04:00:14.178903+0000","last_active":"2024-09-17T04:00:14.181930+0000","last_peered":"2024-09-17T04:00:14.181930+0000","last_clean":"2024-09-17T04:00:14.181930+0000","last_became_active":"2024-09-17T04:00:14.178605+0000","last_became_peered":"2024-09-17T04:00:14.178605+0000","last_unstale":"2024-09-17T04:00:14.181930+0000","last_undegraded":"2024-09-17T04:00:14.181930+0000","last_fullsized":"2024-09-17T04:00:14.181930+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:34.211612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184829+0000","last_change":"2024-09-17T04:00:15.192316+0000","last_active":"2024-09-17T04:00:16.184829+0000","last_peered":"2024-09-17T04:00:16.184829+0000","last_clean":"2024-09-17T04:00:16.184829+0000","last_became_active":"2024-09-17T04:00:15.188357+0000","last_became_peered":"2024-09-17T04:00:15.188357+0000","last_unstale":"2024-09-17T04:00:16.184829+0000","last_undegraded":"2024-09-17T04:00:16.184829+0000","last_fullsized":"2024-09-17T04:00:16.184829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:34.996367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185523+0000","last_change":"2024-09-17T04:00:14.182987+0000","last_active":"2024-09-17T04:00:14.185523+0000","last_peered":"2024-09-17T04:00:14.185523+0000","last_clean":"2024-09-17T04:00:14.185523+0000","last_became_active":"2024-09-17T04:00:14.182665+0000","last_became_peered":"2024-09-17T04:00:14.182665+0000","last_unstale":"2024-09-17T04:00:14.185523+0000","last_undegraded":"2024-09-17T04:00:14.185523+0000","last_fullsized":"2024-09-17T04:00:14.185523+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:22.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185423+0000","last_change":"2024-09-17T04:00:15.188466+0000","last_active":"2024-09-17T04:00:16.185423+0000","last_peered":"2024-09-17T04:00:16.185423+0000","last_clean":"2024-09-17T04:00:16.185423+0000","last_became_active":"2024-09-17T04:00:15.187860+0000","last_became_peered":"2024-09-17T04:00:15.187860+0000","last_unstale":"2024-09-17T04:00:16.185423+0000","last_undegraded":"2024-09-17T04:00:16.185423+0000","last_fullsized":"2024-09-17T04:00:16.185423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:05.989755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183887+0000","last_change":"2024-09-17T04:00:14.179849+0000","last_active":"2024-09-17T04:00:14.183887+0000","last_peered":"2024-09-17T04:00:14.183887+0000","last_clean":"2024-09-17T04:00:14.183887+0000","last_became_active":"2024-09-17T04:00:14.178084+0000","last_became_peered":"2024-09-17T04:00:14.178084+0000","last_unstale":"2024-09-17T04:00:14.183887+0000","last_undegraded":"2024-09-17T04:00:14.183887+0000","last_fullsized":"2024-09-17T04:00:14.183887+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:16.538235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161180+0000","last_change":"2024-09-17T04:00:14.161180+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161180+0000","last_undegraded":"2024-09-17T04:00:14.161180+0000","last_fullsized":"2024-09-17T04:00:14.161180+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.179551+0000","last_change":"2024-09-17T04:00:14.173867+0000","last_active":"2024-09-17T04:00:14.179551+0000","last_peered":"2024-09-17T04:00:14.179551+0000","last_clean":"2024-09-17T04:00:14.179551+0000","last_became_active":"2024-09-17T04:00:14.171614+0000","last_became_peered":"2024-09-17T04:00:14.171614+0000","last_unstale":"2024-09-17T04:00:14.179551+0000","last_undegraded":"2024-09-17T04:00:14.179551+0000","last_fullsized":"2024-09-17T04:00:14.179551+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:22.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158480+0000","last_change":"2024-09-17T04:00:14.158480+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158480+0000","last_undegraded":"2024-09-17T04:00:14.158480+0000","last_fullsized":"2024-09-17T04:00:14.158480+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185558+0000","last_change":"2024-09-17T04:00:14.183120+0000","last_active":"2024-09-17T04:00:14.185558+0000","last_peered":"2024-09-17T04:00:14.185558+0000","last_clean":"2024-09-17T04:00:14.185558+0000","last_became_active":"2024-09-17T04:00:14.182779+0000","last_became_peered":"2024-09-17T04:00:14.182779+0000","last_unstale":"2024-09-17T04:00:14.185558+0000","last_undegraded":"2024-09-17T04:00:14.185558+0000","last_fullsized":"2024-09-17T04:00:14.185558+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:34.211612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977807+0000","last_change":"2024-09-17T04:00:15.204420+0000","last_active":"2024-09-17T04:00:17.977807+0000","last_peered":"2024-09-17T04:00:17.977807+0000","last_clean":"2024-09-17T04:00:17.977807+0000","last_became_active":"2024-09-17T04:00:15.202153+0000","last_became_peered":"2024-09-17T04:00:15.202153+0000","last_unstale":"2024-09-17T04:00:17.977807+0000","last_undegraded":"2024-09-17T04:00:17.977807+0000","last_fullsized":"2024-09-17T04:00:17.977807+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:57.466014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.168322+0000","last_change":"2024-09-17T04:00:14.164836+0000","last_active":"2024-09-17T04:00:14.168322+0000","last_peered":"2024-09-17T04:00:14.168322+0000","last_clean":"2024-09-17T04:00:14.168322+0000","last_became_active":"2024-09-17T04:00:14.164491+0000","last_became_peered":"2024-09-17T04:00:14.164491+0000","last_unstale":"2024-09-17T04:00:14.168322+0000","last_undegraded":"2024-09-17T04:00:14.168322+0000","last_fullsized":"2024-09-17T04:00:14.168322+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:40.826089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185709+0000","last_change":"2024-09-17T04:00:15.203883+0000","last_active":"2024-09-17T04:00:16.185709+0000","last_peered":"2024-09-17T04:00:16.185709+0000","last_clean":"2024-09-17T04:00:16.185709+0000","last_became_active":"2024-09-17T04:00:15.201414+0000","last_became_peered":"2024-09-17T04:00:15.201414+0000","last_unstale":"2024-09-17T04:00:16.185709+0000","last_undegraded":"2024-09-17T04:00:16.185709+0000","last_fullsized":"2024-09-17T04:00:16.185709+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:29.778689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.187098+0000","last_change":"2024-09-17T04:00:14.185714+0000","last_active":"2024-09-17T04:00:14.187098+0000","last_peered":"2024-09-17T04:00:14.187098+0000","last_clean":"2024-09-17T04:00:14.187098+0000","last_became_active":"2024-09-17T04:00:14.185363+0000","last_became_peered":"2024-09-17T04:00:14.185363+0000","last_unstale":"2024-09-17T04:00:14.187098+0000","last_undegraded":"2024-09-17T04:00:14.187098+0000","last_fullsized":"2024-09-17T04:00:14.187098+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:19.679615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157262+0000","last_change":"2024-09-17T04:00:14.157262+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157262+0000","last_undegraded":"2024-09-17T04:00:14.157262+0000","last_fullsized":"2024-09-17T04:00:14.157262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185905+0000","last_change":"2024-09-17T04:00:14.183023+0000","last_active":"2024-09-17T04:00:14.185905+0000","last_peered":"2024-09-17T04:00:14.185905+0000","last_clean":"2024-09-17T04:00:14.185905+0000","last_became_active":"2024-09-17T04:00:14.178678+0000","last_became_peered":"2024-09-17T04:00:14.178678+0000","last_unstale":"2024-09-17T04:00:14.185905+0000","last_undegraded":"2024-09-17T04:00:14.185905+0000","last_fullsized":"2024-09-17T04:00:14.185905+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:51.928961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977340+0000","last_change":"2024-09-17T04:00:15.202518+0000","last_active":"2024-09-17T04:00:17.977340+0000","last_peered":"2024-09-17T04:00:17.977340+0000","last_clean":"2024-09-17T04:00:17.977340+0000","last_became_active":"2024-09-17T04:00:15.202240+0000","last_became_peered":"2024-09-17T04:00:15.202240+0000","last_unstale":"2024-09-17T04:00:17.977340+0000","last_undegraded":"2024-09-17T04:00:17.977340+0000","last_fullsized":"2024-09-17T04:00:17.977340+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:59.197974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183836+0000","last_change":"2024-09-17T04:00:14.178227+0000","last_active":"2024-09-17T04:00:14.183836+0000","last_peered":"2024-09-17T04:00:14.183836+0000","last_clean":"2024-09-17T04:00:14.183836+0000","last_became_active":"2024-09-17T04:00:14.177922+0000","last_became_peered":"2024-09-17T04:00:14.177922+0000","last_unstale":"2024-09-17T04:00:14.183836+0000","last_undegraded":"2024-09-17T04:00:14.183836+0000","last_fullsized":"2024-09-17T04:00:14.183836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:34.330414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.168571+0000","last_change":"2024-09-17T04:00:14.168571+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.168571+0000","last_undegraded":"2024-09-17T04:00:14.168571+0000","last_fullsized":"2024-09-17T04:00:14.168571+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176884+0000","last_change":"2024-09-17T04:00:14.173438+0000","last_active":"2024-09-17T04:00:14.176884+0000","last_peered":"2024-09-17T04:00:14.176884+0000","last_clean":"2024-09-17T04:00:14.176884+0000","last_became_active":"2024-09-17T04:00:14.173090+0000","last_became_peered":"2024-09-17T04:00:14.173090+0000","last_unstale":"2024-09-17T04:00:14.176884+0000","last_undegraded":"2024-09-17T04:00:14.176884+0000","last_fullsized":"2024-09-17T04:00:14.176884+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:51.031094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.168610+0000","last_change":"2024-09-17T04:00:14.168610+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.168610+0000","last_undegraded":"2024-09-17T04:00:14.168610+0000","last_fullsized":"2024-09-17T04:00:14.168610+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.188590+0000","last_change":"2024-09-17T04:00:14.186855+0000","last_active":"2024-09-17T04:00:14.188590+0000","last_peered":"2024-09-17T04:00:14.188590+0000","last_clean":"2024-09-17T04:00:14.188590+0000","last_became_active":"2024-09-17T04:00:14.186535+0000","last_became_peered":"2024-09-17T04:00:14.186535+0000","last_unstale":"2024-09-17T04:00:14.188590+0000","last_undegraded":"2024-09-17T04:00:14.188590+0000","last_fullsized":"2024-09-17T04:00:14.188590+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:31.039000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186153+0000","last_change":"2024-09-17T04:00:15.194286+0000","last_active":"2024-09-17T04:00:16.186153+0000","last_peered":"2024-09-17T04:00:16.186153+0000","last_clean":"2024-09-17T04:00:16.186153+0000","last_became_active":"2024-09-17T04:00:15.193936+0000","last_became_peered":"2024-09-17T04:00:15.193936+0000","last_unstale":"2024-09-17T04:00:16.186153+0000","last_undegraded":"2024-09-17T04:00:16.186153+0000","last_fullsized":"2024-09-17T04:00:16.186153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:03.221443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"163'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185534+0000","last_change":"2024-09-17T04:00:14.176145+0000","last_active":"2024-09-17T04:00:16.185534+0000","last_peered":"2024-09-17T04:00:16.185534+0000","last_clean":"2024-09-17T04:00:16.185534+0000","last_became_active":"2024-09-17T04:00:14.175839+0000","last_became_peered":"2024-09-17T04:00:14.175839+0000","last_unstale":"2024-09-17T04:00:16.185534+0000","last_undegraded":"2024-09-17T04:00:16.185534+0000","last_fullsized":"2024-09-17T04:00:16.185534+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:33.949960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162812+0000","last_change":"2024-09-17T04:00:14.162812+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162812+0000","last_undegraded":"2024-09-17T04:00:14.162812+0000","last_fullsized":"2024-09-17T04:00:14.162812+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184972+0000","last_change":"2024-09-17T04:00:14.182944+0000","last_active":"2024-09-17T04:00:16.184972+0000","last_peered":"2024-09-17T04:00:16.184972+0000","last_clean":"2024-09-17T04:00:16.184972+0000","last_became_active":"2024-09-17T04:00:14.182626+0000","last_became_peered":"2024-09-17T04:00:14.182626+0000","last_unstale":"2024-09-17T04:00:16.184972+0000","last_undegraded":"2024-09-17T04:00:16.184972+0000","last_fullsized":"2024-09-17T04:00:16.184972+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:33.722613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162970+0000","last_change":"2024-09-17T04:00:14.162970+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162970+0000","last_undegraded":"2024-09-17T04:00:14.162970+0000","last_fullsized":"2024-09-17T04:00:14.162970+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176467+0000","last_change":"2024-09-17T04:00:14.172333+0000","last_active":"2024-09-17T04:00:14.176467+0000","last_peered":"2024-09-17T04:00:14.176467+0000","last_clean":"2024-09-17T04:00:14.176467+0000","last_became_active":"2024-09-17T04:00:14.170300+0000","last_became_peered":"2024-09-17T04:00:14.170300+0000","last_unstale":"2024-09-17T04:00:14.176467+0000","last_undegraded":"2024-09-17T04:00:14.176467+0000","last_fullsized":"2024-09-17T04:00:14.176467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:01.751763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163784+0000","last_change":"2024-09-17T04:00:14.163784+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163784+0000","last_undegraded":"2024-09-17T04:00:14.163784+0000","last_fullsized":"2024-09-17T04:00:14.163784+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176228+0000","last_change":"2024-09-17T04:00:14.172200+0000","last_active":"2024-09-17T04:00:14.176228+0000","last_peered":"2024-09-17T04:00:14.176228+0000","last_clean":"2024-09-17T04:00:14.176228+0000","last_became_active":"2024-09-17T04:00:14.171692+0000","last_became_peered":"2024-09-17T04:00:14.171692+0000","last_unstale":"2024-09-17T04:00:14.176228+0000","last_undegraded":"2024-09-17T04:00:14.176228+0000","last_fullsized":"2024-09-17T04:00:14.176228+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:51.737905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166626+0000","last_change":"2024-09-17T04:00:14.166626+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166626+0000","last_undegraded":"2024-09-17T04:00:14.166626+0000","last_fullsized":"2024-09-17T04:00:14.166626+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.178640+0000","last_change":"2024-09-17T04:00:14.175785+0000","last_active":"2024-09-17T04:00:14.178640+0000","last_peered":"2024-09-17T04:00:14.178640+0000","last_clean":"2024-09-17T04:00:14.178640+0000","last_became_active":"2024-09-17T04:00:14.175439+0000","last_became_peered":"2024-09-17T04:00:14.175439+0000","last_unstale":"2024-09-17T04:00:14.178640+0000","last_undegraded":"2024-09-17T04:00:14.178640+0000","last_fullsized":"2024-09-17T04:00:14.178640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:12:16.575657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977474+0000","last_change":"2024-09-17T04:00:15.196798+0000","last_active":"2024-09-17T04:00:17.977474+0000","last_peered":"2024-09-17T04:00:17.977474+0000","last_clean":"2024-09-17T04:00:17.977474+0000","last_became_active":"2024-09-17T04:00:15.191801+0000","last_became_peered":"2024-09-17T04:00:15.191801+0000","last_unstale":"2024-09-17T04:00:17.977474+0000","last_undegraded":"2024-09-17T04:00:17.977474+0000","last_fullsized":"2024-09-17T04:00:17.977474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:14.033347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977378+0000","last_change":"2024-09-17T04:00:14.176595+0000","last_active":"2024-09-17T04:00:17.977378+0000","last_peered":"2024-09-17T04:00:17.977378+0000","last_clean":"2024-09-17T04:00:17.977378+0000","last_became_active":"2024-09-17T04:00:14.176223+0000","last_became_peered":"2024-09-17T04:00:14.176223+0000","last_unstale":"2024-09-17T04:00:17.977378+0000","last_undegraded":"2024-09-17T04:00:17.977378+0000","last_fullsized":"2024-09-17T04:00:17.977378+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:48.426501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167405+0000","last_change":"2024-09-17T04:00:14.167405+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167405+0000","last_undegraded":"2024-09-17T04:00:14.167405+0000","last_fullsized":"2024-09-17T04:00:14.167405+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186198+0000","last_change":"2024-09-17T04:00:14.185208+0000","last_active":"2024-09-17T04:00:14.186198+0000","last_peered":"2024-09-17T04:00:14.186198+0000","last_clean":"2024-09-17T04:00:14.186198+0000","last_became_active":"2024-09-17T04:00:14.184192+0000","last_became_peered":"2024-09-17T04:00:14.184192+0000","last_unstale":"2024-09-17T04:00:14.186198+0000","last_undegraded":"2024-09-17T04:00:14.186198+0000","last_fullsized":"2024-09-17T04:00:14.186198+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:59.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185221+0000","last_change":"2024-09-17T04:00:15.200216+0000","last_active":"2024-09-17T04:00:16.185221+0000","last_peered":"2024-09-17T04:00:16.185221+0000","last_clean":"2024-09-17T04:00:16.185221+0000","last_became_active":"2024-09-17T04:00:15.199102+0000","last_became_peered":"2024-09-17T04:00:15.199102+0000","last_unstale":"2024-09-17T04:00:16.185221+0000","last_undegraded":"2024-09-17T04:00:16.185221+0000","last_fullsized":"2024-09-17T04:00:16.185221+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:46:45.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183892+0000","last_change":"2024-09-17T04:00:14.179639+0000","last_active":"2024-09-17T04:00:14.183892+0000","last_peered":"2024-09-17T04:00:14.183892+0000","last_clean":"2024-09-17T04:00:14.183892+0000","last_became_active":"2024-09-17T04:00:14.179337+0000","last_became_peered":"2024-09-17T04:00:14.179337+0000","last_unstale":"2024-09-17T04:00:14.183892+0000","last_undegraded":"2024-09-17T04:00:14.183892+0000","last_fullsized":"2024-09-17T04:00:14.183892+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:14.364680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":369,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.168937+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:14.168937+0000","last_peered":"2024-09-17T04:00:14.168937+0000","last_clean":"2024-09-17T04:00:14.168937+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:14.168937+0000","last_undegraded":"2024-09-17T04:00:14.168937+0000","last_fullsized":"2024-09-17T04:00:14.168937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37888,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333952,"statfs":{"total":96636764160,"available":96597966848,"internally_reserved":0,"allocated":2600960,"data_stored":1465344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37308,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334532,"statfs":{"total":96636764160,"available":96598560768,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542277,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41492,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330348,"statfs":{"total":96636764160,"available":96594276352,"internally_reserved":0,"allocated":2031616,"data_stored":888026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42092,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329748,"statfs":{"total":96636764160,"available":96593661952,"internally_reserved":0,"allocated":2641920,"data_stored":1484716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:19.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.869+0000 7f92ec400640 1 -- 172.21.15.129:0/1248229126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 msgr2=0x7f92cc004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.869+0000 7f92ec400640 1 --2- 172.21.15.129:0/1248229126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92cc004060 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f92d8009b50 tx=0x7f92cc005830 comp rx=0 tx=0).stop 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.869+0000 7f92ea400640 1 -- 172.21.15.129:0/1248229126 reap_dead start 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.869+0000 7f92ec400640 1 -- 172.21.15.129:0/1248229126 shutdown_connections 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.869+0000 7f92ec400640 1 -- 172.21.15.129:0/1248229126 >> 172.21.15.129:0/1248229126 conn(0x7f92e406d410 msgr2=0x7f92e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.870+0000 7f92ec400640 1 -- 172.21.15.129:0/1248229126 shutdown_connections 2024-09-17T04:00:19.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.870+0000 7f92ec400640 1 -- 172.21.15.129:0/1248229126 wait complete. 2024-09-17T04:00:19.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.872+0000 7f92ec400640 1 Processor -- start 2024-09-17T04:00:19.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.872+0000 7f92ec400640 1 -- start start 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92e409df30 0x7f92e41f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92e415e2e0 0x7f92e41f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92e40cbd70 con 0x7f92cc003c10 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92e40cbbf0 con 0x7f92e415e2e0 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.873+0000 7f92ec400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f92e40cbef0 con 0x7f92e409df30 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35442/0 (socket says 172.21.15.129:35442) 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 -- 172.21.15.129:0/1315969787 learned_addr learned my addr 172.21.15.129:0/1315969787 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:19.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92ea400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92e415e2e0 0x7f92e41f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9000640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92e409df30 0x7f92e41f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 -- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92e409df30 msgr2=0x7f92e41f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92e409df30 0x7f92e41f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 -- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92e415e2e0 msgr2=0x7f92e41f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92e415e2e0 0x7f92e41f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 -- 172.21.15.129:0/1315969787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e415ed30 con 0x7f92cc003c10 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.874+0000 7f92e9a00640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f92d8036040 tx=0x7f92d8033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.875+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8038040 con 0x7f92cc003c10 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.875+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92d8004020 con 0x7f92cc003c10 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.875+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e409a270 con 0x7f92cc003c10 2024-09-17T04:00:19.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.875+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d8004470 con 0x7f92cc003c10 2024-09-17T04:00:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.875+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e41f5280 con 0x7f92cc003c10 2024-09-17T04:00:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.876+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e406a140 con 0x7f92cc003c10 2024-09-17T04:00:19.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.877+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f92d8045020 con 0x7f92cc003c10 2024-09-17T04:00:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.878+0000 7f92e1600640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 0x7f92c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.878+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f92d80bb840 con 0x7f92cc003c10 2024-09-17T04:00:19.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.879+0000 7f92e9000640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 0x7f92c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:19.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.879+0000 7f92e9000640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 0x7f92c407b0d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f92d0001240 tx=0x7f92d0006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:19.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:19.882+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92d8083590 con 0x7f92cc003c10 2024-09-17T04:00:20.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.061+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f92e415ea50 con 0x7f92c4078bc0 2024-09-17T04:00:20.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.066+0000 7f92e1600640 1 -- 172.21.15.129:0/1315969787 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248843 (secure 0 0 0) 0x7f92e415ea50 con 0x7f92c4078bc0 2024-09-17T04:00:20.069 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:20.071 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:20.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.070+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 msgr2=0x7f92c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.070+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 0x7f92c407b0d0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f92d0001240 tx=0x7f92d0006040 comp rx=0 tx=0).stop 2024-09-17T04:00:20.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.070+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 msgr2=0x7f92e4158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.070+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f92d8036040 tx=0x7f92d8033670 comp rx=0 tx=0).stop 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 shutdown_connections 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92c4078bc0 0x7f92c407b0d0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92e415e2e0 0x7f92e41f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92e409df30 0x7f92e41f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 --2- 172.21.15.129:0/1315969787 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92cc003c10 0x7f92e4158770 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 >> 172.21.15.129:0/1315969787 conn(0x7f92e406d410 msgr2=0x7f92e40ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 shutdown_connections 2024-09-17T04:00:20.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.071+0000 7f92ec400640 1 -- 172.21.15.129:0/1315969787 wait complete. 2024-09-17T04:00:20.094 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-17T04:00:18.476783+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":105,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":56,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324652,"kb_used_data":16904,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754650068,"statfs":{"total":773094113280,"available":772761669632,"internally_reserved":0,"allocated":17309696,"data_stored":8212966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.325327"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167060+0000","last_change":"2024-09-17T04:00:14.167060+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167060+0000","last_undegraded":"2024-09-17T04:00:14.167060+0000","last_fullsized":"2024-09-17T04:00:14.167060+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977616+0000","last_change":"2024-09-17T04:00:15.198110+0000","last_active":"2024-09-17T04:00:17.977616+0000","last_peered":"2024-09-17T04:00:17.977616+0000","last_clean":"2024-09-17T04:00:17.977616+0000","last_became_active":"2024-09-17T04:00:15.197746+0000","last_became_peered":"2024-09-17T04:00:15.197746+0000","last_unstale":"2024-09-17T04:00:17.977616+0000","last_undegraded":"2024-09-17T04:00:17.977616+0000","last_fullsized":"2024-09-17T04:00:17.977616+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.169862+0000","last_change":"2024-09-17T04:00:14.169862+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.169862+0000","last_undegraded":"2024-09-17T04:00:14.169862+0000","last_fullsized":"2024-09-17T04:00:14.169862+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977850+0000","last_change":"2024-09-17T04:00:15.198044+0000","last_active":"2024-09-17T04:00:17.977850+0000","last_peered":"2024-09-17T04:00:17.977850+0000","last_clean":"2024-09-17T04:00:17.977850+0000","last_became_active":"2024-09-17T04:00:15.197647+0000","last_became_peered":"2024-09-17T04:00:15.197647+0000","last_unstale":"2024-09-17T04:00:17.977850+0000","last_undegraded":"2024-09-17T04:00:17.977850+0000","last_fullsized":"2024-09-17T04:00:17.977850+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.114514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.173655+0000","last_change":"2024-09-17T04:00:14.173655+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.173655+0000","last_undegraded":"2024-09-17T04:00:14.173655+0000","last_fullsized":"2024-09-17T04:00:14.173655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163297+0000","last_change":"2024-09-17T04:00:14.163297+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163297+0000","last_undegraded":"2024-09-17T04:00:14.163297+0000","last_fullsized":"2024-09-17T04:00:14.163297+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158538+0000","last_change":"2024-09-17T04:00:14.158538+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158538+0000","last_undegraded":"2024-09-17T04:00:14.158538+0000","last_fullsized":"2024-09-17T04:00:14.158538+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163875+0000","last_change":"2024-09-17T04:00:14.163875+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163875+0000","last_undegraded":"2024-09-17T04:00:14.163875+0000","last_fullsized":"2024-09-17T04:00:14.163875+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161997+0000","last_change":"2024-09-17T04:00:14.161997+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161997+0000","last_undegraded":"2024-09-17T04:00:14.161997+0000","last_fullsized":"2024-09-17T04:00:14.161997+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185181+0000","last_change":"2024-09-17T04:00:15.187116+0000","last_active":"2024-09-17T04:00:16.185181+0000","last_peered":"2024-09-17T04:00:16.185181+0000","last_clean":"2024-09-17T04:00:16.185181+0000","last_became_active":"2024-09-17T04:00:15.186491+0000","last_became_peered":"2024-09-17T04:00:15.186491+0000","last_unstale":"2024-09-17T04:00:16.185181+0000","last_undegraded":"2024-09-17T04:00:16.185181+0000","last_fullsized":"2024-09-17T04:00:16.185181+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:20.978854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.159485+0000","last_change":"2024-09-17T04:00:14.159485+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.159485+0000","last_undegraded":"2024-09-17T04:00:14.159485+0000","last_fullsized":"2024-09-17T04:00:14.159485+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977913+0000","last_change":"2024-09-17T04:00:15.202361+0000","last_active":"2024-09-17T04:00:17.977913+0000","last_peered":"2024-09-17T04:00:17.977913+0000","last_clean":"2024-09-17T04:00:17.977913+0000","last_became_active":"2024-09-17T04:00:15.202042+0000","last_became_peered":"2024-09-17T04:00:15.202042+0000","last_unstale":"2024-09-17T04:00:17.977913+0000","last_undegraded":"2024-09-17T04:00:17.977913+0000","last_fullsized":"2024-09-17T04:00:17.977913+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:31.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977369+0000","last_change":"2024-09-17T04:00:15.205859+0000","last_active":"2024-09-17T04:00:17.977369+0000","last_peered":"2024-09-17T04:00:17.977369+0000","last_clean":"2024-09-17T04:00:17.977369+0000","last_became_active":"2024-09-17T04:00:15.205527+0000","last_became_peered":"2024-09-17T04:00:15.205527+0000","last_unstale":"2024-09-17T04:00:17.977369+0000","last_undegraded":"2024-09-17T04:00:17.977369+0000","last_fullsized":"2024-09-17T04:00:17.977369+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:02:13.228284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160861+0000","last_change":"2024-09-17T04:00:14.160861+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160861+0000","last_undegraded":"2024-09-17T04:00:14.160861+0000","last_fullsized":"2024-09-17T04:00:14.160861+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977556+0000","last_change":"2024-09-17T04:00:15.195580+0000","last_active":"2024-09-17T04:00:17.977556+0000","last_peered":"2024-09-17T04:00:17.977556+0000","last_clean":"2024-09-17T04:00:17.977556+0000","last_became_active":"2024-09-17T04:00:15.194025+0000","last_became_peered":"2024-09-17T04:00:15.194025+0000","last_unstale":"2024-09-17T04:00:17.977556+0000","last_undegraded":"2024-09-17T04:00:17.977556+0000","last_fullsized":"2024-09-17T04:00:17.977556+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:34.997014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163519+0000","last_change":"2024-09-17T04:00:14.163519+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163519+0000","last_undegraded":"2024-09-17T04:00:14.163519+0000","last_fullsized":"2024-09-17T04:00:14.163519+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.170293+0000","last_change":"2024-09-17T04:00:14.170293+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.170293+0000","last_undegraded":"2024-09-17T04:00:14.170293+0000","last_fullsized":"2024-09-17T04:00:14.170293+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166156+0000","last_change":"2024-09-17T04:00:14.166156+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166156+0000","last_undegraded":"2024-09-17T04:00:14.166156+0000","last_fullsized":"2024-09-17T04:00:14.166156+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184909+0000","last_change":"2024-09-17T04:00:15.185185+0000","last_active":"2024-09-17T04:00:16.184909+0000","last_peered":"2024-09-17T04:00:16.184909+0000","last_clean":"2024-09-17T04:00:16.184909+0000","last_became_active":"2024-09-17T04:00:15.182697+0000","last_became_peered":"2024-09-17T04:00:15.182697+0000","last_unstale":"2024-09-17T04:00:16.184909+0000","last_undegraded":"2024-09-17T04:00:16.184909+0000","last_fullsized":"2024-09-17T04:00:16.184909+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:06.451276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171386+0000","last_change":"2024-09-17T04:00:14.171386+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171386+0000","last_undegraded":"2024-09-17T04:00:14.171386+0000","last_fullsized":"2024-09-17T04:00:14.171386+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171738+0000","last_change":"2024-09-17T04:00:14.171738+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171738+0000","last_undegraded":"2024-09-17T04:00:14.171738+0000","last_fullsized":"2024-09-17T04:00:14.171738+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184888+0000","last_change":"2024-09-17T04:00:15.187015+0000","last_active":"2024-09-17T04:00:16.184888+0000","last_peered":"2024-09-17T04:00:16.184888+0000","last_clean":"2024-09-17T04:00:16.184888+0000","last_became_active":"2024-09-17T04:00:15.186574+0000","last_became_peered":"2024-09-17T04:00:15.186574+0000","last_unstale":"2024-09-17T04:00:16.184888+0000","last_undegraded":"2024-09-17T04:00:16.184888+0000","last_fullsized":"2024-09-17T04:00:16.184888+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:05.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167448+0000","last_change":"2024-09-17T04:00:14.167448+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167448+0000","last_undegraded":"2024-09-17T04:00:14.167448+0000","last_fullsized":"2024-09-17T04:00:14.167448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161007+0000","last_change":"2024-09-17T04:00:14.161007+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161007+0000","last_undegraded":"2024-09-17T04:00:14.161007+0000","last_fullsized":"2024-09-17T04:00:14.161007+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977962+0000","last_change":"2024-09-17T04:00:15.196465+0000","last_active":"2024-09-17T04:00:17.977962+0000","last_peered":"2024-09-17T04:00:17.977962+0000","last_clean":"2024-09-17T04:00:17.977962+0000","last_became_active":"2024-09-17T04:00:15.195761+0000","last_became_peered":"2024-09-17T04:00:15.195761+0000","last_unstale":"2024-09-17T04:00:17.977962+0000","last_undegraded":"2024-09-17T04:00:17.977962+0000","last_fullsized":"2024-09-17T04:00:17.977962+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:29.557541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157433+0000","last_change":"2024-09-17T04:00:14.157433+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157433+0000","last_undegraded":"2024-09-17T04:00:14.157433+0000","last_fullsized":"2024-09-17T04:00:14.157433+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.169429+0000","last_change":"2024-09-17T04:00:14.169429+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.169429+0000","last_undegraded":"2024-09-17T04:00:14.169429+0000","last_fullsized":"2024-09-17T04:00:14.169429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.170364+0000","last_change":"2024-09-17T04:00:14.170364+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.170364+0000","last_undegraded":"2024-09-17T04:00:14.170364+0000","last_fullsized":"2024-09-17T04:00:14.170364+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977568+0000","last_change":"2024-09-17T04:00:15.203508+0000","last_active":"2024-09-17T04:00:17.977568+0000","last_peered":"2024-09-17T04:00:17.977568+0000","last_clean":"2024-09-17T04:00:17.977568+0000","last_became_active":"2024-09-17T04:00:15.203187+0000","last_became_peered":"2024-09-17T04:00:15.203187+0000","last_unstale":"2024-09-17T04:00:17.977568+0000","last_undegraded":"2024-09-17T04:00:17.977568+0000","last_fullsized":"2024-09-17T04:00:17.977568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:55.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160314+0000","last_change":"2024-09-17T04:00:14.160314+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160314+0000","last_undegraded":"2024-09-17T04:00:14.160314+0000","last_fullsized":"2024-09-17T04:00:14.160314+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166972+0000","last_change":"2024-09-17T04:00:14.166972+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166972+0000","last_undegraded":"2024-09-17T04:00:14.166972+0000","last_fullsized":"2024-09-17T04:00:14.166972+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.171353+0000","last_change":"2024-09-17T04:00:14.171353+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.171353+0000","last_undegraded":"2024-09-17T04:00:14.171353+0000","last_fullsized":"2024-09-17T04:00:14.171353+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161968+0000","last_change":"2024-09-17T04:00:14.161968+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161968+0000","last_undegraded":"2024-09-17T04:00:14.161968+0000","last_fullsized":"2024-09-17T04:00:14.161968+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186077+0000","last_change":"2024-09-17T04:00:14.184544+0000","last_active":"2024-09-17T04:00:14.186077+0000","last_peered":"2024-09-17T04:00:14.186077+0000","last_clean":"2024-09-17T04:00:14.186077+0000","last_became_active":"2024-09-17T04:00:14.184254+0000","last_became_peered":"2024-09-17T04:00:14.184254+0000","last_unstale":"2024-09-17T04:00:14.186077+0000","last_undegraded":"2024-09-17T04:00:14.186077+0000","last_fullsized":"2024-09-17T04:00:14.186077+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:05.130672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186087+0000","last_change":"2024-09-17T04:00:15.203100+0000","last_active":"2024-09-17T04:00:16.186087+0000","last_peered":"2024-09-17T04:00:16.186087+0000","last_clean":"2024-09-17T04:00:16.186087+0000","last_became_active":"2024-09-17T04:00:15.202011+0000","last_became_peered":"2024-09-17T04:00:15.202011+0000","last_unstale":"2024-09-17T04:00:16.186087+0000","last_undegraded":"2024-09-17T04:00:16.186087+0000","last_fullsized":"2024-09-17T04:00:16.186087+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:07.578261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.184031+0000","last_change":"2024-09-17T04:00:14.178919+0000","last_active":"2024-09-17T04:00:14.184031+0000","last_peered":"2024-09-17T04:00:14.184031+0000","last_clean":"2024-09-17T04:00:14.184031+0000","last_became_active":"2024-09-17T04:00:14.177710+0000","last_became_peered":"2024-09-17T04:00:14.177710+0000","last_unstale":"2024-09-17T04:00:14.184031+0000","last_undegraded":"2024-09-17T04:00:14.184031+0000","last_fullsized":"2024-09-17T04:00:14.184031+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:27.461533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.159468+0000","last_change":"2024-09-17T04:00:14.159468+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.159468+0000","last_undegraded":"2024-09-17T04:00:14.159468+0000","last_fullsized":"2024-09-17T04:00:14.159468+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.182124+0000","last_change":"2024-09-17T04:00:14.173566+0000","last_active":"2024-09-17T04:00:14.182124+0000","last_peered":"2024-09-17T04:00:14.182124+0000","last_clean":"2024-09-17T04:00:14.182124+0000","last_became_active":"2024-09-17T04:00:14.173201+0000","last_became_peered":"2024-09-17T04:00:14.173201+0000","last_unstale":"2024-09-17T04:00:14.182124+0000","last_undegraded":"2024-09-17T04:00:14.182124+0000","last_fullsized":"2024-09-17T04:00:14.182124+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:45:33.665800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.160725+0000","last_change":"2024-09-17T04:00:14.160725+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.160725+0000","last_undegraded":"2024-09-17T04:00:14.160725+0000","last_fullsized":"2024-09-17T04:00:14.160725+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977943+0000","last_change":"2024-09-17T04:00:14.180751+0000","last_active":"2024-09-17T04:00:17.977943+0000","last_peered":"2024-09-17T04:00:17.977943+0000","last_clean":"2024-09-17T04:00:17.977943+0000","last_became_active":"2024-09-17T04:00:14.180400+0000","last_became_peered":"2024-09-17T04:00:14.180400+0000","last_unstale":"2024-09-17T04:00:17.977943+0000","last_undegraded":"2024-09-17T04:00:17.977943+0000","last_fullsized":"2024-09-17T04:00:17.977943+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:14.945761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185444+0000","last_change":"2024-09-17T04:00:15.183077+0000","last_active":"2024-09-17T04:00:16.185444+0000","last_peered":"2024-09-17T04:00:16.185444+0000","last_clean":"2024-09-17T04:00:16.185444+0000","last_became_active":"2024-09-17T04:00:15.182579+0000","last_became_peered":"2024-09-17T04:00:15.182579+0000","last_unstale":"2024-09-17T04:00:16.185444+0000","last_undegraded":"2024-09-17T04:00:16.185444+0000","last_fullsized":"2024-09-17T04:00:16.185444+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:32.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185456+0000","last_change":"2024-09-17T04:00:14.178007+0000","last_active":"2024-09-17T04:00:14.185456+0000","last_peered":"2024-09-17T04:00:14.185456+0000","last_clean":"2024-09-17T04:00:14.185456+0000","last_became_active":"2024-09-17T04:00:14.177397+0000","last_became_peered":"2024-09-17T04:00:14.177397+0000","last_unstale":"2024-09-17T04:00:14.185456+0000","last_undegraded":"2024-09-17T04:00:14.185456+0000","last_fullsized":"2024-09-17T04:00:14.185456+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:41.553403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161193+0000","last_change":"2024-09-17T04:00:14.161193+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161193+0000","last_undegraded":"2024-09-17T04:00:14.161193+0000","last_fullsized":"2024-09-17T04:00:14.161193+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185788+0000","last_change":"2024-09-17T04:00:14.181701+0000","last_active":"2024-09-17T04:00:14.185788+0000","last_peered":"2024-09-17T04:00:14.185788+0000","last_clean":"2024-09-17T04:00:14.185788+0000","last_became_active":"2024-09-17T04:00:14.180617+0000","last_became_peered":"2024-09-17T04:00:14.180617+0000","last_unstale":"2024-09-17T04:00:14.185788+0000","last_undegraded":"2024-09-17T04:00:14.185788+0000","last_fullsized":"2024-09-17T04:00:14.185788+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:31.039000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157272+0000","last_change":"2024-09-17T04:00:14.157272+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157272+0000","last_undegraded":"2024-09-17T04:00:14.157272+0000","last_fullsized":"2024-09-17T04:00:14.157272+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.184030+0000","last_change":"2024-09-17T04:00:14.180760+0000","last_active":"2024-09-17T04:00:14.184030+0000","last_peered":"2024-09-17T04:00:14.184030+0000","last_clean":"2024-09-17T04:00:14.184030+0000","last_became_active":"2024-09-17T04:00:14.180063+0000","last_became_peered":"2024-09-17T04:00:14.180063+0000","last_unstale":"2024-09-17T04:00:14.184030+0000","last_undegraded":"2024-09-17T04:00:14.184030+0000","last_fullsized":"2024-09-17T04:00:14.184030+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:01.751763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166153+0000","last_change":"2024-09-17T04:00:14.166153+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166153+0000","last_undegraded":"2024-09-17T04:00:14.166153+0000","last_fullsized":"2024-09-17T04:00:14.166153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.178035+0000","last_change":"2024-09-17T04:00:14.170243+0000","last_active":"2024-09-17T04:00:14.178035+0000","last_peered":"2024-09-17T04:00:14.178035+0000","last_clean":"2024-09-17T04:00:14.178035+0000","last_became_active":"2024-09-17T04:00:14.169878+0000","last_became_peered":"2024-09-17T04:00:14.169878+0000","last_unstale":"2024-09-17T04:00:14.178035+0000","last_undegraded":"2024-09-17T04:00:14.178035+0000","last_fullsized":"2024-09-17T04:00:14.178035+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:08:00.433111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162135+0000","last_change":"2024-09-17T04:00:14.162135+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162135+0000","last_undegraded":"2024-09-17T04:00:14.162135+0000","last_fullsized":"2024-09-17T04:00:14.162135+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181836+0000","last_change":"2024-09-17T04:00:14.178548+0000","last_active":"2024-09-17T04:00:14.181836+0000","last_peered":"2024-09-17T04:00:14.181836+0000","last_clean":"2024-09-17T04:00:14.181836+0000","last_became_active":"2024-09-17T04:00:14.178173+0000","last_became_peered":"2024-09-17T04:00:14.178173+0000","last_unstale":"2024-09-17T04:00:14.181836+0000","last_undegraded":"2024-09-17T04:00:14.181836+0000","last_fullsized":"2024-09-17T04:00:14.181836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:01:42.480246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157748+0000","last_change":"2024-09-17T04:00:14.157748+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157748+0000","last_undegraded":"2024-09-17T04:00:14.157748+0000","last_fullsized":"2024-09-17T04:00:14.157748+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186156+0000","last_change":"2024-09-17T04:00:14.184902+0000","last_active":"2024-09-17T04:00:14.186156+0000","last_peered":"2024-09-17T04:00:14.186156+0000","last_clean":"2024-09-17T04:00:14.186156+0000","last_became_active":"2024-09-17T04:00:14.184624+0000","last_became_peered":"2024-09-17T04:00:14.184624+0000","last_unstale":"2024-09-17T04:00:14.186156+0000","last_undegraded":"2024-09-17T04:00:14.186156+0000","last_fullsized":"2024-09-17T04:00:14.186156+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:59.180634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185090+0000","last_change":"2024-09-17T04:00:15.180729+0000","last_active":"2024-09-17T04:00:16.185090+0000","last_peered":"2024-09-17T04:00:16.185090+0000","last_clean":"2024-09-17T04:00:16.185090+0000","last_became_active":"2024-09-17T04:00:15.180358+0000","last_became_peered":"2024-09-17T04:00:15.180358+0000","last_unstale":"2024-09-17T04:00:16.185090+0000","last_undegraded":"2024-09-17T04:00:16.185090+0000","last_fullsized":"2024-09-17T04:00:16.185090+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:55.762158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181150+0000","last_change":"2024-09-17T04:00:14.177482+0000","last_active":"2024-09-17T04:00:14.181150+0000","last_peered":"2024-09-17T04:00:14.181150+0000","last_clean":"2024-09-17T04:00:14.181150+0000","last_became_active":"2024-09-17T04:00:14.177185+0000","last_became_peered":"2024-09-17T04:00:14.177185+0000","last_unstale":"2024-09-17T04:00:14.181150+0000","last_undegraded":"2024-09-17T04:00:14.181150+0000","last_fullsized":"2024-09-17T04:00:14.181150+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:32.018406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162766+0000","last_change":"2024-09-17T04:00:14.162766+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162766+0000","last_undegraded":"2024-09-17T04:00:14.162766+0000","last_fullsized":"2024-09-17T04:00:14.162766+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183960+0000","last_change":"2024-09-17T04:00:14.180486+0000","last_active":"2024-09-17T04:00:14.183960+0000","last_peered":"2024-09-17T04:00:14.183960+0000","last_clean":"2024-09-17T04:00:14.183960+0000","last_became_active":"2024-09-17T04:00:14.180077+0000","last_became_peered":"2024-09-17T04:00:14.180077+0000","last_unstale":"2024-09-17T04:00:14.183960+0000","last_undegraded":"2024-09-17T04:00:14.183960+0000","last_fullsized":"2024-09-17T04:00:14.183960+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:25.031267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158293+0000","last_change":"2024-09-17T04:00:14.158293+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158293+0000","last_undegraded":"2024-09-17T04:00:14.158293+0000","last_fullsized":"2024-09-17T04:00:14.158293+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185646+0000","last_change":"2024-09-17T04:00:14.179129+0000","last_active":"2024-09-17T04:00:16.185646+0000","last_peered":"2024-09-17T04:00:16.185646+0000","last_clean":"2024-09-17T04:00:16.185646+0000","last_became_active":"2024-09-17T04:00:14.177651+0000","last_became_peered":"2024-09-17T04:00:14.177651+0000","last_unstale":"2024-09-17T04:00:16.185646+0000","last_undegraded":"2024-09-17T04:00:16.185646+0000","last_fullsized":"2024-09-17T04:00:16.185646+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:50.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158568+0000","last_change":"2024-09-17T04:00:14.158568+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158568+0000","last_undegraded":"2024-09-17T04:00:14.158568+0000","last_fullsized":"2024-09-17T04:00:14.158568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.181930+0000","last_change":"2024-09-17T04:00:14.178903+0000","last_active":"2024-09-17T04:00:14.181930+0000","last_peered":"2024-09-17T04:00:14.181930+0000","last_clean":"2024-09-17T04:00:14.181930+0000","last_became_active":"2024-09-17T04:00:14.178605+0000","last_became_peered":"2024-09-17T04:00:14.178605+0000","last_unstale":"2024-09-17T04:00:14.181930+0000","last_undegraded":"2024-09-17T04:00:14.181930+0000","last_fullsized":"2024-09-17T04:00:14.181930+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:34.211612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184829+0000","last_change":"2024-09-17T04:00:15.192316+0000","last_active":"2024-09-17T04:00:16.184829+0000","last_peered":"2024-09-17T04:00:16.184829+0000","last_clean":"2024-09-17T04:00:16.184829+0000","last_became_active":"2024-09-17T04:00:15.188357+0000","last_became_peered":"2024-09-17T04:00:15.188357+0000","last_unstale":"2024-09-17T04:00:16.184829+0000","last_undegraded":"2024-09-17T04:00:16.184829+0000","last_fullsized":"2024-09-17T04:00:16.184829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:34.996367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185523+0000","last_change":"2024-09-17T04:00:14.182987+0000","last_active":"2024-09-17T04:00:14.185523+0000","last_peered":"2024-09-17T04:00:14.185523+0000","last_clean":"2024-09-17T04:00:14.185523+0000","last_became_active":"2024-09-17T04:00:14.182665+0000","last_became_peered":"2024-09-17T04:00:14.182665+0000","last_unstale":"2024-09-17T04:00:14.185523+0000","last_undegraded":"2024-09-17T04:00:14.185523+0000","last_fullsized":"2024-09-17T04:00:14.185523+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:22.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185423+0000","last_change":"2024-09-17T04:00:15.188466+0000","last_active":"2024-09-17T04:00:16.185423+0000","last_peered":"2024-09-17T04:00:16.185423+0000","last_clean":"2024-09-17T04:00:16.185423+0000","last_became_active":"2024-09-17T04:00:15.187860+0000","last_became_peered":"2024-09-17T04:00:15.187860+0000","last_unstale":"2024-09-17T04:00:16.185423+0000","last_undegraded":"2024-09-17T04:00:16.185423+0000","last_fullsized":"2024-09-17T04:00:16.185423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:05.989755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183887+0000","last_change":"2024-09-17T04:00:14.179849+0000","last_active":"2024-09-17T04:00:14.183887+0000","last_peered":"2024-09-17T04:00:14.183887+0000","last_clean":"2024-09-17T04:00:14.183887+0000","last_became_active":"2024-09-17T04:00:14.178084+0000","last_became_peered":"2024-09-17T04:00:14.178084+0000","last_unstale":"2024-09-17T04:00:14.183887+0000","last_undegraded":"2024-09-17T04:00:14.183887+0000","last_fullsized":"2024-09-17T04:00:14.183887+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:16.538235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.161180+0000","last_change":"2024-09-17T04:00:14.161180+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.161180+0000","last_undegraded":"2024-09-17T04:00:14.161180+0000","last_fullsized":"2024-09-17T04:00:14.161180+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.179551+0000","last_change":"2024-09-17T04:00:14.173867+0000","last_active":"2024-09-17T04:00:14.179551+0000","last_peered":"2024-09-17T04:00:14.179551+0000","last_clean":"2024-09-17T04:00:14.179551+0000","last_became_active":"2024-09-17T04:00:14.171614+0000","last_became_peered":"2024-09-17T04:00:14.171614+0000","last_unstale":"2024-09-17T04:00:14.179551+0000","last_undegraded":"2024-09-17T04:00:14.179551+0000","last_fullsized":"2024-09-17T04:00:14.179551+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:22.841142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.158480+0000","last_change":"2024-09-17T04:00:14.158480+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.158480+0000","last_undegraded":"2024-09-17T04:00:14.158480+0000","last_fullsized":"2024-09-17T04:00:14.158480+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185558+0000","last_change":"2024-09-17T04:00:14.183120+0000","last_active":"2024-09-17T04:00:14.185558+0000","last_peered":"2024-09-17T04:00:14.185558+0000","last_clean":"2024-09-17T04:00:14.185558+0000","last_became_active":"2024-09-17T04:00:14.182779+0000","last_became_peered":"2024-09-17T04:00:14.182779+0000","last_unstale":"2024-09-17T04:00:14.185558+0000","last_undegraded":"2024-09-17T04:00:14.185558+0000","last_fullsized":"2024-09-17T04:00:14.185558+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:34.211612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977807+0000","last_change":"2024-09-17T04:00:15.204420+0000","last_active":"2024-09-17T04:00:17.977807+0000","last_peered":"2024-09-17T04:00:17.977807+0000","last_clean":"2024-09-17T04:00:17.977807+0000","last_became_active":"2024-09-17T04:00:15.202153+0000","last_became_peered":"2024-09-17T04:00:15.202153+0000","last_unstale":"2024-09-17T04:00:17.977807+0000","last_undegraded":"2024-09-17T04:00:17.977807+0000","last_fullsized":"2024-09-17T04:00:17.977807+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:57.466014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.168322+0000","last_change":"2024-09-17T04:00:14.164836+0000","last_active":"2024-09-17T04:00:14.168322+0000","last_peered":"2024-09-17T04:00:14.168322+0000","last_clean":"2024-09-17T04:00:14.168322+0000","last_became_active":"2024-09-17T04:00:14.164491+0000","last_became_peered":"2024-09-17T04:00:14.164491+0000","last_unstale":"2024-09-17T04:00:14.168322+0000","last_undegraded":"2024-09-17T04:00:14.168322+0000","last_fullsized":"2024-09-17T04:00:14.168322+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:40.826089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185709+0000","last_change":"2024-09-17T04:00:15.203883+0000","last_active":"2024-09-17T04:00:16.185709+0000","last_peered":"2024-09-17T04:00:16.185709+0000","last_clean":"2024-09-17T04:00:16.185709+0000","last_became_active":"2024-09-17T04:00:15.201414+0000","last_became_peered":"2024-09-17T04:00:15.201414+0000","last_unstale":"2024-09-17T04:00:16.185709+0000","last_undegraded":"2024-09-17T04:00:16.185709+0000","last_fullsized":"2024-09-17T04:00:16.185709+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:29.778689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.187098+0000","last_change":"2024-09-17T04:00:14.185714+0000","last_active":"2024-09-17T04:00:14.187098+0000","last_peered":"2024-09-17T04:00:14.187098+0000","last_clean":"2024-09-17T04:00:14.187098+0000","last_became_active":"2024-09-17T04:00:14.185363+0000","last_became_peered":"2024-09-17T04:00:14.185363+0000","last_unstale":"2024-09-17T04:00:14.187098+0000","last_undegraded":"2024-09-17T04:00:14.187098+0000","last_fullsized":"2024-09-17T04:00:14.187098+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:35:19.679615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.157262+0000","last_change":"2024-09-17T04:00:14.157262+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.157262+0000","last_undegraded":"2024-09-17T04:00:14.157262+0000","last_fullsized":"2024-09-17T04:00:14.157262+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.185905+0000","last_change":"2024-09-17T04:00:14.183023+0000","last_active":"2024-09-17T04:00:14.185905+0000","last_peered":"2024-09-17T04:00:14.185905+0000","last_clean":"2024-09-17T04:00:14.185905+0000","last_became_active":"2024-09-17T04:00:14.178678+0000","last_became_peered":"2024-09-17T04:00:14.178678+0000","last_unstale":"2024-09-17T04:00:14.185905+0000","last_undegraded":"2024-09-17T04:00:14.185905+0000","last_fullsized":"2024-09-17T04:00:14.185905+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:51.928961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977340+0000","last_change":"2024-09-17T04:00:15.202518+0000","last_active":"2024-09-17T04:00:17.977340+0000","last_peered":"2024-09-17T04:00:17.977340+0000","last_clean":"2024-09-17T04:00:17.977340+0000","last_became_active":"2024-09-17T04:00:15.202240+0000","last_became_peered":"2024-09-17T04:00:15.202240+0000","last_unstale":"2024-09-17T04:00:17.977340+0000","last_undegraded":"2024-09-17T04:00:17.977340+0000","last_fullsized":"2024-09-17T04:00:17.977340+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:59.197974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183836+0000","last_change":"2024-09-17T04:00:14.178227+0000","last_active":"2024-09-17T04:00:14.183836+0000","last_peered":"2024-09-17T04:00:14.183836+0000","last_clean":"2024-09-17T04:00:14.183836+0000","last_became_active":"2024-09-17T04:00:14.177922+0000","last_became_peered":"2024-09-17T04:00:14.177922+0000","last_unstale":"2024-09-17T04:00:14.183836+0000","last_undegraded":"2024-09-17T04:00:14.183836+0000","last_fullsized":"2024-09-17T04:00:14.183836+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:34.330414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.168571+0000","last_change":"2024-09-17T04:00:14.168571+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.168571+0000","last_undegraded":"2024-09-17T04:00:14.168571+0000","last_fullsized":"2024-09-17T04:00:14.168571+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176884+0000","last_change":"2024-09-17T04:00:14.173438+0000","last_active":"2024-09-17T04:00:14.176884+0000","last_peered":"2024-09-17T04:00:14.176884+0000","last_clean":"2024-09-17T04:00:14.176884+0000","last_became_active":"2024-09-17T04:00:14.173090+0000","last_became_peered":"2024-09-17T04:00:14.173090+0000","last_unstale":"2024-09-17T04:00:14.176884+0000","last_undegraded":"2024-09-17T04:00:14.176884+0000","last_fullsized":"2024-09-17T04:00:14.176884+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:51.031094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.168610+0000","last_change":"2024-09-17T04:00:14.168610+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.168610+0000","last_undegraded":"2024-09-17T04:00:14.168610+0000","last_fullsized":"2024-09-17T04:00:14.168610+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.188590+0000","last_change":"2024-09-17T04:00:14.186855+0000","last_active":"2024-09-17T04:00:14.188590+0000","last_peered":"2024-09-17T04:00:14.188590+0000","last_clean":"2024-09-17T04:00:14.188590+0000","last_became_active":"2024-09-17T04:00:14.186535+0000","last_became_peered":"2024-09-17T04:00:14.186535+0000","last_unstale":"2024-09-17T04:00:14.188590+0000","last_undegraded":"2024-09-17T04:00:14.188590+0000","last_fullsized":"2024-09-17T04:00:14.188590+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:31.039000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186153+0000","last_change":"2024-09-17T04:00:15.194286+0000","last_active":"2024-09-17T04:00:16.186153+0000","last_peered":"2024-09-17T04:00:16.186153+0000","last_clean":"2024-09-17T04:00:16.186153+0000","last_became_active":"2024-09-17T04:00:15.193936+0000","last_became_peered":"2024-09-17T04:00:15.193936+0000","last_unstale":"2024-09-17T04:00:16.186153+0000","last_undegraded":"2024-09-17T04:00:16.186153+0000","last_fullsized":"2024-09-17T04:00:16.186153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:03.221443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"163'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185534+0000","last_change":"2024-09-17T04:00:14.176145+0000","last_active":"2024-09-17T04:00:16.185534+0000","last_peered":"2024-09-17T04:00:16.185534+0000","last_clean":"2024-09-17T04:00:16.185534+0000","last_became_active":"2024-09-17T04:00:14.175839+0000","last_became_peered":"2024-09-17T04:00:14.175839+0000","last_unstale":"2024-09-17T04:00:16.185534+0000","last_undegraded":"2024-09-17T04:00:16.185534+0000","last_fullsized":"2024-09-17T04:00:16.185534+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:33.949960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162812+0000","last_change":"2024-09-17T04:00:14.162812+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162812+0000","last_undegraded":"2024-09-17T04:00:14.162812+0000","last_fullsized":"2024-09-17T04:00:14.162812+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184972+0000","last_change":"2024-09-17T04:00:14.182944+0000","last_active":"2024-09-17T04:00:16.184972+0000","last_peered":"2024-09-17T04:00:16.184972+0000","last_clean":"2024-09-17T04:00:16.184972+0000","last_became_active":"2024-09-17T04:00:14.182626+0000","last_became_peered":"2024-09-17T04:00:14.182626+0000","last_unstale":"2024-09-17T04:00:16.184972+0000","last_undegraded":"2024-09-17T04:00:16.184972+0000","last_fullsized":"2024-09-17T04:00:16.184972+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:33.722613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.162970+0000","last_change":"2024-09-17T04:00:14.162970+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.162970+0000","last_undegraded":"2024-09-17T04:00:14.162970+0000","last_fullsized":"2024-09-17T04:00:14.162970+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176467+0000","last_change":"2024-09-17T04:00:14.172333+0000","last_active":"2024-09-17T04:00:14.176467+0000","last_peered":"2024-09-17T04:00:14.176467+0000","last_clean":"2024-09-17T04:00:14.176467+0000","last_became_active":"2024-09-17T04:00:14.170300+0000","last_became_peered":"2024-09-17T04:00:14.170300+0000","last_unstale":"2024-09-17T04:00:14.176467+0000","last_undegraded":"2024-09-17T04:00:14.176467+0000","last_fullsized":"2024-09-17T04:00:14.176467+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:01.751763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.163784+0000","last_change":"2024-09-17T04:00:14.163784+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.163784+0000","last_undegraded":"2024-09-17T04:00:14.163784+0000","last_fullsized":"2024-09-17T04:00:14.163784+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.176228+0000","last_change":"2024-09-17T04:00:14.172200+0000","last_active":"2024-09-17T04:00:14.176228+0000","last_peered":"2024-09-17T04:00:14.176228+0000","last_clean":"2024-09-17T04:00:14.176228+0000","last_became_active":"2024-09-17T04:00:14.171692+0000","last_became_peered":"2024-09-17T04:00:14.171692+0000","last_unstale":"2024-09-17T04:00:14.176228+0000","last_undegraded":"2024-09-17T04:00:14.176228+0000","last_fullsized":"2024-09-17T04:00:14.176228+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:51.737905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.166626+0000","last_change":"2024-09-17T04:00:14.166626+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.166626+0000","last_undegraded":"2024-09-17T04:00:14.166626+0000","last_fullsized":"2024-09-17T04:00:14.166626+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.178640+0000","last_change":"2024-09-17T04:00:14.175785+0000","last_active":"2024-09-17T04:00:14.178640+0000","last_peered":"2024-09-17T04:00:14.178640+0000","last_clean":"2024-09-17T04:00:14.178640+0000","last_became_active":"2024-09-17T04:00:14.175439+0000","last_became_peered":"2024-09-17T04:00:14.175439+0000","last_unstale":"2024-09-17T04:00:14.178640+0000","last_undegraded":"2024-09-17T04:00:14.178640+0000","last_fullsized":"2024-09-17T04:00:14.178640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:12:16.575657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977474+0000","last_change":"2024-09-17T04:00:15.196798+0000","last_active":"2024-09-17T04:00:17.977474+0000","last_peered":"2024-09-17T04:00:17.977474+0000","last_clean":"2024-09-17T04:00:17.977474+0000","last_became_active":"2024-09-17T04:00:15.191801+0000","last_became_peered":"2024-09-17T04:00:15.191801+0000","last_unstale":"2024-09-17T04:00:17.977474+0000","last_undegraded":"2024-09-17T04:00:17.977474+0000","last_fullsized":"2024-09-17T04:00:17.977474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:14.033347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977378+0000","last_change":"2024-09-17T04:00:14.176595+0000","last_active":"2024-09-17T04:00:17.977378+0000","last_peered":"2024-09-17T04:00:17.977378+0000","last_clean":"2024-09-17T04:00:17.977378+0000","last_became_active":"2024-09-17T04:00:14.176223+0000","last_became_peered":"2024-09-17T04:00:14.176223+0000","last_unstale":"2024-09-17T04:00:17.977378+0000","last_undegraded":"2024-09-17T04:00:17.977378+0000","last_fullsized":"2024-09-17T04:00:17.977378+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:48.426501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":1,"reported_epoch":161,"state":"creating+peering","last_fresh":"2024-09-17T04:00:14.167405+0000","last_change":"2024-09-17T04:00:14.167405+0000","last_active":"2024-09-17T04:00:14.149927+0000","last_peered":"2024-09-17T04:00:14.149927+0000","last_clean":"2024-09-17T04:00:14.149927+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:14.167405+0000","last_undegraded":"2024-09-17T04:00:14.167405+0000","last_fullsized":"2024-09-17T04:00:14.167405+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.186198+0000","last_change":"2024-09-17T04:00:14.185208+0000","last_active":"2024-09-17T04:00:14.186198+0000","last_peered":"2024-09-17T04:00:14.186198+0000","last_clean":"2024-09-17T04:00:14.186198+0000","last_became_active":"2024-09-17T04:00:14.184192+0000","last_became_peered":"2024-09-17T04:00:14.184192+0000","last_unstale":"2024-09-17T04:00:14.186198+0000","last_undegraded":"2024-09-17T04:00:14.186198+0000","last_fullsized":"2024-09-17T04:00:14.186198+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:26:59.808088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185221+0000","last_change":"2024-09-17T04:00:15.200216+0000","last_active":"2024-09-17T04:00:16.185221+0000","last_peered":"2024-09-17T04:00:16.185221+0000","last_clean":"2024-09-17T04:00:16.185221+0000","last_became_active":"2024-09-17T04:00:15.199102+0000","last_became_peered":"2024-09-17T04:00:15.199102+0000","last_unstale":"2024-09-17T04:00:16.185221+0000","last_undegraded":"2024-09-17T04:00:16.185221+0000","last_fullsized":"2024-09-17T04:00:16.185221+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:46:45.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.183892+0000","last_change":"2024-09-17T04:00:14.179639+0000","last_active":"2024-09-17T04:00:14.183892+0000","last_peered":"2024-09-17T04:00:14.183892+0000","last_clean":"2024-09-17T04:00:14.183892+0000","last_became_active":"2024-09-17T04:00:14.179337+0000","last_became_peered":"2024-09-17T04:00:14.179337+0000","last_unstale":"2024-09-17T04:00:14.183892+0000","last_undegraded":"2024-09-17T04:00:14.183892+0000","last_fullsized":"2024-09-17T04:00:14.183892+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:14.364680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":369,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-17T04:00:14.168937+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:14.168937+0000","last_peered":"2024-09-17T04:00:14.168937+0000","last_clean":"2024-09-17T04:00:14.168937+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:14.168937+0000","last_undegraded":"2024-09-17T04:00:14.168937+0000","last_fullsized":"2024-09-17T04:00:14.168937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509572,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37888,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333952,"statfs":{"total":96636764160,"available":96597966848,"internally_reserved":0,"allocated":2600960,"data_stored":1465344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509575,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37308,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334532,"statfs":{"total":96636764160,"available":96598560768,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41468,"kb_used_data":1960,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330372,"statfs":{"total":96636764160,"available":96594300928,"internally_reserved":0,"allocated":2007040,"data_stored":874976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542277,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41492,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330348,"statfs":{"total":96636764160,"available":96594276352,"internally_reserved":0,"allocated":2031616,"data_stored":888026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42092,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329748,"statfs":{"total":96636764160,"available":96593661952,"internally_reserved":0,"allocated":2641920,"data_stored":1484716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:20.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:20.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 -- 172.21.15.129:0/1756811329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f81524c0 msgr2=0x7fe1f80cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 --2- 172.21.15.129:0/1756811329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f81524c0 0x7fe1f80cf5c0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009830 tx=0x7fe1ec02a510 comp rx=0 tx=0).stop 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 -- 172.21.15.129:0/1756811329 shutdown_connections 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 --2- 172.21.15.129:0/1756811329 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f80cfb00 0x7fe1f815bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 --2- 172.21.15.129:0/1756811329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f81524c0 0x7fe1f80cf5c0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 -- 172.21.15.129:0/1756811329 >> 172.21.15.129:0/1756811329 conn(0x7fe1f806cfc0 msgr2=0x7fe1f806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 -- 172.21.15.129:0/1756811329 shutdown_connections 2024-09-17T04:00:20.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.357+0000 7fe1fee00640 1 -- 172.21.15.129:0/1756811329 wait complete. 2024-09-17T04:00:20.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.359+0000 7fe1fee00640 1 Processor -- start 2024-09-17T04:00:20.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.359+0000 7fe1fee00640 1 -- start start 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1f80cfb00 0x7fe1f80cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f81524c0 0x7fe1f8150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f80d02c0 con 0x7fe1f815c1e0 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f8152b10 con 0x7fe1f80cfb00 2024-09-17T04:00:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f80d0440 con 0x7fe1f81524c0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fe400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fe400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35466/0 (socket says 172.21.15.129:35466) 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fe400640 1 -- 172.21.15.129:0/4189417814 learned_addr learned my addr 172.21.15.129:0/4189417814 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fda00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1f80cfb00 0x7fe1f80cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.361+0000 7fe1fd000640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f81524c0 0x7fe1f8150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 -- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f81524c0 msgr2=0x7fe1f8150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f81524c0 0x7fe1f8150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 -- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1f80cfb00 msgr2=0x7fe1f80cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1f80cfb00 0x7fe1f80cba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 -- 172.21.15.129:0/4189417814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f8152e00 con 0x7fe1f815c1e0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fe400640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000c760 tx=0x7fe1f000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:20.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0003b70 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe1f0003d30 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00052d0 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f81530e0 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.362+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f80ca480 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.363+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1f80d0910 con 0x7fe1f815c1e0 2024-09-17T04:00:20.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.365+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe1f0005470 con 0x7fe1f815c1e0 2024-09-17T04:00:20.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.365+0000 7fe1f5600640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 0x7fe1d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.366+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fe1f009a890 con 0x7fe1f815c1e0 2024-09-17T04:00:20.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.366+0000 7fe1fda00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 0x7fe1d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.367+0000 7fe1fda00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 0x7fe1d807b0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec034000 tx=0x7fe1ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:20.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.370+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe1f0062700 con 0x7fe1f815c1e0 2024-09-17T04:00:20.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.551+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1f81f9880 con 0x7fe1d8078bc0 2024-09-17T04:00:20.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.556+0000 7fe1f5600640 1 -- 172.21.15.129:0/4189417814 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252490 (secure 0 0 0) 0x7fe1f81f9880 con 0x7fe1d8078bc0 2024-09-17T04:00:20.559 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:20.560 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 msgr2=0x7fe1d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 0x7fe1d807b0d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec034000 tx=0x7fe1ec004730 comp rx=0 tx=0).stop 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 msgr2=0x7fe1f806ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000c760 tx=0x7fe1f000cc20 comp rx=0 tx=0).stop 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 shutdown_connections 2024-09-17T04:00:20.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe1d8078bc0 0x7fe1d807b0d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1f815c1e0 0x7fe1f806ecb0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1f81524c0 0x7fe1f8150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 --2- 172.21.15.129:0/4189417814 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1f80cfb00 0x7fe1f80cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 >> 172.21.15.129:0/4189417814 conn(0x7fe1f806cfc0 msgr2=0x7fe1f8075e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.560+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 shutdown_connections 2024-09-17T04:00:20.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.561+0000 7fe1fee00640 1 -- 172.21.15.129:0/4189417814 wait complete. 2024-09-17T04:00:20.586 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-17T04:00:20.477283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324928,"kb_used_data":17180,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754649792,"statfs":{"total":773094113280,"available":772761387008,"internally_reserved":0,"allocated":17592320,"data_stored":8333864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.322782"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978562+0000","last_change":"2024-09-17T04:00:15.205436+0000","last_active":"2024-09-17T04:00:17.978562+0000","last_peered":"2024-09-17T04:00:17.978562+0000","last_clean":"2024-09-17T04:00:17.978562+0000","last_became_active":"2024-09-17T04:00:15.205019+0000","last_became_peered":"2024-09-17T04:00:15.205019+0000","last_unstale":"2024-09-17T04:00:17.978562+0000","last_undegraded":"2024-09-17T04:00:17.978562+0000","last_fullsized":"2024-09-17T04:00:17.978562+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.197723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977616+0000","last_change":"2024-09-17T04:00:15.198110+0000","last_active":"2024-09-17T04:00:17.977616+0000","last_peered":"2024-09-17T04:00:17.977616+0000","last_clean":"2024-09-17T04:00:17.977616+0000","last_became_active":"2024-09-17T04:00:15.197746+0000","last_became_peered":"2024-09-17T04:00:15.197746+0000","last_unstale":"2024-09-17T04:00:17.977616+0000","last_undegraded":"2024-09-17T04:00:17.977616+0000","last_fullsized":"2024-09-17T04:00:17.977616+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181565+0000","last_change":"2024-09-17T04:00:15.195874+0000","last_active":"2024-09-17T04:00:17.181565+0000","last_peered":"2024-09-17T04:00:17.181565+0000","last_clean":"2024-09-17T04:00:17.181565+0000","last_became_active":"2024-09-17T04:00:15.194412+0000","last_became_peered":"2024-09-17T04:00:15.194412+0000","last_unstale":"2024-09-17T04:00:17.181565+0000","last_undegraded":"2024-09-17T04:00:17.181565+0000","last_fullsized":"2024-09-17T04:00:17.181565+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:11.985959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977850+0000","last_change":"2024-09-17T04:00:15.198044+0000","last_active":"2024-09-17T04:00:17.977850+0000","last_peered":"2024-09-17T04:00:17.977850+0000","last_clean":"2024-09-17T04:00:17.977850+0000","last_became_active":"2024-09-17T04:00:15.197647+0000","last_became_peered":"2024-09-17T04:00:15.197647+0000","last_unstale":"2024-09-17T04:00:17.977850+0000","last_undegraded":"2024-09-17T04:00:17.977850+0000","last_fullsized":"2024-09-17T04:00:17.977850+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.114514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354445+0000","last_change":"2024-09-17T04:00:15.206714+0000","last_active":"2024-09-17T04:00:17.354445+0000","last_peered":"2024-09-17T04:00:17.354445+0000","last_clean":"2024-09-17T04:00:17.354445+0000","last_became_active":"2024-09-17T04:00:15.206465+0000","last_became_peered":"2024-09-17T04:00:15.206465+0000","last_unstale":"2024-09-17T04:00:17.354445+0000","last_undegraded":"2024-09-17T04:00:17.354445+0000","last_fullsized":"2024-09-17T04:00:17.354445+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:29.233403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181368+0000","last_change":"2024-09-17T04:00:15.195270+0000","last_active":"2024-09-17T04:00:17.181368+0000","last_peered":"2024-09-17T04:00:17.181368+0000","last_clean":"2024-09-17T04:00:17.181368+0000","last_became_active":"2024-09-17T04:00:15.194918+0000","last_became_peered":"2024-09-17T04:00:15.194918+0000","last_unstale":"2024-09-17T04:00:17.181368+0000","last_undegraded":"2024-09-17T04:00:17.181368+0000","last_fullsized":"2024-09-17T04:00:17.181368+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:12.702219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186097+0000","last_change":"2024-09-17T04:00:15.201371+0000","last_active":"2024-09-17T04:00:16.186097+0000","last_peered":"2024-09-17T04:00:16.186097+0000","last_clean":"2024-09-17T04:00:16.186097+0000","last_became_active":"2024-09-17T04:00:15.200614+0000","last_became_peered":"2024-09-17T04:00:15.200614+0000","last_unstale":"2024-09-17T04:00:16.186097+0000","last_undegraded":"2024-09-17T04:00:16.186097+0000","last_fullsized":"2024-09-17T04:00:16.186097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:15.371085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976448+0000","last_change":"2024-09-17T04:00:15.179118+0000","last_active":"2024-09-17T04:00:17.976448+0000","last_peered":"2024-09-17T04:00:17.976448+0000","last_clean":"2024-09-17T04:00:17.976448+0000","last_became_active":"2024-09-17T04:00:15.178825+0000","last_became_peered":"2024-09-17T04:00:15.178825+0000","last_unstale":"2024-09-17T04:00:17.976448+0000","last_undegraded":"2024-09-17T04:00:17.976448+0000","last_fullsized":"2024-09-17T04:00:17.976448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:10.154034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185605+0000","last_change":"2024-09-17T04:00:15.196095+0000","last_active":"2024-09-17T04:00:16.185605+0000","last_peered":"2024-09-17T04:00:16.185605+0000","last_clean":"2024-09-17T04:00:16.185605+0000","last_became_active":"2024-09-17T04:00:15.195155+0000","last_became_peered":"2024-09-17T04:00:15.195155+0000","last_unstale":"2024-09-17T04:00:16.185605+0000","last_undegraded":"2024-09-17T04:00:16.185605+0000","last_fullsized":"2024-09-17T04:00:16.185605+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:35.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185181+0000","last_change":"2024-09-17T04:00:15.187116+0000","last_active":"2024-09-17T04:00:16.185181+0000","last_peered":"2024-09-17T04:00:16.185181+0000","last_clean":"2024-09-17T04:00:16.185181+0000","last_became_active":"2024-09-17T04:00:15.186491+0000","last_became_peered":"2024-09-17T04:00:15.186491+0000","last_unstale":"2024-09-17T04:00:16.185181+0000","last_undegraded":"2024-09-17T04:00:16.185181+0000","last_fullsized":"2024-09-17T04:00:16.185181+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:20.978854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186343+0000","last_change":"2024-09-17T04:00:15.192294+0000","last_active":"2024-09-17T04:00:16.186343+0000","last_peered":"2024-09-17T04:00:16.186343+0000","last_clean":"2024-09-17T04:00:16.186343+0000","last_became_active":"2024-09-17T04:00:15.191929+0000","last_became_peered":"2024-09-17T04:00:15.191929+0000","last_unstale":"2024-09-17T04:00:16.186343+0000","last_undegraded":"2024-09-17T04:00:16.186343+0000","last_fullsized":"2024-09-17T04:00:16.186343+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:02.758168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977913+0000","last_change":"2024-09-17T04:00:15.202361+0000","last_active":"2024-09-17T04:00:17.977913+0000","last_peered":"2024-09-17T04:00:17.977913+0000","last_clean":"2024-09-17T04:00:17.977913+0000","last_became_active":"2024-09-17T04:00:15.202042+0000","last_became_peered":"2024-09-17T04:00:15.202042+0000","last_unstale":"2024-09-17T04:00:17.977913+0000","last_undegraded":"2024-09-17T04:00:17.977913+0000","last_fullsized":"2024-09-17T04:00:17.977913+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:31.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977369+0000","last_change":"2024-09-17T04:00:15.205859+0000","last_active":"2024-09-17T04:00:17.977369+0000","last_peered":"2024-09-17T04:00:17.977369+0000","last_clean":"2024-09-17T04:00:17.977369+0000","last_became_active":"2024-09-17T04:00:15.205527+0000","last_became_peered":"2024-09-17T04:00:15.205527+0000","last_unstale":"2024-09-17T04:00:17.977369+0000","last_undegraded":"2024-09-17T04:00:17.977369+0000","last_fullsized":"2024-09-17T04:00:17.977369+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:02:13.228284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184876+0000","last_change":"2024-09-17T04:00:15.198473+0000","last_active":"2024-09-17T04:00:16.184876+0000","last_peered":"2024-09-17T04:00:16.184876+0000","last_clean":"2024-09-17T04:00:16.184876+0000","last_became_active":"2024-09-17T04:00:15.194432+0000","last_became_peered":"2024-09-17T04:00:15.194432+0000","last_unstale":"2024-09-17T04:00:16.184876+0000","last_undegraded":"2024-09-17T04:00:16.184876+0000","last_fullsized":"2024-09-17T04:00:16.184876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:07.494846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977556+0000","last_change":"2024-09-17T04:00:15.195580+0000","last_active":"2024-09-17T04:00:17.977556+0000","last_peered":"2024-09-17T04:00:17.977556+0000","last_clean":"2024-09-17T04:00:17.977556+0000","last_became_active":"2024-09-17T04:00:15.194025+0000","last_became_peered":"2024-09-17T04:00:15.194025+0000","last_unstale":"2024-09-17T04:00:17.977556+0000","last_undegraded":"2024-09-17T04:00:17.977556+0000","last_fullsized":"2024-09-17T04:00:17.977556+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:34.997014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976439+0000","last_change":"2024-09-17T04:00:15.188218+0000","last_active":"2024-09-17T04:00:17.976439+0000","last_peered":"2024-09-17T04:00:17.976439+0000","last_clean":"2024-09-17T04:00:17.976439+0000","last_became_active":"2024-09-17T04:00:15.187955+0000","last_became_peered":"2024-09-17T04:00:15.187955+0000","last_unstale":"2024-09-17T04:00:17.976439+0000","last_undegraded":"2024-09-17T04:00:17.976439+0000","last_fullsized":"2024-09-17T04:00:17.976439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354775+0000","last_change":"2024-09-17T04:00:15.208588+0000","last_active":"2024-09-17T04:00:17.354775+0000","last_peered":"2024-09-17T04:00:17.354775+0000","last_clean":"2024-09-17T04:00:17.354775+0000","last_became_active":"2024-09-17T04:00:15.208213+0000","last_became_peered":"2024-09-17T04:00:15.208213+0000","last_unstale":"2024-09-17T04:00:17.354775+0000","last_undegraded":"2024-09-17T04:00:17.354775+0000","last_fullsized":"2024-09-17T04:00:17.354775+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:53.003474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978563+0000","last_change":"2024-09-17T04:00:15.208085+0000","last_active":"2024-09-17T04:00:17.978563+0000","last_peered":"2024-09-17T04:00:17.978563+0000","last_clean":"2024-09-17T04:00:17.978563+0000","last_became_active":"2024-09-17T04:00:15.207762+0000","last_became_peered":"2024-09-17T04:00:15.207762+0000","last_unstale":"2024-09-17T04:00:17.978563+0000","last_undegraded":"2024-09-17T04:00:17.978563+0000","last_fullsized":"2024-09-17T04:00:17.978563+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184909+0000","last_change":"2024-09-17T04:00:15.185185+0000","last_active":"2024-09-17T04:00:16.184909+0000","last_peered":"2024-09-17T04:00:16.184909+0000","last_clean":"2024-09-17T04:00:16.184909+0000","last_became_active":"2024-09-17T04:00:15.182697+0000","last_became_peered":"2024-09-17T04:00:15.182697+0000","last_unstale":"2024-09-17T04:00:16.184909+0000","last_undegraded":"2024-09-17T04:00:16.184909+0000","last_fullsized":"2024-09-17T04:00:16.184909+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:06.451276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354394+0000","last_change":"2024-09-17T04:00:15.210019+0000","last_active":"2024-09-17T04:00:17.354394+0000","last_peered":"2024-09-17T04:00:17.354394+0000","last_clean":"2024-09-17T04:00:17.354394+0000","last_became_active":"2024-09-17T04:00:15.209795+0000","last_became_peered":"2024-09-17T04:00:15.209795+0000","last_unstale":"2024-09-17T04:00:17.354394+0000","last_undegraded":"2024-09-17T04:00:17.354394+0000","last_fullsized":"2024-09-17T04:00:17.354394+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:27.563680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354687+0000","last_change":"2024-09-17T04:00:15.206043+0000","last_active":"2024-09-17T04:00:17.354687+0000","last_peered":"2024-09-17T04:00:17.354687+0000","last_clean":"2024-09-17T04:00:17.354687+0000","last_became_active":"2024-09-17T04:00:15.205686+0000","last_became_peered":"2024-09-17T04:00:15.205686+0000","last_unstale":"2024-09-17T04:00:17.354687+0000","last_undegraded":"2024-09-17T04:00:17.354687+0000","last_fullsized":"2024-09-17T04:00:17.354687+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:40.936992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184888+0000","last_change":"2024-09-17T04:00:15.187015+0000","last_active":"2024-09-17T04:00:16.184888+0000","last_peered":"2024-09-17T04:00:16.184888+0000","last_clean":"2024-09-17T04:00:16.184888+0000","last_became_active":"2024-09-17T04:00:15.186574+0000","last_became_peered":"2024-09-17T04:00:15.186574+0000","last_unstale":"2024-09-17T04:00:16.184888+0000","last_undegraded":"2024-09-17T04:00:16.184888+0000","last_fullsized":"2024-09-17T04:00:16.184888+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:05.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181307+0000","last_change":"2024-09-17T04:00:15.195322+0000","last_active":"2024-09-17T04:00:17.181307+0000","last_peered":"2024-09-17T04:00:17.181307+0000","last_clean":"2024-09-17T04:00:17.181307+0000","last_became_active":"2024-09-17T04:00:15.194994+0000","last_became_peered":"2024-09-17T04:00:15.194994+0000","last_unstale":"2024-09-17T04:00:17.181307+0000","last_undegraded":"2024-09-17T04:00:17.181307+0000","last_fullsized":"2024-09-17T04:00:17.181307+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:36.557958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976714+0000","last_change":"2024-09-17T04:00:15.187550+0000","last_active":"2024-09-17T04:00:17.976714+0000","last_peered":"2024-09-17T04:00:17.976714+0000","last_clean":"2024-09-17T04:00:17.976714+0000","last_became_active":"2024-09-17T04:00:15.183952+0000","last_became_peered":"2024-09-17T04:00:15.183952+0000","last_unstale":"2024-09-17T04:00:17.976714+0000","last_undegraded":"2024-09-17T04:00:17.976714+0000","last_fullsized":"2024-09-17T04:00:17.976714+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:53.762374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977962+0000","last_change":"2024-09-17T04:00:15.196465+0000","last_active":"2024-09-17T04:00:17.977962+0000","last_peered":"2024-09-17T04:00:17.977962+0000","last_clean":"2024-09-17T04:00:17.977962+0000","last_became_active":"2024-09-17T04:00:15.195761+0000","last_became_peered":"2024-09-17T04:00:15.195761+0000","last_unstale":"2024-09-17T04:00:17.977962+0000","last_undegraded":"2024-09-17T04:00:17.977962+0000","last_fullsized":"2024-09-17T04:00:17.977962+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:29.557541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186879+0000","last_change":"2024-09-17T04:00:15.201293+0000","last_active":"2024-09-17T04:00:16.186879+0000","last_peered":"2024-09-17T04:00:16.186879+0000","last_clean":"2024-09-17T04:00:16.186879+0000","last_became_active":"2024-09-17T04:00:15.200722+0000","last_became_peered":"2024-09-17T04:00:15.200722+0000","last_unstale":"2024-09-17T04:00:16.186879+0000","last_undegraded":"2024-09-17T04:00:16.186879+0000","last_fullsized":"2024-09-17T04:00:16.186879+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:21.636014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181655+0000","last_change":"2024-09-17T04:00:15.193142+0000","last_active":"2024-09-17T04:00:17.181655+0000","last_peered":"2024-09-17T04:00:17.181655+0000","last_clean":"2024-09-17T04:00:17.181655+0000","last_became_active":"2024-09-17T04:00:15.192818+0000","last_became_peered":"2024-09-17T04:00:15.192818+0000","last_unstale":"2024-09-17T04:00:17.181655+0000","last_undegraded":"2024-09-17T04:00:17.181655+0000","last_fullsized":"2024-09-17T04:00:17.181655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:51.767875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354448+0000","last_change":"2024-09-17T04:00:15.196835+0000","last_active":"2024-09-17T04:00:17.354448+0000","last_peered":"2024-09-17T04:00:17.354448+0000","last_clean":"2024-09-17T04:00:17.354448+0000","last_became_active":"2024-09-17T04:00:15.196475+0000","last_became_peered":"2024-09-17T04:00:15.196475+0000","last_unstale":"2024-09-17T04:00:17.354448+0000","last_undegraded":"2024-09-17T04:00:17.354448+0000","last_fullsized":"2024-09-17T04:00:17.354448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:19.570741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977568+0000","last_change":"2024-09-17T04:00:15.203508+0000","last_active":"2024-09-17T04:00:17.977568+0000","last_peered":"2024-09-17T04:00:17.977568+0000","last_clean":"2024-09-17T04:00:17.977568+0000","last_became_active":"2024-09-17T04:00:15.203187+0000","last_became_peered":"2024-09-17T04:00:15.203187+0000","last_unstale":"2024-09-17T04:00:17.977568+0000","last_undegraded":"2024-09-17T04:00:17.977568+0000","last_fullsized":"2024-09-17T04:00:17.977568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:55.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184802+0000","last_change":"2024-09-17T04:00:15.194543+0000","last_active":"2024-09-17T04:00:16.184802+0000","last_peered":"2024-09-17T04:00:16.184802+0000","last_clean":"2024-09-17T04:00:16.184802+0000","last_became_active":"2024-09-17T04:00:15.194011+0000","last_became_peered":"2024-09-17T04:00:15.194011+0000","last_unstale":"2024-09-17T04:00:16.184802+0000","last_undegraded":"2024-09-17T04:00:16.184802+0000","last_fullsized":"2024-09-17T04:00:16.184802+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:25.507246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181366+0000","last_change":"2024-09-17T04:00:15.198425+0000","last_active":"2024-09-17T04:00:17.181366+0000","last_peered":"2024-09-17T04:00:17.181366+0000","last_clean":"2024-09-17T04:00:17.181366+0000","last_became_active":"2024-09-17T04:00:15.198098+0000","last_became_peered":"2024-09-17T04:00:15.198098+0000","last_unstale":"2024-09-17T04:00:17.181366+0000","last_undegraded":"2024-09-17T04:00:17.181366+0000","last_fullsized":"2024-09-17T04:00:17.181366+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:09.331234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354139+0000","last_change":"2024-09-17T04:00:15.210848+0000","last_active":"2024-09-17T04:00:17.354139+0000","last_peered":"2024-09-17T04:00:17.354139+0000","last_clean":"2024-09-17T04:00:17.354139+0000","last_became_active":"2024-09-17T04:00:15.210516+0000","last_became_peered":"2024-09-17T04:00:15.210516+0000","last_unstale":"2024-09-17T04:00:17.354139+0000","last_undegraded":"2024-09-17T04:00:17.354139+0000","last_fullsized":"2024-09-17T04:00:17.354139+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:55.037291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977006+0000","last_change":"2024-09-17T04:00:15.185994+0000","last_active":"2024-09-17T04:00:17.977006+0000","last_peered":"2024-09-17T04:00:17.977006+0000","last_clean":"2024-09-17T04:00:17.977006+0000","last_became_active":"2024-09-17T04:00:15.184966+0000","last_became_peered":"2024-09-17T04:00:15.184966+0000","last_unstale":"2024-09-17T04:00:17.977006+0000","last_undegraded":"2024-09-17T04:00:17.977006+0000","last_fullsized":"2024-09-17T04:00:17.977006+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:34.976945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"162'4","reported_seq":19,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354792+0000","last_change":"2024-09-17T04:00:14.184544+0000","last_active":"2024-09-17T04:00:17.354792+0000","last_peered":"2024-09-17T04:00:17.354792+0000","last_clean":"2024-09-17T04:00:17.354792+0000","last_became_active":"2024-09-17T04:00:14.184254+0000","last_became_peered":"2024-09-17T04:00:14.184254+0000","last_unstale":"2024-09-17T04:00:17.354792+0000","last_undegraded":"2024-09-17T04:00:17.354792+0000","last_fullsized":"2024-09-17T04:00:17.354792+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:15:30.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186087+0000","last_change":"2024-09-17T04:00:15.203100+0000","last_active":"2024-09-17T04:00:16.186087+0000","last_peered":"2024-09-17T04:00:16.186087+0000","last_clean":"2024-09-17T04:00:16.186087+0000","last_became_active":"2024-09-17T04:00:15.202011+0000","last_became_peered":"2024-09-17T04:00:15.202011+0000","last_unstale":"2024-09-17T04:00:16.186087+0000","last_undegraded":"2024-09-17T04:00:16.186087+0000","last_fullsized":"2024-09-17T04:00:16.186087+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:07.578261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185365+0000","last_change":"2024-09-17T04:00:14.178919+0000","last_active":"2024-09-17T04:00:16.185365+0000","last_peered":"2024-09-17T04:00:16.185365+0000","last_clean":"2024-09-17T04:00:16.185365+0000","last_became_active":"2024-09-17T04:00:14.177710+0000","last_became_peered":"2024-09-17T04:00:14.177710+0000","last_unstale":"2024-09-17T04:00:16.185365+0000","last_undegraded":"2024-09-17T04:00:16.185365+0000","last_fullsized":"2024-09-17T04:00:16.185365+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:18.564335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186429+0000","last_change":"2024-09-17T04:00:15.202175+0000","last_active":"2024-09-17T04:00:16.186429+0000","last_peered":"2024-09-17T04:00:16.186429+0000","last_clean":"2024-09-17T04:00:16.186429+0000","last_became_active":"2024-09-17T04:00:15.201873+0000","last_became_peered":"2024-09-17T04:00:15.201873+0000","last_unstale":"2024-09-17T04:00:16.186429+0000","last_undegraded":"2024-09-17T04:00:16.186429+0000","last_fullsized":"2024-09-17T04:00:16.186429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:23.847548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181775+0000","last_change":"2024-09-17T04:00:14.173566+0000","last_active":"2024-09-17T04:00:17.181775+0000","last_peered":"2024-09-17T04:00:17.181775+0000","last_clean":"2024-09-17T04:00:17.181775+0000","last_became_active":"2024-09-17T04:00:14.173201+0000","last_became_peered":"2024-09-17T04:00:14.173201+0000","last_unstale":"2024-09-17T04:00:17.181775+0000","last_undegraded":"2024-09-17T04:00:17.181775+0000","last_fullsized":"2024-09-17T04:00:17.181775+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:15.816175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185473+0000","last_change":"2024-09-17T04:00:15.199464+0000","last_active":"2024-09-17T04:00:16.185473+0000","last_peered":"2024-09-17T04:00:16.185473+0000","last_clean":"2024-09-17T04:00:16.185473+0000","last_became_active":"2024-09-17T04:00:15.199050+0000","last_became_peered":"2024-09-17T04:00:15.199050+0000","last_unstale":"2024-09-17T04:00:16.185473+0000","last_undegraded":"2024-09-17T04:00:16.185473+0000","last_fullsized":"2024-09-17T04:00:16.185473+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:33.024811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977943+0000","last_change":"2024-09-17T04:00:14.180751+0000","last_active":"2024-09-17T04:00:17.977943+0000","last_peered":"2024-09-17T04:00:17.977943+0000","last_clean":"2024-09-17T04:00:17.977943+0000","last_became_active":"2024-09-17T04:00:14.180400+0000","last_became_peered":"2024-09-17T04:00:14.180400+0000","last_unstale":"2024-09-17T04:00:17.977943+0000","last_undegraded":"2024-09-17T04:00:17.977943+0000","last_fullsized":"2024-09-17T04:00:17.977943+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:14.945761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185444+0000","last_change":"2024-09-17T04:00:15.183077+0000","last_active":"2024-09-17T04:00:16.185444+0000","last_peered":"2024-09-17T04:00:16.185444+0000","last_clean":"2024-09-17T04:00:16.185444+0000","last_became_active":"2024-09-17T04:00:15.182579+0000","last_became_peered":"2024-09-17T04:00:15.182579+0000","last_unstale":"2024-09-17T04:00:16.185444+0000","last_undegraded":"2024-09-17T04:00:16.185444+0000","last_fullsized":"2024-09-17T04:00:16.185444+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:32.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354752+0000","last_change":"2024-09-17T04:00:14.178007+0000","last_active":"2024-09-17T04:00:17.354752+0000","last_peered":"2024-09-17T04:00:17.354752+0000","last_clean":"2024-09-17T04:00:17.354752+0000","last_became_active":"2024-09-17T04:00:14.177397+0000","last_became_peered":"2024-09-17T04:00:14.177397+0000","last_unstale":"2024-09-17T04:00:17.354752+0000","last_undegraded":"2024-09-17T04:00:17.354752+0000","last_fullsized":"2024-09-17T04:00:17.354752+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:46.574189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976655+0000","last_change":"2024-09-17T04:00:15.184558+0000","last_active":"2024-09-17T04:00:17.976655+0000","last_peered":"2024-09-17T04:00:17.976655+0000","last_clean":"2024-09-17T04:00:17.976655+0000","last_became_active":"2024-09-17T04:00:15.181827+0000","last_became_peered":"2024-09-17T04:00:15.181827+0000","last_unstale":"2024-09-17T04:00:17.976655+0000","last_undegraded":"2024-09-17T04:00:17.976655+0000","last_fullsized":"2024-09-17T04:00:17.976655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:03.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976490+0000","last_change":"2024-09-17T04:00:14.181701+0000","last_active":"2024-09-17T04:00:17.976490+0000","last_peered":"2024-09-17T04:00:17.976490+0000","last_clean":"2024-09-17T04:00:17.976490+0000","last_became_active":"2024-09-17T04:00:14.180617+0000","last_became_peered":"2024-09-17T04:00:14.180617+0000","last_unstale":"2024-09-17T04:00:17.976490+0000","last_undegraded":"2024-09-17T04:00:17.976490+0000","last_fullsized":"2024-09-17T04:00:17.976490+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185980+0000","last_change":"2024-09-17T04:00:15.188106+0000","last_active":"2024-09-17T04:00:16.185980+0000","last_peered":"2024-09-17T04:00:16.185980+0000","last_clean":"2024-09-17T04:00:16.185980+0000","last_became_active":"2024-09-17T04:00:15.187755+0000","last_became_peered":"2024-09-17T04:00:15.187755+0000","last_unstale":"2024-09-17T04:00:16.185980+0000","last_undegraded":"2024-09-17T04:00:16.185980+0000","last_fullsized":"2024-09-17T04:00:16.185980+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:16.200716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976969+0000","last_change":"2024-09-17T04:00:14.180760+0000","last_active":"2024-09-17T04:00:17.976969+0000","last_peered":"2024-09-17T04:00:17.976969+0000","last_clean":"2024-09-17T04:00:17.976969+0000","last_became_active":"2024-09-17T04:00:14.180063+0000","last_became_peered":"2024-09-17T04:00:14.180063+0000","last_unstale":"2024-09-17T04:00:17.976969+0000","last_undegraded":"2024-09-17T04:00:17.976969+0000","last_fullsized":"2024-09-17T04:00:17.976969+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181748+0000","last_change":"2024-09-17T04:00:15.184353+0000","last_active":"2024-09-17T04:00:17.181748+0000","last_peered":"2024-09-17T04:00:17.181748+0000","last_clean":"2024-09-17T04:00:17.181748+0000","last_became_active":"2024-09-17T04:00:15.183526+0000","last_became_peered":"2024-09-17T04:00:15.183526+0000","last_unstale":"2024-09-17T04:00:17.181748+0000","last_undegraded":"2024-09-17T04:00:17.181748+0000","last_fullsized":"2024-09-17T04:00:17.181748+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:08.174367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186810+0000","last_change":"2024-09-17T04:00:14.170243+0000","last_active":"2024-09-17T04:00:16.186810+0000","last_peered":"2024-09-17T04:00:16.186810+0000","last_clean":"2024-09-17T04:00:16.186810+0000","last_became_active":"2024-09-17T04:00:14.169878+0000","last_became_peered":"2024-09-17T04:00:14.169878+0000","last_unstale":"2024-09-17T04:00:16.186810+0000","last_undegraded":"2024-09-17T04:00:16.186810+0000","last_fullsized":"2024-09-17T04:00:16.186810+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:35.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976963+0000","last_change":"2024-09-17T04:00:15.187262+0000","last_active":"2024-09-17T04:00:17.976963+0000","last_peered":"2024-09-17T04:00:17.976963+0000","last_clean":"2024-09-17T04:00:17.976963+0000","last_became_active":"2024-09-17T04:00:15.186910+0000","last_became_peered":"2024-09-17T04:00:15.186910+0000","last_unstale":"2024-09-17T04:00:17.976963+0000","last_undegraded":"2024-09-17T04:00:17.976963+0000","last_fullsized":"2024-09-17T04:00:17.976963+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:21.523618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354640+0000","last_change":"2024-09-17T04:00:14.178548+0000","last_active":"2024-09-17T04:00:17.354640+0000","last_peered":"2024-09-17T04:00:17.354640+0000","last_clean":"2024-09-17T04:00:17.354640+0000","last_became_active":"2024-09-17T04:00:14.178173+0000","last_became_peered":"2024-09-17T04:00:14.178173+0000","last_unstale":"2024-09-17T04:00:17.354640+0000","last_undegraded":"2024-09-17T04:00:17.354640+0000","last_fullsized":"2024-09-17T04:00:17.354640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:05.150582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186273+0000","last_change":"2024-09-17T04:00:15.190476+0000","last_active":"2024-09-17T04:00:16.186273+0000","last_peered":"2024-09-17T04:00:16.186273+0000","last_clean":"2024-09-17T04:00:16.186273+0000","last_became_active":"2024-09-17T04:00:15.189977+0000","last_became_peered":"2024-09-17T04:00:15.189977+0000","last_unstale":"2024-09-17T04:00:16.186273+0000","last_undegraded":"2024-09-17T04:00:16.186273+0000","last_fullsized":"2024-09-17T04:00:16.186273+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:26.037673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"162'3","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354310+0000","last_change":"2024-09-17T04:00:14.184902+0000","last_active":"2024-09-17T04:00:17.354310+0000","last_peered":"2024-09-17T04:00:17.354310+0000","last_clean":"2024-09-17T04:00:17.354310+0000","last_became_active":"2024-09-17T04:00:14.184624+0000","last_became_peered":"2024-09-17T04:00:14.184624+0000","last_unstale":"2024-09-17T04:00:17.354310+0000","last_undegraded":"2024-09-17T04:00:17.354310+0000","last_fullsized":"2024-09-17T04:00:17.354310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:08.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185090+0000","last_change":"2024-09-17T04:00:15.180729+0000","last_active":"2024-09-17T04:00:16.185090+0000","last_peered":"2024-09-17T04:00:16.185090+0000","last_clean":"2024-09-17T04:00:16.185090+0000","last_became_active":"2024-09-17T04:00:15.180358+0000","last_became_peered":"2024-09-17T04:00:15.180358+0000","last_unstale":"2024-09-17T04:00:16.185090+0000","last_undegraded":"2024-09-17T04:00:16.185090+0000","last_fullsized":"2024-09-17T04:00:16.185090+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:55.762158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"162'6","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181426+0000","last_change":"2024-09-17T04:00:14.177482+0000","last_active":"2024-09-17T04:00:17.181426+0000","last_peered":"2024-09-17T04:00:17.181426+0000","last_clean":"2024-09-17T04:00:17.181426+0000","last_became_active":"2024-09-17T04:00:14.177185+0000","last_became_peered":"2024-09-17T04:00:14.177185+0000","last_unstale":"2024-09-17T04:00:17.181426+0000","last_undegraded":"2024-09-17T04:00:17.181426+0000","last_fullsized":"2024-09-17T04:00:17.181426+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:59.681874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181467+0000","last_change":"2024-09-17T04:00:15.197664+0000","last_active":"2024-09-17T04:00:17.181467+0000","last_peered":"2024-09-17T04:00:17.181467+0000","last_clean":"2024-09-17T04:00:17.181467+0000","last_became_active":"2024-09-17T04:00:15.197330+0000","last_became_peered":"2024-09-17T04:00:15.197330+0000","last_unstale":"2024-09-17T04:00:17.181467+0000","last_undegraded":"2024-09-17T04:00:17.181467+0000","last_fullsized":"2024-09-17T04:00:17.181467+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:09.294262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978258+0000","last_change":"2024-09-17T04:00:14.180486+0000","last_active":"2024-09-17T04:00:17.978258+0000","last_peered":"2024-09-17T04:00:17.978258+0000","last_clean":"2024-09-17T04:00:17.978258+0000","last_became_active":"2024-09-17T04:00:14.180077+0000","last_became_peered":"2024-09-17T04:00:14.180077+0000","last_unstale":"2024-09-17T04:00:17.978258+0000","last_undegraded":"2024-09-17T04:00:17.978258+0000","last_fullsized":"2024-09-17T04:00:17.978258+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:01.024929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186622+0000","last_change":"2024-09-17T04:00:15.195036+0000","last_active":"2024-09-17T04:00:16.186622+0000","last_peered":"2024-09-17T04:00:16.186622+0000","last_clean":"2024-09-17T04:00:16.186622+0000","last_became_active":"2024-09-17T04:00:15.193366+0000","last_became_peered":"2024-09-17T04:00:15.193366+0000","last_unstale":"2024-09-17T04:00:16.186622+0000","last_undegraded":"2024-09-17T04:00:16.186622+0000","last_fullsized":"2024-09-17T04:00:16.186622+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:35.218017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185646+0000","last_change":"2024-09-17T04:00:14.179129+0000","last_active":"2024-09-17T04:00:16.185646+0000","last_peered":"2024-09-17T04:00:16.185646+0000","last_clean":"2024-09-17T04:00:16.185646+0000","last_became_active":"2024-09-17T04:00:14.177651+0000","last_became_peered":"2024-09-17T04:00:14.177651+0000","last_unstale":"2024-09-17T04:00:16.185646+0000","last_undegraded":"2024-09-17T04:00:16.185646+0000","last_fullsized":"2024-09-17T04:00:16.185646+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:50.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184621+0000","last_change":"2024-09-17T04:00:15.195850+0000","last_active":"2024-09-17T04:00:16.184621+0000","last_peered":"2024-09-17T04:00:16.184621+0000","last_clean":"2024-09-17T04:00:16.184621+0000","last_became_active":"2024-09-17T04:00:15.194823+0000","last_became_peered":"2024-09-17T04:00:15.194823+0000","last_unstale":"2024-09-17T04:00:16.184621+0000","last_undegraded":"2024-09-17T04:00:16.184621+0000","last_fullsized":"2024-09-17T04:00:16.184621+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:16.156594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976351+0000","last_change":"2024-09-17T04:00:14.178903+0000","last_active":"2024-09-17T04:00:17.976351+0000","last_peered":"2024-09-17T04:00:17.976351+0000","last_clean":"2024-09-17T04:00:17.976351+0000","last_became_active":"2024-09-17T04:00:14.178605+0000","last_became_peered":"2024-09-17T04:00:14.178605+0000","last_unstale":"2024-09-17T04:00:17.976351+0000","last_undegraded":"2024-09-17T04:00:17.976351+0000","last_fullsized":"2024-09-17T04:00:17.976351+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184829+0000","last_change":"2024-09-17T04:00:15.192316+0000","last_active":"2024-09-17T04:00:16.184829+0000","last_peered":"2024-09-17T04:00:16.184829+0000","last_clean":"2024-09-17T04:00:16.184829+0000","last_became_active":"2024-09-17T04:00:15.188357+0000","last_became_peered":"2024-09-17T04:00:15.188357+0000","last_unstale":"2024-09-17T04:00:16.184829+0000","last_undegraded":"2024-09-17T04:00:16.184829+0000","last_fullsized":"2024-09-17T04:00:16.184829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:34.996367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"162'8","reported_seq":23,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977302+0000","last_change":"2024-09-17T04:00:14.182987+0000","last_active":"2024-09-17T04:00:17.977302+0000","last_peered":"2024-09-17T04:00:17.977302+0000","last_clean":"2024-09-17T04:00:17.977302+0000","last_became_active":"2024-09-17T04:00:14.182665+0000","last_became_peered":"2024-09-17T04:00:14.182665+0000","last_unstale":"2024-09-17T04:00:17.977302+0000","last_undegraded":"2024-09-17T04:00:17.977302+0000","last_fullsized":"2024-09-17T04:00:17.977302+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185423+0000","last_change":"2024-09-17T04:00:15.188466+0000","last_active":"2024-09-17T04:00:16.185423+0000","last_peered":"2024-09-17T04:00:16.185423+0000","last_clean":"2024-09-17T04:00:16.185423+0000","last_became_active":"2024-09-17T04:00:15.187860+0000","last_became_peered":"2024-09-17T04:00:15.187860+0000","last_unstale":"2024-09-17T04:00:16.185423+0000","last_undegraded":"2024-09-17T04:00:16.185423+0000","last_fullsized":"2024-09-17T04:00:16.185423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:05.989755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354873+0000","last_change":"2024-09-17T04:00:14.179849+0000","last_active":"2024-09-17T04:00:17.354873+0000","last_peered":"2024-09-17T04:00:17.354873+0000","last_clean":"2024-09-17T04:00:17.354873+0000","last_became_active":"2024-09-17T04:00:14.178084+0000","last_became_peered":"2024-09-17T04:00:14.178084+0000","last_unstale":"2024-09-17T04:00:17.354873+0000","last_undegraded":"2024-09-17T04:00:17.354873+0000","last_fullsized":"2024-09-17T04:00:17.354873+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:26.618084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185623+0000","last_change":"2024-09-17T04:00:15.190551+0000","last_active":"2024-09-17T04:00:16.185623+0000","last_peered":"2024-09-17T04:00:16.185623+0000","last_clean":"2024-09-17T04:00:16.185623+0000","last_became_active":"2024-09-17T04:00:15.190147+0000","last_became_peered":"2024-09-17T04:00:15.190147+0000","last_unstale":"2024-09-17T04:00:16.185623+0000","last_undegraded":"2024-09-17T04:00:16.185623+0000","last_fullsized":"2024-09-17T04:00:16.185623+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:00.814493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977169+0000","last_change":"2024-09-17T04:00:14.173867+0000","last_active":"2024-09-17T04:00:17.977169+0000","last_peered":"2024-09-17T04:00:17.977169+0000","last_clean":"2024-09-17T04:00:17.977169+0000","last_became_active":"2024-09-17T04:00:14.171614+0000","last_became_peered":"2024-09-17T04:00:14.171614+0000","last_unstale":"2024-09-17T04:00:17.977169+0000","last_undegraded":"2024-09-17T04:00:17.977169+0000","last_fullsized":"2024-09-17T04:00:17.977169+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185175+0000","last_change":"2024-09-17T04:00:15.196127+0000","last_active":"2024-09-17T04:00:16.185175+0000","last_peered":"2024-09-17T04:00:16.185175+0000","last_clean":"2024-09-17T04:00:16.185175+0000","last_became_active":"2024-09-17T04:00:15.195799+0000","last_became_peered":"2024-09-17T04:00:15.195799+0000","last_unstale":"2024-09-17T04:00:16.185175+0000","last_undegraded":"2024-09-17T04:00:16.185175+0000","last_fullsized":"2024-09-17T04:00:16.185175+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:45:34.672205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978453+0000","last_change":"2024-09-17T04:00:14.183120+0000","last_active":"2024-09-17T04:00:17.978453+0000","last_peered":"2024-09-17T04:00:17.978453+0000","last_clean":"2024-09-17T04:00:17.978453+0000","last_became_active":"2024-09-17T04:00:14.182779+0000","last_became_peered":"2024-09-17T04:00:14.182779+0000","last_unstale":"2024-09-17T04:00:17.978453+0000","last_undegraded":"2024-09-17T04:00:17.978453+0000","last_fullsized":"2024-09-17T04:00:17.978453+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977807+0000","last_change":"2024-09-17T04:00:15.204420+0000","last_active":"2024-09-17T04:00:17.977807+0000","last_peered":"2024-09-17T04:00:17.977807+0000","last_clean":"2024-09-17T04:00:17.977807+0000","last_became_active":"2024-09-17T04:00:15.202153+0000","last_became_peered":"2024-09-17T04:00:15.202153+0000","last_unstale":"2024-09-17T04:00:17.977807+0000","last_undegraded":"2024-09-17T04:00:17.977807+0000","last_fullsized":"2024-09-17T04:00:17.977807+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:57.466014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186468+0000","last_change":"2024-09-17T04:00:14.164836+0000","last_active":"2024-09-17T04:00:16.186468+0000","last_peered":"2024-09-17T04:00:16.186468+0000","last_clean":"2024-09-17T04:00:16.186468+0000","last_became_active":"2024-09-17T04:00:14.164491+0000","last_became_peered":"2024-09-17T04:00:14.164491+0000","last_unstale":"2024-09-17T04:00:16.186468+0000","last_undegraded":"2024-09-17T04:00:16.186468+0000","last_fullsized":"2024-09-17T04:00:16.186468+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:53:35.894685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185709+0000","last_change":"2024-09-17T04:00:15.203883+0000","last_active":"2024-09-17T04:00:16.185709+0000","last_peered":"2024-09-17T04:00:16.185709+0000","last_clean":"2024-09-17T04:00:16.185709+0000","last_became_active":"2024-09-17T04:00:15.201414+0000","last_became_peered":"2024-09-17T04:00:15.201414+0000","last_unstale":"2024-09-17T04:00:16.185709+0000","last_undegraded":"2024-09-17T04:00:16.185709+0000","last_fullsized":"2024-09-17T04:00:16.185709+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:29.778689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181504+0000","last_change":"2024-09-17T04:00:14.185714+0000","last_active":"2024-09-17T04:00:17.181504+0000","last_peered":"2024-09-17T04:00:17.181504+0000","last_clean":"2024-09-17T04:00:17.181504+0000","last_became_active":"2024-09-17T04:00:14.185363+0000","last_became_peered":"2024-09-17T04:00:14.185363+0000","last_unstale":"2024-09-17T04:00:17.181504+0000","last_undegraded":"2024-09-17T04:00:17.181504+0000","last_fullsized":"2024-09-17T04:00:17.181504+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.866011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186668+0000","last_change":"2024-09-17T04:00:15.179231+0000","last_active":"2024-09-17T04:00:16.186668+0000","last_peered":"2024-09-17T04:00:16.186668+0000","last_clean":"2024-09-17T04:00:16.186668+0000","last_became_active":"2024-09-17T04:00:15.178887+0000","last_became_peered":"2024-09-17T04:00:15.178887+0000","last_unstale":"2024-09-17T04:00:16.186668+0000","last_undegraded":"2024-09-17T04:00:16.186668+0000","last_fullsized":"2024-09-17T04:00:16.186668+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:32.045406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354175+0000","last_change":"2024-09-17T04:00:14.183023+0000","last_active":"2024-09-17T04:00:17.354175+0000","last_peered":"2024-09-17T04:00:17.354175+0000","last_clean":"2024-09-17T04:00:17.354175+0000","last_became_active":"2024-09-17T04:00:14.178678+0000","last_became_peered":"2024-09-17T04:00:14.178678+0000","last_unstale":"2024-09-17T04:00:17.354175+0000","last_undegraded":"2024-09-17T04:00:17.354175+0000","last_fullsized":"2024-09-17T04:00:17.354175+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:09.710450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977340+0000","last_change":"2024-09-17T04:00:15.202518+0000","last_active":"2024-09-17T04:00:17.977340+0000","last_peered":"2024-09-17T04:00:17.977340+0000","last_clean":"2024-09-17T04:00:17.977340+0000","last_became_active":"2024-09-17T04:00:15.202240+0000","last_became_peered":"2024-09-17T04:00:15.202240+0000","last_unstale":"2024-09-17T04:00:17.977340+0000","last_undegraded":"2024-09-17T04:00:17.977340+0000","last_fullsized":"2024-09-17T04:00:17.977340+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:59.197974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181510+0000","last_change":"2024-09-17T04:00:14.178227+0000","last_active":"2024-09-17T04:00:17.181510+0000","last_peered":"2024-09-17T04:00:17.181510+0000","last_clean":"2024-09-17T04:00:17.181510+0000","last_became_active":"2024-09-17T04:00:14.177922+0000","last_became_peered":"2024-09-17T04:00:14.177922+0000","last_unstale":"2024-09-17T04:00:17.181510+0000","last_undegraded":"2024-09-17T04:00:17.181510+0000","last_fullsized":"2024-09-17T04:00:17.181510+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:49.911464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354137+0000","last_change":"2024-09-17T04:00:15.200391+0000","last_active":"2024-09-17T04:00:17.354137+0000","last_peered":"2024-09-17T04:00:17.354137+0000","last_clean":"2024-09-17T04:00:17.354137+0000","last_became_active":"2024-09-17T04:00:15.200095+0000","last_became_peered":"2024-09-17T04:00:15.200095+0000","last_unstale":"2024-09-17T04:00:17.354137+0000","last_undegraded":"2024-09-17T04:00:17.354137+0000","last_fullsized":"2024-09-17T04:00:17.354137+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:18:34.896086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184666+0000","last_change":"2024-09-17T04:00:14.173438+0000","last_active":"2024-09-17T04:00:16.184666+0000","last_peered":"2024-09-17T04:00:16.184666+0000","last_clean":"2024-09-17T04:00:16.184666+0000","last_became_active":"2024-09-17T04:00:14.173090+0000","last_became_peered":"2024-09-17T04:00:14.173090+0000","last_unstale":"2024-09-17T04:00:16.184666+0000","last_undegraded":"2024-09-17T04:00:16.184666+0000","last_fullsized":"2024-09-17T04:00:16.184666+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:21.951204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354500+0000","last_change":"2024-09-17T04:00:15.206400+0000","last_active":"2024-09-17T04:00:17.354500+0000","last_peered":"2024-09-17T04:00:17.354500+0000","last_clean":"2024-09-17T04:00:17.354500+0000","last_became_active":"2024-09-17T04:00:15.205707+0000","last_became_peered":"2024-09-17T04:00:15.205707+0000","last_unstale":"2024-09-17T04:00:17.354500+0000","last_undegraded":"2024-09-17T04:00:17.354500+0000","last_fullsized":"2024-09-17T04:00:17.354500+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:45:36.571087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978691+0000","last_change":"2024-09-17T04:00:14.186855+0000","last_active":"2024-09-17T04:00:17.978691+0000","last_peered":"2024-09-17T04:00:17.978691+0000","last_clean":"2024-09-17T04:00:17.978691+0000","last_became_active":"2024-09-17T04:00:14.186535+0000","last_became_peered":"2024-09-17T04:00:14.186535+0000","last_unstale":"2024-09-17T04:00:17.978691+0000","last_undegraded":"2024-09-17T04:00:17.978691+0000","last_fullsized":"2024-09-17T04:00:17.978691+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:32.914832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186153+0000","last_change":"2024-09-17T04:00:15.194286+0000","last_active":"2024-09-17T04:00:16.186153+0000","last_peered":"2024-09-17T04:00:16.186153+0000","last_clean":"2024-09-17T04:00:16.186153+0000","last_became_active":"2024-09-17T04:00:15.193936+0000","last_became_peered":"2024-09-17T04:00:15.193936+0000","last_unstale":"2024-09-17T04:00:16.186153+0000","last_undegraded":"2024-09-17T04:00:16.186153+0000","last_fullsized":"2024-09-17T04:00:16.186153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:03.221443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"163'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185534+0000","last_change":"2024-09-17T04:00:14.176145+0000","last_active":"2024-09-17T04:00:16.185534+0000","last_peered":"2024-09-17T04:00:16.185534+0000","last_clean":"2024-09-17T04:00:16.185534+0000","last_became_active":"2024-09-17T04:00:14.175839+0000","last_became_peered":"2024-09-17T04:00:14.175839+0000","last_unstale":"2024-09-17T04:00:16.185534+0000","last_undegraded":"2024-09-17T04:00:16.185534+0000","last_fullsized":"2024-09-17T04:00:16.185534+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:33.949960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976660+0000","last_change":"2024-09-17T04:00:15.181165+0000","last_active":"2024-09-17T04:00:17.976660+0000","last_peered":"2024-09-17T04:00:17.976660+0000","last_clean":"2024-09-17T04:00:17.976660+0000","last_became_active":"2024-09-17T04:00:15.180136+0000","last_became_peered":"2024-09-17T04:00:15.180136+0000","last_unstale":"2024-09-17T04:00:17.976660+0000","last_undegraded":"2024-09-17T04:00:17.976660+0000","last_fullsized":"2024-09-17T04:00:17.976660+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:28.049513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184972+0000","last_change":"2024-09-17T04:00:14.182944+0000","last_active":"2024-09-17T04:00:16.184972+0000","last_peered":"2024-09-17T04:00:16.184972+0000","last_clean":"2024-09-17T04:00:16.184972+0000","last_became_active":"2024-09-17T04:00:14.182626+0000","last_became_peered":"2024-09-17T04:00:14.182626+0000","last_unstale":"2024-09-17T04:00:16.184972+0000","last_undegraded":"2024-09-17T04:00:16.184972+0000","last_fullsized":"2024-09-17T04:00:16.184972+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:33.722613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976876+0000","last_change":"2024-09-17T04:00:15.185309+0000","last_active":"2024-09-17T04:00:17.976876+0000","last_peered":"2024-09-17T04:00:17.976876+0000","last_clean":"2024-09-17T04:00:17.976876+0000","last_became_active":"2024-09-17T04:00:15.184979+0000","last_became_peered":"2024-09-17T04:00:15.184979+0000","last_unstale":"2024-09-17T04:00:17.976876+0000","last_undegraded":"2024-09-17T04:00:17.976876+0000","last_fullsized":"2024-09-17T04:00:17.976876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:29.123723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976765+0000","last_change":"2024-09-17T04:00:14.172333+0000","last_active":"2024-09-17T04:00:17.976765+0000","last_peered":"2024-09-17T04:00:17.976765+0000","last_clean":"2024-09-17T04:00:17.976765+0000","last_became_active":"2024-09-17T04:00:14.170300+0000","last_became_peered":"2024-09-17T04:00:14.170300+0000","last_unstale":"2024-09-17T04:00:17.976765+0000","last_undegraded":"2024-09-17T04:00:17.976765+0000","last_fullsized":"2024-09-17T04:00:17.976765+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977992+0000","last_change":"2024-09-17T04:00:15.201456+0000","last_active":"2024-09-17T04:00:17.977992+0000","last_peered":"2024-09-17T04:00:17.977992+0000","last_clean":"2024-09-17T04:00:17.977992+0000","last_became_active":"2024-09-17T04:00:15.201067+0000","last_became_peered":"2024-09-17T04:00:15.201067+0000","last_unstale":"2024-09-17T04:00:17.977992+0000","last_undegraded":"2024-09-17T04:00:17.977992+0000","last_fullsized":"2024-09-17T04:00:17.977992+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:26.401458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184947+0000","last_change":"2024-09-17T04:00:14.172200+0000","last_active":"2024-09-17T04:00:16.184947+0000","last_peered":"2024-09-17T04:00:16.184947+0000","last_clean":"2024-09-17T04:00:16.184947+0000","last_became_active":"2024-09-17T04:00:14.171692+0000","last_became_peered":"2024-09-17T04:00:14.171692+0000","last_unstale":"2024-09-17T04:00:16.184947+0000","last_undegraded":"2024-09-17T04:00:16.184947+0000","last_fullsized":"2024-09-17T04:00:16.184947+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:39.246219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978058+0000","last_change":"2024-09-17T04:00:15.207451+0000","last_active":"2024-09-17T04:00:17.978058+0000","last_peered":"2024-09-17T04:00:17.978058+0000","last_clean":"2024-09-17T04:00:17.978058+0000","last_became_active":"2024-09-17T04:00:15.207157+0000","last_became_peered":"2024-09-17T04:00:15.207157+0000","last_unstale":"2024-09-17T04:00:17.978058+0000","last_undegraded":"2024-09-17T04:00:17.978058+0000","last_fullsized":"2024-09-17T04:00:17.978058+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:33:09.116340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185026+0000","last_change":"2024-09-17T04:00:14.175785+0000","last_active":"2024-09-17T04:00:16.185026+0000","last_peered":"2024-09-17T04:00:16.185026+0000","last_clean":"2024-09-17T04:00:16.185026+0000","last_became_active":"2024-09-17T04:00:14.175439+0000","last_became_peered":"2024-09-17T04:00:14.175439+0000","last_unstale":"2024-09-17T04:00:16.185026+0000","last_undegraded":"2024-09-17T04:00:16.185026+0000","last_fullsized":"2024-09-17T04:00:16.185026+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:12.498724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977474+0000","last_change":"2024-09-17T04:00:15.196798+0000","last_active":"2024-09-17T04:00:17.977474+0000","last_peered":"2024-09-17T04:00:17.977474+0000","last_clean":"2024-09-17T04:00:17.977474+0000","last_became_active":"2024-09-17T04:00:15.191801+0000","last_became_peered":"2024-09-17T04:00:15.191801+0000","last_unstale":"2024-09-17T04:00:17.977474+0000","last_undegraded":"2024-09-17T04:00:17.977474+0000","last_fullsized":"2024-09-17T04:00:17.977474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:14.033347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977378+0000","last_change":"2024-09-17T04:00:14.176595+0000","last_active":"2024-09-17T04:00:17.977378+0000","last_peered":"2024-09-17T04:00:17.977378+0000","last_clean":"2024-09-17T04:00:17.977378+0000","last_became_active":"2024-09-17T04:00:14.176223+0000","last_became_peered":"2024-09-17T04:00:14.176223+0000","last_unstale":"2024-09-17T04:00:17.977378+0000","last_undegraded":"2024-09-17T04:00:17.977378+0000","last_fullsized":"2024-09-17T04:00:17.977378+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:48.426501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354147+0000","last_change":"2024-09-17T04:00:15.207803+0000","last_active":"2024-09-17T04:00:17.354147+0000","last_peered":"2024-09-17T04:00:17.354147+0000","last_clean":"2024-09-17T04:00:17.354147+0000","last_became_active":"2024-09-17T04:00:15.207487+0000","last_became_peered":"2024-09-17T04:00:15.207487+0000","last_unstale":"2024-09-17T04:00:17.354147+0000","last_undegraded":"2024-09-17T04:00:17.354147+0000","last_fullsized":"2024-09-17T04:00:17.354147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:04.107520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181255+0000","last_change":"2024-09-17T04:00:14.185208+0000","last_active":"2024-09-17T04:00:17.181255+0000","last_peered":"2024-09-17T04:00:17.181255+0000","last_clean":"2024-09-17T04:00:17.181255+0000","last_became_active":"2024-09-17T04:00:14.184192+0000","last_became_peered":"2024-09-17T04:00:14.184192+0000","last_unstale":"2024-09-17T04:00:17.181255+0000","last_undegraded":"2024-09-17T04:00:17.181255+0000","last_fullsized":"2024-09-17T04:00:17.181255+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:58.627208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185221+0000","last_change":"2024-09-17T04:00:15.200216+0000","last_active":"2024-09-17T04:00:16.185221+0000","last_peered":"2024-09-17T04:00:16.185221+0000","last_clean":"2024-09-17T04:00:16.185221+0000","last_became_active":"2024-09-17T04:00:15.199102+0000","last_became_peered":"2024-09-17T04:00:15.199102+0000","last_unstale":"2024-09-17T04:00:16.185221+0000","last_undegraded":"2024-09-17T04:00:16.185221+0000","last_fullsized":"2024-09-17T04:00:16.185221+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:46:45.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978059+0000","last_change":"2024-09-17T04:00:14.179639+0000","last_active":"2024-09-17T04:00:17.978059+0000","last_peered":"2024-09-17T04:00:17.978059+0000","last_clean":"2024-09-17T04:00:17.978059+0000","last_became_active":"2024-09-17T04:00:14.179337+0000","last_became_peered":"2024-09-17T04:00:14.179337+0000","last_unstale":"2024-09-17T04:00:17.978059+0000","last_undegraded":"2024-09-17T04:00:17.978059+0000","last_fullsized":"2024-09-17T04:00:17.978059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":373,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977429+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:17.977429+0000","last_peered":"2024-09-17T04:00:17.977429+0000","last_clean":"2024-09-17T04:00:17.977429+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:17.977429+0000","last_undegraded":"2024-09-17T04:00:17.977429+0000","last_fullsized":"2024-09-17T04:00:17.977429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509573,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37932,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333908,"statfs":{"total":96636764160,"available":96597921792,"internally_reserved":0,"allocated":2646016,"data_stored":1484798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509576,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41504,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330336,"statfs":{"total":96636764160,"available":96594264064,"internally_reserved":0,"allocated":2043904,"data_stored":894326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41520,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330320,"statfs":{"total":96636764160,"available":96594247680,"internally_reserved":0,"allocated":2060288,"data_stored":896520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509573,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":894520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509575,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37356,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334484,"statfs":{"total":96636764160,"available":96598511616,"internally_reserved":0,"allocated":2056192,"data_stored":895020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":895938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542277,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41492,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330348,"statfs":{"total":96636764160,"available":96594276352,"internally_reserved":0,"allocated":2031616,"data_stored":888026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42092,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329748,"statfs":{"total":96636764160,"available":96593661952,"internally_reserved":0,"allocated":2641920,"data_stored":1484716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:20.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 -- 172.21.15.129:0/1970263025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f460415ce50 msgr2=0x7f46040af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 --2- 172.21.15.129:0/1970263025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f460415ce50 0x7f46040af7a0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f46040c3cf0 tx=0x7f45f402a560 comp rx=0 tx=0).stop 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 -- 172.21.15.129:0/1970263025 shutdown_connections 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 --2- 172.21.15.129:0/1970263025 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040ccc80 0x7f46040cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 --2- 172.21.15.129:0/1970263025 >> v2:172.21.15.181:3300/0 conn(0x7f46040b4180 0x7f46040ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 --2- 172.21.15.129:0/1970263025 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f460415ce50 0x7f46040af7a0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 -- 172.21.15.129:0/1970263025 >> 172.21.15.129:0/1970263025 conn(0x7f460406d410 msgr2=0x7f46040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:20.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 -- 172.21.15.129:0/1970263025 shutdown_connections 2024-09-17T04:00:20.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.847+0000 7f460a000640 1 -- 172.21.15.129:0/1970263025 wait complete. 2024-09-17T04:00:20.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.850+0000 7f460a000640 1 Processor -- start 2024-09-17T04:00:20.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.850+0000 7f460a000640 1 -- start start 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040b4180 0x7f46040ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 0x7f46040c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f460406f820 con 0x7f46040ccc80 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f460406f6a0 con 0x7f460415ce50 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f460a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f460406f9a0 con 0x7f46040b4180 2024-09-17T04:00:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f4602a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f4602a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35478/0 (socket says 172.21.15.129:35478) 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.851+0000 7f4602a00640 1 -- 172.21.15.129:0/2797903107 learned_addr learned my addr 172.21.15.129:0/2797903107 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4603400640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040b4180 0x7f46040ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4603e00640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 0x7f46040c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 -- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040b4180 msgr2=0x7f46040ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040b4180 0x7f46040ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 -- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 msgr2=0x7f46040c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 0x7f46040c78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 -- 172.21.15.129:0/2797903107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46041f40d0 con 0x7f46040ccc80 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4603e00640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 0x7f46040c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f4602a00640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f45f000d680 tx=0x7f45f000db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0017070 con 0x7f46040ccc80 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.852+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f45f0004310 con 0x7f46040ccc80 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.853+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0005280 con 0x7f46040ccc80 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.853+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46041f43b0 con 0x7f46040ccc80 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.853+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46041f4970 con 0x7f46040ccc80 2024-09-17T04:00:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.854+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f460406ff30 con 0x7f46040ccc80 2024-09-17T04:00:20.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.855+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f45f0005420 con 0x7f46040ccc80 2024-09-17T04:00:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.856+0000 7f45fb400640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 0x7f45e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.856+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f45f009b110 con 0x7f46040ccc80 2024-09-17T04:00:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.856+0000 7f4603400640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 0x7f45e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:20.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.857+0000 7f4603400640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 0x7f45e407b0d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f45f4033000 tx=0x7f45f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:20.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:20.860+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f45f0020020 con 0x7f46040ccc80 2024-09-17T04:00:21.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.113+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f46041f3e20 con 0x7f46040ccc80 2024-09-17T04:00:21.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.113+0000 7f45fb400640 1 -- 172.21.15.129:0/2797903107 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1561 (secure 0 0 0) 0x7f45f00634b0 con 0x7f46040ccc80 2024-09-17T04:00:21.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:21.114 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":672,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":163,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":54},{"state_name":"creating+peering","count":43}],"num_pgs":97,"num_pools":3,"num_objects":4,"data_bytes":590368,"bytes_used":332443648,"bytes_avail":772761669632,"bytes_total":773094113280,"inactive_pgs_ratio":0.44329896569252014,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","id":12,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":12,"rank":0,"name":"d","status":"up:active","gid":8542}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-09-17T04:00:02.472790+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 msgr2=0x7f45e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 0x7f45e407b0d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f45f4033000 tx=0x7f45f4004730 comp rx=0 tx=0).stop 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 msgr2=0x7f46040c2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f45f000d680 tx=0x7f45f000db40 comp rx=0 tx=0).stop 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 shutdown_connections 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f45e4078bc0 0x7f45e407b0d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f460415ce50 0x7f46040c78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46040ccc80 0x7f46040c2d10 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 --2- 172.21.15.129:0/2797903107 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46040b4180 0x7f46040ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 >> 172.21.15.129:0/2797903107 conn(0x7f460406d410 msgr2=0x7f46040ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:21.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 shutdown_connections 2024-09-17T04:00:21.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.116+0000 7f460a000640 1 -- 172.21.15.129:0/2797903107 wait complete. 2024-09-17T04:00:21.138 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:00:21.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 -- 172.21.15.129:0/2792838576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0d4a80 msgr2=0x7ffa1c0d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 --2- 172.21.15.129:0/2792838576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0d4a80 0x7ffa1c0d0fc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffa08005fd0 tx=0x7ffa0802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 -- 172.21.15.129:0/2792838576 shutdown_connections 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 --2- 172.21.15.129:0/2792838576 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0d4a80 0x7ffa1c0d0fc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 --2- 172.21.15.129:0/2792838576 >> v2:172.21.15.181:3300/0 conn(0x7ffa1c0b3220 0x7ffa1c0d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 -- 172.21.15.129:0/2792838576 >> 172.21.15.129:0/2792838576 conn(0x7ffa1c09d8d0 msgr2=0x7ffa1c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 -- 172.21.15.129:0/2792838576 shutdown_connections 2024-09-17T04:00:21.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.360+0000 7ffa1be00640 1 -- 172.21.15.129:0/2792838576 wait complete. 2024-09-17T04:00:21.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.363+0000 7ffa1be00640 1 Processor -- start 2024-09-17T04:00:21.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.363+0000 7ffa1be00640 1 -- start start 2024-09-17T04:00:21.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0b3220 0x7ffa1c1f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:21.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa1c0d1500 0x7ffa1c1f8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c0ba040 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1c0b9ec0 con 0x7ffa1c0d1500 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ffa1c0d0ae0 con 0x7ffa1c0b3220 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.364+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35488/0 (socket says 172.21.15.129:35488) 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 -- 172.21.15.129:0/1076199780 learned_addr learned my addr 172.21.15.129:0/1076199780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1aa00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0b3220 0x7ffa1c1f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:21.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1a000640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa1c0d1500 0x7ffa1c1f8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:21.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 -- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0b3220 msgr2=0x7ffa1c1f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0b3220 0x7ffa1c1f6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 -- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa1c0d1500 msgr2=0x7ffa1c1f8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa1c0d1500 0x7ffa1c1f8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 -- 172.21.15.129:0/1076199780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa1c1f9a20 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa1b400640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400c3c0 tx=0x7ffa1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.365+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14017070 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.366+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa1c1f9d00 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.366+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa140053c0 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.366+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14003850 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.366+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa1c1f5c40 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.367+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1c1f9790 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.368+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffa1401a070 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.369+0000 7ffa12a00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 0x7ff9fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:21.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.369+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7ffa1409a070 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.370+0000 7ffa1aa00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 0x7ff9fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:21.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.370+0000 7ffa1aa00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 0x7ff9fc07b0a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009920 tx=0x7ffa0c0094a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:21.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.373+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ffa1c0d4e90 con 0x7ffa1c0d4a80 2024-09-17T04:00:21.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.553+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ffa1c06a130 con 0x7ff9fc078b90 2024-09-17T04:00:21.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.558+0000 7ffa12a00640 1 -- 172.21.15.129:0/1076199780 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252490 (secure 0 0 0) 0x7ffa1c06a130 con 0x7ff9fc078b90 2024-09-17T04:00:21.563 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:21.564 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:21.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 msgr2=0x7ff9fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 0x7ff9fc07b0a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009920 tx=0x7ffa0c0094a0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 msgr2=0x7ffa1c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:21.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7ffa1400c3c0 tx=0x7ffa1400c880 comp rx=0 tx=0).stop 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 shutdown_connections 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff9fc078b90 0x7ff9fc07b0a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa1c0d4a80 0x7ffa1c158ae0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa1c0d1500 0x7ffa1c1f8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 --2- 172.21.15.129:0/1076199780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa1c0b3220 0x7ffa1c1f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 >> 172.21.15.129:0/1076199780 conn(0x7ffa1c09d8d0 msgr2=0x7ffa1c0d2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 shutdown_connections 2024-09-17T04:00:21.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:21.562+0000 7ffa1be00640 1 -- 172.21.15.129:0/1076199780 wait complete. 2024-09-17T04:00:21.584 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-17T04:00:20.477283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324928,"kb_used_data":17180,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754649792,"statfs":{"total":773094113280,"available":772761387008,"internally_reserved":0,"allocated":17592320,"data_stored":8333864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.322782"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978562+0000","last_change":"2024-09-17T04:00:15.205436+0000","last_active":"2024-09-17T04:00:17.978562+0000","last_peered":"2024-09-17T04:00:17.978562+0000","last_clean":"2024-09-17T04:00:17.978562+0000","last_became_active":"2024-09-17T04:00:15.205019+0000","last_became_peered":"2024-09-17T04:00:15.205019+0000","last_unstale":"2024-09-17T04:00:17.978562+0000","last_undegraded":"2024-09-17T04:00:17.978562+0000","last_fullsized":"2024-09-17T04:00:17.978562+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.197723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977616+0000","last_change":"2024-09-17T04:00:15.198110+0000","last_active":"2024-09-17T04:00:17.977616+0000","last_peered":"2024-09-17T04:00:17.977616+0000","last_clean":"2024-09-17T04:00:17.977616+0000","last_became_active":"2024-09-17T04:00:15.197746+0000","last_became_peered":"2024-09-17T04:00:15.197746+0000","last_unstale":"2024-09-17T04:00:17.977616+0000","last_undegraded":"2024-09-17T04:00:17.977616+0000","last_fullsized":"2024-09-17T04:00:17.977616+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181565+0000","last_change":"2024-09-17T04:00:15.195874+0000","last_active":"2024-09-17T04:00:17.181565+0000","last_peered":"2024-09-17T04:00:17.181565+0000","last_clean":"2024-09-17T04:00:17.181565+0000","last_became_active":"2024-09-17T04:00:15.194412+0000","last_became_peered":"2024-09-17T04:00:15.194412+0000","last_unstale":"2024-09-17T04:00:17.181565+0000","last_undegraded":"2024-09-17T04:00:17.181565+0000","last_fullsized":"2024-09-17T04:00:17.181565+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:11.985959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977850+0000","last_change":"2024-09-17T04:00:15.198044+0000","last_active":"2024-09-17T04:00:17.977850+0000","last_peered":"2024-09-17T04:00:17.977850+0000","last_clean":"2024-09-17T04:00:17.977850+0000","last_became_active":"2024-09-17T04:00:15.197647+0000","last_became_peered":"2024-09-17T04:00:15.197647+0000","last_unstale":"2024-09-17T04:00:17.977850+0000","last_undegraded":"2024-09-17T04:00:17.977850+0000","last_fullsized":"2024-09-17T04:00:17.977850+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.114514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354445+0000","last_change":"2024-09-17T04:00:15.206714+0000","last_active":"2024-09-17T04:00:17.354445+0000","last_peered":"2024-09-17T04:00:17.354445+0000","last_clean":"2024-09-17T04:00:17.354445+0000","last_became_active":"2024-09-17T04:00:15.206465+0000","last_became_peered":"2024-09-17T04:00:15.206465+0000","last_unstale":"2024-09-17T04:00:17.354445+0000","last_undegraded":"2024-09-17T04:00:17.354445+0000","last_fullsized":"2024-09-17T04:00:17.354445+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:29.233403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181368+0000","last_change":"2024-09-17T04:00:15.195270+0000","last_active":"2024-09-17T04:00:17.181368+0000","last_peered":"2024-09-17T04:00:17.181368+0000","last_clean":"2024-09-17T04:00:17.181368+0000","last_became_active":"2024-09-17T04:00:15.194918+0000","last_became_peered":"2024-09-17T04:00:15.194918+0000","last_unstale":"2024-09-17T04:00:17.181368+0000","last_undegraded":"2024-09-17T04:00:17.181368+0000","last_fullsized":"2024-09-17T04:00:17.181368+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:12.702219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186097+0000","last_change":"2024-09-17T04:00:15.201371+0000","last_active":"2024-09-17T04:00:16.186097+0000","last_peered":"2024-09-17T04:00:16.186097+0000","last_clean":"2024-09-17T04:00:16.186097+0000","last_became_active":"2024-09-17T04:00:15.200614+0000","last_became_peered":"2024-09-17T04:00:15.200614+0000","last_unstale":"2024-09-17T04:00:16.186097+0000","last_undegraded":"2024-09-17T04:00:16.186097+0000","last_fullsized":"2024-09-17T04:00:16.186097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:15.371085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976448+0000","last_change":"2024-09-17T04:00:15.179118+0000","last_active":"2024-09-17T04:00:17.976448+0000","last_peered":"2024-09-17T04:00:17.976448+0000","last_clean":"2024-09-17T04:00:17.976448+0000","last_became_active":"2024-09-17T04:00:15.178825+0000","last_became_peered":"2024-09-17T04:00:15.178825+0000","last_unstale":"2024-09-17T04:00:17.976448+0000","last_undegraded":"2024-09-17T04:00:17.976448+0000","last_fullsized":"2024-09-17T04:00:17.976448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:10.154034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185605+0000","last_change":"2024-09-17T04:00:15.196095+0000","last_active":"2024-09-17T04:00:16.185605+0000","last_peered":"2024-09-17T04:00:16.185605+0000","last_clean":"2024-09-17T04:00:16.185605+0000","last_became_active":"2024-09-17T04:00:15.195155+0000","last_became_peered":"2024-09-17T04:00:15.195155+0000","last_unstale":"2024-09-17T04:00:16.185605+0000","last_undegraded":"2024-09-17T04:00:16.185605+0000","last_fullsized":"2024-09-17T04:00:16.185605+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:35.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185181+0000","last_change":"2024-09-17T04:00:15.187116+0000","last_active":"2024-09-17T04:00:16.185181+0000","last_peered":"2024-09-17T04:00:16.185181+0000","last_clean":"2024-09-17T04:00:16.185181+0000","last_became_active":"2024-09-17T04:00:15.186491+0000","last_became_peered":"2024-09-17T04:00:15.186491+0000","last_unstale":"2024-09-17T04:00:16.185181+0000","last_undegraded":"2024-09-17T04:00:16.185181+0000","last_fullsized":"2024-09-17T04:00:16.185181+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:20.978854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186343+0000","last_change":"2024-09-17T04:00:15.192294+0000","last_active":"2024-09-17T04:00:16.186343+0000","last_peered":"2024-09-17T04:00:16.186343+0000","last_clean":"2024-09-17T04:00:16.186343+0000","last_became_active":"2024-09-17T04:00:15.191929+0000","last_became_peered":"2024-09-17T04:00:15.191929+0000","last_unstale":"2024-09-17T04:00:16.186343+0000","last_undegraded":"2024-09-17T04:00:16.186343+0000","last_fullsized":"2024-09-17T04:00:16.186343+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:02.758168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977913+0000","last_change":"2024-09-17T04:00:15.202361+0000","last_active":"2024-09-17T04:00:17.977913+0000","last_peered":"2024-09-17T04:00:17.977913+0000","last_clean":"2024-09-17T04:00:17.977913+0000","last_became_active":"2024-09-17T04:00:15.202042+0000","last_became_peered":"2024-09-17T04:00:15.202042+0000","last_unstale":"2024-09-17T04:00:17.977913+0000","last_undegraded":"2024-09-17T04:00:17.977913+0000","last_fullsized":"2024-09-17T04:00:17.977913+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:31.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977369+0000","last_change":"2024-09-17T04:00:15.205859+0000","last_active":"2024-09-17T04:00:17.977369+0000","last_peered":"2024-09-17T04:00:17.977369+0000","last_clean":"2024-09-17T04:00:17.977369+0000","last_became_active":"2024-09-17T04:00:15.205527+0000","last_became_peered":"2024-09-17T04:00:15.205527+0000","last_unstale":"2024-09-17T04:00:17.977369+0000","last_undegraded":"2024-09-17T04:00:17.977369+0000","last_fullsized":"2024-09-17T04:00:17.977369+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:02:13.228284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184876+0000","last_change":"2024-09-17T04:00:15.198473+0000","last_active":"2024-09-17T04:00:16.184876+0000","last_peered":"2024-09-17T04:00:16.184876+0000","last_clean":"2024-09-17T04:00:16.184876+0000","last_became_active":"2024-09-17T04:00:15.194432+0000","last_became_peered":"2024-09-17T04:00:15.194432+0000","last_unstale":"2024-09-17T04:00:16.184876+0000","last_undegraded":"2024-09-17T04:00:16.184876+0000","last_fullsized":"2024-09-17T04:00:16.184876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:07.494846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977556+0000","last_change":"2024-09-17T04:00:15.195580+0000","last_active":"2024-09-17T04:00:17.977556+0000","last_peered":"2024-09-17T04:00:17.977556+0000","last_clean":"2024-09-17T04:00:17.977556+0000","last_became_active":"2024-09-17T04:00:15.194025+0000","last_became_peered":"2024-09-17T04:00:15.194025+0000","last_unstale":"2024-09-17T04:00:17.977556+0000","last_undegraded":"2024-09-17T04:00:17.977556+0000","last_fullsized":"2024-09-17T04:00:17.977556+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:34.997014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976439+0000","last_change":"2024-09-17T04:00:15.188218+0000","last_active":"2024-09-17T04:00:17.976439+0000","last_peered":"2024-09-17T04:00:17.976439+0000","last_clean":"2024-09-17T04:00:17.976439+0000","last_became_active":"2024-09-17T04:00:15.187955+0000","last_became_peered":"2024-09-17T04:00:15.187955+0000","last_unstale":"2024-09-17T04:00:17.976439+0000","last_undegraded":"2024-09-17T04:00:17.976439+0000","last_fullsized":"2024-09-17T04:00:17.976439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354775+0000","last_change":"2024-09-17T04:00:15.208588+0000","last_active":"2024-09-17T04:00:17.354775+0000","last_peered":"2024-09-17T04:00:17.354775+0000","last_clean":"2024-09-17T04:00:17.354775+0000","last_became_active":"2024-09-17T04:00:15.208213+0000","last_became_peered":"2024-09-17T04:00:15.208213+0000","last_unstale":"2024-09-17T04:00:17.354775+0000","last_undegraded":"2024-09-17T04:00:17.354775+0000","last_fullsized":"2024-09-17T04:00:17.354775+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:53.003474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978563+0000","last_change":"2024-09-17T04:00:15.208085+0000","last_active":"2024-09-17T04:00:17.978563+0000","last_peered":"2024-09-17T04:00:17.978563+0000","last_clean":"2024-09-17T04:00:17.978563+0000","last_became_active":"2024-09-17T04:00:15.207762+0000","last_became_peered":"2024-09-17T04:00:15.207762+0000","last_unstale":"2024-09-17T04:00:17.978563+0000","last_undegraded":"2024-09-17T04:00:17.978563+0000","last_fullsized":"2024-09-17T04:00:17.978563+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184909+0000","last_change":"2024-09-17T04:00:15.185185+0000","last_active":"2024-09-17T04:00:16.184909+0000","last_peered":"2024-09-17T04:00:16.184909+0000","last_clean":"2024-09-17T04:00:16.184909+0000","last_became_active":"2024-09-17T04:00:15.182697+0000","last_became_peered":"2024-09-17T04:00:15.182697+0000","last_unstale":"2024-09-17T04:00:16.184909+0000","last_undegraded":"2024-09-17T04:00:16.184909+0000","last_fullsized":"2024-09-17T04:00:16.184909+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:06.451276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354394+0000","last_change":"2024-09-17T04:00:15.210019+0000","last_active":"2024-09-17T04:00:17.354394+0000","last_peered":"2024-09-17T04:00:17.354394+0000","last_clean":"2024-09-17T04:00:17.354394+0000","last_became_active":"2024-09-17T04:00:15.209795+0000","last_became_peered":"2024-09-17T04:00:15.209795+0000","last_unstale":"2024-09-17T04:00:17.354394+0000","last_undegraded":"2024-09-17T04:00:17.354394+0000","last_fullsized":"2024-09-17T04:00:17.354394+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:27.563680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354687+0000","last_change":"2024-09-17T04:00:15.206043+0000","last_active":"2024-09-17T04:00:17.354687+0000","last_peered":"2024-09-17T04:00:17.354687+0000","last_clean":"2024-09-17T04:00:17.354687+0000","last_became_active":"2024-09-17T04:00:15.205686+0000","last_became_peered":"2024-09-17T04:00:15.205686+0000","last_unstale":"2024-09-17T04:00:17.354687+0000","last_undegraded":"2024-09-17T04:00:17.354687+0000","last_fullsized":"2024-09-17T04:00:17.354687+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:40.936992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184888+0000","last_change":"2024-09-17T04:00:15.187015+0000","last_active":"2024-09-17T04:00:16.184888+0000","last_peered":"2024-09-17T04:00:16.184888+0000","last_clean":"2024-09-17T04:00:16.184888+0000","last_became_active":"2024-09-17T04:00:15.186574+0000","last_became_peered":"2024-09-17T04:00:15.186574+0000","last_unstale":"2024-09-17T04:00:16.184888+0000","last_undegraded":"2024-09-17T04:00:16.184888+0000","last_fullsized":"2024-09-17T04:00:16.184888+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:05.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181307+0000","last_change":"2024-09-17T04:00:15.195322+0000","last_active":"2024-09-17T04:00:17.181307+0000","last_peered":"2024-09-17T04:00:17.181307+0000","last_clean":"2024-09-17T04:00:17.181307+0000","last_became_active":"2024-09-17T04:00:15.194994+0000","last_became_peered":"2024-09-17T04:00:15.194994+0000","last_unstale":"2024-09-17T04:00:17.181307+0000","last_undegraded":"2024-09-17T04:00:17.181307+0000","last_fullsized":"2024-09-17T04:00:17.181307+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:36.557958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976714+0000","last_change":"2024-09-17T04:00:15.187550+0000","last_active":"2024-09-17T04:00:17.976714+0000","last_peered":"2024-09-17T04:00:17.976714+0000","last_clean":"2024-09-17T04:00:17.976714+0000","last_became_active":"2024-09-17T04:00:15.183952+0000","last_became_peered":"2024-09-17T04:00:15.183952+0000","last_unstale":"2024-09-17T04:00:17.976714+0000","last_undegraded":"2024-09-17T04:00:17.976714+0000","last_fullsized":"2024-09-17T04:00:17.976714+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:53.762374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977962+0000","last_change":"2024-09-17T04:00:15.196465+0000","last_active":"2024-09-17T04:00:17.977962+0000","last_peered":"2024-09-17T04:00:17.977962+0000","last_clean":"2024-09-17T04:00:17.977962+0000","last_became_active":"2024-09-17T04:00:15.195761+0000","last_became_peered":"2024-09-17T04:00:15.195761+0000","last_unstale":"2024-09-17T04:00:17.977962+0000","last_undegraded":"2024-09-17T04:00:17.977962+0000","last_fullsized":"2024-09-17T04:00:17.977962+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:29.557541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186879+0000","last_change":"2024-09-17T04:00:15.201293+0000","last_active":"2024-09-17T04:00:16.186879+0000","last_peered":"2024-09-17T04:00:16.186879+0000","last_clean":"2024-09-17T04:00:16.186879+0000","last_became_active":"2024-09-17T04:00:15.200722+0000","last_became_peered":"2024-09-17T04:00:15.200722+0000","last_unstale":"2024-09-17T04:00:16.186879+0000","last_undegraded":"2024-09-17T04:00:16.186879+0000","last_fullsized":"2024-09-17T04:00:16.186879+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:21.636014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181655+0000","last_change":"2024-09-17T04:00:15.193142+0000","last_active":"2024-09-17T04:00:17.181655+0000","last_peered":"2024-09-17T04:00:17.181655+0000","last_clean":"2024-09-17T04:00:17.181655+0000","last_became_active":"2024-09-17T04:00:15.192818+0000","last_became_peered":"2024-09-17T04:00:15.192818+0000","last_unstale":"2024-09-17T04:00:17.181655+0000","last_undegraded":"2024-09-17T04:00:17.181655+0000","last_fullsized":"2024-09-17T04:00:17.181655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:51.767875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354448+0000","last_change":"2024-09-17T04:00:15.196835+0000","last_active":"2024-09-17T04:00:17.354448+0000","last_peered":"2024-09-17T04:00:17.354448+0000","last_clean":"2024-09-17T04:00:17.354448+0000","last_became_active":"2024-09-17T04:00:15.196475+0000","last_became_peered":"2024-09-17T04:00:15.196475+0000","last_unstale":"2024-09-17T04:00:17.354448+0000","last_undegraded":"2024-09-17T04:00:17.354448+0000","last_fullsized":"2024-09-17T04:00:17.354448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:19.570741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977568+0000","last_change":"2024-09-17T04:00:15.203508+0000","last_active":"2024-09-17T04:00:17.977568+0000","last_peered":"2024-09-17T04:00:17.977568+0000","last_clean":"2024-09-17T04:00:17.977568+0000","last_became_active":"2024-09-17T04:00:15.203187+0000","last_became_peered":"2024-09-17T04:00:15.203187+0000","last_unstale":"2024-09-17T04:00:17.977568+0000","last_undegraded":"2024-09-17T04:00:17.977568+0000","last_fullsized":"2024-09-17T04:00:17.977568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:55.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184802+0000","last_change":"2024-09-17T04:00:15.194543+0000","last_active":"2024-09-17T04:00:16.184802+0000","last_peered":"2024-09-17T04:00:16.184802+0000","last_clean":"2024-09-17T04:00:16.184802+0000","last_became_active":"2024-09-17T04:00:15.194011+0000","last_became_peered":"2024-09-17T04:00:15.194011+0000","last_unstale":"2024-09-17T04:00:16.184802+0000","last_undegraded":"2024-09-17T04:00:16.184802+0000","last_fullsized":"2024-09-17T04:00:16.184802+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:25.507246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181366+0000","last_change":"2024-09-17T04:00:15.198425+0000","last_active":"2024-09-17T04:00:17.181366+0000","last_peered":"2024-09-17T04:00:17.181366+0000","last_clean":"2024-09-17T04:00:17.181366+0000","last_became_active":"2024-09-17T04:00:15.198098+0000","last_became_peered":"2024-09-17T04:00:15.198098+0000","last_unstale":"2024-09-17T04:00:17.181366+0000","last_undegraded":"2024-09-17T04:00:17.181366+0000","last_fullsized":"2024-09-17T04:00:17.181366+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:09.331234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354139+0000","last_change":"2024-09-17T04:00:15.210848+0000","last_active":"2024-09-17T04:00:17.354139+0000","last_peered":"2024-09-17T04:00:17.354139+0000","last_clean":"2024-09-17T04:00:17.354139+0000","last_became_active":"2024-09-17T04:00:15.210516+0000","last_became_peered":"2024-09-17T04:00:15.210516+0000","last_unstale":"2024-09-17T04:00:17.354139+0000","last_undegraded":"2024-09-17T04:00:17.354139+0000","last_fullsized":"2024-09-17T04:00:17.354139+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:55.037291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977006+0000","last_change":"2024-09-17T04:00:15.185994+0000","last_active":"2024-09-17T04:00:17.977006+0000","last_peered":"2024-09-17T04:00:17.977006+0000","last_clean":"2024-09-17T04:00:17.977006+0000","last_became_active":"2024-09-17T04:00:15.184966+0000","last_became_peered":"2024-09-17T04:00:15.184966+0000","last_unstale":"2024-09-17T04:00:17.977006+0000","last_undegraded":"2024-09-17T04:00:17.977006+0000","last_fullsized":"2024-09-17T04:00:17.977006+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:34.976945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"162'4","reported_seq":19,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354792+0000","last_change":"2024-09-17T04:00:14.184544+0000","last_active":"2024-09-17T04:00:17.354792+0000","last_peered":"2024-09-17T04:00:17.354792+0000","last_clean":"2024-09-17T04:00:17.354792+0000","last_became_active":"2024-09-17T04:00:14.184254+0000","last_became_peered":"2024-09-17T04:00:14.184254+0000","last_unstale":"2024-09-17T04:00:17.354792+0000","last_undegraded":"2024-09-17T04:00:17.354792+0000","last_fullsized":"2024-09-17T04:00:17.354792+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:15:30.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186087+0000","last_change":"2024-09-17T04:00:15.203100+0000","last_active":"2024-09-17T04:00:16.186087+0000","last_peered":"2024-09-17T04:00:16.186087+0000","last_clean":"2024-09-17T04:00:16.186087+0000","last_became_active":"2024-09-17T04:00:15.202011+0000","last_became_peered":"2024-09-17T04:00:15.202011+0000","last_unstale":"2024-09-17T04:00:16.186087+0000","last_undegraded":"2024-09-17T04:00:16.186087+0000","last_fullsized":"2024-09-17T04:00:16.186087+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:07.578261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185365+0000","last_change":"2024-09-17T04:00:14.178919+0000","last_active":"2024-09-17T04:00:16.185365+0000","last_peered":"2024-09-17T04:00:16.185365+0000","last_clean":"2024-09-17T04:00:16.185365+0000","last_became_active":"2024-09-17T04:00:14.177710+0000","last_became_peered":"2024-09-17T04:00:14.177710+0000","last_unstale":"2024-09-17T04:00:16.185365+0000","last_undegraded":"2024-09-17T04:00:16.185365+0000","last_fullsized":"2024-09-17T04:00:16.185365+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:18.564335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186429+0000","last_change":"2024-09-17T04:00:15.202175+0000","last_active":"2024-09-17T04:00:16.186429+0000","last_peered":"2024-09-17T04:00:16.186429+0000","last_clean":"2024-09-17T04:00:16.186429+0000","last_became_active":"2024-09-17T04:00:15.201873+0000","last_became_peered":"2024-09-17T04:00:15.201873+0000","last_unstale":"2024-09-17T04:00:16.186429+0000","last_undegraded":"2024-09-17T04:00:16.186429+0000","last_fullsized":"2024-09-17T04:00:16.186429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:23.847548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181775+0000","last_change":"2024-09-17T04:00:14.173566+0000","last_active":"2024-09-17T04:00:17.181775+0000","last_peered":"2024-09-17T04:00:17.181775+0000","last_clean":"2024-09-17T04:00:17.181775+0000","last_became_active":"2024-09-17T04:00:14.173201+0000","last_became_peered":"2024-09-17T04:00:14.173201+0000","last_unstale":"2024-09-17T04:00:17.181775+0000","last_undegraded":"2024-09-17T04:00:17.181775+0000","last_fullsized":"2024-09-17T04:00:17.181775+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:15.816175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185473+0000","last_change":"2024-09-17T04:00:15.199464+0000","last_active":"2024-09-17T04:00:16.185473+0000","last_peered":"2024-09-17T04:00:16.185473+0000","last_clean":"2024-09-17T04:00:16.185473+0000","last_became_active":"2024-09-17T04:00:15.199050+0000","last_became_peered":"2024-09-17T04:00:15.199050+0000","last_unstale":"2024-09-17T04:00:16.185473+0000","last_undegraded":"2024-09-17T04:00:16.185473+0000","last_fullsized":"2024-09-17T04:00:16.185473+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:33.024811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977943+0000","last_change":"2024-09-17T04:00:14.180751+0000","last_active":"2024-09-17T04:00:17.977943+0000","last_peered":"2024-09-17T04:00:17.977943+0000","last_clean":"2024-09-17T04:00:17.977943+0000","last_became_active":"2024-09-17T04:00:14.180400+0000","last_became_peered":"2024-09-17T04:00:14.180400+0000","last_unstale":"2024-09-17T04:00:17.977943+0000","last_undegraded":"2024-09-17T04:00:17.977943+0000","last_fullsized":"2024-09-17T04:00:17.977943+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:14.945761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185444+0000","last_change":"2024-09-17T04:00:15.183077+0000","last_active":"2024-09-17T04:00:16.185444+0000","last_peered":"2024-09-17T04:00:16.185444+0000","last_clean":"2024-09-17T04:00:16.185444+0000","last_became_active":"2024-09-17T04:00:15.182579+0000","last_became_peered":"2024-09-17T04:00:15.182579+0000","last_unstale":"2024-09-17T04:00:16.185444+0000","last_undegraded":"2024-09-17T04:00:16.185444+0000","last_fullsized":"2024-09-17T04:00:16.185444+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:32.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354752+0000","last_change":"2024-09-17T04:00:14.178007+0000","last_active":"2024-09-17T04:00:17.354752+0000","last_peered":"2024-09-17T04:00:17.354752+0000","last_clean":"2024-09-17T04:00:17.354752+0000","last_became_active":"2024-09-17T04:00:14.177397+0000","last_became_peered":"2024-09-17T04:00:14.177397+0000","last_unstale":"2024-09-17T04:00:17.354752+0000","last_undegraded":"2024-09-17T04:00:17.354752+0000","last_fullsized":"2024-09-17T04:00:17.354752+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:46.574189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976655+0000","last_change":"2024-09-17T04:00:15.184558+0000","last_active":"2024-09-17T04:00:17.976655+0000","last_peered":"2024-09-17T04:00:17.976655+0000","last_clean":"2024-09-17T04:00:17.976655+0000","last_became_active":"2024-09-17T04:00:15.181827+0000","last_became_peered":"2024-09-17T04:00:15.181827+0000","last_unstale":"2024-09-17T04:00:17.976655+0000","last_undegraded":"2024-09-17T04:00:17.976655+0000","last_fullsized":"2024-09-17T04:00:17.976655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:03.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976490+0000","last_change":"2024-09-17T04:00:14.181701+0000","last_active":"2024-09-17T04:00:17.976490+0000","last_peered":"2024-09-17T04:00:17.976490+0000","last_clean":"2024-09-17T04:00:17.976490+0000","last_became_active":"2024-09-17T04:00:14.180617+0000","last_became_peered":"2024-09-17T04:00:14.180617+0000","last_unstale":"2024-09-17T04:00:17.976490+0000","last_undegraded":"2024-09-17T04:00:17.976490+0000","last_fullsized":"2024-09-17T04:00:17.976490+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185980+0000","last_change":"2024-09-17T04:00:15.188106+0000","last_active":"2024-09-17T04:00:16.185980+0000","last_peered":"2024-09-17T04:00:16.185980+0000","last_clean":"2024-09-17T04:00:16.185980+0000","last_became_active":"2024-09-17T04:00:15.187755+0000","last_became_peered":"2024-09-17T04:00:15.187755+0000","last_unstale":"2024-09-17T04:00:16.185980+0000","last_undegraded":"2024-09-17T04:00:16.185980+0000","last_fullsized":"2024-09-17T04:00:16.185980+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:16.200716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976969+0000","last_change":"2024-09-17T04:00:14.180760+0000","last_active":"2024-09-17T04:00:17.976969+0000","last_peered":"2024-09-17T04:00:17.976969+0000","last_clean":"2024-09-17T04:00:17.976969+0000","last_became_active":"2024-09-17T04:00:14.180063+0000","last_became_peered":"2024-09-17T04:00:14.180063+0000","last_unstale":"2024-09-17T04:00:17.976969+0000","last_undegraded":"2024-09-17T04:00:17.976969+0000","last_fullsized":"2024-09-17T04:00:17.976969+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181748+0000","last_change":"2024-09-17T04:00:15.184353+0000","last_active":"2024-09-17T04:00:17.181748+0000","last_peered":"2024-09-17T04:00:17.181748+0000","last_clean":"2024-09-17T04:00:17.181748+0000","last_became_active":"2024-09-17T04:00:15.183526+0000","last_became_peered":"2024-09-17T04:00:15.183526+0000","last_unstale":"2024-09-17T04:00:17.181748+0000","last_undegraded":"2024-09-17T04:00:17.181748+0000","last_fullsized":"2024-09-17T04:00:17.181748+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:08.174367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186810+0000","last_change":"2024-09-17T04:00:14.170243+0000","last_active":"2024-09-17T04:00:16.186810+0000","last_peered":"2024-09-17T04:00:16.186810+0000","last_clean":"2024-09-17T04:00:16.186810+0000","last_became_active":"2024-09-17T04:00:14.169878+0000","last_became_peered":"2024-09-17T04:00:14.169878+0000","last_unstale":"2024-09-17T04:00:16.186810+0000","last_undegraded":"2024-09-17T04:00:16.186810+0000","last_fullsized":"2024-09-17T04:00:16.186810+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:35.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976963+0000","last_change":"2024-09-17T04:00:15.187262+0000","last_active":"2024-09-17T04:00:17.976963+0000","last_peered":"2024-09-17T04:00:17.976963+0000","last_clean":"2024-09-17T04:00:17.976963+0000","last_became_active":"2024-09-17T04:00:15.186910+0000","last_became_peered":"2024-09-17T04:00:15.186910+0000","last_unstale":"2024-09-17T04:00:17.976963+0000","last_undegraded":"2024-09-17T04:00:17.976963+0000","last_fullsized":"2024-09-17T04:00:17.976963+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:21.523618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354640+0000","last_change":"2024-09-17T04:00:14.178548+0000","last_active":"2024-09-17T04:00:17.354640+0000","last_peered":"2024-09-17T04:00:17.354640+0000","last_clean":"2024-09-17T04:00:17.354640+0000","last_became_active":"2024-09-17T04:00:14.178173+0000","last_became_peered":"2024-09-17T04:00:14.178173+0000","last_unstale":"2024-09-17T04:00:17.354640+0000","last_undegraded":"2024-09-17T04:00:17.354640+0000","last_fullsized":"2024-09-17T04:00:17.354640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:05.150582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186273+0000","last_change":"2024-09-17T04:00:15.190476+0000","last_active":"2024-09-17T04:00:16.186273+0000","last_peered":"2024-09-17T04:00:16.186273+0000","last_clean":"2024-09-17T04:00:16.186273+0000","last_became_active":"2024-09-17T04:00:15.189977+0000","last_became_peered":"2024-09-17T04:00:15.189977+0000","last_unstale":"2024-09-17T04:00:16.186273+0000","last_undegraded":"2024-09-17T04:00:16.186273+0000","last_fullsized":"2024-09-17T04:00:16.186273+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:26.037673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"162'3","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354310+0000","last_change":"2024-09-17T04:00:14.184902+0000","last_active":"2024-09-17T04:00:17.354310+0000","last_peered":"2024-09-17T04:00:17.354310+0000","last_clean":"2024-09-17T04:00:17.354310+0000","last_became_active":"2024-09-17T04:00:14.184624+0000","last_became_peered":"2024-09-17T04:00:14.184624+0000","last_unstale":"2024-09-17T04:00:17.354310+0000","last_undegraded":"2024-09-17T04:00:17.354310+0000","last_fullsized":"2024-09-17T04:00:17.354310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:08.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185090+0000","last_change":"2024-09-17T04:00:15.180729+0000","last_active":"2024-09-17T04:00:16.185090+0000","last_peered":"2024-09-17T04:00:16.185090+0000","last_clean":"2024-09-17T04:00:16.185090+0000","last_became_active":"2024-09-17T04:00:15.180358+0000","last_became_peered":"2024-09-17T04:00:15.180358+0000","last_unstale":"2024-09-17T04:00:16.185090+0000","last_undegraded":"2024-09-17T04:00:16.185090+0000","last_fullsized":"2024-09-17T04:00:16.185090+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:55.762158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"162'6","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181426+0000","last_change":"2024-09-17T04:00:14.177482+0000","last_active":"2024-09-17T04:00:17.181426+0000","last_peered":"2024-09-17T04:00:17.181426+0000","last_clean":"2024-09-17T04:00:17.181426+0000","last_became_active":"2024-09-17T04:00:14.177185+0000","last_became_peered":"2024-09-17T04:00:14.177185+0000","last_unstale":"2024-09-17T04:00:17.181426+0000","last_undegraded":"2024-09-17T04:00:17.181426+0000","last_fullsized":"2024-09-17T04:00:17.181426+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:59.681874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181467+0000","last_change":"2024-09-17T04:00:15.197664+0000","last_active":"2024-09-17T04:00:17.181467+0000","last_peered":"2024-09-17T04:00:17.181467+0000","last_clean":"2024-09-17T04:00:17.181467+0000","last_became_active":"2024-09-17T04:00:15.197330+0000","last_became_peered":"2024-09-17T04:00:15.197330+0000","last_unstale":"2024-09-17T04:00:17.181467+0000","last_undegraded":"2024-09-17T04:00:17.181467+0000","last_fullsized":"2024-09-17T04:00:17.181467+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:09.294262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978258+0000","last_change":"2024-09-17T04:00:14.180486+0000","last_active":"2024-09-17T04:00:17.978258+0000","last_peered":"2024-09-17T04:00:17.978258+0000","last_clean":"2024-09-17T04:00:17.978258+0000","last_became_active":"2024-09-17T04:00:14.180077+0000","last_became_peered":"2024-09-17T04:00:14.180077+0000","last_unstale":"2024-09-17T04:00:17.978258+0000","last_undegraded":"2024-09-17T04:00:17.978258+0000","last_fullsized":"2024-09-17T04:00:17.978258+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:01.024929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186622+0000","last_change":"2024-09-17T04:00:15.195036+0000","last_active":"2024-09-17T04:00:16.186622+0000","last_peered":"2024-09-17T04:00:16.186622+0000","last_clean":"2024-09-17T04:00:16.186622+0000","last_became_active":"2024-09-17T04:00:15.193366+0000","last_became_peered":"2024-09-17T04:00:15.193366+0000","last_unstale":"2024-09-17T04:00:16.186622+0000","last_undegraded":"2024-09-17T04:00:16.186622+0000","last_fullsized":"2024-09-17T04:00:16.186622+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:35.218017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185646+0000","last_change":"2024-09-17T04:00:14.179129+0000","last_active":"2024-09-17T04:00:16.185646+0000","last_peered":"2024-09-17T04:00:16.185646+0000","last_clean":"2024-09-17T04:00:16.185646+0000","last_became_active":"2024-09-17T04:00:14.177651+0000","last_became_peered":"2024-09-17T04:00:14.177651+0000","last_unstale":"2024-09-17T04:00:16.185646+0000","last_undegraded":"2024-09-17T04:00:16.185646+0000","last_fullsized":"2024-09-17T04:00:16.185646+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:50.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184621+0000","last_change":"2024-09-17T04:00:15.195850+0000","last_active":"2024-09-17T04:00:16.184621+0000","last_peered":"2024-09-17T04:00:16.184621+0000","last_clean":"2024-09-17T04:00:16.184621+0000","last_became_active":"2024-09-17T04:00:15.194823+0000","last_became_peered":"2024-09-17T04:00:15.194823+0000","last_unstale":"2024-09-17T04:00:16.184621+0000","last_undegraded":"2024-09-17T04:00:16.184621+0000","last_fullsized":"2024-09-17T04:00:16.184621+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:16.156594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976351+0000","last_change":"2024-09-17T04:00:14.178903+0000","last_active":"2024-09-17T04:00:17.976351+0000","last_peered":"2024-09-17T04:00:17.976351+0000","last_clean":"2024-09-17T04:00:17.976351+0000","last_became_active":"2024-09-17T04:00:14.178605+0000","last_became_peered":"2024-09-17T04:00:14.178605+0000","last_unstale":"2024-09-17T04:00:17.976351+0000","last_undegraded":"2024-09-17T04:00:17.976351+0000","last_fullsized":"2024-09-17T04:00:17.976351+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184829+0000","last_change":"2024-09-17T04:00:15.192316+0000","last_active":"2024-09-17T04:00:16.184829+0000","last_peered":"2024-09-17T04:00:16.184829+0000","last_clean":"2024-09-17T04:00:16.184829+0000","last_became_active":"2024-09-17T04:00:15.188357+0000","last_became_peered":"2024-09-17T04:00:15.188357+0000","last_unstale":"2024-09-17T04:00:16.184829+0000","last_undegraded":"2024-09-17T04:00:16.184829+0000","last_fullsized":"2024-09-17T04:00:16.184829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:34.996367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"162'8","reported_seq":23,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977302+0000","last_change":"2024-09-17T04:00:14.182987+0000","last_active":"2024-09-17T04:00:17.977302+0000","last_peered":"2024-09-17T04:00:17.977302+0000","last_clean":"2024-09-17T04:00:17.977302+0000","last_became_active":"2024-09-17T04:00:14.182665+0000","last_became_peered":"2024-09-17T04:00:14.182665+0000","last_unstale":"2024-09-17T04:00:17.977302+0000","last_undegraded":"2024-09-17T04:00:17.977302+0000","last_fullsized":"2024-09-17T04:00:17.977302+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185423+0000","last_change":"2024-09-17T04:00:15.188466+0000","last_active":"2024-09-17T04:00:16.185423+0000","last_peered":"2024-09-17T04:00:16.185423+0000","last_clean":"2024-09-17T04:00:16.185423+0000","last_became_active":"2024-09-17T04:00:15.187860+0000","last_became_peered":"2024-09-17T04:00:15.187860+0000","last_unstale":"2024-09-17T04:00:16.185423+0000","last_undegraded":"2024-09-17T04:00:16.185423+0000","last_fullsized":"2024-09-17T04:00:16.185423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:05.989755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354873+0000","last_change":"2024-09-17T04:00:14.179849+0000","last_active":"2024-09-17T04:00:17.354873+0000","last_peered":"2024-09-17T04:00:17.354873+0000","last_clean":"2024-09-17T04:00:17.354873+0000","last_became_active":"2024-09-17T04:00:14.178084+0000","last_became_peered":"2024-09-17T04:00:14.178084+0000","last_unstale":"2024-09-17T04:00:17.354873+0000","last_undegraded":"2024-09-17T04:00:17.354873+0000","last_fullsized":"2024-09-17T04:00:17.354873+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:26.618084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185623+0000","last_change":"2024-09-17T04:00:15.190551+0000","last_active":"2024-09-17T04:00:16.185623+0000","last_peered":"2024-09-17T04:00:16.185623+0000","last_clean":"2024-09-17T04:00:16.185623+0000","last_became_active":"2024-09-17T04:00:15.190147+0000","last_became_peered":"2024-09-17T04:00:15.190147+0000","last_unstale":"2024-09-17T04:00:16.185623+0000","last_undegraded":"2024-09-17T04:00:16.185623+0000","last_fullsized":"2024-09-17T04:00:16.185623+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:00.814493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977169+0000","last_change":"2024-09-17T04:00:14.173867+0000","last_active":"2024-09-17T04:00:17.977169+0000","last_peered":"2024-09-17T04:00:17.977169+0000","last_clean":"2024-09-17T04:00:17.977169+0000","last_became_active":"2024-09-17T04:00:14.171614+0000","last_became_peered":"2024-09-17T04:00:14.171614+0000","last_unstale":"2024-09-17T04:00:17.977169+0000","last_undegraded":"2024-09-17T04:00:17.977169+0000","last_fullsized":"2024-09-17T04:00:17.977169+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185175+0000","last_change":"2024-09-17T04:00:15.196127+0000","last_active":"2024-09-17T04:00:16.185175+0000","last_peered":"2024-09-17T04:00:16.185175+0000","last_clean":"2024-09-17T04:00:16.185175+0000","last_became_active":"2024-09-17T04:00:15.195799+0000","last_became_peered":"2024-09-17T04:00:15.195799+0000","last_unstale":"2024-09-17T04:00:16.185175+0000","last_undegraded":"2024-09-17T04:00:16.185175+0000","last_fullsized":"2024-09-17T04:00:16.185175+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:45:34.672205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978453+0000","last_change":"2024-09-17T04:00:14.183120+0000","last_active":"2024-09-17T04:00:17.978453+0000","last_peered":"2024-09-17T04:00:17.978453+0000","last_clean":"2024-09-17T04:00:17.978453+0000","last_became_active":"2024-09-17T04:00:14.182779+0000","last_became_peered":"2024-09-17T04:00:14.182779+0000","last_unstale":"2024-09-17T04:00:17.978453+0000","last_undegraded":"2024-09-17T04:00:17.978453+0000","last_fullsized":"2024-09-17T04:00:17.978453+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977807+0000","last_change":"2024-09-17T04:00:15.204420+0000","last_active":"2024-09-17T04:00:17.977807+0000","last_peered":"2024-09-17T04:00:17.977807+0000","last_clean":"2024-09-17T04:00:17.977807+0000","last_became_active":"2024-09-17T04:00:15.202153+0000","last_became_peered":"2024-09-17T04:00:15.202153+0000","last_unstale":"2024-09-17T04:00:17.977807+0000","last_undegraded":"2024-09-17T04:00:17.977807+0000","last_fullsized":"2024-09-17T04:00:17.977807+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:57.466014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186468+0000","last_change":"2024-09-17T04:00:14.164836+0000","last_active":"2024-09-17T04:00:16.186468+0000","last_peered":"2024-09-17T04:00:16.186468+0000","last_clean":"2024-09-17T04:00:16.186468+0000","last_became_active":"2024-09-17T04:00:14.164491+0000","last_became_peered":"2024-09-17T04:00:14.164491+0000","last_unstale":"2024-09-17T04:00:16.186468+0000","last_undegraded":"2024-09-17T04:00:16.186468+0000","last_fullsized":"2024-09-17T04:00:16.186468+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:53:35.894685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185709+0000","last_change":"2024-09-17T04:00:15.203883+0000","last_active":"2024-09-17T04:00:16.185709+0000","last_peered":"2024-09-17T04:00:16.185709+0000","last_clean":"2024-09-17T04:00:16.185709+0000","last_became_active":"2024-09-17T04:00:15.201414+0000","last_became_peered":"2024-09-17T04:00:15.201414+0000","last_unstale":"2024-09-17T04:00:16.185709+0000","last_undegraded":"2024-09-17T04:00:16.185709+0000","last_fullsized":"2024-09-17T04:00:16.185709+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:29.778689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181504+0000","last_change":"2024-09-17T04:00:14.185714+0000","last_active":"2024-09-17T04:00:17.181504+0000","last_peered":"2024-09-17T04:00:17.181504+0000","last_clean":"2024-09-17T04:00:17.181504+0000","last_became_active":"2024-09-17T04:00:14.185363+0000","last_became_peered":"2024-09-17T04:00:14.185363+0000","last_unstale":"2024-09-17T04:00:17.181504+0000","last_undegraded":"2024-09-17T04:00:17.181504+0000","last_fullsized":"2024-09-17T04:00:17.181504+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.866011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186668+0000","last_change":"2024-09-17T04:00:15.179231+0000","last_active":"2024-09-17T04:00:16.186668+0000","last_peered":"2024-09-17T04:00:16.186668+0000","last_clean":"2024-09-17T04:00:16.186668+0000","last_became_active":"2024-09-17T04:00:15.178887+0000","last_became_peered":"2024-09-17T04:00:15.178887+0000","last_unstale":"2024-09-17T04:00:16.186668+0000","last_undegraded":"2024-09-17T04:00:16.186668+0000","last_fullsized":"2024-09-17T04:00:16.186668+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:32.045406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354175+0000","last_change":"2024-09-17T04:00:14.183023+0000","last_active":"2024-09-17T04:00:17.354175+0000","last_peered":"2024-09-17T04:00:17.354175+0000","last_clean":"2024-09-17T04:00:17.354175+0000","last_became_active":"2024-09-17T04:00:14.178678+0000","last_became_peered":"2024-09-17T04:00:14.178678+0000","last_unstale":"2024-09-17T04:00:17.354175+0000","last_undegraded":"2024-09-17T04:00:17.354175+0000","last_fullsized":"2024-09-17T04:00:17.354175+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:09.710450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977340+0000","last_change":"2024-09-17T04:00:15.202518+0000","last_active":"2024-09-17T04:00:17.977340+0000","last_peered":"2024-09-17T04:00:17.977340+0000","last_clean":"2024-09-17T04:00:17.977340+0000","last_became_active":"2024-09-17T04:00:15.202240+0000","last_became_peered":"2024-09-17T04:00:15.202240+0000","last_unstale":"2024-09-17T04:00:17.977340+0000","last_undegraded":"2024-09-17T04:00:17.977340+0000","last_fullsized":"2024-09-17T04:00:17.977340+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:59.197974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181510+0000","last_change":"2024-09-17T04:00:14.178227+0000","last_active":"2024-09-17T04:00:17.181510+0000","last_peered":"2024-09-17T04:00:17.181510+0000","last_clean":"2024-09-17T04:00:17.181510+0000","last_became_active":"2024-09-17T04:00:14.177922+0000","last_became_peered":"2024-09-17T04:00:14.177922+0000","last_unstale":"2024-09-17T04:00:17.181510+0000","last_undegraded":"2024-09-17T04:00:17.181510+0000","last_fullsized":"2024-09-17T04:00:17.181510+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:49.911464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354137+0000","last_change":"2024-09-17T04:00:15.200391+0000","last_active":"2024-09-17T04:00:17.354137+0000","last_peered":"2024-09-17T04:00:17.354137+0000","last_clean":"2024-09-17T04:00:17.354137+0000","last_became_active":"2024-09-17T04:00:15.200095+0000","last_became_peered":"2024-09-17T04:00:15.200095+0000","last_unstale":"2024-09-17T04:00:17.354137+0000","last_undegraded":"2024-09-17T04:00:17.354137+0000","last_fullsized":"2024-09-17T04:00:17.354137+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:18:34.896086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184666+0000","last_change":"2024-09-17T04:00:14.173438+0000","last_active":"2024-09-17T04:00:16.184666+0000","last_peered":"2024-09-17T04:00:16.184666+0000","last_clean":"2024-09-17T04:00:16.184666+0000","last_became_active":"2024-09-17T04:00:14.173090+0000","last_became_peered":"2024-09-17T04:00:14.173090+0000","last_unstale":"2024-09-17T04:00:16.184666+0000","last_undegraded":"2024-09-17T04:00:16.184666+0000","last_fullsized":"2024-09-17T04:00:16.184666+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:21.951204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354500+0000","last_change":"2024-09-17T04:00:15.206400+0000","last_active":"2024-09-17T04:00:17.354500+0000","last_peered":"2024-09-17T04:00:17.354500+0000","last_clean":"2024-09-17T04:00:17.354500+0000","last_became_active":"2024-09-17T04:00:15.205707+0000","last_became_peered":"2024-09-17T04:00:15.205707+0000","last_unstale":"2024-09-17T04:00:17.354500+0000","last_undegraded":"2024-09-17T04:00:17.354500+0000","last_fullsized":"2024-09-17T04:00:17.354500+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:45:36.571087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978691+0000","last_change":"2024-09-17T04:00:14.186855+0000","last_active":"2024-09-17T04:00:17.978691+0000","last_peered":"2024-09-17T04:00:17.978691+0000","last_clean":"2024-09-17T04:00:17.978691+0000","last_became_active":"2024-09-17T04:00:14.186535+0000","last_became_peered":"2024-09-17T04:00:14.186535+0000","last_unstale":"2024-09-17T04:00:17.978691+0000","last_undegraded":"2024-09-17T04:00:17.978691+0000","last_fullsized":"2024-09-17T04:00:17.978691+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:32.914832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186153+0000","last_change":"2024-09-17T04:00:15.194286+0000","last_active":"2024-09-17T04:00:16.186153+0000","last_peered":"2024-09-17T04:00:16.186153+0000","last_clean":"2024-09-17T04:00:16.186153+0000","last_became_active":"2024-09-17T04:00:15.193936+0000","last_became_peered":"2024-09-17T04:00:15.193936+0000","last_unstale":"2024-09-17T04:00:16.186153+0000","last_undegraded":"2024-09-17T04:00:16.186153+0000","last_fullsized":"2024-09-17T04:00:16.186153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:03.221443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"163'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185534+0000","last_change":"2024-09-17T04:00:14.176145+0000","last_active":"2024-09-17T04:00:16.185534+0000","last_peered":"2024-09-17T04:00:16.185534+0000","last_clean":"2024-09-17T04:00:16.185534+0000","last_became_active":"2024-09-17T04:00:14.175839+0000","last_became_peered":"2024-09-17T04:00:14.175839+0000","last_unstale":"2024-09-17T04:00:16.185534+0000","last_undegraded":"2024-09-17T04:00:16.185534+0000","last_fullsized":"2024-09-17T04:00:16.185534+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:33.949960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976660+0000","last_change":"2024-09-17T04:00:15.181165+0000","last_active":"2024-09-17T04:00:17.976660+0000","last_peered":"2024-09-17T04:00:17.976660+0000","last_clean":"2024-09-17T04:00:17.976660+0000","last_became_active":"2024-09-17T04:00:15.180136+0000","last_became_peered":"2024-09-17T04:00:15.180136+0000","last_unstale":"2024-09-17T04:00:17.976660+0000","last_undegraded":"2024-09-17T04:00:17.976660+0000","last_fullsized":"2024-09-17T04:00:17.976660+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:28.049513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184972+0000","last_change":"2024-09-17T04:00:14.182944+0000","last_active":"2024-09-17T04:00:16.184972+0000","last_peered":"2024-09-17T04:00:16.184972+0000","last_clean":"2024-09-17T04:00:16.184972+0000","last_became_active":"2024-09-17T04:00:14.182626+0000","last_became_peered":"2024-09-17T04:00:14.182626+0000","last_unstale":"2024-09-17T04:00:16.184972+0000","last_undegraded":"2024-09-17T04:00:16.184972+0000","last_fullsized":"2024-09-17T04:00:16.184972+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:33.722613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976876+0000","last_change":"2024-09-17T04:00:15.185309+0000","last_active":"2024-09-17T04:00:17.976876+0000","last_peered":"2024-09-17T04:00:17.976876+0000","last_clean":"2024-09-17T04:00:17.976876+0000","last_became_active":"2024-09-17T04:00:15.184979+0000","last_became_peered":"2024-09-17T04:00:15.184979+0000","last_unstale":"2024-09-17T04:00:17.976876+0000","last_undegraded":"2024-09-17T04:00:17.976876+0000","last_fullsized":"2024-09-17T04:00:17.976876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:29.123723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976765+0000","last_change":"2024-09-17T04:00:14.172333+0000","last_active":"2024-09-17T04:00:17.976765+0000","last_peered":"2024-09-17T04:00:17.976765+0000","last_clean":"2024-09-17T04:00:17.976765+0000","last_became_active":"2024-09-17T04:00:14.170300+0000","last_became_peered":"2024-09-17T04:00:14.170300+0000","last_unstale":"2024-09-17T04:00:17.976765+0000","last_undegraded":"2024-09-17T04:00:17.976765+0000","last_fullsized":"2024-09-17T04:00:17.976765+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977992+0000","last_change":"2024-09-17T04:00:15.201456+0000","last_active":"2024-09-17T04:00:17.977992+0000","last_peered":"2024-09-17T04:00:17.977992+0000","last_clean":"2024-09-17T04:00:17.977992+0000","last_became_active":"2024-09-17T04:00:15.201067+0000","last_became_peered":"2024-09-17T04:00:15.201067+0000","last_unstale":"2024-09-17T04:00:17.977992+0000","last_undegraded":"2024-09-17T04:00:17.977992+0000","last_fullsized":"2024-09-17T04:00:17.977992+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:26.401458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184947+0000","last_change":"2024-09-17T04:00:14.172200+0000","last_active":"2024-09-17T04:00:16.184947+0000","last_peered":"2024-09-17T04:00:16.184947+0000","last_clean":"2024-09-17T04:00:16.184947+0000","last_became_active":"2024-09-17T04:00:14.171692+0000","last_became_peered":"2024-09-17T04:00:14.171692+0000","last_unstale":"2024-09-17T04:00:16.184947+0000","last_undegraded":"2024-09-17T04:00:16.184947+0000","last_fullsized":"2024-09-17T04:00:16.184947+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:39.246219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978058+0000","last_change":"2024-09-17T04:00:15.207451+0000","last_active":"2024-09-17T04:00:17.978058+0000","last_peered":"2024-09-17T04:00:17.978058+0000","last_clean":"2024-09-17T04:00:17.978058+0000","last_became_active":"2024-09-17T04:00:15.207157+0000","last_became_peered":"2024-09-17T04:00:15.207157+0000","last_unstale":"2024-09-17T04:00:17.978058+0000","last_undegraded":"2024-09-17T04:00:17.978058+0000","last_fullsized":"2024-09-17T04:00:17.978058+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:33:09.116340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185026+0000","last_change":"2024-09-17T04:00:14.175785+0000","last_active":"2024-09-17T04:00:16.185026+0000","last_peered":"2024-09-17T04:00:16.185026+0000","last_clean":"2024-09-17T04:00:16.185026+0000","last_became_active":"2024-09-17T04:00:14.175439+0000","last_became_peered":"2024-09-17T04:00:14.175439+0000","last_unstale":"2024-09-17T04:00:16.185026+0000","last_undegraded":"2024-09-17T04:00:16.185026+0000","last_fullsized":"2024-09-17T04:00:16.185026+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:12.498724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977474+0000","last_change":"2024-09-17T04:00:15.196798+0000","last_active":"2024-09-17T04:00:17.977474+0000","last_peered":"2024-09-17T04:00:17.977474+0000","last_clean":"2024-09-17T04:00:17.977474+0000","last_became_active":"2024-09-17T04:00:15.191801+0000","last_became_peered":"2024-09-17T04:00:15.191801+0000","last_unstale":"2024-09-17T04:00:17.977474+0000","last_undegraded":"2024-09-17T04:00:17.977474+0000","last_fullsized":"2024-09-17T04:00:17.977474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:14.033347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977378+0000","last_change":"2024-09-17T04:00:14.176595+0000","last_active":"2024-09-17T04:00:17.977378+0000","last_peered":"2024-09-17T04:00:17.977378+0000","last_clean":"2024-09-17T04:00:17.977378+0000","last_became_active":"2024-09-17T04:00:14.176223+0000","last_became_peered":"2024-09-17T04:00:14.176223+0000","last_unstale":"2024-09-17T04:00:17.977378+0000","last_undegraded":"2024-09-17T04:00:17.977378+0000","last_fullsized":"2024-09-17T04:00:17.977378+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:48.426501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354147+0000","last_change":"2024-09-17T04:00:15.207803+0000","last_active":"2024-09-17T04:00:17.354147+0000","last_peered":"2024-09-17T04:00:17.354147+0000","last_clean":"2024-09-17T04:00:17.354147+0000","last_became_active":"2024-09-17T04:00:15.207487+0000","last_became_peered":"2024-09-17T04:00:15.207487+0000","last_unstale":"2024-09-17T04:00:17.354147+0000","last_undegraded":"2024-09-17T04:00:17.354147+0000","last_fullsized":"2024-09-17T04:00:17.354147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:04.107520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181255+0000","last_change":"2024-09-17T04:00:14.185208+0000","last_active":"2024-09-17T04:00:17.181255+0000","last_peered":"2024-09-17T04:00:17.181255+0000","last_clean":"2024-09-17T04:00:17.181255+0000","last_became_active":"2024-09-17T04:00:14.184192+0000","last_became_peered":"2024-09-17T04:00:14.184192+0000","last_unstale":"2024-09-17T04:00:17.181255+0000","last_undegraded":"2024-09-17T04:00:17.181255+0000","last_fullsized":"2024-09-17T04:00:17.181255+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:58.627208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185221+0000","last_change":"2024-09-17T04:00:15.200216+0000","last_active":"2024-09-17T04:00:16.185221+0000","last_peered":"2024-09-17T04:00:16.185221+0000","last_clean":"2024-09-17T04:00:16.185221+0000","last_became_active":"2024-09-17T04:00:15.199102+0000","last_became_peered":"2024-09-17T04:00:15.199102+0000","last_unstale":"2024-09-17T04:00:16.185221+0000","last_undegraded":"2024-09-17T04:00:16.185221+0000","last_fullsized":"2024-09-17T04:00:16.185221+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:46:45.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978059+0000","last_change":"2024-09-17T04:00:14.179639+0000","last_active":"2024-09-17T04:00:17.978059+0000","last_peered":"2024-09-17T04:00:17.978059+0000","last_clean":"2024-09-17T04:00:17.978059+0000","last_became_active":"2024-09-17T04:00:14.179337+0000","last_became_peered":"2024-09-17T04:00:14.179337+0000","last_unstale":"2024-09-17T04:00:17.978059+0000","last_undegraded":"2024-09-17T04:00:17.978059+0000","last_fullsized":"2024-09-17T04:00:17.978059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":373,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977429+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:17.977429+0000","last_peered":"2024-09-17T04:00:17.977429+0000","last_clean":"2024-09-17T04:00:17.977429+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:17.977429+0000","last_undegraded":"2024-09-17T04:00:17.977429+0000","last_fullsized":"2024-09-17T04:00:17.977429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509573,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37932,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333908,"statfs":{"total":96636764160,"available":96597921792,"internally_reserved":0,"allocated":2646016,"data_stored":1484798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509576,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41504,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330336,"statfs":{"total":96636764160,"available":96594264064,"internally_reserved":0,"allocated":2043904,"data_stored":894326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509575,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41520,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330320,"statfs":{"total":96636764160,"available":96594247680,"internally_reserved":0,"allocated":2060288,"data_stored":896520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509573,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":894520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509575,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37356,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334484,"statfs":{"total":96636764160,"available":96598511616,"internally_reserved":0,"allocated":2056192,"data_stored":895020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":895938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542277,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41492,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330348,"statfs":{"total":96636764160,"available":96594276352,"internally_reserved":0,"allocated":2031616,"data_stored":888026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542278,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42092,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329748,"statfs":{"total":96636764160,"available":96593661952,"internally_reserved":0,"allocated":2641920,"data_stored":1484716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:24.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 -- 172.21.15.129:0/2118946429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0ce170 msgr2=0x7f6a7c0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 --2- 172.21.15.129:0/2118946429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0ce170 0x7f6a7c0cbbf0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c072360 tx=0x7f6a7002a560 comp rx=0 tx=0).stop 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 -- 172.21.15.129:0/2118946429 shutdown_connections 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 --2- 172.21.15.129:0/2118946429 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0b39c0 0x7f6a7c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 --2- 172.21.15.129:0/2118946429 >> v2:172.21.15.181:3300/0 conn(0x7f6a7c0b2950 0x7f6a7c0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 --2- 172.21.15.129:0/2118946429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0ce170 0x7f6a7c0cbbf0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:24.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.820+0000 7f6a83c00640 1 -- 172.21.15.129:0/2118946429 >> 172.21.15.129:0/2118946429 conn(0x7f6a7c06d410 msgr2=0x7f6a7c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:24.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.821+0000 7f6a83c00640 1 -- 172.21.15.129:0/2118946429 shutdown_connections 2024-09-17T04:00:24.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.821+0000 7f6a83c00640 1 -- 172.21.15.129:0/2118946429 wait complete. 2024-09-17T04:00:24.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.823+0000 7f6a83c00640 1 Processor -- start 2024-09-17T04:00:24.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.823+0000 7f6a83c00640 1 -- start start 2024-09-17T04:00:24.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.824+0000 7f6a83c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7c0b2950 0x7f6a7c1e5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.824+0000 7f6a83c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.824+0000 7f6a83c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0ce170 0x7f6a7c09c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.824+0000 7f6a83c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c070850 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a83c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7c0706d0 con 0x7f6a7c0b2950 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a83c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6a7c06f410 con 0x7f6a7c0ce170 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36082/0 (socket says 172.21.15.129:36082) 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 -- 172.21.15.129:0/1856457238 learned_addr learned my addr 172.21.15.129:0/1856457238 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a81c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0ce170 0x7f6a7c09c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a81200640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7c0b2950 0x7f6a7c1e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:24.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 -- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0ce170 msgr2=0x7f6a7c09c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0ce170 0x7f6a7c09c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 -- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7c0b2950 msgr2=0x7f6a7c1e5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7c0b2950 0x7f6a7c1e5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 -- 172.21.15.129:0/1856457238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7c1e5eb0 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.825+0000 7f6a7be00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c970 tx=0x7f6a6800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.826+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a68007bb0 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.826+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6a68007e40 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.826+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6800f6b0 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.826+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7c09e060 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.826+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7c09b500 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.827+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a7c06f900 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.828+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6a6800f850 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.830+0000 7f6a78c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 0x7f6a5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:24.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.830+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f6a6809b3e0 con 0x7f6a7c0b39c0 2024-09-17T04:00:24.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.830+0000 7f6a81200640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 0x7f6a5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:24.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.831+0000 7f6a81200640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 0x7f6a5c07b0a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6a70033000 tx=0x7f6a70004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:24.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:24.835+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6a68014030 con 0x7f6a7c0b39c0 2024-09-17T04:00:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.015+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6a7c06a140 con 0x7f6a5c078b90 2024-09-17T04:00:25.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.020+0000 7f6a78c00640 1 -- 172.21.15.129:0/1856457238 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252414 (secure 0 0 0) 0x7f6a7c06a140 con 0x7f6a5c078b90 2024-09-17T04:00:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:25.024 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 msgr2=0x7f6a5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 0x7f6a5c07b0a0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f6a70033000 tx=0x7f6a70004730 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 msgr2=0x7f6a7c09bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f6a6800c970 tx=0x7f6a6800ce30 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 shutdown_connections 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6a5c078b90 0x7f6a5c07b0a0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6a7c0ce170 0x7f6a7c09c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6a7c0b39c0 0x7f6a7c09bfe0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 --2- 172.21.15.129:0/1856457238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a7c0b2950 0x7f6a7c1e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 >> 172.21.15.129:0/1856457238 conn(0x7f6a7c06d410 msgr2=0x7f6a7c073f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:25.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 shutdown_connections 2024-09-17T04:00:25.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.024+0000 7f6a83c00640 1 -- 172.21.15.129:0/1856457238 wait complete. 2024-09-17T04:00:25.049 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-17T04:00:24.478598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":324940,"kb_used_data":17192,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754649780,"statfs":{"total":773094113280,"available":772761374720,"internally_reserved":0,"allocated":17604608,"data_stored":8340164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.308775"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978562+0000","last_change":"2024-09-17T04:00:15.205436+0000","last_active":"2024-09-17T04:00:17.978562+0000","last_peered":"2024-09-17T04:00:17.978562+0000","last_clean":"2024-09-17T04:00:17.978562+0000","last_became_active":"2024-09-17T04:00:15.205019+0000","last_became_peered":"2024-09-17T04:00:15.205019+0000","last_unstale":"2024-09-17T04:00:17.978562+0000","last_undegraded":"2024-09-17T04:00:17.978562+0000","last_fullsized":"2024-09-17T04:00:17.978562+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:44.197723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977616+0000","last_change":"2024-09-17T04:00:15.198110+0000","last_active":"2024-09-17T04:00:17.977616+0000","last_peered":"2024-09-17T04:00:17.977616+0000","last_clean":"2024-09-17T04:00:17.977616+0000","last_became_active":"2024-09-17T04:00:15.197746+0000","last_became_peered":"2024-09-17T04:00:15.197746+0000","last_unstale":"2024-09-17T04:00:17.977616+0000","last_undegraded":"2024-09-17T04:00:17.977616+0000","last_fullsized":"2024-09-17T04:00:17.977616+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:13:01.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181565+0000","last_change":"2024-09-17T04:00:15.195874+0000","last_active":"2024-09-17T04:00:17.181565+0000","last_peered":"2024-09-17T04:00:17.181565+0000","last_clean":"2024-09-17T04:00:17.181565+0000","last_became_active":"2024-09-17T04:00:15.194412+0000","last_became_peered":"2024-09-17T04:00:15.194412+0000","last_unstale":"2024-09-17T04:00:17.181565+0000","last_undegraded":"2024-09-17T04:00:17.181565+0000","last_fullsized":"2024-09-17T04:00:17.181565+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:11.985959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977850+0000","last_change":"2024-09-17T04:00:15.198044+0000","last_active":"2024-09-17T04:00:17.977850+0000","last_peered":"2024-09-17T04:00:17.977850+0000","last_clean":"2024-09-17T04:00:17.977850+0000","last_became_active":"2024-09-17T04:00:15.197647+0000","last_became_peered":"2024-09-17T04:00:15.197647+0000","last_unstale":"2024-09-17T04:00:17.977850+0000","last_undegraded":"2024-09-17T04:00:17.977850+0000","last_fullsized":"2024-09-17T04:00:17.977850+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:14:14.114514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354445+0000","last_change":"2024-09-17T04:00:15.206714+0000","last_active":"2024-09-17T04:00:17.354445+0000","last_peered":"2024-09-17T04:00:17.354445+0000","last_clean":"2024-09-17T04:00:17.354445+0000","last_became_active":"2024-09-17T04:00:15.206465+0000","last_became_peered":"2024-09-17T04:00:15.206465+0000","last_unstale":"2024-09-17T04:00:17.354445+0000","last_undegraded":"2024-09-17T04:00:17.354445+0000","last_fullsized":"2024-09-17T04:00:17.354445+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:29.233403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181368+0000","last_change":"2024-09-17T04:00:15.195270+0000","last_active":"2024-09-17T04:00:17.181368+0000","last_peered":"2024-09-17T04:00:17.181368+0000","last_clean":"2024-09-17T04:00:17.181368+0000","last_became_active":"2024-09-17T04:00:15.194918+0000","last_became_peered":"2024-09-17T04:00:15.194918+0000","last_unstale":"2024-09-17T04:00:17.181368+0000","last_undegraded":"2024-09-17T04:00:17.181368+0000","last_fullsized":"2024-09-17T04:00:17.181368+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:12.702219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186097+0000","last_change":"2024-09-17T04:00:15.201371+0000","last_active":"2024-09-17T04:00:16.186097+0000","last_peered":"2024-09-17T04:00:16.186097+0000","last_clean":"2024-09-17T04:00:16.186097+0000","last_became_active":"2024-09-17T04:00:15.200614+0000","last_became_peered":"2024-09-17T04:00:15.200614+0000","last_unstale":"2024-09-17T04:00:16.186097+0000","last_undegraded":"2024-09-17T04:00:16.186097+0000","last_fullsized":"2024-09-17T04:00:16.186097+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:15.371085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976448+0000","last_change":"2024-09-17T04:00:15.179118+0000","last_active":"2024-09-17T04:00:17.976448+0000","last_peered":"2024-09-17T04:00:17.976448+0000","last_clean":"2024-09-17T04:00:17.976448+0000","last_became_active":"2024-09-17T04:00:15.178825+0000","last_became_peered":"2024-09-17T04:00:15.178825+0000","last_unstale":"2024-09-17T04:00:17.976448+0000","last_undegraded":"2024-09-17T04:00:17.976448+0000","last_fullsized":"2024-09-17T04:00:17.976448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:10.154034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185605+0000","last_change":"2024-09-17T04:00:15.196095+0000","last_active":"2024-09-17T04:00:16.185605+0000","last_peered":"2024-09-17T04:00:16.185605+0000","last_clean":"2024-09-17T04:00:16.185605+0000","last_became_active":"2024-09-17T04:00:15.195155+0000","last_became_peered":"2024-09-17T04:00:15.195155+0000","last_unstale":"2024-09-17T04:00:16.185605+0000","last_undegraded":"2024-09-17T04:00:16.185605+0000","last_fullsized":"2024-09-17T04:00:16.185605+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:35.336819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185181+0000","last_change":"2024-09-17T04:00:15.187116+0000","last_active":"2024-09-17T04:00:16.185181+0000","last_peered":"2024-09-17T04:00:16.185181+0000","last_clean":"2024-09-17T04:00:16.185181+0000","last_became_active":"2024-09-17T04:00:15.186491+0000","last_became_peered":"2024-09-17T04:00:15.186491+0000","last_unstale":"2024-09-17T04:00:16.185181+0000","last_undegraded":"2024-09-17T04:00:16.185181+0000","last_fullsized":"2024-09-17T04:00:16.185181+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:53:20.978854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186343+0000","last_change":"2024-09-17T04:00:15.192294+0000","last_active":"2024-09-17T04:00:16.186343+0000","last_peered":"2024-09-17T04:00:16.186343+0000","last_clean":"2024-09-17T04:00:16.186343+0000","last_became_active":"2024-09-17T04:00:15.191929+0000","last_became_peered":"2024-09-17T04:00:15.191929+0000","last_unstale":"2024-09-17T04:00:16.186343+0000","last_undegraded":"2024-09-17T04:00:16.186343+0000","last_fullsized":"2024-09-17T04:00:16.186343+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:02.758168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977913+0000","last_change":"2024-09-17T04:00:15.202361+0000","last_active":"2024-09-17T04:00:17.977913+0000","last_peered":"2024-09-17T04:00:17.977913+0000","last_clean":"2024-09-17T04:00:17.977913+0000","last_became_active":"2024-09-17T04:00:15.202042+0000","last_became_peered":"2024-09-17T04:00:15.202042+0000","last_unstale":"2024-09-17T04:00:17.977913+0000","last_undegraded":"2024-09-17T04:00:17.977913+0000","last_fullsized":"2024-09-17T04:00:17.977913+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:31.626804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977369+0000","last_change":"2024-09-17T04:00:15.205859+0000","last_active":"2024-09-17T04:00:17.977369+0000","last_peered":"2024-09-17T04:00:17.977369+0000","last_clean":"2024-09-17T04:00:17.977369+0000","last_became_active":"2024-09-17T04:00:15.205527+0000","last_became_peered":"2024-09-17T04:00:15.205527+0000","last_unstale":"2024-09-17T04:00:17.977369+0000","last_undegraded":"2024-09-17T04:00:17.977369+0000","last_fullsized":"2024-09-17T04:00:17.977369+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:02:13.228284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184876+0000","last_change":"2024-09-17T04:00:15.198473+0000","last_active":"2024-09-17T04:00:16.184876+0000","last_peered":"2024-09-17T04:00:16.184876+0000","last_clean":"2024-09-17T04:00:16.184876+0000","last_became_active":"2024-09-17T04:00:15.194432+0000","last_became_peered":"2024-09-17T04:00:15.194432+0000","last_unstale":"2024-09-17T04:00:16.184876+0000","last_undegraded":"2024-09-17T04:00:16.184876+0000","last_fullsized":"2024-09-17T04:00:16.184876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:07.494846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977556+0000","last_change":"2024-09-17T04:00:15.195580+0000","last_active":"2024-09-17T04:00:17.977556+0000","last_peered":"2024-09-17T04:00:17.977556+0000","last_clean":"2024-09-17T04:00:17.977556+0000","last_became_active":"2024-09-17T04:00:15.194025+0000","last_became_peered":"2024-09-17T04:00:15.194025+0000","last_unstale":"2024-09-17T04:00:17.977556+0000","last_undegraded":"2024-09-17T04:00:17.977556+0000","last_fullsized":"2024-09-17T04:00:17.977556+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:34.997014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976439+0000","last_change":"2024-09-17T04:00:15.188218+0000","last_active":"2024-09-17T04:00:17.976439+0000","last_peered":"2024-09-17T04:00:17.976439+0000","last_clean":"2024-09-17T04:00:17.976439+0000","last_became_active":"2024-09-17T04:00:15.187955+0000","last_became_peered":"2024-09-17T04:00:15.187955+0000","last_unstale":"2024-09-17T04:00:17.976439+0000","last_undegraded":"2024-09-17T04:00:17.976439+0000","last_fullsized":"2024-09-17T04:00:17.976439+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354775+0000","last_change":"2024-09-17T04:00:15.208588+0000","last_active":"2024-09-17T04:00:17.354775+0000","last_peered":"2024-09-17T04:00:17.354775+0000","last_clean":"2024-09-17T04:00:17.354775+0000","last_became_active":"2024-09-17T04:00:15.208213+0000","last_became_peered":"2024-09-17T04:00:15.208213+0000","last_unstale":"2024-09-17T04:00:17.354775+0000","last_undegraded":"2024-09-17T04:00:17.354775+0000","last_fullsized":"2024-09-17T04:00:17.354775+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:53.003474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978563+0000","last_change":"2024-09-17T04:00:15.208085+0000","last_active":"2024-09-17T04:00:17.978563+0000","last_peered":"2024-09-17T04:00:17.978563+0000","last_clean":"2024-09-17T04:00:17.978563+0000","last_became_active":"2024-09-17T04:00:15.207762+0000","last_became_peered":"2024-09-17T04:00:15.207762+0000","last_unstale":"2024-09-17T04:00:17.978563+0000","last_undegraded":"2024-09-17T04:00:17.978563+0000","last_fullsized":"2024-09-17T04:00:17.978563+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184909+0000","last_change":"2024-09-17T04:00:15.185185+0000","last_active":"2024-09-17T04:00:16.184909+0000","last_peered":"2024-09-17T04:00:16.184909+0000","last_clean":"2024-09-17T04:00:16.184909+0000","last_became_active":"2024-09-17T04:00:15.182697+0000","last_became_peered":"2024-09-17T04:00:15.182697+0000","last_unstale":"2024-09-17T04:00:16.184909+0000","last_undegraded":"2024-09-17T04:00:16.184909+0000","last_fullsized":"2024-09-17T04:00:16.184909+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:06.451276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354394+0000","last_change":"2024-09-17T04:00:15.210019+0000","last_active":"2024-09-17T04:00:17.354394+0000","last_peered":"2024-09-17T04:00:17.354394+0000","last_clean":"2024-09-17T04:00:17.354394+0000","last_became_active":"2024-09-17T04:00:15.209795+0000","last_became_peered":"2024-09-17T04:00:15.209795+0000","last_unstale":"2024-09-17T04:00:17.354394+0000","last_undegraded":"2024-09-17T04:00:17.354394+0000","last_fullsized":"2024-09-17T04:00:17.354394+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:27.563680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354687+0000","last_change":"2024-09-17T04:00:15.206043+0000","last_active":"2024-09-17T04:00:17.354687+0000","last_peered":"2024-09-17T04:00:17.354687+0000","last_clean":"2024-09-17T04:00:17.354687+0000","last_became_active":"2024-09-17T04:00:15.205686+0000","last_became_peered":"2024-09-17T04:00:15.205686+0000","last_unstale":"2024-09-17T04:00:17.354687+0000","last_undegraded":"2024-09-17T04:00:17.354687+0000","last_fullsized":"2024-09-17T04:00:17.354687+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:40.936992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184888+0000","last_change":"2024-09-17T04:00:15.187015+0000","last_active":"2024-09-17T04:00:16.184888+0000","last_peered":"2024-09-17T04:00:16.184888+0000","last_clean":"2024-09-17T04:00:16.184888+0000","last_became_active":"2024-09-17T04:00:15.186574+0000","last_became_peered":"2024-09-17T04:00:15.186574+0000","last_unstale":"2024-09-17T04:00:16.184888+0000","last_undegraded":"2024-09-17T04:00:16.184888+0000","last_fullsized":"2024-09-17T04:00:16.184888+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:05.697277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181307+0000","last_change":"2024-09-17T04:00:15.195322+0000","last_active":"2024-09-17T04:00:17.181307+0000","last_peered":"2024-09-17T04:00:17.181307+0000","last_clean":"2024-09-17T04:00:17.181307+0000","last_became_active":"2024-09-17T04:00:15.194994+0000","last_became_peered":"2024-09-17T04:00:15.194994+0000","last_unstale":"2024-09-17T04:00:17.181307+0000","last_undegraded":"2024-09-17T04:00:17.181307+0000","last_fullsized":"2024-09-17T04:00:17.181307+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:36.557958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976714+0000","last_change":"2024-09-17T04:00:15.187550+0000","last_active":"2024-09-17T04:00:17.976714+0000","last_peered":"2024-09-17T04:00:17.976714+0000","last_clean":"2024-09-17T04:00:17.976714+0000","last_became_active":"2024-09-17T04:00:15.183952+0000","last_became_peered":"2024-09-17T04:00:15.183952+0000","last_unstale":"2024-09-17T04:00:17.976714+0000","last_undegraded":"2024-09-17T04:00:17.976714+0000","last_fullsized":"2024-09-17T04:00:17.976714+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:53.762374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977962+0000","last_change":"2024-09-17T04:00:15.196465+0000","last_active":"2024-09-17T04:00:17.977962+0000","last_peered":"2024-09-17T04:00:17.977962+0000","last_clean":"2024-09-17T04:00:17.977962+0000","last_became_active":"2024-09-17T04:00:15.195761+0000","last_became_peered":"2024-09-17T04:00:15.195761+0000","last_unstale":"2024-09-17T04:00:17.977962+0000","last_undegraded":"2024-09-17T04:00:17.977962+0000","last_fullsized":"2024-09-17T04:00:17.977962+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:29.557541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186879+0000","last_change":"2024-09-17T04:00:15.201293+0000","last_active":"2024-09-17T04:00:16.186879+0000","last_peered":"2024-09-17T04:00:16.186879+0000","last_clean":"2024-09-17T04:00:16.186879+0000","last_became_active":"2024-09-17T04:00:15.200722+0000","last_became_peered":"2024-09-17T04:00:15.200722+0000","last_unstale":"2024-09-17T04:00:16.186879+0000","last_undegraded":"2024-09-17T04:00:16.186879+0000","last_fullsized":"2024-09-17T04:00:16.186879+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:21.636014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181655+0000","last_change":"2024-09-17T04:00:15.193142+0000","last_active":"2024-09-17T04:00:17.181655+0000","last_peered":"2024-09-17T04:00:17.181655+0000","last_clean":"2024-09-17T04:00:17.181655+0000","last_became_active":"2024-09-17T04:00:15.192818+0000","last_became_peered":"2024-09-17T04:00:15.192818+0000","last_unstale":"2024-09-17T04:00:17.181655+0000","last_undegraded":"2024-09-17T04:00:17.181655+0000","last_fullsized":"2024-09-17T04:00:17.181655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:51.767875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354448+0000","last_change":"2024-09-17T04:00:15.196835+0000","last_active":"2024-09-17T04:00:17.354448+0000","last_peered":"2024-09-17T04:00:17.354448+0000","last_clean":"2024-09-17T04:00:17.354448+0000","last_became_active":"2024-09-17T04:00:15.196475+0000","last_became_peered":"2024-09-17T04:00:15.196475+0000","last_unstale":"2024-09-17T04:00:17.354448+0000","last_undegraded":"2024-09-17T04:00:17.354448+0000","last_fullsized":"2024-09-17T04:00:17.354448+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:19.570741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977568+0000","last_change":"2024-09-17T04:00:15.203508+0000","last_active":"2024-09-17T04:00:17.977568+0000","last_peered":"2024-09-17T04:00:17.977568+0000","last_clean":"2024-09-17T04:00:17.977568+0000","last_became_active":"2024-09-17T04:00:15.203187+0000","last_became_peered":"2024-09-17T04:00:15.203187+0000","last_unstale":"2024-09-17T04:00:17.977568+0000","last_undegraded":"2024-09-17T04:00:17.977568+0000","last_fullsized":"2024-09-17T04:00:17.977568+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:55.068491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184802+0000","last_change":"2024-09-17T04:00:15.194543+0000","last_active":"2024-09-17T04:00:16.184802+0000","last_peered":"2024-09-17T04:00:16.184802+0000","last_clean":"2024-09-17T04:00:16.184802+0000","last_became_active":"2024-09-17T04:00:15.194011+0000","last_became_peered":"2024-09-17T04:00:15.194011+0000","last_unstale":"2024-09-17T04:00:16.184802+0000","last_undegraded":"2024-09-17T04:00:16.184802+0000","last_fullsized":"2024-09-17T04:00:16.184802+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:25.507246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181366+0000","last_change":"2024-09-17T04:00:15.198425+0000","last_active":"2024-09-17T04:00:17.181366+0000","last_peered":"2024-09-17T04:00:17.181366+0000","last_clean":"2024-09-17T04:00:17.181366+0000","last_became_active":"2024-09-17T04:00:15.198098+0000","last_became_peered":"2024-09-17T04:00:15.198098+0000","last_unstale":"2024-09-17T04:00:17.181366+0000","last_undegraded":"2024-09-17T04:00:17.181366+0000","last_fullsized":"2024-09-17T04:00:17.181366+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:09.331234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354139+0000","last_change":"2024-09-17T04:00:15.210848+0000","last_active":"2024-09-17T04:00:17.354139+0000","last_peered":"2024-09-17T04:00:17.354139+0000","last_clean":"2024-09-17T04:00:17.354139+0000","last_became_active":"2024-09-17T04:00:15.210516+0000","last_became_peered":"2024-09-17T04:00:15.210516+0000","last_unstale":"2024-09-17T04:00:17.354139+0000","last_undegraded":"2024-09-17T04:00:17.354139+0000","last_fullsized":"2024-09-17T04:00:17.354139+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:55.037291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977006+0000","last_change":"2024-09-17T04:00:15.185994+0000","last_active":"2024-09-17T04:00:17.977006+0000","last_peered":"2024-09-17T04:00:17.977006+0000","last_clean":"2024-09-17T04:00:17.977006+0000","last_became_active":"2024-09-17T04:00:15.184966+0000","last_became_peered":"2024-09-17T04:00:15.184966+0000","last_unstale":"2024-09-17T04:00:17.977006+0000","last_undegraded":"2024-09-17T04:00:17.977006+0000","last_fullsized":"2024-09-17T04:00:17.977006+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:34.976945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"162'4","reported_seq":19,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354792+0000","last_change":"2024-09-17T04:00:14.184544+0000","last_active":"2024-09-17T04:00:17.354792+0000","last_peered":"2024-09-17T04:00:17.354792+0000","last_clean":"2024-09-17T04:00:17.354792+0000","last_became_active":"2024-09-17T04:00:14.184254+0000","last_became_peered":"2024-09-17T04:00:14.184254+0000","last_unstale":"2024-09-17T04:00:17.354792+0000","last_undegraded":"2024-09-17T04:00:17.354792+0000","last_fullsized":"2024-09-17T04:00:17.354792+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:15:30.284955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186087+0000","last_change":"2024-09-17T04:00:15.203100+0000","last_active":"2024-09-17T04:00:16.186087+0000","last_peered":"2024-09-17T04:00:16.186087+0000","last_clean":"2024-09-17T04:00:16.186087+0000","last_became_active":"2024-09-17T04:00:15.202011+0000","last_became_peered":"2024-09-17T04:00:15.202011+0000","last_unstale":"2024-09-17T04:00:16.186087+0000","last_undegraded":"2024-09-17T04:00:16.186087+0000","last_fullsized":"2024-09-17T04:00:16.186087+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:07.578261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185365+0000","last_change":"2024-09-17T04:00:14.178919+0000","last_active":"2024-09-17T04:00:16.185365+0000","last_peered":"2024-09-17T04:00:16.185365+0000","last_clean":"2024-09-17T04:00:16.185365+0000","last_became_active":"2024-09-17T04:00:14.177710+0000","last_became_peered":"2024-09-17T04:00:14.177710+0000","last_unstale":"2024-09-17T04:00:16.185365+0000","last_undegraded":"2024-09-17T04:00:16.185365+0000","last_fullsized":"2024-09-17T04:00:16.185365+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:23:18.564335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186429+0000","last_change":"2024-09-17T04:00:15.202175+0000","last_active":"2024-09-17T04:00:16.186429+0000","last_peered":"2024-09-17T04:00:16.186429+0000","last_clean":"2024-09-17T04:00:16.186429+0000","last_became_active":"2024-09-17T04:00:15.201873+0000","last_became_peered":"2024-09-17T04:00:15.201873+0000","last_unstale":"2024-09-17T04:00:16.186429+0000","last_undegraded":"2024-09-17T04:00:16.186429+0000","last_fullsized":"2024-09-17T04:00:16.186429+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:17:23.847548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181775+0000","last_change":"2024-09-17T04:00:14.173566+0000","last_active":"2024-09-17T04:00:17.181775+0000","last_peered":"2024-09-17T04:00:17.181775+0000","last_clean":"2024-09-17T04:00:17.181775+0000","last_became_active":"2024-09-17T04:00:14.173201+0000","last_became_peered":"2024-09-17T04:00:14.173201+0000","last_unstale":"2024-09-17T04:00:17.181775+0000","last_undegraded":"2024-09-17T04:00:17.181775+0000","last_fullsized":"2024-09-17T04:00:17.181775+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:09:15.816175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185473+0000","last_change":"2024-09-17T04:00:15.199464+0000","last_active":"2024-09-17T04:00:16.185473+0000","last_peered":"2024-09-17T04:00:16.185473+0000","last_clean":"2024-09-17T04:00:16.185473+0000","last_became_active":"2024-09-17T04:00:15.199050+0000","last_became_peered":"2024-09-17T04:00:15.199050+0000","last_unstale":"2024-09-17T04:00:16.185473+0000","last_undegraded":"2024-09-17T04:00:16.185473+0000","last_fullsized":"2024-09-17T04:00:16.185473+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:33.024811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977943+0000","last_change":"2024-09-17T04:00:14.180751+0000","last_active":"2024-09-17T04:00:17.977943+0000","last_peered":"2024-09-17T04:00:17.977943+0000","last_clean":"2024-09-17T04:00:17.977943+0000","last_became_active":"2024-09-17T04:00:14.180400+0000","last_became_peered":"2024-09-17T04:00:14.180400+0000","last_unstale":"2024-09-17T04:00:17.977943+0000","last_undegraded":"2024-09-17T04:00:17.977943+0000","last_fullsized":"2024-09-17T04:00:17.977943+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:14.945761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185444+0000","last_change":"2024-09-17T04:00:15.183077+0000","last_active":"2024-09-17T04:00:16.185444+0000","last_peered":"2024-09-17T04:00:16.185444+0000","last_clean":"2024-09-17T04:00:16.185444+0000","last_became_active":"2024-09-17T04:00:15.182579+0000","last_became_peered":"2024-09-17T04:00:15.182579+0000","last_unstale":"2024-09-17T04:00:16.185444+0000","last_undegraded":"2024-09-17T04:00:16.185444+0000","last_fullsized":"2024-09-17T04:00:16.185444+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:24:32.022599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354752+0000","last_change":"2024-09-17T04:00:14.178007+0000","last_active":"2024-09-17T04:00:17.354752+0000","last_peered":"2024-09-17T04:00:17.354752+0000","last_clean":"2024-09-17T04:00:17.354752+0000","last_became_active":"2024-09-17T04:00:14.177397+0000","last_became_peered":"2024-09-17T04:00:14.177397+0000","last_unstale":"2024-09-17T04:00:17.354752+0000","last_undegraded":"2024-09-17T04:00:17.354752+0000","last_fullsized":"2024-09-17T04:00:17.354752+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:51:46.574189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976655+0000","last_change":"2024-09-17T04:00:15.184558+0000","last_active":"2024-09-17T04:00:17.976655+0000","last_peered":"2024-09-17T04:00:17.976655+0000","last_clean":"2024-09-17T04:00:17.976655+0000","last_became_active":"2024-09-17T04:00:15.181827+0000","last_became_peered":"2024-09-17T04:00:15.181827+0000","last_unstale":"2024-09-17T04:00:17.976655+0000","last_undegraded":"2024-09-17T04:00:17.976655+0000","last_fullsized":"2024-09-17T04:00:17.976655+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:03.101845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976490+0000","last_change":"2024-09-17T04:00:14.181701+0000","last_active":"2024-09-17T04:00:17.976490+0000","last_peered":"2024-09-17T04:00:17.976490+0000","last_clean":"2024-09-17T04:00:17.976490+0000","last_became_active":"2024-09-17T04:00:14.180617+0000","last_became_peered":"2024-09-17T04:00:14.180617+0000","last_unstale":"2024-09-17T04:00:17.976490+0000","last_undegraded":"2024-09-17T04:00:17.976490+0000","last_fullsized":"2024-09-17T04:00:17.976490+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185980+0000","last_change":"2024-09-17T04:00:15.188106+0000","last_active":"2024-09-17T04:00:16.185980+0000","last_peered":"2024-09-17T04:00:16.185980+0000","last_clean":"2024-09-17T04:00:16.185980+0000","last_became_active":"2024-09-17T04:00:15.187755+0000","last_became_peered":"2024-09-17T04:00:15.187755+0000","last_unstale":"2024-09-17T04:00:16.185980+0000","last_undegraded":"2024-09-17T04:00:16.185980+0000","last_fullsized":"2024-09-17T04:00:16.185980+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:16.200716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976969+0000","last_change":"2024-09-17T04:00:14.180760+0000","last_active":"2024-09-17T04:00:17.976969+0000","last_peered":"2024-09-17T04:00:17.976969+0000","last_clean":"2024-09-17T04:00:17.976969+0000","last_became_active":"2024-09-17T04:00:14.180063+0000","last_became_peered":"2024-09-17T04:00:14.180063+0000","last_unstale":"2024-09-17T04:00:17.976969+0000","last_undegraded":"2024-09-17T04:00:17.976969+0000","last_fullsized":"2024-09-17T04:00:17.976969+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181748+0000","last_change":"2024-09-17T04:00:15.184353+0000","last_active":"2024-09-17T04:00:17.181748+0000","last_peered":"2024-09-17T04:00:17.181748+0000","last_clean":"2024-09-17T04:00:17.181748+0000","last_became_active":"2024-09-17T04:00:15.183526+0000","last_became_peered":"2024-09-17T04:00:15.183526+0000","last_unstale":"2024-09-17T04:00:17.181748+0000","last_undegraded":"2024-09-17T04:00:17.181748+0000","last_fullsized":"2024-09-17T04:00:17.181748+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:08.174367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186810+0000","last_change":"2024-09-17T04:00:14.170243+0000","last_active":"2024-09-17T04:00:16.186810+0000","last_peered":"2024-09-17T04:00:16.186810+0000","last_clean":"2024-09-17T04:00:16.186810+0000","last_became_active":"2024-09-17T04:00:14.169878+0000","last_became_peered":"2024-09-17T04:00:14.169878+0000","last_unstale":"2024-09-17T04:00:16.186810+0000","last_undegraded":"2024-09-17T04:00:16.186810+0000","last_fullsized":"2024-09-17T04:00:16.186810+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:35.659340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976963+0000","last_change":"2024-09-17T04:00:15.187262+0000","last_active":"2024-09-17T04:00:17.976963+0000","last_peered":"2024-09-17T04:00:17.976963+0000","last_clean":"2024-09-17T04:00:17.976963+0000","last_became_active":"2024-09-17T04:00:15.186910+0000","last_became_peered":"2024-09-17T04:00:15.186910+0000","last_unstale":"2024-09-17T04:00:17.976963+0000","last_undegraded":"2024-09-17T04:00:17.976963+0000","last_fullsized":"2024-09-17T04:00:17.976963+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:21.523618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354640+0000","last_change":"2024-09-17T04:00:14.178548+0000","last_active":"2024-09-17T04:00:17.354640+0000","last_peered":"2024-09-17T04:00:17.354640+0000","last_clean":"2024-09-17T04:00:17.354640+0000","last_became_active":"2024-09-17T04:00:14.178173+0000","last_became_peered":"2024-09-17T04:00:14.178173+0000","last_unstale":"2024-09-17T04:00:17.354640+0000","last_undegraded":"2024-09-17T04:00:17.354640+0000","last_fullsized":"2024-09-17T04:00:17.354640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:05.150582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186273+0000","last_change":"2024-09-17T04:00:15.190476+0000","last_active":"2024-09-17T04:00:16.186273+0000","last_peered":"2024-09-17T04:00:16.186273+0000","last_clean":"2024-09-17T04:00:16.186273+0000","last_became_active":"2024-09-17T04:00:15.189977+0000","last_became_peered":"2024-09-17T04:00:15.189977+0000","last_unstale":"2024-09-17T04:00:16.186273+0000","last_undegraded":"2024-09-17T04:00:16.186273+0000","last_fullsized":"2024-09-17T04:00:16.186273+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:26.037673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"162'3","reported_seq":18,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354310+0000","last_change":"2024-09-17T04:00:14.184902+0000","last_active":"2024-09-17T04:00:17.354310+0000","last_peered":"2024-09-17T04:00:17.354310+0000","last_clean":"2024-09-17T04:00:17.354310+0000","last_became_active":"2024-09-17T04:00:14.184624+0000","last_became_peered":"2024-09-17T04:00:14.184624+0000","last_unstale":"2024-09-17T04:00:17.354310+0000","last_undegraded":"2024-09-17T04:00:17.354310+0000","last_fullsized":"2024-09-17T04:00:17.354310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:08.515942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185090+0000","last_change":"2024-09-17T04:00:15.180729+0000","last_active":"2024-09-17T04:00:16.185090+0000","last_peered":"2024-09-17T04:00:16.185090+0000","last_clean":"2024-09-17T04:00:16.185090+0000","last_became_active":"2024-09-17T04:00:15.180358+0000","last_became_peered":"2024-09-17T04:00:15.180358+0000","last_unstale":"2024-09-17T04:00:16.185090+0000","last_undegraded":"2024-09-17T04:00:16.185090+0000","last_fullsized":"2024-09-17T04:00:16.185090+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:47:55.762158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"162'6","reported_seq":21,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181426+0000","last_change":"2024-09-17T04:00:14.177482+0000","last_active":"2024-09-17T04:00:17.181426+0000","last_peered":"2024-09-17T04:00:17.181426+0000","last_clean":"2024-09-17T04:00:17.181426+0000","last_became_active":"2024-09-17T04:00:14.177185+0000","last_became_peered":"2024-09-17T04:00:14.177185+0000","last_unstale":"2024-09-17T04:00:17.181426+0000","last_undegraded":"2024-09-17T04:00:17.181426+0000","last_fullsized":"2024-09-17T04:00:17.181426+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:59.681874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181467+0000","last_change":"2024-09-17T04:00:15.197664+0000","last_active":"2024-09-17T04:00:17.181467+0000","last_peered":"2024-09-17T04:00:17.181467+0000","last_clean":"2024-09-17T04:00:17.181467+0000","last_became_active":"2024-09-17T04:00:15.197330+0000","last_became_peered":"2024-09-17T04:00:15.197330+0000","last_unstale":"2024-09-17T04:00:17.181467+0000","last_undegraded":"2024-09-17T04:00:17.181467+0000","last_fullsized":"2024-09-17T04:00:17.181467+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:00:09.294262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978258+0000","last_change":"2024-09-17T04:00:14.180486+0000","last_active":"2024-09-17T04:00:17.978258+0000","last_peered":"2024-09-17T04:00:17.978258+0000","last_clean":"2024-09-17T04:00:17.978258+0000","last_became_active":"2024-09-17T04:00:14.180077+0000","last_became_peered":"2024-09-17T04:00:14.180077+0000","last_unstale":"2024-09-17T04:00:17.978258+0000","last_undegraded":"2024-09-17T04:00:17.978258+0000","last_fullsized":"2024-09-17T04:00:17.978258+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:15:01.024929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186622+0000","last_change":"2024-09-17T04:00:15.195036+0000","last_active":"2024-09-17T04:00:16.186622+0000","last_peered":"2024-09-17T04:00:16.186622+0000","last_clean":"2024-09-17T04:00:16.186622+0000","last_became_active":"2024-09-17T04:00:15.193366+0000","last_became_peered":"2024-09-17T04:00:15.193366+0000","last_unstale":"2024-09-17T04:00:16.186622+0000","last_undegraded":"2024-09-17T04:00:16.186622+0000","last_fullsized":"2024-09-17T04:00:16.186622+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:49:35.218017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185646+0000","last_change":"2024-09-17T04:00:14.179129+0000","last_active":"2024-09-17T04:00:16.185646+0000","last_peered":"2024-09-17T04:00:16.185646+0000","last_clean":"2024-09-17T04:00:16.185646+0000","last_became_active":"2024-09-17T04:00:14.177651+0000","last_became_peered":"2024-09-17T04:00:14.177651+0000","last_unstale":"2024-09-17T04:00:16.185646+0000","last_undegraded":"2024-09-17T04:00:16.185646+0000","last_fullsized":"2024-09-17T04:00:16.185646+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:10:50.885154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184621+0000","last_change":"2024-09-17T04:00:15.195850+0000","last_active":"2024-09-17T04:00:16.184621+0000","last_peered":"2024-09-17T04:00:16.184621+0000","last_clean":"2024-09-17T04:00:16.184621+0000","last_became_active":"2024-09-17T04:00:15.194823+0000","last_became_peered":"2024-09-17T04:00:15.194823+0000","last_unstale":"2024-09-17T04:00:16.184621+0000","last_undegraded":"2024-09-17T04:00:16.184621+0000","last_fullsized":"2024-09-17T04:00:16.184621+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:16.156594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976351+0000","last_change":"2024-09-17T04:00:14.178903+0000","last_active":"2024-09-17T04:00:17.976351+0000","last_peered":"2024-09-17T04:00:17.976351+0000","last_clean":"2024-09-17T04:00:17.976351+0000","last_became_active":"2024-09-17T04:00:14.178605+0000","last_became_peered":"2024-09-17T04:00:14.178605+0000","last_unstale":"2024-09-17T04:00:17.976351+0000","last_undegraded":"2024-09-17T04:00:17.976351+0000","last_fullsized":"2024-09-17T04:00:17.976351+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:20.629609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184829+0000","last_change":"2024-09-17T04:00:15.192316+0000","last_active":"2024-09-17T04:00:16.184829+0000","last_peered":"2024-09-17T04:00:16.184829+0000","last_clean":"2024-09-17T04:00:16.184829+0000","last_became_active":"2024-09-17T04:00:15.188357+0000","last_became_peered":"2024-09-17T04:00:15.188357+0000","last_unstale":"2024-09-17T04:00:16.184829+0000","last_undegraded":"2024-09-17T04:00:16.184829+0000","last_fullsized":"2024-09-17T04:00:16.184829+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:34.996367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"162'8","reported_seq":23,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977302+0000","last_change":"2024-09-17T04:00:14.182987+0000","last_active":"2024-09-17T04:00:17.977302+0000","last_peered":"2024-09-17T04:00:17.977302+0000","last_clean":"2024-09-17T04:00:17.977302+0000","last_became_active":"2024-09-17T04:00:14.182665+0000","last_became_peered":"2024-09-17T04:00:14.182665+0000","last_unstale":"2024-09-17T04:00:17.977302+0000","last_undegraded":"2024-09-17T04:00:17.977302+0000","last_fullsized":"2024-09-17T04:00:17.977302+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185423+0000","last_change":"2024-09-17T04:00:15.188466+0000","last_active":"2024-09-17T04:00:16.185423+0000","last_peered":"2024-09-17T04:00:16.185423+0000","last_clean":"2024-09-17T04:00:16.185423+0000","last_became_active":"2024-09-17T04:00:15.187860+0000","last_became_peered":"2024-09-17T04:00:15.187860+0000","last_unstale":"2024-09-17T04:00:16.185423+0000","last_undegraded":"2024-09-17T04:00:16.185423+0000","last_fullsized":"2024-09-17T04:00:16.185423+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:05.989755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354873+0000","last_change":"2024-09-17T04:00:14.179849+0000","last_active":"2024-09-17T04:00:17.354873+0000","last_peered":"2024-09-17T04:00:17.354873+0000","last_clean":"2024-09-17T04:00:17.354873+0000","last_became_active":"2024-09-17T04:00:14.178084+0000","last_became_peered":"2024-09-17T04:00:14.178084+0000","last_unstale":"2024-09-17T04:00:17.354873+0000","last_undegraded":"2024-09-17T04:00:17.354873+0000","last_fullsized":"2024-09-17T04:00:17.354873+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:31:26.618084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185623+0000","last_change":"2024-09-17T04:00:15.190551+0000","last_active":"2024-09-17T04:00:16.185623+0000","last_peered":"2024-09-17T04:00:16.185623+0000","last_clean":"2024-09-17T04:00:16.185623+0000","last_became_active":"2024-09-17T04:00:15.190147+0000","last_became_peered":"2024-09-17T04:00:15.190147+0000","last_unstale":"2024-09-17T04:00:16.185623+0000","last_undegraded":"2024-09-17T04:00:16.185623+0000","last_fullsized":"2024-09-17T04:00:16.185623+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:27:00.814493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977169+0000","last_change":"2024-09-17T04:00:14.173867+0000","last_active":"2024-09-17T04:00:17.977169+0000","last_peered":"2024-09-17T04:00:17.977169+0000","last_clean":"2024-09-17T04:00:17.977169+0000","last_became_active":"2024-09-17T04:00:14.171614+0000","last_became_peered":"2024-09-17T04:00:14.171614+0000","last_unstale":"2024-09-17T04:00:17.977169+0000","last_undegraded":"2024-09-17T04:00:17.977169+0000","last_fullsized":"2024-09-17T04:00:17.977169+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185175+0000","last_change":"2024-09-17T04:00:15.196127+0000","last_active":"2024-09-17T04:00:16.185175+0000","last_peered":"2024-09-17T04:00:16.185175+0000","last_clean":"2024-09-17T04:00:16.185175+0000","last_became_active":"2024-09-17T04:00:15.195799+0000","last_became_peered":"2024-09-17T04:00:15.195799+0000","last_unstale":"2024-09-17T04:00:16.185175+0000","last_undegraded":"2024-09-17T04:00:16.185175+0000","last_fullsized":"2024-09-17T04:00:16.185175+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:45:34.672205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"162'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978453+0000","last_change":"2024-09-17T04:00:14.183120+0000","last_active":"2024-09-17T04:00:17.978453+0000","last_peered":"2024-09-17T04:00:17.978453+0000","last_clean":"2024-09-17T04:00:17.978453+0000","last_became_active":"2024-09-17T04:00:14.182779+0000","last_became_peered":"2024-09-17T04:00:14.182779+0000","last_unstale":"2024-09-17T04:00:17.978453+0000","last_undegraded":"2024-09-17T04:00:17.978453+0000","last_fullsized":"2024-09-17T04:00:17.978453+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:53:19.161386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977807+0000","last_change":"2024-09-17T04:00:15.204420+0000","last_active":"2024-09-17T04:00:17.977807+0000","last_peered":"2024-09-17T04:00:17.977807+0000","last_clean":"2024-09-17T04:00:17.977807+0000","last_became_active":"2024-09-17T04:00:15.202153+0000","last_became_peered":"2024-09-17T04:00:15.202153+0000","last_unstale":"2024-09-17T04:00:17.977807+0000","last_undegraded":"2024-09-17T04:00:17.977807+0000","last_fullsized":"2024-09-17T04:00:17.977807+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:57.466014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186468+0000","last_change":"2024-09-17T04:00:14.164836+0000","last_active":"2024-09-17T04:00:16.186468+0000","last_peered":"2024-09-17T04:00:16.186468+0000","last_clean":"2024-09-17T04:00:16.186468+0000","last_became_active":"2024-09-17T04:00:14.164491+0000","last_became_peered":"2024-09-17T04:00:14.164491+0000","last_unstale":"2024-09-17T04:00:16.186468+0000","last_undegraded":"2024-09-17T04:00:16.186468+0000","last_fullsized":"2024-09-17T04:00:16.186468+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:53:35.894685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185709+0000","last_change":"2024-09-17T04:00:15.203883+0000","last_active":"2024-09-17T04:00:16.185709+0000","last_peered":"2024-09-17T04:00:16.185709+0000","last_clean":"2024-09-17T04:00:16.185709+0000","last_became_active":"2024-09-17T04:00:15.201414+0000","last_became_peered":"2024-09-17T04:00:15.201414+0000","last_unstale":"2024-09-17T04:00:16.185709+0000","last_undegraded":"2024-09-17T04:00:16.185709+0000","last_fullsized":"2024-09-17T04:00:16.185709+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:29.778689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181504+0000","last_change":"2024-09-17T04:00:14.185714+0000","last_active":"2024-09-17T04:00:17.181504+0000","last_peered":"2024-09-17T04:00:17.181504+0000","last_clean":"2024-09-17T04:00:17.181504+0000","last_became_active":"2024-09-17T04:00:14.185363+0000","last_became_peered":"2024-09-17T04:00:14.185363+0000","last_unstale":"2024-09-17T04:00:17.181504+0000","last_undegraded":"2024-09-17T04:00:17.181504+0000","last_fullsized":"2024-09-17T04:00:17.181504+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:56:32.866011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186668+0000","last_change":"2024-09-17T04:00:15.179231+0000","last_active":"2024-09-17T04:00:16.186668+0000","last_peered":"2024-09-17T04:00:16.186668+0000","last_clean":"2024-09-17T04:00:16.186668+0000","last_became_active":"2024-09-17T04:00:15.178887+0000","last_became_peered":"2024-09-17T04:00:15.178887+0000","last_unstale":"2024-09-17T04:00:16.186668+0000","last_undegraded":"2024-09-17T04:00:16.186668+0000","last_fullsized":"2024-09-17T04:00:16.186668+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:32.045406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354175+0000","last_change":"2024-09-17T04:00:14.183023+0000","last_active":"2024-09-17T04:00:17.354175+0000","last_peered":"2024-09-17T04:00:17.354175+0000","last_clean":"2024-09-17T04:00:17.354175+0000","last_became_active":"2024-09-17T04:00:14.178678+0000","last_became_peered":"2024-09-17T04:00:14.178678+0000","last_unstale":"2024-09-17T04:00:17.354175+0000","last_undegraded":"2024-09-17T04:00:17.354175+0000","last_fullsized":"2024-09-17T04:00:17.354175+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:09.710450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977340+0000","last_change":"2024-09-17T04:00:15.202518+0000","last_active":"2024-09-17T04:00:17.977340+0000","last_peered":"2024-09-17T04:00:17.977340+0000","last_clean":"2024-09-17T04:00:17.977340+0000","last_became_active":"2024-09-17T04:00:15.202240+0000","last_became_peered":"2024-09-17T04:00:15.202240+0000","last_unstale":"2024-09-17T04:00:17.977340+0000","last_undegraded":"2024-09-17T04:00:17.977340+0000","last_fullsized":"2024-09-17T04:00:17.977340+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:59.197974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181510+0000","last_change":"2024-09-17T04:00:14.178227+0000","last_active":"2024-09-17T04:00:17.181510+0000","last_peered":"2024-09-17T04:00:17.181510+0000","last_clean":"2024-09-17T04:00:17.181510+0000","last_became_active":"2024-09-17T04:00:14.177922+0000","last_became_peered":"2024-09-17T04:00:14.177922+0000","last_unstale":"2024-09-17T04:00:17.181510+0000","last_undegraded":"2024-09-17T04:00:17.181510+0000","last_fullsized":"2024-09-17T04:00:17.181510+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:49.911464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354137+0000","last_change":"2024-09-17T04:00:15.200391+0000","last_active":"2024-09-17T04:00:17.354137+0000","last_peered":"2024-09-17T04:00:17.354137+0000","last_clean":"2024-09-17T04:00:17.354137+0000","last_became_active":"2024-09-17T04:00:15.200095+0000","last_became_peered":"2024-09-17T04:00:15.200095+0000","last_unstale":"2024-09-17T04:00:17.354137+0000","last_undegraded":"2024-09-17T04:00:17.354137+0000","last_fullsized":"2024-09-17T04:00:17.354137+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:18:34.896086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184666+0000","last_change":"2024-09-17T04:00:14.173438+0000","last_active":"2024-09-17T04:00:16.184666+0000","last_peered":"2024-09-17T04:00:16.184666+0000","last_clean":"2024-09-17T04:00:16.184666+0000","last_became_active":"2024-09-17T04:00:14.173090+0000","last_became_peered":"2024-09-17T04:00:14.173090+0000","last_unstale":"2024-09-17T04:00:16.184666+0000","last_undegraded":"2024-09-17T04:00:16.184666+0000","last_fullsized":"2024-09-17T04:00:16.184666+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:21.951204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354500+0000","last_change":"2024-09-17T04:00:15.206400+0000","last_active":"2024-09-17T04:00:17.354500+0000","last_peered":"2024-09-17T04:00:17.354500+0000","last_clean":"2024-09-17T04:00:17.354500+0000","last_became_active":"2024-09-17T04:00:15.205707+0000","last_became_peered":"2024-09-17T04:00:15.205707+0000","last_unstale":"2024-09-17T04:00:17.354500+0000","last_undegraded":"2024-09-17T04:00:17.354500+0000","last_fullsized":"2024-09-17T04:00:17.354500+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:45:36.571087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"162'2","reported_seq":17,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978691+0000","last_change":"2024-09-17T04:00:14.186855+0000","last_active":"2024-09-17T04:00:17.978691+0000","last_peered":"2024-09-17T04:00:17.978691+0000","last_clean":"2024-09-17T04:00:17.978691+0000","last_became_active":"2024-09-17T04:00:14.186535+0000","last_became_peered":"2024-09-17T04:00:14.186535+0000","last_unstale":"2024-09-17T04:00:17.978691+0000","last_undegraded":"2024-09-17T04:00:17.978691+0000","last_fullsized":"2024-09-17T04:00:17.978691+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:48:32.914832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.186153+0000","last_change":"2024-09-17T04:00:15.194286+0000","last_active":"2024-09-17T04:00:16.186153+0000","last_peered":"2024-09-17T04:00:16.186153+0000","last_clean":"2024-09-17T04:00:16.186153+0000","last_became_active":"2024-09-17T04:00:15.193936+0000","last_became_peered":"2024-09-17T04:00:15.193936+0000","last_unstale":"2024-09-17T04:00:16.186153+0000","last_undegraded":"2024-09-17T04:00:16.186153+0000","last_fullsized":"2024-09-17T04:00:16.186153+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:03.221443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"163'1","reported_seq":16,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185534+0000","last_change":"2024-09-17T04:00:14.176145+0000","last_active":"2024-09-17T04:00:16.185534+0000","last_peered":"2024-09-17T04:00:16.185534+0000","last_clean":"2024-09-17T04:00:16.185534+0000","last_became_active":"2024-09-17T04:00:14.175839+0000","last_became_peered":"2024-09-17T04:00:14.175839+0000","last_unstale":"2024-09-17T04:00:16.185534+0000","last_undegraded":"2024-09-17T04:00:16.185534+0000","last_fullsized":"2024-09-17T04:00:16.185534+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:33.949960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976660+0000","last_change":"2024-09-17T04:00:15.181165+0000","last_active":"2024-09-17T04:00:17.976660+0000","last_peered":"2024-09-17T04:00:17.976660+0000","last_clean":"2024-09-17T04:00:17.976660+0000","last_became_active":"2024-09-17T04:00:15.180136+0000","last_became_peered":"2024-09-17T04:00:15.180136+0000","last_unstale":"2024-09-17T04:00:17.976660+0000","last_undegraded":"2024-09-17T04:00:17.976660+0000","last_fullsized":"2024-09-17T04:00:17.976660+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:28.049513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184972+0000","last_change":"2024-09-17T04:00:14.182944+0000","last_active":"2024-09-17T04:00:16.184972+0000","last_peered":"2024-09-17T04:00:16.184972+0000","last_clean":"2024-09-17T04:00:16.184972+0000","last_became_active":"2024-09-17T04:00:14.182626+0000","last_became_peered":"2024-09-17T04:00:14.182626+0000","last_unstale":"2024-09-17T04:00:16.184972+0000","last_undegraded":"2024-09-17T04:00:16.184972+0000","last_fullsized":"2024-09-17T04:00:16.184972+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:33.722613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976876+0000","last_change":"2024-09-17T04:00:15.185309+0000","last_active":"2024-09-17T04:00:17.976876+0000","last_peered":"2024-09-17T04:00:17.976876+0000","last_clean":"2024-09-17T04:00:17.976876+0000","last_became_active":"2024-09-17T04:00:15.184979+0000","last_became_peered":"2024-09-17T04:00:15.184979+0000","last_unstale":"2024-09-17T04:00:17.976876+0000","last_undegraded":"2024-09-17T04:00:17.976876+0000","last_fullsized":"2024-09-17T04:00:17.976876+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:29.123723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.976765+0000","last_change":"2024-09-17T04:00:14.172333+0000","last_active":"2024-09-17T04:00:17.976765+0000","last_peered":"2024-09-17T04:00:17.976765+0000","last_clean":"2024-09-17T04:00:17.976765+0000","last_became_active":"2024-09-17T04:00:14.170300+0000","last_became_peered":"2024-09-17T04:00:14.170300+0000","last_unstale":"2024-09-17T04:00:17.976765+0000","last_undegraded":"2024-09-17T04:00:17.976765+0000","last_fullsized":"2024-09-17T04:00:17.976765+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:36.599783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977992+0000","last_change":"2024-09-17T04:00:15.201456+0000","last_active":"2024-09-17T04:00:17.977992+0000","last_peered":"2024-09-17T04:00:17.977992+0000","last_clean":"2024-09-17T04:00:17.977992+0000","last_became_active":"2024-09-17T04:00:15.201067+0000","last_became_peered":"2024-09-17T04:00:15.201067+0000","last_unstale":"2024-09-17T04:00:17.977992+0000","last_undegraded":"2024-09-17T04:00:17.977992+0000","last_fullsized":"2024-09-17T04:00:17.977992+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:09:26.401458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.184947+0000","last_change":"2024-09-17T04:00:14.172200+0000","last_active":"2024-09-17T04:00:16.184947+0000","last_peered":"2024-09-17T04:00:16.184947+0000","last_clean":"2024-09-17T04:00:16.184947+0000","last_became_active":"2024-09-17T04:00:14.171692+0000","last_became_peered":"2024-09-17T04:00:14.171692+0000","last_unstale":"2024-09-17T04:00:16.184947+0000","last_undegraded":"2024-09-17T04:00:16.184947+0000","last_fullsized":"2024-09-17T04:00:16.184947+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:39.246219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978058+0000","last_change":"2024-09-17T04:00:15.207451+0000","last_active":"2024-09-17T04:00:17.978058+0000","last_peered":"2024-09-17T04:00:17.978058+0000","last_clean":"2024-09-17T04:00:17.978058+0000","last_became_active":"2024-09-17T04:00:15.207157+0000","last_became_peered":"2024-09-17T04:00:15.207157+0000","last_unstale":"2024-09-17T04:00:17.978058+0000","last_undegraded":"2024-09-17T04:00:17.978058+0000","last_fullsized":"2024-09-17T04:00:17.978058+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:33:09.116340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185026+0000","last_change":"2024-09-17T04:00:14.175785+0000","last_active":"2024-09-17T04:00:16.185026+0000","last_peered":"2024-09-17T04:00:16.185026+0000","last_clean":"2024-09-17T04:00:16.185026+0000","last_became_active":"2024-09-17T04:00:14.175439+0000","last_became_peered":"2024-09-17T04:00:14.175439+0000","last_unstale":"2024-09-17T04:00:16.185026+0000","last_undegraded":"2024-09-17T04:00:16.185026+0000","last_fullsized":"2024-09-17T04:00:16.185026+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:12.498724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977474+0000","last_change":"2024-09-17T04:00:15.196798+0000","last_active":"2024-09-17T04:00:17.977474+0000","last_peered":"2024-09-17T04:00:17.977474+0000","last_clean":"2024-09-17T04:00:17.977474+0000","last_became_active":"2024-09-17T04:00:15.191801+0000","last_became_peered":"2024-09-17T04:00:15.191801+0000","last_unstale":"2024-09-17T04:00:17.977474+0000","last_undegraded":"2024-09-17T04:00:17.977474+0000","last_fullsized":"2024-09-17T04:00:17.977474+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:14.033347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977378+0000","last_change":"2024-09-17T04:00:14.176595+0000","last_active":"2024-09-17T04:00:17.977378+0000","last_peered":"2024-09-17T04:00:17.977378+0000","last_clean":"2024-09-17T04:00:17.977378+0000","last_became_active":"2024-09-17T04:00:14.176223+0000","last_became_peered":"2024-09-17T04:00:14.176223+0000","last_unstale":"2024-09-17T04:00:17.977378+0000","last_undegraded":"2024-09-17T04:00:17.977378+0000","last_fullsized":"2024-09-17T04:00:17.977378+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:48.426501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.354147+0000","last_change":"2024-09-17T04:00:15.207803+0000","last_active":"2024-09-17T04:00:17.354147+0000","last_peered":"2024-09-17T04:00:17.354147+0000","last_clean":"2024-09-17T04:00:17.354147+0000","last_became_active":"2024-09-17T04:00:15.207487+0000","last_became_peered":"2024-09-17T04:00:15.207487+0000","last_unstale":"2024-09-17T04:00:17.354147+0000","last_undegraded":"2024-09-17T04:00:17.354147+0000","last_fullsized":"2024-09-17T04:00:17.354147+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:04.107520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.181255+0000","last_change":"2024-09-17T04:00:14.185208+0000","last_active":"2024-09-17T04:00:17.181255+0000","last_peered":"2024-09-17T04:00:17.181255+0000","last_clean":"2024-09-17T04:00:17.181255+0000","last_became_active":"2024-09-17T04:00:14.184192+0000","last_became_peered":"2024-09-17T04:00:14.184192+0000","last_unstale":"2024-09-17T04:00:17.181255+0000","last_undegraded":"2024-09-17T04:00:17.181255+0000","last_fullsized":"2024-09-17T04:00:17.181255+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:58.627208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":13,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:16.185221+0000","last_change":"2024-09-17T04:00:15.200216+0000","last_active":"2024-09-17T04:00:16.185221+0000","last_peered":"2024-09-17T04:00:16.185221+0000","last_clean":"2024-09-17T04:00:16.185221+0000","last_became_active":"2024-09-17T04:00:15.199102+0000","last_became_peered":"2024-09-17T04:00:15.199102+0000","last_unstale":"2024-09-17T04:00:16.185221+0000","last_undegraded":"2024-09-17T04:00:16.185221+0000","last_fullsized":"2024-09-17T04:00:16.185221+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":161,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:14.149927+0000","last_clean_scrub_stamp":"2024-09-17T04:00:14.149927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:46:45.614483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":15,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.978059+0000","last_change":"2024-09-17T04:00:14.179639+0000","last_active":"2024-09-17T04:00:17.978059+0000","last_peered":"2024-09-17T04:00:17.978059+0000","last_clean":"2024-09-17T04:00:17.978059+0000","last_became_active":"2024-09-17T04:00:14.179337+0000","last_became_peered":"2024-09-17T04:00:14.179337+0000","last_unstale":"2024-09-17T04:00:17.978059+0000","last_undegraded":"2024-09-17T04:00:17.978059+0000","last_fullsized":"2024-09-17T04:00:17.978059+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":160,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:13.143521+0000","last_clean_scrub_stamp":"2024-09-17T04:00:13.143521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:03.804071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":373,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-17T04:00:17.977429+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:17.977429+0000","last_peered":"2024-09-17T04:00:17.977429+0000","last_clean":"2024-09-17T04:00:17.977429+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:17.977429+0000","last_undegraded":"2024-09-17T04:00:17.977429+0000","last_fullsized":"2024-09-17T04:00:17.977429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509574,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37932,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333908,"statfs":{"total":96636764160,"available":96597921792,"internally_reserved":0,"allocated":2646016,"data_stored":1484798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509577,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41504,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330336,"statfs":{"total":96636764160,"available":96594264064,"internally_reserved":0,"allocated":2043904,"data_stored":894326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509576,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41520,"kb_used_data":2012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330320,"statfs":{"total":96636764160,"available":96594247680,"internally_reserved":0,"allocated":2060288,"data_stored":896520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509573,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":894520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509575,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37356,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334484,"statfs":{"total":96636764160,"available":96598511616,"internally_reserved":0,"allocated":2056192,"data_stored":895020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509575,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41516,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330324,"statfs":{"total":96636764160,"available":96594251776,"internally_reserved":0,"allocated":2056192,"data_stored":895938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542278,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41504,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330336,"statfs":{"total":96636764160,"available":96594264064,"internally_reserved":0,"allocated":2043904,"data_stored":894326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542279,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42092,"kb_used_data":2580,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329748,"statfs":{"total":96636764160,"available":96593661952,"internally_reserved":0,"allocated":2641920,"data_stored":1484716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:25.053 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:00:25.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:00:25.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 -- 172.21.15.129:0/2704977871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888156f20 msgr2=0x7f0888160d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 --2- 172.21.15.129:0/2704977871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888156f20 0x7f0888160d50 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f087c005fd0 tx=0x7f087c02a4a0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 -- 172.21.15.129:0/2704977871 shutdown_connections 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 --2- 172.21.15.129:0/2704977871 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 0x7f088806f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 --2- 172.21.15.129:0/2704977871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888156f20 0x7f0888160d50 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 -- 172.21.15.129:0/2704977871 >> 172.21.15.129:0/2704977871 conn(0x7f08880d0120 msgr2=0x7f08880d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 -- 172.21.15.129:0/2704977871 shutdown_connections 2024-09-17T04:00:25.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.318+0000 7f088e200640 1 -- 172.21.15.129:0/2704977871 wait complete. 2024-09-17T04:00:25.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.321+0000 7f088e200640 1 Processor -- start 2024-09-17T04:00:25.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.321+0000 7f088e200640 1 -- start start 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 0x7f08880c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888156f20 0x7f08880c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f08880cf450 con 0x7f0888072ee0 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f08880cf2d0 con 0x7f0888156f20 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f088e200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f08880cf5d0 con 0x7f088806ee70 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f0887e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.322+0000 7f0887e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36090/0 (socket says 172.21.15.129:36090) 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 -- 172.21.15.129:0/1556858518 learned_addr learned my addr 172.21.15.129:0/1556858518 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f088ce00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 0x7f08880c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f088d800640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888156f20 0x7f08880c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 -- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 msgr2=0x7f08880c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 0x7f08880c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 -- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888156f20 msgr2=0x7f08880c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888156f20 0x7f08880c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 -- 172.21.15.129:0/1556858518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08881f9810 con 0x7f0888072ee0 2024-09-17T04:00:25.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0887e00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f087400c940 tx=0x7f087400ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0874007bb0 con 0x7f0888072ee0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08881f9af0 con 0x7f0888072ee0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.323+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0874007e40 con 0x7f0888072ee0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.324+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087400f6b0 con 0x7f0888072ee0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.324+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08881f5a40 con 0x7f0888072ee0 2024-09-17T04:00:25.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.325+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0888073c80 con 0x7f0888072ee0 2024-09-17T04:00:25.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.326+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f087400f850 con 0x7f0888072ee0 2024-09-17T04:00:25.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.327+0000 7f0884c00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 0x7f086807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.327+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f087409b460 con 0x7f0888072ee0 2024-09-17T04:00:25.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.327+0000 7f088ce00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 0x7f086807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.328+0000 7f088ce00640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 0x7f086807b0a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f087c033000 tx=0x7f087c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:25.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.331+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0874014030 con 0x7f0888072ee0 2024-09-17T04:00:25.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.584+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f08881f95a0 con 0x7f0888072ee0 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.587+0000 7f0884c00640 1 -- 172.21.15.129:0/1556858518 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v38) ==== 190+0+147 (secure 0 0 0) 0x7f08740638a0 con 0x7f0888072ee0 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:00:25.587 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:00:25.588 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 msgr2=0x7f086807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 0x7f086807b0a0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f087c033000 tx=0x7f087c004730 comp rx=0 tx=0).stop 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 msgr2=0x7f08881f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f087400c940 tx=0x7f087400ce00 comp rx=0 tx=0).stop 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 shutdown_connections 2024-09-17T04:00:25.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0868078b90 0x7f086807b0a0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0888156f20 0x7f08880c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0888072ee0 0x7f08881f6290 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 --2- 172.21.15.129:0/1556858518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f088806ee70 0x7f08880c82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 >> 172.21.15.129:0/1556858518 conn(0x7f08880d0120 msgr2=0x7f08880732f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 shutdown_connections 2024-09-17T04:00:25.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.590+0000 7f088e200640 1 -- 172.21.15.129:0/1556858518 wait complete. 2024-09-17T04:00:25.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.839+0000 7f118c600640 1 -- 172.21.15.129:0/1362794871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840b3b90 msgr2=0x7f11840cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.839+0000 7f118c600640 1 --2- 172.21.15.129:0/1362794871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840b3b90 0x7f11840cbbf0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f1170009830 tx=0x7f117002a5c0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.839+0000 7f118c600640 1 -- 172.21.15.129:0/1362794871 shutdown_connections 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.839+0000 7f118c600640 1 --2- 172.21.15.129:0/1362794871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840b3b90 0x7f11840cbbf0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.839+0000 7f118c600640 1 -- 172.21.15.129:0/1362794871 >> 172.21.15.129:0/1362794871 conn(0x7f118406d410 msgr2=0x7f11840b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.840+0000 7f118c600640 1 -- 172.21.15.129:0/1362794871 shutdown_connections 2024-09-17T04:00:25.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.840+0000 7f118c600640 1 -- 172.21.15.129:0/1362794871 wait complete. 2024-09-17T04:00:25.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.842+0000 7f118c600640 1 Processor -- start 2024-09-17T04:00:25.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.842+0000 7f118c600640 1 -- start start 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11840b2950 0x7f11841596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 0x7f11841e67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f118406f5e0 con 0x7f11840ce170 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1184070910 con 0x7f11840b2950 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.843+0000 7f118c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f118406f760 con 0x7f11841e63d0 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36102/0 (socket says 172.21.15.129:36102) 2024-09-17T04:00:25.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 -- 172.21.15.129:0/1268378932 learned_addr learned my addr 172.21.15.129:0/1268378932 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189c00640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11840b2950 0x7f11841596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f118a600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 0x7f11841e67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 -- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 msgr2=0x7f11841e67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 0x7f11841e67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 -- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11840b2950 msgr2=0x7f11841596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11840b2950 0x7f11841596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 -- 172.21.15.129:0/1268378932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11841e6d20 con 0x7f11840ce170 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f118a600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 0x7f11841e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.844+0000 7f1189200640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f117002abb0 tx=0x7f1170033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.845+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170031b60 con 0x7f11840ce170 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.845+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1170031df0 con 0x7f11840ce170 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.845+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170005280 con 0x7f11840ce170 2024-09-17T04:00:25.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.845+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11841e4a50 con 0x7f11840ce170 2024-09-17T04:00:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.845+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11841e5010 con 0x7f11840ce170 2024-09-17T04:00:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.846+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11840cbbf0 con 0x7f11840ce170 2024-09-17T04:00:25.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.847+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1170005420 con 0x7f11840ce170 2024-09-17T04:00:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.848+0000 7f1181600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 0x7f116407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.848+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f11700bbfb0 con 0x7f11840ce170 2024-09-17T04:00:25.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.849+0000 7f1189c00640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 0x7f116407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:25.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.849+0000 7f1189c00640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 0x7f116407b0a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f11780015e0 tx=0x7f1178009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:25.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:25.852+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11841e6d20 con 0x7f11840ce170 2024-09-17T04:00:26.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.072+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f118406a140 con 0x7f11840ce170 2024-09-17T04:00:26.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.073+0000 7f1181600640 1 -- 172.21.15.129:0/1268378932 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f117003f030 con 0x7f11840ce170 2024-09-17T04:00:26.074 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:26.074 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 msgr2=0x7f116407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 0x7f116407b0a0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f11780015e0 tx=0x7f1178009040 comp rx=0 tx=0).stop 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 msgr2=0x7f11841e4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f117002abb0 tx=0x7f1170033c10 comp rx=0 tx=0).stop 2024-09-17T04:00:26.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 shutdown_connections 2024-09-17T04:00:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1164078b90 0x7f116407b0a0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f11841e63d0 0x7f11841e67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11840ce170 0x7f11841e4070 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 --2- 172.21.15.129:0/1268378932 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f11840b2950 0x7f11841596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 >> 172.21.15.129:0/1268378932 conn(0x7f118406d410 msgr2=0x7f11840cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 shutdown_connections 2024-09-17T04:00:26.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.075+0000 7f118c600640 1 -- 172.21.15.129:0/1268378932 wait complete. 2024-09-17T04:00:26.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 144, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:00:14.652383+0000', 'modified': '2024-09-17T04:00:17.717373+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8542}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8542': {'gid': 8542, 'name': 'd', 'rank': 0, 'incarnation': 140, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/1344901561', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 1344901561}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 1344901561}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8542, 'qdb_cluster': [8542]} 2024-09-17T04:00:26.501 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:00:26.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T04:00:26.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 -- 172.21.15.129:0/286166454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40b4140 msgr2=0x7f03c40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 --2- 172.21.15.129:0/286166454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40b4140 0x7f03c40ccc80 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f03c4072e00 tx=0x7f03ac02a600 comp rx=0 tx=0).stop 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 -- 172.21.15.129:0/286166454 shutdown_connections 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 --2- 172.21.15.129:0/286166454 >> v2:172.21.15.181:3300/0 conn(0x7f03c40cd1c0 0x7f03c41609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 --2- 172.21.15.129:0/286166454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40b4140 0x7f03c40ccc80 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 -- 172.21.15.129:0/286166454 >> 172.21.15.129:0/286166454 conn(0x7f03c406d410 msgr2=0x7f03c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 -- 172.21.15.129:0/286166454 shutdown_connections 2024-09-17T04:00:26.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.723+0000 7f03caa00640 1 -- 172.21.15.129:0/286166454 wait complete. 2024-09-17T04:00:26.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.725+0000 7f03caa00640 1 Processor -- start 2024-09-17T04:00:26.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.725+0000 7f03caa00640 1 -- start start 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03c40705a0 0x7f03c40c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c40b4140 0x7f03c40ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03c4160260 con 0x7f03c40cd1c0 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41600e0 con 0x7f03c40b4140 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03caa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f03c4160f00 con 0x7f03c40705a0 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36108/0 (socket says 172.21.15.129:36108) 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03c8a00640 1 -- 172.21.15.129:0/737157641 learned_addr learned my addr 172.21.15.129:0/737157641 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03bbe00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c40b4140 0x7f03c40ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.727+0000 7f03c3e00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03c40705a0 0x7f03c40c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 -- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03c40705a0 msgr2=0x7f03c40c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03c40705a0 0x7f03c40c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 -- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c40b4140 msgr2=0x7f03c40ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c40b4140 0x7f03c40ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 -- 172.21.15.129:0/737157641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03c415ba80 con 0x7f03c40cd1c0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c8a00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00ca90 tx=0x7f03bc00cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c0c00640 1 -- 172.21.15.129:0/737157641 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc007df0 con 0x7f03c40cd1c0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c0c00640 1 -- 172.21.15.129:0/737157641 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03bc00f450 con 0x7f03c40cd1c0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03c0c00640 1 -- 172.21.15.129:0/737157641 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00f8c0 con 0x7f03c40cd1c0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.728+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03c40c8740 con 0x7f03c40cd1c0 2024-09-17T04:00:26.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.729+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03c415c040 con 0x7f03c40cd1c0 2024-09-17T04:00:26.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.731+0000 7f03c0c00640 1 -- 172.21.15.129:0/737157641 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f03bc00fa60 con 0x7f03c40cd1c0 2024-09-17T04:00:26.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.732+0000 7f03c0c00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 0x7f03a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.732+0000 7f03c0c00640 1 -- 172.21.15.129:0/737157641 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f03bc09b7c0 con 0x7f03c40cd1c0 2024-09-17T04:00:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.732+0000 7f03c3e00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 0x7f03a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.733+0000 7f03d1a15740 1 Processor -- start 2024-09-17T04:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.733+0000 7f03d1a15740 1 -- start start 2024-09-17T04:00:26.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.733+0000 7f03c3e00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 0x7f03a007b0a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f03b4001720 tx=0x7f03b4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f815970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f7e5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56007f76f430 con 0x56007f41c3b0 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56007f75f5d0 con 0x56007f424590 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.734+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56007f74ea00 con 0x56007f445270 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03c3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f815970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36110/0 (socket says 172.21.15.129:36110) 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 -- 172.21.15.129:0/2389021226 learned_addr learned my addr 172.21.15.129:0/2389021226 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03c8a00640 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f7e5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 -- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 msgr2=0x56007f815970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f815970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 -- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 msgr2=0x56007f7e5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f7e5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.735+0000 7f03bbe00640 1 -- 172.21.15.129:0/2389021226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007f717520 con 0x56007f41c3b0 2024-09-17T04:00:26.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.736+0000 7f03bbe00640 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f03ac034040 tx=0x7f03ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9eddffef89a1de server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.736+0000 7f03b3e00640 1 -- 172.21.15.129:0/2389021226 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac040070 con 0x56007f41c3b0 2024-09-17T04:00:26.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.736+0000 7f03b3e00640 1 -- 172.21.15.129:0/2389021226 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03ac0043b0 con 0x56007f41c3b0 2024-09-17T04:00:26.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.736+0000 7f03b3e00640 1 -- 172.21.15.129:0/2389021226 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac035040 con 0x56007f41c3b0 2024-09-17T04:00:26.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 -- 172.21.15.129:0/2389021226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 msgr2=0x56007f815eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f03ac034040 tx=0x7f03ac0046c0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 -- 172.21.15.129:0/2389021226 shutdown_connections 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f7e5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f815eb0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2389021226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f815970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 -- 172.21.15.129:0/2389021226 >> 172.21.15.129:0/2389021226 conn(0x56007f4393c0 msgr2=0x56007f7eb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 -- 172.21.15.129:0/2389021226 shutdown_connections 2024-09-17T04:00:26.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.737+0000 7f03d1a15740 1 -- 172.21.15.129:0/2389021226 wait complete. 2024-09-17T04:00:26.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.738+0000 7f03d1a15740 1 -- start start 2024-09-17T04:00:26.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.738+0000 7f03d1a15740 1 Processor -- start 2024-09-17T04:00:26.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.739+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f41c3b0 0x56007f80f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.739+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f82a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.739+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.739+0000 7f03d1a15740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56007f6f5330 con 0x56007f445270 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56007f752810 con 0x56007f424590 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56007f4bf620 con 0x56007f41c3b0 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03d1a15740 10 client.0 _ll_register_callbacks cb 0x56007f6fa420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f41c3b0 0x56007f80f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36126/0 (socket says 172.21.15.129:36126) 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 -- 172.21.15.129:0/1864925190 learned_addr learned my addr 172.21.15.129:0/1864925190 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03bbe00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f82a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 -- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f41c3b0 msgr2=0x56007f80f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f41c3b0 0x56007f80f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 -- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 msgr2=0x56007f82a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f82a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 -- 172.21.15.129:0/1864925190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007f4b0850 con 0x56007f445270 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03c8a00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00a820 tx=0x7f03bc0635d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508a46750a079608 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.740+0000 7f03b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc06ce80 con 0x56007f445270 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f03bc062770 con 0x56007f445270 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03ac0043b0 con 0x56007f445270 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f03bc062970 con 0x56007f445270 2024-09-17T04:00:26.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 10 client.9267 fetch_fsmap learned FSMap version 144 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) -- 0x56007f5276b0 con 0x56007f445270 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03ac0043b0 con 0x56007f445270 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 144) ==== 3502+0+0 (secure 0 0 0) 0x7f03bc05ee60 con 0x56007f445270 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 10 client.9267 fetch_fsmap finished waiting for FSMap version 144 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 10 client.9267 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 20 client.9267 populate_metadata read hostname 'smithi129' 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f403e40 0x56007f840050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 4 client.9267 mds_command: new command op to 8542 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.741+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 --> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56007f761520 con 0x56007f403e40 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.742+0000 7f03c3e00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f403e40 0x56007f840050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.742+0000 7f03c3e00640 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f403e40 0x56007f840050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9878c2f75eaf69fc server_cookie=5c9eaac1f74007ae in_seq=0 out_seq=0 2024-09-17T04:00:26.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.742+0000 7f03b0c00640 10 client.9267 ms_handle_connect on v2:172.21.15.181:6835/1344901561 2024-09-17T04:00:26.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03b0c00640 1 -- 172.21.15.129:0/1864925190 <== mds.0 v2:172.21.15.181:6835/1344901561 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56007f761520 con 0x56007f403e40 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03b0c00640 10 client.9267 handle_command_reply: tid=0 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 client.9267 shutdown 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 msgr2=0x56007f843480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00a820 tx=0x7f03bc0635d0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 shutdown_connections 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f403e40 0x56007f840050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f843480 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f82a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1864925190 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f41c3b0 0x56007f80f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03b0c00640 0 client.9267 ms_handle_reset on v2:172.21.15.181:6835/1344901561 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 >> 172.21.15.129:0/1864925190 conn(0x56007f4393c0 msgr2=0x56007f830e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 shutdown_connections 2024-09-17T04:00:26.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.744+0000 7f03d1a15740 1 -- 172.21.15.129:0/1864925190 wait complete. 2024-09-17T04:00:26.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.745+0000 7f03d1a15740 20 client.9267 trim_cache size 0 max 16384 2024-09-17T04:00:26.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.763+0000 7f03d1a15740 1 Processor -- start 2024-09-17T04:00:26.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.763+0000 7f03d1a15740 1 -- start start 2024-09-17T04:00:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f8122c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f812800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03c415d2c0 con 0x56007f41c3b0 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f03c41f5d50 con 0x56007f424590 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f03c41f63a0 con 0x56007f445270 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36132/0 (socket says 172.21.15.129:36132) 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 -- 172.21.15.129:0/1033173813 learned_addr learned my addr 172.21.15.129:0/1033173813 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03bbe00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f8122c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c8a00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f812800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 -- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 msgr2=0x56007f812800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f812800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 -- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 msgr2=0x56007f8122c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f8122c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03c3e00640 1 -- 172.21.15.129:0/1033173813 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56007f834b00 con 0x56007f41c3b0 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.765+0000 7f03bbe00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f8122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:26.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.766+0000 7f03c3e00640 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x56007f8280a0 tx=0x7f03b400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba6a70145a36f8b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.766+0000 7f03b3400640 1 -- 172.21.15.129:0/1033173813 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b402f070 con 0x56007f41c3b0 2024-09-17T04:00:26.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.766+0000 7f03b3400640 1 -- 172.21.15.129:0/1033173813 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03b401b040 con 0x56007f41c3b0 2024-09-17T04:00:26.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.766+0000 7f03b3400640 1 -- 172.21.15.129:0/1033173813 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03b4023c80 con 0x56007f41c3b0 2024-09-17T04:00:26.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 -- 172.21.15.129:0/1033173813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 msgr2=0x56007f82a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x56007f8280a0 tx=0x7f03b400a000 comp rx=0 tx=0).stop 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 -- 172.21.15.129:0/1033173813 shutdown_connections 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f445270 0x56007f812800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f424590 0x56007f8122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 --2- 172.21.15.129:0/1033173813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f41c3b0 0x56007f82a240 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 -- 172.21.15.129:0/1033173813 >> 172.21.15.129:0/1033173813 conn(0x56007f4393c0 msgr2=0x56007f7e1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 -- 172.21.15.129:0/1033173813 shutdown_connections 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.767+0000 7f03d1a15740 1 -- 172.21.15.129:0/1033173813 wait complete. 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.768+0000 7f03d1a15740 1 -- start start 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.768+0000 7f03d1a15740 1 Processor -- start 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f41c3b0 0x56007f852930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f424590 0x56007f852e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56007f828740 con 0x56007f445270 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56007f811800 con 0x56007f41c3b0 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56007f827c70 con 0x56007f424590 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.769+0000 7f03d1a15740 10 client.0 _ll_register_callbacks cb 0x7f03bc062c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36144/0 (socket says 172.21.15.129:36144) 2024-09-17T04:00:26.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 -- 172.21.15.129:0/2361653668 learned_addr learned my addr 172.21.15.129:0/2361653668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03bbe00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f424590 0x56007f852e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c3e00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f41c3b0 0x56007f852930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 -- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f424590 msgr2=0x56007f852e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f424590 0x56007f852e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 -- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f41c3b0 msgr2=0x56007f852930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f41c3b0 0x56007f852930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 -- 172.21.15.129:0/2361653668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56007f7ec9c0 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03c8a00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f03bc05f5c0 tx=0x7f03bc067fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2874f28589e917e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.770+0000 7f03b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc01d040 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f03bc085930 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03b401b040 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=144) ==== 24+0+0 (secure 0 0 0) 0x7f03bc06ce80 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 10 client.9273 fetch_fsmap learned FSMap version 144 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=144,monmap=2+}) -- 0x56007f839e10 con 0x56007f445270 2024-09-17T04:00:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f03b401b040 con 0x56007f445270 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 144) ==== 3502+0+0 (secure 0 0 0) 0x7f03bc0690d0 con 0x56007f445270 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 10 client.9273 fetch_fsmap finished waiting for FSMap version 144 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 10 client.9273 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 20 client.9273 populate_metadata read hostname 'smithi129' 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f8aff30 0x56007f8b0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 4 client.9273 mds_command: new command op to 8542 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.771+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 --> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] -- command(tid 0: {"prefix": "status"}) -- 0x56007f75cfd0 con 0x56007f8aff30 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.772+0000 7f03c3e00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f8aff30 0x56007f8b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.772+0000 7f03c3e00640 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f8aff30 0x56007f8b0340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2491d2bab0fd0238 server_cookie=f552581a8ce6d4ef in_seq=0 out_seq=0 2024-09-17T04:00:26.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.772+0000 7f03b0c00640 10 client.9273 ms_handle_connect on v2:172.21.15.181:6835/1344901561 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.772+0000 7f03b0c00640 1 -- 172.21.15.129:0/2361653668 <== mds.0 v2:172.21.15.181:6835/1344901561 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x56007f75cfd0 con 0x56007f8aff30 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.772+0000 7f03b0c00640 10 client.9273 handle_command_reply: tid=0 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 client.9273 shutdown 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 msgr2=0x56007f8b3e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f03bc05f5c0 tx=0x7f03bc067fc0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 shutdown_connections 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:6835/1344901561,v1:172.21.15.181:6837/1344901561] conn(0x56007f8aff30 0x56007f8b0340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56007f445270 0x56007f8b3e00 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56007f424590 0x56007f852e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 --2- 172.21.15.129:0/2361653668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56007f41c3b0 0x56007f852930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03b0c00640 0 client.9273 ms_handle_reset on v2:172.21.15.181:6835/1344901561 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 >> 172.21.15.129:0/2361653668 conn(0x56007f4393c0 msgr2=0x56007f7e14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 shutdown_connections 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 1 -- 172.21.15.129:0/2361653668 wait complete. 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.773+0000 7f03d1a15740 20 client.9273 trim_cache size 0 max 16384 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 8542, 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:00:26.783 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.605120943999999, 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 144, 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 163, 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 163, 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.631532297 2024-09-17T04:00:26.784 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 msgr2=0x7f03a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 0x7f03a007b0a0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f03b4001720 tx=0x7f03b4009040 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 msgr2=0x7f03c40cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00ca90 tx=0x7f03bc00cf50 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 shutdown_connections 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f03a0078b90 0x7f03a007b0a0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03c40cd1c0 0x7f03c40cbeb0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03c40b4140 0x7f03c40ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 --2- 172.21.15.129:0/737157641 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03c40705a0 0x7f03c40c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 >> 172.21.15.129:0/737157641 conn(0x7f03c406d410 msgr2=0x7f03c40ae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:26.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 shutdown_connections 2024-09-17T04:00:26.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:26.774+0000 7f03caa00640 1 -- 172.21.15.129:0/737157641 wait complete. 2024-09-17T04:00:26.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:00:27.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 -- 172.21.15.129:0/3430835262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e8073ac0 msgr2=0x7fc1e80b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:27.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/3430835262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e8073ac0 0x7fc1e80b3ba0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0005fd0 tx=0x7fc1d002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 -- 172.21.15.129:0/3430835262 shutdown_connections 2024-09-17T04:00:27.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/3430835262 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e8073ac0 0x7fc1e80b3ba0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/3430835262 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b64c0 0x7fc1e8073580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 -- 172.21.15.129:0/3430835262 >> 172.21.15.129:0/3430835262 conn(0x7fc1e80d0970 msgr2=0x7fc1e80d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:27.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 -- 172.21.15.129:0/3430835262 shutdown_connections 2024-09-17T04:00:27.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.019+0000 7fc1e7e00640 1 -- 172.21.15.129:0/3430835262 wait complete. 2024-09-17T04:00:27.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.021+0000 7fc1e7e00640 1 Processor -- start 2024-09-17T04:00:27.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.021+0000 7fc1e7e00640 1 -- start start 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1e8073ac0 0x7fc1e8155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b40e0 0x7fc1e81e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80b4c00 con 0x7fc1e80b64c0 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e815cc00 con 0x7fc1e8073ac0 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80b4d80 con 0x7fc1e80b40e0 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36158/0 (socket says 172.21.15.129:36158) 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 -- 172.21.15.129:0/1485105589 learned_addr learned my addr 172.21.15.129:0/1485105589 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e6a00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1e8073ac0 0x7fc1e8155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:27.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1de000640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b40e0 0x7fc1e81e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 -- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b40e0 msgr2=0x7fc1e81e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b40e0 0x7fc1e81e1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.023+0000 7fc1e7400640 1 -- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1e8073ac0 msgr2=0x7fc1e8155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1e7400640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1e8073ac0 0x7fc1e8155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1e7400640 1 -- 172.21.15.129:0/1485105589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e81f8ee0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1e7400640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c6c0 tx=0x7fc1e000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0003b70 con 0x7fc1e80b64c0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1e0003d30 con 0x7fc1e80b64c0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e81f93e0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0005330 con 0x7fc1e80b64c0 2024-09-17T04:00:27.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.024+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e81f99a0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.025+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e81f9150 con 0x7fc1e80b64c0 2024-09-17T04:00:27.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.027+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc1e00054d0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.027+0000 7fc1df400640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 0x7fc1c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:27.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.028+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fc1e009abc0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.028+0000 7fc1e6a00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 0x7fc1c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:27.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.029+0000 7fc1e6a00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 0x7fc1c407b0a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009ac0 tx=0x7fc1d8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:27.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.032+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc1e8073ed0 con 0x7fc1e80b64c0 2024-09-17T04:00:27.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.248+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc1e80b2130 con 0x7fc1e80b64c0 2024-09-17T04:00:27.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.248+0000 7fc1df400640 1 -- 172.21.15.129:0/1485105589 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v144) ==== 53+0+73 (secure 0 0 0) 0x7fc1e0062f60 con 0x7fc1e80b64c0 2024-09-17T04:00:27.248 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.250+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 msgr2=0x7fc1c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.250+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 0x7fc1c407b0a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009ac0 tx=0x7fc1d8005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 msgr2=0x7fc1e81e1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000c6c0 tx=0x7fc1e000cb80 comp rx=0 tx=0).stop 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 shutdown_connections 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1c4078b90 0x7fc1c407b0a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc1e80b64c0 0x7fc1e81e1fd0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc1e80b40e0 0x7fc1e81e1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 --2- 172.21.15.129:0/1485105589 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc1e8073ac0 0x7fc1e8155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:27.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 >> 172.21.15.129:0/1485105589 conn(0x7fc1e80d0970 msgr2=0x7fc1e80b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:27.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 shutdown_connections 2024-09-17T04:00:27.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27.251+0000 7fc1e7e00640 1 -- 172.21.15.129:0/1485105589 wait complete. 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:00:27.273 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:00:27.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.274 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:00:27.310 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:00:27.311 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42065sec preferred_lft 42065sec 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:00:27.312 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:00:27.313 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:00:27.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.314 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:00:27.404 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:00:27.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.407 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:00:27.444 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:00:27.444 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:00:27.444 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:00:27.444 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:00:27.444 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:00:27.445 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:00:27.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:00:27.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:00:27.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.640 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:00:27.676 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:00:27.676 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:00:27.677 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:00:27.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:00:27.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:00:27.773 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:00:27.809 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:00:27.809 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:00:27.809 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:00:27.884 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:00:27.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:27.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:00:28.079 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:00:28.079 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:00:28.079 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:00:28.099 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:00:28.100 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:00:28.100 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:00:28.100 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:00:28.100 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:00:28.135 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:00:28.136 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:00:28.137 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:00:28.137 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:00:28.138 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:00:28.138 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:00:28.140 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:00:28.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:28.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:00:28.196 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:00:28.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:00:28.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:00:28.297 INFO:teuthology.orchestra.run.smithi129.stdout:client7547 2024-09-17T04:00:28.298 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:28.298 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:28.334 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:28.334 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:28.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:28.411 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file_a bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T04:00:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T04:00:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T04:00:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.126085 s, 66.5 MB/s 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_a").st_ino) 2024-09-17T04:00:28.621 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:00:28.720 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627776 2024-09-17T04:00:28.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:00:28.732 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:28.732 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:28.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:28.808 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file_b bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T04:00:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T04:00:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T04:00:28.985 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.10019 s, 83.7 MB/s 2024-09-17T04:00:28.986 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:00:28.987 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:00:28.987 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T04:00:28.987 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:00:28.987 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/file_b").st_ino) 2024-09-17T04:00:28.987 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:00:29.086 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627777 2024-09-17T04:00:29.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:00:29.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:29.098 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln file_b linkto_b) 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/linkto_b").st_ino) 2024-09-17T04:00:29.177 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:00:29.316 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627777 2024-09-17T04:00:29.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:00:29.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:29.328 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv file_a file_b) 2024-09-17T04:00:29.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:29.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 -- 172.21.15.129:0/1295743459 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1134004d40 msgr2=0x7f1134005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:29.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 --2- 172.21.15.129:0/1295743459 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1134004d40 0x7f1134005150 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f1148005fd0 tx=0x7f1134005830 comp rx=0 tx=0).stop 2024-09-17T04:00:29.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f114f400640 1 -- 172.21.15.129:0/1295743459 reap_dead start 2024-09-17T04:00:29.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 -- 172.21.15.129:0/1295743459 shutdown_connections 2024-09-17T04:00:29.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 -- 172.21.15.129:0/1295743459 >> 172.21.15.129:0/1295743459 conn(0x7f115006d410 msgr2=0x7f11500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:29.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 -- 172.21.15.129:0/1295743459 shutdown_connections 2024-09-17T04:00:29.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.674+0000 7f1155800640 1 -- 172.21.15.129:0/1295743459 wait complete. 2024-09-17T04:00:29.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.676+0000 7f1155800640 1 Processor -- start 2024-09-17T04:00:29.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.677+0000 7f1155800640 1 -- start start 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 0x7f1150159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 0x7f11500c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f11500cd3c0 con 0x7f11500ad8f0 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f11500cd240 con 0x7f1134004d40 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f1155800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f11500cd540 con 0x7f1150155f00 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f114f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f114f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36172/0 (socket says 172.21.15.129:36172) 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.678+0000 7f114f400640 1 -- 172.21.15.129:0/41562255 learned_addr learned my addr 172.21.15.129:0/41562255 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:29.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114ea00640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 0x7f1150159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114e000640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 0x7f11500c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 -- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 msgr2=0x7f11500c9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 0x7f11500c9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 -- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 msgr2=0x7f1150159940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 0x7f1150159940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 -- 172.21.15.129:0/41562255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11500ae240 con 0x7f11500ad8f0 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114ea00640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 0x7f1150159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114e000640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 0x7f11500c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:29.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f114f400640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f114802aea0 tx=0x7f1148033bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148042070 con 0x7f11500ad8f0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1148003970 con 0x7f11500ad8f0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1150159010 con 0x7f11500ad8f0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148038040 con 0x7f11500ad8f0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.679+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11501f6200 con 0x7f11500ad8f0 2024-09-17T04:00:29.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.680+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11500c8200 con 0x7f11500ad8f0 2024-09-17T04:00:29.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.682+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1148004030 con 0x7f11500ad8f0 2024-09-17T04:00:29.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.683+0000 7f1146a00640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 0x7f113007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:29.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.683+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f11480bb330 con 0x7f11500ad8f0 2024-09-17T04:00:29.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.683+0000 7f114ea00640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 0x7f113007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:29.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.684+0000 7f114ea00640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 0x7f113007b0a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1140009b10 tx=0x7f1140005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:29.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.687+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f11500ae240 con 0x7f11500ad8f0 2024-09-17T04:00:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.904+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1150159f20 con 0x7f11500ad8f0 2024-09-17T04:00:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.904+0000 7f1146a00640 1 -- 172.21.15.129:0/41562255 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f1148083000 con 0x7f11500ad8f0 2024-09-17T04:00:29.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:29.904 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 msgr2=0x7f113007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 0x7f113007b0a0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f1140009b10 tx=0x7f1140005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 msgr2=0x7f11500add00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f114802aea0 tx=0x7f1148033bf0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 shutdown_connections 2024-09-17T04:00:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1130078b90 0x7f113007b0a0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f11500ad8f0 0x7f11500add00 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1150155f00 0x7f11500c9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 --2- 172.21.15.129:0/41562255 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1134004d40 0x7f1150159940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 >> 172.21.15.129:0/41562255 conn(0x7f115006d410 msgr2=0x7f11500af070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 shutdown_connections 2024-09-17T04:00:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:29.907+0000 7f1155800640 1 -- 172.21.15.129:0/41562255 wait complete. 2024-09-17T04:00:29.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:00:30.184 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T04:00:30.184 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T04:00:30.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:30.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 -- 172.21.15.129:0/4042046709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 msgr2=0x7fbd800cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 --2- 172.21.15.129:0/4042046709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd800cf6c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c009860 tx=0x7fbd6c02a560 comp rx=0 tx=0).stop 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 -- 172.21.15.129:0/4042046709 shutdown_connections 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 --2- 172.21.15.129:0/4042046709 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd800cf6c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 -- 172.21.15.129:0/4042046709 >> 172.21.15.129:0/4042046709 conn(0x7fbd8006d410 msgr2=0x7fbd800b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 -- 172.21.15.129:0/4042046709 shutdown_connections 2024-09-17T04:00:30.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.408+0000 7fbd87400640 1 -- 172.21.15.129:0/4042046709 wait complete. 2024-09-17T04:00:30.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.410+0000 7fbd87400640 1 Processor -- start 2024-09-17T04:00:30.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.410+0000 7fbd87400640 1 -- start start 2024-09-17T04:00:30.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.411+0000 7fbd87400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:30.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd87400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd800ccc80 0x7fbd801f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:30.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd87400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd8009d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:30.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd87400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80160cb0 con 0x7fbd80099570 2024-09-17T04:00:30.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd87400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80160b30 con 0x7fbd800cf2b0 2024-09-17T04:00:30.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd87400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbd80160e30 con 0x7fbd800ccc80 2024-09-17T04:00:30.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:30.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36182/0 (socket says 172.21.15.129:36182) 2024-09-17T04:00:30.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 -- 172.21.15.129:0/1796318826 learned_addr learned my addr 172.21.15.129:0/1796318826 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:30.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd7fe00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd800ccc80 0x7fbd801f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:30.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd85400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd8009d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 -- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd800ccc80 msgr2=0x7fbd801f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd800ccc80 0x7fbd801f9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 -- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 msgr2=0x7fbd8009d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd8009d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.412+0000 7fbd84a00640 1 -- 172.21.15.129:0/1796318826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd800c9e60 con 0x7fbd80099570 2024-09-17T04:00:30.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd84a00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c3c0 tx=0x7fbd7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74017070 con 0x7fbd80099570 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd740035f0 con 0x7fbd80099570 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74003a80 con 0x7fbd80099570 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8015e000 con 0x7fbd80099570 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.413+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8015e5c0 con 0x7fbd80099570 2024-09-17T04:00:30.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.414+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd800b2750 con 0x7fbd80099570 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.415+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbd7401a070 con 0x7fbd80099570 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.416+0000 7fbd7cc00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 0x7fbd6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.416+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fbd74099fb0 con 0x7fbd80099570 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.417+0000 7fbd7fe00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 0x7fbd6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.417+0000 7fbd7fe00640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 0x7fbd6007b0a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c034000 tx=0x7fbd6c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:30.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.420+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbd800b2750 con 0x7fbd80099570 2024-09-17T04:00:30.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.637+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd800cf6c0 con 0x7fbd80099570 2024-09-17T04:00:30.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.638+0000 7fbd7cc00640 1 -- 172.21.15.129:0/1796318826 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7fbd74061d90 con 0x7fbd80099570 2024-09-17T04:00:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:30.637 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:30.638 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.640+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 msgr2=0x7fbd6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.640+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 0x7fbd6007b0a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c034000 tx=0x7fbd6c004730 comp rx=0 tx=0).stop 2024-09-17T04:00:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.640+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 msgr2=0x7fbd800c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.640+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c3c0 tx=0x7fbd7400c880 comp rx=0 tx=0).stop 2024-09-17T04:00:30.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 shutdown_connections 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd60078b90 0x7fbd6007b0a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd800cf2b0 0x7fbd8009d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd800ccc80 0x7fbd801f9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 --2- 172.21.15.129:0/1796318826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd80099570 0x7fbd800c9530 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 >> 172.21.15.129:0/1796318826 conn(0x7fbd8006d410 msgr2=0x7fbd8006eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 shutdown_connections 2024-09-17T04:00:30.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:30.641+0000 7fbd87400640 1 -- 172.21.15.129:0/1796318826 wait complete. 2024-09-17T04:00:30.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_created 2024-09-17T04:00:30.899 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T04:00:30.900 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T04:00:30.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:31.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 -- 172.21.15.129:0/3194910257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40cc260 msgr2=0x7f02a40b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 --2- 172.21.15.129:0/3194910257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40cc260 0x7f02a40b33f0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f02a406eb20 tx=0x7f028c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 -- 172.21.15.129:0/3194910257 shutdown_connections 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 --2- 172.21.15.129:0/3194910257 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40cc260 0x7f02a40b33f0 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 --2- 172.21.15.129:0/3194910257 >> v2:172.21.15.181:3300/0 conn(0x7f02a40706a0 0x7f02a40cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 -- 172.21.15.129:0/3194910257 >> 172.21.15.129:0/3194910257 conn(0x7f02a406d410 msgr2=0x7f02a40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 -- 172.21.15.129:0/3194910257 shutdown_connections 2024-09-17T04:00:31.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.120+0000 7f02abc00640 1 -- 172.21.15.129:0/3194910257 wait complete. 2024-09-17T04:00:31.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.122+0000 7f02abc00640 1 Processor -- start 2024-09-17T04:00:31.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.123+0000 7f02abc00640 1 -- start start 2024-09-17T04:00:31.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02a40cc260 0x7f02a41f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 0x7f02a41f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02a415fa70 con 0x7f02a40b3930 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f02a40cf390 con 0x7f02a415b0c0 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02abc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f02a415fbf0 con 0x7f02a40cc260 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02a9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02a9200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36212/0 (socket says 172.21.15.129:36212) 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02a9200640 1 -- 172.21.15.129:0/1745996270 learned_addr learned my addr 172.21.15.129:0/1745996270 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f029be00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02a40cc260 0x7f02a41f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.124+0000 7f02a9c00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 0x7f02a41f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 -- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02a40cc260 msgr2=0x7f02a41f8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02a40cc260 0x7f02a41f8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 -- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 msgr2=0x7f02a41f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 0x7f02a41f9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 -- 172.21.15.129:0/1745996270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a415cf50 con 0x7f02a40b3930 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9c00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 0x7f02a41f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a9200640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f029400d770 tx=0x7f029400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0294017070 con 0x7f02a40b3930 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0294009d20 con 0x7f02a40b3930 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940106c0 con 0x7f02a40b3930 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a40d0c50 con 0x7f02a40b3930 2024-09-17T04:00:31.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.125+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a4155670 con 0x7f02a40b3930 2024-09-17T04:00:31.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.126+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a4160110 con 0x7f02a40b3930 2024-09-17T04:00:31.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.128+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f029401a070 con 0x7f02a40b3930 2024-09-17T04:00:31.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.129+0000 7f02a1600640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 0x7f028007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.129+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f029409adc0 con 0x7f02a40b3930 2024-09-17T04:00:31.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.130+0000 7f029be00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 0x7f028007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.130+0000 7f029be00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 0x7f028007b0d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f028c034000 tx=0x7f028c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.134+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0294062ba0 con 0x7f02a40b3930 2024-09-17T04:00:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.359+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02a406a140 con 0x7f02a40b3930 2024-09-17T04:00:31.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.360+0000 7f02a1600640 1 -- 172.21.15.129:0/1745996270 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f02940675d0 con 0x7f02a40b3930 2024-09-17T04:00:31.359 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:31.359 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:31.360 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.362+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 msgr2=0x7f028007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.362+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 0x7f028007b0d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f028c034000 tx=0x7f028c004730 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.362+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 msgr2=0x7f02a4155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.362+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f029400d770 tx=0x7f029400dc30 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 shutdown_connections 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0280078bc0 0x7f028007b0d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02a415b0c0 0x7f02a41f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02a40cc260 0x7f02a41f8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 --2- 172.21.15.129:0/1745996270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02a40b3930 0x7f02a4155130 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 >> 172.21.15.129:0/1745996270 conn(0x7f02a406d410 msgr2=0x7f02a40aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:31.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 shutdown_connections 2024-09-17T04:00:31.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.363+0000 7f02abc00640 1 -- 172.21.15.129:0/1745996270 wait complete. 2024-09-17T04:00:31.385 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:00:31.624 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":1}} 2024-09-17T04:00:31.624 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 1 } } 2024-09-17T04:00:31.625 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T04:00:31.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.766+0000 7f88a1635f40 1 Processor -- start 2024-09-17T04:00:31.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.766+0000 7f88a1635f40 1 -- start start 2024-09-17T04:00:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.767+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f65016cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f650154770 0x55f650169900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f88a1635f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f64ffe5c60 con 0x7f88800022f0 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f88a1635f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f6500441d0 con 0x55f650154770 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f88a1635f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55f65016c250 con 0x55f65016dfd0 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36242/0 (socket says 172.21.15.129:36242) 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 -- 172.21.15.129:0/538015710 learned_addr learned my addr 172.21.15.129:0/538015710 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f8893e00640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f65016cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889b200640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f650154770 0x55f650169900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 -- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 msgr2=0x55f65016cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f65016cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 -- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f650154770 msgr2=0x55f650169900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:31.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f650154770 0x55f650169900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f889a800640 1 -- 172.21.15.129:0/538015710 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6500de900 con 0x7f88800022f0 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.768+0000 7f8893e00640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f65016cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f889a800640 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f888c0262b0 tx=0x7f888c003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add5d4f72bccb3e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f8899400640 1 -- 172.21.15.129:0/538015710 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c062030 con 0x7f88800022f0 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f8899400640 1 -- 172.21.15.129:0/538015710 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f888c04db10 con 0x7f88800022f0 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f8899400640 1 -- 172.21.15.129:0/538015710 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c05e400 con 0x7f88800022f0 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f88a1635f40 1 -- 172.21.15.129:0/538015710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 msgr2=0x55f650153320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.769+0000 7f88a1635f40 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f888c0262b0 tx=0x7f888c003020 comp rx=0 tx=0).stop 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 -- 172.21.15.129:0/538015710 shutdown_connections 2024-09-17T04:00:31.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f650154770 0x55f650169900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f65016cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 --2- 172.21.15.129:0/538015710 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f88800022f0 0x55f650153320 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 -- 172.21.15.129:0/538015710 >> 172.21.15.129:0/538015710 conn(0x55f64ffb6ac0 msgr2=0x55f6500d7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 -- 172.21.15.129:0/538015710 shutdown_connections 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.770+0000 7f88a1635f40 1 -- 172.21.15.129:0/538015710 wait complete. 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.771+0000 7f88a1635f40 1 Processor -- start 2024-09-17T04:00:31.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.771+0000 7f88a1635f40 1 -- start start 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f6500db660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88800022f0 0x55f65016cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55f6500de6b0 con 0x55f650154770 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55f6500d8090 con 0x7f88800022f0 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f88a1635f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55f650168000 con 0x55f65016dfd0 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f889a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.772+0000 7f889a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36254/0 (socket says 172.21.15.129:36254) 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 learned_addr learned my addr 172.21.15.129:0/2887262428 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 msgr2=0x55f6500db660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55f65016dfd0 0x55f6500db660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88800022f0 msgr2=0x55f65016cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88800022f0 0x55f65016cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6500dd250 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f888c00e090 tx=0x7f888c05e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f8893400640 1 -- 172.21.15.129:0/2887262428 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c06b070 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f8893400640 1 -- 172.21.15.129:0/2887262428 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f888c05ed10 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f6500dd4d0 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.773+0000 7f8893400640 1 -- 172.21.15.129:0/2887262428 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0676a0 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.774+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6500dda90 con 0x55f650154770 2024-09-17T04:00:31.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.776+0000 7f8893400640 1 -- 172.21.15.129:0/2887262428 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f888c065070 con 0x55f650154770 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.776+0000 7f8893400640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 0x7f8878079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.777+0000 7f8893400640 1 -- 172.21.15.129:0/2887262428 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f888c0de8e0 con 0x55f650154770 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.777+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 0x7f8878079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.777+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55f650166cc0 0x55f6501670d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.777+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:1 25.0 25:00000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650166cc0 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.777+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55f650166cc0 0x55f6501670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.778+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 0x7f8878079a80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x55f6500db290 tx=0x7f8884003040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.778+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55f650166cc0 0x55f6501670d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.778+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(1 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8894003040 con 0x55f650166cc0 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.778+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x55f650165360 0x7f889400e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.778+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:2 25.20 25:04000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650165360 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.779+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x55f650165360 0x7f889400e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.779+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x55f650165360 0x7f889400e8d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.780+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(2 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.780+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f888c023270 0x7f888c0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.780+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:3 25.10 25:08000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c023270 2024-09-17T04:00:31.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.780+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f888c023270 0x7f888c0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.780+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f888c023270 0x7f888c0af040 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.781+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(3 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8884016040 con 0x7f888c023270 2024-09-17T04:00:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.781+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f888401d910 0x7f888401c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.781+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:4 25.30 25:0c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888401d910 2024-09-17T04:00:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.781+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f888401d910 0x7f888401c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.782+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f888401d910 0x7f888401c390 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.782+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(4 [pgnls start_epoch 163 out=81b] v163'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f8894003040 con 0x7f888401d910 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.782+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:5 25.8 25:10000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650166cc0 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.782+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(5 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8894003040 con 0x55f650166cc0 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.782+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:6 25.28 25:14000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888401d910 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.783+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(6 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8894003040 con 0x7f888401d910 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.783+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55f65016b440 0x7f889401f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.783+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:7 25.18 25:18000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.783+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55f65016b440 0x7f889401f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.784+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55f65016b440 0x7f889401f3b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.784+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(7 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.784+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:8 25.38 25:1c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888401d910 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.784+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(8 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f8894003040 con 0x7f888401d910 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.784+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:9 25.4 25:20000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888401d910 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.785+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(9 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889402b040 con 0x7f888401d910 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.785+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:10 25.24 25:24000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650165360 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.785+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(10 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.785+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f888c01fbf0 0x7f888c0a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.785+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:11 25.14 25:28000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c01fbf0 2024-09-17T04:00:31.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.786+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f888c01fbf0 0x7f888c0a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.786+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f888c01fbf0 0x7f888c0a7a60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.787+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(11 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.787+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 25.34 25:2c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650166cc0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.787+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(12 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889402b040 con 0x55f650166cc0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.787+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:13 25.c 25:30000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c023270 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(13 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:14 25.2c 25:34000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650165360 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(14 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:15 25.1c 25:38000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c01fbf0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(15 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.788+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:16 25.3c 25:3c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650166cc0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.789+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(16 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889402b040 con 0x55f650166cc0 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.789+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f88800030f0 0x7f889402e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.789+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:17 25.2 25:40000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f88800030f0 2024-09-17T04:00:31.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.790+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f88800030f0 0x7f889402e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.790+0000 7f889b200640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f88800030f0 0x7f889402e420 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(17 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f88800030f0 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 25.22 25:44000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c023270 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(18 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f8893e00640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f8884030040 0x7f888402f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:19 25.12 25:48000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f8884030040 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.791+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f8884030040 0x7f888402f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.792+0000 7f889a800640 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f8884030040 0x7f888402f950 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.792+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(19 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.792+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:20 25.32 25:4c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c023270 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(20 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:21 25.a 25:50000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(21 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:22 25.2a 25:54000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(22 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.793+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 25.1a 25:58000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(23 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 25.3a 25:5c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f8884030040 2024-09-17T04:00:31.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(24 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:25 25.6 25:60000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650165360 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(25 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.794+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:26 25.26 25:64000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888c01fbf0 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.795+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(26 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.795+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:27 25.16 25:68000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f8884030040 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.795+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(27 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.795+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:28 25.36 25:6c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.795+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(28 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.796+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:29 25.e 25:70000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f65016b440 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.796+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(29 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.796+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:30 25.2e 25:74000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f88800030f0 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.796+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(30 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f88800030f0 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.796+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:31 25.1e 25:78000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f888401d910 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.797+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(31 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f888401d910 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.797+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:32 25.3e 25:7c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650166cc0 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.797+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(32 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.797+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:33 25.1 25:80000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x55f650165360 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(33 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:34 25.21 25:84000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55f650167610 con 0x7f8884030040 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(34 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:35 25.11 25:88000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f65016b440 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(35 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.798+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:36 25.31 25:8c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650166cc0 2024-09-17T04:00:31.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(36 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:37 25.9 25:90000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650165360 2024-09-17T04:00:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(37 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:38 25.29 25:94000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f8884030040 2024-09-17T04:00:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(38 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.799+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:39 25.19 25:98000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c01fbf0 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.800+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(39 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.800+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:40 25.39 25:9c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c01fbf0 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.800+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(40 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.800+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:41 25.5 25:a0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888401d910 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(41 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f888401d910 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:42 25.25 25:a4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f8884030040 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(42 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:43 25.15 25:a8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f65016b440 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(43 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.801+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:44 25.35 25:ac000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c01fbf0 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.802+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 6 ==== osd_op_reply(44 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.802+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:45 25.d 25:b0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650166cc0 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.802+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 7 ==== osd_op_reply(45 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.802+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:46 25.2d 25:b4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f65016b440 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.803+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(46 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.803+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:47 25.1d 25:b8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c023270 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.803+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(47 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.803+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:48 25.3d 25:bc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f8884030040 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.803+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(48 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x7f8884030040 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.804+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:49 25.3 25:c0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f88800030f0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.804+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(49 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f88800030f0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.804+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:50 25.23 25:c4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650166cc0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.804+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 8 ==== osd_op_reply(50 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.805+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:51 25.13 25:c8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c01fbf0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.805+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 7 ==== osd_op_reply(51 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.805+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:52 25.33 25:cc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650166cc0 2024-09-17T04:00:31.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.806+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 9 ==== osd_op_reply(52 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.806+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:53 25.b 25:d0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c01fbf0 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.4 v2:172.21.15.181:6802/2229307524 8 ==== osd_op_reply(53 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c01fbf0 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:54 25.2b 25:d4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650165360 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(54 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:55 25.1b 25:d8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888401d910 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(55 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f888401d910 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.807+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:56 25.3b 25:dc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650165360 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.808+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(56 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.808+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 25.7 25:e0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f65016b440 2024-09-17T04:00:31.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.808+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(57 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.808+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:58 25.27 25:e4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650166cc0 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.809+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.6 v2:172.21.15.181:6810/4136500368 10 ==== osd_op_reply(58 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x55f650166cc0 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.809+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:59 25.17 25:e8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888401d910 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.809+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(59 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f888401d910 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.809+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:60 25.37 25:ec000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c023270 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(60 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:61 25.f 25:f0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f888c023270 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(61 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888402a040 con 0x7f888c023270 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f8893e00640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:62 25.2f 25:f4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f650165360 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.2 v2:172.21.15.129:6809/283507149 9 ==== osd_op_reply(62 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f650165360 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.810+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:63 25.1f 25:f8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x55f65016b440 2024-09-17T04:00:31.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(63 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f888c0a29a0 con 0x55f65016b440 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f889a800640 1 -- 172.21.15.129:0/2887262428 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:64 25.3f 25:fc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x7f888c0d3040 con 0x7f88800030f0 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(64 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f889403b040 con 0x7f88800030f0 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000000 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000001 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000000 2024-09-17T04:00:31.818 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000001 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55f65016b440 msgr2=0x7f889401f3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x55f65016b440 0x7f889401f3b0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f8884030040 msgr2=0x7f888402f950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f8884030040 0x7f888402f950 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x55f650165360 msgr2=0x7f889400e8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x55f650165360 0x7f889400e8d0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f888401d910 msgr2=0x7f888401c390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 reap_dead start 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f888401d910 0x7f888401c390 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f888c01fbf0 msgr2=0x7f888c0a7a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f888c01fbf0 0x7f888c0a7a60 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f888c023270 msgr2=0x7f888c0af040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f888c023270 0x7f888c0af040 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55f650166cc0 msgr2=0x55f6501670d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55f650166cc0 0x55f6501670d0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f88800030f0 msgr2=0x7f889402e420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.811+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f88800030f0 0x7f889402e420 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f889b200640 1 -- 172.21.15.129:0/2887262428 reap_dead start 2024-09-17T04:00:31.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 msgr2=0x7f8878079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 0x7f8878079a80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x55f6500db290 tx=0x7f8884003040 comp rx=0 tx=0).stop 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 msgr2=0x55f650141660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f888c00e090 tx=0x7f888c05e930 comp rx=0 tx=0).stop 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 shutdown_connections 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8878077570 0x7f8878079a80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 --2- 172.21.15.129:0/2887262428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55f650154770 0x55f650141660 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 >> 172.21.15.129:0/2887262428 conn(0x55f64ffb6ac0 msgr2=0x55f6500d7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 shutdown_connections 2024-09-17T04:00:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.812+0000 7f88a1635f40 1 -- 172.21.15.129:0/2887262428 wait complete. 2024-09-17T04:00:31.823 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2024-09-17T04:00:31.823 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T04:00:31.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.962+0000 7f44d1433f40 1 Processor -- start 2024-09-17T04:00:31.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.963+0000 7f44d1433f40 1 -- start start 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc19d100 0x55cfbc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc1a66c0 0x55cfbc19f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc19d510 con 0x55cfbc0ff190 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc01ec60 con 0x55cfbc1a66c0 2024-09-17T04:00:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44d1433f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cfbc11c7e0 con 0x55cfbc19d100 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44ca600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44ca600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36284/0 (socket says 172.21.15.129:36284) 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44ca600640 1 -- 172.21.15.129:0/3886421365 learned_addr learned my addr 172.21.15.129:0/3886421365 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc19d100 0x55cfbc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.964+0000 7f44cb000640 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc1a66c0 0x55cfbc19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 -- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc19d100 msgr2=0x55cfbc19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc19d100 0x55cfbc19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 -- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc1a66c0 msgr2=0x55cfbc19f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc1a66c0 0x55cfbc19f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 -- 172.21.15.129:0/3886421365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbbff6f10 con 0x55cfbc0ff190 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44ca600640 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0464e0 tx=0x7f44bc002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb3eab2fd1f35ad server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44c3e00640 1 -- 172.21.15.129:0/3886421365 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc04dad0 con 0x55cfbc0ff190 2024-09-17T04:00:31.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44c3e00640 1 -- 172.21.15.129:0/3886421365 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44bc04dc70 con 0x55cfbc0ff190 2024-09-17T04:00:31.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.965+0000 7f44c3e00640 1 -- 172.21.15.129:0/3886421365 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc04de60 con 0x55cfbc0ff190 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 -- 172.21.15.129:0/3886421365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 msgr2=0x55cfbc1949d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f44bc0464e0 tx=0x7f44bc002fb0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 -- 172.21.15.129:0/3886421365 shutdown_connections 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc1a66c0 0x55cfbc19f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc19d100 0x55cfbc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3886421365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1949d0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 -- 172.21.15.129:0/3886421365 >> 172.21.15.129:0/3886421365 conn(0x55cfbbfefac0 msgr2=0x55cfbc0cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 -- 172.21.15.129:0/3886421365 shutdown_connections 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.966+0000 7f44d1433f40 1 -- 172.21.15.129:0/3886421365 wait complete. 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.967+0000 7f44d1433f40 1 Processor -- start 2024-09-17T04:00:31.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.967+0000 7f44d1433f40 1 -- start start 2024-09-17T04:00:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.968+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.968+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc19d100 0x55cfbc1a7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44d1433f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc1a66c0 0x55cfbc1a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44d1433f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc0cdfd0 con 0x55cfbc0ff190 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44d1433f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cfbc19cdc0 con 0x55cfbc19d100 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44d1433f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cfbc19cae0 con 0x55cfbc1a66c0 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36290/0 (socket says 172.21.15.129:36290) 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 learned_addr learned my addr 172.21.15.129:0/3429103342 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc19d100 0x55cfbc1a7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc1a66c0 0x55cfbc1a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc1a66c0 msgr2=0x55cfbc1a7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cfbc1a66c0 0x55cfbc1a7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc19d100 msgr2=0x55cfbc1a7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc19d100 0x55cfbc1a7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cfbc1a3550 con 0x55cfbc0ff190 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.969+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cfbc19d100 0x55cfbc1a7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f44bc004f50 tx=0x7f44bc04df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44c2a00640 1 -- 172.21.15.129:0/3429103342 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc056470 con 0x55cfbc0ff190 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44c2a00640 1 -- 172.21.15.129:0/3429103342 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f44bc056700 con 0x55cfbc0ff190 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55cfbc1a3830 con 0x55cfbc0ff190 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44c2a00640 1 -- 172.21.15.129:0/3429103342 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc056b10 con 0x55cfbc0ff190 2024-09-17T04:00:31.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.970+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cfbc1a3df0 con 0x55cfbc0ff190 2024-09-17T04:00:31.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.972+0000 7f44c2a00640 1 -- 172.21.15.129:0/3429103342 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f44bc06a070 con 0x55cfbc0ff190 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.973+0000 7f44c2a00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f44ac077570 0x7f44ac079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.973+0000 7f44c2a00640 1 -- 172.21.15.129:0/3429103342 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f44bc0de9b0 con 0x55cfbc0ff190 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.973+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f44ac077570 0x7f44ac079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.973+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55cfbc193d20 0x55cfbc1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.973+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:1 25.0 25:00000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.974+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55cfbc193d20 0x55cfbc1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.974+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f44ac077570 0x7f44ac079a80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x55cfbc19f500 tx=0x7f44b800f040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.974+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55cfbc193d20 0x55cfbc1a18a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.975+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(1 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c4003040 con 0x55cfbc193d20 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.975+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f44c400f430 0x7f44c400f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.975+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:2 25.20 25:04000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.975+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f44c400f430 0x7f44c400f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.975+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f44c400f430 0x7f44c400f840 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.976+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(2 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.976+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f44bc022da0 0x7f44bc0a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.976+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:3 25.10 25:08000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:31.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.976+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f44bc022da0 0x7f44bc0a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.977+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f44bc022da0 0x7f44bc0a5600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.977+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(3 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:31.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.977+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f44b8017040 0x7f44b8018970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.977+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:4 25.30 25:0c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.977+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f44b8017040 0x7f44b8018970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.978+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f44b8017040 0x7f44b8018970 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.978+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(4 [pgnls start_epoch 163 out=81b] v163'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f44c4003040 con 0x7f44b8017040 2024-09-17T04:00:31.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.978+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:5 25.8 25:10000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(5 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c4003040 con 0x55cfbc193d20 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:6 25.28 25:14000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(6 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c4003040 con 0x7f44b8017040 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f44c4020420 0x7f44c4020830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:7 25.18 25:18000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.979+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f44c4020420 0x7f44c4020830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.980+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f44c4020420 0x7f44c4020830 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.980+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(7 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.980+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:8 25.38 25:1c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.980+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(8 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c4003040 con 0x7f44b8017040 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.980+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:9 25.4 25:20000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(9 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c402b040 con 0x7f44b8017040 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:10 25.24 25:24000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(10 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f44bc01f720 0x7f44bc0af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:11 25.14 25:28000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.981+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f44bc01f720 0x7f44bc0af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.982+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f44bc01f720 0x7f44bc0af800 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.982+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(11 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.982+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 25.34 25:2c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.983+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(12 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c402b040 con 0x55cfbc193d20 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.983+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:13 25.c 25:30000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.984+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(13 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.984+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:14 25.2c 25:34000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.984+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(14 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.984+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:15 25.1c 25:38000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.985+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(15 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.985+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:16 25.3c 25:3c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.985+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(16 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c402b040 con 0x55cfbc193d20 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.985+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f44c402e930 0x7f44c400e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.985+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:17 25.2 25:40000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c402e930 2024-09-17T04:00:31.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.986+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f44c402e930 0x7f44c400e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.986+0000 7f44cb000640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f44c402e930 0x7f44c400e3c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(17 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44c402e930 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 25.22 25:44000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(18 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44c9c00640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f44b8028410 0x7f44b8028820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:19 25.12 25:48000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.987+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f44b8028410 0x7f44b8028820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:31.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.988+0000 7f44ca600640 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f44b8028410 0x7f44b8028820 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:31.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.988+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(19 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.988+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:20 25.32 25:4c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.988+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(20 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.988+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:21 25.a 25:50000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(21 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:22 25.2a 25:54000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(22 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 25.1a 25:58000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(23 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.989+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 25.3a 25:5c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.990+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(24 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.990+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:25 25.6 25:60000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.990+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(25 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.990+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:26 25.26 25:64000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.990+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(26 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:27 25.16 25:68000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(27 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:28 25.36 25:6c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(28 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:29 25.e 25:70000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.991+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(29 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.992+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:30 25.2e 25:74000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c402e930 2024-09-17T04:00:31.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.992+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(30 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44c402e930 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.992+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:31 25.1e 25:78000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(31 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44b8017040 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:32 25.3e 25:7c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(32 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:33 25.1 25:80000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(33 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.993+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:34 25.21 25:84000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.994+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(34 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.994+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:35 25.11 25:88000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.994+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(35 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.994+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:36 25.31 25:8c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(36 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:37 25.9 25:90000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(37 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:38 25.29 25:94000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(38 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.995+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:39 25.19 25:98000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.996+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(39 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.996+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:40 25.39 25:9c000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.996+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(40 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.996+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:41 25.5 25:a0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(41 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44b8017040 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:42 25.25 25:a4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(42 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:43 25.15 25:a8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(43 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.997+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:44 25.35 25:ac000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.998+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 6 ==== osd_op_reply(44 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.998+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:45 25.d 25:b0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.998+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 7 ==== osd_op_reply(45 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.998+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:46 25.2d 25:b4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.999+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(46 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:31.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:31.999+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:47 25.1d 25:b8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:32.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.000+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(47 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.000+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:48 25.3d 25:bc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8028410 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.000+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.1 v2:172.21.15.129:6800/4215860840 7 ==== osd_op_reply(48 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44b8028410 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.000+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:49 25.3 25:c0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c402e930 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(49 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44c402e930 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:50 25.23 25:c4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 8 ==== osd_op_reply(50 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:51 25.13 25:c8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 7 ==== osd_op_reply(51 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:32.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.001+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:52 25.33 25:cc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.002+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 9 ==== osd_op_reply(52 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.002+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:53 25.b 25:d0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc01f720 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.002+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.4 v2:172.21.15.181:6802/2229307524 8 ==== osd_op_reply(53 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc01f720 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.002+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:54 25.2b 25:d4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.003+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(54 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.003+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:55 25.1b 25:d8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.003+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(55 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44b8017040 2024-09-17T04:00:32.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.003+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:56 25.3b 25:dc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:32.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.004+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(56 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:32.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.004+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 25.7 25:e0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:32.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.004+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(57 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:32.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.004+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:58 25.27 25:e4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x55cfbc193d20 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.005+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.6 v2:172.21.15.181:6810/4136500368 10 ==== osd_op_reply(58 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x55cfbc193d20 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.005+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:59 25.17 25:e8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44b8017040 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.005+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(59 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44b8017040 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.005+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:60 25.37 25:ec000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.006+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(60 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:32.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.006+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:61 25.f 25:f0000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44bc022da0 2024-09-17T04:00:32.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.006+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(61 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44b800a7a0 con 0x7f44bc022da0 2024-09-17T04:00:32.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.006+0000 7f44c9c00640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:62 25.2f 25:f4000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c400f430 2024-09-17T04:00:32.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.007+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.2 v2:172.21.15.129:6809/283507149 9 ==== osd_op_reply(62 [pgnls start_epoch 163 out=81b] v163'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c400f430 2024-09-17T04:00:32.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.007+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:63 25.1f 25:f8000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c4020420 2024-09-17T04:00:32.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.007+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(63 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44bc05f030 con 0x7f44c4020420 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.007+0000 7f44ca600640 1 -- 172.21.15.129:0/3429103342 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:64 25.3f 25:fc000000::::head [pgnls start_epoch 163 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e163) -- 0x55cfbc24a000 con 0x7f44c402e930 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.007+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(64 [pgnls start_epoch 163 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f44c403b040 con 0x7f44c402e930 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000000 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000001 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000000 2024-09-17T04:00:32.010 INFO:teuthology.orchestra.run.smithi129.stdout:10000000001.00000001 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f44c4020420 msgr2=0x7f44c4020830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f44c4020420 0x7f44c4020830 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f44b8028410 msgr2=0x7f44b8028820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f44b8028410 0x7f44b8028820 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f44c400f430 msgr2=0x7f44c400f840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7f44c400f430 0x7f44c400f840 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f44b8017040 msgr2=0x7f44b8018970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f44b8017040 0x7f44b8018970 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f44bc01f720 msgr2=0x7f44bc0af800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 reap_dead start 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f44bc01f720 0x7f44bc0af800 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 reap_dead start 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f44bc022da0 msgr2=0x7f44bc0a5600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f44bc022da0 0x7f44bc0a5600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55cfbc193d20 msgr2=0x55cfbc1a18a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x55cfbc193d20 0x55cfbc1a18a0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f44c402e930 msgr2=0x7f44c400e3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f44c402e930 0x7f44c400e3c0 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f44ac077570 msgr2=0x7f44ac079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f44ac077570 0x7f44ac079a80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x55cfbc19f500 tx=0x7f44b800f040 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 msgr2=0x55cfbc1936a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44cb000640 1 -- 172.21.15.129:0/3429103342 reap_dead start 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f44bc004f50 tx=0x7f44bc04df70 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 shutdown_connections 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 --2- 172.21.15.129:0/3429103342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cfbc0ff190 0x55cfbc1936a0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 >> 172.21.15.129:0/3429103342 conn(0x55cfbbfefac0 msgr2=0x55cfbc0cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:32.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 shutdown_connections 2024-09-17T04:00:32.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.008+0000 7f44d1433f40 1 -- 172.21.15.129:0/3429103342 wait complete. 2024-09-17T04:00:32.020 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627777 size 8388608 found 2024-09-17T04:00:32.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.242+0000 7efc0b400640 1 -- 172.21.15.129:0/1020713150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 msgr2=0x7efc040ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.242+0000 7efc0b400640 1 --2- 172.21.15.129:0/1020713150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc040ccc80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009830 tx=0x7efbf002a540 comp rx=0 tx=0).stop 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.243+0000 7efc0b400640 1 -- 172.21.15.129:0/1020713150 shutdown_connections 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.243+0000 7efc0b400640 1 --2- 172.21.15.129:0/1020713150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc040ccc80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.243+0000 7efc0b400640 1 -- 172.21.15.129:0/1020713150 >> 172.21.15.129:0/1020713150 conn(0x7efc0406d410 msgr2=0x7efc040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:32.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.243+0000 7efc0b400640 1 -- 172.21.15.129:0/1020713150 shutdown_connections 2024-09-17T04:00:32.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.243+0000 7efc0b400640 1 -- 172.21.15.129:0/1020713150 wait complete. 2024-09-17T04:00:32.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.245+0000 7efc0b400640 1 Processor -- start 2024-09-17T04:00:32.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.245+0000 7efc0b400640 1 -- start start 2024-09-17T04:00:32.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 0x7efc04158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc040cd1c0 0x7efc04159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efc04160260 con 0x7efc040705a0 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efc041600e0 con 0x7efc040b4140 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.246+0000 7efc0b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efc041610d0 con 0x7efc040cd1c0 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc08a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 0x7efc04158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:36006/0 (socket says 172.21.15.129:36006) 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 -- 172.21.15.129:0/2138856711 learned_addr learned my addr 172.21.15.129:0/2138856711 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:32.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc09400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc040cd1c0 0x7efc04159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 -- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc040cd1c0 msgr2=0x7efc04159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc040cd1c0 0x7efc04159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 -- 172.21.15.129:0/2138856711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 msgr2=0x7efc04158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 0x7efc04158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc03e00640 1 -- 172.21.15.129:0/2138856711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0415eb90 con 0x7efc040b4140 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.247+0000 7efc08a00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 0x7efc04158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc03e00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009920 tx=0x7efbf0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0005c90 con 0x7efc040b4140 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efbf0005310 con 0x7efc040b4140 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc041f7240 con 0x7efc040b4140 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0003e30 con 0x7efc040b4140 2024-09-17T04:00:32.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.248+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0406fce0 con 0x7efc040b4140 2024-09-17T04:00:32.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.249+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc040ccc80 con 0x7efc040b4140 2024-09-17T04:00:32.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.250+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efbf00317c0 con 0x7efc040b4140 2024-09-17T04:00:32.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.251+0000 7efc00c00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 0x7efbe407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.251+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7efbf00bbc40 con 0x7efc040b4140 2024-09-17T04:00:32.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.252+0000 7efc08a00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 0x7efbe407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.252+0000 7efc08a00640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 0x7efbe407b0d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efbf8001120 tx=0x7efbf8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.256+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efbf003f030 con 0x7efc040b4140 2024-09-17T04:00:32.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.472+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc0406a140 con 0x7efc040b4140 2024-09-17T04:00:32.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.473+0000 7efc00c00640 1 -- 172.21.15.129:0/2138856711 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7efbf0083a20 con 0x7efc040b4140 2024-09-17T04:00:32.472 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:32.473 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 msgr2=0x7efbe407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 0x7efbe407b0d0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7efbf8001120 tx=0x7efbf8009040 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 msgr2=0x7efc041f7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009920 tx=0x7efbf0003c10 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 shutdown_connections 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efbe4078bc0 0x7efbe407b0d0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efc040cd1c0 0x7efc04159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efc040b4140 0x7efc041f7a40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 --2- 172.21.15.129:0/2138856711 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efc040705a0 0x7efc04158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.475+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 >> 172.21.15.129:0/2138856711 conn(0x7efc0406d410 msgr2=0x7efc0406eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:32.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.476+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 shutdown_connections 2024-09-17T04:00:32.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.476+0000 7efc0b400640 1 -- 172.21.15.129:0/2138856711 wait complete. 2024-09-17T04:00:32.497 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-09-17T04:00:32.745 INFO:teuthology.orchestra.run.smithi181.stdout:{"message":"","return_code":0} 2024-09-17T04:00:32.745 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T04:00:32.745 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-17T04:00:32.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.888+0000 7f580520bf40 1 Processor -- start 2024-09-17T04:00:32.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.889+0000 7f580520bf40 1 -- start start 2024-09-17T04:00:32.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e3ab70 0x564746ed78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 0x564746ed7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564746cc9c60 con 0x564746dc1af0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564746d281d0 con 0x564746e517e0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f580520bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564746e3af80 con 0x564746e3ab70 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f57fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f57fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36304/0 (socket says 172.21.15.129:36304) 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.890+0000 7f57fd600640 1 -- 172.21.15.129:0/4124452051 learned_addr learned my addr 172.21.15.129:0/4124452051 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fe000640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 0x564746ed7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fcc00640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e3ab70 0x564746ed78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 -- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e3ab70 msgr2=0x564746ed78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e3ab70 0x564746ed78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 -- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 msgr2=0x564746ed7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 0x564746ed7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 -- 172.21.15.129:0/4124452051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564746dc4f60 con 0x564746dc1af0 2024-09-17T04:00:32.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fe000640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 0x564746ed7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:32.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57fd600640 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f57f0001e40 tx=0x7f57f0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f6f6a0799b3a54 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57f7400640 1 -- 172.21.15.129:0/4124452051 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0016070 con 0x564746dc1af0 2024-09-17T04:00:32.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57f7400640 1 -- 172.21.15.129:0/4124452051 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57f000d7c0 con 0x564746dc1af0 2024-09-17T04:00:32.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.891+0000 7f57f7400640 1 -- 172.21.15.129:0/4124452051 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f000dc10 con 0x564746dc1af0 2024-09-17T04:00:32.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 -- 172.21.15.129:0/4124452051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 msgr2=0x564746e392c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f57f0001e40 tx=0x7f57f0001e70 comp rx=0 tx=0).stop 2024-09-17T04:00:32.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 -- 172.21.15.129:0/4124452051 shutdown_connections 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e517e0 0x564746ed7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e3ab70 0x564746ed78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 --2- 172.21.15.129:0/4124452051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e392c0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 -- 172.21.15.129:0/4124452051 >> 172.21.15.129:0/4124452051 conn(0x564746c9aac0 msgr2=0x564746e39eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 -- 172.21.15.129:0/4124452051 shutdown_connections 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.892+0000 7f580520bf40 1 -- 172.21.15.129:0/4124452051 wait complete. 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.893+0000 7f580520bf40 1 Processor -- start 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.893+0000 7f580520bf40 1 -- start start 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.894+0000 7f580520bf40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e382b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f580520bf40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e3ab70 0x564746ee29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f580520bf40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e517e0 0x564746ee2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f580520bf40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564746e37270 con 0x564746dc1af0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f580520bf40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564746edcf70 con 0x564746e3ab70 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f580520bf40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564746e3a0f0 con 0x564746e517e0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e382b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36318/0 (socket says 172.21.15.129:36318) 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 -- 172.21.15.129:0/3354717510 learned_addr learned my addr 172.21.15.129:0/3354717510 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fe000640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e517e0 0x564746ee2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fcc00640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e3ab70 0x564746ee29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 -- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e517e0 msgr2=0x564746ee2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564746e517e0 0x564746ee2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 -- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e3ab70 msgr2=0x564746ee29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e3ab70 0x564746ee29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fd600640 1 -- 172.21.15.129:0/3354717510 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564746e24af0 con 0x564746dc1af0 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.895+0000 7f57fcc00640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564746e3ab70 0x564746ee29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:32.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f57fd600640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e382b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f57f000bae0 tx=0x7f57f000d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f57f6000640 1 -- 172.21.15.129:0/3354717510 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0022070 con 0x564746dc1af0 2024-09-17T04:00:32.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f57f6000640 1 -- 172.21.15.129:0/3354717510 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57f0017040 con 0x564746dc1af0 2024-09-17T04:00:32.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f57f6000640 1 -- 172.21.15.129:0/3354717510 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0016070 con 0x564746dc1af0 2024-09-17T04:00:32.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564746ee3a20 con 0x564746dc1af0 2024-09-17T04:00:32.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.896+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564746e39d80 con 0x564746dc1af0 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.898+0000 7f57f6000640 1 -- 172.21.15.129:0/3354717510 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f57f0025070 con 0x564746dc1af0 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.899+0000 7f57f6000640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e0077570 0x7f57e0079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.899+0000 7f57f6000640 1 -- 172.21.15.129:0/3354717510 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f57f009a6b0 con 0x564746dc1af0 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.899+0000 7f580520bf40 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x564746e24e60 0x564746eec730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.899+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:1 25.7 25:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e163) -- 0x564746e387f0 con 0x564746e24e60 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.899+0000 7f57fcc00640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e0077570 0x7f57e0079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.900+0000 7f57fe000640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x564746e24e60 0x564746eec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:32.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.900+0000 7f57fcc00640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e0077570 0x7f57e0079a80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f57ec055000 tx=0x7f57ec058040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.900+0000 7f57fe000640 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x564746e24e60 0x564746eec730 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f57fe000640 1 -- 172.21.15.129:0/3354717510 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (64) out=64b] v0'0 uv2 ondisk = 0) ==== 164+0+64 (crc 0 0 0) 0x7f57f800b040 con 0x564746e24e60 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x564746e24e60 msgr2=0x564746eec730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x564746e24e60 0x564746eec730 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e0077570 msgr2=0x7f57e0079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f57e0077570 0x7f57e0079a80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f57ec055000 tx=0x7f57ec058040 comp rx=0 tx=0).stop 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 msgr2=0x564746e382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 --2- 172.21.15.129:0/3354717510 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564746dc1af0 0x564746e382b0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f57f000bae0 tx=0x7f57f000d530 comp rx=0 tx=0).stop 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f57fe000640 1 -- 172.21.15.129:0/3354717510 reap_dead start 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 shutdown_connections 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 >> 172.21.15.129:0/3354717510 conn(0x564746c9aac0 msgr2=0x564746ed8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 shutdown_connections 2024-09-17T04:00:32.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:32.901+0000 7f580520bf40 1 -- 172.21.15.129:0/3354717510 wait complete. 2024-09-17T04:00:32.910 INFO:teuthology.orchestra.run.smithi129.stdout::linkto_b 2024-09-17T04:00:32.910 DEBUG:teuthology.orchestra.run.smithi129:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: "ancestors": [ 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: "dirino": 1, 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: "dname": "linkto_b", 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 23 2024-09-17T04:00:33.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:33.053 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:00:33.053 INFO:teuthology.orchestra.run.smithi129.stdout: "pool": 25, 2024-09-17T04:00:33.054 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [] 2024-09-17T04:00:33.054 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:00:33.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:33.062 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:00:33.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:33.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:33.242 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:00:33.242 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:00:33.242 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:33.242 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:33.278 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:33.279 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:33.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:00:33.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 -- 172.21.15.129:0/1576926051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c09ba70 msgr2=0x7f452c0d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:33.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 --2- 172.21.15.129:0/1576926051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c09ba70 0x7f452c0d01f0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f4518009830 tx=0x7f451802a5c0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 -- 172.21.15.129:0/1576926051 shutdown_connections 2024-09-17T04:00:33.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 --2- 172.21.15.129:0/1576926051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c09ba70 0x7f452c0d01f0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 -- 172.21.15.129:0/1576926051 >> 172.21.15.129:0/1576926051 conn(0x7f452c0b3f00 msgr2=0x7f452c0b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:33.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 -- 172.21.15.129:0/1576926051 shutdown_connections 2024-09-17T04:00:33.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.616+0000 7f4531000640 1 -- 172.21.15.129:0/1576926051 wait complete. 2024-09-17T04:00:33.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.618+0000 7f4531000640 1 Processor -- start 2024-09-17T04:00:33.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.618+0000 7f4531000640 1 -- start start 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f452c09ba70 0x7f452c0c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f452c0c5aa0 0x7f452c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f452c168090 con 0x7f452c0ccfe0 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f452c167f10 con 0x7f452c0c5aa0 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.619+0000 7f4531000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f452c168210 con 0x7f452c09ba70 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f4529400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f452c0c5aa0 0x7f452c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:36344/0 (socket says 172.21.15.129:36344) 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 -- 172.21.15.129:0/713792975 learned_addr learned my addr 172.21.15.129:0/713792975 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:33.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f4529e00640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f452c09ba70 0x7f452c0c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 -- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f452c09ba70 msgr2=0x7f452c0c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f452c09ba70 0x7f452c0c8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 -- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f452c0c5aa0 msgr2=0x7f452c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f452c0c5aa0 0x7f452c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 -- 172.21.15.129:0/713792975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452c0c7f10 con 0x7f452c0ccfe0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.620+0000 7f452a800640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f452400c730 tx=0x7f452400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.621+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4524003b70 con 0x7f452c0ccfe0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.621+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4524003d30 con 0x7f452c0ccfe0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.621+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45240052d0 con 0x7f452c0ccfe0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.621+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452c0c5fd0 con 0x7f452c0ccfe0 2024-09-17T04:00:33.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.621+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452c0c8180 con 0x7f452c0ccfe0 2024-09-17T04:00:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.622+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f452c0c6230 con 0x7f452c0ccfe0 2024-09-17T04:00:33.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.623+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4524005680 con 0x7f452c0ccfe0 2024-09-17T04:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.624+0000 7f4522000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 0x7f450c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.624+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f452409ac60 con 0x7f452c0ccfe0 2024-09-17T04:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.624+0000 7f4529e00640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 0x7f450c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.625+0000 7f4529e00640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 0x7f450c07b0a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f451c001760 tx=0x7f451c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.628+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f45240629b0 con 0x7f452c0ccfe0 2024-09-17T04:00:33.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.819+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f452c0d01f0 con 0x7f452c0ccfe0 2024-09-17T04:00:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.820+0000 7f4522000640 1 -- 172.21.15.129:0/713792975 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v163) ==== 80+0+0 (secure 0 0 0) 0x7f45240673e0 con 0x7f452c0ccfe0 2024-09-17T04:00:33.819 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 msgr2=0x7f450c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 0x7f450c07b0a0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f451c001760 tx=0x7f451c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 msgr2=0x7f452c1f99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f452400c730 tx=0x7f452400cbf0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 shutdown_connections 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f450c078b90 0x7f450c07b0a0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f452c0ccfe0 0x7f452c1f99f0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f452c0c5aa0 0x7f452c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 --2- 172.21.15.129:0/713792975 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f452c09ba70 0x7f452c0c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:33.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.822+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 >> 172.21.15.129:0/713792975 conn(0x7f452c0b3f00 msgr2=0x7f452c0d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:33.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.823+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 shutdown_connections 2024-09-17T04:00:33.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:33.823+0000 7f4531000640 1 -- 172.21.15.129:0/713792975 wait complete. 2024-09-17T04:00:33.844 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:00:33.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:33.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:00:33.972 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:00:33.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:00:33.972 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:00:34.009 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:00:34.009 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:34.009 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:34.044 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:34.045 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:00:34.045 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:34.045 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:34.085 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:00:34.085 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:00:34.085 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:34.085 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:34.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:34.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.347+0000 7fb4e5e00640 1 -- 172.21.15.129:0/1840424629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 msgr2=0x7fb4e0160920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.347+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/1840424629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e0160920 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8009be0 tx=0x7fb4d802a890 comp rx=0 tx=0).stop 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.347+0000 7fb4e5e00640 1 -- 172.21.15.129:0/1840424629 shutdown_connections 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.347+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/1840424629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e0160920 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.347+0000 7fb4e5e00640 1 -- 172.21.15.129:0/1840424629 >> 172.21.15.129:0/1840424629 conn(0x7fb4e006cfc0 msgr2=0x7fb4e006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.348+0000 7fb4e5e00640 1 -- 172.21.15.129:0/1840424629 shutdown_connections 2024-09-17T04:00:34.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.348+0000 7fb4e5e00640 1 -- 172.21.15.129:0/1840424629 wait complete. 2024-09-17T04:00:34.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.350+0000 7fb4e5e00640 1 Processor -- start 2024-09-17T04:00:34.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.350+0000 7fb4e5e00640 1 -- start start 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0145e60 0x7fb4e0138820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e00cef20 con 0x7fb4e0139510 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb4e00ceda0 con 0x7fb4e0145e60 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.351+0000 7fb4e5e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb4e0160e60 con 0x7fb4e01458a0 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43272/0 (socket says 172.21.15.129:43272) 2024-09-17T04:00:34.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e5400640 1 -- 172.21.15.129:0/3230681978 learned_addr learned my addr 172.21.15.129:0/3230681978 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4dfe00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0145e60 0x7fb4e0138820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 -- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0145e60 msgr2=0x7fb4e0138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0145e60 0x7fb4e0138820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 -- 172.21.15.129:0/3230681978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 msgr2=0x7fb4e0138d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 -- 172.21.15.129:0/3230681978 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4e00c74b0 con 0x7fb4e01458a0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e5400640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.352+0000 7fb4e4a00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e015ef50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c3f0 tx=0x7fb4d400c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.353+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4017070 con 0x7fb4e01458a0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.353+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb4d40035f0 con 0x7fb4e01458a0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.353+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4d4003a40 con 0x7fb4e01458a0 2024-09-17T04:00:34.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.353+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4e0136790 con 0x7fb4e01458a0 2024-09-17T04:00:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.353+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4e0136d50 con 0x7fb4e01458a0 2024-09-17T04:00:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.354+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4e00d0340 con 0x7fb4e01458a0 2024-09-17T04:00:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.355+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb4d401a070 con 0x7fb4e01458a0 2024-09-17T04:00:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.356+0000 7fb4dcc00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 0x7fb4c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.356+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fb4d4099dc0 con 0x7fb4e01458a0 2024-09-17T04:00:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.356+0000 7fb4dfe00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 0x7fb4c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.357+0000 7fb4dfe00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 0x7fb4c007b0a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc007fd0 tx=0x7fb4cc001390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.361+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb4d4061d90 con 0x7fb4e01458a0 2024-09-17T04:00:34.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.581+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4e006a140 con 0x7fb4e01458a0 2024-09-17T04:00:34.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.582+0000 7fb4dcc00640 1 -- 172.21.15.129:0/3230681978 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7fb4d40667c0 con 0x7fb4e01458a0 2024-09-17T04:00:34.581 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:34.581 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:34.582 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:34.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.584+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 msgr2=0x7fb4c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.584+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 0x7fb4c007b0a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc007fd0 tx=0x7fb4cc001390 comp rx=0 tx=0).stop 2024-09-17T04:00:34.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 msgr2=0x7fb4e015ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e015ef50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb4d400c3f0 tx=0x7fb4d400c7f0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 shutdown_connections 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb4c0078b90 0x7fb4c007b0a0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb4e0139510 0x7fb4e0138d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb4e0145e60 0x7fb4e0138820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 --2- 172.21.15.129:0/3230681978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb4e01458a0 0x7fb4e015ef50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 >> 172.21.15.129:0/3230681978 conn(0x7fb4e006cfc0 msgr2=0x7fb4e0072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 shutdown_connections 2024-09-17T04:00:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.585+0000 7fb4e5e00640 1 -- 172.21.15.129:0/3230681978 wait complete. 2024-09-17T04:00:34.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 -- 172.21.15.129:0/1907056117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 msgr2=0x7f9da8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/1907056117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9da8004c50 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00c3c0 tx=0x7f9da8005830 comp rx=0 tx=0).stop 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 -- 172.21.15.129:0/1907056117 shutdown_connections 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/1907056117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9da8004c50 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 -- 172.21.15.129:0/1907056117 >> 172.21.15.129:0/1907056117 conn(0x7f9dc006d410 msgr2=0x7f9dc00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.829+0000 7f9dc7e00640 1 -- 172.21.15.129:0/1907056117 shutdown_connections 2024-09-17T04:00:34.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.830+0000 7f9dc7e00640 1 -- 172.21.15.129:0/1907056117 wait complete. 2024-09-17T04:00:34.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.832+0000 7f9dc7e00640 1 Processor -- start 2024-09-17T04:00:34.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.832+0000 7f9dc7e00640 1 -- start start 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc00c8170 0x7f9dc01f90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc00c8580 0x7f9dc01f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc00cfa50 con 0x7f9da8004800 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc00cf8d0 con 0x7f9dc00c8170 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc00cfbd0 con 0x7f9dc00c8580 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.833+0000 7f9dc5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43288/0 (socket says 172.21.15.129:43288) 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 -- 172.21.15.129:0/994237733 learned_addr learned my addr 172.21.15.129:0/994237733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:34.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc00c8580 0x7f9dc01f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc4a00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc00c8170 0x7f9dc01f90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 -- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc00c8580 msgr2=0x7f9dc01f95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc00c8580 0x7f9dc01f95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 -- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc00c8170 msgr2=0x7f9dc01f90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc00c8170 0x7f9dc01f90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 -- 172.21.15.129:0/994237733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc01f9f70 con 0x7f9da8004800 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc5400640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006060 tx=0x7f9db4006460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db4015070 con 0x7f9da8004800 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9db400f960 con 0x7f9da8004800 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.834+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc00c8d50 con 0x7f9da8004800 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.835+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db400fd90 con 0x7f9da8004800 2024-09-17T04:00:34.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.835+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc00c9310 con 0x7f9da8004800 2024-09-17T04:00:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.836+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc006a140 con 0x7f9da8004800 2024-09-17T04:00:34.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.837+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9db4011430 con 0x7f9da8004800 2024-09-17T04:00:34.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.838+0000 7f9dbd600640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 0x7f9da007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:34.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.838+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f9db409bad0 con 0x7f9da8004800 2024-09-17T04:00:34.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.838+0000 7f9dc4a00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 0x7f9da007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:34.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.839+0000 7f9dc4a00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 0x7f9da007b0a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9dac033000 tx=0x7f9dac02d070 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:34.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:34.842+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9db4063820 con 0x7f9da8004800 2024-09-17T04:00:35.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.061+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dc00b29f0 con 0x7f9da8004800 2024-09-17T04:00:35.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.061+0000 7f9dbd600640 1 -- 172.21.15.129:0/994237733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f9db4068250 con 0x7f9da8004800 2024-09-17T04:00:35.061 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:35.061 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:35.061 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 msgr2=0x7f9da007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 0x7f9da007b0a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9dac033000 tx=0x7f9dac02d070 comp rx=0 tx=0).stop 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 msgr2=0x7f9dc015f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f9db4006060 tx=0x7f9db4006460 comp rx=0 tx=0).stop 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 shutdown_connections 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da0078b90 0x7f9da007b0a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc00c8580 0x7f9dc01f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc00c8170 0x7f9dc01f90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 --2- 172.21.15.129:0/994237733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9da8004800 0x7f9dc015f6c0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 >> 172.21.15.129:0/994237733 conn(0x7f9dc006d410 msgr2=0x7f9dc00734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:35.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 shutdown_connections 2024-09-17T04:00:35.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.064+0000 7f9dc7e00640 1 -- 172.21.15.129:0/994237733 wait complete. 2024-09-17T04:00:35.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 -- 172.21.15.129:0/1472952171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240cd1c0 msgr2=0x7fbc241609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 --2- 172.21.15.129:0/1472952171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240cd1c0 0x7fbc241609c0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fbc24072080 tx=0x7fbc1c02aac0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 -- 172.21.15.129:0/1472952171 shutdown_connections 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 --2- 172.21.15.129:0/1472952171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240cd1c0 0x7fbc241609c0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 --2- 172.21.15.129:0/1472952171 >> v2:172.21.15.181:3300/0 conn(0x7fbc240b4140 0x7fbc240ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 -- 172.21.15.129:0/1472952171 >> 172.21.15.129:0/1472952171 conn(0x7fbc2406d410 msgr2=0x7fbc240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.306+0000 7fbc2a200640 1 -- 172.21.15.129:0/1472952171 shutdown_connections 2024-09-17T04:00:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.307+0000 7fbc2a200640 1 -- 172.21.15.129:0/1472952171 wait complete. 2024-09-17T04:00:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.309+0000 7fbc2a200640 1 Processor -- start 2024-09-17T04:00:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.309+0000 7fbc2a200640 1 -- start start 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc240b4140 0x7fbc241f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc240cd1c0 0x7fbc2409bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc24160260 con 0x7fbc240705a0 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc241600e0 con 0x7fbc240cd1c0 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc2a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbc24160f00 con 0x7fbc240b4140 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.310+0000 7fbc23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43312/0 (socket says 172.21.15.129:43312) 2024-09-17T04:00:35.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 -- 172.21.15.129:0/408613438 learned_addr learned my addr 172.21.15.129:0/408613438 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23e00640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc240cd1c0 0x7fbc2409bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc22a00640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc240b4140 0x7fbc241f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 -- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc240b4140 msgr2=0x7fbc241f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc240b4140 0x7fbc241f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 -- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc240cd1c0 msgr2=0x7fbc2409bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc240cd1c0 0x7fbc2409bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 -- 172.21.15.129:0/408613438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc240c9e90 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.311+0000 7fbc23400640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400c3a0 tx=0x7fbc1400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.312+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14017070 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.312+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc140053c0 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.312+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc240ca170 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.312+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14003850 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.312+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc241f56b0 con 0x7fbc240705a0 2024-09-17T04:00:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.313+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc240ccc80 con 0x7fbc240705a0 2024-09-17T04:00:35.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.314+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbc1401a070 con 0x7fbc240705a0 2024-09-17T04:00:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.315+0000 7fbc1b400640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 0x7fbc0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.315+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fbc1409a070 con 0x7fbc240705a0 2024-09-17T04:00:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.316+0000 7fbc22a00640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 0x7fbc0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.316+0000 7fbc22a00640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 0x7fbc0407b0a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbc24073ca0 tx=0x7fbc10005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.320+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbc240cd5d0 con 0x7fbc240705a0 2024-09-17T04:00:35.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.504+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc2406a140 con 0x7fbc240705a0 2024-09-17T04:00:35.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.505+0000 7fbc1b400640 1 -- 172.21.15.129:0/408613438 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+16787 (secure 0 0 0) 0x7fbc14061e50 con 0x7fbc240705a0 2024-09-17T04:00:35.505 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:35.505 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":163,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:16.178377+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:12.563914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:13.619002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 msgr2=0x7fbc0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 0x7fbc0407b0a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fbc24073ca0 tx=0x7fbc10005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 msgr2=0x7fbc241f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fbc1400c3a0 tx=0x7fbc1400c860 comp rx=0 tx=0).stop 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 shutdown_connections 2024-09-17T04:00:35.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc04078b90 0x7fbc0407b0a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc240cd1c0 0x7fbc2409bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc240b4140 0x7fbc241f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 --2- 172.21.15.129:0/408613438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc240705a0 0x7fbc241f6510 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 >> 172.21.15.129:0/408613438 conn(0x7fbc2406d410 msgr2=0x7fbc240ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 shutdown_connections 2024-09-17T04:00:35.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.508+0000 7fbc2a200640 1 -- 172.21.15.129:0/408613438 wait complete. 2024-09-17T04:00:35.530 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:00:35.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:35.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 -- 172.21.15.129:0/388752556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 msgr2=0x7f0510169710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 --2- 172.21.15.129:0/388752556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f0510169710 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f0508005fd0 tx=0x7f050802aa90 comp rx=0 tx=0).stop 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 -- 172.21.15.129:0/388752556 shutdown_connections 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 --2- 172.21.15.129:0/388752556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f0510169710 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 --2- 172.21.15.129:0/388752556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100b3220 0x7f05100d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 -- 172.21.15.129:0/388752556 >> 172.21.15.129:0/388752556 conn(0x7f051009d8d0 msgr2=0x7f051008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 -- 172.21.15.129:0/388752556 shutdown_connections 2024-09-17T04:00:35.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.750+0000 7f0516200640 1 -- 172.21.15.129:0/388752556 wait complete. 2024-09-17T04:00:35.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.752+0000 7f0516200640 1 Processor -- start 2024-09-17T04:00:35.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.752+0000 7f0516200640 1 -- start start 2024-09-17T04:00:35.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05100b3220 0x7f05100a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100d4a80 0x7f051015fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05100d0070 con 0x7f05100d16d0 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05100cfef0 con 0x7f05100b3220 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0516200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05100b9ec0 con 0x7f05100d4a80 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43336/0 (socket says 172.21.15.129:43336) 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 -- 172.21.15.129:0/3947715519 learned_addr learned my addr 172.21.15.129:0/3947715519 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0514e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05100b3220 0x7f05100a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0515800640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100d4a80 0x7f051015fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 -- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100d4a80 msgr2=0x7f051015fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100d4a80 0x7f051015fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 -- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05100b3220 msgr2=0x7f05100a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05100b3220 0x7f05100a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.754+0000 7f0507e00640 1 -- 172.21.15.129:0/3947715519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051015b320 con 0x7f05100d16d0 2024-09-17T04:00:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f0507e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c3c0 tx=0x7f04f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8017070 con 0x7f05100d16d0 2024-09-17T04:00:35.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f04f80035f0 con 0x7f05100d16d0 2024-09-17T04:00:35.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f8003a80 con 0x7f05100d16d0 2024-09-17T04:00:35.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051015b5c0 con 0x7f05100d16d0 2024-09-17T04:00:35.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.755+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f051015bb80 con 0x7f05100d16d0 2024-09-17T04:00:35.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.756+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f051015c240 con 0x7f05100d16d0 2024-09-17T04:00:35.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.757+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f04f801a070 con 0x7f05100d16d0 2024-09-17T04:00:35.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.758+0000 7f050d600640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 0x7f04ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:35.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.758+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f04f809a060 con 0x7f05100d16d0 2024-09-17T04:00:35.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.759+0000 7f0514e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 0x7f04ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:35.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.759+0000 7f0514e00640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 0x7f04ec07b0a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0500005fd0 tx=0x7f0500005e20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:35.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.763+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f051015c240 con 0x7f05100d16d0 2024-09-17T04:00:35.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.980+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05100d0c50 con 0x7f05100d16d0 2024-09-17T04:00:35.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.981+0000 7f050d600640 1 -- 172.21.15.129:0/3947715519 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7f051015b320 con 0x7f05100d16d0 2024-09-17T04:00:35.980 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:35.980 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:35.981 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 msgr2=0x7f04ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 0x7f04ec07b0a0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f0500005fd0 tx=0x7f0500005e20 comp rx=0 tx=0).stop 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 msgr2=0x7f051015f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c3c0 tx=0x7f04f800c880 comp rx=0 tx=0).stop 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 shutdown_connections 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f04ec078b90 0x7f04ec07b0a0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f05100d4a80 0x7f051015fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100d16d0 0x7f051015f680 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 --2- 172.21.15.129:0/3947715519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f05100b3220 0x7f05100a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:35.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 >> 172.21.15.129:0/3947715519 conn(0x7f051009d8d0 msgr2=0x7f051008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:35.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 shutdown_connections 2024-09-17T04:00:35.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:35.983+0000 7f0516200640 1 -- 172.21.15.129:0/3947715519 wait complete. 2024-09-17T04:00:36.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 -- 172.21.15.129:0/3558849886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c0cd390 msgr2=0x7fb12c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 --2- 172.21.15.129:0/3558849886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c0cd390 0x7fb12c070100 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0ce800 tx=0x7fb12402aaa0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 -- 172.21.15.129:0/3558849886 shutdown_connections 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 --2- 172.21.15.129:0/3558849886 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c0cd390 0x7fb12c070100 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 --2- 172.21.15.129:0/3558849886 >> v2:172.21.15.181:3300/0 conn(0x7fb12c15da00 0x7fb12c0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 -- 172.21.15.129:0/3558849886 >> 172.21.15.129:0/3558849886 conn(0x7fb12c06d410 msgr2=0x7fb12c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 -- 172.21.15.129:0/3558849886 shutdown_connections 2024-09-17T04:00:36.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.227+0000 7fb130e00640 1 -- 172.21.15.129:0/3558849886 wait complete. 2024-09-17T04:00:36.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.230+0000 7fb130e00640 1 Processor -- start 2024-09-17T04:00:36.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.230+0000 7fb130e00640 1 -- start start 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb12c0b46c0 0x7fb12c09d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb12c15da00 0x7fb12c156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c0709d0 con 0x7fb12c09dc40 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb12c070850 con 0x7fb12c0b46c0 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.231+0000 7fb130e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb12c070b50 con 0x7fb12c15da00 2024-09-17T04:00:36.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43350/0 (socket says 172.21.15.129:43350) 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 -- 172.21.15.129:0/999761049 learned_addr learned my addr 172.21.15.129:0/999761049 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb129e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb12c0b46c0 0x7fb12c09d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb121400640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb12c15da00 0x7fb12c156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 -- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb12c15da00 msgr2=0x7fb12c156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb12c15da00 0x7fb12c156fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 -- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb12c0b46c0 msgr2=0x7fb12c09d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb12c0b46c0 0x7fb12c09d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 -- 172.21.15.129:0/999761049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12c0b1bd0 con 0x7fb12c09dc40 2024-09-17T04:00:36.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.232+0000 7fb12a800640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fb124005fd0 tx=0x7fb124003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:36.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.233+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124044070 con 0x7fb12c09dc40 2024-09-17T04:00:36.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.233+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb124003e00 con 0x7fb12c09dc40 2024-09-17T04:00:36.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.233+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12c0cecd0 con 0x7fb12c09dc40 2024-09-17T04:00:36.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.233+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1240044b0 con 0x7fb12c09dc40 2024-09-17T04:00:36.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.233+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12c156330 con 0x7fb12c09dc40 2024-09-17T04:00:36.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.234+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb12c099af0 con 0x7fb12c09dc40 2024-09-17T04:00:36.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.235+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb124047070 con 0x7fb12c09dc40 2024-09-17T04:00:36.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.236+0000 7fb122a00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 0x7fb10807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.236+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7fb1240bbd00 con 0x7fb12c09dc40 2024-09-17T04:00:36.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.236+0000 7fb129e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 0x7fb10807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.237+0000 7fb129e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 0x7fb10807b0a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ac0 tx=0x7fb11c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:36.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.240+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb12c099af0 con 0x7fb12c09dc40 2024-09-17T04:00:36.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.460+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb12c070100 con 0x7fb12c09dc40 2024-09-17T04:00:36.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.461+0000 7fb122a00640 1 -- 172.21.15.129:0/999761049 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) ==== 95+0+5245 (secure 0 0 0) 0x7fb124032030 con 0x7fb12c09dc40 2024-09-17T04:00:36.460 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:36.460 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":144,"btime":"2024-09-17T04:00:18:205533+0000","default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:14.652383+0000","modified":"2024-09-17T04:00:17.717373+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8542":{"gid":8542,"name":"d","rank":0,"incarnation":140,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/1344901561","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1344901561},{"type":"v1","addr":"172.21.15.181:6837","nonce":1344901561}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8542,"qdb_cluster":[8542]},"id":12}]} 2024-09-17T04:00:36.461 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 144 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 msgr2=0x7fb10807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 0x7fb10807b0a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009ac0 tx=0x7fb11c005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 msgr2=0x7fb12c157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fb124005fd0 tx=0x7fb124003c10 comp rx=0 tx=0).stop 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 shutdown_connections 2024-09-17T04:00:36.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb108078b90 0x7fb10807b0a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb12c09dc40 0x7fb12c157510 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb12c15da00 0x7fb12c156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 --2- 172.21.15.129:0/999761049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb12c0b46c0 0x7fb12c09d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 >> 172.21.15.129:0/999761049 conn(0x7fb12c06d410 msgr2=0x7fb12c074bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 shutdown_connections 2024-09-17T04:00:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.464+0000 7fb130e00640 1 -- 172.21.15.129:0/999761049 wait complete. 2024-09-17T04:00:36.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 -- 172.21.15.129:0/1575933483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 msgr2=0x7f36c40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 --2- 172.21.15.129:0/1575933483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40ccc80 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f36c40ccbc0 tx=0x7f36bc02a980 comp rx=0 tx=0).stop 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 -- 172.21.15.129:0/1575933483 shutdown_connections 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 --2- 172.21.15.129:0/1575933483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40ccc80 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 --2- 172.21.15.129:0/1575933483 >> v2:172.21.15.181:3300/0 conn(0x7f36c406f410 0x7f36c40cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 --2- 172.21.15.129:0/1575933483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c4099570 0x7f36c406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 -- 172.21.15.129:0/1575933483 >> 172.21.15.129:0/1575933483 conn(0x7f36c406d410 msgr2=0x7f36c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 -- 172.21.15.129:0/1575933483 shutdown_connections 2024-09-17T04:00:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.707+0000 7f36cb800640 1 -- 172.21.15.129:0/1575933483 wait complete. 2024-09-17T04:00:36.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.709+0000 7f36cb800640 1 Processor -- start 2024-09-17T04:00:36.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.710+0000 7f36cb800640 1 -- start start 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c406f410 0x7f36c4150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c4099570 0x7f36c40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f36c4160ba0 con 0x7f36c40cf7f0 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f36c4160a20 con 0x7f36c4099570 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36cb800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f36c4160d20 con 0x7f36c406f410 2024-09-17T04:00:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36c9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36c9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43366/0 (socket says 172.21.15.129:43366) 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.711+0000 7f36c9800640 1 -- 172.21.15.129:0/1341117340 learned_addr learned my addr 172.21.15.129:0/1341117340 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 -- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c406f410 msgr2=0x7f36c4150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c406f410 0x7f36c4150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 -- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c4099570 msgr2=0x7f36c40cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c4099570 0x7f36c40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 -- 172.21.15.129:0/1341117340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36c40c8380 con 0x7f36c40cf7f0 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c9800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009d60 tx=0x7f36bc003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:36.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc034040 con 0x7f36c40cf7f0 2024-09-17T04:00:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36bc004540 con 0x7f36c40cf7f0 2024-09-17T04:00:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36c40c8660 con 0x7f36c40cf7f0 2024-09-17T04:00:36.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.712+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36bc045480 con 0x7f36c40cf7f0 2024-09-17T04:00:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.713+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36c40c8c20 con 0x7f36c40cf7f0 2024-09-17T04:00:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.714+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36c4161240 con 0x7f36c40cf7f0 2024-09-17T04:00:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.715+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f36bc037020 con 0x7f36c40cf7f0 2024-09-17T04:00:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.716+0000 7f36c0c00640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 0x7f36a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.716+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f36bc0bc5a0 con 0x7f36c40cf7f0 2024-09-17T04:00:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.717+0000 7f36c8e00640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 0x7f36a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.717+0000 7f36c8e00640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 0x7f36a407b0a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f36b800b8e0 tx=0x7f36b8005c90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.721+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f36bc084a80 con 0x7f36c40cf7f0 2024-09-17T04:00:36.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.904+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36c406a140 con 0x7f36c40cf7f0 2024-09-17T04:00:36.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.905+0000 7f36c0c00640 1 -- 172.21.15.129:0/1341117340 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) ==== 74+0+16787 (secure 0 0 0) 0x7f36bc084380 con 0x7f36c40cf7f0 2024-09-17T04:00:36.904 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:36.905 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":163,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:16.178377+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:12.563914+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"162","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:13.619002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 msgr2=0x7f36a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 0x7f36a407b0a0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f36b800b8e0 tx=0x7f36b8005c90 comp rx=0 tx=0).stop 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 msgr2=0x7f36c40cb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f36bc009d60 tx=0x7f36bc003f90 comp rx=0 tx=0).stop 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 shutdown_connections 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f36a4078b90 0x7f36a407b0a0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f36c40cf7f0 0x7f36c40cb8d0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f36c4099570 0x7f36c40cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 --2- 172.21.15.129:0/1341117340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f36c406f410 0x7f36c4150250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 >> 172.21.15.129:0/1341117340 conn(0x7f36c406d410 msgr2=0x7f36c40ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 shutdown_connections 2024-09-17T04:00:36.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:36.908+0000 7f36cb800640 1 -- 172.21.15.129:0/1341117340 wait complete. 2024-09-17T04:00:36.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:00:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 -- 172.21.15.129:0/892086239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d406f2e0 msgr2=0x7f69d40cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 --2- 172.21.15.129:0/892086239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d406f2e0 0x7f69d40cd1c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f69d40d1370 tx=0x7f69cc02aa30 comp rx=0 tx=0).stop 2024-09-17T04:00:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 -- 172.21.15.129:0/892086239 shutdown_connections 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 --2- 172.21.15.129:0/892086239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d406f2e0 0x7f69d40cd1c0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 --2- 172.21.15.129:0/892086239 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 0x7f69d40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 --2- 172.21.15.129:0/892086239 >> v2:172.21.15.181:3300/0 conn(0x7f69d40705a0 0x7f69d40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 -- 172.21.15.129:0/892086239 >> 172.21.15.129:0/892086239 conn(0x7f69d406d410 msgr2=0x7f69d40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 -- 172.21.15.129:0/892086239 shutdown_connections 2024-09-17T04:00:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.154+0000 7f69d9c00640 1 -- 172.21.15.129:0/892086239 wait complete. 2024-09-17T04:00:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.156+0000 7f69d9c00640 1 Processor -- start 2024-09-17T04:00:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.156+0000 7f69d9c00640 1 -- start start 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 0x7f69d41f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69d406f2e0 0x7f69d40b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69d4160b40 con 0x7f69d40705a0 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69d41609c0 con 0x7f69d406f2e0 2024-09-17T04:00:37.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d9c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f69d4160cc0 con 0x7f69d406eed0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43388/0 (socket says 172.21.15.129:43388) 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d3400640 1 -- 172.21.15.129:0/1147056438 learned_addr learned my addr 172.21.15.129:0/1147056438 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d2a00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 0x7f69d41f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.158+0000 7f69d2000640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69d406f2e0 0x7f69d40b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 -- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 msgr2=0x7f69d41f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 0x7f69d41f66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 -- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69d406f2e0 msgr2=0x7f69d40b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69d406f2e0 0x7f69d40b1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 -- 172.21.15.129:0/1147056438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d41f3bb0 con 0x7f69d40705a0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d3400640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009d60 tx=0x7f69cc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc033050 con 0x7f69d40705a0 2024-09-17T04:00:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69cc004450 con 0x7f69d40705a0 2024-09-17T04:00:37.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d41f3e90 con 0x7f69d40705a0 2024-09-17T04:00:37.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc038450 con 0x7f69d40705a0 2024-09-17T04:00:37.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.159+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d41f4450 con 0x7f69d40705a0 2024-09-17T04:00:37.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.160+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d4161220 con 0x7f69d40705a0 2024-09-17T04:00:37.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.162+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f69cc036070 con 0x7f69d40705a0 2024-09-17T04:00:37.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.162+0000 7f69caa00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 0x7f69b407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.163+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(163..163 src has 1..163) ==== 5709+0+0 (secure 0 0 0) 0x7f69cc0bc170 con 0x7f69d40705a0 2024-09-17T04:00:37.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.163+0000 7f69d2a00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 0x7f69b407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.164+0000 7f69d2a00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 0x7f69b407b0a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f69c4005fd0 tx=0x7f69c400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:37.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.167+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f69cc083f60 con 0x7f69d40705a0 2024-09-17T04:00:37.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.387+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f69d406a140 con 0x7f69d40705a0 2024-09-17T04:00:37.387 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:00:37.388+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:00:37.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.390+0000 7f69caa00640 1 -- 172.21.15.129:0/1147056438 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v145) ==== 161+0+0 (secure 0 0 0) 0x7f69cc088990 con 0x7f69d40705a0 2024-09-17T04:00:37.389 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:00:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 msgr2=0x7f69b407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 0x7f69b407b0a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f69c4005fd0 tx=0x7f69c400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 msgr2=0x7f69d40b1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f69cc009d60 tx=0x7f69cc003f10 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 shutdown_connections 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69b4078b90 0x7f69b407b0a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69d40705a0 0x7f69d40b1ca0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69d406f2e0 0x7f69d40b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 --2- 172.21.15.129:0/1147056438 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69d406eed0 0x7f69d41f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 >> 172.21.15.129:0/1147056438 conn(0x7f69d406d410 msgr2=0x7f69d40aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.393+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 shutdown_connections 2024-09-17T04:00:37.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.394+0000 7f69d9c00640 1 -- 172.21.15.129:0/1147056438 wait complete. 2024-09-17T04:00:37.405 INFO:tasks.ceph.mds.d.smithi181.stderr: -7338> 2024-09-17T04:00:17.151+0000 7f3033400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:00:37.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:00:37.532 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:00:37.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 -- 172.21.15.129:0/3941849858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 msgr2=0x7f4fd80cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3941849858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd80cd5e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8005fd0 tx=0x7f4fc802a670 comp rx=0 tx=0).stop 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 -- 172.21.15.129:0/3941849858 shutdown_connections 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3941849858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd80cd5e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 -- 172.21.15.129:0/3941849858 >> 172.21.15.129:0/3941849858 conn(0x7f4fd806cfc0 msgr2=0x7f4fd806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 -- 172.21.15.129:0/3941849858 shutdown_connections 2024-09-17T04:00:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.640+0000 7f4fdd600640 1 -- 172.21.15.129:0/3941849858 wait complete. 2024-09-17T04:00:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.642+0000 7f4fdd600640 1 Processor -- start 2024-09-17T04:00:37.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.643+0000 7f4fdd600640 1 -- start start 2024-09-17T04:00:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fd80b4ee0 0x7f4fd81f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fd81e7ab0 0x7f4fd8136c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd815d630 con 0x7f4fd80cdf10 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd80b5500 con 0x7f4fd81e7ab0 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdd600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd815d7b0 con 0x7f4fd80b4ee0 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fcf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fcf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43410/0 (socket says 172.21.15.129:43410) 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fcf400640 1 -- 172.21.15.129:0/3718840410 learned_addr learned my addr 172.21.15.129:0/3718840410 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fd7e00640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fd80b4ee0 0x7f4fd81f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.644+0000 7f4fdcc00640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fd81e7ab0 0x7f4fd8136c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 -- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fd80b4ee0 msgr2=0x7f4fd81f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fd80b4ee0 0x7f4fd81f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 -- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fd81e7ab0 msgr2=0x7f4fd8136c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fd81e7ab0 0x7f4fd8136c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 -- 172.21.15.129:0/3718840410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd81e6a10 con 0x7f4fd80cdf10 2024-09-17T04:00:37.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fcf400640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000c400 tx=0x7f4fc000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0017070 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4fc00053c0 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0003830 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.645+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd81e6cf0 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.646+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd81e72b0 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.646+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd80cd5e0 con 0x7f4fd80cdf10 2024-09-17T04:00:37.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.648+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4fc001a070 con 0x7f4fd80cdf10 2024-09-17T04:00:37.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.649+0000 7f4fd4c00640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 0x7f4fb407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:37.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.650+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f4fc0099ff0 con 0x7f4fd80cdf10 2024-09-17T04:00:37.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.650+0000 7f4fd7e00640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 0x7f4fb407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:37.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.651+0000 7f4fd7e00640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 0x7f4fb407b0d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8033000 tx=0x7f4fc8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:37.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.654+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4fc0061e40 con 0x7f4fd80cdf10 2024-09-17T04:00:37.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:37.873+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f4fd80cf9c0 con 0x7f4fd80cdf10 2024-09-17T04:00:38.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.392+0000 7f4fd4c00640 1 -- 172.21.15.129:0/3718840410 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v146) ==== 104+0+0 (secure 0 0 0) 0x7f4fc0066870 con 0x7f4fd80cdf10 2024-09-17T04:00:38.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 msgr2=0x7f4fb407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 0x7f4fb407b0d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8033000 tx=0x7f4fc8004730 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 msgr2=0x7f4fd81f72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000c400 tx=0x7f4fc000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 shutdown_connections 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4fb4078bc0 0x7f4fb407b0d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4fd81e7ab0 0x7f4fd8136c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4fd80cdf10 0x7f4fd81f72b0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 --2- 172.21.15.129:0/3718840410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4fd80b4ee0 0x7f4fd81f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.394+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 >> 172.21.15.129:0/3718840410 conn(0x7f4fd806cfc0 msgr2=0x7f4fd815ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:38.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.395+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 shutdown_connections 2024-09-17T04:00:38.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.395+0000 7f4fdd600640 1 -- 172.21.15.129:0/3718840410 wait complete. 2024-09-17T04:00:38.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 -- 172.21.15.129:0/1093132531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000d0c50 msgr2=0x7f03000b60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 --2- 172.21.15.129:0/1093132531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000d0c50 0x7f03000b60e0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f02e8005fd0 tx=0x7f02e802a610 comp rx=0 tx=0).stop 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 -- 172.21.15.129:0/1093132531 shutdown_connections 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 --2- 172.21.15.129:0/1093132531 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000d0c50 0x7f03000b60e0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 --2- 172.21.15.129:0/1093132531 >> v2:172.21.15.181:3300/0 conn(0x7f03000b3440 0x7f030006dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 -- 172.21.15.129:0/1093132531 >> 172.21.15.129:0/1093132531 conn(0x7f03000d4660 msgr2=0x7f03000d4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 -- 172.21.15.129:0/1093132531 shutdown_connections 2024-09-17T04:00:38.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.645+0000 7f0305600640 1 -- 172.21.15.129:0/1093132531 wait complete. 2024-09-17T04:00:38.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.647+0000 7f0305600640 1 Processor -- start 2024-09-17T04:00:38.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.647+0000 7f0305600640 1 -- start start 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.648+0000 7f0305600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 0x7f030015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f0305600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f0305600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03000d0c50 0x7f03000c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f0305600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f03000cfe40 con 0x7f03000b9ec0 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f0305600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f03000cfcc0 con 0x7f03000b3440 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f0305600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f03000cffc0 con 0x7f03000d0c50 2024-09-17T04:00:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43422/0 (socket says 172.21.15.129:43422) 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 -- 172.21.15.129:0/2011916804 learned_addr learned my addr 172.21.15.129:0/2011916804 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02fea00640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 0x7f030015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02ff400640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03000d0c50 0x7f03000c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 -- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03000d0c50 msgr2=0x7f03000c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03000d0c50 0x7f03000c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 -- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 msgr2=0x7f030015f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 0x7f030015f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02f6000640 1 -- 172.21.15.129:0/2011916804 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03000c9080 con 0x7f03000b9ec0 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.649+0000 7f02fea00640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 0x7f030015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f02f6000640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f02e8034040 tx=0x7f02e80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:38.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e803e030 con 0x7f03000b9ec0 2024-09-17T04:00:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02e8003ca0 con 0x7f03000b9ec0 2024-09-17T04:00:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e8035040 con 0x7f03000b9ec0 2024-09-17T04:00:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03000ca870 con 0x7f03000b9ec0 2024-09-17T04:00:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.650+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03000c9320 con 0x7f03000b9ec0 2024-09-17T04:00:38.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.651+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03000ca5e0 con 0x7f03000b9ec0 2024-09-17T04:00:38.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.653+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f02e8041030 con 0x7f03000b9ec0 2024-09-17T04:00:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.653+0000 7f02f7400640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 0x7f02dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.654+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(164..164 src has 1..164) ==== 5795+0+0 (secure 0 0 0) 0x7f02e80bb3d0 con 0x7f03000b9ec0 2024-09-17T04:00:38.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.654+0000 7f02fea00640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 0x7f02dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:38.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.655+0000 7f02fea00640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 0x7f02dc07b0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02f0005fd0 tx=0x7f02f000f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:38.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.658+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f02e80837a0 con 0x7f03000b9ec0 2024-09-17T04:00:38.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:38.841+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f03000d0490 con 0x7f03000b9ec0 2024-09-17T04:00:39.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.398+0000 7f02f7400640 1 -- 172.21.15.129:0/2011916804 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v165) ==== 181+0+0 (secure 0 0 0) 0x7f02e80830a0 con 0x7f03000b9ec0 2024-09-17T04:00:39.397 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:00:39.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 msgr2=0x7f02dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:39.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 0x7f02dc07b0d0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f02f0005fd0 tx=0x7f02f000f360 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 msgr2=0x7f03001f9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f02e8034040 tx=0x7f02e80038a0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 shutdown_connections 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f02dc078bc0 0x7f02dc07b0d0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f03000d0c50 0x7f03000c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f03000b9ec0 0x7f03001f9120 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 --2- 172.21.15.129:0/2011916804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03000b3440 0x7f030015f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 >> 172.21.15.129:0/2011916804 conn(0x7f03000d4660 msgr2=0x7f0300075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 shutdown_connections 2024-09-17T04:00:39.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.402+0000 7f0305600640 1 -- 172.21.15.129:0/2011916804 wait complete. 2024-09-17T04:00:39.425 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.650+0000 7f97ec200640 1 -- 172.21.15.129:0/1457523113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 msgr2=0x7f97e40cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.650+0000 7f97ec200640 1 --2- 172.21.15.129:0/1457523113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e40cbea0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005fd0 tx=0x7f97dc02aa10 comp rx=0 tx=0).stop 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 -- 172.21.15.129:0/1457523113 shutdown_connections 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 --2- 172.21.15.129:0/1457523113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e40cbea0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 --2- 172.21.15.129:0/1457523113 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e40b3f70 0x7f97e40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 -- 172.21.15.129:0/1457523113 >> 172.21.15.129:0/1457523113 conn(0x7f97e40ccc40 msgr2=0x7f97e40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 -- 172.21.15.129:0/1457523113 shutdown_connections 2024-09-17T04:00:39.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.651+0000 7f97ec200640 1 -- 172.21.15.129:0/1457523113 wait complete. 2024-09-17T04:00:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.653+0000 7f97ec200640 1 Processor -- start 2024-09-17T04:00:39.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.653+0000 7f97ec200640 1 -- start start 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97e4074580 0x7f97e41585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e41f9120 0x7f97e4156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f97e40cc960 con 0x7f97e40cba90 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f97e40cc7e0 con 0x7f97e4074580 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.654+0000 7f97ec200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f97e4166f20 con 0x7f97e41f9120 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43436/0 (socket says 172.21.15.129:43436) 2024-09-17T04:00:39.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 -- 172.21.15.129:0/4018888713 learned_addr learned my addr 172.21.15.129:0/4018888713 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e9800640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97e4074580 0x7f97e41585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97ea200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e41f9120 0x7f97e4156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 -- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e41f9120 msgr2=0x7f97e4156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e41f9120 0x7f97e4156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 -- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97e4074580 msgr2=0x7f97e41585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97e4074580 0x7f97e41585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.655+0000 7f97e8e00640 1 -- 172.21.15.129:0/4018888713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e409c660 con 0x7f97e40cba90 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97e8e00640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c3d0 tx=0x7f97d000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0017070 con 0x7f97e40cba90 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f97d00053c0 con 0x7f97e40cba90 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e409c940 con 0x7f97e40cba90 2024-09-17T04:00:39.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0003850 con 0x7f97e40cba90 2024-09-17T04:00:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.656+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e409cf00 con 0x7f97e40cba90 2024-09-17T04:00:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.657+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97e40b3f70 con 0x7f97e40cba90 2024-09-17T04:00:39.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.658+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f97d001a070 con 0x7f97e40cba90 2024-09-17T04:00:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.659+0000 7f97e1600640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 0x7f97c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.659+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 5357+0+0 (secure 0 0 0) 0x7f97d0099f00 con 0x7f97e40cba90 2024-09-17T04:00:39.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.660+0000 7f97e9800640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 0x7f97c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:39.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.660+0000 7f97e9800640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 0x7f97c407b0d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f97d800bb50 tx=0x7f97d800b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:39.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.664+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f97d0061e50 con 0x7f97e40cba90 2024-09-17T04:00:39.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:39.846+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f97e40b16b0 con 0x7f97e40cba90 2024-09-17T04:00:40.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.399+0000 7f97e1600640 1 -- 172.21.15.129:0/4018888713 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v166) ==== 169+0+0 (secure 0 0 0) 0x7f97d0066880 con 0x7f97e40cba90 2024-09-17T04:00:40.399 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 msgr2=0x7f97c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 0x7f97c407b0d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f97d800bb50 tx=0x7f97d800b2e0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 msgr2=0x7f97e41f8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c3d0 tx=0x7f97d000c890 comp rx=0 tx=0).stop 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 shutdown_connections 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f97c4078bc0 0x7f97c407b0d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f97e41f9120 0x7f97e4156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f97e40cba90 0x7f97e41f8be0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 --2- 172.21.15.129:0/4018888713 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f97e4074580 0x7f97e41585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.403+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 >> 172.21.15.129:0/4018888713 conn(0x7f97e40ccc40 msgr2=0x7f97e40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.404+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 shutdown_connections 2024-09-17T04:00:40.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.404+0000 7f97ec200640 1 -- 172.21.15.129:0/4018888713 wait complete. 2024-09-17T04:00:40.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.653+0000 7fc42b600640 1 -- 172.21.15.129:0/2210588777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424099570 msgr2=0x7fc42406eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.653+0000 7fc42b600640 1 --2- 172.21.15.129:0/2210588777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424099570 0x7fc42406eed0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc418009830 tx=0x7fc41802a510 comp rx=0 tx=0).stop 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.654+0000 7fc42b600640 1 -- 172.21.15.129:0/2210588777 shutdown_connections 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.654+0000 7fc42b600640 1 --2- 172.21.15.129:0/2210588777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc424099570 0x7fc42406eed0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.654+0000 7fc42b600640 1 -- 172.21.15.129:0/2210588777 >> 172.21.15.129:0/2210588777 conn(0x7fc42406d410 msgr2=0x7fc4240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.654+0000 7fc42b600640 1 -- 172.21.15.129:0/2210588777 shutdown_connections 2024-09-17T04:00:40.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.654+0000 7fc42b600640 1 -- 172.21.15.129:0/2210588777 wait complete. 2024-09-17T04:00:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.656+0000 7fc42b600640 1 Processor -- start 2024-09-17T04:00:40.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.656+0000 7fc42b600640 1 -- start start 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc424099570 0x7fc4240c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4240cf1d0 0x7fc42414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc424160c80 con 0x7fc4240ccc80 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc424160b00 con 0x7fc424099570 2024-09-17T04:00:40.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.657+0000 7fc42b600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc424160e00 con 0x7fc4240cf1d0 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43452/0 (socket says 172.21.15.129:43452) 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 -- 172.21.15.129:0/3761810733 learned_addr learned my addr 172.21.15.129:0/3761810733 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc428c00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc424099570 0x7fc4240c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc429600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4240cf1d0 0x7fc42414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 -- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4240cf1d0 msgr2=0x7fc42414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4240cf1d0 0x7fc42414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 -- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc424099570 msgr2=0x7fc4240c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc424099570 0x7fc4240c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.658+0000 7fc423e00640 1 -- 172.21.15.129:0/3761810733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42415ffd0 con 0x7fc4240ccc80 2024-09-17T04:00:40.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc423e00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc41000c3a0 tx=0x7fc41000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:40.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410011790 con 0x7fc4240ccc80 2024-09-17T04:00:40.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc410017070 con 0x7fc4240ccc80 2024-09-17T04:00:40.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410011ce0 con 0x7fc4240ccc80 2024-09-17T04:00:40.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42414c0d0 con 0x7fc4240ccc80 2024-09-17T04:00:40.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.659+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42414c690 con 0x7fc4240ccc80 2024-09-17T04:00:40.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.660+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4241601a0 con 0x7fc4240ccc80 2024-09-17T04:00:40.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.661+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc41001c460 con 0x7fc4240ccc80 2024-09-17T04:00:40.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.663+0000 7fc420c00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 0x7fc40407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:40.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.663+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7fc41009a790 con 0x7fc4240ccc80 2024-09-17T04:00:40.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.664+0000 7fc428c00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 0x7fc40407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:40.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.664+0000 7fc428c00640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 0x7fc40407b0a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc418034000 tx=0x7fc418004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:40.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.668+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc410011e80 con 0x7fc4240ccc80 2024-09-17T04:00:40.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.866+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7fc42406a140 con 0x7fc4240ccc80 2024-09-17T04:00:40.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.871+0000 7fc420c00640 1 -- 172.21.15.129:0/3761810733 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v57) ==== 71+0+0 (secure 0 0 0) 0x7fc4100627e0 con 0x7fc4240ccc80 2024-09-17T04:00:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 msgr2=0x7fc40407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 0x7fc40407b0a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fc418034000 tx=0x7fc418004730 comp rx=0 tx=0).stop 2024-09-17T04:00:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 msgr2=0x7fc42415fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fc41000c3a0 tx=0x7fc41000c860 comp rx=0 tx=0).stop 2024-09-17T04:00:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 shutdown_connections 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc404078b90 0x7fc40407b0a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4240cf1d0 0x7fc42414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4240ccc80 0x7fc42415fa90 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 --2- 172.21.15.129:0/3761810733 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc424099570 0x7fc4240c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 >> 172.21.15.129:0/3761810733 conn(0x7fc42406d410 msgr2=0x7fc4240ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 shutdown_connections 2024-09-17T04:00:40.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:40.874+0000 7fc42b600640 1 -- 172.21.15.129:0/3761810733 wait complete. 2024-09-17T04:00:40.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup' 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.124+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3018182209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 msgr2=0x7fb0ac0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.124+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3018182209 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0ac0051f0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc001d90 tx=0x7fb0ac005830 comp rx=0 tx=0).stop 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.124+0000 7fb0c9e00640 1 -- 172.21.15.129:0/3018182209 reap_dead start 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.125+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3018182209 shutdown_connections 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.125+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3018182209 >> 172.21.15.129:0/3018182209 conn(0x7fb0c406d410 msgr2=0x7fb0c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:41.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.125+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3018182209 shutdown_connections 2024-09-17T04:00:41.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.125+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3018182209 wait complete. 2024-09-17T04:00:41.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.127+0000 7fb0cbe00640 1 Processor -- start 2024-09-17T04:00:41.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.127+0000 7fb0cbe00640 1 -- start start 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 0x7fb0c41e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0c40992f0 0x7fb0c409b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c40ce2f0 con 0x7fb0ac004d40 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c40ce170 con 0x7fb0c415b5a0 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.128+0000 7fb0cbe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb0c40ce470 con 0x7fb0c40992f0 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43460/0 (socket says 172.21.15.129:43460) 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 -- 172.21.15.129:0/3321429442 learned_addr learned my addr 172.21.15.129:0/3321429442 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9e00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0c40992f0 0x7fb0c409b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c8a00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 0x7fb0c41e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 -- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0c40992f0 msgr2=0x7fb0c409b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0c40992f0 0x7fb0c409b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 -- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 msgr2=0x7fb0c41e9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 0x7fb0c41e9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 -- 172.21.15.129:0/3321429442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c41ea480 con 0x7fb0ac004d40 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c8a00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 0x7fb0c41e9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.129+0000 7fb0c9400640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8011ae0 tx=0x7fb0b8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.130+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b801e070 con 0x7fb0ac004d40 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.130+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0b800fe70 con 0x7fb0ac004d40 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.130+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c41ea760 con 0x7fb0ac004d40 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.130+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b8004290 con 0x7fb0ac004d40 2024-09-17T04:00:41.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.130+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c41ead20 con 0x7fb0ac004d40 2024-09-17T04:00:41.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.131+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c406fc80 con 0x7fb0ac004d40 2024-09-17T04:00:41.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.132+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0b8021070 con 0x7fb0ac004d40 2024-09-17T04:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.133+0000 7fb0c1600640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 0x7fb0a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.134+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7fb0b809b5c0 con 0x7fb0ac004d40 2024-09-17T04:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.134+0000 7fb0c8a00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 0x7fb0a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.135+0000 7fb0c8a00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 0x7fb0a407b0a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40aea50 tx=0x7fb0b00067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:41.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.138+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0b80636d0 con 0x7fb0ac004d40 2024-09-17T04:00:41.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.387+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]} v 0) -- 0x7fb0c406a140 con 0x7fb0ac004d40 2024-09-17T04:00:41.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.406+0000 7fb0c1600640 1 -- 172.21.15.129:0/3321429442 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_mv_hardlink_cleanup"]}]=0 v582) ==== 139+0+0 (secure 0 0 0) 0x7fb0b8068100 con 0x7fb0ac004d40 2024-09-17T04:00:41.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 msgr2=0x7fb0a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 0x7fb0a407b0a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb0c40aea50 tx=0x7fb0b00067a0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 msgr2=0x7fb0c41eb520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8011ae0 tx=0x7fb0b8009e90 comp rx=0 tx=0).stop 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 shutdown_connections 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0a4078b90 0x7fb0a407b0a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0c40992f0 0x7fb0c409b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0c415b5a0 0x7fb0c41e9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 --2- 172.21.15.129:0/3321429442 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0ac004d40 0x7fb0c41eb520 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 >> 172.21.15.129:0/3321429442 conn(0x7fb0c406d410 msgr2=0x7fb0c40caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 shutdown_connections 2024-09-17T04:00:41.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.409+0000 7fb0cbe00640 1 -- 172.21.15.129:0/3321429442 wait complete. 2024-09-17T04:00:41.431 INFO:tasks.cephfs_test_runner:That when doing a rename from A to B, and B has hardlinks, ... ok 2024-09-17T04:00:41.431 INFO:tasks.cephfs_test_runner:Starting test: test_open_inode (tasks.cephfs.test_strays.TestStrays) That the case of a dentry unlinked while a client holds an 2024-09-17T04:00:41.431 INFO:tasks.cephfs_test_runner:test_open_inode (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:00:41.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:41.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 -- 172.21.15.129:0/2713975693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6144003a70 msgr2=0x7f6144003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 --2- 172.21.15.129:0/2713975693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6144003a70 0x7f6144003ee0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f6150009b40 tx=0x7f6144005830 comp rx=0 tx=0).stop 2024-09-17T04:00:41.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6165600640 1 -- 172.21.15.129:0/2713975693 reap_dead start 2024-09-17T04:00:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 -- 172.21.15.129:0/2713975693 shutdown_connections 2024-09-17T04:00:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 -- 172.21.15.129:0/2713975693 >> 172.21.15.129:0/2713975693 conn(0x7f616009d8d0 msgr2=0x7f616008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 -- 172.21.15.129:0/2713975693 shutdown_connections 2024-09-17T04:00:41.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.654+0000 7f6166000640 1 -- 172.21.15.129:0/2713975693 wait complete. 2024-09-17T04:00:41.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.656+0000 7f6166000640 1 Processor -- start 2024-09-17T04:00:41.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.657+0000 7f6166000640 1 -- start start 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6144003a70 0x7f61600caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61600cd0a0 0x7f61600cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f61600d0070 con 0x7f6160156780 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f61600cfef0 con 0x7f61600cd0a0 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6166000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f61600b9ec0 con 0x7f6144003a70 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6165600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6165600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43478/0 (socket says 172.21.15.129:43478) 2024-09-17T04:00:41.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.658+0000 7f6165600640 1 -- 172.21.15.129:0/217431843 learned_addr learned my addr 172.21.15.129:0/217431843 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6164c00640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6144003a70 0x7f61600caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6157e00640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61600cd0a0 0x7f61600cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 -- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6144003a70 msgr2=0x7f61600caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6144003a70 0x7f61600caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 -- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61600cd0a0 msgr2=0x7f61600cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61600cd0a0 0x7f61600cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 -- 172.21.15.129:0/217431843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6160157800 con 0x7f6160156780 2024-09-17T04:00:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6165600640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6158005fd0 tx=0x7f6158011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615800fd70 con 0x7f6160156780 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6158004b20 con 0x7f6160156780 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158003970 con 0x7f6160156780 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.659+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61600cdc60 con 0x7f6160156780 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.660+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6160154c40 con 0x7f6160156780 2024-09-17T04:00:41.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.660+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61600d0c50 con 0x7f6160156780 2024-09-17T04:00:41.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.662+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6158020020 con 0x7f6160156780 2024-09-17T04:00:41.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.663+0000 7f615d600640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 0x7f613c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:41.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.663+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7f615809b190 con 0x7f6160156780 2024-09-17T04:00:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.664+0000 7f6164c00640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 0x7f613c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:41.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.665+0000 7f6164c00640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 0x7f613c07b0d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6150035000 tx=0x7f615002d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:41.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.668+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f61580631e0 con 0x7f6160156780 2024-09-17T04:00:41.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.850+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61600cd9f0 con 0x7f6160156780 2024-09-17T04:00:41.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.851+0000 7f615d600640 1 -- 172.21.15.129:0/217431843 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) ==== 74+0+13092 (secure 0 0 0) 0x7f6158067c10 con 0x7f6160156780 2024-09-17T04:00:41.851 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:41.851 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":166,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:40.398617+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":161,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/1344901561":"2024-09-18T04:00:37.388509+0000","172.21.15.181:6835/1344901561":"2024-09-18T04:00:37.388509+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 msgr2=0x7f613c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 0x7f613c07b0d0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6150035000 tx=0x7f615002d040 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 msgr2=0x7f6160156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6158005fd0 tx=0x7f6158011f90 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 shutdown_connections 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f613c078bc0 0x7f613c07b0d0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6160156780 0x7f6160156b90 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61600cd0a0 0x7f61600cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 --2- 172.21.15.129:0/217431843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6144003a70 0x7f61600caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 >> 172.21.15.129:0/217431843 conn(0x7f616009d8d0 msgr2=0x7f61600d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 shutdown_connections 2024-09-17T04:00:41.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:41.853+0000 7f6166000640 1 -- 172.21.15.129:0/217431843 wait complete. 2024-09-17T04:00:41.880 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:00:41.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 -- 172.21.15.129:0/1285438299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 msgr2=0x7fa0cc06fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 --2- 172.21.15.129:0/1285438299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc06fab0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009830 tx=0x7fa0bc02a530 comp rx=0 tx=0).stop 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 -- 172.21.15.129:0/1285438299 shutdown_connections 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 --2- 172.21.15.129:0/1285438299 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc06fab0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 -- 172.21.15.129:0/1285438299 >> 172.21.15.129:0/1285438299 conn(0x7fa0cc0ccc40 msgr2=0x7fa0cc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 -- 172.21.15.129:0/1285438299 shutdown_connections 2024-09-17T04:00:42.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.104+0000 7fa0d2800640 1 -- 172.21.15.129:0/1285438299 wait complete. 2024-09-17T04:00:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.107+0000 7fa0d2800640 1 Processor -- start 2024-09-17T04:00:42.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.107+0000 7fa0d2800640 1 -- start start 2024-09-17T04:00:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0cc0cbf10 0x7fa0cc1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0cc1611c0 0x7fa0cc15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc0cb0a0 con 0x7fa0cc0705b0 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0cc0caf20 con 0x7fa0cc0cbf10 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0d2800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa0cc0cb220 con 0x7fa0cc1611c0 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0cb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0cb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43488/0 (socket says 172.21.15.129:43488) 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.108+0000 7fa0cb400640 1 -- 172.21.15.129:0/2068642316 learned_addr learned my addr 172.21.15.129:0/2068642316 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0caa00640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0cc0cbf10 0x7fa0cc1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cbe00640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0cc1611c0 0x7fa0cc15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 -- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0cc1611c0 msgr2=0x7fa0cc15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0cc1611c0 0x7fa0cc15efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 -- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0cc0cbf10 msgr2=0x7fa0cc1525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0cc0cbf10 0x7fa0cc1525f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 -- 172.21.15.129:0/2068642316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0cc1f5e80 con 0x7fa0cc0705b0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0cb400640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009950 tx=0x7fa0bc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.109+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc031d60 con 0x7fa0cc0705b0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.110+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0bc033e10 con 0x7fa0cc0705b0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.110+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc033310 con 0x7fa0cc0705b0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.110+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0cc1f6160 con 0x7fa0cc0705b0 2024-09-17T04:00:42.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.110+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0cc0c4f50 con 0x7fa0cc0705b0 2024-09-17T04:00:42.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.111+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0cc0c4d40 con 0x7fa0cc0705b0 2024-09-17T04:00:42.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.112+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa0bc0334b0 con 0x7fa0cc0705b0 2024-09-17T04:00:42.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.113+0000 7fa0c3400640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.114+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7fa0bc0bbd20 con 0x7fa0cc0705b0 2024-09-17T04:00:42.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.114+0000 7fa0caa00640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.115+0000 7fa0caa00640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0ca430 tx=0x7fa0b8007cc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:42.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.118+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa0bc03f030 con 0x7fa0cc0705b0 2024-09-17T04:00:42.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.300+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa0cc06fab0 con 0x7fa0cc0705b0 2024-09-17T04:00:42.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.301+0000 7fa0c3400640 1 -- 172.21.15.129:0/2068642316 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v166) ==== 93+0+10 (secure 0 0 0) 0x7fa0bc083e10 con 0x7fa0cc0705b0 2024-09-17T04:00:42.301 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:00:42.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 msgr2=0x7fa0ac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa0cc0ca430 tx=0x7fa0b8007cc0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 msgr2=0x7fa0cc1f5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009950 tx=0x7fa0bc033c20 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 shutdown_connections 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0cc1611c0 0x7fa0cc15efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0cc0cbf10 0x7fa0cc1525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 --2- 172.21.15.129:0/2068642316 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0cc0705b0 0x7fa0cc1f5630 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 >> 172.21.15.129:0/2068642316 conn(0x7fa0cc0ccc40 msgr2=0x7fa0cc0b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 shutdown_connections 2024-09-17T04:00:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.303+0000 7fa0d2800640 1 -- 172.21.15.129:0/2068642316 wait complete. 2024-09-17T04:00:42.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode' 2024-09-17T04:00:42.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 -- 172.21.15.129:0/974254782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6888003aa0 msgr2=0x7f6888003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 --2- 172.21.15.129:0/974254782 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6888003aa0 0x7f6888003f10 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f689400c2e0 tx=0x7f6888005830 comp rx=0 tx=0).stop 2024-09-17T04:00:42.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68a9a00640 1 -- 172.21.15.129:0/974254782 reap_dead start 2024-09-17T04:00:42.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 -- 172.21.15.129:0/974254782 shutdown_connections 2024-09-17T04:00:42.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 -- 172.21.15.129:0/974254782 >> 172.21.15.129:0/974254782 conn(0x7f68a406cfc0 msgr2=0x7f68a406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:42.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 -- 172.21.15.129:0/974254782 shutdown_connections 2024-09-17T04:00:42.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.552+0000 7f68aa400640 1 -- 172.21.15.129:0/974254782 wait complete. 2024-09-17T04:00:42.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.554+0000 7f68aa400640 1 Processor -- start 2024-09-17T04:00:42.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.554+0000 7f68aa400640 1 -- start start 2024-09-17T04:00:42.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6888003aa0 0x7f68a413e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68a4072d30 0x7f68a4073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68a415cbd0 con 0x7f68a41e87c0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f68a40d0be0 con 0x7f6888003aa0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68aa400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f68a415cd50 con 0x7f68a4072d30 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68a9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68a9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43514/0 (socket says 172.21.15.129:43514) 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68a9a00640 1 -- 172.21.15.129:0/1353543203 learned_addr learned my addr 172.21.15.129:0/1353543203 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f689be00640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68a4072d30 0x7f68a4073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.556+0000 7f68a9000640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6888003aa0 0x7f68a413e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 -- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68a4072d30 msgr2=0x7f68a4073140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68a4072d30 0x7f68a4073140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 -- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6888003aa0 msgr2=0x7f68a413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6888003aa0 0x7f68a413e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 -- 172.21.15.129:0/1353543203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a41e9110 con 0x7f68a41e87c0 2024-09-17T04:00:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a9a00640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f68a40cda50 tx=0x7f689c010f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c003970 con 0x7f68a41e87c0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f689c015070 con 0x7f68a41e87c0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a4073a20 con 0x7f68a41e87c0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c004030 con 0x7f68a41e87c0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.557+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a41f8e30 con 0x7f68a41e87c0 2024-09-17T04:00:42.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.558+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a4073790 con 0x7f68a41e87c0 2024-09-17T04:00:42.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.560+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f689c0041d0 con 0x7f68a41e87c0 2024-09-17T04:00:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.560+0000 7f68a1600640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 0x7f688007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.561+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7f689c09b560 con 0x7f68a41e87c0 2024-09-17T04:00:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.561+0000 7f68a9000640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 0x7f688007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:42.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.562+0000 7f68a9000640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 0x7f688007b0a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6894033000 tx=0x7f68940019f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:42.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.565+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f689c063c00 con 0x7f68a41e87c0 2024-09-17T04:00:42.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:42.815+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode"]} v 0) -- 0x7f68a40b46e0 con 0x7f68a41e87c0 2024-09-17T04:00:43.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.416+0000 7f68a1600640 1 -- 172.21.15.129:0/1353543203 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_open_inode"]}]=0 v584) ==== 133+0+0 (secure 0 0 0) 0x7f689c063500 con 0x7f68a41e87c0 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 msgr2=0x7f688007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 0x7f688007b0a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f6894033000 tx=0x7f68940019f0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 msgr2=0x7f68a41e8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f68a40cda50 tx=0x7f689c010f40 comp rx=0 tx=0).stop 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 shutdown_connections 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6880078b90 0x7f688007b0a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68a41e87c0 0x7f68a41e8bd0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68a4072d30 0x7f68a4073140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 --2- 172.21.15.129:0/1353543203 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6888003aa0 0x7f68a413e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 >> 172.21.15.129:0/1353543203 conn(0x7f68a406cfc0 msgr2=0x7f68a415d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:43.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 shutdown_connections 2024-09-17T04:00:43.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.419+0000 7f68aa400640 1 -- 172.21.15.129:0/1353543203 wait complete. 2024-09-17T04:00:43.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 -- 172.21.15.129:0/2613444432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06eed0 msgr2=0x7f233c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 --2- 172.21.15.129:0/2613444432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06eed0 0x7f233c0ccc80 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f233c0d1370 tx=0x7f232802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 -- 172.21.15.129:0/2613444432 shutdown_connections 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 --2- 172.21.15.129:0/2613444432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06eed0 0x7f233c0ccc80 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 --2- 172.21.15.129:0/2613444432 >> v2:172.21.15.181:3300/0 conn(0x7f233c0705a0 0x7f233c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 -- 172.21.15.129:0/2613444432 >> 172.21.15.129:0/2613444432 conn(0x7f233c06d410 msgr2=0x7f233c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.663+0000 7f2344400640 1 -- 172.21.15.129:0/2613444432 shutdown_connections 2024-09-17T04:00:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.664+0000 7f2344400640 1 -- 172.21.15.129:0/2613444432 wait complete. 2024-09-17T04:00:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.666+0000 7f2344400640 1 Processor -- start 2024-09-17T04:00:43.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.666+0000 7f2344400640 1 -- start start 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 0x7f233c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f233c0705a0 0x7f233c0ba0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f233c161250 con 0x7f233c06f2e0 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f233c1610d0 con 0x7f233c06eed0 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.667+0000 7f2344400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f233c1613d0 con 0x7f233c0705a0 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:43528/0 (socket says 172.21.15.129:43528) 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 -- 172.21.15.129:0/1221842963 learned_addr learned my addr 172.21.15.129:0/1221842963 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2342400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f233c0705a0 0x7f233c0ba0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:43.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341a00640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 0x7f233c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 -- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f233c0705a0 msgr2=0x7f233c0ba0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f233c0705a0 0x7f233c0ba0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 -- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 msgr2=0x7f233c1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 0x7f233c1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 -- 172.21.15.129:0/1221842963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233c1f3d90 con 0x7f233c06f2e0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341a00640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 0x7f233c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.668+0000 7f2341000640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2328034040 tx=0x7f2328003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.669+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232803e030 con 0x7f233c06f2e0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.669+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2328003d10 con 0x7f233c06f2e0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.669+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2328035040 con 0x7f233c06f2e0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.669+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f233c1f4070 con 0x7f233c06f2e0 2024-09-17T04:00:43.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.669+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f233c1f4630 con 0x7f233c06f2e0 2024-09-17T04:00:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.670+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f233c0cd5a0 con 0x7f233c06f2e0 2024-09-17T04:00:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.671+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2328041030 con 0x7f233c06f2e0 2024-09-17T04:00:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.672+0000 7f2339600640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 0x7f231c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.672+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7f23280bb140 con 0x7f233c06f2e0 2024-09-17T04:00:43.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.673+0000 7f2341a00640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 0x7f231c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:43.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.673+0000 7f2341a00640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 0x7f231c07b0d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2330009b80 tx=0x7f2330005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:43.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.676+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2328083900 con 0x7f233c06f2e0 2024-09-17T04:00:43.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.859+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f233c06a140 con 0x7f233c06f2e0 2024-09-17T04:00:43.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.860+0000 7f2339600640 1 -- 172.21.15.129:0/1221842963 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v57) ==== 59+0+2294 (secure 0 0 0) 0x7f2328083200 con 0x7f233c06f2e0 2024-09-17T04:00:43.860 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:00:43.860 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:00:43.860 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:00:43.861 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:00:43.862 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:00:43.863 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:00:43.863 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:00:43.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 msgr2=0x7f231c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 0x7f231c07b0d0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f2330009b80 tx=0x7f2330005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 msgr2=0x7f233c0b9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f2328034040 tx=0x7f2328003820 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 shutdown_connections 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f231c078bc0 0x7f231c07b0d0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f233c0705a0 0x7f233c0ba0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f233c06f2e0 0x7f233c0b9ba0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 --2- 172.21.15.129:0/1221842963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f233c06eed0 0x7f233c1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 >> 172.21.15.129:0/1221842963 conn(0x7f233c06d410 msgr2=0x7f233c0aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:43.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 shutdown_connections 2024-09-17T04:00:43.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:43.863+0000 7f2344400640 1 -- 172.21.15.129:0/1221842963 wait complete. 2024-09-17T04:00:43.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:00:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 -- 172.21.15.129:0/1222649152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 msgr2=0x7f3b840cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 --2- 172.21.15.129:0/1222649152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840cdfb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c005fd0 tx=0x7f3b7c02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 -- 172.21.15.129:0/1222649152 shutdown_connections 2024-09-17T04:00:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 --2- 172.21.15.129:0/1222649152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840cdfb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 --2- 172.21.15.129:0/1222649152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b8415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 -- 172.21.15.129:0/1222649152 >> 172.21.15.129:0/1222649152 conn(0x7f3b840d00b0 msgr2=0x7f3b840d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 -- 172.21.15.129:0/1222649152 shutdown_connections 2024-09-17T04:00:44.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.109+0000 7f3b89c00640 1 -- 172.21.15.129:0/1222649152 wait complete. 2024-09-17T04:00:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.111+0000 7f3b89c00640 1 Processor -- start 2024-09-17T04:00:44.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.111+0000 7f3b89c00640 1 -- start start 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b841560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840ce880 con 0x7f3b84070930 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3b840ce700 con 0x7f3b8415bfd0 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3b840cea00 con 0x7f3b8415b680 2024-09-17T04:00:44.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46690/0 (socket says 172.21.15.129:46690) 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b83e00640 1 -- 172.21.15.129:0/347093892 learned_addr learned my addr 172.21.15.129:0/347093892 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b83400640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b841560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.113+0000 7f3b89200640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 -- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 msgr2=0x7f3b841560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b841560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 -- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 msgr2=0x7f3b840c7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840c7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 -- 172.21.15.129:0/347093892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b840c9790 con 0x7f3b84070930 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83400640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b841560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:44.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b83e00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c460 tx=0x7f3b7400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b740052d0 con 0x7f3b84070930 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b74005490 con 0x7f3b84070930 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b840c9a70 con 0x7f3b84070930 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74012870 con 0x7f3b84070930 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.114+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b841f8c80 con 0x7f3b84070930 2024-09-17T04:00:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.115+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b840ceee0 con 0x7f3b84070930 2024-09-17T04:00:44.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.117+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3b74012a10 con 0x7f3b84070930 2024-09-17T04:00:44.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.118+0000 7f3b7be00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 0x7f3b6407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.118+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7f3b7409a680 con 0x7f3b84070930 2024-09-17T04:00:44.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.118+0000 7f3b83400640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 0x7f3b6407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.119+0000 7f3b83400640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 0x7f3b6407b0a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3b8406a0c0 tx=0x7f3b70009260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:44.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.123+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3b74062ca0 con 0x7f3b84070930 2024-09-17T04:00:44.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.303+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f3b841577b0 con 0x7f3b84070930 2024-09-17T04:00:44.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.311+0000 7f3b7be00640 1 -- 172.21.15.129:0/347093892 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) ==== 122+0+0 (secure 0 0 0) 0x7f3b740625a0 con 0x7f3b84070930 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 msgr2=0x7f3b6407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 0x7f3b6407b0a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3b8406a0c0 tx=0x7f3b70009260 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 msgr2=0x7f3b84073290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400c460 tx=0x7f3b7400c920 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 shutdown_connections 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3b64078b90 0x7f3b6407b0a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3b8415bfd0 0x7f3b840c7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3b8415b680 0x7f3b841560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 --2- 172.21.15.129:0/347093892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3b84070930 0x7f3b84073290 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 >> 172.21.15.129:0/347093892 conn(0x7f3b840d00b0 msgr2=0x7f3b840cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:44.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 shutdown_connections 2024-09-17T04:00:44.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.314+0000 7f3b89c00640 1 -- 172.21.15.129:0/347093892 wait complete. 2024-09-17T04:00:44.337 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:00:44.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:44.418 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:00:44.506 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:00:44.507 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:44.507 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:44.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:00:44.542 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:44.543 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:44.579 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:44.579 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:00:44.579 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:44.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:00:44.618 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:00:44.618 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:00:44.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:00:44.656 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:00:44.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 -- 172.21.15.129:0/3477118542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80cd1c0 msgr2=0x7efca81609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 --2- 172.21.15.129:0/3477118542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80cd1c0 0x7efca81609c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7efca0005fd0 tx=0x7efca002a8b0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 -- 172.21.15.129:0/3477118542 shutdown_connections 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 --2- 172.21.15.129:0/3477118542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80cd1c0 0x7efca81609c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 --2- 172.21.15.129:0/3477118542 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 0x7efca80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.879+0000 7efcad200640 1 -- 172.21.15.129:0/3477118542 >> 172.21.15.129:0/3477118542 conn(0x7efca806d410 msgr2=0x7efca80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.880+0000 7efcad200640 1 -- 172.21.15.129:0/3477118542 shutdown_connections 2024-09-17T04:00:44.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.880+0000 7efcad200640 1 -- 172.21.15.129:0/3477118542 wait complete. 2024-09-17T04:00:44.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.882+0000 7efcad200640 1 Processor -- start 2024-09-17T04:00:44.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.882+0000 7efcad200640 1 -- start start 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 0x7efca81f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 0x7efca814c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efca8160260 con 0x7efca80cd1c0 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efca81600e0 con 0x7efca80705a0 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.883+0000 7efcad200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efca8160f00 con 0x7efca80b4140 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 0x7efca81f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46702/0 (socket says 172.21.15.129:46702) 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 -- 172.21.15.129:0/3612456101 learned_addr learned my addr 172.21.15.129:0/3612456101 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efc9d400640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 0x7efca814c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 -- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 msgr2=0x7efca814c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 0x7efca814c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 -- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 msgr2=0x7efca81f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 0x7efca81f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 -- 172.21.15.129:0/3612456101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca81fa140 con 0x7efca80cd1c0 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca5e00640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 0x7efca81f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.884+0000 7efca6800640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7efca0006010 tx=0x7efca0031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.885+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0003e00 con 0x7efca80cd1c0 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.885+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efca0031a80 con 0x7efca80cd1c0 2024-09-17T04:00:44.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.885+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca814c730 con 0x7efca80cd1c0 2024-09-17T04:00:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.885+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0033070 con 0x7efca80cd1c0 2024-09-17T04:00:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.885+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca814e840 con 0x7efca80cd1c0 2024-09-17T04:00:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.886+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca814c900 con 0x7efca80cd1c0 2024-09-17T04:00:44.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.887+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efca0033210 con 0x7efca80cd1c0 2024-09-17T04:00:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.888+0000 7efc9ea00640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 0x7efc8407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.888+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4943+0+0 (secure 0 0 0) 0x7efca00bbde0 con 0x7efca80cd1c0 2024-09-17T04:00:44.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.889+0000 7efca5e00640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 0x7efc8407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:44.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.889+0000 7efca5e00640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 0x7efc8407b0a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7efc98007fd0 tx=0x7efc98007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:44.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:44.892+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efca81fa140 con 0x7efca80cd1c0 2024-09-17T04:00:45.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.111+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efca806a140 con 0x7efca80cd1c0 2024-09-17T04:00:45.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.111+0000 7efc9ea00640 1 -- 172.21.15.129:0/3612456101 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+3928 (secure 0 0 0) 0x7efca003f030 con 0x7efca80cd1c0 2024-09-17T04:00:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:45.112 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":146,"btime":"2024-09-17T04:00:38:390629+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7427,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3295591553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3295591553},{"type":"v1","addr":"172.21.15.129:6835","nonce":3295591553}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8533,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3569070974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3569070974},{"type":"v1","addr":"172.21.15.129:6834","nonce":3569070974}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":138},{"gid":8539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3212106358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3212106358},{"type":"v1","addr":"172.21.15.181:6836","nonce":3212106358}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":139},{"gid":8779,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1822371677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1822371677},{"type":"v1","addr":"172.21.15.181:6837","nonce":1822371677}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":146}],"filesystems":[]} 2024-09-17T04:00:45.112 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 146 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 msgr2=0x7efc8407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 0x7efc8407b0a0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7efc98007fd0 tx=0x7efc98007cf0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 msgr2=0x7efca80c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7efca0006010 tx=0x7efca0031650 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 shutdown_connections 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc84078b90 0x7efc8407b0a0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80cd1c0 0x7efca80c6720 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b4140 0x7efca814c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 --2- 172.21.15.129:0/3612456101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80705a0 0x7efca81f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 >> 172.21.15.129:0/3612456101 conn(0x7efca806d410 msgr2=0x7efca80cbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:45.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 shutdown_connections 2024-09-17T04:00:45.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.114+0000 7efcad200640 1 -- 172.21.15.129:0/3612456101 wait complete. 2024-09-17T04:00:45.141 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:00:45.141 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:00:45.141 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:00:45.141 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:45.141 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:00:45.141 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:00:45.142 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:00:45.142 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:45.142 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:00:45.142 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:00:45.142 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:00:45.142 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:45.142 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:00:45.143 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:00:45.143 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:00:45.143 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:00:45.166 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:00:45.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:00:45.169 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:00:45.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:00:45.176 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:00:45.176 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:00:45.177 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:00:45.177 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:00:45.212 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:00:45.213 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:45.215 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:00:45.216 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:45.221 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:00:45.221 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:45.224 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:00:45.224 DEBUG:teuthology.parallel:result is None 2024-09-17T04:00:45.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:00:45.407 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:00:45.410 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:00:45.417 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:00:45.418 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:00:45.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 -- 172.21.15.129:0/1967399060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b40cf2b0 msgr2=0x7f18b40cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 --2- 172.21.15.129:0/1967399060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b40cf2b0 0x7f18b40cf6c0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f189c009860 tx=0x7f189c02a560 comp rx=0 tx=0).stop 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 -- 172.21.15.129:0/1967399060 shutdown_connections 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 --2- 172.21.15.129:0/1967399060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b40cf2b0 0x7f18b40cf6c0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 -- 172.21.15.129:0/1967399060 >> 172.21.15.129:0/1967399060 conn(0x7f18b406d410 msgr2=0x7f18b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 -- 172.21.15.129:0/1967399060 shutdown_connections 2024-09-17T04:00:45.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.462+0000 7f18b9a00640 1 -- 172.21.15.129:0/1967399060 wait complete. 2024-09-17T04:00:45.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.465+0000 7f18b9a00640 1 Processor -- start 2024-09-17T04:00:45.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.465+0000 7f18b9a00640 1 -- start start 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18b40ccc80 0x7f18b40c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 0x7f18b40c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4160cb0 con 0x7f18b4099570 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4160b30 con 0x7f18b40cf2b0 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b9a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f18b4160e30 con 0x7f18b40ccc80 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.466+0000 7f18b2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:45.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46752/0 (socket says 172.21.15.129:46752) 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 -- 172.21.15.129:0/131967792 learned_addr learned my addr 172.21.15.129:0/131967792 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b3400640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 0x7f18b40c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18aa000640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18b40ccc80 0x7f18b40c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 -- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18b40ccc80 msgr2=0x7f18b40c93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18b40ccc80 0x7f18b40c93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 -- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 msgr2=0x7f18b40c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 0x7f18b40c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 -- 172.21.15.129:0/131967792 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b40c8c20 con 0x7f18b4099570 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b3400640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 0x7f18b40c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:45.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18b2a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f18a400c3f0 tx=0x7f18a400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4017070 con 0x7f18b4099570 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.467+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f18a40035f0 con 0x7f18b4099570 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.468+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b4157a70 con 0x7f18b4099570 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.468+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4003a40 con 0x7f18b4099570 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.468+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b40cb6c0 con 0x7f18b4099570 2024-09-17T04:00:45.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.469+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b40cf6c0 con 0x7f18b4099570 2024-09-17T04:00:45.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.470+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f18a401a070 con 0x7f18b4099570 2024-09-17T04:00:45.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.471+0000 7f18ab400640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 0x7f189007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:45.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.471+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 4943+0+0 (secure 0 0 0) 0x7f18a4099e20 con 0x7f18b4099570 2024-09-17T04:00:45.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.472+0000 7f18aa000640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 0x7f189007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:45.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.472+0000 7f18aa000640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 0x7f189007b0d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f189c034000 tx=0x7f189c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:45.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.475+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f18a4061e70 con 0x7f18b4099570 2024-09-17T04:00:45.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:45.669+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f18b406a140 con 0x7f18b4099570 2024-09-17T04:00:46.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.448+0000 7f18ab400640 1 -- 172.21.15.129:0/131967792 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v170) ==== 97+0+0 (secure 0 0 0) 0x7f18a40668a0 con 0x7f18b4099570 2024-09-17T04:00:46.449 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.451+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 msgr2=0x7f189007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.451+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 0x7f189007b0d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f189c034000 tx=0x7f189c004730 comp rx=0 tx=0).stop 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.451+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 msgr2=0x7f18b4157530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.451+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f18a400c3f0 tx=0x7f18a400c8b0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 shutdown_connections 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1890078bc0 0x7f189007b0d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f18b40cf2b0 0x7f18b40c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f18b40ccc80 0x7f18b40c93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 --2- 172.21.15.129:0/131967792 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f18b4099570 0x7f18b4157530 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 >> 172.21.15.129:0/131967792 conn(0x7f18b406d410 msgr2=0x7f18b40737a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:46.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 shutdown_connections 2024-09-17T04:00:46.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.452+0000 7f18b9a00640 1 -- 172.21.15.129:0/131967792 wait complete. 2024-09-17T04:00:46.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 -- 172.21.15.129:0/3442260922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 msgr2=0x7f853c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 --2- 172.21.15.129:0/3442260922 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f853c004800 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f8544009b50 tx=0x7f853c005830 comp rx=0 tx=0).stop 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f8557e00640 1 -- 172.21.15.129:0/3442260922 reap_dead start 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 -- 172.21.15.129:0/3442260922 shutdown_connections 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 -- 172.21.15.129:0/3442260922 >> 172.21.15.129:0/3442260922 conn(0x7f855806d410 msgr2=0x7f85580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 -- 172.21.15.129:0/3442260922 shutdown_connections 2024-09-17T04:00:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.700+0000 7f855e000640 1 -- 172.21.15.129:0/3442260922 wait complete. 2024-09-17T04:00:46.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.703+0000 7f855e000640 1 Processor -- start 2024-09-17T04:00:46.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.703+0000 7f855e000640 1 -- start start 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 0x7f8558159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8558159d50 0x7f8558158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85580710e0 con 0x7f853c004060 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8558070f60 con 0x7f8558099610 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f855e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f85580cd090 con 0x7f8558159d50 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f8557400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:46.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f8557400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46770/0 (socket says 172.21.15.129:46770) 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.704+0000 7f8557400640 1 -- 172.21.15.129:0/4203692780 learned_addr learned my addr 172.21.15.129:0/4203692780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8556a00640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 0x7f8558159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557e00640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8558159d50 0x7f8558158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 -- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8558159d50 msgr2=0x7f8558158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8558159d50 0x7f8558158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 -- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 msgr2=0x7f8558159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 0x7f8558159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 -- 172.21.15.129:0/4203692780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855809a5b0 con 0x7f853c004060 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8556a00640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 0x7f8558159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:46.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f8557400640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f8548006610 tx=0x7f85480039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548015070 con 0x7f853c004060 2024-09-17T04:00:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8548011920 con 0x7f853c004060 2024-09-17T04:00:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.705+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855815a820 con 0x7f853c004060 2024-09-17T04:00:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.706+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548011d70 con 0x7f853c004060 2024-09-17T04:00:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.706+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8558158110 con 0x7f853c004060 2024-09-17T04:00:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.707+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8558157c20 con 0x7f853c004060 2024-09-17T04:00:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.708+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8548010420 con 0x7f853c004060 2024-09-17T04:00:46.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.709+0000 7f854f400640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:46.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.709+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 4857+0+0 (secure 0 0 0) 0x7f854809b6f0 con 0x7f853c004060 2024-09-17T04:00:46.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.709+0000 7f8556a00640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:46.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.710+0000 7f8556a00640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 0x7f853807b0a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8544035000 tx=0x7f85440046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:46.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.713+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f85580cd5d0 con 0x7f853c004060 2024-09-17T04:00:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.964+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f855815ae00 con 0x7f853c004060 2024-09-17T04:00:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.965+0000 7f854f400640 1 -- 172.21.15.129:0/4203692780 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v38) ==== 80+0+5304 (secure 0 0 0) 0x7f8548063f00 con 0x7f853c004060 2024-09-17T04:00:46.965 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:46.965 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:00:46.965 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:00:46.966 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:46.967 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:00:46.969 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.970 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.971 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.972 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.973 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:00:46.974 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:46.975 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:46.976 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:00:46.977 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.978 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:00:46.979 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:00:46.980 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.980 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:46.980 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:00:46.980 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:00:46.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 msgr2=0x7f853807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 0x7f853807b0a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f8544035000 tx=0x7f85440046c0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 msgr2=0x7f8558098ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f8548006610 tx=0x7f85480039c0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 shutdown_connections 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8538078b90 0x7f853807b0a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8558159d50 0x7f8558158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8558099610 0x7f8558159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 --2- 172.21.15.129:0/4203692780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f853c004060 0x7f8558098ef0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 >> 172.21.15.129:0/4203692780 conn(0x7f855806d410 msgr2=0x7f85580afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 shutdown_connections 2024-09-17T04:00:46.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:46.968+0000 7f855e000640 1 -- 172.21.15.129:0/4203692780 wait complete. 2024-09-17T04:00:46.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.216+0000 7f6927200640 1 -- 172.21.15.129:0/1286496640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280d16d0 msgr2=0x7f6928169710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.216+0000 7f6927200640 1 --2- 172.21.15.129:0/1286496640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280d16d0 0x7f6928169710 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f69280b7180 tx=0x7f692002aa30 comp rx=0 tx=0).stop 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 -- 172.21.15.129:0/1286496640 shutdown_connections 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 --2- 172.21.15.129:0/1286496640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280d16d0 0x7f6928169710 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 --2- 172.21.15.129:0/1286496640 >> v2:172.21.15.181:3300/0 conn(0x7f69280b3220 0x7f69280d4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 -- 172.21.15.129:0/1286496640 >> 172.21.15.129:0/1286496640 conn(0x7f692809d8d0 msgr2=0x7f692808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:47.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 -- 172.21.15.129:0/1286496640 shutdown_connections 2024-09-17T04:00:47.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.217+0000 7f6927200640 1 -- 172.21.15.129:0/1286496640 wait complete. 2024-09-17T04:00:47.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.219+0000 7f6927200640 1 Processor -- start 2024-09-17T04:00:47.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.219+0000 7f6927200640 1 -- start start 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 0x7f692815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69280d4a80 0x7f692815df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69280d0070 con 0x7f69280b3220 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69280cfef0 con 0x7f69280d16d0 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.220+0000 7f6927200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f69280b9ec0 con 0x7f69280d4a80 2024-09-17T04:00:47.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46786/0 (socket says 172.21.15.129:46786) 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 -- 172.21.15.129:0/1891483030 learned_addr learned my addr 172.21.15.129:0/1891483030 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925400640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 0x7f692815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6926800640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69280d4a80 0x7f692815df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 -- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69280d4a80 msgr2=0x7f692815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69280d4a80 0x7f692815df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 -- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 msgr2=0x7f692815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 0x7f692815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925e00640 1 -- 172.21.15.129:0/1891483030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692815d5c0 con 0x7f69280b3220 2024-09-17T04:00:47.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.221+0000 7f6925400640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 0x7f692815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f6925e00640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f691800d750 tx=0x7f691800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918017070 con 0x7f69280b3220 2024-09-17T04:00:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6918009d40 con 0x7f69280b3220 2024-09-17T04:00:47.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69280cb6e0 con 0x7f69280b3220 2024-09-17T04:00:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69180106c0 con 0x7f69280b3220 2024-09-17T04:00:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.222+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69280cbc60 con 0x7f69280b3220 2024-09-17T04:00:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.223+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69280d0c50 con 0x7f69280b3220 2024-09-17T04:00:47.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.224+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f691801a070 con 0x7f69280b3220 2024-09-17T04:00:47.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.225+0000 7f691e000640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 0x7f690807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.225+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 4857+0+0 (secure 0 0 0) 0x7f691809ab50 con 0x7f69280b3220 2024-09-17T04:00:47.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.226+0000 7f6925400640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 0x7f690807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.226+0000 7f6925400640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 0x7f690807b0a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6914006fd0 tx=0x7f6914008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:47.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.229+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f691800de60 con 0x7f69280b3220 2024-09-17T04:00:47.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.416+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f692806a130 con 0x7f69280b3220 2024-09-17T04:00:47.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.417+0000 7f691e000640 1 -- 172.21.15.129:0/1891483030 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v170) ==== 84+0+65 (secure 0 0 0) 0x7f6918062bf0 con 0x7f69280b3220 2024-09-17T04:00:47.417 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:47.417 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:00:47.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:00:47.418 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:00:47.418 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:00:47.418 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:00:47.418 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 msgr2=0x7f690807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 0x7f690807b0a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6914006fd0 tx=0x7f6914008040 comp rx=0 tx=0).stop 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 msgr2=0x7f69280cd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f691800d750 tx=0x7f691800dc10 comp rx=0 tx=0).stop 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 shutdown_connections 2024-09-17T04:00:47.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6908078b90 0x7f690807b0a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69280d4a80 0x7f692815df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69280d16d0 0x7f692815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 --2- 172.21.15.129:0/1891483030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69280b3220 0x7f69280cd9a0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 >> 172.21.15.129:0/1891483030 conn(0x7f692809d8d0 msgr2=0x7f69280d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 shutdown_connections 2024-09-17T04:00:47.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.420+0000 7f6927200640 1 -- 172.21.15.129:0/1891483030 wait complete. 2024-09-17T04:00:47.449 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:00:47.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:00:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 -- 172.21.15.129:0/917448898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe31c004800 msgr2=0x7fe31c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 --2- 172.21.15.129:0/917448898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe31c004800 0x7fe31c004c10 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7fe3340afb90 tx=0x7fe31c005830 comp rx=0 tx=0).stop 2024-09-17T04:00:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33a000640 1 -- 172.21.15.129:0/917448898 reap_dead start 2024-09-17T04:00:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 -- 172.21.15.129:0/917448898 shutdown_connections 2024-09-17T04:00:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 -- 172.21.15.129:0/917448898 >> 172.21.15.129:0/917448898 conn(0x7fe3340b1c50 msgr2=0x7fe3340b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 -- 172.21.15.129:0/917448898 shutdown_connections 2024-09-17T04:00:47.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.676+0000 7fe33aa00640 1 -- 172.21.15.129:0/917448898 wait complete. 2024-09-17T04:00:47.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.679+0000 7fe33aa00640 1 Processor -- start 2024-09-17T04:00:47.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.679+0000 7fe33aa00640 1 -- start start 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 0x7fe3340b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 0x7fe3341f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe334145430 con 0x7fe3341f6b60 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe3341452b0 con 0x7fe3341f6210 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe3341455b0 con 0x7fe31c004800 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46808/0 (socket says 172.21.15.129:46808) 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.680+0000 7fe33a000640 1 -- 172.21.15.129:0/1383384487 learned_addr learned my addr 172.21.15.129:0/1383384487 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe338c00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 0x7fe3341f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe339600640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 0x7fe3340b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 -- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 msgr2=0x7fe3340b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 0x7fe3340b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 -- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 msgr2=0x7fe3341f6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 0x7fe3341f6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 -- 172.21.15.129:0/1383384487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33415e670 con 0x7fe3341f6b60 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe338c00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 0x7fe3341f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe339600640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 0x7fe3340b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33a000640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fe31c004780 tx=0x7fe32c006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c00fbb0 con 0x7fe3341f6b60 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe32c018070 con 0x7fe3341f6b60 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c006910 con 0x7fe3341f6b60 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.681+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe33415daf0 con 0x7fe3341f6b60 2024-09-17T04:00:47.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.682+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe33415e0b0 con 0x7fe3341f6b60 2024-09-17T04:00:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.682+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3340c4f90 con 0x7fe3341f6b60 2024-09-17T04:00:47.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.684+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe32c01d460 con 0x7fe3341f6b60 2024-09-17T04:00:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.685+0000 7fe331600640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 0x7fe31407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.685+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(170..170 src has 1..170) ==== 4857+0+0 (secure 0 0 0) 0x7fe32c09afe0 con 0x7fe3341f6b60 2024-09-17T04:00:47.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.686+0000 7fe339600640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 0x7fe31407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:47.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.686+0000 7fe339600640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 0x7fe31407b0a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe328009a30 tx=0x7fe328005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:47.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.689+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe334161020 con 0x7fe3341f6b60 2024-09-17T04:00:47.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:00:47.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:00:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:47.878+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fe3340ccfb0 con 0x7fe3341f6b60 2024-09-17T04:00:48.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.453+0000 7fe331600640 1 -- 172.21.15.129:0/1383384487 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v171) ==== 138+0+0 (secure 0 0 0) 0x7fe32c063670 con 0x7fe3341f6b60 2024-09-17T04:00:48.453 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:00:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 msgr2=0x7fe31407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 0x7fe31407b0a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fe328009a30 tx=0x7fe328005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 msgr2=0x7fe334141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fe31c004780 tx=0x7fe32c006720 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 shutdown_connections 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe314078b90 0x7fe31407b0a0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3341f6b60 0x7fe334141ae0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3341f6210 0x7fe3341f6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 --2- 172.21.15.129:0/1383384487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe31c004800 0x7fe3340b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 >> 172.21.15.129:0/1383384487 conn(0x7fe3340b1c50 msgr2=0x7fe3340cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 shutdown_connections 2024-09-17T04:00:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.456+0000 7fe33aa00640 1 -- 172.21.15.129:0/1383384487 wait complete. 2024-09-17T04:00:48.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:00:48.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 -- 172.21.15.129:0/3561554908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 msgr2=0x7f65bc06eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 --2- 172.21.15.129:0/3561554908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc06eee0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65b4005fd0 tx=0x7f65b402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 -- 172.21.15.129:0/3561554908 shutdown_connections 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 --2- 172.21.15.129:0/3561554908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc06eee0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 --2- 172.21.15.129:0/3561554908 >> v2:172.21.15.181:3300/0 conn(0x7f65bc0703b0 0x7f65bc0b3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 -- 172.21.15.129:0/3561554908 >> 172.21.15.129:0/3561554908 conn(0x7f65bc06d7a0 msgr2=0x7f65bc0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 -- 172.21.15.129:0/3561554908 shutdown_connections 2024-09-17T04:00:48.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.715+0000 7f65c4000640 1 -- 172.21.15.129:0/3561554908 wait complete. 2024-09-17T04:00:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.717+0000 7f65c4000640 1 Processor -- start 2024-09-17T04:00:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.717+0000 7f65c4000640 1 -- start start 2024-09-17T04:00:48.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc0cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 0x7f65bc1f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc0b2cb0 con 0x7f65bc0703b0 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f65bc0b2b30 con 0x7f65bc0b4d70 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c4000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f65bc06f420 con 0x7f65bc0b3c90 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46812/0 (socket says 172.21.15.129:46812) 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c1600640 1 -- 172.21.15.129:0/3189882241 learned_addr learned my addr 172.21.15.129:0/3189882241 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c2000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 0x7f65bc1f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.719+0000 7f65c0c00640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc0cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 -- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 msgr2=0x7f65bc0cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc0cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 -- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 msgr2=0x7f65bc1f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 0x7f65bc1f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 -- 172.21.15.129:0/3189882241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65bc15c2c0 con 0x7f65bc0703b0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c2000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 0x7f65bc1f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c1600640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c760 tx=0x7f65b000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0016020 con 0x7f65bc0703b0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f65b000a950 con 0x7f65bc0703b0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65bc0cded0 con 0x7f65bc0703b0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b000ada0 con 0x7f65bc0703b0 2024-09-17T04:00:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.720+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65bc0ce490 con 0x7f65bc0703b0 2024-09-17T04:00:48.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.722+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65bc0af670 con 0x7f65bc0703b0 2024-09-17T04:00:48.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.723+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f65b0019020 con 0x7f65bc0703b0 2024-09-17T04:00:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.724+0000 7f65b9600640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 0x7f659c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.724+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(171..171 src has 1..171) ==== 5227+0+0 (secure 0 0 0) 0x7f65b009a9e0 con 0x7f65bc0703b0 2024-09-17T04:00:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.724+0000 7f65c0c00640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 0x7f659c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:48.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.725+0000 7f65c0c00640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 0x7f659c07b0d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f65bc151d40 tx=0x7f65a8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.729+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f65bc0af670 con 0x7f65bc0703b0 2024-09-17T04:00:48.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:48.915+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f65bc06f8b0 con 0x7f65bc0703b0 2024-09-17T04:00:49.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.461+0000 7f65b9600640 1 -- 172.21.15.129:0/3189882241 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v172) ==== 170+0+0 (secure 0 0 0) 0x7f65b0020020 con 0x7f65bc0703b0 2024-09-17T04:00:49.462 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:00:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 msgr2=0x7f659c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 0x7f659c07b0d0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f65bc151d40 tx=0x7f65a8005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 msgr2=0x7f65bc15bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f65b000c760 tx=0x7f65b000cc20 comp rx=0 tx=0).stop 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 shutdown_connections 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f659c078bc0 0x7f659c07b0d0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65bc0b4d70 0x7f65bc1f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65bc0b3c90 0x7f65bc0cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 --2- 172.21.15.129:0/3189882241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f65bc0703b0 0x7f65bc15bd80 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.465+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 >> 172.21.15.129:0/3189882241 conn(0x7f65bc06d7a0 msgr2=0x7f65bc156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:49.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.466+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 shutdown_connections 2024-09-17T04:00:49.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.467+0000 7f65c4000640 1 -- 172.21.15.129:0/3189882241 wait complete. 2024-09-17T04:00:49.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:00:49.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 -- 172.21.15.129:0/2685058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 msgr2=0x7fdca00704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/2685058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca00704c0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fdc88005fd0 tx=0x7fdc8802a4b0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 -- 172.21.15.129:0/2685058130 shutdown_connections 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/2685058130 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca00704c0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/2685058130 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca015c7b0 0x7fdca00cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 -- 172.21.15.129:0/2685058130 >> 172.21.15.129:0/2685058130 conn(0x7fdca00d00b0 msgr2=0x7fdca00d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 -- 172.21.15.129:0/2685058130 shutdown_connections 2024-09-17T04:00:49.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.729+0000 7fdc9fe00640 1 -- 172.21.15.129:0/2685058130 wait complete. 2024-09-17T04:00:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.732+0000 7fdc9fe00640 1 Processor -- start 2024-09-17T04:00:49.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.732+0000 7fdc9fe00640 1 -- start start 2024-09-17T04:00:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 0x7fdca01f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdca015c7b0 0x7fdca015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdca0160970 con 0x7fdca00700b0 2024-09-17T04:00:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdca00736e0 con 0x7fdca015c7b0 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9fe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdca0160af0 con 0x7fdca0072e10 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46830/0 (socket says 172.21.15.129:46830) 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.733+0000 7fdc9ea00640 1 -- 172.21.15.129:0/1897329202 learned_addr learned my addr 172.21.15.129:0/1897329202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9f400640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdca015c7b0 0x7fdca015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc96000640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 0x7fdca01f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 -- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 msgr2=0x7fdca01f3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 0x7fdca01f3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 -- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdca015c7b0 msgr2=0x7fdca015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdca015c7b0 0x7fdca015efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 -- 172.21.15.129:0/1897329202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca00b4d50 con 0x7fdca00700b0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc96000640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 0x7fdca01f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9ea00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c9a0 tx=0x7fdc9000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc90007bb0 con 0x7fdca00700b0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc90007e40 con 0x7fdca00700b0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca00b5030 con 0x7fdca00700b0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.734+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9000f690 con 0x7fdca00700b0 2024-09-17T04:00:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.735+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca00b55f0 con 0x7fdca00700b0 2024-09-17T04:00:49.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.736+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca0160fc0 con 0x7fdca00700b0 2024-09-17T04:00:49.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.737+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdc90018400 con 0x7fdca00700b0 2024-09-17T04:00:49.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.738+0000 7fdc97400640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 0x7fdc7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:49.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.738+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 5609+0+0 (secure 0 0 0) 0x7fdc9009ba20 con 0x7fdca00700b0 2024-09-17T04:00:49.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.739+0000 7fdc96000640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 0x7fdc7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:49.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.740+0000 7fdc96000640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 0x7fdc7c07b0a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fdc88033000 tx=0x7fdc88004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:49.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.742+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdc90026020 con 0x7fdca00700b0 2024-09-17T04:00:49.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:49.960+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdca00cfc10 con 0x7fdca00700b0 2024-09-17T04:00:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.464+0000 7fdc97400640 1 -- 172.21.15.129:0/1897329202 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v150) ==== 174+0+0 (secure 0 0 0) 0x7fdc90014030 con 0x7fdca00700b0 2024-09-17T04:00:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 26 and data pool 27 2024-09-17T04:00:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 msgr2=0x7fdc7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 0x7fdc7c07b0a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fdc88033000 tx=0x7fdc88004730 comp rx=0 tx=0).stop 2024-09-17T04:00:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 msgr2=0x7fdca015d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fdc9000c9a0 tx=0x7fdc9000ce60 comp rx=0 tx=0).stop 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 shutdown_connections 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc7c078b90 0x7fdc7c07b0a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdca015c7b0 0x7fdca015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdca0072e10 0x7fdca01f3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 --2- 172.21.15.129:0/1897329202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdca00700b0 0x7fdca015d420 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 >> 172.21.15.129:0/1897329202 conn(0x7fdca00d00b0 msgr2=0x7fdca00ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 shutdown_connections 2024-09-17T04:00:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.468+0000 7fdc9fe00640 1 -- 172.21.15.129:0/1897329202 wait complete. 2024-09-17T04:00:50.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:50.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.734+0000 7f3947e00640 1 -- 172.21.15.129:0/3719350754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481528d0 msgr2=0x7f3948072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.734+0000 7f3947e00640 1 --2- 172.21.15.129:0/3719350754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481528d0 0x7f3948072f40 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f39480727e0 tx=0x7f394002aac0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 -- 172.21.15.129:0/3719350754 shutdown_connections 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 --2- 172.21.15.129:0/3719350754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481528d0 0x7f3948072f40 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 --2- 172.21.15.129:0/3719350754 >> v2:172.21.15.181:3300/0 conn(0x7f39481524c0 0x7f39480cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 -- 172.21.15.129:0/3719350754 >> 172.21.15.129:0/3719350754 conn(0x7f394806cfc0 msgr2=0x7f394806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:50.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 -- 172.21.15.129:0/3719350754 shutdown_connections 2024-09-17T04:00:50.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.735+0000 7f3947e00640 1 -- 172.21.15.129:0/3719350754 wait complete. 2024-09-17T04:00:50.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.737+0000 7f3947e00640 1 Processor -- start 2024-09-17T04:00:50.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.737+0000 7f3947e00640 1 -- start start 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 0x7f39480b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 0x7f39480b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f39480d02c0 con 0x7f39481524c0 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f39480737e0 con 0x7f39480cfb00 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f39480d0440 con 0x7f39481528d0 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46860/0 (socket says 172.21.15.129:46860) 2024-09-17T04:00:50.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 -- 172.21.15.129:0/3742687363 learned_addr learned my addr 172.21.15.129:0/3742687363 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946a00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 0x7f39480b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947400640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 0x7f39480b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 -- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 msgr2=0x7f39480b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 0x7f39480b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 -- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 msgr2=0x7f39480b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 0x7f39480b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946000640 1 -- 172.21.15.129:0/3742687363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39481f73b0 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3946a00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 0x7f39480b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.739+0000 7f3947400640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 0x7f39480b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f3946000640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f393400c3a0 tx=0x7f393400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934017070 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f39340053c0 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3948142530 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934003850 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.740+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3948142af0 con 0x7f39481524c0 2024-09-17T04:00:50.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.741+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39480d0960 con 0x7f39481524c0 2024-09-17T04:00:50.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.742+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f393401a070 con 0x7f39481524c0 2024-09-17T04:00:50.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.743+0000 7f393ea00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 0x7f392807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:50.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.743+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5709+0+0 (secure 0 0 0) 0x7f393409a1b0 con 0x7f39481524c0 2024-09-17T04:00:50.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.744+0000 7f3946a00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 0x7f392807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:50.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.744+0000 7f3946a00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 0x7f392807b0a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3938005fd0 tx=0x7f3938005d20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:50.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.748+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3934062550 con 0x7f39481524c0 2024-09-17T04:00:50.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.931+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f394806a140 con 0x7f39481524c0 2024-09-17T04:00:50.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.933+0000 7f393ea00640 1 -- 172.21.15.129:0/3742687363 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) ==== 74+0+16804 (secure 0 0 0) 0x7f3934061e50 con 0x7f39481524c0 2024-09-17T04:00:50.933 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:50.933 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":173,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:49.962350+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:47.893071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:48.928779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:50.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 msgr2=0x7f392807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 0x7f392807b0a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3938005fd0 tx=0x7f3938005d20 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 msgr2=0x7f39481f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f393400c3a0 tx=0x7f393400c860 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 shutdown_connections 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3928078b90 0x7f392807b0a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39481528d0 0x7f39480b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39481524c0 0x7f39481f69a0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 --2- 172.21.15.129:0/3742687363 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f39480cfb00 0x7f39480b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 >> 172.21.15.129:0/3742687363 conn(0x7f394806cfc0 msgr2=0x7f394815a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 shutdown_connections 2024-09-17T04:00:50.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:50.935+0000 7f3947e00640 1 -- 172.21.15.129:0/3742687363 wait complete. 2024-09-17T04:00:50.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 -- 172.21.15.129:0/290315719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c0cfb00 msgr2=0x7f934c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 --2- 172.21.15.129:0/290315719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c0cfb00 0x7f934c15bca0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f934c0727e0 tx=0x7f933402a530 comp rx=0 tx=0).stop 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 -- 172.21.15.129:0/290315719 shutdown_connections 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 --2- 172.21.15.129:0/290315719 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c0cfb00 0x7f934c15bca0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 --2- 172.21.15.129:0/290315719 >> v2:172.21.15.181:3300/0 conn(0x7f934c1524c0 0x7f934c0cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 -- 172.21.15.129:0/290315719 >> 172.21.15.129:0/290315719 conn(0x7f934c06cfc0 msgr2=0x7f934c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 -- 172.21.15.129:0/290315719 shutdown_connections 2024-09-17T04:00:51.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.176+0000 7f9351e00640 1 -- 172.21.15.129:0/290315719 wait complete. 2024-09-17T04:00:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.179+0000 7f9351e00640 1 Processor -- start 2024-09-17T04:00:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.179+0000 7f9351e00640 1 -- start start 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f934c0cfb00 0x7f934c14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f934c1524c0 0x7f934c0ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f934c0d02c0 con 0x7f934c15c1e0 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f934c152ae0 con 0x7f934c1524c0 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f934c0d0440 con 0x7f934c0cfb00 2024-09-17T04:00:51.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46872/0 (socket says 172.21.15.129:46872) 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.180+0000 7f9351400640 1 -- 172.21.15.129:0/1749776599 learned_addr learned my addr 172.21.15.129:0/1749776599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9350a00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f934c0cfb00 0x7f934c14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9343e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f934c1524c0 0x7f934c0ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 -- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f934c0cfb00 msgr2=0x7f934c14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f934c0cfb00 0x7f934c14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 -- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f934c1524c0 msgr2=0x7f934c0ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f934c1524c0 0x7f934c0ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 -- 172.21.15.129:0/1749776599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934c14c6f0 con 0x7f934c15c1e0 2024-09-17T04:00:51.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.181+0000 7f9351400640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f934400c730 tx=0x7f934400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.182+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344003b70 con 0x7f934c15c1e0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.182+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9344003d10 con 0x7f934c15c1e0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.182+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934c0aa440 con 0x7f934c15c1e0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.182+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93440052d0 con 0x7f934c15c1e0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.182+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f934c15b360 con 0x7f934c15c1e0 2024-09-17T04:00:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.183+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f934c0d0960 con 0x7f934c15c1e0 2024-09-17T04:00:51.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.184+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9344005470 con 0x7f934c15c1e0 2024-09-17T04:00:51.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.185+0000 7f9349600640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 0x7f932807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.185+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(173..173 src has 1..173) ==== 5709+0+0 (secure 0 0 0) 0x7f934409b430 con 0x7f934c15c1e0 2024-09-17T04:00:51.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.186+0000 7f9350a00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 0x7f932807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.186+0000 7f9350a00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 0x7f932807b0a0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f933c009b30 tx=0x7f933c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.188+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9344063290 con 0x7f934c15c1e0 2024-09-17T04:00:51.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.381+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f934c06a140 con 0x7f934c15c1e0 2024-09-17T04:00:51.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.382+0000 7f9349600640 1 -- 172.21.15.129:0/1749776599 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v173) ==== 74+0+16804 (secure 0 0 0) 0x7f9344067cc0 con 0x7f934c15c1e0 2024-09-17T04:00:51.383 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:51.383 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":173,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:49.962350+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:47.893071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:48.928779+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.384+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 msgr2=0x7f932807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.384+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 0x7f932807b0a0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f933c009b30 tx=0x7f933c005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.384+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 msgr2=0x7f934c0ab610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.384+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f934400c730 tx=0x7f934400cb30 comp rx=0 tx=0).stop 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 shutdown_connections 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9328078b90 0x7f932807b0a0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f934c15c1e0 0x7f934c0ab610 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f934c1524c0 0x7f934c0ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 --2- 172.21.15.129:0/1749776599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f934c0cfb00 0x7f934c14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 >> 172.21.15.129:0/1749776599 conn(0x7f934c06cfc0 msgr2=0x7f934c151b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:51.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 shutdown_connections 2024-09-17T04:00:51.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.385+0000 7f9351e00640 1 -- 172.21.15.129:0/1749776599 wait complete. 2024-09-17T04:00:51.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.635+0000 7f2da5000640 1 -- 172.21.15.129:0/2452613851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 msgr2=0x7f2d840040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.635+0000 7f2da5000640 1 --2- 172.21.15.129:0/2452613851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2d840040d0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009b50 tx=0x7f2d84005830 comp rx=0 tx=0).stop 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.635+0000 7f2d9fe00640 1 -- 172.21.15.129:0/2452613851 reap_dead start 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.636+0000 7f2da5000640 1 -- 172.21.15.129:0/2452613851 shutdown_connections 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.636+0000 7f2da5000640 1 -- 172.21.15.129:0/2452613851 >> 172.21.15.129:0/2452613851 conn(0x7f2da00d00b0 msgr2=0x7f2da00d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.636+0000 7f2da5000640 1 -- 172.21.15.129:0/2452613851 shutdown_connections 2024-09-17T04:00:51.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.636+0000 7f2da5000640 1 -- 172.21.15.129:0/2452613851 wait complete. 2024-09-17T04:00:51.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.638+0000 7f2da5000640 1 Processor -- start 2024-09-17T04:00:51.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.638+0000 7f2da5000640 1 -- start start 2024-09-17T04:00:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2da00b5700 0x7f2da014e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da00b5b10 0x7f2da014ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2da00cb410 con 0x7f2d84003c80 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2da00cb290 con 0x7f2da00b5b10 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.639+0000 7f2da5000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2da00ceb90 con 0x7f2da00b5700 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46892/0 (socket says 172.21.15.129:46892) 2024-09-17T04:00:51.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 -- 172.21.15.129:0/3947161382 learned_addr learned my addr 172.21.15.129:0/3947161382 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9fe00640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da00b5b10 0x7f2da014ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d96a00640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2da00b5700 0x7f2da014e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 -- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2da00b5700 msgr2=0x7f2da014e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2da00b5700 0x7f2da014e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 -- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da00b5b10 msgr2=0x7f2da014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da00b5b10 0x7f2da014ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 -- 172.21.15.129:0/3947161382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da014c5d0 con 0x7f2d84003c80 2024-09-17T04:00:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.640+0000 7f2d9f400640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f2d90036040 tx=0x7f2d90033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.641+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90038040 con 0x7f2d84003c80 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.641+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d90004020 con 0x7f2d84003c80 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.641+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90004470 con 0x7f2d84003c80 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.641+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da014c8b0 con 0x7f2d84003c80 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.641+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da014ce70 con 0x7f2d84003c80 2024-09-17T04:00:51.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.642+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da00cf0f0 con 0x7f2d84003c80 2024-09-17T04:00:51.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.643+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2d90045020 con 0x7f2d84003c80 2024-09-17T04:00:51.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.644+0000 7f2d97e00640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 0x7f2d7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:51.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.644+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f2d900bb7e0 con 0x7f2d84003c80 2024-09-17T04:00:51.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.645+0000 7f2d96a00640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 0x7f2d7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:51.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.645+0000 7f2d96a00640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 0x7f2d7c07b0a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2da01f6f70 tx=0x7f2d88006c40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:51.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.648+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2d90083530 con 0x7f2d84003c80 2024-09-17T04:00:51.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:51.870+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2da00d0f50 con 0x7f2d84003c80 2024-09-17T04:00:52.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.493+0000 7f2d97e00640 1 -- 172.21.15.129:0/3947161382 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v153) ==== 118+0+0 (secure 0 0 0) 0x7f2d90087f60 con 0x7f2d84003c80 2024-09-17T04:00:52.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 msgr2=0x7f2d7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:52.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 0x7f2d7c07b0a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f2da01f6f70 tx=0x7f2d88006c40 comp rx=0 tx=0).stop 2024-09-17T04:00:52.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 msgr2=0x7f2da009d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:52.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f2d90036040 tx=0x7f2d90033670 comp rx=0 tx=0).stop 2024-09-17T04:00:52.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 shutdown_connections 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d7c078b90 0x7f2d7c07b0a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2da00b5b10 0x7f2da014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2da00b5700 0x7f2da014e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 --2- 172.21.15.129:0/3947161382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d84003c80 0x7f2da009d880 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 >> 172.21.15.129:0/3947161382 conn(0x7f2da00d00b0 msgr2=0x7f2da00afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 shutdown_connections 2024-09-17T04:00:52.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.496+0000 7f2da5000640 1 -- 172.21.15.129:0/3947161382 wait complete. 2024-09-17T04:00:52.522 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:00:52.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 -- 172.21.15.129:0/1071297683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180cfe10 msgr2=0x7f91180d0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 --2- 172.21.15.129:0/1071297683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180cfe10 0x7f91180d0220 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9100009830 tx=0x7f910002a500 comp rx=0 tx=0).stop 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 -- 172.21.15.129:0/1071297683 shutdown_connections 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 --2- 172.21.15.129:0/1071297683 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180d0c30 0x7f91180d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 --2- 172.21.15.129:0/1071297683 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180cfe10 0x7f91180d0220 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 -- 172.21.15.129:0/1071297683 >> 172.21.15.129:0/1071297683 conn(0x7f911808bff0 msgr2=0x7f91180d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:52.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 -- 172.21.15.129:0/1071297683 shutdown_connections 2024-09-17T04:00:52.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.750+0000 7f911ca00640 1 -- 172.21.15.129:0/1071297683 wait complete. 2024-09-17T04:00:52.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.752+0000 7f911ca00640 1 Processor -- start 2024-09-17T04:00:52.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.752+0000 7f911ca00640 1 -- start start 2024-09-17T04:00:52.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180cfe10 0x7f91181f89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:52.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91180d2da0 0x7f911815c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f91180d1700 con 0x7f91180d0c30 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f91180d1580 con 0x7f91180d2da0 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f911ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f91180d1880 con 0x7f91180cfe10 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:52.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46918/0 (socket says 172.21.15.129:46918) 2024-09-17T04:00:52.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 -- 172.21.15.129:0/803060862 learned_addr learned my addr 172.21.15.129:0/803060862 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:52.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f9117e00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91180d2da0 0x7f911815c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:52.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f9117400640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180cfe10 0x7f91181f89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:52.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 -- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180cfe10 msgr2=0x7f91181f89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180cfe10 0x7f91181f89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 -- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91180d2da0 msgr2=0x7f911815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91180d2da0 0x7f911815c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.754+0000 7f910ea00640 1 -- 172.21.15.129:0/803060862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911815b000 con 0x7f91180d0c30 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f910ea00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f910002aaf0 tx=0x7f9100033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:52.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100031d60 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9100033e10 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91181f8f00 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100005260 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.755+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911815b270 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.756+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91180d0220 con 0x7f91180d0c30 2024-09-17T04:00:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.757+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9100005400 con 0x7f91180d0c30 2024-09-17T04:00:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.758+0000 7f910fe00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 0x7f90f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:52.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.758+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f91000bbfb0 con 0x7f91180d0c30 2024-09-17T04:00:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.759+0000 7f9117400640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 0x7f90f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.760+0000 7f9117400640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 0x7f90f407b0a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f91080015b0 tx=0x7f9108009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:52.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.763+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f910003f030 con 0x7f91180d0c30 2024-09-17T04:00:52.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:52.979+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f911806a130 con 0x7f91180d0c30 2024-09-17T04:00:53.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.541+0000 7f910fe00640 1 -- 172.21.15.129:0/803060862 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v154) ==== 122+0+0 (secure 0 0 0) 0x7f9100005d00 con 0x7f91180d0c30 2024-09-17T04:00:53.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 msgr2=0x7f90f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:53.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 0x7f90f407b0a0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f91080015b0 tx=0x7f9108009040 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 msgr2=0x7f911815c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f910002aaf0 tx=0x7f9100033c20 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 shutdown_connections 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f90f4078b90 0x7f90f407b0a0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f91180d2da0 0x7f911815c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f91180d0c30 0x7f911815c030 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 --2- 172.21.15.129:0/803060862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f91180cfe10 0x7f91181f89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 >> 172.21.15.129:0/803060862 conn(0x7f911808bff0 msgr2=0x7f9118160d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 shutdown_connections 2024-09-17T04:00:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.545+0000 7f911ca00640 1 -- 172.21.15.129:0/803060862 wait complete. 2024-09-17T04:00:53.569 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:00:53.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 -- 172.21.15.129:0/933882897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 msgr2=0x7f0f70004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 --2- 172.21.15.129:0/933882897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f70004c10 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009b50 tx=0x7f0f70005830 comp rx=0 tx=0).stop 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 -- 172.21.15.129:0/933882897 shutdown_connections 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 --2- 172.21.15.129:0/933882897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f70004c10 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 -- 172.21.15.129:0/933882897 >> 172.21.15.129:0/933882897 conn(0x7f0f8806d410 msgr2=0x7f0f880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 -- 172.21.15.129:0/933882897 shutdown_connections 2024-09-17T04:00:53.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.792+0000 7f0f90000640 1 -- 172.21.15.129:0/933882897 wait complete. 2024-09-17T04:00:53.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.794+0000 7f0f90000640 1 Processor -- start 2024-09-17T04:00:53.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.795+0000 7f0f90000640 1 -- start start 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 0x7f0f880ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f881617d0 0x7f0f88161be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8815ed90 con 0x7f0f70004800 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8815ec10 con 0x7f0f8815ef10 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f90000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0f880cf390 con 0x7f0f881617d0 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f8d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f8d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35106/0 (socket says 172.21.15.129:35106) 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.796+0000 7f0f8d600640 1 -- 172.21.15.129:0/2266101915 learned_addr learned my addr 172.21.15.129:0/2266101915 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:53.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8cc00640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 0x7f0f880ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8e000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f881617d0 0x7f0f88161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 -- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f881617d0 msgr2=0x7f0f88161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f881617d0 0x7f0f88161be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 -- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 msgr2=0x7f0f880ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 0x7f0f880ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 -- 172.21.15.129:0/2266101915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8815b670 con 0x7f0f70004800 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8cc00640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 0x7f0f880ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:53.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f8d600640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c006600 tx=0x7f0f7c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c015070 con 0x7f0f70004800 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0f7c011900 con 0x7f0f70004800 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7c011d50 con 0x7f0f70004800 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f880aef20 con 0x7f0f70004800 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.797+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f88160fd0 con 0x7f0f70004800 2024-09-17T04:00:53.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.799+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f88162d00 con 0x7f0f70004800 2024-09-17T04:00:53.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.800+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0f7c018070 con 0x7f0f70004800 2024-09-17T04:00:53.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.801+0000 7f0f85600640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 0x7f0f6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:53.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.801+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f0f7c09ba40 con 0x7f0f70004800 2024-09-17T04:00:53.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.801+0000 7f0f8cc00640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 0x7f0f6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:53.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.802+0000 7f0f8cc00640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 0x7f0f6807b0a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0f74035000 tx=0x7f0f740046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:53.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:53.805+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0f880cfce0 con 0x7f0f70004800 2024-09-17T04:00:54.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.025+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f8806a140 con 0x7f0f70004800 2024-09-17T04:00:54.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.026+0000 7f0f85600640 1 -- 172.21.15.129:0/2266101915 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f0f7c063df0 con 0x7f0f70004800 2024-09-17T04:00:54.026 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:54.026 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:00:54.027 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.028+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 msgr2=0x7f0f6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.028+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 0x7f0f6807b0a0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f0f74035000 tx=0x7f0f740046c0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 msgr2=0x7f0f8814c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f0f7c006600 tx=0x7f0f7c0039c0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 shutdown_connections 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0f68078b90 0x7f0f6807b0a0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0f881617d0 0x7f0f88161be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f8815ef10 0x7f0f880ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 --2- 172.21.15.129:0/2266101915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0f70004800 0x7f0f8814c2a0 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 >> 172.21.15.129:0/2266101915 conn(0x7f0f8806d410 msgr2=0x7f0f8806eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:54.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 shutdown_connections 2024-09-17T04:00:54.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.029+0000 7f0f90000640 1 -- 172.21.15.129:0/2266101915 wait complete. 2024-09-17T04:00:54.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:00:54.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 -- 172.21.15.129:0/4236848248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0b9af0 msgr2=0x7f0d9c0d3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 --2- 172.21.15.129:0/4236848248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0b9af0 0x7f0d9c0d3070 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0b42d0 tx=0x7f0d9402a530 comp rx=0 tx=0).stop 2024-09-17T04:00:54.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 -- 172.21.15.129:0/4236848248 shutdown_connections 2024-09-17T04:00:54.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 --2- 172.21.15.129:0/4236848248 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0b9af0 0x7f0d9c0d3070 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 --2- 172.21.15.129:0/4236848248 >> v2:172.21.15.181:3300/0 conn(0x7f0d9c0d4ba0 0x7f0d9c0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 -- 172.21.15.129:0/4236848248 >> 172.21.15.129:0/4236848248 conn(0x7f0d9c08bff0 msgr2=0x7f0d9c09d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:54.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 -- 172.21.15.129:0/4236848248 shutdown_connections 2024-09-17T04:00:54.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.272+0000 7f0d9b200640 1 -- 172.21.15.129:0/4236848248 wait complete. 2024-09-17T04:00:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.274+0000 7f0d9b200640 1 Processor -- start 2024-09-17T04:00:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.274+0000 7f0d9b200640 1 -- start start 2024-09-17T04:00:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.275+0000 7f0d9b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 0x7f0d9c15efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.275+0000 7f0d9b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d9c0d1850 0x7f0d9c0c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c0cf440 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9c0cf2c0 con 0x7f0d9c0d1850 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9c0cf5c0 con 0x7f0d9c0b9af0 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35126/0 (socket says 172.21.15.129:35126) 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 -- 172.21.15.129:0/975469062 learned_addr learned my addr 172.21.15.129:0/975469062 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d99400640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d9c0d1850 0x7f0d9c0c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d99e00640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 0x7f0d9c15efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 -- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 msgr2=0x7f0d9c15efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 0x7f0d9c15efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 -- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d9c0d1850 msgr2=0x7f0d9c0c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d9c0d1850 0x7f0d9c0c7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d9a800640 1 -- 172.21.15.129:0/975469062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c0caf90 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.276+0000 7f0d99e00640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 0x7f0d9c15efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:54.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d9a800640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f0d940098f0 tx=0x7f0d94033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:54.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94031d60 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d94033e10 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9c15f4f0 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d94005260 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.277+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c1f8f60 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.278+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d9c0caa20 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.279+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d94005400 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.280+0000 7f0d92000640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 0x7f0d7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.281+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f0d940bbe40 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.281+0000 7f0d99e00640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 0x7f0d7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.282+0000 7f0d99e00640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 0x7f0d7c07b0a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c007fd0 tx=0x7f0d8c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:54.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.285+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d9403f030 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.470+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d9c0b4300 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.471+0000 7f0d92000640 1 -- 172.21.15.129:0/975469062 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) ==== 74+0+16791 (secure 0 0 0) 0x7f0d94005d00 con 0x7f0d9c0d4ba0 2024-09-17T04:00:54.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:54.472 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":174,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:51.478826+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:47.893071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:48.928779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:00:54.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.473+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 msgr2=0x7f0d7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.473+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 0x7f0d7c07b0a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c007fd0 tx=0x7f0d8c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:54.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.473+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 msgr2=0x7f0d9c0c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.473+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f0d940098f0 tx=0x7f0d94033c20 comp rx=0 tx=0).stop 2024-09-17T04:00:54.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 shutdown_connections 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d7c078b90 0x7f0d7c07b0a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d9c0d4ba0 0x7f0d9c0c88d0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d9c0d1850 0x7f0d9c0c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 --2- 172.21.15.129:0/975469062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d9c0b9af0 0x7f0d9c15efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 >> 172.21.15.129:0/975469062 conn(0x7f0d9c08bff0 msgr2=0x7f0d9c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 shutdown_connections 2024-09-17T04:00:54.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.474+0000 7f0d9b200640 1 -- 172.21.15.129:0/975469062 wait complete. 2024-09-17T04:00:54.495 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:00:54.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:54.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 -- 172.21.15.129:0/4035673522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0b39c0 msgr2=0x7fbc7c06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 --2- 172.21.15.129:0/4035673522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c06eed0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c06dcc0 tx=0x7fbc7402aa30 comp rx=0 tx=0).stop 2024-09-17T04:00:54.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 -- 172.21.15.129:0/4035673522 shutdown_connections 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 --2- 172.21.15.129:0/4035673522 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c06eed0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 --2- 172.21.15.129:0/4035673522 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 0x7fbc7c0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 --2- 172.21.15.129:0/4035673522 >> v2:172.21.15.181:3300/0 conn(0x7fbc7c0ce170 0x7fbc7c0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 -- 172.21.15.129:0/4035673522 >> 172.21.15.129:0/4035673522 conn(0x7fbc7c06d410 msgr2=0x7fbc7c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 -- 172.21.15.129:0/4035673522 shutdown_connections 2024-09-17T04:00:54.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.717+0000 7fbc83e00640 1 -- 172.21.15.129:0/4035673522 wait complete. 2024-09-17T04:00:54.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.720+0000 7fbc83e00640 1 Processor -- start 2024-09-17T04:00:54.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.720+0000 7fbc83e00640 1 -- start start 2024-09-17T04:00:54.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 0x7fbc7c1f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c070850 con 0x7fbc7c0ce170 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c0706d0 con 0x7fbc7c0b39c0 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc83e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c06f410 con 0x7fbc7c0b2950 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.721+0000 7fbc81e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35144/0 (socket says 172.21.15.129:35144) 2024-09-17T04:00:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 -- 172.21.15.129:0/3897849075 learned_addr learned my addr 172.21.15.129:0/3897849075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:54.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81400640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 0x7fbc7c1f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc80a00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 -- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 msgr2=0x7fbc7c1f6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 0x7fbc7c1f6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 -- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc7c0b39c0 msgr2=0x7fbc7c15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c15ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 -- 172.21.15.129:0/3897849075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c15eab0 con 0x7fbc7c0ce170 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.722+0000 7fbc81e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fbc7402b020 tx=0x7fbc74003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.723+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74033050 con 0x7fbc7c0ce170 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.723+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc74004450 con 0x7fbc7c0ce170 2024-09-17T04:00:54.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.723+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74038450 con 0x7fbc7c0ce170 2024-09-17T04:00:54.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.723+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c15c070 con 0x7fbc7c0ce170 2024-09-17T04:00:54.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.723+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c15c630 con 0x7fbc7c0ce170 2024-09-17T04:00:54.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.724+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c06f8f0 con 0x7fbc7c0ce170 2024-09-17T04:00:54.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.725+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbc74036070 con 0x7fbc7c0ce170 2024-09-17T04:00:54.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.726+0000 7fbc79600640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 0x7fbc5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:54.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.727+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fbc740bbfe0 con 0x7fbc7c0ce170 2024-09-17T04:00:54.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.727+0000 7fbc81400640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 0x7fbc5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:54.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.728+0000 7fbc81400640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 0x7fbc5c07b0a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009b30 tx=0x7fbc70005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:54.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.731+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbc7c0b3dd0 con 0x7fbc7c0ce170 2024-09-17T04:00:54.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.912+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc7c15e800 con 0x7fbc5c078b90 2024-09-17T04:00:54.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.918+0000 7fbc79600640 1 -- 172.21.15.129:0/3897849075 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250551 (secure 0 0 0) 0x7fbc7c15e800 con 0x7fbc5c078b90 2024-09-17T04:00:54.921 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:54.922 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 msgr2=0x7fbc5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 0x7fbc5c07b0a0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009b30 tx=0x7fbc70005f90 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 msgr2=0x7fbc7c147f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fbc7402b020 tx=0x7fbc74003f10 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 shutdown_connections 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc5c078b90 0x7fbc5c07b0a0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc7c0ce170 0x7fbc7c147f90 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc7c0b39c0 0x7fbc7c15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 --2- 172.21.15.129:0/3897849075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc7c0b2950 0x7fbc7c1f6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 >> 172.21.15.129:0/3897849075 conn(0x7fbc7c06d410 msgr2=0x7fbc7c0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 shutdown_connections 2024-09-17T04:00:54.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:54.922+0000 7fbc83e00640 1 -- 172.21.15.129:0/3897849075 wait complete. 2024-09-17T04:00:54.945 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-17T04:00:54.487043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592596,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":121,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":325948,"kb_used_data":18184,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754648772,"statfs":{"total":773094113280,"available":772760342528,"internally_reserved":0,"allocated":18620416,"data_stored":8788852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002410"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490698+0000","last_change":"2024-09-17T04:00:50.499023+0000","last_active":"2024-09-17T04:00:51.490698+0000","last_peered":"2024-09-17T04:00:51.490698+0000","last_clean":"2024-09-17T04:00:51.490698+0000","last_became_active":"2024-09-17T04:00:50.498702+0000","last_became_peered":"2024-09-17T04:00:50.498702+0000","last_unstale":"2024-09-17T04:00:51.490698+0000","last_undegraded":"2024-09-17T04:00:51.490698+0000","last_fullsized":"2024-09-17T04:00:51.490698+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:47.660122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493082+0000","last_change":"2024-09-17T04:00:50.481292+0000","last_active":"2024-09-17T04:00:51.493082+0000","last_peered":"2024-09-17T04:00:51.493082+0000","last_clean":"2024-09-17T04:00:51.493082+0000","last_became_active":"2024-09-17T04:00:50.480948+0000","last_became_peered":"2024-09-17T04:00:50.480948+0000","last_unstale":"2024-09-17T04:00:51.493082+0000","last_undegraded":"2024-09-17T04:00:51.493082+0000","last_fullsized":"2024-09-17T04:00:51.493082+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.354995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470097+0000","last_change":"2024-09-17T04:00:49.470097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470097+0000","last_undegraded":"2024-09-17T04:00:49.470097+0000","last_fullsized":"2024-09-17T04:00:49.470097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475438+0000","last_change":"2024-09-17T04:00:50.500751+0000","last_active":"2024-09-17T04:00:52.475438+0000","last_peered":"2024-09-17T04:00:52.475438+0000","last_clean":"2024-09-17T04:00:52.475438+0000","last_became_active":"2024-09-17T04:00:50.500022+0000","last_became_peered":"2024-09-17T04:00:50.500022+0000","last_unstale":"2024-09-17T04:00:52.475438+0000","last_undegraded":"2024-09-17T04:00:52.475438+0000","last_fullsized":"2024-09-17T04:00:52.475438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:04.192066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472481+0000","last_change":"2024-09-17T04:00:49.472481+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472481+0000","last_undegraded":"2024-09-17T04:00:49.472481+0000","last_fullsized":"2024-09-17T04:00:49.472481+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474000+0000","last_change":"2024-09-17T04:00:49.474000+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474000+0000","last_undegraded":"2024-09-17T04:00:49.474000+0000","last_fullsized":"2024-09-17T04:00:49.474000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490063+0000","last_change":"2024-09-17T04:00:50.488121+0000","last_active":"2024-09-17T04:00:51.490063+0000","last_peered":"2024-09-17T04:00:51.490063+0000","last_clean":"2024-09-17T04:00:51.490063+0000","last_became_active":"2024-09-17T04:00:50.487823+0000","last_became_peered":"2024-09-17T04:00:50.487823+0000","last_unstale":"2024-09-17T04:00:51.490063+0000","last_undegraded":"2024-09-17T04:00:51.490063+0000","last_fullsized":"2024-09-17T04:00:51.490063+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:27.505920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472807+0000","last_change":"2024-09-17T04:00:49.472807+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472807+0000","last_undegraded":"2024-09-17T04:00:49.472807+0000","last_fullsized":"2024-09-17T04:00:49.472807+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493544+0000","last_change":"2024-09-17T04:00:50.494343+0000","last_active":"2024-09-17T04:00:51.493544+0000","last_peered":"2024-09-17T04:00:51.493544+0000","last_clean":"2024-09-17T04:00:51.493544+0000","last_became_active":"2024-09-17T04:00:50.493022+0000","last_became_peered":"2024-09-17T04:00:50.493022+0000","last_unstale":"2024-09-17T04:00:51.493544+0000","last_undegraded":"2024-09-17T04:00:51.493544+0000","last_fullsized":"2024-09-17T04:00:51.493544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:20.950951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474579+0000","last_change":"2024-09-17T04:00:49.474579+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474579+0000","last_undegraded":"2024-09-17T04:00:49.474579+0000","last_fullsized":"2024-09-17T04:00:49.474579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468304+0000","last_change":"2024-09-17T04:00:49.468304+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468304+0000","last_undegraded":"2024-09-17T04:00:49.468304+0000","last_fullsized":"2024-09-17T04:00:49.468304+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954479+0000","last_change":"2024-09-17T04:00:50.503620+0000","last_active":"2024-09-17T04:00:51.954479+0000","last_peered":"2024-09-17T04:00:51.954479+0000","last_clean":"2024-09-17T04:00:51.954479+0000","last_became_active":"2024-09-17T04:00:50.503238+0000","last_became_peered":"2024-09-17T04:00:50.503238+0000","last_unstale":"2024-09-17T04:00:51.954479+0000","last_undegraded":"2024-09-17T04:00:51.954479+0000","last_fullsized":"2024-09-17T04:00:51.954479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:10.261218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954637+0000","last_change":"2024-09-17T04:00:50.502335+0000","last_active":"2024-09-17T04:00:51.954637+0000","last_peered":"2024-09-17T04:00:51.954637+0000","last_clean":"2024-09-17T04:00:51.954637+0000","last_became_active":"2024-09-17T04:00:50.502036+0000","last_became_peered":"2024-09-17T04:00:50.502036+0000","last_unstale":"2024-09-17T04:00:51.954637+0000","last_undegraded":"2024-09-17T04:00:51.954637+0000","last_fullsized":"2024-09-17T04:00:51.954637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:37.502247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490230+0000","last_change":"2024-09-17T04:00:50.501289+0000","last_active":"2024-09-17T04:00:51.490230+0000","last_peered":"2024-09-17T04:00:51.490230+0000","last_clean":"2024-09-17T04:00:51.490230+0000","last_became_active":"2024-09-17T04:00:50.501012+0000","last_became_peered":"2024-09-17T04:00:50.501012+0000","last_unstale":"2024-09-17T04:00:51.490230+0000","last_undegraded":"2024-09-17T04:00:51.490230+0000","last_fullsized":"2024-09-17T04:00:51.490230+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:36.532175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490074+0000","last_change":"2024-09-17T04:00:50.490115+0000","last_active":"2024-09-17T04:00:51.490074+0000","last_peered":"2024-09-17T04:00:51.490074+0000","last_clean":"2024-09-17T04:00:51.490074+0000","last_became_active":"2024-09-17T04:00:50.489826+0000","last_became_peered":"2024-09-17T04:00:50.489826+0000","last_unstale":"2024-09-17T04:00:51.490074+0000","last_undegraded":"2024-09-17T04:00:51.490074+0000","last_fullsized":"2024-09-17T04:00:51.490074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:11.406297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.475124+0000","last_change":"2024-09-17T04:00:49.475124+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.475124+0000","last_undegraded":"2024-09-17T04:00:49.475124+0000","last_fullsized":"2024-09-17T04:00:49.475124+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490971+0000","last_change":"2024-09-17T04:00:50.497620+0000","last_active":"2024-09-17T04:00:51.490971+0000","last_peered":"2024-09-17T04:00:51.490971+0000","last_clean":"2024-09-17T04:00:51.490971+0000","last_became_active":"2024-09-17T04:00:50.495270+0000","last_became_peered":"2024-09-17T04:00:50.495270+0000","last_unstale":"2024-09-17T04:00:51.490971+0000","last_undegraded":"2024-09-17T04:00:51.490971+0000","last_fullsized":"2024-09-17T04:00:51.490971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:29.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474097+0000","last_change":"2024-09-17T04:00:49.474097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474097+0000","last_undegraded":"2024-09-17T04:00:49.474097+0000","last_fullsized":"2024-09-17T04:00:49.474097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476999+0000","last_change":"2024-09-17T04:00:49.476999+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476999+0000","last_undegraded":"2024-09-17T04:00:49.476999+0000","last_fullsized":"2024-09-17T04:00:49.476999+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473627+0000","last_change":"2024-09-17T04:00:49.473627+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473627+0000","last_undegraded":"2024-09-17T04:00:49.473627+0000","last_fullsized":"2024-09-17T04:00:49.473627+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475406+0000","last_change":"2024-09-17T04:00:50.499009+0000","last_active":"2024-09-17T04:00:52.475406+0000","last_peered":"2024-09-17T04:00:52.475406+0000","last_clean":"2024-09-17T04:00:52.475406+0000","last_became_active":"2024-09-17T04:00:50.495348+0000","last_became_peered":"2024-09-17T04:00:50.495348+0000","last_unstale":"2024-09-17T04:00:52.475406+0000","last_undegraded":"2024-09-17T04:00:52.475406+0000","last_fullsized":"2024-09-17T04:00:52.475406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:38.854990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558507+0000","last_change":"2024-09-17T04:00:50.483447+0000","last_active":"2024-09-17T04:00:52.558507+0000","last_peered":"2024-09-17T04:00:52.558507+0000","last_clean":"2024-09-17T04:00:52.558507+0000","last_became_active":"2024-09-17T04:00:50.483105+0000","last_became_peered":"2024-09-17T04:00:50.483105+0000","last_unstale":"2024-09-17T04:00:52.558507+0000","last_undegraded":"2024-09-17T04:00:52.558507+0000","last_fullsized":"2024-09-17T04:00:52.558507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:22.999066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558726+0000","last_change":"2024-09-17T04:00:50.488020+0000","last_active":"2024-09-17T04:00:52.558726+0000","last_peered":"2024-09-17T04:00:52.558726+0000","last_clean":"2024-09-17T04:00:52.558726+0000","last_became_active":"2024-09-17T04:00:50.487293+0000","last_became_peered":"2024-09-17T04:00:50.487293+0000","last_unstale":"2024-09-17T04:00:52.558726+0000","last_undegraded":"2024-09-17T04:00:52.558726+0000","last_fullsized":"2024-09-17T04:00:52.558726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:15.011545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475667+0000","last_change":"2024-09-17T04:00:50.493247+0000","last_active":"2024-09-17T04:00:52.475667+0000","last_peered":"2024-09-17T04:00:52.475667+0000","last_clean":"2024-09-17T04:00:52.475667+0000","last_became_active":"2024-09-17T04:00:50.492907+0000","last_became_peered":"2024-09-17T04:00:50.492907+0000","last_unstale":"2024-09-17T04:00:52.475667+0000","last_undegraded":"2024-09-17T04:00:52.475667+0000","last_fullsized":"2024-09-17T04:00:52.475667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:58.410921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493295+0000","last_change":"2024-09-17T04:00:50.487190+0000","last_active":"2024-09-17T04:00:51.493295+0000","last_peered":"2024-09-17T04:00:51.493295+0000","last_clean":"2024-09-17T04:00:51.493295+0000","last_became_active":"2024-09-17T04:00:50.484107+0000","last_became_peered":"2024-09-17T04:00:50.484107+0000","last_unstale":"2024-09-17T04:00:51.493295+0000","last_undegraded":"2024-09-17T04:00:51.493295+0000","last_fullsized":"2024-09-17T04:00:51.493295+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:17.052980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490680+0000","last_change":"2024-09-17T04:00:50.500005+0000","last_active":"2024-09-17T04:00:51.490680+0000","last_peered":"2024-09-17T04:00:51.490680+0000","last_clean":"2024-09-17T04:00:51.490680+0000","last_became_active":"2024-09-17T04:00:50.497315+0000","last_became_peered":"2024-09-17T04:00:50.497315+0000","last_unstale":"2024-09-17T04:00:51.490680+0000","last_undegraded":"2024-09-17T04:00:51.490680+0000","last_fullsized":"2024-09-17T04:00:51.490680+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:50.074066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954424+0000","last_change":"2024-09-17T04:00:50.507420+0000","last_active":"2024-09-17T04:00:51.954424+0000","last_peered":"2024-09-17T04:00:51.954424+0000","last_clean":"2024-09-17T04:00:51.954424+0000","last_became_active":"2024-09-17T04:00:50.506925+0000","last_became_peered":"2024-09-17T04:00:50.506925+0000","last_unstale":"2024-09-17T04:00:51.954424+0000","last_undegraded":"2024-09-17T04:00:51.954424+0000","last_fullsized":"2024-09-17T04:00:51.954424+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.177663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470536+0000","last_change":"2024-09-17T04:00:49.470536+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470536+0000","last_undegraded":"2024-09-17T04:00:49.470536+0000","last_fullsized":"2024-09-17T04:00:49.470536+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954415+0000","last_change":"2024-09-17T04:00:50.507336+0000","last_active":"2024-09-17T04:00:51.954415+0000","last_peered":"2024-09-17T04:00:51.954415+0000","last_clean":"2024-09-17T04:00:51.954415+0000","last_became_active":"2024-09-17T04:00:50.507000+0000","last_became_peered":"2024-09-17T04:00:50.507000+0000","last_unstale":"2024-09-17T04:00:51.954415+0000","last_undegraded":"2024-09-17T04:00:51.954415+0000","last_fullsized":"2024-09-17T04:00:51.954415+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:03.261144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492684+0000","last_change":"2024-09-17T04:00:50.508385+0000","last_active":"2024-09-17T04:00:51.492684+0000","last_peered":"2024-09-17T04:00:51.492684+0000","last_clean":"2024-09-17T04:00:51.492684+0000","last_became_active":"2024-09-17T04:00:50.507803+0000","last_became_peered":"2024-09-17T04:00:50.507803+0000","last_unstale":"2024-09-17T04:00:51.492684+0000","last_undegraded":"2024-09-17T04:00:51.492684+0000","last_fullsized":"2024-09-17T04:00:51.492684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:40.047416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473517+0000","last_change":"2024-09-17T04:00:49.473517+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473517+0000","last_undegraded":"2024-09-17T04:00:49.473517+0000","last_fullsized":"2024-09-17T04:00:49.473517+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490344+0000","last_change":"2024-09-17T04:00:50.504166+0000","last_active":"2024-09-17T04:00:51.490344+0000","last_peered":"2024-09-17T04:00:51.490344+0000","last_clean":"2024-09-17T04:00:51.490344+0000","last_became_active":"2024-09-17T04:00:50.503869+0000","last_became_peered":"2024-09-17T04:00:50.503869+0000","last_unstale":"2024-09-17T04:00:51.490344+0000","last_undegraded":"2024-09-17T04:00:51.490344+0000","last_fullsized":"2024-09-17T04:00:51.490344+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:44.335387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493116+0000","last_change":"2024-09-17T04:00:50.497167+0000","last_active":"2024-09-17T04:00:51.493116+0000","last_peered":"2024-09-17T04:00:51.493116+0000","last_clean":"2024-09-17T04:00:51.493116+0000","last_became_active":"2024-09-17T04:00:50.496798+0000","last_became_peered":"2024-09-17T04:00:50.496798+0000","last_unstale":"2024-09-17T04:00:51.493116+0000","last_undegraded":"2024-09-17T04:00:51.493116+0000","last_fullsized":"2024-09-17T04:00:51.493116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:18.337224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"173'4","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954531+0000","last_change":"2024-09-17T04:00:49.488273+0000","last_active":"2024-09-17T04:00:51.954531+0000","last_peered":"2024-09-17T04:00:51.954531+0000","last_clean":"2024-09-17T04:00:51.954531+0000","last_became_active":"2024-09-17T04:00:49.486926+0000","last_became_peered":"2024-09-17T04:00:49.486926+0000","last_unstale":"2024-09-17T04:00:51.954531+0000","last_undegraded":"2024-09-17T04:00:51.954531+0000","last_fullsized":"2024-09-17T04:00:51.954531+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:23.992360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558954+0000","last_change":"2024-09-17T04:00:50.493935+0000","last_active":"2024-09-17T04:00:52.558954+0000","last_peered":"2024-09-17T04:00:52.558954+0000","last_clean":"2024-09-17T04:00:52.558954+0000","last_became_active":"2024-09-17T04:00:50.490351+0000","last_became_peered":"2024-09-17T04:00:50.490351+0000","last_unstale":"2024-09-17T04:00:52.558954+0000","last_undegraded":"2024-09-17T04:00:52.558954+0000","last_fullsized":"2024-09-17T04:00:52.558954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:16.034449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493225+0000","last_change":"2024-09-17T04:00:49.485660+0000","last_active":"2024-09-17T04:00:51.493225+0000","last_peered":"2024-09-17T04:00:51.493225+0000","last_clean":"2024-09-17T04:00:51.493225+0000","last_became_active":"2024-09-17T04:00:49.484522+0000","last_became_peered":"2024-09-17T04:00:49.484522+0000","last_unstale":"2024-09-17T04:00:51.493225+0000","last_undegraded":"2024-09-17T04:00:51.493225+0000","last_fullsized":"2024-09-17T04:00:51.493225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:17.632102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490880+0000","last_change":"2024-09-17T04:00:50.493759+0000","last_active":"2024-09-17T04:00:51.490880+0000","last_peered":"2024-09-17T04:00:51.490880+0000","last_clean":"2024-09-17T04:00:51.490880+0000","last_became_active":"2024-09-17T04:00:50.493472+0000","last_became_peered":"2024-09-17T04:00:50.493472+0000","last_unstale":"2024-09-17T04:00:51.490880+0000","last_undegraded":"2024-09-17T04:00:51.490880+0000","last_fullsized":"2024-09-17T04:00:51.490880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:36.564425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490615+0000","last_change":"2024-09-17T04:00:49.485026+0000","last_active":"2024-09-17T04:00:49.490615+0000","last_peered":"2024-09-17T04:00:49.490615+0000","last_clean":"2024-09-17T04:00:49.490615+0000","last_became_active":"2024-09-17T04:00:49.484730+0000","last_became_peered":"2024-09-17T04:00:49.484730+0000","last_unstale":"2024-09-17T04:00:49.490615+0000","last_undegraded":"2024-09-17T04:00:49.490615+0000","last_fullsized":"2024-09-17T04:00:49.490615+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:40.328332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476556+0000","last_change":"2024-09-17T04:00:49.476556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476556+0000","last_undegraded":"2024-09-17T04:00:49.476556+0000","last_fullsized":"2024-09-17T04:00:49.476556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475655+0000","last_change":"2024-09-17T04:00:49.492918+0000","last_active":"2024-09-17T04:00:52.475655+0000","last_peered":"2024-09-17T04:00:52.475655+0000","last_clean":"2024-09-17T04:00:52.475655+0000","last_became_active":"2024-09-17T04:00:49.492545+0000","last_became_peered":"2024-09-17T04:00:49.492545+0000","last_unstale":"2024-09-17T04:00:52.475655+0000","last_undegraded":"2024-09-17T04:00:52.475655+0000","last_fullsized":"2024-09-17T04:00:52.475655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:49.593934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475383+0000","last_change":"2024-09-17T04:00:50.498633+0000","last_active":"2024-09-17T04:00:52.475383+0000","last_peered":"2024-09-17T04:00:52.475383+0000","last_clean":"2024-09-17T04:00:52.475383+0000","last_became_active":"2024-09-17T04:00:50.498319+0000","last_became_peered":"2024-09-17T04:00:50.498319+0000","last_unstale":"2024-09-17T04:00:52.475383+0000","last_undegraded":"2024-09-17T04:00:52.475383+0000","last_fullsized":"2024-09-17T04:00:52.475383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:33.289244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486473+0000","last_change":"2024-09-17T04:00:49.484809+0000","last_active":"2024-09-17T04:00:49.486473+0000","last_peered":"2024-09-17T04:00:49.486473+0000","last_clean":"2024-09-17T04:00:49.486473+0000","last_became_active":"2024-09-17T04:00:49.484517+0000","last_became_peered":"2024-09-17T04:00:49.484517+0000","last_unstale":"2024-09-17T04:00:49.486473+0000","last_undegraded":"2024-09-17T04:00:49.486473+0000","last_fullsized":"2024-09-17T04:00:49.486473+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:19.948285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472278+0000","last_change":"2024-09-17T04:00:49.472278+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472278+0000","last_undegraded":"2024-09-17T04:00:49.472278+0000","last_fullsized":"2024-09-17T04:00:49.472278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.488737+0000","last_change":"2024-09-17T04:00:49.483959+0000","last_active":"2024-09-17T04:00:49.488737+0000","last_peered":"2024-09-17T04:00:49.488737+0000","last_clean":"2024-09-17T04:00:49.488737+0000","last_became_active":"2024-09-17T04:00:49.483564+0000","last_became_peered":"2024-09-17T04:00:49.483564+0000","last_unstale":"2024-09-17T04:00:49.488737+0000","last_undegraded":"2024-09-17T04:00:49.488737+0000","last_fullsized":"2024-09-17T04:00:49.488737+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:16.050314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475394+0000","last_change":"2024-09-17T04:00:50.499998+0000","last_active":"2024-09-17T04:00:52.475394+0000","last_peered":"2024-09-17T04:00:52.475394+0000","last_clean":"2024-09-17T04:00:52.475394+0000","last_became_active":"2024-09-17T04:00:50.499605+0000","last_became_peered":"2024-09-17T04:00:50.499605+0000","last_unstale":"2024-09-17T04:00:52.475394+0000","last_undegraded":"2024-09-17T04:00:52.475394+0000","last_fullsized":"2024-09-17T04:00:52.475394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:04.373538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558123+0000","last_change":"2024-09-17T04:00:49.488756+0000","last_active":"2024-09-17T04:00:52.558123+0000","last_peered":"2024-09-17T04:00:52.558123+0000","last_clean":"2024-09-17T04:00:52.558123+0000","last_became_active":"2024-09-17T04:00:49.487367+0000","last_became_peered":"2024-09-17T04:00:49.487367+0000","last_unstale":"2024-09-17T04:00:52.558123+0000","last_undegraded":"2024-09-17T04:00:52.558123+0000","last_fullsized":"2024-09-17T04:00:52.558123+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475774+0000","last_change":"2024-09-17T04:00:50.504866+0000","last_active":"2024-09-17T04:00:52.475774+0000","last_peered":"2024-09-17T04:00:52.475774+0000","last_clean":"2024-09-17T04:00:52.475774+0000","last_became_active":"2024-09-17T04:00:50.504365+0000","last_became_peered":"2024-09-17T04:00:50.504365+0000","last_unstale":"2024-09-17T04:00:52.475774+0000","last_undegraded":"2024-09-17T04:00:52.475774+0000","last_fullsized":"2024-09-17T04:00:52.475774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:23.967019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485996+0000","last_change":"2024-09-17T04:00:49.484439+0000","last_active":"2024-09-17T04:00:49.485996+0000","last_peered":"2024-09-17T04:00:49.485996+0000","last_clean":"2024-09-17T04:00:49.485996+0000","last_became_active":"2024-09-17T04:00:49.483790+0000","last_became_peered":"2024-09-17T04:00:49.483790+0000","last_unstale":"2024-09-17T04:00:49.485996+0000","last_undegraded":"2024-09-17T04:00:49.485996+0000","last_fullsized":"2024-09-17T04:00:49.485996+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.011143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.469674+0000","last_change":"2024-09-17T04:00:49.469674+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.469674+0000","last_undegraded":"2024-09-17T04:00:49.469674+0000","last_fullsized":"2024-09-17T04:00:49.469674+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558774+0000","last_change":"2024-09-17T04:00:49.481548+0000","last_active":"2024-09-17T04:00:52.558774+0000","last_peered":"2024-09-17T04:00:52.558774+0000","last_clean":"2024-09-17T04:00:52.558774+0000","last_became_active":"2024-09-17T04:00:49.479803+0000","last_became_peered":"2024-09-17T04:00:49.479803+0000","last_unstale":"2024-09-17T04:00:52.558774+0000","last_undegraded":"2024-09-17T04:00:52.558774+0000","last_fullsized":"2024-09-17T04:00:52.558774+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475673+0000","last_change":"2024-09-17T04:00:50.501872+0000","last_active":"2024-09-17T04:00:52.475673+0000","last_peered":"2024-09-17T04:00:52.475673+0000","last_clean":"2024-09-17T04:00:52.475673+0000","last_became_active":"2024-09-17T04:00:50.501566+0000","last_became_peered":"2024-09-17T04:00:50.501566+0000","last_unstale":"2024-09-17T04:00:52.475673+0000","last_undegraded":"2024-09-17T04:00:52.475673+0000","last_fullsized":"2024-09-17T04:00:52.475673+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.084351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491568+0000","last_change":"2024-09-17T04:00:49.487399+0000","last_active":"2024-09-17T04:00:49.491568+0000","last_peered":"2024-09-17T04:00:49.491568+0000","last_clean":"2024-09-17T04:00:49.491568+0000","last_became_active":"2024-09-17T04:00:49.487044+0000","last_became_peered":"2024-09-17T04:00:49.487044+0000","last_unstale":"2024-09-17T04:00:49.491568+0000","last_undegraded":"2024-09-17T04:00:49.491568+0000","last_fullsized":"2024-09-17T04:00:49.491568+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:10.745031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490623+0000","last_change":"2024-09-17T04:00:50.500587+0000","last_active":"2024-09-17T04:00:51.490623+0000","last_peered":"2024-09-17T04:00:51.490623+0000","last_clean":"2024-09-17T04:00:51.490623+0000","last_became_active":"2024-09-17T04:00:50.500282+0000","last_became_peered":"2024-09-17T04:00:50.500282+0000","last_unstale":"2024-09-17T04:00:51.490623+0000","last_undegraded":"2024-09-17T04:00:51.490623+0000","last_fullsized":"2024-09-17T04:00:51.490623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:49:57.370683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"173'6","reported_seq":21,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475785+0000","last_change":"2024-09-17T04:00:49.491252+0000","last_active":"2024-09-17T04:00:52.475785+0000","last_peered":"2024-09-17T04:00:52.475785+0000","last_clean":"2024-09-17T04:00:52.475785+0000","last_became_active":"2024-09-17T04:00:49.490892+0000","last_became_peered":"2024-09-17T04:00:49.490892+0000","last_unstale":"2024-09-17T04:00:52.475785+0000","last_undegraded":"2024-09-17T04:00:52.475785+0000","last_fullsized":"2024-09-17T04:00:52.475785+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:02.780252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473556+0000","last_change":"2024-09-17T04:00:49.473556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473556+0000","last_undegraded":"2024-09-17T04:00:49.473556+0000","last_fullsized":"2024-09-17T04:00:49.473556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558379+0000","last_change":"2024-09-17T04:00:49.488763+0000","last_active":"2024-09-17T04:00:52.558379+0000","last_peered":"2024-09-17T04:00:52.558379+0000","last_clean":"2024-09-17T04:00:52.558379+0000","last_became_active":"2024-09-17T04:00:49.487670+0000","last_became_peered":"2024-09-17T04:00:49.487670+0000","last_unstale":"2024-09-17T04:00:52.558379+0000","last_undegraded":"2024-09-17T04:00:52.558379+0000","last_fullsized":"2024-09-17T04:00:52.558379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.500556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471211+0000","last_change":"2024-09-17T04:00:49.471211+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471211+0000","last_undegraded":"2024-09-17T04:00:49.471211+0000","last_fullsized":"2024-09-17T04:00:49.471211+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558318+0000","last_change":"2024-09-17T04:00:49.492017+0000","last_active":"2024-09-17T04:00:52.558318+0000","last_peered":"2024-09-17T04:00:52.558318+0000","last_clean":"2024-09-17T04:00:52.558318+0000","last_became_active":"2024-09-17T04:00:49.491561+0000","last_became_peered":"2024-09-17T04:00:49.491561+0000","last_unstale":"2024-09-17T04:00:52.558318+0000","last_undegraded":"2024-09-17T04:00:52.558318+0000","last_fullsized":"2024-09-17T04:00:52.558318+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467076+0000","last_change":"2024-09-17T04:00:49.467076+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467076+0000","last_undegraded":"2024-09-17T04:00:49.467076+0000","last_fullsized":"2024-09-17T04:00:49.467076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490472+0000","last_change":"2024-09-17T04:00:49.482396+0000","last_active":"2024-09-17T04:00:49.490472+0000","last_peered":"2024-09-17T04:00:49.490472+0000","last_clean":"2024-09-17T04:00:49.490472+0000","last_became_active":"2024-09-17T04:00:49.481991+0000","last_became_peered":"2024-09-17T04:00:49.481991+0000","last_unstale":"2024-09-17T04:00:49.490472+0000","last_undegraded":"2024-09-17T04:00:49.490472+0000","last_fullsized":"2024-09-17T04:00:49.490472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:15.031783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490625+0000","last_change":"2024-09-17T04:00:50.486171+0000","last_active":"2024-09-17T04:00:51.490625+0000","last_peered":"2024-09-17T04:00:51.490625+0000","last_clean":"2024-09-17T04:00:51.490625+0000","last_became_active":"2024-09-17T04:00:50.484422+0000","last_became_peered":"2024-09-17T04:00:50.484422+0000","last_unstale":"2024-09-17T04:00:51.490625+0000","last_undegraded":"2024-09-17T04:00:51.490625+0000","last_fullsized":"2024-09-17T04:00:51.490625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:09.236326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491600+0000","last_change":"2024-09-17T04:00:49.486848+0000","last_active":"2024-09-17T04:00:49.491600+0000","last_peered":"2024-09-17T04:00:49.491600+0000","last_clean":"2024-09-17T04:00:49.491600+0000","last_became_active":"2024-09-17T04:00:49.486463+0000","last_became_peered":"2024-09-17T04:00:49.486463+0000","last_unstale":"2024-09-17T04:00:49.491600+0000","last_undegraded":"2024-09-17T04:00:49.491600+0000","last_fullsized":"2024-09-17T04:00:49.491600+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:50.459427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493377+0000","last_change":"2024-09-17T04:00:50.495171+0000","last_active":"2024-09-17T04:00:51.493377+0000","last_peered":"2024-09-17T04:00:51.493377+0000","last_clean":"2024-09-17T04:00:51.493377+0000","last_became_active":"2024-09-17T04:00:50.494834+0000","last_became_peered":"2024-09-17T04:00:50.494834+0000","last_unstale":"2024-09-17T04:00:51.493377+0000","last_undegraded":"2024-09-17T04:00:51.493377+0000","last_fullsized":"2024-09-17T04:00:51.493377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:22.013808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.492419+0000","last_change":"2024-09-17T04:00:49.487492+0000","last_active":"2024-09-17T04:00:49.492419+0000","last_peered":"2024-09-17T04:00:49.492419+0000","last_clean":"2024-09-17T04:00:49.492419+0000","last_became_active":"2024-09-17T04:00:49.487146+0000","last_became_peered":"2024-09-17T04:00:49.487146+0000","last_unstale":"2024-09-17T04:00:49.492419+0000","last_undegraded":"2024-09-17T04:00:49.492419+0000","last_fullsized":"2024-09-17T04:00:49.492419+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493310+0000","last_change":"2024-09-17T04:00:50.484922+0000","last_active":"2024-09-17T04:00:51.493310+0000","last_peered":"2024-09-17T04:00:51.493310+0000","last_clean":"2024-09-17T04:00:51.493310+0000","last_became_active":"2024-09-17T04:00:50.483724+0000","last_became_peered":"2024-09-17T04:00:50.483724+0000","last_unstale":"2024-09-17T04:00:51.493310+0000","last_undegraded":"2024-09-17T04:00:51.493310+0000","last_fullsized":"2024-09-17T04:00:51.493310+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:40.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493354+0000","last_change":"2024-09-17T04:00:49.475839+0000","last_active":"2024-09-17T04:00:51.493354+0000","last_peered":"2024-09-17T04:00:51.493354+0000","last_clean":"2024-09-17T04:00:51.493354+0000","last_became_active":"2024-09-17T04:00:49.475239+0000","last_became_peered":"2024-09-17T04:00:49.475239+0000","last_unstale":"2024-09-17T04:00:51.493354+0000","last_undegraded":"2024-09-17T04:00:51.493354+0000","last_fullsized":"2024-09-17T04:00:51.493354+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:14.501945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467772+0000","last_change":"2024-09-17T04:00:49.467772+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467772+0000","last_undegraded":"2024-09-17T04:00:49.467772+0000","last_fullsized":"2024-09-17T04:00:49.467772+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491597+0000","last_change":"2024-09-17T04:00:49.485542+0000","last_active":"2024-09-17T04:00:49.491597+0000","last_peered":"2024-09-17T04:00:49.491597+0000","last_clean":"2024-09-17T04:00:49.491597+0000","last_became_active":"2024-09-17T04:00:49.481811+0000","last_became_peered":"2024-09-17T04:00:49.481811+0000","last_unstale":"2024-09-17T04:00:49.491597+0000","last_undegraded":"2024-09-17T04:00:49.491597+0000","last_fullsized":"2024-09-17T04:00:49.491597+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.533653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472723+0000","last_change":"2024-09-17T04:00:49.472723+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472723+0000","last_undegraded":"2024-09-17T04:00:49.472723+0000","last_fullsized":"2024-09-17T04:00:49.472723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490994+0000","last_change":"2024-09-17T04:00:49.485281+0000","last_active":"2024-09-17T04:00:49.490994+0000","last_peered":"2024-09-17T04:00:49.490994+0000","last_clean":"2024-09-17T04:00:49.490994+0000","last_became_active":"2024-09-17T04:00:49.484098+0000","last_became_peered":"2024-09-17T04:00:49.484098+0000","last_unstale":"2024-09-17T04:00:49.490994+0000","last_undegraded":"2024-09-17T04:00:49.490994+0000","last_fullsized":"2024-09-17T04:00:49.490994+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.496374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471100+0000","last_change":"2024-09-17T04:00:49.471100+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471100+0000","last_undegraded":"2024-09-17T04:00:49.471100+0000","last_fullsized":"2024-09-17T04:00:49.471100+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485054+0000","last_change":"2024-09-17T04:00:49.479444+0000","last_active":"2024-09-17T04:00:49.485054+0000","last_peered":"2024-09-17T04:00:49.485054+0000","last_clean":"2024-09-17T04:00:49.485054+0000","last_became_active":"2024-09-17T04:00:49.479095+0000","last_became_peered":"2024-09-17T04:00:49.479095+0000","last_unstale":"2024-09-17T04:00:49.485054+0000","last_undegraded":"2024-09-17T04:00:49.485054+0000","last_fullsized":"2024-09-17T04:00:49.485054+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:09.201712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468009+0000","last_change":"2024-09-17T04:00:49.468009+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468009+0000","last_undegraded":"2024-09-17T04:00:49.468009+0000","last_fullsized":"2024-09-17T04:00:49.468009+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490347+0000","last_change":"2024-09-17T04:00:49.492248+0000","last_active":"2024-09-17T04:00:51.490347+0000","last_peered":"2024-09-17T04:00:51.490347+0000","last_clean":"2024-09-17T04:00:51.490347+0000","last_became_active":"2024-09-17T04:00:49.491744+0000","last_became_peered":"2024-09-17T04:00:49.491744+0000","last_unstale":"2024-09-17T04:00:51.490347+0000","last_undegraded":"2024-09-17T04:00:51.490347+0000","last_fullsized":"2024-09-17T04:00:51.490347+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:45.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492551+0000","last_change":"2024-09-17T04:00:50.485354+0000","last_active":"2024-09-17T04:00:51.492551+0000","last_peered":"2024-09-17T04:00:51.492551+0000","last_clean":"2024-09-17T04:00:51.492551+0000","last_became_active":"2024-09-17T04:00:50.485017+0000","last_became_peered":"2024-09-17T04:00:50.485017+0000","last_unstale":"2024-09-17T04:00:51.492551+0000","last_undegraded":"2024-09-17T04:00:51.492551+0000","last_fullsized":"2024-09-17T04:00:51.492551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:43.479866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490209+0000","last_change":"2024-09-17T04:00:49.490198+0000","last_active":"2024-09-17T04:00:51.490209+0000","last_peered":"2024-09-17T04:00:51.490209+0000","last_clean":"2024-09-17T04:00:51.490209+0000","last_became_active":"2024-09-17T04:00:49.489685+0000","last_became_peered":"2024-09-17T04:00:49.489685+0000","last_unstale":"2024-09-17T04:00:51.490209+0000","last_undegraded":"2024-09-17T04:00:51.490209+0000","last_fullsized":"2024-09-17T04:00:51.490209+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:21.987330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558604+0000","last_change":"2024-09-17T04:00:50.488092+0000","last_active":"2024-09-17T04:00:52.558604+0000","last_peered":"2024-09-17T04:00:52.558604+0000","last_clean":"2024-09-17T04:00:52.558604+0000","last_became_active":"2024-09-17T04:00:50.487718+0000","last_became_peered":"2024-09-17T04:00:50.487718+0000","last_unstale":"2024-09-17T04:00:52.558604+0000","last_undegraded":"2024-09-17T04:00:52.558604+0000","last_fullsized":"2024-09-17T04:00:52.558604+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:12.457060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.494752+0000","last_change":"2024-09-17T04:00:49.492610+0000","last_active":"2024-09-17T04:00:49.494752+0000","last_peered":"2024-09-17T04:00:49.494752+0000","last_clean":"2024-09-17T04:00:49.494752+0000","last_became_active":"2024-09-17T04:00:49.491506+0000","last_became_peered":"2024-09-17T04:00:49.491506+0000","last_unstale":"2024-09-17T04:00:49.494752+0000","last_undegraded":"2024-09-17T04:00:49.494752+0000","last_fullsized":"2024-09-17T04:00:49.494752+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:53.116814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954545+0000","last_change":"2024-09-17T04:00:50.511428+0000","last_active":"2024-09-17T04:00:51.954545+0000","last_peered":"2024-09-17T04:00:51.954545+0000","last_clean":"2024-09-17T04:00:51.954545+0000","last_became_active":"2024-09-17T04:00:50.511114+0000","last_became_peered":"2024-09-17T04:00:50.511114+0000","last_unstale":"2024-09-17T04:00:51.954545+0000","last_undegraded":"2024-09-17T04:00:51.954545+0000","last_fullsized":"2024-09-17T04:00:51.954545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:58.184294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485901+0000","last_change":"2024-09-17T04:00:49.478561+0000","last_active":"2024-09-17T04:00:49.485901+0000","last_peered":"2024-09-17T04:00:49.485901+0000","last_clean":"2024-09-17T04:00:49.485901+0000","last_became_active":"2024-09-17T04:00:49.478248+0000","last_became_peered":"2024-09-17T04:00:49.478248+0000","last_unstale":"2024-09-17T04:00:49.485901+0000","last_undegraded":"2024-09-17T04:00:49.485901+0000","last_fullsized":"2024-09-17T04:00:49.485901+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:13.907873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490983+0000","last_change":"2024-09-17T04:00:50.495573+0000","last_active":"2024-09-17T04:00:51.490983+0000","last_peered":"2024-09-17T04:00:51.490983+0000","last_clean":"2024-09-17T04:00:51.490983+0000","last_became_active":"2024-09-17T04:00:50.494895+0000","last_became_peered":"2024-09-17T04:00:50.494895+0000","last_unstale":"2024-09-17T04:00:51.490983+0000","last_undegraded":"2024-09-17T04:00:51.490983+0000","last_fullsized":"2024-09-17T04:00:51.490983+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:07.480340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"174'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475720+0000","last_change":"2024-09-17T04:00:49.490303+0000","last_active":"2024-09-17T04:00:52.475720+0000","last_peered":"2024-09-17T04:00:52.475720+0000","last_clean":"2024-09-17T04:00:52.475720+0000","last_became_active":"2024-09-17T04:00:49.489882+0000","last_became_peered":"2024-09-17T04:00:49.489882+0000","last_unstale":"2024-09-17T04:00:52.475720+0000","last_undegraded":"2024-09-17T04:00:52.475720+0000","last_fullsized":"2024-09-17T04:00:52.475720+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:46.567261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492851+0000","last_change":"2024-09-17T04:00:50.493244+0000","last_active":"2024-09-17T04:00:51.492851+0000","last_peered":"2024-09-17T04:00:51.492851+0000","last_clean":"2024-09-17T04:00:51.492851+0000","last_became_active":"2024-09-17T04:00:50.492892+0000","last_became_peered":"2024-09-17T04:00:50.492892+0000","last_unstale":"2024-09-17T04:00:51.492851+0000","last_undegraded":"2024-09-17T04:00:51.492851+0000","last_fullsized":"2024-09-17T04:00:51.492851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:19.515809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.484869+0000","last_change":"2024-09-17T04:00:49.480983+0000","last_active":"2024-09-17T04:00:49.484869+0000","last_peered":"2024-09-17T04:00:49.484869+0000","last_clean":"2024-09-17T04:00:49.484869+0000","last_became_active":"2024-09-17T04:00:49.480620+0000","last_became_peered":"2024-09-17T04:00:49.480620+0000","last_unstale":"2024-09-17T04:00:49.484869+0000","last_undegraded":"2024-09-17T04:00:49.484869+0000","last_fullsized":"2024-09-17T04:00:49.484869+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:17.334558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558754+0000","last_change":"2024-09-17T04:00:50.491601+0000","last_active":"2024-09-17T04:00:52.558754+0000","last_peered":"2024-09-17T04:00:52.558754+0000","last_clean":"2024-09-17T04:00:52.558754+0000","last_became_active":"2024-09-17T04:00:50.491279+0000","last_became_peered":"2024-09-17T04:00:50.491279+0000","last_unstale":"2024-09-17T04:00:52.558754+0000","last_undegraded":"2024-09-17T04:00:52.558754+0000","last_fullsized":"2024-09-17T04:00:52.558754+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:03.782918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.482446+0000","last_change":"2024-09-17T04:00:49.477561+0000","last_active":"2024-09-17T04:00:49.482446+0000","last_peered":"2024-09-17T04:00:49.482446+0000","last_clean":"2024-09-17T04:00:49.482446+0000","last_became_active":"2024-09-17T04:00:49.477252+0000","last_became_peered":"2024-09-17T04:00:49.477252+0000","last_unstale":"2024-09-17T04:00:49.482446+0000","last_undegraded":"2024-09-17T04:00:49.482446+0000","last_fullsized":"2024-09-17T04:00:49.482446+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:14.568584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490348+0000","last_change":"2024-09-17T04:00:50.499748+0000","last_active":"2024-09-17T04:00:51.490348+0000","last_peered":"2024-09-17T04:00:51.490348+0000","last_clean":"2024-09-17T04:00:51.490348+0000","last_became_active":"2024-09-17T04:00:50.497933+0000","last_became_peered":"2024-09-17T04:00:50.497933+0000","last_unstale":"2024-09-17T04:00:51.490348+0000","last_undegraded":"2024-09-17T04:00:51.490348+0000","last_fullsized":"2024-09-17T04:00:51.490348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:33.887766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475476+0000","last_change":"2024-09-17T04:00:49.486586+0000","last_active":"2024-09-17T04:00:52.475476+0000","last_peered":"2024-09-17T04:00:52.475476+0000","last_clean":"2024-09-17T04:00:52.475476+0000","last_became_active":"2024-09-17T04:00:49.486099+0000","last_became_peered":"2024-09-17T04:00:49.486099+0000","last_unstale":"2024-09-17T04:00:52.475476+0000","last_undegraded":"2024-09-17T04:00:52.475476+0000","last_fullsized":"2024-09-17T04:00:52.475476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:42.671635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954355+0000","last_change":"2024-09-17T04:00:50.501002+0000","last_active":"2024-09-17T04:00:51.954355+0000","last_peered":"2024-09-17T04:00:51.954355+0000","last_clean":"2024-09-17T04:00:51.954355+0000","last_became_active":"2024-09-17T04:00:50.500621+0000","last_became_peered":"2024-09-17T04:00:50.500621+0000","last_unstale":"2024-09-17T04:00:51.954355+0000","last_undegraded":"2024-09-17T04:00:51.954355+0000","last_fullsized":"2024-09-17T04:00:51.954355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:38.586287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486445+0000","last_change":"2024-09-17T04:00:49.483997+0000","last_active":"2024-09-17T04:00:49.486445+0000","last_peered":"2024-09-17T04:00:49.486445+0000","last_clean":"2024-09-17T04:00:49.486445+0000","last_became_active":"2024-09-17T04:00:49.483458+0000","last_became_peered":"2024-09-17T04:00:49.483458+0000","last_unstale":"2024-09-17T04:00:49.486445+0000","last_undegraded":"2024-09-17T04:00:49.486445+0000","last_fullsized":"2024-09-17T04:00:49.486445+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:10:19.789984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.466540+0000","last_change":"2024-09-17T04:00:49.466540+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.466540+0000","last_undegraded":"2024-09-17T04:00:49.466540+0000","last_fullsized":"2024-09-17T04:00:49.466540+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558691+0000","last_change":"2024-09-17T04:00:49.485410+0000","last_active":"2024-09-17T04:00:52.558691+0000","last_peered":"2024-09-17T04:00:52.558691+0000","last_clean":"2024-09-17T04:00:52.558691+0000","last_became_active":"2024-09-17T04:00:49.484171+0000","last_became_peered":"2024-09-17T04:00:49.484171+0000","last_unstale":"2024-09-17T04:00:52.558691+0000","last_undegraded":"2024-09-17T04:00:52.558691+0000","last_fullsized":"2024-09-17T04:00:52.558691+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492593+0000","last_change":"2024-09-17T04:00:50.490365+0000","last_active":"2024-09-17T04:00:51.492593+0000","last_peered":"2024-09-17T04:00:51.492593+0000","last_clean":"2024-09-17T04:00:51.492593+0000","last_became_active":"2024-09-17T04:00:50.488098+0000","last_became_peered":"2024-09-17T04:00:50.488098+0000","last_unstale":"2024-09-17T04:00:51.492593+0000","last_undegraded":"2024-09-17T04:00:51.492593+0000","last_fullsized":"2024-09-17T04:00:51.492593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:42.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954116+0000","last_change":"2024-09-17T04:00:49.487212+0000","last_active":"2024-09-17T04:00:51.954116+0000","last_peered":"2024-09-17T04:00:51.954116+0000","last_clean":"2024-09-17T04:00:51.954116+0000","last_became_active":"2024-09-17T04:00:49.486849+0000","last_became_peered":"2024-09-17T04:00:49.486849+0000","last_unstale":"2024-09-17T04:00:51.954116+0000","last_undegraded":"2024-09-17T04:00:51.954116+0000","last_fullsized":"2024-09-17T04:00:51.954116+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:31.908526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490138+0000","last_change":"2024-09-17T04:00:50.501206+0000","last_active":"2024-09-17T04:00:51.490138+0000","last_peered":"2024-09-17T04:00:51.490138+0000","last_clean":"2024-09-17T04:00:51.490138+0000","last_became_active":"2024-09-17T04:00:50.500932+0000","last_became_peered":"2024-09-17T04:00:50.500932+0000","last_unstale":"2024-09-17T04:00:51.490138+0000","last_undegraded":"2024-09-17T04:00:51.490138+0000","last_fullsized":"2024-09-17T04:00:51.490138+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:32.987666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954611+0000","last_change":"2024-09-17T04:00:49.489179+0000","last_active":"2024-09-17T04:00:51.954611+0000","last_peered":"2024-09-17T04:00:51.954611+0000","last_clean":"2024-09-17T04:00:51.954611+0000","last_became_active":"2024-09-17T04:00:49.488400+0000","last_became_peered":"2024-09-17T04:00:49.488400+0000","last_unstale":"2024-09-17T04:00:51.954611+0000","last_undegraded":"2024-09-17T04:00:51.954611+0000","last_fullsized":"2024-09-17T04:00:51.954611+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:44:31.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":392,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475314+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:52.475314+0000","last_peered":"2024-09-17T04:00:52.475314+0000","last_clean":"2024-09-17T04:00:52.475314+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:52.475314+0000","last_undegraded":"2024-09-17T04:00:52.475314+0000","last_fullsized":"2024-09-17T04:00:52.475314+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509580,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2793472,"data_stored":1549678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2207744,"data_stored":959414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509579,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542285,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2785280,"data_stored":1548156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:54.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:55.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 -- 172.21.15.129:0/129980335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa6740cd390 msgr2=0x7fa67406f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 --2- 172.21.15.129:0/129980335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa6740cd390 0x7fa67406f640 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa66c005fd0 tx=0x7fa66c02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 -- 172.21.15.129:0/129980335 shutdown_connections 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 --2- 172.21.15.129:0/129980335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa6740cd390 0x7fa67406f640 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 --2- 172.21.15.129:0/129980335 >> v2:172.21.15.181:3300/0 conn(0x7fa6740af4c0 0x7fa67415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 -- 172.21.15.129:0/129980335 >> 172.21.15.129:0/129980335 conn(0x7fa67406d410 msgr2=0x7fa6740b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 -- 172.21.15.129:0/129980335 shutdown_connections 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.204+0000 7fa679a00640 1 -- 172.21.15.129:0/129980335 wait complete. 2024-09-17T04:00:55.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.206+0000 7fa679a00640 1 Processor -- start 2024-09-17T04:00:55.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.206+0000 7fa679a00640 1 -- start start 2024-09-17T04:00:55.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 0x7fa67415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 0x7fa67415ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa67406fb80 con 0x7fa6740af4c0 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa67406fd00 con 0x7fa67415b340 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa679a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa67406fe80 con 0x7fa67415a080 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35160/0 (socket says 172.21.15.129:35160) 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 -- 172.21.15.129:0/228160378 learned_addr learned my addr 172.21.15.129:0/228160378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa673400640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 0x7fa67415ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa66be00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 0x7fa67415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 -- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 msgr2=0x7fa67415ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 0x7fa67415ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 -- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 msgr2=0x7fa67415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.208+0000 7fa672a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 0x7fa67415e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa672a00640 1 -- 172.21.15.129:0/228160378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6740c68e0 con 0x7fa6740af4c0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa673400640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 0x7fa67415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa66be00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 0x7fa67415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa672a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fa66400e9f0 tx=0x7fa66400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa66400cd80 con 0x7fa6740af4c0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa664004020 con 0x7fa6740af4c0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6740c9140 con 0x7fa6740af4c0 2024-09-17T04:00:55.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa664004450 con 0x7fa6740af4c0 2024-09-17T04:00:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.209+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6740c8e10 con 0x7fa6740af4c0 2024-09-17T04:00:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.210+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa674070300 con 0x7fa6740af4c0 2024-09-17T04:00:55.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.211+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa6640035f0 con 0x7fa6740af4c0 2024-09-17T04:00:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.212+0000 7fa66aa00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 0x7fa65007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.212+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fa66409b7a0 con 0x7fa6740af4c0 2024-09-17T04:00:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.213+0000 7fa66be00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 0x7fa65007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.214+0000 7fa66be00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 0x7fa65007b0a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa6740709d0 tx=0x7fa65c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:55.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.217+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa664063b40 con 0x7fa6740af4c0 2024-09-17T04:00:55.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.397+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa6740c9790 con 0x7fa650078b90 2024-09-17T04:00:55.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.402+0000 7fa66aa00640 1 -- 172.21.15.129:0/228160378 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250551 (secure 0 0 0) 0x7fa6740c9790 con 0x7fa650078b90 2024-09-17T04:00:55.405 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:55.406 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 msgr2=0x7fa65007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 0x7fa65007b0a0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa6740709d0 tx=0x7fa65c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 msgr2=0x7fa6741584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7fa66400e9f0 tx=0x7fa66400eeb0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 shutdown_connections 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa650078b90 0x7fa65007b0a0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa67415a080 0x7fa67415ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67415b340 0x7fa67415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 --2- 172.21.15.129:0/228160378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6740af4c0 0x7fa6741584d0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 >> 172.21.15.129:0/228160378 conn(0x7fa67406d410 msgr2=0x7fa67415c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 shutdown_connections 2024-09-17T04:00:55.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.406+0000 7fa679a00640 1 -- 172.21.15.129:0/228160378 wait complete. 2024-09-17T04:00:55.430 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-17T04:00:54.487043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592596,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":121,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":325948,"kb_used_data":18184,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754648772,"statfs":{"total":773094113280,"available":772760342528,"internally_reserved":0,"allocated":18620416,"data_stored":8788852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002410"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490698+0000","last_change":"2024-09-17T04:00:50.499023+0000","last_active":"2024-09-17T04:00:51.490698+0000","last_peered":"2024-09-17T04:00:51.490698+0000","last_clean":"2024-09-17T04:00:51.490698+0000","last_became_active":"2024-09-17T04:00:50.498702+0000","last_became_peered":"2024-09-17T04:00:50.498702+0000","last_unstale":"2024-09-17T04:00:51.490698+0000","last_undegraded":"2024-09-17T04:00:51.490698+0000","last_fullsized":"2024-09-17T04:00:51.490698+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:47.660122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493082+0000","last_change":"2024-09-17T04:00:50.481292+0000","last_active":"2024-09-17T04:00:51.493082+0000","last_peered":"2024-09-17T04:00:51.493082+0000","last_clean":"2024-09-17T04:00:51.493082+0000","last_became_active":"2024-09-17T04:00:50.480948+0000","last_became_peered":"2024-09-17T04:00:50.480948+0000","last_unstale":"2024-09-17T04:00:51.493082+0000","last_undegraded":"2024-09-17T04:00:51.493082+0000","last_fullsized":"2024-09-17T04:00:51.493082+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.354995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470097+0000","last_change":"2024-09-17T04:00:49.470097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470097+0000","last_undegraded":"2024-09-17T04:00:49.470097+0000","last_fullsized":"2024-09-17T04:00:49.470097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475438+0000","last_change":"2024-09-17T04:00:50.500751+0000","last_active":"2024-09-17T04:00:52.475438+0000","last_peered":"2024-09-17T04:00:52.475438+0000","last_clean":"2024-09-17T04:00:52.475438+0000","last_became_active":"2024-09-17T04:00:50.500022+0000","last_became_peered":"2024-09-17T04:00:50.500022+0000","last_unstale":"2024-09-17T04:00:52.475438+0000","last_undegraded":"2024-09-17T04:00:52.475438+0000","last_fullsized":"2024-09-17T04:00:52.475438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:04.192066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472481+0000","last_change":"2024-09-17T04:00:49.472481+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472481+0000","last_undegraded":"2024-09-17T04:00:49.472481+0000","last_fullsized":"2024-09-17T04:00:49.472481+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474000+0000","last_change":"2024-09-17T04:00:49.474000+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474000+0000","last_undegraded":"2024-09-17T04:00:49.474000+0000","last_fullsized":"2024-09-17T04:00:49.474000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490063+0000","last_change":"2024-09-17T04:00:50.488121+0000","last_active":"2024-09-17T04:00:51.490063+0000","last_peered":"2024-09-17T04:00:51.490063+0000","last_clean":"2024-09-17T04:00:51.490063+0000","last_became_active":"2024-09-17T04:00:50.487823+0000","last_became_peered":"2024-09-17T04:00:50.487823+0000","last_unstale":"2024-09-17T04:00:51.490063+0000","last_undegraded":"2024-09-17T04:00:51.490063+0000","last_fullsized":"2024-09-17T04:00:51.490063+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:27.505920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472807+0000","last_change":"2024-09-17T04:00:49.472807+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472807+0000","last_undegraded":"2024-09-17T04:00:49.472807+0000","last_fullsized":"2024-09-17T04:00:49.472807+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493544+0000","last_change":"2024-09-17T04:00:50.494343+0000","last_active":"2024-09-17T04:00:51.493544+0000","last_peered":"2024-09-17T04:00:51.493544+0000","last_clean":"2024-09-17T04:00:51.493544+0000","last_became_active":"2024-09-17T04:00:50.493022+0000","last_became_peered":"2024-09-17T04:00:50.493022+0000","last_unstale":"2024-09-17T04:00:51.493544+0000","last_undegraded":"2024-09-17T04:00:51.493544+0000","last_fullsized":"2024-09-17T04:00:51.493544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:20.950951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474579+0000","last_change":"2024-09-17T04:00:49.474579+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474579+0000","last_undegraded":"2024-09-17T04:00:49.474579+0000","last_fullsized":"2024-09-17T04:00:49.474579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468304+0000","last_change":"2024-09-17T04:00:49.468304+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468304+0000","last_undegraded":"2024-09-17T04:00:49.468304+0000","last_fullsized":"2024-09-17T04:00:49.468304+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954479+0000","last_change":"2024-09-17T04:00:50.503620+0000","last_active":"2024-09-17T04:00:51.954479+0000","last_peered":"2024-09-17T04:00:51.954479+0000","last_clean":"2024-09-17T04:00:51.954479+0000","last_became_active":"2024-09-17T04:00:50.503238+0000","last_became_peered":"2024-09-17T04:00:50.503238+0000","last_unstale":"2024-09-17T04:00:51.954479+0000","last_undegraded":"2024-09-17T04:00:51.954479+0000","last_fullsized":"2024-09-17T04:00:51.954479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:10.261218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954637+0000","last_change":"2024-09-17T04:00:50.502335+0000","last_active":"2024-09-17T04:00:51.954637+0000","last_peered":"2024-09-17T04:00:51.954637+0000","last_clean":"2024-09-17T04:00:51.954637+0000","last_became_active":"2024-09-17T04:00:50.502036+0000","last_became_peered":"2024-09-17T04:00:50.502036+0000","last_unstale":"2024-09-17T04:00:51.954637+0000","last_undegraded":"2024-09-17T04:00:51.954637+0000","last_fullsized":"2024-09-17T04:00:51.954637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:37.502247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490230+0000","last_change":"2024-09-17T04:00:50.501289+0000","last_active":"2024-09-17T04:00:51.490230+0000","last_peered":"2024-09-17T04:00:51.490230+0000","last_clean":"2024-09-17T04:00:51.490230+0000","last_became_active":"2024-09-17T04:00:50.501012+0000","last_became_peered":"2024-09-17T04:00:50.501012+0000","last_unstale":"2024-09-17T04:00:51.490230+0000","last_undegraded":"2024-09-17T04:00:51.490230+0000","last_fullsized":"2024-09-17T04:00:51.490230+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:36.532175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490074+0000","last_change":"2024-09-17T04:00:50.490115+0000","last_active":"2024-09-17T04:00:51.490074+0000","last_peered":"2024-09-17T04:00:51.490074+0000","last_clean":"2024-09-17T04:00:51.490074+0000","last_became_active":"2024-09-17T04:00:50.489826+0000","last_became_peered":"2024-09-17T04:00:50.489826+0000","last_unstale":"2024-09-17T04:00:51.490074+0000","last_undegraded":"2024-09-17T04:00:51.490074+0000","last_fullsized":"2024-09-17T04:00:51.490074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:11.406297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.475124+0000","last_change":"2024-09-17T04:00:49.475124+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.475124+0000","last_undegraded":"2024-09-17T04:00:49.475124+0000","last_fullsized":"2024-09-17T04:00:49.475124+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490971+0000","last_change":"2024-09-17T04:00:50.497620+0000","last_active":"2024-09-17T04:00:51.490971+0000","last_peered":"2024-09-17T04:00:51.490971+0000","last_clean":"2024-09-17T04:00:51.490971+0000","last_became_active":"2024-09-17T04:00:50.495270+0000","last_became_peered":"2024-09-17T04:00:50.495270+0000","last_unstale":"2024-09-17T04:00:51.490971+0000","last_undegraded":"2024-09-17T04:00:51.490971+0000","last_fullsized":"2024-09-17T04:00:51.490971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:29.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474097+0000","last_change":"2024-09-17T04:00:49.474097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474097+0000","last_undegraded":"2024-09-17T04:00:49.474097+0000","last_fullsized":"2024-09-17T04:00:49.474097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476999+0000","last_change":"2024-09-17T04:00:49.476999+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476999+0000","last_undegraded":"2024-09-17T04:00:49.476999+0000","last_fullsized":"2024-09-17T04:00:49.476999+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473627+0000","last_change":"2024-09-17T04:00:49.473627+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473627+0000","last_undegraded":"2024-09-17T04:00:49.473627+0000","last_fullsized":"2024-09-17T04:00:49.473627+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475406+0000","last_change":"2024-09-17T04:00:50.499009+0000","last_active":"2024-09-17T04:00:52.475406+0000","last_peered":"2024-09-17T04:00:52.475406+0000","last_clean":"2024-09-17T04:00:52.475406+0000","last_became_active":"2024-09-17T04:00:50.495348+0000","last_became_peered":"2024-09-17T04:00:50.495348+0000","last_unstale":"2024-09-17T04:00:52.475406+0000","last_undegraded":"2024-09-17T04:00:52.475406+0000","last_fullsized":"2024-09-17T04:00:52.475406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:38.854990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558507+0000","last_change":"2024-09-17T04:00:50.483447+0000","last_active":"2024-09-17T04:00:52.558507+0000","last_peered":"2024-09-17T04:00:52.558507+0000","last_clean":"2024-09-17T04:00:52.558507+0000","last_became_active":"2024-09-17T04:00:50.483105+0000","last_became_peered":"2024-09-17T04:00:50.483105+0000","last_unstale":"2024-09-17T04:00:52.558507+0000","last_undegraded":"2024-09-17T04:00:52.558507+0000","last_fullsized":"2024-09-17T04:00:52.558507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:22.999066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558726+0000","last_change":"2024-09-17T04:00:50.488020+0000","last_active":"2024-09-17T04:00:52.558726+0000","last_peered":"2024-09-17T04:00:52.558726+0000","last_clean":"2024-09-17T04:00:52.558726+0000","last_became_active":"2024-09-17T04:00:50.487293+0000","last_became_peered":"2024-09-17T04:00:50.487293+0000","last_unstale":"2024-09-17T04:00:52.558726+0000","last_undegraded":"2024-09-17T04:00:52.558726+0000","last_fullsized":"2024-09-17T04:00:52.558726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:15.011545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475667+0000","last_change":"2024-09-17T04:00:50.493247+0000","last_active":"2024-09-17T04:00:52.475667+0000","last_peered":"2024-09-17T04:00:52.475667+0000","last_clean":"2024-09-17T04:00:52.475667+0000","last_became_active":"2024-09-17T04:00:50.492907+0000","last_became_peered":"2024-09-17T04:00:50.492907+0000","last_unstale":"2024-09-17T04:00:52.475667+0000","last_undegraded":"2024-09-17T04:00:52.475667+0000","last_fullsized":"2024-09-17T04:00:52.475667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:58.410921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493295+0000","last_change":"2024-09-17T04:00:50.487190+0000","last_active":"2024-09-17T04:00:51.493295+0000","last_peered":"2024-09-17T04:00:51.493295+0000","last_clean":"2024-09-17T04:00:51.493295+0000","last_became_active":"2024-09-17T04:00:50.484107+0000","last_became_peered":"2024-09-17T04:00:50.484107+0000","last_unstale":"2024-09-17T04:00:51.493295+0000","last_undegraded":"2024-09-17T04:00:51.493295+0000","last_fullsized":"2024-09-17T04:00:51.493295+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:17.052980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490680+0000","last_change":"2024-09-17T04:00:50.500005+0000","last_active":"2024-09-17T04:00:51.490680+0000","last_peered":"2024-09-17T04:00:51.490680+0000","last_clean":"2024-09-17T04:00:51.490680+0000","last_became_active":"2024-09-17T04:00:50.497315+0000","last_became_peered":"2024-09-17T04:00:50.497315+0000","last_unstale":"2024-09-17T04:00:51.490680+0000","last_undegraded":"2024-09-17T04:00:51.490680+0000","last_fullsized":"2024-09-17T04:00:51.490680+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:50.074066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954424+0000","last_change":"2024-09-17T04:00:50.507420+0000","last_active":"2024-09-17T04:00:51.954424+0000","last_peered":"2024-09-17T04:00:51.954424+0000","last_clean":"2024-09-17T04:00:51.954424+0000","last_became_active":"2024-09-17T04:00:50.506925+0000","last_became_peered":"2024-09-17T04:00:50.506925+0000","last_unstale":"2024-09-17T04:00:51.954424+0000","last_undegraded":"2024-09-17T04:00:51.954424+0000","last_fullsized":"2024-09-17T04:00:51.954424+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.177663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470536+0000","last_change":"2024-09-17T04:00:49.470536+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470536+0000","last_undegraded":"2024-09-17T04:00:49.470536+0000","last_fullsized":"2024-09-17T04:00:49.470536+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954415+0000","last_change":"2024-09-17T04:00:50.507336+0000","last_active":"2024-09-17T04:00:51.954415+0000","last_peered":"2024-09-17T04:00:51.954415+0000","last_clean":"2024-09-17T04:00:51.954415+0000","last_became_active":"2024-09-17T04:00:50.507000+0000","last_became_peered":"2024-09-17T04:00:50.507000+0000","last_unstale":"2024-09-17T04:00:51.954415+0000","last_undegraded":"2024-09-17T04:00:51.954415+0000","last_fullsized":"2024-09-17T04:00:51.954415+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:03.261144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492684+0000","last_change":"2024-09-17T04:00:50.508385+0000","last_active":"2024-09-17T04:00:51.492684+0000","last_peered":"2024-09-17T04:00:51.492684+0000","last_clean":"2024-09-17T04:00:51.492684+0000","last_became_active":"2024-09-17T04:00:50.507803+0000","last_became_peered":"2024-09-17T04:00:50.507803+0000","last_unstale":"2024-09-17T04:00:51.492684+0000","last_undegraded":"2024-09-17T04:00:51.492684+0000","last_fullsized":"2024-09-17T04:00:51.492684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:40.047416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473517+0000","last_change":"2024-09-17T04:00:49.473517+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473517+0000","last_undegraded":"2024-09-17T04:00:49.473517+0000","last_fullsized":"2024-09-17T04:00:49.473517+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490344+0000","last_change":"2024-09-17T04:00:50.504166+0000","last_active":"2024-09-17T04:00:51.490344+0000","last_peered":"2024-09-17T04:00:51.490344+0000","last_clean":"2024-09-17T04:00:51.490344+0000","last_became_active":"2024-09-17T04:00:50.503869+0000","last_became_peered":"2024-09-17T04:00:50.503869+0000","last_unstale":"2024-09-17T04:00:51.490344+0000","last_undegraded":"2024-09-17T04:00:51.490344+0000","last_fullsized":"2024-09-17T04:00:51.490344+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:44.335387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493116+0000","last_change":"2024-09-17T04:00:50.497167+0000","last_active":"2024-09-17T04:00:51.493116+0000","last_peered":"2024-09-17T04:00:51.493116+0000","last_clean":"2024-09-17T04:00:51.493116+0000","last_became_active":"2024-09-17T04:00:50.496798+0000","last_became_peered":"2024-09-17T04:00:50.496798+0000","last_unstale":"2024-09-17T04:00:51.493116+0000","last_undegraded":"2024-09-17T04:00:51.493116+0000","last_fullsized":"2024-09-17T04:00:51.493116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:18.337224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"173'4","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954531+0000","last_change":"2024-09-17T04:00:49.488273+0000","last_active":"2024-09-17T04:00:51.954531+0000","last_peered":"2024-09-17T04:00:51.954531+0000","last_clean":"2024-09-17T04:00:51.954531+0000","last_became_active":"2024-09-17T04:00:49.486926+0000","last_became_peered":"2024-09-17T04:00:49.486926+0000","last_unstale":"2024-09-17T04:00:51.954531+0000","last_undegraded":"2024-09-17T04:00:51.954531+0000","last_fullsized":"2024-09-17T04:00:51.954531+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:23.992360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558954+0000","last_change":"2024-09-17T04:00:50.493935+0000","last_active":"2024-09-17T04:00:52.558954+0000","last_peered":"2024-09-17T04:00:52.558954+0000","last_clean":"2024-09-17T04:00:52.558954+0000","last_became_active":"2024-09-17T04:00:50.490351+0000","last_became_peered":"2024-09-17T04:00:50.490351+0000","last_unstale":"2024-09-17T04:00:52.558954+0000","last_undegraded":"2024-09-17T04:00:52.558954+0000","last_fullsized":"2024-09-17T04:00:52.558954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:16.034449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493225+0000","last_change":"2024-09-17T04:00:49.485660+0000","last_active":"2024-09-17T04:00:51.493225+0000","last_peered":"2024-09-17T04:00:51.493225+0000","last_clean":"2024-09-17T04:00:51.493225+0000","last_became_active":"2024-09-17T04:00:49.484522+0000","last_became_peered":"2024-09-17T04:00:49.484522+0000","last_unstale":"2024-09-17T04:00:51.493225+0000","last_undegraded":"2024-09-17T04:00:51.493225+0000","last_fullsized":"2024-09-17T04:00:51.493225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:17.632102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490880+0000","last_change":"2024-09-17T04:00:50.493759+0000","last_active":"2024-09-17T04:00:51.490880+0000","last_peered":"2024-09-17T04:00:51.490880+0000","last_clean":"2024-09-17T04:00:51.490880+0000","last_became_active":"2024-09-17T04:00:50.493472+0000","last_became_peered":"2024-09-17T04:00:50.493472+0000","last_unstale":"2024-09-17T04:00:51.490880+0000","last_undegraded":"2024-09-17T04:00:51.490880+0000","last_fullsized":"2024-09-17T04:00:51.490880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:36.564425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490615+0000","last_change":"2024-09-17T04:00:49.485026+0000","last_active":"2024-09-17T04:00:49.490615+0000","last_peered":"2024-09-17T04:00:49.490615+0000","last_clean":"2024-09-17T04:00:49.490615+0000","last_became_active":"2024-09-17T04:00:49.484730+0000","last_became_peered":"2024-09-17T04:00:49.484730+0000","last_unstale":"2024-09-17T04:00:49.490615+0000","last_undegraded":"2024-09-17T04:00:49.490615+0000","last_fullsized":"2024-09-17T04:00:49.490615+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:40.328332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476556+0000","last_change":"2024-09-17T04:00:49.476556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476556+0000","last_undegraded":"2024-09-17T04:00:49.476556+0000","last_fullsized":"2024-09-17T04:00:49.476556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475655+0000","last_change":"2024-09-17T04:00:49.492918+0000","last_active":"2024-09-17T04:00:52.475655+0000","last_peered":"2024-09-17T04:00:52.475655+0000","last_clean":"2024-09-17T04:00:52.475655+0000","last_became_active":"2024-09-17T04:00:49.492545+0000","last_became_peered":"2024-09-17T04:00:49.492545+0000","last_unstale":"2024-09-17T04:00:52.475655+0000","last_undegraded":"2024-09-17T04:00:52.475655+0000","last_fullsized":"2024-09-17T04:00:52.475655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:49.593934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475383+0000","last_change":"2024-09-17T04:00:50.498633+0000","last_active":"2024-09-17T04:00:52.475383+0000","last_peered":"2024-09-17T04:00:52.475383+0000","last_clean":"2024-09-17T04:00:52.475383+0000","last_became_active":"2024-09-17T04:00:50.498319+0000","last_became_peered":"2024-09-17T04:00:50.498319+0000","last_unstale":"2024-09-17T04:00:52.475383+0000","last_undegraded":"2024-09-17T04:00:52.475383+0000","last_fullsized":"2024-09-17T04:00:52.475383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:33.289244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486473+0000","last_change":"2024-09-17T04:00:49.484809+0000","last_active":"2024-09-17T04:00:49.486473+0000","last_peered":"2024-09-17T04:00:49.486473+0000","last_clean":"2024-09-17T04:00:49.486473+0000","last_became_active":"2024-09-17T04:00:49.484517+0000","last_became_peered":"2024-09-17T04:00:49.484517+0000","last_unstale":"2024-09-17T04:00:49.486473+0000","last_undegraded":"2024-09-17T04:00:49.486473+0000","last_fullsized":"2024-09-17T04:00:49.486473+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:19.948285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472278+0000","last_change":"2024-09-17T04:00:49.472278+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472278+0000","last_undegraded":"2024-09-17T04:00:49.472278+0000","last_fullsized":"2024-09-17T04:00:49.472278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.488737+0000","last_change":"2024-09-17T04:00:49.483959+0000","last_active":"2024-09-17T04:00:49.488737+0000","last_peered":"2024-09-17T04:00:49.488737+0000","last_clean":"2024-09-17T04:00:49.488737+0000","last_became_active":"2024-09-17T04:00:49.483564+0000","last_became_peered":"2024-09-17T04:00:49.483564+0000","last_unstale":"2024-09-17T04:00:49.488737+0000","last_undegraded":"2024-09-17T04:00:49.488737+0000","last_fullsized":"2024-09-17T04:00:49.488737+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:16.050314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475394+0000","last_change":"2024-09-17T04:00:50.499998+0000","last_active":"2024-09-17T04:00:52.475394+0000","last_peered":"2024-09-17T04:00:52.475394+0000","last_clean":"2024-09-17T04:00:52.475394+0000","last_became_active":"2024-09-17T04:00:50.499605+0000","last_became_peered":"2024-09-17T04:00:50.499605+0000","last_unstale":"2024-09-17T04:00:52.475394+0000","last_undegraded":"2024-09-17T04:00:52.475394+0000","last_fullsized":"2024-09-17T04:00:52.475394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:04.373538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558123+0000","last_change":"2024-09-17T04:00:49.488756+0000","last_active":"2024-09-17T04:00:52.558123+0000","last_peered":"2024-09-17T04:00:52.558123+0000","last_clean":"2024-09-17T04:00:52.558123+0000","last_became_active":"2024-09-17T04:00:49.487367+0000","last_became_peered":"2024-09-17T04:00:49.487367+0000","last_unstale":"2024-09-17T04:00:52.558123+0000","last_undegraded":"2024-09-17T04:00:52.558123+0000","last_fullsized":"2024-09-17T04:00:52.558123+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475774+0000","last_change":"2024-09-17T04:00:50.504866+0000","last_active":"2024-09-17T04:00:52.475774+0000","last_peered":"2024-09-17T04:00:52.475774+0000","last_clean":"2024-09-17T04:00:52.475774+0000","last_became_active":"2024-09-17T04:00:50.504365+0000","last_became_peered":"2024-09-17T04:00:50.504365+0000","last_unstale":"2024-09-17T04:00:52.475774+0000","last_undegraded":"2024-09-17T04:00:52.475774+0000","last_fullsized":"2024-09-17T04:00:52.475774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:23.967019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485996+0000","last_change":"2024-09-17T04:00:49.484439+0000","last_active":"2024-09-17T04:00:49.485996+0000","last_peered":"2024-09-17T04:00:49.485996+0000","last_clean":"2024-09-17T04:00:49.485996+0000","last_became_active":"2024-09-17T04:00:49.483790+0000","last_became_peered":"2024-09-17T04:00:49.483790+0000","last_unstale":"2024-09-17T04:00:49.485996+0000","last_undegraded":"2024-09-17T04:00:49.485996+0000","last_fullsized":"2024-09-17T04:00:49.485996+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.011143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.469674+0000","last_change":"2024-09-17T04:00:49.469674+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.469674+0000","last_undegraded":"2024-09-17T04:00:49.469674+0000","last_fullsized":"2024-09-17T04:00:49.469674+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558774+0000","last_change":"2024-09-17T04:00:49.481548+0000","last_active":"2024-09-17T04:00:52.558774+0000","last_peered":"2024-09-17T04:00:52.558774+0000","last_clean":"2024-09-17T04:00:52.558774+0000","last_became_active":"2024-09-17T04:00:49.479803+0000","last_became_peered":"2024-09-17T04:00:49.479803+0000","last_unstale":"2024-09-17T04:00:52.558774+0000","last_undegraded":"2024-09-17T04:00:52.558774+0000","last_fullsized":"2024-09-17T04:00:52.558774+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475673+0000","last_change":"2024-09-17T04:00:50.501872+0000","last_active":"2024-09-17T04:00:52.475673+0000","last_peered":"2024-09-17T04:00:52.475673+0000","last_clean":"2024-09-17T04:00:52.475673+0000","last_became_active":"2024-09-17T04:00:50.501566+0000","last_became_peered":"2024-09-17T04:00:50.501566+0000","last_unstale":"2024-09-17T04:00:52.475673+0000","last_undegraded":"2024-09-17T04:00:52.475673+0000","last_fullsized":"2024-09-17T04:00:52.475673+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.084351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491568+0000","last_change":"2024-09-17T04:00:49.487399+0000","last_active":"2024-09-17T04:00:49.491568+0000","last_peered":"2024-09-17T04:00:49.491568+0000","last_clean":"2024-09-17T04:00:49.491568+0000","last_became_active":"2024-09-17T04:00:49.487044+0000","last_became_peered":"2024-09-17T04:00:49.487044+0000","last_unstale":"2024-09-17T04:00:49.491568+0000","last_undegraded":"2024-09-17T04:00:49.491568+0000","last_fullsized":"2024-09-17T04:00:49.491568+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:10.745031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490623+0000","last_change":"2024-09-17T04:00:50.500587+0000","last_active":"2024-09-17T04:00:51.490623+0000","last_peered":"2024-09-17T04:00:51.490623+0000","last_clean":"2024-09-17T04:00:51.490623+0000","last_became_active":"2024-09-17T04:00:50.500282+0000","last_became_peered":"2024-09-17T04:00:50.500282+0000","last_unstale":"2024-09-17T04:00:51.490623+0000","last_undegraded":"2024-09-17T04:00:51.490623+0000","last_fullsized":"2024-09-17T04:00:51.490623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:49:57.370683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"173'6","reported_seq":21,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475785+0000","last_change":"2024-09-17T04:00:49.491252+0000","last_active":"2024-09-17T04:00:52.475785+0000","last_peered":"2024-09-17T04:00:52.475785+0000","last_clean":"2024-09-17T04:00:52.475785+0000","last_became_active":"2024-09-17T04:00:49.490892+0000","last_became_peered":"2024-09-17T04:00:49.490892+0000","last_unstale":"2024-09-17T04:00:52.475785+0000","last_undegraded":"2024-09-17T04:00:52.475785+0000","last_fullsized":"2024-09-17T04:00:52.475785+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:02.780252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473556+0000","last_change":"2024-09-17T04:00:49.473556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473556+0000","last_undegraded":"2024-09-17T04:00:49.473556+0000","last_fullsized":"2024-09-17T04:00:49.473556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558379+0000","last_change":"2024-09-17T04:00:49.488763+0000","last_active":"2024-09-17T04:00:52.558379+0000","last_peered":"2024-09-17T04:00:52.558379+0000","last_clean":"2024-09-17T04:00:52.558379+0000","last_became_active":"2024-09-17T04:00:49.487670+0000","last_became_peered":"2024-09-17T04:00:49.487670+0000","last_unstale":"2024-09-17T04:00:52.558379+0000","last_undegraded":"2024-09-17T04:00:52.558379+0000","last_fullsized":"2024-09-17T04:00:52.558379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.500556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471211+0000","last_change":"2024-09-17T04:00:49.471211+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471211+0000","last_undegraded":"2024-09-17T04:00:49.471211+0000","last_fullsized":"2024-09-17T04:00:49.471211+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558318+0000","last_change":"2024-09-17T04:00:49.492017+0000","last_active":"2024-09-17T04:00:52.558318+0000","last_peered":"2024-09-17T04:00:52.558318+0000","last_clean":"2024-09-17T04:00:52.558318+0000","last_became_active":"2024-09-17T04:00:49.491561+0000","last_became_peered":"2024-09-17T04:00:49.491561+0000","last_unstale":"2024-09-17T04:00:52.558318+0000","last_undegraded":"2024-09-17T04:00:52.558318+0000","last_fullsized":"2024-09-17T04:00:52.558318+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467076+0000","last_change":"2024-09-17T04:00:49.467076+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467076+0000","last_undegraded":"2024-09-17T04:00:49.467076+0000","last_fullsized":"2024-09-17T04:00:49.467076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490472+0000","last_change":"2024-09-17T04:00:49.482396+0000","last_active":"2024-09-17T04:00:49.490472+0000","last_peered":"2024-09-17T04:00:49.490472+0000","last_clean":"2024-09-17T04:00:49.490472+0000","last_became_active":"2024-09-17T04:00:49.481991+0000","last_became_peered":"2024-09-17T04:00:49.481991+0000","last_unstale":"2024-09-17T04:00:49.490472+0000","last_undegraded":"2024-09-17T04:00:49.490472+0000","last_fullsized":"2024-09-17T04:00:49.490472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:15.031783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490625+0000","last_change":"2024-09-17T04:00:50.486171+0000","last_active":"2024-09-17T04:00:51.490625+0000","last_peered":"2024-09-17T04:00:51.490625+0000","last_clean":"2024-09-17T04:00:51.490625+0000","last_became_active":"2024-09-17T04:00:50.484422+0000","last_became_peered":"2024-09-17T04:00:50.484422+0000","last_unstale":"2024-09-17T04:00:51.490625+0000","last_undegraded":"2024-09-17T04:00:51.490625+0000","last_fullsized":"2024-09-17T04:00:51.490625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:09.236326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491600+0000","last_change":"2024-09-17T04:00:49.486848+0000","last_active":"2024-09-17T04:00:49.491600+0000","last_peered":"2024-09-17T04:00:49.491600+0000","last_clean":"2024-09-17T04:00:49.491600+0000","last_became_active":"2024-09-17T04:00:49.486463+0000","last_became_peered":"2024-09-17T04:00:49.486463+0000","last_unstale":"2024-09-17T04:00:49.491600+0000","last_undegraded":"2024-09-17T04:00:49.491600+0000","last_fullsized":"2024-09-17T04:00:49.491600+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:50.459427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493377+0000","last_change":"2024-09-17T04:00:50.495171+0000","last_active":"2024-09-17T04:00:51.493377+0000","last_peered":"2024-09-17T04:00:51.493377+0000","last_clean":"2024-09-17T04:00:51.493377+0000","last_became_active":"2024-09-17T04:00:50.494834+0000","last_became_peered":"2024-09-17T04:00:50.494834+0000","last_unstale":"2024-09-17T04:00:51.493377+0000","last_undegraded":"2024-09-17T04:00:51.493377+0000","last_fullsized":"2024-09-17T04:00:51.493377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:22.013808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.492419+0000","last_change":"2024-09-17T04:00:49.487492+0000","last_active":"2024-09-17T04:00:49.492419+0000","last_peered":"2024-09-17T04:00:49.492419+0000","last_clean":"2024-09-17T04:00:49.492419+0000","last_became_active":"2024-09-17T04:00:49.487146+0000","last_became_peered":"2024-09-17T04:00:49.487146+0000","last_unstale":"2024-09-17T04:00:49.492419+0000","last_undegraded":"2024-09-17T04:00:49.492419+0000","last_fullsized":"2024-09-17T04:00:49.492419+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493310+0000","last_change":"2024-09-17T04:00:50.484922+0000","last_active":"2024-09-17T04:00:51.493310+0000","last_peered":"2024-09-17T04:00:51.493310+0000","last_clean":"2024-09-17T04:00:51.493310+0000","last_became_active":"2024-09-17T04:00:50.483724+0000","last_became_peered":"2024-09-17T04:00:50.483724+0000","last_unstale":"2024-09-17T04:00:51.493310+0000","last_undegraded":"2024-09-17T04:00:51.493310+0000","last_fullsized":"2024-09-17T04:00:51.493310+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:40.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493354+0000","last_change":"2024-09-17T04:00:49.475839+0000","last_active":"2024-09-17T04:00:51.493354+0000","last_peered":"2024-09-17T04:00:51.493354+0000","last_clean":"2024-09-17T04:00:51.493354+0000","last_became_active":"2024-09-17T04:00:49.475239+0000","last_became_peered":"2024-09-17T04:00:49.475239+0000","last_unstale":"2024-09-17T04:00:51.493354+0000","last_undegraded":"2024-09-17T04:00:51.493354+0000","last_fullsized":"2024-09-17T04:00:51.493354+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:14.501945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467772+0000","last_change":"2024-09-17T04:00:49.467772+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467772+0000","last_undegraded":"2024-09-17T04:00:49.467772+0000","last_fullsized":"2024-09-17T04:00:49.467772+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491597+0000","last_change":"2024-09-17T04:00:49.485542+0000","last_active":"2024-09-17T04:00:49.491597+0000","last_peered":"2024-09-17T04:00:49.491597+0000","last_clean":"2024-09-17T04:00:49.491597+0000","last_became_active":"2024-09-17T04:00:49.481811+0000","last_became_peered":"2024-09-17T04:00:49.481811+0000","last_unstale":"2024-09-17T04:00:49.491597+0000","last_undegraded":"2024-09-17T04:00:49.491597+0000","last_fullsized":"2024-09-17T04:00:49.491597+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.533653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472723+0000","last_change":"2024-09-17T04:00:49.472723+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472723+0000","last_undegraded":"2024-09-17T04:00:49.472723+0000","last_fullsized":"2024-09-17T04:00:49.472723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490994+0000","last_change":"2024-09-17T04:00:49.485281+0000","last_active":"2024-09-17T04:00:49.490994+0000","last_peered":"2024-09-17T04:00:49.490994+0000","last_clean":"2024-09-17T04:00:49.490994+0000","last_became_active":"2024-09-17T04:00:49.484098+0000","last_became_peered":"2024-09-17T04:00:49.484098+0000","last_unstale":"2024-09-17T04:00:49.490994+0000","last_undegraded":"2024-09-17T04:00:49.490994+0000","last_fullsized":"2024-09-17T04:00:49.490994+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.496374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471100+0000","last_change":"2024-09-17T04:00:49.471100+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471100+0000","last_undegraded":"2024-09-17T04:00:49.471100+0000","last_fullsized":"2024-09-17T04:00:49.471100+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485054+0000","last_change":"2024-09-17T04:00:49.479444+0000","last_active":"2024-09-17T04:00:49.485054+0000","last_peered":"2024-09-17T04:00:49.485054+0000","last_clean":"2024-09-17T04:00:49.485054+0000","last_became_active":"2024-09-17T04:00:49.479095+0000","last_became_peered":"2024-09-17T04:00:49.479095+0000","last_unstale":"2024-09-17T04:00:49.485054+0000","last_undegraded":"2024-09-17T04:00:49.485054+0000","last_fullsized":"2024-09-17T04:00:49.485054+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:09.201712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468009+0000","last_change":"2024-09-17T04:00:49.468009+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468009+0000","last_undegraded":"2024-09-17T04:00:49.468009+0000","last_fullsized":"2024-09-17T04:00:49.468009+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490347+0000","last_change":"2024-09-17T04:00:49.492248+0000","last_active":"2024-09-17T04:00:51.490347+0000","last_peered":"2024-09-17T04:00:51.490347+0000","last_clean":"2024-09-17T04:00:51.490347+0000","last_became_active":"2024-09-17T04:00:49.491744+0000","last_became_peered":"2024-09-17T04:00:49.491744+0000","last_unstale":"2024-09-17T04:00:51.490347+0000","last_undegraded":"2024-09-17T04:00:51.490347+0000","last_fullsized":"2024-09-17T04:00:51.490347+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:45.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492551+0000","last_change":"2024-09-17T04:00:50.485354+0000","last_active":"2024-09-17T04:00:51.492551+0000","last_peered":"2024-09-17T04:00:51.492551+0000","last_clean":"2024-09-17T04:00:51.492551+0000","last_became_active":"2024-09-17T04:00:50.485017+0000","last_became_peered":"2024-09-17T04:00:50.485017+0000","last_unstale":"2024-09-17T04:00:51.492551+0000","last_undegraded":"2024-09-17T04:00:51.492551+0000","last_fullsized":"2024-09-17T04:00:51.492551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:43.479866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490209+0000","last_change":"2024-09-17T04:00:49.490198+0000","last_active":"2024-09-17T04:00:51.490209+0000","last_peered":"2024-09-17T04:00:51.490209+0000","last_clean":"2024-09-17T04:00:51.490209+0000","last_became_active":"2024-09-17T04:00:49.489685+0000","last_became_peered":"2024-09-17T04:00:49.489685+0000","last_unstale":"2024-09-17T04:00:51.490209+0000","last_undegraded":"2024-09-17T04:00:51.490209+0000","last_fullsized":"2024-09-17T04:00:51.490209+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:21.987330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558604+0000","last_change":"2024-09-17T04:00:50.488092+0000","last_active":"2024-09-17T04:00:52.558604+0000","last_peered":"2024-09-17T04:00:52.558604+0000","last_clean":"2024-09-17T04:00:52.558604+0000","last_became_active":"2024-09-17T04:00:50.487718+0000","last_became_peered":"2024-09-17T04:00:50.487718+0000","last_unstale":"2024-09-17T04:00:52.558604+0000","last_undegraded":"2024-09-17T04:00:52.558604+0000","last_fullsized":"2024-09-17T04:00:52.558604+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:12.457060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.494752+0000","last_change":"2024-09-17T04:00:49.492610+0000","last_active":"2024-09-17T04:00:49.494752+0000","last_peered":"2024-09-17T04:00:49.494752+0000","last_clean":"2024-09-17T04:00:49.494752+0000","last_became_active":"2024-09-17T04:00:49.491506+0000","last_became_peered":"2024-09-17T04:00:49.491506+0000","last_unstale":"2024-09-17T04:00:49.494752+0000","last_undegraded":"2024-09-17T04:00:49.494752+0000","last_fullsized":"2024-09-17T04:00:49.494752+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:53.116814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954545+0000","last_change":"2024-09-17T04:00:50.511428+0000","last_active":"2024-09-17T04:00:51.954545+0000","last_peered":"2024-09-17T04:00:51.954545+0000","last_clean":"2024-09-17T04:00:51.954545+0000","last_became_active":"2024-09-17T04:00:50.511114+0000","last_became_peered":"2024-09-17T04:00:50.511114+0000","last_unstale":"2024-09-17T04:00:51.954545+0000","last_undegraded":"2024-09-17T04:00:51.954545+0000","last_fullsized":"2024-09-17T04:00:51.954545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:58.184294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485901+0000","last_change":"2024-09-17T04:00:49.478561+0000","last_active":"2024-09-17T04:00:49.485901+0000","last_peered":"2024-09-17T04:00:49.485901+0000","last_clean":"2024-09-17T04:00:49.485901+0000","last_became_active":"2024-09-17T04:00:49.478248+0000","last_became_peered":"2024-09-17T04:00:49.478248+0000","last_unstale":"2024-09-17T04:00:49.485901+0000","last_undegraded":"2024-09-17T04:00:49.485901+0000","last_fullsized":"2024-09-17T04:00:49.485901+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:13.907873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490983+0000","last_change":"2024-09-17T04:00:50.495573+0000","last_active":"2024-09-17T04:00:51.490983+0000","last_peered":"2024-09-17T04:00:51.490983+0000","last_clean":"2024-09-17T04:00:51.490983+0000","last_became_active":"2024-09-17T04:00:50.494895+0000","last_became_peered":"2024-09-17T04:00:50.494895+0000","last_unstale":"2024-09-17T04:00:51.490983+0000","last_undegraded":"2024-09-17T04:00:51.490983+0000","last_fullsized":"2024-09-17T04:00:51.490983+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:07.480340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"174'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475720+0000","last_change":"2024-09-17T04:00:49.490303+0000","last_active":"2024-09-17T04:00:52.475720+0000","last_peered":"2024-09-17T04:00:52.475720+0000","last_clean":"2024-09-17T04:00:52.475720+0000","last_became_active":"2024-09-17T04:00:49.489882+0000","last_became_peered":"2024-09-17T04:00:49.489882+0000","last_unstale":"2024-09-17T04:00:52.475720+0000","last_undegraded":"2024-09-17T04:00:52.475720+0000","last_fullsized":"2024-09-17T04:00:52.475720+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:46.567261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492851+0000","last_change":"2024-09-17T04:00:50.493244+0000","last_active":"2024-09-17T04:00:51.492851+0000","last_peered":"2024-09-17T04:00:51.492851+0000","last_clean":"2024-09-17T04:00:51.492851+0000","last_became_active":"2024-09-17T04:00:50.492892+0000","last_became_peered":"2024-09-17T04:00:50.492892+0000","last_unstale":"2024-09-17T04:00:51.492851+0000","last_undegraded":"2024-09-17T04:00:51.492851+0000","last_fullsized":"2024-09-17T04:00:51.492851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:19.515809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.484869+0000","last_change":"2024-09-17T04:00:49.480983+0000","last_active":"2024-09-17T04:00:49.484869+0000","last_peered":"2024-09-17T04:00:49.484869+0000","last_clean":"2024-09-17T04:00:49.484869+0000","last_became_active":"2024-09-17T04:00:49.480620+0000","last_became_peered":"2024-09-17T04:00:49.480620+0000","last_unstale":"2024-09-17T04:00:49.484869+0000","last_undegraded":"2024-09-17T04:00:49.484869+0000","last_fullsized":"2024-09-17T04:00:49.484869+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:17.334558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558754+0000","last_change":"2024-09-17T04:00:50.491601+0000","last_active":"2024-09-17T04:00:52.558754+0000","last_peered":"2024-09-17T04:00:52.558754+0000","last_clean":"2024-09-17T04:00:52.558754+0000","last_became_active":"2024-09-17T04:00:50.491279+0000","last_became_peered":"2024-09-17T04:00:50.491279+0000","last_unstale":"2024-09-17T04:00:52.558754+0000","last_undegraded":"2024-09-17T04:00:52.558754+0000","last_fullsized":"2024-09-17T04:00:52.558754+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:03.782918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.482446+0000","last_change":"2024-09-17T04:00:49.477561+0000","last_active":"2024-09-17T04:00:49.482446+0000","last_peered":"2024-09-17T04:00:49.482446+0000","last_clean":"2024-09-17T04:00:49.482446+0000","last_became_active":"2024-09-17T04:00:49.477252+0000","last_became_peered":"2024-09-17T04:00:49.477252+0000","last_unstale":"2024-09-17T04:00:49.482446+0000","last_undegraded":"2024-09-17T04:00:49.482446+0000","last_fullsized":"2024-09-17T04:00:49.482446+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:14.568584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490348+0000","last_change":"2024-09-17T04:00:50.499748+0000","last_active":"2024-09-17T04:00:51.490348+0000","last_peered":"2024-09-17T04:00:51.490348+0000","last_clean":"2024-09-17T04:00:51.490348+0000","last_became_active":"2024-09-17T04:00:50.497933+0000","last_became_peered":"2024-09-17T04:00:50.497933+0000","last_unstale":"2024-09-17T04:00:51.490348+0000","last_undegraded":"2024-09-17T04:00:51.490348+0000","last_fullsized":"2024-09-17T04:00:51.490348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:33.887766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475476+0000","last_change":"2024-09-17T04:00:49.486586+0000","last_active":"2024-09-17T04:00:52.475476+0000","last_peered":"2024-09-17T04:00:52.475476+0000","last_clean":"2024-09-17T04:00:52.475476+0000","last_became_active":"2024-09-17T04:00:49.486099+0000","last_became_peered":"2024-09-17T04:00:49.486099+0000","last_unstale":"2024-09-17T04:00:52.475476+0000","last_undegraded":"2024-09-17T04:00:52.475476+0000","last_fullsized":"2024-09-17T04:00:52.475476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:42.671635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954355+0000","last_change":"2024-09-17T04:00:50.501002+0000","last_active":"2024-09-17T04:00:51.954355+0000","last_peered":"2024-09-17T04:00:51.954355+0000","last_clean":"2024-09-17T04:00:51.954355+0000","last_became_active":"2024-09-17T04:00:50.500621+0000","last_became_peered":"2024-09-17T04:00:50.500621+0000","last_unstale":"2024-09-17T04:00:51.954355+0000","last_undegraded":"2024-09-17T04:00:51.954355+0000","last_fullsized":"2024-09-17T04:00:51.954355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:38.586287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486445+0000","last_change":"2024-09-17T04:00:49.483997+0000","last_active":"2024-09-17T04:00:49.486445+0000","last_peered":"2024-09-17T04:00:49.486445+0000","last_clean":"2024-09-17T04:00:49.486445+0000","last_became_active":"2024-09-17T04:00:49.483458+0000","last_became_peered":"2024-09-17T04:00:49.483458+0000","last_unstale":"2024-09-17T04:00:49.486445+0000","last_undegraded":"2024-09-17T04:00:49.486445+0000","last_fullsized":"2024-09-17T04:00:49.486445+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:10:19.789984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.466540+0000","last_change":"2024-09-17T04:00:49.466540+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.466540+0000","last_undegraded":"2024-09-17T04:00:49.466540+0000","last_fullsized":"2024-09-17T04:00:49.466540+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558691+0000","last_change":"2024-09-17T04:00:49.485410+0000","last_active":"2024-09-17T04:00:52.558691+0000","last_peered":"2024-09-17T04:00:52.558691+0000","last_clean":"2024-09-17T04:00:52.558691+0000","last_became_active":"2024-09-17T04:00:49.484171+0000","last_became_peered":"2024-09-17T04:00:49.484171+0000","last_unstale":"2024-09-17T04:00:52.558691+0000","last_undegraded":"2024-09-17T04:00:52.558691+0000","last_fullsized":"2024-09-17T04:00:52.558691+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492593+0000","last_change":"2024-09-17T04:00:50.490365+0000","last_active":"2024-09-17T04:00:51.492593+0000","last_peered":"2024-09-17T04:00:51.492593+0000","last_clean":"2024-09-17T04:00:51.492593+0000","last_became_active":"2024-09-17T04:00:50.488098+0000","last_became_peered":"2024-09-17T04:00:50.488098+0000","last_unstale":"2024-09-17T04:00:51.492593+0000","last_undegraded":"2024-09-17T04:00:51.492593+0000","last_fullsized":"2024-09-17T04:00:51.492593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:42.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954116+0000","last_change":"2024-09-17T04:00:49.487212+0000","last_active":"2024-09-17T04:00:51.954116+0000","last_peered":"2024-09-17T04:00:51.954116+0000","last_clean":"2024-09-17T04:00:51.954116+0000","last_became_active":"2024-09-17T04:00:49.486849+0000","last_became_peered":"2024-09-17T04:00:49.486849+0000","last_unstale":"2024-09-17T04:00:51.954116+0000","last_undegraded":"2024-09-17T04:00:51.954116+0000","last_fullsized":"2024-09-17T04:00:51.954116+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:31.908526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490138+0000","last_change":"2024-09-17T04:00:50.501206+0000","last_active":"2024-09-17T04:00:51.490138+0000","last_peered":"2024-09-17T04:00:51.490138+0000","last_clean":"2024-09-17T04:00:51.490138+0000","last_became_active":"2024-09-17T04:00:50.500932+0000","last_became_peered":"2024-09-17T04:00:50.500932+0000","last_unstale":"2024-09-17T04:00:51.490138+0000","last_undegraded":"2024-09-17T04:00:51.490138+0000","last_fullsized":"2024-09-17T04:00:51.490138+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:32.987666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954611+0000","last_change":"2024-09-17T04:00:49.489179+0000","last_active":"2024-09-17T04:00:51.954611+0000","last_peered":"2024-09-17T04:00:51.954611+0000","last_clean":"2024-09-17T04:00:51.954611+0000","last_became_active":"2024-09-17T04:00:49.488400+0000","last_became_peered":"2024-09-17T04:00:49.488400+0000","last_unstale":"2024-09-17T04:00:51.954611+0000","last_undegraded":"2024-09-17T04:00:51.954611+0000","last_fullsized":"2024-09-17T04:00:51.954611+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:44:31.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":392,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475314+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:52.475314+0000","last_peered":"2024-09-17T04:00:52.475314+0000","last_clean":"2024-09-17T04:00:52.475314+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:52.475314+0000","last_undegraded":"2024-09-17T04:00:52.475314+0000","last_fullsized":"2024-09-17T04:00:52.475314+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509580,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2793472,"data_stored":1549678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2207744,"data_stored":959414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509579,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542285,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2785280,"data_stored":1548156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:55.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.692+0000 7f3f51400640 1 -- 172.21.15.129:0/994174013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c15a220 msgr2=0x7f3f4c15d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.692+0000 7f3f51400640 1 --2- 172.21.15.129:0/994174013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c15a220 0x7f3f4c15d660 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005fd0 tx=0x7f3f3c02a540 comp rx=0 tx=0).stop 2024-09-17T04:00:55.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 -- 172.21.15.129:0/994174013 shutdown_connections 2024-09-17T04:00:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 --2- 172.21.15.129:0/994174013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c0cf530 0x7f3f4c0cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 --2- 172.21.15.129:0/994174013 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c15a220 0x7f3f4c15d660 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 -- 172.21.15.129:0/994174013 >> 172.21.15.129:0/994174013 conn(0x7f3f4c069150 msgr2=0x7f3f4c069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 -- 172.21.15.129:0/994174013 shutdown_connections 2024-09-17T04:00:55.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.693+0000 7f3f51400640 1 -- 172.21.15.129:0/994174013 wait complete. 2024-09-17T04:00:55.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.695+0000 7f3f51400640 1 Processor -- start 2024-09-17T04:00:55.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.695+0000 7f3f51400640 1 -- start start 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c15a220 0x7f3f4c0b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c151b30 0x7f3f4c1f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c0d0490 con 0x7f3f4c1602b0 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4c1608d0 con 0x7f3f4c151b30 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.696+0000 7f3f51400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4c0d0610 con 0x7f3f4c15a220 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c15a220 0x7f3f4c0b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35170/0 (socket says 172.21.15.129:35170) 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 -- 172.21.15.129:0/1883691535 learned_addr learned my addr 172.21.15.129:0/1883691535 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f50a00640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c151b30 0x7f3f4c1f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 -- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c15a220 msgr2=0x7f3f4c0b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c15a220 0x7f3f4c0b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 -- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c151b30 msgr2=0x7f3f4c1f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c151b30 0x7f3f4c1f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 -- 172.21.15.129:0/1883691535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4c0c9810 con 0x7f3f4c1602b0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.697+0000 7f3f4b400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800b3c0 tx=0x7f3f3800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.698+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38015020 con 0x7f3f4c1602b0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.698+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f38011920 con 0x7f3f4c1602b0 2024-09-17T04:00:55.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.698+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38011d90 con 0x7f3f4c1602b0 2024-09-17T04:00:55.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.698+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4c0c9af0 con 0x7f3f4c1602b0 2024-09-17T04:00:55.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.698+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4c0ca0b0 con 0x7f3f4c1602b0 2024-09-17T04:00:55.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.699+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f4c0cff50 con 0x7f3f4c1602b0 2024-09-17T04:00:55.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.700+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f38018020 con 0x7f3f4c1602b0 2024-09-17T04:00:55.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.701+0000 7f3f43e00640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 0x7f3f2c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:55.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.701+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f3f3809aa20 con 0x7f3f4c1602b0 2024-09-17T04:00:55.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.702+0000 7f3f4be00640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 0x7f3f2c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:55.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.702+0000 7f3f4be00640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 0x7f3f2c07b0a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c033000 tx=0x7f3f3c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:55.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.706+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3f3801f020 con 0x7f3f4c1602b0 2024-09-17T04:00:55.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.886+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f4c0b6190 con 0x7f3f2c078b90 2024-09-17T04:00:55.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.891+0000 7f3f43e00640 1 -- 172.21.15.129:0/1883691535 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250551 (secure 0 0 0) 0x7f3f3c0047a0 con 0x7f3f2c078b90 2024-09-17T04:00:55.894 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.895+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 msgr2=0x7f3f2c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.895+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 0x7f3f2c07b0a0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c033000 tx=0x7f3f3c004730 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.895+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 msgr2=0x7f3f4c0ca8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.895+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800b3c0 tx=0x7f3f3800b880 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 shutdown_connections 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f2c078b90 0x7f3f2c07b0a0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f4c151b30 0x7f3f4c1f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f4c1602b0 0x7f3f4c0ca8b0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 --2- 172.21.15.129:0/1883691535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f4c15a220 0x7f3f4c0b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 >> 172.21.15.129:0/1883691535 conn(0x7f3f4c069150 msgr2=0x7f3f4c15e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:55.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 shutdown_connections 2024-09-17T04:00:55.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:55.896+0000 7f3f51400640 1 -- 172.21.15.129:0/1883691535 wait complete. 2024-09-17T04:00:55.921 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-17T04:00:54.487043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592596,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":121,"num_write_kb":1639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":325948,"kb_used_data":18184,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754648772,"statfs":{"total":773094113280,"available":772760342528,"internally_reserved":0,"allocated":18620416,"data_stored":8788852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002410"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490698+0000","last_change":"2024-09-17T04:00:50.499023+0000","last_active":"2024-09-17T04:00:51.490698+0000","last_peered":"2024-09-17T04:00:51.490698+0000","last_clean":"2024-09-17T04:00:51.490698+0000","last_became_active":"2024-09-17T04:00:50.498702+0000","last_became_peered":"2024-09-17T04:00:50.498702+0000","last_unstale":"2024-09-17T04:00:51.490698+0000","last_undegraded":"2024-09-17T04:00:51.490698+0000","last_fullsized":"2024-09-17T04:00:51.490698+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:47.660122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493082+0000","last_change":"2024-09-17T04:00:50.481292+0000","last_active":"2024-09-17T04:00:51.493082+0000","last_peered":"2024-09-17T04:00:51.493082+0000","last_clean":"2024-09-17T04:00:51.493082+0000","last_became_active":"2024-09-17T04:00:50.480948+0000","last_became_peered":"2024-09-17T04:00:50.480948+0000","last_unstale":"2024-09-17T04:00:51.493082+0000","last_undegraded":"2024-09-17T04:00:51.493082+0000","last_fullsized":"2024-09-17T04:00:51.493082+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.354995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470097+0000","last_change":"2024-09-17T04:00:49.470097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470097+0000","last_undegraded":"2024-09-17T04:00:49.470097+0000","last_fullsized":"2024-09-17T04:00:49.470097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475438+0000","last_change":"2024-09-17T04:00:50.500751+0000","last_active":"2024-09-17T04:00:52.475438+0000","last_peered":"2024-09-17T04:00:52.475438+0000","last_clean":"2024-09-17T04:00:52.475438+0000","last_became_active":"2024-09-17T04:00:50.500022+0000","last_became_peered":"2024-09-17T04:00:50.500022+0000","last_unstale":"2024-09-17T04:00:52.475438+0000","last_undegraded":"2024-09-17T04:00:52.475438+0000","last_fullsized":"2024-09-17T04:00:52.475438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:04.192066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472481+0000","last_change":"2024-09-17T04:00:49.472481+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472481+0000","last_undegraded":"2024-09-17T04:00:49.472481+0000","last_fullsized":"2024-09-17T04:00:49.472481+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474000+0000","last_change":"2024-09-17T04:00:49.474000+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474000+0000","last_undegraded":"2024-09-17T04:00:49.474000+0000","last_fullsized":"2024-09-17T04:00:49.474000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490063+0000","last_change":"2024-09-17T04:00:50.488121+0000","last_active":"2024-09-17T04:00:51.490063+0000","last_peered":"2024-09-17T04:00:51.490063+0000","last_clean":"2024-09-17T04:00:51.490063+0000","last_became_active":"2024-09-17T04:00:50.487823+0000","last_became_peered":"2024-09-17T04:00:50.487823+0000","last_unstale":"2024-09-17T04:00:51.490063+0000","last_undegraded":"2024-09-17T04:00:51.490063+0000","last_fullsized":"2024-09-17T04:00:51.490063+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:27.505920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472807+0000","last_change":"2024-09-17T04:00:49.472807+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472807+0000","last_undegraded":"2024-09-17T04:00:49.472807+0000","last_fullsized":"2024-09-17T04:00:49.472807+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493544+0000","last_change":"2024-09-17T04:00:50.494343+0000","last_active":"2024-09-17T04:00:51.493544+0000","last_peered":"2024-09-17T04:00:51.493544+0000","last_clean":"2024-09-17T04:00:51.493544+0000","last_became_active":"2024-09-17T04:00:50.493022+0000","last_became_peered":"2024-09-17T04:00:50.493022+0000","last_unstale":"2024-09-17T04:00:51.493544+0000","last_undegraded":"2024-09-17T04:00:51.493544+0000","last_fullsized":"2024-09-17T04:00:51.493544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:20.950951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474579+0000","last_change":"2024-09-17T04:00:49.474579+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474579+0000","last_undegraded":"2024-09-17T04:00:49.474579+0000","last_fullsized":"2024-09-17T04:00:49.474579+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468304+0000","last_change":"2024-09-17T04:00:49.468304+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468304+0000","last_undegraded":"2024-09-17T04:00:49.468304+0000","last_fullsized":"2024-09-17T04:00:49.468304+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954479+0000","last_change":"2024-09-17T04:00:50.503620+0000","last_active":"2024-09-17T04:00:51.954479+0000","last_peered":"2024-09-17T04:00:51.954479+0000","last_clean":"2024-09-17T04:00:51.954479+0000","last_became_active":"2024-09-17T04:00:50.503238+0000","last_became_peered":"2024-09-17T04:00:50.503238+0000","last_unstale":"2024-09-17T04:00:51.954479+0000","last_undegraded":"2024-09-17T04:00:51.954479+0000","last_fullsized":"2024-09-17T04:00:51.954479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:10.261218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954637+0000","last_change":"2024-09-17T04:00:50.502335+0000","last_active":"2024-09-17T04:00:51.954637+0000","last_peered":"2024-09-17T04:00:51.954637+0000","last_clean":"2024-09-17T04:00:51.954637+0000","last_became_active":"2024-09-17T04:00:50.502036+0000","last_became_peered":"2024-09-17T04:00:50.502036+0000","last_unstale":"2024-09-17T04:00:51.954637+0000","last_undegraded":"2024-09-17T04:00:51.954637+0000","last_fullsized":"2024-09-17T04:00:51.954637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:37.502247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490230+0000","last_change":"2024-09-17T04:00:50.501289+0000","last_active":"2024-09-17T04:00:51.490230+0000","last_peered":"2024-09-17T04:00:51.490230+0000","last_clean":"2024-09-17T04:00:51.490230+0000","last_became_active":"2024-09-17T04:00:50.501012+0000","last_became_peered":"2024-09-17T04:00:50.501012+0000","last_unstale":"2024-09-17T04:00:51.490230+0000","last_undegraded":"2024-09-17T04:00:51.490230+0000","last_fullsized":"2024-09-17T04:00:51.490230+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:36.532175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490074+0000","last_change":"2024-09-17T04:00:50.490115+0000","last_active":"2024-09-17T04:00:51.490074+0000","last_peered":"2024-09-17T04:00:51.490074+0000","last_clean":"2024-09-17T04:00:51.490074+0000","last_became_active":"2024-09-17T04:00:50.489826+0000","last_became_peered":"2024-09-17T04:00:50.489826+0000","last_unstale":"2024-09-17T04:00:51.490074+0000","last_undegraded":"2024-09-17T04:00:51.490074+0000","last_fullsized":"2024-09-17T04:00:51.490074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:11.406297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.475124+0000","last_change":"2024-09-17T04:00:49.475124+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.475124+0000","last_undegraded":"2024-09-17T04:00:49.475124+0000","last_fullsized":"2024-09-17T04:00:49.475124+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490971+0000","last_change":"2024-09-17T04:00:50.497620+0000","last_active":"2024-09-17T04:00:51.490971+0000","last_peered":"2024-09-17T04:00:51.490971+0000","last_clean":"2024-09-17T04:00:51.490971+0000","last_became_active":"2024-09-17T04:00:50.495270+0000","last_became_peered":"2024-09-17T04:00:50.495270+0000","last_unstale":"2024-09-17T04:00:51.490971+0000","last_undegraded":"2024-09-17T04:00:51.490971+0000","last_fullsized":"2024-09-17T04:00:51.490971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:29.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.474097+0000","last_change":"2024-09-17T04:00:49.474097+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.474097+0000","last_undegraded":"2024-09-17T04:00:49.474097+0000","last_fullsized":"2024-09-17T04:00:49.474097+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476999+0000","last_change":"2024-09-17T04:00:49.476999+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476999+0000","last_undegraded":"2024-09-17T04:00:49.476999+0000","last_fullsized":"2024-09-17T04:00:49.476999+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473627+0000","last_change":"2024-09-17T04:00:49.473627+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473627+0000","last_undegraded":"2024-09-17T04:00:49.473627+0000","last_fullsized":"2024-09-17T04:00:49.473627+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475406+0000","last_change":"2024-09-17T04:00:50.499009+0000","last_active":"2024-09-17T04:00:52.475406+0000","last_peered":"2024-09-17T04:00:52.475406+0000","last_clean":"2024-09-17T04:00:52.475406+0000","last_became_active":"2024-09-17T04:00:50.495348+0000","last_became_peered":"2024-09-17T04:00:50.495348+0000","last_unstale":"2024-09-17T04:00:52.475406+0000","last_undegraded":"2024-09-17T04:00:52.475406+0000","last_fullsized":"2024-09-17T04:00:52.475406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:38.854990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558507+0000","last_change":"2024-09-17T04:00:50.483447+0000","last_active":"2024-09-17T04:00:52.558507+0000","last_peered":"2024-09-17T04:00:52.558507+0000","last_clean":"2024-09-17T04:00:52.558507+0000","last_became_active":"2024-09-17T04:00:50.483105+0000","last_became_peered":"2024-09-17T04:00:50.483105+0000","last_unstale":"2024-09-17T04:00:52.558507+0000","last_undegraded":"2024-09-17T04:00:52.558507+0000","last_fullsized":"2024-09-17T04:00:52.558507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:22.999066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558726+0000","last_change":"2024-09-17T04:00:50.488020+0000","last_active":"2024-09-17T04:00:52.558726+0000","last_peered":"2024-09-17T04:00:52.558726+0000","last_clean":"2024-09-17T04:00:52.558726+0000","last_became_active":"2024-09-17T04:00:50.487293+0000","last_became_peered":"2024-09-17T04:00:50.487293+0000","last_unstale":"2024-09-17T04:00:52.558726+0000","last_undegraded":"2024-09-17T04:00:52.558726+0000","last_fullsized":"2024-09-17T04:00:52.558726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:15.011545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475667+0000","last_change":"2024-09-17T04:00:50.493247+0000","last_active":"2024-09-17T04:00:52.475667+0000","last_peered":"2024-09-17T04:00:52.475667+0000","last_clean":"2024-09-17T04:00:52.475667+0000","last_became_active":"2024-09-17T04:00:50.492907+0000","last_became_peered":"2024-09-17T04:00:50.492907+0000","last_unstale":"2024-09-17T04:00:52.475667+0000","last_undegraded":"2024-09-17T04:00:52.475667+0000","last_fullsized":"2024-09-17T04:00:52.475667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:58.410921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493295+0000","last_change":"2024-09-17T04:00:50.487190+0000","last_active":"2024-09-17T04:00:51.493295+0000","last_peered":"2024-09-17T04:00:51.493295+0000","last_clean":"2024-09-17T04:00:51.493295+0000","last_became_active":"2024-09-17T04:00:50.484107+0000","last_became_peered":"2024-09-17T04:00:50.484107+0000","last_unstale":"2024-09-17T04:00:51.493295+0000","last_undegraded":"2024-09-17T04:00:51.493295+0000","last_fullsized":"2024-09-17T04:00:51.493295+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:17.052980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490680+0000","last_change":"2024-09-17T04:00:50.500005+0000","last_active":"2024-09-17T04:00:51.490680+0000","last_peered":"2024-09-17T04:00:51.490680+0000","last_clean":"2024-09-17T04:00:51.490680+0000","last_became_active":"2024-09-17T04:00:50.497315+0000","last_became_peered":"2024-09-17T04:00:50.497315+0000","last_unstale":"2024-09-17T04:00:51.490680+0000","last_undegraded":"2024-09-17T04:00:51.490680+0000","last_fullsized":"2024-09-17T04:00:51.490680+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:50.074066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954424+0000","last_change":"2024-09-17T04:00:50.507420+0000","last_active":"2024-09-17T04:00:51.954424+0000","last_peered":"2024-09-17T04:00:51.954424+0000","last_clean":"2024-09-17T04:00:51.954424+0000","last_became_active":"2024-09-17T04:00:50.506925+0000","last_became_peered":"2024-09-17T04:00:50.506925+0000","last_unstale":"2024-09-17T04:00:51.954424+0000","last_undegraded":"2024-09-17T04:00:51.954424+0000","last_fullsized":"2024-09-17T04:00:51.954424+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.177663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.470536+0000","last_change":"2024-09-17T04:00:49.470536+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.470536+0000","last_undegraded":"2024-09-17T04:00:49.470536+0000","last_fullsized":"2024-09-17T04:00:49.470536+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954415+0000","last_change":"2024-09-17T04:00:50.507336+0000","last_active":"2024-09-17T04:00:51.954415+0000","last_peered":"2024-09-17T04:00:51.954415+0000","last_clean":"2024-09-17T04:00:51.954415+0000","last_became_active":"2024-09-17T04:00:50.507000+0000","last_became_peered":"2024-09-17T04:00:50.507000+0000","last_unstale":"2024-09-17T04:00:51.954415+0000","last_undegraded":"2024-09-17T04:00:51.954415+0000","last_fullsized":"2024-09-17T04:00:51.954415+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:03.261144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492684+0000","last_change":"2024-09-17T04:00:50.508385+0000","last_active":"2024-09-17T04:00:51.492684+0000","last_peered":"2024-09-17T04:00:51.492684+0000","last_clean":"2024-09-17T04:00:51.492684+0000","last_became_active":"2024-09-17T04:00:50.507803+0000","last_became_peered":"2024-09-17T04:00:50.507803+0000","last_unstale":"2024-09-17T04:00:51.492684+0000","last_undegraded":"2024-09-17T04:00:51.492684+0000","last_fullsized":"2024-09-17T04:00:51.492684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:40.047416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473517+0000","last_change":"2024-09-17T04:00:49.473517+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473517+0000","last_undegraded":"2024-09-17T04:00:49.473517+0000","last_fullsized":"2024-09-17T04:00:49.473517+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490344+0000","last_change":"2024-09-17T04:00:50.504166+0000","last_active":"2024-09-17T04:00:51.490344+0000","last_peered":"2024-09-17T04:00:51.490344+0000","last_clean":"2024-09-17T04:00:51.490344+0000","last_became_active":"2024-09-17T04:00:50.503869+0000","last_became_peered":"2024-09-17T04:00:50.503869+0000","last_unstale":"2024-09-17T04:00:51.490344+0000","last_undegraded":"2024-09-17T04:00:51.490344+0000","last_fullsized":"2024-09-17T04:00:51.490344+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:44.335387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493116+0000","last_change":"2024-09-17T04:00:50.497167+0000","last_active":"2024-09-17T04:00:51.493116+0000","last_peered":"2024-09-17T04:00:51.493116+0000","last_clean":"2024-09-17T04:00:51.493116+0000","last_became_active":"2024-09-17T04:00:50.496798+0000","last_became_peered":"2024-09-17T04:00:50.496798+0000","last_unstale":"2024-09-17T04:00:51.493116+0000","last_undegraded":"2024-09-17T04:00:51.493116+0000","last_fullsized":"2024-09-17T04:00:51.493116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:18.337224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"173'4","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954531+0000","last_change":"2024-09-17T04:00:49.488273+0000","last_active":"2024-09-17T04:00:51.954531+0000","last_peered":"2024-09-17T04:00:51.954531+0000","last_clean":"2024-09-17T04:00:51.954531+0000","last_became_active":"2024-09-17T04:00:49.486926+0000","last_became_peered":"2024-09-17T04:00:49.486926+0000","last_unstale":"2024-09-17T04:00:51.954531+0000","last_undegraded":"2024-09-17T04:00:51.954531+0000","last_fullsized":"2024-09-17T04:00:51.954531+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:23.992360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558954+0000","last_change":"2024-09-17T04:00:50.493935+0000","last_active":"2024-09-17T04:00:52.558954+0000","last_peered":"2024-09-17T04:00:52.558954+0000","last_clean":"2024-09-17T04:00:52.558954+0000","last_became_active":"2024-09-17T04:00:50.490351+0000","last_became_peered":"2024-09-17T04:00:50.490351+0000","last_unstale":"2024-09-17T04:00:52.558954+0000","last_undegraded":"2024-09-17T04:00:52.558954+0000","last_fullsized":"2024-09-17T04:00:52.558954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:16.034449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493225+0000","last_change":"2024-09-17T04:00:49.485660+0000","last_active":"2024-09-17T04:00:51.493225+0000","last_peered":"2024-09-17T04:00:51.493225+0000","last_clean":"2024-09-17T04:00:51.493225+0000","last_became_active":"2024-09-17T04:00:49.484522+0000","last_became_peered":"2024-09-17T04:00:49.484522+0000","last_unstale":"2024-09-17T04:00:51.493225+0000","last_undegraded":"2024-09-17T04:00:51.493225+0000","last_fullsized":"2024-09-17T04:00:51.493225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:17.632102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490880+0000","last_change":"2024-09-17T04:00:50.493759+0000","last_active":"2024-09-17T04:00:51.490880+0000","last_peered":"2024-09-17T04:00:51.490880+0000","last_clean":"2024-09-17T04:00:51.490880+0000","last_became_active":"2024-09-17T04:00:50.493472+0000","last_became_peered":"2024-09-17T04:00:50.493472+0000","last_unstale":"2024-09-17T04:00:51.490880+0000","last_undegraded":"2024-09-17T04:00:51.490880+0000","last_fullsized":"2024-09-17T04:00:51.490880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:36.564425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490615+0000","last_change":"2024-09-17T04:00:49.485026+0000","last_active":"2024-09-17T04:00:49.490615+0000","last_peered":"2024-09-17T04:00:49.490615+0000","last_clean":"2024-09-17T04:00:49.490615+0000","last_became_active":"2024-09-17T04:00:49.484730+0000","last_became_peered":"2024-09-17T04:00:49.484730+0000","last_unstale":"2024-09-17T04:00:49.490615+0000","last_undegraded":"2024-09-17T04:00:49.490615+0000","last_fullsized":"2024-09-17T04:00:49.490615+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:40.328332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.476556+0000","last_change":"2024-09-17T04:00:49.476556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.476556+0000","last_undegraded":"2024-09-17T04:00:49.476556+0000","last_fullsized":"2024-09-17T04:00:49.476556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475655+0000","last_change":"2024-09-17T04:00:49.492918+0000","last_active":"2024-09-17T04:00:52.475655+0000","last_peered":"2024-09-17T04:00:52.475655+0000","last_clean":"2024-09-17T04:00:52.475655+0000","last_became_active":"2024-09-17T04:00:49.492545+0000","last_became_peered":"2024-09-17T04:00:49.492545+0000","last_unstale":"2024-09-17T04:00:52.475655+0000","last_undegraded":"2024-09-17T04:00:52.475655+0000","last_fullsized":"2024-09-17T04:00:52.475655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:49.593934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475383+0000","last_change":"2024-09-17T04:00:50.498633+0000","last_active":"2024-09-17T04:00:52.475383+0000","last_peered":"2024-09-17T04:00:52.475383+0000","last_clean":"2024-09-17T04:00:52.475383+0000","last_became_active":"2024-09-17T04:00:50.498319+0000","last_became_peered":"2024-09-17T04:00:50.498319+0000","last_unstale":"2024-09-17T04:00:52.475383+0000","last_undegraded":"2024-09-17T04:00:52.475383+0000","last_fullsized":"2024-09-17T04:00:52.475383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:33.289244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486473+0000","last_change":"2024-09-17T04:00:49.484809+0000","last_active":"2024-09-17T04:00:49.486473+0000","last_peered":"2024-09-17T04:00:49.486473+0000","last_clean":"2024-09-17T04:00:49.486473+0000","last_became_active":"2024-09-17T04:00:49.484517+0000","last_became_peered":"2024-09-17T04:00:49.484517+0000","last_unstale":"2024-09-17T04:00:49.486473+0000","last_undegraded":"2024-09-17T04:00:49.486473+0000","last_fullsized":"2024-09-17T04:00:49.486473+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:19.948285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472278+0000","last_change":"2024-09-17T04:00:49.472278+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472278+0000","last_undegraded":"2024-09-17T04:00:49.472278+0000","last_fullsized":"2024-09-17T04:00:49.472278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.488737+0000","last_change":"2024-09-17T04:00:49.483959+0000","last_active":"2024-09-17T04:00:49.488737+0000","last_peered":"2024-09-17T04:00:49.488737+0000","last_clean":"2024-09-17T04:00:49.488737+0000","last_became_active":"2024-09-17T04:00:49.483564+0000","last_became_peered":"2024-09-17T04:00:49.483564+0000","last_unstale":"2024-09-17T04:00:49.488737+0000","last_undegraded":"2024-09-17T04:00:49.488737+0000","last_fullsized":"2024-09-17T04:00:49.488737+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:16.050314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475394+0000","last_change":"2024-09-17T04:00:50.499998+0000","last_active":"2024-09-17T04:00:52.475394+0000","last_peered":"2024-09-17T04:00:52.475394+0000","last_clean":"2024-09-17T04:00:52.475394+0000","last_became_active":"2024-09-17T04:00:50.499605+0000","last_became_peered":"2024-09-17T04:00:50.499605+0000","last_unstale":"2024-09-17T04:00:52.475394+0000","last_undegraded":"2024-09-17T04:00:52.475394+0000","last_fullsized":"2024-09-17T04:00:52.475394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:04.373538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558123+0000","last_change":"2024-09-17T04:00:49.488756+0000","last_active":"2024-09-17T04:00:52.558123+0000","last_peered":"2024-09-17T04:00:52.558123+0000","last_clean":"2024-09-17T04:00:52.558123+0000","last_became_active":"2024-09-17T04:00:49.487367+0000","last_became_peered":"2024-09-17T04:00:49.487367+0000","last_unstale":"2024-09-17T04:00:52.558123+0000","last_undegraded":"2024-09-17T04:00:52.558123+0000","last_fullsized":"2024-09-17T04:00:52.558123+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475774+0000","last_change":"2024-09-17T04:00:50.504866+0000","last_active":"2024-09-17T04:00:52.475774+0000","last_peered":"2024-09-17T04:00:52.475774+0000","last_clean":"2024-09-17T04:00:52.475774+0000","last_became_active":"2024-09-17T04:00:50.504365+0000","last_became_peered":"2024-09-17T04:00:50.504365+0000","last_unstale":"2024-09-17T04:00:52.475774+0000","last_undegraded":"2024-09-17T04:00:52.475774+0000","last_fullsized":"2024-09-17T04:00:52.475774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:23.967019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485996+0000","last_change":"2024-09-17T04:00:49.484439+0000","last_active":"2024-09-17T04:00:49.485996+0000","last_peered":"2024-09-17T04:00:49.485996+0000","last_clean":"2024-09-17T04:00:49.485996+0000","last_became_active":"2024-09-17T04:00:49.483790+0000","last_became_peered":"2024-09-17T04:00:49.483790+0000","last_unstale":"2024-09-17T04:00:49.485996+0000","last_undegraded":"2024-09-17T04:00:49.485996+0000","last_fullsized":"2024-09-17T04:00:49.485996+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.011143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.469674+0000","last_change":"2024-09-17T04:00:49.469674+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.469674+0000","last_undegraded":"2024-09-17T04:00:49.469674+0000","last_fullsized":"2024-09-17T04:00:49.469674+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558774+0000","last_change":"2024-09-17T04:00:49.481548+0000","last_active":"2024-09-17T04:00:52.558774+0000","last_peered":"2024-09-17T04:00:52.558774+0000","last_clean":"2024-09-17T04:00:52.558774+0000","last_became_active":"2024-09-17T04:00:49.479803+0000","last_became_peered":"2024-09-17T04:00:49.479803+0000","last_unstale":"2024-09-17T04:00:52.558774+0000","last_undegraded":"2024-09-17T04:00:52.558774+0000","last_fullsized":"2024-09-17T04:00:52.558774+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475673+0000","last_change":"2024-09-17T04:00:50.501872+0000","last_active":"2024-09-17T04:00:52.475673+0000","last_peered":"2024-09-17T04:00:52.475673+0000","last_clean":"2024-09-17T04:00:52.475673+0000","last_became_active":"2024-09-17T04:00:50.501566+0000","last_became_peered":"2024-09-17T04:00:50.501566+0000","last_unstale":"2024-09-17T04:00:52.475673+0000","last_undegraded":"2024-09-17T04:00:52.475673+0000","last_fullsized":"2024-09-17T04:00:52.475673+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.084351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491568+0000","last_change":"2024-09-17T04:00:49.487399+0000","last_active":"2024-09-17T04:00:49.491568+0000","last_peered":"2024-09-17T04:00:49.491568+0000","last_clean":"2024-09-17T04:00:49.491568+0000","last_became_active":"2024-09-17T04:00:49.487044+0000","last_became_peered":"2024-09-17T04:00:49.487044+0000","last_unstale":"2024-09-17T04:00:49.491568+0000","last_undegraded":"2024-09-17T04:00:49.491568+0000","last_fullsized":"2024-09-17T04:00:49.491568+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:10.745031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490623+0000","last_change":"2024-09-17T04:00:50.500587+0000","last_active":"2024-09-17T04:00:51.490623+0000","last_peered":"2024-09-17T04:00:51.490623+0000","last_clean":"2024-09-17T04:00:51.490623+0000","last_became_active":"2024-09-17T04:00:50.500282+0000","last_became_peered":"2024-09-17T04:00:50.500282+0000","last_unstale":"2024-09-17T04:00:51.490623+0000","last_undegraded":"2024-09-17T04:00:51.490623+0000","last_fullsized":"2024-09-17T04:00:51.490623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:49:57.370683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"173'6","reported_seq":21,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475785+0000","last_change":"2024-09-17T04:00:49.491252+0000","last_active":"2024-09-17T04:00:52.475785+0000","last_peered":"2024-09-17T04:00:52.475785+0000","last_clean":"2024-09-17T04:00:52.475785+0000","last_became_active":"2024-09-17T04:00:49.490892+0000","last_became_peered":"2024-09-17T04:00:49.490892+0000","last_unstale":"2024-09-17T04:00:52.475785+0000","last_undegraded":"2024-09-17T04:00:52.475785+0000","last_fullsized":"2024-09-17T04:00:52.475785+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:02.780252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.473556+0000","last_change":"2024-09-17T04:00:49.473556+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.473556+0000","last_undegraded":"2024-09-17T04:00:49.473556+0000","last_fullsized":"2024-09-17T04:00:49.473556+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558379+0000","last_change":"2024-09-17T04:00:49.488763+0000","last_active":"2024-09-17T04:00:52.558379+0000","last_peered":"2024-09-17T04:00:52.558379+0000","last_clean":"2024-09-17T04:00:52.558379+0000","last_became_active":"2024-09-17T04:00:49.487670+0000","last_became_peered":"2024-09-17T04:00:49.487670+0000","last_unstale":"2024-09-17T04:00:52.558379+0000","last_undegraded":"2024-09-17T04:00:52.558379+0000","last_fullsized":"2024-09-17T04:00:52.558379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.500556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471211+0000","last_change":"2024-09-17T04:00:49.471211+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471211+0000","last_undegraded":"2024-09-17T04:00:49.471211+0000","last_fullsized":"2024-09-17T04:00:49.471211+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558318+0000","last_change":"2024-09-17T04:00:49.492017+0000","last_active":"2024-09-17T04:00:52.558318+0000","last_peered":"2024-09-17T04:00:52.558318+0000","last_clean":"2024-09-17T04:00:52.558318+0000","last_became_active":"2024-09-17T04:00:49.491561+0000","last_became_peered":"2024-09-17T04:00:49.491561+0000","last_unstale":"2024-09-17T04:00:52.558318+0000","last_undegraded":"2024-09-17T04:00:52.558318+0000","last_fullsized":"2024-09-17T04:00:52.558318+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467076+0000","last_change":"2024-09-17T04:00:49.467076+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467076+0000","last_undegraded":"2024-09-17T04:00:49.467076+0000","last_fullsized":"2024-09-17T04:00:49.467076+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490472+0000","last_change":"2024-09-17T04:00:49.482396+0000","last_active":"2024-09-17T04:00:49.490472+0000","last_peered":"2024-09-17T04:00:49.490472+0000","last_clean":"2024-09-17T04:00:49.490472+0000","last_became_active":"2024-09-17T04:00:49.481991+0000","last_became_peered":"2024-09-17T04:00:49.481991+0000","last_unstale":"2024-09-17T04:00:49.490472+0000","last_undegraded":"2024-09-17T04:00:49.490472+0000","last_fullsized":"2024-09-17T04:00:49.490472+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:15.031783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490625+0000","last_change":"2024-09-17T04:00:50.486171+0000","last_active":"2024-09-17T04:00:51.490625+0000","last_peered":"2024-09-17T04:00:51.490625+0000","last_clean":"2024-09-17T04:00:51.490625+0000","last_became_active":"2024-09-17T04:00:50.484422+0000","last_became_peered":"2024-09-17T04:00:50.484422+0000","last_unstale":"2024-09-17T04:00:51.490625+0000","last_undegraded":"2024-09-17T04:00:51.490625+0000","last_fullsized":"2024-09-17T04:00:51.490625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:09.236326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491600+0000","last_change":"2024-09-17T04:00:49.486848+0000","last_active":"2024-09-17T04:00:49.491600+0000","last_peered":"2024-09-17T04:00:49.491600+0000","last_clean":"2024-09-17T04:00:49.491600+0000","last_became_active":"2024-09-17T04:00:49.486463+0000","last_became_peered":"2024-09-17T04:00:49.486463+0000","last_unstale":"2024-09-17T04:00:49.491600+0000","last_undegraded":"2024-09-17T04:00:49.491600+0000","last_fullsized":"2024-09-17T04:00:49.491600+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:50.459427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493377+0000","last_change":"2024-09-17T04:00:50.495171+0000","last_active":"2024-09-17T04:00:51.493377+0000","last_peered":"2024-09-17T04:00:51.493377+0000","last_clean":"2024-09-17T04:00:51.493377+0000","last_became_active":"2024-09-17T04:00:50.494834+0000","last_became_peered":"2024-09-17T04:00:50.494834+0000","last_unstale":"2024-09-17T04:00:51.493377+0000","last_undegraded":"2024-09-17T04:00:51.493377+0000","last_fullsized":"2024-09-17T04:00:51.493377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:22.013808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.492419+0000","last_change":"2024-09-17T04:00:49.487492+0000","last_active":"2024-09-17T04:00:49.492419+0000","last_peered":"2024-09-17T04:00:49.492419+0000","last_clean":"2024-09-17T04:00:49.492419+0000","last_became_active":"2024-09-17T04:00:49.487146+0000","last_became_peered":"2024-09-17T04:00:49.487146+0000","last_unstale":"2024-09-17T04:00:49.492419+0000","last_undegraded":"2024-09-17T04:00:49.492419+0000","last_fullsized":"2024-09-17T04:00:49.492419+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493310+0000","last_change":"2024-09-17T04:00:50.484922+0000","last_active":"2024-09-17T04:00:51.493310+0000","last_peered":"2024-09-17T04:00:51.493310+0000","last_clean":"2024-09-17T04:00:51.493310+0000","last_became_active":"2024-09-17T04:00:50.483724+0000","last_became_peered":"2024-09-17T04:00:50.483724+0000","last_unstale":"2024-09-17T04:00:51.493310+0000","last_undegraded":"2024-09-17T04:00:51.493310+0000","last_fullsized":"2024-09-17T04:00:51.493310+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:40.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493354+0000","last_change":"2024-09-17T04:00:49.475839+0000","last_active":"2024-09-17T04:00:51.493354+0000","last_peered":"2024-09-17T04:00:51.493354+0000","last_clean":"2024-09-17T04:00:51.493354+0000","last_became_active":"2024-09-17T04:00:49.475239+0000","last_became_peered":"2024-09-17T04:00:49.475239+0000","last_unstale":"2024-09-17T04:00:51.493354+0000","last_undegraded":"2024-09-17T04:00:51.493354+0000","last_fullsized":"2024-09-17T04:00:51.493354+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:14.501945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.467772+0000","last_change":"2024-09-17T04:00:49.467772+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.467772+0000","last_undegraded":"2024-09-17T04:00:49.467772+0000","last_fullsized":"2024-09-17T04:00:49.467772+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.491597+0000","last_change":"2024-09-17T04:00:49.485542+0000","last_active":"2024-09-17T04:00:49.491597+0000","last_peered":"2024-09-17T04:00:49.491597+0000","last_clean":"2024-09-17T04:00:49.491597+0000","last_became_active":"2024-09-17T04:00:49.481811+0000","last_became_peered":"2024-09-17T04:00:49.481811+0000","last_unstale":"2024-09-17T04:00:49.491597+0000","last_undegraded":"2024-09-17T04:00:49.491597+0000","last_fullsized":"2024-09-17T04:00:49.491597+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:41.533653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.472723+0000","last_change":"2024-09-17T04:00:49.472723+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.472723+0000","last_undegraded":"2024-09-17T04:00:49.472723+0000","last_fullsized":"2024-09-17T04:00:49.472723+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.490994+0000","last_change":"2024-09-17T04:00:49.485281+0000","last_active":"2024-09-17T04:00:49.490994+0000","last_peered":"2024-09-17T04:00:49.490994+0000","last_clean":"2024-09-17T04:00:49.490994+0000","last_became_active":"2024-09-17T04:00:49.484098+0000","last_became_peered":"2024-09-17T04:00:49.484098+0000","last_unstale":"2024-09-17T04:00:49.490994+0000","last_undegraded":"2024-09-17T04:00:49.490994+0000","last_fullsized":"2024-09-17T04:00:49.490994+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.496374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.471100+0000","last_change":"2024-09-17T04:00:49.471100+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.471100+0000","last_undegraded":"2024-09-17T04:00:49.471100+0000","last_fullsized":"2024-09-17T04:00:49.471100+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485054+0000","last_change":"2024-09-17T04:00:49.479444+0000","last_active":"2024-09-17T04:00:49.485054+0000","last_peered":"2024-09-17T04:00:49.485054+0000","last_clean":"2024-09-17T04:00:49.485054+0000","last_became_active":"2024-09-17T04:00:49.479095+0000","last_became_peered":"2024-09-17T04:00:49.479095+0000","last_unstale":"2024-09-17T04:00:49.485054+0000","last_undegraded":"2024-09-17T04:00:49.485054+0000","last_fullsized":"2024-09-17T04:00:49.485054+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:09.201712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.468009+0000","last_change":"2024-09-17T04:00:49.468009+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.468009+0000","last_undegraded":"2024-09-17T04:00:49.468009+0000","last_fullsized":"2024-09-17T04:00:49.468009+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490347+0000","last_change":"2024-09-17T04:00:49.492248+0000","last_active":"2024-09-17T04:00:51.490347+0000","last_peered":"2024-09-17T04:00:51.490347+0000","last_clean":"2024-09-17T04:00:51.490347+0000","last_became_active":"2024-09-17T04:00:49.491744+0000","last_became_peered":"2024-09-17T04:00:49.491744+0000","last_unstale":"2024-09-17T04:00:51.490347+0000","last_undegraded":"2024-09-17T04:00:51.490347+0000","last_fullsized":"2024-09-17T04:00:51.490347+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:45.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492551+0000","last_change":"2024-09-17T04:00:50.485354+0000","last_active":"2024-09-17T04:00:51.492551+0000","last_peered":"2024-09-17T04:00:51.492551+0000","last_clean":"2024-09-17T04:00:51.492551+0000","last_became_active":"2024-09-17T04:00:50.485017+0000","last_became_peered":"2024-09-17T04:00:50.485017+0000","last_unstale":"2024-09-17T04:00:51.492551+0000","last_undegraded":"2024-09-17T04:00:51.492551+0000","last_fullsized":"2024-09-17T04:00:51.492551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:43.479866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490209+0000","last_change":"2024-09-17T04:00:49.490198+0000","last_active":"2024-09-17T04:00:51.490209+0000","last_peered":"2024-09-17T04:00:51.490209+0000","last_clean":"2024-09-17T04:00:51.490209+0000","last_became_active":"2024-09-17T04:00:49.489685+0000","last_became_peered":"2024-09-17T04:00:49.489685+0000","last_unstale":"2024-09-17T04:00:51.490209+0000","last_undegraded":"2024-09-17T04:00:51.490209+0000","last_fullsized":"2024-09-17T04:00:51.490209+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:21.987330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558604+0000","last_change":"2024-09-17T04:00:50.488092+0000","last_active":"2024-09-17T04:00:52.558604+0000","last_peered":"2024-09-17T04:00:52.558604+0000","last_clean":"2024-09-17T04:00:52.558604+0000","last_became_active":"2024-09-17T04:00:50.487718+0000","last_became_peered":"2024-09-17T04:00:50.487718+0000","last_unstale":"2024-09-17T04:00:52.558604+0000","last_undegraded":"2024-09-17T04:00:52.558604+0000","last_fullsized":"2024-09-17T04:00:52.558604+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:12.457060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.494752+0000","last_change":"2024-09-17T04:00:49.492610+0000","last_active":"2024-09-17T04:00:49.494752+0000","last_peered":"2024-09-17T04:00:49.494752+0000","last_clean":"2024-09-17T04:00:49.494752+0000","last_became_active":"2024-09-17T04:00:49.491506+0000","last_became_peered":"2024-09-17T04:00:49.491506+0000","last_unstale":"2024-09-17T04:00:49.494752+0000","last_undegraded":"2024-09-17T04:00:49.494752+0000","last_fullsized":"2024-09-17T04:00:49.494752+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:53.116814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954545+0000","last_change":"2024-09-17T04:00:50.511428+0000","last_active":"2024-09-17T04:00:51.954545+0000","last_peered":"2024-09-17T04:00:51.954545+0000","last_clean":"2024-09-17T04:00:51.954545+0000","last_became_active":"2024-09-17T04:00:50.511114+0000","last_became_peered":"2024-09-17T04:00:50.511114+0000","last_unstale":"2024-09-17T04:00:51.954545+0000","last_undegraded":"2024-09-17T04:00:51.954545+0000","last_fullsized":"2024-09-17T04:00:51.954545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:58.184294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.485901+0000","last_change":"2024-09-17T04:00:49.478561+0000","last_active":"2024-09-17T04:00:49.485901+0000","last_peered":"2024-09-17T04:00:49.485901+0000","last_clean":"2024-09-17T04:00:49.485901+0000","last_became_active":"2024-09-17T04:00:49.478248+0000","last_became_peered":"2024-09-17T04:00:49.478248+0000","last_unstale":"2024-09-17T04:00:49.485901+0000","last_undegraded":"2024-09-17T04:00:49.485901+0000","last_fullsized":"2024-09-17T04:00:49.485901+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:13.907873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490983+0000","last_change":"2024-09-17T04:00:50.495573+0000","last_active":"2024-09-17T04:00:51.490983+0000","last_peered":"2024-09-17T04:00:51.490983+0000","last_clean":"2024-09-17T04:00:51.490983+0000","last_became_active":"2024-09-17T04:00:50.494895+0000","last_became_peered":"2024-09-17T04:00:50.494895+0000","last_unstale":"2024-09-17T04:00:51.490983+0000","last_undegraded":"2024-09-17T04:00:51.490983+0000","last_fullsized":"2024-09-17T04:00:51.490983+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:07.480340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"174'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475720+0000","last_change":"2024-09-17T04:00:49.490303+0000","last_active":"2024-09-17T04:00:52.475720+0000","last_peered":"2024-09-17T04:00:52.475720+0000","last_clean":"2024-09-17T04:00:52.475720+0000","last_became_active":"2024-09-17T04:00:49.489882+0000","last_became_peered":"2024-09-17T04:00:49.489882+0000","last_unstale":"2024-09-17T04:00:52.475720+0000","last_undegraded":"2024-09-17T04:00:52.475720+0000","last_fullsized":"2024-09-17T04:00:52.475720+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:46.567261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492851+0000","last_change":"2024-09-17T04:00:50.493244+0000","last_active":"2024-09-17T04:00:51.492851+0000","last_peered":"2024-09-17T04:00:51.492851+0000","last_clean":"2024-09-17T04:00:51.492851+0000","last_became_active":"2024-09-17T04:00:50.492892+0000","last_became_peered":"2024-09-17T04:00:50.492892+0000","last_unstale":"2024-09-17T04:00:51.492851+0000","last_undegraded":"2024-09-17T04:00:51.492851+0000","last_fullsized":"2024-09-17T04:00:51.492851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:19.515809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.484869+0000","last_change":"2024-09-17T04:00:49.480983+0000","last_active":"2024-09-17T04:00:49.484869+0000","last_peered":"2024-09-17T04:00:49.484869+0000","last_clean":"2024-09-17T04:00:49.484869+0000","last_became_active":"2024-09-17T04:00:49.480620+0000","last_became_peered":"2024-09-17T04:00:49.480620+0000","last_unstale":"2024-09-17T04:00:49.484869+0000","last_undegraded":"2024-09-17T04:00:49.484869+0000","last_fullsized":"2024-09-17T04:00:49.484869+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:17.334558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558754+0000","last_change":"2024-09-17T04:00:50.491601+0000","last_active":"2024-09-17T04:00:52.558754+0000","last_peered":"2024-09-17T04:00:52.558754+0000","last_clean":"2024-09-17T04:00:52.558754+0000","last_became_active":"2024-09-17T04:00:50.491279+0000","last_became_peered":"2024-09-17T04:00:50.491279+0000","last_unstale":"2024-09-17T04:00:52.558754+0000","last_undegraded":"2024-09-17T04:00:52.558754+0000","last_fullsized":"2024-09-17T04:00:52.558754+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:03.782918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.482446+0000","last_change":"2024-09-17T04:00:49.477561+0000","last_active":"2024-09-17T04:00:49.482446+0000","last_peered":"2024-09-17T04:00:49.482446+0000","last_clean":"2024-09-17T04:00:49.482446+0000","last_became_active":"2024-09-17T04:00:49.477252+0000","last_became_peered":"2024-09-17T04:00:49.477252+0000","last_unstale":"2024-09-17T04:00:49.482446+0000","last_undegraded":"2024-09-17T04:00:49.482446+0000","last_fullsized":"2024-09-17T04:00:49.482446+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:21:14.568584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490348+0000","last_change":"2024-09-17T04:00:50.499748+0000","last_active":"2024-09-17T04:00:51.490348+0000","last_peered":"2024-09-17T04:00:51.490348+0000","last_clean":"2024-09-17T04:00:51.490348+0000","last_became_active":"2024-09-17T04:00:50.497933+0000","last_became_peered":"2024-09-17T04:00:50.497933+0000","last_unstale":"2024-09-17T04:00:51.490348+0000","last_undegraded":"2024-09-17T04:00:51.490348+0000","last_fullsized":"2024-09-17T04:00:51.490348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:33.887766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475476+0000","last_change":"2024-09-17T04:00:49.486586+0000","last_active":"2024-09-17T04:00:52.475476+0000","last_peered":"2024-09-17T04:00:52.475476+0000","last_clean":"2024-09-17T04:00:52.475476+0000","last_became_active":"2024-09-17T04:00:49.486099+0000","last_became_peered":"2024-09-17T04:00:49.486099+0000","last_unstale":"2024-09-17T04:00:52.475476+0000","last_undegraded":"2024-09-17T04:00:52.475476+0000","last_fullsized":"2024-09-17T04:00:52.475476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:42.671635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954355+0000","last_change":"2024-09-17T04:00:50.501002+0000","last_active":"2024-09-17T04:00:51.954355+0000","last_peered":"2024-09-17T04:00:51.954355+0000","last_clean":"2024-09-17T04:00:51.954355+0000","last_became_active":"2024-09-17T04:00:50.500621+0000","last_became_peered":"2024-09-17T04:00:50.500621+0000","last_unstale":"2024-09-17T04:00:51.954355+0000","last_undegraded":"2024-09-17T04:00:51.954355+0000","last_fullsized":"2024-09-17T04:00:51.954355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:38.586287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":10,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-17T04:00:49.486445+0000","last_change":"2024-09-17T04:00:49.483997+0000","last_active":"2024-09-17T04:00:49.486445+0000","last_peered":"2024-09-17T04:00:49.486445+0000","last_clean":"2024-09-17T04:00:49.486445+0000","last_became_active":"2024-09-17T04:00:49.483458+0000","last_became_peered":"2024-09-17T04:00:49.483458+0000","last_unstale":"2024-09-17T04:00:49.486445+0000","last_undegraded":"2024-09-17T04:00:49.486445+0000","last_fullsized":"2024-09-17T04:00:49.486445+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:10:19.789984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":1,"reported_epoch":172,"state":"creating+peering","last_fresh":"2024-09-17T04:00:49.466540+0000","last_change":"2024-09-17T04:00:49.466540+0000","last_active":"2024-09-17T04:00:49.455427+0000","last_peered":"2024-09-17T04:00:49.455427+0000","last_clean":"2024-09-17T04:00:49.455427+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:00:49.466540+0000","last_undegraded":"2024-09-17T04:00:49.466540+0000","last_fullsized":"2024-09-17T04:00:49.466540+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558691+0000","last_change":"2024-09-17T04:00:49.485410+0000","last_active":"2024-09-17T04:00:52.558691+0000","last_peered":"2024-09-17T04:00:52.558691+0000","last_clean":"2024-09-17T04:00:52.558691+0000","last_became_active":"2024-09-17T04:00:49.484171+0000","last_became_peered":"2024-09-17T04:00:49.484171+0000","last_unstale":"2024-09-17T04:00:52.558691+0000","last_undegraded":"2024-09-17T04:00:52.558691+0000","last_fullsized":"2024-09-17T04:00:52.558691+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492593+0000","last_change":"2024-09-17T04:00:50.490365+0000","last_active":"2024-09-17T04:00:51.492593+0000","last_peered":"2024-09-17T04:00:51.492593+0000","last_clean":"2024-09-17T04:00:51.492593+0000","last_became_active":"2024-09-17T04:00:50.488098+0000","last_became_peered":"2024-09-17T04:00:50.488098+0000","last_unstale":"2024-09-17T04:00:51.492593+0000","last_undegraded":"2024-09-17T04:00:51.492593+0000","last_fullsized":"2024-09-17T04:00:51.492593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:42.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954116+0000","last_change":"2024-09-17T04:00:49.487212+0000","last_active":"2024-09-17T04:00:51.954116+0000","last_peered":"2024-09-17T04:00:51.954116+0000","last_clean":"2024-09-17T04:00:51.954116+0000","last_became_active":"2024-09-17T04:00:49.486849+0000","last_became_peered":"2024-09-17T04:00:49.486849+0000","last_unstale":"2024-09-17T04:00:51.954116+0000","last_undegraded":"2024-09-17T04:00:51.954116+0000","last_fullsized":"2024-09-17T04:00:51.954116+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:31.908526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490138+0000","last_change":"2024-09-17T04:00:50.501206+0000","last_active":"2024-09-17T04:00:51.490138+0000","last_peered":"2024-09-17T04:00:51.490138+0000","last_clean":"2024-09-17T04:00:51.490138+0000","last_became_active":"2024-09-17T04:00:50.500932+0000","last_became_peered":"2024-09-17T04:00:50.500932+0000","last_unstale":"2024-09-17T04:00:51.490138+0000","last_undegraded":"2024-09-17T04:00:51.490138+0000","last_fullsized":"2024-09-17T04:00:51.490138+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:32.987666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954611+0000","last_change":"2024-09-17T04:00:49.489179+0000","last_active":"2024-09-17T04:00:51.954611+0000","last_peered":"2024-09-17T04:00:51.954611+0000","last_clean":"2024-09-17T04:00:51.954611+0000","last_became_active":"2024-09-17T04:00:49.488400+0000","last_became_peered":"2024-09-17T04:00:49.488400+0000","last_unstale":"2024-09-17T04:00:51.954611+0000","last_undegraded":"2024-09-17T04:00:51.954611+0000","last_fullsized":"2024-09-17T04:00:51.954611+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:44:31.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":392,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475314+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:52.475314+0000","last_peered":"2024-09-17T04:00:52.475314+0000","last_clean":"2024-09-17T04:00:52.475314+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:52.475314+0000","last_undegraded":"2024-09-17T04:00:52.475314+0000","last_fullsized":"2024-09-17T04:00:52.475314+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2228,"num_objects":11,"num_object_clones":0,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509580,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2793472,"data_stored":1549678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2207744,"data_stored":959414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509579,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41616,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330224,"statfs":{"total":96636764160,"available":96594149376,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37452,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334388,"statfs":{"total":96636764160,"available":96598413312,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509581,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":2150400,"data_stored":938348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542285,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2785280,"data_stored":1548156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:PG 27.3d is not active+clean 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.470097+0000', 'last_change': '2024-09-17T04:00:49.470097+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.470097+0000', 'last_undegraded': '2024-09-17T04:00:49.470097+0000', 'last_fullsized': '2024-09-17T04:00:49.470097+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:PG 27.3b is not active+clean 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.472481+0000', 'last_change': '2024-09-17T04:00:49.472481+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.472481+0000', 'last_undegraded': '2024-09-17T04:00:49.472481+0000', 'last_fullsized': '2024-09-17T04:00:49.472481+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:PG 27.3a is not active+clean 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.474000+0000', 'last_change': '2024-09-17T04:00:49.474000+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.474000+0000', 'last_undegraded': '2024-09-17T04:00:49.474000+0000', 'last_fullsized': '2024-09-17T04:00:49.474000+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:PG 27.38 is not active+clean 2024-09-17T04:00:55.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.472807+0000', 'last_change': '2024-09-17T04:00:49.472807+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.472807+0000', 'last_undegraded': '2024-09-17T04:00:49.472807+0000', 'last_fullsized': '2024-09-17T04:00:49.472807+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.36 is not active+clean 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.474579+0000', 'last_change': '2024-09-17T04:00:49.474579+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.474579+0000', 'last_undegraded': '2024-09-17T04:00:49.474579+0000', 'last_fullsized': '2024-09-17T04:00:49.474579+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.35 is not active+clean 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.468304+0000', 'last_change': '2024-09-17T04:00:49.468304+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.468304+0000', 'last_undegraded': '2024-09-17T04:00:49.468304+0000', 'last_fullsized': '2024-09-17T04:00:49.468304+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.30 is not active+clean 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.475124+0000', 'last_change': '2024-09-17T04:00:49.475124+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.475124+0000', 'last_undegraded': '2024-09-17T04:00:49.475124+0000', 'last_fullsized': '2024-09-17T04:00:49.475124+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.2e is not active+clean 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.474097+0000', 'last_change': '2024-09-17T04:00:49.474097+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.474097+0000', 'last_undegraded': '2024-09-17T04:00:49.474097+0000', 'last_fullsized': '2024-09-17T04:00:49.474097+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.2d is not active+clean 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.476999+0000', 'last_change': '2024-09-17T04:00:49.476999+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.476999+0000', 'last_undegraded': '2024-09-17T04:00:49.476999+0000', 'last_fullsized': '2024-09-17T04:00:49.476999+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.926 INFO:tasks.ceph.ceph_manager.ceph:PG 27.2c is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.473627+0000', 'last_change': '2024-09-17T04:00:49.473627+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.473627+0000', 'last_undegraded': '2024-09-17T04:00:49.473627+0000', 'last_fullsized': '2024-09-17T04:00:49.473627+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.24 is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.470536+0000', 'last_change': '2024-09-17T04:00:49.470536+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.470536+0000', 'last_undegraded': '2024-09-17T04:00:49.470536+0000', 'last_fullsized': '2024-09-17T04:00:49.470536+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.21 is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.473517+0000', 'last_change': '2024-09-17T04:00:49.473517+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.473517+0000', 'last_undegraded': '2024-09-17T04:00:49.473517+0000', 'last_fullsized': '2024-09-17T04:00:49.473517+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.1d is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.476556+0000', 'last_change': '2024-09-17T04:00:49.476556+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.476556+0000', 'last_undegraded': '2024-09-17T04:00:49.476556+0000', 'last_fullsized': '2024-09-17T04:00:49.476556+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.18 is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.472278+0000', 'last_change': '2024-09-17T04:00:49.472278+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.472278+0000', 'last_undegraded': '2024-09-17T04:00:49.472278+0000', 'last_fullsized': '2024-09-17T04:00:49.472278+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.17 is not active+clean 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.469674+0000', 'last_change': '2024-09-17T04:00:49.469674+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.469674+0000', 'last_undegraded': '2024-09-17T04:00:49.469674+0000', 'last_fullsized': '2024-09-17T04:00:49.469674+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.927 INFO:tasks.ceph.ceph_manager.ceph:PG 27.12 is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.473556+0000', 'last_change': '2024-09-17T04:00:49.473556+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.473556+0000', 'last_undegraded': '2024-09-17T04:00:49.473556+0000', 'last_fullsized': '2024-09-17T04:00:49.473556+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:PG 27.13 is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.471211+0000', 'last_change': '2024-09-17T04:00:49.471211+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.471211+0000', 'last_undegraded': '2024-09-17T04:00:49.471211+0000', 'last_fullsized': '2024-09-17T04:00:49.471211+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:PG 27.10 is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.467076+0000', 'last_change': '2024-09-17T04:00:49.467076+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.467076+0000', 'last_undegraded': '2024-09-17T04:00:49.467076+0000', 'last_fullsized': '2024-09-17T04:00:49.467076+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:PG 27.c is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.467772+0000', 'last_change': '2024-09-17T04:00:49.467772+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.467772+0000', 'last_undegraded': '2024-09-17T04:00:49.467772+0000', 'last_fullsized': '2024-09-17T04:00:49.467772+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:PG 27.d is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.472723+0000', 'last_change': '2024-09-17T04:00:49.472723+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.472723+0000', 'last_undegraded': '2024-09-17T04:00:49.472723+0000', 'last_fullsized': '2024-09-17T04:00:49.472723+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:PG 27.a is not active+clean 2024-09-17T04:00:55.928 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.471100+0000', 'last_change': '2024-09-17T04:00:49.471100+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.471100+0000', 'last_undegraded': '2024-09-17T04:00:49.471100+0000', 'last_fullsized': '2024-09-17T04:00:49.471100+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:00:55.929 INFO:tasks.ceph.ceph_manager.ceph:PG 27.b is not active+clean 2024-09-17T04:00:55.929 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.468009+0000', 'last_change': '2024-09-17T04:00:49.468009+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.468009+0000', 'last_undegraded': '2024-09-17T04:00:49.468009+0000', 'last_fullsized': '2024-09-17T04:00:49.468009+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.929 INFO:tasks.ceph.ceph_manager.ceph:PG 27.0 is not active+clean 2024-09-17T04:00:55.929 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '27.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 172, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:00:49.466540+0000', 'last_change': '2024-09-17T04:00:49.466540+0000', 'last_active': '2024-09-17T04:00:49.455427+0000', 'last_peered': '2024-09-17T04:00:49.455427+0000', 'last_clean': '2024-09-17T04:00:49.455427+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:00:49.466540+0000', 'last_undegraded': '2024-09-17T04:00:49.466540+0000', 'last_fullsized': '2024-09-17T04:00:49.466540+0000', 'mapping_epoch': 172, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 172, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'last_clean_scrub_stamp': '2024-09-17T04:00:49.455427+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:00:55.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 -- 172.21.15.129:0/1615793648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 msgr2=0x7f43780b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 --2- 172.21.15.129:0/1615793648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43780b4690 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f4364007350 tx=0x7f436402b1a0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 -- 172.21.15.129:0/1615793648 shutdown_connections 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 --2- 172.21.15.129:0/1615793648 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43780b4bd0 0x7f43780cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 --2- 172.21.15.129:0/1615793648 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43780b4690 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 -- 172.21.15.129:0/1615793648 >> 172.21.15.129:0/1615793648 conn(0x7f437806d410 msgr2=0x7f43780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 -- 172.21.15.129:0/1615793648 shutdown_connections 2024-09-17T04:00:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.180+0000 7f4380400640 1 -- 172.21.15.129:0/1615793648 wait complete. 2024-09-17T04:00:56.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.183+0000 7f4380400640 1 Processor -- start 2024-09-17T04:00:56.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.183+0000 7f4380400640 1 -- start start 2024-09-17T04:00:56.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 0x7f43780adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 0x7f4378159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43780ccfa0 con 0x7f437815ac20 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43780cce20 con 0x7f43780b4bd0 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.184+0000 7f4380400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f43780cd120 con 0x7f4378070a10 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35182/0 (socket says 172.21.15.129:35182) 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 -- 172.21.15.129:0/900340241 learned_addr learned my addr 172.21.15.129:0/900340241 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437da00640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 0x7f43780adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437d000640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 0x7f4378159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 -- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 msgr2=0x7f43780adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 0x7f43780adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 -- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 msgr2=0x7f4378159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 0x7f4378159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 -- 172.21.15.129:0/900340241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43781f5a60 con 0x7f437815ac20 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437da00640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 0x7f43780adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437d000640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 0x7f4378159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f437e400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f437000e990 tx=0x7f437000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370004d20 con 0x7f437815ac20 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.185+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f437000caa0 con 0x7f437815ac20 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.186+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43781f5d00 con 0x7f437815ac20 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.186+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370005020 con 0x7f437815ac20 2024-09-17T04:00:56.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.186+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43781f62c0 con 0x7f437815ac20 2024-09-17T04:00:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.187+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43780ae0a0 con 0x7f437815ac20 2024-09-17T04:00:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.188+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f43700051c0 con 0x7f437815ac20 2024-09-17T04:00:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.189+0000 7f4375600640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.189+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f437009b6e0 con 0x7f437815ac20 2024-09-17T04:00:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.189+0000 7f437da00640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.190+0000 7f437da00640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f436c0016a0 tx=0x7f436c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:56.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.193+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4370016030 con 0x7f437815ac20 2024-09-17T04:00:56.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.445+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f43780cd5f0 con 0x7f437815ac20 2024-09-17T04:00:56.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.446+0000 7f4375600640 1 -- 172.21.15.129:0/900340241 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1565 (secure 0 0 0) 0x7f4370063a80 con 0x7f437815ac20 2024-09-17T04:00:56.446 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:56.446 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":707,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":174,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":74},{"state_name":"creating+peering","count":23}],"num_pgs":97,"num_pools":3,"num_objects":13,"data_bytes":592596,"bytes_used":333770752,"bytes_avail":772760342528,"bytes_total":773094113280,"inactive_pgs_ratio":0.23711340129375458,"write_bytes_sec":1876,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","id":13,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":13,"rank":0,"name":"c","status":"up:active","gid":9492}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-09-17T04:00:02.472790+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 msgr2=0x7f435807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f436c0016a0 tx=0x7f436c009040 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 msgr2=0x7f43781f9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f437000e990 tx=0x7f437000ee50 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 shutdown_connections 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4358078b90 0x7f435807b0a0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f437815ac20 0x7f43781f9890 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780b4bd0 0x7f4378159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 --2- 172.21.15.129:0/900340241 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4378070a10 0x7f43780adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.448+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 >> 172.21.15.129:0/900340241 conn(0x7f437806d410 msgr2=0x7f43780afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.449+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 shutdown_connections 2024-09-17T04:00:56.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.449+0000 7f4380400640 1 -- 172.21.15.129:0/900340241 wait complete. 2024-09-17T04:00:56.450 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T04:00:56.449+0000 7f2963400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:00:56.474 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:00:56.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:00:56.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.697+0000 7f901b200640 1 -- 172.21.15.129:0/4225796245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90140ce600 msgr2=0x7f9014145e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.697+0000 7f901b200640 1 --2- 172.21.15.129:0/4225796245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90140ce600 0x7f9014145e10 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f90140d1170 tx=0x7f900802a420 comp rx=0 tx=0).stop 2024-09-17T04:00:56.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 -- 172.21.15.129:0/4225796245 shutdown_connections 2024-09-17T04:00:56.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 --2- 172.21.15.129:0/4225796245 >> v2:172.21.15.181:3300/0 conn(0x7f90140cffe0 0x7f901415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 --2- 172.21.15.129:0/4225796245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90140ce600 0x7f9014145e10 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 -- 172.21.15.129:0/4225796245 >> 172.21.15.129:0/4225796245 conn(0x7f90140d11a0 msgr2=0x7f90140d15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:56.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 -- 172.21.15.129:0/4225796245 shutdown_connections 2024-09-17T04:00:56.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.698+0000 7f901b200640 1 -- 172.21.15.129:0/4225796245 wait complete. 2024-09-17T04:00:56.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.700+0000 7f901b200640 1 Processor -- start 2024-09-17T04:00:56.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.700+0000 7f901b200640 1 -- start start 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90140ce600 0x7f90141f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90140cffe0 0x7f90141f8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9014073f50 con 0x7f901415d070 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9014073dd0 con 0x7f90140cffe0 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f90140740d0 con 0x7f90140ce600 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35200/0 (socket says 172.21.15.129:35200) 2024-09-17T04:00:56.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 -- 172.21.15.129:0/2456939092 learned_addr learned my addr 172.21.15.129:0/2456939092 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f9019e00640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90140ce600 0x7f90141f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f9019400640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90140cffe0 0x7f90141f8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 -- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90140ce600 msgr2=0x7f90141f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90140ce600 0x7f90141f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 -- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90140cffe0 msgr2=0x7f90141f8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90140cffe0 0x7f90141f8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.702+0000 7f901a800640 1 -- 172.21.15.129:0/2456939092 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90140cc2c0 con 0x7f901415d070 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f901a800640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f901000b9e0 tx=0x7f901000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9010003b70 con 0x7f901415d070 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9010003e00 con 0x7f901415d070 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90140cc5a0 con 0x7f901415d070 2024-09-17T04:00:56.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901000cb80 con 0x7f901415d070 2024-09-17T04:00:56.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.703+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f901415b260 con 0x7f901415d070 2024-09-17T04:00:56.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.704+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9014074580 con 0x7f901415d070 2024-09-17T04:00:56.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.705+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9010004000 con 0x7f901415d070 2024-09-17T04:00:56.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.706+0000 7f900e000640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 0x7f8ff807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:00:56.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.706+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f901009abf0 con 0x7f901415d070 2024-09-17T04:00:56.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.707+0000 7f9019e00640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 0x7f8ff807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:00:56.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.707+0000 7f9019e00640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 0x7f8ff807b0a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9008033000 tx=0x7f9008004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:00:56.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.711+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9010063020 con 0x7f901415d070 2024-09-17T04:00:56.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.890+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90140b0f50 con 0x7f8ff8078b90 2024-09-17T04:00:56.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.895+0000 7f900e000640 1 -- 172.21.15.129:0/2456939092 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252476 (secure 0 0 0) 0x7f90140b0f50 con 0x7f8ff8078b90 2024-09-17T04:00:56.899 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:00:56.900 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:00:56.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.899+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 msgr2=0x7f8ff807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.899+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 0x7f8ff807b0a0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9008033000 tx=0x7f9008004730 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.899+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 msgr2=0x7f90141f9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.899+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f901000b9e0 tx=0x7f901000bea0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 shutdown_connections 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ff8078b90 0x7f8ff807b0a0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f901415d070 0x7f90141f9060 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90140cffe0 0x7f90141f8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 --2- 172.21.15.129:0/2456939092 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90140ce600 0x7f90141f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 >> 172.21.15.129:0/2456939092 conn(0x7f90140d11a0 msgr2=0x7f90140b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 shutdown_connections 2024-09-17T04:00:56.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:00:56.900+0000 7f901b200640 1 -- 172.21.15.129:0/2456939092 wait complete. 2024-09-17T04:00:56.923 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-17T04:00:56.487712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326072,"kb_used_data":18312,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754648648,"statfs":{"total":773094113280,"available":772760215552,"internally_reserved":0,"allocated":18751488,"data_stored":8847140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.022767"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490698+0000","last_change":"2024-09-17T04:00:50.499023+0000","last_active":"2024-09-17T04:00:51.490698+0000","last_peered":"2024-09-17T04:00:51.490698+0000","last_clean":"2024-09-17T04:00:51.490698+0000","last_became_active":"2024-09-17T04:00:50.498702+0000","last_became_peered":"2024-09-17T04:00:50.498702+0000","last_unstale":"2024-09-17T04:00:51.490698+0000","last_undegraded":"2024-09-17T04:00:51.490698+0000","last_fullsized":"2024-09-17T04:00:51.490698+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:47.660122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493082+0000","last_change":"2024-09-17T04:00:50.481292+0000","last_active":"2024-09-17T04:00:51.493082+0000","last_peered":"2024-09-17T04:00:51.493082+0000","last_clean":"2024-09-17T04:00:51.493082+0000","last_became_active":"2024-09-17T04:00:50.480948+0000","last_became_peered":"2024-09-17T04:00:50.480948+0000","last_unstale":"2024-09-17T04:00:51.493082+0000","last_undegraded":"2024-09-17T04:00:51.493082+0000","last_fullsized":"2024-09-17T04:00:51.493082+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.354995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491992+0000","last_change":"2024-09-17T04:00:50.496576+0000","last_active":"2024-09-17T04:00:51.491992+0000","last_peered":"2024-09-17T04:00:51.491992+0000","last_clean":"2024-09-17T04:00:51.491992+0000","last_became_active":"2024-09-17T04:00:50.495701+0000","last_became_peered":"2024-09-17T04:00:50.495701+0000","last_unstale":"2024-09-17T04:00:51.491992+0000","last_undegraded":"2024-09-17T04:00:51.491992+0000","last_fullsized":"2024-09-17T04:00:51.491992+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:50.676585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475438+0000","last_change":"2024-09-17T04:00:50.500751+0000","last_active":"2024-09-17T04:00:52.475438+0000","last_peered":"2024-09-17T04:00:52.475438+0000","last_clean":"2024-09-17T04:00:52.475438+0000","last_became_active":"2024-09-17T04:00:50.500022+0000","last_became_peered":"2024-09-17T04:00:50.500022+0000","last_unstale":"2024-09-17T04:00:52.475438+0000","last_undegraded":"2024-09-17T04:00:52.475438+0000","last_fullsized":"2024-09-17T04:00:52.475438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:04.192066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191098+0000","last_change":"2024-09-17T04:00:50.505448+0000","last_active":"2024-09-17T04:00:52.191098+0000","last_peered":"2024-09-17T04:00:52.191098+0000","last_clean":"2024-09-17T04:00:52.191098+0000","last_became_active":"2024-09-17T04:00:50.505068+0000","last_became_peered":"2024-09-17T04:00:50.505068+0000","last_unstale":"2024-09-17T04:00:52.191098+0000","last_undegraded":"2024-09-17T04:00:52.191098+0000","last_fullsized":"2024-09-17T04:00:52.191098+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:18.315448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190863+0000","last_change":"2024-09-17T04:00:50.502154+0000","last_active":"2024-09-17T04:00:52.190863+0000","last_peered":"2024-09-17T04:00:52.190863+0000","last_clean":"2024-09-17T04:00:52.190863+0000","last_became_active":"2024-09-17T04:00:50.501768+0000","last_became_peered":"2024-09-17T04:00:50.501768+0000","last_unstale":"2024-09-17T04:00:52.190863+0000","last_undegraded":"2024-09-17T04:00:52.190863+0000","last_fullsized":"2024-09-17T04:00:52.190863+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.863583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490063+0000","last_change":"2024-09-17T04:00:50.488121+0000","last_active":"2024-09-17T04:00:51.490063+0000","last_peered":"2024-09-17T04:00:51.490063+0000","last_clean":"2024-09-17T04:00:51.490063+0000","last_became_active":"2024-09-17T04:00:50.487823+0000","last_became_peered":"2024-09-17T04:00:50.487823+0000","last_unstale":"2024-09-17T04:00:51.490063+0000","last_undegraded":"2024-09-17T04:00:51.490063+0000","last_fullsized":"2024-09-17T04:00:51.490063+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:27.505920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190744+0000","last_change":"2024-09-17T04:00:50.495847+0000","last_active":"2024-09-17T04:00:52.190744+0000","last_peered":"2024-09-17T04:00:52.190744+0000","last_clean":"2024-09-17T04:00:52.190744+0000","last_became_active":"2024-09-17T04:00:50.495220+0000","last_became_peered":"2024-09-17T04:00:50.495220+0000","last_unstale":"2024-09-17T04:00:52.190744+0000","last_undegraded":"2024-09-17T04:00:52.190744+0000","last_fullsized":"2024-09-17T04:00:52.190744+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:11.747697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493544+0000","last_change":"2024-09-17T04:00:50.494343+0000","last_active":"2024-09-17T04:00:51.493544+0000","last_peered":"2024-09-17T04:00:51.493544+0000","last_clean":"2024-09-17T04:00:51.493544+0000","last_became_active":"2024-09-17T04:00:50.493022+0000","last_became_peered":"2024-09-17T04:00:50.493022+0000","last_unstale":"2024-09-17T04:00:51.493544+0000","last_undegraded":"2024-09-17T04:00:51.493544+0000","last_fullsized":"2024-09-17T04:00:51.493544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:20.950951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558374+0000","last_change":"2024-09-17T04:00:50.493475+0000","last_active":"2024-09-17T04:00:52.558374+0000","last_peered":"2024-09-17T04:00:52.558374+0000","last_clean":"2024-09-17T04:00:52.558374+0000","last_became_active":"2024-09-17T04:00:50.493182+0000","last_became_peered":"2024-09-17T04:00:50.493182+0000","last_unstale":"2024-09-17T04:00:52.558374+0000","last_undegraded":"2024-09-17T04:00:52.558374+0000","last_fullsized":"2024-09-17T04:00:52.558374+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:03.170026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492137+0000","last_change":"2024-09-17T04:00:50.497695+0000","last_active":"2024-09-17T04:00:51.492137+0000","last_peered":"2024-09-17T04:00:51.492137+0000","last_clean":"2024-09-17T04:00:51.492137+0000","last_became_active":"2024-09-17T04:00:50.497361+0000","last_became_peered":"2024-09-17T04:00:50.497361+0000","last_unstale":"2024-09-17T04:00:51.492137+0000","last_undegraded":"2024-09-17T04:00:51.492137+0000","last_fullsized":"2024-09-17T04:00:51.492137+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:01.343172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954479+0000","last_change":"2024-09-17T04:00:50.503620+0000","last_active":"2024-09-17T04:00:51.954479+0000","last_peered":"2024-09-17T04:00:51.954479+0000","last_clean":"2024-09-17T04:00:51.954479+0000","last_became_active":"2024-09-17T04:00:50.503238+0000","last_became_peered":"2024-09-17T04:00:50.503238+0000","last_unstale":"2024-09-17T04:00:51.954479+0000","last_undegraded":"2024-09-17T04:00:51.954479+0000","last_fullsized":"2024-09-17T04:00:51.954479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:10.261218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954637+0000","last_change":"2024-09-17T04:00:50.502335+0000","last_active":"2024-09-17T04:00:51.954637+0000","last_peered":"2024-09-17T04:00:51.954637+0000","last_clean":"2024-09-17T04:00:51.954637+0000","last_became_active":"2024-09-17T04:00:50.502036+0000","last_became_peered":"2024-09-17T04:00:50.502036+0000","last_unstale":"2024-09-17T04:00:51.954637+0000","last_undegraded":"2024-09-17T04:00:51.954637+0000","last_fullsized":"2024-09-17T04:00:51.954637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:37.502247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490230+0000","last_change":"2024-09-17T04:00:50.501289+0000","last_active":"2024-09-17T04:00:51.490230+0000","last_peered":"2024-09-17T04:00:51.490230+0000","last_clean":"2024-09-17T04:00:51.490230+0000","last_became_active":"2024-09-17T04:00:50.501012+0000","last_became_peered":"2024-09-17T04:00:50.501012+0000","last_unstale":"2024-09-17T04:00:51.490230+0000","last_undegraded":"2024-09-17T04:00:51.490230+0000","last_fullsized":"2024-09-17T04:00:51.490230+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:36.532175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490074+0000","last_change":"2024-09-17T04:00:50.490115+0000","last_active":"2024-09-17T04:00:51.490074+0000","last_peered":"2024-09-17T04:00:51.490074+0000","last_clean":"2024-09-17T04:00:51.490074+0000","last_became_active":"2024-09-17T04:00:50.489826+0000","last_became_peered":"2024-09-17T04:00:50.489826+0000","last_unstale":"2024-09-17T04:00:51.490074+0000","last_undegraded":"2024-09-17T04:00:51.490074+0000","last_fullsized":"2024-09-17T04:00:51.490074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:11.406297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557529+0000","last_change":"2024-09-17T04:00:50.489115+0000","last_active":"2024-09-17T04:00:52.557529+0000","last_peered":"2024-09-17T04:00:52.557529+0000","last_clean":"2024-09-17T04:00:52.557529+0000","last_became_active":"2024-09-17T04:00:50.488820+0000","last_became_peered":"2024-09-17T04:00:50.488820+0000","last_unstale":"2024-09-17T04:00:52.557529+0000","last_undegraded":"2024-09-17T04:00:52.557529+0000","last_fullsized":"2024-09-17T04:00:52.557529+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:50.511222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490971+0000","last_change":"2024-09-17T04:00:50.497620+0000","last_active":"2024-09-17T04:00:51.490971+0000","last_peered":"2024-09-17T04:00:51.490971+0000","last_clean":"2024-09-17T04:00:51.490971+0000","last_became_active":"2024-09-17T04:00:50.495270+0000","last_became_peered":"2024-09-17T04:00:50.495270+0000","last_unstale":"2024-09-17T04:00:51.490971+0000","last_undegraded":"2024-09-17T04:00:51.490971+0000","last_fullsized":"2024-09-17T04:00:51.490971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:29.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558396+0000","last_change":"2024-09-17T04:00:50.496642+0000","last_active":"2024-09-17T04:00:52.558396+0000","last_peered":"2024-09-17T04:00:52.558396+0000","last_clean":"2024-09-17T04:00:52.558396+0000","last_became_active":"2024-09-17T04:00:50.496350+0000","last_became_peered":"2024-09-17T04:00:50.496350+0000","last_unstale":"2024-09-17T04:00:52.558396+0000","last_undegraded":"2024-09-17T04:00:52.558396+0000","last_fullsized":"2024-09-17T04:00:52.558396+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.152814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558136+0000","last_change":"2024-09-17T04:00:50.492275+0000","last_active":"2024-09-17T04:00:52.558136+0000","last_peered":"2024-09-17T04:00:52.558136+0000","last_clean":"2024-09-17T04:00:52.558136+0000","last_became_active":"2024-09-17T04:00:50.491684+0000","last_became_peered":"2024-09-17T04:00:50.491684+0000","last_unstale":"2024-09-17T04:00:52.558136+0000","last_undegraded":"2024-09-17T04:00:52.558136+0000","last_fullsized":"2024-09-17T04:00:52.558136+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:21.696817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191282+0000","last_change":"2024-09-17T04:00:50.496943+0000","last_active":"2024-09-17T04:00:52.191282+0000","last_peered":"2024-09-17T04:00:52.191282+0000","last_clean":"2024-09-17T04:00:52.191282+0000","last_became_active":"2024-09-17T04:00:50.496646+0000","last_became_peered":"2024-09-17T04:00:50.496646+0000","last_unstale":"2024-09-17T04:00:52.191282+0000","last_undegraded":"2024-09-17T04:00:52.191282+0000","last_fullsized":"2024-09-17T04:00:52.191282+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:55.732967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475406+0000","last_change":"2024-09-17T04:00:50.499009+0000","last_active":"2024-09-17T04:00:52.475406+0000","last_peered":"2024-09-17T04:00:52.475406+0000","last_clean":"2024-09-17T04:00:52.475406+0000","last_became_active":"2024-09-17T04:00:50.495348+0000","last_became_peered":"2024-09-17T04:00:50.495348+0000","last_unstale":"2024-09-17T04:00:52.475406+0000","last_undegraded":"2024-09-17T04:00:52.475406+0000","last_fullsized":"2024-09-17T04:00:52.475406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:38.854990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558507+0000","last_change":"2024-09-17T04:00:50.483447+0000","last_active":"2024-09-17T04:00:52.558507+0000","last_peered":"2024-09-17T04:00:52.558507+0000","last_clean":"2024-09-17T04:00:52.558507+0000","last_became_active":"2024-09-17T04:00:50.483105+0000","last_became_peered":"2024-09-17T04:00:50.483105+0000","last_unstale":"2024-09-17T04:00:52.558507+0000","last_undegraded":"2024-09-17T04:00:52.558507+0000","last_fullsized":"2024-09-17T04:00:52.558507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:22.999066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558726+0000","last_change":"2024-09-17T04:00:50.488020+0000","last_active":"2024-09-17T04:00:52.558726+0000","last_peered":"2024-09-17T04:00:52.558726+0000","last_clean":"2024-09-17T04:00:52.558726+0000","last_became_active":"2024-09-17T04:00:50.487293+0000","last_became_peered":"2024-09-17T04:00:50.487293+0000","last_unstale":"2024-09-17T04:00:52.558726+0000","last_undegraded":"2024-09-17T04:00:52.558726+0000","last_fullsized":"2024-09-17T04:00:52.558726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:15.011545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475667+0000","last_change":"2024-09-17T04:00:50.493247+0000","last_active":"2024-09-17T04:00:52.475667+0000","last_peered":"2024-09-17T04:00:52.475667+0000","last_clean":"2024-09-17T04:00:52.475667+0000","last_became_active":"2024-09-17T04:00:50.492907+0000","last_became_peered":"2024-09-17T04:00:50.492907+0000","last_unstale":"2024-09-17T04:00:52.475667+0000","last_undegraded":"2024-09-17T04:00:52.475667+0000","last_fullsized":"2024-09-17T04:00:52.475667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:58.410921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493295+0000","last_change":"2024-09-17T04:00:50.487190+0000","last_active":"2024-09-17T04:00:51.493295+0000","last_peered":"2024-09-17T04:00:51.493295+0000","last_clean":"2024-09-17T04:00:51.493295+0000","last_became_active":"2024-09-17T04:00:50.484107+0000","last_became_peered":"2024-09-17T04:00:50.484107+0000","last_unstale":"2024-09-17T04:00:51.493295+0000","last_undegraded":"2024-09-17T04:00:51.493295+0000","last_fullsized":"2024-09-17T04:00:51.493295+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:17.052980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490680+0000","last_change":"2024-09-17T04:00:50.500005+0000","last_active":"2024-09-17T04:00:51.490680+0000","last_peered":"2024-09-17T04:00:51.490680+0000","last_clean":"2024-09-17T04:00:51.490680+0000","last_became_active":"2024-09-17T04:00:50.497315+0000","last_became_peered":"2024-09-17T04:00:50.497315+0000","last_unstale":"2024-09-17T04:00:51.490680+0000","last_undegraded":"2024-09-17T04:00:51.490680+0000","last_fullsized":"2024-09-17T04:00:51.490680+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:50.074066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954424+0000","last_change":"2024-09-17T04:00:50.507420+0000","last_active":"2024-09-17T04:00:51.954424+0000","last_peered":"2024-09-17T04:00:51.954424+0000","last_clean":"2024-09-17T04:00:51.954424+0000","last_became_active":"2024-09-17T04:00:50.506925+0000","last_became_peered":"2024-09-17T04:00:50.506925+0000","last_unstale":"2024-09-17T04:00:51.954424+0000","last_undegraded":"2024-09-17T04:00:51.954424+0000","last_fullsized":"2024-09-17T04:00:51.954424+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.177663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491345+0000","last_change":"2024-09-17T04:00:50.508554+0000","last_active":"2024-09-17T04:00:51.491345+0000","last_peered":"2024-09-17T04:00:51.491345+0000","last_clean":"2024-09-17T04:00:51.491345+0000","last_became_active":"2024-09-17T04:00:50.508262+0000","last_became_peered":"2024-09-17T04:00:50.508262+0000","last_unstale":"2024-09-17T04:00:51.491345+0000","last_undegraded":"2024-09-17T04:00:51.491345+0000","last_fullsized":"2024-09-17T04:00:51.491345+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:13.497944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954415+0000","last_change":"2024-09-17T04:00:50.507336+0000","last_active":"2024-09-17T04:00:51.954415+0000","last_peered":"2024-09-17T04:00:51.954415+0000","last_clean":"2024-09-17T04:00:51.954415+0000","last_became_active":"2024-09-17T04:00:50.507000+0000","last_became_peered":"2024-09-17T04:00:50.507000+0000","last_unstale":"2024-09-17T04:00:51.954415+0000","last_undegraded":"2024-09-17T04:00:51.954415+0000","last_fullsized":"2024-09-17T04:00:51.954415+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:03.261144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492684+0000","last_change":"2024-09-17T04:00:50.508385+0000","last_active":"2024-09-17T04:00:51.492684+0000","last_peered":"2024-09-17T04:00:51.492684+0000","last_clean":"2024-09-17T04:00:51.492684+0000","last_became_active":"2024-09-17T04:00:50.507803+0000","last_became_peered":"2024-09-17T04:00:50.507803+0000","last_unstale":"2024-09-17T04:00:51.492684+0000","last_undegraded":"2024-09-17T04:00:51.492684+0000","last_fullsized":"2024-09-17T04:00:51.492684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:40.047416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557509+0000","last_change":"2024-09-17T04:00:50.493135+0000","last_active":"2024-09-17T04:00:52.557509+0000","last_peered":"2024-09-17T04:00:52.557509+0000","last_clean":"2024-09-17T04:00:52.557509+0000","last_became_active":"2024-09-17T04:00:50.492849+0000","last_became_peered":"2024-09-17T04:00:50.492849+0000","last_unstale":"2024-09-17T04:00:52.557509+0000","last_undegraded":"2024-09-17T04:00:52.557509+0000","last_fullsized":"2024-09-17T04:00:52.557509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:15.162412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490344+0000","last_change":"2024-09-17T04:00:50.504166+0000","last_active":"2024-09-17T04:00:51.490344+0000","last_peered":"2024-09-17T04:00:51.490344+0000","last_clean":"2024-09-17T04:00:51.490344+0000","last_became_active":"2024-09-17T04:00:50.503869+0000","last_became_peered":"2024-09-17T04:00:50.503869+0000","last_unstale":"2024-09-17T04:00:51.490344+0000","last_undegraded":"2024-09-17T04:00:51.490344+0000","last_fullsized":"2024-09-17T04:00:51.490344+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:44.335387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493116+0000","last_change":"2024-09-17T04:00:50.497167+0000","last_active":"2024-09-17T04:00:51.493116+0000","last_peered":"2024-09-17T04:00:51.493116+0000","last_clean":"2024-09-17T04:00:51.493116+0000","last_became_active":"2024-09-17T04:00:50.496798+0000","last_became_peered":"2024-09-17T04:00:50.496798+0000","last_unstale":"2024-09-17T04:00:51.493116+0000","last_undegraded":"2024-09-17T04:00:51.493116+0000","last_fullsized":"2024-09-17T04:00:51.493116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:18.337224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"173'4","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954531+0000","last_change":"2024-09-17T04:00:49.488273+0000","last_active":"2024-09-17T04:00:51.954531+0000","last_peered":"2024-09-17T04:00:51.954531+0000","last_clean":"2024-09-17T04:00:51.954531+0000","last_became_active":"2024-09-17T04:00:49.486926+0000","last_became_peered":"2024-09-17T04:00:49.486926+0000","last_unstale":"2024-09-17T04:00:51.954531+0000","last_undegraded":"2024-09-17T04:00:51.954531+0000","last_fullsized":"2024-09-17T04:00:51.954531+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:23.992360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558954+0000","last_change":"2024-09-17T04:00:50.493935+0000","last_active":"2024-09-17T04:00:52.558954+0000","last_peered":"2024-09-17T04:00:52.558954+0000","last_clean":"2024-09-17T04:00:52.558954+0000","last_became_active":"2024-09-17T04:00:50.490351+0000","last_became_peered":"2024-09-17T04:00:50.490351+0000","last_unstale":"2024-09-17T04:00:52.558954+0000","last_undegraded":"2024-09-17T04:00:52.558954+0000","last_fullsized":"2024-09-17T04:00:52.558954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:16.034449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493225+0000","last_change":"2024-09-17T04:00:49.485660+0000","last_active":"2024-09-17T04:00:51.493225+0000","last_peered":"2024-09-17T04:00:51.493225+0000","last_clean":"2024-09-17T04:00:51.493225+0000","last_became_active":"2024-09-17T04:00:49.484522+0000","last_became_peered":"2024-09-17T04:00:49.484522+0000","last_unstale":"2024-09-17T04:00:51.493225+0000","last_undegraded":"2024-09-17T04:00:51.493225+0000","last_fullsized":"2024-09-17T04:00:51.493225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:17.632102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490880+0000","last_change":"2024-09-17T04:00:50.493759+0000","last_active":"2024-09-17T04:00:51.490880+0000","last_peered":"2024-09-17T04:00:51.490880+0000","last_clean":"2024-09-17T04:00:51.490880+0000","last_became_active":"2024-09-17T04:00:50.493472+0000","last_became_peered":"2024-09-17T04:00:50.493472+0000","last_unstale":"2024-09-17T04:00:51.490880+0000","last_undegraded":"2024-09-17T04:00:51.490880+0000","last_fullsized":"2024-09-17T04:00:51.490880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:36.564425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492180+0000","last_change":"2024-09-17T04:00:49.485026+0000","last_active":"2024-09-17T04:00:51.492180+0000","last_peered":"2024-09-17T04:00:51.492180+0000","last_clean":"2024-09-17T04:00:51.492180+0000","last_became_active":"2024-09-17T04:00:49.484730+0000","last_became_peered":"2024-09-17T04:00:49.484730+0000","last_unstale":"2024-09-17T04:00:51.492180+0000","last_undegraded":"2024-09-17T04:00:51.492180+0000","last_fullsized":"2024-09-17T04:00:51.492180+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:09.639653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557800+0000","last_change":"2024-09-17T04:00:50.501781+0000","last_active":"2024-09-17T04:00:52.557800+0000","last_peered":"2024-09-17T04:00:52.557800+0000","last_clean":"2024-09-17T04:00:52.557800+0000","last_became_active":"2024-09-17T04:00:50.501192+0000","last_became_peered":"2024-09-17T04:00:50.501192+0000","last_unstale":"2024-09-17T04:00:52.557800+0000","last_undegraded":"2024-09-17T04:00:52.557800+0000","last_fullsized":"2024-09-17T04:00:52.557800+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:10.497603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475655+0000","last_change":"2024-09-17T04:00:49.492918+0000","last_active":"2024-09-17T04:00:52.475655+0000","last_peered":"2024-09-17T04:00:52.475655+0000","last_clean":"2024-09-17T04:00:52.475655+0000","last_became_active":"2024-09-17T04:00:49.492545+0000","last_became_peered":"2024-09-17T04:00:49.492545+0000","last_unstale":"2024-09-17T04:00:52.475655+0000","last_undegraded":"2024-09-17T04:00:52.475655+0000","last_fullsized":"2024-09-17T04:00:52.475655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:49.593934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475383+0000","last_change":"2024-09-17T04:00:50.498633+0000","last_active":"2024-09-17T04:00:52.475383+0000","last_peered":"2024-09-17T04:00:52.475383+0000","last_clean":"2024-09-17T04:00:52.475383+0000","last_became_active":"2024-09-17T04:00:50.498319+0000","last_became_peered":"2024-09-17T04:00:50.498319+0000","last_unstale":"2024-09-17T04:00:52.475383+0000","last_undegraded":"2024-09-17T04:00:52.475383+0000","last_fullsized":"2024-09-17T04:00:52.475383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:33.289244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191058+0000","last_change":"2024-09-17T04:00:49.484809+0000","last_active":"2024-09-17T04:00:52.191058+0000","last_peered":"2024-09-17T04:00:52.191058+0000","last_clean":"2024-09-17T04:00:52.191058+0000","last_became_active":"2024-09-17T04:00:49.484517+0000","last_became_peered":"2024-09-17T04:00:49.484517+0000","last_unstale":"2024-09-17T04:00:52.191058+0000","last_undegraded":"2024-09-17T04:00:52.191058+0000","last_fullsized":"2024-09-17T04:00:52.191058+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:55.826452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190819+0000","last_change":"2024-09-17T04:00:50.495545+0000","last_active":"2024-09-17T04:00:52.190819+0000","last_peered":"2024-09-17T04:00:52.190819+0000","last_clean":"2024-09-17T04:00:52.190819+0000","last_became_active":"2024-09-17T04:00:50.494856+0000","last_became_peered":"2024-09-17T04:00:50.494856+0000","last_unstale":"2024-09-17T04:00:52.190819+0000","last_undegraded":"2024-09-17T04:00:52.190819+0000","last_fullsized":"2024-09-17T04:00:52.190819+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:12.898308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190695+0000","last_change":"2024-09-17T04:00:49.483959+0000","last_active":"2024-09-17T04:00:52.190695+0000","last_peered":"2024-09-17T04:00:52.190695+0000","last_clean":"2024-09-17T04:00:52.190695+0000","last_became_active":"2024-09-17T04:00:49.483564+0000","last_became_peered":"2024-09-17T04:00:49.483564+0000","last_unstale":"2024-09-17T04:00:52.190695+0000","last_undegraded":"2024-09-17T04:00:52.190695+0000","last_fullsized":"2024-09-17T04:00:52.190695+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:03.426557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475394+0000","last_change":"2024-09-17T04:00:50.499998+0000","last_active":"2024-09-17T04:00:52.475394+0000","last_peered":"2024-09-17T04:00:52.475394+0000","last_clean":"2024-09-17T04:00:52.475394+0000","last_became_active":"2024-09-17T04:00:50.499605+0000","last_became_peered":"2024-09-17T04:00:50.499605+0000","last_unstale":"2024-09-17T04:00:52.475394+0000","last_undegraded":"2024-09-17T04:00:52.475394+0000","last_fullsized":"2024-09-17T04:00:52.475394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:04.373538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558123+0000","last_change":"2024-09-17T04:00:49.488756+0000","last_active":"2024-09-17T04:00:52.558123+0000","last_peered":"2024-09-17T04:00:52.558123+0000","last_clean":"2024-09-17T04:00:52.558123+0000","last_became_active":"2024-09-17T04:00:49.487367+0000","last_became_peered":"2024-09-17T04:00:49.487367+0000","last_unstale":"2024-09-17T04:00:52.558123+0000","last_undegraded":"2024-09-17T04:00:52.558123+0000","last_fullsized":"2024-09-17T04:00:52.558123+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475774+0000","last_change":"2024-09-17T04:00:50.504866+0000","last_active":"2024-09-17T04:00:52.475774+0000","last_peered":"2024-09-17T04:00:52.475774+0000","last_clean":"2024-09-17T04:00:52.475774+0000","last_became_active":"2024-09-17T04:00:50.504365+0000","last_became_peered":"2024-09-17T04:00:50.504365+0000","last_unstale":"2024-09-17T04:00:52.475774+0000","last_undegraded":"2024-09-17T04:00:52.475774+0000","last_fullsized":"2024-09-17T04:00:52.475774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:23.967019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191044+0000","last_change":"2024-09-17T04:00:49.484439+0000","last_active":"2024-09-17T04:00:52.191044+0000","last_peered":"2024-09-17T04:00:52.191044+0000","last_clean":"2024-09-17T04:00:52.191044+0000","last_became_active":"2024-09-17T04:00:49.483790+0000","last_became_peered":"2024-09-17T04:00:49.483790+0000","last_unstale":"2024-09-17T04:00:52.191044+0000","last_undegraded":"2024-09-17T04:00:52.191044+0000","last_fullsized":"2024-09-17T04:00:52.191044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492483+0000","last_change":"2024-09-17T04:00:50.490877+0000","last_active":"2024-09-17T04:00:51.492483+0000","last_peered":"2024-09-17T04:00:51.492483+0000","last_clean":"2024-09-17T04:00:51.492483+0000","last_became_active":"2024-09-17T04:00:50.490591+0000","last_became_peered":"2024-09-17T04:00:50.490591+0000","last_unstale":"2024-09-17T04:00:51.492483+0000","last_undegraded":"2024-09-17T04:00:51.492483+0000","last_fullsized":"2024-09-17T04:00:51.492483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:38.063668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558774+0000","last_change":"2024-09-17T04:00:49.481548+0000","last_active":"2024-09-17T04:00:52.558774+0000","last_peered":"2024-09-17T04:00:52.558774+0000","last_clean":"2024-09-17T04:00:52.558774+0000","last_became_active":"2024-09-17T04:00:49.479803+0000","last_became_peered":"2024-09-17T04:00:49.479803+0000","last_unstale":"2024-09-17T04:00:52.558774+0000","last_undegraded":"2024-09-17T04:00:52.558774+0000","last_fullsized":"2024-09-17T04:00:52.558774+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475673+0000","last_change":"2024-09-17T04:00:50.501872+0000","last_active":"2024-09-17T04:00:52.475673+0000","last_peered":"2024-09-17T04:00:52.475673+0000","last_clean":"2024-09-17T04:00:52.475673+0000","last_became_active":"2024-09-17T04:00:50.501566+0000","last_became_peered":"2024-09-17T04:00:50.501566+0000","last_unstale":"2024-09-17T04:00:52.475673+0000","last_undegraded":"2024-09-17T04:00:52.475673+0000","last_fullsized":"2024-09-17T04:00:52.475673+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.084351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"173'3","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558106+0000","last_change":"2024-09-17T04:00:49.487399+0000","last_active":"2024-09-17T04:00:52.558106+0000","last_peered":"2024-09-17T04:00:52.558106+0000","last_clean":"2024-09-17T04:00:52.558106+0000","last_became_active":"2024-09-17T04:00:49.487044+0000","last_became_peered":"2024-09-17T04:00:49.487044+0000","last_unstale":"2024-09-17T04:00:52.558106+0000","last_undegraded":"2024-09-17T04:00:52.558106+0000","last_fullsized":"2024-09-17T04:00:52.558106+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:30.425262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490623+0000","last_change":"2024-09-17T04:00:50.500587+0000","last_active":"2024-09-17T04:00:51.490623+0000","last_peered":"2024-09-17T04:00:51.490623+0000","last_clean":"2024-09-17T04:00:51.490623+0000","last_became_active":"2024-09-17T04:00:50.500282+0000","last_became_peered":"2024-09-17T04:00:50.500282+0000","last_unstale":"2024-09-17T04:00:51.490623+0000","last_undegraded":"2024-09-17T04:00:51.490623+0000","last_fullsized":"2024-09-17T04:00:51.490623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:49:57.370683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"173'6","reported_seq":21,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475785+0000","last_change":"2024-09-17T04:00:49.491252+0000","last_active":"2024-09-17T04:00:52.475785+0000","last_peered":"2024-09-17T04:00:52.475785+0000","last_clean":"2024-09-17T04:00:52.475785+0000","last_became_active":"2024-09-17T04:00:49.490892+0000","last_became_peered":"2024-09-17T04:00:49.490892+0000","last_unstale":"2024-09-17T04:00:52.475785+0000","last_undegraded":"2024-09-17T04:00:52.475785+0000","last_fullsized":"2024-09-17T04:00:52.475785+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:02.780252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190912+0000","last_change":"2024-09-17T04:00:50.503682+0000","last_active":"2024-09-17T04:00:52.190912+0000","last_peered":"2024-09-17T04:00:52.190912+0000","last_clean":"2024-09-17T04:00:52.190912+0000","last_became_active":"2024-09-17T04:00:50.503389+0000","last_became_peered":"2024-09-17T04:00:50.503389+0000","last_unstale":"2024-09-17T04:00:52.190912+0000","last_undegraded":"2024-09-17T04:00:52.190912+0000","last_fullsized":"2024-09-17T04:00:52.190912+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:51.815481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558379+0000","last_change":"2024-09-17T04:00:49.488763+0000","last_active":"2024-09-17T04:00:52.558379+0000","last_peered":"2024-09-17T04:00:52.558379+0000","last_clean":"2024-09-17T04:00:52.558379+0000","last_became_active":"2024-09-17T04:00:49.487670+0000","last_became_peered":"2024-09-17T04:00:49.487670+0000","last_unstale":"2024-09-17T04:00:52.558379+0000","last_undegraded":"2024-09-17T04:00:52.558379+0000","last_fullsized":"2024-09-17T04:00:52.558379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.500556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557927+0000","last_change":"2024-09-17T04:00:50.481453+0000","last_active":"2024-09-17T04:00:52.557927+0000","last_peered":"2024-09-17T04:00:52.557927+0000","last_clean":"2024-09-17T04:00:52.557927+0000","last_became_active":"2024-09-17T04:00:50.481137+0000","last_became_peered":"2024-09-17T04:00:50.481137+0000","last_unstale":"2024-09-17T04:00:52.557927+0000","last_undegraded":"2024-09-17T04:00:52.557927+0000","last_fullsized":"2024-09-17T04:00:52.557927+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:06.279433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558318+0000","last_change":"2024-09-17T04:00:49.492017+0000","last_active":"2024-09-17T04:00:52.558318+0000","last_peered":"2024-09-17T04:00:52.558318+0000","last_clean":"2024-09-17T04:00:52.558318+0000","last_became_active":"2024-09-17T04:00:49.491561+0000","last_became_peered":"2024-09-17T04:00:49.491561+0000","last_unstale":"2024-09-17T04:00:52.558318+0000","last_undegraded":"2024-09-17T04:00:52.558318+0000","last_fullsized":"2024-09-17T04:00:52.558318+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491405+0000","last_change":"2024-09-17T04:00:50.492894+0000","last_active":"2024-09-17T04:00:51.491405+0000","last_peered":"2024-09-17T04:00:51.491405+0000","last_clean":"2024-09-17T04:00:51.491405+0000","last_became_active":"2024-09-17T04:00:50.492614+0000","last_became_peered":"2024-09-17T04:00:50.492614+0000","last_unstale":"2024-09-17T04:00:51.491405+0000","last_undegraded":"2024-09-17T04:00:51.491405+0000","last_fullsized":"2024-09-17T04:00:51.491405+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:14.473607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557441+0000","last_change":"2024-09-17T04:00:49.482396+0000","last_active":"2024-09-17T04:00:52.557441+0000","last_peered":"2024-09-17T04:00:52.557441+0000","last_clean":"2024-09-17T04:00:52.557441+0000","last_became_active":"2024-09-17T04:00:49.481991+0000","last_became_peered":"2024-09-17T04:00:49.481991+0000","last_unstale":"2024-09-17T04:00:52.557441+0000","last_undegraded":"2024-09-17T04:00:52.557441+0000","last_fullsized":"2024-09-17T04:00:52.557441+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.860917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490625+0000","last_change":"2024-09-17T04:00:50.486171+0000","last_active":"2024-09-17T04:00:51.490625+0000","last_peered":"2024-09-17T04:00:51.490625+0000","last_clean":"2024-09-17T04:00:51.490625+0000","last_became_active":"2024-09-17T04:00:50.484422+0000","last_became_peered":"2024-09-17T04:00:50.484422+0000","last_unstale":"2024-09-17T04:00:51.490625+0000","last_undegraded":"2024-09-17T04:00:51.490625+0000","last_fullsized":"2024-09-17T04:00:51.490625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:09.236326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"173'8","reported_seq":23,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491608+0000","last_change":"2024-09-17T04:00:49.486848+0000","last_active":"2024-09-17T04:00:51.491608+0000","last_peered":"2024-09-17T04:00:51.491608+0000","last_clean":"2024-09-17T04:00:51.491608+0000","last_became_active":"2024-09-17T04:00:49.486463+0000","last_became_peered":"2024-09-17T04:00:49.486463+0000","last_unstale":"2024-09-17T04:00:51.491608+0000","last_undegraded":"2024-09-17T04:00:51.491608+0000","last_fullsized":"2024-09-17T04:00:51.491608+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:41.797680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493377+0000","last_change":"2024-09-17T04:00:50.495171+0000","last_active":"2024-09-17T04:00:51.493377+0000","last_peered":"2024-09-17T04:00:51.493377+0000","last_clean":"2024-09-17T04:00:51.493377+0000","last_became_active":"2024-09-17T04:00:50.494834+0000","last_became_peered":"2024-09-17T04:00:50.494834+0000","last_unstale":"2024-09-17T04:00:51.493377+0000","last_undegraded":"2024-09-17T04:00:51.493377+0000","last_fullsized":"2024-09-17T04:00:51.493377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:22.013808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191210+0000","last_change":"2024-09-17T04:00:49.487492+0000","last_active":"2024-09-17T04:00:52.191210+0000","last_peered":"2024-09-17T04:00:52.191210+0000","last_clean":"2024-09-17T04:00:52.191210+0000","last_became_active":"2024-09-17T04:00:49.487146+0000","last_became_peered":"2024-09-17T04:00:49.487146+0000","last_unstale":"2024-09-17T04:00:52.191210+0000","last_undegraded":"2024-09-17T04:00:52.191210+0000","last_fullsized":"2024-09-17T04:00:52.191210+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493310+0000","last_change":"2024-09-17T04:00:50.484922+0000","last_active":"2024-09-17T04:00:51.493310+0000","last_peered":"2024-09-17T04:00:51.493310+0000","last_clean":"2024-09-17T04:00:51.493310+0000","last_became_active":"2024-09-17T04:00:50.483724+0000","last_became_peered":"2024-09-17T04:00:50.483724+0000","last_unstale":"2024-09-17T04:00:51.493310+0000","last_undegraded":"2024-09-17T04:00:51.493310+0000","last_fullsized":"2024-09-17T04:00:51.493310+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:40.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493354+0000","last_change":"2024-09-17T04:00:49.475839+0000","last_active":"2024-09-17T04:00:51.493354+0000","last_peered":"2024-09-17T04:00:51.493354+0000","last_clean":"2024-09-17T04:00:51.493354+0000","last_became_active":"2024-09-17T04:00:49.475239+0000","last_became_peered":"2024-09-17T04:00:49.475239+0000","last_unstale":"2024-09-17T04:00:51.493354+0000","last_undegraded":"2024-09-17T04:00:51.493354+0000","last_fullsized":"2024-09-17T04:00:51.493354+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:14.501945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491480+0000","last_change":"2024-09-17T04:00:50.504359+0000","last_active":"2024-09-17T04:00:51.491480+0000","last_peered":"2024-09-17T04:00:51.491480+0000","last_clean":"2024-09-17T04:00:51.491480+0000","last_became_active":"2024-09-17T04:00:50.501126+0000","last_became_peered":"2024-09-17T04:00:50.501126+0000","last_unstale":"2024-09-17T04:00:51.491480+0000","last_undegraded":"2024-09-17T04:00:51.491480+0000","last_fullsized":"2024-09-17T04:00:51.491480+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:20.943413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557974+0000","last_change":"2024-09-17T04:00:49.485542+0000","last_active":"2024-09-17T04:00:52.557974+0000","last_peered":"2024-09-17T04:00:52.557974+0000","last_clean":"2024-09-17T04:00:52.557974+0000","last_became_active":"2024-09-17T04:00:49.481811+0000","last_became_peered":"2024-09-17T04:00:49.481811+0000","last_unstale":"2024-09-17T04:00:52.557974+0000","last_undegraded":"2024-09-17T04:00:52.557974+0000","last_fullsized":"2024-09-17T04:00:52.557974+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:53.374454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557930+0000","last_change":"2024-09-17T04:00:50.495216+0000","last_active":"2024-09-17T04:00:52.557930+0000","last_peered":"2024-09-17T04:00:52.557930+0000","last_clean":"2024-09-17T04:00:52.557930+0000","last_became_active":"2024-09-17T04:00:50.494947+0000","last_became_peered":"2024-09-17T04:00:50.494947+0000","last_unstale":"2024-09-17T04:00:52.557930+0000","last_undegraded":"2024-09-17T04:00:52.557930+0000","last_fullsized":"2024-09-17T04:00:52.557930+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:24:59.919453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491546+0000","last_change":"2024-09-17T04:00:49.485281+0000","last_active":"2024-09-17T04:00:51.491546+0000","last_peered":"2024-09-17T04:00:51.491546+0000","last_clean":"2024-09-17T04:00:51.491546+0000","last_became_active":"2024-09-17T04:00:49.484098+0000","last_became_peered":"2024-09-17T04:00:49.484098+0000","last_unstale":"2024-09-17T04:00:51.491546+0000","last_undegraded":"2024-09-17T04:00:51.491546+0000","last_fullsized":"2024-09-17T04:00:51.491546+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:08.975039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557656+0000","last_change":"2024-09-17T04:00:50.505179+0000","last_active":"2024-09-17T04:00:52.557656+0000","last_peered":"2024-09-17T04:00:52.557656+0000","last_clean":"2024-09-17T04:00:52.557656+0000","last_became_active":"2024-09-17T04:00:50.504729+0000","last_became_peered":"2024-09-17T04:00:50.504729+0000","last_unstale":"2024-09-17T04:00:52.557656+0000","last_undegraded":"2024-09-17T04:00:52.557656+0000","last_fullsized":"2024-09-17T04:00:52.557656+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:53.716965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558080+0000","last_change":"2024-09-17T04:00:49.479444+0000","last_active":"2024-09-17T04:00:52.558080+0000","last_peered":"2024-09-17T04:00:52.558080+0000","last_clean":"2024-09-17T04:00:52.558080+0000","last_became_active":"2024-09-17T04:00:49.479095+0000","last_became_peered":"2024-09-17T04:00:49.479095+0000","last_unstale":"2024-09-17T04:00:52.558080+0000","last_undegraded":"2024-09-17T04:00:52.558080+0000","last_fullsized":"2024-09-17T04:00:52.558080+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:25.795248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491625+0000","last_change":"2024-09-17T04:00:50.507841+0000","last_active":"2024-09-17T04:00:51.491625+0000","last_peered":"2024-09-17T04:00:51.491625+0000","last_clean":"2024-09-17T04:00:51.491625+0000","last_became_active":"2024-09-17T04:00:50.507577+0000","last_became_peered":"2024-09-17T04:00:50.507577+0000","last_unstale":"2024-09-17T04:00:51.491625+0000","last_undegraded":"2024-09-17T04:00:51.491625+0000","last_fullsized":"2024-09-17T04:00:51.491625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:51.506216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490347+0000","last_change":"2024-09-17T04:00:49.492248+0000","last_active":"2024-09-17T04:00:51.490347+0000","last_peered":"2024-09-17T04:00:51.490347+0000","last_clean":"2024-09-17T04:00:51.490347+0000","last_became_active":"2024-09-17T04:00:49.491744+0000","last_became_peered":"2024-09-17T04:00:49.491744+0000","last_unstale":"2024-09-17T04:00:51.490347+0000","last_undegraded":"2024-09-17T04:00:51.490347+0000","last_fullsized":"2024-09-17T04:00:51.490347+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:45.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492551+0000","last_change":"2024-09-17T04:00:50.485354+0000","last_active":"2024-09-17T04:00:51.492551+0000","last_peered":"2024-09-17T04:00:51.492551+0000","last_clean":"2024-09-17T04:00:51.492551+0000","last_became_active":"2024-09-17T04:00:50.485017+0000","last_became_peered":"2024-09-17T04:00:50.485017+0000","last_unstale":"2024-09-17T04:00:51.492551+0000","last_undegraded":"2024-09-17T04:00:51.492551+0000","last_fullsized":"2024-09-17T04:00:51.492551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:43.479866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490209+0000","last_change":"2024-09-17T04:00:49.490198+0000","last_active":"2024-09-17T04:00:51.490209+0000","last_peered":"2024-09-17T04:00:51.490209+0000","last_clean":"2024-09-17T04:00:51.490209+0000","last_became_active":"2024-09-17T04:00:49.489685+0000","last_became_peered":"2024-09-17T04:00:49.489685+0000","last_unstale":"2024-09-17T04:00:51.490209+0000","last_undegraded":"2024-09-17T04:00:51.490209+0000","last_fullsized":"2024-09-17T04:00:51.490209+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:21.987330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558604+0000","last_change":"2024-09-17T04:00:50.488092+0000","last_active":"2024-09-17T04:00:52.558604+0000","last_peered":"2024-09-17T04:00:52.558604+0000","last_clean":"2024-09-17T04:00:52.558604+0000","last_became_active":"2024-09-17T04:00:50.487718+0000","last_became_peered":"2024-09-17T04:00:50.487718+0000","last_unstale":"2024-09-17T04:00:52.558604+0000","last_undegraded":"2024-09-17T04:00:52.558604+0000","last_fullsized":"2024-09-17T04:00:52.558604+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:12.457060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190954+0000","last_change":"2024-09-17T04:00:49.492610+0000","last_active":"2024-09-17T04:00:52.190954+0000","last_peered":"2024-09-17T04:00:52.190954+0000","last_clean":"2024-09-17T04:00:52.190954+0000","last_became_active":"2024-09-17T04:00:49.491506+0000","last_became_peered":"2024-09-17T04:00:49.491506+0000","last_unstale":"2024-09-17T04:00:52.190954+0000","last_undegraded":"2024-09-17T04:00:52.190954+0000","last_fullsized":"2024-09-17T04:00:52.190954+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954545+0000","last_change":"2024-09-17T04:00:50.511428+0000","last_active":"2024-09-17T04:00:51.954545+0000","last_peered":"2024-09-17T04:00:51.954545+0000","last_clean":"2024-09-17T04:00:51.954545+0000","last_became_active":"2024-09-17T04:00:50.511114+0000","last_became_peered":"2024-09-17T04:00:50.511114+0000","last_unstale":"2024-09-17T04:00:51.954545+0000","last_undegraded":"2024-09-17T04:00:51.954545+0000","last_fullsized":"2024-09-17T04:00:51.954545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:58.184294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492161+0000","last_change":"2024-09-17T04:00:49.478561+0000","last_active":"2024-09-17T04:00:51.492161+0000","last_peered":"2024-09-17T04:00:51.492161+0000","last_clean":"2024-09-17T04:00:51.492161+0000","last_became_active":"2024-09-17T04:00:49.478248+0000","last_became_peered":"2024-09-17T04:00:49.478248+0000","last_unstale":"2024-09-17T04:00:51.492161+0000","last_undegraded":"2024-09-17T04:00:51.492161+0000","last_fullsized":"2024-09-17T04:00:51.492161+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:59.810080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490983+0000","last_change":"2024-09-17T04:00:50.495573+0000","last_active":"2024-09-17T04:00:51.490983+0000","last_peered":"2024-09-17T04:00:51.490983+0000","last_clean":"2024-09-17T04:00:51.490983+0000","last_became_active":"2024-09-17T04:00:50.494895+0000","last_became_peered":"2024-09-17T04:00:50.494895+0000","last_unstale":"2024-09-17T04:00:51.490983+0000","last_undegraded":"2024-09-17T04:00:51.490983+0000","last_fullsized":"2024-09-17T04:00:51.490983+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:07.480340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"174'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475720+0000","last_change":"2024-09-17T04:00:49.490303+0000","last_active":"2024-09-17T04:00:52.475720+0000","last_peered":"2024-09-17T04:00:52.475720+0000","last_clean":"2024-09-17T04:00:52.475720+0000","last_became_active":"2024-09-17T04:00:49.489882+0000","last_became_peered":"2024-09-17T04:00:49.489882+0000","last_unstale":"2024-09-17T04:00:52.475720+0000","last_undegraded":"2024-09-17T04:00:52.475720+0000","last_fullsized":"2024-09-17T04:00:52.475720+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:46.567261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492851+0000","last_change":"2024-09-17T04:00:50.493244+0000","last_active":"2024-09-17T04:00:51.492851+0000","last_peered":"2024-09-17T04:00:51.492851+0000","last_clean":"2024-09-17T04:00:51.492851+0000","last_became_active":"2024-09-17T04:00:50.492892+0000","last_became_peered":"2024-09-17T04:00:50.492892+0000","last_unstale":"2024-09-17T04:00:51.492851+0000","last_undegraded":"2024-09-17T04:00:51.492851+0000","last_fullsized":"2024-09-17T04:00:51.492851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:19.515809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191177+0000","last_change":"2024-09-17T04:00:49.480983+0000","last_active":"2024-09-17T04:00:52.191177+0000","last_peered":"2024-09-17T04:00:52.191177+0000","last_clean":"2024-09-17T04:00:52.191177+0000","last_became_active":"2024-09-17T04:00:49.480620+0000","last_became_peered":"2024-09-17T04:00:49.480620+0000","last_unstale":"2024-09-17T04:00:52.191177+0000","last_undegraded":"2024-09-17T04:00:52.191177+0000","last_fullsized":"2024-09-17T04:00:52.191177+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:28.065208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558754+0000","last_change":"2024-09-17T04:00:50.491601+0000","last_active":"2024-09-17T04:00:52.558754+0000","last_peered":"2024-09-17T04:00:52.558754+0000","last_clean":"2024-09-17T04:00:52.558754+0000","last_became_active":"2024-09-17T04:00:50.491279+0000","last_became_peered":"2024-09-17T04:00:50.491279+0000","last_unstale":"2024-09-17T04:00:52.558754+0000","last_undegraded":"2024-09-17T04:00:52.558754+0000","last_fullsized":"2024-09-17T04:00:52.558754+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:03.782918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491657+0000","last_change":"2024-09-17T04:00:49.477561+0000","last_active":"2024-09-17T04:00:51.491657+0000","last_peered":"2024-09-17T04:00:51.491657+0000","last_clean":"2024-09-17T04:00:51.491657+0000","last_became_active":"2024-09-17T04:00:49.477252+0000","last_became_peered":"2024-09-17T04:00:49.477252+0000","last_unstale":"2024-09-17T04:00:51.491657+0000","last_undegraded":"2024-09-17T04:00:51.491657+0000","last_fullsized":"2024-09-17T04:00:51.491657+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:07.327645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490348+0000","last_change":"2024-09-17T04:00:50.499748+0000","last_active":"2024-09-17T04:00:51.490348+0000","last_peered":"2024-09-17T04:00:51.490348+0000","last_clean":"2024-09-17T04:00:51.490348+0000","last_became_active":"2024-09-17T04:00:50.497933+0000","last_became_peered":"2024-09-17T04:00:50.497933+0000","last_unstale":"2024-09-17T04:00:51.490348+0000","last_undegraded":"2024-09-17T04:00:51.490348+0000","last_fullsized":"2024-09-17T04:00:51.490348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:33.887766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475476+0000","last_change":"2024-09-17T04:00:49.486586+0000","last_active":"2024-09-17T04:00:52.475476+0000","last_peered":"2024-09-17T04:00:52.475476+0000","last_clean":"2024-09-17T04:00:52.475476+0000","last_became_active":"2024-09-17T04:00:49.486099+0000","last_became_peered":"2024-09-17T04:00:49.486099+0000","last_unstale":"2024-09-17T04:00:52.475476+0000","last_undegraded":"2024-09-17T04:00:52.475476+0000","last_fullsized":"2024-09-17T04:00:52.475476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:42.671635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954355+0000","last_change":"2024-09-17T04:00:50.501002+0000","last_active":"2024-09-17T04:00:51.954355+0000","last_peered":"2024-09-17T04:00:51.954355+0000","last_clean":"2024-09-17T04:00:51.954355+0000","last_became_active":"2024-09-17T04:00:50.500621+0000","last_became_peered":"2024-09-17T04:00:50.500621+0000","last_unstale":"2024-09-17T04:00:51.954355+0000","last_undegraded":"2024-09-17T04:00:51.954355+0000","last_fullsized":"2024-09-17T04:00:51.954355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:38.586287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191137+0000","last_change":"2024-09-17T04:00:49.483997+0000","last_active":"2024-09-17T04:00:52.191137+0000","last_peered":"2024-09-17T04:00:52.191137+0000","last_clean":"2024-09-17T04:00:52.191137+0000","last_became_active":"2024-09-17T04:00:49.483458+0000","last_became_peered":"2024-09-17T04:00:49.483458+0000","last_unstale":"2024-09-17T04:00:52.191137+0000","last_undegraded":"2024-09-17T04:00:52.191137+0000","last_fullsized":"2024-09-17T04:00:52.191137+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:09.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491343+0000","last_change":"2024-09-17T04:00:50.492815+0000","last_active":"2024-09-17T04:00:51.491343+0000","last_peered":"2024-09-17T04:00:51.491343+0000","last_clean":"2024-09-17T04:00:51.491343+0000","last_became_active":"2024-09-17T04:00:50.492209+0000","last_became_peered":"2024-09-17T04:00:50.492209+0000","last_unstale":"2024-09-17T04:00:51.491343+0000","last_undegraded":"2024-09-17T04:00:51.491343+0000","last_fullsized":"2024-09-17T04:00:51.491343+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:15.504611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558691+0000","last_change":"2024-09-17T04:00:49.485410+0000","last_active":"2024-09-17T04:00:52.558691+0000","last_peered":"2024-09-17T04:00:52.558691+0000","last_clean":"2024-09-17T04:00:52.558691+0000","last_became_active":"2024-09-17T04:00:49.484171+0000","last_became_peered":"2024-09-17T04:00:49.484171+0000","last_unstale":"2024-09-17T04:00:52.558691+0000","last_undegraded":"2024-09-17T04:00:52.558691+0000","last_fullsized":"2024-09-17T04:00:52.558691+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492593+0000","last_change":"2024-09-17T04:00:50.490365+0000","last_active":"2024-09-17T04:00:51.492593+0000","last_peered":"2024-09-17T04:00:51.492593+0000","last_clean":"2024-09-17T04:00:51.492593+0000","last_became_active":"2024-09-17T04:00:50.488098+0000","last_became_peered":"2024-09-17T04:00:50.488098+0000","last_unstale":"2024-09-17T04:00:51.492593+0000","last_undegraded":"2024-09-17T04:00:51.492593+0000","last_fullsized":"2024-09-17T04:00:51.492593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:42.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954116+0000","last_change":"2024-09-17T04:00:49.487212+0000","last_active":"2024-09-17T04:00:51.954116+0000","last_peered":"2024-09-17T04:00:51.954116+0000","last_clean":"2024-09-17T04:00:51.954116+0000","last_became_active":"2024-09-17T04:00:49.486849+0000","last_became_peered":"2024-09-17T04:00:49.486849+0000","last_unstale":"2024-09-17T04:00:51.954116+0000","last_undegraded":"2024-09-17T04:00:51.954116+0000","last_fullsized":"2024-09-17T04:00:51.954116+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:31.908526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490138+0000","last_change":"2024-09-17T04:00:50.501206+0000","last_active":"2024-09-17T04:00:51.490138+0000","last_peered":"2024-09-17T04:00:51.490138+0000","last_clean":"2024-09-17T04:00:51.490138+0000","last_became_active":"2024-09-17T04:00:50.500932+0000","last_became_peered":"2024-09-17T04:00:50.500932+0000","last_unstale":"2024-09-17T04:00:51.490138+0000","last_undegraded":"2024-09-17T04:00:51.490138+0000","last_fullsized":"2024-09-17T04:00:51.490138+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:32.987666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954611+0000","last_change":"2024-09-17T04:00:49.489179+0000","last_active":"2024-09-17T04:00:51.954611+0000","last_peered":"2024-09-17T04:00:51.954611+0000","last_clean":"2024-09-17T04:00:51.954611+0000","last_became_active":"2024-09-17T04:00:49.488400+0000","last_became_peered":"2024-09-17T04:00:49.488400+0000","last_unstale":"2024-09-17T04:00:51.954611+0000","last_undegraded":"2024-09-17T04:00:51.954611+0000","last_fullsized":"2024-09-17T04:00:51.954611+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:44:31.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":392,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475314+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:52.475314+0000","last_peered":"2024-09-17T04:00:52.475314+0000","last_clean":"2024-09-17T04:00:52.475314+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:52.475314+0000","last_undegraded":"2024-09-17T04:00:52.475314+0000","last_fullsized":"2024-09-17T04:00:52.475314+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509580,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2793472,"data_stored":1549678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2207744,"data_stored":959414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509580,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2199552,"data_stored":957880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509582,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2187264,"data_stored":957698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509582,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2195456,"data_stored":957754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542284,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542285,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2785280,"data_stored":1548156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:00:59.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:01:00.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 -- 172.21.15.129:0/1788537636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c0cfb60 msgr2=0x7f9f4c160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 --2- 172.21.15.129:0/1788537636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c160cf0 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c009830 tx=0x7f9f3c02a520 comp rx=0 tx=0).stop 2024-09-17T04:01:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 -- 172.21.15.129:0/1788537636 shutdown_connections 2024-09-17T04:01:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 --2- 172.21.15.129:0/1788537636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c160cf0 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 -- 172.21.15.129:0/1788537636 >> 172.21.15.129:0/1788537636 conn(0x7f9f4c06cfc0 msgr2=0x7f9f4c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 -- 172.21.15.129:0/1788537636 shutdown_connections 2024-09-17T04:01:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.159+0000 7f9f52000640 1 -- 172.21.15.129:0/1788537636 wait complete. 2024-09-17T04:01:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.161+0000 7f9f52000640 1 Processor -- start 2024-09-17T04:01:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.161+0000 7f9f52000640 1 -- start start 2024-09-17T04:01:00.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f4c0b46e0 0x7f9f4c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c1f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c15d460 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f4c15d2e0 con 0x7f9f4c0cfb60 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f52000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9f4c15d5e0 con 0x7f9f4c0b46e0 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f51600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f51600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35224/0 (socket says 172.21.15.129:35224) 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f51600640 1 -- 172.21.15.129:0/4047692876 learned_addr learned my addr 172.21.15.129:0/4047692876 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f4be00640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c1f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.163+0000 7f9f50c00640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f4c0b46e0 0x7f9f4c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 -- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f4c0b46e0 msgr2=0x7f9f4c15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f4c0b46e0 0x7f9f4c15b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 -- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f4c0cfb60 msgr2=0x7f9f4c1f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c1f6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 -- 172.21.15.129:0/4047692876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f4c1589f0 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f51600640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c700 tx=0x7f9f4400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44003b70 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f44003d10 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400aab0 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.164+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f4c1f4560 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.165+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f4c1f4b20 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.166+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f4c158bc0 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.167+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9f4400ac50 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.168+0000 7f9f48c00640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 0x7f9f2c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.168+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f9f4409aa60 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.168+0000 7f9f50c00640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 0x7f9f2c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.169+0000 7f9f50c00640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 0x7f9f2c07b0a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c034000 tx=0x7f9f3c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:00.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.172+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9f440627b0 con 0x7f9f4c15b8d0 2024-09-17T04:01:00.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.355+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f4c160cf0 con 0x7f9f2c078b90 2024-09-17T04:01:00.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.360+0000 7f9f48c00640 1 -- 172.21.15.129:0/4047692876 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252452 (secure 0 0 0) 0x7f9f4c160cf0 con 0x7f9f2c078b90 2024-09-17T04:01:00.363 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:00.364 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:01:00.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 msgr2=0x7f9f2c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 0x7f9f2c07b0a0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c034000 tx=0x7f9f3c004730 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 msgr2=0x7f9f4c1507f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400c700 tx=0x7f9f4400cbc0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 shutdown_connections 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f2c078b90 0x7f9f2c07b0a0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f4c15b8d0 0x7f9f4c1507f0 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f4c0cfb60 0x7f9f4c1f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 --2- 172.21.15.129:0/4047692876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f4c0b46e0 0x7f9f4c15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 >> 172.21.15.129:0/4047692876 conn(0x7f9f4c06cfc0 msgr2=0x7f9f4c154a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 shutdown_connections 2024-09-17T04:01:00.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.364+0000 7f9f52000640 1 -- 172.21.15.129:0/4047692876 wait complete. 2024-09-17T04:01:00.387 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-17T04:00:58.488404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326072,"kb_used_data":18312,"kb_used_omap":12,"kb_used_meta":307571,"kb_avail":754648648,"statfs":{"total":773094113280,"available":772760215552,"internally_reserved":0,"allocated":18751488,"data_stored":8847140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":314953264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.017409"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490698+0000","last_change":"2024-09-17T04:00:50.499023+0000","last_active":"2024-09-17T04:00:51.490698+0000","last_peered":"2024-09-17T04:00:51.490698+0000","last_clean":"2024-09-17T04:00:51.490698+0000","last_became_active":"2024-09-17T04:00:50.498702+0000","last_became_peered":"2024-09-17T04:00:50.498702+0000","last_unstale":"2024-09-17T04:00:51.490698+0000","last_undegraded":"2024-09-17T04:00:51.490698+0000","last_fullsized":"2024-09-17T04:00:51.490698+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:47.660122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493082+0000","last_change":"2024-09-17T04:00:50.481292+0000","last_active":"2024-09-17T04:00:51.493082+0000","last_peered":"2024-09-17T04:00:51.493082+0000","last_clean":"2024-09-17T04:00:51.493082+0000","last_became_active":"2024-09-17T04:00:50.480948+0000","last_became_peered":"2024-09-17T04:00:50.480948+0000","last_unstale":"2024-09-17T04:00:51.493082+0000","last_undegraded":"2024-09-17T04:00:51.493082+0000","last_fullsized":"2024-09-17T04:00:51.493082+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:02.354995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491992+0000","last_change":"2024-09-17T04:00:50.496576+0000","last_active":"2024-09-17T04:00:51.491992+0000","last_peered":"2024-09-17T04:00:51.491992+0000","last_clean":"2024-09-17T04:00:51.491992+0000","last_became_active":"2024-09-17T04:00:50.495701+0000","last_became_peered":"2024-09-17T04:00:50.495701+0000","last_unstale":"2024-09-17T04:00:51.491992+0000","last_undegraded":"2024-09-17T04:00:51.491992+0000","last_fullsized":"2024-09-17T04:00:51.491992+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:50.676585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475438+0000","last_change":"2024-09-17T04:00:50.500751+0000","last_active":"2024-09-17T04:00:52.475438+0000","last_peered":"2024-09-17T04:00:52.475438+0000","last_clean":"2024-09-17T04:00:52.475438+0000","last_became_active":"2024-09-17T04:00:50.500022+0000","last_became_peered":"2024-09-17T04:00:50.500022+0000","last_unstale":"2024-09-17T04:00:52.475438+0000","last_undegraded":"2024-09-17T04:00:52.475438+0000","last_fullsized":"2024-09-17T04:00:52.475438+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:04.192066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191098+0000","last_change":"2024-09-17T04:00:50.505448+0000","last_active":"2024-09-17T04:00:52.191098+0000","last_peered":"2024-09-17T04:00:52.191098+0000","last_clean":"2024-09-17T04:00:52.191098+0000","last_became_active":"2024-09-17T04:00:50.505068+0000","last_became_peered":"2024-09-17T04:00:50.505068+0000","last_unstale":"2024-09-17T04:00:52.191098+0000","last_undegraded":"2024-09-17T04:00:52.191098+0000","last_fullsized":"2024-09-17T04:00:52.191098+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:18.315448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190863+0000","last_change":"2024-09-17T04:00:50.502154+0000","last_active":"2024-09-17T04:00:52.190863+0000","last_peered":"2024-09-17T04:00:52.190863+0000","last_clean":"2024-09-17T04:00:52.190863+0000","last_became_active":"2024-09-17T04:00:50.501768+0000","last_became_peered":"2024-09-17T04:00:50.501768+0000","last_unstale":"2024-09-17T04:00:52.190863+0000","last_undegraded":"2024-09-17T04:00:52.190863+0000","last_fullsized":"2024-09-17T04:00:52.190863+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:02.863583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490063+0000","last_change":"2024-09-17T04:00:50.488121+0000","last_active":"2024-09-17T04:00:51.490063+0000","last_peered":"2024-09-17T04:00:51.490063+0000","last_clean":"2024-09-17T04:00:51.490063+0000","last_became_active":"2024-09-17T04:00:50.487823+0000","last_became_peered":"2024-09-17T04:00:50.487823+0000","last_unstale":"2024-09-17T04:00:51.490063+0000","last_undegraded":"2024-09-17T04:00:51.490063+0000","last_fullsized":"2024-09-17T04:00:51.490063+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:27.505920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190744+0000","last_change":"2024-09-17T04:00:50.495847+0000","last_active":"2024-09-17T04:00:52.190744+0000","last_peered":"2024-09-17T04:00:52.190744+0000","last_clean":"2024-09-17T04:00:52.190744+0000","last_became_active":"2024-09-17T04:00:50.495220+0000","last_became_peered":"2024-09-17T04:00:50.495220+0000","last_unstale":"2024-09-17T04:00:52.190744+0000","last_undegraded":"2024-09-17T04:00:52.190744+0000","last_fullsized":"2024-09-17T04:00:52.190744+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:11.747697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493544+0000","last_change":"2024-09-17T04:00:50.494343+0000","last_active":"2024-09-17T04:00:51.493544+0000","last_peered":"2024-09-17T04:00:51.493544+0000","last_clean":"2024-09-17T04:00:51.493544+0000","last_became_active":"2024-09-17T04:00:50.493022+0000","last_became_peered":"2024-09-17T04:00:50.493022+0000","last_unstale":"2024-09-17T04:00:51.493544+0000","last_undegraded":"2024-09-17T04:00:51.493544+0000","last_fullsized":"2024-09-17T04:00:51.493544+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:20.950951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558374+0000","last_change":"2024-09-17T04:00:50.493475+0000","last_active":"2024-09-17T04:00:52.558374+0000","last_peered":"2024-09-17T04:00:52.558374+0000","last_clean":"2024-09-17T04:00:52.558374+0000","last_became_active":"2024-09-17T04:00:50.493182+0000","last_became_peered":"2024-09-17T04:00:50.493182+0000","last_unstale":"2024-09-17T04:00:52.558374+0000","last_undegraded":"2024-09-17T04:00:52.558374+0000","last_fullsized":"2024-09-17T04:00:52.558374+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:03.170026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492137+0000","last_change":"2024-09-17T04:00:50.497695+0000","last_active":"2024-09-17T04:00:51.492137+0000","last_peered":"2024-09-17T04:00:51.492137+0000","last_clean":"2024-09-17T04:00:51.492137+0000","last_became_active":"2024-09-17T04:00:50.497361+0000","last_became_peered":"2024-09-17T04:00:50.497361+0000","last_unstale":"2024-09-17T04:00:51.492137+0000","last_undegraded":"2024-09-17T04:00:51.492137+0000","last_fullsized":"2024-09-17T04:00:51.492137+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:01.343172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954479+0000","last_change":"2024-09-17T04:00:50.503620+0000","last_active":"2024-09-17T04:00:51.954479+0000","last_peered":"2024-09-17T04:00:51.954479+0000","last_clean":"2024-09-17T04:00:51.954479+0000","last_became_active":"2024-09-17T04:00:50.503238+0000","last_became_peered":"2024-09-17T04:00:50.503238+0000","last_unstale":"2024-09-17T04:00:51.954479+0000","last_undegraded":"2024-09-17T04:00:51.954479+0000","last_fullsized":"2024-09-17T04:00:51.954479+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:10.261218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954637+0000","last_change":"2024-09-17T04:00:50.502335+0000","last_active":"2024-09-17T04:00:51.954637+0000","last_peered":"2024-09-17T04:00:51.954637+0000","last_clean":"2024-09-17T04:00:51.954637+0000","last_became_active":"2024-09-17T04:00:50.502036+0000","last_became_peered":"2024-09-17T04:00:50.502036+0000","last_unstale":"2024-09-17T04:00:51.954637+0000","last_undegraded":"2024-09-17T04:00:51.954637+0000","last_fullsized":"2024-09-17T04:00:51.954637+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:37.502247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490230+0000","last_change":"2024-09-17T04:00:50.501289+0000","last_active":"2024-09-17T04:00:51.490230+0000","last_peered":"2024-09-17T04:00:51.490230+0000","last_clean":"2024-09-17T04:00:51.490230+0000","last_became_active":"2024-09-17T04:00:50.501012+0000","last_became_peered":"2024-09-17T04:00:50.501012+0000","last_unstale":"2024-09-17T04:00:51.490230+0000","last_undegraded":"2024-09-17T04:00:51.490230+0000","last_fullsized":"2024-09-17T04:00:51.490230+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:36.532175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490074+0000","last_change":"2024-09-17T04:00:50.490115+0000","last_active":"2024-09-17T04:00:51.490074+0000","last_peered":"2024-09-17T04:00:51.490074+0000","last_clean":"2024-09-17T04:00:51.490074+0000","last_became_active":"2024-09-17T04:00:50.489826+0000","last_became_peered":"2024-09-17T04:00:50.489826+0000","last_unstale":"2024-09-17T04:00:51.490074+0000","last_undegraded":"2024-09-17T04:00:51.490074+0000","last_fullsized":"2024-09-17T04:00:51.490074+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:11.406297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557529+0000","last_change":"2024-09-17T04:00:50.489115+0000","last_active":"2024-09-17T04:00:52.557529+0000","last_peered":"2024-09-17T04:00:52.557529+0000","last_clean":"2024-09-17T04:00:52.557529+0000","last_became_active":"2024-09-17T04:00:50.488820+0000","last_became_peered":"2024-09-17T04:00:50.488820+0000","last_unstale":"2024-09-17T04:00:52.557529+0000","last_undegraded":"2024-09-17T04:00:52.557529+0000","last_fullsized":"2024-09-17T04:00:52.557529+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:50.511222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490971+0000","last_change":"2024-09-17T04:00:50.497620+0000","last_active":"2024-09-17T04:00:51.490971+0000","last_peered":"2024-09-17T04:00:51.490971+0000","last_clean":"2024-09-17T04:00:51.490971+0000","last_became_active":"2024-09-17T04:00:50.495270+0000","last_became_peered":"2024-09-17T04:00:50.495270+0000","last_unstale":"2024-09-17T04:00:51.490971+0000","last_undegraded":"2024-09-17T04:00:51.490971+0000","last_fullsized":"2024-09-17T04:00:51.490971+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:29.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558396+0000","last_change":"2024-09-17T04:00:50.496642+0000","last_active":"2024-09-17T04:00:52.558396+0000","last_peered":"2024-09-17T04:00:52.558396+0000","last_clean":"2024-09-17T04:00:52.558396+0000","last_became_active":"2024-09-17T04:00:50.496350+0000","last_became_peered":"2024-09-17T04:00:50.496350+0000","last_unstale":"2024-09-17T04:00:52.558396+0000","last_undegraded":"2024-09-17T04:00:52.558396+0000","last_fullsized":"2024-09-17T04:00:52.558396+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:33.152814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558136+0000","last_change":"2024-09-17T04:00:50.492275+0000","last_active":"2024-09-17T04:00:52.558136+0000","last_peered":"2024-09-17T04:00:52.558136+0000","last_clean":"2024-09-17T04:00:52.558136+0000","last_became_active":"2024-09-17T04:00:50.491684+0000","last_became_peered":"2024-09-17T04:00:50.491684+0000","last_unstale":"2024-09-17T04:00:52.558136+0000","last_undegraded":"2024-09-17T04:00:52.558136+0000","last_fullsized":"2024-09-17T04:00:52.558136+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:21.696817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191282+0000","last_change":"2024-09-17T04:00:50.496943+0000","last_active":"2024-09-17T04:00:52.191282+0000","last_peered":"2024-09-17T04:00:52.191282+0000","last_clean":"2024-09-17T04:00:52.191282+0000","last_became_active":"2024-09-17T04:00:50.496646+0000","last_became_peered":"2024-09-17T04:00:50.496646+0000","last_unstale":"2024-09-17T04:00:52.191282+0000","last_undegraded":"2024-09-17T04:00:52.191282+0000","last_fullsized":"2024-09-17T04:00:52.191282+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:53:55.732967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475406+0000","last_change":"2024-09-17T04:00:50.499009+0000","last_active":"2024-09-17T04:00:52.475406+0000","last_peered":"2024-09-17T04:00:52.475406+0000","last_clean":"2024-09-17T04:00:52.475406+0000","last_became_active":"2024-09-17T04:00:50.495348+0000","last_became_peered":"2024-09-17T04:00:50.495348+0000","last_unstale":"2024-09-17T04:00:52.475406+0000","last_undegraded":"2024-09-17T04:00:52.475406+0000","last_fullsized":"2024-09-17T04:00:52.475406+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:38.854990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558507+0000","last_change":"2024-09-17T04:00:50.483447+0000","last_active":"2024-09-17T04:00:52.558507+0000","last_peered":"2024-09-17T04:00:52.558507+0000","last_clean":"2024-09-17T04:00:52.558507+0000","last_became_active":"2024-09-17T04:00:50.483105+0000","last_became_peered":"2024-09-17T04:00:50.483105+0000","last_unstale":"2024-09-17T04:00:52.558507+0000","last_undegraded":"2024-09-17T04:00:52.558507+0000","last_fullsized":"2024-09-17T04:00:52.558507+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:22.999066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558726+0000","last_change":"2024-09-17T04:00:50.488020+0000","last_active":"2024-09-17T04:00:52.558726+0000","last_peered":"2024-09-17T04:00:52.558726+0000","last_clean":"2024-09-17T04:00:52.558726+0000","last_became_active":"2024-09-17T04:00:50.487293+0000","last_became_peered":"2024-09-17T04:00:50.487293+0000","last_unstale":"2024-09-17T04:00:52.558726+0000","last_undegraded":"2024-09-17T04:00:52.558726+0000","last_fullsized":"2024-09-17T04:00:52.558726+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:15.011545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475667+0000","last_change":"2024-09-17T04:00:50.493247+0000","last_active":"2024-09-17T04:00:52.475667+0000","last_peered":"2024-09-17T04:00:52.475667+0000","last_clean":"2024-09-17T04:00:52.475667+0000","last_became_active":"2024-09-17T04:00:50.492907+0000","last_became_peered":"2024-09-17T04:00:50.492907+0000","last_unstale":"2024-09-17T04:00:52.475667+0000","last_undegraded":"2024-09-17T04:00:52.475667+0000","last_fullsized":"2024-09-17T04:00:52.475667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:58.410921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493295+0000","last_change":"2024-09-17T04:00:50.487190+0000","last_active":"2024-09-17T04:00:51.493295+0000","last_peered":"2024-09-17T04:00:51.493295+0000","last_clean":"2024-09-17T04:00:51.493295+0000","last_became_active":"2024-09-17T04:00:50.484107+0000","last_became_peered":"2024-09-17T04:00:50.484107+0000","last_unstale":"2024-09-17T04:00:51.493295+0000","last_undegraded":"2024-09-17T04:00:51.493295+0000","last_fullsized":"2024-09-17T04:00:51.493295+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:17.052980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490680+0000","last_change":"2024-09-17T04:00:50.500005+0000","last_active":"2024-09-17T04:00:51.490680+0000","last_peered":"2024-09-17T04:00:51.490680+0000","last_clean":"2024-09-17T04:00:51.490680+0000","last_became_active":"2024-09-17T04:00:50.497315+0000","last_became_peered":"2024-09-17T04:00:50.497315+0000","last_unstale":"2024-09-17T04:00:51.490680+0000","last_undegraded":"2024-09-17T04:00:51.490680+0000","last_fullsized":"2024-09-17T04:00:51.490680+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:50.074066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954424+0000","last_change":"2024-09-17T04:00:50.507420+0000","last_active":"2024-09-17T04:00:51.954424+0000","last_peered":"2024-09-17T04:00:51.954424+0000","last_clean":"2024-09-17T04:00:51.954424+0000","last_became_active":"2024-09-17T04:00:50.506925+0000","last_became_peered":"2024-09-17T04:00:50.506925+0000","last_unstale":"2024-09-17T04:00:51.954424+0000","last_undegraded":"2024-09-17T04:00:51.954424+0000","last_fullsized":"2024-09-17T04:00:51.954424+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:21.177663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491345+0000","last_change":"2024-09-17T04:00:50.508554+0000","last_active":"2024-09-17T04:00:51.491345+0000","last_peered":"2024-09-17T04:00:51.491345+0000","last_clean":"2024-09-17T04:00:51.491345+0000","last_became_active":"2024-09-17T04:00:50.508262+0000","last_became_peered":"2024-09-17T04:00:50.508262+0000","last_unstale":"2024-09-17T04:00:51.491345+0000","last_undegraded":"2024-09-17T04:00:51.491345+0000","last_fullsized":"2024-09-17T04:00:51.491345+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:14:13.497944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954415+0000","last_change":"2024-09-17T04:00:50.507336+0000","last_active":"2024-09-17T04:00:51.954415+0000","last_peered":"2024-09-17T04:00:51.954415+0000","last_clean":"2024-09-17T04:00:51.954415+0000","last_became_active":"2024-09-17T04:00:50.507000+0000","last_became_peered":"2024-09-17T04:00:50.507000+0000","last_unstale":"2024-09-17T04:00:51.954415+0000","last_undegraded":"2024-09-17T04:00:51.954415+0000","last_fullsized":"2024-09-17T04:00:51.954415+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:03.261144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492684+0000","last_change":"2024-09-17T04:00:50.508385+0000","last_active":"2024-09-17T04:00:51.492684+0000","last_peered":"2024-09-17T04:00:51.492684+0000","last_clean":"2024-09-17T04:00:51.492684+0000","last_became_active":"2024-09-17T04:00:50.507803+0000","last_became_peered":"2024-09-17T04:00:50.507803+0000","last_unstale":"2024-09-17T04:00:51.492684+0000","last_undegraded":"2024-09-17T04:00:51.492684+0000","last_fullsized":"2024-09-17T04:00:51.492684+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:40:40.047416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557509+0000","last_change":"2024-09-17T04:00:50.493135+0000","last_active":"2024-09-17T04:00:52.557509+0000","last_peered":"2024-09-17T04:00:52.557509+0000","last_clean":"2024-09-17T04:00:52.557509+0000","last_became_active":"2024-09-17T04:00:50.492849+0000","last_became_peered":"2024-09-17T04:00:50.492849+0000","last_unstale":"2024-09-17T04:00:52.557509+0000","last_undegraded":"2024-09-17T04:00:52.557509+0000","last_fullsized":"2024-09-17T04:00:52.557509+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:53:15.162412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490344+0000","last_change":"2024-09-17T04:00:50.504166+0000","last_active":"2024-09-17T04:00:51.490344+0000","last_peered":"2024-09-17T04:00:51.490344+0000","last_clean":"2024-09-17T04:00:51.490344+0000","last_became_active":"2024-09-17T04:00:50.503869+0000","last_became_peered":"2024-09-17T04:00:50.503869+0000","last_unstale":"2024-09-17T04:00:51.490344+0000","last_undegraded":"2024-09-17T04:00:51.490344+0000","last_fullsized":"2024-09-17T04:00:51.490344+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:44.335387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493116+0000","last_change":"2024-09-17T04:00:50.497167+0000","last_active":"2024-09-17T04:00:51.493116+0000","last_peered":"2024-09-17T04:00:51.493116+0000","last_clean":"2024-09-17T04:00:51.493116+0000","last_became_active":"2024-09-17T04:00:50.496798+0000","last_became_peered":"2024-09-17T04:00:50.496798+0000","last_unstale":"2024-09-17T04:00:51.493116+0000","last_undegraded":"2024-09-17T04:00:51.493116+0000","last_fullsized":"2024-09-17T04:00:51.493116+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:18.337224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"173'4","reported_seq":19,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954531+0000","last_change":"2024-09-17T04:00:49.488273+0000","last_active":"2024-09-17T04:00:51.954531+0000","last_peered":"2024-09-17T04:00:51.954531+0000","last_clean":"2024-09-17T04:00:51.954531+0000","last_became_active":"2024-09-17T04:00:49.486926+0000","last_became_peered":"2024-09-17T04:00:49.486926+0000","last_unstale":"2024-09-17T04:00:51.954531+0000","last_undegraded":"2024-09-17T04:00:51.954531+0000","last_fullsized":"2024-09-17T04:00:51.954531+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:23.992360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558954+0000","last_change":"2024-09-17T04:00:50.493935+0000","last_active":"2024-09-17T04:00:52.558954+0000","last_peered":"2024-09-17T04:00:52.558954+0000","last_clean":"2024-09-17T04:00:52.558954+0000","last_became_active":"2024-09-17T04:00:50.490351+0000","last_became_peered":"2024-09-17T04:00:50.490351+0000","last_unstale":"2024-09-17T04:00:52.558954+0000","last_undegraded":"2024-09-17T04:00:52.558954+0000","last_fullsized":"2024-09-17T04:00:52.558954+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:16.034449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493225+0000","last_change":"2024-09-17T04:00:49.485660+0000","last_active":"2024-09-17T04:00:51.493225+0000","last_peered":"2024-09-17T04:00:51.493225+0000","last_clean":"2024-09-17T04:00:51.493225+0000","last_became_active":"2024-09-17T04:00:49.484522+0000","last_became_peered":"2024-09-17T04:00:49.484522+0000","last_unstale":"2024-09-17T04:00:51.493225+0000","last_undegraded":"2024-09-17T04:00:51.493225+0000","last_fullsized":"2024-09-17T04:00:51.493225+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:17.632102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490880+0000","last_change":"2024-09-17T04:00:50.493759+0000","last_active":"2024-09-17T04:00:51.490880+0000","last_peered":"2024-09-17T04:00:51.490880+0000","last_clean":"2024-09-17T04:00:51.490880+0000","last_became_active":"2024-09-17T04:00:50.493472+0000","last_became_peered":"2024-09-17T04:00:50.493472+0000","last_unstale":"2024-09-17T04:00:51.490880+0000","last_undegraded":"2024-09-17T04:00:51.490880+0000","last_fullsized":"2024-09-17T04:00:51.490880+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:36.564425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492180+0000","last_change":"2024-09-17T04:00:49.485026+0000","last_active":"2024-09-17T04:00:51.492180+0000","last_peered":"2024-09-17T04:00:51.492180+0000","last_clean":"2024-09-17T04:00:51.492180+0000","last_became_active":"2024-09-17T04:00:49.484730+0000","last_became_peered":"2024-09-17T04:00:49.484730+0000","last_unstale":"2024-09-17T04:00:51.492180+0000","last_undegraded":"2024-09-17T04:00:51.492180+0000","last_fullsized":"2024-09-17T04:00:51.492180+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:09.639653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557800+0000","last_change":"2024-09-17T04:00:50.501781+0000","last_active":"2024-09-17T04:00:52.557800+0000","last_peered":"2024-09-17T04:00:52.557800+0000","last_clean":"2024-09-17T04:00:52.557800+0000","last_became_active":"2024-09-17T04:00:50.501192+0000","last_became_peered":"2024-09-17T04:00:50.501192+0000","last_unstale":"2024-09-17T04:00:52.557800+0000","last_undegraded":"2024-09-17T04:00:52.557800+0000","last_fullsized":"2024-09-17T04:00:52.557800+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:10.497603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475655+0000","last_change":"2024-09-17T04:00:49.492918+0000","last_active":"2024-09-17T04:00:52.475655+0000","last_peered":"2024-09-17T04:00:52.475655+0000","last_clean":"2024-09-17T04:00:52.475655+0000","last_became_active":"2024-09-17T04:00:49.492545+0000","last_became_peered":"2024-09-17T04:00:49.492545+0000","last_unstale":"2024-09-17T04:00:52.475655+0000","last_undegraded":"2024-09-17T04:00:52.475655+0000","last_fullsized":"2024-09-17T04:00:52.475655+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:49.593934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475383+0000","last_change":"2024-09-17T04:00:50.498633+0000","last_active":"2024-09-17T04:00:52.475383+0000","last_peered":"2024-09-17T04:00:52.475383+0000","last_clean":"2024-09-17T04:00:52.475383+0000","last_became_active":"2024-09-17T04:00:50.498319+0000","last_became_peered":"2024-09-17T04:00:50.498319+0000","last_unstale":"2024-09-17T04:00:52.475383+0000","last_undegraded":"2024-09-17T04:00:52.475383+0000","last_fullsized":"2024-09-17T04:00:52.475383+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:22:33.289244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191058+0000","last_change":"2024-09-17T04:00:49.484809+0000","last_active":"2024-09-17T04:00:52.191058+0000","last_peered":"2024-09-17T04:00:52.191058+0000","last_clean":"2024-09-17T04:00:52.191058+0000","last_became_active":"2024-09-17T04:00:49.484517+0000","last_became_peered":"2024-09-17T04:00:49.484517+0000","last_unstale":"2024-09-17T04:00:52.191058+0000","last_undegraded":"2024-09-17T04:00:52.191058+0000","last_fullsized":"2024-09-17T04:00:52.191058+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:55.826452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190819+0000","last_change":"2024-09-17T04:00:50.495545+0000","last_active":"2024-09-17T04:00:52.190819+0000","last_peered":"2024-09-17T04:00:52.190819+0000","last_clean":"2024-09-17T04:00:52.190819+0000","last_became_active":"2024-09-17T04:00:50.494856+0000","last_became_peered":"2024-09-17T04:00:50.494856+0000","last_unstale":"2024-09-17T04:00:52.190819+0000","last_undegraded":"2024-09-17T04:00:52.190819+0000","last_fullsized":"2024-09-17T04:00:52.190819+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:12.898308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190695+0000","last_change":"2024-09-17T04:00:49.483959+0000","last_active":"2024-09-17T04:00:52.190695+0000","last_peered":"2024-09-17T04:00:52.190695+0000","last_clean":"2024-09-17T04:00:52.190695+0000","last_became_active":"2024-09-17T04:00:49.483564+0000","last_became_peered":"2024-09-17T04:00:49.483564+0000","last_unstale":"2024-09-17T04:00:52.190695+0000","last_undegraded":"2024-09-17T04:00:52.190695+0000","last_fullsized":"2024-09-17T04:00:52.190695+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:03.426557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475394+0000","last_change":"2024-09-17T04:00:50.499998+0000","last_active":"2024-09-17T04:00:52.475394+0000","last_peered":"2024-09-17T04:00:52.475394+0000","last_clean":"2024-09-17T04:00:52.475394+0000","last_became_active":"2024-09-17T04:00:50.499605+0000","last_became_peered":"2024-09-17T04:00:50.499605+0000","last_unstale":"2024-09-17T04:00:52.475394+0000","last_undegraded":"2024-09-17T04:00:52.475394+0000","last_fullsized":"2024-09-17T04:00:52.475394+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:04.373538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558123+0000","last_change":"2024-09-17T04:00:49.488756+0000","last_active":"2024-09-17T04:00:52.558123+0000","last_peered":"2024-09-17T04:00:52.558123+0000","last_clean":"2024-09-17T04:00:52.558123+0000","last_became_active":"2024-09-17T04:00:49.487367+0000","last_became_peered":"2024-09-17T04:00:49.487367+0000","last_unstale":"2024-09-17T04:00:52.558123+0000","last_undegraded":"2024-09-17T04:00:52.558123+0000","last_fullsized":"2024-09-17T04:00:52.558123+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475774+0000","last_change":"2024-09-17T04:00:50.504866+0000","last_active":"2024-09-17T04:00:52.475774+0000","last_peered":"2024-09-17T04:00:52.475774+0000","last_clean":"2024-09-17T04:00:52.475774+0000","last_became_active":"2024-09-17T04:00:50.504365+0000","last_became_peered":"2024-09-17T04:00:50.504365+0000","last_unstale":"2024-09-17T04:00:52.475774+0000","last_undegraded":"2024-09-17T04:00:52.475774+0000","last_fullsized":"2024-09-17T04:00:52.475774+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:23.967019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191044+0000","last_change":"2024-09-17T04:00:49.484439+0000","last_active":"2024-09-17T04:00:52.191044+0000","last_peered":"2024-09-17T04:00:52.191044+0000","last_clean":"2024-09-17T04:00:52.191044+0000","last_became_active":"2024-09-17T04:00:49.483790+0000","last_became_peered":"2024-09-17T04:00:49.483790+0000","last_unstale":"2024-09-17T04:00:52.191044+0000","last_undegraded":"2024-09-17T04:00:52.191044+0000","last_fullsized":"2024-09-17T04:00:52.191044+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:05:39.568646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492483+0000","last_change":"2024-09-17T04:00:50.490877+0000","last_active":"2024-09-17T04:00:51.492483+0000","last_peered":"2024-09-17T04:00:51.492483+0000","last_clean":"2024-09-17T04:00:51.492483+0000","last_became_active":"2024-09-17T04:00:50.490591+0000","last_became_peered":"2024-09-17T04:00:50.490591+0000","last_unstale":"2024-09-17T04:00:51.492483+0000","last_undegraded":"2024-09-17T04:00:51.492483+0000","last_fullsized":"2024-09-17T04:00:51.492483+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:38.063668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558774+0000","last_change":"2024-09-17T04:00:49.481548+0000","last_active":"2024-09-17T04:00:52.558774+0000","last_peered":"2024-09-17T04:00:52.558774+0000","last_clean":"2024-09-17T04:00:52.558774+0000","last_became_active":"2024-09-17T04:00:49.479803+0000","last_became_peered":"2024-09-17T04:00:49.479803+0000","last_unstale":"2024-09-17T04:00:52.558774+0000","last_undegraded":"2024-09-17T04:00:52.558774+0000","last_fullsized":"2024-09-17T04:00:52.558774+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:49.312533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475673+0000","last_change":"2024-09-17T04:00:50.501872+0000","last_active":"2024-09-17T04:00:52.475673+0000","last_peered":"2024-09-17T04:00:52.475673+0000","last_clean":"2024-09-17T04:00:52.475673+0000","last_became_active":"2024-09-17T04:00:50.501566+0000","last_became_peered":"2024-09-17T04:00:50.501566+0000","last_unstale":"2024-09-17T04:00:52.475673+0000","last_undegraded":"2024-09-17T04:00:52.475673+0000","last_fullsized":"2024-09-17T04:00:52.475673+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:14.084351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"173'3","reported_seq":18,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558106+0000","last_change":"2024-09-17T04:00:49.487399+0000","last_active":"2024-09-17T04:00:52.558106+0000","last_peered":"2024-09-17T04:00:52.558106+0000","last_clean":"2024-09-17T04:00:52.558106+0000","last_became_active":"2024-09-17T04:00:49.487044+0000","last_became_peered":"2024-09-17T04:00:49.487044+0000","last_unstale":"2024-09-17T04:00:52.558106+0000","last_undegraded":"2024-09-17T04:00:52.558106+0000","last_fullsized":"2024-09-17T04:00:52.558106+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:13:30.425262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490623+0000","last_change":"2024-09-17T04:00:50.500587+0000","last_active":"2024-09-17T04:00:51.490623+0000","last_peered":"2024-09-17T04:00:51.490623+0000","last_clean":"2024-09-17T04:00:51.490623+0000","last_became_active":"2024-09-17T04:00:50.500282+0000","last_became_peered":"2024-09-17T04:00:50.500282+0000","last_unstale":"2024-09-17T04:00:51.490623+0000","last_undegraded":"2024-09-17T04:00:51.490623+0000","last_fullsized":"2024-09-17T04:00:51.490623+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:49:57.370683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"173'6","reported_seq":21,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475785+0000","last_change":"2024-09-17T04:00:49.491252+0000","last_active":"2024-09-17T04:00:52.475785+0000","last_peered":"2024-09-17T04:00:52.475785+0000","last_clean":"2024-09-17T04:00:52.475785+0000","last_became_active":"2024-09-17T04:00:49.490892+0000","last_became_peered":"2024-09-17T04:00:49.490892+0000","last_unstale":"2024-09-17T04:00:52.475785+0000","last_undegraded":"2024-09-17T04:00:52.475785+0000","last_fullsized":"2024-09-17T04:00:52.475785+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:02.780252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190912+0000","last_change":"2024-09-17T04:00:50.503682+0000","last_active":"2024-09-17T04:00:52.190912+0000","last_peered":"2024-09-17T04:00:52.190912+0000","last_clean":"2024-09-17T04:00:52.190912+0000","last_became_active":"2024-09-17T04:00:50.503389+0000","last_became_peered":"2024-09-17T04:00:50.503389+0000","last_unstale":"2024-09-17T04:00:52.190912+0000","last_undegraded":"2024-09-17T04:00:52.190912+0000","last_fullsized":"2024-09-17T04:00:52.190912+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:51.815481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558379+0000","last_change":"2024-09-17T04:00:49.488763+0000","last_active":"2024-09-17T04:00:52.558379+0000","last_peered":"2024-09-17T04:00:52.558379+0000","last_clean":"2024-09-17T04:00:52.558379+0000","last_became_active":"2024-09-17T04:00:49.487670+0000","last_became_peered":"2024-09-17T04:00:49.487670+0000","last_unstale":"2024-09-17T04:00:52.558379+0000","last_undegraded":"2024-09-17T04:00:52.558379+0000","last_fullsized":"2024-09-17T04:00:52.558379+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:16:18.500556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557927+0000","last_change":"2024-09-17T04:00:50.481453+0000","last_active":"2024-09-17T04:00:52.557927+0000","last_peered":"2024-09-17T04:00:52.557927+0000","last_clean":"2024-09-17T04:00:52.557927+0000","last_became_active":"2024-09-17T04:00:50.481137+0000","last_became_peered":"2024-09-17T04:00:50.481137+0000","last_unstale":"2024-09-17T04:00:52.557927+0000","last_undegraded":"2024-09-17T04:00:52.557927+0000","last_fullsized":"2024-09-17T04:00:52.557927+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:06.279433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558318+0000","last_change":"2024-09-17T04:00:49.492017+0000","last_active":"2024-09-17T04:00:52.558318+0000","last_peered":"2024-09-17T04:00:52.558318+0000","last_clean":"2024-09-17T04:00:52.558318+0000","last_became_active":"2024-09-17T04:00:49.491561+0000","last_became_peered":"2024-09-17T04:00:49.491561+0000","last_unstale":"2024-09-17T04:00:52.558318+0000","last_undegraded":"2024-09-17T04:00:52.558318+0000","last_fullsized":"2024-09-17T04:00:52.558318+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491405+0000","last_change":"2024-09-17T04:00:50.492894+0000","last_active":"2024-09-17T04:00:51.491405+0000","last_peered":"2024-09-17T04:00:51.491405+0000","last_clean":"2024-09-17T04:00:51.491405+0000","last_became_active":"2024-09-17T04:00:50.492614+0000","last_became_peered":"2024-09-17T04:00:50.492614+0000","last_unstale":"2024-09-17T04:00:51.491405+0000","last_undegraded":"2024-09-17T04:00:51.491405+0000","last_fullsized":"2024-09-17T04:00:51.491405+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:14.473607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557441+0000","last_change":"2024-09-17T04:00:49.482396+0000","last_active":"2024-09-17T04:00:52.557441+0000","last_peered":"2024-09-17T04:00:52.557441+0000","last_clean":"2024-09-17T04:00:52.557441+0000","last_became_active":"2024-09-17T04:00:49.481991+0000","last_became_peered":"2024-09-17T04:00:49.481991+0000","last_unstale":"2024-09-17T04:00:52.557441+0000","last_undegraded":"2024-09-17T04:00:52.557441+0000","last_fullsized":"2024-09-17T04:00:52.557441+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:53:01.860917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490625+0000","last_change":"2024-09-17T04:00:50.486171+0000","last_active":"2024-09-17T04:00:51.490625+0000","last_peered":"2024-09-17T04:00:51.490625+0000","last_clean":"2024-09-17T04:00:51.490625+0000","last_became_active":"2024-09-17T04:00:50.484422+0000","last_became_peered":"2024-09-17T04:00:50.484422+0000","last_unstale":"2024-09-17T04:00:51.490625+0000","last_undegraded":"2024-09-17T04:00:51.490625+0000","last_fullsized":"2024-09-17T04:00:51.490625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:09.236326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"173'8","reported_seq":23,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491608+0000","last_change":"2024-09-17T04:00:49.486848+0000","last_active":"2024-09-17T04:00:51.491608+0000","last_peered":"2024-09-17T04:00:51.491608+0000","last_clean":"2024-09-17T04:00:51.491608+0000","last_became_active":"2024-09-17T04:00:49.486463+0000","last_became_peered":"2024-09-17T04:00:49.486463+0000","last_unstale":"2024-09-17T04:00:51.491608+0000","last_undegraded":"2024-09-17T04:00:51.491608+0000","last_fullsized":"2024-09-17T04:00:51.491608+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:41.797680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493377+0000","last_change":"2024-09-17T04:00:50.495171+0000","last_active":"2024-09-17T04:00:51.493377+0000","last_peered":"2024-09-17T04:00:51.493377+0000","last_clean":"2024-09-17T04:00:51.493377+0000","last_became_active":"2024-09-17T04:00:50.494834+0000","last_became_peered":"2024-09-17T04:00:50.494834+0000","last_unstale":"2024-09-17T04:00:51.493377+0000","last_undegraded":"2024-09-17T04:00:51.493377+0000","last_fullsized":"2024-09-17T04:00:51.493377+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:22.013808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191210+0000","last_change":"2024-09-17T04:00:49.487492+0000","last_active":"2024-09-17T04:00:52.191210+0000","last_peered":"2024-09-17T04:00:52.191210+0000","last_clean":"2024-09-17T04:00:52.191210+0000","last_became_active":"2024-09-17T04:00:49.487146+0000","last_became_peered":"2024-09-17T04:00:49.487146+0000","last_unstale":"2024-09-17T04:00:52.191210+0000","last_undegraded":"2024-09-17T04:00:52.191210+0000","last_fullsized":"2024-09-17T04:00:52.191210+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493310+0000","last_change":"2024-09-17T04:00:50.484922+0000","last_active":"2024-09-17T04:00:51.493310+0000","last_peered":"2024-09-17T04:00:51.493310+0000","last_clean":"2024-09-17T04:00:51.493310+0000","last_became_active":"2024-09-17T04:00:50.483724+0000","last_became_peered":"2024-09-17T04:00:50.483724+0000","last_unstale":"2024-09-17T04:00:51.493310+0000","last_undegraded":"2024-09-17T04:00:51.493310+0000","last_fullsized":"2024-09-17T04:00:51.493310+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:49:40.781284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.493354+0000","last_change":"2024-09-17T04:00:49.475839+0000","last_active":"2024-09-17T04:00:51.493354+0000","last_peered":"2024-09-17T04:00:51.493354+0000","last_clean":"2024-09-17T04:00:51.493354+0000","last_became_active":"2024-09-17T04:00:49.475239+0000","last_became_peered":"2024-09-17T04:00:49.475239+0000","last_unstale":"2024-09-17T04:00:51.493354+0000","last_undegraded":"2024-09-17T04:00:51.493354+0000","last_fullsized":"2024-09-17T04:00:51.493354+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:14.501945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491480+0000","last_change":"2024-09-17T04:00:50.504359+0000","last_active":"2024-09-17T04:00:51.491480+0000","last_peered":"2024-09-17T04:00:51.491480+0000","last_clean":"2024-09-17T04:00:51.491480+0000","last_became_active":"2024-09-17T04:00:50.501126+0000","last_became_peered":"2024-09-17T04:00:50.501126+0000","last_unstale":"2024-09-17T04:00:51.491480+0000","last_undegraded":"2024-09-17T04:00:51.491480+0000","last_fullsized":"2024-09-17T04:00:51.491480+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:20.943413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557974+0000","last_change":"2024-09-17T04:00:49.485542+0000","last_active":"2024-09-17T04:00:52.557974+0000","last_peered":"2024-09-17T04:00:52.557974+0000","last_clean":"2024-09-17T04:00:52.557974+0000","last_became_active":"2024-09-17T04:00:49.481811+0000","last_became_peered":"2024-09-17T04:00:49.481811+0000","last_unstale":"2024-09-17T04:00:52.557974+0000","last_undegraded":"2024-09-17T04:00:52.557974+0000","last_fullsized":"2024-09-17T04:00:52.557974+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:53.374454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557930+0000","last_change":"2024-09-17T04:00:50.495216+0000","last_active":"2024-09-17T04:00:52.557930+0000","last_peered":"2024-09-17T04:00:52.557930+0000","last_clean":"2024-09-17T04:00:52.557930+0000","last_became_active":"2024-09-17T04:00:50.494947+0000","last_became_peered":"2024-09-17T04:00:50.494947+0000","last_unstale":"2024-09-17T04:00:52.557930+0000","last_undegraded":"2024-09-17T04:00:52.557930+0000","last_fullsized":"2024-09-17T04:00:52.557930+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:24:59.919453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491546+0000","last_change":"2024-09-17T04:00:49.485281+0000","last_active":"2024-09-17T04:00:51.491546+0000","last_peered":"2024-09-17T04:00:51.491546+0000","last_clean":"2024-09-17T04:00:51.491546+0000","last_became_active":"2024-09-17T04:00:49.484098+0000","last_became_peered":"2024-09-17T04:00:49.484098+0000","last_unstale":"2024-09-17T04:00:51.491546+0000","last_undegraded":"2024-09-17T04:00:51.491546+0000","last_fullsized":"2024-09-17T04:00:51.491546+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:46:08.975039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.557656+0000","last_change":"2024-09-17T04:00:50.505179+0000","last_active":"2024-09-17T04:00:52.557656+0000","last_peered":"2024-09-17T04:00:52.557656+0000","last_clean":"2024-09-17T04:00:52.557656+0000","last_became_active":"2024-09-17T04:00:50.504729+0000","last_became_peered":"2024-09-17T04:00:50.504729+0000","last_unstale":"2024-09-17T04:00:52.557656+0000","last_undegraded":"2024-09-17T04:00:52.557656+0000","last_fullsized":"2024-09-17T04:00:52.557656+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:53.716965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558080+0000","last_change":"2024-09-17T04:00:49.479444+0000","last_active":"2024-09-17T04:00:52.558080+0000","last_peered":"2024-09-17T04:00:52.558080+0000","last_clean":"2024-09-17T04:00:52.558080+0000","last_became_active":"2024-09-17T04:00:49.479095+0000","last_became_peered":"2024-09-17T04:00:49.479095+0000","last_unstale":"2024-09-17T04:00:52.558080+0000","last_undegraded":"2024-09-17T04:00:52.558080+0000","last_fullsized":"2024-09-17T04:00:52.558080+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:42:25.795248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491625+0000","last_change":"2024-09-17T04:00:50.507841+0000","last_active":"2024-09-17T04:00:51.491625+0000","last_peered":"2024-09-17T04:00:51.491625+0000","last_clean":"2024-09-17T04:00:51.491625+0000","last_became_active":"2024-09-17T04:00:50.507577+0000","last_became_peered":"2024-09-17T04:00:50.507577+0000","last_unstale":"2024-09-17T04:00:51.491625+0000","last_undegraded":"2024-09-17T04:00:51.491625+0000","last_fullsized":"2024-09-17T04:00:51.491625+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:51.506216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490347+0000","last_change":"2024-09-17T04:00:49.492248+0000","last_active":"2024-09-17T04:00:51.490347+0000","last_peered":"2024-09-17T04:00:51.490347+0000","last_clean":"2024-09-17T04:00:51.490347+0000","last_became_active":"2024-09-17T04:00:49.491744+0000","last_became_peered":"2024-09-17T04:00:49.491744+0000","last_unstale":"2024-09-17T04:00:51.490347+0000","last_undegraded":"2024-09-17T04:00:51.490347+0000","last_fullsized":"2024-09-17T04:00:51.490347+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:52:45.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492551+0000","last_change":"2024-09-17T04:00:50.485354+0000","last_active":"2024-09-17T04:00:51.492551+0000","last_peered":"2024-09-17T04:00:51.492551+0000","last_clean":"2024-09-17T04:00:51.492551+0000","last_became_active":"2024-09-17T04:00:50.485017+0000","last_became_peered":"2024-09-17T04:00:50.485017+0000","last_unstale":"2024-09-17T04:00:51.492551+0000","last_undegraded":"2024-09-17T04:00:51.492551+0000","last_fullsized":"2024-09-17T04:00:51.492551+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:43.479866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490209+0000","last_change":"2024-09-17T04:00:49.490198+0000","last_active":"2024-09-17T04:00:51.490209+0000","last_peered":"2024-09-17T04:00:51.490209+0000","last_clean":"2024-09-17T04:00:51.490209+0000","last_became_active":"2024-09-17T04:00:49.489685+0000","last_became_peered":"2024-09-17T04:00:49.489685+0000","last_unstale":"2024-09-17T04:00:51.490209+0000","last_undegraded":"2024-09-17T04:00:51.490209+0000","last_fullsized":"2024-09-17T04:00:51.490209+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:39:21.987330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558604+0000","last_change":"2024-09-17T04:00:50.488092+0000","last_active":"2024-09-17T04:00:52.558604+0000","last_peered":"2024-09-17T04:00:52.558604+0000","last_clean":"2024-09-17T04:00:52.558604+0000","last_became_active":"2024-09-17T04:00:50.487718+0000","last_became_peered":"2024-09-17T04:00:50.487718+0000","last_unstale":"2024-09-17T04:00:52.558604+0000","last_undegraded":"2024-09-17T04:00:52.558604+0000","last_fullsized":"2024-09-17T04:00:52.558604+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:09:12.457060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.190954+0000","last_change":"2024-09-17T04:00:49.492610+0000","last_active":"2024-09-17T04:00:52.190954+0000","last_peered":"2024-09-17T04:00:52.190954+0000","last_clean":"2024-09-17T04:00:52.190954+0000","last_became_active":"2024-09-17T04:00:49.491506+0000","last_became_peered":"2024-09-17T04:00:49.491506+0000","last_unstale":"2024-09-17T04:00:52.190954+0000","last_undegraded":"2024-09-17T04:00:52.190954+0000","last_fullsized":"2024-09-17T04:00:52.190954+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.784168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954545+0000","last_change":"2024-09-17T04:00:50.511428+0000","last_active":"2024-09-17T04:00:51.954545+0000","last_peered":"2024-09-17T04:00:51.954545+0000","last_clean":"2024-09-17T04:00:51.954545+0000","last_became_active":"2024-09-17T04:00:50.511114+0000","last_became_peered":"2024-09-17T04:00:50.511114+0000","last_unstale":"2024-09-17T04:00:51.954545+0000","last_undegraded":"2024-09-17T04:00:51.954545+0000","last_fullsized":"2024-09-17T04:00:51.954545+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:00:58.184294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"173'2","reported_seq":17,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492161+0000","last_change":"2024-09-17T04:00:49.478561+0000","last_active":"2024-09-17T04:00:51.492161+0000","last_peered":"2024-09-17T04:00:51.492161+0000","last_clean":"2024-09-17T04:00:51.492161+0000","last_became_active":"2024-09-17T04:00:49.478248+0000","last_became_peered":"2024-09-17T04:00:49.478248+0000","last_unstale":"2024-09-17T04:00:51.492161+0000","last_undegraded":"2024-09-17T04:00:51.492161+0000","last_fullsized":"2024-09-17T04:00:51.492161+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:59.810080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490983+0000","last_change":"2024-09-17T04:00:50.495573+0000","last_active":"2024-09-17T04:00:51.490983+0000","last_peered":"2024-09-17T04:00:51.490983+0000","last_clean":"2024-09-17T04:00:51.490983+0000","last_became_active":"2024-09-17T04:00:50.494895+0000","last_became_peered":"2024-09-17T04:00:50.494895+0000","last_unstale":"2024-09-17T04:00:51.490983+0000","last_undegraded":"2024-09-17T04:00:51.490983+0000","last_fullsized":"2024-09-17T04:00:51.490983+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:07.480340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"174'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475720+0000","last_change":"2024-09-17T04:00:49.490303+0000","last_active":"2024-09-17T04:00:52.475720+0000","last_peered":"2024-09-17T04:00:52.475720+0000","last_clean":"2024-09-17T04:00:52.475720+0000","last_became_active":"2024-09-17T04:00:49.489882+0000","last_became_peered":"2024-09-17T04:00:49.489882+0000","last_unstale":"2024-09-17T04:00:52.475720+0000","last_undegraded":"2024-09-17T04:00:52.475720+0000","last_fullsized":"2024-09-17T04:00:52.475720+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:46.567261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492851+0000","last_change":"2024-09-17T04:00:50.493244+0000","last_active":"2024-09-17T04:00:51.492851+0000","last_peered":"2024-09-17T04:00:51.492851+0000","last_clean":"2024-09-17T04:00:51.492851+0000","last_became_active":"2024-09-17T04:00:50.492892+0000","last_became_peered":"2024-09-17T04:00:50.492892+0000","last_unstale":"2024-09-17T04:00:51.492851+0000","last_undegraded":"2024-09-17T04:00:51.492851+0000","last_fullsized":"2024-09-17T04:00:51.492851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:07:19.515809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191177+0000","last_change":"2024-09-17T04:00:49.480983+0000","last_active":"2024-09-17T04:00:52.191177+0000","last_peered":"2024-09-17T04:00:52.191177+0000","last_clean":"2024-09-17T04:00:52.191177+0000","last_became_active":"2024-09-17T04:00:49.480620+0000","last_became_peered":"2024-09-17T04:00:49.480620+0000","last_unstale":"2024-09-17T04:00:52.191177+0000","last_undegraded":"2024-09-17T04:00:52.191177+0000","last_fullsized":"2024-09-17T04:00:52.191177+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:59:28.065208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558754+0000","last_change":"2024-09-17T04:00:50.491601+0000","last_active":"2024-09-17T04:00:52.558754+0000","last_peered":"2024-09-17T04:00:52.558754+0000","last_clean":"2024-09-17T04:00:52.558754+0000","last_became_active":"2024-09-17T04:00:50.491279+0000","last_became_peered":"2024-09-17T04:00:50.491279+0000","last_unstale":"2024-09-17T04:00:52.558754+0000","last_undegraded":"2024-09-17T04:00:52.558754+0000","last_fullsized":"2024-09-17T04:00:52.558754+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:58:03.782918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491657+0000","last_change":"2024-09-17T04:00:49.477561+0000","last_active":"2024-09-17T04:00:51.491657+0000","last_peered":"2024-09-17T04:00:51.491657+0000","last_clean":"2024-09-17T04:00:51.491657+0000","last_became_active":"2024-09-17T04:00:49.477252+0000","last_became_peered":"2024-09-17T04:00:49.477252+0000","last_unstale":"2024-09-17T04:00:51.491657+0000","last_undegraded":"2024-09-17T04:00:51.491657+0000","last_fullsized":"2024-09-17T04:00:51.491657+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:07.327645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490348+0000","last_change":"2024-09-17T04:00:50.499748+0000","last_active":"2024-09-17T04:00:51.490348+0000","last_peered":"2024-09-17T04:00:51.490348+0000","last_clean":"2024-09-17T04:00:51.490348+0000","last_became_active":"2024-09-17T04:00:50.497933+0000","last_became_peered":"2024-09-17T04:00:50.497933+0000","last_unstale":"2024-09-17T04:00:51.490348+0000","last_undegraded":"2024-09-17T04:00:51.490348+0000","last_fullsized":"2024-09-17T04:00:51.490348+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:29:33.887766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475476+0000","last_change":"2024-09-17T04:00:49.486586+0000","last_active":"2024-09-17T04:00:52.475476+0000","last_peered":"2024-09-17T04:00:52.475476+0000","last_clean":"2024-09-17T04:00:52.475476+0000","last_became_active":"2024-09-17T04:00:49.486099+0000","last_became_peered":"2024-09-17T04:00:49.486099+0000","last_unstale":"2024-09-17T04:00:52.475476+0000","last_undegraded":"2024-09-17T04:00:52.475476+0000","last_fullsized":"2024-09-17T04:00:52.475476+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:42.671635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954355+0000","last_change":"2024-09-17T04:00:50.501002+0000","last_active":"2024-09-17T04:00:51.954355+0000","last_peered":"2024-09-17T04:00:51.954355+0000","last_clean":"2024-09-17T04:00:51.954355+0000","last_became_active":"2024-09-17T04:00:50.500621+0000","last_became_peered":"2024-09-17T04:00:50.500621+0000","last_unstale":"2024-09-17T04:00:51.954355+0000","last_undegraded":"2024-09-17T04:00:51.954355+0000","last_fullsized":"2024-09-17T04:00:51.954355+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:38.586287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.191137+0000","last_change":"2024-09-17T04:00:49.483997+0000","last_active":"2024-09-17T04:00:52.191137+0000","last_peered":"2024-09-17T04:00:52.191137+0000","last_clean":"2024-09-17T04:00:52.191137+0000","last_became_active":"2024-09-17T04:00:49.483458+0000","last_became_peered":"2024-09-17T04:00:49.483458+0000","last_unstale":"2024-09-17T04:00:52.191137+0000","last_undegraded":"2024-09-17T04:00:52.191137+0000","last_fullsized":"2024-09-17T04:00:52.191137+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:09.279779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.491343+0000","last_change":"2024-09-17T04:00:50.492815+0000","last_active":"2024-09-17T04:00:51.491343+0000","last_peered":"2024-09-17T04:00:51.491343+0000","last_clean":"2024-09-17T04:00:51.491343+0000","last_became_active":"2024-09-17T04:00:50.492209+0000","last_became_peered":"2024-09-17T04:00:50.492209+0000","last_unstale":"2024-09-17T04:00:51.491343+0000","last_undegraded":"2024-09-17T04:00:51.491343+0000","last_fullsized":"2024-09-17T04:00:51.491343+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:15.504611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.558691+0000","last_change":"2024-09-17T04:00:49.485410+0000","last_active":"2024-09-17T04:00:52.558691+0000","last_peered":"2024-09-17T04:00:52.558691+0000","last_clean":"2024-09-17T04:00:52.558691+0000","last_became_active":"2024-09-17T04:00:49.484171+0000","last_became_peered":"2024-09-17T04:00:49.484171+0000","last_unstale":"2024-09-17T04:00:52.558691+0000","last_undegraded":"2024-09-17T04:00:52.558691+0000","last_fullsized":"2024-09-17T04:00:52.558691+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:28:02.352346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.492593+0000","last_change":"2024-09-17T04:00:50.490365+0000","last_active":"2024-09-17T04:00:51.492593+0000","last_peered":"2024-09-17T04:00:51.492593+0000","last_clean":"2024-09-17T04:00:51.492593+0000","last_became_active":"2024-09-17T04:00:50.488098+0000","last_became_peered":"2024-09-17T04:00:50.488098+0000","last_unstale":"2024-09-17T04:00:51.492593+0000","last_undegraded":"2024-09-17T04:00:51.492593+0000","last_fullsized":"2024-09-17T04:00:51.492593+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:42.788609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":15,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954116+0000","last_change":"2024-09-17T04:00:49.487212+0000","last_active":"2024-09-17T04:00:51.954116+0000","last_peered":"2024-09-17T04:00:51.954116+0000","last_clean":"2024-09-17T04:00:51.954116+0000","last_became_active":"2024-09-17T04:00:49.486849+0000","last_became_peered":"2024-09-17T04:00:49.486849+0000","last_unstale":"2024-09-17T04:00:51.954116+0000","last_undegraded":"2024-09-17T04:00:51.954116+0000","last_fullsized":"2024-09-17T04:00:51.954116+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:31.908526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":13,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.490138+0000","last_change":"2024-09-17T04:00:50.501206+0000","last_active":"2024-09-17T04:00:51.490138+0000","last_peered":"2024-09-17T04:00:51.490138+0000","last_clean":"2024-09-17T04:00:51.490138+0000","last_became_active":"2024-09-17T04:00:50.500932+0000","last_became_peered":"2024-09-17T04:00:50.500932+0000","last_unstale":"2024-09-17T04:00:51.490138+0000","last_undegraded":"2024-09-17T04:00:51.490138+0000","last_fullsized":"2024-09-17T04:00:51.490138+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":172,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:49.455427+0000","last_clean_scrub_stamp":"2024-09-17T04:00:49.455427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:07:32.987666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"173'1","reported_seq":16,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:51.954611+0000","last_change":"2024-09-17T04:00:49.489179+0000","last_active":"2024-09-17T04:00:51.954611+0000","last_peered":"2024-09-17T04:00:51.954611+0000","last_clean":"2024-09-17T04:00:51.954611+0000","last_became_active":"2024-09-17T04:00:49.488400+0000","last_became_peered":"2024-09-17T04:00:49.488400+0000","last_unstale":"2024-09-17T04:00:51.954611+0000","last_undegraded":"2024-09-17T04:00:51.954611+0000","last_fullsized":"2024-09-17T04:00:51.954611+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:00:48.452761+0000","last_clean_scrub_stamp":"2024-09-17T04:00:48.452761+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:44:31.796435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":392,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-09-17T04:00:52.475314+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:00:52.475314+0000","last_peered":"2024-09-17T04:00:52.475314+0000","last_clean":"2024-09-17T04:00:52.475314+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:00:52.475314+0000","last_undegraded":"2024-09-17T04:00:52.475314+0000","last_fullsized":"2024-09-17T04:00:52.475314+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509580,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38076,"kb_used_data":2728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94333764,"statfs":{"total":96636764160,"available":96597774336,"internally_reserved":0,"allocated":2793472,"data_stored":1549678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509583,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":2207744,"data_stored":959414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509580,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41656,"kb_used_data":2148,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330184,"statfs":{"total":96636764160,"available":96594108416,"internally_reserved":0,"allocated":2199552,"data_stored":957880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509582,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37492,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334348,"statfs":{"total":96636764160,"available":96598372352,"internally_reserved":0,"allocated":2187264,"data_stored":957698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509582,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41652,"kb_used_data":2144,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330188,"statfs":{"total":96636764160,"available":96594112512,"internally_reserved":0,"allocated":2195456,"data_stored":957754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542285,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41648,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330192,"statfs":{"total":96636764160,"available":96594116608,"internally_reserved":0,"allocated":2191360,"data_stored":958280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542286,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42228,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329612,"statfs":{"total":96636764160,"available":96593522688,"internally_reserved":0,"allocated":2785280,"data_stored":1548156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:01:00.391 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:01:00.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 -- 172.21.15.129:0/1931339618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce700ce2c0 msgr2=0x7fce700ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 --2- 172.21.15.129:0/1931339618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce700ce2c0 0x7fce700ce6d0 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fce68005fd0 tx=0x7fce6802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 -- 172.21.15.129:0/1931339618 shutdown_connections 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 --2- 172.21.15.129:0/1931339618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce700ce2c0 0x7fce700ce6d0 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 --2- 172.21.15.129:0/1931339618 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce7006ee70 0x7fce7006f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 -- 172.21.15.129:0/1931339618 >> 172.21.15.129:0/1931339618 conn(0x7fce700d0120 msgr2=0x7fce700d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:00.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.656+0000 7fce75200640 1 -- 172.21.15.129:0/1931339618 shutdown_connections 2024-09-17T04:01:00.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.657+0000 7fce75200640 1 -- 172.21.15.129:0/1931339618 wait complete. 2024-09-17T04:01:00.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.659+0000 7fce75200640 1 Processor -- start 2024-09-17T04:01:00.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.659+0000 7fce75200640 1 -- start start 2024-09-17T04:01:00.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce700ce2c0 0x7fce701f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce70156f20 0x7fce700aa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce700ced90 con 0x7fce7006ee70 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fce700cec10 con 0x7fce700ce2c0 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce75200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fce700cef10 con 0x7fce70156f20 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.660+0000 7fce6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35246/0 (socket says 172.21.15.129:35246) 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 -- 172.21.15.129:0/3899211297 learned_addr learned my addr 172.21.15.129:0/3899211297 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6ea00640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce700ce2c0 0x7fce701f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6fe00640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce70156f20 0x7fce700aa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 -- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce70156f20 msgr2=0x7fce700aa6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce70156f20 0x7fce700aa6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 -- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce700ce2c0 msgr2=0x7fce701f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce700ce2c0 0x7fce701f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 -- 172.21.15.129:0/3899211297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce70158e90 con 0x7fce7006ee70 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.661+0000 7fce6f400640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fce6000c760 tx=0x7fce6000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.662+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60006a90 con 0x7fce7006ee70 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.662+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce60006c30 con 0x7fce7006ee70 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.662+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60007290 con 0x7fce7006ee70 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.662+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce700a9670 con 0x7fce7006ee70 2024-09-17T04:01:00.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.662+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce700a9bf0 con 0x7fce7006ee70 2024-09-17T04:01:00.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.663+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce700cf3e0 con 0x7fce7006ee70 2024-09-17T04:01:00.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.664+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fce60007430 con 0x7fce7006ee70 2024-09-17T04:01:00.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.665+0000 7fce67400640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 0x7fce5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:00.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.665+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fce6009a750 con 0x7fce7006ee70 2024-09-17T04:01:00.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.665+0000 7fce6ea00640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 0x7fce5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:00.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.666+0000 7fce6ea00640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 0x7fce5007b0a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fce7015b460 tx=0x7fce5c00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:00.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.670+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fce7006f7c0 con 0x7fce7006ee70 2024-09-17T04:01:00.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.922+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fce700d0870 con 0x7fce7006ee70 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.925+0000 7fce67400640 1 -- 172.21.15.129:0/3899211297 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v39) ==== 190+0+147 (secure 0 0 0) 0x7fce60006e20 con 0x7fce7006ee70 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:01:00.925 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:01:00.926 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:01:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.927+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 msgr2=0x7fce5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.927+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 0x7fce5007b0a0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fce7015b460 tx=0x7fce5c00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.927+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 msgr2=0x7fce70158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:00.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.927+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fce6000c760 tx=0x7fce6000cc20 comp rx=0 tx=0).stop 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 shutdown_connections 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce50078b90 0x7fce5007b0a0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce70156f20 0x7fce700aa6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce700ce2c0 0x7fce701f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 --2- 172.21.15.129:0/3899211297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce7006ee70 0x7fce70158950 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 >> 172.21.15.129:0/3899211297 conn(0x7fce700d0120 msgr2=0x7fce700b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 shutdown_connections 2024-09-17T04:01:00.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:00.928+0000 7fce75200640 1 -- 172.21.15.129:0/3899211297 wait complete. 2024-09-17T04:01:00.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:01.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 -- 172.21.15.129:0/463474003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c4005250 msgr2=0x7fc5c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:01.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 --2- 172.21.15.129:0/463474003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c4005250 0x7fc5c400b9c0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8005fd0 tx=0x7fc5c4005830 comp rx=0 tx=0).stop 2024-09-17T04:01:01.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 -- 172.21.15.129:0/463474003 shutdown_connections 2024-09-17T04:01:01.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 --2- 172.21.15.129:0/463474003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5c4005250 0x7fc5c400b9c0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 -- 172.21.15.129:0/463474003 >> 172.21.15.129:0/463474003 conn(0x7fc5e006d410 msgr2=0x7fc5e00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:01.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 -- 172.21.15.129:0/463474003 shutdown_connections 2024-09-17T04:01:01.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.170+0000 7fc5e5800640 1 -- 172.21.15.129:0/463474003 wait complete. 2024-09-17T04:01:01.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.172+0000 7fc5e5800640 1 Processor -- start 2024-09-17T04:01:01.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.172+0000 7fc5e5800640 1 -- start start 2024-09-17T04:01:01.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 0x7fc5e00c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:01.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5e015c540 0x7fc5e01f9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00ccea0 con 0x7fc5e015c130 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00ccd20 con 0x7fc5c4005250 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5e5800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e00cd020 con 0x7fc5e015c540 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35256/0 (socket says 172.21.15.129:35256) 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 -- 172.21.15.129:0/371783206 learned_addr learned my addr 172.21.15.129:0/371783206 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5df400640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5e015c540 0x7fc5e01f9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5dea00640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 0x7fc5e00c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 -- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5e015c540 msgr2=0x7fc5e01f9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5e015c540 0x7fc5e01f9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 -- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 msgr2=0x7fc5e00c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:01.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 0x7fc5e00c4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5d6000640 1 -- 172.21.15.129:0/371783206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e01f9ba0 con 0x7fc5e015c130 2024-09-17T04:01:01.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.174+0000 7fc5dea00640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 0x7fc5e00c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:01.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5d6000640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8006680 tx=0x7fc5c80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:01.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8015020 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5c8011900 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e00b0e40 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8011d50 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.175+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e00b1400 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.176+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e015f300 con 0x7fc5e015c130 2024-09-17T04:01:01.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.177+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc5c8018020 con 0x7fc5e015c130 2024-09-17T04:01:01.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.179+0000 7fc5d7400640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 0x7fc5bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:01.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.179+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fc5c809b4f0 con 0x7fc5e015c130 2024-09-17T04:01:01.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.179+0000 7fc5dea00640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 0x7fc5bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:01.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.180+0000 7fc5dea00640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 0x7fc5bc07b0a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0000c00 tx=0x7fc5d0006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:01.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.184+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc5c801f030 con 0x7fc5e015c130 2024-09-17T04:01:01.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.401+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5e006a140 con 0x7fc5e015c130 2024-09-17T04:01:01.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.402+0000 7fc5d7400640 1 -- 172.21.15.129:0/371783206 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fc5c80639b0 con 0x7fc5e015c130 2024-09-17T04:01:01.402 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:01.402 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:01.402 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.404+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 msgr2=0x7fc5bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.404+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 0x7fc5bc07b0a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0000c00 tx=0x7fc5d0006b70 comp rx=0 tx=0).stop 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.404+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 msgr2=0x7fc5e01f8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.404+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8006680 tx=0x7fc5c80039c0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 shutdown_connections 2024-09-17T04:01:01.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5bc078b90 0x7fc5bc07b0a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5e015c540 0x7fc5e01f9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5e015c130 0x7fc5e01f8df0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 --2- 172.21.15.129:0/371783206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5c4005250 0x7fc5e00c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 >> 172.21.15.129:0/371783206 conn(0x7fc5e006d410 msgr2=0x7fc5e00cbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 shutdown_connections 2024-09-17T04:01:01.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:01.405+0000 7fc5e5800640 1 -- 172.21.15.129:0/371783206 wait complete. 2024-09-17T04:01:01.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:00:49.962303+0000', 'modified': '2024-09-17T04:00:52.981107+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9492}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9492': {'gid': 9492, 'name': 'c', 'rank': 0, 'incarnation': 150, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/1228169144', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 1228169144}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 1228169144}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9492, 'qdb_cluster': [9492]} 2024-09-17T04:01:01.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:01:01.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:01:02.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 -- 172.21.15.129:0/1153594030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500cd1c0 msgr2=0x7fbf501609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 --2- 172.21.15.129:0/1153594030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500cd1c0 0x7fbf501609c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbf48005fd0 tx=0x7fbf4802aa90 comp rx=0 tx=0).stop 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 -- 172.21.15.129:0/1153594030 shutdown_connections 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 --2- 172.21.15.129:0/1153594030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500cd1c0 0x7fbf501609c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 --2- 172.21.15.129:0/1153594030 >> v2:172.21.15.181:3300/0 conn(0x7fbf500b4140 0x7fbf500ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 -- 172.21.15.129:0/1153594030 >> 172.21.15.129:0/1153594030 conn(0x7fbf5006d410 msgr2=0x7fbf500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 -- 172.21.15.129:0/1153594030 shutdown_connections 2024-09-17T04:01:02.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.057+0000 7fbf57400640 1 -- 172.21.15.129:0/1153594030 wait complete. 2024-09-17T04:01:02.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.059+0000 7fbf57400640 1 Processor -- start 2024-09-17T04:01:02.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.060+0000 7fbf57400640 1 -- start start 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 0x7fbf500b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf500cd1c0 0x7fbf50156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf50160260 con 0x7fbf500705a0 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbf501600e0 con 0x7fbf500cd1c0 2024-09-17T04:01:02.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf57400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbf50160f00 con 0x7fbf500b4140 2024-09-17T04:01:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf54a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35272/0 (socket says 172.21.15.129:35272) 2024-09-17T04:01:02.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.061+0000 7fbf54a00640 1 -- 172.21.15.129:0/352968159 learned_addr learned my addr 172.21.15.129:0/352968159 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf55400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf500cd1c0 0x7fbf50156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 0x7fbf500b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 -- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 msgr2=0x7fbf500b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 0x7fbf500b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 -- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf500cd1c0 msgr2=0x7fbf50156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf500cd1c0 0x7fbf50156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 -- 172.21.15.129:0/352968159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf500b03e0 con 0x7fbf500705a0 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 0x7fbf500b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:02.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf54a00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400c3a0 tx=0x7fbf4400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf4cc00640 1 -- 172.21.15.129:0/352968159 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf440052d0 con 0x7fbf500705a0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf4cc00640 1 -- 172.21.15.129:0/352968159 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf44007020 con 0x7fbf500705a0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf500b06c0 con 0x7fbf500705a0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf4cc00640 1 -- 172.21.15.129:0/352968159 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44007470 con 0x7fbf500705a0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.062+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf500b0c80 con 0x7fbf500705a0 2024-09-17T04:01:02.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.065+0000 7fbf4cc00640 1 -- 172.21.15.129:0/352968159 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbf440067d0 con 0x7fbf500705a0 2024-09-17T04:01:02.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.066+0000 7fbf4cc00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 0x7fbf3007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.066+0000 7fbf4cc00640 1 -- 172.21.15.129:0/352968159 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fbf4409a2c0 con 0x7fbf500705a0 2024-09-17T04:01:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.066+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 0x7fbf3007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.066+0000 7fbf5e485740 1 Processor -- start 2024-09-17T04:01:02.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.066+0000 7fbf5e485740 1 -- start start 2024-09-17T04:01:02.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.067+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 0x7fbf3007b0a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbf50159030 tx=0x7fbf3c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d95270 0x5594b4165970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d6c3b0 0x5594b4165eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5594b40bf430 con 0x5594b3d74590 2024-09-17T04:01:02.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5594b40af5d0 con 0x5594b3d6c3b0 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5594b409ea00 con 0x5594b3d95270 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35290/0 (socket says 172.21.15.129:35290) 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 -- 172.21.15.129:0/3767181919 learned_addr learned my addr 172.21.15.129:0/3767181919 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf54a00640 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d95270 0x5594b4165970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d6c3b0 0x5594b4165eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 -- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d95270 msgr2=0x5594b4165970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d95270 0x5594b4165970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 -- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d6c3b0 msgr2=0x5594b4165eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d6c3b0 0x5594b4165eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.068+0000 7fbf55400640 1 -- 172.21.15.129:0/3767181919 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594b4067520 con 0x5594b3d74590 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.069+0000 7fbf55400640 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006010 tx=0x7fbf48004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06ae16e5043c9a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.069+0000 7fbf41200640 1 -- 172.21.15.129:0/3767181919 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802eae0 con 0x5594b3d74590 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.069+0000 7fbf41200640 1 -- 172.21.15.129:0/3767181919 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf48044070 con 0x5594b3d74590 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.069+0000 7fbf41200640 1 -- 172.21.15.129:0/3767181919 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48003970 con 0x5594b3d74590 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 -- 172.21.15.129:0/3767181919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 msgr2=0x5594b4135f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fbf48006010 tx=0x7fbf48004ab0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 -- 172.21.15.129:0/3767181919 shutdown_connections 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b4135f60 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d6c3b0 0x5594b4165eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 --2- 172.21.15.129:0/3767181919 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d95270 0x5594b4165970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 -- 172.21.15.129:0/3767181919 >> 172.21.15.129:0/3767181919 conn(0x5594b3d893c0 msgr2=0x5594b413b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 -- 172.21.15.129:0/3767181919 shutdown_connections 2024-09-17T04:01:02.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.070+0000 7fbf5e485740 1 -- 172.21.15.129:0/3767181919 wait complete. 2024-09-17T04:01:02.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.071+0000 7fbf5e485740 1 -- start start 2024-09-17T04:01:02.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.071+0000 7fbf5e485740 1 Processor -- start 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b417b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b415f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5594b4045330 con 0x5594b3d74590 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5594b40a2810 con 0x5594b3d95270 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5594b3e0f620 con 0x5594b3d6c3b0 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf5e485740 10 client.0 _ll_register_callbacks cb 0x5594b404a420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35304/0 (socket says 172.21.15.129:35304) 2024-09-17T04:01:02.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 -- 172.21.15.129:0/2924099684 learned_addr learned my addr 172.21.15.129:0/2924099684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf54a00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b417b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf55400640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b415f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 -- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 msgr2=0x5594b417b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b417b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 -- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 msgr2=0x5594b415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b415f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf4fe00640 1 -- 172.21.15.129:0/2924099684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594b3e00850 con 0x5594b3d74590 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.073+0000 7fbf55400640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b415f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c010fa0 tx=0x7fbf3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287ae2b51588d424 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf35600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c020840 con 0x5594b3d74590 2024-09-17T04:01:02.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf48044070 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fbf3c02ab20 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fbf3c02ad20 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf5e485740 10 client.9645 fetch_fsmap learned FSMap version 154 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x5594b3e776b0 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.074+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf48044070 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 154) ==== 3502+0+0 (secure 0 0 0) 0x7fbf3c00f040 con 0x5594b3d74590 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 10 client.9645 fetch_fsmap finished waiting for FSMap version 154 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 10 client.9645 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 20 client.9645 populate_metadata read hostname 'smithi129' 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b3d53e40 0x5594b419dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 4 client.9645 mds_command: new command op to 9492 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 --> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594b40b1520 con 0x5594b3d53e40 2024-09-17T04:01:02.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf54a00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b3d53e40 0x5594b419dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf54a00640 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b3d53e40 0x5594b419dbc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de5790fc1484930c server_cookie=20332276f96b3e14 in_seq=0 out_seq=0 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.075+0000 7fbf35600640 10 client.9645 ms_handle_connect on v2:172.21.15.129:6834/1228169144 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.076+0000 7fbf35600640 1 -- 172.21.15.129:0/2924099684 <== mds.0 v2:172.21.15.129:6834/1228169144 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5594b40b1520 con 0x5594b3d53e40 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.076+0000 7fbf35600640 10 client.9645 handle_command_reply: tid=0 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.076+0000 7fbf5e485740 1 client.9645 shutdown 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 msgr2=0x5594b415b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c010fa0 tx=0x7fbf3c0046c0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 shutdown_connections 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b3d53e40 0x5594b419dbc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b415f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b415b3c0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 --2- 172.21.15.129:0/2924099684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b417b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 >> 172.21.15.129:0/2924099684 conn(0x5594b3d893c0 msgr2=0x5594b4137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf35600640 0 client.9645 ms_handle_reset on v2:172.21.15.129:6834/1228169144 2024-09-17T04:01:02.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 shutdown_connections 2024-09-17T04:01:02.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 1 -- 172.21.15.129:0/2924099684 wait complete. 2024-09-17T04:01:02.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.077+0000 7fbf5e485740 20 client.9645 trim_cache size 0 max 16384 2024-09-17T04:01:02.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.096+0000 7fbf5e485740 1 Processor -- start 2024-09-17T04:01:02.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.096+0000 7fbf5e485740 1 -- start start 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.097+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b4180990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.097+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 0x5594b4180ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf5e485740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbf502010a0 con 0x5594b3d95270 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbf50200400 con 0x5594b3d74590 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbf50200a50 con 0x5594b3d6c3b0 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35316/0 (socket says 172.21.15.129:35316) 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 -- 172.21.15.129:0/785707512 learned_addr learned my addr 172.21.15.129:0/785707512 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf54a00640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b4180990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 0x5594b4180ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 -- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 msgr2=0x5594b4180990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b4180990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 -- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 msgr2=0x5594b4180ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 0x5594b4180ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf55400640 1 -- 172.21.15.129:0/785707512 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594b413c650 con 0x5594b3d95270 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.098+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 0x5594b4180ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.099+0000 7fbf55400640 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fbf4802aad0 tx=0x7fbf48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c77d8cf4e979bce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.099+0000 7fbf37e00640 1 -- 172.21.15.129:0/785707512 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4802eae0 con 0x5594b3d95270 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.099+0000 7fbf37e00640 1 -- 172.21.15.129:0/785707512 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf48044050 con 0x5594b3d95270 2024-09-17T04:01:02.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.099+0000 7fbf37e00640 1 -- 172.21.15.129:0/785707512 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48003970 con 0x5594b3d95270 2024-09-17T04:01:02.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 -- 172.21.15.129:0/785707512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 msgr2=0x5594b4165130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fbf4802aad0 tx=0x7fbf48004a40 comp rx=0 tx=0).stop 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 -- 172.21.15.129:0/785707512 shutdown_connections 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d95270 0x5594b4165130 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d74590 0x5594b4180ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 --2- 172.21.15.129:0/785707512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b4180990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 -- 172.21.15.129:0/785707512 >> 172.21.15.129:0/785707512 conn(0x5594b3d893c0 msgr2=0x5594b4134c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 -- 172.21.15.129:0/785707512 shutdown_connections 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.100+0000 7fbf5e485740 1 -- 172.21.15.129:0/785707512 wait complete. 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.101+0000 7fbf5e485740 1 -- start start 2024-09-17T04:01:02.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.101+0000 7fbf5e485740 1 Processor -- start 2024-09-17T04:01:02.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.102+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b41967d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.102+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf5e485740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b42002a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf5e485740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5594b4162c30 con 0x5594b3d74590 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5594b415b900 con 0x5594b3d95270 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf5e485740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5594b415f8c0 con 0x5594b3d6c3b0 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf5e485740 10 client.0 _ll_register_callbacks cb 0x7fbf3c010c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35326/0 (socket says 172.21.15.129:35326) 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b41967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf4fe00640 1 -- 172.21.15.129:0/456585 learned_addr learned my addr 172.21.15.129:0/456585 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf55400640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b42002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 -- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 msgr2=0x5594b42002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b42002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 -- 172.21.15.129:0/456585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 msgr2=0x5594b41ffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf54a00640 1 -- 172.21.15.129:0/456585 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x5594b415b120 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.103+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf54a00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b41967d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf44000ff0 tx=0x7fbf44098620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96fcc27f7c99ddc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 10 client.0 ms_handle_connect on v2:172.21.15.181:3301/0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4406b980 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mon.2 v2:172.21.15.181:3301/0 2 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fbf44085e10 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf48044050 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_get_version_reply(handle=1 version=154) ==== 24+0+0 (secure 0 0 0) 0x7fbf4406bbd0 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 10 client.7781 fetch_fsmap learned FSMap version 154 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) -- 0x5594b4195720 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mon.2 v2:172.21.15.181:3301/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbf48044050 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mon.2 v2:172.21.15.181:3301/0 5 ==== fsmap(e 154) ==== 3502+0+0 (secure 0 0 0) 0x7fbf4407c020 con 0x5594b3d6c3b0 2024-09-17T04:01:02.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 10 client.7781 fetch_fsmap finished waiting for FSMap version 154 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 10 client.7781 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.104+0000 7fbf5e485740 20 client.7781 populate_metadata read hostname 'smithi129' 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b4195140 0x5594b4203ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf5e485740 4 client.7781 mds_command: new command op to 9492 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 --> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] -- command(tid 0: {"prefix": "status"}) -- 0x5594b4183ac0 con 0x5594b4195140 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b4195140 0x5594b4203ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf4fe00640 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b4195140 0x5594b4203ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82e55e8935b18c0 server_cookie=f2418fb49380b24e in_seq=0 out_seq=0 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf35600640 10 client.7781 ms_handle_connect on v2:172.21.15.129:6834/1228169144 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf35600640 1 -- 172.21.15.129:0/456585 <== mds.0 v2:172.21.15.129:6834/1228169144 1 ==== command_reply(tid 0: 0 ) ==== 8+0+328 (crc 0 0 0) 0x5594b4183ac0 con 0x5594b4195140 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.105+0000 7fbf35600640 10 client.7781 handle_command_reply: tid=0 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 client.7781 shutdown 2024-09-17T04:01:02.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 msgr2=0x5594b41967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b41967d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbf44000ff0 tx=0x7fbf44098620 comp rx=0 tx=0).stop 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 shutdown_connections 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:6834/1228169144,v1:172.21.15.129:6835/1228169144] conn(0x5594b4195140 0x5594b4203ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5594b3d95270 0x5594b42002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5594b3d74590 0x5594b41ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf35600640 0 client.7781 ms_handle_reset on v2:172.21.15.129:6834/1228169144 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 --2- 172.21.15.129:0/456585 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5594b3d6c3b0 0x5594b41967d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 >> 172.21.15.129:0/456585 conn(0x5594b3d893c0 msgr2=0x5594b4180990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 shutdown_connections 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 1 -- 172.21.15.129:0/456585 wait complete. 2024-09-17T04:01:02.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.106+0000 7fbf5e485740 20 client.7781 trim_cache size 0 max 16384 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 9492, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.640633613, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 154, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 174, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 174, 2024-09-17T04:01:02.111 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.690159906000002 2024-09-17T04:01:02.112 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:01:02.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 msgr2=0x7fbf3007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 0x7fbf3007b0a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbf50159030 tx=0x7fbf3c005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:02.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 msgr2=0x7fbf501594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fbf4400c3a0 tx=0x7fbf4400c860 comp rx=0 tx=0).stop 2024-09-17T04:01:02.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 shutdown_connections 2024-09-17T04:01:02.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbf30078b90 0x7fbf3007b0a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbf500cd1c0 0x7fbf50156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbf500b4140 0x7fbf500b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 --2- 172.21.15.129:0/352968159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbf500705a0 0x7fbf501594b0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 >> 172.21.15.129:0/352968159 conn(0x7fbf5006d410 msgr2=0x7fbf500ad8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 shutdown_connections 2024-09-17T04:01:02.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.107+0000 7fbf57400640 1 -- 172.21.15.129:0/352968159 wait complete. 2024-09-17T04:01:02.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:01:02.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 -- 172.21.15.129:0/217869286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 msgr2=0x7fd02c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 --2- 172.21.15.129:0/217869286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0ccc80 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fd02c0d1370 tx=0x7fd01402a530 comp rx=0 tx=0).stop 2024-09-17T04:01:02.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 -- 172.21.15.129:0/217869286 shutdown_connections 2024-09-17T04:01:02.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 --2- 172.21.15.129:0/217869286 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0ccc80 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 --2- 172.21.15.129:0/217869286 >> v2:172.21.15.181:3300/0 conn(0x7fd02c0705a0 0x7fd02c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 -- 172.21.15.129:0/217869286 >> 172.21.15.129:0/217869286 conn(0x7fd02c06d410 msgr2=0x7fd02c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 -- 172.21.15.129:0/217869286 shutdown_connections 2024-09-17T04:01:02.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.352+0000 7fd031c00640 1 -- 172.21.15.129:0/217869286 wait complete. 2024-09-17T04:01:02.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.354+0000 7fd031c00640 1 Processor -- start 2024-09-17T04:01:02.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.354+0000 7fd031c00640 1 -- start start 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c06f2e0 0x7fd02c157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd02c0705a0 0x7fd02c157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c161250 con 0x7fd02c06eed0 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c1610d0 con 0x7fd02c06f2e0 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd031c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd02c1613d0 con 0x7fd02c0705a0 2024-09-17T04:01:02.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35334/0 (socket says 172.21.15.129:35334) 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 -- 172.21.15.129:0/1621768027 learned_addr learned my addr 172.21.15.129:0/1621768027 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd022000640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c06f2e0 0x7fd02c157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02b400640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd02c0705a0 0x7fd02c157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 -- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd02c0705a0 msgr2=0x7fd02c157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd02c0705a0 0x7fd02c157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 -- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c06f2e0 msgr2=0x7fd02c157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c06f2e0 0x7fd02c157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.356+0000 7fd02aa00640 1 -- 172.21.15.129:0/1621768027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c0c5b80 con 0x7fd02c06eed0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd02aa00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e790 tx=0x7fd01c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c00eea0 con 0x7fd02c06eed0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd01c00cdb0 con 0x7fd02c06eed0 2024-09-17T04:01:02.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c010690 con 0x7fd02c06eed0 2024-09-17T04:01:02.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02c15ac70 con 0x7fd02c06eed0 2024-09-17T04:01:02.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.357+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02c15a190 con 0x7fd02c06eed0 2024-09-17T04:01:02.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.358+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd02c0cd5a0 con 0x7fd02c06eed0 2024-09-17T04:01:02.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.360+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd01c010830 con 0x7fd02c06eed0 2024-09-17T04:01:02.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.361+0000 7fd023400640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 0x7fd00807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:02.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.361+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fd01c09b820 con 0x7fd02c06eed0 2024-09-17T04:01:02.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.362+0000 7fd022000640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 0x7fd00807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:02.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.362+0000 7fd022000640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 0x7fd00807b0a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd014034000 tx=0x7fd014004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:02.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.366+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd01c063bc0 con 0x7fd02c06eed0 2024-09-17T04:01:02.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.588+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd02c06a140 con 0x7fd02c06eed0 2024-09-17T04:01:02.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.589+0000 7fd023400640 1 -- 172.21.15.129:0/1621768027 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v154) ==== 53+0+73 (secure 0 0 0) 0x7fd01c0634c0 con 0x7fd02c06eed0 2024-09-17T04:01:02.589 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:01:02.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 msgr2=0x7fd00807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 0x7fd00807b0a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd014034000 tx=0x7fd014004730 comp rx=0 tx=0).stop 2024-09-17T04:01:02.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 msgr2=0x7fd02c0c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00e790 tx=0x7fd01c00ec50 comp rx=0 tx=0).stop 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 shutdown_connections 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd008078b90 0x7fd00807b0a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd02c0705a0 0x7fd02c157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd02c06f2e0 0x7fd02c157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 --2- 172.21.15.129:0/1621768027 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd02c06eed0 0x7fd02c0c5640 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 >> 172.21.15.129:0/1621768027 conn(0x7fd02c06d410 msgr2=0x7fd02c0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 shutdown_connections 2024-09-17T04:01:02.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02.591+0000 7fd031c00640 1 -- 172.21.15.129:0/1621768027 wait complete. 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:01:02.614 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:02.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:02.615 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:01:02.650 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:01:02.650 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:02.651 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 42029sec preferred_lft 42029sec 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:01:02.652 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:01:02.653 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:02.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:02.654 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:01:02.744 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:01:02.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:02.747 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:01:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:01:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:01:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:01:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:01:02.783 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:01:02.784 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:01:02.785 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:01:02.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:01:02.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:01:02.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:01:02.785 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:01:02.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:01:02.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:02.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:02.984 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:01:03.019 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:01:03.019 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:01:03.019 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:03.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:03.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:01:03.115 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:01:03.115 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:03.150 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:03.150 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:01:03.150 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:03.224 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:01:03.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:03.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:03.419 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:03.419 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:03.419 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:03.440 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:03.440 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:03.440 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:03.440 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:01:03.440 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:01:03.470 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:01:03.472 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:01:03.472 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:01:03.472 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:01:03.472 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:01:03.472 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:01:03.475 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:01:03.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:03.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:03.531 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:01:03.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:01:03.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:03.628 INFO:teuthology.orchestra.run.smithi129.stdout:client9663 2024-09-17T04:01:03.629 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:03.629 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:03.666 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:03.666 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:03.742 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:01:03.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:01:03.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:03.879 INFO:teuthology.orchestra.run.smithi129.stdout:client9663 2024-09-17T04:01:03.880 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:03.880 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> import fcntl 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> import sys 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> import time 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> with open("/home/ubuntu/cephtest/mnt.0/open_file", '"'"'w'"'"') as f: 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> f.write("content") 2024-09-17T04:01:03.916 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> while True: 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> print("open_background: keeping file open", file=sys.stderr) 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> if os.read(0, 4096) == b"": 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> break 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> except BlockingIOError: 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> pass 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> time.sleep(2) 2024-09-17T04:01:03.917 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:01:03.960 DEBUG:teuthology.orchestra.run.smithi129:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file 2024-09-17T04:01:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:01:03.999 INFO:teuthology.orchestra.run.smithi129.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file': No such file or directory 2024-09-17T04:01:04.067 INFO:teuthology.orchestra.run.smithi129.stderr:open_background: keeping file open 2024-09-17T04:01:05.000 DEBUG:teuthology.orchestra.run.smithi129:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file 2024-09-17T04:01:05.036 INFO:teuthology.orchestra.run.smithi129.stdout:7 2024-09-17T04:01:05.036 INFO:tasks.cephfs.mount:File open_file became visible with size 7 from 0 after 1s 2024-09-17T04:01:05.037 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:05.037 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:05.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:05.116 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T04:01:05.293 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T04:01:05.293 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T04:01:05.293 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0971385 s, 86.4 MB/s 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/open_file").st_ino) 2024-09-17T04:01:05.295 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:01:05.394 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627776 2024-09-17T04:01:05.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:05.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:05.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 -- 172.21.15.129:0/2446214153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 msgr2=0x7fb70c0b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 --2- 172.21.15.129:0/2446214153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c0b4180 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fb70c0af320 tx=0x7fb6fc02a560 comp rx=0 tx=0).stop 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 -- 172.21.15.129:0/2446214153 shutdown_connections 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 --2- 172.21.15.129:0/2446214153 >> v2:172.21.15.181:3300/0 conn(0x7fb70c0cd1c0 0x7fb70c070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 --2- 172.21.15.129:0/2446214153 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0b46c0 0x7fb70c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 --2- 172.21.15.129:0/2446214153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c0b4180 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 -- 172.21.15.129:0/2446214153 >> 172.21.15.129:0/2446214153 conn(0x7fb70c06d410 msgr2=0x7fb70c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 -- 172.21.15.129:0/2446214153 shutdown_connections 2024-09-17T04:01:05.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.670+0000 7fb712c00640 1 -- 172.21.15.129:0/2446214153 wait complete. 2024-09-17T04:01:05.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.673+0000 7fb712c00640 1 Processor -- start 2024-09-17T04:01:05.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.673+0000 7fb712c00640 1 -- start start 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb70c0b46c0 0x7fb70c0c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0cd1c0 0x7fb70c0986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c0707c0 con 0x7fb70c15da00 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c070640 con 0x7fb70c0b46c0 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.674+0000 7fb712c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb70c070940 con 0x7fb70c0cd1c0 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45232/0 (socket says 172.21.15.129:45232) 2024-09-17T04:01:05.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 -- 172.21.15.129:0/3703130829 learned_addr learned my addr 172.21.15.129:0/3703130829 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb70be00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb70c0b46c0 0x7fb70c0c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb703400640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0cd1c0 0x7fb70c0986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 -- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0cd1c0 msgr2=0x7fb70c0986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0cd1c0 0x7fb70c0986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 -- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb70c0b46c0 msgr2=0x7fb70c0c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb70c0b46c0 0x7fb70c0c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 -- 172.21.15.129:0/3703130829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c15a3f0 con 0x7fb70c15da00 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.675+0000 7fb710c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb70400d960 tx=0x7fb70400de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.676+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704016070 con 0x7fb70c15da00 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.676+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb70400be20 con 0x7fb70c15da00 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.676+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704003bb0 con 0x7fb70c15da00 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.676+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1532a0 con 0x7fb70c15da00 2024-09-17T04:01:05.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.676+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c074ad0 con 0x7fb70c15da00 2024-09-17T04:01:05.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.677+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb70c076760 con 0x7fb70c15da00 2024-09-17T04:01:05.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.678+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb704010460 con 0x7fb70c15da00 2024-09-17T04:01:05.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.680+0000 7fb708c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 0x7fb6e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:05.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.680+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fb70409b430 con 0x7fb70c15da00 2024-09-17T04:01:05.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.680+0000 7fb70be00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 0x7fb6e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:05.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.681+0000 7fb70be00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 0x7fb6e807b0a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc033000 tx=0x7fb6fc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:05.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.685+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb70c076760 con 0x7fb70c15da00 2024-09-17T04:01:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.901+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb70c0adb80 con 0x7fb70c15da00 2024-09-17T04:01:05.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.902+0000 7fb708c00640 1 -- 172.21.15.129:0/3703130829 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fb704063190 con 0x7fb70c15da00 2024-09-17T04:01:05.902 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:05.902 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:05.902 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 msgr2=0x7fb6e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 0x7fb6e807b0a0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc033000 tx=0x7fb6fc004730 comp rx=0 tx=0).stop 2024-09-17T04:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 msgr2=0x7fb70c098c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb70400d960 tx=0x7fb70400de20 comp rx=0 tx=0).stop 2024-09-17T04:01:05.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 shutdown_connections 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6e8078b90 0x7fb6e807b0a0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb70c15da00 0x7fb70c098c00 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb70c0cd1c0 0x7fb70c0986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 --2- 172.21.15.129:0/3703130829 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb70c0b46c0 0x7fb70c0c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 >> 172.21.15.129:0/3703130829 conn(0x7fb70c06d410 msgr2=0x7fb70c0997c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 shutdown_connections 2024-09-17T04:01:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:05.904+0000 7fb712c00640 1 -- 172.21.15.129:0/3703130829 wait complete. 2024-09-17T04:01:05.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-17T04:01:06.069 INFO:teuthology.orchestra.run.smithi129.stderr:open_background: keeping file open 2024-09-17T04:01:06.168 INFO:teuthology.orchestra.run.smithi129.stdout:[{"id":9663,"entity":{"name":{"type":"client","num":9663},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":3021456163}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":49,"uptime":2.6929419320000001,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":4.9745515551625203,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":51,"last_trim_completed_flushes_tid":4,"delegated_inos":[],"inst":"client.9663 v1:192.168.0.1:0/3021456163","completed_requests":[{"tid":51,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000037fbff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000080ffff"}},"entity_id":"0","hostname":"smithi129","kernel_version":"6.11.0-rc5-g2ca44a2f35ca","root":"/"}}] 2024-09-17T04:01:06.168 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x000000000037fbff" }, "entity_id": "0", "hostname": "smithi129", "kernel_version": "6.11.0-rc5-g2ca44a2f35ca", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000080ffff" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 51 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 3021456163, "type": "v1" }, "name": { "num": 9663, "type": "client" } }, "id": 9663, "inst": "client.9663 v1:192.168.0.1:0/3021456163", "last_trim_completed_flushes_tid": 4, "last_trim_completed_requests_tid": 51, "num_caps": 2, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 49, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 4.97455155516252 }, "state": "open", "uptime": 2.692941932 } ] 2024-09-17T04:01:06.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:06.169 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f open_file) 2024-09-17T04:01:06.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 -- 172.21.15.129:0/2472783841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa35406eed0 msgr2=0x7fa3540ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 --2- 172.21.15.129:0/2472783841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa35406eed0 0x7fa3540ccc80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009830 tx=0x7fa33c02a510 comp rx=0 tx=0).stop 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 -- 172.21.15.129:0/2472783841 shutdown_connections 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 --2- 172.21.15.129:0/2472783841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa35406eed0 0x7fa3540ccc80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 --2- 172.21.15.129:0/2472783841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3540705a0 0x7fa3540b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.468+0000 7fa35a200640 1 -- 172.21.15.129:0/2472783841 >> 172.21.15.129:0/2472783841 conn(0x7fa35406d410 msgr2=0x7fa3540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.469+0000 7fa35a200640 1 -- 172.21.15.129:0/2472783841 shutdown_connections 2024-09-17T04:01:06.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.469+0000 7fa35a200640 1 -- 172.21.15.129:0/2472783841 wait complete. 2024-09-17T04:01:06.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.471+0000 7fa35a200640 1 Processor -- start 2024-09-17T04:01:06.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.471+0000 7fa35a200640 1 -- start start 2024-09-17T04:01:06.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa35406eed0 0x7fa354159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:06.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:06.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3540705a0 0x7fa3541f91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa354160200 con 0x7fa35406f2e0 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa3541612e0 con 0x7fa3540705a0 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.472+0000 7fa35a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa3540cd1c0 con 0x7fa35406eed0 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45248/0 (socket says 172.21.15.129:45248) 2024-09-17T04:01:06.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 -- 172.21.15.129:0/781350940 learned_addr learned my addr 172.21.15.129:0/781350940 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa353400640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa35406eed0 0x7fa354159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa353e00640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3540705a0 0x7fa3541f91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 -- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa35406eed0 msgr2=0x7fa354159590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa35406eed0 0x7fa354159590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 -- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3540705a0 msgr2=0x7fa3541f91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3540705a0 0x7fa3541f91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.473+0000 7fa34aa00640 1 -- 172.21.15.129:0/781350940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa354159d90 con 0x7fa35406f2e0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa34aa00640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009950 tx=0x7fa33c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c005c90 con 0x7fa35406f2e0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa33c031e50 con 0x7fa35406f2e0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa354159ad0 con 0x7fa35406f2e0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa33c005260 con 0x7fa35406f2e0 2024-09-17T04:01:06.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.474+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa35415c100 con 0x7fa35406f2e0 2024-09-17T04:01:06.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.475+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3540ccc80 con 0x7fa35406f2e0 2024-09-17T04:01:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.476+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa33c005400 con 0x7fa35406f2e0 2024-09-17T04:01:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.478+0000 7fa34be00640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 0x7fa33007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.478+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fa33c0bbf20 con 0x7fa35406f2e0 2024-09-17T04:01:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.478+0000 7fa353400640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 0x7fa33007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:06.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.479+0000 7fa353400640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 0x7fa33007b0a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa344005fd0 tx=0x7fa344009260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:06.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.483+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa33c03f030 con 0x7fa35406f2e0 2024-09-17T04:01:06.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.706+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3540ca8c0 con 0x7fa35406f2e0 2024-09-17T04:01:06.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.707+0000 7fa34be00640 1 -- 172.21.15.129:0/781350940 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fa33c084240 con 0x7fa35406f2e0 2024-09-17T04:01:06.707 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:06.707 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:06.707 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:06.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 msgr2=0x7fa33007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:06.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 0x7fa33007b0a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa344005fd0 tx=0x7fa344009260 comp rx=0 tx=0).stop 2024-09-17T04:01:06.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 msgr2=0x7fa3541f8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:06.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fa33c009950 tx=0x7fa33c033c20 comp rx=0 tx=0).stop 2024-09-17T04:01:06.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 shutdown_connections 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa330078b90 0x7fa33007b0a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3540705a0 0x7fa3541f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa35406f2e0 0x7fa3541f8ca0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.709+0000 7fa35a200640 1 --2- 172.21.15.129:0/781350940 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa35406eed0 0x7fa354159590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.710+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 >> 172.21.15.129:0/781350940 conn(0x7fa35406d410 msgr2=0x7fa354073380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.710+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 shutdown_connections 2024-09-17T04:01:06.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:06.710+0000 7fa35a200640 1 -- 172.21.15.129:0/781350940 wait complete. 2024-09-17T04:01:06.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:01:06.972 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":1}} 2024-09-17T04:01:06.973 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 1 } } 2024-09-17T04:01:06.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:07.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.192+0000 7fc414a00640 1 -- 172.21.15.129:0/4041450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006f2e0 msgr2=0x7fc4100cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.192+0000 7fc414a00640 1 --2- 172.21.15.129:0/4041450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006f2e0 0x7fc4100cd1c0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fc4100ccc10 tx=0x7fc40802aa30 comp rx=0 tx=0).stop 2024-09-17T04:01:07.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 -- 172.21.15.129:0/4041450729 shutdown_connections 2024-09-17T04:01:07.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 --2- 172.21.15.129:0/4041450729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006f2e0 0x7fc4100cd1c0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 --2- 172.21.15.129:0/4041450729 >> v2:172.21.15.181:3300/0 conn(0x7fc41006eed0 0x7fc4100ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 --2- 172.21.15.129:0/4041450729 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4100b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 -- 172.21.15.129:0/4041450729 >> 172.21.15.129:0/4041450729 conn(0x7fc41006d410 msgr2=0x7fc4100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:07.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 -- 172.21.15.129:0/4041450729 shutdown_connections 2024-09-17T04:01:07.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.193+0000 7fc414a00640 1 -- 172.21.15.129:0/4041450729 wait complete. 2024-09-17T04:01:07.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.195+0000 7fc414a00640 1 Processor -- start 2024-09-17T04:01:07.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.195+0000 7fc414a00640 1 -- start start 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.196+0000 7fc414a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.196+0000 7fc414a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 0x7fc410158370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc414a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4101588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc414a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc410160b40 con 0x7fc41006eed0 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc414a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc4101609c0 con 0x7fc41006f2e0 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc414a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc410160cc0 con 0x7fc4100705a0 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45260/0 (socket says 172.21.15.129:45260) 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 -- 172.21.15.129:0/2430332511 learned_addr learned my addr 172.21.15.129:0/2430332511 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:07.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc405400640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 0x7fc410158370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40e800640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4101588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 -- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 msgr2=0x7fc4101588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4101588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 -- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 msgr2=0x7fc410158370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 0x7fc410158370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40de00640 1 -- 172.21.15.129:0/2430332511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4100c68d0 con 0x7fc41006eed0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc40e800640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4101588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.197+0000 7fc405400640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 0x7fc410158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc40de00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc40000d770 tx=0x7fc40000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc400017070 con 0x7fc41006eed0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc400009d20 con 0x7fc41006eed0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc410156f00 con 0x7fc41006eed0 2024-09-17T04:01:07.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000106c0 con 0x7fc41006eed0 2024-09-17T04:01:07.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.198+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4101574c0 con 0x7fc41006eed0 2024-09-17T04:01:07.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.199+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc410161220 con 0x7fc41006eed0 2024-09-17T04:01:07.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.200+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc40001a070 con 0x7fc41006eed0 2024-09-17T04:01:07.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.201+0000 7fc406a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 0x7fc3ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.201+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fc40009aeb0 con 0x7fc41006eed0 2024-09-17T04:01:07.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.202+0000 7fc405400640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 0x7fc3ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.203+0000 7fc405400640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 0x7fc3ec07b0a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc4100b2b10 tx=0x7fc3f8005d50 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:07.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.206+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc4000632e0 con 0x7fc41006eed0 2024-09-17T04:01:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.423+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc41006a140 con 0x7fc41006eed0 2024-09-17T04:01:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.423+0000 7fc406a00640 1 -- 172.21.15.129:0/2430332511 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fc400062be0 con 0x7fc41006eed0 2024-09-17T04:01:07.423 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:07.424 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:07.424 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 msgr2=0x7fc3ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 0x7fc3ec07b0a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fc4100b2b10 tx=0x7fc3f8005d50 comp rx=0 tx=0).stop 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 msgr2=0x7fc4100c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fc40000d770 tx=0x7fc40000dc30 comp rx=0 tx=0).stop 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 shutdown_connections 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc3ec078b90 0x7fc3ec07b0a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4100705a0 0x7fc4101588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41006f2e0 0x7fc410158370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 --2- 172.21.15.129:0/2430332511 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc41006eed0 0x7fc4100c6390 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 >> 172.21.15.129:0/2430332511 conn(0x7fc41006d410 msgr2=0x7fc4100aee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 shutdown_connections 2024-09-17T04:01:07.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.426+0000 7fc414a00640 1 -- 172.21.15.129:0/2430332511 wait complete. 2024-09-17T04:01:07.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T04:01:07.687 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":1}} 2024-09-17T04:01:07.687 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 1 } } 2024-09-17T04:01:07.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:07.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 -- 172.21.15.129:0/3740909727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406eed0 msgr2=0x7f96e40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 --2- 172.21.15.129:0/3740909727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406eed0 0x7f96e40ccc80 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f96cc005fd0 tx=0x7f96cc02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 -- 172.21.15.129:0/3740909727 shutdown_connections 2024-09-17T04:01:07.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 --2- 172.21.15.129:0/3740909727 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406eed0 0x7f96e40ccc80 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 --2- 172.21.15.129:0/3740909727 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e40705a0 0x7f96e40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 -- 172.21.15.129:0/3740909727 >> 172.21.15.129:0/3740909727 conn(0x7f96e406d410 msgr2=0x7f96e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:07.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 -- 172.21.15.129:0/3740909727 shutdown_connections 2024-09-17T04:01:07.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.907+0000 7f96eba00640 1 -- 172.21.15.129:0/3740909727 wait complete. 2024-09-17T04:01:07.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.909+0000 7f96eba00640 1 Processor -- start 2024-09-17T04:01:07.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.909+0000 7f96eba00640 1 -- start start 2024-09-17T04:01:07.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.910+0000 7f96eba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e406eed0 0x7f96e4156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.910+0000 7f96eba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.910+0000 7f96eba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 0x7f96e4156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96eba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f96e4160200 con 0x7f96e406f2e0 2024-09-17T04:01:07.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96eba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f96e41612e0 con 0x7f96e40705a0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96eba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f96e40cd1c0 con 0x7f96e406eed0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45268/0 (socket says 172.21.15.129:45268) 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 -- 172.21.15.129:0/2182521071 learned_addr learned my addr 172.21.15.129:0/2182521071 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96e9000640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e406eed0 0x7f96e4156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96e9a00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 0x7f96e4156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 -- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e406eed0 msgr2=0x7f96e4156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e406eed0 0x7f96e4156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 -- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 msgr2=0x7f96e4156c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 0x7f96e4156c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96dbe00640 1 -- 172.21.15.129:0/2182521071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e40c6d50 con 0x7f96e406f2e0 2024-09-17T04:01:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.911+0000 7f96e9a00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 0x7f96e4156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:07.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96dbe00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f96cc006010 tx=0x7f96cc0038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:07.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc03e030 con 0x7f96e406f2e0 2024-09-17T04:01:07.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f96cc003ca0 con 0x7f96e406f2e0 2024-09-17T04:01:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e4159a70 con 0x7f96e406f2e0 2024-09-17T04:01:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96cc035040 con 0x7f96e406f2e0 2024-09-17T04:01:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.912+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e415a030 con 0x7f96e406f2e0 2024-09-17T04:01:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.913+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e40ccc80 con 0x7f96e406f2e0 2024-09-17T04:01:07.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.914+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f96cc041030 con 0x7f96e406f2e0 2024-09-17T04:01:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.915+0000 7f96e1600640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 0x7f96c007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:07.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.915+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f96cc0bb2e0 con 0x7f96e406f2e0 2024-09-17T04:01:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.916+0000 7f96e9000640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 0x7f96c007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.917+0000 7f96e9000640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 0x7f96c007b0d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f96d400bb50 tx=0x7f96d400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:07.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:07.920+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f96cc0830d0 con 0x7f96e406f2e0 2024-09-17T04:01:08.071 INFO:teuthology.orchestra.run.smithi129.stderr:open_background: keeping file open 2024-09-17T04:01:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.136+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96e406a140 con 0x7f96e406f2e0 2024-09-17T04:01:08.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.137+0000 7f96e1600640 1 -- 172.21.15.129:0/2182521071 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f96cc087b00 con 0x7f96e406f2e0 2024-09-17T04:01:08.137 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:08.137 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.139+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 msgr2=0x7f96c007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.139+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 0x7f96c007b0d0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f96d400bb50 tx=0x7f96d400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.139+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 msgr2=0x7f96e41566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.139+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f96cc006010 tx=0x7f96cc0038a0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 shutdown_connections 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f96c0078bc0 0x7f96c007b0d0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f96e40705a0 0x7f96e4156c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f96e406f2e0 0x7f96e41566d0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 --2- 172.21.15.129:0/2182521071 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f96e406eed0 0x7f96e4156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 >> 172.21.15.129:0/2182521071 conn(0x7f96e406d410 msgr2=0x7f96e40d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 shutdown_connections 2024-09-17T04:01:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.140+0000 7f96eba00640 1 -- 172.21.15.129:0/2182521071 wait complete. 2024-09-17T04:01:08.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T04:01:08.408 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T04:01:08.409 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T04:01:08.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:08.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 -- 172.21.15.129:0/3777039785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b13d0 msgr2=0x7f4d600b17e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 --2- 172.21.15.129:0/3777039785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b13d0 0x7f4d600b17e0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f4d600ba660 tx=0x7f4d5802a560 comp rx=0 tx=0).stop 2024-09-17T04:01:08.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 -- 172.21.15.129:0/3777039785 shutdown_connections 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 --2- 172.21.15.129:0/3777039785 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b13d0 0x7f4d600b17e0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 --2- 172.21.15.129:0/3777039785 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 0x7f4d600d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 --2- 172.21.15.129:0/3777039785 >> v2:172.21.15.181:3300/0 conn(0x7f4d600b9be0 0x7f4d600d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 -- 172.21.15.129:0/3777039785 >> 172.21.15.129:0/3777039785 conn(0x7f4d6008bff0 msgr2=0x7f4d6009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 -- 172.21.15.129:0/3777039785 shutdown_connections 2024-09-17T04:01:08.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.632+0000 7f4d64c00640 1 -- 172.21.15.129:0/3777039785 wait complete. 2024-09-17T04:01:08.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.634+0000 7f4d64c00640 1 Processor -- start 2024-09-17T04:01:08.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.634+0000 7f4d64c00640 1 -- start start 2024-09-17T04:01:08.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600b13d0 0x7f4d600ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 0x7f4d6014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4d600b20f0 con 0x7f4d600b9be0 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4d600b1f70 con 0x7f4d600b13d0 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d64c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4d600b2270 con 0x7f4d600b3430 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45282/0 (socket says 172.21.15.129:45282) 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 -- 172.21.15.129:0/809011820 learned_addr learned my addr 172.21.15.129:0/809011820 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:08.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d56a00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 0x7f4d6014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5f400640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600b13d0 0x7f4d600ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 -- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 msgr2=0x7f4d6014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 0x7f4d6014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 -- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600b13d0 msgr2=0x7f4d600ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600b13d0 0x7f4d600ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.636+0000 7f4d5fe00640 1 -- 172.21.15.129:0/809011820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d600cae00 con 0x7f4d600b9be0 2024-09-17T04:01:08.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d5fe00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4d580099c0 tx=0x7f4d58003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5803e030 con 0x7f4d600b9be0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d58003e40 con 0x7f4d600b9be0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d600cd330 con 0x7f4d600b9be0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d58035040 con 0x7f4d600b9be0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.637+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d600cd8f0 con 0x7f4d600b9be0 2024-09-17T04:01:08.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.638+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d600b17e0 con 0x7f4d600b9be0 2024-09-17T04:01:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.640+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4d58041030 con 0x7f4d600b9be0 2024-09-17T04:01:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.640+0000 7f4d57e00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 0x7f4d3c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:08.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.641+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f4d580bb4f0 con 0x7f4d600b9be0 2024-09-17T04:01:08.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.641+0000 7f4d5f400640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 0x7f4d3c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:08.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.642+0000 7f4d5f400640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 0x7f4d3c07b0d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4d600d4000 tx=0x7f4d50005e30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:08.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.645+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4d58083920 con 0x7f4d600b9be0 2024-09-17T04:01:08.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.861+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d6006a130 con 0x7f4d600b9be0 2024-09-17T04:01:08.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.861+0000 7f4d57e00640 1 -- 172.21.15.129:0/809011820 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f4d58083220 con 0x7f4d600b9be0 2024-09-17T04:01:08.862 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:08.862 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:08.862 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 msgr2=0x7f4d3c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 0x7f4d3c07b0d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4d600d4000 tx=0x7f4d50005e30 comp rx=0 tx=0).stop 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 msgr2=0x7f4d600cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f4d580099c0 tx=0x7f4d58003910 comp rx=0 tx=0).stop 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 shutdown_connections 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d3c078bc0 0x7f4d3c07b0d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d600b9be0 0x7f4d600cc680 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d600b3430 0x7f4d6014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 --2- 172.21.15.129:0/809011820 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d600b13d0 0x7f4d600ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 >> 172.21.15.129:0/809011820 conn(0x7f4d6008bff0 msgr2=0x7f4d6006ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 shutdown_connections 2024-09-17T04:01:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:08.864+0000 7f4d64c00640 1 -- 172.21.15.129:0/809011820 wait complete. 2024-09-17T04:01:08.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-17T04:01:09.134 INFO:teuthology.orchestra.run.smithi129.stdout:[{"id":9663,"entity":{"name":{"type":"client","num":9663},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":3021456163}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":48,"uptime":5.6578779990000001,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.9338869364704898,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":52,"last_trim_completed_flushes_tid":5,"delegated_inos":[],"inst":"client.9663 v1:192.168.0.1:0/3021456163","completed_requests":[{"tid":52,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000037fbff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000080ffff"}},"entity_id":"0","hostname":"smithi129","kernel_version":"6.11.0-rc5-g2ca44a2f35ca","root":"/"}}] 2024-09-17T04:01:09.135 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x000000000037fbff" }, "entity_id": "0", "hostname": "smithi129", "kernel_version": "6.11.0-rc5-g2ca44a2f35ca", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000080ffff" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 52 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 3021456163, "type": "v1" }, "name": { "num": 9663, "type": "client" } }, "id": 9663, "inst": "client.9663 v1:192.168.0.1:0/3021456163", "last_trim_completed_flushes_tid": 5, "last_trim_completed_requests_tid": 52, "num_caps": 2, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 48, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.93388693647049 }, "state": "open", "uptime": 5.657877999 } ] 2024-09-17T04:01:09.135 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T04:01:09.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.273+0000 7f2b3f229f40 1 Processor -- start 2024-09-17T04:01:09.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.273+0000 7f2b3f229f40 1 -- start start 2024-09-17T04:01:09.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 0x5602973826a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297385b70 0x5602972b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x560297207c60 con 0x560297390140 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5602972661d0 con 0x5602972fd5c0 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.274+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5602973864c0 con 0x560297385b70 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45292/0 (socket says 172.21.15.129:45292) 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 -- 172.21.15.129:0/340151487 learned_addr learned my addr 172.21.15.129:0/340151487 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38400640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 0x5602973826a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b37a00640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297385b70 0x5602972b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 -- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297385b70 msgr2=0x5602972b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297385b70 0x5602972b7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 -- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 msgr2=0x5602973826a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 0x5602973826a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 -- 172.21.15.129:0/340151487 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602972bae40 con 0x560297390140 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38400640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 0x5602973826a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.275+0000 7f2b38e00640 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f2b30057040 tx=0x7f2b30003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dddfade198a6e0c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b36600640 1 -- 172.21.15.129:0/340151487 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30064030 con 0x560297390140 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b36600640 1 -- 172.21.15.129:0/340151487 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b30004020 con 0x560297390140 2024-09-17T04:01:09.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b36600640 1 -- 172.21.15.129:0/340151487 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30004270 con 0x560297390140 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 -- 172.21.15.129:0/340151487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 msgr2=0x5602972b84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f2b30057040 tx=0x7f2b30003330 comp rx=0 tx=0).stop 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 -- 172.21.15.129:0/340151487 shutdown_connections 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x560297390140 0x5602972b84f0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297385b70 0x5602972b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/340151487 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5602972fd5c0 0x5602973826a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 -- 172.21.15.129:0/340151487 >> 172.21.15.129:0/340151487 conn(0x5602971d8ac0 msgr2=0x5602972b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 -- 172.21.15.129:0/340151487 shutdown_connections 2024-09-17T04:01:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.276+0000 7f2b3f229f40 1 -- 172.21.15.129:0/340151487 wait complete. 2024-09-17T04:01:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.277+0000 7f2b3f229f40 1 Processor -- start 2024-09-17T04:01:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.278+0000 7f2b3f229f40 1 -- start start 2024-09-17T04:01:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560297385b70 0x56029738b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297390140 0x5602972b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56029738f110 con 0x5602972fd5c0 2024-09-17T04:01:09.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x560297384410 con 0x560297385b70 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b3f229f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56029738d8c0 con 0x560297390140 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b38400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b38400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45304/0 (socket says 172.21.15.129:45304) 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 learned_addr learned my addr 172.21.15.129:0/1279201948 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297390140 0x5602972b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.279+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560297385b70 0x56029738b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297390140 msgr2=0x5602972b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x560297390140 0x5602972b8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560297385b70 msgr2=0x56029738b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560297385b70 0x56029738b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56029738d3f0 con 0x5602972fd5c0 2024-09-17T04:01:09.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800a680 tx=0x7f2b2800ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b35200640 1 -- 172.21.15.129:0/1279201948 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2801a070 con 0x5602972fd5c0 2024-09-17T04:01:09.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b35200640 1 -- 172.21.15.129:0/1279201948 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2b28008930 con 0x5602972fd5c0 2024-09-17T04:01:09.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56029738abd0 con 0x5602972fd5c0 2024-09-17T04:01:09.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b35200640 1 -- 172.21.15.129:0/1279201948 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b28008d20 con 0x5602972fd5c0 2024-09-17T04:01:09.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.280+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560297391f40 con 0x5602972fd5c0 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.283+0000 7f2b35200640 1 -- 172.21.15.129:0/1279201948 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2b2801d070 con 0x5602972fd5c0 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.283+0000 7f2b35200640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 0x7f2b18079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.284+0000 7f2b35200640 1 -- 172.21.15.129:0/1279201948 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f2b28099540 con 0x5602972fd5c0 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.284+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 0x7f2b18079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.284+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5602972b92b0 0x560297381600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.284+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:1 27.0 27:00000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x5602972b92b0 2024-09-17T04:01:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.284+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5602972b92b0 0x560297381600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 0x7f2b18079a80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x5602973926a0 tx=0x7f2b2400e040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5602972b92b0 0x560297381600 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(1 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f2b3006a040 0x7f2b30055410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:2 27.20 27:04000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.285+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f2b3006a040 0x7f2b30055410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.286+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f2b3006a040 0x7f2b30055410 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.286+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(2 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.286+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:3 27.10 27:08000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x5602972b92b0 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.287+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(3 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.287+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f2b30022cf0 0x7f2b3006c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.287+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:4 27.30 27:0c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b30022cf0 2024-09-17T04:01:09.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.287+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f2b30022cf0 0x7f2b3006c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.287+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f2b30022cf0 0x7f2b3006c460 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.288+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(4 [pgnls start_epoch 174 out=81b] v174'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f2b2400c0b0 con 0x7f2b30022cf0 2024-09-17T04:01:09.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.288+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f2b24016040 0x7f2b24017970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.288+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:5 27.8 27:10000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.288+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f2b24016040 0x7f2b24017970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.289+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f2b24016040 0x7f2b24017970 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.290+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(5 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.290+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f2b3001f670 0x7f2b300698f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.290+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:6 27.28 27:14000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3001f670 2024-09-17T04:01:09.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.290+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f2b3001f670 0x7f2b300698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.291+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f2b3001f670 0x7f2b300698f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.291+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(6 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.291+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f2b280680b0 0x7f2b28060080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.291+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:7 27.18 27:18000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b280680b0 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.291+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f2b280680b0 0x7f2b28060080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.292+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f2b280680b0 0x7f2b28060080 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.292+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(7 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.292+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:8 27.38 27:1c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b280680b0 2024-09-17T04:01:09.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.292+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(8 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.292+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:9 27.4 27:20000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.293+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(9 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.293+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:10 27.24 27:24000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x5602972b92b0 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.293+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(10 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.293+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:11 27.14 27:28000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3001f670 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.294+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(11 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.294+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f2b28061610 0x7f2b28083a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.294+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 27.34 27:2c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b28061610 2024-09-17T04:01:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.294+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f2b28061610 0x7f2b28083a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.295+0000 7f2b38e00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f2b28061610 0x7f2b28083a70 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.295+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(12 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.295+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:13 27.c 27:30000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x5602972b92b0 2024-09-17T04:01:09.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.296+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(13 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.296+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:14 27.2c 27:34000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b280680b0 2024-09-17T04:01:09.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.296+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(14 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.296+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:15 27.1c 27:38000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(15 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:16 27.3c 27:3c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3001f670 2024-09-17T04:01:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(16 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:17 27.2 27:40000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(17 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.297+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 27.22 27:44000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.298+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(18 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.298+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:19 27.12 27:48000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b280680b0 2024-09-17T04:01:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(19 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:20 27.32 27:4c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(20 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:21 27.a 27:50000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b30022cf0 2024-09-17T04:01:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(21 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b37a00640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f2b2400ab30 0x7f2b24027940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.299+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:22 27.2a 27:54000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b2400ab30 2024-09-17T04:01:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.300+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f2b2400ab30 0x7f2b24027940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.301+0000 7f2b38400640 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f2b2400ab30 0x7f2b24027940 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:09.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.301+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(22 [pgnls start_epoch 174 out=81b] v174'1 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b2400ab30 2024-09-17T04:01:09.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.301+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 27.1a 27:58000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(23 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 27.3a 27:5c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b280680b0 2024-09-17T04:01:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(24 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:25 27.6 27:60000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b28061610 2024-09-17T04:01:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(25 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.302+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 27.26 27:64000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.303+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(26 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.303+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:27 27.16 27:68000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3001f670 2024-09-17T04:01:09.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.303+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(27 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.303+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:28 27.36 27:6c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b30022cf0 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.304+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(28 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.304+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:29 27.e 27:70000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.304+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(29 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.304+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:30 27.2e 27:74000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b30022cf0 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.305+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(30 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.305+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:31 27.1e 27:78000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.305+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(31 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.305+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:32 27.3e 27:7c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.306+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(32 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.306+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:33 27.1 27:80000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b24016040 2024-09-17T04:01:09.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.306+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(33 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.306+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:34 27.21 27:84000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b30022cf0 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.306+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(34 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.307+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:35 27.11 27:88000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x56029737f700 con 0x7f2b3006a040 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.307+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(35 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.307+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:36 27.31 27:8c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.307+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(36 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.307+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:37 27.9 27:90000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b2400ab30 2024-09-17T04:01:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.308+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(37 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b2400ab30 2024-09-17T04:01:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.308+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:38 27.29 27:94000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b2400ab30 2024-09-17T04:01:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.308+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(38 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b2400ab30 2024-09-17T04:01:09.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.308+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:39 27.19 27:98000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3001f670 2024-09-17T04:01:09.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.309+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 5 ==== osd_op_reply(39 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.309+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:40 27.39 27:9c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.309+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(40 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.309+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:41 27.5 27:a0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b2400ab30 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(41 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b2400ab30 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:42 27.25 27:a4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b28061610 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(42 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:43 27.15 27:a8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(43 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.310+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:44 27.35 27:ac000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x5602972b92b0 2024-09-17T04:01:09.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.311+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(44 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.311+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:45 27.d 27:b0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b30022cf0 2024-09-17T04:01:09.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.311+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(45 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.311+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:46 27.2d 27:b4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b30022cf0 2024-09-17T04:01:09.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(46 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:47 27.1d 27:b8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b30022cf0 2024-09-17T04:01:09.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(47 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:48 27.3d 27:bc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x5602972b92b0 2024-09-17T04:01:09.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(48 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.312+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:49 27.3 27:c0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b28061610 2024-09-17T04:01:09.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.313+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(49 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.313+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:50 27.23 27:c4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b28061610 2024-09-17T04:01:09.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.313+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(50 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.313+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:51 27.13 27:c8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b30022cf0 2024-09-17T04:01:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.314+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(51 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b30022cf0 2024-09-17T04:01:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.314+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:52 27.33 27:cc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b28061610 2024-09-17T04:01:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.314+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(52 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b28061610 2024-09-17T04:01:09.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.314+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:53 27.b 27:d0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x5602972b92b0 2024-09-17T04:01:09.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.315+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(53 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.315+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:54 27.2b 27:d4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3001f670 2024-09-17T04:01:09.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.316+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 6 ==== osd_op_reply(54 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.316+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:55 27.1b 27:d8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3001f670 2024-09-17T04:01:09.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.316+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.7 v2:172.21.15.181:6826/580738384 7 ==== osd_op_reply(55 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3001f670 2024-09-17T04:01:09.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.316+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:56 27.3b 27:dc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b280680b0 2024-09-17T04:01:09.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.317+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(56 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b24023040 con 0x7f2b280680b0 2024-09-17T04:01:09.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.317+0000 7f2b37a00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 27.7 27:e0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.317+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(57 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.317+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:58 27.27 27:e4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b24016040 2024-09-17T04:01:09.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.318+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 8 ==== osd_op_reply(58 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.318+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:59 27.17 27:e8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x5602972b92b0 2024-09-17T04:01:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.318+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(59 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x5602972b92b0 2024-09-17T04:01:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.318+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:60 27.37 27:ec000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b24016040 2024-09-17T04:01:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.319+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 9 ==== osd_op_reply(60 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.319+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:61 27.f 27:f0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b24016040 2024-09-17T04:01:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.319+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 <== osd.5 v2:172.21.15.181:6818/3094080342 10 ==== osd_op_reply(61 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b3000b040 con 0x7f2b24016040 2024-09-17T04:01:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.319+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:62 27.2f 27:f4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.319+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 12 ==== osd_op_reply(62 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.320+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:63 27.1f 27:f8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b2400ab30 2024-09-17T04:01:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.320+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(63 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b2400ab30 2024-09-17T04:01:09.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.320+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:64 27.3f 27:fc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f2b28073040 con 0x7f2b3006a040 2024-09-17T04:01:09.320 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000000 2024-09-17T04:01:09.321 INFO:teuthology.orchestra.run.smithi129.stdout:10000000000.00000001 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b38400640 1 -- 172.21.15.129:0/1279201948 <== osd.0 v2:172.21.15.129:6824/1655624368 13 ==== osd_op_reply(64 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f2b2801c030 con 0x7f2b3006a040 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f2b3006a040 msgr2=0x7f2b30055410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f2b3006a040 0x7f2b30055410 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f2b280680b0 msgr2=0x7f2b28060080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f2b280680b0 0x7f2b28060080 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5602972b92b0 msgr2=0x560297381600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x5602972b92b0 0x560297381600 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f2b30022cf0 msgr2=0x7f2b3006c460 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f2b30022cf0 0x7f2b3006c460 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f2b2400ab30 msgr2=0x7f2b24027940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 reap_dead start 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f2b2400ab30 0x7f2b24027940 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f2b24016040 msgr2=0x7f2b24017970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f2b24016040 0x7f2b24017970 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f2b28061610 msgr2=0x7f2b28083a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f2b28061610 0x7f2b28083a70 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f2b3001f670 msgr2=0x7f2b300698f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f2b3001f670 0x7f2b300698f0 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b38e00640 1 -- 172.21.15.129:0/1279201948 reap_dead start 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 msgr2=0x7f2b18079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 0x7f2b18079a80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x5602973926a0 tx=0x7f2b2400e040 comp rx=0 tx=0).stop 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 msgr2=0x56029738ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800a680 tx=0x7f2b2800ab40 comp rx=0 tx=0).stop 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 shutdown_connections 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b18077570 0x7f2b18079a80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 --2- 172.21.15.129:0/1279201948 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5602972fd5c0 0x56029738ceb0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 >> 172.21.15.129:0/1279201948 conn(0x5602971d8ac0 msgr2=0x560297383cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 shutdown_connections 2024-09-17T04:01:09.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09.321+0000 7f2b3f229f40 1 -- 172.21.15.129:0/1279201948 wait complete. 2024-09-17T04:01:09.336 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 found 2024-09-17T04:01:09.337 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi129') 2024-09-17T04:01:09.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:09 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-09-17T04:01:10.073 INFO:teuthology.orchestra.run.smithi129.stderr:open_background: keeping file open 2024-09-17T04:01:10.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:10.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 -- 172.21.15.129:0/3810752647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b2950 msgr2=0x7f81840b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 --2- 172.21.15.129:0/3810752647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b2950 0x7f81840b3480 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f818406dd20 tx=0x7f817002a560 comp rx=0 tx=0).stop 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 -- 172.21.15.129:0/3810752647 shutdown_connections 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 --2- 172.21.15.129:0/3810752647 >> v2:172.21.15.181:3300/0 conn(0x7f81840b39c0 0x7f818406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 --2- 172.21.15.129:0/3810752647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b2950 0x7f81840b3480 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 --2- 172.21.15.129:0/3810752647 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840ce170 0x7f81840cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.351+0000 7f8189400640 1 -- 172.21.15.129:0/3810752647 >> 172.21.15.129:0/3810752647 conn(0x7f818406d410 msgr2=0x7f81840b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:10.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.352+0000 7f8189400640 1 -- 172.21.15.129:0/3810752647 shutdown_connections 2024-09-17T04:01:10.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.352+0000 7f8189400640 1 -- 172.21.15.129:0/3810752647 wait complete. 2024-09-17T04:01:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.354+0000 7f8189400640 1 Processor -- start 2024-09-17T04:01:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.354+0000 7f8189400640 1 -- start start 2024-09-17T04:01:10.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840b2950 0x7f818414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:10.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:10.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 0x7f81841f5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8184070850 con 0x7f81840b39c0 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f81840706d0 con 0x7f81840ce170 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.355+0000 7f8189400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f818406f410 con 0x7f81840b2950 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45320/0 (socket says 172.21.15.129:45320) 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 -- 172.21.15.129:0/757575258 learned_addr learned my addr 172.21.15.129:0/757575258 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8182800640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840b2950 0x7f818414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8183200640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 0x7f81841f5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 -- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840b2950 msgr2=0x7f818414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840b2950 0x7f818414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 -- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 msgr2=0x7f81841f5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 0x7f81841f5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 -- 172.21.15.129:0/757575258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818409cf90 con 0x7f81840b39c0 2024-09-17T04:01:10.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8183200640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 0x7f81841f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:10.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.356+0000 7f8181e00640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f81700099c0 tx=0x7f81700039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:10.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.357+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817003e030 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.357+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f81700047a0 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.357+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818415c4c0 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.357+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8170035040 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.357+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818415ca80 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.358+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81840747b0 con 0x7f81840b39c0 2024-09-17T04:01:10.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.360+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8170041030 con 0x7f81840b39c0 2024-09-17T04:01:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.361+0000 7f817aa00640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 0x7f816407b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:10.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.361+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f81700bb4d0 con 0x7f81840b39c0 2024-09-17T04:01:10.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.361+0000 7f8182800640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 0x7f816407b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:10.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.362+0000 7f8182800640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 0x7f816407b0f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8174005fd0 tx=0x7f8174005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:10.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.365+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f81840747b0 con 0x7f81840b39c0 2024-09-17T04:01:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.583+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f818409d200 con 0x7f81840b39c0 2024-09-17T04:01:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.584+0000 7f817aa00640 1 -- 172.21.15.129:0/757575258 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f818409cf90 con 0x7f81840b39c0 2024-09-17T04:01:10.584 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:10.585 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 msgr2=0x7f816407b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 0x7f816407b0f0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8174005fd0 tx=0x7f8174005850 comp rx=0 tx=0).stop 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 msgr2=0x7f818409ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f81700099c0 tx=0x7f81700039e0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 shutdown_connections 2024-09-17T04:01:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8164078be0 0x7f816407b0f0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81840ce170 0x7f81841f5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81840b39c0 0x7f818409ca50 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 --2- 172.21.15.129:0/757575258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81840b2950 0x7f818414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 >> 172.21.15.129:0/757575258 conn(0x7f818406d410 msgr2=0x7f81840b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 shutdown_connections 2024-09-17T04:01:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:10.587+0000 7f8189400640 1 -- 172.21.15.129:0/757575258 wait complete. 2024-09-17T04:01:10.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-17T04:01:10.853 INFO:teuthology.orchestra.run.smithi129.stdout:[{"id":9663,"entity":{"name":{"type":"client","num":9663},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":3021456163}},"state":"open","num_leases":1,"num_caps":2,"request_load_avg":47,"uptime":7.3778409109999998,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.9103527111402103,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":52,"last_trim_completed_flushes_tid":5,"delegated_inos":[],"inst":"client.9663 v1:192.168.0.1:0/3021456163","completed_requests":[{"tid":52,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000037fbff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000080ffff"}},"entity_id":"0","hostname":"smithi129","kernel_version":"6.11.0-rc5-g2ca44a2f35ca","root":"/"}}] 2024-09-17T04:01:10.854 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x000000000037fbff" }, "entity_id": "0", "hostname": "smithi129", "kernel_version": "6.11.0-rc5-g2ca44a2f35ca", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000080ffff" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 52 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 3021456163, "type": "v1" }, "name": { "num": 9663, "type": "client" } }, "id": 9663, "inst": "client.9663 v1:192.168.0.1:0/3021456163", "last_trim_completed_flushes_tid": 5, "last_trim_completed_requests_tid": 52, "num_caps": 2, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0 }, "request_load_avg": 47, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.91035271114021 }, "state": "open", "uptime": 7.377840911 } ] 2024-09-17T04:01:10.854 DEBUG:tasks.ceph_test_case:wait_until_equal: 2 != 1, waiting (timeout=60)... 2024-09-17T04:01:15.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:16.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 -- 172.21.15.129:0/556988471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 msgr2=0x7f8268160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 --2- 172.21.15.129:0/556988471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268160cf0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f8258005fd0 tx=0x7f825802a540 comp rx=0 tx=0).stop 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 -- 172.21.15.129:0/556988471 shutdown_connections 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 --2- 172.21.15.129:0/556988471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268160cf0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 -- 172.21.15.129:0/556988471 >> 172.21.15.129:0/556988471 conn(0x7f826806cfc0 msgr2=0x7f826806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 -- 172.21.15.129:0/556988471 shutdown_connections 2024-09-17T04:01:16.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.088+0000 7f826e600640 1 -- 172.21.15.129:0/556988471 wait complete. 2024-09-17T04:01:16.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.090+0000 7f826e600640 1 Processor -- start 2024-09-17T04:01:16.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.090+0000 7f826e600640 1 -- start start 2024-09-17T04:01:16.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82680b46e0 0x7f82680aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f826815b620 0x7f826815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f826815d460 con 0x7f82680cfb60 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f826815d2e0 con 0x7f826815b620 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f826815d5e0 con 0x7f82680b46e0 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57912/0 (socket says 172.21.15.129:57912) 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 -- 172.21.15.129:0/3291707392 learned_addr learned my addr 172.21.15.129:0/3291707392 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826d200640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82680b46e0 0x7f82680aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f826dc00640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f826815b620 0x7f826815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 -- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82680b46e0 msgr2=0x7f82680aaee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82680b46e0 0x7f82680aaee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 -- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f826815b620 msgr2=0x7f826815ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f826815b620 0x7f826815ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.092+0000 7f825fe00640 1 -- 172.21.15.129:0/3291707392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826815bf70 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f825fe00640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f825000b420 tx=0x7f825000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250011830 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8250016070 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250011d60 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8268072d30 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.093+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8268073480 con 0x7f82680cfb60 2024-09-17T04:01:16.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.094+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8268160cf0 con 0x7f82680cfb60 2024-09-17T04:01:16.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.095+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f825001b460 con 0x7f82680cfb60 2024-09-17T04:01:16.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.096+0000 7f8265600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 0x7f824407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.096+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f825009a870 con 0x7f82680cfb60 2024-09-17T04:01:16.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.097+0000 7f826d200640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 0x7f824407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.097+0000 7f826d200640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 0x7f824407b0a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8258033000 tx=0x7f8258004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:16.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.101+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8250020020 con 0x7f82680cfb60 2024-09-17T04:01:16.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.320+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8268072f00 con 0x7f82680cfb60 2024-09-17T04:01:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.321+0000 7f8265600640 1 -- 172.21.15.129:0/3291707392 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f826815bf70 con 0x7f82680cfb60 2024-09-17T04:01:16.320 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:16.320 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:16.320 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 msgr2=0x7f824407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 0x7f824407b0a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8258033000 tx=0x7f8258004730 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 msgr2=0x7f8268149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f825000b420 tx=0x7f825000b8e0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 shutdown_connections 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8244078b90 0x7f824407b0a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f826815b620 0x7f826815ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82680cfb60 0x7f8268149c40 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 --2- 172.21.15.129:0/3291707392 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82680b46e0 0x7f82680aaee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.323+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 >> 172.21.15.129:0/3291707392 conn(0x7f826806cfc0 msgr2=0x7f82680cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.324+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 shutdown_connections 2024-09-17T04:01:16.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.324+0000 7f826e600640 1 -- 172.21.15.129:0/3291707392 wait complete. 2024-09-17T04:01:16.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json session ls 2024-09-17T04:01:16.593 INFO:teuthology.orchestra.run.smithi129.stdout:[{"id":9663,"entity":{"name":{"type":"client","num":9663},"addr":{"type":"v1","addr":"192.168.0.1:0","nonce":3021456163}},"state":"open","num_leases":1,"num_caps":1,"request_load_avg":44,"uptime":13.113717230000001,"requests_in_flight":0,"num_completed_requests":1,"num_completed_flushes":1,"reconnecting":false,"recall_caps":{"value":0,"halflife":60},"release_caps":{"value":0.9849919258785782,"halflife":60},"recall_caps_throttle":{"value":0,"halflife":1.5},"recall_caps_throttle2o":{"value":0,"halflife":0.5},"session_cache_liveness":{"value":5.8325413921816169,"halflife":300},"cap_acquisition":{"value":0,"halflife":30},"last_trim_completed_requests_tid":52,"last_trim_completed_flushes_tid":5,"delegated_inos":[],"inst":"client.9663 v1:192.168.0.1:0/3021456163","completed_requests":[{"tid":52,"created_ino":"0x0"}],"prealloc_inos":[{"start":"0x10000000001","length":1000}],"client_metadata":{"client_features":{"feature_bits":"0x000000000037fbff"},"metric_spec":{"metric_flags":{"feature_bits":"0x000000000080ffff"}},"entity_id":"0","hostname":"smithi129","kernel_version":"6.11.0-rc5-g2ca44a2f35ca","root":"/"}}] 2024-09-17T04:01:16.594 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "cap_acquisition": { "halflife": 30, "value": 0 }, "client_metadata": { "client_features": { "feature_bits": "0x000000000037fbff" }, "entity_id": "0", "hostname": "smithi129", "kernel_version": "6.11.0-rc5-g2ca44a2f35ca", "metric_spec": { "metric_flags": { "feature_bits": "0x000000000080ffff" } }, "root": "/" }, "completed_requests": [ { "created_ino": "0x0", "tid": 52 } ], "delegated_inos": [], "entity": { "addr": { "addr": "192.168.0.1:0", "nonce": 3021456163, "type": "v1" }, "name": { "num": 9663, "type": "client" } }, "id": 9663, "inst": "client.9663 v1:192.168.0.1:0/3021456163", "last_trim_completed_flushes_tid": 5, "last_trim_completed_requests_tid": 52, "num_caps": 1, "num_completed_flushes": 1, "num_completed_requests": 1, "num_leases": 1, "prealloc_inos": [ { "length": 1000, "start": "0x10000000001" } ], "recall_caps": { "halflife": 60, "value": 0 }, "recall_caps_throttle": { "halflife": 1.5, "value": 0 }, "recall_caps_throttle2o": { "halflife": 0.5, "value": 0 }, "reconnecting": false, "release_caps": { "halflife": 60, "value": 0.9849919258785782 }, "request_load_avg": 44, "requests_in_flight": 0, "session_cache_liveness": { "halflife": 300, "value": 5.832541392181617 }, "state": "open", "uptime": 13.11371723 } ] 2024-09-17T04:01:16.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 -- 172.21.15.129:0/3861747474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406eed0 msgr2=0x7f83540ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 --2- 172.21.15.129:0/3861747474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406eed0 0x7f83540ccc80 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f8354072e00 tx=0x7f834002a560 comp rx=0 tx=0).stop 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 -- 172.21.15.129:0/3861747474 shutdown_connections 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 --2- 172.21.15.129:0/3861747474 >> v2:172.21.15.181:3300/0 conn(0x7f835406f2e0 0x7f83540cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 --2- 172.21.15.129:0/3861747474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406eed0 0x7f83540ccc80 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 --2- 172.21.15.129:0/3861747474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f83540b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 -- 172.21.15.129:0/3861747474 >> 172.21.15.129:0/3861747474 conn(0x7f835406d410 msgr2=0x7f83540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 -- 172.21.15.129:0/3861747474 shutdown_connections 2024-09-17T04:01:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.815+0000 7f835c600640 1 -- 172.21.15.129:0/3861747474 wait complete. 2024-09-17T04:01:16.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.817+0000 7f835c600640 1 Processor -- start 2024-09-17T04:01:16.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.818+0000 7f835c600640 1 -- start start 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835406eed0 0x7f83540c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f835415d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8354160b40 con 0x7f835406f2e0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f83541609c0 con 0x7f835406eed0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8354160cc0 con 0x7f83540705a0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f8359200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f8359200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57934/0 (socket says 172.21.15.129:57934) 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f8359200640 1 -- 172.21.15.129:0/2754755794 learned_addr learned my addr 172.21.15.129:0/2754755794 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f835a600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f835415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.819+0000 7f8359c00640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835406eed0 0x7f83540c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 -- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 msgr2=0x7f835415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f835415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 -- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835406eed0 msgr2=0x7f83540c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835406eed0 0x7f83540c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 -- 172.21.15.129:0/2754755794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835415cd90 con 0x7f835406f2e0 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f835a600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f835415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8359200640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8340000c00 tx=0x7f8340003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:16.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f834003e030 con 0x7f835406f2e0 2024-09-17T04:01:16.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835415e310 con 0x7f835406f2e0 2024-09-17T04:01:16.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8340003e00 con 0x7f835406f2e0 2024-09-17T04:01:16.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340035040 con 0x7f835406f2e0 2024-09-17T04:01:16.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.820+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835415abe0 con 0x7f835406f2e0 2024-09-17T04:01:16.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.822+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8354161220 con 0x7f835406f2e0 2024-09-17T04:01:16.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.823+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8340041030 con 0x7f835406f2e0 2024-09-17T04:01:16.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.824+0000 7f8351600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 0x7f833407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:16.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.824+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f83400bb5b0 con 0x7f835406f2e0 2024-09-17T04:01:16.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.825+0000 7f8359c00640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 0x7f833407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:16.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.826+0000 7f8359c00640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 0x7f833407b0a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8348005fd0 tx=0x7f83480095e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:16.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:16.829+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8340083310 con 0x7f835406f2e0 2024-09-17T04:01:17.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.046+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f835406a140 con 0x7f835406f2e0 2024-09-17T04:01:17.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.047+0000 7f8351600640 1 -- 172.21.15.129:0/2754755794 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f8340087d40 con 0x7f835406f2e0 2024-09-17T04:01:17.046 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:17.046 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:17.046 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 msgr2=0x7f833407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 0x7f833407b0a0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8348005fd0 tx=0x7f83480095e0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 msgr2=0x7f83540b10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8340000c00 tx=0x7f8340003910 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 shutdown_connections 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8334078b90 0x7f833407b0a0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f83540705a0 0x7f835415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f835406f2e0 0x7f83540b10b0 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 --2- 172.21.15.129:0/2754755794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f835406eed0 0x7f83540c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 >> 172.21.15.129:0/2754755794 conn(0x7f835406d410 msgr2=0x7f83540ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:17.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 shutdown_connections 2024-09-17T04:01:17.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.049+0000 7f835c600640 1 -- 172.21.15.129:0/2754755794 wait complete. 2024-09-17T04:01:17.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T04:01:17.316 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":1}} 2024-09-17T04:01:17.316 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 1 } } 2024-09-17T04:01:17.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:17.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 -- 172.21.15.129:0/4020241120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80ce170 msgr2=0x7f7df80cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 --2- 172.21.15.129:0/4020241120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80ce170 0x7f7df80cbbf0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f7df8072360 tx=0x7f7dec02aa30 comp rx=0 tx=0).stop 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 -- 172.21.15.129:0/4020241120 shutdown_connections 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 --2- 172.21.15.129:0/4020241120 >> v2:172.21.15.181:3300/0 conn(0x7f7df80b2950 0x7f7df80b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 --2- 172.21.15.129:0/4020241120 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80ce170 0x7f7df80cbbf0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 -- 172.21.15.129:0/4020241120 >> 172.21.15.129:0/4020241120 conn(0x7f7df806d410 msgr2=0x7f7df80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 -- 172.21.15.129:0/4020241120 shutdown_connections 2024-09-17T04:01:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.537+0000 7f7e00600640 1 -- 172.21.15.129:0/4020241120 wait complete. 2024-09-17T04:01:17.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.539+0000 7f7e00600640 1 Processor -- start 2024-09-17T04:01:17.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.540+0000 7f7e00600640 1 -- start start 2024-09-17T04:01:17.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:17.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7df80b39c0 0x7f7df815e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:17.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 0x7f7df81f9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7df8070850 con 0x7f7df80b2950 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7df80706d0 con 0x7f7df80ce170 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7e00600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7df806f5e0 con 0x7f7df80b39c0 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7dfdc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7dfdc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57942/0 (socket says 172.21.15.129:57942) 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7dfdc00640 1 -- 172.21.15.129:0/1100596700 learned_addr learned my addr 172.21.15.129:0/1100596700 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7dfd200640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7df80b39c0 0x7f7df815e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.541+0000 7f7dfe600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 0x7f7df81f9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 -- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7df80b39c0 msgr2=0x7f7df815e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7df80b39c0 0x7f7df815e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 -- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 msgr2=0x7f7df81f9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 0x7f7df81f9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 -- 172.21.15.129:0/1100596700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df815c220 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfe600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 0x7f7df81f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7dfdc00640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009d60 tx=0x7f7dec003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec033050 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7dec0044f0 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df80affa0 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.542+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec038450 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.543+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df80c87d0 con 0x7f7df80b2950 2024-09-17T04:01:17.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.544+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df806fab0 con 0x7f7df80b2950 2024-09-17T04:01:17.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.545+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7dec036070 con 0x7f7df80b2950 2024-09-17T04:01:17.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.546+0000 7f7df5600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 0x7f7dd807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:17.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.547+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f7dec0bc290 con 0x7f7df80b2950 2024-09-17T04:01:17.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.547+0000 7f7dfd200640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 0x7f7dd807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:17.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.547+0000 7f7dfd200640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 0x7f7dd807b0d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7df80b22b0 tx=0x7f7de400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:17.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.551+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7df80b3dd0 con 0x7f7df80b2950 2024-09-17T04:01:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.771+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7df806a140 con 0x7f7df80b2950 2024-09-17T04:01:17.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.772+0000 7f7df5600640 1 -- 172.21.15.129:0/1100596700 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f7dec0846b0 con 0x7f7df80b2950 2024-09-17T04:01:17.771 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:17.771 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:17.771 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 msgr2=0x7f7dd807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 0x7f7dd807b0d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f7df80b22b0 tx=0x7f7de400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 msgr2=0x7f7df8099b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:17.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009d60 tx=0x7f7dec003f10 comp rx=0 tx=0).stop 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 shutdown_connections 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7dd8078bc0 0x7f7dd807b0d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7df80ce170 0x7f7df81f9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7df80b39c0 0x7f7df815e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 --2- 172.21.15.129:0/1100596700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7df80b2950 0x7f7df8099b40 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 >> 172.21.15.129:0/1100596700 conn(0x7f7df806d410 msgr2=0x7f7df80d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 shutdown_connections 2024-09-17T04:01:17.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:17.774+0000 7f7e00600640 1 -- 172.21.15.129:0/1100596700 wait complete. 2024-09-17T04:01:17.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:01:17.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:01:17.953 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:01:18.040 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T04:01:18.041 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T04:01:18.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 -- 172.21.15.129:0/2463237429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33406eed0 msgr2=0x7fb3340ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 --2- 172.21.15.129:0/2463237429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33406eed0 0x7fb3340ccc80 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fb3340d1370 tx=0x7fb31c02a560 comp rx=0 tx=0).stop 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 -- 172.21.15.129:0/2463237429 shutdown_connections 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 --2- 172.21.15.129:0/2463237429 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb3340cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 --2- 172.21.15.129:0/2463237429 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb33406eed0 0x7fb3340ccc80 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 --2- 172.21.15.129:0/2463237429 >> v2:172.21.15.181:3300/0 conn(0x7fb3340705a0 0x7fb3340b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 -- 172.21.15.129:0/2463237429 >> 172.21.15.129:0/2463237429 conn(0x7fb33406d410 msgr2=0x7fb3340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 -- 172.21.15.129:0/2463237429 shutdown_connections 2024-09-17T04:01:18.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.302+0000 7fb33b400640 1 -- 172.21.15.129:0/2463237429 wait complete. 2024-09-17T04:01:18.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.305+0000 7fb33b400640 1 Processor -- start 2024-09-17T04:01:18.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.305+0000 7fb33b400640 1 -- start start 2024-09-17T04:01:18.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 0x7fb3341f9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb33414ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb334160b40 con 0x7fb3340705a0 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb3341609c0 con 0x7fb33406eed0 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb33b400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb334160cc0 con 0x7fb33406f2e0 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.306+0000 7fb339400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57954/0 (socket says 172.21.15.129:57954) 2024-09-17T04:01:18.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 -- 172.21.15.129:0/1459951736 learned_addr learned my addr 172.21.15.129:0/1459951736 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb338a00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 0x7fb3341f9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb32be00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb33414ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 -- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 msgr2=0x7fb33414ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb33414ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 -- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 msgr2=0x7fb3341f9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 0x7fb3341f9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 -- 172.21.15.129:0/1459951736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb334150ca0 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb338a00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 0x7fb3341f9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb32be00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb33414ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb339400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00ca80 tx=0x7fb32c00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c007d30 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb32c00f450 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00f880 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.307+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33414d5d0 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.308+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3341f9700 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.309+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb334161250 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.310+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb32c00fa20 con 0x7fb3340705a0 2024-09-17T04:01:18.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.311+0000 7fb331600640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 0x7fb31007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.312+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fb32c09b700 con 0x7fb3340705a0 2024-09-17T04:01:18.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.312+0000 7fb338a00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 0x7fb31007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.313+0000 7fb338a00640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 0x7fb31007b0a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb32400ba60 tx=0x7fb324005cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.316+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb32c014070 con 0x7fb3340705a0 2024-09-17T04:01:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.534+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb33406a170 con 0x7fb3340705a0 2024-09-17T04:01:18.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.534+0000 7fb331600640 1 -- 172.21.15.129:0/1459951736 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fb32c063b30 con 0x7fb3340705a0 2024-09-17T04:01:18.534 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:18.534 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:18.534 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 msgr2=0x7fb31007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 0x7fb31007b0a0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb32400ba60 tx=0x7fb324005cf0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 msgr2=0x7fb3340c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00ca80 tx=0x7fb32c00cf40 comp rx=0 tx=0).stop 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 shutdown_connections 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb310078b90 0x7fb31007b0a0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3340705a0 0x7fb3340c6720 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb33406f2e0 0x7fb33414ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 --2- 172.21.15.129:0/1459951736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb33406eed0 0x7fb3341f9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 >> 172.21.15.129:0/1459951736 conn(0x7fb33406d410 msgr2=0x7fb3340b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:18.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 shutdown_connections 2024-09-17T04:01:18.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.537+0000 7fb33b400640 1 -- 172.21.15.129:0/1459951736 wait complete. 2024-09-17T04:01:18.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-09-17T04:01:18.801 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":1}} 2024-09-17T04:01:18.801 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 1 } } 2024-09-17T04:01:18.802 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T04:01:18.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.942+0000 7f0d84f90f40 1 Processor -- start 2024-09-17T04:01:18.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.942+0000 7f0d84f90f40 1 -- start start 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5cf4a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5cf5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556da5ae1c60 con 0x556da5bdd5c0 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556da5b401d0 con 0x556da5bd75c0 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.943+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x556da5bdcf20 con 0x556da5c5fb70 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57958/0 (socket says 172.21.15.129:57958) 2024-09-17T04:01:18.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 -- 172.21.15.129:0/3592307880 learned_addr learned my addr 172.21.15.129:0/3592307880 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d7e200640 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5cf4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5cf5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 -- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 msgr2=0x556da5cf5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5cf5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 -- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 msgr2=0x556da5cf4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5cf4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.944+0000 7f0d77e00640 1 -- 172.21.15.129:0/3592307880 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da5bdcd00 con 0x556da5bdd5c0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d77e00640 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f0d68046430 tx=0x7f0d68002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd74d4d02caf575c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d7ce00640 1 -- 172.21.15.129:0/3592307880 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68066070 con 0x556da5bdd5c0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d7ce00640 1 -- 172.21.15.129:0/3592307880 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d6804eba0 con 0x556da5bdd5c0 2024-09-17T04:01:18.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d7ce00640 1 -- 172.21.15.129:0/3592307880 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68056470 con 0x556da5bdd5c0 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 -- 172.21.15.129:0/3592307880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 msgr2=0x556da5cf4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f0d68046430 tx=0x7f0d68002fb0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 -- 172.21.15.129:0/3592307880 shutdown_connections 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5cf5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5cf4f50 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/3592307880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5cf4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.945+0000 7f0d84f90f40 1 -- 172.21.15.129:0/3592307880 >> 172.21.15.129:0/3592307880 conn(0x556da5ab2ac0 msgr2=0x556da5cf6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.946+0000 7f0d84f90f40 1 -- 172.21.15.129:0/3592307880 shutdown_connections 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.946+0000 7f0d84f90f40 1 -- 172.21.15.129:0/3592307880 wait complete. 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.947+0000 7f0d84f90f40 1 Processor -- start 2024-09-17T04:01:18.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.947+0000 7f0d84f90f40 1 -- start start 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5c520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5c56e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x556da5c6c710 con 0x556da5bdd5c0 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x556da5bdca20 con 0x556da5bd75c0 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d84f90f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x556da5cf3cd0 con 0x556da5c5fb70 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57966/0 (socket says 172.21.15.129:57966) 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.948+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 learned_addr learned my addr 172.21.15.129:0/1722109034 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5c520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5c56e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 msgr2=0x556da5c56e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5c56e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 msgr2=0x556da5c520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5c520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556da5cf5b30 con 0x556da5bdd5c0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556da5bd75c0 0x556da5c520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x556da5c5fb70 0x556da5c56e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f0d68004e90 tx=0x7f0d68004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d76a00640 1 -- 172.21.15.129:0/1722109034 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68066040 con 0x556da5bdd5c0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d76a00640 1 -- 172.21.15.129:0/1722109034 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d6804e330 con 0x556da5bdd5c0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d76a00640 1 -- 172.21.15.129:0/1722109034 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6804d590 con 0x556da5bdd5c0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556da5b92e90 con 0x556da5bdd5c0 2024-09-17T04:01:18.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.949+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556da5b93690 con 0x556da5bdd5c0 2024-09-17T04:01:18.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.951+0000 7f0d76a00640 1 -- 172.21.15.129:0/1722109034 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d6804d750 con 0x556da5bdd5c0 2024-09-17T04:01:18.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.952+0000 7f0d76a00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 0x7f0d5c079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.952+0000 7f0d76a00640 1 -- 172.21.15.129:0/1722109034 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f0d680de690 con 0x556da5bdd5c0 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.953+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x556da5c56440 0x556da5cf2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.953+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:1 27.0 27:00000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x556da5c56440 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.953+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 0x7f0d5c079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.953+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x556da5c56440 0x556da5cf2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 0x7f0d5c079ab0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000f540 tx=0x7f0d70002e80 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x556da5c56440 0x556da5cf2cd0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(1 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f0d78011ad0 0x7f0d7800d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:2 27.20 27:04000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.954+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f0d78011ad0 0x7f0d7800d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f0d78011ad0 0x7f0d7800d390 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(2 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:3 27.10 27:08000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x556da5c56440 2024-09-17T04:01:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(3 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0d7800e3c0 0x7f0d7800e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.955+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:4 27.30 27:0c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7800e3c0 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.956+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0d7800e3c0 0x7f0d7800e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.956+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0d7800e3c0 0x7f0d7800e7d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.956+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(4 [pgnls start_epoch 174 out=49b] v174'4 uv4 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.956+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0d68022cf0 0x7f0d680a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.957+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:5 27.8 27:10000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.957+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0d68022cf0 0x7f0d680a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.958+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0d68022cf0 0x7f0d680a52e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.958+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(5 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.958+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0d780213b0 0x7f0d780217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.958+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:6 27.28 27:14000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d780213b0 2024-09-17T04:01:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.959+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0d780213b0 0x7f0d780217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.960+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0d780213b0 0x7f0d780217c0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.960+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(6 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.960+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0d70019380 0x7f0d70019790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.960+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:7 27.18 27:18000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d70019380 2024-09-17T04:01:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.961+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0d70019380 0x7f0d70019790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.961+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0d70019380 0x7f0d70019790 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.962+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(7 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.962+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:8 27.38 27:1c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d70019380 2024-09-17T04:01:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.962+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(8 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.962+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:9 27.4 27:20000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.963+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(9 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.963+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:10 27.24 27:24000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x556da5c56440 2024-09-17T04:01:18.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.963+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(10 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.963+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:11 27.14 27:28000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d780213b0 2024-09-17T04:01:18.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.964+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(11 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.964+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0d7001b8b0 0x7f0d7000d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.964+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:12 27.34 27:2c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7001b8b0 2024-09-17T04:01:18.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.964+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0d7001b8b0 0x7f0d7000d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.965+0000 7f0d7ec00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0d7001b8b0 0x7f0d7000d390 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.965+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(12 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.965+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:13 27.c 27:30000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x556da5c56440 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.966+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(13 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.966+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:14 27.2c 27:34000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d70019380 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.966+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(14 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.966+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:15 27.1c 27:38000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.967+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(15 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.967+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:16 27.3c 27:3c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d780213b0 2024-09-17T04:01:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.967+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(16 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.967+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:17 27.2 27:40000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.968+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(17 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.968+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:18 27.22 27:44000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.968+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(18 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.968+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:19 27.12 27:48000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d70019380 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(19 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:20 27.32 27:4c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(20 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:21 27.a 27:50000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7800e3c0 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(21 [pgnls start_epoch 174 out=49b] v174'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d77e00640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0d6801f670 0x7f0d680de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:18.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.969+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:22 27.2a 27:54000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d6801f670 2024-09-17T04:01:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.970+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0d6801f670 0x7f0d680de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.971+0000 7f0d7e200640 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0d6801f670 0x7f0d680de0b0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.971+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(22 [pgnls start_epoch 174 out=49b] v174'2 uv2 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d6801f670 2024-09-17T04:01:18.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.971+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:23 27.1a 27:58000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(23 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:24 27.3a 27:5c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d70019380 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(24 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:25 27.6 27:60000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7001b8b0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(25 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.972+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:26 27.26 27:64000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.973+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(26 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.973+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:27 27.16 27:68000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d780213b0 2024-09-17T04:01:18.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.973+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(27 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.973+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:28 27.36 27:6c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7800e3c0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.974+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(28 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.974+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:29 27.e 27:70000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.974+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(29 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.974+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:30 27.2e 27:74000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7800e3c0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.975+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(30 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.975+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:31 27.1e 27:78000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.975+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(31 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.975+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:32 27.3e 27:7c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(32 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:33 27.1 27:80000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d68022cf0 2024-09-17T04:01:18.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(33 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:34 27.21 27:84000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7800e3c0 2024-09-17T04:01:18.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(34 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.976+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:35 27.11 27:88000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.977+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(35 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.977+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:36 27.31 27:8c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.977+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(36 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.977+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:37 27.9 27:90000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d6801f670 2024-09-17T04:01:18.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.978+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(37 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d6801f670 2024-09-17T04:01:18.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.978+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:38 27.29 27:94000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d6801f670 2024-09-17T04:01:18.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.978+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(38 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d6801f670 2024-09-17T04:01:18.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.978+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:39 27.19 27:98000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d780213b0 2024-09-17T04:01:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.979+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 5 ==== osd_op_reply(39 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.979+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:40 27.39 27:9c000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.979+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(40 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.979+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:41 27.5 27:a0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d6801f670 2024-09-17T04:01:18.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(41 [pgnls start_epoch 174 out=49b] v174'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d6801f670 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:42 27.25 27:a4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d7001b8b0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(42 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:43 27.15 27:a8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x7f0d78011ad0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 10 ==== osd_op_reply(43 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.980+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:44 27.35 27:ac000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x556da5cf3210 con 0x556da5c56440 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(44 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:45 27.d 27:b0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7800e3c0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(45 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:46 27.2d 27:b4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7800e3c0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 7 ==== osd_op_reply(46 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.981+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:47 27.1d 27:b8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7800e3c0 2024-09-17T04:01:18.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.982+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 8 ==== osd_op_reply(47 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.982+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:48 27.3d 27:bc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x556da5c56440 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.982+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 6 ==== osd_op_reply(48 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.982+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:49 27.3 27:c0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.983+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(49 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.983+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:50 27.23 27:c4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.983+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(50 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.983+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:51 27.13 27:c8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7800e3c0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.983+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.3 v2:172.21.15.129:6808/2225231477 9 ==== osd_op_reply(51 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d7800e3c0 2024-09-17T04:01:18.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.984+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:52 27.33 27:cc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.984+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(52 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d7001b8b0 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.984+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:53 27.b 27:d0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x556da5c56440 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.985+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 7 ==== osd_op_reply(53 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.985+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:54 27.2b 27:d4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d780213b0 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.985+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 6 ==== osd_op_reply(54 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.985+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:55 27.1b 27:d8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d780213b0 2024-09-17T04:01:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.986+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.7 v2:172.21.15.181:6826/580738384 7 ==== osd_op_reply(55 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d780213b0 2024-09-17T04:01:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.986+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:56 27.3b 27:dc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d70019380 2024-09-17T04:01:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.986+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(56 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d68071020 con 0x7f0d70019380 2024-09-17T04:01:18.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.986+0000 7f0d77e00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:57 27.7 27:e0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d78011ad0 2024-09-17T04:01:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 11 ==== osd_op_reply(57 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:58 27.27 27:e4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d68022cf0 2024-09-17T04:01:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 8 ==== osd_op_reply(58 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:59 27.17 27:e8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x556da5c56440 2024-09-17T04:01:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.2 v2:172.21.15.129:6809/283507149 8 ==== osd_op_reply(59 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x556da5c56440 2024-09-17T04:01:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.987+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:60 27.37 27:ec000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d68022cf0 2024-09-17T04:01:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.988+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 9 ==== osd_op_reply(60 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.988+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:61 27.f 27:f0000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d68022cf0 2024-09-17T04:01:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.989+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 <== osd.5 v2:172.21.15.181:6818/3094080342 10 ==== osd_op_reply(61 [pgnls start_epoch 174 out=49b] v174'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d78003040 con 0x7f0d68022cf0 2024-09-17T04:01:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.989+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:62 27.2f 27:f4000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d78011ad0 2024-09-17T04:01:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.989+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 12 ==== osd_op_reply(62 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.989+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:63 27.1f 27:f8000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d6801f670 2024-09-17T04:01:18.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(63 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d6801f670 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:64 27.3f 27:fc000000::::head [pgnls start_epoch 174 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e174) -- 0x7f0d7803d040 con 0x7f0d78011ad0 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d7e200640 1 -- 172.21.15.129:0/1722109034 <== osd.0 v2:172.21.15.129:6824/1655624368 13 ==== osd_op_reply(64 [pgnls start_epoch 174 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7f0d700105b0 con 0x7f0d78011ad0 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f0d78011ad0 msgr2=0x7f0d7800d390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f0d78011ad0 0x7f0d7800d390 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0d70019380 msgr2=0x7f0d70019790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.990+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7f0d70019380 0x7f0d70019790 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x556da5c56440 msgr2=0x556da5cf2cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x556da5c56440 0x556da5cf2cd0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0d7800e3c0 msgr2=0x7f0d7800e7d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f0d7800e3c0 0x7f0d7800e7d0 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0d6801f670 msgr2=0x7f0d680de0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7f0d6801f670 0x7f0d680de0b0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 reap_dead start 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0d68022cf0 msgr2=0x7f0d680a52e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7f0d68022cf0 0x7f0d680a52e0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0d7001b8b0 msgr2=0x7f0d7000d390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7f0d7001b8b0 0x7f0d7000d390 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0d780213b0 msgr2=0x7f0d780217c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f0d780213b0 0x7f0d780217c0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d7ec00640 1 -- 172.21.15.129:0/1722109034 reap_dead start 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 msgr2=0x7f0d5c079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 0x7f0d5c079ab0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000f540 tx=0x7f0d70002e80 comp rx=0 tx=0).stop 2024-09-17T04:01:18.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 msgr2=0x556da5c56920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f0d68004e90 tx=0x7f0d68004580 comp rx=0 tx=0).stop 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 shutdown_connections 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d5c0775a0 0x7f0d5c079ab0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 --2- 172.21.15.129:0/1722109034 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x556da5bdd5c0 0x556da5c56920 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 >> 172.21.15.129:0/1722109034 conn(0x556da5ab2ac0 msgr2=0x556da5cf6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 shutdown_connections 2024-09-17T04:01:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:18.991+0000 7f0d84f90f40 1 -- 172.21.15.129:0/1722109034 wait complete. 2024-09-17T04:01:19.001 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627776 size 8388608 are absent 2024-09-17T04:01:19.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T04:01:19.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 -- 172.21.15.129:0/316810681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 msgr2=0x7f4ab80cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:19.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 --2- 172.21.15.129:0/316810681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80cadb0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8005fd0 tx=0x7f4aa802a940 comp rx=0 tx=0).stop 2024-09-17T04:01:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 -- 172.21.15.129:0/316810681 shutdown_connections 2024-09-17T04:01:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 --2- 172.21.15.129:0/316810681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80cadb0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 -- 172.21.15.129:0/316810681 >> 172.21.15.129:0/316810681 conn(0x7f4ab806ee70 msgr2=0x7f4ab806f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 -- 172.21.15.129:0/316810681 shutdown_connections 2024-09-17T04:01:19.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.220+0000 7f4abd800640 1 -- 172.21.15.129:0/316810681 wait complete. 2024-09-17T04:01:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.222+0000 7f4abd800640 1 Processor -- start 2024-09-17T04:01:19.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.223+0000 7f4abd800640 1 -- start start 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ab80cb2f0 0x7f4ab80b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ab80cd260 0x7f4ab80c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab8073ab0 con 0x7f4ab815c510 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab8073930 con 0x7f4ab80cb2f0 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abd800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab8073c30 con 0x7f4ab80cd260 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.224+0000 7f4abce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57988/0 (socket says 172.21.15.129:57988) 2024-09-17T04:01:19.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 -- 172.21.15.129:0/1866085353 learned_addr learned my addr 172.21.15.129:0/1866085353 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4ab7e00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ab80cb2f0 0x7f4ab80b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4ab7400640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ab80cd260 0x7f4ab80c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 -- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ab80cd260 msgr2=0x7f4ab80c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ab80cd260 0x7f4ab80c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 -- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ab80cb2f0 msgr2=0x7f4ab80b24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ab80cb2f0 0x7f4ab80b24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 -- 172.21.15.129:0/1866085353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab80b6100 con 0x7f4ab815c510 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4abce00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000c3f0 tx=0x7f4ab000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0017070 con 0x7f4ab815c510 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.225+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ab00053c0 con 0x7f4ab815c510 2024-09-17T04:01:19.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.226+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab80b0470 con 0x7f4ab815c510 2024-09-17T04:01:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.226+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0003850 con 0x7f4ab815c510 2024-09-17T04:01:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.226+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab80b3280 con 0x7f4ab815c510 2024-09-17T04:01:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.227+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab80b5e70 con 0x7f4ab815c510 2024-09-17T04:01:19.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.228+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4ab001a070 con 0x7f4ab815c510 2024-09-17T04:01:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.229+0000 7f4aafe00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 0x7f4a9807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.229+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f4ab0099ff0 con 0x7f4ab815c510 2024-09-17T04:01:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.230+0000 7f4ab7e00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 0x7f4a9807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:19.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.230+0000 7f4ab7e00640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 0x7f4a9807b0a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8034000 tx=0x7f4aa8004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:19.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.234+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4ab0062510 con 0x7f4ab815c510 2024-09-17T04:01:19.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.488+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f4ab815ff20 con 0x7f4ab815c510 2024-09-17T04:01:19.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.488+0000 7f4aafe00640 1 -- 172.21.15.129:0/1866085353 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1694 (secure 0 0 0) 0x7f4ab0061e10 con 0x7f4ab815c510 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772747616256, 2024-09-17T04:01:19.488 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 346497024, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 346497024, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00044819514732807875, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:19.489 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772747616256, 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 346497024, 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 346497024, 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00044819514732807875 2024-09-17T04:01:19.490 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:01:19.491 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6182395938812988e-06, 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367015526400 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T04:01:19.492 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 26, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 19413, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 22, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 100, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 102400, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.3950361221759522e-07, 2024-09-17T04:01:19.493 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367015526400 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 27, 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 6291456, 2024-09-17T04:01:19.494 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:01:19.496 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 12288, 2024-09-17T04:01:19.496 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 12582912, 2024-09-17T04:01:19.496 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.7141912394436076e-05, 2024-09-17T04:01:19.496 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367015526400 2024-09-17T04:01:19.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:19.497 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:19.497 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:01:19.497 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:01:19.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.490+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 msgr2=0x7f4a9807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:19.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.490+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 0x7f4a9807b0a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8034000 tx=0x7f4aa8004ab0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.490+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 msgr2=0x7f4ab80c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.490+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f4ab000c3f0 tx=0x7f4ab000c8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 shutdown_connections 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4a98078b90 0x7f4a9807b0a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ab815c510 0x7f4ab80c8fa0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ab80cd260 0x7f4ab80c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 --2- 172.21.15.129:0/1866085353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ab80cb2f0 0x7f4ab80b24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:19.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 >> 172.21.15.129:0/1866085353 conn(0x7f4ab806ee70 msgr2=0x7f4ab80c63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:19.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 shutdown_connections 2024-09-17T04:01:19.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:19.491+0000 7f4abd800640 1 -- 172.21.15.129:0/1866085353 wait complete. 2024-09-17T04:01:19.510 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-17T04:01:24.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T04:01:24.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 -- 172.21.15.129:0/195170158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 msgr2=0x7f5a48004f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:24.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 --2- 172.21.15.129:0/195170158 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a48004f70 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5a54009b50 tx=0x7f5a48006940 comp rx=0 tx=0).stop 2024-09-17T04:01:24.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a66600640 1 -- 172.21.15.129:0/195170158 reap_dead start 2024-09-17T04:01:24.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 -- 172.21.15.129:0/195170158 shutdown_connections 2024-09-17T04:01:24.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 -- 172.21.15.129:0/195170158 >> 172.21.15.129:0/195170158 conn(0x7f5a6006d410 msgr2=0x7f5a600b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:24.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 -- 172.21.15.129:0/195170158 shutdown_connections 2024-09-17T04:01:24.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.745+0000 7f5a68600640 1 -- 172.21.15.129:0/195170158 wait complete. 2024-09-17T04:01:24.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.747+0000 7f5a68600640 1 Processor -- start 2024-09-17T04:01:24.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.747+0000 7f5a68600640 1 -- start start 2024-09-17T04:01:24.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 0x7f5a60099020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 0x7f5a601fa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6015ac20 con 0x7f5a48004b40 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5a60074200 con 0x7f5a60098c10 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a68600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6015ada0 con 0x7f5a601fa360 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57340/0 (socket says 172.21.15.129:57340) 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 -- 172.21.15.129:0/682894747 learned_addr learned my addr 172.21.15.129:0/682894747 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65200640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 0x7f5a60099020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a66600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 0x7f5a601fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:24.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 -- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 msgr2=0x7f5a601fa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 0x7f5a601fa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 -- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 msgr2=0x7f5a60099020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 0x7f5a60099020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65c00640 1 -- 172.21.15.129:0/682894747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a601e7050 con 0x7f5a48004b40 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a65200640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 0x7f5a60099020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.749+0000 7f5a66600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 0x7f5a601fa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a65c00640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5a54036040 tx=0x7f5a54033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54038050 con 0x7f5a48004b40 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a54039040 con 0x7f5a48004b40 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a54033c50 con 0x7f5a48004b40 2024-09-17T04:01:24.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a60099560 con 0x7f5a48004b40 2024-09-17T04:01:24.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.750+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a601e58f0 con 0x7f5a48004b40 2024-09-17T04:01:24.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.751+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a600b4690 con 0x7f5a48004b40 2024-09-17T04:01:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.752+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5a54046020 con 0x7f5a48004b40 2024-09-17T04:01:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.753+0000 7f5a5d600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 0x7f5a4007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.753+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f5a540bb810 con 0x7f5a48004b40 2024-09-17T04:01:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.754+0000 7f5a65200640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 0x7f5a4007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:24.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.755+0000 7f5a65200640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 0x7f5a4007b0a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5a600cd4f0 tx=0x7f5a4c006f10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:24.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:24.758+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5a5404d020 con 0x7f5a48004b40 2024-09-17T04:01:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.007+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f5a60073d20 con 0x7f5a48004b40 2024-09-17T04:01:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.008+0000 7f5a5d600640 1 -- 172.21.15.129:0/682894747 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1656 (secure 0 0 0) 0x7f5a5408a020 con 0x7f5a48004b40 2024-09-17T04:01:25.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:25.007 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:01:25.007 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772760223744, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 333889536, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 333889536, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00043188731069676578, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:25.008 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772760223744, 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 333889536, 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 333889536, 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00043188731069676578 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:25.009 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6181019191208179e-06, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367046754304 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:25.010 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 26, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 21714, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 23, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 104, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 106496, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.4507141088415665e-07, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367046754304 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 27, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 0, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 0, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 0, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 0, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 0, 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 367046754304 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:01:25.013 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:01:25.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 msgr2=0x7f5a4007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 0x7f5a4007b0a0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f5a600cd4f0 tx=0x7f5a4c006f10 comp rx=0 tx=0).stop 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 msgr2=0x7f5a601e6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f5a54036040 tx=0x7f5a54033670 comp rx=0 tx=0).stop 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 shutdown_connections 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a40078b90 0x7f5a4007b0a0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a601fa360 0x7f5a601fa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a60098c10 0x7f5a60099020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 --2- 172.21.15.129:0/682894747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a48004b40 0x7f5a601e6b10 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 >> 172.21.15.129:0/682894747 conn(0x7f5a6006d410 msgr2=0x7f5a600b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:25.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 shutdown_connections 2024-09-17T04:01:25.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.010+0000 7f5a68600640 1 -- 172.21.15.129:0/682894747 wait complete. 2024-09-17T04:01:25.034 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-17T04:01:25.034 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:01:25.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:01:25.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:01:25.224 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:01:25.224 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:01:25.224 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:25.224 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:25.260 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:25.260 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:25.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:01:25.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 -- 172.21.15.129:0/3274419476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 msgr2=0x7ff374073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 --2- 172.21.15.129:0/3274419476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff374073d40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7ff364005fd0 tx=0x7ff36402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 -- 172.21.15.129:0/3274419476 shutdown_connections 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 --2- 172.21.15.129:0/3274419476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff3741609a0 0x7ff3740cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 --2- 172.21.15.129:0/3274419476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff374073d40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 -- 172.21.15.129:0/3274419476 >> 172.21.15.129:0/3274419476 conn(0x7ff3740cd0c0 msgr2=0x7ff3740cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 -- 172.21.15.129:0/3274419476 shutdown_connections 2024-09-17T04:01:25.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.600+0000 7ff378e00640 1 -- 172.21.15.129:0/3274419476 wait complete. 2024-09-17T04:01:25.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.602+0000 7ff378e00640 1 Processor -- start 2024-09-17T04:01:25.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.603+0000 7ff378e00640 1 -- start start 2024-09-17T04:01:25.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 0x7ff3741f9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:25.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff3740b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740cb560 con 0x7ff3740b16d0 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3740cb3e0 con 0x7ff3740afd70 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff378e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff3740cb6e0 con 0x7ff374074280 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff372800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff372800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57342/0 (socket says 172.21.15.129:57342) 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.604+0000 7ff372800640 1 -- 172.21.15.129:0/1928130875 learned_addr learned my addr 172.21.15.129:0/1928130875 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff371400640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff3740b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:25.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff371e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 0x7ff3741f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 -- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 msgr2=0x7ff3741f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 0x7ff3741f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 -- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 msgr2=0x7ff3740b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff3740b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 -- 172.21.15.129:0/1928130875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3741f9850 con 0x7ff3740b16d0 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff371e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 0x7ff3741f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff372800640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00e740 tx=0x7ff36c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:25.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c00ccc0 con 0x7ff3740b16d0 2024-09-17T04:01:25.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff36c00ee50 con 0x7ff3740b16d0 2024-09-17T04:01:25.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36c00ce60 con 0x7ff3740b16d0 2024-09-17T04:01:25.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3741f5900 con 0x7ff3740b16d0 2024-09-17T04:01:25.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.605+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3741f5ec0 con 0x7ff3740b16d0 2024-09-17T04:01:25.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.607+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3740cbc00 con 0x7ff3740b16d0 2024-09-17T04:01:25.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.608+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff36c005280 con 0x7ff3740b16d0 2024-09-17T04:01:25.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.609+0000 7ff36a000640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 0x7ff35407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:25.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.609+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7ff36c09b710 con 0x7ff3740b16d0 2024-09-17T04:01:25.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.609+0000 7ff371e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 0x7ff35407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:25.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.610+0000 7ff371e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 0x7ff35407b0d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ff364034000 tx=0x7ff364004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:25.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.613+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff3741f9850 con 0x7ff3740b16d0 2024-09-17T04:01:25.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.808+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff3740af040 con 0x7ff3740b16d0 2024-09-17T04:01:25.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.808+0000 7ff36a000640 1 -- 172.21.15.129:0/1928130875 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v174) ==== 80+0+0 (secure 0 0 0) 0x7ff36c063b70 con 0x7ff3740b16d0 2024-09-17T04:01:25.809 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:01:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 msgr2=0x7ff35407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 0x7ff35407b0d0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ff364034000 tx=0x7ff364004ab0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 msgr2=0x7ff3740b57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7ff36c00e740 tx=0x7ff36c00ec00 comp rx=0 tx=0).stop 2024-09-17T04:01:25.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 shutdown_connections 2024-09-17T04:01:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff354078bc0 0x7ff35407b0d0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff3740b16d0 0x7ff3740b57c0 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff3740afd70 0x7ff3740b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 --2- 172.21.15.129:0/1928130875 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff374074280 0x7ff3741f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:25.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 >> 172.21.15.129:0/1928130875 conn(0x7ff3740cd0c0 msgr2=0x7ff374075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 shutdown_connections 2024-09-17T04:01:25.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:25.811+0000 7ff378e00640 1 -- 172.21.15.129:0/1928130875 wait complete. 2024-09-17T04:01:25.834 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:01:25.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:25.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:25.898 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:01:25.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:25.898 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:01:25.935 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:01:25.935 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:01:25.935 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:25.977 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:01:25.977 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:01:25.977 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:25.977 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:26.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:01:26.013 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:01:26.013 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:01:26.013 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:26.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:01:26.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:26.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 -- 172.21.15.129:0/3087655706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80cd1c0 msgr2=0x7f82e81609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 --2- 172.21.15.129:0/3087655706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80cd1c0 0x7f82e81609c0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f82e8072080 tx=0x7f82e002aac0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 -- 172.21.15.129:0/3087655706 shutdown_connections 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 --2- 172.21.15.129:0/3087655706 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80cd1c0 0x7f82e81609c0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 --2- 172.21.15.129:0/3087655706 >> v2:172.21.15.181:3300/0 conn(0x7f82e80b4140 0x7f82e80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 -- 172.21.15.129:0/3087655706 >> 172.21.15.129:0/3087655706 conn(0x7f82e806d410 msgr2=0x7f82e80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 -- 172.21.15.129:0/3087655706 shutdown_connections 2024-09-17T04:01:26.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.276+0000 7f82eda00640 1 -- 172.21.15.129:0/3087655706 wait complete. 2024-09-17T04:01:26.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.278+0000 7f82eda00640 1 Processor -- start 2024-09-17T04:01:26.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.279+0000 7f82eda00640 1 -- start start 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82e80b4140 0x7f82e81f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82e80cd1c0 0x7f82e809bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8160260 con 0x7f82e80705a0 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f82e81600e0 con 0x7f82e80cd1c0 2024-09-17T04:01:26.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82eda00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f82e8160f00 con 0x7f82e80b4140 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82e6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82e6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57360/0 (socket says 172.21.15.129:57360) 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.280+0000 7f82e6a00640 1 -- 172.21.15.129:0/532155573 learned_addr learned my addr 172.21.15.129:0/532155573 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 -- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82e80b4140 msgr2=0x7f82e81f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82e80b4140 0x7f82e81f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 -- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82e80cd1c0 msgr2=0x7f82e809bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82e80cd1c0 0x7f82e809bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 -- 172.21.15.129:0/532155573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e80c9e90 con 0x7f82e80705a0 2024-09-17T04:01:26.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82e6a00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c3a0 tx=0x7f82d800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8017070 con 0x7f82e80705a0 2024-09-17T04:01:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82d80053c0 con 0x7f82e80705a0 2024-09-17T04:01:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e80ca170 con 0x7f82e80705a0 2024-09-17T04:01:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d8003850 con 0x7f82e80705a0 2024-09-17T04:01:26.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.281+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e81f56b0 con 0x7f82e80705a0 2024-09-17T04:01:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.282+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82e80ccc80 con 0x7f82e80705a0 2024-09-17T04:01:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.283+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f82d801a070 con 0x7f82e80705a0 2024-09-17T04:01:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.284+0000 7f82df400640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 0x7f82c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.285+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f82d809a120 con 0x7f82e80705a0 2024-09-17T04:01:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.285+0000 7f82de000640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 0x7f82c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.286+0000 7f82de000640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 0x7f82c407b0a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f82e8073ca0 tx=0x7f82d0005b50 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:26.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.289+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f82e80cd5d0 con 0x7f82e80705a0 2024-09-17T04:01:26.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.508+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82e806a140 con 0x7f82e80705a0 2024-09-17T04:01:26.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.509+0000 7f82df400640 1 -- 172.21.15.129:0/532155573 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f82d8062550 con 0x7f82e80705a0 2024-09-17T04:01:26.509 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:26.509 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:26.509 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:26.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 msgr2=0x7f82c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 0x7f82c407b0a0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f82e8073ca0 tx=0x7f82d0005b50 comp rx=0 tx=0).stop 2024-09-17T04:01:26.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 msgr2=0x7f82e81f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f82d800c3a0 tx=0x7f82d800c860 comp rx=0 tx=0).stop 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 shutdown_connections 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f82c4078b90 0x7f82c407b0a0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82e80cd1c0 0x7f82e809bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82e80b4140 0x7f82e81f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 --2- 172.21.15.129:0/532155573 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82e80705a0 0x7f82e81f6510 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 >> 172.21.15.129:0/532155573 conn(0x7f82e806d410 msgr2=0x7f82e80ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 shutdown_connections 2024-09-17T04:01:26.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.511+0000 7f82eda00640 1 -- 172.21.15.129:0/532155573 wait complete. 2024-09-17T04:01:26.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 -- 172.21.15.129:0/3079544867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00b4750 msgr2=0x7fc7d00d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 --2- 172.21.15.129:0/3079544867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00b4750 0x7fc7d00d0320 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fc7d00d0fb0 tx=0x7fc7bc02a540 comp rx=0 tx=0).stop 2024-09-17T04:01:26.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 -- 172.21.15.129:0/3079544867 shutdown_connections 2024-09-17T04:01:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 --2- 172.21.15.129:0/3079544867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00b4750 0x7fc7d00d0320 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 --2- 172.21.15.129:0/3079544867 >> v2:172.21.15.181:3300/0 conn(0x7fc7d00ccdf0 0x7fc7d0160c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 -- 172.21.15.129:0/3079544867 >> 172.21.15.129:0/3079544867 conn(0x7fc7d006cfc0 msgr2=0x7fc7d006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 -- 172.21.15.129:0/3079544867 shutdown_connections 2024-09-17T04:01:26.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.759+0000 7fc7d5400640 1 -- 172.21.15.129:0/3079544867 wait complete. 2024-09-17T04:01:26.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.761+0000 7fc7d5400640 1 Processor -- start 2024-09-17T04:01:26.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.761+0000 7fc7d5400640 1 -- start start 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d00b4750 0x7fc7d01f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 0x7fc7d01535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d015d220 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d00d0c70 con 0x7fc7d00b4750 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d5400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d015d3a0 con 0x7fc7d00b4b60 2024-09-17T04:01:26.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57382/0 (socket says 172.21.15.129:57382) 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d4a00640 1 -- 172.21.15.129:0/1468287483 learned_addr learned my addr 172.21.15.129:0/1468287483 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7cf400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 0x7fc7d01535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7cfe00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d00b4750 0x7fc7d01f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d4a00640 1 -- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 msgr2=0x7fc7d01535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.763+0000 7fc7d4a00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 0x7fc7d01535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d4a00640 1 -- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d00b4750 msgr2=0x7fc7d01f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d4a00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d00b4750 0x7fc7d01f5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d4a00640 1 -- 172.21.15.129:0/1468287483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d01f5f10 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7cf400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 0x7fc7d01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d4a00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c390 tx=0x7fc7c800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8011790 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc7c8017070 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8011c60 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d01f61f0 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.764+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d0156310 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.765+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d0156570 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.767+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc7c801c460 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.768+0000 7fc7c7e00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 0x7fc7b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:26.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.768+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fc7c809aa20 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.769+0000 7fc7cfe00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 0x7fc7b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:26.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.769+0000 7fc7cfe00640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 0x7fc7b007b0a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005fd0 tx=0x7fc7c00095e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:26.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.773+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc7d0156570 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.991+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7d01f5ca0 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.991+0000 7fc7c7e00640 1 -- 172.21.15.129:0/1468287483 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7fc7c8062dc0 con 0x7fc7d00ccdf0 2024-09-17T04:01:26.992 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:26.992 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:26.992 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:26.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 msgr2=0x7fc7b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 0x7fc7b007b0a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0005fd0 tx=0x7fc7c00095e0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 msgr2=0x7fc7d0134670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c390 tx=0x7fc7c800c850 comp rx=0 tx=0).stop 2024-09-17T04:01:26.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 shutdown_connections 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc7b0078b90 0x7fc7b007b0a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc7d00ccdf0 0x7fc7d0134670 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc7d00b4b60 0x7fc7d01535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 --2- 172.21.15.129:0/1468287483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7d00b4750 0x7fc7d01f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 >> 172.21.15.129:0/1468287483 conn(0x7fc7d006cfc0 msgr2=0x7fc7d00cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 shutdown_connections 2024-09-17T04:01:26.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:26.994+0000 7fc7d5400640 1 -- 172.21.15.129:0/1468287483 wait complete. 2024-09-17T04:01:27.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.239+0000 7fe90cc00640 1 -- 172.21.15.129:0/1754951650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe90815da00 msgr2=0x7fe9080b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.239+0000 7fe90cc00640 1 --2- 172.21.15.129:0/1754951650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe90815da00 0x7fe9080b4180 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8005fd0 tx=0x7fe8f802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 -- 172.21.15.129:0/1754951650 shutdown_connections 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 --2- 172.21.15.129:0/1754951650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080cd390 0x7fe908070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 --2- 172.21.15.129:0/1754951650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe90815da00 0x7fe9080b4180 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 -- 172.21.15.129:0/1754951650 >> 172.21.15.129:0/1754951650 conn(0x7fe90806d410 msgr2=0x7fe9080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:27.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 -- 172.21.15.129:0/1754951650 shutdown_connections 2024-09-17T04:01:27.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.240+0000 7fe90cc00640 1 -- 172.21.15.129:0/1754951650 wait complete. 2024-09-17T04:01:27.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.242+0000 7fe90cc00640 1 Processor -- start 2024-09-17T04:01:27.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.242+0000 7fe90cc00640 1 -- start start 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.243+0000 7fe90cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080b46c0 0x7fe908159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe90cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe90cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9081f4d30 0x7fe908157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe90cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9080709d0 con 0x7fe90815da00 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe90cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe908070850 con 0x7fe9081f4d30 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe90cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe908070b50 con 0x7fe9080b46c0 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57388/0 (socket says 172.21.15.129:57388) 2024-09-17T04:01:27.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 -- 172.21.15.129:0/3582301694 learned_addr learned my addr 172.21.15.129:0/3582301694 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:27.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe905e00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080b46c0 0x7fe908159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe906800640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9081f4d30 0x7fe908157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 -- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080b46c0 msgr2=0x7fe908159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080b46c0 0x7fe908159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 -- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9081f4d30 msgr2=0x7fe908157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9081f4d30 0x7fe908157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.244+0000 7fe8fd400640 1 -- 172.21.15.129:0/3582301694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9080c8720 con 0x7fe90815da00 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe8fd400640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c400 tx=0x7fe8f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0017070 con 0x7fe90815da00 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe8f00053c0 con 0x7fe90815da00 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9080c8a00 con 0x7fe90815da00 2024-09-17T04:01:27.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0003850 con 0x7fe90815da00 2024-09-17T04:01:27.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.245+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9080c8fc0 con 0x7fe90815da00 2024-09-17T04:01:27.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.246+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe908070100 con 0x7fe90815da00 2024-09-17T04:01:27.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.248+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe8f001a070 con 0x7fe90815da00 2024-09-17T04:01:27.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.248+0000 7fe8fea00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 0x7fe8e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.249+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fe8f009a110 con 0x7fe90815da00 2024-09-17T04:01:27.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.249+0000 7fe905e00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 0x7fe8e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.250+0000 7fe905e00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 0x7fe8e407b0a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8033000 tx=0x7fe8f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:27.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.253+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe8f0061ef0 con 0x7fe90815da00 2024-09-17T04:01:27.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.438+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe9081579c0 con 0x7fe90815da00 2024-09-17T04:01:27.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.439+0000 7fe8fea00640 1 -- 172.21.15.129:0/3582301694 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) ==== 74+0+16791 (secure 0 0 0) 0x7fe8f0066920 con 0x7fe90815da00 2024-09-17T04:01:27.439 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:27.439 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":174,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:51.478826+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:47.893071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:48.928779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 msgr2=0x7fe8e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 0x7fe8e407b0a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe8f8033000 tx=0x7fe8f8004730 comp rx=0 tx=0).stop 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 msgr2=0x7fe908157400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c400 tx=0x7fe8f000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 shutdown_connections 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe8e4078b90 0x7fe8e407b0a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9081f4d30 0x7fe908157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe90815da00 0x7fe908157400 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 --2- 172.21.15.129:0/3582301694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9080b46c0 0x7fe908159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 >> 172.21.15.129:0/3582301694 conn(0x7fe90806d410 msgr2=0x7fe9080ad260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:27.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 shutdown_connections 2024-09-17T04:01:27.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.442+0000 7fe90cc00640 1 -- 172.21.15.129:0/3582301694 wait complete. 2024-09-17T04:01:27.464 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:01:27.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 -- 172.21.15.129:0/3081000594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0b4140 msgr2=0x7f154c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 --2- 172.21.15.129:0/3081000594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0b4140 0x7f154c0ccc80 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f1538009830 tx=0x7f153802a510 comp rx=0 tx=0).stop 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 -- 172.21.15.129:0/3081000594 shutdown_connections 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 --2- 172.21.15.129:0/3081000594 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0cd1c0 0x7f154c1609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 --2- 172.21.15.129:0/3081000594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0b4140 0x7f154c0ccc80 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.686+0000 7f1552a00640 1 -- 172.21.15.129:0/3081000594 >> 172.21.15.129:0/3081000594 conn(0x7f154c06d410 msgr2=0x7f154c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.687+0000 7f1552a00640 1 -- 172.21.15.129:0/3081000594 shutdown_connections 2024-09-17T04:01:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.687+0000 7f1552a00640 1 -- 172.21.15.129:0/3081000594 wait complete. 2024-09-17T04:01:27.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.689+0000 7f1552a00640 1 Processor -- start 2024-09-17T04:01:27.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.689+0000 7f1552a00640 1 -- start start 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0705a0 0x7f154c1f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f154c0b4140 0x7f154c0c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f154c160260 con 0x7f154c0cd1c0 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f154c1600e0 con 0x7f154c0b4140 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.690+0000 7f1552a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f154c160f00 con 0x7f154c0705a0 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57398/0 (socket says 172.21.15.129:57398) 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 -- 172.21.15.129:0/3103532396 learned_addr learned my addr 172.21.15.129:0/3103532396 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f154b400640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f154c0b4140 0x7f154c0c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f154be00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0705a0 0x7f154c1f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 -- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0705a0 msgr2=0x7f154c1f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0705a0 0x7f154c1f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 -- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f154c0b4140 msgr2=0x7f154c0c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f154c0b4140 0x7f154c0c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 -- 172.21.15.129:0/3103532396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f154c0c6050 con 0x7f154c0cd1c0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.691+0000 7f1550a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f154400e9e0 tx=0x7f154400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.692+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154400cdb0 con 0x7f154c0cd1c0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.692+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1544004c10 con 0x7f154c0cd1c0 2024-09-17T04:01:27.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.692+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544005020 con 0x7f154c0cd1c0 2024-09-17T04:01:27.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.692+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f154c0c6330 con 0x7f154c0cd1c0 2024-09-17T04:01:27.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.692+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f154c0c68f0 con 0x7f154c0cd1c0 2024-09-17T04:01:27.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.693+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f154c0ccc80 con 0x7f154c0cd1c0 2024-09-17T04:01:27.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.694+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f15440051c0 con 0x7f154c0cd1c0 2024-09-17T04:01:27.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.695+0000 7f1543e00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 0x7f152c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:27.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.695+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f154409b8f0 con 0x7f154c0cd1c0 2024-09-17T04:01:27.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.695+0000 7f154be00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 0x7f152c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:27.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.696+0000 7f154be00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 0x7f152c07b0d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f154c072c70 tx=0x7f153c007eb0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:27.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.699+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f15440637e0 con 0x7f154c0cd1c0 2024-09-17T04:01:27.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.918+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f154c06a140 con 0x7f154c0cd1c0 2024-09-17T04:01:27.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.919+0000 7f1543e00640 1 -- 172.21.15.129:0/3103532396 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f1544068210 con 0x7f154c0cd1c0 2024-09-17T04:01:27.919 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:27.919 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:27.919 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 msgr2=0x7f152c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 0x7f152c07b0d0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f154c072c70 tx=0x7f153c007eb0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 msgr2=0x7f154c0c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f154400e9e0 tx=0x7f154400eea0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 shutdown_connections 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.921+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f152c078bc0 0x7f152c07b0d0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f154c0cd1c0 0x7f154c0c9480 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f154c0b4140 0x7f154c0c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 --2- 172.21.15.129:0/3103532396 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f154c0705a0 0x7f154c1f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:27.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 >> 172.21.15.129:0/3103532396 conn(0x7f154c06d410 msgr2=0x7f154c0b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:27.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 shutdown_connections 2024-09-17T04:01:27.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:27.922+0000 7f1552a00640 1 -- 172.21.15.129:0/3103532396 wait complete. 2024-09-17T04:01:27.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.162+0000 7f4e0b200640 1 -- 172.21.15.129:0/1367676870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c145e60 msgr2=0x7f4e0c0d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.162+0000 7f4e0b200640 1 --2- 172.21.15.129:0/1367676870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c145e60 0x7f4e0c0d0340 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0b4220 tx=0x7f4dfc02a4d0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 -- 172.21.15.129:0/1367676870 shutdown_connections 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 --2- 172.21.15.129:0/1367676870 >> v2:172.21.15.181:3300/0 conn(0x7f4e0c06ecb0 0x7f4e0c06f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 --2- 172.21.15.129:0/1367676870 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4e0c1458a0 0x7f4e0c0d0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 --2- 172.21.15.129:0/1367676870 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c145e60 0x7f4e0c0d0340 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 -- 172.21.15.129:0/1367676870 >> 172.21.15.129:0/1367676870 conn(0x7f4e0c06cfc0 msgr2=0x7f4e0c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:28.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 -- 172.21.15.129:0/1367676870 shutdown_connections 2024-09-17T04:01:28.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.163+0000 7f4e0b200640 1 -- 172.21.15.129:0/1367676870 wait complete. 2024-09-17T04:01:28.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.165+0000 7f4e0b200640 1 Processor -- start 2024-09-17T04:01:28.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.165+0000 7f4e0b200640 1 -- start start 2024-09-17T04:01:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.166+0000 7f4e0b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.166+0000 7f4e0b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 0x7f4e0c15f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e0b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4e0c145e60 0x7f4e0c0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e0b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c0cef20 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e0b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0c0ceda0 con 0x7f4e0c1458a0 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e0b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0c06f600 con 0x7f4e0c145e60 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57424/0 (socket says 172.21.15.129:57424) 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 -- 172.21.15.129:0/2012382994 learned_addr learned my addr 172.21.15.129:0/2012382994 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09400640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 0x7f4e0c15f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 -- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4e0c145e60 msgr2=0x7f4e0c0b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4e0c145e60 0x7f4e0c0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 -- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 msgr2=0x7f4e0c15f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 0x7f4e0c15f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09e00640 1 -- 172.21.15.129:0/2012382994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e0c0ca1c0 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.167+0000 7f4e09400640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 0x7f4e0c15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e09e00640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc000c00 tx=0x7f4dfc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc03e030 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4dfc003d10 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0c0ca4a0 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc035040 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.168+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e0c0caa60 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.169+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e0c06f0c0 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.170+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4dfc041030 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.171+0000 7f4e02000640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 0x7f4dec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.171+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f4dfc0bb510 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.172+0000 7f4e09400640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 0x7f4dec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.172+0000 7f4e09400640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 0x7f4dec07b0d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0c8c40 tx=0x7f4df8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:28.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.176+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4dfc0838b0 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.392+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e0c06a140 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.392+0000 7f4e02000640 1 -- 172.21.15.129:0/2012382994 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5245 (secure 0 0 0) 0x7f4dfc0831b0 con 0x7f4e0c06ecb0 2024-09-17T04:01:28.393 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:28.393 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":154,"btime":"2024-09-17T04:00:53:540416+0000","default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:00:49.962303+0000","modified":"2024-09-17T04:00:52.981107+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9492},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9492":{"gid":9492,"name":"c","rank":0,"incarnation":150,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/1228169144","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":1228169144},{"type":"v1","addr":"172.21.15.129:6835","nonce":1228169144}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9492,"qdb_cluster":[9492]},"id":13}]} 2024-09-17T04:01:28.393 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 154 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 msgr2=0x7f4dec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 0x7f4dec07b0d0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0c8c40 tx=0x7f4df8005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 msgr2=0x7f4e0c0c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc000c00 tx=0x7f4dfc003820 comp rx=0 tx=0).stop 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 shutdown_connections 2024-09-17T04:01:28.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4dec078bc0 0x7f4dec07b0d0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4e0c145e60 0x7f4e0c0b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4e0c1458a0 0x7f4e0c15f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 --2- 172.21.15.129:0/2012382994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4e0c06ecb0 0x7f4e0c0c96c0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 >> 172.21.15.129:0/2012382994 conn(0x7f4e0c06cfc0 msgr2=0x7f4e0c0747d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 shutdown_connections 2024-09-17T04:01:28.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.395+0000 7f4e0b200640 1 -- 172.21.15.129:0/2012382994 wait complete. 2024-09-17T04:01:28.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:28.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 -- 172.21.15.129:0/3140810878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940cd1c0 msgr2=0x7fe3941609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 --2- 172.21.15.129:0/3140810878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940cd1c0 0x7fe3941609c0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fe38c005fd0 tx=0x7fe38c02aa00 comp rx=0 tx=0).stop 2024-09-17T04:01:28.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 -- 172.21.15.129:0/3140810878 shutdown_connections 2024-09-17T04:01:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 --2- 172.21.15.129:0/3140810878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940cd1c0 0x7fe3941609c0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 --2- 172.21.15.129:0/3140810878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940b4140 0x7fe3940ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 -- 172.21.15.129:0/3140810878 >> 172.21.15.129:0/3140810878 conn(0x7fe39406d410 msgr2=0x7fe3940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 -- 172.21.15.129:0/3140810878 shutdown_connections 2024-09-17T04:01:28.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.642+0000 7fe39c400640 1 -- 172.21.15.129:0/3140810878 wait complete. 2024-09-17T04:01:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.644+0000 7fe39c400640 1 Processor -- start 2024-09-17T04:01:28.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.644+0000 7fe39c400640 1 -- start start 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3940705a0 0x7fe3941f5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940cd1c0 0x7fe3940c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe394160260 con 0x7fe3940b4140 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe3941600e0 con 0x7fe3940705a0 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39c400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe394160f00 con 0x7fe3940cd1c0 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe399000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe399000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57440/0 (socket says 172.21.15.129:57440) 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe399000640 1 -- 172.21.15.129:0/1816148505 learned_addr learned my addr 172.21.15.129:0/1816148505 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:28.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe399a00640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3940705a0 0x7fe3941f5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.646+0000 7fe39a400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940cd1c0 0x7fe3940c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 -- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940cd1c0 msgr2=0x7fe3940c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940cd1c0 0x7fe3940c7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 -- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3940705a0 msgr2=0x7fe3941f5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3940705a0 0x7fe3941f5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 -- 172.21.15.129:0/1816148505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3940c6a20 con 0x7fe3940b4140 2024-09-17T04:01:28.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe399000640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c940 tx=0x7fe38000ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:28.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.647+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380004220 con 0x7fe3940b4140 2024-09-17T04:01:28.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.648+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe3800079b0 con 0x7fe3940b4140 2024-09-17T04:01:28.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.648+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe380007de0 con 0x7fe3940b4140 2024-09-17T04:01:28.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.648+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3940c4e10 con 0x7fe3940b4140 2024-09-17T04:01:28.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.648+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3940c64b0 con 0x7fe3940b4140 2024-09-17T04:01:28.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.649+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3940ccc80 con 0x7fe3940b4140 2024-09-17T04:01:28.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.650+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe380005020 con 0x7fe3940b4140 2024-09-17T04:01:28.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.651+0000 7fe391600640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 0x7fe37407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:28.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.651+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7fe38009b070 con 0x7fe3940b4140 2024-09-17T04:01:28.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.652+0000 7fe399a00640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 0x7fe37407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:28.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.652+0000 7fe399a00640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 0x7fe37407b0a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe388006fd0 tx=0x7fe388009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:28.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.657+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe380014030 con 0x7fe3940b4140 2024-09-17T04:01:28.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.843+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3940c67b0 con 0x7fe3940b4140 2024-09-17T04:01:28.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.844+0000 7fe391600640 1 -- 172.21.15.129:0/1816148505 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) ==== 74+0+16791 (secure 0 0 0) 0x7fe380062e60 con 0x7fe3940b4140 2024-09-17T04:01:28.844 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:28.844 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":174,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:00:51.478826+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:00:47.893071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"173","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-17T04:00:48.928779+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"174","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 msgr2=0x7fe37407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 0x7fe37407b0a0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe388006fd0 tx=0x7fe388009060 comp rx=0 tx=0).stop 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 msgr2=0x7fe3940c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c940 tx=0x7fe38000ce00 comp rx=0 tx=0).stop 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 shutdown_connections 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe374078b90 0x7fe37407b0a0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe3940cd1c0 0x7fe3940c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe3940b4140 0x7fe3940c7600 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 --2- 172.21.15.129:0/1816148505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3940705a0 0x7fe3941f5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 >> 172.21.15.129:0/1816148505 conn(0x7fe39406d410 msgr2=0x7fe3940cb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 shutdown_connections 2024-09-17T04:01:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:28.847+0000 7fe39c400640 1 -- 172.21.15.129:0/1816148505 wait complete. 2024-09-17T04:01:28.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:01:29.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 -- 172.21.15.129:0/15337345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e30070100 msgr2=0x7f9e30070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 --2- 172.21.15.129:0/15337345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e30070100 0x7f9e30070510 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f9e28005fd0 tx=0x7f9e2802aa90 comp rx=0 tx=0).stop 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 -- 172.21.15.129:0/15337345 shutdown_connections 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 --2- 172.21.15.129:0/15337345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e30070100 0x7f9e30070510 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 --2- 172.21.15.129:0/15337345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e300ccc80 0x7f9e300cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 -- 172.21.15.129:0/15337345 >> 172.21.15.129:0/15337345 conn(0x7f9e3006d410 msgr2=0x7f9e300b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 -- 172.21.15.129:0/15337345 shutdown_connections 2024-09-17T04:01:29.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.093+0000 7f9e36e00640 1 -- 172.21.15.129:0/15337345 wait complete. 2024-09-17T04:01:29.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.095+0000 7f9e36e00640 1 Processor -- start 2024-09-17T04:01:29.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.096+0000 7f9e36e00640 1 -- start start 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e30070100 0x7f9e301ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3015da00 0x7f9e301f9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e30070bd0 con 0x7f9e300ccc80 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e30070a50 con 0x7f9e3015da00 2024-09-17T04:01:29.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e36e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e30070d50 con 0x7f9e30070100 2024-09-17T04:01:29.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e27400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e27400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57450/0 (socket says 172.21.15.129:57450) 2024-09-17T04:01:29.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e27400640 1 -- 172.21.15.129:0/3671968297 learned_addr learned my addr 172.21.15.129:0/3671968297 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:29.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e34e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3015da00 0x7f9e301f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.097+0000 7f9e2fe00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e30070100 0x7f9e301ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 -- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e30070100 msgr2=0x7f9e301ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e30070100 0x7f9e301ef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 -- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3015da00 msgr2=0x7f9e301f9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3015da00 0x7f9e301f9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 -- 172.21.15.129:0/3671968297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e301efe20 con 0x7f9e300ccc80 2024-09-17T04:01:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e27400640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800e790 tx=0x7f9e1800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1800eea0 con 0x7f9e300ccc80 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e1800cdb0 con 0x7f9e300ccc80 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e30099430 con 0x7f9e300ccc80 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18010690 con 0x7f9e300ccc80 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.098+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e300cc270 con 0x7f9e300ccc80 2024-09-17T04:01:29.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.100+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e30070510 con 0x7f9e300ccc80 2024-09-17T04:01:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.101+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e18010830 con 0x7f9e300ccc80 2024-09-17T04:01:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.102+0000 7f9e2cc00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.102+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(174..174 src has 1..174) ==== 5709+0+0 (secure 0 0 0) 0x7f9e1809b790 con 0x7f9e300ccc80 2024-09-17T04:01:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.102+0000 7f9e2fe00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.103+0000 7f9e2fe00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9e20007fd0 tx=0x7f9e20009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:29.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.106+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e180634e0 con 0x7f9e300ccc80 2024-09-17T04:01:29.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.321+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f9e3006a140 con 0x7f9e300ccc80 2024-09-17T04:01:29.322 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:01:29.322+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:01:29.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.324+0000 7f9e2cc00640 1 -- 172.21.15.129:0/3671968297 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v155) ==== 161+0+0 (secure 0 0 0) 0x7f9e18067f10 con 0x7f9e300ccc80 2024-09-17T04:01:29.324 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:01:29.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.328+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 msgr2=0x7f9e0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.328+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f9e20007fd0 tx=0x7f9e20009040 comp rx=0 tx=0).stop 2024-09-17T04:01:29.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.328+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 msgr2=0x7f9e301f9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.328+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f9e1800e790 tx=0x7f9e1800ec50 comp rx=0 tx=0).stop 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 shutdown_connections 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e0c078b90 0x7f9e0c07b0a0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e3015da00 0x7f9e301f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e300ccc80 0x7f9e301f9020 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 --2- 172.21.15.129:0/3671968297 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e30070100 0x7f9e301ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 >> 172.21.15.129:0/3671968297 conn(0x7f9e3006d410 msgr2=0x7f9e300ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:29.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 shutdown_connections 2024-09-17T04:01:29.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.329+0000 7f9e36e00640 1 -- 172.21.15.129:0/3671968297 wait complete. 2024-09-17T04:01:29.338 INFO:tasks.ceph.mds.c.smithi129.stderr: -5873> 2024-09-17T04:00:56.449+0000 7f2963400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:01:29.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:01:29.465 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:01:29.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 -- 172.21.15.129:0/1994403833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b3010 msgr2=0x7f37d00b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 --2- 172.21.15.129:0/1994403833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b3010 0x7f37d00b3420 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f37c8005fd0 tx=0x7f37c802a540 comp rx=0 tx=0).stop 2024-09-17T04:01:29.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 -- 172.21.15.129:0/1994403833 shutdown_connections 2024-09-17T04:01:29.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 --2- 172.21.15.129:0/1994403833 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b3010 0x7f37d00b3420 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 --2- 172.21.15.129:0/1994403833 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d00b1c40 0x7f37d00b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 -- 172.21.15.129:0/1994403833 >> 172.21.15.129:0/1994403833 conn(0x7f37d00ccc40 msgr2=0x7f37d00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:29.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 -- 172.21.15.129:0/1994403833 shutdown_connections 2024-09-17T04:01:29.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.587+0000 7f37d6200640 1 -- 172.21.15.129:0/1994403833 wait complete. 2024-09-17T04:01:29.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.589+0000 7f37d6200640 1 Processor -- start 2024-09-17T04:01:29.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.589+0000 7f37d6200640 1 -- start start 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d0074800 0x7f37d01f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37d00b3010 0x7f37d015e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00b3ae0 con 0x7f37d00b1c40 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f37d00b3960 con 0x7f37d00b3010 2024-09-17T04:01:29.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37d6200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f37d00b3c60 con 0x7f37d0074800 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37cea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37cea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57496/0 (socket says 172.21.15.129:57496) 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37cea00640 1 -- 172.21.15.129:0/418862113 learned_addr learned my addr 172.21.15.129:0/418862113 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37cf400640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d0074800 0x7f37d01f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.591+0000 7f37cfe00640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37d00b3010 0x7f37d015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 -- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d0074800 msgr2=0x7f37d01f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d0074800 0x7f37d01f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 -- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37d00b3010 msgr2=0x7f37d015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37d00b3010 0x7f37d015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 -- 172.21.15.129:0/418862113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d01f6000 con 0x7f37d00b1c40 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37cea00640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e7c0 tx=0x7f37bc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc00ccc0 con 0x7f37d00b1c40 2024-09-17T04:01:29.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f37bc00ce60 con 0x7f37d00b1c40 2024-09-17T04:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc005280 con 0x7f37d00b1c40 2024-09-17T04:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37d01f62e0 con 0x7f37d00b1c40 2024-09-17T04:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.592+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37d00c81a0 con 0x7f37d00b1c40 2024-09-17T04:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.593+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37d00b4130 con 0x7f37d00b1c40 2024-09-17T04:01:29.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.595+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f37bc005420 con 0x7f37d00b1c40 2024-09-17T04:01:29.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.595+0000 7f37c7400640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 0x7f37b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:29.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.596+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 5795+0+0 (secure 0 0 0) 0x7f37bc09b6a0 con 0x7f37d00b1c40 2024-09-17T04:01:29.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.596+0000 7f37cf400640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 0x7f37b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:29.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.597+0000 7f37cf400640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 0x7f37b007b0a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007fd0 tx=0x7f37c0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:29.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.600+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f37bc063b00 con 0x7f37d00b1c40 2024-09-17T04:01:29.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:29.818+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f37d00b2050 con 0x7f37d00b1c40 2024-09-17T04:01:30.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.326+0000 7f37c7400640 1 -- 172.21.15.129:0/418862113 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v156) ==== 104+0+0 (secure 0 0 0) 0x7f37bc063400 con 0x7f37d00b1c40 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 msgr2=0x7f37b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 0x7f37b007b0a0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f37c0007fd0 tx=0x7f37c0009040 comp rx=0 tx=0).stop 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 msgr2=0x7f37d01593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e7c0 tx=0x7f37bc00ec80 comp rx=0 tx=0).stop 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 shutdown_connections 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f37b0078b90 0x7f37b007b0a0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37d00b3010 0x7f37d015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37d00b1c40 0x7f37d01593a0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 --2- 172.21.15.129:0/418862113 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37d0074800 0x7f37d01f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 >> 172.21.15.129:0/418862113 conn(0x7f37d00ccc40 msgr2=0x7f37d00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:30.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 shutdown_connections 2024-09-17T04:01:30.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.329+0000 7f37d6200640 1 -- 172.21.15.129:0/418862113 wait complete. 2024-09-17T04:01:30.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:01:30.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 -- 172.21.15.129:0/2007833901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 msgr2=0x7fef740cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:30.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 --2- 172.21.15.129:0/2007833901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef740cbea0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7fef6c005fd0 tx=0x7fef6c02aa10 comp rx=0 tx=0).stop 2024-09-17T04:01:30.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 -- 172.21.15.129:0/2007833901 shutdown_connections 2024-09-17T04:01:30.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 --2- 172.21.15.129:0/2007833901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef740cbea0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 --2- 172.21.15.129:0/2007833901 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef74074580 0x7fef740b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 -- 172.21.15.129:0/2007833901 >> 172.21.15.129:0/2007833901 conn(0x7fef740ccc40 msgr2=0x7fef740cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:30.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 -- 172.21.15.129:0/2007833901 shutdown_connections 2024-09-17T04:01:30.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.583+0000 7fef7aa00640 1 -- 172.21.15.129:0/2007833901 wait complete. 2024-09-17T04:01:30.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.585+0000 7fef7aa00640 1 Processor -- start 2024-09-17T04:01:30.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.585+0000 7fef7aa00640 1 -- start start 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef74074580 0x7fef74157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef74158770 0x7fef741f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef740cc930 con 0x7fef740cba90 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef740cc7b0 con 0x7fef74074580 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef7aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fef74166f20 con 0x7fef74158770 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57512/0 (socket says 172.21.15.129:57512) 2024-09-17T04:01:30.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 -- 172.21.15.129:0/264488878 learned_addr learned my addr 172.21.15.129:0/264488878 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 -- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef74158770 msgr2=0x7fef741f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef74158770 0x7fef741f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 -- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef74074580 msgr2=0x7fef74157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef74074580 0x7fef74157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.587+0000 7fef73400640 1 -- 172.21.15.129:0/264488878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef740cf0c0 con 0x7fef740cba90 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef73400640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d750 tx=0x7fef6000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60017070 con 0x7fef740cba90 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef60004310 con 0x7fef740cba90 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef740cede0 con 0x7fef740cba90 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60009c30 con 0x7fef740cba90 2024-09-17T04:01:30.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.588+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef741f8b50 con 0x7fef740cba90 2024-09-17T04:01:30.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.589+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef740b3f70 con 0x7fef740cba90 2024-09-17T04:01:30.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.591+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef60005020 con 0x7fef740cba90 2024-09-17T04:01:30.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.592+0000 7fef6be00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 0x7fef5407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:30.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.592+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 5795+0+0 (secure 0 0 0) 0x7fef6009b180 con 0x7fef740cba90 2024-09-17T04:01:30.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.592+0000 7fef73e00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 0x7fef5407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:30.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.593+0000 7fef73e00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 0x7fef5407b0d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fef64000f60 tx=0x7fef640079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.597+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef60020020 con 0x7fef740cba90 2024-09-17T04:01:30.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:30.780+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fef740b1fc0 con 0x7fef740cba90 2024-09-17T04:01:31.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.670+0000 7fef6be00640 1 -- 172.21.15.129:0/264488878 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v176) ==== 181+0+0 (secure 0 0 0) 0x7fef60063550 con 0x7fef740cba90 2024-09-17T04:01:31.670 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:01:31.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 msgr2=0x7fef5407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:31.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 0x7fef5407b0d0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fef64000f60 tx=0x7fef640079b0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 msgr2=0x7fef741f9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d750 tx=0x7fef6000dc10 comp rx=0 tx=0).stop 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 shutdown_connections 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef54078bc0 0x7fef5407b0d0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef74158770 0x7fef741f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef740cba90 0x7fef741f9350 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 --2- 172.21.15.129:0/264488878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef74074580 0x7fef74157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.674+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 >> 172.21.15.129:0/264488878 conn(0x7fef740ccc40 msgr2=0x7fef740afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.675+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 shutdown_connections 2024-09-17T04:01:31.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.675+0000 7fef7aa00640 1 -- 172.21.15.129:0/264488878 wait complete. 2024-09-17T04:01:31.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:01:31.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 -- 172.21.15.129:0/2992184428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 msgr2=0x7f43080b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 --2- 172.21.15.129:0/2992184428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f43080b49a0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f42f4005fd0 tx=0x7f42f402a540 comp rx=0 tx=0).stop 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 -- 172.21.15.129:0/2992184428 shutdown_connections 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 --2- 172.21.15.129:0/2992184428 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 0x7f43080cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 --2- 172.21.15.129:0/2992184428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f43080b49a0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 -- 172.21.15.129:0/2992184428 >> 172.21.15.129:0/2992184428 conn(0x7f43080ccc40 msgr2=0x7f43080cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 -- 172.21.15.129:0/2992184428 shutdown_connections 2024-09-17T04:01:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.932+0000 7f430fc00640 1 -- 172.21.15.129:0/2992184428 wait complete. 2024-09-17T04:01:31.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.935+0000 7f430fc00640 1 Processor -- start 2024-09-17T04:01:31.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.935+0000 7f430fc00640 1 -- start start 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 0x7f430814d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f430815b260 0x7f430815b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43080cc8e0 con 0x7f4308074580 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43080cc760 con 0x7f430815b260 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.936+0000 7f430fc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4308166f20 con 0x7f43080cba90 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57520/0 (socket says 172.21.15.129:57520) 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 -- 172.21.15.129:0/2307536541 learned_addr learned my addr 172.21.15.129:0/2307536541 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430dc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f430815b260 0x7f430815b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f4307e00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 0x7f430814d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 -- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 msgr2=0x7f430814d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 0x7f430814d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 -- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f430815b260 msgr2=0x7f430815b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f430815b260 0x7f430815b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 -- 172.21.15.129:0/2307536541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43080ce070 con 0x7f4308074580 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.937+0000 7f430d200640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c3a0 tx=0x7f42fc00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.938+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc017070 con 0x7f4308074580 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.938+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42fc0053c0 con 0x7f4308074580 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.938+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42fc003850 con 0x7f4308074580 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.938+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43080ce350 con 0x7f4308074580 2024-09-17T04:01:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.938+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43080ce910 con 0x7f4308074580 2024-09-17T04:01:31.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.939+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43080b3f70 con 0x7f4308074580 2024-09-17T04:01:31.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.940+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f42fc01a070 con 0x7f4308074580 2024-09-17T04:01:31.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.941+0000 7f4304c00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 0x7f42e807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:31.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.941+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(176..176 src has 1..176) ==== 5357+0+0 (secure 0 0 0) 0x7f42fc099f10 con 0x7f4308074580 2024-09-17T04:01:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.942+0000 7f4307e00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 0x7f42e807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:31.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.943+0000 7f4307e00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 0x7f42e807b0d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f42f4033000 tx=0x7f42f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:31.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:31.946+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f42fc062520 con 0x7f4308074580 2024-09-17T04:01:32.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.129+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f43080b0240 con 0x7f4308074580 2024-09-17T04:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.676+0000 7f4304c00640 1 -- 172.21.15.129:0/2307536541 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v177) ==== 169+0+0 (secure 0 0 0) 0x7f42fc061e20 con 0x7f4308074580 2024-09-17T04:01:32.676 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:01:32.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 msgr2=0x7f42e807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 0x7f42e807b0d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f42f4033000 tx=0x7f42f4004730 comp rx=0 tx=0).stop 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 msgr2=0x7f430814c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f42fc00c3a0 tx=0x7f42fc00c860 comp rx=0 tx=0).stop 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 shutdown_connections 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42e8078bc0 0x7f42e807b0d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f430815b260 0x7f430815b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43080cba90 0x7f430814d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 --2- 172.21.15.129:0/2307536541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4308074580 0x7f430814c600 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.679+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 >> 172.21.15.129:0/2307536541 conn(0x7f43080ccc40 msgr2=0x7f4308155e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:32.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.680+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 shutdown_connections 2024-09-17T04:01:32.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.681+0000 7f430fc00640 1 -- 172.21.15.129:0/2307536541 wait complete. 2024-09-17T04:01:32.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 -- 172.21.15.129:0/3734693599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 msgr2=0x7fdd3415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 --2- 172.21.15.129:0/3734693599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd3415bca0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fdd340711d0 tx=0x7fdd2002a560 comp rx=0 tx=0).stop 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 -- 172.21.15.129:0/3734693599 shutdown_connections 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 --2- 172.21.15.129:0/3734693599 >> v2:172.21.15.181:3300/0 conn(0x7fdd3415c1e0 0x7fdd34072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 --2- 172.21.15.129:0/3734693599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd3415bca0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 --2- 172.21.15.129:0/3734693599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd341524c0 0x7fdd340cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 -- 172.21.15.129:0/3734693599 >> 172.21.15.129:0/3734693599 conn(0x7fdd3406cfc0 msgr2=0x7fdd3406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.934+0000 7fdd33200640 1 -- 172.21.15.129:0/3734693599 shutdown_connections 2024-09-17T04:01:32.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.935+0000 7fdd33200640 1 -- 172.21.15.129:0/3734693599 wait complete. 2024-09-17T04:01:32.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.937+0000 7fdd33200640 1 Processor -- start 2024-09-17T04:01:32.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.937+0000 7fdd33200640 1 -- start start 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd341524c0 0x7fdd34153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd3415c1e0 0x7fdd3415d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34073810 con 0x7fdd340cfb00 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34073690 con 0x7fdd341524c0 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd33200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdd340d02c0 con 0x7fdd3415c1e0 2024-09-17T04:01:32.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd31e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57528/0 (socket says 172.21.15.129:57528) 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.938+0000 7fdd31e00640 1 -- 172.21.15.129:0/1323000267 learned_addr learned my addr 172.21.15.129:0/1323000267 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31400640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd341524c0 0x7fdd34153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd32800640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd3415c1e0 0x7fdd3415d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 -- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd3415c1e0 msgr2=0x7fdd3415d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd3415c1e0 0x7fdd3415d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 -- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd341524c0 msgr2=0x7fdd34153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd341524c0 0x7fdd34153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 -- 172.21.15.129:0/1323000267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd341541c0 con 0x7fdd340cfb00 2024-09-17T04:01:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd31e00640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400b410 tx=0x7fdd2400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:32.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24015020 con 0x7fdd340cfb00 2024-09-17T04:01:32.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdd24011940 con 0x7fdd340cfb00 2024-09-17T04:01:32.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd341f9970 con 0x7fdd340cfb00 2024-09-17T04:01:32.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.939+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24011d90 con 0x7fdd340cfb00 2024-09-17T04:01:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.940+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd341f9f30 con 0x7fdd340cfb00 2024-09-17T04:01:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.941+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd340cf5c0 con 0x7fdd340cfb00 2024-09-17T04:01:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.942+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdd24018020 con 0x7fdd340cfb00 2024-09-17T04:01:32.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.943+0000 7fdd2a000640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 0x7fdd1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:32.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.943+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7fdd2409a990 con 0x7fdd340cfb00 2024-09-17T04:01:32.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.943+0000 7fdd31400640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 0x7fdd1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:32.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.944+0000 7fdd31400640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 0x7fdd1407b0a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdd20033000 tx=0x7fdd20004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:32.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:32.947+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdd2401f020 con 0x7fdd340cfb00 2024-09-17T04:01:33.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.130+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7fdd3406a140 con 0x7fdd340cfb00 2024-09-17T04:01:33.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.135+0000 7fdd2a000640 1 -- 172.21.15.129:0/1323000267 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v59) ==== 71+0+0 (secure 0 0 0) 0x7fdd240630c0 con 0x7fdd340cfb00 2024-09-17T04:01:33.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 msgr2=0x7fdd1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 0x7fdd1407b0a0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fdd20033000 tx=0x7fdd20004730 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 msgr2=0x7fdd34154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400b410 tx=0x7fdd2400b8d0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 shutdown_connections 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdd14078b90 0x7fdd1407b0a0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdd3415c1e0 0x7fdd3415d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdd341524c0 0x7fdd34153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 --2- 172.21.15.129:0/1323000267 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdd340cfb00 0x7fdd34154de0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 >> 172.21.15.129:0/1323000267 conn(0x7fdd3406cfc0 msgr2=0x7fdd340cca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 shutdown_connections 2024-09-17T04:01:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.137+0000 7fdd33200640 1 -- 172.21.15.129:0/1323000267 wait complete. 2024-09-17T04:01:33.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode' 2024-09-17T04:01:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 -- 172.21.15.129:0/179004517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740ce170 msgr2=0x7f9a740cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 --2- 172.21.15.129:0/179004517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740ce170 0x7f9a740cbbf0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f9a64009830 tx=0x7f9a6402a510 comp rx=0 tx=0).stop 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 -- 172.21.15.129:0/179004517 shutdown_connections 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 --2- 172.21.15.129:0/179004517 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b3b90 0x7f9a7406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 --2- 172.21.15.129:0/179004517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a740ce170 0x7f9a740cbbf0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 -- 172.21.15.129:0/179004517 >> 172.21.15.129:0/179004517 conn(0x7f9a7406d410 msgr2=0x7f9a740b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 -- 172.21.15.129:0/179004517 shutdown_connections 2024-09-17T04:01:33.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.386+0000 7f9a78a00640 1 -- 172.21.15.129:0/179004517 wait complete. 2024-09-17T04:01:33.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.388+0000 7f9a78a00640 1 Processor -- start 2024-09-17T04:01:33.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.388+0000 7f9a78a00640 1 -- start start 2024-09-17T04:01:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 0x7f9a74157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 0x7f9a7415c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7406f410 con 0x7f9a74158350 2024-09-17T04:01:33.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9a74070910 con 0x7f9a740ce170 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a78a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9a7406f590 con 0x7f9a740b2950 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57542/0 (socket says 172.21.15.129:57542) 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 -- 172.21.15.129:0/4266075018 learned_addr learned my addr 172.21.15.129:0/4266075018 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a71e00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 0x7f9a74157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 -- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 msgr2=0x7f9a74157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a69400640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 0x7f9a7415c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 0x7f9a74157630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 -- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 msgr2=0x7f9a7415c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 0x7f9a7415c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a72800640 1 -- 172.21.15.129:0/4266075018 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7415e990 con 0x7f9a74158350 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a71e00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 0x7f9a74157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:33.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.390+0000 7f9a69400640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 0x7f9a7415c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a72800640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00d9d0 tx=0x7f9a6c00de90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c017070 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9a6c003970 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7415ec70 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00be60 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.391+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a741f9b80 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.392+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a7406fa40 con 0x7f9a74158350 2024-09-17T04:01:33.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.393+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9a6c010460 con 0x7f9a74158350 2024-09-17T04:01:33.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.394+0000 7f9a6aa00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 0x7f9a5007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.394+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7f9a6c09ae30 con 0x7f9a74158350 2024-09-17T04:01:33.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.395+0000 7f9a71e00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 0x7f9a5007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.395+0000 7f9a71e00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 0x7f9a5007b0d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a64034000 tx=0x7f9a64004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:33.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.399+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9a6c063560 con 0x7f9a74158350 2024-09-17T04:01:33.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.651+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode"]} v 0) -- 0x7f9a7406efd0 con 0x7f9a74158350 2024-09-17T04:01:33.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.678+0000 7f9a6aa00640 1 -- 172.21.15.129:0/4266075018 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_open_inode"]}]=0 v625) ==== 130+0+0 (secure 0 0 0) 0x7f9a6c062e60 con 0x7f9a74158350 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.680+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 msgr2=0x7f9a5007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.680+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 0x7f9a5007b0d0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9a64034000 tx=0x7f9a64004730 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.680+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 msgr2=0x7f9a7415cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.680+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00d9d0 tx=0x7f9a6c00de90 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 shutdown_connections 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a50078bc0 0x7f9a5007b0d0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a74158350 0x7f9a7415cdf0 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a740ce170 0x7f9a7415c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 --2- 172.21.15.129:0/4266075018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9a740b2950 0x7f9a74157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 >> 172.21.15.129:0/4266075018 conn(0x7f9a7406d410 msgr2=0x7f9a7415b320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:33.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 shutdown_connections 2024-09-17T04:01:33.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.681+0000 7f9a78a00640 1 -- 172.21.15.129:0/4266075018 wait complete. 2024-09-17T04:01:33.707 INFO:tasks.cephfs_test_runner:That the case of a dentry unlinked while a client holds an ... ok 2024-09-17T04:01:33.707 INFO:tasks.cephfs_test_runner:Starting test: test_ops_throttle (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:01:33.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 -- 172.21.15.129:0/3402445204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080b39c0 msgr2=0x7f6e0806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 --2- 172.21.15.129:0/3402445204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080b39c0 0x7f6e0806eed0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6e08071050 tx=0x7f6e0002aa30 comp rx=0 tx=0).stop 2024-09-17T04:01:33.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 -- 172.21.15.129:0/3402445204 shutdown_connections 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 --2- 172.21.15.129:0/3402445204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080b39c0 0x7f6e0806eed0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 --2- 172.21.15.129:0/3402445204 >> v2:172.21.15.181:3300/0 conn(0x7f6e080b2950 0x7f6e080b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 --2- 172.21.15.129:0/3402445204 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080ce170 0x7f6e080cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 -- 172.21.15.129:0/3402445204 >> 172.21.15.129:0/3402445204 conn(0x7f6e0806d410 msgr2=0x7f6e080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 -- 172.21.15.129:0/3402445204 shutdown_connections 2024-09-17T04:01:33.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.928+0000 7f6e10200640 1 -- 172.21.15.129:0/3402445204 wait complete. 2024-09-17T04:01:33.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.930+0000 7f6e10200640 1 Processor -- start 2024-09-17T04:01:33.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.930+0000 7f6e10200640 1 -- start start 2024-09-17T04:01:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e080b2950 0x7f6e081e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080b39c0 0x7f6e0815e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08070850 con 0x7f6e080ce170 2024-09-17T04:01:33.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6e080706d0 con 0x7f6e080b2950 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e10200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6e0806f410 con 0x7f6e080b39c0 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50444/0 (socket says 172.21.15.129:50444) 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 -- 172.21.15.129:0/708945143 learned_addr learned my addr 172.21.15.129:0/708945143 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0ce00640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080b39c0 0x7f6e0815e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0d800640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e080b2950 0x7f6e081e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 -- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080b39c0 msgr2=0x7f6e0815e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080b39c0 0x7f6e0815e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 -- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e080b2950 msgr2=0x7f6e081e3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e080b2950 0x7f6e081e3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.932+0000 7f6e0e200640 1 -- 172.21.15.129:0/708945143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e081e4510 con 0x7f6e080ce170 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e0e200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009d60 tx=0x7f6e00003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:33.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00033050 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6e00004450 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00038450 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e081f3b90 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.933+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081f4150 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.935+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e0806f8f0 con 0x7f6e080ce170 2024-09-17T04:01:33.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.936+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6e00036070 con 0x7f6e080ce170 2024-09-17T04:01:33.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.936+0000 7f6e05600640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 0x7f6de807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:33.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.937+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7f6e000bbd80 con 0x7f6e080ce170 2024-09-17T04:01:33.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.937+0000 7f6e0d800640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 0x7f6de807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:33.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.938+0000 7f6e0d800640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 0x7f6de807b0a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00bb50 tx=0x7f6dfc00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:33.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:33.941+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6e080b3dd0 con 0x7f6e080ce170 2024-09-17T04:01:34.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.123+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e0815a520 con 0x7f6e080ce170 2024-09-17T04:01:34.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.124+0000 7f6e05600640 1 -- 172.21.15.129:0/708945143 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) ==== 74+0+13092 (secure 0 0 0) 0x7f6e00083e70 con 0x7f6e080ce170 2024-09-17T04:01:34.124 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:34.124 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":177,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:01:32.671287+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":172,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/1228169144":"2024-09-18T04:01:29.323476+0000","172.21.15.129:6834/1228169144":"2024-09-18T04:01:29.323476+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 msgr2=0x7f6de807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 0x7f6de807b0a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00bb50 tx=0x7f6dfc00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 msgr2=0x7f6e0815e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009d60 tx=0x7f6e00003f10 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 shutdown_connections 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6de8078b90 0x7f6de807b0a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6e080ce170 0x7f6e0815e5e0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6e080b39c0 0x7f6e0815e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 --2- 172.21.15.129:0/708945143 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6e080b2950 0x7f6e081e3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.126+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 >> 172.21.15.129:0/708945143 conn(0x7f6e0806d410 msgr2=0x7f6e080af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:34.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.127+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 shutdown_connections 2024-09-17T04:01:34.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.127+0000 7f6e10200640 1 -- 172.21.15.129:0/708945143 wait complete. 2024-09-17T04:01:34.149 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:01:34.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 -- 172.21.15.129:0/1834120681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0cc170 msgr2=0x7fb2cc0ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 --2- 172.21.15.129:0/1834120681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc0ce5f0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0b51b0 tx=0x7fb2b402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 -- 172.21.15.129:0/1834120681 shutdown_connections 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 --2- 172.21.15.129:0/1834120681 >> v2:172.21.15.181:3300/0 conn(0x7fb2cc0ceb30 0x7fb2cc0ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 --2- 172.21.15.129:0/1834120681 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc0ce5f0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 -- 172.21.15.129:0/1834120681 >> 172.21.15.129:0/1834120681 conn(0x7fb2cc0ccc40 msgr2=0x7fb2cc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 -- 172.21.15.129:0/1834120681 shutdown_connections 2024-09-17T04:01:34.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.373+0000 7fb2d2200640 1 -- 172.21.15.129:0/1834120681 wait complete. 2024-09-17T04:01:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.376+0000 7fb2d2200640 1 Processor -- start 2024-09-17T04:01:34.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.376+0000 7fb2d2200640 1 -- start start 2024-09-17T04:01:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2cc070b20 0x7fb2cc160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0cb2c0 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2cc0cb140 con 0x7fb2cc0cc170 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.377+0000 7fb2d2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb2cc0cb440 con 0x7fb2cc070b20 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50464/0 (socket says 172.21.15.129:50464) 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 -- 172.21.15.129:0/2362861368 learned_addr learned my addr 172.21.15.129:0/2362861368 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cb400640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2cc070b20 0x7fb2cc160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2c2a00640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 -- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2cc070b20 msgr2=0x7fb2cc160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2cc070b20 0x7fb2cc160580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 -- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2cc0cc170 msgr2=0x7fb2cc1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc1517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 -- 172.21.15.129:0/2362861368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2cc163140 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.378+0000 7fb2cbe00640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d690 tx=0x7fb2c400db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.379+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4017070 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.379+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2c4005310 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.379+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4009820 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.379+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2cc163420 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.379+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2cc151d00 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.380+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2cc0ce5f0 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.381+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb2c40099c0 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.382+0000 7fb2c3e00640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 0x7fb2a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.382+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7fb2c409aaa0 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.382+0000 7fb2cb400640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 0x7fb2a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.383+0000 7fb2cb400640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 0x7fb2a807b0a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc001e20 tx=0x7fb2bc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.386+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb2c409ce20 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.568+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2cc1635f0 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.568+0000 7fb2c3e00640 1 -- 172.21.15.129:0/2362861368 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v177) ==== 93+0+10 (secure 0 0 0) 0x7fb2c4062b80 con 0x7fb2cc0ceb30 2024-09-17T04:01:34.569 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:01:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 msgr2=0x7fb2a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 0x7fb2a807b0a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc001e20 tx=0x7fb2bc009040 comp rx=0 tx=0).stop 2024-09-17T04:01:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 msgr2=0x7fb2cc0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d690 tx=0x7fb2c400db50 comp rx=0 tx=0).stop 2024-09-17T04:01:34.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 shutdown_connections 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2a8078b90 0x7fb2a807b0a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2cc0ceb30 0x7fb2cc0b20f0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2cc0cc170 0x7fb2cc1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 --2- 172.21.15.129:0/2362861368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2cc070b20 0x7fb2cc160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 >> 172.21.15.129:0/2362861368 conn(0x7fb2cc0ccc40 msgr2=0x7fb2cc15f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 shutdown_connections 2024-09-17T04:01:34.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.571+0000 7fb2d2200640 1 -- 172.21.15.129:0/2362861368 wait complete. 2024-09-17T04:01:34.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle' 2024-09-17T04:01:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 -- 172.21.15.129:0/1228588218 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 msgr2=0x7fe17c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 --2- 172.21.15.129:0/1228588218 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0ccc80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe168009830 tx=0x7fe16802a510 comp rx=0 tx=0).stop 2024-09-17T04:01:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 -- 172.21.15.129:0/1228588218 shutdown_connections 2024-09-17T04:01:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 --2- 172.21.15.129:0/1228588218 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0ccc80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 --2- 172.21.15.129:0/1228588218 >> v2:172.21.15.181:3300/0 conn(0x7fe17c0705a0 0x7fe17c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 -- 172.21.15.129:0/1228588218 >> 172.21.15.129:0/1228588218 conn(0x7fe17c06d410 msgr2=0x7fe17c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:34.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 -- 172.21.15.129:0/1228588218 shutdown_connections 2024-09-17T04:01:34.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.813+0000 7fe183000640 1 -- 172.21.15.129:0/1228588218 wait complete. 2024-09-17T04:01:34.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.815+0000 7fe183000640 1 Processor -- start 2024-09-17T04:01:34.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.815+0000 7fe183000640 1 -- start start 2024-09-17T04:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0705a0 0x7fe17c15a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c160200 con 0x7fe17c06f2e0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c1612e0 con 0x7fe17c0705a0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe183000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c0cd1c0 con 0x7fe17c06eed0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe17b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe17b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50480/0 (socket says 172.21.15.129:50480) 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe17b400640 1 -- 172.21.15.129:0/1708755065 learned_addr learned my addr 172.21.15.129:0/1708755065 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe17be00640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.817+0000 7fe181000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0705a0 0x7fe17c15a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 -- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 msgr2=0x7fe17c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 -- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0705a0 msgr2=0x7fe17c15a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0705a0 0x7fe17c15a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 -- 172.21.15.129:0/1708755065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0c7320 con 0x7fe17c06f2e0 2024-09-17T04:01:34.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17be00640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe17b400640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe16802ab00 tx=0x7fe168033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168031d60 con 0x7fe17c06f2e0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe168033e10 con 0x7fe17c06f2e0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c157970 con 0x7fe17c06f2e0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168005260 con 0x7fe17c06f2e0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.818+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c1f7140 con 0x7fe17c06f2e0 2024-09-17T04:01:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.820+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe17c1f6d20 con 0x7fe17c06f2e0 2024-09-17T04:01:34.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.820+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe168005400 con 0x7fe17c06f2e0 2024-09-17T04:01:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.821+0000 7fe173e00640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 0x7fe15c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.822+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7fe1680bbc40 con 0x7fe17c06f2e0 2024-09-17T04:01:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.822+0000 7fe17be00640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 0x7fe15c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.823+0000 7fe17be00640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 0x7fe15c07b0d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe16c005fd0 tx=0x7fe16c0095c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:34.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:34.826+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe16803f030 con 0x7fe17c06f2e0 2024-09-17T04:01:35.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.074+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]} v 0) -- 0x7fe17c0ccc80 con 0x7fe17c06f2e0 2024-09-17T04:01:35.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.685+0000 7fe173e00640 1 -- 172.21.15.129:0/1708755065 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]}]=0 v627) ==== 135+0+0 (secure 0 0 0) 0x7fe1680842f0 con 0x7fe17c06f2e0 2024-09-17T04:01:35.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 msgr2=0x7fe15c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:35.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 0x7fe15c07b0d0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe16c005fd0 tx=0x7fe16c0095c0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 msgr2=0x7fe17c157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:35.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe16802ab00 tx=0x7fe168033c20 comp rx=0 tx=0).stop 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 shutdown_connections 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe15c078bc0 0x7fe15c07b0d0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe17c0705a0 0x7fe17c15a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe17c06f2e0 0x7fe17c157430 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 --2- 172.21.15.129:0/1708755065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe17c06eed0 0x7fe17c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 >> 172.21.15.129:0/1708755065 conn(0x7fe17c06d410 msgr2=0x7fe17c0ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 shutdown_connections 2024-09-17T04:01:35.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.688+0000 7fe183000640 1 -- 172.21.15.129:0/1708755065 wait complete. 2024-09-17T04:01:35.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 -- 172.21.15.129:0/3429384695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9880cd370 msgr2=0x7fa9880b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 --2- 172.21.15.129:0/3429384695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9880cd370 0x7fa9880b4290 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fa97c005fd0 tx=0x7fa97c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 -- 172.21.15.129:0/3429384695 shutdown_connections 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 --2- 172.21.15.129:0/3429384695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9880cd370 0x7fa9880b4290 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 -- 172.21.15.129:0/3429384695 >> 172.21.15.129:0/3429384695 conn(0x7fa98806d410 msgr2=0x7fa9880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 -- 172.21.15.129:0/3429384695 shutdown_connections 2024-09-17T04:01:35.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.934+0000 7fa98f400640 1 -- 172.21.15.129:0/3429384695 wait complete. 2024-09-17T04:01:35.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.937+0000 7fa98f400640 1 Processor -- start 2024-09-17T04:01:35.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.937+0000 7fa98f400640 1 -- start start 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9880cd370 0x7fa98815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa988162d00 0x7fa98815b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa9880cb1e0 con 0x7fa98815bc20 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa988163110 con 0x7fa988162d00 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.938+0000 7fa98f400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa98815c910 con 0x7fa9880cd370 2024-09-17T04:01:35.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50494/0 (socket says 172.21.15.129:50494) 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 -- 172.21.15.129:0/658215777 learned_addr learned my addr 172.21.15.129:0/658215777 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa987e00640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa988162d00 0x7fa98815b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98ca00640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9880cd370 0x7fa98815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 -- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9880cd370 msgr2=0x7fa98815cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9880cd370 0x7fa98815cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 -- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa988162d00 msgr2=0x7fa98815b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa988162d00 0x7fa98815b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:35.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 -- 172.21.15.129:0/658215777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa988161db0 con 0x7fa98815bc20 2024-09-17T04:01:35.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.939+0000 7fa98d400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c6b0 tx=0x7fa98000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:35.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.940+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980003b70 con 0x7fa98815bc20 2024-09-17T04:01:35.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.940+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa980003d30 con 0x7fa98815bc20 2024-09-17T04:01:35.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.940+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980005330 con 0x7fa98815bc20 2024-09-17T04:01:35.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.940+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9880c93f0 con 0x7fa98815bc20 2024-09-17T04:01:35.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.940+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9880c99b0 con 0x7fa98815bc20 2024-09-17T04:01:35.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.941+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9880b4290 con 0x7fa98815bc20 2024-09-17T04:01:35.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.942+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa9800054d0 con 0x7fa98815bc20 2024-09-17T04:01:35.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.943+0000 7fa984c00640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 0x7fa96807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:35.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.943+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7fa98009a750 con 0x7fa98815bc20 2024-09-17T04:01:35.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.944+0000 7fa98ca00640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 0x7fa96807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:35.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.945+0000 7fa98ca00640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 0x7fa96807b0a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa97c033000 tx=0x7fa97c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:35.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:35.947+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa9800627a0 con 0x7fa98815bc20 2024-09-17T04:01:36.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.131+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fa98806a140 con 0x7fa98815bc20 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.132+0000 7fa984c00640 1 -- 172.21.15.129:0/658215777 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v59) ==== 59+0+2294 (secure 0 0 0) 0x7fa9800671d0 con 0x7fa98815bc20 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:01:36.132 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:01:36.133 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:01:36.134 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:01:36.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 msgr2=0x7fa96807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 0x7fa96807b0a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa97c033000 tx=0x7fa97c004730 comp rx=0 tx=0).stop 2024-09-17T04:01:36.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 msgr2=0x7fa98815c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c6b0 tx=0x7fa98000cb70 comp rx=0 tx=0).stop 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 shutdown_connections 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa968078b90 0x7fa96807b0a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa98815bc20 0x7fa98815c030 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa988162d00 0x7fa98815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 --2- 172.21.15.129:0/658215777 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9880cd370 0x7fa98815cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 >> 172.21.15.129:0/658215777 conn(0x7fa98806d410 msgr2=0x7fa988073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 shutdown_connections 2024-09-17T04:01:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.134+0000 7fa98f400640 1 -- 172.21.15.129:0/658215777 wait complete. 2024-09-17T04:01:36.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 -- 172.21.15.129:0/1694959046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de80cb080 msgr2=0x7f8de8072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 --2- 172.21.15.129:0/1694959046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de80cb080 0x7f8de8072e10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0009830 tx=0x7f8dd002a530 comp rx=0 tx=0).stop 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 -- 172.21.15.129:0/1694959046 shutdown_connections 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 --2- 172.21.15.129:0/1694959046 >> v2:172.21.15.181:3300/0 conn(0x7f8de8073350 0x7f8de80ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 --2- 172.21.15.129:0/1694959046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de80cb080 0x7f8de8072e10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.380+0000 7f8de7e00640 1 -- 172.21.15.129:0/1694959046 >> 172.21.15.129:0/1694959046 conn(0x7f8de80d00b0 msgr2=0x7f8de80d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.381+0000 7f8de7e00640 1 -- 172.21.15.129:0/1694959046 shutdown_connections 2024-09-17T04:01:36.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.381+0000 7f8de7e00640 1 -- 172.21.15.129:0/1694959046 wait complete. 2024-09-17T04:01:36.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.383+0000 7f8de7e00640 1 Processor -- start 2024-09-17T04:01:36.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.383+0000 7f8de7e00640 1 -- start start 2024-09-17T04:01:36.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8de8073350 0x7f8de8150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de815c7b0 0x7f8de80b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8de80ced50 con 0x7f8de80cb080 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8de80cebd0 con 0x7f8de8073350 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.384+0000 7f8de7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8de80ceed0 con 0x7f8de815c7b0 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50498/0 (socket says 172.21.15.129:50498) 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 -- 172.21.15.129:0/3706849509 learned_addr learned my addr 172.21.15.129:0/3706849509 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8de7400640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de815c7b0 0x7f8de80b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:36.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8de6a00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8de8073350 0x7f8de8150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 -- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de815c7b0 msgr2=0x7f8de80b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de815c7b0 0x7f8de80b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 -- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8de8073350 msgr2=0x7f8de8150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8de8073350 0x7f8de8150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 -- 172.21.15.129:0/3706849509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de814e9c0 con 0x7f8de80cb080 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.385+0000 7f8dde000640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f8dd002ab20 tx=0x7f8dd0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.386+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0031d60 con 0x7f8de80cb080 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.386+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8dd0033e10 con 0x7f8de80cb080 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.386+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de80b3500 con 0x7f8de80cb080 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.386+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd0005260 con 0x7f8de80cb080 2024-09-17T04:01:36.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.386+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de80b37c0 con 0x7f8de80cb080 2024-09-17T04:01:36.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.387+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8151610 con 0x7f8de80cb080 2024-09-17T04:01:36.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.388+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8dd0005400 con 0x7f8de80cb080 2024-09-17T04:01:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.389+0000 7f8ddfe00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 0x7f8dc407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.389+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7f8dd00bba20 con 0x7f8de80cb080 2024-09-17T04:01:36.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.390+0000 7f8de6a00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 0x7f8dc407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:36.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.391+0000 7f8de6a00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 0x7f8dc407b0a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8001720 tx=0x7f8dd8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:36.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.394+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8dd003f030 con 0x7f8de80cb080 2024-09-17T04:01:36.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.581+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8de80d0f50 con 0x7f8de80cb080 2024-09-17T04:01:36.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.585+0000 7f8ddfe00640 1 -- 172.21.15.129:0/3706849509 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v60)=0 v60) ==== 122+0+0 (secure 0 0 0) 0x7f8dd0083ba0 con 0x7f8de80cb080 2024-09-17T04:01:36.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 msgr2=0x7f8dc407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 0x7f8dc407b0a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8001720 tx=0x7f8dd8009040 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 msgr2=0x7f8de8151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f8dd002ab20 tx=0x7f8dd0033c20 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 shutdown_connections 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8dc4078b90 0x7f8dc407b0a0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8de815c7b0 0x7f8de80b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8de80cb080 0x7f8de8151010 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 --2- 172.21.15.129:0/3706849509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8de8073350 0x7f8de8150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 >> 172.21.15.129:0/3706849509 conn(0x7f8de80d00b0 msgr2=0x7f8de80b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:36.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 shutdown_connections 2024-09-17T04:01:36.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:36.588+0000 7f8de7e00640 1 -- 172.21.15.129:0/3706849509 wait complete. 2024-09-17T04:01:36.610 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:01:36.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:01:36.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:01:36.777 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:01:36.777 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:36.777 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:36.812 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:01:36.813 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:01:36.813 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:36.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:01:36.850 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:36.850 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:36.888 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:01:36.888 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:01:36.888 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:36.926 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:01:36.926 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:01:36.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:37.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 -- 172.21.15.129:0/59337769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 msgr2=0x7f51dc0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 --2- 172.21.15.129:0/59337769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc0ccc80 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f51dc072e00 tx=0x7f51c802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 -- 172.21.15.129:0/59337769 shutdown_connections 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 --2- 172.21.15.129:0/59337769 >> v2:172.21.15.181:3300/0 conn(0x7f51dc0cd1c0 0x7f51dc1609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 --2- 172.21.15.129:0/59337769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc0ccc80 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 -- 172.21.15.129:0/59337769 >> 172.21.15.129:0/59337769 conn(0x7f51dc06d410 msgr2=0x7f51dc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 -- 172.21.15.129:0/59337769 shutdown_connections 2024-09-17T04:01:37.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.147+0000 7f51e2e00640 1 -- 172.21.15.129:0/59337769 wait complete. 2024-09-17T04:01:37.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.150+0000 7f51e2e00640 1 Processor -- start 2024-09-17T04:01:37.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.150+0000 7f51e2e00640 1 -- start start 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0705a0 0x7f51dc15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f51dc0cd1c0 0x7f51dc1f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc160260 con 0x7f51dc0b4140 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc1600e0 con 0x7f51dc0705a0 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51e2e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f51dc160f00 con 0x7f51dc0cd1c0 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51db400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51db400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50524/0 (socket says 172.21.15.129:50524) 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.151+0000 7f51db400640 1 -- 172.21.15.129:0/1138053650 learned_addr learned my addr 172.21.15.129:0/1138053650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51dbe00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0705a0 0x7f51dc15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51e0e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f51dc0cd1c0 0x7f51dc1f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 -- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f51dc0cd1c0 msgr2=0x7f51dc1f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f51dc0cd1c0 0x7f51dc1f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 -- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0705a0 msgr2=0x7f51dc15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0705a0 0x7f51dc15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 -- 172.21.15.129:0/1138053650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc15bec0 con 0x7f51dc0b4140 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51db400640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f51c8000c00 tx=0x7f51c8003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.152+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c803e030 con 0x7f51dc0b4140 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.153+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f51c8003d10 con 0x7f51dc0b4140 2024-09-17T04:01:37.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.153+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51dc1586a0 con 0x7f51dc0b4140 2024-09-17T04:01:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.153+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8035040 con 0x7f51dc0b4140 2024-09-17T04:01:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.153+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51dc0c8c10 con 0x7f51dc0b4140 2024-09-17T04:01:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.154+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51dc0ccc80 con 0x7f51dc0b4140 2024-09-17T04:01:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.155+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f51c8041030 con 0x7f51dc0b4140 2024-09-17T04:01:37.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.156+0000 7f51d3e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 0x7f51bc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.156+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(177..177 src has 1..177) ==== 4943+0+0 (secure 0 0 0) 0x7f51c80bb270 con 0x7f51dc0b4140 2024-09-17T04:01:37.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.157+0000 7f51dbe00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 0x7f51bc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.157+0000 7f51dbe00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 0x7f51bc07b0d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f51cc001720 tx=0x7f51cc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:37.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.160+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f51c8083240 con 0x7f51dc0b4140 2024-09-17T04:01:37.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.377+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51dc06a140 con 0x7f51dc0b4140 2024-09-17T04:01:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.378+0000 7f51d3e00640 1 -- 172.21.15.129:0/1138053650 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+3925 (secure 0 0 0) 0x7f51c80b9e20 con 0x7f51dc0b4140 2024-09-17T04:01:37.378 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:37.378 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":156,"btime":"2024-09-17T04:01:30:325688+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7682,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/3265500655","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":3265500655},{"type":"v1","addr":"172.21.15.181:6836","nonce":3265500655}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":148},{"gid":7685,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2788512891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2788512891},{"type":"v1","addr":"172.21.15.181:6837","nonce":2788512891}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":149},{"gid":9483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2226211263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2226211263},{"type":"v1","addr":"172.21.15.129:6833","nonce":2226211263}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147},{"gid":9813,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/577436364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":577436364},{"type":"v1","addr":"172.21.15.129:6835","nonce":577436364}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":156}],"filesystems":[]} 2024-09-17T04:01:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 156 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 msgr2=0x7f51bc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 0x7f51bc07b0d0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f51cc001720 tx=0x7f51cc009040 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 msgr2=0x7f51dc1f52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f51c8000c00 tx=0x7f51c8003820 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 shutdown_connections 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f51bc078bc0 0x7f51bc07b0d0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f51dc0cd1c0 0x7f51dc1f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f51dc0b4140 0x7f51dc1f52b0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 --2- 172.21.15.129:0/1138053650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f51dc0705a0 0x7f51dc15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.380+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 >> 172.21.15.129:0/1138053650 conn(0x7f51dc06d410 msgr2=0x7f51dc0d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:37.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.381+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 shutdown_connections 2024-09-17T04:01:37.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.381+0000 7f51e2e00640 1 -- 172.21.15.129:0/1138053650 wait complete. 2024-09-17T04:01:37.402 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:01:37.403 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:01:37.403 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:01:37.403 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:37.403 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:01:37.403 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:01:37.403 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:01:37.403 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:37.404 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:01:37.404 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:01:37.404 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:01:37.404 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:37.404 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:01:37.404 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:01:37.405 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:01:37.405 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:37.429 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:01:37.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:01:37.431 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:01:37.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:01:37.431 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:01:37.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:01:37.432 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:01:37.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:01:37.474 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:01:37.474 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:37.475 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:01:37.475 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:37.477 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:01:37.477 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:37.477 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:01:37.478 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:37.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:01:37.672 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:01:37.672 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:01:37.673 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:01:37.673 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 -- 172.21.15.129:0/2913634901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 msgr2=0x7f29d00ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 --2- 172.21.15.129:0/2913634901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d00ccc80 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f29d0072e00 tx=0x7f29bc02a5f0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 -- 172.21.15.129:0/2913634901 shutdown_connections 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 --2- 172.21.15.129:0/2913634901 >> v2:172.21.15.181:3300/0 conn(0x7f29d00cd1c0 0x7f29d01609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 --2- 172.21.15.129:0/2913634901 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d00ccc80 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 -- 172.21.15.129:0/2913634901 >> 172.21.15.129:0/2913634901 conn(0x7f29d006d410 msgr2=0x7f29d00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 -- 172.21.15.129:0/2913634901 shutdown_connections 2024-09-17T04:01:37.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.710+0000 7f29d5600640 1 -- 172.21.15.129:0/2913634901 wait complete. 2024-09-17T04:01:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.713+0000 7f29d5600640 1 Processor -- start 2024-09-17T04:01:37.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.713+0000 7f29d5600640 1 -- start start 2024-09-17T04:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d00705a0 0x7f29d00cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29d00cd1c0 0x7f29d00d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29d0160260 con 0x7f29d00b4140 2024-09-17T04:01:37.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29d01600e0 con 0x7f29d00705a0 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.714+0000 7f29d5600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f29d0160f00 con 0x7f29d00cd1c0 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50574/0 (socket says 172.21.15.129:50574) 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 -- 172.21.15.129:0/3998577571 learned_addr learned my addr 172.21.15.129:0/3998577571 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29cf400640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29d00cd1c0 0x7f29d00d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29cea00640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d00705a0 0x7f29d00cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 -- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29d00cd1c0 msgr2=0x7f29d00d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29d00cd1c0 0x7f29d00d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 -- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d00705a0 msgr2=0x7f29d00cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d00705a0 0x7f29d00cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 -- 172.21.15.129:0/3998577571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d015dc10 con 0x7f29d00b4140 2024-09-17T04:01:37.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.715+0000 7f29ce000640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f29bc034040 tx=0x7f29bc003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:37.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.716+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc03e030 con 0x7f29d00b4140 2024-09-17T04:01:37.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.716+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d015c610 con 0x7f29d00b4140 2024-09-17T04:01:37.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.716+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d01f92a0 con 0x7f29d00b4140 2024-09-17T04:01:37.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.717+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d00ccc80 con 0x7f29d00b4140 2024-09-17T04:01:37.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.721+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29bc005310 con 0x7f29d00b4140 2024-09-17T04:01:37.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.721+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29bc048060 con 0x7f29d00b4140 2024-09-17T04:01:37.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.724+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f29bc041030 con 0x7f29d00b4140 2024-09-17T04:01:37.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.725+0000 7f29c6a00640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 0x7f29b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:37.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.725+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 4943+0+0 (secure 0 0 0) 0x7f29bc0bb5b0 con 0x7f29d00b4140 2024-09-17T04:01:37.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.726+0000 7f29cea00640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 0x7f29b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:37.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.726+0000 7f29cea00640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 0x7f29b007b0a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f29c0001720 tx=0x7f29c0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:37.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.728+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f29bc083680 con 0x7f29d00b4140 2024-09-17T04:01:37.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:37.923+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f29d006a140 con 0x7f29d00b4140 2024-09-17T04:01:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.747+0000 7f29c6a00640 1 -- 172.21.15.129:0/3998577571 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v182) ==== 97+0+0 (secure 0 0 0) 0x7f29bc0880b0 con 0x7f29d00b4140 2024-09-17T04:01:38.747 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.749+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 msgr2=0x7f29b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.749+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 0x7f29b007b0a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f29c0001720 tx=0x7f29c0009040 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 msgr2=0x7f29d015b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f29bc034040 tx=0x7f29bc003820 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 shutdown_connections 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29b0078b90 0x7f29b007b0a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29d00cd1c0 0x7f29d00d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29d00b4140 0x7f29d015b950 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 --2- 172.21.15.129:0/3998577571 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29d00705a0 0x7f29d00cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 >> 172.21.15.129:0/3998577571 conn(0x7f29d006d410 msgr2=0x7f29d0074970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:38.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 shutdown_connections 2024-09-17T04:01:38.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.750+0000 7f29d5600640 1 -- 172.21.15.129:0/3998577571 wait complete. 2024-09-17T04:01:38.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 -- 172.21.15.129:0/608191875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 msgr2=0x7f9dc80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 --2- 172.21.15.129:0/608191875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80ccc80 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009830 tx=0x7f9db002a5b0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 -- 172.21.15.129:0/608191875 shutdown_connections 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 --2- 172.21.15.129:0/608191875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80ccc80 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 -- 172.21.15.129:0/608191875 >> 172.21.15.129:0/608191875 conn(0x7f9dc806d410 msgr2=0x7f9dc80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.998+0000 7f9dcde00640 1 -- 172.21.15.129:0/608191875 shutdown_connections 2024-09-17T04:01:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:38.999+0000 7f9dcde00640 1 -- 172.21.15.129:0/608191875 wait complete. 2024-09-17T04:01:39.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.001+0000 7f9dcde00640 1 Processor -- start 2024-09-17T04:01:39.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.001+0000 7f9dcde00640 1 -- start start 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 0x7f9dc81f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc80cd1c0 0x7f9dc815a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc8160260 con 0x7f9dc80b4140 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc81600e0 con 0x7f9dc80cd1c0 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.002+0000 7f9dcde00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc81610d0 con 0x7f9dc80705a0 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50598/0 (socket says 172.21.15.129:50598) 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 -- 172.21.15.129:0/4005762402 learned_addr learned my addr 172.21.15.129:0/4005762402 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:39.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dc7400640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc80cd1c0 0x7f9dc815a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dc6a00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 0x7f9dc81f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 -- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 msgr2=0x7f9dc81f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 0x7f9dc81f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 -- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc80cd1c0 msgr2=0x7f9dc815a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc80cd1c0 0x7f9dc815a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 -- 172.21.15.129:0/4005762402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc81f6a00 con 0x7f9dc80b4140 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dc6a00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 0x7f9dc81f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.003+0000 7f9dbfe00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9db002aba0 tx=0x7f9db0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.004+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db0005c90 con 0x7f9dc80b4140 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.004+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9db0031e50 con 0x7f9dc80b4140 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.004+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db0005260 con 0x7f9dc80b4140 2024-09-17T04:01:39.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.004+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc80c3ec0 con 0x7f9dc80b4140 2024-09-17T04:01:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.004+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc80c5b70 con 0x7f9dc80b4140 2024-09-17T04:01:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.005+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc80ccc80 con 0x7f9dc80b4140 2024-09-17T04:01:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.006+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9db0005400 con 0x7f9dc80b4140 2024-09-17T04:01:39.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.008+0000 7f9dbf400640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 0x7f9da407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.008+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 4857+0+0 (secure 0 0 0) 0x7f9db00bbaf0 con 0x7f9dc80b4140 2024-09-17T04:01:39.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.008+0000 7f9dc6a00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 0x7f9da407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.009+0000 7f9dc6a00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 0x7f9da407b0a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9db8001120 tx=0x7f9db8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.012+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9dc81f6a00 con 0x7f9dc80b4140 2024-09-17T04:01:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.264+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f9dc806a140 con 0x7f9dc80b4140 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.265+0000 7f9dbf400640 1 -- 172.21.15.129:0/4005762402 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) ==== 80+0+5304 (secure 0 0 0) 0x7f9db003f030 con 0x7f9dc80b4140 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.265 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:01:39.266 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:01:39.268 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.272 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.274 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:01:39.276 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:01:39.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.278 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:01:39.279 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.280 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:01:39.281 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:01:39.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 msgr2=0x7f9da407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 0x7f9da407b0a0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9db8001120 tx=0x7f9db8009040 comp rx=0 tx=0).stop 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 msgr2=0x7f9dc80990c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f9db002aba0 tx=0x7f9db0033c20 comp rx=0 tx=0).stop 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 shutdown_connections 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9da4078b90 0x7f9da407b0a0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9dc80cd1c0 0x7f9dc815a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9dc80b4140 0x7f9dc80990c0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 --2- 172.21.15.129:0/4005762402 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9dc80705a0 0x7f9dc81f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.267+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 >> 172.21.15.129:0/4005762402 conn(0x7f9dc806d410 msgr2=0x7f9dc806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.268+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 shutdown_connections 2024-09-17T04:01:39.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.268+0000 7f9dcde00640 1 -- 172.21.15.129:0/4005762402 wait complete. 2024-09-17T04:01:39.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 -- 172.21.15.129:0/2899484985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e240b2950 msgr2=0x7f3e240cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 --2- 172.21.15.129:0/2899484985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e240b2950 0x7f3e240cbbf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c005fd0 tx=0x7f3e1c02a930 comp rx=0 tx=0).stop 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 -- 172.21.15.129:0/2899484985 shutdown_connections 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 --2- 172.21.15.129:0/2899484985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e240b2950 0x7f3e240cbbf0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 -- 172.21.15.129:0/2899484985 >> 172.21.15.129:0/2899484985 conn(0x7f3e2406d410 msgr2=0x7f3e240b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 -- 172.21.15.129:0/2899484985 shutdown_connections 2024-09-17T04:01:39.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.510+0000 7f3e2c000640 1 -- 172.21.15.129:0/2899484985 wait complete. 2024-09-17T04:01:39.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.512+0000 7f3e2c000640 1 Processor -- start 2024-09-17T04:01:39.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.512+0000 7f3e2c000640 1 -- start start 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.513+0000 7f3e2c000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.513+0000 7f3e2c000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e240ce170 0x7f3e241f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e2c000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e241f66d0 0x7f3e2415dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e2c000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2406f5e0 con 0x7f3e240b2950 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e2c000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3e24070910 con 0x7f3e241f66d0 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e2c000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3e2406f760 con 0x7f3e240ce170 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50614/0 (socket says 172.21.15.129:50614) 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 -- 172.21.15.129:0/3773030315 learned_addr learned my addr 172.21.15.129:0/3773030315 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:39.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e2a000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e241f66d0 0x7f3e2415dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e28c00640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e240ce170 0x7f3e241f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 -- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e240ce170 msgr2=0x7f3e241f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e240ce170 0x7f3e241f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 -- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e241f66d0 msgr2=0x7f3e2415dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e241f66d0 0x7f3e2415dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.514+0000 7f3e29600640 1 -- 172.21.15.129:0/3773030315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e241f8f30 con 0x7f3e240b2950 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e29600640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c3a0 tx=0x7f3e1800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18017070 con 0x7f3e240b2950 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e241f91d0 con 0x7f3e240b2950 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e180053c0 con 0x7f3e240b2950 2024-09-17T04:01:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18003850 con 0x7f3e240b2950 2024-09-17T04:01:39.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.515+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e241f9790 con 0x7f3e240b2950 2024-09-17T04:01:39.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.516+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e240cbbf0 con 0x7f3e240b2950 2024-09-17T04:01:39.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.517+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3e1801a070 con 0x7f3e240b2950 2024-09-17T04:01:39.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.518+0000 7f3e21600640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 0x7f3e0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.519+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 4857+0+0 (secure 0 0 0) 0x7f3e18099e20 con 0x7f3e240b2950 2024-09-17T04:01:39.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.519+0000 7f3e28c00640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 0x7f3e0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.520+0000 7f3e28c00640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 0x7f3e0407b0a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3e10007fd0 tx=0x7f3e10001390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.523+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3e18061e40 con 0x7f3e240b2950 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.710+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3e2406a140 con 0x7f3e240b2950 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.710+0000 7f3e21600640 1 -- 172.21.15.129:0/3773030315 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v182) ==== 84+0+65 (secure 0 0 0) 0x7f3e18066870 con 0x7f3e240b2950 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:01:39.710 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:01:39.711 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:01:39.711 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:01:39.711 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 msgr2=0x7f3e0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 0x7f3e0407b0a0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f3e10007fd0 tx=0x7f3e10001390 comp rx=0 tx=0).stop 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 msgr2=0x7f3e240c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c3a0 tx=0x7f3e1800c860 comp rx=0 tx=0).stop 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 shutdown_connections 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3e04078b90 0x7f3e0407b0a0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e241f66d0 0x7f3e2415dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3e240ce170 0x7f3e241f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 --2- 172.21.15.129:0/3773030315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3e240b2950 0x7f3e240c92c0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.712+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 >> 172.21.15.129:0/3773030315 conn(0x7f3e2406d410 msgr2=0x7f3e240af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.713+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 shutdown_connections 2024-09-17T04:01:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.713+0000 7f3e2c000640 1 -- 172.21.15.129:0/3773030315 wait complete. 2024-09-17T04:01:39.735 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:01:39.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 -- 172.21.15.129:0/3047656376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c80ae7f0 msgr2=0x7fc0c80cc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 --2- 172.21.15.129:0/3047656376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c80cc9f0 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8072710 tx=0x7fc0c002aa30 comp rx=0 tx=0).stop 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 -- 172.21.15.129:0/3047656376 shutdown_connections 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 --2- 172.21.15.129:0/3047656376 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c80cc9f0 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 --2- 172.21.15.129:0/3047656376 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c815f2f0 0x7fc0c80ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 --2- 172.21.15.129:0/3047656376 >> v2:172.21.15.181:3300/0 conn(0x7fc0c8155660 0x7fc0c815edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 -- 172.21.15.129:0/3047656376 >> 172.21.15.129:0/3047656376 conn(0x7fc0c80b1c50 msgr2=0x7fc0c80b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 -- 172.21.15.129:0/3047656376 shutdown_connections 2024-09-17T04:01:39.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.961+0000 7fc0cd200640 1 -- 172.21.15.129:0/3047656376 wait complete. 2024-09-17T04:01:39.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.964+0000 7fc0cd200640 1 Processor -- start 2024-09-17T04:01:39.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.964+0000 7fc0cd200640 1 -- start start 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c8154c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c8155660 0x7fc0c80c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80ce070 con 0x7fc0c815f2f0 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c80cd2b0 con 0x7fc0c80ae7f0 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0cd200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c80ce1f0 con 0x7fc0c8155660 2024-09-17T04:01:39.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0c7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50624/0 (socket says 172.21.15.129:50624) 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0c7e00640 1 -- 172.21.15.129:0/2944573982 learned_addr learned my addr 172.21.15.129:0/2944573982 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.965+0000 7fc0c7400640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c8154c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0bea00640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c8155660 0x7fc0c80c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 -- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c8155660 msgr2=0x7fc0c80c8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c8155660 0x7fc0c80c8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 -- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0c80ae7f0 msgr2=0x7fc0c8154c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c8154c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 -- 172.21.15.129:0/2944573982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c815cb90 con 0x7fc0c815f2f0 2024-09-17T04:01:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.966+0000 7fc0c7e00640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0005fd0 tx=0x7fc0c0003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.967+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0033050 con 0x7fc0c815f2f0 2024-09-17T04:01:39.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.967+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc0c0004450 con 0x7fc0c815f2f0 2024-09-17T04:01:39.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.967+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0004840 con 0x7fc0c815f2f0 2024-09-17T04:01:39.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.967+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c815ce70 con 0x7fc0c815f2f0 2024-09-17T04:01:39.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.967+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c815d430 con 0x7fc0c815f2f0 2024-09-17T04:01:39.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.968+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c80b38a0 con 0x7fc0c815f2f0 2024-09-17T04:01:39.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.969+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc0c0036070 con 0x7fc0c815f2f0 2024-09-17T04:01:39.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.970+0000 7fc0bfe00640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 0x7fc0a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:39.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.970+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 4857+0+0 (secure 0 0 0) 0x7fc0c00bbdd0 con 0x7fc0c815f2f0 2024-09-17T04:01:39.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.971+0000 7fc0c7400640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 0x7fc0a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:39.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.971+0000 7fc0c7400640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 0x7fc0a407b0d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8005fd0 tx=0x7fc0b800b260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:39.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:39.974+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc0c80b38a0 con 0x7fc0c815f2f0 2024-09-17T04:01:40.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.157+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc0c815edb0 con 0x7fc0c815f2f0 2024-09-17T04:01:40.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.723+0000 7fc0bfe00640 1 -- 172.21.15.129:0/2944573982 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v183) ==== 138+0+0 (secure 0 0 0) 0x7fc0c0083f00 con 0x7fc0c815f2f0 2024-09-17T04:01:40.723 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:01:40.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 msgr2=0x7fc0a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:40.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 0x7fc0a407b0d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8005fd0 tx=0x7fc0b800b260 comp rx=0 tx=0).stop 2024-09-17T04:01:40.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 msgr2=0x7fc0c815e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:40.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0005fd0 tx=0x7fc0c0003f10 comp rx=0 tx=0).stop 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 shutdown_connections 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc0a4078bc0 0x7fc0a407b0d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0c815f2f0 0x7fc0c815e1b0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0c8155660 0x7fc0c80c8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 --2- 172.21.15.129:0/2944573982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0c80ae7f0 0x7fc0c8154c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 >> 172.21.15.129:0/2944573982 conn(0x7fc0c80b1c50 msgr2=0x7fc0c8162330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 shutdown_connections 2024-09-17T04:01:40.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.727+0000 7fc0cd200640 1 -- 172.21.15.129:0/2944573982 wait complete. 2024-09-17T04:01:40.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:01:40.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 -- 172.21.15.129:0/1479029513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 msgr2=0x7fc9c00cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:40.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 --2- 172.21.15.129:0/1479029513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c00cf040 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fc9b8005fd0 tx=0x7fc9b802aaa0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 -- 172.21.15.129:0/1479029513 shutdown_connections 2024-09-17T04:01:40.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 --2- 172.21.15.129:0/1479029513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c00cf040 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 --2- 172.21.15.129:0/1479029513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d4dd0 0x7fc9c006dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 -- 172.21.15.129:0/1479029513 >> 172.21.15.129:0/1479029513 conn(0x7fc9c009da40 msgr2=0x7fc9c008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:40.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 -- 172.21.15.129:0/1479029513 shutdown_connections 2024-09-17T04:01:40.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.986+0000 7fc9c7800640 1 -- 172.21.15.129:0/1479029513 wait complete. 2024-09-17T04:01:40.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.988+0000 7fc9c7800640 1 Processor -- start 2024-09-17T04:01:40.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.988+0000 7fc9c7800640 1 -- start start 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d0c20 0x7fc9c00a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9c015f040 0x7fc9c015f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c00cfb80 con 0x7fc9c00b97d0 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc9c00cfa00 con 0x7fc9c015f040 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c7800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc9c00cfd00 con 0x7fc9c00d0c20 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:40.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50646/0 (socket says 172.21.15.129:50646) 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 -- 172.21.15.129:0/2470196466 learned_addr learned my addr 172.21.15.129:0/2470196466 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c5800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9c015f040 0x7fc9c015f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9b7e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d0c20 0x7fc9c00a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 -- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d0c20 msgr2=0x7fc9c00a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d0c20 0x7fc9c00a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 -- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9c015f040 msgr2=0x7fc9c015f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9c015f040 0x7fc9c015f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.990+0000 7fc9c4e00640 1 -- 172.21.15.129:0/2470196466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9c015f990 con 0x7fc9c00b97d0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9c4e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c3c0 tx=0x7fc9b000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0017070 con 0x7fc9c00b97d0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9b00053c0 con 0x7fc9c00b97d0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9b00038a0 con 0x7fc9c00b97d0 2024-09-17T04:01:40.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9c01f8df0 con 0x7fc9c00b97d0 2024-09-17T04:01:40.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.991+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9c01f9540 con 0x7fc9c00b97d0 2024-09-17T04:01:40.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.992+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9c00d01b0 con 0x7fc9c00b97d0 2024-09-17T04:01:40.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.994+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9b001a070 con 0x7fc9c00b97d0 2024-09-17T04:01:40.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.995+0000 7fc9bd600640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:40.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.995+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 5227+0+0 (secure 0 0 0) 0x7fc9b0099f70 con 0x7fc9c00b97d0 2024-09-17T04:01:40.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.995+0000 7fc9b7e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:40.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:40.996+0000 7fc9b7e00640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00b3bd0 tx=0x7fc9a800b390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:40.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.000+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc9c015f990 con 0x7fc9c00b97d0 2024-09-17T04:01:41.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.183+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc9c006a130 con 0x7fc9c00b97d0 2024-09-17T04:01:41.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.726+0000 7fc9bd600640 1 -- 172.21.15.129:0/2470196466 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v184) ==== 170+0+0 (secure 0 0 0) 0x7fc9b0061ea0 con 0x7fc9c00b97d0 2024-09-17T04:01:41.727 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:01:41.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 msgr2=0x7fc99c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00b3bd0 tx=0x7fc9a800b390 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 msgr2=0x7fc9c01eaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000c3c0 tx=0x7fc9b000c880 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 shutdown_connections 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc99c078bc0 0x7fc99c07b0d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9c015f040 0x7fc9c015f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9c00d0c20 0x7fc9c00a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 --2- 172.21.15.129:0/2470196466 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9c00b97d0 0x7fc9c01eaef0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.729+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 >> 172.21.15.129:0/2470196466 conn(0x7fc9c009da40 msgr2=0x7fc9c00b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.730+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 shutdown_connections 2024-09-17T04:01:41.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.730+0000 7fc9c7800640 1 -- 172.21.15.129:0/2470196466 wait complete. 2024-09-17T04:01:41.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.992+0000 7fef3ce00640 1 -- 172.21.15.129:0/3091588254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380c5b10 msgr2=0x7fef380cab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.992+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3091588254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380c5b10 0x7fef380cab20 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fef30005fd0 tx=0x7fef3002a970 comp rx=0 tx=0).stop 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 -- 172.21.15.129:0/3091588254 shutdown_connections 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3091588254 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380c5b10 0x7fef380cab20 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3091588254 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380c4fc0 0x7fef3815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 -- 172.21.15.129:0/3091588254 >> 172.21.15.129:0/3091588254 conn(0x7fef380d00b0 msgr2=0x7fef380d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 -- 172.21.15.129:0/3091588254 shutdown_connections 2024-09-17T04:01:41.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.993+0000 7fef3ce00640 1 -- 172.21.15.129:0/3091588254 wait complete. 2024-09-17T04:01:41.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.995+0000 7fef3ce00640 1 Processor -- start 2024-09-17T04:01:41.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.995+0000 7fef3ce00640 1 -- start start 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380ae3c0 0x7fef380b1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef380b1660 0x7fef38163990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fef38165970 con 0x7fef380ce2f0 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fef381657f0 con 0x7fef380b1660 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.996+0000 7fef3ce00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fef38165af0 con 0x7fef380ae3c0 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50676/0 (socket says 172.21.15.129:50676) 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 -- 172.21.15.129:0/3154251841 learned_addr learned my addr 172.21.15.129:0/3154251841 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef2ea00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380ae3c0 0x7fef380b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37e00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef380b1660 0x7fef38163990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 -- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380ae3c0 msgr2=0x7fef380b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380ae3c0 0x7fef380b1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 -- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef380b1660 msgr2=0x7fef38163990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef380b1660 0x7fef38163990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 -- 172.21.15.129:0/3154251841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef380b1a70 con 0x7fef380ce2f0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.997+0000 7fef37400640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fef2800d780 tx=0x7fef2800dc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.998+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28017070 con 0x7fef380ce2f0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.998+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fef2800de90 con 0x7fef380ce2f0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.998+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef38157b60 con 0x7fef380ce2f0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.998+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef280099d0 con 0x7fef380ce2f0 2024-09-17T04:01:41.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.998+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef38158120 con 0x7fef380ce2f0 2024-09-17T04:01:42.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:41.999+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef380c4fc0 con 0x7fef380ce2f0 2024-09-17T04:01:42.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.000+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fef28009b70 con 0x7fef380ce2f0 2024-09-17T04:01:42.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.001+0000 7fef2fe00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 0x7fef1407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:42.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.001+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(184..184 src has 1..184) ==== 5609+0+0 (secure 0 0 0) 0x7fef2809b570 con 0x7fef380ce2f0 2024-09-17T04:01:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.002+0000 7fef2ea00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 0x7fef1407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:42.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.002+0000 7fef2ea00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 0x7fef1407b0d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fef380afad0 tx=0x7fef20009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:42.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.005+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fef28020020 con 0x7fef380ce2f0 2024-09-17T04:01:42.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.222+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fef38166a00 con 0x7fef380ce2f0 2024-09-17T04:01:42.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.749+0000 7fef2fe00640 1 -- 172.21.15.129:0/3154251841 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 28 and data pool 29 v159) ==== 174+0+0 (secure 0 0 0) 0x7fef380b1a70 con 0x7fef380ce2f0 2024-09-17T04:01:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 28 and data pool 29 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.758+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 msgr2=0x7fef1407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.758+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 0x7fef1407b0d0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fef380afad0 tx=0x7fef20009040 comp rx=0 tx=0).stop 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.758+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 msgr2=0x7fef38155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.758+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fef2800d780 tx=0x7fef2800dc40 comp rx=0 tx=0).stop 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 shutdown_connections 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fef14078bc0 0x7fef1407b0d0 secure :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fef380afad0 tx=0x7fef20009040 comp rx=0 tx=0).stop 2024-09-17T04:01:42.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fef380b1660 0x7fef38163990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fef380ae3c0 0x7fef380b1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 --2- 172.21.15.129:0/3154251841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fef380ce2f0 0x7fef38155db0 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.759+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 >> 172.21.15.129:0/3154251841 conn(0x7fef380d00b0 msgr2=0x7fef380c51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.762+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 shutdown_connections 2024-09-17T04:01:42.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:42.763+0000 7fef3ce00640 1 -- 172.21.15.129:0/3154251841 wait complete. 2024-09-17T04:01:42.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 -- 172.21.15.129:0/2349085340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d440d0a50 msgr2=0x7f2d44160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 --2- 172.21.15.129:0/2349085340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d440d0a50 0x7f2d44160210 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f2d441463d0 tx=0x7f2d2c02a580 comp rx=0 tx=0).stop 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 -- 172.21.15.129:0/2349085340 shutdown_connections 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 --2- 172.21.15.129:0/2349085340 >> v2:172.21.15.181:3300/0 conn(0x7f2d441458a0 0x7f2d44160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 --2- 172.21.15.129:0/2349085340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d440d0a50 0x7f2d44160210 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 -- 172.21.15.129:0/2349085340 >> 172.21.15.129:0/2349085340 conn(0x7f2d4406cfc0 msgr2=0x7f2d4406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:43.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 -- 172.21.15.129:0/2349085340 shutdown_connections 2024-09-17T04:01:43.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.020+0000 7f2d49c00640 1 -- 172.21.15.129:0/2349085340 wait complete. 2024-09-17T04:01:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.022+0000 7f2d49c00640 1 Processor -- start 2024-09-17T04:01:43.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.022+0000 7f2d49c00640 1 -- start start 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d44145e60 0x7f2d441330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d44133630 0x7f2d44133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440cef20 con 0x7f2d441458a0 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2d440ceda0 con 0x7f2d44133630 2024-09-17T04:01:43.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.023+0000 7f2d49c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2d44160c90 con 0x7f2d44145e60 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50680/0 (socket says 172.21.15.129:50680) 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 -- 172.21.15.129:0/1370687173 learned_addr learned my addr 172.21.15.129:0/1370687173 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d3be00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d44145e60 0x7f2d441330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d49200640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d44133630 0x7f2d44133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 -- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d44145e60 msgr2=0x7f2d441330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d44145e60 0x7f2d441330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 -- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d44133630 msgr2=0x7f2d44133a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d44133630 0x7f2d44133a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.024+0000 7f2d43e00640 1 -- 172.21.15.129:0/1370687173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d440cbcd0 con 0x7f2d441458a0 2024-09-17T04:01:43.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d43e00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400c3a0 tx=0x7f2d3400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34017070 con 0x7f2d441458a0 2024-09-17T04:01:43.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d340053c0 con 0x7f2d441458a0 2024-09-17T04:01:43.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d440c5120 con 0x7f2d441458a0 2024-09-17T04:01:43.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d34003850 con 0x7f2d441458a0 2024-09-17T04:01:43.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.025+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d44134230 con 0x7f2d441458a0 2024-09-17T04:01:43.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.026+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d44160210 con 0x7f2d441458a0 2024-09-17T04:01:43.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.027+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2d3401a070 con 0x7f2d441458a0 2024-09-17T04:01:43.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.028+0000 7f2d40c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.028+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5709+0+0 (secure 0 0 0) 0x7f2d3409a060 con 0x7f2d441458a0 2024-09-17T04:01:43.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.029+0000 7f2d3be00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.029+0000 7f2d3be00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c033000 tx=0x7f2d2c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.032+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2d34061e40 con 0x7f2d441458a0 2024-09-17T04:01:43.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.219+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d4406a140 con 0x7f2d441458a0 2024-09-17T04:01:43.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.221+0000 7f2d40c00640 1 -- 172.21.15.129:0/1370687173 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) ==== 74+0+16800 (secure 0 0 0) 0x7f2d34066870 con 0x7f2d441458a0 2024-09-17T04:01:43.221 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:43.221 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":185,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:01:42.223998+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:01:40.170274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-17T04:01:41.196845+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:43.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 msgr2=0x7f2d2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c033000 tx=0x7f2d2c004730 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 msgr2=0x7f2d4415f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f2d3400c3a0 tx=0x7f2d3400c860 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 shutdown_connections 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2d20078b90 0x7f2d2007b0a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2d44133630 0x7f2d44133a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2d44145e60 0x7f2d441330f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 --2- 172.21.15.129:0/1370687173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2d441458a0 0x7f2d4415f8a0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.223+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 >> 172.21.15.129:0/1370687173 conn(0x7f2d4406cfc0 msgr2=0x7f2d440b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.224+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 shutdown_connections 2024-09-17T04:01:43.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.224+0000 7f2d49c00640 1 -- 172.21.15.129:0/1370687173 wait complete. 2024-09-17T04:01:43.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 -- 172.21.15.129:0/1496547518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100cd390 msgr2=0x7fac101609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 --2- 172.21.15.129:0/1496547518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100cd390 0x7fac101609c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7fac08005fd0 tx=0x7fac0802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 -- 172.21.15.129:0/1496547518 shutdown_connections 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 --2- 172.21.15.129:0/1496547518 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100cd390 0x7fac101609c0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 --2- 172.21.15.129:0/1496547518 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac100705a0 0x7fac100b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 -- 172.21.15.129:0/1496547518 >> 172.21.15.129:0/1496547518 conn(0x7fac1006d410 msgr2=0x7fac100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 -- 172.21.15.129:0/1496547518 shutdown_connections 2024-09-17T04:01:43.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.465+0000 7fac15200640 1 -- 172.21.15.129:0/1496547518 wait complete. 2024-09-17T04:01:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.468+0000 7fac15200640 1 Processor -- start 2024-09-17T04:01:43.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.468+0000 7fac15200640 1 -- start start 2024-09-17T04:01:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac1006eed0 0x7fac1015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 0x7fac10074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac10160f00 con 0x7fac100705a0 2024-09-17T04:01:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac1006f4f0 con 0x7fac1015ad10 2024-09-17T04:01:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac15200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fac10161080 con 0x7fac1006eed0 2024-09-17T04:01:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac0d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac0d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:50698/0 (socket says 172.21.15.129:50698) 2024-09-17T04:01:43.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.469+0000 7fac0d400640 1 -- 172.21.15.129:0/1542149645 learned_addr learned my addr 172.21.15.129:0/1542149645 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:43.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0e800640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 0x7fac10074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0de00640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac1006eed0 0x7fac1015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 -- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac1006eed0 msgr2=0x7fac1015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac1006eed0 0x7fac1015a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 -- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 msgr2=0x7fac10074310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 0x7fac10074310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 -- 172.21.15.129:0/1542149645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1015b570 con 0x7fac100705a0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0e800640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 0x7fac10074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac0d400640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007fd0 tx=0x7fabfc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.470+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011790 con 0x7fac100705a0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.471+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabfc017070 con 0x7fac100705a0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.471+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc011d00 con 0x7fac100705a0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.471+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1015ab10 con 0x7fac100705a0 2024-09-17T04:01:43.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.471+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac100c9610 con 0x7fac100705a0 2024-09-17T04:01:43.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.472+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac100c8e00 con 0x7fac100705a0 2024-09-17T04:01:43.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.473+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fabfc005020 con 0x7fac100705a0 2024-09-17T04:01:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.474+0000 7fac06000640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 0x7fabf007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.474+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(185..185 src has 1..185) ==== 5709+0+0 (secure 0 0 0) 0x7fabfc09b4e0 con 0x7fac100705a0 2024-09-17T04:01:43.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.475+0000 7fac0de00640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 0x7fabf007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.475+0000 7fac0de00640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 0x7fabf007b0d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fac00009ac0 tx=0x7fac00005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.477+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fac1015ab10 con 0x7fac100705a0 2024-09-17T04:01:43.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.660+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac100b4140 con 0x7fac100705a0 2024-09-17T04:01:43.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.661+0000 7fac06000640 1 -- 172.21.15.129:0/1542149645 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v185) ==== 74+0+16800 (secure 0 0 0) 0x7fabfc063a20 con 0x7fac100705a0 2024-09-17T04:01:43.662 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:43.662 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":185,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:01:42.223998+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:01:40.170274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-17T04:01:41.196845+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 msgr2=0x7fabf007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 0x7fabf007b0d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fac00009ac0 tx=0x7fac00005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 msgr2=0x7fac10073dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7fabfc007fd0 tx=0x7fabfc00c820 comp rx=0 tx=0).stop 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 shutdown_connections 2024-09-17T04:01:43.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabf0078bc0 0x7fabf007b0d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac1015ad10 0x7fac10074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac100705a0 0x7fac10073dd0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 --2- 172.21.15.129:0/1542149645 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac1006eed0 0x7fac1015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 >> 172.21.15.129:0/1542149645 conn(0x7fac1006d410 msgr2=0x7fac100ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 shutdown_connections 2024-09-17T04:01:43.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.664+0000 7fac15200640 1 -- 172.21.15.129:0/1542149645 wait complete. 2024-09-17T04:01:43.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 -- 172.21.15.129:0/1145359577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c06eea0 msgr2=0x7fe74c06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 --2- 172.21.15.129:0/1145359577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c06eea0 0x7fe74c06f2b0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7fe74c069b60 tx=0x7fe74402aa10 comp rx=0 tx=0).stop 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 -- 172.21.15.129:0/1145359577 shutdown_connections 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 --2- 172.21.15.129:0/1145359577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c06eea0 0x7fe74c06f2b0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 --2- 172.21.15.129:0/1145359577 >> v2:172.21.15.181:3300/0 conn(0x7fe74c0b1c40 0x7fe74c0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 --2- 172.21.15.129:0/1145359577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c074800 0x7fe74c0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 -- 172.21.15.129:0/1145359577 >> 172.21.15.129:0/1145359577 conn(0x7fe74c0ccc40 msgr2=0x7fe74c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 -- 172.21.15.129:0/1145359577 shutdown_connections 2024-09-17T04:01:43.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.912+0000 7fe751000640 1 -- 172.21.15.129:0/1145359577 wait complete. 2024-09-17T04:01:43.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.915+0000 7fe751000640 1 Processor -- start 2024-09-17T04:01:43.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.915+0000 7fe751000640 1 -- start start 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe74c06eea0 0x7fe74c150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c0b1c40 0x7fe74c15af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c06f970 con 0x7fe74c074800 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe74c06f7f0 con 0x7fe74c06eea0 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe751000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe74c06faf0 con 0x7fe74c0b1c40 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe749400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe749400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44462/0 (socket says 172.21.15.129:44462) 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.916+0000 7fe749400640 1 -- 172.21.15.129:0/3499920447 learned_addr learned my addr 172.21.15.129:0/3499920447 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749e00640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe74c06eea0 0x7fe74c150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe74a800640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c0b1c40 0x7fe74c15af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 -- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c0b1c40 msgr2=0x7fe74c15af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c0b1c40 0x7fe74c15af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 -- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe74c06eea0 msgr2=0x7fe74c150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe74c06eea0 0x7fe74c150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 -- 172.21.15.129:0/3499920447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74c157a10 con 0x7fe74c074800 2024-09-17T04:01:43.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe749400640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d680 tx=0x7fe73800db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.917+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738017070 con 0x7fe74c074800 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.918+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe738009b10 con 0x7fe74c074800 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.918+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74c157cf0 con 0x7fe74c074800 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.918+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738010460 con 0x7fe74c074800 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.918+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74c1582b0 con 0x7fe74c074800 2024-09-17T04:01:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.919+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe74c159ae0 con 0x7fe74c074800 2024-09-17T04:01:43.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.920+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe73801a070 con 0x7fe74c074800 2024-09-17T04:01:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.921+0000 7fe742000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 0x7fe72c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.921+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fe73809b200 con 0x7fe74c074800 2024-09-17T04:01:43.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.922+0000 7fe749e00640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 0x7fe72c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:43.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.923+0000 7fe749e00640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 0x7fe72c07b0d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe73c004040 tx=0x7fe73c00f6f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:43.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:43.925+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe738062fd0 con 0x7fe74c074800 2024-09-17T04:01:44.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.143+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe74c0af480 con 0x7fe74c074800 2024-09-17T04:01:44.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.776+0000 7fe742000640 1 -- 172.21.15.129:0/3499920447 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v162) ==== 118+0+0 (secure 0 0 0) 0x7fe738067a00 con 0x7fe74c074800 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 msgr2=0x7fe72c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 0x7fe72c07b0d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe73c004040 tx=0x7fe73c00f6f0 comp rx=0 tx=0).stop 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 msgr2=0x7fe74c159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7fe73800d680 tx=0x7fe73800db40 comp rx=0 tx=0).stop 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 shutdown_connections 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe72c078bc0 0x7fe72c07b0d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:44.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe74c0b1c40 0x7fe74c15af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe74c074800 0x7fe74c159400 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 --2- 172.21.15.129:0/3499920447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe74c06eea0 0x7fe74c150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 >> 172.21.15.129:0/3499920447 conn(0x7fe74c0ccc40 msgr2=0x7fe74c0739a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 shutdown_connections 2024-09-17T04:01:44.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:44.779+0000 7fe751000640 1 -- 172.21.15.129:0/3499920447 wait complete. 2024-09-17T04:01:44.806 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:01:44.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.031+0000 7f70c9600640 1 -- 172.21.15.129:0/1753045721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 msgr2=0x7f70c40b4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.031+0000 7f70c9600640 1 --2- 172.21.15.129:0/1753045721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c40b4630 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f70c40d1290 tx=0x7f70bc02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 -- 172.21.15.129:0/1753045721 shutdown_connections 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 --2- 172.21.15.129:0/1753045721 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c40b4630 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 --2- 172.21.15.129:0/1753045721 >> v2:172.21.15.181:3300/0 conn(0x7f70c40b2650 0x7f70c40cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 -- 172.21.15.129:0/1753045721 >> 172.21.15.129:0/1753045721 conn(0x7f70c40ccc40 msgr2=0x7f70c40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 -- 172.21.15.129:0/1753045721 shutdown_connections 2024-09-17T04:01:45.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.032+0000 7f70c9600640 1 -- 172.21.15.129:0/1753045721 wait complete. 2024-09-17T04:01:45.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.034+0000 7f70c9600640 1 Processor -- start 2024-09-17T04:01:45.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.034+0000 7f70c9600640 1 -- start start 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70c40b2650 0x7f70c40c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70c40ce500 0x7f70c40c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40cab70 con 0x7f70c40b4220 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40ca9f0 con 0x7f70c40b2650 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.035+0000 7f70c9600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70c40cacf0 con 0x7f70c40ce500 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70c40b2650 0x7f70c40c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44474/0 (socket says 172.21.15.129:44474) 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 -- 172.21.15.129:0/655406412 learned_addr learned my addr 172.21.15.129:0/655406412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:45.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c3400640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70c40ce500 0x7f70c40c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 -- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70c40ce500 msgr2=0x7f70c40c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70c40ce500 0x7f70c40c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 -- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70c40b2650 msgr2=0x7f70c40c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70c40b2650 0x7f70c40c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.036+0000 7f70c2000640 1 -- 172.21.15.129:0/655406412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c41e99b0 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70c2000640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f70b000c3a0 tx=0x7f70b000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b0011790 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70b0017070 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c41e9c70 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b0011cc0 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.037+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c41e9f50 con 0x7f70c40b4220 2024-09-17T04:01:45.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.038+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c40cb1c0 con 0x7f70c40b4220 2024-09-17T04:01:45.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.039+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70b001c460 con 0x7f70c40b4220 2024-09-17T04:01:45.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.040+0000 7f70baa00640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 0x7f70a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:45.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.040+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f70b009aa20 con 0x7f70c40b4220 2024-09-17T04:01:45.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.041+0000 7f70c2a00640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 0x7f70a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:45.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.041+0000 7f70c2a00640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 0x7f70a407b0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f70c40726b0 tx=0x7f70b400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:45.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.045+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f70c40b4b70 con 0x7f70c40b4220 2024-09-17T04:01:45.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.262+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f70c409e1c0 con 0x7f70c40b4220 2024-09-17T04:01:45.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.778+0000 7f70baa00640 1 -- 172.21.15.129:0/655406412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v163) ==== 122+0+0 (secure 0 0 0) 0x7f70b0062780 con 0x7f70c40b4220 2024-09-17T04:01:45.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 msgr2=0x7f70a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:45.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 0x7f70a407b0d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f70c40726b0 tx=0x7f70b400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 msgr2=0x7f70c415a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f70b000c3a0 tx=0x7f70b000c860 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 shutdown_connections 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70a4078bc0 0x7f70a407b0d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70c40ce500 0x7f70c40c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70c40b4220 0x7f70c415a600 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 --2- 172.21.15.129:0/655406412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70c40b2650 0x7f70c40c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 >> 172.21.15.129:0/655406412 conn(0x7f70c40ccc40 msgr2=0x7f70c40ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:45.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.780+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 shutdown_connections 2024-09-17T04:01:45.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:45.781+0000 7f70c9600640 1 -- 172.21.15.129:0/655406412 wait complete. 2024-09-17T04:01:45.807 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:01:45.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:46.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 -- 172.21.15.129:0/2568481843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 msgr2=0x7f3db40b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 --2- 172.21.15.129:0/2568481843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db40b3440 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f3db40b41b0 tx=0x7f3da402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 -- 172.21.15.129:0/2568481843 shutdown_connections 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 --2- 172.21.15.129:0/2568481843 >> v2:172.21.15.181:3300/0 conn(0x7f3db40d4ba0 0x7f3db40d3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 --2- 172.21.15.129:0/2568481843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db40b3440 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 -- 172.21.15.129:0/2568481843 >> 172.21.15.129:0/2568481843 conn(0x7f3db408bff0 msgr2=0x7f3db409d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 -- 172.21.15.129:0/2568481843 shutdown_connections 2024-09-17T04:01:46.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.027+0000 7f3db3e00640 1 -- 172.21.15.129:0/2568481843 wait complete. 2024-09-17T04:01:46.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.029+0000 7f3db3e00640 1 Processor -- start 2024-09-17T04:01:46.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.029+0000 7f3db3e00640 1 -- start start 2024-09-17T04:01:46.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 0x7f3db4151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 0x7f3db41f93d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40cf440 con 0x7f3db40d1850 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40cf2c0 con 0x7f3db40d4ba0 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.030+0000 7f3db3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3db40cf5c0 con 0x7f3db40b9af0 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44506/0 (socket says 172.21.15.129:44506) 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 -- 172.21.15.129:0/3807257602 learned_addr learned my addr 172.21.15.129:0/3807257602 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2a00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 0x7f3db4151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 -- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 msgr2=0x7f3db4151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db3400640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 0x7f3db41f93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 0x7f3db4151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 -- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 msgr2=0x7f3db41f93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 0x7f3db41f93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 -- 172.21.15.129:0/3807257602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db40cd6f0 con 0x7f3db40d1850 2024-09-17T04:01:46.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2a00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 0x7f3db4151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db3400640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 0x7f3db41f93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.031+0000 7f3db2000640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f3da000c3d0 tx=0x7f3da000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.032+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0017070 con 0x7f3db40d1850 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.032+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3da00053c0 con 0x7f3db40d1850 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.032+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db40cd9d0 con 0x7f3db40d1850 2024-09-17T04:01:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.032+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0003850 con 0x7f3db40d1850 2024-09-17T04:01:46.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.032+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db40cdf90 con 0x7f3db40d1850 2024-09-17T04:01:46.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.033+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db40b3440 con 0x7f3db40d1850 2024-09-17T04:01:46.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.034+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3da001a070 con 0x7f3db40d1850 2024-09-17T04:01:46.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.036+0000 7f3daaa00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 0x7f3d9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.036+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f3da009a400 con 0x7f3db40d1850 2024-09-17T04:01:46.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.037+0000 7f3db2a00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 0x7f3d9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.037+0000 7f3db2a00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 0x7f3d9407b0a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f3da4033000 tx=0x7f3da4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.039+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3da0021020 con 0x7f3db40d1850 2024-09-17T04:01:46.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.256+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3db40b4300 con 0x7f3db40d1850 2024-09-17T04:01:46.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.257+0000 7f3daaa00640 1 -- 172.21.15.129:0/3807257602 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5245 (secure 0 0 0) 0x7f3da0062930 con 0x7f3db40d1850 2024-09-17T04:01:46.257 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:46.257 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":163,"btime":"2024-09-17T04:01:45:777482+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2944792499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2944792499},{"type":"v1","addr":"172.21.15.181:6837","nonce":2944792499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158},{"gid":9205,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1569162292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1569162292},{"type":"v1","addr":"172.21.15.181:6836","nonce":1569162292}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157},{"gid":9208,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3011518213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3011518213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3011518213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:01:45.263805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9903},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9903":{"gid":9903,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3531032601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3531032601},{"type":"v1","addr":"172.21.15.129:6835","nonce":3531032601}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9903,"qdb_cluster":[9903]},"id":14}]} 2024-09-17T04:01:46.257 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 163 2024-09-17T04:01:46.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 msgr2=0x7f3d9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 0x7f3d9407b0a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f3da4033000 tx=0x7f3da4004730 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 msgr2=0x7f3db41f8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f3da000c3d0 tx=0x7f3da000c890 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 shutdown_connections 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d94078b90 0x7f3d9407b0a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db40d4ba0 0x7f3db41f93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3db40d1850 0x7f3db41f8e90 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 --2- 172.21.15.129:0/3807257602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3db40b9af0 0x7f3db4151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 >> 172.21.15.129:0/3807257602 conn(0x7f3db408bff0 msgr2=0x7f3db4161e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 shutdown_connections 2024-09-17T04:01:46.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.259+0000 7f3db3e00640 1 -- 172.21.15.129:0/3807257602 wait complete. 2024-09-17T04:01:46.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:01:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.501+0000 7f38f8800640 1 -- 172.21.15.129:0/3733202791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f0074730 msgr2=0x7f38f00cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 --2- 172.21.15.129:0/3733202791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f0074730 0x7f38f00cc590 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f38f00514f0 tx=0x7f38ec02aa30 comp rx=0 tx=0).stop 2024-09-17T04:01:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 -- 172.21.15.129:0/3733202791 shutdown_connections 2024-09-17T04:01:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 --2- 172.21.15.129:0/3733202791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f0074730 0x7f38f00cc590 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 --2- 172.21.15.129:0/3733202791 >> v2:172.21.15.181:3300/0 conn(0x7f38f0068d70 0x7f38f00b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 -- 172.21.15.129:0/3733202791 >> 172.21.15.129:0/3733202791 conn(0x7f38f00d0a70 msgr2=0x7f38f00d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:46.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 -- 172.21.15.129:0/3733202791 shutdown_connections 2024-09-17T04:01:46.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.502+0000 7f38f8800640 1 -- 172.21.15.129:0/3733202791 wait complete. 2024-09-17T04:01:46.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.504+0000 7f38f8800640 1 Processor -- start 2024-09-17T04:01:46.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.504+0000 7f38f8800640 1 -- start start 2024-09-17T04:01:46.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 0x7f38f01f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38f0074730 0x7f38f014f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f38f00cce60 con 0x7f38f015c950 2024-09-17T04:01:46.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f38f00ccce0 con 0x7f38f0074730 2024-09-17T04:01:46.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f8800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f38f00ccfe0 con 0x7f38f0068d70 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44512/0 (socket says 172.21.15.129:44512) 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 -- 172.21.15.129:0/813622412 learned_addr learned my addr 172.21.15.129:0/813622412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f5400640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38f0074730 0x7f38f014f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f5e00640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 0x7f38f01f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 -- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 msgr2=0x7f38f01f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 0x7f38f01f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 -- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38f0074730 msgr2=0x7f38f014f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38f0074730 0x7f38f014f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f6800640 1 -- 172.21.15.129:0/813622412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f00c5170 con 0x7f38f015c950 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.506+0000 7f38f5e00640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 0x7f38f01f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38f6800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009d60 tx=0x7f38ec003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec033050 con 0x7f38f015c950 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38ec004450 con 0x7f38f015c950 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f00c5450 con 0x7f38f015c950 2024-09-17T04:01:46.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ec038450 con 0x7f38f015c950 2024-09-17T04:01:46.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.507+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f00c5a10 con 0x7f38f015c950 2024-09-17T04:01:46.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.508+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38f00c4ee0 con 0x7f38f015c950 2024-09-17T04:01:46.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.510+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f38ec036070 con 0x7f38f015c950 2024-09-17T04:01:46.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.511+0000 7f38ea000640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 0x7f38d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.511+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f38ec0bc260 con 0x7f38f015c950 2024-09-17T04:01:46.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.511+0000 7f38f5e00640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 0x7f38d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.512+0000 7f38f5e00640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 0x7f38d407b0a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f38e4005fd0 tx=0x7f38e400f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.515+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f38ec084600 con 0x7f38f015c950 2024-09-17T04:01:46.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.703+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38f00ce290 con 0x7f38f015c950 2024-09-17T04:01:46.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.704+0000 7f38ea000640 1 -- 172.21.15.129:0/813622412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v186) ==== 74+0+16787 (secure 0 0 0) 0x7f38ec083f00 con 0x7f38f015c950 2024-09-17T04:01:46.704 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:46.705 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":186,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:01:43.750299+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:01:40.170274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-17T04:01:41.196845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.706+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 msgr2=0x7f38d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.706+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 0x7f38d407b0a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f38e4005fd0 tx=0x7f38e400f360 comp rx=0 tx=0).stop 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.706+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 msgr2=0x7f38f014fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.706+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f38ec009d60 tx=0x7f38ec003f10 comp rx=0 tx=0).stop 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 shutdown_connections 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f38d4078b90 0x7f38d407b0a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f38f015c950 0x7f38f014fc70 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38f0074730 0x7f38f014f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 --2- 172.21.15.129:0/813622412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f38f0068d70 0x7f38f01f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 >> 172.21.15.129:0/813622412 conn(0x7f38f00d0a70 msgr2=0x7f38f0075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 shutdown_connections 2024-09-17T04:01:46.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.707+0000 7f38f8800640 1 -- 172.21.15.129:0/813622412 wait complete. 2024-09-17T04:01:46.733 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:01:46.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 -- 172.21.15.129:0/2288853550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 msgr2=0x7f051000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 --2- 172.21.15.129:0/2288853550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f051000ba20 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f052815dd20 tx=0x7f0510005830 comp rx=0 tx=0).stop 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 -- 172.21.15.129:0/2288853550 shutdown_connections 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 --2- 172.21.15.129:0/2288853550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f051000ba20 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 -- 172.21.15.129:0/2288853550 >> 172.21.15.129:0/2288853550 conn(0x7f052806cfc0 msgr2=0x7f052806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 -- 172.21.15.129:0/2288853550 shutdown_connections 2024-09-17T04:01:46.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.954+0000 7f052e800640 1 -- 172.21.15.129:0/2288853550 wait complete. 2024-09-17T04:01:46.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.957+0000 7f052e800640 1 Processor -- start 2024-09-17T04:01:46.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.957+0000 7f052e800640 1 -- start start 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 0x7f0528157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0528157ba0 0x7f05281446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f05280b46e0 con 0x7f05100053a0 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f05280d0520 con 0x7f0528157ba0 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.958+0000 7f052e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f05280b4860 con 0x7f0528157250 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44524/0 (socket says 172.21.15.129:44524) 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 -- 172.21.15.129:0/3851161996 learned_addr learned my addr 172.21.15.129:0/3851161996 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052ca00640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 0x7f0528157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052de00640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0528157ba0 0x7f05281446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 -- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 msgr2=0x7f0528157660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 0x7f0528157660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 -- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0528157ba0 msgr2=0x7f05281446c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0528157ba0 0x7f05281446c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 -- 172.21.15.129:0/3851161996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05281f7630 con 0x7f05100053a0 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052ca00640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 0x7f0528157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.959+0000 7f052d400640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f051c004db0 tx=0x7f051c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.960+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c001d40 con 0x7f05100053a0 2024-09-17T04:01:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.960+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f051c016070 con 0x7f05100053a0 2024-09-17T04:01:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.960+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c010b40 con 0x7f05100053a0 2024-09-17T04:01:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.960+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05281f78d0 con 0x7f05100053a0 2024-09-17T04:01:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.960+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f052815c230 con 0x7f05100053a0 2024-09-17T04:01:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.961+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05280b4dc0 con 0x7f05100053a0 2024-09-17T04:01:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.962+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f051c010ce0 con 0x7f05100053a0 2024-09-17T04:01:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.963+0000 7f0525600640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 0x7f050807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.963+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f051c09b430 con 0x7f05100053a0 2024-09-17T04:01:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.964+0000 7f052ca00640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 0x7f050807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.964+0000 7f052ca00640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 0x7f050807b0d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0528156040 tx=0x7f0514005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:46.968+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f051c063860 con 0x7f05100053a0 2024-09-17T04:01:47.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.147+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0528142370 con 0x7f0508078bc0 2024-09-17T04:01:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.153+0000 7f0525600640 1 -- 172.21.15.129:0/3851161996 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f0514013ba0 con 0x7f0508078bc0 2024-09-17T04:01:47.156 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:01:47.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 msgr2=0x7f050807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 0x7f050807b0d0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0528156040 tx=0x7f0514005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 msgr2=0x7f05281f7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f051c004db0 tx=0x7f051c001630 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 shutdown_connections 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0508078bc0 0x7f050807b0d0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0528157ba0 0x7f05281446c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0528157250 0x7f0528157660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 --2- 172.21.15.129:0/3851161996 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f05100053a0 0x7f05281f7020 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 >> 172.21.15.129:0/3851161996 conn(0x7f052806cfc0 msgr2=0x7f052815cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 shutdown_connections 2024-09-17T04:01:47.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.157+0000 7f052e800640 1 -- 172.21.15.129:0/3851161996 wait complete. 2024-09-17T04:01:47.181 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-17T04:01:46.502079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331412,"kb_used_data":19484,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754643308,"statfs":{"total":773094113280,"available":772754747392,"internally_reserved":0,"allocated":19951616,"data_stored":9375213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.771528"},"pg_stats":[{"pgid":"29.3f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767884+0000","last_change":"2024-09-17T04:01:42.754485+0000","last_active":"2024-09-17T04:01:43.767884+0000","last_peered":"2024-09-17T04:01:43.767884+0000","last_clean":"2024-09-17T04:01:43.767884+0000","last_became_active":"2024-09-17T04:01:42.753255+0000","last_became_peered":"2024-09-17T04:01:42.753255+0000","last_unstale":"2024-09-17T04:01:43.767884+0000","last_undegraded":"2024-09-17T04:01:43.767884+0000","last_fullsized":"2024-09-17T04:01:43.767884+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:03.067830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928770+0000","last_change":"2024-09-17T04:01:42.785451+0000","last_active":"2024-09-17T04:01:43.928770+0000","last_peered":"2024-09-17T04:01:43.928770+0000","last_clean":"2024-09-17T04:01:43.928770+0000","last_became_active":"2024-09-17T04:01:42.784265+0000","last_became_peered":"2024-09-17T04:01:42.784265+0000","last_unstale":"2024-09-17T04:01:43.928770+0000","last_undegraded":"2024-09-17T04:01:43.928770+0000","last_fullsized":"2024-09-17T04:01:43.928770+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:55.439278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780763+0000","last_change":"2024-09-17T04:01:42.770419+0000","last_active":"2024-09-17T04:01:42.780763+0000","last_peered":"2024-09-17T04:01:42.780763+0000","last_clean":"2024-09-17T04:01:42.780763+0000","last_became_active":"2024-09-17T04:01:42.769275+0000","last_became_peered":"2024-09-17T04:01:42.769275+0000","last_unstale":"2024-09-17T04:01:42.780763+0000","last_undegraded":"2024-09-17T04:01:42.780763+0000","last_fullsized":"2024-09-17T04:01:42.780763+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:05.207595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929357+0000","last_change":"2024-09-17T04:01:42.759946+0000","last_active":"2024-09-17T04:01:43.929357+0000","last_peered":"2024-09-17T04:01:43.929357+0000","last_clean":"2024-09-17T04:01:43.929357+0000","last_became_active":"2024-09-17T04:01:42.759614+0000","last_became_peered":"2024-09-17T04:01:42.759614+0000","last_unstale":"2024-09-17T04:01:43.929357+0000","last_undegraded":"2024-09-17T04:01:43.929357+0000","last_fullsized":"2024-09-17T04:01:43.929357+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:39.034951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928536+0000","last_change":"2024-09-17T04:01:42.761604+0000","last_active":"2024-09-17T04:01:43.928536+0000","last_peered":"2024-09-17T04:01:43.928536+0000","last_clean":"2024-09-17T04:01:43.928536+0000","last_became_active":"2024-09-17T04:01:42.761303+0000","last_became_peered":"2024-09-17T04:01:42.761303+0000","last_unstale":"2024-09-17T04:01:43.928536+0000","last_undegraded":"2024-09-17T04:01:43.928536+0000","last_fullsized":"2024-09-17T04:01:43.928536+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:50.550720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929117+0000","last_change":"2024-09-17T04:01:42.762544+0000","last_active":"2024-09-17T04:01:43.929117+0000","last_peered":"2024-09-17T04:01:43.929117+0000","last_clean":"2024-09-17T04:01:43.929117+0000","last_became_active":"2024-09-17T04:01:42.762157+0000","last_became_peered":"2024-09-17T04:01:42.762157+0000","last_unstale":"2024-09-17T04:01:43.929117+0000","last_undegraded":"2024-09-17T04:01:43.929117+0000","last_fullsized":"2024-09-17T04:01:43.929117+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:10.939599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781454+0000","last_change":"2024-09-17T04:01:42.767972+0000","last_active":"2024-09-17T04:01:42.781454+0000","last_peered":"2024-09-17T04:01:42.781454+0000","last_clean":"2024-09-17T04:01:42.781454+0000","last_became_active":"2024-09-17T04:01:42.767668+0000","last_became_peered":"2024-09-17T04:01:42.767668+0000","last_unstale":"2024-09-17T04:01:42.781454+0000","last_undegraded":"2024-09-17T04:01:42.781454+0000","last_fullsized":"2024-09-17T04:01:42.781454+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:34.770623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763311+0000","last_change":"2024-09-17T04:01:42.770322+0000","last_active":"2024-09-17T04:01:43.763311+0000","last_peered":"2024-09-17T04:01:43.763311+0000","last_clean":"2024-09-17T04:01:43.763311+0000","last_became_active":"2024-09-17T04:01:42.770035+0000","last_became_peered":"2024-09-17T04:01:42.770035+0000","last_unstale":"2024-09-17T04:01:43.763311+0000","last_undegraded":"2024-09-17T04:01:43.763311+0000","last_fullsized":"2024-09-17T04:01:43.763311+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:49.056085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786891+0000","last_change":"2024-09-17T04:01:42.780240+0000","last_active":"2024-09-17T04:01:42.786891+0000","last_peered":"2024-09-17T04:01:42.786891+0000","last_clean":"2024-09-17T04:01:42.786891+0000","last_became_active":"2024-09-17T04:01:42.779875+0000","last_became_peered":"2024-09-17T04:01:42.779875+0000","last_unstale":"2024-09-17T04:01:42.786891+0000","last_undegraded":"2024-09-17T04:01:42.786891+0000","last_fullsized":"2024-09-17T04:01:42.786891+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:19.775171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929819+0000","last_change":"2024-09-17T04:01:42.762434+0000","last_active":"2024-09-17T04:01:43.929819+0000","last_peered":"2024-09-17T04:01:43.929819+0000","last_clean":"2024-09-17T04:01:43.929819+0000","last_became_active":"2024-09-17T04:01:42.762102+0000","last_became_peered":"2024-09-17T04:01:42.762102+0000","last_unstale":"2024-09-17T04:01:43.929819+0000","last_undegraded":"2024-09-17T04:01:43.929819+0000","last_fullsized":"2024-09-17T04:01:43.929819+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:01:29.965573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767827+0000","last_change":"2024-09-17T04:01:42.756123+0000","last_active":"2024-09-17T04:01:43.767827+0000","last_peered":"2024-09-17T04:01:43.767827+0000","last_clean":"2024-09-17T04:01:43.767827+0000","last_became_active":"2024-09-17T04:01:42.755416+0000","last_became_peered":"2024-09-17T04:01:42.755416+0000","last_unstale":"2024-09-17T04:01:43.767827+0000","last_undegraded":"2024-09-17T04:01:43.767827+0000","last_fullsized":"2024-09-17T04:01:43.767827+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:30.705447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928702+0000","last_change":"2024-09-17T04:01:42.766800+0000","last_active":"2024-09-17T04:01:43.928702+0000","last_peered":"2024-09-17T04:01:43.928702+0000","last_clean":"2024-09-17T04:01:43.928702+0000","last_became_active":"2024-09-17T04:01:42.766479+0000","last_became_peered":"2024-09-17T04:01:42.766479+0000","last_unstale":"2024-09-17T04:01:43.928702+0000","last_undegraded":"2024-09-17T04:01:43.928702+0000","last_fullsized":"2024-09-17T04:01:43.928702+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:40.521166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928928+0000","last_change":"2024-09-17T04:01:42.769768+0000","last_active":"2024-09-17T04:01:43.928928+0000","last_peered":"2024-09-17T04:01:43.928928+0000","last_clean":"2024-09-17T04:01:43.928928+0000","last_became_active":"2024-09-17T04:01:42.769426+0000","last_became_peered":"2024-09-17T04:01:42.769426+0000","last_unstale":"2024-09-17T04:01:43.928928+0000","last_undegraded":"2024-09-17T04:01:43.928928+0000","last_fullsized":"2024-09-17T04:01:43.928928+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:18.702585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763693+0000","last_change":"2024-09-17T04:01:42.766746+0000","last_active":"2024-09-17T04:01:43.763693+0000","last_peered":"2024-09-17T04:01:43.763693+0000","last_clean":"2024-09-17T04:01:43.763693+0000","last_became_active":"2024-09-17T04:01:42.762258+0000","last_became_peered":"2024-09-17T04:01:42.762258+0000","last_unstale":"2024-09-17T04:01:43.763693+0000","last_undegraded":"2024-09-17T04:01:43.763693+0000","last_fullsized":"2024-09-17T04:01:43.763693+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:17.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.767630+0000","last_change":"2024-09-17T04:01:42.765061+0000","last_active":"2024-09-17T04:01:42.767630+0000","last_peered":"2024-09-17T04:01:42.767630+0000","last_clean":"2024-09-17T04:01:42.767630+0000","last_became_active":"2024-09-17T04:01:42.764781+0000","last_became_peered":"2024-09-17T04:01:42.764781+0000","last_unstale":"2024-09-17T04:01:42.767630+0000","last_undegraded":"2024-09-17T04:01:42.767630+0000","last_fullsized":"2024-09-17T04:01:42.767630+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:31.784422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928755+0000","last_change":"2024-09-17T04:01:42.753912+0000","last_active":"2024-09-17T04:01:43.928755+0000","last_peered":"2024-09-17T04:01:43.928755+0000","last_clean":"2024-09-17T04:01:43.928755+0000","last_became_active":"2024-09-17T04:01:42.753606+0000","last_became_peered":"2024-09-17T04:01:42.753606+0000","last_unstale":"2024-09-17T04:01:43.928755+0000","last_undegraded":"2024-09-17T04:01:43.928755+0000","last_fullsized":"2024-09-17T04:01:43.928755+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:52.188705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764105+0000","last_change":"2024-09-17T04:01:42.768700+0000","last_active":"2024-09-17T04:01:43.764105+0000","last_peered":"2024-09-17T04:01:43.764105+0000","last_clean":"2024-09-17T04:01:43.764105+0000","last_became_active":"2024-09-17T04:01:42.768338+0000","last_became_peered":"2024-09-17T04:01:42.768338+0000","last_unstale":"2024-09-17T04:01:43.764105+0000","last_undegraded":"2024-09-17T04:01:43.764105+0000","last_fullsized":"2024-09-17T04:01:43.764105+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:14.198048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.930035+0000","last_change":"2024-09-17T04:01:42.770307+0000","last_active":"2024-09-17T04:01:43.930035+0000","last_peered":"2024-09-17T04:01:43.930035+0000","last_clean":"2024-09-17T04:01:43.930035+0000","last_became_active":"2024-09-17T04:01:42.769936+0000","last_became_peered":"2024-09-17T04:01:42.769936+0000","last_unstale":"2024-09-17T04:01:43.930035+0000","last_undegraded":"2024-09-17T04:01:43.930035+0000","last_fullsized":"2024-09-17T04:01:43.930035+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:04.899511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929793+0000","last_change":"2024-09-17T04:01:42.755567+0000","last_active":"2024-09-17T04:01:43.929793+0000","last_peered":"2024-09-17T04:01:43.929793+0000","last_clean":"2024-09-17T04:01:43.929793+0000","last_became_active":"2024-09-17T04:01:42.755244+0000","last_became_peered":"2024-09-17T04:01:42.755244+0000","last_unstale":"2024-09-17T04:01:43.929793+0000","last_undegraded":"2024-09-17T04:01:43.929793+0000","last_fullsized":"2024-09-17T04:01:43.929793+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:10.408327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929419+0000","last_change":"2024-09-17T04:01:42.754625+0000","last_active":"2024-09-17T04:01:43.929419+0000","last_peered":"2024-09-17T04:01:43.929419+0000","last_clean":"2024-09-17T04:01:43.929419+0000","last_became_active":"2024-09-17T04:01:42.754277+0000","last_became_peered":"2024-09-17T04:01:42.754277+0000","last_unstale":"2024-09-17T04:01:43.929419+0000","last_undegraded":"2024-09-17T04:01:43.929419+0000","last_fullsized":"2024-09-17T04:01:43.929419+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:56.783108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928758+0000","last_change":"2024-09-17T04:01:42.755662+0000","last_active":"2024-09-17T04:01:43.928758+0000","last_peered":"2024-09-17T04:01:43.928758+0000","last_clean":"2024-09-17T04:01:43.928758+0000","last_became_active":"2024-09-17T04:01:42.755139+0000","last_became_peered":"2024-09-17T04:01:42.755139+0000","last_unstale":"2024-09-17T04:01:43.928758+0000","last_undegraded":"2024-09-17T04:01:43.928758+0000","last_fullsized":"2024-09-17T04:01:43.928758+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:26.887685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780855+0000","last_change":"2024-09-17T04:01:42.776413+0000","last_active":"2024-09-17T04:01:42.780855+0000","last_peered":"2024-09-17T04:01:42.780855+0000","last_clean":"2024-09-17T04:01:42.780855+0000","last_became_active":"2024-09-17T04:01:42.776083+0000","last_became_peered":"2024-09-17T04:01:42.776083+0000","last_unstale":"2024-09-17T04:01:42.780855+0000","last_undegraded":"2024-09-17T04:01:42.780855+0000","last_fullsized":"2024-09-17T04:01:42.780855+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:28.497472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767357+0000","last_change":"2024-09-17T04:01:42.763674+0000","last_active":"2024-09-17T04:01:43.767357+0000","last_peered":"2024-09-17T04:01:43.767357+0000","last_clean":"2024-09-17T04:01:43.767357+0000","last_became_active":"2024-09-17T04:01:42.760549+0000","last_became_peered":"2024-09-17T04:01:42.760549+0000","last_unstale":"2024-09-17T04:01:43.767357+0000","last_undegraded":"2024-09-17T04:01:43.767357+0000","last_fullsized":"2024-09-17T04:01:43.767357+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:27.528607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928276+0000","last_change":"2024-09-17T04:01:42.760393+0000","last_active":"2024-09-17T04:01:43.928276+0000","last_peered":"2024-09-17T04:01:43.928276+0000","last_clean":"2024-09-17T04:01:43.928276+0000","last_became_active":"2024-09-17T04:01:42.759501+0000","last_became_peered":"2024-09-17T04:01:42.759501+0000","last_unstale":"2024-09-17T04:01:43.928276+0000","last_undegraded":"2024-09-17T04:01:43.928276+0000","last_fullsized":"2024-09-17T04:01:43.928276+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:19:32.416747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768010+0000","last_change":"2024-09-17T04:01:42.760598+0000","last_active":"2024-09-17T04:01:43.768010+0000","last_peered":"2024-09-17T04:01:43.768010+0000","last_clean":"2024-09-17T04:01:43.768010+0000","last_became_active":"2024-09-17T04:01:42.759775+0000","last_became_peered":"2024-09-17T04:01:42.759775+0000","last_unstale":"2024-09-17T04:01:43.768010+0000","last_undegraded":"2024-09-17T04:01:43.768010+0000","last_fullsized":"2024-09-17T04:01:43.768010+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:14.625621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763650+0000","last_change":"2024-09-17T04:01:42.778451+0000","last_active":"2024-09-17T04:01:43.763650+0000","last_peered":"2024-09-17T04:01:43.763650+0000","last_clean":"2024-09-17T04:01:43.763650+0000","last_became_active":"2024-09-17T04:01:42.777195+0000","last_became_peered":"2024-09-17T04:01:42.777195+0000","last_unstale":"2024-09-17T04:01:43.763650+0000","last_undegraded":"2024-09-17T04:01:43.763650+0000","last_fullsized":"2024-09-17T04:01:43.763650+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:56.698474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768305+0000","last_change":"2024-09-17T04:01:42.756470+0000","last_active":"2024-09-17T04:01:43.768305+0000","last_peered":"2024-09-17T04:01:43.768305+0000","last_clean":"2024-09-17T04:01:43.768305+0000","last_became_active":"2024-09-17T04:01:42.755821+0000","last_became_peered":"2024-09-17T04:01:42.755821+0000","last_unstale":"2024-09-17T04:01:43.768305+0000","last_undegraded":"2024-09-17T04:01:43.768305+0000","last_fullsized":"2024-09-17T04:01:43.768305+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:56.165743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929499+0000","last_change":"2024-09-17T04:01:42.753795+0000","last_active":"2024-09-17T04:01:43.929499+0000","last_peered":"2024-09-17T04:01:43.929499+0000","last_clean":"2024-09-17T04:01:43.929499+0000","last_became_active":"2024-09-17T04:01:42.753070+0000","last_became_peered":"2024-09-17T04:01:42.753070+0000","last_unstale":"2024-09-17T04:01:43.929499+0000","last_undegraded":"2024-09-17T04:01:43.929499+0000","last_fullsized":"2024-09-17T04:01:43.929499+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:57.895981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763766+0000","last_change":"2024-09-17T04:01:42.774411+0000","last_active":"2024-09-17T04:01:43.763766+0000","last_peered":"2024-09-17T04:01:43.763766+0000","last_clean":"2024-09-17T04:01:43.763766+0000","last_became_active":"2024-09-17T04:01:42.774092+0000","last_became_peered":"2024-09-17T04:01:42.774092+0000","last_unstale":"2024-09-17T04:01:43.763766+0000","last_undegraded":"2024-09-17T04:01:43.763766+0000","last_fullsized":"2024-09-17T04:01:43.763766+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:56.461318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767490+0000","last_change":"2024-09-17T04:01:42.771279+0000","last_active":"2024-09-17T04:01:43.767490+0000","last_peered":"2024-09-17T04:01:43.767490+0000","last_clean":"2024-09-17T04:01:43.767490+0000","last_became_active":"2024-09-17T04:01:42.770882+0000","last_became_peered":"2024-09-17T04:01:42.770882+0000","last_unstale":"2024-09-17T04:01:43.767490+0000","last_undegraded":"2024-09-17T04:01:43.767490+0000","last_fullsized":"2024-09-17T04:01:43.767490+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:34.277187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928408+0000","last_change":"2024-09-17T04:01:42.761573+0000","last_active":"2024-09-17T04:01:43.928408+0000","last_peered":"2024-09-17T04:01:43.928408+0000","last_clean":"2024-09-17T04:01:43.928408+0000","last_became_active":"2024-09-17T04:01:42.760888+0000","last_became_peered":"2024-09-17T04:01:42.760888+0000","last_unstale":"2024-09-17T04:01:43.928408+0000","last_undegraded":"2024-09-17T04:01:43.928408+0000","last_fullsized":"2024-09-17T04:01:43.928408+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:44.756271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.771583+0000","last_change":"2024-09-17T04:01:42.770126+0000","last_active":"2024-09-17T04:01:42.771583+0000","last_peered":"2024-09-17T04:01:42.771583+0000","last_clean":"2024-09-17T04:01:42.771583+0000","last_became_active":"2024-09-17T04:01:42.769786+0000","last_became_peered":"2024-09-17T04:01:42.769786+0000","last_unstale":"2024-09-17T04:01:42.771583+0000","last_undegraded":"2024-09-17T04:01:42.771583+0000","last_fullsized":"2024-09-17T04:01:42.771583+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:57.455421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786818+0000","last_change":"2024-09-17T04:01:42.781821+0000","last_active":"2024-09-17T04:01:42.786818+0000","last_peered":"2024-09-17T04:01:42.786818+0000","last_clean":"2024-09-17T04:01:42.786818+0000","last_became_active":"2024-09-17T04:01:42.777720+0000","last_became_peered":"2024-09-17T04:01:42.777720+0000","last_unstale":"2024-09-17T04:01:42.786818+0000","last_undegraded":"2024-09-17T04:01:42.786818+0000","last_fullsized":"2024-09-17T04:01:42.786818+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:25.256917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"185'4","reported_seq":17,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785859+0000","last_change":"2024-09-17T04:01:41.746448+0000","last_active":"2024-09-17T04:01:42.785859+0000","last_peered":"2024-09-17T04:01:42.785859+0000","last_clean":"2024-09-17T04:01:42.785859+0000","last_became_active":"2024-09-17T04:01:41.746146+0000","last_became_peered":"2024-09-17T04:01:41.746146+0000","last_unstale":"2024-09-17T04:01:42.785859+0000","last_undegraded":"2024-09-17T04:01:42.785859+0000","last_fullsized":"2024-09-17T04:01:42.785859+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:36.991555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763802+0000","last_change":"2024-09-17T04:01:42.762618+0000","last_active":"2024-09-17T04:01:43.763802+0000","last_peered":"2024-09-17T04:01:43.763802+0000","last_clean":"2024-09-17T04:01:43.763802+0000","last_became_active":"2024-09-17T04:01:42.762264+0000","last_became_peered":"2024-09-17T04:01:42.762264+0000","last_unstale":"2024-09-17T04:01:43.763802+0000","last_undegraded":"2024-09-17T04:01:43.763802+0000","last_fullsized":"2024-09-17T04:01:43.763802+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:02.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928880+0000","last_change":"2024-09-17T04:01:41.753761+0000","last_active":"2024-09-17T04:01:43.928880+0000","last_peered":"2024-09-17T04:01:43.928880+0000","last_clean":"2024-09-17T04:01:43.928880+0000","last_became_active":"2024-09-17T04:01:41.753432+0000","last_became_peered":"2024-09-17T04:01:41.753432+0000","last_unstale":"2024-09-17T04:01:43.928880+0000","last_undegraded":"2024-09-17T04:01:43.928880+0000","last_fullsized":"2024-09-17T04:01:43.928880+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:15.229671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764110+0000","last_change":"2024-09-17T04:01:42.781189+0000","last_active":"2024-09-17T04:01:43.764110+0000","last_peered":"2024-09-17T04:01:43.764110+0000","last_clean":"2024-09-17T04:01:43.764110+0000","last_became_active":"2024-09-17T04:01:42.780868+0000","last_became_peered":"2024-09-17T04:01:42.780868+0000","last_unstale":"2024-09-17T04:01:43.764110+0000","last_undegraded":"2024-09-17T04:01:43.764110+0000","last_fullsized":"2024-09-17T04:01:43.764110+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:07.494776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1c","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928977+0000","last_change":"2024-09-17T04:01:41.742401+0000","last_active":"2024-09-17T04:01:43.928977+0000","last_peered":"2024-09-17T04:01:43.928977+0000","last_clean":"2024-09-17T04:01:43.928977+0000","last_became_active":"2024-09-17T04:01:41.741938+0000","last_became_peered":"2024-09-17T04:01:41.741938+0000","last_unstale":"2024-09-17T04:01:43.928977+0000","last_undegraded":"2024-09-17T04:01:43.928977+0000","last_fullsized":"2024-09-17T04:01:43.928977+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:13.191449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767602+0000","last_change":"2024-09-17T04:01:42.778848+0000","last_active":"2024-09-17T04:01:43.767602+0000","last_peered":"2024-09-17T04:01:43.767602+0000","last_clean":"2024-09-17T04:01:43.767602+0000","last_became_active":"2024-09-17T04:01:42.778550+0000","last_became_peered":"2024-09-17T04:01:42.778550+0000","last_unstale":"2024-09-17T04:01:43.767602+0000","last_undegraded":"2024-09-17T04:01:43.767602+0000","last_fullsized":"2024-09-17T04:01:43.767602+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:16.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763677+0000","last_change":"2024-09-17T04:01:41.753169+0000","last_active":"2024-09-17T04:01:43.763677+0000","last_peered":"2024-09-17T04:01:43.763677+0000","last_clean":"2024-09-17T04:01:43.763677+0000","last_became_active":"2024-09-17T04:01:41.752374+0000","last_became_peered":"2024-09-17T04:01:41.752374+0000","last_unstale":"2024-09-17T04:01:43.763677+0000","last_undegraded":"2024-09-17T04:01:43.763677+0000","last_fullsized":"2024-09-17T04:01:43.763677+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:32.043936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928863+0000","last_change":"2024-09-17T04:01:42.766186+0000","last_active":"2024-09-17T04:01:43.928863+0000","last_peered":"2024-09-17T04:01:43.928863+0000","last_clean":"2024-09-17T04:01:43.928863+0000","last_became_active":"2024-09-17T04:01:42.765826+0000","last_became_peered":"2024-09-17T04:01:42.765826+0000","last_unstale":"2024-09-17T04:01:43.928863+0000","last_undegraded":"2024-09-17T04:01:43.928863+0000","last_fullsized":"2024-09-17T04:01:43.928863+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:29.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1a","version":"185'1","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785954+0000","last_change":"2024-09-17T04:01:41.756151+0000","last_active":"2024-09-17T04:01:42.785954+0000","last_peered":"2024-09-17T04:01:42.785954+0000","last_clean":"2024-09-17T04:01:42.785954+0000","last_became_active":"2024-09-17T04:01:41.754993+0000","last_became_peered":"2024-09-17T04:01:41.754993+0000","last_unstale":"2024-09-17T04:01:42.785954+0000","last_undegraded":"2024-09-17T04:01:42.785954+0000","last_fullsized":"2024-09-17T04:01:42.785954+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:37.786276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928622+0000","last_change":"2024-09-17T04:01:42.752218+0000","last_active":"2024-09-17T04:01:43.928622+0000","last_peered":"2024-09-17T04:01:43.928622+0000","last_clean":"2024-09-17T04:01:43.928622+0000","last_became_active":"2024-09-17T04:01:42.751907+0000","last_became_peered":"2024-09-17T04:01:42.751907+0000","last_unstale":"2024-09-17T04:01:43.928622+0000","last_undegraded":"2024-09-17T04:01:43.928622+0000","last_fullsized":"2024-09-17T04:01:43.928622+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:28.826108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763349+0000","last_change":"2024-09-17T04:01:41.743886+0000","last_active":"2024-09-17T04:01:43.763349+0000","last_peered":"2024-09-17T04:01:43.763349+0000","last_clean":"2024-09-17T04:01:43.763349+0000","last_became_active":"2024-09-17T04:01:41.743545+0000","last_became_peered":"2024-09-17T04:01:41.743545+0000","last_unstale":"2024-09-17T04:01:43.763349+0000","last_undegraded":"2024-09-17T04:01:43.763349+0000","last_fullsized":"2024-09-17T04:01:43.763349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:06.274197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929114+0000","last_change":"2024-09-17T04:01:42.766977+0000","last_active":"2024-09-17T04:01:43.929114+0000","last_peered":"2024-09-17T04:01:43.929114+0000","last_clean":"2024-09-17T04:01:43.929114+0000","last_became_active":"2024-09-17T04:01:42.766577+0000","last_became_peered":"2024-09-17T04:01:42.766577+0000","last_unstale":"2024-09-17T04:01:43.929114+0000","last_undegraded":"2024-09-17T04:01:43.929114+0000","last_fullsized":"2024-09-17T04:01:43.929114+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:49.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928747+0000","last_change":"2024-09-17T04:01:41.740632+0000","last_active":"2024-09-17T04:01:43.928747+0000","last_peered":"2024-09-17T04:01:43.928747+0000","last_clean":"2024-09-17T04:01:43.928747+0000","last_became_active":"2024-09-17T04:01:41.739898+0000","last_became_peered":"2024-09-17T04:01:41.739898+0000","last_unstale":"2024-09-17T04:01:43.928747+0000","last_undegraded":"2024-09-17T04:01:43.928747+0000","last_fullsized":"2024-09-17T04:01:43.928747+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:05.721314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929007+0000","last_change":"2024-09-17T04:01:42.784572+0000","last_active":"2024-09-17T04:01:43.929007+0000","last_peered":"2024-09-17T04:01:43.929007+0000","last_clean":"2024-09-17T04:01:43.929007+0000","last_became_active":"2024-09-17T04:01:42.783908+0000","last_became_peered":"2024-09-17T04:01:42.783908+0000","last_unstale":"2024-09-17T04:01:43.929007+0000","last_undegraded":"2024-09-17T04:01:43.929007+0000","last_fullsized":"2024-09-17T04:01:43.929007+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.656123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768194+0000","last_change":"2024-09-17T04:01:41.757459+0000","last_active":"2024-09-17T04:01:43.768194+0000","last_peered":"2024-09-17T04:01:43.768194+0000","last_clean":"2024-09-17T04:01:43.768194+0000","last_became_active":"2024-09-17T04:01:41.756976+0000","last_became_peered":"2024-09-17T04:01:41.756976+0000","last_unstale":"2024-09-17T04:01:43.768194+0000","last_undegraded":"2024-09-17T04:01:43.768194+0000","last_fullsized":"2024-09-17T04:01:43.768194+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:12.122874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768157+0000","last_change":"2024-09-17T04:01:42.771577+0000","last_active":"2024-09-17T04:01:43.768157+0000","last_peered":"2024-09-17T04:01:43.768157+0000","last_clean":"2024-09-17T04:01:43.768157+0000","last_became_active":"2024-09-17T04:01:42.770786+0000","last_became_peered":"2024-09-17T04:01:42.770786+0000","last_unstale":"2024-09-17T04:01:43.768157+0000","last_undegraded":"2024-09-17T04:01:43.768157+0000","last_fullsized":"2024-09-17T04:01:43.768157+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:04.962338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.749242+0000","last_change":"2024-09-17T04:01:41.751864+0000","last_active":"2024-09-17T04:01:42.749242+0000","last_peered":"2024-09-17T04:01:42.749242+0000","last_clean":"2024-09-17T04:01:42.749242+0000","last_became_active":"2024-09-17T04:01:41.751558+0000","last_became_peered":"2024-09-17T04:01:41.751558+0000","last_unstale":"2024-09-17T04:01:42.749242+0000","last_undegraded":"2024-09-17T04:01:42.749242+0000","last_fullsized":"2024-09-17T04:01:42.749242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:12.506890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763854+0000","last_change":"2024-09-17T04:01:42.771648+0000","last_active":"2024-09-17T04:01:43.763854+0000","last_peered":"2024-09-17T04:01:43.763854+0000","last_clean":"2024-09-17T04:01:43.763854+0000","last_became_active":"2024-09-17T04:01:42.771328+0000","last_became_peered":"2024-09-17T04:01:42.771328+0000","last_unstale":"2024-09-17T04:01:43.763854+0000","last_undegraded":"2024-09-17T04:01:43.763854+0000","last_fullsized":"2024-09-17T04:01:43.763854+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:06.353603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.15","version":"185'3","reported_seq":18,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929835+0000","last_change":"2024-09-17T04:01:41.753062+0000","last_active":"2024-09-17T04:01:43.929835+0000","last_peered":"2024-09-17T04:01:43.929835+0000","last_clean":"2024-09-17T04:01:43.929835+0000","last_became_active":"2024-09-17T04:01:41.752724+0000","last_became_peered":"2024-09-17T04:01:41.752724+0000","last_unstale":"2024-09-17T04:01:43.929835+0000","last_undegraded":"2024-09-17T04:01:43.929835+0000","last_fullsized":"2024-09-17T04:01:43.929835+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:35.015903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767948+0000","last_change":"2024-09-17T04:01:42.767460+0000","last_active":"2024-09-17T04:01:43.767948+0000","last_peered":"2024-09-17T04:01:43.767948+0000","last_clean":"2024-09-17T04:01:43.767948+0000","last_became_active":"2024-09-17T04:01:42.766109+0000","last_became_peered":"2024-09-17T04:01:42.766109+0000","last_unstale":"2024-09-17T04:01:43.767948+0000","last_undegraded":"2024-09-17T04:01:43.767948+0000","last_fullsized":"2024-09-17T04:01:43.767948+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:41.230265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"185'6","reported_seq":21,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929555+0000","last_change":"2024-09-17T04:01:41.755339+0000","last_active":"2024-09-17T04:01:43.929555+0000","last_peered":"2024-09-17T04:01:43.929555+0000","last_clean":"2024-09-17T04:01:43.929555+0000","last_became_active":"2024-09-17T04:01:41.754514+0000","last_became_peered":"2024-09-17T04:01:41.754514+0000","last_unstale":"2024-09-17T04:01:43.929555+0000","last_undegraded":"2024-09-17T04:01:43.929555+0000","last_fullsized":"2024-09-17T04:01:43.929555+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:49.451867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765887+0000","last_change":"2024-09-17T04:01:42.760950+0000","last_active":"2024-09-17T04:01:42.765887+0000","last_peered":"2024-09-17T04:01:42.765887+0000","last_clean":"2024-09-17T04:01:42.765887+0000","last_became_active":"2024-09-17T04:01:42.760631+0000","last_became_peered":"2024-09-17T04:01:42.760631+0000","last_unstale":"2024-09-17T04:01:42.765887+0000","last_undegraded":"2024-09-17T04:01:42.765887+0000","last_fullsized":"2024-09-17T04:01:42.765887+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:11.054394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763796+0000","last_change":"2024-09-17T04:01:41.756584+0000","last_active":"2024-09-17T04:01:43.763796+0000","last_peered":"2024-09-17T04:01:43.763796+0000","last_clean":"2024-09-17T04:01:43.763796+0000","last_became_active":"2024-09-17T04:01:41.755819+0000","last_became_peered":"2024-09-17T04:01:41.755819+0000","last_unstale":"2024-09-17T04:01:43.763796+0000","last_undegraded":"2024-09-17T04:01:43.763796+0000","last_fullsized":"2024-09-17T04:01:43.763796+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:13.276461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785167+0000","last_change":"2024-09-17T04:01:42.777226+0000","last_active":"2024-09-17T04:01:42.785167+0000","last_peered":"2024-09-17T04:01:42.785167+0000","last_clean":"2024-09-17T04:01:42.785167+0000","last_became_active":"2024-09-17T04:01:42.767386+0000","last_became_peered":"2024-09-17T04:01:42.767386+0000","last_unstale":"2024-09-17T04:01:42.785167+0000","last_undegraded":"2024-09-17T04:01:42.785167+0000","last_fullsized":"2024-09-17T04:01:42.785167+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:19.847104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.750670+0000","last_change":"2024-09-17T04:01:41.752773+0000","last_active":"2024-09-17T04:01:42.750670+0000","last_peered":"2024-09-17T04:01:42.750670+0000","last_clean":"2024-09-17T04:01:42.750670+0000","last_became_active":"2024-09-17T04:01:41.752369+0000","last_became_peered":"2024-09-17T04:01:41.752369+0000","last_unstale":"2024-09-17T04:01:42.750670+0000","last_undegraded":"2024-09-17T04:01:42.750670+0000","last_fullsized":"2024-09-17T04:01:42.750670+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:51:50.688529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786996+0000","last_change":"2024-09-17T04:01:42.784206+0000","last_active":"2024-09-17T04:01:42.786996+0000","last_peered":"2024-09-17T04:01:42.786996+0000","last_clean":"2024-09-17T04:01:42.786996+0000","last_became_active":"2024-09-17T04:01:42.783874+0000","last_became_peered":"2024-09-17T04:01:42.783874+0000","last_unstale":"2024-09-17T04:01:42.786996+0000","last_undegraded":"2024-09-17T04:01:42.786996+0000","last_fullsized":"2024-09-17T04:01:42.786996+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:28.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928622+0000","last_change":"2024-09-17T04:01:41.756092+0000","last_active":"2024-09-17T04:01:43.928622+0000","last_peered":"2024-09-17T04:01:43.928622+0000","last_clean":"2024-09-17T04:01:43.928622+0000","last_became_active":"2024-09-17T04:01:41.754932+0000","last_became_peered":"2024-09-17T04:01:41.754932+0000","last_unstale":"2024-09-17T04:01:43.928622+0000","last_undegraded":"2024-09-17T04:01:43.928622+0000","last_fullsized":"2024-09-17T04:01:43.928622+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:55:06.681949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763498+0000","last_change":"2024-09-17T04:01:42.765088+0000","last_active":"2024-09-17T04:01:43.763498+0000","last_peered":"2024-09-17T04:01:43.763498+0000","last_clean":"2024-09-17T04:01:43.763498+0000","last_became_active":"2024-09-17T04:01:42.764764+0000","last_became_peered":"2024-09-17T04:01:42.764764+0000","last_unstale":"2024-09-17T04:01:43.763498+0000","last_undegraded":"2024-09-17T04:01:43.763498+0000","last_fullsized":"2024-09-17T04:01:43.763498+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.375890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.10","version":"185'8","reported_seq":23,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928711+0000","last_change":"2024-09-17T04:01:41.748879+0000","last_active":"2024-09-17T04:01:43.928711+0000","last_peered":"2024-09-17T04:01:43.928711+0000","last_clean":"2024-09-17T04:01:43.928711+0000","last_became_active":"2024-09-17T04:01:41.747046+0000","last_became_peered":"2024-09-17T04:01:41.747046+0000","last_unstale":"2024-09-17T04:01:43.928711+0000","last_undegraded":"2024-09-17T04:01:43.928711+0000","last_fullsized":"2024-09-17T04:01:43.928711+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:05.347003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763721+0000","last_change":"2024-09-17T04:01:42.767825+0000","last_active":"2024-09-17T04:01:43.763721+0000","last_peered":"2024-09-17T04:01:43.763721+0000","last_clean":"2024-09-17T04:01:43.763721+0000","last_became_active":"2024-09-17T04:01:42.767519+0000","last_became_peered":"2024-09-17T04:01:42.767519+0000","last_unstale":"2024-09-17T04:01:43.763721+0000","last_undegraded":"2024-09-17T04:01:43.763721+0000","last_fullsized":"2024-09-17T04:01:43.763721+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:51:49.098369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"185'1","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785186+0000","last_change":"2024-09-17T04:01:41.756724+0000","last_active":"2024-09-17T04:01:42.785186+0000","last_peered":"2024-09-17T04:01:42.785186+0000","last_clean":"2024-09-17T04:01:42.785186+0000","last_became_active":"2024-09-17T04:01:41.756397+0000","last_became_peered":"2024-09-17T04:01:41.756397+0000","last_unstale":"2024-09-17T04:01:42.785186+0000","last_undegraded":"2024-09-17T04:01:42.785186+0000","last_fullsized":"2024-09-17T04:01:42.785186+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:37.545097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929524+0000","last_change":"2024-09-17T04:01:42.770508+0000","last_active":"2024-09-17T04:01:43.929524+0000","last_peered":"2024-09-17T04:01:43.929524+0000","last_clean":"2024-09-17T04:01:43.929524+0000","last_became_active":"2024-09-17T04:01:42.769764+0000","last_became_peered":"2024-09-17T04:01:42.769764+0000","last_unstale":"2024-09-17T04:01:43.929524+0000","last_undegraded":"2024-09-17T04:01:43.929524+0000","last_fullsized":"2024-09-17T04:01:43.929524+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:39.353840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"185'2","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780753+0000","last_change":"2024-09-17T04:01:41.751012+0000","last_active":"2024-09-17T04:01:42.780753+0000","last_peered":"2024-09-17T04:01:42.780753+0000","last_clean":"2024-09-17T04:01:42.780753+0000","last_became_active":"2024-09-17T04:01:41.749862+0000","last_became_peered":"2024-09-17T04:01:41.749862+0000","last_unstale":"2024-09-17T04:01:42.780753+0000","last_undegraded":"2024-09-17T04:01:42.780753+0000","last_fullsized":"2024-09-17T04:01:42.780753+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:06.418878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786448+0000","last_change":"2024-09-17T04:01:42.778201+0000","last_active":"2024-09-17T04:01:42.786448+0000","last_peered":"2024-09-17T04:01:42.786448+0000","last_clean":"2024-09-17T04:01:42.786448+0000","last_became_active":"2024-09-17T04:01:42.777811+0000","last_became_peered":"2024-09-17T04:01:42.777811+0000","last_unstale":"2024-09-17T04:01:42.786448+0000","last_undegraded":"2024-09-17T04:01:42.786448+0000","last_fullsized":"2024-09-17T04:01:42.786448+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:29.314779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.d","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768114+0000","last_change":"2024-09-17T04:01:41.753579+0000","last_active":"2024-09-17T04:01:43.768114+0000","last_peered":"2024-09-17T04:01:43.768114+0000","last_clean":"2024-09-17T04:01:43.768114+0000","last_became_active":"2024-09-17T04:01:41.753259+0000","last_became_peered":"2024-09-17T04:01:41.753259+0000","last_unstale":"2024-09-17T04:01:43.768114+0000","last_undegraded":"2024-09-17T04:01:43.768114+0000","last_fullsized":"2024-09-17T04:01:43.768114+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:42.324911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786868+0000","last_change":"2024-09-17T04:01:42.778845+0000","last_active":"2024-09-17T04:01:42.786868+0000","last_peered":"2024-09-17T04:01:42.786868+0000","last_clean":"2024-09-17T04:01:42.786868+0000","last_became_active":"2024-09-17T04:01:42.778269+0000","last_became_peered":"2024-09-17T04:01:42.778269+0000","last_unstale":"2024-09-17T04:01:42.786868+0000","last_undegraded":"2024-09-17T04:01:42.786868+0000","last_fullsized":"2024-09-17T04:01:42.786868+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:30.141000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929641+0000","last_change":"2024-09-17T04:01:41.746140+0000","last_active":"2024-09-17T04:01:43.929641+0000","last_peered":"2024-09-17T04:01:43.929641+0000","last_clean":"2024-09-17T04:01:43.929641+0000","last_became_active":"2024-09-17T04:01:41.744931+0000","last_became_peered":"2024-09-17T04:01:41.744931+0000","last_unstale":"2024-09-17T04:01:43.929641+0000","last_undegraded":"2024-09-17T04:01:43.929641+0000","last_fullsized":"2024-09-17T04:01:43.929641+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:03.651538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928387+0000","last_change":"2024-09-17T04:01:42.775567+0000","last_active":"2024-09-17T04:01:43.928387+0000","last_peered":"2024-09-17T04:01:43.928387+0000","last_clean":"2024-09-17T04:01:43.928387+0000","last_became_active":"2024-09-17T04:01:42.775147+0000","last_became_peered":"2024-09-17T04:01:42.775147+0000","last_unstale":"2024-09-17T04:01:43.928387+0000","last_undegraded":"2024-09-17T04:01:43.928387+0000","last_fullsized":"2024-09-17T04:01:43.928387+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:05.524401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.b","version":"185'2","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.787252+0000","last_change":"2024-09-17T04:01:41.756084+0000","last_active":"2024-09-17T04:01:42.787252+0000","last_peered":"2024-09-17T04:01:42.787252+0000","last_clean":"2024-09-17T04:01:42.787252+0000","last_became_active":"2024-09-17T04:01:41.755765+0000","last_became_peered":"2024-09-17T04:01:41.755765+0000","last_unstale":"2024-09-17T04:01:42.787252+0000","last_undegraded":"2024-09-17T04:01:42.787252+0000","last_fullsized":"2024-09-17T04:01:42.787252+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:49.605022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786857+0000","last_change":"2024-09-17T04:01:42.780860+0000","last_active":"2024-09-17T04:01:42.786857+0000","last_peered":"2024-09-17T04:01:42.786857+0000","last_clean":"2024-09-17T04:01:42.786857+0000","last_became_active":"2024-09-17T04:01:42.780520+0000","last_became_peered":"2024-09-17T04:01:42.780520+0000","last_unstale":"2024-09-17T04:01:42.786857+0000","last_undegraded":"2024-09-17T04:01:42.786857+0000","last_fullsized":"2024-09-17T04:01:42.786857+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:00.582115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763835+0000","last_change":"2024-09-17T04:01:41.745037+0000","last_active":"2024-09-17T04:01:43.763835+0000","last_peered":"2024-09-17T04:01:43.763835+0000","last_clean":"2024-09-17T04:01:43.763835+0000","last_became_active":"2024-09-17T04:01:41.744675+0000","last_became_peered":"2024-09-17T04:01:41.744675+0000","last_unstale":"2024-09-17T04:01:43.763835+0000","last_undegraded":"2024-09-17T04:01:43.763835+0000","last_fullsized":"2024-09-17T04:01:43.763835+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:01.467030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763356+0000","last_change":"2024-09-17T04:01:42.767153+0000","last_active":"2024-09-17T04:01:43.763356+0000","last_peered":"2024-09-17T04:01:43.763356+0000","last_clean":"2024-09-17T04:01:43.763356+0000","last_became_active":"2024-09-17T04:01:42.766844+0000","last_became_peered":"2024-09-17T04:01:42.766844+0000","last_unstale":"2024-09-17T04:01:43.763356+0000","last_undegraded":"2024-09-17T04:01:43.763356+0000","last_fullsized":"2024-09-17T04:01:43.763356+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:21.337125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763998+0000","last_change":"2024-09-17T04:01:41.754201+0000","last_active":"2024-09-17T04:01:43.763998+0000","last_peered":"2024-09-17T04:01:43.763998+0000","last_clean":"2024-09-17T04:01:43.763998+0000","last_became_active":"2024-09-17T04:01:41.753871+0000","last_became_peered":"2024-09-17T04:01:41.753871+0000","last_unstale":"2024-09-17T04:01:43.763998+0000","last_undegraded":"2024-09-17T04:01:43.763998+0000","last_fullsized":"2024-09-17T04:01:43.763998+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:07.297101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928642+0000","last_change":"2024-09-17T04:01:42.759404+0000","last_active":"2024-09-17T04:01:43.928642+0000","last_peered":"2024-09-17T04:01:43.928642+0000","last_clean":"2024-09-17T04:01:43.928642+0000","last_became_active":"2024-09-17T04:01:42.758839+0000","last_became_peered":"2024-09-17T04:01:42.758839+0000","last_unstale":"2024-09-17T04:01:43.928642+0000","last_undegraded":"2024-09-17T04:01:43.928642+0000","last_fullsized":"2024-09-17T04:01:43.928642+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:08.698710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765276+0000","last_change":"2024-09-17T04:01:41.754990+0000","last_active":"2024-09-17T04:01:42.765276+0000","last_peered":"2024-09-17T04:01:42.765276+0000","last_clean":"2024-09-17T04:01:42.765276+0000","last_became_active":"2024-09-17T04:01:41.754659+0000","last_became_peered":"2024-09-17T04:01:41.754659+0000","last_unstale":"2024-09-17T04:01:42.765276+0000","last_undegraded":"2024-09-17T04:01:42.765276+0000","last_fullsized":"2024-09-17T04:01:42.765276+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:14.252648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929845+0000","last_change":"2024-09-17T04:01:42.772854+0000","last_active":"2024-09-17T04:01:43.929845+0000","last_peered":"2024-09-17T04:01:43.929845+0000","last_clean":"2024-09-17T04:01:43.929845+0000","last_became_active":"2024-09-17T04:01:42.772538+0000","last_became_peered":"2024-09-17T04:01:42.772538+0000","last_unstale":"2024-09-17T04:01:43.929845+0000","last_undegraded":"2024-09-17T04:01:43.929845+0000","last_fullsized":"2024-09-17T04:01:43.929845+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:28.826108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.7","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929938+0000","last_change":"2024-09-17T04:01:41.759215+0000","last_active":"2024-09-17T04:01:43.929938+0000","last_peered":"2024-09-17T04:01:43.929938+0000","last_clean":"2024-09-17T04:01:43.929938+0000","last_became_active":"2024-09-17T04:01:41.758958+0000","last_became_peered":"2024-09-17T04:01:41.758958+0000","last_unstale":"2024-09-17T04:01:43.929938+0000","last_undegraded":"2024-09-17T04:01:43.929938+0000","last_fullsized":"2024-09-17T04:01:43.929938+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:26.610681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781839+0000","last_change":"2024-09-17T04:01:42.771571+0000","last_active":"2024-09-17T04:01:42.781839+0000","last_peered":"2024-09-17T04:01:42.781839+0000","last_clean":"2024-09-17T04:01:42.781839+0000","last_became_active":"2024-09-17T04:01:42.771264+0000","last_became_peered":"2024-09-17T04:01:42.771264+0000","last_unstale":"2024-09-17T04:01:42.781839+0000","last_undegraded":"2024-09-17T04:01:42.781839+0000","last_fullsized":"2024-09-17T04:01:42.781839+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:04.247007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.6","version":"186'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.769667+0000","last_change":"2024-09-17T04:01:41.759965+0000","last_active":"2024-09-17T04:01:43.769667+0000","last_peered":"2024-09-17T04:01:43.769667+0000","last_clean":"2024-09-17T04:01:43.769667+0000","last_became_active":"2024-09-17T04:01:41.759584+0000","last_became_peered":"2024-09-17T04:01:41.759584+0000","last_unstale":"2024-09-17T04:01:43.769667+0000","last_undegraded":"2024-09-17T04:01:43.769667+0000","last_fullsized":"2024-09-17T04:01:43.769667+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:39.152086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.782855+0000","last_change":"2024-09-17T04:01:42.775297+0000","last_active":"2024-09-17T04:01:42.782855+0000","last_peered":"2024-09-17T04:01:42.782855+0000","last_clean":"2024-09-17T04:01:42.782855+0000","last_became_active":"2024-09-17T04:01:42.773237+0000","last_became_peered":"2024-09-17T04:01:42.773237+0000","last_unstale":"2024-09-17T04:01:42.782855+0000","last_undegraded":"2024-09-17T04:01:42.782855+0000","last_fullsized":"2024-09-17T04:01:42.782855+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:39.396816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768260+0000","last_change":"2024-09-17T04:01:41.758099+0000","last_active":"2024-09-17T04:01:43.768260+0000","last_peered":"2024-09-17T04:01:43.768260+0000","last_clean":"2024-09-17T04:01:43.768260+0000","last_became_active":"2024-09-17T04:01:41.756992+0000","last_became_peered":"2024-09-17T04:01:41.756992+0000","last_unstale":"2024-09-17T04:01:43.768260+0000","last_undegraded":"2024-09-17T04:01:43.768260+0000","last_fullsized":"2024-09-17T04:01:43.768260+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:22.619173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929711+0000","last_change":"2024-09-17T04:01:42.761322+0000","last_active":"2024-09-17T04:01:43.929711+0000","last_peered":"2024-09-17T04:01:43.929711+0000","last_clean":"2024-09-17T04:01:43.929711+0000","last_became_active":"2024-09-17T04:01:42.761023+0000","last_became_peered":"2024-09-17T04:01:42.761023+0000","last_unstale":"2024-09-17T04:01:43.929711+0000","last_undegraded":"2024-09-17T04:01:43.929711+0000","last_fullsized":"2024-09-17T04:01:43.929711+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:16.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.772280+0000","last_change":"2024-09-17T04:01:41.754588+0000","last_active":"2024-09-17T04:01:42.772280+0000","last_peered":"2024-09-17T04:01:42.772280+0000","last_clean":"2024-09-17T04:01:42.772280+0000","last_became_active":"2024-09-17T04:01:41.754254+0000","last_became_peered":"2024-09-17T04:01:41.754254+0000","last_unstale":"2024-09-17T04:01:42.772280+0000","last_undegraded":"2024-09-17T04:01:42.772280+0000","last_fullsized":"2024-09-17T04:01:42.772280+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:10.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928885+0000","last_change":"2024-09-17T04:01:42.761957+0000","last_active":"2024-09-17T04:01:43.928885+0000","last_peered":"2024-09-17T04:01:43.928885+0000","last_clean":"2024-09-17T04:01:43.928885+0000","last_became_active":"2024-09-17T04:01:42.760954+0000","last_became_peered":"2024-09-17T04:01:42.760954+0000","last_unstale":"2024-09-17T04:01:43.928885+0000","last_undegraded":"2024-09-17T04:01:43.928885+0000","last_fullsized":"2024-09-17T04:01:43.928885+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:12:25.422065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763702+0000","last_change":"2024-09-17T04:01:41.746613+0000","last_active":"2024-09-17T04:01:43.763702+0000","last_peered":"2024-09-17T04:01:43.763702+0000","last_clean":"2024-09-17T04:01:43.763702+0000","last_became_active":"2024-09-17T04:01:41.745078+0000","last_became_peered":"2024-09-17T04:01:41.745078+0000","last_unstale":"2024-09-17T04:01:43.763702+0000","last_undegraded":"2024-09-17T04:01:43.763702+0000","last_fullsized":"2024-09-17T04:01:43.763702+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:03.719713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781647+0000","last_change":"2024-09-17T04:01:42.769926+0000","last_active":"2024-09-17T04:01:42.781647+0000","last_peered":"2024-09-17T04:01:42.781647+0000","last_clean":"2024-09-17T04:01:42.781647+0000","last_became_active":"2024-09-17T04:01:42.766047+0000","last_became_peered":"2024-09-17T04:01:42.766047+0000","last_unstale":"2024-09-17T04:01:42.781647+0000","last_undegraded":"2024-09-17T04:01:42.781647+0000","last_fullsized":"2024-09-17T04:01:42.781647+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:58.970219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.750899+0000","last_change":"2024-09-17T04:01:41.757010+0000","last_active":"2024-09-17T04:01:42.750899+0000","last_peered":"2024-09-17T04:01:42.750899+0000","last_clean":"2024-09-17T04:01:42.750899+0000","last_became_active":"2024-09-17T04:01:41.756668+0000","last_became_peered":"2024-09-17T04:01:41.756668+0000","last_unstale":"2024-09-17T04:01:42.750899+0000","last_undegraded":"2024-09-17T04:01:42.750899+0000","last_fullsized":"2024-09-17T04:01:42.750899+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:13.736891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781817+0000","last_change":"2024-09-17T04:01:42.774458+0000","last_active":"2024-09-17T04:01:42.781817+0000","last_peered":"2024-09-17T04:01:42.781817+0000","last_clean":"2024-09-17T04:01:42.781817+0000","last_became_active":"2024-09-17T04:01:42.771159+0000","last_became_peered":"2024-09-17T04:01:42.771159+0000","last_unstale":"2024-09-17T04:01:42.781817+0000","last_undegraded":"2024-09-17T04:01:42.781817+0000","last_fullsized":"2024-09-17T04:01:42.781817+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:03.676708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928725+0000","last_change":"2024-09-17T04:01:41.755626+0000","last_active":"2024-09-17T04:01:43.928725+0000","last_peered":"2024-09-17T04:01:43.928725+0000","last_clean":"2024-09-17T04:01:43.928725+0000","last_became_active":"2024-09-17T04:01:41.755345+0000","last_became_peered":"2024-09-17T04:01:41.755345+0000","last_unstale":"2024-09-17T04:01:43.928725+0000","last_undegraded":"2024-09-17T04:01:43.928725+0000","last_fullsized":"2024-09-17T04:01:43.928725+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:39.514567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767474+0000","last_change":"2024-09-17T04:01:42.771823+0000","last_active":"2024-09-17T04:01:43.767474+0000","last_peered":"2024-09-17T04:01:43.767474+0000","last_clean":"2024-09-17T04:01:43.767474+0000","last_became_active":"2024-09-17T04:01:42.769101+0000","last_became_peered":"2024-09-17T04:01:42.769101+0000","last_unstale":"2024-09-17T04:01:43.767474+0000","last_undegraded":"2024-09-17T04:01:43.767474+0000","last_fullsized":"2024-09-17T04:01:43.767474+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:07.630156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765509+0000","last_change":"2024-09-17T04:01:41.753411+0000","last_active":"2024-09-17T04:01:42.765509+0000","last_peered":"2024-09-17T04:01:42.765509+0000","last_clean":"2024-09-17T04:01:42.765509+0000","last_became_active":"2024-09-17T04:01:41.752972+0000","last_became_peered":"2024-09-17T04:01:41.752972+0000","last_unstale":"2024-09-17T04:01:42.765509+0000","last_undegraded":"2024-09-17T04:01:42.765509+0000","last_fullsized":"2024-09-17T04:01:42.765509+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:32.697239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767594+0000","last_change":"2024-09-17T04:01:42.764031+0000","last_active":"2024-09-17T04:01:43.767594+0000","last_peered":"2024-09-17T04:01:43.767594+0000","last_clean":"2024-09-17T04:01:43.767594+0000","last_became_active":"2024-09-17T04:01:42.762106+0000","last_became_peered":"2024-09-17T04:01:42.762106+0000","last_unstale":"2024-09-17T04:01:43.767594+0000","last_undegraded":"2024-09-17T04:01:43.767594+0000","last_fullsized":"2024-09-17T04:01:43.767594+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:56.512369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764055+0000","last_change":"2024-09-17T04:01:41.755631+0000","last_active":"2024-09-17T04:01:43.764055+0000","last_peered":"2024-09-17T04:01:43.764055+0000","last_clean":"2024-09-17T04:01:43.764055+0000","last_became_active":"2024-09-17T04:01:41.755282+0000","last_became_peered":"2024-09-17T04:01:41.755282+0000","last_unstale":"2024-09-17T04:01:43.764055+0000","last_undegraded":"2024-09-17T04:01:43.764055+0000","last_fullsized":"2024-09-17T04:01:43.764055+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:33.798971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":414,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767109+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:01:43.767109+0000","last_peered":"2024-09-17T04:01:43.767109+0000","last_clean":"2024-09-17T04:01:43.767109+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:01:43.767109+0000","last_undegraded":"2024-09-17T04:01:43.767109+0000","last_fullsized":"2024-09-17T04:01:43.767109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509590,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42384,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329456,"statfs":{"total":96636764160,"available":96593362944,"internally_reserved":0,"allocated":2940928,"data_stored":1616651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509593,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2342912,"data_stored":1026213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509592,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":2347008,"data_stored":1026795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509590,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330024,"statfs":{"total":96636764160,"available":96593944576,"internally_reserved":0,"allocated":2359296,"data_stored":1027895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37660,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334180,"statfs":{"total":96636764160,"available":96598200320,"internally_reserved":0,"allocated":2363392,"data_stored":1027881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509592,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41804,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330036,"statfs":{"total":96636764160,"available":96593956864,"internally_reserved":0,"allocated":2347008,"data_stored":1026303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2338816,"data_stored":1019969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":2912256,"data_stored":1603506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:01:47.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:01:47.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 -- 172.21.15.129:0/4113884852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 msgr2=0x7f94e40b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 --2- 172.21.15.129:0/4113884852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e40b4180 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009830 tx=0x7f94d402a510 comp rx=0 tx=0).stop 2024-09-17T04:01:47.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 -- 172.21.15.129:0/4113884852 shutdown_connections 2024-09-17T04:01:47.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 --2- 172.21.15.129:0/4113884852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40b46c0 0x7f94e40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 --2- 172.21.15.129:0/4113884852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e40b4180 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 -- 172.21.15.129:0/4113884852 >> 172.21.15.129:0/4113884852 conn(0x7f94e406d410 msgr2=0x7f94e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:47.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 -- 172.21.15.129:0/4113884852 shutdown_connections 2024-09-17T04:01:47.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.441+0000 7f94eac00640 1 -- 172.21.15.129:0/4113884852 wait complete. 2024-09-17T04:01:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.443+0000 7f94eac00640 1 Processor -- start 2024-09-17T04:01:47.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.443+0000 7f94eac00640 1 -- start start 2024-09-17T04:01:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40cd1c0 0x7f94e415a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e4098750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4070990 con 0x7f94e40b46c0 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94e4070810 con 0x7f94e415da00 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94eac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f94e4070b10 con 0x7f94e40cd1c0 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44532/0 (socket says 172.21.15.129:44532) 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e3e00640 1 -- 172.21.15.129:0/1656506670 learned_addr learned my addr 172.21.15.129:0/1656506670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e3400640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40cd1c0 0x7f94e415a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e8c00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e4098750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.445+0000 7f94e3e00640 1 -- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40cd1c0 msgr2=0x7f94e415a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94e3e00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40cd1c0 0x7f94e415a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94e3e00640 1 -- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 msgr2=0x7f94e4098750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94e3e00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e4098750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94e3e00640 1 -- 172.21.15.129:0/1656506670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e41f4d80 con 0x7f94e40b46c0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94e3e00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009950 tx=0x7f94d4033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4005c90 con 0x7f94e40b46c0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94d4031e50 con 0x7f94e40b46c0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e41f44a0 con 0x7f94e40b46c0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4005260 con 0x7f94e40b46c0 2024-09-17T04:01:47.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.446+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41f4a60 con 0x7f94e40b46c0 2024-09-17T04:01:47.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.447+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e40ccc80 con 0x7f94e40b46c0 2024-09-17T04:01:47.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.449+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f94d4005400 con 0x7f94e40b46c0 2024-09-17T04:01:47.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.450+0000 7f94dbe00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 0x7f94c407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.450+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f94d40bbe30 con 0x7f94e40b46c0 2024-09-17T04:01:47.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.450+0000 7f94e3400640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 0x7f94c407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.451+0000 7f94e3400640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 0x7f94c407b0d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f94e40c53c0 tx=0x7f94d0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:47.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.454+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f94d403f030 con 0x7f94e40b46c0 2024-09-17T04:01:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.636+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f94e406a140 con 0x7f94c4078bc0 2024-09-17T04:01:47.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.641+0000 7f94dbe00640 1 -- 172.21.15.129:0/1656506670 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252478 (secure 0 0 0) 0x7f94e406a140 con 0x7f94c4078bc0 2024-09-17T04:01:47.644 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:47.646 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:01:47.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 msgr2=0x7f94c407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 0x7f94c407b0d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f94e40c53c0 tx=0x7f94d0005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:47.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 msgr2=0x7f94e41f3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009950 tx=0x7f94d4033c20 comp rx=0 tx=0).stop 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 shutdown_connections 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f94c4078bc0 0x7f94c407b0d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94e415da00 0x7f94e4098750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f94e40cd1c0 0x7f94e415a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 --2- 172.21.15.129:0/1656506670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f94e40b46c0 0x7f94e41f3c50 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 >> 172.21.15.129:0/1656506670 conn(0x7f94e406d410 msgr2=0x7f94e40b1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 shutdown_connections 2024-09-17T04:01:47.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.645+0000 7f94eac00640 1 -- 172.21.15.129:0/1656506670 wait complete. 2024-09-17T04:01:47.668 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-17T04:01:46.502079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":331412,"kb_used_data":19484,"kb_used_omap":12,"kb_used_meta":311731,"kb_avail":754643308,"statfs":{"total":773094113280,"available":772754747392,"internally_reserved":0,"allocated":19951616,"data_stored":9375213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":319213104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.771528"},"pg_stats":[{"pgid":"29.3f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767884+0000","last_change":"2024-09-17T04:01:42.754485+0000","last_active":"2024-09-17T04:01:43.767884+0000","last_peered":"2024-09-17T04:01:43.767884+0000","last_clean":"2024-09-17T04:01:43.767884+0000","last_became_active":"2024-09-17T04:01:42.753255+0000","last_became_peered":"2024-09-17T04:01:42.753255+0000","last_unstale":"2024-09-17T04:01:43.767884+0000","last_undegraded":"2024-09-17T04:01:43.767884+0000","last_fullsized":"2024-09-17T04:01:43.767884+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:03.067830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928770+0000","last_change":"2024-09-17T04:01:42.785451+0000","last_active":"2024-09-17T04:01:43.928770+0000","last_peered":"2024-09-17T04:01:43.928770+0000","last_clean":"2024-09-17T04:01:43.928770+0000","last_became_active":"2024-09-17T04:01:42.784265+0000","last_became_peered":"2024-09-17T04:01:42.784265+0000","last_unstale":"2024-09-17T04:01:43.928770+0000","last_undegraded":"2024-09-17T04:01:43.928770+0000","last_fullsized":"2024-09-17T04:01:43.928770+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:49:55.439278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780763+0000","last_change":"2024-09-17T04:01:42.770419+0000","last_active":"2024-09-17T04:01:42.780763+0000","last_peered":"2024-09-17T04:01:42.780763+0000","last_clean":"2024-09-17T04:01:42.780763+0000","last_became_active":"2024-09-17T04:01:42.769275+0000","last_became_peered":"2024-09-17T04:01:42.769275+0000","last_unstale":"2024-09-17T04:01:42.780763+0000","last_undegraded":"2024-09-17T04:01:42.780763+0000","last_fullsized":"2024-09-17T04:01:42.780763+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:05.207595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929357+0000","last_change":"2024-09-17T04:01:42.759946+0000","last_active":"2024-09-17T04:01:43.929357+0000","last_peered":"2024-09-17T04:01:43.929357+0000","last_clean":"2024-09-17T04:01:43.929357+0000","last_became_active":"2024-09-17T04:01:42.759614+0000","last_became_peered":"2024-09-17T04:01:42.759614+0000","last_unstale":"2024-09-17T04:01:43.929357+0000","last_undegraded":"2024-09-17T04:01:43.929357+0000","last_fullsized":"2024-09-17T04:01:43.929357+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:55:39.034951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928536+0000","last_change":"2024-09-17T04:01:42.761604+0000","last_active":"2024-09-17T04:01:43.928536+0000","last_peered":"2024-09-17T04:01:43.928536+0000","last_clean":"2024-09-17T04:01:43.928536+0000","last_became_active":"2024-09-17T04:01:42.761303+0000","last_became_peered":"2024-09-17T04:01:42.761303+0000","last_unstale":"2024-09-17T04:01:43.928536+0000","last_undegraded":"2024-09-17T04:01:43.928536+0000","last_fullsized":"2024-09-17T04:01:43.928536+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:50.550720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929117+0000","last_change":"2024-09-17T04:01:42.762544+0000","last_active":"2024-09-17T04:01:43.929117+0000","last_peered":"2024-09-17T04:01:43.929117+0000","last_clean":"2024-09-17T04:01:43.929117+0000","last_became_active":"2024-09-17T04:01:42.762157+0000","last_became_peered":"2024-09-17T04:01:42.762157+0000","last_unstale":"2024-09-17T04:01:43.929117+0000","last_undegraded":"2024-09-17T04:01:43.929117+0000","last_fullsized":"2024-09-17T04:01:43.929117+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:10.939599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781454+0000","last_change":"2024-09-17T04:01:42.767972+0000","last_active":"2024-09-17T04:01:42.781454+0000","last_peered":"2024-09-17T04:01:42.781454+0000","last_clean":"2024-09-17T04:01:42.781454+0000","last_became_active":"2024-09-17T04:01:42.767668+0000","last_became_peered":"2024-09-17T04:01:42.767668+0000","last_unstale":"2024-09-17T04:01:42.781454+0000","last_undegraded":"2024-09-17T04:01:42.781454+0000","last_fullsized":"2024-09-17T04:01:42.781454+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:57:34.770623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763311+0000","last_change":"2024-09-17T04:01:42.770322+0000","last_active":"2024-09-17T04:01:43.763311+0000","last_peered":"2024-09-17T04:01:43.763311+0000","last_clean":"2024-09-17T04:01:43.763311+0000","last_became_active":"2024-09-17T04:01:42.770035+0000","last_became_peered":"2024-09-17T04:01:42.770035+0000","last_unstale":"2024-09-17T04:01:43.763311+0000","last_undegraded":"2024-09-17T04:01:43.763311+0000","last_fullsized":"2024-09-17T04:01:43.763311+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:49.056085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786891+0000","last_change":"2024-09-17T04:01:42.780240+0000","last_active":"2024-09-17T04:01:42.786891+0000","last_peered":"2024-09-17T04:01:42.786891+0000","last_clean":"2024-09-17T04:01:42.786891+0000","last_became_active":"2024-09-17T04:01:42.779875+0000","last_became_peered":"2024-09-17T04:01:42.779875+0000","last_unstale":"2024-09-17T04:01:42.786891+0000","last_undegraded":"2024-09-17T04:01:42.786891+0000","last_fullsized":"2024-09-17T04:01:42.786891+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:19.775171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929819+0000","last_change":"2024-09-17T04:01:42.762434+0000","last_active":"2024-09-17T04:01:43.929819+0000","last_peered":"2024-09-17T04:01:43.929819+0000","last_clean":"2024-09-17T04:01:43.929819+0000","last_became_active":"2024-09-17T04:01:42.762102+0000","last_became_peered":"2024-09-17T04:01:42.762102+0000","last_unstale":"2024-09-17T04:01:43.929819+0000","last_undegraded":"2024-09-17T04:01:43.929819+0000","last_fullsized":"2024-09-17T04:01:43.929819+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:01:29.965573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767827+0000","last_change":"2024-09-17T04:01:42.756123+0000","last_active":"2024-09-17T04:01:43.767827+0000","last_peered":"2024-09-17T04:01:43.767827+0000","last_clean":"2024-09-17T04:01:43.767827+0000","last_became_active":"2024-09-17T04:01:42.755416+0000","last_became_peered":"2024-09-17T04:01:42.755416+0000","last_unstale":"2024-09-17T04:01:43.767827+0000","last_undegraded":"2024-09-17T04:01:43.767827+0000","last_fullsized":"2024-09-17T04:01:43.767827+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:37:30.705447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928702+0000","last_change":"2024-09-17T04:01:42.766800+0000","last_active":"2024-09-17T04:01:43.928702+0000","last_peered":"2024-09-17T04:01:43.928702+0000","last_clean":"2024-09-17T04:01:43.928702+0000","last_became_active":"2024-09-17T04:01:42.766479+0000","last_became_peered":"2024-09-17T04:01:42.766479+0000","last_unstale":"2024-09-17T04:01:43.928702+0000","last_undegraded":"2024-09-17T04:01:43.928702+0000","last_fullsized":"2024-09-17T04:01:43.928702+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:40.521166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928928+0000","last_change":"2024-09-17T04:01:42.769768+0000","last_active":"2024-09-17T04:01:43.928928+0000","last_peered":"2024-09-17T04:01:43.928928+0000","last_clean":"2024-09-17T04:01:43.928928+0000","last_became_active":"2024-09-17T04:01:42.769426+0000","last_became_peered":"2024-09-17T04:01:42.769426+0000","last_unstale":"2024-09-17T04:01:43.928928+0000","last_undegraded":"2024-09-17T04:01:43.928928+0000","last_fullsized":"2024-09-17T04:01:43.928928+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:11:18.702585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763693+0000","last_change":"2024-09-17T04:01:42.766746+0000","last_active":"2024-09-17T04:01:43.763693+0000","last_peered":"2024-09-17T04:01:43.763693+0000","last_clean":"2024-09-17T04:01:43.763693+0000","last_became_active":"2024-09-17T04:01:42.762258+0000","last_became_peered":"2024-09-17T04:01:42.762258+0000","last_unstale":"2024-09-17T04:01:43.763693+0000","last_undegraded":"2024-09-17T04:01:43.763693+0000","last_fullsized":"2024-09-17T04:01:43.763693+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:17.469808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.767630+0000","last_change":"2024-09-17T04:01:42.765061+0000","last_active":"2024-09-17T04:01:42.767630+0000","last_peered":"2024-09-17T04:01:42.767630+0000","last_clean":"2024-09-17T04:01:42.767630+0000","last_became_active":"2024-09-17T04:01:42.764781+0000","last_became_peered":"2024-09-17T04:01:42.764781+0000","last_unstale":"2024-09-17T04:01:42.767630+0000","last_undegraded":"2024-09-17T04:01:42.767630+0000","last_fullsized":"2024-09-17T04:01:42.767630+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:31.784422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928755+0000","last_change":"2024-09-17T04:01:42.753912+0000","last_active":"2024-09-17T04:01:43.928755+0000","last_peered":"2024-09-17T04:01:43.928755+0000","last_clean":"2024-09-17T04:01:43.928755+0000","last_became_active":"2024-09-17T04:01:42.753606+0000","last_became_peered":"2024-09-17T04:01:42.753606+0000","last_unstale":"2024-09-17T04:01:43.928755+0000","last_undegraded":"2024-09-17T04:01:43.928755+0000","last_fullsized":"2024-09-17T04:01:43.928755+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:52.188705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764105+0000","last_change":"2024-09-17T04:01:42.768700+0000","last_active":"2024-09-17T04:01:43.764105+0000","last_peered":"2024-09-17T04:01:43.764105+0000","last_clean":"2024-09-17T04:01:43.764105+0000","last_became_active":"2024-09-17T04:01:42.768338+0000","last_became_peered":"2024-09-17T04:01:42.768338+0000","last_unstale":"2024-09-17T04:01:43.764105+0000","last_undegraded":"2024-09-17T04:01:43.764105+0000","last_fullsized":"2024-09-17T04:01:43.764105+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:14.198048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.930035+0000","last_change":"2024-09-17T04:01:42.770307+0000","last_active":"2024-09-17T04:01:43.930035+0000","last_peered":"2024-09-17T04:01:43.930035+0000","last_clean":"2024-09-17T04:01:43.930035+0000","last_became_active":"2024-09-17T04:01:42.769936+0000","last_became_peered":"2024-09-17T04:01:42.769936+0000","last_unstale":"2024-09-17T04:01:43.930035+0000","last_undegraded":"2024-09-17T04:01:43.930035+0000","last_fullsized":"2024-09-17T04:01:43.930035+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:04.899511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929793+0000","last_change":"2024-09-17T04:01:42.755567+0000","last_active":"2024-09-17T04:01:43.929793+0000","last_peered":"2024-09-17T04:01:43.929793+0000","last_clean":"2024-09-17T04:01:43.929793+0000","last_became_active":"2024-09-17T04:01:42.755244+0000","last_became_peered":"2024-09-17T04:01:42.755244+0000","last_unstale":"2024-09-17T04:01:43.929793+0000","last_undegraded":"2024-09-17T04:01:43.929793+0000","last_fullsized":"2024-09-17T04:01:43.929793+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:10.408327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929419+0000","last_change":"2024-09-17T04:01:42.754625+0000","last_active":"2024-09-17T04:01:43.929419+0000","last_peered":"2024-09-17T04:01:43.929419+0000","last_clean":"2024-09-17T04:01:43.929419+0000","last_became_active":"2024-09-17T04:01:42.754277+0000","last_became_peered":"2024-09-17T04:01:42.754277+0000","last_unstale":"2024-09-17T04:01:43.929419+0000","last_undegraded":"2024-09-17T04:01:43.929419+0000","last_fullsized":"2024-09-17T04:01:43.929419+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:56.783108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928758+0000","last_change":"2024-09-17T04:01:42.755662+0000","last_active":"2024-09-17T04:01:43.928758+0000","last_peered":"2024-09-17T04:01:43.928758+0000","last_clean":"2024-09-17T04:01:43.928758+0000","last_became_active":"2024-09-17T04:01:42.755139+0000","last_became_peered":"2024-09-17T04:01:42.755139+0000","last_unstale":"2024-09-17T04:01:43.928758+0000","last_undegraded":"2024-09-17T04:01:43.928758+0000","last_fullsized":"2024-09-17T04:01:43.928758+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:26.887685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780855+0000","last_change":"2024-09-17T04:01:42.776413+0000","last_active":"2024-09-17T04:01:42.780855+0000","last_peered":"2024-09-17T04:01:42.780855+0000","last_clean":"2024-09-17T04:01:42.780855+0000","last_became_active":"2024-09-17T04:01:42.776083+0000","last_became_peered":"2024-09-17T04:01:42.776083+0000","last_unstale":"2024-09-17T04:01:42.780855+0000","last_undegraded":"2024-09-17T04:01:42.780855+0000","last_fullsized":"2024-09-17T04:01:42.780855+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:06:28.497472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767357+0000","last_change":"2024-09-17T04:01:42.763674+0000","last_active":"2024-09-17T04:01:43.767357+0000","last_peered":"2024-09-17T04:01:43.767357+0000","last_clean":"2024-09-17T04:01:43.767357+0000","last_became_active":"2024-09-17T04:01:42.760549+0000","last_became_peered":"2024-09-17T04:01:42.760549+0000","last_unstale":"2024-09-17T04:01:43.767357+0000","last_undegraded":"2024-09-17T04:01:43.767357+0000","last_fullsized":"2024-09-17T04:01:43.767357+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:45:27.528607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928276+0000","last_change":"2024-09-17T04:01:42.760393+0000","last_active":"2024-09-17T04:01:43.928276+0000","last_peered":"2024-09-17T04:01:43.928276+0000","last_clean":"2024-09-17T04:01:43.928276+0000","last_became_active":"2024-09-17T04:01:42.759501+0000","last_became_peered":"2024-09-17T04:01:42.759501+0000","last_unstale":"2024-09-17T04:01:43.928276+0000","last_undegraded":"2024-09-17T04:01:43.928276+0000","last_fullsized":"2024-09-17T04:01:43.928276+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:19:32.416747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768010+0000","last_change":"2024-09-17T04:01:42.760598+0000","last_active":"2024-09-17T04:01:43.768010+0000","last_peered":"2024-09-17T04:01:43.768010+0000","last_clean":"2024-09-17T04:01:43.768010+0000","last_became_active":"2024-09-17T04:01:42.759775+0000","last_became_peered":"2024-09-17T04:01:42.759775+0000","last_unstale":"2024-09-17T04:01:43.768010+0000","last_undegraded":"2024-09-17T04:01:43.768010+0000","last_fullsized":"2024-09-17T04:01:43.768010+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:14.625621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763650+0000","last_change":"2024-09-17T04:01:42.778451+0000","last_active":"2024-09-17T04:01:43.763650+0000","last_peered":"2024-09-17T04:01:43.763650+0000","last_clean":"2024-09-17T04:01:43.763650+0000","last_became_active":"2024-09-17T04:01:42.777195+0000","last_became_peered":"2024-09-17T04:01:42.777195+0000","last_unstale":"2024-09-17T04:01:43.763650+0000","last_undegraded":"2024-09-17T04:01:43.763650+0000","last_fullsized":"2024-09-17T04:01:43.763650+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:56.698474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768305+0000","last_change":"2024-09-17T04:01:42.756470+0000","last_active":"2024-09-17T04:01:43.768305+0000","last_peered":"2024-09-17T04:01:43.768305+0000","last_clean":"2024-09-17T04:01:43.768305+0000","last_became_active":"2024-09-17T04:01:42.755821+0000","last_became_peered":"2024-09-17T04:01:42.755821+0000","last_unstale":"2024-09-17T04:01:43.768305+0000","last_undegraded":"2024-09-17T04:01:43.768305+0000","last_fullsized":"2024-09-17T04:01:43.768305+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:46:56.165743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929499+0000","last_change":"2024-09-17T04:01:42.753795+0000","last_active":"2024-09-17T04:01:43.929499+0000","last_peered":"2024-09-17T04:01:43.929499+0000","last_clean":"2024-09-17T04:01:43.929499+0000","last_became_active":"2024-09-17T04:01:42.753070+0000","last_became_peered":"2024-09-17T04:01:42.753070+0000","last_unstale":"2024-09-17T04:01:43.929499+0000","last_undegraded":"2024-09-17T04:01:43.929499+0000","last_fullsized":"2024-09-17T04:01:43.929499+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:27:57.895981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763766+0000","last_change":"2024-09-17T04:01:42.774411+0000","last_active":"2024-09-17T04:01:43.763766+0000","last_peered":"2024-09-17T04:01:43.763766+0000","last_clean":"2024-09-17T04:01:43.763766+0000","last_became_active":"2024-09-17T04:01:42.774092+0000","last_became_peered":"2024-09-17T04:01:42.774092+0000","last_unstale":"2024-09-17T04:01:43.763766+0000","last_undegraded":"2024-09-17T04:01:43.763766+0000","last_fullsized":"2024-09-17T04:01:43.763766+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:17:56.461318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767490+0000","last_change":"2024-09-17T04:01:42.771279+0000","last_active":"2024-09-17T04:01:43.767490+0000","last_peered":"2024-09-17T04:01:43.767490+0000","last_clean":"2024-09-17T04:01:43.767490+0000","last_became_active":"2024-09-17T04:01:42.770882+0000","last_became_peered":"2024-09-17T04:01:42.770882+0000","last_unstale":"2024-09-17T04:01:43.767490+0000","last_undegraded":"2024-09-17T04:01:43.767490+0000","last_fullsized":"2024-09-17T04:01:43.767490+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:48:34.277187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928408+0000","last_change":"2024-09-17T04:01:42.761573+0000","last_active":"2024-09-17T04:01:43.928408+0000","last_peered":"2024-09-17T04:01:43.928408+0000","last_clean":"2024-09-17T04:01:43.928408+0000","last_became_active":"2024-09-17T04:01:42.760888+0000","last_became_peered":"2024-09-17T04:01:42.760888+0000","last_unstale":"2024-09-17T04:01:43.928408+0000","last_undegraded":"2024-09-17T04:01:43.928408+0000","last_fullsized":"2024-09-17T04:01:43.928408+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:44.756271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.771583+0000","last_change":"2024-09-17T04:01:42.770126+0000","last_active":"2024-09-17T04:01:42.771583+0000","last_peered":"2024-09-17T04:01:42.771583+0000","last_clean":"2024-09-17T04:01:42.771583+0000","last_became_active":"2024-09-17T04:01:42.769786+0000","last_became_peered":"2024-09-17T04:01:42.769786+0000","last_unstale":"2024-09-17T04:01:42.771583+0000","last_undegraded":"2024-09-17T04:01:42.771583+0000","last_fullsized":"2024-09-17T04:01:42.771583+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:57.455421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786818+0000","last_change":"2024-09-17T04:01:42.781821+0000","last_active":"2024-09-17T04:01:42.786818+0000","last_peered":"2024-09-17T04:01:42.786818+0000","last_clean":"2024-09-17T04:01:42.786818+0000","last_became_active":"2024-09-17T04:01:42.777720+0000","last_became_peered":"2024-09-17T04:01:42.777720+0000","last_unstale":"2024-09-17T04:01:42.786818+0000","last_undegraded":"2024-09-17T04:01:42.786818+0000","last_fullsized":"2024-09-17T04:01:42.786818+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:25.256917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"185'4","reported_seq":17,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785859+0000","last_change":"2024-09-17T04:01:41.746448+0000","last_active":"2024-09-17T04:01:42.785859+0000","last_peered":"2024-09-17T04:01:42.785859+0000","last_clean":"2024-09-17T04:01:42.785859+0000","last_became_active":"2024-09-17T04:01:41.746146+0000","last_became_peered":"2024-09-17T04:01:41.746146+0000","last_unstale":"2024-09-17T04:01:42.785859+0000","last_undegraded":"2024-09-17T04:01:42.785859+0000","last_fullsized":"2024-09-17T04:01:42.785859+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:36.991555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763802+0000","last_change":"2024-09-17T04:01:42.762618+0000","last_active":"2024-09-17T04:01:43.763802+0000","last_peered":"2024-09-17T04:01:43.763802+0000","last_clean":"2024-09-17T04:01:43.763802+0000","last_became_active":"2024-09-17T04:01:42.762264+0000","last_became_peered":"2024-09-17T04:01:42.762264+0000","last_unstale":"2024-09-17T04:01:43.763802+0000","last_undegraded":"2024-09-17T04:01:43.763802+0000","last_fullsized":"2024-09-17T04:01:43.763802+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:02.551696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928880+0000","last_change":"2024-09-17T04:01:41.753761+0000","last_active":"2024-09-17T04:01:43.928880+0000","last_peered":"2024-09-17T04:01:43.928880+0000","last_clean":"2024-09-17T04:01:43.928880+0000","last_became_active":"2024-09-17T04:01:41.753432+0000","last_became_peered":"2024-09-17T04:01:41.753432+0000","last_unstale":"2024-09-17T04:01:43.928880+0000","last_undegraded":"2024-09-17T04:01:43.928880+0000","last_fullsized":"2024-09-17T04:01:43.928880+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:15.229671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764110+0000","last_change":"2024-09-17T04:01:42.781189+0000","last_active":"2024-09-17T04:01:43.764110+0000","last_peered":"2024-09-17T04:01:43.764110+0000","last_clean":"2024-09-17T04:01:43.764110+0000","last_became_active":"2024-09-17T04:01:42.780868+0000","last_became_peered":"2024-09-17T04:01:42.780868+0000","last_unstale":"2024-09-17T04:01:43.764110+0000","last_undegraded":"2024-09-17T04:01:43.764110+0000","last_fullsized":"2024-09-17T04:01:43.764110+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:10:07.494776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1c","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928977+0000","last_change":"2024-09-17T04:01:41.742401+0000","last_active":"2024-09-17T04:01:43.928977+0000","last_peered":"2024-09-17T04:01:43.928977+0000","last_clean":"2024-09-17T04:01:43.928977+0000","last_became_active":"2024-09-17T04:01:41.741938+0000","last_became_peered":"2024-09-17T04:01:41.741938+0000","last_unstale":"2024-09-17T04:01:43.928977+0000","last_undegraded":"2024-09-17T04:01:43.928977+0000","last_fullsized":"2024-09-17T04:01:43.928977+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:47:13.191449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767602+0000","last_change":"2024-09-17T04:01:42.778848+0000","last_active":"2024-09-17T04:01:43.767602+0000","last_peered":"2024-09-17T04:01:43.767602+0000","last_clean":"2024-09-17T04:01:43.767602+0000","last_became_active":"2024-09-17T04:01:42.778550+0000","last_became_peered":"2024-09-17T04:01:42.778550+0000","last_unstale":"2024-09-17T04:01:43.767602+0000","last_undegraded":"2024-09-17T04:01:43.767602+0000","last_fullsized":"2024-09-17T04:01:43.767602+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:16.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763677+0000","last_change":"2024-09-17T04:01:41.753169+0000","last_active":"2024-09-17T04:01:43.763677+0000","last_peered":"2024-09-17T04:01:43.763677+0000","last_clean":"2024-09-17T04:01:43.763677+0000","last_became_active":"2024-09-17T04:01:41.752374+0000","last_became_peered":"2024-09-17T04:01:41.752374+0000","last_unstale":"2024-09-17T04:01:43.763677+0000","last_undegraded":"2024-09-17T04:01:43.763677+0000","last_fullsized":"2024-09-17T04:01:43.763677+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:32.043936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928863+0000","last_change":"2024-09-17T04:01:42.766186+0000","last_active":"2024-09-17T04:01:43.928863+0000","last_peered":"2024-09-17T04:01:43.928863+0000","last_clean":"2024-09-17T04:01:43.928863+0000","last_became_active":"2024-09-17T04:01:42.765826+0000","last_became_peered":"2024-09-17T04:01:42.765826+0000","last_unstale":"2024-09-17T04:01:43.928863+0000","last_undegraded":"2024-09-17T04:01:43.928863+0000","last_fullsized":"2024-09-17T04:01:43.928863+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:29.135439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1a","version":"185'1","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785954+0000","last_change":"2024-09-17T04:01:41.756151+0000","last_active":"2024-09-17T04:01:42.785954+0000","last_peered":"2024-09-17T04:01:42.785954+0000","last_clean":"2024-09-17T04:01:42.785954+0000","last_became_active":"2024-09-17T04:01:41.754993+0000","last_became_peered":"2024-09-17T04:01:41.754993+0000","last_unstale":"2024-09-17T04:01:42.785954+0000","last_undegraded":"2024-09-17T04:01:42.785954+0000","last_fullsized":"2024-09-17T04:01:42.785954+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:37.786276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928622+0000","last_change":"2024-09-17T04:01:42.752218+0000","last_active":"2024-09-17T04:01:43.928622+0000","last_peered":"2024-09-17T04:01:43.928622+0000","last_clean":"2024-09-17T04:01:43.928622+0000","last_became_active":"2024-09-17T04:01:42.751907+0000","last_became_peered":"2024-09-17T04:01:42.751907+0000","last_unstale":"2024-09-17T04:01:43.928622+0000","last_undegraded":"2024-09-17T04:01:43.928622+0000","last_fullsized":"2024-09-17T04:01:43.928622+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:28.826108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763349+0000","last_change":"2024-09-17T04:01:41.743886+0000","last_active":"2024-09-17T04:01:43.763349+0000","last_peered":"2024-09-17T04:01:43.763349+0000","last_clean":"2024-09-17T04:01:43.763349+0000","last_became_active":"2024-09-17T04:01:41.743545+0000","last_became_peered":"2024-09-17T04:01:41.743545+0000","last_unstale":"2024-09-17T04:01:43.763349+0000","last_undegraded":"2024-09-17T04:01:43.763349+0000","last_fullsized":"2024-09-17T04:01:43.763349+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:06.274197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929114+0000","last_change":"2024-09-17T04:01:42.766977+0000","last_active":"2024-09-17T04:01:43.929114+0000","last_peered":"2024-09-17T04:01:43.929114+0000","last_clean":"2024-09-17T04:01:43.929114+0000","last_became_active":"2024-09-17T04:01:42.766577+0000","last_became_peered":"2024-09-17T04:01:42.766577+0000","last_unstale":"2024-09-17T04:01:43.929114+0000","last_undegraded":"2024-09-17T04:01:43.929114+0000","last_fullsized":"2024-09-17T04:01:43.929114+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:49.114199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928747+0000","last_change":"2024-09-17T04:01:41.740632+0000","last_active":"2024-09-17T04:01:43.928747+0000","last_peered":"2024-09-17T04:01:43.928747+0000","last_clean":"2024-09-17T04:01:43.928747+0000","last_became_active":"2024-09-17T04:01:41.739898+0000","last_became_peered":"2024-09-17T04:01:41.739898+0000","last_unstale":"2024-09-17T04:01:43.928747+0000","last_undegraded":"2024-09-17T04:01:43.928747+0000","last_fullsized":"2024-09-17T04:01:43.928747+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:02:05.721314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929007+0000","last_change":"2024-09-17T04:01:42.784572+0000","last_active":"2024-09-17T04:01:43.929007+0000","last_peered":"2024-09-17T04:01:43.929007+0000","last_clean":"2024-09-17T04:01:43.929007+0000","last_became_active":"2024-09-17T04:01:42.783908+0000","last_became_peered":"2024-09-17T04:01:42.783908+0000","last_unstale":"2024-09-17T04:01:43.929007+0000","last_undegraded":"2024-09-17T04:01:43.929007+0000","last_fullsized":"2024-09-17T04:01:43.929007+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.656123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768194+0000","last_change":"2024-09-17T04:01:41.757459+0000","last_active":"2024-09-17T04:01:43.768194+0000","last_peered":"2024-09-17T04:01:43.768194+0000","last_clean":"2024-09-17T04:01:43.768194+0000","last_became_active":"2024-09-17T04:01:41.756976+0000","last_became_peered":"2024-09-17T04:01:41.756976+0000","last_unstale":"2024-09-17T04:01:43.768194+0000","last_undegraded":"2024-09-17T04:01:43.768194+0000","last_fullsized":"2024-09-17T04:01:43.768194+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:12.122874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768157+0000","last_change":"2024-09-17T04:01:42.771577+0000","last_active":"2024-09-17T04:01:43.768157+0000","last_peered":"2024-09-17T04:01:43.768157+0000","last_clean":"2024-09-17T04:01:43.768157+0000","last_became_active":"2024-09-17T04:01:42.770786+0000","last_became_peered":"2024-09-17T04:01:42.770786+0000","last_unstale":"2024-09-17T04:01:43.768157+0000","last_undegraded":"2024-09-17T04:01:43.768157+0000","last_fullsized":"2024-09-17T04:01:43.768157+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:04.962338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.749242+0000","last_change":"2024-09-17T04:01:41.751864+0000","last_active":"2024-09-17T04:01:42.749242+0000","last_peered":"2024-09-17T04:01:42.749242+0000","last_clean":"2024-09-17T04:01:42.749242+0000","last_became_active":"2024-09-17T04:01:41.751558+0000","last_became_peered":"2024-09-17T04:01:41.751558+0000","last_unstale":"2024-09-17T04:01:42.749242+0000","last_undegraded":"2024-09-17T04:01:42.749242+0000","last_fullsized":"2024-09-17T04:01:42.749242+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:25:12.506890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763854+0000","last_change":"2024-09-17T04:01:42.771648+0000","last_active":"2024-09-17T04:01:43.763854+0000","last_peered":"2024-09-17T04:01:43.763854+0000","last_clean":"2024-09-17T04:01:43.763854+0000","last_became_active":"2024-09-17T04:01:42.771328+0000","last_became_peered":"2024-09-17T04:01:42.771328+0000","last_unstale":"2024-09-17T04:01:43.763854+0000","last_undegraded":"2024-09-17T04:01:43.763854+0000","last_fullsized":"2024-09-17T04:01:43.763854+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:06.353603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.15","version":"185'3","reported_seq":18,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929835+0000","last_change":"2024-09-17T04:01:41.753062+0000","last_active":"2024-09-17T04:01:43.929835+0000","last_peered":"2024-09-17T04:01:43.929835+0000","last_clean":"2024-09-17T04:01:43.929835+0000","last_became_active":"2024-09-17T04:01:41.752724+0000","last_became_peered":"2024-09-17T04:01:41.752724+0000","last_unstale":"2024-09-17T04:01:43.929835+0000","last_undegraded":"2024-09-17T04:01:43.929835+0000","last_fullsized":"2024-09-17T04:01:43.929835+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:35.015903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767948+0000","last_change":"2024-09-17T04:01:42.767460+0000","last_active":"2024-09-17T04:01:43.767948+0000","last_peered":"2024-09-17T04:01:43.767948+0000","last_clean":"2024-09-17T04:01:43.767948+0000","last_became_active":"2024-09-17T04:01:42.766109+0000","last_became_peered":"2024-09-17T04:01:42.766109+0000","last_unstale":"2024-09-17T04:01:43.767948+0000","last_undegraded":"2024-09-17T04:01:43.767948+0000","last_fullsized":"2024-09-17T04:01:43.767948+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:41.230265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"185'6","reported_seq":21,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929555+0000","last_change":"2024-09-17T04:01:41.755339+0000","last_active":"2024-09-17T04:01:43.929555+0000","last_peered":"2024-09-17T04:01:43.929555+0000","last_clean":"2024-09-17T04:01:43.929555+0000","last_became_active":"2024-09-17T04:01:41.754514+0000","last_became_peered":"2024-09-17T04:01:41.754514+0000","last_unstale":"2024-09-17T04:01:43.929555+0000","last_undegraded":"2024-09-17T04:01:43.929555+0000","last_fullsized":"2024-09-17T04:01:43.929555+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:49.451867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765887+0000","last_change":"2024-09-17T04:01:42.760950+0000","last_active":"2024-09-17T04:01:42.765887+0000","last_peered":"2024-09-17T04:01:42.765887+0000","last_clean":"2024-09-17T04:01:42.765887+0000","last_became_active":"2024-09-17T04:01:42.760631+0000","last_became_peered":"2024-09-17T04:01:42.760631+0000","last_unstale":"2024-09-17T04:01:42.765887+0000","last_undegraded":"2024-09-17T04:01:42.765887+0000","last_fullsized":"2024-09-17T04:01:42.765887+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:20:11.054394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763796+0000","last_change":"2024-09-17T04:01:41.756584+0000","last_active":"2024-09-17T04:01:43.763796+0000","last_peered":"2024-09-17T04:01:43.763796+0000","last_clean":"2024-09-17T04:01:43.763796+0000","last_became_active":"2024-09-17T04:01:41.755819+0000","last_became_peered":"2024-09-17T04:01:41.755819+0000","last_unstale":"2024-09-17T04:01:43.763796+0000","last_undegraded":"2024-09-17T04:01:43.763796+0000","last_fullsized":"2024-09-17T04:01:43.763796+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:25:13.276461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785167+0000","last_change":"2024-09-17T04:01:42.777226+0000","last_active":"2024-09-17T04:01:42.785167+0000","last_peered":"2024-09-17T04:01:42.785167+0000","last_clean":"2024-09-17T04:01:42.785167+0000","last_became_active":"2024-09-17T04:01:42.767386+0000","last_became_peered":"2024-09-17T04:01:42.767386+0000","last_unstale":"2024-09-17T04:01:42.785167+0000","last_undegraded":"2024-09-17T04:01:42.785167+0000","last_fullsized":"2024-09-17T04:01:42.785167+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:27:19.847104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.750670+0000","last_change":"2024-09-17T04:01:41.752773+0000","last_active":"2024-09-17T04:01:42.750670+0000","last_peered":"2024-09-17T04:01:42.750670+0000","last_clean":"2024-09-17T04:01:42.750670+0000","last_became_active":"2024-09-17T04:01:41.752369+0000","last_became_peered":"2024-09-17T04:01:41.752369+0000","last_unstale":"2024-09-17T04:01:42.750670+0000","last_undegraded":"2024-09-17T04:01:42.750670+0000","last_fullsized":"2024-09-17T04:01:42.750670+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:51:50.688529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786996+0000","last_change":"2024-09-17T04:01:42.784206+0000","last_active":"2024-09-17T04:01:42.786996+0000","last_peered":"2024-09-17T04:01:42.786996+0000","last_clean":"2024-09-17T04:01:42.786996+0000","last_became_active":"2024-09-17T04:01:42.783874+0000","last_became_peered":"2024-09-17T04:01:42.783874+0000","last_unstale":"2024-09-17T04:01:42.786996+0000","last_undegraded":"2024-09-17T04:01:42.786996+0000","last_fullsized":"2024-09-17T04:01:42.786996+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:28.801426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928622+0000","last_change":"2024-09-17T04:01:41.756092+0000","last_active":"2024-09-17T04:01:43.928622+0000","last_peered":"2024-09-17T04:01:43.928622+0000","last_clean":"2024-09-17T04:01:43.928622+0000","last_became_active":"2024-09-17T04:01:41.754932+0000","last_became_peered":"2024-09-17T04:01:41.754932+0000","last_unstale":"2024-09-17T04:01:43.928622+0000","last_undegraded":"2024-09-17T04:01:43.928622+0000","last_fullsized":"2024-09-17T04:01:43.928622+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:55:06.681949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763498+0000","last_change":"2024-09-17T04:01:42.765088+0000","last_active":"2024-09-17T04:01:43.763498+0000","last_peered":"2024-09-17T04:01:43.763498+0000","last_clean":"2024-09-17T04:01:43.763498+0000","last_became_active":"2024-09-17T04:01:42.764764+0000","last_became_peered":"2024-09-17T04:01:42.764764+0000","last_unstale":"2024-09-17T04:01:43.763498+0000","last_undegraded":"2024-09-17T04:01:43.763498+0000","last_fullsized":"2024-09-17T04:01:43.763498+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:55.375890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.10","version":"185'8","reported_seq":23,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928711+0000","last_change":"2024-09-17T04:01:41.748879+0000","last_active":"2024-09-17T04:01:43.928711+0000","last_peered":"2024-09-17T04:01:43.928711+0000","last_clean":"2024-09-17T04:01:43.928711+0000","last_became_active":"2024-09-17T04:01:41.747046+0000","last_became_peered":"2024-09-17T04:01:41.747046+0000","last_unstale":"2024-09-17T04:01:43.928711+0000","last_undegraded":"2024-09-17T04:01:43.928711+0000","last_fullsized":"2024-09-17T04:01:43.928711+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:12:05.347003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763721+0000","last_change":"2024-09-17T04:01:42.767825+0000","last_active":"2024-09-17T04:01:43.763721+0000","last_peered":"2024-09-17T04:01:43.763721+0000","last_clean":"2024-09-17T04:01:43.763721+0000","last_became_active":"2024-09-17T04:01:42.767519+0000","last_became_peered":"2024-09-17T04:01:42.767519+0000","last_unstale":"2024-09-17T04:01:43.763721+0000","last_undegraded":"2024-09-17T04:01:43.763721+0000","last_fullsized":"2024-09-17T04:01:43.763721+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:51:49.098369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"185'1","reported_seq":14,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.785186+0000","last_change":"2024-09-17T04:01:41.756724+0000","last_active":"2024-09-17T04:01:42.785186+0000","last_peered":"2024-09-17T04:01:42.785186+0000","last_clean":"2024-09-17T04:01:42.785186+0000","last_became_active":"2024-09-17T04:01:41.756397+0000","last_became_peered":"2024-09-17T04:01:41.756397+0000","last_unstale":"2024-09-17T04:01:42.785186+0000","last_undegraded":"2024-09-17T04:01:42.785186+0000","last_fullsized":"2024-09-17T04:01:42.785186+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:53:37.545097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929524+0000","last_change":"2024-09-17T04:01:42.770508+0000","last_active":"2024-09-17T04:01:43.929524+0000","last_peered":"2024-09-17T04:01:43.929524+0000","last_clean":"2024-09-17T04:01:43.929524+0000","last_became_active":"2024-09-17T04:01:42.769764+0000","last_became_peered":"2024-09-17T04:01:42.769764+0000","last_unstale":"2024-09-17T04:01:43.929524+0000","last_undegraded":"2024-09-17T04:01:43.929524+0000","last_fullsized":"2024-09-17T04:01:43.929524+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:39.353840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"185'2","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.780753+0000","last_change":"2024-09-17T04:01:41.751012+0000","last_active":"2024-09-17T04:01:42.780753+0000","last_peered":"2024-09-17T04:01:42.780753+0000","last_clean":"2024-09-17T04:01:42.780753+0000","last_became_active":"2024-09-17T04:01:41.749862+0000","last_became_peered":"2024-09-17T04:01:41.749862+0000","last_unstale":"2024-09-17T04:01:42.780753+0000","last_undegraded":"2024-09-17T04:01:42.780753+0000","last_fullsized":"2024-09-17T04:01:42.780753+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:06.418878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786448+0000","last_change":"2024-09-17T04:01:42.778201+0000","last_active":"2024-09-17T04:01:42.786448+0000","last_peered":"2024-09-17T04:01:42.786448+0000","last_clean":"2024-09-17T04:01:42.786448+0000","last_became_active":"2024-09-17T04:01:42.777811+0000","last_became_peered":"2024-09-17T04:01:42.777811+0000","last_unstale":"2024-09-17T04:01:42.786448+0000","last_undegraded":"2024-09-17T04:01:42.786448+0000","last_fullsized":"2024-09-17T04:01:42.786448+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:01:29.314779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.d","version":"185'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768114+0000","last_change":"2024-09-17T04:01:41.753579+0000","last_active":"2024-09-17T04:01:43.768114+0000","last_peered":"2024-09-17T04:01:43.768114+0000","last_clean":"2024-09-17T04:01:43.768114+0000","last_became_active":"2024-09-17T04:01:41.753259+0000","last_became_peered":"2024-09-17T04:01:41.753259+0000","last_unstale":"2024-09-17T04:01:43.768114+0000","last_undegraded":"2024-09-17T04:01:43.768114+0000","last_fullsized":"2024-09-17T04:01:43.768114+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:21:42.324911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786868+0000","last_change":"2024-09-17T04:01:42.778845+0000","last_active":"2024-09-17T04:01:42.786868+0000","last_peered":"2024-09-17T04:01:42.786868+0000","last_clean":"2024-09-17T04:01:42.786868+0000","last_became_active":"2024-09-17T04:01:42.778269+0000","last_became_peered":"2024-09-17T04:01:42.778269+0000","last_unstale":"2024-09-17T04:01:42.786868+0000","last_undegraded":"2024-09-17T04:01:42.786868+0000","last_fullsized":"2024-09-17T04:01:42.786868+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:20:30.141000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929641+0000","last_change":"2024-09-17T04:01:41.746140+0000","last_active":"2024-09-17T04:01:43.929641+0000","last_peered":"2024-09-17T04:01:43.929641+0000","last_clean":"2024-09-17T04:01:43.929641+0000","last_became_active":"2024-09-17T04:01:41.744931+0000","last_became_peered":"2024-09-17T04:01:41.744931+0000","last_unstale":"2024-09-17T04:01:43.929641+0000","last_undegraded":"2024-09-17T04:01:43.929641+0000","last_fullsized":"2024-09-17T04:01:43.929641+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:03.651538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928387+0000","last_change":"2024-09-17T04:01:42.775567+0000","last_active":"2024-09-17T04:01:43.928387+0000","last_peered":"2024-09-17T04:01:43.928387+0000","last_clean":"2024-09-17T04:01:43.928387+0000","last_became_active":"2024-09-17T04:01:42.775147+0000","last_became_peered":"2024-09-17T04:01:42.775147+0000","last_unstale":"2024-09-17T04:01:43.928387+0000","last_undegraded":"2024-09-17T04:01:43.928387+0000","last_fullsized":"2024-09-17T04:01:43.928387+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:05.524401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.b","version":"185'2","reported_seq":15,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.787252+0000","last_change":"2024-09-17T04:01:41.756084+0000","last_active":"2024-09-17T04:01:42.787252+0000","last_peered":"2024-09-17T04:01:42.787252+0000","last_clean":"2024-09-17T04:01:42.787252+0000","last_became_active":"2024-09-17T04:01:41.755765+0000","last_became_peered":"2024-09-17T04:01:41.755765+0000","last_unstale":"2024-09-17T04:01:42.787252+0000","last_undegraded":"2024-09-17T04:01:42.787252+0000","last_fullsized":"2024-09-17T04:01:42.787252+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:49.605022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.786857+0000","last_change":"2024-09-17T04:01:42.780860+0000","last_active":"2024-09-17T04:01:42.786857+0000","last_peered":"2024-09-17T04:01:42.786857+0000","last_clean":"2024-09-17T04:01:42.786857+0000","last_became_active":"2024-09-17T04:01:42.780520+0000","last_became_peered":"2024-09-17T04:01:42.780520+0000","last_unstale":"2024-09-17T04:01:42.786857+0000","last_undegraded":"2024-09-17T04:01:42.786857+0000","last_fullsized":"2024-09-17T04:01:42.786857+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:00.582115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763835+0000","last_change":"2024-09-17T04:01:41.745037+0000","last_active":"2024-09-17T04:01:43.763835+0000","last_peered":"2024-09-17T04:01:43.763835+0000","last_clean":"2024-09-17T04:01:43.763835+0000","last_became_active":"2024-09-17T04:01:41.744675+0000","last_became_peered":"2024-09-17T04:01:41.744675+0000","last_unstale":"2024-09-17T04:01:43.763835+0000","last_undegraded":"2024-09-17T04:01:43.763835+0000","last_fullsized":"2024-09-17T04:01:43.763835+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:06:01.467030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763356+0000","last_change":"2024-09-17T04:01:42.767153+0000","last_active":"2024-09-17T04:01:43.763356+0000","last_peered":"2024-09-17T04:01:43.763356+0000","last_clean":"2024-09-17T04:01:43.763356+0000","last_became_active":"2024-09-17T04:01:42.766844+0000","last_became_peered":"2024-09-17T04:01:42.766844+0000","last_unstale":"2024-09-17T04:01:43.763356+0000","last_undegraded":"2024-09-17T04:01:43.763356+0000","last_fullsized":"2024-09-17T04:01:43.763356+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:00:21.337125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763998+0000","last_change":"2024-09-17T04:01:41.754201+0000","last_active":"2024-09-17T04:01:43.763998+0000","last_peered":"2024-09-17T04:01:43.763998+0000","last_clean":"2024-09-17T04:01:43.763998+0000","last_became_active":"2024-09-17T04:01:41.753871+0000","last_became_peered":"2024-09-17T04:01:41.753871+0000","last_unstale":"2024-09-17T04:01:43.763998+0000","last_undegraded":"2024-09-17T04:01:43.763998+0000","last_fullsized":"2024-09-17T04:01:43.763998+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:08:07.297101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928642+0000","last_change":"2024-09-17T04:01:42.759404+0000","last_active":"2024-09-17T04:01:43.928642+0000","last_peered":"2024-09-17T04:01:43.928642+0000","last_clean":"2024-09-17T04:01:43.928642+0000","last_became_active":"2024-09-17T04:01:42.758839+0000","last_became_peered":"2024-09-17T04:01:42.758839+0000","last_unstale":"2024-09-17T04:01:43.928642+0000","last_undegraded":"2024-09-17T04:01:43.928642+0000","last_fullsized":"2024-09-17T04:01:43.928642+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:26:08.698710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765276+0000","last_change":"2024-09-17T04:01:41.754990+0000","last_active":"2024-09-17T04:01:42.765276+0000","last_peered":"2024-09-17T04:01:42.765276+0000","last_clean":"2024-09-17T04:01:42.765276+0000","last_became_active":"2024-09-17T04:01:41.754659+0000","last_became_peered":"2024-09-17T04:01:41.754659+0000","last_unstale":"2024-09-17T04:01:42.765276+0000","last_undegraded":"2024-09-17T04:01:42.765276+0000","last_fullsized":"2024-09-17T04:01:42.765276+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:40:14.252648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929845+0000","last_change":"2024-09-17T04:01:42.772854+0000","last_active":"2024-09-17T04:01:43.929845+0000","last_peered":"2024-09-17T04:01:43.929845+0000","last_clean":"2024-09-17T04:01:43.929845+0000","last_became_active":"2024-09-17T04:01:42.772538+0000","last_became_peered":"2024-09-17T04:01:42.772538+0000","last_unstale":"2024-09-17T04:01:43.929845+0000","last_undegraded":"2024-09-17T04:01:43.929845+0000","last_fullsized":"2024-09-17T04:01:43.929845+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:25:28.826108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.7","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929938+0000","last_change":"2024-09-17T04:01:41.759215+0000","last_active":"2024-09-17T04:01:43.929938+0000","last_peered":"2024-09-17T04:01:43.929938+0000","last_clean":"2024-09-17T04:01:43.929938+0000","last_became_active":"2024-09-17T04:01:41.758958+0000","last_became_peered":"2024-09-17T04:01:41.758958+0000","last_unstale":"2024-09-17T04:01:43.929938+0000","last_undegraded":"2024-09-17T04:01:43.929938+0000","last_fullsized":"2024-09-17T04:01:43.929938+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:31:26.610681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781839+0000","last_change":"2024-09-17T04:01:42.771571+0000","last_active":"2024-09-17T04:01:42.781839+0000","last_peered":"2024-09-17T04:01:42.781839+0000","last_clean":"2024-09-17T04:01:42.781839+0000","last_became_active":"2024-09-17T04:01:42.771264+0000","last_became_peered":"2024-09-17T04:01:42.771264+0000","last_unstale":"2024-09-17T04:01:42.781839+0000","last_undegraded":"2024-09-17T04:01:42.781839+0000","last_fullsized":"2024-09-17T04:01:42.781839+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:50:04.247007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.6","version":"186'1","reported_seq":16,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.769667+0000","last_change":"2024-09-17T04:01:41.759965+0000","last_active":"2024-09-17T04:01:43.769667+0000","last_peered":"2024-09-17T04:01:43.769667+0000","last_clean":"2024-09-17T04:01:43.769667+0000","last_became_active":"2024-09-17T04:01:41.759584+0000","last_became_peered":"2024-09-17T04:01:41.759584+0000","last_unstale":"2024-09-17T04:01:43.769667+0000","last_undegraded":"2024-09-17T04:01:43.769667+0000","last_fullsized":"2024-09-17T04:01:43.769667+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:39.152086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.782855+0000","last_change":"2024-09-17T04:01:42.775297+0000","last_active":"2024-09-17T04:01:42.782855+0000","last_peered":"2024-09-17T04:01:42.782855+0000","last_clean":"2024-09-17T04:01:42.782855+0000","last_became_active":"2024-09-17T04:01:42.773237+0000","last_became_peered":"2024-09-17T04:01:42.773237+0000","last_unstale":"2024-09-17T04:01:42.782855+0000","last_undegraded":"2024-09-17T04:01:42.782855+0000","last_fullsized":"2024-09-17T04:01:42.782855+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:39.396816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.768260+0000","last_change":"2024-09-17T04:01:41.758099+0000","last_active":"2024-09-17T04:01:43.768260+0000","last_peered":"2024-09-17T04:01:43.768260+0000","last_clean":"2024-09-17T04:01:43.768260+0000","last_became_active":"2024-09-17T04:01:41.756992+0000","last_became_peered":"2024-09-17T04:01:41.756992+0000","last_unstale":"2024-09-17T04:01:43.768260+0000","last_undegraded":"2024-09-17T04:01:43.768260+0000","last_fullsized":"2024-09-17T04:01:43.768260+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:22.619173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.929711+0000","last_change":"2024-09-17T04:01:42.761322+0000","last_active":"2024-09-17T04:01:43.929711+0000","last_peered":"2024-09-17T04:01:43.929711+0000","last_clean":"2024-09-17T04:01:43.929711+0000","last_became_active":"2024-09-17T04:01:42.761023+0000","last_became_peered":"2024-09-17T04:01:42.761023+0000","last_unstale":"2024-09-17T04:01:43.929711+0000","last_undegraded":"2024-09-17T04:01:43.929711+0000","last_fullsized":"2024-09-17T04:01:43.929711+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:48:16.178281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.772280+0000","last_change":"2024-09-17T04:01:41.754588+0000","last_active":"2024-09-17T04:01:42.772280+0000","last_peered":"2024-09-17T04:01:42.772280+0000","last_clean":"2024-09-17T04:01:42.772280+0000","last_became_active":"2024-09-17T04:01:41.754254+0000","last_became_peered":"2024-09-17T04:01:41.754254+0000","last_unstale":"2024-09-17T04:01:42.772280+0000","last_undegraded":"2024-09-17T04:01:42.772280+0000","last_fullsized":"2024-09-17T04:01:42.772280+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:10.541971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928885+0000","last_change":"2024-09-17T04:01:42.761957+0000","last_active":"2024-09-17T04:01:43.928885+0000","last_peered":"2024-09-17T04:01:43.928885+0000","last_clean":"2024-09-17T04:01:43.928885+0000","last_became_active":"2024-09-17T04:01:42.760954+0000","last_became_peered":"2024-09-17T04:01:42.760954+0000","last_unstale":"2024-09-17T04:01:43.928885+0000","last_undegraded":"2024-09-17T04:01:43.928885+0000","last_fullsized":"2024-09-17T04:01:43.928885+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:12:25.422065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.763702+0000","last_change":"2024-09-17T04:01:41.746613+0000","last_active":"2024-09-17T04:01:43.763702+0000","last_peered":"2024-09-17T04:01:43.763702+0000","last_clean":"2024-09-17T04:01:43.763702+0000","last_became_active":"2024-09-17T04:01:41.745078+0000","last_became_peered":"2024-09-17T04:01:41.745078+0000","last_unstale":"2024-09-17T04:01:43.763702+0000","last_undegraded":"2024-09-17T04:01:43.763702+0000","last_fullsized":"2024-09-17T04:01:43.763702+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:03.719713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781647+0000","last_change":"2024-09-17T04:01:42.769926+0000","last_active":"2024-09-17T04:01:42.781647+0000","last_peered":"2024-09-17T04:01:42.781647+0000","last_clean":"2024-09-17T04:01:42.781647+0000","last_became_active":"2024-09-17T04:01:42.766047+0000","last_became_peered":"2024-09-17T04:01:42.766047+0000","last_unstale":"2024-09-17T04:01:42.781647+0000","last_undegraded":"2024-09-17T04:01:42.781647+0000","last_fullsized":"2024-09-17T04:01:42.781647+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:37:58.970219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.750899+0000","last_change":"2024-09-17T04:01:41.757010+0000","last_active":"2024-09-17T04:01:42.750899+0000","last_peered":"2024-09-17T04:01:42.750899+0000","last_clean":"2024-09-17T04:01:42.750899+0000","last_became_active":"2024-09-17T04:01:41.756668+0000","last_became_peered":"2024-09-17T04:01:41.756668+0000","last_unstale":"2024-09-17T04:01:42.750899+0000","last_undegraded":"2024-09-17T04:01:42.750899+0000","last_fullsized":"2024-09-17T04:01:42.750899+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:14:13.736891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.781817+0000","last_change":"2024-09-17T04:01:42.774458+0000","last_active":"2024-09-17T04:01:42.781817+0000","last_peered":"2024-09-17T04:01:42.781817+0000","last_clean":"2024-09-17T04:01:42.781817+0000","last_became_active":"2024-09-17T04:01:42.771159+0000","last_became_peered":"2024-09-17T04:01:42.771159+0000","last_unstale":"2024-09-17T04:01:42.781817+0000","last_undegraded":"2024-09-17T04:01:42.781817+0000","last_fullsized":"2024-09-17T04:01:42.781817+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:03.676708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":15,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.928725+0000","last_change":"2024-09-17T04:01:41.755626+0000","last_active":"2024-09-17T04:01:43.928725+0000","last_peered":"2024-09-17T04:01:43.928725+0000","last_clean":"2024-09-17T04:01:43.928725+0000","last_became_active":"2024-09-17T04:01:41.755345+0000","last_became_peered":"2024-09-17T04:01:41.755345+0000","last_unstale":"2024-09-17T04:01:43.928725+0000","last_undegraded":"2024-09-17T04:01:43.928725+0000","last_fullsized":"2024-09-17T04:01:43.928725+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:39.514567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767474+0000","last_change":"2024-09-17T04:01:42.771823+0000","last_active":"2024-09-17T04:01:43.767474+0000","last_peered":"2024-09-17T04:01:43.767474+0000","last_clean":"2024-09-17T04:01:43.767474+0000","last_became_active":"2024-09-17T04:01:42.769101+0000","last_became_peered":"2024-09-17T04:01:42.769101+0000","last_unstale":"2024-09-17T04:01:43.767474+0000","last_undegraded":"2024-09-17T04:01:43.767474+0000","last_fullsized":"2024-09-17T04:01:43.767474+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:07.630156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":13,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-17T04:01:42.765509+0000","last_change":"2024-09-17T04:01:41.753411+0000","last_active":"2024-09-17T04:01:42.765509+0000","last_peered":"2024-09-17T04:01:42.765509+0000","last_clean":"2024-09-17T04:01:42.765509+0000","last_became_active":"2024-09-17T04:01:41.752972+0000","last_became_peered":"2024-09-17T04:01:41.752972+0000","last_unstale":"2024-09-17T04:01:42.765509+0000","last_undegraded":"2024-09-17T04:01:42.765509+0000","last_fullsized":"2024-09-17T04:01:42.765509+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:56:32.697239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":13,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767594+0000","last_change":"2024-09-17T04:01:42.764031+0000","last_active":"2024-09-17T04:01:43.767594+0000","last_peered":"2024-09-17T04:01:43.767594+0000","last_clean":"2024-09-17T04:01:43.767594+0000","last_became_active":"2024-09-17T04:01:42.762106+0000","last_became_peered":"2024-09-17T04:01:42.762106+0000","last_unstale":"2024-09-17T04:01:43.767594+0000","last_undegraded":"2024-09-17T04:01:43.767594+0000","last_fullsized":"2024-09-17T04:01:43.767594+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":184,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:41.724678+0000","last_clean_scrub_stamp":"2024-09-17T04:01:41.724678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:56.512369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"185'2","reported_seq":17,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.764055+0000","last_change":"2024-09-17T04:01:41.755631+0000","last_active":"2024-09-17T04:01:43.764055+0000","last_peered":"2024-09-17T04:01:43.764055+0000","last_clean":"2024-09-17T04:01:43.764055+0000","last_became_active":"2024-09-17T04:01:41.755282+0000","last_became_peered":"2024-09-17T04:01:41.755282+0000","last_unstale":"2024-09-17T04:01:43.764055+0000","last_undegraded":"2024-09-17T04:01:43.764055+0000","last_fullsized":"2024-09-17T04:01:43.764055+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":183,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:01:40.718079+0000","last_clean_scrub_stamp":"2024-09-17T04:01:40.718079+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:04:33.798971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":414,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-17T04:01:43.767109+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:01:43.767109+0000","last_peered":"2024-09-17T04:01:43.767109+0000","last_clean":"2024-09-17T04:01:43.767109+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:01:43.767109+0000","last_undegraded":"2024-09-17T04:01:43.767109+0000","last_fullsized":"2024-09-17T04:01:43.767109+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509590,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42384,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329456,"statfs":{"total":96636764160,"available":96593362944,"internally_reserved":0,"allocated":2940928,"data_stored":1616651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509593,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2342912,"data_stored":1026213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509592,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":2347008,"data_stored":1026795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509590,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41816,"kb_used_data":2304,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330024,"statfs":{"total":96636764160,"available":96593944576,"internally_reserved":0,"allocated":2359296,"data_stored":1027895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37660,"kb_used_data":2308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334180,"statfs":{"total":96636764160,"available":96598200320,"internally_reserved":0,"allocated":2363392,"data_stored":1027881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509592,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41804,"kb_used_data":2292,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330036,"statfs":{"total":96636764160,"available":96593956864,"internally_reserved":0,"allocated":2347008,"data_stored":1026303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41796,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330044,"statfs":{"total":96636764160,"available":96593965056,"internally_reserved":0,"allocated":2338816,"data_stored":1019969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":2844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":2912256,"data_stored":1603506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:01:47.672 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:01:47.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:01:47.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 -- 172.21.15.129:0/1298725818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940703b0 msgr2=0x7fe0940b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 --2- 172.21.15.129:0/1298725818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940703b0 0x7fe0940b3750 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe07c005fd0 tx=0x7fe07c02aa90 comp rx=0 tx=0).stop 2024-09-17T04:01:47.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 -- 172.21.15.129:0/1298725818 shutdown_connections 2024-09-17T04:01:47.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 --2- 172.21.15.129:0/1298725818 >> v2:172.21.15.181:3300/0 conn(0x7fe0940b3c90 0x7fe09406eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 --2- 172.21.15.129:0/1298725818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940703b0 0x7fe0940b3750 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 -- 172.21.15.129:0/1298725818 >> 172.21.15.129:0/1298725818 conn(0x7fe09406d7a0 msgr2=0x7fe0940b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:47.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 -- 172.21.15.129:0/1298725818 shutdown_connections 2024-09-17T04:01:47.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.939+0000 7fe09a400640 1 -- 172.21.15.129:0/1298725818 wait complete. 2024-09-17T04:01:47.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.941+0000 7fe09a400640 1 Processor -- start 2024-09-17T04:01:47.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.941+0000 7fe09a400640 1 -- start start 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.942+0000 7fe09a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 0x7fe09409a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe09a400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940b3c90 0x7fe09414c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe09a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe09a400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe0940b2cb0 con 0x7fe0940b4d70 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe09a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe0940b2b30 con 0x7fe0940703b0 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe09a400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe09406f420 con 0x7fe0940b3c90 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44542/0 (socket says 172.21.15.129:44542) 2024-09-17T04:01:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 -- 172.21.15.129:0/2588165335 learned_addr learned my addr 172.21.15.129:0/2588165335 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:47.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe08aa00640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940b3c90 0x7fe09414c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 0x7fe09409a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 -- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940b3c90 msgr2=0x7fe09414c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940b3c90 0x7fe09414c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 -- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 msgr2=0x7fe09409a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 0x7fe09409a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093e00640 1 -- 172.21.15.129:0/2588165335 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09409ab30 con 0x7fe0940b4d70 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.943+0000 7fe093400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 0x7fe09409a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe093e00640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e940 tx=0x7fe08c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c00cd80 con 0x7fe0940b4d70 2024-09-17T04:01:47.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe08c004020 con 0x7fe0940b4d70 2024-09-17T04:01:47.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08c004470 con 0x7fe0940b4d70 2024-09-17T04:01:47.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09409ae10 con 0x7fe0940b4d70 2024-09-17T04:01:47.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.944+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09409b3d0 con 0x7fe0940b4d70 2024-09-17T04:01:47.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.945+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe09406f810 con 0x7fe0940b4d70 2024-09-17T04:01:47.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.947+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe08c0035f0 con 0x7fe0940b4d70 2024-09-17T04:01:47.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.948+0000 7fe08be00640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 0x7fe07007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:47.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.948+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fe08c09b810 con 0x7fe0940b4d70 2024-09-17T04:01:47.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.948+0000 7fe093400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 0x7fe07007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.949+0000 7fe093400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 0x7fe07007b0a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe0840016c0 tx=0x7fe084009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:47.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:47.952+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe08c063b30 con 0x7fe0940b4d70 2024-09-17T04:01:48.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:01:48.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:01:48.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.213+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fe09409a8c0 con 0x7fe0940b4d70 2024-09-17T04:01:48.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.216+0000 7fe08be00640 1 -- 172.21.15.129:0/2588165335 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v41) ==== 190+0+147 (secure 0 0 0) 0x7fe08c063430 con 0x7fe0940b4d70 2024-09-17T04:01:48.216 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:01:48.216 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:01:48.216 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:01:48.216 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:01:48.217 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:01:48.217 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 msgr2=0x7fe07007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 0x7fe07007b0a0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe0840016c0 tx=0x7fe084009040 comp rx=0 tx=0).stop 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 msgr2=0x7fe09415b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fe08c00e940 tx=0x7fe08c00ee00 comp rx=0 tx=0).stop 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 shutdown_connections 2024-09-17T04:01:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe070078b90 0x7fe07007b0a0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe0940b4d70 0x7fe09415b900 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe0940b3c90 0x7fe09414c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 --2- 172.21.15.129:0/2588165335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0940703b0 0x7fe09409a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 >> 172.21.15.129:0/2588165335 conn(0x7fe09406d7a0 msgr2=0x7fe0940b0fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 shutdown_connections 2024-09-17T04:01:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.219+0000 7fe09a400640 1 -- 172.21.15.129:0/2588165335 wait complete. 2024-09-17T04:01:48.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:48.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 -- 172.21.15.129:0/1270181722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0b4140 msgr2=0x7fd43c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 --2- 172.21.15.129:0/1270181722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0b4140 0x7fd43c0ccc80 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7fd43c072e00 tx=0x7fd42402a540 comp rx=0 tx=0).stop 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 -- 172.21.15.129:0/1270181722 shutdown_connections 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 --2- 172.21.15.129:0/1270181722 >> v2:172.21.15.181:3300/0 conn(0x7fd43c0cd1c0 0x7fd43c1609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 --2- 172.21.15.129:0/1270181722 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0b4140 0x7fd43c0ccc80 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 -- 172.21.15.129:0/1270181722 >> 172.21.15.129:0/1270181722 conn(0x7fd43c06d410 msgr2=0x7fd43c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 -- 172.21.15.129:0/1270181722 shutdown_connections 2024-09-17T04:01:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.461+0000 7fd440a00640 1 -- 172.21.15.129:0/1270181722 wait complete. 2024-09-17T04:01:48.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.464+0000 7fd440a00640 1 Processor -- start 2024-09-17T04:01:48.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.464+0000 7fd440a00640 1 -- start start 2024-09-17T04:01:48.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 0x7fd43c09d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 0x7fd43c09d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c160260 con 0x7fd43c0705a0 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd43c1600e0 con 0x7fd43c0b4140 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd440a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd43c160f00 con 0x7fd43c0cd1c0 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.465+0000 7fd439e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44560/0 (socket says 172.21.15.129:44560) 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 -- 172.21.15.129:0/2958235471 learned_addr learned my addr 172.21.15.129:0/2958235471 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd431400640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 0x7fd43c09d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd43a800640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 0x7fd43c09d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 -- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 msgr2=0x7fd43c09d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 0x7fd43c09d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 -- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 msgr2=0x7fd43c09d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 0x7fd43c09d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 -- 172.21.15.129:0/2958235471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43c0c7320 con 0x7fd43c0705a0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd43a800640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 0x7fd43c09d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd431400640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 0x7fd43c09d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd439e00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00c7d0 tx=0x7fd42c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.466+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c017070 con 0x7fd43c0705a0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.467+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd42c005070 con 0x7fd43c0705a0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.467+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0054a0 con 0x7fd43c0705a0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.467+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43c1f41d0 con 0x7fd43c0705a0 2024-09-17T04:01:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.467+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd43c1f4790 con 0x7fd43c0705a0 2024-09-17T04:01:48.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.468+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43c0ccc80 con 0x7fd43c0705a0 2024-09-17T04:01:48.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.469+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd42c01a070 con 0x7fd43c0705a0 2024-09-17T04:01:48.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.470+0000 7fd432a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 0x7fd41807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:48.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.470+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fd42c09a3d0 con 0x7fd43c0705a0 2024-09-17T04:01:48.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.470+0000 7fd431400640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 0x7fd41807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:48.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.471+0000 7fd431400640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 0x7fd41807b0a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd424034000 tx=0x7fd424004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:48.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.475+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd43c0cd5d0 con 0x7fd43c0705a0 2024-09-17T04:01:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.692+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd43c06a140 con 0x7fd43c0705a0 2024-09-17T04:01:48.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.693+0000 7fd432a00640 1 -- 172.21.15.129:0/2958235471 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5245 (secure 0 0 0) 0x7fd42c062800 con 0x7fd43c0705a0 2024-09-17T04:01:48.693 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:48.693 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":163,"btime":"2024-09-17T04:01:45:777482+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2944792499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2944792499},{"type":"v1","addr":"172.21.15.181:6837","nonce":2944792499}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158},{"gid":9205,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1569162292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1569162292},{"type":"v1","addr":"172.21.15.181:6836","nonce":1569162292}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157},{"gid":9208,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3011518213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3011518213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3011518213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":158}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:01:45.263805+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9903},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9903":{"gid":9903,"name":"a","rank":0,"incarnation":159,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3531032601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3531032601},{"type":"v1","addr":"172.21.15.129:6835","nonce":3531032601}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9903,"qdb_cluster":[9903]},"id":14}]} 2024-09-17T04:01:48.693 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 163 2024-09-17T04:01:48.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 msgr2=0x7fd41807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 0x7fd41807b0a0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd424034000 tx=0x7fd424004730 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 msgr2=0x7fd43c0c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00c7d0 tx=0x7fd42c00cc90 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 shutdown_connections 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd418078b90 0x7fd41807b0a0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd43c0cd1c0 0x7fd43c09d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd43c0b4140 0x7fd43c09d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 --2- 172.21.15.129:0/2958235471 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd43c0705a0 0x7fd43c0c9a30 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 >> 172.21.15.129:0/2958235471 conn(0x7fd43c06d410 msgr2=0x7fd43c072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 shutdown_connections 2024-09-17T04:01:48.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:48.695+0000 7fd440a00640 1 -- 172.21.15.129:0/2958235471 wait complete. 2024-09-17T04:01:48.723 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T04:01:48.724+0000 7f3667200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:01:49.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 163, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:01:42.223951+0000', 'modified': '2024-09-17T04:01:45.263805+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9903}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9903': {'gid': 9903, 'name': 'a', 'rank': 0, 'incarnation': 159, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3531032601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 3531032601}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3531032601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9903, 'qdb_cluster': [9903]} 2024-09-17T04:01:49.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:01:49.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T04:01:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 -- 172.21.15.129:0/2646716383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e00b2950 msgr2=0x7f78e006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 --2- 172.21.15.129:0/2646716383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e00b2950 0x7f78e006eed0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005fd0 tx=0x7f78dc02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 -- 172.21.15.129:0/2646716383 shutdown_connections 2024-09-17T04:01:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 --2- 172.21.15.129:0/2646716383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e00b2950 0x7f78e006eed0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 --2- 172.21.15.129:0/2646716383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b3b90 0x7f78e00cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 -- 172.21.15.129:0/2646716383 >> 172.21.15.129:0/2646716383 conn(0x7f78e006d410 msgr2=0x7f78e00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 -- 172.21.15.129:0/2646716383 shutdown_connections 2024-09-17T04:01:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.346+0000 7f78e8800640 1 -- 172.21.15.129:0/2646716383 wait complete. 2024-09-17T04:01:49.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.348+0000 7f78e8800640 1 Processor -- start 2024-09-17T04:01:49.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.348+0000 7f78e8800640 1 -- start start 2024-09-17T04:01:49.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b2950 0x7f78e01f29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 0x7f78e0155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78e006f590 con 0x7f78e01572a0 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f78e006f410 con 0x7f78e00ce170 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e8800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f78e006f710 con 0x7f78e00b2950 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44578/0 (socket says 172.21.15.129:44578) 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e6800640 1 -- 172.21.15.129:0/528072270 learned_addr learned my addr 172.21.15.129:0/528072270 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e5e00640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b2950 0x7f78e01f29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.350+0000 7f78e5400640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 0x7f78e0155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 -- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b2950 msgr2=0x7f78e01f29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b2950 0x7f78e01f29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 -- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 msgr2=0x7f78e0155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 0x7f78e0155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 -- 172.21.15.129:0/528072270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e01551d0 con 0x7f78e01572a0 2024-09-17T04:01:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e5400640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 0x7f78e0155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e6800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009cd0 tx=0x7f78dc0316d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78da000640 1 -- 172.21.15.129:0/528072270 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc031990 con 0x7f78e01572a0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78da000640 1 -- 172.21.15.129:0/528072270 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78dc031c20 con 0x7f78e01572a0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e01554b0 con 0x7f78e01572a0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78da000640 1 -- 172.21.15.129:0/528072270 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0330d0 con 0x7f78e01572a0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.351+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e01576e0 con 0x7f78e01572a0 2024-09-17T04:01:49.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.353+0000 7f78da000640 1 -- 172.21.15.129:0/528072270 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f78dc033270 con 0x7f78e01572a0 2024-09-17T04:01:49.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.354+0000 7f78da000640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 0x7f78c407b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.355+0000 7f78da000640 1 -- 172.21.15.129:0/528072270 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f78dc0bc1a0 con 0x7f78e01572a0 2024-09-17T04:01:49.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.355+0000 7f78e5e00640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 0x7f78c407b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.355+0000 7f78ef86d740 1 Processor -- start 2024-09-17T04:01:49.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.355+0000 7f78ef86d740 1 -- start start 2024-09-17T04:01:49.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.356+0000 7f78e5e00640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 0x7f78c407b0c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f78d4007fd0 tx=0x7f78d4001370 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x56359142f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359142feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x563591389430 con 0x56359103e590 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5635913795d0 con 0x5635910363b0 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x563591368a00 con 0x56359105f270 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44584/0 (socket says 172.21.15.129:44584) 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 -- 172.21.15.129:0/2467144252 learned_addr learned my addr 172.21.15.129:0/2467144252 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 -- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 msgr2=0x56359142f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x56359142f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 -- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 msgr2=0x56359142feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359142feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 -- 172.21.15.129:0/2467144252 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563591331520 con 0x56359103e590 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.357+0000 7f78e6800640 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005cc0 tx=0x7f78dc083dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60e560bf62fcb67 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.358+0000 7f78ce600640 1 -- 172.21.15.129:0/2467144252 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0839e0 con 0x56359103e590 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.358+0000 7f78ce600640 1 -- 172.21.15.129:0/2467144252 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78dc0845d0 con 0x56359103e590 2024-09-17T04:01:49.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.358+0000 7f78ce600640 1 -- 172.21.15.129:0/2467144252 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc083ed0 con 0x56359103e590 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.358+0000 7f78ef86d740 1 -- 172.21.15.129:0/2467144252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 msgr2=0x5635913fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.358+0000 7f78ef86d740 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005cc0 tx=0x7f78dc083dd0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 -- 172.21.15.129:0/2467144252 shutdown_connections 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x5635913fff60 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359142feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 --2- 172.21.15.129:0/2467144252 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x56359142f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 -- 172.21.15.129:0/2467144252 >> 172.21.15.129:0/2467144252 conn(0x5635910533c0 msgr2=0x563591405970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 -- 172.21.15.129:0/2467144252 shutdown_connections 2024-09-17T04:01:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.359+0000 7f78ef86d740 1 -- 172.21.15.129:0/2467144252 wait complete. 2024-09-17T04:01:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.360+0000 7f78ef86d740 1 -- start start 2024-09-17T04:01:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.360+0000 7f78ef86d740 1 Processor -- start 2024-09-17T04:01:49.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359141a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x563591443c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56359130f330 con 0x56359103e590 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56359136c810 con 0x5635910363b0 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5635910d9620 con 0x56359105f270 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.361+0000 7f78ef86d740 10 client.0 _ll_register_callbacks cb 0x563591314420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359141a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44596/0 (socket says 172.21.15.129:44596) 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 -- 172.21.15.129:0/4129339634 learned_addr learned my addr 172.21.15.129:0/4129339634 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e6800640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x563591443c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 -- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 msgr2=0x563591443c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x563591443c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 -- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 msgr2=0x56359141a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359141a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 -- 172.21.15.129:0/4129339634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635910ca850 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78e5400640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f78d000ddd0 tx=0x7f78d000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ee3ed17904a749 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.362+0000 7f78c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d0011650 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f78d00117f0 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78dc0845d0 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=163) ==== 24+0+0 (secure 0 0 0) 0x7f78d0022050 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 10 client.10017 fetch_fsmap learned FSMap version 163 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=163,monmap=2+}) -- 0x5635911416b0 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78dc0845d0 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 163) ==== 3502+0+0 (secure 0 0 0) 0x7f78d0025020 con 0x56359103e590 2024-09-17T04:01:49.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 10 client.10017 fetch_fsmap finished waiting for FSMap version 163 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 10 client.10017 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 20 client.10017 populate_metadata read hostname 'smithi129' 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x56359101de40 0x563591445880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 4 client.10017 mds_command: new command op to 9903 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.363+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 --> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56359137b520 con 0x56359101de40 2024-09-17T04:01:49.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.364+0000 7f78e5e00640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x56359101de40 0x563591445880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.364+0000 7f78e5e00640 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x56359101de40 0x563591445880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=393ba2bb8f83bab4 server_cookie=9060d626c9963bea in_seq=0 out_seq=0 2024-09-17T04:01:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.364+0000 7f78c2a00640 10 client.10017 ms_handle_connect on v2:172.21.15.129:6832/3531032601 2024-09-17T04:01:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.365+0000 7f78c2a00640 1 -- 172.21.15.129:0/4129339634 <== mds.0 v2:172.21.15.129:6832/3531032601 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56359137b520 con 0x56359101de40 2024-09-17T04:01:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.365+0000 7f78c2a00640 10 client.10017 handle_command_reply: tid=0 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.365+0000 7f78ef86d740 1 client.10017 shutdown 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.365+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 msgr2=0x563591460b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.365+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f78d000ddd0 tx=0x7f78d000f730 comp rx=0 tx=0).stop 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 shutdown_connections 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x56359101de40 0x563591445880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359105f270 0x563591443c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359103e590 0x563591460b80 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78c2a00640 0 client.10017 ms_handle_reset on v2:172.21.15.129:6832/3531032601 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 --2- 172.21.15.129:0/4129339634 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x56359141a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 >> 172.21.15.129:0/4129339634 conn(0x5635910533c0 msgr2=0x56359144dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 shutdown_connections 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 1 -- 172.21.15.129:0/4129339634 wait complete. 2024-09-17T04:01:49.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.366+0000 7f78ef86d740 20 client.10017 trim_cache size 0 max 16384 2024-09-17T04:01:49.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.385+0000 7f78ef86d740 1 Processor -- start 2024-09-17T04:01:49.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.385+0000 7f78ef86d740 1 -- start start 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5635910363b0 0x563591426fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56359103e590 0x563591427520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f78e00b1430 con 0x56359105f270 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f78e01f70f0 con 0x56359103e590 2024-09-17T04:01:49.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f78e01f7740 con 0x5635910363b0 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.386+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44600/0 (socket says 172.21.15.129:44600) 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 -- 172.21.15.129:0/1146813656 learned_addr learned my addr 172.21.15.129:0/1146813656 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e5400640 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56359103e590 0x563591427520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e5e00640 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5635910363b0 0x563591426fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 -- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5635910363b0 msgr2=0x563591426fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5635910363b0 0x563591426fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 -- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56359103e590 msgr2=0x563591427520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56359103e590 0x563591427520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 -- 172.21.15.129:0/1146813656 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563591403950 con 0x56359105f270 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78e6800640 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005cc0 tx=0x7f78dc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7713d0a99112dc server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78cdc00640 1 -- 172.21.15.129:0/1146813656 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc09e190 con 0x56359105f270 2024-09-17T04:01:49.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78cdc00640 1 -- 172.21.15.129:0/1146813656 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78dc0845d0 con 0x56359105f270 2024-09-17T04:01:49.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.387+0000 7f78cdc00640 1 -- 172.21.15.129:0/1146813656 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0889b0 con 0x56359105f270 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 -- 172.21.15.129:0/1146813656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 msgr2=0x563591472b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f78dc005cc0 tx=0x7f78dc009500 comp rx=0 tx=0).stop 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 -- 172.21.15.129:0/1146813656 shutdown_connections 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x563591472b80 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56359103e590 0x563591427520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 --2- 172.21.15.129:0/1146813656 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5635910363b0 0x563591426fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 -- 172.21.15.129:0/1146813656 >> 172.21.15.129:0/1146813656 conn(0x5635910533c0 msgr2=0x56359144dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 -- 172.21.15.129:0/1146813656 shutdown_connections 2024-09-17T04:01:49.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.388+0000 7f78ef86d740 1 -- 172.21.15.129:0/1146813656 wait complete. 2024-09-17T04:01:49.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.389+0000 7f78ef86d740 1 -- start start 2024-09-17T04:01:49.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.389+0000 7f78ef86d740 1 Processor -- start 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x5635914722e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359103e590 0x5635914d2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56359141b660 con 0x56359105f270 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x563591444d40 con 0x5635910363b0 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5635914619d0 con 0x56359103e590 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78ef86d740 10 client.0 _ll_register_callbacks cb 0x7f78d000d930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44612/0 (socket says 172.21.15.129:44612) 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 -- 172.21.15.129:0/3662456417 learned_addr learned my addr 172.21.15.129:0/3662456417 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e5e00640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x5635914722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e5400640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359103e590 0x5635914d2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 -- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359103e590 msgr2=0x5635914d2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359103e590 0x5635914d2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 -- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 msgr2=0x5635914722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x5635914722e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.391+0000 7f78e6800640 1 -- 172.21.15.129:0/3662456417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563591426bd0 con 0x56359105f270 2024-09-17T04:01:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78e6800640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0ba630 tx=0x7f78dc0803f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831a2aca49780e69 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc0a5060 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f78dc08d3d0 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f78dc0845d0 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=163) ==== 24+0+0 (secure 0 0 0) 0x7f78dc0839e0 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 10 client.10023 fetch_fsmap learned FSMap version 163 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=163,monmap=2+}) -- 0x56359145fd10 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f78dc0845d0 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 163) ==== 3502+0+0 (secure 0 0 0) 0x7f78dc04b030 con 0x56359105f270 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 10 client.10023 fetch_fsmap finished waiting for FSMap version 163 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 10 client.10023 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:01:49.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.392+0000 7f78ef86d740 20 client.10023 populate_metadata read hostname 'smithi129' 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x563591472b80 0x563591472f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78ef86d740 4 client.10023 mds_command: new command op to 9903 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 --> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] -- command(tid 0: {"prefix": "status"}) -- 0x563591376fd0 con 0x563591472b80 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78e5e00640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x563591472b80 0x563591472f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78e5e00640 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x563591472b80 0x563591472f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb3bac4d286e4426 server_cookie=a773660af495d922 in_seq=0 out_seq=0 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.393+0000 7f78c2a00640 10 client.10023 ms_handle_connect on v2:172.21.15.129:6832/3531032601 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78c2a00640 1 -- 172.21.15.129:0/3662456417 <== mds.0 v2:172.21.15.129:6832/3531032601 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x563591376fd0 con 0x563591472b80 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78c2a00640 10 client.10023 handle_command_reply: tid=0 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 client.10023 shutdown 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 msgr2=0x5635914ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f78dc0ba630 tx=0x7f78dc0803f0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 shutdown_connections 2024-09-17T04:01:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:6832/3531032601,v1:172.21.15.129:6835/3531032601] conn(0x563591472b80 0x563591472f90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56359105f270 0x5635914ce2a0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56359103e590 0x5635914d2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 --2- 172.21.15.129:0/3662456417 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5635910363b0 0x5635914722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78c2a00640 0 client.10023 ms_handle_reset on v2:172.21.15.129:6832/3531032601 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 >> 172.21.15.129:0/3662456417 conn(0x5635910533c0 msgr2=0x563591401250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 shutdown_connections 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 1 -- 172.21.15.129:0/3662456417 wait complete. 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.394+0000 7f78ef86d740 20 client.10023 trim_cache size 0 max 16384 2024-09-17T04:01:49.400 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 9903, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.643493264, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 163, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 186, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 186, 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.717864232 2024-09-17T04:01:49.401 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:01:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 msgr2=0x7f78c407b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 0x7f78c407b0c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f78d4007fd0 tx=0x7f78d4001370 comp rx=0 tx=0).stop 2024-09-17T04:01:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 msgr2=0x7f78e01564d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f78dc009cd0 tx=0x7f78dc0316d0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 shutdown_connections 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f78c4078bb0 0x7f78c407b0c0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f78e01572a0 0x7f78e01564d0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f78e00ce170 0x7f78e0155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 --2- 172.21.15.129:0/528072270 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f78e00b2950 0x7f78e01f29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 >> 172.21.15.129:0/528072270 conn(0x7f78e006d410 msgr2=0x7f78e0073930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 shutdown_connections 2024-09-17T04:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.395+0000 7f78e8800640 1 -- 172.21.15.129:0/528072270 wait complete. 2024-09-17T04:01:49.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 -- 172.21.15.129:0/3576886017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 msgr2=0x7f646c0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 --2- 172.21.15.129:0/3576886017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c0cbbf0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f6464005fd0 tx=0x7f646402a940 comp rx=0 tx=0).stop 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 -- 172.21.15.129:0/3576886017 shutdown_connections 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 --2- 172.21.15.129:0/3576886017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c0cbbf0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 -- 172.21.15.129:0/3576886017 >> 172.21.15.129:0/3576886017 conn(0x7f646c06d410 msgr2=0x7f646c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.639+0000 7f6472e00640 1 -- 172.21.15.129:0/3576886017 shutdown_connections 2024-09-17T04:01:49.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.640+0000 7f6472e00640 1 -- 172.21.15.129:0/3576886017 wait complete. 2024-09-17T04:01:49.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.642+0000 7f6472e00640 1 Processor -- start 2024-09-17T04:01:49.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.642+0000 7f6472e00640 1 -- start start 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f646c0ce170 0x7f646c0ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 0x7f646c0ceb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f646c06f5e0 con 0x7f646c0b2950 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f646c070910 con 0x7f646c158a40 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f6472e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f646c06f760 con 0x7f646c0ce170 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.643+0000 7f646be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44632/0 (socket says 172.21.15.129:44632) 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 -- 172.21.15.129:0/1187922353 learned_addr learned my addr 172.21.15.129:0/1187922353 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646b400640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f646c0ce170 0x7f646c0ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f6470e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 0x7f646c0ceb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 -- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f646c0ce170 msgr2=0x7f646c0ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f646c0ce170 0x7f646c0ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 -- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 msgr2=0x7f646c0ceb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 0x7f646c0ceb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 -- 172.21.15.129:0/1187922353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646c1f5a20 con 0x7f646c0b2950 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f6470e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 0x7f646c0ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f646be00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c3a0 tx=0x7f645c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.644+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c017070 con 0x7f646c0b2950 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.645+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f645c0035f0 con 0x7f646c0b2950 2024-09-17T04:01:49.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.645+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c003a40 con 0x7f646c0b2950 2024-09-17T04:01:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.645+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646c1f94e0 con 0x7f646c0b2950 2024-09-17T04:01:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.645+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646c1f9aa0 con 0x7f646c0b2950 2024-09-17T04:01:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.646+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f646c0cbbf0 con 0x7f646c0b2950 2024-09-17T04:01:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.647+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f645c01a070 con 0x7f646c0b2950 2024-09-17T04:01:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.648+0000 7f6463e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 0x7f644c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:49.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.649+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f645c09a050 con 0x7f646c0b2950 2024-09-17T04:01:49.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.649+0000 7f646b400640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 0x7f644c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:49.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.650+0000 7f646b400640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 0x7f644c07b0a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f60 tx=0x7f6458009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:49.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.653+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f645c0623f0 con 0x7f646c0b2950 2024-09-17T04:01:49.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.875+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f646c06a140 con 0x7f646c0b2950 2024-09-17T04:01:49.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.876+0000 7f6463e00640 1 -- 172.21.15.129:0/1187922353 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v163) ==== 53+0+73 (secure 0 0 0) 0x7f645c061cf0 con 0x7f646c0b2950 2024-09-17T04:01:49.876 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:01:49.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 msgr2=0x7f644c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 0x7f644c07b0a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f60 tx=0x7f6458009040 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 msgr2=0x7f646c099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f645c00c3a0 tx=0x7f645c00c860 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 shutdown_connections 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f644c078b90 0x7f644c07b0a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f646c158a40 0x7f646c0ceb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f646c0ce170 0x7f646c0ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 --2- 172.21.15.129:0/1187922353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f646c0b2950 0x7f646c099a20 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 >> 172.21.15.129:0/1187922353 conn(0x7f646c06d410 msgr2=0x7f646c0af4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:49.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.878+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 shutdown_connections 2024-09-17T04:01:49.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:49.879+0000 7f6472e00640 1 -- 172.21.15.129:0/1187922353 wait complete. 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:01:49.902 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:49.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:49.903 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:01:49.939 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41982sec preferred_lft 41982sec 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:01:49.940 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:01:49.941 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:01:49.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:49.942 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:01:50.032 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:01:50.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:50.036 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:01:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:01:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:01:50.071 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:01:50.072 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:01:50.072 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:01:50.072 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:01:50.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:01:50.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:50.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:50.269 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:01:50.304 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:01:50.305 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:01:50.305 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:50.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:50.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:01:50.402 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:01:50.402 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:50.437 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:01:50.438 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:01:50.438 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:50.512 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:01:50.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:50.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:50.707 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:50.707 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:50.707 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:50.727 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:50.727 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:50.728 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:50.728 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:01:50.728 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:01:50.760 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:01:50.761 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:01:50.762 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:01:50.762 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:01:50.763 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:01:50.763 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:01:50.763 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:01:50.766 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:01:50.766 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:01:50.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:01:50.824 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:01:50.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:01:50.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:01:50.920 INFO:teuthology.orchestra.run.smithi129.stdout:client9343 2024-09-17T04:01:50.920 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:01:50.921 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:01:50.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.001 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.094 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:01:51.094 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:01:51.095 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.096 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.097 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.098 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.099 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:01:51.100 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.101 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.102 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.103 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.104 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:01:51.105 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:01:51.106 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:01:51.108 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.109 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:01:51.110 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:01:51.111 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:01:51.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.113 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:01:51.114 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:01:51.115 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.116 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.117 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:01:51.118 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.232 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.233 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:01:51.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.235 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.237 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:01:51.238 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:01:51.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.240 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.241 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.241 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.242 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.243 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.244 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.245 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.246 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.247 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.247 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:01:51.248 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:01:51.249 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:01:51.250 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.250 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 16 2024-09-17T04:01:51.251 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:01:51.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:01:51.253 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.254 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:01:51.255 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.256 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:01:51.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.258 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.259 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.260 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.261 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.262 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:01:51.263 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.264 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 16 2024-09-17T04:01:51.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.491+0000 7fd8c0e00640 1 -- 172.21.15.129:0/879694774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc0b4c30 msgr2=0x7fd8bc15cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.491+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/879694774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc0b4c30 0x7fd8bc15cd30 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4005fd0 tx=0x7fd8b402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.491+0000 7fd8c0e00640 1 -- 172.21.15.129:0/879694774 shutdown_connections 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.491+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/879694774 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc0b4c30 0x7fd8bc15cd30 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.491+0000 7fd8c0e00640 1 -- 172.21.15.129:0/879694774 >> 172.21.15.129:0/879694774 conn(0x7fd8bc06d330 msgr2=0x7fd8bc0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.492+0000 7fd8c0e00640 1 -- 172.21.15.129:0/879694774 shutdown_connections 2024-09-17T04:01:51.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.492+0000 7fd8c0e00640 1 -- 172.21.15.129:0/879694774 wait complete. 2024-09-17T04:01:51.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.494+0000 7fd8c0e00640 1 Processor -- start 2024-09-17T04:01:51.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.494+0000 7fd8c0e00640 1 -- start start 2024-09-17T04:01:51.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:51.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8bc15c810 0x7fd8bc15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:51.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8bc149b70 0x7fd8bc15ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc15e880 con 0x7fd8bc1527b0 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc15e700 con 0x7fd8bc149b70 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8c0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd8bc15ea00 con 0x7fd8bc15c810 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8bb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8bb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44660/0 (socket says 172.21.15.129:44660) 2024-09-17T04:01:51.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.495+0000 7fd8bb400640 1 -- 172.21.15.129:0/2942255497 learned_addr learned my addr 172.21.15.129:0/2942255497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 -- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8bc15c810 msgr2=0x7fd8bc15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8bc15c810 0x7fd8bc15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 -- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8bc149b70 msgr2=0x7fd8bc15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8bc149b70 0x7fd8bc15ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 -- 172.21.15.129:0/2942255497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc14dd10 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8bb400640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00c3f0 tx=0x7fd8ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac017070 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8ac0035f0 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac003a40 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.496+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8bc14dff0 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.497+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8bc14e5b0 con 0x7fd8bc1527b0 2024-09-17T04:01:51.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.498+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8bc15eeb0 con 0x7fd8bc1527b0 2024-09-17T04:01:51.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.499+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd8ac01a070 con 0x7fd8bc1527b0 2024-09-17T04:01:51.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.500+0000 7fd8b3400640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 0x7fd89c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:51.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.500+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fd8ac099fd0 con 0x7fd8bc1527b0 2024-09-17T04:01:51.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.501+0000 7fd8baa00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 0x7fd89c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:51.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.501+0000 7fd8baa00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 0x7fd89c07b0d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc14d930 tx=0x7fd8a8007a30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:51.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.505+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd8ac062490 con 0x7fd8bc1527b0 2024-09-17T04:01:51.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.685+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd8bc0aff20 con 0x7fd8bc1527b0 2024-09-17T04:01:51.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.686+0000 7fd8b3400640 1 -- 172.21.15.129:0/2942255497 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v186) ==== 100+0+11 (secure 0 0 0) 0x7fd8ac061d90 con 0x7fd8bc1527b0 2024-09-17T04:01:51.687 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 64 2024-09-17T04:01:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 msgr2=0x7fd89c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:51.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 0x7fd89c07b0d0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc14d930 tx=0x7fd8a8007a30 comp rx=0 tx=0).stop 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 msgr2=0x7fd8bc1f48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00c3f0 tx=0x7fd8ac00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 shutdown_connections 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd89c078bc0 0x7fd89c07b0d0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8bc149b70 0x7fd8bc15ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8bc15c810 0x7fd8bc15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 --2- 172.21.15.129:0/2942255497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8bc1527b0 0x7fd8bc1f48c0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 >> 172.21.15.129:0/2942255497 conn(0x7fd8bc06d330 msgr2=0x7fd8bc0d0600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 shutdown_connections 2024-09-17T04:01:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:51.689+0000 7fd8c0e00640 1 -- 172.21.15.129:0/2942255497 wait complete. 2024-09-17T04:01:51.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:01:51.814 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:01:51.814 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:01:51.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.816 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:01:51.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.818 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.819 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.820 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:01:51.821 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:01:51.822 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.823 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:01:51.824 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 16 2024-09-17T04:01:51.825 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops_per_pg = 0.25 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:01:51.826 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:01:51.827 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:01:51.828 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:01:51.829 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:01:51.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:01:51.831 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:01:51.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:01:51.833 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:01:51.834 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:01:51.835 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 100000000 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 16 2024-09-17T04:01:51.836 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops_per_pg = 0.25 2024-09-17T04:01:51.843 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:01:51.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:51.844 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:01:51.844 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:51.844 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:01:51.844 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:51.845 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:01:51.845 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:01:51.867 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:01:51.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T04:01:51.868 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:01:51.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T04:01:51.870 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:01:51.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T04:01:52.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 -- 172.21.15.129:0/2715568597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 msgr2=0x7f6da4073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 --2- 172.21.15.129:0/2715568597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4073bc0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f6da40b6060 tx=0x7f6d9402a560 comp rx=0 tx=0).stop 2024-09-17T04:01:52.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 -- 172.21.15.129:0/2715568597 shutdown_connections 2024-09-17T04:01:52.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 --2- 172.21.15.129:0/2715568597 >> v2:172.21.15.181:3300/0 conn(0x7f6da415f880 0x7f6da415fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 --2- 172.21.15.129:0/2715568597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4073bc0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 -- 172.21.15.129:0/2715568597 >> 172.21.15.129:0/2715568597 conn(0x7f6da40b23c0 msgr2=0x7f6da40b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:52.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 -- 172.21.15.129:0/2715568597 shutdown_connections 2024-09-17T04:01:52.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.101+0000 7f6daa600640 1 -- 172.21.15.129:0/2715568597 wait complete. 2024-09-17T04:01:52.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 -- 172.21.15.129:0/3679603239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0cd190 msgr2=0x7fb10c15cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 --2- 172.21.15.129:0/3679603239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0cd190 0x7fb10c15cf70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb10c072150 tx=0x7fb0fc02a4d0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 -- 172.21.15.129:0/3679603239 shutdown_connections 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 --2- 172.21.15.129:0/3679603239 >> v2:172.21.15.129:3300/0 conn(0x7fb10c0cfd80 0x7fb10c072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 --2- 172.21.15.129:0/3679603239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0cd190 0x7fb10c15cf70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 -- 172.21.15.129:0/3679603239 >> 172.21.15.129:0/3679603239 conn(0x7fb10c0b23c0 msgr2=0x7fb10c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 -- 172.21.15.129:0/3679603239 shutdown_connections 2024-09-17T04:01:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.102+0000 7fb110a00640 1 -- 172.21.15.129:0/3679603239 wait complete. 2024-09-17T04:01:52.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.104+0000 7fb110a00640 1 Processor -- start 2024-09-17T04:01:52.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.104+0000 7fb110a00640 1 -- start start 2024-09-17T04:01:52.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.105+0000 7f6daa600640 1 Processor -- start 2024-09-17T04:01:52.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.105+0000 7f6daa600640 1 -- start start 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0b59d0 0x7fb10c0af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb10c0cfd80 0x7fb10c160950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c06fe50 con 0x7fb10c0cd190 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c06fcd0 con 0x7fb10c0b59d0 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb110a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c06ffd0 con 0x7fb10c0cfd80 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb109400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb109400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44676/0 (socket says 172.21.15.129:44676) 2024-09-17T04:01:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb109400640 1 -- 172.21.15.129:0/3624237049 learned_addr learned my addr 172.21.15.129:0/3624237049 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb109e00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0b59d0 0x7fb10c0af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.106+0000 7fb10a800640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb10c0cfd80 0x7fb10c160950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 -- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb10c0cfd80 msgr2=0x7fb10c160950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb10c0cfd80 0x7fb10c160950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 -- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0b59d0 msgr2=0x7fb10c0af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0b59d0 0x7fb10c0af5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 -- 172.21.15.129:0/3624237049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0afb10 con 0x7fb10c0cd190 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb109400640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009fd0 tx=0x7fb0f800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f800ccc0 con 0x7fb10c0cd190 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0f800ee70 con 0x7fb10c0cd190 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c132360 con 0x7fb10c0cd190 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f800ce60 con 0x7fb10c0cd190 2024-09-17T04:01:52.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c132920 con 0x7fb10c0cd190 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da415f880 0x7f6da4145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6da40cfe00 con 0x7f6da40737b0 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6da415bf70 con 0x7f6da415f880 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.107+0000 7f6daa600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40cff80 con 0x7f6da406f710 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:35456/0 (socket says 172.21.15.129:35456) 2024-09-17T04:01:52.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 -- 172.21.15.129:0/884942323 learned_addr learned my addr 172.21.15.129:0/884942323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da2a00640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3e00640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da415f880 0x7f6da4145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 -- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da415f880 msgr2=0x7f6da4145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da415f880 0x7f6da4145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 -- 172.21.15.129:0/884942323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 msgr2=0x7f6da4157bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4157bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.108+0000 7f6da3400640 1 -- 172.21.15.129:0/884942323 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da4154070 con 0x7f6da406f710 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6da3400640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d94034040 tx=0x7f6d94003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d9403e030 con 0x7f6da406f710 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6d94003d10 con 0x7f6da406f710 2024-09-17T04:01:52.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94035040 con 0x7f6da406f710 2024-09-17T04:01:52.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c070480 con 0x7fb10c0cd190 2024-09-17T04:01:52.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da4154310 con 0x7f6da406f710 2024-09-17T04:01:52.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.109+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da41548d0 con 0x7f6da406f710 2024-09-17T04:01:52.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.111+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da40703f0 con 0x7f6da406f710 2024-09-17T04:01:52.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.111+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0f8005260 con 0x7fb10c0cd190 2024-09-17T04:01:52.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.112+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6d94041030 con 0x7f6da406f710 2024-09-17T04:01:52.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.112+0000 7fb102000640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 0x7fb0ec07f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.113+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fb0f809b500 con 0x7fb10c0cd190 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.113+0000 7fb109e00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 0x7fb0ec07f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.113+0000 7fb109e00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 0x7fb0ec07f4e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc033000 tx=0x7fb0fc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.114+0000 7f6d9b400640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 0x7f6d8407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.114+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7f6d940bb410 con 0x7f6da406f710 2024-09-17T04:01:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.114+0000 7f6da2a00640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 0x7f6d8407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.115+0000 7f6da2a00640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 0x7f6d8407b0d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6da41592c0 tx=0x7f6d90005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.117+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0f80632f0 con 0x7fb10c0cd190 2024-09-17T04:01:52.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.118+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6d94083160 con 0x7f6da406f710 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.152+0000 7fc622a00640 1 -- 172.21.15.129:0/582801857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15f230 msgr2=0x7fc61c0b53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.152+0000 7fc622a00640 1 --2- 172.21.15.129:0/582801857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15f230 0x7fc61c0b53d0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7fc614005fd0 tx=0x7fc61402bc80 comp rx=0 tx=0).stop 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.153+0000 7fc622a00640 1 -- 172.21.15.129:0/582801857 shutdown_connections 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.153+0000 7fc622a00640 1 --2- 172.21.15.129:0/582801857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15f230 0x7fc61c0b53d0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.153+0000 7fc622a00640 1 -- 172.21.15.129:0/582801857 >> 172.21.15.129:0/582801857 conn(0x7fc61c0b23c0 msgr2=0x7fc61c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.153+0000 7fc622a00640 1 -- 172.21.15.129:0/582801857 shutdown_connections 2024-09-17T04:01:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.153+0000 7fc622a00640 1 -- 172.21.15.129:0/582801857 wait complete. 2024-09-17T04:01:52.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.155+0000 7fc622a00640 1 Processor -- start 2024-09-17T04:01:52.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.155+0000 7fc622a00640 1 -- start start 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 0x7fc61c145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc61c1544b0 0x7fc61c1548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c06fb50 con 0x7fc61c15c200 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c06f9d0 con 0x7fc61c15f230 2024-09-17T04:01:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.156+0000 7fc622a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c06fcd0 con 0x7fc61c1544b0 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 0x7fc61c145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:44702/0 (socket says 172.21.15.129:44702) 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 -- 172.21.15.129:0/287549748 learned_addr learned my addr 172.21.15.129:0/287549748 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc620a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc61c1544b0 0x7fc61c1548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 -- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc61c1544b0 msgr2=0x7fc61c1548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc61c1544b0 0x7fc61c1548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 -- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 msgr2=0x7fc61c145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 0x7fc61c145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61be00640 1 -- 172.21.15.129:0/287549748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c1f9810 con 0x7fc61c15c200 2024-09-17T04:01:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.157+0000 7fc61b400640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 0x7fc61c145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:01:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc61be00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c430 tx=0x7fc60c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c017070 con 0x7fc61c15c200 2024-09-17T04:01:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc60c0053e0 con 0x7fc61c15c200 2024-09-17T04:01:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61c154e00 con 0x7fc61c15c200 2024-09-17T04:01:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60c003850 con 0x7fc61c15c200 2024-09-17T04:01:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.158+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc61c1f8d30 con 0x7fc61c15c200 2024-09-17T04:01:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.159+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61c0b53d0 con 0x7fc61c15c200 2024-09-17T04:01:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.161+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc60c01a070 con 0x7fc61c15c200 2024-09-17T04:01:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.162+0000 7fc613e00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 0x7fc5fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.162+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(186..186 src has 1..186) ==== 5709+0+0 (secure 0 0 0) 0x7fc60c099fd0 con 0x7fc61c15c200 2024-09-17T04:01:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.162+0000 7fc61b400640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 0x7fc5fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.163+0000 7fc61b400640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 0x7fc5fc07b0a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc608000c00 tx=0x7fc608007f70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:52.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.166+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc60c061e50 con 0x7fc61c15c200 2024-09-17T04:01:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.383+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7f6da41525a0 con 0x7f6da406f710 2024-09-17T04:01:52.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.386+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fb10c072f00 con 0x7fb10c0cd190 2024-09-17T04:01:52.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:52.429+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7fc61c0b4740 con 0x7fc61c15c200 2024-09-17T04:01:53.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.236+0000 7f6d9b400640 1 -- 172.21.15.129:0/884942323 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 9205 v164) ==== 95+0+0 (secure 0 0 0) 0x7f6d94087b90 con 0x7f6da406f710 2024-09-17T04:01:53.236 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 9205 2024-09-17T04:01:53.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 msgr2=0x7f6d8407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:53.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 0x7f6d8407b0d0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6da41592c0 tx=0x7f6d90005f90 comp rx=0 tx=0).stop 2024-09-17T04:01:53.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 msgr2=0x7f6da41538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:53.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6d94034040 tx=0x7f6d94003820 comp rx=0 tx=0).stop 2024-09-17T04:01:53.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 shutdown_connections 2024-09-17T04:01:53.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6d84078bc0 0x7f6d8407b0d0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:53.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6da415f880 0x7f6da4145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:53.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6da40737b0 0x7f6da4157bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:53.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 --2- 172.21.15.129:0/884942323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6da406f710 0x7f6da41538e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:53.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.239+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 >> 172.21.15.129:0/884942323 conn(0x7f6da40b23c0 msgr2=0x7f6da40b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:53.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.240+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 shutdown_connections 2024-09-17T04:01:53.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:53.240+0000 7f6daa600640 1 -- 172.21.15.129:0/884942323 wait complete. 2024-09-17T04:01:53.272 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:01:53.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:01:53.276 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:01:53.277 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:53.469 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:01:54.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.234+0000 7fb102000640 1 -- 172.21.15.129:0/3624237049 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7946 v165) ==== 95+0+0 (secure 0 0 0) 0x7fb0f8067d20 con 0x7fb10c0cd190 2024-09-17T04:01:54.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.235+0000 7fc613e00640 1 -- 172.21.15.129:0/287549748 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 9208 v165) ==== 95+0+0 (secure 0 0 0) 0x7fc60c066880 con 0x7fc61c15c200 2024-09-17T04:01:54.235 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 7946 2024-09-17T04:01:54.236 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 9208 2024-09-17T04:01:54.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 msgr2=0x7fc5fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:54.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 0x7fc5fc07b0a0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fc608000c00 tx=0x7fc608007f70 comp rx=0 tx=0).stop 2024-09-17T04:01:54.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 msgr2=0x7fc61c0ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:54.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7fc60c00c430 tx=0x7fc60c00c8f0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 msgr2=0x7fb0ec07f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 0x7fb0ec07f4e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc033000 tx=0x7fb0fc004730 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 msgr2=0x7fb10c0ad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009fd0 tx=0x7fb0f800ec20 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 shutdown_connections 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5fc078b90 0x7fc5fc07b0a0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc61c1544b0 0x7fc61c1548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc61c15f230 0x7fc61c145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 --2- 172.21.15.129:0/287549748 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc61c15c200 0x7fc61c0ce1b0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 >> 172.21.15.129:0/287549748 conn(0x7fc61c0b23c0 msgr2=0x7fc61c072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:54.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 shutdown_connections 2024-09-17T04:01:54.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0ec07cfd0 0x7fb0ec07f4e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb10c0cfd80 0x7fb10c160950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10c0cd190 0x7fb10c0ad630 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 shutdown_connections 2024-09-17T04:01:54.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 --2- 172.21.15.129:0/3624237049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb10c0b59d0 0x7fb10c0af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:54.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.239+0000 7fc622a00640 1 -- 172.21.15.129:0/287549748 wait complete. 2024-09-17T04:01:54.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.238+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 >> 172.21.15.129:0/3624237049 conn(0x7fb10c0b23c0 msgr2=0x7fb10c0cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:54.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.239+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 shutdown_connections 2024-09-17T04:01:54.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:54.239+0000 7fb110a00640 1 -- 172.21.15.129:0/3624237049 wait complete. 2024-09-17T04:01:54.271 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:01:54.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:01:54.275 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:01:54.275 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:54.276 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:01:54.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:01:54.281 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:01:54.281 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:54.465 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:01:54.473 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:01:56.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-09-17T04:01:57.870 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:01:57.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-17T04:01:58.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.101+0000 7feb2a000640 1 -- 172.21.15.129:0/1813168771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb2415cf70 msgr2=0x7feb24072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.101+0000 7feb2a000640 1 --2- 172.21.15.129:0/1813168771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb2415cf70 0x7feb24072d70 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7feb240cf600 tx=0x7feb1002a4a0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.101+0000 7feb2a000640 1 -- 172.21.15.129:0/1813168771 shutdown_connections 2024-09-17T04:01:58.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.101+0000 7feb2a000640 1 --2- 172.21.15.129:0/1813168771 >> v2:172.21.15.181:3300/0 conn(0x7feb240b59d0 0x7feb240b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.101+0000 7feb2a000640 1 --2- 172.21.15.129:0/1813168771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb2415cf70 0x7feb24072d70 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.102+0000 7feb2a000640 1 -- 172.21.15.129:0/1813168771 >> 172.21.15.129:0/1813168771 conn(0x7feb240b23c0 msgr2=0x7feb240b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:58.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.102+0000 7feb2a000640 1 -- 172.21.15.129:0/1813168771 shutdown_connections 2024-09-17T04:01:58.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.102+0000 7feb2a000640 1 -- 172.21.15.129:0/1813168771 wait complete. 2024-09-17T04:01:58.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.104+0000 7feb2a000640 1 Processor -- start 2024-09-17T04:01:58.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.104+0000 7feb2a000640 1 -- start start 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 0x7feb240c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb2415cf70 0x7feb240c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feb2406fe50 con 0x7feb240b59d0 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feb2406fcd0 con 0x7feb2415cf70 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.105+0000 7feb2a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7feb2406ffd0 con 0x7feb240cd190 2024-09-17T04:01:58.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51138/0 (socket says 172.21.15.129:51138) 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 -- 172.21.15.129:0/1251034279 learned_addr learned my addr 172.21.15.129:0/1251034279 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb22a00640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 0x7feb240c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23e00640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb2415cf70 0x7feb240c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 -- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 msgr2=0x7feb240c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 0x7feb240c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 -- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb2415cf70 msgr2=0x7feb240c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb2415cf70 0x7feb240c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 -- 172.21.15.129:0/1251034279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb240ac770 con 0x7feb240b59d0 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb22a00640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 0x7feb240c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.106+0000 7feb23400640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c3c0 tx=0x7feb1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:58.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.107+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14017070 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.107+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feb140035f0 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.107+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14003a80 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.107+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb240c6580 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.107+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb240afa60 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.108+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb240704c0 con 0x7feb240b59d0 2024-09-17T04:01:58.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.109+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feb1401a070 con 0x7feb240b59d0 2024-09-17T04:01:58.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.110+0000 7feb1b400640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 0x7feb0407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.110+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 5709+0+0 (secure 0 0 0) 0x7feb1409a0d0 con 0x7feb240b59d0 2024-09-17T04:01:58.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.110+0000 7feb22a00640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 0x7feb0407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.111+0000 7feb22a00640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 0x7feb0407b0a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7feb10034000 tx=0x7feb10004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:58.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.115+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feb240ac770 con 0x7feb240b59d0 2024-09-17T04:01:58.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.366+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7feb240b27d0 con 0x7feb240b59d0 2024-09-17T04:01:58.367 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:01:58.367+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:01:58.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.369+0000 7feb1b400640 1 -- 172.21.15.129:0/1251034279 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 9903 v167) ==== 95+0+0 (secure 0 0 0) 0x7feb14062500 con 0x7feb240b59d0 2024-09-17T04:01:58.369 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 9903 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.373+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 msgr2=0x7feb0407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.373+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 0x7feb0407b0a0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7feb10034000 tx=0x7feb10004730 comp rx=0 tx=0).stop 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.373+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 msgr2=0x7feb24143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.373+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7feb1400c3c0 tx=0x7feb1400c880 comp rx=0 tx=0).stop 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 shutdown_connections 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feb04078b90 0x7feb0407b0a0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feb2415cf70 0x7feb240c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feb240cd190 0x7feb240c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 --2- 172.21.15.129:0/1251034279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feb240b59d0 0x7feb24143600 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 >> 172.21.15.129:0/1251034279 conn(0x7feb240b23c0 msgr2=0x7feb24144f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:58.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 shutdown_connections 2024-09-17T04:01:58.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.374+0000 7feb2a000640 1 -- 172.21.15.129:0/1251034279 wait complete. 2024-09-17T04:01:58.403 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:01:58.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:01:58.407 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:01:58.408 DEBUG:teuthology.parallel:result is None 2024-09-17T04:01:58.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:01:58.601 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 -- 172.21.15.129:0/2568167976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f017815da40 msgr2=0x7f017806e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 --2- 172.21.15.129:0/2568167976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f017815da40 0x7f017806e310 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f0178161390 tx=0x7f016c02a5f0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 -- 172.21.15.129:0/2568167976 shutdown_connections 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 --2- 172.21.15.129:0/2568167976 >> v2:172.21.15.181:3300/0 conn(0x7f0178156f90 0x7f01780722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 --2- 172.21.15.129:0/2568167976 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f017815da40 0x7f017806e310 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 -- 172.21.15.129:0/2568167976 >> 172.21.15.129:0/2568167976 conn(0x7f017806d560 msgr2=0x7f01780b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 -- 172.21.15.129:0/2568167976 shutdown_connections 2024-09-17T04:01:58.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.634+0000 7f017e600640 1 -- 172.21.15.129:0/2568167976 wait complete. 2024-09-17T04:01:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.637+0000 7f017e600640 1 Processor -- start 2024-09-17T04:01:58.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.637+0000 7f017e600640 1 -- start start 2024-09-17T04:01:58.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0178156f90 0x7f01781f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 0x7f0178092780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0178074e90 con 0x7f0178141f60 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0178074d10 con 0x7f017815da40 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0178075010 con 0x7f0178156f90 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51176/0 (socket says 172.21.15.129:51176) 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.638+0000 7f017d200640 1 -- 172.21.15.129:0/4105137878 learned_addr learned my addr 172.21.15.129:0/4105137878 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f0177e00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0178156f90 0x7f01781f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017dc00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 0x7f0178092780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 -- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0178156f90 msgr2=0x7f01781f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0178156f90 0x7f01781f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 -- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 msgr2=0x7f0178092780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 0x7f0178092780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 -- 172.21.15.129:0/4105137878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0178158c90 con 0x7f0178141f60 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017dc00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 0x7f0178092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f017d200640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f016c0099c0 tx=0x7f016c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.639+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c03e030 con 0x7f0178141f60 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.640+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f016c0047a0 con 0x7f0178141f60 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.640+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0178159bd0 con 0x7f0178141f60 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.640+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f016c035040 con 0x7f0178141f60 2024-09-17T04:01:58.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.640+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0178157f50 con 0x7f0178141f60 2024-09-17T04:01:58.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.641+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01780b21f0 con 0x7f0178141f60 2024-09-17T04:01:58.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.642+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f016c041030 con 0x7f0178141f60 2024-09-17T04:01:58.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.643+0000 7f0174c00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 0x7f015807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:01:58.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.643+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f016c0bb4f0 con 0x7f0178141f60 2024-09-17T04:01:58.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.644+0000 7f0177e00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 0x7f015807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:01:58.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.645+0000 7f0177e00640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 0x7f015807b0a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f01781e4310 tx=0x7f016400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:01:58.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.648+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f016c083270 con 0x7f0178141f60 2024-09-17T04:01:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.872+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01780754c0 con 0x7f0178141f60 2024-09-17T04:01:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.873+0000 7f0174c00640 1 -- 172.21.15.129:0/4105137878 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) ==== 95+0+4414 (secure 0 0 0) 0x7f016c087ca0 con 0x7f0178141f60 2024-09-17T04:01:58.873 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:01:58.873 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":168,"btime":"2024-09-17T04:01:58:375100+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:01:58.375081+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:replay","state_seq":1,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-09-17T04:01:58.873 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 168 2024-09-17T04:01:58.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 msgr2=0x7f015807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 0x7f015807b0a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f01781e4310 tx=0x7f016400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 msgr2=0x7f0178158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f016c0099c0 tx=0x7f016c0039e0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 shutdown_connections 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0158078b90 0x7f015807b0a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f017815da40 0x7f0178092780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0178156f90 0x7f01781f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 --2- 172.21.15.129:0/4105137878 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0178141f60 0x7f0178158750 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 >> 172.21.15.129:0/4105137878 conn(0x7f017806d560 msgr2=0x7f01780b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 shutdown_connections 2024-09-17T04:01:58.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:01:58.875+0000 7f017e600640 1 -- 172.21.15.129:0/4105137878 wait complete. 2024-09-17T04:01:59.304 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 168, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:01:42.223951+0000', 'modified': '2024-09-17T04:01:58.375081+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 190, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.129:6833/2632602393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2632602393}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 2632602393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:01:59.304 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10077:up:replay 2024-09-17T04:02:00.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:00.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 -- 172.21.15.129:0/2890751063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b7806f930 msgr2=0x7f8b780af3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:00.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 --2- 172.21.15.129:0/2890751063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b7806f930 0x7f8b780af3f0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f8b68005fd0 tx=0x7f8b6802aa10 comp rx=0 tx=0).stop 2024-09-17T04:02:00.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 -- 172.21.15.129:0/2890751063 shutdown_connections 2024-09-17T04:02:00.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 --2- 172.21.15.129:0/2890751063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b7806f930 0x7f8b780af3f0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 -- 172.21.15.129:0/2890751063 >> 172.21.15.129:0/2890751063 conn(0x7f8b7806d560 msgr2=0x7f8b780b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:00.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 -- 172.21.15.129:0/2890751063 shutdown_connections 2024-09-17T04:02:00.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.532+0000 7f8b7d600640 1 -- 172.21.15.129:0/2890751063 wait complete. 2024-09-17T04:02:00.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.534+0000 7f8b7d600640 1 Processor -- start 2024-09-17T04:02:00.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.534+0000 7f8b7d600640 1 -- start start 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b7806f930 0x7f8b780ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b7815f150 0x7f8b7814af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b780b3c20 con 0x7f8b780b3100 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7815fde0 con 0x7f8b7815f150 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8b780b3da0 con 0x7f8b7806f930 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51198/0 (socket says 172.21.15.129:51198) 2024-09-17T04:02:00.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 -- 172.21.15.129:0/1691565523 learned_addr learned my addr 172.21.15.129:0/1691565523 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77e00640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b7806f930 0x7f8b780ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b7cc00640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b7815f150 0x7f8b7814af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 -- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b7806f930 msgr2=0x7f8b780ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b7806f930 0x7f8b780ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 -- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b7815f150 msgr2=0x7f8b7814af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b7815f150 0x7f8b7814af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.536+0000 7f8b77400640 1 -- 172.21.15.129:0/1691565523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b781f5c10 con 0x7f8b780b3100 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b77400640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400c390 tx=0x7f8b6400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64017070 con 0x7f8b780b3100 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b640035f0 con 0x7f8b780b3100 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b781f6110 con 0x7f8b780b3100 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64003a20 con 0x7f8b780b3100 2024-09-17T04:02:00.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.537+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b781f66d0 con 0x7f8b780b3100 2024-09-17T04:02:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.538+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b781f5e80 con 0x7f8b780b3100 2024-09-17T04:02:00.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.539+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b6401a070 con 0x7f8b780b3100 2024-09-17T04:02:00.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.541+0000 7f8b6fe00640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 0x7f8b5807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:00.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.541+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f8b64099f30 con 0x7f8b780b3100 2024-09-17T04:02:00.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.542+0000 7f8b77e00640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 0x7f8b5807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:00.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.542+0000 7f8b77e00640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 0x7f8b5807b0d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8b68035000 tx=0x7f8b6803b040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:00.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.546+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8b64061d50 con 0x7f8b780b3100 2024-09-17T04:02:00.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.764+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b780af3f0 con 0x7f8b780b3100 2024-09-17T04:02:00.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.765+0000 7f8b6fe00640 1 -- 172.21.15.129:0/1691565523 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+5244 (secure 0 0 0) 0x7f8b64066780 con 0x7f8b780b3100 2024-09-17T04:02:00.765 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:00.766 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":170,"btime":"2024-09-17T04:02:00:384808+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:01:59.394623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":14}]} 2024-09-17T04:02:00.766 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 170 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.767+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 msgr2=0x7f8b5807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.767+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 0x7f8b5807b0d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8b68035000 tx=0x7f8b6803b040 comp rx=0 tx=0).stop 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.767+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 msgr2=0x7f8b78152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.767+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400c390 tx=0x7f8b6400c850 comp rx=0 tx=0).stop 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 shutdown_connections 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b58078bc0 0x7f8b5807b0d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b7815f150 0x7f8b7814af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b780b3100 0x7f8b78152910 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 --2- 172.21.15.129:0/1691565523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b7806f930 0x7f8b780ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:00.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 >> 172.21.15.129:0/1691565523 conn(0x7f8b7806d560 msgr2=0x7f8b7815cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:00.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 shutdown_connections 2024-09-17T04:02:00.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:00.768+0000 7f8b7d600640 1 -- 172.21.15.129:0/1691565523 wait complete. 2024-09-17T04:02:01.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 170, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:01:42.223951+0000', 'modified': '2024-09-17T04:01:59.394623+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 190, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.129:6833/2632602393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2632602393}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 2632602393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:02:01.193 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_10077:up:rejoin 2024-09-17T04:02:02.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 -- 172.21.15.129:0/2583902483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 msgr2=0x7f1e88005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 --2- 172.21.15.129:0/2583902483 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1e88005150 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f1e98005fd0 tx=0x7f1e88005830 comp rx=0 tx=0).stop 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea5e00640 1 -- 172.21.15.129:0/2583902483 reap_dead start 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 -- 172.21.15.129:0/2583902483 shutdown_connections 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 -- 172.21.15.129:0/2583902483 >> 172.21.15.129:0/2583902483 conn(0x7f1ea006d560 msgr2=0x7f1ea00b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 -- 172.21.15.129:0/2583902483 shutdown_connections 2024-09-17T04:02:02.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.431+0000 7f1ea6800640 1 -- 172.21.15.129:0/2583902483 wait complete. 2024-09-17T04:02:02.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.434+0000 7f1ea6800640 1 Processor -- start 2024-09-17T04:02:02.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.434+0000 7f1ea6800640 1 -- start start 2024-09-17T04:02:02.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ea014c810 0x7f1ea009b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ea014cc20 0x7f1ea009b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea00749c0 con 0x7f1e88004d40 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea006e760 con 0x7f1ea014c810 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea0074b40 con 0x7f1ea014cc20 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51216/0 (socket says 172.21.15.129:51216) 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.435+0000 7f1ea5400640 1 -- 172.21.15.129:0/306915775 learned_addr learned my addr 172.21.15.129:0/306915775 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea4a00640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ea014c810 0x7f1ea009b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5e00640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ea014cc20 0x7f1ea009b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:02.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 -- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ea014cc20 msgr2=0x7f1ea009b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ea014cc20 0x7f1ea009b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 -- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ea014c810 msgr2=0x7f1ea009b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ea014c810 0x7f1ea009b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 -- 172.21.15.129:0/306915775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea009a2d0 con 0x7f1e88004d40 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea5400640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1e94011ae0 tx=0x7f1e94009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9401e070 con 0x7f1e88004d40 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1e9400fe70 con 0x7f1e88004d40 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea009a5b0 con 0x7f1e88004d40 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.436+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94004290 con 0x7f1e88004d40 2024-09-17T04:02:02.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.437+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea0153e70 con 0x7f1e88004d40 2024-09-17T04:02:02.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.438+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea0142370 con 0x7f1e88004d40 2024-09-17T04:02:02.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.439+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1e94021070 con 0x7f1e88004d40 2024-09-17T04:02:02.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.440+0000 7f1e9d600640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 0x7f1e8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:02.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.440+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f1e9409b7c0 con 0x7f1e88004d40 2024-09-17T04:02:02.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.440+0000 7f1ea4a00640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 0x7f1e8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:02.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.441+0000 7f1ea4a00640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 0x7f1e8007b0a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0154280 tx=0x7f1e8c0011c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:02.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.444+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1e940636a0 con 0x7f1e88004d40 2024-09-17T04:02:02.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.663+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea01544b0 con 0x7f1e88004d40 2024-09-17T04:02:02.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.663+0000 7f1e9d600640 1 -- 172.21.15.129:0/306915775 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f1e940680d0 con 0x7f1e88004d40 2024-09-17T04:02:02.663 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:02.663 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:02.663 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 msgr2=0x7f1e8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 0x7f1e8007b0a0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0154280 tx=0x7f1e8c0011c0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 msgr2=0x7f1ea0156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f1e94011ae0 tx=0x7f1e94009e90 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 shutdown_connections 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e80078b90 0x7f1e8007b0a0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1ea014cc20 0x7f1ea009b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ea014c810 0x7f1ea009b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 --2- 172.21.15.129:0/306915775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1e88004d40 0x7f1ea0156040 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 >> 172.21.15.129:0/306915775 conn(0x7f1ea006d560 msgr2=0x7f1ea00b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 shutdown_connections 2024-09-17T04:02:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:02.666+0000 7f1ea6800640 1 -- 172.21.15.129:0/306915775 wait complete. 2024-09-17T04:02:03.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2024-09-17T04:02:03.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:01:42.223951+0000', 'modified': '2024-09-17T04:02:02.398276+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 190, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10077}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10077': {'gid': 10077, 'name': 'c', 'rank': 0, 'incarnation': 168, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.129:6833/2632602393', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 2632602393}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 2632602393}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10077, 'qdb_cluster': [10077]} 2024-09-17T04:02:03.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:02:03.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:02:03.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 -- 172.21.15.129:0/3447460513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41340ae890 msgr2=0x7f4134162f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 --2- 172.21.15.129:0/3447460513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41340ae890 0x7f4134162f30 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f4134159710 tx=0x7f412002a530 comp rx=0 tx=0).stop 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 -- 172.21.15.129:0/3447460513 shutdown_connections 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 --2- 172.21.15.129:0/3447460513 >> v2:172.21.15.181:3300/0 conn(0x7f41340aeca0 0x7f413406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 --2- 172.21.15.129:0/3447460513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f41340ae890 0x7f4134162f30 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 -- 172.21.15.129:0/3447460513 >> 172.21.15.129:0/3447460513 conn(0x7f41340b23c0 msgr2=0x7f41340b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 -- 172.21.15.129:0/3447460513 shutdown_connections 2024-09-17T04:02:03.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.314+0000 7f4139c00640 1 -- 172.21.15.129:0/3447460513 wait complete. 2024-09-17T04:02:03.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.317+0000 7f4139c00640 1 Processor -- start 2024-09-17T04:02:03.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.317+0000 7f4139c00640 1 -- start start 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41340ae890 0x7f41340cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41340aeca0 0x7f41341f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41340b5d30 con 0x7f413415f110 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f41340700a0 con 0x7f41340aeca0 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4139c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f41340b5eb0 con 0x7f41340ae890 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4133400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4133400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51226/0 (socket says 172.21.15.129:51226) 2024-09-17T04:02:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.318+0000 7f4133400640 1 -- 172.21.15.129:0/1412181495 learned_addr learned my addr 172.21.15.129:0/1412181495 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4132000640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41340aeca0 0x7f41341f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4132a00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41340ae890 0x7f41340cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 -- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41340ae890 msgr2=0x7f41340cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41340ae890 0x7f41340cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 -- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41340aeca0 msgr2=0x7f41341f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41340aeca0 0x7f41341f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 -- 172.21.15.129:0/1412181495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41341584f0 con 0x7f413415f110 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f4133400640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f412c00e990 tx=0x7f412c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.319+0000 7f412aa00640 1 -- 172.21.15.129:0/1412181495 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c004d20 con 0x7f413415f110 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.320+0000 7f412aa00640 1 -- 172.21.15.129:0/1412181495 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f412c00caa0 con 0x7f413415f110 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.320+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41341587b0 con 0x7f413415f110 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.320+0000 7f412aa00640 1 -- 172.21.15.129:0/1412181495 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c005020 con 0x7f413415f110 2024-09-17T04:02:03.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.320+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4134158d70 con 0x7f413415f110 2024-09-17T04:02:03.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.322+0000 7f412aa00640 1 -- 172.21.15.129:0/1412181495 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f412c0051c0 con 0x7f413415f110 2024-09-17T04:02:03.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.323+0000 7f412aa00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 0x7f411407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.323+0000 7f412aa00640 1 -- 172.21.15.129:0/1412181495 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f412c09b730 con 0x7f413415f110 2024-09-17T04:02:03.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.324+0000 7f4140c1c740 1 Processor -- start 2024-09-17T04:02:03.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.324+0000 7f4132a00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 0x7f411407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.324+0000 7f4140c1c740 1 -- start start 2024-09-17T04:02:03.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.324+0000 7f4132a00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 0x7f411407b0a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4124000f60 tx=0x7f4124009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c67e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c64ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c5d8430 con 0x55ad9c2853b0 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c5c85d0 con 0x55ad9c28d590 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ad9c5b7a00 con 0x55ad9c2ae270 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4132000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4132000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51238/0 (socket says 172.21.15.129:51238) 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4132000640 1 -- 172.21.15.129:0/4108088223 learned_addr learned my addr 172.21.15.129:0/4108088223 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4132a00640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c67e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.325+0000 7f4133400640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c64ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 -- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 msgr2=0x55ad9c67e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c67e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 -- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 msgr2=0x55ad9c64ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c64ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 -- 172.21.15.129:0/4108088223 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad9c580520 con 0x55ad9c2853b0 2024-09-17T04:02:03.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132a00640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c67e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f4132000640 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4120009920 tx=0x7f41200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fdc1ad2b795071 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f411f400640 1 -- 172.21.15.129:0/4108088223 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120005020 con 0x55ad9c2853b0 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f411f400640 1 -- 172.21.15.129:0/4108088223 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41200052b0 con 0x55ad9c2853b0 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.326+0000 7f411f400640 1 -- 172.21.15.129:0/4108088223 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41200047a0 con 0x55ad9c2853b0 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 -- 172.21.15.129:0/4108088223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 msgr2=0x55ad9c67eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f4120009920 tx=0x7f41200046c0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 -- 172.21.15.129:0/4108088223 shutdown_connections 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c64ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67eeb0 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 --2- 172.21.15.129:0/4108088223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c67e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 -- 172.21.15.129:0/4108088223 >> 172.21.15.129:0/4108088223 conn(0x55ad9c2a23c0 msgr2=0x55ad9c654970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 -- 172.21.15.129:0/4108088223 shutdown_connections 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.327+0000 7f4140c1c740 1 -- 172.21.15.129:0/4108088223 wait complete. 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.328+0000 7f4140c1c740 1 -- start start 2024-09-17T04:02:03.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.328+0000 7f4140c1c740 1 Processor -- start 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c675480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c69bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c55e330 con 0x55ad9c2ae270 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c5bb810 con 0x55ad9c28d590 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ad9c328620 con 0x55ad9c2853b0 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4140c1c740 10 client.0 _ll_register_callbacks cb 0x55ad9c563420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4133400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4133400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51252/0 (socket says 172.21.15.129:51252) 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4133400640 1 -- 172.21.15.129:0/645735311 learned_addr learned my addr 172.21.15.129:0/645735311 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:03.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4132a00640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c675480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.330+0000 7f4132000640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c69bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 -- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 msgr2=0x55ad9c675480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c675480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 -- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 msgr2=0x55ad9c69bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c69bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 -- 172.21.15.129:0/645735311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad9c319850 con 0x55ad9c2ae270 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4133400640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f412c0630a0 tx=0x7f412c063920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63915dddab0f28e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4113e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c089020 con 0x55ad9c2ae270 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f412c05ecf0 con 0x55ad9c2ae270 2024-09-17T04:02:03.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41200052b0 con 0x55ad9c2ae270 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f412c01f040 con 0x55ad9c2ae270 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4140c1c740 10 client.10131 fetch_fsmap learned FSMap version 172 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) -- 0x55ad9c3906b0 con 0x55ad9c2ae270 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.331+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f41200052b0 con 0x55ad9c2ae270 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 172) ==== 3502+0+0 (secure 0 0 0) 0x7f412c04ace0 con 0x55ad9c2ae270 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 10 client.10131 fetch_fsmap finished waiting for FSMap version 172 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 10 client.10131 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 20 client.10131 populate_metadata read hostname 'smithi129' 2024-09-17T04:02:03.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c26ce40 0x55ad9c6a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 4 client.10131 mds_command: new command op to 10077 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 --> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ad9c5ca520 con 0x55ad9c26ce40 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4132a00640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c26ce40 0x55ad9c6a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4132a00640 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c26ce40 0x55ad9c6a94c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=804227d06d81413a server_cookie=65210d39a553ef19 in_seq=0 out_seq=0 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.332+0000 7f4113e00640 10 client.10131 ms_handle_connect on v2:172.21.15.129:6832/2632602393 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.333+0000 7f4113e00640 1 -- 172.21.15.129:0/645735311 <== mds.0 v2:172.21.15.129:6832/2632602393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ad9c5ca520 con 0x55ad9c26ce40 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.333+0000 7f4113e00640 10 client.10131 handle_command_reply: tid=0 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.333+0000 7f4140c1c740 1 client.10131 shutdown 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 msgr2=0x55ad9c69c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f412c0630a0 tx=0x7f412c063920 comp rx=0 tx=0).stop 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 shutdown_connections 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c26ce40 0x55ad9c6a94c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2ae270 0x55ad9c69c490 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c69bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 --2- 172.21.15.129:0/645735311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c675480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 >> 172.21.15.129:0/645735311 conn(0x55ad9c2a23c0 msgr2=0x55ad9c69cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4113e00640 0 client.10131 ms_handle_reset on v2:172.21.15.129:6832/2632602393 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 shutdown_connections 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 1 -- 172.21.15.129:0/645735311 wait complete. 2024-09-17T04:02:03.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.334+0000 7f4140c1c740 20 client.10131 trim_cache size 0 max 16384 2024-09-17T04:02:03.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.353+0000 7f4140c1c740 1 Processor -- start 2024-09-17T04:02:03.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.353+0000 7f4140c1c740 1 -- start start 2024-09-17T04:02:03.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c6bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c6bce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f41341322b0 con 0x55ad9c2853b0 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f41341595c0 con 0x55ad9c28d590 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.354+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4134159c10 con 0x55ad9c2ae270 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51262/0 (socket says 172.21.15.129:51262) 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 -- 172.21.15.129:0/1976580639 learned_addr learned my addr 172.21.15.129:0/1976580639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:03.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132000640 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c6bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4133400640 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c6bce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 -- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 msgr2=0x55ad9c6bce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c6bce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 -- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 msgr2=0x55ad9c6bc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c6bc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 -- 172.21.15.129:0/1976580639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ad9c655da0 con 0x55ad9c2853b0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.355+0000 7f4132a00640 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x55ad9c679fe0 tx=0x7f4124012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94d66736bd02746 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f411ea00640 1 -- 172.21.15.129:0/1976580639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124024af0 con 0x55ad9c2853b0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f411ea00640 1 -- 172.21.15.129:0/1976580639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f412402f070 con 0x55ad9c2853b0 2024-09-17T04:02:03.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f411ea00640 1 -- 172.21.15.129:0/1976580639 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4124014450 con 0x55ad9c2853b0 2024-09-17T04:02:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 -- 172.21.15.129:0/1976580639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 msgr2=0x55ad9c67e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x55ad9c679fe0 tx=0x7f4124012000 comp rx=0 tx=0).stop 2024-09-17T04:02:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 -- 172.21.15.129:0/1976580639 shutdown_connections 2024-09-17T04:02:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2ae270 0x55ad9c6bce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c28d590 0x55ad9c6bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 --2- 172.21.15.129:0/1976580639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c2853b0 0x55ad9c67e7e0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.356+0000 7f4140c1c740 1 -- 172.21.15.129:0/1976580639 >> 172.21.15.129:0/1976580639 conn(0x55ad9c2a23c0 msgr2=0x55ad9c6503e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.357+0000 7f4140c1c740 1 -- 172.21.15.129:0/1976580639 shutdown_connections 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.357+0000 7f4140c1c740 1 -- 172.21.15.129:0/1976580639 wait complete. 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.357+0000 7f4140c1c740 1 -- start start 2024-09-17T04:02:03.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.358+0000 7f4140c1c740 1 Processor -- start 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c718b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c2ae270 0x55ad9c69ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c67acf0 con 0x55ad9c28d590 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ad9c6aa400 con 0x55ad9c2ae270 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ad9c678720 con 0x55ad9c2853b0 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4140c1c740 10 client.0 _ll_register_callbacks cb 0x7f412c063260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4132000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4132000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51274/0 (socket says 172.21.15.129:51274) 2024-09-17T04:02:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.359+0000 7f4132000640 1 -- 172.21.15.129:0/664095783 learned_addr learned my addr 172.21.15.129:0/664095783 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4133400640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c2ae270 0x55ad9c69ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132a00640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c718b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 -- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 msgr2=0x55ad9c718b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c718b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 -- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c2ae270 msgr2=0x55ad9c69ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c2ae270 0x55ad9c69ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 -- 172.21.15.129:0/664095783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ad9c6a84b0 con 0x55ad9c28d590 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132a00640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c718b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4132000640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f4120005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed4e98a7d10220f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4113e00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4120056060 con 0x55ad9c28d590 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(190..190 src has 1..190) ==== 5795+0+0 (secure 0 0 0) 0x7f41200054b0 con 0x55ad9c28d590 2024-09-17T04:02:03.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f412402f070 con 0x55ad9c28d590 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) ==== 24+0+0 (secure 0 0 0) 0x7f412005c050 con 0x55ad9c28d590 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4140c1c740 10 client.10137 fetch_fsmap learned FSMap version 172 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.360+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) -- 0x55ad9c6a9cf0 con 0x55ad9c28d590 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f412402f070 con 0x55ad9c28d590 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 172) ==== 3502+0+0 (secure 0 0 0) 0x7f412005f020 con 0x55ad9c28d590 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 10 client.10137 fetch_fsmap finished waiting for FSMap version 172 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 10 client.10137 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 20 client.10137 populate_metadata read hostname 'smithi129' 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c71d290 0x55ad9c6aa670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 4 client.10137 mds_command: new command op to 10077 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 --> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] -- command(tid 0: {"prefix": "status"}) -- 0x55ad9c5c5fd0 con 0x55ad9c71d290 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.361+0000 7f4132a00640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c71d290 0x55ad9c6aa670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:03.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4113e00640 10 client.10137 ms_handle_connect on v2:172.21.15.129:6832/2632602393 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4132a00640 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c71d290 0x55ad9c6aa670 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7ca3112b7920445 server_cookie=70366f96cad841ff in_seq=0 out_seq=0 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4113e00640 1 -- 172.21.15.129:0/664095783 <== mds.0 v2:172.21.15.129:6832/2632602393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55ad9c5c5fd0 con 0x55ad9c71d290 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4113e00640 10 client.10137 handle_command_reply: tid=0 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 client.10137 shutdown 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 msgr2=0x55ad9c719060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f4120000c00 tx=0x7f4120005d80 comp rx=0 tx=0).stop 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 shutdown_connections 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:6832/2632602393,v1:172.21.15.129:6833/2632602393] conn(0x55ad9c71d290 0x55ad9c6aa670 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ad9c2ae270 0x55ad9c69ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ad9c28d590 0x55ad9c719060 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 --2- 172.21.15.129:0/664095783 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ad9c2853b0 0x55ad9c718b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4113e00640 0 client.10137 ms_handle_reset on v2:172.21.15.129:6832/2632602393 2024-09-17T04:02:03.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 >> 172.21.15.129:0/664095783 conn(0x55ad9c2a23c0 msgr2=0x55ad9c6503a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 shutdown_connections 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.362+0000 7f4140c1c740 1 -- 172.21.15.129:0/664095783 wait complete. 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4140c1c740 20 client.10137 trim_cache size 0 max 16384 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 10077, 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:02:03.366 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 4.9848306500000001, 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 172, 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 190, 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 190, 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 8.8928790370000002 2024-09-17T04:02:03.367 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 msgr2=0x7f411407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 0x7f411407b0a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4124000f60 tx=0x7f4124009040 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 msgr2=0x7f41341f8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f412c00e990 tx=0x7f412c00ee50 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 shutdown_connections 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4114078b90 0x7f411407b0a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f413415f110 0x7f41341f8d70 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41340aeca0 0x7f41341f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 --2- 172.21.15.129:0/1412181495 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f41340ae890 0x7f41340cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 >> 172.21.15.129:0/1412181495 conn(0x7f41340b23c0 msgr2=0x7f4134141800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:03.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 shutdown_connections 2024-09-17T04:02:03.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:03.363+0000 7f4139c00640 1 -- 172.21.15.129:0/1412181495 wait complete. 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> subdir = "delete_me" 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> size_unit = 1048576 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> file_multiplier = 100 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> os.mkdir(os.path.join(mountpoint, subdir)) 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(0, file_multiplier): 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> for size in range(0, 16*size_unit, size_unit): 2024-09-17T04:02:03.387 DEBUG:teuthology.orchestra.run.smithi129:> filename = "{0}_{1}.bin".format(i, size // size_unit) 2024-09-17T04:02:03.388 DEBUG:teuthology.orchestra.run.smithi129:> with open(os.path.join(mountpoint, subdir, filename), '"'"'w'"'"') as f: 2024-09-17T04:02:03.388 DEBUG:teuthology.orchestra.run.smithi129:> f.write(size * '"'"'x'"'"') 2024-09-17T04:02:03.388 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:02:05.257 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T04:02:05.258+0000 7f49b2000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:02:18.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:02:18.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:02:46.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:02:46.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_max_purge_ops 2024-09-17T04:02:46.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:02:46.540 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me) 2024-09-17T04:02:46.906 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_max_purge_ops":"16"} 2024-09-17T04:02:46.906 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_ops": "16" } 2024-09-17T04:02:46.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config get mds_max_purge_files 2024-09-17T04:02:47.146 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_max_purge_files":"100000000"} 2024-09-17T04:02:47.146 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_max_purge_files": "100000000" } 2024-09-17T04:02:47.146 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:47.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 -- 172.21.15.129:0/851744161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef00d1d70 msgr2=0x7f5ef00b6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/851744161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef00d1d70 0x7f5ef00b6620 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0073f50 tx=0x7f5ee802a820 comp rx=0 tx=0).stop 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 -- 172.21.15.129:0/851744161 shutdown_connections 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/851744161 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef00d1d70 0x7f5ef00b6620 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/851744161 >> v2:172.21.15.181:3300/0 conn(0x7f5ef0160250 0x7f5ef00b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 -- 172.21.15.129:0/851744161 >> 172.21.15.129:0/851744161 conn(0x7f5ef00b4420 msgr2=0x7f5ef00b4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 -- 172.21.15.129:0/851744161 shutdown_connections 2024-09-17T04:02:47.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.381+0000 7f5ef4a00640 1 -- 172.21.15.129:0/851744161 wait complete. 2024-09-17T04:02:47.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.384+0000 7f5ef4a00640 1 Processor -- start 2024-09-17T04:02:47.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.384+0000 7f5ef4a00640 1 -- start start 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ef00d1d70 0x7f5ef015da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef0145260 0x7f5ef015f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00b6ef0 con 0x7f5ef0160250 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef00b6d70 con 0x7f5ef0145260 2024-09-17T04:02:47.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.385+0000 7f5ef4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef00b7070 con 0x7f5ef00d1d70 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40412/0 (socket says 172.21.15.129:40412) 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 -- 172.21.15.129:0/1335023470 learned_addr learned my addr 172.21.15.129:0/1335023470 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eef400640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ef00d1d70 0x7f5ef015da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eeea00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef0145260 0x7f5ef015f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 -- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ef00d1d70 msgr2=0x7f5ef015da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ef00d1d70 0x7f5ef015da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 -- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef0145260 msgr2=0x7f5ef015f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef0145260 0x7f5ef015f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.386+0000 7f5eefe00640 1 -- 172.21.15.129:0/1335023470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef015fef0 con 0x7f5ef0160250 2024-09-17T04:02:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5eefe00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009ca0 tx=0x7f5ee8032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee803f020 con 0x7f5ef0160250 2024-09-17T04:02:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ee8004040 con 0x7f5ef0160250 2024-09-17T04:02:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef01f6b30 con 0x7f5ef0160250 2024-09-17T04:02:47.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8037040 con 0x7f5ef0160250 2024-09-17T04:02:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.387+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef015cf70 con 0x7f5ef0160250 2024-09-17T04:02:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.388+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef01e6df0 con 0x7f5ef0160250 2024-09-17T04:02:47.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.390+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5ee8042020 con 0x7f5ef0160250 2024-09-17T04:02:47.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.390+0000 7f5ee7400640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 0x7f5ed007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:47.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.391+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f5ee80bb950 con 0x7f5ef0160250 2024-09-17T04:02:47.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.391+0000 7f5eef400640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 0x7f5ed007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:47.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.391+0000 7f5eef400640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 0x7f5ed007b0a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009b30 tx=0x7f5ee0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:47.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.399+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5ee8049020 con 0x7f5ef0160250 2024-09-17T04:02:47.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.635+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ef00724f0 con 0x7f5ef0160250 2024-09-17T04:02:47.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.636+0000 7f5ee7400640 1 -- 172.21.15.129:0/1335023470 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f5ef00724f0 con 0x7f5ef0160250 2024-09-17T04:02:47.636 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:47.636 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:47.636 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:47.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 msgr2=0x7f5ed007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 0x7f5ed007b0a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5ee0009b30 tx=0x7f5ee0005f90 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 msgr2=0x7f5ef01f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8009ca0 tx=0x7f5ee8032c90 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 shutdown_connections 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ed0078b90 0x7f5ed007b0a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ef0160250 0x7f5ef01f65f0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ef0145260 0x7f5ef015f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 --2- 172.21.15.129:0/1335023470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ef00d1d70 0x7f5ef015da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 >> 172.21.15.129:0/1335023470 conn(0x7f5ef00b4420 msgr2=0x7f5ef00b5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 shutdown_connections 2024-09-17T04:02:47.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:47.638+0000 7f5ef4a00640 1 -- 172.21.15.129:0/1335023470 wait complete. 2024-09-17T04:02:47.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:47.930 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":100,"msgr_send_messages":94,"msgr_recv_bytes":148531,"msgr_send_bytes":182032,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.025567341,"msgr_running_send_time":0.007869758,"msgr_running_recv_time":0.013436920,"msgr_running_fast_dispatch_time":0.002897343,"msgr_send_messages_queue_lat":{"avgcount":94,"sum":0.019084188,"avgtime":0.000203023},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125104,"msgr_send_encrypted_bytes":13376},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7262,"msgr_send_messages":9747,"msgr_recv_bytes":2399809,"msgr_send_bytes":9281818,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.478428369,"msgr_running_send_time":1.063229906,"msgr_running_recv_time":0.628899084,"msgr_running_fast_dispatch_time":0.038904189,"msgr_send_messages_queue_lat":{"avgcount":9745,"sum":1.052676533,"avgtime":0.000108022},"msgr_handle_ack_lat":{"avgcount":7514,"sum":0.003664407,"avgtime":0.000000487},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":74528},"AsyncMessenger::Worker-2":{"msgr_recv_messages":672,"msgr_send_messages":675,"msgr_recv_bytes":175363,"msgr_send_bytes":4408945,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.128762681,"msgr_running_send_time":0.061043511,"msgr_running_recv_time":0.028232776,"msgr_running_fast_dispatch_time":0.030763747,"msgr_send_messages_queue_lat":{"avgcount":672,"sum":0.066976605,"avgtime":0.000099667},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":1495,"complete_latency":{"avgcount":3038,"sum":1.653410618,"avgtime":0.000544243}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000105520,"avgtime":0.000026380}},"mds":{"request":2407,"reply":2405,"reply_latency":{"avgcount":2405,"sum":3.218412578,"avgtime":0.001338217},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":2412,"inodes_top":0,"inodes_bottom":2412,"inodes_pin_tail":0,"inodes_pinned":2412,"inodes_expired":0,"inodes_with_caps":1602,"caps":1602,"subtrees":2,"traverse":2407,"traverse_hit":2407,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1362,"root_rbytes":8393850880,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":3360,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":3924,"handle_client_caps_dirty":1328,"handle_client_cap_release":0,"process_request_cap_release":805},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":801,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":801,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":5920,"evex":0,"evtrm":0,"ev":5923,"evexg":0,"evexd":0,"segadd":6,"segex":0,"segtrm":0,"seg":8,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":16909018,"rdpos":4195244,"jlat":{"avgcount":1516,"sum":6.877835629,"avgtime":0.004536830},"replayed":3},"mds_mem":{"ino":1614,"ino+":1614,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":2412,"dn+":2412,"dn-":0,"cap":1602,"cap+":1602,"cap-":0,"rss":148468,"heap":381640704},"mds_server":{"dispatch_client_request":2406,"dispatch_server_request":0,"handle_client_request":2407,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":801,"sum":0.937023264,"avgtime":0.001169816},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2018,"average_load":2018,"avg_session_uptime":1293,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":336580,"buffer_anon_items":1659,"buffer_meta_bytes":2024,"buffer_meta_items":23,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":5303397,"mds_co_items":68663,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":1610,"op_send_bytes":12702970,"op_resend":0,"op_reply":1610,"op_latency":{"avgcount":1610,"sum":7.044847980,"avgtime":0.004375681},"op_inflight":0,"oplen_avg":{"avgcount":1610,"sum":1737},"op":1610,"op_r":36,"op_w":1574,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":1516,"osdop_writefull":4,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":8,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":89,"omap_rd":48,"omap_del":37},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":8027,"get_sum":2098975,"get_or_fail_fail":0,"get_or_fail_success":8027,"take":0,"take_sum":0,"put":8027,"put_sum":2098975,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1610,"take_sum":12703905,"put":1610,"put_sum":12703905,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1610,"take_sum":1610,"put":1610,"put_sum":1610,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":131802,"max":3758096384,"get_started":0,"get":5920,"get_sum":12713515,"get_or_fail_fail":0,"get_or_fail_success":5920,"take":0,"take_sum":0,"put":1516,"put_sum":12581713,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:47.932 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 148531, "msgr_recv_encrypted_bytes": 125104, "msgr_recv_messages": 100, "msgr_running_fast_dispatch_time": 0.002897343, "msgr_running_recv_time": 0.01343692, "msgr_running_send_time": 0.007869758, "msgr_running_total_time": 0.025567341, "msgr_send_bytes": 182032, "msgr_send_encrypted_bytes": 13376, "msgr_send_messages": 94, "msgr_send_messages_queue_lat": { "avgcount": 94, "avgtime": 0.000203023, "sum": 0.019084188 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 7514, "avgtime": 4.87e-07, "sum": 0.003664407 }, "msgr_recv_bytes": 2399809, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7262, "msgr_running_fast_dispatch_time": 0.038904189, "msgr_running_recv_time": 0.628899084, "msgr_running_send_time": 1.063229906, "msgr_running_total_time": 1.478428369, "msgr_send_bytes": 9281818, "msgr_send_encrypted_bytes": 74528, "msgr_send_messages": 9747, "msgr_send_messages_queue_lat": { "avgcount": 9745, "avgtime": 0.000108022, "sum": 1.052676533 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 175363, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 672, "msgr_running_fast_dispatch_time": 0.030763747, "msgr_running_recv_time": 0.028232776, "msgr_running_send_time": 0.061043511, "msgr_running_total_time": 0.128762681, "msgr_send_bytes": 4408945, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 675, "msgr_send_messages_queue_lat": { "avgcount": 672, "avgtime": 9.9667e-05, "sum": 0.066976605 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3038, "avgtime": 0.000544243, "sum": 1.653410618 }, "queue_len": 1495 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.638e-05, "sum": 0.00010552 }, "queue_len": 0 }, "mds": { "caps": 1602, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 3360, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 3924, "handle_client_caps_dirty": 1328, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2412, "inodes_bottom": 2412, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2412, "inodes_top": 0, "inodes_with_caps": 1602, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 805, "q": 0, "reply": 2405, "reply_latency": { "avgcount": 2405, "avgtime": 0.001338217, "sum": 3.218412578 }, "request": 2407, "root_rbytes": 8393850880, "root_rfiles": 1362, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2407, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2407, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 801, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 801, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 5923, "evadd": 5920, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1516, "avgtime": 0.00453683, "sum": 6.877835629 }, "rdpos": 4195244, "replayed": 3, "seg": 8, "segadd": 6, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 16909018 }, "mds_mem": { "cap": 1602, "cap+": 1602, "cap-": 0, "dir": 13, "dir+": 13, "dir-": 0, "dn": 2412, "dn+": 2412, "dn-": 0, "heap": 381640704, "ino": 1614, "ino+": 1614, "ino-": 0, "rss": 148468 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2406, "dispatch_server_request": 0, "handle_client_request": 2407, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 801, "avgtime": 0.001169816, "sum": 0.937023264 } }, "mds_sessions": { "average_load": 2018, "avg_session_uptime": 1293, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2018 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 336580, "buffer_anon_items": 1659, "buffer_meta_bytes": 2024, "buffer_meta_items": 23, "mds_co_bytes": 5303397, "mds_co_items": 68663, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 37, "omap_rd": 48, "omap_wr": 89, "op": 1610, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 1610, "avgtime": 0.004375681, "sum": 7.04484798 }, "op_pg": 0, "op_r": 36, "op_reply": 1610, "op_resend": 0, "op_rmw": 0, "op_send": 1610, "op_send_bytes": 12702970, "op_w": 1574, "oplen_avg": { "avgcount": 1610, "sum": 1737 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 8, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1516, "osdop_writefull": 4, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8027, "get_or_fail_fail": 0, "get_or_fail_success": 8027, "get_started": 0, "get_sum": 2098975, "max": 104857600, "put": 8027, "put_sum": 2098975, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1610, "put_sum": 12703905, "take": 1610, "take_sum": 12703905, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1610, "put_sum": 1610, "take": 1610, "take_sum": 1610, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 5920, "get_or_fail_fail": 0, "get_or_fail_success": 5920, "get_started": 0, "get_sum": 12713515, "max": 3758096384, "put": 1516, "put_sum": 12581713, "take": 0, "take_sum": 0, "val": 131802, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:47.932 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/801, 0/801 2024-09-17T04:02:48.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:02:48.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:02:48.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 -- 172.21.15.129:0/233812303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30400ae890 msgr2=0x7f3040162f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 --2- 172.21.15.129:0/233812303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30400ae890 0x7f3040162f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3030009860 tx=0x7f303002a5a0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 -- 172.21.15.129:0/233812303 shutdown_connections 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 --2- 172.21.15.129:0/233812303 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f30400ae890 0x7f3040162f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 -- 172.21.15.129:0/233812303 >> 172.21.15.129:0/233812303 conn(0x7f30400b23c0 msgr2=0x7f30400b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 -- 172.21.15.129:0/233812303 shutdown_connections 2024-09-17T04:02:49.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.168+0000 7f3048800640 1 -- 172.21.15.129:0/233812303 wait complete. 2024-09-17T04:02:49.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.170+0000 7f3048800640 1 Processor -- start 2024-09-17T04:02:49.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.170+0000 7f3048800640 1 -- start start 2024-09-17T04:02:49.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30400ae890 0x7f30401f8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f304015f110 0x7f30401f9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f30400b60c0 con 0x7f30400aeca0 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f30400b5f40 con 0x7f30400ae890 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3048800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f30400b6240 con 0x7f304015f110 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3045400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3045400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40422/0 (socket says 172.21.15.129:40422) 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3045400640 1 -- 172.21.15.129:0/2116183036 learned_addr learned my addr 172.21.15.129:0/2116183036 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:49.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3045e00640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30400ae890 0x7f30401f8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.172+0000 7f3046800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f304015f110 0x7f30401f9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 -- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f304015f110 msgr2=0x7f30401f9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f304015f110 0x7f30401f9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 -- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30400ae890 msgr2=0x7f30401f8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30400ae890 0x7f30401f8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 -- 172.21.15.129:0/2116183036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304015a330 con 0x7f30400aeca0 2024-09-17T04:02:49.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3045400640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f303002ab90 tx=0x7f3030033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:49.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030005c90 con 0x7f30400aeca0 2024-09-17T04:02:49.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3030031e50 con 0x7f30400aeca0 2024-09-17T04:02:49.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030005260 con 0x7f30400aeca0 2024-09-17T04:02:49.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.173+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304015a610 con 0x7f30400aeca0 2024-09-17T04:02:49.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.174+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304015abd0 con 0x7f30400aeca0 2024-09-17T04:02:49.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.175+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30400b6730 con 0x7f30400aeca0 2024-09-17T04:02:49.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.176+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3030005400 con 0x7f30400aeca0 2024-09-17T04:02:49.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.177+0000 7f303a000640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 0x7f302407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:49.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.177+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f30300bbe30 con 0x7f30400aeca0 2024-09-17T04:02:49.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.178+0000 7f3045e00640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 0x7f302407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:49.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.178+0000 7f3045e00640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 0x7f302407b0a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3034001120 tx=0x7f3034009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:49.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.183+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f303003f030 con 0x7f30400aeca0 2024-09-17T04:02:49.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.414+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30400b1f60 con 0x7f30400aeca0 2024-09-17T04:02:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.415+0000 7f303a000640 1 -- 172.21.15.129:0/2116183036 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f30300840f0 con 0x7f30400aeca0 2024-09-17T04:02:49.414 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:49.414 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:49.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 msgr2=0x7f302407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:49.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 0x7f302407b0a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f3034001120 tx=0x7f3034009040 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 msgr2=0x7f30401f9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f303002ab90 tx=0x7f3030033c20 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 shutdown_connections 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3024078b90 0x7f302407b0a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f304015f110 0x7f30401f9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f30400aeca0 0x7f30401f9040 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 --2- 172.21.15.129:0/2116183036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30400ae890 0x7f30401f8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 >> 172.21.15.129:0/2116183036 conn(0x7f30400b23c0 msgr2=0x7f304006f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 shutdown_connections 2024-09-17T04:02:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:49.417+0000 7f3048800640 1 -- 172.21.15.129:0/2116183036 wait complete. 2024-09-17T04:02:49.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:49.781 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":144,"msgr_send_messages":141,"msgr_recv_bytes":159747,"msgr_send_bytes":211280,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.029958540,"msgr_running_send_time":0.009839979,"msgr_running_recv_time":0.014577009,"msgr_running_fast_dispatch_time":0.003916837,"msgr_send_messages_queue_lat":{"avgcount":141,"sum":0.048891490,"avgtime":0.000346748},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125376,"msgr_send_encrypted_bytes":14016},"AsyncMessenger::Worker-1":{"msgr_recv_messages":7870,"msgr_send_messages":12035,"msgr_recv_bytes":2608751,"msgr_send_bytes":9951116,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.602725543,"msgr_running_send_time":1.158916279,"msgr_running_recv_time":0.657286138,"msgr_running_fast_dispatch_time":0.041979584,"msgr_send_messages_queue_lat":{"avgcount":12028,"sum":1.264592607,"avgtime":0.000105137},"msgr_handle_ack_lat":{"avgcount":7974,"sum":0.003957269,"avgtime":0.000000496},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":74528},"AsyncMessenger::Worker-2":{"msgr_recv_messages":743,"msgr_send_messages":753,"msgr_recv_bytes":251594,"msgr_send_bytes":6624952,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.146535439,"msgr_running_send_time":0.071977743,"msgr_running_recv_time":0.030060533,"msgr_running_fast_dispatch_time":0.035380300,"msgr_send_messages_queue_lat":{"avgcount":749,"sum":0.084048883,"avgtime":0.000112214},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":981,"complete_latency":{"avgcount":3041,"sum":2.031233883,"avgtime":0.000667949}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":192,"sum":0.220616905,"avgtime":0.001149046}},"mds":{"request":2858,"reply":2856,"reply_latency":{"avgcount":2856,"sum":3.746182256,"avgtime":0.001311688},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":2863,"inodes_top":0,"inodes_bottom":2863,"inodes_pin_tail":0,"inodes_pinned":2863,"inodes_expired":0,"inodes_with_caps":1025,"caps":1025,"subtrees":2,"traverse":2858,"traverse_hit":2858,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":798,"root_rbytes":4848615424,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":4174,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4020,"handle_client_caps_dirty":1424,"handle_client_cap_release":4,"process_request_cap_release":1256},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":675,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1252,"strays_enqueued":577,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":7367,"evex":0,"evtrm":0,"ev":7370,"evexg":0,"evexd":0,"segadd":7,"segex":0,"segtrm":0,"seg":9,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":19305433,"rdpos":4195244,"jlat":{"avgcount":1517,"sum":6.903224932,"avgtime":0.004550576},"replayed":3},"mds_mem":{"ino":1614,"ino+":1614,"ino-":0,"dir":13,"dir+":13,"dir-":0,"dn":2863,"dn+":2863,"dn-":0,"cap":1025,"cap+":1602,"cap-":577,"rss":160628,"heap":394223616},"mds_server":{"dispatch_client_request":2857,"dispatch_server_request":0,"handle_client_request":2858,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1252,"sum":1.464792942,"avgtime":0.001169962},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2427,"average_load":2427,"avg_session_uptime":1298,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13809170,"buffer_anon_items":5473,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8110424,"mds_co_items":131982,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":16,"op_laggy":0,"op_send":1795,"op_send_bytes":14907875,"op_resend":0,"op_reply":1779,"op_latency":{"avgcount":1779,"sum":8.496816637,"avgtime":0.004776175},"op_inflight":16,"oplen_avg":{"avgcount":1795,"sum":1923},"op":1795,"op_r":39,"op_w":1756,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":11,"osdop_write":1520,"osdop_writefull":5,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":184,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":91,"omap_rd":48,"omap_del":37},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":18,"pq_executing_ops_high_water":19,"pq_executing":9,"pq_executing_high_water":10,"pq_executed_ops":152,"pq_executed":72,"pq_item_in_journal":505},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":8750,"get_sum":2339090,"get_or_fail_fail":0,"get_or_fail_success":8750,"take":0,"take_sum":0,"put":8750,"put_sum":2339090,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1795,"take_sum":14967087,"put":1779,"put_sum":14967087,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":16,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":1795,"take_sum":1795,"put":1779,"put_sum":1779,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":577,"get_sum":58277,"get_or_fail_fail":0,"get_or_fail_success":577,"take":0,"take_sum":0,"put":3,"put_sum":58277,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":395273,"max":3758096384,"get_started":0,"get":7367,"get_sum":15109930,"get_or_fail_fail":0,"get_or_fail_success":7367,"take":0,"take_sum":0,"put":1517,"put_sum":14714657,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:49.783 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 159747, "msgr_recv_encrypted_bytes": 125376, "msgr_recv_messages": 144, "msgr_running_fast_dispatch_time": 0.003916837, "msgr_running_recv_time": 0.014577009, "msgr_running_send_time": 0.009839979, "msgr_running_total_time": 0.02995854, "msgr_send_bytes": 211280, "msgr_send_encrypted_bytes": 14016, "msgr_send_messages": 141, "msgr_send_messages_queue_lat": { "avgcount": 141, "avgtime": 0.000346748, "sum": 0.04889149 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 7974, "avgtime": 4.96e-07, "sum": 0.003957269 }, "msgr_recv_bytes": 2608751, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 7870, "msgr_running_fast_dispatch_time": 0.041979584, "msgr_running_recv_time": 0.657286138, "msgr_running_send_time": 1.158916279, "msgr_running_total_time": 1.602725543, "msgr_send_bytes": 9951116, "msgr_send_encrypted_bytes": 74528, "msgr_send_messages": 12035, "msgr_send_messages_queue_lat": { "avgcount": 12028, "avgtime": 0.000105137, "sum": 1.264592607 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 251594, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 743, "msgr_running_fast_dispatch_time": 0.0353803, "msgr_running_recv_time": 0.030060533, "msgr_running_send_time": 0.071977743, "msgr_running_total_time": 0.146535439, "msgr_send_bytes": 6624952, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 753, "msgr_send_messages_queue_lat": { "avgcount": 749, "avgtime": 0.000112214, "sum": 0.084048883 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3041, "avgtime": 0.000667949, "sum": 2.031233883 }, "queue_len": 981 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 192, "avgtime": 0.001149046, "sum": 0.220616905 }, "queue_len": 0 }, "mds": { "caps": 1025, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4174, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 4, "handle_client_caps": 4020, "handle_client_caps_dirty": 1424, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2863, "inodes_bottom": 2863, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2863, "inodes_top": 0, "inodes_with_caps": 1025, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1256, "q": 0, "reply": 2856, "reply_latency": { "avgcount": 2856, "avgtime": 0.001311688, "sum": 3.746182256 }, "request": 2858, "root_rbytes": 4848615424, "root_rfiles": 798, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2858, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2858, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 675, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1252, "strays_enqueued": 577, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 7370, "evadd": 7367, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1517, "avgtime": 0.004550576, "sum": 6.903224932 }, "rdpos": 4195244, "replayed": 3, "seg": 9, "segadd": 7, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 19305433 }, "mds_mem": { "cap": 1025, "cap+": 1602, "cap-": 577, "dir": 13, "dir+": 13, "dir-": 0, "dn": 2863, "dn+": 2863, "dn-": 0, "heap": 394223616, "ino": 1614, "ino+": 1614, "ino-": 0, "rss": 160628 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2857, "dispatch_server_request": 0, "handle_client_request": 2858, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1252, "avgtime": 0.001169962, "sum": 1.464792942 } }, "mds_sessions": { "average_load": 2427, "avg_session_uptime": 1298, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2427 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13809170, "buffer_anon_items": 5473, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 8110424, "mds_co_items": 131982, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 37, "omap_rd": 48, "omap_wr": 91, "op": 1795, "op_Infinitylight": 16, "op_active": 16, "op_laggy": 0, "op_latency": { "avgcount": 1779, "avgtime": 0.004776175, "sum": 8.496816637 }, "op_pg": 0, "op_r": 39, "op_reply": 1779, "op_resend": 0, "op_rmw": 0, "op_send": 1795, "op_send_bytes": 14907875, "op_w": 1756, "oplen_avg": { "avgcount": 1795, "sum": 1923 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 184, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 11, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1520, "osdop_writefull": 5, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 72, "pq_executed_ops": 152, "pq_executing": 9, "pq_executing_high_water": 10, "pq_executing_ops": 18, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 505 }, "throttle-msgr_dispatch_throttler-mds": { "get": 8750, "get_or_fail_fail": 0, "get_or_fail_success": 8750, "get_started": 0, "get_sum": 2339090, "max": 104857600, "put": 8750, "put_sum": 2339090, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 1779, "put_sum": 14967087, "take": 1795, "take_sum": 14967087, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 1779, "put_sum": 1779, "take": 1795, "take_sum": 1795, "val": 16, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 577, "get_or_fail_fail": 0, "get_or_fail_success": 577, "get_started": 0, "get_sum": 58277, "max": 3758096384, "put": 3, "put_sum": 58277, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 7367, "get_or_fail_fail": 0, "get_or_fail_success": 7367, "get_started": 0, "get_sum": 15109930, "max": 3758096384, "put": 1517, "put_sum": 14714657, "take": 0, "take_sum": 0, "val": 395273, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:49.783 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 9/675, 72/1252 2024-09-17T04:02:50.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.023+0000 7f6748a00640 1 -- 172.21.15.129:0/2132751354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67440cf2e0 msgr2=0x7f67440d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.023+0000 7f6748a00640 1 --2- 172.21.15.129:0/2132751354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67440cf2e0 0x7f67440d2290 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f672c009830 tx=0x7f672c02a530 comp rx=0 tx=0).stop 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 -- 172.21.15.129:0/2132751354 shutdown_connections 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 --2- 172.21.15.129:0/2132751354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f674415ea40 0x7f67440b7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 --2- 172.21.15.129:0/2132751354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f67440cf2e0 0x7f67440d2290 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 -- 172.21.15.129:0/2132751354 >> 172.21.15.129:0/2132751354 conn(0x7f67440d1770 msgr2=0x7f67440d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:51.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 -- 172.21.15.129:0/2132751354 shutdown_connections 2024-09-17T04:02:51.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.024+0000 7f6748a00640 1 -- 172.21.15.129:0/2132751354 wait complete. 2024-09-17T04:02:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.026+0000 7f6748a00640 1 Processor -- start 2024-09-17T04:02:51.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.026+0000 7f6748a00640 1 -- start start 2024-09-17T04:02:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 0x7f67441f9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 0x7f67441f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 0x7f674414a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f67440d3be0 con 0x7f674415ea40 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f67440d3a60 con 0x7f67440b8150 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6748a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f67440d3d60 con 0x7f67440cf2e0 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6743e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 0x7f674414a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 0x7f67441f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6743e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 0x7f674414a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40438/0 (socket says 172.21.15.129:40438) 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6743e00640 1 -- 172.21.15.129:0/3627969234 learned_addr learned my addr 172.21.15.129:0/3627969234 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6743400640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 0x7f67441f9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 -- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 msgr2=0x7f67441f9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 0x7f67441f9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 -- 172.21.15.129:0/3627969234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 msgr2=0x7f674414a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 0x7f674414a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f673be00640 1 -- 172.21.15.129:0/3627969234 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67441fa140 con 0x7f67440cf2e0 2024-09-17T04:02:51.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.028+0000 7f6743400640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 0x7f67441f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f673be00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 0x7f67441f5e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f673400e700 tx=0x7f673400ebc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673400cab0 con 0x7f67440cf2e0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f673400cd40 con 0x7f67440cf2e0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67440c9620 con 0x7f67440cf2e0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734010690 con 0x7f67440cf2e0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.029+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67440c9be0 con 0x7f67440cf2e0 2024-09-17T04:02:51.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.031+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67440d4210 con 0x7f67440cf2e0 2024-09-17T04:02:51.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.032+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6734010830 con 0x7f67440cf2e0 2024-09-17T04:02:51.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.033+0000 7f673b400640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 0x7f672007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.033+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f673409b7a0 con 0x7f67440cf2e0 2024-09-17T04:02:51.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.033+0000 7f6743400640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 0x7f672007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.034+0000 7f6743400640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 0x7f672007b0a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f672c034000 tx=0x7f672c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:51.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.037+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f67441fa140 con 0x7f67440cf2e0 2024-09-17T04:02:51.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.264+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f67441e75e0 con 0x7f67440cf2e0 2024-09-17T04:02:51.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.265+0000 7f673b400640 1 -- 172.21.15.129:0/3627969234 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f6734063410 con 0x7f67440cf2e0 2024-09-17T04:02:51.265 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:51.265 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:51.265 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 msgr2=0x7f672007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 0x7f672007b0a0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f672c034000 tx=0x7f672c004730 comp rx=0 tx=0).stop 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 msgr2=0x7f67441f5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 0x7f67441f5e90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f673400e700 tx=0x7f673400ebc0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 shutdown_connections 2024-09-17T04:02:51.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6720078b90 0x7f672007b0a0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f674415ea40 0x7f674414a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f67440cf2e0 0x7f67441f5e90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 --2- 172.21.15.129:0/3627969234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f67440b8150 0x7f67441f9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 >> 172.21.15.129:0/3627969234 conn(0x7f67440d1770 msgr2=0x7f6744161170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 shutdown_connections 2024-09-17T04:02:51.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.268+0000 7f6748a00640 1 -- 172.21.15.129:0/3627969234 wait complete. 2024-09-17T04:02:51.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:51.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:51.579 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":471,"msgr_send_messages":470,"msgr_recv_bytes":242923,"msgr_send_bytes":335447,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.067729682,"msgr_running_send_time":0.024203659,"msgr_running_recv_time":0.023308471,"msgr_running_fast_dispatch_time":0.016039709,"msgr_send_messages_queue_lat":{"avgcount":470,"sum":0.093136907,"avgtime":0.000198163},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125376,"msgr_send_encrypted_bytes":14016},"AsyncMessenger::Worker-1":{"msgr_recv_messages":8658,"msgr_send_messages":13932,"msgr_recv_bytes":2855268,"msgr_send_bytes":10589158,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.745689718,"msgr_running_send_time":1.251997071,"msgr_running_recv_time":0.685702037,"msgr_running_fast_dispatch_time":0.055839381,"msgr_send_messages_queue_lat":{"avgcount":13930,"sum":1.634373451,"avgtime":0.000117327},"msgr_handle_ack_lat":{"avgcount":8326,"sum":0.004168076,"avgtime":0.000000500},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":76368},"AsyncMessenger::Worker-2":{"msgr_recv_messages":1338,"msgr_send_messages":1346,"msgr_recv_bytes":436960,"msgr_send_bytes":8914608,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.255876406,"msgr_running_send_time":0.135556287,"msgr_running_recv_time":0.045723569,"msgr_running_fast_dispatch_time":0.060381719,"msgr_send_messages_queue_lat":{"avgcount":1343,"sum":0.214114366,"avgtime":0.000159429},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3056,"sum":2.745523685,"avgtime":0.000898404}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":1798,"sum":0.327905472,"avgtime":0.000182372}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":2626,"inodes_top":0,"inodes_bottom":2626,"inodes_pin_tail":0,"inodes_pinned":2626,"inodes_expired":0,"inodes_with_caps":421,"caps":421,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4710,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4098,"handle_client_caps_dirty":1500,"handle_client_cap_release":8,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":684,"num_strays_delayed":264,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":917,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":8385,"evex":0,"evtrm":0,"ev":8388,"evexg":0,"evexd":0,"segadd":9,"segex":0,"segtrm":0,"seg":11,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":21102606,"rdpos":4195244,"jlat":{"avgcount":1523,"sum":6.978345559,"avgtime":0.004581973},"replayed":3},"mds_mem":{"ino":1037,"ino+":1614,"ino-":577,"dir":13,"dir+":13,"dir-":0,"dn":2635,"dn+":3212,"dn-":577,"cap":421,"cap+":1602,"cap-":1181,"rss":160756,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1298,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13809170,"buffer_anon_items":5473,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8110424,"mds_co_items":131982,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":10,"op_laggy":0,"op_send":3065,"op_send_bytes":17076084,"op_resend":0,"op_reply":3055,"op_latency":{"avgcount":3055,"sum":17.158629729,"avgtime":0.005616572},"op_inflight":10,"oplen_avg":{"avgcount":3065,"sum":3196},"op":3065,"op_r":42,"op_w":3023,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":14,"osdop_write":1529,"osdop_writefull":6,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":1439,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":95,"omap_rd":48,"omap_del":38},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":16,"pq_executing_ops_high_water":19,"pq_executing":6,"pq_executing_high_water":11,"pq_executed_ops":1408,"pq_executed":617,"pq_item_in_journal":301},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":10460,"get_sum":2707193,"get_or_fail_fail":0,"get_or_fail_success":10460,"take":0,"take_sum":0,"put":10460,"put_sum":2707193,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3065,"take_sum":17169636,"put":3055,"put_sum":17169636,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":10,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":3065,"take_sum":3065,"put":3055,"put_sum":3055,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":917,"get_sum":92617,"get_or_fail_fail":0,"get_or_fail_success":917,"take":0,"take_sum":0,"put":6,"put_sum":92617,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":75474,"max":3758096384,"get_started":0,"get":8385,"get_sum":16907103,"get_or_fail_fail":0,"get_or_fail_success":8385,"take":0,"take_sum":0,"put":1523,"put_sum":16831629,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:51.581 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 242923, "msgr_recv_encrypted_bytes": 125376, "msgr_recv_messages": 471, "msgr_running_fast_dispatch_time": 0.016039709, "msgr_running_recv_time": 0.023308471, "msgr_running_send_time": 0.024203659, "msgr_running_total_time": 0.067729682, "msgr_send_bytes": 335447, "msgr_send_encrypted_bytes": 14016, "msgr_send_messages": 470, "msgr_send_messages_queue_lat": { "avgcount": 470, "avgtime": 0.000198163, "sum": 0.093136907 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8326, "avgtime": 5e-07, "sum": 0.004168076 }, "msgr_recv_bytes": 2855268, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 8658, "msgr_running_fast_dispatch_time": 0.055839381, "msgr_running_recv_time": 0.685702037, "msgr_running_send_time": 1.251997071, "msgr_running_total_time": 1.745689718, "msgr_send_bytes": 10589158, "msgr_send_encrypted_bytes": 76368, "msgr_send_messages": 13932, "msgr_send_messages_queue_lat": { "avgcount": 13930, "avgtime": 0.000117327, "sum": 1.634373451 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 436960, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 1338, "msgr_running_fast_dispatch_time": 0.060381719, "msgr_running_recv_time": 0.045723569, "msgr_running_send_time": 0.135556287, "msgr_running_total_time": 0.255876406, "msgr_send_bytes": 8914608, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 1346, "msgr_send_messages_queue_lat": { "avgcount": 1343, "avgtime": 0.000159429, "sum": 0.214114366 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3056, "avgtime": 0.000898404, "sum": 2.745523685 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 1798, "avgtime": 0.000182372, "sum": 0.327905472 }, "queue_len": 0 }, "mds": { "caps": 421, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4710, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 8, "handle_client_caps": 4098, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2626, "inodes_bottom": 2626, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2626, "inodes_top": 0, "inodes_with_caps": 421, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 684, "num_strays_delayed": 264, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 917, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 8388, "evadd": 8385, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 1523, "avgtime": 0.004581973, "sum": 6.978345559 }, "rdpos": 4195244, "replayed": 3, "seg": 11, "segadd": 9, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 21102606 }, "mds_mem": { "cap": 421, "cap+": 1602, "cap-": 1181, "dir": 13, "dir+": 13, "dir-": 0, "dn": 2635, "dn+": 3212, "dn-": 577, "heap": 394223616, "ino": 1037, "ino+": 1614, "ino-": 577, "rss": 160756 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1298, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13809170, "buffer_anon_items": 5473, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 8110424, "mds_co_items": 131982, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 38, "omap_rd": 48, "omap_wr": 95, "op": 3065, "op_Infinitylight": 10, "op_active": 10, "op_laggy": 0, "op_latency": { "avgcount": 3055, "avgtime": 0.005616572, "sum": 17.158629729 }, "op_pg": 0, "op_r": 42, "op_reply": 3055, "op_resend": 0, "op_rmw": 0, "op_send": 3065, "op_send_bytes": 17076084, "op_w": 3023, "oplen_avg": { "avgcount": 3065, "sum": 3196 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 1439, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 14, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1529, "osdop_writefull": 6, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 617, "pq_executed_ops": 1408, "pq_executing": 6, "pq_executing_high_water": 11, "pq_executing_ops": 16, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 301 }, "throttle-msgr_dispatch_throttler-mds": { "get": 10460, "get_or_fail_fail": 0, "get_or_fail_success": 10460, "get_started": 0, "get_sum": 2707193, "max": 104857600, "put": 10460, "put_sum": 2707193, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 3055, "put_sum": 17169636, "take": 3065, "take_sum": 17169636, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 3055, "put_sum": 3055, "take": 3065, "take_sum": 3065, "val": 10, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 917, "get_or_fail_fail": 0, "get_or_fail_success": 917, "get_started": 0, "get_sum": 92617, "max": 3758096384, "put": 6, "put_sum": 92617, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 8385, "get_or_fail_fail": 0, "get_or_fail_success": 8385, "get_started": 0, "get_sum": 16907103, "max": 3758096384, "put": 1523, "put_sum": 16831629, "take": 0, "take_sum": 0, "val": 75474, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:51.581 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 6/684, 617/1601 2024-09-17T04:02:51.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 -- 172.21.15.129:0/3797019904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa04006fe20 msgr2=0x7fa040151720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 --2- 172.21.15.129:0/3797019904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa04006fe20 0x7fa040151720 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009830 tx=0x7fa02c02a520 comp rx=0 tx=0).stop 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 -- 172.21.15.129:0/3797019904 shutdown_connections 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 --2- 172.21.15.129:0/3797019904 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa040151c60 0x7fa0400726a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 --2- 172.21.15.129:0/3797019904 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa04006fe20 0x7fa040151720 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.814+0000 7fa045c00640 1 -- 172.21.15.129:0/3797019904 >> 172.21.15.129:0/3797019904 conn(0x7fa0400b23c0 msgr2=0x7fa0400b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.815+0000 7fa045c00640 1 -- 172.21.15.129:0/3797019904 shutdown_connections 2024-09-17T04:02:51.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.815+0000 7fa045c00640 1 -- 172.21.15.129:0/3797019904 wait complete. 2024-09-17T04:02:51.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.820+0000 7fa045c00640 1 Processor -- start 2024-09-17T04:02:51.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.820+0000 7fa045c00640 1 -- start start 2024-09-17T04:02:51.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0400b4340 0x7fa040140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0401508b0 0x7fa040150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa040072f90 con 0x7fa040151c60 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa04015b6e0 con 0x7fa0400b4340 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa045c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa0400cf150 con 0x7fa0401508b0 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa03e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa03e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:40464/0 (socket says 172.21.15.129:40464) 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.822+0000 7fa03e000640 1 -- 172.21.15.129:0/4124141082 learned_addr learned my addr 172.21.15.129:0/4124141082 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03ea00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0400b4340 0x7fa040140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03f400640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0401508b0 0x7fa040150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03e000640 1 -- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0401508b0 msgr2=0x7fa040150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03e000640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0401508b0 0x7fa040150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03e000640 1 -- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0400b4340 msgr2=0x7fa040140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03e000640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0400b4340 0x7fa040140ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.823+0000 7fa03e000640 1 -- 172.21.15.129:0/4124141082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa040151200 con 0x7fa040151c60 2024-09-17T04:02:51.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.824+0000 7fa03e000640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009950 tx=0x7fa02c033ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:51.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.824+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c031d60 con 0x7fa040151c60 2024-09-17T04:02:51.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.824+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa02c0330d0 con 0x7fa040151c60 2024-09-17T04:02:51.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.824+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa02c033520 con 0x7fa040151c60 2024-09-17T04:02:51.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.826+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa04014f160 con 0x7fa040151c60 2024-09-17T04:02:51.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.826+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa040148f90 con 0x7fa040151c60 2024-09-17T04:02:51.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.828+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa040151720 con 0x7fa040151c60 2024-09-17T04:02:51.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.832+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa02c0035f0 con 0x7fa040151c60 2024-09-17T04:02:51.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.834+0000 7fa036a00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 0x7fa02007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:51.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.834+0000 7fa03ea00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 0x7fa02007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:51.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.834+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7fa02c0bc2b0 con 0x7fa040151c60 2024-09-17T04:02:51.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.835+0000 7fa03ea00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 0x7fa02007b0a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa030000f60 tx=0x7fa030009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:51.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:51.840+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa02c04a030 con 0x7fa040151c60 2024-09-17T04:02:52.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.147+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa040069b10 con 0x7fa040151c60 2024-09-17T04:02:52.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.148+0000 7fa036a00640 1 -- 172.21.15.129:0/4124141082 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7fa040151200 con 0x7fa040151c60 2024-09-17T04:02:52.147 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:52.147 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:52.148 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.151+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 msgr2=0x7fa02007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.151+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 0x7fa02007b0a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa030000f60 tx=0x7fa030009040 comp rx=0 tx=0).stop 2024-09-17T04:02:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.151+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 msgr2=0x7fa04014ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.151+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009950 tx=0x7fa02c033ce0 comp rx=0 tx=0).stop 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 shutdown_connections 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa020078b90 0x7fa02007b0a0 secure :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fa030000f60 tx=0x7fa030009040 comp rx=0 tx=0).stop 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0401508b0 0x7fa040150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa040151c60 0x7fa04014ec20 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 --2- 172.21.15.129:0/4124141082 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0400b4340 0x7fa040140ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:52.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.152+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 >> 172.21.15.129:0/4124141082 conn(0x7fa0400b23c0 msgr2=0x7fa0400b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.153+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 shutdown_connections 2024-09-17T04:02:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:52.153+0000 7fa045c00640 1 -- 172.21.15.129:0/4124141082 wait complete. 2024-09-17T04:02:52.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-17T04:02:52.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:52.897 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T04:02:52.898 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T04:02:53.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.092+0000 7fbc21a00640 1 -- 172.21.15.129:0/3125420703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c162850 msgr2=0x7fbc1c162c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:53.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.094+0000 7fbc1b400640 1 -- 172.21.15.129:0/3125420703 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04031600 con 0x7fbc1c162850 2024-09-17T04:02:53.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.094+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3125420703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c162850 0x7fbc1c162c60 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009880 tx=0x7fbc0402a4d0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.096+0000 7fbc21a00640 1 -- 172.21.15.129:0/3125420703 shutdown_connections 2024-09-17T04:02:53.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.096+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3125420703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c162850 0x7fbc1c162c60 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.096+0000 7fbc21a00640 1 -- 172.21.15.129:0/3125420703 >> 172.21.15.129:0/3125420703 conn(0x7fbc1c0d4b70 msgr2=0x7fbc1c0d4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:53.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.097+0000 7fbc21a00640 1 -- 172.21.15.129:0/3125420703 shutdown_connections 2024-09-17T04:02:53.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.097+0000 7fbc21a00640 1 -- 172.21.15.129:0/3125420703 wait complete. 2024-09-17T04:02:53.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.106+0000 7fbc21a00640 1 Processor -- start 2024-09-17T04:02:53.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.107+0000 7fbc21a00640 1 -- start start 2024-09-17T04:02:53.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.114+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 0x7fbc1c1f96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:53.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.115+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc13e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.115+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 0x7fbc1c1f96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc13e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:36504/0 (socket says 172.21.15.129:36504) 2024-09-17T04:02:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.115+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 0x7fbc1c1ea810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:53.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc21a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c15f9b0 con 0x7fbc1c06ea50 2024-09-17T04:02:53.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc13e00640 1 -- 172.21.15.129:0/3761769272 learned_addr learned my addr 172.21.15.129:0/3761769272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:53.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c15f830 con 0x7fbc1c0d28a0 2024-09-17T04:02:53.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.116+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1c15fb30 con 0x7fbc1c162850 2024-09-17T04:02:53.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.117+0000 7fbc21000640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 0x7fbc1c1ea810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:53.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.117+0000 7fbc21000640 1 -- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 msgr2=0x7fbc1c0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.117+0000 7fbc21000640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.117+0000 7fbc21000640 1 -- 172.21.15.129:0/3761769272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 msgr2=0x7fbc1c1f96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.118+0000 7fbc21000640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 0x7fbc1c1f96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.118+0000 7fbc21000640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c149720 con 0x7fbc1c162850 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.118+0000 7fbc13e00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:02:53.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.118+0000 7fbc1be00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 0x7fbc1c1f96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:02:53.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.120+0000 7fbc21000640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 0x7fbc1c1ea810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc14014950 tx=0x7fbc14014e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:53.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.121+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc14022070 con 0x7fbc1c162850 2024-09-17T04:02:53.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.121+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc14015b40 con 0x7fbc1c162850 2024-09-17T04:02:53.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.121+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1401d440 con 0x7fbc1c162850 2024-09-17T04:02:53.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.121+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c1499c0 con 0x7fbc1c162850 2024-09-17T04:02:53.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.121+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c149f80 con 0x7fbc1c162850 2024-09-17T04:02:53.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.125+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c15f480 con 0x7fbc1c162850 2024-09-17T04:02:53.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.126+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbc14025070 con 0x7fbc1c162850 2024-09-17T04:02:53.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.129+0000 7fbc18c00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 0x7fbbf807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:53.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.129+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7fbc140a3a10 con 0x7fbc1c162850 2024-09-17T04:02:53.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.132+0000 7fbc1be00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 0x7fbbf807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:53.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.133+0000 7fbc1be00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 0x7fbbf807b0d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c0016a0 tx=0x7fbc0c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:53.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.136+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbc1406bde0 con 0x7fbc1c162850 2024-09-17T04:02:53.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.581+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc1c1eb6a0 con 0x7fbc1c162850 2024-09-17T04:02:53.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.582+0000 7fbc18c00640 1 -- 172.21.15.129:0/3761769272 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7fbc1406b6e0 con 0x7fbc1c162850 2024-09-17T04:02:53.583 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:53.584 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:53.584 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:53.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.586+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 msgr2=0x7fbbf807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:53.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.586+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 0x7fbbf807b0d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c0016a0 tx=0x7fbc0c009040 comp rx=0 tx=0).stop 2024-09-17T04:02:53.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.586+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 msgr2=0x7fbc1c1ea810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.586+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 0x7fbc1c1ea810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc14014950 tx=0x7fbc14014e10 comp rx=0 tx=0).stop 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 shutdown_connections 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbf8078bc0 0x7fbbf807b0d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc1c162850 0x7fbc1c1ea810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc1c0d28a0 0x7fbc1c0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 --2- 172.21.15.129:0/3761769272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc1c06ea50 0x7fbc1c1f96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 >> 172.21.15.129:0/3761769272 conn(0x7fbc1c0d4b70 msgr2=0x7fbc1c0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 shutdown_connections 2024-09-17T04:02:53.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:53.587+0000 7fbc21a00640 1 -- 172.21.15.129:0/3761769272 wait complete. 2024-09-17T04:02:53.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:54.050 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1041,"msgr_send_messages":1035,"msgr_recv_bytes":398161,"msgr_send_bytes":608105,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.178977917,"msgr_running_send_time":0.084404629,"msgr_running_recv_time":0.037469498,"msgr_running_fast_dispatch_time":0.049447570,"msgr_send_messages_queue_lat":{"avgcount":1035,"sum":2.247963923,"avgtime":0.002171945},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125648,"msgr_send_encrypted_bytes":14656},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9285,"msgr_send_messages":14492,"msgr_recv_bytes":3031883,"msgr_send_bytes":11155472,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.886274723,"msgr_running_send_time":1.332773275,"msgr_running_recv_time":0.707514442,"msgr_running_fast_dispatch_time":0.092664174,"msgr_send_messages_queue_lat":{"avgcount":14490,"sum":2.570968731,"avgtime":0.000177430},"msgr_handle_ack_lat":{"avgcount":8329,"sum":0.004210993,"avgtime":0.000000505},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":76368},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2209,"msgr_send_messages":2212,"msgr_recv_bytes":710422,"msgr_send_bytes":9414157,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.447295187,"msgr_running_send_time":0.249610324,"msgr_running_recv_time":0.069579782,"msgr_running_fast_dispatch_time":0.107176146,"msgr_send_messages_queue_lat":{"avgcount":2209,"sum":2.801254847,"avgtime":0.001268109},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3081,"sum":2.926649704,"avgtime":0.000949902}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3755,"sum":0.484485023,"avgtime":0.000129023}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":2626,"inodes_top":0,"inodes_bottom":2626,"inodes_pin_tail":0,"inodes_pinned":2626,"inodes_expired":0,"inodes_with_caps":421,"caps":421,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4733,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4170,"handle_client_caps_dirty":1500,"handle_client_cap_release":23,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":323,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1278,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":8819,"evex":8670,"evtrm":8670,"ev":152,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21463404,"rdpos":4195244,"jlat":{"avgcount":1525,"sum":7.037246640,"avgtime":0.004614587},"replayed":3},"mds_mem":{"ino":433,"ino+":1614,"ino-":1181,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":318,"cap+":1602,"cap-":1284,"rss":150044,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1298,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":13809170,"buffer_anon_items":5473,"buffer_meta_bytes":176,"buffer_meta_items":2,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":8110424,"mds_co_items":131982,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5032,"op_send_bytes":17737876,"op_resend":0,"op_reply":5032,"op_latency":{"avgcount":5032,"sum":69.991489675,"avgtime":0.013909278},"op_inflight":0,"oplen_avg":{"avgcount":5032,"sum":6037},"op":5032,"op_r":48,"op_w":4984,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":20,"osdop_write":1537,"osdop_writefull":14,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2942,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":119,"omap_rd":48,"omap_del":42},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":2923,"pq_executed":1278,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12528,"get_sum":3127776,"get_or_fail_fail":0,"get_or_fail_success":12528,"take":0,"take_sum":0,"put":12528,"put_sum":3127776,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5032,"take_sum":17867889,"put":5032,"put_sum":17867889,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5032,"take_sum":5032,"put":5032,"put_sum":5032,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1278,"get_sum":129078,"get_or_fail_fail":0,"get_or_fail_success":1278,"take":0,"take_sum":0,"put":12,"put_sum":129078,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":155878,"max":3758096384,"get_started":0,"get":8819,"get_sum":17267901,"get_or_fail_fail":0,"get_or_fail_success":8819,"take":0,"take_sum":0,"put":1525,"put_sum":17112023,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:54.052 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 398161, "msgr_recv_encrypted_bytes": 125648, "msgr_recv_messages": 1041, "msgr_running_fast_dispatch_time": 0.04944757, "msgr_running_recv_time": 0.037469498, "msgr_running_send_time": 0.084404629, "msgr_running_total_time": 0.178977917, "msgr_send_bytes": 608105, "msgr_send_encrypted_bytes": 14656, "msgr_send_messages": 1035, "msgr_send_messages_queue_lat": { "avgcount": 1035, "avgtime": 0.002171945, "sum": 2.247963923 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8329, "avgtime": 5.05e-07, "sum": 0.004210993 }, "msgr_recv_bytes": 3031883, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9285, "msgr_running_fast_dispatch_time": 0.092664174, "msgr_running_recv_time": 0.707514442, "msgr_running_send_time": 1.332773275, "msgr_running_total_time": 1.886274723, "msgr_send_bytes": 11155472, "msgr_send_encrypted_bytes": 76368, "msgr_send_messages": 14492, "msgr_send_messages_queue_lat": { "avgcount": 14490, "avgtime": 0.00017743, "sum": 2.570968731 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 710422, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2209, "msgr_running_fast_dispatch_time": 0.107176146, "msgr_running_recv_time": 0.069579782, "msgr_running_send_time": 0.249610324, "msgr_running_total_time": 0.447295187, "msgr_send_bytes": 9414157, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2212, "msgr_send_messages_queue_lat": { "avgcount": 2209, "avgtime": 0.001268109, "sum": 2.801254847 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3081, "avgtime": 0.000949902, "sum": 2.926649704 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3755, "avgtime": 0.000129023, "sum": 0.484485023 }, "queue_len": 0 }, "mds": { "caps": 421, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4733, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 23, "handle_client_caps": 4170, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 2626, "inodes_bottom": 2626, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 2626, "inodes_top": 0, "inodes_with_caps": 421, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 323, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1278, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 152, "evadd": 8819, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1525, "avgtime": 0.004614587, "sum": 7.03724664 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21463404 }, "mds_mem": { "cap": 318, "cap+": 1602, "cap-": 1284, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 433, "ino+": 1614, "ino-": 1181, "rss": 150044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1298, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 13809170, "buffer_anon_items": 5473, "buffer_meta_bytes": 176, "buffer_meta_items": 2, "mds_co_bytes": 8110424, "mds_co_items": 131982, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 42, "omap_rd": 48, "omap_wr": 119, "op": 5032, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5032, "avgtime": 0.013909278, "sum": 69.991489675 }, "op_pg": 0, "op_r": 48, "op_reply": 5032, "op_resend": 0, "op_rmw": 0, "op_send": 5032, "op_send_bytes": 17737876, "op_w": 4984, "oplen_avg": { "avgcount": 5032, "sum": 6037 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 2942, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 20, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1537, "osdop_writefull": 14, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1278, "pq_executed_ops": 2923, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12528, "get_or_fail_fail": 0, "get_or_fail_success": 12528, "get_started": 0, "get_sum": 3127776, "max": 104857600, "put": 12528, "put_sum": 3127776, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5032, "put_sum": 17867889, "take": 5032, "take_sum": 17867889, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5032, "put_sum": 5032, "take": 5032, "take_sum": 5032, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1278, "get_or_fail_fail": 0, "get_or_fail_success": 1278, "get_started": 0, "get_sum": 129078, "max": 3758096384, "put": 12, "put_sum": 129078, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 8819, "get_or_fail_fail": 0, "get_or_fail_success": 8819, "get_started": 0, "get_sum": 17267901, "max": 3758096384, "put": 1525, "put_sum": 17112023, "take": 0, "take_sum": 0, "val": 155878, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:54.052 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/323, 1278/1601 2024-09-17T04:02:55.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.302+0000 7fd500200640 1 -- 172.21.15.129:0/3617508381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f815f880 msgr2=0x7fd4f815fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.302+0000 7fd500200640 1 --2- 172.21.15.129:0/3617508381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f815f880 0x7fd4f815fc90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd4e4009830 tx=0x7fd4e402a530 comp rx=0 tx=0).stop 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.302+0000 7fd500200640 1 -- 172.21.15.129:0/3617508381 shutdown_connections 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.302+0000 7fd500200640 1 --2- 172.21.15.129:0/3617508381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f815f880 0x7fd4f815fc90 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.302+0000 7fd500200640 1 -- 172.21.15.129:0/3617508381 >> 172.21.15.129:0/3617508381 conn(0x7fd4f80b23c0 msgr2=0x7fd4f80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.303+0000 7fd500200640 1 -- 172.21.15.129:0/3617508381 shutdown_connections 2024-09-17T04:02:55.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.303+0000 7fd500200640 1 -- 172.21.15.129:0/3617508381 wait complete. 2024-09-17T04:02:55.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.305+0000 7fd500200640 1 Processor -- start 2024-09-17T04:02:55.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.305+0000 7fd500200640 1 -- start start 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 0x7fd4f81551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 0x7fd4f8142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4f815f880 0x7fd4f8143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80cfca0 con 0x7fd4f80cedf0 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd4f80cfb20 con 0x7fd4f80737b0 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.306+0000 7fd500200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd4f80cfe20 con 0x7fd4f815f880 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 0x7fd4f8142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 0x7fd4f81551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 0x7fd4f8142b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49244/0 (socket says 172.21.15.129:49244) 2024-09-17T04:02:55.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fce00640 1 -- 172.21.15.129:0/553035825 learned_addr learned my addr 172.21.15.129:0/553035825 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:55.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fe200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4f815f880 0x7fd4f8143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 -- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4f815f880 msgr2=0x7fd4f8143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4f815f880 0x7fd4f8143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 -- 172.21.15.129:0/553035825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 msgr2=0x7fd4f8142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 0x7fd4f8142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.307+0000 7fd4fd800640 1 -- 172.21.15.129:0/553035825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4f8156970 con 0x7fd4f80737b0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd4fd800640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 0x7fd4f81551a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c3c0 tx=0x7fd4ec00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec011790 con 0x7fd4f80737b0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4ec017070 con 0x7fd4f80737b0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4f8156c50 con 0x7fd4f80737b0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec011cc0 con 0x7fd4f80737b0 2024-09-17T04:02:55.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.308+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4f8157210 con 0x7fd4f80737b0 2024-09-17T04:02:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4f815fc90 con 0x7fd4f80737b0 2024-09-17T04:02:55.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd4ec01c460 con 0x7fd4f80737b0 2024-09-17T04:02:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd4f5600640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 0x7fd4d807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7fd4ec09aac0 con 0x7fd4f80737b0 2024-09-17T04:02:55.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd4fce00640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 0x7fd4d807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:55.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.309+0000 7fd4fce00640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 0x7fd4d807b0a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40095f0 tx=0x7fd4e4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:55.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.314+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4f80ce930 con 0x7fd4f80737b0 2024-09-17T04:02:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.639+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4f8069b10 con 0x7fd4f80737b0 2024-09-17T04:02:55.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.640+0000 7fd4f5600640 1 -- 172.21.15.129:0/553035825 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7fd4ec062730 con 0x7fd4f80737b0 2024-09-17T04:02:55.639 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:55.640 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:55.640 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:55.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.645+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 msgr2=0x7fd4d807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:55.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.645+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 0x7fd4d807b0a0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fd4e40095f0 tx=0x7fd4e4004730 comp rx=0 tx=0).stop 2024-09-17T04:02:55.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.645+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 msgr2=0x7fd4f81551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:55.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.645+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 0x7fd4f81551a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec00c3c0 tx=0x7fd4ec00c880 comp rx=0 tx=0).stop 2024-09-17T04:02:55.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 shutdown_connections 2024-09-17T04:02:55.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4d8078b90 0x7fd4d807b0a0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4f815f880 0x7fd4f8143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd4f80cedf0 0x7fd4f8142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 --2- 172.21.15.129:0/553035825 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4f80737b0 0x7fd4f81551a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:55.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.648+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 >> 172.21.15.129:0/553035825 conn(0x7fd4f80b23c0 msgr2=0x7fd4f80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:55.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.650+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 shutdown_connections 2024-09-17T04:02:55.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:55.650+0000 7fd500200640 1 -- 172.21.15.129:0/553035825 wait complete. 2024-09-17T04:02:55.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:56.008 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1047,"msgr_send_messages":1041,"msgr_recv_bytes":399761,"msgr_send_bytes":610971,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.179894479,"msgr_running_send_time":0.084769217,"msgr_running_recv_time":0.037653094,"msgr_running_fast_dispatch_time":0.049656074,"msgr_send_messages_queue_lat":{"avgcount":1041,"sum":2.249727061,"avgtime":0.002161121},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125648,"msgr_send_encrypted_bytes":14656},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9414,"msgr_send_messages":14563,"msgr_recv_bytes":3067129,"msgr_send_bytes":11374640,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.912114355,"msgr_running_send_time":1.351376130,"msgr_running_recv_time":0.832604530,"msgr_running_fast_dispatch_time":0.093519426,"msgr_send_messages_queue_lat":{"avgcount":14561,"sum":2.736349386,"avgtime":0.000187923},"msgr_handle_ack_lat":{"avgcount":8330,"sum":0.004225296,"avgtime":0.000000507},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":78208},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2220,"msgr_send_messages":2223,"msgr_recv_bytes":713782,"msgr_send_bytes":9418399,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.448584509,"msgr_running_send_time":0.250205336,"msgr_running_recv_time":0.069866341,"msgr_running_fast_dispatch_time":0.107471867,"msgr_send_messages_queue_lat":{"avgcount":2220,"sum":2.807943369,"avgtime":0.001264839},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3086,"sum":2.974959750,"avgtime":0.000964018}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3786,"sum":0.493392291,"avgtime":0.000130320}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":318,"caps":318,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4793,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4241,"handle_client_caps_dirty":1500,"handle_client_cap_release":69,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":317,"num_strays_delayed":60,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1284,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":8897,"evex":8670,"evtrm":8670,"ev":230,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21612959,"rdpos":4195244,"jlat":{"avgcount":1527,"sum":7.047674376,"avgtime":0.004615372},"replayed":3},"mds_mem":{"ino":336,"ino+":1614,"ino-":1278,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":217,"cap+":1602,"cap-":1385,"rss":150044,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":3,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1303,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2482267,"buffer_anon_items":1065,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1658746,"mds_co_items":33062,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5059,"op_send_bytes":17932918,"op_resend":0,"op_reply":5059,"op_latency":{"avgcount":5059,"sum":70.317482645,"avgtime":0.013899482},"op_inflight":0,"oplen_avg":{"avgcount":5059,"sum":6066},"op":5059,"op_r":50,"op_w":5009,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":22,"osdop_write":1541,"osdop_writefull":15,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":2960,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":121,"omap_rd":48,"omap_del":44},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":2941,"pq_executed":1284,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12674,"get_sum":3156746,"get_or_fail_fail":0,"get_or_fail_success":12674,"take":0,"take_sum":0,"put":12674,"put_sum":3156746,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5059,"take_sum":18063537,"put":5059,"put_sum":18063537,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5059,"take_sum":5059,"put":5059,"put_sum":5059,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1284,"get_sum":129684,"get_or_fail_fail":0,"get_or_fail_success":1284,"take":0,"take_sum":0,"put":14,"put_sum":129684,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":111929,"max":3758096384,"get_started":0,"get":8897,"get_sum":17417456,"get_or_fail_fail":0,"get_or_fail_success":8897,"take":0,"take_sum":0,"put":1527,"put_sum":17305527,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:56.010 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 399761, "msgr_recv_encrypted_bytes": 125648, "msgr_recv_messages": 1047, "msgr_running_fast_dispatch_time": 0.049656074, "msgr_running_recv_time": 0.037653094, "msgr_running_send_time": 0.084769217, "msgr_running_total_time": 0.179894479, "msgr_send_bytes": 610971, "msgr_send_encrypted_bytes": 14656, "msgr_send_messages": 1041, "msgr_send_messages_queue_lat": { "avgcount": 1041, "avgtime": 0.002161121, "sum": 2.249727061 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8330, "avgtime": 5.07e-07, "sum": 0.004225296 }, "msgr_recv_bytes": 3067129, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9414, "msgr_running_fast_dispatch_time": 0.093519426, "msgr_running_recv_time": 0.83260453, "msgr_running_send_time": 1.35137613, "msgr_running_total_time": 1.912114355, "msgr_send_bytes": 11374640, "msgr_send_encrypted_bytes": 78208, "msgr_send_messages": 14563, "msgr_send_messages_queue_lat": { "avgcount": 14561, "avgtime": 0.000187923, "sum": 2.736349386 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 713782, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2220, "msgr_running_fast_dispatch_time": 0.107471867, "msgr_running_recv_time": 0.069866341, "msgr_running_send_time": 0.250205336, "msgr_running_total_time": 0.448584509, "msgr_send_bytes": 9418399, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2223, "msgr_send_messages_queue_lat": { "avgcount": 2220, "avgtime": 0.001264839, "sum": 2.807943369 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3086, "avgtime": 0.000964018, "sum": 2.97495975 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3786, "avgtime": 0.00013032, "sum": 0.493392291 }, "queue_len": 0 }, "mds": { "caps": 318, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4793, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 69, "handle_client_caps": 4241, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 318, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 317, "num_strays_delayed": 60, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1284, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 230, "evadd": 8897, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1527, "avgtime": 0.004615372, "sum": 7.047674376 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21612959 }, "mds_mem": { "cap": 217, "cap+": 1602, "cap-": 1385, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 336, "ino+": 1614, "ino-": 1278, "rss": 150044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 3, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1303, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2482267, "buffer_anon_items": 1065, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1658746, "mds_co_items": 33062, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 44, "omap_rd": 48, "omap_wr": 121, "op": 5059, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5059, "avgtime": 0.013899482, "sum": 70.317482645 }, "op_pg": 0, "op_r": 50, "op_reply": 5059, "op_resend": 0, "op_rmw": 0, "op_send": 5059, "op_send_bytes": 17932918, "op_w": 5009, "oplen_avg": { "avgcount": 5059, "sum": 6066 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 2960, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 22, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1541, "osdop_writefull": 15, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1284, "pq_executed_ops": 2941, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12674, "get_or_fail_fail": 0, "get_or_fail_success": 12674, "get_started": 0, "get_sum": 3156746, "max": 104857600, "put": 12674, "put_sum": 3156746, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5059, "put_sum": 18063537, "take": 5059, "take_sum": 18063537, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5059, "put_sum": 5059, "take": 5059, "take_sum": 5059, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1284, "get_or_fail_fail": 0, "get_or_fail_success": 1284, "get_started": 0, "get_sum": 129684, "max": 3758096384, "put": 14, "put_sum": 129684, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 8897, "get_or_fail_fail": 0, "get_or_fail_success": 8897, "get_started": 0, "get_sum": 17417456, "max": 3758096384, "put": 1527, "put_sum": 17305527, "take": 0, "take_sum": 0, "val": 111929, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:56.010 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/317, 1284/1601 2024-09-17T04:02:57.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:57.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 -- 172.21.15.129:0/729718223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c06deb0 msgr2=0x7f499c06e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:57.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 --2- 172.21.15.129:0/729718223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c06deb0 0x7f499c06e2c0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f4994005fd0 tx=0x7f499402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 -- 172.21.15.129:0/729718223 shutdown_connections 2024-09-17T04:02:57.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 --2- 172.21.15.129:0/729718223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c06deb0 0x7f499c06e2c0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 --2- 172.21.15.129:0/729718223 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 0x7f499c158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 -- 172.21.15.129:0/729718223 >> 172.21.15.129:0/729718223 conn(0x7f499c070730 msgr2=0x7f499c070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:57.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 -- 172.21.15.129:0/729718223 shutdown_connections 2024-09-17T04:02:57.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.240+0000 7f499b200640 1 -- 172.21.15.129:0/729718223 wait complete. 2024-09-17T04:02:57.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.242+0000 7f499b200640 1 Processor -- start 2024-09-17T04:02:57.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.242+0000 7f499b200640 1 -- start start 2024-09-17T04:02:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f499c06deb0 0x7f499c1f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:57.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 0x7f499c0cfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f499c0d2950 con 0x7f499c1624a0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f499c15fed0 con 0x7f499c06deb0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f499c0d2ad0 con 0x7f499c0b54d0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49260/0 (socket says 172.21.15.129:49260) 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 -- 172.21.15.129:0/956594208 learned_addr learned my addr 172.21.15.129:0/956594208 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f4999e00640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f499c06deb0 0x7f499c1f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f4999400640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 0x7f499c0cfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 -- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 msgr2=0x7f499c0cfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:57.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 0x7f499c0cfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 -- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f499c06deb0 msgr2=0x7f499c1f8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:57.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f499c06deb0 0x7f499c1f8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.244+0000 7f499a800640 1 -- 172.21.15.129:0/956594208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499c1f9420 con 0x7f499c1624a0 2024-09-17T04:02:57.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f499a800640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4994009cd0 tx=0x7f49940316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994031da0 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4994003a60 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994033070 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499c1f9700 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.245+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499c0b3b30 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.246+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f499c0d3060 con 0x7f499c1624a0 2024-09-17T04:02:57.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.248+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4994033210 con 0x7f499c1624a0 2024-09-17T04:02:57.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.249+0000 7f4992000640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 0x7f497c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:57.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.249+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f49940bbbc0 con 0x7f499c1624a0 2024-09-17T04:02:57.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.250+0000 7f4999e00640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 0x7f497c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:57.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.250+0000 7f4999e00640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 0x7f497c07b0a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f498c009b30 tx=0x7f498c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:57.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.254+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f499404a070 con 0x7f499c1624a0 2024-09-17T04:02:57.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.476+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f499c1f9180 con 0x7f499c1624a0 2024-09-17T04:02:57.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.477+0000 7f4992000640 1 -- 172.21.15.129:0/956594208 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f499403f030 con 0x7f499c1624a0 2024-09-17T04:02:57.476 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:57.476 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:57.476 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 msgr2=0x7f497c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 0x7f497c07b0a0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f498c009b30 tx=0x7f498c005f90 comp rx=0 tx=0).stop 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 msgr2=0x7f499c158040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f4994009cd0 tx=0x7f49940316f0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 shutdown_connections 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f497c078b90 0x7f497c07b0a0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f499c1624a0 0x7f499c158040 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f499c0b54d0 0x7f499c0cfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 --2- 172.21.15.129:0/956594208 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f499c06deb0 0x7f499c1f8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 >> 172.21.15.129:0/956594208 conn(0x7f499c070730 msgr2=0x7f499c15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.479+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 shutdown_connections 2024-09-17T04:02:57.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:57.480+0000 7f499b200640 1 -- 172.21.15.129:0/956594208 wait complete. 2024-09-17T04:02:57.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1100,"msgr_send_messages":1094,"msgr_recv_bytes":413317,"msgr_send_bytes":630088,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.193423734,"msgr_running_send_time":0.093915018,"msgr_running_recv_time":0.039131397,"msgr_running_fast_dispatch_time":0.052233612,"msgr_send_messages_queue_lat":{"avgcount":1094,"sum":2.270863128,"avgtime":0.002075743},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125920,"msgr_send_encrypted_bytes":15296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9483,"msgr_send_messages":14593,"msgr_recv_bytes":3088319,"msgr_send_bytes":11384197,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.922929730,"msgr_running_send_time":1.356739594,"msgr_running_recv_time":0.837008651,"msgr_running_fast_dispatch_time":0.096043192,"msgr_send_messages_queue_lat":{"avgcount":14591,"sum":2.747475920,"avgtime":0.000188299},"msgr_handle_ack_lat":{"avgcount":8331,"sum":0.004237871,"avgtime":0.000000508},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":78208},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2290,"msgr_send_messages":2293,"msgr_recv_bytes":737582,"msgr_send_bytes":9447329,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.464855534,"msgr_running_send_time":0.258016138,"msgr_running_recv_time":0.072236599,"msgr_running_fast_dispatch_time":0.112637355,"msgr_send_messages_queue_lat":{"avgcount":2290,"sum":2.830323559,"avgtime":0.001235949},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3088,"sum":2.974989855,"avgtime":0.000963403}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3972,"sum":0.507164021,"avgtime":0.000127684}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":318,"caps":318,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4793,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":70,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":257,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1344,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":8994,"evex":8670,"evtrm":8670,"ev":327,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21723316,"rdpos":4195244,"jlat":{"avgcount":1527,"sum":7.047674376,"avgtime":0.004615372},"replayed":3},"mds_mem":{"ino":336,"ino+":1614,"ino-":1278,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":199,"cap+":1602,"cap-":1403,"rss":150044,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1303,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2482267,"buffer_anon_items":1065,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1658746,"mds_co_items":33062,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5210,"op_send_bytes":17940525,"op_resend":0,"op_reply":5210,"op_latency":{"avgcount":5210,"sum":71.808450490,"avgtime":0.013782811},"op_inflight":0,"oplen_avg":{"avgcount":5210,"sum":6219},"op":5210,"op_r":52,"op_w":5158,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":24,"osdop_write":1543,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3104,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3085,"pq_executed":1344,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12866,"get_sum":3198600,"get_or_fail_fail":0,"get_or_fail_success":12866,"take":0,"take_sum":0,"put":12866,"put_sum":3198600,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":18077204,"put":5210,"put_sum":18077204,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":5210,"put":5210,"put_sum":5210,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1344,"get_sum":135744,"get_or_fail_fail":0,"get_or_fail_success":1344,"take":0,"take_sum":0,"put":16,"put_sum":135744,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":222286,"max":3758096384,"get_started":0,"get":8994,"get_sum":17527813,"get_or_fail_fail":0,"get_or_fail_success":8994,"take":0,"take_sum":0,"put":1527,"put_sum":17305527,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:57.745 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 413317, "msgr_recv_encrypted_bytes": 125920, "msgr_recv_messages": 1100, "msgr_running_fast_dispatch_time": 0.052233612, "msgr_running_recv_time": 0.039131397, "msgr_running_send_time": 0.093915018, "msgr_running_total_time": 0.193423734, "msgr_send_bytes": 630088, "msgr_send_encrypted_bytes": 15296, "msgr_send_messages": 1094, "msgr_send_messages_queue_lat": { "avgcount": 1094, "avgtime": 0.002075743, "sum": 2.270863128 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8331, "avgtime": 5.08e-07, "sum": 0.004237871 }, "msgr_recv_bytes": 3088319, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9483, "msgr_running_fast_dispatch_time": 0.096043192, "msgr_running_recv_time": 0.837008651, "msgr_running_send_time": 1.356739594, "msgr_running_total_time": 1.92292973, "msgr_send_bytes": 11384197, "msgr_send_encrypted_bytes": 78208, "msgr_send_messages": 14593, "msgr_send_messages_queue_lat": { "avgcount": 14591, "avgtime": 0.000188299, "sum": 2.74747592 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 737582, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2290, "msgr_running_fast_dispatch_time": 0.112637355, "msgr_running_recv_time": 0.072236599, "msgr_running_send_time": 0.258016138, "msgr_running_total_time": 0.464855534, "msgr_send_bytes": 9447329, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2293, "msgr_send_messages_queue_lat": { "avgcount": 2290, "avgtime": 0.001235949, "sum": 2.830323559 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3088, "avgtime": 0.000963403, "sum": 2.974989855 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3972, "avgtime": 0.000127684, "sum": 0.507164021 }, "queue_len": 0 }, "mds": { "caps": 318, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4793, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 70, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 318, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 257, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1344, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 327, "evadd": 8994, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1527, "avgtime": 0.004615372, "sum": 7.047674376 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21723316 }, "mds_mem": { "cap": 199, "cap+": 1602, "cap-": 1403, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 336, "ino+": 1614, "ino-": 1278, "rss": 150044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1303, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2482267, "buffer_anon_items": 1065, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1658746, "mds_co_items": 33062, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5210, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5210, "avgtime": 0.013782811, "sum": 71.80845049 }, "op_pg": 0, "op_r": 52, "op_reply": 5210, "op_resend": 0, "op_rmw": 0, "op_send": 5210, "op_send_bytes": 17940525, "op_w": 5158, "oplen_avg": { "avgcount": 5210, "sum": 6219 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3104, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1543, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1344, "pq_executed_ops": 3085, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12866, "get_or_fail_fail": 0, "get_or_fail_success": 12866, "get_started": 0, "get_sum": 3198600, "max": 104857600, "put": 12866, "put_sum": 3198600, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5210, "put_sum": 18077204, "take": 5210, "take_sum": 18077204, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5210, "put_sum": 5210, "take": 5210, "take_sum": 5210, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1344, "get_or_fail_fail": 0, "get_or_fail_success": 1344, "get_started": 0, "get_sum": 135744, "max": 3758096384, "put": 16, "put_sum": 135744, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 8994, "get_or_fail_fail": 0, "get_or_fail_success": 8994, "get_started": 0, "get_sum": 17527813, "max": 3758096384, "put": 1527, "put_sum": 17305527, "take": 0, "take_sum": 0, "val": 222286, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:57.745 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/257, 1344/1601 2024-09-17T04:02:58.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:02:58.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 -- 172.21.15.129:0/684735850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cfd80 msgr2=0x7f3ec8072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:58.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 --2- 172.21.15.129:0/684735850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cfd80 0x7f3ec8072d70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4005fd0 tx=0x7f3eb402a540 comp rx=0 tx=0).stop 2024-09-17T04:02:58.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 -- 172.21.15.129:0/684735850 shutdown_connections 2024-09-17T04:02:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 --2- 172.21.15.129:0/684735850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cfd80 0x7f3ec8072d70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 --2- 172.21.15.129:0/684735850 >> v2:172.21.15.181:3300/0 conn(0x7f3ec80cd190 0x7f3ec815cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 -- 172.21.15.129:0/684735850 >> 172.21.15.129:0/684735850 conn(0x7f3ec80b23c0 msgr2=0x7f3ec80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 -- 172.21.15.129:0/684735850 shutdown_connections 2024-09-17T04:02:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.978+0000 7f3ecf800640 1 -- 172.21.15.129:0/684735850 wait complete. 2024-09-17T04:02:58.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.980+0000 7f3ecf800640 1 Processor -- start 2024-09-17T04:02:58.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.980+0000 7f3ecf800640 1 -- start start 2024-09-17T04:02:58.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ec80b59d0 0x7f3ec8143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cd190 0x7f3ec813e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8070060 con 0x7f3ec80cfd80 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec806fee0 con 0x7f3ec80b59d0 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecf800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec80701e0 con 0x7f3ec80cd190 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecd800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecd800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49266/0 (socket says 172.21.15.129:49266) 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecd800640 1 -- 172.21.15.129:0/1578122942 learned_addr learned my addr 172.21.15.129:0/1578122942 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ecce00640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ec80b59d0 0x7f3ec8143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.982+0000 7f3ec7e00640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cd190 0x7f3ec813e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 -- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cd190 msgr2=0x7f3ec813e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:58.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cd190 0x7f3ec813e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:58.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 -- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ec80b59d0 msgr2=0x7f3ec8143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:58.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ec80b59d0 0x7f3ec8143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:58.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 -- 172.21.15.129:0/1578122942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec813e5a0 con 0x7f3ec80cfd80 2024-09-17T04:02:58.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecd800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c730 tx=0x7f3ec000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0003b70 con 0x7f3ec80cfd80 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec8140560 con 0x7f3ec80cfd80 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ec0003d10 con 0x7f3ec80cfd80 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec000aab0 con 0x7f3ec80cfd80 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.983+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec813f260 con 0x7f3ec80cfd80 2024-09-17T04:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.984+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec813ecf0 con 0x7f3ec80cfd80 2024-09-17T04:02:58.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.986+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3ec000ac50 con 0x7f3ec80cfd80 2024-09-17T04:02:58.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.987+0000 7f3ec4c00640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 0x7f3ea807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:02:58.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.987+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f3ec009b730 con 0x7f3ec80cfd80 2024-09-17T04:02:58.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.988+0000 7f3ecce00640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 0x7f3ea807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:02:58.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.989+0000 7f3ecce00640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 0x7f3ea807b0a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0095f0 tx=0x7f3ebc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:02:58.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:58.991+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3ec813e5a0 con 0x7f3ec80cfd80 2024-09-17T04:02:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.210+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ec8074230 con 0x7f3ec80cfd80 2024-09-17T04:02:59.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.211+0000 7f3ec4c00640 1 -- 172.21.15.129:0/1578122942 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f3ec0063390 con 0x7f3ec80cfd80 2024-09-17T04:02:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:02:59.212 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:02:59.212 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 msgr2=0x7f3ea807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 0x7f3ea807b0a0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f3ebc0095f0 tx=0x7f3ebc005f90 comp rx=0 tx=0).stop 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 msgr2=0x7f3ec8140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000c730 tx=0x7f3ec000cbf0 comp rx=0 tx=0).stop 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 shutdown_connections 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ea8078b90 0x7f3ea807b0a0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec80cfd80 0x7f3ec8140020 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ec80cd190 0x7f3ec813e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 --2- 172.21.15.129:0/1578122942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ec80b59d0 0x7f3ec8143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:02:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 >> 172.21.15.129:0/1578122942 conn(0x7f3ec80b23c0 msgr2=0x7f3ec8145080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:02:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 shutdown_connections 2024-09-17T04:02:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:02:59.214+0000 7f3ecf800640 1 -- 172.21.15.129:0/1578122942 wait complete. 2024-09-17T04:02:59.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:02:59.478 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1100,"msgr_send_messages":1094,"msgr_recv_bytes":413317,"msgr_send_bytes":630088,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.193423734,"msgr_running_send_time":0.093915018,"msgr_running_recv_time":0.039131397,"msgr_running_fast_dispatch_time":0.052233612,"msgr_send_messages_queue_lat":{"avgcount":1094,"sum":2.270863128,"avgtime":0.002075743},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125920,"msgr_send_encrypted_bytes":15296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9485,"msgr_send_messages":14593,"msgr_recv_bytes":3089123,"msgr_send_bytes":11384197,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.923256403,"msgr_running_send_time":1.356864156,"msgr_running_recv_time":0.837088868,"msgr_running_fast_dispatch_time":0.096158077,"msgr_send_messages_queue_lat":{"avgcount":14591,"sum":2.747475920,"avgtime":0.000188299},"msgr_handle_ack_lat":{"avgcount":8331,"sum":0.004237871,"avgtime":0.000000508},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":78208},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2290,"msgr_send_messages":2293,"msgr_recv_bytes":737582,"msgr_send_bytes":9447329,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.464855534,"msgr_running_send_time":0.258016138,"msgr_running_recv_time":0.072236599,"msgr_running_fast_dispatch_time":0.112637355,"msgr_send_messages_queue_lat":{"avgcount":2290,"sum":2.830323559,"avgtime":0.001235949},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3088,"sum":2.974989855,"avgtime":0.000963403}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":3972,"sum":0.507164021,"avgtime":0.000127684}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":318,"caps":318,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4793,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":70,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":257,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1344,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":8994,"evex":8670,"evtrm":8670,"ev":327,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21723316,"rdpos":4195244,"jlat":{"avgcount":1527,"sum":7.047674376,"avgtime":0.004615372},"replayed":3},"mds_mem":{"ino":336,"ino+":1614,"ino-":1278,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":199,"cap+":1602,"cap-":1403,"rss":150044,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1303,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":2482267,"buffer_anon_items":1065,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1658746,"mds_co_items":33062,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5210,"op_send_bytes":17940525,"op_resend":0,"op_reply":5210,"op_latency":{"avgcount":5210,"sum":71.808450490,"avgtime":0.013782811},"op_inflight":0,"oplen_avg":{"avgcount":5210,"sum":6219},"op":5210,"op_r":52,"op_w":5158,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":24,"osdop_write":1543,"osdop_writefull":16,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3104,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3085,"pq_executed":1344,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":12868,"get_sum":3199256,"get_or_fail_fail":0,"get_or_fail_success":12868,"take":0,"take_sum":0,"put":12868,"put_sum":3199256,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":18077204,"put":5210,"put_sum":18077204,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5210,"take_sum":5210,"put":5210,"put_sum":5210,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1344,"get_sum":135744,"get_or_fail_fail":0,"get_or_fail_success":1344,"take":0,"take_sum":0,"put":16,"put_sum":135744,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":222286,"max":3758096384,"get_started":0,"get":8994,"get_sum":17527813,"get_or_fail_fail":0,"get_or_fail_success":8994,"take":0,"take_sum":0,"put":1527,"put_sum":17305527,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:02:59.480 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 413317, "msgr_recv_encrypted_bytes": 125920, "msgr_recv_messages": 1100, "msgr_running_fast_dispatch_time": 0.052233612, "msgr_running_recv_time": 0.039131397, "msgr_running_send_time": 0.093915018, "msgr_running_total_time": 0.193423734, "msgr_send_bytes": 630088, "msgr_send_encrypted_bytes": 15296, "msgr_send_messages": 1094, "msgr_send_messages_queue_lat": { "avgcount": 1094, "avgtime": 0.002075743, "sum": 2.270863128 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8331, "avgtime": 5.08e-07, "sum": 0.004237871 }, "msgr_recv_bytes": 3089123, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9485, "msgr_running_fast_dispatch_time": 0.096158077, "msgr_running_recv_time": 0.837088868, "msgr_running_send_time": 1.356864156, "msgr_running_total_time": 1.923256403, "msgr_send_bytes": 11384197, "msgr_send_encrypted_bytes": 78208, "msgr_send_messages": 14593, "msgr_send_messages_queue_lat": { "avgcount": 14591, "avgtime": 0.000188299, "sum": 2.74747592 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 737582, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2290, "msgr_running_fast_dispatch_time": 0.112637355, "msgr_running_recv_time": 0.072236599, "msgr_running_send_time": 0.258016138, "msgr_running_total_time": 0.464855534, "msgr_send_bytes": 9447329, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2293, "msgr_send_messages_queue_lat": { "avgcount": 2290, "avgtime": 0.001235949, "sum": 2.830323559 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3088, "avgtime": 0.000963403, "sum": 2.974989855 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 3972, "avgtime": 0.000127684, "sum": 0.507164021 }, "queue_len": 0 }, "mds": { "caps": 318, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4793, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 70, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 318, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 257, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1344, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 327, "evadd": 8994, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1527, "avgtime": 0.004615372, "sum": 7.047674376 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21723316 }, "mds_mem": { "cap": 199, "cap+": 1602, "cap-": 1403, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 336, "ino+": 1614, "ino-": 1278, "rss": 150044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1303, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 2482267, "buffer_anon_items": 1065, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1658746, "mds_co_items": 33062, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5210, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5210, "avgtime": 0.013782811, "sum": 71.80845049 }, "op_pg": 0, "op_r": 52, "op_reply": 5210, "op_resend": 0, "op_rmw": 0, "op_send": 5210, "op_send_bytes": 17940525, "op_w": 5158, "oplen_avg": { "avgcount": 5210, "sum": 6219 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3104, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 24, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1543, "osdop_writefull": 16, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1344, "pq_executed_ops": 3085, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 12868, "get_or_fail_fail": 0, "get_or_fail_success": 12868, "get_started": 0, "get_sum": 3199256, "max": 104857600, "put": 12868, "put_sum": 3199256, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5210, "put_sum": 18077204, "take": 5210, "take_sum": 18077204, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5210, "put_sum": 5210, "take": 5210, "take_sum": 5210, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1344, "get_or_fail_fail": 0, "get_or_fail_success": 1344, "get_started": 0, "get_sum": 135744, "max": 3758096384, "put": 16, "put_sum": 135744, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 8994, "get_or_fail_fail": 0, "get_or_fail_success": 8994, "get_started": 0, "get_sum": 17527813, "max": 3758096384, "put": 1527, "put_sum": 17305527, "take": 0, "take_sum": 0, "val": 222286, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:02:59.480 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/257, 1344/1601 2024-09-17T04:03:00.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:00.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 -- 172.21.15.129:0/3355646144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0b59d0 msgr2=0x7f25bc0b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 --2- 172.21.15.129:0/3355646144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0b59d0 0x7f25bc0b5de0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0d0490 tx=0x7f25b402a980 comp rx=0 tx=0).stop 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 -- 172.21.15.129:0/3355646144 shutdown_connections 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 --2- 172.21.15.129:0/3355646144 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0b59d0 0x7f25bc0b5de0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 --2- 172.21.15.129:0/3355646144 >> v2:172.21.15.181:3300/0 conn(0x7f25bc15cf70 0x7f25bc072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 -- 172.21.15.129:0/3355646144 >> 172.21.15.129:0/3355646144 conn(0x7f25bc0b23c0 msgr2=0x7f25bc0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 -- 172.21.15.129:0/3355646144 shutdown_connections 2024-09-17T04:03:00.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.721+0000 7f25c2200640 1 -- 172.21.15.129:0/3355646144 wait complete. 2024-09-17T04:03:00.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.723+0000 7f25c2200640 1 Processor -- start 2024-09-17T04:03:00.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.724+0000 7f25c2200640 1 -- start start 2024-09-17T04:03:00.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25bc0b59d0 0x7f25bc1433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25bc15cf70 0x7f25bc15e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc06fe50 con 0x7f25bc0cd190 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25bc06fcd0 con 0x7f25bc0b59d0 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25c2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f25bc06ffd0 con 0x7f25bc15cf70 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25baa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25baa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49284/0 (socket says 172.21.15.129:49284) 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.725+0000 7f25baa00640 1 -- 172.21.15.129:0/3216892540 learned_addr learned my addr 172.21.15.129:0/3216892540 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25bb400640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25bc0b59d0 0x7f25bc1433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25bbe00640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25bc15cf70 0x7f25bc15e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 -- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25bc15cf70 msgr2=0x7f25bc15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:00.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25bc15cf70 0x7f25bc15e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 -- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25bc0b59d0 msgr2=0x7f25bc1433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25bc0b59d0 0x7f25bc1433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 -- 172.21.15.129:0/3216892540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25bc143920 con 0x7f25bc0cd190 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25baa00640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f25a800d6d0 tx=0x7f25a800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8017070 con 0x7f25bc0cd190 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25a8005330 con 0x7f25bc0cd190 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8009820 con 0x7f25bc0cd190 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.726+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25bc0b56b0 con 0x7f25bc0cd190 2024-09-17T04:03:00.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.727+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25bc0c9dc0 con 0x7f25bc0cd190 2024-09-17T04:03:00.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.728+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25bc0b5420 con 0x7f25bc0cd190 2024-09-17T04:03:00.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.729+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f25a8009bd0 con 0x7f25bc0cd190 2024-09-17T04:03:00.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.730+0000 7f25b3400640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 0x7f259c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:00.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.730+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f25a809b0f0 con 0x7f25bc0cd190 2024-09-17T04:03:00.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.730+0000 7f25bb400640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 0x7f259c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:00.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.731+0000 7f25bb400640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 0x7f259c07b0a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f25ac006fd0 tx=0x7f25ac009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:00.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.734+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f25a8062de0 con 0x7f25bc0cd190 2024-09-17T04:03:00.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.952+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25bc070500 con 0x7f25bc0cd190 2024-09-17T04:03:00.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.953+0000 7f25b3400640 1 -- 172.21.15.129:0/3216892540 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f25a8067810 con 0x7f25bc0cd190 2024-09-17T04:03:00.953 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:00.954 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:00.954 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 msgr2=0x7f259c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 0x7f259c07b0a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f25ac006fd0 tx=0x7f25ac009060 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 msgr2=0x7f25bc15e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f25a800d6d0 tx=0x7f25a800db90 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 shutdown_connections 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f259c078b90 0x7f259c07b0a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25bc15cf70 0x7f25bc15e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25bc0cd190 0x7f25bc15e110 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 --2- 172.21.15.129:0/3216892540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25bc0b59d0 0x7f25bc1433e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:00.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.955+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 >> 172.21.15.129:0/3216892540 conn(0x7f25bc0b23c0 msgr2=0x7f25bc144e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:00.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.956+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 shutdown_connections 2024-09-17T04:03:00.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:00.956+0000 7f25c2200640 1 -- 172.21.15.129:0/3216892540 wait complete. 2024-09-17T04:03:00.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:01.216 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1156,"msgr_send_messages":1150,"msgr_recv_bytes":427541,"msgr_send_bytes":648344,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198194485,"msgr_running_send_time":0.095976603,"msgr_running_recv_time":0.040286726,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1150,"sum":2.287034129,"avgtime":0.001988725},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":125920,"msgr_send_encrypted_bytes":15296},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9567,"msgr_send_messages":14680,"msgr_recv_bytes":3107303,"msgr_send_bytes":11636895,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.932088457,"msgr_running_send_time":1.361456879,"msgr_running_recv_time":0.843052326,"msgr_running_fast_dispatch_time":0.098057151,"msgr_send_messages_queue_lat":{"avgcount":14678,"sum":2.753026462,"avgtime":0.000187561},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":80048},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3090,"sum":2.992131825,"avgtime":0.000968327}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":199,"caps":199,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9085,"evex":8670,"evtrm":8670,"ev":418,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21777825,"rdpos":4195244,"jlat":{"avgcount":1528,"sum":7.052642635,"avgtime":0.004615603},"replayed":3},"mds_mem":{"ino":270,"ino+":1614,"ino-":1344,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":150044,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1308,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1889502,"buffer_anon_items":821,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1427528,"mds_co_items":27234,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5441,"op_send_bytes":18172182,"op_resend":0,"op_reply":5441,"op_latency":{"avgcount":5441,"sum":73.288418554,"avgtime":0.013469659},"op_inflight":0,"oplen_avg":{"avgcount":5441,"sum":6450},"op":5441,"op_r":56,"op_w":5385,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1548,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13127,"get_sum":3248643,"get_or_fail_fail":0,"get_or_fail_success":13127,"take":0,"take_sum":0,"put":13127,"put_sum":3248643,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5441,"take_sum":18318052,"put":5441,"put_sum":18318052,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5441,"take_sum":5441,"put":5441,"put_sum":5441,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":54509,"max":3758096384,"get_started":0,"get":9085,"get_sum":17582322,"get_or_fail_fail":0,"get_or_fail_success":9085,"take":0,"take_sum":0,"put":1528,"put_sum":17527813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:01.218 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 427541, "msgr_recv_encrypted_bytes": 125920, "msgr_recv_messages": 1156, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040286726, "msgr_running_send_time": 0.095976603, "msgr_running_total_time": 0.198194485, "msgr_send_bytes": 648344, "msgr_send_encrypted_bytes": 15296, "msgr_send_messages": 1150, "msgr_send_messages_queue_lat": { "avgcount": 1150, "avgtime": 0.001988725, "sum": 2.287034129 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3107303, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9567, "msgr_running_fast_dispatch_time": 0.098057151, "msgr_running_recv_time": 0.843052326, "msgr_running_send_time": 1.361456879, "msgr_running_total_time": 1.932088457, "msgr_send_bytes": 11636895, "msgr_send_encrypted_bytes": 80048, "msgr_send_messages": 14680, "msgr_send_messages_queue_lat": { "avgcount": 14678, "avgtime": 0.000187561, "sum": 2.753026462 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3090, "avgtime": 0.000968327, "sum": 2.992131825 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 199, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 199, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 418, "evadd": 9085, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1528, "avgtime": 0.004615603, "sum": 7.052642635 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21777825 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 270, "ino+": 1614, "ino-": 1344, "rss": 150044 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1308, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1889502, "buffer_anon_items": 821, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1427528, "mds_co_items": 27234, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5441, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5441, "avgtime": 0.013469659, "sum": 73.288418554 }, "op_pg": 0, "op_r": 56, "op_reply": 5441, "op_resend": 0, "op_rmw": 0, "op_send": 5441, "op_send_bytes": 18172182, "op_w": 5385, "oplen_avg": { "avgcount": 5441, "sum": 6450 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1548, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13127, "get_or_fail_fail": 0, "get_or_fail_success": 13127, "get_started": 0, "get_sum": 3248643, "max": 104857600, "put": 13127, "put_sum": 3248643, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5441, "put_sum": 18318052, "take": 5441, "take_sum": 18318052, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5441, "put_sum": 5441, "take": 5441, "take_sum": 5441, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9085, "get_or_fail_fail": 0, "get_or_fail_success": 9085, "get_started": 0, "get_sum": 17582322, "max": 3758096384, "put": 1528, "put_sum": 17527813, "take": 0, "take_sum": 0, "val": 54509, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:01.218 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:02.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 -- 172.21.15.129:0/2809396636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f815da40 msgr2=0x7fb9f806e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 --2- 172.21.15.129:0/2809396636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f815da40 0x7fb9f806e310 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009830 tx=0x7fb9ec02a520 comp rx=0 tx=0).stop 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 -- 172.21.15.129:0/2809396636 shutdown_connections 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 --2- 172.21.15.129:0/2809396636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f8141f60 0x7fb9f8142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 --2- 172.21.15.129:0/2809396636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f815da40 0x7fb9f806e310 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 -- 172.21.15.129:0/2809396636 >> 172.21.15.129:0/2809396636 conn(0x7fb9f806d560 msgr2=0x7fb9f80b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 -- 172.21.15.129:0/2809396636 shutdown_connections 2024-09-17T04:03:02.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.443+0000 7fb9fea00640 1 -- 172.21.15.129:0/2809396636 wait complete. 2024-09-17T04:03:02.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.446+0000 7fb9fea00640 1 Processor -- start 2024-09-17T04:03:02.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.446+0000 7fb9fea00640 1 -- start start 2024-09-17T04:03:02.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 0x7fb9f81f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f815da40 0x7fb9f81f6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8074e90 con 0x7fb9f8141f60 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8074d10 con 0x7fb9f8156f90 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fea00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f8075010 con 0x7fb9f815da40 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49288/0 (socket says 172.21.15.129:49288) 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.447+0000 7fb9fd600640 1 -- 172.21.15.129:0/1733474229 learned_addr learned my addr 172.21.15.129:0/1733474229 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fe000640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f815da40 0x7fb9f81f6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fcc00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 0x7fb9f81f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 -- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f815da40 msgr2=0x7fb9f81f6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f815da40 0x7fb9f81f6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 -- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 msgr2=0x7fb9f81f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 0x7fb9f81f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 -- 172.21.15.129:0/1733474229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f81573a0 con 0x7fb9f8141f60 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fcc00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 0x7fb9f81f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9fd600640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000c00 tx=0x7fb9ec033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec031d60 con 0x7fb9f8141f60 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9ec033e10 con 0x7fb9f8141f60 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.448+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec033310 con 0x7fb9f8141f60 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.449+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f8157680 con 0x7fb9f8141f60 2024-09-17T04:03:02.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.449+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f8157c40 con 0x7fb9f8141f60 2024-09-17T04:03:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.450+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9f8075530 con 0x7fb9f8141f60 2024-09-17T04:03:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.451+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb9ec0334b0 con 0x7fb9f8141f60 2024-09-17T04:03:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.452+0000 7fb9f5600640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 0x7fb9d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.452+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7fb9ec0bbea0 con 0x7fb9f8141f60 2024-09-17T04:03:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.452+0000 7fb9fcc00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 0x7fb9d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.453+0000 7fb9fcc00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 0x7fb9d807b0d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8072560 tx=0x7fb9e4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:02.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.456+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb9ec04a030 con 0x7fb9f8141f60 2024-09-17T04:03:02.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.676+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9f8072590 con 0x7fb9f8141f60 2024-09-17T04:03:02.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.676+0000 7fb9f5600640 1 -- 172.21.15.129:0/1733474229 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7fb9ec03f030 con 0x7fb9f8141f60 2024-09-17T04:03:02.677 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:02.677 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:02.677 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:02.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 msgr2=0x7fb9d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:02.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 0x7fb9d807b0d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb9f8072560 tx=0x7fb9e4009040 comp rx=0 tx=0).stop 2024-09-17T04:03:02.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 msgr2=0x7fb9f8158b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:02.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec000c00 tx=0x7fb9ec033c20 comp rx=0 tx=0).stop 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 shutdown_connections 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb9d8078bc0 0x7fb9d807b0d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9f815da40 0x7fb9f81f6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9f8156f90 0x7fb9f81f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 --2- 172.21.15.129:0/1733474229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb9f8141f60 0x7fb9f8158b70 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 >> 172.21.15.129:0/1733474229 conn(0x7fb9f806d560 msgr2=0x7fb9f80ad9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 shutdown_connections 2024-09-17T04:03:02.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:02.679+0000 7fb9fea00640 1 -- 172.21.15.129:0/1733474229 wait complete. 2024-09-17T04:03:02.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:02.952 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1157,"msgr_send_messages":1151,"msgr_recv_bytes":427813,"msgr_send_bytes":648984,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198343083,"msgr_running_send_time":0.096072379,"msgr_running_recv_time":0.040333571,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1151,"sum":2.287144182,"avgtime":0.001987093},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126192,"msgr_send_encrypted_bytes":15936},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9569,"msgr_send_messages":14680,"msgr_recv_bytes":3108107,"msgr_send_bytes":11636895,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.932446577,"msgr_running_send_time":1.361621240,"msgr_running_recv_time":0.843143061,"msgr_running_fast_dispatch_time":0.098150327,"msgr_send_messages_queue_lat":{"avgcount":14678,"sum":2.753026462,"avgtime":0.000187561},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":80048},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3090,"sum":2.992131825,"avgtime":0.000968327}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":199,"caps":199,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9085,"evex":8670,"evtrm":8670,"ev":418,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21777825,"rdpos":4195244,"jlat":{"avgcount":1528,"sum":7.052642635,"avgtime":0.004615603},"replayed":3},"mds_mem":{"ino":270,"ino+":1614,"ino-":1344,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141812,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1308,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1889502,"buffer_anon_items":821,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1427528,"mds_co_items":27234,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5441,"op_send_bytes":18172182,"op_resend":0,"op_reply":5441,"op_latency":{"avgcount":5441,"sum":73.288418554,"avgtime":0.013469659},"op_inflight":0,"oplen_avg":{"avgcount":5441,"sum":6450},"op":5441,"op_r":56,"op_w":5385,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1548,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13130,"get_sum":3249429,"get_or_fail_fail":0,"get_or_fail_success":13130,"take":0,"take_sum":0,"put":13130,"put_sum":3249429,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5441,"take_sum":18318052,"put":5441,"put_sum":18318052,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5441,"take_sum":5441,"put":5441,"put_sum":5441,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":54509,"max":3758096384,"get_started":0,"get":9085,"get_sum":17582322,"get_or_fail_fail":0,"get_or_fail_success":9085,"take":0,"take_sum":0,"put":1528,"put_sum":17527813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:02.953 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 427813, "msgr_recv_encrypted_bytes": 126192, "msgr_recv_messages": 1157, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040333571, "msgr_running_send_time": 0.096072379, "msgr_running_total_time": 0.198343083, "msgr_send_bytes": 648984, "msgr_send_encrypted_bytes": 15936, "msgr_send_messages": 1151, "msgr_send_messages_queue_lat": { "avgcount": 1151, "avgtime": 0.001987093, "sum": 2.287144182 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3108107, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9569, "msgr_running_fast_dispatch_time": 0.098150327, "msgr_running_recv_time": 0.843143061, "msgr_running_send_time": 1.36162124, "msgr_running_total_time": 1.932446577, "msgr_send_bytes": 11636895, "msgr_send_encrypted_bytes": 80048, "msgr_send_messages": 14680, "msgr_send_messages_queue_lat": { "avgcount": 14678, "avgtime": 0.000187561, "sum": 2.753026462 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3090, "avgtime": 0.000968327, "sum": 2.992131825 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 199, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 199, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 418, "evadd": 9085, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1528, "avgtime": 0.004615603, "sum": 7.052642635 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21777825 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 270, "ino+": 1614, "ino-": 1344, "rss": 141812 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1308, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1889502, "buffer_anon_items": 821, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1427528, "mds_co_items": 27234, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5441, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5441, "avgtime": 0.013469659, "sum": 73.288418554 }, "op_pg": 0, "op_r": 56, "op_reply": 5441, "op_resend": 0, "op_rmw": 0, "op_send": 5441, "op_send_bytes": 18172182, "op_w": 5385, "oplen_avg": { "avgcount": 5441, "sum": 6450 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1548, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13130, "get_or_fail_fail": 0, "get_or_fail_success": 13130, "get_started": 0, "get_sum": 3249429, "max": 104857600, "put": 13130, "put_sum": 3249429, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5441, "put_sum": 18318052, "take": 5441, "take_sum": 18318052, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5441, "put_sum": 5441, "take": 5441, "take_sum": 5441, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9085, "get_or_fail_fail": 0, "get_or_fail_success": 9085, "get_started": 0, "get_sum": 17582322, "max": 3758096384, "put": 1528, "put_sum": 17527813, "take": 0, "take_sum": 0, "val": 54509, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:02.953 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:03.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:04.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 -- 172.21.15.129:0/1726385533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 msgr2=0x7ff4a806f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 --2- 172.21.15.129:0/1726385533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a806f710 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0005fd0 tx=0x7ff4a002aaa0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 -- 172.21.15.129:0/1726385533 shutdown_connections 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 --2- 172.21.15.129:0/1726385533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a806f710 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 -- 172.21.15.129:0/1726385533 >> 172.21.15.129:0/1726385533 conn(0x7ff4a80b23c0 msgr2=0x7ff4a80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 -- 172.21.15.129:0/1726385533 shutdown_connections 2024-09-17T04:03:04.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.178+0000 7ff4ad200640 1 -- 172.21.15.129:0/1726385533 wait complete. 2024-09-17T04:03:04.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.180+0000 7ff4ad200640 1 Processor -- start 2024-09-17T04:03:04.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.181+0000 7ff4ad200640 1 -- start start 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 0x7ff4a8155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 0x7ff4a8156db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80b6100 con 0x7ff4a80aeca0 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a80b5f80 con 0x7ff4a815f110 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4ad200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a80b6280 con 0x7ff4a80ae890 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:04.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34016/0 (socket says 172.21.15.129:34016) 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4a5400640 1 -- 172.21.15.129:0/3400863827 learned_addr learned my addr 172.21.15.129:0/3400863827 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4a5e00640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 0x7ff4a8155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.182+0000 7ff4a6800640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 0x7ff4a8156db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 -- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 msgr2=0x7ff4a8155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 0x7ff4a8155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 -- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 msgr2=0x7ff4a8156db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 0x7ff4a8156db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 -- 172.21.15.129:0/3400863827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a815e000 con 0x7ff4a80aeca0 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5e00640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 0x7ff4a8155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a6800640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 0x7ff4a8156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:04.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4a5400640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff49400b3c0 tx=0x7ff49400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:04.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494011830 con 0x7ff4a80aeca0 2024-09-17T04:03:04.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff494016070 con 0x7ff4a80aeca0 2024-09-17T04:03:04.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a81562d0 con 0x7ff4a80aeca0 2024-09-17T04:03:04.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494011da0 con 0x7ff4a80aeca0 2024-09-17T04:03:04.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.183+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a8156890 con 0x7ff4a80aeca0 2024-09-17T04:03:04.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.184+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a815dd70 con 0x7ff4a80aeca0 2024-09-17T04:03:04.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.186+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff49401b460 con 0x7ff4a80aeca0 2024-09-17T04:03:04.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.187+0000 7ff49e000640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 0x7ff48807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:04.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.187+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7ff49409a930 con 0x7ff4a80aeca0 2024-09-17T04:03:04.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.188+0000 7ff4a5e00640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 0x7ff48807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:04.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.188+0000 7ff4a5e00640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 0x7ff48807b0a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff498007fd0 tx=0x7ff498009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:04.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.192+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff494020020 con 0x7ff4a80aeca0 2024-09-17T04:03:04.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.409+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4a80b6740 con 0x7ff4a80aeca0 2024-09-17T04:03:04.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.410+0000 7ff49e000640 1 -- 172.21.15.129:0/3400863827 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7ff494062d00 con 0x7ff4a80aeca0 2024-09-17T04:03:04.410 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:04.410 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:04.411 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:04.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 msgr2=0x7ff48807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:04.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 0x7ff48807b0a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff498007fd0 tx=0x7ff498009040 comp rx=0 tx=0).stop 2024-09-17T04:03:04.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 msgr2=0x7ff4a8155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:04.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7ff49400b3c0 tx=0x7ff49400b880 comp rx=0 tx=0).stop 2024-09-17T04:03:04.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 shutdown_connections 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff488078b90 0x7ff48807b0a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4a815f110 0x7ff4a8156db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4a80aeca0 0x7ff4a8155b60 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 --2- 172.21.15.129:0/3400863827 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4a80ae890 0x7ff4a8155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.412+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 >> 172.21.15.129:0/3400863827 conn(0x7ff4a80b23c0 msgr2=0x7ff4a80735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:04.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.413+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 shutdown_connections 2024-09-17T04:03:04.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:04.413+0000 7ff4ad200640 1 -- 172.21.15.129:0/3400863827 wait complete. 2024-09-17T04:03:04.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:04.678 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1157,"msgr_send_messages":1151,"msgr_recv_bytes":427813,"msgr_send_bytes":648984,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198343083,"msgr_running_send_time":0.096072379,"msgr_running_recv_time":0.040333571,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1151,"sum":2.287144182,"avgtime":0.001987093},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126192,"msgr_send_encrypted_bytes":15936},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9572,"msgr_send_messages":14682,"msgr_recv_bytes":3109001,"msgr_send_bytes":11693178,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.932932186,"msgr_running_send_time":1.361891767,"msgr_running_recv_time":0.843225071,"msgr_running_fast_dispatch_time":0.098266762,"msgr_send_messages_queue_lat":{"avgcount":14680,"sum":2.753165836,"avgtime":0.000187545},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":80048},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3093,"sum":2.996487716,"avgtime":0.000968796}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9086,"evex":8670,"evtrm":8670,"ev":419,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21778963,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141812,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1313,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":985260,"buffer_anon_items":530,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1064758,"mds_co_items":19706,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5443,"op_send_bytes":18227829,"op_resend":0,"op_reply":5443,"op_latency":{"avgcount":5443,"sum":73.295418403,"avgtime":0.013465996},"op_inflight":0,"oplen_avg":{"avgcount":5443,"sum":6452},"op":5443,"op_r":56,"op_w":5387,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13133,"get_sum":3250069,"get_or_fail_fail":0,"get_or_fail_success":13133,"take":0,"take_sum":0,"put":13133,"put_sum":3250069,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":18373699,"put":5443,"put_sum":18373699,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":5443,"put":5443,"put_sum":5443,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":0,"max":3758096384,"get_started":0,"get":9086,"get_sum":17583460,"get_or_fail_fail":0,"get_or_fail_success":9086,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:04.680 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 427813, "msgr_recv_encrypted_bytes": 126192, "msgr_recv_messages": 1157, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040333571, "msgr_running_send_time": 0.096072379, "msgr_running_total_time": 0.198343083, "msgr_send_bytes": 648984, "msgr_send_encrypted_bytes": 15936, "msgr_send_messages": 1151, "msgr_send_messages_queue_lat": { "avgcount": 1151, "avgtime": 0.001987093, "sum": 2.287144182 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3109001, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9572, "msgr_running_fast_dispatch_time": 0.098266762, "msgr_running_recv_time": 0.843225071, "msgr_running_send_time": 1.361891767, "msgr_running_total_time": 1.932932186, "msgr_send_bytes": 11693178, "msgr_send_encrypted_bytes": 80048, "msgr_send_messages": 14682, "msgr_send_messages_queue_lat": { "avgcount": 14680, "avgtime": 0.000187545, "sum": 2.753165836 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3093, "avgtime": 0.000968796, "sum": 2.996487716 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 419, "evadd": 9086, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21778963 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141812 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1313, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 985260, "buffer_anon_items": 530, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1064758, "mds_co_items": 19706, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5443, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5443, "avgtime": 0.013465996, "sum": 73.295418403 }, "op_pg": 0, "op_r": 56, "op_reply": 5443, "op_resend": 0, "op_rmw": 0, "op_send": 5443, "op_send_bytes": 18227829, "op_w": 5387, "oplen_avg": { "avgcount": 5443, "sum": 6452 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13133, "get_or_fail_fail": 0, "get_or_fail_success": 13133, "get_started": 0, "get_sum": 3250069, "max": 104857600, "put": 13133, "put_sum": 3250069, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5443, "put_sum": 18373699, "take": 5443, "take_sum": 18373699, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5443, "put_sum": 5443, "take": 5443, "take_sum": 5443, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9086, "get_or_fail_fail": 0, "get_or_fail_success": 9086, "get_started": 0, "get_sum": 17583460, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:04.681 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:05.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 -- 172.21.15.129:0/2532006651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cfd80 msgr2=0x7f548c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 --2- 172.21.15.129:0/2532006651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cfd80 0x7f548c072d70 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f548c072150 tx=0x7f547c02a4d0 comp rx=0 tx=0).stop 2024-09-17T04:03:05.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 -- 172.21.15.129:0/2532006651 shutdown_connections 2024-09-17T04:03:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 --2- 172.21.15.129:0/2532006651 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cfd80 0x7f548c072d70 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 --2- 172.21.15.129:0/2532006651 >> v2:172.21.15.181:3300/0 conn(0x7f548c0cd190 0x7f548c15cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 -- 172.21.15.129:0/2532006651 >> 172.21.15.129:0/2532006651 conn(0x7f548c0b23c0 msgr2=0x7f548c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 -- 172.21.15.129:0/2532006651 shutdown_connections 2024-09-17T04:03:05.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.914+0000 7f5494800640 1 -- 172.21.15.129:0/2532006651 wait complete. 2024-09-17T04:03:05.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.917+0000 7f5494800640 1 Processor -- start 2024-09-17T04:03:05.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.917+0000 7f5494800640 1 -- start start 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548c0b59d0 0x7f548c141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f548c0cfd80 0x7f548c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f548c06fe50 con 0x7f548c0cd190 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f548c06fcd0 con 0x7f548c0b59d0 2024-09-17T04:03:05.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.918+0000 7f5494800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f548c06ffd0 con 0x7f548c0cfd80 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34038/0 (socket says 172.21.15.129:34038) 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 -- 172.21.15.129:0/513542586 learned_addr learned my addr 172.21.15.129:0/513542586 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491e00640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548c0b59d0 0x7f548c141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5492800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f548c0cfd80 0x7f548c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 -- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f548c0cfd80 msgr2=0x7f548c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f548c0cfd80 0x7f548c13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 -- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548c0b59d0 msgr2=0x7f548c141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548c0b59d0 0x7f548c141b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 -- 172.21.15.129:0/513542586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f548c13b770 con 0x7f548c0cd190 2024-09-17T04:03:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.919+0000 7f5491400640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f547c0099c0 tx=0x7f547c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:05.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.920+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c03e030 con 0x7f548c0cd190 2024-09-17T04:03:05.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.920+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f548c140f00 con 0x7f548c0cd190 2024-09-17T04:03:05.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.920+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f548c1414c0 con 0x7f548c0cd190 2024-09-17T04:03:05.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.920+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f547c0047a0 con 0x7f548c0cd190 2024-09-17T04:03:05.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.920+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547c048060 con 0x7f548c0cd190 2024-09-17T04:03:05.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.921+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5468005290 con 0x7f548c0cd190 2024-09-17T04:03:05.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.922+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f547c041030 con 0x7f548c0cd190 2024-09-17T04:03:05.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.924+0000 7f5486000640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 0x7f547007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:05.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.924+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f547c0bb3d0 con 0x7f548c0cd190 2024-09-17T04:03:05.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.924+0000 7f5491e00640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 0x7f547007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:05.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.925+0000 7f5491e00640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 0x7f547007b0a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f548000bb50 tx=0x7f548000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:05.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:05.928+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f547c083040 con 0x7f548c0cd190 2024-09-17T04:03:06.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.148+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5468005580 con 0x7f548c0cd190 2024-09-17T04:03:06.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.149+0000 7f5486000640 1 -- 172.21.15.129:0/513542586 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f547c087a70 con 0x7f548c0cd190 2024-09-17T04:03:06.149 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:06.149 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:06.150 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 msgr2=0x7f547007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 0x7f547007b0a0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f548000bb50 tx=0x7f548000b2e0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 msgr2=0x7f548c13b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f547c0099c0 tx=0x7f547c0039e0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 shutdown_connections 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5470078b90 0x7f547007b0a0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f548c0cfd80 0x7f548c13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f548c0cd190 0x7f548c13b230 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 --2- 172.21.15.129:0/513542586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f548c0b59d0 0x7f548c141b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 >> 172.21.15.129:0/513542586 conn(0x7f548c0b23c0 msgr2=0x7f548c0b18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 shutdown_connections 2024-09-17T04:03:06.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:06.152+0000 7f5494800640 1 -- 172.21.15.129:0/513542586 wait complete. 2024-09-17T04:03:06.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:06.412 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1158,"msgr_send_messages":1152,"msgr_recv_bytes":428085,"msgr_send_bytes":649624,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198447955,"msgr_running_send_time":0.096139357,"msgr_running_recv_time":0.040380147,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1152,"sum":2.287209222,"avgtime":0.001985424},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126464,"msgr_send_encrypted_bytes":16576},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9574,"msgr_send_messages":14683,"msgr_recv_bytes":3109805,"msgr_send_bytes":11695018,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.933266644,"msgr_running_send_time":1.362051231,"msgr_running_recv_time":0.843297560,"msgr_running_fast_dispatch_time":0.098353832,"msgr_send_messages_queue_lat":{"avgcount":14681,"sum":2.753222511,"avgtime":0.000187536},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":81888},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3093,"sum":2.996487716,"avgtime":0.000968796}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9086,"evex":8670,"evtrm":8670,"ev":419,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21778963,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141812,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1313,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":985260,"buffer_anon_items":530,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1064758,"mds_co_items":19706,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5443,"op_send_bytes":18227829,"op_resend":0,"op_reply":5443,"op_latency":{"avgcount":5443,"sum":73.295418403,"avgtime":0.013465996},"op_inflight":0,"oplen_avg":{"avgcount":5443,"sum":6452},"op":5443,"op_r":56,"op_w":5387,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13136,"get_sum":3250855,"get_or_fail_fail":0,"get_or_fail_success":13136,"take":0,"take_sum":0,"put":13136,"put_sum":3250855,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":18373699,"put":5443,"put_sum":18373699,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":5443,"put":5443,"put_sum":5443,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":0,"max":3758096384,"get_started":0,"get":9086,"get_sum":17583460,"get_or_fail_fail":0,"get_or_fail_success":9086,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:06.413 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 428085, "msgr_recv_encrypted_bytes": 126464, "msgr_recv_messages": 1158, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040380147, "msgr_running_send_time": 0.096139357, "msgr_running_total_time": 0.198447955, "msgr_send_bytes": 649624, "msgr_send_encrypted_bytes": 16576, "msgr_send_messages": 1152, "msgr_send_messages_queue_lat": { "avgcount": 1152, "avgtime": 0.001985424, "sum": 2.287209222 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3109805, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9574, "msgr_running_fast_dispatch_time": 0.098353832, "msgr_running_recv_time": 0.84329756, "msgr_running_send_time": 1.362051231, "msgr_running_total_time": 1.933266644, "msgr_send_bytes": 11695018, "msgr_send_encrypted_bytes": 81888, "msgr_send_messages": 14683, "msgr_send_messages_queue_lat": { "avgcount": 14681, "avgtime": 0.000187536, "sum": 2.753222511 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3093, "avgtime": 0.000968796, "sum": 2.996487716 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 419, "evadd": 9086, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21778963 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141812 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1313, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 985260, "buffer_anon_items": 530, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1064758, "mds_co_items": 19706, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5443, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5443, "avgtime": 0.013465996, "sum": 73.295418403 }, "op_pg": 0, "op_r": 56, "op_reply": 5443, "op_resend": 0, "op_rmw": 0, "op_send": 5443, "op_send_bytes": 18227829, "op_w": 5387, "oplen_avg": { "avgcount": 5443, "sum": 6452 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13136, "get_or_fail_fail": 0, "get_or_fail_success": 13136, "get_started": 0, "get_sum": 3250855, "max": 104857600, "put": 13136, "put_sum": 3250855, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5443, "put_sum": 18373699, "take": 5443, "take_sum": 18373699, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5443, "put_sum": 5443, "take": 5443, "take_sum": 5443, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9086, "get_or_fail_fail": 0, "get_or_fail_success": 9086, "get_started": 0, "get_sum": 17583460, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:06.413 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:07.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:07.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 -- 172.21.15.129:0/653627422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 msgr2=0x7f73000b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 --2- 172.21.15.129:0/653627422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f73000b5de0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f72f8005fd0 tx=0x7f72f802a770 comp rx=0 tx=0).stop 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 -- 172.21.15.129:0/653627422 shutdown_connections 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 --2- 172.21.15.129:0/653627422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f73000b5de0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 -- 172.21.15.129:0/653627422 >> 172.21.15.129:0/653627422 conn(0x7f73000b23c0 msgr2=0x7f73000b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 -- 172.21.15.129:0/653627422 shutdown_connections 2024-09-17T04:03:07.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.639+0000 7f7307400640 1 -- 172.21.15.129:0/653627422 wait complete. 2024-09-17T04:03:07.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.641+0000 7f7307400640 1 Processor -- start 2024-09-17T04:03:07.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.641+0000 7f7307400640 1 -- start start 2024-09-17T04:03:07.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f7300142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:07.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:07.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f730015cf70 0x7f7300142040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:07.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7300070060 con 0x7f73000cd190 2024-09-17T04:03:07.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f730006fee0 con 0x7f73000b59d0 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7307400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f73000701e0 con 0x7f730015cf70 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f72ffe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f72ffe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34054/0 (socket says 172.21.15.129:34054) 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f72ffe00640 1 -- 172.21.15.129:0/4132337412 learned_addr learned my addr 172.21.15.129:0/4132337412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7305400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f730015cf70 0x7f7300142040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:07.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.643+0000 7f7304a00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f7300142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:07.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 -- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f730015cf70 msgr2=0x7f7300142040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:07.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f730015cf70 0x7f7300142040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 -- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 msgr2=0x7f7300142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f7300142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 -- 172.21.15.129:0/4132337412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73000c5630 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f7304a00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f7300142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72ffe00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00c6c0 tx=0x7f72ec00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec005270 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f72ec005430 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7300134330 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec00a8a0 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.644+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73001348f0 con 0x7f73000cd190 2024-09-17T04:03:07.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.646+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7300143b90 con 0x7f73000cd190 2024-09-17T04:03:07.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.647+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f72ec00aa40 con 0x7f73000cd190 2024-09-17T04:03:07.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.648+0000 7f72fcc00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 0x7f72e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:07.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.648+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f72ec09abd0 con 0x7f73000cd190 2024-09-17T04:03:07.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.648+0000 7f7304a00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 0x7f72e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:07.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.649+0000 7f7304a00640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 0x7f72e007b0a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f72f4001220 tx=0x7f72f4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:07.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.652+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f73000b6320 con 0x7f73000cd190 2024-09-17T04:03:07.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.870+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73000b5de0 con 0x7f73000cd190 2024-09-17T04:03:07.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.870+0000 7f72fcc00640 1 -- 172.21.15.129:0/4132337412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f72ec062fb0 con 0x7f73000cd190 2024-09-17T04:03:07.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:07.871 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:07.871 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 msgr2=0x7f72e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 0x7f72e007b0a0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f72f4001220 tx=0x7f72f4009040 comp rx=0 tx=0).stop 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 msgr2=0x7f7300141b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f72ec00c6c0 tx=0x7f72ec00cb80 comp rx=0 tx=0).stop 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 shutdown_connections 2024-09-17T04:03:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f72e0078b90 0x7f72e007b0a0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f730015cf70 0x7f7300142040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f73000cd190 0x7f7300141b00 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 --2- 172.21.15.129:0/4132337412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73000b59d0 0x7f7300142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 >> 172.21.15.129:0/4132337412 conn(0x7f73000b23c0 msgr2=0x7f73000ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 shutdown_connections 2024-09-17T04:03:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:07.873+0000 7f7307400640 1 -- 172.21.15.129:0/4132337412 wait complete. 2024-09-17T04:03:07.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:08.138 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1158,"msgr_send_messages":1152,"msgr_recv_bytes":428085,"msgr_send_bytes":649624,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198554317,"msgr_running_send_time":0.096191338,"msgr_running_recv_time":0.040407103,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1152,"sum":2.287209222,"avgtime":0.001985424},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126464,"msgr_send_encrypted_bytes":16576},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9576,"msgr_send_messages":14683,"msgr_recv_bytes":3110609,"msgr_send_bytes":11695018,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.933623545,"msgr_running_send_time":1.362208691,"msgr_running_recv_time":0.843386789,"msgr_running_fast_dispatch_time":0.098454225,"msgr_send_messages_queue_lat":{"avgcount":14681,"sum":2.753222511,"avgtime":0.000187536},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":81888},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3093,"sum":2.996487716,"avgtime":0.000968796}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9086,"evex":8670,"evtrm":8670,"ev":419,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21778963,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141812,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1313,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":985260,"buffer_anon_items":530,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":1064758,"mds_co_items":19706,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5443,"op_send_bytes":18227829,"op_resend":0,"op_reply":5443,"op_latency":{"avgcount":5443,"sum":73.295418403,"avgtime":0.013465996},"op_inflight":0,"oplen_avg":{"avgcount":5443,"sum":6452},"op":5443,"op_r":56,"op_w":5387,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13138,"get_sum":3251511,"get_or_fail_fail":0,"get_or_fail_success":13138,"take":0,"take_sum":0,"put":13138,"put_sum":3251511,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":18373699,"put":5443,"put_sum":18373699,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":5443,"put":5443,"put_sum":5443,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":0,"max":3758096384,"get_started":0,"get":9086,"get_sum":17583460,"get_or_fail_fail":0,"get_or_fail_success":9086,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:08.140 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 428085, "msgr_recv_encrypted_bytes": 126464, "msgr_recv_messages": 1158, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040407103, "msgr_running_send_time": 0.096191338, "msgr_running_total_time": 0.198554317, "msgr_send_bytes": 649624, "msgr_send_encrypted_bytes": 16576, "msgr_send_messages": 1152, "msgr_send_messages_queue_lat": { "avgcount": 1152, "avgtime": 0.001985424, "sum": 2.287209222 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3110609, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9576, "msgr_running_fast_dispatch_time": 0.098454225, "msgr_running_recv_time": 0.843386789, "msgr_running_send_time": 1.362208691, "msgr_running_total_time": 1.933623545, "msgr_send_bytes": 11695018, "msgr_send_encrypted_bytes": 81888, "msgr_send_messages": 14683, "msgr_send_messages_queue_lat": { "avgcount": 14681, "avgtime": 0.000187536, "sum": 2.753222511 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3093, "avgtime": 0.000968796, "sum": 2.996487716 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 419, "evadd": 9086, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21778963 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141812 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1313, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 985260, "buffer_anon_items": 530, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 1064758, "mds_co_items": 19706, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5443, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5443, "avgtime": 0.013465996, "sum": 73.295418403 }, "op_pg": 0, "op_r": 56, "op_reply": 5443, "op_resend": 0, "op_rmw": 0, "op_send": 5443, "op_send_bytes": 18227829, "op_w": 5387, "oplen_avg": { "avgcount": 5443, "sum": 6452 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13138, "get_or_fail_fail": 0, "get_or_fail_success": 13138, "get_started": 0, "get_sum": 3251511, "max": 104857600, "put": 13138, "put_sum": 3251511, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5443, "put_sum": 18373699, "take": 5443, "take_sum": 18373699, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5443, "put_sum": 5443, "take": 5443, "take_sum": 5443, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9086, "get_or_fail_fail": 0, "get_or_fail_success": 9086, "get_started": 0, "get_sum": 17583460, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:08.140 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:09.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.366+0000 7f1f28c00640 1 -- 172.21.15.129:0/3962412690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 msgr2=0x7f1f2406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.366+0000 7f1f28c00640 1 --2- 172.21.15.129:0/3962412690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2406e310 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f1f14005fd0 tx=0x7f1f1402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 -- 172.21.15.129:0/3962412690 shutdown_connections 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 --2- 172.21.15.129:0/3962412690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 0x7f1f240722b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 --2- 172.21.15.129:0/3962412690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2406e310 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 -- 172.21.15.129:0/3962412690 >> 172.21.15.129:0/3962412690 conn(0x7f1f2406d560 msgr2=0x7f1f240b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 -- 172.21.15.129:0/3962412690 shutdown_connections 2024-09-17T04:03:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.367+0000 7f1f28c00640 1 -- 172.21.15.129:0/3962412690 wait complete. 2024-09-17T04:03:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.369+0000 7f1f28c00640 1 Processor -- start 2024-09-17T04:03:09.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.369+0000 7f1f28c00640 1 -- start start 2024-09-17T04:03:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f24141f60 0x7f1f240ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 0x7f1f2415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24074ee0 con 0x7f1f2415da40 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1f24074d60 con 0x7f1f24141f60 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f28c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1f24075060 con 0x7f1f24156f90 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f1aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 0x7f1f2415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f24141f60 0x7f1f240ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34076/0 (socket says 172.21.15.129:34076) 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 -- 172.21.15.129:0/4215635999 learned_addr learned my addr 172.21.15.129:0/4215635999 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 -- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 msgr2=0x7f1f2415bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 0x7f1f2415bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 -- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f24141f60 msgr2=0x7f1f240ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f24141f60 0x7f1f240ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.371+0000 7f1f23e00640 1 -- 172.21.15.129:0/4215635999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f240cf420 con 0x7f1f2415da40 2024-09-17T04:03:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f23e00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00c730 tx=0x7f1f1c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:09.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c003b70 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1f1c003d10 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f240cf700 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c005330 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.372+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f240cfcc0 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.373+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f240d1360 con 0x7f1f2415da40 2024-09-17T04:03:09.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.374+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1f1c0054d0 con 0x7f1f2415da40 2024-09-17T04:03:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.375+0000 7f1f1be00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.375+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f1f1c09ab30 con 0x7f1f2415da40 2024-09-17T04:03:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.376+0000 7f1f23400640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.377+0000 7f1f23400640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1f14033000 tx=0x7f1f14004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:09.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.380+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1f240727f0 con 0x7f1f2415da40 2024-09-17T04:03:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.597+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f24154b90 con 0x7f1f2415da40 2024-09-17T04:03:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.598+0000 7f1f1be00640 1 -- 172.21.15.129:0/4215635999 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f1f1c062790 con 0x7f1f2415da40 2024-09-17T04:03:09.598 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:09.598 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 msgr2=0x7f1f0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1f14033000 tx=0x7f1f14004730 comp rx=0 tx=0).stop 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 msgr2=0x7f1f2415c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00c730 tx=0x7f1f1c00cb30 comp rx=0 tx=0).stop 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 shutdown_connections 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1f00078b90 0x7f1f0007b0a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1f2415da40 0x7f1f2415c130 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1f24156f90 0x7f1f2415bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 --2- 172.21.15.129:0/4215635999 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1f24141f60 0x7f1f240ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.600+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 >> 172.21.15.129:0/4215635999 conn(0x7f1f2406d560 msgr2=0x7f1f240b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.601+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 shutdown_connections 2024-09-17T04:03:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:09.601+0000 7f1f28c00640 1 -- 172.21.15.129:0/4215635999 wait complete. 2024-09-17T04:03:09.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:09.871 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1159,"msgr_send_messages":1153,"msgr_recv_bytes":428357,"msgr_send_bytes":650264,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198693447,"msgr_running_send_time":0.096283174,"msgr_running_recv_time":0.040466802,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1153,"sum":2.287248446,"avgtime":0.001983736},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126736,"msgr_send_encrypted_bytes":17216},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9578,"msgr_send_messages":14683,"msgr_recv_bytes":3111413,"msgr_send_bytes":11695018,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.933865899,"msgr_running_send_time":1.362299759,"msgr_running_recv_time":0.843448503,"msgr_running_fast_dispatch_time":0.098538739,"msgr_send_messages_queue_lat":{"avgcount":14681,"sum":2.753222511,"avgtime":0.000187536},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":81888},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3093,"sum":2.996487716,"avgtime":0.000968796}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9086,"evex":8670,"evtrm":8670,"ev":419,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21778963,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141812,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1318,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248429,"buffer_anon_items":256,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":815290,"mds_co_items":14118,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5443,"op_send_bytes":18227829,"op_resend":0,"op_reply":5443,"op_latency":{"avgcount":5443,"sum":73.295418403,"avgtime":0.013465996},"op_inflight":0,"oplen_avg":{"avgcount":5443,"sum":6452},"op":5443,"op_r":56,"op_w":5387,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13141,"get_sum":3252297,"get_or_fail_fail":0,"get_or_fail_success":13141,"take":0,"take_sum":0,"put":13141,"put_sum":3252297,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":18373699,"put":5443,"put_sum":18373699,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":5443,"put":5443,"put_sum":5443,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":0,"max":3758096384,"get_started":0,"get":9086,"get_sum":17583460,"get_or_fail_fail":0,"get_or_fail_success":9086,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:09.873 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 428357, "msgr_recv_encrypted_bytes": 126736, "msgr_recv_messages": 1159, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040466802, "msgr_running_send_time": 0.096283174, "msgr_running_total_time": 0.198693447, "msgr_send_bytes": 650264, "msgr_send_encrypted_bytes": 17216, "msgr_send_messages": 1153, "msgr_send_messages_queue_lat": { "avgcount": 1153, "avgtime": 0.001983736, "sum": 2.287248446 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3111413, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9578, "msgr_running_fast_dispatch_time": 0.098538739, "msgr_running_recv_time": 0.843448503, "msgr_running_send_time": 1.362299759, "msgr_running_total_time": 1.933865899, "msgr_send_bytes": 11695018, "msgr_send_encrypted_bytes": 81888, "msgr_send_messages": 14683, "msgr_send_messages_queue_lat": { "avgcount": 14681, "avgtime": 0.000187536, "sum": 2.753222511 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3093, "avgtime": 0.000968796, "sum": 2.996487716 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 419, "evadd": 9086, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21778963 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141812 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1318, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248429, "buffer_anon_items": 256, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 815290, "mds_co_items": 14118, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5443, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5443, "avgtime": 0.013465996, "sum": 73.295418403 }, "op_pg": 0, "op_r": 56, "op_reply": 5443, "op_resend": 0, "op_rmw": 0, "op_send": 5443, "op_send_bytes": 18227829, "op_w": 5387, "oplen_avg": { "avgcount": 5443, "sum": 6452 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13141, "get_or_fail_fail": 0, "get_or_fail_success": 13141, "get_started": 0, "get_sum": 3252297, "max": 104857600, "put": 13141, "put_sum": 3252297, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5443, "put_sum": 18373699, "take": 5443, "take_sum": 18373699, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5443, "put_sum": 5443, "take": 5443, "take_sum": 5443, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9086, "get_or_fail_fail": 0, "get_or_fail_success": 9086, "get_started": 0, "get_sum": 17583460, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:09.873 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:10.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:11.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 -- 172.21.15.129:0/1451121384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 msgr2=0x7f1994141800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 --2- 172.21.15.129:0/1451121384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994141800 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f1984005fd0 tx=0x7f198402a670 comp rx=0 tx=0).stop 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 -- 172.21.15.129:0/1451121384 shutdown_connections 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 --2- 172.21.15.129:0/1451121384 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994141800 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 -- 172.21.15.129:0/1451121384 >> 172.21.15.129:0/1451121384 conn(0x7f19940b23c0 msgr2=0x7f19940b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 -- 172.21.15.129:0/1451121384 shutdown_connections 2024-09-17T04:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.104+0000 7f1999800640 1 -- 172.21.15.129:0/1451121384 wait complete. 2024-09-17T04:03:11.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.106+0000 7f1999800640 1 Processor -- start 2024-09-17T04:03:11.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.107+0000 7f1999800640 1 -- start start 2024-09-17T04:03:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19940ae890 0x7f1994159710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19940aeca0 0x7f1994159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:11.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19940b5f40 con 0x7f199415f110 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19940700c0 con 0x7f19940aeca0 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1999800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f19940b60c0 con 0x7f19940ae890 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1993400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1993400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:34092/0 (socket says 172.21.15.129:34092) 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1993400640 1 -- 172.21.15.129:0/2316175684 learned_addr learned my addr 172.21.15.129:0/2316175684 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f1992a00640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19940ae890 0x7f1994159710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:11.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.108+0000 7f198a000640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19940aeca0 0x7f1994159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 -- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19940ae890 msgr2=0x7f1994159710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19940ae890 0x7f1994159710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 -- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19940aeca0 msgr2=0x7f1994159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19940aeca0 0x7f1994159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 -- 172.21.15.129:0/2316175684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1994159010 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1993400640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f198c002230 tx=0x7f198c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c003b70 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f198c003d10 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1994146c20 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.109+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198c005330 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.110+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19941471e0 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.111+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199413c470 con 0x7f199415f110 2024-09-17T04:03:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.112+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f198c0054d0 con 0x7f199415f110 2024-09-17T04:03:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.113+0000 7f198b400640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 0x7f197007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.113+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f198c09aa80 con 0x7f199415f110 2024-09-17T04:03:11.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.113+0000 7f1992a00640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 0x7f197007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.114+0000 7f1992a00640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 0x7f197007b0a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1984033000 tx=0x7f1984004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.117+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f199413c470 con 0x7f199415f110 2024-09-17T04:03:11.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.333+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19940b6570 con 0x7f199415f110 2024-09-17T04:03:11.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.334+0000 7f198b400640 1 -- 172.21.15.129:0/2316175684 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f198c09ce20 con 0x7f199415f110 2024-09-17T04:03:11.334 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:11.334 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:11.335 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 msgr2=0x7f197007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 0x7f197007b0a0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1984033000 tx=0x7f1984004730 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 msgr2=0x7f1994157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f198c002230 tx=0x7f198c00cb90 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 shutdown_connections 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1970078b90 0x7f197007b0a0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f199415f110 0x7f1994157e60 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19940aeca0 0x7f1994159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.336+0000 7f1999800640 1 --2- 172.21.15.129:0/2316175684 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19940ae890 0x7f1994159710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.337+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 >> 172.21.15.129:0/2316175684 conn(0x7f19940b23c0 msgr2=0x7f199406f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.337+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 shutdown_connections 2024-09-17T04:03:11.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:11.337+0000 7f1999800640 1 -- 172.21.15.129:0/2316175684 wait complete. 2024-09-17T04:03:11.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:11.600 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1159,"msgr_send_messages":1153,"msgr_recv_bytes":428357,"msgr_send_bytes":650264,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198693447,"msgr_running_send_time":0.096283174,"msgr_running_recv_time":0.040466802,"msgr_running_fast_dispatch_time":0.053466873,"msgr_send_messages_queue_lat":{"avgcount":1153,"sum":2.287248446,"avgtime":0.001983736},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":126736,"msgr_send_encrypted_bytes":17216},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9579,"msgr_send_messages":14684,"msgr_recv_bytes":3111815,"msgr_send_bytes":11696858,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.934114425,"msgr_running_send_time":1.362444851,"msgr_running_recv_time":0.843484595,"msgr_running_fast_dispatch_time":0.098593469,"msgr_send_messages_queue_lat":{"avgcount":14682,"sum":2.753283245,"avgtime":0.000187527},"msgr_handle_ack_lat":{"avgcount":8332,"sum":0.004242123,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":83728},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3093,"sum":2.996487716,"avgtime":0.000968796}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4825,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4278,"handle_client_caps_dirty":1500,"handle_client_cap_release":96,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9086,"evex":8670,"evtrm":8670,"ev":419,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21778963,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":167,"cap+":1602,"cap-":1435,"rss":141660,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":4,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1318,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248429,"buffer_anon_items":256,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":815290,"mds_co_items":14118,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5443,"op_send_bytes":18227829,"op_resend":0,"op_reply":5443,"op_latency":{"avgcount":5443,"sum":73.295418403,"avgtime":0.013465996},"op_inflight":0,"oplen_avg":{"avgcount":5443,"sum":6452},"op":5443,"op_r":56,"op_w":5387,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":123,"omap_rd":48,"omap_del":46},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13142,"get_sum":3252625,"get_or_fail_fail":0,"get_or_fail_success":13142,"take":0,"take_sum":0,"put":13142,"put_sum":3252625,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":18373699,"put":5443,"put_sum":18373699,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5443,"take_sum":5443,"put":5443,"put_sum":5443,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":0,"max":3758096384,"get_started":0,"get":9086,"get_sum":17583460,"get_or_fail_fail":0,"get_or_fail_success":9086,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:11.601 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 428357, "msgr_recv_encrypted_bytes": 126736, "msgr_recv_messages": 1159, "msgr_running_fast_dispatch_time": 0.053466873, "msgr_running_recv_time": 0.040466802, "msgr_running_send_time": 0.096283174, "msgr_running_total_time": 0.198693447, "msgr_send_bytes": 650264, "msgr_send_encrypted_bytes": 17216, "msgr_send_messages": 1153, "msgr_send_messages_queue_lat": { "avgcount": 1153, "avgtime": 0.001983736, "sum": 2.287248446 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8332, "avgtime": 5.09e-07, "sum": 0.004242123 }, "msgr_recv_bytes": 3111815, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9579, "msgr_running_fast_dispatch_time": 0.098593469, "msgr_running_recv_time": 0.843484595, "msgr_running_send_time": 1.362444851, "msgr_running_total_time": 1.934114425, "msgr_send_bytes": 11696858, "msgr_send_encrypted_bytes": 83728, "msgr_send_messages": 14684, "msgr_send_messages_queue_lat": { "avgcount": 14682, "avgtime": 0.000187527, "sum": 2.753283245 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3093, "avgtime": 0.000968796, "sum": 2.996487716 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4825, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 96, "handle_client_caps": 4278, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 419, "evadd": 9086, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21778963 }, "mds_mem": { "cap": 167, "cap+": 1602, "cap-": 1435, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141660 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 4, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1318, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248429, "buffer_anon_items": 256, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 815290, "mds_co_items": 14118, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 46, "omap_rd": 48, "omap_wr": 123, "op": 5443, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5443, "avgtime": 0.013465996, "sum": 73.295418403 }, "op_pg": 0, "op_r": 56, "op_reply": 5443, "op_resend": 0, "op_rmw": 0, "op_send": 5443, "op_send_bytes": 18227829, "op_w": 5387, "oplen_avg": { "avgcount": 5443, "sum": 6452 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13142, "get_or_fail_fail": 0, "get_or_fail_success": 13142, "get_started": 0, "get_sum": 3252625, "max": 104857600, "put": 13142, "put_sum": 3252625, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5443, "put_sum": 18373699, "take": 5443, "take_sum": 18373699, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5443, "put_sum": 5443, "take": 5443, "take_sum": 5443, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9086, "get_or_fail_fail": 0, "get_or_fail_success": 9086, "get_started": 0, "get_sum": 17583460, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:11.601 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:12.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:13.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 -- 172.21.15.129:0/1991667776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30070790 msgr2=0x7f6c3006f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 --2- 172.21.15.129:0/1991667776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30070790 0x7f6c3006f710 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6c20005fd0 tx=0x7f6c2002a5a0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 -- 172.21.15.129:0/1991667776 shutdown_connections 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 --2- 172.21.15.129:0/1991667776 >> v2:172.21.15.129:3300/0 conn(0x7f6c300b4340 0x7f6c3006fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 --2- 172.21.15.129:0/1991667776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30070790 0x7f6c3006f710 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 -- 172.21.15.129:0/1991667776 >> 172.21.15.129:0/1991667776 conn(0x7f6c300b23c0 msgr2=0x7f6c300b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 -- 172.21.15.129:0/1991667776 shutdown_connections 2024-09-17T04:03:13.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.028+0000 7f6c36c00640 1 -- 172.21.15.129:0/1991667776 wait complete. 2024-09-17T04:03:13.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.033+0000 7f6c36c00640 1 Processor -- start 2024-09-17T04:03:13.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.034+0000 7f6c36c00640 1 -- start start 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c300b4340 0x7f6c30152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30151720 0x7f6c30152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c30073080 con 0x7f6c300b4340 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c30072f00 con 0x7f6c30151720 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c36c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3015b6e0 con 0x7f6c30070790 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c2fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c2fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:52796/0 (socket says 172.21.15.129:52796) 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c2fe00640 1 -- 172.21.15.129:0/24633272 learned_addr learned my addr 172.21.15.129:0/24633272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c2f400640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c300b4340 0x7f6c30152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:13.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.037+0000 7f6c34c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30151720 0x7f6c30152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 -- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30151720 msgr2=0x7f6c30152ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30151720 0x7f6c30152ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 -- 172.21.15.129:0/24633272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c300b4340 msgr2=0x7f6c30152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c300b4340 0x7f6c30152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 -- 172.21.15.129:0/24633272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3014a650 con 0x7f6c30070790 2024-09-17T04:03:13.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c2fe00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c200038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2003e030 con 0x7f6c30070790 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.038+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c20003d90 con 0x7f6c30070790 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.039+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3014a930 con 0x7f6c30070790 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.039+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20035040 con 0x7f6c30070790 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.039+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3014aef0 con 0x7f6c30070790 2024-09-17T04:03:13.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.040+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c30151b30 con 0x7f6c30070790 2024-09-17T04:03:13.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.042+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c20041030 con 0x7f6c30070790 2024-09-17T04:03:13.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.044+0000 7f6c27e00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 0x7f6c1007b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:13.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.044+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f6c200bb320 con 0x7f6c30070790 2024-09-17T04:03:13.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.044+0000 7f6c2f400640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 0x7f6c1007b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:13.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.046+0000 7f6c2f400640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 0x7f6c1007b0f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:13.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.056+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c20083090 con 0x7f6c30070790 2024-09-17T04:03:13.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.396+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c301543d0 con 0x7f6c30070790 2024-09-17T04:03:13.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.398+0000 7f6c27e00640 1 -- 172.21.15.129:0/24633272 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f6c20087ac0 con 0x7f6c30070790 2024-09-17T04:03:13.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:13.401 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.402+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 msgr2=0x7f6c1007b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.402+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 0x7f6c1007b0f0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c005fd0 tx=0x7f6c1c005850 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.402+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 msgr2=0x7f6c3014f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.402+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c200038a0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 shutdown_connections 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c10078be0 0x7f6c1007b0f0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c30151720 0x7f6c30152ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c300b4340 0x7f6c30152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 --2- 172.21.15.129:0/24633272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c30070790 0x7f6c3014f200 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 >> 172.21.15.129:0/24633272 conn(0x7f6c300b23c0 msgr2=0x7f6c300b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 shutdown_connections 2024-09-17T04:03:13.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:13.403+0000 7f6c36c00640 1 -- 172.21.15.129:0/24633272 wait complete. 2024-09-17T04:03:13.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:13.740 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1161,"msgr_send_messages":1155,"msgr_recv_bytes":428921,"msgr_send_bytes":651880,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.198938238,"msgr_running_send_time":0.096405273,"msgr_running_recv_time":0.040543879,"msgr_running_fast_dispatch_time":0.053512923,"msgr_send_messages_queue_lat":{"avgcount":1155,"sum":2.287310789,"avgtime":0.001980355},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127008,"msgr_send_encrypted_bytes":17856},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9634,"msgr_send_messages":14691,"msgr_recv_bytes":3130037,"msgr_send_bytes":11699019,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":1.947220350,"msgr_running_send_time":1.372849407,"msgr_running_recv_time":0.848786884,"msgr_running_fast_dispatch_time":0.098696171,"msgr_send_messages_queue_lat":{"avgcount":14689,"sum":2.753877042,"avgtime":0.000187478},"msgr_handle_ack_lat":{"avgcount":8339,"sum":0.004248738,"avgtime":0.000000509},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":83728},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2411,"msgr_send_messages":2414,"msgr_recv_bytes":777427,"msgr_send_bytes":9496066,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.478147307,"msgr_running_send_time":0.264822559,"msgr_running_recv_time":0.074999269,"msgr_running_fast_dispatch_time":0.115481044,"msgr_send_messages_queue_lat":{"avgcount":2411,"sum":2.845604082,"avgtime":0.001180258},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3094,"sum":2.996497504,"avgtime":0.000968486}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4260,"sum":0.518309937,"avgtime":0.000121668}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":167,"caps":167,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4831,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4323,"handle_client_caps_dirty":1500,"handle_client_cap_release":103,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":166,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1435,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9131,"evex":8670,"evtrm":8670,"ev":464,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":21869924,"rdpos":4195244,"jlat":{"avgcount":1530,"sum":7.059810113,"avgtime":0.004614254},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":123,"cap+":1602,"cap-":1479,"rss":141660,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1318,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":248429,"buffer_anon_items":256,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":815290,"mds_co_items":14118,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5444,"op_send_bytes":18228445,"op_resend":0,"op_reply":5444,"op_latency":{"avgcount":5444,"sum":73.319417885,"avgtime":0.013467931},"op_inflight":0,"oplen_avg":{"avgcount":5444,"sum":6454},"op":5444,"op_r":56,"op_w":5388,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":28,"osdop_write":1550,"osdop_writefull":18,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3324,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":124,"omap_rd":48,"omap_del":47},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3305,"pq_executed":1435,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13199,"get_sum":3267109,"get_or_fail_fail":0,"get_or_fail_success":13199,"take":0,"take_sum":0,"put":13199,"put_sum":3267109,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5444,"take_sum":18374315,"put":5444,"put_sum":18374315,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5444,"take_sum":5444,"put":5444,"put_sum":5444,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1435,"get_sum":144935,"get_or_fail_fail":0,"get_or_fail_success":1435,"take":0,"take_sum":0,"put":20,"put_sum":144935,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":90961,"max":3758096384,"get_started":0,"get":9131,"get_sum":17674421,"get_or_fail_fail":0,"get_or_fail_success":9131,"take":0,"take_sum":0,"put":1530,"put_sum":17583460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:13.741 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 428921, "msgr_recv_encrypted_bytes": 127008, "msgr_recv_messages": 1161, "msgr_running_fast_dispatch_time": 0.053512923, "msgr_running_recv_time": 0.040543879, "msgr_running_send_time": 0.096405273, "msgr_running_total_time": 0.198938238, "msgr_send_bytes": 651880, "msgr_send_encrypted_bytes": 17856, "msgr_send_messages": 1155, "msgr_send_messages_queue_lat": { "avgcount": 1155, "avgtime": 0.001980355, "sum": 2.287310789 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8339, "avgtime": 5.09e-07, "sum": 0.004248738 }, "msgr_recv_bytes": 3130037, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9634, "msgr_running_fast_dispatch_time": 0.098696171, "msgr_running_recv_time": 0.848786884, "msgr_running_send_time": 1.372849407, "msgr_running_total_time": 1.94722035, "msgr_send_bytes": 11699019, "msgr_send_encrypted_bytes": 83728, "msgr_send_messages": 14691, "msgr_send_messages_queue_lat": { "avgcount": 14689, "avgtime": 0.000187478, "sum": 2.753877042 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 777427, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2411, "msgr_running_fast_dispatch_time": 0.115481044, "msgr_running_recv_time": 0.074999269, "msgr_running_send_time": 0.264822559, "msgr_running_total_time": 0.478147307, "msgr_send_bytes": 9496066, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2414, "msgr_send_messages_queue_lat": { "avgcount": 2411, "avgtime": 0.001180258, "sum": 2.845604082 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3094, "avgtime": 0.000968486, "sum": 2.996497504 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4260, "avgtime": 0.000121668, "sum": 0.518309937 }, "queue_len": 0 }, "mds": { "caps": 167, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4831, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 103, "handle_client_caps": 4323, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 167, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 166, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1435, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 464, "evadd": 9131, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1530, "avgtime": 0.004614254, "sum": 7.059810113 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 21869924 }, "mds_mem": { "cap": 123, "cap+": 1602, "cap-": 1479, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141660 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1318, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 248429, "buffer_anon_items": 256, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 815290, "mds_co_items": 14118, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 47, "omap_rd": 48, "omap_wr": 124, "op": 5444, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5444, "avgtime": 0.013467931, "sum": 73.319417885 }, "op_pg": 0, "op_r": 56, "op_reply": 5444, "op_resend": 0, "op_rmw": 0, "op_send": 5444, "op_send_bytes": 18228445, "op_w": 5388, "oplen_avg": { "avgcount": 5444, "sum": 6454 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3324, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 28, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1550, "osdop_writefull": 18, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1435, "pq_executed_ops": 3305, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13199, "get_or_fail_fail": 0, "get_or_fail_success": 13199, "get_started": 0, "get_sum": 3267109, "max": 104857600, "put": 13199, "put_sum": 3267109, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5444, "put_sum": 18374315, "take": 5444, "take_sum": 18374315, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5444, "put_sum": 5444, "take": 5444, "take_sum": 5444, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1435, "get_or_fail_fail": 0, "get_or_fail_success": 1435, "get_started": 0, "get_sum": 144935, "max": 3758096384, "put": 20, "put_sum": 144935, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9131, "get_or_fail_fail": 0, "get_or_fail_success": 9131, "get_started": 0, "get_sum": 17674421, "max": 3758096384, "put": 1530, "put_sum": 17583460, "take": 0, "take_sum": 0, "val": 90961, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:13.742 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/166, 1435/1601 2024-09-17T04:03:14.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:15.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.038+0000 7f2018e00640 1 -- 172.21.15.129:0/343298345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 msgr2=0x7f20140d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:15.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.038+0000 7f2018e00640 1 --2- 172.21.15.129:0/343298345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f20140d2b60 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f201408d630 tx=0x7f200402a490 comp rx=0 tx=0).stop 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 -- 172.21.15.129:0/343298345 shutdown_connections 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 --2- 172.21.15.129:0/343298345 >> v2:172.21.15.129:3300/0 conn(0x7f20140b6db0 0x7f20140b7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 --2- 172.21.15.129:0/343298345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f20140d2b60 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 -- 172.21.15.129:0/343298345 >> 172.21.15.129:0/343298345 conn(0x7f20140d4a70 msgr2=0x7f20140d4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 -- 172.21.15.129:0/343298345 shutdown_connections 2024-09-17T04:03:15.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.039+0000 7f2018e00640 1 -- 172.21.15.129:0/343298345 wait complete. 2024-09-17T04:03:15.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.041+0000 7f2018e00640 1 Processor -- start 2024-09-17T04:03:15.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.042+0000 7f2018e00640 1 -- start start 2024-09-17T04:03:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 0x7f20140cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f201414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f20140b8870 con 0x7f20140b6db0 2024-09-17T04:03:15.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f20140b86f0 con 0x7f201415ef60 2024-09-17T04:03:15.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.044+0000 7f2018e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f20140b89f0 con 0x7f20140d00e0 2024-09-17T04:03:15.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.045+0000 7f2011e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:15.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.045+0000 7f2011e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47172/0 (socket says 172.21.15.129:47172) 2024-09-17T04:03:15.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.045+0000 7f2011e00640 1 -- 172.21.15.129:0/1631824484 learned_addr learned my addr 172.21.15.129:0/1631824484 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:15.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.046+0000 7f2012800640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f201414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:15.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.046+0000 7f2011e00640 1 -- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 msgr2=0x7f20140cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:15.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011400640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 0x7f20140cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:15.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 0x7f20140cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011e00640 1 -- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 msgr2=0x7f201414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:15.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f201414ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011e00640 1 -- 172.21.15.129:0/1631824484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201415f910 con 0x7f20140b6db0 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.047+0000 7f2011400640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 0x7f20140cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f2011e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f2004009780 tx=0x7f2004033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004031d60 con 0x7f20140b6db0 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2004033e10 con 0x7f20140b6db0 2024-09-17T04:03:15.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2004005280 con 0x7f20140b6db0 2024-09-17T04:03:15.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201415fbf0 con 0x7f20140b6db0 2024-09-17T04:03:15.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.048+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20141601b0 con 0x7f20140b6db0 2024-09-17T04:03:15.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.051+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20140b8f10 con 0x7f20140b6db0 2024-09-17T04:03:15.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.052+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2004005420 con 0x7f20140b6db0 2024-09-17T04:03:15.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.054+0000 7f200a000640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 0x7f1ff407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:15.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.054+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f20040bbf90 con 0x7f20140b6db0 2024-09-17T04:03:15.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.055+0000 7f2011400640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 0x7f1ff407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:15.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.056+0000 7f2011400640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 0x7f1ff407b0a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f20141ea660 tx=0x7f2000009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:15.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.060+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f200403f030 con 0x7f20140b6db0 2024-09-17T04:03:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.492+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20140b3ab0 con 0x7f20140b6db0 2024-09-17T04:03:15.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.492+0000 7f200a000640 1 -- 172.21.15.129:0/1631824484 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f2004084240 con 0x7f20140b6db0 2024-09-17T04:03:15.494 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:15.494 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:15.494 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:15.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.495+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 msgr2=0x7f1ff407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:15.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.495+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 0x7f1ff407b0a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f20141ea660 tx=0x7f2000009040 comp rx=0 tx=0).stop 2024-09-17T04:03:15.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.495+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 msgr2=0x7f20141609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.495+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f2004009780 tx=0x7f2004033c20 comp rx=0 tx=0).stop 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 shutdown_connections 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1ff4078b90 0x7f1ff407b0a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f201415ef60 0x7f201414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f20140d00e0 0x7f20140cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 --2- 172.21.15.129:0/1631824484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20140b6db0 0x7f20141609b0 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 >> 172.21.15.129:0/1631824484 conn(0x7f20140d4a70 msgr2=0x7f20140b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:15.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 shutdown_connections 2024-09-17T04:03:15.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:15.496+0000 7f2018e00640 1 -- 172.21.15.129:0/1631824484 wait complete. 2024-09-17T04:03:15.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:15.925 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1216,"msgr_send_messages":1210,"msgr_recv_bytes":442967,"msgr_send_bytes":671020,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.205053170,"msgr_running_send_time":0.099035905,"msgr_running_recv_time":0.041965816,"msgr_running_fast_dispatch_time":0.055060713,"msgr_send_messages_queue_lat":{"avgcount":1210,"sum":2.303270630,"avgtime":0.001903529},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127008,"msgr_send_encrypted_bytes":17856},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9825,"msgr_send_messages":14798,"msgr_recv_bytes":3181947,"msgr_send_bytes":11893082,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.012593762,"msgr_running_send_time":1.427787838,"msgr_running_recv_time":0.862274182,"msgr_running_fast_dispatch_time":0.100991213,"msgr_send_messages_queue_lat":{"avgcount":14796,"sum":2.783762690,"avgtime":0.000188142},"msgr_handle_ack_lat":{"avgcount":8377,"sum":0.004289173,"avgtime":0.000000512},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":85568},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2494,"msgr_send_messages":2497,"msgr_recv_bytes":806323,"msgr_send_bytes":9531118,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.493462412,"msgr_running_send_time":0.274121171,"msgr_running_recv_time":0.077379757,"msgr_running_fast_dispatch_time":0.118268668,"msgr_send_messages_queue_lat":{"avgcount":2494,"sum":2.866083342,"avgtime":0.001149191},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3099,"sum":3.014496616,"avgtime":0.000972732}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4512,"sum":0.542619006,"avgtime":0.000120261}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":90,"caps":90,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4877,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4407,"handle_client_caps_dirty":1500,"handle_client_cap_release":147,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":88,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1513,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9293,"evex":8670,"evtrm":8670,"ev":626,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":22085429,"rdpos":4195244,"jlat":{"avgcount":1531,"sum":7.065134873,"avgtime":0.004614719},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":38,"cap+":1602,"cap-":1564,"rss":141788,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1323,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1178285,"buffer_anon_items":488,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":958576,"mds_co_items":16821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5642,"op_send_bytes":18394546,"op_resend":0,"op_reply":5642,"op_latency":{"avgcount":5642,"sum":74.821385465,"avgtime":0.013261500},"op_inflight":0,"oplen_avg":{"avgcount":5642,"sum":6654},"op":5642,"op_r":59,"op_w":5583,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":31,"osdop_write":1554,"osdop_writefull":21,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3510,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":126,"omap_rd":48,"omap_del":49},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3491,"pq_executed":1513,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13528,"get_sum":3334447,"get_or_fail_fail":0,"get_or_fail_success":13528,"take":0,"take_sum":0,"put":13528,"put_sum":3334447,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5642,"take_sum":18548294,"put":5642,"put_sum":18548294,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5642,"take_sum":5642,"put":5642,"put_sum":5642,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1513,"get_sum":152813,"get_or_fail_fail":0,"get_or_fail_success":1513,"take":0,"take_sum":0,"put":23,"put_sum":152813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":149655,"max":3758096384,"get_started":0,"get":9293,"get_sum":17889926,"get_or_fail_fail":0,"get_or_fail_success":9293,"take":0,"take_sum":0,"put":1531,"put_sum":17740271,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:15.926 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 442967, "msgr_recv_encrypted_bytes": 127008, "msgr_recv_messages": 1216, "msgr_running_fast_dispatch_time": 0.055060713, "msgr_running_recv_time": 0.041965816, "msgr_running_send_time": 0.099035905, "msgr_running_total_time": 0.20505317, "msgr_send_bytes": 671020, "msgr_send_encrypted_bytes": 17856, "msgr_send_messages": 1210, "msgr_send_messages_queue_lat": { "avgcount": 1210, "avgtime": 0.001903529, "sum": 2.30327063 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8377, "avgtime": 5.12e-07, "sum": 0.004289173 }, "msgr_recv_bytes": 3181947, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9825, "msgr_running_fast_dispatch_time": 0.100991213, "msgr_running_recv_time": 0.862274182, "msgr_running_send_time": 1.427787838, "msgr_running_total_time": 2.012593762, "msgr_send_bytes": 11893082, "msgr_send_encrypted_bytes": 85568, "msgr_send_messages": 14798, "msgr_send_messages_queue_lat": { "avgcount": 14796, "avgtime": 0.000188142, "sum": 2.78376269 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 806323, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2494, "msgr_running_fast_dispatch_time": 0.118268668, "msgr_running_recv_time": 0.077379757, "msgr_running_send_time": 0.274121171, "msgr_running_total_time": 0.493462412, "msgr_send_bytes": 9531118, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2497, "msgr_send_messages_queue_lat": { "avgcount": 2494, "avgtime": 0.001149191, "sum": 2.866083342 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3099, "avgtime": 0.000972732, "sum": 3.014496616 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4512, "avgtime": 0.000120261, "sum": 0.542619006 }, "queue_len": 0 }, "mds": { "caps": 90, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4877, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 147, "handle_client_caps": 4407, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 90, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 88, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1513, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 626, "evadd": 9293, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1531, "avgtime": 0.004614719, "sum": 7.065134873 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 22085429 }, "mds_mem": { "cap": 38, "cap+": 1602, "cap-": 1564, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 141788 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1323, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1178285, "buffer_anon_items": 488, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 958576, "mds_co_items": 16821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 49, "omap_rd": 48, "omap_wr": 126, "op": 5642, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5642, "avgtime": 0.0132615, "sum": 74.821385465 }, "op_pg": 0, "op_r": 59, "op_reply": 5642, "op_resend": 0, "op_rmw": 0, "op_send": 5642, "op_send_bytes": 18394546, "op_w": 5583, "oplen_avg": { "avgcount": 5642, "sum": 6654 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3510, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1554, "osdop_writefull": 21, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1513, "pq_executed_ops": 3491, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13528, "get_or_fail_fail": 0, "get_or_fail_success": 13528, "get_started": 0, "get_sum": 3334447, "max": 104857600, "put": 13528, "put_sum": 3334447, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5642, "put_sum": 18548294, "take": 5642, "take_sum": 18548294, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5642, "put_sum": 5642, "take": 5642, "take_sum": 5642, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1513, "get_or_fail_fail": 0, "get_or_fail_success": 1513, "get_started": 0, "get_sum": 152813, "max": 3758096384, "put": 23, "put_sum": 152813, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9293, "get_or_fail_fail": 0, "get_or_fail_success": 9293, "get_started": 0, "get_sum": 17889926, "max": 3758096384, "put": 1531, "put_sum": 17740271, "take": 0, "take_sum": 0, "val": 149655, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:15.926 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/88, 1513/1601 2024-09-17T04:03:16.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.151+0000 7f10f3600640 1 -- 172.21.15.129:0/1449624846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec0b59d0 msgr2=0x7f10ec0b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.151+0000 7f10f3600640 1 --2- 172.21.15.129:0/1449624846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5de0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f10e4005fd0 tx=0x7f10e402a9f0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 -- 172.21.15.129:0/1449624846 shutdown_connections 2024-09-17T04:03:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 --2- 172.21.15.129:0/1449624846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5de0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 --2- 172.21.15.129:0/1449624846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec15cf70 0x7f10ec072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 -- 172.21.15.129:0/1449624846 >> 172.21.15.129:0/1449624846 conn(0x7f10ec0b23c0 msgr2=0x7f10ec0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:17.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 -- 172.21.15.129:0/1449624846 shutdown_connections 2024-09-17T04:03:17.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.152+0000 7f10f3600640 1 -- 172.21.15.129:0/1449624846 wait complete. 2024-09-17T04:03:17.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.154+0000 7f10f3600640 1 Processor -- start 2024-09-17T04:03:17.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.154+0000 7f10f3600640 1 -- start start 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec0cd190 0x7f10ec15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec06fe50 con 0x7f10ec15cf70 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec06fcd0 con 0x7f10ec0b59d0 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.155+0000 7f10f3600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10ec06ffd0 con 0x7f10ec0cd190 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47192/0 (socket says 172.21.15.129:47192) 2024-09-17T04:03:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 -- 172.21.15.129:0/2674220412 learned_addr learned my addr 172.21.15.129:0/2674220412 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10ebe00640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec0cd190 0x7f10ec15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 -- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec0cd190 msgr2=0x7f10ec15c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec0cd190 0x7f10ec15c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 -- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec0b59d0 msgr2=0x7f10ec0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.156+0000 7f10f1600640 1 -- 172.21.15.129:0/2674220412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec1f9d00 con 0x7f10ec15cf70 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10f1600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009d90 tx=0x7f10e4009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4033050 con 0x7f10ec15cf70 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10e4004450 con 0x7f10ec15cf70 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ec0b4870 con 0x7f10ec15cf70 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4004840 con 0x7f10ec15cf70 2024-09-17T04:03:17.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.157+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ec0b4e30 con 0x7f10ec15cf70 2024-09-17T04:03:17.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.158+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec0704c0 con 0x7f10ec15cf70 2024-09-17T04:03:17.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.159+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10e4036070 con 0x7f10ec15cf70 2024-09-17T04:03:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.160+0000 7f10e8c00640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 0x7f10cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.160+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f10e40bbfd0 con 0x7f10ec15cf70 2024-09-17T04:03:17.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.161+0000 7f10f0c00640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 0x7f10cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:17.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.161+0000 7f10f0c00640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 0x7f10cc07b0a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f10e0001720 tx=0x7f10e0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:17.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.165+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10e4083cd0 con 0x7f10ec15cf70 2024-09-17T04:03:17.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.380+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10ec069b10 con 0x7f10ec15cf70 2024-09-17T04:03:17.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.381+0000 7f10e8c00640 1 -- 172.21.15.129:0/2674220412 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f10e4088700 con 0x7f10ec15cf70 2024-09-17T04:03:17.381 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:17.381 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:17.382 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.383+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 msgr2=0x7f10cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.383+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 0x7f10cc07b0a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f10e0001720 tx=0x7f10e0009040 comp rx=0 tx=0).stop 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.383+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 msgr2=0x7f10ec1f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.383+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f10e4009d90 tx=0x7f10e4009360 comp rx=0 tx=0).stop 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.383+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 shutdown_connections 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10cc078b90 0x7f10cc07b0a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10ec15cf70 0x7f10ec1f8e30 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10ec0cd190 0x7f10ec15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 --2- 172.21.15.129:0/2674220412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10ec0b59d0 0x7f10ec0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:17.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 >> 172.21.15.129:0/2674220412 conn(0x7f10ec0b23c0 msgr2=0x7f10ec0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:17.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 shutdown_connections 2024-09-17T04:03:17.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:17.384+0000 7f10f3600640 1 -- 172.21.15.129:0/2674220412 wait complete. 2024-09-17T04:03:17.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:17.646 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1219,"msgr_send_messages":1213,"msgr_recv_bytes":443823,"msgr_send_bytes":673384,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.205603676,"msgr_running_send_time":0.099318894,"msgr_running_recv_time":0.042105536,"msgr_running_fast_dispatch_time":0.055157594,"msgr_send_messages_queue_lat":{"avgcount":1213,"sum":2.303519672,"avgtime":0.001899026},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127280,"msgr_send_encrypted_bytes":18496},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9890,"msgr_send_messages":14831,"msgr_recv_bytes":3198897,"msgr_send_bytes":11904401,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.029409964,"msgr_running_send_time":1.439542076,"msgr_running_recv_time":0.869051115,"msgr_running_fast_dispatch_time":0.101020816,"msgr_send_messages_queue_lat":{"avgcount":14829,"sum":2.807085219,"avgtime":0.000189297},"msgr_handle_ack_lat":{"avgcount":8396,"sum":0.004308616,"avgtime":0.000000513},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":85568},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2494,"msgr_send_messages":2497,"msgr_recv_bytes":806323,"msgr_send_bytes":9531118,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.493462412,"msgr_running_send_time":0.274121171,"msgr_running_recv_time":0.077379757,"msgr_running_fast_dispatch_time":0.118268668,"msgr_send_messages_queue_lat":{"avgcount":2494,"sum":2.866083342,"avgtime":0.001149191},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3101,"sum":3.014526484,"avgtime":0.000972114}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4512,"sum":0.542619006,"avgtime":0.000120261}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":90,"caps":90,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4910,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4444,"handle_client_caps_dirty":1500,"handle_client_cap_release":174,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":88,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1513,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9330,"evex":8670,"evtrm":8670,"ev":663,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":22159847,"rdpos":4195244,"jlat":{"avgcount":1531,"sum":7.065134873,"avgtime":0.004614719},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":1,"cap+":1602,"cap-":1601,"rss":142300,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1323,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1178285,"buffer_anon_items":488,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":958576,"mds_co_items":16821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5644,"op_send_bytes":18395550,"op_resend":0,"op_reply":5644,"op_latency":{"avgcount":5644,"sum":74.830385270,"avgtime":0.013258395},"op_inflight":0,"oplen_avg":{"avgcount":5644,"sum":6658},"op":5644,"op_r":59,"op_w":5585,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":31,"osdop_write":1554,"osdop_writefull":21,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3510,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":128,"omap_rd":48,"omap_del":51},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3491,"pq_executed":1513,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13596,"get_sum":3347121,"get_or_fail_fail":0,"get_or_fail_success":13596,"take":0,"take_sum":0,"put":13596,"put_sum":3347121,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5644,"take_sum":18549298,"put":5644,"put_sum":18549298,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5644,"take_sum":5644,"put":5644,"put_sum":5644,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1513,"get_sum":152813,"get_or_fail_fail":0,"get_or_fail_success":1513,"take":0,"take_sum":0,"put":23,"put_sum":152813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":224073,"max":3758096384,"get_started":0,"get":9330,"get_sum":17964344,"get_or_fail_fail":0,"get_or_fail_success":9330,"take":0,"take_sum":0,"put":1531,"put_sum":17740271,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:17.648 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 443823, "msgr_recv_encrypted_bytes": 127280, "msgr_recv_messages": 1219, "msgr_running_fast_dispatch_time": 0.055157594, "msgr_running_recv_time": 0.042105536, "msgr_running_send_time": 0.099318894, "msgr_running_total_time": 0.205603676, "msgr_send_bytes": 673384, "msgr_send_encrypted_bytes": 18496, "msgr_send_messages": 1213, "msgr_send_messages_queue_lat": { "avgcount": 1213, "avgtime": 0.001899026, "sum": 2.303519672 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8396, "avgtime": 5.13e-07, "sum": 0.004308616 }, "msgr_recv_bytes": 3198897, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9890, "msgr_running_fast_dispatch_time": 0.101020816, "msgr_running_recv_time": 0.869051115, "msgr_running_send_time": 1.439542076, "msgr_running_total_time": 2.029409964, "msgr_send_bytes": 11904401, "msgr_send_encrypted_bytes": 85568, "msgr_send_messages": 14831, "msgr_send_messages_queue_lat": { "avgcount": 14829, "avgtime": 0.000189297, "sum": 2.807085219 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 806323, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2494, "msgr_running_fast_dispatch_time": 0.118268668, "msgr_running_recv_time": 0.077379757, "msgr_running_send_time": 0.274121171, "msgr_running_total_time": 0.493462412, "msgr_send_bytes": 9531118, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2497, "msgr_send_messages_queue_lat": { "avgcount": 2494, "avgtime": 0.001149191, "sum": 2.866083342 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3101, "avgtime": 0.000972114, "sum": 3.014526484 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4512, "avgtime": 0.000120261, "sum": 0.542619006 }, "queue_len": 0 }, "mds": { "caps": 90, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4910, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 174, "handle_client_caps": 4444, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 90, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 88, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1513, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 663, "evadd": 9330, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1531, "avgtime": 0.004614719, "sum": 7.065134873 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 22159847 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 142300 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1323, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1178285, "buffer_anon_items": 488, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 958576, "mds_co_items": 16821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 51, "omap_rd": 48, "omap_wr": 128, "op": 5644, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5644, "avgtime": 0.013258395, "sum": 74.83038527 }, "op_pg": 0, "op_r": 59, "op_reply": 5644, "op_resend": 0, "op_rmw": 0, "op_send": 5644, "op_send_bytes": 18395550, "op_w": 5585, "oplen_avg": { "avgcount": 5644, "sum": 6658 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3510, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1554, "osdop_writefull": 21, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1513, "pq_executed_ops": 3491, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13596, "get_or_fail_fail": 0, "get_or_fail_success": 13596, "get_started": 0, "get_sum": 3347121, "max": 104857600, "put": 13596, "put_sum": 3347121, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5644, "put_sum": 18549298, "take": 5644, "take_sum": 18549298, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5644, "put_sum": 5644, "take": 5644, "take_sum": 5644, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1513, "get_or_fail_fail": 0, "get_or_fail_success": 1513, "get_started": 0, "get_sum": 152813, "max": 3758096384, "put": 23, "put_sum": 152813, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9330, "get_or_fail_fail": 0, "get_or_fail_success": 9330, "get_started": 0, "get_sum": 17964344, "max": 3758096384, "put": 1531, "put_sum": 17740271, "take": 0, "take_sum": 0, "val": 224073, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:17.649 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/88, 1513/1601 2024-09-17T04:03:18.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:18.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:03:18.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:03:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.892+0000 7f4570c00640 1 -- 172.21.15.129:0/2613657869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 msgr2=0x7f456c142370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.892+0000 7f4570c00640 1 --2- 172.21.15.129:0/2613657869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c142370 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f4564005fd0 tx=0x7f456402aa90 comp rx=0 tx=0).stop 2024-09-17T04:03:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 -- 172.21.15.129:0/2613657869 shutdown_connections 2024-09-17T04:03:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 --2- 172.21.15.129:0/2613657869 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c142370 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:18.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 --2- 172.21.15.129:0/2613657869 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 0x7f456c06e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 -- 172.21.15.129:0/2613657869 >> 172.21.15.129:0/2613657869 conn(0x7f456c06d560 msgr2=0x7f456c0b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 -- 172.21.15.129:0/2613657869 shutdown_connections 2024-09-17T04:03:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.893+0000 7f4570c00640 1 -- 172.21.15.129:0/2613657869 wait complete. 2024-09-17T04:03:18.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.895+0000 7f4570c00640 1 Processor -- start 2024-09-17T04:03:18.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.895+0000 7f4570c00640 1 -- start start 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 0x7f456c143590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f456c15da40 0x7f456c1e4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f456c074ec0 con 0x7f456c141f60 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f456c074d40 con 0x7f456c15da40 2024-09-17T04:03:18.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.896+0000 7f4570c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f456c075040 con 0x7f456c156f90 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47200/0 (socket says 172.21.15.129:47200) 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 -- 172.21.15.129:0/1459638229 learned_addr learned my addr 172.21.15.129:0/1459638229 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f4562a00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 0x7f456c143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456be00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f456c15da40 0x7f456c1e4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 -- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 msgr2=0x7f456c143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 0x7f456c143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 -- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f456c15da40 msgr2=0x7f456c1e4be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:18.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f456c15da40 0x7f456c1e4be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.897+0000 7f456b400640 1 -- 172.21.15.129:0/1459638229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456c0cfee0 con 0x7f456c141f60 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f456b400640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f455c00c3c0 tx=0x7f455c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c017070 con 0x7f456c141f60 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f455c0035f0 con 0x7f456c141f60 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455c003a40 con 0x7f456c141f60 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f456c15c420 con 0x7f456c141f60 2024-09-17T04:03:18.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.898+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f456c1e4100 con 0x7f456c141f60 2024-09-17T04:03:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.899+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f456c075560 con 0x7f456c141f60 2024-09-17T04:03:18.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.900+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f455c01a070 con 0x7f456c141f60 2024-09-17T04:03:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.901+0000 7f4563e00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 0x7f454807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.901+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f455c09a210 con 0x7f456c141f60 2024-09-17T04:03:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.902+0000 7f4562a00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 0x7f454807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:18.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.902+0000 7f4562a00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 0x7f454807b0a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f456c0d0e60 tx=0x7f4554005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:18.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:18.905+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f456c1428b0 con 0x7f456c141f60 2024-09-17T04:03:19.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.121+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f456c142370 con 0x7f456c141f60 2024-09-17T04:03:19.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.121+0000 7f4563e00640 1 -- 172.21.15.129:0/1459638229 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f455c061df0 con 0x7f456c141f60 2024-09-17T04:03:19.121 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:19.122 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:19.122 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.123+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 msgr2=0x7f454807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.123+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 0x7f454807b0a0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f456c0d0e60 tx=0x7f4554005f90 comp rx=0 tx=0).stop 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.123+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 msgr2=0x7f456c143050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.123+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f455c00c3c0 tx=0x7f455c00c880 comp rx=0 tx=0).stop 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 shutdown_connections 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4548078b90 0x7f454807b0a0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f456c15da40 0x7f456c1e4be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:19.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f456c156f90 0x7f456c143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 --2- 172.21.15.129:0/1459638229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f456c141f60 0x7f456c143050 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 >> 172.21.15.129:0/1459638229 conn(0x7f456c06d560 msgr2=0x7f456c154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 shutdown_connections 2024-09-17T04:03:19.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:19.124+0000 7f4570c00640 1 -- 172.21.15.129:0/1459638229 wait complete. 2024-09-17T04:03:19.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:19.388 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1219,"msgr_send_messages":1213,"msgr_recv_bytes":443823,"msgr_send_bytes":673384,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.205603676,"msgr_running_send_time":0.099318894,"msgr_running_recv_time":0.042105536,"msgr_running_fast_dispatch_time":0.055157594,"msgr_send_messages_queue_lat":{"avgcount":1213,"sum":2.303519672,"avgtime":0.001899026},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127280,"msgr_send_encrypted_bytes":18496},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9892,"msgr_send_messages":14831,"msgr_recv_bytes":3199701,"msgr_send_bytes":11904401,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.029756986,"msgr_running_send_time":1.439705710,"msgr_running_recv_time":0.869141972,"msgr_running_fast_dispatch_time":0.101103143,"msgr_send_messages_queue_lat":{"avgcount":14829,"sum":2.807085219,"avgtime":0.000189297},"msgr_handle_ack_lat":{"avgcount":8396,"sum":0.004308616,"avgtime":0.000000513},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":85568},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2494,"msgr_send_messages":2497,"msgr_recv_bytes":806323,"msgr_send_bytes":9531118,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.493462412,"msgr_running_send_time":0.274121171,"msgr_running_recv_time":0.077379757,"msgr_running_fast_dispatch_time":0.118268668,"msgr_send_messages_queue_lat":{"avgcount":2494,"sum":2.866083342,"avgtime":0.001149191},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3101,"sum":3.014526484,"avgtime":0.000972114}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4512,"sum":0.542619006,"avgtime":0.000120261}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":90,"caps":90,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4910,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4444,"handle_client_caps_dirty":1500,"handle_client_cap_release":174,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":88,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1513,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9330,"evex":8670,"evtrm":8670,"ev":663,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":22159847,"rdpos":4195244,"jlat":{"avgcount":1531,"sum":7.065134873,"avgtime":0.004614719},"replayed":3},"mds_mem":{"ino":179,"ino+":1614,"ino-":1435,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":1,"cap+":1602,"cap-":1601,"rss":142300,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1323,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1178285,"buffer_anon_items":488,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":958576,"mds_co_items":16821,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5644,"op_send_bytes":18395550,"op_resend":0,"op_reply":5644,"op_latency":{"avgcount":5644,"sum":74.830385270,"avgtime":0.013258395},"op_inflight":0,"oplen_avg":{"avgcount":5644,"sum":6658},"op":5644,"op_r":59,"op_w":5585,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":31,"osdop_write":1554,"osdop_writefull":21,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3510,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":128,"omap_rd":48,"omap_del":51},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3491,"pq_executed":1513,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13598,"get_sum":3347777,"get_or_fail_fail":0,"get_or_fail_success":13598,"take":0,"take_sum":0,"put":13598,"put_sum":3347777,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5644,"take_sum":18549298,"put":5644,"put_sum":18549298,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5644,"take_sum":5644,"put":5644,"put_sum":5644,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1513,"get_sum":152813,"get_or_fail_fail":0,"get_or_fail_success":1513,"take":0,"take_sum":0,"put":23,"put_sum":152813,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":224073,"max":3758096384,"get_started":0,"get":9330,"get_sum":17964344,"get_or_fail_fail":0,"get_or_fail_success":9330,"take":0,"take_sum":0,"put":1531,"put_sum":17740271,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:19.389 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 443823, "msgr_recv_encrypted_bytes": 127280, "msgr_recv_messages": 1219, "msgr_running_fast_dispatch_time": 0.055157594, "msgr_running_recv_time": 0.042105536, "msgr_running_send_time": 0.099318894, "msgr_running_total_time": 0.205603676, "msgr_send_bytes": 673384, "msgr_send_encrypted_bytes": 18496, "msgr_send_messages": 1213, "msgr_send_messages_queue_lat": { "avgcount": 1213, "avgtime": 0.001899026, "sum": 2.303519672 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8396, "avgtime": 5.13e-07, "sum": 0.004308616 }, "msgr_recv_bytes": 3199701, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9892, "msgr_running_fast_dispatch_time": 0.101103143, "msgr_running_recv_time": 0.869141972, "msgr_running_send_time": 1.43970571, "msgr_running_total_time": 2.029756986, "msgr_send_bytes": 11904401, "msgr_send_encrypted_bytes": 85568, "msgr_send_messages": 14831, "msgr_send_messages_queue_lat": { "avgcount": 14829, "avgtime": 0.000189297, "sum": 2.807085219 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 806323, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2494, "msgr_running_fast_dispatch_time": 0.118268668, "msgr_running_recv_time": 0.077379757, "msgr_running_send_time": 0.274121171, "msgr_running_total_time": 0.493462412, "msgr_send_bytes": 9531118, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2497, "msgr_send_messages_queue_lat": { "avgcount": 2494, "avgtime": 0.001149191, "sum": 2.866083342 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3101, "avgtime": 0.000972114, "sum": 3.014526484 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4512, "avgtime": 0.000120261, "sum": 0.542619006 }, "queue_len": 0 }, "mds": { "caps": 90, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4910, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 174, "handle_client_caps": 4444, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 90, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 88, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1513, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 663, "evadd": 9330, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1531, "avgtime": 0.004614719, "sum": 7.065134873 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 22159847 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 179, "ino+": 1614, "ino-": 1435, "rss": 142300 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1323, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1178285, "buffer_anon_items": 488, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 958576, "mds_co_items": 16821, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 51, "omap_rd": 48, "omap_wr": 128, "op": 5644, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5644, "avgtime": 0.013258395, "sum": 74.83038527 }, "op_pg": 0, "op_r": 59, "op_reply": 5644, "op_resend": 0, "op_rmw": 0, "op_send": 5644, "op_send_bytes": 18395550, "op_w": 5585, "oplen_avg": { "avgcount": 5644, "sum": 6658 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3510, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 31, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1554, "osdop_writefull": 21, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1513, "pq_executed_ops": 3491, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13598, "get_or_fail_fail": 0, "get_or_fail_success": 13598, "get_started": 0, "get_sum": 3347777, "max": 104857600, "put": 13598, "put_sum": 3347777, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5644, "put_sum": 18549298, "take": 5644, "take_sum": 18549298, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5644, "put_sum": 5644, "take": 5644, "take_sum": 5644, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1513, "get_or_fail_fail": 0, "get_or_fail_success": 1513, "get_started": 0, "get_sum": 152813, "max": 3758096384, "put": 23, "put_sum": 152813, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9330, "get_or_fail_fail": 0, "get_or_fail_success": 9330, "get_started": 0, "get_sum": 17964344, "max": 3758096384, "put": 1531, "put_sum": 17740271, "take": 0, "take_sum": 0, "val": 224073, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:19.390 INFO:tasks.cephfs.test_strays:Waiting for purge to complete 0/88, 1513/1601 2024-09-17T04:03:20.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 -- 172.21.15.129:0/3559426780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 msgr2=0x7f66280d2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 --2- 172.21.15.129:0/3559426780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66280d2290 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f66280d0320 tx=0x7f661802a560 comp rx=0 tx=0).stop 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 -- 172.21.15.129:0/3559426780 shutdown_connections 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 --2- 172.21.15.129:0/3559426780 >> v2:172.21.15.181:3300/0 conn(0x7f662806d270 0x7f662806d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 --2- 172.21.15.129:0/3559426780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66280d2290 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 -- 172.21.15.129:0/3559426780 >> 172.21.15.129:0/3559426780 conn(0x7f66280d1770 msgr2=0x7f66280d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 -- 172.21.15.129:0/3559426780 shutdown_connections 2024-09-17T04:03:20.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.620+0000 7f6627e00640 1 -- 172.21.15.129:0/3559426780 wait complete. 2024-09-17T04:03:20.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.622+0000 7f6627e00640 1 Processor -- start 2024-09-17T04:03:20.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.622+0000 7f6627e00640 1 -- start start 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f662806d270 0x7f66281f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 0x7f662814a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f66280d3be0 con 0x7f66280cf2e0 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f66280d3a60 con 0x7f662815ea40 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f66280d3d60 con 0x7f662806d270 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f661e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f661e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47226/0 (socket says 172.21.15.129:47226) 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f661e000640 1 -- 172.21.15.129:0/2905663795 learned_addr learned my addr 172.21.15.129:0/2905663795 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6626a00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f662806d270 0x7f66281f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.624+0000 7f6627400640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 0x7f662814a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 -- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f662806d270 msgr2=0x7f66281f6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f662806d270 0x7f66281f6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 -- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 msgr2=0x7f662814a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 0x7f662814a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 -- 172.21.15.129:0/2905663795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66281f6990 con 0x7f66280cf2e0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f6627400640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 0x7f662814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661e000640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f661000b3f0 tx=0x7f661000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610015020 con 0x7f66280cf2e0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6610011920 con 0x7f66280cf2e0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610011d70 con 0x7f66280cf2e0 2024-09-17T04:03:20.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66281e7830 con 0x7f66280cf2e0 2024-09-17T04:03:20.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.625+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66280c99a0 con 0x7f66280cf2e0 2024-09-17T04:03:20.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.626+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66280d4230 con 0x7f66280cf2e0 2024-09-17T04:03:20.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.628+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6610018020 con 0x7f66280cf2e0 2024-09-17T04:03:20.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.629+0000 7f661fe00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 0x7f660407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:20.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.629+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f661009a950 con 0x7f66280cf2e0 2024-09-17T04:03:20.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.629+0000 7f6626a00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 0x7f660407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:20.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.630+0000 7f6626a00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 0x7f660407b0a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f6618033000 tx=0x7f6618004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:20.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.633+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f661001f030 con 0x7f66280cf2e0 2024-09-17T04:03:20.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.852+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f662806de30 con 0x7f66280cf2e0 2024-09-17T04:03:20.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.853+0000 7f661fe00640 1 -- 172.21.15.129:0/2905663795 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f6610062d20 con 0x7f66280cf2e0 2024-09-17T04:03:20.853 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:20.853 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:20.854 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.855+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 msgr2=0x7f660407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.855+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 0x7f660407b0a0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f6618033000 tx=0x7f6618004730 comp rx=0 tx=0).stop 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.855+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 msgr2=0x7f66281f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.855+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f661000b3f0 tx=0x7f661000b8b0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.855+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 shutdown_connections 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6604078b90 0x7f660407b0a0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f662815ea40 0x7f662814a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f66280cf2e0 0x7f66281f91f0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 --2- 172.21.15.129:0/2905663795 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f662806d270 0x7f66281f6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:20.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 >> 172.21.15.129:0/2905663795 conn(0x7f66280d1770 msgr2=0x7f66281613e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:20.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 shutdown_connections 2024-09-17T04:03:20.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:20.856+0000 7f6627e00640 1 -- 172.21.15.129:0/2905663795 wait complete. 2024-09-17T04:03:20.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:21.119 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1272,"msgr_send_messages":1266,"msgr_recv_bytes":457285,"msgr_send_bytes":690662,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.210661732,"msgr_running_send_time":0.101367974,"msgr_running_recv_time":0.043369364,"msgr_running_fast_dispatch_time":0.056593546,"msgr_send_messages_queue_lat":{"avgcount":1266,"sum":2.308207388,"avgtime":0.001823228},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127280,"msgr_send_encrypted_bytes":18496},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9957,"msgr_send_messages":14895,"msgr_recv_bytes":3216491,"msgr_send_bytes":12151974,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.038399874,"msgr_running_send_time":1.444170464,"msgr_running_recv_time":0.870747115,"msgr_running_fast_dispatch_time":0.103043267,"msgr_send_messages_queue_lat":{"avgcount":14893,"sum":2.822509848,"avgtime":0.000189519},"msgr_handle_ack_lat":{"avgcount":8396,"sum":0.004308616,"avgtime":0.000000513},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":87408},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2599,"msgr_send_messages":2602,"msgr_recv_bytes":841841,"msgr_send_bytes":9574286,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.505408573,"msgr_running_send_time":0.280365450,"msgr_running_recv_time":0.079814365,"msgr_running_fast_dispatch_time":0.120841810,"msgr_send_messages_queue_lat":{"avgcount":2599,"sum":2.878654983,"avgtime":0.001107600},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3105,"sum":3.041231009,"avgtime":0.000979462}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4823,"sum":0.553110835,"avgtime":0.000114681}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4910,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4444,"handle_client_caps_dirty":1500,"handle_client_cap_release":174,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1601,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9419,"evex":8670,"evtrm":8670,"ev":752,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":22213697,"rdpos":4195244,"jlat":{"avgcount":1533,"sum":7.072556477,"avgtime":0.004613539},"replayed":3},"mds_mem":{"ino":101,"ino+":1614,"ino-":1513,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":1,"cap+":1602,"cap-":1601,"rss":141788,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1328,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1947821,"buffer_anon_items":680,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":975136,"mds_co_items":17104,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5865,"op_send_bytes":18629739,"op_resend":0,"op_reply":5865,"op_latency":{"avgcount":5865,"sum":76.168356398,"avgtime":0.012986932},"op_inflight":0,"oplen_avg":{"avgcount":5865,"sum":6879},"op":5865,"op_r":61,"op_w":5804,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":33,"osdop_write":1558,"osdop_writefull":22,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3724,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":128,"omap_rd":48,"omap_del":51},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3705,"pq_executed":1601,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13821,"get_sum":3393509,"get_or_fail_fail":0,"get_or_fail_success":13821,"take":0,"take_sum":0,"put":13821,"put_sum":3393509,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5865,"take_sum":18792375,"put":5865,"put_sum":18792375,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5865,"take_sum":5865,"put":5865,"put_sum":5865,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1601,"get_sum":161701,"get_or_fail_fail":0,"get_or_fail_success":1601,"take":0,"take_sum":0,"put":25,"put_sum":161701,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":52712,"max":3758096384,"get_started":0,"get":9419,"get_sum":18018194,"get_or_fail_fail":0,"get_or_fail_success":9419,"take":0,"take_sum":0,"put":1533,"put_sum":17965482,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:21.121 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 457285, "msgr_recv_encrypted_bytes": 127280, "msgr_recv_messages": 1272, "msgr_running_fast_dispatch_time": 0.056593546, "msgr_running_recv_time": 0.043369364, "msgr_running_send_time": 0.101367974, "msgr_running_total_time": 0.210661732, "msgr_send_bytes": 690662, "msgr_send_encrypted_bytes": 18496, "msgr_send_messages": 1266, "msgr_send_messages_queue_lat": { "avgcount": 1266, "avgtime": 0.001823228, "sum": 2.308207388 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8396, "avgtime": 5.13e-07, "sum": 0.004308616 }, "msgr_recv_bytes": 3216491, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9957, "msgr_running_fast_dispatch_time": 0.103043267, "msgr_running_recv_time": 0.870747115, "msgr_running_send_time": 1.444170464, "msgr_running_total_time": 2.038399874, "msgr_send_bytes": 12151974, "msgr_send_encrypted_bytes": 87408, "msgr_send_messages": 14895, "msgr_send_messages_queue_lat": { "avgcount": 14893, "avgtime": 0.000189519, "sum": 2.822509848 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 841841, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2599, "msgr_running_fast_dispatch_time": 0.12084181, "msgr_running_recv_time": 0.079814365, "msgr_running_send_time": 0.28036545, "msgr_running_total_time": 0.505408573, "msgr_send_bytes": 9574286, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2602, "msgr_send_messages_queue_lat": { "avgcount": 2599, "avgtime": 0.0011076, "sum": 2.878654983 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3105, "avgtime": 0.000979462, "sum": 3.041231009 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4823, "avgtime": 0.000114681, "sum": 0.553110835 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4910, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 174, "handle_client_caps": 4444, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1601, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 752, "evadd": 9419, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1533, "avgtime": 0.004613539, "sum": 7.072556477 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 22213697 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 101, "ino+": 1614, "ino-": 1513, "rss": 141788 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1328, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1947821, "buffer_anon_items": 680, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 975136, "mds_co_items": 17104, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 51, "omap_rd": 48, "omap_wr": 128, "op": 5865, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5865, "avgtime": 0.012986932, "sum": 76.168356398 }, "op_pg": 0, "op_r": 61, "op_reply": 5865, "op_resend": 0, "op_rmw": 0, "op_send": 5865, "op_send_bytes": 18629739, "op_w": 5804, "oplen_avg": { "avgcount": 5865, "sum": 6879 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3724, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1558, "osdop_writefull": 22, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1601, "pq_executed_ops": 3705, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13821, "get_or_fail_fail": 0, "get_or_fail_success": 13821, "get_started": 0, "get_sum": 3393509, "max": 104857600, "put": 13821, "put_sum": 3393509, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5865, "put_sum": 18792375, "take": 5865, "take_sum": 18792375, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5865, "put_sum": 5865, "take": 5865, "take_sum": 5865, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1601, "get_or_fail_fail": 0, "get_or_fail_success": 1601, "get_started": 0, "get_sum": 161701, "max": 3758096384, "put": 25, "put_sum": 161701, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9419, "get_or_fail_fail": 0, "get_or_fail_success": 9419, "get_started": 0, "get_sum": 18018194, "max": 3758096384, "put": 1533, "put_sum": 17965482, "take": 0, "take_sum": 0, "val": 52712, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:21.121 INFO:tasks.cephfs.test_strays:Complete purge in 18 seconds 2024-09-17T04:03:21.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 -- 172.21.15.129:0/1985884496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 msgr2=0x7f46fc072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 --2- 172.21.15.129:0/1985884496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc072d70 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f46e4005fd0 tx=0x7f46e402a540 comp rx=0 tx=0).stop 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 -- 172.21.15.129:0/1985884496 shutdown_connections 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 --2- 172.21.15.129:0/1985884496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0b59d0 0x7f46fc0b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 --2- 172.21.15.129:0/1985884496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc072d70 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 -- 172.21.15.129:0/1985884496 >> 172.21.15.129:0/1985884496 conn(0x7f46fc0b23c0 msgr2=0x7f46fc0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 -- 172.21.15.129:0/1985884496 shutdown_connections 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.341+0000 7f4702800640 1 -- 172.21.15.129:0/1985884496 wait complete. 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.344+0000 7f4702800640 1 Processor -- start 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.344+0000 7f4702800640 1 -- start start 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46fc0b59d0 0x7f46fc0b5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0cd190 0x7f46fc0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:21.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc06fe50 con 0x7f46fc15cf70 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc06fcd0 con 0x7f46fc0b59d0 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.345+0000 7f4702800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f46fc06ffd0 con 0x7f46fc0cd190 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47246/0 (socket says 172.21.15.129:47246) 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 -- 172.21.15.129:0/232188826 learned_addr learned my addr 172.21.15.129:0/232188826 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fb400640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46fc0b59d0 0x7f46fc0b5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46f2a00640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0cd190 0x7f46fc0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 -- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0cd190 msgr2=0x7f46fc0aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0cd190 0x7f46fc0aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 -- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46fc0b59d0 msgr2=0x7f46fc0b5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46fc0b59d0 0x7f46fc0b5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 -- 172.21.15.129:0/232188826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46fc0b3860 con 0x7f46fc15cf70 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46fbe00640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f46f400e960 tx=0x7f46f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.346+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4004d20 con 0x7f46fc15cf70 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.347+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f46f400cb20 con 0x7f46fc15cf70 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.347+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46fc0b3b40 con 0x7f46fc15cf70 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.347+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4005020 con 0x7f46fc15cf70 2024-09-17T04:03:21.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.347+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46fc0b4100 con 0x7f46fc15cf70 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.348+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46fc0704c0 con 0x7f46fc15cf70 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.349+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f46f40051c0 con 0x7f46fc15cf70 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.350+0000 7f46f3e00640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 0x7f46d807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.350+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f46f409b760 con 0x7f46fc15cf70 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.350+0000 7f46fb400640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 0x7f46d807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.351+0000 7f46fb400640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 0x7f46d807b0a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f46ec001720 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:21.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.354+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f46f4016030 con 0x7f46fc15cf70 2024-09-17T04:03:21.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.572+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46fc145180 con 0x7f46fc15cf70 2024-09-17T04:03:21.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.572+0000 7f46f3e00640 1 -- 172.21.15.129:0/232188826 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f46f40633c0 con 0x7f46fc15cf70 2024-09-17T04:03:21.606 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:21.607 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.574+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 msgr2=0x7f46d807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 0x7f46d807b0a0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f46ec001720 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 msgr2=0x7f46fc0cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f46f400e960 tx=0x7f46f400ee20 comp rx=0 tx=0).stop 2024-09-17T04:03:21.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 shutdown_connections 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f46d8078b90 0x7f46d807b0a0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f46fc15cf70 0x7f46fc0cba80 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f46fc0cd190 0x7f46fc0aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 --2- 172.21.15.129:0/232188826 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46fc0b59d0 0x7f46fc0b5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 >> 172.21.15.129:0/232188826 conn(0x7f46fc0b23c0 msgr2=0x7f46fc0cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 shutdown_connections 2024-09-17T04:03:21.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:21.575+0000 7f4702800640 1 -- 172.21.15.129:0/232188826 wait complete. 2024-09-17T04:03:21.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump 2024-09-17T04:03:21.849 INFO:teuthology.orchestra.run.smithi129.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":1273,"msgr_send_messages":1267,"msgr_recv_bytes":457557,"msgr_send_bytes":691302,"msgr_created_connections":9,"msgr_active_connections":4,"msgr_running_total_time":0.210833966,"msgr_running_send_time":0.101474140,"msgr_running_recv_time":0.043445608,"msgr_running_fast_dispatch_time":0.056593546,"msgr_send_messages_queue_lat":{"avgcount":1267,"sum":2.308254834,"avgtime":0.001821827},"msgr_handle_ack_lat":{"avgcount":2,"sum":0.000000241,"avgtime":0.000000120},"msgr_recv_encrypted_bytes":127552,"msgr_send_encrypted_bytes":19136},"AsyncMessenger::Worker-1":{"msgr_recv_messages":9958,"msgr_send_messages":14895,"msgr_recv_bytes":3216893,"msgr_send_bytes":12151974,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":2.038545264,"msgr_running_send_time":1.444225965,"msgr_running_recv_time":0.870787363,"msgr_running_fast_dispatch_time":0.103089715,"msgr_send_messages_queue_lat":{"avgcount":14893,"sum":2.822509848,"avgtime":0.000189519},"msgr_handle_ack_lat":{"avgcount":8396,"sum":0.004308616,"avgtime":0.000000513},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":87408},"AsyncMessenger::Worker-2":{"msgr_recv_messages":2599,"msgr_send_messages":2602,"msgr_recv_bytes":841841,"msgr_send_bytes":9574286,"msgr_created_connections":7,"msgr_active_connections":5,"msgr_running_total_time":0.505408573,"msgr_running_send_time":0.280365450,"msgr_running_recv_time":0.079814365,"msgr_running_fast_dispatch_time":0.120841810,"msgr_send_messages_queue_lat":{"avgcount":2599,"sum":2.878654983,"avgtime":0.001107600},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":0,"msgr_send_encrypted_bytes":0},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":3105,"sum":3.041231009,"avgtime":0.000979462}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4823,"sum":0.553110835,"avgtime":0.000114681}},"mds":{"request":3207,"reply":3205,"reply_latency":{"avgcount":3205,"sum":4.674819336,"avgtime":0.001458601},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":12,"dir_split":0,"dir_merge":0,"inodes":431,"inodes_top":0,"inodes_bottom":431,"inodes_pin_tail":0,"inodes_pinned":431,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":3207,"traverse_hit":3207,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":2,"ceph_cap_op_grant":4910,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":4444,"handle_client_caps_dirty":1500,"handle_client_cap_release":174,"process_request_cap_release":1606},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":1601,"strays_enqueued":1601,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0,"uninline_started":0,"uninline_succeeded":0,"uninline_write_failed":0},"mds_log":{"evlrg":0,"evadd":9419,"evex":8670,"evtrm":8670,"ev":752,"evexg":0,"evexd":8669,"segadd":10,"segex":11,"segtrm":11,"seg":1,"segmjr":1,"segexg":0,"segexd":10,"expos":21296659,"wrpos":22213697,"rdpos":4195244,"jlat":{"avgcount":1533,"sum":7.072556477,"avgtime":0.004613539},"replayed":3},"mds_mem":{"ino":101,"ino+":1614,"ino-":1513,"dir":12,"dir+":13,"dir-":1,"dn":431,"dn+":3212,"dn-":2781,"cap":1,"cap+":1602,"cap-":1601,"rss":141788,"heap":394223616},"mds_server":{"dispatch_client_request":3209,"dispatch_server_request":0,"handle_client_request":3207,"handle_client_session":5,"handle_peer_request":0,"req_create_latency":{"avgcount":1600,"sum":2.253621121,"avgtime":0.001408513},"req_getattr_latency":{"avgcount":1,"sum":0.000538302,"avgtime":0.000538302},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":1,"sum":0.003195940,"avgtime":0.003195940},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":2,"sum":0.024033951,"avgtime":0.012016975},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":1,"sum":0.524969509,"avgtime":0.524969509},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":1600,"sum":1.868460513,"avgtime":0.001167787},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":2720,"average_load":2720,"avg_session_uptime":1328,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":1947821,"buffer_anon_items":680,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":6024,"osdmap_items":100,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":975136,"mds_co_items":17104,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":5865,"op_send_bytes":18629739,"op_resend":0,"op_reply":5865,"op_latency":{"avgcount":5865,"sum":76.168356398,"avgtime":0.012986932},"op_inflight":0,"oplen_avg":{"avgcount":5865,"sum":6879},"op":5865,"op_r":61,"op_w":5804,"op_rmw":0,"op_pg":0,"osdop_stat":12,"osdop_create":426,"osdop_read":33,"osdop_write":1558,"osdop_writefull":22,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":3724,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":852,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":192,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":128,"omap_rd":48,"omap_del":51},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":19,"pq_executing":0,"pq_executing_high_water":11,"pq_executed_ops":3705,"pq_executed":1601,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":13823,"get_sum":3393967,"get_or_fail_fail":0,"get_or_fail_success":13823,"take":0,"take_sum":0,"put":13823,"put_sum":3393967,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5865,"take_sum":18792375,"put":5865,"put_sum":18792375,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":5865,"take_sum":5865,"put":5865,"put_sum":5865,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":1601,"get_sum":161701,"get_or_fail_fail":0,"get_or_fail_success":1601,"take":0,"take_sum":0,"put":25,"put_sum":161701,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5559efc69520":{"val":52712,"max":3758096384,"get_started":0,"get":9419,"get_sum":18018194,"get_or_fail_fail":0,"get_or_fail_success":9419,"take":0,"take_sum":0,"put":1533,"put_sum":17965482,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"trackedop":{"slow_ops_count":0}} 2024-09-17T04:03:21.850 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 4, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 2, "avgtime": 1.2e-07, "sum": 2.41e-07 }, "msgr_recv_bytes": 457557, "msgr_recv_encrypted_bytes": 127552, "msgr_recv_messages": 1273, "msgr_running_fast_dispatch_time": 0.056593546, "msgr_running_recv_time": 0.043445608, "msgr_running_send_time": 0.10147414, "msgr_running_total_time": 0.210833966, "msgr_send_bytes": 691302, "msgr_send_encrypted_bytes": 19136, "msgr_send_messages": 1267, "msgr_send_messages_queue_lat": { "avgcount": 1267, "avgtime": 0.001821827, "sum": 2.308254834 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 8396, "avgtime": 5.13e-07, "sum": 0.004308616 }, "msgr_recv_bytes": 3216893, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 9958, "msgr_running_fast_dispatch_time": 0.103089715, "msgr_running_recv_time": 0.870787363, "msgr_running_send_time": 1.444225965, "msgr_running_total_time": 2.038545264, "msgr_send_bytes": 12151974, "msgr_send_encrypted_bytes": 87408, "msgr_send_messages": 14895, "msgr_send_messages_queue_lat": { "avgcount": 14893, "avgtime": 0.000189519, "sum": 2.822509848 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 7, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 841841, "msgr_recv_encrypted_bytes": 0, "msgr_recv_messages": 2599, "msgr_running_fast_dispatch_time": 0.12084181, "msgr_running_recv_time": 0.079814365, "msgr_running_send_time": 0.28036545, "msgr_running_total_time": 0.505408573, "msgr_send_bytes": 9574286, "msgr_send_encrypted_bytes": 0, "msgr_send_messages": 2602, "msgr_send_messages_queue_lat": { "avgcount": 2599, "avgtime": 0.0011076, "sum": 2.878654983 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 3105, "avgtime": 0.000979462, "sum": 3.041231009 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4823, "avgtime": 0.000114681, "sum": 0.553110835 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 4910, "ceph_cap_op_revoke": 2, "ceph_cap_op_trunc": 0, "dir_commit": 12, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 174, "handle_client_caps": 4444, "handle_client_caps_dirty": 1500, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 431, "inodes_bottom": 431, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 431, "inodes_top": 0, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1606, "q": 0, "reply": 3205, "reply_latency": { "avgcount": 3205, "avgtime": 0.001458601, "sum": 4.674819336 }, "request": 3207, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 3207, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 3207, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 1601, "strays_enqueued": 1601, "strays_migrated": 0, "strays_reintegrated": 0, "uninline_started": 0, "uninline_succeeded": 0, "uninline_write_failed": 0 }, "mds_log": { "ev": 752, "evadd": 9419, "evex": 8670, "evexd": 8669, "evexg": 0, "evlrg": 0, "evtrm": 8670, "expos": 21296659, "jlat": { "avgcount": 1533, "avgtime": 0.004613539, "sum": 7.072556477 }, "rdpos": 4195244, "replayed": 3, "seg": 1, "segadd": 10, "segex": 11, "segexd": 10, "segexg": 0, "segmjr": 1, "segtrm": 11, "wrpos": 22213697 }, "mds_mem": { "cap": 1, "cap+": 1602, "cap-": 1601, "dir": 12, "dir+": 13, "dir-": 1, "dn": 431, "dn+": 3212, "dn-": 2781, "heap": 394223616, "ino": 101, "ino+": 1614, "ino-": 1513, "rss": 141788 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 3209, "dispatch_server_request": 0, "handle_client_request": 3207, "handle_client_session": 5, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1600, "avgtime": 0.001408513, "sum": 2.253621121 }, "req_getattr_latency": { "avgcount": 1, "avgtime": 0.000538302, "sum": 0.000538302 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 1, "avgtime": 0.00319594, "sum": 0.00319594 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 2, "avgtime": 0.012016975, "sum": 0.024033951 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 1, "avgtime": 0.524969509, "sum": 0.524969509 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 1600, "avgtime": 0.001167787, "sum": 1.868460513 } }, "mds_sessions": { "average_load": 2720, "avg_session_uptime": 1328, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 2720 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 1947821, "buffer_anon_items": 680, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 975136, "mds_co_items": 17104, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 6024, "osdmap_items": 100, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 192, "map_full": 0, "map_inc": 2, "omap_del": 51, "omap_rd": 48, "omap_wr": 128, "op": 5865, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 5865, "avgtime": 0.012986932, "sum": 76.168356398 }, "op_pg": 0, "op_r": 61, "op_reply": 5865, "op_resend": 0, "op_rmw": 0, "op_send": 5865, "op_send_bytes": 18629739, "op_w": 5804, "oplen_avg": { "avgcount": 5865, "sum": 6879 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 426, "osdop_delete": 3724, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 33, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 852, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 12, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 1558, "osdop_writefull": 22, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "purge_queue": { "pq_executed": 1601, "pq_executed_ops": 3705, "pq_executing": 0, "pq_executing_high_water": 11, "pq_executing_ops": 0, "pq_executing_ops_high_water": 19, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 13823, "get_or_fail_fail": 0, "get_or_fail_success": 13823, "get_started": 0, "get_sum": 3393967, "max": 104857600, "put": 13823, "put_sum": 3393967, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 5865, "put_sum": 18792375, "take": 5865, "take_sum": 18792375, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 5865, "put_sum": 5865, "take": 5865, "take_sum": 5865, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 1601, "get_or_fail_fail": 0, "get_or_fail_success": 1601, "get_started": 0, "get_sum": 161701, "max": 3758096384, "put": 25, "put_sum": 161701, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5559efc69520": { "get": 9419, "get_or_fail_fail": 0, "get_or_fail_success": 9419, "get_started": 0, "get_sum": 18018194, "max": 3758096384, "put": 1533, "put_sum": 17965482, "take": 0, "take_sum": 0, "val": 52712, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "trackedop": { "slow_ops_count": 0 } } 2024-09-17T04:03:21.851 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:03:21.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:03:21.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:03:22.038 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:03:22.038 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:03:22.038 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:22.038 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:22.074 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:22.074 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:22.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:03:22.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 -- 172.21.15.129:0/2239765793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 msgr2=0x7f794c072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 --2- 172.21.15.129:0/2239765793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c072d70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7934009860 tx=0x7f793402a510 comp rx=0 tx=0).stop 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 -- 172.21.15.129:0/2239765793 shutdown_connections 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 --2- 172.21.15.129:0/2239765793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c072d70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 -- 172.21.15.129:0/2239765793 >> 172.21.15.129:0/2239765793 conn(0x7f794c0b23c0 msgr2=0x7f794c0b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 -- 172.21.15.129:0/2239765793 shutdown_connections 2024-09-17T04:03:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.411+0000 7f7951800640 1 -- 172.21.15.129:0/2239765793 wait complete. 2024-09-17T04:03:22.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.413+0000 7f7951800640 1 Processor -- start 2024-09-17T04:03:22.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.414+0000 7f7951800640 1 -- start start 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 0x7f794c14eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c15f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f794c0700a0 con 0x7f794c0cd190 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f794c06ff20 con 0x7f794c0b59d0 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7951800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f794c070220 con 0x7f794c15cf70 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7943e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7943e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47256/0 (socket says 172.21.15.129:47256) 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f7943e00640 1 -- 172.21.15.129:0/2946222672 learned_addr learned my addr 172.21.15.129:0/2946222672 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f794aa00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 0x7f794c14eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.415+0000 7f794b400640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c15f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 -- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 msgr2=0x7f794c15f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c15f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 -- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 msgr2=0x7f794c14eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 0x7f794c14eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 -- 172.21.15.129:0/2946222672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f794c151690 con 0x7f794c0cd190 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f794aa00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 0x7f794c14eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f794b400640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c15f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:22.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7943e00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f793402ab00 tx=0x7f7934033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:22.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934031d60 con 0x7f794c0cd190 2024-09-17T04:03:22.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7934033e10 con 0x7f794c0cd190 2024-09-17T04:03:22.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7934005260 con 0x7f794c0cd190 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f794c151970 con 0x7f794c0cd190 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.416+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f794c151f30 con 0x7f794c0cd190 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.418+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f794c1513e0 con 0x7f794c0cd190 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.419+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7934005400 con 0x7f794c0cd190 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.420+0000 7f7942a00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 0x7f792807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.420+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f79340bbf40 con 0x7f794c0cd190 2024-09-17T04:03:22.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.420+0000 7f794aa00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 0x7f792807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:22.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.421+0000 7f794aa00640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 0x7f792807b0d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f793c007fd0 tx=0x7f793c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:22.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.424+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f793403f030 con 0x7f794c0cd190 2024-09-17T04:03:22.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.621+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f794c072d70 con 0x7f794c0cd190 2024-09-17T04:03:22.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.621+0000 7f7942a00640 1 -- 172.21.15.129:0/2946222672 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v192) ==== 80+0+124 (secure 0 0 0) 0x7f79340841f0 con 0x7f794c0cd190 2024-09-17T04:03:22.622 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6835/3531032601 2024-09-18T04:01:58.368557+0000 2024-09-17T04:03:22.622 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6832/3531032601 2024-09-18T04:01:58.368557+0000 2024-09-17T04:03:22.622 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T04:03:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 msgr2=0x7f792807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 0x7f792807b0d0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f793c007fd0 tx=0x7f793c009040 comp rx=0 tx=0).stop 2024-09-17T04:03:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 msgr2=0x7f794c143de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f793402ab00 tx=0x7f7934033c20 comp rx=0 tx=0).stop 2024-09-17T04:03:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 shutdown_connections 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7928078bc0 0x7f792807b0d0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f794c15cf70 0x7f794c15f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f794c0cd190 0x7f794c143de0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 --2- 172.21.15.129:0/2946222672 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f794c0b59d0 0x7f794c14eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 >> 172.21.15.129:0/2946222672 conn(0x7f794c0b23c0 msgr2=0x7f794c0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 shutdown_connections 2024-09-17T04:03:22.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:22.624+0000 7f7951800640 1 -- 172.21.15.129:0/2946222672 wait complete. 2024-09-17T04:03:22.650 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:03:22.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:22.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:03:22.742 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:03:22.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:22.742 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:03:22.795 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:03:22.795 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:03:22.796 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:22.833 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:03:22.834 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:03:22.834 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:22.834 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:22.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:03:22.870 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:03:22.870 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:03:22.870 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:22.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:03:22.910 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 -- 172.21.15.129:0/1205614299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540cd190 msgr2=0x7f7d5415cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 --2- 172.21.15.129:0/1205614299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540cd190 0x7f7d5415cf70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009830 tx=0x7f7d4402a480 comp rx=0 tx=0).stop 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 -- 172.21.15.129:0/1205614299 shutdown_connections 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 --2- 172.21.15.129:0/1205614299 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540b59d0 0x7f7d540b5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 --2- 172.21.15.129:0/1205614299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540cd190 0x7f7d5415cf70 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 -- 172.21.15.129:0/1205614299 >> 172.21.15.129:0/1205614299 conn(0x7f7d540b23c0 msgr2=0x7f7d540b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 -- 172.21.15.129:0/1205614299 shutdown_connections 2024-09-17T04:03:23.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.129+0000 7f7d59a00640 1 -- 172.21.15.129:0/1205614299 wait complete. 2024-09-17T04:03:23.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.132+0000 7f7d59a00640 1 Processor -- start 2024-09-17T04:03:23.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.132+0000 7f7d59a00640 1 -- start start 2024-09-17T04:03:23.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540b59d0 0x7f7d540b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540cfd80 0x7f7d540c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5406fe50 con 0x7f7d540cd190 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5406fcd0 con 0x7f7d540b59d0 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d59a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5406ffd0 con 0x7f7d540cfd80 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47264/0 (socket says 172.21.15.129:47264) 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.133+0000 7f7d52000640 1 -- 172.21.15.129:0/2706601665 learned_addr learned my addr 172.21.15.129:0/2706601665 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540b59d0 0x7f7d540b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d53400640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540cfd80 0x7f7d540c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 -- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540cfd80 msgr2=0x7f7d540c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540cfd80 0x7f7d540c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 -- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540b59d0 msgr2=0x7f7d540b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540b59d0 0x7f7d540b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 -- 172.21.15.129:0/2706601665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d541f5970 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d52000640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000c400 tx=0x7f7d4000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40011790 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d40017070 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d541f5c50 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.134+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40011cc0 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.135+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d541f6210 con 0x7f7d540cd190 2024-09-17T04:03:23.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.136+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d540704f0 con 0x7f7d540cd190 2024-09-17T04:03:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.137+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7d4001c460 con 0x7f7d540cd190 2024-09-17T04:03:23.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.138+0000 7f7d4aa00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 0x7f7d3407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.138+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f7d4009abf0 con 0x7f7d540cd190 2024-09-17T04:03:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.138+0000 7f7d52a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 0x7f7d3407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.139+0000 7f7d52a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 0x7f7d3407b0a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7d44034000 tx=0x7f7d44004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:23.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.142+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7d40062ff0 con 0x7f7d540cd190 2024-09-17T04:03:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.376+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d541f56e0 con 0x7f7d540cd190 2024-09-17T04:03:23.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.377+0000 7f7d4aa00640 1 -- 172.21.15.129:0/2706601665 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f7d400628f0 con 0x7f7d540cd190 2024-09-17T04:03:23.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:23.377 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:23.378 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 msgr2=0x7f7d3407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 0x7f7d3407b0a0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f7d44034000 tx=0x7f7d44004730 comp rx=0 tx=0).stop 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 msgr2=0x7f7d540c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f7d4000c400 tx=0x7f7d4000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 shutdown_connections 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d34078b90 0x7f7d3407b0a0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d540cfd80 0x7f7d540c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d540cd190 0x7f7d540c7040 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 --2- 172.21.15.129:0/2706601665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d540b59d0 0x7f7d540b1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 >> 172.21.15.129:0/2706601665 conn(0x7f7d540b23c0 msgr2=0x7f7d54075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 shutdown_connections 2024-09-17T04:03:23.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.380+0000 7f7d59a00640 1 -- 172.21.15.129:0/2706601665 wait complete. 2024-09-17T04:03:23.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 -- 172.21.15.129:0/821824161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a4815cf70 msgr2=0x7f5a48072d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/821824161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a4815cf70 0x7f5a48072d70 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009860 tx=0x7f5a3002a510 comp rx=0 tx=0).stop 2024-09-17T04:03:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 -- 172.21.15.129:0/821824161 shutdown_connections 2024-09-17T04:03:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/821824161 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a4815cf70 0x7f5a48072d70 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 -- 172.21.15.129:0/821824161 >> 172.21.15.129:0/821824161 conn(0x7f5a480b23c0 msgr2=0x7f5a480b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 -- 172.21.15.129:0/821824161 shutdown_connections 2024-09-17T04:03:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.629+0000 7f5a4ec00640 1 -- 172.21.15.129:0/821824161 wait complete. 2024-09-17T04:03:23.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.631+0000 7f5a4ec00640 1 Processor -- start 2024-09-17T04:03:23.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.631+0000 7f5a4ec00640 1 -- start start 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a480cd190 0x7f5a48136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a4815cf70 0x7f5a48136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5a480700a0 con 0x7f5a480b59d0 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4806ff20 con 0x7f5a4815cf70 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.632+0000 7f5a4ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5a48070220 con 0x7f5a480cd190 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47276/0 (socket says 172.21.15.129:47276) 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 -- 172.21.15.129:0/3636230767 learned_addr learned my addr 172.21.15.129:0/3636230767 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:23.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a3f400640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a480cd190 0x7f5a48136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a4cc00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a4815cf70 0x7f5a48136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 -- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a480cd190 msgr2=0x7f5a48136570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a480cd190 0x7f5a48136570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 -- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a4815cf70 msgr2=0x7f5a48136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a4815cf70 0x7f5a48136ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.633+0000 7f5a47e00640 1 -- 172.21.15.129:0/3636230767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a481f71e0 con 0x7f5a480b59d0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a47e00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c430 tx=0x7f5a3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38017070 con 0x7f5a480b59d0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a481f74c0 con 0x7f5a480b59d0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5a380053e0 con 0x7f5a480b59d0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38003850 con 0x7f5a480b59d0 2024-09-17T04:03:23.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.634+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a481f7a80 con 0x7f5a480b59d0 2024-09-17T04:03:23.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.635+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a48072d70 con 0x7f5a480b59d0 2024-09-17T04:03:23.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.637+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5a3801a070 con 0x7f5a480b59d0 2024-09-17T04:03:23.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.637+0000 7f5a44c00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 0x7f5a2407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:23.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.638+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f5a3809a2a0 con 0x7f5a480b59d0 2024-09-17T04:03:23.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.638+0000 7f5a3f400640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 0x7f5a2407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:23.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.639+0000 7f5a3f400640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 0x7f5a2407b0a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5a30034000 tx=0x7f5a30004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:23.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.642+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5a38061e80 con 0x7f5a480b59d0 2024-09-17T04:03:23.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.862+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a480b0050 con 0x7f5a480b59d0 2024-09-17T04:03:23.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.863+0000 7f5a44c00640 1 -- 172.21.15.129:0/3636230767 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f5a380668b0 con 0x7f5a480b59d0 2024-09-17T04:03:23.863 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:23.863 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:23.864 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.865+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 msgr2=0x7f5a2407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.865+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 0x7f5a2407b0a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5a30034000 tx=0x7f5a30004730 comp rx=0 tx=0).stop 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.865+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 msgr2=0x7f5a481eade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.865+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c430 tx=0x7f5a3800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 shutdown_connections 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a24078b90 0x7f5a2407b0a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a4815cf70 0x7f5a48136ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5a480cd190 0x7f5a48136570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 --2- 172.21.15.129:0/3636230767 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5a480b59d0 0x7f5a481eade0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 >> 172.21.15.129:0/3636230767 conn(0x7f5a480b23c0 msgr2=0x7f5a48073580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 shutdown_connections 2024-09-17T04:03:23.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:23.866+0000 7f5a4ec00640 1 -- 172.21.15.129:0/3636230767 wait complete. 2024-09-17T04:03:23.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 -- 172.21.15.129:0/422239532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 msgr2=0x7f684c06e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 --2- 172.21.15.129:0/422239532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c06e310 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f684c0b2860 tx=0x7f683c02a530 comp rx=0 tx=0).stop 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 -- 172.21.15.129:0/422239532 shutdown_connections 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 --2- 172.21.15.129:0/422239532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c06e310 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 -- 172.21.15.129:0/422239532 >> 172.21.15.129:0/422239532 conn(0x7f684c06d560 msgr2=0x7f684c0b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.108+0000 7f6852600640 1 -- 172.21.15.129:0/422239532 shutdown_connections 2024-09-17T04:03:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.109+0000 7f6852600640 1 -- 172.21.15.129:0/422239532 wait complete. 2024-09-17T04:03:24.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.111+0000 7f6852600640 1 Processor -- start 2024-09-17T04:03:24.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.111+0000 7f6852600640 1 -- start start 2024-09-17T04:03:24.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f684c141f60 0x7f684c1f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c15f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f684c075110 con 0x7f684c156f90 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f684c074f90 con 0x7f684c15da40 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.112+0000 7f6852600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f684c075290 con 0x7f684c141f60 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39070/0 (socket says 172.21.15.129:39070) 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 -- 172.21.15.129:0/1760704971 learned_addr learned my addr 172.21.15.129:0/1760704971 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f6851200640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f684c141f60 0x7f684c1f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f6851c00640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c15f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 -- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f684c141f60 msgr2=0x7f684c1f55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f684c141f60 0x7f684c1f55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 -- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 msgr2=0x7f684c15f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c15f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.113+0000 7f684be00640 1 -- 172.21.15.129:0/1760704971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684c1f5f40 con 0x7f684c156f90 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f684be00640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f683800c400 tx=0x7f683800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838017070 con 0x7f684c156f90 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68380053c0 con 0x7f684c156f90 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684c1f6220 con 0x7f684c156f90 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838003850 con 0x7f684c156f90 2024-09-17T04:03:24.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.114+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684c1f6830 con 0x7f684c156f90 2024-09-17T04:03:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.115+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684c1f5cb0 con 0x7f684c156f90 2024-09-17T04:03:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.116+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f683801a070 con 0x7f684c156f90 2024-09-17T04:03:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.117+0000 7f6848c00640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 0x7f682c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.117+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f683809a210 con 0x7f684c156f90 2024-09-17T04:03:24.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.118+0000 7f6851200640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 0x7f682c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.118+0000 7f6851200640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 0x7f682c07b0d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f683c034000 tx=0x7f683c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:24.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.122+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6838061e80 con 0x7f684c156f90 2024-09-17T04:03:24.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.307+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f684c06e310 con 0x7f684c156f90 2024-09-17T04:03:24.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.309+0000 7f6848c00640 1 -- 172.21.15.129:0/1760704971 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) ==== 74+0+17160 (secure 0 0 0) 0x7f68380668b0 con 0x7f684c156f90 2024-09-17T04:03:24.309 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:24.309 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":192,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:02:18.611682+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:01:40.170274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-17T04:01:41.196845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.10","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.1b","mappings":[{"from":6,"to":0}]},{"pgid":"29.31","mappings":[{"from":6,"to":2}]},{"pgid":"29.3d","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/3531032601":"2024-09-18T04:01:58.368557+0000","172.21.15.129:6832/3531032601":"2024-09-18T04:01:58.368557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:24.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 msgr2=0x7f682c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 0x7f682c07b0d0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f683c034000 tx=0x7f683c004730 comp rx=0 tx=0).stop 2024-09-17T04:03:24.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 msgr2=0x7f684c09d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f683800c400 tx=0x7f683800c8c0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 shutdown_connections 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f682c078bc0 0x7f682c07b0d0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f684c15da40 0x7f684c15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f684c156f90 0x7f684c09d360 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 --2- 172.21.15.129:0/1760704971 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f684c141f60 0x7f684c1f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 >> 172.21.15.129:0/1760704971 conn(0x7f684c06d560 msgr2=0x7f684c0ad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 shutdown_connections 2024-09-17T04:03:24.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.311+0000 7f6852600640 1 -- 172.21.15.129:0/1760704971 wait complete. 2024-09-17T04:03:24.338 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:03:24.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:24.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 -- 172.21.15.129:0/2600230871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34240737b0 msgr2=0x7f3424073bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 --2- 172.21.15.129:0/2600230871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34240737b0 0x7f3424073bc0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f3418009830 tx=0x7f341802a5c0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 -- 172.21.15.129:0/2600230871 shutdown_connections 2024-09-17T04:03:24.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 --2- 172.21.15.129:0/2600230871 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34240737b0 0x7f3424073bc0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 -- 172.21.15.129:0/2600230871 >> 172.21.15.129:0/2600230871 conn(0x7f34240b23c0 msgr2=0x7f34240b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:24.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 -- 172.21.15.129:0/2600230871 shutdown_connections 2024-09-17T04:03:24.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.558+0000 7f342b800640 1 -- 172.21.15.129:0/2600230871 wait complete. 2024-09-17T04:03:24.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.560+0000 7f342b800640 1 Processor -- start 2024-09-17T04:03:24.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.560+0000 7f342b800640 1 -- start start 2024-09-17T04:03:24.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34240737b0 0x7f3424156390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34240cedf0 0x7f3424143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f34240cfca0 con 0x7f342415f880 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f34240cfb20 con 0x7f34240cedf0 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f342b800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f34240cfe20 con 0x7f34240737b0 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f3429800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f3429800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39078/0 (socket says 172.21.15.129:39078) 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f3429800640 1 -- 172.21.15.129:0/1185875342 learned_addr learned my addr 172.21.15.129:0/1185875342 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f3428e00640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34240737b0 0x7f3424156390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.562+0000 7f3423e00640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34240cedf0 0x7f3424143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 -- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34240737b0 msgr2=0x7f3424156390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34240737b0 0x7f3424156390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 -- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34240cedf0 msgr2=0x7f3424143e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34240cedf0 0x7f3424143e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 -- 172.21.15.129:0/1185875342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3424156be0 con 0x7f342415f880 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3429800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f341c00c730 tx=0x7f341c00cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c003b70 con 0x7f342415f880 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f341c003d10 con 0x7f342415f880 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c0052d0 con 0x7f342415f880 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3424156ec0 con 0x7f342415f880 2024-09-17T04:03:24.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.563+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3424157480 con 0x7f342415f880 2024-09-17T04:03:24.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.564+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342415f0a0 con 0x7f342415f880 2024-09-17T04:03:24.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.566+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f341c005470 con 0x7f342415f880 2024-09-17T04:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.567+0000 7f3420c00640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 0x7f340407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.567+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f341c09aa40 con 0x7f342415f880 2024-09-17T04:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.567+0000 7f3428e00640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 0x7f340407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:24.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.568+0000 7f3428e00640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 0x7f340407b0a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3418034000 tx=0x7f3418004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:24.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.571+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f342415f0a0 con 0x7f342415f880 2024-09-17T04:03:24.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.791+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3424156970 con 0x7f342415f880 2024-09-17T04:03:24.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.791+0000 7f3420c00640 1 -- 172.21.15.129:0/1185875342 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7f34240ce930 con 0x7f342415f880 2024-09-17T04:03:24.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:24.792 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:24.792 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 msgr2=0x7f340407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 0x7f340407b0a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f3418034000 tx=0x7f3418004730 comp rx=0 tx=0).stop 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 msgr2=0x7f3424158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7f341c00c730 tx=0x7f341c00cb30 comp rx=0 tx=0).stop 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 shutdown_connections 2024-09-17T04:03:24.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3404078b90 0x7f340407b0a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f342415f880 0x7f3424158af0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34240cedf0 0x7f3424143e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 --2- 172.21.15.129:0/1185875342 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34240737b0 0x7f3424156390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 >> 172.21.15.129:0/1185875342 conn(0x7f34240b23c0 msgr2=0x7f3424150ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 shutdown_connections 2024-09-17T04:03:24.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:24.794+0000 7f342b800640 1 -- 172.21.15.129:0/1185875342 wait complete. 2024-09-17T04:03:24.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 -- 172.21.15.129:0/3689044189 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0d2290 msgr2=0x7fb68c15ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 --2- 172.21.15.129:0/3689044189 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0d2290 0x7fb68c15ea40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb674009860 tx=0x7fb67402a530 comp rx=0 tx=0).stop 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 -- 172.21.15.129:0/3689044189 shutdown_connections 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 --2- 172.21.15.129:0/3689044189 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0d2290 0x7fb68c15ea40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 -- 172.21.15.129:0/3689044189 >> 172.21.15.129:0/3689044189 conn(0x7fb68c0d1770 msgr2=0x7fb68c0d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.042+0000 7fb691e00640 1 -- 172.21.15.129:0/3689044189 shutdown_connections 2024-09-17T04:03:25.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.043+0000 7fb691e00640 1 -- 172.21.15.129:0/3689044189 wait complete. 2024-09-17T04:03:25.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.045+0000 7fb691e00640 1 Processor -- start 2024-09-17T04:03:25.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.045+0000 7fb691e00640 1 -- start start 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0cf2e0 0x7fb68c0b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb68c0d2290 0x7fb68c0b5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c0d3df0 con 0x7fb68c06d270 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb68c0d3c70 con 0x7fb68c0d2290 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.046+0000 7fb691e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb68c0d3f70 con 0x7fb68c0cf2e0 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39080/0 (socket says 172.21.15.129:39080) 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 -- 172.21.15.129:0/1475278799 learned_addr learned my addr 172.21.15.129:0/1475278799 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:25.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb691400640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb68c0d2290 0x7fb68c0b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb683e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0cf2e0 0x7fb68c0b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 -- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0cf2e0 msgr2=0x7fb68c0b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0cf2e0 0x7fb68c0b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 -- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb68c0d2290 msgr2=0x7fb68c0b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb68c0d2290 0x7fb68c0b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 -- 172.21.15.129:0/1475278799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68c1e7a00 con 0x7fb68c06d270 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb690a00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00c460 tx=0x7fb67c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.047+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0052d0 con 0x7fb68c06d270 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.048+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb67c005490 con 0x7fb68c06d270 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.048+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c012870 con 0x7fb68c06d270 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.048+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68c1e7ce0 con 0x7fb68c06d270 2024-09-17T04:03:25.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.048+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb68c0cce20 con 0x7fb68c06d270 2024-09-17T04:03:25.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.049+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb68c15ea40 con 0x7fb68c06d270 2024-09-17T04:03:25.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.050+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb67c012a10 con 0x7fb68c06d270 2024-09-17T04:03:25.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.051+0000 7fb689600640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 0x7fb66807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.051+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7fb67c09a910 con 0x7fb68c06d270 2024-09-17T04:03:25.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.051+0000 7fb683e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 0x7fb66807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.052+0000 7fb683e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 0x7fb66807b0d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb674034000 tx=0x7fb674004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.055+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb67c062ce0 con 0x7fb68c06d270 2024-09-17T04:03:25.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.273+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb68c0b3dc0 con 0x7fb68c06d270 2024-09-17T04:03:25.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.274+0000 7fb689600640 1 -- 172.21.15.129:0/1475278799 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+5253 (secure 0 0 0) 0x7fb67c0625e0 con 0x7fb68c06d270 2024-09-17T04:03:25.274 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:25.274 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":172,"btime":"2024-09-17T04:02:02:398283+0000","default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:01:42.223951+0000","modified":"2024-09-17T04:02:02.398276+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":190,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10077},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10077":{"gid":10077,"name":"c","rank":0,"incarnation":168,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/2632602393","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2632602393},{"type":"v1","addr":"172.21.15.129:6833","nonce":2632602393}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10077,"qdb_cluster":[10077]},"id":14}]} 2024-09-17T04:03:25.274 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 172 2024-09-17T04:03:25.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 msgr2=0x7fb66807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 0x7fb66807b0d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb674034000 tx=0x7fb674004730 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 msgr2=0x7fb68c14eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00c460 tx=0x7fb67c00c920 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 shutdown_connections 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb668078bc0 0x7fb66807b0d0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb68c0d2290 0x7fb68c0b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb68c0cf2e0 0x7fb68c0b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 --2- 172.21.15.129:0/1475278799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb68c06d270 0x7fb68c14eb70 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 >> 172.21.15.129:0/1475278799 conn(0x7fb68c0d1770 msgr2=0x7fb68c075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 shutdown_connections 2024-09-17T04:03:25.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.276+0000 7fb691e00640 1 -- 172.21.15.129:0/1475278799 wait complete. 2024-09-17T04:03:25.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 -- 172.21.15.129:0/49293453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 msgr2=0x7f8234162f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 --2- 172.21.15.129:0/49293453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234162f30 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f82340b5a20 tx=0x7f822002a5c0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 -- 172.21.15.129:0/49293453 shutdown_connections 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 --2- 172.21.15.129:0/49293453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234162f30 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 -- 172.21.15.129:0/49293453 >> 172.21.15.129:0/49293453 conn(0x7f82340b23c0 msgr2=0x7f82340b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 -- 172.21.15.129:0/49293453 shutdown_connections 2024-09-17T04:03:25.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.519+0000 7f8239400640 1 -- 172.21.15.129:0/49293453 wait complete. 2024-09-17T04:03:25.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.521+0000 7f8239400640 1 Processor -- start 2024-09-17T04:03:25.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.522+0000 7f8239400640 1 -- start start 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82340aeca0 0x7f82341f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f823415f110 0x7f82341f9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82340b5f80 con 0x7f82340ae890 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f82340700c0 con 0x7f823415f110 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8239400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f82340b6100 con 0x7f82340aeca0 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8232800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8232800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39094/0 (socket says 172.21.15.129:39094) 2024-09-17T04:03:25.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.523+0000 7f8232800640 1 -- 172.21.15.129:0/1365204391 learned_addr learned my addr 172.21.15.129:0/1365204391 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8233200640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f823415f110 0x7f82341f9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8231e00640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82340aeca0 0x7f82341f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 -- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82340aeca0 msgr2=0x7f82341f8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82340aeca0 0x7f82341f8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 -- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f823415f110 msgr2=0x7f82341f9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f823415f110 0x7f82341f9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 -- 172.21.15.129:0/1365204391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82341e8570 con 0x7f82340ae890 2024-09-17T04:03:25.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8232800640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f822400c730 tx=0x7f822400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224003b70 con 0x7f82340ae890 2024-09-17T04:03:25.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8224003d30 con 0x7f82340ae890 2024-09-17T04:03:25.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.524+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82341e8850 con 0x7f82340ae890 2024-09-17T04:03:25.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.525+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822400aab0 con 0x7f82340ae890 2024-09-17T04:03:25.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.525+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341e8b30 con 0x7f82340ae890 2024-09-17T04:03:25.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.526+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82340b6580 con 0x7f82340ae890 2024-09-17T04:03:25.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.527+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f822400ac50 con 0x7f82340ae890 2024-09-17T04:03:25.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.528+0000 7f822aa00640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 0x7f821407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.529+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f822409aa50 con 0x7f82340ae890 2024-09-17T04:03:25.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.529+0000 7f8231e00640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 0x7f821407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.530+0000 7f8231e00640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 0x7f821407b0a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8220034000 tx=0x7f8220004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.533+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f82240627e0 con 0x7f82340ae890 2024-09-17T04:03:25.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.717+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82341410e0 con 0x7f82340ae890 2024-09-17T04:03:25.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.719+0000 7f822aa00640 1 -- 172.21.15.129:0/1365204391 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) ==== 74+0+17160 (secure 0 0 0) 0x7f8224067210 con 0x7f82340ae890 2024-09-17T04:03:25.720 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:25.720 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":192,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:02:18.611682+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:01:40.170274+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"185","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-17T04:01:41.196845+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"186","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.10","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.1b","mappings":[{"from":6,"to":0}]},{"pgid":"29.31","mappings":[{"from":6,"to":2}]},{"pgid":"29.3d","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/3531032601":"2024-09-18T04:01:58.368557+0000","172.21.15.129:6832/3531032601":"2024-09-18T04:01:58.368557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:25.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 msgr2=0x7f821407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 0x7f821407b0a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f8220034000 tx=0x7f8220004730 comp rx=0 tx=0).stop 2024-09-17T04:03:25.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 msgr2=0x7f8234142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f822400c730 tx=0x7f822400cbf0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 shutdown_connections 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8214078b90 0x7f821407b0a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f823415f110 0x7f82341f9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82340aeca0 0x7f82341f8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 --2- 172.21.15.129:0/1365204391 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82340ae890 0x7f8234142310 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.721+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 >> 172.21.15.129:0/1365204391 conn(0x7f82340b23c0 msgr2=0x7f8234160170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:25.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.722+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 shutdown_connections 2024-09-17T04:03:25.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.722+0000 7f8239400640 1 -- 172.21.15.129:0/1365204391 wait complete. 2024-09-17T04:03:25.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:03:25.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 -- 172.21.15.129:0/1369493689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0156f90 msgr2=0x7f85f00722b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 --2- 172.21.15.129:0/1369493689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0156f90 0x7f85f00722b0 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f85f006e9d0 tx=0x7f85d802a540 comp rx=0 tx=0).stop 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 -- 172.21.15.129:0/1369493689 shutdown_connections 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 --2- 172.21.15.129:0/1369493689 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f0141f60 0x7f85f0142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 --2- 172.21.15.129:0/1369493689 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0156f90 0x7f85f00722b0 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 --2- 172.21.15.129:0/1369493689 >> v2:172.21.15.181:3300/0 conn(0x7f85f015da40 0x7f85f006e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 -- 172.21.15.129:0/1369493689 >> 172.21.15.129:0/1369493689 conn(0x7f85f006d560 msgr2=0x7f85f00b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 -- 172.21.15.129:0/1369493689 shutdown_connections 2024-09-17T04:03:25.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.970+0000 7f85f5200640 1 -- 172.21.15.129:0/1369493689 wait complete. 2024-09-17T04:03:25.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.972+0000 7f85f5200640 1 Processor -- start 2024-09-17T04:03:25.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.972+0000 7f85f5200640 1 -- start start 2024-09-17T04:03:25.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 0x7f85f009d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 0x7f85f01603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f85f0074d50 con 0x7f85f0141f60 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f85f0074bd0 con 0x7f85f0156f90 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85f5200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f85f0074ed0 con 0x7f85f015da40 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39114/0 (socket says 172.21.15.129:39114) 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 -- 172.21.15.129:0/1737853484 learned_addr learned my addr 172.21.15.129:0/1737853484 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85efe00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 0x7f85f01603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85e6a00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 0x7f85f009d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 -- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 msgr2=0x7f85f01603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 0x7f85f01603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 -- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 msgr2=0x7f85f009d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 0x7f85f009d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85ef400640 1 -- 172.21.15.129:0/1737853484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f01578e0 con 0x7f85f0141f60 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85e6a00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 0x7f85f009d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.974+0000 7f85efe00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 0x7f85f01603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:25.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85ef400640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d740 tx=0x7f85e000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0017070 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f85e0009d20 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f0157d20 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e00106c0 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.975+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f01582e0 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.976+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f00753f0 con 0x7f85f0141f60 2024-09-17T04:03:25.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.977+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f85e001a070 con 0x7f85f0141f60 2024-09-17T04:03:25.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.978+0000 7f85e7e00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 0x7f85cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:25.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.979+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 5940+0+0 (secure 0 0 0) 0x7f85e009af10 con 0x7f85f0141f60 2024-09-17T04:03:25.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.979+0000 7f85e6a00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 0x7f85cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:25.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.980+0000 7f85e6a00640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 0x7f85cc07b0a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f85d8033000 tx=0x7f85d8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:25.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:25.983+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f85e0063250 con 0x7f85f0141f60 2024-09-17T04:03:26.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.202+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f85f00b4400 con 0x7f85f0141f60 2024-09-17T04:03:26.203 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:03:26.203+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:03:26.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.207+0000 7f85e7e00640 1 -- 172.21.15.129:0/1737853484 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v173) ==== 161+0+0 (secure 0 0 0) 0x7f85e0062b50 con 0x7f85f0141f60 2024-09-17T04:03:26.207 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 msgr2=0x7f85cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 0x7f85cc07b0a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f85d8033000 tx=0x7f85d8004730 comp rx=0 tx=0).stop 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 msgr2=0x7f85f01573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d740 tx=0x7f85e000dc00 comp rx=0 tx=0).stop 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 shutdown_connections 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f85cc078b90 0x7f85cc07b0a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f85f015da40 0x7f85f01603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f85f0156f90 0x7f85f009d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 --2- 172.21.15.129:0/1737853484 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f85f0141f60 0x7f85f01573a0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.211+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 >> 172.21.15.129:0/1737853484 conn(0x7f85f006d560 msgr2=0x7f85f00b1d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:26.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.212+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 shutdown_connections 2024-09-17T04:03:26.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.212+0000 7f85f5200640 1 -- 172.21.15.129:0/1737853484 wait complete. 2024-09-17T04:03:26.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:03:26.370 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.483+0000 7fdfa7000640 1 -- 172.21.15.129:0/3575934766 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa015da40 msgr2=0x7fdfa006e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.483+0000 7fdfa7000640 1 --2- 172.21.15.129:0/3575934766 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa015da40 0x7fdfa006e310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009860 tx=0x7fdf9402a530 comp rx=0 tx=0).stop 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.483+0000 7fdfa7000640 1 -- 172.21.15.129:0/3575934766 shutdown_connections 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.483+0000 7fdfa7000640 1 --2- 172.21.15.129:0/3575934766 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa015da40 0x7fdfa006e310 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.484+0000 7fdfa7000640 1 -- 172.21.15.129:0/3575934766 >> 172.21.15.129:0/3575934766 conn(0x7fdfa006d560 msgr2=0x7fdfa00b1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.484+0000 7fdfa7000640 1 -- 172.21.15.129:0/3575934766 shutdown_connections 2024-09-17T04:03:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.484+0000 7fdfa7000640 1 -- 172.21.15.129:0/3575934766 wait complete. 2024-09-17T04:03:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.486+0000 7fdfa7000640 1 Processor -- start 2024-09-17T04:03:26.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.486+0000 7fdfa7000640 1 -- start start 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa0156f90 0x7fdfa015fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa015da40 0x7fdfa015bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa0075110 con 0x7fdfa0141f60 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa0074f90 con 0x7fdfa015da40 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.487+0000 7fdfa7000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa0075290 con 0x7fdfa0156f90 2024-09-17T04:03:26.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39138/0 (socket says 172.21.15.129:39138) 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 -- 172.21.15.129:0/1980335540 learned_addr learned my addr 172.21.15.129:0/1980335540 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5200640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa0156f90 0x7fdfa015fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa6600640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa015da40 0x7fdfa015bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 -- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa0156f90 msgr2=0x7fdfa015fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa0156f90 0x7fdfa015fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 -- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa015da40 msgr2=0x7fdfa015bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa015da40 0x7fdfa015bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:26.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 -- 172.21.15.129:0/1980335540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa0150570 con 0x7fdfa0141f60 2024-09-17T04:03:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.488+0000 7fdfa5c00640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402ab20 tx=0x7fdf94030650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.489+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9403f070 con 0x7fdfa0141f60 2024-09-17T04:03:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.489+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf94030d60 con 0x7fdfa0141f60 2024-09-17T04:03:26.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.489+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa0150850 con 0x7fdfa0141f60 2024-09-17T04:03:26.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.489+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94004280 con 0x7fdfa0141f60 2024-09-17T04:03:26.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.489+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa0150e10 con 0x7fdfa0141f60 2024-09-17T04:03:26.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.490+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa0151250 con 0x7fdfa0141f60 2024-09-17T04:03:26.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.491+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdf940035f0 con 0x7fdfa0141f60 2024-09-17T04:03:26.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.492+0000 7fdf9d600640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 0x7fdf8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:26.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.492+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 6026+0+0 (secure 0 0 0) 0x7fdf940bbdb0 con 0x7fdfa0141f60 2024-09-17T04:03:26.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.492+0000 7fdfa5200640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 0x7fdf8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:26.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.493+0000 7fdfa5200640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 0x7fdf8007b0a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fdfa01610e0 tx=0x7fdf8c001720 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:26.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.496+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdf940c0070 con 0x7fdfa0141f60 2024-09-17T04:03:26.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:26.719+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fdfa006e310 con 0x7fdfa0141f60 2024-09-17T04:03:27.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.210+0000 7fdf9d600640 1 -- 172.21.15.129:0/1980335540 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v174) ==== 104+0+0 (secure 0 0 0) 0x7fdf94083ae0 con 0x7fdfa0141f60 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 msgr2=0x7fdf8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 0x7fdf8007b0a0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fdfa01610e0 tx=0x7fdf8c001720 comp rx=0 tx=0).stop 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 msgr2=0x7fdfa0157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402ab20 tx=0x7fdf94030650 comp rx=0 tx=0).stop 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 shutdown_connections 2024-09-17T04:03:27.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf80078b90 0x7fdf8007b0a0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfa015da40 0x7fdfa015bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdfa0156f90 0x7fdfa015fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 --2- 172.21.15.129:0/1980335540 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdfa0141f60 0x7fdfa0157da0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 >> 172.21.15.129:0/1980335540 conn(0x7fdfa006d560 msgr2=0x7fdfa0154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 shutdown_connections 2024-09-17T04:03:27.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.213+0000 7fdfa7000640 1 -- 172.21.15.129:0/1980335540 wait complete. 2024-09-17T04:03:27.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:03:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 -- 172.21.15.129:0/976708287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 msgr2=0x7fa80c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 --2- 172.21.15.129:0/976708287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa80c003e70 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fa818009b50 tx=0x7fa80c005830 comp rx=0 tx=0).stop 2024-09-17T04:03:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa826800640 1 -- 172.21.15.129:0/976708287 reap_dead start 2024-09-17T04:03:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 -- 172.21.15.129:0/976708287 shutdown_connections 2024-09-17T04:03:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 -- 172.21.15.129:0/976708287 >> 172.21.15.129:0/976708287 conn(0x7fa8280b23c0 msgr2=0x7fa8280b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:27.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 -- 172.21.15.129:0/976708287 shutdown_connections 2024-09-17T04:03:27.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.463+0000 7fa82ce00640 1 -- 172.21.15.129:0/976708287 wait complete. 2024-09-17T04:03:27.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.465+0000 7fa82ce00640 1 Processor -- start 2024-09-17T04:03:27.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.465+0000 7fa82ce00640 1 -- start start 2024-09-17T04:03:27.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:27.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa828142d60 0x7fa828073980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:27.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa828143170 0x7fa828073ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa82815d0f0 con 0x7fa80c003a00 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa82815cf70 con 0x7fa828142d60 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa82ce00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa82815d270 con 0x7fa828143170 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39158/0 (socket says 172.21.15.129:39158) 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 -- 172.21.15.129:0/2244071039 learned_addr learned my addr 172.21.15.129:0/2244071039 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa81d400640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa828142d60 0x7fa828073980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa826800640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa828143170 0x7fa828073ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 -- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa828143170 msgr2=0x7fa828073ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa828143170 0x7fa828073ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 -- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa828142d60 msgr2=0x7fa828073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:27.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa828142d60 0x7fa828073980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:27.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.467+0000 7fa825e00640 1 -- 172.21.15.129:0/2244071039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8281f32e0 con 0x7fa80c003a00 2024-09-17T04:03:27.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa825e00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa818036040 tx=0x7fa818033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:27.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818038040 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa81802b7e0 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa818004260 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8281f6a70 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.468+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8281f7030 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.469+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8280b5de0 con 0x7fa80c003a00 2024-09-17T04:03:27.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.470+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa818045020 con 0x7fa80c003a00 2024-09-17T04:03:27.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.471+0000 7fa81ea00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 0x7fa80407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:27.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.471+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(193..193 src has 1..193) ==== 6026+0+0 (secure 0 0 0) 0x7fa8180bb7a0 con 0x7fa80c003a00 2024-09-17T04:03:27.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.472+0000 7fa81d400640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 0x7fa80407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:27.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.472+0000 7fa81d400640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 0x7fa80407b0a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa810001860 tx=0x7fa810000ed0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:27.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.475+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa8281f32e0 con 0x7fa80c003a00 2024-09-17T04:03:27.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:27.657+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa8280ce240 con 0x7fa80c003a00 2024-09-17T04:03:28.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.222+0000 7fa81ea00640 1 -- 172.21.15.129:0/2244071039 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v194) ==== 181+0+0 (secure 0 0 0) 0x7fa818083a90 con 0x7fa80c003a00 2024-09-17T04:03:28.222 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.225+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 msgr2=0x7fa80407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.225+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 0x7fa80407b0a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa810001860 tx=0x7fa810000ed0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.225+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 msgr2=0x7fa8280cb120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.225+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fa818036040 tx=0x7fa818033670 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 shutdown_connections 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa804078b90 0x7fa80407b0a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa828143170 0x7fa828073ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa828142d60 0x7fa828073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 --2- 172.21.15.129:0/2244071039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa80c003a00 0x7fa8280cb120 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 >> 172.21.15.129:0/2244071039 conn(0x7fa8280b23c0 msgr2=0x7fa82815ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 shutdown_connections 2024-09-17T04:03:28.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.226+0000 7fa82ce00640 1 -- 172.21.15.129:0/2244071039 wait complete. 2024-09-17T04:03:28.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:03:28.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 -- 172.21.15.129:0/3480049532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 msgr2=0x7fd03c004080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 --2- 172.21.15.129:0/3480049532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd03c004080 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fd048001830 tx=0x7fd03c005830 comp rx=0 tx=0).stop 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 -- 172.21.15.129:0/3480049532 shutdown_connections 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 --2- 172.21.15.129:0/3480049532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd03c004080 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 -- 172.21.15.129:0/3480049532 >> 172.21.15.129:0/3480049532 conn(0x7fd0580b23c0 msgr2=0x7fd0580b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 -- 172.21.15.129:0/3480049532 shutdown_connections 2024-09-17T04:03:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.486+0000 7fd05e600640 1 -- 172.21.15.129:0/3480049532 wait complete. 2024-09-17T04:03:28.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.488+0000 7fd05e600640 1 Processor -- start 2024-09-17T04:03:28.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.488+0000 7fd05e600640 1 -- start start 2024-09-17T04:03:28.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:28.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd058144020 0x7fd05815c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0581f67a0 0x7fd0580b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0580d02d0 con 0x7fd03c003c30 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd0580d0150 con 0x7fd058144020 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd05e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd0580b5de0 con 0x7fd0581f67a0 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39180/0 (socket says 172.21.15.129:39180) 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 -- 172.21.15.129:0/1271948866 learned_addr learned my addr 172.21.15.129:0/1271948866 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 -- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0581f67a0 msgr2=0x7fd0580b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0581f67a0 0x7fd0580b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 -- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd058144020 msgr2=0x7fd05815c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd058144020 0x7fd05815c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.490+0000 7fd057400640 1 -- 172.21.15.129:0/1271948866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05815c7b0 con 0x7fd03c003c30 2024-09-17T04:03:28.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd057400640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd04802a9f0 tx=0x7fd0480060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:28.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048043070 con 0x7fd03c003c30 2024-09-17T04:03:28.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0480038e0 con 0x7fd03c003c30 2024-09-17T04:03:28.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0580705d0 con 0x7fd03c003c30 2024-09-17T04:03:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048003cd0 con 0x7fd03c003c30 2024-09-17T04:03:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.491+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581f5fa0 con 0x7fd03c003c30 2024-09-17T04:03:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.492+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0581f5ae0 con 0x7fd03c003c30 2024-09-17T04:03:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.493+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd048046070 con 0x7fd03c003c30 2024-09-17T04:03:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.494+0000 7fd04fe00640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 0x7fd03407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.494+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(194..194 src has 1..194) ==== 5588+0+0 (secure 0 0 0) 0x7fd0480baac0 con 0x7fd03c003c30 2024-09-17T04:03:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.495+0000 7fd04ea00640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 0x7fd03407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.495+0000 7fd04ea00640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 0x7fd03407b0a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd0580aed70 tx=0x7fd040009810 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:28.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.498+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0581f5ae0 con 0x7fd03c003c30 2024-09-17T04:03:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:28.682+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd058072d70 con 0x7fd03c003c30 2024-09-17T04:03:29.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.228+0000 7fd04fe00640 1 -- 172.21.15.129:0/1271948866 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v195) ==== 169+0+0 (secure 0 0 0) 0x7fd05815c7b0 con 0x7fd03c003c30 2024-09-17T04:03:29.229 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:03:29.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.233+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 msgr2=0x7fd03407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:29.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.233+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 0x7fd03407b0a0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fd0580aed70 tx=0x7fd040009810 comp rx=0 tx=0).stop 2024-09-17T04:03:29.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.233+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 msgr2=0x7fd05806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:29.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.233+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fd04802a9f0 tx=0x7fd0480060e0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.235+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 shutdown_connections 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.235+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd034078b90 0x7fd03407b0a0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0581f67a0 0x7fd0580b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd058144020 0x7fd05815c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 --2- 172.21.15.129:0/1271948866 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd03c003c30 0x7fd05806fcd0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 >> 172.21.15.129:0/1271948866 conn(0x7fd0580b23c0 msgr2=0x7fd0580cc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 shutdown_connections 2024-09-17T04:03:29.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.236+0000 7fd05e600640 1 -- 172.21.15.129:0/1271948866 wait complete. 2024-09-17T04:03:29.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.368 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:03:29.368 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:03:29.368 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.368 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:03:29.368 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.369 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.370 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.371 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.372 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:03:29.373 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.374 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.375 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.378 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.379 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.380 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 100000000 2024-09-17T04:03:29.381 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_ops = 16 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.382 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.383 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.384 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:03:29.385 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.386 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.387 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.388 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.389 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.390 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:03:29.391 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:03:29.392 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 100000000 2024-09-17T04:03:29.393 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_ops = 16 2024-09-17T04:03:29.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.509 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.510 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.511 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:03:29.512 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.513 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.514 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.515 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:03:29.516 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.517 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.518 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:03:29.519 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:03:29.520 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 100000000 2024-09-17T04:03:29.522 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.523 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.524 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.525 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.526 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:03:29.527 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:03:29.528 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.529 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.530 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.531 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.532 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:03:29.533 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:03:29.534 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:03:29.534 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:03:29.534 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 100000000 2024-09-17T04:03:29.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.647 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.648 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.649 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:03:29.650 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.651 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.652 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.653 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.654 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.655 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.656 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:03:29.657 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:03:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:03:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:03:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:03:29.658 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.659 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:03:29.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.661 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.662 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:03:29.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.664 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:03:29.665 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.666 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:03:29.667 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:03:29.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:03:29.669 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:03:29.670 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:03:29.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:03:29.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 -- 172.21.15.129:0/435163321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60500705a0 msgr2=0x7f60500b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:29.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 --2- 172.21.15.129:0/435163321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60500705a0 0x7f60500b4140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6040005fd0 tx=0x7f604002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 -- 172.21.15.129:0/435163321 shutdown_connections 2024-09-17T04:03:29.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 --2- 172.21.15.129:0/435163321 >> v2:172.21.15.181:3300/0 conn(0x7f605006eed0 0x7f60500ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 --2- 172.21.15.129:0/435163321 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f60500705a0 0x7f60500b4140 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 -- 172.21.15.129:0/435163321 >> 172.21.15.129:0/435163321 conn(0x7f605006d410 msgr2=0x7f60500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:29.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 -- 172.21.15.129:0/435163321 shutdown_connections 2024-09-17T04:03:29.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.899+0000 7f6057600640 1 -- 172.21.15.129:0/435163321 wait complete. 2024-09-17T04:03:29.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.901+0000 7f6057600640 1 Processor -- start 2024-09-17T04:03:29.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.901+0000 7f6057600640 1 -- start start 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.902+0000 7f6057600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 0x7f60500c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.902+0000 7f6057600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f605006f2e0 0x7f60501f42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6057600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 0x7f60500c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6057600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6050160200 con 0x7f60500705a0 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6057600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f60501612e0 con 0x7f605006f2e0 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6057600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f60500cd1c0 con 0x7f605006eed0 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6055600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 0x7f60500c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 0x7f60500c7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6055600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 0x7f60500c6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39190/0 (socket says 172.21.15.129:39190) 2024-09-17T04:03:29.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6055600640 1 -- 172.21.15.129:0/1178531481 learned_addr learned my addr 172.21.15.129:0/1178531481 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:29.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6047e00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f605006f2e0 0x7f60501f42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 -- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f605006f2e0 msgr2=0x7f60501f42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f605006f2e0 0x7f60501f42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 -- 172.21.15.129:0/1178531481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 msgr2=0x7f60500c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 0x7f60500c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.903+0000 7f6054c00640 1 -- 172.21.15.129:0/1178531481 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60500c7a90 con 0x7f605006eed0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f6054c00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 0x7f60500c7550 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6040031000 tx=0x7f6040005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f604003e030 con 0x7f605006eed0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6040003a70 con 0x7f605006eed0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040035040 con 0x7f605006eed0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60501f4820 con 0x7f605006eed0 2024-09-17T04:03:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.904+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6050156420 con 0x7f605006eed0 2024-09-17T04:03:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.905+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60500ccc80 con 0x7f605006eed0 2024-09-17T04:03:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.906+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6040041030 con 0x7f605006eed0 2024-09-17T04:03:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.907+0000 7f604d600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 0x7f602c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.907+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7f60400bb1f0 con 0x7f605006eed0 2024-09-17T04:03:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.908+0000 7f6047e00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 0x7f602c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:29.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.908+0000 7f6047e00640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 0x7f602c07b0d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6050159540 tx=0x7f6038005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:29.912+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6040083300 con 0x7f605006eed0 2024-09-17T04:03:30.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.096+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f605006a140 con 0x7f605006eed0 2024-09-17T04:03:30.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.104+0000 7f604d600640 1 -- 172.21.15.129:0/1178531481 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v61) ==== 71+0+0 (secure 0 0 0) 0x7f6040087d30 con 0x7f605006eed0 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 msgr2=0x7f602c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 0x7f602c07b0d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f6050159540 tx=0x7f6038005f90 comp rx=0 tx=0).stop 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 msgr2=0x7f60500c7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 0x7f60500c7550 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6040031000 tx=0x7f6040005750 comp rx=0 tx=0).stop 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 shutdown_connections 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f602c078bc0 0x7f602c07b0d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60500705a0 0x7f60500c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f605006f2e0 0x7f60501f42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 --2- 172.21.15.129:0/1178531481 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f605006eed0 0x7f60500c7550 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 >> 172.21.15.129:0/1178531481 conn(0x7f605006d410 msgr2=0x7f60501609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:30.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 shutdown_connections 2024-09-17T04:03:30.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.106+0000 7f6057600640 1 -- 172.21.15.129:0/1178531481 wait complete. 2024-09-17T04:03:30.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle' 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 -- 172.21.15.129:0/542954155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0b46e0 msgr2=0x7f0c3c0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 --2- 172.21.15.129:0/542954155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0b46e0 0x7f0c3c0b4af0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f0c24005fd0 tx=0x7f0c2402a540 comp rx=0 tx=0).stop 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 -- 172.21.15.129:0/542954155 shutdown_connections 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 --2- 172.21.15.129:0/542954155 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0b46e0 0x7f0c3c0b4af0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 --2- 172.21.15.129:0/542954155 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0cfb60 0x7f0c3c160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 -- 172.21.15.129:0/542954155 >> 172.21.15.129:0/542954155 conn(0x7f0c3c06cfc0 msgr2=0x7f0c3c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 -- 172.21.15.129:0/542954155 shutdown_connections 2024-09-17T04:03:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.354+0000 7f0c41800640 1 -- 172.21.15.129:0/542954155 wait complete. 2024-09-17T04:03:30.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.356+0000 7f0c41800640 1 Processor -- start 2024-09-17T04:03:30.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.356+0000 7f0c41800640 1 -- start start 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0b46e0 0x7f0c3c1429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0cfb60 0x7f0c3c1f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c15d270 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c15d0f0 con 0x7f0c3c0cfb60 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c41800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0c3c15d3f0 con 0x7f0c3c0b46e0 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c40e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c40e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39194/0 (socket says 172.21.15.129:39194) 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c40e00640 1 -- 172.21.15.129:0/3634133063 learned_addr learned my addr 172.21.15.129:0/3634133063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c3be00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0b46e0 0x7f0c3c1429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.358+0000 7f0c33400640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0cfb60 0x7f0c3c1f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 -- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0b46e0 msgr2=0x7f0c3c1429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0b46e0 0x7f0c3c1429d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 -- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0cfb60 msgr2=0x7f0c3c1f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0cfb60 0x7f0c3c1f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 -- 172.21.15.129:0/3634133063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3c0c5b50 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c40e00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c760 tx=0x7f0c3400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34003b70 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0c34003d10 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3c0c5e30 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34005330 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.359+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c3c1f7230 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.360+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c3c15d8a0 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.362+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0c340054d0 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.362+0000 7f0c38c00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 0x7f0c1807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.363+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7f0c3409a680 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.363+0000 7f0c3be00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 0x7f0c1807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:30.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.363+0000 7f0c3be00640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 0x7f0c1807b0d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0b4450 tx=0x7f0c2c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:30.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.367+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0c34062710 con 0x7f0c3c0d02c0 2024-09-17T04:03:30.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:30.618+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]} v 0) -- 0x7f0c3c06a140 con 0x7f0c3c0d02c0 2024-09-17T04:03:31.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.238+0000 7f0c38c00640 1 -- 172.21.15.129:0/3634133063 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_ops_throttle"]}]=0 v706) ==== 132+0+0 (secure 0 0 0) 0x7f0c34067140 con 0x7f0c3c0d02c0 2024-09-17T04:03:31.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 msgr2=0x7f0c1807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 0x7f0c1807b0d0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0b4450 tx=0x7f0c2c005f90 comp rx=0 tx=0).stop 2024-09-17T04:03:31.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 msgr2=0x7f0c3c1e8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c760 tx=0x7f0c3400cb60 comp rx=0 tx=0).stop 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 shutdown_connections 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0c18078bc0 0x7f0c1807b0d0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0c3c0d02c0 0x7f0c3c1e8960 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0c3c0cfb60 0x7f0c3c1f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 --2- 172.21.15.129:0/3634133063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0c3c0b46e0 0x7f0c3c1429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 >> 172.21.15.129:0/3634133063 conn(0x7f0c3c06cfc0 msgr2=0x7f0c3c0cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 shutdown_connections 2024-09-17T04:03:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.241+0000 7f0c41800640 1 -- 172.21.15.129:0/3634133063 wait complete. 2024-09-17T04:03:31.263 INFO:tasks.cephfs_test_runner:test_ops_throttle (tasks.cephfs.test_strays.TestStrays) ... ok 2024-09-17T04:03:31.264 INFO:tasks.cephfs_test_runner:Starting test: test_purge_on_shutdown (tasks.cephfs.test_strays.TestStrays) That when an MDS rank is shut down, its purge queue is 2024-09-17T04:03:31.264 INFO:tasks.cephfs_test_runner:test_purge_on_shutdown (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:03:31.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:31.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 -- 172.21.15.129:0/1740016390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b81426c0 msgr2=0x7fd9b8142ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/1740016390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b81426c0 0x7fd9b8142ad0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009ca0 tx=0x7fd9b002a9a0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 -- 172.21.15.129:0/1740016390 shutdown_connections 2024-09-17T04:03:31.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/1740016390 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b81426c0 0x7fd9b8142ad0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/1740016390 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b80ccd60 0x7fd9b8160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 -- 172.21.15.129:0/1740016390 >> 172.21.15.129:0/1740016390 conn(0x7fd9b806cfc0 msgr2=0x7fd9b806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:31.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 -- 172.21.15.129:0/1740016390 shutdown_connections 2024-09-17T04:03:31.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.486+0000 7fd9bcc00640 1 -- 172.21.15.129:0/1740016390 wait complete. 2024-09-17T04:03:31.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.488+0000 7fd9bcc00640 1 Processor -- start 2024-09-17T04:03:31.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.488+0000 7fd9bcc00640 1 -- start start 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9b80ccd60 0x7fd9b813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 0x7fd9b81f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80ce310 con 0x7fd9b80b51e0 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80ce190 con 0x7fd9b80ccd60 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9bcc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b80ce490 con 0x7fd9b81426c0 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39220/0 (socket says 172.21.15.129:39220) 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.490+0000 7fd9b7400640 1 -- 172.21.15.129:0/3032491913 learned_addr learned my addr 172.21.15.129:0/3032491913 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:31.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7e00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 0x7fd9b81f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9aea00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9b80ccd60 0x7fd9b813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 -- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 msgr2=0x7fd9b81f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 0x7fd9b81f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 -- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9b80ccd60 msgr2=0x7fd9b813eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9b80ccd60 0x7fd9b813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 -- 172.21.15.129:0/3032491913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b813f0d0 con 0x7fd9b80b51e0 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7e00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 0x7fd9b81f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:31.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9b7400640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b3f0 tx=0x7fd9a800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8015020 con 0x7fd9b80b51e0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9a8011920 con 0x7fd9b80b51e0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b80ca190 con 0x7fd9b80b51e0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8011d70 con 0x7fd9b80b51e0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.491+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b80ca750 con 0x7fd9b80b51e0 2024-09-17T04:03:31.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.493+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b80ce920 con 0x7fd9b80b51e0 2024-09-17T04:03:31.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.494+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd9a8018020 con 0x7fd9b80b51e0 2024-09-17T04:03:31.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.494+0000 7fd9afe00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 0x7fd99407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.495+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7fd9a809a690 con 0x7fd9b80b51e0 2024-09-17T04:03:31.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.495+0000 7fd9aea00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 0x7fd99407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.496+0000 7fd9aea00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 0x7fd99407b0a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00058c0 tx=0x7fd9a0005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:31.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.499+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd9a801f030 con 0x7fd9b80b51e0 2024-09-17T04:03:31.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.684+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9b806a140 con 0x7fd9b80b51e0 2024-09-17T04:03:31.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.685+0000 7fd9afe00640 1 -- 172.21.15.129:0/3032491913 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v195) ==== 74+0+13224 (secure 0 0 0) 0x7fd9b813f0d0 con 0x7fd9b80b51e0 2024-09-17T04:03:31.685 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:31.685 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":195,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:03:29.223819+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":184,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6833/2632602393":"2024-09-18T04:03:26.204367+0000","172.21.15.129:6832/2632602393":"2024-09-18T04:03:26.204367+0000","172.21.15.129:6835/3531032601":"2024-09-18T04:01:58.368557+0000","172.21.15.129:6832/3531032601":"2024-09-18T04:01:58.368557+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:31.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 msgr2=0x7fd99407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 0x7fd99407b0a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd9a00058c0 tx=0x7fd9a0005850 comp rx=0 tx=0).stop 2024-09-17T04:03:31.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 msgr2=0x7fd9b80c8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800b3f0 tx=0x7fd9a800b8b0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 shutdown_connections 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd994078b90 0x7fd99407b0a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9b81426c0 0x7fd9b81f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9b80ccd60 0x7fd9b813eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 --2- 172.21.15.129:0/3032491913 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9b80b51e0 0x7fd9b80c8750 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 >> 172.21.15.129:0/3032491913 conn(0x7fd9b806cfc0 msgr2=0x7fd9b81408c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 shutdown_connections 2024-09-17T04:03:31.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.687+0000 7fd9bcc00640 1 -- 172.21.15.129:0/3032491913 wait complete. 2024-09-17T04:03:31.713 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:03:31.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 -- 172.21.15.129:0/16443413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 msgr2=0x7fc2100b1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 --2- 172.21.15.129:0/16443413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2100b1e20 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7fc21015ce40 tx=0x7fc1f802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 -- 172.21.15.129:0/16443413 shutdown_connections 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 --2- 172.21.15.129:0/16443413 >> v2:172.21.15.181:3300/0 conn(0x7fc2100b2360 0x7fc21006eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 --2- 172.21.15.129:0/16443413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2100b1e20 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 --2- 172.21.15.129:0/16443413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b3f70 0x7fc2100b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 -- 172.21.15.129:0/16443413 >> 172.21.15.129:0/16443413 conn(0x7fc2100ccc40 msgr2=0x7fc2100cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 -- 172.21.15.129:0/16443413 shutdown_connections 2024-09-17T04:03:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.937+0000 7fc214c00640 1 -- 172.21.15.129:0/16443413 wait complete. 2024-09-17T04:03:31.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.940+0000 7fc214c00640 1 Processor -- start 2024-09-17T04:03:31.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.940+0000 7fc214c00640 1 -- start start 2024-09-17T04:03:31.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b2360 0x7fc2101f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 0x7fc2101f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc21006fae0 con 0x7fc210074580 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc21006f960 con 0x7fc2101f5260 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc214c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc21006fc60 con 0x7fc2100b2360 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.941+0000 7fc20de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39238/0 (socket says 172.21.15.129:39238) 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 -- 172.21.15.129:0/3339298014 learned_addr learned my addr 172.21.15.129:0/3339298014 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc207e00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b2360 0x7fc2101f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20e800640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 0x7fc2101f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 -- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b2360 msgr2=0x7fc2101f59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b2360 0x7fc2101f59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 -- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 msgr2=0x7fc2101f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 0x7fc2101f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 -- 172.21.15.129:0/3339298014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2101f8c00 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20e800640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 0x7fc2101f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.942+0000 7fc20de00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e7c0 tx=0x7fc20000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.943+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc20000ccc0 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.943+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc20000ce80 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.943+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc200005280 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.943+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2101f8ec0 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.943+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2101f9480 con 0x7fc210074580 2024-09-17T04:03:31.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.944+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2100b3f70 con 0x7fc210074580 2024-09-17T04:03:31.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.945+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc200005420 con 0x7fc210074580 2024-09-17T04:03:31.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.946+0000 7fc206000640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 0x7fc1ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:31.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.947+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7fc20009b420 con 0x7fc210074580 2024-09-17T04:03:31.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.947+0000 7fc207e00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 0x7fc1ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:31.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.948+0000 7fc207e00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 0x7fc1ec07b0a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8033000 tx=0x7fc1f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:31.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:31.951+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc200063a60 con 0x7fc210074580 2024-09-17T04:03:32.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.134+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc210074340 con 0x7fc210074580 2024-09-17T04:03:32.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.134+0000 7fc206000640 1 -- 172.21.15.129:0/3339298014 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v195) ==== 93+0+10 (secure 0 0 0) 0x7fc200063360 con 0x7fc210074580 2024-09-17T04:03:32.134 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:03:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 msgr2=0x7fc1ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 0x7fc1ec07b0a0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8033000 tx=0x7fc1f8004730 comp rx=0 tx=0).stop 2024-09-17T04:03:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 msgr2=0x7fc2101f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fc20000e7c0 tx=0x7fc20000ec80 comp rx=0 tx=0).stop 2024-09-17T04:03:32.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 shutdown_connections 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc1ec078b90 0x7fc1ec07b0a0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2101f5260 0x7fc2101f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2100b2360 0x7fc2101f59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 --2- 172.21.15.129:0/3339298014 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc210074580 0x7fc2101f4d20 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 >> 172.21.15.129:0/3339298014 conn(0x7fc2100ccc40 msgr2=0x7fc2100c9ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 shutdown_connections 2024-09-17T04:03:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.137+0000 7fc214c00640 1 -- 172.21.15.129:0/3339298014 wait complete. 2024-09-17T04:03:32.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown' 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 -- 172.21.15.129:0/3860328513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 msgr2=0x7fb2e00cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 --2- 172.21.15.129:0/3860328513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cb7c0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8005fd0 tx=0x7fb2c802a530 comp rx=0 tx=0).stop 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 -- 172.21.15.129:0/3860328513 shutdown_connections 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 --2- 172.21.15.129:0/3860328513 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00cc3c0 0x7fb2e00cb7c0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 --2- 172.21.15.129:0/3860328513 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00b5fa0 0x7fb2e00cd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 -- 172.21.15.129:0/3860328513 >> 172.21.15.129:0/3860328513 conn(0x7fb2e006ee70 msgr2=0x7fb2e006f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 -- 172.21.15.129:0/3860328513 shutdown_connections 2024-09-17T04:03:32.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.382+0000 7fb2e5400640 1 -- 172.21.15.129:0/3860328513 wait complete. 2024-09-17T04:03:32.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.384+0000 7fb2e5400640 1 Processor -- start 2024-09-17T04:03:32.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.385+0000 7fb2e5400640 1 -- start start 2024-09-17T04:03:32.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:32.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00cc3c0 0x7fb2e014c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:32.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00aef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e00b3670 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e00b34f0 con 0x7fb2e015c730 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2e5400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e00b37f0 con 0x7fb2e00cc3c0 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39250/0 (socket says 172.21.15.129:39250) 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.386+0000 7fb2dfe00640 1 -- 172.21.15.129:0/1668110863 learned_addr learned my addr 172.21.15.129:0/1668110863 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2e4a00640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2d7400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00cc3c0 0x7fb2e014c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 -- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00cc3c0 msgr2=0x7fb2e014c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00cc3c0 0x7fb2e014c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 -- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 msgr2=0x7fb2e00aef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:32.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00aef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 -- 172.21.15.129:0/1668110863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e0154a60 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dfe00640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000b440 tx=0x7fb2d000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0015020 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2d0011980 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e0154d40 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.387+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0011dd0 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.388+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e0155300 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.388+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e00cd290 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.390+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb2d0018020 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.391+0000 7fb2dcc00640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 0x7fb2bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:32.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.391+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7fb2d009ac40 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.391+0000 7fb2d7400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 0x7fb2bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:32.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.392+0000 7fb2d7400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 0x7fb2bc07b0a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8033000 tx=0x7fb2c8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:32.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.395+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb2d001f020 con 0x7fb2e00b5fa0 2024-09-17T04:03:32.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:32.649+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]} v 0) -- 0x7fb2e006fa20 con 0x7fb2e00b5fa0 2024-09-17T04:03:33.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.246+0000 7fb2dcc00640 1 -- 172.21.15.129:0/1668110863 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]}]=0 v708) ==== 140+0+0 (secure 0 0 0) 0x7fb2d000d040 con 0x7fb2e00b5fa0 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 msgr2=0x7fb2bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 0x7fb2bc07b0a0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8033000 tx=0x7fb2c8004730 comp rx=0 tx=0).stop 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 msgr2=0x7fb2e0156710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000b440 tx=0x7fb2d000b900 comp rx=0 tx=0).stop 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 shutdown_connections 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb2bc078b90 0x7fb2bc07b0a0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2e015c730 0x7fb2e00aef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2e00cc3c0 0x7fb2e014c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 --2- 172.21.15.129:0/1668110863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2e00b5fa0 0x7fb2e0156710 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 >> 172.21.15.129:0/1668110863 conn(0x7fb2e006ee70 msgr2=0x7fb2e0153de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 shutdown_connections 2024-09-17T04:03:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.250+0000 7fb2e5400640 1 -- 172.21.15.129:0/1668110863 wait complete. 2024-09-17T04:03:33.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:03:33.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 -- 172.21.15.129:0/2173532139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f0160cc0 msgr2=0x7f16f00d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 --2- 172.21.15.129:0/2173532139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f0160cc0 0x7f16f00d0600 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f16f00727e0 tx=0x7f16dc02a560 comp rx=0 tx=0).stop 2024-09-17T04:03:33.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 -- 172.21.15.129:0/2173532139 shutdown_connections 2024-09-17T04:03:33.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 --2- 172.21.15.129:0/2173532139 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f0160cc0 0x7f16f00d0600 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 --2- 172.21.15.129:0/2173532139 >> v2:172.21.15.181:3300/0 conn(0x7f16f00b51e0 0x7f16f00ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 -- 172.21.15.129:0/2173532139 >> 172.21.15.129:0/2173532139 conn(0x7f16f006cfc0 msgr2=0x7f16f006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:33.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 -- 172.21.15.129:0/2173532139 shutdown_connections 2024-09-17T04:03:33.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.496+0000 7f16f6600640 1 -- 172.21.15.129:0/2173532139 wait complete. 2024-09-17T04:03:33.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.498+0000 7f16f6600640 1 Processor -- start 2024-09-17T04:03:33.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.498+0000 7f16f6600640 1 -- start start 2024-09-17T04:03:33.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 0x7f16f00b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16f0160cc0 0x7f16f0130960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f16f00ce310 con 0x7f16f01426c0 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f16f00ce190 con 0x7f16f00b51e0 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f6600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f16f00ce490 con 0x7f16f0160cc0 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39268/0 (socket says 172.21.15.129:39268) 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16efe00640 1 -- 172.21.15.129:0/1838836374 learned_addr learned my addr 172.21.15.129:0/1838836374 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:33.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f5c00640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16f0160cc0 0x7f16f0130960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.500+0000 7f16f5200640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 0x7f16f00b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 -- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16f0160cc0 msgr2=0x7f16f0130960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16f0160cc0 0x7f16f0130960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 -- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 msgr2=0x7f16f00b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 0x7f16f00b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 -- 172.21.15.129:0/1838836374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f00b2c20 con 0x7f16f01426c0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16f5200640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 0x7f16f00b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16efe00640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f16dc034040 tx=0x7f16dc0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc03e030 con 0x7f16f01426c0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f16dc0047a0 con 0x7f16f01426c0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f00c7f80 con 0x7f16f01426c0 2024-09-17T04:03:33.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc035040 con 0x7f16f01426c0 2024-09-17T04:03:33.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.501+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f012f940 con 0x7f16f01426c0 2024-09-17T04:03:33.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.502+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f00ce940 con 0x7f16f01426c0 2024-09-17T04:03:33.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.504+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16dc041030 con 0x7f16f01426c0 2024-09-17T04:03:33.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.505+0000 7f16ecc00640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 0x7f16d007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.505+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7f16dc0bb340 con 0x7f16f01426c0 2024-09-17T04:03:33.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.505+0000 7f16f5200640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 0x7f16d007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.506+0000 7f16f5200640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 0x7f16d007b0a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f16e0005fd0 tx=0x7f16e000f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:33.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.509+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16dc083a10 con 0x7f16f01426c0 2024-09-17T04:03:33.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.691+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f16f00c7a30 con 0x7f16f01426c0 2024-09-17T04:03:33.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.691+0000 7f16ecc00640 1 -- 172.21.15.129:0/1838836374 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v61) ==== 59+0+2294 (secure 0 0 0) 0x7f16dc083310 con 0x7f16f01426c0 2024-09-17T04:03:33.691 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:03:33.691 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:03:33.692 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:03:33.693 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:03:33.694 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 msgr2=0x7f16d007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 0x7f16d007b0a0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f16e0005fd0 tx=0x7f16e000f360 comp rx=0 tx=0).stop 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 msgr2=0x7f16f0130420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f16dc034040 tx=0x7f16dc0039e0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 shutdown_connections 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f16d0078b90 0x7f16d007b0a0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f16f0160cc0 0x7f16f0130960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16f01426c0 0x7f16f0130420 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 --2- 172.21.15.129:0/1838836374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16f00b51e0 0x7f16f00b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 >> 172.21.15.129:0/1838836374 conn(0x7f16f006cfc0 msgr2=0x7f16f015f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 shutdown_connections 2024-09-17T04:03:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.694+0000 7f16f6600640 1 -- 172.21.15.129:0/1838836374 wait complete. 2024-09-17T04:03:33.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:03:33.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 -- 172.21.15.129:0/2096895287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 msgr2=0x7f7c4c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 --2- 172.21.15.129:0/2096895287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c4c004c10 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7f7c5400c290 tx=0x7f7c4c005830 comp rx=0 tx=0).stop 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 -- 172.21.15.129:0/2096895287 shutdown_connections 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 --2- 172.21.15.129:0/2096895287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c4c004c10 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 -- 172.21.15.129:0/2096895287 >> 172.21.15.129:0/2096895287 conn(0x7f7c680ccc40 msgr2=0x7f7c680cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 -- 172.21.15.129:0/2096895287 shutdown_connections 2024-09-17T04:03:33.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.945+0000 7f7c6de00640 1 -- 172.21.15.129:0/2096895287 wait complete. 2024-09-17T04:03:33.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.947+0000 7f7c6de00640 1 Processor -- start 2024-09-17T04:03:33.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.947+0000 7f7c6de00640 1 -- start start 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 0x7f7c6814f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 0x7f7c6809d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680b49c0 con 0x7f7c4c004800 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680b4840 con 0x7f7c680c81e0 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c6de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7c680cbf30 con 0x7f7c6814f560 2024-09-17T04:03:33.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c66a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c66a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52912/0 (socket says 172.21.15.129:52912) 2024-09-17T04:03:33.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.949+0000 7f7c66a00640 1 -- 172.21.15.129:0/1451840050 learned_addr learned my addr 172.21.15.129:0/1451840050 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66000640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 0x7f7c6814f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c67400640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 0x7f7c6809d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 -- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 msgr2=0x7f7c6809d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 0x7f7c6809d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 -- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 msgr2=0x7f7c6814f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:33.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 0x7f7c6814f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 -- 172.21.15.129:0/1451840050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c681525c0 con 0x7f7c4c004800 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66000640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 0x7f7c6814f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c67400640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 0x7f7c6809d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c66a00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f80 tx=0x7f7c58006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58015070 con 0x7f7c4c004800 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c58006850 con 0x7f7c4c004800 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c681528a0 con 0x7f7c4c004800 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58001c30 con 0x7f7c4c004800 2024-09-17T04:03:33.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.950+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6814f970 con 0x7f7c4c004800 2024-09-17T04:03:33.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.951+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c680cc440 con 0x7f7c4c004800 2024-09-17T04:03:33.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.952+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7c58011430 con 0x7f7c4c004800 2024-09-17T04:03:33.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.953+0000 7f7c5f400640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 0x7f7c4807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:33.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.953+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7f7c5809b9c0 con 0x7f7c4c004800 2024-09-17T04:03:33.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.954+0000 7f7c66000640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 0x7f7c4807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:33.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.955+0000 7f7c66000640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 0x7f7c4807b0d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7c54033000 tx=0x7f7c5402d000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:33.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:33.958+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c680b4b70 con 0x7f7c4c004800 2024-09-17T04:03:34.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.141+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f7c680704a0 con 0x7f7c4c004800 2024-09-17T04:03:34.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.147+0000 7f7c5f400640 1 -- 172.21.15.129:0/1451840050 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v62)=0 v62) ==== 122+0+0 (secure 0 0 0) 0x7f7c58064000 con 0x7f7c4c004800 2024-09-17T04:03:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 msgr2=0x7f7c4807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 0x7f7c4807b0d0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7c54033000 tx=0x7f7c5402d000 comp rx=0 tx=0).stop 2024-09-17T04:03:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 msgr2=0x7f7c68151d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f7c58000f80 tx=0x7f7c58006720 comp rx=0 tx=0).stop 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 shutdown_connections 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c48078bc0 0x7f7c4807b0d0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c6814f560 0x7f7c6809d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c680c81e0 0x7f7c6814f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 --2- 172.21.15.129:0/1451840050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c004800 0x7f7c68151d70 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 >> 172.21.15.129:0/1451840050 conn(0x7f7c680ccc40 msgr2=0x7f7c680af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 shutdown_connections 2024-09-17T04:03:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.151+0000 7f7c6de00640 1 -- 172.21.15.129:0/1451840050 wait complete. 2024-09-17T04:03:34.176 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:03:34.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:03:34.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:03:34.356 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:03:34.356 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:34.356 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:34.392 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:03:34.392 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:03:34.392 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:34.429 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:03:34.430 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:34.430 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:34.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:03:34.468 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:03:34.468 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:34.506 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:03:34.506 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:03:34.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:34.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 -- 172.21.15.129:0/2166505067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd406eed0 msgr2=0x7f9fd40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 --2- 172.21.15.129:0/2166505067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd406eed0 0x7f9fd40ccc80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4005fd0 tx=0x7f9fc402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 -- 172.21.15.129:0/2166505067 shutdown_connections 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 --2- 172.21.15.129:0/2166505067 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd406eed0 0x7f9fd40ccc80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 --2- 172.21.15.129:0/2166505067 >> v2:172.21.15.181:3300/0 conn(0x7f9fd40705a0 0x7f9fd40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 -- 172.21.15.129:0/2166505067 >> 172.21.15.129:0/2166505067 conn(0x7f9fd406d410 msgr2=0x7f9fd40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 -- 172.21.15.129:0/2166505067 shutdown_connections 2024-09-17T04:03:34.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.731+0000 7f9fdc800640 1 -- 172.21.15.129:0/2166505067 wait complete. 2024-09-17T04:03:34.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.734+0000 7f9fdc800640 1 Processor -- start 2024-09-17T04:03:34.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.734+0000 7f9fdc800640 1 -- start start 2024-09-17T04:03:34.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fd406eed0 0x7f9fd40c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 0x7f9fd4157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd4160200 con 0x7f9fd406f2e0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd41612e0 con 0x7f9fd406eed0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.735+0000 7f9fdc800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd40cd1c0 con 0x7f9fd40705a0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fd9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 0x7f9fd4157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fd9e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fd406eed0 0x7f9fd40c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:56764/0 (socket says 172.21.15.129:56764) 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fd9400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 0x7f9fd4157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52914/0 (socket says 172.21.15.129:52914) 2024-09-17T04:03:34.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 -- 172.21.15.129:0/1881299848 learned_addr learned my addr 172.21.15.129:0/1881299848 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 -- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fd406eed0 msgr2=0x7f9fd40c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fd406eed0 0x7f9fd40c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 -- 172.21.15.129:0/1881299848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 msgr2=0x7f9fd4157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 0x7f9fd4157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 -- 172.21.15.129:0/1881299848 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd40c7320 con 0x7f9fd40705a0 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.736+0000 7f9fda800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c700 tx=0x7f9fd000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.737+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0003b70 con 0x7f9fd40705a0 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.737+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fd0003d10 con 0x7f9fd40705a0 2024-09-17T04:03:34.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.737+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000aab0 con 0x7f9fd40705a0 2024-09-17T04:03:34.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.737+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd4157970 con 0x7f9fd40705a0 2024-09-17T04:03:34.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.737+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd41f7140 con 0x7f9fd40705a0 2024-09-17T04:03:34.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.738+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd41f6d20 con 0x7f9fd40705a0 2024-09-17T04:03:34.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.739+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9fd000ac50 con 0x7f9fd40705a0 2024-09-17T04:03:34.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.740+0000 7f9fce000640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 0x7f9fb807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:34.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.740+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7f9fd009a780 con 0x7f9fd40705a0 2024-09-17T04:03:34.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.740+0000 7f9fd9e00640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 0x7f9fb807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:34.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.741+0000 7f9fd9e00640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 0x7f9fb807b0d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009b30 tx=0x7f9fc8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:34.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.745+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9fd0062800 con 0x7f9fd40705a0 2024-09-17T04:03:34.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.966+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fd40ccc80 con 0x7f9fd40705a0 2024-09-17T04:03:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.967+0000 7f9fce000640 1 -- 172.21.15.129:0/1881299848 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) ==== 95+0+3930 (secure 0 0 0) 0x7f9fd0067230 con 0x7f9fd40705a0 2024-09-17T04:03:34.967 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:34.967 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":174,"btime":"2024-09-17T04:03:27:208078+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8045,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4110004642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4110004642},{"type":"v1","addr":"172.21.15.181:6835","nonce":4110004642}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":165},{"gid":9397,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/1392269036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":1392269036},{"type":"v1","addr":"172.21.15.181:6837","nonce":1392269036}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":166},{"gid":10095,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3284912885","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3284912885},{"type":"v1","addr":"172.21.15.129:6835","nonce":3284912885}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":169},{"gid":10347,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/2432853058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2432853058},{"type":"v1","addr":"172.21.15.129:6833","nonce":2432853058}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":174}],"filesystems":[]} 2024-09-17T04:03:34.968 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 174 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 msgr2=0x7f9fb807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 0x7f9fb807b0d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009b30 tx=0x7f9fc8005f90 comp rx=0 tx=0).stop 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 msgr2=0x7f9fd415a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000c700 tx=0x7f9fd000cbc0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 shutdown_connections 2024-09-17T04:03:34.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fb8078bc0 0x7f9fb807b0d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fd40705a0 0x7f9fd415a2c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fd406f2e0 0x7f9fd4157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 --2- 172.21.15.129:0/1881299848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fd406eed0 0x7f9fd40c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 >> 172.21.15.129:0/1881299848 conn(0x7f9fd406d410 msgr2=0x7f9fd40ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 shutdown_connections 2024-09-17T04:03:34.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:34.970+0000 7f9fdc800640 1 -- 172.21.15.129:0/1881299848 wait complete. 2024-09-17T04:03:34.995 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:03:34.996 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:03:34.996 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:03:34.996 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:03:34.996 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:03:34.996 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:03:34.996 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:03:34.997 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:03:34.997 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:03:34.997 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:03:34.997 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:03:34.997 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:03:34.997 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:03:34.997 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:03:34.998 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:03:34.998 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:03:35.017 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:03:35.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:03:35.021 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:03:35.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:03:35.030 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:03:35.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:03:35.031 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:03:35.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:03:35.065 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:03:35.065 DEBUG:teuthology.parallel:result is None 2024-09-17T04:03:35.068 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:03:35.068 DEBUG:teuthology.parallel:result is None 2024-09-17T04:03:35.075 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:03:35.075 DEBUG:teuthology.parallel:result is None 2024-09-17T04:03:35.077 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:03:35.077 DEBUG:teuthology.parallel:result is None 2024-09-17T04:03:35.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:03:35.263 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:03:35.266 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:03:35.271 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:03:35.272 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 -- 172.21.15.129:0/1096911950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400b4140 msgr2=0x7fa9400ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 --2- 172.21.15.129:0/1096911950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400b4140 0x7fa9400ccc80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009830 tx=0x7fa92c02a510 comp rx=0 tx=0).stop 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 -- 172.21.15.129:0/1096911950 shutdown_connections 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 --2- 172.21.15.129:0/1096911950 >> v2:172.21.15.181:3300/0 conn(0x7fa9400cd1c0 0x7fa9401609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 --2- 172.21.15.129:0/1096911950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400b4140 0x7fa9400ccc80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 -- 172.21.15.129:0/1096911950 >> 172.21.15.129:0/1096911950 conn(0x7fa94006d410 msgr2=0x7fa9400b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 -- 172.21.15.129:0/1096911950 shutdown_connections 2024-09-17T04:03:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.314+0000 7fa946600640 1 -- 172.21.15.129:0/1096911950 wait complete. 2024-09-17T04:03:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.317+0000 7fa946600640 1 Processor -- start 2024-09-17T04:03:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.317+0000 7fa946600640 1 -- start start 2024-09-17T04:03:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 0x7fa9401f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:35.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400cd1c0 0x7fa94014fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa940160260 con 0x7fa9400705a0 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa9401600e0 con 0x7fa9400b4140 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa946600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa940160f00 con 0x7fa9400cd1c0 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa93f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa93f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52946/0 (socket says 172.21.15.129:52946) 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.318+0000 7fa93f400640 1 -- 172.21.15.129:0/236427055 learned_addr learned my addr 172.21.15.129:0/236427055 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93fe00640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400cd1c0 0x7fa94014fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:35.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93ea00640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 0x7fa9401f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 -- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400cd1c0 msgr2=0x7fa94014fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400cd1c0 0x7fa94014fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 -- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 msgr2=0x7fa9401f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 0x7fa9401f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 -- 172.21.15.129:0/236427055 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9400c6c60 con 0x7fa9400705a0 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93ea00640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 0x7fa9401f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa93f400640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa93000cbc0 tx=0x7fa93000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:35.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.319+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930016020 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.320+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9400c76f0 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.320+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa93000ad90 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.320+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930012680 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.320+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9400c7cb0 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.321+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9400ccc80 con 0x7fa9400705a0 2024-09-17T04:03:35.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.322+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa930019020 con 0x7fa9400705a0 2024-09-17T04:03:35.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.323+0000 7fa937400640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 0x7fa92007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:35.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.323+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(195..195 src has 1..195) ==== 5029+0+0 (secure 0 0 0) 0x7fa93009af80 con 0x7fa9400705a0 2024-09-17T04:03:35.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.323+0000 7fa93ea00640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 0x7fa92007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:35.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.324+0000 7fa93ea00640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 0x7fa92007b0a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa92c034000 tx=0x7fa92c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:35.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.328+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa930020020 con 0x7fa9400705a0 2024-09-17T04:03:35.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:35.524+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fa94006a140 con 0x7fa9400705a0 2024-09-17T04:03:36.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.272+0000 7fa937400640 1 -- 172.21.15.129:0/236427055 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v197) ==== 97+0+0 (secure 0 0 0) 0x7fa9400cd5d0 con 0x7fa9400705a0 2024-09-17T04:03:36.272 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 msgr2=0x7fa92007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 0x7fa92007b0a0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fa92c034000 tx=0x7fa92c004730 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 msgr2=0x7fa9400c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa93000cbc0 tx=0x7fa93000a060 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 shutdown_connections 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa920078b90 0x7fa92007b0a0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa9400cd1c0 0x7fa94014fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa9400b4140 0x7fa9401f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 --2- 172.21.15.129:0/236427055 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa9400705a0 0x7fa9400c6720 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.276+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 >> 172.21.15.129:0/236427055 conn(0x7fa94006d410 msgr2=0x7fa9400ad830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:36.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.277+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 shutdown_connections 2024-09-17T04:03:36.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.277+0000 7fa946600640 1 -- 172.21.15.129:0/236427055 wait complete. 2024-09-17T04:03:36.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:03:36.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 -- 172.21.15.129:0/3227604975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c0705a0 msgr2=0x7fa81c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 --2- 172.21.15.129:0/3227604975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c0705a0 0x7fa81c0b4140 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa80c005fd0 tx=0x7fa80c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 -- 172.21.15.129:0/3227604975 shutdown_connections 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 --2- 172.21.15.129:0/3227604975 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c06eed0 0x7fa81c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 --2- 172.21.15.129:0/3227604975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c0705a0 0x7fa81c0b4140 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 -- 172.21.15.129:0/3227604975 >> 172.21.15.129:0/3227604975 conn(0x7fa81c06d410 msgr2=0x7fa81c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 -- 172.21.15.129:0/3227604975 shutdown_connections 2024-09-17T04:03:36.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.526+0000 7fa822000640 1 -- 172.21.15.129:0/3227604975 wait complete. 2024-09-17T04:03:36.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.528+0000 7fa822000640 1 Processor -- start 2024-09-17T04:03:36.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.528+0000 7fa822000640 1 -- start start 2024-09-17T04:03:36.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c06eed0 0x7fa81c1e7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c0705a0 0x7fa81c0c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c160200 con 0x7fa81c06f2e0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c1612e0 con 0x7fa81c06eed0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa822000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa81c0cd1c0 con 0x7fa81c0705a0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa81aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa81aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52948/0 (socket says 172.21.15.129:52948) 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa81aa00640 1 -- 172.21.15.129:0/3836159988 learned_addr learned my addr 172.21.15.129:0/3836159988 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.530+0000 7fa81b400640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c06eed0 0x7fa81c1e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81be00640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c0705a0 0x7fa81c0c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 -- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c0705a0 msgr2=0x7fa81c0c8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c0705a0 0x7fa81c0c8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 -- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c06eed0 msgr2=0x7fa81c1e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c06eed0 0x7fa81c1e7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 -- 172.21.15.129:0/3836159988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c1e63c0 con 0x7fa81c06f2e0 2024-09-17T04:03:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa81aa00640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fa80800e9f0 tx=0x7fa80800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80800ccc0 con 0x7fa81c06f2e0 2024-09-17T04:03:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa808004020 con 0x7fa81c06f2e0 2024-09-17T04:03:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80800ce60 con 0x7fa81c06f2e0 2024-09-17T04:03:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.531+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c1e66a0 con 0x7fa81c06f2e0 2024-09-17T04:03:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.532+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1e6c60 con 0x7fa81c06f2e0 2024-09-17T04:03:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.533+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa81c0ccc80 con 0x7fa81c06f2e0 2024-09-17T04:03:36.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.534+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa8080035f0 con 0x7fa81c06f2e0 2024-09-17T04:03:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.535+0000 7fa813400640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 0x7fa7fc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.535+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 4857+0+0 (secure 0 0 0) 0x7fa80809b230 con 0x7fa81c06f2e0 2024-09-17T04:03:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.535+0000 7fa81b400640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 0x7fa7fc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.536+0000 7fa81b400640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 0x7fa7fc07b0d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa80c033000 tx=0x7fa80c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:36.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.539+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa8080632d0 con 0x7fa81c06f2e0 2024-09-17T04:03:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.792+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fa81c06a140 con 0x7fa81c06f2e0 2024-09-17T04:03:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.793+0000 7fa813400640 1 -- 172.21.15.129:0/3836159988 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) ==== 80+0+5304 (secure 0 0 0) 0x7fa808067d00 con 0x7fa81c06f2e0 2024-09-17T04:03:36.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:36.793 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:03:36.793 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:03:36.793 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.794 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:03:36.794 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:03:36.794 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.794 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.794 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.795 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:03:36.796 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:03:36.796 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.796 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.796 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:03:36.796 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:03:36.797 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.797 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.797 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.797 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.798 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:03:36.798 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:03:36.798 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.798 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.798 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:03:36.799 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:03:36.799 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.799 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.799 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.800 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.800 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:03:36.800 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:03:36.800 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.800 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.801 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.802 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.803 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.803 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.803 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.803 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:03:36.803 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.804 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.805 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.805 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:03:36.805 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:03:36.805 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.806 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:03:36.807 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.808 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:03:36.809 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:03:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.811 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.812 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:03:36.813 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:03:36.814 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:03:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:03:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:03:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:03:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:03:36.816 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:03:36.816 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.816 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:36.816 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:03:36.816 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:03:36.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 msgr2=0x7fa7fc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 0x7fa7fc07b0d0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa80c033000 tx=0x7fa80c004730 comp rx=0 tx=0).stop 2024-09-17T04:03:36.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 msgr2=0x7fa81c1f67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fa80800e9f0 tx=0x7fa80800eeb0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 shutdown_connections 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7fc078bc0 0x7fa7fc07b0d0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa81c0705a0 0x7fa81c0c8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa81c06f2e0 0x7fa81c1f67b0 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 --2- 172.21.15.129:0/3836159988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa81c06eed0 0x7fa81c1e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 >> 172.21.15.129:0/3836159988 conn(0x7fa81c06d410 msgr2=0x7fa81c074960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 shutdown_connections 2024-09-17T04:03:36.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:36.796+0000 7fa822000640 1 -- 172.21.15.129:0/3836159988 wait complete. 2024-09-17T04:03:36.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:03:37.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 -- 172.21.15.129:0/3707804822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40cd390 msgr2=0x7f7ad41609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 --2- 172.21.15.129:0/3707804822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40cd390 0x7f7ad41609c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc02a8a0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 -- 172.21.15.129:0/3707804822 shutdown_connections 2024-09-17T04:03:37.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 --2- 172.21.15.129:0/3707804822 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40cd390 0x7f7ad41609c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 --2- 172.21.15.129:0/3707804822 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40705a0 0x7f7ad40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.043+0000 7f7adc000640 1 -- 172.21.15.129:0/3707804822 >> 172.21.15.129:0/3707804822 conn(0x7f7ad406d410 msgr2=0x7f7ad40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:37.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.044+0000 7f7adc000640 1 -- 172.21.15.129:0/3707804822 shutdown_connections 2024-09-17T04:03:37.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.044+0000 7f7adc000640 1 -- 172.21.15.129:0/3707804822 wait complete. 2024-09-17T04:03:37.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.046+0000 7f7adc000640 1 Processor -- start 2024-09-17T04:03:37.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.046+0000 7f7adc000640 1 -- start start 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 0x7f7ad415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40c5c50 0x7f7ad40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad4160f00 con 0x7f7ad406eed0 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad406f4f0 con 0x7f7ad40705a0 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.047+0000 7f7adc000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad4161080 con 0x7f7ad40c5c50 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52954/0 (socket says 172.21.15.129:52954) 2024-09-17T04:03:37.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 -- 172.21.15.129:0/2306231377 learned_addr learned my addr 172.21.15.129:0/2306231377 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad8c00640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 0x7f7ad415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ada000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40c5c50 0x7f7ad40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 -- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40c5c50 msgr2=0x7f7ad40ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40c5c50 0x7f7ad40ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 -- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 msgr2=0x7f7ad415b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 0x7f7ad415b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 -- 172.21.15.129:0/2306231377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad40c73c0 con 0x7f7ad406eed0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad8c00640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 0x7f7ad415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.048+0000 7f7ad9600640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800e7c0 tx=0x7f7ac800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.049+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800ccc0 con 0x7f7ad406eed0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.049+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ac8010430 con 0x7f7ad406eed0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.049+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac8010860 con 0x7f7ad406eed0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.049+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad40c7630 con 0x7f7ad406eed0 2024-09-17T04:03:37.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.049+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad40c7bf0 con 0x7f7ad406eed0 2024-09-17T04:03:37.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.050+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad40b4140 con 0x7f7ad406eed0 2024-09-17T04:03:37.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.051+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7ac8010a00 con 0x7f7ad406eed0 2024-09-17T04:03:37.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.052+0000 7f7ad1600640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 0x7f7ab407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.052+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 4857+0+0 (secure 0 0 0) 0x7f7ac809b460 con 0x7f7ad406eed0 2024-09-17T04:03:37.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.053+0000 7f7ad8c00640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 0x7f7ab407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.053+0000 7f7ad8c00640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 0x7f7ab407b0d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7ad41f31b0 tx=0x7f7ac0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:37.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.057+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7ac8063b70 con 0x7f7ad406eed0 2024-09-17T04:03:37.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.242+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7ad406a140 con 0x7f7ad406eed0 2024-09-17T04:03:37.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.242+0000 7f7ad1600640 1 -- 172.21.15.129:0/2306231377 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v199) ==== 84+0+65 (secure 0 0 0) 0x7f7ac8063470 con 0x7f7ad406eed0 2024-09-17T04:03:37.242 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:37.242 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:03:37.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:03:37.243 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:03:37.243 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:03:37.243 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:03:37.243 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:03:37.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.244+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 msgr2=0x7f7ab407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.244+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 0x7f7ab407b0d0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f7ad41f31b0 tx=0x7f7ac0009040 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.244+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 msgr2=0x7f7ad40c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.244+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800e7c0 tx=0x7f7ac800ec80 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 shutdown_connections 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ab4078bc0 0x7f7ab407b0d0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ad40c5c50 0x7f7ad40ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ad40705a0 0x7f7ad415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 --2- 172.21.15.129:0/2306231377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ad406eed0 0x7f7ad40c5710 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 >> 172.21.15.129:0/2306231377 conn(0x7f7ad406d410 msgr2=0x7f7ad40ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 shutdown_connections 2024-09-17T04:03:37.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.245+0000 7f7adc000640 1 -- 172.21.15.129:0/2306231377 wait complete. 2024-09-17T04:03:37.267 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:03:37.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:03:37.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 -- 172.21.15.129:0/4177579003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6780cba90 msgr2=0x7fa6780cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 --2- 172.21.15.129:0/4177579003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6780cba90 0x7fa6780cbea0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7fa670005fd0 tx=0x7fa67002aa10 comp rx=0 tx=0).stop 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 -- 172.21.15.129:0/4177579003 shutdown_connections 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 --2- 172.21.15.129:0/4177579003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa6780cba90 0x7fa6780cbea0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 --2- 172.21.15.129:0/4177579003 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 0x7fa6780b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 -- 172.21.15.129:0/4177579003 >> 172.21.15.129:0/4177579003 conn(0x7fa6780ccc40 msgr2=0x7fa6780cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 -- 172.21.15.129:0/4177579003 shutdown_connections 2024-09-17T04:03:37.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.494+0000 7fa67f000640 1 -- 172.21.15.129:0/4177579003 wait complete. 2024-09-17T04:03:37.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.496+0000 7fa67f000640 1 Processor -- start 2024-09-17T04:03:37.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.496+0000 7fa67f000640 1 -- start start 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 0x7fa67815f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6780cba90 0x7fa678073410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cc930 con 0x7fa678073950 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa6780cc7b0 con 0x7fa6780cba90 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa678166f20 con 0x7fa678074580 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52962/0 (socket says 172.21.15.129:52962) 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.498+0000 7fa67d000640 1 -- 172.21.15.129:0/3781067451 learned_addr learned my addr 172.21.15.129:0/3781067451 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa66f400640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6780cba90 0x7fa678073410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa677e00640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 0x7fa67815f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 -- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 msgr2=0x7fa67815f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 0x7fa67815f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 -- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6780cba90 msgr2=0x7fa678073410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:37.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6780cba90 0x7fa678073410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:37.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 -- 172.21.15.129:0/3781067451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa678073d60 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67d000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fa670009d90 tx=0x7fa670003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670033050 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa670004450 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa670004860 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6781653f0 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.499+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6781659b0 con 0x7fa678073950 2024-09-17T04:03:37.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.501+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa678165140 con 0x7fa678073950 2024-09-17T04:03:37.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.502+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa670036070 con 0x7fa678073950 2024-09-17T04:03:37.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.503+0000 7fa674c00640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 0x7fa65407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:37.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.503+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 4857+0+0 (secure 0 0 0) 0x7fa6700bbf00 con 0x7fa678073950 2024-09-17T04:03:37.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.503+0000 7fa677e00640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 0x7fa65407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:37.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.504+0000 7fa677e00640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 0x7fa65407b0a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa6680015e0 tx=0x7fa668009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:37.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.507+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa6700845f0 con 0x7fa678073950 2024-09-17T04:03:37.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:37.690+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa6780b3f70 con 0x7fa678073950 2024-09-17T04:03:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.295+0000 7fa674c00640 1 -- 172.21.15.129:0/3781067451 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v200) ==== 138+0+0 (secure 0 0 0) 0x7fa670083ef0 con 0x7fa678073950 2024-09-17T04:03:38.294 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.297+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 msgr2=0x7fa65407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.297+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 0x7fa65407b0a0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa6680015e0 tx=0x7fa668009040 comp rx=0 tx=0).stop 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.297+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 msgr2=0x7fa6780acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.297+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7fa670009d90 tx=0x7fa670003f10 comp rx=0 tx=0).stop 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 shutdown_connections 2024-09-17T04:03:38.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa654078b90 0x7fa65407b0a0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa678073950 0x7fa6780acf50 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa6780cba90 0x7fa678073410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 --2- 172.21.15.129:0/3781067451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa678074580 0x7fa67815f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 >> 172.21.15.129:0/3781067451 conn(0x7fa6780ccc40 msgr2=0x7fa678160940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 shutdown_connections 2024-09-17T04:03:38.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.298+0000 7fa67f000640 1 -- 172.21.15.129:0/3781067451 wait complete. 2024-09-17T04:03:38.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 -- 172.21.15.129:0/1180822637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 msgr2=0x7fb1c415b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1180822637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415b6f0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4005fd0 tx=0x7fb1b402a670 comp rx=0 tx=0).stop 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 -- 172.21.15.129:0/1180822637 shutdown_connections 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1180822637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415b6f0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 -- 172.21.15.129:0/1180822637 >> 172.21.15.129:0/1180822637 conn(0x7fb1c406d7a0 msgr2=0x7fb1c40b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 -- 172.21.15.129:0/1180822637 shutdown_connections 2024-09-17T04:03:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.557+0000 7fb1ca600640 1 -- 172.21.15.129:0/1180822637 wait complete. 2024-09-17T04:03:38.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.560+0000 7fb1ca600640 1 Processor -- start 2024-09-17T04:03:38.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.560+0000 7fb1ca600640 1 -- start start 2024-09-17T04:03:38.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1c40af550 0x7fb1c414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:38.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1c40af960 0x7fb1c415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c415faf0 con 0x7fb1c40cc500 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c415f970 con 0x7fb1c40af550 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1ca600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c415fc70 con 0x7fb1c40af960 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52974/0 (socket says 172.21.15.129:52974) 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.561+0000 7fb1c3e00640 1 -- 172.21.15.129:0/1613820933 learned_addr learned my addr 172.21.15.129:0/1613820933 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1baa00640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1c40af960 0x7fb1c415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3400640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1c40af550 0x7fb1c414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 -- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1c40af960 msgr2=0x7fb1c415e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1c40af960 0x7fb1c415e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 -- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1c40af550 msgr2=0x7fb1c414ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1c40af550 0x7fb1c414ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 -- 172.21.15.129:0/1613820933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c4159550 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1c3e00640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c730 tx=0x7fb1bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.562+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc003b70 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.563+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb1bc003d10 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.563+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c4159830 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.563+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc00aab0 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.563+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c4159df0 con 0x7fb1c40cc500 2024-09-17T04:03:38.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.564+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c41600c0 con 0x7fb1c40cc500 2024-09-17T04:03:38.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.565+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb1bc00ac50 con 0x7fb1c40cc500 2024-09-17T04:03:38.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.566+0000 7fb1bbe00640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 0x7fb1a007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:38.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.567+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(200..200 src has 1..200) ==== 5227+0+0 (secure 0 0 0) 0x7fb1bc09b400 con 0x7fb1c40cc500 2024-09-17T04:03:38.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.567+0000 7fb1c3400640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 0x7fb1a007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:38.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.568+0000 7fb1c3400640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 0x7fb1a007b0d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4033000 tx=0x7fb1b4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:38.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.571+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb1bc063a90 con 0x7fb1c40cc500 2024-09-17T04:03:38.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:38.759+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb1c41592a0 con 0x7fb1c40cc500 2024-09-17T04:03:39.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.305+0000 7fb1bbe00640 1 -- 172.21.15.129:0/1613820933 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v201) ==== 170+0+0 (secure 0 0 0) 0x7fb1bc063390 con 0x7fb1c40cc500 2024-09-17T04:03:39.304 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:03:39.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.308+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 msgr2=0x7fb1a007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:39.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.308+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 0x7fb1a007b0d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4033000 tx=0x7fb1b4004730 comp rx=0 tx=0).stop 2024-09-17T04:03:39.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.308+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 msgr2=0x7fb1c415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:39.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.308+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00c730 tx=0x7fb1bc00cbf0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 shutdown_connections 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb1a0078bc0 0x7fb1a007b0d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1c40cc500 0x7fb1c415e9e0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1c40af960 0x7fb1c415e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 --2- 172.21.15.129:0/1613820933 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1c40af550 0x7fb1c414ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 >> 172.21.15.129:0/1613820933 conn(0x7fb1c406d7a0 msgr2=0x7fb1c406f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 shutdown_connections 2024-09-17T04:03:39.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.309+0000 7fb1ca600640 1 -- 172.21.15.129:0/1613820933 wait complete. 2024-09-17T04:03:39.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:03:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 -- 172.21.15.129:0/2105243010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff20074800 msgr2=0x7eff200af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 --2- 172.21.15.129:0/2105243010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff20074800 0x7eff200af480 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7eff200b2a20 tx=0x7eff1402aa10 comp rx=0 tx=0).stop 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 -- 172.21.15.129:0/2105243010 shutdown_connections 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 --2- 172.21.15.129:0/2105243010 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2006f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 --2- 172.21.15.129:0/2105243010 >> v2:172.21.15.181:3300/0 conn(0x7eff200b1c40 0x7eff200b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 --2- 172.21.15.129:0/2105243010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff20074800 0x7eff200af480 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 -- 172.21.15.129:0/2105243010 >> 172.21.15.129:0/2105243010 conn(0x7eff200ccc40 msgr2=0x7eff200cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 -- 172.21.15.129:0/2105243010 shutdown_connections 2024-09-17T04:03:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.570+0000 7eff28200640 1 -- 172.21.15.129:0/2105243010 wait complete. 2024-09-17T04:03:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.572+0000 7eff28200640 1 Processor -- start 2024-09-17T04:03:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.572+0000 7eff28200640 1 -- start start 2024-09-17T04:03:39.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:39.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 0x7eff2014e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7eff2006f7f0 con 0x7eff200b1c40 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7eff200b4b50 con 0x7eff20074800 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff28200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7eff2006f970 con 0x7eff2006eea0 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52990/0 (socket says 172.21.15.129:52990) 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 -- 172.21.15.129:0/1295094974 learned_addr learned my addr 172.21.15.129:0/1295094974 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff24e00640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 0x7eff2014e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff25800640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 -- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 msgr2=0x7eff2009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 -- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 msgr2=0x7eff2014e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 0x7eff2014e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff26200640 1 -- 172.21.15.129:0/1295094974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff20151bc0 con 0x7eff200b1c40 2024-09-17T04:03:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff25800640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.574+0000 7eff24e00640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 0x7eff2014e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff26200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7eff1800b410 tx=0x7eff1800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18015020 con 0x7eff200b1c40 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7eff18011940 con 0x7eff200b1c40 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2009c7b0 con 0x7eff200b1c40 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff18011d90 con 0x7eff200b1c40 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.575+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2009afa0 con 0x7eff200b1c40 2024-09-17T04:03:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.576+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff200af480 con 0x7eff200b1c40 2024-09-17T04:03:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.577+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7eff18018020 con 0x7eff200b1c40 2024-09-17T04:03:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.578+0000 7eff1d600640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 0x7eff0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.579+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(201..201 src has 1..201) ==== 5609+0+0 (secure 0 0 0) 0x7eff1809abe0 con 0x7eff200b1c40 2024-09-17T04:03:39.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.579+0000 7eff25800640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 0x7eff0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:39.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.580+0000 7eff25800640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 0x7eff0007b0a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7eff14035000 tx=0x7eff14004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:39.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.584+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7eff1801f020 con 0x7eff200b1c40 2024-09-17T04:03:39.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:39.804+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7eff2006f340 con 0x7eff200b1c40 2024-09-17T04:03:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.307+0000 7eff1d600640 1 -- 172.21.15.129:0/1295094974 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v177) ==== 174+0+0 (secure 0 0 0) 0x7eff18063070 con 0x7eff200b1c40 2024-09-17T04:03:40.307 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 30 and data pool 31 2024-09-17T04:03:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.310+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 msgr2=0x7eff0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.310+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 0x7eff0007b0a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7eff14035000 tx=0x7eff14004ab0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.310+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 msgr2=0x7eff2014ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.310+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7eff1800b410 tx=0x7eff1800b8d0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 shutdown_connections 2024-09-17T04:03:40.310 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7eff00078b90 0x7eff0007b0a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7eff200b1c40 0x7eff2014ed60 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff20074800 0x7eff2014e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 --2- 172.21.15.129:0/1295094974 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7eff2006eea0 0x7eff2009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.311+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 >> 172.21.15.129:0/1295094974 conn(0x7eff200ccc40 msgr2=0x7eff2015c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:40.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.312+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 shutdown_connections 2024-09-17T04:03:40.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.313+0000 7eff28200640 1 -- 172.21.15.129:0/1295094974 wait complete. 2024-09-17T04:03:40.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 -- 172.21.15.129:0/702596560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad341524c0 msgr2=0x7fad340cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 --2- 172.21.15.129:0/702596560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad341524c0 0x7fad340cf5c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fad3415f1d0 tx=0x7fad2402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 -- 172.21.15.129:0/702596560 shutdown_connections 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 --2- 172.21.15.129:0/702596560 >> v2:172.21.15.181:3300/0 conn(0x7fad341528d0 0x7fad34072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 --2- 172.21.15.129:0/702596560 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad341524c0 0x7fad340cf5c0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 -- 172.21.15.129:0/702596560 >> 172.21.15.129:0/702596560 conn(0x7fad3406cfc0 msgr2=0x7fad3406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 -- 172.21.15.129:0/702596560 shutdown_connections 2024-09-17T04:03:40.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.587+0000 7fad3a400640 1 -- 172.21.15.129:0/702596560 wait complete. 2024-09-17T04:03:40.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.590+0000 7fad3a400640 1 Processor -- start 2024-09-17T04:03:40.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.590+0000 7fad3a400640 1 -- start start 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad341524c0 0x7fad3415af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad341528d0 0x7fad340aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad340d02c0 con 0x7fad340cfb00 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fad340737d0 con 0x7fad341528d0 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad3a400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fad340d0440 con 0x7fad341524c0 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad39000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.591+0000 7fad39000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53008/0 (socket says 172.21.15.129:53008) 2024-09-17T04:03:40.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 -- 172.21.15.129:0/1723752354 learned_addr learned my addr 172.21.15.129:0/1723752354 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 -- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad341524c0 msgr2=0x7fad3415af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad341524c0 0x7fad3415af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 -- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad341528d0 msgr2=0x7fad340aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad341528d0 0x7fad340aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 -- 172.21.15.129:0/1723752354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad341f0fb0 con 0x7fad340cfb00 2024-09-17T04:03:40.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad39000640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fad240099c0 tx=0x7fad240039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.592+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2403e030 con 0x7fad340cfb00 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.593+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad240047a0 con 0x7fad340cfb00 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.593+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad3415b490 con 0x7fad340cfb00 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.593+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24035040 con 0x7fad340cfb00 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.593+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad340a97a0 con 0x7fad340cfb00 2024-09-17T04:03:40.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.594+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad340747b0 con 0x7fad340cfb00 2024-09-17T04:03:40.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.595+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fad24041030 con 0x7fad340cfb00 2024-09-17T04:03:40.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.596+0000 7fad30c00640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 0x7fad1407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:40.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.596+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 5709+0+0 (secure 0 0 0) 0x7fad240bb6a0 con 0x7fad340cfb00 2024-09-17T04:03:40.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.597+0000 7fad33e00640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 0x7fad1407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:40.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.598+0000 7fad33e00640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 0x7fad1407b0d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fad20000f60 tx=0x7fad200079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.601+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fad340747b0 con 0x7fad340cfb00 2024-09-17T04:03:40.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.785+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad340d0960 con 0x7fad340cfb00 2024-09-17T04:03:40.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.786+0000 7fad30c00640 1 -- 172.21.15.129:0/1723752354 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) ==== 74+0+16740 (secure 0 0 0) 0x7fad24083a50 con 0x7fad340cfb00 2024-09-17T04:03:40.786 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:40.786 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":202,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:03:39.805150+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:03:37.702365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-17T04:03:38.771949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.788+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 msgr2=0x7fad1407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.788+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 0x7fad1407b0d0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fad20000f60 tx=0x7fad200079b0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 msgr2=0x7fad341f0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fad240099c0 tx=0x7fad240039e0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 shutdown_connections 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078bc0 0x7fad1407b0d0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad341528d0 0x7fad340aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad341524c0 0x7fad3415af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 --2- 172.21.15.129:0/1723752354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad340cfb00 0x7fad341f0a70 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:40.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 >> 172.21.15.129:0/1723752354 conn(0x7fad3406cfc0 msgr2=0x7fad340b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:40.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 shutdown_connections 2024-09-17T04:03:40.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:40.789+0000 7fad3a400640 1 -- 172.21.15.129:0/1723752354 wait complete. 2024-09-17T04:03:40.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:41.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 -- 172.21.15.129:0/827324815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0cf530 msgr2=0x7efe9c0cff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 --2- 172.21.15.129:0/827324815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0cf530 0x7efe9c0cff50 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7efe94009830 tx=0x7efe9402a530 comp rx=0 tx=0).stop 2024-09-17T04:03:41.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 -- 172.21.15.129:0/827324815 shutdown_connections 2024-09-17T04:03:41.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 --2- 172.21.15.129:0/827324815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0cf530 0x7efe9c0cff50 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 -- 172.21.15.129:0/827324815 >> 172.21.15.129:0/827324815 conn(0x7efe9c069150 msgr2=0x7efe9c069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:41.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 -- 172.21.15.129:0/827324815 shutdown_connections 2024-09-17T04:03:41.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.032+0000 7efea0c00640 1 -- 172.21.15.129:0/827324815 wait complete. 2024-09-17T04:03:41.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.034+0000 7efea0c00640 1 Processor -- start 2024-09-17T04:03:41.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.035+0000 7efea0c00640 1 -- start start 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9c1602b0 0x7efe9c0b7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 0x7efe9c1f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c0d0610 con 0x7efe9c15a220 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efe9c0d0490 con 0x7efe9c0c9760 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efea0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efe9c0d0790 con 0x7efe9c1602b0 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efe9b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efe9b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53010/0 (socket says 172.21.15.129:53010) 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.036+0000 7efe9b400640 1 -- 172.21.15.129:0/1235129680 learned_addr learned my addr 172.21.15.129:0/1235129680 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 -- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9c1602b0 msgr2=0x7efe9c0b7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9c1602b0 0x7efe9c0b7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9be00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 0x7efe9c1f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 -- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 msgr2=0x7efe9c1f8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 0x7efe9c1f8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 -- 172.21.15.129:0/1235129680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9c0cc2f0 con 0x7efe9c15a220 2024-09-17T04:03:41.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9be00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 0x7efe9c1f8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.037+0000 7efe9b400640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00cbb0 tx=0x7efe8c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.038+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c016020 con 0x7efe9c15a220 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.038+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe8c00ad90 con 0x7efe9c15a220 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.038+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe9c0cc5d0 con 0x7efe9c15a220 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.038+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c012680 con 0x7efe9c15a220 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.038+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe9c06e3b0 con 0x7efe9c15a220 2024-09-17T04:03:41.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.039+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe9c06def0 con 0x7efe9c15a220 2024-09-17T04:03:41.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.040+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efe8c019020 con 0x7efe9c15a220 2024-09-17T04:03:41.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.041+0000 7efe93e00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.041+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(202..202 src has 1..202) ==== 5709+0+0 (secure 0 0 0) 0x7efe8c09b380 con 0x7efe9c15a220 2024-09-17T04:03:41.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.041+0000 7efe92a00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.042+0000 7efe92a00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efe84000f60 tx=0x7efe840079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:41.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.046+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efe9c06def0 con 0x7efe9c15a220 2024-09-17T04:03:41.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.229+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe9c0cd8d0 con 0x7efe9c15a220 2024-09-17T04:03:41.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.230+0000 7efe93e00640 1 -- 172.21.15.129:0/1235129680 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v202) ==== 74+0+16740 (secure 0 0 0) 0x7efe9c0cd8d0 con 0x7efe9c15a220 2024-09-17T04:03:41.229 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:41.230 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":202,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:03:39.805150+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:03:37.702365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-17T04:03:38.771949+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.232+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 msgr2=0x7efe7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.232+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7efe84000f60 tx=0x7efe840079b0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.232+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 msgr2=0x7efe9c0cb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.232+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7efe8c00cbb0 tx=0x7efe8c00a060 comp rx=0 tx=0).stop 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 shutdown_connections 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efe78078b90 0x7efe7807b0a0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efe9c0c9760 0x7efe9c1f8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efe9c1602b0 0x7efe9c0b7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 --2- 172.21.15.129:0/1235129680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efe9c15a220 0x7efe9c0cb9f0 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 >> 172.21.15.129:0/1235129680 conn(0x7efe9c069150 msgr2=0x7efe9c159710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:41.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 shutdown_connections 2024-09-17T04:03:41.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.233+0000 7efea0c00640 1 -- 172.21.15.129:0/1235129680 wait complete. 2024-09-17T04:03:41.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:03:41.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 -- 172.21.15.129:0/3388945429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63581655e0 msgr2=0x7f6358167a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 --2- 172.21.15.129:0/3388945429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63581655e0 0x7f6358167a00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6350005fd0 tx=0x7f635002a8d0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 -- 172.21.15.129:0/3388945429 shutdown_connections 2024-09-17T04:03:41.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 --2- 172.21.15.129:0/3388945429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63581655e0 0x7f6358167a00 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 -- 172.21.15.129:0/3388945429 >> 172.21.15.129:0/3388945429 conn(0x7f63580d00b0 msgr2=0x7f63580d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:41.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 -- 172.21.15.129:0/3388945429 shutdown_connections 2024-09-17T04:03:41.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.483+0000 7f635d600640 1 -- 172.21.15.129:0/3388945429 wait complete. 2024-09-17T04:03:41.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.485+0000 7f635d600640 1 Processor -- start 2024-09-17T04:03:41.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.486+0000 7f635d600640 1 -- start start 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63580cb080 0x7f63581f40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f635815c7b0 0x7f635814c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f63581680c0 con 0x7f63581655e0 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6358167f40 con 0x7f63580cb080 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6358168240 con 0x7f635815c7b0 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53012/0 (socket says 172.21.15.129:53012) 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.487+0000 7f635cc00640 1 -- 172.21.15.129:0/1007752998 learned_addr learned my addr 172.21.15.129:0/1007752998 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f634f400640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f635815c7b0 0x7f635814c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f6357e00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63580cb080 0x7f63581f40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 -- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f635815c7b0 msgr2=0x7f635814c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f635815c7b0 0x7f635814c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 -- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63580cb080 msgr2=0x7f63581f40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63580cb080 0x7f63581f40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 -- 172.21.15.129:0/1007752998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635814f1b0 con 0x7f63581655e0 2024-09-17T04:03:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f635cc00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f6350005fd0 tx=0x7f6350005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:41.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.488+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6350037040 con 0x7f63581655e0 2024-09-17T04:03:41.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.489+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6350003c60 con 0x7f63581655e0 2024-09-17T04:03:41.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.489+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63580ae150 con 0x7f63581655e0 2024-09-17T04:03:41.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.489+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635003c990 con 0x7f63581655e0 2024-09-17T04:03:41.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.489+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6358166df0 con 0x7f63581655e0 2024-09-17T04:03:41.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.490+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6358072e10 con 0x7f63581655e0 2024-09-17T04:03:41.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.491+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f635003a070 con 0x7f63581655e0 2024-09-17T04:03:41.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.492+0000 7f6354c00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 0x7f633407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:41.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.492+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f63500bbbe0 con 0x7f63581655e0 2024-09-17T04:03:41.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.493+0000 7f6357e00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 0x7f633407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:41.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.493+0000 7f6357e00640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 0x7f633407b0a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6348000f60 tx=0x7f6348009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:41.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.497+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6350041020 con 0x7f63581655e0 2024-09-17T04:03:41.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:41.715+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f63580d0f50 con 0x7f63581655e0 2024-09-17T04:03:42.278 INFO:tasks.ceph.mds.d.smithi181.stderr:2024-09-17T04:03:42.278+0000 7fd151200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:03:42.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.336+0000 7f6354c00640 1 -- 172.21.15.129:0/1007752998 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v180) ==== 118+0+0 (secure 0 0 0) 0x7f63500838b0 con 0x7f63581655e0 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 msgr2=0x7f633407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 0x7f633407b0a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f6348000f60 tx=0x7f6348009040 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 msgr2=0x7f63581675f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f6350005fd0 tx=0x7f6350005c30 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 shutdown_connections 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6334078b90 0x7f633407b0a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f63581655e0 0x7f63581675f0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f635815c7b0 0x7f635814c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 --2- 172.21.15.129:0/1007752998 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f63580cb080 0x7f63581f40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.339+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 >> 172.21.15.129:0/1007752998 conn(0x7f63580d00b0 msgr2=0x7f63581584b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:42.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.340+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 shutdown_connections 2024-09-17T04:03:42.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.340+0000 7f635d600640 1 -- 172.21.15.129:0/1007752998 wait complete. 2024-09-17T04:03:42.366 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:03:42.366 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 -- 172.21.15.129:0/2813968450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80b2180 msgr2=0x7fa3f815efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 --2- 172.21.15.129:0/2813968450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80b2180 0x7fa3f815efd0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8069b60 tx=0x7fa3f002a980 comp rx=0 tx=0).stop 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 -- 172.21.15.129:0/2813968450 shutdown_connections 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 --2- 172.21.15.129:0/2813968450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80b2180 0x7fa3f815efd0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 --2- 172.21.15.129:0/2813968450 >> v2:172.21.15.181:3300/0 conn(0x7fa3f80cbf10 0x7fa3f80b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 --2- 172.21.15.129:0/2813968450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80cfca0 0x7fa3f80705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 -- 172.21.15.129:0/2813968450 >> 172.21.15.129:0/2813968450 conn(0x7fa3f80ccc40 msgr2=0x7fa3f80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.592+0000 7fa3ff800640 1 -- 172.21.15.129:0/2813968450 shutdown_connections 2024-09-17T04:03:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.593+0000 7fa3ff800640 1 -- 172.21.15.129:0/2813968450 wait complete. 2024-09-17T04:03:42.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.595+0000 7fa3ff800640 1 Processor -- start 2024-09-17T04:03:42.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.595+0000 7fa3ff800640 1 -- start start 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80b2180 0x7fa3f8074010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3f80cfca0 0x7fa3f8158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f815f8a0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f815f720 con 0x7fa3f80cfca0 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.596+0000 7fa3ff800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa3f815fa20 con 0x7fa3f80b2180 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53020/0 (socket says 172.21.15.129:53020) 2024-09-17T04:03:42.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 -- 172.21.15.129:0/4240911731 learned_addr learned my addr 172.21.15.129:0/4240911731 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3fce00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80b2180 0x7fa3f8074010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3fd800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3f80cfca0 0x7fa3f8158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 -- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80b2180 msgr2=0x7fa3f8074010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80b2180 0x7fa3f8074010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 -- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3f80cfca0 msgr2=0x7fa3f8158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3f80cfca0 0x7fa3f8158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.597+0000 7fa3efe00640 1 -- 172.21.15.129:0/4240911731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f8160140 con 0x7fa3f80cbf10 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3efe00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000e710 tx=0x7fa3e000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e000cab0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa3e000cd40 con 0x7fa3f80cbf10 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0010690 con 0x7fa3f80cbf10 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f8160420 con 0x7fa3f80cbf10 2024-09-17T04:03:42.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.598+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f81609e0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.599+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3f80cc4f0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.600+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa3e0010a40 con 0x7fa3f80cbf10 2024-09-17T04:03:42.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.601+0000 7fa3f5600640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 0x7fa3d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:42.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.601+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7fa3e009b8c0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.602+0000 7fa3fce00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 0x7fa3d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:42.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.602+0000 7fa3fce00640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 0x7fa3d407b0a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8005fd0 tx=0x7fa3e800b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:42.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.605+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa3f80cc4f0 con 0x7fa3f80cbf10 2024-09-17T04:03:42.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:42.826+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa3f80b1c40 con 0x7fa3f80cbf10 2024-09-17T04:03:43.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.344+0000 7fa3f5600640 1 -- 172.21.15.129:0/4240911731 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v181) ==== 122+0+0 (secure 0 0 0) 0x7fa3e00636a0 con 0x7fa3f80cbf10 2024-09-17T04:03:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 msgr2=0x7fa3d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 0x7fa3d407b0a0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fa3e8005fd0 tx=0x7fa3e800b2e0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 msgr2=0x7fa3f8161160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fa3e000e710 tx=0x7fa3e000ebd0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 shutdown_connections 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa3d4078b90 0x7fa3d407b0a0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3f80cfca0 0x7fa3f8158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa3f80cbf10 0x7fa3f8161160 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 --2- 172.21.15.129:0/4240911731 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa3f80b2180 0x7fa3f8074010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 >> 172.21.15.129:0/4240911731 conn(0x7fa3f80ccc40 msgr2=0x7fa3f815e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 shutdown_connections 2024-09-17T04:03:43.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.347+0000 7fa3ff800640 1 -- 172.21.15.129:0/4240911731 wait complete. 2024-09-17T04:03:43.369 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:03:43.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 -- 172.21.15.129:0/3903676868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 msgr2=0x7f14e0160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 --2- 172.21.15.129:0/3903676868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0160cf0 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f14e00cc9e0 tx=0x7f14d002a980 comp rx=0 tx=0).stop 2024-09-17T04:03:43.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 -- 172.21.15.129:0/3903676868 shutdown_connections 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 --2- 172.21.15.129:0/3903676868 >> v2:172.21.15.181:3300/0 conn(0x7f14e00d02c0 0x7f14e00d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 --2- 172.21.15.129:0/3903676868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 0x7f14e00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 --2- 172.21.15.129:0/3903676868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0160cf0 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 -- 172.21.15.129:0/3903676868 >> 172.21.15.129:0/3903676868 conn(0x7f14e006cfc0 msgr2=0x7f14e006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 -- 172.21.15.129:0/3903676868 shutdown_connections 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.592+0000 7f14e4a00640 1 -- 172.21.15.129:0/3903676868 wait complete. 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.594+0000 7f14e4a00640 1 Processor -- start 2024-09-17T04:03:43.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.594+0000 7f14e4a00640 1 -- start start 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 0x7f14e00c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e00d02c0 0x7f14e0155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14e015d040 con 0x7f14e00cfb60 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14e015cec0 con 0x7f14e00d02c0 2024-09-17T04:03:43.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14e4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14e015d1c0 con 0x7f14e00b46e0 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53034/0 (socket says 172.21.15.129:53034) 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 -- 172.21.15.129:0/2173175470 learned_addr learned my addr 172.21.15.129:0/2173175470 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14de800640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 0x7f14e00c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14df200640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e00d02c0 0x7f14e0155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 -- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 msgr2=0x7f14e00c55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 0x7f14e00c55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 -- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e00d02c0 msgr2=0x7f14e0155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e00d02c0 0x7f14e0155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.596+0000 7f14d5e00640 1 -- 172.21.15.129:0/2173175470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e01f5720 con 0x7f14e00cfb60 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14d5e00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f14c800d6d0 tx=0x7f14c800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8017070 con 0x7f14e00cfb60 2024-09-17T04:03:43.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14c8005330 con 0x7f14e00cfb60 2024-09-17T04:03:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e01f5a00 con 0x7f14e00cfb60 2024-09-17T04:03:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8009820 con 0x7f14e00cfb60 2024-09-17T04:03:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.597+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e01f5fc0 con 0x7f14e00cfb60 2024-09-17T04:03:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.598+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e015d6e0 con 0x7f14e00cfb60 2024-09-17T04:03:43.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.599+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14c8009bd0 con 0x7f14e00cfb60 2024-09-17T04:03:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.600+0000 7f14d7400640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 0x7f14bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.600+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f14c809afe0 con 0x7f14e00cfb60 2024-09-17T04:03:43.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.601+0000 7f14de800640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 0x7f14bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:43.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.601+0000 7f14de800640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 0x7f14bc07b0a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f14d0035000 tx=0x7f14d0004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:43.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.605+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14c8063500 con 0x7f14e00cfb60 2024-09-17T04:03:43.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.822+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14e006a140 con 0x7f14e00cfb60 2024-09-17T04:03:43.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.823+0000 7f14d7400640 1 -- 172.21.15.129:0/2173175470 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5251 (secure 0 0 0) 0x7f14c8062e00 con 0x7f14e00cfb60 2024-09-17T04:03:43.822 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:43.822 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":181,"btime":"2024-09-17T04:03:43:337368+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3964402375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3964402375},{"type":"v1","addr":"172.21.15.129:6833","nonce":3964402375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175},{"gid":8276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2137304743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2137304743},{"type":"v1","addr":"172.21.15.181:6836","nonce":2137304743}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176},{"gid":10434,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3759513659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3759513659},{"type":"v1","addr":"172.21.15.129:6835","nonce":3759513659}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:03:42.827055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:03:43.823 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 181 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 msgr2=0x7f14bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 0x7f14bc07b0a0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f14d0035000 tx=0x7f14d0004ab0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 msgr2=0x7f14e0150310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f14c800d6d0 tx=0x7f14c800db90 comp rx=0 tx=0).stop 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 shutdown_connections 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14bc078b90 0x7f14bc07b0a0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e00d02c0 0x7f14e0155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e00cfb60 0x7f14e0150310 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 --2- 172.21.15.129:0/2173175470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14e00b46e0 0x7f14e00c55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:43.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 >> 172.21.15.129:0/2173175470 conn(0x7f14e006cfc0 msgr2=0x7f14e0072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:43.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 shutdown_connections 2024-09-17T04:03:43.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:43.826+0000 7f14e4a00640 1 -- 172.21.15.129:0/2173175470 wait complete. 2024-09-17T04:03:43.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:03:44.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 -- 172.21.15.129:0/3770485143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500cf260 msgr2=0x7ff5500ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 --2- 172.21.15.129:0/3770485143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500cf260 0x7ff5500ccb00 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7ff5500743a0 tx=0x7ff53c02a5f0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 -- 172.21.15.129:0/3770485143 shutdown_connections 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 --2- 172.21.15.129:0/3770485143 >> v2:172.21.15.181:3300/0 conn(0x7ff5500cd040 0x7ff5500cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 --2- 172.21.15.129:0/3770485143 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500cf260 0x7ff5500ccb00 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 -- 172.21.15.129:0/3770485143 >> 172.21.15.129:0/3770485143 conn(0x7ff5500d0550 msgr2=0x7ff5500d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 -- 172.21.15.129:0/3770485143 shutdown_connections 2024-09-17T04:03:44.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.072+0000 7ff555c00640 1 -- 172.21.15.129:0/3770485143 wait complete. 2024-09-17T04:03:44.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.074+0000 7ff555c00640 1 Processor -- start 2024-09-17T04:03:44.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.074+0000 7ff555c00640 1 -- start start 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 0x7ff5501f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5500cd040 0x7ff5500c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500b56e0 con 0x7ff5500c6000 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff5500d1480 con 0x7ff5500cd040 2024-09-17T04:03:44.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff5500b5860 con 0x7ff5500b5c10 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46032/0 (socket says 172.21.15.129:46032) 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff555200640 1 -- 172.21.15.129:0/40509437 learned_addr learned my addr 172.21.15.129:0/40509437 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff54f400640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5500cd040 0x7ff5500c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.076+0000 7ff54fe00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 0x7ff5501f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 -- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 msgr2=0x7ff5501f5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 0x7ff5501f5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 -- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5500cd040 msgr2=0x7ff5500c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5500cd040 0x7ff5500c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 -- 172.21.15.129:0/40509437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5500c9ed0 con 0x7ff5500c6000 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff54fe00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 0x7ff5501f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:44.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555200640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff54800e740 tx=0x7ff54800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff54800ccc0 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff54800ce80 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548005280 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5500ca1b0 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.077+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5500ca770 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.078+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5500ccb00 con 0x7ff5500c6000 2024-09-17T04:03:44.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.080+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff548005420 con 0x7ff5500c6000 2024-09-17T04:03:44.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.080+0000 7ff547e00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 0x7ff53007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.081+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7ff54809b630 con 0x7ff5500c6000 2024-09-17T04:03:44.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.081+0000 7ff54fe00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 0x7ff53007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.082+0000 7ff54fe00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 0x7ff53007b0d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff540007fd0 tx=0x7ff540009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:44.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.085+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff548063a60 con 0x7ff5500c6000 2024-09-17T04:03:44.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.268+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5500c9c60 con 0x7ff5500c6000 2024-09-17T04:03:44.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.269+0000 7ff547e00640 1 -- 172.21.15.129:0/40509437 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v203) ==== 74+0+16727 (secure 0 0 0) 0x7ff548063360 con 0x7ff5500c6000 2024-09-17T04:03:44.268 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:44.268 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":203,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:03:41.321136+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:03:37.702365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-17T04:03:38.771949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 msgr2=0x7ff53007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 0x7ff53007b0d0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7ff540007fd0 tx=0x7ff540009040 comp rx=0 tx=0).stop 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 msgr2=0x7ff5500c7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff54800e740 tx=0x7ff54800ec00 comp rx=0 tx=0).stop 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 shutdown_connections 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff530078bc0 0x7ff53007b0d0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5500c6000 0x7ff5500c7600 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5500cd040 0x7ff5500c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 --2- 172.21.15.129:0/40509437 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5500b5c10 0x7ff5501f5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 >> 172.21.15.129:0/40509437 conn(0x7ff5500d0550 msgr2=0x7ff55006f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:44.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 shutdown_connections 2024-09-17T04:03:44.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.271+0000 7ff555c00640 1 -- 172.21.15.129:0/40509437 wait complete. 2024-09-17T04:03:44.293 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:03:44.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:03:44.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 -- 172.21.15.129:0/426717690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba440705a0 msgr2=0x7fba440b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 --2- 172.21.15.129:0/426717690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba440705a0 0x7fba440b4140 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fba34005fd0 tx=0x7fba3402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 -- 172.21.15.129:0/426717690 shutdown_connections 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 --2- 172.21.15.129:0/426717690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba440cd390 0x7fba441609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 --2- 172.21.15.129:0/426717690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba440705a0 0x7fba440b4140 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 -- 172.21.15.129:0/426717690 >> 172.21.15.129:0/426717690 conn(0x7fba4406d410 msgr2=0x7fba440b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 -- 172.21.15.129:0/426717690 shutdown_connections 2024-09-17T04:03:44.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.517+0000 7fba4bc00640 1 -- 172.21.15.129:0/426717690 wait complete. 2024-09-17T04:03:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.519+0000 7fba4bc00640 1 Processor -- start 2024-09-17T04:03:44.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.520+0000 7fba4bc00640 1 -- start start 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba4406eed0 0x7fba44158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba440705a0 0x7fba4415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fba44160f00 con 0x7fba4415bdb0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fba4406f4f0 con 0x7fba440705a0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba4bc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fba44161080 con 0x7fba4406eed0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba49c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba49c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46054/0 (socket says 172.21.15.129:46054) 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba49c00640 1 -- 172.21.15.129:0/2421415950 learned_addr learned my addr 172.21.15.129:0/2421415950 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba49200640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba4406eed0 0x7fba44158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.521+0000 7fba3be00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba440705a0 0x7fba4415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 -- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba4406eed0 msgr2=0x7fba44158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba4406eed0 0x7fba44158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 -- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba440705a0 msgr2=0x7fba4415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba440705a0 0x7fba4415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 -- 172.21.15.129:0/2421415950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba44159670 con 0x7fba4415bdb0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba49c00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009fd0 tx=0x7fba3c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c016070 con 0x7fba4415bdb0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba3c00be00 con 0x7fba4415bdb0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba44159950 con 0x7fba4415bdb0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c003bd0 con 0x7fba4415bdb0 2024-09-17T04:03:44.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.522+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba44159f10 con 0x7fba4415bdb0 2024-09-17T04:03:44.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.524+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba440b4140 con 0x7fba4415bdb0 2024-09-17T04:03:44.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.525+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fba3c010460 con 0x7fba4415bdb0 2024-09-17T04:03:44.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.526+0000 7fba41600640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 0x7fba2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:44.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.526+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7fba3c09b350 con 0x7fba4415bdb0 2024-09-17T04:03:44.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.526+0000 7fba49200640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 0x7fba2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:44.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.527+0000 7fba49200640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 0x7fba2007b0a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba34033000 tx=0x7fba34004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:44.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.530+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fba3c0630a0 con 0x7fba4415bdb0 2024-09-17T04:03:44.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.712+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fba440af7f0 con 0x7fba20078b90 2024-09-17T04:03:44.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.718+0000 7fba41600640 1 -- 172.21.15.129:0/2421415950 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250367 (secure 0 0 0) 0x7fba440af7f0 con 0x7fba20078b90 2024-09-17T04:03:44.720 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:44.721 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:03:44.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 msgr2=0x7fba2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 0x7fba2007b0a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fba34033000 tx=0x7fba34004730 comp rx=0 tx=0).stop 2024-09-17T04:03:44.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 msgr2=0x7fba440c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:44.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009fd0 tx=0x7fba3c00dde0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 shutdown_connections 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba20078b90 0x7fba2007b0a0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba4415bdb0 0x7fba440c7320 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba440705a0 0x7fba4415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 --2- 172.21.15.129:0/2421415950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba4406eed0 0x7fba44158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 >> 172.21.15.129:0/2421415950 conn(0x7fba4406d410 msgr2=0x7fba4415f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 shutdown_connections 2024-09-17T04:03:44.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:44.722+0000 7fba4bc00640 1 -- 172.21.15.129:0/2421415950 wait complete. 2024-09-17T04:03:44.748 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-17T04:03:44.536475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590460,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374740,"kb_used_data":21100,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754599980,"statfs":{"total":773094113280,"available":772710379520,"internally_reserved":0,"allocated":21606400,"data_stored":10143045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":361942576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001886"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315975+0000","last_change":"2024-09-17T04:03:39.315975+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315975+0000","last_undegraded":"2024-09-17T04:03:39.315975+0000","last_fullsized":"2024-09-17T04:03:39.315975+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314425+0000","last_change":"2024-09-17T04:03:39.314425+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314425+0000","last_undegraded":"2024-09-17T04:03:39.314425+0000","last_fullsized":"2024-09-17T04:03:39.314425+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313411+0000","last_change":"2024-09-17T04:03:39.313411+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313411+0000","last_undegraded":"2024-09-17T04:03:39.313411+0000","last_fullsized":"2024-09-17T04:03:39.313411+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318949+0000","last_change":"2024-09-17T04:03:39.318949+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318949+0000","last_undegraded":"2024-09-17T04:03:39.318949+0000","last_fullsized":"2024-09-17T04:03:39.318949+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333497+0000","last_change":"2024-09-17T04:03:40.335355+0000","last_active":"2024-09-17T04:03:41.333497+0000","last_peered":"2024-09-17T04:03:41.333497+0000","last_clean":"2024-09-17T04:03:41.333497+0000","last_became_active":"2024-09-17T04:03:40.334994+0000","last_became_peered":"2024-09-17T04:03:40.334994+0000","last_unstale":"2024-09-17T04:03:41.333497+0000","last_undegraded":"2024-09-17T04:03:41.333497+0000","last_fullsized":"2024-09-17T04:03:41.333497+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:02.228911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317948+0000","last_change":"2024-09-17T04:03:39.317948+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317948+0000","last_undegraded":"2024-09-17T04:03:39.317948+0000","last_fullsized":"2024-09-17T04:03:39.317948+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482089+0000","last_change":"2024-09-17T04:03:40.350322+0000","last_active":"2024-09-17T04:03:41.482089+0000","last_peered":"2024-09-17T04:03:41.482089+0000","last_clean":"2024-09-17T04:03:41.482089+0000","last_became_active":"2024-09-17T04:03:40.349953+0000","last_became_peered":"2024-09-17T04:03:40.349953+0000","last_unstale":"2024-09-17T04:03:41.482089+0000","last_undegraded":"2024-09-17T04:03:41.482089+0000","last_fullsized":"2024-09-17T04:03:41.482089+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:03:12.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741207+0000","last_change":"2024-09-17T04:03:40.334851+0000","last_active":"2024-09-17T04:03:42.741207+0000","last_peered":"2024-09-17T04:03:42.741207+0000","last_clean":"2024-09-17T04:03:42.741207+0000","last_became_active":"2024-09-17T04:03:40.331337+0000","last_became_peered":"2024-09-17T04:03:40.331337+0000","last_unstale":"2024-09-17T04:03:42.741207+0000","last_undegraded":"2024-09-17T04:03:42.741207+0000","last_fullsized":"2024-09-17T04:03:42.741207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:52.463737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521788+0000","last_change":"2024-09-17T04:03:40.349511+0000","last_active":"2024-09-17T04:03:43.521788+0000","last_peered":"2024-09-17T04:03:43.521788+0000","last_clean":"2024-09-17T04:03:43.521788+0000","last_became_active":"2024-09-17T04:03:40.349113+0000","last_became_peered":"2024-09-17T04:03:40.349113+0000","last_unstale":"2024-09-17T04:03:43.521788+0000","last_undegraded":"2024-09-17T04:03:43.521788+0000","last_fullsized":"2024-09-17T04:03:43.521788+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:50.920504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482695+0000","last_change":"2024-09-17T04:03:40.340480+0000","last_active":"2024-09-17T04:03:41.482695+0000","last_peered":"2024-09-17T04:03:41.482695+0000","last_clean":"2024-09-17T04:03:41.482695+0000","last_became_active":"2024-09-17T04:03:40.338521+0000","last_became_peered":"2024-09-17T04:03:40.338521+0000","last_unstale":"2024-09-17T04:03:41.482695+0000","last_undegraded":"2024-09-17T04:03:41.482695+0000","last_fullsized":"2024-09-17T04:03:41.482695+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:10.458690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410478+0000","last_change":"2024-09-17T04:03:40.348952+0000","last_active":"2024-09-17T04:03:42.410478+0000","last_peered":"2024-09-17T04:03:42.410478+0000","last_clean":"2024-09-17T04:03:42.410478+0000","last_became_active":"2024-09-17T04:03:40.348656+0000","last_became_peered":"2024-09-17T04:03:40.348656+0000","last_unstale":"2024-09-17T04:03:42.410478+0000","last_undegraded":"2024-09-17T04:03:42.410478+0000","last_fullsized":"2024-09-17T04:03:42.410478+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:47.744159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741493+0000","last_change":"2024-09-17T04:03:40.354695+0000","last_active":"2024-09-17T04:03:42.741493+0000","last_peered":"2024-09-17T04:03:42.741493+0000","last_clean":"2024-09-17T04:03:42.741493+0000","last_became_active":"2024-09-17T04:03:40.354274+0000","last_became_peered":"2024-09-17T04:03:40.354274+0000","last_unstale":"2024-09-17T04:03:42.741493+0000","last_undegraded":"2024-09-17T04:03:42.741493+0000","last_fullsized":"2024-09-17T04:03:42.741493+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:15.701377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482731+0000","last_change":"2024-09-17T04:03:40.350373+0000","last_active":"2024-09-17T04:03:41.482731+0000","last_peered":"2024-09-17T04:03:41.482731+0000","last_clean":"2024-09-17T04:03:41.482731+0000","last_became_active":"2024-09-17T04:03:40.350000+0000","last_became_peered":"2024-09-17T04:03:40.350000+0000","last_unstale":"2024-09-17T04:03:41.482731+0000","last_undegraded":"2024-09-17T04:03:41.482731+0000","last_fullsized":"2024-09-17T04:03:41.482731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:29.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312559+0000","last_change":"2024-09-17T04:03:39.312559+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312559+0000","last_undegraded":"2024-09-17T04:03:39.312559+0000","last_fullsized":"2024-09-17T04:03:39.312559+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521536+0000","last_change":"2024-09-17T04:03:40.344294+0000","last_active":"2024-09-17T04:03:43.521536+0000","last_peered":"2024-09-17T04:03:43.521536+0000","last_clean":"2024-09-17T04:03:43.521536+0000","last_became_active":"2024-09-17T04:03:40.343923+0000","last_became_peered":"2024-09-17T04:03:40.343923+0000","last_unstale":"2024-09-17T04:03:43.521536+0000","last_undegraded":"2024-09-17T04:03:43.521536+0000","last_fullsized":"2024-09-17T04:03:43.521536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:35.362812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313125+0000","last_change":"2024-09-17T04:03:39.313125+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313125+0000","last_undegraded":"2024-09-17T04:03:39.313125+0000","last_fullsized":"2024-09-17T04:03:39.313125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741536+0000","last_change":"2024-09-17T04:03:40.343695+0000","last_active":"2024-09-17T04:03:42.741536+0000","last_peered":"2024-09-17T04:03:42.741536+0000","last_clean":"2024-09-17T04:03:42.741536+0000","last_became_active":"2024-09-17T04:03:40.343413+0000","last_became_peered":"2024-09-17T04:03:40.343413+0000","last_unstale":"2024-09-17T04:03:42.741536+0000","last_undegraded":"2024-09-17T04:03:42.741536+0000","last_fullsized":"2024-09-17T04:03:42.741536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:54.092079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313567+0000","last_change":"2024-09-17T04:03:39.313567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313567+0000","last_undegraded":"2024-09-17T04:03:39.313567+0000","last_fullsized":"2024-09-17T04:03:39.313567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:40.351995+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:40.351649+0000","last_became_peered":"2024-09-17T04:03:40.351649+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:03.942253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313824+0000","last_change":"2024-09-17T04:03:39.313824+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313824+0000","last_undegraded":"2024-09-17T04:03:39.313824+0000","last_fullsized":"2024-09-17T04:03:39.313824+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741813+0000","last_change":"2024-09-17T04:03:40.333277+0000","last_active":"2024-09-17T04:03:42.741813+0000","last_peered":"2024-09-17T04:03:42.741813+0000","last_clean":"2024-09-17T04:03:42.741813+0000","last_became_active":"2024-09-17T04:03:40.332882+0000","last_became_peered":"2024-09-17T04:03:40.332882+0000","last_unstale":"2024-09-17T04:03:42.741813+0000","last_undegraded":"2024-09-17T04:03:42.741813+0000","last_fullsized":"2024-09-17T04:03:42.741813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:44.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482369+0000","last_change":"2024-09-17T04:03:40.341616+0000","last_active":"2024-09-17T04:03:41.482369+0000","last_peered":"2024-09-17T04:03:41.482369+0000","last_clean":"2024-09-17T04:03:41.482369+0000","last_became_active":"2024-09-17T04:03:40.341269+0000","last_became_peered":"2024-09-17T04:03:40.341269+0000","last_unstale":"2024-09-17T04:03:41.482369+0000","last_undegraded":"2024-09-17T04:03:41.482369+0000","last_fullsized":"2024-09-17T04:03:41.482369+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:28.676331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741352+0000","last_change":"2024-09-17T04:03:40.337777+0000","last_active":"2024-09-17T04:03:42.741352+0000","last_peered":"2024-09-17T04:03:42.741352+0000","last_clean":"2024-09-17T04:03:42.741352+0000","last_became_active":"2024-09-17T04:03:40.335850+0000","last_became_peered":"2024-09-17T04:03:40.335850+0000","last_unstale":"2024-09-17T04:03:42.741352+0000","last_undegraded":"2024-09-17T04:03:42.741352+0000","last_fullsized":"2024-09-17T04:03:42.741352+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:37.334408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316858+0000","last_change":"2024-09-17T04:03:39.316858+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316858+0000","last_undegraded":"2024-09-17T04:03:39.316858+0000","last_fullsized":"2024-09-17T04:03:39.316858+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.324689+0000","last_change":"2024-09-17T04:03:39.324689+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.324689+0000","last_undegraded":"2024-09-17T04:03:39.324689+0000","last_fullsized":"2024-09-17T04:03:39.324689+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482816+0000","last_change":"2024-09-17T04:03:40.350740+0000","last_active":"2024-09-17T04:03:41.482816+0000","last_peered":"2024-09-17T04:03:41.482816+0000","last_clean":"2024-09-17T04:03:41.482816+0000","last_became_active":"2024-09-17T04:03:40.350415+0000","last_became_peered":"2024-09-17T04:03:40.350415+0000","last_unstale":"2024-09-17T04:03:41.482816+0000","last_undegraded":"2024-09-17T04:03:41.482816+0000","last_fullsized":"2024-09-17T04:03:41.482816+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:04.588026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410543+0000","last_change":"2024-09-17T04:03:40.347688+0000","last_active":"2024-09-17T04:03:42.410543+0000","last_peered":"2024-09-17T04:03:42.410543+0000","last_clean":"2024-09-17T04:03:42.410543+0000","last_became_active":"2024-09-17T04:03:40.347381+0000","last_became_peered":"2024-09-17T04:03:40.347381+0000","last_unstale":"2024-09-17T04:03:42.410543+0000","last_undegraded":"2024-09-17T04:03:42.410543+0000","last_fullsized":"2024-09-17T04:03:42.410543+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:51.255876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312330+0000","last_change":"2024-09-17T04:03:39.312330+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312330+0000","last_undegraded":"2024-09-17T04:03:39.312330+0000","last_fullsized":"2024-09-17T04:03:39.312330+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410325+0000","last_change":"2024-09-17T04:03:40.350667+0000","last_active":"2024-09-17T04:03:42.410325+0000","last_peered":"2024-09-17T04:03:42.410325+0000","last_clean":"2024-09-17T04:03:42.410325+0000","last_became_active":"2024-09-17T04:03:40.350312+0000","last_became_peered":"2024-09-17T04:03:40.350312+0000","last_unstale":"2024-09-17T04:03:42.410325+0000","last_undegraded":"2024-09-17T04:03:42.410325+0000","last_fullsized":"2024-09-17T04:03:42.410325+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:21.193199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333504+0000","last_change":"2024-09-17T04:03:40.331612+0000","last_active":"2024-09-17T04:03:41.333504+0000","last_peered":"2024-09-17T04:03:41.333504+0000","last_clean":"2024-09-17T04:03:41.333504+0000","last_became_active":"2024-09-17T04:03:40.329978+0000","last_became_peered":"2024-09-17T04:03:40.329978+0000","last_unstale":"2024-09-17T04:03:41.333504+0000","last_undegraded":"2024-09-17T04:03:41.333504+0000","last_fullsized":"2024-09-17T04:03:41.333504+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:48.029240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.321401+0000","last_change":"2024-09-17T04:03:39.321401+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.321401+0000","last_undegraded":"2024-09-17T04:03:39.321401+0000","last_fullsized":"2024-09-17T04:03:39.321401+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316435+0000","last_change":"2024-09-17T04:03:39.316435+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316435+0000","last_undegraded":"2024-09-17T04:03:39.316435+0000","last_fullsized":"2024-09-17T04:03:39.316435+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312535+0000","last_change":"2024-09-17T04:03:39.312535+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312535+0000","last_undegraded":"2024-09-17T04:03:39.312535+0000","last_fullsized":"2024-09-17T04:03:39.312535+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482954+0000","last_change":"2024-09-17T04:03:39.330078+0000","last_active":"2024-09-17T04:03:41.482954+0000","last_peered":"2024-09-17T04:03:41.482954+0000","last_clean":"2024-09-17T04:03:41.482954+0000","last_became_active":"2024-09-17T04:03:39.329766+0000","last_became_peered":"2024-09-17T04:03:39.329766+0000","last_unstale":"2024-09-17T04:03:41.482954+0000","last_undegraded":"2024-09-17T04:03:41.482954+0000","last_fullsized":"2024-09-17T04:03:41.482954+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:11.195328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410186+0000","last_change":"2024-09-17T04:03:40.351683+0000","last_active":"2024-09-17T04:03:42.410186+0000","last_peered":"2024-09-17T04:03:42.410186+0000","last_clean":"2024-09-17T04:03:42.410186+0000","last_became_active":"2024-09-17T04:03:40.351288+0000","last_became_peered":"2024-09-17T04:03:40.351288+0000","last_unstale":"2024-09-17T04:03:42.410186+0000","last_undegraded":"2024-09-17T04:03:42.410186+0000","last_fullsized":"2024-09-17T04:03:42.410186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:11:57.454665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326983+0000","last_change":"2024-09-17T04:03:39.322960+0000","last_active":"2024-09-17T04:03:39.326983+0000","last_peered":"2024-09-17T04:03:39.326983+0000","last_clean":"2024-09-17T04:03:39.326983+0000","last_became_active":"2024-09-17T04:03:39.322704+0000","last_became_peered":"2024-09-17T04:03:39.322704+0000","last_unstale":"2024-09-17T04:03:39.326983+0000","last_undegraded":"2024-09-17T04:03:39.326983+0000","last_fullsized":"2024-09-17T04:03:39.326983+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741839+0000","last_change":"2024-09-17T04:03:40.343840+0000","last_active":"2024-09-17T04:03:42.741839+0000","last_peered":"2024-09-17T04:03:42.741839+0000","last_clean":"2024-09-17T04:03:42.741839+0000","last_became_active":"2024-09-17T04:03:40.343539+0000","last_became_peered":"2024-09-17T04:03:40.343539+0000","last_unstale":"2024-09-17T04:03:42.741839+0000","last_undegraded":"2024-09-17T04:03:42.741839+0000","last_fullsized":"2024-09-17T04:03:42.741839+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:04.501121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.325189+0000","last_change":"2024-09-17T04:03:39.320734+0000","last_active":"2024-09-17T04:03:39.325189+0000","last_peered":"2024-09-17T04:03:39.325189+0000","last_clean":"2024-09-17T04:03:39.325189+0000","last_became_active":"2024-09-17T04:03:39.320431+0000","last_became_peered":"2024-09-17T04:03:39.320431+0000","last_unstale":"2024-09-17T04:03:39.325189+0000","last_undegraded":"2024-09-17T04:03:39.325189+0000","last_fullsized":"2024-09-17T04:03:39.325189+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:51.187815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410125+0000","last_change":"2024-09-17T04:03:40.335363+0000","last_active":"2024-09-17T04:03:42.410125+0000","last_peered":"2024-09-17T04:03:42.410125+0000","last_clean":"2024-09-17T04:03:42.410125+0000","last_became_active":"2024-09-17T04:03:40.333333+0000","last_became_peered":"2024-09-17T04:03:40.333333+0000","last_unstale":"2024-09-17T04:03:42.410125+0000","last_undegraded":"2024-09-17T04:03:42.410125+0000","last_fullsized":"2024-09-17T04:03:42.410125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:35.331891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334393+0000","last_change":"2024-09-17T04:03:39.329538+0000","last_active":"2024-09-17T04:03:39.334393+0000","last_peered":"2024-09-17T04:03:39.334393+0000","last_clean":"2024-09-17T04:03:39.334393+0000","last_became_active":"2024-09-17T04:03:39.329257+0000","last_became_peered":"2024-09-17T04:03:39.329257+0000","last_unstale":"2024-09-17T04:03:39.334393+0000","last_undegraded":"2024-09-17T04:03:39.334393+0000","last_fullsized":"2024-09-17T04:03:39.334393+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:25.093614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521725+0000","last_change":"2024-09-17T04:03:40.344266+0000","last_active":"2024-09-17T04:03:43.521725+0000","last_peered":"2024-09-17T04:03:43.521725+0000","last_clean":"2024-09-17T04:03:43.521725+0000","last_became_active":"2024-09-17T04:03:40.343914+0000","last_became_peered":"2024-09-17T04:03:40.343914+0000","last_unstale":"2024-09-17T04:03:43.521725+0000","last_undegraded":"2024-09-17T04:03:43.521725+0000","last_fullsized":"2024-09-17T04:03:43.521725+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:49.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326649+0000","last_change":"2024-09-17T04:03:39.321662+0000","last_active":"2024-09-17T04:03:39.326649+0000","last_peered":"2024-09-17T04:03:39.326649+0000","last_clean":"2024-09-17T04:03:39.326649+0000","last_became_active":"2024-09-17T04:03:39.321278+0000","last_became_peered":"2024-09-17T04:03:39.321278+0000","last_unstale":"2024-09-17T04:03:39.326649+0000","last_undegraded":"2024-09-17T04:03:39.326649+0000","last_fullsized":"2024-09-17T04:03:39.326649+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.772774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312542+0000","last_change":"2024-09-17T04:03:39.312542+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312542+0000","last_undegraded":"2024-09-17T04:03:39.312542+0000","last_fullsized":"2024-09-17T04:03:39.312542+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482232+0000","last_change":"2024-09-17T04:03:39.332398+0000","last_active":"2024-09-17T04:03:41.482232+0000","last_peered":"2024-09-17T04:03:41.482232+0000","last_clean":"2024-09-17T04:03:41.482232+0000","last_became_active":"2024-09-17T04:03:39.331945+0000","last_became_peered":"2024-09-17T04:03:39.331945+0000","last_unstale":"2024-09-17T04:03:41.482232+0000","last_undegraded":"2024-09-17T04:03:41.482232+0000","last_fullsized":"2024-09-17T04:03:41.482232+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.951852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314983+0000","last_change":"2024-09-17T04:03:39.314983+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314983+0000","last_undegraded":"2024-09-17T04:03:39.314983+0000","last_fullsized":"2024-09-17T04:03:39.314983+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.324526+0000","last_change":"2024-09-17T04:03:39.317105+0000","last_active":"2024-09-17T04:03:39.324526+0000","last_peered":"2024-09-17T04:03:39.324526+0000","last_clean":"2024-09-17T04:03:39.324526+0000","last_became_active":"2024-09-17T04:03:39.316779+0000","last_became_peered":"2024-09-17T04:03:39.316779+0000","last_unstale":"2024-09-17T04:03:39.324526+0000","last_undegraded":"2024-09-17T04:03:39.324526+0000","last_fullsized":"2024-09-17T04:03:39.324526+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:21.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313113+0000","last_change":"2024-09-17T04:03:39.313113+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313113+0000","last_undegraded":"2024-09-17T04:03:39.313113+0000","last_fullsized":"2024-09-17T04:03:39.313113+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:39.326124+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:39.325797+0000","last_became_peered":"2024-09-17T04:03:39.325797+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:04.499200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410736+0000","last_change":"2024-09-17T04:03:40.351885+0000","last_active":"2024-09-17T04:03:42.410736+0000","last_peered":"2024-09-17T04:03:42.410736+0000","last_clean":"2024-09-17T04:03:42.410736+0000","last_became_active":"2024-09-17T04:03:40.351494+0000","last_became_peered":"2024-09-17T04:03:40.351494+0000","last_unstale":"2024-09-17T04:03:42.410736+0000","last_undegraded":"2024-09-17T04:03:42.410736+0000","last_fullsized":"2024-09-17T04:03:42.410736+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:13.184588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333987+0000","last_change":"2024-09-17T04:03:39.328133+0000","last_active":"2024-09-17T04:03:39.333987+0000","last_peered":"2024-09-17T04:03:39.333987+0000","last_clean":"2024-09-17T04:03:39.333987+0000","last_became_active":"2024-09-17T04:03:39.327863+0000","last_became_peered":"2024-09-17T04:03:39.327863+0000","last_unstale":"2024-09-17T04:03:39.333987+0000","last_undegraded":"2024-09-17T04:03:39.333987+0000","last_fullsized":"2024-09-17T04:03:39.333987+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:55.809447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311933+0000","last_change":"2024-09-17T04:03:39.311933+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311933+0000","last_undegraded":"2024-09-17T04:03:39.311933+0000","last_fullsized":"2024-09-17T04:03:39.311933+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332935+0000","last_change":"2024-09-17T04:03:39.331097+0000","last_active":"2024-09-17T04:03:39.332935+0000","last_peered":"2024-09-17T04:03:39.332935+0000","last_clean":"2024-09-17T04:03:39.332935+0000","last_became_active":"2024-09-17T04:03:39.330131+0000","last_became_peered":"2024-09-17T04:03:39.330131+0000","last_unstale":"2024-09-17T04:03:39.332935+0000","last_undegraded":"2024-09-17T04:03:39.332935+0000","last_fullsized":"2024-09-17T04:03:39.332935+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:57:40.153994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482540+0000","last_change":"2024-09-17T04:03:40.332717+0000","last_active":"2024-09-17T04:03:41.482540+0000","last_peered":"2024-09-17T04:03:41.482540+0000","last_clean":"2024-09-17T04:03:41.482540+0000","last_became_active":"2024-09-17T04:03:40.332421+0000","last_became_peered":"2024-09-17T04:03:40.332421+0000","last_unstale":"2024-09-17T04:03:41.482540+0000","last_undegraded":"2024-09-17T04:03:41.482540+0000","last_fullsized":"2024-09-17T04:03:41.482540+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:42.530742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332906+0000","last_change":"2024-09-17T04:03:39.329612+0000","last_active":"2024-09-17T04:03:39.332906+0000","last_peered":"2024-09-17T04:03:39.332906+0000","last_clean":"2024-09-17T04:03:39.332906+0000","last_became_active":"2024-09-17T04:03:39.329342+0000","last_became_peered":"2024-09-17T04:03:39.329342+0000","last_unstale":"2024-09-17T04:03:39.332906+0000","last_undegraded":"2024-09-17T04:03:39.332906+0000","last_fullsized":"2024-09-17T04:03:39.332906+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:14.980173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482365+0000","last_change":"2024-09-17T04:03:40.338842+0000","last_active":"2024-09-17T04:03:41.482365+0000","last_peered":"2024-09-17T04:03:41.482365+0000","last_clean":"2024-09-17T04:03:41.482365+0000","last_became_active":"2024-09-17T04:03:40.338449+0000","last_became_peered":"2024-09-17T04:03:40.338449+0000","last_unstale":"2024-09-17T04:03:41.482365+0000","last_undegraded":"2024-09-17T04:03:41.482365+0000","last_fullsized":"2024-09-17T04:03:41.482365+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:44.963272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.328955+0000","last_change":"2024-09-17T04:03:39.324560+0000","last_active":"2024-09-17T04:03:39.328955+0000","last_peered":"2024-09-17T04:03:39.328955+0000","last_clean":"2024-09-17T04:03:39.328955+0000","last_became_active":"2024-09-17T04:03:39.324206+0000","last_became_peered":"2024-09-17T04:03:39.324206+0000","last_unstale":"2024-09-17T04:03:39.328955+0000","last_undegraded":"2024-09-17T04:03:39.328955+0000","last_fullsized":"2024-09-17T04:03:39.328955+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:40.444284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521913+0000","last_change":"2024-09-17T04:03:40.344849+0000","last_active":"2024-09-17T04:03:43.521913+0000","last_peered":"2024-09-17T04:03:43.521913+0000","last_clean":"2024-09-17T04:03:43.521913+0000","last_became_active":"2024-09-17T04:03:40.344173+0000","last_became_peered":"2024-09-17T04:03:40.344173+0000","last_unstale":"2024-09-17T04:03:43.521913+0000","last_undegraded":"2024-09-17T04:03:43.521913+0000","last_fullsized":"2024-09-17T04:03:43.521913+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:12.072676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333711+0000","last_change":"2024-09-17T04:03:39.327456+0000","last_active":"2024-09-17T04:03:39.333711+0000","last_peered":"2024-09-17T04:03:39.333711+0000","last_clean":"2024-09-17T04:03:39.333711+0000","last_became_active":"2024-09-17T04:03:39.326151+0000","last_became_peered":"2024-09-17T04:03:39.326151+0000","last_unstale":"2024-09-17T04:03:39.333711+0000","last_undegraded":"2024-09-17T04:03:39.333711+0000","last_fullsized":"2024-09-17T04:03:39.333711+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317212+0000","last_change":"2024-09-17T04:03:39.317212+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317212+0000","last_undegraded":"2024-09-17T04:03:39.317212+0000","last_fullsized":"2024-09-17T04:03:39.317212+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482297+0000","last_change":"2024-09-17T04:03:39.330613+0000","last_active":"2024-09-17T04:03:41.482297+0000","last_peered":"2024-09-17T04:03:41.482297+0000","last_clean":"2024-09-17T04:03:41.482297+0000","last_became_active":"2024-09-17T04:03:39.330283+0000","last_became_peered":"2024-09-17T04:03:39.330283+0000","last_unstale":"2024-09-17T04:03:41.482297+0000","last_undegraded":"2024-09-17T04:03:41.482297+0000","last_fullsized":"2024-09-17T04:03:41.482297+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:46.502036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482092+0000","last_change":"2024-09-17T04:03:40.341659+0000","last_active":"2024-09-17T04:03:41.482092+0000","last_peered":"2024-09-17T04:03:41.482092+0000","last_clean":"2024-09-17T04:03:41.482092+0000","last_became_active":"2024-09-17T04:03:40.341344+0000","last_became_peered":"2024-09-17T04:03:40.341344+0000","last_unstale":"2024-09-17T04:03:41.482092+0000","last_undegraded":"2024-09-17T04:03:41.482092+0000","last_fullsized":"2024-09-17T04:03:41.482092+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:24.293693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"202'8","reported_seq":23,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.409812+0000","last_change":"2024-09-17T04:03:39.332639+0000","last_active":"2024-09-17T04:03:42.409812+0000","last_peered":"2024-09-17T04:03:42.409812+0000","last_clean":"2024-09-17T04:03:42.409812+0000","last_became_active":"2024-09-17T04:03:39.332409+0000","last_became_peered":"2024-09-17T04:03:39.332409+0000","last_unstale":"2024-09-17T04:03:42.409812+0000","last_undegraded":"2024-09-17T04:03:42.409812+0000","last_fullsized":"2024-09-17T04:03:42.409812+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:03.585348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318088+0000","last_change":"2024-09-17T04:03:39.318088+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318088+0000","last_undegraded":"2024-09-17T04:03:39.318088+0000","last_fullsized":"2024-09-17T04:03:39.318088+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482664+0000","last_change":"2024-09-17T04:03:39.333660+0000","last_active":"2024-09-17T04:03:41.482664+0000","last_peered":"2024-09-17T04:03:41.482664+0000","last_clean":"2024-09-17T04:03:41.482664+0000","last_became_active":"2024-09-17T04:03:39.333330+0000","last_became_peered":"2024-09-17T04:03:39.333330+0000","last_unstale":"2024-09-17T04:03:41.482664+0000","last_undegraded":"2024-09-17T04:03:41.482664+0000","last_fullsized":"2024-09-17T04:03:41.482664+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:03.955054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311836+0000","last_change":"2024-09-17T04:03:39.311836+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311836+0000","last_undegraded":"2024-09-17T04:03:39.311836+0000","last_fullsized":"2024-09-17T04:03:39.311836+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326510+0000","last_change":"2024-09-17T04:03:39.322064+0000","last_active":"2024-09-17T04:03:39.326510+0000","last_peered":"2024-09-17T04:03:39.326510+0000","last_clean":"2024-09-17T04:03:39.326510+0000","last_became_active":"2024-09-17T04:03:39.321750+0000","last_became_peered":"2024-09-17T04:03:39.321750+0000","last_unstale":"2024-09-17T04:03:39.326510+0000","last_undegraded":"2024-09-17T04:03:39.326510+0000","last_fullsized":"2024-09-17T04:03:39.326510+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521646+0000","last_change":"2024-09-17T04:03:40.340280+0000","last_active":"2024-09-17T04:03:43.521646+0000","last_peered":"2024-09-17T04:03:43.521646+0000","last_clean":"2024-09-17T04:03:43.521646+0000","last_became_active":"2024-09-17T04:03:40.339904+0000","last_became_peered":"2024-09-17T04:03:40.339904+0000","last_unstale":"2024-09-17T04:03:43.521646+0000","last_undegraded":"2024-09-17T04:03:43.521646+0000","last_fullsized":"2024-09-17T04:03:43.521646+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:09.142973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521655+0000","last_change":"2024-09-17T04:03:39.335875+0000","last_active":"2024-09-17T04:03:43.521655+0000","last_peered":"2024-09-17T04:03:43.521655+0000","last_clean":"2024-09-17T04:03:43.521655+0000","last_became_active":"2024-09-17T04:03:39.335580+0000","last_became_peered":"2024-09-17T04:03:39.335580+0000","last_unstale":"2024-09-17T04:03:43.521655+0000","last_undegraded":"2024-09-17T04:03:43.521655+0000","last_fullsized":"2024-09-17T04:03:43.521655+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741903+0000","last_change":"2024-09-17T04:03:40.342551+0000","last_active":"2024-09-17T04:03:42.741903+0000","last_peered":"2024-09-17T04:03:42.741903+0000","last_clean":"2024-09-17T04:03:42.741903+0000","last_became_active":"2024-09-17T04:03:40.339096+0000","last_became_peered":"2024-09-17T04:03:40.339096+0000","last_unstale":"2024-09-17T04:03:42.741903+0000","last_undegraded":"2024-09-17T04:03:42.741903+0000","last_fullsized":"2024-09-17T04:03:42.741903+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:42.841396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410300+0000","last_change":"2024-09-17T04:03:39.331133+0000","last_active":"2024-09-17T04:03:42.410300+0000","last_peered":"2024-09-17T04:03:42.410300+0000","last_clean":"2024-09-17T04:03:42.410300+0000","last_became_active":"2024-09-17T04:03:39.330849+0000","last_became_peered":"2024-09-17T04:03:39.330849+0000","last_unstale":"2024-09-17T04:03:42.410300+0000","last_undegraded":"2024-09-17T04:03:42.410300+0000","last_fullsized":"2024-09-17T04:03:42.410300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:19.933450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741382+0000","last_change":"2024-09-17T04:03:40.339612+0000","last_active":"2024-09-17T04:03:42.741382+0000","last_peered":"2024-09-17T04:03:42.741382+0000","last_clean":"2024-09-17T04:03:42.741382+0000","last_became_active":"2024-09-17T04:03:40.339297+0000","last_became_peered":"2024-09-17T04:03:40.339297+0000","last_unstale":"2024-09-17T04:03:42.741382+0000","last_undegraded":"2024-09-17T04:03:42.741382+0000","last_fullsized":"2024-09-17T04:03:42.741382+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:14.950266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.335737+0000","last_change":"2024-09-17T04:03:39.334073+0000","last_active":"2024-09-17T04:03:39.335737+0000","last_peered":"2024-09-17T04:03:39.335737+0000","last_clean":"2024-09-17T04:03:39.335737+0000","last_became_active":"2024-09-17T04:03:39.333739+0000","last_became_peered":"2024-09-17T04:03:39.333739+0000","last_unstale":"2024-09-17T04:03:39.335737+0000","last_undegraded":"2024-09-17T04:03:39.335737+0000","last_fullsized":"2024-09-17T04:03:39.335737+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:00.772342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312567+0000","last_change":"2024-09-17T04:03:39.312567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312567+0000","last_undegraded":"2024-09-17T04:03:39.312567+0000","last_fullsized":"2024-09-17T04:03:39.312567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410122+0000","last_change":"2024-09-17T04:03:39.331198+0000","last_active":"2024-09-17T04:03:42.410122+0000","last_peered":"2024-09-17T04:03:42.410122+0000","last_clean":"2024-09-17T04:03:42.410122+0000","last_became_active":"2024-09-17T04:03:39.330893+0000","last_became_peered":"2024-09-17T04:03:39.330893+0000","last_unstale":"2024-09-17T04:03:42.410122+0000","last_undegraded":"2024-09-17T04:03:42.410122+0000","last_fullsized":"2024-09-17T04:03:42.410122+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:28.427519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410029+0000","last_change":"2024-09-17T04:03:40.347641+0000","last_active":"2024-09-17T04:03:42.410029+0000","last_peered":"2024-09-17T04:03:42.410029+0000","last_clean":"2024-09-17T04:03:42.410029+0000","last_became_active":"2024-09-17T04:03:40.347326+0000","last_became_peered":"2024-09-17T04:03:40.347326+0000","last_unstale":"2024-09-17T04:03:42.410029+0000","last_undegraded":"2024-09-17T04:03:42.410029+0000","last_fullsized":"2024-09-17T04:03:42.410029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:56.328537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521658+0000","last_change":"2024-09-17T04:03:39.335709+0000","last_active":"2024-09-17T04:03:43.521658+0000","last_peered":"2024-09-17T04:03:43.521658+0000","last_clean":"2024-09-17T04:03:43.521658+0000","last_became_active":"2024-09-17T04:03:39.334739+0000","last_became_peered":"2024-09-17T04:03:39.334739+0000","last_unstale":"2024-09-17T04:03:43.521658+0000","last_undegraded":"2024-09-17T04:03:43.521658+0000","last_fullsized":"2024-09-17T04:03:43.521658+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741466+0000","last_change":"2024-09-17T04:03:40.335288+0000","last_active":"2024-09-17T04:03:42.741466+0000","last_peered":"2024-09-17T04:03:42.741466+0000","last_clean":"2024-09-17T04:03:42.741466+0000","last_became_active":"2024-09-17T04:03:40.334907+0000","last_became_peered":"2024-09-17T04:03:40.334907+0000","last_unstale":"2024-09-17T04:03:42.741466+0000","last_undegraded":"2024-09-17T04:03:42.741466+0000","last_fullsized":"2024-09-17T04:03:42.741466+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:15.982851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482130+0000","last_change":"2024-09-17T04:03:39.331744+0000","last_active":"2024-09-17T04:03:41.482130+0000","last_peered":"2024-09-17T04:03:41.482130+0000","last_clean":"2024-09-17T04:03:41.482130+0000","last_became_active":"2024-09-17T04:03:39.330827+0000","last_became_peered":"2024-09-17T04:03:39.330827+0000","last_unstale":"2024-09-17T04:03:41.482130+0000","last_undegraded":"2024-09-17T04:03:41.482130+0000","last_fullsized":"2024-09-17T04:03:41.482130+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:57.121963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482922+0000","last_change":"2024-09-17T04:03:40.334111+0000","last_active":"2024-09-17T04:03:41.482922+0000","last_peered":"2024-09-17T04:03:41.482922+0000","last_clean":"2024-09-17T04:03:41.482922+0000","last_became_active":"2024-09-17T04:03:40.333244+0000","last_became_peered":"2024-09-17T04:03:40.333244+0000","last_unstale":"2024-09-17T04:03:41.482922+0000","last_undegraded":"2024-09-17T04:03:41.482922+0000","last_fullsized":"2024-09-17T04:03:41.482922+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:20.936128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332507+0000","last_change":"2024-09-17T04:03:39.327400+0000","last_active":"2024-09-17T04:03:39.332507+0000","last_peered":"2024-09-17T04:03:39.332507+0000","last_clean":"2024-09-17T04:03:39.332507+0000","last_became_active":"2024-09-17T04:03:39.327045+0000","last_became_peered":"2024-09-17T04:03:39.327045+0000","last_unstale":"2024-09-17T04:03:39.332507+0000","last_undegraded":"2024-09-17T04:03:39.332507+0000","last_fullsized":"2024-09-17T04:03:39.332507+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:04.062871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315550+0000","last_change":"2024-09-17T04:03:39.315550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315550+0000","last_undegraded":"2024-09-17T04:03:39.315550+0000","last_fullsized":"2024-09-17T04:03:39.315550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"203'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333769+0000","last_change":"2024-09-17T04:03:39.332034+0000","last_active":"2024-09-17T04:03:41.333769+0000","last_peered":"2024-09-17T04:03:41.333769+0000","last_clean":"2024-09-17T04:03:41.333769+0000","last_became_active":"2024-09-17T04:03:39.331723+0000","last_became_peered":"2024-09-17T04:03:39.331723+0000","last_unstale":"2024-09-17T04:03:41.333769+0000","last_undegraded":"2024-09-17T04:03:41.333769+0000","last_fullsized":"2024-09-17T04:03:41.333769+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:01:32.008051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410209+0000","last_change":"2024-09-17T04:03:40.343426+0000","last_active":"2024-09-17T04:03:42.410209+0000","last_peered":"2024-09-17T04:03:42.410209+0000","last_clean":"2024-09-17T04:03:42.410209+0000","last_became_active":"2024-09-17T04:03:40.343092+0000","last_became_peered":"2024-09-17T04:03:40.343092+0000","last_unstale":"2024-09-17T04:03:42.410209+0000","last_undegraded":"2024-09-17T04:03:42.410209+0000","last_fullsized":"2024-09-17T04:03:42.410209+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:19.250495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521877+0000","last_change":"2024-09-17T04:03:39.335065+0000","last_active":"2024-09-17T04:03:43.521877+0000","last_peered":"2024-09-17T04:03:43.521877+0000","last_clean":"2024-09-17T04:03:43.521877+0000","last_became_active":"2024-09-17T04:03:39.334604+0000","last_became_peered":"2024-09-17T04:03:39.334604+0000","last_unstale":"2024-09-17T04:03:43.521877+0000","last_undegraded":"2024-09-17T04:03:43.521877+0000","last_fullsized":"2024-09-17T04:03:43.521877+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410698+0000","last_change":"2024-09-17T04:03:40.352477+0000","last_active":"2024-09-17T04:03:42.410698+0000","last_peered":"2024-09-17T04:03:42.410698+0000","last_clean":"2024-09-17T04:03:42.410698+0000","last_became_active":"2024-09-17T04:03:40.351366+0000","last_became_peered":"2024-09-17T04:03:40.351366+0000","last_unstale":"2024-09-17T04:03:42.410698+0000","last_undegraded":"2024-09-17T04:03:42.410698+0000","last_fullsized":"2024-09-17T04:03:42.410698+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:17:14.000165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410341+0000","last_change":"2024-09-17T04:03:39.335924+0000","last_active":"2024-09-17T04:03:42.410341+0000","last_peered":"2024-09-17T04:03:42.410341+0000","last_clean":"2024-09-17T04:03:42.410341+0000","last_became_active":"2024-09-17T04:03:39.335629+0000","last_became_peered":"2024-09-17T04:03:39.335629+0000","last_unstale":"2024-09-17T04:03:42.410341+0000","last_undegraded":"2024-09-17T04:03:42.410341+0000","last_fullsized":"2024-09-17T04:03:42.410341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:52.490212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482687+0000","last_change":"2024-09-17T04:03:40.339516+0000","last_active":"2024-09-17T04:03:41.482687+0000","last_peered":"2024-09-17T04:03:41.482687+0000","last_clean":"2024-09-17T04:03:41.482687+0000","last_became_active":"2024-09-17T04:03:40.339040+0000","last_became_peered":"2024-09-17T04:03:40.339040+0000","last_unstale":"2024-09-17T04:03:41.482687+0000","last_undegraded":"2024-09-17T04:03:41.482687+0000","last_fullsized":"2024-09-17T04:03:41.482687+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:33.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482477+0000","last_change":"2024-09-17T04:03:39.331355+0000","last_active":"2024-09-17T04:03:41.482477+0000","last_peered":"2024-09-17T04:03:41.482477+0000","last_clean":"2024-09-17T04:03:41.482477+0000","last_became_active":"2024-09-17T04:03:39.331003+0000","last_became_peered":"2024-09-17T04:03:39.331003+0000","last_unstale":"2024-09-17T04:03:41.482477+0000","last_undegraded":"2024-09-17T04:03:41.482477+0000","last_fullsized":"2024-09-17T04:03:41.482477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:52.147204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741734+0000","last_change":"2024-09-17T04:03:40.337242+0000","last_active":"2024-09-17T04:03:42.741734+0000","last_peered":"2024-09-17T04:03:42.741734+0000","last_clean":"2024-09-17T04:03:42.741734+0000","last_became_active":"2024-09-17T04:03:40.336866+0000","last_became_peered":"2024-09-17T04:03:40.336866+0000","last_unstale":"2024-09-17T04:03:42.741734+0000","last_undegraded":"2024-09-17T04:03:42.741734+0000","last_fullsized":"2024-09-17T04:03:42.741734+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:52.701613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.336329+0000","last_change":"2024-09-17T04:03:39.332196+0000","last_active":"2024-09-17T04:03:39.336329+0000","last_peered":"2024-09-17T04:03:39.336329+0000","last_clean":"2024-09-17T04:03:39.336329+0000","last_became_active":"2024-09-17T04:03:39.331897+0000","last_became_peered":"2024-09-17T04:03:39.331897+0000","last_unstale":"2024-09-17T04:03:39.336329+0000","last_undegraded":"2024-09-17T04:03:39.336329+0000","last_fullsized":"2024-09-17T04:03:39.336329+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:10.918846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741362+0000","last_change":"2024-09-17T04:03:40.331229+0000","last_active":"2024-09-17T04:03:42.741362+0000","last_peered":"2024-09-17T04:03:42.741362+0000","last_clean":"2024-09-17T04:03:42.741362+0000","last_became_active":"2024-09-17T04:03:40.330857+0000","last_became_peered":"2024-09-17T04:03:40.330857+0000","last_unstale":"2024-09-17T04:03:42.741362+0000","last_undegraded":"2024-09-17T04:03:42.741362+0000","last_fullsized":"2024-09-17T04:03:42.741362+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:18.666966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334106+0000","last_change":"2024-09-17T04:03:39.329983+0000","last_active":"2024-09-17T04:03:39.334106+0000","last_peered":"2024-09-17T04:03:39.334106+0000","last_clean":"2024-09-17T04:03:39.334106+0000","last_became_active":"2024-09-17T04:03:39.325670+0000","last_became_peered":"2024-09-17T04:03:39.325670+0000","last_unstale":"2024-09-17T04:03:39.334106+0000","last_undegraded":"2024-09-17T04:03:39.334106+0000","last_fullsized":"2024-09-17T04:03:39.334106+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:20.865772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314550+0000","last_change":"2024-09-17T04:03:39.314550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314550+0000","last_undegraded":"2024-09-17T04:03:39.314550+0000","last_fullsized":"2024-09-17T04:03:39.314550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334879+0000","last_change":"2024-09-17T04:03:39.332559+0000","last_active":"2024-09-17T04:03:39.334879+0000","last_peered":"2024-09-17T04:03:39.334879+0000","last_clean":"2024-09-17T04:03:39.334879+0000","last_became_active":"2024-09-17T04:03:39.332265+0000","last_became_peered":"2024-09-17T04:03:39.332265+0000","last_unstale":"2024-09-17T04:03:39.334879+0000","last_undegraded":"2024-09-17T04:03:39.334879+0000","last_fullsized":"2024-09-17T04:03:39.334879+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:07.087568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333644+0000","last_change":"2024-09-17T04:03:40.352076+0000","last_active":"2024-09-17T04:03:41.333644+0000","last_peered":"2024-09-17T04:03:41.333644+0000","last_clean":"2024-09-17T04:03:41.333644+0000","last_became_active":"2024-09-17T04:03:40.351735+0000","last_became_peered":"2024-09-17T04:03:40.351735+0000","last_unstale":"2024-09-17T04:03:41.333644+0000","last_undegraded":"2024-09-17T04:03:41.333644+0000","last_fullsized":"2024-09-17T04:03:41.333644+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:16.809754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334891+0000","last_change":"2024-09-17T04:03:39.331554+0000","last_active":"2024-09-17T04:03:39.334891+0000","last_peered":"2024-09-17T04:03:39.334891+0000","last_clean":"2024-09-17T04:03:39.334891+0000","last_became_active":"2024-09-17T04:03:39.331212+0000","last_became_peered":"2024-09-17T04:03:39.331212+0000","last_unstale":"2024-09-17T04:03:39.334891+0000","last_undegraded":"2024-09-17T04:03:39.334891+0000","last_fullsized":"2024-09-17T04:03:39.334891+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:04.328735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":446,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521595+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:03:43.521595+0000","last_peered":"2024-09-17T04:03:43.521595+0000","last_clean":"2024-09-17T04:03:43.521595+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:03:43.521595+0000","last_undegraded":"2024-09-17T04:03:43.521595+0000","last_fullsized":"2024-09-17T04:03:43.521595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52320,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319520,"statfs":{"total":96636764160,"available":96583188480,"internally_reserved":0,"allocated":3158016,"data_stored":1716897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29528,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342312,"statfs":{"total":96636764160,"available":96606527488,"internally_reserved":0,"allocated":2560000,"data_stored":1126459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509616,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51716,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320124,"statfs":{"total":96636764160,"available":96583806976,"internally_reserved":0,"allocated":2535424,"data_stored":1113384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509615,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542318,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1127063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542319,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3158016,"data_stored":1716861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:03:44.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:03:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 -- 172.21.15.129:0/2315044610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0cfb60 msgr2=0x7f43cc160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 --2- 172.21.15.129:0/2315044610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0cfb60 0x7f43cc160cf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f43bc009830 tx=0x7f43bc02a480 comp rx=0 tx=0).stop 2024-09-17T04:03:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 -- 172.21.15.129:0/2315044610 shutdown_connections 2024-09-17T04:03:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 --2- 172.21.15.129:0/2315044610 >> v2:172.21.15.181:3300/0 conn(0x7f43cc0b46e0 0x7f43cc0b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 --2- 172.21.15.129:0/2315044610 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0cfb60 0x7f43cc160cf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 -- 172.21.15.129:0/2315044610 >> 172.21.15.129:0/2315044610 conn(0x7f43cc06cfc0 msgr2=0x7f43cc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 -- 172.21.15.129:0/2315044610 shutdown_connections 2024-09-17T04:03:45.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.012+0000 7f43d2400640 1 -- 172.21.15.129:0/2315044610 wait complete. 2024-09-17T04:03:45.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.014+0000 7f43d2400640 1 Processor -- start 2024-09-17T04:03:45.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.014+0000 7f43d2400640 1 -- start start 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0b46e0 0x7f43cc0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43cc0d02c0 0x7f43cc1e8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f43cc15d270 con 0x7f43cc0cfb60 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43cc15d0f0 con 0x7f43cc0d02c0 2024-09-17T04:03:45.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d2400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f43cc15d3f0 con 0x7f43cc0b46e0 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46068/0 (socket says 172.21.15.129:46068) 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 -- 172.21.15.129:0/2460345791 learned_addr learned my addr 172.21.15.129:0/2460345791 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d1000640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0b46e0 0x7f43cc0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43d1a00640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43cc0d02c0 0x7f43cc1e8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 -- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0b46e0 msgr2=0x7f43cc0c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0b46e0 0x7f43cc0c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 -- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43cc0d02c0 msgr2=0x7f43cc1e8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43cc0d02c0 0x7f43cc1e8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.016+0000 7f43c3e00640 1 -- 172.21.15.129:0/2460345791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43cc0b2630 con 0x7f43cc0cfb60 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43c3e00640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f43b400e9f0 tx=0x7f43b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b400cd20 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f43b4004020 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43cc072d30 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4004450 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.017+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43cc1f9ad0 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.018+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43cc072fd0 con 0x7f43cc0cfb60 2024-09-17T04:03:45.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.019+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f43b40035f0 con 0x7f43cc0cfb60 2024-09-17T04:03:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.020+0000 7f43c9600640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 0x7f43a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.021+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f43b409b5d0 con 0x7f43cc0cfb60 2024-09-17T04:03:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.021+0000 7f43d1000640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 0x7f43a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.022+0000 7f43d1000640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 0x7f43a807b0a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f43bc034000 tx=0x7f43bc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.025+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f43b40633b0 con 0x7f43cc0cfb60 2024-09-17T04:03:45.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.201+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f43cc0b2b10 con 0x7f43a8078b90 2024-09-17T04:03:45.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.206+0000 7f43c9600640 1 -- 172.21.15.129:0/2460345791 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250367 (secure 0 0 0) 0x7f43cc0b2b10 con 0x7f43a8078b90 2024-09-17T04:03:45.209 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:45.210 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:03:45.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 msgr2=0x7f43a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 0x7f43a807b0a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f43bc034000 tx=0x7f43bc004730 comp rx=0 tx=0).stop 2024-09-17T04:03:45.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 msgr2=0x7f43cc1f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f43b400e9f0 tx=0x7f43b400eeb0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 shutdown_connections 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f43a8078b90 0x7f43a807b0a0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43cc0d02c0 0x7f43cc1e8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43cc0cfb60 0x7f43cc1f90d0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 --2- 172.21.15.129:0/2460345791 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f43cc0b46e0 0x7f43cc0c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 >> 172.21.15.129:0/2460345791 conn(0x7f43cc06cfc0 msgr2=0x7f43cc158cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 shutdown_connections 2024-09-17T04:03:45.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.210+0000 7f43d2400640 1 -- 172.21.15.129:0/2460345791 wait complete. 2024-09-17T04:03:45.236 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-17T04:03:44.536475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590460,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374740,"kb_used_data":21100,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754599980,"statfs":{"total":773094113280,"available":772710379520,"internally_reserved":0,"allocated":21606400,"data_stored":10143045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":361942576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001886"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315975+0000","last_change":"2024-09-17T04:03:39.315975+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315975+0000","last_undegraded":"2024-09-17T04:03:39.315975+0000","last_fullsized":"2024-09-17T04:03:39.315975+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314425+0000","last_change":"2024-09-17T04:03:39.314425+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314425+0000","last_undegraded":"2024-09-17T04:03:39.314425+0000","last_fullsized":"2024-09-17T04:03:39.314425+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313411+0000","last_change":"2024-09-17T04:03:39.313411+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313411+0000","last_undegraded":"2024-09-17T04:03:39.313411+0000","last_fullsized":"2024-09-17T04:03:39.313411+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318949+0000","last_change":"2024-09-17T04:03:39.318949+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318949+0000","last_undegraded":"2024-09-17T04:03:39.318949+0000","last_fullsized":"2024-09-17T04:03:39.318949+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333497+0000","last_change":"2024-09-17T04:03:40.335355+0000","last_active":"2024-09-17T04:03:41.333497+0000","last_peered":"2024-09-17T04:03:41.333497+0000","last_clean":"2024-09-17T04:03:41.333497+0000","last_became_active":"2024-09-17T04:03:40.334994+0000","last_became_peered":"2024-09-17T04:03:40.334994+0000","last_unstale":"2024-09-17T04:03:41.333497+0000","last_undegraded":"2024-09-17T04:03:41.333497+0000","last_fullsized":"2024-09-17T04:03:41.333497+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:02.228911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317948+0000","last_change":"2024-09-17T04:03:39.317948+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317948+0000","last_undegraded":"2024-09-17T04:03:39.317948+0000","last_fullsized":"2024-09-17T04:03:39.317948+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482089+0000","last_change":"2024-09-17T04:03:40.350322+0000","last_active":"2024-09-17T04:03:41.482089+0000","last_peered":"2024-09-17T04:03:41.482089+0000","last_clean":"2024-09-17T04:03:41.482089+0000","last_became_active":"2024-09-17T04:03:40.349953+0000","last_became_peered":"2024-09-17T04:03:40.349953+0000","last_unstale":"2024-09-17T04:03:41.482089+0000","last_undegraded":"2024-09-17T04:03:41.482089+0000","last_fullsized":"2024-09-17T04:03:41.482089+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:03:12.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741207+0000","last_change":"2024-09-17T04:03:40.334851+0000","last_active":"2024-09-17T04:03:42.741207+0000","last_peered":"2024-09-17T04:03:42.741207+0000","last_clean":"2024-09-17T04:03:42.741207+0000","last_became_active":"2024-09-17T04:03:40.331337+0000","last_became_peered":"2024-09-17T04:03:40.331337+0000","last_unstale":"2024-09-17T04:03:42.741207+0000","last_undegraded":"2024-09-17T04:03:42.741207+0000","last_fullsized":"2024-09-17T04:03:42.741207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:52.463737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521788+0000","last_change":"2024-09-17T04:03:40.349511+0000","last_active":"2024-09-17T04:03:43.521788+0000","last_peered":"2024-09-17T04:03:43.521788+0000","last_clean":"2024-09-17T04:03:43.521788+0000","last_became_active":"2024-09-17T04:03:40.349113+0000","last_became_peered":"2024-09-17T04:03:40.349113+0000","last_unstale":"2024-09-17T04:03:43.521788+0000","last_undegraded":"2024-09-17T04:03:43.521788+0000","last_fullsized":"2024-09-17T04:03:43.521788+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:50.920504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482695+0000","last_change":"2024-09-17T04:03:40.340480+0000","last_active":"2024-09-17T04:03:41.482695+0000","last_peered":"2024-09-17T04:03:41.482695+0000","last_clean":"2024-09-17T04:03:41.482695+0000","last_became_active":"2024-09-17T04:03:40.338521+0000","last_became_peered":"2024-09-17T04:03:40.338521+0000","last_unstale":"2024-09-17T04:03:41.482695+0000","last_undegraded":"2024-09-17T04:03:41.482695+0000","last_fullsized":"2024-09-17T04:03:41.482695+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:10.458690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410478+0000","last_change":"2024-09-17T04:03:40.348952+0000","last_active":"2024-09-17T04:03:42.410478+0000","last_peered":"2024-09-17T04:03:42.410478+0000","last_clean":"2024-09-17T04:03:42.410478+0000","last_became_active":"2024-09-17T04:03:40.348656+0000","last_became_peered":"2024-09-17T04:03:40.348656+0000","last_unstale":"2024-09-17T04:03:42.410478+0000","last_undegraded":"2024-09-17T04:03:42.410478+0000","last_fullsized":"2024-09-17T04:03:42.410478+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:47.744159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741493+0000","last_change":"2024-09-17T04:03:40.354695+0000","last_active":"2024-09-17T04:03:42.741493+0000","last_peered":"2024-09-17T04:03:42.741493+0000","last_clean":"2024-09-17T04:03:42.741493+0000","last_became_active":"2024-09-17T04:03:40.354274+0000","last_became_peered":"2024-09-17T04:03:40.354274+0000","last_unstale":"2024-09-17T04:03:42.741493+0000","last_undegraded":"2024-09-17T04:03:42.741493+0000","last_fullsized":"2024-09-17T04:03:42.741493+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:15.701377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482731+0000","last_change":"2024-09-17T04:03:40.350373+0000","last_active":"2024-09-17T04:03:41.482731+0000","last_peered":"2024-09-17T04:03:41.482731+0000","last_clean":"2024-09-17T04:03:41.482731+0000","last_became_active":"2024-09-17T04:03:40.350000+0000","last_became_peered":"2024-09-17T04:03:40.350000+0000","last_unstale":"2024-09-17T04:03:41.482731+0000","last_undegraded":"2024-09-17T04:03:41.482731+0000","last_fullsized":"2024-09-17T04:03:41.482731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:29.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312559+0000","last_change":"2024-09-17T04:03:39.312559+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312559+0000","last_undegraded":"2024-09-17T04:03:39.312559+0000","last_fullsized":"2024-09-17T04:03:39.312559+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521536+0000","last_change":"2024-09-17T04:03:40.344294+0000","last_active":"2024-09-17T04:03:43.521536+0000","last_peered":"2024-09-17T04:03:43.521536+0000","last_clean":"2024-09-17T04:03:43.521536+0000","last_became_active":"2024-09-17T04:03:40.343923+0000","last_became_peered":"2024-09-17T04:03:40.343923+0000","last_unstale":"2024-09-17T04:03:43.521536+0000","last_undegraded":"2024-09-17T04:03:43.521536+0000","last_fullsized":"2024-09-17T04:03:43.521536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:35.362812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313125+0000","last_change":"2024-09-17T04:03:39.313125+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313125+0000","last_undegraded":"2024-09-17T04:03:39.313125+0000","last_fullsized":"2024-09-17T04:03:39.313125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741536+0000","last_change":"2024-09-17T04:03:40.343695+0000","last_active":"2024-09-17T04:03:42.741536+0000","last_peered":"2024-09-17T04:03:42.741536+0000","last_clean":"2024-09-17T04:03:42.741536+0000","last_became_active":"2024-09-17T04:03:40.343413+0000","last_became_peered":"2024-09-17T04:03:40.343413+0000","last_unstale":"2024-09-17T04:03:42.741536+0000","last_undegraded":"2024-09-17T04:03:42.741536+0000","last_fullsized":"2024-09-17T04:03:42.741536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:54.092079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313567+0000","last_change":"2024-09-17T04:03:39.313567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313567+0000","last_undegraded":"2024-09-17T04:03:39.313567+0000","last_fullsized":"2024-09-17T04:03:39.313567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:40.351995+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:40.351649+0000","last_became_peered":"2024-09-17T04:03:40.351649+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:03.942253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313824+0000","last_change":"2024-09-17T04:03:39.313824+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313824+0000","last_undegraded":"2024-09-17T04:03:39.313824+0000","last_fullsized":"2024-09-17T04:03:39.313824+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741813+0000","last_change":"2024-09-17T04:03:40.333277+0000","last_active":"2024-09-17T04:03:42.741813+0000","last_peered":"2024-09-17T04:03:42.741813+0000","last_clean":"2024-09-17T04:03:42.741813+0000","last_became_active":"2024-09-17T04:03:40.332882+0000","last_became_peered":"2024-09-17T04:03:40.332882+0000","last_unstale":"2024-09-17T04:03:42.741813+0000","last_undegraded":"2024-09-17T04:03:42.741813+0000","last_fullsized":"2024-09-17T04:03:42.741813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:44.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482369+0000","last_change":"2024-09-17T04:03:40.341616+0000","last_active":"2024-09-17T04:03:41.482369+0000","last_peered":"2024-09-17T04:03:41.482369+0000","last_clean":"2024-09-17T04:03:41.482369+0000","last_became_active":"2024-09-17T04:03:40.341269+0000","last_became_peered":"2024-09-17T04:03:40.341269+0000","last_unstale":"2024-09-17T04:03:41.482369+0000","last_undegraded":"2024-09-17T04:03:41.482369+0000","last_fullsized":"2024-09-17T04:03:41.482369+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:28.676331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741352+0000","last_change":"2024-09-17T04:03:40.337777+0000","last_active":"2024-09-17T04:03:42.741352+0000","last_peered":"2024-09-17T04:03:42.741352+0000","last_clean":"2024-09-17T04:03:42.741352+0000","last_became_active":"2024-09-17T04:03:40.335850+0000","last_became_peered":"2024-09-17T04:03:40.335850+0000","last_unstale":"2024-09-17T04:03:42.741352+0000","last_undegraded":"2024-09-17T04:03:42.741352+0000","last_fullsized":"2024-09-17T04:03:42.741352+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:37.334408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316858+0000","last_change":"2024-09-17T04:03:39.316858+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316858+0000","last_undegraded":"2024-09-17T04:03:39.316858+0000","last_fullsized":"2024-09-17T04:03:39.316858+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.324689+0000","last_change":"2024-09-17T04:03:39.324689+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.324689+0000","last_undegraded":"2024-09-17T04:03:39.324689+0000","last_fullsized":"2024-09-17T04:03:39.324689+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482816+0000","last_change":"2024-09-17T04:03:40.350740+0000","last_active":"2024-09-17T04:03:41.482816+0000","last_peered":"2024-09-17T04:03:41.482816+0000","last_clean":"2024-09-17T04:03:41.482816+0000","last_became_active":"2024-09-17T04:03:40.350415+0000","last_became_peered":"2024-09-17T04:03:40.350415+0000","last_unstale":"2024-09-17T04:03:41.482816+0000","last_undegraded":"2024-09-17T04:03:41.482816+0000","last_fullsized":"2024-09-17T04:03:41.482816+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:04.588026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410543+0000","last_change":"2024-09-17T04:03:40.347688+0000","last_active":"2024-09-17T04:03:42.410543+0000","last_peered":"2024-09-17T04:03:42.410543+0000","last_clean":"2024-09-17T04:03:42.410543+0000","last_became_active":"2024-09-17T04:03:40.347381+0000","last_became_peered":"2024-09-17T04:03:40.347381+0000","last_unstale":"2024-09-17T04:03:42.410543+0000","last_undegraded":"2024-09-17T04:03:42.410543+0000","last_fullsized":"2024-09-17T04:03:42.410543+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:51.255876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312330+0000","last_change":"2024-09-17T04:03:39.312330+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312330+0000","last_undegraded":"2024-09-17T04:03:39.312330+0000","last_fullsized":"2024-09-17T04:03:39.312330+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410325+0000","last_change":"2024-09-17T04:03:40.350667+0000","last_active":"2024-09-17T04:03:42.410325+0000","last_peered":"2024-09-17T04:03:42.410325+0000","last_clean":"2024-09-17T04:03:42.410325+0000","last_became_active":"2024-09-17T04:03:40.350312+0000","last_became_peered":"2024-09-17T04:03:40.350312+0000","last_unstale":"2024-09-17T04:03:42.410325+0000","last_undegraded":"2024-09-17T04:03:42.410325+0000","last_fullsized":"2024-09-17T04:03:42.410325+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:21.193199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333504+0000","last_change":"2024-09-17T04:03:40.331612+0000","last_active":"2024-09-17T04:03:41.333504+0000","last_peered":"2024-09-17T04:03:41.333504+0000","last_clean":"2024-09-17T04:03:41.333504+0000","last_became_active":"2024-09-17T04:03:40.329978+0000","last_became_peered":"2024-09-17T04:03:40.329978+0000","last_unstale":"2024-09-17T04:03:41.333504+0000","last_undegraded":"2024-09-17T04:03:41.333504+0000","last_fullsized":"2024-09-17T04:03:41.333504+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:48.029240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.321401+0000","last_change":"2024-09-17T04:03:39.321401+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.321401+0000","last_undegraded":"2024-09-17T04:03:39.321401+0000","last_fullsized":"2024-09-17T04:03:39.321401+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316435+0000","last_change":"2024-09-17T04:03:39.316435+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316435+0000","last_undegraded":"2024-09-17T04:03:39.316435+0000","last_fullsized":"2024-09-17T04:03:39.316435+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312535+0000","last_change":"2024-09-17T04:03:39.312535+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312535+0000","last_undegraded":"2024-09-17T04:03:39.312535+0000","last_fullsized":"2024-09-17T04:03:39.312535+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482954+0000","last_change":"2024-09-17T04:03:39.330078+0000","last_active":"2024-09-17T04:03:41.482954+0000","last_peered":"2024-09-17T04:03:41.482954+0000","last_clean":"2024-09-17T04:03:41.482954+0000","last_became_active":"2024-09-17T04:03:39.329766+0000","last_became_peered":"2024-09-17T04:03:39.329766+0000","last_unstale":"2024-09-17T04:03:41.482954+0000","last_undegraded":"2024-09-17T04:03:41.482954+0000","last_fullsized":"2024-09-17T04:03:41.482954+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:11.195328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410186+0000","last_change":"2024-09-17T04:03:40.351683+0000","last_active":"2024-09-17T04:03:42.410186+0000","last_peered":"2024-09-17T04:03:42.410186+0000","last_clean":"2024-09-17T04:03:42.410186+0000","last_became_active":"2024-09-17T04:03:40.351288+0000","last_became_peered":"2024-09-17T04:03:40.351288+0000","last_unstale":"2024-09-17T04:03:42.410186+0000","last_undegraded":"2024-09-17T04:03:42.410186+0000","last_fullsized":"2024-09-17T04:03:42.410186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:11:57.454665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326983+0000","last_change":"2024-09-17T04:03:39.322960+0000","last_active":"2024-09-17T04:03:39.326983+0000","last_peered":"2024-09-17T04:03:39.326983+0000","last_clean":"2024-09-17T04:03:39.326983+0000","last_became_active":"2024-09-17T04:03:39.322704+0000","last_became_peered":"2024-09-17T04:03:39.322704+0000","last_unstale":"2024-09-17T04:03:39.326983+0000","last_undegraded":"2024-09-17T04:03:39.326983+0000","last_fullsized":"2024-09-17T04:03:39.326983+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741839+0000","last_change":"2024-09-17T04:03:40.343840+0000","last_active":"2024-09-17T04:03:42.741839+0000","last_peered":"2024-09-17T04:03:42.741839+0000","last_clean":"2024-09-17T04:03:42.741839+0000","last_became_active":"2024-09-17T04:03:40.343539+0000","last_became_peered":"2024-09-17T04:03:40.343539+0000","last_unstale":"2024-09-17T04:03:42.741839+0000","last_undegraded":"2024-09-17T04:03:42.741839+0000","last_fullsized":"2024-09-17T04:03:42.741839+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:04.501121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.325189+0000","last_change":"2024-09-17T04:03:39.320734+0000","last_active":"2024-09-17T04:03:39.325189+0000","last_peered":"2024-09-17T04:03:39.325189+0000","last_clean":"2024-09-17T04:03:39.325189+0000","last_became_active":"2024-09-17T04:03:39.320431+0000","last_became_peered":"2024-09-17T04:03:39.320431+0000","last_unstale":"2024-09-17T04:03:39.325189+0000","last_undegraded":"2024-09-17T04:03:39.325189+0000","last_fullsized":"2024-09-17T04:03:39.325189+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:51.187815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410125+0000","last_change":"2024-09-17T04:03:40.335363+0000","last_active":"2024-09-17T04:03:42.410125+0000","last_peered":"2024-09-17T04:03:42.410125+0000","last_clean":"2024-09-17T04:03:42.410125+0000","last_became_active":"2024-09-17T04:03:40.333333+0000","last_became_peered":"2024-09-17T04:03:40.333333+0000","last_unstale":"2024-09-17T04:03:42.410125+0000","last_undegraded":"2024-09-17T04:03:42.410125+0000","last_fullsized":"2024-09-17T04:03:42.410125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:35.331891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334393+0000","last_change":"2024-09-17T04:03:39.329538+0000","last_active":"2024-09-17T04:03:39.334393+0000","last_peered":"2024-09-17T04:03:39.334393+0000","last_clean":"2024-09-17T04:03:39.334393+0000","last_became_active":"2024-09-17T04:03:39.329257+0000","last_became_peered":"2024-09-17T04:03:39.329257+0000","last_unstale":"2024-09-17T04:03:39.334393+0000","last_undegraded":"2024-09-17T04:03:39.334393+0000","last_fullsized":"2024-09-17T04:03:39.334393+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:25.093614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521725+0000","last_change":"2024-09-17T04:03:40.344266+0000","last_active":"2024-09-17T04:03:43.521725+0000","last_peered":"2024-09-17T04:03:43.521725+0000","last_clean":"2024-09-17T04:03:43.521725+0000","last_became_active":"2024-09-17T04:03:40.343914+0000","last_became_peered":"2024-09-17T04:03:40.343914+0000","last_unstale":"2024-09-17T04:03:43.521725+0000","last_undegraded":"2024-09-17T04:03:43.521725+0000","last_fullsized":"2024-09-17T04:03:43.521725+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:49.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326649+0000","last_change":"2024-09-17T04:03:39.321662+0000","last_active":"2024-09-17T04:03:39.326649+0000","last_peered":"2024-09-17T04:03:39.326649+0000","last_clean":"2024-09-17T04:03:39.326649+0000","last_became_active":"2024-09-17T04:03:39.321278+0000","last_became_peered":"2024-09-17T04:03:39.321278+0000","last_unstale":"2024-09-17T04:03:39.326649+0000","last_undegraded":"2024-09-17T04:03:39.326649+0000","last_fullsized":"2024-09-17T04:03:39.326649+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.772774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312542+0000","last_change":"2024-09-17T04:03:39.312542+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312542+0000","last_undegraded":"2024-09-17T04:03:39.312542+0000","last_fullsized":"2024-09-17T04:03:39.312542+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482232+0000","last_change":"2024-09-17T04:03:39.332398+0000","last_active":"2024-09-17T04:03:41.482232+0000","last_peered":"2024-09-17T04:03:41.482232+0000","last_clean":"2024-09-17T04:03:41.482232+0000","last_became_active":"2024-09-17T04:03:39.331945+0000","last_became_peered":"2024-09-17T04:03:39.331945+0000","last_unstale":"2024-09-17T04:03:41.482232+0000","last_undegraded":"2024-09-17T04:03:41.482232+0000","last_fullsized":"2024-09-17T04:03:41.482232+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.951852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314983+0000","last_change":"2024-09-17T04:03:39.314983+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314983+0000","last_undegraded":"2024-09-17T04:03:39.314983+0000","last_fullsized":"2024-09-17T04:03:39.314983+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.324526+0000","last_change":"2024-09-17T04:03:39.317105+0000","last_active":"2024-09-17T04:03:39.324526+0000","last_peered":"2024-09-17T04:03:39.324526+0000","last_clean":"2024-09-17T04:03:39.324526+0000","last_became_active":"2024-09-17T04:03:39.316779+0000","last_became_peered":"2024-09-17T04:03:39.316779+0000","last_unstale":"2024-09-17T04:03:39.324526+0000","last_undegraded":"2024-09-17T04:03:39.324526+0000","last_fullsized":"2024-09-17T04:03:39.324526+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:21.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313113+0000","last_change":"2024-09-17T04:03:39.313113+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313113+0000","last_undegraded":"2024-09-17T04:03:39.313113+0000","last_fullsized":"2024-09-17T04:03:39.313113+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:39.326124+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:39.325797+0000","last_became_peered":"2024-09-17T04:03:39.325797+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:04.499200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410736+0000","last_change":"2024-09-17T04:03:40.351885+0000","last_active":"2024-09-17T04:03:42.410736+0000","last_peered":"2024-09-17T04:03:42.410736+0000","last_clean":"2024-09-17T04:03:42.410736+0000","last_became_active":"2024-09-17T04:03:40.351494+0000","last_became_peered":"2024-09-17T04:03:40.351494+0000","last_unstale":"2024-09-17T04:03:42.410736+0000","last_undegraded":"2024-09-17T04:03:42.410736+0000","last_fullsized":"2024-09-17T04:03:42.410736+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:13.184588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333987+0000","last_change":"2024-09-17T04:03:39.328133+0000","last_active":"2024-09-17T04:03:39.333987+0000","last_peered":"2024-09-17T04:03:39.333987+0000","last_clean":"2024-09-17T04:03:39.333987+0000","last_became_active":"2024-09-17T04:03:39.327863+0000","last_became_peered":"2024-09-17T04:03:39.327863+0000","last_unstale":"2024-09-17T04:03:39.333987+0000","last_undegraded":"2024-09-17T04:03:39.333987+0000","last_fullsized":"2024-09-17T04:03:39.333987+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:55.809447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311933+0000","last_change":"2024-09-17T04:03:39.311933+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311933+0000","last_undegraded":"2024-09-17T04:03:39.311933+0000","last_fullsized":"2024-09-17T04:03:39.311933+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332935+0000","last_change":"2024-09-17T04:03:39.331097+0000","last_active":"2024-09-17T04:03:39.332935+0000","last_peered":"2024-09-17T04:03:39.332935+0000","last_clean":"2024-09-17T04:03:39.332935+0000","last_became_active":"2024-09-17T04:03:39.330131+0000","last_became_peered":"2024-09-17T04:03:39.330131+0000","last_unstale":"2024-09-17T04:03:39.332935+0000","last_undegraded":"2024-09-17T04:03:39.332935+0000","last_fullsized":"2024-09-17T04:03:39.332935+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:57:40.153994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482540+0000","last_change":"2024-09-17T04:03:40.332717+0000","last_active":"2024-09-17T04:03:41.482540+0000","last_peered":"2024-09-17T04:03:41.482540+0000","last_clean":"2024-09-17T04:03:41.482540+0000","last_became_active":"2024-09-17T04:03:40.332421+0000","last_became_peered":"2024-09-17T04:03:40.332421+0000","last_unstale":"2024-09-17T04:03:41.482540+0000","last_undegraded":"2024-09-17T04:03:41.482540+0000","last_fullsized":"2024-09-17T04:03:41.482540+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:42.530742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332906+0000","last_change":"2024-09-17T04:03:39.329612+0000","last_active":"2024-09-17T04:03:39.332906+0000","last_peered":"2024-09-17T04:03:39.332906+0000","last_clean":"2024-09-17T04:03:39.332906+0000","last_became_active":"2024-09-17T04:03:39.329342+0000","last_became_peered":"2024-09-17T04:03:39.329342+0000","last_unstale":"2024-09-17T04:03:39.332906+0000","last_undegraded":"2024-09-17T04:03:39.332906+0000","last_fullsized":"2024-09-17T04:03:39.332906+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:14.980173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482365+0000","last_change":"2024-09-17T04:03:40.338842+0000","last_active":"2024-09-17T04:03:41.482365+0000","last_peered":"2024-09-17T04:03:41.482365+0000","last_clean":"2024-09-17T04:03:41.482365+0000","last_became_active":"2024-09-17T04:03:40.338449+0000","last_became_peered":"2024-09-17T04:03:40.338449+0000","last_unstale":"2024-09-17T04:03:41.482365+0000","last_undegraded":"2024-09-17T04:03:41.482365+0000","last_fullsized":"2024-09-17T04:03:41.482365+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:44.963272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.328955+0000","last_change":"2024-09-17T04:03:39.324560+0000","last_active":"2024-09-17T04:03:39.328955+0000","last_peered":"2024-09-17T04:03:39.328955+0000","last_clean":"2024-09-17T04:03:39.328955+0000","last_became_active":"2024-09-17T04:03:39.324206+0000","last_became_peered":"2024-09-17T04:03:39.324206+0000","last_unstale":"2024-09-17T04:03:39.328955+0000","last_undegraded":"2024-09-17T04:03:39.328955+0000","last_fullsized":"2024-09-17T04:03:39.328955+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:40.444284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521913+0000","last_change":"2024-09-17T04:03:40.344849+0000","last_active":"2024-09-17T04:03:43.521913+0000","last_peered":"2024-09-17T04:03:43.521913+0000","last_clean":"2024-09-17T04:03:43.521913+0000","last_became_active":"2024-09-17T04:03:40.344173+0000","last_became_peered":"2024-09-17T04:03:40.344173+0000","last_unstale":"2024-09-17T04:03:43.521913+0000","last_undegraded":"2024-09-17T04:03:43.521913+0000","last_fullsized":"2024-09-17T04:03:43.521913+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:12.072676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333711+0000","last_change":"2024-09-17T04:03:39.327456+0000","last_active":"2024-09-17T04:03:39.333711+0000","last_peered":"2024-09-17T04:03:39.333711+0000","last_clean":"2024-09-17T04:03:39.333711+0000","last_became_active":"2024-09-17T04:03:39.326151+0000","last_became_peered":"2024-09-17T04:03:39.326151+0000","last_unstale":"2024-09-17T04:03:39.333711+0000","last_undegraded":"2024-09-17T04:03:39.333711+0000","last_fullsized":"2024-09-17T04:03:39.333711+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317212+0000","last_change":"2024-09-17T04:03:39.317212+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317212+0000","last_undegraded":"2024-09-17T04:03:39.317212+0000","last_fullsized":"2024-09-17T04:03:39.317212+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482297+0000","last_change":"2024-09-17T04:03:39.330613+0000","last_active":"2024-09-17T04:03:41.482297+0000","last_peered":"2024-09-17T04:03:41.482297+0000","last_clean":"2024-09-17T04:03:41.482297+0000","last_became_active":"2024-09-17T04:03:39.330283+0000","last_became_peered":"2024-09-17T04:03:39.330283+0000","last_unstale":"2024-09-17T04:03:41.482297+0000","last_undegraded":"2024-09-17T04:03:41.482297+0000","last_fullsized":"2024-09-17T04:03:41.482297+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:46.502036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482092+0000","last_change":"2024-09-17T04:03:40.341659+0000","last_active":"2024-09-17T04:03:41.482092+0000","last_peered":"2024-09-17T04:03:41.482092+0000","last_clean":"2024-09-17T04:03:41.482092+0000","last_became_active":"2024-09-17T04:03:40.341344+0000","last_became_peered":"2024-09-17T04:03:40.341344+0000","last_unstale":"2024-09-17T04:03:41.482092+0000","last_undegraded":"2024-09-17T04:03:41.482092+0000","last_fullsized":"2024-09-17T04:03:41.482092+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:24.293693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"202'8","reported_seq":23,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.409812+0000","last_change":"2024-09-17T04:03:39.332639+0000","last_active":"2024-09-17T04:03:42.409812+0000","last_peered":"2024-09-17T04:03:42.409812+0000","last_clean":"2024-09-17T04:03:42.409812+0000","last_became_active":"2024-09-17T04:03:39.332409+0000","last_became_peered":"2024-09-17T04:03:39.332409+0000","last_unstale":"2024-09-17T04:03:42.409812+0000","last_undegraded":"2024-09-17T04:03:42.409812+0000","last_fullsized":"2024-09-17T04:03:42.409812+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:03.585348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318088+0000","last_change":"2024-09-17T04:03:39.318088+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318088+0000","last_undegraded":"2024-09-17T04:03:39.318088+0000","last_fullsized":"2024-09-17T04:03:39.318088+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482664+0000","last_change":"2024-09-17T04:03:39.333660+0000","last_active":"2024-09-17T04:03:41.482664+0000","last_peered":"2024-09-17T04:03:41.482664+0000","last_clean":"2024-09-17T04:03:41.482664+0000","last_became_active":"2024-09-17T04:03:39.333330+0000","last_became_peered":"2024-09-17T04:03:39.333330+0000","last_unstale":"2024-09-17T04:03:41.482664+0000","last_undegraded":"2024-09-17T04:03:41.482664+0000","last_fullsized":"2024-09-17T04:03:41.482664+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:03.955054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311836+0000","last_change":"2024-09-17T04:03:39.311836+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311836+0000","last_undegraded":"2024-09-17T04:03:39.311836+0000","last_fullsized":"2024-09-17T04:03:39.311836+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326510+0000","last_change":"2024-09-17T04:03:39.322064+0000","last_active":"2024-09-17T04:03:39.326510+0000","last_peered":"2024-09-17T04:03:39.326510+0000","last_clean":"2024-09-17T04:03:39.326510+0000","last_became_active":"2024-09-17T04:03:39.321750+0000","last_became_peered":"2024-09-17T04:03:39.321750+0000","last_unstale":"2024-09-17T04:03:39.326510+0000","last_undegraded":"2024-09-17T04:03:39.326510+0000","last_fullsized":"2024-09-17T04:03:39.326510+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521646+0000","last_change":"2024-09-17T04:03:40.340280+0000","last_active":"2024-09-17T04:03:43.521646+0000","last_peered":"2024-09-17T04:03:43.521646+0000","last_clean":"2024-09-17T04:03:43.521646+0000","last_became_active":"2024-09-17T04:03:40.339904+0000","last_became_peered":"2024-09-17T04:03:40.339904+0000","last_unstale":"2024-09-17T04:03:43.521646+0000","last_undegraded":"2024-09-17T04:03:43.521646+0000","last_fullsized":"2024-09-17T04:03:43.521646+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:09.142973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521655+0000","last_change":"2024-09-17T04:03:39.335875+0000","last_active":"2024-09-17T04:03:43.521655+0000","last_peered":"2024-09-17T04:03:43.521655+0000","last_clean":"2024-09-17T04:03:43.521655+0000","last_became_active":"2024-09-17T04:03:39.335580+0000","last_became_peered":"2024-09-17T04:03:39.335580+0000","last_unstale":"2024-09-17T04:03:43.521655+0000","last_undegraded":"2024-09-17T04:03:43.521655+0000","last_fullsized":"2024-09-17T04:03:43.521655+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741903+0000","last_change":"2024-09-17T04:03:40.342551+0000","last_active":"2024-09-17T04:03:42.741903+0000","last_peered":"2024-09-17T04:03:42.741903+0000","last_clean":"2024-09-17T04:03:42.741903+0000","last_became_active":"2024-09-17T04:03:40.339096+0000","last_became_peered":"2024-09-17T04:03:40.339096+0000","last_unstale":"2024-09-17T04:03:42.741903+0000","last_undegraded":"2024-09-17T04:03:42.741903+0000","last_fullsized":"2024-09-17T04:03:42.741903+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:42.841396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410300+0000","last_change":"2024-09-17T04:03:39.331133+0000","last_active":"2024-09-17T04:03:42.410300+0000","last_peered":"2024-09-17T04:03:42.410300+0000","last_clean":"2024-09-17T04:03:42.410300+0000","last_became_active":"2024-09-17T04:03:39.330849+0000","last_became_peered":"2024-09-17T04:03:39.330849+0000","last_unstale":"2024-09-17T04:03:42.410300+0000","last_undegraded":"2024-09-17T04:03:42.410300+0000","last_fullsized":"2024-09-17T04:03:42.410300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:19.933450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741382+0000","last_change":"2024-09-17T04:03:40.339612+0000","last_active":"2024-09-17T04:03:42.741382+0000","last_peered":"2024-09-17T04:03:42.741382+0000","last_clean":"2024-09-17T04:03:42.741382+0000","last_became_active":"2024-09-17T04:03:40.339297+0000","last_became_peered":"2024-09-17T04:03:40.339297+0000","last_unstale":"2024-09-17T04:03:42.741382+0000","last_undegraded":"2024-09-17T04:03:42.741382+0000","last_fullsized":"2024-09-17T04:03:42.741382+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:14.950266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.335737+0000","last_change":"2024-09-17T04:03:39.334073+0000","last_active":"2024-09-17T04:03:39.335737+0000","last_peered":"2024-09-17T04:03:39.335737+0000","last_clean":"2024-09-17T04:03:39.335737+0000","last_became_active":"2024-09-17T04:03:39.333739+0000","last_became_peered":"2024-09-17T04:03:39.333739+0000","last_unstale":"2024-09-17T04:03:39.335737+0000","last_undegraded":"2024-09-17T04:03:39.335737+0000","last_fullsized":"2024-09-17T04:03:39.335737+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:00.772342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312567+0000","last_change":"2024-09-17T04:03:39.312567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312567+0000","last_undegraded":"2024-09-17T04:03:39.312567+0000","last_fullsized":"2024-09-17T04:03:39.312567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410122+0000","last_change":"2024-09-17T04:03:39.331198+0000","last_active":"2024-09-17T04:03:42.410122+0000","last_peered":"2024-09-17T04:03:42.410122+0000","last_clean":"2024-09-17T04:03:42.410122+0000","last_became_active":"2024-09-17T04:03:39.330893+0000","last_became_peered":"2024-09-17T04:03:39.330893+0000","last_unstale":"2024-09-17T04:03:42.410122+0000","last_undegraded":"2024-09-17T04:03:42.410122+0000","last_fullsized":"2024-09-17T04:03:42.410122+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:28.427519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410029+0000","last_change":"2024-09-17T04:03:40.347641+0000","last_active":"2024-09-17T04:03:42.410029+0000","last_peered":"2024-09-17T04:03:42.410029+0000","last_clean":"2024-09-17T04:03:42.410029+0000","last_became_active":"2024-09-17T04:03:40.347326+0000","last_became_peered":"2024-09-17T04:03:40.347326+0000","last_unstale":"2024-09-17T04:03:42.410029+0000","last_undegraded":"2024-09-17T04:03:42.410029+0000","last_fullsized":"2024-09-17T04:03:42.410029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:56.328537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521658+0000","last_change":"2024-09-17T04:03:39.335709+0000","last_active":"2024-09-17T04:03:43.521658+0000","last_peered":"2024-09-17T04:03:43.521658+0000","last_clean":"2024-09-17T04:03:43.521658+0000","last_became_active":"2024-09-17T04:03:39.334739+0000","last_became_peered":"2024-09-17T04:03:39.334739+0000","last_unstale":"2024-09-17T04:03:43.521658+0000","last_undegraded":"2024-09-17T04:03:43.521658+0000","last_fullsized":"2024-09-17T04:03:43.521658+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741466+0000","last_change":"2024-09-17T04:03:40.335288+0000","last_active":"2024-09-17T04:03:42.741466+0000","last_peered":"2024-09-17T04:03:42.741466+0000","last_clean":"2024-09-17T04:03:42.741466+0000","last_became_active":"2024-09-17T04:03:40.334907+0000","last_became_peered":"2024-09-17T04:03:40.334907+0000","last_unstale":"2024-09-17T04:03:42.741466+0000","last_undegraded":"2024-09-17T04:03:42.741466+0000","last_fullsized":"2024-09-17T04:03:42.741466+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:15.982851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482130+0000","last_change":"2024-09-17T04:03:39.331744+0000","last_active":"2024-09-17T04:03:41.482130+0000","last_peered":"2024-09-17T04:03:41.482130+0000","last_clean":"2024-09-17T04:03:41.482130+0000","last_became_active":"2024-09-17T04:03:39.330827+0000","last_became_peered":"2024-09-17T04:03:39.330827+0000","last_unstale":"2024-09-17T04:03:41.482130+0000","last_undegraded":"2024-09-17T04:03:41.482130+0000","last_fullsized":"2024-09-17T04:03:41.482130+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:57.121963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482922+0000","last_change":"2024-09-17T04:03:40.334111+0000","last_active":"2024-09-17T04:03:41.482922+0000","last_peered":"2024-09-17T04:03:41.482922+0000","last_clean":"2024-09-17T04:03:41.482922+0000","last_became_active":"2024-09-17T04:03:40.333244+0000","last_became_peered":"2024-09-17T04:03:40.333244+0000","last_unstale":"2024-09-17T04:03:41.482922+0000","last_undegraded":"2024-09-17T04:03:41.482922+0000","last_fullsized":"2024-09-17T04:03:41.482922+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:20.936128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332507+0000","last_change":"2024-09-17T04:03:39.327400+0000","last_active":"2024-09-17T04:03:39.332507+0000","last_peered":"2024-09-17T04:03:39.332507+0000","last_clean":"2024-09-17T04:03:39.332507+0000","last_became_active":"2024-09-17T04:03:39.327045+0000","last_became_peered":"2024-09-17T04:03:39.327045+0000","last_unstale":"2024-09-17T04:03:39.332507+0000","last_undegraded":"2024-09-17T04:03:39.332507+0000","last_fullsized":"2024-09-17T04:03:39.332507+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:04.062871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315550+0000","last_change":"2024-09-17T04:03:39.315550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315550+0000","last_undegraded":"2024-09-17T04:03:39.315550+0000","last_fullsized":"2024-09-17T04:03:39.315550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"203'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333769+0000","last_change":"2024-09-17T04:03:39.332034+0000","last_active":"2024-09-17T04:03:41.333769+0000","last_peered":"2024-09-17T04:03:41.333769+0000","last_clean":"2024-09-17T04:03:41.333769+0000","last_became_active":"2024-09-17T04:03:39.331723+0000","last_became_peered":"2024-09-17T04:03:39.331723+0000","last_unstale":"2024-09-17T04:03:41.333769+0000","last_undegraded":"2024-09-17T04:03:41.333769+0000","last_fullsized":"2024-09-17T04:03:41.333769+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:01:32.008051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410209+0000","last_change":"2024-09-17T04:03:40.343426+0000","last_active":"2024-09-17T04:03:42.410209+0000","last_peered":"2024-09-17T04:03:42.410209+0000","last_clean":"2024-09-17T04:03:42.410209+0000","last_became_active":"2024-09-17T04:03:40.343092+0000","last_became_peered":"2024-09-17T04:03:40.343092+0000","last_unstale":"2024-09-17T04:03:42.410209+0000","last_undegraded":"2024-09-17T04:03:42.410209+0000","last_fullsized":"2024-09-17T04:03:42.410209+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:19.250495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521877+0000","last_change":"2024-09-17T04:03:39.335065+0000","last_active":"2024-09-17T04:03:43.521877+0000","last_peered":"2024-09-17T04:03:43.521877+0000","last_clean":"2024-09-17T04:03:43.521877+0000","last_became_active":"2024-09-17T04:03:39.334604+0000","last_became_peered":"2024-09-17T04:03:39.334604+0000","last_unstale":"2024-09-17T04:03:43.521877+0000","last_undegraded":"2024-09-17T04:03:43.521877+0000","last_fullsized":"2024-09-17T04:03:43.521877+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410698+0000","last_change":"2024-09-17T04:03:40.352477+0000","last_active":"2024-09-17T04:03:42.410698+0000","last_peered":"2024-09-17T04:03:42.410698+0000","last_clean":"2024-09-17T04:03:42.410698+0000","last_became_active":"2024-09-17T04:03:40.351366+0000","last_became_peered":"2024-09-17T04:03:40.351366+0000","last_unstale":"2024-09-17T04:03:42.410698+0000","last_undegraded":"2024-09-17T04:03:42.410698+0000","last_fullsized":"2024-09-17T04:03:42.410698+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:17:14.000165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410341+0000","last_change":"2024-09-17T04:03:39.335924+0000","last_active":"2024-09-17T04:03:42.410341+0000","last_peered":"2024-09-17T04:03:42.410341+0000","last_clean":"2024-09-17T04:03:42.410341+0000","last_became_active":"2024-09-17T04:03:39.335629+0000","last_became_peered":"2024-09-17T04:03:39.335629+0000","last_unstale":"2024-09-17T04:03:42.410341+0000","last_undegraded":"2024-09-17T04:03:42.410341+0000","last_fullsized":"2024-09-17T04:03:42.410341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:52.490212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482687+0000","last_change":"2024-09-17T04:03:40.339516+0000","last_active":"2024-09-17T04:03:41.482687+0000","last_peered":"2024-09-17T04:03:41.482687+0000","last_clean":"2024-09-17T04:03:41.482687+0000","last_became_active":"2024-09-17T04:03:40.339040+0000","last_became_peered":"2024-09-17T04:03:40.339040+0000","last_unstale":"2024-09-17T04:03:41.482687+0000","last_undegraded":"2024-09-17T04:03:41.482687+0000","last_fullsized":"2024-09-17T04:03:41.482687+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:33.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482477+0000","last_change":"2024-09-17T04:03:39.331355+0000","last_active":"2024-09-17T04:03:41.482477+0000","last_peered":"2024-09-17T04:03:41.482477+0000","last_clean":"2024-09-17T04:03:41.482477+0000","last_became_active":"2024-09-17T04:03:39.331003+0000","last_became_peered":"2024-09-17T04:03:39.331003+0000","last_unstale":"2024-09-17T04:03:41.482477+0000","last_undegraded":"2024-09-17T04:03:41.482477+0000","last_fullsized":"2024-09-17T04:03:41.482477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:52.147204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741734+0000","last_change":"2024-09-17T04:03:40.337242+0000","last_active":"2024-09-17T04:03:42.741734+0000","last_peered":"2024-09-17T04:03:42.741734+0000","last_clean":"2024-09-17T04:03:42.741734+0000","last_became_active":"2024-09-17T04:03:40.336866+0000","last_became_peered":"2024-09-17T04:03:40.336866+0000","last_unstale":"2024-09-17T04:03:42.741734+0000","last_undegraded":"2024-09-17T04:03:42.741734+0000","last_fullsized":"2024-09-17T04:03:42.741734+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:52.701613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.336329+0000","last_change":"2024-09-17T04:03:39.332196+0000","last_active":"2024-09-17T04:03:39.336329+0000","last_peered":"2024-09-17T04:03:39.336329+0000","last_clean":"2024-09-17T04:03:39.336329+0000","last_became_active":"2024-09-17T04:03:39.331897+0000","last_became_peered":"2024-09-17T04:03:39.331897+0000","last_unstale":"2024-09-17T04:03:39.336329+0000","last_undegraded":"2024-09-17T04:03:39.336329+0000","last_fullsized":"2024-09-17T04:03:39.336329+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:10.918846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741362+0000","last_change":"2024-09-17T04:03:40.331229+0000","last_active":"2024-09-17T04:03:42.741362+0000","last_peered":"2024-09-17T04:03:42.741362+0000","last_clean":"2024-09-17T04:03:42.741362+0000","last_became_active":"2024-09-17T04:03:40.330857+0000","last_became_peered":"2024-09-17T04:03:40.330857+0000","last_unstale":"2024-09-17T04:03:42.741362+0000","last_undegraded":"2024-09-17T04:03:42.741362+0000","last_fullsized":"2024-09-17T04:03:42.741362+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:18.666966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334106+0000","last_change":"2024-09-17T04:03:39.329983+0000","last_active":"2024-09-17T04:03:39.334106+0000","last_peered":"2024-09-17T04:03:39.334106+0000","last_clean":"2024-09-17T04:03:39.334106+0000","last_became_active":"2024-09-17T04:03:39.325670+0000","last_became_peered":"2024-09-17T04:03:39.325670+0000","last_unstale":"2024-09-17T04:03:39.334106+0000","last_undegraded":"2024-09-17T04:03:39.334106+0000","last_fullsized":"2024-09-17T04:03:39.334106+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:20.865772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314550+0000","last_change":"2024-09-17T04:03:39.314550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314550+0000","last_undegraded":"2024-09-17T04:03:39.314550+0000","last_fullsized":"2024-09-17T04:03:39.314550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334879+0000","last_change":"2024-09-17T04:03:39.332559+0000","last_active":"2024-09-17T04:03:39.334879+0000","last_peered":"2024-09-17T04:03:39.334879+0000","last_clean":"2024-09-17T04:03:39.334879+0000","last_became_active":"2024-09-17T04:03:39.332265+0000","last_became_peered":"2024-09-17T04:03:39.332265+0000","last_unstale":"2024-09-17T04:03:39.334879+0000","last_undegraded":"2024-09-17T04:03:39.334879+0000","last_fullsized":"2024-09-17T04:03:39.334879+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:07.087568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333644+0000","last_change":"2024-09-17T04:03:40.352076+0000","last_active":"2024-09-17T04:03:41.333644+0000","last_peered":"2024-09-17T04:03:41.333644+0000","last_clean":"2024-09-17T04:03:41.333644+0000","last_became_active":"2024-09-17T04:03:40.351735+0000","last_became_peered":"2024-09-17T04:03:40.351735+0000","last_unstale":"2024-09-17T04:03:41.333644+0000","last_undegraded":"2024-09-17T04:03:41.333644+0000","last_fullsized":"2024-09-17T04:03:41.333644+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:16.809754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334891+0000","last_change":"2024-09-17T04:03:39.331554+0000","last_active":"2024-09-17T04:03:39.334891+0000","last_peered":"2024-09-17T04:03:39.334891+0000","last_clean":"2024-09-17T04:03:39.334891+0000","last_became_active":"2024-09-17T04:03:39.331212+0000","last_became_peered":"2024-09-17T04:03:39.331212+0000","last_unstale":"2024-09-17T04:03:39.334891+0000","last_undegraded":"2024-09-17T04:03:39.334891+0000","last_fullsized":"2024-09-17T04:03:39.334891+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:04.328735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":446,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521595+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:03:43.521595+0000","last_peered":"2024-09-17T04:03:43.521595+0000","last_clean":"2024-09-17T04:03:43.521595+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:03:43.521595+0000","last_undegraded":"2024-09-17T04:03:43.521595+0000","last_fullsized":"2024-09-17T04:03:43.521595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52320,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319520,"statfs":{"total":96636764160,"available":96583188480,"internally_reserved":0,"allocated":3158016,"data_stored":1716897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29528,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342312,"statfs":{"total":96636764160,"available":96606527488,"internally_reserved":0,"allocated":2560000,"data_stored":1126459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509616,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51716,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320124,"statfs":{"total":96636764160,"available":96583806976,"internally_reserved":0,"allocated":2535424,"data_stored":1113384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509615,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542318,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1127063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542319,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3158016,"data_stored":1716861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:03:45.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:03:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 -- 172.21.15.129:0/364168639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c80005210 msgr2=0x7f6c8000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 --2- 172.21.15.129:0/364168639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c80005210 0x7f6c8000b9c0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f6c9806eba0 tx=0x7f6c80005830 comp rx=0 tx=0).stop 2024-09-17T04:03:45.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6c9e800640 1 -- 172.21.15.129:0/364168639 reap_dead start 2024-09-17T04:03:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 -- 172.21.15.129:0/364168639 shutdown_connections 2024-09-17T04:03:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 -- 172.21.15.129:0/364168639 >> 172.21.15.129:0/364168639 conn(0x7f6c9806d410 msgr2=0x7f6c980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 -- 172.21.15.129:0/364168639 shutdown_connections 2024-09-17T04:03:45.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.499+0000 7f6ca0800640 1 -- 172.21.15.129:0/364168639 wait complete. 2024-09-17T04:03:45.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.501+0000 7f6ca0800640 1 Processor -- start 2024-09-17T04:03:45.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.501+0000 7f6ca0800640 1 -- start start 2024-09-17T04:03:45.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 0x7f6c981572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c980adc30 0x7f6c98073a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c980cd390 con 0x7f6c980c3d80 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c98070d60 con 0x7f6c80005210 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6ca0800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c980cd510 con 0x7f6c980adc30 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46094/0 (socket says 172.21.15.129:46094) 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 -- 172.21.15.129:0/57756858 learned_addr learned my addr 172.21.15.129:0/57756858 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 -- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c980adc30 msgr2=0x7f6c98073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c980adc30 0x7f6c98073a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 -- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 msgr2=0x7f6c981572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9de00640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 0x7f6c981572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 0x7f6c981572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9e800640 1 -- 172.21.15.129:0/57756858 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c980c6df0 con 0x7f6c980c3d80 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.503+0000 7f6c9de00640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 0x7f6c981572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6c9e800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f6c9402b810 tx=0x7f6c94009330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94004b20 con 0x7f6c980c3d80 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c94044070 con 0x7f6c980c3d80 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9815cc80 con 0x7f6c980c3d80 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94033cd0 con 0x7f6c980c3d80 2024-09-17T04:03:45.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.504+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c980c65f0 con 0x7f6c980c3d80 2024-09-17T04:03:45.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.505+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9815ed40 con 0x7f6c980c3d80 2024-09-17T04:03:45.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.506+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c94049460 con 0x7f6c980c3d80 2024-09-17T04:03:45.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.507+0000 7f6c92000640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 0x7f6c7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.507+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f6c940bc2c0 con 0x7f6c980c3d80 2024-09-17T04:03:45.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.508+0000 7f6c9de00640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 0x7f6c7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.509+0000 7f6c9de00640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 0x7f6c7c07b0a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c001860 tx=0x7f6c8c000ed0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.512+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c9815ed40 con 0x7f6c980c3d80 2024-09-17T04:03:45.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.690+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6c9815ea80 con 0x7f6c7c078b90 2024-09-17T04:03:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.695+0000 7f6c92000640 1 -- 172.21.15.129:0/57756858 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250367 (secure 0 0 0) 0x7f6c9815ea80 con 0x7f6c7c078b90 2024-09-17T04:03:45.698 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:45.699 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:03:45.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 msgr2=0x7f6c7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 0x7f6c7c07b0a0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c001860 tx=0x7f6c8c000ed0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 msgr2=0x7f6c98073f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f6c9402b810 tx=0x7f6c94009330 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 shutdown_connections 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c7c078b90 0x7f6c7c07b0a0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c980c3d80 0x7f6c98073f60 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c980adc30 0x7f6c98073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 --2- 172.21.15.129:0/57756858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c80005210 0x7f6c981572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 >> 172.21.15.129:0/57756858 conn(0x7f6c9806d410 msgr2=0x7f6c98159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:45.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 shutdown_connections 2024-09-17T04:03:45.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.699+0000 7f6ca0800640 1 -- 172.21.15.129:0/57756858 wait complete. 2024-09-17T04:03:45.721 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-17T04:03:44.536475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590460,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":116,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":374740,"kb_used_data":21100,"kb_used_omap":12,"kb_used_meta":353459,"kb_avail":754599980,"statfs":{"total":773094113280,"available":772710379520,"internally_reserved":0,"allocated":21606400,"data_stored":10143045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":361942576},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001886"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315975+0000","last_change":"2024-09-17T04:03:39.315975+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315975+0000","last_undegraded":"2024-09-17T04:03:39.315975+0000","last_fullsized":"2024-09-17T04:03:39.315975+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314425+0000","last_change":"2024-09-17T04:03:39.314425+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314425+0000","last_undegraded":"2024-09-17T04:03:39.314425+0000","last_fullsized":"2024-09-17T04:03:39.314425+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313411+0000","last_change":"2024-09-17T04:03:39.313411+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313411+0000","last_undegraded":"2024-09-17T04:03:39.313411+0000","last_fullsized":"2024-09-17T04:03:39.313411+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318949+0000","last_change":"2024-09-17T04:03:39.318949+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318949+0000","last_undegraded":"2024-09-17T04:03:39.318949+0000","last_fullsized":"2024-09-17T04:03:39.318949+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333497+0000","last_change":"2024-09-17T04:03:40.335355+0000","last_active":"2024-09-17T04:03:41.333497+0000","last_peered":"2024-09-17T04:03:41.333497+0000","last_clean":"2024-09-17T04:03:41.333497+0000","last_became_active":"2024-09-17T04:03:40.334994+0000","last_became_peered":"2024-09-17T04:03:40.334994+0000","last_unstale":"2024-09-17T04:03:41.333497+0000","last_undegraded":"2024-09-17T04:03:41.333497+0000","last_fullsized":"2024-09-17T04:03:41.333497+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:02.228911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317948+0000","last_change":"2024-09-17T04:03:39.317948+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317948+0000","last_undegraded":"2024-09-17T04:03:39.317948+0000","last_fullsized":"2024-09-17T04:03:39.317948+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482089+0000","last_change":"2024-09-17T04:03:40.350322+0000","last_active":"2024-09-17T04:03:41.482089+0000","last_peered":"2024-09-17T04:03:41.482089+0000","last_clean":"2024-09-17T04:03:41.482089+0000","last_became_active":"2024-09-17T04:03:40.349953+0000","last_became_peered":"2024-09-17T04:03:40.349953+0000","last_unstale":"2024-09-17T04:03:41.482089+0000","last_undegraded":"2024-09-17T04:03:41.482089+0000","last_fullsized":"2024-09-17T04:03:41.482089+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:03:12.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741207+0000","last_change":"2024-09-17T04:03:40.334851+0000","last_active":"2024-09-17T04:03:42.741207+0000","last_peered":"2024-09-17T04:03:42.741207+0000","last_clean":"2024-09-17T04:03:42.741207+0000","last_became_active":"2024-09-17T04:03:40.331337+0000","last_became_peered":"2024-09-17T04:03:40.331337+0000","last_unstale":"2024-09-17T04:03:42.741207+0000","last_undegraded":"2024-09-17T04:03:42.741207+0000","last_fullsized":"2024-09-17T04:03:42.741207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:52.463737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521788+0000","last_change":"2024-09-17T04:03:40.349511+0000","last_active":"2024-09-17T04:03:43.521788+0000","last_peered":"2024-09-17T04:03:43.521788+0000","last_clean":"2024-09-17T04:03:43.521788+0000","last_became_active":"2024-09-17T04:03:40.349113+0000","last_became_peered":"2024-09-17T04:03:40.349113+0000","last_unstale":"2024-09-17T04:03:43.521788+0000","last_undegraded":"2024-09-17T04:03:43.521788+0000","last_fullsized":"2024-09-17T04:03:43.521788+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:50.920504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482695+0000","last_change":"2024-09-17T04:03:40.340480+0000","last_active":"2024-09-17T04:03:41.482695+0000","last_peered":"2024-09-17T04:03:41.482695+0000","last_clean":"2024-09-17T04:03:41.482695+0000","last_became_active":"2024-09-17T04:03:40.338521+0000","last_became_peered":"2024-09-17T04:03:40.338521+0000","last_unstale":"2024-09-17T04:03:41.482695+0000","last_undegraded":"2024-09-17T04:03:41.482695+0000","last_fullsized":"2024-09-17T04:03:41.482695+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:10.458690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410478+0000","last_change":"2024-09-17T04:03:40.348952+0000","last_active":"2024-09-17T04:03:42.410478+0000","last_peered":"2024-09-17T04:03:42.410478+0000","last_clean":"2024-09-17T04:03:42.410478+0000","last_became_active":"2024-09-17T04:03:40.348656+0000","last_became_peered":"2024-09-17T04:03:40.348656+0000","last_unstale":"2024-09-17T04:03:42.410478+0000","last_undegraded":"2024-09-17T04:03:42.410478+0000","last_fullsized":"2024-09-17T04:03:42.410478+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:47.744159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741493+0000","last_change":"2024-09-17T04:03:40.354695+0000","last_active":"2024-09-17T04:03:42.741493+0000","last_peered":"2024-09-17T04:03:42.741493+0000","last_clean":"2024-09-17T04:03:42.741493+0000","last_became_active":"2024-09-17T04:03:40.354274+0000","last_became_peered":"2024-09-17T04:03:40.354274+0000","last_unstale":"2024-09-17T04:03:42.741493+0000","last_undegraded":"2024-09-17T04:03:42.741493+0000","last_fullsized":"2024-09-17T04:03:42.741493+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:15.701377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482731+0000","last_change":"2024-09-17T04:03:40.350373+0000","last_active":"2024-09-17T04:03:41.482731+0000","last_peered":"2024-09-17T04:03:41.482731+0000","last_clean":"2024-09-17T04:03:41.482731+0000","last_became_active":"2024-09-17T04:03:40.350000+0000","last_became_peered":"2024-09-17T04:03:40.350000+0000","last_unstale":"2024-09-17T04:03:41.482731+0000","last_undegraded":"2024-09-17T04:03:41.482731+0000","last_fullsized":"2024-09-17T04:03:41.482731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:29.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312559+0000","last_change":"2024-09-17T04:03:39.312559+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312559+0000","last_undegraded":"2024-09-17T04:03:39.312559+0000","last_fullsized":"2024-09-17T04:03:39.312559+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521536+0000","last_change":"2024-09-17T04:03:40.344294+0000","last_active":"2024-09-17T04:03:43.521536+0000","last_peered":"2024-09-17T04:03:43.521536+0000","last_clean":"2024-09-17T04:03:43.521536+0000","last_became_active":"2024-09-17T04:03:40.343923+0000","last_became_peered":"2024-09-17T04:03:40.343923+0000","last_unstale":"2024-09-17T04:03:43.521536+0000","last_undegraded":"2024-09-17T04:03:43.521536+0000","last_fullsized":"2024-09-17T04:03:43.521536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:35.362812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313125+0000","last_change":"2024-09-17T04:03:39.313125+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313125+0000","last_undegraded":"2024-09-17T04:03:39.313125+0000","last_fullsized":"2024-09-17T04:03:39.313125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741536+0000","last_change":"2024-09-17T04:03:40.343695+0000","last_active":"2024-09-17T04:03:42.741536+0000","last_peered":"2024-09-17T04:03:42.741536+0000","last_clean":"2024-09-17T04:03:42.741536+0000","last_became_active":"2024-09-17T04:03:40.343413+0000","last_became_peered":"2024-09-17T04:03:40.343413+0000","last_unstale":"2024-09-17T04:03:42.741536+0000","last_undegraded":"2024-09-17T04:03:42.741536+0000","last_fullsized":"2024-09-17T04:03:42.741536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:54.092079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313567+0000","last_change":"2024-09-17T04:03:39.313567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313567+0000","last_undegraded":"2024-09-17T04:03:39.313567+0000","last_fullsized":"2024-09-17T04:03:39.313567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:40.351995+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:40.351649+0000","last_became_peered":"2024-09-17T04:03:40.351649+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:03.942253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313824+0000","last_change":"2024-09-17T04:03:39.313824+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313824+0000","last_undegraded":"2024-09-17T04:03:39.313824+0000","last_fullsized":"2024-09-17T04:03:39.313824+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741813+0000","last_change":"2024-09-17T04:03:40.333277+0000","last_active":"2024-09-17T04:03:42.741813+0000","last_peered":"2024-09-17T04:03:42.741813+0000","last_clean":"2024-09-17T04:03:42.741813+0000","last_became_active":"2024-09-17T04:03:40.332882+0000","last_became_peered":"2024-09-17T04:03:40.332882+0000","last_unstale":"2024-09-17T04:03:42.741813+0000","last_undegraded":"2024-09-17T04:03:42.741813+0000","last_fullsized":"2024-09-17T04:03:42.741813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:44.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482369+0000","last_change":"2024-09-17T04:03:40.341616+0000","last_active":"2024-09-17T04:03:41.482369+0000","last_peered":"2024-09-17T04:03:41.482369+0000","last_clean":"2024-09-17T04:03:41.482369+0000","last_became_active":"2024-09-17T04:03:40.341269+0000","last_became_peered":"2024-09-17T04:03:40.341269+0000","last_unstale":"2024-09-17T04:03:41.482369+0000","last_undegraded":"2024-09-17T04:03:41.482369+0000","last_fullsized":"2024-09-17T04:03:41.482369+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:28.676331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741352+0000","last_change":"2024-09-17T04:03:40.337777+0000","last_active":"2024-09-17T04:03:42.741352+0000","last_peered":"2024-09-17T04:03:42.741352+0000","last_clean":"2024-09-17T04:03:42.741352+0000","last_became_active":"2024-09-17T04:03:40.335850+0000","last_became_peered":"2024-09-17T04:03:40.335850+0000","last_unstale":"2024-09-17T04:03:42.741352+0000","last_undegraded":"2024-09-17T04:03:42.741352+0000","last_fullsized":"2024-09-17T04:03:42.741352+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:37.334408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316858+0000","last_change":"2024-09-17T04:03:39.316858+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316858+0000","last_undegraded":"2024-09-17T04:03:39.316858+0000","last_fullsized":"2024-09-17T04:03:39.316858+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.324689+0000","last_change":"2024-09-17T04:03:39.324689+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.324689+0000","last_undegraded":"2024-09-17T04:03:39.324689+0000","last_fullsized":"2024-09-17T04:03:39.324689+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482816+0000","last_change":"2024-09-17T04:03:40.350740+0000","last_active":"2024-09-17T04:03:41.482816+0000","last_peered":"2024-09-17T04:03:41.482816+0000","last_clean":"2024-09-17T04:03:41.482816+0000","last_became_active":"2024-09-17T04:03:40.350415+0000","last_became_peered":"2024-09-17T04:03:40.350415+0000","last_unstale":"2024-09-17T04:03:41.482816+0000","last_undegraded":"2024-09-17T04:03:41.482816+0000","last_fullsized":"2024-09-17T04:03:41.482816+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:04.588026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410543+0000","last_change":"2024-09-17T04:03:40.347688+0000","last_active":"2024-09-17T04:03:42.410543+0000","last_peered":"2024-09-17T04:03:42.410543+0000","last_clean":"2024-09-17T04:03:42.410543+0000","last_became_active":"2024-09-17T04:03:40.347381+0000","last_became_peered":"2024-09-17T04:03:40.347381+0000","last_unstale":"2024-09-17T04:03:42.410543+0000","last_undegraded":"2024-09-17T04:03:42.410543+0000","last_fullsized":"2024-09-17T04:03:42.410543+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:51.255876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312330+0000","last_change":"2024-09-17T04:03:39.312330+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312330+0000","last_undegraded":"2024-09-17T04:03:39.312330+0000","last_fullsized":"2024-09-17T04:03:39.312330+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410325+0000","last_change":"2024-09-17T04:03:40.350667+0000","last_active":"2024-09-17T04:03:42.410325+0000","last_peered":"2024-09-17T04:03:42.410325+0000","last_clean":"2024-09-17T04:03:42.410325+0000","last_became_active":"2024-09-17T04:03:40.350312+0000","last_became_peered":"2024-09-17T04:03:40.350312+0000","last_unstale":"2024-09-17T04:03:42.410325+0000","last_undegraded":"2024-09-17T04:03:42.410325+0000","last_fullsized":"2024-09-17T04:03:42.410325+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:21.193199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333504+0000","last_change":"2024-09-17T04:03:40.331612+0000","last_active":"2024-09-17T04:03:41.333504+0000","last_peered":"2024-09-17T04:03:41.333504+0000","last_clean":"2024-09-17T04:03:41.333504+0000","last_became_active":"2024-09-17T04:03:40.329978+0000","last_became_peered":"2024-09-17T04:03:40.329978+0000","last_unstale":"2024-09-17T04:03:41.333504+0000","last_undegraded":"2024-09-17T04:03:41.333504+0000","last_fullsized":"2024-09-17T04:03:41.333504+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:48.029240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.321401+0000","last_change":"2024-09-17T04:03:39.321401+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.321401+0000","last_undegraded":"2024-09-17T04:03:39.321401+0000","last_fullsized":"2024-09-17T04:03:39.321401+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.316435+0000","last_change":"2024-09-17T04:03:39.316435+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.316435+0000","last_undegraded":"2024-09-17T04:03:39.316435+0000","last_fullsized":"2024-09-17T04:03:39.316435+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312535+0000","last_change":"2024-09-17T04:03:39.312535+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312535+0000","last_undegraded":"2024-09-17T04:03:39.312535+0000","last_fullsized":"2024-09-17T04:03:39.312535+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482954+0000","last_change":"2024-09-17T04:03:39.330078+0000","last_active":"2024-09-17T04:03:41.482954+0000","last_peered":"2024-09-17T04:03:41.482954+0000","last_clean":"2024-09-17T04:03:41.482954+0000","last_became_active":"2024-09-17T04:03:39.329766+0000","last_became_peered":"2024-09-17T04:03:39.329766+0000","last_unstale":"2024-09-17T04:03:41.482954+0000","last_undegraded":"2024-09-17T04:03:41.482954+0000","last_fullsized":"2024-09-17T04:03:41.482954+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:11.195328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410186+0000","last_change":"2024-09-17T04:03:40.351683+0000","last_active":"2024-09-17T04:03:42.410186+0000","last_peered":"2024-09-17T04:03:42.410186+0000","last_clean":"2024-09-17T04:03:42.410186+0000","last_became_active":"2024-09-17T04:03:40.351288+0000","last_became_peered":"2024-09-17T04:03:40.351288+0000","last_unstale":"2024-09-17T04:03:42.410186+0000","last_undegraded":"2024-09-17T04:03:42.410186+0000","last_fullsized":"2024-09-17T04:03:42.410186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:11:57.454665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326983+0000","last_change":"2024-09-17T04:03:39.322960+0000","last_active":"2024-09-17T04:03:39.326983+0000","last_peered":"2024-09-17T04:03:39.326983+0000","last_clean":"2024-09-17T04:03:39.326983+0000","last_became_active":"2024-09-17T04:03:39.322704+0000","last_became_peered":"2024-09-17T04:03:39.322704+0000","last_unstale":"2024-09-17T04:03:39.326983+0000","last_undegraded":"2024-09-17T04:03:39.326983+0000","last_fullsized":"2024-09-17T04:03:39.326983+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741839+0000","last_change":"2024-09-17T04:03:40.343840+0000","last_active":"2024-09-17T04:03:42.741839+0000","last_peered":"2024-09-17T04:03:42.741839+0000","last_clean":"2024-09-17T04:03:42.741839+0000","last_became_active":"2024-09-17T04:03:40.343539+0000","last_became_peered":"2024-09-17T04:03:40.343539+0000","last_unstale":"2024-09-17T04:03:42.741839+0000","last_undegraded":"2024-09-17T04:03:42.741839+0000","last_fullsized":"2024-09-17T04:03:42.741839+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:04.501121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.325189+0000","last_change":"2024-09-17T04:03:39.320734+0000","last_active":"2024-09-17T04:03:39.325189+0000","last_peered":"2024-09-17T04:03:39.325189+0000","last_clean":"2024-09-17T04:03:39.325189+0000","last_became_active":"2024-09-17T04:03:39.320431+0000","last_became_peered":"2024-09-17T04:03:39.320431+0000","last_unstale":"2024-09-17T04:03:39.325189+0000","last_undegraded":"2024-09-17T04:03:39.325189+0000","last_fullsized":"2024-09-17T04:03:39.325189+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:51.187815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410125+0000","last_change":"2024-09-17T04:03:40.335363+0000","last_active":"2024-09-17T04:03:42.410125+0000","last_peered":"2024-09-17T04:03:42.410125+0000","last_clean":"2024-09-17T04:03:42.410125+0000","last_became_active":"2024-09-17T04:03:40.333333+0000","last_became_peered":"2024-09-17T04:03:40.333333+0000","last_unstale":"2024-09-17T04:03:42.410125+0000","last_undegraded":"2024-09-17T04:03:42.410125+0000","last_fullsized":"2024-09-17T04:03:42.410125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:35.331891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334393+0000","last_change":"2024-09-17T04:03:39.329538+0000","last_active":"2024-09-17T04:03:39.334393+0000","last_peered":"2024-09-17T04:03:39.334393+0000","last_clean":"2024-09-17T04:03:39.334393+0000","last_became_active":"2024-09-17T04:03:39.329257+0000","last_became_peered":"2024-09-17T04:03:39.329257+0000","last_unstale":"2024-09-17T04:03:39.334393+0000","last_undegraded":"2024-09-17T04:03:39.334393+0000","last_fullsized":"2024-09-17T04:03:39.334393+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:25.093614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521725+0000","last_change":"2024-09-17T04:03:40.344266+0000","last_active":"2024-09-17T04:03:43.521725+0000","last_peered":"2024-09-17T04:03:43.521725+0000","last_clean":"2024-09-17T04:03:43.521725+0000","last_became_active":"2024-09-17T04:03:40.343914+0000","last_became_peered":"2024-09-17T04:03:40.343914+0000","last_unstale":"2024-09-17T04:03:43.521725+0000","last_undegraded":"2024-09-17T04:03:43.521725+0000","last_fullsized":"2024-09-17T04:03:43.521725+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:49.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326649+0000","last_change":"2024-09-17T04:03:39.321662+0000","last_active":"2024-09-17T04:03:39.326649+0000","last_peered":"2024-09-17T04:03:39.326649+0000","last_clean":"2024-09-17T04:03:39.326649+0000","last_became_active":"2024-09-17T04:03:39.321278+0000","last_became_peered":"2024-09-17T04:03:39.321278+0000","last_unstale":"2024-09-17T04:03:39.326649+0000","last_undegraded":"2024-09-17T04:03:39.326649+0000","last_fullsized":"2024-09-17T04:03:39.326649+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.772774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312542+0000","last_change":"2024-09-17T04:03:39.312542+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312542+0000","last_undegraded":"2024-09-17T04:03:39.312542+0000","last_fullsized":"2024-09-17T04:03:39.312542+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482232+0000","last_change":"2024-09-17T04:03:39.332398+0000","last_active":"2024-09-17T04:03:41.482232+0000","last_peered":"2024-09-17T04:03:41.482232+0000","last_clean":"2024-09-17T04:03:41.482232+0000","last_became_active":"2024-09-17T04:03:39.331945+0000","last_became_peered":"2024-09-17T04:03:39.331945+0000","last_unstale":"2024-09-17T04:03:41.482232+0000","last_undegraded":"2024-09-17T04:03:41.482232+0000","last_fullsized":"2024-09-17T04:03:41.482232+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.951852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314983+0000","last_change":"2024-09-17T04:03:39.314983+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314983+0000","last_undegraded":"2024-09-17T04:03:39.314983+0000","last_fullsized":"2024-09-17T04:03:39.314983+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.324526+0000","last_change":"2024-09-17T04:03:39.317105+0000","last_active":"2024-09-17T04:03:39.324526+0000","last_peered":"2024-09-17T04:03:39.324526+0000","last_clean":"2024-09-17T04:03:39.324526+0000","last_became_active":"2024-09-17T04:03:39.316779+0000","last_became_peered":"2024-09-17T04:03:39.316779+0000","last_unstale":"2024-09-17T04:03:39.324526+0000","last_undegraded":"2024-09-17T04:03:39.324526+0000","last_fullsized":"2024-09-17T04:03:39.324526+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:21.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.313113+0000","last_change":"2024-09-17T04:03:39.313113+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.313113+0000","last_undegraded":"2024-09-17T04:03:39.313113+0000","last_fullsized":"2024-09-17T04:03:39.313113+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:39.326124+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:39.325797+0000","last_became_peered":"2024-09-17T04:03:39.325797+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:04.499200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410736+0000","last_change":"2024-09-17T04:03:40.351885+0000","last_active":"2024-09-17T04:03:42.410736+0000","last_peered":"2024-09-17T04:03:42.410736+0000","last_clean":"2024-09-17T04:03:42.410736+0000","last_became_active":"2024-09-17T04:03:40.351494+0000","last_became_peered":"2024-09-17T04:03:40.351494+0000","last_unstale":"2024-09-17T04:03:42.410736+0000","last_undegraded":"2024-09-17T04:03:42.410736+0000","last_fullsized":"2024-09-17T04:03:42.410736+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:13.184588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333987+0000","last_change":"2024-09-17T04:03:39.328133+0000","last_active":"2024-09-17T04:03:39.333987+0000","last_peered":"2024-09-17T04:03:39.333987+0000","last_clean":"2024-09-17T04:03:39.333987+0000","last_became_active":"2024-09-17T04:03:39.327863+0000","last_became_peered":"2024-09-17T04:03:39.327863+0000","last_unstale":"2024-09-17T04:03:39.333987+0000","last_undegraded":"2024-09-17T04:03:39.333987+0000","last_fullsized":"2024-09-17T04:03:39.333987+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:56:55.809447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311933+0000","last_change":"2024-09-17T04:03:39.311933+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311933+0000","last_undegraded":"2024-09-17T04:03:39.311933+0000","last_fullsized":"2024-09-17T04:03:39.311933+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332935+0000","last_change":"2024-09-17T04:03:39.331097+0000","last_active":"2024-09-17T04:03:39.332935+0000","last_peered":"2024-09-17T04:03:39.332935+0000","last_clean":"2024-09-17T04:03:39.332935+0000","last_became_active":"2024-09-17T04:03:39.330131+0000","last_became_peered":"2024-09-17T04:03:39.330131+0000","last_unstale":"2024-09-17T04:03:39.332935+0000","last_undegraded":"2024-09-17T04:03:39.332935+0000","last_fullsized":"2024-09-17T04:03:39.332935+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:57:40.153994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482540+0000","last_change":"2024-09-17T04:03:40.332717+0000","last_active":"2024-09-17T04:03:41.482540+0000","last_peered":"2024-09-17T04:03:41.482540+0000","last_clean":"2024-09-17T04:03:41.482540+0000","last_became_active":"2024-09-17T04:03:40.332421+0000","last_became_peered":"2024-09-17T04:03:40.332421+0000","last_unstale":"2024-09-17T04:03:41.482540+0000","last_undegraded":"2024-09-17T04:03:41.482540+0000","last_fullsized":"2024-09-17T04:03:41.482540+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:42.530742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332906+0000","last_change":"2024-09-17T04:03:39.329612+0000","last_active":"2024-09-17T04:03:39.332906+0000","last_peered":"2024-09-17T04:03:39.332906+0000","last_clean":"2024-09-17T04:03:39.332906+0000","last_became_active":"2024-09-17T04:03:39.329342+0000","last_became_peered":"2024-09-17T04:03:39.329342+0000","last_unstale":"2024-09-17T04:03:39.332906+0000","last_undegraded":"2024-09-17T04:03:39.332906+0000","last_fullsized":"2024-09-17T04:03:39.332906+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:14.980173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482365+0000","last_change":"2024-09-17T04:03:40.338842+0000","last_active":"2024-09-17T04:03:41.482365+0000","last_peered":"2024-09-17T04:03:41.482365+0000","last_clean":"2024-09-17T04:03:41.482365+0000","last_became_active":"2024-09-17T04:03:40.338449+0000","last_became_peered":"2024-09-17T04:03:40.338449+0000","last_unstale":"2024-09-17T04:03:41.482365+0000","last_undegraded":"2024-09-17T04:03:41.482365+0000","last_fullsized":"2024-09-17T04:03:41.482365+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:44.963272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.328955+0000","last_change":"2024-09-17T04:03:39.324560+0000","last_active":"2024-09-17T04:03:39.328955+0000","last_peered":"2024-09-17T04:03:39.328955+0000","last_clean":"2024-09-17T04:03:39.328955+0000","last_became_active":"2024-09-17T04:03:39.324206+0000","last_became_peered":"2024-09-17T04:03:39.324206+0000","last_unstale":"2024-09-17T04:03:39.328955+0000","last_undegraded":"2024-09-17T04:03:39.328955+0000","last_fullsized":"2024-09-17T04:03:39.328955+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:40.444284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521913+0000","last_change":"2024-09-17T04:03:40.344849+0000","last_active":"2024-09-17T04:03:43.521913+0000","last_peered":"2024-09-17T04:03:43.521913+0000","last_clean":"2024-09-17T04:03:43.521913+0000","last_became_active":"2024-09-17T04:03:40.344173+0000","last_became_peered":"2024-09-17T04:03:40.344173+0000","last_unstale":"2024-09-17T04:03:43.521913+0000","last_undegraded":"2024-09-17T04:03:43.521913+0000","last_fullsized":"2024-09-17T04:03:43.521913+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:12.072676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.333711+0000","last_change":"2024-09-17T04:03:39.327456+0000","last_active":"2024-09-17T04:03:39.333711+0000","last_peered":"2024-09-17T04:03:39.333711+0000","last_clean":"2024-09-17T04:03:39.333711+0000","last_became_active":"2024-09-17T04:03:39.326151+0000","last_became_peered":"2024-09-17T04:03:39.326151+0000","last_unstale":"2024-09-17T04:03:39.333711+0000","last_undegraded":"2024-09-17T04:03:39.333711+0000","last_fullsized":"2024-09-17T04:03:39.333711+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.317212+0000","last_change":"2024-09-17T04:03:39.317212+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.317212+0000","last_undegraded":"2024-09-17T04:03:39.317212+0000","last_fullsized":"2024-09-17T04:03:39.317212+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482297+0000","last_change":"2024-09-17T04:03:39.330613+0000","last_active":"2024-09-17T04:03:41.482297+0000","last_peered":"2024-09-17T04:03:41.482297+0000","last_clean":"2024-09-17T04:03:41.482297+0000","last_became_active":"2024-09-17T04:03:39.330283+0000","last_became_peered":"2024-09-17T04:03:39.330283+0000","last_unstale":"2024-09-17T04:03:41.482297+0000","last_undegraded":"2024-09-17T04:03:41.482297+0000","last_fullsized":"2024-09-17T04:03:41.482297+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:46.502036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482092+0000","last_change":"2024-09-17T04:03:40.341659+0000","last_active":"2024-09-17T04:03:41.482092+0000","last_peered":"2024-09-17T04:03:41.482092+0000","last_clean":"2024-09-17T04:03:41.482092+0000","last_became_active":"2024-09-17T04:03:40.341344+0000","last_became_peered":"2024-09-17T04:03:40.341344+0000","last_unstale":"2024-09-17T04:03:41.482092+0000","last_undegraded":"2024-09-17T04:03:41.482092+0000","last_fullsized":"2024-09-17T04:03:41.482092+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:24.293693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"202'8","reported_seq":23,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.409812+0000","last_change":"2024-09-17T04:03:39.332639+0000","last_active":"2024-09-17T04:03:42.409812+0000","last_peered":"2024-09-17T04:03:42.409812+0000","last_clean":"2024-09-17T04:03:42.409812+0000","last_became_active":"2024-09-17T04:03:39.332409+0000","last_became_peered":"2024-09-17T04:03:39.332409+0000","last_unstale":"2024-09-17T04:03:42.409812+0000","last_undegraded":"2024-09-17T04:03:42.409812+0000","last_fullsized":"2024-09-17T04:03:42.409812+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:03.585348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.318088+0000","last_change":"2024-09-17T04:03:39.318088+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.318088+0000","last_undegraded":"2024-09-17T04:03:39.318088+0000","last_fullsized":"2024-09-17T04:03:39.318088+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482664+0000","last_change":"2024-09-17T04:03:39.333660+0000","last_active":"2024-09-17T04:03:41.482664+0000","last_peered":"2024-09-17T04:03:41.482664+0000","last_clean":"2024-09-17T04:03:41.482664+0000","last_became_active":"2024-09-17T04:03:39.333330+0000","last_became_peered":"2024-09-17T04:03:39.333330+0000","last_unstale":"2024-09-17T04:03:41.482664+0000","last_undegraded":"2024-09-17T04:03:41.482664+0000","last_fullsized":"2024-09-17T04:03:41.482664+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:03.955054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.311836+0000","last_change":"2024-09-17T04:03:39.311836+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.311836+0000","last_undegraded":"2024-09-17T04:03:39.311836+0000","last_fullsized":"2024-09-17T04:03:39.311836+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.326510+0000","last_change":"2024-09-17T04:03:39.322064+0000","last_active":"2024-09-17T04:03:39.326510+0000","last_peered":"2024-09-17T04:03:39.326510+0000","last_clean":"2024-09-17T04:03:39.326510+0000","last_became_active":"2024-09-17T04:03:39.321750+0000","last_became_peered":"2024-09-17T04:03:39.321750+0000","last_unstale":"2024-09-17T04:03:39.326510+0000","last_undegraded":"2024-09-17T04:03:39.326510+0000","last_fullsized":"2024-09-17T04:03:39.326510+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521646+0000","last_change":"2024-09-17T04:03:40.340280+0000","last_active":"2024-09-17T04:03:43.521646+0000","last_peered":"2024-09-17T04:03:43.521646+0000","last_clean":"2024-09-17T04:03:43.521646+0000","last_became_active":"2024-09-17T04:03:40.339904+0000","last_became_peered":"2024-09-17T04:03:40.339904+0000","last_unstale":"2024-09-17T04:03:43.521646+0000","last_undegraded":"2024-09-17T04:03:43.521646+0000","last_fullsized":"2024-09-17T04:03:43.521646+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:09.142973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521655+0000","last_change":"2024-09-17T04:03:39.335875+0000","last_active":"2024-09-17T04:03:43.521655+0000","last_peered":"2024-09-17T04:03:43.521655+0000","last_clean":"2024-09-17T04:03:43.521655+0000","last_became_active":"2024-09-17T04:03:39.335580+0000","last_became_peered":"2024-09-17T04:03:39.335580+0000","last_unstale":"2024-09-17T04:03:43.521655+0000","last_undegraded":"2024-09-17T04:03:43.521655+0000","last_fullsized":"2024-09-17T04:03:43.521655+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741903+0000","last_change":"2024-09-17T04:03:40.342551+0000","last_active":"2024-09-17T04:03:42.741903+0000","last_peered":"2024-09-17T04:03:42.741903+0000","last_clean":"2024-09-17T04:03:42.741903+0000","last_became_active":"2024-09-17T04:03:40.339096+0000","last_became_peered":"2024-09-17T04:03:40.339096+0000","last_unstale":"2024-09-17T04:03:42.741903+0000","last_undegraded":"2024-09-17T04:03:42.741903+0000","last_fullsized":"2024-09-17T04:03:42.741903+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:42.841396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410300+0000","last_change":"2024-09-17T04:03:39.331133+0000","last_active":"2024-09-17T04:03:42.410300+0000","last_peered":"2024-09-17T04:03:42.410300+0000","last_clean":"2024-09-17T04:03:42.410300+0000","last_became_active":"2024-09-17T04:03:39.330849+0000","last_became_peered":"2024-09-17T04:03:39.330849+0000","last_unstale":"2024-09-17T04:03:42.410300+0000","last_undegraded":"2024-09-17T04:03:42.410300+0000","last_fullsized":"2024-09-17T04:03:42.410300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:19.933450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741382+0000","last_change":"2024-09-17T04:03:40.339612+0000","last_active":"2024-09-17T04:03:42.741382+0000","last_peered":"2024-09-17T04:03:42.741382+0000","last_clean":"2024-09-17T04:03:42.741382+0000","last_became_active":"2024-09-17T04:03:40.339297+0000","last_became_peered":"2024-09-17T04:03:40.339297+0000","last_unstale":"2024-09-17T04:03:42.741382+0000","last_undegraded":"2024-09-17T04:03:42.741382+0000","last_fullsized":"2024-09-17T04:03:42.741382+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:14.950266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.335737+0000","last_change":"2024-09-17T04:03:39.334073+0000","last_active":"2024-09-17T04:03:39.335737+0000","last_peered":"2024-09-17T04:03:39.335737+0000","last_clean":"2024-09-17T04:03:39.335737+0000","last_became_active":"2024-09-17T04:03:39.333739+0000","last_became_peered":"2024-09-17T04:03:39.333739+0000","last_unstale":"2024-09-17T04:03:39.335737+0000","last_undegraded":"2024-09-17T04:03:39.335737+0000","last_fullsized":"2024-09-17T04:03:39.335737+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:36:00.772342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.312567+0000","last_change":"2024-09-17T04:03:39.312567+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.312567+0000","last_undegraded":"2024-09-17T04:03:39.312567+0000","last_fullsized":"2024-09-17T04:03:39.312567+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410122+0000","last_change":"2024-09-17T04:03:39.331198+0000","last_active":"2024-09-17T04:03:42.410122+0000","last_peered":"2024-09-17T04:03:42.410122+0000","last_clean":"2024-09-17T04:03:42.410122+0000","last_became_active":"2024-09-17T04:03:39.330893+0000","last_became_peered":"2024-09-17T04:03:39.330893+0000","last_unstale":"2024-09-17T04:03:42.410122+0000","last_undegraded":"2024-09-17T04:03:42.410122+0000","last_fullsized":"2024-09-17T04:03:42.410122+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:28.427519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410029+0000","last_change":"2024-09-17T04:03:40.347641+0000","last_active":"2024-09-17T04:03:42.410029+0000","last_peered":"2024-09-17T04:03:42.410029+0000","last_clean":"2024-09-17T04:03:42.410029+0000","last_became_active":"2024-09-17T04:03:40.347326+0000","last_became_peered":"2024-09-17T04:03:40.347326+0000","last_unstale":"2024-09-17T04:03:42.410029+0000","last_undegraded":"2024-09-17T04:03:42.410029+0000","last_fullsized":"2024-09-17T04:03:42.410029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:56.328537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521658+0000","last_change":"2024-09-17T04:03:39.335709+0000","last_active":"2024-09-17T04:03:43.521658+0000","last_peered":"2024-09-17T04:03:43.521658+0000","last_clean":"2024-09-17T04:03:43.521658+0000","last_became_active":"2024-09-17T04:03:39.334739+0000","last_became_peered":"2024-09-17T04:03:39.334739+0000","last_unstale":"2024-09-17T04:03:43.521658+0000","last_undegraded":"2024-09-17T04:03:43.521658+0000","last_fullsized":"2024-09-17T04:03:43.521658+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741466+0000","last_change":"2024-09-17T04:03:40.335288+0000","last_active":"2024-09-17T04:03:42.741466+0000","last_peered":"2024-09-17T04:03:42.741466+0000","last_clean":"2024-09-17T04:03:42.741466+0000","last_became_active":"2024-09-17T04:03:40.334907+0000","last_became_peered":"2024-09-17T04:03:40.334907+0000","last_unstale":"2024-09-17T04:03:42.741466+0000","last_undegraded":"2024-09-17T04:03:42.741466+0000","last_fullsized":"2024-09-17T04:03:42.741466+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:15.982851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482130+0000","last_change":"2024-09-17T04:03:39.331744+0000","last_active":"2024-09-17T04:03:41.482130+0000","last_peered":"2024-09-17T04:03:41.482130+0000","last_clean":"2024-09-17T04:03:41.482130+0000","last_became_active":"2024-09-17T04:03:39.330827+0000","last_became_peered":"2024-09-17T04:03:39.330827+0000","last_unstale":"2024-09-17T04:03:41.482130+0000","last_undegraded":"2024-09-17T04:03:41.482130+0000","last_fullsized":"2024-09-17T04:03:41.482130+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:57.121963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482922+0000","last_change":"2024-09-17T04:03:40.334111+0000","last_active":"2024-09-17T04:03:41.482922+0000","last_peered":"2024-09-17T04:03:41.482922+0000","last_clean":"2024-09-17T04:03:41.482922+0000","last_became_active":"2024-09-17T04:03:40.333244+0000","last_became_peered":"2024-09-17T04:03:40.333244+0000","last_unstale":"2024-09-17T04:03:41.482922+0000","last_undegraded":"2024-09-17T04:03:41.482922+0000","last_fullsized":"2024-09-17T04:03:41.482922+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:20.936128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.332507+0000","last_change":"2024-09-17T04:03:39.327400+0000","last_active":"2024-09-17T04:03:39.332507+0000","last_peered":"2024-09-17T04:03:39.332507+0000","last_clean":"2024-09-17T04:03:39.332507+0000","last_became_active":"2024-09-17T04:03:39.327045+0000","last_became_peered":"2024-09-17T04:03:39.327045+0000","last_unstale":"2024-09-17T04:03:39.332507+0000","last_undegraded":"2024-09-17T04:03:39.332507+0000","last_fullsized":"2024-09-17T04:03:39.332507+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:04.062871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.315550+0000","last_change":"2024-09-17T04:03:39.315550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.315550+0000","last_undegraded":"2024-09-17T04:03:39.315550+0000","last_fullsized":"2024-09-17T04:03:39.315550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"203'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333769+0000","last_change":"2024-09-17T04:03:39.332034+0000","last_active":"2024-09-17T04:03:41.333769+0000","last_peered":"2024-09-17T04:03:41.333769+0000","last_clean":"2024-09-17T04:03:41.333769+0000","last_became_active":"2024-09-17T04:03:39.331723+0000","last_became_peered":"2024-09-17T04:03:39.331723+0000","last_unstale":"2024-09-17T04:03:41.333769+0000","last_undegraded":"2024-09-17T04:03:41.333769+0000","last_fullsized":"2024-09-17T04:03:41.333769+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:01:32.008051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410209+0000","last_change":"2024-09-17T04:03:40.343426+0000","last_active":"2024-09-17T04:03:42.410209+0000","last_peered":"2024-09-17T04:03:42.410209+0000","last_clean":"2024-09-17T04:03:42.410209+0000","last_became_active":"2024-09-17T04:03:40.343092+0000","last_became_peered":"2024-09-17T04:03:40.343092+0000","last_unstale":"2024-09-17T04:03:42.410209+0000","last_undegraded":"2024-09-17T04:03:42.410209+0000","last_fullsized":"2024-09-17T04:03:42.410209+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:19.250495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521877+0000","last_change":"2024-09-17T04:03:39.335065+0000","last_active":"2024-09-17T04:03:43.521877+0000","last_peered":"2024-09-17T04:03:43.521877+0000","last_clean":"2024-09-17T04:03:43.521877+0000","last_became_active":"2024-09-17T04:03:39.334604+0000","last_became_peered":"2024-09-17T04:03:39.334604+0000","last_unstale":"2024-09-17T04:03:43.521877+0000","last_undegraded":"2024-09-17T04:03:43.521877+0000","last_fullsized":"2024-09-17T04:03:43.521877+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410698+0000","last_change":"2024-09-17T04:03:40.352477+0000","last_active":"2024-09-17T04:03:42.410698+0000","last_peered":"2024-09-17T04:03:42.410698+0000","last_clean":"2024-09-17T04:03:42.410698+0000","last_became_active":"2024-09-17T04:03:40.351366+0000","last_became_peered":"2024-09-17T04:03:40.351366+0000","last_unstale":"2024-09-17T04:03:42.410698+0000","last_undegraded":"2024-09-17T04:03:42.410698+0000","last_fullsized":"2024-09-17T04:03:42.410698+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:17:14.000165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410341+0000","last_change":"2024-09-17T04:03:39.335924+0000","last_active":"2024-09-17T04:03:42.410341+0000","last_peered":"2024-09-17T04:03:42.410341+0000","last_clean":"2024-09-17T04:03:42.410341+0000","last_became_active":"2024-09-17T04:03:39.335629+0000","last_became_peered":"2024-09-17T04:03:39.335629+0000","last_unstale":"2024-09-17T04:03:42.410341+0000","last_undegraded":"2024-09-17T04:03:42.410341+0000","last_fullsized":"2024-09-17T04:03:42.410341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:52.490212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482687+0000","last_change":"2024-09-17T04:03:40.339516+0000","last_active":"2024-09-17T04:03:41.482687+0000","last_peered":"2024-09-17T04:03:41.482687+0000","last_clean":"2024-09-17T04:03:41.482687+0000","last_became_active":"2024-09-17T04:03:40.339040+0000","last_became_peered":"2024-09-17T04:03:40.339040+0000","last_unstale":"2024-09-17T04:03:41.482687+0000","last_undegraded":"2024-09-17T04:03:41.482687+0000","last_fullsized":"2024-09-17T04:03:41.482687+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:33.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482477+0000","last_change":"2024-09-17T04:03:39.331355+0000","last_active":"2024-09-17T04:03:41.482477+0000","last_peered":"2024-09-17T04:03:41.482477+0000","last_clean":"2024-09-17T04:03:41.482477+0000","last_became_active":"2024-09-17T04:03:39.331003+0000","last_became_peered":"2024-09-17T04:03:39.331003+0000","last_unstale":"2024-09-17T04:03:41.482477+0000","last_undegraded":"2024-09-17T04:03:41.482477+0000","last_fullsized":"2024-09-17T04:03:41.482477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:52.147204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741734+0000","last_change":"2024-09-17T04:03:40.337242+0000","last_active":"2024-09-17T04:03:42.741734+0000","last_peered":"2024-09-17T04:03:42.741734+0000","last_clean":"2024-09-17T04:03:42.741734+0000","last_became_active":"2024-09-17T04:03:40.336866+0000","last_became_peered":"2024-09-17T04:03:40.336866+0000","last_unstale":"2024-09-17T04:03:42.741734+0000","last_undegraded":"2024-09-17T04:03:42.741734+0000","last_fullsized":"2024-09-17T04:03:42.741734+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:52.701613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.336329+0000","last_change":"2024-09-17T04:03:39.332196+0000","last_active":"2024-09-17T04:03:39.336329+0000","last_peered":"2024-09-17T04:03:39.336329+0000","last_clean":"2024-09-17T04:03:39.336329+0000","last_became_active":"2024-09-17T04:03:39.331897+0000","last_became_peered":"2024-09-17T04:03:39.331897+0000","last_unstale":"2024-09-17T04:03:39.336329+0000","last_undegraded":"2024-09-17T04:03:39.336329+0000","last_fullsized":"2024-09-17T04:03:39.336329+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:53:10.918846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741362+0000","last_change":"2024-09-17T04:03:40.331229+0000","last_active":"2024-09-17T04:03:42.741362+0000","last_peered":"2024-09-17T04:03:42.741362+0000","last_clean":"2024-09-17T04:03:42.741362+0000","last_became_active":"2024-09-17T04:03:40.330857+0000","last_became_peered":"2024-09-17T04:03:40.330857+0000","last_unstale":"2024-09-17T04:03:42.741362+0000","last_undegraded":"2024-09-17T04:03:42.741362+0000","last_fullsized":"2024-09-17T04:03:42.741362+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:18.666966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334106+0000","last_change":"2024-09-17T04:03:39.329983+0000","last_active":"2024-09-17T04:03:39.334106+0000","last_peered":"2024-09-17T04:03:39.334106+0000","last_clean":"2024-09-17T04:03:39.334106+0000","last_became_active":"2024-09-17T04:03:39.325670+0000","last_became_peered":"2024-09-17T04:03:39.325670+0000","last_unstale":"2024-09-17T04:03:39.334106+0000","last_undegraded":"2024-09-17T04:03:39.334106+0000","last_fullsized":"2024-09-17T04:03:39.334106+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:20.865772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":1,"reported_epoch":201,"state":"creating+peering","last_fresh":"2024-09-17T04:03:39.314550+0000","last_change":"2024-09-17T04:03:39.314550+0000","last_active":"2024-09-17T04:03:39.295451+0000","last_peered":"2024-09-17T04:03:39.295451+0000","last_clean":"2024-09-17T04:03:39.295451+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:03:39.314550+0000","last_undegraded":"2024-09-17T04:03:39.314550+0000","last_fullsized":"2024-09-17T04:03:39.314550+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334879+0000","last_change":"2024-09-17T04:03:39.332559+0000","last_active":"2024-09-17T04:03:39.334879+0000","last_peered":"2024-09-17T04:03:39.334879+0000","last_clean":"2024-09-17T04:03:39.334879+0000","last_became_active":"2024-09-17T04:03:39.332265+0000","last_became_peered":"2024-09-17T04:03:39.332265+0000","last_unstale":"2024-09-17T04:03:39.334879+0000","last_undegraded":"2024-09-17T04:03:39.334879+0000","last_fullsized":"2024-09-17T04:03:39.334879+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:07.087568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333644+0000","last_change":"2024-09-17T04:03:40.352076+0000","last_active":"2024-09-17T04:03:41.333644+0000","last_peered":"2024-09-17T04:03:41.333644+0000","last_clean":"2024-09-17T04:03:41.333644+0000","last_became_active":"2024-09-17T04:03:40.351735+0000","last_became_peered":"2024-09-17T04:03:40.351735+0000","last_unstale":"2024-09-17T04:03:41.333644+0000","last_undegraded":"2024-09-17T04:03:41.333644+0000","last_fullsized":"2024-09-17T04:03:41.333644+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:16.809754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"0'0","reported_seq":10,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-09-17T04:03:39.334891+0000","last_change":"2024-09-17T04:03:39.331554+0000","last_active":"2024-09-17T04:03:39.334891+0000","last_peered":"2024-09-17T04:03:39.334891+0000","last_clean":"2024-09-17T04:03:39.334891+0000","last_became_active":"2024-09-17T04:03:39.331212+0000","last_became_peered":"2024-09-17T04:03:39.331212+0000","last_unstale":"2024-09-17T04:03:39.334891+0000","last_undegraded":"2024-09-17T04:03:39.334891+0000","last_fullsized":"2024-09-17T04:03:39.334891+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:11:04.328735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":446,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521595+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:03:43.521595+0000","last_peered":"2024-09-17T04:03:43.521595+0000","last_clean":"2024-09-17T04:03:43.521595+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:03:43.521595+0000","last_undegraded":"2024-09-17T04:03:43.521595+0000","last_fullsized":"2024-09-17T04:03:43.521595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":92,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52320,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319520,"statfs":{"total":96636764160,"available":96583188480,"internally_reserved":0,"allocated":3158016,"data_stored":1716897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29528,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342312,"statfs":{"total":96636764160,"available":96606527488,"internally_reserved":0,"allocated":2560000,"data_stored":1126459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509616,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509613,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51716,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320124,"statfs":{"total":96636764160,"available":96583806976,"internally_reserved":0,"allocated":2535424,"data_stored":1113384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509615,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509615,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51700,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320140,"statfs":{"total":96636764160,"available":96583823360,"internally_reserved":0,"allocated":2523136,"data_stored":1107109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542318,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1127063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542319,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3158016,"data_stored":1716861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:03:45.725 INFO:tasks.ceph.ceph_manager.ceph:PG 31.3f is not active+clean 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.315975+0000', 'last_change': '2024-09-17T04:03:39.315975+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.315975+0000', 'last_undegraded': '2024-09-17T04:03:39.315975+0000', 'last_fullsized': '2024-09-17T04:03:39.315975+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:PG 31.3e is not active+clean 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.314425+0000', 'last_change': '2024-09-17T04:03:39.314425+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.314425+0000', 'last_undegraded': '2024-09-17T04:03:39.314425+0000', 'last_fullsized': '2024-09-17T04:03:39.314425+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:PG 31.3d is not active+clean 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.313411+0000', 'last_change': '2024-09-17T04:03:39.313411+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.313411+0000', 'last_undegraded': '2024-09-17T04:03:39.313411+0000', 'last_fullsized': '2024-09-17T04:03:39.313411+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.726 INFO:tasks.ceph.ceph_manager.ceph:PG 31.3c is not active+clean 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.318949+0000', 'last_change': '2024-09-17T04:03:39.318949+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.318949+0000', 'last_undegraded': '2024-09-17T04:03:39.318949+0000', 'last_fullsized': '2024-09-17T04:03:39.318949+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:PG 31.3a is not active+clean 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.317948+0000', 'last_change': '2024-09-17T04:03:39.317948+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.317948+0000', 'last_undegraded': '2024-09-17T04:03:39.317948+0000', 'last_fullsized': '2024-09-17T04:03:39.317948+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:PG 31.32 is not active+clean 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.312559+0000', 'last_change': '2024-09-17T04:03:39.312559+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.312559+0000', 'last_undegraded': '2024-09-17T04:03:39.312559+0000', 'last_fullsized': '2024-09-17T04:03:39.312559+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:PG 31.30 is not active+clean 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.313125+0000', 'last_change': '2024-09-17T04:03:39.313125+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.313125+0000', 'last_undegraded': '2024-09-17T04:03:39.313125+0000', 'last_fullsized': '2024-09-17T04:03:39.313125+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:PG 31.2e is not active+clean 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.313567+0000', 'last_change': '2024-09-17T04:03:39.313567+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.313567+0000', 'last_undegraded': '2024-09-17T04:03:39.313567+0000', 'last_fullsized': '2024-09-17T04:03:39.313567+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.727 INFO:tasks.ceph.ceph_manager.ceph:PG 31.2c is not active+clean 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.313824+0000', 'last_change': '2024-09-17T04:03:39.313824+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.313824+0000', 'last_undegraded': '2024-09-17T04:03:39.313824+0000', 'last_fullsized': '2024-09-17T04:03:39.313824+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:PG 31.28 is not active+clean 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.316858+0000', 'last_change': '2024-09-17T04:03:39.316858+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.316858+0000', 'last_undegraded': '2024-09-17T04:03:39.316858+0000', 'last_fullsized': '2024-09-17T04:03:39.316858+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:PG 31.27 is not active+clean 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.324689+0000', 'last_change': '2024-09-17T04:03:39.324689+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.324689+0000', 'last_undegraded': '2024-09-17T04:03:39.324689+0000', 'last_fullsized': '2024-09-17T04:03:39.324689+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:PG 31.24 is not active+clean 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.312330+0000', 'last_change': '2024-09-17T04:03:39.312330+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.312330+0000', 'last_undegraded': '2024-09-17T04:03:39.312330+0000', 'last_fullsized': '2024-09-17T04:03:39.312330+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:PG 31.21 is not active+clean 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.321401+0000', 'last_change': '2024-09-17T04:03:39.321401+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.321401+0000', 'last_undegraded': '2024-09-17T04:03:39.321401+0000', 'last_fullsized': '2024-09-17T04:03:39.321401+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.728 INFO:tasks.ceph.ceph_manager.ceph:PG 31.20 is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.316435+0000', 'last_change': '2024-09-17T04:03:39.316435+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.316435+0000', 'last_undegraded': '2024-09-17T04:03:39.316435+0000', 'last_fullsized': '2024-09-17T04:03:39.316435+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:PG 31.1f is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.312535+0000', 'last_change': '2024-09-17T04:03:39.312535+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.312535+0000', 'last_undegraded': '2024-09-17T04:03:39.312535+0000', 'last_fullsized': '2024-09-17T04:03:39.312535+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:PG 31.18 is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.312542+0000', 'last_change': '2024-09-17T04:03:39.312542+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.312542+0000', 'last_undegraded': '2024-09-17T04:03:39.312542+0000', 'last_fullsized': '2024-09-17T04:03:39.312542+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:PG 31.19 is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.314983+0000', 'last_change': '2024-09-17T04:03:39.314983+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.314983+0000', 'last_undegraded': '2024-09-17T04:03:39.314983+0000', 'last_fullsized': '2024-09-17T04:03:39.314983+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:PG 31.16 is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.313113+0000', 'last_change': '2024-09-17T04:03:39.313113+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.313113+0000', 'last_undegraded': '2024-09-17T04:03:39.313113+0000', 'last_fullsized': '2024-09-17T04:03:39.313113+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:PG 31.14 is not active+clean 2024-09-17T04:03:45.729 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.311933+0000', 'last_change': '2024-09-17T04:03:39.311933+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.311933+0000', 'last_undegraded': '2024-09-17T04:03:39.311933+0000', 'last_fullsized': '2024-09-17T04:03:39.311933+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.10 is not active+clean 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.317212+0000', 'last_change': '2024-09-17T04:03:39.317212+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.317212+0000', 'last_undegraded': '2024-09-17T04:03:39.317212+0000', 'last_fullsized': '2024-09-17T04:03:39.317212+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.e is not active+clean 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.318088+0000', 'last_change': '2024-09-17T04:03:39.318088+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.318088+0000', 'last_undegraded': '2024-09-17T04:03:39.318088+0000', 'last_fullsized': '2024-09-17T04:03:39.318088+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.f is not active+clean 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.311836+0000', 'last_change': '2024-09-17T04:03:39.311836+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.311836+0000', 'last_undegraded': '2024-09-17T04:03:39.311836+0000', 'last_fullsized': '2024-09-17T04:03:39.311836+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.b is not active+clean 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.312567+0000', 'last_change': '2024-09-17T04:03:39.312567+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.312567+0000', 'last_undegraded': '2024-09-17T04:03:39.312567+0000', 'last_fullsized': '2024-09-17T04:03:39.312567+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.7 is not active+clean 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.315550+0000', 'last_change': '2024-09-17T04:03:39.315550+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.315550+0000', 'last_undegraded': '2024-09-17T04:03:39.315550+0000', 'last_fullsized': '2024-09-17T04:03:39.315550+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.730 INFO:tasks.ceph.ceph_manager.ceph:PG 31.1 is not active+clean 2024-09-17T04:03:45.731 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '31.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 201, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:03:39.314550+0000', 'last_change': '2024-09-17T04:03:39.314550+0000', 'last_active': '2024-09-17T04:03:39.295451+0000', 'last_peered': '2024-09-17T04:03:39.295451+0000', 'last_clean': '2024-09-17T04:03:39.295451+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:03:39.314550+0000', 'last_undegraded': '2024-09-17T04:03:39.314550+0000', 'last_fullsized': '2024-09-17T04:03:39.314550+0000', 'mapping_epoch': 201, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 201, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'last_clean_scrub_stamp': '2024-09-17T04:03:39.295451+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:03:45.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:03:45.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 -- 172.21.15.129:0/772418217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a815ce50 msgr2=0x7f47a80af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 --2- 172.21.15.129:0/772418217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a815ce50 0x7f47a80af7a0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f4798005fd0 tx=0x7f479802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 -- 172.21.15.129:0/772418217 shutdown_connections 2024-09-17T04:03:45.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 --2- 172.21.15.129:0/772418217 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80b4180 0x7f47a80ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 --2- 172.21.15.129:0/772418217 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a815ce50 0x7f47a80af7a0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 -- 172.21.15.129:0/772418217 >> 172.21.15.129:0/772418217 conn(0x7f47a806d410 msgr2=0x7f47a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:45.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 -- 172.21.15.129:0/772418217 shutdown_connections 2024-09-17T04:03:45.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.978+0000 7f47ae600640 1 -- 172.21.15.129:0/772418217 wait complete. 2024-09-17T04:03:45.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.980+0000 7f47ae600640 1 Processor -- start 2024-09-17T04:03:45.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.981+0000 7f47ae600640 1 -- start start 2024-09-17T04:03:45.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80ccc80 0x7f47a80b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47a815ce50 0x7f47a81580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47a806f930 con 0x7f47a80b4180 2024-09-17T04:03:45.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47a806f7b0 con 0x7f47a815ce50 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47ae600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f47a806fab0 con 0x7f47a80ccc80 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47a7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46114/0 (socket says 172.21.15.129:46114) 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.982+0000 7f47a7400640 1 -- 172.21.15.129:0/4230527272 learned_addr learned my addr 172.21.15.129:0/4230527272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a6a00640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80ccc80 0x7f47a80b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7e00640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47a815ce50 0x7f47a81580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 -- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80ccc80 msgr2=0x7f47a80b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80ccc80 0x7f47a80b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 -- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47a815ce50 msgr2=0x7f47a81580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47a815ce50 0x7f47a81580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 -- 172.21.15.129:0/4230527272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a8157040 con 0x7f47a80b4180 2024-09-17T04:03:45.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47a7400640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f4798034040 tx=0x7f47980038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479803e030 con 0x7f47a80b4180 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4798003ca0 con 0x7f47a80b4180 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a8157320 con 0x7f47a80b4180 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.983+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4798035040 con 0x7f47a80b4180 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.984+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a81578e0 con 0x7f47a80b4180 2024-09-17T04:03:45.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.985+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a806ffd0 con 0x7f47a80b4180 2024-09-17T04:03:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.986+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4798041030 con 0x7f47a80b4180 2024-09-17T04:03:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.987+0000 7f479f400640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 0x7f478807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.988+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f47980bb450 con 0x7f47a80b4180 2024-09-17T04:03:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.988+0000 7f47a6a00640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 0x7f478807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:45.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.989+0000 7f47a6a00640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 0x7f478807b0a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4794005d70 tx=0x7f4794005d00 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:45.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:45.992+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4798083880 con 0x7f47a80b4180 2024-09-17T04:03:46.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.243+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f47a80b2980 con 0x7f47a80b4180 2024-09-17T04:03:46.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.244+0000 7f479f400640 1 -- 172.21.15.129:0/4230527272 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1145 (secure 0 0 0) 0x7f4798083180 con 0x7f47a80b4180 2024-09-17T04:03:46.244 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:46.244 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":877,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":203,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":72},{"state_name":"creating+peering","count":25}],"num_pgs":97,"num_pools":3,"num_objects":9,"data_bytes":590460,"bytes_used":383733760,"bytes_avail":772710379520,"bytes_total":773094113280,"inactive_pgs_ratio":0.25773194432258606,"write_bytes_sec":341,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":181,"btime":"2024-09-17T04:03:43:337368+0000","id":15,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":15,"rank":0,"name":"d","status":"up:active","gid":10440}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":27,"modified":"2024-09-17T04:03:06.526412+0000","services":{}},"progress_events":{}} 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 msgr2=0x7f478807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 0x7f478807b0a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f4794005d70 tx=0x7f4794005d00 comp rx=0 tx=0).stop 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 msgr2=0x7f47a81f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f4798034040 tx=0x7f47980038a0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 shutdown_connections 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4788078b90 0x7f478807b0a0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47a815ce50 0x7f47a81580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47a80ccc80 0x7f47a80b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 --2- 172.21.15.129:0/4230527272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47a80b4180 0x7f47a81f5bb0 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 >> 172.21.15.129:0/4230527272 conn(0x7f47a806d410 msgr2=0x7f47a80744d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:46.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 shutdown_connections 2024-09-17T04:03:46.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.246+0000 7f47ae600640 1 -- 172.21.15.129:0/4230527272 wait complete. 2024-09-17T04:03:46.271 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:03:46.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 -- 172.21.15.129:0/1046560323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00cd390 msgr2=0x7f2bb0070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 --2- 172.21.15.129:0/1046560323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00cd390 0x7f2bb0070100 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f2bb00ce800 tx=0x7f2ba802aa10 comp rx=0 tx=0).stop 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 -- 172.21.15.129:0/1046560323 shutdown_connections 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 --2- 172.21.15.129:0/1046560323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00cd390 0x7f2bb0070100 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 --2- 172.21.15.129:0/1046560323 >> v2:172.21.15.181:3300/0 conn(0x7f2bb015da00 0x7f2bb00b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 -- 172.21.15.129:0/1046560323 >> 172.21.15.129:0/1046560323 conn(0x7f2bb006d410 msgr2=0x7f2bb00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 -- 172.21.15.129:0/1046560323 shutdown_connections 2024-09-17T04:03:46.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.493+0000 7f2bb6800640 1 -- 172.21.15.129:0/1046560323 wait complete. 2024-09-17T04:03:46.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.496+0000 7f2bb6800640 1 Processor -- start 2024-09-17T04:03:46.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.496+0000 7f2bb6800640 1 -- start start 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.497+0000 7f2bb6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b46c0 0x7f2bb009db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.497+0000 7f2bb6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb015da00 0x7f2bb00c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.497+0000 7f2bb6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bb6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb00709d0 con 0x7f2bb01f39e0 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bb6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb0070850 con 0x7f2bb00b46c0 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bb6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb0070b50 con 0x7f2bb015da00 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:46.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46128/0 (socket says 172.21.15.129:46128) 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 -- 172.21.15.129:0/3796140750 learned_addr learned my addr 172.21.15.129:0/3796140750 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2ba6a00640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb015da00 0x7f2bb00c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2baf400640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b46c0 0x7f2bb009db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 -- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb015da00 msgr2=0x7f2bb00c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb015da00 0x7f2bb00c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 -- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b46c0 msgr2=0x7f2bb009db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b46c0 0x7f2bb009db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.498+0000 7f2bafe00640 1 -- 172.21.15.129:0/3796140750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb01f7220 con 0x7f2bb01f39e0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2bafe00640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f2ba802b000 tx=0x7f2ba8003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8033050 con 0x7f2bb01f39e0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ba8004550 con 0x7f2bb01f39e0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb009e0a0 con 0x7f2bb01f39e0 2024-09-17T04:03:46.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8038450 con 0x7f2bb01f39e0 2024-09-17T04:03:46.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.499+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb01f77e0 con 0x7f2bb01f39e0 2024-09-17T04:03:46.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.500+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb0070100 con 0x7f2bb01f39e0 2024-09-17T04:03:46.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.501+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2ba8036070 con 0x7f2bb01f39e0 2024-09-17T04:03:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.502+0000 7f2ba7e00640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:46.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.502+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f2ba80bc1f0 con 0x7f2bb01f39e0 2024-09-17T04:03:46.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.503+0000 7f2baf400640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:46.503 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.503+0000 7f2baf400640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0005fd0 tx=0x7f2ba000a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:46.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.507+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2ba8083fd0 con 0x7f2bb01f39e0 2024-09-17T04:03:46.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.689+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2bb01f6fd0 con 0x7f2b8c078bc0 2024-09-17T04:03:46.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.694+0000 7f2ba7e00640 1 -- 172.21.15.129:0/3796140750 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252482 (secure 0 0 0) 0x7f2bb01f6fd0 con 0x7f2b8c078bc0 2024-09-17T04:03:46.697 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:46.698 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:03:46.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 msgr2=0x7f2b8c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0005fd0 tx=0x7f2ba000a250 comp rx=0 tx=0).stop 2024-09-17T04:03:46.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 msgr2=0x7f2bb015bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:46.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f2ba802b000 tx=0x7f2ba8003f10 comp rx=0 tx=0).stop 2024-09-17T04:03:46.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 shutdown_connections 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb01f39e0 0x7f2bb015bd30 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb015da00 0x7f2bb00c91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 --2- 172.21.15.129:0/3796140750 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b46c0 0x7f2bb009db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 >> 172.21.15.129:0/3796140750 conn(0x7f2bb006d410 msgr2=0x7f2bb00ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 shutdown_connections 2024-09-17T04:03:46.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:46.698+0000 7f2bb6800640 1 -- 172.21.15.129:0/3796140750 wait complete. 2024-09-17T04:03:46.724 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2024-09-17T04:03:46.537025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":379020,"kb_used_data":21224,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754595700,"statfs":{"total":773094113280,"available":772705996800,"internally_reserved":0,"allocated":21733376,"data_stored":10197228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":366202416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.227173"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330963+0000","last_change":"2024-09-17T04:03:40.330095+0000","last_active":"2024-09-17T04:03:41.330963+0000","last_peered":"2024-09-17T04:03:41.330963+0000","last_clean":"2024-09-17T04:03:41.330963+0000","last_became_active":"2024-09-17T04:03:40.329761+0000","last_became_peered":"2024-09-17T04:03:40.329761+0000","last_unstale":"2024-09-17T04:03:41.330963+0000","last_undegraded":"2024-09-17T04:03:41.330963+0000","last_fullsized":"2024-09-17T04:03:41.330963+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:38.735272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330025+0000","last_change":"2024-09-17T04:03:40.340996+0000","last_active":"2024-09-17T04:03:41.330025+0000","last_peered":"2024-09-17T04:03:41.330025+0000","last_clean":"2024-09-17T04:03:41.330025+0000","last_became_active":"2024-09-17T04:03:40.340458+0000","last_became_peered":"2024-09-17T04:03:40.340458+0000","last_unstale":"2024-09-17T04:03:41.330025+0000","last_undegraded":"2024-09-17T04:03:41.330025+0000","last_fullsized":"2024-09-17T04:03:41.330025+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:17.195721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497110+0000","last_change":"2024-09-17T04:03:40.347883+0000","last_active":"2024-09-17T04:03:41.497110+0000","last_peered":"2024-09-17T04:03:41.497110+0000","last_clean":"2024-09-17T04:03:41.497110+0000","last_became_active":"2024-09-17T04:03:40.347571+0000","last_became_peered":"2024-09-17T04:03:40.347571+0000","last_unstale":"2024-09-17T04:03:41.497110+0000","last_undegraded":"2024-09-17T04:03:41.497110+0000","last_fullsized":"2024-09-17T04:03:41.497110+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:58.174465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332582+0000","last_change":"2024-09-17T04:03:40.332143+0000","last_active":"2024-09-17T04:03:41.332582+0000","last_peered":"2024-09-17T04:03:41.332582+0000","last_clean":"2024-09-17T04:03:41.332582+0000","last_became_active":"2024-09-17T04:03:40.331789+0000","last_became_peered":"2024-09-17T04:03:40.331789+0000","last_unstale":"2024-09-17T04:03:41.332582+0000","last_undegraded":"2024-09-17T04:03:41.332582+0000","last_fullsized":"2024-09-17T04:03:41.332582+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:06.672381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333497+0000","last_change":"2024-09-17T04:03:40.335355+0000","last_active":"2024-09-17T04:03:41.333497+0000","last_peered":"2024-09-17T04:03:41.333497+0000","last_clean":"2024-09-17T04:03:41.333497+0000","last_became_active":"2024-09-17T04:03:40.334994+0000","last_became_peered":"2024-09-17T04:03:40.334994+0000","last_unstale":"2024-09-17T04:03:41.333497+0000","last_undegraded":"2024-09-17T04:03:41.333497+0000","last_fullsized":"2024-09-17T04:03:41.333497+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:02.228911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332515+0000","last_change":"2024-09-17T04:03:40.355785+0000","last_active":"2024-09-17T04:03:41.332515+0000","last_peered":"2024-09-17T04:03:41.332515+0000","last_clean":"2024-09-17T04:03:41.332515+0000","last_became_active":"2024-09-17T04:03:40.355371+0000","last_became_peered":"2024-09-17T04:03:40.355371+0000","last_unstale":"2024-09-17T04:03:41.332515+0000","last_undegraded":"2024-09-17T04:03:41.332515+0000","last_fullsized":"2024-09-17T04:03:41.332515+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:38.226897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482089+0000","last_change":"2024-09-17T04:03:40.350322+0000","last_active":"2024-09-17T04:03:41.482089+0000","last_peered":"2024-09-17T04:03:41.482089+0000","last_clean":"2024-09-17T04:03:41.482089+0000","last_became_active":"2024-09-17T04:03:40.349953+0000","last_became_peered":"2024-09-17T04:03:40.349953+0000","last_unstale":"2024-09-17T04:03:41.482089+0000","last_undegraded":"2024-09-17T04:03:41.482089+0000","last_fullsized":"2024-09-17T04:03:41.482089+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:03:12.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741207+0000","last_change":"2024-09-17T04:03:40.334851+0000","last_active":"2024-09-17T04:03:42.741207+0000","last_peered":"2024-09-17T04:03:42.741207+0000","last_clean":"2024-09-17T04:03:42.741207+0000","last_became_active":"2024-09-17T04:03:40.331337+0000","last_became_peered":"2024-09-17T04:03:40.331337+0000","last_unstale":"2024-09-17T04:03:42.741207+0000","last_undegraded":"2024-09-17T04:03:42.741207+0000","last_fullsized":"2024-09-17T04:03:42.741207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:52.463737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521788+0000","last_change":"2024-09-17T04:03:40.349511+0000","last_active":"2024-09-17T04:03:43.521788+0000","last_peered":"2024-09-17T04:03:43.521788+0000","last_clean":"2024-09-17T04:03:43.521788+0000","last_became_active":"2024-09-17T04:03:40.349113+0000","last_became_peered":"2024-09-17T04:03:40.349113+0000","last_unstale":"2024-09-17T04:03:43.521788+0000","last_undegraded":"2024-09-17T04:03:43.521788+0000","last_fullsized":"2024-09-17T04:03:43.521788+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:50.920504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482695+0000","last_change":"2024-09-17T04:03:40.340480+0000","last_active":"2024-09-17T04:03:41.482695+0000","last_peered":"2024-09-17T04:03:41.482695+0000","last_clean":"2024-09-17T04:03:41.482695+0000","last_became_active":"2024-09-17T04:03:40.338521+0000","last_became_peered":"2024-09-17T04:03:40.338521+0000","last_unstale":"2024-09-17T04:03:41.482695+0000","last_undegraded":"2024-09-17T04:03:41.482695+0000","last_fullsized":"2024-09-17T04:03:41.482695+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:10.458690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410478+0000","last_change":"2024-09-17T04:03:40.348952+0000","last_active":"2024-09-17T04:03:42.410478+0000","last_peered":"2024-09-17T04:03:42.410478+0000","last_clean":"2024-09-17T04:03:42.410478+0000","last_became_active":"2024-09-17T04:03:40.348656+0000","last_became_peered":"2024-09-17T04:03:40.348656+0000","last_unstale":"2024-09-17T04:03:42.410478+0000","last_undegraded":"2024-09-17T04:03:42.410478+0000","last_fullsized":"2024-09-17T04:03:42.410478+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:47.744159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741493+0000","last_change":"2024-09-17T04:03:40.354695+0000","last_active":"2024-09-17T04:03:42.741493+0000","last_peered":"2024-09-17T04:03:42.741493+0000","last_clean":"2024-09-17T04:03:42.741493+0000","last_became_active":"2024-09-17T04:03:40.354274+0000","last_became_peered":"2024-09-17T04:03:40.354274+0000","last_unstale":"2024-09-17T04:03:42.741493+0000","last_undegraded":"2024-09-17T04:03:42.741493+0000","last_fullsized":"2024-09-17T04:03:42.741493+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:15.701377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482731+0000","last_change":"2024-09-17T04:03:40.350373+0000","last_active":"2024-09-17T04:03:41.482731+0000","last_peered":"2024-09-17T04:03:41.482731+0000","last_clean":"2024-09-17T04:03:41.482731+0000","last_became_active":"2024-09-17T04:03:40.350000+0000","last_became_peered":"2024-09-17T04:03:40.350000+0000","last_unstale":"2024-09-17T04:03:41.482731+0000","last_undegraded":"2024-09-17T04:03:41.482731+0000","last_fullsized":"2024-09-17T04:03:41.482731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:29.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329902+0000","last_change":"2024-09-17T04:03:40.320319+0000","last_active":"2024-09-17T04:03:41.329902+0000","last_peered":"2024-09-17T04:03:41.329902+0000","last_clean":"2024-09-17T04:03:41.329902+0000","last_became_active":"2024-09-17T04:03:40.320061+0000","last_became_peered":"2024-09-17T04:03:40.320061+0000","last_unstale":"2024-09-17T04:03:41.329902+0000","last_undegraded":"2024-09-17T04:03:41.329902+0000","last_fullsized":"2024-09-17T04:03:41.329902+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:23.008632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521536+0000","last_change":"2024-09-17T04:03:40.344294+0000","last_active":"2024-09-17T04:03:43.521536+0000","last_peered":"2024-09-17T04:03:43.521536+0000","last_clean":"2024-09-17T04:03:43.521536+0000","last_became_active":"2024-09-17T04:03:40.343923+0000","last_became_peered":"2024-09-17T04:03:40.343923+0000","last_unstale":"2024-09-17T04:03:43.521536+0000","last_undegraded":"2024-09-17T04:03:43.521536+0000","last_fullsized":"2024-09-17T04:03:43.521536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:35.362812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496287+0000","last_change":"2024-09-17T04:03:40.328076+0000","last_active":"2024-09-17T04:03:41.496287+0000","last_peered":"2024-09-17T04:03:41.496287+0000","last_clean":"2024-09-17T04:03:41.496287+0000","last_became_active":"2024-09-17T04:03:40.327770+0000","last_became_peered":"2024-09-17T04:03:40.327770+0000","last_unstale":"2024-09-17T04:03:41.496287+0000","last_undegraded":"2024-09-17T04:03:41.496287+0000","last_fullsized":"2024-09-17T04:03:41.496287+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:40.277989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741536+0000","last_change":"2024-09-17T04:03:40.343695+0000","last_active":"2024-09-17T04:03:42.741536+0000","last_peered":"2024-09-17T04:03:42.741536+0000","last_clean":"2024-09-17T04:03:42.741536+0000","last_became_active":"2024-09-17T04:03:40.343413+0000","last_became_peered":"2024-09-17T04:03:40.343413+0000","last_unstale":"2024-09-17T04:03:42.741536+0000","last_undegraded":"2024-09-17T04:03:42.741536+0000","last_fullsized":"2024-09-17T04:03:42.741536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:54.092079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330462+0000","last_change":"2024-09-17T04:03:40.334723+0000","last_active":"2024-09-17T04:03:41.330462+0000","last_peered":"2024-09-17T04:03:41.330462+0000","last_clean":"2024-09-17T04:03:41.330462+0000","last_became_active":"2024-09-17T04:03:40.334418+0000","last_became_peered":"2024-09-17T04:03:40.334418+0000","last_unstale":"2024-09-17T04:03:41.330462+0000","last_undegraded":"2024-09-17T04:03:41.330462+0000","last_fullsized":"2024-09-17T04:03:41.330462+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:00.430811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:40.351995+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:40.351649+0000","last_became_peered":"2024-09-17T04:03:40.351649+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:03.942253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496692+0000","last_change":"2024-09-17T04:03:40.319052+0000","last_active":"2024-09-17T04:03:41.496692+0000","last_peered":"2024-09-17T04:03:41.496692+0000","last_clean":"2024-09-17T04:03:41.496692+0000","last_became_active":"2024-09-17T04:03:40.318761+0000","last_became_peered":"2024-09-17T04:03:40.318761+0000","last_unstale":"2024-09-17T04:03:41.496692+0000","last_undegraded":"2024-09-17T04:03:41.496692+0000","last_fullsized":"2024-09-17T04:03:41.496692+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:33.494049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741813+0000","last_change":"2024-09-17T04:03:40.333277+0000","last_active":"2024-09-17T04:03:42.741813+0000","last_peered":"2024-09-17T04:03:42.741813+0000","last_clean":"2024-09-17T04:03:42.741813+0000","last_became_active":"2024-09-17T04:03:40.332882+0000","last_became_peered":"2024-09-17T04:03:40.332882+0000","last_unstale":"2024-09-17T04:03:42.741813+0000","last_undegraded":"2024-09-17T04:03:42.741813+0000","last_fullsized":"2024-09-17T04:03:42.741813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:44.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482369+0000","last_change":"2024-09-17T04:03:40.341616+0000","last_active":"2024-09-17T04:03:41.482369+0000","last_peered":"2024-09-17T04:03:41.482369+0000","last_clean":"2024-09-17T04:03:41.482369+0000","last_became_active":"2024-09-17T04:03:40.341269+0000","last_became_peered":"2024-09-17T04:03:40.341269+0000","last_unstale":"2024-09-17T04:03:41.482369+0000","last_undegraded":"2024-09-17T04:03:41.482369+0000","last_fullsized":"2024-09-17T04:03:41.482369+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:28.676331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741352+0000","last_change":"2024-09-17T04:03:40.337777+0000","last_active":"2024-09-17T04:03:42.741352+0000","last_peered":"2024-09-17T04:03:42.741352+0000","last_clean":"2024-09-17T04:03:42.741352+0000","last_became_active":"2024-09-17T04:03:40.335850+0000","last_became_peered":"2024-09-17T04:03:40.335850+0000","last_unstale":"2024-09-17T04:03:42.741352+0000","last_undegraded":"2024-09-17T04:03:42.741352+0000","last_fullsized":"2024-09-17T04:03:42.741352+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:37.334408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329845+0000","last_change":"2024-09-17T04:03:40.344001+0000","last_active":"2024-09-17T04:03:41.329845+0000","last_peered":"2024-09-17T04:03:41.329845+0000","last_clean":"2024-09-17T04:03:41.329845+0000","last_became_active":"2024-09-17T04:03:40.339574+0000","last_became_peered":"2024-09-17T04:03:40.339574+0000","last_unstale":"2024-09-17T04:03:41.329845+0000","last_undegraded":"2024-09-17T04:03:41.329845+0000","last_fullsized":"2024-09-17T04:03:41.329845+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.778853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333133+0000","last_change":"2024-09-17T04:03:40.330798+0000","last_active":"2024-09-17T04:03:41.333133+0000","last_peered":"2024-09-17T04:03:41.333133+0000","last_clean":"2024-09-17T04:03:41.333133+0000","last_became_active":"2024-09-17T04:03:40.330363+0000","last_became_peered":"2024-09-17T04:03:40.330363+0000","last_unstale":"2024-09-17T04:03:41.333133+0000","last_undegraded":"2024-09-17T04:03:41.333133+0000","last_fullsized":"2024-09-17T04:03:41.333133+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:53.191430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482816+0000","last_change":"2024-09-17T04:03:40.350740+0000","last_active":"2024-09-17T04:03:41.482816+0000","last_peered":"2024-09-17T04:03:41.482816+0000","last_clean":"2024-09-17T04:03:41.482816+0000","last_became_active":"2024-09-17T04:03:40.350415+0000","last_became_peered":"2024-09-17T04:03:40.350415+0000","last_unstale":"2024-09-17T04:03:41.482816+0000","last_undegraded":"2024-09-17T04:03:41.482816+0000","last_fullsized":"2024-09-17T04:03:41.482816+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:04.588026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410543+0000","last_change":"2024-09-17T04:03:40.347688+0000","last_active":"2024-09-17T04:03:42.410543+0000","last_peered":"2024-09-17T04:03:42.410543+0000","last_clean":"2024-09-17T04:03:42.410543+0000","last_became_active":"2024-09-17T04:03:40.347381+0000","last_became_peered":"2024-09-17T04:03:40.347381+0000","last_unstale":"2024-09-17T04:03:42.410543+0000","last_undegraded":"2024-09-17T04:03:42.410543+0000","last_fullsized":"2024-09-17T04:03:42.410543+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:51.255876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330673+0000","last_change":"2024-09-17T04:03:40.356708+0000","last_active":"2024-09-17T04:03:41.330673+0000","last_peered":"2024-09-17T04:03:41.330673+0000","last_clean":"2024-09-17T04:03:41.330673+0000","last_became_active":"2024-09-17T04:03:40.352402+0000","last_became_peered":"2024-09-17T04:03:40.352402+0000","last_unstale":"2024-09-17T04:03:41.330673+0000","last_undegraded":"2024-09-17T04:03:41.330673+0000","last_fullsized":"2024-09-17T04:03:41.330673+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:19.133678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410325+0000","last_change":"2024-09-17T04:03:40.350667+0000","last_active":"2024-09-17T04:03:42.410325+0000","last_peered":"2024-09-17T04:03:42.410325+0000","last_clean":"2024-09-17T04:03:42.410325+0000","last_became_active":"2024-09-17T04:03:40.350312+0000","last_became_peered":"2024-09-17T04:03:40.350312+0000","last_unstale":"2024-09-17T04:03:42.410325+0000","last_undegraded":"2024-09-17T04:03:42.410325+0000","last_fullsized":"2024-09-17T04:03:42.410325+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:21.193199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333504+0000","last_change":"2024-09-17T04:03:40.331612+0000","last_active":"2024-09-17T04:03:41.333504+0000","last_peered":"2024-09-17T04:03:41.333504+0000","last_clean":"2024-09-17T04:03:41.333504+0000","last_became_active":"2024-09-17T04:03:40.329978+0000","last_became_peered":"2024-09-17T04:03:40.329978+0000","last_unstale":"2024-09-17T04:03:41.333504+0000","last_undegraded":"2024-09-17T04:03:41.333504+0000","last_fullsized":"2024-09-17T04:03:41.333504+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:48.029240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332513+0000","last_change":"2024-09-17T04:03:40.329393+0000","last_active":"2024-09-17T04:03:41.332513+0000","last_peered":"2024-09-17T04:03:41.332513+0000","last_clean":"2024-09-17T04:03:41.332513+0000","last_became_active":"2024-09-17T04:03:40.329011+0000","last_became_peered":"2024-09-17T04:03:40.329011+0000","last_unstale":"2024-09-17T04:03:41.332513+0000","last_undegraded":"2024-09-17T04:03:41.332513+0000","last_fullsized":"2024-09-17T04:03:41.332513+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:43:46.684973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329902+0000","last_change":"2024-09-17T04:03:40.328538+0000","last_active":"2024-09-17T04:03:41.329902+0000","last_peered":"2024-09-17T04:03:41.329902+0000","last_clean":"2024-09-17T04:03:41.329902+0000","last_became_active":"2024-09-17T04:03:40.328220+0000","last_became_peered":"2024-09-17T04:03:40.328220+0000","last_unstale":"2024-09-17T04:03:41.329902+0000","last_undegraded":"2024-09-17T04:03:41.329902+0000","last_fullsized":"2024-09-17T04:03:41.329902+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:07.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497139+0000","last_change":"2024-09-17T04:03:40.334720+0000","last_active":"2024-09-17T04:03:41.497139+0000","last_peered":"2024-09-17T04:03:41.497139+0000","last_clean":"2024-09-17T04:03:41.497139+0000","last_became_active":"2024-09-17T04:03:40.334427+0000","last_became_peered":"2024-09-17T04:03:40.334427+0000","last_unstale":"2024-09-17T04:03:41.497139+0000","last_undegraded":"2024-09-17T04:03:41.497139+0000","last_fullsized":"2024-09-17T04:03:41.497139+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:29.796643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482954+0000","last_change":"2024-09-17T04:03:39.330078+0000","last_active":"2024-09-17T04:03:41.482954+0000","last_peered":"2024-09-17T04:03:41.482954+0000","last_clean":"2024-09-17T04:03:41.482954+0000","last_became_active":"2024-09-17T04:03:39.329766+0000","last_became_peered":"2024-09-17T04:03:39.329766+0000","last_unstale":"2024-09-17T04:03:41.482954+0000","last_undegraded":"2024-09-17T04:03:41.482954+0000","last_fullsized":"2024-09-17T04:03:41.482954+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:11.195328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410186+0000","last_change":"2024-09-17T04:03:40.351683+0000","last_active":"2024-09-17T04:03:42.410186+0000","last_peered":"2024-09-17T04:03:42.410186+0000","last_clean":"2024-09-17T04:03:42.410186+0000","last_became_active":"2024-09-17T04:03:40.351288+0000","last_became_peered":"2024-09-17T04:03:40.351288+0000","last_unstale":"2024-09-17T04:03:42.410186+0000","last_undegraded":"2024-09-17T04:03:42.410186+0000","last_fullsized":"2024-09-17T04:03:42.410186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:11:57.454665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497043+0000","last_change":"2024-09-17T04:03:39.322960+0000","last_active":"2024-09-17T04:03:41.497043+0000","last_peered":"2024-09-17T04:03:41.497043+0000","last_clean":"2024-09-17T04:03:41.497043+0000","last_became_active":"2024-09-17T04:03:39.322704+0000","last_became_peered":"2024-09-17T04:03:39.322704+0000","last_unstale":"2024-09-17T04:03:41.497043+0000","last_undegraded":"2024-09-17T04:03:41.497043+0000","last_fullsized":"2024-09-17T04:03:41.497043+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:19.046905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741839+0000","last_change":"2024-09-17T04:03:40.343840+0000","last_active":"2024-09-17T04:03:42.741839+0000","last_peered":"2024-09-17T04:03:42.741839+0000","last_clean":"2024-09-17T04:03:42.741839+0000","last_became_active":"2024-09-17T04:03:40.343539+0000","last_became_peered":"2024-09-17T04:03:40.343539+0000","last_unstale":"2024-09-17T04:03:42.741839+0000","last_undegraded":"2024-09-17T04:03:42.741839+0000","last_fullsized":"2024-09-17T04:03:42.741839+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:04.501121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496914+0000","last_change":"2024-09-17T04:03:39.320734+0000","last_active":"2024-09-17T04:03:41.496914+0000","last_peered":"2024-09-17T04:03:41.496914+0000","last_clean":"2024-09-17T04:03:41.496914+0000","last_became_active":"2024-09-17T04:03:39.320431+0000","last_became_peered":"2024-09-17T04:03:39.320431+0000","last_unstale":"2024-09-17T04:03:41.496914+0000","last_undegraded":"2024-09-17T04:03:41.496914+0000","last_fullsized":"2024-09-17T04:03:41.496914+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:05.564947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410125+0000","last_change":"2024-09-17T04:03:40.335363+0000","last_active":"2024-09-17T04:03:42.410125+0000","last_peered":"2024-09-17T04:03:42.410125+0000","last_clean":"2024-09-17T04:03:42.410125+0000","last_became_active":"2024-09-17T04:03:40.333333+0000","last_became_peered":"2024-09-17T04:03:40.333333+0000","last_unstale":"2024-09-17T04:03:42.410125+0000","last_undegraded":"2024-09-17T04:03:42.410125+0000","last_fullsized":"2024-09-17T04:03:42.410125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:35.331891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332477+0000","last_change":"2024-09-17T04:03:39.329538+0000","last_active":"2024-09-17T04:03:41.332477+0000","last_peered":"2024-09-17T04:03:41.332477+0000","last_clean":"2024-09-17T04:03:41.332477+0000","last_became_active":"2024-09-17T04:03:39.329257+0000","last_became_peered":"2024-09-17T04:03:39.329257+0000","last_unstale":"2024-09-17T04:03:41.332477+0000","last_undegraded":"2024-09-17T04:03:41.332477+0000","last_fullsized":"2024-09-17T04:03:41.332477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:56:03.999758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521725+0000","last_change":"2024-09-17T04:03:40.344266+0000","last_active":"2024-09-17T04:03:43.521725+0000","last_peered":"2024-09-17T04:03:43.521725+0000","last_clean":"2024-09-17T04:03:43.521725+0000","last_became_active":"2024-09-17T04:03:40.343914+0000","last_became_peered":"2024-09-17T04:03:40.343914+0000","last_unstale":"2024-09-17T04:03:43.521725+0000","last_undegraded":"2024-09-17T04:03:43.521725+0000","last_fullsized":"2024-09-17T04:03:43.521725+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:49.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330144+0000","last_change":"2024-09-17T04:03:39.321662+0000","last_active":"2024-09-17T04:03:41.330144+0000","last_peered":"2024-09-17T04:03:41.330144+0000","last_clean":"2024-09-17T04:03:41.330144+0000","last_became_active":"2024-09-17T04:03:39.321278+0000","last_became_peered":"2024-09-17T04:03:39.321278+0000","last_unstale":"2024-09-17T04:03:41.330144+0000","last_undegraded":"2024-09-17T04:03:41.330144+0000","last_fullsized":"2024-09-17T04:03:41.330144+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:11.779647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496290+0000","last_change":"2024-09-17T04:03:40.331538+0000","last_active":"2024-09-17T04:03:41.496290+0000","last_peered":"2024-09-17T04:03:41.496290+0000","last_clean":"2024-09-17T04:03:41.496290+0000","last_became_active":"2024-09-17T04:03:40.331210+0000","last_became_peered":"2024-09-17T04:03:40.331210+0000","last_unstale":"2024-09-17T04:03:41.496290+0000","last_undegraded":"2024-09-17T04:03:41.496290+0000","last_fullsized":"2024-09-17T04:03:41.496290+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:00.716824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482232+0000","last_change":"2024-09-17T04:03:39.332398+0000","last_active":"2024-09-17T04:03:41.482232+0000","last_peered":"2024-09-17T04:03:41.482232+0000","last_clean":"2024-09-17T04:03:41.482232+0000","last_became_active":"2024-09-17T04:03:39.331945+0000","last_became_peered":"2024-09-17T04:03:39.331945+0000","last_unstale":"2024-09-17T04:03:41.482232+0000","last_undegraded":"2024-09-17T04:03:41.482232+0000","last_fullsized":"2024-09-17T04:03:41.482232+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.951852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329863+0000","last_change":"2024-09-17T04:03:40.334437+0000","last_active":"2024-09-17T04:03:41.329863+0000","last_peered":"2024-09-17T04:03:41.329863+0000","last_clean":"2024-09-17T04:03:41.329863+0000","last_became_active":"2024-09-17T04:03:40.334121+0000","last_became_peered":"2024-09-17T04:03:40.334121+0000","last_unstale":"2024-09-17T04:03:41.329863+0000","last_undegraded":"2024-09-17T04:03:41.329863+0000","last_fullsized":"2024-09-17T04:03:41.329863+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:12:54.843913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496562+0000","last_change":"2024-09-17T04:03:39.317105+0000","last_active":"2024-09-17T04:03:41.496562+0000","last_peered":"2024-09-17T04:03:41.496562+0000","last_clean":"2024-09-17T04:03:41.496562+0000","last_became_active":"2024-09-17T04:03:39.316779+0000","last_became_peered":"2024-09-17T04:03:39.316779+0000","last_unstale":"2024-09-17T04:03:41.496562+0000","last_undegraded":"2024-09-17T04:03:41.496562+0000","last_fullsized":"2024-09-17T04:03:41.496562+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:11.069998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330936+0000","last_change":"2024-09-17T04:03:40.339195+0000","last_active":"2024-09-17T04:03:41.330936+0000","last_peered":"2024-09-17T04:03:41.330936+0000","last_clean":"2024-09-17T04:03:41.330936+0000","last_became_active":"2024-09-17T04:03:40.336174+0000","last_became_peered":"2024-09-17T04:03:40.336174+0000","last_unstale":"2024-09-17T04:03:41.330936+0000","last_undegraded":"2024-09-17T04:03:41.330936+0000","last_fullsized":"2024-09-17T04:03:41.330936+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:18:54.704094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:39.326124+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:39.325797+0000","last_became_peered":"2024-09-17T04:03:39.325797+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:04.499200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410736+0000","last_change":"2024-09-17T04:03:40.351885+0000","last_active":"2024-09-17T04:03:42.410736+0000","last_peered":"2024-09-17T04:03:42.410736+0000","last_clean":"2024-09-17T04:03:42.410736+0000","last_became_active":"2024-09-17T04:03:40.351494+0000","last_became_peered":"2024-09-17T04:03:40.351494+0000","last_unstale":"2024-09-17T04:03:42.410736+0000","last_undegraded":"2024-09-17T04:03:42.410736+0000","last_fullsized":"2024-09-17T04:03:42.410736+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:13.184588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497062+0000","last_change":"2024-09-17T04:03:39.328133+0000","last_active":"2024-09-17T04:03:41.497062+0000","last_peered":"2024-09-17T04:03:41.497062+0000","last_clean":"2024-09-17T04:03:41.497062+0000","last_became_active":"2024-09-17T04:03:39.327863+0000","last_became_peered":"2024-09-17T04:03:39.327863+0000","last_unstale":"2024-09-17T04:03:41.497062+0000","last_undegraded":"2024-09-17T04:03:41.497062+0000","last_fullsized":"2024-09-17T04:03:41.497062+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.776175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496825+0000","last_change":"2024-09-17T04:03:40.345889+0000","last_active":"2024-09-17T04:03:41.496825+0000","last_peered":"2024-09-17T04:03:41.496825+0000","last_clean":"2024-09-17T04:03:41.496825+0000","last_became_active":"2024-09-17T04:03:40.345589+0000","last_became_peered":"2024-09-17T04:03:40.345589+0000","last_unstale":"2024-09-17T04:03:41.496825+0000","last_undegraded":"2024-09-17T04:03:41.496825+0000","last_fullsized":"2024-09-17T04:03:41.496825+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:20.025729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"202'3","reported_seq":18,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330142+0000","last_change":"2024-09-17T04:03:39.331097+0000","last_active":"2024-09-17T04:03:41.330142+0000","last_peered":"2024-09-17T04:03:41.330142+0000","last_clean":"2024-09-17T04:03:41.330142+0000","last_became_active":"2024-09-17T04:03:39.330131+0000","last_became_peered":"2024-09-17T04:03:39.330131+0000","last_unstale":"2024-09-17T04:03:41.330142+0000","last_undegraded":"2024-09-17T04:03:41.330142+0000","last_fullsized":"2024-09-17T04:03:41.330142+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482540+0000","last_change":"2024-09-17T04:03:40.332717+0000","last_active":"2024-09-17T04:03:41.482540+0000","last_peered":"2024-09-17T04:03:41.482540+0000","last_clean":"2024-09-17T04:03:41.482540+0000","last_became_active":"2024-09-17T04:03:40.332421+0000","last_became_peered":"2024-09-17T04:03:40.332421+0000","last_unstale":"2024-09-17T04:03:41.482540+0000","last_undegraded":"2024-09-17T04:03:41.482540+0000","last_fullsized":"2024-09-17T04:03:41.482540+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:42.530742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"202'6","reported_seq":21,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330379+0000","last_change":"2024-09-17T04:03:39.329612+0000","last_active":"2024-09-17T04:03:41.330379+0000","last_peered":"2024-09-17T04:03:41.330379+0000","last_clean":"2024-09-17T04:03:41.330379+0000","last_became_active":"2024-09-17T04:03:39.329342+0000","last_became_peered":"2024-09-17T04:03:39.329342+0000","last_unstale":"2024-09-17T04:03:41.330379+0000","last_undegraded":"2024-09-17T04:03:41.330379+0000","last_fullsized":"2024-09-17T04:03:41.330379+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:51.187815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482365+0000","last_change":"2024-09-17T04:03:40.338842+0000","last_active":"2024-09-17T04:03:41.482365+0000","last_peered":"2024-09-17T04:03:41.482365+0000","last_clean":"2024-09-17T04:03:41.482365+0000","last_became_active":"2024-09-17T04:03:40.338449+0000","last_became_peered":"2024-09-17T04:03:40.338449+0000","last_unstale":"2024-09-17T04:03:41.482365+0000","last_undegraded":"2024-09-17T04:03:41.482365+0000","last_fullsized":"2024-09-17T04:03:41.482365+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:44.963272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330165+0000","last_change":"2024-09-17T04:03:39.324560+0000","last_active":"2024-09-17T04:03:41.330165+0000","last_peered":"2024-09-17T04:03:41.330165+0000","last_clean":"2024-09-17T04:03:41.330165+0000","last_became_active":"2024-09-17T04:03:39.324206+0000","last_became_peered":"2024-09-17T04:03:39.324206+0000","last_unstale":"2024-09-17T04:03:41.330165+0000","last_undegraded":"2024-09-17T04:03:41.330165+0000","last_fullsized":"2024-09-17T04:03:41.330165+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:21.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521913+0000","last_change":"2024-09-17T04:03:40.344849+0000","last_active":"2024-09-17T04:03:43.521913+0000","last_peered":"2024-09-17T04:03:43.521913+0000","last_clean":"2024-09-17T04:03:43.521913+0000","last_became_active":"2024-09-17T04:03:40.344173+0000","last_became_peered":"2024-09-17T04:03:40.344173+0000","last_unstale":"2024-09-17T04:03:43.521913+0000","last_undegraded":"2024-09-17T04:03:43.521913+0000","last_fullsized":"2024-09-17T04:03:43.521913+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:12.072676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496514+0000","last_change":"2024-09-17T04:03:39.327456+0000","last_active":"2024-09-17T04:03:41.496514+0000","last_peered":"2024-09-17T04:03:41.496514+0000","last_clean":"2024-09-17T04:03:41.496514+0000","last_became_active":"2024-09-17T04:03:39.326151+0000","last_became_peered":"2024-09-17T04:03:39.326151+0000","last_unstale":"2024-09-17T04:03:41.496514+0000","last_undegraded":"2024-09-17T04:03:41.496514+0000","last_fullsized":"2024-09-17T04:03:41.496514+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:48.907118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332322+0000","last_change":"2024-09-17T04:03:40.347714+0000","last_active":"2024-09-17T04:03:41.332322+0000","last_peered":"2024-09-17T04:03:41.332322+0000","last_clean":"2024-09-17T04:03:41.332322+0000","last_became_active":"2024-09-17T04:03:40.347413+0000","last_became_peered":"2024-09-17T04:03:40.347413+0000","last_unstale":"2024-09-17T04:03:41.332322+0000","last_undegraded":"2024-09-17T04:03:41.332322+0000","last_fullsized":"2024-09-17T04:03:41.332322+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:16.273359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482297+0000","last_change":"2024-09-17T04:03:39.330613+0000","last_active":"2024-09-17T04:03:41.482297+0000","last_peered":"2024-09-17T04:03:41.482297+0000","last_clean":"2024-09-17T04:03:41.482297+0000","last_became_active":"2024-09-17T04:03:39.330283+0000","last_became_peered":"2024-09-17T04:03:39.330283+0000","last_unstale":"2024-09-17T04:03:41.482297+0000","last_undegraded":"2024-09-17T04:03:41.482297+0000","last_fullsized":"2024-09-17T04:03:41.482297+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:46.502036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482092+0000","last_change":"2024-09-17T04:03:40.341659+0000","last_active":"2024-09-17T04:03:41.482092+0000","last_peered":"2024-09-17T04:03:41.482092+0000","last_clean":"2024-09-17T04:03:41.482092+0000","last_became_active":"2024-09-17T04:03:40.341344+0000","last_became_peered":"2024-09-17T04:03:40.341344+0000","last_unstale":"2024-09-17T04:03:41.482092+0000","last_undegraded":"2024-09-17T04:03:41.482092+0000","last_fullsized":"2024-09-17T04:03:41.482092+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:24.293693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"202'8","reported_seq":23,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.409812+0000","last_change":"2024-09-17T04:03:39.332639+0000","last_active":"2024-09-17T04:03:42.409812+0000","last_peered":"2024-09-17T04:03:42.409812+0000","last_clean":"2024-09-17T04:03:42.409812+0000","last_became_active":"2024-09-17T04:03:39.332409+0000","last_became_peered":"2024-09-17T04:03:39.332409+0000","last_unstale":"2024-09-17T04:03:42.409812+0000","last_undegraded":"2024-09-17T04:03:42.409812+0000","last_fullsized":"2024-09-17T04:03:42.409812+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:03.585348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333079+0000","last_change":"2024-09-17T04:03:40.332345+0000","last_active":"2024-09-17T04:03:41.333079+0000","last_peered":"2024-09-17T04:03:41.333079+0000","last_clean":"2024-09-17T04:03:41.333079+0000","last_became_active":"2024-09-17T04:03:40.332029+0000","last_became_peered":"2024-09-17T04:03:40.332029+0000","last_unstale":"2024-09-17T04:03:41.333079+0000","last_undegraded":"2024-09-17T04:03:41.333079+0000","last_fullsized":"2024-09-17T04:03:41.333079+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:57.501108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482664+0000","last_change":"2024-09-17T04:03:39.333660+0000","last_active":"2024-09-17T04:03:41.482664+0000","last_peered":"2024-09-17T04:03:41.482664+0000","last_clean":"2024-09-17T04:03:41.482664+0000","last_became_active":"2024-09-17T04:03:39.333330+0000","last_became_peered":"2024-09-17T04:03:39.333330+0000","last_unstale":"2024-09-17T04:03:41.482664+0000","last_undegraded":"2024-09-17T04:03:41.482664+0000","last_fullsized":"2024-09-17T04:03:41.482664+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:03.955054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497298+0000","last_change":"2024-09-17T04:03:40.328129+0000","last_active":"2024-09-17T04:03:41.497298+0000","last_peered":"2024-09-17T04:03:41.497298+0000","last_clean":"2024-09-17T04:03:41.497298+0000","last_became_active":"2024-09-17T04:03:40.327790+0000","last_became_peered":"2024-09-17T04:03:40.327790+0000","last_unstale":"2024-09-17T04:03:41.497298+0000","last_undegraded":"2024-09-17T04:03:41.497298+0000","last_fullsized":"2024-09-17T04:03:41.497298+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:43.906106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497076+0000","last_change":"2024-09-17T04:03:39.322064+0000","last_active":"2024-09-17T04:03:41.497076+0000","last_peered":"2024-09-17T04:03:41.497076+0000","last_clean":"2024-09-17T04:03:41.497076+0000","last_became_active":"2024-09-17T04:03:39.321750+0000","last_became_peered":"2024-09-17T04:03:39.321750+0000","last_unstale":"2024-09-17T04:03:41.497076+0000","last_undegraded":"2024-09-17T04:03:41.497076+0000","last_fullsized":"2024-09-17T04:03:41.497076+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:22.005954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521646+0000","last_change":"2024-09-17T04:03:40.340280+0000","last_active":"2024-09-17T04:03:43.521646+0000","last_peered":"2024-09-17T04:03:43.521646+0000","last_clean":"2024-09-17T04:03:43.521646+0000","last_became_active":"2024-09-17T04:03:40.339904+0000","last_became_peered":"2024-09-17T04:03:40.339904+0000","last_unstale":"2024-09-17T04:03:43.521646+0000","last_undegraded":"2024-09-17T04:03:43.521646+0000","last_fullsized":"2024-09-17T04:03:43.521646+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:09.142973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521655+0000","last_change":"2024-09-17T04:03:39.335875+0000","last_active":"2024-09-17T04:03:43.521655+0000","last_peered":"2024-09-17T04:03:43.521655+0000","last_clean":"2024-09-17T04:03:43.521655+0000","last_became_active":"2024-09-17T04:03:39.335580+0000","last_became_peered":"2024-09-17T04:03:39.335580+0000","last_unstale":"2024-09-17T04:03:43.521655+0000","last_undegraded":"2024-09-17T04:03:43.521655+0000","last_fullsized":"2024-09-17T04:03:43.521655+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741903+0000","last_change":"2024-09-17T04:03:40.342551+0000","last_active":"2024-09-17T04:03:42.741903+0000","last_peered":"2024-09-17T04:03:42.741903+0000","last_clean":"2024-09-17T04:03:42.741903+0000","last_became_active":"2024-09-17T04:03:40.339096+0000","last_became_peered":"2024-09-17T04:03:40.339096+0000","last_unstale":"2024-09-17T04:03:42.741903+0000","last_undegraded":"2024-09-17T04:03:42.741903+0000","last_fullsized":"2024-09-17T04:03:42.741903+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:42.841396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410300+0000","last_change":"2024-09-17T04:03:39.331133+0000","last_active":"2024-09-17T04:03:42.410300+0000","last_peered":"2024-09-17T04:03:42.410300+0000","last_clean":"2024-09-17T04:03:42.410300+0000","last_became_active":"2024-09-17T04:03:39.330849+0000","last_became_peered":"2024-09-17T04:03:39.330849+0000","last_unstale":"2024-09-17T04:03:42.410300+0000","last_undegraded":"2024-09-17T04:03:42.410300+0000","last_fullsized":"2024-09-17T04:03:42.410300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:19.933450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741382+0000","last_change":"2024-09-17T04:03:40.339612+0000","last_active":"2024-09-17T04:03:42.741382+0000","last_peered":"2024-09-17T04:03:42.741382+0000","last_clean":"2024-09-17T04:03:42.741382+0000","last_became_active":"2024-09-17T04:03:40.339297+0000","last_became_peered":"2024-09-17T04:03:40.339297+0000","last_unstale":"2024-09-17T04:03:42.741382+0000","last_undegraded":"2024-09-17T04:03:42.741382+0000","last_fullsized":"2024-09-17T04:03:42.741382+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:14.950266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496646+0000","last_change":"2024-09-17T04:03:39.334073+0000","last_active":"2024-09-17T04:03:41.496646+0000","last_peered":"2024-09-17T04:03:41.496646+0000","last_clean":"2024-09-17T04:03:41.496646+0000","last_became_active":"2024-09-17T04:03:39.333739+0000","last_became_peered":"2024-09-17T04:03:39.333739+0000","last_unstale":"2024-09-17T04:03:41.496646+0000","last_undegraded":"2024-09-17T04:03:41.496646+0000","last_fullsized":"2024-09-17T04:03:41.496646+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.360134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330131+0000","last_change":"2024-09-17T04:03:40.337036+0000","last_active":"2024-09-17T04:03:41.330131+0000","last_peered":"2024-09-17T04:03:41.330131+0000","last_clean":"2024-09-17T04:03:41.330131+0000","last_became_active":"2024-09-17T04:03:40.336743+0000","last_became_peered":"2024-09-17T04:03:40.336743+0000","last_unstale":"2024-09-17T04:03:41.330131+0000","last_undegraded":"2024-09-17T04:03:41.330131+0000","last_fullsized":"2024-09-17T04:03:41.330131+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:45:34.712374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410122+0000","last_change":"2024-09-17T04:03:39.331198+0000","last_active":"2024-09-17T04:03:42.410122+0000","last_peered":"2024-09-17T04:03:42.410122+0000","last_clean":"2024-09-17T04:03:42.410122+0000","last_became_active":"2024-09-17T04:03:39.330893+0000","last_became_peered":"2024-09-17T04:03:39.330893+0000","last_unstale":"2024-09-17T04:03:42.410122+0000","last_undegraded":"2024-09-17T04:03:42.410122+0000","last_fullsized":"2024-09-17T04:03:42.410122+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:28.427519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410029+0000","last_change":"2024-09-17T04:03:40.347641+0000","last_active":"2024-09-17T04:03:42.410029+0000","last_peered":"2024-09-17T04:03:42.410029+0000","last_clean":"2024-09-17T04:03:42.410029+0000","last_became_active":"2024-09-17T04:03:40.347326+0000","last_became_peered":"2024-09-17T04:03:40.347326+0000","last_unstale":"2024-09-17T04:03:42.410029+0000","last_undegraded":"2024-09-17T04:03:42.410029+0000","last_fullsized":"2024-09-17T04:03:42.410029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:56.328537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521658+0000","last_change":"2024-09-17T04:03:39.335709+0000","last_active":"2024-09-17T04:03:43.521658+0000","last_peered":"2024-09-17T04:03:43.521658+0000","last_clean":"2024-09-17T04:03:43.521658+0000","last_became_active":"2024-09-17T04:03:39.334739+0000","last_became_peered":"2024-09-17T04:03:39.334739+0000","last_unstale":"2024-09-17T04:03:43.521658+0000","last_undegraded":"2024-09-17T04:03:43.521658+0000","last_fullsized":"2024-09-17T04:03:43.521658+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741466+0000","last_change":"2024-09-17T04:03:40.335288+0000","last_active":"2024-09-17T04:03:42.741466+0000","last_peered":"2024-09-17T04:03:42.741466+0000","last_clean":"2024-09-17T04:03:42.741466+0000","last_became_active":"2024-09-17T04:03:40.334907+0000","last_became_peered":"2024-09-17T04:03:40.334907+0000","last_unstale":"2024-09-17T04:03:42.741466+0000","last_undegraded":"2024-09-17T04:03:42.741466+0000","last_fullsized":"2024-09-17T04:03:42.741466+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:15.982851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482130+0000","last_change":"2024-09-17T04:03:39.331744+0000","last_active":"2024-09-17T04:03:41.482130+0000","last_peered":"2024-09-17T04:03:41.482130+0000","last_clean":"2024-09-17T04:03:41.482130+0000","last_became_active":"2024-09-17T04:03:39.330827+0000","last_became_peered":"2024-09-17T04:03:39.330827+0000","last_unstale":"2024-09-17T04:03:41.482130+0000","last_undegraded":"2024-09-17T04:03:41.482130+0000","last_fullsized":"2024-09-17T04:03:41.482130+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:57.121963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482922+0000","last_change":"2024-09-17T04:03:40.334111+0000","last_active":"2024-09-17T04:03:41.482922+0000","last_peered":"2024-09-17T04:03:41.482922+0000","last_clean":"2024-09-17T04:03:41.482922+0000","last_became_active":"2024-09-17T04:03:40.333244+0000","last_became_peered":"2024-09-17T04:03:40.333244+0000","last_unstale":"2024-09-17T04:03:41.482922+0000","last_undegraded":"2024-09-17T04:03:41.482922+0000","last_fullsized":"2024-09-17T04:03:41.482922+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:20.936128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333140+0000","last_change":"2024-09-17T04:03:39.327400+0000","last_active":"2024-09-17T04:03:41.333140+0000","last_peered":"2024-09-17T04:03:41.333140+0000","last_clean":"2024-09-17T04:03:41.333140+0000","last_became_active":"2024-09-17T04:03:39.327045+0000","last_became_peered":"2024-09-17T04:03:39.327045+0000","last_unstale":"2024-09-17T04:03:41.333140+0000","last_undegraded":"2024-09-17T04:03:41.333140+0000","last_fullsized":"2024-09-17T04:03:41.333140+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:42.554312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.331177+0000","last_change":"2024-09-17T04:03:40.352872+0000","last_active":"2024-09-17T04:03:41.331177+0000","last_peered":"2024-09-17T04:03:41.331177+0000","last_clean":"2024-09-17T04:03:41.331177+0000","last_became_active":"2024-09-17T04:03:40.352511+0000","last_became_peered":"2024-09-17T04:03:40.352511+0000","last_unstale":"2024-09-17T04:03:41.331177+0000","last_undegraded":"2024-09-17T04:03:41.331177+0000","last_fullsized":"2024-09-17T04:03:41.331177+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:54.181921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"203'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333769+0000","last_change":"2024-09-17T04:03:39.332034+0000","last_active":"2024-09-17T04:03:41.333769+0000","last_peered":"2024-09-17T04:03:41.333769+0000","last_clean":"2024-09-17T04:03:41.333769+0000","last_became_active":"2024-09-17T04:03:39.331723+0000","last_became_peered":"2024-09-17T04:03:39.331723+0000","last_unstale":"2024-09-17T04:03:41.333769+0000","last_undegraded":"2024-09-17T04:03:41.333769+0000","last_fullsized":"2024-09-17T04:03:41.333769+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:01:32.008051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410209+0000","last_change":"2024-09-17T04:03:40.343426+0000","last_active":"2024-09-17T04:03:42.410209+0000","last_peered":"2024-09-17T04:03:42.410209+0000","last_clean":"2024-09-17T04:03:42.410209+0000","last_became_active":"2024-09-17T04:03:40.343092+0000","last_became_peered":"2024-09-17T04:03:40.343092+0000","last_unstale":"2024-09-17T04:03:42.410209+0000","last_undegraded":"2024-09-17T04:03:42.410209+0000","last_fullsized":"2024-09-17T04:03:42.410209+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:19.250495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521877+0000","last_change":"2024-09-17T04:03:39.335065+0000","last_active":"2024-09-17T04:03:43.521877+0000","last_peered":"2024-09-17T04:03:43.521877+0000","last_clean":"2024-09-17T04:03:43.521877+0000","last_became_active":"2024-09-17T04:03:39.334604+0000","last_became_peered":"2024-09-17T04:03:39.334604+0000","last_unstale":"2024-09-17T04:03:43.521877+0000","last_undegraded":"2024-09-17T04:03:43.521877+0000","last_fullsized":"2024-09-17T04:03:43.521877+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410698+0000","last_change":"2024-09-17T04:03:40.352477+0000","last_active":"2024-09-17T04:03:42.410698+0000","last_peered":"2024-09-17T04:03:42.410698+0000","last_clean":"2024-09-17T04:03:42.410698+0000","last_became_active":"2024-09-17T04:03:40.351366+0000","last_became_peered":"2024-09-17T04:03:40.351366+0000","last_unstale":"2024-09-17T04:03:42.410698+0000","last_undegraded":"2024-09-17T04:03:42.410698+0000","last_fullsized":"2024-09-17T04:03:42.410698+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:17:14.000165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410341+0000","last_change":"2024-09-17T04:03:39.335924+0000","last_active":"2024-09-17T04:03:42.410341+0000","last_peered":"2024-09-17T04:03:42.410341+0000","last_clean":"2024-09-17T04:03:42.410341+0000","last_became_active":"2024-09-17T04:03:39.335629+0000","last_became_peered":"2024-09-17T04:03:39.335629+0000","last_unstale":"2024-09-17T04:03:42.410341+0000","last_undegraded":"2024-09-17T04:03:42.410341+0000","last_fullsized":"2024-09-17T04:03:42.410341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:52.490212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482687+0000","last_change":"2024-09-17T04:03:40.339516+0000","last_active":"2024-09-17T04:03:41.482687+0000","last_peered":"2024-09-17T04:03:41.482687+0000","last_clean":"2024-09-17T04:03:41.482687+0000","last_became_active":"2024-09-17T04:03:40.339040+0000","last_became_peered":"2024-09-17T04:03:40.339040+0000","last_unstale":"2024-09-17T04:03:41.482687+0000","last_undegraded":"2024-09-17T04:03:41.482687+0000","last_fullsized":"2024-09-17T04:03:41.482687+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:33.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482477+0000","last_change":"2024-09-17T04:03:39.331355+0000","last_active":"2024-09-17T04:03:41.482477+0000","last_peered":"2024-09-17T04:03:41.482477+0000","last_clean":"2024-09-17T04:03:41.482477+0000","last_became_active":"2024-09-17T04:03:39.331003+0000","last_became_peered":"2024-09-17T04:03:39.331003+0000","last_unstale":"2024-09-17T04:03:41.482477+0000","last_undegraded":"2024-09-17T04:03:41.482477+0000","last_fullsized":"2024-09-17T04:03:41.482477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:52.147204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741734+0000","last_change":"2024-09-17T04:03:40.337242+0000","last_active":"2024-09-17T04:03:42.741734+0000","last_peered":"2024-09-17T04:03:42.741734+0000","last_clean":"2024-09-17T04:03:42.741734+0000","last_became_active":"2024-09-17T04:03:40.336866+0000","last_became_peered":"2024-09-17T04:03:40.336866+0000","last_unstale":"2024-09-17T04:03:42.741734+0000","last_undegraded":"2024-09-17T04:03:42.741734+0000","last_fullsized":"2024-09-17T04:03:42.741734+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:52.701613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332623+0000","last_change":"2024-09-17T04:03:39.332196+0000","last_active":"2024-09-17T04:03:41.332623+0000","last_peered":"2024-09-17T04:03:41.332623+0000","last_clean":"2024-09-17T04:03:41.332623+0000","last_became_active":"2024-09-17T04:03:39.331897+0000","last_became_peered":"2024-09-17T04:03:39.331897+0000","last_unstale":"2024-09-17T04:03:41.332623+0000","last_undegraded":"2024-09-17T04:03:41.332623+0000","last_fullsized":"2024-09-17T04:03:41.332623+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:39.348568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741362+0000","last_change":"2024-09-17T04:03:40.331229+0000","last_active":"2024-09-17T04:03:42.741362+0000","last_peered":"2024-09-17T04:03:42.741362+0000","last_clean":"2024-09-17T04:03:42.741362+0000","last_became_active":"2024-09-17T04:03:40.330857+0000","last_became_peered":"2024-09-17T04:03:40.330857+0000","last_unstale":"2024-09-17T04:03:42.741362+0000","last_undegraded":"2024-09-17T04:03:42.741362+0000","last_fullsized":"2024-09-17T04:03:42.741362+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:18.666966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496366+0000","last_change":"2024-09-17T04:03:39.329983+0000","last_active":"2024-09-17T04:03:41.496366+0000","last_peered":"2024-09-17T04:03:41.496366+0000","last_clean":"2024-09-17T04:03:41.496366+0000","last_became_active":"2024-09-17T04:03:39.325670+0000","last_became_peered":"2024-09-17T04:03:39.325670+0000","last_unstale":"2024-09-17T04:03:41.496366+0000","last_undegraded":"2024-09-17T04:03:41.496366+0000","last_fullsized":"2024-09-17T04:03:41.496366+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.252323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329936+0000","last_change":"2024-09-17T04:03:40.337809+0000","last_active":"2024-09-17T04:03:41.329936+0000","last_peered":"2024-09-17T04:03:41.329936+0000","last_clean":"2024-09-17T04:03:41.329936+0000","last_became_active":"2024-09-17T04:03:40.337519+0000","last_became_peered":"2024-09-17T04:03:40.337519+0000","last_unstale":"2024-09-17T04:03:41.329936+0000","last_undegraded":"2024-09-17T04:03:41.329936+0000","last_fullsized":"2024-09-17T04:03:41.329936+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:20.049583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496512+0000","last_change":"2024-09-17T04:03:39.332559+0000","last_active":"2024-09-17T04:03:41.496512+0000","last_peered":"2024-09-17T04:03:41.496512+0000","last_clean":"2024-09-17T04:03:41.496512+0000","last_became_active":"2024-09-17T04:03:39.332265+0000","last_became_peered":"2024-09-17T04:03:39.332265+0000","last_unstale":"2024-09-17T04:03:41.496512+0000","last_undegraded":"2024-09-17T04:03:41.496512+0000","last_fullsized":"2024-09-17T04:03:41.496512+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:49.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333644+0000","last_change":"2024-09-17T04:03:40.352076+0000","last_active":"2024-09-17T04:03:41.333644+0000","last_peered":"2024-09-17T04:03:41.333644+0000","last_clean":"2024-09-17T04:03:41.333644+0000","last_became_active":"2024-09-17T04:03:40.351735+0000","last_became_peered":"2024-09-17T04:03:40.351735+0000","last_unstale":"2024-09-17T04:03:41.333644+0000","last_undegraded":"2024-09-17T04:03:41.333644+0000","last_fullsized":"2024-09-17T04:03:41.333644+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:16.809754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"202'4","reported_seq":19,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497332+0000","last_change":"2024-09-17T04:03:39.331554+0000","last_active":"2024-09-17T04:03:41.497332+0000","last_peered":"2024-09-17T04:03:41.497332+0000","last_clean":"2024-09-17T04:03:41.497332+0000","last_became_active":"2024-09-17T04:03:39.331212+0000","last_became_peered":"2024-09-17T04:03:39.331212+0000","last_unstale":"2024-09-17T04:03:41.497332+0000","last_undegraded":"2024-09-17T04:03:41.497332+0000","last_fullsized":"2024-09-17T04:03:41.497332+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:16.193043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":446,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521595+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:03:43.521595+0000","last_peered":"2024-09-17T04:03:43.521595+0000","last_clean":"2024-09-17T04:03:43.521595+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:03:43.521595+0000","last_undegraded":"2024-09-17T04:03:43.521595+0000","last_fullsized":"2024-09-17T04:03:43.521595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52320,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319520,"statfs":{"total":96636764160,"available":96583188480,"internally_reserved":0,"allocated":3158016,"data_stored":1716897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29528,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342312,"statfs":{"total":96636764160,"available":96606527488,"internally_reserved":0,"allocated":2560000,"data_stored":1126459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509616,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509614,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55900,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315940,"statfs":{"total":96636764160,"available":96579522560,"internally_reserved":0,"allocated":2564096,"data_stored":1127041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509616,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51740,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320100,"statfs":{"total":96636764160,"available":96583782400,"internally_reserved":0,"allocated":2564096,"data_stored":1126549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509616,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542318,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1127063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542319,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3158016,"data_stored":1716861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:03:48.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:03:48.818 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:03:49.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 -- 172.21.15.129:0/853669065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc8160cf0 msgr2=0x7f8cc80d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 --2- 172.21.15.129:0/853669065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc8160cf0 0x7f8cc80d02c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009830 tx=0x7f8cb402a480 comp rx=0 tx=0).stop 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 -- 172.21.15.129:0/853669065 shutdown_connections 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 --2- 172.21.15.129:0/853669065 >> v2:172.21.15.181:3300/0 conn(0x7f8cc80d0800 0x7f8cc815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 --2- 172.21.15.129:0/853669065 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc8160cf0 0x7f8cc80d02c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.961+0000 7f8cc7200640 1 -- 172.21.15.129:0/853669065 >> 172.21.15.129:0/853669065 conn(0x7f8cc806cfc0 msgr2=0x7f8cc806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.962+0000 7f8cc7200640 1 -- 172.21.15.129:0/853669065 shutdown_connections 2024-09-17T04:03:49.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.962+0000 7f8cc7200640 1 -- 172.21.15.129:0/853669065 wait complete. 2024-09-17T04:03:49.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.964+0000 7f8cc7200640 1 Processor -- start 2024-09-17T04:03:49.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.964+0000 7f8cc7200640 1 -- start start 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 0x7f8cc80a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cc8160cf0 0x7f8cc80a9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc815d4a0 con 0x7f8cc80cfb60 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc815d320 con 0x7f8cc8160cf0 2024-09-17T04:03:49.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.965+0000 7f8cc7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc815d620 con 0x7f8cc80d0800 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46138/0 (socket says 172.21.15.129:46138) 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 -- 172.21.15.129:0/3378373892 learned_addr learned my addr 172.21.15.129:0/3378373892 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5400640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 0x7f8cc80a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc6800640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cc8160cf0 0x7f8cc80a9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 -- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 msgr2=0x7f8cc80a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 0x7f8cc80a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 -- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cc8160cf0 msgr2=0x7f8cc80a9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cc8160cf0 0x7f8cc80a9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5e00640 1 -- 172.21.15.129:0/3378373892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc80aa460 con 0x7f8cc80cfb60 2024-09-17T04:03:49.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.966+0000 7f8cc5400640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 0x7f8cc80a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:49.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cc5e00640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c730 tx=0x7f8cb800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:49.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8006890 con 0x7f8cc80cfb60 2024-09-17T04:03:49.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8cb8006a30 con 0x7f8cc80cfb60 2024-09-17T04:03:49.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc80aa740 con 0x7f8cc80cfb60 2024-09-17T04:03:49.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8007030 con 0x7f8cc80cfb60 2024-09-17T04:03:49.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.967+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc80aad00 con 0x7f8cc80cfb60 2024-09-17T04:03:49.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.968+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc80b2950 con 0x7f8cc80cfb60 2024-09-17T04:03:49.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.969+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8cb80071d0 con 0x7f8cc80cfb60 2024-09-17T04:03:49.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.970+0000 7f8cbe000640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 0x7f8ca807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:49.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.970+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f8cb809a740 con 0x7f8cc80cfb60 2024-09-17T04:03:49.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.971+0000 7f8cc5400640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 0x7f8ca807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:49.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.972+0000 7f8cc5400640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 0x7f8ca807b0a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4034000 tx=0x7f8cb4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:49.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:49.975+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8cc80b2950 con 0x7f8cc80cfb60 2024-09-17T04:03:50.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.154+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8cc815cbd0 con 0x7f8ca8078b90 2024-09-17T04:03:50.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.159+0000 7f8cbe000640 1 -- 172.21.15.129:0/3378373892 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252458 (secure 0 0 0) 0x7f8cc815cbd0 con 0x7f8ca8078b90 2024-09-17T04:03:50.160 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:50.163 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:03:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 msgr2=0x7f8ca807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 0x7f8ca807b0a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4034000 tx=0x7f8cb4004730 comp rx=0 tx=0).stop 2024-09-17T04:03:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 msgr2=0x7f8cc81f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800c730 tx=0x7f8cb800cbf0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 shutdown_connections 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ca8078b90 0x7f8ca807b0a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8cc8160cf0 0x7f8cc80a9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8cc80d0800 0x7f8cc80a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 --2- 172.21.15.129:0/3378373892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8cc80cfb60 0x7f8cc81f6e40 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 >> 172.21.15.129:0/3378373892 conn(0x7f8cc806cfc0 msgr2=0x7f8cc806f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 shutdown_connections 2024-09-17T04:03:50.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.164+0000 7f8cc7200640 1 -- 172.21.15.129:0/3378373892 wait complete. 2024-09-17T04:03:50.186 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-09-17T04:03:48.537631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":379020,"kb_used_data":21224,"kb_used_omap":12,"kb_used_meta":357619,"kb_avail":754595700,"statfs":{"total":773094113280,"available":772705996800,"internally_reserved":0,"allocated":21733376,"data_stored":10197228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":366202416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.227438"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330963+0000","last_change":"2024-09-17T04:03:40.330095+0000","last_active":"2024-09-17T04:03:41.330963+0000","last_peered":"2024-09-17T04:03:41.330963+0000","last_clean":"2024-09-17T04:03:41.330963+0000","last_became_active":"2024-09-17T04:03:40.329761+0000","last_became_peered":"2024-09-17T04:03:40.329761+0000","last_unstale":"2024-09-17T04:03:41.330963+0000","last_undegraded":"2024-09-17T04:03:41.330963+0000","last_fullsized":"2024-09-17T04:03:41.330963+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:44:38.735272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330025+0000","last_change":"2024-09-17T04:03:40.340996+0000","last_active":"2024-09-17T04:03:41.330025+0000","last_peered":"2024-09-17T04:03:41.330025+0000","last_clean":"2024-09-17T04:03:41.330025+0000","last_became_active":"2024-09-17T04:03:40.340458+0000","last_became_peered":"2024-09-17T04:03:40.340458+0000","last_unstale":"2024-09-17T04:03:41.330025+0000","last_undegraded":"2024-09-17T04:03:41.330025+0000","last_fullsized":"2024-09-17T04:03:41.330025+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:17.195721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497110+0000","last_change":"2024-09-17T04:03:40.347883+0000","last_active":"2024-09-17T04:03:41.497110+0000","last_peered":"2024-09-17T04:03:41.497110+0000","last_clean":"2024-09-17T04:03:41.497110+0000","last_became_active":"2024-09-17T04:03:40.347571+0000","last_became_peered":"2024-09-17T04:03:40.347571+0000","last_unstale":"2024-09-17T04:03:41.497110+0000","last_undegraded":"2024-09-17T04:03:41.497110+0000","last_fullsized":"2024-09-17T04:03:41.497110+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:58.174465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332582+0000","last_change":"2024-09-17T04:03:40.332143+0000","last_active":"2024-09-17T04:03:41.332582+0000","last_peered":"2024-09-17T04:03:41.332582+0000","last_clean":"2024-09-17T04:03:41.332582+0000","last_became_active":"2024-09-17T04:03:40.331789+0000","last_became_peered":"2024-09-17T04:03:40.331789+0000","last_unstale":"2024-09-17T04:03:41.332582+0000","last_undegraded":"2024-09-17T04:03:41.332582+0000","last_fullsized":"2024-09-17T04:03:41.332582+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:06.672381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333497+0000","last_change":"2024-09-17T04:03:40.335355+0000","last_active":"2024-09-17T04:03:41.333497+0000","last_peered":"2024-09-17T04:03:41.333497+0000","last_clean":"2024-09-17T04:03:41.333497+0000","last_became_active":"2024-09-17T04:03:40.334994+0000","last_became_peered":"2024-09-17T04:03:40.334994+0000","last_unstale":"2024-09-17T04:03:41.333497+0000","last_undegraded":"2024-09-17T04:03:41.333497+0000","last_fullsized":"2024-09-17T04:03:41.333497+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:02.228911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332515+0000","last_change":"2024-09-17T04:03:40.355785+0000","last_active":"2024-09-17T04:03:41.332515+0000","last_peered":"2024-09-17T04:03:41.332515+0000","last_clean":"2024-09-17T04:03:41.332515+0000","last_became_active":"2024-09-17T04:03:40.355371+0000","last_became_peered":"2024-09-17T04:03:40.355371+0000","last_unstale":"2024-09-17T04:03:41.332515+0000","last_undegraded":"2024-09-17T04:03:41.332515+0000","last_fullsized":"2024-09-17T04:03:41.332515+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:38.226897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482089+0000","last_change":"2024-09-17T04:03:40.350322+0000","last_active":"2024-09-17T04:03:41.482089+0000","last_peered":"2024-09-17T04:03:41.482089+0000","last_clean":"2024-09-17T04:03:41.482089+0000","last_became_active":"2024-09-17T04:03:40.349953+0000","last_became_peered":"2024-09-17T04:03:40.349953+0000","last_unstale":"2024-09-17T04:03:41.482089+0000","last_undegraded":"2024-09-17T04:03:41.482089+0000","last_fullsized":"2024-09-17T04:03:41.482089+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:03:12.060737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741207+0000","last_change":"2024-09-17T04:03:40.334851+0000","last_active":"2024-09-17T04:03:42.741207+0000","last_peered":"2024-09-17T04:03:42.741207+0000","last_clean":"2024-09-17T04:03:42.741207+0000","last_became_active":"2024-09-17T04:03:40.331337+0000","last_became_peered":"2024-09-17T04:03:40.331337+0000","last_unstale":"2024-09-17T04:03:42.741207+0000","last_undegraded":"2024-09-17T04:03:42.741207+0000","last_fullsized":"2024-09-17T04:03:42.741207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:52.463737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521788+0000","last_change":"2024-09-17T04:03:40.349511+0000","last_active":"2024-09-17T04:03:43.521788+0000","last_peered":"2024-09-17T04:03:43.521788+0000","last_clean":"2024-09-17T04:03:43.521788+0000","last_became_active":"2024-09-17T04:03:40.349113+0000","last_became_peered":"2024-09-17T04:03:40.349113+0000","last_unstale":"2024-09-17T04:03:43.521788+0000","last_undegraded":"2024-09-17T04:03:43.521788+0000","last_fullsized":"2024-09-17T04:03:43.521788+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:50.920504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482695+0000","last_change":"2024-09-17T04:03:40.340480+0000","last_active":"2024-09-17T04:03:41.482695+0000","last_peered":"2024-09-17T04:03:41.482695+0000","last_clean":"2024-09-17T04:03:41.482695+0000","last_became_active":"2024-09-17T04:03:40.338521+0000","last_became_peered":"2024-09-17T04:03:40.338521+0000","last_unstale":"2024-09-17T04:03:41.482695+0000","last_undegraded":"2024-09-17T04:03:41.482695+0000","last_fullsized":"2024-09-17T04:03:41.482695+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:11:10.458690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410478+0000","last_change":"2024-09-17T04:03:40.348952+0000","last_active":"2024-09-17T04:03:42.410478+0000","last_peered":"2024-09-17T04:03:42.410478+0000","last_clean":"2024-09-17T04:03:42.410478+0000","last_became_active":"2024-09-17T04:03:40.348656+0000","last_became_peered":"2024-09-17T04:03:40.348656+0000","last_unstale":"2024-09-17T04:03:42.410478+0000","last_undegraded":"2024-09-17T04:03:42.410478+0000","last_fullsized":"2024-09-17T04:03:42.410478+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:47.744159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741493+0000","last_change":"2024-09-17T04:03:40.354695+0000","last_active":"2024-09-17T04:03:42.741493+0000","last_peered":"2024-09-17T04:03:42.741493+0000","last_clean":"2024-09-17T04:03:42.741493+0000","last_became_active":"2024-09-17T04:03:40.354274+0000","last_became_peered":"2024-09-17T04:03:40.354274+0000","last_unstale":"2024-09-17T04:03:42.741493+0000","last_undegraded":"2024-09-17T04:03:42.741493+0000","last_fullsized":"2024-09-17T04:03:42.741493+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:15.701377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482731+0000","last_change":"2024-09-17T04:03:40.350373+0000","last_active":"2024-09-17T04:03:41.482731+0000","last_peered":"2024-09-17T04:03:41.482731+0000","last_clean":"2024-09-17T04:03:41.482731+0000","last_became_active":"2024-09-17T04:03:40.350000+0000","last_became_peered":"2024-09-17T04:03:40.350000+0000","last_unstale":"2024-09-17T04:03:41.482731+0000","last_undegraded":"2024-09-17T04:03:41.482731+0000","last_fullsized":"2024-09-17T04:03:41.482731+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:29.430197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329902+0000","last_change":"2024-09-17T04:03:40.320319+0000","last_active":"2024-09-17T04:03:41.329902+0000","last_peered":"2024-09-17T04:03:41.329902+0000","last_clean":"2024-09-17T04:03:41.329902+0000","last_became_active":"2024-09-17T04:03:40.320061+0000","last_became_peered":"2024-09-17T04:03:40.320061+0000","last_unstale":"2024-09-17T04:03:41.329902+0000","last_undegraded":"2024-09-17T04:03:41.329902+0000","last_fullsized":"2024-09-17T04:03:41.329902+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:23.008632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521536+0000","last_change":"2024-09-17T04:03:40.344294+0000","last_active":"2024-09-17T04:03:43.521536+0000","last_peered":"2024-09-17T04:03:43.521536+0000","last_clean":"2024-09-17T04:03:43.521536+0000","last_became_active":"2024-09-17T04:03:40.343923+0000","last_became_peered":"2024-09-17T04:03:40.343923+0000","last_unstale":"2024-09-17T04:03:43.521536+0000","last_undegraded":"2024-09-17T04:03:43.521536+0000","last_fullsized":"2024-09-17T04:03:43.521536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:35.362812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496287+0000","last_change":"2024-09-17T04:03:40.328076+0000","last_active":"2024-09-17T04:03:41.496287+0000","last_peered":"2024-09-17T04:03:41.496287+0000","last_clean":"2024-09-17T04:03:41.496287+0000","last_became_active":"2024-09-17T04:03:40.327770+0000","last_became_peered":"2024-09-17T04:03:40.327770+0000","last_unstale":"2024-09-17T04:03:41.496287+0000","last_undegraded":"2024-09-17T04:03:41.496287+0000","last_fullsized":"2024-09-17T04:03:41.496287+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:16:40.277989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741536+0000","last_change":"2024-09-17T04:03:40.343695+0000","last_active":"2024-09-17T04:03:42.741536+0000","last_peered":"2024-09-17T04:03:42.741536+0000","last_clean":"2024-09-17T04:03:42.741536+0000","last_became_active":"2024-09-17T04:03:40.343413+0000","last_became_peered":"2024-09-17T04:03:40.343413+0000","last_unstale":"2024-09-17T04:03:42.741536+0000","last_undegraded":"2024-09-17T04:03:42.741536+0000","last_fullsized":"2024-09-17T04:03:42.741536+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:34:54.092079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330462+0000","last_change":"2024-09-17T04:03:40.334723+0000","last_active":"2024-09-17T04:03:41.330462+0000","last_peered":"2024-09-17T04:03:41.330462+0000","last_clean":"2024-09-17T04:03:41.330462+0000","last_became_active":"2024-09-17T04:03:40.334418+0000","last_became_peered":"2024-09-17T04:03:40.334418+0000","last_unstale":"2024-09-17T04:03:41.330462+0000","last_undegraded":"2024-09-17T04:03:41.330462+0000","last_fullsized":"2024-09-17T04:03:41.330462+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:35:00.430811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:40.351995+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:40.351649+0000","last_became_peered":"2024-09-17T04:03:40.351649+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:58:03.942253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496692+0000","last_change":"2024-09-17T04:03:40.319052+0000","last_active":"2024-09-17T04:03:41.496692+0000","last_peered":"2024-09-17T04:03:41.496692+0000","last_clean":"2024-09-17T04:03:41.496692+0000","last_became_active":"2024-09-17T04:03:40.318761+0000","last_became_peered":"2024-09-17T04:03:40.318761+0000","last_unstale":"2024-09-17T04:03:41.496692+0000","last_undegraded":"2024-09-17T04:03:41.496692+0000","last_fullsized":"2024-09-17T04:03:41.496692+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:33.494049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741813+0000","last_change":"2024-09-17T04:03:40.333277+0000","last_active":"2024-09-17T04:03:42.741813+0000","last_peered":"2024-09-17T04:03:42.741813+0000","last_clean":"2024-09-17T04:03:42.741813+0000","last_became_active":"2024-09-17T04:03:40.332882+0000","last_became_peered":"2024-09-17T04:03:40.332882+0000","last_unstale":"2024-09-17T04:03:42.741813+0000","last_undegraded":"2024-09-17T04:03:42.741813+0000","last_fullsized":"2024-09-17T04:03:42.741813+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:44.245808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482369+0000","last_change":"2024-09-17T04:03:40.341616+0000","last_active":"2024-09-17T04:03:41.482369+0000","last_peered":"2024-09-17T04:03:41.482369+0000","last_clean":"2024-09-17T04:03:41.482369+0000","last_became_active":"2024-09-17T04:03:40.341269+0000","last_became_peered":"2024-09-17T04:03:40.341269+0000","last_unstale":"2024-09-17T04:03:41.482369+0000","last_undegraded":"2024-09-17T04:03:41.482369+0000","last_fullsized":"2024-09-17T04:03:41.482369+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:28.676331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741352+0000","last_change":"2024-09-17T04:03:40.337777+0000","last_active":"2024-09-17T04:03:42.741352+0000","last_peered":"2024-09-17T04:03:42.741352+0000","last_clean":"2024-09-17T04:03:42.741352+0000","last_became_active":"2024-09-17T04:03:40.335850+0000","last_became_peered":"2024-09-17T04:03:40.335850+0000","last_unstale":"2024-09-17T04:03:42.741352+0000","last_undegraded":"2024-09-17T04:03:42.741352+0000","last_fullsized":"2024-09-17T04:03:42.741352+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:37.334408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329845+0000","last_change":"2024-09-17T04:03:40.344001+0000","last_active":"2024-09-17T04:03:41.329845+0000","last_peered":"2024-09-17T04:03:41.329845+0000","last_clean":"2024-09-17T04:03:41.329845+0000","last_became_active":"2024-09-17T04:03:40.339574+0000","last_became_peered":"2024-09-17T04:03:40.339574+0000","last_unstale":"2024-09-17T04:03:41.329845+0000","last_undegraded":"2024-09-17T04:03:41.329845+0000","last_fullsized":"2024-09-17T04:03:41.329845+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:03.778853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333133+0000","last_change":"2024-09-17T04:03:40.330798+0000","last_active":"2024-09-17T04:03:41.333133+0000","last_peered":"2024-09-17T04:03:41.333133+0000","last_clean":"2024-09-17T04:03:41.333133+0000","last_became_active":"2024-09-17T04:03:40.330363+0000","last_became_peered":"2024-09-17T04:03:40.330363+0000","last_unstale":"2024-09-17T04:03:41.333133+0000","last_undegraded":"2024-09-17T04:03:41.333133+0000","last_fullsized":"2024-09-17T04:03:41.333133+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:53.191430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482816+0000","last_change":"2024-09-17T04:03:40.350740+0000","last_active":"2024-09-17T04:03:41.482816+0000","last_peered":"2024-09-17T04:03:41.482816+0000","last_clean":"2024-09-17T04:03:41.482816+0000","last_became_active":"2024-09-17T04:03:40.350415+0000","last_became_peered":"2024-09-17T04:03:40.350415+0000","last_unstale":"2024-09-17T04:03:41.482816+0000","last_undegraded":"2024-09-17T04:03:41.482816+0000","last_fullsized":"2024-09-17T04:03:41.482816+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:04.588026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410543+0000","last_change":"2024-09-17T04:03:40.347688+0000","last_active":"2024-09-17T04:03:42.410543+0000","last_peered":"2024-09-17T04:03:42.410543+0000","last_clean":"2024-09-17T04:03:42.410543+0000","last_became_active":"2024-09-17T04:03:40.347381+0000","last_became_peered":"2024-09-17T04:03:40.347381+0000","last_unstale":"2024-09-17T04:03:42.410543+0000","last_undegraded":"2024-09-17T04:03:42.410543+0000","last_fullsized":"2024-09-17T04:03:42.410543+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:17:51.255876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330673+0000","last_change":"2024-09-17T04:03:40.356708+0000","last_active":"2024-09-17T04:03:41.330673+0000","last_peered":"2024-09-17T04:03:41.330673+0000","last_clean":"2024-09-17T04:03:41.330673+0000","last_became_active":"2024-09-17T04:03:40.352402+0000","last_became_peered":"2024-09-17T04:03:40.352402+0000","last_unstale":"2024-09-17T04:03:41.330673+0000","last_undegraded":"2024-09-17T04:03:41.330673+0000","last_fullsized":"2024-09-17T04:03:41.330673+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:03:19.133678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410325+0000","last_change":"2024-09-17T04:03:40.350667+0000","last_active":"2024-09-17T04:03:42.410325+0000","last_peered":"2024-09-17T04:03:42.410325+0000","last_clean":"2024-09-17T04:03:42.410325+0000","last_became_active":"2024-09-17T04:03:40.350312+0000","last_became_peered":"2024-09-17T04:03:40.350312+0000","last_unstale":"2024-09-17T04:03:42.410325+0000","last_undegraded":"2024-09-17T04:03:42.410325+0000","last_fullsized":"2024-09-17T04:03:42.410325+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:48:21.193199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333504+0000","last_change":"2024-09-17T04:03:40.331612+0000","last_active":"2024-09-17T04:03:41.333504+0000","last_peered":"2024-09-17T04:03:41.333504+0000","last_clean":"2024-09-17T04:03:41.333504+0000","last_became_active":"2024-09-17T04:03:40.329978+0000","last_became_peered":"2024-09-17T04:03:40.329978+0000","last_unstale":"2024-09-17T04:03:41.333504+0000","last_undegraded":"2024-09-17T04:03:41.333504+0000","last_fullsized":"2024-09-17T04:03:41.333504+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:46:48.029240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332513+0000","last_change":"2024-09-17T04:03:40.329393+0000","last_active":"2024-09-17T04:03:41.332513+0000","last_peered":"2024-09-17T04:03:41.332513+0000","last_clean":"2024-09-17T04:03:41.332513+0000","last_became_active":"2024-09-17T04:03:40.329011+0000","last_became_peered":"2024-09-17T04:03:40.329011+0000","last_unstale":"2024-09-17T04:03:41.332513+0000","last_undegraded":"2024-09-17T04:03:41.332513+0000","last_fullsized":"2024-09-17T04:03:41.332513+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:43:46.684973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329902+0000","last_change":"2024-09-17T04:03:40.328538+0000","last_active":"2024-09-17T04:03:41.329902+0000","last_peered":"2024-09-17T04:03:41.329902+0000","last_clean":"2024-09-17T04:03:41.329902+0000","last_became_active":"2024-09-17T04:03:40.328220+0000","last_became_peered":"2024-09-17T04:03:40.328220+0000","last_unstale":"2024-09-17T04:03:41.329902+0000","last_undegraded":"2024-09-17T04:03:41.329902+0000","last_fullsized":"2024-09-17T04:03:41.329902+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:07.783912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497139+0000","last_change":"2024-09-17T04:03:40.334720+0000","last_active":"2024-09-17T04:03:41.497139+0000","last_peered":"2024-09-17T04:03:41.497139+0000","last_clean":"2024-09-17T04:03:41.497139+0000","last_became_active":"2024-09-17T04:03:40.334427+0000","last_became_peered":"2024-09-17T04:03:40.334427+0000","last_unstale":"2024-09-17T04:03:41.497139+0000","last_undegraded":"2024-09-17T04:03:41.497139+0000","last_fullsized":"2024-09-17T04:03:41.497139+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:29.796643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482954+0000","last_change":"2024-09-17T04:03:39.330078+0000","last_active":"2024-09-17T04:03:41.482954+0000","last_peered":"2024-09-17T04:03:41.482954+0000","last_clean":"2024-09-17T04:03:41.482954+0000","last_became_active":"2024-09-17T04:03:39.329766+0000","last_became_peered":"2024-09-17T04:03:39.329766+0000","last_unstale":"2024-09-17T04:03:41.482954+0000","last_undegraded":"2024-09-17T04:03:41.482954+0000","last_fullsized":"2024-09-17T04:03:41.482954+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:58:11.195328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410186+0000","last_change":"2024-09-17T04:03:40.351683+0000","last_active":"2024-09-17T04:03:42.410186+0000","last_peered":"2024-09-17T04:03:42.410186+0000","last_clean":"2024-09-17T04:03:42.410186+0000","last_became_active":"2024-09-17T04:03:40.351288+0000","last_became_peered":"2024-09-17T04:03:40.351288+0000","last_unstale":"2024-09-17T04:03:42.410186+0000","last_undegraded":"2024-09-17T04:03:42.410186+0000","last_fullsized":"2024-09-17T04:03:42.410186+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:11:57.454665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497043+0000","last_change":"2024-09-17T04:03:39.322960+0000","last_active":"2024-09-17T04:03:41.497043+0000","last_peered":"2024-09-17T04:03:41.497043+0000","last_clean":"2024-09-17T04:03:41.497043+0000","last_became_active":"2024-09-17T04:03:39.322704+0000","last_became_peered":"2024-09-17T04:03:39.322704+0000","last_unstale":"2024-09-17T04:03:41.497043+0000","last_undegraded":"2024-09-17T04:03:41.497043+0000","last_fullsized":"2024-09-17T04:03:41.497043+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:19.046905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741839+0000","last_change":"2024-09-17T04:03:40.343840+0000","last_active":"2024-09-17T04:03:42.741839+0000","last_peered":"2024-09-17T04:03:42.741839+0000","last_clean":"2024-09-17T04:03:42.741839+0000","last_became_active":"2024-09-17T04:03:40.343539+0000","last_became_peered":"2024-09-17T04:03:40.343539+0000","last_unstale":"2024-09-17T04:03:42.741839+0000","last_undegraded":"2024-09-17T04:03:42.741839+0000","last_fullsized":"2024-09-17T04:03:42.741839+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:04.501121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496914+0000","last_change":"2024-09-17T04:03:39.320734+0000","last_active":"2024-09-17T04:03:41.496914+0000","last_peered":"2024-09-17T04:03:41.496914+0000","last_clean":"2024-09-17T04:03:41.496914+0000","last_became_active":"2024-09-17T04:03:39.320431+0000","last_became_peered":"2024-09-17T04:03:39.320431+0000","last_unstale":"2024-09-17T04:03:41.496914+0000","last_undegraded":"2024-09-17T04:03:41.496914+0000","last_fullsized":"2024-09-17T04:03:41.496914+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:26:05.564947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410125+0000","last_change":"2024-09-17T04:03:40.335363+0000","last_active":"2024-09-17T04:03:42.410125+0000","last_peered":"2024-09-17T04:03:42.410125+0000","last_clean":"2024-09-17T04:03:42.410125+0000","last_became_active":"2024-09-17T04:03:40.333333+0000","last_became_peered":"2024-09-17T04:03:40.333333+0000","last_unstale":"2024-09-17T04:03:42.410125+0000","last_undegraded":"2024-09-17T04:03:42.410125+0000","last_fullsized":"2024-09-17T04:03:42.410125+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:35.331891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332477+0000","last_change":"2024-09-17T04:03:39.329538+0000","last_active":"2024-09-17T04:03:41.332477+0000","last_peered":"2024-09-17T04:03:41.332477+0000","last_clean":"2024-09-17T04:03:41.332477+0000","last_became_active":"2024-09-17T04:03:39.329257+0000","last_became_peered":"2024-09-17T04:03:39.329257+0000","last_unstale":"2024-09-17T04:03:41.332477+0000","last_undegraded":"2024-09-17T04:03:41.332477+0000","last_fullsized":"2024-09-17T04:03:41.332477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:56:03.999758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521725+0000","last_change":"2024-09-17T04:03:40.344266+0000","last_active":"2024-09-17T04:03:43.521725+0000","last_peered":"2024-09-17T04:03:43.521725+0000","last_clean":"2024-09-17T04:03:43.521725+0000","last_became_active":"2024-09-17T04:03:40.343914+0000","last_became_peered":"2024-09-17T04:03:40.343914+0000","last_unstale":"2024-09-17T04:03:43.521725+0000","last_undegraded":"2024-09-17T04:03:43.521725+0000","last_fullsized":"2024-09-17T04:03:43.521725+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:49.909796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330144+0000","last_change":"2024-09-17T04:03:39.321662+0000","last_active":"2024-09-17T04:03:41.330144+0000","last_peered":"2024-09-17T04:03:41.330144+0000","last_clean":"2024-09-17T04:03:41.330144+0000","last_became_active":"2024-09-17T04:03:39.321278+0000","last_became_peered":"2024-09-17T04:03:39.321278+0000","last_unstale":"2024-09-17T04:03:41.330144+0000","last_undegraded":"2024-09-17T04:03:41.330144+0000","last_fullsized":"2024-09-17T04:03:41.330144+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:11.779647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496290+0000","last_change":"2024-09-17T04:03:40.331538+0000","last_active":"2024-09-17T04:03:41.496290+0000","last_peered":"2024-09-17T04:03:41.496290+0000","last_clean":"2024-09-17T04:03:41.496290+0000","last_became_active":"2024-09-17T04:03:40.331210+0000","last_became_peered":"2024-09-17T04:03:40.331210+0000","last_unstale":"2024-09-17T04:03:41.496290+0000","last_undegraded":"2024-09-17T04:03:41.496290+0000","last_fullsized":"2024-09-17T04:03:41.496290+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:00.716824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482232+0000","last_change":"2024-09-17T04:03:39.332398+0000","last_active":"2024-09-17T04:03:41.482232+0000","last_peered":"2024-09-17T04:03:41.482232+0000","last_clean":"2024-09-17T04:03:41.482232+0000","last_became_active":"2024-09-17T04:03:39.331945+0000","last_became_peered":"2024-09-17T04:03:39.331945+0000","last_unstale":"2024-09-17T04:03:41.482232+0000","last_undegraded":"2024-09-17T04:03:41.482232+0000","last_fullsized":"2024-09-17T04:03:41.482232+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:50.951852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329863+0000","last_change":"2024-09-17T04:03:40.334437+0000","last_active":"2024-09-17T04:03:41.329863+0000","last_peered":"2024-09-17T04:03:41.329863+0000","last_clean":"2024-09-17T04:03:41.329863+0000","last_became_active":"2024-09-17T04:03:40.334121+0000","last_became_peered":"2024-09-17T04:03:40.334121+0000","last_unstale":"2024-09-17T04:03:41.329863+0000","last_undegraded":"2024-09-17T04:03:41.329863+0000","last_fullsized":"2024-09-17T04:03:41.329863+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:12:54.843913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496562+0000","last_change":"2024-09-17T04:03:39.317105+0000","last_active":"2024-09-17T04:03:41.496562+0000","last_peered":"2024-09-17T04:03:41.496562+0000","last_clean":"2024-09-17T04:03:41.496562+0000","last_became_active":"2024-09-17T04:03:39.316779+0000","last_became_peered":"2024-09-17T04:03:39.316779+0000","last_unstale":"2024-09-17T04:03:41.496562+0000","last_undegraded":"2024-09-17T04:03:41.496562+0000","last_fullsized":"2024-09-17T04:03:41.496562+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:11.069998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330936+0000","last_change":"2024-09-17T04:03:40.339195+0000","last_active":"2024-09-17T04:03:41.330936+0000","last_peered":"2024-09-17T04:03:41.330936+0000","last_clean":"2024-09-17T04:03:41.330936+0000","last_became_active":"2024-09-17T04:03:40.336174+0000","last_became_peered":"2024-09-17T04:03:40.336174+0000","last_unstale":"2024-09-17T04:03:41.330936+0000","last_undegraded":"2024-09-17T04:03:41.330936+0000","last_fullsized":"2024-09-17T04:03:41.330936+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:18:54.704094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741594+0000","last_change":"2024-09-17T04:03:39.326124+0000","last_active":"2024-09-17T04:03:42.741594+0000","last_peered":"2024-09-17T04:03:42.741594+0000","last_clean":"2024-09-17T04:03:42.741594+0000","last_became_active":"2024-09-17T04:03:39.325797+0000","last_became_peered":"2024-09-17T04:03:39.325797+0000","last_unstale":"2024-09-17T04:03:42.741594+0000","last_undegraded":"2024-09-17T04:03:42.741594+0000","last_fullsized":"2024-09-17T04:03:42.741594+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:01:04.499200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410736+0000","last_change":"2024-09-17T04:03:40.351885+0000","last_active":"2024-09-17T04:03:42.410736+0000","last_peered":"2024-09-17T04:03:42.410736+0000","last_clean":"2024-09-17T04:03:42.410736+0000","last_became_active":"2024-09-17T04:03:40.351494+0000","last_became_peered":"2024-09-17T04:03:40.351494+0000","last_unstale":"2024-09-17T04:03:42.410736+0000","last_undegraded":"2024-09-17T04:03:42.410736+0000","last_fullsized":"2024-09-17T04:03:42.410736+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:34:13.184588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497062+0000","last_change":"2024-09-17T04:03:39.328133+0000","last_active":"2024-09-17T04:03:41.497062+0000","last_peered":"2024-09-17T04:03:41.497062+0000","last_clean":"2024-09-17T04:03:41.497062+0000","last_became_active":"2024-09-17T04:03:39.327863+0000","last_became_peered":"2024-09-17T04:03:39.327863+0000","last_unstale":"2024-09-17T04:03:41.497062+0000","last_undegraded":"2024-09-17T04:03:41.497062+0000","last_fullsized":"2024-09-17T04:03:41.497062+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:48:02.776175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496825+0000","last_change":"2024-09-17T04:03:40.345889+0000","last_active":"2024-09-17T04:03:41.496825+0000","last_peered":"2024-09-17T04:03:41.496825+0000","last_clean":"2024-09-17T04:03:41.496825+0000","last_became_active":"2024-09-17T04:03:40.345589+0000","last_became_peered":"2024-09-17T04:03:40.345589+0000","last_unstale":"2024-09-17T04:03:41.496825+0000","last_undegraded":"2024-09-17T04:03:41.496825+0000","last_fullsized":"2024-09-17T04:03:41.496825+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:20.025729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"202'3","reported_seq":18,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330142+0000","last_change":"2024-09-17T04:03:39.331097+0000","last_active":"2024-09-17T04:03:41.330142+0000","last_peered":"2024-09-17T04:03:41.330142+0000","last_clean":"2024-09-17T04:03:41.330142+0000","last_became_active":"2024-09-17T04:03:39.330131+0000","last_became_peered":"2024-09-17T04:03:39.330131+0000","last_unstale":"2024-09-17T04:03:41.330142+0000","last_undegraded":"2024-09-17T04:03:41.330142+0000","last_fullsized":"2024-09-17T04:03:41.330142+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482540+0000","last_change":"2024-09-17T04:03:40.332717+0000","last_active":"2024-09-17T04:03:41.482540+0000","last_peered":"2024-09-17T04:03:41.482540+0000","last_clean":"2024-09-17T04:03:41.482540+0000","last_became_active":"2024-09-17T04:03:40.332421+0000","last_became_peered":"2024-09-17T04:03:40.332421+0000","last_unstale":"2024-09-17T04:03:41.482540+0000","last_undegraded":"2024-09-17T04:03:41.482540+0000","last_fullsized":"2024-09-17T04:03:41.482540+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:42.530742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"202'6","reported_seq":21,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330379+0000","last_change":"2024-09-17T04:03:39.329612+0000","last_active":"2024-09-17T04:03:41.330379+0000","last_peered":"2024-09-17T04:03:41.330379+0000","last_clean":"2024-09-17T04:03:41.330379+0000","last_became_active":"2024-09-17T04:03:39.329342+0000","last_became_peered":"2024-09-17T04:03:39.329342+0000","last_unstale":"2024-09-17T04:03:41.330379+0000","last_undegraded":"2024-09-17T04:03:41.330379+0000","last_fullsized":"2024-09-17T04:03:41.330379+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:51.187815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482365+0000","last_change":"2024-09-17T04:03:40.338842+0000","last_active":"2024-09-17T04:03:41.482365+0000","last_peered":"2024-09-17T04:03:41.482365+0000","last_clean":"2024-09-17T04:03:41.482365+0000","last_became_active":"2024-09-17T04:03:40.338449+0000","last_became_peered":"2024-09-17T04:03:40.338449+0000","last_unstale":"2024-09-17T04:03:41.482365+0000","last_undegraded":"2024-09-17T04:03:41.482365+0000","last_fullsized":"2024-09-17T04:03:41.482365+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:57:44.963272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330165+0000","last_change":"2024-09-17T04:03:39.324560+0000","last_active":"2024-09-17T04:03:41.330165+0000","last_peered":"2024-09-17T04:03:41.330165+0000","last_clean":"2024-09-17T04:03:41.330165+0000","last_became_active":"2024-09-17T04:03:39.324206+0000","last_became_peered":"2024-09-17T04:03:39.324206+0000","last_unstale":"2024-09-17T04:03:41.330165+0000","last_undegraded":"2024-09-17T04:03:41.330165+0000","last_fullsized":"2024-09-17T04:03:41.330165+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:15:21.064030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521913+0000","last_change":"2024-09-17T04:03:40.344849+0000","last_active":"2024-09-17T04:03:43.521913+0000","last_peered":"2024-09-17T04:03:43.521913+0000","last_clean":"2024-09-17T04:03:43.521913+0000","last_became_active":"2024-09-17T04:03:40.344173+0000","last_became_peered":"2024-09-17T04:03:40.344173+0000","last_unstale":"2024-09-17T04:03:43.521913+0000","last_undegraded":"2024-09-17T04:03:43.521913+0000","last_fullsized":"2024-09-17T04:03:43.521913+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:12.072676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496514+0000","last_change":"2024-09-17T04:03:39.327456+0000","last_active":"2024-09-17T04:03:41.496514+0000","last_peered":"2024-09-17T04:03:41.496514+0000","last_clean":"2024-09-17T04:03:41.496514+0000","last_became_active":"2024-09-17T04:03:39.326151+0000","last_became_peered":"2024-09-17T04:03:39.326151+0000","last_unstale":"2024-09-17T04:03:41.496514+0000","last_undegraded":"2024-09-17T04:03:41.496514+0000","last_fullsized":"2024-09-17T04:03:41.496514+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:48.907118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332322+0000","last_change":"2024-09-17T04:03:40.347714+0000","last_active":"2024-09-17T04:03:41.332322+0000","last_peered":"2024-09-17T04:03:41.332322+0000","last_clean":"2024-09-17T04:03:41.332322+0000","last_became_active":"2024-09-17T04:03:40.347413+0000","last_became_peered":"2024-09-17T04:03:40.347413+0000","last_unstale":"2024-09-17T04:03:41.332322+0000","last_undegraded":"2024-09-17T04:03:41.332322+0000","last_fullsized":"2024-09-17T04:03:41.332322+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:16.273359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482297+0000","last_change":"2024-09-17T04:03:39.330613+0000","last_active":"2024-09-17T04:03:41.482297+0000","last_peered":"2024-09-17T04:03:41.482297+0000","last_clean":"2024-09-17T04:03:41.482297+0000","last_became_active":"2024-09-17T04:03:39.330283+0000","last_became_peered":"2024-09-17T04:03:39.330283+0000","last_unstale":"2024-09-17T04:03:41.482297+0000","last_undegraded":"2024-09-17T04:03:41.482297+0000","last_fullsized":"2024-09-17T04:03:41.482297+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:46.502036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482092+0000","last_change":"2024-09-17T04:03:40.341659+0000","last_active":"2024-09-17T04:03:41.482092+0000","last_peered":"2024-09-17T04:03:41.482092+0000","last_clean":"2024-09-17T04:03:41.482092+0000","last_became_active":"2024-09-17T04:03:40.341344+0000","last_became_peered":"2024-09-17T04:03:40.341344+0000","last_unstale":"2024-09-17T04:03:41.482092+0000","last_undegraded":"2024-09-17T04:03:41.482092+0000","last_fullsized":"2024-09-17T04:03:41.482092+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:05:24.293693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"202'8","reported_seq":23,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.409812+0000","last_change":"2024-09-17T04:03:39.332639+0000","last_active":"2024-09-17T04:03:42.409812+0000","last_peered":"2024-09-17T04:03:42.409812+0000","last_clean":"2024-09-17T04:03:42.409812+0000","last_became_active":"2024-09-17T04:03:39.332409+0000","last_became_peered":"2024-09-17T04:03:39.332409+0000","last_unstale":"2024-09-17T04:03:42.409812+0000","last_undegraded":"2024-09-17T04:03:42.409812+0000","last_fullsized":"2024-09-17T04:03:42.409812+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:58:03.585348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333079+0000","last_change":"2024-09-17T04:03:40.332345+0000","last_active":"2024-09-17T04:03:41.333079+0000","last_peered":"2024-09-17T04:03:41.333079+0000","last_clean":"2024-09-17T04:03:41.333079+0000","last_became_active":"2024-09-17T04:03:40.332029+0000","last_became_peered":"2024-09-17T04:03:40.332029+0000","last_unstale":"2024-09-17T04:03:41.333079+0000","last_undegraded":"2024-09-17T04:03:41.333079+0000","last_fullsized":"2024-09-17T04:03:41.333079+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:57.501108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482664+0000","last_change":"2024-09-17T04:03:39.333660+0000","last_active":"2024-09-17T04:03:41.482664+0000","last_peered":"2024-09-17T04:03:41.482664+0000","last_clean":"2024-09-17T04:03:41.482664+0000","last_became_active":"2024-09-17T04:03:39.333330+0000","last_became_peered":"2024-09-17T04:03:39.333330+0000","last_unstale":"2024-09-17T04:03:41.482664+0000","last_undegraded":"2024-09-17T04:03:41.482664+0000","last_fullsized":"2024-09-17T04:03:41.482664+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:03.955054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497298+0000","last_change":"2024-09-17T04:03:40.328129+0000","last_active":"2024-09-17T04:03:41.497298+0000","last_peered":"2024-09-17T04:03:41.497298+0000","last_clean":"2024-09-17T04:03:41.497298+0000","last_became_active":"2024-09-17T04:03:40.327790+0000","last_became_peered":"2024-09-17T04:03:40.327790+0000","last_unstale":"2024-09-17T04:03:41.497298+0000","last_undegraded":"2024-09-17T04:03:41.497298+0000","last_fullsized":"2024-09-17T04:03:41.497298+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:43.906106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497076+0000","last_change":"2024-09-17T04:03:39.322064+0000","last_active":"2024-09-17T04:03:41.497076+0000","last_peered":"2024-09-17T04:03:41.497076+0000","last_clean":"2024-09-17T04:03:41.497076+0000","last_became_active":"2024-09-17T04:03:39.321750+0000","last_became_peered":"2024-09-17T04:03:39.321750+0000","last_unstale":"2024-09-17T04:03:41.497076+0000","last_undegraded":"2024-09-17T04:03:41.497076+0000","last_fullsized":"2024-09-17T04:03:41.497076+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:27:22.005954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521646+0000","last_change":"2024-09-17T04:03:40.340280+0000","last_active":"2024-09-17T04:03:43.521646+0000","last_peered":"2024-09-17T04:03:43.521646+0000","last_clean":"2024-09-17T04:03:43.521646+0000","last_became_active":"2024-09-17T04:03:40.339904+0000","last_became_peered":"2024-09-17T04:03:40.339904+0000","last_unstale":"2024-09-17T04:03:43.521646+0000","last_undegraded":"2024-09-17T04:03:43.521646+0000","last_fullsized":"2024-09-17T04:03:43.521646+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:59:09.142973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"202'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521655+0000","last_change":"2024-09-17T04:03:39.335875+0000","last_active":"2024-09-17T04:03:43.521655+0000","last_peered":"2024-09-17T04:03:43.521655+0000","last_clean":"2024-09-17T04:03:43.521655+0000","last_became_active":"2024-09-17T04:03:39.335580+0000","last_became_peered":"2024-09-17T04:03:39.335580+0000","last_unstale":"2024-09-17T04:03:43.521655+0000","last_undegraded":"2024-09-17T04:03:43.521655+0000","last_fullsized":"2024-09-17T04:03:43.521655+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:32:43.243129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741903+0000","last_change":"2024-09-17T04:03:40.342551+0000","last_active":"2024-09-17T04:03:42.741903+0000","last_peered":"2024-09-17T04:03:42.741903+0000","last_clean":"2024-09-17T04:03:42.741903+0000","last_became_active":"2024-09-17T04:03:40.339096+0000","last_became_peered":"2024-09-17T04:03:40.339096+0000","last_unstale":"2024-09-17T04:03:42.741903+0000","last_undegraded":"2024-09-17T04:03:42.741903+0000","last_fullsized":"2024-09-17T04:03:42.741903+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:42.841396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410300+0000","last_change":"2024-09-17T04:03:39.331133+0000","last_active":"2024-09-17T04:03:42.410300+0000","last_peered":"2024-09-17T04:03:42.410300+0000","last_clean":"2024-09-17T04:03:42.410300+0000","last_became_active":"2024-09-17T04:03:39.330849+0000","last_became_peered":"2024-09-17T04:03:39.330849+0000","last_unstale":"2024-09-17T04:03:42.410300+0000","last_undegraded":"2024-09-17T04:03:42.410300+0000","last_fullsized":"2024-09-17T04:03:42.410300+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:19.933450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741382+0000","last_change":"2024-09-17T04:03:40.339612+0000","last_active":"2024-09-17T04:03:42.741382+0000","last_peered":"2024-09-17T04:03:42.741382+0000","last_clean":"2024-09-17T04:03:42.741382+0000","last_became_active":"2024-09-17T04:03:40.339297+0000","last_became_peered":"2024-09-17T04:03:40.339297+0000","last_unstale":"2024-09-17T04:03:42.741382+0000","last_undegraded":"2024-09-17T04:03:42.741382+0000","last_fullsized":"2024-09-17T04:03:42.741382+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:00:14.950266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496646+0000","last_change":"2024-09-17T04:03:39.334073+0000","last_active":"2024-09-17T04:03:41.496646+0000","last_peered":"2024-09-17T04:03:41.496646+0000","last_clean":"2024-09-17T04:03:41.496646+0000","last_became_active":"2024-09-17T04:03:39.333739+0000","last_became_peered":"2024-09-17T04:03:39.333739+0000","last_unstale":"2024-09-17T04:03:41.496646+0000","last_undegraded":"2024-09-17T04:03:41.496646+0000","last_fullsized":"2024-09-17T04:03:41.496646+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:34.360134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.330131+0000","last_change":"2024-09-17T04:03:40.337036+0000","last_active":"2024-09-17T04:03:41.330131+0000","last_peered":"2024-09-17T04:03:41.330131+0000","last_clean":"2024-09-17T04:03:41.330131+0000","last_became_active":"2024-09-17T04:03:40.336743+0000","last_became_peered":"2024-09-17T04:03:40.336743+0000","last_unstale":"2024-09-17T04:03:41.330131+0000","last_undegraded":"2024-09-17T04:03:41.330131+0000","last_fullsized":"2024-09-17T04:03:41.330131+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:45:34.712374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410122+0000","last_change":"2024-09-17T04:03:39.331198+0000","last_active":"2024-09-17T04:03:42.410122+0000","last_peered":"2024-09-17T04:03:42.410122+0000","last_clean":"2024-09-17T04:03:42.410122+0000","last_became_active":"2024-09-17T04:03:39.330893+0000","last_became_peered":"2024-09-17T04:03:39.330893+0000","last_unstale":"2024-09-17T04:03:42.410122+0000","last_undegraded":"2024-09-17T04:03:42.410122+0000","last_fullsized":"2024-09-17T04:03:42.410122+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:04:28.427519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410029+0000","last_change":"2024-09-17T04:03:40.347641+0000","last_active":"2024-09-17T04:03:42.410029+0000","last_peered":"2024-09-17T04:03:42.410029+0000","last_clean":"2024-09-17T04:03:42.410029+0000","last_became_active":"2024-09-17T04:03:40.347326+0000","last_became_peered":"2024-09-17T04:03:40.347326+0000","last_unstale":"2024-09-17T04:03:42.410029+0000","last_undegraded":"2024-09-17T04:03:42.410029+0000","last_fullsized":"2024-09-17T04:03:42.410029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:56.328537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521658+0000","last_change":"2024-09-17T04:03:39.335709+0000","last_active":"2024-09-17T04:03:43.521658+0000","last_peered":"2024-09-17T04:03:43.521658+0000","last_clean":"2024-09-17T04:03:43.521658+0000","last_became_active":"2024-09-17T04:03:39.334739+0000","last_became_peered":"2024-09-17T04:03:39.334739+0000","last_unstale":"2024-09-17T04:03:43.521658+0000","last_undegraded":"2024-09-17T04:03:43.521658+0000","last_fullsized":"2024-09-17T04:03:43.521658+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:10.959670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741466+0000","last_change":"2024-09-17T04:03:40.335288+0000","last_active":"2024-09-17T04:03:42.741466+0000","last_peered":"2024-09-17T04:03:42.741466+0000","last_clean":"2024-09-17T04:03:42.741466+0000","last_became_active":"2024-09-17T04:03:40.334907+0000","last_became_peered":"2024-09-17T04:03:40.334907+0000","last_unstale":"2024-09-17T04:03:42.741466+0000","last_undegraded":"2024-09-17T04:03:42.741466+0000","last_fullsized":"2024-09-17T04:03:42.741466+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:15.982851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482130+0000","last_change":"2024-09-17T04:03:39.331744+0000","last_active":"2024-09-17T04:03:41.482130+0000","last_peered":"2024-09-17T04:03:41.482130+0000","last_clean":"2024-09-17T04:03:41.482130+0000","last_became_active":"2024-09-17T04:03:39.330827+0000","last_became_peered":"2024-09-17T04:03:39.330827+0000","last_unstale":"2024-09-17T04:03:41.482130+0000","last_undegraded":"2024-09-17T04:03:41.482130+0000","last_fullsized":"2024-09-17T04:03:41.482130+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:57.121963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482922+0000","last_change":"2024-09-17T04:03:40.334111+0000","last_active":"2024-09-17T04:03:41.482922+0000","last_peered":"2024-09-17T04:03:41.482922+0000","last_clean":"2024-09-17T04:03:41.482922+0000","last_became_active":"2024-09-17T04:03:40.333244+0000","last_became_peered":"2024-09-17T04:03:40.333244+0000","last_unstale":"2024-09-17T04:03:41.482922+0000","last_undegraded":"2024-09-17T04:03:41.482922+0000","last_fullsized":"2024-09-17T04:03:41.482922+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:53:20.936128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"202'2","reported_seq":17,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333140+0000","last_change":"2024-09-17T04:03:39.327400+0000","last_active":"2024-09-17T04:03:41.333140+0000","last_peered":"2024-09-17T04:03:41.333140+0000","last_clean":"2024-09-17T04:03:41.333140+0000","last_became_active":"2024-09-17T04:03:39.327045+0000","last_became_peered":"2024-09-17T04:03:39.327045+0000","last_unstale":"2024-09-17T04:03:41.333140+0000","last_undegraded":"2024-09-17T04:03:41.333140+0000","last_fullsized":"2024-09-17T04:03:41.333140+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:42.554312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.331177+0000","last_change":"2024-09-17T04:03:40.352872+0000","last_active":"2024-09-17T04:03:41.331177+0000","last_peered":"2024-09-17T04:03:41.331177+0000","last_clean":"2024-09-17T04:03:41.331177+0000","last_became_active":"2024-09-17T04:03:40.352511+0000","last_became_peered":"2024-09-17T04:03:40.352511+0000","last_unstale":"2024-09-17T04:03:41.331177+0000","last_undegraded":"2024-09-17T04:03:41.331177+0000","last_fullsized":"2024-09-17T04:03:41.331177+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:54.181921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"203'1","reported_seq":16,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333769+0000","last_change":"2024-09-17T04:03:39.332034+0000","last_active":"2024-09-17T04:03:41.333769+0000","last_peered":"2024-09-17T04:03:41.333769+0000","last_clean":"2024-09-17T04:03:41.333769+0000","last_became_active":"2024-09-17T04:03:39.331723+0000","last_became_peered":"2024-09-17T04:03:39.331723+0000","last_unstale":"2024-09-17T04:03:41.333769+0000","last_undegraded":"2024-09-17T04:03:41.333769+0000","last_fullsized":"2024-09-17T04:03:41.333769+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:01:32.008051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410209+0000","last_change":"2024-09-17T04:03:40.343426+0000","last_active":"2024-09-17T04:03:42.410209+0000","last_peered":"2024-09-17T04:03:42.410209+0000","last_clean":"2024-09-17T04:03:42.410209+0000","last_became_active":"2024-09-17T04:03:40.343092+0000","last_became_peered":"2024-09-17T04:03:40.343092+0000","last_unstale":"2024-09-17T04:03:42.410209+0000","last_undegraded":"2024-09-17T04:03:42.410209+0000","last_fullsized":"2024-09-17T04:03:42.410209+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:19.250495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521877+0000","last_change":"2024-09-17T04:03:39.335065+0000","last_active":"2024-09-17T04:03:43.521877+0000","last_peered":"2024-09-17T04:03:43.521877+0000","last_clean":"2024-09-17T04:03:43.521877+0000","last_became_active":"2024-09-17T04:03:39.334604+0000","last_became_peered":"2024-09-17T04:03:39.334604+0000","last_unstale":"2024-09-17T04:03:43.521877+0000","last_undegraded":"2024-09-17T04:03:43.521877+0000","last_fullsized":"2024-09-17T04:03:43.521877+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:07:06.943264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410698+0000","last_change":"2024-09-17T04:03:40.352477+0000","last_active":"2024-09-17T04:03:42.410698+0000","last_peered":"2024-09-17T04:03:42.410698+0000","last_clean":"2024-09-17T04:03:42.410698+0000","last_became_active":"2024-09-17T04:03:40.351366+0000","last_became_peered":"2024-09-17T04:03:40.351366+0000","last_unstale":"2024-09-17T04:03:42.410698+0000","last_undegraded":"2024-09-17T04:03:42.410698+0000","last_fullsized":"2024-09-17T04:03:42.410698+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:17:14.000165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.410341+0000","last_change":"2024-09-17T04:03:39.335924+0000","last_active":"2024-09-17T04:03:42.410341+0000","last_peered":"2024-09-17T04:03:42.410341+0000","last_clean":"2024-09-17T04:03:42.410341+0000","last_became_active":"2024-09-17T04:03:39.335629+0000","last_became_peered":"2024-09-17T04:03:39.335629+0000","last_unstale":"2024-09-17T04:03:42.410341+0000","last_undegraded":"2024-09-17T04:03:42.410341+0000","last_fullsized":"2024-09-17T04:03:42.410341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:26:52.490212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482687+0000","last_change":"2024-09-17T04:03:40.339516+0000","last_active":"2024-09-17T04:03:41.482687+0000","last_peered":"2024-09-17T04:03:41.482687+0000","last_clean":"2024-09-17T04:03:41.482687+0000","last_became_active":"2024-09-17T04:03:40.339040+0000","last_became_peered":"2024-09-17T04:03:40.339040+0000","last_unstale":"2024-09-17T04:03:41.482687+0000","last_undegraded":"2024-09-17T04:03:41.482687+0000","last_fullsized":"2024-09-17T04:03:41.482687+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:48:33.297830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.482477+0000","last_change":"2024-09-17T04:03:39.331355+0000","last_active":"2024-09-17T04:03:41.482477+0000","last_peered":"2024-09-17T04:03:41.482477+0000","last_clean":"2024-09-17T04:03:41.482477+0000","last_became_active":"2024-09-17T04:03:39.331003+0000","last_became_peered":"2024-09-17T04:03:39.331003+0000","last_unstale":"2024-09-17T04:03:41.482477+0000","last_undegraded":"2024-09-17T04:03:41.482477+0000","last_fullsized":"2024-09-17T04:03:41.482477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:52.147204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741734+0000","last_change":"2024-09-17T04:03:40.337242+0000","last_active":"2024-09-17T04:03:42.741734+0000","last_peered":"2024-09-17T04:03:42.741734+0000","last_clean":"2024-09-17T04:03:42.741734+0000","last_became_active":"2024-09-17T04:03:40.336866+0000","last_became_peered":"2024-09-17T04:03:40.336866+0000","last_unstale":"2024-09-17T04:03:42.741734+0000","last_undegraded":"2024-09-17T04:03:42.741734+0000","last_fullsized":"2024-09-17T04:03:42.741734+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:05:52.701613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.332623+0000","last_change":"2024-09-17T04:03:39.332196+0000","last_active":"2024-09-17T04:03:41.332623+0000","last_peered":"2024-09-17T04:03:41.332623+0000","last_clean":"2024-09-17T04:03:41.332623+0000","last_became_active":"2024-09-17T04:03:39.331897+0000","last_became_peered":"2024-09-17T04:03:39.331897+0000","last_unstale":"2024-09-17T04:03:41.332623+0000","last_undegraded":"2024-09-17T04:03:41.332623+0000","last_fullsized":"2024-09-17T04:03:41.332623+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:39.348568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:42.741362+0000","last_change":"2024-09-17T04:03:40.331229+0000","last_active":"2024-09-17T04:03:42.741362+0000","last_peered":"2024-09-17T04:03:42.741362+0000","last_clean":"2024-09-17T04:03:42.741362+0000","last_became_active":"2024-09-17T04:03:40.330857+0000","last_became_peered":"2024-09-17T04:03:40.330857+0000","last_unstale":"2024-09-17T04:03:42.741362+0000","last_undegraded":"2024-09-17T04:03:42.741362+0000","last_fullsized":"2024-09-17T04:03:42.741362+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:18.666966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496366+0000","last_change":"2024-09-17T04:03:39.329983+0000","last_active":"2024-09-17T04:03:41.496366+0000","last_peered":"2024-09-17T04:03:41.496366+0000","last_clean":"2024-09-17T04:03:41.496366+0000","last_became_active":"2024-09-17T04:03:39.325670+0000","last_became_peered":"2024-09-17T04:03:39.325670+0000","last_unstale":"2024-09-17T04:03:41.496366+0000","last_undegraded":"2024-09-17T04:03:41.496366+0000","last_fullsized":"2024-09-17T04:03:41.496366+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:13.252323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.329936+0000","last_change":"2024-09-17T04:03:40.337809+0000","last_active":"2024-09-17T04:03:41.329936+0000","last_peered":"2024-09-17T04:03:41.329936+0000","last_clean":"2024-09-17T04:03:41.329936+0000","last_became_active":"2024-09-17T04:03:40.337519+0000","last_became_peered":"2024-09-17T04:03:40.337519+0000","last_unstale":"2024-09-17T04:03:41.329936+0000","last_undegraded":"2024-09-17T04:03:41.329936+0000","last_fullsized":"2024-09-17T04:03:41.329936+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:20.049583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":15,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.496512+0000","last_change":"2024-09-17T04:03:39.332559+0000","last_active":"2024-09-17T04:03:41.496512+0000","last_peered":"2024-09-17T04:03:41.496512+0000","last_clean":"2024-09-17T04:03:41.496512+0000","last_became_active":"2024-09-17T04:03:39.332265+0000","last_became_peered":"2024-09-17T04:03:39.332265+0000","last_unstale":"2024-09-17T04:03:41.496512+0000","last_undegraded":"2024-09-17T04:03:41.496512+0000","last_fullsized":"2024-09-17T04:03:41.496512+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:49.917826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":13,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.333644+0000","last_change":"2024-09-17T04:03:40.352076+0000","last_active":"2024-09-17T04:03:41.333644+0000","last_peered":"2024-09-17T04:03:41.333644+0000","last_clean":"2024-09-17T04:03:41.333644+0000","last_became_active":"2024-09-17T04:03:40.351735+0000","last_became_peered":"2024-09-17T04:03:40.351735+0000","last_unstale":"2024-09-17T04:03:41.333644+0000","last_undegraded":"2024-09-17T04:03:41.333644+0000","last_fullsized":"2024-09-17T04:03:41.333644+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":201,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:39.295451+0000","last_clean_scrub_stamp":"2024-09-17T04:03:39.295451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:43:16.809754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"202'4","reported_seq":19,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:41.497332+0000","last_change":"2024-09-17T04:03:39.331554+0000","last_active":"2024-09-17T04:03:41.497332+0000","last_peered":"2024-09-17T04:03:41.497332+0000","last_clean":"2024-09-17T04:03:41.497332+0000","last_became_active":"2024-09-17T04:03:39.331212+0000","last_became_peered":"2024-09-17T04:03:39.331212+0000","last_unstale":"2024-09-17T04:03:41.497332+0000","last_undegraded":"2024-09-17T04:03:41.497332+0000","last_fullsized":"2024-09-17T04:03:41.497332+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":200,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:03:38.292773+0000","last_clean_scrub_stamp":"2024-09-17T04:03:38.292773+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:52:16.193043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":446,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-17T04:03:43.521595+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:03:43.521595+0000","last_peered":"2024-09-17T04:03:43.521595+0000","last_clean":"2024-09-17T04:03:43.521595+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:03:43.521595+0000","last_undegraded":"2024-09-17T04:03:43.521595+0000","last_fullsized":"2024-09-17T04:03:43.521595+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52320,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319520,"statfs":{"total":96636764160,"available":96583188480,"internally_reserved":0,"allocated":3158016,"data_stored":1716897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509617,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29528,"kb_used_data":2500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342312,"statfs":{"total":96636764160,"available":96606527488,"internally_reserved":0,"allocated":2560000,"data_stored":1126459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509616,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509614,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55900,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315940,"statfs":{"total":96636764160,"available":96579522560,"internally_reserved":0,"allocated":2564096,"data_stored":1127041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509616,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51740,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320100,"statfs":{"total":96636764160,"available":96583782400,"internally_reserved":0,"allocated":2564096,"data_stored":1126549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509616,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":2580480,"data_stored":1128195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542319,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51744,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320096,"statfs":{"total":96636764160,"available":96583778304,"internally_reserved":0,"allocated":2568192,"data_stored":1127063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542320,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34276,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337564,"statfs":{"total":96636764160,"available":96601665536,"internally_reserved":0,"allocated":3158016,"data_stored":1716861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:03:50.190 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:03:50.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 -- 172.21.15.129:0/998720458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe4003a00 msgr2=0x7f1fe4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 --2- 172.21.15.129:0/998720458 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fe4003a00 0x7f1fe4003e70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000c4a0 tx=0x7f1fe4005830 comp rx=0 tx=0).stop 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2002400640 1 -- 172.21.15.129:0/998720458 reap_dead start 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 -- 172.21.15.129:0/998720458 shutdown_connections 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 -- 172.21.15.129:0/998720458 >> 172.21.15.129:0/998720458 conn(0x7f1ffc06d7a0 msgr2=0x7f1ffc0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 -- 172.21.15.129:0/998720458 shutdown_connections 2024-09-17T04:03:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.456+0000 7f2004400640 1 -- 172.21.15.129:0/998720458 wait complete. 2024-09-17T04:03:50.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.459+0000 7f2004400640 1 Processor -- start 2024-09-17T04:03:50.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.459+0000 7f2004400640 1 -- start start 2024-09-17T04:03:50.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe4003a00 0x7f1ffc1543f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ffc0b0f30 0x7f1ffc09b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc06f620 con 0x7f1ffc09ce50 2024-09-17T04:03:50.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc162a20 con 0x7f1ffc0b0f30 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.460+0000 7f2004400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc06f7a0 con 0x7f1fe4003a00 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46168/0 (socket says 172.21.15.129:46168) 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 -- 172.21.15.129:0/4115070910 learned_addr learned my addr 172.21.15.129:0/4115070910 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001a00640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe4003a00 0x7f1ffc1543f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2002400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ffc0b0f30 0x7f1ffc09b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 -- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe4003a00 msgr2=0x7f1ffc1543f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe4003a00 0x7f1ffc1543f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 -- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ffc0b0f30 msgr2=0x7f1ffc09b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ffc0b0f30 0x7f1ffc09b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 -- 172.21.15.129:0/4115070910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc0b1340 con 0x7f1ffc09ce50 2024-09-17T04:03:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.461+0000 7f2001000640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8011b10 tx=0x7f1fe8009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.462+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe801e070 con 0x7f1ffc09ce50 2024-09-17T04:03:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.462+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1fe800fe40 con 0x7f1ffc09ce50 2024-09-17T04:03:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.462+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fe80042b0 con 0x7f1ffc09ce50 2024-09-17T04:03:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.462+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc1f93e0 con 0x7f1ffc09ce50 2024-09-17T04:03:50.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.462+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc1f99a0 con 0x7f1ffc09ce50 2024-09-17T04:03:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.463+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc06f0e0 con 0x7f1ffc09ce50 2024-09-17T04:03:50.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.464+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1fe8021070 con 0x7f1ffc09ce50 2024-09-17T04:03:50.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.465+0000 7f1ff9600640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 0x7f1fdc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.465+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f1fe809b760 con 0x7f1ffc09ce50 2024-09-17T04:03:50.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.465+0000 7f2001a00640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 0x7f1fdc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.466+0000 7f2001a00640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 0x7f1fdc07b0a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0033000 tx=0x7f1ff0038040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:50.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.469+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1ffc0b1340 con 0x7f1ffc09ce50 2024-09-17T04:03:50.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.720+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1ffc0b33c0 con 0x7f1ffc09ce50 2024-09-17T04:03:50.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.725+0000 7f1ff9600640 1 -- 172.21.15.129:0/4115070910 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) ==== 190+0+147 (secure 0 0 0) 0x7f1fe8063540 con 0x7f1ffc09ce50 2024-09-17T04:03:50.725 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:03:50.726 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:03:50.726 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:03:50.726 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:03:50.726 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:03:50.726 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 msgr2=0x7f1fdc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 0x7f1fdc07b0a0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0033000 tx=0x7f1ff0038040 comp rx=0 tx=0).stop 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 msgr2=0x7f1ffc0b09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8011b10 tx=0x7f1fe8009e40 comp rx=0 tx=0).stop 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 shutdown_connections 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1fdc078b90 0x7f1fdc07b0a0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ffc0b0f30 0x7f1ffc09b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1ffc09ce50 0x7f1ffc0b09f0 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 --2- 172.21.15.129:0/4115070910 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1fe4003a00 0x7f1ffc1543f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 >> 172.21.15.129:0/4115070910 conn(0x7f1ffc06d7a0 msgr2=0x7f1ffc1615f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:50.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 shutdown_connections 2024-09-17T04:03:50.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.728+0000 7f2004400640 1 -- 172.21.15.129:0/4115070910 wait complete. 2024-09-17T04:03:50.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:50.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 -- 172.21.15.129:0/1771891147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 msgr2=0x7f52f4004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 --2- 172.21.15.129:0/1771891147 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f52f4004c70 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f53100b5030 tx=0x7f52f4005830 comp rx=0 tx=0).stop 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5314e00640 1 -- 172.21.15.129:0/1771891147 reap_dead start 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 -- 172.21.15.129:0/1771891147 shutdown_connections 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 -- 172.21.15.129:0/1771891147 >> 172.21.15.129:0/1771891147 conn(0x7f531006cfc0 msgr2=0x7f531006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 -- 172.21.15.129:0/1771891147 shutdown_connections 2024-09-17T04:03:50.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.974+0000 7f5315800640 1 -- 172.21.15.129:0/1771891147 wait complete. 2024-09-17T04:03:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.976+0000 7f5315800640 1 Processor -- start 2024-09-17T04:03:50.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.977+0000 7f5315800640 1 -- start start 2024-09-17T04:03:50.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5310155380 0x7f5310155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f531015aa40 0x7f531015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f53100cf740 con 0x7f52f4004800 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f53100cf5c0 con 0x7f5310155380 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5315800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f53100cf8c0 con 0x7f531015aa40 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f530fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f530fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46176/0 (socket says 172.21.15.129:46176) 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f530fe00640 1 -- 172.21.15.129:0/1100051386 learned_addr learned my addr 172.21.15.129:0/1100051386 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f530f400640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5310155380 0x7f5310155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.978+0000 7f5314e00640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f531015aa40 0x7f531015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 -- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f531015aa40 msgr2=0x7f531015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f531015aa40 0x7f531015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 -- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5310155380 msgr2=0x7f5310155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5310155380 0x7f5310155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 -- 172.21.15.129:0/1100051386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531015b390 con 0x7f52f4004800 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f530fe00640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f52f8011ae0 tx=0x7f52f8009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f801e070 con 0x7f52f4004800 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f52f800fe70 con 0x7f52f4004800 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5310144210 con 0x7f52f4004800 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8004290 con 0x7f52f4004800 2024-09-17T04:03:50.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.979+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53101447d0 con 0x7f52f4004800 2024-09-17T04:03:50.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.980+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f531006a140 con 0x7f52f4004800 2024-09-17T04:03:50.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.981+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52f8021070 con 0x7f52f4004800 2024-09-17T04:03:50.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.982+0000 7f5307e00640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 0x7f52ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:50.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.983+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f52f809b7e0 con 0x7f52f4004800 2024-09-17T04:03:50.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.983+0000 7f530f400640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 0x7f52ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:50.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.984+0000 7f530f400640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 0x7f52ec07b0a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5300033000 tx=0x7f531015cbf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:50.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:50.987+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f531015b390 con 0x7f52f4004800 2024-09-17T04:03:51.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.205+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5310151df0 con 0x7f52f4004800 2024-09-17T04:03:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.206+0000 7f5307e00640 1 -- 172.21.15.129:0/1100051386 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5251 (secure 0 0 0) 0x7f52f8063680 con 0x7f52f4004800 2024-09-17T04:03:51.206 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:51.206 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":181,"btime":"2024-09-17T04:03:43:337368+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3964402375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3964402375},{"type":"v1","addr":"172.21.15.129:6833","nonce":3964402375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175},{"gid":8276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2137304743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2137304743},{"type":"v1","addr":"172.21.15.181:6836","nonce":2137304743}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176},{"gid":10434,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3759513659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3759513659},{"type":"v1","addr":"172.21.15.129:6835","nonce":3759513659}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:03:42.827055+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:03:51.206 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 181 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 msgr2=0x7f52ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 0x7f52ec07b0a0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5300033000 tx=0x7f531015cbf0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 msgr2=0x7f53101415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f52f8011ae0 tx=0x7f52f8009e90 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 shutdown_connections 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f52ec078b90 0x7f52ec07b0a0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f531015aa40 0x7f531015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5310155380 0x7f5310155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 --2- 172.21.15.129:0/1100051386 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f52f4004800 0x7f53101415e0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.208+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 >> 172.21.15.129:0/1100051386 conn(0x7f531006cfc0 msgr2=0x7f5310075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.209+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 shutdown_connections 2024-09-17T04:03:51.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.209+0000 7f5315800640 1 -- 172.21.15.129:0/1100051386 wait complete. 2024-09-17T04:03:51.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 181, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:03:42.827055+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10440}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:03:51.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:03:51.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 -- 172.21.15.129:0/745751255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0cd390 msgr2=0x7f3fbc1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 --2- 172.21.15.129:0/745751255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0cd390 0x7f3fbc1609c0 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0d1370 tx=0x7f3fb402aa30 comp rx=0 tx=0).stop 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 -- 172.21.15.129:0/745751255 shutdown_connections 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 --2- 172.21.15.129:0/745751255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0cd390 0x7f3fbc1609c0 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 --2- 172.21.15.129:0/745751255 >> v2:172.21.15.181:3300/0 conn(0x7f3fbc0705a0 0x7f3fbc0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 -- 172.21.15.129:0/745751255 >> 172.21.15.129:0/745751255 conn(0x7f3fbc06d410 msgr2=0x7f3fbc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 -- 172.21.15.129:0/745751255 shutdown_connections 2024-09-17T04:03:51.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.856+0000 7f3fc3200640 1 -- 172.21.15.129:0/745751255 wait complete. 2024-09-17T04:03:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.859+0000 7f3fc3200640 1 Processor -- start 2024-09-17T04:03:51.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.859+0000 7f3fc3200640 1 -- start start 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fbc06eed0 0x7f3fbc0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fbc0cb540 0x7f3fbc074000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc160250 con 0x7f3fbc0705a0 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc06f4f0 con 0x7f3fbc06eed0 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fc3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3fbc160f00 con 0x7f3fbc0cb540 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fbbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fbc06eed0 0x7f3fbc0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.860+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46192/0 (socket says 172.21.15.129:46192) 2024-09-17T04:03:51.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 -- 172.21.15.129:0/3770062517 learned_addr learned my addr 172.21.15.129:0/3770062517 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fc1200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fbc0cb540 0x7f3fbc074000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 -- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fbc0cb540 msgr2=0x7f3fbc074000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fbc0cb540 0x7f3fbc074000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 -- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fbc06eed0 msgr2=0x7f3fbc0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fbc06eed0 0x7f3fbc0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 -- 172.21.15.129:0/3770062517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc0cc870 con 0x7f3fbc0705a0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.861+0000 7f3fbb400640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b3c0 tx=0x7f3fa800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.862+0000 7f3fb3e00640 1 -- 172.21.15.129:0/3770062517 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8011830 con 0x7f3fbc0705a0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.862+0000 7f3fb3e00640 1 -- 172.21.15.129:0/3770062517 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fa8016070 con 0x7f3fbc0705a0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.862+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc159770 con 0x7f3fbc0705a0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.862+0000 7f3fb3e00640 1 -- 172.21.15.129:0/3770062517 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8011da0 con 0x7f3fbc0705a0 2024-09-17T04:03:51.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.862+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fbc1f5fa0 con 0x7f3fbc0705a0 2024-09-17T04:03:51.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.864+0000 7f3fb3e00640 1 -- 172.21.15.129:0/3770062517 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3fa801b460 con 0x7f3fbc0705a0 2024-09-17T04:03:51.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.866+0000 7f3fb3e00640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 0x7f3f9c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.866+0000 7f3fb3e00640 1 -- 172.21.15.129:0/3770062517 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f3fa809a740 con 0x7f3fbc0705a0 2024-09-17T04:03:51.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.866+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 0x7f3f9c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.866+0000 7f3fca288740 1 Processor -- start 2024-09-17T04:03:51.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.866+0000 7f3fca288740 1 -- start start 2024-09-17T04:03:51.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.867+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 0x7f3f9c07b0a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3fac004630 tx=0x7f3fac00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d09eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b17cff8430 con 0x55b17ccad590 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b17cfe85d0 con 0x55b17ccce270 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55b17cfd7a00 con 0x55b17cca53b0 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46208/0 (socket says 172.21.15.129:46208) 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 -- 172.21.15.129:0/3578418375 learned_addr learned my addr 172.21.15.129:0/3578418375 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fbb400640 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d09eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 -- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 msgr2=0x55b17d09eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d09eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 -- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 msgr2=0x55b17d09e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.868+0000 7f3fc1200640 1 -- 172.21.15.129:0/3578418375 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b17cfa0520 con 0x55b17ccad590 2024-09-17T04:03:51.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.869+0000 7f3fc1200640 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f3fb402b020 tx=0x7f3fb4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b75475193670887 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.869+0000 7f3fb0c00640 1 -- 172.21.15.129:0/3578418375 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4047070 con 0x55b17ccad590 2024-09-17T04:03:51.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.869+0000 7f3fb0c00640 1 -- 172.21.15.129:0/3578418375 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fb402fdc0 con 0x55b17ccad590 2024-09-17T04:03:51.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.869+0000 7f3fb0c00640 1 -- 172.21.15.129:0/3578418375 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4033070 con 0x55b17ccad590 2024-09-17T04:03:51.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 -- 172.21.15.129:0/3578418375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 msgr2=0x55b17d06ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f3fb402b020 tx=0x7f3fb4004ab0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 -- 172.21.15.129:0/3578418375 shutdown_connections 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d06ef60 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d09eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 --2- 172.21.15.129:0/3578418375 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 -- 172.21.15.129:0/3578418375 >> 172.21.15.129:0/3578418375 conn(0x55b17ccc23c0 msgr2=0x55b17d074970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 -- 172.21.15.129:0/3578418375 shutdown_connections 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.870+0000 7f3fca288740 1 -- 172.21.15.129:0/3578418375 wait complete. 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.871+0000 7f3fca288740 1 -- start start 2024-09-17T04:03:51.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.871+0000 7f3fca288740 1 Processor -- start 2024-09-17T04:03:51.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b17cf7e330 con 0x55b17ccad590 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b17cfdb810 con 0x55b17cca53b0 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55b17cd48620 con 0x55b17ccce270 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fca288740 10 client.0 _ll_register_callbacks cb 0x55b17cf83420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:03:51.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46218/0 (socket says 172.21.15.129:46218) 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 -- 172.21.15.129:0/2910682332 learned_addr learned my addr 172.21.15.129:0/2910682332 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fc1200640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 -- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 msgr2=0x55b17d0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d0b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 -- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 msgr2=0x55b17d0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbb400640 1 -- 172.21.15.129:0/2910682332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b17cd39850 con 0x55b17ccad590 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fc1200640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.873+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b390 tx=0x7f3fa8062490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f717f25fadf6ff58 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8088070 con 0x55b17ccad590 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f3fa80616f0 con 0x55b17ccad590 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fb402fdc0 con 0x55b17ccad590 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f3fa8083050 con 0x55b17ccad590 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fca288740 10 client.10575 fetch_fsmap learned FSMap version 181 2024-09-17T04:03:51.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) -- 0x55b17cdb06b0 con 0x55b17ccad590 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fb402fdc0 con 0x55b17ccad590 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 181) ==== 3502+0+0 (secure 0 0 0) 0x7f3fa805dde0 con 0x55b17ccad590 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fca288740 10 client.10575 fetch_fsmap finished waiting for FSMap version 181 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.874+0000 7f3fca288740 10 client.10575 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.875+0000 7f3fca288740 20 client.10575 populate_metadata read hostname 'smithi129' 2024-09-17T04:03:51.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.875+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17cc8ce40 0x55b17d0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.875+0000 7f3fca288740 4 client.10575 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:03:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.875+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b17cfea520 con 0x55b17cc8ce40 2024-09-17T04:03:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.875+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17cc8ce40 0x55b17d0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.876+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17cc8ce40 0x55b17d0cbee0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d02fee53b4d304ac server_cookie=64affef92a5ace80 in_seq=0 out_seq=0 2024-09-17T04:03:51.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.876+0000 7f3fa5600640 10 client.10575 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:03:51.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fa5600640 1 -- 172.21.15.129:0/2910682332 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55b17cfea520 con 0x55b17cc8ce40 2024-09-17T04:03:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fa5600640 10 client.10575 handle_command_reply: tid=0 2024-09-17T04:03:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 client.10575 shutdown 2024-09-17T04:03:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 msgr2=0x55b17d098640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b390 tx=0x7f3fa8062490 comp rx=0 tx=0).stop 2024-09-17T04:03:51.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 shutdown_connections 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17cc8ce40 0x55b17d0cbee0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d098640 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fa5600640 0 client.10575 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 --2- 172.21.15.129:0/2910682332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 >> 172.21.15.129:0/2910682332 conn(0x55b17ccc23c0 msgr2=0x55b17d0702f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 shutdown_connections 2024-09-17T04:03:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 1 -- 172.21.15.129:0/2910682332 wait complete. 2024-09-17T04:03:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.877+0000 7f3fca288740 20 client.10575 trim_cache size 0 max 16384 2024-09-17T04:03:51.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.896+0000 7f3fca288740 1 Processor -- start 2024-09-17T04:03:51.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.896+0000 7f3fca288740 1 -- start start 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d0bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc0c34f0 con 0x55b17ccad590 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc0c2850 con 0x55b17ccce270 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3fbc0c2ea0 con 0x55b17cca53b0 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46234/0 (socket says 172.21.15.129:46234) 2024-09-17T04:03:51.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 -- 172.21.15.129:0/2027591439 learned_addr learned my addr 172.21.15.129:0/2027591439 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d0bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fc1200640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 -- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 msgr2=0x55b17d0bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d0bcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 -- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 msgr2=0x55b17d09e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbb400640 1 -- 172.21.15.129:0/2027591439 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b17d075650 con 0x55b17ccad590 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.898+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d0bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.899+0000 7f3fbb400640 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8062190 tx=0x7f3fa8098b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d1cd2d74a7047e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.899+0000 7f3fa7e00640 1 -- 172.21.15.129:0/2027591439 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8088070 con 0x55b17ccad590 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.899+0000 7f3fa7e00640 1 -- 172.21.15.129:0/2027591439 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fa806be00 con 0x55b17ccad590 2024-09-17T04:03:51.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.899+0000 7f3fa7e00640 1 -- 172.21.15.129:0/2027591439 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8061160 con 0x55b17ccad590 2024-09-17T04:03:51.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 -- 172.21.15.129:0/2027591439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 msgr2=0x55b17d0bd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8062190 tx=0x7f3fa8098b10 comp rx=0 tx=0).stop 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 -- 172.21.15.129:0/2027591439 shutdown_connections 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17ccce270 0x55b17d09e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d0bd030 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 --2- 172.21.15.129:0/2027591439 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17cca53b0 0x55b17d0bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 -- 172.21.15.129:0/2027591439 >> 172.21.15.129:0/2027591439 conn(0x55b17ccc23c0 msgr2=0x55b17d0b51b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 -- 172.21.15.129:0/2027591439 shutdown_connections 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.900+0000 7f3fca288740 1 -- 172.21.15.129:0/2027591439 wait complete. 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.901+0000 7f3fca288740 1 -- start start 2024-09-17T04:03:51.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.901+0000 7f3fca288740 1 Processor -- start 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0db6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d13d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55b17d099d20 con 0x55b17ccad590 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55b17d09a300 con 0x55b17cca53b0 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55b17d0bbd70 con 0x55b17ccce270 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.902+0000 7f3fca288740 10 client.0 _ll_register_callbacks cb 0x7f3fa8061c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46238/0 (socket says 172.21.15.129:46238) 2024-09-17T04:03:51.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 -- 172.21.15.129:0/1696255981 learned_addr learned my addr 172.21.15.129:0/1696255981 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0db6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fc1200640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d13d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 -- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 msgr2=0x55b17d13d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d13d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 -- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 msgr2=0x55b17d0db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0db6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 -- 172.21.15.129:0/1696255981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b17d0985a0 con 0x55b17ccad590 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0db6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fbb400640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b390 tx=0x7f3fa8062100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6238932d5a4011 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fa5600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa8088070 con 0x55b17ccad590 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f3fa8061360 con 0x55b17ccad590 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.903+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fa806be00 con 0x55b17ccad590 2024-09-17T04:03:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) ==== 24+0+0 (secure 0 0 0) 0x7f3fa8068050 con 0x55b17ccad590 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 10 client.10581 fetch_fsmap learned FSMap version 181 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) -- 0x55b17d097ba0 con 0x55b17ccad590 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3fa806be00 con 0x55b17ccad590 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 181) ==== 3502+0+0 (secure 0 0 0) 0x7f3fa80681f0 con 0x55b17ccad590 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 10 client.10581 fetch_fsmap finished waiting for FSMap version 181 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 10 client.10581 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 20 client.10581 populate_metadata read hostname 'smithi129' 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17d096b70 0x55b17d096f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 4 client.10581 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "status"}) -- 0x55b17cfe5fd0 con 0x55b17d096b70 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.904+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17d096b70 0x55b17d096f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fbbe00640 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17d096b70 0x55b17d096f80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b931fdfa1d7e939 server_cookie=7a5d17ff02323278 in_seq=0 out_seq=0 2024-09-17T04:03:51.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fa5600640 10 client.10581 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:03:51.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fa5600640 1 -- 172.21.15.129:0/1696255981 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55b17cfe5fd0 con 0x55b17d096b70 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fa5600640 10 client.10581 handle_command_reply: tid=0 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fca288740 1 client.10581 shutdown 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 msgr2=0x55b17d122e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.905+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b390 tx=0x7f3fa8062100 comp rx=0 tx=0).stop 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 shutdown_connections 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55b17d096b70 0x55b17d096f80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55b17ccce270 0x55b17d13d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55b17ccad590 0x55b17d122e60 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 --2- 172.21.15.129:0/1696255981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b17cca53b0 0x55b17d0db6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 >> 172.21.15.129:0/1696255981 conn(0x55b17ccc23c0 msgr2=0x55b17d0bd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fa5600640 0 client.10581 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 shutdown_connections 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 1 -- 172.21.15.129:0/1696255981 wait complete. 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fca288740 20 client.10581 trim_cache size 0 max 16384 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 10440, 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:03:51.910 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.597878457, 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 181, 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 203, 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 203, 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.628798123999999 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:03:51.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 msgr2=0x7f3f9c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 0x7f3f9c07b0a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f3fac004630 tx=0x7f3fac00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 msgr2=0x7f3fbc0cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.906+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f3fa800b3c0 tx=0x7f3fa800b880 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 shutdown_connections 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f9c078b90 0x7f3f9c07b0a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3fbc0cb540 0x7f3fbc074000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3fbc0705a0 0x7f3fbc0cb000 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 --2- 172.21.15.129:0/3770062517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3fbc06eed0 0x7f3fbc0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 >> 172.21.15.129:0/3770062517 conn(0x7f3fbc06d410 msgr2=0x7f3fbc0ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 shutdown_connections 2024-09-17T04:03:51.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:51.907+0000 7f3fc3200640 1 -- 172.21.15.129:0/3770062517 wait complete. 2024-09-17T04:03:51.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 -- 172.21.15.129:0/428981975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80b4140 msgr2=0x7f75a80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 --2- 172.21.15.129:0/428981975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80b4140 0x7f75a80ccc80 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f75a8072e00 tx=0x7f759402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 -- 172.21.15.129:0/428981975 shutdown_connections 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 --2- 172.21.15.129:0/428981975 >> v2:172.21.15.181:3300/0 conn(0x7f75a80cd1c0 0x7f75a81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 --2- 172.21.15.129:0/428981975 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80b4140 0x7f75a80ccc80 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 -- 172.21.15.129:0/428981975 >> 172.21.15.129:0/428981975 conn(0x7f75a806d410 msgr2=0x7f75a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 -- 172.21.15.129:0/428981975 shutdown_connections 2024-09-17T04:03:52.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.151+0000 7f75afa00640 1 -- 172.21.15.129:0/428981975 wait complete. 2024-09-17T04:03:52.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.154+0000 7f75afa00640 1 Processor -- start 2024-09-17T04:03:52.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.154+0000 7f75afa00640 1 -- start start 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75a80b4140 0x7f75a809d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 0x7f75a809d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8160260 con 0x7f75a80705a0 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f75a81600e0 con 0x7f75a80b4140 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75afa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f75a8160f00 con 0x7f75a80cd1c0 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.155+0000 7f75ad000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46256/0 (socket says 172.21.15.129:46256) 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 -- 172.21.15.129:0/1937255226 learned_addr learned my addr 172.21.15.129:0/1937255226 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75a7e00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75a80b4140 0x7f75a809d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ada00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 0x7f75a809d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 -- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 msgr2=0x7f75a809d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 0x7f75a809d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 -- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75a80b4140 msgr2=0x7f75a809d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75a80b4140 0x7f75a809d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 -- 172.21.15.129:0/1937255226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a80c7320 con 0x7f75a80705a0 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ada00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 0x7f75a809d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75ad000640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c3f0 tx=0x7f759c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.156+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c017070 con 0x7f75a80705a0 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.157+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f759c0035f0 con 0x7f75a80705a0 2024-09-17T04:03:52.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.157+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a81f41d0 con 0x7f75a80705a0 2024-09-17T04:03:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.157+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f759c003a40 con 0x7f75a80705a0 2024-09-17T04:03:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.157+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a81f4790 con 0x7f75a80705a0 2024-09-17T04:03:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.158+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a80ccc80 con 0x7f75a80705a0 2024-09-17T04:03:52.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.159+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f759c01a070 con 0x7f75a80705a0 2024-09-17T04:03:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.161+0000 7f75a4c00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 0x7f758807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.161+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f759c09a090 con 0x7f75a80705a0 2024-09-17T04:03:52.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.161+0000 7f75a7e00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 0x7f758807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:52.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.162+0000 7f75a7e00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 0x7f758807b0a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f7594004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:52.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.165+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75a80cd5d0 con 0x7f75a80705a0 2024-09-17T04:03:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.385+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75a806a140 con 0x7f75a80705a0 2024-09-17T04:03:52.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.385+0000 7f75a4c00640 1 -- 172.21.15.129:0/1937255226 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v181) ==== 53+0+73 (secure 0 0 0) 0x7f759c061df0 con 0x7f75a80705a0 2024-09-17T04:03:52.385 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.387+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 msgr2=0x7f758807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 0x7f758807b0a0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f7594004730 comp rx=0 tx=0).stop 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 msgr2=0x7f75a80c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f759c00c3f0 tx=0x7f759c00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 shutdown_connections 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7588078b90 0x7f758807b0a0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75a80cd1c0 0x7f75a809d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75a80b4140 0x7f75a809d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 --2- 172.21.15.129:0/1937255226 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75a80705a0 0x7f75a80c9a30 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:52.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 >> 172.21.15.129:0/1937255226 conn(0x7f75a806d410 msgr2=0x7f75a8072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:52.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 shutdown_connections 2024-09-17T04:03:52.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52.388+0000 7f75afa00640 1 -- 172.21.15.129:0/1937255226 wait complete. 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:03:52.415 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:03:52.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:52.416 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:03:52.453 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41860sec preferred_lft 41860sec 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.454 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:03:52.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:52.456 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:03:52.546 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:03:52.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:52.549 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:03:52.585 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:03:52.585 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:03:52.585 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:03:52.585 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:03:52.585 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:03:52.586 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:03:52.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:03:52.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:03:52.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:52.782 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:03:52.817 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:03:52.817 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:03:52.818 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:03:52.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:52.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:03:52.914 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:03:52.914 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:03:52.949 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:03:52.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:03:52.949 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:03:53.024 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:03:53.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:53.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:03:53.220 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:03:53.221 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:03:53.221 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:03:53.241 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:03:53.241 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:03:53.241 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:03:53.241 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:03:53.241 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:03:53.275 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:03:53.275 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:03:53.275 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:03:53.275 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:03:53.276 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:03:53.276 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:03:53.276 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:03:53.276 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:03:53.276 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:03:53.277 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:03:53.277 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:03:53.277 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:03:53.277 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:03:53.279 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:03:53.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:03:53.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:03:53.336 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:03:53.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:03:53.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:03:53.440 INFO:teuthology.orchestra.run.smithi129.stdout:client9838 2024-09-17T04:03:53.441 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:03:53.441 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:03:53.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-09-17T04:03:53.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 -- 172.21.15.129:0/1454489454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2340cb080 msgr2=0x7fd234072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 --2- 172.21.15.129:0/1454489454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2340cb080 0x7fd234072e10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fd224009830 tx=0x7fd22402a5b0 comp rx=0 tx=0).stop 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 -- 172.21.15.129:0/1454489454 shutdown_connections 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 --2- 172.21.15.129:0/1454489454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2340cb080 0x7fd234072e10 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 -- 172.21.15.129:0/1454489454 >> 172.21.15.129:0/1454489454 conn(0x7fd2340d00b0 msgr2=0x7fd2340d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 -- 172.21.15.129:0/1454489454 shutdown_connections 2024-09-17T04:03:53.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.742+0000 7fd23b200640 1 -- 172.21.15.129:0/1454489454 wait complete. 2024-09-17T04:03:53.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.744+0000 7fd23b200640 1 Processor -- start 2024-09-17T04:03:53.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.744+0000 7fd23b200640 1 -- start start 2024-09-17T04:03:53.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd234073350 0x7fd23409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2340cb080 0x7fd23414ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2340cef20 con 0x7fd23415c7b0 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd2340ceda0 con 0x7fd234073350 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd2340cf0a0 con 0x7fd2340cb080 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46272/0 (socket says 172.21.15.129:46272) 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd23a800640 1 -- 172.21.15.129:0/4167135015 learned_addr learned my addr 172.21.15.129:0/4167135015 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:53.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd239e00640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd234073350 0x7fd23409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.746+0000 7fd239400640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2340cb080 0x7fd23414ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 -- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2340cb080 msgr2=0x7fd23414ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2340cb080 0x7fd23414ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 -- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd234073350 msgr2=0x7fd23409aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd234073350 0x7fd23409aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 -- 172.21.15.129:0/4167135015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2340c66c0 con 0x7fd23415c7b0 2024-09-17T04:03:53.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23a800640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fd23000c730 tx=0x7fd23000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd230003b70 con 0x7fd23415c7b0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd230003d10 con 0x7fd23415c7b0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2340c69a0 con 0x7fd23415c7b0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23000aab0 con 0x7fd23415c7b0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.747+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd23409afa0 con 0x7fd23415c7b0 2024-09-17T04:03:53.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.749+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2340c6430 con 0x7fd23415c7b0 2024-09-17T04:03:53.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.750+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd23000ac50 con 0x7fd23415c7b0 2024-09-17T04:03:53.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.750+0000 7fd22e000640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 0x7fd21807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:53.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.751+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7fd23009a950 con 0x7fd23415c7b0 2024-09-17T04:03:53.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.751+0000 7fd239e00640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 0x7fd21807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:53.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.752+0000 7fd239e00640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 0x7fd21807b0a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd2280046b0 tx=0x7fd228007cc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:53.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.755+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd2300627c0 con 0x7fd23415c7b0 2024-09-17T04:03:53.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:53.978+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7fd2340d0f50 con 0x7fd23415c7b0 2024-09-17T04:03:54.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.618+0000 7fd22e000640 1 -- 172.21.15.129:0/4167135015 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v182) ==== 105+0+0 (secure 0 0 0) 0x7fd2300671f0 con 0x7fd23415c7b0 2024-09-17T04:03:54.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 msgr2=0x7fd21807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:54.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 0x7fd21807b0a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fd2280046b0 tx=0x7fd228007cc0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 msgr2=0x7fd23409b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fd23000c730 tx=0x7fd23000cbf0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 shutdown_connections 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd218078b90 0x7fd21807b0a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd23415c7b0 0x7fd23409b830 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2340cb080 0x7fd23414ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 --2- 172.21.15.129:0/4167135015 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd234073350 0x7fd23409aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 >> 172.21.15.129:0/4167135015 conn(0x7fd2340d00b0 msgr2=0x7fd2340ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 shutdown_connections 2024-09-17T04:03:54.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.623+0000 7fd23b200640 1 -- 172.21.15.129:0/4167135015 wait complete. 2024-09-17T04:03:54.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 -- 172.21.15.129:0/2521347803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f40cd390 msgr2=0x7f09f41609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 --2- 172.21.15.129:0/2521347803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f40cd390 0x7f09f41609c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7f09ec005fd0 tx=0x7f09ec02aac0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 -- 172.21.15.129:0/2521347803 shutdown_connections 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 --2- 172.21.15.129:0/2521347803 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f40cd390 0x7f09f41609c0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 --2- 172.21.15.129:0/2521347803 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 0x7f09f40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.880+0000 7f09fb400640 1 -- 172.21.15.129:0/2521347803 >> 172.21.15.129:0/2521347803 conn(0x7f09f406d410 msgr2=0x7f09f40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:54.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.881+0000 7f09fb400640 1 -- 172.21.15.129:0/2521347803 shutdown_connections 2024-09-17T04:03:54.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.881+0000 7f09fb400640 1 -- 172.21.15.129:0/2521347803 wait complete. 2024-09-17T04:03:54.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.883+0000 7f09fb400640 1 Processor -- start 2024-09-17T04:03:54.883 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.883+0000 7f09fb400640 1 -- start start 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 0x7f09f415bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09f40cbc90 0x7f09f4098c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f09f4160f00 con 0x7f09f406eed0 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f09f406f4f0 con 0x7f09f40cbc90 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09fb400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f09f4161080 con 0x7f09f40705a0 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.884+0000 7f09f8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46592/0 (socket says 172.21.15.129:46592) 2024-09-17T04:03:54.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 -- 172.21.15.129:0/1540764405 learned_addr learned my addr 172.21.15.129:0/1540764405 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09ebe00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 0x7f09f415bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f9400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09f40cbc90 0x7f09f4098c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 -- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 msgr2=0x7f09f415bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 0x7f09f415bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 -- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09f40cbc90 msgr2=0x7f09f4098c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09f40cbc90 0x7f09f4098c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 -- 172.21.15.129:0/1540764405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f40b0f20 con 0x7f09f406eed0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.885+0000 7f09f8a00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f09e400d740 tx=0x7f09e400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.886+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4017070 con 0x7f09f406eed0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.886+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09e4009d20 con 0x7f09f406eed0 2024-09-17T04:03:54.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.886+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f40b1190 con 0x7f09f406eed0 2024-09-17T04:03:54.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.886+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e40106c0 con 0x7f09f406eed0 2024-09-17T04:03:54.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.886+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f40b1750 con 0x7f09f406eed0 2024-09-17T04:03:54.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.887+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f40b4140 con 0x7f09f406eed0 2024-09-17T04:03:54.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.888+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f09e401a070 con 0x7f09f406eed0 2024-09-17T04:03:54.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.889+0000 7f09f1600640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 0x7f09d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:03:54.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.889+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f09e409adf0 con 0x7f09f406eed0 2024-09-17T04:03:54.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.889+0000 7f09ebe00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 0x7f09d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:03:54.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.890+0000 7f09ebe00640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 0x7f09d007b0d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f09f41e6500 tx=0x7f09dc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:03:54.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:54.893+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f09e4062bd0 con 0x7f09f406eed0 2024-09-17T04:03:55.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.112+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f406a140 con 0x7f09f406eed0 2024-09-17T04:03:55.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.112+0000 7f09f1600640 1 -- 172.21.15.129:0/1540764405 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) ==== 95+0+5269 (secure 0 0 0) 0x7f09e4067600 con 0x7f09f406eed0 2024-09-17T04:03:55.113 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:03:55.113 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":183,"btime":"2024-09-17T04:03:54:624541+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3964402375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3964402375},{"type":"v1","addr":"172.21.15.129:6833","nonce":3964402375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175},{"gid":8276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2137304743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2137304743},{"type":"v1","addr":"172.21.15.181:6836","nonce":2137304743}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:03:54.624524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":10434},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10434":{"gid":10434,"name":"c","rank":1,"incarnation":183,"state":"up:creating","state_seq":1,"addr":"172.21.15.129:6835/3759513659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3759513659},{"type":"v1","addr":"172.21.15.129:6835","nonce":3759513659}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:03:55.113 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 183 2024-09-17T04:03:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 msgr2=0x7f09d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 0x7f09d007b0d0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f09f41e6500 tx=0x7f09dc009040 comp rx=0 tx=0).stop 2024-09-17T04:03:55.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 msgr2=0x7f09f40cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7f09e400d740 tx=0x7f09e400dc00 comp rx=0 tx=0).stop 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 shutdown_connections 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f09d0078bc0 0x7f09d007b0d0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09f40cbc90 0x7f09f4098c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09f40705a0 0x7f09f415bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 --2- 172.21.15.129:0/1540764405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09f406eed0 0x7f09f40cc340 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 >> 172.21.15.129:0/1540764405 conn(0x7f09f406d410 msgr2=0x7f09f415e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 shutdown_connections 2024-09-17T04:03:55.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:03:55.115+0000 7f09fb400640 1 -- 172.21.15.129:0/1540764405 wait complete. 2024-09-17T04:03:55.137 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-09-17T04:04:00.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 -- 172.21.15.129:0/1685717594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 msgr2=0x7fc2b415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 --2- 172.21.15.129:0/1685717594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b415bca0 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7fc2b415f1d0 tx=0x7fc2a002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 -- 172.21.15.129:0/1685717594 shutdown_connections 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 --2- 172.21.15.129:0/1685717594 >> v2:172.21.15.181:3300/0 conn(0x7fc2b415c1e0 0x7fc2b4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 --2- 172.21.15.129:0/1685717594 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b415bca0 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 -- 172.21.15.129:0/1685717594 >> 172.21.15.129:0/1685717594 conn(0x7fc2b406cfc0 msgr2=0x7fc2b406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 -- 172.21.15.129:0/1685717594 shutdown_connections 2024-09-17T04:04:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.368+0000 7fc2bac00640 1 -- 172.21.15.129:0/1685717594 wait complete. 2024-09-17T04:04:00.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.371+0000 7fc2bac00640 1 Processor -- start 2024-09-17T04:04:00.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.371+0000 7fc2bac00640 1 -- start start 2024-09-17T04:04:00.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2b41524c0 0x7fc2b4149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2b415c1e0 0x7fc2b41e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b40d02c0 con 0x7fc2b41528d0 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4073830 con 0x7fc2b415c1e0 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2bac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b40d0440 con 0x7fc2b41524c0 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.372+0000 7fc2b8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46606/0 (socket says 172.21.15.129:46606) 2024-09-17T04:04:00.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 -- 172.21.15.129:0/880739700 learned_addr learned my addr 172.21.15.129:0/880739700 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2ba200640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2b415c1e0 0x7fc2b41e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b9800640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2b41524c0 0x7fc2b4149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 -- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2b41524c0 msgr2=0x7fc2b4149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2b41524c0 0x7fc2b4149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 -- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2b415c1e0 msgr2=0x7fc2b41e9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2b415c1e0 0x7fc2b41e9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 -- 172.21.15.129:0/880739700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b414a8f0 con 0x7fc2b41528d0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b8e00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0000c00 tx=0x7fc2a0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.373+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a003e030 con 0x7fc2b41528d0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.374+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc2a0003d10 con 0x7fc2b41528d0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.374+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0035040 con 0x7fc2b41528d0 2024-09-17T04:04:00.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.374+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b414abd0 con 0x7fc2b41528d0 2024-09-17T04:04:00.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.374+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b414b190 con 0x7fc2b41528d0 2024-09-17T04:04:00.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.375+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b406a140 con 0x7fc2b41528d0 2024-09-17T04:04:00.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.376+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc2a0041030 con 0x7fc2b41528d0 2024-09-17T04:04:00.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.377+0000 7fc2b1600640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 0x7fc29407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.377+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7fc2a00bb3d0 con 0x7fc2b41528d0 2024-09-17T04:04:00.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.378+0000 7fc2b9800640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 0x7fc29407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.378+0000 7fc2b9800640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 0x7fc29407b0d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80015e0 tx=0x7fc2a8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:00.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.381+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc2a00831b0 con 0x7fc2b41528d0 2024-09-17T04:04:00.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.596+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b414a680 con 0x7fc2b41528d0 2024-09-17T04:04:00.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.597+0000 7fc2b1600640 1 -- 172.21.15.129:0/880739700 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5273 (secure 0 0 0) 0x7fc2a0087be0 con 0x7fc2b41528d0 2024-09-17T04:04:00.597 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:00.597 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":184,"btime":"2024-09-17T04:03:55:626213+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3964402375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3964402375},{"type":"v1","addr":"172.21.15.129:6833","nonce":3964402375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175},{"gid":8276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2137304743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2137304743},{"type":"v1","addr":"172.21.15.181:6836","nonce":2137304743}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:03:55.626207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":10434},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10434":{"gid":10434,"name":"c","rank":1,"incarnation":183,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3759513659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3759513659},{"type":"v1","addr":"172.21.15.129:6835","nonce":3759513659}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10434,10440]},"id":15}]} 2024-09-17T04:04:00.598 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 184 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.599+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 msgr2=0x7fc29407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.599+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 0x7fc29407b0d0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80015e0 tx=0x7fc2a8009040 comp rx=0 tx=0).stop 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.599+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 msgr2=0x7fc2b41e94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.599+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0000c00 tx=0x7fc2a0003820 comp rx=0 tx=0).stop 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 shutdown_connections 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc294078bc0 0x7fc29407b0d0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2b415c1e0 0x7fc2b41e9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc2b41528d0 0x7fc2b41e94e0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 --2- 172.21.15.129:0/880739700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2b41524c0 0x7fc2b4149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 >> 172.21.15.129:0/880739700 conn(0x7fc2b406cfc0 msgr2=0x7fc2b4151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 shutdown_connections 2024-09-17T04:04:00.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.600+0000 7fc2bac00640 1 -- 172.21.15.129:0/880739700 wait complete. 2024-09-17T04:04:00.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.848+0000 7f17f4a00640 1 -- 172.21.15.129:0/2519234419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00cd1c0 msgr2=0x7f17f01609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.848+0000 7f17f4a00640 1 --2- 172.21.15.129:0/2519234419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00cd1c0 0x7f17f01609c0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f17f0072080 tx=0x7f17e802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 -- 172.21.15.129:0/2519234419 shutdown_connections 2024-09-17T04:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 --2- 172.21.15.129:0/2519234419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00cd1c0 0x7f17f01609c0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 --2- 172.21.15.129:0/2519234419 >> v2:172.21.15.181:3300/0 conn(0x7f17f00b4140 0x7f17f00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 -- 172.21.15.129:0/2519234419 >> 172.21.15.129:0/2519234419 conn(0x7f17f006d410 msgr2=0x7f17f00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:00.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 -- 172.21.15.129:0/2519234419 shutdown_connections 2024-09-17T04:04:00.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.849+0000 7f17f4a00640 1 -- 172.21.15.129:0/2519234419 wait complete. 2024-09-17T04:04:00.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.851+0000 7f17f4a00640 1 Processor -- start 2024-09-17T04:04:00.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.851+0000 7f17f4a00640 1 -- start start 2024-09-17T04:04:00.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.852+0000 7f17f4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f00705a0 0x7f17f009d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.852+0000 7f17f4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17f4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f00cd1c0 0x7f17f015bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17f4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0160260 con 0x7f17f00b4140 2024-09-17T04:04:00.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17f4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17f01600e0 con 0x7f17f00705a0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17f4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17f0160f00 con 0x7f17f00cd1c0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46624/0 (socket says 172.21.15.129:46624) 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 -- 172.21.15.129:0/1304978077 learned_addr learned my addr 172.21.15.129:0/1304978077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17ede00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f00705a0 0x7f17f009d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17ee800640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f00cd1c0 0x7f17f015bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 -- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f00cd1c0 msgr2=0x7f17f015bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f00cd1c0 0x7f17f015bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 -- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f00705a0 msgr2=0x7f17f009d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f00705a0 0x7f17f009d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.853+0000 7f17e5400640 1 -- 172.21.15.129:0/1304978077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f01f93a0 con 0x7f17f00b4140 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17e5400640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009fd0 tx=0x7f17d800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:00.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d800ccc0 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17d800ee70 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d800ce60 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f01f9680 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.854+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01f9c40 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.855+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f00cc050 con 0x7f17f00b4140 2024-09-17T04:04:00.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.856+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f17d8005260 con 0x7f17f00b4140 2024-09-17T04:04:00.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.857+0000 7f17e6a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 0x7f17cc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:00.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.857+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7f17d809b650 con 0x7f17f00b4140 2024-09-17T04:04:00.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.858+0000 7f17ede00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 0x7f17cc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:00.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.858+0000 7f17ede00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 0x7f17cc07b0a0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001720 tx=0x7f17e0007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:00.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:00.862+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f17d80633a0 con 0x7f17f00b4140 2024-09-17T04:04:01.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.078+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17f00ccc80 con 0x7f17f00b4140 2024-09-17T04:04:01.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.078+0000 7f17e6a00640 1 -- 172.21.15.129:0/1304978077 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5273 (secure 0 0 0) 0x7f17d8067dd0 con 0x7f17f00b4140 2024-09-17T04:04:01.079 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:01.079 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":184,"btime":"2024-09-17T04:03:55:626213+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/3964402375","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3964402375},{"type":"v1","addr":"172.21.15.129:6833","nonce":3964402375}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":175},{"gid":8276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2137304743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2137304743},{"type":"v1","addr":"172.21.15.181:6836","nonce":2137304743}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":176}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:03:55.626207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":10434},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10434":{"gid":10434,"name":"c","rank":1,"incarnation":183,"state":"up:active","state_seq":6,"addr":"172.21.15.129:6835/3759513659","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3759513659},{"type":"v1","addr":"172.21.15.129:6835","nonce":3759513659}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10434,10440]},"id":15}]} 2024-09-17T04:04:01.079 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 184 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 msgr2=0x7f17cc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 0x7f17cc07b0a0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001720 tx=0x7f17e0007e90 comp rx=0 tx=0).stop 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 msgr2=0x7f17f015ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009fd0 tx=0x7f17d800ec20 comp rx=0 tx=0).stop 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 shutdown_connections 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17cc078b90 0x7f17cc07b0a0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f00cd1c0 0x7f17f015bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f00b4140 0x7f17f015ba90 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 --2- 172.21.15.129:0/1304978077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f00705a0 0x7f17f009d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 >> 172.21.15.129:0/1304978077 conn(0x7f17f006d410 msgr2=0x7f17f00ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:01.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 shutdown_connections 2024-09-17T04:04:01.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.081+0000 7f17f4a00640 1 -- 172.21.15.129:0/1304978077 wait complete. 2024-09-17T04:04:01.103 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are d and c 2024-09-17T04:04:01.103 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:04:01.103 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:04:01.125 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:04:01.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-17T04:04:01.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 -- 172.21.15.129:0/608471576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780cd390 msgr2=0x7fd2781609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:01.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 --2- 172.21.15.129:0/608471576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780cd390 0x7fd2781609c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fd270005fd0 tx=0x7fd27002a940 comp rx=0 tx=0).stop 2024-09-17T04:04:01.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 -- 172.21.15.129:0/608471576 shutdown_connections 2024-09-17T04:04:01.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 --2- 172.21.15.129:0/608471576 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780cd390 0x7fd2781609c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 -- 172.21.15.129:0/608471576 >> 172.21.15.129:0/608471576 conn(0x7fd27806d410 msgr2=0x7fd2780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:01.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 -- 172.21.15.129:0/608471576 shutdown_connections 2024-09-17T04:04:01.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.389+0000 7fd27ee00640 1 -- 172.21.15.129:0/608471576 wait complete. 2024-09-17T04:04:01.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.391+0000 7fd27ee00640 1 Processor -- start 2024-09-17T04:04:01.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.391+0000 7fd27ee00640 1 -- start start 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 0x7fd278158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2781f3950 0x7fd2781f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd278160f00 con 0x7fd2780b4140 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd278070d40 con 0x7fd2780705a0 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd278161080 con 0x7fd2781f3950 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46628/0 (socket says 172.21.15.129:46628) 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 -- 172.21.15.129:0/561945784 learned_addr learned my addr 172.21.15.129:0/561945784 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd27ce00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2781f3950 0x7fd2781f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277e00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 0x7fd278158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:01.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 -- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2781f3950 msgr2=0x7fd2781f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2781f3950 0x7fd2781f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 -- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 msgr2=0x7fd278158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 0x7fd278158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277400640 1 -- 172.21.15.129:0/561945784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27815bd50 con 0x7fd2780b4140 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.393+0000 7fd277e00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 0x7fd278158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd277400640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd26400c400 tx=0x7fd26400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264017070 con 0x7fd2780b4140 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2640053c0 con 0x7fd2780b4140 2024-09-17T04:04:01.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27815ba70 con 0x7fd2780b4140 2024-09-17T04:04:01.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264003830 con 0x7fd2780b4140 2024-09-17T04:04:01.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.394+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd278156ae0 con 0x7fd2780b4140 2024-09-17T04:04:01.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.395+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2781609c0 con 0x7fd2780b4140 2024-09-17T04:04:01.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.396+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd26401a070 con 0x7fd2780b4140 2024-09-17T04:04:01.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.397+0000 7fd26fe00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 0x7fd25807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:01.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.397+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5709+0+0 (secure 0 0 0) 0x7fd26409a0b0 con 0x7fd2780b4140 2024-09-17T04:04:01.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.398+0000 7fd277e00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 0x7fd25807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:01.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.398+0000 7fd277e00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 0x7fd25807b0d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd2680046b0 tx=0x7fd268009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:01.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.402+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd264061e10 con 0x7fd2780b4140 2024-09-17T04:04:01.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:01.656+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7fd27806a140 con 0x7fd2780b4140 2024-09-17T04:04:02.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.661+0000 7fd26fe00640 1 -- 172.21.15.129:0/561945784 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 8270 v185) ==== 95+0+0 (secure 0 0 0) 0x7fd264066840 con 0x7fd2780b4140 2024-09-17T04:04:02.661 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 8270 2024-09-17T04:04:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 msgr2=0x7fd25807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 0x7fd25807b0d0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd2680046b0 tx=0x7fd268009040 comp rx=0 tx=0).stop 2024-09-17T04:04:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 msgr2=0x7fd2781572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:02.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fd26400c400 tx=0x7fd26400c8c0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 shutdown_connections 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd258078bc0 0x7fd25807b0d0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2781f3950 0x7fd2781f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2780b4140 0x7fd2781572e0 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 --2- 172.21.15.129:0/561945784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2780705a0 0x7fd278158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 >> 172.21.15.129:0/561945784 conn(0x7fd27806d410 msgr2=0x7fd2780ae470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 shutdown_connections 2024-09-17T04:04:02.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.665+0000 7fd27ee00640 1 -- 172.21.15.129:0/561945784 wait complete. 2024-09-17T04:04:02.696 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:04:02.696 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:04:02.721 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:04:02.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T04:04:02.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 -- 172.21.15.129:0/3368180963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad1c004d40 msgr2=0x7fad1c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 --2- 172.21.15.129:0/3368180963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad1c004d40 0x7fad1c005150 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fad30009b50 tx=0x7fad1c005830 comp rx=0 tx=0).stop 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad37e00640 1 -- 172.21.15.129:0/3368180963 reap_dead start 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 -- 172.21.15.129:0/3368180963 shutdown_connections 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 -- 172.21.15.129:0/3368180963 >> 172.21.15.129:0/3368180963 conn(0x7fad3806cfc0 msgr2=0x7fad3806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 -- 172.21.15.129:0/3368180963 shutdown_connections 2024-09-17T04:04:02.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.944+0000 7fad3cc00640 1 -- 172.21.15.129:0/3368180963 wait complete. 2024-09-17T04:04:02.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.946+0000 7fad3cc00640 1 Processor -- start 2024-09-17T04:04:02.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.946+0000 7fad3cc00640 1 -- start start 2024-09-17T04:04:02.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:02.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad381f91e0 0x7fad380c5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad380c5d80 0x7fad381eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fad3815cd50 con 0x7fad380c6860 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fad3815cbd0 con 0x7fad380c5d80 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad3cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fad3815ced0 con 0x7fad381f91e0 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46654/0 (socket says 172.21.15.129:46654) 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad37400640 1 -- 172.21.15.129:0/1436896642 learned_addr learned my addr 172.21.15.129:0/1436896642 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad2ea00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad381f91e0 0x7fad380c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.948+0000 7fad37e00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad380c5d80 0x7fad381eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 -- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad381f91e0 msgr2=0x7fad380c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:02.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad381f91e0 0x7fad380c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 -- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad380c5d80 msgr2=0x7fad381eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad380c5d80 0x7fad381eaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 -- 172.21.15.129:0/1436896642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad381f9b30 con 0x7fad380c6860 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad37400640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fad280013d0 tx=0x7fad28006790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2800fd10 con 0x7fad380c6860 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad28011430 con 0x7fad380c6860 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad28011880 con 0x7fad380c6860 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad381f9e70 con 0x7fad380c6860 2024-09-17T04:04:02.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.949+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad380c5320 con 0x7fad380c6860 2024-09-17T04:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.951+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad380d02c0 con 0x7fad380c6860 2024-09-17T04:04:02.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.952+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fad28018020 con 0x7fad380c6860 2024-09-17T04:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.953+0000 7fad2fe00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 0x7fad1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.953+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 5709+0+0 (secure 0 0 0) 0x7fad2809b7d0 con 0x7fad380c6860 2024-09-17T04:04:02.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.953+0000 7fad2ea00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 0x7fad1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:02.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.954+0000 7fad2ea00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 0x7fad1407b0a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fad380c6260 tx=0x7fad20005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:02.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:02.957+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fad381f9e70 con 0x7fad380c6860 2024-09-17T04:04:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.208+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fad3806a140 con 0x7fad380c6860 2024-09-17T04:04:03.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.662+0000 7fad2fe00640 1 -- 172.21.15.129:0/1436896642 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 8276 v186) ==== 95+0+0 (secure 0 0 0) 0x7fad28015030 con 0x7fad380c6860 2024-09-17T04:04:03.663 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 8276 2024-09-17T04:04:03.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 msgr2=0x7fad1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 0x7fad1407b0a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fad380c6260 tx=0x7fad20005f90 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 msgr2=0x7fad38142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fad280013d0 tx=0x7fad28006790 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 shutdown_connections 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fad14078b90 0x7fad1407b0a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fad380c5d80 0x7fad381eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fad381f91e0 0x7fad380c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 --2- 172.21.15.129:0/1436896642 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fad380c6860 0x7fad38142d60 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.665+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 >> 172.21.15.129:0/1436896642 conn(0x7fad3806cfc0 msgr2=0x7fad380ce940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.666+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 shutdown_connections 2024-09-17T04:04:03.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:03.666+0000 7fad3cc00640 1 -- 172.21.15.129:0/1436896642 wait complete. 2024-09-17T04:04:03.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:04:03.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.789 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.790 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:04:03.791 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:04:03.792 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:04:03.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:04:03.794 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:04:03.795 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:04:03.796 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:04:03.797 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:04:03.798 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 0 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:04:03.799 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:04:03.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:04:03.801 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:04:03.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:04:03.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:04:03.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:04:03.805 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:04:03.806 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:04:03.807 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:04:03.808 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:04:03.809 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 0 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:04:03.821 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:04:03.822 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:04:03.844 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:04:03.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T04:04:04.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 -- 172.21.15.129:0/3379106306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60015cd30 msgr2=0x7fc6000cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 --2- 172.21.15.129:0/3379106306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60015cd30 0x7fc6000cfd10 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fc60015a870 tx=0x7fc5f002a560 comp rx=0 tx=0).stop 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 -- 172.21.15.129:0/3379106306 shutdown_connections 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 --2- 172.21.15.129:0/3379106306 >> v2:172.21.15.181:3300/0 conn(0x7fc60006fc10 0x7fc6000d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 --2- 172.21.15.129:0/3379106306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 --2- 172.21.15.129:0/3379106306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60015cd30 0x7fc6000cfd10 unknown :-1 s=CLOSED pgs=2217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 -- 172.21.15.129:0/3379106306 >> 172.21.15.129:0/3379106306 conn(0x7fc60006cfc0 msgr2=0x7fc60006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 -- 172.21.15.129:0/3379106306 shutdown_connections 2024-09-17T04:04:04.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.104+0000 7fc604c00640 1 -- 172.21.15.129:0/3379106306 wait complete. 2024-09-17T04:04:04.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.106+0000 7fc604c00640 1 Processor -- start 2024-09-17T04:04:04.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.107+0000 7fc604c00640 1 -- start start 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc60015cd30 0x7fc6001e90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc600073710 con 0x7fc60006fc10 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6000d0f10 con 0x7fc60015cd30 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc604c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc600073890 con 0x7fc60006f800 2024-09-17T04:04:04.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc5f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc5fde00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc5f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38636/0 (socket says 172.21.15.129:38636) 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc5f5400640 1 -- 172.21.15.129:0/1117433063 learned_addr learned my addr 172.21.15.129:0/1117433063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.108+0000 7fc5fe800640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc60015cd30 0x7fc6001e90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 -- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc60015cd30 msgr2=0x7fc6001e90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc60015cd30 0x7fc6001e90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 -- 172.21.15.129:0/1117433063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 msgr2=0x7fc6001f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 -- 172.21.15.129:0/1117433063 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6001f4680 con 0x7fc60006f800 2024-09-17T04:04:04.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5f5400640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:04:04.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5fde00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000aa000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0031000 tx=0x7fc5f00038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.109+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f003e030 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.110+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5f0003d90 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.110+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0035040 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.110+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6001f4960 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.110+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6000a9800 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.111+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60015fca0 con 0x7fc60006f800 2024-09-17T04:04:04.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.112+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc5f0041030 con 0x7fc60006f800 2024-09-17T04:04:04.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.113+0000 7fc5f6a00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 0x7fc5dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.113+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(205..205 src has 1..205) ==== 5709+0+0 (secure 0 0 0) 0x7fc5f00bba30 con 0x7fc60006f800 2024-09-17T04:04:04.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.114+0000 7fc5f5400640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 0x7fc5dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.114+0000 7fc5f5400640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 0x7fc5dc07b0a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc60015f320 tx=0x7fc5e800b410 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:04.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.117+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc60015fca0 con 0x7fc60006f800 2024-09-17T04:04:04.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.375+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7fc6000d0620 con 0x7fc60006f800 2024-09-17T04:04:04.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.383+0000 7fc5f6a00640 1 -- 172.21.15.129:0/1117433063 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 10434 v187) ==== 96+0+0 (secure 0 0 0) 0x7fc5f00838c0 con 0x7fc60006f800 2024-09-17T04:04:04.384 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 10434 2024-09-17T04:04:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 msgr2=0x7fc5dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 0x7fc5dc07b0a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fc60015f320 tx=0x7fc5e800b410 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 msgr2=0x7fc6000aa000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000aa000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0031000 tx=0x7fc5f00038a0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 shutdown_connections 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5dc078b90 0x7fc5dc07b0a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc60015cd30 0x7fc6001e90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc60006fc10 0x7fc6001f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 --2- 172.21.15.129:0/1117433063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc60006f800 0x7fc6000aa000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 >> 172.21.15.129:0/1117433063 conn(0x7fc60006cfc0 msgr2=0x7fc60015de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 shutdown_connections 2024-09-17T04:04:04.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.386+0000 7fc604c00640 1 -- 172.21.15.129:0/1117433063 wait complete. 2024-09-17T04:04:04.412 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:04:04.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:04:04.417 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:04:04.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:04.612 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:04:04.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 -- 172.21.15.129:0/375434357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2415cd30 msgr2=0x7f0d240cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 --2- 172.21.15.129:0/375434357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2415cd30 0x7f0d240cfd10 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f0d14005fd0 tx=0x7f0d1402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 -- 172.21.15.129:0/375434357 shutdown_connections 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 --2- 172.21.15.129:0/375434357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2406fc10 0x7f0d240d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 --2- 172.21.15.129:0/375434357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2415cd30 0x7f0d240cfd10 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 -- 172.21.15.129:0/375434357 >> 172.21.15.129:0/375434357 conn(0x7f0d2406cfc0 msgr2=0x7f0d2406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 -- 172.21.15.129:0/375434357 shutdown_connections 2024-09-17T04:04:04.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.643+0000 7f0d2a000640 1 -- 172.21.15.129:0/375434357 wait complete. 2024-09-17T04:04:04.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.645+0000 7f0d2a000640 1 Processor -- start 2024-09-17T04:04:04.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.645+0000 7f0d2a000640 1 -- start start 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2406f800 0x7f0d2414bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 0x7f0d24157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24073890 con 0x7f0d2406fc10 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d24073710 con 0x7f0d2406f800 2024-09-17T04:04:04.647 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d2a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0d24073a10 con 0x7f0d2415cd30 2024-09-17T04:04:04.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38680/0 (socket says 172.21.15.129:38680) 2024-09-17T04:04:04.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d1aa00640 1 -- 172.21.15.129:0/3186054480 learned_addr learned my addr 172.21.15.129:0/3186054480 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:04.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d23400640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2406f800 0x7f0d2414bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.648 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.647+0000 7f0d23e00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 0x7f0d24157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 -- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 msgr2=0x7f0d24157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 0x7f0d24157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 -- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2406f800 msgr2=0x7f0d2414bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2406f800 0x7f0d2414bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 -- 172.21.15.129:0/3186054480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d241562c0 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d23e00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 0x7f0d24157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1aa00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c3f0 tx=0x7f0d0c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c017070 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d0c0035f0 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d24156560 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c003a40 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.648+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d24154870 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.649+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d240ced60 con 0x7f0d2406fc10 2024-09-17T04:04:04.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.651+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d0c01a070 con 0x7f0d2406fc10 2024-09-17T04:04:04.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.651+0000 7f0d1be00640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 0x7f0d0007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:04.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.652+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f0d0c09a090 con 0x7f0d2406fc10 2024-09-17T04:04:04.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.652+0000 7f0d23400640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 0x7f0d0007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:04.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.653+0000 7f0d23400640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 0x7f0d0007b0d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0d14033000 tx=0x7f0d14004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:04.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.656+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d240ced60 con 0x7f0d2406fc10 2024-09-17T04:04:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.874+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d24073f00 con 0x7f0d2406fc10 2024-09-17T04:04:04.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.875+0000 7f0d1be00640 1 -- 172.21.15.129:0/3186054480 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+2769 (secure 0 0 0) 0x7f0d0c061e40 con 0x7f0d2406fc10 2024-09-17T04:04:04.875 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:04.875 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":187,"btime":"2024-09-17T04:04:04:378486+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:04.378481+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:04.876 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 187 2024-09-17T04:04:04.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 msgr2=0x7f0d0007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 0x7f0d0007b0d0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0d14033000 tx=0x7f0d14004730 comp rx=0 tx=0).stop 2024-09-17T04:04:04.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 msgr2=0x7f0d24155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c3f0 tx=0x7f0d0c00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 shutdown_connections 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d00078bc0 0x7f0d0007b0d0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d2415cd30 0x7f0d24157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d2406fc10 0x7f0d24155350 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 --2- 172.21.15.129:0/3186054480 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d2406f800 0x7f0d2414bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 >> 172.21.15.129:0/3186054480 conn(0x7f0d2406cfc0 msgr2=0x7f0d2415e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 shutdown_connections 2024-09-17T04:04:04.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:04.878+0000 7f0d2a000640 1 -- 172.21.15.129:0/3186054480 wait complete. 2024-09-17T04:04:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 187, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:04.378481+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10440}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:05.101 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2024-09-17T04:04:06.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:06.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 -- 172.21.15.129:0/368759383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 msgr2=0x7f2e90004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:06.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 --2- 172.21.15.129:0/368759383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2e90004800 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009b50 tx=0x7f2e90005830 comp rx=0 tx=0).stop 2024-09-17T04:04:06.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eab400640 1 -- 172.21.15.129:0/368759383 reap_dead start 2024-09-17T04:04:06.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 -- 172.21.15.129:0/368759383 shutdown_connections 2024-09-17T04:04:06.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 -- 172.21.15.129:0/368759383 >> 172.21.15.129:0/368759383 conn(0x7f2eac0b2510 msgr2=0x7f2eac0d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:06.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 -- 172.21.15.129:0/368759383 shutdown_connections 2024-09-17T04:04:06.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.332+0000 7f2eabe00640 1 -- 172.21.15.129:0/368759383 wait complete. 2024-09-17T04:04:06.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.334+0000 7f2eabe00640 1 Processor -- start 2024-09-17T04:04:06.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.334+0000 7f2eabe00640 1 -- start start 2024-09-17T04:04:06.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:06.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0cfcf0 0x7f2eac15f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:06.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac09b870 0x7f2eac15f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac06fe30 con 0x7f2e90003ff0 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac06fcb0 con 0x7f2eac09b870 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eabe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac06ffb0 con 0x7f2eac0cfcf0 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38698/0 (socket says 172.21.15.129:38698) 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 -- 172.21.15.129:0/3443938345 learned_addr learned my addr 172.21.15.129:0/3443938345 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2ea2000640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0cfcf0 0x7f2eac15f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eab400640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac09b870 0x7f2eac15f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 -- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0cfcf0 msgr2=0x7f2eac15f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0cfcf0 0x7f2eac15f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 -- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac09b870 msgr2=0x7f2eac15f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:06.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac09b870 0x7f2eac15f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.336+0000 7f2eaaa00640 1 -- 172.21.15.129:0/3443938345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0c9e90 con 0x7f2e90003ff0 2024-09-17T04:04:06.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2eaaa00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c011a50 tx=0x7f2e9c011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c01f070 con 0x7f2e90003ff0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e9c00fe00 con 0x7f2e90003ff0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c012690 con 0x7f2e90003ff0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac15e410 con 0x7f2e90003ff0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.337+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac15e9d0 con 0x7f2e90003ff0 2024-09-17T04:04:06.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.338+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac15c830 con 0x7f2e90003ff0 2024-09-17T04:04:06.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.339+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2e9c022070 con 0x7f2e90003ff0 2024-09-17T04:04:06.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.340+0000 7f2ea3e00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:06.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.340+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f2e9c09b810 con 0x7f2e90003ff0 2024-09-17T04:04:06.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.341+0000 7f2ea2000640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:06.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.341+0000 7f2ea2000640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2e94035000 tx=0x7f2e940046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:06.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.345+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2e9c063500 con 0x7f2e90003ff0 2024-09-17T04:04:06.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.566+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eac15cad0 con 0x7f2e90003ff0 2024-09-17T04:04:06.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.566+0000 7f2ea3e00640 1 -- 172.21.15.129:0/3443938345 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) ==== 95+0+3610 (secure 0 0 0) 0x7f2e9c067f30 con 0x7f2e90003ff0 2024-09-17T04:04:06.567 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:06.568 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":190,"btime":"2024-09-17T04:04:06:390311+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:05.403963+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:resolve","state_seq":2,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:06.568 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 190 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 msgr2=0x7f2e8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f2e94035000 tx=0x7f2e940046c0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 msgr2=0x7f2eac0cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c011a50 tx=0x7f2e9c011f10 comp rx=0 tx=0).stop 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 shutdown_connections 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac09b870 0x7f2eac15f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0cfcf0 0x7f2eac15f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 --2- 172.21.15.129:0/3443938345 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2e90003ff0 0x7f2eac0cf7b0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 >> 172.21.15.129:0/3443938345 conn(0x7f2eac0b2510 msgr2=0x7f2eac0ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 shutdown_connections 2024-09-17T04:04:06.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:06.569+0000 7f2eabe00640 1 -- 172.21.15.129:0/3443938345 wait complete. 2024-09-17T04:04:06.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 190, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:05.403963+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:06.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:resolve 2024-09-17T04:04:07.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.019+0000 7f9447000640 1 -- 172.21.15.129:0/2630133083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944006fde0 msgr2=0x7f9440072fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.019+0000 7f9447000640 1 --2- 172.21.15.129:0/2630133083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944006fde0 0x7f9440072fa0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9438005fd0 tx=0x7f943802aa90 comp rx=0 tx=0).stop 2024-09-17T04:04:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 -- 172.21.15.129:0/2630133083 shutdown_connections 2024-09-17T04:04:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 --2- 172.21.15.129:0/2630133083 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944006fde0 0x7f9440072fa0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 --2- 172.21.15.129:0/2630133083 >> v2:172.21.15.181:3300/0 conn(0x7f944015c830 0x7f944006f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 -- 172.21.15.129:0/2630133083 >> 172.21.15.129:0/2630133083 conn(0x7f94400b2510 msgr2=0x7f94400d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 -- 172.21.15.129:0/2630133083 shutdown_connections 2024-09-17T04:04:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.020+0000 7f9447000640 1 -- 172.21.15.129:0/2630133083 wait complete. 2024-09-17T04:04:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.022+0000 7f9447000640 1 Processor -- start 2024-09-17T04:04:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.022+0000 7f9447000640 1 -- start start 2024-09-17T04:04:08.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:08.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94400d0f20 0x7f94401e5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:08.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944015c830 0x7f94401f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f94400b4570 con 0x7f944006fde0 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f94400b43f0 con 0x7f94400d0f20 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.023+0000 7f9447000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f94400b46f0 con 0x7f944015c830 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38706/0 (socket says 172.21.15.129:38706) 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 -- 172.21.15.129:0/987108373 learned_addr learned my addr 172.21.15.129:0/987108373 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445200640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94400d0f20 0x7f94401e5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9446600640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944015c830 0x7f94401f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 -- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944015c830 msgr2=0x7f94401f95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944015c830 0x7f94401f95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 -- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94400d0f20 msgr2=0x7f94401e5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94400d0f20 0x7f94401e5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.024+0000 7f9445c00640 1 -- 172.21.15.129:0/987108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94401e7ad0 con 0x7f944006fde0 2024-09-17T04:04:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f9445c00640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f943400c430 tx=0x7f943400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434017070 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f94340053e0 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9434003850 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94401e7db0 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.025+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944015a950 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.026+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94400b4c10 con 0x7f944006fde0 2024-09-17T04:04:08.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.027+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f943401a070 con 0x7f944006fde0 2024-09-17T04:04:08.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.028+0000 7f943d600640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 0x7f942007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:08.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.028+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f943409a090 con 0x7f944006fde0 2024-09-17T04:04:08.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.029+0000 7f9445200640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 0x7f942007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:08.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.030+0000 7f9445200640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 0x7f942007b0a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f94400c9fa0 tx=0x7f942c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:08.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.033+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9434061e10 con 0x7f944006fde0 2024-09-17T04:04:08.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.249+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9440074cc0 con 0x7f944006fde0 2024-09-17T04:04:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.250+0000 7f943d600640 1 -- 172.21.15.129:0/987108373 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+3612 (secure 0 0 0) 0x7f9434066840 con 0x7f944006fde0 2024-09-17T04:04:08.250 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:08.250 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":191,"btime":"2024-09-17T04:04:07:392766+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:06.396513+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 191 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 msgr2=0x7f942007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 0x7f942007b0a0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f94400c9fa0 tx=0x7f942c005f90 comp rx=0 tx=0).stop 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 msgr2=0x7f944015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f943400c430 tx=0x7f943400c8f0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 shutdown_connections 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9420078b90 0x7f942007b0a0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f944015c830 0x7f94401f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f94400d0f20 0x7f94401e5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 --2- 172.21.15.129:0/987108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f944006fde0 0x7f944015a410 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:08.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.252+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 >> 172.21.15.129:0/987108373 conn(0x7f94400b2510 msgr2=0x7f944015baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:08.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.253+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 shutdown_connections 2024-09-17T04:04:08.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:08.253+0000 7f9447000640 1 -- 172.21.15.129:0/987108373 wait complete. 2024-09-17T04:04:08.481 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 191, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:06.396513+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:reconnect', 'state_seq': 3, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:08.481 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:reconnect 2024-09-17T04:04:09.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 -- 172.21.15.129:0/3664080062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c0cfd10 msgr2=0x7ff21c0cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 --2- 172.21.15.129:0/3664080062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c0cfd10 0x7ff21c0cd8e0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7ff208005fd0 tx=0x7ff20802a5c0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 -- 172.21.15.129:0/3664080062 shutdown_connections 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 --2- 172.21.15.129:0/3664080062 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0d0620 0x7ff21c0d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 --2- 172.21.15.129:0/3664080062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c0cfd10 0x7ff21c0cd8e0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.703+0000 7ff220c00640 1 -- 172.21.15.129:0/3664080062 >> 172.21.15.129:0/3664080062 conn(0x7ff21c06cfc0 msgr2=0x7ff21c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:09.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.704+0000 7ff220c00640 1 -- 172.21.15.129:0/3664080062 shutdown_connections 2024-09-17T04:04:09.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.704+0000 7ff220c00640 1 -- 172.21.15.129:0/3664080062 wait complete. 2024-09-17T04:04:09.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.706+0000 7ff220c00640 1 Processor -- start 2024-09-17T04:04:09.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.706+0000 7ff220c00640 1 -- start start 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 0x7ff21c14aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 0x7ff21c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c073890 con 0x7ff21c15cd30 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff21c073710 con 0x7ff21c0d0620 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.707+0000 7ff220c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff21c073a10 con 0x7ff21c0cfd10 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:09.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38724/0 (socket says 172.21.15.129:38724) 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 -- 172.21.15.129:0/3712233558 learned_addr learned my addr 172.21.15.129:0/3712233558 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff219e00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 0x7ff21c14aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff219400640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 0x7ff21c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 -- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 msgr2=0x7ff21c14aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 0x7ff21c14aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 -- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 msgr2=0x7ff21c158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 0x7ff21c158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 -- 172.21.15.129:0/3712233558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21c156400 con 0x7ff21c15cd30 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff219e00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 0x7ff21c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:09.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff219400640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 0x7ff21c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.708+0000 7ff21a800640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7ff21400e990 tx=0x7ff21400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.709+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214004d20 con 0x7ff21c15cd30 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.709+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff21400caa0 con 0x7ff21c15cd30 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.709+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff214005020 con 0x7ff21c15cd30 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.709+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff21c1566c0 con 0x7ff21c15cd30 2024-09-17T04:04:09.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.709+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21c159ba0 con 0x7ff21c15cd30 2024-09-17T04:04:09.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.710+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff21c073f50 con 0x7ff21c15cd30 2024-09-17T04:04:09.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.711+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff2140051c0 con 0x7ff21c15cd30 2024-09-17T04:04:09.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.712+0000 7ff212000640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 0x7ff1fc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:09.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.712+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7ff21409b6d0 con 0x7ff21c15cd30 2024-09-17T04:04:09.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.713+0000 7ff219e00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 0x7ff1fc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:09.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.713+0000 7ff219e00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 0x7ff1fc07b0a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff20c000f60 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:09.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.717+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff214016030 con 0x7ff21c15cd30 2024-09-17T04:04:09.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.935+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff21c0cd4d0 con 0x7ff21c15cd30 2024-09-17T04:04:09.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.936+0000 7ff212000640 1 -- 172.21.15.129:0/3712233558 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3614 (secure 0 0 0) 0x7ff214063aa0 con 0x7ff21c15cd30 2024-09-17T04:04:09.936 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:09.936 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":193,"btime":"2024-09-17T04:04:09:409438+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:09.409432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[9895,10440]},"id":15}]} 2024-09-17T04:04:09.936 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 193 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 msgr2=0x7ff1fc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 0x7ff1fc07b0a0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7ff20c000f60 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 msgr2=0x7ff21c159060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7ff21400e990 tx=0x7ff21400ee50 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 shutdown_connections 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1fc078b90 0x7ff1fc07b0a0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff21c15cd30 0x7ff21c159060 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff21c0d0620 0x7ff21c158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 --2- 172.21.15.129:0/3712233558 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff21c0cfd10 0x7ff21c14aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 >> 172.21.15.129:0/3712233558 conn(0x7ff21c06cfc0 msgr2=0x7ff21c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:09.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 shutdown_connections 2024-09-17T04:04:09.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:09.938+0000 7ff220c00640 1 -- 172.21.15.129:0/3712233558 wait complete. 2024-09-17T04:04:10.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 193, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:09.409432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [9895, 10440]} 2024-09-17T04:04:10.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-09-17T04:04:10.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:04:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 -- 172.21.15.129:0/4096565266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e44004d40 msgr2=0x7f9e44005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 --2- 172.21.15.129:0/4096565266 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e44004d40 0x7f9e44005150 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c610 tx=0x7f9e44005830 comp rx=0 tx=0).stop 2024-09-17T04:04:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5e800640 1 -- 172.21.15.129:0/4096565266 reap_dead start 2024-09-17T04:04:10.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 -- 172.21.15.129:0/4096565266 shutdown_connections 2024-09-17T04:04:10.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 -- 172.21.15.129:0/4096565266 >> 172.21.15.129:0/4096565266 conn(0x7f9e6008bfb0 msgr2=0x7f9e600d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 -- 172.21.15.129:0/4096565266 shutdown_connections 2024-09-17T04:04:10.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.389+0000 7f9e5f200640 1 -- 172.21.15.129:0/4096565266 wait complete. 2024-09-17T04:04:10.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.391+0000 7f9e5f200640 1 Processor -- start 2024-09-17T04:04:10.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.391+0000 7f9e5f200640 1 -- start start 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e44004d40 0x7f9e601f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e60158c80 0x7f9e601f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600d2e60 con 0x7f9e60158350 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e600b6d30 con 0x7f9e60158c80 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e600d2fe0 con 0x7f9e44004d40 2024-09-17T04:04:10.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38742/0 (socket says 172.21.15.129:38742) 2024-09-17T04:04:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 -- 172.21.15.129:0/3048728298 learned_addr learned my addr 172.21.15.129:0/3048728298 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 -- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e44004d40 msgr2=0x7f9e601f5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e44004d40 0x7f9e601f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 -- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e60158c80 msgr2=0x7f9e601f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e60158c80 0x7f9e601f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.393+0000 7f9e5e800640 1 -- 172.21.15.129:0/3048728298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e600cf0e0 con 0x7f9e60158350 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e5e800640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c700 tx=0x7f9e58003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e56a00640 1 -- 172.21.15.129:0/3048728298 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58039070 con 0x7f9e60158350 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e56a00640 1 -- 172.21.15.129:0/3048728298 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e5802bca0 con 0x7f9e60158350 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e600cf3c0 con 0x7f9e60158350 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e56a00640 1 -- 172.21.15.129:0/3048728298 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e580063d0 con 0x7f9e60158350 2024-09-17T04:04:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.394+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e600cf980 con 0x7f9e60158350 2024-09-17T04:04:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.397+0000 7f9e56a00640 1 -- 172.21.15.129:0/3048728298 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e58006570 con 0x7f9e60158350 2024-09-17T04:04:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.398+0000 7f9e56a00640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 0x7f9e3c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.398+0000 7f9e56a00640 1 -- 172.21.15.129:0/3048728298 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f9e580bb610 con 0x7f9e60158350 2024-09-17T04:04:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.398+0000 7f9e6be68740 1 Processor -- start 2024-09-17T04:04:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.398+0000 7f9e6be68740 1 -- start start 2024-09-17T04:04:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.398+0000 7f9e5de00640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 0x7f9e3c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.399+0000 7f9e5de00640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 0x7f9e3c07b0a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009aa0 tx=0x7f9e50009540 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b02f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005afff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56005af88430 con 0x56005ac5e270 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56005af785d0 con 0x56005ac353b0 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56005af67a00 con 0x56005ac3d590 2024-09-17T04:04:10.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38752/0 (socket says 172.21.15.129:38752) 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 -- 172.21.15.129:0/1917455654 learned_addr learned my addr 172.21.15.129:0/1917455654 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e55400640 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b02f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 -- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 msgr2=0x56005afff150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005afff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 -- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 msgr2=0x56005b02f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b02f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.400+0000 7f9e5de00640 1 -- 172.21.15.129:0/1917455654 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56005af30520 con 0x56005ac5e270 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.401+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f9e50011f30 tx=0x7f9e50014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea06c8549e7ce81d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.401+0000 7f9e4ea00640 1 -- 172.21.15.129:0/1917455654 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e500047a0 con 0x56005ac5e270 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.401+0000 7f9e4ea00640 1 -- 172.21.15.129:0/1917455654 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e50022030 con 0x56005ac5e270 2024-09-17T04:04:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.401+0000 7f9e4ea00640 1 -- 172.21.15.129:0/1917455654 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50004c90 con 0x56005ac5e270 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 -- 172.21.15.129:0/1917455654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 msgr2=0x56005b02eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f9e50011f30 tx=0x7f9e50014000 comp rx=0 tx=0).stop 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 -- 172.21.15.129:0/1917455654 shutdown_connections 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005afff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b02f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1917455654 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac5e270 0x56005b02eb60 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 -- 172.21.15.129:0/1917455654 >> 172.21.15.129:0/1917455654 conn(0x56005ac523c0 msgr2=0x56005b004b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 -- 172.21.15.129:0/1917455654 shutdown_connections 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.402+0000 7f9e6be68740 1 -- 172.21.15.129:0/1917455654 wait complete. 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.403+0000 7f9e6be68740 1 -- start start 2024-09-17T04:04:10.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.403+0000 7f9e6be68740 1 Processor -- start 2024-09-17T04:04:10.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005b040af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 0x56005b02c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56005af0e330 con 0x56005ac353b0 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56005af6b810 con 0x56005ac5e270 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56005acd8620 con 0x56005ac3d590 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e6be68740 10 client.0 _ll_register_callbacks cb 0x56005af13420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38766/0 (socket says 172.21.15.129:38766) 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 -- 172.21.15.129:0/3385271172 learned_addr learned my addr 172.21.15.129:0/3385271172 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e55400640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005b040af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5e800640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 0x56005b02c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 -- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 msgr2=0x56005b040af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005b040af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 -- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 msgr2=0x56005b02c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 0x56005b02c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5de00640 1 -- 172.21.15.129:0/3385271172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56005acc9850 con 0x56005ac353b0 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.405+0000 7f9e5e800640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 0x56005b02c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e5de00640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f9e50004f70 tx=0x7f9e50004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9432e6fae322bb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50005690 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f9e50012450 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e50022030 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7f9e50039050 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e6be68740 10 client.10677 fetch_fsmap learned FSMap version 193 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) -- 0x56005ad406b0 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e50022030 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 193) ==== 2312+0+0 (secure 0 0 0) 0x7f9e5003c020 con 0x56005ac353b0 2024-09-17T04:04:10.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e6be68740 10 client.10677 fetch_fsmap finished waiting for FSMap version 193 2024-09-17T04:04:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.406+0000 7f9e6be68740 10 client.10677 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:04:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e6be68740 20 client.10677 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005ac1ce40 0x56005b05c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e6be68740 4 client.10677 mds_command: new command op to 9895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:04:10.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 --> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56005af7a520 con 0x56005ac1ce40 2024-09-17T04:04:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e55400640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005ac1ce40 0x56005b05c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e55400640 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005ac1ce40 0x56005b05c080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfafa9f84cc41d25 server_cookie=77c74d171c10d0c1 in_seq=0 out_seq=0 2024-09-17T04:04:10.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.407+0000 7f9e43400640 10 client.10677 ms_handle_connect on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:10.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.408+0000 7f9e43400640 1 -- 172.21.15.129:0/3385271172 <== mds.1 v2:172.21.15.129:6832/1714996749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56005af7a520 con 0x56005ac1ce40 2024-09-17T04:04:10.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.408+0000 7f9e43400640 10 client.10677 handle_command_reply: tid=0 2024-09-17T04:04:10.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.408+0000 7f9e6be68740 1 client.10677 shutdown 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 msgr2=0x56005b043780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f9e50004f70 tx=0x7f9e50004fa0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 shutdown_connections 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac5e270 0x56005b02c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac3d590 0x56005b040af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005ac1ce40 0x56005b05c080 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 --2- 172.21.15.129:0/3385271172 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b043780 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e43400640 0 client.10677 ms_handle_reset on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:10.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 >> 172.21.15.129:0/3385271172 conn(0x56005ac523c0 msgr2=0x56005b040060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 shutdown_connections 2024-09-17T04:04:10.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 1 -- 172.21.15.129:0/3385271172 wait complete. 2024-09-17T04:04:10.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.409+0000 7f9e6be68740 20 client.10677 trim_cache size 0 max 16384 2024-09-17T04:04:10.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.429+0000 7f9e6be68740 1 Processor -- start 2024-09-17T04:04:10.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.429+0000 7f9e6be68740 1 -- start start 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b05c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60201020 con 0x56005ac3d590 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e602016b0 con 0x56005ac353b0 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.430+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56005b041030 con 0x56005ac5e270 2024-09-17T04:04:10.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38772/0 (socket says 172.21.15.129:38772) 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 -- 172.21.15.129:0/649695234 learned_addr learned my addr 172.21.15.129:0/649695234 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e5de00640 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b05c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e5e800640 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 -- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 msgr2=0x56005b06c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 -- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 msgr2=0x56005b05c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b05c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 -- 172.21.15.129:0/649695234 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56005b000b30 con 0x56005ac3d590 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e55400640 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f9e600b8920 tx=0x7f9e48014470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1440e743114e043 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e4e000640 1 -- 172.21.15.129:0/649695234 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e480141d0 con 0x56005ac3d590 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e4e000640 1 -- 172.21.15.129:0/649695234 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e48014b90 con 0x56005ac3d590 2024-09-17T04:04:10.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.431+0000 7f9e4e000640 1 -- 172.21.15.129:0/649695234 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48004030 con 0x56005ac3d590 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 -- 172.21.15.129:0/649695234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 msgr2=0x56005b04ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 secure :-1 s=READY pgs=2232 cs=0 l=1 rev1=1 crypto rx=0x7f9e600b8920 tx=0x7f9e48014470 comp rx=0 tx=0).stop 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 -- 172.21.15.129:0/649695234 shutdown_connections 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac3d590 0x56005b04ac00 unknown :-1 s=CLOSED pgs=2232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 --2- 172.21.15.129:0/649695234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac353b0 0x56005b05c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 -- 172.21.15.129:0/649695234 >> 172.21.15.129:0/649695234 conn(0x56005ac523c0 msgr2=0x56005b042d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 -- 172.21.15.129:0/649695234 shutdown_connections 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.432+0000 7f9e6be68740 1 -- 172.21.15.129:0/649695234 wait complete. 2024-09-17T04:04:10.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.433+0000 7f9e6be68740 1 -- start start 2024-09-17T04:04:10.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.434+0000 7f9e6be68740 1 Processor -- start 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 0x56005b0bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56005b05c820 con 0x56005ac353b0 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56005b02c640 con 0x56005ac3d590 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56005b03fce0 con 0x56005ac5e270 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e6be68740 10 client.0 _ll_register_callbacks cb 0x7f9e50011e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38782/0 (socket says 172.21.15.129:38782) 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.435+0000 7f9e5de00640 1 -- 172.21.15.129:0/1115170046 learned_addr learned my addr 172.21.15.129:0/1115170046 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5e800640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e55400640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 0x56005b0bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 -- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 msgr2=0x56005b06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 -- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 msgr2=0x56005b0bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 0x56005b0bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 -- 172.21.15.129:0/1115170046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56005b0266a0 con 0x56005ac353b0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e55400640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 0x56005b0bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e5de00640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f9e50004f70 tx=0x7f9e50011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c4211787a53af server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e43400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e50032a50 con 0x56005ac353b0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f9e5003eb00 con 0x56005ac353b0 2024-09-17T04:04:10.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.436+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e48014b90 con 0x56005ac353b0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7f9e5003ed00 con 0x56005ac353b0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 10 client.10683 fetch_fsmap learned FSMap version 193 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) -- 0x56005b03fae0 con 0x56005ac353b0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e48014b90 con 0x56005ac353b0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 193) ==== 2312+0+0 (secure 0 0 0) 0x7f9e5003c070 con 0x56005ac353b0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 10 client.10683 fetch_fsmap finished waiting for FSMap version 193 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 10 client.10683 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 20 client.10683 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005b0c96f0 0x56005b0c9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 4 client.10683 mds_command: new command op to 9895 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 --> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] -- command(tid 0: {"prefix": "status"}) -- 0x56005af75fd0 con 0x56005b0c96f0 2024-09-17T04:04:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.437+0000 7f9e55400640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005b0c96f0 0x56005b0c9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e55400640 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005b0c96f0 0x56005b0c9b00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96f0efafec4ebf3f server_cookie=3779571f20a8c829 in_seq=0 out_seq=0 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e43400640 10 client.10683 ms_handle_connect on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e43400640 1 -- 172.21.15.129:0/1115170046 <== mds.1 v2:172.21.15.129:6832/1714996749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56005af75fd0 con 0x56005b0c96f0 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e43400640 10 client.10683 handle_command_reply: tid=0 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 client.10683 shutdown 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 msgr2=0x56005b05d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f9e50004f70 tx=0x7f9e50011d30 comp rx=0 tx=0).stop 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 shutdown_connections 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56005ac5e270 0x56005b06c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56005ac3d590 0x56005b0bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x56005b0c96f0 0x56005b0c9b00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 --2- 172.21.15.129:0/1115170046 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56005ac353b0 0x56005b05d100 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 >> 172.21.15.129:0/1115170046 conn(0x56005ac523c0 msgr2=0x56005b0049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.438+0000 7f9e43400640 0 client.10683 ms_handle_reset on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 shutdown_connections 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e6be68740 1 -- 172.21.15.129:0/1115170046 wait complete. 2024-09-17T04:04:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e6be68740 20 client.10683 trim_cache size 0 max 16384 2024-09-17T04:04:10.448 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:04:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:04:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 1, 2024-09-17T04:04:10.448 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 9895, 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 5.0449793319999996, 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 193, 2024-09-17T04:04:10.449 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 206, 2024-09-17T04:04:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 206, 2024-09-17T04:04:10.450 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 5.822611277 2024-09-17T04:04:10.450 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:04:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 msgr2=0x7f9e3c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 0x7f9e3c07b0a0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009aa0 tx=0x7f9e50009540 comp rx=0 tx=0).stop 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 msgr2=0x7f9e600d0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f9e5800c700 tx=0x7f9e58003ba0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 shutdown_connections 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e3c078b90 0x7f9e3c07b0a0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e60158350 0x7f9e600d0180 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e60158c80 0x7f9e601f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 --2- 172.21.15.129:0/3048728298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e44004d40 0x7f9e601f5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.439+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 >> 172.21.15.129:0/3048728298 conn(0x7f9e6008bfb0 msgr2=0x7f9e600b8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.440+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 shutdown_connections 2024-09-17T04:04:10.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.440+0000 7f9e5f200640 1 -- 172.21.15.129:0/3048728298 wait complete. 2024-09-17T04:04:10.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.683+0000 7f8018600640 1 -- 172.21.15.129:0/1435005950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 msgr2=0x7f80100cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.683+0000 7f8018600640 1 --2- 172.21.15.129:0/1435005950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80100cd8e0 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc005fd0 tx=0x7f7ffc02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.683+0000 7f8018600640 1 -- 172.21.15.129:0/1435005950 shutdown_connections 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.683+0000 7f8018600640 1 --2- 172.21.15.129:0/1435005950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80100cd8e0 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.683+0000 7f8018600640 1 --2- 172.21.15.129:0/1435005950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801015cd30 0x7f80100cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.684+0000 7f8018600640 1 -- 172.21.15.129:0/1435005950 >> 172.21.15.129:0/1435005950 conn(0x7f801006cfc0 msgr2=0x7f801006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.684+0000 7f8018600640 1 -- 172.21.15.129:0/1435005950 shutdown_connections 2024-09-17T04:04:10.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.684+0000 7f8018600640 1 -- 172.21.15.129:0/1435005950 wait complete. 2024-09-17T04:04:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.686+0000 7f8018600640 1 Processor -- start 2024-09-17T04:04:10.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.686+0000 7f8018600640 1 -- start start 2024-09-17T04:04:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801006fc10 0x7f8010143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 0x7f80101f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8010073890 con 0x7f801006f800 2024-09-17T04:04:10.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8010073710 con 0x7f801015cd30 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.687+0000 7f8018600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8010073a10 con 0x7f801006fc10 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38790/0 (socket says 172.21.15.129:38790) 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 -- 172.21.15.129:0/1126633152 learned_addr learned my addr 172.21.15.129:0/1126633152 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015200640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801006fc10 0x7f8010143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8016600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 0x7f80101f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 -- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801006fc10 msgr2=0x7f8010143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801006fc10 0x7f8010143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 -- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 msgr2=0x7f80101f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 0x7f80101f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 -- 172.21.15.129:0/1126633152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80100ad930 con 0x7f801006f800 2024-09-17T04:04:10.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8016600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 0x7f80101f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.688+0000 7f8015c00640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f800400e790 tx=0x7f800400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.689+0000 7f800d600640 1 -- 172.21.15.129:0/1126633152 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800400eea0 con 0x7f801006f800 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.689+0000 7f800d600640 1 -- 172.21.15.129:0/1126633152 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f800400cdb0 con 0x7f801006f800 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.689+0000 7f800d600640 1 -- 172.21.15.129:0/1126633152 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8004010690 con 0x7f801006f800 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.689+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80100adc10 con 0x7f801006f800 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.689+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80100ae1d0 con 0x7f801006f800 2024-09-17T04:04:10.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.691+0000 7f800d600640 1 -- 172.21.15.129:0/1126633152 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8004010830 con 0x7f801006f800 2024-09-17T04:04:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.692+0000 7f800d600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 0x7f7ff007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.692+0000 7f800d600640 1 -- 172.21.15.129:0/1126633152 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f800409b6c0 con 0x7f801006f800 2024-09-17T04:04:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.693+0000 7f801f80b740 1 Processor -- start 2024-09-17T04:04:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.693+0000 7f8015200640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 0x7f7ff007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.693+0000 7f801f80b740 1 -- start start 2024-09-17T04:04:10.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.693+0000 7f8015200640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 0x7f7ff007b0d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc033000 tx=0x7f7ffc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d906970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d906eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55707d860430 con 0x55707d515590 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55707d8505d0 con 0x55707d536270 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55707d83fa00 con 0x55707d50d3b0 2024-09-17T04:04:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38796/0 (socket says 172.21.15.129:38796) 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f8016600640 1 -- 172.21.15.129:0/4026976369 learned_addr learned my addr 172.21.15.129:0/4026976369 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f8015c00640 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d906970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.694+0000 7f8015200640 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d906eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 -- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 msgr2=0x55707d906eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d906eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 -- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 msgr2=0x55707d906970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d906970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 -- 172.21.15.129:0/4026976369 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707d808520 con 0x55707d515590 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8016600640 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f8008001c90 tx=0x7f8008002150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d035d891bea430 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8002000640 1 -- 172.21.15.129:0/4026976369 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008019070 con 0x55707d515590 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8002000640 1 -- 172.21.15.129:0/4026976369 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8008003b80 con 0x55707d515590 2024-09-17T04:04:10.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.695+0000 7f8002000640 1 -- 172.21.15.129:0/4026976369 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800800e8c0 con 0x55707d515590 2024-09-17T04:04:10.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 -- 172.21.15.129:0/4026976369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 msgr2=0x55707d8d6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f8008001c90 tx=0x7f8008002150 comp rx=0 tx=0).stop 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 -- 172.21.15.129:0/4026976369 shutdown_connections 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d515590 0x55707d8d6f60 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d906eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 --2- 172.21.15.129:0/4026976369 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d906970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 -- 172.21.15.129:0/4026976369 >> 172.21.15.129:0/4026976369 conn(0x55707d52a3c0 msgr2=0x55707d8dc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 -- 172.21.15.129:0/4026976369 shutdown_connections 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.696+0000 7f801f80b740 1 -- 172.21.15.129:0/4026976369 wait complete. 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.697+0000 7f801f80b740 1 -- start start 2024-09-17T04:04:10.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.697+0000 7f801f80b740 1 Processor -- start 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d9313a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d515590 0x55707d900640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55707d7e6330 con 0x55707d536270 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55707d843810 con 0x55707d515590 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55707d5b0620 con 0x55707d50d3b0 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f801f80b740 10 client.0 _ll_register_callbacks cb 0x55707d7eb420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38800/0 (socket says 172.21.15.129:38800) 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.699+0000 7f8016600640 1 -- 172.21.15.129:0/3517398802 learned_addr learned my addr 172.21.15.129:0/3517398802 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8015c00640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d9313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8015200640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d515590 0x55707d900640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 -- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 msgr2=0x55707d9313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d9313a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 -- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d515590 msgr2=0x55707d900640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d515590 0x55707d900640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 -- 172.21.15.129:0/3517398802 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707d5a1850 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f8016600640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f8008009010 tx=0x7f8008006b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b7583c012660227 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f7ff6a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008019040 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f800800e8c0 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.700+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8008003b80 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7f800802a050 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 10 client.10695 fetch_fsmap learned FSMap version 193 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) -- 0x55707d6186b0 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8008003b80 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 193) ==== 2312+0+0 (secure 0 0 0) 0x7f800802d020 con 0x55707d536270 2024-09-17T04:04:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 10 client.10695 fetch_fsmap finished waiting for FSMap version 193 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 10 client.10695 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 20 client.10695 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d4f4e40 0x55707d923d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 4 client.10695 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.701+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55707d852520 con 0x55707d4f4e40 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.702+0000 7f8015c00640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d4f4e40 0x55707d923d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.702+0000 7f7ff6a00640 10 client.10695 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.702+0000 7f8015c00640 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d4f4e40 0x55707d923d70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56347f6fc8648566 server_cookie=87cef581a97692c8 in_seq=0 out_seq=0 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f7ff6a00640 1 -- 172.21.15.129:0/3517398802 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55707d852520 con 0x55707d4f4e40 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f7ff6a00640 10 client.10695 handle_command_reply: tid=0 2024-09-17T04:04:10.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 client.10695 shutdown 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 msgr2=0x55707d91b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f8008009010 tx=0x7f8008006b50 comp rx=0 tx=0).stop 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 shutdown_connections 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d4f4e40 0x55707d923d70 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d91b460 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d515590 0x55707d900640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 --2- 172.21.15.129:0/3517398802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d50d3b0 0x55707d9313a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.703+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 >> 172.21.15.129:0/3517398802 conn(0x55707d52a3c0 msgr2=0x55707d8d82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.704+0000 7f7ff6a00640 0 client.10695 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.704+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 shutdown_connections 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.704+0000 7f801f80b740 1 -- 172.21.15.129:0/3517398802 wait complete. 2024-09-17T04:04:10.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.704+0000 7f801f80b740 20 client.10695 trim_cache size 0 max 16384 2024-09-17T04:04:10.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.723+0000 7f801f80b740 1 Processor -- start 2024-09-17T04:04:10.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.723+0000 7f801f80b740 1 -- start start 2024-09-17T04:04:10.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d50d3b0 0x55707d8fc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d924ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f80102011d0 con 0x55707d536270 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8010201820 con 0x55707d50d3b0 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55707d9022b0 con 0x55707d515590 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f8016600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38802/0 (socket says 172.21.15.129:38802) 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.724+0000 7f8016600640 1 -- 172.21.15.129:0/1159645371 learned_addr learned my addr 172.21.15.129:0/1159645371 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8015c00640 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d50d3b0 0x55707d8fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8015200640 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d924ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 -- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 msgr2=0x55707d924ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d924ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 -- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d50d3b0 msgr2=0x55707d8fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d50d3b0 0x55707d8fc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 -- 172.21.15.129:0/1159645371 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55707d921ca0 con 0x55707d536270 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8016600640 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f8008001220 tx=0x7f8008006bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c160aed28924 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.725+0000 7f8001600640 1 -- 172.21.15.129:0/1159645371 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008033070 con 0x55707d536270 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f8001600640 1 -- 172.21.15.129:0/1159645371 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8008022070 con 0x55707d536270 2024-09-17T04:04:10.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f8001600640 1 -- 172.21.15.129:0/1159645371 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8008019070 con 0x55707d536270 2024-09-17T04:04:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 -- 172.21.15.129:0/1159645371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 msgr2=0x55707d938da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 secure :-1 s=READY pgs=2238 cs=0 l=1 rev1=1 crypto rx=0x7f8008001220 tx=0x7f8008006bc0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 -- 172.21.15.129:0/1159645371 shutdown_connections 2024-09-17T04:04:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d536270 0x55707d938da0 unknown :-1 s=CLOSED pgs=2238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d924ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 --2- 172.21.15.129:0/1159645371 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d50d3b0 0x55707d8fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 -- 172.21.15.129:0/1159645371 >> 172.21.15.129:0/1159645371 conn(0x55707d52a3c0 msgr2=0x55707d900400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.726+0000 7f801f80b740 1 -- 172.21.15.129:0/1159645371 shutdown_connections 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.727+0000 7f801f80b740 1 -- 172.21.15.129:0/1159645371 wait complete. 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.727+0000 7f801f80b740 1 -- start start 2024-09-17T04:04:10.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.728+0000 7f801f80b740 1 Processor -- start 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d99e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d91cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55707d91bb30 con 0x55707d50d3b0 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55707d922500 con 0x55707d536270 2024-09-17T04:04:10.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55707d9a2220 con 0x55707d515590 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f801f80b740 10 client.0 _ll_register_callbacks cb 0x7f7ffc040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f8015c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f8015c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38818/0 (socket says 172.21.15.129:38818) 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.729+0000 7f8015c00640 1 -- 172.21.15.129:0/780749159 learned_addr learned my addr 172.21.15.129:0/780749159 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015200640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d99e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8016600640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d91cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 -- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 msgr2=0x55707d99e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d99e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 -- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 msgr2=0x55707d91cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d91cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 -- 172.21.15.129:0/780749159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55707d9a4520 con 0x55707d50d3b0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8016600640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d91cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f8015c00640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f800405f450 tx=0x7f800401a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2e855ee1b256c18 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f7ff6a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f800406a2b0 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f80040626a0 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8008022070 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7f80040628a0 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f801f80b740 10 client.10701 fetch_fsmap learned FSMap version 193 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.730+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=193,monmap=2+}) -- 0x55707d9a1f60 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8008022070 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 193) ==== 2312+0+0 (secure 0 0 0) 0x7f8004010b20 con 0x55707d50d3b0 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 10 client.10701 fetch_fsmap finished waiting for FSMap version 193 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 10 client.10701 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 20 client.10701 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d9434e0 0x55707d9438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:10.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 4 client.10701 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "status"}) -- 0x55707d84dfd0 con 0x55707d9434e0 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.731+0000 7f8015200640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d9434e0 0x55707d9438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f7ff6a00640 10 client.10701 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f8015200640 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d9434e0 0x55707d9438f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a7ec29de63c6eaf server_cookie=b4149323954078ab in_seq=0 out_seq=0 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f7ff6a00640 1 -- 172.21.15.129:0/780749159 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55707d84dfd0 con 0x55707d9434e0 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f7ff6a00640 10 client.10701 handle_command_reply: tid=0 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 client.10701 shutdown 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 msgr2=0x55707d9052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f800405f450 tx=0x7f800401a040 comp rx=0 tx=0).stop 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 shutdown_connections 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55707d9434e0 0x55707d9438f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55707d536270 0x55707d91cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55707d515590 0x55707d99e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 --2- 172.21.15.129:0/780749159 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55707d50d3b0 0x55707d9052b0 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f7ff6a00640 0 client.10701 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.732+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 >> 172.21.15.129:0/780749159 conn(0x55707d52a3c0 msgr2=0x55707d905a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 shutdown_connections 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f801f80b740 1 -- 172.21.15.129:0/780749159 wait complete. 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f801f80b740 20 client.10701 trim_cache size 0 max 16384 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:04:10.737 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 10440, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 30.424679274999999, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 193, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 206, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 206, 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 35.455598201000001 2024-09-17T04:04:10.738 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 msgr2=0x7f7ff007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 0x7f7ff007b0d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc033000 tx=0x7f7ffc004730 comp rx=0 tx=0).stop 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 msgr2=0x7f80101f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f800400e790 tx=0x7f800400ec50 comp rx=0 tx=0).stop 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 shutdown_connections 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7ff0078bc0 0x7f7ff007b0d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f801015cd30 0x7f80101f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f801006fc10 0x7f8010143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 --2- 172.21.15.129:0/1126633152 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f801006f800 0x7f80101f6b50 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:10.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 >> 172.21.15.129:0/1126633152 conn(0x7f801006cfc0 msgr2=0x7f801015ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:10.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.733+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 shutdown_connections 2024-09-17T04:04:10.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10.734+0000 7f8018600640 1 -- 172.21.15.129:0/1126633152 wait complete. 2024-09-17T04:04:10.755 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:04:10.755 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> n = 5 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/delete_me/file" 2024-09-17T04:04:10.791 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:04:10.792 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T04:04:10.793 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:04:10.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:04:10.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:04:10.965 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 delete_me) 2024-09-17T04:04:11.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.312+0000 7f9597a00640 1 -- 172.21.15.129:0/1480989240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9578005e50 msgr2=0x7f9578006320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.312+0000 7f9597a00640 1 --2- 172.21.15.129:0/1480989240 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9578005e50 0x7f9578006320 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f9588005fd0 tx=0x7f9578006940 comp rx=0 tx=0).stop 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.312+0000 7f9595a00640 1 -- 172.21.15.129:0/1480989240 reap_dead start 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.312+0000 7f9597a00640 1 -- 172.21.15.129:0/1480989240 shutdown_connections 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.312+0000 7f9597a00640 1 -- 172.21.15.129:0/1480989240 >> 172.21.15.129:0/1480989240 conn(0x7f959006cfc0 msgr2=0x7f959006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.313+0000 7f9597a00640 1 -- 172.21.15.129:0/1480989240 shutdown_connections 2024-09-17T04:04:11.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.313+0000 7f9597a00640 1 -- 172.21.15.129:0/1480989240 wait complete. 2024-09-17T04:04:11.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.315+0000 7f9597a00640 1 Processor -- start 2024-09-17T04:04:11.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.315+0000 7f9597a00640 1 -- start start 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.316+0000 7f9597a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9578005e50 0x7f95900b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.316+0000 7f9597a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9597a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95900a9480 0x7f95900a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9597a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f959015b5d0 con 0x7f95900aa9d0 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9597a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f95900714a0 con 0x7f95900a9480 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9597a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f959015b750 con 0x7f9578005e50 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38836/0 (socket says 172.21.15.129:38836) 2024-09-17T04:04:11.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 -- 172.21.15.129:0/1124202454 learned_addr learned my addr 172.21.15.129:0/1124202454 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9595000640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9578005e50 0x7f95900b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f9595a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95900a9480 0x7f95900a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 -- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9578005e50 msgr2=0x7f95900b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9578005e50 0x7f95900b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 -- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95900a9480 msgr2=0x7f95900a9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95900a9480 0x7f95900a9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 -- 172.21.15.129:0/1124202454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95900a9dd0 con 0x7f95900aa9d0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.317+0000 7f958fe00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f957c003f20 tx=0x7f957c0016f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.318+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c011640 con 0x7f95900aa9d0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.318+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f957c016020 con 0x7f95900aa9d0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.318+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95901f6630 con 0x7f95900aa9d0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.318+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c011a80 con 0x7f95900aa9d0 2024-09-17T04:04:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.318+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95900aff80 con 0x7f95900aa9d0 2024-09-17T04:04:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.319+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95900afcd0 con 0x7f95900aa9d0 2024-09-17T04:04:11.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.320+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f957c011c20 con 0x7f95900aa9d0 2024-09-17T04:04:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.321+0000 7f958cc00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 0x7f957007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:11.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.322+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f957c09b930 con 0x7f95900aa9d0 2024-09-17T04:04:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.322+0000 7f9595000640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 0x7f957007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:11.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.323+0000 7f9595000640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 0x7f957007b0a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9584001010 tx=0x7f9584006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:11.326 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.326+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f957c020020 con 0x7f95900aa9d0 2024-09-17T04:04:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.546+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95900aa280 con 0x7f95900aa9d0 2024-09-17T04:04:11.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.547+0000 7f958cc00640 1 -- 172.21.15.129:0/1124202454 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3614 (secure 0 0 0) 0x7f95901f6630 con 0x7f95900aa9d0 2024-09-17T04:04:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:11.547 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":193,"btime":"2024-09-17T04:04:09:409438+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:09.409432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[9895,10440]},"id":15}]} 2024-09-17T04:04:11.547 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 193 2024-09-17T04:04:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 msgr2=0x7f957007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:11.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 0x7f957007b0a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f9584001010 tx=0x7f9584006040 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 msgr2=0x7f95901f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 secure :-1 s=READY pgs=2242 cs=0 l=1 rev1=1 crypto rx=0x7f957c003f20 tx=0x7f957c0016f0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 shutdown_connections 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9570078b90 0x7f957007b0a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95900a9480 0x7f95900a9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f95900aa9d0 0x7f95901f60f0 unknown :-1 s=CLOSED pgs=2242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 --2- 172.21.15.129:0/1124202454 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9578005e50 0x7f95900b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 >> 172.21.15.129:0/1124202454 conn(0x7f959006cfc0 msgr2=0x7f95900d0580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 shutdown_connections 2024-09-17T04:04:11.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:11.549+0000 7f9597a00640 1 -- 172.21.15.129:0/1124202454 wait complete. 2024-09-17T04:04:11.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:04:11.816 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:04:11.816 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:04:11.816 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:04:13.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 -- 172.21.15.129:0/2882187575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 msgr2=0x7f35100d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 --2- 172.21.15.129:0/2882187575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100d0620 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f35100d1240 tx=0x7f350802aa30 comp rx=0 tx=0).stop 2024-09-17T04:04:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 -- 172.21.15.129:0/2882187575 shutdown_connections 2024-09-17T04:04:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 --2- 172.21.15.129:0/2882187575 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100d0620 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 --2- 172.21.15.129:0/2882187575 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351006f800 0x7f35100cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 --2- 172.21.15.129:0/2882187575 >> v2:172.21.15.181:3300/0 conn(0x7f351015cd30 0x7f35100cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.048+0000 7f3515c00640 1 -- 172.21.15.129:0/2882187575 >> 172.21.15.129:0/2882187575 conn(0x7f351006cfc0 msgr2=0x7f351006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:14.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.049+0000 7f3515c00640 1 -- 172.21.15.129:0/2882187575 shutdown_connections 2024-09-17T04:04:14.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.049+0000 7f3515c00640 1 -- 172.21.15.129:0/2882187575 wait complete. 2024-09-17T04:04:14.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.051+0000 7f3515c00640 1 Processor -- start 2024-09-17T04:04:14.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.051+0000 7f3515c00640 1 -- start start 2024-09-17T04:04:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.052+0000 7f3515c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 0x7f351014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.052+0000 7f3515c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.052+0000 7f3515c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 0x7f35100ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3515c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3510073710 con 0x7f351006fc10 2024-09-17T04:04:14.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3515c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f35100d0f10 con 0x7f351006f800 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3515c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3510073890 con 0x7f351015cd30 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37232/0 (socket says 172.21.15.129:37232) 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 -- 172.21.15.129:0/407063941 learned_addr learned my addr 172.21.15.129:0/407063941 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f350f400640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 0x7f35100ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f350ea00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 0x7f351014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 -- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 msgr2=0x7f35100ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 0x7f35100ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 -- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 msgr2=0x7f351014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 0x7f351014ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f3507e00640 1 -- 172.21.15.129:0/407063941 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35101f65a0 con 0x7f351006fc10 2024-09-17T04:04:14.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f350f400640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 0x7f35100ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.053+0000 7f350ea00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 0x7f351014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3507e00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f34f800d770 tx=0x7f34f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8017070 con 0x7f351006fc10 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f34f8009d20 con 0x7f351006fc10 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35100ac4e0 con 0x7f351006fc10 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8004280 con 0x7f351006fc10 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.054+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f351014d350 con 0x7f351006fc10 2024-09-17T04:04:14.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.055+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35100ac750 con 0x7f351006fc10 2024-09-17T04:04:14.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.056+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f34f8005020 con 0x7f351006fc10 2024-09-17T04:04:14.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.057+0000 7f3506a00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 0x7f34ec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:14.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.057+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f34f809b1f0 con 0x7f351006fc10 2024-09-17T04:04:14.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.058+0000 7f350ea00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 0x7f34ec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:14.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.059+0000 7f350ea00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 0x7f34ec07b0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3500009a30 tx=0x7f3500005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:14.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.062+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f34f8020020 con 0x7f351006fc10 2024-09-17T04:04:14.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.279+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35100d0620 con 0x7f351006fc10 2024-09-17T04:04:14.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.280+0000 7f3506a00640 1 -- 172.21.15.129:0/407063941 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3614 (secure 0 0 0) 0x7f34f8063650 con 0x7f351006fc10 2024-09-17T04:04:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:14.280 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":193,"btime":"2024-09-17T04:04:09:409438+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:09.409432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[9895,10440]},"id":15}]} 2024-09-17T04:04:14.281 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 193 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.282+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 msgr2=0x7f34ec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.282+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 0x7f34ec07b0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3500009a30 tx=0x7f3500005f90 comp rx=0 tx=0).stop 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.282+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 msgr2=0x7f35100ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.282+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f34f800d770 tx=0x7f34f800dc30 comp rx=0 tx=0).stop 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 shutdown_connections 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f34ec078b90 0x7f34ec07b0a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f351015cd30 0x7f35100ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f351006fc10 0x7f35100ad080 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 --2- 172.21.15.129:0/407063941 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f351006f800 0x7f351014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:14.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 >> 172.21.15.129:0/407063941 conn(0x7f351006cfc0 msgr2=0x7f351015ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:14.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 shutdown_connections 2024-09-17T04:04:14.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:14.283+0000 7f3515c00640 1 -- 172.21.15.129:0/407063941 wait complete. 2024-09-17T04:04:14.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:04:14.542 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:04:14.542 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:04:14.542 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:04:16.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 -- 172.21.15.129:0/1768013794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d38072fa0 msgr2=0x7f0d380733b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/1768013794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d38072fa0 0x7f0d380733b0 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f0d30005fd0 tx=0x7f0d3002a940 comp rx=0 tx=0).stop 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 -- 172.21.15.129:0/1768013794 shutdown_connections 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/1768013794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d38072fa0 0x7f0d380733b0 unknown :-1 s=CLOSED pgs=2246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 -- 172.21.15.129:0/1768013794 >> 172.21.15.129:0/1768013794 conn(0x7f0d380b2510 msgr2=0x7f0d380d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 -- 172.21.15.129:0/1768013794 shutdown_connections 2024-09-17T04:04:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.776+0000 7f0d3ec00640 1 -- 172.21.15.129:0/1768013794 wait complete. 2024-09-17T04:04:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.778+0000 7f0d3ec00640 1 Processor -- start 2024-09-17T04:04:16.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.778+0000 7f0d3ec00640 1 -- start start 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d38072fa0 0x7f0d380cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d380d0f20 0x7f0d381f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d380b4780 con 0x7f0d3815c830 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d380b4600 con 0x7f0d38072fa0 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0d380b4900 con 0x7f0d380d0f20 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:16.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37248/0 (socket says 172.21.15.129:37248) 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 -- 172.21.15.129:0/154944950 learned_addr learned my addr 172.21.15.129:0/154944950 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3d800640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d38072fa0 0x7f0d380cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3ce00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d380d0f20 0x7f0d381f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 -- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d380d0f20 msgr2=0x7f0d381f0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d380d0f20 0x7f0d381f0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 -- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d38072fa0 msgr2=0x7f0d380cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d38072fa0 0x7f0d380cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.780+0000 7f0d3e200640 1 -- 172.21.15.129:0/154944950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d380ad720 con 0x7f0d3815c830 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d3e200640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009cd0 tx=0x7f0d30031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3003e070 con 0x7f0d3815c830 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d30004560 con 0x7f0d3815c830 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d381f4fd0 con 0x7f0d3815c830 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30043460 con 0x7f0d3815c830 2024-09-17T04:04:16.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.781+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d381f5590 con 0x7f0d3815c830 2024-09-17T04:04:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.782+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d380733b0 con 0x7f0d3815c830 2024-09-17T04:04:16.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.783+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d3003a440 con 0x7f0d3815c830 2024-09-17T04:04:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.784+0000 7f0d35600640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 0x7f0d1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.784+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5795+0+0 (secure 0 0 0) 0x7f0d300bc990 con 0x7f0d3815c830 2024-09-17T04:04:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.785+0000 7f0d3d800640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 0x7f0d1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:16.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.785+0000 7f0d3d800640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 0x7f0d1807b0a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c001610 tx=0x7f0d2c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:16.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:16.789+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d30084cd0 con 0x7f0d3815c830 2024-09-17T04:04:17.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.005+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d3815cff0 con 0x7f0d3815c830 2024-09-17T04:04:17.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.005+0000 7f0d35600640 1 -- 172.21.15.129:0/154944950 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3614 (secure 0 0 0) 0x7f0d300845d0 con 0x7f0d3815c830 2024-09-17T04:04:17.006 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:17.006 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":193,"btime":"2024-09-17T04:04:09:409438+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:09.409432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[9895,10440]},"id":15}]} 2024-09-17T04:04:17.006 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 193 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.007+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 msgr2=0x7f0d1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.007+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 0x7f0d1807b0a0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c001610 tx=0x7f0d2c009040 comp rx=0 tx=0).stop 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.007+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 msgr2=0x7f0d381f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.007+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009cd0 tx=0x7f0d30031ea0 comp rx=0 tx=0).stop 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 shutdown_connections 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d18078b90 0x7f0d1807b0a0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d3815c830 0x7f0d381f67a0 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:17.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d380d0f20 0x7f0d381f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:17.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 --2- 172.21.15.129:0/154944950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d38072fa0 0x7f0d380cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:17.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 >> 172.21.15.129:0/154944950 conn(0x7f0d380b2510 msgr2=0x7f0d38070140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:17.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 shutdown_connections 2024-09-17T04:04:17.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:17.008+0000 7f0d3ec00640 1 -- 172.21.15.129:0/154944950 wait complete. 2024-09-17T04:04:17.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:04:17.269 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:04:17.270 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:04:17.270 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:04:19.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:04:19.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:04:19.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 -- 172.21.15.129:0/114786593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 msgr2=0x7f69fc15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 --2- 172.21.15.129:0/114786593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc15bd70 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f69ec005fd0 tx=0x7f69ec02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 -- 172.21.15.129:0/114786593 shutdown_connections 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 --2- 172.21.15.129:0/114786593 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 0x7f69fc0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 --2- 172.21.15.129:0/114786593 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc15bd70 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.498+0000 7f6a00e00640 1 -- 172.21.15.129:0/114786593 >> 172.21.15.129:0/114786593 conn(0x7f69fc06cfc0 msgr2=0x7f69fc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.499+0000 7f6a00e00640 1 -- 172.21.15.129:0/114786593 shutdown_connections 2024-09-17T04:04:19.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.499+0000 7f6a00e00640 1 -- 172.21.15.129:0/114786593 wait complete. 2024-09-17T04:04:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.501+0000 7f6a00e00640 1 Processor -- start 2024-09-17T04:04:19.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.501+0000 7f6a00e00640 1 -- start start 2024-09-17T04:04:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc155a70 0x7f69fc0ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 0x7f69fc0cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:19.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc0d0620 con 0x7f69fc15f360 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f69fc0b4ed0 con 0x7f69fc155a70 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.502+0000 7f6a00e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f69fc0d07a0 con 0x7f69fc155e80 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37254/0 (socket says 172.21.15.129:37254) 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 -- 172.21.15.129:0/1690042244 learned_addr learned my addr 172.21.15.129:0/1690042244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69f9e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc155a70 0x7f69fc0ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69f9400640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 0x7f69fc0cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:19.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 -- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 msgr2=0x7f69fc0cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 0x7f69fc0cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 -- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc155a70 msgr2=0x7f69fc0ae300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc155a70 0x7f69fc0ae300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 -- 172.21.15.129:0/1690042244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69fc0aece0 con 0x7f69fc15f360 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.503+0000 7f69fa800640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f69f400e960 tx=0x7f69f400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.504+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4004d20 con 0x7f69fc15f360 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.504+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f69f400cb20 con 0x7f69fc15f360 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.504+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69fc0aefc0 con 0x7f69fc15f360 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.504+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69f4005020 con 0x7f69fc15f360 2024-09-17T04:04:19.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.504+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69fc159df0 con 0x7f69fc15f360 2024-09-17T04:04:19.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.505+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69fc0aea50 con 0x7f69fc15f360 2024-09-17T04:04:19.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.506+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f69f40051c0 con 0x7f69fc15f360 2024-09-17T04:04:19.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.507+0000 7f69f2000640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 0x7f69dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:19.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.507+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5824+0+0 (secure 0 0 0) 0x7f69f409b740 con 0x7f69fc15f360 2024-09-17T04:04:19.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.508+0000 7f69f9e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 0x7f69dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:19.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.508+0000 7f69f9e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 0x7f69dc07b0a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f69ec033000 tx=0x7f69ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:19.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.512+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f69f4016030 con 0x7f69fc15f360 2024-09-17T04:04:19.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.736+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69fc0af4e0 con 0x7f69fc15f360 2024-09-17T04:04:19.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.737+0000 7f69f2000640 1 -- 172.21.15.129:0/1690042244 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+3614 (secure 0 0 0) 0x7f69fc0af4e0 con 0x7f69fc15f360 2024-09-17T04:04:19.738 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:19.738 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":193,"btime":"2024-09-17T04:04:09:409438+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:09.409432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[9895,10440]},"id":15}]} 2024-09-17T04:04:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 193 2024-09-17T04:04:19.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 msgr2=0x7f69dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 0x7f69dc07b0a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f69ec033000 tx=0x7f69ec004730 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 msgr2=0x7f69fc148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f69f400e960 tx=0x7f69f400ee20 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 shutdown_connections 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f69dc078b90 0x7f69dc07b0a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f69fc15f360 0x7f69fc148690 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f69fc155e80 0x7f69fc0cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 --2- 172.21.15.129:0/1690042244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69fc155a70 0x7f69fc0ae300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.739+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 >> 172.21.15.129:0/1690042244 conn(0x7f69fc06cfc0 msgr2=0x7f69fc068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.740+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 shutdown_connections 2024-09-17T04:04:19.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:19.740+0000 7f6a00e00640 1 -- 172.21.15.129:0/1690042244 wait complete. 2024-09-17T04:04:19.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:04:20.004 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:04:20.005 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:04:20.005 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:04:22.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 -- 172.21.15.129:0/855594411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 msgr2=0x7f42e0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 --2- 172.21.15.129:0/855594411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f42fc073100 tx=0x7f42e0005830 comp rx=0 tx=0).stop 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 -- 172.21.15.129:0/855594411 shutdown_connections 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 --2- 172.21.15.129:0/855594411 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42e0004c10 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 -- 172.21.15.129:0/855594411 >> 172.21.15.129:0/855594411 conn(0x7f42fc06cfc0 msgr2=0x7f42fc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 -- 172.21.15.129:0/855594411 shutdown_connections 2024-09-17T04:04:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.232+0000 7f4303c00640 1 -- 172.21.15.129:0/855594411 wait complete. 2024-09-17T04:04:22.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.235+0000 7f4303c00640 1 Processor -- start 2024-09-17T04:04:22.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.235+0000 7f4303c00640 1 -- start start 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42fc0b4380 0x7f42fc0b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc0cba70 0x7f42fc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc0cda60 con 0x7f42e0004800 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42fc0cd8e0 con 0x7f42fc0cba70 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4303c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f42fc0cdbe0 con 0x7f42fc0b4380 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4301200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4301200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37258/0 (socket says 172.21.15.129:37258) 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.236+0000 7f4301200640 1 -- 172.21.15.129:0/3847167102 learned_addr learned my addr 172.21.15.129:0/3847167102 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f42fbe00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42fc0b4380 0x7f42fc0b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:22.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc0cba70 0x7f42fc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 -- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42fc0b4380 msgr2=0x7f42fc0b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42fc0b4380 0x7f42fc0b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 -- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc0cba70 msgr2=0x7f42fc0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc0cba70 0x7f42fc0cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 -- 172.21.15.129:0/3847167102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42fc0ae580 con 0x7f42e0004800 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4301200640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f42ec011ab0 tx=0x7f42ec011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec01e070 con 0x7f42e0004800 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42ec00fc70 con 0x7f42e0004800 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec019670 con 0x7f42e0004800 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.237+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42fc0ae860 con 0x7f42e0004800 2024-09-17T04:04:22.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.238+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42fc0aee20 con 0x7f42e0004800 2024-09-17T04:04:22.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.239+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42fc0d0790 con 0x7f42e0004800 2024-09-17T04:04:22.239 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.240+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f42ec021070 con 0x7f42e0004800 2024-09-17T04:04:22.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.241+0000 7f42f8c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 0x7f42dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:22.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.241+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f42ec09b5c0 con 0x7f42e0004800 2024-09-17T04:04:22.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.242+0000 7f42fbe00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 0x7f42dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:22.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.242+0000 7f42fbe00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 0x7f42dc07b0d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42fc15d4e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:22.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.246+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f42ec00fe60 con 0x7f42e0004800 2024-09-17T04:04:22.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.464+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42fc0cc3c0 con 0x7f42e0004800 2024-09-17T04:04:22.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.464+0000 7f42f8c00640 1 -- 172.21.15.129:0/3847167102 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+3615 (secure 0 0 0) 0x7f42ec099e20 con 0x7f42e0004800 2024-09-17T04:04:22.464 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:22.464 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":194,"btime":"2024-09-17T04:04:20:614992+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:20.283023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440,9895]},"id":15}]} 2024-09-17T04:04:22.464 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 194 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 msgr2=0x7f42dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 0x7f42dc07b0d0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f42e8033000 tx=0x7f42fc15d4e0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 msgr2=0x7f42fc1f8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f42ec011ab0 tx=0x7f42ec011f70 comp rx=0 tx=0).stop 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 shutdown_connections 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42dc078bc0 0x7f42dc07b0d0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42fc0cba70 0x7f42fc0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f42fc0b4380 0x7f42fc0b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 --2- 172.21.15.129:0/3847167102 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f42e0004800 0x7f42fc1f8d60 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 >> 172.21.15.129:0/3847167102 conn(0x7f42fc06cfc0 msgr2=0x7f42fc15bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 shutdown_connections 2024-09-17T04:04:22.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:22.467+0000 7f4303c00640 1 -- 172.21.15.129:0/3847167102 wait complete. 2024-09-17T04:04:22.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:04:22.733 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"26","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/delete_me","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"21","version":"21","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-09-17T04:04:22.734 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "26" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/delete_me", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "21", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "21" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:04:22.734 INFO:tasks.cephfs.cephfs_test_case:[('/delete_me', 1)] =?= [('/delete_me', 1)] 2024-09-17T04:04:22.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:04:22.734 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me/*) 2024-09-17T04:04:22.785 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:04:22.785 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:04:22.820 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:04:22.820 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:04:22.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 -- 172.21.15.129:0/867145039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ce310 msgr2=0x7fe7380ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 --2- 172.21.15.129:0/867145039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ce310 0x7fe7380ce720 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe720005fd0 tx=0x7fe72002a540 comp rx=0 tx=0).stop 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 -- 172.21.15.129:0/867145039 shutdown_connections 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 --2- 172.21.15.129:0/867145039 >> v2:172.21.15.181:3300/0 conn(0x7fe7380ca890 0x7fe7380caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 --2- 172.21.15.129:0/867145039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ce310 0x7fe7380ce720 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 -- 172.21.15.129:0/867145039 >> 172.21.15.129:0/867145039 conn(0x7fe7380b2510 msgr2=0x7fe7380d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 -- 172.21.15.129:0/867145039 shutdown_connections 2024-09-17T04:04:23.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.157+0000 7fe737e00640 1 -- 172.21.15.129:0/867145039 wait complete. 2024-09-17T04:04:23.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.160+0000 7fe737e00640 1 Processor -- start 2024-09-17T04:04:23.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.160+0000 7fe737e00640 1 -- start start 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ca890 0x7fe7381f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 0x7fe7380c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe7380cb5f0 con 0x7fe7380cc390 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe7380cb470 con 0x7fe7380ce310 2024-09-17T04:04:23.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe737e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe7380cb770 con 0x7fe7380ca890 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.161+0000 7fe72e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37272/0 (socket says 172.21.15.129:37272) 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 -- 172.21.15.129:0/10619623 learned_addr learned my addr 172.21.15.129:0/10619623 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe736a00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ca890 0x7fe7381f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe737400640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 0x7fe7380c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 -- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ca890 msgr2=0x7fe7381f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ca890 0x7fe7381f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 -- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 msgr2=0x7fe7380c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 0x7fe7380c9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 -- 172.21.15.129:0/10619623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe738074df0 con 0x7fe7380cc390 2024-09-17T04:04:23.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe737400640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 0x7fe7380c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.162+0000 7fe72e000640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe720009230 tx=0x7fe720005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.163+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe72003e030 con 0x7fe7380cc390 2024-09-17T04:04:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.163+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe720031700 con 0x7fe7380cc390 2024-09-17T04:04:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.163+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720035040 con 0x7fe7380cc390 2024-09-17T04:04:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.163+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7381f55e0 con 0x7fe7380cc390 2024-09-17T04:04:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.163+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7381f5ba0 con 0x7fe7380cc390 2024-09-17T04:04:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.164+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7381f9940 con 0x7fe7380cc390 2024-09-17T04:04:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.165+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe720041030 con 0x7fe7380cc390 2024-09-17T04:04:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.166+0000 7fe72f400640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 0x7fe71407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.166+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fe7200bb480 con 0x7fe7380cc390 2024-09-17T04:04:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.167+0000 7fe736a00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 0x7fe71407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.167+0000 7fe736a00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 0x7fe71407b0a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe728000f60 tx=0x7fe728009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:23.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.171+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe738074df0 con 0x7fe7380cc390 2024-09-17T04:04:23.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.366+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe7380cbc90 con 0x7fe7380cc390 2024-09-17T04:04:23.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.367+0000 7fe72f400640 1 -- 172.21.15.129:0/10619623 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v208) ==== 80+0+124 (secure 0 0 0) 0x7fe720083210 con 0x7fe7380cc390 2024-09-17T04:04:23.366 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6835/3759513659 2024-09-18T04:04:04.378256+0000 2024-09-17T04:04:23.366 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6834/3759513659 2024-09-18T04:04:04.378256+0000 2024-09-17T04:04:23.366 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T04:04:23.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 msgr2=0x7fe71407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 0x7fe71407b0a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe728000f60 tx=0x7fe728009040 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 msgr2=0x7fe7380c9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7fe720009230 tx=0x7fe720005770 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 shutdown_connections 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078b90 0x7fe71407b0a0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7380ce310 0x7fe7380c9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7380cc390 0x7fe7380c9840 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 --2- 172.21.15.129:0/10619623 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7380ca890 0x7fe7381f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 >> 172.21.15.129:0/10619623 conn(0x7fe7380b2510 msgr2=0x7fe7380cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 shutdown_connections 2024-09-17T04:04:23.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:23.369+0000 7fe737e00640 1 -- 172.21.15.129:0/10619623 wait complete. 2024-09-17T04:04:23.392 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:04:23.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:04:23.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:04:23.492 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:04:23.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:04:23.493 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:04:23.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_created 2024-09-17T04:04:23.818 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":5}} 2024-09-17T04:04:23.819 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 5 } } 2024-09-17T04:04:23.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T04:04:24.063 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":0}} 2024-09-17T04:04:24.063 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 0 } } 2024-09-17T04:04:24.064 DEBUG:tasks.ceph_test_case:wait_until_equal: 0 != 5, waiting (timeout=60)... 2024-09-17T04:04:29.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T04:04:29.314 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":5}} 2024-09-17T04:04:29.314 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 5 } } 2024-09-17T04:04:29.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:04:29.557 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T04:04:29.557 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T04:04:39.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-09-17T04:04:39.804 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":0}} 2024-09-17T04:04:39.805 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 0 } } 2024-09-17T04:04:39.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 -- 172.21.15.129:0/1143284823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 msgr2=0x7f32e415cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 --2- 172.21.15.129:0/1143284823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e415cf80 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7f32e40711e0 tx=0x7f32dc02a820 comp rx=0 tx=0).stop 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 -- 172.21.15.129:0/1143284823 shutdown_connections 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 --2- 172.21.15.129:0/1143284823 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e415cf80 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 --2- 172.21.15.129:0/1143284823 >> v2:172.21.15.181:3300/0 conn(0x7f32e406fa30 0x7f32e40d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 -- 172.21.15.129:0/1143284823 >> 172.21.15.129:0/1143284823 conn(0x7f32e406cfc0 msgr2=0x7f32e406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 -- 172.21.15.129:0/1143284823 shutdown_connections 2024-09-17T04:04:40.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.029+0000 7f32eba00640 1 -- 172.21.15.129:0/1143284823 wait complete. 2024-09-17T04:04:40.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.032+0000 7f32eba00640 1 Processor -- start 2024-09-17T04:04:40.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.032+0000 7f32eba00640 1 -- start start 2024-09-17T04:04:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 0x7f32e41448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:40.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32e415a460 0x7f32e40b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32e415d640 con 0x7f32e40d0c00 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f32e415d4c0 con 0x7f32e406fa30 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32eba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f32e415d7c0 con 0x7f32e415a460 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57464/0 (socket says 172.21.15.129:57464) 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.033+0000 7f32e3e00640 1 -- 172.21.15.129:0/1997300954 learned_addr learned my addr 172.21.15.129:0/1997300954 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e9a00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32e415a460 0x7f32e40b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e9000640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 0x7f32e41448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 -- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32e415a460 msgr2=0x7f32e40b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32e415a460 0x7f32e40b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 -- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 msgr2=0x7f32e41448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 0x7f32e41448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:40.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 -- 172.21.15.129:0/1997300954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e40b4180 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e9000640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 0x7f32e41448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e3e00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f32d000d750 tx=0x7f32d000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0017070 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32d0004310 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e40b4460 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d000de60 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.034+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e40c5810 con 0x7f32e40d0c00 2024-09-17T04:04:40.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.035+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32e415dca0 con 0x7f32e40d0c00 2024-09-17T04:04:40.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.037+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f32d0009c30 con 0x7f32e40d0c00 2024-09-17T04:04:40.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.038+0000 7f32e0c00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 0x7f32c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:40.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.038+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f32d009b210 con 0x7f32e40d0c00 2024-09-17T04:04:40.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.038+0000 7f32e9000640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 0x7f32c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:40.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.039+0000 7f32e9000640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 0x7f32c407b0a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f32d8000f60 tx=0x7f32d8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:40.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.042+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f32d0020020 con 0x7f32e40d0c00 2024-09-17T04:04:40.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:40.261+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) -- 0x7f32e40b3f10 con 0x7f32e40d0c00 2024-09-17T04:04:41.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.011+0000 7f32e0c00640 1 -- 172.21.15.129:0/1997300954 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v196) ==== 105+0+0 (secure 0 0 0) 0x7f32d00635f0 con 0x7f32e40d0c00 2024-09-17T04:04:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 msgr2=0x7f32c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 0x7f32c407b0a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f32d8000f60 tx=0x7f32d8009040 comp rx=0 tx=0).stop 2024-09-17T04:04:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 msgr2=0x7f32e40b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:41.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 secure :-1 s=READY pgs=2257 cs=0 l=1 rev1=1 crypto rx=0x7f32d000d750 tx=0x7f32d000dc10 comp rx=0 tx=0).stop 2024-09-17T04:04:41.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 shutdown_connections 2024-09-17T04:04:41.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32c4078b90 0x7f32c407b0a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:41.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32e415a460 0x7f32e40b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:41.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32e40d0c00 0x7f32e40b3470 unknown :-1 s=CLOSED pgs=2257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:41.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 --2- 172.21.15.129:0/1997300954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32e406fa30 0x7f32e41448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:41.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 >> 172.21.15.129:0/1997300954 conn(0x7f32e406cfc0 msgr2=0x7f32e4070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:41.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 shutdown_connections 2024-09-17T04:04:41.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:41.014+0000 7f32eba00640 1 -- 172.21.15.129:0/1997300954 wait complete. 2024-09-17T04:04:49.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:04:49.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:04:51.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json perf dump purge_queue pq_executed 2024-09-17T04:04:51.292 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":0}} 2024-09-17T04:04:51.293 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 0 } } 2024-09-17T04:04:51.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 -- 172.21.15.129:0/4244628151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6148004800 msgr2=0x7f6148004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 --2- 172.21.15.129:0/4244628151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6148004800 0x7f6148004c10 secure :-1 s=READY pgs=2259 cs=0 l=1 rev1=1 crypto rx=0x7f6164073100 tx=0x7f6148005830 comp rx=0 tx=0).stop 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f6168e00640 1 -- 172.21.15.129:0/4244628151 reap_dead start 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 -- 172.21.15.129:0/4244628151 shutdown_connections 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 -- 172.21.15.129:0/4244628151 >> 172.21.15.129:0/4244628151 conn(0x7f616406cfc0 msgr2=0x7f616406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 -- 172.21.15.129:0/4244628151 shutdown_connections 2024-09-17T04:04:51.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.515+0000 7f616ae00640 1 -- 172.21.15.129:0/4244628151 wait complete. 2024-09-17T04:04:51.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.518+0000 7f616ae00640 1 Processor -- start 2024-09-17T04:04:51.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.518+0000 7f616ae00640 1 -- start start 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6148004800 0x7f6164145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61641480c0 0x7f61641484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f61640d0f50 con 0x7f6164155ec0 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f61640d0dd0 con 0x7f6148004800 2024-09-17T04:04:51.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f616ae00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6164073710 con 0x7f61641480c0 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.519+0000 7f6168e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38804/0 (socket says 172.21.15.129:38804) 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 -- 172.21.15.129:0/3461833535 learned_addr learned my addr 172.21.15.129:0/3461833535 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f615b400640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61641480c0 0x7f61641484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6163e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6148004800 0x7f6164145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 -- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61641480c0 msgr2=0x7f61641484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61641480c0 0x7f61641484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 -- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6148004800 msgr2=0x7f6164145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6148004800 0x7f6164145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 -- 172.21.15.129:0/3461833535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6164156810 con 0x7f6164155ec0 2024-09-17T04:04:51.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6168e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f615c005fd0 tx=0x7f615c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:51.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.520+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c004b20 con 0x7f6164155ec0 2024-09-17T04:04:51.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.521+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f615c004cc0 con 0x7f6164155ec0 2024-09-17T04:04:51.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.521+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61640aab80 con 0x7f6164155ec0 2024-09-17T04:04:51.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.521+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615c003bd0 con 0x7f6164155ec0 2024-09-17T04:04:51.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.521+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6164146280 con 0x7f6164155ec0 2024-09-17T04:04:51.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.522+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61640aad50 con 0x7f6164155ec0 2024-09-17T04:04:51.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.523+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f615c004030 con 0x7f6164155ec0 2024-09-17T04:04:51.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.524+0000 7f6160c00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 0x7f614007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.524+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f615c09b1e0 con 0x7f6164155ec0 2024-09-17T04:04:51.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.524+0000 7f6163e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 0x7f614007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.525+0000 7f6163e00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 0x7f614007b0d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6154001050 tx=0x7f61540067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:51.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.528+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6164156810 con 0x7f6164155ec0 2024-09-17T04:04:51.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.743+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6164073c70 con 0x7f6164155ec0 2024-09-17T04:04:51.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.744+0000 7f6160c00640 1 -- 172.21.15.129:0/3461833535 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7f615c062f70 con 0x7f6164155ec0 2024-09-17T04:04:51.744 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:51.744 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:51.744 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 msgr2=0x7f614007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 0x7f614007b0d0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6154001050 tx=0x7f61540067a0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 msgr2=0x7f61641562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f615c005fd0 tx=0x7f615c00f3c0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 shutdown_connections 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6140078bc0 0x7f614007b0d0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6164155ec0 0x7f61641562d0 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61641480c0 0x7f61641484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 --2- 172.21.15.129:0/3461833535 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6148004800 0x7f6164145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 >> 172.21.15.129:0/3461833535 conn(0x7f616406cfc0 msgr2=0x7f61640b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:51.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.746+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 shutdown_connections 2024-09-17T04:04:51.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.747+0000 7f616ae00640 1 -- 172.21.15.129:0/3461833535 wait complete. 2024-09-17T04:04:51.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c injectargs '--mds_max_purge_files 100' 2024-09-17T04:04:51.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 -- 172.21.15.129:0/1254084162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7815cd30 msgr2=0x7fdc780cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 --2- 172.21.15.129:0/1254084162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7815cd30 0x7fdc780cfd10 secure :-1 s=READY pgs=2262 cs=0 l=1 rev1=1 crypto rx=0x7fdc7806de00 tx=0x7fdc6802a4a0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 -- 172.21.15.129:0/1254084162 shutdown_connections 2024-09-17T04:04:51.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 --2- 172.21.15.129:0/1254084162 >> v2:172.21.15.181:3300/0 conn(0x7fdc7806f800 0x7fdc780cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 --2- 172.21.15.129:0/1254084162 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7815cd30 0x7fdc780cfd10 unknown :-1 s=CLOSED pgs=2262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 -- 172.21.15.129:0/1254084162 >> 172.21.15.129:0/1254084162 conn(0x7fdc7806cfc0 msgr2=0x7fdc7806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:51.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 -- 172.21.15.129:0/1254084162 shutdown_connections 2024-09-17T04:04:51.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.990+0000 7fdc7f200640 1 -- 172.21.15.129:0/1254084162 wait complete. 2024-09-17T04:04:51.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.992+0000 7fdc7f200640 1 Processor -- start 2024-09-17T04:04:51.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.992+0000 7fdc7f200640 1 -- start start 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 0x7fdc781f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdc7815cd30 0x7fdc781f8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78073890 con 0x7fdc7806fc10 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78073710 con 0x7fdc7806f800 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc7f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdc78073a10 con 0x7fdc7815cd30 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc77400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38808/0 (socket says 172.21.15.129:38808) 2024-09-17T04:04:51.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.994+0000 7fdc77400640 1 -- 172.21.15.129:0/3145696453 learned_addr learned my addr 172.21.15.129:0/3145696453 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 0x7fdc781f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdc7815cd30 0x7fdc781f8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 -- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdc7815cd30 msgr2=0x7fdc781f8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdc7815cd30 0x7fdc781f8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 -- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 msgr2=0x7fdc781f8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 0x7fdc781f8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 -- 172.21.15.129:0/3145696453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc780af390 con 0x7fdc7806fc10 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 0x7fdc781f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:51.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc77400640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400d720 tx=0x7fdc6400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:51.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc6fe00640 1 -- 172.21.15.129:0/3145696453 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc64017070 con 0x7fdc7806fc10 2024-09-17T04:04:51.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc6fe00640 1 -- 172.21.15.129:0/3145696453 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc64009cc0 con 0x7fdc7806fc10 2024-09-17T04:04:51.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc780af0b0 con 0x7fdc7806fc10 2024-09-17T04:04:51.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.995+0000 7fdc6fe00640 1 -- 172.21.15.129:0/3145696453 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc640106c0 con 0x7fdc7806fc10 2024-09-17T04:04:51.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.996+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc781548b0 con 0x7fdc7806fc10 2024-09-17T04:04:51.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.998+0000 7fdc6fe00640 1 -- 172.21.15.129:0/3145696453 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdc6401a070 con 0x7fdc7806fc10 2024-09-17T04:04:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.999+0000 7fdc6fe00640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 0x7fdc5807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.999+0000 7fdc6fe00640 1 -- 172.21.15.129:0/3145696453 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fdc6409ae20 con 0x7fdc7806fc10 2024-09-17T04:04:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:51.999+0000 7fdc8625a740 1 Processor -- start 2024-09-17T04:04:51.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.000+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 0x7fdc5807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.000+0000 7fdc8625a740 1 -- start start 2024-09-17T04:04:52.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.000+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 0x7fdc5807b0a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdc68034000 tx=0x7fdc68004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea1595ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1593faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ea15888700 con 0x55ea1554d680 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ea15877b30 con 0x55ea1556e360 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ea1587b940 con 0x55ea155454a0 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38822/0 (socket says 172.21.15.129:38822) 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc7d200640 1 -- 172.21.15.129:0/2869733550 learned_addr learned my addr 172.21.15.129:0/2869733550 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc77e00640 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea1595ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.001+0000 7fdc77400640 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1593faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 -- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 msgr2=0x55ea1593faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1593faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 -- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 msgr2=0x55ea1595ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea1595ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 -- 172.21.15.129:0/2869733550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea155d9940 con 0x55ea1554d680 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc7d200640 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdc70001be0 tx=0x7fdc700020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbca44b769ad183 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc6cc00640 1 -- 172.21.15.129:0/2869733550 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70019070 con 0x55ea1554d680 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc6cc00640 1 -- 172.21.15.129:0/2869733550 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc70003b80 con 0x55ea1554d680 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.002+0000 7fdc6cc00640 1 -- 172.21.15.129:0/2869733550 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7000e800 con 0x55ea1554d680 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 -- 172.21.15.129:0/2869733550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 msgr2=0x55ea15940030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fdc70001be0 tx=0x7fdc700020a0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 -- 172.21.15.129:0/2869733550 shutdown_connections 2024-09-17T04:04:52.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1554d680 0x55ea15940030 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1593faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 --2- 172.21.15.129:0/2869733550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea1595ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 -- 172.21.15.129:0/2869733550 >> 172.21.15.129:0/2869733550 conn(0x55ea155624b0 msgr2=0x55ea15915d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 -- 172.21.15.129:0/2869733550 shutdown_connections 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.003+0000 7fdc8625a740 1 -- 172.21.15.129:0/2869733550 wait complete. 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.004+0000 7fdc8625a740 1 -- start start 2024-09-17T04:04:52.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.005+0000 7fdc8625a740 1 Processor -- start 2024-09-17T04:04:52.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea1554d680 0x55ea159698c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea15953450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ea155e8710 con 0x55ea155454a0 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ea1581e370 con 0x55ea1556e360 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ea15955cc0 con 0x55ea1554d680 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc8625a740 10 client.0 _ll_register_callbacks cb 0x55ea158235c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc77e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38828/0 (socket says 172.21.15.129:38828) 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.006+0000 7fdc77e00640 1 -- 172.21.15.129:0/799742763 learned_addr learned my addr 172.21.15.129:0/799742763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77400640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea1554d680 0x55ea159698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc7d200640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea15953450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 -- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea1554d680 msgr2=0x55ea159698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea1554d680 0x55ea159698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 -- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 msgr2=0x55ea15953450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea15953450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 -- 172.21.15.129:0/799742763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea156507a0 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc77e00640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802aa90 tx=0x7fdc68005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5eca04b4c96063 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc61600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.007+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6803cb70 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fdc6803cd10 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc70003b80 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=197) ==== 24+0+0 (secure 0 0 0) 0x7fdc68053470 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 10 client.10770 fetch_fsmap learned FSMap version 197 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=197,monmap=2+}) -- 0x55ea15955a00 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc70003b80 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 197) ==== 2312+0+0 (secure 0 0 0) 0x7fdc6805e080 con 0x55ea155454a0 2024-09-17T04:04:52.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 10 client.10770 fetch_fsmap finished waiting for FSMap version 197 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 10 client.10770 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 20 client.10770 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea1552d140 0x55ea159157b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 4 client.10770 mds_command: new command op to 9895 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 --> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea1588a650 con 0x55ea1552d140 2024-09-17T04:04:52.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.008+0000 7fdc77400640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea1552d140 0x55ea159157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.009+0000 7fdc61600640 10 client.10770 ms_handle_connect on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:52.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.009+0000 7fdc77400640 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea1552d140 0x55ea159157b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=933dc206aa25cbf4 server_cookie=7e2ef5d88162a3c2 in_seq=0 out_seq=0 2024-09-17T04:04:52.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc61600640 1 -- 172.21.15.129:0/799742763 <== mds.1 v2:172.21.15.129:6832/1714996749 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55ea1588a650 con 0x55ea1552d140 2024-09-17T04:04:52.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc61600640 10 client.10770 handle_command_reply: tid=0 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 client.10770 shutdown 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 msgr2=0x55ea1592e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 secure :-1 s=READY pgs=2265 cs=0 l=1 rev1=1 crypto rx=0x7fdc6802aa90 tx=0x7fdc68005f90 comp rx=0 tx=0).stop 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 shutdown_connections 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1556e360 0x55ea15953450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea1554d680 0x55ea159698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea1552d140 0x55ea159157b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 --2- 172.21.15.129:0/799742763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea155454a0 0x55ea1592e420 unknown :-1 s=CLOSED pgs=2265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 >> 172.21.15.129:0/799742763 conn(0x55ea155624b0 msgr2=0x55ea1593ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc61600640 0 client.10770 ms_handle_reset on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 shutdown_connections 2024-09-17T04:04:52.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 1 -- 172.21.15.129:0/799742763 wait complete. 2024-09-17T04:04:52.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.010+0000 7fdc8625a740 20 client.10770 trim_cache size 0 max 16384 2024-09-17T04:04:52.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.026+0000 7fdc8625a740 1 Processor -- start 2024-09-17T04:04:52.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.026+0000 7fdc8625a740 1 -- start start 2024-09-17T04:04:52.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1596d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15916be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdc781f64a0 con 0x55ea1556e360 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdc78201450 con 0x55ea1554d680 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ea15954ec0 con 0x55ea155454a0 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38840/0 (socket says 172.21.15.129:38840) 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 -- 172.21.15.129:0/3336562271 learned_addr learned my addr 172.21.15.129:0/3336562271 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc77400640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15916be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1596d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 -- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 msgr2=0x55ea1596d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1596d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 -- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 msgr2=0x55ea15916be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15916be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.028+0000 7fdc7d200640 1 -- 172.21.15.129:0/3336562271 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea15950d70 con 0x55ea1556e360 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1596d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc77400640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15916be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:52.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdc70013000 tx=0x7fdc70011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8a488acd197fba server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc63e00640 1 -- 172.21.15.129:0/3336562271 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70024070 con 0x55ea1556e360 2024-09-17T04:04:52.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc63e00640 1 -- 172.21.15.129:0/3336562271 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc7000ebf0 con 0x55ea1556e360 2024-09-17T04:04:52.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.029+0000 7fdc63e00640 1 -- 172.21.15.129:0/3336562271 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70019040 con 0x55ea1556e360 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 -- 172.21.15.129:0/3336562271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 msgr2=0x55ea15917120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fdc70013000 tx=0x7fdc70011000 comp rx=0 tx=0).stop 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 -- 172.21.15.129:0/3336562271 shutdown_connections 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15917120 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15916be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3336562271 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea1596d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 -- 172.21.15.129:0/3336562271 >> 172.21.15.129:0/3336562271 conn(0x55ea155624b0 msgr2=0x55ea1593fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 -- 172.21.15.129:0/3336562271 shutdown_connections 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.030+0000 7fdc8625a740 1 -- 172.21.15.129:0/3336562271 wait complete. 2024-09-17T04:04:52.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.031+0000 7fdc8625a740 1 -- start start 2024-09-17T04:04:52.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.031+0000 7fdc8625a740 1 Processor -- start 2024-09-17T04:04:52.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.032+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea159d4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.032+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15955e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc8625a740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc8625a740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55ea15955310 con 0x55ea1556e360 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55ea15953520 con 0x55ea1554d680 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc8625a740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55ea159def70 con 0x55ea155454a0 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc8625a740 10 client.0 _ll_register_callbacks cb 0x7fdc68003a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38856/0 (socket says 172.21.15.129:38856) 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 -- 172.21.15.129:0/3399112149 learned_addr learned my addr 172.21.15.129:0/3399112149 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea159d4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc77400640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15955e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 -- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 msgr2=0x55ea159d4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea159d4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 -- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 msgr2=0x55ea15955e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:04:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15955e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 -- 172.21.15.129:0/3399112149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea15956a20 con 0x55ea1556e360 2024-09-17T04:04:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc77400640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15955e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc7d200640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fdc70022040 tx=0x7fdc700017f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899a025f95bde6cb server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.033+0000 7fdc61600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7002d060 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fdc700050d0 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc7000ebf0 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=197) ==== 24+0+0 (secure 0 0 0) 0x7fdc70033050 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 10 client.10776 fetch_fsmap learned FSMap version 197 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=197,monmap=2+}) -- 0x55ea159d5320 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdc7000ebf0 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 197) ==== 2312+0+0 (secure 0 0 0) 0x7fdc70036020 con 0x55ea1556e360 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 10 client.10776 fetch_fsmap finished waiting for FSMap version 197 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 10 client.10776 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 20 client.10776 populate_metadata read hostname 'smithi129' 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea159554e0 0x55ea159daf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 4 client.10776 mds_command: new command op to 9895 tid=0 multi_id=0 [{"prefix": "injectargs", "injected_args": ["--mds_max_purge_files 100"]}] 2024-09-17T04:04:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.034+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 --> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] -- command(tid 0: {"prefix": "injectargs", "injected_args": ["--mds_max_purge_files 100"]}) -- 0x55ea15886100 con 0x55ea159554e0 2024-09-17T04:04:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.035+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea159554e0 0x55ea159daf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.035+0000 7fdc77e00640 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea159554e0 0x55ea159daf70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee892f8626fd158c server_cookie=63bd5bb13aa9c2b7 in_seq=0 out_seq=0 2024-09-17T04:04:52.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.035+0000 7fdc61600640 10 client.10776 ms_handle_connect on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.036+0000 7fdc61600640 1 -- 172.21.15.129:0/3399112149 <== mds.1 v2:172.21.15.129:6832/1714996749 1 ==== command_reply(tid 0: 0 mds_max_purge_files = '' ) ==== 33+0+3 (crc 0 0 0) 0x55ea15886100 con 0x55ea159554e0 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.036+0000 7fdc61600640 10 client.10776 handle_command_reply: tid=0 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 client.10776 shutdown 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 msgr2=0x55ea15956390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fdc70022040 tx=0x7fdc700017f0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 shutdown_connections 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:6832/1714996749,v1:172.21.15.129:6833/1714996749] conn(0x55ea159554e0 0x55ea159daf70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55ea1556e360 0x55ea15956390 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea1554d680 0x55ea15955e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 --2- 172.21.15.129:0/3399112149 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55ea155454a0 0x55ea159d4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 >> 172.21.15.129:0/3399112149 conn(0x55ea155624b0 msgr2=0x55ea1593fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc61600640 0 client.10776 ms_handle_reset on v2:172.21.15.129:6832/1714996749 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 shutdown_connections 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 1 -- 172.21.15.129:0/3399112149 wait complete. 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.037+0000 7fdc8625a740 20 client.10776 trim_cache size 0 max 16384 2024-09-17T04:04:52.041 INFO:teuthology.orchestra.run.smithi129.stdout:{} 2024-09-17T04:04:52.042 INFO:teuthology.orchestra.run.smithi129.stderr:mds_max_purge_files = '' 2024-09-17T04:04:52.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 msgr2=0x7fdc5807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 0x7fdc5807b0a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fdc68034000 tx=0x7fdc68004730 comp rx=0 tx=0).stop 2024-09-17T04:04:52.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 msgr2=0x7fdc780ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fdc6400d720 tx=0x7fdc6400dbe0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 shutdown_connections 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdc58078b90 0x7fdc5807b0a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdc7815cd30 0x7fdc781f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdc7806fc10 0x7fdc780ae540 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 --2- 172.21.15.129:0/3145696453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdc7806f800 0x7fdc781f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.038+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 >> 172.21.15.129:0/3145696453 conn(0x7fdc7806cfc0 msgr2=0x7fdc78070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.039+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 shutdown_connections 2024-09-17T04:04:52.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.039+0000 7fdc7f200640 1 -- 172.21.15.129:0/3145696453 wait complete. 2024-09-17T04:04:52.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 -- 172.21.15.129:0/1580380474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2660155e80 msgr2=0x7f26600b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 --2- 172.21.15.129:0/1580380474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2660155e80 0x7f26600b4690 secure :-1 s=READY pgs=2268 cs=0 l=1 rev1=1 crypto rx=0x7f2658005fd0 tx=0x7f265802aa00 comp rx=0 tx=0).stop 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 -- 172.21.15.129:0/1580380474 shutdown_connections 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 --2- 172.21.15.129:0/1580380474 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2660155e80 0x7f26600b4690 unknown :-1 s=CLOSED pgs=2268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 --2- 172.21.15.129:0/1580380474 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f266015f360 0x7f266015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 -- 172.21.15.129:0/1580380474 >> 172.21.15.129:0/1580380474 conn(0x7f266006cfc0 msgr2=0x7f266006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 -- 172.21.15.129:0/1580380474 shutdown_connections 2024-09-17T04:04:52.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.286+0000 7f2664c00640 1 -- 172.21.15.129:0/1580380474 wait complete. 2024-09-17T04:04:52.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.289+0000 7f2664c00640 1 Processor -- start 2024-09-17T04:04:52.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.289+0000 7f2664c00640 1 -- start start 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2660155a70 0x7f26601512a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2660155e80 0x7f26601e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f26600d0620 con 0x7f266015f360 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f26600b4ef0 con 0x7f2660155a70 2024-09-17T04:04:52.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f2664c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f26600d07a0 con 0x7f2660155e80 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f265e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f265e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38874/0 (socket says 172.21.15.129:38874) 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.290+0000 7f265e800640 1 -- 172.21.15.129:0/1436741340 learned_addr learned my addr 172.21.15.129:0/1436741340 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265d400640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2660155e80 0x7f26601e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265de00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2660155a70 0x7f26601512a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 -- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2660155e80 msgr2=0x7f26601e73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2660155e80 0x7f26601e73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 -- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2660155a70 msgr2=0x7f26601512a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2660155a70 0x7f26601512a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 -- 172.21.15.129:0/1436741340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26601f4f00 con 0x7f266015f360 2024-09-17T04:04:52.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f265e800640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f265802aff0 tx=0x7f265802f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.291+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658033050 con 0x7f266015f360 2024-09-17T04:04:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.292+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f26580044d0 con 0x7f266015f360 2024-09-17T04:04:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.292+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26601f51e0 con 0x7f266015f360 2024-09-17T04:04:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.292+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658038450 con 0x7f266015f360 2024-09-17T04:04:52.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.292+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26601f57a0 con 0x7f266015f360 2024-09-17T04:04:52.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.293+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26600708d0 con 0x7f266015f360 2024-09-17T04:04:52.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.294+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2658036070 con 0x7f266015f360 2024-09-17T04:04:52.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.295+0000 7f2656000640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 0x7f264007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:52.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.295+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f26580bc0d0 con 0x7f266015f360 2024-09-17T04:04:52.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.295+0000 7f265de00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 0x7f264007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:52.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.296+0000 7f265de00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 0x7f264007b0d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2650005fd0 tx=0x7f265000a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:52.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.299+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2658083e60 con 0x7f266015f360 2024-09-17T04:04:52.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.527+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2660068fe0 con 0x7f266015f360 2024-09-17T04:04:52.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.528+0000 7f2656000640 1 -- 172.21.15.129:0/1436741340 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7f2658088890 con 0x7f266015f360 2024-09-17T04:04:52.528 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:52.528 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:52.529 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 msgr2=0x7f264007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 0x7f264007b0d0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f2650005fd0 tx=0x7f265000a250 comp rx=0 tx=0).stop 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 msgr2=0x7f2660156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f265802aff0 tx=0x7f265802f6d0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 shutdown_connections 2024-09-17T04:04:52.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2640078bc0 0x7f264007b0d0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f266015f360 0x7f2660156b00 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2660155e80 0x7f26601e73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 --2- 172.21.15.129:0/1436741340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2660155a70 0x7f26601512a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 >> 172.21.15.129:0/1436741340 conn(0x7f266006cfc0 msgr2=0x7f266015cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 shutdown_connections 2024-09-17T04:04:52.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:52.531+0000 7f2664c00640 1 -- 172.21.15.129:0/1436741340 wait complete. 2024-09-17T04:04:52.758 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:52.758 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:04:53.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 -- 172.21.15.129:0/2137779096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340cfd10 msgr2=0x7f17340cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 --2- 172.21.15.129:0/2137779096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340cfd10 0x7f17340cd8e0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f17341572f0 tx=0x7f172002a4a0 comp rx=0 tx=0).stop 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 -- 172.21.15.129:0/2137779096 shutdown_connections 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 --2- 172.21.15.129:0/2137779096 >> v2:172.21.15.181:3300/0 conn(0x7f17340d0620 0x7f17340d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 --2- 172.21.15.129:0/2137779096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340cfd10 0x7f17340cd8e0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 -- 172.21.15.129:0/2137779096 >> 172.21.15.129:0/2137779096 conn(0x7f173406cfc0 msgr2=0x7f173406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 -- 172.21.15.129:0/2137779096 shutdown_connections 2024-09-17T04:04:53.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.986+0000 7f173ac00640 1 -- 172.21.15.129:0/2137779096 wait complete. 2024-09-17T04:04:53.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.988+0000 7f173ac00640 1 Processor -- start 2024-09-17T04:04:53.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.989+0000 7f173ac00640 1 -- start start 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17340cfd10 0x7f17341583a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173415cd30 0x7f17340b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1734073890 con 0x7f17340d0620 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1734073710 con 0x7f173415cd30 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f173ac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1734073a10 con 0x7f17340cfd10 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f1733400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f1733400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35500/0 (socket says 172.21.15.129:35500) 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f1733400640 1 -- 172.21.15.129:0/2750512174 learned_addr learned my addr 172.21.15.129:0/2750512174 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f1738c00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173415cd30 0x7f17340b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:53.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.990+0000 7f1733e00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17340cfd10 0x7f17341583a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 -- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17340cfd10 msgr2=0x7f17341583a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17340cfd10 0x7f17341583a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 -- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173415cd30 msgr2=0x7f17340b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173415cd30 0x7f17340b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:53.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 -- 172.21.15.129:0/2750512174 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1734149310 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f1733400640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f172002aa90 tx=0x7f1720033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.991+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720031d60 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.992+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1720033e10 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.992+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17340af1c0 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.992+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1720005260 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.992+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17340ac4b0 con 0x7f17340d0620 2024-09-17T04:04:53.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.993+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17340abfc0 con 0x7f17340d0620 2024-09-17T04:04:53.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.994+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1720005400 con 0x7f17340d0620 2024-09-17T04:04:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.995+0000 7f172be00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 0x7f171407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.995+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f17200bbf50 con 0x7f17340d0620 2024-09-17T04:04:53.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.995+0000 7f1733e00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 0x7f171407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.996+0000 7f1733e00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 0x7f171407b0a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1724000f60 tx=0x7f1724009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:53.999+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f172003f030 con 0x7f17340d0620 2024-09-17T04:04:54.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.218+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1734073f50 con 0x7f17340d0620 2024-09-17T04:04:54.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.219+0000 7f172be00640 1 -- 172.21.15.129:0/2750512174 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7f1720005d00 con 0x7f17340d0620 2024-09-17T04:04:54.219 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:54.219 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:54.219 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:54.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 msgr2=0x7f171407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 0x7f171407b0a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1724000f60 tx=0x7f1724009040 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 msgr2=0x7f1734148dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f172002aa90 tx=0x7f1720033c20 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 shutdown_connections 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1714078b90 0x7f171407b0a0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f173415cd30 0x7f17340b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17340d0620 0x7f1734148dd0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 --2- 172.21.15.129:0/2750512174 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17340cfd10 0x7f17341583a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 >> 172.21.15.129:0/2750512174 conn(0x7f173406cfc0 msgr2=0x7f173406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 shutdown_connections 2024-09-17T04:04:54.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:54.221+0000 7f173ac00640 1 -- 172.21.15.129:0/2750512174 wait complete. 2024-09-17T04:04:54.448 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:54.449 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:04:55.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 -- 172.21.15.129:0/2129782814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c15cd30 msgr2=0x7fdf9c0cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/2129782814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c15cd30 0x7fdf9c0cfd10 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c06de00 tx=0x7fdf8c02a4a0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 -- 172.21.15.129:0/2129782814 shutdown_connections 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/2129782814 >> v2:172.21.15.181:3300/0 conn(0x7fdf9c06f800 0x7fdf9c0cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/2129782814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c15cd30 0x7fdf9c0cfd10 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.681+0000 7fdfa1c00640 1 -- 172.21.15.129:0/2129782814 >> 172.21.15.129:0/2129782814 conn(0x7fdf9c06cfc0 msgr2=0x7fdf9c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:55.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.682+0000 7fdfa1c00640 1 -- 172.21.15.129:0/2129782814 shutdown_connections 2024-09-17T04:04:55.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.682+0000 7fdfa1c00640 1 -- 172.21.15.129:0/2129782814 wait complete. 2024-09-17T04:04:55.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.684+0000 7fdfa1c00640 1 Processor -- start 2024-09-17T04:04:55.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.684+0000 7fdfa1c00640 1 -- start start 2024-09-17T04:04:55.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:55.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c06fc10 0x7fdf9c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:55.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 0x7fdf9c1f8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:55.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c073890 con 0x7fdf9c06f800 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c073710 con 0x7fdf9c06fc10 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.685+0000 7fdfa1c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9c073a10 con 0x7fdf9c15cd30 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35522/0 (socket says 172.21.15.129:35522) 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 -- 172.21.15.129:0/1716784368 learned_addr learned my addr 172.21.15.129:0/1716784368 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf92000640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c06fc10 0x7fdf9c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9b400640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 0x7fdf9c1f8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 -- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 msgr2=0x7fdf9c1f8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 0x7fdf9c1f8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 -- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c06fc10 msgr2=0x7fdf9c0ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c06fc10 0x7fdf9c0ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 -- 172.21.15.129:0/1716784368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c0af390 con 0x7fdf9c06f800 2024-09-17T04:04:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9b400640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 0x7fdf9c1f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:04:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.686+0000 7fdf9aa00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c02aa90 tx=0x7fdf8c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:55.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.687+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c031d60 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.687+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdf8c033e10 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.687+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c005260 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.687+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c0af0b0 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.687+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c1548b0 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.688+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9c0b2960 con 0x7fdf9c06f800 2024-09-17T04:04:55.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.689+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdf8c005400 con 0x7fdf9c06f800 2024-09-17T04:04:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.690+0000 7fdf93400640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 0x7fdf7807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.690+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fdf8c0bbe70 con 0x7fdf9c06f800 2024-09-17T04:04:55.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.691+0000 7fdf92000640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 0x7fdf7807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.691+0000 7fdf92000640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 0x7fdf7807b0a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c157220 tx=0x7fdf8400b390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:55.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.695+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdf9c0b2960 con 0x7fdf9c06f800 2024-09-17T04:04:55.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.914+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf9c073ec0 con 0x7fdf9c06f800 2024-09-17T04:04:55.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.914+0000 7fdf93400640 1 -- 172.21.15.129:0/1716784368 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7fdf8c03f030 con 0x7fdf9c06f800 2024-09-17T04:04:55.914 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:55.915 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:55.915 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 msgr2=0x7fdf7807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 0x7fdf7807b0a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c157220 tx=0x7fdf8400b390 comp rx=0 tx=0).stop 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 msgr2=0x7fdf9c1f8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c02aa90 tx=0x7fdf8c033c20 comp rx=0 tx=0).stop 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 shutdown_connections 2024-09-17T04:04:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdf78078b90 0x7fdf7807b0a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdf9c15cd30 0x7fdf9c1f8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf9c06fc10 0x7fdf9c0ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 --2- 172.21.15.129:0/1716784368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdf9c06f800 0x7fdf9c1f8960 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 >> 172.21.15.129:0/1716784368 conn(0x7fdf9c06cfc0 msgr2=0x7fdf9c070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 shutdown_connections 2024-09-17T04:04:55.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:55.917+0000 7fdfa1c00640 1 -- 172.21.15.129:0/1716784368 wait complete. 2024-09-17T04:04:56.140 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:56.141 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:04:57.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 -- 172.21.15.129:0/3335838723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 msgr2=0x7fec0c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 --2- 172.21.15.129:0/3335838723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec0c00b9c0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 crypto rx=0x7fec1c005fd0 tx=0x7fec0c005830 comp rx=0 tx=0).stop 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 -- 172.21.15.129:0/3335838723 shutdown_connections 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 --2- 172.21.15.129:0/3335838723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec0c00b9c0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 -- 172.21.15.129:0/3335838723 >> 172.21.15.129:0/3335838723 conn(0x7fec2406cfc0 msgr2=0x7fec2406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 -- 172.21.15.129:0/3335838723 shutdown_connections 2024-09-17T04:04:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.370+0000 7fec2c400640 1 -- 172.21.15.129:0/3335838723 wait complete. 2024-09-17T04:04:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.373+0000 7fec2c400640 1 Processor -- start 2024-09-17T04:04:57.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.373+0000 7fec2c400640 1 -- start start 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec241f5c40 0x7fec240ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 0x7fec241f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec240cdcc0 con 0x7fec0c005350 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fec240cdb40 con 0x7fec241f5c40 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec2c400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fec240cde40 con 0x7fec240aee70 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.374+0000 7fec29a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35542/0 (socket says 172.21.15.129:35542) 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 -- 172.21.15.129:0/1521803494 learned_addr learned my addr 172.21.15.129:0/1521803494 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:57.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29000640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec241f5c40 0x7fec240ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec2a400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 0x7fec241f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 -- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 msgr2=0x7fec241f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 0x7fec241f87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 -- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec241f5c40 msgr2=0x7fec240ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec241f5c40 0x7fec240ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 -- 172.21.15.129:0/1521803494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec240af280 con 0x7fec0c005350 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec2a400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 0x7fec241f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec29a00640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fec18006680 tx=0x7fec180039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18015070 con 0x7fec0c005350 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec18011900 con 0x7fec0c005350 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec240aa630 con 0x7fec0c005350 2024-09-17T04:04:57.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.375+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec18011d50 con 0x7fec0c005350 2024-09-17T04:04:57.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.376+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec240acfd0 con 0x7fec0c005350 2024-09-17T04:04:57.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.377+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec2406fbe0 con 0x7fec0c005350 2024-09-17T04:04:57.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.378+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fec18018070 con 0x7fec0c005350 2024-09-17T04:04:57.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.379+0000 7fec21600640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 0x7fec0407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:57.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.379+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fec1809ba70 con 0x7fec0c005350 2024-09-17T04:04:57.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.380+0000 7fec29000640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 0x7fec0407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:57.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.380+0000 7fec29000640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 0x7fec0407b0d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.383+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec240af280 con 0x7fec0c005350 2024-09-17T04:04:57.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.601+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec24156e20 con 0x7fec0c005350 2024-09-17T04:04:57.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.602+0000 7fec21600640 1 -- 172.21.15.129:0/1521803494 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7fec18063e90 con 0x7fec0c005350 2024-09-17T04:04:57.602 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:57.602 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:57.602 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:57.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 msgr2=0x7fec0407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:57.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 0x7fec0407b0d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10006b70 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 msgr2=0x7fec241f27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7fec18006680 tx=0x7fec180039c0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 shutdown_connections 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec04078bc0 0x7fec0407b0d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec240aee70 0x7fec241f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec241f5c40 0x7fec240ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 --2- 172.21.15.129:0/1521803494 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec0c005350 0x7fec241f27f0 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 >> 172.21.15.129:0/1521803494 conn(0x7fec2406cfc0 msgr2=0x7fec240b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 shutdown_connections 2024-09-17T04:04:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:57.604+0000 7fec2c400640 1 -- 172.21.15.129:0/1521803494 wait complete. 2024-09-17T04:04:57.831 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:57.832 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:04:58.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 -- 172.21.15.129:0/2594077914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 msgr2=0x7fca400d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 --2- 172.21.15.129:0/2594077914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca400d0ad0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7fca400d1410 tx=0x7fca3802a980 comp rx=0 tx=0).stop 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 -- 172.21.15.129:0/2594077914 shutdown_connections 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 --2- 172.21.15.129:0/2594077914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca400d0ad0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 --2- 172.21.15.129:0/2594077914 >> v2:172.21.15.181:3300/0 conn(0x7fca400cd780 0x7fca400cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 --2- 172.21.15.129:0/2594077914 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca4015a460 0x7fca4006fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 -- 172.21.15.129:0/2594077914 >> 172.21.15.129:0/2594077914 conn(0x7fca4006cfc0 msgr2=0x7fca4006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 -- 172.21.15.129:0/2594077914 shutdown_connections 2024-09-17T04:04:59.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.064+0000 7fca47600640 1 -- 172.21.15.129:0/2594077914 wait complete. 2024-09-17T04:04:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.066+0000 7fca47600640 1 Processor -- start 2024-09-17T04:04:59.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.066+0000 7fca47600640 1 -- start start 2024-09-17T04:04:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca400cd780 0x7fca40159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca4015a460 0x7fca40142870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca4015d450 con 0x7fca400d06c0 2024-09-17T04:04:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca4015d2d0 con 0x7fca4015a460 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca47600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fca4015d5d0 con 0x7fca400cd780 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca3fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca3fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35562/0 (socket says 172.21.15.129:35562) 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca3fe00640 1 -- 172.21.15.129:0/920229365 learned_addr learned my addr 172.21.15.129:0/920229365 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca44c00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca400cd780 0x7fca40159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.068+0000 7fca45600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca4015a460 0x7fca40142870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 -- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca400cd780 msgr2=0x7fca40159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca400cd780 0x7fca40159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 -- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca4015a460 msgr2=0x7fca40142870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca4015a460 0x7fca40142870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 -- 172.21.15.129:0/920229365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca40159750 con 0x7fca400d06c0 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3fe00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fca3000e710 tx=0x7fca3000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:59.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3000cab0 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca3000cd40 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca400ad9b0 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30010690 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.069+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca400af090 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.070+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca4015daf0 con 0x7fca400d06c0 2024-09-17T04:04:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.072+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fca30010a40 con 0x7fca400d06c0 2024-09-17T04:04:59.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.073+0000 7fca3cc00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 0x7fca2407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:04:59.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.073+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fca3009b950 con 0x7fca400d06c0 2024-09-17T04:04:59.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.074+0000 7fca44c00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 0x7fca2407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:04:59.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.074+0000 7fca44c00640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 0x7fca2407b0d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fca400cec40 tx=0x7fca34005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:04:59.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.078+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fca40159750 con 0x7fca400d06c0 2024-09-17T04:04:59.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.294+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca4015dfe0 con 0x7fca400d06c0 2024-09-17T04:04:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.295+0000 7fca3cc00640 1 -- 172.21.15.129:0/920229365 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7fca30063db0 con 0x7fca400d06c0 2024-09-17T04:04:59.295 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:04:59.295 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:04:59.295 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:04:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 msgr2=0x7fca2407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 0x7fca2407b0d0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fca400cec40 tx=0x7fca34005f90 comp rx=0 tx=0).stop 2024-09-17T04:04:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 msgr2=0x7fca40142330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:04:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 secure :-1 s=READY pgs=2280 cs=0 l=1 rev1=1 crypto rx=0x7fca3000e710 tx=0x7fca3000ebd0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 shutdown_connections 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fca24078bc0 0x7fca2407b0d0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca4015a460 0x7fca40142870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca400d06c0 0x7fca40142330 unknown :-1 s=CLOSED pgs=2280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 --2- 172.21.15.129:0/920229365 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fca400cd780 0x7fca40159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 >> 172.21.15.129:0/920229365 conn(0x7fca4006cfc0 msgr2=0x7fca4006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 shutdown_connections 2024-09-17T04:04:59.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:04:59.297+0000 7fca47600640 1 -- 172.21.15.129:0/920229365 wait complete. 2024-09-17T04:04:59.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:04:59.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:05:00.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:00.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.753+0000 7f4285600640 1 -- 172.21.15.129:0/2810618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4264003a50 msgr2=0x7f4264003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:00.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.753+0000 7f4285600640 1 --2- 172.21.15.129:0/2810618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4264003a50 0x7f4264003ec0 secure :-1 s=READY pgs=2282 cs=0 l=1 rev1=1 crypto rx=0x7f427000c310 tx=0x7f4264005830 comp rx=0 tx=0).stop 2024-09-17T04:05:00.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.753+0000 7f4285600640 1 -- 172.21.15.129:0/2810618914 shutdown_connections 2024-09-17T04:05:00.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.753+0000 7f4285600640 1 --2- 172.21.15.129:0/2810618914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4264003a50 0x7f4264003ec0 unknown :-1 s=CLOSED pgs=2282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.753+0000 7f4285600640 1 -- 172.21.15.129:0/2810618914 >> 172.21.15.129:0/2810618914 conn(0x7f428006cfc0 msgr2=0x7f428006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:00.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.754+0000 7f4285600640 1 -- 172.21.15.129:0/2810618914 shutdown_connections 2024-09-17T04:05:00.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.754+0000 7f4285600640 1 -- 172.21.15.129:0/2810618914 wait complete. 2024-09-17T04:05:00.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.756+0000 7f4285600640 1 Processor -- start 2024-09-17T04:05:00.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.756+0000 7f4285600640 1 -- start start 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 0x7f42801f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 0x7f42801ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42800b4ee0 con 0x7f428014e710 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42800b4d60 con 0x7f4280152e20 2024-09-17T04:05:00.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.757+0000 7f4285600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4280155a70 con 0x7f4264003a50 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35584/0 (socket says 172.21.15.129:35584) 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 -- 172.21.15.129:0/1747210780 learned_addr learned my addr 172.21.15.129:0/1747210780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 -- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 msgr2=0x7f42801f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427f400640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 0x7f42801ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427ea00640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 0x7f42801f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 0x7f42801f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 -- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 msgr2=0x7f42801ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 0x7f42801ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 -- 172.21.15.129:0/1747210780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42801f6550 con 0x7f428014e710 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427ea00640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 0x7f42801f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427f400640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 0x7f42801ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:00.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.758+0000 7f427e000640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f426c004d90 tx=0x7f426c001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.759+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c001d40 con 0x7f428014e710 2024-09-17T04:05:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.759+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f426c016070 con 0x7f428014e710 2024-09-17T04:05:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.759+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42801f6830 con 0x7f428014e710 2024-09-17T04:05:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.759+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c010b60 con 0x7f428014e710 2024-09-17T04:05:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.759+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42800b10e0 con 0x7f428014e710 2024-09-17T04:05:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.760+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42801f62c0 con 0x7f428014e710 2024-09-17T04:05:00.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.761+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f426c010d00 con 0x7f428014e710 2024-09-17T04:05:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.762+0000 7f4276a00640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 0x7f426007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.762+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f426c09b590 con 0x7f428014e710 2024-09-17T04:05:00.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.763+0000 7f427ea00640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 0x7f426007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:00.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.764+0000 7f427ea00640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 0x7f426007b0a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4270033000 tx=0x7f42700019f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:00.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.766+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f426c0638e0 con 0x7f428014e710 2024-09-17T04:05:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.982+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42800b1520 con 0x7f428014e710 2024-09-17T04:05:00.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.983+0000 7f4276a00640 1 -- 172.21.15.129:0/1747210780 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7f426c0631e0 con 0x7f428014e710 2024-09-17T04:05:00.983 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:00.984 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:00.984 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 msgr2=0x7f426007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 0x7f426007b0a0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f4270033000 tx=0x7f42700019f0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 msgr2=0x7f42801528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 secure :-1 s=READY pgs=2283 cs=0 l=1 rev1=1 crypto rx=0x7f426c004d90 tx=0x7f426c001630 comp rx=0 tx=0).stop 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 shutdown_connections 2024-09-17T04:05:00.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4260078b90 0x7f426007b0a0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4280152e20 0x7f42801ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f428014e710 0x7f42801528e0 unknown :-1 s=CLOSED pgs=2283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 --2- 172.21.15.129:0/1747210780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4264003a50 0x7f42801f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 >> 172.21.15.129:0/1747210780 conn(0x7f428006cfc0 msgr2=0x7f42800d0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 shutdown_connections 2024-09-17T04:05:00.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:00.986+0000 7f4285600640 1 -- 172.21.15.129:0/1747210780 wait complete. 2024-09-17T04:05:01.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:05:01.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:05:02.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:02.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 -- 172.21.15.129:0/232323461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c6c004d90 msgr2=0x7f6c6c005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/232323461 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c6c004d90 0x7f6c6c005220 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f6c80005fd0 tx=0x7f6c6c005830 comp rx=0 tx=0).stop 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8da00640 1 -- 172.21.15.129:0/232323461 reap_dead start 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 -- 172.21.15.129:0/232323461 shutdown_connections 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 -- 172.21.15.129:0/232323461 >> 172.21.15.129:0/232323461 conn(0x7f6c8806cfc0 msgr2=0x7f6c8806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 -- 172.21.15.129:0/232323461 shutdown_connections 2024-09-17T04:05:02.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.446+0000 7f6c8fa00640 1 -- 172.21.15.129:0/232323461 wait complete. 2024-09-17T04:05:02.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.448+0000 7f6c8fa00640 1 Processor -- start 2024-09-17T04:05:02.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.448+0000 7f6c8fa00640 1 -- start start 2024-09-17T04:05:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c6c004d90 0x7f6c881522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c881e67c0 0x7f6c88155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:02.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6c880d0ed0 con 0x7f6c8815a100 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6c880d0d50 con 0x7f6c881e67c0 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6c88073710 con 0x7f6c6c004d90 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c87e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c87e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:35596/0 (socket says 172.21.15.129:35596) 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c87e00640 1 -- 172.21.15.129:0/195763747 learned_addr learned my addr 172.21.15.129:0/195763747 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8d000640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c6c004d90 0x7f6c881522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c8da00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c881e67c0 0x7f6c88155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c87e00640 1 -- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c6c004d90 msgr2=0x7f6c881522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.450+0000 7f6c87e00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c6c004d90 0x7f6c881522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c87e00640 1 -- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c881e67c0 msgr2=0x7f6c88155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c87e00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c881e67c0 0x7f6c88155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c87e00640 1 -- 172.21.15.129:0/195763747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c88159660 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c87e00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6c74003e00 tx=0x7f6c740017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74011640 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6c74016020 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c74011aa0 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c88152810 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.451+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c88159900 con 0x7f6c8815a100 2024-09-17T04:05:02.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.452+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c88073c70 con 0x7f6c8815a100 2024-09-17T04:05:02.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.453+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6c74011c40 con 0x7f6c8815a100 2024-09-17T04:05:02.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.454+0000 7f6c84c00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 0x7f6c6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:02.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.455+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f6c7409bbf0 con 0x7f6c8815a100 2024-09-17T04:05:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.455+0000 7f6c8d000640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 0x7f6c6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:02.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.455+0000 7f6c8d000640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 0x7f6c6807b0a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6c78006b50 tx=0x7f6c78000bc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:02.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.459+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6c74020020 con 0x7f6c8815a100 2024-09-17T04:05:02.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.678+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c88156e70 con 0x7f6c8815a100 2024-09-17T04:05:02.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.678+0000 7f6c84c00640 1 -- 172.21.15.129:0/195763747 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7f6c74063fd0 con 0x7f6c8815a100 2024-09-17T04:05:02.679 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:02.679 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:02.679 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:05:02.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 msgr2=0x7f6c6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:02.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 0x7f6c6807b0a0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6c78006b50 tx=0x7f6c78000bc0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 msgr2=0x7f6c88155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f6c74003e00 tx=0x7f6c740017a0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 shutdown_connections 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6c68078b90 0x7f6c6807b0a0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c881e67c0 0x7f6c88155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6c8815a100 0x7f6c88155a90 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 --2- 172.21.15.129:0/195763747 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6c6c004d90 0x7f6c881522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 >> 172.21.15.129:0/195763747 conn(0x7f6c8806cfc0 msgr2=0x7f6c8806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 shutdown_connections 2024-09-17T04:05:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:02.681+0000 7f6c8fa00640 1 -- 172.21.15.129:0/195763747 wait complete. 2024-09-17T04:05:02.906 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:05:02.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:05:03.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 -- 172.21.15.129:0/2164582491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c0cfd10 msgr2=0x7fe24c0cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 --2- 172.21.15.129:0/2164582491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c0cfd10 0x7fe24c0cd8e0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7fe234005fd0 tx=0x7fe23402a5c0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 -- 172.21.15.129:0/2164582491 shutdown_connections 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 --2- 172.21.15.129:0/2164582491 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 0x7fe24c0d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 --2- 172.21.15.129:0/2164582491 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c0cfd10 0x7fe24c0cd8e0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 -- 172.21.15.129:0/2164582491 >> 172.21.15.129:0/2164582491 conn(0x7fe24c06cfc0 msgr2=0x7fe24c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.140+0000 7fe251800640 1 -- 172.21.15.129:0/2164582491 shutdown_connections 2024-09-17T04:05:04.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.141+0000 7fe251800640 1 -- 172.21.15.129:0/2164582491 wait complete. 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.143+0000 7fe251800640 1 Processor -- start 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.143+0000 7fe251800640 1 -- start start 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe24c0cfd10 0x7fe24c1f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 0x7fe24c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe24c073890 con 0x7fe24c15cd30 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe24c073710 con 0x7fe24c0cfd10 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe251800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe24c073a10 con 0x7fe24c0d0620 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.144+0000 7fe24b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47374/0 (socket says 172.21.15.129:47374) 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 -- 172.21.15.129:0/4238551590 learned_addr learned my addr 172.21.15.129:0/4238551590 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe242000640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 0x7fe24c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:04.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24aa00640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe24c0cfd10 0x7fe24c1f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 -- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 msgr2=0x7fe24c153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 0x7fe24c153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 -- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe24c0cfd10 msgr2=0x7fe24c1f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe24c0cfd10 0x7fe24c1f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 -- 172.21.15.129:0/4238551590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24c1f3a10 con 0x7fe24c15cd30 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe24b400640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fe24400caa0 tx=0x7fe24400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.145+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24400adf0 con 0x7fe24c15cd30 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.146+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe24400f540 con 0x7fe24c15cd30 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.146+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24400f990 con 0x7fe24c15cd30 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.146+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24c151f10 con 0x7fe24c15cd30 2024-09-17T04:05:04.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.146+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24c1f3c80 con 0x7fe24c15cd30 2024-09-17T04:05:04.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.147+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe24c151c80 con 0x7fe24c15cd30 2024-09-17T04:05:04.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.148+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe24400fb30 con 0x7fe24c15cd30 2024-09-17T04:05:04.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.149+0000 7fe243400640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 0x7fe22807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:04.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.149+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fe24409b880 con 0x7fe24c15cd30 2024-09-17T04:05:04.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.149+0000 7fe24aa00640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 0x7fe22807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:04.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.150+0000 7fe24aa00640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 0x7fe22807b0d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe23c007fd0 tx=0x7fe23c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:04.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.154+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe244014030 con 0x7fe24c15cd30 2024-09-17T04:05:04.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.370+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe24c0b2f80 con 0x7fe24c15cd30 2024-09-17T04:05:04.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.371+0000 7fe243400640 1 -- 172.21.15.129:0/4238551590 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+3613 (secure 0 0 0) 0x7fe24c0b2f80 con 0x7fe24c15cd30 2024-09-17T04:05:04.371 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:04.371 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":197,"btime":"2024-09-17T04:04:41:015299+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:04:41.015292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0,1],"up":{"mds_0":10440,"mds_1":9895},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9895":{"gid":9895,"name":"c","rank":1,"incarnation":189,"state":"up:stopping","state_seq":5,"addr":"172.21.15.129:6833/1714996749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1714996749},{"type":"v1","addr":"172.21.15.129:6833","nonce":1714996749}]},"join_fscid":-1,"export_targets":[0],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:04.371 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 197 2024-09-17T04:05:04.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 msgr2=0x7fe22807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 0x7fe22807b0d0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fe23c007fd0 tx=0x7fe23c009040 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 msgr2=0x7fe24c156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7fe24400caa0 tx=0x7fe24400cf60 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 shutdown_connections 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe228078bc0 0x7fe22807b0d0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe24c15cd30 0x7fe24c156810 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe24c0d0620 0x7fe24c153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 --2- 172.21.15.129:0/4238551590 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe24c0cfd10 0x7fe24c1f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 >> 172.21.15.129:0/4238551590 conn(0x7fe24c06cfc0 msgr2=0x7fe24c15b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 shutdown_connections 2024-09-17T04:05:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:04.373+0000 7fe251800640 1 -- 172.21.15.129:0/4238551590 wait complete. 2024-09-17T04:05:04.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 197, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:04:41.015292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0, 1], 'up': {'mds_0': 10440, 'mds_1': 9895}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9895': {'gid': 9895, 'name': 'c', 'rank': 1, 'incarnation': 189, 'state': 'up:stopping', 'state_seq': 5, 'addr': '172.21.15.129:6833/1714996749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 1714996749}, {'type': 'v1', 'addr': '172.21.15.129:6833', 'nonce': 1714996749}]}, 'join_fscid': -1, 'export_targets': [0], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:05:04.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9895:up:stopping 2024-09-17T04:05:05.222 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:05:05.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 -- 172.21.15.129:0/1749352404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806815cd30 msgr2=0x7f80680cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 --2- 172.21.15.129:0/1749352404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806815cd30 0x7f80680cfd10 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f806806de00 tx=0x7f805802a4a0 comp rx=0 tx=0).stop 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 -- 172.21.15.129:0/1749352404 shutdown_connections 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 --2- 172.21.15.129:0/1749352404 >> v2:172.21.15.181:3300/0 conn(0x7f806806f800 0x7f80680cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 --2- 172.21.15.129:0/1749352404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806815cd30 0x7f80680cfd10 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 -- 172.21.15.129:0/1749352404 >> 172.21.15.129:0/1749352404 conn(0x7f806806cfc0 msgr2=0x7f806806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.828+0000 7f806e400640 1 -- 172.21.15.129:0/1749352404 shutdown_connections 2024-09-17T04:05:05.829 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.829+0000 7f806e400640 1 -- 172.21.15.129:0/1749352404 wait complete. 2024-09-17T04:05:05.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.831+0000 7f806e400640 1 Processor -- start 2024-09-17T04:05:05.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.831+0000 7f806e400640 1 -- start start 2024-09-17T04:05:05.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806806f800 0x7f80681f98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:05.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:05.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f806815cd30 0x7f8068152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:05.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8068073890 con 0x7f806806fc10 2024-09-17T04:05:05.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8068073710 con 0x7f806806f800 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.832+0000 7f806e400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8068073a10 con 0x7f806815cd30 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47396/0 (socket says 172.21.15.129:47396) 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 -- 172.21.15.129:0/2793925010 learned_addr learned my addr 172.21.15.129:0/2793925010 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8067400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806806f800 0x7f80681f98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8067e00640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f806815cd30 0x7f8068152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 -- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f806815cd30 msgr2=0x7f8068152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f806815cd30 0x7f8068152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 -- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806806f800 msgr2=0x7f80681f98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806806f800 0x7f80681f98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 -- 172.21.15.129:0/2793925010 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80681f6070 con 0x7f806806fc10 2024-09-17T04:05:05.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.833+0000 7f8066a00640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f805400d6d0 tx=0x7f805400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.834+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054017070 con 0x7f806806fc10 2024-09-17T04:05:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.834+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8054004310 con 0x7f806806fc10 2024-09-17T04:05:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.834+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8054009a80 con 0x7f806806fc10 2024-09-17T04:05:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.834+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80681f6350 con 0x7f806806fc10 2024-09-17T04:05:05.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.834+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80680ab180 con 0x7f806806fc10 2024-09-17T04:05:05.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.835+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8068073ec0 con 0x7f806806fc10 2024-09-17T04:05:05.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.836+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8054009c20 con 0x7f806806fc10 2024-09-17T04:05:05.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.838+0000 7f805f400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 0x7f804807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:05.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.838+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f805409ad80 con 0x7f806806fc10 2024-09-17T04:05:05.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.838+0000 7f8067400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 0x7f804807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:05.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.839+0000 7f8067400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 0x7f804807b0d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8058034000 tx=0x7f8058004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:05.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:05.842+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8054062b10 con 0x7f806806fc10 2024-09-17T04:05:06.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.058+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80681f5e00 con 0x7f806806fc10 2024-09-17T04:05:06.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.059+0000 7f805f400640 1 -- 172.21.15.129:0/2793925010 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+2768 (secure 0 0 0) 0x7f8054067540 con 0x7f806806fc10 2024-09-17T04:05:06.059 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:06.059 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":198,"btime":"2024-09-17T04:05:05:081422+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:05:04.622165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:06.059 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 198 2024-09-17T04:05:06.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 msgr2=0x7f804807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 0x7f804807b0d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f8058034000 tx=0x7f8058004730 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 msgr2=0x7f80680ae820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f805400d6d0 tx=0x7f805400db90 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 shutdown_connections 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8048078bc0 0x7f804807b0d0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f806815cd30 0x7f8068152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f806806fc10 0x7f80680ae820 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 --2- 172.21.15.129:0/2793925010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f806806f800 0x7f80681f98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 >> 172.21.15.129:0/2793925010 conn(0x7f806806cfc0 msgr2=0x7f8068070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 shutdown_connections 2024-09-17T04:05:06.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.061+0000 7f806e400640 1 -- 172.21.15.129:0/2793925010 wait complete. 2024-09-17T04:05:06.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 198, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:03:39.805108+0000', 'modified': '2024-09-17T04:05:04.622165+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 206, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10440}, 'failed': [], 'damaged': [], 'stopped': [1], 'info': {'gid_10440': {'gid': 10440, 'name': 'd', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6837/2583867786', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2583867786}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2583867786}]}, 'join_fscid': -1, 'export_targets': [1], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 10440, 'qdb_cluster': [10440]} 2024-09-17T04:05:06.285 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:05:06.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T04:05:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 -- 172.21.15.129:0/2439633618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0090003a00 msgr2=0x7f0090003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 --2- 172.21.15.129:0/2439633618 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0090003a00 0x7f0090003e70 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f009c009b50 tx=0x7f0090005830 comp rx=0 tx=0).stop 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00abe00640 1 -- 172.21.15.129:0/2439633618 reap_dead start 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 -- 172.21.15.129:0/2439633618 shutdown_connections 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 -- 172.21.15.129:0/2439633618 >> 172.21.15.129:0/2439633618 conn(0x7f00ac06cfc0 msgr2=0x7f00ac06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 -- 172.21.15.129:0/2439633618 shutdown_connections 2024-09-17T04:05:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.507+0000 7f00b2200640 1 -- 172.21.15.129:0/2439633618 wait complete. 2024-09-17T04:05:06.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.509+0000 7f00b2200640 1 Processor -- start 2024-09-17T04:05:06.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.510+0000 7f00b2200640 1 -- start start 2024-09-17T04:05:06.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 0x7f00ac1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.512 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1f6f70 0x7f00ac1f4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0d0dc0 con 0x7f00ac152e00 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0d0c40 con 0x7f00ac1f6f70 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00b2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f00ac0d0f40 con 0x7f0090003a00 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00a2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00a2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47412/0 (socket says 172.21.15.129:47412) 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.511+0000 7f00a2a00640 1 -- 172.21.15.129:0/1358594322 learned_addr learned my addr 172.21.15.129:0/1358594322 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00ab400640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 0x7f00ac1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00abe00640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1f6f70 0x7f00ac1f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 -- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 msgr2=0x7f00ac1566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 0x7f00ac1566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 -- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1f6f70 msgr2=0x7f00ac1f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1f6f70 0x7f00ac1f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 -- 172.21.15.129:0/1358594322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac1f7380 con 0x7f00ac152e00 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00ab400640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 0x7f00ac1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f009400c700 tx=0x7f009400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a3e00640 1 -- 172.21.15.129:0/1358594322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094017070 con 0x7f00ac152e00 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a3e00640 1 -- 172.21.15.129:0/1358594322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0094005380 con 0x7f00ac152e00 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ac154b20 con 0x7f00ac152e00 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.512+0000 7f00a3e00640 1 -- 172.21.15.129:0/1358594322 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009400a8a0 con 0x7f00ac152e00 2024-09-17T04:05:06.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.513+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ac1550e0 con 0x7f00ac152e00 2024-09-17T04:05:06.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.515+0000 7f00a3e00640 1 -- 172.21.15.129:0/1358594322 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f009401a070 con 0x7f00ac152e00 2024-09-17T04:05:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.516+0000 7f00a3e00640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 0x7f008807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.516+0000 7f00a3e00640 1 -- 172.21.15.129:0/1358594322 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f009409a2b0 con 0x7f00ac152e00 2024-09-17T04:05:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.516+0000 7f00b9265740 1 Processor -- start 2024-09-17T04:05:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.516+0000 7f00ab400640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 0x7f008807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.516+0000 7f00b9265740 1 -- start start 2024-09-17T04:05:06.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.517+0000 7f00ab400640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 0x7f008807b0a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f009c009b50 tx=0x7f009c02d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f555d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f526370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d40f4ab430 con 0x55d40f1583b0 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d40f49b5d0 con 0x55d40f181270 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d40f48aa00 con 0x55d40f160590 2024-09-17T04:05:06.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47416/0 (socket says 172.21.15.129:47416) 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 -- 172.21.15.129:0/1690730864 learned_addr learned my addr 172.21.15.129:0/1690730864 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00ab400640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f555d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00abe00640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f526370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 -- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 msgr2=0x55d40f526370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f526370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 -- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 msgr2=0x55d40f555d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f555d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00a2a00640 1 -- 172.21.15.129:0/1690730864 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40f453520 con 0x55d40f1583b0 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00ab400640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f555d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.518+0000 7f00abe00640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f526370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:06.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.519+0000 7f00a2a00640 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f009400a430 tx=0x7f0094061730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80eb2f871d5021ce server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.519+0000 7f009b400640 1 -- 172.21.15.129:0/1690730864 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094088020 con 0x55d40f1583b0 2024-09-17T04:05:06.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.519+0000 7f009b400640 1 -- 172.21.15.129:0/1690730864 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0094061f00 con 0x55d40f1583b0 2024-09-17T04:05:06.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.519+0000 7f009b400640 1 -- 172.21.15.129:0/1690730864 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940669e0 con 0x55d40f1583b0 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 -- 172.21.15.129:0/1690730864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 msgr2=0x55d40f5562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f009400a430 tx=0x7f0094061730 comp rx=0 tx=0).stop 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 -- 172.21.15.129:0/1690730864 shutdown_connections 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f526370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f5562c0 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 --2- 172.21.15.129:0/1690730864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f555d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 -- 172.21.15.129:0/1690730864 >> 172.21.15.129:0/1690730864 conn(0x55d40f1753c0 msgr2=0x55d40f52bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 -- 172.21.15.129:0/1690730864 shutdown_connections 2024-09-17T04:05:06.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.520+0000 7f00b9265740 1 -- 172.21.15.129:0/1690730864 wait complete. 2024-09-17T04:05:06.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.521+0000 7f00b9265740 1 -- start start 2024-09-17T04:05:06.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.521+0000 7f00b9265740 1 Processor -- start 2024-09-17T04:05:06.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f54c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f5529f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d40f431330 con 0x55d40f1583b0 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d40f48e810 con 0x55d40f181270 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d40f1fb620 con 0x55d40f160590 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00b9265740 10 client.0 _ll_register_callbacks cb 0x55d40f436420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47432/0 (socket says 172.21.15.129:47432) 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 -- 172.21.15.129:0/2495437566 learned_addr learned my addr 172.21.15.129:0/2495437566 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00a2a00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f54c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00abe00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f5529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 -- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 msgr2=0x55d40f54c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f54c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 -- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 msgr2=0x55d40f5529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f5529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00ab400640 1 -- 172.21.15.129:0/2495437566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40f1ec850 con 0x55d40f1583b0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00a2a00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f54c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.523+0000 7f00abe00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f5529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f00ab400640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f009c009c40 tx=0x7f009c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2394e9ab7d8cae server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f008fe00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c04e070 con 0x55d40f1583b0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f009c03ed40 con 0x55d40f1583b0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0094061f00 con 0x55d40f1583b0 2024-09-17T04:05:06.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) ==== 24+0+0 (secure 0 0 0) 0x7f009c059050 con 0x55d40f1583b0 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f00b9265740 10 client.10863 fetch_fsmap learned FSMap version 199 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) -- 0x55d40f2636b0 con 0x55d40f1583b0 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.524+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0094061f00 con 0x55d40f1583b0 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 199) ==== 2300+0+0 (secure 0 0 0) 0x7f009c05c020 con 0x55d40f1583b0 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 10 client.10863 fetch_fsmap finished waiting for FSMap version 199 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 10 client.10863 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 20 client.10863 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f13fe40 0x55d40f552f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 4 client.10863 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d40f49d520 con 0x55d40f13fe40 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.525+0000 7f00a2a00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f13fe40 0x55d40f552f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.526+0000 7f00a2a00640 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f13fe40 0x55d40f552f30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b73c5ee8de78a7 server_cookie=1a92439b86e07a5b in_seq=0 out_seq=0 2024-09-17T04:05:06.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.526+0000 7f008fe00640 10 client.10863 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.526+0000 7f008fe00640 1 -- 172.21.15.129:0/2495437566 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d40f49d520 con 0x55d40f13fe40 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.526+0000 7f008fe00640 10 client.10863 handle_command_reply: tid=0 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.526+0000 7f00b9265740 1 client.10863 shutdown 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 msgr2=0x55d40f56a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f009c009c40 tx=0x7f009c0046c0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 shutdown_connections 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f13fe40 0x55d40f552f30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f181270 0x55d40f5529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f54c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 --2- 172.21.15.129:0/2495437566 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f56a7d0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f008fe00640 0 client.10863 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:05:06.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 >> 172.21.15.129:0/2495437566 conn(0x55d40f1753c0 msgr2=0x55d40f521770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 shutdown_connections 2024-09-17T04:05:06.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 1 -- 172.21.15.129:0/2495437566 wait complete. 2024-09-17T04:05:06.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.527+0000 7f00b9265740 20 client.10863 trim_cache size 0 max 16384 2024-09-17T04:05:06.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.547+0000 7f00b9265740 1 Processor -- start 2024-09-17T04:05:06.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.547+0000 7f00b9265740 1 -- start start 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f1583b0 0x55d40f550e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f552b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0b5500 con 0x55d40f181270 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0b4630 con 0x55d40f1583b0 2024-09-17T04:05:06.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d40f54c820 con 0x55d40f160590 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47442/0 (socket says 172.21.15.129:47442) 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.548+0000 7f00abe00640 1 -- 172.21.15.129:0/3834938156 learned_addr learned my addr 172.21.15.129:0/3834938156 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f552b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00ab400640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f1583b0 0x55d40f550e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 -- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 msgr2=0x55d40f552b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f552b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 -- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f1583b0 msgr2=0x55d40f550e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f1583b0 0x55d40f550e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 -- 172.21.15.129:0/3834938156 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40f52ca60 con 0x55d40f181270 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f552b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f00abe00640 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ed0 tx=0x7f00a4009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e072000d384246 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f009aa00640 1 -- 172.21.15.129:0/3834938156 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4003b70 con 0x55d40f181270 2024-09-17T04:05:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f009aa00640 1 -- 172.21.15.129:0/3834938156 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00a4003e20 con 0x55d40f181270 2024-09-17T04:05:06.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.549+0000 7f009aa00640 1 -- 172.21.15.129:0/3834938156 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4004270 con 0x55d40f181270 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 -- 172.21.15.129:0/3834938156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 msgr2=0x55d40f555a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ed0 tx=0x7f00a4009500 comp rx=0 tx=0).stop 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 -- 172.21.15.129:0/3834938156 shutdown_connections 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f181270 0x55d40f555a70 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f160590 0x55d40f552b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 --2- 172.21.15.129:0/3834938156 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f1583b0 0x55d40f550e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 -- 172.21.15.129:0/3834938156 >> 172.21.15.129:0/3834938156 conn(0x55d40f1753c0 msgr2=0x55d40f5505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 -- 172.21.15.129:0/3834938156 shutdown_connections 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.550+0000 7f00b9265740 1 -- 172.21.15.129:0/3834938156 wait complete. 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.551+0000 7f00b9265740 1 -- start start 2024-09-17T04:05:06.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.551+0000 7f00b9265740 1 Processor -- start 2024-09-17T04:05:06.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 0x55d40f5d6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f181270 0x55d40f5f46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d40f52bb10 con 0x55d40f1583b0 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d40f527f00 con 0x55d40f160590 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d40f551380 con 0x55d40f181270 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00b9265740 10 client.0 _ll_register_callbacks cb 0x7f009000b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47446/0 (socket says 172.21.15.129:47446) 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00ab400640 1 -- 172.21.15.129:0/3116419587 learned_addr learned my addr 172.21.15.129:0/3116419587 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00abe00640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f181270 0x55d40f5f46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.553+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 0x55d40f5d6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 -- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f181270 msgr2=0x55d40f5f46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f181270 0x55d40f5f46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 -- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 msgr2=0x55d40f5d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 0x55d40f5d6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 -- 172.21.15.129:0/3116419587 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40f5931b0 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 0x55d40f5d6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00ab400640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f009c036040 tx=0x7f009c033cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b9920b516073d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f008fe00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c04e030 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f009c053440 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00a4003e20 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=199) ==== 24+0+0 (secure 0 0 0) 0x7f009c067050 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00b9265740 10 client.10869 fetch_fsmap learned FSMap version 199 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.554+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=199,monmap=2+}) -- 0x55d40f5f4c90 con 0x55d40f1583b0 2024-09-17T04:05:06.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f00a4003e20 con 0x55d40f1583b0 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 199) ==== 2300+0+0 (secure 0 0 0) 0x7f009c06a020 con 0x55d40f1583b0 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 10 client.10869 fetch_fsmap finished waiting for FSMap version 199 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 10 client.10869 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 20 client.10869 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f5da2f0 0x55d40f5528d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 4 client.10869 mds_command: new command op to 10440 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.555+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 --> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] -- command(tid 0: {"prefix": "status"}) -- 0x55d40f498fd0 con 0x55d40f5da2f0 2024-09-17T04:05:06.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f5da2f0 0x55d40f5528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f008fe00640 10 client.10869 ms_handle_connect on v2:172.21.15.181:6835/2583867786 2024-09-17T04:05:06.559 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f00a2a00640 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f5da2f0 0x55d40f5528d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d36e3ca986ebaedc server_cookie=f3fff2e8014cbb7b in_seq=0 out_seq=0 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f008fe00640 1 -- 172.21.15.129:0/3116419587 <== mds.0 v2:172.21.15.181:6835/2583867786 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55d40f498fd0 con 0x55d40f5da2f0 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f008fe00640 10 client.10869 handle_command_reply: tid=0 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f00b9265740 1 client.10869 shutdown 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 msgr2=0x55d40f58a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.556+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f009c036040 tx=0x7f009c033cc0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 shutdown_connections 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:6835/2583867786,v1:172.21.15.181:6837/2583867786] conn(0x55d40f5da2f0 0x55d40f5528d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d40f181270 0x55d40f5f46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d40f160590 0x55d40f5d6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 --2- 172.21.15.129:0/3116419587 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d40f1583b0 0x55d40f58a330 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f008fe00640 0 client.10869 ms_handle_reset on v2:172.21.15.181:6835/2583867786 2024-09-17T04:05:06.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 >> 172.21.15.129:0/3116419587 conn(0x55d40f1753c0 msgr2=0x55d40f5505c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 shutdown_connections 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 1 -- 172.21.15.129:0/3116419587 wait complete. 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b9265740 20 client.10869 trim_cache size 0 max 16384 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 10440, 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:05:06.561 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 86.248906195999993, 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 198, 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 208, 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 208, 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 91.279824583999996 2024-09-17T04:05:06.562 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 msgr2=0x7f008807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 0x7f008807b0a0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f009c009b50 tx=0x7f009c02d040 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 msgr2=0x7f00ac1f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.557+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f009400c700 tx=0x7f009400cbc0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 shutdown_connections 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0088078b90 0x7f008807b0a0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00ac1f6f70 0x7f00ac1f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00ac152e00 0x7f00ac1f6a30 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 --2- 172.21.15.129:0/1358594322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0090003a00 0x7f00ac1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 >> 172.21.15.129:0/1358594322 conn(0x7f00ac06cfc0 msgr2=0x7f00ac15ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 shutdown_connections 2024-09-17T04:05:06.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.558+0000 7f00b2200640 1 -- 172.21.15.129:0/1358594322 wait complete. 2024-09-17T04:05:06.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T04:05:06.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 -- 172.21.15.129:0/2394458351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80cb960 msgr2=0x7f0cf80cbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 --2- 172.21.15.129:0/2394458351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80cb960 0x7f0cf80cbd70 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8071ed0 tx=0x7f0ce002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 -- 172.21.15.129:0/2394458351 shutdown_connections 2024-09-17T04:05:06.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 --2- 172.21.15.129:0/2394458351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80cb960 0x7f0cf80cbd70 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 --2- 172.21.15.129:0/2394458351 >> v2:172.21.15.181:3300/0 conn(0x7f0cf80d0f20 0x7f0cf815c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 -- 172.21.15.129:0/2394458351 >> 172.21.15.129:0/2394458351 conn(0x7f0cf80b2510 msgr2=0x7f0cf80d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:06.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 -- 172.21.15.129:0/2394458351 shutdown_connections 2024-09-17T04:05:06.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.802+0000 7f0cf7200640 1 -- 172.21.15.129:0/2394458351 wait complete. 2024-09-17T04:05:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.804+0000 7f0cf7200640 1 Processor -- start 2024-09-17T04:05:06.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.804+0000 7f0cf7200640 1 -- start start 2024-09-17T04:05:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 0x7f0cf81f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 0x7f0cf80b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf80b4570 con 0x7f0cf80d0f20 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf80b43f0 con 0x7f0cf806f8a0 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0cf80b46f0 con 0x7f0cf80cb960 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47464/0 (socket says 172.21.15.129:47464) 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 -- 172.21.15.129:0/3270890328 learned_addr learned my addr 172.21.15.129:0/3270890328 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf5e00640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 0x7f0cf81f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0ced400640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 0x7f0cf80b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 -- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 msgr2=0x7f0cf80b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 0x7f0cf80b0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 -- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 msgr2=0x7f0cf81f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 0x7f0cf81f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf6800640 1 -- 172.21.15.129:0/3270890328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf80af990 con 0x7f0cf80d0f20 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0ced400640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 0x7f0cf80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.806+0000 7f0cf5e00640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 0x7f0cf81f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0cf6800640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c730 tx=0x7f0cf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0003b70 con 0x7f0cf80d0f20 2024-09-17T04:05:06.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0cf0003d30 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00052d0 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf80afc70 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.807+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf80b0230 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.808+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf80b4c80 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.810+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0cf0005680 con 0x7f0cf80d0f20 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.810+0000 7f0ceea00640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 0x7f0cd407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:06.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.811+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f0cf009aca0 con 0x7f0cf80d0f20 2024-09-17T04:05:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.811+0000 7f0cf5e00640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 0x7f0cd407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.812+0000 7f0cf5e00640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 0x7f0cd407b0a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8005fd0 tx=0x7f0ce800f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:06.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:06.815+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0cf0063100 con 0x7f0cf80d0f20 2024-09-17T04:05:07.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.066+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f0cf8075170 con 0x7f0cf80d0f20 2024-09-17T04:05:07.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.066+0000 7f0ceea00640 1 -- 172.21.15.129:0/3270890328 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1656 (secure 0 0 0) 0x7f0cf0062a00 con 0x7f0cf80d0f20 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772705378304, 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 388734976, 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 388734976, 2024-09-17T04:05:07.067 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00050283008022233844, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772705378304, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 388734976, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 388734976, 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00050283008022233844 2024-09-17T04:05:07.068 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6183499838007265e-06, 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366990491648 2024-09-17T04:05:07.069 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 30, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 61836, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 42, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 224, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 229376, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 3.1250937126969802e-07, 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366990491648 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:07.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 31, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 0, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 0, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 0, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 0, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 0, 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366990491648 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:05:07.071 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:05:07.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 msgr2=0x7f0cd407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 0x7f0cd407b0a0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8005fd0 tx=0x7f0ce800f360 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 msgr2=0x7f0cf81f3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c730 tx=0x7f0cf000cbf0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 shutdown_connections 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0cd4078b90 0x7f0cd407b0a0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0cf80d0f20 0x7f0cf81f3070 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0cf80cb960 0x7f0cf80b0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 --2- 172.21.15.129:0/3270890328 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0cf806f8a0 0x7f0cf81f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 >> 172.21.15.129:0/3270890328 conn(0x7f0cf80b2510 msgr2=0x7f0cf80747c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 shutdown_connections 2024-09-17T04:05:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.070+0000 7f0cf7200640 1 -- 172.21.15.129:0/3270890328 wait complete. 2024-09-17T04:05:07.095 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-17T04:05:07.095 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:05:07.095 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:07.181 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:07.282 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:05:07.282 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:07.282 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:07.283 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:07.318 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:07.318 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:07.318 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:07.318 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:07.356 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:07.356 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:07.356 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:07.357 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:07.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:07.394 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:07.394 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:07.394 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:07.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:07.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.655+0000 7fa030400640 1 -- 172.21.15.129:0/1020666334 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa02815a460 msgr2=0x7fa02806fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.655+0000 7fa030400640 1 --2- 172.21.15.129:0/1020666334 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa02815a460 0x7fa02806fa30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009860 tx=0x7fa01c02a510 comp rx=0 tx=0).stop 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.656+0000 7fa030400640 1 -- 172.21.15.129:0/1020666334 shutdown_connections 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.656+0000 7fa030400640 1 --2- 172.21.15.129:0/1020666334 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa02815a460 0x7fa02806fa30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.656+0000 7fa030400640 1 -- 172.21.15.129:0/1020666334 >> 172.21.15.129:0/1020666334 conn(0x7fa02806cfc0 msgr2=0x7fa02806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.656+0000 7fa030400640 1 -- 172.21.15.129:0/1020666334 shutdown_connections 2024-09-17T04:05:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.656+0000 7fa030400640 1 -- 172.21.15.129:0/1020666334 wait complete. 2024-09-17T04:05:07.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.658+0000 7fa030400640 1 Processor -- start 2024-09-17T04:05:07.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.658+0000 7fa030400640 1 -- start start 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0280cd780 0x7fa0280c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa02815a460 0x7fa028137f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa02815da20 con 0x7fa0280d0dd0 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa02815d8a0 con 0x7fa02815a460 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.659+0000 7fa030400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa02815dba0 con 0x7fa0280cd780 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47476/0 (socket says 172.21.15.129:47476) 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 -- 172.21.15.129:0/3435201124 learned_addr learned my addr 172.21.15.129:0/3435201124 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:07.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02da00640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0280cd780 0x7fa0280c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02e400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa02815a460 0x7fa028137f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 -- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0280cd780 msgr2=0x7fa0280c5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0280cd780 0x7fa0280c5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 -- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa02815a460 msgr2=0x7fa028137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa02815a460 0x7fa028137f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 -- 172.21.15.129:0/3435201124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0280ac070 con 0x7fa0280d0dd0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.660+0000 7fa02d000640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c460 tx=0x7fa01400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.661+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014016020 con 0x7fa0280d0dd0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.661+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa014012a00 con 0x7fa0280d0dd0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.661+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014012e50 con 0x7fa0280d0dd0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.661+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa028135e30 con 0x7fa0280d0dd0 2024-09-17T04:05:07.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.661+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa028137990 con 0x7fa0280d0dd0 2024-09-17T04:05:07.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.662+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0280abde0 con 0x7fa0280d0dd0 2024-09-17T04:05:07.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.663+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa014019020 con 0x7fa0280d0dd0 2024-09-17T04:05:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.664+0000 7fa025600640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 0x7fa00807b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.664+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fa01409a960 con 0x7fa0280d0dd0 2024-09-17T04:05:07.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.664+0000 7fa02da00640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 0x7fa00807b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:07.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.665+0000 7fa02da00640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 0x7fa00807b240 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa01c034000 tx=0x7fa01c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:07.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.668+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa014020030 con 0x7fa0280d0dd0 2024-09-17T04:05:07.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.885+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa028069080 con 0x7fa0280d0dd0 2024-09-17T04:05:07.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.886+0000 7fa025600640 1 -- 172.21.15.129:0/3435201124 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3597 (secure 0 0 0) 0x7fa014062670 con 0x7fa0280d0dd0 2024-09-17T04:05:07.885 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:07.885 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":199,"btime":"2024-09-17T04:05:06:087044+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1344703948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1344703948},{"type":"v1","addr":"172.21.15.129:6833","nonce":1344703948}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:05:04.622165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:07.886 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 199 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.888+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 msgr2=0x7fa00807b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.888+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 0x7fa00807b240 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa01c034000 tx=0x7fa01c004730 comp rx=0 tx=0).stop 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.888+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 msgr2=0x7fa0280c5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.888+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fa01400c460 tx=0x7fa01400c920 comp rx=0 tx=0).stop 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.888+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 shutdown_connections 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa008078d30 0x7fa00807b240 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa02815a460 0x7fa028137f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0280d0dd0 0x7fa0280c5940 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 --2- 172.21.15.129:0/3435201124 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0280cd780 0x7fa0280c5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:07.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 >> 172.21.15.129:0/3435201124 conn(0x7fa02806cfc0 msgr2=0x7fa02815c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:07.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 shutdown_connections 2024-09-17T04:05:07.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:07.889+0000 7fa030400640 1 -- 172.21.15.129:0/3435201124 wait complete. 2024-09-17T04:05:07.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 -- 172.21.15.129:0/3962697831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c0cfd10 msgr2=0x7f315c0cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 --2- 172.21.15.129:0/3962697831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c0cfd10 0x7f315c0cd8e0 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f3148009830 tx=0x7f314802a500 comp rx=0 tx=0).stop 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 -- 172.21.15.129:0/3962697831 shutdown_connections 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 --2- 172.21.15.129:0/3962697831 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0d0620 0x7f315c0d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 --2- 172.21.15.129:0/3962697831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c0cfd10 0x7f315c0cd8e0 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 -- 172.21.15.129:0/3962697831 >> 172.21.15.129:0/3962697831 conn(0x7f315c06cfc0 msgr2=0x7f315c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 -- 172.21.15.129:0/3962697831 shutdown_connections 2024-09-17T04:05:08.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.135+0000 7f3164600640 1 -- 172.21.15.129:0/3962697831 wait complete. 2024-09-17T04:05:08.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.137+0000 7f3164600640 1 Processor -- start 2024-09-17T04:05:08.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.138+0000 7f3164600640 1 -- start start 2024-09-17T04:05:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 0x7f315c0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c0d0620 0x7f315c1f81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f315c073890 con 0x7f315c15cd30 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f315c073710 con 0x7f315c0d0620 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3164600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f315c073a10 con 0x7f315c0cfd10 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3162600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3162600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47494/0 (socket says 172.21.15.129:47494) 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.139+0000 7f3162600640 1 -- 172.21.15.129:0/616436187 learned_addr learned my addr 172.21.15.129:0/616436187 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3161200640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c0d0620 0x7f315c1f81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3161c00640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 0x7f315c0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 -- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 msgr2=0x7f315c0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 0x7f315c0b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 -- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c0d0620 msgr2=0x7f315c1f81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c0d0620 0x7f315c1f81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 -- 172.21.15.129:0/616436187 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c154010 con 0x7f315c15cd30 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3161c00640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 0x7f315c0b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3162600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f315400daf0 tx=0x7f3154009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154017070 con 0x7f315c15cd30 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3154003970 con 0x7f315c15cd30 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c1542f0 con 0x7f315c15cd30 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.140+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154003da0 con 0x7f315c15cd30 2024-09-17T04:05:08.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.141+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c1548b0 con 0x7f315c15cd30 2024-09-17T04:05:08.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.142+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315c1582c0 con 0x7f315c15cd30 2024-09-17T04:05:08.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.143+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3154003f40 con 0x7f315c15cd30 2024-09-17T04:05:08.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.144+0000 7f3159600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 0x7f313c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.145+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f315409ad60 con 0x7f315c15cd30 2024-09-17T04:05:08.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.145+0000 7f3161c00640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 0x7f313c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.145+0000 7f3161c00640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 0x7f313c07b0d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3150000f60 tx=0x7f3150009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:08.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.149+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3154062b70 con 0x7f315c15cd30 2024-09-17T04:05:08.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.373+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f315c0cd1f0 con 0x7f315c15cd30 2024-09-17T04:05:08.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.374+0000 7f3159600640 1 -- 172.21.15.129:0/616436187 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3597 (secure 0 0 0) 0x7f315c0cd1f0 con 0x7f315c15cd30 2024-09-17T04:05:08.374 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:08.374 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":199,"btime":"2024-09-17T04:05:06:087044+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1344703948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1344703948},{"type":"v1","addr":"172.21.15.129:6833","nonce":1344703948}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:05:04.622165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:08.374 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 199 2024-09-17T04:05:08.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 msgr2=0x7f313c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 0x7f313c07b0d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f3150000f60 tx=0x7f3150009040 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 msgr2=0x7f315c1f86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f315400daf0 tx=0x7f3154009f60 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 shutdown_connections 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f313c078bc0 0x7f313c07b0d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f315c15cd30 0x7f315c1f86e0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f315c0d0620 0x7f315c1f81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 --2- 172.21.15.129:0/616436187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f315c0cfd10 0x7f315c0b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 >> 172.21.15.129:0/616436187 conn(0x7f315c06cfc0 msgr2=0x7f315c06f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 shutdown_connections 2024-09-17T04:05:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.376+0000 7f3164600640 1 -- 172.21.15.129:0/616436187 wait complete. 2024-09-17T04:05:08.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:08.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.619+0000 7fead6400640 1 -- 172.21.15.129:0/1965196234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 msgr2=0x7fead015c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.619+0000 7fead6400640 1 --2- 172.21.15.129:0/1965196234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead015c830 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7feac0005fd0 tx=0x7feac002a530 comp rx=0 tx=0).stop 2024-09-17T04:05:08.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 -- 172.21.15.129:0/1965196234 shutdown_connections 2024-09-17T04:05:08.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 --2- 172.21.15.129:0/1965196234 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead00cb960 0x7fead00cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 --2- 172.21.15.129:0/1965196234 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead015c830 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 -- 172.21.15.129:0/1965196234 >> 172.21.15.129:0/1965196234 conn(0x7fead00b2510 msgr2=0x7fead00d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:08.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 -- 172.21.15.129:0/1965196234 shutdown_connections 2024-09-17T04:05:08.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.620+0000 7fead6400640 1 -- 172.21.15.129:0/1965196234 wait complete. 2024-09-17T04:05:08.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.622+0000 7fead6400640 1 Processor -- start 2024-09-17T04:05:08.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.622+0000 7fead6400640 1 -- start start 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead006f8a0 0x7fead01f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead00cb960 0x7fead00ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fead00b4570 con 0x7fead00d0f20 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fead00b43f0 con 0x7fead00cb960 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.623+0000 7fead6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fead00b46f0 con 0x7fead006f8a0 2024-09-17T04:05:08.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47512/0 (socket says 172.21.15.129:47512) 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7feac7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead00cb960 0x7fead00ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 -- 172.21.15.129:0/1504396577 learned_addr learned my addr 172.21.15.129:0/1504396577 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5000640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead006f8a0 0x7fead01f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 -- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead006f8a0 msgr2=0x7fead01f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead006f8a0 0x7fead01f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 -- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead00cb960 msgr2=0x7fead00ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead00cb960 0x7fead00ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.624+0000 7fead5a00640 1 -- 172.21.15.129:0/1504396577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead009ae70 con 0x7fead00d0f20 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7fead5a00640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7feac800be90 tx=0x7feac800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8019070 con 0x7fead00d0f20 2024-09-17T04:05:08.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feac8014440 con 0x7fead00d0f20 2024-09-17T04:05:08.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead009ab90 con 0x7fead00d0f20 2024-09-17T04:05:08.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8014890 con 0x7fead00d0f20 2024-09-17T04:05:08.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.625+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead009c830 con 0x7fead00d0f20 2024-09-17T04:05:08.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.626+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fead009a900 con 0x7fead00d0f20 2024-09-17T04:05:08.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.627+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feac801c070 con 0x7fead00d0f20 2024-09-17T04:05:08.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.628+0000 7feacd600640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 0x7feaac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:08.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.628+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7feac809ad60 con 0x7fead00d0f20 2024-09-17T04:05:08.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.629+0000 7fead5000640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 0x7feaac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:08.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.630+0000 7fead5000640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 0x7feaac07b0a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7feac0033000 tx=0x7feac0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:08.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.633+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feac8062a60 con 0x7fead00d0f20 2024-09-17T04:05:08.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.816+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fead00b4c50 con 0x7fead00d0f20 2024-09-17T04:05:08.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.818+0000 7feacd600640 1 -- 172.21.15.129:0/1504396577 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) ==== 74+0+16907 (secure 0 0 0) 0x7feac8067490 con 0x7fead00d0f20 2024-09-17T04:05:08.817 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:08.817 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":208,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:04:20.614835+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:03:37.702365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-17T04:03:38.771949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"31.7","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/3759513659":"2024-09-18T04:04:04.378256+0000","172.21.15.129:6834/3759513659":"2024-09-18T04:04:04.378256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 msgr2=0x7feaac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 0x7feaac07b0a0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7feac0033000 tx=0x7feac0004730 comp rx=0 tx=0).stop 2024-09-17T04:05:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 msgr2=0x7fead00adb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:08.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7feac800be90 tx=0x7feac800bf80 comp rx=0 tx=0).stop 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 shutdown_connections 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7feaac078b90 0x7feaac07b0a0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead00d0f20 0x7fead00adb10 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead00cb960 0x7fead00ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 --2- 172.21.15.129:0/1504396577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fead006f8a0 0x7fead01f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 >> 172.21.15.129:0/1504396577 conn(0x7fead00b2510 msgr2=0x7fead00b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 shutdown_connections 2024-09-17T04:05:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:08.820+0000 7fead6400640 1 -- 172.21.15.129:0/1504396577 wait complete. 2024-09-17T04:05:08.841 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:05:08.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:09.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 -- 172.21.15.129:0/1548236386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 msgr2=0x7f82280d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 --2- 172.21.15.129:0/1548236386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280d0a30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8220009830 tx=0x7f822002a510 comp rx=0 tx=0).stop 2024-09-17T04:05:09.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 -- 172.21.15.129:0/1548236386 shutdown_connections 2024-09-17T04:05:09.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 --2- 172.21.15.129:0/1548236386 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280d0a30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 --2- 172.21.15.129:0/1548236386 >> v2:172.21.15.181:3300/0 conn(0x7f82280cfd10 0x7f82280cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 -- 172.21.15.129:0/1548236386 >> 172.21.15.129:0/1548236386 conn(0x7f822806cfc0 msgr2=0x7f822806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:09.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 -- 172.21.15.129:0/1548236386 shutdown_connections 2024-09-17T04:05:09.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.064+0000 7f822ce00640 1 -- 172.21.15.129:0/1548236386 wait complete. 2024-09-17T04:05:09.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.066+0000 7f822ce00640 1 Processor -- start 2024-09-17T04:05:09.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.066+0000 7f822ce00640 1 -- start start 2024-09-17T04:05:09.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.067+0000 7f822ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82280cfd10 0x7f82281547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.067+0000 7f822ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f822ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f822ce00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8228073890 con 0x7f822815cd30 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f822ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8228073710 con 0x7f82280cfd10 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f822ce00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8228073a10 con 0x7f82280d0620 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47514/0 (socket says 172.21.15.129:47514) 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 -- 172.21.15.129:0/4173686017 learned_addr learned my addr 172.21.15.129:0/4173686017 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f821d400640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8225e00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82280cfd10 0x7f82281547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 -- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 msgr2=0x7f82280aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280aab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 -- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82280cfd10 msgr2=0x7f82281547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82280cfd10 0x7f82281547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.068+0000 7f8226800640 1 -- 172.21.15.129:0/4173686017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82280ab6c0 con 0x7f822815cd30 2024-09-17T04:05:09.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f8226800640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f8220009920 tx=0x7f8220031720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:09.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220005020 con 0x7f822815cd30 2024-09-17T04:05:09.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82280ab9a0 con 0x7f822815cd30 2024-09-17T04:05:09.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f82200052b0 con 0x7f822815cd30 2024-09-17T04:05:09.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82200047a0 con 0x7f822815cd30 2024-09-17T04:05:09.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.069+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82280abf60 con 0x7f822815cd30 2024-09-17T04:05:09.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.070+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82280ab430 con 0x7f822815cd30 2024-09-17T04:05:09.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.071+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8220004940 con 0x7f822815cd30 2024-09-17T04:05:09.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.072+0000 7f821ea00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 0x7f820407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.072+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f82200bbf60 con 0x7f822815cd30 2024-09-17T04:05:09.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.073+0000 7f8225e00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 0x7f820407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.073+0000 7f8225e00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 0x7f820407b0a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f82180017a0 tx=0x7f8218001730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:09.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.077+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f822003f030 con 0x7f822815cd30 2024-09-17T04:05:09.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.294+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8228073f30 con 0x7f822815cd30 2024-09-17T04:05:09.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.295+0000 7f821ea00640 1 -- 172.21.15.129:0/4173686017 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3597 (secure 0 0 0) 0x7f8220083cf0 con 0x7f822815cd30 2024-09-17T04:05:09.294 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:09.294 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":199,"btime":"2024-09-17T04:05:06:087044+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1344703948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1344703948},{"type":"v1","addr":"172.21.15.129:6833","nonce":1344703948}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:05:04.622165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:09.295 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 199 2024-09-17T04:05:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 msgr2=0x7f820407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 0x7f820407b0a0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f82180017a0 tx=0x7f8218001730 comp rx=0 tx=0).stop 2024-09-17T04:05:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 msgr2=0x7f82281f6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f8220009920 tx=0x7f8220031720 comp rx=0 tx=0).stop 2024-09-17T04:05:09.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 shutdown_connections 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8204078b90 0x7f820407b0a0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f822815cd30 0x7f82281f6960 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82280d0620 0x7f82280aab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 --2- 172.21.15.129:0/4173686017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82280cfd10 0x7f82281547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 >> 172.21.15.129:0/4173686017 conn(0x7f822806cfc0 msgr2=0x7f8228152430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:09.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 shutdown_connections 2024-09-17T04:05:09.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.297+0000 7f822ce00640 1 -- 172.21.15.129:0/4173686017 wait complete. 2024-09-17T04:05:09.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 -- 172.21.15.129:0/3637114338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9d0004800 msgr2=0x7fe9d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 --2- 172.21.15.129:0/3637114338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9d0004800 0x7fe9d0004c10 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4009b50 tx=0x7fe9d0005830 comp rx=0 tx=0).stop 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 -- 172.21.15.129:0/3637114338 shutdown_connections 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 --2- 172.21.15.129:0/3637114338 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9d0004800 0x7fe9d0004c10 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 -- 172.21.15.129:0/3637114338 >> 172.21.15.129:0/3637114338 conn(0x7fe9ec06cfc0 msgr2=0x7fe9ec06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 -- 172.21.15.129:0/3637114338 shutdown_connections 2024-09-17T04:05:09.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.539+0000 7fe9f2000640 1 -- 172.21.15.129:0/3637114338 wait complete. 2024-09-17T04:05:09.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.542+0000 7fe9f2000640 1 Processor -- start 2024-09-17T04:05:09.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.542+0000 7fe9f2000640 1 -- start start 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 0x7fe9ec0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ec1f5620 0x7fe9ec0af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec0b3d80 con 0x7fe9ec151d40 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec0b3c00 con 0x7fe9d0004800 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.543+0000 7fe9f2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ec155a70 con 0x7fe9ec1f5620 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9eb400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 0x7fe9ec0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47528/0 (socket says 172.21.15.129:47528) 2024-09-17T04:05:09.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 -- 172.21.15.129:0/4204448761 learned_addr learned my addr 172.21.15.129:0/4204448761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9e2a00640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ec1f5620 0x7fe9ec0af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 -- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ec1f5620 msgr2=0x7fe9ec0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ec1f5620 0x7fe9ec0af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 -- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 msgr2=0x7fe9ec0ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 0x7fe9ec0ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 -- 172.21.15.129:0/4204448761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec1f5e00 con 0x7fe9ec151d40 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9eb400640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 0x7fe9ec0ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.544+0000 7fe9ebe00640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4002230 tx=0x7fe9e400eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.545+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4015070 con 0x7fe9ec151d40 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.545+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9e400cb00 con 0x7fe9ec151d40 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.545+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9e4011460 con 0x7fe9ec151d40 2024-09-17T04:05:09.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.545+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec1f60e0 con 0x7fe9ec151d40 2024-09-17T04:05:09.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.545+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec1529a0 con 0x7fe9ec151d40 2024-09-17T04:05:09.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.546+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ec1f5b70 con 0x7fe9ec151d40 2024-09-17T04:05:09.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.547+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe9e4018070 con 0x7fe9ec151d40 2024-09-17T04:05:09.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.548+0000 7fe9e3e00640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 0x7fe9c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:09.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.548+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7fe9e409b190 con 0x7fe9ec151d40 2024-09-17T04:05:09.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.548+0000 7fe9eb400640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 0x7fe9c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:09.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.549+0000 7fe9eb400640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 0x7fe9c807b0a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc006d50 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:09.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.552+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe9e4062f30 con 0x7fe9ec151d40 2024-09-17T04:05:09.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.768+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9ec0b4690 con 0x7fe9ec151d40 2024-09-17T04:05:09.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.768+0000 7fe9e3e00640 1 -- 172.21.15.129:0/4204448761 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+3597 (secure 0 0 0) 0x7fe9e4067960 con 0x7fe9ec151d40 2024-09-17T04:05:09.768 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:09.768 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":199,"btime":"2024-09-17T04:05:06:087044+0000","default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1344703948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1344703948},{"type":"v1","addr":"172.21.15.129:6833","nonce":1344703948}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":199}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:03:39.805108+0000","modified":"2024-09-17T04:05:04.622165+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":206,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":10440},"failed":[],"damaged":[],"stopped":[1],"info":{"gid_10440":{"gid":10440,"name":"d","rank":0,"incarnation":177,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6837/2583867786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2583867786},{"type":"v1","addr":"172.21.15.181:6837","nonce":2583867786}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":10440,"qdb_cluster":[10440]},"id":15}]} 2024-09-17T04:05:09.768 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 199 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 msgr2=0x7fe9c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 0x7fe9c807b0a0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc000c00 tx=0x7fe9dc006d50 comp rx=0 tx=0).stop 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 msgr2=0x7fe9ec152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4002230 tx=0x7fe9e400eb60 comp rx=0 tx=0).stop 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 shutdown_connections 2024-09-17T04:05:09.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe9c8078b90 0x7fe9c807b0a0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9ec151d40 0x7fe9ec152ec0 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9ec1f5620 0x7fe9ec0af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 --2- 172.21.15.129:0/4204448761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9d0004800 0x7fe9ec0ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 >> 172.21.15.129:0/4204448761 conn(0x7fe9ec06cfc0 msgr2=0x7fe9ec0cf240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 shutdown_connections 2024-09-17T04:05:09.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:09.771+0000 7fe9f2000640 1 -- 172.21.15.129:0/4204448761 wait complete. 2024-09-17T04:05:09.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 -- 172.21.15.129:0/3913691966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33809b130 msgr2=0x7ff3380cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 --2- 172.21.15.129:0/3913691966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33809b130 0x7ff3380cd6b0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff330005fd0 tx=0x7ff33002a7c0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 -- 172.21.15.129:0/3913691966 shutdown_connections 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 --2- 172.21.15.129:0/3913691966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33809b130 0x7ff3380cd6b0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 -- 172.21.15.129:0/3913691966 >> 172.21.15.129:0/3913691966 conn(0x7ff3380b3e40 msgr2=0x7ff3380b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 -- 172.21.15.129:0/3913691966 shutdown_connections 2024-09-17T04:05:10.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.012+0000 7ff337200640 1 -- 172.21.15.129:0/3913691966 wait complete. 2024-09-17T04:05:10.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.015+0000 7ff337200640 1 Processor -- start 2024-09-17T04:05:10.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.015+0000 7ff337200640 1 -- start start 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33809b130 0x7ff33815e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33815fe20 0x7ff338150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff3380cdbf0 con 0x7ff33809ba60 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff3380cec00 con 0x7ff33815fe20 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff337200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff3380cdd70 con 0x7ff33809b130 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff335400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff335400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47540/0 (socket says 172.21.15.129:47540) 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff335400640 1 -- 172.21.15.129:0/1925649553 learned_addr learned my addr 172.21.15.129:0/1925649553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.016+0000 7ff335e00640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33809b130 0x7ff33815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff336800640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33815fe20 0x7ff338150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 -- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33809b130 msgr2=0x7ff33815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33809b130 0x7ff33815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 -- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33815fe20 msgr2=0x7ff338150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33815fe20 0x7ff338150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 -- 172.21.15.129:0/1925649553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3380b3b10 con 0x7ff33809ba60 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff335400640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7ff324001220 tx=0x7ff32400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324016020 con 0x7ff33809ba60 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff324012920 con 0x7ff33809ba60 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324012db0 con 0x7ff33809ba60 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.017+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3381f6e90 con 0x7ff33809ba60 2024-09-17T04:05:10.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.018+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3381f7450 con 0x7ff33809ba60 2024-09-17T04:05:10.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.019+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3380ce160 con 0x7ff33809ba60 2024-09-17T04:05:10.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.020+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff324019020 con 0x7ff33809ba60 2024-09-17T04:05:10.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.021+0000 7ff32e000640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 0x7ff31807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.021+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7ff32409aa00 con 0x7ff33809ba60 2024-09-17T04:05:10.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.022+0000 7ff335e00640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 0x7ff31807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.022+0000 7ff335e00640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 0x7ff31807b0a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff328000fb0 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.026+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff324020030 con 0x7ff33809ba60 2024-09-17T04:05:10.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.213+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3380b38a0 con 0x7ff33809ba60 2024-09-17T04:05:10.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.214+0000 7ff32e000640 1 -- 172.21.15.129:0/1925649553 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v208) ==== 74+0+16907 (secure 0 0 0) 0x7ff324062820 con 0x7ff33809ba60 2024-09-17T04:05:10.214 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:10.214 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":208,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:04:20.614835+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:03:37.702365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"202","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-17T04:03:38.771949+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"203","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"31.7","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/3759513659":"2024-09-18T04:04:04.378256+0000","172.21.15.129:6834/3759513659":"2024-09-18T04:04:04.378256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:10.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 msgr2=0x7ff31807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 0x7ff31807b0a0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff328000fb0 tx=0x7ff328009040 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 msgr2=0x7ff338150560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7ff324001220 tx=0x7ff32400c940 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 shutdown_connections 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff318078b90 0x7ff31807b0a0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33815fe20 0x7ff338150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33809ba60 0x7ff338150560 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 --2- 172.21.15.129:0/1925649553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33809b130 0x7ff33815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 >> 172.21.15.129:0/1925649553 conn(0x7ff3380b3e40 msgr2=0x7ff338071500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 shutdown_connections 2024-09-17T04:05:10.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.217+0000 7ff337200640 1 -- 172.21.15.129:0/1925649553 wait complete. 2024-09-17T04:05:10.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.464+0000 7f75e3e00640 1 -- 172.21.15.129:0/4077618064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75c4004d40 msgr2=0x7f75c4005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.464+0000 7f75e3e00640 1 --2- 172.21.15.129:0/4077618064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75c4004d40 0x7f75c4005150 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7f75d4005fd0 tx=0x7f75c4005830 comp rx=0 tx=0).stop 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.464+0000 7f75e1e00640 1 -- 172.21.15.129:0/4077618064 reap_dead start 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.464+0000 7f75e3e00640 1 -- 172.21.15.129:0/4077618064 shutdown_connections 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.464+0000 7f75e3e00640 1 -- 172.21.15.129:0/4077618064 >> 172.21.15.129:0/4077618064 conn(0x7f75dc06cfc0 msgr2=0x7f75dc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.465+0000 7f75e3e00640 1 -- 172.21.15.129:0/4077618064 shutdown_connections 2024-09-17T04:05:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.465+0000 7f75e3e00640 1 -- 172.21.15.129:0/4077618064 wait complete. 2024-09-17T04:05:10.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.467+0000 7f75e3e00640 1 Processor -- start 2024-09-17T04:05:10.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.467+0000 7f75e3e00640 1 -- start start 2024-09-17T04:05:10.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75c4004d40 0x7f75dc0c8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75dc0ca080 0x7f75dc133590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc0d0840 con 0x7f75dc133f70 2024-09-17T04:05:10.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc0d06c0 con 0x7f75dc0ca080 2024-09-17T04:05:10.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.468+0000 7f75e3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f75dc0d09c0 con 0x7f75c4004d40 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e0a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75dc0ca080 0x7f75dc133590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47560/0 (socket says 172.21.15.129:47560) 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 -- 172.21.15.129:0/3290931118 learned_addr learned my addr 172.21.15.129:0/3290931118 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75c4004d40 0x7f75dc0c8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 -- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75c4004d40 msgr2=0x7f75dc0c8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75c4004d40 0x7f75dc0c8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 -- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75dc0ca080 msgr2=0x7f75dc133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75dc0ca080 0x7f75dc133590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 -- 172.21.15.129:0/3290931118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc0a9760 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.469+0000 7f75e1e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f75d402b860 tx=0x7f75d4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.470+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4042070 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.470+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75d4003970 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.470+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc0a9a40 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.470+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d4038040 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.470+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc0c82e0 con 0x7f75dc133f70 2024-09-17T04:05:10.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.471+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc0a94d0 con 0x7f75dc133f70 2024-09-17T04:05:10.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.472+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f75d4033d10 con 0x7f75dc133f70 2024-09-17T04:05:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.473+0000 7f75d9600640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 0x7f75bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.473+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(208..208 src has 1..208) ==== 5824+0+0 (secure 0 0 0) 0x7f75d40baeb0 con 0x7f75dc133f70 2024-09-17T04:05:10.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.473+0000 7f75e1400640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 0x7f75bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.474+0000 7f75e1400640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 0x7f75bc07b0a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0cec40 tx=0x7f75d00067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.477+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75d4082bb0 con 0x7f75dc133f70 2024-09-17T04:05:10.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.696+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f75dc074ae0 con 0x7f75dc133f70 2024-09-17T04:05:10.695 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:05:10.697+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:05:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.699+0000 7f75d9600640 1 -- 172.21.15.129:0/3290931118 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v200) ==== 161+0+0 (secure 0 0 0) 0x7f75dc074ae0 con 0x7f75dc133f70 2024-09-17T04:05:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 msgr2=0x7f75bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 0x7f75bc07b0a0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f75dc0cec40 tx=0x7f75d00067a0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 msgr2=0x7f75dc134380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f75d402b860 tx=0x7f75d4033ba0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 shutdown_connections 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f75bc078b90 0x7f75bc07b0a0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75dc133f70 0x7f75dc134380 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75dc0ca080 0x7f75dc133590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 --2- 172.21.15.129:0/3290931118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75c4004d40 0x7f75dc0c8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 >> 172.21.15.129:0/3290931118 conn(0x7f75dc06cfc0 msgr2=0x7f75dc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:10.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 shutdown_connections 2024-09-17T04:05:10.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.703+0000 7f75e3e00640 1 -- 172.21.15.129:0/3290931118 wait complete. 2024-09-17T04:05:10.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:05:10.840 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:05:10.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 -- 172.21.15.129:0/2902441762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980160940 msgr2=0x7fe9800cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 --2- 172.21.15.129:0/2902441762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980160940 0x7fe9800cc430 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7fe968009830 tx=0x7fe96802a420 comp rx=0 tx=0).stop 2024-09-17T04:05:10.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 -- 172.21.15.129:0/2902441762 shutdown_connections 2024-09-17T04:05:10.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 --2- 172.21.15.129:0/2902441762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980160940 0x7fe9800cc430 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 -- 172.21.15.129:0/2902441762 >> 172.21.15.129:0/2902441762 conn(0x7fe9800b25f0 msgr2=0x7fe9800d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:10.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 -- 172.21.15.129:0/2902441762 shutdown_connections 2024-09-17T04:05:10.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.954+0000 7fe986600640 1 -- 172.21.15.129:0/2902441762 wait complete. 2024-09-17T04:05:10.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.956+0000 7fe986600640 1 Processor -- start 2024-09-17T04:05:10.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.956+0000 7fe986600640 1 -- start start 2024-09-17T04:05:10.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 0x7fe98014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 0x7fe9801f9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800ae100 con 0x7fe980156d60 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9800adf80 con 0x7fe9800af690 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe986600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9800ae280 con 0x7fe980155350 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe977e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe977e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47578/0 (socket says 172.21.15.129:47578) 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe977e00640 1 -- 172.21.15.129:0/466260160 learned_addr learned my addr 172.21.15.129:0/466260160 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe985200640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 0x7fe98014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.958+0000 7fe985c00640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 0x7fe9801f9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 -- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 msgr2=0x7fe9801f9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 0x7fe9801f9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 -- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 msgr2=0x7fe98014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:10.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 0x7fe98014e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 -- 172.21.15.129:0/466260160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe980073ca0 con 0x7fe980156d60 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe985200640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 0x7fe98014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe985c00640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 0x7fe9801f9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe977e00640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fe968009780 tx=0x7fe968033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96803f020 con 0x7fe980156d60 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe968033e10 con 0x7fe980156d60 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968037040 con 0x7fe980156d60 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9800736e0 con 0x7fe980156d60 2024-09-17T04:05:10.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.959+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9800c71e0 con 0x7fe980156d60 2024-09-17T04:05:10.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.960+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9801f99b0 con 0x7fe980156d60 2024-09-17T04:05:10.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.962+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe968042020 con 0x7fe980156d60 2024-09-17T04:05:10.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.963+0000 7fe97d600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 0x7fe95c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.963+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 5910+0+0 (secure 0 0 0) 0x7fe9680bafe0 con 0x7fe980156d60 2024-09-17T04:05:10.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.963+0000 7fe985200640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 0x7fe95c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:10.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.964+0000 7fe985200640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 0x7fe95c07b0d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe970007fd0 tx=0x7fe970009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:10.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:10.967+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe968049030 con 0x7fe980156d60 2024-09-17T04:05:11.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.188+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe9800ae760 con 0x7fe980156d60 2024-09-17T04:05:11.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.701+0000 7fe97d600640 1 -- 172.21.15.129:0/466260160 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v201) ==== 104+0+0 (secure 0 0 0) 0x7fe980073ca0 con 0x7fe980156d60 2024-09-17T04:05:11.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.703+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 msgr2=0x7fe95c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:11.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.703+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 0x7fe95c07b0d0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe970007fd0 tx=0x7fe970009040 comp rx=0 tx=0).stop 2024-09-17T04:05:11.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.703+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 msgr2=0x7fe9801f8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:11.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.703+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7fe968009780 tx=0x7fe968033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:11.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 shutdown_connections 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe95c078bc0 0x7fe95c07b0d0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe980155350 0x7fe9801f9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe980156d60 0x7fe9801f8b30 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 --2- 172.21.15.129:0/466260160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9800af690 0x7fe98014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 >> 172.21.15.129:0/466260160 conn(0x7fe9800b25f0 msgr2=0x7fe980156400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 shutdown_connections 2024-09-17T04:05:11.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.704+0000 7fe986600640 1 -- 172.21.15.129:0/466260160 wait complete. 2024-09-17T04:05:11.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:05:11.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 -- 172.21.15.129:0/3570193459 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c153080 msgr2=0x7fd06c153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 --2- 172.21.15.129:0/3570193459 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c153080 0x7fd06c153490 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7fd06c06dd90 tx=0x7fd05402a4a0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 -- 172.21.15.129:0/3570193459 shutdown_connections 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 --2- 172.21.15.129:0/3570193459 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c153080 0x7fd06c153490 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 --2- 172.21.15.129:0/3570193459 >> v2:172.21.15.181:3300/0 conn(0x7fd06c0cfc80 0x7fd06c0af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 -- 172.21.15.129:0/3570193459 >> 172.21.15.129:0/3570193459 conn(0x7fd06c06d7a0 msgr2=0x7fd06c0b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 -- 172.21.15.129:0/3570193459 shutdown_connections 2024-09-17T04:05:11.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.956+0000 7fd071c00640 1 -- 172.21.15.129:0/3570193459 wait complete. 2024-09-17T04:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.958+0000 7fd071c00640 1 Processor -- start 2024-09-17T04:05:11.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.958+0000 7fd071c00640 1 -- start start 2024-09-17T04:05:11.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd06c0cfc80 0x7fd06c1f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd06c153080 0x7fd06c1f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c160b60 con 0x7fd06c1539d0 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c070b40 con 0x7fd06c153080 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd071c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c160ce0 con 0x7fd06c0cfc80 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd06b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd06b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47604/0 (socket says 172.21.15.129:47604) 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd06b400640 1 -- 172.21.15.129:0/2952971505 learned_addr learned my addr 172.21.15.129:0/2952971505 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd062000640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd06c153080 0x7fd06c1f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:11.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.960+0000 7fd06aa00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd06c0cfc80 0x7fd06c1f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 -- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd06c0cfc80 msgr2=0x7fd06c1f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd06c0cfc80 0x7fd06c1f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 -- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd06c153080 msgr2=0x7fd06c1f98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd06c153080 0x7fd06c1f98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 -- 172.21.15.129:0/2952971505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c1f5820 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd06b400640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fd06400cb10 tx=0x7fd06400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064016020 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd06400ad80 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd064012680 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c1f5ac0 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.961+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c1f6080 con 0x7fd06c1539d0 2024-09-17T04:05:11.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.962+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06c1611a0 con 0x7fd06c1539d0 2024-09-17T04:05:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.964+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd064019020 con 0x7fd06c1539d0 2024-09-17T04:05:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.964+0000 7fd063400640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 0x7fd04807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.965+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(209..209 src has 1..209) ==== 5910+0+0 (secure 0 0 0) 0x7fd06409b340 con 0x7fd06c1539d0 2024-09-17T04:05:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.965+0000 7fd06aa00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 0x7fd04807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:11.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.966+0000 7fd06aa00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 0x7fd04807b0a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009b30 tx=0x7fd05c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:11.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:11.969+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd064020020 con 0x7fd06c1539d0 2024-09-17T04:05:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:12.154+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd06c0b4d60 con 0x7fd06c1539d0 2024-09-17T04:05:13.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.139+0000 7fd063400640 1 -- 172.21.15.129:0/2952971505 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v210) ==== 181+0+0 (secure 0 0 0) 0x7fd06c1f5820 con 0x7fd06c1539d0 2024-09-17T04:05:13.138 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:05:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 msgr2=0x7fd04807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 0x7fd04807b0a0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd05c009b30 tx=0x7fd05c005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 msgr2=0x7fd06c0b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7fd06400cb10 tx=0x7fd06400a060 comp rx=0 tx=0).stop 2024-09-17T04:05:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 shutdown_connections 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd048078b90 0x7fd04807b0a0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd06c1539d0 0x7fd06c0b19d0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd06c153080 0x7fd06c1f98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 --2- 172.21.15.129:0/2952971505 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd06c0cfc80 0x7fd06c1f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 >> 172.21.15.129:0/2952971505 conn(0x7fd06c06d7a0 msgr2=0x7fd06c1527e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 shutdown_connections 2024-09-17T04:05:13.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.142+0000 7fd071c00640 1 -- 172.21.15.129:0/2952971505 wait complete. 2024-09-17T04:05:13.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:05:13.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 -- 172.21.15.129:0/3768489561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 msgr2=0x7ff414073520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:13.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 --2- 172.21.15.129:0/3768489561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff414073520 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff41409e4b0 tx=0x7ff40402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 -- 172.21.15.129:0/3768489561 shutdown_connections 2024-09-17T04:05:13.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 --2- 172.21.15.129:0/3768489561 >> v2:172.21.15.181:3300/0 conn(0x7ff4140b4060 0x7ff41406f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 --2- 172.21.15.129:0/3768489561 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff414073520 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 -- 172.21.15.129:0/3768489561 >> 172.21.15.129:0/3768489561 conn(0x7ff41406d7a0 msgr2=0x7ff4140b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:13.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 -- 172.21.15.129:0/3768489561 shutdown_connections 2024-09-17T04:05:13.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.397+0000 7ff419a00640 1 -- 172.21.15.129:0/3768489561 wait complete. 2024-09-17T04:05:13.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.399+0000 7ff419a00640 1 Processor -- start 2024-09-17T04:05:13.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.399+0000 7ff419a00640 1 -- start start 2024-09-17T04:05:13.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.400+0000 7ff419a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff414073a60 0x7ff414148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:13.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.400+0000 7ff419a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 0x7ff4140c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:13.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.400+0000 7ff419a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff419a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff414070390 con 0x7ff414155c20 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff419a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff414070210 con 0x7ff414073a60 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff419a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff414070510 con 0x7ff4140b4060 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47618/0 (socket says 172.21.15.129:47618) 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 -- 172.21.15.129:0/3801322548 learned_addr learned my addr 172.21.15.129:0/3801322548 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff412a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff414073a60 0x7ff414148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff412000640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 0x7ff4140c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 -- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 msgr2=0x7ff4140c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 0x7ff4140c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 -- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff414073a60 msgr2=0x7ff414148930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff414073a60 0x7ff414148930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:13.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff413400640 1 -- 172.21.15.129:0/3801322548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4140c9ab0 con 0x7ff414155c20 2024-09-17T04:05:13.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.401+0000 7ff412000640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 0x7ff4140c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:13.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff413400640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00d620 tx=0x7ff40c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40c017070 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff40c005310 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4140c9f30 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff40c0047a0 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.402+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4140ca4f0 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.403+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4140745d0 con 0x7ff414155c20 2024-09-17T04:05:13.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.404+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff40c004940 con 0x7ff414155c20 2024-09-17T04:05:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.406+0000 7ff40aa00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 0x7ff3f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.406+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 5472+0+0 (secure 0 0 0) 0x7ff40c09abf0 con 0x7ff414155c20 2024-09-17T04:05:13.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.406+0000 7ff412a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 0x7ff3f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:13.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.407+0000 7ff412a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 0x7ff3f407b0a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff404034000 tx=0x7ff404004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:13.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.410+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff4140745d0 con 0x7ff414155c20 2024-09-17T04:05:13.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:13.591+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff414070aa0 con 0x7ff414155c20 2024-09-17T04:05:14.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.145+0000 7ff40aa00640 1 -- 172.21.15.129:0/3801322548 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v211) ==== 169+0+0 (secure 0 0 0) 0x7ff40c062a50 con 0x7ff414155c20 2024-09-17T04:05:14.145 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:05:14.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.149+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 msgr2=0x7ff3f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.149+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 0x7ff3f407b0a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff404034000 tx=0x7ff404004ab0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.149+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 msgr2=0x7ff4140cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.149+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00d620 tx=0x7ff40c00dae0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 shutdown_connections 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff3f4078b90 0x7ff3f407b0a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff414155c20 0x7ff4140cb760 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4140b4060 0x7ff4140c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 --2- 172.21.15.129:0/3801322548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff414073a60 0x7ff414148930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 >> 172.21.15.129:0/3801322548 conn(0x7ff41406d7a0 msgr2=0x7ff4140afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 shutdown_connections 2024-09-17T04:05:14.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.150+0000 7ff419a00640 1 -- 172.21.15.129:0/3801322548 wait complete. 2024-09-17T04:05:14.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:05:14.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:05:14.276 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.277 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:05:14.278 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:05:14.279 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:05:14.280 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.281 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.282 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:05:14.283 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:05:14.284 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:05:14.285 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:05:14.286 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:05:14.287 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:05:14.288 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.289 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:05:14.290 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.291 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.292 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:05:14.293 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:05:14.294 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:05:14.295 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:05:14.296 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:05:14.297 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:05:14.298 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:05:14.299 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:05:14.300 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:05:14.300 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:05:14.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 -- 172.21.15.129:0/3875730978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0b2950 msgr2=0x7f2eac0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/3875730978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0b2950 0x7f2eac0b3480 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009830 tx=0x7f2e9402a510 comp rx=0 tx=0).stop 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 -- 172.21.15.129:0/3875730978 shutdown_connections 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/3875730978 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0b2950 0x7f2eac0b3480 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/3875730978 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0ce170 0x7f2eac0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 -- 172.21.15.129:0/3875730978 >> 172.21.15.129:0/3875730978 conn(0x7f2eac06d410 msgr2=0x7f2eac0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 -- 172.21.15.129:0/3875730978 shutdown_connections 2024-09-17T04:05:14.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.532+0000 7f2eb0e00640 1 -- 172.21.15.129:0/3875730978 wait complete. 2024-09-17T04:05:14.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.535+0000 7f2eb0e00640 1 Processor -- start 2024-09-17T04:05:14.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.535+0000 7f2eb0e00640 1 -- start start 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0b2950 0x7f2eac09ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0ce170 0x7f2eac15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac06f5e0 con 0x7f2eac0b39c0 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac0708e0 con 0x7f2eac0ce170 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2eb0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac06f760 con 0x7f2eac0b2950 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.536+0000 7f2ea1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41624/0 (socket says 172.21.15.129:41624) 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 -- 172.21.15.129:0/2803963761 learned_addr learned my addr 172.21.15.129:0/2803963761 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea9e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0b2950 0x7f2eac09ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2eaa800640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0ce170 0x7f2eac15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.537 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 -- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0b2950 msgr2=0x7f2eac09ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0b2950 0x7f2eac09ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 -- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0ce170 msgr2=0x7f2eac15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0ce170 0x7f2eac15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 -- 172.21.15.129:0/2803963761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac150fd0 con 0x7f2eac0b39c0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.537+0000 7f2ea1400640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e94033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.538+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94005c90 con 0x7f2eac0b39c0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.538+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2e94031e50 con 0x7f2eac0b39c0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.538+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac09c140 con 0x7f2eac0b39c0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.538+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e940332f0 con 0x7f2eac0b39c0 2024-09-17T04:05:14.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.538+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac09c700 con 0x7f2eac0b39c0 2024-09-17T04:05:14.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.539+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac09beb0 con 0x7f2eac0b39c0 2024-09-17T04:05:14.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.540+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2e94033490 con 0x7f2eac0b39c0 2024-09-17T04:05:14.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.541+0000 7f2ea2a00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.541+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7f2e940bbcb0 con 0x7f2eac0b39c0 2024-09-17T04:05:14.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.542+0000 7f2ea9e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.542+0000 7f2ea9e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009b30 tx=0x7f2e9c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:14.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.546+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2e9403f030 con 0x7f2eac0b39c0 2024-09-17T04:05:14.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.728+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f2eac074b70 con 0x7f2eac0b39c0 2024-09-17T04:05:14.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.733+0000 7f2ea2a00640 1 -- 172.21.15.129:0/2803963761 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v63) ==== 71+0+0 (secure 0 0 0) 0x7f2eac074b70 con 0x7f2eac0b39c0 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 msgr2=0x7f2e8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009b30 tx=0x7f2e9c005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 msgr2=0x7f2eac0c9d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f2e94000c00 tx=0x7f2e94033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 shutdown_connections 2024-09-17T04:05:14.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2e88078b90 0x7f2e8807b0a0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2eac0ce170 0x7f2eac15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2eac0b39c0 0x7f2eac0c9d40 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 --2- 172.21.15.129:0/2803963761 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2eac0b2950 0x7f2eac09ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 >> 172.21.15.129:0/2803963761 conn(0x7f2eac06d410 msgr2=0x7f2eac0996a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 shutdown_connections 2024-09-17T04:05:14.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.736+0000 7f2eb0e00640 1 -- 172.21.15.129:0/2803963761 wait complete. 2024-09-17T04:05:14.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown' 2024-09-17T04:05:14.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 -- 172.21.15.129:0/3722251977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 msgr2=0x7f376c15e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 --2- 172.21.15.129:0/3722251977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c15e880 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f375c009830 tx=0x7f375c02a540 comp rx=0 tx=0).stop 2024-09-17T04:05:14.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 -- 172.21.15.129:0/3722251977 shutdown_connections 2024-09-17T04:05:14.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 --2- 172.21.15.129:0/3722251977 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c15e880 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 -- 172.21.15.129:0/3722251977 >> 172.21.15.129:0/3722251977 conn(0x7f376c06cfc0 msgr2=0x7f376c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:14.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 -- 172.21.15.129:0/3722251977 shutdown_connections 2024-09-17T04:05:14.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.979+0000 7f376be00640 1 -- 172.21.15.129:0/3722251977 wait complete. 2024-09-17T04:05:14.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.981+0000 7f376be00640 1 Processor -- start 2024-09-17T04:05:14.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.981+0000 7f376be00640 1 -- start start 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f376c0cf910 0x7f376c1e9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f376c162f10 con 0x7f376c14dda0 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f376c162d90 con 0x7f376c0ad8d0 2024-09-17T04:05:14.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f376c163090 con 0x7f376c0cf910 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41630/0 (socket says 172.21.15.129:41630) 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f3762000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f376c0cf910 0x7f376c1e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:52680/0 (socket says 172.21.15.129:52680) 2024-09-17T04:05:14.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 -- 172.21.15.129:0/1885137682 learned_addr learned my addr 172.21.15.129:0/1885137682 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:14.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 -- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f376c0cf910 msgr2=0x7f376c1e9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f376c0cf910 0x7f376c1e9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 -- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 msgr2=0x7f376c14e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:14.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:14.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376b400640 1 -- 172.21.15.129:0/1885137682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c0b1a60 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.983+0000 7f376aa00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f376b400640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f376400c730 tx=0x7f376400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764003b70 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3764003d10 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376400aab0 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c14d580 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.984+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c1e8d30 con 0x7f376c14dda0 2024-09-17T04:05:14.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.985+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c15e880 con 0x7f376c14dda0 2024-09-17T04:05:14.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.987+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f376400ac50 con 0x7f376c14dda0 2024-09-17T04:05:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.988+0000 7f3763e00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 0x7f374807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.988+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7f376409a690 con 0x7f376c14dda0 2024-09-17T04:05:14.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.988+0000 7f376aa00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 0x7f374807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:14.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.989+0000 7f376aa00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 0x7f374807b0a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f375c034000 tx=0x7f375c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:14.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:14.993+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f37640627b0 con 0x7f376c14dda0 2024-09-17T04:05:15.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:15.250+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]} v 0) -- 0x7f376c06a140 con 0x7f376c14dda0 2024-09-17T04:05:16.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.153+0000 7f3763e00640 1 -- 172.21.15.129:0/1885137682 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_on_shutdown"]}]=0 v791) ==== 137+0+0 (secure 0 0 0) 0x7f37640671e0 con 0x7f376c14dda0 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 msgr2=0x7f374807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 0x7f374807b0a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f375c034000 tx=0x7f375c004730 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 msgr2=0x7f376c0b44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f376400c730 tx=0x7f376400cbf0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 shutdown_connections 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3748078b90 0x7f374807b0a0 secure :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f375c034000 tx=0x7f375c004730 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f376c14dda0 0x7f376c0b44c0 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f376c0cf910 0x7f376c1e9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 --2- 172.21.15.129:0/1885137682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f376c0ad8d0 0x7f376c14e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.156+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 >> 172.21.15.129:0/1885137682 conn(0x7f376c06cfc0 msgr2=0x7f376c154830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:16.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.157+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 shutdown_connections 2024-09-17T04:05:16.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.157+0000 7f376be00640 1 -- 172.21.15.129:0/1885137682 wait complete. 2024-09-17T04:05:16.178 INFO:tasks.cephfs_test_runner:That when an MDS rank is shut down, its purge queue is ... ok 2024-09-17T04:05:16.178 INFO:tasks.cephfs_test_runner:Starting test: test_purge_queue_upgrade (tasks.cephfs.test_strays.TestStrays) That when starting on a system with no purge queue in the metadata 2024-09-17T04:05:16.178 INFO:tasks.cephfs_test_runner:test_purge_queue_upgrade (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:05:16.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 -- 172.21.15.129:0/1602585781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 msgr2=0x7fd0500cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 --2- 172.21.15.129:0/1602585781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd0500cbbf0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fd040005fd0 tx=0x7fd04002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 -- 172.21.15.129:0/1602585781 shutdown_connections 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 --2- 172.21.15.129:0/1602585781 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0500b3b90 0x7fd05006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 --2- 172.21.15.129:0/1602585781 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd0500cbbf0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 -- 172.21.15.129:0/1602585781 >> 172.21.15.129:0/1602585781 conn(0x7fd05006d410 msgr2=0x7fd0500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 -- 172.21.15.129:0/1602585781 shutdown_connections 2024-09-17T04:05:16.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.402+0000 7fd057a00640 1 -- 172.21.15.129:0/1602585781 wait complete. 2024-09-17T04:05:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.404+0000 7fd057a00640 1 Processor -- start 2024-09-17T04:05:16.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.405+0000 7fd057a00640 1 -- start start 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0500b2950 0x7fd05009d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0501f4c60 0x7fd05015c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd05006f410 con 0x7fd0500ce170 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd050070910 con 0x7fd0500b2950 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd057a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd05006f590 con 0x7fd0501f4c60 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd047e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd047e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41648/0 (socket says 172.21.15.129:41648) 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd047e00640 1 -- 172.21.15.129:0/2722762457 learned_addr learned my addr 172.21.15.129:0/2722762457 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd055a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0501f4c60 0x7fd05015c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.406+0000 7fd055000640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0500b2950 0x7fd05009d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 -- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0501f4c60 msgr2=0x7fd05015c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0501f4c60 0x7fd05015c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 -- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0500b2950 msgr2=0x7fd05009d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0500b2950 0x7fd05009d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 -- 172.21.15.129:0/2722762457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05015dd60 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd047e00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c430 tx=0x7fd03800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038017070 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0380053c0 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038003850 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.407+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05015bf70 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.408+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05009ddf0 con 0x7fd0500ce170 2024-09-17T04:05:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.408+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd050076760 con 0x7fd0500ce170 2024-09-17T04:05:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.410+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd03801a070 con 0x7fd0500ce170 2024-09-17T04:05:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.411+0000 7fd04d600640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 0x7fd02c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.411+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7fd038099e70 con 0x7fd0500ce170 2024-09-17T04:05:16.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.411+0000 7fd055000640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 0x7fd02c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.412+0000 7fd055000640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 0x7fd02c07b0a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd040033000 tx=0x7fd040004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.415+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd050076760 con 0x7fd0500ce170 2024-09-17T04:05:16.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.599+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd05009d8a0 con 0x7fd0500ce170 2024-09-17T04:05:16.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.600+0000 7fd04d600640 1 -- 172.21.15.129:0/2722762457 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v211) ==== 74+0+13224 (secure 0 0 0) 0x7fd038061ef0 con 0x7fd0500ce170 2024-09-17T04:05:16.599 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:16.600 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":211,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:14.139399+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":207,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/2583867786":"2024-09-18T04:05:10.697107+0000","172.21.15.181:6835/2583867786":"2024-09-18T04:05:10.697107+0000","172.21.15.129:6835/3759513659":"2024-09-18T04:04:04.378256+0000","172.21.15.129:6834/3759513659":"2024-09-18T04:04:04.378256+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 msgr2=0x7fd02c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 0x7fd02c07b0a0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fd040033000 tx=0x7fd040004730 comp rx=0 tx=0).stop 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 msgr2=0x7fd05015d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7fd03800c430 tx=0x7fd03800c8f0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 shutdown_connections 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd02c078b90 0x7fd02c07b0a0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0501f4c60 0x7fd05015c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0500ce170 0x7fd05015d820 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 --2- 172.21.15.129:0/2722762457 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0500b2950 0x7fd05009d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 >> 172.21.15.129:0/2722762457 conn(0x7fd05006d410 msgr2=0x7fd0500741e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:16.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 shutdown_connections 2024-09-17T04:05:16.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.603+0000 7fd057a00640 1 -- 172.21.15.129:0/2722762457 wait complete. 2024-09-17T04:05:16.627 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:05:16.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:05:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 -- 172.21.15.129:0/3877258405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d240cba90 msgr2=0x7f3d240cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3877258405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d240cba90 0x7f3d240cbea0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c005fd0 tx=0x7f3d1c02aa10 comp rx=0 tx=0).stop 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 -- 172.21.15.129:0/3877258405 shutdown_connections 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3877258405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d240cba90 0x7f3d240cbea0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3877258405 >> v2:172.21.15.181:3300/0 conn(0x7f3d240b3f70 0x7f3d240b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 -- 172.21.15.129:0/3877258405 >> 172.21.15.129:0/3877258405 conn(0x7f3d240ccc40 msgr2=0x7f3d240cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 -- 172.21.15.129:0/3877258405 shutdown_connections 2024-09-17T04:05:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.851+0000 7f3d2a600640 1 -- 172.21.15.129:0/3877258405 wait complete. 2024-09-17T04:05:16.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.853+0000 7f3d2a600640 1 Processor -- start 2024-09-17T04:05:16.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.853+0000 7f3d2a600640 1 -- start start 2024-09-17T04:05:16.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d24074580 0x7f3d241571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d24157810 0x7f3d24166850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d240cc8e0 con 0x7f3d240cba90 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d240cc760 con 0x7f3d24074580 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d2a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d24166f20 con 0x7f3d24157810 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d1aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41658/0 (socket says 172.21.15.129:41658) 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d1aa00640 1 -- 172.21.15.129:0/3082912274 learned_addr learned my addr 172.21.15.129:0/3082912274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d23e00640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d24157810 0x7f3d24166850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.855+0000 7f3d23400640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d24074580 0x7f3d241571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 -- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d24157810 msgr2=0x7f3d24166850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d24157810 0x7f3d24166850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 -- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d24074580 msgr2=0x7f3d241571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d24074580 0x7f3d241571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 -- 172.21.15.129:0/3082912274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d240cdd90 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1aa00640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00c3d0 tx=0x7f3d0c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c017070 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d0c0053c0 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d241f5cd0 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c003850 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.856+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d241f6290 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.858+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d241f5a40 con 0x7f3d240cba90 2024-09-17T04:05:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.859+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d0c01a070 con 0x7f3d240cba90 2024-09-17T04:05:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.860+0000 7f3d1be00640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 0x7f3d0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.860+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7f3d0c099e90 con 0x7f3d240cba90 2024-09-17T04:05:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.860+0000 7f3d23400640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 0x7f3d0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:16.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.861+0000 7f3d23400640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 0x7f3d0007b0a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3d140095f0 tx=0x7f3d14005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:16.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:16.864+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d0c061e80 con 0x7f3d240cba90 2024-09-17T04:05:17.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.045+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d240b3f70 con 0x7f3d240cba90 2024-09-17T04:05:17.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.045+0000 7f3d1be00640 1 -- 172.21.15.129:0/3082912274 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v211) ==== 93+0+10 (secure 0 0 0) 0x7f3d0c09b0a0 con 0x7f3d240cba90 2024-09-17T04:05:17.044 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 msgr2=0x7f3d0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 0x7f3d0007b0a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3d140095f0 tx=0x7f3d14005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 msgr2=0x7f3d24166310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c00c3d0 tx=0x7f3d0c00c890 comp rx=0 tx=0).stop 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 shutdown_connections 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d00078b90 0x7f3d0007b0a0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d24157810 0x7f3d24166850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d240cba90 0x7f3d24166310 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 --2- 172.21.15.129:0/3082912274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d24074580 0x7f3d241571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.047+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 >> 172.21.15.129:0/3082912274 conn(0x7f3d240ccc40 msgr2=0x7f3d240cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.048+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 shutdown_connections 2024-09-17T04:05:17.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.048+0000 7f3d2a600640 1 -- 172.21.15.129:0/3082912274 wait complete. 2024-09-17T04:05:17.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade' 2024-09-17T04:05:17.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 -- 172.21.15.129:0/105408708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 msgr2=0x7fd3fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 --2- 172.21.15.129:0/105408708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd3fc00b9c0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fd410005fd0 tx=0x7fd3fc005830 comp rx=0 tx=0).stop 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd416800640 1 -- 172.21.15.129:0/105408708 reap_dead start 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 -- 172.21.15.129:0/105408708 shutdown_connections 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 -- 172.21.15.129:0/105408708 >> 172.21.15.129:0/105408708 conn(0x7fd41806cfc0 msgr2=0x7fd41806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 -- 172.21.15.129:0/105408708 shutdown_connections 2024-09-17T04:05:17.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.297+0000 7fd417200640 1 -- 172.21.15.129:0/105408708 wait complete. 2024-09-17T04:05:17.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.299+0000 7fd417200640 1 Processor -- start 2024-09-17T04:05:17.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.300+0000 7fd417200640 1 -- start start 2024-09-17T04:05:17.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:17.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4180c52b0 0x7fd4181f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:17.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4181f6d40 0x7fd41815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd41815cd50 con 0x7fd3fc005150 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd41815cbd0 con 0x7fd4181f6d40 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd417200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd41815ced0 con 0x7fd4180c52b0 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd415e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd415e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41662/0 (socket says 172.21.15.129:41662) 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd415e00640 1 -- 172.21.15.129:0/412376274 learned_addr learned my addr 172.21.15.129:0/412376274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd40d400640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4180c52b0 0x7fd4181f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.301+0000 7fd416800640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4181f6d40 0x7fd41815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 -- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4180c52b0 msgr2=0x7fd4181f6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4180c52b0 0x7fd4181f6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 -- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4181f6d40 msgr2=0x7fd41815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:17.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4181f6d40 0x7fd41815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 -- 172.21.15.129:0/412376274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41815a500 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd415e00640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fd4080117e0 tx=0x7fd408011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40801e070 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd40800fe10 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd408019690 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.302+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41815a7e0 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.303+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41815ada0 con 0x7fd3fc005150 2024-09-17T04:05:17.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.304+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4180b4af0 con 0x7fd3fc005150 2024-09-17T04:05:17.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.305+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd408021070 con 0x7fd3fc005150 2024-09-17T04:05:17.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.306+0000 7fd40ea00640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 0x7fd3f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:17.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.306+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7fd40809b320 con 0x7fd3fc005150 2024-09-17T04:05:17.305 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.306+0000 7fd40d400640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 0x7fd3f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:17.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.307+0000 7fd40d400640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 0x7fd3f407b0d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd4180d0d60 tx=0x7fd4000067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:17.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.310+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4080633a0 con 0x7fd3fc005150 2024-09-17T04:05:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:17.561+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]} v 0) -- 0x7fd41806a140 con 0x7fd3fc005150 2024-09-17T04:05:18.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.161+0000 7fd40ea00640 1 -- 172.21.15.129:0/412376274 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]}]=0 v793) ==== 142+0+0 (secure 0 0 0) 0x7fd408067dd0 con 0x7fd3fc005150 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 msgr2=0x7fd3f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 0x7fd3f407b0d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fd4180d0d60 tx=0x7fd4000067a0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 msgr2=0x7fd418135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fd4080117e0 tx=0x7fd408011ca0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 shutdown_connections 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd3f4078bc0 0x7fd3f407b0d0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd4181f6d40 0x7fd41815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd4180c52b0 0x7fd4181f6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 --2- 172.21.15.129:0/412376274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd3fc005150 0x7fd418135440 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.164+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 >> 172.21.15.129:0/412376274 conn(0x7fd41806cfc0 msgr2=0x7fd418156a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.165+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 shutdown_connections 2024-09-17T04:05:18.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.165+0000 7fd417200640 1 -- 172.21.15.129:0/412376274 wait complete. 2024-09-17T04:05:18.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 -- 172.21.15.129:0/1415199100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc0705a0 msgr2=0x7fc5cc0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 --2- 172.21.15.129:0/1415199100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc0705a0 0x7fc5cc0b4140 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc072080 tx=0x7fc5bc02a530 comp rx=0 tx=0).stop 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 -- 172.21.15.129:0/1415199100 shutdown_connections 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 --2- 172.21.15.129:0/1415199100 >> v2:172.21.15.181:3300/0 conn(0x7fc5cc06eed0 0x7fc5cc0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 --2- 172.21.15.129:0/1415199100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc0705a0 0x7fc5cc0b4140 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 -- 172.21.15.129:0/1415199100 >> 172.21.15.129:0/1415199100 conn(0x7fc5cc06d410 msgr2=0x7fc5cc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 -- 172.21.15.129:0/1415199100 shutdown_connections 2024-09-17T04:05:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.411+0000 7fc5d3000640 1 -- 172.21.15.129:0/1415199100 wait complete. 2024-09-17T04:05:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.413+0000 7fc5d3000640 1 Processor -- start 2024-09-17T04:05:18.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.414+0000 7fc5d3000640 1 -- start start 2024-09-17T04:05:18.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5cc06eed0 0x7fc5cc0c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 0x7fc5cc0b1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc161250 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc5cc1610d0 con 0x7fc5cc06eed0 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5d3000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc5cc1613d0 con 0x7fc5cc0705a0 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5c3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41680/0 (socket says 172.21.15.129:41680) 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.415+0000 7fc5c3400640 1 -- 172.21.15.129:0/2941124524 learned_addr learned my addr 172.21.15.129:0/2941124524 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5cbe00640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5cc06eed0 0x7fc5cc0c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5d1000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 0x7fc5cc0b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 -- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 msgr2=0x7fc5cc0b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 0x7fc5cc0b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 -- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5cc06eed0 msgr2=0x7fc5cc0c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5cc06eed0 0x7fc5cc0c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 -- 172.21.15.129:0/2941124524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5cc1f72c0 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5d1000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 0x7fc5cc0b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c3400640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400d720 tx=0x7fc5b400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4017070 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc5b4004310 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5cc1f75a0 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b4005280 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.416+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5cc1f7b60 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.417+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5cc158420 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.419+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc5b4005420 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.420+0000 7fc5c8c00640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 0x7fc5a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.420+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7fc5b409aed0 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.420+0000 7fc5cbe00640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 0x7fc5a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.421+0000 7fc5cbe00640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 0x7fc5a807b0a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc034000 tx=0x7fc5bc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:18.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.424+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc5b4020020 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.604+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fc5cc0cd5a0 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.605+0000 7fc5c8c00640 1 -- 172.21.15.129:0/2941124524 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v63) ==== 59+0+2294 (secure 0 0 0) 0x7fc5b4063510 con 0x7fc5cc06f2e0 2024-09-17T04:05:18.605 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:05:18.605 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:05:18.605 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:05:18.606 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:05:18.607 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:05:18.608 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:05:18.608 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:05:18.608 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 msgr2=0x7fc5a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 0x7fc5a807b0a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc034000 tx=0x7fc5bc004730 comp rx=0 tx=0).stop 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 msgr2=0x7fc5cc157b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7fc5b400d720 tx=0x7fc5b400dbe0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 shutdown_connections 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc5a8078b90 0x7fc5a807b0a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc5cc0705a0 0x7fc5cc0b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5cc06f2e0 0x7fc5cc157b40 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 --2- 172.21.15.129:0/2941124524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc5cc06eed0 0x7fc5cc0c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.607+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 >> 172.21.15.129:0/2941124524 conn(0x7fc5cc06d410 msgr2=0x7fc5cc074930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:18.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.608+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 shutdown_connections 2024-09-17T04:05:18.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.608+0000 7fc5d3000640 1 -- 172.21.15.129:0/2941124524 wait complete. 2024-09-17T04:05:18.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:05:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.857+0000 7f024fe00640 1 -- 172.21.15.129:0/1876413324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f025006fe80 msgr2=0x7f0250070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.857+0000 7f024fe00640 1 --2- 172.21.15.129:0/1876413324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f025006fe80 0x7f0250070290 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f0248005fd0 tx=0x7f024802aa90 comp rx=0 tx=0).stop 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 -- 172.21.15.129:0/1876413324 shutdown_connections 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 --2- 172.21.15.129:0/1876413324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f025006fe80 0x7f0250070290 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 --2- 172.21.15.129:0/1876413324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02500cb280 0x7f02500c5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 -- 172.21.15.129:0/1876413324 >> 172.21.15.129:0/1876413324 conn(0x7f02500d00b0 msgr2=0x7f02500d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 -- 172.21.15.129:0/1876413324 shutdown_connections 2024-09-17T04:05:18.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.858+0000 7f024fe00640 1 -- 172.21.15.129:0/1876413324 wait complete. 2024-09-17T04:05:18.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.860+0000 7f024fe00640 1 Processor -- start 2024-09-17T04:05:18.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.860+0000 7f024fe00640 1 -- start start 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f025006fe80 0x7f0250154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 0x7f02500b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02500ceee0 con 0x7f02500cb280 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f02500ced60 con 0x7f025006fe80 2024-09-17T04:05:18.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.861+0000 7f024fe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f02500cf060 con 0x7f025015e970 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41690/0 (socket says 172.21.15.129:41690) 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 -- 172.21.15.129:0/3738474037 learned_addr learned my addr 172.21.15.129:0/3738474037 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024ea00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f025006fe80 0x7f0250154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024f400640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 0x7f02500b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 -- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 msgr2=0x7f02500b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 0x7f02500b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 -- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f025006fe80 msgr2=0x7f0250154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f025006fe80 0x7f0250154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:18.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 -- 172.21.15.129:0/3738474037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f025015f090 con 0x7f02500cb280 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024f400640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 0x7f02500b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.862+0000 7f024e000640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c3d0 tx=0x7f023c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.863+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c017070 con 0x7f02500cb280 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.863+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f023c0053c0 con 0x7f02500cb280 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.863+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02500c7e90 con 0x7f02500cb280 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.863+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023c003850 con 0x7f02500cb280 2024-09-17T04:05:18.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.863+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02501f6b40 con 0x7f02500cb280 2024-09-17T04:05:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.864+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02500cf580 con 0x7f02500cb280 2024-09-17T04:05:18.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.865+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f023c01a070 con 0x7f02500cb280 2024-09-17T04:05:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.866+0000 7f0246a00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 0x7f023007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.867+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7f023c099da0 con 0x7f02500cb280 2024-09-17T04:05:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.867+0000 7f024ea00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 0x7f023007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:18.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.868+0000 7f024ea00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 0x7f023007b0d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0240009ac0 tx=0x7f0240005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:18.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:18.871+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f023c062500 con 0x7f02500cb280 2024-09-17T04:05:19.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.055+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f025015eec0 con 0x7f02500cb280 2024-09-17T04:05:19.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.059+0000 7f0246a00640 1 -- 172.21.15.129:0/3738474037 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) ==== 122+0+0 (secure 0 0 0) 0x7f023c061e00 con 0x7f02500cb280 2024-09-17T04:05:19.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 msgr2=0x7f023007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 0x7f023007b0d0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f0240009ac0 tx=0x7f0240005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:19.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 msgr2=0x7f02500c7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f023c00c3d0 tx=0x7f023c00c890 comp rx=0 tx=0).stop 2024-09-17T04:05:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 shutdown_connections 2024-09-17T04:05:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0230078bc0 0x7f023007b0d0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f025015e970 0x7f02500b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02500cb280 0x7f02500c7120 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 --2- 172.21.15.129:0/3738474037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f025006fe80 0x7f0250154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.061+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 >> 172.21.15.129:0/3738474037 conn(0x7f02500d00b0 msgr2=0x7f02500ca870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:19.064 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.063+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 shutdown_connections 2024-09-17T04:05:19.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.063+0000 7f024fe00640 1 -- 172.21.15.129:0/3738474037 wait complete. 2024-09-17T04:05:19.090 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:05:19.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:19.186 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:19.279 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:05:19.279 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:19.279 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:19.314 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:19.314 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:19.315 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:19.350 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:19.350 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:19.350 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:19.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:05:19.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:05:19.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:19.393 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:19.393 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:19.432 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:19.432 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:05:19.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:19.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 -- 172.21.15.129:0/1388493760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801528d0 msgr2=0x7f5f8015bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 --2- 172.21.15.129:0/1388493760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801528d0 0x7f5f8015bca0 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009830 tx=0x7f5f6802a5b0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 -- 172.21.15.129:0/1388493760 shutdown_connections 2024-09-17T04:05:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 --2- 172.21.15.129:0/1388493760 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801528d0 0x7f5f8015bca0 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 -- 172.21.15.129:0/1388493760 >> 172.21.15.129:0/1388493760 conn(0x7f5f8006cfc0 msgr2=0x7f5f8006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 -- 172.21.15.129:0/1388493760 shutdown_connections 2024-09-17T04:05:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.653+0000 7f5f84a00640 1 -- 172.21.15.129:0/1388493760 wait complete. 2024-09-17T04:05:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.655+0000 7f5f84a00640 1 Processor -- start 2024-09-17T04:05:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.655+0000 7f5f84a00640 1 -- start start 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f801528d0 0x7f5f801f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 0x7f5f8014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d0440 con 0x7f5f801524c0 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5f800d02c0 con 0x7f5f801528d0 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f84a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5f800d05c0 con 0x7f5f8015c1e0 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:19.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41708/0 (socket says 172.21.15.129:41708) 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 -- 172.21.15.129:0/2865902659 learned_addr learned my addr 172.21.15.129:0/2865902659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f75e00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f801528d0 0x7f5f801f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 -- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 msgr2=0x7f5f8014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7f200640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 0x7f5f8014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 0x7f5f8014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 -- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f801528d0 msgr2=0x7f5f801f8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f801528d0 0x7f5f801f8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7e800640 1 -- 172.21.15.129:0/2865902659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f801f9120 con 0x7f5f801524c0 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.657+0000 7f5f7f200640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 0x7f5f8014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f7e800640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c400 tx=0x7f5f7000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:19.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70011790 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f70017070 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f801f5630 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70011ce0 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.658+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f801591d0 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.659+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f800d0a70 con 0x7f5f801524c0 2024-09-17T04:05:19.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.662+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5f7001c460 con 0x7f5f801524c0 2024-09-17T04:05:19.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.662+0000 7f5f77400640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 0x7f5f5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:19.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.663+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 5029+0+0 (secure 0 0 0) 0x7f5f7009b400 con 0x7f5f801524c0 2024-09-17T04:05:19.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.663+0000 7f5f75e00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 0x7f5f5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:19.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.664+0000 7f5f75e00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 0x7f5f5c07b0a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5f68034000 tx=0x7f5f68004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:19.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.666+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5f70063b50 con 0x7f5f801524c0 2024-09-17T04:05:19.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.885+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f8006a140 con 0x7f5f801524c0 2024-09-17T04:05:19.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.885+0000 7f5f77400640 1 -- 172.21.15.129:0/2865902659 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) ==== 95+0+2272 (secure 0 0 0) 0x7f5f70063450 con 0x7f5f801524c0 2024-09-17T04:05:19.886 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:19.886 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":201,"btime":"2024-09-17T04:05:11:699514+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":10135,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/4184582844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":4184582844},{"type":"v1","addr":"172.21.15.181:6835","nonce":4184582844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":201},{"gid":10842,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1344703948","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1344703948},{"type":"v1","addr":"172.21.15.129:6833","nonce":1344703948}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":199}],"filesystems":[]} 2024-09-17T04:05:19.886 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 201 2024-09-17T04:05:19.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 msgr2=0x7f5f5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 0x7f5f5c07b0a0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5f68034000 tx=0x7f5f68004730 comp rx=0 tx=0).stop 2024-09-17T04:05:19.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 msgr2=0x7f5f8015f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:19.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000c400 tx=0x7f5f7000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 shutdown_connections 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f5c078b90 0x7f5f5c07b0a0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f8015c1e0 0x7f5f8014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f801528d0 0x7f5f801f8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 --2- 172.21.15.129:0/2865902659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f801524c0 0x7f5f8015f9d0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 >> 172.21.15.129:0/2865902659 conn(0x7f5f8006cfc0 msgr2=0x7f5f8006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 shutdown_connections 2024-09-17T04:05:19.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:19.888+0000 7f5f84a00640 1 -- 172.21.15.129:0/2865902659 wait complete. 2024-09-17T04:05:19.914 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:05:19.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:05:19.915 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:05:19.915 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:05:19.915 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:05:19.915 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:19.915 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:05:19.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:05:19.916 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:05:19.916 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:05:19.916 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:05:19.916 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:19.938 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:05:19.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:05:19.940 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:05:19.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:05:19.941 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:05:19.941 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:19.942 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:05:19.942 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:19.944 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:05:19.944 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:19.945 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:05:19.945 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:19.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:05:20.136 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:05:20.136 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:05:20.138 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:05:20.138 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:05:20.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 -- 172.21.15.129:0/2030441951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 msgr2=0x7f7108070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:20.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 --2- 172.21.15.129:0/2030441951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f7108070510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f71080ce800 tx=0x7f710002a920 comp rx=0 tx=0).stop 2024-09-17T04:05:20.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 -- 172.21.15.129:0/2030441951 shutdown_connections 2024-09-17T04:05:20.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 --2- 172.21.15.129:0/2030441951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f7108070510 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:20.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 -- 172.21.15.129:0/2030441951 >> 172.21.15.129:0/2030441951 conn(0x7f710806d410 msgr2=0x7f71080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:20.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 -- 172.21.15.129:0/2030441951 shutdown_connections 2024-09-17T04:05:20.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.179+0000 7f710ec00640 1 -- 172.21.15.129:0/2030441951 wait complete. 2024-09-17T04:05:20.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.182+0000 7f710ec00640 1 Processor -- start 2024-09-17T04:05:20.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.182+0000 7f710ec00640 1 -- start start 2024-09-17T04:05:20.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f71080c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:20.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:20.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f710815da00 0x7f7108099740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:20.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7108070bd0 con 0x7f71080ccc80 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7108070a50 con 0x7f710815da00 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f710ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7108070d50 con 0x7f7108070100 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f7107400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f7107400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41740/0 (socket says 172.21.15.129:41740) 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f7107400640 1 -- 172.21.15.129:0/4227399959 learned_addr learned my addr 172.21.15.129:0/4227399959 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:20.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.185+0000 7f7107e00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f71080c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 -- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 msgr2=0x7f71080c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f71080c08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 -- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f710815da00 msgr2=0x7f7108099740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f710815da00 0x7f7108099740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 -- 172.21.15.129:0/4227399959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71080c13e0 con 0x7f71080ccc80 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.186+0000 7f7107400640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f70f400c3d0 tx=0x7f70f400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.187+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4017070 con 0x7f71080ccc80 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.187+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f70f40053c0 con 0x7f71080ccc80 2024-09-17T04:05:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.187+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71080976b0 con 0x7f71080ccc80 2024-09-17T04:05:20.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.187+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4003850 con 0x7f71080ccc80 2024-09-17T04:05:20.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.187+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7108153370 con 0x7f71080ccc80 2024-09-17T04:05:20.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.189+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71080994d0 con 0x7f71080ccc80 2024-09-17T04:05:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.191+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70f401a070 con 0x7f71080ccc80 2024-09-17T04:05:20.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.191+0000 7f70ffe00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 0x7f70e807b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.192+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 5029+0+0 (secure 0 0 0) 0x7f70f409a2a0 con 0x7f71080ccc80 2024-09-17T04:05:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.192+0000 7f7107e00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 0x7f70e807b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.192+0000 7f7107e00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 0x7f70e807b0f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f70f80046b0 tx=0x7f70f8007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.194+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f71080994d0 con 0x7f71080ccc80 2024-09-17T04:05:20.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:20.389+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f7108070510 con 0x7f71080ccc80 2024-09-17T04:05:21.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.189+0000 7f70ffe00640 1 -- 172.21.15.129:0/4227399959 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v214) ==== 97+0+0 (secure 0 0 0) 0x7f70f4021020 con 0x7f71080ccc80 2024-09-17T04:05:21.190 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 msgr2=0x7f70e807b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 0x7f70e807b0f0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f70f80046b0 tx=0x7f70f8007e90 comp rx=0 tx=0).stop 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 msgr2=0x7f71080c0ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f70f400c3d0 tx=0x7f70f400c890 comp rx=0 tx=0).stop 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 shutdown_connections 2024-09-17T04:05:21.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f70e8078be0 0x7f70e807b0f0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f710815da00 0x7f7108099740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f71080ccc80 0x7f71080c0ea0 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 --2- 172.21.15.129:0/4227399959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7108070100 0x7f71080c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 >> 172.21.15.129:0/4227399959 conn(0x7f710806d410 msgr2=0x7f71080927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 shutdown_connections 2024-09-17T04:05:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.193+0000 7f710ec00640 1 -- 172.21.15.129:0/4227399959 wait complete. 2024-09-17T04:05:21.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 -- 172.21.15.129:0/1531898068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d68160cf0 msgr2=0x7f3d680d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1531898068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d68160cf0 0x7f3d680d02c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009830 tx=0x7f3d5002a480 comp rx=0 tx=0).stop 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 -- 172.21.15.129:0/1531898068 shutdown_connections 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1531898068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680d0800 0x7f3d6815cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1531898068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d68160cf0 0x7f3d680d02c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 -- 172.21.15.129:0/1531898068 >> 172.21.15.129:0/1531898068 conn(0x7f3d6806cfc0 msgr2=0x7f3d6806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.442+0000 7f3d67e00640 1 -- 172.21.15.129:0/1531898068 shutdown_connections 2024-09-17T04:05:21.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.443+0000 7f3d67e00640 1 -- 172.21.15.129:0/1531898068 wait complete. 2024-09-17T04:05:21.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.445+0000 7f3d67e00640 1 Processor -- start 2024-09-17T04:05:21.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.445+0000 7f3d67e00640 1 -- start start 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 0x7f3d68158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 0x7f3d680c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6815d290 con 0x7f3d68160cf0 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6815d110 con 0x7f3d680d0800 2024-09-17T04:05:21.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.446+0000 7f3d67e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6815d410 con 0x7f3d680cfb60 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41746/0 (socket says 172.21.15.129:41746) 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 -- 172.21.15.129:0/1471102961 learned_addr learned my addr 172.21.15.129:0/1471102961 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d66a00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 0x7f3d68158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d5e000640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 0x7f3d680c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 -- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 msgr2=0x7f3d68158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 0x7f3d68158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 -- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 msgr2=0x7f3d680c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 0x7f3d680c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 -- 172.21.15.129:0/1471102961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d681577c0 con 0x7f3d68160cf0 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d66a00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 0x7f3d68158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:21.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d5e000640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 0x7f3d680c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:21.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.447+0000 7f3d67400640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000ec50 tx=0x7f3d6000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:21.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.448+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60003b70 con 0x7f3d68160cf0 2024-09-17T04:05:21.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.448+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3d60003d10 con 0x7f3d68160cf0 2024-09-17T04:05:21.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.448+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d60004280 con 0x7f3d68160cf0 2024-09-17T04:05:21.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.448+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d68157aa0 con 0x7f3d68160cf0 2024-09-17T04:05:21.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.448+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d68158060 con 0x7f3d68160cf0 2024-09-17T04:05:21.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.449+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d68157530 con 0x7f3d68160cf0 2024-09-17T04:05:21.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.450+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3d60004420 con 0x7f3d68160cf0 2024-09-17T04:05:21.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.451+0000 7f3d5f400640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 0x7f3d4407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.452+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 4857+0+0 (secure 0 0 0) 0x7f3d6009b930 con 0x7f3d68160cf0 2024-09-17T04:05:21.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.452+0000 7f3d66a00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 0x7f3d4407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.453+0000 7f3d66a00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 0x7f3d4407b0a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3d580013f0 tx=0x7f3d58009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:21.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.456+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3d600640b0 con 0x7f3d68160cf0 2024-09-17T04:05:21.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.703+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3d6815d930 con 0x7f3d68160cf0 2024-09-17T04:05:21.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.704+0000 7f3d5f400640 1 -- 172.21.15.129:0/1471102961 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) ==== 80+0+5304 (secure 0 0 0) 0x7f3d600639b0 con 0x7f3d68160cf0 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:05:21.704 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.705 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:05:21.706 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:05:21.707 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.708 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.709 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.710 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.711 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:05:21.712 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.713 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.714 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:05:21.715 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:05:21.716 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.717 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:05:21.718 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:05:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 msgr2=0x7f3d4407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 0x7f3d4407b0a0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f3d580013f0 tx=0x7f3d58009040 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 msgr2=0x7f3d681599b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000ec50 tx=0x7f3d6000c6a0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 shutdown_connections 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3d44078b90 0x7f3d4407b0a0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3d68160cf0 0x7f3d681599b0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d680d0800 0x7f3d680c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 --2- 172.21.15.129:0/1471102961 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3d680cfb60 0x7f3d68158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 >> 172.21.15.129:0/1471102961 conn(0x7f3d6806cfc0 msgr2=0x7f3d68153e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 shutdown_connections 2024-09-17T04:05:21.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.706+0000 7f3d67e00640 1 -- 172.21.15.129:0/1471102961 wait complete. 2024-09-17T04:05:21.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 -- 172.21.15.129:0/2698077925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0b4140 msgr2=0x7f1bac0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2698077925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0b4140 0x7f1bac0ccc80 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7f1bac072e00 tx=0x7f1b9802a530 comp rx=0 tx=0).stop 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 -- 172.21.15.129:0/2698077925 shutdown_connections 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2698077925 >> v2:172.21.15.181:3300/0 conn(0x7f1bac0cd1c0 0x7f1bac1609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2698077925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0b4140 0x7f1bac0ccc80 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 -- 172.21.15.129:0/2698077925 >> 172.21.15.129:0/2698077925 conn(0x7f1bac06d410 msgr2=0x7f1bac0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 -- 172.21.15.129:0/2698077925 shutdown_connections 2024-09-17T04:05:21.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.947+0000 7f1bb4200640 1 -- 172.21.15.129:0/2698077925 wait complete. 2024-09-17T04:05:21.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.949+0000 7f1bb4200640 1 Processor -- start 2024-09-17T04:05:21.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.950+0000 7f1bb4200640 1 -- start start 2024-09-17T04:05:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 0x7f1bac0c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 0x7f1bac1f5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac160260 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1bac1600e0 con 0x7f1bac0705a0 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb4200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1bac160f00 con 0x7f1bac0b4140 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41762/0 (socket says 172.21.15.129:41762) 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.951+0000 7f1bb2200640 1 -- 172.21.15.129:0/2075041674 learned_addr learned my addr 172.21.15.129:0/2075041674 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:21.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb0e00640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 0x7f1bac1f5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb1800640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 0x7f1bac0c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 -- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 msgr2=0x7f1bac1f5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 0x7f1bac1f5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 -- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 msgr2=0x7f1bac0c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 0x7f1bac0c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 -- 172.21.15.129:0/2075041674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bac1fa3a0 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb0e00640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 0x7f1bac1f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb1800640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 0x7f1bac0c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb2200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e990 tx=0x7f1ba400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4004d20 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ba400ca40 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4005020 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bac0d0d90 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.952+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bac0c69f0 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.953+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bac0ccc80 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.955+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1ba40051c0 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.956+0000 7f1ba9600640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 0x7f1b8c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:21.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.956+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 4857+0+0 (secure 0 0 0) 0x7f1ba409b2d0 con 0x7f1bac0cd1c0 2024-09-17T04:05:21.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.956+0000 7f1bb1800640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 0x7f1b8c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:21.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.957+0000 7f1bb1800640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 0x7f1b8c07b0a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0000f60 tx=0x7f1ba0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:21.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:21.960+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1ba4016030 con 0x7f1bac0cd1c0 2024-09-17T04:05:22.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.145+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f1bac06a140 con 0x7f1bac0cd1c0 2024-09-17T04:05:22.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.146+0000 7f1ba9600640 1 -- 172.21.15.129:0/2075041674 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v214) ==== 84+0+65 (secure 0 0 0) 0x7f1ba4063a50 con 0x7f1bac0cd1c0 2024-09-17T04:05:22.146 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:22.146 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:05:22.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:05:22.147 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:05:22.147 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:05:22.147 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:05:22.147 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 msgr2=0x7f1b8c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 0x7f1b8c07b0a0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0000f60 tx=0x7f1ba0009040 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 msgr2=0x7f1bac0c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e990 tx=0x7f1ba400ee50 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 shutdown_connections 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1b8c078b90 0x7f1b8c07b0a0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1bac0cd1c0 0x7f1bac0c9320 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1bac0b4140 0x7f1bac1f5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 --2- 172.21.15.129:0/2075041674 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1bac0705a0 0x7f1bac0c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 >> 172.21.15.129:0/2075041674 conn(0x7f1bac06d410 msgr2=0x7f1bac0b1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:22.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.148+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 shutdown_connections 2024-09-17T04:05:22.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.149+0000 7f1bb4200640 1 -- 172.21.15.129:0/2075041674 wait complete. 2024-09-17T04:05:22.174 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:05:22.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:05:22.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 -- 172.21.15.129:0/1801293467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc840cba90 msgr2=0x7fbc840cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 --2- 172.21.15.129:0/1801293467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc840cba90 0x7fbc840cbea0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbc80005fd0 tx=0x7fbc8002a810 comp rx=0 tx=0).stop 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 -- 172.21.15.129:0/1801293467 shutdown_connections 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 --2- 172.21.15.129:0/1801293467 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc840cba90 0x7fbc840cbea0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 --2- 172.21.15.129:0/1801293467 >> v2:172.21.15.181:3300/0 conn(0x7fbc840b3f70 0x7fbc840b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 -- 172.21.15.129:0/1801293467 >> 172.21.15.129:0/1801293467 conn(0x7fbc840ccc40 msgr2=0x7fbc840cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 -- 172.21.15.129:0/1801293467 shutdown_connections 2024-09-17T04:05:22.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.402+0000 7fbc8c800640 1 -- 172.21.15.129:0/1801293467 wait complete. 2024-09-17T04:05:22.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.404+0000 7fbc8c800640 1 Processor -- start 2024-09-17T04:05:22.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.405+0000 7fbc8c800640 1 -- start start 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 0x7fbc841567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 0x7fbc8415b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbc840cc930 con 0x7fbc840cba90 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbc840cc7b0 con 0x7fbc84074580 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc8c800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbc84166f20 con 0x7fbc84157630 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.406+0000 7fbc89400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41772/0 (socket says 172.21.15.129:41772) 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 -- 172.21.15.129:0/2547026213 learned_addr learned my addr 172.21.15.129:0/2547026213 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89e00640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 0x7fbc841567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:22.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc8a800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 0x7fbc8415b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 -- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 msgr2=0x7fbc8415b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 0x7fbc8415b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 -- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 msgr2=0x7fbc841567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 0x7fbc841567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 -- 172.21.15.129:0/2547026213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc8409bfc0 con 0x7fbc840cba90 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc8a800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 0x7fbc8415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89e00640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 0x7fbc841567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc89400640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c430 tx=0x7fbc7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74011790 con 0x7fbc840cba90 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc74017070 con 0x7fbc840cba90 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.407+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc84159bd0 con 0x7fbc840cba90 2024-09-17T04:05:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.408+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc74011ce0 con 0x7fbc840cba90 2024-09-17T04:05:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.408+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc8409d540 con 0x7fbc840cba90 2024-09-17T04:05:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.409+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc840b3f70 con 0x7fbc840cba90 2024-09-17T04:05:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.410+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbc7401c460 con 0x7fbc840cba90 2024-09-17T04:05:22.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.411+0000 7fbc7e000640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 0x7fbc6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:22.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.411+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 4857+0+0 (secure 0 0 0) 0x7fbc7409a8b0 con 0x7fbc840cba90 2024-09-17T04:05:22.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.411+0000 7fbc89e00640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 0x7fbc6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:22.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.412+0000 7fbc89e00640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 0x7fbc6807b0a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009b80 tx=0x7fbc78005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:22.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.415+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbc74062fa0 con 0x7fbc840cba90 2024-09-17T04:05:22.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:22.598+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fbc840706b0 con 0x7fbc840cba90 2024-09-17T04:05:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.190+0000 7fbc7e000640 1 -- 172.21.15.129:0/2547026213 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v215) ==== 138+0+0 (secure 0 0 0) 0x7fbc740628a0 con 0x7fbc840cba90 2024-09-17T04:05:23.190 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.192+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 msgr2=0x7fbc6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.192+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 0x7fbc6807b0a0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009b80 tx=0x7fbc78005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.192+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 msgr2=0x7fbc8415b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.192+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c430 tx=0x7fbc7400c8f0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 shutdown_connections 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc68078b90 0x7fbc6807b0a0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbc84157630 0x7fbc8415b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbc840cba90 0x7fbc8415b080 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 --2- 172.21.15.129:0/2547026213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc84074580 0x7fbc841567a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 >> 172.21.15.129:0/2547026213 conn(0x7fbc840ccc40 msgr2=0x7fbc840cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:23.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 shutdown_connections 2024-09-17T04:05:23.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.193+0000 7fbc8c800640 1 -- 172.21.15.129:0/2547026213 wait complete. 2024-09-17T04:05:23.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 -- 172.21.15.129:0/106979804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 msgr2=0x7fbb0c0cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 --2- 172.21.15.129:0/106979804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c0cdcb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009830 tx=0x7fbafc02a3f0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 -- 172.21.15.129:0/106979804 shutdown_connections 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 --2- 172.21.15.129:0/106979804 >> v2:172.21.15.181:3300/0 conn(0x7fbb0c0b5470 0x7fbb0c0b5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 --2- 172.21.15.129:0/106979804 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c0cdcb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.447+0000 7fbb11c00640 1 -- 172.21.15.129:0/106979804 >> 172.21.15.129:0/106979804 conn(0x7fbb0c0d1300 msgr2=0x7fbb0c0d1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.448+0000 7fbb11c00640 1 -- 172.21.15.129:0/106979804 shutdown_connections 2024-09-17T04:05:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.448+0000 7fbb11c00640 1 -- 172.21.15.129:0/106979804 wait complete. 2024-09-17T04:05:23.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.450+0000 7fbb11c00640 1 Processor -- start 2024-09-17T04:05:23.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.450+0000 7fbb11c00640 1 -- start start 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 0x7fbb0c1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c1659a0 con 0x7fbb0c0b5470 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c165820 con 0x7fbb0c0b6b70 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.451+0000 7fbb11c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0c165b20 con 0x7fbb0c0b6760 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:23.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:41788/0 (socket says 172.21.15.129:41788) 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 -- 172.21.15.129:0/3632054121 learned_addr learned my addr 172.21.15.129:0/3632054121 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb02000640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0b400640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 0x7fbb0c1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 -- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 msgr2=0x7fbb0c1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c1591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 -- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 msgr2=0x7fbb0c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 0x7fbb0c1596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0aa00640 1 -- 172.21.15.129:0/3632054121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c1589c0 con 0x7fbb0c0b5470 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.452+0000 7fbb0b400640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 0x7fbb0c1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb0aa00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009950 tx=0x7fbafc033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc005c90 con 0x7fbb0c0b5470 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbafc031e50 con 0x7fbb0c0b5470 2024-09-17T04:05:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c0cd110 con 0x7fbb0c0b5470 2024-09-17T04:05:23.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc005260 con 0x7fbb0c0b5470 2024-09-17T04:05:23.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.453+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c0cd6d0 con 0x7fbb0c0b5470 2024-09-17T04:05:23.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.454+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb0c0b5880 con 0x7fbb0c0b5470 2024-09-17T04:05:23.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.455+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbafc005400 con 0x7fbb0c0b5470 2024-09-17T04:05:23.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.456+0000 7fbb03400640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 0x7fbae807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:23.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.456+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 5227+0+0 (secure 0 0 0) 0x7fbafc0bbc00 con 0x7fbb0c0b5470 2024-09-17T04:05:23.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.457+0000 7fbb02000640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 0x7fbae807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:23.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.457+0000 7fbb02000640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 0x7fbae807b0a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4007fd0 tx=0x7fbaf4001390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:23.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.461+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbafc03f030 con 0x7fbb0c0b5470 2024-09-17T04:05:23.646 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:23.646+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbb0c0cf420 con 0x7fbb0c0b5470 2024-09-17T04:05:24.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.201+0000 7fbb03400640 1 -- 172.21.15.129:0/3632054121 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v216) ==== 170+0+0 (secure 0 0 0) 0x7fbafc083bc0 con 0x7fbb0c0b5470 2024-09-17T04:05:24.201 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.203+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 msgr2=0x7fbae807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.203+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 0x7fbae807b0a0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4007fd0 tx=0x7fbaf4001390 comp rx=0 tx=0).stop 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.203+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 msgr2=0x7fbb0c147550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.203+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fbafc009950 tx=0x7fbafc033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 shutdown_connections 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbae8078b90 0x7fbae807b0a0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb0c0b6b70 0x7fbb0c1596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb0c0b6760 0x7fbb0c1591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 --2- 172.21.15.129:0/3632054121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb0c0b5470 0x7fbb0c147550 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.204+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 >> 172.21.15.129:0/3632054121 conn(0x7fbb0c0d1300 msgr2=0x7fbb0c06cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.205+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 shutdown_connections 2024-09-17T04:05:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.205+0000 7fbb11c00640 1 -- 172.21.15.129:0/3632054121 wait complete. 2024-09-17T04:05:24.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 -- 172.21.15.129:0/3587827165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc074800 msgr2=0x7f6ddc0af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 --2- 172.21.15.129:0/3587827165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc074800 0x7f6ddc0af480 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc0d1260 tx=0x7f6dcc02a560 comp rx=0 tx=0).stop 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 -- 172.21.15.129:0/3587827165 shutdown_connections 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 --2- 172.21.15.129:0/3587827165 >> v2:172.21.15.181:3300/0 conn(0x7f6ddc0b3010 0x7f6ddc0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 --2- 172.21.15.129:0/3587827165 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc074800 0x7f6ddc0af480 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.468+0000 7f6de1c00640 1 -- 172.21.15.129:0/3587827165 >> 172.21.15.129:0/3587827165 conn(0x7f6ddc0ccc40 msgr2=0x7f6ddc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:24.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.469+0000 7f6de1c00640 1 -- 172.21.15.129:0/3587827165 shutdown_connections 2024-09-17T04:05:24.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.469+0000 7f6de1c00640 1 -- 172.21.15.129:0/3587827165 wait complete. 2024-09-17T04:05:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.471+0000 7f6de1c00640 1 Processor -- start 2024-09-17T04:05:24.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.471+0000 7f6de1c00640 1 -- start start 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ddc074800 0x7f6ddc09c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ddc0b1c40 0x7f6ddc1e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b3ae0 con 0x7f6ddc0b3010 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b3960 con 0x7f6ddc0b1c40 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.472+0000 7f6de1c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ddc0b3c60 con 0x7f6ddc074800 2024-09-17T04:05:24.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56672/0 (socket says 172.21.15.129:56672) 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 -- 172.21.15.129:0/4128867542 learned_addr learned my addr 172.21.15.129:0/4128867542 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddaa00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ddc074800 0x7f6ddc09c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6dda000640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ddc0b1c40 0x7f6ddc1e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 -- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ddc074800 msgr2=0x7f6ddc09c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ddc074800 0x7f6ddc09c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 -- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ddc0b1c40 msgr2=0x7f6ddc1e69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ddc0b1c40 0x7f6ddc1e69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:24.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.473+0000 7f6ddb400640 1 -- 172.21.15.129:0/4128867542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc1e50b0 con 0x7f6ddc0b3010 2024-09-17T04:05:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6ddb400640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b9e0 tx=0x7f6dd400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4019070 con 0x7f6ddc0b3010 2024-09-17T04:05:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dd400cc30 con 0x7f6ddc0b3010 2024-09-17T04:05:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc1e5390 con 0x7f6ddc0b3010 2024-09-17T04:05:24.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd40146a0 con 0x7f6ddc0b3010 2024-09-17T04:05:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.474+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc1e5950 con 0x7f6ddc0b3010 2024-09-17T04:05:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.475+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ddc1f4420 con 0x7f6ddc0b3010 2024-09-17T04:05:24.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.477+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6dd401c070 con 0x7f6ddc0b3010 2024-09-17T04:05:24.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.477+0000 7f6dd2a00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 0x7f6dbc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:24.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.478+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 5609+0+0 (secure 0 0 0) 0x7f6dd409add0 con 0x7f6ddc0b3010 2024-09-17T04:05:24.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.478+0000 7f6ddaa00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 0x7f6dbc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:24.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.479+0000 7f6ddaa00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 0x7f6dbc07b0d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc033000 tx=0x7f6dcc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:24.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.481+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6dd4023020 con 0x7f6ddc0b3010 2024-09-17T04:05:24.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:24.701+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f6ddc0b4200 con 0x7f6ddc0b3010 2024-09-17T04:05:25.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.220+0000 7f6dd2a00640 1 -- 172.21.15.129:0/4128867542 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 32 and data pool 33 v204) ==== 174+0+0 (secure 0 0 0) 0x7f6dd400ce20 con 0x7f6ddc0b3010 2024-09-17T04:05:25.224 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 32 and data pool 33 2024-09-17T04:05:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.232+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 msgr2=0x7f6dbc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.232+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 0x7f6dbc07b0d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc033000 tx=0x7f6dcc004730 comp rx=0 tx=0).stop 2024-09-17T04:05:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.232+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 msgr2=0x7f6ddc1e6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.232+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400b9e0 tx=0x7f6dd400bea0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 shutdown_connections 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6dbc078bc0 0x7f6dbc07b0d0 secure :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc033000 tx=0x7f6dcc004730 comp rx=0 tx=0).stop 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6ddc0b3010 0x7f6ddc1e6ee0 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ddc0b1c40 0x7f6ddc1e69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 --2- 172.21.15.129:0/4128867542 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ddc074800 0x7f6ddc09c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.233 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.233+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 >> 172.21.15.129:0/4128867542 conn(0x7f6ddc0ccc40 msgr2=0x7f6ddc0d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.235+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 shutdown_connections 2024-09-17T04:05:25.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.235+0000 7f6de1c00640 1 -- 172.21.15.129:0/4128867542 wait complete. 2024-09-17T04:05:25.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:25.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 -- 172.21.15.129:0/2860963091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040cf5f0 msgr2=0x7f4a040cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/2860963091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040cf5f0 0x7f4a040cfa00 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f49fc005fd0 tx=0x7f49fc02ab40 comp rx=0 tx=0).stop 2024-09-17T04:05:25.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 -- 172.21.15.129:0/2860963091 shutdown_connections 2024-09-17T04:05:25.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/2860963091 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040cf5f0 0x7f4a040cfa00 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 -- 172.21.15.129:0/2860963091 >> 172.21.15.129:0/2860963091 conn(0x7f4a0409d8d0 msgr2=0x7f4a0408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:25.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 -- 172.21.15.129:0/2860963091 shutdown_connections 2024-09-17T04:05:25.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.486+0000 7f4a0aa00640 1 -- 172.21.15.129:0/2860963091 wait complete. 2024-09-17T04:05:25.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.488+0000 7f4a0aa00640 1 Processor -- start 2024-09-17T04:05:25.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.488+0000 7f4a0aa00640 1 -- start start 2024-09-17T04:05:25.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a040cf5f0 0x7f4a0414eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a041f8970 0x7f4a041f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a040ba040 con 0x7f4a040b3220 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4a040b9ec0 con 0x7f4a041f8970 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4a040d0ae0 con 0x7f4a040cf5f0 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56688/0 (socket says 172.21.15.129:56688) 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 -- 172.21.15.129:0/1255587854 learned_addr learned my addr 172.21.15.129:0/1255587854 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a08c00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a040cf5f0 0x7f4a0414eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a0a000640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a041f8970 0x7f4a041f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 -- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a040cf5f0 msgr2=0x7f4a0414eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a040cf5f0 0x7f4a0414eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 -- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a041f8970 msgr2=0x7f4a041f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a041f8970 0x7f4a041f8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.490+0000 7f4a09600640 1 -- 172.21.15.129:0/1255587854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a041f92c0 con 0x7f4a040b3220 2024-09-17T04:05:25.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a09600640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c3c0 tx=0x7f49f800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:25.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8017070 con 0x7f4a040b3220 2024-09-17T04:05:25.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49f80035f0 con 0x7f4a040b3220 2024-09-17T04:05:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8003a80 con 0x7f4a040b3220 2024-09-17T04:05:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a040cdb40 con 0x7f4a040b3220 2024-09-17T04:05:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.491+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a040ce330 con 0x7f4a040b3220 2024-09-17T04:05:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.492+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a040cfa00 con 0x7f4a040b3220 2024-09-17T04:05:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.494+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f49f801a070 con 0x7f4a040b3220 2024-09-17T04:05:25.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.494+0000 7f4a01600640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 0x7f49e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.495+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 5709+0+0 (secure 0 0 0) 0x7f49f809a090 con 0x7f4a040b3220 2024-09-17T04:05:25.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.495+0000 7f4a08c00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 0x7f49e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.496+0000 7f4a08c00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 0x7f49e407b0a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4a040b90d0 tx=0x7f49f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:25.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.499+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4a041f92c0 con 0x7f4a040b3220 2024-09-17T04:05:25.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.681+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a0406a130 con 0x7f4a040b3220 2024-09-17T04:05:25.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.682+0000 7f4a01600640 1 -- 172.21.15.129:0/1255587854 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) ==== 74+0+16744 (secure 0 0 0) 0x7f49f8062430 con 0x7f4a040b3220 2024-09-17T04:05:25.683 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:25.683 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":217,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:24.702869+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:05:22.611745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-17T04:05:23.660603+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 msgr2=0x7f49e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 0x7f49e407b0a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f4a040b90d0 tx=0x7f49f0009040 comp rx=0 tx=0).stop 2024-09-17T04:05:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 msgr2=0x7f4a040cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f49f800c3c0 tx=0x7f49f800c880 comp rx=0 tx=0).stop 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 shutdown_connections 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49e4078b90 0x7f49e407b0a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a041f8970 0x7f4a041f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a040cf5f0 0x7f4a0414eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 --2- 172.21.15.129:0/1255587854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a040b3220 0x7f4a040cd600 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 >> 172.21.15.129:0/1255587854 conn(0x7f4a0409d8d0 msgr2=0x7f4a04161b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 shutdown_connections 2024-09-17T04:05:25.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.685+0000 7f4a0aa00640 1 -- 172.21.15.129:0/1255587854 wait complete. 2024-09-17T04:05:25.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 -- 172.21.15.129:0/2013819813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 msgr2=0x7f9f8415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 --2- 172.21.15.129:0/2013819813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f8415bca0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c005fd0 tx=0x7f9f6c02a530 comp rx=0 tx=0).stop 2024-09-17T04:05:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 -- 172.21.15.129:0/2013819813 shutdown_connections 2024-09-17T04:05:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 --2- 172.21.15.129:0/2013819813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f8415bca0 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 --2- 172.21.15.129:0/2013819813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 0x7f9f840cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 -- 172.21.15.129:0/2013819813 >> 172.21.15.129:0/2013819813 conn(0x7f9f8406cfc0 msgr2=0x7f9f8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 -- 172.21.15.129:0/2013819813 shutdown_connections 2024-09-17T04:05:25.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.927+0000 7f9f88c00640 1 -- 172.21.15.129:0/2013819813 wait complete. 2024-09-17T04:05:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.929+0000 7f9f88c00640 1 Processor -- start 2024-09-17T04:05:25.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.929+0000 7f9f88c00640 1 -- start start 2024-09-17T04:05:25.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.930+0000 7f9f88c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f88c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 0x7f9f841504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f88c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 0x7f9f8414e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f88c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f840d02c0 con 0x7f9f840cfb00 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f88c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f84152b60 con 0x7f9f8415c1e0 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f88c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9f840d0440 con 0x7f9f841524c0 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56710/0 (socket says 172.21.15.129:56710) 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 -- 172.21.15.129:0/286465556 learned_addr learned my addr 172.21.15.129:0/286465556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f7aa00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 0x7f9f841504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83e00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 0x7f9f8414e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 -- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 msgr2=0x7f9f841504a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 0x7f9f841504a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 -- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 msgr2=0x7f9f8414e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:25.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 0x7f9f8414e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:25.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83400640 1 -- 172.21.15.129:0/286465556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f84145260 con 0x7f9f840cfb00 2024-09-17T04:05:25.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.931+0000 7f9f83e00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 0x7f9f8414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:25.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f83400640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400d700 tx=0x7f9f7400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74017070 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f74005310 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f840aa8f0 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74009820 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.932+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8414fca0 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.933+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f840d0910 con 0x7f9f840cfb00 2024-09-17T04:05:25.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.934+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9f740099c0 con 0x7f9f840cfb00 2024-09-17T04:05:25.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.935+0000 7f9f7be00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:25.935 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.936+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 5709+0+0 (secure 0 0 0) 0x7f9f7409add0 con 0x7f9f840cfb00 2024-09-17T04:05:25.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.936+0000 7f9f7aa00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:25.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.937+0000 7f9f7aa00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c033000 tx=0x7f9f6c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:25.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:25.940+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9f74062bc0 con 0x7f9f840cfb00 2024-09-17T04:05:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.125+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f8406a140 con 0x7f9f840cfb00 2024-09-17T04:05:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.126+0000 7f9f7be00640 1 -- 172.21.15.129:0/286465556 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v217) ==== 74+0+16744 (secure 0 0 0) 0x7f9f7409c020 con 0x7f9f840cfb00 2024-09-17T04:05:26.126 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:26.126 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":217,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:24.702869+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:05:22.611745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-17T04:05:23.660603+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 msgr2=0x7f9f6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c033000 tx=0x7f9f6c004730 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 msgr2=0x7f9f840ad1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f9f7400d700 tx=0x7f9f7400dbc0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 shutdown_connections 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f60078b90 0x7f9f6007b0a0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f8415c1e0 0x7f9f8414e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f841524c0 0x7f9f841504a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 --2- 172.21.15.129:0/286465556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f840cfb00 0x7f9f840ad1f0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.128+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 >> 172.21.15.129:0/286465556 conn(0x7f9f8406cfc0 msgr2=0x7f9f8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.129+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 shutdown_connections 2024-09-17T04:05:26.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.129+0000 7f9f88c00640 1 -- 172.21.15.129:0/286465556 wait complete. 2024-09-17T04:05:26.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:05:26.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 -- 172.21.15.129:0/4135674592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65400b4e00 msgr2=0x7f654006eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:26.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 --2- 172.21.15.129:0/4135674592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65400b4e00 0x7f654006eea0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6530005fd0 tx=0x7f653002a540 comp rx=0 tx=0).stop 2024-09-17T04:05:26.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 -- 172.21.15.129:0/4135674592 shutdown_connections 2024-09-17T04:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 --2- 172.21.15.129:0/4135674592 >> v2:172.21.15.181:3300/0 conn(0x7f65400b2350 0x7f65400b3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 --2- 172.21.15.129:0/4135674592 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f65400b4e00 0x7f654006eea0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 -- 172.21.15.129:0/4135674592 >> 172.21.15.129:0/4135674592 conn(0x7f65400ccc40 msgr2=0x7f65400cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 -- 172.21.15.129:0/4135674592 shutdown_connections 2024-09-17T04:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.381+0000 7f6545600640 1 -- 172.21.15.129:0/4135674592 wait complete. 2024-09-17T04:05:26.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.383+0000 7f6545600640 1 Processor -- start 2024-09-17T04:05:26.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.383+0000 7f6545600640 1 -- start start 2024-09-17T04:05:26.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.384+0000 7f6545600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6540070a00 0x7f6540145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f6545600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65400b4e00 0x7f65401f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f6545600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f6545600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f65400af3a0 con 0x7f6540153570 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f6545600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f65400711e0 con 0x7f65400b4e00 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f6545600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f65400af520 con 0x7f6540070a00 2024-09-17T04:05:26.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56720/0 (socket says 172.21.15.129:56720) 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 -- 172.21.15.129:0/676793688 learned_addr learned my addr 172.21.15.129:0/676793688 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653e000640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65400b4e00 0x7f65401f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653ea00640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6540070a00 0x7f6540145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 -- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6540070a00 msgr2=0x7f6540145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:26.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6540070a00 0x7f6540145740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 -- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65400b4e00 msgr2=0x7f65401f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65400b4e00 0x7f65401f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.385+0000 7f653f400640 1 -- 172.21.15.129:0/676793688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f654009d960 con 0x7f6540153570 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f653f400640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f653800ebf0 tx=0x7f653800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538003b70 con 0x7f6540153570 2024-09-17T04:05:26.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6538003d10 con 0x7f6540153570 2024-09-17T04:05:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65401f9750 con 0x7f6540153570 2024-09-17T04:05:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6538004280 con 0x7f6540153570 2024-09-17T04:05:26.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.386+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f654009cb70 con 0x7f6540153570 2024-09-17T04:05:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.387+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f654006eea0 con 0x7f6540153570 2024-09-17T04:05:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.388+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6538004420 con 0x7f6540153570 2024-09-17T04:05:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.389+0000 7f6536a00640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 0x7f652007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.389+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f653809b890 con 0x7f6540153570 2024-09-17T04:05:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.390+0000 7f653ea00640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 0x7f652007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:26.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.391+0000 7f653ea00640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 0x7f652007b0d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6530033000 tx=0x7f6530004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:26.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.393+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6538063670 con 0x7f6540153570 2024-09-17T04:05:26.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:26.615+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6540156f80 con 0x7f6540153570 2024-09-17T04:05:27.227 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.227+0000 7f6536a00640 1 -- 172.21.15.129:0/676793688 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v207) ==== 118+0+0 (secure 0 0 0) 0x7f65380680a0 con 0x7f6540153570 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 msgr2=0x7f652007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 0x7f652007b0d0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f6530033000 tx=0x7f6530004730 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 msgr2=0x7f65401f8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 secure :-1 s=READY pgs=2370 cs=0 l=1 rev1=1 crypto rx=0x7f653800ebf0 tx=0x7f653800c6a0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 shutdown_connections 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6520078bc0 0x7f652007b0d0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6540153570 0x7f65401f8e50 unknown :-1 s=CLOSED pgs=2370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65400b4e00 0x7f65401f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 --2- 172.21.15.129:0/676793688 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6540070a00 0x7f6540145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 >> 172.21.15.129:0/676793688 conn(0x7f65400ccc40 msgr2=0x7f65400adb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.230+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 shutdown_connections 2024-09-17T04:05:27.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.231+0000 7f6545600640 1 -- 172.21.15.129:0/676793688 wait complete. 2024-09-17T04:05:27.255 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:05:27.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 -- 172.21.15.129:0/3069325888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20580700b0 msgr2=0x7f20580704c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 --2- 172.21.15.129:0/3069325888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20580700b0 0x7f20580704c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f205806f500 tx=0x7f204402a4b0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 -- 172.21.15.129:0/3069325888 shutdown_connections 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 --2- 172.21.15.129:0/3069325888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 0x7f2058073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 --2- 172.21.15.129:0/3069325888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f20580700b0 0x7f20580704c0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 --2- 172.21.15.129:0/3069325888 >> v2:172.21.15.181:3300/0 conn(0x7f205815c7b0 0x7f20580cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 -- 172.21.15.129:0/3069325888 >> 172.21.15.129:0/3069325888 conn(0x7f20580d00b0 msgr2=0x7f20580d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 -- 172.21.15.129:0/3069325888 shutdown_connections 2024-09-17T04:05:27.482 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.481+0000 7f205f000640 1 -- 172.21.15.129:0/3069325888 wait complete. 2024-09-17T04:05:27.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.484+0000 7f205f000640 1 Processor -- start 2024-09-17T04:05:27.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.484+0000 7f205f000640 1 -- start start 2024-09-17T04:05:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 0x7f20580b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 0x7f20581583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 0x7f20581588f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2058160700 con 0x7f205815c7b0 2024-09-17T04:05:27.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2058160580 con 0x7f20580700b0 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.485+0000 7f205f000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2058160880 con 0x7f2058072e10 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 0x7f20581588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205dc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 0x7f20580b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205dc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 0x7f20580b59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:47684/0 (socket says 172.21.15.129:47684) 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205dc00640 1 -- 172.21.15.129:0/2554897024 learned_addr learned my addr 172.21.15.129:0/2554897024 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205d200640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 0x7f20581583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 -- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 msgr2=0x7f20581583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 0x7f20581583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 -- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 msgr2=0x7f20580b59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 0x7f20580b59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 -- 172.21.15.129:0/2554897024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20581568f0 con 0x7f205815c7b0 2024-09-17T04:05:27.487 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.486+0000 7f205e600640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 0x7f20581588f0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f205000ca80 tx=0x7f205000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:27.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.487+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2050007d30 con 0x7f205815c7b0 2024-09-17T04:05:27.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.487+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f205000f450 con 0x7f205815c7b0 2024-09-17T04:05:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.487+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2058156bd0 con 0x7f205815c7b0 2024-09-17T04:05:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.487+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000f880 con 0x7f205815c7b0 2024-09-17T04:05:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.487+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20580c5480 con 0x7f205815c7b0 2024-09-17T04:05:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.488+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20580766d0 con 0x7f205815c7b0 2024-09-17T04:05:27.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.489+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f205000fa20 con 0x7f205815c7b0 2024-09-17T04:05:27.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.490+0000 7f2055600640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 0x7f203807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.490+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f205009b610 con 0x7f205815c7b0 2024-09-17T04:05:27.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.491+0000 7f205dc00640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 0x7f203807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:27.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.491+0000 7f205dc00640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 0x7f203807b0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f204c00abf0 tx=0x7f204c00a300 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:27.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.494+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f20580766d0 con 0x7f205815c7b0 2024-09-17T04:05:27.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:27.713+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2058160da0 con 0x7f205815c7b0 2024-09-17T04:05:28.238 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.238+0000 7f2055600640 1 -- 172.21.15.129:0/2554897024 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v208) ==== 122+0+0 (secure 0 0 0) 0x7f2050014030 con 0x7f205815c7b0 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 msgr2=0x7f203807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 0x7f203807b0d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f204c00abf0 tx=0x7f204c00a300 comp rx=0 tx=0).stop 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 msgr2=0x7f20581588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 0x7f20581588f0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f205000ca80 tx=0x7f205000cf40 comp rx=0 tx=0).stop 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 shutdown_connections 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2038078bc0 0x7f203807b0d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f205815c7b0 0x7f20581588f0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2058072e10 0x7f20581583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 --2- 172.21.15.129:0/2554897024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f20580700b0 0x7f20580b59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 >> 172.21.15.129:0/2554897024 conn(0x7f20580d00b0 msgr2=0x7f20580ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 shutdown_connections 2024-09-17T04:05:28.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.241+0000 7f205f000640 1 -- 172.21.15.129:0/2554897024 wait complete. 2024-09-17T04:05:28.268 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:05:28.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:28.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 -- 172.21.15.129:0/787355276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c300705a0 msgr2=0x7f7c300b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 --2- 172.21.15.129:0/787355276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c300705a0 0x7f7c300b4140 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f7c30072080 tx=0x7f7c2402a560 comp rx=0 tx=0).stop 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 -- 172.21.15.129:0/787355276 shutdown_connections 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 --2- 172.21.15.129:0/787355276 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c3006f2e0 0x7f7c300cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 --2- 172.21.15.129:0/787355276 >> v2:172.21.15.181:3300/0 conn(0x7f7c3006eed0 0x7f7c300ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 --2- 172.21.15.129:0/787355276 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c300705a0 0x7f7c300b4140 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 -- 172.21.15.129:0/787355276 >> 172.21.15.129:0/787355276 conn(0x7f7c3006d410 msgr2=0x7f7c300b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 -- 172.21.15.129:0/787355276 shutdown_connections 2024-09-17T04:05:28.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.488+0000 7f7c38600640 1 -- 172.21.15.129:0/787355276 wait complete. 2024-09-17T04:05:28.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.490+0000 7f7c38600640 1 Processor -- start 2024-09-17T04:05:28.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.490+0000 7f7c38600640 1 -- start start 2024-09-17T04:05:28.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c3006f2e0 0x7f7c301f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c300705a0 0x7f7c300b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c30160b40 con 0x7f7c3006eed0 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c301609c0 con 0x7f7c3006f2e0 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c38600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7c30160cc0 con 0x7f7c300705a0 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56758/0 (socket says 172.21.15.129:56758) 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 -- 172.21.15.129:0/4141945451 learned_addr learned my addr 172.21.15.129:0/4141945451 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 -- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c300705a0 msgr2=0x7f7c300b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c300705a0 0x7f7c300b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 -- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c3006f2e0 msgr2=0x7f7c301f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c3006f2e0 0x7f7c301f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.492+0000 7f7c35c00640 1 -- 172.21.15.129:0/4141945451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c301f5980 con 0x7f7c3006eed0 2024-09-17T04:05:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c35c00640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f7c24034040 tx=0x7f7c240038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2403e030 con 0x7f7c3006eed0 2024-09-17T04:05:28.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c24003ca0 con 0x7f7c3006eed0 2024-09-17T04:05:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c24035040 con 0x7f7c3006eed0 2024-09-17T04:05:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c301f95a0 con 0x7f7c3006eed0 2024-09-17T04:05:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.493+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c301f9b60 con 0x7f7c3006eed0 2024-09-17T04:05:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.494+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c30161250 con 0x7f7c3006eed0 2024-09-17T04:05:28.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.495+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7c24041030 con 0x7f7c3006eed0 2024-09-17T04:05:28.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.496+0000 7f7c2d600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 0x7f7c1007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.496+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f7c240bb440 con 0x7f7c3006eed0 2024-09-17T04:05:28.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.497+0000 7f7c35200640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 0x7f7c1007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.497+0000 7f7c35200640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 0x7f7c1007b0a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7c3015fde0 tx=0x7f7c1c0094f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:28.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.501+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c24083870 con 0x7f7c3006eed0 2024-09-17T04:05:28.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.720+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c3006a140 con 0x7f7c3006eed0 2024-09-17T04:05:28.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.721+0000 7f7c2d600640 1 -- 172.21.15.129:0/4141945451 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5249 (secure 0 0 0) 0x7f7c24083170 con 0x7f7c3006eed0 2024-09-17T04:05:28.721 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:28.721 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":208,"btime":"2024-09-17T04:05:28:229503+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2764153925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2764153925},{"type":"v1","addr":"172.21.15.181:6837","nonce":2764153925}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202},{"gid":10231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2457838594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2457838594},{"type":"v1","addr":"172.21.15.181:6836","nonce":2457838594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202},{"gid":10237,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3249193545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3249193545},{"type":"v1","addr":"172.21.15.129:6834","nonce":3249193545}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:27.714822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"c","rank":0,"incarnation":204,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/711019236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":711019236},{"type":"v1","addr":"172.21.15.129:6835","nonce":711019236}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":16}]} 2024-09-17T04:05:28.721 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 208 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 msgr2=0x7f7c1007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 0x7f7c1007b0a0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f7c3015fde0 tx=0x7f7c1c0094f0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 msgr2=0x7f7c300ca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f7c24034040 tx=0x7f7c240038a0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 shutdown_connections 2024-09-17T04:05:28.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c10078b90 0x7f7c1007b0a0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c300705a0 0x7f7c300b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c3006f2e0 0x7f7c301f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 --2- 172.21.15.129:0/4141945451 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c3006eed0 0x7f7c300ca2e0 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 >> 172.21.15.129:0/4141945451 conn(0x7f7c3006d410 msgr2=0x7f7c30073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 shutdown_connections 2024-09-17T04:05:28.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.723+0000 7f7c38600640 1 -- 172.21.15.129:0/4141945451 wait complete. 2024-09-17T04:05:28.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 -- 172.21.15.129:0/346133018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 msgr2=0x7f39280cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 --2- 172.21.15.129:0/346133018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39280cf6c0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f3914005fd0 tx=0x7f391402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 -- 172.21.15.129:0/346133018 shutdown_connections 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 --2- 172.21.15.129:0/346133018 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 0x7f39280cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 --2- 172.21.15.129:0/346133018 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39280cf6c0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 -- 172.21.15.129:0/346133018 >> 172.21.15.129:0/346133018 conn(0x7f392806d410 msgr2=0x7f39280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:28.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 -- 172.21.15.129:0/346133018 shutdown_connections 2024-09-17T04:05:28.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.966+0000 7f392e200640 1 -- 172.21.15.129:0/346133018 wait complete. 2024-09-17T04:05:28.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.969+0000 7f392e200640 1 Processor -- start 2024-09-17T04:05:28.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.969+0000 7f392e200640 1 -- start start 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3928099570 0x7f39280985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 0x7f3928098b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3928160ba0 con 0x7f39280cf2b0 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3928160a20 con 0x7f3928099570 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f392e200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3928160d20 con 0x7f39280ccc80 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f3927e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.970+0000 7f3927e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56776/0 (socket says 172.21.15.129:56776) 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 -- 172.21.15.129:0/2789564037 learned_addr learned my addr 172.21.15.129:0/2789564037 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927400640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3928099570 0x7f39280985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3926a00640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 0x7f3928098b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 -- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 msgr2=0x7f3928098b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 0x7f3928098b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 -- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3928099570 msgr2=0x7f39280985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3928099570 0x7f39280985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 -- 172.21.15.129:0/2789564037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39281e6340 con 0x7f39280cf2b0 2024-09-17T04:05:28.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f3927e00640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f392000e960 tx=0x7f392000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:28.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.971+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920004d20 con 0x7f39280cf2b0 2024-09-17T04:05:28.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.972+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f392000cac0 con 0x7f39280cf2b0 2024-09-17T04:05:28.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.972+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39281e6620 con 0x7f39280cf2b0 2024-09-17T04:05:28.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.972+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920005020 con 0x7f39280cf2b0 2024-09-17T04:05:28.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.972+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39280b0f30 con 0x7f39280cf2b0 2024-09-17T04:05:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.973+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39280b1210 con 0x7f39280cf2b0 2024-09-17T04:05:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.974+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f39200051c0 con 0x7f39280cf2b0 2024-09-17T04:05:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.975+0000 7f391f400640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 0x7f390807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.975+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f392009b580 con 0x7f39280cf2b0 2024-09-17T04:05:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.976+0000 7f3927400640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 0x7f390807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:28.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.976+0000 7f3927400640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 0x7f390807b0a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3918001720 tx=0x7f3918009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:28.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:28.979+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3920016030 con 0x7f39280cf2b0 2024-09-17T04:05:29.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.163+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39281611d0 con 0x7f39280cf2b0 2024-09-17T04:05:29.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.164+0000 7f391f400640 1 -- 172.21.15.129:0/2789564037 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) ==== 74+0+16731 (secure 0 0 0) 0x7f3920063360 con 0x7f39280cf2b0 2024-09-17T04:05:29.165 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:29.165 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":218,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:26.214772+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:05:22.611745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-17T04:05:23.660603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:29.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 msgr2=0x7f390807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 0x7f390807b0a0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f3918001720 tx=0x7f3918009040 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 msgr2=0x7f39281f3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f392000e960 tx=0x7f392000ee20 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 shutdown_connections 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3908078b90 0x7f390807b0a0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f39280cf2b0 0x7f39281f3c10 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f39280ccc80 0x7f3928098b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 --2- 172.21.15.129:0/2789564037 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3928099570 0x7f39280985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 >> 172.21.15.129:0/2789564037 conn(0x7f392806d410 msgr2=0x7f3928070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 shutdown_connections 2024-09-17T04:05:29.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.167+0000 7f392e200640 1 -- 172.21.15.129:0/2789564037 wait complete. 2024-09-17T04:05:29.194 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:05:29.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 -- 172.21.15.129:0/2408943057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580b4140 msgr2=0x7fe5580ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 --2- 172.21.15.129:0/2408943057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580b4140 0x7fe5580ccc80 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7fe558072080 tx=0x7fe54402a530 comp rx=0 tx=0).stop 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 -- 172.21.15.129:0/2408943057 shutdown_connections 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 --2- 172.21.15.129:0/2408943057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580cd1c0 0x7fe5581609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 --2- 172.21.15.129:0/2408943057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580b4140 0x7fe5580ccc80 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 -- 172.21.15.129:0/2408943057 >> 172.21.15.129:0/2408943057 conn(0x7fe55806d410 msgr2=0x7fe5580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:29.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 -- 172.21.15.129:0/2408943057 shutdown_connections 2024-09-17T04:05:29.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.415+0000 7fe55f400640 1 -- 172.21.15.129:0/2408943057 wait complete. 2024-09-17T04:05:29.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.417+0000 7fe55f400640 1 Processor -- start 2024-09-17T04:05:29.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.418+0000 7fe55f400640 1 -- start start 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5580705a0 0x7fe558154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580b4140 0x7fe5581f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe558160260 con 0x7fe5580cd1c0 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe5581600e0 con 0x7fe5580705a0 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55f400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe558160f00 con 0x7fe5580b4140 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56792/0 (socket says 172.21.15.129:56792) 2024-09-17T04:05:29.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55d400640 1 -- 172.21.15.129:0/858764675 learned_addr learned my addr 172.21.15.129:0/858764675 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.419+0000 7fe55ca00640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5580705a0 0x7fe558154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe557e00640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580b4140 0x7fe5581f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 -- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580b4140 msgr2=0x7fe5581f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580b4140 0x7fe5581f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 -- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5580705a0 msgr2=0x7fe558154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5580705a0 0x7fe558154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 -- 172.21.15.129:0/858764675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe558154580 con 0x7fe5580cd1c0 2024-09-17T04:05:29.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55d400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fe55000b410 tx=0x7fe55000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe550015020 con 0x7fe5580cd1c0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe550011920 con 0x7fe5580cd1c0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe550011d50 con 0x7fe5580cd1c0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.420+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5580add50 con 0x7fe5580cd1c0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.421+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5581f7240 con 0x7fe5580cd1c0 2024-09-17T04:05:29.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.422+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5580ccc80 con 0x7fe5580cd1c0 2024-09-17T04:05:29.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.423+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe550018020 con 0x7fe5580cd1c0 2024-09-17T04:05:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.424+0000 7fe554c00640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 0x7fe53807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.424+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fe55009aae0 con 0x7fe5580cd1c0 2024-09-17T04:05:29.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.424+0000 7fe55ca00640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 0x7fe53807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.425+0000 7fe55ca00640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 0x7fe53807b0d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b450 tx=0x7fe54800d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:29.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.428+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe55001f020 con 0x7fe5580cd1c0 2024-09-17T04:05:29.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.610+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe55806a140 con 0x7fe538078bc0 2024-09-17T04:05:29.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.616+0000 7fe554c00640 1 -- 172.21.15.129:0/858764675 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248522 (secure 0 0 0) 0x7fe55806a140 con 0x7fe538078bc0 2024-09-17T04:05:29.619 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:29.621 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 msgr2=0x7fe53807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 0x7fe53807b0d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe54800b450 tx=0x7fe54800d040 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 msgr2=0x7fe5581f8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fe55000b410 tx=0x7fe55000b8d0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 shutdown_connections 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe538078bc0 0x7fe53807b0d0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe5580cd1c0 0x7fe5581f8540 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe5580b4140 0x7fe5581f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 --2- 172.21.15.129:0/858764675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5580705a0 0x7fe558154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 >> 172.21.15.129:0/858764675 conn(0x7fe55806d410 msgr2=0x7fe558072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.620+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 shutdown_connections 2024-09-17T04:05:29.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.621+0000 7fe55f400640 1 -- 172.21.15.129:0/858764675 wait complete. 2024-09-17T04:05:29.643 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-09-17T04:05:28.569229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402512,"kb_used_data":22492,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754572208,"statfs":{"total":773094113280,"available":772681940992,"internally_reserved":0,"allocated":23031808,"data_stored":10805549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.376500"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212732+0000","last_change":"2024-09-17T04:05:24.212732+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212732+0000","last_undegraded":"2024-09-17T04:05:24.212732+0000","last_fullsized":"2024-09-17T04:05:24.212732+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.216070+0000","last_change":"2024-09-17T04:05:24.216070+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.216070+0000","last_undegraded":"2024-09-17T04:05:24.216070+0000","last_fullsized":"2024-09-17T04:05:24.216070+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213097+0000","last_change":"2024-09-17T04:05:24.213097+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213097+0000","last_undegraded":"2024-09-17T04:05:24.213097+0000","last_fullsized":"2024-09-17T04:05:24.213097+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214496+0000","last_change":"2024-09-17T04:05:24.214496+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214496+0000","last_undegraded":"2024-09-17T04:05:24.214496+0000","last_fullsized":"2024-09-17T04:05:24.214496+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209881+0000","last_change":"2024-09-17T04:05:24.209881+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209881+0000","last_undegraded":"2024-09-17T04:05:24.209881+0000","last_fullsized":"2024-09-17T04:05:24.209881+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214316+0000","last_change":"2024-09-17T04:05:24.214316+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214316+0000","last_undegraded":"2024-09-17T04:05:24.214316+0000","last_fullsized":"2024-09-17T04:05:24.214316+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500029+0000","last_change":"2024-09-17T04:05:25.217581+0000","last_active":"2024-09-17T04:05:26.500029+0000","last_peered":"2024-09-17T04:05:26.500029+0000","last_clean":"2024-09-17T04:05:26.500029+0000","last_became_active":"2024-09-17T04:05:25.217074+0000","last_became_peered":"2024-09-17T04:05:25.217074+0000","last_unstale":"2024-09-17T04:05:26.500029+0000","last_undegraded":"2024-09-17T04:05:26.500029+0000","last_fullsized":"2024-09-17T04:05:26.500029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:30.412641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212075+0000","last_change":"2024-09-17T04:05:24.212075+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212075+0000","last_undegraded":"2024-09-17T04:05:24.212075+0000","last_fullsized":"2024-09-17T04:05:24.212075+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425646+0000","last_change":"2024-09-17T04:05:25.224713+0000","last_active":"2024-09-17T04:05:26.425646+0000","last_peered":"2024-09-17T04:05:26.425646+0000","last_clean":"2024-09-17T04:05:26.425646+0000","last_became_active":"2024-09-17T04:05:25.224391+0000","last_became_peered":"2024-09-17T04:05:25.224391+0000","last_unstale":"2024-09-17T04:05:26.425646+0000","last_undegraded":"2024-09-17T04:05:26.425646+0000","last_fullsized":"2024-09-17T04:05:26.425646+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:16.385982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500080+0000","last_change":"2024-09-17T04:05:25.226863+0000","last_active":"2024-09-17T04:05:26.500080+0000","last_peered":"2024-09-17T04:05:26.500080+0000","last_clean":"2024-09-17T04:05:26.500080+0000","last_became_active":"2024-09-17T04:05:25.226508+0000","last_became_peered":"2024-09-17T04:05:25.226508+0000","last_unstale":"2024-09-17T04:05:26.500080+0000","last_undegraded":"2024-09-17T04:05:26.500080+0000","last_fullsized":"2024-09-17T04:05:26.500080+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:56.644223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500120+0000","last_change":"2024-09-17T04:05:25.228022+0000","last_active":"2024-09-17T04:05:26.500120+0000","last_peered":"2024-09-17T04:05:26.500120+0000","last_clean":"2024-09-17T04:05:26.500120+0000","last_became_active":"2024-09-17T04:05:25.227659+0000","last_became_peered":"2024-09-17T04:05:25.227659+0000","last_unstale":"2024-09-17T04:05:26.500120+0000","last_undegraded":"2024-09-17T04:05:26.500120+0000","last_fullsized":"2024-09-17T04:05:26.500120+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:51.022680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214060+0000","last_change":"2024-09-17T04:05:24.214060+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214060+0000","last_undegraded":"2024-09-17T04:05:24.214060+0000","last_fullsized":"2024-09-17T04:05:24.214060+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212465+0000","last_change":"2024-09-17T04:05:24.212465+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212465+0000","last_undegraded":"2024-09-17T04:05:24.212465+0000","last_fullsized":"2024-09-17T04:05:24.212465+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215425+0000","last_change":"2024-09-17T04:05:24.215425+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215425+0000","last_undegraded":"2024-09-17T04:05:24.215425+0000","last_fullsized":"2024-09-17T04:05:24.215425+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.217791+0000","last_change":"2024-09-17T04:05:24.217791+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.217791+0000","last_undegraded":"2024-09-17T04:05:24.217791+0000","last_fullsized":"2024-09-17T04:05:24.217791+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211378+0000","last_change":"2024-09-17T04:05:24.211378+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211378+0000","last_undegraded":"2024-09-17T04:05:24.211378+0000","last_fullsized":"2024-09-17T04:05:24.211378+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211382+0000","last_change":"2024-09-17T04:05:24.211382+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211382+0000","last_undegraded":"2024-09-17T04:05:24.211382+0000","last_fullsized":"2024-09-17T04:05:24.211382+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500764+0000","last_change":"2024-09-17T04:05:25.217557+0000","last_active":"2024-09-17T04:05:26.500764+0000","last_peered":"2024-09-17T04:05:26.500764+0000","last_clean":"2024-09-17T04:05:26.500764+0000","last_became_active":"2024-09-17T04:05:25.217163+0000","last_became_peered":"2024-09-17T04:05:25.217163+0000","last_unstale":"2024-09-17T04:05:26.500764+0000","last_undegraded":"2024-09-17T04:05:26.500764+0000","last_fullsized":"2024-09-17T04:05:26.500764+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:28.881707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500523+0000","last_change":"2024-09-17T04:05:25.219655+0000","last_active":"2024-09-17T04:05:26.500523+0000","last_peered":"2024-09-17T04:05:26.500523+0000","last_clean":"2024-09-17T04:05:26.500523+0000","last_became_active":"2024-09-17T04:05:25.219345+0000","last_became_peered":"2024-09-17T04:05:25.219345+0000","last_unstale":"2024-09-17T04:05:26.500523+0000","last_undegraded":"2024-09-17T04:05:26.500523+0000","last_fullsized":"2024-09-17T04:05:26.500523+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:59.868383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.217364+0000","last_change":"2024-09-17T04:05:24.217364+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.217364+0000","last_undegraded":"2024-09-17T04:05:24.217364+0000","last_fullsized":"2024-09-17T04:05:24.217364+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425500+0000","last_change":"2024-09-17T04:05:25.231383+0000","last_active":"2024-09-17T04:05:26.425500+0000","last_peered":"2024-09-17T04:05:26.425500+0000","last_clean":"2024-09-17T04:05:26.425500+0000","last_became_active":"2024-09-17T04:05:25.231058+0000","last_became_peered":"2024-09-17T04:05:25.231058+0000","last_unstale":"2024-09-17T04:05:26.425500+0000","last_undegraded":"2024-09-17T04:05:26.425500+0000","last_fullsized":"2024-09-17T04:05:26.425500+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.757807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425336+0000","last_change":"2024-09-17T04:05:25.223284+0000","last_active":"2024-09-17T04:05:26.425336+0000","last_peered":"2024-09-17T04:05:26.425336+0000","last_clean":"2024-09-17T04:05:26.425336+0000","last_became_active":"2024-09-17T04:05:25.221020+0000","last_became_peered":"2024-09-17T04:05:25.221020+0000","last_unstale":"2024-09-17T04:05:26.425336+0000","last_undegraded":"2024-09-17T04:05:26.425336+0000","last_fullsized":"2024-09-17T04:05:26.425336+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:09.253541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500358+0000","last_change":"2024-09-17T04:05:25.226879+0000","last_active":"2024-09-17T04:05:26.500358+0000","last_peered":"2024-09-17T04:05:26.500358+0000","last_clean":"2024-09-17T04:05:26.500358+0000","last_became_active":"2024-09-17T04:05:25.226571+0000","last_became_peered":"2024-09-17T04:05:25.226571+0000","last_unstale":"2024-09-17T04:05:26.500358+0000","last_undegraded":"2024-09-17T04:05:26.500358+0000","last_fullsized":"2024-09-17T04:05:26.500358+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:45.293995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212257+0000","last_change":"2024-09-17T04:05:24.212257+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212257+0000","last_undegraded":"2024-09-17T04:05:24.212257+0000","last_fullsized":"2024-09-17T04:05:24.212257+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.216822+0000","last_change":"2024-09-17T04:05:24.216822+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.216822+0000","last_undegraded":"2024-09-17T04:05:24.216822+0000","last_fullsized":"2024-09-17T04:05:24.216822+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212029+0000","last_change":"2024-09-17T04:05:24.212029+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212029+0000","last_undegraded":"2024-09-17T04:05:24.212029+0000","last_fullsized":"2024-09-17T04:05:24.212029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500441+0000","last_change":"2024-09-17T04:05:25.230257+0000","last_active":"2024-09-17T04:05:26.500441+0000","last_peered":"2024-09-17T04:05:26.500441+0000","last_clean":"2024-09-17T04:05:26.500441+0000","last_became_active":"2024-09-17T04:05:25.229043+0000","last_became_peered":"2024-09-17T04:05:25.229043+0000","last_unstale":"2024-09-17T04:05:26.500441+0000","last_undegraded":"2024-09-17T04:05:26.500441+0000","last_fullsized":"2024-09-17T04:05:26.500441+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:03.038463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.218979+0000","last_change":"2024-09-17T04:05:24.218979+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.218979+0000","last_undegraded":"2024-09-17T04:05:24.218979+0000","last_fullsized":"2024-09-17T04:05:24.218979+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210651+0000","last_change":"2024-09-17T04:05:24.210651+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210651+0000","last_undegraded":"2024-09-17T04:05:24.210651+0000","last_fullsized":"2024-09-17T04:05:24.210651+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214973+0000","last_change":"2024-09-17T04:05:24.214973+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214973+0000","last_undegraded":"2024-09-17T04:05:24.214973+0000","last_fullsized":"2024-09-17T04:05:24.214973+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212337+0000","last_change":"2024-09-17T04:05:24.212337+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212337+0000","last_undegraded":"2024-09-17T04:05:24.212337+0000","last_fullsized":"2024-09-17T04:05:24.212337+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500413+0000","last_change":"2024-09-17T04:05:25.231448+0000","last_active":"2024-09-17T04:05:26.500413+0000","last_peered":"2024-09-17T04:05:26.500413+0000","last_clean":"2024-09-17T04:05:26.500413+0000","last_became_active":"2024-09-17T04:05:25.229864+0000","last_became_peered":"2024-09-17T04:05:25.229864+0000","last_unstale":"2024-09-17T04:05:26.500413+0000","last_undegraded":"2024-09-17T04:05:26.500413+0000","last_fullsized":"2024-09-17T04:05:26.500413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:31.295888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227395+0000","last_change":"2024-09-17T04:05:24.220787+0000","last_active":"2024-09-17T04:05:24.227395+0000","last_peered":"2024-09-17T04:05:24.227395+0000","last_clean":"2024-09-17T04:05:24.227395+0000","last_became_active":"2024-09-17T04:05:24.220468+0000","last_became_peered":"2024-09-17T04:05:24.220468+0000","last_unstale":"2024-09-17T04:05:24.227395+0000","last_undegraded":"2024-09-17T04:05:24.227395+0000","last_fullsized":"2024-09-17T04:05:24.227395+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:17.532872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214787+0000","last_change":"2024-09-17T04:05:24.214787+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214787+0000","last_undegraded":"2024-09-17T04:05:24.214787+0000","last_fullsized":"2024-09-17T04:05:24.214787+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.238859+0000","last_change":"2024-09-17T04:05:24.231055+0000","last_active":"2024-09-17T04:05:24.238859+0000","last_peered":"2024-09-17T04:05:24.238859+0000","last_clean":"2024-09-17T04:05:24.238859+0000","last_became_active":"2024-09-17T04:05:24.222063+0000","last_became_peered":"2024-09-17T04:05:24.222063+0000","last_unstale":"2024-09-17T04:05:24.238859+0000","last_undegraded":"2024-09-17T04:05:24.238859+0000","last_fullsized":"2024-09-17T04:05:24.238859+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:56.085206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500629+0000","last_change":"2024-09-17T04:05:25.227705+0000","last_active":"2024-09-17T04:05:26.500629+0000","last_peered":"2024-09-17T04:05:26.500629+0000","last_clean":"2024-09-17T04:05:26.500629+0000","last_became_active":"2024-09-17T04:05:25.223453+0000","last_became_peered":"2024-09-17T04:05:25.223453+0000","last_unstale":"2024-09-17T04:05:26.500629+0000","last_undegraded":"2024-09-17T04:05:26.500629+0000","last_fullsized":"2024-09-17T04:05:26.500629+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:26.267555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233033+0000","last_change":"2024-09-17T04:05:24.228243+0000","last_active":"2024-09-17T04:05:24.233033+0000","last_peered":"2024-09-17T04:05:24.233033+0000","last_clean":"2024-09-17T04:05:24.233033+0000","last_became_active":"2024-09-17T04:05:24.227852+0000","last_became_peered":"2024-09-17T04:05:24.227852+0000","last_unstale":"2024-09-17T04:05:24.233033+0000","last_undegraded":"2024-09-17T04:05:24.233033+0000","last_fullsized":"2024-09-17T04:05:24.233033+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:14:19.206049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213736+0000","last_change":"2024-09-17T04:05:24.213736+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213736+0000","last_undegraded":"2024-09-17T04:05:24.213736+0000","last_fullsized":"2024-09-17T04:05:24.213736+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232937+0000","last_change":"2024-09-17T04:05:24.226740+0000","last_active":"2024-09-17T04:05:24.232937+0000","last_peered":"2024-09-17T04:05:24.232937+0000","last_clean":"2024-09-17T04:05:24.232937+0000","last_became_active":"2024-09-17T04:05:24.226406+0000","last_became_peered":"2024-09-17T04:05:24.226406+0000","last_unstale":"2024-09-17T04:05:24.232937+0000","last_undegraded":"2024-09-17T04:05:24.232937+0000","last_fullsized":"2024-09-17T04:05:24.232937+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:07.670673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425319+0000","last_change":"2024-09-17T04:05:25.232667+0000","last_active":"2024-09-17T04:05:26.425319+0000","last_peered":"2024-09-17T04:05:26.425319+0000","last_clean":"2024-09-17T04:05:26.425319+0000","last_became_active":"2024-09-17T04:05:25.232379+0000","last_became_peered":"2024-09-17T04:05:25.232379+0000","last_unstale":"2024-09-17T04:05:26.425319+0000","last_undegraded":"2024-09-17T04:05:26.425319+0000","last_fullsized":"2024-09-17T04:05:26.425319+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:20.607889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499657+0000","last_change":"2024-09-17T04:05:24.224011+0000","last_active":"2024-09-17T04:05:26.499657+0000","last_peered":"2024-09-17T04:05:26.499657+0000","last_clean":"2024-09-17T04:05:26.499657+0000","last_became_active":"2024-09-17T04:05:24.223670+0000","last_became_peered":"2024-09-17T04:05:24.223670+0000","last_unstale":"2024-09-17T04:05:26.499657+0000","last_undegraded":"2024-09-17T04:05:26.499657+0000","last_fullsized":"2024-09-17T04:05:26.499657+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.853028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215489+0000","last_change":"2024-09-17T04:05:24.215489+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215489+0000","last_undegraded":"2024-09-17T04:05:24.215489+0000","last_fullsized":"2024-09-17T04:05:24.215489+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227346+0000","last_change":"2024-09-17T04:05:24.221715+0000","last_active":"2024-09-17T04:05:24.227346+0000","last_peered":"2024-09-17T04:05:24.227346+0000","last_clean":"2024-09-17T04:05:24.227346+0000","last_became_active":"2024-09-17T04:05:24.221380+0000","last_became_peered":"2024-09-17T04:05:24.221380+0000","last_unstale":"2024-09-17T04:05:24.227346+0000","last_undegraded":"2024-09-17T04:05:24.227346+0000","last_fullsized":"2024-09-17T04:05:24.227346+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:02.226834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210326+0000","last_change":"2024-09-17T04:05:24.210326+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210326+0000","last_undegraded":"2024-09-17T04:05:24.210326+0000","last_fullsized":"2024-09-17T04:05:24.210326+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.235031+0000","last_change":"2024-09-17T04:05:24.234352+0000","last_active":"2024-09-17T04:05:24.235031+0000","last_peered":"2024-09-17T04:05:24.235031+0000","last_clean":"2024-09-17T04:05:24.235031+0000","last_became_active":"2024-09-17T04:05:24.234071+0000","last_became_peered":"2024-09-17T04:05:24.234071+0000","last_unstale":"2024-09-17T04:05:24.235031+0000","last_undegraded":"2024-09-17T04:05:24.235031+0000","last_fullsized":"2024-09-17T04:05:24.235031+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:18.444880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500626+0000","last_change":"2024-09-17T04:05:25.219129+0000","last_active":"2024-09-17T04:05:26.500626+0000","last_peered":"2024-09-17T04:05:26.500626+0000","last_clean":"2024-09-17T04:05:26.500626+0000","last_became_active":"2024-09-17T04:05:25.217644+0000","last_became_peered":"2024-09-17T04:05:25.217644+0000","last_unstale":"2024-09-17T04:05:26.500626+0000","last_undegraded":"2024-09-17T04:05:26.500626+0000","last_fullsized":"2024-09-17T04:05:26.500626+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:52.028296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499665+0000","last_change":"2024-09-17T04:05:24.224568+0000","last_active":"2024-09-17T04:05:26.499665+0000","last_peered":"2024-09-17T04:05:26.499665+0000","last_clean":"2024-09-17T04:05:26.499665+0000","last_became_active":"2024-09-17T04:05:24.223638+0000","last_became_peered":"2024-09-17T04:05:24.223638+0000","last_unstale":"2024-09-17T04:05:26.499665+0000","last_undegraded":"2024-09-17T04:05:26.499665+0000","last_fullsized":"2024-09-17T04:05:26.499665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:49.701292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500661+0000","last_change":"2024-09-17T04:05:25.218519+0000","last_active":"2024-09-17T04:05:26.500661+0000","last_peered":"2024-09-17T04:05:26.500661+0000","last_clean":"2024-09-17T04:05:26.500661+0000","last_became_active":"2024-09-17T04:05:25.217254+0000","last_became_peered":"2024-09-17T04:05:25.217254+0000","last_unstale":"2024-09-17T04:05:26.500661+0000","last_undegraded":"2024-09-17T04:05:26.500661+0000","last_fullsized":"2024-09-17T04:05:26.500661+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:49.680789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227870+0000","last_change":"2024-09-17T04:05:24.223519+0000","last_active":"2024-09-17T04:05:24.227870+0000","last_peered":"2024-09-17T04:05:24.227870+0000","last_clean":"2024-09-17T04:05:24.227870+0000","last_became_active":"2024-09-17T04:05:24.223231+0000","last_became_peered":"2024-09-17T04:05:24.223231+0000","last_unstale":"2024-09-17T04:05:24.227870+0000","last_undegraded":"2024-09-17T04:05:24.227870+0000","last_fullsized":"2024-09-17T04:05:24.227870+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:12.017426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209088+0000","last_change":"2024-09-17T04:05:24.209088+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209088+0000","last_undegraded":"2024-09-17T04:05:24.209088+0000","last_fullsized":"2024-09-17T04:05:24.209088+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233132+0000","last_change":"2024-09-17T04:05:24.226942+0000","last_active":"2024-09-17T04:05:24.233132+0000","last_peered":"2024-09-17T04:05:24.233132+0000","last_clean":"2024-09-17T04:05:24.233132+0000","last_became_active":"2024-09-17T04:05:24.226597+0000","last_became_peered":"2024-09-17T04:05:24.226597+0000","last_unstale":"2024-09-17T04:05:24.233132+0000","last_undegraded":"2024-09-17T04:05:24.233132+0000","last_fullsized":"2024-09-17T04:05:24.233132+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:47:19.422554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.220589+0000","last_change":"2024-09-17T04:05:24.220589+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.220589+0000","last_undegraded":"2024-09-17T04:05:24.220589+0000","last_fullsized":"2024-09-17T04:05:24.220589+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233169+0000","last_change":"2024-09-17T04:05:24.230056+0000","last_active":"2024-09-17T04:05:24.233169+0000","last_peered":"2024-09-17T04:05:24.233169+0000","last_clean":"2024-09-17T04:05:24.233169+0000","last_became_active":"2024-09-17T04:05:24.229748+0000","last_became_peered":"2024-09-17T04:05:24.229748+0000","last_unstale":"2024-09-17T04:05:24.233169+0000","last_undegraded":"2024-09-17T04:05:24.233169+0000","last_fullsized":"2024-09-17T04:05:24.233169+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:29.381428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210187+0000","last_change":"2024-09-17T04:05:24.210187+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210187+0000","last_undegraded":"2024-09-17T04:05:24.210187+0000","last_fullsized":"2024-09-17T04:05:24.210187+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.229675+0000","last_change":"2024-09-17T04:05:24.225084+0000","last_active":"2024-09-17T04:05:24.229675+0000","last_peered":"2024-09-17T04:05:24.229675+0000","last_clean":"2024-09-17T04:05:24.229675+0000","last_became_active":"2024-09-17T04:05:24.224781+0000","last_became_peered":"2024-09-17T04:05:24.224781+0000","last_unstale":"2024-09-17T04:05:24.229675+0000","last_undegraded":"2024-09-17T04:05:24.229675+0000","last_fullsized":"2024-09-17T04:05:24.229675+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:24.613188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209032+0000","last_change":"2024-09-17T04:05:24.209032+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209032+0000","last_undegraded":"2024-09-17T04:05:24.209032+0000","last_fullsized":"2024-09-17T04:05:24.209032+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227339+0000","last_change":"2024-09-17T04:05:24.223126+0000","last_active":"2024-09-17T04:05:24.227339+0000","last_peered":"2024-09-17T04:05:24.227339+0000","last_clean":"2024-09-17T04:05:24.227339+0000","last_became_active":"2024-09-17T04:05:24.221114+0000","last_became_peered":"2024-09-17T04:05:24.221114+0000","last_unstale":"2024-09-17T04:05:24.227339+0000","last_undegraded":"2024-09-17T04:05:24.227339+0000","last_fullsized":"2024-09-17T04:05:24.227339+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:29:18.283661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215086+0000","last_change":"2024-09-17T04:05:24.215086+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215086+0000","last_undegraded":"2024-09-17T04:05:24.215086+0000","last_fullsized":"2024-09-17T04:05:24.215086+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425400+0000","last_change":"2024-09-17T04:05:24.223560+0000","last_active":"2024-09-17T04:05:26.425400+0000","last_peered":"2024-09-17T04:05:26.425400+0000","last_clean":"2024-09-17T04:05:26.425400+0000","last_became_active":"2024-09-17T04:05:24.223302+0000","last_became_peered":"2024-09-17T04:05:24.223302+0000","last_unstale":"2024-09-17T04:05:26.425400+0000","last_undegraded":"2024-09-17T04:05:26.425400+0000","last_fullsized":"2024-09-17T04:05:26.425400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:25.259928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209309+0000","last_change":"2024-09-17T04:05:24.209309+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209309+0000","last_undegraded":"2024-09-17T04:05:24.209309+0000","last_fullsized":"2024-09-17T04:05:24.209309+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232775+0000","last_change":"2024-09-17T04:05:24.231560+0000","last_active":"2024-09-17T04:05:24.232775+0000","last_peered":"2024-09-17T04:05:24.232775+0000","last_clean":"2024-09-17T04:05:24.232775+0000","last_became_active":"2024-09-17T04:05:24.231172+0000","last_became_peered":"2024-09-17T04:05:24.231172+0000","last_unstale":"2024-09-17T04:05:24.232775+0000","last_undegraded":"2024-09-17T04:05:24.232775+0000","last_fullsized":"2024-09-17T04:05:24.232775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:39.213576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209752+0000","last_change":"2024-09-17T04:05:24.209752+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209752+0000","last_undegraded":"2024-09-17T04:05:24.209752+0000","last_fullsized":"2024-09-17T04:05:24.209752+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233192+0000","last_change":"2024-09-17T04:05:24.229283+0000","last_active":"2024-09-17T04:05:24.233192+0000","last_peered":"2024-09-17T04:05:24.233192+0000","last_clean":"2024-09-17T04:05:24.233192+0000","last_became_active":"2024-09-17T04:05:24.228948+0000","last_became_peered":"2024-09-17T04:05:24.228948+0000","last_unstale":"2024-09-17T04:05:24.233192+0000","last_undegraded":"2024-09-17T04:05:24.233192+0000","last_fullsized":"2024-09-17T04:05:24.233192+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:33.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211360+0000","last_change":"2024-09-17T04:05:24.211360+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211360+0000","last_undegraded":"2024-09-17T04:05:24.211360+0000","last_fullsized":"2024-09-17T04:05:24.211360+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.224229+0000","last_change":"2024-09-17T04:05:24.222556+0000","last_active":"2024-09-17T04:05:24.224229+0000","last_peered":"2024-09-17T04:05:24.224229+0000","last_clean":"2024-09-17T04:05:24.224229+0000","last_became_active":"2024-09-17T04:05:24.222221+0000","last_became_peered":"2024-09-17T04:05:24.222221+0000","last_unstale":"2024-09-17T04:05:24.224229+0000","last_undegraded":"2024-09-17T04:05:24.224229+0000","last_fullsized":"2024-09-17T04:05:24.224229+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:09.099725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215819+0000","last_change":"2024-09-17T04:05:24.215819+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215819+0000","last_undegraded":"2024-09-17T04:05:24.215819+0000","last_fullsized":"2024-09-17T04:05:24.215819+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233106+0000","last_change":"2024-09-17T04:05:24.226050+0000","last_active":"2024-09-17T04:05:24.233106+0000","last_peered":"2024-09-17T04:05:24.233106+0000","last_clean":"2024-09-17T04:05:24.233106+0000","last_became_active":"2024-09-17T04:05:24.225367+0000","last_became_peered":"2024-09-17T04:05:24.225367+0000","last_unstale":"2024-09-17T04:05:24.233106+0000","last_undegraded":"2024-09-17T04:05:24.233106+0000","last_fullsized":"2024-09-17T04:05:24.233106+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:18.957797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.219860+0000","last_change":"2024-09-17T04:05:24.219860+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.219860+0000","last_undegraded":"2024-09-17T04:05:24.219860+0000","last_fullsized":"2024-09-17T04:05:24.219860+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.229599+0000","last_change":"2024-09-17T04:05:24.226944+0000","last_active":"2024-09-17T04:05:24.229599+0000","last_peered":"2024-09-17T04:05:24.229599+0000","last_clean":"2024-09-17T04:05:24.229599+0000","last_became_active":"2024-09-17T04:05:24.225543+0000","last_became_peered":"2024-09-17T04:05:24.225543+0000","last_unstale":"2024-09-17T04:05:24.229599+0000","last_undegraded":"2024-09-17T04:05:24.229599+0000","last_fullsized":"2024-09-17T04:05:24.229599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:32.931323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212615+0000","last_change":"2024-09-17T04:05:24.212615+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212615+0000","last_undegraded":"2024-09-17T04:05:24.212615+0000","last_fullsized":"2024-09-17T04:05:24.212615+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233774+0000","last_change":"2024-09-17T04:05:24.229174+0000","last_active":"2024-09-17T04:05:24.233774+0000","last_peered":"2024-09-17T04:05:24.233774+0000","last_clean":"2024-09-17T04:05:24.233774+0000","last_became_active":"2024-09-17T04:05:24.228858+0000","last_became_peered":"2024-09-17T04:05:24.228858+0000","last_unstale":"2024-09-17T04:05:24.233774+0000","last_undegraded":"2024-09-17T04:05:24.233774+0000","last_fullsized":"2024-09-17T04:05:24.233774+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:44.527416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213318+0000","last_change":"2024-09-17T04:05:24.213318+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213318+0000","last_undegraded":"2024-09-17T04:05:24.213318+0000","last_fullsized":"2024-09-17T04:05:24.213318+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227436+0000","last_change":"2024-09-17T04:05:24.225036+0000","last_active":"2024-09-17T04:05:24.227436+0000","last_peered":"2024-09-17T04:05:24.227436+0000","last_clean":"2024-09-17T04:05:24.227436+0000","last_became_active":"2024-09-17T04:05:24.223209+0000","last_became_peered":"2024-09-17T04:05:24.223209+0000","last_unstale":"2024-09-17T04:05:24.227436+0000","last_undegraded":"2024-09-17T04:05:24.227436+0000","last_fullsized":"2024-09-17T04:05:24.227436+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:13.835186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214644+0000","last_change":"2024-09-17T04:05:24.214644+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214644+0000","last_undegraded":"2024-09-17T04:05:24.214644+0000","last_fullsized":"2024-09-17T04:05:24.214644+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227394+0000","last_change":"2024-09-17T04:05:24.224549+0000","last_active":"2024-09-17T04:05:24.227394+0000","last_peered":"2024-09-17T04:05:24.227394+0000","last_clean":"2024-09-17T04:05:24.227394+0000","last_became_active":"2024-09-17T04:05:24.224066+0000","last_became_peered":"2024-09-17T04:05:24.224066+0000","last_unstale":"2024-09-17T04:05:24.227394+0000","last_undegraded":"2024-09-17T04:05:24.227394+0000","last_fullsized":"2024-09-17T04:05:24.227394+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:16.113356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.208267+0000","last_change":"2024-09-17T04:05:24.208267+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.208267+0000","last_undegraded":"2024-09-17T04:05:24.208267+0000","last_fullsized":"2024-09-17T04:05:24.208267+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232336+0000","last_change":"2024-09-17T04:05:24.229963+0000","last_active":"2024-09-17T04:05:24.232336+0000","last_peered":"2024-09-17T04:05:24.232336+0000","last_clean":"2024-09-17T04:05:24.232336+0000","last_became_active":"2024-09-17T04:05:24.229600+0000","last_became_peered":"2024-09-17T04:05:24.229600+0000","last_unstale":"2024-09-17T04:05:24.232336+0000","last_undegraded":"2024-09-17T04:05:24.232336+0000","last_fullsized":"2024-09-17T04:05:24.232336+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:44.081757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425310+0000","last_change":"2024-09-17T04:05:25.238027+0000","last_active":"2024-09-17T04:05:26.425310+0000","last_peered":"2024-09-17T04:05:26.425310+0000","last_clean":"2024-09-17T04:05:26.425310+0000","last_became_active":"2024-09-17T04:05:25.236850+0000","last_became_peered":"2024-09-17T04:05:25.236850+0000","last_unstale":"2024-09-17T04:05:26.425310+0000","last_undegraded":"2024-09-17T04:05:26.425310+0000","last_fullsized":"2024-09-17T04:05:26.425310+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:04.588444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227167+0000","last_change":"2024-09-17T04:05:24.224522+0000","last_active":"2024-09-17T04:05:24.227167+0000","last_peered":"2024-09-17T04:05:24.227167+0000","last_clean":"2024-09-17T04:05:24.227167+0000","last_became_active":"2024-09-17T04:05:24.224171+0000","last_became_peered":"2024-09-17T04:05:24.224171+0000","last_unstale":"2024-09-17T04:05:24.227167+0000","last_undegraded":"2024-09-17T04:05:24.227167+0000","last_fullsized":"2024-09-17T04:05:24.227167+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:34.227715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210915+0000","last_change":"2024-09-17T04:05:24.210915+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210915+0000","last_undegraded":"2024-09-17T04:05:24.210915+0000","last_fullsized":"2024-09-17T04:05:24.210915+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.238850+0000","last_change":"2024-09-17T04:05:24.232297+0000","last_active":"2024-09-17T04:05:24.238850+0000","last_peered":"2024-09-17T04:05:24.238850+0000","last_clean":"2024-09-17T04:05:24.238850+0000","last_became_active":"2024-09-17T04:05:24.229674+0000","last_became_peered":"2024-09-17T04:05:24.229674+0000","last_unstale":"2024-09-17T04:05:24.238850+0000","last_undegraded":"2024-09-17T04:05:24.238850+0000","last_fullsized":"2024-09-17T04:05:24.238850+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:46.421923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.208267+0000","last_change":"2024-09-17T04:05:24.208267+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.208267+0000","last_undegraded":"2024-09-17T04:05:24.208267+0000","last_fullsized":"2024-09-17T04:05:24.208267+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233225+0000","last_change":"2024-09-17T04:05:24.232345+0000","last_active":"2024-09-17T04:05:24.233225+0000","last_peered":"2024-09-17T04:05:24.233225+0000","last_clean":"2024-09-17T04:05:24.233225+0000","last_became_active":"2024-09-17T04:05:24.231921+0000","last_became_peered":"2024-09-17T04:05:24.231921+0000","last_unstale":"2024-09-17T04:05:24.233225+0000","last_undegraded":"2024-09-17T04:05:24.233225+0000","last_fullsized":"2024-09-17T04:05:24.233225+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:31.267336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211625+0000","last_change":"2024-09-17T04:05:24.211625+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211625+0000","last_undegraded":"2024-09-17T04:05:24.211625+0000","last_fullsized":"2024-09-17T04:05:24.211625+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500526+0000","last_change":"2024-09-17T04:05:24.227599+0000","last_active":"2024-09-17T04:05:26.500526+0000","last_peered":"2024-09-17T04:05:26.500526+0000","last_clean":"2024-09-17T04:05:26.500526+0000","last_became_active":"2024-09-17T04:05:24.225987+0000","last_became_peered":"2024-09-17T04:05:24.225987+0000","last_unstale":"2024-09-17T04:05:26.500526+0000","last_undegraded":"2024-09-17T04:05:26.500526+0000","last_fullsized":"2024-09-17T04:05:26.500526+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:05.381905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209870+0000","last_change":"2024-09-17T04:05:24.209870+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209870+0000","last_undegraded":"2024-09-17T04:05:24.209870+0000","last_fullsized":"2024-09-17T04:05:24.209870+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227803+0000","last_change":"2024-09-17T04:05:24.222480+0000","last_active":"2024-09-17T04:05:24.227803+0000","last_peered":"2024-09-17T04:05:24.227803+0000","last_clean":"2024-09-17T04:05:24.227803+0000","last_became_active":"2024-09-17T04:05:24.222139+0000","last_became_peered":"2024-09-17T04:05:24.222139+0000","last_unstale":"2024-09-17T04:05:24.227803+0000","last_undegraded":"2024-09-17T04:05:24.227803+0000","last_fullsized":"2024-09-17T04:05:24.227803+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:19.117891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214533+0000","last_change":"2024-09-17T04:05:24.214533+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214533+0000","last_undegraded":"2024-09-17T04:05:24.214533+0000","last_fullsized":"2024-09-17T04:05:24.214533+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425138+0000","last_change":"2024-09-17T04:05:24.228230+0000","last_active":"2024-09-17T04:05:26.425138+0000","last_peered":"2024-09-17T04:05:26.425138+0000","last_clean":"2024-09-17T04:05:26.425138+0000","last_became_active":"2024-09-17T04:05:24.227943+0000","last_became_peered":"2024-09-17T04:05:24.227943+0000","last_unstale":"2024-09-17T04:05:26.425138+0000","last_undegraded":"2024-09-17T04:05:26.425138+0000","last_fullsized":"2024-09-17T04:05:26.425138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:30.288261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211987+0000","last_change":"2024-09-17T04:05:24.211987+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211987+0000","last_undegraded":"2024-09-17T04:05:24.211987+0000","last_fullsized":"2024-09-17T04:05:24.211987+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425458+0000","last_change":"2024-09-17T04:05:24.223526+0000","last_active":"2024-09-17T04:05:26.425458+0000","last_peered":"2024-09-17T04:05:26.425458+0000","last_clean":"2024-09-17T04:05:26.425458+0000","last_became_active":"2024-09-17T04:05:24.223182+0000","last_became_peered":"2024-09-17T04:05:24.223182+0000","last_unstale":"2024-09-17T04:05:26.425458+0000","last_undegraded":"2024-09-17T04:05:26.425458+0000","last_fullsized":"2024-09-17T04:05:26.425458+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:02.030836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.222115+0000","last_change":"2024-09-17T04:05:24.222115+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.222115+0000","last_undegraded":"2024-09-17T04:05:24.222115+0000","last_fullsized":"2024-09-17T04:05:24.222115+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233671+0000","last_change":"2024-09-17T04:05:24.231774+0000","last_active":"2024-09-17T04:05:24.233671+0000","last_peered":"2024-09-17T04:05:24.233671+0000","last_clean":"2024-09-17T04:05:24.233671+0000","last_became_active":"2024-09-17T04:05:24.231438+0000","last_became_peered":"2024-09-17T04:05:24.231438+0000","last_unstale":"2024-09-17T04:05:24.233671+0000","last_undegraded":"2024-09-17T04:05:24.233671+0000","last_fullsized":"2024-09-17T04:05:24.233671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:22.689850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209833+0000","last_change":"2024-09-17T04:05:24.209833+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209833+0000","last_undegraded":"2024-09-17T04:05:24.209833+0000","last_fullsized":"2024-09-17T04:05:24.209833+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227383+0000","last_change":"2024-09-17T04:05:24.220172+0000","last_active":"2024-09-17T04:05:24.227383+0000","last_peered":"2024-09-17T04:05:24.227383+0000","last_clean":"2024-09-17T04:05:24.227383+0000","last_became_active":"2024-09-17T04:05:24.219869+0000","last_became_peered":"2024-09-17T04:05:24.219869+0000","last_unstale":"2024-09-17T04:05:24.227383+0000","last_undegraded":"2024-09-17T04:05:24.227383+0000","last_fullsized":"2024-09-17T04:05:24.227383+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:51.681172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499599+0000","last_change":"2024-09-17T04:05:25.230118+0000","last_active":"2024-09-17T04:05:26.499599+0000","last_peered":"2024-09-17T04:05:26.499599+0000","last_clean":"2024-09-17T04:05:26.499599+0000","last_became_active":"2024-09-17T04:05:25.229777+0000","last_became_peered":"2024-09-17T04:05:25.229777+0000","last_unstale":"2024-09-17T04:05:26.499599+0000","last_undegraded":"2024-09-17T04:05:26.499599+0000","last_fullsized":"2024-09-17T04:05:26.499599+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:04.076093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":471,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.218819+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:05:24.218819+0000","last_peered":"2024-09-17T04:05:24.218819+0000","last_clean":"2024-09-17T04:05:24.218819+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:05:24.218819+0000","last_undegraded":"2024-09-17T04:05:24.218819+0000","last_fullsized":"2024-09-17T04:05:24.218819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52484,"kb_used_data":3244,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319356,"statfs":{"total":96636764160,"available":96583020544,"internally_reserved":0,"allocated":3321856,"data_stored":1792540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509637,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29680,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342160,"statfs":{"total":96636764160,"available":96606371840,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51888,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319952,"statfs":{"total":96636764160,"available":96583630848,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56048,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315792,"statfs":{"total":96636764160,"available":96579371008,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509636,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56064,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315776,"statfs":{"total":96636764160,"available":96579354624,"internally_reserved":0,"allocated":2727936,"data_stored":1202172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509636,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51904,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319936,"statfs":{"total":96636764160,"available":96583614464,"internally_reserved":0,"allocated":2727936,"data_stored":1202172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2760704,"data_stored":1215359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3346432,"data_stored":1805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:05:29.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:05:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 -- 172.21.15.129:0/3205786163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680d0a90 msgr2=0x7fea680b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 --2- 172.21.15.129:0/3205786163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680d0a90 0x7fea680b4ad0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fea60005fd0 tx=0x7fea6002aab0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 -- 172.21.15.129:0/3205786163 shutdown_connections 2024-09-17T04:05:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 --2- 172.21.15.129:0/3205786163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680d0a90 0x7fea680b4ad0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 --2- 172.21.15.129:0/3205786163 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea680cf8e0 0x7fea68160ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 -- 172.21.15.129:0/3205786163 >> 172.21.15.129:0/3205786163 conn(0x7fea6806cfc0 msgr2=0x7fea6806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 -- 172.21.15.129:0/3205786163 shutdown_connections 2024-09-17T04:05:29.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.905+0000 7fea6d600640 1 -- 172.21.15.129:0/3205786163 wait complete. 2024-09-17T04:05:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.907+0000 7fea6d600640 1 Processor -- start 2024-09-17T04:05:29.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.907+0000 7fea6d600640 1 -- start start 2024-09-17T04:05:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 0x7fea680b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea68155780 0x7fea68155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fea6815d120 con 0x7fea680b46c0 2024-09-17T04:05:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fea6815cfa0 con 0x7fea680cf8e0 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fea6815d2a0 con 0x7fea68155780 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56814/0 (socket says 172.21.15.129:56814) 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea67e00640 1 -- 172.21.15.129:0/1153524556 learned_addr learned my addr 172.21.15.129:0/1153524556 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.909+0000 7fea6cc00640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea68155780 0x7fea68155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67400640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 0x7fea680b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 -- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea68155780 msgr2=0x7fea68155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea68155780 0x7fea68155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 -- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 msgr2=0x7fea680b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 0x7fea680b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 -- 172.21.15.129:0/1153524556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea68074590 con 0x7fea680b46c0 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67400640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 0x7fea680b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:29.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea67e00640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fea5800d740 tx=0x7fea5800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58017070 con 0x7fea680b46c0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fea58009d20 con 0x7fea680b46c0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea68074870 con 0x7fea680b46c0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580106c0 con 0x7fea680b46c0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.910+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea68154f80 con 0x7fea680b46c0 2024-09-17T04:05:29.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.912+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6815d750 con 0x7fea680b46c0 2024-09-17T04:05:29.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.913+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fea5801a070 con 0x7fea680b46c0 2024-09-17T04:05:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.914+0000 7fea5fe00640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 0x7fea4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.914+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fea5809aee0 con 0x7fea680b46c0 2024-09-17T04:05:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.914+0000 7fea67400640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 0x7fea4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:29.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.915+0000 7fea67400640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 0x7fea4807b0a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fea681ec520 tx=0x7fea54009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:29.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:29.918+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fea58063320 con 0x7fea680b46c0 2024-09-17T04:05:30.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.097+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea6806a140 con 0x7fea48078b90 2024-09-17T04:05:30.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.102+0000 7fea5fe00640 1 -- 172.21.15.129:0/1153524556 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248522 (secure 0 0 0) 0x7fea6806a140 con 0x7fea48078b90 2024-09-17T04:05:30.107 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:30.109 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:05:30.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 msgr2=0x7fea4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 0x7fea4807b0a0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fea681ec520 tx=0x7fea54009040 comp rx=0 tx=0).stop 2024-09-17T04:05:30.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 msgr2=0x7fea680b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 secure :-1 s=READY pgs=2381 cs=0 l=1 rev1=1 crypto rx=0x7fea5800d740 tx=0x7fea5800dc00 comp rx=0 tx=0).stop 2024-09-17T04:05:30.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 shutdown_connections 2024-09-17T04:05:30.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fea48078b90 0x7fea4807b0a0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fea68155780 0x7fea68155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea680cf8e0 0x7fea680b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 --2- 172.21.15.129:0/1153524556 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fea680b46c0 0x7fea680b2060 unknown :-1 s=CLOSED pgs=2381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 >> 172.21.15.129:0/1153524556 conn(0x7fea6806cfc0 msgr2=0x7fea68145480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 shutdown_connections 2024-09-17T04:05:30.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.106+0000 7fea6d600640 1 -- 172.21.15.129:0/1153524556 wait complete. 2024-09-17T04:05:30.129 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":694,"stamp":"2024-09-17T04:05:28.569229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":61,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402512,"kb_used_data":22492,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754572208,"statfs":{"total":773094113280,"available":772681940992,"internally_reserved":0,"allocated":23031808,"data_stored":10805549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.376500"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212732+0000","last_change":"2024-09-17T04:05:24.212732+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212732+0000","last_undegraded":"2024-09-17T04:05:24.212732+0000","last_fullsized":"2024-09-17T04:05:24.212732+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.216070+0000","last_change":"2024-09-17T04:05:24.216070+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.216070+0000","last_undegraded":"2024-09-17T04:05:24.216070+0000","last_fullsized":"2024-09-17T04:05:24.216070+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213097+0000","last_change":"2024-09-17T04:05:24.213097+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213097+0000","last_undegraded":"2024-09-17T04:05:24.213097+0000","last_fullsized":"2024-09-17T04:05:24.213097+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214496+0000","last_change":"2024-09-17T04:05:24.214496+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214496+0000","last_undegraded":"2024-09-17T04:05:24.214496+0000","last_fullsized":"2024-09-17T04:05:24.214496+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209881+0000","last_change":"2024-09-17T04:05:24.209881+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209881+0000","last_undegraded":"2024-09-17T04:05:24.209881+0000","last_fullsized":"2024-09-17T04:05:24.209881+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214316+0000","last_change":"2024-09-17T04:05:24.214316+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214316+0000","last_undegraded":"2024-09-17T04:05:24.214316+0000","last_fullsized":"2024-09-17T04:05:24.214316+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500029+0000","last_change":"2024-09-17T04:05:25.217581+0000","last_active":"2024-09-17T04:05:26.500029+0000","last_peered":"2024-09-17T04:05:26.500029+0000","last_clean":"2024-09-17T04:05:26.500029+0000","last_became_active":"2024-09-17T04:05:25.217074+0000","last_became_peered":"2024-09-17T04:05:25.217074+0000","last_unstale":"2024-09-17T04:05:26.500029+0000","last_undegraded":"2024-09-17T04:05:26.500029+0000","last_fullsized":"2024-09-17T04:05:26.500029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:30.412641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212075+0000","last_change":"2024-09-17T04:05:24.212075+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212075+0000","last_undegraded":"2024-09-17T04:05:24.212075+0000","last_fullsized":"2024-09-17T04:05:24.212075+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425646+0000","last_change":"2024-09-17T04:05:25.224713+0000","last_active":"2024-09-17T04:05:26.425646+0000","last_peered":"2024-09-17T04:05:26.425646+0000","last_clean":"2024-09-17T04:05:26.425646+0000","last_became_active":"2024-09-17T04:05:25.224391+0000","last_became_peered":"2024-09-17T04:05:25.224391+0000","last_unstale":"2024-09-17T04:05:26.425646+0000","last_undegraded":"2024-09-17T04:05:26.425646+0000","last_fullsized":"2024-09-17T04:05:26.425646+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:16.385982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500080+0000","last_change":"2024-09-17T04:05:25.226863+0000","last_active":"2024-09-17T04:05:26.500080+0000","last_peered":"2024-09-17T04:05:26.500080+0000","last_clean":"2024-09-17T04:05:26.500080+0000","last_became_active":"2024-09-17T04:05:25.226508+0000","last_became_peered":"2024-09-17T04:05:25.226508+0000","last_unstale":"2024-09-17T04:05:26.500080+0000","last_undegraded":"2024-09-17T04:05:26.500080+0000","last_fullsized":"2024-09-17T04:05:26.500080+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:56.644223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500120+0000","last_change":"2024-09-17T04:05:25.228022+0000","last_active":"2024-09-17T04:05:26.500120+0000","last_peered":"2024-09-17T04:05:26.500120+0000","last_clean":"2024-09-17T04:05:26.500120+0000","last_became_active":"2024-09-17T04:05:25.227659+0000","last_became_peered":"2024-09-17T04:05:25.227659+0000","last_unstale":"2024-09-17T04:05:26.500120+0000","last_undegraded":"2024-09-17T04:05:26.500120+0000","last_fullsized":"2024-09-17T04:05:26.500120+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:51.022680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214060+0000","last_change":"2024-09-17T04:05:24.214060+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214060+0000","last_undegraded":"2024-09-17T04:05:24.214060+0000","last_fullsized":"2024-09-17T04:05:24.214060+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212465+0000","last_change":"2024-09-17T04:05:24.212465+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212465+0000","last_undegraded":"2024-09-17T04:05:24.212465+0000","last_fullsized":"2024-09-17T04:05:24.212465+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215425+0000","last_change":"2024-09-17T04:05:24.215425+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215425+0000","last_undegraded":"2024-09-17T04:05:24.215425+0000","last_fullsized":"2024-09-17T04:05:24.215425+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.217791+0000","last_change":"2024-09-17T04:05:24.217791+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.217791+0000","last_undegraded":"2024-09-17T04:05:24.217791+0000","last_fullsized":"2024-09-17T04:05:24.217791+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211378+0000","last_change":"2024-09-17T04:05:24.211378+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211378+0000","last_undegraded":"2024-09-17T04:05:24.211378+0000","last_fullsized":"2024-09-17T04:05:24.211378+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211382+0000","last_change":"2024-09-17T04:05:24.211382+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211382+0000","last_undegraded":"2024-09-17T04:05:24.211382+0000","last_fullsized":"2024-09-17T04:05:24.211382+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500764+0000","last_change":"2024-09-17T04:05:25.217557+0000","last_active":"2024-09-17T04:05:26.500764+0000","last_peered":"2024-09-17T04:05:26.500764+0000","last_clean":"2024-09-17T04:05:26.500764+0000","last_became_active":"2024-09-17T04:05:25.217163+0000","last_became_peered":"2024-09-17T04:05:25.217163+0000","last_unstale":"2024-09-17T04:05:26.500764+0000","last_undegraded":"2024-09-17T04:05:26.500764+0000","last_fullsized":"2024-09-17T04:05:26.500764+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:28.881707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500523+0000","last_change":"2024-09-17T04:05:25.219655+0000","last_active":"2024-09-17T04:05:26.500523+0000","last_peered":"2024-09-17T04:05:26.500523+0000","last_clean":"2024-09-17T04:05:26.500523+0000","last_became_active":"2024-09-17T04:05:25.219345+0000","last_became_peered":"2024-09-17T04:05:25.219345+0000","last_unstale":"2024-09-17T04:05:26.500523+0000","last_undegraded":"2024-09-17T04:05:26.500523+0000","last_fullsized":"2024-09-17T04:05:26.500523+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:59.868383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.217364+0000","last_change":"2024-09-17T04:05:24.217364+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.217364+0000","last_undegraded":"2024-09-17T04:05:24.217364+0000","last_fullsized":"2024-09-17T04:05:24.217364+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425500+0000","last_change":"2024-09-17T04:05:25.231383+0000","last_active":"2024-09-17T04:05:26.425500+0000","last_peered":"2024-09-17T04:05:26.425500+0000","last_clean":"2024-09-17T04:05:26.425500+0000","last_became_active":"2024-09-17T04:05:25.231058+0000","last_became_peered":"2024-09-17T04:05:25.231058+0000","last_unstale":"2024-09-17T04:05:26.425500+0000","last_undegraded":"2024-09-17T04:05:26.425500+0000","last_fullsized":"2024-09-17T04:05:26.425500+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.757807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425336+0000","last_change":"2024-09-17T04:05:25.223284+0000","last_active":"2024-09-17T04:05:26.425336+0000","last_peered":"2024-09-17T04:05:26.425336+0000","last_clean":"2024-09-17T04:05:26.425336+0000","last_became_active":"2024-09-17T04:05:25.221020+0000","last_became_peered":"2024-09-17T04:05:25.221020+0000","last_unstale":"2024-09-17T04:05:26.425336+0000","last_undegraded":"2024-09-17T04:05:26.425336+0000","last_fullsized":"2024-09-17T04:05:26.425336+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:09.253541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500358+0000","last_change":"2024-09-17T04:05:25.226879+0000","last_active":"2024-09-17T04:05:26.500358+0000","last_peered":"2024-09-17T04:05:26.500358+0000","last_clean":"2024-09-17T04:05:26.500358+0000","last_became_active":"2024-09-17T04:05:25.226571+0000","last_became_peered":"2024-09-17T04:05:25.226571+0000","last_unstale":"2024-09-17T04:05:26.500358+0000","last_undegraded":"2024-09-17T04:05:26.500358+0000","last_fullsized":"2024-09-17T04:05:26.500358+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:45.293995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212257+0000","last_change":"2024-09-17T04:05:24.212257+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212257+0000","last_undegraded":"2024-09-17T04:05:24.212257+0000","last_fullsized":"2024-09-17T04:05:24.212257+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.216822+0000","last_change":"2024-09-17T04:05:24.216822+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.216822+0000","last_undegraded":"2024-09-17T04:05:24.216822+0000","last_fullsized":"2024-09-17T04:05:24.216822+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212029+0000","last_change":"2024-09-17T04:05:24.212029+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212029+0000","last_undegraded":"2024-09-17T04:05:24.212029+0000","last_fullsized":"2024-09-17T04:05:24.212029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500441+0000","last_change":"2024-09-17T04:05:25.230257+0000","last_active":"2024-09-17T04:05:26.500441+0000","last_peered":"2024-09-17T04:05:26.500441+0000","last_clean":"2024-09-17T04:05:26.500441+0000","last_became_active":"2024-09-17T04:05:25.229043+0000","last_became_peered":"2024-09-17T04:05:25.229043+0000","last_unstale":"2024-09-17T04:05:26.500441+0000","last_undegraded":"2024-09-17T04:05:26.500441+0000","last_fullsized":"2024-09-17T04:05:26.500441+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:03.038463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.218979+0000","last_change":"2024-09-17T04:05:24.218979+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.218979+0000","last_undegraded":"2024-09-17T04:05:24.218979+0000","last_fullsized":"2024-09-17T04:05:24.218979+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210651+0000","last_change":"2024-09-17T04:05:24.210651+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210651+0000","last_undegraded":"2024-09-17T04:05:24.210651+0000","last_fullsized":"2024-09-17T04:05:24.210651+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214973+0000","last_change":"2024-09-17T04:05:24.214973+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214973+0000","last_undegraded":"2024-09-17T04:05:24.214973+0000","last_fullsized":"2024-09-17T04:05:24.214973+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212337+0000","last_change":"2024-09-17T04:05:24.212337+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212337+0000","last_undegraded":"2024-09-17T04:05:24.212337+0000","last_fullsized":"2024-09-17T04:05:24.212337+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500413+0000","last_change":"2024-09-17T04:05:25.231448+0000","last_active":"2024-09-17T04:05:26.500413+0000","last_peered":"2024-09-17T04:05:26.500413+0000","last_clean":"2024-09-17T04:05:26.500413+0000","last_became_active":"2024-09-17T04:05:25.229864+0000","last_became_peered":"2024-09-17T04:05:25.229864+0000","last_unstale":"2024-09-17T04:05:26.500413+0000","last_undegraded":"2024-09-17T04:05:26.500413+0000","last_fullsized":"2024-09-17T04:05:26.500413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:31.295888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227395+0000","last_change":"2024-09-17T04:05:24.220787+0000","last_active":"2024-09-17T04:05:24.227395+0000","last_peered":"2024-09-17T04:05:24.227395+0000","last_clean":"2024-09-17T04:05:24.227395+0000","last_became_active":"2024-09-17T04:05:24.220468+0000","last_became_peered":"2024-09-17T04:05:24.220468+0000","last_unstale":"2024-09-17T04:05:24.227395+0000","last_undegraded":"2024-09-17T04:05:24.227395+0000","last_fullsized":"2024-09-17T04:05:24.227395+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:17.532872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214787+0000","last_change":"2024-09-17T04:05:24.214787+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214787+0000","last_undegraded":"2024-09-17T04:05:24.214787+0000","last_fullsized":"2024-09-17T04:05:24.214787+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.238859+0000","last_change":"2024-09-17T04:05:24.231055+0000","last_active":"2024-09-17T04:05:24.238859+0000","last_peered":"2024-09-17T04:05:24.238859+0000","last_clean":"2024-09-17T04:05:24.238859+0000","last_became_active":"2024-09-17T04:05:24.222063+0000","last_became_peered":"2024-09-17T04:05:24.222063+0000","last_unstale":"2024-09-17T04:05:24.238859+0000","last_undegraded":"2024-09-17T04:05:24.238859+0000","last_fullsized":"2024-09-17T04:05:24.238859+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:49:56.085206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500629+0000","last_change":"2024-09-17T04:05:25.227705+0000","last_active":"2024-09-17T04:05:26.500629+0000","last_peered":"2024-09-17T04:05:26.500629+0000","last_clean":"2024-09-17T04:05:26.500629+0000","last_became_active":"2024-09-17T04:05:25.223453+0000","last_became_peered":"2024-09-17T04:05:25.223453+0000","last_unstale":"2024-09-17T04:05:26.500629+0000","last_undegraded":"2024-09-17T04:05:26.500629+0000","last_fullsized":"2024-09-17T04:05:26.500629+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:26.267555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233033+0000","last_change":"2024-09-17T04:05:24.228243+0000","last_active":"2024-09-17T04:05:24.233033+0000","last_peered":"2024-09-17T04:05:24.233033+0000","last_clean":"2024-09-17T04:05:24.233033+0000","last_became_active":"2024-09-17T04:05:24.227852+0000","last_became_peered":"2024-09-17T04:05:24.227852+0000","last_unstale":"2024-09-17T04:05:24.233033+0000","last_undegraded":"2024-09-17T04:05:24.233033+0000","last_fullsized":"2024-09-17T04:05:24.233033+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:14:19.206049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213736+0000","last_change":"2024-09-17T04:05:24.213736+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213736+0000","last_undegraded":"2024-09-17T04:05:24.213736+0000","last_fullsized":"2024-09-17T04:05:24.213736+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232937+0000","last_change":"2024-09-17T04:05:24.226740+0000","last_active":"2024-09-17T04:05:24.232937+0000","last_peered":"2024-09-17T04:05:24.232937+0000","last_clean":"2024-09-17T04:05:24.232937+0000","last_became_active":"2024-09-17T04:05:24.226406+0000","last_became_peered":"2024-09-17T04:05:24.226406+0000","last_unstale":"2024-09-17T04:05:24.232937+0000","last_undegraded":"2024-09-17T04:05:24.232937+0000","last_fullsized":"2024-09-17T04:05:24.232937+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:36:07.670673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425319+0000","last_change":"2024-09-17T04:05:25.232667+0000","last_active":"2024-09-17T04:05:26.425319+0000","last_peered":"2024-09-17T04:05:26.425319+0000","last_clean":"2024-09-17T04:05:26.425319+0000","last_became_active":"2024-09-17T04:05:25.232379+0000","last_became_peered":"2024-09-17T04:05:25.232379+0000","last_unstale":"2024-09-17T04:05:26.425319+0000","last_undegraded":"2024-09-17T04:05:26.425319+0000","last_fullsized":"2024-09-17T04:05:26.425319+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:20.607889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499657+0000","last_change":"2024-09-17T04:05:24.224011+0000","last_active":"2024-09-17T04:05:26.499657+0000","last_peered":"2024-09-17T04:05:26.499657+0000","last_clean":"2024-09-17T04:05:26.499657+0000","last_became_active":"2024-09-17T04:05:24.223670+0000","last_became_peered":"2024-09-17T04:05:24.223670+0000","last_unstale":"2024-09-17T04:05:26.499657+0000","last_undegraded":"2024-09-17T04:05:26.499657+0000","last_fullsized":"2024-09-17T04:05:26.499657+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.853028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215489+0000","last_change":"2024-09-17T04:05:24.215489+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215489+0000","last_undegraded":"2024-09-17T04:05:24.215489+0000","last_fullsized":"2024-09-17T04:05:24.215489+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227346+0000","last_change":"2024-09-17T04:05:24.221715+0000","last_active":"2024-09-17T04:05:24.227346+0000","last_peered":"2024-09-17T04:05:24.227346+0000","last_clean":"2024-09-17T04:05:24.227346+0000","last_became_active":"2024-09-17T04:05:24.221380+0000","last_became_peered":"2024-09-17T04:05:24.221380+0000","last_unstale":"2024-09-17T04:05:24.227346+0000","last_undegraded":"2024-09-17T04:05:24.227346+0000","last_fullsized":"2024-09-17T04:05:24.227346+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:02.226834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210326+0000","last_change":"2024-09-17T04:05:24.210326+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210326+0000","last_undegraded":"2024-09-17T04:05:24.210326+0000","last_fullsized":"2024-09-17T04:05:24.210326+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.235031+0000","last_change":"2024-09-17T04:05:24.234352+0000","last_active":"2024-09-17T04:05:24.235031+0000","last_peered":"2024-09-17T04:05:24.235031+0000","last_clean":"2024-09-17T04:05:24.235031+0000","last_became_active":"2024-09-17T04:05:24.234071+0000","last_became_peered":"2024-09-17T04:05:24.234071+0000","last_unstale":"2024-09-17T04:05:24.235031+0000","last_undegraded":"2024-09-17T04:05:24.235031+0000","last_fullsized":"2024-09-17T04:05:24.235031+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:18.444880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500626+0000","last_change":"2024-09-17T04:05:25.219129+0000","last_active":"2024-09-17T04:05:26.500626+0000","last_peered":"2024-09-17T04:05:26.500626+0000","last_clean":"2024-09-17T04:05:26.500626+0000","last_became_active":"2024-09-17T04:05:25.217644+0000","last_became_peered":"2024-09-17T04:05:25.217644+0000","last_unstale":"2024-09-17T04:05:26.500626+0000","last_undegraded":"2024-09-17T04:05:26.500626+0000","last_fullsized":"2024-09-17T04:05:26.500626+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:52.028296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499665+0000","last_change":"2024-09-17T04:05:24.224568+0000","last_active":"2024-09-17T04:05:26.499665+0000","last_peered":"2024-09-17T04:05:26.499665+0000","last_clean":"2024-09-17T04:05:26.499665+0000","last_became_active":"2024-09-17T04:05:24.223638+0000","last_became_peered":"2024-09-17T04:05:24.223638+0000","last_unstale":"2024-09-17T04:05:26.499665+0000","last_undegraded":"2024-09-17T04:05:26.499665+0000","last_fullsized":"2024-09-17T04:05:26.499665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:49.701292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500661+0000","last_change":"2024-09-17T04:05:25.218519+0000","last_active":"2024-09-17T04:05:26.500661+0000","last_peered":"2024-09-17T04:05:26.500661+0000","last_clean":"2024-09-17T04:05:26.500661+0000","last_became_active":"2024-09-17T04:05:25.217254+0000","last_became_peered":"2024-09-17T04:05:25.217254+0000","last_unstale":"2024-09-17T04:05:26.500661+0000","last_undegraded":"2024-09-17T04:05:26.500661+0000","last_fullsized":"2024-09-17T04:05:26.500661+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:49.680789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227870+0000","last_change":"2024-09-17T04:05:24.223519+0000","last_active":"2024-09-17T04:05:24.227870+0000","last_peered":"2024-09-17T04:05:24.227870+0000","last_clean":"2024-09-17T04:05:24.227870+0000","last_became_active":"2024-09-17T04:05:24.223231+0000","last_became_peered":"2024-09-17T04:05:24.223231+0000","last_unstale":"2024-09-17T04:05:24.227870+0000","last_undegraded":"2024-09-17T04:05:24.227870+0000","last_fullsized":"2024-09-17T04:05:24.227870+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:12.017426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209088+0000","last_change":"2024-09-17T04:05:24.209088+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209088+0000","last_undegraded":"2024-09-17T04:05:24.209088+0000","last_fullsized":"2024-09-17T04:05:24.209088+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233132+0000","last_change":"2024-09-17T04:05:24.226942+0000","last_active":"2024-09-17T04:05:24.233132+0000","last_peered":"2024-09-17T04:05:24.233132+0000","last_clean":"2024-09-17T04:05:24.233132+0000","last_became_active":"2024-09-17T04:05:24.226597+0000","last_became_peered":"2024-09-17T04:05:24.226597+0000","last_unstale":"2024-09-17T04:05:24.233132+0000","last_undegraded":"2024-09-17T04:05:24.233132+0000","last_fullsized":"2024-09-17T04:05:24.233132+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:47:19.422554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.220589+0000","last_change":"2024-09-17T04:05:24.220589+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.220589+0000","last_undegraded":"2024-09-17T04:05:24.220589+0000","last_fullsized":"2024-09-17T04:05:24.220589+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233169+0000","last_change":"2024-09-17T04:05:24.230056+0000","last_active":"2024-09-17T04:05:24.233169+0000","last_peered":"2024-09-17T04:05:24.233169+0000","last_clean":"2024-09-17T04:05:24.233169+0000","last_became_active":"2024-09-17T04:05:24.229748+0000","last_became_peered":"2024-09-17T04:05:24.229748+0000","last_unstale":"2024-09-17T04:05:24.233169+0000","last_undegraded":"2024-09-17T04:05:24.233169+0000","last_fullsized":"2024-09-17T04:05:24.233169+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:29.381428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210187+0000","last_change":"2024-09-17T04:05:24.210187+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210187+0000","last_undegraded":"2024-09-17T04:05:24.210187+0000","last_fullsized":"2024-09-17T04:05:24.210187+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.229675+0000","last_change":"2024-09-17T04:05:24.225084+0000","last_active":"2024-09-17T04:05:24.229675+0000","last_peered":"2024-09-17T04:05:24.229675+0000","last_clean":"2024-09-17T04:05:24.229675+0000","last_became_active":"2024-09-17T04:05:24.224781+0000","last_became_peered":"2024-09-17T04:05:24.224781+0000","last_unstale":"2024-09-17T04:05:24.229675+0000","last_undegraded":"2024-09-17T04:05:24.229675+0000","last_fullsized":"2024-09-17T04:05:24.229675+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:24.613188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209032+0000","last_change":"2024-09-17T04:05:24.209032+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209032+0000","last_undegraded":"2024-09-17T04:05:24.209032+0000","last_fullsized":"2024-09-17T04:05:24.209032+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227339+0000","last_change":"2024-09-17T04:05:24.223126+0000","last_active":"2024-09-17T04:05:24.227339+0000","last_peered":"2024-09-17T04:05:24.227339+0000","last_clean":"2024-09-17T04:05:24.227339+0000","last_became_active":"2024-09-17T04:05:24.221114+0000","last_became_peered":"2024-09-17T04:05:24.221114+0000","last_unstale":"2024-09-17T04:05:24.227339+0000","last_undegraded":"2024-09-17T04:05:24.227339+0000","last_fullsized":"2024-09-17T04:05:24.227339+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:29:18.283661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215086+0000","last_change":"2024-09-17T04:05:24.215086+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215086+0000","last_undegraded":"2024-09-17T04:05:24.215086+0000","last_fullsized":"2024-09-17T04:05:24.215086+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425400+0000","last_change":"2024-09-17T04:05:24.223560+0000","last_active":"2024-09-17T04:05:26.425400+0000","last_peered":"2024-09-17T04:05:26.425400+0000","last_clean":"2024-09-17T04:05:26.425400+0000","last_became_active":"2024-09-17T04:05:24.223302+0000","last_became_peered":"2024-09-17T04:05:24.223302+0000","last_unstale":"2024-09-17T04:05:26.425400+0000","last_undegraded":"2024-09-17T04:05:26.425400+0000","last_fullsized":"2024-09-17T04:05:26.425400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:25.259928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209309+0000","last_change":"2024-09-17T04:05:24.209309+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209309+0000","last_undegraded":"2024-09-17T04:05:24.209309+0000","last_fullsized":"2024-09-17T04:05:24.209309+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232775+0000","last_change":"2024-09-17T04:05:24.231560+0000","last_active":"2024-09-17T04:05:24.232775+0000","last_peered":"2024-09-17T04:05:24.232775+0000","last_clean":"2024-09-17T04:05:24.232775+0000","last_became_active":"2024-09-17T04:05:24.231172+0000","last_became_peered":"2024-09-17T04:05:24.231172+0000","last_unstale":"2024-09-17T04:05:24.232775+0000","last_undegraded":"2024-09-17T04:05:24.232775+0000","last_fullsized":"2024-09-17T04:05:24.232775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:39.213576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209752+0000","last_change":"2024-09-17T04:05:24.209752+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209752+0000","last_undegraded":"2024-09-17T04:05:24.209752+0000","last_fullsized":"2024-09-17T04:05:24.209752+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233192+0000","last_change":"2024-09-17T04:05:24.229283+0000","last_active":"2024-09-17T04:05:24.233192+0000","last_peered":"2024-09-17T04:05:24.233192+0000","last_clean":"2024-09-17T04:05:24.233192+0000","last_became_active":"2024-09-17T04:05:24.228948+0000","last_became_peered":"2024-09-17T04:05:24.228948+0000","last_unstale":"2024-09-17T04:05:24.233192+0000","last_undegraded":"2024-09-17T04:05:24.233192+0000","last_fullsized":"2024-09-17T04:05:24.233192+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:08:33.178822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211360+0000","last_change":"2024-09-17T04:05:24.211360+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211360+0000","last_undegraded":"2024-09-17T04:05:24.211360+0000","last_fullsized":"2024-09-17T04:05:24.211360+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.224229+0000","last_change":"2024-09-17T04:05:24.222556+0000","last_active":"2024-09-17T04:05:24.224229+0000","last_peered":"2024-09-17T04:05:24.224229+0000","last_clean":"2024-09-17T04:05:24.224229+0000","last_became_active":"2024-09-17T04:05:24.222221+0000","last_became_peered":"2024-09-17T04:05:24.222221+0000","last_unstale":"2024-09-17T04:05:24.224229+0000","last_undegraded":"2024-09-17T04:05:24.224229+0000","last_fullsized":"2024-09-17T04:05:24.224229+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:26:09.099725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.215819+0000","last_change":"2024-09-17T04:05:24.215819+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.215819+0000","last_undegraded":"2024-09-17T04:05:24.215819+0000","last_fullsized":"2024-09-17T04:05:24.215819+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233106+0000","last_change":"2024-09-17T04:05:24.226050+0000","last_active":"2024-09-17T04:05:24.233106+0000","last_peered":"2024-09-17T04:05:24.233106+0000","last_clean":"2024-09-17T04:05:24.233106+0000","last_became_active":"2024-09-17T04:05:24.225367+0000","last_became_peered":"2024-09-17T04:05:24.225367+0000","last_unstale":"2024-09-17T04:05:24.233106+0000","last_undegraded":"2024-09-17T04:05:24.233106+0000","last_fullsized":"2024-09-17T04:05:24.233106+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:43:18.957797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.219860+0000","last_change":"2024-09-17T04:05:24.219860+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.219860+0000","last_undegraded":"2024-09-17T04:05:24.219860+0000","last_fullsized":"2024-09-17T04:05:24.219860+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.229599+0000","last_change":"2024-09-17T04:05:24.226944+0000","last_active":"2024-09-17T04:05:24.229599+0000","last_peered":"2024-09-17T04:05:24.229599+0000","last_clean":"2024-09-17T04:05:24.229599+0000","last_became_active":"2024-09-17T04:05:24.225543+0000","last_became_peered":"2024-09-17T04:05:24.225543+0000","last_unstale":"2024-09-17T04:05:24.229599+0000","last_undegraded":"2024-09-17T04:05:24.229599+0000","last_fullsized":"2024-09-17T04:05:24.229599+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:49:32.931323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.212615+0000","last_change":"2024-09-17T04:05:24.212615+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.212615+0000","last_undegraded":"2024-09-17T04:05:24.212615+0000","last_fullsized":"2024-09-17T04:05:24.212615+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233774+0000","last_change":"2024-09-17T04:05:24.229174+0000","last_active":"2024-09-17T04:05:24.233774+0000","last_peered":"2024-09-17T04:05:24.233774+0000","last_clean":"2024-09-17T04:05:24.233774+0000","last_became_active":"2024-09-17T04:05:24.228858+0000","last_became_peered":"2024-09-17T04:05:24.228858+0000","last_unstale":"2024-09-17T04:05:24.233774+0000","last_undegraded":"2024-09-17T04:05:24.233774+0000","last_fullsized":"2024-09-17T04:05:24.233774+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:44.527416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.213318+0000","last_change":"2024-09-17T04:05:24.213318+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.213318+0000","last_undegraded":"2024-09-17T04:05:24.213318+0000","last_fullsized":"2024-09-17T04:05:24.213318+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227436+0000","last_change":"2024-09-17T04:05:24.225036+0000","last_active":"2024-09-17T04:05:24.227436+0000","last_peered":"2024-09-17T04:05:24.227436+0000","last_clean":"2024-09-17T04:05:24.227436+0000","last_became_active":"2024-09-17T04:05:24.223209+0000","last_became_peered":"2024-09-17T04:05:24.223209+0000","last_unstale":"2024-09-17T04:05:24.227436+0000","last_undegraded":"2024-09-17T04:05:24.227436+0000","last_fullsized":"2024-09-17T04:05:24.227436+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:36:13.835186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214644+0000","last_change":"2024-09-17T04:05:24.214644+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214644+0000","last_undegraded":"2024-09-17T04:05:24.214644+0000","last_fullsized":"2024-09-17T04:05:24.214644+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227394+0000","last_change":"2024-09-17T04:05:24.224549+0000","last_active":"2024-09-17T04:05:24.227394+0000","last_peered":"2024-09-17T04:05:24.227394+0000","last_clean":"2024-09-17T04:05:24.227394+0000","last_became_active":"2024-09-17T04:05:24.224066+0000","last_became_peered":"2024-09-17T04:05:24.224066+0000","last_unstale":"2024-09-17T04:05:24.227394+0000","last_undegraded":"2024-09-17T04:05:24.227394+0000","last_fullsized":"2024-09-17T04:05:24.227394+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:16.113356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.208267+0000","last_change":"2024-09-17T04:05:24.208267+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.208267+0000","last_undegraded":"2024-09-17T04:05:24.208267+0000","last_fullsized":"2024-09-17T04:05:24.208267+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.232336+0000","last_change":"2024-09-17T04:05:24.229963+0000","last_active":"2024-09-17T04:05:24.232336+0000","last_peered":"2024-09-17T04:05:24.232336+0000","last_clean":"2024-09-17T04:05:24.232336+0000","last_became_active":"2024-09-17T04:05:24.229600+0000","last_became_peered":"2024-09-17T04:05:24.229600+0000","last_unstale":"2024-09-17T04:05:24.232336+0000","last_undegraded":"2024-09-17T04:05:24.232336+0000","last_fullsized":"2024-09-17T04:05:24.232336+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:44.081757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425310+0000","last_change":"2024-09-17T04:05:25.238027+0000","last_active":"2024-09-17T04:05:26.425310+0000","last_peered":"2024-09-17T04:05:26.425310+0000","last_clean":"2024-09-17T04:05:26.425310+0000","last_became_active":"2024-09-17T04:05:25.236850+0000","last_became_peered":"2024-09-17T04:05:25.236850+0000","last_unstale":"2024-09-17T04:05:26.425310+0000","last_undegraded":"2024-09-17T04:05:26.425310+0000","last_fullsized":"2024-09-17T04:05:26.425310+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:04.588444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227167+0000","last_change":"2024-09-17T04:05:24.224522+0000","last_active":"2024-09-17T04:05:24.227167+0000","last_peered":"2024-09-17T04:05:24.227167+0000","last_clean":"2024-09-17T04:05:24.227167+0000","last_became_active":"2024-09-17T04:05:24.224171+0000","last_became_peered":"2024-09-17T04:05:24.224171+0000","last_unstale":"2024-09-17T04:05:24.227167+0000","last_undegraded":"2024-09-17T04:05:24.227167+0000","last_fullsized":"2024-09-17T04:05:24.227167+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:58:34.227715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.210915+0000","last_change":"2024-09-17T04:05:24.210915+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.210915+0000","last_undegraded":"2024-09-17T04:05:24.210915+0000","last_fullsized":"2024-09-17T04:05:24.210915+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.238850+0000","last_change":"2024-09-17T04:05:24.232297+0000","last_active":"2024-09-17T04:05:24.238850+0000","last_peered":"2024-09-17T04:05:24.238850+0000","last_clean":"2024-09-17T04:05:24.238850+0000","last_became_active":"2024-09-17T04:05:24.229674+0000","last_became_peered":"2024-09-17T04:05:24.229674+0000","last_unstale":"2024-09-17T04:05:24.238850+0000","last_undegraded":"2024-09-17T04:05:24.238850+0000","last_fullsized":"2024-09-17T04:05:24.238850+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:46.421923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.208267+0000","last_change":"2024-09-17T04:05:24.208267+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.208267+0000","last_undegraded":"2024-09-17T04:05:24.208267+0000","last_fullsized":"2024-09-17T04:05:24.208267+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233225+0000","last_change":"2024-09-17T04:05:24.232345+0000","last_active":"2024-09-17T04:05:24.233225+0000","last_peered":"2024-09-17T04:05:24.233225+0000","last_clean":"2024-09-17T04:05:24.233225+0000","last_became_active":"2024-09-17T04:05:24.231921+0000","last_became_peered":"2024-09-17T04:05:24.231921+0000","last_unstale":"2024-09-17T04:05:24.233225+0000","last_undegraded":"2024-09-17T04:05:24.233225+0000","last_fullsized":"2024-09-17T04:05:24.233225+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:30:31.267336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211625+0000","last_change":"2024-09-17T04:05:24.211625+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211625+0000","last_undegraded":"2024-09-17T04:05:24.211625+0000","last_fullsized":"2024-09-17T04:05:24.211625+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500526+0000","last_change":"2024-09-17T04:05:24.227599+0000","last_active":"2024-09-17T04:05:26.500526+0000","last_peered":"2024-09-17T04:05:26.500526+0000","last_clean":"2024-09-17T04:05:26.500526+0000","last_became_active":"2024-09-17T04:05:24.225987+0000","last_became_peered":"2024-09-17T04:05:24.225987+0000","last_unstale":"2024-09-17T04:05:26.500526+0000","last_undegraded":"2024-09-17T04:05:26.500526+0000","last_fullsized":"2024-09-17T04:05:26.500526+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:05.381905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209870+0000","last_change":"2024-09-17T04:05:24.209870+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209870+0000","last_undegraded":"2024-09-17T04:05:24.209870+0000","last_fullsized":"2024-09-17T04:05:24.209870+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227803+0000","last_change":"2024-09-17T04:05:24.222480+0000","last_active":"2024-09-17T04:05:24.227803+0000","last_peered":"2024-09-17T04:05:24.227803+0000","last_clean":"2024-09-17T04:05:24.227803+0000","last_became_active":"2024-09-17T04:05:24.222139+0000","last_became_peered":"2024-09-17T04:05:24.222139+0000","last_unstale":"2024-09-17T04:05:24.227803+0000","last_undegraded":"2024-09-17T04:05:24.227803+0000","last_fullsized":"2024-09-17T04:05:24.227803+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:19.117891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.214533+0000","last_change":"2024-09-17T04:05:24.214533+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.214533+0000","last_undegraded":"2024-09-17T04:05:24.214533+0000","last_fullsized":"2024-09-17T04:05:24.214533+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425138+0000","last_change":"2024-09-17T04:05:24.228230+0000","last_active":"2024-09-17T04:05:26.425138+0000","last_peered":"2024-09-17T04:05:26.425138+0000","last_clean":"2024-09-17T04:05:26.425138+0000","last_became_active":"2024-09-17T04:05:24.227943+0000","last_became_peered":"2024-09-17T04:05:24.227943+0000","last_unstale":"2024-09-17T04:05:26.425138+0000","last_undegraded":"2024-09-17T04:05:26.425138+0000","last_fullsized":"2024-09-17T04:05:26.425138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:30.288261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.211987+0000","last_change":"2024-09-17T04:05:24.211987+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.211987+0000","last_undegraded":"2024-09-17T04:05:24.211987+0000","last_fullsized":"2024-09-17T04:05:24.211987+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425458+0000","last_change":"2024-09-17T04:05:24.223526+0000","last_active":"2024-09-17T04:05:26.425458+0000","last_peered":"2024-09-17T04:05:26.425458+0000","last_clean":"2024-09-17T04:05:26.425458+0000","last_became_active":"2024-09-17T04:05:24.223182+0000","last_became_peered":"2024-09-17T04:05:24.223182+0000","last_unstale":"2024-09-17T04:05:26.425458+0000","last_undegraded":"2024-09-17T04:05:26.425458+0000","last_fullsized":"2024-09-17T04:05:26.425458+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:02.030836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.222115+0000","last_change":"2024-09-17T04:05:24.222115+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.222115+0000","last_undegraded":"2024-09-17T04:05:24.222115+0000","last_fullsized":"2024-09-17T04:05:24.222115+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.233671+0000","last_change":"2024-09-17T04:05:24.231774+0000","last_active":"2024-09-17T04:05:24.233671+0000","last_peered":"2024-09-17T04:05:24.233671+0000","last_clean":"2024-09-17T04:05:24.233671+0000","last_became_active":"2024-09-17T04:05:24.231438+0000","last_became_peered":"2024-09-17T04:05:24.231438+0000","last_unstale":"2024-09-17T04:05:24.233671+0000","last_undegraded":"2024-09-17T04:05:24.233671+0000","last_fullsized":"2024-09-17T04:05:24.233671+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:22.689850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":1,"reported_epoch":216,"state":"creating+peering","last_fresh":"2024-09-17T04:05:24.209833+0000","last_change":"2024-09-17T04:05:24.209833+0000","last_active":"2024-09-17T04:05:24.191890+0000","last_peered":"2024-09-17T04:05:24.191890+0000","last_clean":"2024-09-17T04:05:24.191890+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:05:24.209833+0000","last_undegraded":"2024-09-17T04:05:24.209833+0000","last_fullsized":"2024-09-17T04:05:24.209833+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":10,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.227383+0000","last_change":"2024-09-17T04:05:24.220172+0000","last_active":"2024-09-17T04:05:24.227383+0000","last_peered":"2024-09-17T04:05:24.227383+0000","last_clean":"2024-09-17T04:05:24.227383+0000","last_became_active":"2024-09-17T04:05:24.219869+0000","last_became_peered":"2024-09-17T04:05:24.219869+0000","last_unstale":"2024-09-17T04:05:24.227383+0000","last_undegraded":"2024-09-17T04:05:24.227383+0000","last_fullsized":"2024-09-17T04:05:24.227383+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:51.681172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499599+0000","last_change":"2024-09-17T04:05:25.230118+0000","last_active":"2024-09-17T04:05:26.499599+0000","last_peered":"2024-09-17T04:05:26.499599+0000","last_clean":"2024-09-17T04:05:26.499599+0000","last_became_active":"2024-09-17T04:05:25.229777+0000","last_became_peered":"2024-09-17T04:05:25.229777+0000","last_unstale":"2024-09-17T04:05:26.499599+0000","last_undegraded":"2024-09-17T04:05:26.499599+0000","last_fullsized":"2024-09-17T04:05:26.499599+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:04.076093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":471,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-17T04:05:24.218819+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:05:24.218819+0000","last_peered":"2024-09-17T04:05:24.218819+0000","last_clean":"2024-09-17T04:05:24.218819+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:05:24.218819+0000","last_undegraded":"2024-09-17T04:05:24.218819+0000","last_fullsized":"2024-09-17T04:05:24.218819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509634,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52484,"kb_used_data":3244,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319356,"statfs":{"total":96636764160,"available":96583020544,"internally_reserved":0,"allocated":3321856,"data_stored":1792540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509637,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29680,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342160,"statfs":{"total":96636764160,"available":96606371840,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509636,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51888,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319952,"statfs":{"total":96636764160,"available":96583630848,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509634,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56048,"kb_used_data":2652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315792,"statfs":{"total":96636764160,"available":96579371008,"internally_reserved":0,"allocated":2715648,"data_stored":1195897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509636,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56064,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315776,"statfs":{"total":96636764160,"available":96579354624,"internally_reserved":0,"allocated":2727936,"data_stored":1202172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509636,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51904,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319936,"statfs":{"total":96636764160,"available":96583614464,"internally_reserved":0,"allocated":2727936,"data_stored":1202172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2760704,"data_stored":1215359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3346432,"data_stored":1805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:05:30.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 -- 172.21.15.129:0/1386652826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b7415da00 msgr2=0x7f9b740b4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 --2- 172.21.15.129:0/1386652826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b7415da00 0x7f9b740b4180 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f9b64005fd0 tx=0x7f9b6402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 -- 172.21.15.129:0/1386652826 shutdown_connections 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 --2- 172.21.15.129:0/1386652826 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b740b46c0 0x7f9b740ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 --2- 172.21.15.129:0/1386652826 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b7415da00 0x7f9b740b4180 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 -- 172.21.15.129:0/1386652826 >> 172.21.15.129:0/1386652826 conn(0x7f9b7406d410 msgr2=0x7f9b740b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 -- 172.21.15.129:0/1386652826 shutdown_connections 2024-09-17T04:05:30.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.393+0000 7f9b7a800640 1 -- 172.21.15.129:0/1386652826 wait complete. 2024-09-17T04:05:30.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.396+0000 7f9b7a800640 1 Processor -- start 2024-09-17T04:05:30.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.396+0000 7f9b7a800640 1 -- start start 2024-09-17T04:05:30.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 0x7f9b741f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 0x7f9b741f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9b74070990 con 0x7f9b740cd1c0 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9b74070810 con 0x7f9b740b46c0 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b7a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9b74070b10 con 0x7f9b7415da00 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56830/0 (socket says 172.21.15.129:56830) 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.397+0000 7f9b72a00640 1 -- 172.21.15.129:0/3879064414 learned_addr learned my addr 172.21.15.129:0/3879064414 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b73400640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 0x7f9b741f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b73e00640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 0x7f9b741f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 -- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 msgr2=0x7f9b741f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 0x7f9b741f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 -- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 msgr2=0x7f9b741f3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 0x7f9b741f3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 -- 172.21.15.129:0/3879064414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b740cadf0 con 0x7f9b740cd1c0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b73400640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 0x7f9b741f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b73e00640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 0x7f9b741f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b72a00640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000d730 tx=0x7f9b6000dbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60017070 con 0x7f9b740cd1c0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9b60009b40 con 0x7f9b740cd1c0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b60005020 con 0x7f9b740cd1c0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.398+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b740cc860 con 0x7f9b740cd1c0 2024-09-17T04:05:30.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.399+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b740cc4f0 con 0x7f9b740cd1c0 2024-09-17T04:05:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.399+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b740ccc80 con 0x7f9b740cd1c0 2024-09-17T04:05:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.401+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9b600051c0 con 0x7f9b740cd1c0 2024-09-17T04:05:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.402+0000 7f9b6b400640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 0x7f9b5407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.402+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f9b6009add0 con 0x7f9b740cd1c0 2024-09-17T04:05:30.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.402+0000 7f9b73400640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 0x7f9b5407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.403+0000 7f9b73400640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 0x7f9b5407b0d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9b64033000 tx=0x7f9b64004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:30.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.406+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9b60063200 con 0x7f9b740cd1c0 2024-09-17T04:05:30.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.586+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9b740ca8c0 con 0x7f9b54078bc0 2024-09-17T04:05:30.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.591+0000 7f9b6b400640 1 -- 172.21.15.129:0/3879064414 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7f9b740ca8c0 con 0x7f9b54078bc0 2024-09-17T04:05:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:30.595 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:05:30.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 msgr2=0x7f9b5407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 0x7f9b5407b0d0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f9b64033000 tx=0x7f9b64004730 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 msgr2=0x7f9b741f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f9b6000d730 tx=0x7f9b6000dbf0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 shutdown_connections 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9b54078bc0 0x7f9b5407b0d0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9b7415da00 0x7f9b741f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9b740cd1c0 0x7f9b741f5f50 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 --2- 172.21.15.129:0/3879064414 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9b740b46c0 0x7f9b741f3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 >> 172.21.15.129:0/3879064414 conn(0x7f9b7406d410 msgr2=0x7f9b7406f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 shutdown_connections 2024-09-17T04:05:30.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.595+0000 7f9b7a800640 1 -- 172.21.15.129:0/3879064414 wait complete. 2024-09-17T04:05:30.618 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-09-17T04:05:30.569710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402736,"kb_used_data":22728,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754571984,"statfs":{"total":773094113280,"available":772681711616,"internally_reserved":0,"allocated":23273472,"data_stored":10907532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.362373"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500684+0000","last_change":"2024-09-17T04:05:25.221229+0000","last_active":"2024-09-17T04:05:26.500684+0000","last_peered":"2024-09-17T04:05:26.500684+0000","last_clean":"2024-09-17T04:05:26.500684+0000","last_became_active":"2024-09-17T04:05:25.220873+0000","last_became_peered":"2024-09-17T04:05:25.220873+0000","last_unstale":"2024-09-17T04:05:26.500684+0000","last_undegraded":"2024-09-17T04:05:26.500684+0000","last_fullsized":"2024-09-17T04:05:26.500684+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:38:14.801493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500862+0000","last_change":"2024-09-17T04:05:25.222366+0000","last_active":"2024-09-17T04:05:26.500862+0000","last_peered":"2024-09-17T04:05:26.500862+0000","last_clean":"2024-09-17T04:05:26.500862+0000","last_became_active":"2024-09-17T04:05:25.220987+0000","last_became_peered":"2024-09-17T04:05:25.220987+0000","last_unstale":"2024-09-17T04:05:26.500862+0000","last_undegraded":"2024-09-17T04:05:26.500862+0000","last_fullsized":"2024-09-17T04:05:26.500862+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:23.139209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228756+0000","last_change":"2024-09-17T04:05:25.235978+0000","last_active":"2024-09-17T04:05:26.228756+0000","last_peered":"2024-09-17T04:05:26.228756+0000","last_clean":"2024-09-17T04:05:26.228756+0000","last_became_active":"2024-09-17T04:05:25.234686+0000","last_became_peered":"2024-09-17T04:05:25.234686+0000","last_unstale":"2024-09-17T04:05:26.228756+0000","last_undegraded":"2024-09-17T04:05:26.228756+0000","last_fullsized":"2024-09-17T04:05:26.228756+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:06.764889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228350+0000","last_change":"2024-09-17T04:05:25.234958+0000","last_active":"2024-09-17T04:05:26.228350+0000","last_peered":"2024-09-17T04:05:26.228350+0000","last_clean":"2024-09-17T04:05:26.228350+0000","last_became_active":"2024-09-17T04:05:25.233705+0000","last_became_peered":"2024-09-17T04:05:25.233705+0000","last_unstale":"2024-09-17T04:05:26.228350+0000","last_undegraded":"2024-09-17T04:05:26.228350+0000","last_fullsized":"2024-09-17T04:05:26.228350+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:29.142246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228296+0000","last_change":"2024-09-17T04:05:25.234011+0000","last_active":"2024-09-17T04:05:26.228296+0000","last_peered":"2024-09-17T04:05:26.228296+0000","last_clean":"2024-09-17T04:05:26.228296+0000","last_became_active":"2024-09-17T04:05:25.233676+0000","last_became_peered":"2024-09-17T04:05:25.233676+0000","last_unstale":"2024-09-17T04:05:26.228296+0000","last_undegraded":"2024-09-17T04:05:26.228296+0000","last_fullsized":"2024-09-17T04:05:26.228296+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436570+0000","last_change":"2024-09-17T04:05:25.232509+0000","last_active":"2024-09-17T04:05:26.436570+0000","last_peered":"2024-09-17T04:05:26.436570+0000","last_clean":"2024-09-17T04:05:26.436570+0000","last_became_active":"2024-09-17T04:05:25.232170+0000","last_became_peered":"2024-09-17T04:05:25.232170+0000","last_unstale":"2024-09-17T04:05:26.436570+0000","last_undegraded":"2024-09-17T04:05:26.436570+0000","last_fullsized":"2024-09-17T04:05:26.436570+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:40.882023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500029+0000","last_change":"2024-09-17T04:05:25.217581+0000","last_active":"2024-09-17T04:05:26.500029+0000","last_peered":"2024-09-17T04:05:26.500029+0000","last_clean":"2024-09-17T04:05:26.500029+0000","last_became_active":"2024-09-17T04:05:25.217074+0000","last_became_peered":"2024-09-17T04:05:25.217074+0000","last_unstale":"2024-09-17T04:05:26.500029+0000","last_undegraded":"2024-09-17T04:05:26.500029+0000","last_fullsized":"2024-09-17T04:05:26.500029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:30.412641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499223+0000","last_change":"2024-09-17T04:05:25.226793+0000","last_active":"2024-09-17T04:05:26.499223+0000","last_peered":"2024-09-17T04:05:26.499223+0000","last_clean":"2024-09-17T04:05:26.499223+0000","last_became_active":"2024-09-17T04:05:25.226356+0000","last_became_peered":"2024-09-17T04:05:25.226356+0000","last_unstale":"2024-09-17T04:05:26.499223+0000","last_undegraded":"2024-09-17T04:05:26.499223+0000","last_fullsized":"2024-09-17T04:05:26.499223+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:40.221203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425646+0000","last_change":"2024-09-17T04:05:25.224713+0000","last_active":"2024-09-17T04:05:26.425646+0000","last_peered":"2024-09-17T04:05:26.425646+0000","last_clean":"2024-09-17T04:05:26.425646+0000","last_became_active":"2024-09-17T04:05:25.224391+0000","last_became_peered":"2024-09-17T04:05:25.224391+0000","last_unstale":"2024-09-17T04:05:26.425646+0000","last_undegraded":"2024-09-17T04:05:26.425646+0000","last_fullsized":"2024-09-17T04:05:26.425646+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:16.385982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500080+0000","last_change":"2024-09-17T04:05:25.226863+0000","last_active":"2024-09-17T04:05:26.500080+0000","last_peered":"2024-09-17T04:05:26.500080+0000","last_clean":"2024-09-17T04:05:26.500080+0000","last_became_active":"2024-09-17T04:05:25.226508+0000","last_became_peered":"2024-09-17T04:05:25.226508+0000","last_unstale":"2024-09-17T04:05:26.500080+0000","last_undegraded":"2024-09-17T04:05:26.500080+0000","last_fullsized":"2024-09-17T04:05:26.500080+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:56.644223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500120+0000","last_change":"2024-09-17T04:05:25.228022+0000","last_active":"2024-09-17T04:05:26.500120+0000","last_peered":"2024-09-17T04:05:26.500120+0000","last_clean":"2024-09-17T04:05:26.500120+0000","last_became_active":"2024-09-17T04:05:25.227659+0000","last_became_peered":"2024-09-17T04:05:25.227659+0000","last_unstale":"2024-09-17T04:05:26.500120+0000","last_undegraded":"2024-09-17T04:05:26.500120+0000","last_fullsized":"2024-09-17T04:05:26.500120+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:51.022680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500106+0000","last_change":"2024-09-17T04:05:25.230634+0000","last_active":"2024-09-17T04:05:26.500106+0000","last_peered":"2024-09-17T04:05:26.500106+0000","last_clean":"2024-09-17T04:05:26.500106+0000","last_became_active":"2024-09-17T04:05:25.229125+0000","last_became_peered":"2024-09-17T04:05:25.229125+0000","last_unstale":"2024-09-17T04:05:26.500106+0000","last_undegraded":"2024-09-17T04:05:26.500106+0000","last_fullsized":"2024-09-17T04:05:26.500106+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:13.538395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228383+0000","last_change":"2024-09-17T04:05:25.238984+0000","last_active":"2024-09-17T04:05:26.228383+0000","last_peered":"2024-09-17T04:05:26.228383+0000","last_clean":"2024-09-17T04:05:26.228383+0000","last_became_active":"2024-09-17T04:05:25.233999+0000","last_became_peered":"2024-09-17T04:05:25.233999+0000","last_unstale":"2024-09-17T04:05:26.228383+0000","last_undegraded":"2024-09-17T04:05:26.228383+0000","last_fullsized":"2024-09-17T04:05:26.228383+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228099+0000","last_change":"2024-09-17T04:05:25.232964+0000","last_active":"2024-09-17T04:05:26.228099+0000","last_peered":"2024-09-17T04:05:26.228099+0000","last_clean":"2024-09-17T04:05:26.228099+0000","last_became_active":"2024-09-17T04:05:25.232684+0000","last_became_peered":"2024-09-17T04:05:25.232684+0000","last_unstale":"2024-09-17T04:05:26.228099+0000","last_undegraded":"2024-09-17T04:05:26.228099+0000","last_fullsized":"2024-09-17T04:05:26.228099+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:37.598051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228177+0000","last_change":"2024-09-17T04:05:25.226026+0000","last_active":"2024-09-17T04:05:26.228177+0000","last_peered":"2024-09-17T04:05:26.228177+0000","last_clean":"2024-09-17T04:05:26.228177+0000","last_became_active":"2024-09-17T04:05:25.225661+0000","last_became_peered":"2024-09-17T04:05:25.225661+0000","last_unstale":"2024-09-17T04:05:26.228177+0000","last_undegraded":"2024-09-17T04:05:26.228177+0000","last_fullsized":"2024-09-17T04:05:26.228177+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.988518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436505+0000","last_change":"2024-09-17T04:05:25.230601+0000","last_active":"2024-09-17T04:05:26.436505+0000","last_peered":"2024-09-17T04:05:26.436505+0000","last_clean":"2024-09-17T04:05:26.436505+0000","last_became_active":"2024-09-17T04:05:25.230028+0000","last_became_peered":"2024-09-17T04:05:25.230028+0000","last_unstale":"2024-09-17T04:05:26.436505+0000","last_undegraded":"2024-09-17T04:05:26.436505+0000","last_fullsized":"2024-09-17T04:05:26.436505+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:52:57.752381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225608+0000","last_change":"2024-09-17T04:05:25.227454+0000","last_active":"2024-09-17T04:05:26.225608+0000","last_peered":"2024-09-17T04:05:26.225608+0000","last_clean":"2024-09-17T04:05:26.225608+0000","last_became_active":"2024-09-17T04:05:25.226074+0000","last_became_peered":"2024-09-17T04:05:25.226074+0000","last_unstale":"2024-09-17T04:05:26.225608+0000","last_undegraded":"2024-09-17T04:05:26.225608+0000","last_fullsized":"2024-09-17T04:05:26.225608+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:20.806967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500764+0000","last_change":"2024-09-17T04:05:25.217557+0000","last_active":"2024-09-17T04:05:26.500764+0000","last_peered":"2024-09-17T04:05:26.500764+0000","last_clean":"2024-09-17T04:05:26.500764+0000","last_became_active":"2024-09-17T04:05:25.217163+0000","last_became_peered":"2024-09-17T04:05:25.217163+0000","last_unstale":"2024-09-17T04:05:26.500764+0000","last_undegraded":"2024-09-17T04:05:26.500764+0000","last_fullsized":"2024-09-17T04:05:26.500764+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:28.881707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500523+0000","last_change":"2024-09-17T04:05:25.219655+0000","last_active":"2024-09-17T04:05:26.500523+0000","last_peered":"2024-09-17T04:05:26.500523+0000","last_clean":"2024-09-17T04:05:26.500523+0000","last_became_active":"2024-09-17T04:05:25.219345+0000","last_became_peered":"2024-09-17T04:05:25.219345+0000","last_unstale":"2024-09-17T04:05:26.500523+0000","last_undegraded":"2024-09-17T04:05:26.500523+0000","last_fullsized":"2024-09-17T04:05:26.500523+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:59.868383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436713+0000","last_change":"2024-09-17T04:05:25.227398+0000","last_active":"2024-09-17T04:05:26.436713+0000","last_peered":"2024-09-17T04:05:26.436713+0000","last_clean":"2024-09-17T04:05:26.436713+0000","last_became_active":"2024-09-17T04:05:25.227065+0000","last_became_peered":"2024-09-17T04:05:25.227065+0000","last_unstale":"2024-09-17T04:05:26.436713+0000","last_undegraded":"2024-09-17T04:05:26.436713+0000","last_fullsized":"2024-09-17T04:05:26.436713+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:41.966823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425500+0000","last_change":"2024-09-17T04:05:25.231383+0000","last_active":"2024-09-17T04:05:26.425500+0000","last_peered":"2024-09-17T04:05:26.425500+0000","last_clean":"2024-09-17T04:05:26.425500+0000","last_became_active":"2024-09-17T04:05:25.231058+0000","last_became_peered":"2024-09-17T04:05:25.231058+0000","last_unstale":"2024-09-17T04:05:26.425500+0000","last_undegraded":"2024-09-17T04:05:26.425500+0000","last_fullsized":"2024-09-17T04:05:26.425500+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.757807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425336+0000","last_change":"2024-09-17T04:05:25.223284+0000","last_active":"2024-09-17T04:05:26.425336+0000","last_peered":"2024-09-17T04:05:26.425336+0000","last_clean":"2024-09-17T04:05:26.425336+0000","last_became_active":"2024-09-17T04:05:25.221020+0000","last_became_peered":"2024-09-17T04:05:25.221020+0000","last_unstale":"2024-09-17T04:05:26.425336+0000","last_undegraded":"2024-09-17T04:05:26.425336+0000","last_fullsized":"2024-09-17T04:05:26.425336+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:09.253541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500358+0000","last_change":"2024-09-17T04:05:25.226879+0000","last_active":"2024-09-17T04:05:26.500358+0000","last_peered":"2024-09-17T04:05:26.500358+0000","last_clean":"2024-09-17T04:05:26.500358+0000","last_became_active":"2024-09-17T04:05:25.226571+0000","last_became_peered":"2024-09-17T04:05:25.226571+0000","last_unstale":"2024-09-17T04:05:26.500358+0000","last_undegraded":"2024-09-17T04:05:26.500358+0000","last_fullsized":"2024-09-17T04:05:26.500358+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:45.293995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225827+0000","last_change":"2024-09-17T04:05:25.229114+0000","last_active":"2024-09-17T04:05:26.225827+0000","last_peered":"2024-09-17T04:05:26.225827+0000","last_clean":"2024-09-17T04:05:26.225827+0000","last_became_active":"2024-09-17T04:05:25.228819+0000","last_became_peered":"2024-09-17T04:05:25.228819+0000","last_unstale":"2024-09-17T04:05:26.225827+0000","last_undegraded":"2024-09-17T04:05:26.225827+0000","last_fullsized":"2024-09-17T04:05:26.225827+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:48.170272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228576+0000","last_change":"2024-09-17T04:05:25.244112+0000","last_active":"2024-09-17T04:05:26.228576+0000","last_peered":"2024-09-17T04:05:26.228576+0000","last_clean":"2024-09-17T04:05:26.228576+0000","last_became_active":"2024-09-17T04:05:25.243746+0000","last_became_peered":"2024-09-17T04:05:25.243746+0000","last_unstale":"2024-09-17T04:05:26.228576+0000","last_undegraded":"2024-09-17T04:05:26.228576+0000","last_fullsized":"2024-09-17T04:05:26.228576+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228461+0000","last_change":"2024-09-17T04:05:25.236197+0000","last_active":"2024-09-17T04:05:26.228461+0000","last_peered":"2024-09-17T04:05:26.228461+0000","last_clean":"2024-09-17T04:05:26.228461+0000","last_became_active":"2024-09-17T04:05:25.233570+0000","last_became_peered":"2024-09-17T04:05:25.233570+0000","last_unstale":"2024-09-17T04:05:26.228461+0000","last_undegraded":"2024-09-17T04:05:26.228461+0000","last_fullsized":"2024-09-17T04:05:26.228461+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:52.842381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500441+0000","last_change":"2024-09-17T04:05:25.230257+0000","last_active":"2024-09-17T04:05:26.500441+0000","last_peered":"2024-09-17T04:05:26.500441+0000","last_clean":"2024-09-17T04:05:26.500441+0000","last_became_active":"2024-09-17T04:05:25.229043+0000","last_became_peered":"2024-09-17T04:05:25.229043+0000","last_unstale":"2024-09-17T04:05:26.500441+0000","last_undegraded":"2024-09-17T04:05:26.500441+0000","last_fullsized":"2024-09-17T04:05:26.500441+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:03.038463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228701+0000","last_change":"2024-09-17T04:05:25.242725+0000","last_active":"2024-09-17T04:05:26.228701+0000","last_peered":"2024-09-17T04:05:26.228701+0000","last_clean":"2024-09-17T04:05:26.228701+0000","last_became_active":"2024-09-17T04:05:25.239325+0000","last_became_peered":"2024-09-17T04:05:25.239325+0000","last_unstale":"2024-09-17T04:05:26.228701+0000","last_undegraded":"2024-09-17T04:05:26.228701+0000","last_fullsized":"2024-09-17T04:05:26.228701+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:29.392591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226066+0000","last_change":"2024-09-17T04:05:25.226877+0000","last_active":"2024-09-17T04:05:26.226066+0000","last_peered":"2024-09-17T04:05:26.226066+0000","last_clean":"2024-09-17T04:05:26.226066+0000","last_became_active":"2024-09-17T04:05:25.226520+0000","last_became_peered":"2024-09-17T04:05:25.226520+0000","last_unstale":"2024-09-17T04:05:26.226066+0000","last_undegraded":"2024-09-17T04:05:26.226066+0000","last_fullsized":"2024-09-17T04:05:26.226066+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:15.166447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228380+0000","last_change":"2024-09-17T04:05:25.232439+0000","last_active":"2024-09-17T04:05:26.228380+0000","last_peered":"2024-09-17T04:05:26.228380+0000","last_clean":"2024-09-17T04:05:26.228380+0000","last_became_active":"2024-09-17T04:05:25.230995+0000","last_became_peered":"2024-09-17T04:05:25.230995+0000","last_unstale":"2024-09-17T04:05:26.228380+0000","last_undegraded":"2024-09-17T04:05:26.228380+0000","last_fullsized":"2024-09-17T04:05:26.228380+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:49.397560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499086+0000","last_change":"2024-09-17T04:05:25.225290+0000","last_active":"2024-09-17T04:05:26.499086+0000","last_peered":"2024-09-17T04:05:26.499086+0000","last_clean":"2024-09-17T04:05:26.499086+0000","last_became_active":"2024-09-17T04:05:25.224935+0000","last_became_peered":"2024-09-17T04:05:25.224935+0000","last_unstale":"2024-09-17T04:05:26.499086+0000","last_undegraded":"2024-09-17T04:05:26.499086+0000","last_fullsized":"2024-09-17T04:05:26.499086+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:00.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500413+0000","last_change":"2024-09-17T04:05:25.231448+0000","last_active":"2024-09-17T04:05:26.500413+0000","last_peered":"2024-09-17T04:05:26.500413+0000","last_clean":"2024-09-17T04:05:26.500413+0000","last_became_active":"2024-09-17T04:05:25.229864+0000","last_became_peered":"2024-09-17T04:05:25.229864+0000","last_unstale":"2024-09-17T04:05:26.500413+0000","last_undegraded":"2024-09-17T04:05:26.500413+0000","last_fullsized":"2024-09-17T04:05:26.500413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:31.295888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"217'4","reported_seq":19,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226006+0000","last_change":"2024-09-17T04:05:24.220787+0000","last_active":"2024-09-17T04:05:26.226006+0000","last_peered":"2024-09-17T04:05:26.226006+0000","last_clean":"2024-09-17T04:05:26.226006+0000","last_became_active":"2024-09-17T04:05:24.220468+0000","last_became_peered":"2024-09-17T04:05:24.220468+0000","last_unstale":"2024-09-17T04:05:26.226006+0000","last_undegraded":"2024-09-17T04:05:26.226006+0000","last_fullsized":"2024-09-17T04:05:26.226006+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:23.868440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228637+0000","last_change":"2024-09-17T04:05:25.235813+0000","last_active":"2024-09-17T04:05:26.228637+0000","last_peered":"2024-09-17T04:05:26.228637+0000","last_clean":"2024-09-17T04:05:26.228637+0000","last_became_active":"2024-09-17T04:05:25.235483+0000","last_became_peered":"2024-09-17T04:05:25.235483+0000","last_unstale":"2024-09-17T04:05:26.228637+0000","last_undegraded":"2024-09-17T04:05:26.228637+0000","last_fullsized":"2024-09-17T04:05:26.228637+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:06.428970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228581+0000","last_change":"2024-09-17T04:05:24.231055+0000","last_active":"2024-09-17T04:05:26.228581+0000","last_peered":"2024-09-17T04:05:26.228581+0000","last_clean":"2024-09-17T04:05:26.228581+0000","last_became_active":"2024-09-17T04:05:24.222063+0000","last_became_peered":"2024-09-17T04:05:24.222063+0000","last_unstale":"2024-09-17T04:05:26.228581+0000","last_undegraded":"2024-09-17T04:05:26.228581+0000","last_fullsized":"2024-09-17T04:05:26.228581+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:35.224862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500629+0000","last_change":"2024-09-17T04:05:25.227705+0000","last_active":"2024-09-17T04:05:26.500629+0000","last_peered":"2024-09-17T04:05:26.500629+0000","last_clean":"2024-09-17T04:05:26.500629+0000","last_became_active":"2024-09-17T04:05:25.223453+0000","last_became_peered":"2024-09-17T04:05:25.223453+0000","last_unstale":"2024-09-17T04:05:26.500629+0000","last_undegraded":"2024-09-17T04:05:26.500629+0000","last_fullsized":"2024-09-17T04:05:26.500629+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:26.267555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437169+0000","last_change":"2024-09-17T04:05:24.228243+0000","last_active":"2024-09-17T04:05:26.437169+0000","last_peered":"2024-09-17T04:05:26.437169+0000","last_clean":"2024-09-17T04:05:26.437169+0000","last_became_active":"2024-09-17T04:05:24.227852+0000","last_became_peered":"2024-09-17T04:05:24.227852+0000","last_unstale":"2024-09-17T04:05:26.437169+0000","last_undegraded":"2024-09-17T04:05:26.437169+0000","last_fullsized":"2024-09-17T04:05:26.437169+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:02.226834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500033+0000","last_change":"2024-09-17T04:05:25.225950+0000","last_active":"2024-09-17T04:05:26.500033+0000","last_peered":"2024-09-17T04:05:26.500033+0000","last_clean":"2024-09-17T04:05:26.500033+0000","last_became_active":"2024-09-17T04:05:25.225457+0000","last_became_peered":"2024-09-17T04:05:25.225457+0000","last_unstale":"2024-09-17T04:05:26.500033+0000","last_undegraded":"2024-09-17T04:05:26.500033+0000","last_fullsized":"2024-09-17T04:05:26.500033+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:41.166197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437014+0000","last_change":"2024-09-17T04:05:24.226740+0000","last_active":"2024-09-17T04:05:26.437014+0000","last_peered":"2024-09-17T04:05:26.437014+0000","last_clean":"2024-09-17T04:05:26.437014+0000","last_became_active":"2024-09-17T04:05:24.226406+0000","last_became_peered":"2024-09-17T04:05:24.226406+0000","last_unstale":"2024-09-17T04:05:26.437014+0000","last_undegraded":"2024-09-17T04:05:26.437014+0000","last_fullsized":"2024-09-17T04:05:26.437014+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:51.681172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425319+0000","last_change":"2024-09-17T04:05:25.232667+0000","last_active":"2024-09-17T04:05:26.425319+0000","last_peered":"2024-09-17T04:05:26.425319+0000","last_clean":"2024-09-17T04:05:26.425319+0000","last_became_active":"2024-09-17T04:05:25.232379+0000","last_became_peered":"2024-09-17T04:05:25.232379+0000","last_unstale":"2024-09-17T04:05:26.425319+0000","last_undegraded":"2024-09-17T04:05:26.425319+0000","last_fullsized":"2024-09-17T04:05:26.425319+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:20.607889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499657+0000","last_change":"2024-09-17T04:05:24.224011+0000","last_active":"2024-09-17T04:05:26.499657+0000","last_peered":"2024-09-17T04:05:26.499657+0000","last_clean":"2024-09-17T04:05:26.499657+0000","last_became_active":"2024-09-17T04:05:24.223670+0000","last_became_peered":"2024-09-17T04:05:24.223670+0000","last_unstale":"2024-09-17T04:05:26.499657+0000","last_undegraded":"2024-09-17T04:05:26.499657+0000","last_fullsized":"2024-09-17T04:05:26.499657+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.853028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436826+0000","last_change":"2024-09-17T04:05:25.220380+0000","last_active":"2024-09-17T04:05:26.436826+0000","last_peered":"2024-09-17T04:05:26.436826+0000","last_clean":"2024-09-17T04:05:26.436826+0000","last_became_active":"2024-09-17T04:05:25.217028+0000","last_became_peered":"2024-09-17T04:05:25.217028+0000","last_unstale":"2024-09-17T04:05:26.436826+0000","last_undegraded":"2024-09-17T04:05:26.436826+0000","last_fullsized":"2024-09-17T04:05:26.436826+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:44.696425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225677+0000","last_change":"2024-09-17T04:05:24.221715+0000","last_active":"2024-09-17T04:05:26.225677+0000","last_peered":"2024-09-17T04:05:26.225677+0000","last_clean":"2024-09-17T04:05:26.225677+0000","last_became_active":"2024-09-17T04:05:24.221380+0000","last_became_peered":"2024-09-17T04:05:24.221380+0000","last_unstale":"2024-09-17T04:05:26.225677+0000","last_undegraded":"2024-09-17T04:05:26.225677+0000","last_fullsized":"2024-09-17T04:05:26.225677+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:33.998165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436516+0000","last_change":"2024-09-17T04:05:25.213295+0000","last_active":"2024-09-17T04:05:26.436516+0000","last_peered":"2024-09-17T04:05:26.436516+0000","last_clean":"2024-09-17T04:05:26.436516+0000","last_became_active":"2024-09-17T04:05:25.212970+0000","last_became_peered":"2024-09-17T04:05:25.212970+0000","last_unstale":"2024-09-17T04:05:26.436516+0000","last_undegraded":"2024-09-17T04:05:26.436516+0000","last_fullsized":"2024-09-17T04:05:26.436516+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:01.839472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228120+0000","last_change":"2024-09-17T04:05:24.234352+0000","last_active":"2024-09-17T04:05:26.228120+0000","last_peered":"2024-09-17T04:05:26.228120+0000","last_clean":"2024-09-17T04:05:26.228120+0000","last_became_active":"2024-09-17T04:05:24.234071+0000","last_became_peered":"2024-09-17T04:05:24.234071+0000","last_unstale":"2024-09-17T04:05:26.228120+0000","last_undegraded":"2024-09-17T04:05:26.228120+0000","last_fullsized":"2024-09-17T04:05:26.228120+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:37.269558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500626+0000","last_change":"2024-09-17T04:05:25.219129+0000","last_active":"2024-09-17T04:05:26.500626+0000","last_peered":"2024-09-17T04:05:26.500626+0000","last_clean":"2024-09-17T04:05:26.500626+0000","last_became_active":"2024-09-17T04:05:25.217644+0000","last_became_peered":"2024-09-17T04:05:25.217644+0000","last_unstale":"2024-09-17T04:05:26.500626+0000","last_undegraded":"2024-09-17T04:05:26.500626+0000","last_fullsized":"2024-09-17T04:05:26.500626+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:52.028296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499665+0000","last_change":"2024-09-17T04:05:24.224568+0000","last_active":"2024-09-17T04:05:26.499665+0000","last_peered":"2024-09-17T04:05:26.499665+0000","last_clean":"2024-09-17T04:05:26.499665+0000","last_became_active":"2024-09-17T04:05:24.223638+0000","last_became_peered":"2024-09-17T04:05:24.223638+0000","last_unstale":"2024-09-17T04:05:26.499665+0000","last_undegraded":"2024-09-17T04:05:26.499665+0000","last_fullsized":"2024-09-17T04:05:26.499665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:49.701292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500661+0000","last_change":"2024-09-17T04:05:25.218519+0000","last_active":"2024-09-17T04:05:26.500661+0000","last_peered":"2024-09-17T04:05:26.500661+0000","last_clean":"2024-09-17T04:05:26.500661+0000","last_became_active":"2024-09-17T04:05:25.217254+0000","last_became_peered":"2024-09-17T04:05:25.217254+0000","last_unstale":"2024-09-17T04:05:26.500661+0000","last_undegraded":"2024-09-17T04:05:26.500661+0000","last_fullsized":"2024-09-17T04:05:26.500661+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:49.680789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226069+0000","last_change":"2024-09-17T04:05:24.223519+0000","last_active":"2024-09-17T04:05:26.226069+0000","last_peered":"2024-09-17T04:05:26.226069+0000","last_clean":"2024-09-17T04:05:26.226069+0000","last_became_active":"2024-09-17T04:05:24.223231+0000","last_became_peered":"2024-09-17T04:05:24.223231+0000","last_unstale":"2024-09-17T04:05:26.226069+0000","last_undegraded":"2024-09-17T04:05:26.226069+0000","last_fullsized":"2024-09-17T04:05:26.226069+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:42.041674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225715+0000","last_change":"2024-09-17T04:05:25.221512+0000","last_active":"2024-09-17T04:05:26.225715+0000","last_peered":"2024-09-17T04:05:26.225715+0000","last_clean":"2024-09-17T04:05:26.225715+0000","last_became_active":"2024-09-17T04:05:25.220458+0000","last_became_peered":"2024-09-17T04:05:25.220458+0000","last_unstale":"2024-09-17T04:05:26.225715+0000","last_undegraded":"2024-09-17T04:05:26.225715+0000","last_fullsized":"2024-09-17T04:05:26.225715+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:55.416231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436765+0000","last_change":"2024-09-17T04:05:24.226942+0000","last_active":"2024-09-17T04:05:26.436765+0000","last_peered":"2024-09-17T04:05:26.436765+0000","last_clean":"2024-09-17T04:05:26.436765+0000","last_became_active":"2024-09-17T04:05:24.226597+0000","last_became_peered":"2024-09-17T04:05:24.226597+0000","last_unstale":"2024-09-17T04:05:26.436765+0000","last_undegraded":"2024-09-17T04:05:26.436765+0000","last_fullsized":"2024-09-17T04:05:26.436765+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:12.017426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228414+0000","last_change":"2024-09-17T04:05:25.227178+0000","last_active":"2024-09-17T04:05:26.228414+0000","last_peered":"2024-09-17T04:05:26.228414+0000","last_clean":"2024-09-17T04:05:26.228414+0000","last_became_active":"2024-09-17T04:05:25.226791+0000","last_became_peered":"2024-09-17T04:05:25.226791+0000","last_unstale":"2024-09-17T04:05:26.228414+0000","last_undegraded":"2024-09-17T04:05:26.228414+0000","last_fullsized":"2024-09-17T04:05:26.228414+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:24.402863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"217'3","reported_seq":18,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437223+0000","last_change":"2024-09-17T04:05:24.230056+0000","last_active":"2024-09-17T04:05:26.437223+0000","last_peered":"2024-09-17T04:05:26.437223+0000","last_clean":"2024-09-17T04:05:26.437223+0000","last_became_active":"2024-09-17T04:05:24.229748+0000","last_became_peered":"2024-09-17T04:05:24.229748+0000","last_unstale":"2024-09-17T04:05:26.437223+0000","last_undegraded":"2024-09-17T04:05:26.437223+0000","last_fullsized":"2024-09-17T04:05:26.437223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:19.117891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225488+0000","last_change":"2024-09-17T04:05:25.220710+0000","last_active":"2024-09-17T04:05:26.225488+0000","last_peered":"2024-09-17T04:05:26.225488+0000","last_clean":"2024-09-17T04:05:26.225488+0000","last_became_active":"2024-09-17T04:05:25.220415+0000","last_became_peered":"2024-09-17T04:05:25.220415+0000","last_unstale":"2024-09-17T04:05:26.225488+0000","last_undegraded":"2024-09-17T04:05:26.225488+0000","last_fullsized":"2024-09-17T04:05:26.225488+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:46.853834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"217'6","reported_seq":21,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225389+0000","last_change":"2024-09-17T04:05:24.225084+0000","last_active":"2024-09-17T04:05:26.225389+0000","last_peered":"2024-09-17T04:05:26.225389+0000","last_clean":"2024-09-17T04:05:26.225389+0000","last_became_active":"2024-09-17T04:05:24.224781+0000","last_became_peered":"2024-09-17T04:05:24.224781+0000","last_unstale":"2024-09-17T04:05:26.225389+0000","last_undegraded":"2024-09-17T04:05:26.225389+0000","last_fullsized":"2024-09-17T04:05:26.225389+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:21.718712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225506+0000","last_change":"2024-09-17T04:05:25.219009+0000","last_active":"2024-09-17T04:05:26.225506+0000","last_peered":"2024-09-17T04:05:26.225506+0000","last_clean":"2024-09-17T04:05:26.225506+0000","last_became_active":"2024-09-17T04:05:25.218697+0000","last_became_peered":"2024-09-17T04:05:25.218697+0000","last_unstale":"2024-09-17T04:05:26.225506+0000","last_undegraded":"2024-09-17T04:05:26.225506+0000","last_fullsized":"2024-09-17T04:05:26.225506+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:46.772065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500062+0000","last_change":"2024-09-17T04:05:24.223126+0000","last_active":"2024-09-17T04:05:26.500062+0000","last_peered":"2024-09-17T04:05:26.500062+0000","last_clean":"2024-09-17T04:05:26.500062+0000","last_became_active":"2024-09-17T04:05:24.221114+0000","last_became_peered":"2024-09-17T04:05:24.221114+0000","last_unstale":"2024-09-17T04:05:26.500062+0000","last_undegraded":"2024-09-17T04:05:26.500062+0000","last_fullsized":"2024-09-17T04:05:26.500062+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:13.552456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436876+0000","last_change":"2024-09-17T04:05:25.213282+0000","last_active":"2024-09-17T04:05:26.436876+0000","last_peered":"2024-09-17T04:05:26.436876+0000","last_clean":"2024-09-17T04:05:26.436876+0000","last_became_active":"2024-09-17T04:05:25.212938+0000","last_became_peered":"2024-09-17T04:05:25.212938+0000","last_unstale":"2024-09-17T04:05:26.436876+0000","last_undegraded":"2024-09-17T04:05:26.436876+0000","last_fullsized":"2024-09-17T04:05:26.436876+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:26.820397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425400+0000","last_change":"2024-09-17T04:05:24.223560+0000","last_active":"2024-09-17T04:05:26.425400+0000","last_peered":"2024-09-17T04:05:26.425400+0000","last_clean":"2024-09-17T04:05:26.425400+0000","last_became_active":"2024-09-17T04:05:24.223302+0000","last_became_peered":"2024-09-17T04:05:24.223302+0000","last_unstale":"2024-09-17T04:05:26.425400+0000","last_undegraded":"2024-09-17T04:05:26.425400+0000","last_fullsized":"2024-09-17T04:05:26.425400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:25.259928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225342+0000","last_change":"2024-09-17T04:05:25.231922+0000","last_active":"2024-09-17T04:05:26.225342+0000","last_peered":"2024-09-17T04:05:26.225342+0000","last_clean":"2024-09-17T04:05:26.225342+0000","last_became_active":"2024-09-17T04:05:25.231099+0000","last_became_peered":"2024-09-17T04:05:25.231099+0000","last_unstale":"2024-09-17T04:05:26.225342+0000","last_undegraded":"2024-09-17T04:05:26.225342+0000","last_fullsized":"2024-09-17T04:05:26.225342+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:29:35.958163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499298+0000","last_change":"2024-09-17T04:05:24.231560+0000","last_active":"2024-09-17T04:05:26.499298+0000","last_peered":"2024-09-17T04:05:26.499298+0000","last_clean":"2024-09-17T04:05:26.499298+0000","last_became_active":"2024-09-17T04:05:24.231172+0000","last_became_peered":"2024-09-17T04:05:24.231172+0000","last_unstale":"2024-09-17T04:05:26.499298+0000","last_undegraded":"2024-09-17T04:05:26.499298+0000","last_fullsized":"2024-09-17T04:05:26.499298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:17.113825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225269+0000","last_change":"2024-09-17T04:05:25.228578+0000","last_active":"2024-09-17T04:05:26.225269+0000","last_peered":"2024-09-17T04:05:26.225269+0000","last_clean":"2024-09-17T04:05:26.225269+0000","last_became_active":"2024-09-17T04:05:25.228179+0000","last_became_peered":"2024-09-17T04:05:25.228179+0000","last_unstale":"2024-09-17T04:05:26.225269+0000","last_undegraded":"2024-09-17T04:05:26.225269+0000","last_fullsized":"2024-09-17T04:05:26.225269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:08.402904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"217'8","reported_seq":23,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436709+0000","last_change":"2024-09-17T04:05:24.229283+0000","last_active":"2024-09-17T04:05:26.436709+0000","last_peered":"2024-09-17T04:05:26.436709+0000","last_clean":"2024-09-17T04:05:26.436709+0000","last_became_active":"2024-09-17T04:05:24.228948+0000","last_became_peered":"2024-09-17T04:05:24.228948+0000","last_unstale":"2024-09-17T04:05:26.436709+0000","last_undegraded":"2024-09-17T04:05:26.436709+0000","last_fullsized":"2024-09-17T04:05:26.436709+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:17.532872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228381+0000","last_change":"2024-09-17T04:05:25.229771+0000","last_active":"2024-09-17T04:05:26.228381+0000","last_peered":"2024-09-17T04:05:26.228381+0000","last_clean":"2024-09-17T04:05:26.228381+0000","last_became_active":"2024-09-17T04:05:25.229452+0000","last_became_peered":"2024-09-17T04:05:25.229452+0000","last_unstale":"2024-09-17T04:05:26.228381+0000","last_undegraded":"2024-09-17T04:05:26.228381+0000","last_fullsized":"2024-09-17T04:05:26.228381+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:39.818419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500697+0000","last_change":"2024-09-17T04:05:24.222556+0000","last_active":"2024-09-17T04:05:26.500697+0000","last_peered":"2024-09-17T04:05:26.500697+0000","last_clean":"2024-09-17T04:05:26.500697+0000","last_became_active":"2024-09-17T04:05:24.222221+0000","last_became_peered":"2024-09-17T04:05:24.222221+0000","last_unstale":"2024-09-17T04:05:26.500697+0000","last_undegraded":"2024-09-17T04:05:26.500697+0000","last_fullsized":"2024-09-17T04:05:26.500697+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:20.842936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.501041+0000","last_change":"2024-09-17T04:05:25.244458+0000","last_active":"2024-09-17T04:05:26.501041+0000","last_peered":"2024-09-17T04:05:26.501041+0000","last_clean":"2024-09-17T04:05:26.501041+0000","last_became_active":"2024-09-17T04:05:25.244033+0000","last_became_peered":"2024-09-17T04:05:25.244033+0000","last_unstale":"2024-09-17T04:05:26.501041+0000","last_undegraded":"2024-09-17T04:05:26.501041+0000","last_fullsized":"2024-09-17T04:05:26.501041+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:00.213217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436875+0000","last_change":"2024-09-17T04:05:24.226050+0000","last_active":"2024-09-17T04:05:26.436875+0000","last_peered":"2024-09-17T04:05:26.436875+0000","last_clean":"2024-09-17T04:05:26.436875+0000","last_became_active":"2024-09-17T04:05:24.225367+0000","last_became_peered":"2024-09-17T04:05:24.225367+0000","last_unstale":"2024-09-17T04:05:26.436875+0000","last_undegraded":"2024-09-17T04:05:26.436875+0000","last_fullsized":"2024-09-17T04:05:26.436875+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:24.613188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228770+0000","last_change":"2024-09-17T04:05:25.226348+0000","last_active":"2024-09-17T04:05:26.228770+0000","last_peered":"2024-09-17T04:05:26.228770+0000","last_clean":"2024-09-17T04:05:26.228770+0000","last_became_active":"2024-09-17T04:05:25.225598+0000","last_became_peered":"2024-09-17T04:05:25.225598+0000","last_unstale":"2024-09-17T04:05:26.228770+0000","last_undegraded":"2024-09-17T04:05:26.228770+0000","last_fullsized":"2024-09-17T04:05:26.228770+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:59.151440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500308+0000","last_change":"2024-09-17T04:05:24.226944+0000","last_active":"2024-09-17T04:05:26.500308+0000","last_peered":"2024-09-17T04:05:26.500308+0000","last_clean":"2024-09-17T04:05:26.500308+0000","last_became_active":"2024-09-17T04:05:24.225543+0000","last_became_peered":"2024-09-17T04:05:24.225543+0000","last_unstale":"2024-09-17T04:05:26.500308+0000","last_undegraded":"2024-09-17T04:05:26.500308+0000","last_fullsized":"2024-09-17T04:05:26.500308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.257988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228745+0000","last_change":"2024-09-17T04:05:25.242208+0000","last_active":"2024-09-17T04:05:26.228745+0000","last_peered":"2024-09-17T04:05:26.228745+0000","last_clean":"2024-09-17T04:05:26.228745+0000","last_became_active":"2024-09-17T04:05:25.241886+0000","last_became_peered":"2024-09-17T04:05:25.241886+0000","last_unstale":"2024-09-17T04:05:26.228745+0000","last_undegraded":"2024-09-17T04:05:26.228745+0000","last_fullsized":"2024-09-17T04:05:26.228745+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228869+0000","last_change":"2024-09-17T04:05:24.229174+0000","last_active":"2024-09-17T04:05:26.228869+0000","last_peered":"2024-09-17T04:05:26.228869+0000","last_clean":"2024-09-17T04:05:26.228869+0000","last_became_active":"2024-09-17T04:05:24.228858+0000","last_became_peered":"2024-09-17T04:05:24.228858+0000","last_unstale":"2024-09-17T04:05:26.228869+0000","last_undegraded":"2024-09-17T04:05:26.228869+0000","last_fullsized":"2024-09-17T04:05:26.228869+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.505474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436624+0000","last_change":"2024-09-17T04:05:25.226304+0000","last_active":"2024-09-17T04:05:26.436624+0000","last_peered":"2024-09-17T04:05:26.436624+0000","last_clean":"2024-09-17T04:05:26.436624+0000","last_became_active":"2024-09-17T04:05:25.224061+0000","last_became_peered":"2024-09-17T04:05:25.224061+0000","last_unstale":"2024-09-17T04:05:26.436624+0000","last_undegraded":"2024-09-17T04:05:26.436624+0000","last_fullsized":"2024-09-17T04:05:26.436624+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:55:49.229679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228504+0000","last_change":"2024-09-17T04:05:24.225036+0000","last_active":"2024-09-17T04:05:26.228504+0000","last_peered":"2024-09-17T04:05:26.228504+0000","last_clean":"2024-09-17T04:05:26.228504+0000","last_became_active":"2024-09-17T04:05:24.223209+0000","last_became_peered":"2024-09-17T04:05:24.223209+0000","last_unstale":"2024-09-17T04:05:26.228504+0000","last_undegraded":"2024-09-17T04:05:26.228504+0000","last_fullsized":"2024-09-17T04:05:26.228504+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:21.294118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437062+0000","last_change":"2024-09-17T04:05:25.221299+0000","last_active":"2024-09-17T04:05:26.437062+0000","last_peered":"2024-09-17T04:05:26.437062+0000","last_clean":"2024-09-17T04:05:26.437062+0000","last_became_active":"2024-09-17T04:05:25.220970+0000","last_became_peered":"2024-09-17T04:05:25.220970+0000","last_unstale":"2024-09-17T04:05:26.437062+0000","last_undegraded":"2024-09-17T04:05:26.437062+0000","last_fullsized":"2024-09-17T04:05:26.437062+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:28.900393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499673+0000","last_change":"2024-09-17T04:05:24.224549+0000","last_active":"2024-09-17T04:05:26.499673+0000","last_peered":"2024-09-17T04:05:26.499673+0000","last_clean":"2024-09-17T04:05:26.499673+0000","last_became_active":"2024-09-17T04:05:24.224066+0000","last_became_peered":"2024-09-17T04:05:24.224066+0000","last_unstale":"2024-09-17T04:05:26.499673+0000","last_undegraded":"2024-09-17T04:05:26.499673+0000","last_fullsized":"2024-09-17T04:05:26.499673+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:26.556703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225575+0000","last_change":"2024-09-17T04:05:25.221813+0000","last_active":"2024-09-17T04:05:26.225575+0000","last_peered":"2024-09-17T04:05:26.225575+0000","last_clean":"2024-09-17T04:05:26.225575+0000","last_became_active":"2024-09-17T04:05:25.221550+0000","last_became_peered":"2024-09-17T04:05:25.221550+0000","last_unstale":"2024-09-17T04:05:26.225575+0000","last_undegraded":"2024-09-17T04:05:26.225575+0000","last_fullsized":"2024-09-17T04:05:26.225575+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:42.072358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499425+0000","last_change":"2024-09-17T04:05:24.229963+0000","last_active":"2024-09-17T04:05:26.499425+0000","last_peered":"2024-09-17T04:05:26.499425+0000","last_clean":"2024-09-17T04:05:26.499425+0000","last_became_active":"2024-09-17T04:05:24.229600+0000","last_became_peered":"2024-09-17T04:05:24.229600+0000","last_unstale":"2024-09-17T04:05:26.499425+0000","last_undegraded":"2024-09-17T04:05:26.499425+0000","last_fullsized":"2024-09-17T04:05:26.499425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:45.305405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425310+0000","last_change":"2024-09-17T04:05:25.238027+0000","last_active":"2024-09-17T04:05:26.425310+0000","last_peered":"2024-09-17T04:05:26.425310+0000","last_clean":"2024-09-17T04:05:26.425310+0000","last_became_active":"2024-09-17T04:05:25.236850+0000","last_became_peered":"2024-09-17T04:05:25.236850+0000","last_unstale":"2024-09-17T04:05:26.425310+0000","last_undegraded":"2024-09-17T04:05:26.425310+0000","last_fullsized":"2024-09-17T04:05:26.425310+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:04.588444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499682+0000","last_change":"2024-09-17T04:05:24.224522+0000","last_active":"2024-09-17T04:05:26.499682+0000","last_peered":"2024-09-17T04:05:26.499682+0000","last_clean":"2024-09-17T04:05:26.499682+0000","last_became_active":"2024-09-17T04:05:24.224171+0000","last_became_peered":"2024-09-17T04:05:24.224171+0000","last_unstale":"2024-09-17T04:05:26.499682+0000","last_undegraded":"2024-09-17T04:05:26.499682+0000","last_fullsized":"2024-09-17T04:05:26.499682+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:08.223211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228423+0000","last_change":"2024-09-17T04:05:25.238743+0000","last_active":"2024-09-17T04:05:26.228423+0000","last_peered":"2024-09-17T04:05:26.228423+0000","last_clean":"2024-09-17T04:05:26.228423+0000","last_became_active":"2024-09-17T04:05:25.233859+0000","last_became_peered":"2024-09-17T04:05:25.233859+0000","last_unstale":"2024-09-17T04:05:26.228423+0000","last_undegraded":"2024-09-17T04:05:26.228423+0000","last_fullsized":"2024-09-17T04:05:26.228423+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.941861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228763+0000","last_change":"2024-09-17T04:05:24.232297+0000","last_active":"2024-09-17T04:05:26.228763+0000","last_peered":"2024-09-17T04:05:26.228763+0000","last_clean":"2024-09-17T04:05:26.228763+0000","last_became_active":"2024-09-17T04:05:24.229674+0000","last_became_peered":"2024-09-17T04:05:24.229674+0000","last_unstale":"2024-09-17T04:05:26.228763+0000","last_undegraded":"2024-09-17T04:05:26.228763+0000","last_fullsized":"2024-09-17T04:05:26.228763+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:16.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225858+0000","last_change":"2024-09-17T04:05:25.225732+0000","last_active":"2024-09-17T04:05:26.225858+0000","last_peered":"2024-09-17T04:05:26.225858+0000","last_clean":"2024-09-17T04:05:26.225858+0000","last_became_active":"2024-09-17T04:05:25.225325+0000","last_became_peered":"2024-09-17T04:05:25.225325+0000","last_unstale":"2024-09-17T04:05:26.225858+0000","last_undegraded":"2024-09-17T04:05:26.225858+0000","last_fullsized":"2024-09-17T04:05:26.225858+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:08.117034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"218'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228547+0000","last_change":"2024-09-17T04:05:24.232345+0000","last_active":"2024-09-17T04:05:26.228547+0000","last_peered":"2024-09-17T04:05:26.228547+0000","last_clean":"2024-09-17T04:05:26.228547+0000","last_became_active":"2024-09-17T04:05:24.231921+0000","last_became_peered":"2024-09-17T04:05:24.231921+0000","last_unstale":"2024-09-17T04:05:26.228547+0000","last_undegraded":"2024-09-17T04:05:26.228547+0000","last_fullsized":"2024-09-17T04:05:26.228547+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:22:05.051896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228618+0000","last_change":"2024-09-17T04:05:25.241564+0000","last_active":"2024-09-17T04:05:26.228618+0000","last_peered":"2024-09-17T04:05:26.228618+0000","last_clean":"2024-09-17T04:05:26.228618+0000","last_became_active":"2024-09-17T04:05:25.241265+0000","last_became_peered":"2024-09-17T04:05:25.241265+0000","last_unstale":"2024-09-17T04:05:26.228618+0000","last_undegraded":"2024-09-17T04:05:26.228618+0000","last_fullsized":"2024-09-17T04:05:26.228618+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:41.708564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500526+0000","last_change":"2024-09-17T04:05:24.227599+0000","last_active":"2024-09-17T04:05:26.500526+0000","last_peered":"2024-09-17T04:05:26.500526+0000","last_clean":"2024-09-17T04:05:26.500526+0000","last_became_active":"2024-09-17T04:05:24.225987+0000","last_became_peered":"2024-09-17T04:05:24.225987+0000","last_unstale":"2024-09-17T04:05:26.500526+0000","last_undegraded":"2024-09-17T04:05:26.500526+0000","last_fullsized":"2024-09-17T04:05:26.500526+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:05.381905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228927+0000","last_change":"2024-09-17T04:05:25.235165+0000","last_active":"2024-09-17T04:05:26.228927+0000","last_peered":"2024-09-17T04:05:26.228927+0000","last_clean":"2024-09-17T04:05:26.228927+0000","last_became_active":"2024-09-17T04:05:25.233933+0000","last_became_peered":"2024-09-17T04:05:25.233933+0000","last_unstale":"2024-09-17T04:05:26.228927+0000","last_undegraded":"2024-09-17T04:05:26.228927+0000","last_fullsized":"2024-09-17T04:05:26.228927+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:52.986685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225870+0000","last_change":"2024-09-17T04:05:24.222480+0000","last_active":"2024-09-17T04:05:26.225870+0000","last_peered":"2024-09-17T04:05:26.225870+0000","last_clean":"2024-09-17T04:05:26.225870+0000","last_became_active":"2024-09-17T04:05:24.222139+0000","last_became_peered":"2024-09-17T04:05:24.222139+0000","last_unstale":"2024-09-17T04:05:26.225870+0000","last_undegraded":"2024-09-17T04:05:26.225870+0000","last_fullsized":"2024-09-17T04:05:26.225870+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:06.341706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228474+0000","last_change":"2024-09-17T04:05:25.235547+0000","last_active":"2024-09-17T04:05:26.228474+0000","last_peered":"2024-09-17T04:05:26.228474+0000","last_clean":"2024-09-17T04:05:26.228474+0000","last_became_active":"2024-09-17T04:05:25.235183+0000","last_became_peered":"2024-09-17T04:05:25.235183+0000","last_unstale":"2024-09-17T04:05:26.228474+0000","last_undegraded":"2024-09-17T04:05:26.228474+0000","last_fullsized":"2024-09-17T04:05:26.228474+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:06.927656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425138+0000","last_change":"2024-09-17T04:05:24.228230+0000","last_active":"2024-09-17T04:05:26.425138+0000","last_peered":"2024-09-17T04:05:26.425138+0000","last_clean":"2024-09-17T04:05:26.425138+0000","last_became_active":"2024-09-17T04:05:24.227943+0000","last_became_peered":"2024-09-17T04:05:24.227943+0000","last_unstale":"2024-09-17T04:05:26.425138+0000","last_undegraded":"2024-09-17T04:05:26.425138+0000","last_fullsized":"2024-09-17T04:05:26.425138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:30.288261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228341+0000","last_change":"2024-09-17T04:05:25.226060+0000","last_active":"2024-09-17T04:05:26.228341+0000","last_peered":"2024-09-17T04:05:26.228341+0000","last_clean":"2024-09-17T04:05:26.228341+0000","last_became_active":"2024-09-17T04:05:25.225719+0000","last_became_peered":"2024-09-17T04:05:25.225719+0000","last_unstale":"2024-09-17T04:05:26.228341+0000","last_undegraded":"2024-09-17T04:05:26.228341+0000","last_fullsized":"2024-09-17T04:05:26.228341+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:03.563405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425458+0000","last_change":"2024-09-17T04:05:24.223526+0000","last_active":"2024-09-17T04:05:26.425458+0000","last_peered":"2024-09-17T04:05:26.425458+0000","last_clean":"2024-09-17T04:05:26.425458+0000","last_became_active":"2024-09-17T04:05:24.223182+0000","last_became_peered":"2024-09-17T04:05:24.223182+0000","last_unstale":"2024-09-17T04:05:26.425458+0000","last_undegraded":"2024-09-17T04:05:26.425458+0000","last_fullsized":"2024-09-17T04:05:26.425458+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:02.030836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228162+0000","last_change":"2024-09-17T04:05:25.231952+0000","last_active":"2024-09-17T04:05:26.228162+0000","last_peered":"2024-09-17T04:05:26.228162+0000","last_clean":"2024-09-17T04:05:26.228162+0000","last_became_active":"2024-09-17T04:05:25.231626+0000","last_became_peered":"2024-09-17T04:05:25.231626+0000","last_unstale":"2024-09-17T04:05:26.228162+0000","last_undegraded":"2024-09-17T04:05:26.228162+0000","last_fullsized":"2024-09-17T04:05:26.228162+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:50.434212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228121+0000","last_change":"2024-09-17T04:05:24.231774+0000","last_active":"2024-09-17T04:05:26.228121+0000","last_peered":"2024-09-17T04:05:26.228121+0000","last_clean":"2024-09-17T04:05:26.228121+0000","last_became_active":"2024-09-17T04:05:24.231438+0000","last_became_peered":"2024-09-17T04:05:24.231438+0000","last_unstale":"2024-09-17T04:05:26.228121+0000","last_undegraded":"2024-09-17T04:05:26.228121+0000","last_fullsized":"2024-09-17T04:05:26.228121+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:37.625328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228104+0000","last_change":"2024-09-17T04:05:25.248748+0000","last_active":"2024-09-17T04:05:26.228104+0000","last_peered":"2024-09-17T04:05:26.228104+0000","last_clean":"2024-09-17T04:05:26.228104+0000","last_became_active":"2024-09-17T04:05:25.248296+0000","last_became_peered":"2024-09-17T04:05:25.248296+0000","last_unstale":"2024-09-17T04:05:26.228104+0000","last_undegraded":"2024-09-17T04:05:26.228104+0000","last_fullsized":"2024-09-17T04:05:26.228104+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:37.086932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225734+0000","last_change":"2024-09-17T04:05:24.220172+0000","last_active":"2024-09-17T04:05:26.225734+0000","last_peered":"2024-09-17T04:05:26.225734+0000","last_clean":"2024-09-17T04:05:26.225734+0000","last_became_active":"2024-09-17T04:05:24.219869+0000","last_became_peered":"2024-09-17T04:05:24.219869+0000","last_unstale":"2024-09-17T04:05:26.225734+0000","last_undegraded":"2024-09-17T04:05:26.225734+0000","last_fullsized":"2024-09-17T04:05:26.225734+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:18:39.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499599+0000","last_change":"2024-09-17T04:05:25.230118+0000","last_active":"2024-09-17T04:05:26.499599+0000","last_peered":"2024-09-17T04:05:26.499599+0000","last_clean":"2024-09-17T04:05:26.499599+0000","last_became_active":"2024-09-17T04:05:25.229777+0000","last_became_peered":"2024-09-17T04:05:25.229777+0000","last_unstale":"2024-09-17T04:05:26.499599+0000","last_undegraded":"2024-09-17T04:05:26.499599+0000","last_fullsized":"2024-09-17T04:05:26.499599+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:04.076093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":475,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.498879+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:05:26.498879+0000","last_peered":"2024-09-17T04:05:26.498879+0000","last_clean":"2024-09-17T04:05:26.498879+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:05:26.498879+0000","last_undegraded":"2024-09-17T04:05:26.498879+0000","last_fullsized":"2024-09-17T04:05:26.498879+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509635,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3350528,"data_stored":1805685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29716,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342124,"statfs":{"total":96636764160,"available":96606334976,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509637,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2772992,"data_stored":1216915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509635,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56100,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315740,"statfs":{"total":96636764160,"available":96579317760,"internally_reserved":0,"allocated":2768896,"data_stored":1217441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509637,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509637,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2768896,"data_stored":1216023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2760704,"data_stored":1215359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3346432,"data_stored":1805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:05:30.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:05:30.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 -- 172.21.15.129:0/424082206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a815ce50 msgr2=0x7f37a80af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 --2- 172.21.15.129:0/424082206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a815ce50 0x7f37a80af7a0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f3798005fd0 tx=0x7f379802aaa0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 -- 172.21.15.129:0/424082206 shutdown_connections 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 --2- 172.21.15.129:0/424082206 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80cd390 0x7f37a80ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 --2- 172.21.15.129:0/424082206 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a815ce50 0x7f37a80af7a0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 -- 172.21.15.129:0/424082206 >> 172.21.15.129:0/424082206 conn(0x7f37a806d410 msgr2=0x7f37a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 -- 172.21.15.129:0/424082206 shutdown_connections 2024-09-17T04:05:30.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.878+0000 7f37adc00640 1 -- 172.21.15.129:0/424082206 wait complete. 2024-09-17T04:05:30.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.880+0000 7f37adc00640 1 Processor -- start 2024-09-17T04:05:30.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.881+0000 7f37adc00640 1 -- start start 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37a815ce50 0x7f37a81f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80c9bc0 0x7f37a80c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f37a806f8d0 con 0x7f37a80b4180 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f37a806f750 con 0x7f37a815ce50 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37adc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f37a806fa50 con 0x7f37a80c9bc0 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.884 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56850/0 (socket says 172.21.15.129:56850) 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.882+0000 7f37a6a00640 1 -- 172.21.15.129:0/790657356 learned_addr learned my addr 172.21.15.129:0/790657356 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a7400640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80c9bc0 0x7f37a80c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6000640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37a815ce50 0x7f37a81f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 -- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80c9bc0 msgr2=0x7f37a80c8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80c9bc0 0x7f37a80c8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 -- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37a815ce50 msgr2=0x7f37a81f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37a815ce50 0x7f37a81f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 -- 172.21.15.129:0/790657356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81fa0e0 con 0x7f37a80b4180 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37a6a00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f3798000f80 tx=0x7f3798003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798044070 con 0x7f37a80b4180 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3798004250 con 0x7f37a80b4180 2024-09-17T04:05:30.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a809c1b0 con 0x7f37a80b4180 2024-09-17T04:05:30.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.883+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798004660 con 0x7f37a80b4180 2024-09-17T04:05:30.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.884+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a809c780 con 0x7f37a80b4180 2024-09-17T04:05:30.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.885+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a806ff70 con 0x7f37a80b4180 2024-09-17T04:05:30.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.886+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3798047070 con 0x7f37a80b4180 2024-09-17T04:05:30.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.887+0000 7f379ea00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 0x7f378807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:30.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.887+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f37980bbdb0 con 0x7f37a80b4180 2024-09-17T04:05:30.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.887+0000 7f37a6000640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 0x7f378807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:30.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.888+0000 7f37a6000640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 0x7f378807b0a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3794000f60 tx=0x7f3794009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:30.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:30.891+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f37980c0070 con 0x7f37a80b4180 2024-09-17T04:05:31.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.142+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f37a806a140 con 0x7f37a80b4180 2024-09-17T04:05:31.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.143+0000 7f379ea00640 1 -- 172.21.15.129:0/790657356 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1630 (secure 0 0 0) 0x7f37a81fa0e0 con 0x7f37a80b4180 2024-09-17T04:05:31.143 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:31.143 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":982,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":218,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":50},{"state_name":"creating+peering","count":47}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":412172288,"bytes_avail":772681940992,"bytes_total":773094113280,"inactive_pgs_ratio":0.48453608155250549},"fsmap":{"epoch":208,"btime":"2024-09-17T04:05:28:229503+0000","id":16,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":16,"rank":0,"name":"c","status":"up:active","gid":11028}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":31,"modified":"2024-09-17T04:05:24.568269+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 msgr2=0x7f378807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 0x7f378807b0a0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3794000f60 tx=0x7f3794009040 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 msgr2=0x7f37a81f9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7f3798000f80 tx=0x7f3798003f10 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 shutdown_connections 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3788078b90 0x7f378807b0a0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f37a80c9bc0 0x7f37a80c8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f37a815ce50 0x7f37a81f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 --2- 172.21.15.129:0/790657356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f37a80b4180 0x7f37a81f9ba0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 >> 172.21.15.129:0/790657356 conn(0x7f37a806d410 msgr2=0x7f37a80ccbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 shutdown_connections 2024-09-17T04:05:31.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.145+0000 7f37adc00640 1 -- 172.21.15.129:0/790657356 wait complete. 2024-09-17T04:05:31.171 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:05:31.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:05:31.189 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T04:05:31.189+0000 7f269a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 -- 172.21.15.129:0/4266023863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a64005150 msgr2=0x7f2a6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 --2- 172.21.15.129:0/4266023863 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a64005150 0x7f2a6400b9c0 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f2a78002230 tx=0x7f2a64005830 comp rx=0 tx=0).stop 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a7e800640 1 -- 172.21.15.129:0/4266023863 reap_dead start 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 -- 172.21.15.129:0/4266023863 shutdown_connections 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 -- 172.21.15.129:0/4266023863 >> 172.21.15.129:0/4266023863 conn(0x7f2a8006d410 msgr2=0x7f2a800b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 -- 172.21.15.129:0/4266023863 shutdown_connections 2024-09-17T04:05:31.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.394+0000 7f2a85000640 1 -- 172.21.15.129:0/4266023863 wait complete. 2024-09-17T04:05:31.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.397+0000 7f2a85000640 1 Processor -- start 2024-09-17T04:05:31.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.397+0000 7f2a85000640 1 -- start start 2024-09-17T04:05:31.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a64005150 0x7f2a800b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:31.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a800c4f60 0x7f2a800b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:31.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a80161250 con 0x7f2a801f5f80 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a801610d0 con 0x7f2a800c4f60 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.398+0000 7f2a85000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2a801613d0 con 0x7f2a64005150 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:56868/0 (socket says 172.21.15.129:56868) 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 -- 172.21.15.129:0/391178637 learned_addr learned my addr 172.21.15.129:0/391178637 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a75400640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a800c4f60 0x7f2a800b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7de00640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a64005150 0x7f2a800b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 -- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a64005150 msgr2=0x7f2a800b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a64005150 0x7f2a800b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 -- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a800c4f60 msgr2=0x7f2a800b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a800c4f60 0x7f2a800b16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.399+0000 7f2a7e800640 1 -- 172.21.15.129:0/391178637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a801f5a80 con 0x7f2a801f5f80 2024-09-17T04:05:31.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a7e800640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f2a7802b490 tx=0x7f2a78004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:31.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7802b9e0 con 0x7f2a801f5f80 2024-09-17T04:05:31.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a7802bc90 con 0x7f2a801f5f80 2024-09-17T04:05:31.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a78006300 con 0x7f2a801f5f80 2024-09-17T04:05:31.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a801f9070 con 0x7f2a801f5f80 2024-09-17T04:05:31.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.400+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a801f97f0 con 0x7f2a801f5f80 2024-09-17T04:05:31.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.401+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a800b4140 con 0x7f2a801f5f80 2024-09-17T04:05:31.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.402+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a780064a0 con 0x7f2a801f5f80 2024-09-17T04:05:31.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.403+0000 7f2a76a00640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 0x7f2a5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:31.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.403+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f2a780baee0 con 0x7f2a801f5f80 2024-09-17T04:05:31.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.404+0000 7f2a7de00640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 0x7f2a5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:31.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.404+0000 7f2a7de00640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 0x7f2a5c07b0a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009cd0 tx=0x7f2a70005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:31.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.407+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a78082c30 con 0x7f2a801f5f80 2024-09-17T04:05:31.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.588+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a8006a140 con 0x7f2a5c078b90 2024-09-17T04:05:31.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.593+0000 7f2a76a00640 1 -- 172.21.15.129:0/391178637 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7f2a7005ea50 con 0x7f2a5c078b90 2024-09-17T04:05:31.596 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:31.597 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 msgr2=0x7f2a5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 0x7f2a5c07b0a0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009cd0 tx=0x7f2a70005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 msgr2=0x7f2a801f6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f2a7802b490 tx=0x7f2a78004bb0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 shutdown_connections 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a5c078b90 0x7f2a5c07b0a0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a801f5f80 0x7f2a801f6390 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a800c4f60 0x7f2a800b16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 --2- 172.21.15.129:0/391178637 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a64005150 0x7f2a800b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:31.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 >> 172.21.15.129:0/391178637 conn(0x7f2a8006d410 msgr2=0x7f2a80072ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:31.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.597+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 shutdown_connections 2024-09-17T04:05:31.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:31.598+0000 7f2a85000640 1 -- 172.21.15.129:0/391178637 wait complete. 2024-09-17T04:05:31.624 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-09-17T04:05:30.569710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402736,"kb_used_data":22728,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754571984,"statfs":{"total":773094113280,"available":772681711616,"internally_reserved":0,"allocated":23273472,"data_stored":10907532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.362373"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500684+0000","last_change":"2024-09-17T04:05:25.221229+0000","last_active":"2024-09-17T04:05:26.500684+0000","last_peered":"2024-09-17T04:05:26.500684+0000","last_clean":"2024-09-17T04:05:26.500684+0000","last_became_active":"2024-09-17T04:05:25.220873+0000","last_became_peered":"2024-09-17T04:05:25.220873+0000","last_unstale":"2024-09-17T04:05:26.500684+0000","last_undegraded":"2024-09-17T04:05:26.500684+0000","last_fullsized":"2024-09-17T04:05:26.500684+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:38:14.801493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500862+0000","last_change":"2024-09-17T04:05:25.222366+0000","last_active":"2024-09-17T04:05:26.500862+0000","last_peered":"2024-09-17T04:05:26.500862+0000","last_clean":"2024-09-17T04:05:26.500862+0000","last_became_active":"2024-09-17T04:05:25.220987+0000","last_became_peered":"2024-09-17T04:05:25.220987+0000","last_unstale":"2024-09-17T04:05:26.500862+0000","last_undegraded":"2024-09-17T04:05:26.500862+0000","last_fullsized":"2024-09-17T04:05:26.500862+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:23.139209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228756+0000","last_change":"2024-09-17T04:05:25.235978+0000","last_active":"2024-09-17T04:05:26.228756+0000","last_peered":"2024-09-17T04:05:26.228756+0000","last_clean":"2024-09-17T04:05:26.228756+0000","last_became_active":"2024-09-17T04:05:25.234686+0000","last_became_peered":"2024-09-17T04:05:25.234686+0000","last_unstale":"2024-09-17T04:05:26.228756+0000","last_undegraded":"2024-09-17T04:05:26.228756+0000","last_fullsized":"2024-09-17T04:05:26.228756+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:06.764889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228350+0000","last_change":"2024-09-17T04:05:25.234958+0000","last_active":"2024-09-17T04:05:26.228350+0000","last_peered":"2024-09-17T04:05:26.228350+0000","last_clean":"2024-09-17T04:05:26.228350+0000","last_became_active":"2024-09-17T04:05:25.233705+0000","last_became_peered":"2024-09-17T04:05:25.233705+0000","last_unstale":"2024-09-17T04:05:26.228350+0000","last_undegraded":"2024-09-17T04:05:26.228350+0000","last_fullsized":"2024-09-17T04:05:26.228350+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:29.142246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228296+0000","last_change":"2024-09-17T04:05:25.234011+0000","last_active":"2024-09-17T04:05:26.228296+0000","last_peered":"2024-09-17T04:05:26.228296+0000","last_clean":"2024-09-17T04:05:26.228296+0000","last_became_active":"2024-09-17T04:05:25.233676+0000","last_became_peered":"2024-09-17T04:05:25.233676+0000","last_unstale":"2024-09-17T04:05:26.228296+0000","last_undegraded":"2024-09-17T04:05:26.228296+0000","last_fullsized":"2024-09-17T04:05:26.228296+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436570+0000","last_change":"2024-09-17T04:05:25.232509+0000","last_active":"2024-09-17T04:05:26.436570+0000","last_peered":"2024-09-17T04:05:26.436570+0000","last_clean":"2024-09-17T04:05:26.436570+0000","last_became_active":"2024-09-17T04:05:25.232170+0000","last_became_peered":"2024-09-17T04:05:25.232170+0000","last_unstale":"2024-09-17T04:05:26.436570+0000","last_undegraded":"2024-09-17T04:05:26.436570+0000","last_fullsized":"2024-09-17T04:05:26.436570+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:40.882023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500029+0000","last_change":"2024-09-17T04:05:25.217581+0000","last_active":"2024-09-17T04:05:26.500029+0000","last_peered":"2024-09-17T04:05:26.500029+0000","last_clean":"2024-09-17T04:05:26.500029+0000","last_became_active":"2024-09-17T04:05:25.217074+0000","last_became_peered":"2024-09-17T04:05:25.217074+0000","last_unstale":"2024-09-17T04:05:26.500029+0000","last_undegraded":"2024-09-17T04:05:26.500029+0000","last_fullsized":"2024-09-17T04:05:26.500029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:30.412641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499223+0000","last_change":"2024-09-17T04:05:25.226793+0000","last_active":"2024-09-17T04:05:26.499223+0000","last_peered":"2024-09-17T04:05:26.499223+0000","last_clean":"2024-09-17T04:05:26.499223+0000","last_became_active":"2024-09-17T04:05:25.226356+0000","last_became_peered":"2024-09-17T04:05:25.226356+0000","last_unstale":"2024-09-17T04:05:26.499223+0000","last_undegraded":"2024-09-17T04:05:26.499223+0000","last_fullsized":"2024-09-17T04:05:26.499223+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:40.221203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425646+0000","last_change":"2024-09-17T04:05:25.224713+0000","last_active":"2024-09-17T04:05:26.425646+0000","last_peered":"2024-09-17T04:05:26.425646+0000","last_clean":"2024-09-17T04:05:26.425646+0000","last_became_active":"2024-09-17T04:05:25.224391+0000","last_became_peered":"2024-09-17T04:05:25.224391+0000","last_unstale":"2024-09-17T04:05:26.425646+0000","last_undegraded":"2024-09-17T04:05:26.425646+0000","last_fullsized":"2024-09-17T04:05:26.425646+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:16.385982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500080+0000","last_change":"2024-09-17T04:05:25.226863+0000","last_active":"2024-09-17T04:05:26.500080+0000","last_peered":"2024-09-17T04:05:26.500080+0000","last_clean":"2024-09-17T04:05:26.500080+0000","last_became_active":"2024-09-17T04:05:25.226508+0000","last_became_peered":"2024-09-17T04:05:25.226508+0000","last_unstale":"2024-09-17T04:05:26.500080+0000","last_undegraded":"2024-09-17T04:05:26.500080+0000","last_fullsized":"2024-09-17T04:05:26.500080+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:56.644223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500120+0000","last_change":"2024-09-17T04:05:25.228022+0000","last_active":"2024-09-17T04:05:26.500120+0000","last_peered":"2024-09-17T04:05:26.500120+0000","last_clean":"2024-09-17T04:05:26.500120+0000","last_became_active":"2024-09-17T04:05:25.227659+0000","last_became_peered":"2024-09-17T04:05:25.227659+0000","last_unstale":"2024-09-17T04:05:26.500120+0000","last_undegraded":"2024-09-17T04:05:26.500120+0000","last_fullsized":"2024-09-17T04:05:26.500120+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:51.022680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500106+0000","last_change":"2024-09-17T04:05:25.230634+0000","last_active":"2024-09-17T04:05:26.500106+0000","last_peered":"2024-09-17T04:05:26.500106+0000","last_clean":"2024-09-17T04:05:26.500106+0000","last_became_active":"2024-09-17T04:05:25.229125+0000","last_became_peered":"2024-09-17T04:05:25.229125+0000","last_unstale":"2024-09-17T04:05:26.500106+0000","last_undegraded":"2024-09-17T04:05:26.500106+0000","last_fullsized":"2024-09-17T04:05:26.500106+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:13.538395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228383+0000","last_change":"2024-09-17T04:05:25.238984+0000","last_active":"2024-09-17T04:05:26.228383+0000","last_peered":"2024-09-17T04:05:26.228383+0000","last_clean":"2024-09-17T04:05:26.228383+0000","last_became_active":"2024-09-17T04:05:25.233999+0000","last_became_peered":"2024-09-17T04:05:25.233999+0000","last_unstale":"2024-09-17T04:05:26.228383+0000","last_undegraded":"2024-09-17T04:05:26.228383+0000","last_fullsized":"2024-09-17T04:05:26.228383+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228099+0000","last_change":"2024-09-17T04:05:25.232964+0000","last_active":"2024-09-17T04:05:26.228099+0000","last_peered":"2024-09-17T04:05:26.228099+0000","last_clean":"2024-09-17T04:05:26.228099+0000","last_became_active":"2024-09-17T04:05:25.232684+0000","last_became_peered":"2024-09-17T04:05:25.232684+0000","last_unstale":"2024-09-17T04:05:26.228099+0000","last_undegraded":"2024-09-17T04:05:26.228099+0000","last_fullsized":"2024-09-17T04:05:26.228099+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:37.598051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228177+0000","last_change":"2024-09-17T04:05:25.226026+0000","last_active":"2024-09-17T04:05:26.228177+0000","last_peered":"2024-09-17T04:05:26.228177+0000","last_clean":"2024-09-17T04:05:26.228177+0000","last_became_active":"2024-09-17T04:05:25.225661+0000","last_became_peered":"2024-09-17T04:05:25.225661+0000","last_unstale":"2024-09-17T04:05:26.228177+0000","last_undegraded":"2024-09-17T04:05:26.228177+0000","last_fullsized":"2024-09-17T04:05:26.228177+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.988518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436505+0000","last_change":"2024-09-17T04:05:25.230601+0000","last_active":"2024-09-17T04:05:26.436505+0000","last_peered":"2024-09-17T04:05:26.436505+0000","last_clean":"2024-09-17T04:05:26.436505+0000","last_became_active":"2024-09-17T04:05:25.230028+0000","last_became_peered":"2024-09-17T04:05:25.230028+0000","last_unstale":"2024-09-17T04:05:26.436505+0000","last_undegraded":"2024-09-17T04:05:26.436505+0000","last_fullsized":"2024-09-17T04:05:26.436505+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:52:57.752381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225608+0000","last_change":"2024-09-17T04:05:25.227454+0000","last_active":"2024-09-17T04:05:26.225608+0000","last_peered":"2024-09-17T04:05:26.225608+0000","last_clean":"2024-09-17T04:05:26.225608+0000","last_became_active":"2024-09-17T04:05:25.226074+0000","last_became_peered":"2024-09-17T04:05:25.226074+0000","last_unstale":"2024-09-17T04:05:26.225608+0000","last_undegraded":"2024-09-17T04:05:26.225608+0000","last_fullsized":"2024-09-17T04:05:26.225608+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:20.806967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500764+0000","last_change":"2024-09-17T04:05:25.217557+0000","last_active":"2024-09-17T04:05:26.500764+0000","last_peered":"2024-09-17T04:05:26.500764+0000","last_clean":"2024-09-17T04:05:26.500764+0000","last_became_active":"2024-09-17T04:05:25.217163+0000","last_became_peered":"2024-09-17T04:05:25.217163+0000","last_unstale":"2024-09-17T04:05:26.500764+0000","last_undegraded":"2024-09-17T04:05:26.500764+0000","last_fullsized":"2024-09-17T04:05:26.500764+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:28.881707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500523+0000","last_change":"2024-09-17T04:05:25.219655+0000","last_active":"2024-09-17T04:05:26.500523+0000","last_peered":"2024-09-17T04:05:26.500523+0000","last_clean":"2024-09-17T04:05:26.500523+0000","last_became_active":"2024-09-17T04:05:25.219345+0000","last_became_peered":"2024-09-17T04:05:25.219345+0000","last_unstale":"2024-09-17T04:05:26.500523+0000","last_undegraded":"2024-09-17T04:05:26.500523+0000","last_fullsized":"2024-09-17T04:05:26.500523+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:59.868383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436713+0000","last_change":"2024-09-17T04:05:25.227398+0000","last_active":"2024-09-17T04:05:26.436713+0000","last_peered":"2024-09-17T04:05:26.436713+0000","last_clean":"2024-09-17T04:05:26.436713+0000","last_became_active":"2024-09-17T04:05:25.227065+0000","last_became_peered":"2024-09-17T04:05:25.227065+0000","last_unstale":"2024-09-17T04:05:26.436713+0000","last_undegraded":"2024-09-17T04:05:26.436713+0000","last_fullsized":"2024-09-17T04:05:26.436713+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:41.966823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425500+0000","last_change":"2024-09-17T04:05:25.231383+0000","last_active":"2024-09-17T04:05:26.425500+0000","last_peered":"2024-09-17T04:05:26.425500+0000","last_clean":"2024-09-17T04:05:26.425500+0000","last_became_active":"2024-09-17T04:05:25.231058+0000","last_became_peered":"2024-09-17T04:05:25.231058+0000","last_unstale":"2024-09-17T04:05:26.425500+0000","last_undegraded":"2024-09-17T04:05:26.425500+0000","last_fullsized":"2024-09-17T04:05:26.425500+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.757807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425336+0000","last_change":"2024-09-17T04:05:25.223284+0000","last_active":"2024-09-17T04:05:26.425336+0000","last_peered":"2024-09-17T04:05:26.425336+0000","last_clean":"2024-09-17T04:05:26.425336+0000","last_became_active":"2024-09-17T04:05:25.221020+0000","last_became_peered":"2024-09-17T04:05:25.221020+0000","last_unstale":"2024-09-17T04:05:26.425336+0000","last_undegraded":"2024-09-17T04:05:26.425336+0000","last_fullsized":"2024-09-17T04:05:26.425336+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:09.253541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500358+0000","last_change":"2024-09-17T04:05:25.226879+0000","last_active":"2024-09-17T04:05:26.500358+0000","last_peered":"2024-09-17T04:05:26.500358+0000","last_clean":"2024-09-17T04:05:26.500358+0000","last_became_active":"2024-09-17T04:05:25.226571+0000","last_became_peered":"2024-09-17T04:05:25.226571+0000","last_unstale":"2024-09-17T04:05:26.500358+0000","last_undegraded":"2024-09-17T04:05:26.500358+0000","last_fullsized":"2024-09-17T04:05:26.500358+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:45.293995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225827+0000","last_change":"2024-09-17T04:05:25.229114+0000","last_active":"2024-09-17T04:05:26.225827+0000","last_peered":"2024-09-17T04:05:26.225827+0000","last_clean":"2024-09-17T04:05:26.225827+0000","last_became_active":"2024-09-17T04:05:25.228819+0000","last_became_peered":"2024-09-17T04:05:25.228819+0000","last_unstale":"2024-09-17T04:05:26.225827+0000","last_undegraded":"2024-09-17T04:05:26.225827+0000","last_fullsized":"2024-09-17T04:05:26.225827+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:48.170272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228576+0000","last_change":"2024-09-17T04:05:25.244112+0000","last_active":"2024-09-17T04:05:26.228576+0000","last_peered":"2024-09-17T04:05:26.228576+0000","last_clean":"2024-09-17T04:05:26.228576+0000","last_became_active":"2024-09-17T04:05:25.243746+0000","last_became_peered":"2024-09-17T04:05:25.243746+0000","last_unstale":"2024-09-17T04:05:26.228576+0000","last_undegraded":"2024-09-17T04:05:26.228576+0000","last_fullsized":"2024-09-17T04:05:26.228576+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228461+0000","last_change":"2024-09-17T04:05:25.236197+0000","last_active":"2024-09-17T04:05:26.228461+0000","last_peered":"2024-09-17T04:05:26.228461+0000","last_clean":"2024-09-17T04:05:26.228461+0000","last_became_active":"2024-09-17T04:05:25.233570+0000","last_became_peered":"2024-09-17T04:05:25.233570+0000","last_unstale":"2024-09-17T04:05:26.228461+0000","last_undegraded":"2024-09-17T04:05:26.228461+0000","last_fullsized":"2024-09-17T04:05:26.228461+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:52.842381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500441+0000","last_change":"2024-09-17T04:05:25.230257+0000","last_active":"2024-09-17T04:05:26.500441+0000","last_peered":"2024-09-17T04:05:26.500441+0000","last_clean":"2024-09-17T04:05:26.500441+0000","last_became_active":"2024-09-17T04:05:25.229043+0000","last_became_peered":"2024-09-17T04:05:25.229043+0000","last_unstale":"2024-09-17T04:05:26.500441+0000","last_undegraded":"2024-09-17T04:05:26.500441+0000","last_fullsized":"2024-09-17T04:05:26.500441+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:03.038463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228701+0000","last_change":"2024-09-17T04:05:25.242725+0000","last_active":"2024-09-17T04:05:26.228701+0000","last_peered":"2024-09-17T04:05:26.228701+0000","last_clean":"2024-09-17T04:05:26.228701+0000","last_became_active":"2024-09-17T04:05:25.239325+0000","last_became_peered":"2024-09-17T04:05:25.239325+0000","last_unstale":"2024-09-17T04:05:26.228701+0000","last_undegraded":"2024-09-17T04:05:26.228701+0000","last_fullsized":"2024-09-17T04:05:26.228701+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:29.392591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226066+0000","last_change":"2024-09-17T04:05:25.226877+0000","last_active":"2024-09-17T04:05:26.226066+0000","last_peered":"2024-09-17T04:05:26.226066+0000","last_clean":"2024-09-17T04:05:26.226066+0000","last_became_active":"2024-09-17T04:05:25.226520+0000","last_became_peered":"2024-09-17T04:05:25.226520+0000","last_unstale":"2024-09-17T04:05:26.226066+0000","last_undegraded":"2024-09-17T04:05:26.226066+0000","last_fullsized":"2024-09-17T04:05:26.226066+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:15.166447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228380+0000","last_change":"2024-09-17T04:05:25.232439+0000","last_active":"2024-09-17T04:05:26.228380+0000","last_peered":"2024-09-17T04:05:26.228380+0000","last_clean":"2024-09-17T04:05:26.228380+0000","last_became_active":"2024-09-17T04:05:25.230995+0000","last_became_peered":"2024-09-17T04:05:25.230995+0000","last_unstale":"2024-09-17T04:05:26.228380+0000","last_undegraded":"2024-09-17T04:05:26.228380+0000","last_fullsized":"2024-09-17T04:05:26.228380+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:49.397560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499086+0000","last_change":"2024-09-17T04:05:25.225290+0000","last_active":"2024-09-17T04:05:26.499086+0000","last_peered":"2024-09-17T04:05:26.499086+0000","last_clean":"2024-09-17T04:05:26.499086+0000","last_became_active":"2024-09-17T04:05:25.224935+0000","last_became_peered":"2024-09-17T04:05:25.224935+0000","last_unstale":"2024-09-17T04:05:26.499086+0000","last_undegraded":"2024-09-17T04:05:26.499086+0000","last_fullsized":"2024-09-17T04:05:26.499086+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:00.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500413+0000","last_change":"2024-09-17T04:05:25.231448+0000","last_active":"2024-09-17T04:05:26.500413+0000","last_peered":"2024-09-17T04:05:26.500413+0000","last_clean":"2024-09-17T04:05:26.500413+0000","last_became_active":"2024-09-17T04:05:25.229864+0000","last_became_peered":"2024-09-17T04:05:25.229864+0000","last_unstale":"2024-09-17T04:05:26.500413+0000","last_undegraded":"2024-09-17T04:05:26.500413+0000","last_fullsized":"2024-09-17T04:05:26.500413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:31.295888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"217'4","reported_seq":19,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226006+0000","last_change":"2024-09-17T04:05:24.220787+0000","last_active":"2024-09-17T04:05:26.226006+0000","last_peered":"2024-09-17T04:05:26.226006+0000","last_clean":"2024-09-17T04:05:26.226006+0000","last_became_active":"2024-09-17T04:05:24.220468+0000","last_became_peered":"2024-09-17T04:05:24.220468+0000","last_unstale":"2024-09-17T04:05:26.226006+0000","last_undegraded":"2024-09-17T04:05:26.226006+0000","last_fullsized":"2024-09-17T04:05:26.226006+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:23.868440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228637+0000","last_change":"2024-09-17T04:05:25.235813+0000","last_active":"2024-09-17T04:05:26.228637+0000","last_peered":"2024-09-17T04:05:26.228637+0000","last_clean":"2024-09-17T04:05:26.228637+0000","last_became_active":"2024-09-17T04:05:25.235483+0000","last_became_peered":"2024-09-17T04:05:25.235483+0000","last_unstale":"2024-09-17T04:05:26.228637+0000","last_undegraded":"2024-09-17T04:05:26.228637+0000","last_fullsized":"2024-09-17T04:05:26.228637+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:06.428970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228581+0000","last_change":"2024-09-17T04:05:24.231055+0000","last_active":"2024-09-17T04:05:26.228581+0000","last_peered":"2024-09-17T04:05:26.228581+0000","last_clean":"2024-09-17T04:05:26.228581+0000","last_became_active":"2024-09-17T04:05:24.222063+0000","last_became_peered":"2024-09-17T04:05:24.222063+0000","last_unstale":"2024-09-17T04:05:26.228581+0000","last_undegraded":"2024-09-17T04:05:26.228581+0000","last_fullsized":"2024-09-17T04:05:26.228581+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:35.224862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500629+0000","last_change":"2024-09-17T04:05:25.227705+0000","last_active":"2024-09-17T04:05:26.500629+0000","last_peered":"2024-09-17T04:05:26.500629+0000","last_clean":"2024-09-17T04:05:26.500629+0000","last_became_active":"2024-09-17T04:05:25.223453+0000","last_became_peered":"2024-09-17T04:05:25.223453+0000","last_unstale":"2024-09-17T04:05:26.500629+0000","last_undegraded":"2024-09-17T04:05:26.500629+0000","last_fullsized":"2024-09-17T04:05:26.500629+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:26.267555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437169+0000","last_change":"2024-09-17T04:05:24.228243+0000","last_active":"2024-09-17T04:05:26.437169+0000","last_peered":"2024-09-17T04:05:26.437169+0000","last_clean":"2024-09-17T04:05:26.437169+0000","last_became_active":"2024-09-17T04:05:24.227852+0000","last_became_peered":"2024-09-17T04:05:24.227852+0000","last_unstale":"2024-09-17T04:05:26.437169+0000","last_undegraded":"2024-09-17T04:05:26.437169+0000","last_fullsized":"2024-09-17T04:05:26.437169+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:02.226834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500033+0000","last_change":"2024-09-17T04:05:25.225950+0000","last_active":"2024-09-17T04:05:26.500033+0000","last_peered":"2024-09-17T04:05:26.500033+0000","last_clean":"2024-09-17T04:05:26.500033+0000","last_became_active":"2024-09-17T04:05:25.225457+0000","last_became_peered":"2024-09-17T04:05:25.225457+0000","last_unstale":"2024-09-17T04:05:26.500033+0000","last_undegraded":"2024-09-17T04:05:26.500033+0000","last_fullsized":"2024-09-17T04:05:26.500033+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:41.166197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437014+0000","last_change":"2024-09-17T04:05:24.226740+0000","last_active":"2024-09-17T04:05:26.437014+0000","last_peered":"2024-09-17T04:05:26.437014+0000","last_clean":"2024-09-17T04:05:26.437014+0000","last_became_active":"2024-09-17T04:05:24.226406+0000","last_became_peered":"2024-09-17T04:05:24.226406+0000","last_unstale":"2024-09-17T04:05:26.437014+0000","last_undegraded":"2024-09-17T04:05:26.437014+0000","last_fullsized":"2024-09-17T04:05:26.437014+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:51.681172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425319+0000","last_change":"2024-09-17T04:05:25.232667+0000","last_active":"2024-09-17T04:05:26.425319+0000","last_peered":"2024-09-17T04:05:26.425319+0000","last_clean":"2024-09-17T04:05:26.425319+0000","last_became_active":"2024-09-17T04:05:25.232379+0000","last_became_peered":"2024-09-17T04:05:25.232379+0000","last_unstale":"2024-09-17T04:05:26.425319+0000","last_undegraded":"2024-09-17T04:05:26.425319+0000","last_fullsized":"2024-09-17T04:05:26.425319+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:20.607889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499657+0000","last_change":"2024-09-17T04:05:24.224011+0000","last_active":"2024-09-17T04:05:26.499657+0000","last_peered":"2024-09-17T04:05:26.499657+0000","last_clean":"2024-09-17T04:05:26.499657+0000","last_became_active":"2024-09-17T04:05:24.223670+0000","last_became_peered":"2024-09-17T04:05:24.223670+0000","last_unstale":"2024-09-17T04:05:26.499657+0000","last_undegraded":"2024-09-17T04:05:26.499657+0000","last_fullsized":"2024-09-17T04:05:26.499657+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.853028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436826+0000","last_change":"2024-09-17T04:05:25.220380+0000","last_active":"2024-09-17T04:05:26.436826+0000","last_peered":"2024-09-17T04:05:26.436826+0000","last_clean":"2024-09-17T04:05:26.436826+0000","last_became_active":"2024-09-17T04:05:25.217028+0000","last_became_peered":"2024-09-17T04:05:25.217028+0000","last_unstale":"2024-09-17T04:05:26.436826+0000","last_undegraded":"2024-09-17T04:05:26.436826+0000","last_fullsized":"2024-09-17T04:05:26.436826+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:44.696425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225677+0000","last_change":"2024-09-17T04:05:24.221715+0000","last_active":"2024-09-17T04:05:26.225677+0000","last_peered":"2024-09-17T04:05:26.225677+0000","last_clean":"2024-09-17T04:05:26.225677+0000","last_became_active":"2024-09-17T04:05:24.221380+0000","last_became_peered":"2024-09-17T04:05:24.221380+0000","last_unstale":"2024-09-17T04:05:26.225677+0000","last_undegraded":"2024-09-17T04:05:26.225677+0000","last_fullsized":"2024-09-17T04:05:26.225677+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:33.998165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436516+0000","last_change":"2024-09-17T04:05:25.213295+0000","last_active":"2024-09-17T04:05:26.436516+0000","last_peered":"2024-09-17T04:05:26.436516+0000","last_clean":"2024-09-17T04:05:26.436516+0000","last_became_active":"2024-09-17T04:05:25.212970+0000","last_became_peered":"2024-09-17T04:05:25.212970+0000","last_unstale":"2024-09-17T04:05:26.436516+0000","last_undegraded":"2024-09-17T04:05:26.436516+0000","last_fullsized":"2024-09-17T04:05:26.436516+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:01.839472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228120+0000","last_change":"2024-09-17T04:05:24.234352+0000","last_active":"2024-09-17T04:05:26.228120+0000","last_peered":"2024-09-17T04:05:26.228120+0000","last_clean":"2024-09-17T04:05:26.228120+0000","last_became_active":"2024-09-17T04:05:24.234071+0000","last_became_peered":"2024-09-17T04:05:24.234071+0000","last_unstale":"2024-09-17T04:05:26.228120+0000","last_undegraded":"2024-09-17T04:05:26.228120+0000","last_fullsized":"2024-09-17T04:05:26.228120+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:37.269558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500626+0000","last_change":"2024-09-17T04:05:25.219129+0000","last_active":"2024-09-17T04:05:26.500626+0000","last_peered":"2024-09-17T04:05:26.500626+0000","last_clean":"2024-09-17T04:05:26.500626+0000","last_became_active":"2024-09-17T04:05:25.217644+0000","last_became_peered":"2024-09-17T04:05:25.217644+0000","last_unstale":"2024-09-17T04:05:26.500626+0000","last_undegraded":"2024-09-17T04:05:26.500626+0000","last_fullsized":"2024-09-17T04:05:26.500626+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:52.028296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499665+0000","last_change":"2024-09-17T04:05:24.224568+0000","last_active":"2024-09-17T04:05:26.499665+0000","last_peered":"2024-09-17T04:05:26.499665+0000","last_clean":"2024-09-17T04:05:26.499665+0000","last_became_active":"2024-09-17T04:05:24.223638+0000","last_became_peered":"2024-09-17T04:05:24.223638+0000","last_unstale":"2024-09-17T04:05:26.499665+0000","last_undegraded":"2024-09-17T04:05:26.499665+0000","last_fullsized":"2024-09-17T04:05:26.499665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:49.701292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500661+0000","last_change":"2024-09-17T04:05:25.218519+0000","last_active":"2024-09-17T04:05:26.500661+0000","last_peered":"2024-09-17T04:05:26.500661+0000","last_clean":"2024-09-17T04:05:26.500661+0000","last_became_active":"2024-09-17T04:05:25.217254+0000","last_became_peered":"2024-09-17T04:05:25.217254+0000","last_unstale":"2024-09-17T04:05:26.500661+0000","last_undegraded":"2024-09-17T04:05:26.500661+0000","last_fullsized":"2024-09-17T04:05:26.500661+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:49.680789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226069+0000","last_change":"2024-09-17T04:05:24.223519+0000","last_active":"2024-09-17T04:05:26.226069+0000","last_peered":"2024-09-17T04:05:26.226069+0000","last_clean":"2024-09-17T04:05:26.226069+0000","last_became_active":"2024-09-17T04:05:24.223231+0000","last_became_peered":"2024-09-17T04:05:24.223231+0000","last_unstale":"2024-09-17T04:05:26.226069+0000","last_undegraded":"2024-09-17T04:05:26.226069+0000","last_fullsized":"2024-09-17T04:05:26.226069+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:42.041674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225715+0000","last_change":"2024-09-17T04:05:25.221512+0000","last_active":"2024-09-17T04:05:26.225715+0000","last_peered":"2024-09-17T04:05:26.225715+0000","last_clean":"2024-09-17T04:05:26.225715+0000","last_became_active":"2024-09-17T04:05:25.220458+0000","last_became_peered":"2024-09-17T04:05:25.220458+0000","last_unstale":"2024-09-17T04:05:26.225715+0000","last_undegraded":"2024-09-17T04:05:26.225715+0000","last_fullsized":"2024-09-17T04:05:26.225715+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:55.416231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436765+0000","last_change":"2024-09-17T04:05:24.226942+0000","last_active":"2024-09-17T04:05:26.436765+0000","last_peered":"2024-09-17T04:05:26.436765+0000","last_clean":"2024-09-17T04:05:26.436765+0000","last_became_active":"2024-09-17T04:05:24.226597+0000","last_became_peered":"2024-09-17T04:05:24.226597+0000","last_unstale":"2024-09-17T04:05:26.436765+0000","last_undegraded":"2024-09-17T04:05:26.436765+0000","last_fullsized":"2024-09-17T04:05:26.436765+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:12.017426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228414+0000","last_change":"2024-09-17T04:05:25.227178+0000","last_active":"2024-09-17T04:05:26.228414+0000","last_peered":"2024-09-17T04:05:26.228414+0000","last_clean":"2024-09-17T04:05:26.228414+0000","last_became_active":"2024-09-17T04:05:25.226791+0000","last_became_peered":"2024-09-17T04:05:25.226791+0000","last_unstale":"2024-09-17T04:05:26.228414+0000","last_undegraded":"2024-09-17T04:05:26.228414+0000","last_fullsized":"2024-09-17T04:05:26.228414+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:24.402863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"217'3","reported_seq":18,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437223+0000","last_change":"2024-09-17T04:05:24.230056+0000","last_active":"2024-09-17T04:05:26.437223+0000","last_peered":"2024-09-17T04:05:26.437223+0000","last_clean":"2024-09-17T04:05:26.437223+0000","last_became_active":"2024-09-17T04:05:24.229748+0000","last_became_peered":"2024-09-17T04:05:24.229748+0000","last_unstale":"2024-09-17T04:05:26.437223+0000","last_undegraded":"2024-09-17T04:05:26.437223+0000","last_fullsized":"2024-09-17T04:05:26.437223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:19.117891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225488+0000","last_change":"2024-09-17T04:05:25.220710+0000","last_active":"2024-09-17T04:05:26.225488+0000","last_peered":"2024-09-17T04:05:26.225488+0000","last_clean":"2024-09-17T04:05:26.225488+0000","last_became_active":"2024-09-17T04:05:25.220415+0000","last_became_peered":"2024-09-17T04:05:25.220415+0000","last_unstale":"2024-09-17T04:05:26.225488+0000","last_undegraded":"2024-09-17T04:05:26.225488+0000","last_fullsized":"2024-09-17T04:05:26.225488+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:46.853834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"217'6","reported_seq":21,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225389+0000","last_change":"2024-09-17T04:05:24.225084+0000","last_active":"2024-09-17T04:05:26.225389+0000","last_peered":"2024-09-17T04:05:26.225389+0000","last_clean":"2024-09-17T04:05:26.225389+0000","last_became_active":"2024-09-17T04:05:24.224781+0000","last_became_peered":"2024-09-17T04:05:24.224781+0000","last_unstale":"2024-09-17T04:05:26.225389+0000","last_undegraded":"2024-09-17T04:05:26.225389+0000","last_fullsized":"2024-09-17T04:05:26.225389+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:21.718712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225506+0000","last_change":"2024-09-17T04:05:25.219009+0000","last_active":"2024-09-17T04:05:26.225506+0000","last_peered":"2024-09-17T04:05:26.225506+0000","last_clean":"2024-09-17T04:05:26.225506+0000","last_became_active":"2024-09-17T04:05:25.218697+0000","last_became_peered":"2024-09-17T04:05:25.218697+0000","last_unstale":"2024-09-17T04:05:26.225506+0000","last_undegraded":"2024-09-17T04:05:26.225506+0000","last_fullsized":"2024-09-17T04:05:26.225506+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:46.772065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500062+0000","last_change":"2024-09-17T04:05:24.223126+0000","last_active":"2024-09-17T04:05:26.500062+0000","last_peered":"2024-09-17T04:05:26.500062+0000","last_clean":"2024-09-17T04:05:26.500062+0000","last_became_active":"2024-09-17T04:05:24.221114+0000","last_became_peered":"2024-09-17T04:05:24.221114+0000","last_unstale":"2024-09-17T04:05:26.500062+0000","last_undegraded":"2024-09-17T04:05:26.500062+0000","last_fullsized":"2024-09-17T04:05:26.500062+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:13.552456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436876+0000","last_change":"2024-09-17T04:05:25.213282+0000","last_active":"2024-09-17T04:05:26.436876+0000","last_peered":"2024-09-17T04:05:26.436876+0000","last_clean":"2024-09-17T04:05:26.436876+0000","last_became_active":"2024-09-17T04:05:25.212938+0000","last_became_peered":"2024-09-17T04:05:25.212938+0000","last_unstale":"2024-09-17T04:05:26.436876+0000","last_undegraded":"2024-09-17T04:05:26.436876+0000","last_fullsized":"2024-09-17T04:05:26.436876+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:26.820397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425400+0000","last_change":"2024-09-17T04:05:24.223560+0000","last_active":"2024-09-17T04:05:26.425400+0000","last_peered":"2024-09-17T04:05:26.425400+0000","last_clean":"2024-09-17T04:05:26.425400+0000","last_became_active":"2024-09-17T04:05:24.223302+0000","last_became_peered":"2024-09-17T04:05:24.223302+0000","last_unstale":"2024-09-17T04:05:26.425400+0000","last_undegraded":"2024-09-17T04:05:26.425400+0000","last_fullsized":"2024-09-17T04:05:26.425400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:25.259928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225342+0000","last_change":"2024-09-17T04:05:25.231922+0000","last_active":"2024-09-17T04:05:26.225342+0000","last_peered":"2024-09-17T04:05:26.225342+0000","last_clean":"2024-09-17T04:05:26.225342+0000","last_became_active":"2024-09-17T04:05:25.231099+0000","last_became_peered":"2024-09-17T04:05:25.231099+0000","last_unstale":"2024-09-17T04:05:26.225342+0000","last_undegraded":"2024-09-17T04:05:26.225342+0000","last_fullsized":"2024-09-17T04:05:26.225342+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:29:35.958163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499298+0000","last_change":"2024-09-17T04:05:24.231560+0000","last_active":"2024-09-17T04:05:26.499298+0000","last_peered":"2024-09-17T04:05:26.499298+0000","last_clean":"2024-09-17T04:05:26.499298+0000","last_became_active":"2024-09-17T04:05:24.231172+0000","last_became_peered":"2024-09-17T04:05:24.231172+0000","last_unstale":"2024-09-17T04:05:26.499298+0000","last_undegraded":"2024-09-17T04:05:26.499298+0000","last_fullsized":"2024-09-17T04:05:26.499298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:17.113825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225269+0000","last_change":"2024-09-17T04:05:25.228578+0000","last_active":"2024-09-17T04:05:26.225269+0000","last_peered":"2024-09-17T04:05:26.225269+0000","last_clean":"2024-09-17T04:05:26.225269+0000","last_became_active":"2024-09-17T04:05:25.228179+0000","last_became_peered":"2024-09-17T04:05:25.228179+0000","last_unstale":"2024-09-17T04:05:26.225269+0000","last_undegraded":"2024-09-17T04:05:26.225269+0000","last_fullsized":"2024-09-17T04:05:26.225269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:08.402904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"217'8","reported_seq":23,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436709+0000","last_change":"2024-09-17T04:05:24.229283+0000","last_active":"2024-09-17T04:05:26.436709+0000","last_peered":"2024-09-17T04:05:26.436709+0000","last_clean":"2024-09-17T04:05:26.436709+0000","last_became_active":"2024-09-17T04:05:24.228948+0000","last_became_peered":"2024-09-17T04:05:24.228948+0000","last_unstale":"2024-09-17T04:05:26.436709+0000","last_undegraded":"2024-09-17T04:05:26.436709+0000","last_fullsized":"2024-09-17T04:05:26.436709+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:17.532872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228381+0000","last_change":"2024-09-17T04:05:25.229771+0000","last_active":"2024-09-17T04:05:26.228381+0000","last_peered":"2024-09-17T04:05:26.228381+0000","last_clean":"2024-09-17T04:05:26.228381+0000","last_became_active":"2024-09-17T04:05:25.229452+0000","last_became_peered":"2024-09-17T04:05:25.229452+0000","last_unstale":"2024-09-17T04:05:26.228381+0000","last_undegraded":"2024-09-17T04:05:26.228381+0000","last_fullsized":"2024-09-17T04:05:26.228381+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:39.818419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500697+0000","last_change":"2024-09-17T04:05:24.222556+0000","last_active":"2024-09-17T04:05:26.500697+0000","last_peered":"2024-09-17T04:05:26.500697+0000","last_clean":"2024-09-17T04:05:26.500697+0000","last_became_active":"2024-09-17T04:05:24.222221+0000","last_became_peered":"2024-09-17T04:05:24.222221+0000","last_unstale":"2024-09-17T04:05:26.500697+0000","last_undegraded":"2024-09-17T04:05:26.500697+0000","last_fullsized":"2024-09-17T04:05:26.500697+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:20.842936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.501041+0000","last_change":"2024-09-17T04:05:25.244458+0000","last_active":"2024-09-17T04:05:26.501041+0000","last_peered":"2024-09-17T04:05:26.501041+0000","last_clean":"2024-09-17T04:05:26.501041+0000","last_became_active":"2024-09-17T04:05:25.244033+0000","last_became_peered":"2024-09-17T04:05:25.244033+0000","last_unstale":"2024-09-17T04:05:26.501041+0000","last_undegraded":"2024-09-17T04:05:26.501041+0000","last_fullsized":"2024-09-17T04:05:26.501041+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:00.213217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436875+0000","last_change":"2024-09-17T04:05:24.226050+0000","last_active":"2024-09-17T04:05:26.436875+0000","last_peered":"2024-09-17T04:05:26.436875+0000","last_clean":"2024-09-17T04:05:26.436875+0000","last_became_active":"2024-09-17T04:05:24.225367+0000","last_became_peered":"2024-09-17T04:05:24.225367+0000","last_unstale":"2024-09-17T04:05:26.436875+0000","last_undegraded":"2024-09-17T04:05:26.436875+0000","last_fullsized":"2024-09-17T04:05:26.436875+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:24.613188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228770+0000","last_change":"2024-09-17T04:05:25.226348+0000","last_active":"2024-09-17T04:05:26.228770+0000","last_peered":"2024-09-17T04:05:26.228770+0000","last_clean":"2024-09-17T04:05:26.228770+0000","last_became_active":"2024-09-17T04:05:25.225598+0000","last_became_peered":"2024-09-17T04:05:25.225598+0000","last_unstale":"2024-09-17T04:05:26.228770+0000","last_undegraded":"2024-09-17T04:05:26.228770+0000","last_fullsized":"2024-09-17T04:05:26.228770+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:59.151440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500308+0000","last_change":"2024-09-17T04:05:24.226944+0000","last_active":"2024-09-17T04:05:26.500308+0000","last_peered":"2024-09-17T04:05:26.500308+0000","last_clean":"2024-09-17T04:05:26.500308+0000","last_became_active":"2024-09-17T04:05:24.225543+0000","last_became_peered":"2024-09-17T04:05:24.225543+0000","last_unstale":"2024-09-17T04:05:26.500308+0000","last_undegraded":"2024-09-17T04:05:26.500308+0000","last_fullsized":"2024-09-17T04:05:26.500308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.257988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228745+0000","last_change":"2024-09-17T04:05:25.242208+0000","last_active":"2024-09-17T04:05:26.228745+0000","last_peered":"2024-09-17T04:05:26.228745+0000","last_clean":"2024-09-17T04:05:26.228745+0000","last_became_active":"2024-09-17T04:05:25.241886+0000","last_became_peered":"2024-09-17T04:05:25.241886+0000","last_unstale":"2024-09-17T04:05:26.228745+0000","last_undegraded":"2024-09-17T04:05:26.228745+0000","last_fullsized":"2024-09-17T04:05:26.228745+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228869+0000","last_change":"2024-09-17T04:05:24.229174+0000","last_active":"2024-09-17T04:05:26.228869+0000","last_peered":"2024-09-17T04:05:26.228869+0000","last_clean":"2024-09-17T04:05:26.228869+0000","last_became_active":"2024-09-17T04:05:24.228858+0000","last_became_peered":"2024-09-17T04:05:24.228858+0000","last_unstale":"2024-09-17T04:05:26.228869+0000","last_undegraded":"2024-09-17T04:05:26.228869+0000","last_fullsized":"2024-09-17T04:05:26.228869+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.505474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436624+0000","last_change":"2024-09-17T04:05:25.226304+0000","last_active":"2024-09-17T04:05:26.436624+0000","last_peered":"2024-09-17T04:05:26.436624+0000","last_clean":"2024-09-17T04:05:26.436624+0000","last_became_active":"2024-09-17T04:05:25.224061+0000","last_became_peered":"2024-09-17T04:05:25.224061+0000","last_unstale":"2024-09-17T04:05:26.436624+0000","last_undegraded":"2024-09-17T04:05:26.436624+0000","last_fullsized":"2024-09-17T04:05:26.436624+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:55:49.229679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228504+0000","last_change":"2024-09-17T04:05:24.225036+0000","last_active":"2024-09-17T04:05:26.228504+0000","last_peered":"2024-09-17T04:05:26.228504+0000","last_clean":"2024-09-17T04:05:26.228504+0000","last_became_active":"2024-09-17T04:05:24.223209+0000","last_became_peered":"2024-09-17T04:05:24.223209+0000","last_unstale":"2024-09-17T04:05:26.228504+0000","last_undegraded":"2024-09-17T04:05:26.228504+0000","last_fullsized":"2024-09-17T04:05:26.228504+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:21.294118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437062+0000","last_change":"2024-09-17T04:05:25.221299+0000","last_active":"2024-09-17T04:05:26.437062+0000","last_peered":"2024-09-17T04:05:26.437062+0000","last_clean":"2024-09-17T04:05:26.437062+0000","last_became_active":"2024-09-17T04:05:25.220970+0000","last_became_peered":"2024-09-17T04:05:25.220970+0000","last_unstale":"2024-09-17T04:05:26.437062+0000","last_undegraded":"2024-09-17T04:05:26.437062+0000","last_fullsized":"2024-09-17T04:05:26.437062+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:28.900393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499673+0000","last_change":"2024-09-17T04:05:24.224549+0000","last_active":"2024-09-17T04:05:26.499673+0000","last_peered":"2024-09-17T04:05:26.499673+0000","last_clean":"2024-09-17T04:05:26.499673+0000","last_became_active":"2024-09-17T04:05:24.224066+0000","last_became_peered":"2024-09-17T04:05:24.224066+0000","last_unstale":"2024-09-17T04:05:26.499673+0000","last_undegraded":"2024-09-17T04:05:26.499673+0000","last_fullsized":"2024-09-17T04:05:26.499673+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:26.556703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225575+0000","last_change":"2024-09-17T04:05:25.221813+0000","last_active":"2024-09-17T04:05:26.225575+0000","last_peered":"2024-09-17T04:05:26.225575+0000","last_clean":"2024-09-17T04:05:26.225575+0000","last_became_active":"2024-09-17T04:05:25.221550+0000","last_became_peered":"2024-09-17T04:05:25.221550+0000","last_unstale":"2024-09-17T04:05:26.225575+0000","last_undegraded":"2024-09-17T04:05:26.225575+0000","last_fullsized":"2024-09-17T04:05:26.225575+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:42.072358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499425+0000","last_change":"2024-09-17T04:05:24.229963+0000","last_active":"2024-09-17T04:05:26.499425+0000","last_peered":"2024-09-17T04:05:26.499425+0000","last_clean":"2024-09-17T04:05:26.499425+0000","last_became_active":"2024-09-17T04:05:24.229600+0000","last_became_peered":"2024-09-17T04:05:24.229600+0000","last_unstale":"2024-09-17T04:05:26.499425+0000","last_undegraded":"2024-09-17T04:05:26.499425+0000","last_fullsized":"2024-09-17T04:05:26.499425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:45.305405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425310+0000","last_change":"2024-09-17T04:05:25.238027+0000","last_active":"2024-09-17T04:05:26.425310+0000","last_peered":"2024-09-17T04:05:26.425310+0000","last_clean":"2024-09-17T04:05:26.425310+0000","last_became_active":"2024-09-17T04:05:25.236850+0000","last_became_peered":"2024-09-17T04:05:25.236850+0000","last_unstale":"2024-09-17T04:05:26.425310+0000","last_undegraded":"2024-09-17T04:05:26.425310+0000","last_fullsized":"2024-09-17T04:05:26.425310+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:04.588444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499682+0000","last_change":"2024-09-17T04:05:24.224522+0000","last_active":"2024-09-17T04:05:26.499682+0000","last_peered":"2024-09-17T04:05:26.499682+0000","last_clean":"2024-09-17T04:05:26.499682+0000","last_became_active":"2024-09-17T04:05:24.224171+0000","last_became_peered":"2024-09-17T04:05:24.224171+0000","last_unstale":"2024-09-17T04:05:26.499682+0000","last_undegraded":"2024-09-17T04:05:26.499682+0000","last_fullsized":"2024-09-17T04:05:26.499682+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:08.223211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228423+0000","last_change":"2024-09-17T04:05:25.238743+0000","last_active":"2024-09-17T04:05:26.228423+0000","last_peered":"2024-09-17T04:05:26.228423+0000","last_clean":"2024-09-17T04:05:26.228423+0000","last_became_active":"2024-09-17T04:05:25.233859+0000","last_became_peered":"2024-09-17T04:05:25.233859+0000","last_unstale":"2024-09-17T04:05:26.228423+0000","last_undegraded":"2024-09-17T04:05:26.228423+0000","last_fullsized":"2024-09-17T04:05:26.228423+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.941861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228763+0000","last_change":"2024-09-17T04:05:24.232297+0000","last_active":"2024-09-17T04:05:26.228763+0000","last_peered":"2024-09-17T04:05:26.228763+0000","last_clean":"2024-09-17T04:05:26.228763+0000","last_became_active":"2024-09-17T04:05:24.229674+0000","last_became_peered":"2024-09-17T04:05:24.229674+0000","last_unstale":"2024-09-17T04:05:26.228763+0000","last_undegraded":"2024-09-17T04:05:26.228763+0000","last_fullsized":"2024-09-17T04:05:26.228763+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:16.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225858+0000","last_change":"2024-09-17T04:05:25.225732+0000","last_active":"2024-09-17T04:05:26.225858+0000","last_peered":"2024-09-17T04:05:26.225858+0000","last_clean":"2024-09-17T04:05:26.225858+0000","last_became_active":"2024-09-17T04:05:25.225325+0000","last_became_peered":"2024-09-17T04:05:25.225325+0000","last_unstale":"2024-09-17T04:05:26.225858+0000","last_undegraded":"2024-09-17T04:05:26.225858+0000","last_fullsized":"2024-09-17T04:05:26.225858+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:08.117034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"218'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228547+0000","last_change":"2024-09-17T04:05:24.232345+0000","last_active":"2024-09-17T04:05:26.228547+0000","last_peered":"2024-09-17T04:05:26.228547+0000","last_clean":"2024-09-17T04:05:26.228547+0000","last_became_active":"2024-09-17T04:05:24.231921+0000","last_became_peered":"2024-09-17T04:05:24.231921+0000","last_unstale":"2024-09-17T04:05:26.228547+0000","last_undegraded":"2024-09-17T04:05:26.228547+0000","last_fullsized":"2024-09-17T04:05:26.228547+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:22:05.051896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228618+0000","last_change":"2024-09-17T04:05:25.241564+0000","last_active":"2024-09-17T04:05:26.228618+0000","last_peered":"2024-09-17T04:05:26.228618+0000","last_clean":"2024-09-17T04:05:26.228618+0000","last_became_active":"2024-09-17T04:05:25.241265+0000","last_became_peered":"2024-09-17T04:05:25.241265+0000","last_unstale":"2024-09-17T04:05:26.228618+0000","last_undegraded":"2024-09-17T04:05:26.228618+0000","last_fullsized":"2024-09-17T04:05:26.228618+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:41.708564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500526+0000","last_change":"2024-09-17T04:05:24.227599+0000","last_active":"2024-09-17T04:05:26.500526+0000","last_peered":"2024-09-17T04:05:26.500526+0000","last_clean":"2024-09-17T04:05:26.500526+0000","last_became_active":"2024-09-17T04:05:24.225987+0000","last_became_peered":"2024-09-17T04:05:24.225987+0000","last_unstale":"2024-09-17T04:05:26.500526+0000","last_undegraded":"2024-09-17T04:05:26.500526+0000","last_fullsized":"2024-09-17T04:05:26.500526+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:05.381905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228927+0000","last_change":"2024-09-17T04:05:25.235165+0000","last_active":"2024-09-17T04:05:26.228927+0000","last_peered":"2024-09-17T04:05:26.228927+0000","last_clean":"2024-09-17T04:05:26.228927+0000","last_became_active":"2024-09-17T04:05:25.233933+0000","last_became_peered":"2024-09-17T04:05:25.233933+0000","last_unstale":"2024-09-17T04:05:26.228927+0000","last_undegraded":"2024-09-17T04:05:26.228927+0000","last_fullsized":"2024-09-17T04:05:26.228927+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:52.986685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225870+0000","last_change":"2024-09-17T04:05:24.222480+0000","last_active":"2024-09-17T04:05:26.225870+0000","last_peered":"2024-09-17T04:05:26.225870+0000","last_clean":"2024-09-17T04:05:26.225870+0000","last_became_active":"2024-09-17T04:05:24.222139+0000","last_became_peered":"2024-09-17T04:05:24.222139+0000","last_unstale":"2024-09-17T04:05:26.225870+0000","last_undegraded":"2024-09-17T04:05:26.225870+0000","last_fullsized":"2024-09-17T04:05:26.225870+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:06.341706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228474+0000","last_change":"2024-09-17T04:05:25.235547+0000","last_active":"2024-09-17T04:05:26.228474+0000","last_peered":"2024-09-17T04:05:26.228474+0000","last_clean":"2024-09-17T04:05:26.228474+0000","last_became_active":"2024-09-17T04:05:25.235183+0000","last_became_peered":"2024-09-17T04:05:25.235183+0000","last_unstale":"2024-09-17T04:05:26.228474+0000","last_undegraded":"2024-09-17T04:05:26.228474+0000","last_fullsized":"2024-09-17T04:05:26.228474+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:06.927656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425138+0000","last_change":"2024-09-17T04:05:24.228230+0000","last_active":"2024-09-17T04:05:26.425138+0000","last_peered":"2024-09-17T04:05:26.425138+0000","last_clean":"2024-09-17T04:05:26.425138+0000","last_became_active":"2024-09-17T04:05:24.227943+0000","last_became_peered":"2024-09-17T04:05:24.227943+0000","last_unstale":"2024-09-17T04:05:26.425138+0000","last_undegraded":"2024-09-17T04:05:26.425138+0000","last_fullsized":"2024-09-17T04:05:26.425138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:30.288261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228341+0000","last_change":"2024-09-17T04:05:25.226060+0000","last_active":"2024-09-17T04:05:26.228341+0000","last_peered":"2024-09-17T04:05:26.228341+0000","last_clean":"2024-09-17T04:05:26.228341+0000","last_became_active":"2024-09-17T04:05:25.225719+0000","last_became_peered":"2024-09-17T04:05:25.225719+0000","last_unstale":"2024-09-17T04:05:26.228341+0000","last_undegraded":"2024-09-17T04:05:26.228341+0000","last_fullsized":"2024-09-17T04:05:26.228341+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:03.563405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425458+0000","last_change":"2024-09-17T04:05:24.223526+0000","last_active":"2024-09-17T04:05:26.425458+0000","last_peered":"2024-09-17T04:05:26.425458+0000","last_clean":"2024-09-17T04:05:26.425458+0000","last_became_active":"2024-09-17T04:05:24.223182+0000","last_became_peered":"2024-09-17T04:05:24.223182+0000","last_unstale":"2024-09-17T04:05:26.425458+0000","last_undegraded":"2024-09-17T04:05:26.425458+0000","last_fullsized":"2024-09-17T04:05:26.425458+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:02.030836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228162+0000","last_change":"2024-09-17T04:05:25.231952+0000","last_active":"2024-09-17T04:05:26.228162+0000","last_peered":"2024-09-17T04:05:26.228162+0000","last_clean":"2024-09-17T04:05:26.228162+0000","last_became_active":"2024-09-17T04:05:25.231626+0000","last_became_peered":"2024-09-17T04:05:25.231626+0000","last_unstale":"2024-09-17T04:05:26.228162+0000","last_undegraded":"2024-09-17T04:05:26.228162+0000","last_fullsized":"2024-09-17T04:05:26.228162+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:50.434212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228121+0000","last_change":"2024-09-17T04:05:24.231774+0000","last_active":"2024-09-17T04:05:26.228121+0000","last_peered":"2024-09-17T04:05:26.228121+0000","last_clean":"2024-09-17T04:05:26.228121+0000","last_became_active":"2024-09-17T04:05:24.231438+0000","last_became_peered":"2024-09-17T04:05:24.231438+0000","last_unstale":"2024-09-17T04:05:26.228121+0000","last_undegraded":"2024-09-17T04:05:26.228121+0000","last_fullsized":"2024-09-17T04:05:26.228121+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:37.625328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228104+0000","last_change":"2024-09-17T04:05:25.248748+0000","last_active":"2024-09-17T04:05:26.228104+0000","last_peered":"2024-09-17T04:05:26.228104+0000","last_clean":"2024-09-17T04:05:26.228104+0000","last_became_active":"2024-09-17T04:05:25.248296+0000","last_became_peered":"2024-09-17T04:05:25.248296+0000","last_unstale":"2024-09-17T04:05:26.228104+0000","last_undegraded":"2024-09-17T04:05:26.228104+0000","last_fullsized":"2024-09-17T04:05:26.228104+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:37.086932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225734+0000","last_change":"2024-09-17T04:05:24.220172+0000","last_active":"2024-09-17T04:05:26.225734+0000","last_peered":"2024-09-17T04:05:26.225734+0000","last_clean":"2024-09-17T04:05:26.225734+0000","last_became_active":"2024-09-17T04:05:24.219869+0000","last_became_peered":"2024-09-17T04:05:24.219869+0000","last_unstale":"2024-09-17T04:05:26.225734+0000","last_undegraded":"2024-09-17T04:05:26.225734+0000","last_fullsized":"2024-09-17T04:05:26.225734+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:18:39.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499599+0000","last_change":"2024-09-17T04:05:25.230118+0000","last_active":"2024-09-17T04:05:26.499599+0000","last_peered":"2024-09-17T04:05:26.499599+0000","last_clean":"2024-09-17T04:05:26.499599+0000","last_became_active":"2024-09-17T04:05:25.229777+0000","last_became_peered":"2024-09-17T04:05:25.229777+0000","last_unstale":"2024-09-17T04:05:26.499599+0000","last_undegraded":"2024-09-17T04:05:26.499599+0000","last_fullsized":"2024-09-17T04:05:26.499599+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:04.076093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":475,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.498879+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:05:26.498879+0000","last_peered":"2024-09-17T04:05:26.498879+0000","last_clean":"2024-09-17T04:05:26.498879+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:05:26.498879+0000","last_undegraded":"2024-09-17T04:05:26.498879+0000","last_fullsized":"2024-09-17T04:05:26.498879+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509635,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3350528,"data_stored":1805685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509638,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29716,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342124,"statfs":{"total":96636764160,"available":96606334976,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509637,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2772992,"data_stored":1216915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509635,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56100,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315740,"statfs":{"total":96636764160,"available":96579317760,"internally_reserved":0,"allocated":2768896,"data_stored":1217441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509637,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509637,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2768896,"data_stored":1216023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542339,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2760704,"data_stored":1215359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542340,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3346432,"data_stored":1805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:05:34.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:05:34.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 -- 172.21.15.129:0/3086275753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f8003fd0 msgr2=0x7f60f8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 --2- 172.21.15.129:0/3086275753 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f60f8003fd0 0x7f60f8004800 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f61140ca870 tx=0x7f60f8005830 comp rx=0 tx=0).stop 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f6113e00640 1 -- 172.21.15.129:0/3086275753 reap_dead start 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 -- 172.21.15.129:0/3086275753 shutdown_connections 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 -- 172.21.15.129:0/3086275753 >> 172.21.15.129:0/3086275753 conn(0x7f611406d410 msgr2=0x7f61140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 -- 172.21.15.129:0/3086275753 shutdown_connections 2024-09-17T04:05:34.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.863+0000 7f611a400640 1 -- 172.21.15.129:0/3086275753 wait complete. 2024-09-17T04:05:34.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.865+0000 7f611a400640 1 Processor -- start 2024-09-17T04:05:34.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.866+0000 7f611a400640 1 -- start start 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 0x7f61141f7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61140ae420 0x7f61140ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6114160240 con 0x7f61140c6430 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f61141612e0 con 0x7f60f8003fd0 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f611a400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f61140cd1c0 con 0x7f61140ae420 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f6113e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f6113e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53238/0 (socket says 172.21.15.129:53238) 2024-09-17T04:05:34.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.867+0000 7f6113e00640 1 -- 172.21.15.129:0/3922301089 learned_addr learned my addr 172.21.15.129:0/3922301089 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 0x7f61141f7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6112a00640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61140ae420 0x7f61140ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 -- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61140ae420 msgr2=0x7f61140ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61140ae420 0x7f61140ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 -- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 msgr2=0x7f61141f7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 0x7f61141f7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 -- 172.21.15.129:0/3922301089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61141f7dd0 con 0x7f61140c6430 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 0x7f61141f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f6113e00640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f610c002230 tx=0x7f610c006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c00fa20 con 0x7f61140c6430 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f610c018070 con 0x7f61140c6430 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61140aed70 con 0x7f61140c6430 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f610c00fe80 con 0x7f61140c6430 2024-09-17T04:05:34.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.868+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61140b9f00 con 0x7f61140c6430 2024-09-17T04:05:34.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.870+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61140ccc80 con 0x7f61140c6430 2024-09-17T04:05:34.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.871+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f610c01d460 con 0x7f61140c6430 2024-09-17T04:05:34.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.872+0000 7f610b400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 0x7f60f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:34.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.872+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f610c09b620 con 0x7f61140c6430 2024-09-17T04:05:34.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.872+0000 7f6113400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 0x7f60f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:34.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.873+0000 7f6113400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 0x7f60f407b0d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6104009c50 tx=0x7f6104005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:34.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:34.876+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f61140aed70 con 0x7f61140c6430 2024-09-17T04:05:35.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.057+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f611406a140 con 0x7f60f4078bc0 2024-09-17T04:05:35.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.062+0000 7f610b400640 1 -- 172.21.15.129:0/3922301089 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252399 (secure 0 0 0) 0x7f610405ea00 con 0x7f60f4078bc0 2024-09-17T04:05:35.065 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:35.067 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:05:35.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 msgr2=0x7f60f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 0x7f60f407b0d0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6104009c50 tx=0x7f6104005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 msgr2=0x7f61140c6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f610c002230 tx=0x7f610c006720 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 shutdown_connections 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f60f4078bc0 0x7f60f407b0d0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61140c6430 0x7f61140c6840 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61140ae420 0x7f61140ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 --2- 172.21.15.129:0/3922301089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f60f8003fd0 0x7f61141f7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 >> 172.21.15.129:0/3922301089 conn(0x7f611406d410 msgr2=0x7f611415e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 shutdown_connections 2024-09-17T04:05:35.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.067+0000 7f611a400640 1 -- 172.21.15.129:0/3922301089 wait complete. 2024-09-17T04:05:35.089 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":697,"stamp":"2024-09-17T04:05:34.570826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402736,"kb_used_data":22728,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754571984,"statfs":{"total":773094113280,"available":772681711616,"internally_reserved":0,"allocated":23273472,"data_stored":10907532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.363763"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500684+0000","last_change":"2024-09-17T04:05:25.221229+0000","last_active":"2024-09-17T04:05:26.500684+0000","last_peered":"2024-09-17T04:05:26.500684+0000","last_clean":"2024-09-17T04:05:26.500684+0000","last_became_active":"2024-09-17T04:05:25.220873+0000","last_became_peered":"2024-09-17T04:05:25.220873+0000","last_unstale":"2024-09-17T04:05:26.500684+0000","last_undegraded":"2024-09-17T04:05:26.500684+0000","last_fullsized":"2024-09-17T04:05:26.500684+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:38:14.801493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500862+0000","last_change":"2024-09-17T04:05:25.222366+0000","last_active":"2024-09-17T04:05:26.500862+0000","last_peered":"2024-09-17T04:05:26.500862+0000","last_clean":"2024-09-17T04:05:26.500862+0000","last_became_active":"2024-09-17T04:05:25.220987+0000","last_became_peered":"2024-09-17T04:05:25.220987+0000","last_unstale":"2024-09-17T04:05:26.500862+0000","last_undegraded":"2024-09-17T04:05:26.500862+0000","last_fullsized":"2024-09-17T04:05:26.500862+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:23.139209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228756+0000","last_change":"2024-09-17T04:05:25.235978+0000","last_active":"2024-09-17T04:05:26.228756+0000","last_peered":"2024-09-17T04:05:26.228756+0000","last_clean":"2024-09-17T04:05:26.228756+0000","last_became_active":"2024-09-17T04:05:25.234686+0000","last_became_peered":"2024-09-17T04:05:25.234686+0000","last_unstale":"2024-09-17T04:05:26.228756+0000","last_undegraded":"2024-09-17T04:05:26.228756+0000","last_fullsized":"2024-09-17T04:05:26.228756+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:38:06.764889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228350+0000","last_change":"2024-09-17T04:05:25.234958+0000","last_active":"2024-09-17T04:05:26.228350+0000","last_peered":"2024-09-17T04:05:26.228350+0000","last_clean":"2024-09-17T04:05:26.228350+0000","last_became_active":"2024-09-17T04:05:25.233705+0000","last_became_peered":"2024-09-17T04:05:25.233705+0000","last_unstale":"2024-09-17T04:05:26.228350+0000","last_undegraded":"2024-09-17T04:05:26.228350+0000","last_fullsized":"2024-09-17T04:05:26.228350+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:29.142246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228296+0000","last_change":"2024-09-17T04:05:25.234011+0000","last_active":"2024-09-17T04:05:26.228296+0000","last_peered":"2024-09-17T04:05:26.228296+0000","last_clean":"2024-09-17T04:05:26.228296+0000","last_became_active":"2024-09-17T04:05:25.233676+0000","last_became_peered":"2024-09-17T04:05:25.233676+0000","last_unstale":"2024-09-17T04:05:26.228296+0000","last_undegraded":"2024-09-17T04:05:26.228296+0000","last_fullsized":"2024-09-17T04:05:26.228296+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436570+0000","last_change":"2024-09-17T04:05:25.232509+0000","last_active":"2024-09-17T04:05:26.436570+0000","last_peered":"2024-09-17T04:05:26.436570+0000","last_clean":"2024-09-17T04:05:26.436570+0000","last_became_active":"2024-09-17T04:05:25.232170+0000","last_became_peered":"2024-09-17T04:05:25.232170+0000","last_unstale":"2024-09-17T04:05:26.436570+0000","last_undegraded":"2024-09-17T04:05:26.436570+0000","last_fullsized":"2024-09-17T04:05:26.436570+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:40.882023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500029+0000","last_change":"2024-09-17T04:05:25.217581+0000","last_active":"2024-09-17T04:05:26.500029+0000","last_peered":"2024-09-17T04:05:26.500029+0000","last_clean":"2024-09-17T04:05:26.500029+0000","last_became_active":"2024-09-17T04:05:25.217074+0000","last_became_peered":"2024-09-17T04:05:25.217074+0000","last_unstale":"2024-09-17T04:05:26.500029+0000","last_undegraded":"2024-09-17T04:05:26.500029+0000","last_fullsized":"2024-09-17T04:05:26.500029+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:34:30.412641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499223+0000","last_change":"2024-09-17T04:05:25.226793+0000","last_active":"2024-09-17T04:05:26.499223+0000","last_peered":"2024-09-17T04:05:26.499223+0000","last_clean":"2024-09-17T04:05:26.499223+0000","last_became_active":"2024-09-17T04:05:25.226356+0000","last_became_peered":"2024-09-17T04:05:25.226356+0000","last_unstale":"2024-09-17T04:05:26.499223+0000","last_undegraded":"2024-09-17T04:05:26.499223+0000","last_fullsized":"2024-09-17T04:05:26.499223+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:10:40.221203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425646+0000","last_change":"2024-09-17T04:05:25.224713+0000","last_active":"2024-09-17T04:05:26.425646+0000","last_peered":"2024-09-17T04:05:26.425646+0000","last_clean":"2024-09-17T04:05:26.425646+0000","last_became_active":"2024-09-17T04:05:25.224391+0000","last_became_peered":"2024-09-17T04:05:25.224391+0000","last_unstale":"2024-09-17T04:05:26.425646+0000","last_undegraded":"2024-09-17T04:05:26.425646+0000","last_fullsized":"2024-09-17T04:05:26.425646+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:01:16.385982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500080+0000","last_change":"2024-09-17T04:05:25.226863+0000","last_active":"2024-09-17T04:05:26.500080+0000","last_peered":"2024-09-17T04:05:26.500080+0000","last_clean":"2024-09-17T04:05:26.500080+0000","last_became_active":"2024-09-17T04:05:25.226508+0000","last_became_peered":"2024-09-17T04:05:25.226508+0000","last_unstale":"2024-09-17T04:05:26.500080+0000","last_undegraded":"2024-09-17T04:05:26.500080+0000","last_fullsized":"2024-09-17T04:05:26.500080+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:56.644223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500120+0000","last_change":"2024-09-17T04:05:25.228022+0000","last_active":"2024-09-17T04:05:26.500120+0000","last_peered":"2024-09-17T04:05:26.500120+0000","last_clean":"2024-09-17T04:05:26.500120+0000","last_became_active":"2024-09-17T04:05:25.227659+0000","last_became_peered":"2024-09-17T04:05:25.227659+0000","last_unstale":"2024-09-17T04:05:26.500120+0000","last_undegraded":"2024-09-17T04:05:26.500120+0000","last_fullsized":"2024-09-17T04:05:26.500120+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:16:51.022680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500106+0000","last_change":"2024-09-17T04:05:25.230634+0000","last_active":"2024-09-17T04:05:26.500106+0000","last_peered":"2024-09-17T04:05:26.500106+0000","last_clean":"2024-09-17T04:05:26.500106+0000","last_became_active":"2024-09-17T04:05:25.229125+0000","last_became_peered":"2024-09-17T04:05:25.229125+0000","last_unstale":"2024-09-17T04:05:26.500106+0000","last_undegraded":"2024-09-17T04:05:26.500106+0000","last_fullsized":"2024-09-17T04:05:26.500106+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:41:13.538395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228383+0000","last_change":"2024-09-17T04:05:25.238984+0000","last_active":"2024-09-17T04:05:26.228383+0000","last_peered":"2024-09-17T04:05:26.228383+0000","last_clean":"2024-09-17T04:05:26.228383+0000","last_became_active":"2024-09-17T04:05:25.233999+0000","last_became_peered":"2024-09-17T04:05:25.233999+0000","last_unstale":"2024-09-17T04:05:26.228383+0000","last_undegraded":"2024-09-17T04:05:26.228383+0000","last_fullsized":"2024-09-17T04:05:26.228383+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:54:56.858787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228099+0000","last_change":"2024-09-17T04:05:25.232964+0000","last_active":"2024-09-17T04:05:26.228099+0000","last_peered":"2024-09-17T04:05:26.228099+0000","last_clean":"2024-09-17T04:05:26.228099+0000","last_became_active":"2024-09-17T04:05:25.232684+0000","last_became_peered":"2024-09-17T04:05:25.232684+0000","last_unstale":"2024-09-17T04:05:26.228099+0000","last_undegraded":"2024-09-17T04:05:26.228099+0000","last_fullsized":"2024-09-17T04:05:26.228099+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:07:37.598051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228177+0000","last_change":"2024-09-17T04:05:25.226026+0000","last_active":"2024-09-17T04:05:26.228177+0000","last_peered":"2024-09-17T04:05:26.228177+0000","last_clean":"2024-09-17T04:05:26.228177+0000","last_became_active":"2024-09-17T04:05:25.225661+0000","last_became_peered":"2024-09-17T04:05:25.225661+0000","last_unstale":"2024-09-17T04:05:26.228177+0000","last_undegraded":"2024-09-17T04:05:26.228177+0000","last_fullsized":"2024-09-17T04:05:26.228177+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:38.988518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436505+0000","last_change":"2024-09-17T04:05:25.230601+0000","last_active":"2024-09-17T04:05:26.436505+0000","last_peered":"2024-09-17T04:05:26.436505+0000","last_clean":"2024-09-17T04:05:26.436505+0000","last_became_active":"2024-09-17T04:05:25.230028+0000","last_became_peered":"2024-09-17T04:05:25.230028+0000","last_unstale":"2024-09-17T04:05:26.436505+0000","last_undegraded":"2024-09-17T04:05:26.436505+0000","last_fullsized":"2024-09-17T04:05:26.436505+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:52:57.752381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225608+0000","last_change":"2024-09-17T04:05:25.227454+0000","last_active":"2024-09-17T04:05:26.225608+0000","last_peered":"2024-09-17T04:05:26.225608+0000","last_clean":"2024-09-17T04:05:26.225608+0000","last_became_active":"2024-09-17T04:05:25.226074+0000","last_became_peered":"2024-09-17T04:05:25.226074+0000","last_unstale":"2024-09-17T04:05:26.225608+0000","last_undegraded":"2024-09-17T04:05:26.225608+0000","last_fullsized":"2024-09-17T04:05:26.225608+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:20.806967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500764+0000","last_change":"2024-09-17T04:05:25.217557+0000","last_active":"2024-09-17T04:05:26.500764+0000","last_peered":"2024-09-17T04:05:26.500764+0000","last_clean":"2024-09-17T04:05:26.500764+0000","last_became_active":"2024-09-17T04:05:25.217163+0000","last_became_peered":"2024-09-17T04:05:25.217163+0000","last_unstale":"2024-09-17T04:05:26.500764+0000","last_undegraded":"2024-09-17T04:05:26.500764+0000","last_fullsized":"2024-09-17T04:05:26.500764+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:28.881707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500523+0000","last_change":"2024-09-17T04:05:25.219655+0000","last_active":"2024-09-17T04:05:26.500523+0000","last_peered":"2024-09-17T04:05:26.500523+0000","last_clean":"2024-09-17T04:05:26.500523+0000","last_became_active":"2024-09-17T04:05:25.219345+0000","last_became_peered":"2024-09-17T04:05:25.219345+0000","last_unstale":"2024-09-17T04:05:26.500523+0000","last_undegraded":"2024-09-17T04:05:26.500523+0000","last_fullsized":"2024-09-17T04:05:26.500523+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:59.868383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436713+0000","last_change":"2024-09-17T04:05:25.227398+0000","last_active":"2024-09-17T04:05:26.436713+0000","last_peered":"2024-09-17T04:05:26.436713+0000","last_clean":"2024-09-17T04:05:26.436713+0000","last_became_active":"2024-09-17T04:05:25.227065+0000","last_became_peered":"2024-09-17T04:05:25.227065+0000","last_unstale":"2024-09-17T04:05:26.436713+0000","last_undegraded":"2024-09-17T04:05:26.436713+0000","last_fullsized":"2024-09-17T04:05:26.436713+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:28:41.966823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425500+0000","last_change":"2024-09-17T04:05:25.231383+0000","last_active":"2024-09-17T04:05:26.425500+0000","last_peered":"2024-09-17T04:05:26.425500+0000","last_clean":"2024-09-17T04:05:26.425500+0000","last_became_active":"2024-09-17T04:05:25.231058+0000","last_became_peered":"2024-09-17T04:05:25.231058+0000","last_unstale":"2024-09-17T04:05:26.425500+0000","last_undegraded":"2024-09-17T04:05:26.425500+0000","last_fullsized":"2024-09-17T04:05:26.425500+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:03:13.757807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425336+0000","last_change":"2024-09-17T04:05:25.223284+0000","last_active":"2024-09-17T04:05:26.425336+0000","last_peered":"2024-09-17T04:05:26.425336+0000","last_clean":"2024-09-17T04:05:26.425336+0000","last_became_active":"2024-09-17T04:05:25.221020+0000","last_became_peered":"2024-09-17T04:05:25.221020+0000","last_unstale":"2024-09-17T04:05:26.425336+0000","last_undegraded":"2024-09-17T04:05:26.425336+0000","last_fullsized":"2024-09-17T04:05:26.425336+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:09.253541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500358+0000","last_change":"2024-09-17T04:05:25.226879+0000","last_active":"2024-09-17T04:05:26.500358+0000","last_peered":"2024-09-17T04:05:26.500358+0000","last_clean":"2024-09-17T04:05:26.500358+0000","last_became_active":"2024-09-17T04:05:25.226571+0000","last_became_peered":"2024-09-17T04:05:25.226571+0000","last_unstale":"2024-09-17T04:05:26.500358+0000","last_undegraded":"2024-09-17T04:05:26.500358+0000","last_fullsized":"2024-09-17T04:05:26.500358+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:45.293995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225827+0000","last_change":"2024-09-17T04:05:25.229114+0000","last_active":"2024-09-17T04:05:26.225827+0000","last_peered":"2024-09-17T04:05:26.225827+0000","last_clean":"2024-09-17T04:05:26.225827+0000","last_became_active":"2024-09-17T04:05:25.228819+0000","last_became_peered":"2024-09-17T04:05:25.228819+0000","last_unstale":"2024-09-17T04:05:26.225827+0000","last_undegraded":"2024-09-17T04:05:26.225827+0000","last_fullsized":"2024-09-17T04:05:26.225827+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:48.170272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228576+0000","last_change":"2024-09-17T04:05:25.244112+0000","last_active":"2024-09-17T04:05:26.228576+0000","last_peered":"2024-09-17T04:05:26.228576+0000","last_clean":"2024-09-17T04:05:26.228576+0000","last_became_active":"2024-09-17T04:05:25.243746+0000","last_became_peered":"2024-09-17T04:05:25.243746+0000","last_unstale":"2024-09-17T04:05:26.228576+0000","last_undegraded":"2024-09-17T04:05:26.228576+0000","last_fullsized":"2024-09-17T04:05:26.228576+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228461+0000","last_change":"2024-09-17T04:05:25.236197+0000","last_active":"2024-09-17T04:05:26.228461+0000","last_peered":"2024-09-17T04:05:26.228461+0000","last_clean":"2024-09-17T04:05:26.228461+0000","last_became_active":"2024-09-17T04:05:25.233570+0000","last_became_peered":"2024-09-17T04:05:25.233570+0000","last_unstale":"2024-09-17T04:05:26.228461+0000","last_undegraded":"2024-09-17T04:05:26.228461+0000","last_fullsized":"2024-09-17T04:05:26.228461+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:08:52.842381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500441+0000","last_change":"2024-09-17T04:05:25.230257+0000","last_active":"2024-09-17T04:05:26.500441+0000","last_peered":"2024-09-17T04:05:26.500441+0000","last_clean":"2024-09-17T04:05:26.500441+0000","last_became_active":"2024-09-17T04:05:25.229043+0000","last_became_peered":"2024-09-17T04:05:25.229043+0000","last_unstale":"2024-09-17T04:05:26.500441+0000","last_undegraded":"2024-09-17T04:05:26.500441+0000","last_fullsized":"2024-09-17T04:05:26.500441+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:03.038463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228701+0000","last_change":"2024-09-17T04:05:25.242725+0000","last_active":"2024-09-17T04:05:26.228701+0000","last_peered":"2024-09-17T04:05:26.228701+0000","last_clean":"2024-09-17T04:05:26.228701+0000","last_became_active":"2024-09-17T04:05:25.239325+0000","last_became_peered":"2024-09-17T04:05:25.239325+0000","last_unstale":"2024-09-17T04:05:26.228701+0000","last_undegraded":"2024-09-17T04:05:26.228701+0000","last_fullsized":"2024-09-17T04:05:26.228701+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:06:29.392591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226066+0000","last_change":"2024-09-17T04:05:25.226877+0000","last_active":"2024-09-17T04:05:26.226066+0000","last_peered":"2024-09-17T04:05:26.226066+0000","last_clean":"2024-09-17T04:05:26.226066+0000","last_became_active":"2024-09-17T04:05:25.226520+0000","last_became_peered":"2024-09-17T04:05:25.226520+0000","last_unstale":"2024-09-17T04:05:26.226066+0000","last_undegraded":"2024-09-17T04:05:26.226066+0000","last_fullsized":"2024-09-17T04:05:26.226066+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:15.166447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228380+0000","last_change":"2024-09-17T04:05:25.232439+0000","last_active":"2024-09-17T04:05:26.228380+0000","last_peered":"2024-09-17T04:05:26.228380+0000","last_clean":"2024-09-17T04:05:26.228380+0000","last_became_active":"2024-09-17T04:05:25.230995+0000","last_became_peered":"2024-09-17T04:05:25.230995+0000","last_unstale":"2024-09-17T04:05:26.228380+0000","last_undegraded":"2024-09-17T04:05:26.228380+0000","last_fullsized":"2024-09-17T04:05:26.228380+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:49.397560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499086+0000","last_change":"2024-09-17T04:05:25.225290+0000","last_active":"2024-09-17T04:05:26.499086+0000","last_peered":"2024-09-17T04:05:26.499086+0000","last_clean":"2024-09-17T04:05:26.499086+0000","last_became_active":"2024-09-17T04:05:25.224935+0000","last_became_peered":"2024-09-17T04:05:25.224935+0000","last_unstale":"2024-09-17T04:05:26.499086+0000","last_undegraded":"2024-09-17T04:05:26.499086+0000","last_fullsized":"2024-09-17T04:05:26.499086+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:00.397591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500413+0000","last_change":"2024-09-17T04:05:25.231448+0000","last_active":"2024-09-17T04:05:26.500413+0000","last_peered":"2024-09-17T04:05:26.500413+0000","last_clean":"2024-09-17T04:05:26.500413+0000","last_became_active":"2024-09-17T04:05:25.229864+0000","last_became_peered":"2024-09-17T04:05:25.229864+0000","last_unstale":"2024-09-17T04:05:26.500413+0000","last_undegraded":"2024-09-17T04:05:26.500413+0000","last_fullsized":"2024-09-17T04:05:26.500413+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:31.295888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"217'4","reported_seq":19,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226006+0000","last_change":"2024-09-17T04:05:24.220787+0000","last_active":"2024-09-17T04:05:26.226006+0000","last_peered":"2024-09-17T04:05:26.226006+0000","last_clean":"2024-09-17T04:05:26.226006+0000","last_became_active":"2024-09-17T04:05:24.220468+0000","last_became_peered":"2024-09-17T04:05:24.220468+0000","last_unstale":"2024-09-17T04:05:26.226006+0000","last_undegraded":"2024-09-17T04:05:26.226006+0000","last_fullsized":"2024-09-17T04:05:26.226006+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:23.868440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228637+0000","last_change":"2024-09-17T04:05:25.235813+0000","last_active":"2024-09-17T04:05:26.228637+0000","last_peered":"2024-09-17T04:05:26.228637+0000","last_clean":"2024-09-17T04:05:26.228637+0000","last_became_active":"2024-09-17T04:05:25.235483+0000","last_became_peered":"2024-09-17T04:05:25.235483+0000","last_unstale":"2024-09-17T04:05:26.228637+0000","last_undegraded":"2024-09-17T04:05:26.228637+0000","last_fullsized":"2024-09-17T04:05:26.228637+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:08:06.428970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228581+0000","last_change":"2024-09-17T04:05:24.231055+0000","last_active":"2024-09-17T04:05:26.228581+0000","last_peered":"2024-09-17T04:05:26.228581+0000","last_clean":"2024-09-17T04:05:26.228581+0000","last_became_active":"2024-09-17T04:05:24.222063+0000","last_became_peered":"2024-09-17T04:05:24.222063+0000","last_unstale":"2024-09-17T04:05:26.228581+0000","last_undegraded":"2024-09-17T04:05:26.228581+0000","last_fullsized":"2024-09-17T04:05:26.228581+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:56:35.224862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500629+0000","last_change":"2024-09-17T04:05:25.227705+0000","last_active":"2024-09-17T04:05:26.500629+0000","last_peered":"2024-09-17T04:05:26.500629+0000","last_clean":"2024-09-17T04:05:26.500629+0000","last_became_active":"2024-09-17T04:05:25.223453+0000","last_became_peered":"2024-09-17T04:05:25.223453+0000","last_unstale":"2024-09-17T04:05:26.500629+0000","last_undegraded":"2024-09-17T04:05:26.500629+0000","last_fullsized":"2024-09-17T04:05:26.500629+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:26.267555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437169+0000","last_change":"2024-09-17T04:05:24.228243+0000","last_active":"2024-09-17T04:05:26.437169+0000","last_peered":"2024-09-17T04:05:26.437169+0000","last_clean":"2024-09-17T04:05:26.437169+0000","last_became_active":"2024-09-17T04:05:24.227852+0000","last_became_peered":"2024-09-17T04:05:24.227852+0000","last_unstale":"2024-09-17T04:05:26.437169+0000","last_undegraded":"2024-09-17T04:05:26.437169+0000","last_fullsized":"2024-09-17T04:05:26.437169+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:02.226834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500033+0000","last_change":"2024-09-17T04:05:25.225950+0000","last_active":"2024-09-17T04:05:26.500033+0000","last_peered":"2024-09-17T04:05:26.500033+0000","last_clean":"2024-09-17T04:05:26.500033+0000","last_became_active":"2024-09-17T04:05:25.225457+0000","last_became_peered":"2024-09-17T04:05:25.225457+0000","last_unstale":"2024-09-17T04:05:26.500033+0000","last_undegraded":"2024-09-17T04:05:26.500033+0000","last_fullsized":"2024-09-17T04:05:26.500033+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:00:41.166197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437014+0000","last_change":"2024-09-17T04:05:24.226740+0000","last_active":"2024-09-17T04:05:26.437014+0000","last_peered":"2024-09-17T04:05:26.437014+0000","last_clean":"2024-09-17T04:05:26.437014+0000","last_became_active":"2024-09-17T04:05:24.226406+0000","last_became_peered":"2024-09-17T04:05:24.226406+0000","last_unstale":"2024-09-17T04:05:26.437014+0000","last_undegraded":"2024-09-17T04:05:26.437014+0000","last_fullsized":"2024-09-17T04:05:26.437014+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:00:51.681172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425319+0000","last_change":"2024-09-17T04:05:25.232667+0000","last_active":"2024-09-17T04:05:26.425319+0000","last_peered":"2024-09-17T04:05:26.425319+0000","last_clean":"2024-09-17T04:05:26.425319+0000","last_became_active":"2024-09-17T04:05:25.232379+0000","last_became_peered":"2024-09-17T04:05:25.232379+0000","last_unstale":"2024-09-17T04:05:26.425319+0000","last_undegraded":"2024-09-17T04:05:26.425319+0000","last_fullsized":"2024-09-17T04:05:26.425319+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:55:20.607889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499657+0000","last_change":"2024-09-17T04:05:24.224011+0000","last_active":"2024-09-17T04:05:26.499657+0000","last_peered":"2024-09-17T04:05:26.499657+0000","last_clean":"2024-09-17T04:05:26.499657+0000","last_became_active":"2024-09-17T04:05:24.223670+0000","last_became_peered":"2024-09-17T04:05:24.223670+0000","last_unstale":"2024-09-17T04:05:26.499657+0000","last_undegraded":"2024-09-17T04:05:26.499657+0000","last_fullsized":"2024-09-17T04:05:26.499657+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:49.853028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436826+0000","last_change":"2024-09-17T04:05:25.220380+0000","last_active":"2024-09-17T04:05:26.436826+0000","last_peered":"2024-09-17T04:05:26.436826+0000","last_clean":"2024-09-17T04:05:26.436826+0000","last_became_active":"2024-09-17T04:05:25.217028+0000","last_became_peered":"2024-09-17T04:05:25.217028+0000","last_unstale":"2024-09-17T04:05:26.436826+0000","last_undegraded":"2024-09-17T04:05:26.436826+0000","last_fullsized":"2024-09-17T04:05:26.436826+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:36:44.696425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225677+0000","last_change":"2024-09-17T04:05:24.221715+0000","last_active":"2024-09-17T04:05:26.225677+0000","last_peered":"2024-09-17T04:05:26.225677+0000","last_clean":"2024-09-17T04:05:26.225677+0000","last_became_active":"2024-09-17T04:05:24.221380+0000","last_became_peered":"2024-09-17T04:05:24.221380+0000","last_unstale":"2024-09-17T04:05:26.225677+0000","last_undegraded":"2024-09-17T04:05:26.225677+0000","last_fullsized":"2024-09-17T04:05:26.225677+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:22:33.998165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436516+0000","last_change":"2024-09-17T04:05:25.213295+0000","last_active":"2024-09-17T04:05:26.436516+0000","last_peered":"2024-09-17T04:05:26.436516+0000","last_clean":"2024-09-17T04:05:26.436516+0000","last_became_active":"2024-09-17T04:05:25.212970+0000","last_became_peered":"2024-09-17T04:05:25.212970+0000","last_unstale":"2024-09-17T04:05:26.436516+0000","last_undegraded":"2024-09-17T04:05:26.436516+0000","last_fullsized":"2024-09-17T04:05:26.436516+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:30:01.839472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228120+0000","last_change":"2024-09-17T04:05:24.234352+0000","last_active":"2024-09-17T04:05:26.228120+0000","last_peered":"2024-09-17T04:05:26.228120+0000","last_clean":"2024-09-17T04:05:26.228120+0000","last_became_active":"2024-09-17T04:05:24.234071+0000","last_became_peered":"2024-09-17T04:05:24.234071+0000","last_unstale":"2024-09-17T04:05:26.228120+0000","last_undegraded":"2024-09-17T04:05:26.228120+0000","last_fullsized":"2024-09-17T04:05:26.228120+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:37.269558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500626+0000","last_change":"2024-09-17T04:05:25.219129+0000","last_active":"2024-09-17T04:05:26.500626+0000","last_peered":"2024-09-17T04:05:26.500626+0000","last_clean":"2024-09-17T04:05:26.500626+0000","last_became_active":"2024-09-17T04:05:25.217644+0000","last_became_peered":"2024-09-17T04:05:25.217644+0000","last_unstale":"2024-09-17T04:05:26.500626+0000","last_undegraded":"2024-09-17T04:05:26.500626+0000","last_fullsized":"2024-09-17T04:05:26.500626+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:52.028296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499665+0000","last_change":"2024-09-17T04:05:24.224568+0000","last_active":"2024-09-17T04:05:26.499665+0000","last_peered":"2024-09-17T04:05:26.499665+0000","last_clean":"2024-09-17T04:05:26.499665+0000","last_became_active":"2024-09-17T04:05:24.223638+0000","last_became_peered":"2024-09-17T04:05:24.223638+0000","last_unstale":"2024-09-17T04:05:26.499665+0000","last_undegraded":"2024-09-17T04:05:26.499665+0000","last_fullsized":"2024-09-17T04:05:26.499665+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:49.701292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500661+0000","last_change":"2024-09-17T04:05:25.218519+0000","last_active":"2024-09-17T04:05:26.500661+0000","last_peered":"2024-09-17T04:05:26.500661+0000","last_clean":"2024-09-17T04:05:26.500661+0000","last_became_active":"2024-09-17T04:05:25.217254+0000","last_became_peered":"2024-09-17T04:05:25.217254+0000","last_unstale":"2024-09-17T04:05:26.500661+0000","last_undegraded":"2024-09-17T04:05:26.500661+0000","last_fullsized":"2024-09-17T04:05:26.500661+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:49.680789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.226069+0000","last_change":"2024-09-17T04:05:24.223519+0000","last_active":"2024-09-17T04:05:26.226069+0000","last_peered":"2024-09-17T04:05:26.226069+0000","last_clean":"2024-09-17T04:05:26.226069+0000","last_became_active":"2024-09-17T04:05:24.223231+0000","last_became_peered":"2024-09-17T04:05:24.223231+0000","last_unstale":"2024-09-17T04:05:26.226069+0000","last_undegraded":"2024-09-17T04:05:26.226069+0000","last_fullsized":"2024-09-17T04:05:26.226069+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:42.041674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225715+0000","last_change":"2024-09-17T04:05:25.221512+0000","last_active":"2024-09-17T04:05:26.225715+0000","last_peered":"2024-09-17T04:05:26.225715+0000","last_clean":"2024-09-17T04:05:26.225715+0000","last_became_active":"2024-09-17T04:05:25.220458+0000","last_became_peered":"2024-09-17T04:05:25.220458+0000","last_unstale":"2024-09-17T04:05:26.225715+0000","last_undegraded":"2024-09-17T04:05:26.225715+0000","last_fullsized":"2024-09-17T04:05:26.225715+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:41:55.416231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436765+0000","last_change":"2024-09-17T04:05:24.226942+0000","last_active":"2024-09-17T04:05:26.436765+0000","last_peered":"2024-09-17T04:05:26.436765+0000","last_clean":"2024-09-17T04:05:26.436765+0000","last_became_active":"2024-09-17T04:05:24.226597+0000","last_became_peered":"2024-09-17T04:05:24.226597+0000","last_unstale":"2024-09-17T04:05:26.436765+0000","last_undegraded":"2024-09-17T04:05:26.436765+0000","last_fullsized":"2024-09-17T04:05:26.436765+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:12.017426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228414+0000","last_change":"2024-09-17T04:05:25.227178+0000","last_active":"2024-09-17T04:05:26.228414+0000","last_peered":"2024-09-17T04:05:26.228414+0000","last_clean":"2024-09-17T04:05:26.228414+0000","last_became_active":"2024-09-17T04:05:25.226791+0000","last_became_peered":"2024-09-17T04:05:25.226791+0000","last_unstale":"2024-09-17T04:05:26.228414+0000","last_undegraded":"2024-09-17T04:05:26.228414+0000","last_fullsized":"2024-09-17T04:05:26.228414+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:24.402863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"217'3","reported_seq":18,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437223+0000","last_change":"2024-09-17T04:05:24.230056+0000","last_active":"2024-09-17T04:05:26.437223+0000","last_peered":"2024-09-17T04:05:26.437223+0000","last_clean":"2024-09-17T04:05:26.437223+0000","last_became_active":"2024-09-17T04:05:24.229748+0000","last_became_peered":"2024-09-17T04:05:24.229748+0000","last_unstale":"2024-09-17T04:05:26.437223+0000","last_undegraded":"2024-09-17T04:05:26.437223+0000","last_fullsized":"2024-09-17T04:05:26.437223+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:19.117891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225488+0000","last_change":"2024-09-17T04:05:25.220710+0000","last_active":"2024-09-17T04:05:26.225488+0000","last_peered":"2024-09-17T04:05:26.225488+0000","last_clean":"2024-09-17T04:05:26.225488+0000","last_became_active":"2024-09-17T04:05:25.220415+0000","last_became_peered":"2024-09-17T04:05:25.220415+0000","last_unstale":"2024-09-17T04:05:26.225488+0000","last_undegraded":"2024-09-17T04:05:26.225488+0000","last_fullsized":"2024-09-17T04:05:26.225488+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:36:46.853834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"217'6","reported_seq":21,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225389+0000","last_change":"2024-09-17T04:05:24.225084+0000","last_active":"2024-09-17T04:05:26.225389+0000","last_peered":"2024-09-17T04:05:26.225389+0000","last_clean":"2024-09-17T04:05:26.225389+0000","last_became_active":"2024-09-17T04:05:24.224781+0000","last_became_peered":"2024-09-17T04:05:24.224781+0000","last_unstale":"2024-09-17T04:05:26.225389+0000","last_undegraded":"2024-09-17T04:05:26.225389+0000","last_fullsized":"2024-09-17T04:05:26.225389+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:21.718712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225506+0000","last_change":"2024-09-17T04:05:25.219009+0000","last_active":"2024-09-17T04:05:26.225506+0000","last_peered":"2024-09-17T04:05:26.225506+0000","last_clean":"2024-09-17T04:05:26.225506+0000","last_became_active":"2024-09-17T04:05:25.218697+0000","last_became_peered":"2024-09-17T04:05:25.218697+0000","last_unstale":"2024-09-17T04:05:26.225506+0000","last_undegraded":"2024-09-17T04:05:26.225506+0000","last_fullsized":"2024-09-17T04:05:26.225506+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:46.772065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500062+0000","last_change":"2024-09-17T04:05:24.223126+0000","last_active":"2024-09-17T04:05:26.500062+0000","last_peered":"2024-09-17T04:05:26.500062+0000","last_clean":"2024-09-17T04:05:26.500062+0000","last_became_active":"2024-09-17T04:05:24.221114+0000","last_became_peered":"2024-09-17T04:05:24.221114+0000","last_unstale":"2024-09-17T04:05:26.500062+0000","last_undegraded":"2024-09-17T04:05:26.500062+0000","last_fullsized":"2024-09-17T04:05:26.500062+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:13.552456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436876+0000","last_change":"2024-09-17T04:05:25.213282+0000","last_active":"2024-09-17T04:05:26.436876+0000","last_peered":"2024-09-17T04:05:26.436876+0000","last_clean":"2024-09-17T04:05:26.436876+0000","last_became_active":"2024-09-17T04:05:25.212938+0000","last_became_peered":"2024-09-17T04:05:25.212938+0000","last_unstale":"2024-09-17T04:05:26.436876+0000","last_undegraded":"2024-09-17T04:05:26.436876+0000","last_fullsized":"2024-09-17T04:05:26.436876+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:26.820397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425400+0000","last_change":"2024-09-17T04:05:24.223560+0000","last_active":"2024-09-17T04:05:26.425400+0000","last_peered":"2024-09-17T04:05:26.425400+0000","last_clean":"2024-09-17T04:05:26.425400+0000","last_became_active":"2024-09-17T04:05:24.223302+0000","last_became_peered":"2024-09-17T04:05:24.223302+0000","last_unstale":"2024-09-17T04:05:26.425400+0000","last_undegraded":"2024-09-17T04:05:26.425400+0000","last_fullsized":"2024-09-17T04:05:26.425400+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:03:25.259928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225342+0000","last_change":"2024-09-17T04:05:25.231922+0000","last_active":"2024-09-17T04:05:26.225342+0000","last_peered":"2024-09-17T04:05:26.225342+0000","last_clean":"2024-09-17T04:05:26.225342+0000","last_became_active":"2024-09-17T04:05:25.231099+0000","last_became_peered":"2024-09-17T04:05:25.231099+0000","last_unstale":"2024-09-17T04:05:26.225342+0000","last_undegraded":"2024-09-17T04:05:26.225342+0000","last_fullsized":"2024-09-17T04:05:26.225342+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:29:35.958163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499298+0000","last_change":"2024-09-17T04:05:24.231560+0000","last_active":"2024-09-17T04:05:26.499298+0000","last_peered":"2024-09-17T04:05:26.499298+0000","last_clean":"2024-09-17T04:05:26.499298+0000","last_became_active":"2024-09-17T04:05:24.231172+0000","last_became_peered":"2024-09-17T04:05:24.231172+0000","last_unstale":"2024-09-17T04:05:26.499298+0000","last_undegraded":"2024-09-17T04:05:26.499298+0000","last_fullsized":"2024-09-17T04:05:26.499298+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:17.113825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225269+0000","last_change":"2024-09-17T04:05:25.228578+0000","last_active":"2024-09-17T04:05:26.225269+0000","last_peered":"2024-09-17T04:05:26.225269+0000","last_clean":"2024-09-17T04:05:26.225269+0000","last_became_active":"2024-09-17T04:05:25.228179+0000","last_became_peered":"2024-09-17T04:05:25.228179+0000","last_unstale":"2024-09-17T04:05:26.225269+0000","last_undegraded":"2024-09-17T04:05:26.225269+0000","last_fullsized":"2024-09-17T04:05:26.225269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:04:08.402904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"217'8","reported_seq":23,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436709+0000","last_change":"2024-09-17T04:05:24.229283+0000","last_active":"2024-09-17T04:05:26.436709+0000","last_peered":"2024-09-17T04:05:26.436709+0000","last_clean":"2024-09-17T04:05:26.436709+0000","last_became_active":"2024-09-17T04:05:24.228948+0000","last_became_peered":"2024-09-17T04:05:24.228948+0000","last_unstale":"2024-09-17T04:05:26.436709+0000","last_undegraded":"2024-09-17T04:05:26.436709+0000","last_fullsized":"2024-09-17T04:05:26.436709+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:17.532872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228381+0000","last_change":"2024-09-17T04:05:25.229771+0000","last_active":"2024-09-17T04:05:26.228381+0000","last_peered":"2024-09-17T04:05:26.228381+0000","last_clean":"2024-09-17T04:05:26.228381+0000","last_became_active":"2024-09-17T04:05:25.229452+0000","last_became_peered":"2024-09-17T04:05:25.229452+0000","last_unstale":"2024-09-17T04:05:26.228381+0000","last_undegraded":"2024-09-17T04:05:26.228381+0000","last_fullsized":"2024-09-17T04:05:26.228381+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:39.818419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500697+0000","last_change":"2024-09-17T04:05:24.222556+0000","last_active":"2024-09-17T04:05:26.500697+0000","last_peered":"2024-09-17T04:05:26.500697+0000","last_clean":"2024-09-17T04:05:26.500697+0000","last_became_active":"2024-09-17T04:05:24.222221+0000","last_became_peered":"2024-09-17T04:05:24.222221+0000","last_unstale":"2024-09-17T04:05:26.500697+0000","last_undegraded":"2024-09-17T04:05:26.500697+0000","last_fullsized":"2024-09-17T04:05:26.500697+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:24:20.842936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.501041+0000","last_change":"2024-09-17T04:05:25.244458+0000","last_active":"2024-09-17T04:05:26.501041+0000","last_peered":"2024-09-17T04:05:26.501041+0000","last_clean":"2024-09-17T04:05:26.501041+0000","last_became_active":"2024-09-17T04:05:25.244033+0000","last_became_peered":"2024-09-17T04:05:25.244033+0000","last_unstale":"2024-09-17T04:05:26.501041+0000","last_undegraded":"2024-09-17T04:05:26.501041+0000","last_fullsized":"2024-09-17T04:05:26.501041+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:00.213217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436875+0000","last_change":"2024-09-17T04:05:24.226050+0000","last_active":"2024-09-17T04:05:26.436875+0000","last_peered":"2024-09-17T04:05:26.436875+0000","last_clean":"2024-09-17T04:05:26.436875+0000","last_became_active":"2024-09-17T04:05:24.225367+0000","last_became_peered":"2024-09-17T04:05:24.225367+0000","last_unstale":"2024-09-17T04:05:26.436875+0000","last_undegraded":"2024-09-17T04:05:26.436875+0000","last_fullsized":"2024-09-17T04:05:26.436875+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:24.613188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228770+0000","last_change":"2024-09-17T04:05:25.226348+0000","last_active":"2024-09-17T04:05:26.228770+0000","last_peered":"2024-09-17T04:05:26.228770+0000","last_clean":"2024-09-17T04:05:26.228770+0000","last_became_active":"2024-09-17T04:05:25.225598+0000","last_became_peered":"2024-09-17T04:05:25.225598+0000","last_unstale":"2024-09-17T04:05:26.228770+0000","last_undegraded":"2024-09-17T04:05:26.228770+0000","last_fullsized":"2024-09-17T04:05:26.228770+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:05:59.151440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"217'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500308+0000","last_change":"2024-09-17T04:05:24.226944+0000","last_active":"2024-09-17T04:05:26.500308+0000","last_peered":"2024-09-17T04:05:26.500308+0000","last_clean":"2024-09-17T04:05:26.500308+0000","last_became_active":"2024-09-17T04:05:24.225543+0000","last_became_peered":"2024-09-17T04:05:24.225543+0000","last_unstale":"2024-09-17T04:05:26.500308+0000","last_undegraded":"2024-09-17T04:05:26.500308+0000","last_fullsized":"2024-09-17T04:05:26.500308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.257988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228745+0000","last_change":"2024-09-17T04:05:25.242208+0000","last_active":"2024-09-17T04:05:26.228745+0000","last_peered":"2024-09-17T04:05:26.228745+0000","last_clean":"2024-09-17T04:05:26.228745+0000","last_became_active":"2024-09-17T04:05:25.241886+0000","last_became_peered":"2024-09-17T04:05:25.241886+0000","last_unstale":"2024-09-17T04:05:26.228745+0000","last_undegraded":"2024-09-17T04:05:26.228745+0000","last_fullsized":"2024-09-17T04:05:26.228745+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:50.227852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228869+0000","last_change":"2024-09-17T04:05:24.229174+0000","last_active":"2024-09-17T04:05:26.228869+0000","last_peered":"2024-09-17T04:05:26.228869+0000","last_clean":"2024-09-17T04:05:26.228869+0000","last_became_active":"2024-09-17T04:05:24.228858+0000","last_became_peered":"2024-09-17T04:05:24.228858+0000","last_unstale":"2024-09-17T04:05:26.228869+0000","last_undegraded":"2024-09-17T04:05:26.228869+0000","last_fullsized":"2024-09-17T04:05:26.228869+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.505474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.436624+0000","last_change":"2024-09-17T04:05:25.226304+0000","last_active":"2024-09-17T04:05:26.436624+0000","last_peered":"2024-09-17T04:05:26.436624+0000","last_clean":"2024-09-17T04:05:26.436624+0000","last_became_active":"2024-09-17T04:05:25.224061+0000","last_became_peered":"2024-09-17T04:05:25.224061+0000","last_unstale":"2024-09-17T04:05:26.436624+0000","last_undegraded":"2024-09-17T04:05:26.436624+0000","last_fullsized":"2024-09-17T04:05:26.436624+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:55:49.229679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228504+0000","last_change":"2024-09-17T04:05:24.225036+0000","last_active":"2024-09-17T04:05:26.228504+0000","last_peered":"2024-09-17T04:05:26.228504+0000","last_clean":"2024-09-17T04:05:26.228504+0000","last_became_active":"2024-09-17T04:05:24.223209+0000","last_became_peered":"2024-09-17T04:05:24.223209+0000","last_unstale":"2024-09-17T04:05:26.228504+0000","last_undegraded":"2024-09-17T04:05:26.228504+0000","last_fullsized":"2024-09-17T04:05:26.228504+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:23:21.294118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.437062+0000","last_change":"2024-09-17T04:05:25.221299+0000","last_active":"2024-09-17T04:05:26.437062+0000","last_peered":"2024-09-17T04:05:26.437062+0000","last_clean":"2024-09-17T04:05:26.437062+0000","last_became_active":"2024-09-17T04:05:25.220970+0000","last_became_peered":"2024-09-17T04:05:25.220970+0000","last_unstale":"2024-09-17T04:05:26.437062+0000","last_undegraded":"2024-09-17T04:05:26.437062+0000","last_fullsized":"2024-09-17T04:05:26.437062+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:16:28.900393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499673+0000","last_change":"2024-09-17T04:05:24.224549+0000","last_active":"2024-09-17T04:05:26.499673+0000","last_peered":"2024-09-17T04:05:26.499673+0000","last_clean":"2024-09-17T04:05:26.499673+0000","last_became_active":"2024-09-17T04:05:24.224066+0000","last_became_peered":"2024-09-17T04:05:24.224066+0000","last_unstale":"2024-09-17T04:05:26.499673+0000","last_undegraded":"2024-09-17T04:05:26.499673+0000","last_fullsized":"2024-09-17T04:05:26.499673+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:26.556703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225575+0000","last_change":"2024-09-17T04:05:25.221813+0000","last_active":"2024-09-17T04:05:26.225575+0000","last_peered":"2024-09-17T04:05:26.225575+0000","last_clean":"2024-09-17T04:05:26.225575+0000","last_became_active":"2024-09-17T04:05:25.221550+0000","last_became_peered":"2024-09-17T04:05:25.221550+0000","last_unstale":"2024-09-17T04:05:26.225575+0000","last_undegraded":"2024-09-17T04:05:26.225575+0000","last_fullsized":"2024-09-17T04:05:26.225575+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:42.072358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499425+0000","last_change":"2024-09-17T04:05:24.229963+0000","last_active":"2024-09-17T04:05:26.499425+0000","last_peered":"2024-09-17T04:05:26.499425+0000","last_clean":"2024-09-17T04:05:26.499425+0000","last_became_active":"2024-09-17T04:05:24.229600+0000","last_became_peered":"2024-09-17T04:05:24.229600+0000","last_unstale":"2024-09-17T04:05:26.499425+0000","last_undegraded":"2024-09-17T04:05:26.499425+0000","last_fullsized":"2024-09-17T04:05:26.499425+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:44:45.305405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425310+0000","last_change":"2024-09-17T04:05:25.238027+0000","last_active":"2024-09-17T04:05:26.425310+0000","last_peered":"2024-09-17T04:05:26.425310+0000","last_clean":"2024-09-17T04:05:26.425310+0000","last_became_active":"2024-09-17T04:05:25.236850+0000","last_became_peered":"2024-09-17T04:05:25.236850+0000","last_unstale":"2024-09-17T04:05:26.425310+0000","last_undegraded":"2024-09-17T04:05:26.425310+0000","last_fullsized":"2024-09-17T04:05:26.425310+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:21:04.588444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499682+0000","last_change":"2024-09-17T04:05:24.224522+0000","last_active":"2024-09-17T04:05:26.499682+0000","last_peered":"2024-09-17T04:05:26.499682+0000","last_clean":"2024-09-17T04:05:26.499682+0000","last_became_active":"2024-09-17T04:05:24.224171+0000","last_became_peered":"2024-09-17T04:05:24.224171+0000","last_unstale":"2024-09-17T04:05:26.499682+0000","last_undegraded":"2024-09-17T04:05:26.499682+0000","last_fullsized":"2024-09-17T04:05:26.499682+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:08.223211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228423+0000","last_change":"2024-09-17T04:05:25.238743+0000","last_active":"2024-09-17T04:05:26.228423+0000","last_peered":"2024-09-17T04:05:26.228423+0000","last_clean":"2024-09-17T04:05:26.228423+0000","last_became_active":"2024-09-17T04:05:25.233859+0000","last_became_peered":"2024-09-17T04:05:25.233859+0000","last_unstale":"2024-09-17T04:05:26.228423+0000","last_undegraded":"2024-09-17T04:05:26.228423+0000","last_fullsized":"2024-09-17T04:05:26.228423+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.941861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"217'2","reported_seq":17,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228763+0000","last_change":"2024-09-17T04:05:24.232297+0000","last_active":"2024-09-17T04:05:26.228763+0000","last_peered":"2024-09-17T04:05:26.228763+0000","last_clean":"2024-09-17T04:05:26.228763+0000","last_became_active":"2024-09-17T04:05:24.229674+0000","last_became_peered":"2024-09-17T04:05:24.229674+0000","last_unstale":"2024-09-17T04:05:26.228763+0000","last_undegraded":"2024-09-17T04:05:26.228763+0000","last_fullsized":"2024-09-17T04:05:26.228763+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:58:16.479645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225858+0000","last_change":"2024-09-17T04:05:25.225732+0000","last_active":"2024-09-17T04:05:26.225858+0000","last_peered":"2024-09-17T04:05:26.225858+0000","last_clean":"2024-09-17T04:05:26.225858+0000","last_became_active":"2024-09-17T04:05:25.225325+0000","last_became_peered":"2024-09-17T04:05:25.225325+0000","last_unstale":"2024-09-17T04:05:26.225858+0000","last_undegraded":"2024-09-17T04:05:26.225858+0000","last_fullsized":"2024-09-17T04:05:26.225858+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:08.117034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"218'1","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228547+0000","last_change":"2024-09-17T04:05:24.232345+0000","last_active":"2024-09-17T04:05:26.228547+0000","last_peered":"2024-09-17T04:05:26.228547+0000","last_clean":"2024-09-17T04:05:26.228547+0000","last_became_active":"2024-09-17T04:05:24.231921+0000","last_became_peered":"2024-09-17T04:05:24.231921+0000","last_unstale":"2024-09-17T04:05:26.228547+0000","last_undegraded":"2024-09-17T04:05:26.228547+0000","last_fullsized":"2024-09-17T04:05:26.228547+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:22:05.051896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228618+0000","last_change":"2024-09-17T04:05:25.241564+0000","last_active":"2024-09-17T04:05:26.228618+0000","last_peered":"2024-09-17T04:05:26.228618+0000","last_clean":"2024-09-17T04:05:26.228618+0000","last_became_active":"2024-09-17T04:05:25.241265+0000","last_became_peered":"2024-09-17T04:05:25.241265+0000","last_unstale":"2024-09-17T04:05:26.228618+0000","last_undegraded":"2024-09-17T04:05:26.228618+0000","last_fullsized":"2024-09-17T04:05:26.228618+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:58:41.708564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.500526+0000","last_change":"2024-09-17T04:05:24.227599+0000","last_active":"2024-09-17T04:05:26.500526+0000","last_peered":"2024-09-17T04:05:26.500526+0000","last_clean":"2024-09-17T04:05:26.500526+0000","last_became_active":"2024-09-17T04:05:24.225987+0000","last_became_peered":"2024-09-17T04:05:24.225987+0000","last_unstale":"2024-09-17T04:05:26.500526+0000","last_undegraded":"2024-09-17T04:05:26.500526+0000","last_fullsized":"2024-09-17T04:05:26.500526+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:05.381905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228927+0000","last_change":"2024-09-17T04:05:25.235165+0000","last_active":"2024-09-17T04:05:26.228927+0000","last_peered":"2024-09-17T04:05:26.228927+0000","last_clean":"2024-09-17T04:05:26.228927+0000","last_became_active":"2024-09-17T04:05:25.233933+0000","last_became_peered":"2024-09-17T04:05:25.233933+0000","last_unstale":"2024-09-17T04:05:26.228927+0000","last_undegraded":"2024-09-17T04:05:26.228927+0000","last_fullsized":"2024-09-17T04:05:26.228927+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:11:52.986685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225870+0000","last_change":"2024-09-17T04:05:24.222480+0000","last_active":"2024-09-17T04:05:26.225870+0000","last_peered":"2024-09-17T04:05:26.225870+0000","last_clean":"2024-09-17T04:05:26.225870+0000","last_became_active":"2024-09-17T04:05:24.222139+0000","last_became_peered":"2024-09-17T04:05:24.222139+0000","last_unstale":"2024-09-17T04:05:26.225870+0000","last_undegraded":"2024-09-17T04:05:26.225870+0000","last_fullsized":"2024-09-17T04:05:26.225870+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:06.341706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228474+0000","last_change":"2024-09-17T04:05:25.235547+0000","last_active":"2024-09-17T04:05:26.228474+0000","last_peered":"2024-09-17T04:05:26.228474+0000","last_clean":"2024-09-17T04:05:26.228474+0000","last_became_active":"2024-09-17T04:05:25.235183+0000","last_became_peered":"2024-09-17T04:05:25.235183+0000","last_unstale":"2024-09-17T04:05:26.228474+0000","last_undegraded":"2024-09-17T04:05:26.228474+0000","last_fullsized":"2024-09-17T04:05:26.228474+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:06.927656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425138+0000","last_change":"2024-09-17T04:05:24.228230+0000","last_active":"2024-09-17T04:05:26.425138+0000","last_peered":"2024-09-17T04:05:26.425138+0000","last_clean":"2024-09-17T04:05:26.425138+0000","last_became_active":"2024-09-17T04:05:24.227943+0000","last_became_peered":"2024-09-17T04:05:24.227943+0000","last_unstale":"2024-09-17T04:05:26.425138+0000","last_undegraded":"2024-09-17T04:05:26.425138+0000","last_fullsized":"2024-09-17T04:05:26.425138+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:30.288261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228341+0000","last_change":"2024-09-17T04:05:25.226060+0000","last_active":"2024-09-17T04:05:26.228341+0000","last_peered":"2024-09-17T04:05:26.228341+0000","last_clean":"2024-09-17T04:05:26.228341+0000","last_became_active":"2024-09-17T04:05:25.225719+0000","last_became_peered":"2024-09-17T04:05:25.225719+0000","last_unstale":"2024-09-17T04:05:26.228341+0000","last_undegraded":"2024-09-17T04:05:26.228341+0000","last_fullsized":"2024-09-17T04:05:26.228341+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:02:03.563405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.425458+0000","last_change":"2024-09-17T04:05:24.223526+0000","last_active":"2024-09-17T04:05:26.425458+0000","last_peered":"2024-09-17T04:05:26.425458+0000","last_clean":"2024-09-17T04:05:26.425458+0000","last_became_active":"2024-09-17T04:05:24.223182+0000","last_became_peered":"2024-09-17T04:05:24.223182+0000","last_unstale":"2024-09-17T04:05:26.425458+0000","last_undegraded":"2024-09-17T04:05:26.425458+0000","last_fullsized":"2024-09-17T04:05:26.425458+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:36:02.030836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228162+0000","last_change":"2024-09-17T04:05:25.231952+0000","last_active":"2024-09-17T04:05:26.228162+0000","last_peered":"2024-09-17T04:05:26.228162+0000","last_clean":"2024-09-17T04:05:26.228162+0000","last_became_active":"2024-09-17T04:05:25.231626+0000","last_became_peered":"2024-09-17T04:05:25.231626+0000","last_unstale":"2024-09-17T04:05:26.228162+0000","last_undegraded":"2024-09-17T04:05:26.228162+0000","last_fullsized":"2024-09-17T04:05:26.228162+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:21:50.434212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228121+0000","last_change":"2024-09-17T04:05:24.231774+0000","last_active":"2024-09-17T04:05:26.228121+0000","last_peered":"2024-09-17T04:05:26.228121+0000","last_clean":"2024-09-17T04:05:26.228121+0000","last_became_active":"2024-09-17T04:05:24.231438+0000","last_became_peered":"2024-09-17T04:05:24.231438+0000","last_unstale":"2024-09-17T04:05:26.228121+0000","last_undegraded":"2024-09-17T04:05:26.228121+0000","last_fullsized":"2024-09-17T04:05:26.228121+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:37.625328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.228104+0000","last_change":"2024-09-17T04:05:25.248748+0000","last_active":"2024-09-17T04:05:26.228104+0000","last_peered":"2024-09-17T04:05:26.228104+0000","last_clean":"2024-09-17T04:05:26.228104+0000","last_became_active":"2024-09-17T04:05:25.248296+0000","last_became_peered":"2024-09-17T04:05:25.248296+0000","last_unstale":"2024-09-17T04:05:26.228104+0000","last_undegraded":"2024-09-17T04:05:26.228104+0000","last_fullsized":"2024-09-17T04:05:26.228104+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:20:37.086932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":15,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.225734+0000","last_change":"2024-09-17T04:05:24.220172+0000","last_active":"2024-09-17T04:05:26.225734+0000","last_peered":"2024-09-17T04:05:26.225734+0000","last_clean":"2024-09-17T04:05:26.225734+0000","last_became_active":"2024-09-17T04:05:24.219869+0000","last_became_peered":"2024-09-17T04:05:24.219869+0000","last_unstale":"2024-09-17T04:05:26.225734+0000","last_undegraded":"2024-09-17T04:05:26.225734+0000","last_fullsized":"2024-09-17T04:05:26.225734+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":215,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:23.184263+0000","last_clean_scrub_stamp":"2024-09-17T04:05:23.184263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:18:39.944003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.499599+0000","last_change":"2024-09-17T04:05:25.230118+0000","last_active":"2024-09-17T04:05:26.499599+0000","last_peered":"2024-09-17T04:05:26.499599+0000","last_clean":"2024-09-17T04:05:26.499599+0000","last_became_active":"2024-09-17T04:05:25.229777+0000","last_became_peered":"2024-09-17T04:05:25.229777+0000","last_unstale":"2024-09-17T04:05:26.499599+0000","last_undegraded":"2024-09-17T04:05:26.499599+0000","last_fullsized":"2024-09-17T04:05:26.499599+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:05:24.191890+0000","last_clean_scrub_stamp":"2024-09-17T04:05:24.191890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:04.076093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":476,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-17T04:05:26.498957+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:05:26.498957+0000","last_peered":"2024-09-17T04:05:26.498957+0000","last_clean":"2024-09-17T04:05:26.498957+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:05:26.498957+0000","last_undegraded":"2024-09-17T04:05:26.498957+0000","last_fullsized":"2024-09-17T04:05:26.498957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509636,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52508,"kb_used_data":3272,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319332,"statfs":{"total":96636764160,"available":96582995968,"internally_reserved":0,"allocated":3350528,"data_stored":1805685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509639,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29716,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342124,"statfs":{"total":96636764160,"available":96606334976,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509638,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51944,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319896,"statfs":{"total":96636764160,"available":96583573504,"internally_reserved":0,"allocated":2772992,"data_stored":1216915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509635,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56100,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315740,"statfs":{"total":96636764160,"available":96579317760,"internally_reserved":0,"allocated":2768896,"data_stored":1217441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509637,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56084,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315756,"statfs":{"total":96636764160,"available":96579334144,"internally_reserved":0,"allocated":2752512,"data_stored":1215247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509638,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2768896,"data_stored":1216023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542340,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51940,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319900,"statfs":{"total":96636764160,"available":96583577600,"internally_reserved":0,"allocated":2760704,"data_stored":1215359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542341,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52504,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319336,"statfs":{"total":96636764160,"available":96583000064,"internally_reserved":0,"allocated":3346432,"data_stored":1805615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:05:35.093 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:05:35.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:05:35.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 -- 172.21.15.129:0/398175816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 msgr2=0x7f3c7c0b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 --2- 172.21.15.129:0/398175816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0b3750 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009830 tx=0x7f3c6802a480 comp rx=0 tx=0).stop 2024-09-17T04:05:35.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 -- 172.21.15.129:0/398175816 shutdown_connections 2024-09-17T04:05:35.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 --2- 172.21.15.129:0/398175816 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0b3750 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 --2- 172.21.15.129:0/398175816 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 0x7f3c7c160810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 -- 172.21.15.129:0/398175816 >> 172.21.15.129:0/398175816 conn(0x7f3c7c06d7a0 msgr2=0x7f3c7c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:35.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 -- 172.21.15.129:0/398175816 shutdown_connections 2024-09-17T04:05:35.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.358+0000 7f3c84000640 1 -- 172.21.15.129:0/398175816 wait complete. 2024-09-17T04:05:35.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.360+0000 7f3c84000640 1 Processor -- start 2024-09-17T04:05:35.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.360+0000 7f3c84000640 1 -- start start 2024-09-17T04:05:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.361+0000 7f3c84000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c7c0b3c90 0x7f3c7c1f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.361+0000 7f3c84000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 0x7f3c7c15d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c84000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c84000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c0cfb70 con 0x7f3c7c160d50 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c84000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7c0cf9f0 con 0x7f3c7c0b3c90 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c84000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3c7c0b40a0 con 0x7f3c7c160400 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53244/0 (socket says 172.21.15.129:53244) 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 -- 172.21.15.129:0/489067117 learned_addr learned my addr 172.21.15.129:0/489067117 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c80c00640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 0x7f3c7c15d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c81600640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c7c0b3c90 0x7f3c7c1f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 -- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 msgr2=0x7f3c7c15d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 0x7f3c7c15d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 -- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c7c0b3c90 msgr2=0x7f3c7c1f9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c7c0b3c90 0x7f3c7c1f9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.362+0000 7f3c82000640 1 -- 172.21.15.129:0/489067117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c7c15ce80 con 0x7f3c7c160d50 2024-09-17T04:05:35.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c82000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c3c0 tx=0x7f3c7400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011790 con 0x7f3c7c160d50 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c74017070 con 0x7f3c7c160d50 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7c1fa020 con 0x7f3c7c160d50 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c74011ca0 con 0x7f3c7c160d50 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.363+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c7c1f6190 con 0x7f3c7c160d50 2024-09-17T04:05:35.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.364+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c7c0cf5c0 con 0x7f3c7c160d50 2024-09-17T04:05:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.365+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3c7401c460 con 0x7f3c7c160d50 2024-09-17T04:05:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.366+0000 7f3c79600640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 0x7f3c5c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.366+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f3c7409a9c0 con 0x7f3c7c160d50 2024-09-17T04:05:35.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.367+0000 7f3c81600640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 0x7f3c5c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.367+0000 7f3c81600640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 0x7f3c5c07b0a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005fd0 tx=0x7f3c700095e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:35.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.370+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3c7c0cf5c0 con 0x7f3c7c160d50 2024-09-17T04:05:35.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.623+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f3c7c160810 con 0x7f3c7c160d50 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.627+0000 7f3c79600640 1 -- 172.21.15.129:0/489067117 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v44) ==== 190+0+147 (secure 0 0 0) 0x7f3c740627a0 con 0x7f3c7c160d50 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:05:35.627 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 msgr2=0x7f3c5c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 0x7f3c5c07b0a0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f3c70005fd0 tx=0x7f3c700095e0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 msgr2=0x7f3c7c0cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7f3c7400c3c0 tx=0x7f3c7400c880 comp rx=0 tx=0).stop 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 shutdown_connections 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3c5c078b90 0x7f3c5c07b0a0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3c7c160d50 0x7f3c7c0cdb10 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3c7c160400 0x7f3c7c15d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 --2- 172.21.15.129:0/489067117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3c7c0b3c90 0x7f3c7c1f9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.629+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 >> 172.21.15.129:0/489067117 conn(0x7f3c7c06d7a0 msgr2=0x7f3c7c073b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.630+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 shutdown_connections 2024-09-17T04:05:35.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.630+0000 7f3c84000640 1 -- 172.21.15.129:0/489067117 wait complete. 2024-09-17T04:05:35.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 -- 172.21.15.129:0/4099508211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 msgr2=0x7fb89c0cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/4099508211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c0cf6c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fb888009830 tx=0x7fb88802a4a0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 -- 172.21.15.129:0/4099508211 shutdown_connections 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/4099508211 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c0ccc80 0x7fb89c0cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/4099508211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c0cf6c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 -- 172.21.15.129:0/4099508211 >> 172.21.15.129:0/4099508211 conn(0x7fb89c06d410 msgr2=0x7fb89c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 -- 172.21.15.129:0/4099508211 shutdown_connections 2024-09-17T04:05:35.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.872+0000 7fb8a1e00640 1 -- 172.21.15.129:0/4099508211 wait complete. 2024-09-17T04:05:35.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.875+0000 7fb8a1e00640 1 Processor -- start 2024-09-17T04:05:35.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.875+0000 7fb8a1e00640 1 -- start start 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 0x7fb89c09a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c160ba0 con 0x7fb89c0ccc80 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c160a20 con 0x7fb89c0cf2b0 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb8a1e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb89c160d20 con 0x7fb89c099570 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.876+0000 7fb89a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53250/0 (socket says 172.21.15.129:53250) 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 -- 172.21.15.129:0/130644776 learned_addr learned my addr 172.21.15.129:0/130644776 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89b400640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89aa00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 0x7fb89c09a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 -- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 msgr2=0x7fb89c09a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 0x7fb89c09a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 -- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 msgr2=0x7fb89c15d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c15d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 -- 172.21.15.129:0/130644776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c15d8d0 con 0x7fb89c0ccc80 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89aa00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 0x7fb89c09a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb89a000640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fb888009950 tx=0x7fb888003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.877+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888005c90 con 0x7fb89c0ccc80 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.878+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb888005020 con 0x7fb89c0ccc80 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.878+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888005450 con 0x7fb89c0ccc80 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.878+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c0b0800 con 0x7fb89c0ccc80 2024-09-17T04:05:35.879 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.878+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c0b0d80 con 0x7fb89c0ccc80 2024-09-17T04:05:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.879+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb89c161230 con 0x7fb89c0ccc80 2024-09-17T04:05:35.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.880+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8880317c0 con 0x7fb89c0ccc80 2024-09-17T04:05:35.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.881+0000 7fb893400640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 0x7fb87c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:35.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.881+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fb8880bbd30 con 0x7fb89c0ccc80 2024-09-17T04:05:35.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.881+0000 7fb89aa00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 0x7fb87c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:35.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.882+0000 7fb89aa00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 0x7fb87c07b0a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb88c007960 tx=0x7fb88c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:35.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:35.886+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb89c15d8d0 con 0x7fb89c0ccc80 2024-09-17T04:05:36.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.104+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb89c06a140 con 0x7fb89c0ccc80 2024-09-17T04:05:36.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.105+0000 7fb893400640 1 -- 172.21.15.129:0/130644776 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5249 (secure 0 0 0) 0x7fb88803f030 con 0x7fb89c0ccc80 2024-09-17T04:05:36.105 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:36.105 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":208,"btime":"2024-09-17T04:05:28:229503+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8657,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2764153925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2764153925},{"type":"v1","addr":"172.21.15.181:6837","nonce":2764153925}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202},{"gid":10231,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/2457838594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2457838594},{"type":"v1","addr":"172.21.15.181:6836","nonce":2457838594}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202},{"gid":10237,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3249193545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3249193545},{"type":"v1","addr":"172.21.15.129:6834","nonce":3249193545}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:27.714822+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11028},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11028":{"gid":11028,"name":"c","rank":0,"incarnation":204,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/711019236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":711019236},{"type":"v1","addr":"172.21.15.129:6835","nonce":711019236}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11028,"qdb_cluster":[11028]},"id":16}]} 2024-09-17T04:05:36.106 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 208 2024-09-17T04:05:36.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 msgr2=0x7fb87c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 0x7fb87c07b0a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb88c007960 tx=0x7fb88c009040 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 msgr2=0x7fb89c0c31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7fb888009950 tx=0x7fb888003c10 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 shutdown_connections 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb87c078b90 0x7fb87c07b0a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb89c0cf2b0 0x7fb89c15d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb89c0ccc80 0x7fb89c0c31f0 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 --2- 172.21.15.129:0/130644776 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb89c099570 0x7fb89c09a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 >> 172.21.15.129:0/130644776 conn(0x7fb89c06d410 msgr2=0x7fb89c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 shutdown_connections 2024-09-17T04:05:36.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.107+0000 7fb8a1e00640 1 -- 172.21.15.129:0/130644776 wait complete. 2024-09-17T04:05:36.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 208, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:05:24.702816+0000', 'modified': '2024-09-17T04:05:27.714822+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11028}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11028': {'gid': 11028, 'name': 'c', 'rank': 0, 'incarnation': 204, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/711019236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 711019236}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 711019236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11028, 'qdb_cluster': [11028]} 2024-09-17T04:05:36.532 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:05:36.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:05:36.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3235805708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbd4003a30 msgr2=0x7fbbd4003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3235805708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbd4003a30 0x7fbbd4003e60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009ed0 tx=0x7fbbd4005830 comp rx=0 tx=0).stop 2024-09-17T04:05:36.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf5c00640 1 -- 172.21.15.129:0/3235805708 reap_dead start 2024-09-17T04:05:36.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3235805708 shutdown_connections 2024-09-17T04:05:36.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3235805708 >> 172.21.15.129:0/3235805708 conn(0x7fbbf006d410 msgr2=0x7fbbf00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3235805708 shutdown_connections 2024-09-17T04:05:36.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.757+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3235805708 wait complete. 2024-09-17T04:05:36.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.759+0000 7fbbf7c00640 1 Processor -- start 2024-09-17T04:05:36.759 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.759+0000 7fbbf7c00640 1 -- start start 2024-09-17T04:05:36.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbd4003a30 0x7fbbf015ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 0x7fbbf0073eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf00ccb40 con 0x7fbbf00c6310 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf00cc9c0 con 0x7fbbf00c5f00 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf7c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf00708e0 con 0x7fbbd4003a30 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53264/0 (socket says 172.21.15.129:53264) 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf5c00640 1 -- 172.21.15.129:0/3417828502 learned_addr learned my addr 172.21.15.129:0/3417828502 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 0x7fbbf0073eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbd4003a30 0x7fbbf015ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.761+0000 7fbbf5c00640 1 -- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbd4003a30 msgr2=0x7fbbf015ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbd4003a30 0x7fbbf015ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf5c00640 1 -- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 msgr2=0x7fbbf0073eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 0x7fbbf0073eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.763 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf5c00640 1 -- 172.21.15.129:0/3417828502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf015f250 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 0x7fbbf0073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0160420 tx=0x7fbbe80039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbecc00640 1 -- 172.21.15.129:0/3417828502 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc0013e0 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbecc00640 1 -- 172.21.15.129:0/3417828502 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe8018070 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf00c7570 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbecc00640 1 -- 172.21.15.129:0/3417828502 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8011ae0 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.762+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf00c7b30 con 0x7fbbf00c6310 2024-09-17T04:05:36.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.765+0000 7fbbecc00640 1 -- 172.21.15.129:0/3417828502 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbbe8011c80 con 0x7fbbf00c6310 2024-09-17T04:05:36.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.766+0000 7fbbecc00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 0x7fbbd007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.766+0000 7fbbecc00640 1 -- 172.21.15.129:0/3417828502 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe809b2f0 con 0x7fbbf00c6310 2024-09-17T04:05:36.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.766+0000 7fbbfec99740 1 Processor -- start 2024-09-17T04:05:36.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.766+0000 7fbbfec99740 1 -- start start 2024-09-17T04:05:36.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.766+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 0x7fbbd007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.767+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 0x7fbbd007b0d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0035000 tx=0x7fbbe0004a40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a2132b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a2132beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a21285430 con 0x557a20f3a590 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557a212755d0 con 0x557a20f5b270 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x557a21264a00 con 0x557a20f323b0 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53274/0 (socket says 172.21.15.129:53274) 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 -- 172.21.15.129:0/1045040129 learned_addr learned my addr 172.21.15.129:0/1045040129 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:36.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5200640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a2132b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbefe00640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a2132beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 -- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 msgr2=0x557a2132beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a2132beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 -- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 msgr2=0x557a2132b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a2132b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5c00640 1 -- 172.21.15.129:0/1045040129 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a2122d520 con 0x557a20f3a590 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbf5200640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a2132b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.768+0000 7fbbefe00640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a2132beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.769+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8099780 tx=0x7fbbe8062970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dec941fb6319ec9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.769+0000 7fbbe5600640 1 -- 172.21.15.129:0/1045040129 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8063720 con 0x557a20f3a590 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.769+0000 7fbbe5600640 1 -- 172.21.15.129:0/1045040129 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe8063020 con 0x557a20f3a590 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.769+0000 7fbbe5600640 1 -- 172.21.15.129:0/1045040129 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8067b00 con 0x557a20f3a590 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 -- 172.21.15.129:0/1045040129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 msgr2=0x557a212fbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8099780 tx=0x7fbbe8062970 comp rx=0 tx=0).stop 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 -- 172.21.15.129:0/1045040129 shutdown_connections 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a212fbf60 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a2132beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1045040129 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a2132b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 -- 172.21.15.129:0/1045040129 >> 172.21.15.129:0/1045040129 conn(0x557a20f4f3c0 msgr2=0x557a21301970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 -- 172.21.15.129:0/1045040129 shutdown_connections 2024-09-17T04:05:36.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.770+0000 7fbbfec99740 1 -- 172.21.15.129:0/1045040129 wait complete. 2024-09-17T04:05:36.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.771+0000 7fbbfec99740 1 -- start start 2024-09-17T04:05:36.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.771+0000 7fbbfec99740 1 Processor -- start 2024-09-17T04:05:36.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21310c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a2120b330 con 0x557a20f323b0 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557a21268810 con 0x557a20f3a590 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x557a20fd5620 con 0x557a20f5b270 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.772+0000 7fbbfec99740 10 client.0 _ll_register_callbacks cb 0x557a21210420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53286/0 (socket says 172.21.15.129:53286) 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 -- 172.21.15.129:0/3557109586 learned_addr learned my addr 172.21.15.129:0/3557109586 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21310c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 -- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 msgr2=0x557a21310c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21310c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 -- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 msgr2=0x557a21340560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 -- 172.21.15.129:0/3557109586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a20fc6850 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21310c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fbbe002aed0 tx=0x7fbbd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3e509dcb478cbe server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.773+0000 7fbbda000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0045040 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe004b470 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe8063020 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7fbbe0058050 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 10 client.11166 fetch_fsmap learned FSMap version 208 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=2+}) -- 0x557a2103d6b0 con 0x557a20f323b0 2024-09-17T04:05:36.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe8063020 con 0x557a20f323b0 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 208) ==== 3502+0+0 (secure 0 0 0) 0x7fbbe005b020 con 0x557a20f323b0 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 10 client.11166 fetch_fsmap finished waiting for FSMap version 208 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 10 client.11166 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 20 client.11166 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a20f19e40 0x557a21359930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 4 client.11166 mds_command: new command op to 11028 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.774+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 --> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a21277520 con 0x557a20f19e40 2024-09-17T04:05:36.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.775+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a20f19e40 0x557a21359930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.775+0000 7fbbda000640 10 client.11166 ms_handle_connect on v2:172.21.15.129:6833/711019236 2024-09-17T04:05:36.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.775+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a20f19e40 0x557a21359930 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ad50776e467c33a server_cookie=e085c6395b1e2fea in_seq=0 out_seq=0 2024-09-17T04:05:36.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbda000640 1 -- 172.21.15.129:0/3557109586 <== mds.0 v2:172.21.15.129:6833/711019236 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x557a21277520 con 0x557a20f19e40 2024-09-17T04:05:36.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbda000640 10 client.11166 handle_command_reply: tid=0 2024-09-17T04:05:36.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 client.11166 shutdown 2024-09-17T04:05:36.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 msgr2=0x557a21326100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 secure :-1 s=READY pgs=2401 cs=0 l=1 rev1=1 crypto rx=0x7fbbe002aed0 tx=0x7fbbd4005830 comp rx=0 tx=0).stop 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 shutdown_connections 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a20f19e40 0x557a21359930 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21310c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3557109586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21326100 unknown :-1 s=CLOSED pgs=2401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbda000640 0 client.11166 ms_handle_reset on v2:172.21.15.129:6833/711019236 2024-09-17T04:05:36.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.776+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 >> 172.21.15.129:0/3557109586 conn(0x557a20f4f3c0 msgr2=0x557a21346fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.777+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 shutdown_connections 2024-09-17T04:05:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.777+0000 7fbbfec99740 1 -- 172.21.15.129:0/3557109586 wait complete. 2024-09-17T04:05:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.777+0000 7fbbfec99740 20 client.11166 trim_cache size 0 max 16384 2024-09-17T04:05:36.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.796+0000 7fbbfec99740 1 Processor -- start 2024-09-17T04:05:36.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.796+0000 7fbbfec99740 1 -- start start 2024-09-17T04:05:36.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0200a50 con 0x557a20f323b0 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf015ce90 con 0x557a20f3a590 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf0200400 con 0x557a20f5b270 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbf5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbf5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53292/0 (socket says 172.21.15.129:53292) 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.797+0000 7fbbf5200640 1 -- 172.21.15.129:0/1959274150 learned_addr learned my addr 172.21.15.129:0/1959274150 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 -- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 msgr2=0x557a21341280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 -- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 msgr2=0x557a21340d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 -- 172.21.15.129:0/1959274150 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a21341940 con 0x557a20f323b0 2024-09-17T04:05:36.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbf5200640 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00093e0 tx=0x7fbbe00094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8c881abe489d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbe4c00640 1 -- 172.21.15.129:0/1959274150 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0060070 con 0x557a20f323b0 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbe4c00640 1 -- 172.21.15.129:0/1959274150 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe0050070 con 0x557a20f323b0 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.798+0000 7fbbe4c00640 1 -- 172.21.15.129:0/1959274150 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe004b470 con 0x557a20f323b0 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 -- 172.21.15.129:0/1959274150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 msgr2=0x557a21322620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00093e0 tx=0x7fbbe00094d0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 -- 172.21.15.129:0/1959274150 shutdown_connections 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 --2- 172.21.15.129:0/1959274150 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f323b0 0x557a21322620 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 -- 172.21.15.129:0/1959274150 >> 172.21.15.129:0/1959274150 conn(0x557a20f4f3c0 msgr2=0x557a21349d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 -- 172.21.15.129:0/1959274150 shutdown_connections 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.799+0000 7fbbfec99740 1 -- 172.21.15.129:0/1959274150 wait complete. 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.800+0000 7fbbfec99740 1 -- start start 2024-09-17T04:05:36.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.800+0000 7fbbfec99740 1 Processor -- start 2024-09-17T04:05:36.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a213ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x557a21311700 con 0x557a20f3a590 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x557a21325be0 con 0x557a20f5b270 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x557a213132d0 con 0x557a20f323b0 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbfec99740 10 client.0 _ll_register_callbacks cb 0x7fbbd4005150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbefe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53296/0 (socket says 172.21.15.129:53296) 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbefe00640 1 -- 172.21.15.129:0/3915070221 learned_addr learned my addr 172.21.15.129:0/3915070221 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbf5c00640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.802+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a213ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 -- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 msgr2=0x557a213ac6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a213ac6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 -- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 msgr2=0x557a21341280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 -- 172.21.15.129:0/3915070221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a21302650 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbefe00640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00ca420 tx=0x7fbbdc001b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d67a86f0b3ce42 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbda000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc013450 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fbbdc01c040 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0050070 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=208) ==== 24+0+0 (secure 0 0 0) 0x7fbbdc02b050 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbfec99740 10 client.11172 fetch_fsmap learned FSMap version 208 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.803+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=208,monmap=2+}) -- 0x557a213c7350 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbe0050070 con 0x557a20f3a590 2024-09-17T04:05:36.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 208) ==== 3502+0+0 (secure 0 0 0) 0x7fbbdc02e020 con 0x557a20f3a590 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 10 client.11172 fetch_fsmap finished waiting for FSMap version 208 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 10 client.11172 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 20 client.11172 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a2136a660 0x557a213cab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 4 client.11172 mds_command: new command op to 11028 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 --> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] -- command(tid 0: {"prefix": "status"}) -- 0x557a212fe4f0 con 0x557a2136a660 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a2136a660 0x557a213cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbda000640 10 client.11172 ms_handle_connect on v2:172.21.15.129:6833/711019236 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.804+0000 7fbbf5200640 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a2136a660 0x557a213cab00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79da2a9ddce0af5 server_cookie=c2eb0c161f2418c9 in_seq=0 out_seq=0 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbda000640 1 -- 172.21.15.129:0/3915070221 <== mds.0 v2:172.21.15.129:6833/711019236 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x557a212fe4f0 con 0x557a2136a660 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbda000640 10 client.11172 handle_command_reply: tid=0 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 client.11172 shutdown 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 msgr2=0x557a21340d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7fbbf00ca420 tx=0x7fbbdc001b60 comp rx=0 tx=0).stop 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 shutdown_connections 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:6833/711019236,v1:172.21.15.129:6835/711019236] conn(0x557a2136a660 0x557a213cab00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557a20f5b270 0x557a21341280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x557a20f3a590 0x557a21340d40 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbda000640 0 client.11172 ms_handle_reset on v2:172.21.15.129:6833/711019236 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 --2- 172.21.15.129:0/3915070221 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x557a20f323b0 0x557a213ac6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 >> 172.21.15.129:0/3915070221 conn(0x557a20f4f3c0 msgr2=0x557a21349d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 shutdown_connections 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 1 -- 172.21.15.129:0/3915070221 wait complete. 2024-09-17T04:05:36.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.805+0000 7fbbfec99740 20 client.11172 trim_cache size 0 max 16384 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 11028, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.590694238999999, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 208, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 218, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 218, 2024-09-17T04:05:36.810 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.662299054000002 2024-09-17T04:05:36.811 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:05:36.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 msgr2=0x7fbbd007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 0x7fbbd007b0d0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0035000 tx=0x7fbbe0004a40 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 msgr2=0x7fbbf00743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0160420 tx=0x7fbbe80039c0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 shutdown_connections 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbbd0078bc0 0x7fbbd007b0d0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbf00c6310 0x7fbbf00743f0 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbf00c5f00 0x7fbbf0073eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 --2- 172.21.15.129:0/3417828502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbd4003a30 0x7fbbf015ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 >> 172.21.15.129:0/3417828502 conn(0x7fbbf006d410 msgr2=0x7fbbf006f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 shutdown_connections 2024-09-17T04:05:36.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:36.806+0000 7fbbf7c00640 1 -- 172.21.15.129:0/3417828502 wait complete. 2024-09-17T04:05:36.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:05:37.051 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 -- 172.21.15.129:0/1477653414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 msgr2=0x7f77b006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 --2- 172.21.15.129:0/1477653414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b006eed0 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f77a0005fd0 tx=0x7f77a002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 -- 172.21.15.129:0/1477653414 shutdown_connections 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 --2- 172.21.15.129:0/1477653414 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 0x7f77b00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 --2- 172.21.15.129:0/1477653414 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b006eed0 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 -- 172.21.15.129:0/1477653414 >> 172.21.15.129:0/1477653414 conn(0x7f77b006d410 msgr2=0x7f77b00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 -- 172.21.15.129:0/1477653414 shutdown_connections 2024-09-17T04:05:37.052 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.051+0000 7f77b7600640 1 -- 172.21.15.129:0/1477653414 wait complete. 2024-09-17T04:05:37.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.053+0000 7f77b7600640 1 Processor -- start 2024-09-17T04:05:37.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.053+0000 7f77b7600640 1 -- start start 2024-09-17T04:05:37.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 0x7f77b01e60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:37.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 0x7f77b015f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f77b0160ba0 con 0x7f77b0099570 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f77b0160a20 con 0x7f77b006f410 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b7600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f77b0160d20 con 0x7f77b00cf9c0 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53310/0 (socket says 172.21.15.129:53310) 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77a7e00640 1 -- 172.21.15.129:0/1431088001 learned_addr learned my addr 172.21.15.129:0/1431088001 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b4c00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 0x7f77b01e60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.055+0000 7f77b5600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 0x7f77b015f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 -- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 msgr2=0x7f77b015f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 0x7f77b015f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 -- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 msgr2=0x7f77b01e60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:37.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 0x7f77b01e60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 -- 172.21.15.129:0/1431088001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77b00990e0 con 0x7f77b0099570 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77b4c00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 0x7f77b01e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77a7e00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f779800c3a0 tx=0x7f779800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798017070 con 0x7f77b0099570 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f77980053c0 con 0x7f77b0099570 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77b014d630 con 0x7f77b0099570 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7798003850 con 0x7f77b0099570 2024-09-17T04:05:37.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.056+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77b014e260 con 0x7f77b0099570 2024-09-17T04:05:37.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.057+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77b014d3a0 con 0x7f77b0099570 2024-09-17T04:05:37.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.059+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f779801a070 con 0x7f77b0099570 2024-09-17T04:05:37.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.060+0000 7f77ad600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 0x7f778c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:37.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.060+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f779809a070 con 0x7f77b0099570 2024-09-17T04:05:37.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.061+0000 7f77b4c00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 0x7f778c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:37.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.061+0000 7f77b4c00640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 0x7f778c07b0a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f77a0033000 tx=0x7f77a0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:37.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.065+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7798061e50 con 0x7f77b0099570 2024-09-17T04:05:37.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.285+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f77b01611d0 con 0x7f77b0099570 2024-09-17T04:05:37.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.286+0000 7f77ad600640 1 -- 172.21.15.129:0/1431088001 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v208) ==== 53+0+73 (secure 0 0 0) 0x7f7798066880 con 0x7f77b0099570 2024-09-17T04:05:37.286 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:05:37.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 msgr2=0x7f778c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:37.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 0x7f778c07b0a0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f77a0033000 tx=0x7f77a0004730 comp rx=0 tx=0).stop 2024-09-17T04:05:37.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 msgr2=0x7f77b0098ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7f779800c3a0 tx=0x7f779800c860 comp rx=0 tx=0).stop 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 shutdown_connections 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f778c078b90 0x7f778c07b0a0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f77b00cf9c0 0x7f77b015f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f77b0099570 0x7f77b0098ba0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 --2- 172.21.15.129:0/1431088001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f77b006f410 0x7f77b01e60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:37.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 >> 172.21.15.129:0/1431088001 conn(0x7f77b006d410 msgr2=0x7f77b00b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:37.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 shutdown_connections 2024-09-17T04:05:37.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37.288+0000 7f77b7600640 1 -- 172.21.15.129:0/1431088001 wait complete. 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:05:37.311 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:05:37.312 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:05:37.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.312 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:05:37.348 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:05:37.349 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41755sec preferred_lft 41755sec 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.350 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:05:37.351 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:05:37.352 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:05:37.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.352 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:05:37.444 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:05:37.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.447 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:05:37.483 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:05:37.483 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:05:37.483 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:05:37.483 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:05:37.483 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:05:37.484 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:05:37.484 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:05:37.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:05:37.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:05:37.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:05:37.485 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:05:37.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:05:37.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:05:37.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.681 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:05:37.716 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:05:37.716 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:05:37.716 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:05:37.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:05:37.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:05:37.813 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:05:37.848 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:05:37.848 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:05:37.848 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:05:37.923 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:05:37.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:37.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:05:38.117 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:05:38.118 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:05:38.118 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:05:38.138 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:05:38.138 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:05:38.138 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:05:38.138 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:05:38.138 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:05:38.168 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:05:38.168 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:05:38.168 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:05:38.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:05:38.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:05:38.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:05:38.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:05:38.169 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:05:38.170 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:05:38.170 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:05:38.170 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:05:38.170 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:05:38.170 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:05:38.173 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:05:38.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:38.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:05:38.230 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:05:38.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:05:38.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:05:38.329 INFO:teuthology.orchestra.run.smithi129.stdout:client11181 2024-09-17T04:05:38.329 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:38.329 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:38.366 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:05:38.366 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:38.366 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:05:38.366 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:38.366 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:05:38.366 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:38.367 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:05:38.367 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:38.392 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:05:38.392 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:38.415 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:05:38.415 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:38.431 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:05:38.432 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:38.436 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:05:38.436 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:38.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-17T04:05:38.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T04:05:38.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T04:05:38.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 -- 172.21.15.129:0/3160024329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc80cf420 msgr2=0x7fdbc80cf830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 --2- 172.21.15.129:0/3160024329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc80cf420 0x7fdbc80cf830 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fdbc806f7a0 tx=0x7fdbc002a8b0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 -- 172.21.15.129:0/3160024329 shutdown_connections 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 --2- 172.21.15.129:0/3160024329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc80cf420 0x7fdbc80cf830 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 --2- 172.21.15.129:0/3160024329 >> v2:172.21.15.129:3300/0 conn(0x7fdbc80d4c50 0x7fdbc80ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7fdbcca00640 1 -- 172.21.15.129:0/3160024329 >> 172.21.15.129:0/3160024329 conn(0x7fdbc809d8d0 msgr2=0x7fdbc808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7faa0b200640 1 -- 172.21.15.129:0/3688121013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 msgr2=0x7faa040d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.713+0000 7faa0b200640 1 --2- 172.21.15.129:0/3688121013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa040d0320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4009830 tx=0x7fa9f402a510 comp rx=0 tx=0).stop 2024-09-17T04:05:38.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 -- 172.21.15.129:0/3688121013 shutdown_connections 2024-09-17T04:05:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7fdbcca00640 1 -- 172.21.15.129:0/3160024329 shutdown_connections 2024-09-17T04:05:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 --2- 172.21.15.129:0/3688121013 >> v2:172.21.15.181:3300/0 conn(0x7faa040d0860 0x7faa0415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 --2- 172.21.15.129:0/3688121013 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa040d0320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.716 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 -- 172.21.15.129:0/3688121013 >> 172.21.15.129:0/3688121013 conn(0x7faa0406cfc0 msgr2=0x7faa0406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 -- 172.21.15.129:0/1076387136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa80d12a0 msgr2=0x7f8fa8166540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 --2- 172.21.15.129:0/1076387136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa80d12a0 0x7f8fa8166540 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80d2020 tx=0x7f8f9402a4a0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 -- 172.21.15.129:0/3688121013 shutdown_connections 2024-09-17T04:05:38.717 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7fdbcca00640 1 -- 172.21.15.129:0/3160024329 wait complete. 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7faa0b200640 1 -- 172.21.15.129:0/3688121013 wait complete. 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 -- 172.21.15.129:0/1076387136 shutdown_connections 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 --2- 172.21.15.129:0/1076387136 >> v2:172.21.15.129:3300/0 conn(0x7f8fa8166a80 0x7f8fa8168ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 --2- 172.21.15.129:0/1076387136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa80d12a0 0x7f8fa8166540 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 -- 172.21.15.129:0/1076387136 >> 172.21.15.129:0/1076387136 conn(0x7f8fa809d790 msgr2=0x7f8fa80d44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:38.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 -- 172.21.15.129:0/1076387136 shutdown_connections 2024-09-17T04:05:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.714+0000 7f8faca00640 1 -- 172.21.15.129:0/1076387136 wait complete. 2024-09-17T04:05:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.716+0000 7faa0b200640 1 Processor -- start 2024-09-17T04:05:38.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.716+0000 7faa0b200640 1 -- start start 2024-09-17T04:05:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.716+0000 7f8faca00640 1 Processor -- start 2024-09-17T04:05:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.716+0000 7f8faca00640 1 -- start start 2024-09-17T04:05:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.717+0000 7fdbcca00640 1 Processor -- start 2024-09-17T04:05:38.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.717+0000 7fdbcca00640 1 -- start start 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 0x7faa040cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa041f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa0415d3e0 con 0x7faa040ccdf0 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa0415d260 con 0x7faa040d0860 2024-09-17T04:05:38.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faa0415d560 con 0x7faa04160c30 2024-09-17T04:05:38.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 0x7f8fa815f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa8166a80 0x7f8fa815fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa80d1830 con 0x7f8fa80b32e0 2024-09-17T04:05:38.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa80d16b0 con 0x7f8fa8166a80 2024-09-17T04:05:38.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8faca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8fa80d0330 con 0x7f8fa80d12a0 2024-09-17T04:05:38.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa09e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa09e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53336/0 (socket says 172.21.15.129:53336) 2024-09-17T04:05:38.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa09e00640 1 -- 172.21.15.129:0/70652381 learned_addr learned my addr 172.21.15.129:0/70652381 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8fa7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53346/0 (socket says 172.21.15.129:53346) 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8fa7400640 1 -- 172.21.15.129:0/2563749964 learned_addr learned my addr 172.21.15.129:0/2563749964 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8fa6a00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 0x7f8fa815f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7faa0a800640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa041f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 0x7faa040cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc80b3220 0x7fdbc80ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.718+0000 7f8fa7e00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa8166a80 0x7f8fa815fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc81f5e80 0x7fdbc815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc80d0ae0 con 0x7fdbc80cf420 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc80ba0d0 con 0x7fdbc81f5e80 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbcca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc80d0c60 con 0x7fdbc80b3220 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 -- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 msgr2=0x7faa041f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa041f6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 -- 172.21.15.129:0/70652381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 msgr2=0x7faa040c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09400640 1 -- 172.21.15.129:0/70652381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa041e7f00 con 0x7faa040d0860 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbc7400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc80b3220 0x7fdbc80ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbc6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbc6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53354/0 (socket says 172.21.15.129:53354) 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbc6a00640 1 -- 172.21.15.129:0/1984508650 learned_addr learned my addr 172.21.15.129:0/1984508650 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7fdbc7e00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc81f5e80 0x7fdbc815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa6a00640 1 -- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa8166a80 msgr2=0x7f8fa815fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa6a00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa8166a80 0x7f8fa815fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa6a00640 1 -- 172.21.15.129:0/2563749964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 msgr2=0x7f8fa8157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa6a00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa6a00640 1 -- 172.21.15.129:0/2563749964 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa80b7400 con 0x7f8fa80d12a0 2024-09-17T04:05:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa09e00640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7faa0a800640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa041f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.719+0000 7f8fa7400640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:38.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 -- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc80b3220 msgr2=0x7fdbc80ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc80b3220 0x7fdbc80ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7faa09400640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 0x7faa040cb790 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4031000 tx=0x7fa9f4033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 -- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc81f5e80 msgr2=0x7fdbc815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc81f5e80 0x7fdbc815e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 -- 172.21.15.129:0/1984508650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc815c970 con 0x7fdbc80cf420 2024-09-17T04:05:38.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7f8fa6a00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 0x7f8fa815f870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80d5120 tx=0x7f8f940052f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fdbc6a00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d750 tx=0x7fdbb400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94005c30 con 0x7f8fa80d12a0 2024-09-17T04:05:38.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f94004340 con 0x7f8fa80d12a0 2024-09-17T04:05:38.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f94031bb0 con 0x7f8fa80d12a0 2024-09-17T04:05:38.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4005c90 con 0x7faa040d0860 2024-09-17T04:05:38.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9f4031e50 con 0x7faa040d0860 2024-09-17T04:05:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4005280 con 0x7faa040d0860 2024-09-17T04:05:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa80b76e0 con 0x7f8fa80d12a0 2024-09-17T04:05:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4017070 con 0x7fdbc80cf420 2024-09-17T04:05:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdbb4004310 con 0x7fdbc80cf420 2024-09-17T04:05:38.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4009c30 con 0x7fdbc80cf420 2024-09-17T04:05:38.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.720+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa041e81e0 con 0x7faa040d0860 2024-09-17T04:05:38.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa041e87a0 con 0x7faa040d0860 2024-09-17T04:05:38.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa80b7ca0 con 0x7f8fa80d12a0 2024-09-17T04:05:38.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc80ce620 con 0x7fdbc80cf420 2024-09-17T04:05:38.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.721+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc815cef0 con 0x7fdbc80cf420 2024-09-17T04:05:38.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.722+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa0415eca0 con 0x7faa040d0860 2024-09-17T04:05:38.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.722+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa80b9fb0 con 0x7f8fa80d12a0 2024-09-17T04:05:38.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.723+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc80ce850 con 0x7fdbc80cf420 2024-09-17T04:05:38.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.724+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa9f4005420 con 0x7faa040d0860 2024-09-17T04:05:38.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.724+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8f940047a0 con 0x7f8fa80d12a0 2024-09-17T04:05:38.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.724+0000 7fa9fe000640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 0x7fa9e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.724+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdbb4005020 con 0x7fdbc80cf420 2024-09-17T04:05:38.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fa9f40bbef0 con 0x7faa040d0860 2024-09-17T04:05:38.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7faa09e00640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 0x7fa9e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7f8f9f400640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 0x7f8f8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7f8f940bbe80 con 0x7f8fa80d12a0 2024-09-17T04:05:38.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7fe189c00640 1 -- 172.21.15.129:0/1637663023 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe18409ba70 msgr2=0x7fe1840d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7fe189c00640 1 --2- 172.21.15.129:0/1637663023 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe18409ba70 0x7fe1840d01f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe170009860 tx=0x7fe17002a580 comp rx=0 tx=0).stop 2024-09-17T04:05:38.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.725+0000 7f8fa7400640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 0x7f8f8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fe189c00640 1 -- 172.21.15.129:0/1637663023 shutdown_connections 2024-09-17T04:05:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fe189c00640 1 --2- 172.21.15.129:0/1637663023 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe18409ba70 0x7fe1840d01f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fe189c00640 1 -- 172.21.15.129:0/1637663023 >> 172.21.15.129:0/1637663023 conn(0x7fe1840b3f00 msgr2=0x7fe1840b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fe189c00640 1 -- 172.21.15.129:0/1637663023 shutdown_connections 2024-09-17T04:05:38.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7faa09e00640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 0x7fa9e807b0a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8001c90 tx=0x7fa9f8007be0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fe189c00640 1 -- 172.21.15.129:0/1637663023 wait complete. 2024-09-17T04:05:38.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fdbbf400640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 0x7fdba807b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7f8fa7400640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 0x7f8f8807b0a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8f98007fd0 tx=0x7f8f98007f20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.726+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fdbb409b080 con 0x7fdbc80cf420 2024-09-17T04:05:38.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.727+0000 7fdbc7400640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 0x7fdba807b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.727+0000 7fdbc7400640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 0x7fdba807b0f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8001760 tx=0x7fdbb8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.728+0000 7fe189c00640 1 Processor -- start 2024-09-17T04:05:38.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.728+0000 7fe189c00640 1 -- start start 2024-09-17T04:05:38.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.728+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faa0415eca0 con 0x7faa040d0860 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 0x7fe184150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1840c5aa0 0x7fe18415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe18406e280 con 0x7fe1840ccfe0 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe18406e100 con 0x7fe18409ba70 2024-09-17T04:05:38.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe189c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe18406e400 con 0x7fe1840c5aa0 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe183400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe183400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53362/0 (socket says 172.21.15.129:53362) 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe183400640 1 -- 172.21.15.129:0/730915854 learned_addr learned my addr 172.21.15.129:0/730915854 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.730+0000 7fe182000640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1840c5aa0 0x7fe18415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe182a00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 0x7fe184150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8f9403f070 con 0x7f8fa80d12a0 2024-09-17T04:05:38.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdbb4020020 con 0x7fdbc80cf420 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 -- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1840c5aa0 msgr2=0x7fe18415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1840c5aa0 0x7fe18415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 -- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 msgr2=0x7fe184150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 0x7fe184150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 -- 172.21.15.129:0/730915854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe184151430 con 0x7fe1840ccfe0 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe182a00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 0x7fe184150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.731+0000 7fe183400640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c6c0 tx=0x7fe17c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.732+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c017070 con 0x7fe1840ccfe0 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.732+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe17c003c60 con 0x7fe1840ccfe0 2024-09-17T04:05:38.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.732+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17c0052d0 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.732+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe184159ef0 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.732+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18415a4b0 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.733+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1840d01f0 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.736+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe17c01a070 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.738+0000 7fe17aa00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 0x7fe16407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.738+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 5709+0+0 (secure 0 0 0) 0x7fe17c09a670 con 0x7fe1840ccfe0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.739+0000 7fe182a00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 0x7fe16407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.740+0000 7fe182a00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 0x7fe16407b0d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe174007a60 tx=0x7fe174009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:38.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.740+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe17c021020 con 0x7fe1840ccfe0 2024-09-17T04:05:38.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:38.999+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7fdbc80cf830 con 0x7fdbc80cf420 2024-09-17T04:05:39.000 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:05:39.000+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:05:39.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.001+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) -- 0x7faa041e8ca0 con 0x7faa040d0860 2024-09-17T04:05:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.002+0000 7fdbbf400640 1 -- 172.21.15.129:0/1984508650 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 11028 v209) ==== 96+0+0 (secure 0 0 0) 0x7fdbb400de60 con 0x7fdbc80cf420 2024-09-17T04:05:39.002 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 11028 2024-09-17T04:05:39.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.004+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7f8fa815bfb0 con 0x7f8fa80d12a0 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 msgr2=0x7fdba807b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 0x7fdba807b0f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8001760 tx=0x7fdbb8009040 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 msgr2=0x7fdbc81f5940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 secure :-1 s=READY pgs=2409 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d750 tx=0x7fdbb400dc10 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 shutdown_connections 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdba8078be0 0x7fdba807b0f0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc81f5e80 0x7fdbc815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdbc80cf420 0x7fdbc81f5940 unknown :-1 s=CLOSED pgs=2409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 --2- 172.21.15.129:0/1984508650 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdbc80b3220 0x7fdbc80ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 >> 172.21.15.129:0/1984508650 conn(0x7fdbc809d8d0 msgr2=0x7fdbc80b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.005+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 shutdown_connections 2024-09-17T04:05:39.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.006+0000 7fdbcca00640 1 -- 172.21.15.129:0/1984508650 wait complete. 2024-09-17T04:05:39.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.011+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7fe18406f6c0 con 0x7fe1840ccfe0 2024-09-17T04:05:39.015 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:05:39.015+0000 7f409c400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:05:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.027+0000 7fa9fe000640 1 -- 172.21.15.129:0/70652381 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 10237 v211) ==== 96+0+0 (secure 0 0 0) 0x7fa9f403f030 con 0x7faa040d0860 2024-09-17T04:05:39.027 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 10237 2024-09-17T04:05:39.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.028+0000 7fe17aa00640 1 -- 172.21.15.129:0/730915854 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 10231 v212) ==== 96+0+0 (secure 0 0 0) 0x7fe17c0624d0 con 0x7fe1840ccfe0 2024-09-17T04:05:39.028 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 10231 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 msgr2=0x7fa9e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 0x7fa9e807b0a0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8001c90 tx=0x7fa9f8007be0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 msgr2=0x7faa040cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 0x7faa040cb790 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4031000 tx=0x7fa9f4033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 shutdown_connections 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa9e8078b90 0x7fa9e807b0a0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa04160c30 0x7faa041f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa040d0860 0x7faa040cb790 secure :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4031000 tx=0x7fa9f4033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 --2- 172.21.15.129:0/70652381 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa040ccdf0 0x7faa040c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 >> 172.21.15.129:0/70652381 conn(0x7faa0406cfc0 msgr2=0x7faa04072d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 shutdown_connections 2024-09-17T04:05:39.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.030+0000 7faa0b200640 1 -- 172.21.15.129:0/70652381 wait complete. 2024-09-17T04:05:39.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 msgr2=0x7fe16407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 0x7fe16407b0d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe174007a60 tx=0x7fe174009040 comp rx=0 tx=0).stop 2024-09-17T04:05:39.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 msgr2=0x7fe1841524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00c6c0 tx=0x7fe17c00cb80 comp rx=0 tx=0).stop 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 shutdown_connections 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe164078bc0 0x7fe16407b0d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe1840ccfe0 0x7fe1841524a0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe1840c5aa0 0x7fe18415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 --2- 172.21.15.129:0/730915854 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe18409ba70 0x7fe184150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 >> 172.21.15.129:0/730915854 conn(0x7fe1840b3f00 msgr2=0x7fe18406e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 shutdown_connections 2024-09-17T04:05:39.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.032+0000 7fe189c00640 1 -- 172.21.15.129:0/730915854 wait complete. 2024-09-17T04:05:39.039 DEBUG:teuthology.parallel:result is '' 2024-09-17T04:05:39.057 DEBUG:teuthology.parallel:result is '' 2024-09-17T04:05:39.057 DEBUG:teuthology.parallel:result is '' 2024-09-17T04:05:39.281 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:05:39.281+0000 7f409c400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:05:39.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.288+0000 7f8f9f400640 1 -- 172.21.15.129:0/2563749964 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 8657 v213) ==== 95+0+0 (secure 0 0 0) 0x7f8f94031770 con 0x7f8fa80d12a0 2024-09-17T04:05:39.289 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 8657 2024-09-17T04:05:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 msgr2=0x7f8f8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 0x7f8f8807b0a0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f8f98007fd0 tx=0x7f8f98007f20 comp rx=0 tx=0).stop 2024-09-17T04:05:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 msgr2=0x7f8fa815f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 0x7f8fa815f870 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fa80d5120 tx=0x7f8f940052f0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 shutdown_connections 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8f88078b90 0x7f8f8807b0a0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8fa8166a80 0x7f8fa815fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8fa80d12a0 0x7f8fa815f870 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 --2- 172.21.15.129:0/2563749964 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8fa80b32e0 0x7f8fa8157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 >> 172.21.15.129:0/2563749964 conn(0x7f8fa809d790 msgr2=0x7f8fa806f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 shutdown_connections 2024-09-17T04:05:39.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.292+0000 7f8faca00640 1 -- 172.21.15.129:0/2563749964 wait complete. 2024-09-17T04:05:39.320 DEBUG:teuthology.parallel:result is '' 2024-09-17T04:05:39.321 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 500.00000000 2024-09-17T04:05:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.460+0000 7fd557343f40 1 Processor -- start 2024-09-17T04:05:39.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.460+0000 7fd557343f40 1 -- start start 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d7640 0x55da2f5ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f648150 0x55da2f646a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da2f4dcc60 con 0x55da2f5d9610 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55da2f53b1d0 con 0x55da2f648150 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd557343f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55da2f662b00 con 0x55da2f5d7640 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd547e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd547e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53380/0 (socket says 172.21.15.129:53380) 2024-09-17T04:05:39.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.461+0000 7fd547e00640 1 -- 172.21.15.129:0/224501755 learned_addr learned my addr 172.21.15.129:0/224501755 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd550400640 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d7640 0x55da2f5ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd550e00640 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f648150 0x55da2f646a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 -- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d7640 msgr2=0x55da2f5ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d7640 0x55da2f5ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 -- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f648150 msgr2=0x55da2f646a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f648150 0x55da2f646a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 -- 172.21.15.129:0/224501755 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2f5cfb60 con 0x55da2f5d9610 2024-09-17T04:05:39.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd547e00640 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd53800b520 tx=0x7fd53800b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea637fe72b39f055 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd54f000640 1 -- 172.21.15.129:0/224501755 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5380099f0 con 0x55da2f5d9610 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd54f000640 1 -- 172.21.15.129:0/224501755 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd538009b90 con 0x55da2f5d9610 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.462+0000 7fd54f000640 1 -- 172.21.15.129:0/224501755 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd538014440 con 0x55da2f5d9610 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 -- 172.21.15.129:0/224501755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 msgr2=0x55da2f663740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd53800b520 tx=0x7fd53800b9e0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 -- 172.21.15.129:0/224501755 shutdown_connections 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f648150 0x55da2f646a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f5d9610 0x55da2f663740 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 --2- 172.21.15.129:0/224501755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d7640 0x55da2f5ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 -- 172.21.15.129:0/224501755 >> 172.21.15.129:0/224501755 conn(0x55da2f4adac0 msgr2=0x55da2f661020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 -- 172.21.15.129:0/224501755 shutdown_connections 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.463+0000 7fd557343f40 1 -- 172.21.15.129:0/224501755 wait complete. 2024-09-17T04:05:39.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.464+0000 7fd557343f40 1 Processor -- start 2024-09-17T04:05:39.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.464+0000 7fd557343f40 1 -- start start 2024-09-17T04:05:39.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f5d7640 0x55da2f6447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d9610 0x55da2f644ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 0x55da2f6491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55da2f661730 con 0x55da2f648150 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55da2f6640b0 con 0x55da2f5d7640 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd557343f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55da2f5cf400 con 0x55da2f5d9610 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 0x55da2f6491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 0x55da2f6491c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53396/0 (socket says 172.21.15.129:53396) 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 -- 172.21.15.129:0/2451776631 learned_addr learned my addr 172.21.15.129:0/2451776631 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550400640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f5d7640 0x55da2f6447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd547e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d9610 0x55da2f644ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 -- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d9610 msgr2=0x55da2f644ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55da2f5d9610 0x55da2f644ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 -- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f5d7640 msgr2=0x55da2f6447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f5d7640 0x55da2f6447a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550e00640 1 -- 172.21.15.129:0/2451776631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da2f5d6cc0 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.466+0000 7fd550400640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55da2f5d7640 0x55da2f6447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd550e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 0x55da2f6491c0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fd54800a110 tx=0x7fd54800a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd54dc00640 1 -- 172.21.15.129:0/2451776631 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548015020 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55da2f663a90 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd54dc00640 1 -- 172.21.15.129:0/2451776631 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd54800ae00 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd54dc00640 1 -- 172.21.15.129:0/2451776631 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd548011890 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.467+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da2f58d7b0 con 0x55da2f648150 2024-09-17T04:05:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.469+0000 7fd54dc00640 1 -- 172.21.15.129:0/2451776631 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd548018020 con 0x55da2f648150 2024-09-17T04:05:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.470+0000 7fd54dc00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd52c077570 0x7fd52c079a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.470+0000 7fd54dc00640 1 -- 172.21.15.129:0/2451776631 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fd548099b80 con 0x55da2f648150 2024-09-17T04:05:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.471+0000 7fd557343f40 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x55da2f5ce710 0x55da2f649c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.471+0000 7fd550400640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd52c077570 0x7fd52c079a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.471+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:1 32.15 32:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e222) -- 0x55da2f58dcd0 con 0x55da2f5ce710 2024-09-17T04:05:39.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.471+0000 7fd547e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x55da2f5ce710 0x55da2f649c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.471+0000 7fd550400640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd52c077570 0x7fd52c079a80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd540055000 tx=0x7fd540058040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.472 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.472+0000 7fd547e00640 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x55da2f5ce710 0x55da2f649c10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd547e00640 1 -- 172.21.15.129:0/2451776631 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(1 500.00000000 [delete] v222'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7fd538009b50 con 0x55da2f5ce710 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x55da2f5ce710 msgr2=0x55da2f649c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x55da2f5ce710 0x55da2f649c10 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd52c077570 msgr2=0x7fd52c079a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd52c077570 0x7fd52c079a80 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd540055000 tx=0x7fd540058040 comp rx=0 tx=0).stop 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 msgr2=0x55da2f6491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 --2- 172.21.15.129:0/2451776631 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55da2f648150 0x55da2f6491c0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fd54800a110 tx=0x7fd54800a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd550e00640 1 -- 172.21.15.129:0/2451776631 reap_dead start 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 shutdown_connections 2024-09-17T04:05:39.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 >> 172.21.15.129:0/2451776631 conn(0x55da2f4adac0 msgr2=0x55da2f661020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 shutdown_connections 2024-09-17T04:05:39.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.477+0000 7fd557343f40 1 -- 172.21.15.129:0/2451776631 wait complete. 2024-09-17T04:05:39.493 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:05:39.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:05:39.493 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:05:39.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:05:39.494 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:05:39.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:05:39.494 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:05:39.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:05:39.499 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:05:39.499 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:39.499 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:05:39.500 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:39.501 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:05:39.501 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:39.502 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:05:39.502 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:39.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:39.695 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:05:39.696 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:05:39.697 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:05:39.700 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:05:39.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.740+0000 7fb105c00640 1 -- 172.21.15.129:0/1736068503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1000705a0 msgr2=0x7fb1000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 --2- 172.21.15.129:0/1736068503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1000705a0 0x7fb1000b4140 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0005fd0 tx=0x7fb0f002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 -- 172.21.15.129:0/1736068503 shutdown_connections 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 --2- 172.21.15.129:0/1736068503 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000cd390 0x7fb1001609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 --2- 172.21.15.129:0/1736068503 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1000705a0 0x7fb1000b4140 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 -- 172.21.15.129:0/1736068503 >> 172.21.15.129:0/1736068503 conn(0x7fb10006d410 msgr2=0x7fb1000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 -- 172.21.15.129:0/1736068503 shutdown_connections 2024-09-17T04:05:39.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.741+0000 7fb105c00640 1 -- 172.21.15.129:0/1736068503 wait complete. 2024-09-17T04:05:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.743+0000 7fb105c00640 1 Processor -- start 2024-09-17T04:05:39.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.743+0000 7fb105c00640 1 -- start start 2024-09-17T04:05:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 0x7fb100159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000bce10 0x7fb1001599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb100160f00 con 0x7fb10006eed0 2024-09-17T04:05:39.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb10006f4f0 con 0x7fb1000705a0 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.744+0000 7fb105c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb100161080 con 0x7fb1000bce10 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53460/0 (socket says 172.21.15.129:53460) 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fe000640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 0x7fb100159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0ff400640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000bce10 0x7fb1001599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 -- 172.21.15.129:0/585676256 learned_addr learned my addr 172.21.15.129:0/585676256 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 -- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000bce10 msgr2=0x7fb1001599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000bce10 0x7fb1001599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 -- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 msgr2=0x7fb100159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 0x7fb100159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fea00640 1 -- 172.21.15.129:0/585676256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb100156600 con 0x7fb10006eed0 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.745+0000 7fb0fe000640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 0x7fb100159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:39.747 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb0fea00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0034040 tx=0x7fb0f00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f003e030 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0f0003ca0 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f0035040 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb100156ab0 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.746+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb100157070 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.747+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1000b4140 con 0x7fb10006eed0 2024-09-17T04:05:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.749+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0f0041030 con 0x7fb10006eed0 2024-09-17T04:05:39.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.749+0000 7fb0f6a00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 0x7fb0e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:39.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.750+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fb0f00bb310 con 0x7fb10006eed0 2024-09-17T04:05:39.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.750+0000 7fb0fe000640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 0x7fb0e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:39.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.751+0000 7fb0fe000640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 0x7fb0e007b0a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb100158ed0 tx=0x7fb0ec009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:39.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.754+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0f0083080 con 0x7fb10006eed0 2024-09-17T04:05:39.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.971+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb10006a140 con 0x7fb10006eed0 2024-09-17T04:05:39.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.972+0000 7fb0f6a00640 1 -- 172.21.15.129:0/585676256 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) ==== 95+0+1916 (secure 0 0 0) 0x7fb0f0087ab0 con 0x7fb10006eed0 2024-09-17T04:05:39.972 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:39.973 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":213,"btime":"2024-09-17T04:05:39:282281+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:39.282052+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":16}]} 2024-09-17T04:05:39.973 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 213 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 msgr2=0x7fb0e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 0x7fb0e007b0a0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fb100158ed0 tx=0x7fb0ec009040 comp rx=0 tx=0).stop 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 msgr2=0x7fb1000bc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0034040 tx=0x7fb0f00038a0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 shutdown_connections 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0e0078b90 0x7fb0e007b0a0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb1000bce10 0x7fb1001599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1000705a0 0x7fb100159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 --2- 172.21.15.129:0/585676256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb10006eed0 0x7fb1000bc8d0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 >> 172.21.15.129:0/585676256 conn(0x7fb10006d410 msgr2=0x7fb1000c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 shutdown_connections 2024-09-17T04:05:39.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:39.975+0000 7fb105c00640 1 -- 172.21.15.129:0/585676256 wait complete. 2024-09-17T04:05:40.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 213, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:05:24.702816+0000', 'modified': '2024-09-17T04:05:39.282052+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 222, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:05:40.400 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-09-17T04:05:41.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 -- 172.21.15.129:0/2321368990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 msgr2=0x7f533c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 --2- 172.21.15.129:0/2321368990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c15bca0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f533c0727e0 tx=0x7f532802a530 comp rx=0 tx=0).stop 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 -- 172.21.15.129:0/2321368990 shutdown_connections 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 --2- 172.21.15.129:0/2321368990 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c15bca0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 --2- 172.21.15.129:0/2321368990 >> v2:172.21.15.181:3300/0 conn(0x7f533c1524c0 0x7f533c0cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 -- 172.21.15.129:0/2321368990 >> 172.21.15.129:0/2321368990 conn(0x7f533c06cfc0 msgr2=0x7f533c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 -- 172.21.15.129:0/2321368990 shutdown_connections 2024-09-17T04:05:41.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.630+0000 7f5343000640 1 -- 172.21.15.129:0/2321368990 wait complete. 2024-09-17T04:05:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.633+0000 7f5343000640 1 Processor -- start 2024-09-17T04:05:41.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.633+0000 7f5343000640 1 -- start start 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 0x7f533c0ab0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f533c15c1e0 0x7f533c0ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f533c0d02c0 con 0x7f533c0cfb00 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f533c152ae0 con 0x7f533c1524c0 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.634+0000 7f5343000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f533c0d0440 con 0x7f533c15c1e0 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:41.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53486/0 (socket says 172.21.15.129:53486) 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 -- 172.21.15.129:0/763844698 learned_addr learned my addr 172.21.15.129:0/763844698 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341200640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 0x7f533c0ab0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5342600640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f533c15c1e0 0x7f533c0ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 -- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f533c15c1e0 msgr2=0x7f533c0ab610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f533c15c1e0 0x7f533c0ab610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 -- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 msgr2=0x7f533c0ab0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 0x7f533c0ab0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 -- 172.21.15.129:0/763844698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533c14c6f0 con 0x7f533c0cfb00 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341200640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 0x7f533c0ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.635+0000 7f5341c00640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f533000e790 tx=0x7f533000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.636+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533000eea0 con 0x7f533c0cfb00 2024-09-17T04:05:41.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.636+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f533000cdb0 con 0x7f533c0cfb00 2024-09-17T04:05:41.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.636+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533c0aa440 con 0x7f533c0cfb00 2024-09-17T04:05:41.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.636+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330010690 con 0x7f533c0cfb00 2024-09-17T04:05:41.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.636+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f533c15b360 con 0x7f533c0cfb00 2024-09-17T04:05:41.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.637+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f533c0d0960 con 0x7f533c0cfb00 2024-09-17T04:05:41.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.638+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5330010830 con 0x7f533c0cfb00 2024-09-17T04:05:41.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.639+0000 7f5339600640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 0x7f531c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:41.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.639+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f533009b8d0 con 0x7f533c0cfb00 2024-09-17T04:05:41.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.639+0000 7f5341200640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 0x7f531c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:41.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.640+0000 7f5341200640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 0x7f531c07b0a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5328034000 tx=0x7f5328004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:41.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.643+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5330063c00 con 0x7f533c0cfb00 2024-09-17T04:05:41.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.864+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f533c06a140 con 0x7f533c0cfb00 2024-09-17T04:05:41.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.865+0000 7f5339600640 1 -- 172.21.15.129:0/763844698 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+5248 (secure 0 0 0) 0x7f5330063500 con 0x7f533c0cfb00 2024-09-17T04:05:41.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:41.866 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":216,"btime":"2024-09-17T04:05:41:294908+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:40.306642+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":16}]} 2024-09-17T04:05:41.866 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 216 2024-09-17T04:05:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 msgr2=0x7f531c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 0x7f531c07b0a0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f5328034000 tx=0x7f5328004730 comp rx=0 tx=0).stop 2024-09-17T04:05:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 msgr2=0x7f533c14c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:41.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f533000e790 tx=0x7f533000ec50 comp rx=0 tx=0).stop 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 shutdown_connections 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f531c078b90 0x7f531c07b0a0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f533c15c1e0 0x7f533c0ab610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f533c1524c0 0x7f533c0ab0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 --2- 172.21.15.129:0/763844698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f533c0cfb00 0x7f533c14c1b0 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.867+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 >> 172.21.15.129:0/763844698 conn(0x7f533c06cfc0 msgr2=0x7f533c151b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.868+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 shutdown_connections 2024-09-17T04:05:41.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:41.868+0000 7f5343000640 1 -- 172.21.15.129:0/763844698 wait complete. 2024-09-17T04:05:42.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 216, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:05:24.702816+0000', 'modified': '2024-09-17T04:05:40.306642+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 222, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11253': {'gid': 11253, 'name': 'c', 'rank': 0, 'incarnation': 215, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.129:6835/3706217946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3706217946}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3706217946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:05:42.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11253:up:reconnect 2024-09-17T04:05:43.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:43.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 -- 172.21.15.129:0/2274646812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb84c004800 msgr2=0x7fb84c004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2274646812 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb84c004800 0x7fb84c004c70 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7fb850001f00 tx=0x7fb84c005830 comp rx=0 tx=0).stop 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb867e00640 1 -- 172.21.15.129:0/2274646812 reap_dead start 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 -- 172.21.15.129:0/2274646812 shutdown_connections 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 -- 172.21.15.129:0/2274646812 >> 172.21.15.129:0/2274646812 conn(0x7fb868069150 msgr2=0x7fb868069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 -- 172.21.15.129:0/2274646812 shutdown_connections 2024-09-17T04:05:43.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.533+0000 7fb86ca00640 1 -- 172.21.15.129:0/2274646812 wait complete. 2024-09-17T04:05:43.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.535+0000 7fb86ca00640 1 Processor -- start 2024-09-17T04:05:43.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.535+0000 7fb86ca00640 1 -- start start 2024-09-17T04:05:43.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 0x7fb868159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8680b33f0 0x7fb8681e7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb868160840 con 0x7fb8680b64c0 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8681606c0 con 0x7fb8680b33f0 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb86ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8681609c0 con 0x7fb84c004800 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb867e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb867e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53498/0 (socket says 172.21.15.129:53498) 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb867e00640 1 -- 172.21.15.129:0/2087199322 learned_addr learned my addr 172.21.15.129:0/2087199322 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb867400640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 0x7fb868159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.537+0000 7fb85ea00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8680b33f0 0x7fb8681e7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 -- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 msgr2=0x7fb868159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:43.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 0x7fb868159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 -- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8680b33f0 msgr2=0x7fb8681e7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8680b33f0 0x7fb8681e7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 -- 172.21.15.129:0/2087199322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8680b3b40 con 0x7fb8680b64c0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867400640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 0x7fb868159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb867e00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fb8680ced00 tx=0x7fb860011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860004230 con 0x7fb8680b64c0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb860016070 con 0x7fb8680b64c0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8680b3e20 con 0x7fb8680b64c0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86000fa90 con 0x7fb8680b64c0 2024-09-17T04:05:43.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.538+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8680b2040 con 0x7fb8680b64c0 2024-09-17T04:05:43.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.540+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb86815d660 con 0x7fb8680b64c0 2024-09-17T04:05:43.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.541+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8600035f0 con 0x7fb8680b64c0 2024-09-17T04:05:43.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.542+0000 7fb85fe00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 0x7fb84407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.542+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fb86009bac0 con 0x7fb8680b64c0 2024-09-17T04:05:43.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.542+0000 7fb867400640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 0x7fb84407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:43.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.543+0000 7fb867400640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 0x7fb84407b0a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb8580018e0 tx=0x7fb858001cc0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:43.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.546+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb860063d60 con 0x7fb8680b64c0 2024-09-17T04:05:43.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.764+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb868075f30 con 0x7fb8680b64c0 2024-09-17T04:05:43.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.765+0000 7fb85fe00640 1 -- 172.21.15.129:0/2087199322 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) ==== 95+0+5251 (secure 0 0 0) 0x7fb860063660 con 0x7fb8680b64c0 2024-09-17T04:05:43.766 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:43.766 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":218,"btime":"2024-09-17T04:05:43:308452+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:42.310950+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:clientreplay","state_seq":4,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":16}]} 2024-09-17T04:05:43.766 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 218 2024-09-17T04:05:43.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.767+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 msgr2=0x7fb84407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:43.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.767+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 0x7fb84407b0a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb8580018e0 tx=0x7fb858001cc0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.767+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 msgr2=0x7fb8680b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:43.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.767+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fb8680ced00 tx=0x7fb860011b10 comp rx=0 tx=0).stop 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 shutdown_connections 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb844078b90 0x7fb84407b0a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8680b64c0 0x7fb8680b2840 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8680b33f0 0x7fb8681e7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 --2- 172.21.15.129:0/2087199322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb84c004800 0x7fb868159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 >> 172.21.15.129:0/2087199322 conn(0x7fb868069150 msgr2=0x7fb86815e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 shutdown_connections 2024-09-17T04:05:43.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:43.768+0000 7fb86ca00640 1 -- 172.21.15.129:0/2087199322 wait complete. 2024-09-17T04:05:44.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 218, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:05:24.702816+0000', 'modified': '2024-09-17T04:05:42.310950+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 222, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11253': {'gid': 11253, 'name': 'c', 'rank': 0, 'incarnation': 215, 'state': 'up:clientreplay', 'state_seq': 4, 'addr': '172.21.15.129:6835/3706217946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3706217946}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3706217946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:05:44.197 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11253:up:clientreplay 2024-09-17T04:05:45.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 -- 172.21.15.129:0/1574924480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff44406f410 msgr2=0x7ff4440cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 --2- 172.21.15.129:0/1574924480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff44406f410 0x7ff4440cf2b0 secure :-1 s=READY pgs=2431 cs=0 l=1 rev1=1 crypto rx=0x7ff430009830 tx=0x7ff43002a480 comp rx=0 tx=0).stop 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 -- 172.21.15.129:0/1574924480 shutdown_connections 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 --2- 172.21.15.129:0/1574924480 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff44406f410 0x7ff4440cf2b0 unknown :-1 s=CLOSED pgs=2431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 --2- 172.21.15.129:0/1574924480 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff44406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 -- 172.21.15.129:0/1574924480 >> 172.21.15.129:0/1574924480 conn(0x7ff44406d410 msgr2=0x7ff4440b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 -- 172.21.15.129:0/1574924480 shutdown_connections 2024-09-17T04:05:45.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.430+0000 7ff449400640 1 -- 172.21.15.129:0/1574924480 wait complete. 2024-09-17T04:05:45.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.432+0000 7ff449400640 1 Processor -- start 2024-09-17T04:05:45.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.433+0000 7ff449400640 1 -- start start 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406f410 0x7ff4440c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff4441f5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff444160ba0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff444160a20 con 0x7ff44406f410 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff449400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff444160d20 con 0x7ff444099570 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff443200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.434+0000 7ff443200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39582/0 (socket says 172.21.15.129:39582) 2024-09-17T04:05:45.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 -- 172.21.15.129:0/1260985103 learned_addr learned my addr 172.21.15.129:0/1260985103 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff442800640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406f410 0x7ff4440c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff441e00640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff4441f5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 -- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 msgr2=0x7ff4441f5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff4441f5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 -- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406f410 msgr2=0x7ff4440c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406f410 0x7ff4440c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 -- 172.21.15.129:0/1260985103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44409a0d0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff441e00640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff4441f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.435+0000 7ff443200640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c730 tx=0x7ff43c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.436+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c017070 con 0x7ff4440cf7f0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.436+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff43c003c60 con 0x7ff4440cf7f0 2024-09-17T04:05:45.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.436+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c0052d0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.436+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44409a390 con 0x7ff4440cf7f0 2024-09-17T04:05:45.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.436+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4441505b0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.437+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4441611d0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.438+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff43c01a070 con 0x7ff4440cf7f0 2024-09-17T04:05:45.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.439+0000 7ff43aa00640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 0x7ff42407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:45.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.439+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7ff43c09a4f0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.440+0000 7ff442800640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 0x7ff42407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:45.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.440+0000 7ff442800640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 0x7ff42407b0a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff43400bb50 tx=0x7ff43400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:45.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.444+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff43c0621d0 con 0x7ff4440cf7f0 2024-09-17T04:05:45.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.661+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff44406a140 con 0x7ff4440cf7f0 2024-09-17T04:05:45.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.662+0000 7ff43aa00640 1 -- 172.21.15.129:0/1260985103 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5254 (secure 0 0 0) 0x7ff43c066c00 con 0x7ff4440cf7f0 2024-09-17T04:05:45.662 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:45.663 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":219,"btime":"2024-09-17T04:05:44:314407+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:44.314397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11253,"qdb_cluster":[11253]},"id":16}]} 2024-09-17T04:05:45.663 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 219 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.664+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 msgr2=0x7ff42407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.664+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 0x7ff42407b0a0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff43400bb50 tx=0x7ff43400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.664+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 msgr2=0x7ff444150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.664+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00c730 tx=0x7ff43c00cbf0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 shutdown_connections 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff424078b90 0x7ff42407b0a0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4440cf7f0 0x7ff444150ad0 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff444099570 0x7ff4441f5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 --2- 172.21.15.129:0/1260985103 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406f410 0x7ff4440c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 >> 172.21.15.129:0/1260985103 conn(0x7ff44406d410 msgr2=0x7ff444073030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 shutdown_connections 2024-09-17T04:05:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:45.665+0000 7ff449400640 1 -- 172.21.15.129:0/1260985103 wait complete. 2024-09-17T04:05:46.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 219, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:05:24.702816+0000', 'modified': '2024-09-17T04:05:44.314397+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 222, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11253}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11253': {'gid': 11253, 'name': 'c', 'rank': 0, 'incarnation': 215, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.129:6835/3706217946', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6834', 'nonce': 3706217946}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3706217946}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11253, 'qdb_cluster': [11253]} 2024-09-17T04:05:46.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:05:46.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:05:46.298 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T04:05:46.298+0000 7f1c5c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:05:46.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.315+0000 7f54ad000640 1 -- 172.21.15.129:0/1393555119 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a806eed0 msgr2=0x7f54a80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.315+0000 7f54ad000640 1 --2- 172.21.15.129:0/1393555119 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a806eed0 0x7f54a80ccc80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5494005fd0 tx=0x7f549402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 -- 172.21.15.129:0/1393555119 shutdown_connections 2024-09-17T04:05:46.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 --2- 172.21.15.129:0/1393555119 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a806eed0 0x7f54a80ccc80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 --2- 172.21.15.129:0/1393555119 >> v2:172.21.15.181:3300/0 conn(0x7f54a80705a0 0x7f54a80b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 -- 172.21.15.129:0/1393555119 >> 172.21.15.129:0/1393555119 conn(0x7f54a806d410 msgr2=0x7f54a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 -- 172.21.15.129:0/1393555119 shutdown_connections 2024-09-17T04:05:46.317 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.316+0000 7f54ad000640 1 -- 172.21.15.129:0/1393555119 wait complete. 2024-09-17T04:05:46.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.318+0000 7f54ad000640 1 Processor -- start 2024-09-17T04:05:46.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.318+0000 7f54ad000640 1 -- start start 2024-09-17T04:05:46.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a806eed0 0x7f54a80c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a80705a0 0x7f54a81f9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8160200 con 0x7f54a806f2e0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54a81612e0 con 0x7f54a806eed0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.319+0000 7f54ad000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f54a80cd1c0 con 0x7f54a80705a0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39596/0 (socket says 172.21.15.129:39596) 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 -- 172.21.15.129:0/2956815110 learned_addr learned my addr 172.21.15.129:0/2956815110 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a6800640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a80705a0 0x7f54a81f9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5e00640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a806eed0 0x7f54a80c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 -- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a80705a0 msgr2=0x7f54a81f9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a80705a0 0x7f54a81f9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 -- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a806eed0 msgr2=0x7f54a80c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a806eed0 0x7f54a80c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 -- 172.21.15.129:0/2956815110 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a81563c0 con 0x7f54a806f2e0 2024-09-17T04:05:46.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.320+0000 7f54a5400640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f5494034040 tx=0x7f54940035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.321+0000 7f549e000640 1 -- 172.21.15.129:0/2956815110 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549403e030 con 0x7f54a806f2e0 2024-09-17T04:05:46.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.321+0000 7f549e000640 1 -- 172.21.15.129:0/2956815110 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5494003ae0 con 0x7f54a806f2e0 2024-09-17T04:05:46.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.321+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a81fa890 con 0x7f54a806f2e0 2024-09-17T04:05:46.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.321+0000 7f549e000640 1 -- 172.21.15.129:0/2956815110 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494035040 con 0x7f54a806f2e0 2024-09-17T04:05:46.322 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.321+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a81fa5b0 con 0x7f54a806f2e0 2024-09-17T04:05:46.323 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.323+0000 7f549e000640 1 -- 172.21.15.129:0/2956815110 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5494041030 con 0x7f54a806f2e0 2024-09-17T04:05:46.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.324+0000 7f549e000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 0x7f548807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.324 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.324+0000 7f549e000640 1 -- 172.21.15.129:0/2956815110 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f54940bb430 con 0x7f54a806f2e0 2024-09-17T04:05:46.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.325+0000 7f54a5e00640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 0x7f548807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.325+0000 7f54b41c1740 1 Processor -- start 2024-09-17T04:05:46.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.325+0000 7f54b41c1740 1 -- start start 2024-09-17T04:05:46.325 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.325+0000 7f54a5e00640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 0x7f548807b0a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5498009b40 tx=0x7f5498005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d91970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522d91eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a522ceb430 con 0x55a5229a0590 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a522cdb5d0 con 0x55a5229c1270 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.326+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55a522ccaa00 con 0x55a5229983b0 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39612/0 (socket says 172.21.15.129:39612) 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 -- 172.21.15.129:0/716728659 learned_addr learned my addr 172.21.15.129:0/716728659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a5e00640 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d91970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.327 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a5400640 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522d91eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 -- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 msgr2=0x55a522d91eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522d91eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 -- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 msgr2=0x55a522d91970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d91970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 -- 172.21.15.129:0/716728659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a522c93520 con 0x55a5229a0590 2024-09-17T04:05:46.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.327+0000 7f54a6800640 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f54a0001b90 tx=0x7f54a0002050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24f522cc1fdde2a server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.331+0000 7f5492a00640 1 -- 172.21.15.129:0/716728659 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0019070 con 0x55a5229a0590 2024-09-17T04:05:46.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.331+0000 7f5492a00640 1 -- 172.21.15.129:0/716728659 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54a0003b80 con 0x55a5229a0590 2024-09-17T04:05:46.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.331+0000 7f5492a00640 1 -- 172.21.15.129:0/716728659 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a000e800 con 0x55a5229a0590 2024-09-17T04:05:46.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 -- 172.21.15.129:0/716728659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 msgr2=0x55a522d61f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f54a0001b90 tx=0x7f54a0002050 comp rx=0 tx=0).stop 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 -- 172.21.15.129:0/716728659 shutdown_connections 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d61f60 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522d91eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 --2- 172.21.15.129:0/716728659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d91970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 -- 172.21.15.129:0/716728659 >> 172.21.15.129:0/716728659 conn(0x55a5229b53c0 msgr2=0x55a522d67970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 -- 172.21.15.129:0/716728659 shutdown_connections 2024-09-17T04:05:46.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.332+0000 7f54b41c1740 1 -- 172.21.15.129:0/716728659 wait complete. 2024-09-17T04:05:46.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.333+0000 7f54b41c1740 1 -- start start 2024-09-17T04:05:46.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.334+0000 7f54b41c1740 1 Processor -- start 2024-09-17T04:05:46.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229983b0 0x55a522dbc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229c1270 0x55a522da6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a522c71330 con 0x55a5229a0590 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a522cce810 con 0x55a5229983b0 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55a522a3b620 con 0x55a5229c1270 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54b41c1740 10 client.0 _ll_register_callbacks cb 0x55a522c76420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.335+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39622/0 (socket says 172.21.15.129:39622) 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 -- 172.21.15.129:0/1571173322 learned_addr learned my addr 172.21.15.129:0/1571173322 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5e00640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229983b0 0x55a522dbc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a6800640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229c1270 0x55a522da6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 -- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229c1270 msgr2=0x55a522da6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229c1270 0x55a522da6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 -- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229983b0 msgr2=0x55a522dbc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229983b0 0x55a522dbc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 -- 172.21.15.129:0/1571173322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a522a2c850 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54a5400640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f5494006010 tx=0x7f54940829c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b415ae55a67ceb20 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f5487400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494048040 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f549408c770 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.336+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54a0003b80 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=219) ==== 24+0+0 (secure 0 0 0) 0x7f5494083770 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 10 client.11292 fetch_fsmap learned FSMap version 219 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=219,monmap=2+}) -- 0x55a522aa36b0 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54a0003b80 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 219) ==== 3502+0+0 (secure 0 0 0) 0x7f5494088ce0 con 0x55a5229a0590 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 10 client.11292 fetch_fsmap finished waiting for FSMap version 219 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 10 client.11292 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:05:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 20 client.11292 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:46.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a52297fe40 0x55a522dbee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 4 client.11292 mds_command: new command op to 11253 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:05:46.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 --> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a522cdd520 con 0x55a52297fe40 2024-09-17T04:05:46.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.337+0000 7f54a5e00640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a52297fe40 0x55a522dbee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.338+0000 7f5487400640 10 client.11292 ms_handle_connect on v2:172.21.15.129:6834/3706217946 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.338+0000 7f54a5e00640 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a52297fe40 0x55a522dbee70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f02ff346775d2e63 server_cookie=80ca24b02b36fa86 in_seq=0 out_seq=0 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f5487400640 1 -- 172.21.15.129:0/1571173322 <== mds.0 v2:172.21.15.129:6834/3706217946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55a522cdd520 con 0x55a52297fe40 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f5487400640 10 client.11292 handle_command_reply: tid=0 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 client.11292 shutdown 2024-09-17T04:05:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 msgr2=0x55a522d8b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f5494006010 tx=0x7f54940829c0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 shutdown_connections 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a52297fe40 0x55a522dbee70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229c1270 0x55a522da6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522d8b640 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 --2- 172.21.15.129:0/1571173322 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229983b0 0x55a522dbc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f5487400640 0 client.11292 ms_handle_reset on v2:172.21.15.129:6834/3706217946 2024-09-17T04:05:46.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 >> 172.21.15.129:0/1571173322 conn(0x55a5229b53c0 msgr2=0x55a522d632f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 shutdown_connections 2024-09-17T04:05:46.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 1 -- 172.21.15.129:0/1571173322 wait complete. 2024-09-17T04:05:46.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.339+0000 7f54b41c1740 20 client.11292 trim_cache size 0 max 16384 2024-09-17T04:05:46.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.358+0000 7f54b41c1740 1 Processor -- start 2024-09-17T04:05:46.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.359+0000 7f54b41c1740 1 -- start start 2024-09-17T04:05:46.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522dafce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522dcf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54a80c97d0 con 0x55a5229a0590 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54a8159050 con 0x55a5229c1270 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f54a80c9180 con 0x55a5229983b0 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39624/0 (socket says 172.21.15.129:39624) 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54a5400640 1 -- 172.21.15.129:0/4138705058 learned_addr learned my addr 172.21.15.129:0/4138705058 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5e00640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522dafce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.360+0000 7f54a6800640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522dcf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 -- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 msgr2=0x55a522dafce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522dafce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 -- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 msgr2=0x55a522dcf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522dcf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 -- 172.21.15.129:0/4138705058 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a522d5c8d0 con 0x55a5229a0590 2024-09-17T04:05:46.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5e00640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522dafce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:46.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f54a5400640 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f54940a5010 tx=0x7f549407efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f47049861ec849 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f5492000640 1 -- 172.21.15.129:0/4138705058 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5494048070 con 0x55a5229a0590 2024-09-17T04:05:46.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f5492000640 1 -- 172.21.15.129:0/4138705058 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5494083070 con 0x55a5229a0590 2024-09-17T04:05:46.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.361+0000 7f5492000640 1 -- 172.21.15.129:0/4138705058 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940827e0 con 0x55a5229a0590 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 -- 172.21.15.129:0/4138705058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 msgr2=0x55a522db0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f54940a5010 tx=0x7f549407efb0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 -- 172.21.15.129:0/4138705058 shutdown_connections 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522dcf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229a0590 0x55a522db0220 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 --2- 172.21.15.129:0/4138705058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229983b0 0x55a522dafce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 -- 172.21.15.129:0/4138705058 >> 172.21.15.129:0/4138705058 conn(0x55a5229b53c0 msgr2=0x55a522d8bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 -- 172.21.15.129:0/4138705058 shutdown_connections 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.362+0000 7f54b41c1740 1 -- 172.21.15.129:0/4138705058 wait complete. 2024-09-17T04:05:46.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.363+0000 7f54b41c1740 1 -- start start 2024-09-17T04:05:46.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.363+0000 7f54b41c1740 1 Processor -- start 2024-09-17T04:05:46.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229a0590 0x55a522d8bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d8c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55a522da6ae0 con 0x55a5229983b0 2024-09-17T04:05:46.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55a522d8d2b0 con 0x55a5229c1270 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55a522daed70 con 0x55a5229a0590 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54b41c1740 10 client.0 _ll_register_callbacks cb 0x7f5494082e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39626/0 (socket says 172.21.15.129:39626) 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 -- 172.21.15.129:0/3239790762 learned_addr learned my addr 172.21.15.129:0/3239790762 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5400640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229a0590 0x55a522d8bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a6800640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d8c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 -- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229a0590 msgr2=0x55a522d8bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229a0590 0x55a522d8bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 -- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 msgr2=0x55a522d8c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d8c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 -- 172.21.15.129:0/3239790762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a522e300f0 con 0x55a5229983b0 2024-09-17T04:05:46.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a6800640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d8c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f54a5e00640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x55a522daf540 tx=0x7f5498014070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d2b1166d85c1287 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.365+0000 7f5487400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498005220 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f54980053c0 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5494083070 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=219) ==== 24+0+0 (secure 0 0 0) 0x7f5498039050 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 10 client.11298 fetch_fsmap learned FSMap version 219 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=219,monmap=2+}) -- 0x55a522d8ac40 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5494083070 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 219) ==== 3502+0+0 (secure 0 0 0) 0x7f54980047a0 con 0x55a5229983b0 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 10 client.11298 fetch_fsmap finished waiting for FSMap version 219 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 10 client.11298 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:05:46.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 20 client.11298 populate_metadata read hostname 'smithi129' 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a522e2cb60 0x55a522dbce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 4 client.11298 mds_command: new command op to 11253 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.366+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 --> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] -- command(tid 0: {"prefix": "status"}) -- 0x55a522cd8fd0 con 0x55a522e2cb60 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f54a5400640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a522e2cb60 0x55a522dbce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f54a5400640 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a522e2cb60 0x55a522dbce70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c408270e565acd9d server_cookie=29a2a0984130e1f1 in_seq=0 out_seq=0 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f5487400640 10 client.11298 ms_handle_connect on v2:172.21.15.129:6834/3706217946 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f5487400640 1 -- 172.21.15.129:0/3239790762 <== mds.0 v2:172.21.15.129:6834/3706217946 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x55a522cd8fd0 con 0x55a522e2cb60 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f5487400640 10 client.11298 handle_command_reply: tid=0 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.367+0000 7f54b41c1740 1 client.11298 shutdown 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 msgr2=0x55a522e11de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x55a522daf540 tx=0x7f5498014070 comp rx=0 tx=0).stop 2024-09-17T04:05:46.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 shutdown_connections 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:6834/3706217946,v1:172.21.15.129:6835/3706217946] conn(0x55a522e2cb60 0x55a522dbce70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55a5229c1270 0x55a522d8c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55a5229a0590 0x55a522d8bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 --2- 172.21.15.129:0/3239790762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55a5229983b0 0x55a522e11de0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f5487400640 0 client.11298 ms_handle_reset on v2:172.21.15.129:6834/3706217946 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 >> 172.21.15.129:0/3239790762 conn(0x55a5229b53c0 msgr2=0x55a522da7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 shutdown_connections 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 1 -- 172.21.15.129:0/3239790762 wait complete. 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54b41c1740 20 client.11298 trim_cache size 0 max 16384 2024-09-17T04:05:46.372 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 11253, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.0705697949999999, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 219, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 222, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 222, 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 6.6619759350000001 2024-09-17T04:05:46.373 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:05:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 msgr2=0x7f548807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.368+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 0x7f548807b0a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f5498009b40 tx=0x7f5498005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 msgr2=0x7f54a81f9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f5494034040 tx=0x7f54940035f0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 shutdown_connections 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5488078b90 0x7f548807b0a0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54a80705a0 0x7f54a81f9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54a806f2e0 0x7f54a81f9950 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 --2- 172.21.15.129:0/2956815110 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54a806eed0 0x7f54a80c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 >> 172.21.15.129:0/2956815110 conn(0x7f54a806d410 msgr2=0x7f54a806f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 shutdown_connections 2024-09-17T04:05:46.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.369+0000 7f54ad000640 1 -- 172.21.15.129:0/2956815110 wait complete. 2024-09-17T04:05:46.391 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:05:46.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:46.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:46.572 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:05:46.572 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:46.572 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:46.573 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:46.608 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:46.608 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:46.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:05:46.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 -- 172.21.15.129:0/3889088163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 msgr2=0x7fb994003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 --2- 172.21.15.129:0/3889088163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb994003e50 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fb9a000baf0 tx=0x7fb994005830 comp rx=0 tx=0).stop 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 -- 172.21.15.129:0/3889088163 shutdown_connections 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 --2- 172.21.15.129:0/3889088163 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb994003e50 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 -- 172.21.15.129:0/3889088163 >> 172.21.15.129:0/3889088163 conn(0x7fb9b006d410 msgr2=0x7fb9b00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 -- 172.21.15.129:0/3889088163 shutdown_connections 2024-09-17T04:05:46.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.944+0000 7fb9b5200640 1 -- 172.21.15.129:0/3889088163 wait complete. 2024-09-17T04:05:46.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.946+0000 7fb9b5200640 1 Processor -- start 2024-09-17T04:05:46.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.947+0000 7fb9b5200640 1 -- start start 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 0x7fb9b0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 0x7fb9b015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b015f270 con 0x7fb994003a00 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b00b4430 con 0x7fb9b01611a0 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9b5200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b015f3f0 con 0x7fb9b015d9b0 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9ade00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9ade00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39650/0 (socket says 172.21.15.129:39650) 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9ade00640 1 -- 172.21.15.129:0/770559550 learned_addr learned my addr 172.21.15.129:0/770559550 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:46.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9a5400640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 0x7fb9b0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.948+0000 7fb9ae800640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 0x7fb9b015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 -- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 msgr2=0x7fb9b015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 0x7fb9b015ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 -- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 msgr2=0x7fb9b0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 0x7fb9b0152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 -- 172.21.15.129:0/770559550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b015e300 con 0x7fb994003a00 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9a5400640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 0x7fb9b0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ae800640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 0x7fb9b015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9ade00640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0034040 tx=0x7fb9a0032760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0041070 con 0x7fb994003a00 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb9a0032d60 con 0x7fb994003a00 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0004290 con 0x7fb994003a00 2024-09-17T04:05:46.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b01525d0 con 0x7fb994003a00 2024-09-17T04:05:46.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.949+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b01606c0 con 0x7fb994003a00 2024-09-17T04:05:46.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.950+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b00ccf30 con 0x7fb994003a00 2024-09-17T04:05:46.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.951+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb9a00035f0 con 0x7fb994003a00 2024-09-17T04:05:46.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.952+0000 7fb9a6a00640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 0x7fb98c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:46.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.953+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fb9a00bbbe0 con 0x7fb994003a00 2024-09-17T04:05:46.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.953+0000 7fb9a5400640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 0x7fb98c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:46.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.954+0000 7fb9a5400640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 0x7fb98c07b0d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb9b006fb00 tx=0x7fb9980067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:46.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:46.957+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb9b015e300 con 0x7fb994003a00 2024-09-17T04:05:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.151+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb9b006a140 con 0x7fb994003a00 2024-09-17T04:05:47.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.152+0000 7fb9a6a00640 1 -- 172.21.15.129:0/770559550 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v222) ==== 80+0+370 (secure 0 0 0) 0x7fb9a0083e90 con 0x7fb994003a00 2024-09-17T04:05:47.152 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.181:6837/2764153925 2024-09-18T04:05:39.282040+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.181:6835/2764153925 2024-09-18T04:05:39.282040+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6834/3249193545 2024-09-18T04:05:39.016692+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6835/711019236 2024-09-18T04:05:39.001120+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6832/3249193545 2024-09-18T04:05:39.016692+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6833/711019236 2024-09-18T04:05:39.001120+0000 2024-09-17T04:05:47.153 INFO:teuthology.orchestra.run.smithi129.stderr:listed 6 entries 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 msgr2=0x7fb98c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 0x7fb98c07b0d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb9b006fb00 tx=0x7fb9980067a0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 msgr2=0x7fb9b0162090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0034040 tx=0x7fb9a0032760 comp rx=0 tx=0).stop 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 shutdown_connections 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb98c078bc0 0x7fb98c07b0d0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb9b015d9b0 0x7fb9b015ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9b01611a0 0x7fb9b0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 --2- 172.21.15.129:0/770559550 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb994003a00 0x7fb9b0162090 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.154+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 >> 172.21.15.129:0/770559550 conn(0x7fb9b006d410 msgr2=0x7fb9b015c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.155+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 shutdown_connections 2024-09-17T04:05:47.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.155+0000 7fb9b5200640 1 -- 172.21.15.129:0/770559550 wait complete. 2024-09-17T04:05:47.176 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:05:47.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:47.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:05:47.238 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:05:47.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:05:47.239 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:05:47.275 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:47.275 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:47.275 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:47.315 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:47.315 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:47.315 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:47.315 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:47.351 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:47.351 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:05:47.352 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:47.352 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:47.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:47.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 -- 172.21.15.129:0/3673996549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f356806eed0 msgr2=0x7f35680ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 --2- 172.21.15.129:0/3673996549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f356806eed0 0x7f35680ccc80 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f3554005fd0 tx=0x7f355402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 -- 172.21.15.129:0/3673996549 shutdown_connections 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 --2- 172.21.15.129:0/3673996549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f356806eed0 0x7f35680ccc80 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 --2- 172.21.15.129:0/3673996549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f35680705a0 0x7f35680b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 -- 172.21.15.129:0/3673996549 >> 172.21.15.129:0/3673996549 conn(0x7f356806d410 msgr2=0x7f35680b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 -- 172.21.15.129:0/3673996549 shutdown_connections 2024-09-17T04:05:47.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.612+0000 7f3570400640 1 -- 172.21.15.129:0/3673996549 wait complete. 2024-09-17T04:05:47.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.615+0000 7f3570400640 1 Processor -- start 2024-09-17T04:05:47.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.615+0000 7f3570400640 1 -- start start 2024-09-17T04:05:47.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356806eed0 0x7f35681594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:47.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f356806f2e0 0x7f35681f5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3568160200 con 0x7f35680705a0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f35681612e0 con 0x7f356806eed0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f3570400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f35680cd1c0 con 0x7f356806f2e0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.616+0000 7f356e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39670/0 (socket says 172.21.15.129:39670) 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 -- 172.21.15.129:0/3424231311 learned_addr learned my addr 172.21.15.129:0/3424231311 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356d000640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f356806f2e0 0x7f35681f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356da00640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356806eed0 0x7f35681594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 -- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f356806f2e0 msgr2=0x7f35681f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f356806f2e0 0x7f35681f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 -- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356806eed0 msgr2=0x7f35681594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356806eed0 0x7f35681594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 -- 172.21.15.129:0/3424231311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356815bd00 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f356e400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f356000c760 tx=0x7f356000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3560017070 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f356000ce10 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.617+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35681f5980 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.618+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35600052b0 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.618+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356815c2c0 con 0x7f35680705a0 2024-09-17T04:05:47.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.619+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35680ccc80 con 0x7f35680705a0 2024-09-17T04:05:47.620 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.620+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f356001a070 con 0x7f35680705a0 2024-09-17T04:05:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.621+0000 7f3565600640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 0x7f354807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.621+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f356009a3e0 con 0x7f35680705a0 2024-09-17T04:05:47.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.621+0000 7f356da00640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 0x7f354807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:47.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.622+0000 7f356da00640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 0x7f354807b0d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f355c005fd0 tx=0x7f355c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:47.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.625+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f35600627b0 con 0x7f35680705a0 2024-09-17T04:05:47.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.845+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f356806a140 con 0x7f35680705a0 2024-09-17T04:05:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.846+0000 7f3565600640 1 -- 172.21.15.129:0/3424231311 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5254 (secure 0 0 0) 0x7f35600620b0 con 0x7f35680705a0 2024-09-17T04:05:47.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:47.846 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":219,"btime":"2024-09-17T04:05:44:314407+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:44.314397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11253,"qdb_cluster":[11253]},"id":16}]} 2024-09-17T04:05:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 219 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 msgr2=0x7f354807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 0x7f354807b0d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f355c005fd0 tx=0x7f355c005850 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 msgr2=0x7f35681f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f356000c760 tx=0x7f356000cc20 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 shutdown_connections 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3548078bc0 0x7f354807b0d0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f35680705a0 0x7f35681f9130 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f356806f2e0 0x7f35681f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 --2- 172.21.15.129:0/3424231311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f356806eed0 0x7f35681594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 >> 172.21.15.129:0/3424231311 conn(0x7f356806d410 msgr2=0x7f35680ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 shutdown_connections 2024-09-17T04:05:47.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:47.848+0000 7f3570400640 1 -- 172.21.15.129:0/3424231311 wait complete. 2024-09-17T04:05:47.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:48.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 -- 172.21.15.129:0/2111183370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0300705a0 msgr2=0x7fc0300b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 --2- 172.21.15.129:0/2111183370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0300705a0 0x7fc0300b4140 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fc020009830 tx=0x7fc02002a510 comp rx=0 tx=0).stop 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 -- 172.21.15.129:0/2111183370 shutdown_connections 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 --2- 172.21.15.129:0/2111183370 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc03006eed0 0x7fc0300ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 --2- 172.21.15.129:0/2111183370 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc0300705a0 0x7fc0300b4140 unknown :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 -- 172.21.15.129:0/2111183370 >> 172.21.15.129:0/2111183370 conn(0x7fc03006d410 msgr2=0x7fc0300b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 -- 172.21.15.129:0/2111183370 shutdown_connections 2024-09-17T04:05:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.093+0000 7fc035200640 1 -- 172.21.15.129:0/2111183370 wait complete. 2024-09-17T04:05:48.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.095+0000 7fc035200640 1 Processor -- start 2024-09-17T04:05:48.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.095+0000 7fc035200640 1 -- start start 2024-09-17T04:05:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 0x7fc0301e6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 0x7fc0301e6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc030160200 con 0x7fc03006eed0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc0301612e0 con 0x7fc03006f2e0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc035200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc0300cd1c0 con 0x7fc0300705a0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39690/0 (socket says 172.21.15.129:39690) 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 -- 172.21.15.129:0/2096036024 learned_addr learned my addr 172.21.15.129:0/2096036024 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 -- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 msgr2=0x7fc0301e6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02d400640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 0x7fc0301e6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02e800640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 0x7fc0301e6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 0x7fc0301e6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 -- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 msgr2=0x7fc0301e6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 0x7fc0301e6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02de00640 1 -- 172.21.15.129:0/2096036024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc03015e890 con 0x7fc03006eed0 2024-09-17T04:05:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02d400640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 0x7fc0301e6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.097+0000 7fc02e800640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 0x7fc0301e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc02de00640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc020009950 tx=0x7fc020033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020005c90 con 0x7fc03006eed0 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc020031e50 con 0x7fc03006eed0 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020005260 con 0x7fc03006eed0 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0301e8ab0 con 0x7fc03006eed0 2024-09-17T04:05:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.098+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0300c8410 con 0x7fc03006eed0 2024-09-17T04:05:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.099+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0300ccc80 con 0x7fc03006eed0 2024-09-17T04:05:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.101+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc020005400 con 0x7fc03006eed0 2024-09-17T04:05:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.101+0000 7fc026000640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 0x7fc01007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.102+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fc0200bbf80 con 0x7fc03006eed0 2024-09-17T04:05:48.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.102+0000 7fc02d400640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 0x7fc01007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.103+0000 7fc02d400640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 0x7fc01007b0d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc0301e7d80 tx=0x7fc01c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:48.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.106+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc03015e890 con 0x7fc03006eed0 2024-09-17T04:05:48.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.328+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc03006a140 con 0x7fc03006eed0 2024-09-17T04:05:48.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.329+0000 7fc026000640 1 -- 172.21.15.129:0/2096036024 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5254 (secure 0 0 0) 0x7fc02003f030 con 0x7fc03006eed0 2024-09-17T04:05:48.329 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:48.329 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":219,"btime":"2024-09-17T04:05:44:314407+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:44.314397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11253,"qdb_cluster":[11253]},"id":16}]} 2024-09-17T04:05:48.330 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 219 2024-09-17T04:05:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 msgr2=0x7fc01007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 0x7fc01007b0d0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc0301e7d80 tx=0x7fc01c005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 msgr2=0x7fc030073d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7fc020009950 tx=0x7fc020033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 shutdown_connections 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc010078bc0 0x7fc01007b0d0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc0300705a0 0x7fc0301e6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc03006f2e0 0x7fc0301e6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 --2- 172.21.15.129:0/2096036024 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc03006eed0 0x7fc030073d40 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 >> 172.21.15.129:0/2096036024 conn(0x7fc03006d410 msgr2=0x7fc030074390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 shutdown_connections 2024-09-17T04:05:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.332+0000 7fc035200640 1 -- 172.21.15.129:0/2096036024 wait complete. 2024-09-17T04:05:48.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 -- 172.21.15.129:0/2346281840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 msgr2=0x7facf415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 --2- 172.21.15.129:0/2346281840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf415bca0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7facf40727e0 tx=0x7facdc02a560 comp rx=0 tx=0).stop 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 -- 172.21.15.129:0/2346281840 shutdown_connections 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 --2- 172.21.15.129:0/2346281840 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf415c1e0 0x7facf4072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 --2- 172.21.15.129:0/2346281840 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf415bca0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 --2- 172.21.15.129:0/2346281840 >> v2:172.21.15.181:3300/0 conn(0x7facf41524c0 0x7facf40cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 -- 172.21.15.129:0/2346281840 >> 172.21.15.129:0/2346281840 conn(0x7facf406cfc0 msgr2=0x7facf406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 -- 172.21.15.129:0/2346281840 shutdown_connections 2024-09-17T04:05:48.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.578+0000 7facf3e00640 1 -- 172.21.15.129:0/2346281840 wait complete. 2024-09-17T04:05:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.580+0000 7facf3e00640 1 Processor -- start 2024-09-17T04:05:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.580+0000 7facf3e00640 1 -- start start 2024-09-17T04:05:48.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf41524c0 0x7facf4145b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf415c1e0 0x7facf41e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7facf4073810 con 0x7facf40cfb00 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7facf4073690 con 0x7facf415c1e0 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7facf40d02c0 con 0x7facf41524c0 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39706/0 (socket says 172.21.15.129:39706) 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 -- 172.21.15.129:0/873466289 learned_addr learned my addr 172.21.15.129:0/873466289 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facea000640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf41524c0 0x7facf4145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf3400640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf415c1e0 0x7facf41e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 -- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf41524c0 msgr2=0x7facf4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf41524c0 0x7facf4145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 -- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf415c1e0 msgr2=0x7facf41e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf415c1e0 0x7facf41e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.582+0000 7facf2a00640 1 -- 172.21.15.129:0/873466289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf41460b0 con 0x7facf40cfb00 2024-09-17T04:05:48.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facf2a00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7face400d720 tx=0x7face400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4017070 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7face4009ab0 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face4010460 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf41570d0 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.583+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf4157370 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.584+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf40cf5c0 con 0x7facf40cfb00 2024-09-17T04:05:48.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.585+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7face401a070 con 0x7facf40cfb00 2024-09-17T04:05:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.586+0000 7facebe00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.587+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7face409afd0 con 0x7facf40cfb00 2024-09-17T04:05:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.587+0000 7facea000640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.588+0000 7facea000640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7facdc033000 tx=0x7facdc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.591+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7facf41460b0 con 0x7facf40cfb00 2024-09-17T04:05:48.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.779+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7facf406a140 con 0x7facf40cfb00 2024-09-17T04:05:48.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.781+0000 7facebe00640 1 -- 172.21.15.129:0/873466289 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+17124 (secure 0 0 0) 0x7face4062c20 con 0x7facf40cfb00 2024-09-17T04:05:48.781 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:48.782 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":222,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:39.282065+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:05:22.611745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-17T04:05:23.660603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.181:6835/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.129:6834/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6835/711019236":"2024-09-18T04:05:39.001120+0000","172.21.15.129:6832/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6833/711019236":"2024-09-18T04:05:39.001120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 msgr2=0x7facd007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7facdc033000 tx=0x7facdc004730 comp rx=0 tx=0).stop 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 msgr2=0x7facf41e6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7face400d720 tx=0x7face400dbe0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 shutdown_connections 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7facd0078b90 0x7facd007b0a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7facf415c1e0 0x7facf41e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7facf41524c0 0x7facf4145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 --2- 172.21.15.129:0/873466289 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7facf40cfb00 0x7facf41e6120 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 >> 172.21.15.129:0/873466289 conn(0x7facf406cfc0 msgr2=0x7facf406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.783+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 shutdown_connections 2024-09-17T04:05:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:48.784+0000 7facf3e00640 1 -- 172.21.15.129:0/873466289 wait complete. 2024-09-17T04:05:48.810 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:05:48.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 -- 172.21.15.129:0/2980269477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88140cd370 msgr2=0x7f88140b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 --2- 172.21.15.129:0/2980269477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88140cd370 0x7f88140b4290 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8804009860 tx=0x7f880402a5a0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 -- 172.21.15.129:0/2980269477 shutdown_connections 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 --2- 172.21.15.129:0/2980269477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f88140cd370 0x7f88140b4290 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 -- 172.21.15.129:0/2980269477 >> 172.21.15.129:0/2980269477 conn(0x7f881406d410 msgr2=0x7f88140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 -- 172.21.15.129:0/2980269477 shutdown_connections 2024-09-17T04:05:49.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.030+0000 7f881a200640 1 -- 172.21.15.129:0/2980269477 wait complete. 2024-09-17T04:05:49.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.032+0000 7f881a200640 1 Processor -- start 2024-09-17T04:05:49.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.033+0000 7f881a200640 1 -- start start 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88140cd370 0x7f88140c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 0x7f88140999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8814163110 con 0x7f8814074370 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f88140cb1e0 con 0x7f88140cd370 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f881a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8814163290 con 0x7f8814162d00 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f8813e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f8813e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39708/0 (socket says 172.21.15.129:39708) 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.034+0000 7f8813e00640 1 -- 172.21.15.129:0/2426210281 learned_addr learned my addr 172.21.15.129:0/2426210281 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f880aa00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 0x7f88140999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813400640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88140cd370 0x7f88140c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 -- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 msgr2=0x7f88140999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 0x7f88140999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 -- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88140cd370 msgr2=0x7f88140c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88140cd370 0x7f88140c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 -- 172.21.15.129:0/2426210281 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8814099ef0 con 0x7f8814074370 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f880aa00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 0x7f88140999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f8813e00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f880c00cad0 tx=0x7f880c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c016020 con 0x7f8814074370 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f880c00ad20 con 0x7f8814074370 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.035+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c012680 con 0x7f8814074370 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.036+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881415e3d0 con 0x7f8814074370 2024-09-17T04:05:49.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.036+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88140b1b20 con 0x7f8814074370 2024-09-17T04:05:49.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.037+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88140b4290 con 0x7f8814074370 2024-09-17T04:05:49.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.038+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f880c019020 con 0x7f8814074370 2024-09-17T04:05:49.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.039+0000 7f880be00640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 0x7f87f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.040+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f880c09b3e0 con 0x7f8814074370 2024-09-17T04:05:49.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.040+0000 7f8813400640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 0x7f87f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.041+0000 7f8813400640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 0x7f87f007b0a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f8804034000 tx=0x7f8804004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:49.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.044+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f880c020020 con 0x7f8814074370 2024-09-17T04:05:49.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.268+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f881406a140 con 0x7f8814074370 2024-09-17T04:05:49.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.269+0000 7f880be00640 1 -- 172.21.15.129:0/2426210281 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5254 (secure 0 0 0) 0x7f8814099ef0 con 0x7f8814074370 2024-09-17T04:05:49.269 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:49.269 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":219,"btime":"2024-09-17T04:05:44:314407+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:44.314397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11253,"qdb_cluster":[11253]},"id":16}]} 2024-09-17T04:05:49.269 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 219 2024-09-17T04:05:49.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 msgr2=0x7f87f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 0x7f87f007b0a0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f8804034000 tx=0x7f8804004730 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 msgr2=0x7f8814154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f880c00cad0 tx=0x7f880c00cf90 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 shutdown_connections 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f87f0078b90 0x7f87f007b0a0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8814074370 0x7f8814154ec0 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8814162d00 0x7f88140999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 --2- 172.21.15.129:0/2426210281 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f88140cd370 0x7f88140c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 >> 172.21.15.129:0/2426210281 conn(0x7f881406d410 msgr2=0x7f8814156620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 shutdown_connections 2024-09-17T04:05:49.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.271+0000 7f881a200640 1 -- 172.21.15.129:0/2426210281 wait complete. 2024-09-17T04:05:49.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:49.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:05:49.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:05:49.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 -- 172.21.15.129:0/1487423463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d2015ca30 msgr2=0x7f1d200b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 --2- 172.21.15.129:0/1487423463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d2015ca30 0x7f1d200b3e70 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f1d10009830 tx=0x7f1d1002a510 comp rx=0 tx=0).stop 2024-09-17T04:05:49.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 -- 172.21.15.129:0/1487423463 shutdown_connections 2024-09-17T04:05:49.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 --2- 172.21.15.129:0/1487423463 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d200ccc50 0x7f1d200b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 --2- 172.21.15.129:0/1487423463 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d2015ca30 0x7f1d200b3e70 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 -- 172.21.15.129:0/1487423463 >> 172.21.15.129:0/1487423463 conn(0x7f1d200cee70 msgr2=0x7f1d200cf280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:49.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 -- 172.21.15.129:0/1487423463 shutdown_connections 2024-09-17T04:05:49.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.520+0000 7f1d26000640 1 -- 172.21.15.129:0/1487423463 wait complete. 2024-09-17T04:05:49.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.522+0000 7f1d26000640 1 Processor -- start 2024-09-17T04:05:49.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.522+0000 7f1d26000640 1 -- start start 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d200b7b40 0x7f1d201f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d2015ca30 0x7f1d2014bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20073310 con 0x7f1d200ccc50 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20073190 con 0x7f1d200b7b40 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d26000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1d20073490 con 0x7f1d2015ca30 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d1ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d1ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39732/0 (socket says 172.21.15.129:39732) 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d1ea00640 1 -- 172.21.15.129:0/2531920652 learned_addr learned my addr 172.21.15.129:0/2531920652 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d1f400640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d200b7b40 0x7f1d201f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.524+0000 7f1d1fe00640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d2015ca30 0x7f1d2014bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 -- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d2015ca30 msgr2=0x7f1d2014bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d2015ca30 0x7f1d2014bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 -- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d200b7b40 msgr2=0x7f1d201f79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d200b7b40 0x7f1d201f79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 -- 172.21.15.129:0/2531920652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2014c660 con 0x7f1d200ccc50 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d1ea00640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e7c0 tx=0x7f1d0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c00cc60 con 0x7f1d200ccc50 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d0c010430 con 0x7f1d200ccc50 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c010860 con 0x7f1d200ccc50 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.525+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2014c940 con 0x7f1d200ccc50 2024-09-17T04:05:49.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.526+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d201f71f0 con 0x7f1d200ccc50 2024-09-17T04:05:49.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.527+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d20072a80 con 0x7f1d200ccc50 2024-09-17T04:05:49.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.528+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1d0c010a00 con 0x7f1d200ccc50 2024-09-17T04:05:49.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.530+0000 7f1d17400640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 0x7f1d0007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:49.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.530+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f1d0c09b7e0 con 0x7f1d200ccc50 2024-09-17T04:05:49.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.530+0000 7f1d1f400640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 0x7f1d0007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:49.531 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.531+0000 7f1d1f400640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 0x7f1d0007b0d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1d10034000 tx=0x7f1d10004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:49.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.534+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1d0c063430 con 0x7f1d200ccc50 2024-09-17T04:05:49.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.757+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d200b55a0 con 0x7f1d200ccc50 2024-09-17T04:05:49.758 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.758+0000 7f1d17400640 1 -- 172.21.15.129:0/2531920652 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5254 (secure 0 0 0) 0x7f1d0c067e60 con 0x7f1d200ccc50 2024-09-17T04:05:49.758 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:49.758 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":219,"btime":"2024-09-17T04:05:44:314407+0000","default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:05:24.702816+0000","modified":"2024-09-17T04:05:44.314397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":222,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11253},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11253":{"gid":11253,"name":"c","rank":0,"incarnation":215,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6835/3706217946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3706217946},{"type":"v1","addr":"172.21.15.129:6835","nonce":3706217946}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11253,"qdb_cluster":[11253]},"id":16}]} 2024-09-17T04:05:49.759 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 219 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 msgr2=0x7f1d0007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 0x7f1d0007b0d0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1d10034000 tx=0x7f1d10004730 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 msgr2=0x7f1d20149320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c00e7c0 tx=0x7f1d0c00ec80 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 shutdown_connections 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1d00078bc0 0x7f1d0007b0d0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1d2015ca30 0x7f1d2014bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1d200ccc50 0x7f1d20149320 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 --2- 172.21.15.129:0/2531920652 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d200b7b40 0x7f1d201f79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 >> 172.21.15.129:0/2531920652 conn(0x7f1d200cee70 msgr2=0x7f1d200b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:49.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 shutdown_connections 2024-09-17T04:05:49.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:49.760+0000 7f1d26000640 1 -- 172.21.15.129:0/2531920652 wait complete. 2024-09-17T04:05:49.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:50.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 -- 172.21.15.129:0/3137404053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840b4140 msgr2=0x7f01840ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 --2- 172.21.15.129:0/3137404053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840b4140 0x7f01840ccc80 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7f0184072e00 tx=0x7f017002a560 comp rx=0 tx=0).stop 2024-09-17T04:05:50.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 -- 172.21.15.129:0/3137404053 shutdown_connections 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 --2- 172.21.15.129:0/3137404053 >> v2:172.21.15.181:3300/0 conn(0x7f01840cd1c0 0x7f01841609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 --2- 172.21.15.129:0/3137404053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840b4140 0x7f01840ccc80 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 -- 172.21.15.129:0/3137404053 >> 172.21.15.129:0/3137404053 conn(0x7f018406d410 msgr2=0x7f01840b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 -- 172.21.15.129:0/3137404053 shutdown_connections 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.001+0000 7f018c400640 1 -- 172.21.15.129:0/3137404053 wait complete. 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.003+0000 7f018c400640 1 Processor -- start 2024-09-17T04:05:50.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.004+0000 7f018c400640 1 -- start start 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01840b4140 0x7f01840c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01840cd1c0 0x7f01840c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0184160260 con 0x7f01840705a0 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f01841600e0 con 0x7f01840cd1c0 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f018c400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0184160f00 con 0x7f01840b4140 2024-09-17T04:05:50.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f0189a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f0189a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39756/0 (socket says 172.21.15.129:39756) 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.005+0000 7f0189a00640 1 -- 172.21.15.129:0/3842784928 learned_addr learned my addr 172.21.15.129:0/3842784928 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189000640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01840b4140 0x7f01840c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f018a400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01840cd1c0 0x7f01840c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 -- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01840b4140 msgr2=0x7f01840c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01840b4140 0x7f01840c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 -- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01840cd1c0 msgr2=0x7f01840c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01840cd1c0 0x7f01840c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 -- 172.21.15.129:0/3842784928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01840cb3b0 con 0x7f01840705a0 2024-09-17T04:05:50.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0189a00640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178015020 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0178011920 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.006+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0178011d70 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.007+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01840cb690 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.007+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0184159e50 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.008+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01840ccc80 con 0x7f01840705a0 2024-09-17T04:05:50.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.009+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0178018020 con 0x7f01840705a0 2024-09-17T04:05:50.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.010+0000 7f0181600640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 0x7f016407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.010+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7f017809a920 con 0x7f01840705a0 2024-09-17T04:05:50.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.011+0000 7f0189000640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 0x7f016407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.011+0000 7f0189000640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 0x7f016407b0a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0170033000 tx=0x7f0170004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.014+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f017801f030 con 0x7f01840705a0 2024-09-17T04:05:50.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.196+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f018406a140 con 0x7f01840705a0 2024-09-17T04:05:50.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.198+0000 7f0181600640 1 -- 172.21.15.129:0/3842784928 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v222) ==== 74+0+17124 (secure 0 0 0) 0x7f0178062690 con 0x7f01840705a0 2024-09-17T04:05:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:50.198 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":222,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:39.282065+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:05:22.611745+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"217","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-17T04:05:23.660603+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.181:6835/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.129:6834/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6835/711019236":"2024-09-18T04:05:39.001120+0000","172.21.15.129:6832/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6833/711019236":"2024-09-18T04:05:39.001120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:50.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 msgr2=0x7f016407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 0x7f016407b0a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0170033000 tx=0x7f0170004730 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 msgr2=0x7f018415c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 shutdown_connections 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0164078b90 0x7f016407b0a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f01840cd1c0 0x7f01840c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f01840b4140 0x7f01840c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 --2- 172.21.15.129:0/3842784928 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f01840705a0 0x7f018415c630 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 >> 172.21.15.129:0/3842784928 conn(0x7f018406d410 msgr2=0x7f01840ad3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 shutdown_connections 2024-09-17T04:05:50.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.200+0000 7f018c400640 1 -- 172.21.15.129:0/3842784928 wait complete. 2024-09-17T04:05:50.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:05:50.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 -- 172.21.15.129:0/4247051167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d0004800 msgr2=0x7fd2d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 --2- 172.21.15.129:0/4247051167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d0004800 0x7fd2d0004c10 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400c2e0 tx=0x7fd2d0005830 comp rx=0 tx=0).stop 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 -- 172.21.15.129:0/4247051167 shutdown_connections 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 --2- 172.21.15.129:0/4247051167 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2d0004800 0x7fd2d0004c10 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 -- 172.21.15.129:0/4247051167 >> 172.21.15.129:0/4247051167 conn(0x7fd2ec06d410 msgr2=0x7fd2ec0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 -- 172.21.15.129:0/4247051167 shutdown_connections 2024-09-17T04:05:50.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.449+0000 7fd2f1600640 1 -- 172.21.15.129:0/4247051167 wait complete. 2024-09-17T04:05:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.451+0000 7fd2f1600640 1 Processor -- start 2024-09-17T04:05:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.451+0000 7fd2f1600640 1 -- start start 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2d0004800 0x7fd2ec158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2ec06fca0 0x7fd2ec1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec0b4140 con 0x7fd2ec158d40 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ec06f4f0 con 0x7fd2d0004800 2024-09-17T04:05:50.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2f1600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ec0b42c0 con 0x7fd2ec06fca0 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2e2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2e2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39770/0 (socket says 172.21.15.129:39770) 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2e2000640 1 -- 172.21.15.129:0/708973939 learned_addr learned my addr 172.21.15.129:0/708973939 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2eaa00640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2d0004800 0x7fd2ec158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.453+0000 7fd2eb400640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2ec06fca0 0x7fd2ec1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 -- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2ec06fca0 msgr2=0x7fd2ec1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2ec06fca0 0x7fd2ec1540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 -- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2d0004800 msgr2=0x7fd2ec158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2d0004800 0x7fd2ec158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 -- 172.21.15.129:0/708973939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ec0c8ca0 con 0x7fd2ec158d40 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e2000640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400c400 tx=0x7fd2d4001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4031040 con 0x7fd2ec158d40 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2d40065e0 con 0x7fd2ec158d40 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4003a40 con 0x7fd2ec158d40 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ec0c8f80 con 0x7fd2ec158d40 2024-09-17T04:05:50.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.454+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ec0c9540 con 0x7fd2ec158d40 2024-09-17T04:05:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.455+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ec0ccc80 con 0x7fd2ec158d40 2024-09-17T04:05:50.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.456+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd2d403e030 con 0x7fd2ec158d40 2024-09-17T04:05:50.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.457+0000 7fd2e3400640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 0x7fd2c807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.458+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 5967+0+0 (secure 0 0 0) 0x7fd2d40bac30 con 0x7fd2ec158d40 2024-09-17T04:05:50.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.458+0000 7fd2eaa00640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 0x7fd2c807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.459+0000 7fd2eaa00640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 0x7fd2c807b0a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009c10 tx=0x7fd2dc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.462+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd2d4082f60 con 0x7fd2ec158d40 2024-09-17T04:05:50.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.683+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd2ec1f6d30 con 0x7fd2ec158d40 2024-09-17T04:05:50.684 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:05:50.684+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:05:50.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.686+0000 7fd2e3400640 1 -- 172.21.15.129:0/708973939 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v220) ==== 161+0+0 (secure 0 0 0) 0x7fd2d4082860 con 0x7fd2ec158d40 2024-09-17T04:05:50.687 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 msgr2=0x7fd2c807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 0x7fd2c807b0a0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc009c10 tx=0x7fd2dc005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 msgr2=0x7fd2ec153bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fd2d400c400 tx=0x7fd2d4001da0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 shutdown_connections 2024-09-17T04:05:50.690 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd2c8078b90 0x7fd2c807b0a0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2ec06fca0 0x7fd2ec1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd2ec158d40 0x7fd2ec153bb0 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 --2- 172.21.15.129:0/708973939 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd2d0004800 0x7fd2ec158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 >> 172.21.15.129:0/708973939 conn(0x7fd2ec06d410 msgr2=0x7fd2ec159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 shutdown_connections 2024-09-17T04:05:50.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.690+0000 7fd2f1600640 1 -- 172.21.15.129:0/708973939 wait complete. 2024-09-17T04:05:50.695 INFO:tasks.ceph.mds.c.smithi129.stderr: -252> 2024-09-17T04:05:46.298+0000 7f1c5c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:05:50.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:05:50.810 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:05:50.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 -- 172.21.15.129:0/221623424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b740b1c40 msgr2=0x7f7b740b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 --2- 172.21.15.129:0/221623424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b740b1c40 0x7f7b740b2050 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f7b74071e10 tx=0x7f7b6002a530 comp rx=0 tx=0).stop 2024-09-17T04:05:50.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 -- 172.21.15.129:0/221623424 shutdown_connections 2024-09-17T04:05:50.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 --2- 172.21.15.129:0/221623424 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b740b1c40 0x7f7b740b2050 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 --2- 172.21.15.129:0/221623424 >> v2:172.21.15.181:3300/0 conn(0x7f7b74074800 0x7f7b740af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 -- 172.21.15.129:0/221623424 >> 172.21.15.129:0/221623424 conn(0x7f7b740ccc40 msgr2=0x7f7b740cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:50.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 -- 172.21.15.129:0/221623424 shutdown_connections 2024-09-17T04:05:50.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.943+0000 7f7b7a200640 1 -- 172.21.15.129:0/221623424 wait complete. 2024-09-17T04:05:50.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.945+0000 7f7b7a200640 1 Processor -- start 2024-09-17T04:05:50.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.945+0000 7f7b7a200640 1 -- start start 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.946+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b7406eea0 0x7f7b740c4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b7a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 0x7f7b7415e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b7a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b740b4020 con 0x7f7b74074800 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b7a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b740b41a0 con 0x7f7b740b1c40 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b7a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7b740b4320 con 0x7f7b7406eea0 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39804/0 (socket says 172.21.15.129:39804) 2024-09-17T04:05:50.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 -- 172.21.15.129:0/828559154 learned_addr learned my addr 172.21.15.129:0/828559154 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b73400640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b7406eea0 0x7f7b740c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b73e00640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 0x7f7b7415e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 -- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b7406eea0 msgr2=0x7f7b740c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b7406eea0 0x7f7b740c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 -- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 msgr2=0x7f7b7415e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 0x7f7b7415e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b72a00640 1 -- 172.21.15.129:0/828559154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b7409bd20 con 0x7f7b74074800 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.947+0000 7f7b73e00640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 0x7f7b7415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b72a00640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002ab20 tx=0x7f7b60033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60031d60 con 0x7f7b74074800 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b60033e10 con 0x7f7b74074800 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60005260 con 0x7f7b74074800 2024-09-17T04:05:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7409bfc0 con 0x7f7b74074800 2024-09-17T04:05:50.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.948+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7409c580 con 0x7f7b74074800 2024-09-17T04:05:50.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.949+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b740c48f0 con 0x7f7b74074800 2024-09-17T04:05:50.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.950+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7b60005400 con 0x7f7b74074800 2024-09-17T04:05:50.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.951+0000 7f7b6b400640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 0x7f7b5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:50.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.951+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 6053+0+0 (secure 0 0 0) 0x7f7b600bc090 con 0x7f7b74074800 2024-09-17T04:05:50.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.952+0000 7f7b73400640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 0x7f7b5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:50.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.952+0000 7f7b73400640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 0x7f7b5407b0a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7b64005fd0 tx=0x7f7b64005c50 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:50.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:50.956+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7b6003f030 con 0x7f7b74074800 2024-09-17T04:05:51.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.178+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7b7406fba0 con 0x7f7b74074800 2024-09-17T04:05:51.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.688+0000 7f7b6b400640 1 -- 172.21.15.129:0/828559154 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v221) ==== 104+0+0 (secure 0 0 0) 0x7f7b60005d00 con 0x7f7b74074800 2024-09-17T04:05:51.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 msgr2=0x7f7b5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 0x7f7b5407b0a0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7b64005fd0 tx=0x7f7b64005c50 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 msgr2=0x7f7b741596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f7b6002ab20 tx=0x7f7b60033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 shutdown_connections 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b54078b90 0x7f7b5407b0a0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b740b1c40 0x7f7b7415e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b74074800 0x7f7b741596d0 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 --2- 172.21.15.129:0/828559154 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b7406eea0 0x7f7b740c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 >> 172.21.15.129:0/828559154 conn(0x7f7b740ccc40 msgr2=0x7f7b740cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.691+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 shutdown_connections 2024-09-17T04:05:51.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.692+0000 7f7b7a200640 1 -- 172.21.15.129:0/828559154 wait complete. 2024-09-17T04:05:51.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:05:51.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 -- 172.21.15.129:0/104301582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb500b3f70 msgr2=0x7fbb500b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:51.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 --2- 172.21.15.129:0/104301582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb500b3f70 0x7fbb500b49a0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009830 tx=0x7fbb4402a520 comp rx=0 tx=0).stop 2024-09-17T04:05:51.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 -- 172.21.15.129:0/104301582 shutdown_connections 2024-09-17T04:05:51.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 --2- 172.21.15.129:0/104301582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb500b3f70 0x7fbb500b49a0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 -- 172.21.15.129:0/104301582 >> 172.21.15.129:0/104301582 conn(0x7fbb500ccc40 msgr2=0x7fbb500cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:51.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 -- 172.21.15.129:0/104301582 shutdown_connections 2024-09-17T04:05:51.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.947+0000 7fbb58800640 1 -- 172.21.15.129:0/104301582 wait complete. 2024-09-17T04:05:51.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.949+0000 7fbb58800640 1 Processor -- start 2024-09-17T04:05:51.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.949+0000 7fbb58800640 1 -- start start 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb500cba90 0x7fbb500cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 0x7fbb50151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50167f30 con 0x7fbb50074580 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb500cc9b0 con 0x7fbb501676f0 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.950+0000 7fbb58800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbb501680b0 con 0x7fbb500cba90 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39820/0 (socket says 172.21.15.129:39820) 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 -- 172.21.15.129:0/1920037574 learned_addr learned my addr 172.21.15.129:0/1920037574 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb56800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 0x7fbb50151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:51.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 -- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb500cba90 msgr2=0x7fbb500cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb500cba90 0x7fbb500cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 -- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 msgr2=0x7fbb50151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 0x7fbb50151fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 -- 172.21.15.129:0/1920037574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb50151850 con 0x7fbb50074580 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb56800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 0x7fbb50151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.951+0000 7fbb55e00640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009950 tx=0x7fbb44033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.952+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44031d60 con 0x7fbb50074580 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.952+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb44033070 con 0x7fbb50074580 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.952+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb440334c0 con 0x7fbb50074580 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.952+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb501592d0 con 0x7fbb50074580 2024-09-17T04:05:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.952+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb501562e0 con 0x7fbb50074580 2024-09-17T04:05:51.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.953+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb500aef60 con 0x7fbb50074580 2024-09-17T04:05:51.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.954+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbb44033660 con 0x7fbb50074580 2024-09-17T04:05:51.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.955+0000 7fbb4a000640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 0x7fbb3407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.955+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(223..223 src has 1..223) ==== 6053+0+0 (secure 0 0 0) 0x7fbb440bc470 con 0x7fbb50074580 2024-09-17T04:05:51.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.956+0000 7fbb55400640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 0x7fbb3407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:51.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.956+0000 7fbb55400640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 0x7fbb3407b0d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbb40000f60 tx=0x7fbb400079b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:51.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:51.959+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbb500aef60 con 0x7fbb50074580 2024-09-17T04:05:52.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.143+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbb500b3f70 con 0x7fbb50074580 2024-09-17T04:05:52.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.690+0000 7fbb4a000640 1 -- 172.21.15.129:0/1920037574 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v224) ==== 181+0+0 (secure 0 0 0) 0x7fbb4403f030 con 0x7fbb50074580 2024-09-17T04:05:52.691 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 msgr2=0x7fbb3407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 0x7fbb3407b0d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbb40000f60 tx=0x7fbb400079b0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 msgr2=0x7fbb50158bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009950 tx=0x7fbb44033d60 comp rx=0 tx=0).stop 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 shutdown_connections 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb34078bc0 0x7fbb3407b0d0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb501676f0 0x7fbb50151fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb500cba90 0x7fbb500cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 --2- 172.21.15.129:0/1920037574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb50074580 0x7fbb50158bb0 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 >> 172.21.15.129:0/1920037574 conn(0x7fbb500ccc40 msgr2=0x7fbb500afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:52.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 shutdown_connections 2024-09-17T04:05:52.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.694+0000 7fbb58800640 1 -- 172.21.15.129:0/1920037574 wait complete. 2024-09-17T04:05:52.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:05:52.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 -- 172.21.15.129:0/595467458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0b3f70 msgr2=0x7f29bc0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:52.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 --2- 172.21.15.129:0/595467458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0b3f70 0x7f29bc0b49a0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009830 tx=0x7f29b002a480 comp rx=0 tx=0).stop 2024-09-17T04:05:52.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 -- 172.21.15.129:0/595467458 shutdown_connections 2024-09-17T04:05:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 --2- 172.21.15.129:0/595467458 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc0cba90 0x7f29bc0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 --2- 172.21.15.129:0/595467458 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0b3f70 0x7f29bc0b49a0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 -- 172.21.15.129:0/595467458 >> 172.21.15.129:0/595467458 conn(0x7f29bc0ccc40 msgr2=0x7f29bc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 -- 172.21.15.129:0/595467458 shutdown_connections 2024-09-17T04:05:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.943+0000 7f29c4800640 1 -- 172.21.15.129:0/595467458 wait complete. 2024-09-17T04:05:52.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.945+0000 7f29c4800640 1 Processor -- start 2024-09-17T04:05:52.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.945+0000 7f29c4800640 1 -- start start 2024-09-17T04:05:52.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc074580 0x7f29bc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29bc0cba90 0x7f29bc14b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc0cc8e0 con 0x7f29bc0c9c90 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc0cc760 con 0x7f29bc0cba90 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c4800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f29bc166f20 con 0x7f29bc074580 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:39830/0 (socket says 172.21.15.129:39830) 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c2800640 1 -- 172.21.15.129:0/4086004670 learned_addr learned my addr 172.21.15.129:0/4086004670 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c1e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc074580 0x7f29bc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.947+0000 7f29c1400640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29bc0cba90 0x7f29bc14b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 -- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc074580 msgr2=0x7f29bc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:52.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc074580 0x7f29bc158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 -- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29bc0cba90 msgr2=0x7f29bc14b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:52.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29bc0cba90 0x7f29bc14b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:52.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 -- 172.21.15.129:0/4086004670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc070920 con 0x7f29bc0c9c90 2024-09-17T04:05:52.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c2800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e990 tx=0x7f29b800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8004d20 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f29b800caa0 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8005020 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc070640 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.948+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29bc0c8340 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.949+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29bc1f9790 con 0x7f29bc0c9c90 2024-09-17T04:05:52.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.950+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f29b80051c0 con 0x7f29bc0c9c90 2024-09-17T04:05:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.951+0000 7f29b6000640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 0x7f29a007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:52.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.952+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(224..224 src has 1..224) ==== 5615+0+0 (secure 0 0 0) 0x7f29b809b5e0 con 0x7f29bc0c9c90 2024-09-17T04:05:52.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.952+0000 7f29c1e00640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 0x7f29a007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:52.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.952+0000 7f29c1e00640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 0x7f29a007b0a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f29b0034000 tx=0x7f29b0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:52.956+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f29b8016030 con 0x7f29bc0c9c90 2024-09-17T04:05:53.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.139+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f29bc0b3f70 con 0x7f29bc0c9c90 2024-09-17T04:05:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.693+0000 7f29b6000640 1 -- 172.21.15.129:0/4086004670 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v225) ==== 169+0+0 (secure 0 0 0) 0x7f29b80633a0 con 0x7f29bc0c9c90 2024-09-17T04:05:53.693 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:05:53.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 msgr2=0x7f29a007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:53.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 0x7f29a007b0a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f29b0034000 tx=0x7f29b0004730 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 msgr2=0x7f29bc09bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f29b800e990 tx=0x7f29b800ee50 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 shutdown_connections 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f29a0078b90 0x7f29a007b0a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29bc0c9c90 0x7f29bc09bbe0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29bc0cba90 0x7f29bc14b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 --2- 172.21.15.129:0/4086004670 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29bc074580 0x7f29bc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 >> 172.21.15.129:0/4086004670 conn(0x7f29bc0ccc40 msgr2=0x7f29bc0b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 shutdown_connections 2024-09-17T04:05:53.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.696+0000 7f29c4800640 1 -- 172.21.15.129:0/4086004670 wait complete. 2024-09-17T04:05:53.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:05:53.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 -- 172.21.15.129:0/2280455731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac80705a0 msgr2=0x7fdac80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:53.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 --2- 172.21.15.129:0/2280455731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac80705a0 0x7fdac80b4140 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fdab8005fd0 tx=0x7fdab802a530 comp rx=0 tx=0).stop 2024-09-17T04:05:53.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 -- 172.21.15.129:0/2280455731 shutdown_connections 2024-09-17T04:05:53.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 --2- 172.21.15.129:0/2280455731 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdac806eed0 0x7fdac80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 --2- 172.21.15.129:0/2280455731 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac80705a0 0x7fdac80b4140 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 -- 172.21.15.129:0/2280455731 >> 172.21.15.129:0/2280455731 conn(0x7fdac806d410 msgr2=0x7fdac80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:53.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 -- 172.21.15.129:0/2280455731 shutdown_connections 2024-09-17T04:05:53.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.951+0000 7fdacca00640 1 -- 172.21.15.129:0/2280455731 wait complete. 2024-09-17T04:05:53.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.953+0000 7fdacca00640 1 Processor -- start 2024-09-17T04:05:53.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.953+0000 7fdacca00640 1 -- start start 2024-09-17T04:05:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdac806f2e0 0x7fdac815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:53.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdac80705a0 0x7fdac815bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fdac8160200 con 0x7fdac806eed0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fdac81612e0 con 0x7fdac806f2e0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdacca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fdac80cd1c0 con 0x7fdac80705a0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48882/0 (socket says 172.21.15.129:48882) 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 -- 172.21.15.129:0/85223298 learned_addr learned my addr 172.21.15.129:0/85223298 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdabd400640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdac806f2e0 0x7fdac815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 -- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdac80705a0 msgr2=0x7fdac815bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdac80705a0 0x7fdac815bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 -- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdac806f2e0 msgr2=0x7fdac815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdac806f2e0 0x7fdac815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 -- 172.21.15.129:0/85223298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac81f6ac0 con 0x7fdac806eed0 2024-09-17T04:05:53.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.955+0000 7fdac5e00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fdab8034040 tx=0x7fdab80039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:53.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.956+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab803e030 con 0x7fdac806eed0 2024-09-17T04:05:53.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.956+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdab80047a0 con 0x7fdac806eed0 2024-09-17T04:05:53.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.956+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac81f6fc0 con 0x7fdac806eed0 2024-09-17T04:05:53.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.956+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab8035040 con 0x7fdac806eed0 2024-09-17T04:05:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.956+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac81f7580 con 0x7fdac806eed0 2024-09-17T04:05:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.957+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdac80ccc80 con 0x7fdac806eed0 2024-09-17T04:05:53.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.958+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fdab8041030 con 0x7fdac806eed0 2024-09-17T04:05:53.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.959+0000 7fdabea00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 0x7fdaa407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:53.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.959+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7fdab80bb4f0 con 0x7fdac806eed0 2024-09-17T04:05:53.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.960+0000 7fdabd400640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 0x7fdaa407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:53.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.960+0000 7fdabd400640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 0x7fdaa407b0a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdac8073700 tx=0x7fdab0005c20 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:53.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:53.963+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fdab8083b10 con 0x7fdac806eed0 2024-09-17T04:05:54.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.158+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7fdac806a140 con 0x7fdac806eed0 2024-09-17T04:05:54.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.163+0000 7fdabea00640 1 -- 172.21.15.129:0/85223298 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v65) ==== 71+0+0 (secure 0 0 0) 0x7fdab8083410 con 0x7fdac806eed0 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 msgr2=0x7fdaa407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 0x7fdaa407b0a0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdac8073700 tx=0x7fdab0005c20 comp rx=0 tx=0).stop 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 msgr2=0x7fdac80b0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7fdab8034040 tx=0x7fdab80039e0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 shutdown_connections 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fdaa4078b90 0x7fdaa407b0a0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fdac80705a0 0x7fdac815bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdac806f2e0 0x7fdac815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 --2- 172.21.15.129:0/85223298 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fdac806eed0 0x7fdac80b0f80 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 >> 172.21.15.129:0/85223298 conn(0x7fdac806d410 msgr2=0x7fdac8073970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.165+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 shutdown_connections 2024-09-17T04:05:54.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.166+0000 7fdacca00640 1 -- 172.21.15.129:0/85223298 wait complete. 2024-09-17T04:05:54.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade' 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 -- 172.21.15.129:0/4232421882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d00b4180 msgr2=0x7fa0d00ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 --2- 172.21.15.129:0/4232421882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d00b4180 0x7fa0d00ce170 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00c3cf0 tx=0x7fa0c802aac0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 -- 172.21.15.129:0/4232421882 shutdown_connections 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 --2- 172.21.15.129:0/4232421882 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d00b4180 0x7fa0d00ce170 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 --2- 172.21.15.129:0/4232421882 >> v2:172.21.15.181:3300/0 conn(0x7fa0d00cd390 0x7fa0d00af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 -- 172.21.15.129:0/4232421882 >> 172.21.15.129:0/4232421882 conn(0x7fa0d006d410 msgr2=0x7fa0d00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 -- 172.21.15.129:0/4232421882 shutdown_connections 2024-09-17T04:05:54.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.407+0000 7fa0d6800640 1 -- 172.21.15.129:0/4232421882 wait complete. 2024-09-17T04:05:54.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.410+0000 7fa0d6800640 1 Processor -- start 2024-09-17T04:05:54.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.410+0000 7fa0d6800640 1 -- start start 2024-09-17T04:05:54.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d00b4180 0x7fa0d0099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0d0074810 0x7fa0d01f4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d006f8d0 con 0x7fa0d015ce50 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d006f750 con 0x7fa0d00b4180 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.411+0000 7fa0d6800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d006fa50 con 0x7fa0d0074810 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48894/0 (socket says 172.21.15.129:48894) 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 -- 172.21.15.129:0/1078595908 learned_addr learned my addr 172.21.15.129:0/1078595908 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0cf400640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d00b4180 0x7fa0d0099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0cfe00640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0d0074810 0x7fa0d01f4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 -- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0d0074810 msgr2=0x7fa0d01f4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0d0074810 0x7fa0d01f4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 -- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d00b4180 msgr2=0x7fa0d0099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d00b4180 0x7fa0d0099ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 -- 172.21.15.129:0/1078595908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d01f3780 con 0x7fa0d015ce50 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.412+0000 7fa0c6a00640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800d770 tx=0x7fa0b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.413+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8017070 con 0x7fa0d015ce50 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.413+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa0b8009d20 con 0x7fa0d015ce50 2024-09-17T04:05:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.413+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8004280 con 0x7fa0d015ce50 2024-09-17T04:05:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.413+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01f3a20 con 0x7fa0d015ce50 2024-09-17T04:05:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.413+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d015cba0 con 0x7fa0d015ce50 2024-09-17T04:05:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.414+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0d006ff00 con 0x7fa0d015ce50 2024-09-17T04:05:54.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.415+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa0b8005020 con 0x7fa0d015ce50 2024-09-17T04:05:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.416+0000 7fa0c7e00640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.416+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7fa0b809ae70 con 0x7fa0d015ce50 2024-09-17T04:05:54.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.416+0000 7fa0cf400640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.417+0000 7fa0cf400640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0001720 tx=0x7fa0c0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:54.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.421+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa0b8020020 con 0x7fa0d015ce50 2024-09-17T04:05:54.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.675+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]} v 0) -- 0x7fa0d006a140 con 0x7fa0d015ce50 2024-09-17T04:05:54.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.708+0000 7fa0c7e00640 1 -- 172.21.15.129:0/1078595908 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_purge_queue_upgrade"]}]=0 v827) ==== 139+0+0 (secure 0 0 0) 0x7fa0b8062e40 con 0x7fa0d015ce50 2024-09-17T04:05:54.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 msgr2=0x7fa0ac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa0c0001720 tx=0x7fa0c0009040 comp rx=0 tx=0).stop 2024-09-17T04:05:54.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 msgr2=0x7fa0d01f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800d770 tx=0x7fa0b800dc30 comp rx=0 tx=0).stop 2024-09-17T04:05:54.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 shutdown_connections 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa0ac078bc0 0x7fa0ac07b0d0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa0d0074810 0x7fa0d01f4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa0d015ce50 0x7fa0d01f4830 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 --2- 172.21.15.129:0/1078595908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa0d00b4180 0x7fa0d0099ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 >> 172.21.15.129:0/1078595908 conn(0x7fa0d006d410 msgr2=0x7fa0d01e5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 shutdown_connections 2024-09-17T04:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.711+0000 7fa0d6800640 1 -- 172.21.15.129:0/1078595908 wait complete. 2024-09-17T04:05:54.734 INFO:tasks.cephfs_test_runner:That when starting on a system with no purge queue in the metadata ... ok 2024-09-17T04:05:54.734 INFO:tasks.cephfs_test_runner:Starting test: test_reintegration_limit (tasks.cephfs.test_strays.TestStrays) That the reintegration is not blocked by full directories. 2024-09-17T04:05:54.734 INFO:tasks.cephfs_test_runner:test_reintegration_limit (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:05:54.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 -- 172.21.15.129:0/3541479253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0b46e0 msgr2=0x7f32cc0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 --2- 172.21.15.129:0/3541479253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0b46e0 0x7f32cc0b4af0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f32cc0727e0 tx=0x7f32bc02a4d0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 -- 172.21.15.129:0/3541479253 shutdown_connections 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 --2- 172.21.15.129:0/3541479253 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0d02c0 0x7f32cc0d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 --2- 172.21.15.129:0/3541479253 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0b46e0 0x7f32cc0b4af0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 --2- 172.21.15.129:0/3541479253 >> v2:172.21.15.181:3300/0 conn(0x7f32cc0cfb60 0x7f32cc160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 -- 172.21.15.129:0/3541479253 >> 172.21.15.129:0/3541479253 conn(0x7f32cc06cfc0 msgr2=0x7f32cc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 -- 172.21.15.129:0/3541479253 shutdown_connections 2024-09-17T04:05:54.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.957+0000 7f32d3200640 1 -- 172.21.15.129:0/3541479253 wait complete. 2024-09-17T04:05:54.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.959+0000 7f32d3200640 1 Processor -- start 2024-09-17T04:05:54.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.960+0000 7f32d3200640 1 -- start start 2024-09-17T04:05:54.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32cc0b46e0 0x7f32cc1e9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0cfb60 0x7f32cc156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc15d040 con 0x7f32cc0d02c0 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc15cec0 con 0x7f32cc0b46e0 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f32cc15d1c0 con 0x7f32cc0cfb60 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.961+0000 7f32d2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48906/0 (socket says 172.21.15.129:48906) 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 -- 172.21.15.129:0/4172188629 learned_addr learned my addr 172.21.15.129:0/4172188629 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d1e00640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32cc0b46e0 0x7f32cc1e9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d1400640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0cfb60 0x7f32cc156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 -- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0cfb60 msgr2=0x7f32cc156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0cfb60 0x7f32cc156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 -- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32cc0b46e0 msgr2=0x7f32cc1e9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32cc0b46e0 0x7f32cc1e9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 -- 172.21.15.129:0/4172188629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc155af0 con 0x7f32cc0d02c0 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.962+0000 7f32d2800640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f32c800eab0 tx=0x7f32c800ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.963+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c800cea0 con 0x7f32cc0d02c0 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.963+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f32c8004b20 con 0x7f32cc0d02c0 2024-09-17T04:05:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.963+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8005020 con 0x7f32cc0d02c0 2024-09-17T04:05:54.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.963+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc155d90 con 0x7f32cc0d02c0 2024-09-17T04:05:54.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.963+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc156350 con 0x7f32cc0d02c0 2024-09-17T04:05:54.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.964+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32cc1e9b10 con 0x7f32cc0d02c0 2024-09-17T04:05:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.965+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f32c80051c0 con 0x7f32cc0d02c0 2024-09-17T04:05:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.966+0000 7f32c6000640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 0x7f32b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.966+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f32c809b680 con 0x7f32cc0d02c0 2024-09-17T04:05:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.966+0000 7f32d1e00640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 0x7f32b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.967+0000 7f32d1e00640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 0x7f32b007b0a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f32cc0b4450 tx=0x7f32c0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:54.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:54.970+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f32c80635c0 con 0x7f32cc0d02c0 2024-09-17T04:05:55.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.155+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32cc06a140 con 0x7f32cc0d02c0 2024-09-17T04:05:55.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.156+0000 7f32c6000640 1 -- 172.21.15.129:0/4172188629 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) ==== 74+0+13486 (secure 0 0 0) 0x7f32c8067ff0 con 0x7f32cc0d02c0 2024-09-17T04:05:55.186 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:55.186 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":225,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:05:53.691931+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":216,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/3706217946":"2024-09-18T04:05:50.685417+0000","172.21.15.181:6837/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.181:6835/2764153925":"2024-09-18T04:05:39.282040+0000","172.21.15.129:6834/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6835/711019236":"2024-09-18T04:05:39.001120+0000","172.21.15.129:6835/3706217946":"2024-09-18T04:05:50.685417+0000","172.21.15.129:6832/3249193545":"2024-09-18T04:05:39.016692+0000","172.21.15.129:6833/711019236":"2024-09-18T04:05:39.001120+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 msgr2=0x7f32b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 0x7f32b007b0a0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f32cc0b4450 tx=0x7f32c0005f90 comp rx=0 tx=0).stop 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 msgr2=0x7f32cc1e76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f32c800eab0 tx=0x7f32c800ef70 comp rx=0 tx=0).stop 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 shutdown_connections 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f32b0078b90 0x7f32b007b0a0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f32cc0d02c0 0x7f32cc1e76c0 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f32cc0cfb60 0x7f32cc156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 --2- 172.21.15.129:0/4172188629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32cc0b46e0 0x7f32cc1e9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 >> 172.21.15.129:0/4172188629 conn(0x7f32cc06cfc0 msgr2=0x7f32cc15f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 shutdown_connections 2024-09-17T04:05:55.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.159+0000 7f32d3200640 1 -- 172.21.15.129:0/4172188629 wait complete. 2024-09-17T04:05:55.188 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:05:55.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:05:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 -- 172.21.15.129:0/1692598986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 msgr2=0x7f2be80cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 --2- 172.21.15.129:0/1692598986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80cbea0 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7f2be80d1260 tx=0x7f2be002a820 comp rx=0 tx=0).stop 2024-09-17T04:05:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 -- 172.21.15.129:0/1692598986 shutdown_connections 2024-09-17T04:05:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 --2- 172.21.15.129:0/1692598986 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80cbea0 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 --2- 172.21.15.129:0/1692598986 >> v2:172.21.15.181:3300/0 conn(0x7f2be80b3f70 0x7f2be80b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 -- 172.21.15.129:0/1692598986 >> 172.21.15.129:0/1692598986 conn(0x7f2be80ccc40 msgr2=0x7f2be80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 -- 172.21.15.129:0/1692598986 shutdown_connections 2024-09-17T04:05:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.412+0000 7f2beee00640 1 -- 172.21.15.129:0/1692598986 wait complete. 2024-09-17T04:05:55.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.415+0000 7f2beee00640 1 Processor -- start 2024-09-17T04:05:55.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.415+0000 7f2beee00640 1 -- start start 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be8074580 0x7f2be809d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 0x7f2be80ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80cc880 con 0x7f2be80cba90 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80cc700 con 0x7f2be81590b0 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2beee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80cca00 con 0x7f2be8074580 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.416+0000 7f2be7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48928/0 (socket says 172.21.15.129:48928) 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 -- 172.21.15.129:0/875982668 learned_addr learned my addr 172.21.15.129:0/875982668 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:55.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2bece00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 0x7f2be80ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7e00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be8074580 0x7f2be809d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 -- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be8074580 msgr2=0x7f2be809d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be8074580 0x7f2be809d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 -- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 msgr2=0x7f2be80ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 0x7f2be80ae540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 -- 172.21.15.129:0/875982668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be815bef0 con 0x7f2be80cba90 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2bece00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 0x7f2be80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2be7400640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c460 tx=0x7f2bd400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4016020 con 0x7f2be80cba90 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2bd4012940 con 0x7f2be80cba90 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4012d90 con 0x7f2be80cba90 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.417+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be815c1d0 con 0x7f2be80cba90 2024-09-17T04:05:55.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.418+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be8144390 con 0x7f2be80cba90 2024-09-17T04:05:55.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.419+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be8159550 con 0x7f2be80cba90 2024-09-17T04:05:55.419 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.420+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2bd4019020 con 0x7f2be80cba90 2024-09-17T04:05:55.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.421+0000 7f2bdfe00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 0x7f2bc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.420 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.422+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f2bd409aaa0 con 0x7f2be80cba90 2024-09-17T04:05:55.421 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.422+0000 7f2be7e00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 0x7f2bc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.423+0000 7f2be7e00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 0x7f2bc807b0a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8005fd0 tx=0x7f2bd8005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:55.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.426+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2bd4020020 con 0x7f2be80cba90 2024-09-17T04:05:55.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.607+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2be80b3f70 con 0x7f2be80cba90 2024-09-17T04:05:55.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.607+0000 7f2bdfe00640 1 -- 172.21.15.129:0/875982668 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) ==== 93+0+10 (secure 0 0 0) 0x7f2bd40630c0 con 0x7f2be80cba90 2024-09-17T04:05:55.607 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:05:55.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 msgr2=0x7f2bc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 0x7f2bc807b0a0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8005fd0 tx=0x7f2bd8005850 comp rx=0 tx=0).stop 2024-09-17T04:05:55.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 msgr2=0x7f2be80ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f2bd400c460 tx=0x7f2bd400c920 comp rx=0 tx=0).stop 2024-09-17T04:05:55.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 shutdown_connections 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2bc8078b90 0x7f2bc807b0a0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2be81590b0 0x7f2be80ae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2be80cba90 0x7f2be80ae000 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 --2- 172.21.15.129:0/875982668 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2be8074580 0x7f2be809d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 >> 172.21.15.129:0/875982668 conn(0x7f2be80ccc40 msgr2=0x7f2be80704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 shutdown_connections 2024-09-17T04:05:55.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.610+0000 7f2beee00640 1 -- 172.21.15.129:0/875982668 wait complete. 2024-09-17T04:05:55.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit' 2024-09-17T04:05:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 -- 172.21.15.129:0/3112115393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 msgr2=0x7f14680b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 --2- 172.21.15.129:0/3112115393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680b4140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1458009830 tx=0x7f145802a510 comp rx=0 tx=0).stop 2024-09-17T04:05:55.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 -- 172.21.15.129:0/3112115393 shutdown_connections 2024-09-17T04:05:55.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 --2- 172.21.15.129:0/3112115393 >> v2:172.21.15.181:3300/0 conn(0x7f146806eed0 0x7f14680ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 --2- 172.21.15.129:0/3112115393 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680b4140 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 -- 172.21.15.129:0/3112115393 >> 172.21.15.129:0/3112115393 conn(0x7f146806d410 msgr2=0x7f14680b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:55.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 -- 172.21.15.129:0/3112115393 shutdown_connections 2024-09-17T04:05:55.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.858+0000 7f146f200640 1 -- 172.21.15.129:0/3112115393 wait complete. 2024-09-17T04:05:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.860+0000 7f146f200640 1 Processor -- start 2024-09-17T04:05:55.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.860+0000 7f146f200640 1 -- start start 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f146806eed0 0x7f14681f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1468160200 con 0x7f146806f2e0 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14681612e0 con 0x7f146806eed0 2024-09-17T04:05:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14680cd1c0 con 0x7f14680705a0 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48938/0 (socket says 172.21.15.129:48938) 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 -- 172.21.15.129:0/1984766477 learned_addr learned my addr 172.21.15.129:0/1984766477 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467e00640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f146806eed0 0x7f14681f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146d200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 -- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 msgr2=0x7f14680c64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680c64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 -- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f146806eed0 msgr2=0x7f14681f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f146806eed0 0x7f14681f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f1467400640 1 -- 172.21.15.129:0/1984766477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14680c78e0 con 0x7f146806f2e0 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.862+0000 7f146d200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f1467400640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f145400e790 tx=0x7f145400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f145400cac0 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f145400cc60 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14680c7bc0 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454005280 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.863+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14680c8180 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.864+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14680ccc80 con 0x7f146806f2e0 2024-09-17T04:05:55.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.865+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1454005420 con 0x7f146806f2e0 2024-09-17T04:05:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.866+0000 7f145fe00640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 0x7f144807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.866+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f145409b4d0 con 0x7f146806f2e0 2024-09-17T04:05:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.867+0000 7f1467e00640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 0x7f144807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:55.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.868+0000 7f1467e00640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 0x7f144807b0d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1458034000 tx=0x7f1458004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:55.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:55.870+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1454063ae0 con 0x7f146806f2e0 2024-09-17T04:05:56.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.120+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit"]} v 0) -- 0x7f146806a140 con 0x7f146806f2e0 2024-09-17T04:05:56.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.716+0000 7f145fe00640 1 -- 172.21.15.129:0/1984766477 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit"]}]=0 v829) ==== 142+0+0 (secure 0 0 0) 0x7f14540633e0 con 0x7f146806f2e0 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 msgr2=0x7f144807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 0x7f144807b0d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f1458034000 tx=0x7f1458004730 comp rx=0 tx=0).stop 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 msgr2=0x7f14681560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f145400e790 tx=0x7f145400ec50 comp rx=0 tx=0).stop 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 shutdown_connections 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1448078bc0 0x7f144807b0d0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14680705a0 0x7f14680c64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f146806f2e0 0x7f14681560a0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 --2- 172.21.15.129:0/1984766477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f146806eed0 0x7f14681f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 >> 172.21.15.129:0/1984766477 conn(0x7f146806d410 msgr2=0x7f14680737d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 shutdown_connections 2024-09-17T04:05:56.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.720+0000 7f146f200640 1 -- 172.21.15.129:0/1984766477 wait complete. 2024-09-17T04:05:56.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 -- 172.21.15.129:0/2020957994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 msgr2=0x7f68cc15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 --2- 172.21.15.129:0/2020957994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc15ae00 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0c72d0 tx=0x7f68c002a530 comp rx=0 tx=0).stop 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 -- 172.21.15.129:0/2020957994 shutdown_connections 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 --2- 172.21.15.129:0/2020957994 >> v2:172.21.15.181:3300/0 conn(0x7f68cc0cd390 0x7f68cc06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 --2- 172.21.15.129:0/2020957994 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc15ae00 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 -- 172.21.15.129:0/2020957994 >> 172.21.15.129:0/2020957994 conn(0x7f68cc06d410 msgr2=0x7f68cc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 -- 172.21.15.129:0/2020957994 shutdown_connections 2024-09-17T04:05:56.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.962+0000 7f68d3600640 1 -- 172.21.15.129:0/2020957994 wait complete. 2024-09-17T04:05:56.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.965+0000 7f68d3600640 1 Processor -- start 2024-09-17T04:05:56.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.965+0000 7f68d3600640 1 -- start start 2024-09-17T04:05:56.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:56.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68cc15b340 0x7f68cc15a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 0x7f68cc09c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc06fb80 con 0x7f68cc0af4c0 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc071170 con 0x7f68cc1f66b0 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d3600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f68cc06fd00 con 0x7f68cc15b340 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48950/0 (socket says 172.21.15.129:48950) 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.966+0000 7f68d0c00640 1 -- 172.21.15.129:0/3283458806 learned_addr learned my addr 172.21.15.129:0/3283458806 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d1600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 0x7f68cc09c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68cbe00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68cc15b340 0x7f68cc15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 -- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68cc15b340 msgr2=0x7f68cc15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68cc15b340 0x7f68cc15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 -- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 msgr2=0x7f68cc09c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:56.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 0x7f68cc09c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 -- 172.21.15.129:0/3283458806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc1f7150 con 0x7f68cc0af4c0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d1600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 0x7f68cc09c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d0c00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f68c002ab20 tx=0x7f68c0033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0005c90 con 0x7f68cc0af4c0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f68c0031e50 con 0x7f68cc0af4c0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0005260 con 0x7f68cc0af4c0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.967+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc1f7430 con 0x7f68cc0af4c0 2024-09-17T04:05:56.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.968+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc09cb20 con 0x7f68cc0af4c0 2024-09-17T04:05:56.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.969+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc09cd80 con 0x7f68cc0af4c0 2024-09-17T04:05:56.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.970+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f68c0005400 con 0x7f68cc0af4c0 2024-09-17T04:05:56.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.971+0000 7f68c8c00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 0x7f68ac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:56.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.971+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f68c00bbbd0 con 0x7f68cc0af4c0 2024-09-17T04:05:56.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.971+0000 7f68cbe00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 0x7f68ac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:56.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.972+0000 7f68cbe00640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 0x7f68ac07b0d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f68cc15ef40 tx=0x7f68b8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:56.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:56.975+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f68c003f030 con 0x7f68cc0af4c0 2024-09-17T04:05:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.157+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f68cc15ae00 con 0x7f68cc0af4c0 2024-09-17T04:05:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.158+0000 7f68c8c00640 1 -- 172.21.15.129:0/3283458806 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v65) ==== 59+0+2294 (secure 0 0 0) 0x7f68c00841f0 con 0x7f68cc0af4c0 2024-09-17T04:05:57.157 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:05:57.157 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:05:57.157 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:05:57.157 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:05:57.158 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:05:57.159 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:05:57.160 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 msgr2=0x7f68ac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 0x7f68ac07b0d0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f68cc15ef40 tx=0x7f68b8009040 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 msgr2=0x7f68cc1f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f68c002ab20 tx=0x7f68c0033c20 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 shutdown_connections 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f68ac078bc0 0x7f68ac07b0d0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68cc1f66b0 0x7f68cc09c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68cc15b340 0x7f68cc15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 --2- 172.21.15.129:0/3283458806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68cc0af4c0 0x7f68cc1f3bd0 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 >> 172.21.15.129:0/3283458806 conn(0x7f68cc06d410 msgr2=0x7f68cc0cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 shutdown_connections 2024-09-17T04:05:57.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.160+0000 7f68d3600640 1 -- 172.21.15.129:0/3283458806 wait complete. 2024-09-17T04:05:57.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:05:57.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 -- 172.21.15.129:0/1606102852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc06eea0 msgr2=0x7f9adc06f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 --2- 172.21.15.129:0/1606102852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc06eea0 0x7f9adc06f2b0 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f9adc069b60 tx=0x7f9ad402aa30 comp rx=0 tx=0).stop 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 -- 172.21.15.129:0/1606102852 shutdown_connections 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 --2- 172.21.15.129:0/1606102852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc06eea0 0x7f9adc06f2b0 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 --2- 172.21.15.129:0/1606102852 >> v2:172.21.15.181:3300/0 conn(0x7f9adc0b1c40 0x7f9adc0b2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 --2- 172.21.15.129:0/1606102852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc074800 0x7f9adc0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 -- 172.21.15.129:0/1606102852 >> 172.21.15.129:0/1606102852 conn(0x7f9adc0ccc40 msgr2=0x7f9adc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 -- 172.21.15.129:0/1606102852 shutdown_connections 2024-09-17T04:05:57.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.406+0000 7f9ae2400640 1 -- 172.21.15.129:0/1606102852 wait complete. 2024-09-17T04:05:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.408+0000 7f9ae2400640 1 Processor -- start 2024-09-17T04:05:57.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.408+0000 7f9ae2400640 1 -- start start 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.409+0000 7f9ae2400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 0x7f9adc0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.409+0000 7f9ae2400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ae2400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 0x7f9adc1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ae2400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc06f970 con 0x7f9adc074800 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ae2400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9adc06f7f0 con 0x7f9adc06eea0 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ae2400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9adc06faf0 con 0x7f9adc0b1c40 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48976/0 (socket says 172.21.15.129:48976) 2024-09-17T04:05:57.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 -- 172.21.15.129:0/3499844157 learned_addr learned my addr 172.21.15.129:0/3499844157 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9adbe00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 0x7f9adc1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9adb400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 0x7f9adc0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 -- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 msgr2=0x7f9adc1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 0x7f9adc1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 -- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 msgr2=0x7f9adc0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 0x7f9adc0ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 -- 172.21.15.129:0/3499844157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9adc0cee60 con 0x7f9adc074800 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9adbe00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 0x7f9adc1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9adb400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 0x7f9adc0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.410+0000 7f9ad2a00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e7c0 tx=0x7f9ac400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.411+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400ccc0 con 0x7f9adc074800 2024-09-17T04:05:57.411 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.411+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ac400ce60 con 0x7f9adc074800 2024-09-17T04:05:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.411+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4005280 con 0x7f9adc074800 2024-09-17T04:05:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.411+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9adc0cf140 con 0x7f9adc074800 2024-09-17T04:05:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.411+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9adc0cf700 con 0x7f9adc074800 2024-09-17T04:05:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.412+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9adc0cebd0 con 0x7f9adc074800 2024-09-17T04:05:57.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.413+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9ac4005420 con 0x7f9adc074800 2024-09-17T04:05:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.414+0000 7f9ad3e00640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 0x7f9ab807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.415+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f9ac409b450 con 0x7f9adc074800 2024-09-17T04:05:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.415+0000 7f9adb400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 0x7f9ab807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:57.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.416+0000 7f9adb400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 0x7f9ab807b0a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9acc005d00 tx=0x7f9acc005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:57.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.419+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9ac4063a70 con 0x7f9adc074800 2024-09-17T04:05:57.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.602+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f9adc0af480 con 0x7f9adc074800 2024-09-17T04:05:57.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.606+0000 7f9ad3e00640 1 -- 172.21.15.129:0/3499844157 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v66)=0 v66) ==== 122+0+0 (secure 0 0 0) 0x7f9ac4063370 con 0x7f9adc074800 2024-09-17T04:05:57.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 msgr2=0x7f9ab807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 0x7f9ab807b0a0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9acc005d00 tx=0x7f9acc005db0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 msgr2=0x7f9adc09af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:57.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400e7c0 tx=0x7f9ac400ec80 comp rx=0 tx=0).stop 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 shutdown_connections 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ab8078b90 0x7f9ab807b0a0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9adc0b1c40 0x7f9adc1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9adc074800 0x7f9adc09af70 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 --2- 172.21.15.129:0/3499844157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9adc06eea0 0x7f9adc0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 >> 172.21.15.129:0/3499844157 conn(0x7f9adc0ccc40 msgr2=0x7f9adc15c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 shutdown_connections 2024-09-17T04:05:57.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:57.609+0000 7f9ae2400640 1 -- 172.21.15.129:0/3499844157 wait complete. 2024-09-17T04:05:57.630 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:05:57.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:57.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:05:57.822 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:05:57.822 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:57.822 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:57.858 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:57.858 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:57.858 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:57.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:57.895 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:05:57.895 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:57.933 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:05:57.933 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:05:57.934 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:05:57.972 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:05:57.972 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:05:57.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:05:58.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.195+0000 7f8752000640 1 -- 172.21.15.129:0/722249925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8730003a00 msgr2=0x7f8730003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.195+0000 7f8752000640 1 --2- 172.21.15.129:0/722249925 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8730003a00 0x7f8730003e70 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7f873c009b50 tx=0x7f8730005830 comp rx=0 tx=0).stop 2024-09-17T04:05:58.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.196+0000 7f874be00640 1 -- 172.21.15.129:0/722249925 reap_dead start 2024-09-17T04:05:58.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.196+0000 7f8752000640 1 -- 172.21.15.129:0/722249925 shutdown_connections 2024-09-17T04:05:58.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.196+0000 7f8752000640 1 -- 172.21.15.129:0/722249925 >> 172.21.15.129:0/722249925 conn(0x7f874c06d410 msgr2=0x7f874c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:58.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.196+0000 7f8752000640 1 -- 172.21.15.129:0/722249925 shutdown_connections 2024-09-17T04:05:58.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.196+0000 7f8752000640 1 -- 172.21.15.129:0/722249925 wait complete. 2024-09-17T04:05:58.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.198+0000 7f8752000640 1 Processor -- start 2024-09-17T04:05:58.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.198+0000 7f8752000640 1 -- start start 2024-09-17T04:05:58.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.199+0000 7f8752000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8730003a00 0x7f874c1f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f8752000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f874c099a30 0x7f874c0b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f8752000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f8752000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f874c0b3b90 con 0x7f874c0b0e10 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f8752000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f874c0708e0 con 0x7f8730003a00 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f8752000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f874c0b3d10 con 0x7f874c099a30 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48994/0 (socket says 172.21.15.129:48994) 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 -- 172.21.15.129:0/1613472436 learned_addr learned my addr 172.21.15.129:0/1613472436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874aa00640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f874c099a30 0x7f874c0b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874b400640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8730003a00 0x7f874c1f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 -- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f874c099a30 msgr2=0x7f874c0b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f874c099a30 0x7f874c0b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 -- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8730003a00 msgr2=0x7f874c1f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8730003a00 0x7f874c1f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.200+0000 7f874be00640 1 -- 172.21.15.129:0/1613472436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c1f57a0 con 0x7f874c0b0e10 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f874be00640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f874c072e20 tx=0x7f874400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744004b20 con 0x7f874c0b0e10 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8744004cc0 con 0x7f874c0b0e10 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874c0afdf0 con 0x7f874c0b0e10 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8744003bd0 con 0x7f874c0b0e10 2024-09-17T04:05:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.201+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874c15bed0 con 0x7f874c0b0e10 2024-09-17T04:05:58.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.202+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874c159ab0 con 0x7f874c0b0e10 2024-09-17T04:05:58.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.204+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8744004030 con 0x7f874c0b0e10 2024-09-17T04:05:58.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.205+0000 7f8743400640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 0x7f872c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.205+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f874409b330 con 0x7f874c0b0e10 2024-09-17T04:05:58.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.205+0000 7f874b400640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 0x7f872c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.206+0000 7f874b400640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 0x7f872c07b0a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f873c035000 tx=0x7f873c02d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:58.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.209+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8744063300 con 0x7f874c0b0e10 2024-09-17T04:05:58.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.428+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f874c15a8a0 con 0x7f874c0b0e10 2024-09-17T04:05:58.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.429+0000 7f8743400640 1 -- 172.21.15.129:0/1613472436 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+3931 (secure 0 0 0) 0x7f874c15a8a0 con 0x7f874c0b0e10 2024-09-17T04:05:58.428 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:05:58.428 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":221,"btime":"2024-09-17T04:05:51:687858+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8822,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1634503697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":1634503697},{"type":"v1","addr":"172.21.15.181:6836","nonce":1634503697}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10384,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6833/1394473387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1394473387},{"type":"v1","addr":"172.21.15.129:6833","nonce":1394473387}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":10390,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2440348812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2440348812},{"type":"v1","addr":"172.21.15.181:6837","nonce":2440348812}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":214},{"gid":11361,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/3755044684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6834","nonce":3755044684},{"type":"v1","addr":"172.21.15.129:6835","nonce":3755044684}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":221}],"filesystems":[]} 2024-09-17T04:05:58.428 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 221 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.431+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 msgr2=0x7f872c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.431+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 0x7f872c07b0a0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f873c035000 tx=0x7f873c02d040 comp rx=0 tx=0).stop 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.431+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 msgr2=0x7f874c0b1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.431+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f874c072e20 tx=0x7f874400f3c0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 shutdown_connections 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f872c078b90 0x7f872c07b0a0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f874c0b0e10 0x7f874c0b1220 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f874c099a30 0x7f874c0b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 --2- 172.21.15.129:0/1613472436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8730003a00 0x7f874c1f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 >> 172.21.15.129:0/1613472436 conn(0x7f874c06d410 msgr2=0x7f874c0af100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:58.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 shutdown_connections 2024-09-17T04:05:58.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.432+0000 7f8752000640 1 -- 172.21.15.129:0/1613472436 wait complete. 2024-09-17T04:05:58.454 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:05:58.454 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:05:58.454 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:05:58.454 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:58.455 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:05:58.455 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:05:58.455 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:05:58.455 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:58.455 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:05:58.455 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:05:58.455 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:05:58.456 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:58.456 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:05:58.456 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:05:58.456 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:05:58.456 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:05:58.479 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:05:58.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:05:58.480 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:05:58.481 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:05:58.485 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:05:58.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:05:58.487 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:05:58.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:05:58.524 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:05:58.525 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:58.527 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:05:58.527 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:58.531 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:05:58.532 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:58.535 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:05:58.535 DEBUG:teuthology.parallel:result is None 2024-09-17T04:05:58.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:05:58.725 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:05:58.725 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:05:58.726 INFO:tasks.ceph.mds.b.smithi181.stderr:2024-09-17T04:05:58.727+0000 7f3abaa00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-17T04:05:58.730 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:05:58.731 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 -- 172.21.15.129:0/2283931694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f500b2950 msgr2=0x7f9f500cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 --2- 172.21.15.129:0/2283931694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f500b2950 0x7f9f500cbbf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9f48009be0 tx=0x7f9f4802a920 comp rx=0 tx=0).stop 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 -- 172.21.15.129:0/2283931694 shutdown_connections 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 --2- 172.21.15.129:0/2283931694 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f500b2950 0x7f9f500cbbf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 -- 172.21.15.129:0/2283931694 >> 172.21.15.129:0/2283931694 conn(0x7f9f5006d410 msgr2=0x7f9f500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 -- 172.21.15.129:0/2283931694 shutdown_connections 2024-09-17T04:05:58.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.770+0000 7f9f55600640 1 -- 172.21.15.129:0/2283931694 wait complete. 2024-09-17T04:05:58.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.773+0000 7f9f55600640 1 Processor -- start 2024-09-17T04:05:58.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.773+0000 7f9f55600640 1 -- start start 2024-09-17T04:05:58.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 0x7f9f50099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f501f5220 0x7f9f501e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5006f760 con 0x7f9f500ce170 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9f5006f5e0 con 0x7f9f500b2950 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f55600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9f5006f8e0 con 0x7f9f501f5220 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.774+0000 7f9f4e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49030/0 (socket says 172.21.15.129:49030) 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4e000640 1 -- 172.21.15.129:0/3418223821 learned_addr learned my addr 172.21.15.129:0/3418223821 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 0x7f9f50099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4f400640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f501f5220 0x7f9f501e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 -- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f501f5220 msgr2=0x7f9f501e45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f501f5220 0x7f9f501e45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 -- 172.21.15.129:0/3418223821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 msgr2=0x7f9f501e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 -- 172.21.15.129:0/3418223821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f50099ed0 con 0x7f9f500b2950 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4e000640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.775+0000 7f9f4ea00640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 0x7f9f50099990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000c3f0 tx=0x7f9f4000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.776+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f400052d0 con 0x7f9f500b2950 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.776+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f40007020 con 0x7f9f500b2950 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.776+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f40007470 con 0x7f9f500b2950 2024-09-17T04:05:58.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.776+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f501f5900 con 0x7f9f500b2950 2024-09-17T04:05:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.776+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f501f5ec0 con 0x7f9f500b2950 2024-09-17T04:05:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.777+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f500cbbf0 con 0x7f9f500b2950 2024-09-17T04:05:58.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.778+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9f400067d0 con 0x7f9f500b2950 2024-09-17T04:05:58.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.779+0000 7f9f46a00640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 0x7f9f3007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:05:58.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.779+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(225..225 src has 1..225) ==== 5201+0+0 (secure 0 0 0) 0x7f9f4009a0b0 con 0x7f9f500b2950 2024-09-17T04:05:58.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.779+0000 7f9f4e000640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 0x7f9f3007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:05:58.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.780+0000 7f9f4e000640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 0x7f9f3007b0d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9f500ced70 tx=0x7f9f3c001840 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:05:58.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.782+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9f40062100 con 0x7f9f500b2950 2024-09-17T04:05:58.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:58.978+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9f5006a140 con 0x7f9f500b2950 2024-09-17T04:05:59.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.750+0000 7f9f46a00640 1 -- 172.21.15.129:0/3418223821 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v227) ==== 97+0+0 (secure 0 0 0) 0x7f9f40066b30 con 0x7f9f500b2950 2024-09-17T04:05:59.749 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 msgr2=0x7f9f3007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 0x7f9f3007b0d0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9f500ced70 tx=0x7f9f3c001840 comp rx=0 tx=0).stop 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 msgr2=0x7f9f50099990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 0x7f9f50099990 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f9f4000c3f0 tx=0x7f9f4000c8b0 comp rx=0 tx=0).stop 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 shutdown_connections 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9f30078bc0 0x7f9f3007b0d0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9f501f5220 0x7f9f501e45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:59.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9f500ce170 0x7f9f501e4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:59.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.753+0000 7f9f55600640 1 --2- 172.21.15.129:0/3418223821 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9f500b2950 0x7f9f50099990 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:05:59.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.754+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 >> 172.21.15.129:0/3418223821 conn(0x7f9f5006d410 msgr2=0x7f9f500cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:05:59.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.754+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 shutdown_connections 2024-09-17T04:05:59.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:05:59.754+0000 7f9f55600640 1 -- 172.21.15.129:0/3418223821 wait complete. 2024-09-17T04:05:59.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:06:00.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 -- 172.21.15.129:0/739248131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 msgr2=0x7f2bb00b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/739248131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb00b3480 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f2b98005fd0 tx=0x7f2b9802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 -- 172.21.15.129:0/739248131 shutdown_connections 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/739248131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb00b3480 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/739248131 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00ce170 0x7f2bb00cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 -- 172.21.15.129:0/739248131 >> 172.21.15.129:0/739248131 conn(0x7f2bb006d410 msgr2=0x7f2bb00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 -- 172.21.15.129:0/739248131 shutdown_connections 2024-09-17T04:06:00.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.002+0000 7f2bb4a00640 1 -- 172.21.15.129:0/739248131 wait complete. 2024-09-17T04:06:00.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.004+0000 7f2bb4a00640 1 Processor -- start 2024-09-17T04:06:00.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.004+0000 7f2bb4a00640 1 -- start start 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb01583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00b39c0 0x7f2bb015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb006f5e0 con 0x7f2bb00ce170 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb00708e0 con 0x7f2bb00b2950 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bb4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb006f760 con 0x7f2bb00b39c0 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49042/0 (socket says 172.21.15.129:49042) 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.006+0000 7f2bae800640 1 -- 172.21.15.129:0/1492759888 learned_addr learned my addr 172.21.15.129:0/1492759888 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bade00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb01583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2ba5400640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00b39c0 0x7f2bb015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 -- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00b39c0 msgr2=0x7f2bb015a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00b39c0 0x7f2bb015a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 -- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 msgr2=0x7f2bb01583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb01583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 -- 172.21.15.129:0/1492759888 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb0155960 con 0x7f2bb00ce170 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bade00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:00.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bae800640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c760 tx=0x7f2ba800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8003da0 con 0x7f2bb00ce170 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2ba800ce10 con 0x7f2bb00ce170 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb00afed0 con 0x7f2bb00ce170 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.007+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80052d0 con 0x7f2bb00ce170 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.008+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb00b0470 con 0x7f2bb00ce170 2024-09-17T04:06:00.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.009+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb0076430 con 0x7f2bb00ce170 2024-09-17T04:06:00.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.010+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2ba8005470 con 0x7f2bb00ce170 2024-09-17T04:06:00.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.011+0000 7f2ba6a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.011+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4857+0+0 (secure 0 0 0) 0x7f2ba809a7e0 con 0x7f2bb00ce170 2024-09-17T04:06:00.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.011+0000 7f2bade00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.012+0000 7f2bade00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009b80 tx=0x7f2ba0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.015+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2bb0076430 con 0x7f2bb00ce170 2024-09-17T04:06:00.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.263+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f2bb0158ac0 con 0x7f2bb00ce170 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.264+0000 7f2ba6a00640 1 -- 172.21.15.129:0/1492759888 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v44) ==== 80+0+5304 (secure 0 0 0) 0x7f2bb00b3dd0 con 0x7f2bb00ce170 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.263 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:06:00.264 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:06:00.265 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.266 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.267 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:06:00.269 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.270 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.273 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:06:00.274 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.275 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:00.276 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.277 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.278 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:06:00.279 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:06:00.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 msgr2=0x7f2b8c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009b80 tx=0x7f2ba0005f90 comp rx=0 tx=0).stop 2024-09-17T04:06:00.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 msgr2=0x7f2bb015a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7f2ba800c760 tx=0x7f2ba800cc20 comp rx=0 tx=0).stop 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 shutdown_connections 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2b8c078bc0 0x7f2b8c07b0d0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2bb00ce170 0x7f2bb015a6b0 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2bb00b39c0 0x7f2bb015a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 --2- 172.21.15.129:0/1492759888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2bb00b2950 0x7f2bb01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 >> 172.21.15.129:0/1492759888 conn(0x7f2bb006d410 msgr2=0x7f2bb00cccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.266+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 shutdown_connections 2024-09-17T04:06:00.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.267+0000 7f2bb4a00640 1 -- 172.21.15.129:0/1492759888 wait complete. 2024-09-17T04:06:00.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:06:00.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 -- 172.21.15.129:0/4189272261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7301524c0 msgr2=0x7fb7300cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 --2- 172.21.15.129:0/4189272261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7301524c0 0x7fb7300cf5c0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fb720005fd0 tx=0x7fb72002a540 comp rx=0 tx=0).stop 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 -- 172.21.15.129:0/4189272261 shutdown_connections 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 --2- 172.21.15.129:0/4189272261 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7300cfb00 0x7fb73015bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 --2- 172.21.15.129:0/4189272261 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7301524c0 0x7fb7300cf5c0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 -- 172.21.15.129:0/4189272261 >> 172.21.15.129:0/4189272261 conn(0x7fb73006cfc0 msgr2=0x7fb73006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 -- 172.21.15.129:0/4189272261 shutdown_connections 2024-09-17T04:06:00.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.509+0000 7fb736600640 1 -- 172.21.15.129:0/4189272261 wait complete. 2024-09-17T04:06:00.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.511+0000 7fb736600640 1 Processor -- start 2024-09-17T04:06:00.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.511+0000 7fb736600640 1 -- start start 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7300cfb00 0x7fb7300c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7301524c0 0x7fb73014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb7300d02c0 con 0x7fb73015c1e0 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb730152b10 con 0x7fb7300cfb00 2024-09-17T04:06:00.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb736600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb7300d0440 con 0x7fb7301524c0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49060/0 (socket says 172.21.15.129:49060) 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 -- 172.21.15.129:0/2103604361 learned_addr learned my addr 172.21.15.129:0/2103604361 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735200640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7300cfb00 0x7fb7300c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb727e00640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7301524c0 0x7fb73014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 -- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7301524c0 msgr2=0x7fb73014aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7301524c0 0x7fb73014aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 -- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7300cfb00 msgr2=0x7fb7300c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7300cfb00 0x7fb7300c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.513+0000 7fb735c00640 1 -- 172.21.15.129:0/2103604361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7300c62b0 con 0x7fb73015c1e0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb735c00640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fb72800be90 tx=0x7fb72800bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728019070 con 0x7fb73015c1e0 2024-09-17T04:06:00.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb728014440 con 0x7fb73015c1e0 2024-09-17T04:06:00.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7300c6590 con 0x7fb73015c1e0 2024-09-17T04:06:00.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7280148d0 con 0x7fb73015c1e0 2024-09-17T04:06:00.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.514+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb730149ed0 con 0x7fb73015c1e0 2024-09-17T04:06:00.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.515+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7300c6020 con 0x7fb73015c1e0 2024-09-17T04:06:00.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.516+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb72801c070 con 0x7fb73015c1e0 2024-09-17T04:06:00.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.517+0000 7fb72d600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 0x7fb70c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.517+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4857+0+0 (secure 0 0 0) 0x7fb72809aa80 con 0x7fb73015c1e0 2024-09-17T04:06:00.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.518+0000 7fb735200640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 0x7fb70c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.518+0000 7fb735200640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 0x7fb70c07b0d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb720033000 tx=0x7fb720004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.522+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb728063280 con 0x7fb73015c1e0 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.709+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb7300c6ab0 con 0x7fb73015c1e0 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.710+0000 7fb72d600640 1 -- 172.21.15.129:0/2103604361 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v229) ==== 84+0+65 (secure 0 0 0) 0x7fb728062b80 con 0x7fb73015c1e0 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:06:00.709 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:00.710 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 msgr2=0x7fb70c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 0x7fb70c07b0d0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fb720033000 tx=0x7fb720004730 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 msgr2=0x7fb730148e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7fb72800be90 tx=0x7fb72800bf80 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 shutdown_connections 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb70c078bc0 0x7fb70c07b0d0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb73015c1e0 0x7fb730148e50 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7301524c0 0x7fb73014aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 --2- 172.21.15.129:0/2103604361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7300cfb00 0x7fb7300c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 >> 172.21.15.129:0/2103604361 conn(0x7fb73006cfc0 msgr2=0x7fb7300b4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 shutdown_connections 2024-09-17T04:06:00.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.712+0000 7fb736600640 1 -- 172.21.15.129:0/2103604361 wait complete. 2024-09-17T04:06:00.737 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:06:00.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.963+0000 7f5ba2200640 1 -- 172.21.15.129:0/3784312647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 msgr2=0x7f5b80003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.963+0000 7f5ba2200640 1 --2- 172.21.15.129:0/3784312647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b80003ec0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009b50 tx=0x7f5b80005830 comp rx=0 tx=0).stop 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.964+0000 7f5ba2200640 1 -- 172.21.15.129:0/3784312647 shutdown_connections 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.964+0000 7f5ba2200640 1 --2- 172.21.15.129:0/3784312647 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b80003ec0 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.964+0000 7f5ba2200640 1 -- 172.21.15.129:0/3784312647 >> 172.21.15.129:0/3784312647 conn(0x7f5b9c0ccc40 msgr2=0x7f5b9c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.964+0000 7f5ba2200640 1 -- 172.21.15.129:0/3784312647 shutdown_connections 2024-09-17T04:06:00.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.964+0000 7f5ba2200640 1 -- 172.21.15.129:0/3784312647 wait complete. 2024-09-17T04:06:00.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.966+0000 7f5ba2200640 1 Processor -- start 2024-09-17T04:06:00.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.966+0000 7f5ba2200640 1 -- start start 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b9c158560 0x7f5b9c15be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 0x7f5b9c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c0b4d90 con 0x7f5b80003a50 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5b9c0b4c10 con 0x7f5b9c158560 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.967+0000 7f5ba2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5b9c0b1e20 con 0x7f5b9c158970 2024-09-17T04:06:00.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49080/0 (socket says 172.21.15.129:49080) 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 -- 172.21.15.129:0/2542589797 learned_addr learned my addr 172.21.15.129:0/2542589797 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9aa00640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b9c158560 0x7f5b9c15be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9be00640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 0x7f5b9c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 -- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 msgr2=0x7f5b9c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 0x7f5b9c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 -- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b9c158560 msgr2=0x7f5b9c15be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b9c158560 0x7f5b9c15be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 -- 172.21.15.129:0/2542589797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9c1e2c40 con 0x7f5b80003a50 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9be00640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 0x7f5b9c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.968+0000 7f5b9b400640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009a70 tx=0x7f5b8c0336f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.969+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c038040 con 0x7f5b80003a50 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.969+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5b8c0339d0 con 0x7f5b80003a50 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.969+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b9c157490 con 0x7f5b80003a50 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.969+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c033e20 con 0x7f5b80003a50 2024-09-17T04:06:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.969+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b9c157a50 con 0x7f5b80003a50 2024-09-17T04:06:00.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.970+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9c0b2380 con 0x7f5b80003a50 2024-09-17T04:06:00.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.971+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5b8c043020 con 0x7f5b80003a50 2024-09-17T04:06:00.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.973+0000 7f5b93400640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 0x7f5b7c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:00.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.973+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(229..229 src has 1..229) ==== 4857+0+0 (secure 0 0 0) 0x7f5b8c0bab50 con 0x7f5b80003a50 2024-09-17T04:06:00.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.973+0000 7f5b9aa00640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 0x7f5b7c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:00.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.974+0000 7f5b9aa00640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 0x7f5b7c07b0a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0752a0 tx=0x7f5b88006040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:00.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:00.977+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5b8c04a020 con 0x7f5b80003a50 2024-09-17T04:06:01.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.158+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5b9c0affd0 con 0x7f5b80003a50 2024-09-17T04:06:01.764 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.765+0000 7f5b93400640 1 -- 172.21.15.129:0/2542589797 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v230) ==== 138+0+0 (secure 0 0 0) 0x7f5b8c082c80 con 0x7f5b80003a50 2024-09-17T04:06:01.765 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 msgr2=0x7f5b7c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 0x7f5b7c07b0a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c0752a0 tx=0x7f5b88006040 comp rx=0 tx=0).stop 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 msgr2=0x7f5b9c150540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c009a70 tx=0x7f5b8c0336f0 comp rx=0 tx=0).stop 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 shutdown_connections 2024-09-17T04:06:01.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5b7c078b90 0x7f5b7c07b0a0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5b9c158970 0x7f5b9c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b9c158560 0x7f5b9c15be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 --2- 172.21.15.129:0/2542589797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5b80003a50 0x7f5b9c150540 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 >> 172.21.15.129:0/2542589797 conn(0x7f5b9c0ccc40 msgr2=0x7f5b9c0c9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 shutdown_connections 2024-09-17T04:06:01.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:01.768+0000 7f5ba2200640 1 -- 172.21.15.129:0/2542589797 wait complete. 2024-09-17T04:06:01.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:06:02.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 -- 172.21.15.129:0/929847091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 msgr2=0x7f7d3c1696f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 --2- 172.21.15.129:0/929847091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1696f0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7d34009be0 tx=0x7f7d3402a870 comp rx=0 tx=0).stop 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 -- 172.21.15.129:0/929847091 shutdown_connections 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 --2- 172.21.15.129:0/929847091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1696f0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 -- 172.21.15.129:0/929847091 >> 172.21.15.129:0/929847091 conn(0x7f7d3c0d0120 msgr2=0x7f7d3c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 -- 172.21.15.129:0/929847091 shutdown_connections 2024-09-17T04:06:02.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.028+0000 7f7d42800640 1 -- 172.21.15.129:0/929847091 wait complete. 2024-09-17T04:06:02.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.030+0000 7f7d42800640 1 Processor -- start 2024-09-17T04:06:02.029 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.031+0000 7f7d42800640 1 -- start start 2024-09-17T04:06:02.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d3c06ee70 0x7f7d3c0b4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:02.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:02.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:02.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c169e40 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3c0adb50 con 0x7f7d3c0cbf00 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d42800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3c169fc0 con 0x7f7d3c06ee70 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d41e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d41e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49082/0 (socket says 172.21.15.129:49082) 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.032+0000 7f7d41e00640 1 -- 172.21.15.129:0/448168832 learned_addr learned my addr 172.21.15.129:0/448168832 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d40a00640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41400640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d3c06ee70 0x7f7d3c0b4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 -- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d3c06ee70 msgr2=0x7f7d3c0b4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d3c06ee70 0x7f7d3c0b4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 -- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 msgr2=0x7f7d3c1579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 -- 172.21.15.129:0/448168832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3c0b4e90 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.033+0000 7f7d41e00640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f7d3402ae60 tx=0x7f7d34031f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.034+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3403f070 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.034+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d3c0b5170 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.034+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7d34003c60 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.034+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d34036040 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.034+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d3c0b5730 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.035+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d3c0b4c00 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.037+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7d34039070 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.037+0000 7f7d39600640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 0x7f7d1c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:02.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.038+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(230..230 src has 1..230) ==== 5227+0+0 (secure 0 0 0) 0x7f7d340bb830 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.038+0000 7f7d41400640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 0x7f7d1c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:02.038 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.039+0000 7f7d41400640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 0x7f7d1c07b0a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7d30000f60 tx=0x7f7d30009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:02.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.041+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7d34083760 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.226+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7d3c15ed60 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.774+0000 7f7d39600640 1 -- 172.21.15.129:0/448168832 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v231) ==== 170+0+0 (secure 0 0 0) 0x7f7d34088190 con 0x7f7d3c15f9e0 2024-09-17T04:06:02.773 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.777+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 msgr2=0x7f7d1c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.777+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 0x7f7d1c07b0a0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7d30000f60 tx=0x7f7d30009040 comp rx=0 tx=0).stop 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.777+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 msgr2=0x7f7d3c157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.777+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f7d3402ae60 tx=0x7f7d34031f60 comp rx=0 tx=0).stop 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 shutdown_connections 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7d1c078b90 0x7f7d1c07b0a0 secure :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7d30000f60 tx=0x7f7d30009040 comp rx=0 tx=0).stop 2024-09-17T04:06:02.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7d3c15f9e0 0x7f7d3c157ee0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d3c0cbf00 0x7f7d3c1579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 --2- 172.21.15.129:0/448168832 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7d3c06ee70 0x7f7d3c0b4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 >> 172.21.15.129:0/448168832 conn(0x7f7d3c0d0120 msgr2=0x7f7d3c0b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 shutdown_connections 2024-09-17T04:06:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:02.778+0000 7f7d42800640 1 -- 172.21.15.129:0/448168832 wait complete. 2024-09-17T04:06:02.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:06:03.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 -- 172.21.15.129:0/589258845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 msgr2=0x7f4d2c072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 --2- 172.21.15.129:0/589258845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c072e10 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c06dd90 tx=0x7f4d1402a560 comp rx=0 tx=0).stop 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 -- 172.21.15.129:0/589258845 shutdown_connections 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 --2- 172.21.15.129:0/589258845 >> v2:172.21.15.181:3300/0 conn(0x7f4d2c073350 0x7f4d2c0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 --2- 172.21.15.129:0/589258845 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c072e10 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 -- 172.21.15.129:0/589258845 >> 172.21.15.129:0/589258845 conn(0x7f4d2c0d00b0 msgr2=0x7f4d2c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 -- 172.21.15.129:0/589258845 shutdown_connections 2024-09-17T04:06:03.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.048+0000 7f4d2b200640 1 -- 172.21.15.129:0/589258845 wait complete. 2024-09-17T04:06:03.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.051+0000 7f4d2b200640 1 Processor -- start 2024-09-17T04:06:03.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.051+0000 7f4d2b200640 1 -- start start 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d2c073350 0x7f4d2c09cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d2c15c7b0 0x7f4d2c0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c0ceb40 con 0x7f4d2c0cb080 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2c0ce9c0 con 0x7f4d2c073350 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d2b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2c0cecc0 con 0x7f4d2c15c7b0 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d21400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d21400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49096/0 (socket says 172.21.15.129:49096) 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.052+0000 7f4d21400640 1 -- 172.21.15.129:0/1382256308 learned_addr learned my addr 172.21.15.129:0/1382256308 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d29e00640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d2c073350 0x7f4d2c09cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d2a800640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d2c15c7b0 0x7f4d2c0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:03.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 -- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d2c15c7b0 msgr2=0x7f4d2c0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d2c15c7b0 0x7f4d2c0b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 -- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d2c073350 msgr2=0x7f4d2c09cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d2c073350 0x7f4d2c09cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 -- 172.21.15.129:0/1382256308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2c15f9a0 con 0x7f4d2c0cb080 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.053+0000 7f4d21400640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f4d14034040 tx=0x7f4d14003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.054+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1403e030 con 0x7f4d2c0cb080 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.054+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4d14003e00 con 0x7f4d2c0cb080 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.054+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d14035040 con 0x7f4d2c0cb080 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.054+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2c09d6c0 con 0x7f4d2c0cb080 2024-09-17T04:06:03.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.054+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d2c09ad40 con 0x7f4d2c0cb080 2024-09-17T04:06:03.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.055+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d2c09d430 con 0x7f4d2c0cb080 2024-09-17T04:06:03.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.056+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4d14041030 con 0x7f4d2c0cb080 2024-09-17T04:06:03.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.057+0000 7f4d22a00640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 0x7f4d0807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:03.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.057+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(231..231 src has 1..231) ==== 5609+0+0 (secure 0 0 0) 0x7f4d140bb450 con 0x7f4d2c0cb080 2024-09-17T04:06:03.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.058+0000 7f4d29e00640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 0x7f4d0807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:03.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.059+0000 7f4d29e00640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 0x7f4d0807b0a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006fd0 tx=0x7f4d1c009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:03.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.062+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4d2c15f9a0 con 0x7f4d2c0cb080 2024-09-17T04:06:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.280+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4d2c0cf170 con 0x7f4d2c0cb080 2024-09-17T04:06:03.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.776+0000 7f4d22a00640 1 -- 172.21.15.129:0/1382256308 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v224) ==== 174+0+0 (secure 0 0 0) 0x7f4d14083330 con 0x7f4d2c0cb080 2024-09-17T04:06:03.775 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 34 and data pool 35 2024-09-17T04:06:03.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 msgr2=0x7f4d0807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:03.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 0x7f4d0807b0a0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c006fd0 tx=0x7f4d1c009060 comp rx=0 tx=0).stop 2024-09-17T04:06:03.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 msgr2=0x7f4d2c15f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:03.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f4d14034040 tx=0x7f4d14003910 comp rx=0 tx=0).stop 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 shutdown_connections 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4d08078b90 0x7f4d0807b0a0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4d2c15c7b0 0x7f4d2c0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4d2c0cb080 0x7f4d2c15f460 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 --2- 172.21.15.129:0/1382256308 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d2c073350 0x7f4d2c09cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:03.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.782+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 >> 172.21.15.129:0/1382256308 conn(0x7f4d2c0d00b0 msgr2=0x7f4d2c1601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:03.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.785+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 shutdown_connections 2024-09-17T04:06:03.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:03.785+0000 7f4d2b200640 1 -- 172.21.15.129:0/1382256308 wait complete. 2024-09-17T04:06:03.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 -- 172.21.15.129:0/2411499694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c0d09d0 msgr2=0x7f106c15cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 --2- 172.21.15.129:0/2411499694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c0d09d0 0x7f106c15cbd0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1064005fd0 tx=0x7f106402a8a0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 -- 172.21.15.129:0/2411499694 shutdown_connections 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 --2- 172.21.15.129:0/2411499694 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c0d09d0 0x7f106c15cbd0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 -- 172.21.15.129:0/2411499694 >> 172.21.15.129:0/2411499694 conn(0x7f106c06cfc0 msgr2=0x7f106c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 -- 172.21.15.129:0/2411499694 shutdown_connections 2024-09-17T04:06:04.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.049+0000 7f1071000640 1 -- 172.21.15.129:0/2411499694 wait complete. 2024-09-17T04:06:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.051+0000 7f1071000640 1 Processor -- start 2024-09-17T04:06:04.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.052+0000 7f1071000640 1 -- start start 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f106c0cfb60 0x7f106c153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 0x7f106c156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f106c15d4a0 con 0x7f106c160cf0 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f106c15d320 con 0x7f106c15a900 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1071000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f106c15d620 con 0x7f106c0cfb60 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1062a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1062a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51876/0 (socket says 172.21.15.129:51876) 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.053+0000 7f1062a00640 1 -- 172.21.15.129:0/4228087754 learned_addr learned my addr 172.21.15.129:0/4228087754 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f106be00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 0x7f106c156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f106b400640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f106c0cfb60 0x7f106c153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 -- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f106c0cfb60 msgr2=0x7f106c153660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f106c0cfb60 0x7f106c153660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 -- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 msgr2=0x7f106c156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 0x7f106c156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 -- 172.21.15.129:0/4228087754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c1f8670 con 0x7f106c160cf0 2024-09-17T04:06:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f106be00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 0x7f106c156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1062a00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f105400c460 tx=0x7f105400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.054+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054016020 con 0x7f106c160cf0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.055+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1054012940 con 0x7f106c160cf0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.055+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054012d90 con 0x7f106c160cf0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.055+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c1f8950 con 0x7f106c160cf0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.055+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c1f8f10 con 0x7f106c160cf0 2024-09-17T04:06:04.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.056+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f106c15cbd0 con 0x7f106c160cf0 2024-09-17T04:06:04.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.057+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1054019020 con 0x7f106c160cf0 2024-09-17T04:06:04.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.058+0000 7f1063e00640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 0x7f104807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.059+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 5709+0+0 (secure 0 0 0) 0x7f105409ac50 con 0x7f106c160cf0 2024-09-17T04:06:04.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.059+0000 7f106b400640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 0x7f104807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.060+0000 7f106b400640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 0x7f104807b0a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f106c0b4450 tx=0x7f105c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.063+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1054020020 con 0x7f106c160cf0 2024-09-17T04:06:04.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.246+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f106c06a140 con 0x7f106c160cf0 2024-09-17T04:06:04.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.247+0000 7f1063e00640 1 -- 172.21.15.129:0/4228087754 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) ==== 74+0+16748 (secure 0 0 0) 0x7f10540629a0 con 0x7f106c160cf0 2024-09-17T04:06:04.247 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:04.247 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":232,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:03.281621+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:01.171609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:02.239670+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 msgr2=0x7f104807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 0x7f104807b0a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f106c0b4450 tx=0x7f105c009040 comp rx=0 tx=0).stop 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 msgr2=0x7f106c15a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f105400c460 tx=0x7f105400c920 comp rx=0 tx=0).stop 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 shutdown_connections 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1048078b90 0x7f104807b0a0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f106c15a900 0x7f106c156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f106c160cf0 0x7f106c15a3c0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 --2- 172.21.15.129:0/4228087754 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f106c0cfb60 0x7f106c153660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 >> 172.21.15.129:0/4228087754 conn(0x7f106c06cfc0 msgr2=0x7f106c15f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:04.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 shutdown_connections 2024-09-17T04:06:04.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.250+0000 7f1071000640 1 -- 172.21.15.129:0/4228087754 wait complete. 2024-09-17T04:06:04.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.491+0000 7fb81c600640 1 -- 172.21.15.129:0/703833349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140b4140 msgr2=0x7fb8140ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.491+0000 7fb81c600640 1 --2- 172.21.15.129:0/703833349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140b4140 0x7fb8140ccc80 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb800009830 tx=0x7fb80002a510 comp rx=0 tx=0).stop 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.491+0000 7fb81c600640 1 -- 172.21.15.129:0/703833349 shutdown_connections 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.492+0000 7fb81c600640 1 --2- 172.21.15.129:0/703833349 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 0x7fb8141609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.492+0000 7fb81c600640 1 --2- 172.21.15.129:0/703833349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140b4140 0x7fb8140ccc80 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.492+0000 7fb81c600640 1 -- 172.21.15.129:0/703833349 >> 172.21.15.129:0/703833349 conn(0x7fb81406d410 msgr2=0x7fb8140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:04.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.492+0000 7fb81c600640 1 -- 172.21.15.129:0/703833349 shutdown_connections 2024-09-17T04:06:04.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.492+0000 7fb81c600640 1 -- 172.21.15.129:0/703833349 wait complete. 2024-09-17T04:06:04.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.494+0000 7fb81c600640 1 Processor -- start 2024-09-17T04:06:04.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.494+0000 7fb81c600640 1 -- start start 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140705a0 0x7fb814159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 0x7fb8141f9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb814160260 con 0x7fb8140b4140 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8141600e0 con 0x7fb8140705a0 2024-09-17T04:06:04.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.495+0000 7fb81c600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb814160f00 con 0x7fb8140cd1c0 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51878/0 (socket says 172.21.15.129:51878) 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 -- 172.21.15.129:0/2000474708 learned_addr learned my addr 172.21.15.129:0/2000474708 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819c00640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140705a0 0x7fb814159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb81a600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 0x7fb8141f9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 -- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 msgr2=0x7fb8141f9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 0x7fb8141f9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 -- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140705a0 msgr2=0x7fb814159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140705a0 0x7fb814159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 -- 172.21.15.129:0/2000474708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8140cc8a0 con 0x7fb8140b4140 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.496+0000 7fb819200640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fb80002ab00 tx=0x7fb800033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.497+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800005c90 con 0x7fb8140b4140 2024-09-17T04:06:04.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.497+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb800031e50 con 0x7fb8140b4140 2024-09-17T04:06:04.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.497+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81415c300 con 0x7fb8140b4140 2024-09-17T04:06:04.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.497+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb800005260 con 0x7fb8140b4140 2024-09-17T04:06:04.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.497+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8141fa6c0 con 0x7fb8140b4140 2024-09-17T04:06:04.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.498+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb814160c10 con 0x7fb8140b4140 2024-09-17T04:06:04.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.499+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb800005400 con 0x7fb8140b4140 2024-09-17T04:06:04.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.500+0000 7fb811600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 0x7fb7f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.500+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(232..232 src has 1..232) ==== 5709+0+0 (secure 0 0 0) 0x7fb8000bbe40 con 0x7fb8140b4140 2024-09-17T04:06:04.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.501+0000 7fb819c00640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 0x7fb7f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.501+0000 7fb819c00640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 0x7fb7f407b0d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb808007d60 tx=0x7fb808007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.505+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb814160c10 con 0x7fb8140b4140 2024-09-17T04:06:04.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.687+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb8141fa450 con 0x7fb8140b4140 2024-09-17T04:06:04.688 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.688+0000 7fb811600640 1 -- 172.21.15.129:0/2000474708 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) ==== 74+0+16748 (secure 0 0 0) 0x7fb80003f030 con 0x7fb8140b4140 2024-09-17T04:06:04.689 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:04.689 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":232,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:03.281621+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:01.171609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:02.239670+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 msgr2=0x7fb7f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 0x7fb7f407b0d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb808007d60 tx=0x7fb808007cf0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 msgr2=0x7fb8140cc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7fb80002ab00 tx=0x7fb800033c20 comp rx=0 tx=0).stop 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 shutdown_connections 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7f4078bc0 0x7fb7f407b0d0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8140cd1c0 0x7fb8141f9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.691 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8140b4140 0x7fb8140cc360 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 --2- 172.21.15.129:0/2000474708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8140705a0 0x7fb814159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 >> 172.21.15.129:0/2000474708 conn(0x7fb81406d410 msgr2=0x7fb8140b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:04.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 shutdown_connections 2024-09-17T04:06:04.692 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.691+0000 7fb81c600640 1 -- 172.21.15.129:0/2000474708 wait complete. 2024-09-17T04:06:04.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:06:04.736 INFO:tasks.ceph.mds.b.smithi181.stderr:2024-09-17T04:06:04.736+0000 7f3ab3200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:06:04.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 -- 172.21.15.129:0/3327976315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c004d40 msgr2=0x7f244c0051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 --2- 172.21.15.129:0/3327976315 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f244c004d40 0x7f244c0051f0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7f2460009ed0 tx=0x7f244c005830 comp rx=0 tx=0).stop 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2466800640 1 -- 172.21.15.129:0/3327976315 reap_dead start 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 -- 172.21.15.129:0/3327976315 shutdown_connections 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 -- 172.21.15.129:0/3327976315 >> 172.21.15.129:0/3327976315 conn(0x7f24680d00b0 msgr2=0x7f24680d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 -- 172.21.15.129:0/3327976315 shutdown_connections 2024-09-17T04:06:04.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.944+0000 7f2467200640 1 -- 172.21.15.129:0/3327976315 wait complete. 2024-09-17T04:06:04.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.946+0000 7f2467200640 1 Processor -- start 2024-09-17T04:06:04.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.947+0000 7f2467200640 1 -- start start 2024-09-17T04:06:04.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f244c004d40 0x7f246815b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f246809d4c0 0x7f24680adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2468165b70 con 0x7f24680ae110 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f24681659f0 con 0x7f246809d4c0 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2467200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2468165cf0 con 0x7f244c004d40 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2466800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2466800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51890/0 (socket says 172.21.15.129:51890) 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2466800640 1 -- 172.21.15.129:0/3143437475 learned_addr learned my addr 172.21.15.129:0/3143437475 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:04.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f2465e00640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f244c004d40 0x7f246815b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.948+0000 7f245d400640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f246809d4c0 0x7f24680adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 -- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f244c004d40 msgr2=0x7f246815b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f244c004d40 0x7f246815b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 -- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f246809d4c0 msgr2=0x7f24680adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f246809d4c0 0x7f24680adbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 -- 172.21.15.129:0/3143437475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24680ae520 con 0x7f24680ae110 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2466800640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f2460009ed0 tx=0x7f2460033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460043070 con 0x7f24680ae110 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2460033c80 con 0x7f24680ae110 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.949+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24681f5f60 con 0x7f24680ae110 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.950+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460038040 con 0x7f24680ae110 2024-09-17T04:06:04.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.950+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24681f6520 con 0x7f24680ae110 2024-09-17T04:06:04.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.951+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2468072e10 con 0x7f24680ae110 2024-09-17T04:06:04.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.952+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2460003bd0 con 0x7f24680ae110 2024-09-17T04:06:04.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.953+0000 7f245ea00640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 0x7f244407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:04.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.953+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f24600bae40 con 0x7f24680ae110 2024-09-17T04:06:04.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.953+0000 7f2465e00640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 0x7f244407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:04.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.954+0000 7f2465e00640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 0x7f244407b0a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f2458001220 tx=0x7f24580067d0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:04.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:04.957+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f24680ae520 con 0x7f24680ae110 2024-09-17T04:06:05.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.175+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f24681f5cf0 con 0x7f24680ae110 2024-09-17T04:06:05.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.991+0000 7f245ea00640 1 -- 172.21.15.129:0/3143437475 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v226) ==== 118+0+0 (secure 0 0 0) 0x7f2460082c30 con 0x7f24680ae110 2024-09-17T04:06:05.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 msgr2=0x7f244407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:05.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 0x7f244407b0a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f2458001220 tx=0x7f24580067d0 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 msgr2=0x7f24681f8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f2460009ed0 tx=0x7f2460033c10 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 shutdown_connections 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2444078b90 0x7f244407b0a0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f24680ae110 0x7f24681f8f00 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f246809d4c0 0x7f24680adbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 --2- 172.21.15.129:0/3143437475 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f244c004d40 0x7f246815b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 >> 172.21.15.129:0/3143437475 conn(0x7f24680d00b0 msgr2=0x7f24680737c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 shutdown_connections 2024-09-17T04:06:05.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:05.994+0000 7f2467200640 1 -- 172.21.15.129:0/3143437475 wait complete. 2024-09-17T04:06:06.018 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:06:06.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:06:06.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 -- 172.21.15.129:0/3866689956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c0b1c40 msgr2=0x7ff33c0b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 --2- 172.21.15.129:0/3866689956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c0b1c40 0x7ff33c0b2050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff328005fd0 tx=0x7ff32802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 -- 172.21.15.129:0/3866689956 shutdown_connections 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 --2- 172.21.15.129:0/3866689956 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c0b1c40 0x7ff33c0b2050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 --2- 172.21.15.129:0/3866689956 >> v2:172.21.15.181:3300/0 conn(0x7ff33c074800 0x7ff33c0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 -- 172.21.15.129:0/3866689956 >> 172.21.15.129:0/3866689956 conn(0x7ff33c0ccc40 msgr2=0x7ff33c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 -- 172.21.15.129:0/3866689956 shutdown_connections 2024-09-17T04:06:06.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.241+0000 7ff340c00640 1 -- 172.21.15.129:0/3866689956 wait complete. 2024-09-17T04:06:06.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.244+0000 7ff340c00640 1 Processor -- start 2024-09-17T04:06:06.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.244+0000 7ff340c00640 1 -- start start 2024-09-17T04:06:06.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c06eea0 0x7ff33c1f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33c074800 0x7ff33c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c0b41a0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff33c0b4020 con 0x7ff33c074800 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.245+0000 7ff340c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff33c0b4320 con 0x7ff33c06eea0 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51902/0 (socket says 172.21.15.129:51902) 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 -- 172.21.15.129:0/706008527 learned_addr learned my addr 172.21.15.129:0/706008527 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff339e00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c06eea0 0x7ff33c1f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff339400640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33c074800 0x7ff33c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 -- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c06eea0 msgr2=0x7ff33c1f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:06.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c06eea0 0x7ff33c1f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 -- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33c074800 msgr2=0x7ff33c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33c074800 0x7ff33c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 -- 172.21.15.129:0/706008527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c1571c0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.246+0000 7ff33a800640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c810 tx=0x7ff33400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.247+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33400ce60 con 0x7ff33c0b1c40 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.247+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33c1574a0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.247+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff334003b90 con 0x7ff33c0b1c40 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.247+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33400a850 con 0x7ff33c0b1c40 2024-09-17T04:06:06.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.247+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff33c1559e0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.248+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff33c06fb30 con 0x7ff33c0b1c40 2024-09-17T04:06:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.249+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff33400a9f0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.250+0000 7ff332000640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 0x7ff31c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.250+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7ff33409a9f0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.250+0000 7ff339e00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 0x7ff31c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:06.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.251+0000 7ff339e00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 0x7ff31c07b0a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0095f0 tx=0x7ff32c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:06.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.255+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff33c0cbaf0 con 0x7ff33c0b1c40 2024-09-17T04:06:06.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.473+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff33c15cb00 con 0x7ff33c0b1c40 2024-09-17T04:06:06.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:06.999+0000 7ff332000640 1 -- 172.21.15.129:0/706008527 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v227) ==== 122+0+0 (secure 0 0 0) 0x7ff334062750 con 0x7ff33c0b1c40 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 msgr2=0x7ff31c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 0x7ff31c07b0a0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7ff32c0095f0 tx=0x7ff32c005f90 comp rx=0 tx=0).stop 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 msgr2=0x7ff33c153a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7ff33400c810 tx=0x7ff33400cc10 comp rx=0 tx=0).stop 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 shutdown_connections 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff31c078b90 0x7ff31c07b0a0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff33c0b1c40 0x7ff33c153a20 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff33c074800 0x7ff33c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 --2- 172.21.15.129:0/706008527 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff33c06eea0 0x7ff33c1f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 >> 172.21.15.129:0/706008527 conn(0x7ff33c0ccc40 msgr2=0x7ff33c15f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:07.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 shutdown_connections 2024-09-17T04:06:07.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.002+0000 7ff340c00640 1 -- 172.21.15.129:0/706008527 wait complete. 2024-09-17T04:06:07.029 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:06:07.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:07.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 -- 172.21.15.129:0/757752375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380b46c0 msgr2=0x7f86380b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 --2- 172.21.15.129:0/757752375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380b46c0 0x7f86380b4ad0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f86381602d0 tx=0x7f862002a5b0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 -- 172.21.15.129:0/757752375 shutdown_connections 2024-09-17T04:06:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 --2- 172.21.15.129:0/757752375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380b46c0 0x7f86380b4ad0 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 --2- 172.21.15.129:0/757752375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380cf8e0 0x7f8638160ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 -- 172.21.15.129:0/757752375 >> 172.21.15.129:0/757752375 conn(0x7f863806cfc0 msgr2=0x7f863806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 -- 172.21.15.129:0/757752375 shutdown_connections 2024-09-17T04:06:07.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.251+0000 7f8637200640 1 -- 172.21.15.129:0/757752375 wait complete. 2024-09-17T04:06:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.253+0000 7f8637200640 1 Processor -- start 2024-09-17T04:06:07.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.253+0000 7f8637200640 1 -- start start 2024-09-17T04:06:07.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380b46c0 0x7f86380b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86380cf8e0 0x7f86381f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f863815d150 con 0x7f86380d0380 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f863815cfd0 con 0x7f86380cf8e0 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8637200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f863815d2d0 con 0x7f86380b46c0 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51914/0 (socket says 172.21.15.129:51914) 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 -- 172.21.15.129:0/3506964186 learned_addr learned my addr 172.21.15.129:0/3506964186 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8635e00640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380b46c0 0x7f86380b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f862d400640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86380cf8e0 0x7f86381f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 -- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380b46c0 msgr2=0x7f86380b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380b46c0 0x7f86380b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 -- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86380cf8e0 msgr2=0x7f86381f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86380cf8e0 0x7f86381f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.255+0000 7f8636800640 1 -- 172.21.15.129:0/3506964186 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86381f34a0 con 0x7f86380d0380 2024-09-17T04:06:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f8636800640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f863000c730 tx=0x7f863000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:07.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630003b70 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8630003d10 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86380afa40 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8630005330 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.256+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86380b0000 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.257+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f863815d780 con 0x7f86380d0380 2024-09-17T04:06:07.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.258+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f86300054d0 con 0x7f86380d0380 2024-09-17T04:06:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.260+0000 7f862ea00640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 0x7f861407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.260+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f863009a940 con 0x7f86380d0380 2024-09-17T04:06:07.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.260+0000 7f8635e00640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 0x7f861407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.261+0000 7f8635e00640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 0x7f861407b0a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f8628009b30 tx=0x7f8628005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:07.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.265+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8630062720 con 0x7f86380d0380 2024-09-17T04:06:07.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.484+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f863806a140 con 0x7f86380d0380 2024-09-17T04:06:07.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.485+0000 7f862ea00640 1 -- 172.21.15.129:0/3506964186 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f8630067150 con 0x7f86380d0380 2024-09-17T04:06:07.485 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:07.485 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:07.486 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.487+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 msgr2=0x7f861407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 0x7f861407b0a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f8628009b30 tx=0x7f8628005f90 comp rx=0 tx=0).stop 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 msgr2=0x7f8638158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f863000c730 tx=0x7f863000cb30 comp rx=0 tx=0).stop 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 shutdown_connections 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8614078b90 0x7f861407b0a0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f86380d0380 0x7f8638158e90 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f86380cf8e0 0x7f86381f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 --2- 172.21.15.129:0/3506964186 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f86380b46c0 0x7f86380b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 >> 172.21.15.129:0/3506964186 conn(0x7f863806cfc0 msgr2=0x7f863815a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:07.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 shutdown_connections 2024-09-17T04:06:07.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.488+0000 7f8637200640 1 -- 172.21.15.129:0/3506964186 wait complete. 2024-09-17T04:06:07.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:07.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 -- 172.21.15.129:0/1265835029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e40705a0 msgr2=0x7ff5e40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 --2- 172.21.15.129:0/1265835029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e40705a0 0x7ff5e40b4140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009860 tx=0x7ff5d402a5a0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 -- 172.21.15.129:0/1265835029 shutdown_connections 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 --2- 172.21.15.129:0/1265835029 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e40705a0 0x7ff5e40b4140 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 -- 172.21.15.129:0/1265835029 >> 172.21.15.129:0/1265835029 conn(0x7ff5e406d410 msgr2=0x7ff5e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 -- 172.21.15.129:0/1265835029 shutdown_connections 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.731+0000 7ff5eb800640 1 -- 172.21.15.129:0/1265835029 wait complete. 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.733+0000 7ff5eb800640 1 Processor -- start 2024-09-17T04:06:07.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.734+0000 7ff5eb800640 1 -- start start 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e40705a0 0x7ff5e4167390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e415c980 0x7ff5e40ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e406f2e0 con 0x7ff5e406eed0 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e41612e0 con 0x7ff5e40705a0 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5eb800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e406f460 con 0x7ff5e415c980 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5e8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5e8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51926/0 (socket says 172.21.15.129:51926) 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.735+0000 7ff5e8e00640 1 -- 172.21.15.129:0/3005773378 learned_addr learned my addr 172.21.15.129:0/3005773378 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:07.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5dbe00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e40705a0 0x7ff5e4167390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e9800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e415c980 0x7ff5e40ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 -- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e415c980 msgr2=0x7ff5e40ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e415c980 0x7ff5e40ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 -- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e40705a0 msgr2=0x7ff5e4167390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e40705a0 0x7ff5e4167390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 -- 172.21.15.129:0/3005773378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e41678d0 con 0x7ff5e406eed0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e8e00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009980 tx=0x7ff5d4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d4031b50 con 0x7ff5e406eed0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.736+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff5d4031de0 con 0x7ff5e406eed0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.737+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d40332d0 con 0x7ff5e406eed0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.737+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e4165db0 con 0x7ff5e406eed0 2024-09-17T04:06:07.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.737+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e4166370 con 0x7ff5e406eed0 2024-09-17T04:06:07.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.738+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e40b4140 con 0x7ff5e406eed0 2024-09-17T04:06:07.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.739+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff5d4033470 con 0x7ff5e406eed0 2024-09-17T04:06:07.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.740+0000 7ff5e1600640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 0x7ff5c007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:07.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.740+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7ff5d40bbf00 con 0x7ff5e406eed0 2024-09-17T04:06:07.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.740+0000 7ff5dbe00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 0x7ff5c007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:07.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.741+0000 7ff5dbe00640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 0x7ff5c007b0d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff5e41685d0 tx=0x7ff5cc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:07.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.744+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff5d403f030 con 0x7ff5e406eed0 2024-09-17T04:06:07.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.929+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5e406a140 con 0x7ff5e406eed0 2024-09-17T04:06:07.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.930+0000 7ff5e1600640 1 -- 172.21.15.129:0/3005773378 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) ==== 74+0+16735 (secure 0 0 0) 0x7ff5d4084330 con 0x7ff5e406eed0 2024-09-17T04:06:07.930 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:07.931 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":233,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:04.786130+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:01.171609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:02.239670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 msgr2=0x7ff5c007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 0x7ff5c007b0d0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff5e41685d0 tx=0x7ff5cc009040 comp rx=0 tx=0).stop 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 msgr2=0x7ff5e4166e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009980 tx=0x7ff5d4033ba0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 shutdown_connections 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff5c0078bc0 0x7ff5c007b0d0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff5e415c980 0x7ff5e40ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff5e40705a0 0x7ff5e4167390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.932+0000 7ff5eb800640 1 --2- 172.21.15.129:0/3005773378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff5e406eed0 0x7ff5e4166e50 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.933+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 >> 172.21.15.129:0/3005773378 conn(0x7ff5e406d410 msgr2=0x7ff5e406f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.933+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 shutdown_connections 2024-09-17T04:06:07.934 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:07.933+0000 7ff5eb800640 1 -- 172.21.15.129:0/3005773378 wait complete. 2024-09-17T04:06:07.955 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:06:07.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 -- 172.21.15.129:0/2794979797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a9006eed0 msgr2=0x7f2a900ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 --2- 172.21.15.129:0/2794979797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a9006eed0 0x7f2a900ccc80 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c005fd0 tx=0x7f2a7c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 -- 172.21.15.129:0/2794979797 shutdown_connections 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 --2- 172.21.15.129:0/2794979797 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a9006eed0 0x7f2a900ccc80 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 --2- 172.21.15.129:0/2794979797 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a900705a0 0x7f2a900b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 -- 172.21.15.129:0/2794979797 >> 172.21.15.129:0/2794979797 conn(0x7f2a9006d410 msgr2=0x7f2a900b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 -- 172.21.15.129:0/2794979797 shutdown_connections 2024-09-17T04:06:08.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.174+0000 7f2a97e00640 1 -- 172.21.15.129:0/2794979797 wait complete. 2024-09-17T04:06:08.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.176+0000 7f2a97e00640 1 Processor -- start 2024-09-17T04:06:08.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.177+0000 7f2a97e00640 1 -- start start 2024-09-17T04:06:08.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a9006eed0 0x7f2a9015de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a9006f2e0 0x7f2a9008bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a90160200 con 0x7f2a900705a0 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a901612e0 con 0x7f2a9006eed0 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a97e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2a900cd1c0 con 0x7f2a9006f2e0 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a95e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a95e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51938/0 (socket says 172.21.15.129:51938) 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.178+0000 7f2a95e00640 1 -- 172.21.15.129:0/3236604969 learned_addr learned my addr 172.21.15.129:0/3236604969 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95400640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a9006eed0 0x7f2a9015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a94a00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a9006f2e0 0x7f2a9008bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 -- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a9006f2e0 msgr2=0x7f2a9008bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a9006f2e0 0x7f2a9008bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 -- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a9006eed0 msgr2=0x7f2a9015de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a9006eed0 0x7f2a9015de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 -- 172.21.15.129:0/3236604969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a901f70a0 con 0x7f2a900705a0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a95e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800c390 tx=0x7f2a8800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.179+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88017070 con 0x7f2a900705a0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.180+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a880053c0 con 0x7f2a900705a0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.180+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a901f7380 con 0x7f2a900705a0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.180+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88003830 con 0x7f2a900705a0 2024-09-17T04:06:08.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.180+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a901f7940 con 0x7f2a900705a0 2024-09-17T04:06:08.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.181+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9008ca90 con 0x7f2a900705a0 2024-09-17T04:06:08.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.182+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a8801a070 con 0x7f2a900705a0 2024-09-17T04:06:08.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.183+0000 7f2a8d600640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 0x7f2a7007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.183+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f2a88099fc0 con 0x7f2a900705a0 2024-09-17T04:06:08.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.183+0000 7f2a95400640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 0x7f2a7007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.184+0000 7f2a95400640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 0x7f2a7007b0a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005fd0 tx=0x7f2a840095e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:08.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.187+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a88061db0 con 0x7f2a900705a0 2024-09-17T04:06:08.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.368+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2a901f6e50 con 0x7f2a70078b90 2024-09-17T04:06:08.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.373+0000 7f2a8d600640 1 -- 172.21.15.129:0/3236604969 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250735 (secure 0 0 0) 0x7f2a901f6e50 con 0x7f2a70078b90 2024-09-17T04:06:08.376 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:08.378 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.377+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 msgr2=0x7f2a7007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.377+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 0x7f2a7007b0a0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f2a84005fd0 tx=0x7f2a840095e0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.377+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 msgr2=0x7f2a900b0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.377+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800c390 tx=0x7f2a8800c850 comp rx=0 tx=0).stop 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 shutdown_connections 2024-09-17T04:06:08.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a70078b90 0x7f2a7007b0a0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a900705a0 0x7f2a900b0a40 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a9006f2e0 0x7f2a9008bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 --2- 172.21.15.129:0/3236604969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a9006eed0 0x7f2a9015de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 >> 172.21.15.129:0/3236604969 conn(0x7f2a9006d410 msgr2=0x7f2a900d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 shutdown_connections 2024-09-17T04:06:08.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.378+0000 7f2a97e00640 1 -- 172.21.15.129:0/3236604969 wait complete. 2024-09-17T04:06:08.403 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-09-17T04:06:06.579654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":137,"num_write_kb":1641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404068,"kb_used_data":24048,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754570652,"statfs":{"total":773094113280,"available":772680347648,"internally_reserved":0,"allocated":24625152,"data_stored":11539198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.809110"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.786688+0000","last_change":"2024-09-17T04:06:02.786688+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.786688+0000","last_undegraded":"2024-09-17T04:06:02.786688+0000","last_fullsized":"2024-09-17T04:06:02.786688+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813384+0000","last_change":"2024-09-17T04:06:03.806736+0000","last_active":"2024-09-17T04:06:03.813384+0000","last_peered":"2024-09-17T04:06:03.813384+0000","last_clean":"2024-09-17T04:06:03.813384+0000","last_became_active":"2024-09-17T04:06:03.806423+0000","last_became_peered":"2024-09-17T04:06:03.806423+0000","last_unstale":"2024-09-17T04:06:03.813384+0000","last_undegraded":"2024-09-17T04:06:03.813384+0000","last_fullsized":"2024-09-17T04:06:03.813384+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:47.077633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801873+0000","last_change":"2024-09-17T04:06:03.796399+0000","last_active":"2024-09-17T04:06:03.801873+0000","last_peered":"2024-09-17T04:06:03.801873+0000","last_clean":"2024-09-17T04:06:03.801873+0000","last_became_active":"2024-09-17T04:06:03.796105+0000","last_became_peered":"2024-09-17T04:06:03.796105+0000","last_unstale":"2024-09-17T04:06:03.801873+0000","last_undegraded":"2024-09-17T04:06:03.801873+0000","last_fullsized":"2024-09-17T04:06:03.801873+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:12.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.819203+0000","last_change":"2024-09-17T04:06:03.817407+0000","last_active":"2024-09-17T04:06:03.819203+0000","last_peered":"2024-09-17T04:06:03.819203+0000","last_clean":"2024-09-17T04:06:03.819203+0000","last_became_active":"2024-09-17T04:06:03.817010+0000","last_became_peered":"2024-09-17T04:06:03.817010+0000","last_unstale":"2024-09-17T04:06:03.819203+0000","last_undegraded":"2024-09-17T04:06:03.819203+0000","last_fullsized":"2024-09-17T04:06:03.819203+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:47.218078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813775+0000","last_change":"2024-09-17T04:06:03.808299+0000","last_active":"2024-09-17T04:06:03.813775+0000","last_peered":"2024-09-17T04:06:03.813775+0000","last_clean":"2024-09-17T04:06:03.813775+0000","last_became_active":"2024-09-17T04:06:03.806953+0000","last_became_peered":"2024-09-17T04:06:03.806953+0000","last_unstale":"2024-09-17T04:06:03.813775+0000","last_undegraded":"2024-09-17T04:06:03.813775+0000","last_fullsized":"2024-09-17T04:06:03.813775+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:25.478342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813806+0000","last_change":"2024-09-17T04:06:03.807167+0000","last_active":"2024-09-17T04:06:03.813806+0000","last_peered":"2024-09-17T04:06:03.813806+0000","last_clean":"2024-09-17T04:06:03.813806+0000","last_became_active":"2024-09-17T04:06:03.806876+0000","last_became_peered":"2024-09-17T04:06:03.806876+0000","last_unstale":"2024-09-17T04:06:03.813806+0000","last_undegraded":"2024-09-17T04:06:03.813806+0000","last_fullsized":"2024-09-17T04:06:03.813806+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:52.692494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.786073+0000","last_change":"2024-09-17T04:06:02.786073+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.786073+0000","last_undegraded":"2024-09-17T04:06:02.786073+0000","last_fullsized":"2024-09-17T04:06:02.786073+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.782946+0000","last_change":"2024-09-17T04:06:02.782946+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.782946+0000","last_undegraded":"2024-09-17T04:06:02.782946+0000","last_fullsized":"2024-09-17T04:06:02.782946+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817820+0000","last_change":"2024-09-17T04:06:03.816441+0000","last_active":"2024-09-17T04:06:03.817820+0000","last_peered":"2024-09-17T04:06:03.817820+0000","last_clean":"2024-09-17T04:06:03.817820+0000","last_became_active":"2024-09-17T04:06:03.816103+0000","last_became_peered":"2024-09-17T04:06:03.816103+0000","last_unstale":"2024-09-17T04:06:03.817820+0000","last_undegraded":"2024-09-17T04:06:03.817820+0000","last_fullsized":"2024-09-17T04:06:03.817820+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:25:00.425180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817345+0000","last_change":"2024-09-17T04:06:03.810393+0000","last_active":"2024-09-17T04:06:03.817345+0000","last_peered":"2024-09-17T04:06:03.817345+0000","last_clean":"2024-09-17T04:06:03.817345+0000","last_became_active":"2024-09-17T04:06:03.805893+0000","last_became_peered":"2024-09-17T04:06:03.805893+0000","last_unstale":"2024-09-17T04:06:03.817345+0000","last_undegraded":"2024-09-17T04:06:03.817345+0000","last_fullsized":"2024-09-17T04:06:03.817345+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:45.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817661+0000","last_change":"2024-09-17T04:06:03.815419+0000","last_active":"2024-09-17T04:06:03.817661+0000","last_peered":"2024-09-17T04:06:03.817661+0000","last_clean":"2024-09-17T04:06:03.817661+0000","last_became_active":"2024-09-17T04:06:03.815077+0000","last_became_peered":"2024-09-17T04:06:03.815077+0000","last_unstale":"2024-09-17T04:06:03.817661+0000","last_undegraded":"2024-09-17T04:06:03.817661+0000","last_fullsized":"2024-09-17T04:06:03.817661+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:44.349015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.786991+0000","last_change":"2024-09-17T04:06:02.786991+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.786991+0000","last_undegraded":"2024-09-17T04:06:02.786991+0000","last_fullsized":"2024-09-17T04:06:02.786991+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.786117+0000","last_change":"2024-09-17T04:06:02.786117+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.786117+0000","last_undegraded":"2024-09-17T04:06:02.786117+0000","last_fullsized":"2024-09-17T04:06:02.786117+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813847+0000","last_change":"2024-09-17T04:06:03.809183+0000","last_active":"2024-09-17T04:06:03.813847+0000","last_peered":"2024-09-17T04:06:03.813847+0000","last_clean":"2024-09-17T04:06:03.813847+0000","last_became_active":"2024-09-17T04:06:03.807957+0000","last_became_peered":"2024-09-17T04:06:03.807957+0000","last_unstale":"2024-09-17T04:06:03.813847+0000","last_undegraded":"2024-09-17T04:06:03.813847+0000","last_fullsized":"2024-09-17T04:06:03.813847+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:01.300956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.784658+0000","last_change":"2024-09-17T04:06:02.784658+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.784658+0000","last_undegraded":"2024-09-17T04:06:02.784658+0000","last_fullsized":"2024-09-17T04:06:02.784658+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781265+0000","last_change":"2024-09-17T04:06:02.781265+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781265+0000","last_undegraded":"2024-09-17T04:06:02.781265+0000","last_fullsized":"2024-09-17T04:06:02.781265+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.805508+0000","last_change":"2024-09-17T04:06:03.797198+0000","last_active":"2024-09-17T04:06:03.805508+0000","last_peered":"2024-09-17T04:06:03.805508+0000","last_clean":"2024-09-17T04:06:03.805508+0000","last_became_active":"2024-09-17T04:06:03.796885+0000","last_became_peered":"2024-09-17T04:06:03.796885+0000","last_unstale":"2024-09-17T04:06:03.805508+0000","last_undegraded":"2024-09-17T04:06:03.805508+0000","last_fullsized":"2024-09-17T04:06:03.805508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:06.138948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811501+0000","last_change":"2024-09-17T04:06:03.801881+0000","last_active":"2024-09-17T04:06:03.811501+0000","last_peered":"2024-09-17T04:06:03.811501+0000","last_clean":"2024-09-17T04:06:03.811501+0000","last_became_active":"2024-09-17T04:06:03.801360+0000","last_became_peered":"2024-09-17T04:06:03.801360+0000","last_unstale":"2024-09-17T04:06:03.811501+0000","last_undegraded":"2024-09-17T04:06:03.811501+0000","last_fullsized":"2024-09-17T04:06:03.811501+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:23.901868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.782545+0000","last_change":"2024-09-17T04:06:02.782545+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.782545+0000","last_undegraded":"2024-09-17T04:06:02.782545+0000","last_fullsized":"2024-09-17T04:06:02.782545+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.782363+0000","last_change":"2024-09-17T04:06:02.782363+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.782363+0000","last_undegraded":"2024-09-17T04:06:02.782363+0000","last_fullsized":"2024-09-17T04:06:02.782363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.785504+0000","last_change":"2024-09-17T04:06:02.785504+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.785504+0000","last_undegraded":"2024-09-17T04:06:02.785504+0000","last_fullsized":"2024-09-17T04:06:02.785504+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.799908+0000","last_change":"2024-09-17T04:06:03.795897+0000","last_active":"2024-09-17T04:06:03.799908+0000","last_peered":"2024-09-17T04:06:03.799908+0000","last_clean":"2024-09-17T04:06:03.799908+0000","last_became_active":"2024-09-17T04:06:03.795418+0000","last_became_peered":"2024-09-17T04:06:03.795418+0000","last_unstale":"2024-09-17T04:06:03.799908+0000","last_undegraded":"2024-09-17T04:06:03.799908+0000","last_fullsized":"2024-09-17T04:06:03.799908+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:17.494967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.810107+0000","last_change":"2024-09-17T04:06:03.805850+0000","last_active":"2024-09-17T04:06:03.810107+0000","last_peered":"2024-09-17T04:06:03.810107+0000","last_clean":"2024-09-17T04:06:03.810107+0000","last_became_active":"2024-09-17T04:06:03.804799+0000","last_became_peered":"2024-09-17T04:06:03.804799+0000","last_unstale":"2024-09-17T04:06:03.810107+0000","last_undegraded":"2024-09-17T04:06:03.810107+0000","last_fullsized":"2024-09-17T04:06:03.810107+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:55.693687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802306+0000","last_change":"2024-09-17T04:06:03.797753+0000","last_active":"2024-09-17T04:06:03.802306+0000","last_peered":"2024-09-17T04:06:03.802306+0000","last_clean":"2024-09-17T04:06:03.802306+0000","last_became_active":"2024-09-17T04:06:03.797408+0000","last_became_peered":"2024-09-17T04:06:03.797408+0000","last_unstale":"2024-09-17T04:06:03.802306+0000","last_undegraded":"2024-09-17T04:06:03.802306+0000","last_fullsized":"2024-09-17T04:06:03.802306+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:59.498210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.810797+0000","last_change":"2024-09-17T04:06:03.806787+0000","last_active":"2024-09-17T04:06:03.810797+0000","last_peered":"2024-09-17T04:06:03.810797+0000","last_clean":"2024-09-17T04:06:03.810797+0000","last_became_active":"2024-09-17T04:06:03.806508+0000","last_became_peered":"2024-09-17T04:06:03.806508+0000","last_unstale":"2024-09-17T04:06:03.810797+0000","last_undegraded":"2024-09-17T04:06:03.810797+0000","last_fullsized":"2024-09-17T04:06:03.810797+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:42.486394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.789092+0000","last_change":"2024-09-17T04:06:02.789092+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.789092+0000","last_undegraded":"2024-09-17T04:06:02.789092+0000","last_fullsized":"2024-09-17T04:06:02.789092+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781553+0000","last_change":"2024-09-17T04:06:02.781553+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781553+0000","last_undegraded":"2024-09-17T04:06:02.781553+0000","last_fullsized":"2024-09-17T04:06:02.781553+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812154+0000","last_change":"2024-09-17T04:06:03.799602+0000","last_active":"2024-09-17T04:06:03.812154+0000","last_peered":"2024-09-17T04:06:03.812154+0000","last_clean":"2024-09-17T04:06:03.812154+0000","last_became_active":"2024-09-17T04:06:03.799302+0000","last_became_peered":"2024-09-17T04:06:03.799302+0000","last_unstale":"2024-09-17T04:06:03.812154+0000","last_undegraded":"2024-09-17T04:06:03.812154+0000","last_fullsized":"2024-09-17T04:06:03.812154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:08.963673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.783126+0000","last_change":"2024-09-17T04:06:02.783126+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.783126+0000","last_undegraded":"2024-09-17T04:06:02.783126+0000","last_fullsized":"2024-09-17T04:06:02.783126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.784111+0000","last_change":"2024-09-17T04:06:02.784111+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.784111+0000","last_undegraded":"2024-09-17T04:06:02.784111+0000","last_fullsized":"2024-09-17T04:06:02.784111+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806485+0000","last_change":"2024-09-17T04:06:03.804093+0000","last_active":"2024-09-17T04:06:03.806485+0000","last_peered":"2024-09-17T04:06:03.806485+0000","last_clean":"2024-09-17T04:06:03.806485+0000","last_became_active":"2024-09-17T04:06:03.803367+0000","last_became_peered":"2024-09-17T04:06:03.803367+0000","last_unstale":"2024-09-17T04:06:03.806485+0000","last_undegraded":"2024-09-17T04:06:03.806485+0000","last_fullsized":"2024-09-17T04:06:03.806485+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:53.134700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.786199+0000","last_change":"2024-09-17T04:06:02.786199+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.786199+0000","last_undegraded":"2024-09-17T04:06:02.786199+0000","last_fullsized":"2024-09-17T04:06:02.786199+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.805991+0000","last_change":"2024-09-17T04:06:02.803265+0000","last_active":"2024-09-17T04:06:02.805991+0000","last_peered":"2024-09-17T04:06:02.805991+0000","last_clean":"2024-09-17T04:06:02.805991+0000","last_became_active":"2024-09-17T04:06:02.801302+0000","last_became_peered":"2024-09-17T04:06:02.801302+0000","last_unstale":"2024-09-17T04:06:02.805991+0000","last_undegraded":"2024-09-17T04:06:02.805991+0000","last_fullsized":"2024-09-17T04:06:02.805991+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:30.205532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817624+0000","last_change":"2024-09-17T04:06:03.815673+0000","last_active":"2024-09-17T04:06:03.817624+0000","last_peered":"2024-09-17T04:06:03.817624+0000","last_clean":"2024-09-17T04:06:03.817624+0000","last_became_active":"2024-09-17T04:06:03.815312+0000","last_became_peered":"2024-09-17T04:06:03.815312+0000","last_unstale":"2024-09-17T04:06:03.817624+0000","last_undegraded":"2024-09-17T04:06:03.817624+0000","last_fullsized":"2024-09-17T04:06:03.817624+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:09.824297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.805381+0000","last_change":"2024-09-17T04:06:02.797312+0000","last_active":"2024-09-17T04:06:03.805381+0000","last_peered":"2024-09-17T04:06:03.805381+0000","last_clean":"2024-09-17T04:06:03.805381+0000","last_became_active":"2024-09-17T04:06:02.795549+0000","last_became_peered":"2024-09-17T04:06:02.795549+0000","last_unstale":"2024-09-17T04:06:03.805381+0000","last_undegraded":"2024-09-17T04:06:03.805381+0000","last_fullsized":"2024-09-17T04:06:03.805381+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:54.691901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817228+0000","last_change":"2024-09-17T04:06:03.812526+0000","last_active":"2024-09-17T04:06:03.817228+0000","last_peered":"2024-09-17T04:06:03.817228+0000","last_clean":"2024-09-17T04:06:03.817228+0000","last_became_active":"2024-09-17T04:06:03.812263+0000","last_became_peered":"2024-09-17T04:06:03.812263+0000","last_unstale":"2024-09-17T04:06:03.817228+0000","last_undegraded":"2024-09-17T04:06:03.817228+0000","last_fullsized":"2024-09-17T04:06:03.817228+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:10.219033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816029+0000","last_change":"2024-09-17T04:06:02.802766+0000","last_active":"2024-09-17T04:06:03.816029+0000","last_peered":"2024-09-17T04:06:03.816029+0000","last_clean":"2024-09-17T04:06:03.816029+0000","last_became_active":"2024-09-17T04:06:02.802486+0000","last_became_peered":"2024-09-17T04:06:02.802486+0000","last_unstale":"2024-09-17T04:06:03.816029+0000","last_undegraded":"2024-09-17T04:06:03.816029+0000","last_fullsized":"2024-09-17T04:06:03.816029+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:29:56.862206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.814528+0000","last_change":"2024-09-17T04:06:03.811870+0000","last_active":"2024-09-17T04:06:03.814528+0000","last_peered":"2024-09-17T04:06:03.814528+0000","last_clean":"2024-09-17T04:06:03.814528+0000","last_became_active":"2024-09-17T04:06:03.811552+0000","last_became_peered":"2024-09-17T04:06:03.811552+0000","last_unstale":"2024-09-17T04:06:03.814528+0000","last_undegraded":"2024-09-17T04:06:03.814528+0000","last_fullsized":"2024-09-17T04:06:03.814528+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:14.281666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.802921+0000","last_change":"2024-09-17T04:06:02.801393+0000","last_active":"2024-09-17T04:06:02.802921+0000","last_peered":"2024-09-17T04:06:02.802921+0000","last_clean":"2024-09-17T04:06:02.802921+0000","last_became_active":"2024-09-17T04:06:02.800404+0000","last_became_peered":"2024-09-17T04:06:02.800404+0000","last_unstale":"2024-09-17T04:06:02.802921+0000","last_undegraded":"2024-09-17T04:06:02.802921+0000","last_fullsized":"2024-09-17T04:06:02.802921+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:30.619442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808540+0000","last_change":"2024-09-17T04:06:03.802716+0000","last_active":"2024-09-17T04:06:03.808540+0000","last_peered":"2024-09-17T04:06:03.808540+0000","last_clean":"2024-09-17T04:06:03.808540+0000","last_became_active":"2024-09-17T04:06:03.802367+0000","last_became_peered":"2024-09-17T04:06:03.802367+0000","last_unstale":"2024-09-17T04:06:03.808540+0000","last_undegraded":"2024-09-17T04:06:03.808540+0000","last_fullsized":"2024-09-17T04:06:03.808540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:54.201988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816774+0000","last_change":"2024-09-17T04:06:02.792176+0000","last_active":"2024-09-17T04:06:03.816774+0000","last_peered":"2024-09-17T04:06:03.816774+0000","last_clean":"2024-09-17T04:06:03.816774+0000","last_became_active":"2024-09-17T04:06:02.791849+0000","last_became_peered":"2024-09-17T04:06:02.791849+0000","last_unstale":"2024-09-17T04:06:03.816774+0000","last_undegraded":"2024-09-17T04:06:03.816774+0000","last_fullsized":"2024-09-17T04:06:03.816774+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:20:37.125910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.782660+0000","last_change":"2024-09-17T04:06:02.782660+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.782660+0000","last_undegraded":"2024-09-17T04:06:02.782660+0000","last_fullsized":"2024-09-17T04:06:02.782660+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817698+0000","last_change":"2024-09-17T04:06:02.796869+0000","last_active":"2024-09-17T04:06:03.817698+0000","last_peered":"2024-09-17T04:06:03.817698+0000","last_clean":"2024-09-17T04:06:03.817698+0000","last_became_active":"2024-09-17T04:06:02.796123+0000","last_became_peered":"2024-09-17T04:06:02.796123+0000","last_unstale":"2024-09-17T04:06:03.817698+0000","last_undegraded":"2024-09-17T04:06:03.817698+0000","last_fullsized":"2024-09-17T04:06:03.817698+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:41.930024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806454+0000","last_change":"2024-09-17T04:06:03.803582+0000","last_active":"2024-09-17T04:06:03.806454+0000","last_peered":"2024-09-17T04:06:03.806454+0000","last_clean":"2024-09-17T04:06:03.806454+0000","last_became_active":"2024-09-17T04:06:03.803283+0000","last_became_peered":"2024-09-17T04:06:03.803283+0000","last_unstale":"2024-09-17T04:06:03.806454+0000","last_undegraded":"2024-09-17T04:06:03.806454+0000","last_fullsized":"2024-09-17T04:06:03.806454+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:47.805456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.805029+0000","last_change":"2024-09-17T04:06:02.802659+0000","last_active":"2024-09-17T04:06:02.805029+0000","last_peered":"2024-09-17T04:06:02.805029+0000","last_clean":"2024-09-17T04:06:02.805029+0000","last_became_active":"2024-09-17T04:06:02.802349+0000","last_became_peered":"2024-09-17T04:06:02.802349+0000","last_unstale":"2024-09-17T04:06:02.805029+0000","last_undegraded":"2024-09-17T04:06:02.805029+0000","last_fullsized":"2024-09-17T04:06:02.805029+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:38:55.943232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.803766+0000","last_change":"2024-09-17T04:06:03.793281+0000","last_active":"2024-09-17T04:06:03.803766+0000","last_peered":"2024-09-17T04:06:03.803766+0000","last_clean":"2024-09-17T04:06:03.803766+0000","last_became_active":"2024-09-17T04:06:03.791163+0000","last_became_peered":"2024-09-17T04:06:03.791163+0000","last_unstale":"2024-09-17T04:06:03.803766+0000","last_undegraded":"2024-09-17T04:06:03.803766+0000","last_fullsized":"2024-09-17T04:06:03.803766+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:48.471016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.799621+0000","last_change":"2024-09-17T04:06:02.796371+0000","last_active":"2024-09-17T04:06:02.799621+0000","last_peered":"2024-09-17T04:06:02.799621+0000","last_clean":"2024-09-17T04:06:02.799621+0000","last_became_active":"2024-09-17T04:06:02.796081+0000","last_became_peered":"2024-09-17T04:06:02.796081+0000","last_unstale":"2024-09-17T04:06:02.799621+0000","last_undegraded":"2024-09-17T04:06:02.799621+0000","last_fullsized":"2024-09-17T04:06:02.799621+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:38:23.195561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801703+0000","last_change":"2024-09-17T04:06:03.795816+0000","last_active":"2024-09-17T04:06:03.801703+0000","last_peered":"2024-09-17T04:06:03.801703+0000","last_clean":"2024-09-17T04:06:03.801703+0000","last_became_active":"2024-09-17T04:06:03.795543+0000","last_became_peered":"2024-09-17T04:06:03.795543+0000","last_unstale":"2024-09-17T04:06:03.801703+0000","last_undegraded":"2024-09-17T04:06:03.801703+0000","last_fullsized":"2024-09-17T04:06:03.801703+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:15.130160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808033+0000","last_change":"2024-09-17T04:06:02.803758+0000","last_active":"2024-09-17T04:06:03.808033+0000","last_peered":"2024-09-17T04:06:03.808033+0000","last_clean":"2024-09-17T04:06:03.808033+0000","last_became_active":"2024-09-17T04:06:02.800956+0000","last_became_peered":"2024-09-17T04:06:02.800956+0000","last_unstale":"2024-09-17T04:06:03.808033+0000","last_undegraded":"2024-09-17T04:06:03.808033+0000","last_fullsized":"2024-09-17T04:06:03.808033+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:03.191732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812214+0000","last_change":"2024-09-17T04:06:03.805584+0000","last_active":"2024-09-17T04:06:03.812214+0000","last_peered":"2024-09-17T04:06:03.812214+0000","last_clean":"2024-09-17T04:06:03.812214+0000","last_became_active":"2024-09-17T04:06:03.805253+0000","last_became_peered":"2024-09-17T04:06:03.805253+0000","last_unstale":"2024-09-17T04:06:03.812214+0000","last_undegraded":"2024-09-17T04:06:03.812214+0000","last_fullsized":"2024-09-17T04:06:03.812214+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:58.700136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801101+0000","last_change":"2024-09-17T04:06:02.809561+0000","last_active":"2024-09-17T04:06:03.801101+0000","last_peered":"2024-09-17T04:06:03.801101+0000","last_clean":"2024-09-17T04:06:03.801101+0000","last_became_active":"2024-09-17T04:06:02.809205+0000","last_became_peered":"2024-09-17T04:06:02.809205+0000","last_unstale":"2024-09-17T04:06:03.801101+0000","last_undegraded":"2024-09-17T04:06:03.801101+0000","last_fullsized":"2024-09-17T04:06:03.801101+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:57.306478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.822844+0000","last_change":"2024-09-17T04:06:03.821651+0000","last_active":"2024-09-17T04:06:03.822844+0000","last_peered":"2024-09-17T04:06:03.822844+0000","last_clean":"2024-09-17T04:06:03.822844+0000","last_became_active":"2024-09-17T04:06:03.821041+0000","last_became_peered":"2024-09-17T04:06:03.821041+0000","last_unstale":"2024-09-17T04:06:03.822844+0000","last_undegraded":"2024-09-17T04:06:03.822844+0000","last_fullsized":"2024-09-17T04:06:03.822844+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.264621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"232'3","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815889+0000","last_change":"2024-09-17T04:06:02.796662+0000","last_active":"2024-09-17T04:06:03.815889+0000","last_peered":"2024-09-17T04:06:03.815889+0000","last_clean":"2024-09-17T04:06:03.815889+0000","last_became_active":"2024-09-17T04:06:02.796336+0000","last_became_peered":"2024-09-17T04:06:02.796336+0000","last_unstale":"2024-09-17T04:06:03.815889+0000","last_undegraded":"2024-09-17T04:06:03.815889+0000","last_fullsized":"2024-09-17T04:06:03.815889+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:12.806260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818666+0000","last_change":"2024-09-17T04:06:03.810719+0000","last_active":"2024-09-17T04:06:03.818666+0000","last_peered":"2024-09-17T04:06:03.818666+0000","last_clean":"2024-09-17T04:06:03.818666+0000","last_became_active":"2024-09-17T04:06:03.807985+0000","last_became_peered":"2024-09-17T04:06:03.807985+0000","last_unstale":"2024-09-17T04:06:03.818666+0000","last_undegraded":"2024-09-17T04:06:03.818666+0000","last_fullsized":"2024-09-17T04:06:03.818666+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:13.580410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"232'6","reported_seq":19,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.824178+0000","last_change":"2024-09-17T04:06:02.797156+0000","last_active":"2024-09-17T04:06:03.824178+0000","last_peered":"2024-09-17T04:06:03.824178+0000","last_clean":"2024-09-17T04:06:03.824178+0000","last_became_active":"2024-09-17T04:06:02.796817+0000","last_became_peered":"2024-09-17T04:06:02.796817+0000","last_unstale":"2024-09-17T04:06:03.824178+0000","last_undegraded":"2024-09-17T04:06:03.824178+0000","last_fullsized":"2024-09-17T04:06:03.824178+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:12.025611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781229+0000","last_change":"2024-09-17T04:06:02.781229+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781229+0000","last_undegraded":"2024-09-17T04:06:02.781229+0000","last_fullsized":"2024-09-17T04:06:02.781229+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815281+0000","last_change":"2024-09-17T04:06:02.793880+0000","last_active":"2024-09-17T04:06:03.815281+0000","last_peered":"2024-09-17T04:06:03.815281+0000","last_clean":"2024-09-17T04:06:03.815281+0000","last_became_active":"2024-09-17T04:06:02.792660+0000","last_became_peered":"2024-09-17T04:06:02.792660+0000","last_unstale":"2024-09-17T04:06:03.815281+0000","last_undegraded":"2024-09-17T04:06:03.815281+0000","last_fullsized":"2024-09-17T04:06:03.815281+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:44.744990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816348+0000","last_change":"2024-09-17T04:06:03.813764+0000","last_active":"2024-09-17T04:06:03.816348+0000","last_peered":"2024-09-17T04:06:03.816348+0000","last_clean":"2024-09-17T04:06:03.816348+0000","last_became_active":"2024-09-17T04:06:03.812094+0000","last_became_peered":"2024-09-17T04:06:03.812094+0000","last_unstale":"2024-09-17T04:06:03.816348+0000","last_undegraded":"2024-09-17T04:06:03.816348+0000","last_fullsized":"2024-09-17T04:06:03.816348+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.749045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.787062+0000","last_change":"2024-09-17T04:06:02.800294+0000","last_active":"2024-09-17T04:06:03.787062+0000","last_peered":"2024-09-17T04:06:03.787062+0000","last_clean":"2024-09-17T04:06:03.787062+0000","last_became_active":"2024-09-17T04:06:02.799725+0000","last_became_peered":"2024-09-17T04:06:02.799725+0000","last_unstale":"2024-09-17T04:06:03.787062+0000","last_undegraded":"2024-09-17T04:06:03.787062+0000","last_fullsized":"2024-09-17T04:06:03.787062+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:54.285672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808478+0000","last_change":"2024-09-17T04:06:03.807107+0000","last_active":"2024-09-17T04:06:03.808478+0000","last_peered":"2024-09-17T04:06:03.808478+0000","last_clean":"2024-09-17T04:06:03.808478+0000","last_became_active":"2024-09-17T04:06:03.806775+0000","last_became_peered":"2024-09-17T04:06:03.806775+0000","last_unstale":"2024-09-17T04:06:03.808478+0000","last_undegraded":"2024-09-17T04:06:03.808478+0000","last_fullsized":"2024-09-17T04:06:03.808478+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:30.210756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802616+0000","last_change":"2024-09-17T04:06:02.802486+0000","last_active":"2024-09-17T04:06:03.802616+0000","last_peered":"2024-09-17T04:06:03.802616+0000","last_clean":"2024-09-17T04:06:03.802616+0000","last_became_active":"2024-09-17T04:06:02.801193+0000","last_became_peered":"2024-09-17T04:06:02.801193+0000","last_unstale":"2024-09-17T04:06:03.802616+0000","last_undegraded":"2024-09-17T04:06:03.802616+0000","last_fullsized":"2024-09-17T04:06:03.802616+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:56.111417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781271+0000","last_change":"2024-09-17T04:06:02.781271+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781271+0000","last_undegraded":"2024-09-17T04:06:02.781271+0000","last_fullsized":"2024-09-17T04:06:02.781271+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"232'8","reported_seq":21,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817461+0000","last_change":"2024-09-17T04:06:02.794914+0000","last_active":"2024-09-17T04:06:03.817461+0000","last_peered":"2024-09-17T04:06:03.817461+0000","last_clean":"2024-09-17T04:06:03.817461+0000","last_became_active":"2024-09-17T04:06:02.794626+0000","last_became_peered":"2024-09-17T04:06:02.794626+0000","last_unstale":"2024-09-17T04:06:03.817461+0000","last_undegraded":"2024-09-17T04:06:03.817461+0000","last_fullsized":"2024-09-17T04:06:03.817461+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:28.405822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796154+0000","last_change":"2024-09-17T04:06:03.795419+0000","last_active":"2024-09-17T04:06:03.796154+0000","last_peered":"2024-09-17T04:06:03.796154+0000","last_clean":"2024-09-17T04:06:03.796154+0000","last_became_active":"2024-09-17T04:06:03.795073+0000","last_became_peered":"2024-09-17T04:06:03.795073+0000","last_unstale":"2024-09-17T04:06:03.796154+0000","last_undegraded":"2024-09-17T04:06:03.796154+0000","last_fullsized":"2024-09-17T04:06:03.796154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:55.646893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.821357+0000","last_change":"2024-09-17T04:06:02.805998+0000","last_active":"2024-09-17T04:06:03.821357+0000","last_peered":"2024-09-17T04:06:03.821357+0000","last_clean":"2024-09-17T04:06:03.821357+0000","last_became_active":"2024-09-17T04:06:02.803503+0000","last_became_peered":"2024-09-17T04:06:02.803503+0000","last_unstale":"2024-09-17T04:06:03.821357+0000","last_undegraded":"2024-09-17T04:06:03.821357+0000","last_fullsized":"2024-09-17T04:06:03.821357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:18.522548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796057+0000","last_change":"2024-09-17T04:06:03.793336+0000","last_active":"2024-09-17T04:06:03.796057+0000","last_peered":"2024-09-17T04:06:03.796057+0000","last_clean":"2024-09-17T04:06:03.796057+0000","last_became_active":"2024-09-17T04:06:03.793040+0000","last_became_peered":"2024-09-17T04:06:03.793040+0000","last_unstale":"2024-09-17T04:06:03.796057+0000","last_undegraded":"2024-09-17T04:06:03.796057+0000","last_fullsized":"2024-09-17T04:06:03.796057+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:09.417510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.819357+0000","last_change":"2024-09-17T04:06:02.801756+0000","last_active":"2024-09-17T04:06:03.819357+0000","last_peered":"2024-09-17T04:06:03.819357+0000","last_clean":"2024-09-17T04:06:03.819357+0000","last_became_active":"2024-09-17T04:06:02.801436+0000","last_became_peered":"2024-09-17T04:06:02.801436+0000","last_unstale":"2024-09-17T04:06:03.819357+0000","last_undegraded":"2024-09-17T04:06:03.819357+0000","last_fullsized":"2024-09-17T04:06:03.819357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:37.941773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815106+0000","last_change":"2024-09-17T04:06:03.810169+0000","last_active":"2024-09-17T04:06:03.815106+0000","last_peered":"2024-09-17T04:06:03.815106+0000","last_clean":"2024-09-17T04:06:03.815106+0000","last_became_active":"2024-09-17T04:06:03.809843+0000","last_became_peered":"2024-09-17T04:06:03.809843+0000","last_unstale":"2024-09-17T04:06:03.815106+0000","last_undegraded":"2024-09-17T04:06:03.815106+0000","last_fullsized":"2024-09-17T04:06:03.815106+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:17.652978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818962+0000","last_change":"2024-09-17T04:06:02.807475+0000","last_active":"2024-09-17T04:06:03.818962+0000","last_peered":"2024-09-17T04:06:03.818962+0000","last_clean":"2024-09-17T04:06:03.818962+0000","last_became_active":"2024-09-17T04:06:02.806218+0000","last_became_peered":"2024-09-17T04:06:02.806218+0000","last_unstale":"2024-09-17T04:06:03.818962+0000","last_undegraded":"2024-09-17T04:06:03.818962+0000","last_fullsized":"2024-09-17T04:06:03.818962+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:50.595970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813736+0000","last_change":"2024-09-17T04:06:03.806964+0000","last_active":"2024-09-17T04:06:03.813736+0000","last_peered":"2024-09-17T04:06:03.813736+0000","last_clean":"2024-09-17T04:06:03.813736+0000","last_became_active":"2024-09-17T04:06:03.806679+0000","last_became_peered":"2024-09-17T04:06:03.806679+0000","last_unstale":"2024-09-17T04:06:03.813736+0000","last_undegraded":"2024-09-17T04:06:03.813736+0000","last_fullsized":"2024-09-17T04:06:03.813736+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:59:39.256382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806822+0000","last_change":"2024-09-17T04:06:02.806497+0000","last_active":"2024-09-17T04:06:03.806822+0000","last_peered":"2024-09-17T04:06:03.806822+0000","last_clean":"2024-09-17T04:06:03.806822+0000","last_became_active":"2024-09-17T04:06:02.804443+0000","last_became_peered":"2024-09-17T04:06:02.804443+0000","last_unstale":"2024-09-17T04:06:03.806822+0000","last_undegraded":"2024-09-17T04:06:03.806822+0000","last_fullsized":"2024-09-17T04:06:03.806822+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:30.259717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820289+0000","last_change":"2024-09-17T04:06:03.817813+0000","last_active":"2024-09-17T04:06:03.820289+0000","last_peered":"2024-09-17T04:06:03.820289+0000","last_clean":"2024-09-17T04:06:03.820289+0000","last_became_active":"2024-09-17T04:06:03.817578+0000","last_became_peered":"2024-09-17T04:06:03.817578+0000","last_unstale":"2024-09-17T04:06:03.820289+0000","last_undegraded":"2024-09-17T04:06:03.820289+0000","last_fullsized":"2024-09-17T04:06:03.820289+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:24.187880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.801342+0000","last_change":"2024-09-17T04:06:02.798596+0000","last_active":"2024-09-17T04:06:02.801342+0000","last_peered":"2024-09-17T04:06:02.801342+0000","last_clean":"2024-09-17T04:06:02.801342+0000","last_became_active":"2024-09-17T04:06:02.798278+0000","last_became_peered":"2024-09-17T04:06:02.798278+0000","last_unstale":"2024-09-17T04:06:02.801342+0000","last_undegraded":"2024-09-17T04:06:02.801342+0000","last_fullsized":"2024-09-17T04:06:02.801342+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:28.052006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.803472+0000","last_change":"2024-09-17T04:06:03.796410+0000","last_active":"2024-09-17T04:06:03.803472+0000","last_peered":"2024-09-17T04:06:03.803472+0000","last_clean":"2024-09-17T04:06:03.803472+0000","last_became_active":"2024-09-17T04:06:03.792472+0000","last_became_peered":"2024-09-17T04:06:03.792472+0000","last_unstale":"2024-09-17T04:06:03.803472+0000","last_undegraded":"2024-09-17T04:06:03.803472+0000","last_fullsized":"2024-09-17T04:06:03.803472+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:58.183431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.782129+0000","last_change":"2024-09-17T04:06:02.800955+0000","last_active":"2024-09-17T04:06:03.782129+0000","last_peered":"2024-09-17T04:06:03.782129+0000","last_clean":"2024-09-17T04:06:03.782129+0000","last_became_active":"2024-09-17T04:06:02.800645+0000","last_became_peered":"2024-09-17T04:06:02.800645+0000","last_unstale":"2024-09-17T04:06:03.782129+0000","last_undegraded":"2024-09-17T04:06:03.782129+0000","last_fullsized":"2024-09-17T04:06:03.782129+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:58.167449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811665+0000","last_change":"2024-09-17T04:06:03.804237+0000","last_active":"2024-09-17T04:06:03.811665+0000","last_peered":"2024-09-17T04:06:03.811665+0000","last_clean":"2024-09-17T04:06:03.811665+0000","last_became_active":"2024-09-17T04:06:03.803910+0000","last_became_peered":"2024-09-17T04:06:03.803910+0000","last_unstale":"2024-09-17T04:06:03.811665+0000","last_undegraded":"2024-09-17T04:06:03.811665+0000","last_fullsized":"2024-09-17T04:06:03.811665+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:18.175150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802839+0000","last_change":"2024-09-17T04:06:02.803084+0000","last_active":"2024-09-17T04:06:03.802839+0000","last_peered":"2024-09-17T04:06:03.802839+0000","last_clean":"2024-09-17T04:06:03.802839+0000","last_became_active":"2024-09-17T04:06:02.802699+0000","last_became_peered":"2024-09-17T04:06:02.802699+0000","last_unstale":"2024-09-17T04:06:03.802839+0000","last_undegraded":"2024-09-17T04:06:03.802839+0000","last_fullsized":"2024-09-17T04:06:03.802839+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.805379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.785032+0000","last_change":"2024-09-17T04:06:02.785032+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.785032+0000","last_undegraded":"2024-09-17T04:06:02.785032+0000","last_fullsized":"2024-09-17T04:06:02.785032+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.805959+0000","last_change":"2024-09-17T04:06:02.803099+0000","last_active":"2024-09-17T04:06:02.805959+0000","last_peered":"2024-09-17T04:06:02.805959+0000","last_clean":"2024-09-17T04:06:02.805959+0000","last_became_active":"2024-09-17T04:06:02.802781+0000","last_became_peered":"2024-09-17T04:06:02.802781+0000","last_unstale":"2024-09-17T04:06:02.805959+0000","last_undegraded":"2024-09-17T04:06:02.805959+0000","last_fullsized":"2024-09-17T04:06:02.805959+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:35.466816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781986+0000","last_change":"2024-09-17T04:06:02.781986+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781986+0000","last_undegraded":"2024-09-17T04:06:02.781986+0000","last_fullsized":"2024-09-17T04:06:02.781986+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820510+0000","last_change":"2024-09-17T04:06:02.809075+0000","last_active":"2024-09-17T04:06:03.820510+0000","last_peered":"2024-09-17T04:06:03.820510+0000","last_clean":"2024-09-17T04:06:03.820510+0000","last_became_active":"2024-09-17T04:06:02.808749+0000","last_became_peered":"2024-09-17T04:06:02.808749+0000","last_unstale":"2024-09-17T04:06:03.820510+0000","last_undegraded":"2024-09-17T04:06:03.820510+0000","last_fullsized":"2024-09-17T04:06:03.820510+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:14.014465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813152+0000","last_change":"2024-09-17T04:06:03.806209+0000","last_active":"2024-09-17T04:06:03.813152+0000","last_peered":"2024-09-17T04:06:03.813152+0000","last_clean":"2024-09-17T04:06:03.813152+0000","last_became_active":"2024-09-17T04:06:03.803114+0000","last_became_peered":"2024-09-17T04:06:03.803114+0000","last_unstale":"2024-09-17T04:06:03.813152+0000","last_undegraded":"2024-09-17T04:06:03.813152+0000","last_fullsized":"2024-09-17T04:06:03.813152+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:02.206329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.790903+0000","last_change":"2024-09-17T04:06:02.800746+0000","last_active":"2024-09-17T04:06:03.790903+0000","last_peered":"2024-09-17T04:06:03.790903+0000","last_clean":"2024-09-17T04:06:03.790903+0000","last_became_active":"2024-09-17T04:06:02.800425+0000","last_became_peered":"2024-09-17T04:06:02.800425+0000","last_unstale":"2024-09-17T04:06:03.790903+0000","last_undegraded":"2024-09-17T04:06:03.790903+0000","last_fullsized":"2024-09-17T04:06:03.790903+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:58.913127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":1,"reported_epoch":231,"state":"creating+peering","last_fresh":"2024-09-17T04:06:02.781347+0000","last_change":"2024-09-17T04:06:02.781347+0000","last_active":"2024-09-17T04:06:02.766508+0000","last_peered":"2024-09-17T04:06:02.766508+0000","last_clean":"2024-09-17T04:06:02.766508+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:02.781347+0000","last_undegraded":"2024-09-17T04:06:02.781347+0000","last_fullsized":"2024-09-17T04:06:02.781347+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801098+0000","last_change":"2024-09-17T04:06:02.806083+0000","last_active":"2024-09-17T04:06:03.801098+0000","last_peered":"2024-09-17T04:06:03.801098+0000","last_clean":"2024-09-17T04:06:03.801098+0000","last_became_active":"2024-09-17T04:06:02.804581+0000","last_became_peered":"2024-09-17T04:06:02.804581+0000","last_unstale":"2024-09-17T04:06:03.801098+0000","last_undegraded":"2024-09-17T04:06:03.801098+0000","last_fullsized":"2024-09-17T04:06:03.801098+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.805379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812338+0000","last_change":"2024-09-17T04:06:03.805922+0000","last_active":"2024-09-17T04:06:03.812338+0000","last_peered":"2024-09-17T04:06:03.812338+0000","last_clean":"2024-09-17T04:06:03.812338+0000","last_became_active":"2024-09-17T04:06:03.805635+0000","last_became_peered":"2024-09-17T04:06:03.805635+0000","last_unstale":"2024-09-17T04:06:03.812338+0000","last_undegraded":"2024-09-17T04:06:03.812338+0000","last_fullsized":"2024-09-17T04:06:03.812338+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:05.884244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.804946+0000","last_change":"2024-09-17T04:06:02.806327+0000","last_active":"2024-09-17T04:06:03.804946+0000","last_peered":"2024-09-17T04:06:03.804946+0000","last_clean":"2024-09-17T04:06:03.804946+0000","last_became_active":"2024-09-17T04:06:02.805995+0000","last_became_peered":"2024-09-17T04:06:02.805995+0000","last_unstale":"2024-09-17T04:06:03.804946+0000","last_undegraded":"2024-09-17T04:06:03.804946+0000","last_fullsized":"2024-09-17T04:06:03.804946+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:37.924620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820866+0000","last_change":"2024-09-17T04:06:03.818557+0000","last_active":"2024-09-17T04:06:03.820866+0000","last_peered":"2024-09-17T04:06:03.820866+0000","last_clean":"2024-09-17T04:06:03.820866+0000","last_became_active":"2024-09-17T04:06:03.818308+0000","last_became_peered":"2024-09-17T04:06:03.818308+0000","last_unstale":"2024-09-17T04:06:03.820866+0000","last_undegraded":"2024-09-17T04:06:03.820866+0000","last_fullsized":"2024-09-17T04:06:03.820866+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.224591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.800848+0000","last_change":"2024-09-17T04:06:02.797925+0000","last_active":"2024-09-17T04:06:02.800848+0000","last_peered":"2024-09-17T04:06:02.800848+0000","last_clean":"2024-09-17T04:06:02.800848+0000","last_became_active":"2024-09-17T04:06:02.797589+0000","last_became_peered":"2024-09-17T04:06:02.797589+0000","last_unstale":"2024-09-17T04:06:02.800848+0000","last_undegraded":"2024-09-17T04:06:02.800848+0000","last_fullsized":"2024-09-17T04:06:02.800848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:51.176128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.809534+0000","last_change":"2024-09-17T04:06:03.801247+0000","last_active":"2024-09-17T04:06:03.809534+0000","last_peered":"2024-09-17T04:06:03.809534+0000","last_clean":"2024-09-17T04:06:03.809534+0000","last_became_active":"2024-09-17T04:06:03.800917+0000","last_became_peered":"2024-09-17T04:06:03.800917+0000","last_unstale":"2024-09-17T04:06:03.809534+0000","last_undegraded":"2024-09-17T04:06:03.809534+0000","last_fullsized":"2024-09-17T04:06:03.809534+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:31.254968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.799467+0000","last_change":"2024-09-17T04:06:02.794443+0000","last_active":"2024-09-17T04:06:02.799467+0000","last_peered":"2024-09-17T04:06:02.799467+0000","last_clean":"2024-09-17T04:06:02.799467+0000","last_became_active":"2024-09-17T04:06:02.794093+0000","last_became_peered":"2024-09-17T04:06:02.794093+0000","last_unstale":"2024-09-17T04:06:02.799467+0000","last_undegraded":"2024-09-17T04:06:02.799467+0000","last_fullsized":"2024-09-17T04:06:02.799467+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:01:29.521189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811634+0000","last_change":"2024-09-17T04:06:03.804057+0000","last_active":"2024-09-17T04:06:03.811634+0000","last_peered":"2024-09-17T04:06:03.811634+0000","last_clean":"2024-09-17T04:06:03.811634+0000","last_became_active":"2024-09-17T04:06:03.803711+0000","last_became_peered":"2024-09-17T04:06:03.803711+0000","last_unstale":"2024-09-17T04:06:03.811634+0000","last_undegraded":"2024-09-17T04:06:03.811634+0000","last_fullsized":"2024-09-17T04:06:03.811634+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:42.604735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":10,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-17T04:06:02.801231+0000","last_change":"2024-09-17T04:06:02.798831+0000","last_active":"2024-09-17T04:06:02.801231+0000","last_peered":"2024-09-17T04:06:02.801231+0000","last_clean":"2024-09-17T04:06:02.801231+0000","last_became_active":"2024-09-17T04:06:02.798500+0000","last_became_peered":"2024-09-17T04:06:02.798500+0000","last_unstale":"2024-09-17T04:06:02.801231+0000","last_undegraded":"2024-09-17T04:06:02.801231+0000","last_fullsized":"2024-09-17T04:06:02.801231+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:32:22.846474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818562+0000","last_change":"2024-09-17T04:06:03.811222+0000","last_active":"2024-09-17T04:06:03.818562+0000","last_peered":"2024-09-17T04:06:03.818562+0000","last_clean":"2024-09-17T04:06:03.818562+0000","last_became_active":"2024-09-17T04:06:03.810800+0000","last_became_peered":"2024-09-17T04:06:03.810800+0000","last_unstale":"2024-09-17T04:06:03.818562+0000","last_undegraded":"2024-09-17T04:06:03.818562+0000","last_fullsized":"2024-09-17T04:06:03.818562+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:21.623918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.780447+0000","last_change":"2024-09-17T04:06:02.786094+0000","last_active":"2024-09-17T04:06:03.780447+0000","last_peered":"2024-09-17T04:06:03.780447+0000","last_clean":"2024-09-17T04:06:03.780447+0000","last_became_active":"2024-09-17T04:06:02.785792+0000","last_became_peered":"2024-09-17T04:06:02.785792+0000","last_unstale":"2024-09-17T04:06:03.780447+0000","last_undegraded":"2024-09-17T04:06:03.780447+0000","last_fullsized":"2024-09-17T04:06:03.780447+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:36.650214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817062+0000","last_change":"2024-09-17T04:06:03.811229+0000","last_active":"2024-09-17T04:06:03.817062+0000","last_peered":"2024-09-17T04:06:03.817062+0000","last_clean":"2024-09-17T04:06:03.817062+0000","last_became_active":"2024-09-17T04:06:03.810843+0000","last_became_peered":"2024-09-17T04:06:03.810843+0000","last_unstale":"2024-09-17T04:06:03.817062+0000","last_undegraded":"2024-09-17T04:06:03.817062+0000","last_fullsized":"2024-09-17T04:06:03.817062+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:27.620977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":499,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796429+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:03.796429+0000","last_peered":"2024-09-17T04:06:03.796429+0000","last_clean":"2024-09-17T04:06:03.796429+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:03.796429+0000","last_undegraded":"2024-09-17T04:06:03.796429+0000","last_fullsized":"2024-09-17T04:06:03.796429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":10,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52684,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3526656,"data_stored":1888339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509645,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29880,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341960,"statfs":{"total":96636764160,"available":96606167040,"internally_reserved":0,"allocated":2920448,"data_stored":1291196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52112,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319728,"statfs":{"total":96636764160,"available":96583401472,"internally_reserved":0,"allocated":2940928,"data_stored":1298063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56276,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315564,"statfs":{"total":96636764160,"available":96579137536,"internally_reserved":0,"allocated":2945024,"data_stored":1299583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56268,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315572,"statfs":{"total":96636764160,"available":96579145728,"internally_reserved":0,"allocated":2936832,"data_stored":1298061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52088,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319752,"statfs":{"total":96636764160,"available":96583426048,"internally_reserved":0,"allocated":2920448,"data_stored":1291196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542346,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52088,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319752,"statfs":{"total":96636764160,"available":96583426048,"internally_reserved":0,"allocated":2920448,"data_stored":1291196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542347,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52672,"kb_used_data":3432,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319168,"statfs":{"total":96636764160,"available":96582828032,"internally_reserved":0,"allocated":3514368,"data_stored":1881564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:08.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 -- 172.21.15.129:0/1220101524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 msgr2=0x7f908c003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 --2- 172.21.15.129:0/1220101524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f908c003ec0 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f909800c310 tx=0x7f908c005830 comp rx=0 tx=0).stop 2024-09-17T04:06:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 -- 172.21.15.129:0/1220101524 shutdown_connections 2024-09-17T04:06:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 --2- 172.21.15.129:0/1220101524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f908c003ec0 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 -- 172.21.15.129:0/1220101524 >> 172.21.15.129:0/1220101524 conn(0x7f90a806d410 msgr2=0x7f90a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 -- 172.21.15.129:0/1220101524 shutdown_connections 2024-09-17T04:06:08.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.663+0000 7f90aec00640 1 -- 172.21.15.129:0/1220101524 wait complete. 2024-09-17T04:06:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.665+0000 7f90aec00640 1 Processor -- start 2024-09-17T04:06:08.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.665+0000 7f90aec00640 1 -- start start 2024-09-17T04:06:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90a81f9f60 0x7f90a815a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90a80cc320 0x7f90a815db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90a8160260 con 0x7f908c003a50 2024-09-17T04:06:08.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f90a81600e0 con 0x7f90a80cc320 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90aec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f90a806eed0 con 0x7f90a81f9f60 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90a7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51966/0 (socket says 172.21.15.129:51966) 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90a7e00640 1 -- 172.21.15.129:0/1309916329 learned_addr learned my addr 172.21.15.129:0/1309916329 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f909f400640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90a81f9f60 0x7f90a815a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.667+0000 7f90acc00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90a80cc320 0x7f90a815db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 -- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90a81f9f60 msgr2=0x7f90a815a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90a81f9f60 0x7f90a815a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 -- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90a80cc320 msgr2=0x7f90a815db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90a80cc320 0x7f90a815db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 -- 172.21.15.129:0/1309916329 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a815acd0 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a7e00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f909800c430 tx=0x7f9098004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9098044070 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9098006420 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a81f9480 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9098031040 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.668+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a815afb0 con 0x7f908c003a50 2024-09-17T04:06:08.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.669+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a806f3e0 con 0x7f908c003a50 2024-09-17T04:06:08.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.671+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9098034070 con 0x7f908c003a50 2024-09-17T04:06:08.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.672+0000 7f90a4c00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 0x7f908407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:08.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.672+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f90980ba910 con 0x7f908c003a50 2024-09-17T04:06:08.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.673+0000 7f909f400640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 0x7f908407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:08.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.673+0000 7f909f400640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 0x7f908407b0a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f90a815bdb0 tx=0x7f9090006ba0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:08.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.677+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9098082d40 con 0x7f908c003a50 2024-09-17T04:06:08.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.857+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90a806a140 con 0x7f9084078b90 2024-09-17T04:06:08.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.862+0000 7f90a4c00640 1 -- 172.21.15.129:0/1309916329 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252477 (secure 0 0 0) 0x7f90a806a140 con 0x7f9084078b90 2024-09-17T04:06:08.865 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:08.866 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 msgr2=0x7f908407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 0x7f908407b0a0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f90a815bdb0 tx=0x7f9090006ba0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 msgr2=0x7f90a8099b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f909800c430 tx=0x7f9098004be0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 shutdown_connections 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9084078b90 0x7f908407b0a0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90a80cc320 0x7f90a815db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90a81f9f60 0x7f90a815a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 --2- 172.21.15.129:0/1309916329 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f908c003a50 0x7f90a8099b70 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 >> 172.21.15.129:0/1309916329 conn(0x7f90a806d410 msgr2=0x7f90a80b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 shutdown_connections 2024-09-17T04:06:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:08.867+0000 7f90aec00640 1 -- 172.21.15.129:0/1309916329 wait complete. 2024-09-17T04:06:08.890 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-17T04:06:08.580119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":404136,"kb_used_data":24112,"kb_used_omap":12,"kb_used_meta":379827,"kb_avail":754570584,"statfs":{"total":773094113280,"available":772680278016,"internally_reserved":0,"allocated":24690688,"data_stored":11566076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":388943408},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001484"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.005189+0000","last_change":"2024-09-17T04:06:03.802800+0000","last_active":"2024-09-17T04:06:06.005189+0000","last_peered":"2024-09-17T04:06:06.005189+0000","last_clean":"2024-09-17T04:06:06.005189+0000","last_became_active":"2024-09-17T04:06:03.802474+0000","last_became_peered":"2024-09-17T04:06:03.802474+0000","last_unstale":"2024-09-17T04:06:06.005189+0000","last_undegraded":"2024-09-17T04:06:06.005189+0000","last_fullsized":"2024-09-17T04:06:06.005189+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:09.252767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813384+0000","last_change":"2024-09-17T04:06:03.806736+0000","last_active":"2024-09-17T04:06:03.813384+0000","last_peered":"2024-09-17T04:06:03.813384+0000","last_clean":"2024-09-17T04:06:03.813384+0000","last_became_active":"2024-09-17T04:06:03.806423+0000","last_became_peered":"2024-09-17T04:06:03.806423+0000","last_unstale":"2024-09-17T04:06:03.813384+0000","last_undegraded":"2024-09-17T04:06:03.813384+0000","last_fullsized":"2024-09-17T04:06:03.813384+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:39:47.077633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801873+0000","last_change":"2024-09-17T04:06:03.796399+0000","last_active":"2024-09-17T04:06:03.801873+0000","last_peered":"2024-09-17T04:06:03.801873+0000","last_clean":"2024-09-17T04:06:03.801873+0000","last_became_active":"2024-09-17T04:06:03.796105+0000","last_became_peered":"2024-09-17T04:06:03.796105+0000","last_unstale":"2024-09-17T04:06:03.801873+0000","last_undegraded":"2024-09-17T04:06:03.801873+0000","last_fullsized":"2024-09-17T04:06:03.801873+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:33:12.238192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.819203+0000","last_change":"2024-09-17T04:06:03.817407+0000","last_active":"2024-09-17T04:06:03.819203+0000","last_peered":"2024-09-17T04:06:03.819203+0000","last_clean":"2024-09-17T04:06:03.819203+0000","last_became_active":"2024-09-17T04:06:03.817010+0000","last_became_peered":"2024-09-17T04:06:03.817010+0000","last_unstale":"2024-09-17T04:06:03.819203+0000","last_undegraded":"2024-09-17T04:06:03.819203+0000","last_fullsized":"2024-09-17T04:06:03.819203+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:47.218078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813775+0000","last_change":"2024-09-17T04:06:03.808299+0000","last_active":"2024-09-17T04:06:03.813775+0000","last_peered":"2024-09-17T04:06:03.813775+0000","last_clean":"2024-09-17T04:06:03.813775+0000","last_became_active":"2024-09-17T04:06:03.806953+0000","last_became_peered":"2024-09-17T04:06:03.806953+0000","last_unstale":"2024-09-17T04:06:03.813775+0000","last_undegraded":"2024-09-17T04:06:03.813775+0000","last_fullsized":"2024-09-17T04:06:03.813775+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:25:25.478342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813806+0000","last_change":"2024-09-17T04:06:03.807167+0000","last_active":"2024-09-17T04:06:03.813806+0000","last_peered":"2024-09-17T04:06:03.813806+0000","last_clean":"2024-09-17T04:06:03.813806+0000","last_became_active":"2024-09-17T04:06:03.806876+0000","last_became_peered":"2024-09-17T04:06:03.806876+0000","last_unstale":"2024-09-17T04:06:03.813806+0000","last_undegraded":"2024-09-17T04:06:03.813806+0000","last_fullsized":"2024-09-17T04:06:03.813806+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:52.692494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707529+0000","last_change":"2024-09-17T04:06:03.821976+0000","last_active":"2024-09-17T04:06:05.707529+0000","last_peered":"2024-09-17T04:06:05.707529+0000","last_clean":"2024-09-17T04:06:05.707529+0000","last_became_active":"2024-09-17T04:06:03.821519+0000","last_became_peered":"2024-09-17T04:06:03.821519+0000","last_unstale":"2024-09-17T04:06:05.707529+0000","last_undegraded":"2024-09-17T04:06:05.707529+0000","last_fullsized":"2024-09-17T04:06:05.707529+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:29:12.788250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707517+0000","last_change":"2024-09-17T04:06:03.802365+0000","last_active":"2024-09-17T04:06:05.707517+0000","last_peered":"2024-09-17T04:06:05.707517+0000","last_clean":"2024-09-17T04:06:05.707517+0000","last_became_active":"2024-09-17T04:06:03.799364+0000","last_became_peered":"2024-09-17T04:06:03.799364+0000","last_unstale":"2024-09-17T04:06:05.707517+0000","last_undegraded":"2024-09-17T04:06:05.707517+0000","last_fullsized":"2024-09-17T04:06:05.707517+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:32:19.256076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817820+0000","last_change":"2024-09-17T04:06:03.816441+0000","last_active":"2024-09-17T04:06:03.817820+0000","last_peered":"2024-09-17T04:06:03.817820+0000","last_clean":"2024-09-17T04:06:03.817820+0000","last_became_active":"2024-09-17T04:06:03.816103+0000","last_became_peered":"2024-09-17T04:06:03.816103+0000","last_unstale":"2024-09-17T04:06:03.817820+0000","last_undegraded":"2024-09-17T04:06:03.817820+0000","last_fullsized":"2024-09-17T04:06:03.817820+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:25:00.425180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817345+0000","last_change":"2024-09-17T04:06:03.810393+0000","last_active":"2024-09-17T04:06:03.817345+0000","last_peered":"2024-09-17T04:06:03.817345+0000","last_clean":"2024-09-17T04:06:03.817345+0000","last_became_active":"2024-09-17T04:06:03.805893+0000","last_became_peered":"2024-09-17T04:06:03.805893+0000","last_unstale":"2024-09-17T04:06:03.817345+0000","last_undegraded":"2024-09-17T04:06:03.817345+0000","last_fullsized":"2024-09-17T04:06:03.817345+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:18:45.923951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817661+0000","last_change":"2024-09-17T04:06:03.815419+0000","last_active":"2024-09-17T04:06:03.817661+0000","last_peered":"2024-09-17T04:06:03.817661+0000","last_clean":"2024-09-17T04:06:03.817661+0000","last_became_active":"2024-09-17T04:06:03.815077+0000","last_became_peered":"2024-09-17T04:06:03.815077+0000","last_unstale":"2024-09-17T04:06:03.817661+0000","last_undegraded":"2024-09-17T04:06:03.817661+0000","last_fullsized":"2024-09-17T04:06:03.817661+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:15:44.349015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004908+0000","last_change":"2024-09-17T04:06:03.799938+0000","last_active":"2024-09-17T04:06:06.004908+0000","last_peered":"2024-09-17T04:06:06.004908+0000","last_clean":"2024-09-17T04:06:06.004908+0000","last_became_active":"2024-09-17T04:06:03.798906+0000","last_became_peered":"2024-09-17T04:06:03.798906+0000","last_unstale":"2024-09-17T04:06:06.004908+0000","last_undegraded":"2024-09-17T04:06:06.004908+0000","last_fullsized":"2024-09-17T04:06:06.004908+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:21.719996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004739+0000","last_change":"2024-09-17T04:06:03.806119+0000","last_active":"2024-09-17T04:06:06.004739+0000","last_peered":"2024-09-17T04:06:06.004739+0000","last_clean":"2024-09-17T04:06:06.004739+0000","last_became_active":"2024-09-17T04:06:03.805797+0000","last_became_peered":"2024-09-17T04:06:03.805797+0000","last_unstale":"2024-09-17T04:06:06.004739+0000","last_undegraded":"2024-09-17T04:06:06.004739+0000","last_fullsized":"2024-09-17T04:06:06.004739+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:36:24.614392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813847+0000","last_change":"2024-09-17T04:06:03.809183+0000","last_active":"2024-09-17T04:06:03.813847+0000","last_peered":"2024-09-17T04:06:03.813847+0000","last_clean":"2024-09-17T04:06:03.813847+0000","last_became_active":"2024-09-17T04:06:03.807957+0000","last_became_peered":"2024-09-17T04:06:03.807957+0000","last_unstale":"2024-09-17T04:06:03.813847+0000","last_undegraded":"2024-09-17T04:06:03.813847+0000","last_fullsized":"2024-09-17T04:06:03.813847+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:01.300956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707571+0000","last_change":"2024-09-17T04:06:03.803225+0000","last_active":"2024-09-17T04:06:05.707571+0000","last_peered":"2024-09-17T04:06:05.707571+0000","last_clean":"2024-09-17T04:06:05.707571+0000","last_became_active":"2024-09-17T04:06:03.800410+0000","last_became_peered":"2024-09-17T04:06:03.800410+0000","last_unstale":"2024-09-17T04:06:05.707571+0000","last_undegraded":"2024-09-17T04:06:05.707571+0000","last_fullsized":"2024-09-17T04:06:05.707571+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:10:59.073717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707483+0000","last_change":"2024-09-17T04:06:03.796891+0000","last_active":"2024-09-17T04:06:05.707483+0000","last_peered":"2024-09-17T04:06:05.707483+0000","last_clean":"2024-09-17T04:06:05.707483+0000","last_became_active":"2024-09-17T04:06:03.796573+0000","last_became_peered":"2024-09-17T04:06:03.796573+0000","last_unstale":"2024-09-17T04:06:05.707483+0000","last_undegraded":"2024-09-17T04:06:05.707483+0000","last_fullsized":"2024-09-17T04:06:05.707483+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:28:35.720588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.805508+0000","last_change":"2024-09-17T04:06:03.797198+0000","last_active":"2024-09-17T04:06:03.805508+0000","last_peered":"2024-09-17T04:06:03.805508+0000","last_clean":"2024-09-17T04:06:03.805508+0000","last_became_active":"2024-09-17T04:06:03.796885+0000","last_became_peered":"2024-09-17T04:06:03.796885+0000","last_unstale":"2024-09-17T04:06:03.805508+0000","last_undegraded":"2024-09-17T04:06:03.805508+0000","last_fullsized":"2024-09-17T04:06:03.805508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:06.138948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811501+0000","last_change":"2024-09-17T04:06:03.801881+0000","last_active":"2024-09-17T04:06:03.811501+0000","last_peered":"2024-09-17T04:06:03.811501+0000","last_clean":"2024-09-17T04:06:03.811501+0000","last_became_active":"2024-09-17T04:06:03.801360+0000","last_became_peered":"2024-09-17T04:06:03.801360+0000","last_unstale":"2024-09-17T04:06:03.811501+0000","last_undegraded":"2024-09-17T04:06:03.811501+0000","last_fullsized":"2024-09-17T04:06:03.811501+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:23.901868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707874+0000","last_change":"2024-09-17T04:06:03.821263+0000","last_active":"2024-09-17T04:06:05.707874+0000","last_peered":"2024-09-17T04:06:05.707874+0000","last_clean":"2024-09-17T04:06:05.707874+0000","last_became_active":"2024-09-17T04:06:03.820834+0000","last_became_peered":"2024-09-17T04:06:03.820834+0000","last_unstale":"2024-09-17T04:06:05.707874+0000","last_undegraded":"2024-09-17T04:06:05.707874+0000","last_fullsized":"2024-09-17T04:06:05.707874+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:50.817447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.708060+0000","last_change":"2024-09-17T04:06:03.807819+0000","last_active":"2024-09-17T04:06:05.708060+0000","last_peered":"2024-09-17T04:06:05.708060+0000","last_clean":"2024-09-17T04:06:05.708060+0000","last_became_active":"2024-09-17T04:06:03.806620+0000","last_became_peered":"2024-09-17T04:06:03.806620+0000","last_unstale":"2024-09-17T04:06:05.708060+0000","last_undegraded":"2024-09-17T04:06:05.708060+0000","last_fullsized":"2024-09-17T04:06:05.708060+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:05.533535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004753+0000","last_change":"2024-09-17T04:06:03.804117+0000","last_active":"2024-09-17T04:06:06.004753+0000","last_peered":"2024-09-17T04:06:06.004753+0000","last_clean":"2024-09-17T04:06:06.004753+0000","last_became_active":"2024-09-17T04:06:03.801933+0000","last_became_peered":"2024-09-17T04:06:03.801933+0000","last_unstale":"2024-09-17T04:06:06.004753+0000","last_undegraded":"2024-09-17T04:06:06.004753+0000","last_fullsized":"2024-09-17T04:06:06.004753+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:28:50.127585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.799908+0000","last_change":"2024-09-17T04:06:03.795897+0000","last_active":"2024-09-17T04:06:03.799908+0000","last_peered":"2024-09-17T04:06:03.799908+0000","last_clean":"2024-09-17T04:06:03.799908+0000","last_became_active":"2024-09-17T04:06:03.795418+0000","last_became_peered":"2024-09-17T04:06:03.795418+0000","last_unstale":"2024-09-17T04:06:03.799908+0000","last_undegraded":"2024-09-17T04:06:03.799908+0000","last_fullsized":"2024-09-17T04:06:03.799908+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:39:17.494967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.810107+0000","last_change":"2024-09-17T04:06:03.805850+0000","last_active":"2024-09-17T04:06:03.810107+0000","last_peered":"2024-09-17T04:06:03.810107+0000","last_clean":"2024-09-17T04:06:03.810107+0000","last_became_active":"2024-09-17T04:06:03.804799+0000","last_became_peered":"2024-09-17T04:06:03.804799+0000","last_unstale":"2024-09-17T04:06:03.810107+0000","last_undegraded":"2024-09-17T04:06:03.810107+0000","last_fullsized":"2024-09-17T04:06:03.810107+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:55.693687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802306+0000","last_change":"2024-09-17T04:06:03.797753+0000","last_active":"2024-09-17T04:06:03.802306+0000","last_peered":"2024-09-17T04:06:03.802306+0000","last_clean":"2024-09-17T04:06:03.802306+0000","last_became_active":"2024-09-17T04:06:03.797408+0000","last_became_peered":"2024-09-17T04:06:03.797408+0000","last_unstale":"2024-09-17T04:06:03.802306+0000","last_undegraded":"2024-09-17T04:06:03.802306+0000","last_fullsized":"2024-09-17T04:06:03.802306+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:59.498210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.810797+0000","last_change":"2024-09-17T04:06:03.806787+0000","last_active":"2024-09-17T04:06:03.810797+0000","last_peered":"2024-09-17T04:06:03.810797+0000","last_clean":"2024-09-17T04:06:03.810797+0000","last_became_active":"2024-09-17T04:06:03.806508+0000","last_became_peered":"2024-09-17T04:06:03.806508+0000","last_unstale":"2024-09-17T04:06:03.810797+0000","last_undegraded":"2024-09-17T04:06:03.810797+0000","last_fullsized":"2024-09-17T04:06:03.810797+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:15:42.486394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.005007+0000","last_change":"2024-09-17T04:06:03.805042+0000","last_active":"2024-09-17T04:06:06.005007+0000","last_peered":"2024-09-17T04:06:06.005007+0000","last_clean":"2024-09-17T04:06:06.005007+0000","last_became_active":"2024-09-17T04:06:03.797026+0000","last_became_peered":"2024-09-17T04:06:03.797026+0000","last_unstale":"2024-09-17T04:06:06.005007+0000","last_undegraded":"2024-09-17T04:06:06.005007+0000","last_fullsized":"2024-09-17T04:06:06.005007+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.708158+0000","last_change":"2024-09-17T04:06:03.811729+0000","last_active":"2024-09-17T04:06:05.708158+0000","last_peered":"2024-09-17T04:06:05.708158+0000","last_clean":"2024-09-17T04:06:05.708158+0000","last_became_active":"2024-09-17T04:06:03.811243+0000","last_became_peered":"2024-09-17T04:06:03.811243+0000","last_unstale":"2024-09-17T04:06:05.708158+0000","last_undegraded":"2024-09-17T04:06:05.708158+0000","last_fullsized":"2024-09-17T04:06:05.708158+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:29:58.318718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812154+0000","last_change":"2024-09-17T04:06:03.799602+0000","last_active":"2024-09-17T04:06:03.812154+0000","last_peered":"2024-09-17T04:06:03.812154+0000","last_clean":"2024-09-17T04:06:03.812154+0000","last_became_active":"2024-09-17T04:06:03.799302+0000","last_became_peered":"2024-09-17T04:06:03.799302+0000","last_unstale":"2024-09-17T04:06:03.812154+0000","last_undegraded":"2024-09-17T04:06:03.812154+0000","last_fullsized":"2024-09-17T04:06:03.812154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:08.963673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707518+0000","last_change":"2024-09-17T04:06:03.821048+0000","last_active":"2024-09-17T04:06:05.707518+0000","last_peered":"2024-09-17T04:06:05.707518+0000","last_clean":"2024-09-17T04:06:05.707518+0000","last_became_active":"2024-09-17T04:06:03.820734+0000","last_became_peered":"2024-09-17T04:06:03.820734+0000","last_unstale":"2024-09-17T04:06:05.707518+0000","last_undegraded":"2024-09-17T04:06:05.707518+0000","last_fullsized":"2024-09-17T04:06:05.707518+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:37:20.157383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707964+0000","last_change":"2024-09-17T04:06:03.820288+0000","last_active":"2024-09-17T04:06:05.707964+0000","last_peered":"2024-09-17T04:06:05.707964+0000","last_clean":"2024-09-17T04:06:05.707964+0000","last_became_active":"2024-09-17T04:06:03.819807+0000","last_became_peered":"2024-09-17T04:06:03.819807+0000","last_unstale":"2024-09-17T04:06:05.707964+0000","last_undegraded":"2024-09-17T04:06:05.707964+0000","last_fullsized":"2024-09-17T04:06:05.707964+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:35:26.004038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806485+0000","last_change":"2024-09-17T04:06:03.804093+0000","last_active":"2024-09-17T04:06:03.806485+0000","last_peered":"2024-09-17T04:06:03.806485+0000","last_clean":"2024-09-17T04:06:03.806485+0000","last_became_active":"2024-09-17T04:06:03.803367+0000","last_became_peered":"2024-09-17T04:06:03.803367+0000","last_unstale":"2024-09-17T04:06:03.806485+0000","last_undegraded":"2024-09-17T04:06:03.806485+0000","last_fullsized":"2024-09-17T04:06:03.806485+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:21:53.134700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.005083+0000","last_change":"2024-09-17T04:06:03.807910+0000","last_active":"2024-09-17T04:06:06.005083+0000","last_peered":"2024-09-17T04:06:06.005083+0000","last_clean":"2024-09-17T04:06:06.005083+0000","last_became_active":"2024-09-17T04:06:03.807606+0000","last_became_peered":"2024-09-17T04:06:03.807606+0000","last_unstale":"2024-09-17T04:06:06.005083+0000","last_undegraded":"2024-09-17T04:06:06.005083+0000","last_fullsized":"2024-09-17T04:06:06.005083+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:29:10.352766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"232'4","reported_seq":19,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.005362+0000","last_change":"2024-09-17T04:06:02.803265+0000","last_active":"2024-09-17T04:06:06.005362+0000","last_peered":"2024-09-17T04:06:06.005362+0000","last_clean":"2024-09-17T04:06:06.005362+0000","last_became_active":"2024-09-17T04:06:02.801302+0000","last_became_peered":"2024-09-17T04:06:02.801302+0000","last_unstale":"2024-09-17T04:06:06.005362+0000","last_undegraded":"2024-09-17T04:06:06.005362+0000","last_fullsized":"2024-09-17T04:06:06.005362+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:19.220001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817624+0000","last_change":"2024-09-17T04:06:03.815673+0000","last_active":"2024-09-17T04:06:03.817624+0000","last_peered":"2024-09-17T04:06:03.817624+0000","last_clean":"2024-09-17T04:06:03.817624+0000","last_became_active":"2024-09-17T04:06:03.815312+0000","last_became_peered":"2024-09-17T04:06:03.815312+0000","last_unstale":"2024-09-17T04:06:03.817624+0000","last_undegraded":"2024-09-17T04:06:03.817624+0000","last_fullsized":"2024-09-17T04:06:03.817624+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:09.824297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.805381+0000","last_change":"2024-09-17T04:06:02.797312+0000","last_active":"2024-09-17T04:06:03.805381+0000","last_peered":"2024-09-17T04:06:03.805381+0000","last_clean":"2024-09-17T04:06:03.805381+0000","last_became_active":"2024-09-17T04:06:02.795549+0000","last_became_peered":"2024-09-17T04:06:02.795549+0000","last_unstale":"2024-09-17T04:06:03.805381+0000","last_undegraded":"2024-09-17T04:06:03.805381+0000","last_fullsized":"2024-09-17T04:06:03.805381+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:48:54.691901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817228+0000","last_change":"2024-09-17T04:06:03.812526+0000","last_active":"2024-09-17T04:06:03.817228+0000","last_peered":"2024-09-17T04:06:03.817228+0000","last_clean":"2024-09-17T04:06:03.817228+0000","last_became_active":"2024-09-17T04:06:03.812263+0000","last_became_peered":"2024-09-17T04:06:03.812263+0000","last_unstale":"2024-09-17T04:06:03.817228+0000","last_undegraded":"2024-09-17T04:06:03.817228+0000","last_fullsized":"2024-09-17T04:06:03.817228+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:10.219033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816029+0000","last_change":"2024-09-17T04:06:02.802766+0000","last_active":"2024-09-17T04:06:03.816029+0000","last_peered":"2024-09-17T04:06:03.816029+0000","last_clean":"2024-09-17T04:06:03.816029+0000","last_became_active":"2024-09-17T04:06:02.802486+0000","last_became_peered":"2024-09-17T04:06:02.802486+0000","last_unstale":"2024-09-17T04:06:03.816029+0000","last_undegraded":"2024-09-17T04:06:03.816029+0000","last_fullsized":"2024-09-17T04:06:03.816029+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:29:56.862206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.814528+0000","last_change":"2024-09-17T04:06:03.811870+0000","last_active":"2024-09-17T04:06:03.814528+0000","last_peered":"2024-09-17T04:06:03.814528+0000","last_clean":"2024-09-17T04:06:03.814528+0000","last_became_active":"2024-09-17T04:06:03.811552+0000","last_became_peered":"2024-09-17T04:06:03.811552+0000","last_unstale":"2024-09-17T04:06:03.814528+0000","last_undegraded":"2024-09-17T04:06:03.814528+0000","last_fullsized":"2024-09-17T04:06:03.814528+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:42:14.281666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"232'2","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.708226+0000","last_change":"2024-09-17T04:06:02.801393+0000","last_active":"2024-09-17T04:06:05.708226+0000","last_peered":"2024-09-17T04:06:05.708226+0000","last_clean":"2024-09-17T04:06:05.708226+0000","last_became_active":"2024-09-17T04:06:02.800404+0000","last_became_peered":"2024-09-17T04:06:02.800404+0000","last_unstale":"2024-09-17T04:06:05.708226+0000","last_undegraded":"2024-09-17T04:06:05.708226+0000","last_fullsized":"2024-09-17T04:06:05.708226+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:11:22.863334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808540+0000","last_change":"2024-09-17T04:06:03.802716+0000","last_active":"2024-09-17T04:06:03.808540+0000","last_peered":"2024-09-17T04:06:03.808540+0000","last_clean":"2024-09-17T04:06:03.808540+0000","last_became_active":"2024-09-17T04:06:03.802367+0000","last_became_peered":"2024-09-17T04:06:03.802367+0000","last_unstale":"2024-09-17T04:06:03.808540+0000","last_undegraded":"2024-09-17T04:06:03.808540+0000","last_fullsized":"2024-09-17T04:06:03.808540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:54.201988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816774+0000","last_change":"2024-09-17T04:06:02.792176+0000","last_active":"2024-09-17T04:06:03.816774+0000","last_peered":"2024-09-17T04:06:03.816774+0000","last_clean":"2024-09-17T04:06:03.816774+0000","last_became_active":"2024-09-17T04:06:02.791849+0000","last_became_peered":"2024-09-17T04:06:02.791849+0000","last_unstale":"2024-09-17T04:06:03.816774+0000","last_undegraded":"2024-09-17T04:06:03.816774+0000","last_fullsized":"2024-09-17T04:06:03.816774+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:20:37.125910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707827+0000","last_change":"2024-09-17T04:06:03.805617+0000","last_active":"2024-09-17T04:06:05.707827+0000","last_peered":"2024-09-17T04:06:05.707827+0000","last_clean":"2024-09-17T04:06:05.707827+0000","last_became_active":"2024-09-17T04:06:03.805315+0000","last_became_peered":"2024-09-17T04:06:03.805315+0000","last_unstale":"2024-09-17T04:06:05.707827+0000","last_undegraded":"2024-09-17T04:06:05.707827+0000","last_fullsized":"2024-09-17T04:06:05.707827+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:08.203534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817698+0000","last_change":"2024-09-17T04:06:02.796869+0000","last_active":"2024-09-17T04:06:03.817698+0000","last_peered":"2024-09-17T04:06:03.817698+0000","last_clean":"2024-09-17T04:06:03.817698+0000","last_became_active":"2024-09-17T04:06:02.796123+0000","last_became_peered":"2024-09-17T04:06:02.796123+0000","last_unstale":"2024-09-17T04:06:03.817698+0000","last_undegraded":"2024-09-17T04:06:03.817698+0000","last_fullsized":"2024-09-17T04:06:03.817698+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:43:41.930024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806454+0000","last_change":"2024-09-17T04:06:03.803582+0000","last_active":"2024-09-17T04:06:03.806454+0000","last_peered":"2024-09-17T04:06:03.806454+0000","last_clean":"2024-09-17T04:06:03.806454+0000","last_became_active":"2024-09-17T04:06:03.803283+0000","last_became_peered":"2024-09-17T04:06:03.803283+0000","last_unstale":"2024-09-17T04:06:03.806454+0000","last_undegraded":"2024-09-17T04:06:03.806454+0000","last_fullsized":"2024-09-17T04:06:03.806454+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:06:47.805456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004498+0000","last_change":"2024-09-17T04:06:02.802659+0000","last_active":"2024-09-17T04:06:06.004498+0000","last_peered":"2024-09-17T04:06:06.004498+0000","last_clean":"2024-09-17T04:06:06.004498+0000","last_became_active":"2024-09-17T04:06:02.802349+0000","last_became_peered":"2024-09-17T04:06:02.802349+0000","last_unstale":"2024-09-17T04:06:06.004498+0000","last_undegraded":"2024-09-17T04:06:06.004498+0000","last_fullsized":"2024-09-17T04:06:06.004498+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:12:29.516738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.803766+0000","last_change":"2024-09-17T04:06:03.793281+0000","last_active":"2024-09-17T04:06:03.803766+0000","last_peered":"2024-09-17T04:06:03.803766+0000","last_clean":"2024-09-17T04:06:03.803766+0000","last_became_active":"2024-09-17T04:06:03.791163+0000","last_became_peered":"2024-09-17T04:06:03.791163+0000","last_unstale":"2024-09-17T04:06:03.803766+0000","last_undegraded":"2024-09-17T04:06:03.803766+0000","last_fullsized":"2024-09-17T04:06:03.803766+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:48.471016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004407+0000","last_change":"2024-09-17T04:06:02.796371+0000","last_active":"2024-09-17T04:06:06.004407+0000","last_peered":"2024-09-17T04:06:06.004407+0000","last_clean":"2024-09-17T04:06:06.004407+0000","last_became_active":"2024-09-17T04:06:02.796081+0000","last_became_peered":"2024-09-17T04:06:02.796081+0000","last_unstale":"2024-09-17T04:06:06.004407+0000","last_undegraded":"2024-09-17T04:06:06.004407+0000","last_fullsized":"2024-09-17T04:06:06.004407+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:52:17.097464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801703+0000","last_change":"2024-09-17T04:06:03.795816+0000","last_active":"2024-09-17T04:06:03.801703+0000","last_peered":"2024-09-17T04:06:03.801703+0000","last_clean":"2024-09-17T04:06:03.801703+0000","last_became_active":"2024-09-17T04:06:03.795543+0000","last_became_peered":"2024-09-17T04:06:03.795543+0000","last_unstale":"2024-09-17T04:06:03.801703+0000","last_undegraded":"2024-09-17T04:06:03.801703+0000","last_fullsized":"2024-09-17T04:06:03.801703+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:15.130160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808033+0000","last_change":"2024-09-17T04:06:02.803758+0000","last_active":"2024-09-17T04:06:03.808033+0000","last_peered":"2024-09-17T04:06:03.808033+0000","last_clean":"2024-09-17T04:06:03.808033+0000","last_became_active":"2024-09-17T04:06:02.800956+0000","last_became_peered":"2024-09-17T04:06:02.800956+0000","last_unstale":"2024-09-17T04:06:03.808033+0000","last_undegraded":"2024-09-17T04:06:03.808033+0000","last_fullsized":"2024-09-17T04:06:03.808033+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:03.191732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812214+0000","last_change":"2024-09-17T04:06:03.805584+0000","last_active":"2024-09-17T04:06:03.812214+0000","last_peered":"2024-09-17T04:06:03.812214+0000","last_clean":"2024-09-17T04:06:03.812214+0000","last_became_active":"2024-09-17T04:06:03.805253+0000","last_became_peered":"2024-09-17T04:06:03.805253+0000","last_unstale":"2024-09-17T04:06:03.812214+0000","last_undegraded":"2024-09-17T04:06:03.812214+0000","last_fullsized":"2024-09-17T04:06:03.812214+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:51:58.700136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801101+0000","last_change":"2024-09-17T04:06:02.809561+0000","last_active":"2024-09-17T04:06:03.801101+0000","last_peered":"2024-09-17T04:06:03.801101+0000","last_clean":"2024-09-17T04:06:03.801101+0000","last_became_active":"2024-09-17T04:06:02.809205+0000","last_became_peered":"2024-09-17T04:06:02.809205+0000","last_unstale":"2024-09-17T04:06:03.801101+0000","last_undegraded":"2024-09-17T04:06:03.801101+0000","last_fullsized":"2024-09-17T04:06:03.801101+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:22:57.306478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.822844+0000","last_change":"2024-09-17T04:06:03.821651+0000","last_active":"2024-09-17T04:06:03.822844+0000","last_peered":"2024-09-17T04:06:03.822844+0000","last_clean":"2024-09-17T04:06:03.822844+0000","last_became_active":"2024-09-17T04:06:03.821041+0000","last_became_peered":"2024-09-17T04:06:03.821041+0000","last_unstale":"2024-09-17T04:06:03.822844+0000","last_undegraded":"2024-09-17T04:06:03.822844+0000","last_fullsized":"2024-09-17T04:06:03.822844+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:44.264621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"232'3","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815889+0000","last_change":"2024-09-17T04:06:02.796662+0000","last_active":"2024-09-17T04:06:03.815889+0000","last_peered":"2024-09-17T04:06:03.815889+0000","last_clean":"2024-09-17T04:06:03.815889+0000","last_became_active":"2024-09-17T04:06:02.796336+0000","last_became_peered":"2024-09-17T04:06:02.796336+0000","last_unstale":"2024-09-17T04:06:03.815889+0000","last_undegraded":"2024-09-17T04:06:03.815889+0000","last_fullsized":"2024-09-17T04:06:03.815889+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:12.806260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818666+0000","last_change":"2024-09-17T04:06:03.810719+0000","last_active":"2024-09-17T04:06:03.818666+0000","last_peered":"2024-09-17T04:06:03.818666+0000","last_clean":"2024-09-17T04:06:03.818666+0000","last_became_active":"2024-09-17T04:06:03.807985+0000","last_became_peered":"2024-09-17T04:06:03.807985+0000","last_unstale":"2024-09-17T04:06:03.818666+0000","last_undegraded":"2024-09-17T04:06:03.818666+0000","last_fullsized":"2024-09-17T04:06:03.818666+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:13.580410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"232'6","reported_seq":19,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.824178+0000","last_change":"2024-09-17T04:06:02.797156+0000","last_active":"2024-09-17T04:06:03.824178+0000","last_peered":"2024-09-17T04:06:03.824178+0000","last_clean":"2024-09-17T04:06:03.824178+0000","last_became_active":"2024-09-17T04:06:02.796817+0000","last_became_peered":"2024-09-17T04:06:02.796817+0000","last_unstale":"2024-09-17T04:06:03.824178+0000","last_undegraded":"2024-09-17T04:06:03.824178+0000","last_fullsized":"2024-09-17T04:06:03.824178+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:08:12.025611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707508+0000","last_change":"2024-09-17T04:06:03.807133+0000","last_active":"2024-09-17T04:06:05.707508+0000","last_peered":"2024-09-17T04:06:05.707508+0000","last_clean":"2024-09-17T04:06:05.707508+0000","last_became_active":"2024-09-17T04:06:03.806796+0000","last_became_peered":"2024-09-17T04:06:03.806796+0000","last_unstale":"2024-09-17T04:06:05.707508+0000","last_undegraded":"2024-09-17T04:06:05.707508+0000","last_fullsized":"2024-09-17T04:06:05.707508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:17:11.147406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815281+0000","last_change":"2024-09-17T04:06:02.793880+0000","last_active":"2024-09-17T04:06:03.815281+0000","last_peered":"2024-09-17T04:06:03.815281+0000","last_clean":"2024-09-17T04:06:03.815281+0000","last_became_active":"2024-09-17T04:06:02.792660+0000","last_became_peered":"2024-09-17T04:06:02.792660+0000","last_unstale":"2024-09-17T04:06:03.815281+0000","last_undegraded":"2024-09-17T04:06:03.815281+0000","last_fullsized":"2024-09-17T04:06:03.815281+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:03:44.744990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.816348+0000","last_change":"2024-09-17T04:06:03.813764+0000","last_active":"2024-09-17T04:06:03.816348+0000","last_peered":"2024-09-17T04:06:03.816348+0000","last_clean":"2024-09-17T04:06:03.816348+0000","last_became_active":"2024-09-17T04:06:03.812094+0000","last_became_peered":"2024-09-17T04:06:03.812094+0000","last_unstale":"2024-09-17T04:06:03.816348+0000","last_undegraded":"2024-09-17T04:06:03.816348+0000","last_fullsized":"2024-09-17T04:06:03.816348+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:03.749045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.787062+0000","last_change":"2024-09-17T04:06:02.800294+0000","last_active":"2024-09-17T04:06:03.787062+0000","last_peered":"2024-09-17T04:06:03.787062+0000","last_clean":"2024-09-17T04:06:03.787062+0000","last_became_active":"2024-09-17T04:06:02.799725+0000","last_became_peered":"2024-09-17T04:06:02.799725+0000","last_unstale":"2024-09-17T04:06:03.787062+0000","last_undegraded":"2024-09-17T04:06:03.787062+0000","last_fullsized":"2024-09-17T04:06:03.787062+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:54.285672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.808478+0000","last_change":"2024-09-17T04:06:03.807107+0000","last_active":"2024-09-17T04:06:03.808478+0000","last_peered":"2024-09-17T04:06:03.808478+0000","last_clean":"2024-09-17T04:06:03.808478+0000","last_became_active":"2024-09-17T04:06:03.806775+0000","last_became_peered":"2024-09-17T04:06:03.806775+0000","last_unstale":"2024-09-17T04:06:03.808478+0000","last_undegraded":"2024-09-17T04:06:03.808478+0000","last_fullsized":"2024-09-17T04:06:03.808478+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:30.210756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802616+0000","last_change":"2024-09-17T04:06:02.802486+0000","last_active":"2024-09-17T04:06:03.802616+0000","last_peered":"2024-09-17T04:06:03.802616+0000","last_clean":"2024-09-17T04:06:03.802616+0000","last_became_active":"2024-09-17T04:06:02.801193+0000","last_became_peered":"2024-09-17T04:06:02.801193+0000","last_unstale":"2024-09-17T04:06:03.802616+0000","last_undegraded":"2024-09-17T04:06:03.802616+0000","last_fullsized":"2024-09-17T04:06:03.802616+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:56.111417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707896+0000","last_change":"2024-09-17T04:06:03.801459+0000","last_active":"2024-09-17T04:06:05.707896+0000","last_peered":"2024-09-17T04:06:05.707896+0000","last_clean":"2024-09-17T04:06:05.707896+0000","last_became_active":"2024-09-17T04:06:03.796297+0000","last_became_peered":"2024-09-17T04:06:03.796297+0000","last_unstale":"2024-09-17T04:06:05.707896+0000","last_undegraded":"2024-09-17T04:06:05.707896+0000","last_fullsized":"2024-09-17T04:06:05.707896+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:08:37.002697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"232'8","reported_seq":21,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817461+0000","last_change":"2024-09-17T04:06:02.794914+0000","last_active":"2024-09-17T04:06:03.817461+0000","last_peered":"2024-09-17T04:06:03.817461+0000","last_clean":"2024-09-17T04:06:03.817461+0000","last_became_active":"2024-09-17T04:06:02.794626+0000","last_became_peered":"2024-09-17T04:06:02.794626+0000","last_unstale":"2024-09-17T04:06:03.817461+0000","last_undegraded":"2024-09-17T04:06:03.817461+0000","last_fullsized":"2024-09-17T04:06:03.817461+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:28.405822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796154+0000","last_change":"2024-09-17T04:06:03.795419+0000","last_active":"2024-09-17T04:06:03.796154+0000","last_peered":"2024-09-17T04:06:03.796154+0000","last_clean":"2024-09-17T04:06:03.796154+0000","last_became_active":"2024-09-17T04:06:03.795073+0000","last_became_peered":"2024-09-17T04:06:03.795073+0000","last_unstale":"2024-09-17T04:06:03.796154+0000","last_undegraded":"2024-09-17T04:06:03.796154+0000","last_fullsized":"2024-09-17T04:06:03.796154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:55.646893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.821357+0000","last_change":"2024-09-17T04:06:02.805998+0000","last_active":"2024-09-17T04:06:03.821357+0000","last_peered":"2024-09-17T04:06:03.821357+0000","last_clean":"2024-09-17T04:06:03.821357+0000","last_became_active":"2024-09-17T04:06:02.803503+0000","last_became_peered":"2024-09-17T04:06:02.803503+0000","last_unstale":"2024-09-17T04:06:03.821357+0000","last_undegraded":"2024-09-17T04:06:03.821357+0000","last_fullsized":"2024-09-17T04:06:03.821357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:19:18.522548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796057+0000","last_change":"2024-09-17T04:06:03.793336+0000","last_active":"2024-09-17T04:06:03.796057+0000","last_peered":"2024-09-17T04:06:03.796057+0000","last_clean":"2024-09-17T04:06:03.796057+0000","last_became_active":"2024-09-17T04:06:03.793040+0000","last_became_peered":"2024-09-17T04:06:03.793040+0000","last_unstale":"2024-09-17T04:06:03.796057+0000","last_undegraded":"2024-09-17T04:06:03.796057+0000","last_fullsized":"2024-09-17T04:06:03.796057+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:09.417510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.819357+0000","last_change":"2024-09-17T04:06:02.801756+0000","last_active":"2024-09-17T04:06:03.819357+0000","last_peered":"2024-09-17T04:06:03.819357+0000","last_clean":"2024-09-17T04:06:03.819357+0000","last_became_active":"2024-09-17T04:06:02.801436+0000","last_became_peered":"2024-09-17T04:06:02.801436+0000","last_unstale":"2024-09-17T04:06:03.819357+0000","last_undegraded":"2024-09-17T04:06:03.819357+0000","last_fullsized":"2024-09-17T04:06:03.819357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:37.941773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.815106+0000","last_change":"2024-09-17T04:06:03.810169+0000","last_active":"2024-09-17T04:06:03.815106+0000","last_peered":"2024-09-17T04:06:03.815106+0000","last_clean":"2024-09-17T04:06:03.815106+0000","last_became_active":"2024-09-17T04:06:03.809843+0000","last_became_peered":"2024-09-17T04:06:03.809843+0000","last_unstale":"2024-09-17T04:06:03.815106+0000","last_undegraded":"2024-09-17T04:06:03.815106+0000","last_fullsized":"2024-09-17T04:06:03.815106+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:17.652978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"232'1","reported_seq":14,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818962+0000","last_change":"2024-09-17T04:06:02.807475+0000","last_active":"2024-09-17T04:06:03.818962+0000","last_peered":"2024-09-17T04:06:03.818962+0000","last_clean":"2024-09-17T04:06:03.818962+0000","last_became_active":"2024-09-17T04:06:02.806218+0000","last_became_peered":"2024-09-17T04:06:02.806218+0000","last_unstale":"2024-09-17T04:06:03.818962+0000","last_undegraded":"2024-09-17T04:06:03.818962+0000","last_fullsized":"2024-09-17T04:06:03.818962+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:38:50.595970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813736+0000","last_change":"2024-09-17T04:06:03.806964+0000","last_active":"2024-09-17T04:06:03.813736+0000","last_peered":"2024-09-17T04:06:03.813736+0000","last_clean":"2024-09-17T04:06:03.813736+0000","last_became_active":"2024-09-17T04:06:03.806679+0000","last_became_peered":"2024-09-17T04:06:03.806679+0000","last_unstale":"2024-09-17T04:06:03.813736+0000","last_undegraded":"2024-09-17T04:06:03.813736+0000","last_fullsized":"2024-09-17T04:06:03.813736+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:59:39.256382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.806822+0000","last_change":"2024-09-17T04:06:02.806497+0000","last_active":"2024-09-17T04:06:03.806822+0000","last_peered":"2024-09-17T04:06:03.806822+0000","last_clean":"2024-09-17T04:06:03.806822+0000","last_became_active":"2024-09-17T04:06:02.804443+0000","last_became_peered":"2024-09-17T04:06:02.804443+0000","last_unstale":"2024-09-17T04:06:03.806822+0000","last_undegraded":"2024-09-17T04:06:03.806822+0000","last_fullsized":"2024-09-17T04:06:03.806822+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:01:30.259717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820289+0000","last_change":"2024-09-17T04:06:03.817813+0000","last_active":"2024-09-17T04:06:03.820289+0000","last_peered":"2024-09-17T04:06:03.820289+0000","last_clean":"2024-09-17T04:06:03.820289+0000","last_became_active":"2024-09-17T04:06:03.817578+0000","last_became_peered":"2024-09-17T04:06:03.817578+0000","last_unstale":"2024-09-17T04:06:03.820289+0000","last_undegraded":"2024-09-17T04:06:03.820289+0000","last_fullsized":"2024-09-17T04:06:03.820289+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:24.187880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"232'2","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004930+0000","last_change":"2024-09-17T04:06:02.798596+0000","last_active":"2024-09-17T04:06:06.004930+0000","last_peered":"2024-09-17T04:06:06.004930+0000","last_clean":"2024-09-17T04:06:06.004930+0000","last_became_active":"2024-09-17T04:06:02.798278+0000","last_became_peered":"2024-09-17T04:06:02.798278+0000","last_unstale":"2024-09-17T04:06:06.004930+0000","last_undegraded":"2024-09-17T04:06:06.004930+0000","last_fullsized":"2024-09-17T04:06:06.004930+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:27:23.078848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.803472+0000","last_change":"2024-09-17T04:06:03.796410+0000","last_active":"2024-09-17T04:06:03.803472+0000","last_peered":"2024-09-17T04:06:03.803472+0000","last_clean":"2024-09-17T04:06:03.803472+0000","last_became_active":"2024-09-17T04:06:03.792472+0000","last_became_peered":"2024-09-17T04:06:03.792472+0000","last_unstale":"2024-09-17T04:06:03.803472+0000","last_undegraded":"2024-09-17T04:06:03.803472+0000","last_fullsized":"2024-09-17T04:06:03.803472+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:47:58.183431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.782129+0000","last_change":"2024-09-17T04:06:02.800955+0000","last_active":"2024-09-17T04:06:03.782129+0000","last_peered":"2024-09-17T04:06:03.782129+0000","last_clean":"2024-09-17T04:06:03.782129+0000","last_became_active":"2024-09-17T04:06:02.800645+0000","last_became_peered":"2024-09-17T04:06:02.800645+0000","last_unstale":"2024-09-17T04:06:03.782129+0000","last_undegraded":"2024-09-17T04:06:03.782129+0000","last_fullsized":"2024-09-17T04:06:03.782129+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:55:58.167449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811665+0000","last_change":"2024-09-17T04:06:03.804237+0000","last_active":"2024-09-17T04:06:03.811665+0000","last_peered":"2024-09-17T04:06:03.811665+0000","last_clean":"2024-09-17T04:06:03.811665+0000","last_became_active":"2024-09-17T04:06:03.803910+0000","last_became_peered":"2024-09-17T04:06:03.803910+0000","last_unstale":"2024-09-17T04:06:03.811665+0000","last_undegraded":"2024-09-17T04:06:03.811665+0000","last_fullsized":"2024-09-17T04:06:03.811665+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:18.175150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.802839+0000","last_change":"2024-09-17T04:06:02.803084+0000","last_active":"2024-09-17T04:06:03.802839+0000","last_peered":"2024-09-17T04:06:03.802839+0000","last_clean":"2024-09-17T04:06:03.802839+0000","last_became_active":"2024-09-17T04:06:02.802699+0000","last_became_peered":"2024-09-17T04:06:02.802699+0000","last_unstale":"2024-09-17T04:06:03.802839+0000","last_undegraded":"2024-09-17T04:06:03.802839+0000","last_fullsized":"2024-09-17T04:06:03.802839+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.805379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004498+0000","last_change":"2024-09-17T04:06:03.816615+0000","last_active":"2024-09-17T04:06:06.004498+0000","last_peered":"2024-09-17T04:06:06.004498+0000","last_clean":"2024-09-17T04:06:06.004498+0000","last_became_active":"2024-09-17T04:06:03.816230+0000","last_became_peered":"2024-09-17T04:06:03.816230+0000","last_unstale":"2024-09-17T04:06:06.004498+0000","last_undegraded":"2024-09-17T04:06:06.004498+0000","last_fullsized":"2024-09-17T04:06:06.004498+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:36.646238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004421+0000","last_change":"2024-09-17T04:06:02.803099+0000","last_active":"2024-09-17T04:06:06.004421+0000","last_peered":"2024-09-17T04:06:06.004421+0000","last_clean":"2024-09-17T04:06:06.004421+0000","last_became_active":"2024-09-17T04:06:02.802781+0000","last_became_peered":"2024-09-17T04:06:02.802781+0000","last_unstale":"2024-09-17T04:06:06.004421+0000","last_undegraded":"2024-09-17T04:06:06.004421+0000","last_fullsized":"2024-09-17T04:06:06.004421+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:48:17.168758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.708027+0000","last_change":"2024-09-17T04:06:03.812530+0000","last_active":"2024-09-17T04:06:05.708027+0000","last_peered":"2024-09-17T04:06:05.708027+0000","last_clean":"2024-09-17T04:06:05.708027+0000","last_became_active":"2024-09-17T04:06:03.812208+0000","last_became_peered":"2024-09-17T04:06:03.812208+0000","last_unstale":"2024-09-17T04:06:05.708027+0000","last_undegraded":"2024-09-17T04:06:05.708027+0000","last_fullsized":"2024-09-17T04:06:05.708027+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:54:34.711499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"232'2","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820510+0000","last_change":"2024-09-17T04:06:02.809075+0000","last_active":"2024-09-17T04:06:03.820510+0000","last_peered":"2024-09-17T04:06:03.820510+0000","last_clean":"2024-09-17T04:06:03.820510+0000","last_became_active":"2024-09-17T04:06:02.808749+0000","last_became_peered":"2024-09-17T04:06:02.808749+0000","last_unstale":"2024-09-17T04:06:03.820510+0000","last_undegraded":"2024-09-17T04:06:03.820510+0000","last_fullsized":"2024-09-17T04:06:03.820510+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:14.014465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.813152+0000","last_change":"2024-09-17T04:06:03.806209+0000","last_active":"2024-09-17T04:06:03.813152+0000","last_peered":"2024-09-17T04:06:03.813152+0000","last_clean":"2024-09-17T04:06:03.813152+0000","last_became_active":"2024-09-17T04:06:03.803114+0000","last_became_peered":"2024-09-17T04:06:03.803114+0000","last_unstale":"2024-09-17T04:06:03.813152+0000","last_undegraded":"2024-09-17T04:06:03.813152+0000","last_fullsized":"2024-09-17T04:06:03.813152+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:47:02.206329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.790903+0000","last_change":"2024-09-17T04:06:02.800746+0000","last_active":"2024-09-17T04:06:03.790903+0000","last_peered":"2024-09-17T04:06:03.790903+0000","last_clean":"2024-09-17T04:06:03.790903+0000","last_became_active":"2024-09-17T04:06:02.800425+0000","last_became_peered":"2024-09-17T04:06:02.800425+0000","last_unstale":"2024-09-17T04:06:03.790903+0000","last_undegraded":"2024-09-17T04:06:03.790903+0000","last_fullsized":"2024-09-17T04:06:03.790903+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:39:58.913127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.707770+0000","last_change":"2024-09-17T04:06:03.817742+0000","last_active":"2024-09-17T04:06:05.707770+0000","last_peered":"2024-09-17T04:06:05.707770+0000","last_clean":"2024-09-17T04:06:05.707770+0000","last_became_active":"2024-09-17T04:06:03.817322+0000","last_became_peered":"2024-09-17T04:06:03.817322+0000","last_unstale":"2024-09-17T04:06:05.707770+0000","last_undegraded":"2024-09-17T04:06:05.707770+0000","last_fullsized":"2024-09-17T04:06:05.707770+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:33:04.400169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.801098+0000","last_change":"2024-09-17T04:06:02.806083+0000","last_active":"2024-09-17T04:06:03.801098+0000","last_peered":"2024-09-17T04:06:03.801098+0000","last_clean":"2024-09-17T04:06:03.801098+0000","last_became_active":"2024-09-17T04:06:02.804581+0000","last_became_peered":"2024-09-17T04:06:02.804581+0000","last_unstale":"2024-09-17T04:06:03.801098+0000","last_undegraded":"2024-09-17T04:06:03.801098+0000","last_fullsized":"2024-09-17T04:06:03.801098+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:40.805379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.812338+0000","last_change":"2024-09-17T04:06:03.805922+0000","last_active":"2024-09-17T04:06:03.812338+0000","last_peered":"2024-09-17T04:06:03.812338+0000","last_clean":"2024-09-17T04:06:03.812338+0000","last_became_active":"2024-09-17T04:06:03.805635+0000","last_became_peered":"2024-09-17T04:06:03.805635+0000","last_unstale":"2024-09-17T04:06:03.812338+0000","last_undegraded":"2024-09-17T04:06:03.812338+0000","last_fullsized":"2024-09-17T04:06:03.812338+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:12:05.884244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.804946+0000","last_change":"2024-09-17T04:06:02.806327+0000","last_active":"2024-09-17T04:06:03.804946+0000","last_peered":"2024-09-17T04:06:03.804946+0000","last_clean":"2024-09-17T04:06:03.804946+0000","last_became_active":"2024-09-17T04:06:02.805995+0000","last_became_peered":"2024-09-17T04:06:02.805995+0000","last_unstale":"2024-09-17T04:06:03.804946+0000","last_undegraded":"2024-09-17T04:06:03.804946+0000","last_fullsized":"2024-09-17T04:06:03.804946+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:20:37.924620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.820866+0000","last_change":"2024-09-17T04:06:03.818557+0000","last_active":"2024-09-17T04:06:03.820866+0000","last_peered":"2024-09-17T04:06:03.820866+0000","last_clean":"2024-09-17T04:06:03.820866+0000","last_became_active":"2024-09-17T04:06:03.818308+0000","last_became_peered":"2024-09-17T04:06:03.818308+0000","last_unstale":"2024-09-17T04:06:03.820866+0000","last_undegraded":"2024-09-17T04:06:03.820866+0000","last_fullsized":"2024-09-17T04:06:03.820866+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:10:16.224591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:05.708079+0000","last_change":"2024-09-17T04:06:02.797925+0000","last_active":"2024-09-17T04:06:05.708079+0000","last_peered":"2024-09-17T04:06:05.708079+0000","last_clean":"2024-09-17T04:06:05.708079+0000","last_became_active":"2024-09-17T04:06:02.797589+0000","last_became_peered":"2024-09-17T04:06:02.797589+0000","last_unstale":"2024-09-17T04:06:05.708079+0000","last_undegraded":"2024-09-17T04:06:05.708079+0000","last_fullsized":"2024-09-17T04:06:05.708079+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:42:19.382211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.809534+0000","last_change":"2024-09-17T04:06:03.801247+0000","last_active":"2024-09-17T04:06:03.809534+0000","last_peered":"2024-09-17T04:06:03.809534+0000","last_clean":"2024-09-17T04:06:03.809534+0000","last_became_active":"2024-09-17T04:06:03.800917+0000","last_became_peered":"2024-09-17T04:06:03.800917+0000","last_unstale":"2024-09-17T04:06:03.809534+0000","last_undegraded":"2024-09-17T04:06:03.809534+0000","last_fullsized":"2024-09-17T04:06:03.809534+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:31.254968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004609+0000","last_change":"2024-09-17T04:06:02.794443+0000","last_active":"2024-09-17T04:06:06.004609+0000","last_peered":"2024-09-17T04:06:06.004609+0000","last_clean":"2024-09-17T04:06:06.004609+0000","last_became_active":"2024-09-17T04:06:02.794093+0000","last_became_peered":"2024-09-17T04:06:02.794093+0000","last_unstale":"2024-09-17T04:06:06.004609+0000","last_undegraded":"2024-09-17T04:06:06.004609+0000","last_fullsized":"2024-09-17T04:06:06.004609+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:51:41.100886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.811634+0000","last_change":"2024-09-17T04:06:03.804057+0000","last_active":"2024-09-17T04:06:03.811634+0000","last_peered":"2024-09-17T04:06:03.811634+0000","last_clean":"2024-09-17T04:06:03.811634+0000","last_became_active":"2024-09-17T04:06:03.803711+0000","last_became_peered":"2024-09-17T04:06:03.803711+0000","last_unstale":"2024-09-17T04:06:03.811634+0000","last_undegraded":"2024-09-17T04:06:03.811634+0000","last_fullsized":"2024-09-17T04:06:03.811634+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:05:42.604735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-17T04:06:06.004620+0000","last_change":"2024-09-17T04:06:02.798831+0000","last_active":"2024-09-17T04:06:06.004620+0000","last_peered":"2024-09-17T04:06:06.004620+0000","last_clean":"2024-09-17T04:06:06.004620+0000","last_became_active":"2024-09-17T04:06:02.798500+0000","last_became_peered":"2024-09-17T04:06:02.798500+0000","last_unstale":"2024-09-17T04:06:06.004620+0000","last_undegraded":"2024-09-17T04:06:06.004620+0000","last_fullsized":"2024-09-17T04:06:06.004620+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:24:11.859915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.818562+0000","last_change":"2024-09-17T04:06:03.811222+0000","last_active":"2024-09-17T04:06:03.818562+0000","last_peered":"2024-09-17T04:06:03.818562+0000","last_clean":"2024-09-17T04:06:03.818562+0000","last_became_active":"2024-09-17T04:06:03.810800+0000","last_became_peered":"2024-09-17T04:06:03.810800+0000","last_unstale":"2024-09-17T04:06:03.818562+0000","last_undegraded":"2024-09-17T04:06:03.818562+0000","last_fullsized":"2024-09-17T04:06:03.818562+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:43:21.623918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":13,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.780447+0000","last_change":"2024-09-17T04:06:02.786094+0000","last_active":"2024-09-17T04:06:03.780447+0000","last_peered":"2024-09-17T04:06:03.780447+0000","last_clean":"2024-09-17T04:06:03.780447+0000","last_became_active":"2024-09-17T04:06:02.785792+0000","last_became_peered":"2024-09-17T04:06:02.785792+0000","last_unstale":"2024-09-17T04:06:03.780447+0000","last_undegraded":"2024-09-17T04:06:03.780447+0000","last_fullsized":"2024-09-17T04:06:03.780447+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:01.762808+0000","last_clean_scrub_stamp":"2024-09-17T04:06:01.762808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:59:36.650214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":10,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.817062+0000","last_change":"2024-09-17T04:06:03.811229+0000","last_active":"2024-09-17T04:06:03.817062+0000","last_peered":"2024-09-17T04:06:03.817062+0000","last_clean":"2024-09-17T04:06:03.817062+0000","last_became_active":"2024-09-17T04:06:03.810843+0000","last_became_peered":"2024-09-17T04:06:03.810843+0000","last_unstale":"2024-09-17T04:06:03.817062+0000","last_undegraded":"2024-09-17T04:06:03.817062+0000","last_fullsized":"2024-09-17T04:06:03.817062+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:02.766508+0000","last_clean_scrub_stamp":"2024-09-17T04:06:02.766508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:27.620977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":499,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-17T04:06:03.796429+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:03.796429+0000","last_peered":"2024-09-17T04:06:03.796429+0000","last_clean":"2024-09-17T04:06:03.796429+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:03.796429+0000","last_undegraded":"2024-09-17T04:06:03.796429+0000","last_fullsized":"2024-09-17T04:06:03.796429+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509642,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52684,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319156,"statfs":{"total":96636764160,"available":96582815744,"internally_reserved":0,"allocated":3526656,"data_stored":1888339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509645,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29880,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341960,"statfs":{"total":96636764160,"available":96606167040,"internally_reserved":0,"allocated":2920448,"data_stored":1291196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52112,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319728,"statfs":{"total":96636764160,"available":96583401472,"internally_reserved":0,"allocated":2940928,"data_stored":1298063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509642,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56276,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315564,"statfs":{"total":96636764160,"available":96579137536,"internally_reserved":0,"allocated":2945024,"data_stored":1299583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56268,"kb_used_data":2868,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315572,"statfs":{"total":96636764160,"available":96579145728,"internally_reserved":0,"allocated":2936832,"data_stored":1298061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509644,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52088,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319752,"statfs":{"total":96636764160,"available":96583426048,"internally_reserved":0,"allocated":2920448,"data_stored":1291196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542347,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52124,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319716,"statfs":{"total":96636764160,"available":96583389184,"internally_reserved":0,"allocated":2949120,"data_stored":1304361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542348,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52704,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319136,"statfs":{"total":96636764160,"available":96582795264,"internally_reserved":0,"allocated":3551232,"data_stored":1895277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:08.894 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:06:08.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:06:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 -- 172.21.15.129:0/2753127283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b340ad8b0 msgr2=0x7f4b340b3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/2753127283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b340ad8b0 0x7f4b340b3180 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f4b34146530 tx=0x7f4b2c02aa30 comp rx=0 tx=0).stop 2024-09-17T04:06:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 -- 172.21.15.129:0/2753127283 shutdown_connections 2024-09-17T04:06:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/2753127283 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b340ad8b0 0x7f4b340b3180 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/2753127283 >> v2:172.21.15.181:3300/0 conn(0x7f4b3406eee0 0x7f4b340ad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 -- 172.21.15.129:0/2753127283 >> 172.21.15.129:0/2753127283 conn(0x7f4b3406d7a0 msgr2=0x7f4b340b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 -- 172.21.15.129:0/2753127283 shutdown_connections 2024-09-17T04:06:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.159+0000 7f4b3bc00640 1 -- 172.21.15.129:0/2753127283 wait complete. 2024-09-17T04:06:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.161+0000 7f4b3bc00640 1 Processor -- start 2024-09-17T04:06:09.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.161+0000 7f4b3bc00640 1 -- start start 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b3406eee0 0x7f4b3413c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b340ad8b0 0x7f4b3413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340b3840 con 0x7f4b3409a4a0 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4b340b36c0 con 0x7f4b340ad8b0 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.162+0000 7f4b3bc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4b340b39c0 con 0x7f4b3406eee0 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51990/0 (socket says 172.21.15.129:51990) 2024-09-17T04:06:09.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 -- 172.21.15.129:0/818345022 learned_addr learned my addr 172.21.15.129:0/818345022 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b33e00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b340ad8b0 0x7f4b3413eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39200640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b3406eee0 0x7f4b3413c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 -- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b3406eee0 msgr2=0x7f4b3413c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b3406eee0 0x7f4b3413c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 -- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b340ad8b0 msgr2=0x7f4b3413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b340ad8b0 0x7f4b3413eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.163+0000 7f4b39c00640 1 -- 172.21.15.129:0/818345022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3413c9a0 con 0x7f4b3409a4a0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b39c00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c005fd0 tx=0x7f4b2c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c033050 con 0x7f4b3409a4a0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4b2c004450 con 0x7f4b3409a4a0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b2c004840 con 0x7f4b3409a4a0 2024-09-17T04:06:09.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b34150590 con 0x7f4b3409a4a0 2024-09-17T04:06:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.164+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b34150830 con 0x7f4b3409a4a0 2024-09-17T04:06:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.165+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b340b3ee0 con 0x7f4b3409a4a0 2024-09-17T04:06:09.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.166+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4b2c036070 con 0x7f4b3409a4a0 2024-09-17T04:06:09.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.167+0000 7f4b30c00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 0x7f4b1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.167+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f4b2c0bc1b0 con 0x7f4b3409a4a0 2024-09-17T04:06:09.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.168+0000 7f4b39200640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 0x7f4b1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.169+0000 7f4b39200640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 0x7f4b1407b0a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000f60 tx=0x7f4b24009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:09.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.172+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4b2c083f00 con 0x7f4b3409a4a0 2024-09-17T04:06:09.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.423+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f4b340b2c80 con 0x7f4b3409a4a0 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.427+0000 7f4b30c00640 1 -- 172.21.15.129:0/818345022 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v45) ==== 190+0+147 (secure 0 0 0) 0x7f4b2c088930 con 0x7f4b3409a4a0 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:06:09.427 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:06:09.428 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 msgr2=0x7f4b1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 0x7f4b1407b0a0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4b24000f60 tx=0x7f4b24009040 comp rx=0 tx=0).stop 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 msgr2=0x7f4b341515f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c005fd0 tx=0x7f4b2c003f10 comp rx=0 tx=0).stop 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 shutdown_connections 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4b14078b90 0x7f4b1407b0a0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4b3409a4a0 0x7f4b341515f0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b340ad8b0 0x7f4b3413eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 --2- 172.21.15.129:0/818345022 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4b3406eee0 0x7f4b3413c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.429+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 >> 172.21.15.129:0/818345022 conn(0x7f4b3406d7a0 msgr2=0x7f4b3415f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.430+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 shutdown_connections 2024-09-17T04:06:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.430+0000 7f4b3bc00640 1 -- 172.21.15.129:0/818345022 wait complete. 2024-09-17T04:06:09.453 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:09.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 -- 172.21.15.129:0/71273662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 msgr2=0x7f8154005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 --2- 172.21.15.129:0/71273662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8154005150 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f8168005fd0 tx=0x7f8154005830 comp rx=0 tx=0).stop 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175000640 1 -- 172.21.15.129:0/71273662 reap_dead start 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 -- 172.21.15.129:0/71273662 shutdown_connections 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 -- 172.21.15.129:0/71273662 >> 172.21.15.129:0/71273662 conn(0x7f817008bff0 msgr2=0x7f817009d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 -- 172.21.15.129:0/71273662 shutdown_connections 2024-09-17T04:06:09.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.676+0000 7f8175a00640 1 -- 172.21.15.129:0/71273662 wait complete. 2024-09-17T04:06:09.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.678+0000 7f8175a00640 1 Processor -- start 2024-09-17T04:06:09.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.678+0000 7f8175a00640 1 -- start start 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f817015eaa0 0x7f817015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8170160b20 0x7f8170160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81700d3400 con 0x7f8154004d40 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f81700d3280 con 0x7f817015eaa0 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f817015dd30 con 0x7f8170160b20 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52014/0 (socket says 172.21.15.129:52014) 2024-09-17T04:06:09.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 -- 172.21.15.129:0/3829987001 learned_addr learned my addr 172.21.15.129:0/3829987001 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f8175000640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8170160b20 0x7f8170160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816f400640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f817015eaa0 0x7f817015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 -- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8170160b20 msgr2=0x7f8170160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8170160b20 0x7f8170160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 -- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f817015eaa0 msgr2=0x7f817015eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f817015eaa0 0x7f817015eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.680+0000 7f816fe00640 1 -- 172.21.15.129:0/3829987001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8170161470 con 0x7f8154004d40 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f816fe00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f8160011ae0 tx=0x7f8160009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816001e070 con 0x7f8154004d40 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f816000fc70 con 0x7f8154004d40 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160019410 con 0x7f8154004d40 2024-09-17T04:06:09.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f817015a7d0 con 0x7f8154004d40 2024-09-17T04:06:09.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.681+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f817015ad90 con 0x7f8154004d40 2024-09-17T04:06:09.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.682+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8170076fb0 con 0x7f8154004d40 2024-09-17T04:06:09.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.684+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8160021070 con 0x7f8154004d40 2024-09-17T04:06:09.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.684+0000 7f8167e00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 0x7f815007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:09.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.685+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f816009b5d0 con 0x7f8154004d40 2024-09-17T04:06:09.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.685+0000 7f816f400640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 0x7f815007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:09.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.686+0000 7f816f400640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 0x7f815007b0d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f817015ea50 tx=0x7f815c0014f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:09.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.689+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8170076fb0 con 0x7f8154004d40 2024-09-17T04:06:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.907+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81700b3440 con 0x7f8154004d40 2024-09-17T04:06:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.908+0000 7f8167e00640 1 -- 172.21.15.129:0/3829987001 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f8170161470 con 0x7f8154004d40 2024-09-17T04:06:09.909 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:09.909 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:09.909 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:09.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 msgr2=0x7f815007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 0x7f815007b0d0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f817015ea50 tx=0x7f815c0014f0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 msgr2=0x7f8170150c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:09.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f8160011ae0 tx=0x7f8160009ed0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 shutdown_connections 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8150078bc0 0x7f815007b0d0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8170160b20 0x7f8170160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f817015eaa0 0x7f817015eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 --2- 172.21.15.129:0/3829987001 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8154004d40 0x7f8170150c90 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 >> 172.21.15.129:0/3829987001 conn(0x7f817008bff0 msgr2=0x7f8170161dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 shutdown_connections 2024-09-17T04:06:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:09.911+0000 7f8175a00640 1 -- 172.21.15.129:0/3829987001 wait complete. 2024-09-17T04:06:10.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 227, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:06:03.281568+0000', 'modified': '2024-09-17T04:06:06.475395+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11460}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11460': {'gid': 11460, 'name': 'b', 'rank': 0, 'incarnation': 224, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.181:6836/4204505060', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 4204505060}, {'type': 'v1', 'addr': '172.21.15.181:6836', 'nonce': 4204505060}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11460, 'qdb_cluster': [11460]} 2024-09-17T04:06:10.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:06:10.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-17T04:06:10.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 -- 172.21.15.129:0/3487944563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b9be0 msgr2=0x7fb8e00d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 --2- 172.21.15.129:0/3487944563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b9be0 0x7fb8e00d0f90 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0005fd0 tx=0x7fb8d002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 -- 172.21.15.129:0/3487944563 shutdown_connections 2024-09-17T04:06:10.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 --2- 172.21.15.129:0/3487944563 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8e00d0050 0x7fb8e00d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 --2- 172.21.15.129:0/3487944563 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b9be0 0x7fb8e00d0f90 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 -- 172.21.15.129:0/3487944563 >> 172.21.15.129:0/3487944563 conn(0x7fb8e008bff0 msgr2=0x7fb8e009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 -- 172.21.15.129:0/3487944563 shutdown_connections 2024-09-17T04:06:10.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.563+0000 7fb8e6600640 1 -- 172.21.15.129:0/3487944563 wait complete. 2024-09-17T04:06:10.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.565+0000 7fb8e6600640 1 Processor -- start 2024-09-17T04:06:10.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.565+0000 7fb8e6600640 1 -- start start 2024-09-17T04:06:10.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8e00b9be0 0x7fb8e00cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 0x7fb8e00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b1c60 con 0x7fb8e00b3430 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00b1ae0 con 0x7fb8e00d0050 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e6600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e00b1de0 con 0x7fb8e00b9be0 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52032/0 (socket says 172.21.15.129:52032) 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 -- 172.21.15.129:0/2125900450 learned_addr learned my addr 172.21.15.129:0/2125900450 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 -- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8e00b9be0 msgr2=0x7fb8e00cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8e00b9be0 0x7fb8e00cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 -- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 msgr2=0x7fb8e00cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 0x7fb8e00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 0x7fb8e00cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5200640 1 -- 172.21.15.129:0/2125900450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e00cc600 con 0x7fb8e00b3430 2024-09-17T04:06:10.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.567+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 0x7fb8e00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:10.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0034040 tx=0x7fb8d0004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8dcc00640 1 -- 172.21.15.129:0/2125900450 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d003e030 con 0x7fb8e00b3430 2024-09-17T04:06:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8dcc00640 1 -- 172.21.15.129:0/2125900450 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8d0004e80 con 0x7fb8e00b3430 2024-09-17T04:06:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8dcc00640 1 -- 172.21.15.129:0/2125900450 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0035040 con 0x7fb8e00b3430 2024-09-17T04:06:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e00cadd0 con 0x7fb8e00b3430 2024-09-17T04:06:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.568+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e00cd690 con 0x7fb8e00b3430 2024-09-17T04:06:10.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.570+0000 7fb8dcc00640 1 -- 172.21.15.129:0/2125900450 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8d0041030 con 0x7fb8e00b3430 2024-09-17T04:06:10.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.571+0000 7fb8dcc00640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 0x7fb8c007b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.571+0000 7fb8dcc00640 1 -- 172.21.15.129:0/2125900450 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7fb8d00bb360 con 0x7fb8e00b3430 2024-09-17T04:06:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.572+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 0x7fb8c007b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.572+0000 7fb8eeddb740 1 Processor -- start 2024-09-17T04:06:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.572+0000 7fb8eeddb740 1 -- start start 2024-09-17T04:06:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 0x7fb8c007b100 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb8e006e130 tx=0x7fb8cc007930 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92e2270 0x55efc96b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92c1590 0x55efc9683150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55efc960c430 con 0x55efc92b93b0 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55efc95fc5d0 con 0x55efc92c1590 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.573+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55efc95eba00 con 0x55efc92e2270 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52040/0 (socket says 172.21.15.129:52040) 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 -- 172.21.15.129:0/2180601202 learned_addr learned my addr 172.21.15.129:0/2180601202 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92e2270 0x55efc96b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92c1590 0x55efc9683150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 -- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92e2270 msgr2=0x55efc96b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92e2270 0x55efc96b2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 -- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92c1590 msgr2=0x55efc9683150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92c1590 0x55efc9683150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 -- 172.21.15.129:0/2180601202 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc95b4520 con 0x55efc92b93b0 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.574+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc005550 tx=0x7fb8cc00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644c05d064f4cd3f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.575+0000 7fb8d5600640 1 -- 172.21.15.129:0/2180601202 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc016510 con 0x55efc92b93b0 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.575+0000 7fb8d5600640 1 -- 172.21.15.129:0/2180601202 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8cc0166b0 con 0x55efc92b93b0 2024-09-17T04:06:10.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.575+0000 7fb8d5600640 1 -- 172.21.15.129:0/2180601202 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc016ac0 con 0x55efc92b93b0 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.575+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2180601202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 msgr2=0x55efc96b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.575+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc005550 tx=0x7fb8cc00f000 comp rx=0 tx=0).stop 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2180601202 shutdown_connections 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92c1590 0x55efc9683150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92b93b0 0x55efc96b30a0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2180601202 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92e2270 0x55efc96b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2180601202 >> 172.21.15.129:0/2180601202 conn(0x55efc92d63c0 msgr2=0x55efc9688b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2180601202 shutdown_connections 2024-09-17T04:06:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.576+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2180601202 wait complete. 2024-09-17T04:06:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.577+0000 7fb8eeddb740 1 -- start start 2024-09-17T04:06:10.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.577+0000 7fb8eeddb740 1 Processor -- start 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.578+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9697fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.578+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.578+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55efc9592330 con 0x55efc92e2270 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55efc95ef810 con 0x55efc92b93b0 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55efc935c620 con 0x55efc92c1590 2024-09-17T04:06:10.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8eeddb740 10 client.0 _ll_register_callbacks cb 0x55efc9597420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52052/0 (socket says 172.21.15.129:52052) 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2705901364 learned_addr learned my addr 172.21.15.129:0/2705901364 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9697fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 msgr2=0x55efc96c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 msgr2=0x55efc9697fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9697fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.579+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2705901364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efc934d850 con 0x55efc92e2270 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8010a40 tx=0x7fb8d8010f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6f82e75438ba99 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8ca000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d801e060 con 0x55efc92e2270 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7fb8d800ed40 con 0x55efc92e2270 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8cc0166b0 con 0x55efc92e2270 2024-09-17T04:06:10.582 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=227) ==== 24+0+0 (secure 0 0 0) 0x7fb8d8024050 con 0x55efc92e2270 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8eeddb740 10 client.11574 fetch_fsmap learned FSMap version 227 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.580+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=227,monmap=2+}) -- 0x55efc93c46b0 con 0x55efc92e2270 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8cc0166b0 con 0x55efc92e2270 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 227) ==== 3502+0+0 (secure 0 0 0) 0x7fb8d8027020 con 0x55efc92e2270 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 10 client.11574 fetch_fsmap finished waiting for FSMap version 227 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 10 client.11574 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 20 client.11574 populate_metadata read hostname 'smithi129' 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc92a0e40 0x55efc96af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 4 client.11574 mds_command: new command op to 11460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 --> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efc95fe520 con 0x55efc92a0e40 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.581+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc92a0e40 0x55efc96af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.582+0000 7fb8ca000640 10 client.11574 ms_handle_connect on v2:172.21.15.181:6835/4204505060 2024-09-17T04:06:10.583 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.582+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc92a0e40 0x55efc96af5c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21900d7cea8b4136 server_cookie=1accf3529fc22c23 in_seq=0 out_seq=0 2024-09-17T04:06:10.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8ca000640 1 -- 172.21.15.129:0/2705901364 <== mds.0 v2:172.21.15.181:6835/4204505060 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55efc95fe520 con 0x55efc92a0e40 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8ca000640 10 client.11574 handle_command_reply: tid=0 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 client.11574 shutdown 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 msgr2=0x55efc96ad840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8010a40 tx=0x7fb8d8010f00 comp rx=0 tx=0).stop 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 shutdown_connections 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc92a0e40 0x55efc96af5c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96ad840 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2705901364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9697fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8ca000640 0 client.11574 ms_handle_reset on v2:172.21.15.181:6835/4204505060 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 >> 172.21.15.129:0/2705901364 conn(0x55efc92d63c0 msgr2=0x55efc96aa0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 shutdown_connections 2024-09-17T04:06:10.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.583+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2705901364 wait complete. 2024-09-17T04:06:10.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.584+0000 7fb8eeddb740 20 client.11574 trim_cache size 0 max 16384 2024-09-17T04:06:10.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.602+0000 7fb8eeddb740 1 Processor -- start 2024-09-17T04:06:10.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.602+0000 7fb8eeddb740 1 -- start start 2024-09-17T04:06:10.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 0x55efc96b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01f9d60 con 0x55efc92c1590 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e01f90c0 con 0x55efc92e2270 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e01f9710 con 0x55efc92b93b0 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52062/0 (socket says 172.21.15.129:52062) 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 -- 172.21.15.129:0/3664431564 learned_addr learned my addr 172.21.15.129:0/3664431564 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8e5200640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 0x55efc96b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 -- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 msgr2=0x55efc96b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 0x55efc96b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 -- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92e2270 msgr2=0x55efc96cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8dfe00640 1 -- 172.21.15.129:0/3664431564 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efc96cf8c0 con 0x55efc92c1590 2024-09-17T04:06:10.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.604+0000 7fb8e5200640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 0x55efc96b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.605+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc003e50 tx=0x7fb8cc0010f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a63fd64cbaa244 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.605+0000 7fb8d4c00640 1 -- 172.21.15.129:0/3664431564 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc006870 con 0x55efc92c1590 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.605+0000 7fb8d4c00640 1 -- 172.21.15.129:0/3664431564 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8cc006a10 con 0x55efc92c1590 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.605+0000 7fb8d4c00640 1 -- 172.21.15.129:0/3664431564 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc006e40 con 0x55efc92c1590 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 -- 172.21.15.129:0/3664431564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 msgr2=0x55efc96cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc003e50 tx=0x7fb8cc0010f0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 -- 172.21.15.129:0/3664431564 shutdown_connections 2024-09-17T04:06:10.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/3664431564 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92b93b0 0x55efc96b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 -- 172.21.15.129:0/3664431564 >> 172.21.15.129:0/3664431564 conn(0x55efc92d63c0 msgr2=0x55efc9688b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 -- 172.21.15.129:0/3664431564 shutdown_connections 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.606+0000 7fb8eeddb740 1 -- 172.21.15.129:0/3664431564 wait complete. 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.607+0000 7fb8eeddb740 1 -- start start 2024-09-17T04:06:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.607+0000 7fb8eeddb740 1 Processor -- start 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.608+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9744370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.608+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8eeddb740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55efc96adfb0 con 0x55efc92e2270 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55efc96987c0 con 0x55efc92b93b0 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8eeddb740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55efc969a3b0 con 0x55efc92c1590 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8eeddb740 10 client.0 _ll_register_callbacks cb 0x7fb8d8010600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52078/0 (socket says 172.21.15.129:52078) 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2810763852 learned_addr learned my addr 172.21.15.129:0/2810763852 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9744370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8dfe00640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 msgr2=0x55efc96cfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 msgr2=0x55efc9744370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9744370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.609+0000 7fb8e5c00640 1 -- 172.21.15.129:0/2810763852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efc9689840 con 0x55efc92e2270 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8e5c00640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x55efc968a620 tx=0x7fb8d800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3436602aeb27f79 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d802c070 con 0x55efc92e2270 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7fb8d8012c10 con 0x55efc92e2270 2024-09-17T04:06:10.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8cc006a10 con 0x55efc92e2270 2024-09-17T04:06:10.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=227) ==== 24+0+0 (secure 0 0 0) 0x7fb8d8012e10 con 0x55efc92e2270 2024-09-17T04:06:10.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8eeddb740 10 client.11580 fetch_fsmap learned FSMap version 227 2024-09-17T04:06:10.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=227,monmap=2+}) -- 0x55efc96f1300 con 0x55efc92e2270 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8cc006a10 con 0x55efc92e2270 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 227) ==== 3502+0+0 (secure 0 0 0) 0x7fb8d803a020 con 0x55efc92e2270 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.610+0000 7fb8eeddb740 10 client.11580 fetch_fsmap finished waiting for FSMap version 227 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8eeddb740 10 client.11580 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8eeddb740 20 client.11580 populate_metadata read hostname 'smithi129' 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc96995c0 0x55efc96ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8eeddb740 4 client.11580 mds_command: new command op to 11460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 --> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] -- command(tid 0: {"prefix": "status"}) -- 0x55efc95f9fd0 con 0x55efc96995c0 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.611+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc96995c0 0x55efc96ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8e5200640 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc96995c0 0x55efc96ab3f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3651b81c84d688c1 server_cookie=40270778841c1abc in_seq=0 out_seq=0 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8ca000640 10 client.11580 ms_handle_connect on v2:172.21.15.181:6835/4204505060 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8ca000640 1 -- 172.21.15.129:0/2810763852 <== mds.0 v2:172.21.15.181:6835/4204505060 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55efc95f9fd0 con 0x55efc96995c0 2024-09-17T04:06:10.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8ca000640 10 client.11580 handle_command_reply: tid=0 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 client.11580 shutdown 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 msgr2=0x55efc96cfff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x55efc968a620 tx=0x7fb8d800be10 comp rx=0 tx=0).stop 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 shutdown_connections 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:6835/4204505060,v1:172.21.15.181:6836/4204505060] conn(0x55efc96995c0 0x55efc96ab3f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55efc92e2270 0x55efc96cfff0 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55efc92c1590 0x55efc96cfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8ca000640 0 client.11580 ms_handle_reset on v2:172.21.15.181:6835/4204505060 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 --2- 172.21.15.129:0/2810763852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55efc92b93b0 0x55efc9744370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 >> 172.21.15.129:0/2810763852 conn(0x55efc92d63c0 msgr2=0x55efc96a9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 shutdown_connections 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.612+0000 7fb8eeddb740 1 -- 172.21.15.129:0/2810763852 wait complete. 2024-09-17T04:06:10.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8eeddb740 20 client.11580 trim_cache size 0 max 16384 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 11460, 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.8357149760000002, 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 227, 2024-09-17T04:06:10.616 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 233, 2024-09-17T04:06:10.617 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 233, 2024-09-17T04:06:10.617 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.883134153 2024-09-17T04:06:10.617 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 msgr2=0x7fb8c007b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 0x7fb8c007b100 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fb8e006e130 tx=0x7fb8cc007930 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 msgr2=0x7fb8e015b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0034040 tx=0x7fb8d0004990 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 shutdown_connections 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8c0078bf0 0x7fb8c007b100 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8e00d0050 0x7fb8e00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8e00b9be0 0x7fb8e00cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 --2- 172.21.15.129:0/2125900450 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8e00b3430 0x7fb8e015b190 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 >> 172.21.15.129:0/2125900450 conn(0x7fb8e008bff0 msgr2=0x7fb8e00d45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 shutdown_connections 2024-09-17T04:06:10.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.613+0000 7fb8e6600640 1 -- 172.21.15.129:0/2125900450 wait complete. 2024-09-17T04:06:10.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:06:10.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 -- 172.21.15.129:0/2627603337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 msgr2=0x7f997c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 --2- 172.21.15.129:0/2627603337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f997c00b9c0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f9990005fd0 tx=0x7f997c005830 comp rx=0 tx=0).stop 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 -- 172.21.15.129:0/2627603337 shutdown_connections 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 --2- 172.21.15.129:0/2627603337 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f997c00b9c0 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 -- 172.21.15.129:0/2627603337 >> 172.21.15.129:0/2627603337 conn(0x7f999406d410 msgr2=0x7f99940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 -- 172.21.15.129:0/2627603337 shutdown_connections 2024-09-17T04:06:10.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.865+0000 7f999c800640 1 -- 172.21.15.129:0/2627603337 wait complete. 2024-09-17T04:06:10.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.867+0000 7f999c800640 1 Processor -- start 2024-09-17T04:06:10.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.868+0000 7f999c800640 1 -- start start 2024-09-17T04:06:10.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99941f5ef0 0x7f99941f6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9994154dd0 0x7f99941551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99940cd420 con 0x7f997c005270 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f99940cd2a0 con 0x7f99941f5ef0 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f999c800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f99940cd5a0 con 0x7f9994154dd0 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f9999e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f9999e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52094/0 (socket says 172.21.15.129:52094) 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f9999e00640 1 -- 172.21.15.129:0/3839300272 learned_addr learned my addr 172.21.15.129:0/3839300272 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.869+0000 7f9999400640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99941f5ef0 0x7f99941f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f999a800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9994154dd0 0x7f99941551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 -- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9994154dd0 msgr2=0x7f99941551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9994154dd0 0x7f99941551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 -- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99941f5ef0 msgr2=0x7f99941f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99941f5ef0 0x7f99941f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 -- 172.21.15.129:0/3839300272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999409d6f0 con 0x7f997c005270 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f9999e00640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f9988006450 tx=0x7f99880039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988015070 con 0x7f997c005270 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9988011840 con 0x7f997c005270 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.870+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9988011c30 con 0x7f997c005270 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.871+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9994155720 con 0x7f997c005270 2024-09-17T04:06:10.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.871+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9994153080 con 0x7f997c005270 2024-09-17T04:06:10.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.872+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9994070840 con 0x7f997c005270 2024-09-17T04:06:10.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.873+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9988018070 con 0x7f997c005270 2024-09-17T04:06:10.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.874+0000 7f998e000640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 0x7f997807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:10.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.874+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f998809b8e0 con 0x7f997c005270 2024-09-17T04:06:10.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.874+0000 7f9999400640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 0x7f997807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:10.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.875+0000 7f9999400640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 0x7f997807b0a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9984000c00 tx=0x7f9984006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:10.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:10.878+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9994155720 con 0x7f997c005270 2024-09-17T04:06:11.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.097+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f999406a140 con 0x7f997c005270 2024-09-17T04:06:11.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.097+0000 7f998e000640 1 -- 172.21.15.129:0/3839300272 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v227) ==== 53+0+73 (secure 0 0 0) 0x7f99880636d0 con 0x7f997c005270 2024-09-17T04:06:11.098 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:06:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 msgr2=0x7f997807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 0x7f997807b0a0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f9984000c00 tx=0x7f9984006b70 comp rx=0 tx=0).stop 2024-09-17T04:06:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 msgr2=0x7f99941f32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f9988006450 tx=0x7f99880039c0 comp rx=0 tx=0).stop 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 shutdown_connections 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9978078b90 0x7f997807b0a0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9994154dd0 0x7f99941551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99941f5ef0 0x7f99941f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 --2- 172.21.15.129:0/3839300272 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f997c005270 0x7f99941f32d0 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 >> 172.21.15.129:0/3839300272 conn(0x7f999406d410 msgr2=0x7f99940cb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 shutdown_connections 2024-09-17T04:06:11.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11.100+0000 7f999c800640 1 -- 172.21.15.129:0/3839300272 wait complete. 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:06:11.127 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:06:11.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.128 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:06:11.164 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:06:11.164 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:06:11.164 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:06:11.164 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.164 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:06:11.165 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41721sec preferred_lft 41721sec 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.166 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:06:11.167 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:06:11.167 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:06:11.167 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:06:11.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.167 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:06:11.258 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:06:11.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.261 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:06:11.297 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:06:11.297 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:06:11.297 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:06:11.297 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:06:11.297 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:06:11.298 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:06:11.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:06:11.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:06:11.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.494 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:06:11.530 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:06:11.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:06:11.626 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:06:11.626 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:06:11.661 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:06:11.661 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:06:11.661 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:06:11.736 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:06:11.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:06:11.933 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:06:11.933 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:06:11.933 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:06:11.954 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:06:11.954 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:06:11.954 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:06:11.954 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:06:11.954 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:06:11.985 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:06:11.986 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:06:11.986 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:06:11.986 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:06:11.987 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:06:11.987 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:06:11.989 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:06:11.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:11.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:06:12.047 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:06:12.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:06:12.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:06:12.146 INFO:teuthology.orchestra.run.smithi129.stdout:client10675 2024-09-17T04:06:12.146 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:12.146 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:12.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_fragment_size_max 50 2024-09-17T04:06:12.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 -- 172.21.15.129:0/1480749100 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 msgr2=0x7f19ac0cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 --2- 172.21.15.129:0/1480749100 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0cb080 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f199c009830 tx=0x7f199c02a410 comp rx=0 tx=0).stop 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 -- 172.21.15.129:0/1480749100 shutdown_connections 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 --2- 172.21.15.129:0/1480749100 >> v2:172.21.15.181:3300/0 conn(0x7f19ac0704c0 0x7f19ac0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 --2- 172.21.15.129:0/1480749100 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0cb080 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 -- 172.21.15.129:0/1480749100 >> 172.21.15.129:0/1480749100 conn(0x7f19ac0d00b0 msgr2=0x7f19ac0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 -- 172.21.15.129:0/1480749100 shutdown_connections 2024-09-17T04:06:12.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.446+0000 7f19b0e00640 1 -- 172.21.15.129:0/1480749100 wait complete. 2024-09-17T04:06:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.448+0000 7f19b0e00640 1 Processor -- start 2024-09-17T04:06:12.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.448+0000 7f19b0e00640 1 -- start start 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19ac0704c0 0x7f19ac0c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0ceb40 con 0x7f19ac0700b0 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f19ac0ce9c0 con 0x7f19ac0704c0 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19b0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f19ac0cecc0 con 0x7f19ac15c7b0 2024-09-17T04:06:12.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52096/0 (socket says 172.21.15.129:52096) 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 -- 172.21.15.129:0/2613554101 learned_addr learned my addr 172.21.15.129:0/2613554101 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19abe00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19a2a00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19ac0704c0 0x7f19ac0c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 -- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 msgr2=0x7f19ac0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 -- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19ac0704c0 msgr2=0x7f19ac0c87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19ac0704c0 0x7f19ac0c87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.450+0000 7f19ab400640 1 -- 172.21.15.129:0/2613554101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19ac0c4f50 con 0x7f19ac0700b0 2024-09-17T04:06:12.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19ab400640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f199c02aa00 tx=0x7f199c003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c03f030 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f199c004510 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19ac156020 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199c037040 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.451+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19ac1565e0 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.452+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19ac0cf190 con 0x7f19ac0700b0 2024-09-17T04:06:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.453+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f199c042070 con 0x7f19ac0700b0 2024-09-17T04:06:12.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.455+0000 7f19a3e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 0x7f198807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:12.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.455+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(233..233 src has 1..233) ==== 5709+0+0 (secure 0 0 0) 0x7f199c0ba740 con 0x7f19ac0700b0 2024-09-17T04:06:12.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.455+0000 7f19a2a00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 0x7f198807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:12.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.456+0000 7f19a2a00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 0x7f198807b0d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1994001040 tx=0x7f1994000fd0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:12.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.460+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f199c049070 con 0x7f19ac0700b0 2024-09-17T04:06:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.644+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f19ac0d0f50 con 0x7f19ac0700b0 2024-09-17T04:06:12.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.649+0000 7f19a3e00640 1 -- 172.21.15.129:0/2613554101 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v67)=0 v67) ==== 124+0+0 (secure 0 0 0) 0x7f199c082530 con 0x7f19ac0700b0 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 msgr2=0x7f198807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 0x7f198807b0d0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f1994001040 tx=0x7f1994000fd0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 msgr2=0x7f19ac0c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f199c02aa00 tx=0x7f199c003b60 comp rx=0 tx=0).stop 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 shutdown_connections 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1988078bc0 0x7f198807b0d0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f19ac15c7b0 0x7f19ac0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19ac0704c0 0x7f19ac0c87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 --2- 172.21.15.129:0/2613554101 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f19ac0700b0 0x7f19ac0c69d0 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.651+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 >> 172.21.15.129:0/2613554101 conn(0x7f19ac0d00b0 msgr2=0x7f19ac075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.652+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 shutdown_connections 2024-09-17T04:06:12.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:12.652+0000 7f19b0e00640 1 -- 172.21.15.129:0/2613554101 wait complete. 2024-09-17T04:06:19.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:06:19.596 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:06:22.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:22.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.912+0000 7ff654600640 1 -- 172.21.15.129:0/3327689124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 msgr2=0x7ff64c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:22.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.912+0000 7ff654600640 1 --2- 172.21.15.129:0/3327689124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c0b4140 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7ff640005fd0 tx=0x7ff64002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:22.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.913+0000 7ff654600640 1 -- 172.21.15.129:0/3327689124 shutdown_connections 2024-09-17T04:06:22.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.913+0000 7ff654600640 1 --2- 172.21.15.129:0/3327689124 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c0b4140 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:22.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.913+0000 7ff654600640 1 -- 172.21.15.129:0/3327689124 >> 172.21.15.129:0/3327689124 conn(0x7ff64c06d410 msgr2=0x7ff64c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:22.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.913+0000 7ff654600640 1 -- 172.21.15.129:0/3327689124 shutdown_connections 2024-09-17T04:06:22.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.913+0000 7ff654600640 1 -- 172.21.15.129:0/3327689124 wait complete. 2024-09-17T04:06:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.915+0000 7ff654600640 1 Processor -- start 2024-09-17T04:06:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.915+0000 7ff654600640 1 -- start start 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff64c06eed0 0x7ff64c15c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff64c0c6290 0x7ff64c0c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c06f2e0 con 0x7ff64c0705a0 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff64c1612e0 con 0x7ff64c0c6290 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.916+0000 7ff654600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff64c06f460 con 0x7ff64c06eed0 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:55264/0 (socket says 172.21.15.129:55264) 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 -- 172.21.15.129:0/796390116 learned_addr learned my addr 172.21.15.129:0/796390116 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651c00640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff64c06eed0 0x7ff64c15c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff652600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff64c0c6290 0x7ff64c0c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 -- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff64c06eed0 msgr2=0x7ff64c15c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff64c06eed0 0x7ff64c15c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 -- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff64c0c6290 msgr2=0x7ff64c0c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff64c0c6290 0x7ff64c0c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 -- 172.21.15.129:0/796390116 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64c15b020 con 0x7ff64c0705a0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.917+0000 7ff651200640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b3c0 tx=0x7ff63800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.918+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638011830 con 0x7ff64c0705a0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.918+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff638016070 con 0x7ff64c0705a0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.918+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638011d80 con 0x7ff64c0705a0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.918+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64c15b300 con 0x7ff64c0705a0 2024-09-17T04:06:22.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.918+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64c15b8c0 con 0x7ff64c0705a0 2024-09-17T04:06:22.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.919+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff64c0b4140 con 0x7ff64c0705a0 2024-09-17T04:06:22.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.920+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff63801b460 con 0x7ff64c0705a0 2024-09-17T04:06:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.921+0000 7ff649600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 0x7ff62c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.921+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7ff63809abb0 con 0x7ff64c0705a0 2024-09-17T04:06:22.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.921+0000 7ff651c00640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 0x7ff62c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:22.922 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.922+0000 7ff651c00640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 0x7ff62c07b0a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:22.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:22.925+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff638062ec0 con 0x7ff64c0705a0 2024-09-17T04:06:23.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.143+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff64c06a140 con 0x7ff64c0705a0 2024-09-17T04:06:23.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.144+0000 7ff649600640 1 -- 172.21.15.129:0/796390116 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7ff6380627c0 con 0x7ff64c0705a0 2024-09-17T04:06:23.145 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:23.145 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:23.145 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 msgr2=0x7ff62c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 0x7ff62c07b0a0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7ff640033000 tx=0x7ff640004730 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 msgr2=0x7ff64c15a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7ff63800b3c0 tx=0x7ff63800b880 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 shutdown_connections 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff62c078b90 0x7ff62c07b0a0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff64c0c6290 0x7ff64c0c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff64c0705a0 0x7ff64c15a580 unknown :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.146+0000 7ff654600640 1 --2- 172.21.15.129:0/796390116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff64c06eed0 0x7ff64c15c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.147+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 >> 172.21.15.129:0/796390116 conn(0x7ff64c06d410 msgr2=0x7ff64c0cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:23.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.147+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 shutdown_connections 2024-09-17T04:06:23.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23.147+0000 7ff654600640 1 -- 172.21.15.129:0/796390116 wait complete. 2024-09-17T04:06:23.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:06:23.424 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-09-17T04:06:23.424 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-09-17T04:06:23.424 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c ' 2024-09-17T04:06:23.424 DEBUG:teuthology.orchestra.run.smithi129:> mkdir a b 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> for i in `seq 1 50`; do 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> touch a/"$i" 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> ln a/"$i" b/"$i" 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> done 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> sync -f a b 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> rm a/* 2024-09-17T04:06:23.425 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:06:23.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:06:23.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:06:23.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 -- 172.21.15.129:0/970109977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 msgr2=0x7f25fc0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 --2- 172.21.15.129:0/970109977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc0cbbf0 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005fd0 tx=0x7f25ec02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 -- 172.21.15.129:0/970109977 shutdown_connections 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 --2- 172.21.15.129:0/970109977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 0x7f25fc0b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 --2- 172.21.15.129:0/970109977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc0cbbf0 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 -- 172.21.15.129:0/970109977 >> 172.21.15.129:0/970109977 conn(0x7f25fc06d410 msgr2=0x7f25fc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 -- 172.21.15.129:0/970109977 shutdown_connections 2024-09-17T04:06:24.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.208+0000 7f2603200640 1 -- 172.21.15.129:0/970109977 wait complete. 2024-09-17T04:06:24.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.211+0000 7f2603200640 1 Processor -- start 2024-09-17T04:06:24.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.211+0000 7f2603200640 1 -- start start 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 0x7f25fc1e9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25fc0b39c0 0x7f25fc0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc06f5e0 con 0x7f25fc0ce170 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc0708e0 con 0x7f25fc0b39c0 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2603200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f25fc06f760 con 0x7f25fc0b2950 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.212+0000 7f2601200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46058/0 (socket says 172.21.15.129:46058) 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 -- 172.21.15.129:0/2056590679 learned_addr learned my addr 172.21.15.129:0/2056590679 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:24.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f25fbe00640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 0x7f25fc1e9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f25fb400640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25fc0b39c0 0x7f25fc0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 -- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 msgr2=0x7f25fc1e9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 0x7f25fc1e9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 -- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25fc0b39c0 msgr2=0x7f25fc0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25fc0b39c0 0x7f25fc0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 -- 172.21.15.129:0/2056590679 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc1f63e0 con 0x7f25fc0ce170 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f2601200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f25f400c730 tx=0x7f25f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.213+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f4003b70 con 0x7f25fc0ce170 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.214+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f25f4003d30 con 0x7f25fc0ce170 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.214+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25fc1e8620 con 0x7f25fc0ce170 2024-09-17T04:06:24.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.214+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f40052d0 con 0x7f25fc0ce170 2024-09-17T04:06:24.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.214+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25fc1e8be0 con 0x7f25fc0ce170 2024-09-17T04:06:24.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.215+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25fc06a140 con 0x7f25fc0ce170 2024-09-17T04:06:24.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.216+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f25f4005680 con 0x7f25fc0ce170 2024-09-17T04:06:24.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.217+0000 7f25f3e00640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 0x7f25dc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:24.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.217+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f25f409ac70 con 0x7f25fc0ce170 2024-09-17T04:06:24.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.217+0000 7f25fbe00640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 0x7f25dc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:24.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.218+0000 7f25fbe00640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 0x7f25dc07b0d0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f25ec033000 tx=0x7f25ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:24.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.221+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f25fc0b3dd0 con 0x7f25fc0ce170 2024-09-17T04:06:24.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.440+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25fc0cd850 con 0x7f25fc0ce170 2024-09-17T04:06:24.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.440+0000 7f25f3e00640 1 -- 172.21.15.129:0/2056590679 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f25f4062920 con 0x7f25fc0ce170 2024-09-17T04:06:24.441 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:24.441 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:24.441 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 msgr2=0x7f25dc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 0x7f25dc07b0d0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f25ec033000 tx=0x7f25ec004730 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 msgr2=0x7f25fc09d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f25f400c730 tx=0x7f25f400cbf0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 shutdown_connections 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f25dc078bc0 0x7f25dc07b0d0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25fc0ce170 0x7f25fc09d0d0 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25fc0b39c0 0x7f25fc0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 --2- 172.21.15.129:0/2056590679 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25fc0b2950 0x7f25fc1e9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 >> 172.21.15.129:0/2056590679 conn(0x7f25fc06d410 msgr2=0x7f25fc06eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 shutdown_connections 2024-09-17T04:06:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:24.443+0000 7f2603200640 1 -- 172.21.15.129:0/2056590679 wait complete. 2024-09-17T04:06:24.466 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:06:24.705 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"num_strays":50}} 2024-09-17T04:06:24.705 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 50 } } 2024-09-17T04:06:24.705 DEBUG:tasks.ceph_test_case:wait_until_equal: 50 != 0, waiting (timeout=60)... 2024-09-17T04:06:29.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:29.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 -- 172.21.15.129:0/1017512592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0b46c0 msgr2=0x7f7a1c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 --2- 172.21.15.129:0/1017512592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0b46c0 0x7f7a1c0ccc80 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c0ce800 tx=0x7f7a0402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 -- 172.21.15.129:0/1017512592 shutdown_connections 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 --2- 172.21.15.129:0/1017512592 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0b46c0 0x7f7a1c0ccc80 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 --2- 172.21.15.129:0/1017512592 >> v2:172.21.15.181:3300/0 conn(0x7f7a1c15da00 0x7f7a1c0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 -- 172.21.15.129:0/1017512592 >> 172.21.15.129:0/1017512592 conn(0x7f7a1c06d410 msgr2=0x7f7a1c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 -- 172.21.15.129:0/1017512592 shutdown_connections 2024-09-17T04:06:29.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.937+0000 7f7a21a00640 1 -- 172.21.15.129:0/1017512592 wait complete. 2024-09-17T04:06:29.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.939+0000 7f7a21a00640 1 Processor -- start 2024-09-17T04:06:29.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.939+0000 7f7a21a00640 1 -- start start 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 0x7f7a1c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 0x7f7a1c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c070990 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7a1c070810 con 0x7f7a1c15da00 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a21a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1c070b10 con 0x7f7a1c0b46c0 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a12000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:29.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a12000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46072/0 (socket says 172.21.15.129:46072) 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.941+0000 7f7a12000640 1 -- 172.21.15.129:0/2871591263 learned_addr learned my addr 172.21.15.129:0/2871591263 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a1aa00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 0x7f7a1c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 -- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 msgr2=0x7f7a1c156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 0x7f7a1c156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 -- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 msgr2=0x7f7a1c0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a1b400640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 0x7f7a1c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 0x7f7a1c0744f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 -- 172.21.15.129:0/2871591263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a1c1f72d0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a1aa00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 0x7f7a1c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a1b400640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 0x7f7a1c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:29.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a12000640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f7a04034040 tx=0x7f7a04003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:29.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0403e030 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a04003e00 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a1c1f75b0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04035040 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.942+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a1c1f7b70 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.944+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a1c06a140 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.945+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7a04041030 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.946+0000 7f7a13400640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 0x7f79f807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:29.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.946+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f7a040bb6e0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:29.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.946+0000 7f7a1aa00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 0x7f79f807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:29.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.947+0000 7f7a1aa00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 0x7f79f807b0d0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009aa0 tx=0x7f7a0c005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:29.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:29.951+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7a040839e0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:30.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.169+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a1c0b29f0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:30.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.169+0000 7f7a13400640 1 -- 172.21.15.129:0/2871591263 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f7a040832e0 con 0x7f7a1c0cd1c0 2024-09-17T04:06:30.169 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:30.170 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 msgr2=0x7f79f807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 0x7f79f807b0d0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c009aa0 tx=0x7f7a0c005db0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 msgr2=0x7f7a1c073fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f7a04034040 tx=0x7f7a04003910 comp rx=0 tx=0).stop 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 shutdown_connections 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f79f8078bc0 0x7f79f807b0d0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a1c15da00 0x7f7a1c0744f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7a1c0cd1c0 0x7f7a1c073fb0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 --2- 172.21.15.129:0/2871591263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7a1c0b46c0 0x7f7a1c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 >> 172.21.15.129:0/2871591263 conn(0x7f7a1c06d410 msgr2=0x7f7a1c0c9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:30.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 shutdown_connections 2024-09-17T04:06:30.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.172+0000 7f7a21a00640 1 -- 172.21.15.129:0/2871591263 wait complete. 2024-09-17T04:06:30.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:06:30.444 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T04:06:30.444 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T04:06:30.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:30.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 -- 172.21.15.129:0/2807466106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00d0a70 msgr2=0x7f02c01558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 --2- 172.21.15.129:0/2807466106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00d0a70 0x7f02c01558b0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f02a8005fd0 tx=0x7f02a802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 -- 172.21.15.129:0/2807466106 shutdown_connections 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 --2- 172.21.15.129:0/2807466106 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c00af280 0x7f02c015ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 --2- 172.21.15.129:0/2807466106 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00d0a70 0x7f02c01558b0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 -- 172.21.15.129:0/2807466106 >> 172.21.15.129:0/2807466106 conn(0x7f02c006cfc0 msgr2=0x7f02c006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 -- 172.21.15.129:0/2807466106 shutdown_connections 2024-09-17T04:06:30.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.666+0000 7f02c5000640 1 -- 172.21.15.129:0/2807466106 wait complete. 2024-09-17T04:06:30.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.668+0000 7f02c5000640 1 Processor -- start 2024-09-17T04:06:30.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.668+0000 7f02c5000640 1 -- start start 2024-09-17T04:06:30.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c00ad8d0 0x7f02c01f95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c01f6150 0x7f02c0151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f02c015f740 con 0x7f02c00af280 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f02c015f5c0 con 0x7f02c00ad8d0 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02c5000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f02c015f8c0 con 0x7f02c01f6150 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02b6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02b6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46096/0 (socket says 172.21.15.129:46096) 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02b6a00640 1 -- 172.21.15.129:0/138630095 learned_addr learned my addr 172.21.15.129:0/138630095 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02bfe00640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c01f6150 0x7f02c0151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:30.672 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.670+0000 7f02bf400640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c00ad8d0 0x7f02c01f95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 -- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c01f6150 msgr2=0x7f02c0151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c01f6150 0x7f02c0151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 -- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c00ad8d0 msgr2=0x7f02c01f95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c00ad8d0 0x7f02c01f95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 -- 172.21.15.129:0/138630095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c00b1db0 con 0x7f02c00af280 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b6a00640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c00 tx=0x7f02a80038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a803e030 con 0x7f02c00af280 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f02a8003ca0 con 0x7f02c00af280 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c00b1820 con 0x7f02c00af280 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a8035040 con 0x7f02c00af280 2024-09-17T04:06:30.673 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.671+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c01f5670 con 0x7f02c00af280 2024-09-17T04:06:30.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.672+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c0076590 con 0x7f02c00af280 2024-09-17T04:06:30.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.674+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f02a8041030 con 0x7f02c00af280 2024-09-17T04:06:30.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.675+0000 7f02b7e00640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 0x7f029c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:30.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.675+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f02a80bb2e0 con 0x7f02c00af280 2024-09-17T04:06:30.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.675+0000 7f02bf400640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 0x7f029c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:30.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.676+0000 7f02bf400640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 0x7f029c07b0d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f02b0001720 tx=0x7f02b0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:30.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.679+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f02a8083010 con 0x7f02c00af280 2024-09-17T04:06:30.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.897+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02c015fd70 con 0x7f02c00af280 2024-09-17T04:06:30.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.897+0000 7f02b7e00640 1 -- 172.21.15.129:0/138630095 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f02a8087a40 con 0x7f02c00af280 2024-09-17T04:06:30.897 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:30.898 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:30.898 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 msgr2=0x7f029c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 0x7f029c07b0d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f02b0001720 tx=0x7f02b0009040 comp rx=0 tx=0).stop 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 msgr2=0x7f02c00b12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c00 tx=0x7f02a80038a0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 shutdown_connections 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f029c078bc0 0x7f029c07b0d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f02c01f6150 0x7f02c0151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f02c00af280 0x7f02c00b12e0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 --2- 172.21.15.129:0/138630095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f02c00ad8d0 0x7f02c01f95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 >> 172.21.15.129:0/138630095 conn(0x7f02c006cfc0 msgr2=0x7f02c006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 shutdown_connections 2024-09-17T04:06:30.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:30.900+0000 7f02c5000640 1 -- 172.21.15.129:0/138630095 wait complete. 2024-09-17T04:06:30.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:06:31.164 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":50}} 2024-09-17T04:06:31.164 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 50 } } 2024-09-17T04:06:31.164 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:06:31.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:06:31.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:06:31.348 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:06:31.348 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:06:31.349 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:31.349 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:31.385 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:31.385 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:31.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 -- 172.21.15.129:0/65976416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd07406eed0 msgr2=0x7fd0740ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 --2- 172.21.15.129:0/65976416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd07406eed0 0x7fd0740ccc80 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd060009830 tx=0x7fd06002a520 comp rx=0 tx=0).stop 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 -- 172.21.15.129:0/65976416 shutdown_connections 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 --2- 172.21.15.129:0/65976416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd07406eed0 0x7fd0740ccc80 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 --2- 172.21.15.129:0/65976416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0740705a0 0x7fd0740b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.721+0000 7fd07c200640 1 -- 172.21.15.129:0/65976416 >> 172.21.15.129:0/65976416 conn(0x7fd07406d410 msgr2=0x7fd0740b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.722+0000 7fd07c200640 1 -- 172.21.15.129:0/65976416 shutdown_connections 2024-09-17T04:06:31.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.722+0000 7fd07c200640 1 -- 172.21.15.129:0/65976416 wait complete. 2024-09-17T04:06:31.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.724+0000 7fd07c200640 1 Processor -- start 2024-09-17T04:06:31.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.724+0000 7fd07c200640 1 -- start start 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 0x7fd0740c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0740705a0 0x7fd0740c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd074160200 con 0x7fd07406eed0 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd0741612e0 con 0x7fd0740705a0 2024-09-17T04:06:31.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.725+0000 7fd07c200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd0740cd1c0 con 0x7fd07406f2e0 2024-09-17T04:06:31.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46104/0 (socket says 172.21.15.129:46104) 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 -- 172.21.15.129:0/2772244496 learned_addr learned my addr 172.21.15.129:0/2772244496 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd078e00640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 0x7fd0740c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd07a200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0740705a0 0x7fd0740c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 -- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 msgr2=0x7fd0740c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 0x7fd0740c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 -- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0740705a0 msgr2=0x7fd0740c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0740705a0 0x7fd0740c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 -- 172.21.15.129:0/2772244496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0741586c0 con 0x7fd07406eed0 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd078e00640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 0x7fd0740c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.726+0000 7fd079800640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9f0 tx=0x7fd06800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:31.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.727+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06800cd80 con 0x7fd07406eed0 2024-09-17T04:06:31.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.727+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd068004020 con 0x7fd07406eed0 2024-09-17T04:06:31.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.727+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd068004470 con 0x7fd07406eed0 2024-09-17T04:06:31.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.727+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd074156680 con 0x7fd07406eed0 2024-09-17T04:06:31.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.727+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd074158140 con 0x7fd07406eed0 2024-09-17T04:06:31.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.728+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0740ccc80 con 0x7fd07406eed0 2024-09-17T04:06:31.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.729+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0680035f0 con 0x7fd07406eed0 2024-09-17T04:06:31.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.731+0000 7fd071600640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 0x7fd05407b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:31.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.731+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7fd06809b7c0 con 0x7fd07406eed0 2024-09-17T04:06:31.731 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.731+0000 7fd078e00640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 0x7fd05407b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:31.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.732+0000 7fd078e00640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 0x7fd05407b100 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd060034000 tx=0x7fd060004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:31.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.735+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0680634f0 con 0x7fd07406eed0 2024-09-17T04:06:31.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.928+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd07406a140 con 0x7fd07406eed0 2024-09-17T04:06:31.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.929+0000 7fd071600640 1 -- 172.21.15.129:0/2772244496 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v235) ==== 80+0+0 (secure 0 0 0) 0x7fd068067f20 con 0x7fd07406eed0 2024-09-17T04:06:31.930 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:06:31.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 msgr2=0x7fd05407b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 0x7fd05407b100 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fd060034000 tx=0x7fd060004730 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 msgr2=0x7fd074157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7fd06800e9f0 tx=0x7fd06800eeb0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 shutdown_connections 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd054078bf0 0x7fd05407b100 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0740705a0 0x7fd0740c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd07406f2e0 0x7fd0740c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 --2- 172.21.15.129:0/2772244496 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd07406eed0 0x7fd074157c00 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 >> 172.21.15.129:0/2772244496 conn(0x7fd07406d410 msgr2=0x7fd0740ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.931+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 shutdown_connections 2024-09-17T04:06:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:31.932+0000 7fd07c200640 1 -- 172.21.15.129:0/2772244496 wait complete. 2024-09-17T04:06:31.956 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:06:31.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:31.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:06:32.086 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:06:32.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:06:32.087 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:06:32.123 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:06:32.123 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:06:32.123 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:32.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:06:32.159 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:06:32.159 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:32.159 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:32.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:06:32.198 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:06:32.198 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:06:32.198 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:32.237 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:06:32.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 -- 172.21.15.129:0/4123227574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f421c004800 msgr2=0x7f421c004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 --2- 172.21.15.129:0/4123227574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f421c004800 0x7f421c004c70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4220006f90 tx=0x7f421c005830 comp rx=0 tx=0).stop 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f4237200640 1 -- 172.21.15.129:0/4123227574 reap_dead start 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 -- 172.21.15.129:0/4123227574 shutdown_connections 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 -- 172.21.15.129:0/4123227574 >> 172.21.15.129:0/4123227574 conn(0x7f423806cfc0 msgr2=0x7f423806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 -- 172.21.15.129:0/4123227574 shutdown_connections 2024-09-17T04:06:32.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.460+0000 7f423ca00640 1 -- 172.21.15.129:0/4123227574 wait complete. 2024-09-17T04:06:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.463+0000 7f423ca00640 1 Processor -- start 2024-09-17T04:06:32.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.463+0000 7f423ca00640 1 -- start start 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 0x7f4238138ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f423815f7e0 0x7f42380d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f42380cef20 con 0x7f423815fbf0 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f42380ceda0 con 0x7f421c004800 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.464+0000 7f423ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4238160920 con 0x7f423815f7e0 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46118/0 (socket says 172.21.15.129:46118) 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 -- 172.21.15.129:0/957842256 learned_addr learned my addr 172.21.15.129:0/957842256 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f422de00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f423815f7e0 0x7f42380d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4236800640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 0x7f4238138ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 -- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f423815f7e0 msgr2=0x7f42380d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f423815f7e0 0x7f42380d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 -- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 msgr2=0x7f4238138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 0x7f4238138ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 -- 172.21.15.129:0/957842256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42381f8c40 con 0x7f423815fbf0 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4236800640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 0x7f4238138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.465+0000 7f4237200640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f423806dd70 tx=0x7f423000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:32.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.466+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230003d80 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.466+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4230004b20 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.466+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42381f8f20 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.466+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230004030 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.466+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42381f94e0 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.467+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4238160eb0 con 0x7f423815fbf0 2024-09-17T04:06:32.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.468+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f42300041d0 con 0x7f423815fbf0 2024-09-17T04:06:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.469+0000 7f422f400640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 0x7f421407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.469+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f423009bbd0 con 0x7f423815fbf0 2024-09-17T04:06:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.469+0000 7f4236800640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 0x7f421407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.470+0000 7f4236800640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 0x7f421407b0d0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4228006520 tx=0x7f4228006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:32.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.474+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4230063f50 con 0x7f423815fbf0 2024-09-17T04:06:32.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.694+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f423806a140 con 0x7f423815fbf0 2024-09-17T04:06:32.694 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.694+0000 7f422f400640 1 -- 172.21.15.129:0/957842256 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f4230063850 con 0x7f423815fbf0 2024-09-17T04:06:32.695 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:32.695 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:32.695 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:32.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 msgr2=0x7f421407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 0x7f421407b0d0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f4228006520 tx=0x7f4228006b10 comp rx=0 tx=0).stop 2024-09-17T04:06:32.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 msgr2=0x7f42380d07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7f423806dd70 tx=0x7f423000f790 comp rx=0 tx=0).stop 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 shutdown_connections 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4214078bc0 0x7f421407b0d0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f423815fbf0 0x7f42380d07a0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f423815f7e0 0x7f42380d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 --2- 172.21.15.129:0/957842256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f421c004800 0x7f4238138ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 >> 172.21.15.129:0/957842256 conn(0x7f423806cfc0 msgr2=0x7f42380ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 shutdown_connections 2024-09-17T04:06:32.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.697+0000 7f423ca00640 1 -- 172.21.15.129:0/957842256 wait complete. 2024-09-17T04:06:32.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 -- 172.21.15.129:0/1004485712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40af4c0 msgr2=0x7f8ef415ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 --2- 172.21.15.129:0/1004485712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40af4c0 0x7f8ef415ae00 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8ef406f050 tx=0x7f8ee802a500 comp rx=0 tx=0).stop 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 -- 172.21.15.129:0/1004485712 shutdown_connections 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 --2- 172.21.15.129:0/1004485712 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef415b340 0x7f8ef40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 --2- 172.21.15.129:0/1004485712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40af4c0 0x7f8ef415ae00 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 -- 172.21.15.129:0/1004485712 >> 172.21.15.129:0/1004485712 conn(0x7f8ef406d410 msgr2=0x7f8ef40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 -- 172.21.15.129:0/1004485712 shutdown_connections 2024-09-17T04:06:32.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.941+0000 7f8efc400640 1 -- 172.21.15.129:0/1004485712 wait complete. 2024-09-17T04:06:32.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.944+0000 7f8efc400640 1 Processor -- start 2024-09-17T04:06:32.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.944+0000 7f8efc400640 1 -- start start 2024-09-17T04:06:32.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef40af4c0 0x7f8ef4155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 0x7f8ef40b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef40710e0 con 0x7f8ef415b340 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef4070f60 con 0x7f8ef40cd1c0 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efc400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef415ed50 con 0x7f8ef40af4c0 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.945+0000 7f8efa400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46130/0 (socket says 172.21.15.129:46130) 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 -- 172.21.15.129:0/3448069355 learned_addr learned my addr 172.21.15.129:0/3448069355 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8ef9000640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 0x7f8ef40b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.946 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8ef9a00640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef40af4c0 0x7f8ef4155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 -- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef40af4c0 msgr2=0x7f8ef4155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef40af4c0 0x7f8ef4155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.947 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 -- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 msgr2=0x7f8ef40b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 0x7f8ef40b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 -- 172.21.15.129:0/3448069355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef40b67c0 con 0x7f8ef415b340 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8ef9000640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 0x7f8ef40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.946+0000 7f8efa400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c6f0 tx=0x7f8eec00cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.947+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec017070 con 0x7f8ef415b340 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.947+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8eec003c90 con 0x7f8ef415b340 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.947+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec003e60 con 0x7f8ef415b340 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.947+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef40b6aa0 con 0x7f8ef415b340 2024-09-17T04:06:32.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.947+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef40b7060 con 0x7f8ef415b340 2024-09-17T04:06:32.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.948+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef415f1d0 con 0x7f8ef415b340 2024-09-17T04:06:32.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.949+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8eec01a070 con 0x7f8ef415b340 2024-09-17T04:06:32.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.950+0000 7f8ef1600640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 0x7f8ed407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:32.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.950+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f8eec09a380 con 0x7f8ef415b340 2024-09-17T04:06:32.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.951+0000 7f8ef9a00640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 0x7f8ed407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:32.951 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.951+0000 7f8ef9a00640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 0x7f8ed407b0a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8034000 tx=0x7f8ee8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:32.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:32.955+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8ef40cd5d0 con 0x7f8ef415b340 2024-09-17T04:06:33.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.177+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ef406a140 con 0x7f8ef415b340 2024-09-17T04:06:33.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.178+0000 7f8ef1600640 1 -- 172.21.15.129:0/3448069355 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f8eec0620c0 con 0x7f8ef415b340 2024-09-17T04:06:33.178 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:33.178 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:33.178 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 msgr2=0x7f8ed407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 0x7f8ed407b0a0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8034000 tx=0x7f8ee8004730 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 msgr2=0x7f8ef40b15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00c6f0 tx=0x7f8eec00cbb0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 shutdown_connections 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8ed4078b90 0x7f8ed407b0a0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef415b340 0x7f8ef40b15b0 unknown :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8ef40cd1c0 0x7f8ef40b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 --2- 172.21.15.129:0/3448069355 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8ef40af4c0 0x7f8ef4155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 >> 172.21.15.129:0/3448069355 conn(0x7f8ef406d410 msgr2=0x7f8ef4073160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 shutdown_connections 2024-09-17T04:06:33.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.180+0000 7f8efc400640 1 -- 172.21.15.129:0/3448069355 wait complete. 2024-09-17T04:06:33.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:33.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 -- 172.21.15.129:0/3850383824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d0070100 msgr2=0x7f93d0070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 --2- 172.21.15.129:0/3850383824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d0070100 0x7f93d0070510 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f93d00ce800 tx=0x7f93c802a670 comp rx=0 tx=0).stop 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 -- 172.21.15.129:0/3850383824 shutdown_connections 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 --2- 172.21.15.129:0/3850383824 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d0070100 0x7f93d0070510 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 -- 172.21.15.129:0/3850383824 >> 172.21.15.129:0/3850383824 conn(0x7f93d006d410 msgr2=0x7f93d00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 -- 172.21.15.129:0/3850383824 shutdown_connections 2024-09-17T04:06:33.424 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.423+0000 7f93d5c00640 1 -- 172.21.15.129:0/3850383824 wait complete. 2024-09-17T04:06:33.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.425+0000 7f93d5c00640 1 Processor -- start 2024-09-17T04:06:33.425 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.425+0000 7f93d5c00640 1 -- start start 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d0070100 0x7f93d009da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 0x7f93d00d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f93d00b4a10 con 0x7f93d015da00 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f93d00b4890 con 0x7f93d0070100 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93d5c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f93d0070a50 con 0x7f93d00ccc80 2024-09-17T04:06:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93cf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93cf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46150/0 (socket says 172.21.15.129:46150) 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93cf400640 1 -- 172.21.15.129:0/1901903488 learned_addr learned my addr 172.21.15.129:0/1901903488 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93ce000640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 0x7f93d00d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.427+0000 7f93cea00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d0070100 0x7f93d009da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 -- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 msgr2=0x7f93d00d0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 0x7f93d00d0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 -- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d0070100 msgr2=0x7f93d009da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d0070100 0x7f93d009da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 -- 172.21.15.129:0/1901903488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d01f9fe0 con 0x7f93d015da00 2024-09-17T04:06:33.429 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93ce000640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 0x7f93d00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93cf400640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034040 tx=0x7f93c8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c803e030 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f93c8003e00 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8035040 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d01fa2c0 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.428+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d01fa880 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.430+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d0073a10 con 0x7f93d015da00 2024-09-17T04:06:33.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.431+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f93c8041030 con 0x7f93d015da00 2024-09-17T04:06:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.432+0000 7f93c6a00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 0x7f93b007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.432+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f93c80bb770 con 0x7f93d015da00 2024-09-17T04:06:33.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.432+0000 7f93cea00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 0x7f93b007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.433+0000 7f93cea00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 0x7f93b007b0a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f93c00012e0 tx=0x7f93c0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:33.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.436+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f93d0073a10 con 0x7f93d015da00 2024-09-17T04:06:33.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.622+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93d01f9d70 con 0x7f93d015da00 2024-09-17T04:06:33.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.624+0000 7f93c6a00640 1 -- 172.21.15.129:0/1901903488 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) ==== 74+0+17080 (secure 0 0 0) 0x7f93c8083a60 con 0x7f93d015da00 2024-09-17T04:06:33.624 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:33.625 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":235,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:20.014760+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:01.171609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:02.239670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"35.5","mappings":[{"from":4,"to":2}]},{"pgid":"35.28","mappings":[{"from":6,"to":2}]},{"pgid":"35.30","mappings":[{"from":6,"to":1}]},{"pgid":"35.32","mappings":[{"from":4,"to":1}]},{"pgid":"35.36","mappings":[{"from":4,"to":1}]},{"pgid":"35.3c","mappings":[{"from":4,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 msgr2=0x7f93b007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 0x7f93b007b0a0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f93c00012e0 tx=0x7f93c0009040 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 msgr2=0x7f93d01effc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f93c8034040 tx=0x7f93c8003910 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 shutdown_connections 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f93b0078b90 0x7f93b007b0a0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f93d015da00 0x7f93d01effc0 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f93d00ccc80 0x7f93d00d0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 --2- 172.21.15.129:0/1901903488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f93d0070100 0x7f93d009da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 >> 172.21.15.129:0/1901903488 conn(0x7f93d006d410 msgr2=0x7f93d00ada10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 shutdown_connections 2024-09-17T04:06:33.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.626+0000 7f93d5c00640 1 -- 172.21.15.129:0/1901903488 wait complete. 2024-09-17T04:06:33.649 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:06:33.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:33.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 -- 172.21.15.129:0/3081502007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000cd1c0 msgr2=0x7f4a001609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 --2- 172.21.15.129:0/3081502007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000cd1c0 0x7f4a001609c0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f4a00072080 tx=0x7f49f802aac0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 -- 172.21.15.129:0/3081502007 shutdown_connections 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 --2- 172.21.15.129:0/3081502007 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000cd1c0 0x7f4a001609c0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 --2- 172.21.15.129:0/3081502007 >> v2:172.21.15.181:3300/0 conn(0x7f4a000b4140 0x7f4a000ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 -- 172.21.15.129:0/3081502007 >> 172.21.15.129:0/3081502007 conn(0x7f4a0006d410 msgr2=0x7f4a000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 -- 172.21.15.129:0/3081502007 shutdown_connections 2024-09-17T04:06:33.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.868+0000 7f4a06800640 1 -- 172.21.15.129:0/3081502007 wait complete. 2024-09-17T04:06:33.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.871+0000 7f4a06800640 1 Processor -- start 2024-09-17T04:06:33.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.871+0000 7f4a06800640 1 -- start start 2024-09-17T04:06:33.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a000b4140 0x7f4a001f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a000cd1c0 0x7f4a0009bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00160260 con 0x7f4a000705a0 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4a001600e0 con 0x7f4a000cd1c0 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f4a06800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4a00160f00 con 0x7f4a000b4140 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f49ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f49ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37416/0 (socket says 172.21.15.129:37416) 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.872+0000 7f49ff400640 1 -- 172.21.15.129:0/768662533 learned_addr learned my addr 172.21.15.129:0/768662533 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49f6a00640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a000b4140 0x7f4a001f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ffe00640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a000cd1c0 0x7f4a0009bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 -- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a000b4140 msgr2=0x7f4a001f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a000b4140 0x7f4a001f5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 -- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a000cd1c0 msgr2=0x7f4a0009bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a000cd1c0 0x7f4a0009bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 -- 172.21.15.129:0/768662533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a000c9e90 con 0x7f4a000705a0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49ff400640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c3a0 tx=0x7f49f000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0017070 con 0x7f4a000705a0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49f00053c0 con 0x7f4a000705a0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f0003850 con 0x7f4a000705a0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.873+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a000ca170 con 0x7f4a000705a0 2024-09-17T04:06:33.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.874+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a001f56b0 con 0x7f4a000705a0 2024-09-17T04:06:33.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.874+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a000ccc80 con 0x7f4a000705a0 2024-09-17T04:06:33.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.876+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f49f001a070 con 0x7f4a000705a0 2024-09-17T04:06:33.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.877+0000 7f49f7e00640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 0x7f49dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:33.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.877+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f49f009a1d0 con 0x7f4a000705a0 2024-09-17T04:06:33.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.877+0000 7f49f6a00640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 0x7f49dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:33.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.878+0000 7f49f6a00640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 0x7f49dc07b0a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4a00073ca0 tx=0x7f49e8005e30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:33.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:33.881+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4a000cd5d0 con 0x7f4a000705a0 2024-09-17T04:06:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.100+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a0006a140 con 0x7f4a000705a0 2024-09-17T04:06:34.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.100+0000 7f49f7e00640 1 -- 172.21.15.129:0/768662533 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7f49f0062550 con 0x7f4a000705a0 2024-09-17T04:06:34.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:34.101 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:34.101 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 msgr2=0x7f49dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 0x7f49dc07b0a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f4a00073ca0 tx=0x7f49e8005e30 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 msgr2=0x7f4a001f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f49f000c3a0 tx=0x7f49f000c860 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 shutdown_connections 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49dc078b90 0x7f49dc07b0a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4a000cd1c0 0x7f4a0009bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4a000b4140 0x7f4a001f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 --2- 172.21.15.129:0/768662533 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4a000705a0 0x7f4a001f6510 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 >> 172.21.15.129:0/768662533 conn(0x7f4a0006d410 msgr2=0x7f4a000ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:34.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 shutdown_connections 2024-09-17T04:06:34.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.103+0000 7f4a06800640 1 -- 172.21.15.129:0/768662533 wait complete. 2024-09-17T04:06:34.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 -- 172.21.15.129:0/1903596950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc06eed0 msgr2=0x7fd0cc0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 --2- 172.21.15.129:0/1903596950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc06eed0 0x7fd0cc0ccc80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4009830 tx=0x7fd0b402a510 comp rx=0 tx=0).stop 2024-09-17T04:06:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 -- 172.21.15.129:0/1903596950 shutdown_connections 2024-09-17T04:06:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 --2- 172.21.15.129:0/1903596950 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc06eed0 0x7fd0cc0ccc80 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 --2- 172.21.15.129:0/1903596950 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc0705a0 0x7fd0cc0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 -- 172.21.15.129:0/1903596950 >> 172.21.15.129:0/1903596950 conn(0x7fd0cc06d410 msgr2=0x7fd0cc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 -- 172.21.15.129:0/1903596950 shutdown_connections 2024-09-17T04:06:34.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.353+0000 7fd0d2200640 1 -- 172.21.15.129:0/1903596950 wait complete. 2024-09-17T04:06:34.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.356+0000 7fd0d2200640 1 Processor -- start 2024-09-17T04:06:34.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.356+0000 7fd0d2200640 1 -- start start 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 0x7fd0cc0c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc0705a0 0x7fd0cc1f6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc160200 con 0x7fd0cc06f2e0 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc1612e0 con 0x7fd0cc0705a0 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.357+0000 7fd0d2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd0cc0cd1c0 con 0x7fd0cc06eed0 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0c3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37424/0 (socket says 172.21.15.129:37424) 2024-09-17T04:06:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0c3e00640 1 -- 172.21.15.129:0/478935915 learned_addr learned my addr 172.21.15.129:0/478935915 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 0x7fd0cc0c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cbe00640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc0705a0 0x7fd0cc1f6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 -- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc0705a0 msgr2=0x7fd0cc1f6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc0705a0 0x7fd0cc1f6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 -- 172.21.15.129:0/478935915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 msgr2=0x7fd0cc1f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0cb400640 1 -- 172.21.15.129:0/478935915 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc159ca0 con 0x7fd0cc06eed0 2024-09-17T04:06:34.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.358+0000 7fd0c3e00640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0cb400640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 0x7fd0cc0c5dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc005fd0 tx=0x7fd0bc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc004220 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd0bc0043c0 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc15e720 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc00cc20 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.359+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc15a220 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.360+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc0c7550 con 0x7fd0cc06eed0 2024-09-17T04:06:34.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.362+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd0bc0035f0 con 0x7fd0cc06eed0 2024-09-17T04:06:34.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.363+0000 7fd0c3400640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 0x7fd0a807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.363+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7fd0bc09b720 con 0x7fd0cc06eed0 2024-09-17T04:06:34.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.363+0000 7fd0c3e00640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 0x7fd0a807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.364+0000 7fd0c3e00640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 0x7fd0a807b0a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0d0e60 tx=0x7fd0b4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:34.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.366+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd0bc009510 con 0x7fd0cc06eed0 2024-09-17T04:06:34.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.584+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0cc0ccc80 con 0x7fd0cc06eed0 2024-09-17T04:06:34.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.586+0000 7fd0c3400640 1 -- 172.21.15.129:0/478935915 <== mon.2 v2:172.21.15.181:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+5251 (secure 0 0 0) 0x7fd0bc063560 con 0x7fd0cc06eed0 2024-09-17T04:06:34.586 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:34.586 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":227,"btime":"2024-09-17T04:06:06:993362+0000","default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[{"mdsmap":{"epoch":227,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:03.281568+0000","modified":"2024-09-17T04:06:06.475395+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11460":{"gid":11460,"name":"b","rank":0,"incarnation":224,"state":"up:active","state_seq":3,"addr":"172.21.15.181:6836/4204505060","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4204505060},{"type":"v1","addr":"172.21.15.181:6836","nonce":4204505060}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11460,"qdb_cluster":[11460]},"id":17}]} 2024-09-17T04:06:34.586 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 227 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 msgr2=0x7fd0a807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 0x7fd0a807b0a0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0d0e60 tx=0x7fd0b4004730 comp rx=0 tx=0).stop 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 msgr2=0x7fd0cc0c5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 0x7fd0cc0c5dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc005fd0 tx=0x7fd0bc00ee50 comp rx=0 tx=0).stop 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 shutdown_connections 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd0a8078b90 0x7fd0a807b0a0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd0cc0705a0 0x7fd0cc1f6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd0cc06f2e0 0x7fd0cc1f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.589 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 --2- 172.21.15.129:0/478935915 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd0cc06eed0 0x7fd0cc0c5dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.588+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 >> 172.21.15.129:0/478935915 conn(0x7fd0cc06d410 msgr2=0x7fd0cc072e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:34.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.589+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 shutdown_connections 2024-09-17T04:06:34.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.589+0000 7fd0d2200640 1 -- 172.21.15.129:0/478935915 wait complete. 2024-09-17T04:06:34.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:34.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 -- 172.21.15.129:0/4054985793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3198004d40 msgr2=0x7f3198005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 --2- 172.21.15.129:0/4054985793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3198004d40 0x7f3198005150 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f31ac00c730 tx=0x7f3198005830 comp rx=0 tx=0).stop 2024-09-17T04:06:34.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b3e00640 1 -- 172.21.15.129:0/4054985793 reap_dead start 2024-09-17T04:06:34.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 -- 172.21.15.129:0/4054985793 shutdown_connections 2024-09-17T04:06:34.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 -- 172.21.15.129:0/4054985793 >> 172.21.15.129:0/4054985793 conn(0x7f31b409d8d0 msgr2=0x7f31b408bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:34.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 -- 172.21.15.129:0/4054985793 shutdown_connections 2024-09-17T04:06:34.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.840+0000 7f31b8e00640 1 -- 172.21.15.129:0/4054985793 wait complete. 2024-09-17T04:06:34.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.842+0000 7f31b8e00640 1 Processor -- start 2024-09-17T04:06:34.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.842+0000 7f31b8e00640 1 -- start start 2024-09-17T04:06:34.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.843+0000 7f31b8e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3198004d40 0x7f31b41f91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b8e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31b415b680 0x7f31b414a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b8e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31b40ba040 con 0x7f31b41f9720 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b8e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f31b40b9ec0 con 0x7f3198004d40 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b8e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f31b40d0ae0 con 0x7f31b415b680 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37430/0 (socket says 172.21.15.129:37430) 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 -- 172.21.15.129:0/2768100899 learned_addr learned my addr 172.21.15.129:0/2768100899 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b3400640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3198004d40 0x7f31b41f91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31b3e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31b415b680 0x7f31b414a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 -- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31b415b680 msgr2=0x7f31b414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31b415b680 0x7f31b414a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 -- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3198004d40 msgr2=0x7f31b41f91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:34.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3198004d40 0x7f31b41f91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:34.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.844+0000 7f31abe00640 1 -- 172.21.15.129:0/2768100899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b41f9b30 con 0x7f31b41f9720 2024-09-17T04:06:34.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31abe00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f319c011b10 tx=0x7f319c011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:34.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c00fc20 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f319c00fdc0 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b415bf70 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319c004280 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.845+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b415c530 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.846+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31b40d5700 con 0x7f31b41f9720 2024-09-17T04:06:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.847+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f319c0035f0 con 0x7f31b41f9720 2024-09-17T04:06:34.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.848+0000 7f31ab400640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 0x7f319007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:34.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.848+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7f319c09b820 con 0x7f31b41f9720 2024-09-17T04:06:34.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.849+0000 7f31b3400640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 0x7f319007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:34.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.850+0000 7f31b3400640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 0x7f319007b0d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009c40 tx=0x7f31a4005f30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:34.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:34.853+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f31b40d5700 con 0x7f31b41f9720 2024-09-17T04:06:35.036 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.036+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31b40d4620 con 0x7f31b41f9720 2024-09-17T04:06:35.037 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.037+0000 7f31ab400640 1 -- 172.21.15.129:0/2768100899 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) ==== 74+0+17080 (secure 0 0 0) 0x7f31b41f9b30 con 0x7f31b41f9720 2024-09-17T04:06:35.037 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:35.038 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":235,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:20.014760+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:01.171609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:02.239670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"35.5","mappings":[{"from":4,"to":2}]},{"pgid":"35.28","mappings":[{"from":6,"to":2}]},{"pgid":"35.30","mappings":[{"from":6,"to":1}]},{"pgid":"35.32","mappings":[{"from":4,"to":1}]},{"pgid":"35.36","mappings":[{"from":4,"to":1}]},{"pgid":"35.3c","mappings":[{"from":4,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 msgr2=0x7f319007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 0x7f319007b0d0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009c40 tx=0x7f31a4005f30 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 msgr2=0x7f31b415b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f319c011b10 tx=0x7f319c011f10 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 shutdown_connections 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3190078bc0 0x7f319007b0d0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31b415b680 0x7f31b414a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31b41f9720 0x7f31b415b140 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 --2- 172.21.15.129:0/2768100899 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3198004d40 0x7f31b41f91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.039+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 >> 172.21.15.129:0/2768100899 conn(0x7f31b409d8d0 msgr2=0x7f31b40b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:35.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.040+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 shutdown_connections 2024-09-17T04:06:35.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.040+0000 7f31b8e00640 1 -- 172.21.15.129:0/2768100899 wait complete. 2024-09-17T04:06:35.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:06:35.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 -- 172.21.15.129:0/1740104353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0af4c0 msgr2=0x7fee5c15ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 --2- 172.21.15.129:0/1740104353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0af4c0 0x7fee5c15ae00 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009830 tx=0x7fee4c02a520 comp rx=0 tx=0).stop 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 -- 172.21.15.129:0/1740104353 shutdown_connections 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 --2- 172.21.15.129:0/1740104353 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0cd390 0x7fee5c06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 --2- 172.21.15.129:0/1740104353 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0af4c0 0x7fee5c15ae00 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 -- 172.21.15.129:0/1740104353 >> 172.21.15.129:0/1740104353 conn(0x7fee5c06d410 msgr2=0x7fee5c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 -- 172.21.15.129:0/1740104353 shutdown_connections 2024-09-17T04:06:35.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.290+0000 7fee62e00640 1 -- 172.21.15.129:0/1740104353 wait complete. 2024-09-17T04:06:35.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.292+0000 7fee62e00640 1 Processor -- start 2024-09-17T04:06:35.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.292+0000 7fee62e00640 1 -- start start 2024-09-17T04:06:35.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0af4c0 0x7fee5c09d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0b1180 0x7fee5c0cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c06fb80 con 0x7fee5c15b340 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c06fd00 con 0x7fee5c0b1180 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee62e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fee5c06fe80 con 0x7fee5c0af4c0 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee5b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee5b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37444/0 (socket says 172.21.15.129:37444) 2024-09-17T04:06:35.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee5b400640 1 -- 172.21.15.129:0/1552781708 learned_addr learned my addr 172.21.15.129:0/1552781708 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee60e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0b1180 0x7fee5c0cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.294+0000 7fee5be00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0af4c0 0x7fee5c09d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 -- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0af4c0 msgr2=0x7fee5c09d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0af4c0 0x7fee5c09d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 -- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0b1180 msgr2=0x7fee5c0cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0b1180 0x7fee5c0cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 -- 172.21.15.129:0/1552781708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c1f3900 con 0x7fee5c15b340 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee5b400640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c390 tx=0x7fee4800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48017070 con 0x7fee5c15b340 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee480053e0 con 0x7fee5c15b340 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5c1f3be0 con 0x7fee5c15b340 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee48003850 con 0x7fee5c15b340 2024-09-17T04:06:35.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.295+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee5c1f41a0 con 0x7fee5c15b340 2024-09-17T04:06:35.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.296+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee5c070300 con 0x7fee5c15b340 2024-09-17T04:06:35.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.298+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fee4801a070 con 0x7fee5c15b340 2024-09-17T04:06:35.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.299+0000 7fee53e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 0x7fee3c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.299+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(235..235 src has 1..235) ==== 5883+0+0 (secure 0 0 0) 0x7fee4809a0b0 con 0x7fee5c15b340 2024-09-17T04:06:35.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.300+0000 7fee5be00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 0x7fee3c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.300+0000 7fee5be00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 0x7fee3c07b0a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fee4c034000 tx=0x7fee4c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:35.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.304+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fee48061de0 con 0x7fee5c15b340 2024-09-17T04:06:35.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.522+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fee5c06a140 con 0x7fee5c15b340 2024-09-17T04:06:35.523 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:06:35.523+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:06:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.527+0000 7fee53e00640 1 -- 172.21.15.129:0/1552781708 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v228) ==== 161+0+0 (secure 0 0 0) 0x7fee48066810 con 0x7fee5c15b340 2024-09-17T04:06:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:06:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 msgr2=0x7fee3c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 0x7fee3c07b0a0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fee4c034000 tx=0x7fee4c004730 comp rx=0 tx=0).stop 2024-09-17T04:06:35.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 msgr2=0x7fee5c0b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c390 tx=0x7fee4800c850 comp rx=0 tx=0).stop 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 shutdown_connections 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fee3c078b90 0x7fee3c07b0a0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee5c0b1180 0x7fee5c0cbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fee5c15b340 0x7fee5c0b0c40 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 --2- 172.21.15.129:0/1552781708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fee5c0af4c0 0x7fee5c09d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 >> 172.21.15.129:0/1552781708 conn(0x7fee5c06d410 msgr2=0x7fee5c15a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 shutdown_connections 2024-09-17T04:06:35.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.529+0000 7fee62e00640 1 -- 172.21.15.129:0/1552781708 wait complete. 2024-09-17T04:06:35.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:06:35.669 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 -- 172.21.15.129:0/2453535629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 msgr2=0x7f8b880b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/2453535629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b880b2050 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f8b880d1260 tx=0x7f8b7402a4b0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 -- 172.21.15.129:0/2453535629 shutdown_connections 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/2453535629 >> v2:172.21.15.181:3300/0 conn(0x7f8b880b3010 0x7f8b880b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/2453535629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b880b2050 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.776+0000 7f8b8cc00640 1 -- 172.21.15.129:0/2453535629 >> 172.21.15.129:0/2453535629 conn(0x7f8b880ccc40 msgr2=0x7f8b880cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:35.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.777+0000 7f8b8cc00640 1 -- 172.21.15.129:0/2453535629 shutdown_connections 2024-09-17T04:06:35.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.777+0000 7f8b8cc00640 1 -- 172.21.15.129:0/2453535629 wait complete. 2024-09-17T04:06:35.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.779+0000 7f8b8cc00640 1 Processor -- start 2024-09-17T04:06:35.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.779+0000 7f8b8cc00640 1 -- start start 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 0x7f8b88149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b880b3010 0x7f8b881e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880b3ae0 con 0x7f8b880b1c40 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880b3960 con 0x7f8b88074800 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.780+0000 7f8b8cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8b880b3c60 con 0x7f8b880b3010 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37452/0 (socket says 172.21.15.129:37452) 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 -- 172.21.15.129:0/4153032763 learned_addr learned my addr 172.21.15.129:0/4153032763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85e00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 0x7f8b88149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b86800640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b880b3010 0x7f8b881e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 -- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b880b3010 msgr2=0x7f8b881e9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b880b3010 0x7f8b881e9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 -- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 msgr2=0x7f8b88149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 0x7f8b88149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 -- 172.21.15.129:0/4153032763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8814d720 con 0x7f8b880b1c40 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85e00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 0x7f8b88149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.781+0000 7f8b85400640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009920 tx=0x7f8b74033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.782+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74031d60 con 0x7f8b880b1c40 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.782+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b74033e10 con 0x7f8b880b1c40 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.782+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74005260 con 0x7f8b880b1c40 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.782+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b880ce0c0 con 0x7f8b880b1c40 2024-09-17T04:06:35.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.782+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b880ce680 con 0x7f8b880b1c40 2024-09-17T04:06:35.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.783+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b880cde20 con 0x7f8b880b1c40 2024-09-17T04:06:35.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.784+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b74005400 con 0x7f8b880b1c40 2024-09-17T04:06:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.785+0000 7f8b7e000640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 0x7f8b6807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.785+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 5969+0+0 (secure 0 0 0) 0x7f8b740bc0b0 con 0x7f8b880b1c40 2024-09-17T04:06:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.785+0000 7f8b85e00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 0x7f8b6807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:35.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.786+0000 7f8b85e00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 0x7f8b6807b0d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8b780016a0 tx=0x7f8b78009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:35.789+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8b880cde20 con 0x7f8b880b1c40 2024-09-17T04:06:36.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.011+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8b880b4180 con 0x7f8b880b1c40 2024-09-17T04:06:36.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.530+0000 7f8b7e000640 1 -- 172.21.15.129:0/4153032763 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v229) ==== 104+0+0 (secure 0 0 0) 0x7f8b8814d720 con 0x7f8b880b1c40 2024-09-17T04:06:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 msgr2=0x7f8b6807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 0x7f8b6807b0d0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f8b780016a0 tx=0x7f8b78009040 comp rx=0 tx=0).stop 2024-09-17T04:06:36.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 msgr2=0x7f8b8814d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009920 tx=0x7f8b74033c20 comp rx=0 tx=0).stop 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 shutdown_connections 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b68078bc0 0x7f8b6807b0d0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b880b3010 0x7f8b881e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b880b1c40 0x7f8b8814d1e0 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 --2- 172.21.15.129:0/4153032763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b88074800 0x7f8b88149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 >> 172.21.15.129:0/4153032763 conn(0x7f8b880ccc40 msgr2=0x7f8b880cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 shutdown_connections 2024-09-17T04:06:36.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.534+0000 7f8b8cc00640 1 -- 172.21.15.129:0/4153032763 wait complete. 2024-09-17T04:06:36.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:06:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 -- 172.21.15.129:0/3223801181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300cbf10 msgr2=0x7f0e300b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 --2- 172.21.15.129:0/3223801181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300cbf10 0x7f0e300b1c40 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f0e18005fd0 tx=0x7f0e1802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 -- 172.21.15.129:0/3223801181 shutdown_connections 2024-09-17T04:06:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 --2- 172.21.15.129:0/3223801181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300cbf10 0x7f0e300b1c40 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 -- 172.21.15.129:0/3223801181 >> 172.21.15.129:0/3223801181 conn(0x7f0e300ccc40 msgr2=0x7f0e300cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:36.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 -- 172.21.15.129:0/3223801181 shutdown_connections 2024-09-17T04:06:36.784 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.783+0000 7f0e35000640 1 -- 172.21.15.129:0/3223801181 wait complete. 2024-09-17T04:06:36.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.785+0000 7f0e35000640 1 Processor -- start 2024-09-17T04:06:36.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.785+0000 7f0e35000640 1 -- start start 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 0x7f0e300c5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 0x7f0e300c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300cb0a0 con 0x7f0e300705b0 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0e300caf20 con 0x7f0e300cbf10 2024-09-17T04:06:36.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.786+0000 7f0e35000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0e300cb220 con 0x7f0e300b2180 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37466/0 (socket says 172.21.15.129:37466) 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 -- 172.21.15.129:0/2088531952 learned_addr learned my addr 172.21.15.129:0/2088531952 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 -- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 msgr2=0x7f0e300c5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e25400640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 0x7f0e300c5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 0x7f0e300c5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 -- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 msgr2=0x7f0e300c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2e800640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 0x7f0e300c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 0x7f0e300c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2de00640 1 -- 172.21.15.129:0/2088531952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e300c8110 con 0x7f0e300705b0 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e25400640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 0x7f0e300c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:36.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.787+0000 7f0e2e800640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 0x7f0e300c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e2de00640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000c400 tx=0x7f0e2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20017070 con 0x7f0e300705b0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e200053c0 con 0x7f0e300705b0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e300c83f0 con 0x7f0e300705b0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e20003850 con 0x7f0e300705b0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.788+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e300c8e00 con 0x7f0e300705b0 2024-09-17T04:06:36.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.789+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e300cc9b0 con 0x7f0e300705b0 2024-09-17T04:06:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.790+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0e2001a070 con 0x7f0e300705b0 2024-09-17T04:06:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.791+0000 7f0e26a00640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 0x7f0e0c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:36.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.792+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(236..236 src has 1..236) ==== 5969+0+0 (secure 0 0 0) 0x7f0e2009a1f0 con 0x7f0e300705b0 2024-09-17T04:06:36.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.792+0000 7f0e25400640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 0x7f0e0c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:36.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.793+0000 7f0e25400640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 0x7f0e0c07b0a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0e18033000 tx=0x7f0e18004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:36.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.796+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0e20062510 con 0x7f0e300705b0 2024-09-17T04:06:36.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:36.978+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0e300c88d0 con 0x7f0e300705b0 2024-09-17T04:06:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.539+0000 7f0e26a00640 1 -- 172.21.15.129:0/2088531952 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v237) ==== 181+0+0 (secure 0 0 0) 0x7f0e20061e10 con 0x7f0e300705b0 2024-09-17T04:06:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 msgr2=0x7f0e0c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 0x7f0e0c07b0a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0e18033000 tx=0x7f0e18004730 comp rx=0 tx=0).stop 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 msgr2=0x7f0e3015b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000c400 tx=0x7f0e2000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 shutdown_connections 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e0c078b90 0x7f0e0c07b0a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e300cbf10 0x7f0e300c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e300b2180 0x7f0e300c5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 --2- 172.21.15.129:0/2088531952 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e300705b0 0x7f0e3015b910 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 >> 172.21.15.129:0/2088531952 conn(0x7f0e300ccc40 msgr2=0x7f0e300b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:37.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 shutdown_connections 2024-09-17T04:06:37.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.542+0000 7f0e35000640 1 -- 172.21.15.129:0/2088531952 wait complete. 2024-09-17T04:06:37.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 -- 172.21.15.129:0/2514241204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c0cba90 msgr2=0x7f8c0c0cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 --2- 172.21.15.129:0/2514241204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c0cba90 0x7f8c0c0cbea0 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0d1260 tx=0x7f8c0402a9a0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 -- 172.21.15.129:0/2514241204 shutdown_connections 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 --2- 172.21.15.129:0/2514241204 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c0cba90 0x7f8c0c0cbea0 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 --2- 172.21.15.129:0/2514241204 >> v2:172.21.15.181:3300/0 conn(0x7f8c0c0b3f70 0x7f8c0c0b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 -- 172.21.15.129:0/2514241204 >> 172.21.15.129:0/2514241204 conn(0x7f8c0c0ccc40 msgr2=0x7f8c0c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 -- 172.21.15.129:0/2514241204 shutdown_connections 2024-09-17T04:06:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.799+0000 7f8c13200640 1 -- 172.21.15.129:0/2514241204 wait complete. 2024-09-17T04:06:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.801+0000 7f8c13200640 1 Processor -- start 2024-09-17T04:06:37.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.801+0000 7f8c13200640 1 -- start start 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c0c0cba90 0x7f8c0c0c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 0x7f8c0c0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c0cc8e0 con 0x7f8c0c074580 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c0cc760 con 0x7f8c0c157870 2024-09-17T04:06:37.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c13200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8c0c166f20 con 0x7f8c0c0cba90 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c0be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37486/0 (socket says 172.21.15.129:37486) 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c0be00640 1 -- 172.21.15.129:0/1054020050 learned_addr learned my addr 172.21.15.129:0/1054020050 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c11200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 0x7f8c0c0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.803+0000 7f8c0b400640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c0c0cba90 0x7f8c0c0c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 -- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c0c0cba90 msgr2=0x7f8c0c0c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c0c0cba90 0x7f8c0c0c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 -- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 msgr2=0x7f8c0c0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:37.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 0x7f8c0c0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 -- 172.21.15.129:0/1054020050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c1f91c0 con 0x7f8c0c074580 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c11200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 0x7f8c0c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c0be00640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00d6d0 tx=0x7f8bfc00db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc017070 con 0x7f8c0c074580 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8bfc005330 con 0x7f8c0c074580 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0c1f94a0 con 0x7f8c0c074580 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc009820 con 0x7f8c0c074580 2024-09-17T04:06:37.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.804+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0c15a940 con 0x7f8c0c074580 2024-09-17T04:06:37.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.805+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c0c0b3f70 con 0x7f8c0c074580 2024-09-17T04:06:37.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.807+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8bfc009bd0 con 0x7f8c0c074580 2024-09-17T04:06:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.808+0000 7f8c03e00640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.808+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(237..237 src has 1..237) ==== 5531+0+0 (secure 0 0 0) 0x7f8bfc09afa0 con 0x7f8c0c074580 2024-09-17T04:06:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.808+0000 7f8c0b400640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:37.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.809+0000 7f8c0b400640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8006fd0 tx=0x7f8bf8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:37.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.812+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8bfc063480 con 0x7f8c0c074580 2024-09-17T04:06:37.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:37.994+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8c0c157320 con 0x7f8c0c074580 2024-09-17T04:06:38.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.541+0000 7f8c03e00640 1 -- 172.21.15.129:0/1054020050 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v238) ==== 169+0+0 (secure 0 0 0) 0x7f8bfc062d80 con 0x7f8c0c074580 2024-09-17T04:06:38.541 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.545+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 msgr2=0x7f8bec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.545+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8006fd0 tx=0x7f8bf8008040 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.545+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 msgr2=0x7f8c0c156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.545+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00d6d0 tx=0x7f8bfc00db90 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 shutdown_connections 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8bec078bc0 0x7f8bec07b0d0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8c0c157870 0x7f8c0c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8c0c0cba90 0x7f8c0c0c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 --2- 172.21.15.129:0/1054020050 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8c0c074580 0x7f8c0c156a30 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 >> 172.21.15.129:0/1054020050 conn(0x7f8c0c0ccc40 msgr2=0x7f8c0c0b04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:38.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.546+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 shutdown_connections 2024-09-17T04:06:38.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.547+0000 7f8c13200640 1 -- 172.21.15.129:0/1054020050 wait complete. 2024-09-17T04:06:38.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.799+0000 7fabe3000640 1 -- 172.21.15.129:0/1007145954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 msgr2=0x7fabdc0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.799+0000 7fabe3000640 1 --2- 172.21.15.129:0/1007145954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc0ccc80 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fabdc072e00 tx=0x7fabc802a530 comp rx=0 tx=0).stop 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 -- 172.21.15.129:0/1007145954 shutdown_connections 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 --2- 172.21.15.129:0/1007145954 >> v2:172.21.15.181:3300/0 conn(0x7fabdc0cd1c0 0x7fabdc1609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 --2- 172.21.15.129:0/1007145954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc0ccc80 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 -- 172.21.15.129:0/1007145954 >> 172.21.15.129:0/1007145954 conn(0x7fabdc06d410 msgr2=0x7fabdc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 -- 172.21.15.129:0/1007145954 shutdown_connections 2024-09-17T04:06:38.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.800+0000 7fabe3000640 1 -- 172.21.15.129:0/1007145954 wait complete. 2024-09-17T04:06:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.802+0000 7fabe3000640 1 Processor -- start 2024-09-17T04:06:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.802+0000 7fabe3000640 1 -- start start 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabdc0705a0 0x7fabdc09d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabdc0cd1c0 0x7fabdc1f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc160260 con 0x7fabdc0b4140 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fabdc1600e0 con 0x7fabdc0cd1c0 2024-09-17T04:06:38.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.803+0000 7fabe3000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fabdc160f00 con 0x7fabdc0705a0 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37502/0 (socket says 172.21.15.129:37502) 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 -- 172.21.15.129:0/1675803602 learned_addr learned my addr 172.21.15.129:0/1675803602 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabe1000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabdc0cd1c0 0x7fabdc1f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdbe00640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabdc0705a0 0x7fabdc09d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 -- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabdc0705a0 msgr2=0x7fabdc09d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabdc0705a0 0x7fabdc09d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 -- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabdc0cd1c0 msgr2=0x7fabdc1f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabdc0cd1c0 0x7fabdc1f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:38.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.804+0000 7fabdb400640 1 -- 172.21.15.129:0/1675803602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabdc1f44c0 con 0x7fabdc0b4140 2024-09-17T04:06:38.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabdb400640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fabc80098f0 tx=0x7fabc8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:38.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8031d60 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabc8033e10 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabdc1f6ed0 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc8005260 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.805+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabdc09da50 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.806+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabdc1f4230 con 0x7fabdc0b4140 2024-09-17T04:06:38.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.807+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fabc8005400 con 0x7fabdc0b4140 2024-09-17T04:06:38.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.808+0000 7fabd3e00640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 0x7fabbc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:38.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.808+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7fabc80bbb40 con 0x7fabdc0b4140 2024-09-17T04:06:38.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.809+0000 7fabdbe00640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 0x7fabbc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:38.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.809+0000 7fabdbe00640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 0x7fabbc07b0a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fabcc000f60 tx=0x7fabcc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:38.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:38.813+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fabc803f030 con 0x7fabdc0b4140 2024-09-17T04:06:39.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.006+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7fabdc0ccc80 con 0x7fabdc0b4140 2024-09-17T04:06:39.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.011+0000 7fabd3e00640 1 -- 172.21.15.129:0/1675803602 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v68) ==== 71+0+0 (secure 0 0 0) 0x7fabc8005d00 con 0x7fabdc0b4140 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 msgr2=0x7fabbc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 0x7fabbc07b0a0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fabcc000f60 tx=0x7fabcc009040 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 msgr2=0x7fabdc1f3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7fabc80098f0 tx=0x7fabc8033c20 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 shutdown_connections 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fabbc078b90 0x7fabbc07b0a0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabdc0cd1c0 0x7fabdc1f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabdc0b4140 0x7fabdc1f3730 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 --2- 172.21.15.129:0/1675803602 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabdc0705a0 0x7fabdc09d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 >> 172.21.15.129:0/1675803602 conn(0x7fabdc06d410 msgr2=0x7fabdc0ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.013+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 shutdown_connections 2024-09-17T04:06:39.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.014+0000 7fabe3000640 1 -- 172.21.15.129:0/1675803602 wait complete. 2024-09-17T04:06:39.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit' 2024-09-17T04:06:39.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 -- 172.21.15.129:0/4225249407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700cd390 msgr2=0x7ffb700af7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 --2- 172.21.15.129:0/4225249407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700cd390 0x7ffb700af7a0 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7ffb700ca850 tx=0x7ffb5802a540 comp rx=0 tx=0).stop 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 -- 172.21.15.129:0/4225249407 shutdown_connections 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 --2- 172.21.15.129:0/4225249407 >> v2:172.21.15.181:3300/0 conn(0x7ffb700b4180 0x7ffb700ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 --2- 172.21.15.129:0/4225249407 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700cd390 0x7ffb700af7a0 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 -- 172.21.15.129:0/4225249407 >> 172.21.15.129:0/4225249407 conn(0x7ffb7006d410 msgr2=0x7ffb700b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 -- 172.21.15.129:0/4225249407 shutdown_connections 2024-09-17T04:06:39.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.258+0000 7ffb77000640 1 -- 172.21.15.129:0/4225249407 wait complete. 2024-09-17T04:06:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.260+0000 7ffb77000640 1 Processor -- start 2024-09-17T04:06:39.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.260+0000 7ffb77000640 1 -- start start 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb7015ce50 0x7ffb7009dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffb701f62d0 0x7ffb700ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7006f820 con 0x7ffb700b4180 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffb7006f6a0 con 0x7ffb7015ce50 2024-09-17T04:06:39.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb77000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ffb7006f9a0 con 0x7ffb701f62d0 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37524/0 (socket says 172.21.15.129:37524) 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 -- 172.21.15.129:0/2328638085 learned_addr learned my addr 172.21.15.129:0/2328638085 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb67400640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb7015ce50 0x7ffb7009dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb75000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffb701f62d0 0x7ffb700ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 -- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffb701f62d0 msgr2=0x7ffb700ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffb701f62d0 0x7ffb700ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 -- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb7015ce50 msgr2=0x7ffb7009dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb7015ce50 0x7ffb7009dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.262+0000 7ffb6fe00640 1 -- 172.21.15.129:0/2328638085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb7009e100 con 0x7ffb700b4180 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb6fe00640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c3d0 tx=0x7ffb6000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011790 con 0x7ffb700b4180 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffb60017070 con 0x7ffb700b4180 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb60011cc0 con 0x7ffb700b4180 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb701f5b10 con 0x7ffb700b4180 2024-09-17T04:06:39.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.263+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb700adec0 con 0x7ffb700b4180 2024-09-17T04:06:39.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.264+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb701f5900 con 0x7ffb700b4180 2024-09-17T04:06:39.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.265+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffb6001c460 con 0x7ffb700b4180 2024-09-17T04:06:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.266+0000 7ffb6cc00640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 0x7ffb4c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.266+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7ffb6009a6a0 con 0x7ffb700b4180 2024-09-17T04:06:39.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.267+0000 7ffb67400640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 0x7ffb4c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.267+0000 7ffb67400640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 0x7ffb4c07b0a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ffb58034000 tx=0x7ffb58004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:39.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.271+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ffb600626f0 con 0x7ffb700b4180 2024-09-17T04:06:39.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.520+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit"]} v 0) -- 0x7ffb7006fe80 con 0x7ffb700b4180 2024-09-17T04:06:39.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.552+0000 7ffb6cc00640 1 -- 172.21.15.129:0/2328638085 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_limit"]}]=0 v863) ==== 139+0+0 (secure 0 0 0) 0x7ffb60067120 con 0x7ffb700b4180 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 msgr2=0x7ffb4c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 0x7ffb4c07b0a0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7ffb58034000 tx=0x7ffb58004730 comp rx=0 tx=0).stop 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 msgr2=0x7ffb70099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7ffb6000c3d0 tx=0x7ffb6000c890 comp rx=0 tx=0).stop 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 shutdown_connections 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffb4c078b90 0x7ffb4c07b0a0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffb701f62d0 0x7ffb700ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb7015ce50 0x7ffb7009dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 --2- 172.21.15.129:0/2328638085 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffb700b4180 0x7ffb70099ab0 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 >> 172.21.15.129:0/2328638085 conn(0x7ffb7006d410 msgr2=0x7ffb700b2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:39.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 shutdown_connections 2024-09-17T04:06:39.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.555+0000 7ffb77000640 1 -- 172.21.15.129:0/2328638085 wait complete. 2024-09-17T04:06:39.582 INFO:tasks.cephfs_test_runner:That the reintegration is not blocked by full directories. ... ok 2024-09-17T04:06:39.582 INFO:tasks.cephfs_test_runner:Starting test: test_reintegration_via_scrub (tasks.cephfs.test_strays.TestStrays) That reintegration is triggered via recursive scrub. 2024-09-17T04:06:39.582 INFO:tasks.cephfs_test_runner:test_reintegration_via_scrub (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:06:39.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.804+0000 7fa7ae400640 1 -- 172.21.15.129:0/388131738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c005250 msgr2=0x7fa78c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.804+0000 7fa7ae400640 1 --2- 172.21.15.129:0/388131738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c005250 0x7fa78c00b9c0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0005fd0 tx=0x7fa78c005830 comp rx=0 tx=0).stop 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.804+0000 7fa7ae400640 1 -- 172.21.15.129:0/388131738 shutdown_connections 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.804+0000 7fa7ae400640 1 --2- 172.21.15.129:0/388131738 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa78c005250 0x7fa78c00b9c0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.804+0000 7fa7ae400640 1 -- 172.21.15.129:0/388131738 >> 172.21.15.129:0/388131738 conn(0x7fa7a806d410 msgr2=0x7fa7a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:39.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.805+0000 7fa7ae400640 1 -- 172.21.15.129:0/388131738 shutdown_connections 2024-09-17T04:06:39.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.806+0000 7fa7ae400640 1 -- 172.21.15.129:0/388131738 wait complete. 2024-09-17T04:06:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.808+0000 7fa7ae400640 1 Processor -- start 2024-09-17T04:06:39.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.808+0000 7fa7ae400640 1 -- start start 2024-09-17T04:06:39.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa78c005250 0x7fa7a80b1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 0x7fa7a80ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a80cbbf0 con 0x7fa7a81591f0 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a80708e0 con 0x7fa7a809daf0 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a80cbd70 con 0x7fa78c005250 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37538/0 (socket says 172.21.15.129:37538) 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 -- 172.21.15.129:0/3456676532 learned_addr learned my addr 172.21.15.129:0/3456676532 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a7400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa78c005250 0x7fa7a80b1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a7e00640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 0x7fa7a80ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 -- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa78c005250 msgr2=0x7fa7a80b1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa78c005250 0x7fa7a80b1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 -- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 msgr2=0x7fa7a80ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 0x7fa7a80ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 -- 172.21.15.129:0/3456676532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a809df00 con 0x7fa7a81591f0 2024-09-17T04:06:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a7e00640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 0x7fa7a80ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:39.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7a6a00640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fa794003f00 tx=0x7fa794001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794011640 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa794016020 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a81f4c00 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794011a80 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.811+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a81f51c0 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.812+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a81f4970 con 0x7fa7a81591f0 2024-09-17T04:06:39.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.814+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa794011c20 con 0x7fa7a81591f0 2024-09-17T04:06:39.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.814+0000 7fa79f400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 0x7fa78807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:39.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.815+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7fa79409b4e0 con 0x7fa7a81591f0 2024-09-17T04:06:39.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.815+0000 7fa7a7400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 0x7fa78807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:39.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.816+0000 7fa7a7400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 0x7fa78807b0a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa798006bd0 tx=0x7fa7980067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:39.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:39.819+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa794020020 con 0x7fa7a81591f0 2024-09-17T04:06:40.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.001+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7a80b3560 con 0x7fa7a81591f0 2024-09-17T04:06:40.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.002+0000 7fa79f400640 1 -- 172.21.15.129:0/3456676532 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) ==== 74+0+13092 (secure 0 0 0) 0x7fa794063ca0 con 0x7fa7a81591f0 2024-09-17T04:06:40.004 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:40.004 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":238,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:38.540553+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":234,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6836/4204505060":"2024-09-18T04:06:35.524600+0000","172.21.15.181:6835/4204505060":"2024-09-18T04:06:35.524600+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 msgr2=0x7fa78807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 0x7fa78807b0a0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa798006bd0 tx=0x7fa7980067a0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 msgr2=0x7fa7a809d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7fa794003f00 tx=0x7fa794001630 comp rx=0 tx=0).stop 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 shutdown_connections 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa788078b90 0x7fa78807b0a0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7a809daf0 0x7fa7a80ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7a81591f0 0x7fa7a809d5b0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 --2- 172.21.15.129:0/3456676532 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa78c005250 0x7fa7a80b1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 >> 172.21.15.129:0/3456676532 conn(0x7fa7a806d410 msgr2=0x7fa7a815b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 shutdown_connections 2024-09-17T04:06:40.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.004+0000 7fa7ae400640 1 -- 172.21.15.129:0/3456676532 wait complete. 2024-09-17T04:06:40.029 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:06:40.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 -- 172.21.15.129:0/3234327211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a00b3f70 msgr2=0x7f70a00b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 --2- 172.21.15.129:0/3234327211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a00b3f70 0x7f70a00b49a0 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f7090005fd0 tx=0x7f709002a540 comp rx=0 tx=0).stop 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 -- 172.21.15.129:0/3234327211 shutdown_connections 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 --2- 172.21.15.129:0/3234327211 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a0074580 0x7f70a00b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 --2- 172.21.15.129:0/3234327211 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a00b3f70 0x7f70a00b49a0 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.254+0000 7f70a6c00640 1 -- 172.21.15.129:0/3234327211 >> 172.21.15.129:0/3234327211 conn(0x7f70a00ccc40 msgr2=0x7f70a00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:40.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.255+0000 7f70a6c00640 1 -- 172.21.15.129:0/3234327211 shutdown_connections 2024-09-17T04:06:40.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.255+0000 7f70a6c00640 1 -- 172.21.15.129:0/3234327211 wait complete. 2024-09-17T04:06:40.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.257+0000 7f70a6c00640 1 Processor -- start 2024-09-17T04:06:40.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.257+0000 7f70a6c00640 1 -- start start 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70a0074580 0x7f70a01f91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a00b2360 0x7f70a0156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f70a006fc20 con 0x7f70a0157730 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f70a006faa0 con 0x7f70a0074580 2024-09-17T04:06:40.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f70a0166f20 con 0x7f70a00b2360 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.258+0000 7f70a4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37560/0 (socket says 172.21.15.129:37560) 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 -- 172.21.15.129:0/86242244 learned_addr learned my addr 172.21.15.129:0/86242244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f709fe00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70a0074580 0x7f70a01f91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f709f400640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a00b2360 0x7f70a0156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 -- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a00b2360 msgr2=0x7f70a0156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a00b2360 0x7f70a0156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 -- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70a0074580 msgr2=0x7f70a01f91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70a0074580 0x7f70a01f91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 -- 172.21.15.129:0/86242244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00aeb10 con 0x7f70a0157730 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.259+0000 7f70a4c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f709800c6e0 tx=0x7f709800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:40.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.260+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098003b70 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.260+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7098003d30 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.260+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7098005330 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.260+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a00aedf0 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.260+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a01f8c80 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.261+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a00b3f70 con 0x7f70a0157730 2024-09-17T04:06:40.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.262+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f70980054d0 con 0x7f70a0157730 2024-09-17T04:06:40.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.263+0000 7f7097e00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 0x7f708007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.263+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7f709809a760 con 0x7f70a0157730 2024-09-17T04:06:40.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.263+0000 7f709fe00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 0x7f708007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.264+0000 7f709fe00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 0x7f708007b0a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f7090033000 tx=0x7f7090004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:40.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.267+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f70980627b0 con 0x7f70a0157730 2024-09-17T04:06:40.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.449+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70a006f160 con 0x7f70a0157730 2024-09-17T04:06:40.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.450+0000 7f7097e00640 1 -- 172.21.15.129:0/86242244 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v238) ==== 93+0+10 (secure 0 0 0) 0x7f70980671e0 con 0x7f70a0157730 2024-09-17T04:06:40.449 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:06:40.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 msgr2=0x7f708007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 0x7f708007b0a0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f7090033000 tx=0x7f7090004730 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 msgr2=0x7f70a00acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f709800c6e0 tx=0x7f709800cba0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 shutdown_connections 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7080078b90 0x7f708007b0a0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f70a0157730 0x7f70a00acf50 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f70a00b2360 0x7f70a0156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 --2- 172.21.15.129:0/86242244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f70a0074580 0x7f70a01f91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 >> 172.21.15.129:0/86242244 conn(0x7f70a00ccc40 msgr2=0x7f70a00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 shutdown_connections 2024-09-17T04:06:40.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.452+0000 7f70a6c00640 1 -- 172.21.15.129:0/86242244 wait complete. 2024-09-17T04:06:40.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub' 2024-09-17T04:06:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.697+0000 7ff1e8000640 1 -- 172.21.15.129:0/1176089909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e0068d70 msgr2=0x7ff1e00b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.697+0000 7ff1e8000640 1 --2- 172.21.15.129:0/1176089909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e0068d70 0x7ff1e00b3960 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0069180 tx=0x7ff1d402a4d0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 -- 172.21.15.129:0/1176089909 shutdown_connections 2024-09-17T04:06:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 --2- 172.21.15.129:0/1176089909 >> v2:172.21.15.181:3300/0 conn(0x7ff1e015c950 0x7ff1e0071490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 --2- 172.21.15.129:0/1176089909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e0068d70 0x7ff1e00b3960 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 -- 172.21.15.129:0/1176089909 >> 172.21.15.129:0/1176089909 conn(0x7ff1e00d0a70 msgr2=0x7ff1e00d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:40.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 -- 172.21.15.129:0/1176089909 shutdown_connections 2024-09-17T04:06:40.698 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.698+0000 7ff1e8000640 1 -- 172.21.15.129:0/1176089909 wait complete. 2024-09-17T04:06:40.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.700+0000 7ff1e8000640 1 Processor -- start 2024-09-17T04:06:40.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.700+0000 7ff1e8000640 1 -- start start 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e0068d70 0x7ff1e00c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1e00719d0 0x7ff1e009f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e00cce20 con 0x7ff1e015c950 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e00ccca0 con 0x7ff1e0068d70 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.701+0000 7ff1e8000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e00ccfa0 con 0x7ff1e00719d0 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37572/0 (socket says 172.21.15.129:37572) 2024-09-17T04:06:40.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e0068d70 0x7ff1e00c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 -- 172.21.15.129:0/2285393096 learned_addr learned my addr 172.21.15.129:0/2285393096 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e4c00640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1e00719d0 0x7ff1e009f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 -- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1e00719d0 msgr2=0x7ff1e009f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1e00719d0 0x7ff1e009f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 -- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e0068d70 msgr2=0x7ff1e00c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e0068d70 0x7ff1e00c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 -- 172.21.15.129:0/2285393096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e009fef0 con 0x7ff1e015c950 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.702+0000 7ff1e6000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800ba10 tx=0x7ff1d800bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.703+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8019070 con 0x7ff1e015c950 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.703+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1d800cc90 con 0x7ff1e015c950 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.703+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e00a01d0 con 0x7ff1e015c950 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.703+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80146a0 con 0x7ff1e015c950 2024-09-17T04:06:40.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.703+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e00a0790 con 0x7ff1e015c950 2024-09-17T04:06:40.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.704+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e009fc60 con 0x7ff1e015c950 2024-09-17T04:06:40.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.705+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff1d801c070 con 0x7ff1e015c950 2024-09-17T04:06:40.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.706+0000 7ff1dd600640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 0x7ff1c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:40.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.706+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7ff1d809a5d0 con 0x7ff1e015c950 2024-09-17T04:06:40.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.707+0000 7ff1e5600640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 0x7ff1c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:40.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.707+0000 7ff1e5600640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 0x7ff1c007b0a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4033000 tx=0x7ff1d4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:40.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.711+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff1d8062740 con 0x7ff1e015c950 2024-09-17T04:06:40.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:40.962+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub"]} v 0) -- 0x7ff1e00b3960 con 0x7ff1e015c950 2024-09-17T04:06:41.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.563+0000 7ff1dd600640 1 -- 172.21.15.129:0/2285393096 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub"]}]=0 v865) ==== 146+0+0 (secure 0 0 0) 0x7ff1d8067170 con 0x7ff1e015c950 2024-09-17T04:06:41.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 msgr2=0x7ff1c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:41.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 0x7ff1c007b0a0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4033000 tx=0x7ff1d4004730 comp rx=0 tx=0).stop 2024-09-17T04:06:41.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 msgr2=0x7ff1e009f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:41.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7ff1d800ba10 tx=0x7ff1d800bed0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 shutdown_connections 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.565+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1c0078b90 0x7ff1c007b0a0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff1e015c950 0x7ff1e009f670 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff1e00719d0 0x7ff1e009f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 --2- 172.21.15.129:0/2285393096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1e0068d70 0x7ff1e00c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 >> 172.21.15.129:0/2285393096 conn(0x7ff1e00d0a70 msgr2=0x7ff1e00b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 shutdown_connections 2024-09-17T04:06:41.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.566+0000 7ff1e8000640 1 -- 172.21.15.129:0/2285393096 wait complete. 2024-09-17T04:06:41.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:06:41.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 -- 172.21.15.129:0/3765382406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 msgr2=0x7fc8a00b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3765382406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a00b33f0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00c72d0 tx=0x7fc88802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 -- 172.21.15.129:0/3765382406 shutdown_connections 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3765382406 >> v2:172.21.15.181:3300/0 conn(0x7fc8a00b3930 0x7fc8a015f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3765382406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a00b33f0 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3765382406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00706a0 0x7fc8a00cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 -- 172.21.15.129:0/3765382406 >> 172.21.15.129:0/3765382406 conn(0x7fc8a006d410 msgr2=0x7fc8a00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 -- 172.21.15.129:0/3765382406 shutdown_connections 2024-09-17T04:06:41.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.813+0000 7fc8a5800640 1 -- 172.21.15.129:0/3765382406 wait complete. 2024-09-17T04:06:41.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.815+0000 7fc8a5800640 1 Processor -- start 2024-09-17T04:06:41.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.815+0000 7fc8a5800640 1 -- start start 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 0x7fc8a015e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 0x7fc8a01f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a015fc30 con 0x7fc8a00cc260 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a015fab0 con 0x7fc8a00c67b0 2024-09-17T04:06:41.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc8a5800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a015fdb0 con 0x7fc8a00b3930 2024-09-17T04:06:41.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:41.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37600/0 (socket says 172.21.15.129:37600) 2024-09-17T04:06:41.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 -- 172.21.15.129:0/3492910043 learned_addr learned my addr 172.21.15.129:0/3492910043 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc89ea00640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 0x7fc8a015e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc89f400640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 0x7fc8a01f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 -- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 msgr2=0x7fc8a015e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 0x7fc8a015e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.817+0000 7fc896000640 1 -- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 msgr2=0x7fc8a01f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc896000640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 0x7fc8a01f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:41.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc896000640 1 -- 172.21.15.129:0/3492910043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a00c75c0 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc89ea00640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 0x7fc8a015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc89f400640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 0x7fc8a01f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc896000640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fc8880099c0 tx=0x7fc8880039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc88803e030 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc8880047a0 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a00c78a0 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888035040 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.818+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a00c7e60 con 0x7fc8a00cc260 2024-09-17T04:06:41.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.819+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a00b22b0 con 0x7fc8a00cc260 2024-09-17T04:06:41.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.820+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc888041030 con 0x7fc8a00cc260 2024-09-17T04:06:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.822+0000 7fc897400640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 0x7fc87c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.822+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7fc8880bb410 con 0x7fc8a00cc260 2024-09-17T04:06:41.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.823+0000 7fc89ea00640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 0x7fc87c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.823+0000 7fc89ea00640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 0x7fc87c07b0d0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc89000a870 tx=0x7fc890008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:41.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:41.827+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc8a00b22b0 con 0x7fc8a00cc260 2024-09-17T04:06:42.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.008+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fc8a00706a0 con 0x7fc8a00cc260 2024-09-17T04:06:42.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.009+0000 7fc897400640 1 -- 172.21.15.129:0/3492910043 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v68) ==== 59+0+2294 (secure 0 0 0) 0x7fc888083ab0 con 0x7fc8a00cc260 2024-09-17T04:06:42.008 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:06:42.008 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:06:42.008 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:06:42.009 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:06:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:06:42.011 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.011+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 msgr2=0x7fc87c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.011+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 0x7fc87c07b0d0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc89000a870 tx=0x7fc890008040 comp rx=0 tx=0).stop 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.011+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 msgr2=0x7fc8a01f6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.011+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fc8880099c0 tx=0x7fc8880039e0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 shutdown_connections 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc87c078bc0 0x7fc87c07b0d0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8a00c67b0 0x7fc8a01f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc8a00cc260 0x7fc8a01f6b10 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 --2- 172.21.15.129:0/3492910043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc8a00b3930 0x7fc8a015e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 >> 172.21.15.129:0/3492910043 conn(0x7fc8a006d410 msgr2=0x7fc8a015b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:42.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 shutdown_connections 2024-09-17T04:06:42.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.012+0000 7fc8a5800640 1 -- 172.21.15.129:0/3492910043 wait complete. 2024-09-17T04:06:42.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 -- 172.21.15.129:0/1232714151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b2650 msgr2=0x7f9fe40cbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 --2- 172.21.15.129:0/1232714151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b2650 0x7f9fe40cbf30 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4005fd0 tx=0x7f9fd402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 -- 172.21.15.129:0/1232714151 shutdown_connections 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 --2- 172.21.15.129:0/1232714151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 0x7f9fe40ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 --2- 172.21.15.129:0/1232714151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b2650 0x7f9fe40cbf30 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.257+0000 7f9fe9000640 1 -- 172.21.15.129:0/1232714151 >> 172.21.15.129:0/1232714151 conn(0x7f9fe40ccc40 msgr2=0x7f9fe40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.258+0000 7f9fe9000640 1 -- 172.21.15.129:0/1232714151 shutdown_connections 2024-09-17T04:06:42.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.258+0000 7f9fe9000640 1 -- 172.21.15.129:0/1232714151 wait complete. 2024-09-17T04:06:42.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.260+0000 7f9fe9000640 1 Processor -- start 2024-09-17T04:06:42.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.260+0000 7f9fe9000640 1 -- start start 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe40b2650 0x7f9fe41e8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 0x7f9fe40c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe40cad80 con 0x7f9fe40b4220 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe40cac00 con 0x7f9fe40b2650 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.261+0000 7f9fe9000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9fe40caf00 con 0x7f9fe40ce500 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37618/0 (socket says 172.21.15.129:37618) 2024-09-17T04:06:42.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 -- 172.21.15.129:0/4238952069 learned_addr learned my addr 172.21.15.129:0/4238952069 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe2800640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 0x7f9fe40c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1e00640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe40b2650 0x7f9fe41e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 -- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 msgr2=0x7f9fe40c7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 0x7f9fe40c7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 -- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe40b2650 msgr2=0x7f9fe41e8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe40b2650 0x7f9fe41e8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 -- 172.21.15.129:0/4238952069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe40c8ca0 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.262+0000 7f9fe1400640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000d6d0 tx=0x7f9fd000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.263+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0017070 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.263+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9fd0009ab0 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.263+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe40c8f80 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.263+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0010460 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.263+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe41e8800 con 0x7f9fe40b4220 2024-09-17T04:06:42.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.264+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe40cbf30 con 0x7f9fe40b4220 2024-09-17T04:06:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.265+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9fd001a070 con 0x7f9fe40b4220 2024-09-17T04:06:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.266+0000 7f9fda000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 0x7f9fc407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.266+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7f9fd009aa50 con 0x7f9fe40b4220 2024-09-17T04:06:42.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.267+0000 7f9fe1e00640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 0x7f9fc407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:42.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.267+0000 7f9fe1e00640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 0x7f9fc407b0a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4033000 tx=0x7f9fd4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:42.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.270+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9fd0062bd0 con 0x7f9fe40b4220 2024-09-17T04:06:42.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.453+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f9fe40cd330 con 0x7f9fe40b4220 2024-09-17T04:06:42.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.458+0000 7f9fda000640 1 -- 172.21.15.129:0/4238952069 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v69)=0 v69) ==== 122+0+0 (secure 0 0 0) 0x7f9fd0067600 con 0x7f9fe40b4220 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 msgr2=0x7f9fc407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 0x7f9fc407b0a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4033000 tx=0x7f9fd4004730 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 msgr2=0x7f9fe41f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000d6d0 tx=0x7f9fd000db90 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 shutdown_connections 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9fc4078b90 0x7f9fc407b0a0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9fe40ce500 0x7f9fe40c7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9fe40b4220 0x7f9fe41f67a0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 --2- 172.21.15.129:0/4238952069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9fe40b2650 0x7f9fe41e8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 >> 172.21.15.129:0/4238952069 conn(0x7f9fe40ccc40 msgr2=0x7f9fe40aff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:42.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.460+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 shutdown_connections 2024-09-17T04:06:42.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:42.461+0000 7f9fe9000640 1 -- 172.21.15.129:0/4238952069 wait complete. 2024-09-17T04:06:42.483 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:06:42.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:06:42.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:06:42.660 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:06:42.660 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:42.660 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:42.696 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:06:42.696 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:06:42.697 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:42.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:06:42.734 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:06:42.734 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:42.772 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:06:42.773 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:06:42.773 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:06:42.809 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:06:42.809 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:06:42.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:43.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 -- 172.21.15.129:0/1668747908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c0162d00 msgr2=0x7fb2c00cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 --2- 172.21.15.129:0/1668747908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c0162d00 0x7fb2c00cf390 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009830 tx=0x7fb2a802a510 comp rx=0 tx=0).stop 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 -- 172.21.15.129:0/1668747908 shutdown_connections 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 --2- 172.21.15.129:0/1668747908 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c0162d00 0x7fb2c00cf390 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 --2- 172.21.15.129:0/1668747908 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c00cd370 0x7fb2c00b4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 -- 172.21.15.129:0/1668747908 >> 172.21.15.129:0/1668747908 conn(0x7fb2c006d410 msgr2=0x7fb2c00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 -- 172.21.15.129:0/1668747908 shutdown_connections 2024-09-17T04:06:43.034 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.034+0000 7fb2c7200640 1 -- 172.21.15.129:0/1668747908 wait complete. 2024-09-17T04:06:43.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.036+0000 7fb2c7200640 1 Processor -- start 2024-09-17T04:06:43.035 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.036+0000 7fb2c7200640 1 -- start start 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c00cd370 0x7fb2c0099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c0163110 0x7fb2c00c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c00cb150 con 0x7fb2c0162d00 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c00cafd0 con 0x7fb2c00cd370 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c00b03e0 con 0x7fb2c0163110 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2b7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37624/0 (socket says 172.21.15.129:37624) 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2b7400640 1 -- 172.21.15.129:0/2757787784 learned_addr learned my addr 172.21.15.129:0/2757787784 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2c5200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c0163110 0x7fb2c00c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2bfe00640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c00cd370 0x7fb2c0099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2b7400640 1 -- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c0163110 msgr2=0x7fb2c00c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.038+0000 7fb2b7400640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c0163110 0x7fb2c00c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2b7400640 1 -- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c00cd370 msgr2=0x7fb2c0099ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2b7400640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c00cd370 0x7fb2c0099ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2b7400640 1 -- 172.21.15.129:0/2757787784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c00c6ce0 con 0x7fb2c0162d00 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2b7400640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009950 tx=0x7fb2a8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:43.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8005c90 con 0x7fb2c0162d00 2024-09-17T04:06:43.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb2a8031e50 con 0x7fb2c0162d00 2024-09-17T04:06:43.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c01f9c10 con 0x7fb2c0162d00 2024-09-17T04:06:43.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8005260 con 0x7fb2c0162d00 2024-09-17T04:06:43.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.039+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c00c5780 con 0x7fb2c0162d00 2024-09-17T04:06:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.040+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c01f9980 con 0x7fb2c0162d00 2024-09-17T04:06:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.042+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb2a8005400 con 0x7fb2c0162d00 2024-09-17T04:06:43.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.042+0000 7fb2bcc00640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 0x7fb29c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.043+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(238..238 src has 1..238) ==== 4943+0+0 (secure 0 0 0) 0x7fb2a80bb9d0 con 0x7fb2c0162d00 2024-09-17T04:06:43.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.043+0000 7fb2bfe00640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 0x7fb29c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.044+0000 7fb2bfe00640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 0x7fb29c07b0a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009530 tx=0x7fb2b0009260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:43.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.047+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb2a803f030 con 0x7fb2c0162d00 2024-09-17T04:06:43.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.263+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2c00cfc40 con 0x7fb2c0162d00 2024-09-17T04:06:43.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.263+0000 7fb2bcc00640 1 -- 172.21.15.129:0/2757787784 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) ==== 95+0+3929 (secure 0 0 0) 0x7fb2a8083b40 con 0x7fb2c0162d00 2024-09-17T04:06:43.263 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:43.263 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":229,"btime":"2024-09-17T04:06:36:528193+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8948,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2160659315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":2160659315},{"type":"v1","addr":"172.21.15.129:6835","nonce":2160659315}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":8951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/3853997213","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3853997213},{"type":"v1","addr":"172.21.15.129:6834","nonce":3853997213}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223},{"gid":9113,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/4245822035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":4245822035},{"type":"v1","addr":"172.21.15.181:6836","nonce":4245822035}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":229},{"gid":11457,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2719015850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2719015850},{"type":"v1","addr":"172.21.15.181:6837","nonce":2719015850}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":223}],"filesystems":[]} 2024-09-17T04:06:43.264 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 229 2024-09-17T04:06:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 msgr2=0x7fb29c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 0x7fb29c07b0a0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009530 tx=0x7fb2b0009260 comp rx=0 tx=0).stop 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 msgr2=0x7fb2c00c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009950 tx=0x7fb2a8033c20 comp rx=0 tx=0).stop 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 shutdown_connections 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb29c078b90 0x7fb29c07b0a0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb2c0163110 0x7fb2c00c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb2c0162d00 0x7fb2c00c6260 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 --2- 172.21.15.129:0/2757787784 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb2c00cd370 0x7fb2c0099ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 >> 172.21.15.129:0/2757787784 conn(0x7fb2c006d410 msgr2=0x7fb2c006f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:43.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 shutdown_connections 2024-09-17T04:06:43.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.266+0000 7fb2c7200640 1 -- 172.21.15.129:0/2757787784 wait complete. 2024-09-17T04:06:43.288 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:06:43.289 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:06:43.289 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:06:43.289 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:06:43.289 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:06:43.289 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:06:43.289 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:06:43.290 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:06:43.290 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:06:43.290 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:06:43.290 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:06:43.290 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:06:43.290 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:06:43.290 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:06:43.290 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:06:43.291 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:06:43.311 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:06:43.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:06:43.313 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:06:43.314 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:06:43.314 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:06:43.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:06:43.316 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:06:43.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:06:43.358 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:06:43.358 DEBUG:teuthology.parallel:result is None 2024-09-17T04:06:43.360 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:06:43.360 DEBUG:teuthology.parallel:result is None 2024-09-17T04:06:43.361 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:06:43.361 DEBUG:teuthology.parallel:result is None 2024-09-17T04:06:43.364 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:06:43.364 DEBUG:teuthology.parallel:result is None 2024-09-17T04:06:43.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:06:43.552 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:06:43.553 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:06:43.558 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:06:43.558 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:06:43.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 -- 172.21.15.129:0/970904125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 msgr2=0x7f234c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 --2- 172.21.15.129:0/970904125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c0cd090 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7f234c0c72d0 tx=0x7f233802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 -- 172.21.15.129:0/970904125 shutdown_connections 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 --2- 172.21.15.129:0/970904125 >> v2:172.21.15.181:3300/0 conn(0x7f234c06f640 0x7f234c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 --2- 172.21.15.129:0/970904125 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c0cd090 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 -- 172.21.15.129:0/970904125 >> 172.21.15.129:0/970904125 conn(0x7f234c06d410 msgr2=0x7f234c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 -- 172.21.15.129:0/970904125 shutdown_connections 2024-09-17T04:06:43.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.606+0000 7f2350e00640 1 -- 172.21.15.129:0/970904125 wait complete. 2024-09-17T04:06:43.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.608+0000 7f2350e00640 1 Processor -- start 2024-09-17T04:06:43.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.608+0000 7f2350e00640 1 -- start start 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f234c06f640 0x7f234c1f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 0x7f234c15f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f234c0710e0 con 0x7f234c0ccc80 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f234c070f60 con 0x7f234c06f640 2024-09-17T04:06:43.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.610+0000 7f2350e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f234c06ff90 con 0x7f234c0af4c0 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f2349400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 0x7f234c15f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:37672/0 (socket says 172.21.15.129:37672) 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f2349e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f234c06f640 0x7f234c1f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 -- 172.21.15.129:0/20590292 learned_addr learned my addr 172.21.15.129:0/20590292 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f2349400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 0x7f234c15f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:52072/0 (socket says 172.21.15.129:52072) 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 -- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 msgr2=0x7f234c15f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 0x7f234c15f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 -- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f234c06f640 msgr2=0x7f234c1f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f234c06f640 0x7f234c1f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 -- 172.21.15.129:0/20590292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234c09cee0 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.611+0000 7f234a800640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f234400ea80 tx=0x7f234400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.612+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f234400ce40 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.612+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2344004b20 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.612+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f234c09d3c0 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.612+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344005020 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.612+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f234c09d980 con 0x7f234c0ccc80 2024-09-17T04:06:43.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.613+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f234c070410 con 0x7f234c0ccc80 2024-09-17T04:06:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.614+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f23440051c0 con 0x7f234c0ccc80 2024-09-17T04:06:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.615+0000 7f2342000640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 0x7f232c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.615+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(242..242 src has 1..242) ==== 4943+0+0 (secure 0 0 0) 0x7f234409b550 con 0x7f234c0ccc80 2024-09-17T04:06:43.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.616+0000 7f2349e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 0x7f232c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:43.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.616+0000 7f2349e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 0x7f232c07b0d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f233c001720 tx=0x7f233c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:43.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.619+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f234c09cee0 con 0x7f234c0ccc80 2024-09-17T04:06:43.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:43.815+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f234c09d150 con 0x7f234c0ccc80 2024-09-17T04:06:44.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.604+0000 7f2342000640 1 -- 172.21.15.129:0/20590292 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v243) ==== 97+0+0 (secure 0 0 0) 0x7f2344063630 con 0x7f234c0ccc80 2024-09-17T04:06:44.603 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:06:44.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 msgr2=0x7f232c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 0x7f232c07b0d0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f233c001720 tx=0x7f233c009040 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 msgr2=0x7f234c1468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7f234400ea80 tx=0x7f234400ef40 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 shutdown_connections 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f232c078bc0 0x7f232c07b0d0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f234c0ccc80 0x7f234c1468b0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f234c0af4c0 0x7f234c15f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 --2- 172.21.15.129:0/20590292 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f234c06f640 0x7f234c1f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 >> 172.21.15.129:0/20590292 conn(0x7f234c06d410 msgr2=0x7f234c0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 shutdown_connections 2024-09-17T04:06:44.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.607+0000 7f2350e00640 1 -- 172.21.15.129:0/20590292 wait complete. 2024-09-17T04:06:44.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 -- 172.21.15.129:0/1109634060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 msgr2=0x7ffbec06eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 --2- 172.21.15.129:0/1109634060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec06eed0 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4005fd0 tx=0x7ffbe402aa90 comp rx=0 tx=0).stop 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 -- 172.21.15.129:0/1109634060 shutdown_connections 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 --2- 172.21.15.129:0/1109634060 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec06eed0 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 --2- 172.21.15.129:0/1109634060 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0b3b90 0x7ffbec0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 -- 172.21.15.129:0/1109634060 >> 172.21.15.129:0/1109634060 conn(0x7ffbec06d410 msgr2=0x7ffbec0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 -- 172.21.15.129:0/1109634060 shutdown_connections 2024-09-17T04:06:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.859+0000 7ffbf2400640 1 -- 172.21.15.129:0/1109634060 wait complete. 2024-09-17T04:06:44.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.862+0000 7ffbf2400640 1 Processor -- start 2024-09-17T04:06:44.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.862+0000 7ffbf2400640 1 -- start start 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0ce170 0x7ffbec1558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 0x7ffbec1f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec06f590 con 0x7ffbec0b2950 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffbec06f410 con 0x7ffbec1f8b70 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbf2400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ffbec06f710 con 0x7ffbec0ce170 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.863+0000 7ffbeb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:44.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52100/0 (socket says 172.21.15.129:52100) 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 -- 172.21.15.129:0/2486929669 learned_addr learned my addr 172.21.15.129:0/2486929669 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeaa00640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0ce170 0x7ffbec1558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbebe00640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 0x7ffbec1f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 -- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0ce170 msgr2=0x7ffbec1558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0ce170 0x7ffbec1558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 -- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 msgr2=0x7ffbec1f8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:44.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 0x7ffbec1f8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 -- 172.21.15.129:0/2486929669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbec156190 con 0x7ffbec0b2950 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbebe00640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 0x7ffbec1f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbeb400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c3d0 tx=0x7ffbdc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.864+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc017070 con 0x7ffbec0b2950 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.865+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffbdc0053c0 con 0x7ffbec0b2950 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.865+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbec156470 con 0x7ffbec0b2950 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.865+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc003850 con 0x7ffbec0b2950 2024-09-17T04:06:44.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.865+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbec156a30 con 0x7ffbec0b2950 2024-09-17T04:06:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.866+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbec0cca60 con 0x7ffbec0b2950 2024-09-17T04:06:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.867+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffbdc01a070 con 0x7ffbec0b2950 2024-09-17T04:06:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.868+0000 7ffbe3400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 0x7ffbcc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:44.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.868+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 4857+0+0 (secure 0 0 0) 0x7ffbdc099e30 con 0x7ffbec0b2950 2024-09-17T04:06:44.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.869+0000 7ffbeaa00640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 0x7ffbcc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:44.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.869+0000 7ffbeaa00640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 0x7ffbcc07b0a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8005fd0 tx=0x7ffbd8005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:44.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:44.873+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ffbec0cca60 con 0x7ffbec0b2950 2024-09-17T04:06:45.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.122+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ffbec06a140 con 0x7ffbec0b2950 2024-09-17T04:06:45.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.123+0000 7ffbe3400640 1 -- 172.21.15.129:0/2486929669 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v45) ==== 80+0+5304 (secure 0 0 0) 0x7ffbdc062520 con 0x7ffbec0b2950 2024-09-17T04:06:45.122 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:45.122 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.124 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:06:45.125 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.126 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.127 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:06:45.129 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.130 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:06:45.132 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:06:45.133 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.133 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.133 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:06:45.133 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:06:45.133 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:45.134 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:45.135 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:06:45.136 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.137 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:06:45.138 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:06:45.139 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:06:45.140 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.125+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 msgr2=0x7ffbcc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.125+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 0x7ffbcc07b0a0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8005fd0 tx=0x7ffbd8005850 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.125+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 msgr2=0x7ffbec153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.125+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc00c3d0 tx=0x7ffbdc00c890 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 shutdown_connections 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbcc078b90 0x7ffbcc07b0a0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbec1f8b70 0x7ffbec1f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbec0ce170 0x7ffbec1558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 --2- 172.21.15.129:0/2486929669 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbec0b2950 0x7ffbec153ba0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 >> 172.21.15.129:0/2486929669 conn(0x7ffbec06d410 msgr2=0x7ffbec0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:45.141 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 shutdown_connections 2024-09-17T04:06:45.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.126+0000 7ffbf2400640 1 -- 172.21.15.129:0/2486929669 wait complete. 2024-09-17T04:06:45.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:06:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 -- 172.21.15.129:0/1997318153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480d16a0 msgr2=0x7f81480cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 --2- 172.21.15.129:0/1997318153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480d16a0 0x7f81480cf040 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f814808db40 tx=0x7f813802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 -- 172.21.15.129:0/1997318153 shutdown_connections 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 --2- 172.21.15.129:0/1997318153 >> v2:172.21.15.181:3300/0 conn(0x7f81480b23e0 0x7f81480cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 --2- 172.21.15.129:0/1997318153 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480d16a0 0x7f81480cf040 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 -- 172.21.15.129:0/1997318153 >> 172.21.15.129:0/1997318153 conn(0x7f814808bff0 msgr2=0x7f814809d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 -- 172.21.15.129:0/1997318153 shutdown_connections 2024-09-17T04:06:45.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.370+0000 7f814f200640 1 -- 172.21.15.129:0/1997318153 wait complete. 2024-09-17T04:06:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.372+0000 7f814f200640 1 Processor -- start 2024-09-17T04:06:45.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.372+0000 7f814f200640 1 -- start start 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81480b23e0 0x7f81481f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 0x7f8148160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f81480cfac0 con 0x7f81480b9be0 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f81480b3670 con 0x7f81480b23e0 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f81480cfc40 con 0x7f81480ccdf0 2024-09-17T04:06:45.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52120/0 (socket says 172.21.15.129:52120) 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814d400640 1 -- 172.21.15.129:0/1603196867 learned_addr learned my addr 172.21.15.129:0/1603196867 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814de00640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81480b23e0 0x7f81481f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.374+0000 7f814e800640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 0x7f8148160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 -- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 msgr2=0x7f8148160ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 0x7f8148160ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 -- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81480b23e0 msgr2=0x7f81481f6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81480b23e0 0x7f81481f6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 -- 172.21.15.129:0/1603196867 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8148147dd0 con 0x7f81480b9be0 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814e800640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 0x7f8148160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:45.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f814d400640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f8138034040 tx=0x7f81380038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:45.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.375+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813803e030 con 0x7f81480b9be0 2024-09-17T04:06:45.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.376+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8138003db0 con 0x7f81480b9be0 2024-09-17T04:06:45.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.376+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81481480b0 con 0x7f81480b9be0 2024-09-17T04:06:45.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.376+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8138035040 con 0x7f81480b9be0 2024-09-17T04:06:45.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.376+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8148148670 con 0x7f81480b9be0 2024-09-17T04:06:45.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.377+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81480d0120 con 0x7f81480b9be0 2024-09-17T04:06:45.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.378+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8138041030 con 0x7f81480b9be0 2024-09-17T04:06:45.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.379+0000 7f8142000640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 0x7f812c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.379+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 4857+0+0 (secure 0 0 0) 0x7f81380bb400 con 0x7f81480b9be0 2024-09-17T04:06:45.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.379+0000 7f814de00640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 0x7f812c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.380+0000 7f814de00640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 0x7f812c07b0d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f813c000f60 tx=0x7f813c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:45.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.383+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8138083530 con 0x7f81480b9be0 2024-09-17T04:06:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.568+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f81481f8f20 con 0x7f81480b9be0 2024-09-17T04:06:45.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.568+0000 7f8142000640 1 -- 172.21.15.129:0/1603196867 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v243) ==== 84+0+65 (secure 0 0 0) 0x7f8138087f60 con 0x7f81480b9be0 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:06:45.568 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 msgr2=0x7f812c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 0x7f812c07b0d0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f813c000f60 tx=0x7f813c009040 comp rx=0 tx=0).stop 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 msgr2=0x7f81480cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7f8138034040 tx=0x7f81380038a0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 shutdown_connections 2024-09-17T04:06:45.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f812c078bc0 0x7f812c07b0d0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f81480ccdf0 0x7f8148160ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f81480b9be0 0x7f81480cc8b0 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 --2- 172.21.15.129:0/1603196867 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f81480b23e0 0x7f81481f6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 >> 172.21.15.129:0/1603196867 conn(0x7f814808bff0 msgr2=0x7f8148158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 shutdown_connections 2024-09-17T04:06:45.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.571+0000 7f814f200640 1 -- 172.21.15.129:0/1603196867 wait complete. 2024-09-17T04:06:45.594 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:06:45.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:06:45.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 -- 172.21.15.129:0/4163267290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b3f70 msgr2=0x7f255c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 --2- 172.21.15.129:0/4163267290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b3f70 0x7f255c0b49a0 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7f255c0c3af0 tx=0x7f254c02a4d0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 -- 172.21.15.129:0/4163267290 shutdown_connections 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 --2- 172.21.15.129:0/4163267290 >> v2:172.21.15.181:3300/0 conn(0x7f255c074580 0x7f255c0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 --2- 172.21.15.129:0/4163267290 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b3f70 0x7f255c0b49a0 unknown :-1 s=CLOSED pgs=2607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 -- 172.21.15.129:0/4163267290 >> 172.21.15.129:0/4163267290 conn(0x7f255c0ccc40 msgr2=0x7f255c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 -- 172.21.15.129:0/4163267290 shutdown_connections 2024-09-17T04:06:45.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.818+0000 7f2560a00640 1 -- 172.21.15.129:0/4163267290 wait complete. 2024-09-17T04:06:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.820+0000 7f2560a00640 1 Processor -- start 2024-09-17T04:06:45.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.820+0000 7f2560a00640 1 -- start start 2024-09-17T04:06:45.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 0x7f255c1623e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 0x7f255c164360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f255c06fc00 con 0x7f255c0b2360 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f255c06fa80 con 0x7f255c163cd0 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2560a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f255c166f20 con 0x7f255c074580 2024-09-17T04:06:45.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2551400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2551400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52150/0 (socket says 172.21.15.129:52150) 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2551400640 1 -- 172.21.15.129:0/4235648865 learned_addr learned my addr 172.21.15.129:0/4235648865 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f255a800640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 0x7f255c164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.822+0000 7f2559e00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 0x7f255c1623e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 -- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 msgr2=0x7f255c1623e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 0x7f255c1623e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 -- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 msgr2=0x7f255c164360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 0x7f255c164360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 -- 172.21.15.129:0/4235648865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255c162c30 con 0x7f255c0b2360 2024-09-17T04:06:45.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2559e00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 0x7f255c1623e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f255a800640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 0x7f255c164360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2551400640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f254400d700 tx=0x7f254400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544017070 con 0x7f255c0b2360 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2544005310 con 0x7f255c0b2360 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2544009820 con 0x7f255c0b2360 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255c162f10 con 0x7f255c0b2360 2024-09-17T04:06:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.823+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f255c1634d0 con 0x7f255c0b2360 2024-09-17T04:06:45.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.824+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f255c0b3f70 con 0x7f255c0b2360 2024-09-17T04:06:45.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.826+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f25440099c0 con 0x7f255c0b2360 2024-09-17T04:06:45.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.826+0000 7f2552a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 0x7f253807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:45.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.827+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 4857+0+0 (secure 0 0 0) 0x7f254409ab50 con 0x7f255c0b2360 2024-09-17T04:06:45.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.827+0000 7f2559e00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 0x7f253807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:45.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.828+0000 7f2559e00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 0x7f253807b0d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f254c033000 tx=0x7f254c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:45.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:45.831+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2544063240 con 0x7f255c0b2360 2024-09-17T04:06:46.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.017+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f255c1629c0 con 0x7f255c0b2360 2024-09-17T04:06:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.601+0000 7f2552a00640 1 -- 172.21.15.129:0/4235648865 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v244) ==== 138+0+0 (secure 0 0 0) 0x7f2544062b40 con 0x7f255c0b2360 2024-09-17T04:06:46.600 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:06:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 msgr2=0x7f253807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 0x7f253807b0d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f254c033000 tx=0x7f254c004730 comp rx=0 tx=0).stop 2024-09-17T04:06:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 msgr2=0x7f255c154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:46.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f254400d700 tx=0x7f254400dbc0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 shutdown_connections 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2538078bc0 0x7f253807b0d0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f255c163cd0 0x7f255c164360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f255c0b2360 0x7f255c154760 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 --2- 172.21.15.129:0/4235648865 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f255c074580 0x7f255c1623e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 >> 172.21.15.129:0/4235648865 conn(0x7f255c0ccc40 msgr2=0x7f255c0d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 shutdown_connections 2024-09-17T04:06:46.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.604+0000 7f2560a00640 1 -- 172.21.15.129:0/4235648865 wait complete. 2024-09-17T04:06:46.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 -- 172.21.15.129:0/447824100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c06ee70 msgr2=0x7f204c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 --2- 172.21.15.129:0/447824100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c06ee70 0x7f204c06f280 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f204c161460 tx=0x7f203402aa30 comp rx=0 tx=0).stop 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 -- 172.21.15.129:0/447824100 shutdown_connections 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 --2- 172.21.15.129:0/447824100 >> v2:172.21.15.181:3300/0 conn(0x7f204c0ce2c0 0x7f204c0ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 --2- 172.21.15.129:0/447824100 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c06ee70 0x7f204c06f280 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 -- 172.21.15.129:0/447824100 >> 172.21.15.129:0/447824100 conn(0x7f204c0d0120 msgr2=0x7f204c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 -- 172.21.15.129:0/447824100 shutdown_connections 2024-09-17T04:06:46.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.864+0000 7f2050a00640 1 -- 172.21.15.129:0/447824100 wait complete. 2024-09-17T04:06:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.866+0000 7f2050a00640 1 Processor -- start 2024-09-17T04:06:46.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.867+0000 7f2050a00640 1 -- start start 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f204c06ee70 0x7f204c1f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f204c156f20 0x7f204c15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f204c0ced90 con 0x7f204c0ce2c0 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f204c0cec10 con 0x7f204c156f20 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2050a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f204c0cef10 con 0x7f204c06ee70 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2042a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2042a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52168/0 (socket says 172.21.15.129:52168) 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f2042a00640 1 -- 172.21.15.129:0/2314909043 learned_addr learned my addr 172.21.15.129:0/2314909043 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.868+0000 7f204be00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f204c156f20 0x7f204c15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:46.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f204b400640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f204c06ee70 0x7f204c1f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 -- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f204c06ee70 msgr2=0x7f204c1f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f204c06ee70 0x7f204c1f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 -- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f204c156f20 msgr2=0x7f204c15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f204c156f20 0x7f204c15b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 -- 172.21.15.129:0/2314909043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204c1f8dd0 con 0x7f204c0ce2c0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2042a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f2034009d60 tx=0x7f2034003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034033050 con 0x7f204c0ce2c0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f20340045c0 con 0x7f204c0ce2c0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2034038450 con 0x7f204c0ce2c0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.869+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f204c1f90b0 con 0x7f204c0ce2c0 2024-09-17T04:06:46.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.870+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f204c1f9670 con 0x7f204c0ce2c0 2024-09-17T04:06:46.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.871+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f204c0cf3f0 con 0x7f204c0ce2c0 2024-09-17T04:06:46.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.873+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2034036070 con 0x7f204c0ce2c0 2024-09-17T04:06:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.874+0000 7f2043e00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 0x7f202807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.874+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(244..244 src has 1..244) ==== 5227+0+0 (secure 0 0 0) 0x7f20340bc0e0 con 0x7f204c0ce2c0 2024-09-17T04:06:46.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.874+0000 7f204b400640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 0x7f202807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.875+0000 7f204b400640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 0x7f202807b0a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f204c0afb70 tx=0x7f203c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:46.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:46.878+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2034084770 con 0x7f204c0ce2c0 2024-09-17T04:06:47.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.063+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f204c076480 con 0x7f204c0ce2c0 2024-09-17T04:06:47.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.610+0000 7f2043e00640 1 -- 172.21.15.129:0/2314909043 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v245) ==== 170+0+0 (secure 0 0 0) 0x7f2034084070 con 0x7f204c0ce2c0 2024-09-17T04:06:47.609 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 msgr2=0x7f202807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 0x7f202807b0a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f204c0afb70 tx=0x7f203c009040 comp rx=0 tx=0).stop 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 msgr2=0x7f204c0c57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f2034009d60 tx=0x7f2034003f90 comp rx=0 tx=0).stop 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 shutdown_connections 2024-09-17T04:06:47.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2028078b90 0x7f202807b0a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f204c156f20 0x7f204c15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f204c0ce2c0 0x7f204c0c57a0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 --2- 172.21.15.129:0/2314909043 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f204c06ee70 0x7f204c1f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 >> 172.21.15.129:0/2314909043 conn(0x7f204c0d0120 msgr2=0x7f204c0b57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 shutdown_connections 2024-09-17T04:06:47.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.613+0000 7f2050a00640 1 -- 172.21.15.129:0/2314909043 wait complete. 2024-09-17T04:06:47.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 -- 172.21.15.129:0/3131898530 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 msgr2=0x7f40280b9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 --2- 172.21.15.129:0/3131898530 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40280b9e80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4018005fd0 tx=0x7f401802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 -- 172.21.15.129:0/3131898530 shutdown_connections 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 --2- 172.21.15.129:0/3131898530 >> v2:172.21.15.181:3300/0 conn(0x7f40280d1720 0x7f40280b46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 --2- 172.21.15.129:0/3131898530 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40280b9e80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 -- 172.21.15.129:0/3131898530 >> 172.21.15.129:0/3131898530 conn(0x7f402808bff0 msgr2=0x7f402809d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 -- 172.21.15.129:0/3131898530 shutdown_connections 2024-09-17T04:06:47.880 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.880+0000 7f402de00640 1 -- 172.21.15.129:0/3131898530 wait complete. 2024-09-17T04:06:47.881 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.882+0000 7f402de00640 1 Processor -- start 2024-09-17T04:06:47.882 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.883+0000 7f402de00640 1 -- start start 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40281594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 0x7f40280d4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f40280b5100 con 0x7f40280d2f90 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f40280b4f80 con 0x7f40280b2ed0 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f402de00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f40280b5280 con 0x7f40280cf1c0 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f401fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f401fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52172/0 (socket says 172.21.15.129:52172) 2024-09-17T04:06:47.885 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.884+0000 7f401fe00640 1 -- 172.21.15.129:0/1188768969 learned_addr learned my addr 172.21.15.129:0/1188768969 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f402ca00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40281594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f402d400640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 0x7f40280d4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 -- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 msgr2=0x7f40281594e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40281594e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 -- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 msgr2=0x7f40280d4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 0x7f40280d4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 -- 172.21.15.129:0/1188768969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40281f6610 con 0x7f40280d2f90 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f402ca00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40281594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f402d400640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 0x7f40280d4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.885+0000 7f401fe00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f401000c400 tx=0x7f401000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.886+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010017070 con 0x7f40280d2f90 2024-09-17T04:06:47.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.886+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40100053c0 con 0x7f40280d2f90 2024-09-17T04:06:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.886+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40280b23f0 con 0x7f40280d2f90 2024-09-17T04:06:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.886+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4010003830 con 0x7f40280d2f90 2024-09-17T04:06:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.886+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f402815ab90 con 0x7f40280d2f90 2024-09-17T04:06:47.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.887+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40281f6380 con 0x7f40280d2f90 2024-09-17T04:06:47.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.889+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f401001a070 con 0x7f40280d2f90 2024-09-17T04:06:47.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.890+0000 7f4025600640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 0x7f400407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:47.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.890+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 5609+0+0 (secure 0 0 0) 0x7f401009a330 con 0x7f40280d2f90 2024-09-17T04:06:47.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.891+0000 7f402ca00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 0x7f400407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:47.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.892+0000 7f402ca00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 0x7f400407b0a0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4018033000 tx=0x7f4018004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:47.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:47.894+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4010021020 con 0x7f40280d2f90 2024-09-17T04:06:48.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.110+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f40280b21e0 con 0x7f40280d2f90 2024-09-17T04:06:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.622+0000 7f4025600640 1 -- 172.21.15.129:0/1188768969 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 36 and data pool 37 v232) ==== 174+0+0 (secure 0 0 0) 0x7f4010062270 con 0x7f40280d2f90 2024-09-17T04:06:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 36 and data pool 37 2024-09-17T04:06:48.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.633+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 msgr2=0x7f400407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.633+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 0x7f400407b0a0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f4018033000 tx=0x7f4018004730 comp rx=0 tx=0).stop 2024-09-17T04:06:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.633+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 msgr2=0x7f40281f58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.633+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f401000c400 tx=0x7f401000c8c0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 shutdown_connections 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4004078b90 0x7f400407b0a0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f40280b2ed0 0x7f40280d4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f40280d2f90 0x7f40281f58e0 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 --2- 172.21.15.129:0/1188768969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f40280cf1c0 0x7f40281594e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.636+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 >> 172.21.15.129:0/1188768969 conn(0x7f402808bff0 msgr2=0x7f40280d2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.637+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 shutdown_connections 2024-09-17T04:06:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.637+0000 7f402de00640 1 -- 172.21.15.129:0/1188768969 wait complete. 2024-09-17T04:06:48.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 -- 172.21.15.129:0/3386776516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c0705a0 msgr2=0x7faa2c0b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 --2- 172.21.15.129:0/3386776516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c0705a0 0x7faa2c0b4140 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7faa1c005fd0 tx=0x7faa1c02a510 comp rx=0 tx=0).stop 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 -- 172.21.15.129:0/3386776516 shutdown_connections 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 --2- 172.21.15.129:0/3386776516 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c0705a0 0x7faa2c0b4140 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 -- 172.21.15.129:0/3386776516 >> 172.21.15.129:0/3386776516 conn(0x7faa2c06d410 msgr2=0x7faa2c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.894+0000 7faa32000640 1 -- 172.21.15.129:0/3386776516 shutdown_connections 2024-09-17T04:06:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.895+0000 7faa32000640 1 -- 172.21.15.129:0/3386776516 wait complete. 2024-09-17T04:06:48.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.897+0000 7faa32000640 1 Processor -- start 2024-09-17T04:06:48.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.897+0000 7faa32000640 1 -- start start 2024-09-17T04:06:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa2c0705a0 0x7faa2c15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa2c159d50 0x7faa2c15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c06f2e0 con 0x7faa2c06eed0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c1612e0 con 0x7faa2c0705a0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.898+0000 7faa32000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faa2c06f460 con 0x7faa2c159d50 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52182/0 (socket says 172.21.15.129:52182) 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 -- 172.21.15.129:0/2072596403 learned_addr learned my addr 172.21.15.129:0/2072596403 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa22a00640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa2c0705a0 0x7faa2c15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2be00640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa2c159d50 0x7faa2c15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 -- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa2c159d50 msgr2=0x7faa2c15a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa2c159d50 0x7faa2c15a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 -- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa2c0705a0 msgr2=0x7faa2c15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa2c0705a0 0x7faa2c15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 -- 172.21.15.129:0/2072596403 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c15be90 con 0x7faa2c06eed0 2024-09-17T04:06:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.899+0000 7faa2b400640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:48.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.900+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c03e030 con 0x7faa2c06eed0 2024-09-17T04:06:48.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.900+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faa1c0047a0 con 0x7faa2c06eed0 2024-09-17T04:06:48.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.900+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c035040 con 0x7faa2c06eed0 2024-09-17T04:06:48.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.900+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c15c170 con 0x7faa2c06eed0 2024-09-17T04:06:48.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.900+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c159550 con 0x7faa2c06eed0 2024-09-17T04:06:48.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.901+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa2c0b4140 con 0x7faa2c06eed0 2024-09-17T04:06:48.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.902+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faa1c041030 con 0x7faa2c06eed0 2024-09-17T04:06:48.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.903+0000 7faa23e00640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 0x7faa0807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:48.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.903+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 5709+0+0 (secure 0 0 0) 0x7faa1c0bb5a0 con 0x7faa2c06eed0 2024-09-17T04:06:48.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.904+0000 7faa22a00640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 0x7faa0807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:48.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.904+0000 7faa22a00640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 0x7faa0807b0a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7faa2c160ec0 tx=0x7faa14009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:48.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:48.908+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faa1c0832f0 con 0x7faa2c06eed0 2024-09-17T04:06:49.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.092+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa2c0af7f0 con 0x7faa2c06eed0 2024-09-17T04:06:49.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.093+0000 7faa23e00640 1 -- 172.21.15.129:0/2072596403 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) ==== 74+0+16748 (secure 0 0 0) 0x7faa1c087d20 con 0x7faa2c06eed0 2024-09-17T04:06:49.093 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:49.093 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":246,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:48.110802+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:46.030146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:47.075519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 msgr2=0x7faa0807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 0x7faa0807b0a0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7faa2c160ec0 tx=0x7faa14009040 comp rx=0 tx=0).stop 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 msgr2=0x7faa2c15af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7faa1c000c00 tx=0x7faa1c0039e0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 shutdown_connections 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faa08078b90 0x7faa0807b0a0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faa2c159d50 0x7faa2c15a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa2c0705a0 0x7faa2c15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 --2- 172.21.15.129:0/2072596403 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faa2c06eed0 0x7faa2c15af80 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 >> 172.21.15.129:0/2072596403 conn(0x7faa2c06d410 msgr2=0x7faa2c06f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 shutdown_connections 2024-09-17T04:06:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.096+0000 7faa32000640 1 -- 172.21.15.129:0/2072596403 wait complete. 2024-09-17T04:06:49.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 -- 172.21.15.129:0/171115053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06eed0 msgr2=0x7f900c0ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 --2- 172.21.15.129:0/171115053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06eed0 0x7f900c0ccc80 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f900c0d1370 tx=0x7f8ff802a560 comp rx=0 tx=0).stop 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 -- 172.21.15.129:0/171115053 shutdown_connections 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 --2- 172.21.15.129:0/171115053 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06eed0 0x7f900c0ccc80 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 --2- 172.21.15.129:0/171115053 >> v2:172.21.15.181:3300/0 conn(0x7f900c0705a0 0x7f900c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 -- 172.21.15.129:0/171115053 >> 172.21.15.129:0/171115053 conn(0x7f900c06d410 msgr2=0x7f900c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 -- 172.21.15.129:0/171115053 shutdown_connections 2024-09-17T04:06:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.338+0000 7f9011a00640 1 -- 172.21.15.129:0/171115053 wait complete. 2024-09-17T04:06:49.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.341+0000 7f9011a00640 1 Processor -- start 2024-09-17T04:06:49.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.341+0000 7f9011a00640 1 -- start start 2024-09-17T04:06:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f900c06eed0 0x7f900c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f900c0705a0 0x7f900c15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f900c161250 con 0x7f900c06f2e0 2024-09-17T04:06:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f900c1610d0 con 0x7f900c0705a0 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f9011a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f900c1613d0 con 0x7f900c06eed0 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f900a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f900a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52202/0 (socket says 172.21.15.129:52202) 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.342+0000 7f900a000640 1 -- 172.21.15.129:0/2573613057 learned_addr learned my addr 172.21.15.129:0/2573613057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900aa00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f900c06eed0 0x7f900c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900b400640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f900c0705a0 0x7f900c15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 -- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f900c06eed0 msgr2=0x7f900c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f900c06eed0 0x7f900c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 -- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f900c0705a0 msgr2=0x7f900c15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f900c0705a0 0x7f900c15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 -- 172.21.15.129:0/2573613057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c1562d0 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f900a000640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8000c00 tx=0x7f8ff8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.343+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff803e030 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.344+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ff8003e00 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.344+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8035040 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.344+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f900c1565b0 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.344+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f900c156b70 con 0x7f900c06f2e0 2024-09-17T04:06:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.345+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f900c0cd5a0 con 0x7f900c06f2e0 2024-09-17T04:06:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.346+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8ff8041030 con 0x7f900c06f2e0 2024-09-17T04:06:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.347+0000 7f9002a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 0x7f8fec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.347+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(246..246 src has 1..246) ==== 5709+0+0 (secure 0 0 0) 0x7f8ff80bb420 con 0x7f900c06f2e0 2024-09-17T04:06:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.348+0000 7f900aa00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 0x7f8fec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.348+0000 7f900aa00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 0x7f8fec07b0d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00bb50 tx=0x7f8ffc00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:49.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.352+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8ff8083290 con 0x7f900c06f2e0 2024-09-17T04:06:49.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.534+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f900c1f79e0 con 0x7f900c06f2e0 2024-09-17T04:06:49.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.536+0000 7f9002a00640 1 -- 172.21.15.129:0/2573613057 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) ==== 74+0+16748 (secure 0 0 0) 0x7f8ff8087cc0 con 0x7f900c06f2e0 2024-09-17T04:06:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:49.535 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":246,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:48.110802+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:46.030146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:47.075519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 msgr2=0x7f8fec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 0x7f8fec07b0d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc00bb50 tx=0x7f8ffc00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 msgr2=0x7f900c15b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f8ff8000c00 tx=0x7f8ff8003910 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 shutdown_connections 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8fec078bc0 0x7f8fec07b0d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f900c0705a0 0x7f900c15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f900c06f2e0 0x7f900c15b0b0 unknown :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 --2- 172.21.15.129:0/2573613057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f900c06eed0 0x7f900c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 >> 172.21.15.129:0/2573613057 conn(0x7f900c06d410 msgr2=0x7f900c0adba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:49.538 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 shutdown_connections 2024-09-17T04:06:49.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.538+0000 7f9011a00640 1 -- 172.21.15.129:0/2573613057 wait complete. 2024-09-17T04:06:49.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:06:49.588 INFO:tasks.ceph.mds.c.smithi129.stderr:2024-09-17T04:06:49.589+0000 7fe454e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:06:49.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:06:49.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:06:49.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 -- 172.21.15.129:0/816865427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc80704c0 msgr2=0x7fabc80ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 --2- 172.21.15.129:0/816865427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc80704c0 0x7fabc80ce480 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fabc0005fd0 tx=0x7fabc002a820 comp rx=0 tx=0).stop 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 -- 172.21.15.129:0/816865427 shutdown_connections 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 --2- 172.21.15.129:0/816865427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc80704c0 0x7fabc80ce480 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 --2- 172.21.15.129:0/816865427 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc815c7b0 0x7fabc80cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 -- 172.21.15.129:0/816865427 >> 172.21.15.129:0/816865427 conn(0x7fabc80d00b0 msgr2=0x7fabc80d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 -- 172.21.15.129:0/816865427 shutdown_connections 2024-09-17T04:06:49.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.793+0000 7fabce800640 1 -- 172.21.15.129:0/816865427 wait complete. 2024-09-17T04:06:49.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.795+0000 7fabce800640 1 Processor -- start 2024-09-17T04:06:49.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.795+0000 7fabce800640 1 -- start start 2024-09-17T04:06:49.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 0x7fabc81f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fabc80ced50 con 0x7fabc80704c0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fabc80cebd0 con 0x7fabc815c7b0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabce800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fabc80ceed0 con 0x7fabc80700b0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 0x7fabc81f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcd400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 0x7fabc81f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52210/0 (socket says 172.21.15.129:52210) 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcca00640 1 -- 172.21.15.129:0/720371690 learned_addr learned my addr 172.21.15.129:0/720371690 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcd400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:47926/0 (socket says 172.21.15.129:47926) 2024-09-17T04:06:49.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.797+0000 7fabcde00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:53106/0 (socket says 172.21.15.129:53106) 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 -- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 msgr2=0x7fabc80c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 -- 172.21.15.129:0/720371690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 msgr2=0x7fabc81f3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 0x7fabc81f3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 -- 172.21.15.129:0/720371690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc81f3fe0 con 0x7fabc815c7b0 2024-09-17T04:06:49.798 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcd400640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:06:49.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabcde00640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fabc80d11b0 tx=0x7fabc0032c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:49.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc003f020 con 0x7fabc815c7b0 2024-09-17T04:06:49.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabc0004040 con 0x7fabc815c7b0 2024-09-17T04:06:49.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc81f42c0 con 0x7fabc815c7b0 2024-09-17T04:06:49.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0037040 con 0x7fabc815c7b0 2024-09-17T04:06:49.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.798+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc809c970 con 0x7fabc815c7b0 2024-09-17T04:06:49.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.799+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc80d0f50 con 0x7fabc815c7b0 2024-09-17T04:06:49.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.801+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fabc0042020 con 0x7fabc815c7b0 2024-09-17T04:06:49.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.802+0000 7fabc5600640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 0x7faba807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:49.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.802+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7fabc00bb560 con 0x7fabc815c7b0 2024-09-17T04:06:49.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.802+0000 7fabcd400640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 0x7faba807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:49.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.803+0000 7fabcd400640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 0x7faba807b0a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fabbc005fd0 tx=0x7fabbc009260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:49.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:49.806+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fabc0049030 con 0x7fabc815c7b0 2024-09-17T04:06:50.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.023+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fabc80ad5b0 con 0x7fabc815c7b0 2024-09-17T04:06:50.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.639+0000 7fabc5600640 1 -- 172.21.15.129:0/720371690 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v234) ==== 118+0+0 (secure 0 0 0) 0x7fabc0083ba0 con 0x7fabc815c7b0 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 msgr2=0x7faba807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 0x7faba807b0a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fabbc005fd0 tx=0x7fabbc009260 comp rx=0 tx=0).stop 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 msgr2=0x7fabc8158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fabc80d11b0 tx=0x7fabc0032c90 comp rx=0 tx=0).stop 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 shutdown_connections 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faba8078b90 0x7faba807b0a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fabc815c7b0 0x7fabc8158ba0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fabc80704c0 0x7fabc81f3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 --2- 172.21.15.129:0/720371690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fabc80700b0 0x7fabc80c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 >> 172.21.15.129:0/720371690 conn(0x7fabc80d00b0 msgr2=0x7fabc80b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:50.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 shutdown_connections 2024-09-17T04:06:50.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.642+0000 7fabce800640 1 -- 172.21.15.129:0/720371690 wait complete. 2024-09-17T04:06:50.666 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:06:50.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:06:50.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 -- 172.21.15.129:0/85933582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 msgr2=0x7fd76c15ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:50.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 --2- 172.21.15.129:0/85933582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c15ba90 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fd758009860 tx=0x7fd75802a5e0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 -- 172.21.15.129:0/85933582 shutdown_connections 2024-09-17T04:06:50.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 --2- 172.21.15.129:0/85933582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c15ba90 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 -- 172.21.15.129:0/85933582 >> 172.21.15.129:0/85933582 conn(0x7fd76c0d00b0 msgr2=0x7fd76c0d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:50.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 -- 172.21.15.129:0/85933582 shutdown_connections 2024-09-17T04:06:50.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.892+0000 7fd76b200640 1 -- 172.21.15.129:0/85933582 wait complete. 2024-09-17T04:06:50.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.894+0000 7fd76b200640 1 Processor -- start 2024-09-17T04:06:50.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.894+0000 7fd76b200640 1 -- start start 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 0x7fd76c1f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 0x7fd76c0c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c0cea50 con 0x7fd76c15b680 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd76c0ce8d0 con 0x7fd76c070930 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd76c0cebd0 con 0x7fd76c15bfd0 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd769400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd769400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52228/0 (socket says 172.21.15.129:52228) 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd769400640 1 -- 172.21.15.129:0/341187885 learned_addr learned my addr 172.21.15.129:0/341187885 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:50.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd769e00640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 0x7fd76c1f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.896+0000 7fd76a800640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 0x7fd76c0c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 -- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 msgr2=0x7fd76c0c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 0x7fd76c0c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 -- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 msgr2=0x7fd76c1f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 0x7fd76c1f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 -- 172.21.15.129:0/341187885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76c1f47c0 con 0x7fd76c15b680 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769e00640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 0x7fd76c1f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd76a800640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 0x7fd76c0c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:50.898 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd769400640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fd758000c00 tx=0x7fd758033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758031d60 con 0x7fd76c15b680 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7580330d0 con 0x7fd76c15b680 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76c158cc0 con 0x7fd76c15b680 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd758033520 con 0x7fd76c15b680 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.897+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd76c159280 con 0x7fd76c15b680 2024-09-17T04:06:50.899 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.898+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd76c0c73b0 con 0x7fd76c15b680 2024-09-17T04:06:50.900 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.900+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd7580035f0 con 0x7fd76c15b680 2024-09-17T04:06:50.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.901+0000 7fd762000640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 0x7fd74c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:50.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.901+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7fd7580bc2d0 con 0x7fd76c15b680 2024-09-17T04:06:50.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.902+0000 7fd769e00640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 0x7fd74c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:50.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.902+0000 7fd769e00640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 0x7fd74c07b0d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd75c007fd0 tx=0x7fd75c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:50.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:50.905+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd76c1f47c0 con 0x7fd76c15b680 2024-09-17T04:06:51.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.123+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd76c0cf080 con 0x7fd76c15b680 2024-09-17T04:06:51.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.636+0000 7fd762000640 1 -- 172.21.15.129:0/341187885 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v235) ==== 122+0+0 (secure 0 0 0) 0x7fd75803f030 con 0x7fd76c15b680 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 msgr2=0x7fd74c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 0x7fd74c07b0d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd75c007fd0 tx=0x7fd75c009040 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 msgr2=0x7fd76c1f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fd758000c00 tx=0x7fd758033d60 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 shutdown_connections 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd74c078bc0 0x7fd74c07b0d0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd76c15bfd0 0x7fd76c0c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd76c15b680 0x7fd76c1f6560 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 --2- 172.21.15.129:0/341187885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd76c070930 0x7fd76c1f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.639+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 >> 172.21.15.129:0/341187885 conn(0x7fd76c0d00b0 msgr2=0x7fd76c0c61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.640+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 shutdown_connections 2024-09-17T04:06:51.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.640+0000 7fd76b200640 1 -- 172.21.15.129:0/341187885 wait complete. 2024-09-17T04:06:51.664 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:06:51.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 -- 172.21.15.129:0/2375079400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80ad8d0 msgr2=0x7f3be81558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 --2- 172.21.15.129:0/2375079400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80ad8d0 0x7f3be81558b0 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8005fd0 tx=0x7f3bd802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.886 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 -- 172.21.15.129:0/2375079400 shutdown_connections 2024-09-17T04:06:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 --2- 172.21.15.129:0/2375079400 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be80d0a70 0x7f3be815ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 --2- 172.21.15.129:0/2375079400 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80ad8d0 0x7f3be81558b0 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 -- 172.21.15.129:0/2375079400 >> 172.21.15.129:0/2375079400 conn(0x7f3be806cfc0 msgr2=0x7f3be806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 -- 172.21.15.129:0/2375079400 shutdown_connections 2024-09-17T04:06:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.886+0000 7f3be7e00640 1 -- 172.21.15.129:0/2375079400 wait complete. 2024-09-17T04:06:51.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.888+0000 7f3be7e00640 1 Processor -- start 2024-09-17T04:06:51.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.888+0000 7f3be7e00640 1 -- start start 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3be80ad8d0 0x7f3be8157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be8146320 0x7f3be8146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3be815f740 con 0x7f3be80af280 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3be815f5c0 con 0x7f3be80ad8d0 2024-09-17T04:06:51.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3be815f8c0 con 0x7f3be8146320 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52252/0 (socket says 172.21.15.129:52252) 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be6000640 1 -- 172.21.15.129:0/807344341 learned_addr learned my addr 172.21.15.129:0/807344341 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be6a00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3be80ad8d0 0x7f3be8157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.890+0000 7f3be7400640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be8146320 0x7f3be8146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 -- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be8146320 msgr2=0x7f3be8146730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be8146320 0x7f3be8146730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 -- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3be80ad8d0 msgr2=0x7f3be8157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3be80ad8d0 0x7f3be8157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 -- 172.21.15.129:0/807344341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be81565e0 con 0x7f3be80af280 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be6000640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400c400 tx=0x7f3bd400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:51.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4017070 con 0x7f3be80af280 2024-09-17T04:06:51.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be81568c0 con 0x7f3be80af280 2024-09-17T04:06:51.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3bd40053c0 con 0x7f3be80af280 2024-09-17T04:06:51.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4003850 con 0x7f3be80af280 2024-09-17T04:06:51.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.891+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be8156e80 con 0x7f3be80af280 2024-09-17T04:06:51.893 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.892+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be8156350 con 0x7f3be80af280 2024-09-17T04:06:51.894 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.894+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3bd401a070 con 0x7f3be80af280 2024-09-17T04:06:51.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.895+0000 7f3bdf400640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 0x7f3bc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:51.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.895+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f3bd409a5d0 con 0x7f3be80af280 2024-09-17T04:06:51.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.895+0000 7f3be6a00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 0x7f3bc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:51.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.896+0000 7f3be6a00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 0x7f3bc807b0a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8033000 tx=0x7f3bd8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:51.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:51.898+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3bd4021020 con 0x7f3be80af280 2024-09-17T04:06:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.115+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3be81558f0 con 0x7f3be80af280 2024-09-17T04:06:52.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.115+0000 7f3bdf400640 1 -- 172.21.15.129:0/807344341 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) ==== 95+0+5246 (secure 0 0 0) 0x7f3be81558f0 con 0x7f3be80af280 2024-09-17T04:06:52.116 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:52.116 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":235,"btime":"2024-09-17T04:06:51:636027+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:06:51.125080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11778":{"gid":11778,"name":"c","rank":0,"incarnation":232,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/968815434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":968815434},{"type":"v1","addr":"172.21.15.129:6835","nonce":968815434}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11778,"qdb_cluster":[11778]},"id":18}]} 2024-09-17T04:06:52.116 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 235 2024-09-17T04:06:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 msgr2=0x7f3bc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 0x7f3bc807b0a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8033000 tx=0x7f3bd8004730 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 msgr2=0x7f3be8145a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400c400 tx=0x7f3bd400c8c0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 shutdown_connections 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3bc8078b90 0x7f3bc807b0a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3be8146320 0x7f3be8146730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3be80af280 0x7f3be8145a70 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 --2- 172.21.15.129:0/807344341 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3be80ad8d0 0x7f3be8157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 >> 172.21.15.129:0/807344341 conn(0x7f3be806cfc0 msgr2=0x7f3be8075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 shutdown_connections 2024-09-17T04:06:52.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.118+0000 7f3be7e00640 1 -- 172.21.15.129:0/807344341 wait complete. 2024-09-17T04:06:52.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:06:52.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.362+0000 7ff215c00640 1 -- 172.21.15.129:0/1390617850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 msgr2=0x7ff21006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.362+0000 7ff215c00640 1 --2- 172.21.15.129:0/1390617850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff21006eed0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff208005fd0 tx=0x7ff20802a830 comp rx=0 tx=0).stop 2024-09-17T04:06:52.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 -- 172.21.15.129:0/1390617850 shutdown_connections 2024-09-17T04:06:52.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 --2- 172.21.15.129:0/1390617850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff21006eed0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 --2- 172.21.15.129:0/1390617850 >> v2:172.21.15.181:3300/0 conn(0x7ff2100b3b90 0x7ff2100cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 -- 172.21.15.129:0/1390617850 >> 172.21.15.129:0/1390617850 conn(0x7ff21006d410 msgr2=0x7ff2100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:52.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 -- 172.21.15.129:0/1390617850 shutdown_connections 2024-09-17T04:06:52.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.363+0000 7ff215c00640 1 -- 172.21.15.129:0/1390617850 wait complete. 2024-09-17T04:06:52.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.365+0000 7ff215c00640 1 Processor -- start 2024-09-17T04:06:52.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.365+0000 7ff215c00640 1 -- start start 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff2101f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2100ad050 0x7ff21015dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff21006f410 con 0x7ff2100ce170 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff2100708e0 con 0x7ff2100ad050 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.366+0000 7ff215c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff21006f590 con 0x7ff2100b2950 2024-09-17T04:06:52.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52260/0 (socket says 172.21.15.129:52260) 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 -- 172.21.15.129:0/816281549 learned_addr learned my addr 172.21.15.129:0/816281549 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff20ea00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff2101f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff20f400640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2100ad050 0x7ff21015dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 -- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 msgr2=0x7ff2101f8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff2101f8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 -- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2100ad050 msgr2=0x7ff21015dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2100ad050 0x7ff21015dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 -- 172.21.15.129:0/816281549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2101f97f0 con 0x7ff2100ce170 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.367+0000 7ff206000640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800e790 tx=0x7ff1f800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.368+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f800eea0 con 0x7ff2100ce170 2024-09-17T04:06:52.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.368+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff1f800cdb0 con 0x7ff2100ce170 2024-09-17T04:06:52.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.368+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8010690 con 0x7ff2100ce170 2024-09-17T04:06:52.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.368+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2101f9ad0 con 0x7ff2100ce170 2024-09-17T04:06:52.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.368+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff21015c400 con 0x7ff2100ce170 2024-09-17T04:06:52.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.369+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2101f9ca0 con 0x7ff2100ce170 2024-09-17T04:06:52.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.370+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff1f8010830 con 0x7ff2100ce170 2024-09-17T04:06:52.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.371+0000 7ff207400640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 0x7ff1ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.371+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7ff1f809b780 con 0x7ff2100ce170 2024-09-17T04:06:52.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.372+0000 7ff20ea00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 0x7ff1ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.372+0000 7ff20ea00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 0x7ff1ec07b0d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff200007960 tx=0x7ff200001d50 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:52.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.375+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff1f80634d0 con 0x7ff2100ce170 2024-09-17T04:06:52.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.558+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff21006fa10 con 0x7ff2100ce170 2024-09-17T04:06:52.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.560+0000 7ff207400640 1 -- 172.21.15.129:0/816281549 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v247) ==== 74+0+16735 (secure 0 0 0) 0x7ff1f8067f00 con 0x7ff2100ce170 2024-09-17T04:06:52.560 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:52.560 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":247,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:06:49.622559+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:46.030146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:47.075519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 msgr2=0x7ff1ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 0x7ff1ec07b0d0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff200007960 tx=0x7ff200001d50 comp rx=0 tx=0).stop 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 msgr2=0x7ff21015da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 secure :-1 s=READY pgs=2623 cs=0 l=1 rev1=1 crypto rx=0x7ff1f800e790 tx=0x7ff1f800ec50 comp rx=0 tx=0).stop 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 shutdown_connections 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff1ec078bc0 0x7ff1ec07b0d0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff2100ad050 0x7ff21015dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff2100ce170 0x7ff21015da80 unknown :-1 s=CLOSED pgs=2623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 --2- 172.21.15.129:0/816281549 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff2100b2950 0x7ff2101f8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.562+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 >> 172.21.15.129:0/816281549 conn(0x7ff21006d410 msgr2=0x7ff2100c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.563+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 shutdown_connections 2024-09-17T04:06:52.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.563+0000 7ff215c00640 1 -- 172.21.15.129:0/816281549 wait complete. 2024-09-17T04:06:52.585 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:06:52.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:52.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 -- 172.21.15.129:0/1965318616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87040cd390 msgr2=0x7f87041609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 --2- 172.21.15.129:0/1965318616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87040cd390 0x7f87041609c0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005fd0 tx=0x7f86fc02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 -- 172.21.15.129:0/1965318616 shutdown_connections 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 --2- 172.21.15.129:0/1965318616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f87040cd390 0x7f87041609c0 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 --2- 172.21.15.129:0/1965318616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 0x7f87040b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 -- 172.21.15.129:0/1965318616 >> 172.21.15.129:0/1965318616 conn(0x7f870406d410 msgr2=0x7f87040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 -- 172.21.15.129:0/1965318616 shutdown_connections 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.810+0000 7f870a800640 1 -- 172.21.15.129:0/1965318616 wait complete. 2024-09-17T04:06:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.812+0000 7f870a800640 1 Processor -- start 2024-09-17T04:06:52.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.812+0000 7f870a800640 1 -- start start 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.813+0000 7f870a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.813+0000 7f870a800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 0x7f87040c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f870a800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8704157290 0x7f87040afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f870a800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8704160f00 con 0x7f870406eed0 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f870a800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f870406f4f0 con 0x7f8704157290 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f870a800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8704161080 con 0x7f87040705a0 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52286/0 (socket says 172.21.15.129:52286) 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 -- 172.21.15.129:0/1127310497 learned_addr learned my addr 172.21.15.129:0/1127310497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8702a00640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 0x7f87040c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703e00640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8704157290 0x7f87040afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 -- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 msgr2=0x7f87040c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 0x7f87040c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 -- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8704157290 msgr2=0x7f87040afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8704157290 0x7f87040afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.814+0000 7f8703400640 1 -- 172.21.15.129:0/1127310497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87040c9980 con 0x7f870406eed0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f8703400640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f86f400e790 tx=0x7f86f400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f400eea0 con 0x7f870406eed0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f86f400cdb0 con 0x7f870406eed0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87040c9c60 con 0x7f870406eed0 2024-09-17T04:06:52.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f4010690 con 0x7f870406eed0 2024-09-17T04:06:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.815+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87040ca220 con 0x7f870406eed0 2024-09-17T04:06:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.816+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87040b4140 con 0x7f870406eed0 2024-09-17T04:06:52.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.817+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f86f4010830 con 0x7f870406eed0 2024-09-17T04:06:52.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.818+0000 7f86fb400640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 0x7f86e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:52.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.818+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f86f409b7a0 con 0x7f870406eed0 2024-09-17T04:06:52.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.819+0000 7f8702a00640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 0x7f86e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:52.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.819+0000 7f8702a00640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 0x7f86e407b0a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f87041eaba0 tx=0x7f86f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:52.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:52.823+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f86f4063bd0 con 0x7f870406eed0 2024-09-17T04:06:53.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.003+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f870406a140 con 0x7f86e4078b90 2024-09-17T04:06:53.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.008+0000 7f86fb400640 1 -- 172.21.15.129:0/1127310497 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251135 (secure 0 0 0) 0x7f870406a140 con 0x7f86e4078b90 2024-09-17T04:06:53.012 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:53.014 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:53.015 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 msgr2=0x7f86e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 0x7f86e407b0a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f87041eaba0 tx=0x7f86f0009040 comp rx=0 tx=0).stop 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 msgr2=0x7f87040c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f86f400e790 tx=0x7f86f400ec50 comp rx=0 tx=0).stop 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 shutdown_connections 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f86e4078b90 0x7f86e407b0a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8704157290 0x7f87040afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f87040705a0 0x7f87040c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 --2- 172.21.15.129:0/1127310497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f870406eed0 0x7f87040c8c00 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 >> 172.21.15.129:0/1127310497 conn(0x7f870406d410 msgr2=0x7f87040d09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:53.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 shutdown_connections 2024-09-17T04:06:53.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.013+0000 7f870a800640 1 -- 172.21.15.129:0/1127310497 wait complete. 2024-09-17T04:06:53.035 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":765,"stamp":"2024-09-17T04:06:52.592477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":135,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413796,"kb_used_data":25456,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754560924,"statfs":{"total":773094113280,"available":772670386176,"internally_reserved":0,"allocated":26066944,"data_stored":12184049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":397463088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.988601"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642545+0000","last_change":"2024-09-17T04:06:48.639552+0000","last_active":"2024-09-17T04:06:48.642545+0000","last_peered":"2024-09-17T04:06:48.642545+0000","last_clean":"2024-09-17T04:06:48.642545+0000","last_became_active":"2024-09-17T04:06:48.639000+0000","last_became_peered":"2024-09-17T04:06:48.639000+0000","last_unstale":"2024-09-17T04:06:48.642545+0000","last_undegraded":"2024-09-17T04:06:48.642545+0000","last_fullsized":"2024-09-17T04:06:48.642545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:43.840034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645686+0000","last_change":"2024-09-17T04:06:48.639479+0000","last_active":"2024-09-17T04:06:48.645686+0000","last_peered":"2024-09-17T04:06:48.645686+0000","last_clean":"2024-09-17T04:06:48.645686+0000","last_became_active":"2024-09-17T04:06:48.632914+0000","last_became_peered":"2024-09-17T04:06:48.632914+0000","last_unstale":"2024-09-17T04:06:48.645686+0000","last_undegraded":"2024-09-17T04:06:48.645686+0000","last_fullsized":"2024-09-17T04:06:48.645686+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:41.325611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656386+0000","last_change":"2024-09-17T04:06:48.653840+0000","last_active":"2024-09-17T04:06:48.656386+0000","last_peered":"2024-09-17T04:06:48.656386+0000","last_clean":"2024-09-17T04:06:48.656386+0000","last_became_active":"2024-09-17T04:06:48.653613+0000","last_became_peered":"2024-09-17T04:06:48.653613+0000","last_unstale":"2024-09-17T04:06:48.656386+0000","last_undegraded":"2024-09-17T04:06:48.656386+0000","last_fullsized":"2024-09-17T04:06:48.656386+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:48.801325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656192+0000","last_change":"2024-09-17T04:06:48.653715+0000","last_active":"2024-09-17T04:06:48.656192+0000","last_peered":"2024-09-17T04:06:48.656192+0000","last_clean":"2024-09-17T04:06:48.656192+0000","last_became_active":"2024-09-17T04:06:48.653119+0000","last_became_peered":"2024-09-17T04:06:48.653119+0000","last_unstale":"2024-09-17T04:06:48.656192+0000","last_undegraded":"2024-09-17T04:06:48.656192+0000","last_fullsized":"2024-09-17T04:06:48.656192+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:16.543513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650744+0000","last_change":"2024-09-17T04:06:48.645621+0000","last_active":"2024-09-17T04:06:48.650744+0000","last_peered":"2024-09-17T04:06:48.650744+0000","last_clean":"2024-09-17T04:06:48.650744+0000","last_became_active":"2024-09-17T04:06:48.644873+0000","last_became_peered":"2024-09-17T04:06:48.644873+0000","last_unstale":"2024-09-17T04:06:48.650744+0000","last_undegraded":"2024-09-17T04:06:48.650744+0000","last_fullsized":"2024-09-17T04:06:48.650744+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:06.459154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619151+0000","last_change":"2024-09-17T04:06:47.619151+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619151+0000","last_undegraded":"2024-09-17T04:06:47.619151+0000","last_fullsized":"2024-09-17T04:06:47.619151+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650635+0000","last_change":"2024-09-17T04:06:48.645454+0000","last_active":"2024-09-17T04:06:48.650635+0000","last_peered":"2024-09-17T04:06:48.650635+0000","last_clean":"2024-09-17T04:06:48.650635+0000","last_became_active":"2024-09-17T04:06:48.645060+0000","last_became_peered":"2024-09-17T04:06:48.645060+0000","last_unstale":"2024-09-17T04:06:48.650635+0000","last_undegraded":"2024-09-17T04:06:48.650635+0000","last_fullsized":"2024-09-17T04:06:48.650635+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:41.950353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637543+0000","last_change":"2024-09-17T04:06:48.636070+0000","last_active":"2024-09-17T04:06:48.637543+0000","last_peered":"2024-09-17T04:06:48.637543+0000","last_clean":"2024-09-17T04:06:48.637543+0000","last_became_active":"2024-09-17T04:06:48.635753+0000","last_became_peered":"2024-09-17T04:06:48.635753+0000","last_unstale":"2024-09-17T04:06:48.637543+0000","last_undegraded":"2024-09-17T04:06:48.637543+0000","last_fullsized":"2024-09-17T04:06:48.637543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:22.423358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645657+0000","last_change":"2024-09-17T04:06:48.640090+0000","last_active":"2024-09-17T04:06:48.645657+0000","last_peered":"2024-09-17T04:06:48.645657+0000","last_clean":"2024-09-17T04:06:48.645657+0000","last_became_active":"2024-09-17T04:06:48.635312+0000","last_became_peered":"2024-09-17T04:06:48.635312+0000","last_unstale":"2024-09-17T04:06:48.645657+0000","last_undegraded":"2024-09-17T04:06:48.645657+0000","last_fullsized":"2024-09-17T04:06:48.645657+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:06.548364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635377+0000","last_change":"2024-09-17T04:06:48.632182+0000","last_active":"2024-09-17T04:06:48.635377+0000","last_peered":"2024-09-17T04:06:48.635377+0000","last_clean":"2024-09-17T04:06:48.635377+0000","last_became_active":"2024-09-17T04:06:48.631841+0000","last_became_peered":"2024-09-17T04:06:48.631841+0000","last_unstale":"2024-09-17T04:06:48.635377+0000","last_undegraded":"2024-09-17T04:06:48.635377+0000","last_fullsized":"2024-09-17T04:06:48.635377+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:32.088154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637508+0000","last_change":"2024-09-17T04:06:48.635273+0000","last_active":"2024-09-17T04:06:48.637508+0000","last_peered":"2024-09-17T04:06:48.637508+0000","last_clean":"2024-09-17T04:06:48.637508+0000","last_became_active":"2024-09-17T04:06:48.634916+0000","last_became_peered":"2024-09-17T04:06:48.634916+0000","last_unstale":"2024-09-17T04:06:48.637508+0000","last_undegraded":"2024-09-17T04:06:48.637508+0000","last_fullsized":"2024-09-17T04:06:48.637508+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:17.777375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655592+0000","last_change":"2024-09-17T04:06:48.654714+0000","last_active":"2024-09-17T04:06:48.655592+0000","last_peered":"2024-09-17T04:06:48.655592+0000","last_clean":"2024-09-17T04:06:48.655592+0000","last_became_active":"2024-09-17T04:06:48.654168+0000","last_became_peered":"2024-09-17T04:06:48.654168+0000","last_unstale":"2024-09-17T04:06:48.655592+0000","last_undegraded":"2024-09-17T04:06:48.655592+0000","last_fullsized":"2024-09-17T04:06:48.655592+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:43.110635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629202+0000","last_change":"2024-09-17T04:06:47.629202+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629202+0000","last_undegraded":"2024-09-17T04:06:47.629202+0000","last_fullsized":"2024-09-17T04:06:47.629202+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638822+0000","last_change":"2024-09-17T04:06:48.635524+0000","last_active":"2024-09-17T04:06:48.638822+0000","last_peered":"2024-09-17T04:06:48.638822+0000","last_clean":"2024-09-17T04:06:48.638822+0000","last_became_active":"2024-09-17T04:06:48.634740+0000","last_became_peered":"2024-09-17T04:06:48.634740+0000","last_unstale":"2024-09-17T04:06:48.638822+0000","last_undegraded":"2024-09-17T04:06:48.638822+0000","last_fullsized":"2024-09-17T04:06:48.638822+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.745727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.648208+0000","last_change":"2024-09-17T04:06:48.639136+0000","last_active":"2024-09-17T04:06:48.648208+0000","last_peered":"2024-09-17T04:06:48.648208+0000","last_clean":"2024-09-17T04:06:48.648208+0000","last_became_active":"2024-09-17T04:06:48.638849+0000","last_became_peered":"2024-09-17T04:06:48.638849+0000","last_unstale":"2024-09-17T04:06:48.648208+0000","last_undegraded":"2024-09-17T04:06:48.648208+0000","last_fullsized":"2024-09-17T04:06:48.648208+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.361484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644287+0000","last_change":"2024-09-17T04:06:48.635932+0000","last_active":"2024-09-17T04:06:48.644287+0000","last_peered":"2024-09-17T04:06:48.644287+0000","last_clean":"2024-09-17T04:06:48.644287+0000","last_became_active":"2024-09-17T04:06:48.634277+0000","last_became_peered":"2024-09-17T04:06:48.634277+0000","last_unstale":"2024-09-17T04:06:48.644287+0000","last_undegraded":"2024-09-17T04:06:48.644287+0000","last_fullsized":"2024-09-17T04:06:48.644287+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:45.140616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629285+0000","last_change":"2024-09-17T04:06:47.629285+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629285+0000","last_undegraded":"2024-09-17T04:06:47.629285+0000","last_fullsized":"2024-09-17T04:06:47.629285+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.627532+0000","last_change":"2024-09-17T04:06:47.627532+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.627532+0000","last_undegraded":"2024-09-17T04:06:47.627532+0000","last_fullsized":"2024-09-17T04:06:47.627532+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645355+0000","last_change":"2024-09-17T04:06:48.643193+0000","last_active":"2024-09-17T04:06:48.645355+0000","last_peered":"2024-09-17T04:06:48.645355+0000","last_clean":"2024-09-17T04:06:48.645355+0000","last_became_active":"2024-09-17T04:06:48.641658+0000","last_became_peered":"2024-09-17T04:06:48.641658+0000","last_unstale":"2024-09-17T04:06:48.645355+0000","last_undegraded":"2024-09-17T04:06:48.645355+0000","last_fullsized":"2024-09-17T04:06:48.645355+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.874073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645968+0000","last_change":"2024-09-17T04:06:48.641052+0000","last_active":"2024-09-17T04:06:48.645968+0000","last_peered":"2024-09-17T04:06:48.645968+0000","last_clean":"2024-09-17T04:06:48.645968+0000","last_became_active":"2024-09-17T04:06:48.640727+0000","last_became_peered":"2024-09-17T04:06:48.640727+0000","last_unstale":"2024-09-17T04:06:48.645968+0000","last_undegraded":"2024-09-17T04:06:48.645968+0000","last_fullsized":"2024-09-17T04:06:48.645968+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:57.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644189+0000","last_change":"2024-09-17T04:06:48.638889+0000","last_active":"2024-09-17T04:06:48.644189+0000","last_peered":"2024-09-17T04:06:48.644189+0000","last_clean":"2024-09-17T04:06:48.644189+0000","last_became_active":"2024-09-17T04:06:48.638320+0000","last_became_peered":"2024-09-17T04:06:48.638320+0000","last_unstale":"2024-09-17T04:06:48.644189+0000","last_undegraded":"2024-09-17T04:06:48.644189+0000","last_fullsized":"2024-09-17T04:06:48.644189+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:19.054145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650679+0000","last_change":"2024-09-17T04:06:48.640395+0000","last_active":"2024-09-17T04:06:48.650679+0000","last_peered":"2024-09-17T04:06:48.650679+0000","last_clean":"2024-09-17T04:06:48.650679+0000","last_became_active":"2024-09-17T04:06:48.640063+0000","last_became_peered":"2024-09-17T04:06:48.640063+0000","last_unstale":"2024-09-17T04:06:48.650679+0000","last_undegraded":"2024-09-17T04:06:48.650679+0000","last_fullsized":"2024-09-17T04:06:48.650679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:19:30.759186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629478+0000","last_change":"2024-09-17T04:06:47.629478+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629478+0000","last_undegraded":"2024-09-17T04:06:47.629478+0000","last_fullsized":"2024-09-17T04:06:47.629478+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642754+0000","last_change":"2024-09-17T04:06:48.641161+0000","last_active":"2024-09-17T04:06:48.642754+0000","last_peered":"2024-09-17T04:06:48.642754+0000","last_clean":"2024-09-17T04:06:48.642754+0000","last_became_active":"2024-09-17T04:06:48.640811+0000","last_became_peered":"2024-09-17T04:06:48.640811+0000","last_unstale":"2024-09-17T04:06:48.642754+0000","last_undegraded":"2024-09-17T04:06:48.642754+0000","last_fullsized":"2024-09-17T04:06:48.642754+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.092195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628734+0000","last_change":"2024-09-17T04:06:47.628734+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628734+0000","last_undegraded":"2024-09-17T04:06:47.628734+0000","last_fullsized":"2024-09-17T04:06:47.628734+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651843+0000","last_change":"2024-09-17T04:06:48.646124+0000","last_active":"2024-09-17T04:06:48.651843+0000","last_peered":"2024-09-17T04:06:48.651843+0000","last_clean":"2024-09-17T04:06:48.651843+0000","last_became_active":"2024-09-17T04:06:48.645673+0000","last_became_peered":"2024-09-17T04:06:48.645673+0000","last_unstale":"2024-09-17T04:06:48.651843+0000","last_undegraded":"2024-09-17T04:06:48.651843+0000","last_fullsized":"2024-09-17T04:06:48.651843+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:26.644315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645666+0000","last_change":"2024-09-17T04:06:48.637878+0000","last_active":"2024-09-17T04:06:48.645666+0000","last_peered":"2024-09-17T04:06:48.645666+0000","last_clean":"2024-09-17T04:06:48.645666+0000","last_became_active":"2024-09-17T04:06:48.637580+0000","last_became_peered":"2024-09-17T04:06:48.637580+0000","last_unstale":"2024-09-17T04:06:48.645666+0000","last_undegraded":"2024-09-17T04:06:48.645666+0000","last_fullsized":"2024-09-17T04:06:48.645666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:38.576300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618364+0000","last_change":"2024-09-17T04:06:47.618364+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618364+0000","last_undegraded":"2024-09-17T04:06:47.618364+0000","last_fullsized":"2024-09-17T04:06:47.618364+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633778+0000","last_change":"2024-09-17T04:06:47.633778+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633778+0000","last_undegraded":"2024-09-17T04:06:47.633778+0000","last_fullsized":"2024-09-17T04:06:47.633778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645544+0000","last_change":"2024-09-17T04:06:48.643887+0000","last_active":"2024-09-17T04:06:48.645544+0000","last_peered":"2024-09-17T04:06:48.645544+0000","last_clean":"2024-09-17T04:06:48.645544+0000","last_became_active":"2024-09-17T04:06:48.643541+0000","last_became_peered":"2024-09-17T04:06:48.643541+0000","last_unstale":"2024-09-17T04:06:48.645544+0000","last_undegraded":"2024-09-17T04:06:48.645544+0000","last_fullsized":"2024-09-17T04:06:48.645544+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:51.740525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650895+0000","last_change":"2024-09-17T04:06:48.646487+0000","last_active":"2024-09-17T04:06:48.650895+0000","last_peered":"2024-09-17T04:06:48.650895+0000","last_clean":"2024-09-17T04:06:48.650895+0000","last_became_active":"2024-09-17T04:06:48.646200+0000","last_became_peered":"2024-09-17T04:06:48.646200+0000","last_unstale":"2024-09-17T04:06:48.650895+0000","last_undegraded":"2024-09-17T04:06:48.650895+0000","last_fullsized":"2024-09-17T04:06:48.650895+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.053314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645614+0000","last_change":"2024-09-17T04:06:48.638450+0000","last_active":"2024-09-17T04:06:48.645614+0000","last_peered":"2024-09-17T04:06:48.645614+0000","last_clean":"2024-09-17T04:06:48.645614+0000","last_became_active":"2024-09-17T04:06:48.638148+0000","last_became_peered":"2024-09-17T04:06:48.638148+0000","last_unstale":"2024-09-17T04:06:48.645614+0000","last_undegraded":"2024-09-17T04:06:48.645614+0000","last_fullsized":"2024-09-17T04:06:48.645614+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.182838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"246'4","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656184+0000","last_change":"2024-09-17T04:06:47.630685+0000","last_active":"2024-09-17T04:06:48.656184+0000","last_peered":"2024-09-17T04:06:48.656184+0000","last_clean":"2024-09-17T04:06:48.656184+0000","last_became_active":"2024-09-17T04:06:47.630367+0000","last_became_peered":"2024-09-17T04:06:47.630367+0000","last_unstale":"2024-09-17T04:06:48.656184+0000","last_undegraded":"2024-09-17T04:06:48.656184+0000","last_fullsized":"2024-09-17T04:06:48.656184+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:56.891603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640751+0000","last_change":"2024-09-17T04:06:48.636369+0000","last_active":"2024-09-17T04:06:48.640751+0000","last_peered":"2024-09-17T04:06:48.640751+0000","last_clean":"2024-09-17T04:06:48.640751+0000","last_became_active":"2024-09-17T04:06:48.636001+0000","last_became_peered":"2024-09-17T04:06:48.636001+0000","last_unstale":"2024-09-17T04:06:48.640751+0000","last_undegraded":"2024-09-17T04:06:48.640751+0000","last_fullsized":"2024-09-17T04:06:48.640751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:21.692754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642287+0000","last_change":"2024-09-17T04:06:47.640788+0000","last_active":"2024-09-17T04:06:47.642287+0000","last_peered":"2024-09-17T04:06:47.642287+0000","last_clean":"2024-09-17T04:06:47.642287+0000","last_became_active":"2024-09-17T04:06:47.640507+0000","last_became_peered":"2024-09-17T04:06:47.640507+0000","last_unstale":"2024-09-17T04:06:47.642287+0000","last_undegraded":"2024-09-17T04:06:47.642287+0000","last_fullsized":"2024-09-17T04:06:47.642287+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:24.521931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.625495+0000","last_change":"2024-09-17T04:06:47.625495+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.625495+0000","last_undegraded":"2024-09-17T04:06:47.625495+0000","last_fullsized":"2024-09-17T04:06:47.625495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642354+0000","last_change":"2024-09-17T04:06:47.640338+0000","last_active":"2024-09-17T04:06:47.642354+0000","last_peered":"2024-09-17T04:06:47.642354+0000","last_clean":"2024-09-17T04:06:47.642354+0000","last_became_active":"2024-09-17T04:06:47.639781+0000","last_became_peered":"2024-09-17T04:06:47.639781+0000","last_unstale":"2024-09-17T04:06:47.642354+0000","last_undegraded":"2024-09-17T04:06:47.642354+0000","last_fullsized":"2024-09-17T04:06:47.642354+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:20.160754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645958+0000","last_change":"2024-09-17T04:06:48.641644+0000","last_active":"2024-09-17T04:06:48.645958+0000","last_peered":"2024-09-17T04:06:48.645958+0000","last_clean":"2024-09-17T04:06:48.645958+0000","last_became_active":"2024-09-17T04:06:48.641054+0000","last_became_peered":"2024-09-17T04:06:48.641054+0000","last_unstale":"2024-09-17T04:06:48.645958+0000","last_undegraded":"2024-09-17T04:06:48.645958+0000","last_fullsized":"2024-09-17T04:06:48.645958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:50.540964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660581+0000","last_change":"2024-09-17T04:06:47.634854+0000","last_active":"2024-09-17T04:06:48.660581+0000","last_peered":"2024-09-17T04:06:48.660581+0000","last_clean":"2024-09-17T04:06:48.660581+0000","last_became_active":"2024-09-17T04:06:47.634556+0000","last_became_peered":"2024-09-17T04:06:47.634556+0000","last_unstale":"2024-09-17T04:06:48.660581+0000","last_undegraded":"2024-09-17T04:06:48.660581+0000","last_fullsized":"2024-09-17T04:06:48.660581+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:55:59.294589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637459+0000","last_change":"2024-09-17T04:06:48.633413+0000","last_active":"2024-09-17T04:06:48.637459+0000","last_peered":"2024-09-17T04:06:48.637459+0000","last_clean":"2024-09-17T04:06:48.637459+0000","last_became_active":"2024-09-17T04:06:48.632689+0000","last_became_peered":"2024-09-17T04:06:48.632689+0000","last_unstale":"2024-09-17T04:06:48.637459+0000","last_undegraded":"2024-09-17T04:06:48.637459+0000","last_fullsized":"2024-09-17T04:06:48.637459+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.165427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653343+0000","last_change":"2024-09-17T04:06:47.637532+0000","last_active":"2024-09-17T04:06:48.653343+0000","last_peered":"2024-09-17T04:06:48.653343+0000","last_clean":"2024-09-17T04:06:48.653343+0000","last_became_active":"2024-09-17T04:06:47.637150+0000","last_became_peered":"2024-09-17T04:06:47.637150+0000","last_unstale":"2024-09-17T04:06:48.653343+0000","last_undegraded":"2024-09-17T04:06:48.653343+0000","last_fullsized":"2024-09-17T04:06:48.653343+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:04.474254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.632926+0000","last_change":"2024-09-17T04:06:48.625119+0000","last_active":"2024-09-17T04:06:48.632926+0000","last_peered":"2024-09-17T04:06:48.632926+0000","last_clean":"2024-09-17T04:06:48.632926+0000","last_became_active":"2024-09-17T04:06:48.624823+0000","last_became_peered":"2024-09-17T04:06:48.624823+0000","last_unstale":"2024-09-17T04:06:48.632926+0000","last_undegraded":"2024-09-17T04:06:48.632926+0000","last_fullsized":"2024-09-17T04:06:48.632926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:21.840133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639235+0000","last_change":"2024-09-17T04:06:47.636767+0000","last_active":"2024-09-17T04:06:47.639235+0000","last_peered":"2024-09-17T04:06:47.639235+0000","last_clean":"2024-09-17T04:06:47.639235+0000","last_became_active":"2024-09-17T04:06:47.635877+0000","last_became_peered":"2024-09-17T04:06:47.635877+0000","last_unstale":"2024-09-17T04:06:47.639235+0000","last_undegraded":"2024-09-17T04:06:47.639235+0000","last_fullsized":"2024-09-17T04:06:47.639235+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633498+0000","last_change":"2024-09-17T04:06:47.633498+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633498+0000","last_undegraded":"2024-09-17T04:06:47.633498+0000","last_fullsized":"2024-09-17T04:06:47.633498+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630330+0000","last_change":"2024-09-17T04:06:47.641518+0000","last_active":"2024-09-17T04:06:48.630330+0000","last_peered":"2024-09-17T04:06:48.630330+0000","last_clean":"2024-09-17T04:06:48.630330+0000","last_became_active":"2024-09-17T04:06:47.641158+0000","last_became_peered":"2024-09-17T04:06:47.641158+0000","last_unstale":"2024-09-17T04:06:48.630330+0000","last_undegraded":"2024-09-17T04:06:48.630330+0000","last_fullsized":"2024-09-17T04:06:48.630330+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:43.208862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.616816+0000","last_change":"2024-09-17T04:06:47.616816+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.616816+0000","last_undegraded":"2024-09-17T04:06:47.616816+0000","last_fullsized":"2024-09-17T04:06:47.616816+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.638712+0000","last_change":"2024-09-17T04:06:47.637605+0000","last_active":"2024-09-17T04:06:47.638712+0000","last_peered":"2024-09-17T04:06:47.638712+0000","last_clean":"2024-09-17T04:06:47.638712+0000","last_became_active":"2024-09-17T04:06:47.637251+0000","last_became_peered":"2024-09-17T04:06:47.637251+0000","last_unstale":"2024-09-17T04:06:47.638712+0000","last_undegraded":"2024-09-17T04:06:47.638712+0000","last_fullsized":"2024-09-17T04:06:47.638712+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619494+0000","last_change":"2024-09-17T04:06:47.619494+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619494+0000","last_undegraded":"2024-09-17T04:06:47.619494+0000","last_fullsized":"2024-09-17T04:06:47.619494+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630182+0000","last_change":"2024-09-17T04:06:47.639900+0000","last_active":"2024-09-17T04:06:48.630182+0000","last_peered":"2024-09-17T04:06:48.630182+0000","last_clean":"2024-09-17T04:06:48.630182+0000","last_became_active":"2024-09-17T04:06:47.638771+0000","last_became_peered":"2024-09-17T04:06:47.638771+0000","last_unstale":"2024-09-17T04:06:48.630182+0000","last_undegraded":"2024-09-17T04:06:48.630182+0000","last_fullsized":"2024-09-17T04:06:48.630182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:42.102677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640781+0000","last_change":"2024-09-17T04:06:48.637971+0000","last_active":"2024-09-17T04:06:48.640781+0000","last_peered":"2024-09-17T04:06:48.640781+0000","last_clean":"2024-09-17T04:06:48.640781+0000","last_became_active":"2024-09-17T04:06:48.637655+0000","last_became_peered":"2024-09-17T04:06:48.637655+0000","last_unstale":"2024-09-17T04:06:48.640781+0000","last_undegraded":"2024-09-17T04:06:48.640781+0000","last_fullsized":"2024-09-17T04:06:48.640781+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:54.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624781+0000","last_change":"2024-09-17T04:06:47.628105+0000","last_active":"2024-09-17T04:06:48.624781+0000","last_peered":"2024-09-17T04:06:48.624781+0000","last_clean":"2024-09-17T04:06:48.624781+0000","last_became_active":"2024-09-17T04:06:47.627756+0000","last_became_peered":"2024-09-17T04:06:47.627756+0000","last_unstale":"2024-09-17T04:06:48.624781+0000","last_undegraded":"2024-09-17T04:06:48.624781+0000","last_fullsized":"2024-09-17T04:06:48.624781+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:38.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653352+0000","last_change":"2024-09-17T04:06:48.646524+0000","last_active":"2024-09-17T04:06:48.653352+0000","last_peered":"2024-09-17T04:06:48.653352+0000","last_clean":"2024-09-17T04:06:48.653352+0000","last_became_active":"2024-09-17T04:06:48.646148+0000","last_became_peered":"2024-09-17T04:06:48.646148+0000","last_unstale":"2024-09-17T04:06:48.653352+0000","last_undegraded":"2024-09-17T04:06:48.653352+0000","last_fullsized":"2024-09-17T04:06:48.653352+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:29:48.285920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"246'3","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.661105+0000","last_change":"2024-09-17T04:06:47.635826+0000","last_active":"2024-09-17T04:06:48.661105+0000","last_peered":"2024-09-17T04:06:48.661105+0000","last_clean":"2024-09-17T04:06:48.661105+0000","last_became_active":"2024-09-17T04:06:47.633375+0000","last_became_peered":"2024-09-17T04:06:47.633375+0000","last_unstale":"2024-09-17T04:06:48.661105+0000","last_undegraded":"2024-09-17T04:06:48.661105+0000","last_fullsized":"2024-09-17T04:06:48.661105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:13.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622935+0000","last_change":"2024-09-17T04:06:47.622935+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622935+0000","last_undegraded":"2024-09-17T04:06:47.622935+0000","last_fullsized":"2024-09-17T04:06:47.622935+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639789+0000","last_change":"2024-09-17T04:06:47.638185+0000","last_active":"2024-09-17T04:06:47.639789+0000","last_peered":"2024-09-17T04:06:47.639789+0000","last_clean":"2024-09-17T04:06:47.639789+0000","last_became_active":"2024-09-17T04:06:47.637416+0000","last_became_peered":"2024-09-17T04:06:47.637416+0000","last_unstale":"2024-09-17T04:06:47.639789+0000","last_undegraded":"2024-09-17T04:06:47.639789+0000","last_fullsized":"2024-09-17T04:06:47.639789+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649907+0000","last_change":"2024-09-17T04:06:48.647751+0000","last_active":"2024-09-17T04:06:48.649907+0000","last_peered":"2024-09-17T04:06:48.649907+0000","last_clean":"2024-09-17T04:06:48.649907+0000","last_became_active":"2024-09-17T04:06:48.647416+0000","last_became_peered":"2024-09-17T04:06:48.647416+0000","last_unstale":"2024-09-17T04:06:48.649907+0000","last_undegraded":"2024-09-17T04:06:48.649907+0000","last_fullsized":"2024-09-17T04:06:48.649907+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.537308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652112+0000","last_change":"2024-09-17T04:06:47.640722+0000","last_active":"2024-09-17T04:06:48.652112+0000","last_peered":"2024-09-17T04:06:48.652112+0000","last_clean":"2024-09-17T04:06:48.652112+0000","last_became_active":"2024-09-17T04:06:47.640406+0000","last_became_peered":"2024-09-17T04:06:47.640406+0000","last_unstale":"2024-09-17T04:06:48.652112+0000","last_undegraded":"2024-09-17T04:06:48.652112+0000","last_fullsized":"2024-09-17T04:06:48.652112+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:40.791049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.633929+0000","last_change":"2024-09-17T04:06:48.628835+0000","last_active":"2024-09-17T04:06:48.633929+0000","last_peered":"2024-09-17T04:06:48.633929+0000","last_clean":"2024-09-17T04:06:48.633929+0000","last_became_active":"2024-09-17T04:06:48.628468+0000","last_became_peered":"2024-09-17T04:06:48.628468+0000","last_unstale":"2024-09-17T04:06:48.633929+0000","last_undegraded":"2024-09-17T04:06:48.633929+0000","last_fullsized":"2024-09-17T04:06:48.633929+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:43.375277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637573+0000","last_change":"2024-09-17T04:06:47.629947+0000","last_active":"2024-09-17T04:06:48.637573+0000","last_peered":"2024-09-17T04:06:48.637573+0000","last_clean":"2024-09-17T04:06:48.637573+0000","last_became_active":"2024-09-17T04:06:47.628403+0000","last_became_peered":"2024-09-17T04:06:47.628403+0000","last_unstale":"2024-09-17T04:06:48.637573+0000","last_undegraded":"2024-09-17T04:06:48.637573+0000","last_fullsized":"2024-09-17T04:06:48.637573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:15.090695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652110+0000","last_change":"2024-09-17T04:06:48.650761+0000","last_active":"2024-09-17T04:06:48.652110+0000","last_peered":"2024-09-17T04:06:48.652110+0000","last_clean":"2024-09-17T04:06:48.652110+0000","last_became_active":"2024-09-17T04:06:48.650386+0000","last_became_peered":"2024-09-17T04:06:48.650386+0000","last_unstale":"2024-09-17T04:06:48.652110+0000","last_undegraded":"2024-09-17T04:06:48.652110+0000","last_fullsized":"2024-09-17T04:06:48.652110+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:57.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639196+0000","last_change":"2024-09-17T04:06:47.637094+0000","last_active":"2024-09-17T04:06:47.639196+0000","last_peered":"2024-09-17T04:06:47.639196+0000","last_clean":"2024-09-17T04:06:47.639196+0000","last_became_active":"2024-09-17T04:06:47.636787+0000","last_became_peered":"2024-09-17T04:06:47.636787+0000","last_unstale":"2024-09-17T04:06:47.639196+0000","last_undegraded":"2024-09-17T04:06:47.639196+0000","last_fullsized":"2024-09-17T04:06:47.639196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644926+0000","last_change":"2024-09-17T04:06:48.642186+0000","last_active":"2024-09-17T04:06:48.644926+0000","last_peered":"2024-09-17T04:06:48.644926+0000","last_clean":"2024-09-17T04:06:48.644926+0000","last_became_active":"2024-09-17T04:06:48.640270+0000","last_became_peered":"2024-09-17T04:06:48.640270+0000","last_unstale":"2024-09-17T04:06:48.644926+0000","last_undegraded":"2024-09-17T04:06:48.644926+0000","last_fullsized":"2024-09-17T04:06:48.644926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:57.596302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"246'8","reported_seq":21,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.665782+0000","last_change":"2024-09-17T04:06:47.639284+0000","last_active":"2024-09-17T04:06:48.665782+0000","last_peered":"2024-09-17T04:06:48.665782+0000","last_clean":"2024-09-17T04:06:48.665782+0000","last_became_active":"2024-09-17T04:06:47.638987+0000","last_became_peered":"2024-09-17T04:06:47.638987+0000","last_unstale":"2024-09-17T04:06:48.665782+0000","last_undegraded":"2024-09-17T04:06:48.665782+0000","last_fullsized":"2024-09-17T04:06:48.665782+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:25.636357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650612+0000","last_change":"2024-09-17T04:06:48.640025+0000","last_active":"2024-09-17T04:06:48.650612+0000","last_peered":"2024-09-17T04:06:48.650612+0000","last_clean":"2024-09-17T04:06:48.650612+0000","last_became_active":"2024-09-17T04:06:48.639713+0000","last_became_peered":"2024-09-17T04:06:48.639713+0000","last_unstale":"2024-09-17T04:06:48.650612+0000","last_undegraded":"2024-09-17T04:06:48.650612+0000","last_fullsized":"2024-09-17T04:06:48.650612+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:46.136192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650031+0000","last_change":"2024-09-17T04:06:47.627885+0000","last_active":"2024-09-17T04:06:48.650031+0000","last_peered":"2024-09-17T04:06:48.650031+0000","last_clean":"2024-09-17T04:06:48.650031+0000","last_became_active":"2024-09-17T04:06:47.627314+0000","last_became_peered":"2024-09-17T04:06:47.627314+0000","last_unstale":"2024-09-17T04:06:48.650031+0000","last_undegraded":"2024-09-17T04:06:48.650031+0000","last_fullsized":"2024-09-17T04:06:48.650031+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:32.996351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646180+0000","last_change":"2024-09-17T04:06:48.640039+0000","last_active":"2024-09-17T04:06:48.646180+0000","last_peered":"2024-09-17T04:06:48.646180+0000","last_clean":"2024-09-17T04:06:48.646180+0000","last_became_active":"2024-09-17T04:06:48.639062+0000","last_became_peered":"2024-09-17T04:06:48.639062+0000","last_unstale":"2024-09-17T04:06:48.646180+0000","last_undegraded":"2024-09-17T04:06:48.646180+0000","last_fullsized":"2024-09-17T04:06:48.646180+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:39.282547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655638+0000","last_change":"2024-09-17T04:06:47.626021+0000","last_active":"2024-09-17T04:06:48.655638+0000","last_peered":"2024-09-17T04:06:48.655638+0000","last_clean":"2024-09-17T04:06:48.655638+0000","last_became_active":"2024-09-17T04:06:47.625353+0000","last_became_peered":"2024-09-17T04:06:47.625353+0000","last_unstale":"2024-09-17T04:06:48.655638+0000","last_undegraded":"2024-09-17T04:06:48.655638+0000","last_fullsized":"2024-09-17T04:06:48.655638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622812+0000","last_change":"2024-09-17T04:06:47.622812+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622812+0000","last_undegraded":"2024-09-17T04:06:47.622812+0000","last_fullsized":"2024-09-17T04:06:47.622812+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.632014+0000","last_change":"2024-09-17T04:06:47.630827+0000","last_active":"2024-09-17T04:06:47.632014+0000","last_peered":"2024-09-17T04:06:47.632014+0000","last_clean":"2024-09-17T04:06:47.632014+0000","last_became_active":"2024-09-17T04:06:47.630146+0000","last_became_peered":"2024-09-17T04:06:47.630146+0000","last_unstale":"2024-09-17T04:06:47.632014+0000","last_undegraded":"2024-09-17T04:06:47.632014+0000","last_fullsized":"2024-09-17T04:06:47.632014+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656054+0000","last_change":"2024-09-17T04:06:48.655091+0000","last_active":"2024-09-17T04:06:48.656054+0000","last_peered":"2024-09-17T04:06:48.656054+0000","last_clean":"2024-09-17T04:06:48.656054+0000","last_became_active":"2024-09-17T04:06:48.654782+0000","last_became_peered":"2024-09-17T04:06:48.654782+0000","last_unstale":"2024-09-17T04:06:48.656054+0000","last_undegraded":"2024-09-17T04:06:48.656054+0000","last_fullsized":"2024-09-17T04:06:48.656054+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:06.889990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642059+0000","last_change":"2024-09-17T04:06:47.639967+0000","last_active":"2024-09-17T04:06:47.642059+0000","last_peered":"2024-09-17T04:06:47.642059+0000","last_clean":"2024-09-17T04:06:47.642059+0000","last_became_active":"2024-09-17T04:06:47.639606+0000","last_became_peered":"2024-09-17T04:06:47.639606+0000","last_unstale":"2024-09-17T04:06:47.642059+0000","last_undegraded":"2024-09-17T04:06:47.642059+0000","last_fullsized":"2024-09-17T04:06:47.642059+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:36.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649830+0000","last_change":"2024-09-17T04:06:48.642222+0000","last_active":"2024-09-17T04:06:48.649830+0000","last_peered":"2024-09-17T04:06:48.649830+0000","last_clean":"2024-09-17T04:06:48.649830+0000","last_became_active":"2024-09-17T04:06:48.641845+0000","last_became_peered":"2024-09-17T04:06:48.641845+0000","last_unstale":"2024-09-17T04:06:48.649830+0000","last_undegraded":"2024-09-17T04:06:48.649830+0000","last_fullsized":"2024-09-17T04:06:48.649830+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:51.088460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.659677+0000","last_change":"2024-09-17T04:06:47.635036+0000","last_active":"2024-09-17T04:06:48.659677+0000","last_peered":"2024-09-17T04:06:48.659677+0000","last_clean":"2024-09-17T04:06:48.659677+0000","last_became_active":"2024-09-17T04:06:47.634721+0000","last_became_peered":"2024-09-17T04:06:47.634721+0000","last_unstale":"2024-09-17T04:06:48.659677+0000","last_undegraded":"2024-09-17T04:06:48.659677+0000","last_fullsized":"2024-09-17T04:06:48.659677+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:48.022710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649772+0000","last_change":"2024-09-17T04:06:48.647597+0000","last_active":"2024-09-17T04:06:48.649772+0000","last_peered":"2024-09-17T04:06:48.649772+0000","last_clean":"2024-09-17T04:06:48.649772+0000","last_became_active":"2024-09-17T04:06:48.647265+0000","last_became_peered":"2024-09-17T04:06:48.647265+0000","last_unstale":"2024-09-17T04:06:48.649772+0000","last_undegraded":"2024-09-17T04:06:48.649772+0000","last_fullsized":"2024-09-17T04:06:48.649772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:31.880487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.621482+0000","last_change":"2024-09-17T04:06:47.627934+0000","last_active":"2024-09-17T04:06:48.621482+0000","last_peered":"2024-09-17T04:06:48.621482+0000","last_clean":"2024-09-17T04:06:48.621482+0000","last_became_active":"2024-09-17T04:06:47.627651+0000","last_became_peered":"2024-09-17T04:06:47.627651+0000","last_unstale":"2024-09-17T04:06:48.621482+0000","last_undegraded":"2024-09-17T04:06:48.621482+0000","last_fullsized":"2024-09-17T04:06:48.621482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:28.266741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635500+0000","last_change":"2024-09-17T04:06:48.630489+0000","last_active":"2024-09-17T04:06:48.635500+0000","last_peered":"2024-09-17T04:06:48.635500+0000","last_clean":"2024-09-17T04:06:48.635500+0000","last_became_active":"2024-09-17T04:06:48.630199+0000","last_became_peered":"2024-09-17T04:06:48.630199+0000","last_unstale":"2024-09-17T04:06:48.635500+0000","last_undegraded":"2024-09-17T04:06:48.635500+0000","last_fullsized":"2024-09-17T04:06:48.635500+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:49.945070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.637806+0000","last_change":"2024-09-17T04:06:47.636455+0000","last_active":"2024-09-17T04:06:47.637806+0000","last_peered":"2024-09-17T04:06:47.637806+0000","last_clean":"2024-09-17T04:06:47.637806+0000","last_became_active":"2024-09-17T04:06:47.636161+0000","last_became_peered":"2024-09-17T04:06:47.636161+0000","last_unstale":"2024-09-17T04:06:47.637806+0000","last_undegraded":"2024-09-17T04:06:47.637806+0000","last_fullsized":"2024-09-17T04:06:47.637806+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630666+0000","last_change":"2024-09-17T04:06:48.626073+0000","last_active":"2024-09-17T04:06:48.630666+0000","last_peered":"2024-09-17T04:06:48.630666+0000","last_clean":"2024-09-17T04:06:48.630666+0000","last_became_active":"2024-09-17T04:06:48.624659+0000","last_became_peered":"2024-09-17T04:06:48.624659+0000","last_unstale":"2024-09-17T04:06:48.630666+0000","last_undegraded":"2024-09-17T04:06:48.630666+0000","last_fullsized":"2024-09-17T04:06:48.630666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:31.519896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.622226+0000","last_change":"2024-09-17T04:06:47.631678+0000","last_active":"2024-09-17T04:06:48.622226+0000","last_peered":"2024-09-17T04:06:48.622226+0000","last_clean":"2024-09-17T04:06:48.622226+0000","last_became_active":"2024-09-17T04:06:47.629357+0000","last_became_peered":"2024-09-17T04:06:47.629357+0000","last_unstale":"2024-09-17T04:06:48.622226+0000","last_undegraded":"2024-09-17T04:06:48.622226+0000","last_fullsized":"2024-09-17T04:06:48.622226+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:58.960464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646523+0000","last_change":"2024-09-17T04:06:48.641309+0000","last_active":"2024-09-17T04:06:48.646523+0000","last_peered":"2024-09-17T04:06:48.646523+0000","last_clean":"2024-09-17T04:06:48.646523+0000","last_became_active":"2024-09-17T04:06:48.640780+0000","last_became_peered":"2024-09-17T04:06:48.640780+0000","last_unstale":"2024-09-17T04:06:48.646523+0000","last_undegraded":"2024-09-17T04:06:48.646523+0000","last_fullsized":"2024-09-17T04:06:48.646523+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:38.559159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660451+0000","last_change":"2024-09-17T04:06:47.634843+0000","last_active":"2024-09-17T04:06:48.660451+0000","last_peered":"2024-09-17T04:06:48.660451+0000","last_clean":"2024-09-17T04:06:48.660451+0000","last_became_active":"2024-09-17T04:06:47.634529+0000","last_became_peered":"2024-09-17T04:06:47.634529+0000","last_unstale":"2024-09-17T04:06:48.660451+0000","last_undegraded":"2024-09-17T04:06:48.660451+0000","last_fullsized":"2024-09-17T04:06:48.660451+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:24.241367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628250+0000","last_change":"2024-09-17T04:06:47.628250+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628250+0000","last_undegraded":"2024-09-17T04:06:47.628250+0000","last_fullsized":"2024-09-17T04:06:47.628250+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.619096+0000","last_change":"2024-09-17T04:06:47.635594+0000","last_active":"2024-09-17T04:06:48.619096+0000","last_peered":"2024-09-17T04:06:48.619096+0000","last_clean":"2024-09-17T04:06:48.619096+0000","last_became_active":"2024-09-17T04:06:47.633247+0000","last_became_peered":"2024-09-17T04:06:47.633247+0000","last_unstale":"2024-09-17T04:06:48.619096+0000","last_undegraded":"2024-09-17T04:06:48.619096+0000","last_fullsized":"2024-09-17T04:06:48.619096+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:29.409434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656026+0000","last_change":"2024-09-17T04:06:48.654761+0000","last_active":"2024-09-17T04:06:48.656026+0000","last_peered":"2024-09-17T04:06:48.656026+0000","last_clean":"2024-09-17T04:06:48.656026+0000","last_became_active":"2024-09-17T04:06:48.654228+0000","last_became_peered":"2024-09-17T04:06:48.654228+0000","last_unstale":"2024-09-17T04:06:48.656026+0000","last_undegraded":"2024-09-17T04:06:48.656026+0000","last_fullsized":"2024-09-17T04:06:48.656026+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:46.467512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650149+0000","last_change":"2024-09-17T04:06:47.631080+0000","last_active":"2024-09-17T04:06:48.650149+0000","last_peered":"2024-09-17T04:06:48.650149+0000","last_clean":"2024-09-17T04:06:48.650149+0000","last_became_active":"2024-09-17T04:06:47.630810+0000","last_became_peered":"2024-09-17T04:06:47.630810+0000","last_unstale":"2024-09-17T04:06:48.650149+0000","last_undegraded":"2024-09-17T04:06:48.650149+0000","last_fullsized":"2024-09-17T04:06:48.650149+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.601411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644148+0000","last_change":"2024-09-17T04:06:48.640563+0000","last_active":"2024-09-17T04:06:48.644148+0000","last_peered":"2024-09-17T04:06:48.644148+0000","last_clean":"2024-09-17T04:06:48.644148+0000","last_became_active":"2024-09-17T04:06:48.640193+0000","last_became_peered":"2024-09-17T04:06:48.640193+0000","last_unstale":"2024-09-17T04:06:48.644148+0000","last_undegraded":"2024-09-17T04:06:48.644148+0000","last_fullsized":"2024-09-17T04:06:48.644148+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:43.623529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631108+0000","last_change":"2024-09-17T04:06:47.640808+0000","last_active":"2024-09-17T04:06:48.631108+0000","last_peered":"2024-09-17T04:06:48.631108+0000","last_clean":"2024-09-17T04:06:48.631108+0000","last_became_active":"2024-09-17T04:06:47.639846+0000","last_became_peered":"2024-09-17T04:06:47.639846+0000","last_unstale":"2024-09-17T04:06:48.631108+0000","last_undegraded":"2024-09-17T04:06:48.631108+0000","last_fullsized":"2024-09-17T04:06:48.631108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:15.837166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618723+0000","last_change":"2024-09-17T04:06:47.618723+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618723+0000","last_undegraded":"2024-09-17T04:06:47.618723+0000","last_fullsized":"2024-09-17T04:06:47.618723+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630437+0000","last_change":"2024-09-17T04:06:47.639290+0000","last_active":"2024-09-17T04:06:48.630437+0000","last_peered":"2024-09-17T04:06:48.630437+0000","last_clean":"2024-09-17T04:06:48.630437+0000","last_became_active":"2024-09-17T04:06:47.638370+0000","last_became_peered":"2024-09-17T04:06:47.638370+0000","last_unstale":"2024-09-17T04:06:48.630437+0000","last_undegraded":"2024-09-17T04:06:48.630437+0000","last_fullsized":"2024-09-17T04:06:48.630437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:17.818126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651574+0000","last_change":"2024-09-17T04:06:48.645144+0000","last_active":"2024-09-17T04:06:48.651574+0000","last_peered":"2024-09-17T04:06:48.651574+0000","last_clean":"2024-09-17T04:06:48.651574+0000","last_became_active":"2024-09-17T04:06:48.644506+0000","last_became_peered":"2024-09-17T04:06:48.644506+0000","last_unstale":"2024-09-17T04:06:48.651574+0000","last_undegraded":"2024-09-17T04:06:48.651574+0000","last_fullsized":"2024-09-17T04:06:48.651574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:58.415646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638296+0000","last_change":"2024-09-17T04:06:47.637796+0000","last_active":"2024-09-17T04:06:48.638296+0000","last_peered":"2024-09-17T04:06:48.638296+0000","last_clean":"2024-09-17T04:06:48.638296+0000","last_became_active":"2024-09-17T04:06:47.637473+0000","last_became_peered":"2024-09-17T04:06:47.637473+0000","last_unstale":"2024-09-17T04:06:48.638296+0000","last_undegraded":"2024-09-17T04:06:48.638296+0000","last_fullsized":"2024-09-17T04:06:48.638296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:35.426948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635294+0000","last_change":"2024-09-17T04:06:48.626386+0000","last_active":"2024-09-17T04:06:48.635294+0000","last_peered":"2024-09-17T04:06:48.635294+0000","last_clean":"2024-09-17T04:06:48.635294+0000","last_became_active":"2024-09-17T04:06:48.623872+0000","last_became_peered":"2024-09-17T04:06:48.623872+0000","last_unstale":"2024-09-17T04:06:48.635294+0000","last_undegraded":"2024-09-17T04:06:48.635294+0000","last_fullsized":"2024-09-17T04:06:48.635294+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.050814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624231+0000","last_change":"2024-09-17T04:06:47.632594+0000","last_active":"2024-09-17T04:06:48.624231+0000","last_peered":"2024-09-17T04:06:48.624231+0000","last_clean":"2024-09-17T04:06:48.624231+0000","last_became_active":"2024-09-17T04:06:47.632229+0000","last_became_peered":"2024-09-17T04:06:47.632229+0000","last_unstale":"2024-09-17T04:06:48.624231+0000","last_undegraded":"2024-09-17T04:06:48.624231+0000","last_fullsized":"2024-09-17T04:06:48.624231+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:10.174881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650605+0000","last_change":"2024-09-17T04:06:48.640651+0000","last_active":"2024-09-17T04:06:48.650605+0000","last_peered":"2024-09-17T04:06:48.650605+0000","last_clean":"2024-09-17T04:06:48.650605+0000","last_became_active":"2024-09-17T04:06:48.639761+0000","last_became_peered":"2024-09-17T04:06:48.639761+0000","last_unstale":"2024-09-17T04:06:48.650605+0000","last_undegraded":"2024-09-17T04:06:48.650605+0000","last_fullsized":"2024-09-17T04:06:48.650605+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:53.798909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638592+0000","last_change":"2024-09-17T04:06:47.639117+0000","last_active":"2024-09-17T04:06:48.638592+0000","last_peered":"2024-09-17T04:06:48.638592+0000","last_clean":"2024-09-17T04:06:48.638592+0000","last_became_active":"2024-09-17T04:06:47.638778+0000","last_became_peered":"2024-09-17T04:06:47.638778+0000","last_unstale":"2024-09-17T04:06:48.638592+0000","last_undegraded":"2024-09-17T04:06:48.638592+0000","last_fullsized":"2024-09-17T04:06:48.638592+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:42.527414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653501+0000","last_change":"2024-09-17T04:06:48.648394+0000","last_active":"2024-09-17T04:06:48.653501+0000","last_peered":"2024-09-17T04:06:48.653501+0000","last_clean":"2024-09-17T04:06:48.653501+0000","last_became_active":"2024-09-17T04:06:48.647962+0000","last_became_peered":"2024-09-17T04:06:48.647962+0000","last_unstale":"2024-09-17T04:06:48.653501+0000","last_undegraded":"2024-09-17T04:06:48.653501+0000","last_fullsized":"2024-09-17T04:06:48.653501+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:16.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":525,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631126+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:48.631126+0000","last_peered":"2024-09-17T04:06:48.631126+0000","last_clean":"2024-09-17T04:06:48.631126+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:48.631126+0000","last_undegraded":"2024-09-17T04:06:48.631126+0000","last_fullsized":"2024-09-17T04:06:48.631126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509651,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509654,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34248,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337592,"statfs":{"total":96636764160,"available":96601694208,"internally_reserved":0,"allocated":3129344,"data_stored":1380978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56424,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315416,"statfs":{"total":96636764160,"available":96578985984,"internally_reserved":0,"allocated":3100672,"data_stored":1372501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56444,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315396,"statfs":{"total":96636764160,"available":96578965504,"internally_reserved":0,"allocated":3117056,"data_stored":1379858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509653,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56448,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315392,"statfs":{"total":96636764160,"available":96578961408,"internally_reserved":0,"allocated":3121152,"data_stored":1379400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509653,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3121152,"data_stored":1379332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542355,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52256,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319584,"statfs":{"total":96636764160,"available":96583254016,"internally_reserved":0,"allocated":3088384,"data_stored":1366242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:53.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:53.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 -- 172.21.15.129:0/2293601526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 msgr2=0x7f0e380cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 --2- 172.21.15.129:0/2293601526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e380cd1c0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f0e380ccc10 tx=0x7f0e3002aa10 comp rx=0 tx=0).stop 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 -- 172.21.15.129:0/2293601526 shutdown_connections 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 --2- 172.21.15.129:0/2293601526 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e380cd1c0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 --2- 172.21.15.129:0/2293601526 >> v2:172.21.15.181:3300/0 conn(0x7f0e3806eed0 0x7f0e380ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 --2- 172.21.15.129:0/2293601526 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e380705a0 0x7f0e380b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 -- 172.21.15.129:0/2293601526 >> 172.21.15.129:0/2293601526 conn(0x7f0e3806d410 msgr2=0x7f0e380b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:53.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 -- 172.21.15.129:0/2293601526 shutdown_connections 2024-09-17T04:06:53.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.294+0000 7f0e3e400640 1 -- 172.21.15.129:0/2293601526 wait complete. 2024-09-17T04:06:53.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.297+0000 7f0e3e400640 1 Processor -- start 2024-09-17T04:06:53.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.297+0000 7f0e3e400640 1 -- start start 2024-09-17T04:06:53.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3806eed0 0x7f0e381f84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 0x7f0e381f9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e38160b40 con 0x7f0e3806f2e0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0e381609c0 con 0x7f0e380705a0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e3e400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0e38160cc0 con 0x7f0e3806eed0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.298+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52310/0 (socket says 172.21.15.129:52310) 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 -- 172.21.15.129:0/3235437406 learned_addr learned my addr 172.21.15.129:0/3235437406 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e37400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3806eed0 0x7f0e381f84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e37e00640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 0x7f0e381f9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 -- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3806eed0 msgr2=0x7f0e381f84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3806eed0 0x7f0e381f84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 -- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 msgr2=0x7f0e381f9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 0x7f0e381f9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 -- 172.21.15.129:0/3235437406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e380cab10 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e37e00640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 0x7f0e381f9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e36a00640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005940 tx=0x7f0e2400cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e24007bb0 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e24007e40 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e380cadf0 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.299+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2400f6b0 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.300+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e380cb3b0 con 0x7f0e3806f2e0 2024-09-17T04:06:53.303 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.301+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e38158d70 con 0x7f0e3806f2e0 2024-09-17T04:06:53.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.302+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0e2400f850 con 0x7f0e3806f2e0 2024-09-17T04:06:53.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.303+0000 7f0e2f400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.303+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f0e2409b380 con 0x7f0e3806f2e0 2024-09-17T04:06:53.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.303+0000 7f0e37400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.304 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.304+0000 7f0e37400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004630 tx=0x7f0e2800b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:53.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.307+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0e24014030 con 0x7f0e3806f2e0 2024-09-17T04:06:53.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.486+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e380ca8c0 con 0x7f0e18078b90 2024-09-17T04:06:53.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.491+0000 7f0e2f400640 1 -- 172.21.15.129:0/3235437406 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251135 (secure 0 0 0) 0x7f0e380ca8c0 con 0x7f0e18078b90 2024-09-17T04:06:53.495 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:53.496 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.495+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 msgr2=0x7f0e1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.495+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004630 tx=0x7f0e2800b2e0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.495+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 msgr2=0x7f0e381f89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.495+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f0e24005940 tx=0x7f0e2400cdd0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 shutdown_connections 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e380705a0 0x7f0e381f9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3806f2e0 0x7f0e381f89f0 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 --2- 172.21.15.129:0/3235437406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3806eed0 0x7f0e381f84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 >> 172.21.15.129:0/3235437406 conn(0x7f0e3806d410 msgr2=0x7f0e380ad500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 shutdown_connections 2024-09-17T04:06:53.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.496+0000 7f0e3e400640 1 -- 172.21.15.129:0/3235437406 wait complete. 2024-09-17T04:06:53.520 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":765,"stamp":"2024-09-17T04:06:52.592477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":135,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413796,"kb_used_data":25456,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754560924,"statfs":{"total":773094113280,"available":772670386176,"internally_reserved":0,"allocated":26066944,"data_stored":12184049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":397463088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.988601"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642545+0000","last_change":"2024-09-17T04:06:48.639552+0000","last_active":"2024-09-17T04:06:48.642545+0000","last_peered":"2024-09-17T04:06:48.642545+0000","last_clean":"2024-09-17T04:06:48.642545+0000","last_became_active":"2024-09-17T04:06:48.639000+0000","last_became_peered":"2024-09-17T04:06:48.639000+0000","last_unstale":"2024-09-17T04:06:48.642545+0000","last_undegraded":"2024-09-17T04:06:48.642545+0000","last_fullsized":"2024-09-17T04:06:48.642545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:43.840034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645686+0000","last_change":"2024-09-17T04:06:48.639479+0000","last_active":"2024-09-17T04:06:48.645686+0000","last_peered":"2024-09-17T04:06:48.645686+0000","last_clean":"2024-09-17T04:06:48.645686+0000","last_became_active":"2024-09-17T04:06:48.632914+0000","last_became_peered":"2024-09-17T04:06:48.632914+0000","last_unstale":"2024-09-17T04:06:48.645686+0000","last_undegraded":"2024-09-17T04:06:48.645686+0000","last_fullsized":"2024-09-17T04:06:48.645686+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:41.325611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656386+0000","last_change":"2024-09-17T04:06:48.653840+0000","last_active":"2024-09-17T04:06:48.656386+0000","last_peered":"2024-09-17T04:06:48.656386+0000","last_clean":"2024-09-17T04:06:48.656386+0000","last_became_active":"2024-09-17T04:06:48.653613+0000","last_became_peered":"2024-09-17T04:06:48.653613+0000","last_unstale":"2024-09-17T04:06:48.656386+0000","last_undegraded":"2024-09-17T04:06:48.656386+0000","last_fullsized":"2024-09-17T04:06:48.656386+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:48.801325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656192+0000","last_change":"2024-09-17T04:06:48.653715+0000","last_active":"2024-09-17T04:06:48.656192+0000","last_peered":"2024-09-17T04:06:48.656192+0000","last_clean":"2024-09-17T04:06:48.656192+0000","last_became_active":"2024-09-17T04:06:48.653119+0000","last_became_peered":"2024-09-17T04:06:48.653119+0000","last_unstale":"2024-09-17T04:06:48.656192+0000","last_undegraded":"2024-09-17T04:06:48.656192+0000","last_fullsized":"2024-09-17T04:06:48.656192+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:16.543513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650744+0000","last_change":"2024-09-17T04:06:48.645621+0000","last_active":"2024-09-17T04:06:48.650744+0000","last_peered":"2024-09-17T04:06:48.650744+0000","last_clean":"2024-09-17T04:06:48.650744+0000","last_became_active":"2024-09-17T04:06:48.644873+0000","last_became_peered":"2024-09-17T04:06:48.644873+0000","last_unstale":"2024-09-17T04:06:48.650744+0000","last_undegraded":"2024-09-17T04:06:48.650744+0000","last_fullsized":"2024-09-17T04:06:48.650744+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:06.459154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619151+0000","last_change":"2024-09-17T04:06:47.619151+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619151+0000","last_undegraded":"2024-09-17T04:06:47.619151+0000","last_fullsized":"2024-09-17T04:06:47.619151+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650635+0000","last_change":"2024-09-17T04:06:48.645454+0000","last_active":"2024-09-17T04:06:48.650635+0000","last_peered":"2024-09-17T04:06:48.650635+0000","last_clean":"2024-09-17T04:06:48.650635+0000","last_became_active":"2024-09-17T04:06:48.645060+0000","last_became_peered":"2024-09-17T04:06:48.645060+0000","last_unstale":"2024-09-17T04:06:48.650635+0000","last_undegraded":"2024-09-17T04:06:48.650635+0000","last_fullsized":"2024-09-17T04:06:48.650635+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:41.950353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637543+0000","last_change":"2024-09-17T04:06:48.636070+0000","last_active":"2024-09-17T04:06:48.637543+0000","last_peered":"2024-09-17T04:06:48.637543+0000","last_clean":"2024-09-17T04:06:48.637543+0000","last_became_active":"2024-09-17T04:06:48.635753+0000","last_became_peered":"2024-09-17T04:06:48.635753+0000","last_unstale":"2024-09-17T04:06:48.637543+0000","last_undegraded":"2024-09-17T04:06:48.637543+0000","last_fullsized":"2024-09-17T04:06:48.637543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:22.423358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645657+0000","last_change":"2024-09-17T04:06:48.640090+0000","last_active":"2024-09-17T04:06:48.645657+0000","last_peered":"2024-09-17T04:06:48.645657+0000","last_clean":"2024-09-17T04:06:48.645657+0000","last_became_active":"2024-09-17T04:06:48.635312+0000","last_became_peered":"2024-09-17T04:06:48.635312+0000","last_unstale":"2024-09-17T04:06:48.645657+0000","last_undegraded":"2024-09-17T04:06:48.645657+0000","last_fullsized":"2024-09-17T04:06:48.645657+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:06.548364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635377+0000","last_change":"2024-09-17T04:06:48.632182+0000","last_active":"2024-09-17T04:06:48.635377+0000","last_peered":"2024-09-17T04:06:48.635377+0000","last_clean":"2024-09-17T04:06:48.635377+0000","last_became_active":"2024-09-17T04:06:48.631841+0000","last_became_peered":"2024-09-17T04:06:48.631841+0000","last_unstale":"2024-09-17T04:06:48.635377+0000","last_undegraded":"2024-09-17T04:06:48.635377+0000","last_fullsized":"2024-09-17T04:06:48.635377+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:32.088154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637508+0000","last_change":"2024-09-17T04:06:48.635273+0000","last_active":"2024-09-17T04:06:48.637508+0000","last_peered":"2024-09-17T04:06:48.637508+0000","last_clean":"2024-09-17T04:06:48.637508+0000","last_became_active":"2024-09-17T04:06:48.634916+0000","last_became_peered":"2024-09-17T04:06:48.634916+0000","last_unstale":"2024-09-17T04:06:48.637508+0000","last_undegraded":"2024-09-17T04:06:48.637508+0000","last_fullsized":"2024-09-17T04:06:48.637508+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:17.777375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655592+0000","last_change":"2024-09-17T04:06:48.654714+0000","last_active":"2024-09-17T04:06:48.655592+0000","last_peered":"2024-09-17T04:06:48.655592+0000","last_clean":"2024-09-17T04:06:48.655592+0000","last_became_active":"2024-09-17T04:06:48.654168+0000","last_became_peered":"2024-09-17T04:06:48.654168+0000","last_unstale":"2024-09-17T04:06:48.655592+0000","last_undegraded":"2024-09-17T04:06:48.655592+0000","last_fullsized":"2024-09-17T04:06:48.655592+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:43.110635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629202+0000","last_change":"2024-09-17T04:06:47.629202+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629202+0000","last_undegraded":"2024-09-17T04:06:47.629202+0000","last_fullsized":"2024-09-17T04:06:47.629202+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638822+0000","last_change":"2024-09-17T04:06:48.635524+0000","last_active":"2024-09-17T04:06:48.638822+0000","last_peered":"2024-09-17T04:06:48.638822+0000","last_clean":"2024-09-17T04:06:48.638822+0000","last_became_active":"2024-09-17T04:06:48.634740+0000","last_became_peered":"2024-09-17T04:06:48.634740+0000","last_unstale":"2024-09-17T04:06:48.638822+0000","last_undegraded":"2024-09-17T04:06:48.638822+0000","last_fullsized":"2024-09-17T04:06:48.638822+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.745727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.648208+0000","last_change":"2024-09-17T04:06:48.639136+0000","last_active":"2024-09-17T04:06:48.648208+0000","last_peered":"2024-09-17T04:06:48.648208+0000","last_clean":"2024-09-17T04:06:48.648208+0000","last_became_active":"2024-09-17T04:06:48.638849+0000","last_became_peered":"2024-09-17T04:06:48.638849+0000","last_unstale":"2024-09-17T04:06:48.648208+0000","last_undegraded":"2024-09-17T04:06:48.648208+0000","last_fullsized":"2024-09-17T04:06:48.648208+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.361484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644287+0000","last_change":"2024-09-17T04:06:48.635932+0000","last_active":"2024-09-17T04:06:48.644287+0000","last_peered":"2024-09-17T04:06:48.644287+0000","last_clean":"2024-09-17T04:06:48.644287+0000","last_became_active":"2024-09-17T04:06:48.634277+0000","last_became_peered":"2024-09-17T04:06:48.634277+0000","last_unstale":"2024-09-17T04:06:48.644287+0000","last_undegraded":"2024-09-17T04:06:48.644287+0000","last_fullsized":"2024-09-17T04:06:48.644287+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:45.140616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629285+0000","last_change":"2024-09-17T04:06:47.629285+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629285+0000","last_undegraded":"2024-09-17T04:06:47.629285+0000","last_fullsized":"2024-09-17T04:06:47.629285+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.627532+0000","last_change":"2024-09-17T04:06:47.627532+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.627532+0000","last_undegraded":"2024-09-17T04:06:47.627532+0000","last_fullsized":"2024-09-17T04:06:47.627532+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645355+0000","last_change":"2024-09-17T04:06:48.643193+0000","last_active":"2024-09-17T04:06:48.645355+0000","last_peered":"2024-09-17T04:06:48.645355+0000","last_clean":"2024-09-17T04:06:48.645355+0000","last_became_active":"2024-09-17T04:06:48.641658+0000","last_became_peered":"2024-09-17T04:06:48.641658+0000","last_unstale":"2024-09-17T04:06:48.645355+0000","last_undegraded":"2024-09-17T04:06:48.645355+0000","last_fullsized":"2024-09-17T04:06:48.645355+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.874073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645968+0000","last_change":"2024-09-17T04:06:48.641052+0000","last_active":"2024-09-17T04:06:48.645968+0000","last_peered":"2024-09-17T04:06:48.645968+0000","last_clean":"2024-09-17T04:06:48.645968+0000","last_became_active":"2024-09-17T04:06:48.640727+0000","last_became_peered":"2024-09-17T04:06:48.640727+0000","last_unstale":"2024-09-17T04:06:48.645968+0000","last_undegraded":"2024-09-17T04:06:48.645968+0000","last_fullsized":"2024-09-17T04:06:48.645968+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:57.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644189+0000","last_change":"2024-09-17T04:06:48.638889+0000","last_active":"2024-09-17T04:06:48.644189+0000","last_peered":"2024-09-17T04:06:48.644189+0000","last_clean":"2024-09-17T04:06:48.644189+0000","last_became_active":"2024-09-17T04:06:48.638320+0000","last_became_peered":"2024-09-17T04:06:48.638320+0000","last_unstale":"2024-09-17T04:06:48.644189+0000","last_undegraded":"2024-09-17T04:06:48.644189+0000","last_fullsized":"2024-09-17T04:06:48.644189+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:19.054145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650679+0000","last_change":"2024-09-17T04:06:48.640395+0000","last_active":"2024-09-17T04:06:48.650679+0000","last_peered":"2024-09-17T04:06:48.650679+0000","last_clean":"2024-09-17T04:06:48.650679+0000","last_became_active":"2024-09-17T04:06:48.640063+0000","last_became_peered":"2024-09-17T04:06:48.640063+0000","last_unstale":"2024-09-17T04:06:48.650679+0000","last_undegraded":"2024-09-17T04:06:48.650679+0000","last_fullsized":"2024-09-17T04:06:48.650679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:19:30.759186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629478+0000","last_change":"2024-09-17T04:06:47.629478+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629478+0000","last_undegraded":"2024-09-17T04:06:47.629478+0000","last_fullsized":"2024-09-17T04:06:47.629478+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642754+0000","last_change":"2024-09-17T04:06:48.641161+0000","last_active":"2024-09-17T04:06:48.642754+0000","last_peered":"2024-09-17T04:06:48.642754+0000","last_clean":"2024-09-17T04:06:48.642754+0000","last_became_active":"2024-09-17T04:06:48.640811+0000","last_became_peered":"2024-09-17T04:06:48.640811+0000","last_unstale":"2024-09-17T04:06:48.642754+0000","last_undegraded":"2024-09-17T04:06:48.642754+0000","last_fullsized":"2024-09-17T04:06:48.642754+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.092195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628734+0000","last_change":"2024-09-17T04:06:47.628734+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628734+0000","last_undegraded":"2024-09-17T04:06:47.628734+0000","last_fullsized":"2024-09-17T04:06:47.628734+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651843+0000","last_change":"2024-09-17T04:06:48.646124+0000","last_active":"2024-09-17T04:06:48.651843+0000","last_peered":"2024-09-17T04:06:48.651843+0000","last_clean":"2024-09-17T04:06:48.651843+0000","last_became_active":"2024-09-17T04:06:48.645673+0000","last_became_peered":"2024-09-17T04:06:48.645673+0000","last_unstale":"2024-09-17T04:06:48.651843+0000","last_undegraded":"2024-09-17T04:06:48.651843+0000","last_fullsized":"2024-09-17T04:06:48.651843+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:26.644315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645666+0000","last_change":"2024-09-17T04:06:48.637878+0000","last_active":"2024-09-17T04:06:48.645666+0000","last_peered":"2024-09-17T04:06:48.645666+0000","last_clean":"2024-09-17T04:06:48.645666+0000","last_became_active":"2024-09-17T04:06:48.637580+0000","last_became_peered":"2024-09-17T04:06:48.637580+0000","last_unstale":"2024-09-17T04:06:48.645666+0000","last_undegraded":"2024-09-17T04:06:48.645666+0000","last_fullsized":"2024-09-17T04:06:48.645666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:38.576300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618364+0000","last_change":"2024-09-17T04:06:47.618364+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618364+0000","last_undegraded":"2024-09-17T04:06:47.618364+0000","last_fullsized":"2024-09-17T04:06:47.618364+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633778+0000","last_change":"2024-09-17T04:06:47.633778+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633778+0000","last_undegraded":"2024-09-17T04:06:47.633778+0000","last_fullsized":"2024-09-17T04:06:47.633778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645544+0000","last_change":"2024-09-17T04:06:48.643887+0000","last_active":"2024-09-17T04:06:48.645544+0000","last_peered":"2024-09-17T04:06:48.645544+0000","last_clean":"2024-09-17T04:06:48.645544+0000","last_became_active":"2024-09-17T04:06:48.643541+0000","last_became_peered":"2024-09-17T04:06:48.643541+0000","last_unstale":"2024-09-17T04:06:48.645544+0000","last_undegraded":"2024-09-17T04:06:48.645544+0000","last_fullsized":"2024-09-17T04:06:48.645544+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:51.740525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650895+0000","last_change":"2024-09-17T04:06:48.646487+0000","last_active":"2024-09-17T04:06:48.650895+0000","last_peered":"2024-09-17T04:06:48.650895+0000","last_clean":"2024-09-17T04:06:48.650895+0000","last_became_active":"2024-09-17T04:06:48.646200+0000","last_became_peered":"2024-09-17T04:06:48.646200+0000","last_unstale":"2024-09-17T04:06:48.650895+0000","last_undegraded":"2024-09-17T04:06:48.650895+0000","last_fullsized":"2024-09-17T04:06:48.650895+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.053314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645614+0000","last_change":"2024-09-17T04:06:48.638450+0000","last_active":"2024-09-17T04:06:48.645614+0000","last_peered":"2024-09-17T04:06:48.645614+0000","last_clean":"2024-09-17T04:06:48.645614+0000","last_became_active":"2024-09-17T04:06:48.638148+0000","last_became_peered":"2024-09-17T04:06:48.638148+0000","last_unstale":"2024-09-17T04:06:48.645614+0000","last_undegraded":"2024-09-17T04:06:48.645614+0000","last_fullsized":"2024-09-17T04:06:48.645614+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.182838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"246'4","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656184+0000","last_change":"2024-09-17T04:06:47.630685+0000","last_active":"2024-09-17T04:06:48.656184+0000","last_peered":"2024-09-17T04:06:48.656184+0000","last_clean":"2024-09-17T04:06:48.656184+0000","last_became_active":"2024-09-17T04:06:47.630367+0000","last_became_peered":"2024-09-17T04:06:47.630367+0000","last_unstale":"2024-09-17T04:06:48.656184+0000","last_undegraded":"2024-09-17T04:06:48.656184+0000","last_fullsized":"2024-09-17T04:06:48.656184+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:56.891603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640751+0000","last_change":"2024-09-17T04:06:48.636369+0000","last_active":"2024-09-17T04:06:48.640751+0000","last_peered":"2024-09-17T04:06:48.640751+0000","last_clean":"2024-09-17T04:06:48.640751+0000","last_became_active":"2024-09-17T04:06:48.636001+0000","last_became_peered":"2024-09-17T04:06:48.636001+0000","last_unstale":"2024-09-17T04:06:48.640751+0000","last_undegraded":"2024-09-17T04:06:48.640751+0000","last_fullsized":"2024-09-17T04:06:48.640751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:21.692754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642287+0000","last_change":"2024-09-17T04:06:47.640788+0000","last_active":"2024-09-17T04:06:47.642287+0000","last_peered":"2024-09-17T04:06:47.642287+0000","last_clean":"2024-09-17T04:06:47.642287+0000","last_became_active":"2024-09-17T04:06:47.640507+0000","last_became_peered":"2024-09-17T04:06:47.640507+0000","last_unstale":"2024-09-17T04:06:47.642287+0000","last_undegraded":"2024-09-17T04:06:47.642287+0000","last_fullsized":"2024-09-17T04:06:47.642287+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:24.521931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.625495+0000","last_change":"2024-09-17T04:06:47.625495+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.625495+0000","last_undegraded":"2024-09-17T04:06:47.625495+0000","last_fullsized":"2024-09-17T04:06:47.625495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642354+0000","last_change":"2024-09-17T04:06:47.640338+0000","last_active":"2024-09-17T04:06:47.642354+0000","last_peered":"2024-09-17T04:06:47.642354+0000","last_clean":"2024-09-17T04:06:47.642354+0000","last_became_active":"2024-09-17T04:06:47.639781+0000","last_became_peered":"2024-09-17T04:06:47.639781+0000","last_unstale":"2024-09-17T04:06:47.642354+0000","last_undegraded":"2024-09-17T04:06:47.642354+0000","last_fullsized":"2024-09-17T04:06:47.642354+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:20.160754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645958+0000","last_change":"2024-09-17T04:06:48.641644+0000","last_active":"2024-09-17T04:06:48.645958+0000","last_peered":"2024-09-17T04:06:48.645958+0000","last_clean":"2024-09-17T04:06:48.645958+0000","last_became_active":"2024-09-17T04:06:48.641054+0000","last_became_peered":"2024-09-17T04:06:48.641054+0000","last_unstale":"2024-09-17T04:06:48.645958+0000","last_undegraded":"2024-09-17T04:06:48.645958+0000","last_fullsized":"2024-09-17T04:06:48.645958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:50.540964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660581+0000","last_change":"2024-09-17T04:06:47.634854+0000","last_active":"2024-09-17T04:06:48.660581+0000","last_peered":"2024-09-17T04:06:48.660581+0000","last_clean":"2024-09-17T04:06:48.660581+0000","last_became_active":"2024-09-17T04:06:47.634556+0000","last_became_peered":"2024-09-17T04:06:47.634556+0000","last_unstale":"2024-09-17T04:06:48.660581+0000","last_undegraded":"2024-09-17T04:06:48.660581+0000","last_fullsized":"2024-09-17T04:06:48.660581+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:55:59.294589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637459+0000","last_change":"2024-09-17T04:06:48.633413+0000","last_active":"2024-09-17T04:06:48.637459+0000","last_peered":"2024-09-17T04:06:48.637459+0000","last_clean":"2024-09-17T04:06:48.637459+0000","last_became_active":"2024-09-17T04:06:48.632689+0000","last_became_peered":"2024-09-17T04:06:48.632689+0000","last_unstale":"2024-09-17T04:06:48.637459+0000","last_undegraded":"2024-09-17T04:06:48.637459+0000","last_fullsized":"2024-09-17T04:06:48.637459+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.165427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653343+0000","last_change":"2024-09-17T04:06:47.637532+0000","last_active":"2024-09-17T04:06:48.653343+0000","last_peered":"2024-09-17T04:06:48.653343+0000","last_clean":"2024-09-17T04:06:48.653343+0000","last_became_active":"2024-09-17T04:06:47.637150+0000","last_became_peered":"2024-09-17T04:06:47.637150+0000","last_unstale":"2024-09-17T04:06:48.653343+0000","last_undegraded":"2024-09-17T04:06:48.653343+0000","last_fullsized":"2024-09-17T04:06:48.653343+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:04.474254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.632926+0000","last_change":"2024-09-17T04:06:48.625119+0000","last_active":"2024-09-17T04:06:48.632926+0000","last_peered":"2024-09-17T04:06:48.632926+0000","last_clean":"2024-09-17T04:06:48.632926+0000","last_became_active":"2024-09-17T04:06:48.624823+0000","last_became_peered":"2024-09-17T04:06:48.624823+0000","last_unstale":"2024-09-17T04:06:48.632926+0000","last_undegraded":"2024-09-17T04:06:48.632926+0000","last_fullsized":"2024-09-17T04:06:48.632926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:21.840133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639235+0000","last_change":"2024-09-17T04:06:47.636767+0000","last_active":"2024-09-17T04:06:47.639235+0000","last_peered":"2024-09-17T04:06:47.639235+0000","last_clean":"2024-09-17T04:06:47.639235+0000","last_became_active":"2024-09-17T04:06:47.635877+0000","last_became_peered":"2024-09-17T04:06:47.635877+0000","last_unstale":"2024-09-17T04:06:47.639235+0000","last_undegraded":"2024-09-17T04:06:47.639235+0000","last_fullsized":"2024-09-17T04:06:47.639235+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633498+0000","last_change":"2024-09-17T04:06:47.633498+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633498+0000","last_undegraded":"2024-09-17T04:06:47.633498+0000","last_fullsized":"2024-09-17T04:06:47.633498+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630330+0000","last_change":"2024-09-17T04:06:47.641518+0000","last_active":"2024-09-17T04:06:48.630330+0000","last_peered":"2024-09-17T04:06:48.630330+0000","last_clean":"2024-09-17T04:06:48.630330+0000","last_became_active":"2024-09-17T04:06:47.641158+0000","last_became_peered":"2024-09-17T04:06:47.641158+0000","last_unstale":"2024-09-17T04:06:48.630330+0000","last_undegraded":"2024-09-17T04:06:48.630330+0000","last_fullsized":"2024-09-17T04:06:48.630330+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:43.208862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.616816+0000","last_change":"2024-09-17T04:06:47.616816+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.616816+0000","last_undegraded":"2024-09-17T04:06:47.616816+0000","last_fullsized":"2024-09-17T04:06:47.616816+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.638712+0000","last_change":"2024-09-17T04:06:47.637605+0000","last_active":"2024-09-17T04:06:47.638712+0000","last_peered":"2024-09-17T04:06:47.638712+0000","last_clean":"2024-09-17T04:06:47.638712+0000","last_became_active":"2024-09-17T04:06:47.637251+0000","last_became_peered":"2024-09-17T04:06:47.637251+0000","last_unstale":"2024-09-17T04:06:47.638712+0000","last_undegraded":"2024-09-17T04:06:47.638712+0000","last_fullsized":"2024-09-17T04:06:47.638712+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619494+0000","last_change":"2024-09-17T04:06:47.619494+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619494+0000","last_undegraded":"2024-09-17T04:06:47.619494+0000","last_fullsized":"2024-09-17T04:06:47.619494+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630182+0000","last_change":"2024-09-17T04:06:47.639900+0000","last_active":"2024-09-17T04:06:48.630182+0000","last_peered":"2024-09-17T04:06:48.630182+0000","last_clean":"2024-09-17T04:06:48.630182+0000","last_became_active":"2024-09-17T04:06:47.638771+0000","last_became_peered":"2024-09-17T04:06:47.638771+0000","last_unstale":"2024-09-17T04:06:48.630182+0000","last_undegraded":"2024-09-17T04:06:48.630182+0000","last_fullsized":"2024-09-17T04:06:48.630182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:42.102677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640781+0000","last_change":"2024-09-17T04:06:48.637971+0000","last_active":"2024-09-17T04:06:48.640781+0000","last_peered":"2024-09-17T04:06:48.640781+0000","last_clean":"2024-09-17T04:06:48.640781+0000","last_became_active":"2024-09-17T04:06:48.637655+0000","last_became_peered":"2024-09-17T04:06:48.637655+0000","last_unstale":"2024-09-17T04:06:48.640781+0000","last_undegraded":"2024-09-17T04:06:48.640781+0000","last_fullsized":"2024-09-17T04:06:48.640781+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:54.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624781+0000","last_change":"2024-09-17T04:06:47.628105+0000","last_active":"2024-09-17T04:06:48.624781+0000","last_peered":"2024-09-17T04:06:48.624781+0000","last_clean":"2024-09-17T04:06:48.624781+0000","last_became_active":"2024-09-17T04:06:47.627756+0000","last_became_peered":"2024-09-17T04:06:47.627756+0000","last_unstale":"2024-09-17T04:06:48.624781+0000","last_undegraded":"2024-09-17T04:06:48.624781+0000","last_fullsized":"2024-09-17T04:06:48.624781+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:38.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653352+0000","last_change":"2024-09-17T04:06:48.646524+0000","last_active":"2024-09-17T04:06:48.653352+0000","last_peered":"2024-09-17T04:06:48.653352+0000","last_clean":"2024-09-17T04:06:48.653352+0000","last_became_active":"2024-09-17T04:06:48.646148+0000","last_became_peered":"2024-09-17T04:06:48.646148+0000","last_unstale":"2024-09-17T04:06:48.653352+0000","last_undegraded":"2024-09-17T04:06:48.653352+0000","last_fullsized":"2024-09-17T04:06:48.653352+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:29:48.285920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"246'3","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.661105+0000","last_change":"2024-09-17T04:06:47.635826+0000","last_active":"2024-09-17T04:06:48.661105+0000","last_peered":"2024-09-17T04:06:48.661105+0000","last_clean":"2024-09-17T04:06:48.661105+0000","last_became_active":"2024-09-17T04:06:47.633375+0000","last_became_peered":"2024-09-17T04:06:47.633375+0000","last_unstale":"2024-09-17T04:06:48.661105+0000","last_undegraded":"2024-09-17T04:06:48.661105+0000","last_fullsized":"2024-09-17T04:06:48.661105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:13.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622935+0000","last_change":"2024-09-17T04:06:47.622935+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622935+0000","last_undegraded":"2024-09-17T04:06:47.622935+0000","last_fullsized":"2024-09-17T04:06:47.622935+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639789+0000","last_change":"2024-09-17T04:06:47.638185+0000","last_active":"2024-09-17T04:06:47.639789+0000","last_peered":"2024-09-17T04:06:47.639789+0000","last_clean":"2024-09-17T04:06:47.639789+0000","last_became_active":"2024-09-17T04:06:47.637416+0000","last_became_peered":"2024-09-17T04:06:47.637416+0000","last_unstale":"2024-09-17T04:06:47.639789+0000","last_undegraded":"2024-09-17T04:06:47.639789+0000","last_fullsized":"2024-09-17T04:06:47.639789+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649907+0000","last_change":"2024-09-17T04:06:48.647751+0000","last_active":"2024-09-17T04:06:48.649907+0000","last_peered":"2024-09-17T04:06:48.649907+0000","last_clean":"2024-09-17T04:06:48.649907+0000","last_became_active":"2024-09-17T04:06:48.647416+0000","last_became_peered":"2024-09-17T04:06:48.647416+0000","last_unstale":"2024-09-17T04:06:48.649907+0000","last_undegraded":"2024-09-17T04:06:48.649907+0000","last_fullsized":"2024-09-17T04:06:48.649907+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.537308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652112+0000","last_change":"2024-09-17T04:06:47.640722+0000","last_active":"2024-09-17T04:06:48.652112+0000","last_peered":"2024-09-17T04:06:48.652112+0000","last_clean":"2024-09-17T04:06:48.652112+0000","last_became_active":"2024-09-17T04:06:47.640406+0000","last_became_peered":"2024-09-17T04:06:47.640406+0000","last_unstale":"2024-09-17T04:06:48.652112+0000","last_undegraded":"2024-09-17T04:06:48.652112+0000","last_fullsized":"2024-09-17T04:06:48.652112+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:40.791049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.633929+0000","last_change":"2024-09-17T04:06:48.628835+0000","last_active":"2024-09-17T04:06:48.633929+0000","last_peered":"2024-09-17T04:06:48.633929+0000","last_clean":"2024-09-17T04:06:48.633929+0000","last_became_active":"2024-09-17T04:06:48.628468+0000","last_became_peered":"2024-09-17T04:06:48.628468+0000","last_unstale":"2024-09-17T04:06:48.633929+0000","last_undegraded":"2024-09-17T04:06:48.633929+0000","last_fullsized":"2024-09-17T04:06:48.633929+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:43.375277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637573+0000","last_change":"2024-09-17T04:06:47.629947+0000","last_active":"2024-09-17T04:06:48.637573+0000","last_peered":"2024-09-17T04:06:48.637573+0000","last_clean":"2024-09-17T04:06:48.637573+0000","last_became_active":"2024-09-17T04:06:47.628403+0000","last_became_peered":"2024-09-17T04:06:47.628403+0000","last_unstale":"2024-09-17T04:06:48.637573+0000","last_undegraded":"2024-09-17T04:06:48.637573+0000","last_fullsized":"2024-09-17T04:06:48.637573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:15.090695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652110+0000","last_change":"2024-09-17T04:06:48.650761+0000","last_active":"2024-09-17T04:06:48.652110+0000","last_peered":"2024-09-17T04:06:48.652110+0000","last_clean":"2024-09-17T04:06:48.652110+0000","last_became_active":"2024-09-17T04:06:48.650386+0000","last_became_peered":"2024-09-17T04:06:48.650386+0000","last_unstale":"2024-09-17T04:06:48.652110+0000","last_undegraded":"2024-09-17T04:06:48.652110+0000","last_fullsized":"2024-09-17T04:06:48.652110+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:57.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639196+0000","last_change":"2024-09-17T04:06:47.637094+0000","last_active":"2024-09-17T04:06:47.639196+0000","last_peered":"2024-09-17T04:06:47.639196+0000","last_clean":"2024-09-17T04:06:47.639196+0000","last_became_active":"2024-09-17T04:06:47.636787+0000","last_became_peered":"2024-09-17T04:06:47.636787+0000","last_unstale":"2024-09-17T04:06:47.639196+0000","last_undegraded":"2024-09-17T04:06:47.639196+0000","last_fullsized":"2024-09-17T04:06:47.639196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644926+0000","last_change":"2024-09-17T04:06:48.642186+0000","last_active":"2024-09-17T04:06:48.644926+0000","last_peered":"2024-09-17T04:06:48.644926+0000","last_clean":"2024-09-17T04:06:48.644926+0000","last_became_active":"2024-09-17T04:06:48.640270+0000","last_became_peered":"2024-09-17T04:06:48.640270+0000","last_unstale":"2024-09-17T04:06:48.644926+0000","last_undegraded":"2024-09-17T04:06:48.644926+0000","last_fullsized":"2024-09-17T04:06:48.644926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:57.596302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"246'8","reported_seq":21,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.665782+0000","last_change":"2024-09-17T04:06:47.639284+0000","last_active":"2024-09-17T04:06:48.665782+0000","last_peered":"2024-09-17T04:06:48.665782+0000","last_clean":"2024-09-17T04:06:48.665782+0000","last_became_active":"2024-09-17T04:06:47.638987+0000","last_became_peered":"2024-09-17T04:06:47.638987+0000","last_unstale":"2024-09-17T04:06:48.665782+0000","last_undegraded":"2024-09-17T04:06:48.665782+0000","last_fullsized":"2024-09-17T04:06:48.665782+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:25.636357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650612+0000","last_change":"2024-09-17T04:06:48.640025+0000","last_active":"2024-09-17T04:06:48.650612+0000","last_peered":"2024-09-17T04:06:48.650612+0000","last_clean":"2024-09-17T04:06:48.650612+0000","last_became_active":"2024-09-17T04:06:48.639713+0000","last_became_peered":"2024-09-17T04:06:48.639713+0000","last_unstale":"2024-09-17T04:06:48.650612+0000","last_undegraded":"2024-09-17T04:06:48.650612+0000","last_fullsized":"2024-09-17T04:06:48.650612+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:46.136192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650031+0000","last_change":"2024-09-17T04:06:47.627885+0000","last_active":"2024-09-17T04:06:48.650031+0000","last_peered":"2024-09-17T04:06:48.650031+0000","last_clean":"2024-09-17T04:06:48.650031+0000","last_became_active":"2024-09-17T04:06:47.627314+0000","last_became_peered":"2024-09-17T04:06:47.627314+0000","last_unstale":"2024-09-17T04:06:48.650031+0000","last_undegraded":"2024-09-17T04:06:48.650031+0000","last_fullsized":"2024-09-17T04:06:48.650031+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:32.996351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646180+0000","last_change":"2024-09-17T04:06:48.640039+0000","last_active":"2024-09-17T04:06:48.646180+0000","last_peered":"2024-09-17T04:06:48.646180+0000","last_clean":"2024-09-17T04:06:48.646180+0000","last_became_active":"2024-09-17T04:06:48.639062+0000","last_became_peered":"2024-09-17T04:06:48.639062+0000","last_unstale":"2024-09-17T04:06:48.646180+0000","last_undegraded":"2024-09-17T04:06:48.646180+0000","last_fullsized":"2024-09-17T04:06:48.646180+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:39.282547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655638+0000","last_change":"2024-09-17T04:06:47.626021+0000","last_active":"2024-09-17T04:06:48.655638+0000","last_peered":"2024-09-17T04:06:48.655638+0000","last_clean":"2024-09-17T04:06:48.655638+0000","last_became_active":"2024-09-17T04:06:47.625353+0000","last_became_peered":"2024-09-17T04:06:47.625353+0000","last_unstale":"2024-09-17T04:06:48.655638+0000","last_undegraded":"2024-09-17T04:06:48.655638+0000","last_fullsized":"2024-09-17T04:06:48.655638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622812+0000","last_change":"2024-09-17T04:06:47.622812+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622812+0000","last_undegraded":"2024-09-17T04:06:47.622812+0000","last_fullsized":"2024-09-17T04:06:47.622812+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.632014+0000","last_change":"2024-09-17T04:06:47.630827+0000","last_active":"2024-09-17T04:06:47.632014+0000","last_peered":"2024-09-17T04:06:47.632014+0000","last_clean":"2024-09-17T04:06:47.632014+0000","last_became_active":"2024-09-17T04:06:47.630146+0000","last_became_peered":"2024-09-17T04:06:47.630146+0000","last_unstale":"2024-09-17T04:06:47.632014+0000","last_undegraded":"2024-09-17T04:06:47.632014+0000","last_fullsized":"2024-09-17T04:06:47.632014+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656054+0000","last_change":"2024-09-17T04:06:48.655091+0000","last_active":"2024-09-17T04:06:48.656054+0000","last_peered":"2024-09-17T04:06:48.656054+0000","last_clean":"2024-09-17T04:06:48.656054+0000","last_became_active":"2024-09-17T04:06:48.654782+0000","last_became_peered":"2024-09-17T04:06:48.654782+0000","last_unstale":"2024-09-17T04:06:48.656054+0000","last_undegraded":"2024-09-17T04:06:48.656054+0000","last_fullsized":"2024-09-17T04:06:48.656054+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:06.889990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642059+0000","last_change":"2024-09-17T04:06:47.639967+0000","last_active":"2024-09-17T04:06:47.642059+0000","last_peered":"2024-09-17T04:06:47.642059+0000","last_clean":"2024-09-17T04:06:47.642059+0000","last_became_active":"2024-09-17T04:06:47.639606+0000","last_became_peered":"2024-09-17T04:06:47.639606+0000","last_unstale":"2024-09-17T04:06:47.642059+0000","last_undegraded":"2024-09-17T04:06:47.642059+0000","last_fullsized":"2024-09-17T04:06:47.642059+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:36.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649830+0000","last_change":"2024-09-17T04:06:48.642222+0000","last_active":"2024-09-17T04:06:48.649830+0000","last_peered":"2024-09-17T04:06:48.649830+0000","last_clean":"2024-09-17T04:06:48.649830+0000","last_became_active":"2024-09-17T04:06:48.641845+0000","last_became_peered":"2024-09-17T04:06:48.641845+0000","last_unstale":"2024-09-17T04:06:48.649830+0000","last_undegraded":"2024-09-17T04:06:48.649830+0000","last_fullsized":"2024-09-17T04:06:48.649830+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:51.088460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.659677+0000","last_change":"2024-09-17T04:06:47.635036+0000","last_active":"2024-09-17T04:06:48.659677+0000","last_peered":"2024-09-17T04:06:48.659677+0000","last_clean":"2024-09-17T04:06:48.659677+0000","last_became_active":"2024-09-17T04:06:47.634721+0000","last_became_peered":"2024-09-17T04:06:47.634721+0000","last_unstale":"2024-09-17T04:06:48.659677+0000","last_undegraded":"2024-09-17T04:06:48.659677+0000","last_fullsized":"2024-09-17T04:06:48.659677+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:48.022710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649772+0000","last_change":"2024-09-17T04:06:48.647597+0000","last_active":"2024-09-17T04:06:48.649772+0000","last_peered":"2024-09-17T04:06:48.649772+0000","last_clean":"2024-09-17T04:06:48.649772+0000","last_became_active":"2024-09-17T04:06:48.647265+0000","last_became_peered":"2024-09-17T04:06:48.647265+0000","last_unstale":"2024-09-17T04:06:48.649772+0000","last_undegraded":"2024-09-17T04:06:48.649772+0000","last_fullsized":"2024-09-17T04:06:48.649772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:31.880487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.621482+0000","last_change":"2024-09-17T04:06:47.627934+0000","last_active":"2024-09-17T04:06:48.621482+0000","last_peered":"2024-09-17T04:06:48.621482+0000","last_clean":"2024-09-17T04:06:48.621482+0000","last_became_active":"2024-09-17T04:06:47.627651+0000","last_became_peered":"2024-09-17T04:06:47.627651+0000","last_unstale":"2024-09-17T04:06:48.621482+0000","last_undegraded":"2024-09-17T04:06:48.621482+0000","last_fullsized":"2024-09-17T04:06:48.621482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:28.266741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635500+0000","last_change":"2024-09-17T04:06:48.630489+0000","last_active":"2024-09-17T04:06:48.635500+0000","last_peered":"2024-09-17T04:06:48.635500+0000","last_clean":"2024-09-17T04:06:48.635500+0000","last_became_active":"2024-09-17T04:06:48.630199+0000","last_became_peered":"2024-09-17T04:06:48.630199+0000","last_unstale":"2024-09-17T04:06:48.635500+0000","last_undegraded":"2024-09-17T04:06:48.635500+0000","last_fullsized":"2024-09-17T04:06:48.635500+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:49.945070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.637806+0000","last_change":"2024-09-17T04:06:47.636455+0000","last_active":"2024-09-17T04:06:47.637806+0000","last_peered":"2024-09-17T04:06:47.637806+0000","last_clean":"2024-09-17T04:06:47.637806+0000","last_became_active":"2024-09-17T04:06:47.636161+0000","last_became_peered":"2024-09-17T04:06:47.636161+0000","last_unstale":"2024-09-17T04:06:47.637806+0000","last_undegraded":"2024-09-17T04:06:47.637806+0000","last_fullsized":"2024-09-17T04:06:47.637806+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630666+0000","last_change":"2024-09-17T04:06:48.626073+0000","last_active":"2024-09-17T04:06:48.630666+0000","last_peered":"2024-09-17T04:06:48.630666+0000","last_clean":"2024-09-17T04:06:48.630666+0000","last_became_active":"2024-09-17T04:06:48.624659+0000","last_became_peered":"2024-09-17T04:06:48.624659+0000","last_unstale":"2024-09-17T04:06:48.630666+0000","last_undegraded":"2024-09-17T04:06:48.630666+0000","last_fullsized":"2024-09-17T04:06:48.630666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:31.519896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.622226+0000","last_change":"2024-09-17T04:06:47.631678+0000","last_active":"2024-09-17T04:06:48.622226+0000","last_peered":"2024-09-17T04:06:48.622226+0000","last_clean":"2024-09-17T04:06:48.622226+0000","last_became_active":"2024-09-17T04:06:47.629357+0000","last_became_peered":"2024-09-17T04:06:47.629357+0000","last_unstale":"2024-09-17T04:06:48.622226+0000","last_undegraded":"2024-09-17T04:06:48.622226+0000","last_fullsized":"2024-09-17T04:06:48.622226+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:58.960464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646523+0000","last_change":"2024-09-17T04:06:48.641309+0000","last_active":"2024-09-17T04:06:48.646523+0000","last_peered":"2024-09-17T04:06:48.646523+0000","last_clean":"2024-09-17T04:06:48.646523+0000","last_became_active":"2024-09-17T04:06:48.640780+0000","last_became_peered":"2024-09-17T04:06:48.640780+0000","last_unstale":"2024-09-17T04:06:48.646523+0000","last_undegraded":"2024-09-17T04:06:48.646523+0000","last_fullsized":"2024-09-17T04:06:48.646523+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:38.559159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660451+0000","last_change":"2024-09-17T04:06:47.634843+0000","last_active":"2024-09-17T04:06:48.660451+0000","last_peered":"2024-09-17T04:06:48.660451+0000","last_clean":"2024-09-17T04:06:48.660451+0000","last_became_active":"2024-09-17T04:06:47.634529+0000","last_became_peered":"2024-09-17T04:06:47.634529+0000","last_unstale":"2024-09-17T04:06:48.660451+0000","last_undegraded":"2024-09-17T04:06:48.660451+0000","last_fullsized":"2024-09-17T04:06:48.660451+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:24.241367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628250+0000","last_change":"2024-09-17T04:06:47.628250+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628250+0000","last_undegraded":"2024-09-17T04:06:47.628250+0000","last_fullsized":"2024-09-17T04:06:47.628250+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.619096+0000","last_change":"2024-09-17T04:06:47.635594+0000","last_active":"2024-09-17T04:06:48.619096+0000","last_peered":"2024-09-17T04:06:48.619096+0000","last_clean":"2024-09-17T04:06:48.619096+0000","last_became_active":"2024-09-17T04:06:47.633247+0000","last_became_peered":"2024-09-17T04:06:47.633247+0000","last_unstale":"2024-09-17T04:06:48.619096+0000","last_undegraded":"2024-09-17T04:06:48.619096+0000","last_fullsized":"2024-09-17T04:06:48.619096+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:29.409434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656026+0000","last_change":"2024-09-17T04:06:48.654761+0000","last_active":"2024-09-17T04:06:48.656026+0000","last_peered":"2024-09-17T04:06:48.656026+0000","last_clean":"2024-09-17T04:06:48.656026+0000","last_became_active":"2024-09-17T04:06:48.654228+0000","last_became_peered":"2024-09-17T04:06:48.654228+0000","last_unstale":"2024-09-17T04:06:48.656026+0000","last_undegraded":"2024-09-17T04:06:48.656026+0000","last_fullsized":"2024-09-17T04:06:48.656026+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:46.467512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650149+0000","last_change":"2024-09-17T04:06:47.631080+0000","last_active":"2024-09-17T04:06:48.650149+0000","last_peered":"2024-09-17T04:06:48.650149+0000","last_clean":"2024-09-17T04:06:48.650149+0000","last_became_active":"2024-09-17T04:06:47.630810+0000","last_became_peered":"2024-09-17T04:06:47.630810+0000","last_unstale":"2024-09-17T04:06:48.650149+0000","last_undegraded":"2024-09-17T04:06:48.650149+0000","last_fullsized":"2024-09-17T04:06:48.650149+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.601411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644148+0000","last_change":"2024-09-17T04:06:48.640563+0000","last_active":"2024-09-17T04:06:48.644148+0000","last_peered":"2024-09-17T04:06:48.644148+0000","last_clean":"2024-09-17T04:06:48.644148+0000","last_became_active":"2024-09-17T04:06:48.640193+0000","last_became_peered":"2024-09-17T04:06:48.640193+0000","last_unstale":"2024-09-17T04:06:48.644148+0000","last_undegraded":"2024-09-17T04:06:48.644148+0000","last_fullsized":"2024-09-17T04:06:48.644148+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:43.623529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631108+0000","last_change":"2024-09-17T04:06:47.640808+0000","last_active":"2024-09-17T04:06:48.631108+0000","last_peered":"2024-09-17T04:06:48.631108+0000","last_clean":"2024-09-17T04:06:48.631108+0000","last_became_active":"2024-09-17T04:06:47.639846+0000","last_became_peered":"2024-09-17T04:06:47.639846+0000","last_unstale":"2024-09-17T04:06:48.631108+0000","last_undegraded":"2024-09-17T04:06:48.631108+0000","last_fullsized":"2024-09-17T04:06:48.631108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:15.837166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618723+0000","last_change":"2024-09-17T04:06:47.618723+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618723+0000","last_undegraded":"2024-09-17T04:06:47.618723+0000","last_fullsized":"2024-09-17T04:06:47.618723+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630437+0000","last_change":"2024-09-17T04:06:47.639290+0000","last_active":"2024-09-17T04:06:48.630437+0000","last_peered":"2024-09-17T04:06:48.630437+0000","last_clean":"2024-09-17T04:06:48.630437+0000","last_became_active":"2024-09-17T04:06:47.638370+0000","last_became_peered":"2024-09-17T04:06:47.638370+0000","last_unstale":"2024-09-17T04:06:48.630437+0000","last_undegraded":"2024-09-17T04:06:48.630437+0000","last_fullsized":"2024-09-17T04:06:48.630437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:17.818126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651574+0000","last_change":"2024-09-17T04:06:48.645144+0000","last_active":"2024-09-17T04:06:48.651574+0000","last_peered":"2024-09-17T04:06:48.651574+0000","last_clean":"2024-09-17T04:06:48.651574+0000","last_became_active":"2024-09-17T04:06:48.644506+0000","last_became_peered":"2024-09-17T04:06:48.644506+0000","last_unstale":"2024-09-17T04:06:48.651574+0000","last_undegraded":"2024-09-17T04:06:48.651574+0000","last_fullsized":"2024-09-17T04:06:48.651574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:58.415646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638296+0000","last_change":"2024-09-17T04:06:47.637796+0000","last_active":"2024-09-17T04:06:48.638296+0000","last_peered":"2024-09-17T04:06:48.638296+0000","last_clean":"2024-09-17T04:06:48.638296+0000","last_became_active":"2024-09-17T04:06:47.637473+0000","last_became_peered":"2024-09-17T04:06:47.637473+0000","last_unstale":"2024-09-17T04:06:48.638296+0000","last_undegraded":"2024-09-17T04:06:48.638296+0000","last_fullsized":"2024-09-17T04:06:48.638296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:35.426948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635294+0000","last_change":"2024-09-17T04:06:48.626386+0000","last_active":"2024-09-17T04:06:48.635294+0000","last_peered":"2024-09-17T04:06:48.635294+0000","last_clean":"2024-09-17T04:06:48.635294+0000","last_became_active":"2024-09-17T04:06:48.623872+0000","last_became_peered":"2024-09-17T04:06:48.623872+0000","last_unstale":"2024-09-17T04:06:48.635294+0000","last_undegraded":"2024-09-17T04:06:48.635294+0000","last_fullsized":"2024-09-17T04:06:48.635294+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.050814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624231+0000","last_change":"2024-09-17T04:06:47.632594+0000","last_active":"2024-09-17T04:06:48.624231+0000","last_peered":"2024-09-17T04:06:48.624231+0000","last_clean":"2024-09-17T04:06:48.624231+0000","last_became_active":"2024-09-17T04:06:47.632229+0000","last_became_peered":"2024-09-17T04:06:47.632229+0000","last_unstale":"2024-09-17T04:06:48.624231+0000","last_undegraded":"2024-09-17T04:06:48.624231+0000","last_fullsized":"2024-09-17T04:06:48.624231+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:10.174881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650605+0000","last_change":"2024-09-17T04:06:48.640651+0000","last_active":"2024-09-17T04:06:48.650605+0000","last_peered":"2024-09-17T04:06:48.650605+0000","last_clean":"2024-09-17T04:06:48.650605+0000","last_became_active":"2024-09-17T04:06:48.639761+0000","last_became_peered":"2024-09-17T04:06:48.639761+0000","last_unstale":"2024-09-17T04:06:48.650605+0000","last_undegraded":"2024-09-17T04:06:48.650605+0000","last_fullsized":"2024-09-17T04:06:48.650605+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:53.798909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638592+0000","last_change":"2024-09-17T04:06:47.639117+0000","last_active":"2024-09-17T04:06:48.638592+0000","last_peered":"2024-09-17T04:06:48.638592+0000","last_clean":"2024-09-17T04:06:48.638592+0000","last_became_active":"2024-09-17T04:06:47.638778+0000","last_became_peered":"2024-09-17T04:06:47.638778+0000","last_unstale":"2024-09-17T04:06:48.638592+0000","last_undegraded":"2024-09-17T04:06:48.638592+0000","last_fullsized":"2024-09-17T04:06:48.638592+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:42.527414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653501+0000","last_change":"2024-09-17T04:06:48.648394+0000","last_active":"2024-09-17T04:06:48.653501+0000","last_peered":"2024-09-17T04:06:48.653501+0000","last_clean":"2024-09-17T04:06:48.653501+0000","last_became_active":"2024-09-17T04:06:48.647962+0000","last_became_peered":"2024-09-17T04:06:48.647962+0000","last_unstale":"2024-09-17T04:06:48.653501+0000","last_undegraded":"2024-09-17T04:06:48.653501+0000","last_fullsized":"2024-09-17T04:06:48.653501+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:16.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":525,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631126+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:48.631126+0000","last_peered":"2024-09-17T04:06:48.631126+0000","last_clean":"2024-09-17T04:06:48.631126+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:48.631126+0000","last_undegraded":"2024-09-17T04:06:48.631126+0000","last_fullsized":"2024-09-17T04:06:48.631126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509651,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509654,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34248,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337592,"statfs":{"total":96636764160,"available":96601694208,"internally_reserved":0,"allocated":3129344,"data_stored":1380978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56424,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315416,"statfs":{"total":96636764160,"available":96578985984,"internally_reserved":0,"allocated":3100672,"data_stored":1372501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56444,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315396,"statfs":{"total":96636764160,"available":96578965504,"internally_reserved":0,"allocated":3117056,"data_stored":1379858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509653,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56448,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315392,"statfs":{"total":96636764160,"available":96578961408,"internally_reserved":0,"allocated":3121152,"data_stored":1379400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509653,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3121152,"data_stored":1379332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542355,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52256,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319584,"statfs":{"total":96636764160,"available":96583254016,"internally_reserved":0,"allocated":3088384,"data_stored":1366242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:53.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 -- 172.21.15.129:0/1402026369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140ce170 msgr2=0x7f34140cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 --2- 172.21.15.129:0/1402026369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140ce170 0x7f34140cbbf0 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f3414072360 tx=0x7f340402a560 comp rx=0 tx=0).stop 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 -- 172.21.15.129:0/1402026369 shutdown_connections 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 --2- 172.21.15.129:0/1402026369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140b39c0 0x7f341406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 --2- 172.21.15.129:0/1402026369 >> v2:172.21.15.181:3300/0 conn(0x7f34140b2950 0x7f34140b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 --2- 172.21.15.129:0/1402026369 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140ce170 0x7f34140cbbf0 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 -- 172.21.15.129:0/1402026369 >> 172.21.15.129:0/1402026369 conn(0x7f341406d410 msgr2=0x7f34140b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 -- 172.21.15.129:0/1402026369 shutdown_connections 2024-09-17T04:06:53.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.780+0000 7f3418c00640 1 -- 172.21.15.129:0/1402026369 wait complete. 2024-09-17T04:06:53.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.782+0000 7f3418c00640 1 Processor -- start 2024-09-17T04:06:53.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.783+0000 7f3418c00640 1 -- start start 2024-09-17T04:06:53.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34140b2950 0x7f3414099990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 0x7f3414155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.785 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3414070850 con 0x7f34140b39c0 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f34140706d0 con 0x7f34140b2950 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3418c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f341406f410 con 0x7f34140ce170 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3411400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3411400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52206/0 (socket says 172.21.15.129:52206) 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3411400640 1 -- 172.21.15.129:0/1494010895 learned_addr learned my addr 172.21.15.129:0/1494010895 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3411e00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34140b2950 0x7f3414099990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.784+0000 7f3412800640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 0x7f3414155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 -- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 msgr2=0x7f3414155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 0x7f3414155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 -- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34140b2950 msgr2=0x7f3414099990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34140b2950 0x7f3414099990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 -- 172.21.15.129:0/1494010895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34141f6a70 con 0x7f34140b39c0 2024-09-17T04:06:53.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3412800640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 0x7f3414155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3411400640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f340000d650 tx=0x7f340000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400017070 con 0x7f34140b39c0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3400004310 con 0x7f34140b39c0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34141fa000 con 0x7f34140b39c0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3400005280 con 0x7f34140b39c0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.785+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34141f9520 con 0x7f34140b39c0 2024-09-17T04:06:53.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.786+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341406f8e0 con 0x7f34140b39c0 2024-09-17T04:06:53.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.788+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3400005420 con 0x7f34140b39c0 2024-09-17T04:06:53.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.789+0000 7f340a000640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 0x7f33f407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:53.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.789+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f340009aed0 con 0x7f34140b39c0 2024-09-17T04:06:53.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.790+0000 7f3411e00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 0x7f33f407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:53.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.791+0000 7f3411e00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 0x7f33f407b0d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f3404033000 tx=0x7f3404004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:53.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.794+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3400020020 con 0x7f34140b39c0 2024-09-17T04:06:53.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.974+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f34141673b0 con 0x7f33f4078bc0 2024-09-17T04:06:53.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.979+0000 7f340a000640 1 -- 172.21.15.129:0/1494010895 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251135 (secure 0 0 0) 0x7f3404005550 con 0x7f33f4078bc0 2024-09-17T04:06:53.982 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:53.983 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 msgr2=0x7f33f407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 0x7f33f407b0d0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f3404033000 tx=0x7f3404004730 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 msgr2=0x7f3414166e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f340000d650 tx=0x7f340000db10 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 shutdown_connections 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f33f4078bc0 0x7f33f407b0d0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f34140ce170 0x7f3414155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f34140b39c0 0x7f3414166e70 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 --2- 172.21.15.129:0/1494010895 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f34140b2950 0x7f3414099990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:53.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 >> 172.21.15.129:0/1494010895 conn(0x7f341406d410 msgr2=0x7f34140b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 shutdown_connections 2024-09-17T04:06:53.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:53.983+0000 7f3418c00640 1 -- 172.21.15.129:0/1494010895 wait complete. 2024-09-17T04:06:54.009 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":765,"stamp":"2024-09-17T04:06:52.592477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":135,"num_write_kb":1638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":413796,"kb_used_data":25456,"kb_used_omap":12,"kb_used_meta":388147,"kb_avail":754560924,"statfs":{"total":773094113280,"available":772670386176,"internally_reserved":0,"allocated":26066944,"data_stored":12184049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":397463088},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.988601"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642545+0000","last_change":"2024-09-17T04:06:48.639552+0000","last_active":"2024-09-17T04:06:48.642545+0000","last_peered":"2024-09-17T04:06:48.642545+0000","last_clean":"2024-09-17T04:06:48.642545+0000","last_became_active":"2024-09-17T04:06:48.639000+0000","last_became_peered":"2024-09-17T04:06:48.639000+0000","last_unstale":"2024-09-17T04:06:48.642545+0000","last_undegraded":"2024-09-17T04:06:48.642545+0000","last_fullsized":"2024-09-17T04:06:48.642545+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:43.840034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645686+0000","last_change":"2024-09-17T04:06:48.639479+0000","last_active":"2024-09-17T04:06:48.645686+0000","last_peered":"2024-09-17T04:06:48.645686+0000","last_clean":"2024-09-17T04:06:48.645686+0000","last_became_active":"2024-09-17T04:06:48.632914+0000","last_became_peered":"2024-09-17T04:06:48.632914+0000","last_unstale":"2024-09-17T04:06:48.645686+0000","last_undegraded":"2024-09-17T04:06:48.645686+0000","last_fullsized":"2024-09-17T04:06:48.645686+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:41.325611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656386+0000","last_change":"2024-09-17T04:06:48.653840+0000","last_active":"2024-09-17T04:06:48.656386+0000","last_peered":"2024-09-17T04:06:48.656386+0000","last_clean":"2024-09-17T04:06:48.656386+0000","last_became_active":"2024-09-17T04:06:48.653613+0000","last_became_peered":"2024-09-17T04:06:48.653613+0000","last_unstale":"2024-09-17T04:06:48.656386+0000","last_undegraded":"2024-09-17T04:06:48.656386+0000","last_fullsized":"2024-09-17T04:06:48.656386+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:48.801325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656192+0000","last_change":"2024-09-17T04:06:48.653715+0000","last_active":"2024-09-17T04:06:48.656192+0000","last_peered":"2024-09-17T04:06:48.656192+0000","last_clean":"2024-09-17T04:06:48.656192+0000","last_became_active":"2024-09-17T04:06:48.653119+0000","last_became_peered":"2024-09-17T04:06:48.653119+0000","last_unstale":"2024-09-17T04:06:48.656192+0000","last_undegraded":"2024-09-17T04:06:48.656192+0000","last_fullsized":"2024-09-17T04:06:48.656192+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:16.543513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650744+0000","last_change":"2024-09-17T04:06:48.645621+0000","last_active":"2024-09-17T04:06:48.650744+0000","last_peered":"2024-09-17T04:06:48.650744+0000","last_clean":"2024-09-17T04:06:48.650744+0000","last_became_active":"2024-09-17T04:06:48.644873+0000","last_became_peered":"2024-09-17T04:06:48.644873+0000","last_unstale":"2024-09-17T04:06:48.650744+0000","last_undegraded":"2024-09-17T04:06:48.650744+0000","last_fullsized":"2024-09-17T04:06:48.650744+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:06.459154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619151+0000","last_change":"2024-09-17T04:06:47.619151+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619151+0000","last_undegraded":"2024-09-17T04:06:47.619151+0000","last_fullsized":"2024-09-17T04:06:47.619151+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650635+0000","last_change":"2024-09-17T04:06:48.645454+0000","last_active":"2024-09-17T04:06:48.650635+0000","last_peered":"2024-09-17T04:06:48.650635+0000","last_clean":"2024-09-17T04:06:48.650635+0000","last_became_active":"2024-09-17T04:06:48.645060+0000","last_became_peered":"2024-09-17T04:06:48.645060+0000","last_unstale":"2024-09-17T04:06:48.650635+0000","last_undegraded":"2024-09-17T04:06:48.650635+0000","last_fullsized":"2024-09-17T04:06:48.650635+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:13:41.950353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637543+0000","last_change":"2024-09-17T04:06:48.636070+0000","last_active":"2024-09-17T04:06:48.637543+0000","last_peered":"2024-09-17T04:06:48.637543+0000","last_clean":"2024-09-17T04:06:48.637543+0000","last_became_active":"2024-09-17T04:06:48.635753+0000","last_became_peered":"2024-09-17T04:06:48.635753+0000","last_unstale":"2024-09-17T04:06:48.637543+0000","last_undegraded":"2024-09-17T04:06:48.637543+0000","last_fullsized":"2024-09-17T04:06:48.637543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:22.423358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645657+0000","last_change":"2024-09-17T04:06:48.640090+0000","last_active":"2024-09-17T04:06:48.645657+0000","last_peered":"2024-09-17T04:06:48.645657+0000","last_clean":"2024-09-17T04:06:48.645657+0000","last_became_active":"2024-09-17T04:06:48.635312+0000","last_became_peered":"2024-09-17T04:06:48.635312+0000","last_unstale":"2024-09-17T04:06:48.645657+0000","last_undegraded":"2024-09-17T04:06:48.645657+0000","last_fullsized":"2024-09-17T04:06:48.645657+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:06.548364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635377+0000","last_change":"2024-09-17T04:06:48.632182+0000","last_active":"2024-09-17T04:06:48.635377+0000","last_peered":"2024-09-17T04:06:48.635377+0000","last_clean":"2024-09-17T04:06:48.635377+0000","last_became_active":"2024-09-17T04:06:48.631841+0000","last_became_peered":"2024-09-17T04:06:48.631841+0000","last_unstale":"2024-09-17T04:06:48.635377+0000","last_undegraded":"2024-09-17T04:06:48.635377+0000","last_fullsized":"2024-09-17T04:06:48.635377+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:37:32.088154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637508+0000","last_change":"2024-09-17T04:06:48.635273+0000","last_active":"2024-09-17T04:06:48.637508+0000","last_peered":"2024-09-17T04:06:48.637508+0000","last_clean":"2024-09-17T04:06:48.637508+0000","last_became_active":"2024-09-17T04:06:48.634916+0000","last_became_peered":"2024-09-17T04:06:48.634916+0000","last_unstale":"2024-09-17T04:06:48.637508+0000","last_undegraded":"2024-09-17T04:06:48.637508+0000","last_fullsized":"2024-09-17T04:06:48.637508+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:45:17.777375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655592+0000","last_change":"2024-09-17T04:06:48.654714+0000","last_active":"2024-09-17T04:06:48.655592+0000","last_peered":"2024-09-17T04:06:48.655592+0000","last_clean":"2024-09-17T04:06:48.655592+0000","last_became_active":"2024-09-17T04:06:48.654168+0000","last_became_peered":"2024-09-17T04:06:48.654168+0000","last_unstale":"2024-09-17T04:06:48.655592+0000","last_undegraded":"2024-09-17T04:06:48.655592+0000","last_fullsized":"2024-09-17T04:06:48.655592+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:43.110635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629202+0000","last_change":"2024-09-17T04:06:47.629202+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629202+0000","last_undegraded":"2024-09-17T04:06:47.629202+0000","last_fullsized":"2024-09-17T04:06:47.629202+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638822+0000","last_change":"2024-09-17T04:06:48.635524+0000","last_active":"2024-09-17T04:06:48.638822+0000","last_peered":"2024-09-17T04:06:48.638822+0000","last_clean":"2024-09-17T04:06:48.638822+0000","last_became_active":"2024-09-17T04:06:48.634740+0000","last_became_peered":"2024-09-17T04:06:48.634740+0000","last_unstale":"2024-09-17T04:06:48.638822+0000","last_undegraded":"2024-09-17T04:06:48.638822+0000","last_fullsized":"2024-09-17T04:06:48.638822+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.745727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.648208+0000","last_change":"2024-09-17T04:06:48.639136+0000","last_active":"2024-09-17T04:06:48.648208+0000","last_peered":"2024-09-17T04:06:48.648208+0000","last_clean":"2024-09-17T04:06:48.648208+0000","last_became_active":"2024-09-17T04:06:48.638849+0000","last_became_peered":"2024-09-17T04:06:48.638849+0000","last_unstale":"2024-09-17T04:06:48.648208+0000","last_undegraded":"2024-09-17T04:06:48.648208+0000","last_fullsized":"2024-09-17T04:06:48.648208+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.361484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644287+0000","last_change":"2024-09-17T04:06:48.635932+0000","last_active":"2024-09-17T04:06:48.644287+0000","last_peered":"2024-09-17T04:06:48.644287+0000","last_clean":"2024-09-17T04:06:48.644287+0000","last_became_active":"2024-09-17T04:06:48.634277+0000","last_became_peered":"2024-09-17T04:06:48.634277+0000","last_unstale":"2024-09-17T04:06:48.644287+0000","last_undegraded":"2024-09-17T04:06:48.644287+0000","last_fullsized":"2024-09-17T04:06:48.644287+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:45.140616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629285+0000","last_change":"2024-09-17T04:06:47.629285+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629285+0000","last_undegraded":"2024-09-17T04:06:47.629285+0000","last_fullsized":"2024-09-17T04:06:47.629285+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.627532+0000","last_change":"2024-09-17T04:06:47.627532+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.627532+0000","last_undegraded":"2024-09-17T04:06:47.627532+0000","last_fullsized":"2024-09-17T04:06:47.627532+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645355+0000","last_change":"2024-09-17T04:06:48.643193+0000","last_active":"2024-09-17T04:06:48.645355+0000","last_peered":"2024-09-17T04:06:48.645355+0000","last_clean":"2024-09-17T04:06:48.645355+0000","last_became_active":"2024-09-17T04:06:48.641658+0000","last_became_peered":"2024-09-17T04:06:48.641658+0000","last_unstale":"2024-09-17T04:06:48.645355+0000","last_undegraded":"2024-09-17T04:06:48.645355+0000","last_fullsized":"2024-09-17T04:06:48.645355+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.874073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645968+0000","last_change":"2024-09-17T04:06:48.641052+0000","last_active":"2024-09-17T04:06:48.645968+0000","last_peered":"2024-09-17T04:06:48.645968+0000","last_clean":"2024-09-17T04:06:48.645968+0000","last_became_active":"2024-09-17T04:06:48.640727+0000","last_became_peered":"2024-09-17T04:06:48.640727+0000","last_unstale":"2024-09-17T04:06:48.645968+0000","last_undegraded":"2024-09-17T04:06:48.645968+0000","last_fullsized":"2024-09-17T04:06:48.645968+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:57.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644189+0000","last_change":"2024-09-17T04:06:48.638889+0000","last_active":"2024-09-17T04:06:48.644189+0000","last_peered":"2024-09-17T04:06:48.644189+0000","last_clean":"2024-09-17T04:06:48.644189+0000","last_became_active":"2024-09-17T04:06:48.638320+0000","last_became_peered":"2024-09-17T04:06:48.638320+0000","last_unstale":"2024-09-17T04:06:48.644189+0000","last_undegraded":"2024-09-17T04:06:48.644189+0000","last_fullsized":"2024-09-17T04:06:48.644189+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:03:19.054145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650679+0000","last_change":"2024-09-17T04:06:48.640395+0000","last_active":"2024-09-17T04:06:48.650679+0000","last_peered":"2024-09-17T04:06:48.650679+0000","last_clean":"2024-09-17T04:06:48.650679+0000","last_became_active":"2024-09-17T04:06:48.640063+0000","last_became_peered":"2024-09-17T04:06:48.640063+0000","last_unstale":"2024-09-17T04:06:48.650679+0000","last_undegraded":"2024-09-17T04:06:48.650679+0000","last_fullsized":"2024-09-17T04:06:48.650679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:19:30.759186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.629478+0000","last_change":"2024-09-17T04:06:47.629478+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.629478+0000","last_undegraded":"2024-09-17T04:06:47.629478+0000","last_fullsized":"2024-09-17T04:06:47.629478+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.642754+0000","last_change":"2024-09-17T04:06:48.641161+0000","last_active":"2024-09-17T04:06:48.642754+0000","last_peered":"2024-09-17T04:06:48.642754+0000","last_clean":"2024-09-17T04:06:48.642754+0000","last_became_active":"2024-09-17T04:06:48.640811+0000","last_became_peered":"2024-09-17T04:06:48.640811+0000","last_unstale":"2024-09-17T04:06:48.642754+0000","last_undegraded":"2024-09-17T04:06:48.642754+0000","last_fullsized":"2024-09-17T04:06:48.642754+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.092195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628734+0000","last_change":"2024-09-17T04:06:47.628734+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628734+0000","last_undegraded":"2024-09-17T04:06:47.628734+0000","last_fullsized":"2024-09-17T04:06:47.628734+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651843+0000","last_change":"2024-09-17T04:06:48.646124+0000","last_active":"2024-09-17T04:06:48.651843+0000","last_peered":"2024-09-17T04:06:48.651843+0000","last_clean":"2024-09-17T04:06:48.651843+0000","last_became_active":"2024-09-17T04:06:48.645673+0000","last_became_peered":"2024-09-17T04:06:48.645673+0000","last_unstale":"2024-09-17T04:06:48.651843+0000","last_undegraded":"2024-09-17T04:06:48.651843+0000","last_fullsized":"2024-09-17T04:06:48.651843+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:26.644315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645666+0000","last_change":"2024-09-17T04:06:48.637878+0000","last_active":"2024-09-17T04:06:48.645666+0000","last_peered":"2024-09-17T04:06:48.645666+0000","last_clean":"2024-09-17T04:06:48.645666+0000","last_became_active":"2024-09-17T04:06:48.637580+0000","last_became_peered":"2024-09-17T04:06:48.637580+0000","last_unstale":"2024-09-17T04:06:48.645666+0000","last_undegraded":"2024-09-17T04:06:48.645666+0000","last_fullsized":"2024-09-17T04:06:48.645666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:28:38.576300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618364+0000","last_change":"2024-09-17T04:06:47.618364+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618364+0000","last_undegraded":"2024-09-17T04:06:47.618364+0000","last_fullsized":"2024-09-17T04:06:47.618364+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633778+0000","last_change":"2024-09-17T04:06:47.633778+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633778+0000","last_undegraded":"2024-09-17T04:06:47.633778+0000","last_fullsized":"2024-09-17T04:06:47.633778+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645544+0000","last_change":"2024-09-17T04:06:48.643887+0000","last_active":"2024-09-17T04:06:48.645544+0000","last_peered":"2024-09-17T04:06:48.645544+0000","last_clean":"2024-09-17T04:06:48.645544+0000","last_became_active":"2024-09-17T04:06:48.643541+0000","last_became_peered":"2024-09-17T04:06:48.643541+0000","last_unstale":"2024-09-17T04:06:48.645544+0000","last_undegraded":"2024-09-17T04:06:48.645544+0000","last_fullsized":"2024-09-17T04:06:48.645544+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:44:51.740525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650895+0000","last_change":"2024-09-17T04:06:48.646487+0000","last_active":"2024-09-17T04:06:48.650895+0000","last_peered":"2024-09-17T04:06:48.650895+0000","last_clean":"2024-09-17T04:06:48.650895+0000","last_became_active":"2024-09-17T04:06:48.646200+0000","last_became_peered":"2024-09-17T04:06:48.646200+0000","last_unstale":"2024-09-17T04:06:48.650895+0000","last_undegraded":"2024-09-17T04:06:48.650895+0000","last_fullsized":"2024-09-17T04:06:48.650895+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.053314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645614+0000","last_change":"2024-09-17T04:06:48.638450+0000","last_active":"2024-09-17T04:06:48.645614+0000","last_peered":"2024-09-17T04:06:48.645614+0000","last_clean":"2024-09-17T04:06:48.645614+0000","last_became_active":"2024-09-17T04:06:48.638148+0000","last_became_peered":"2024-09-17T04:06:48.638148+0000","last_unstale":"2024-09-17T04:06:48.645614+0000","last_undegraded":"2024-09-17T04:06:48.645614+0000","last_fullsized":"2024-09-17T04:06:48.645614+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:11.182838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"246'4","reported_seq":17,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656184+0000","last_change":"2024-09-17T04:06:47.630685+0000","last_active":"2024-09-17T04:06:48.656184+0000","last_peered":"2024-09-17T04:06:48.656184+0000","last_clean":"2024-09-17T04:06:48.656184+0000","last_became_active":"2024-09-17T04:06:47.630367+0000","last_became_peered":"2024-09-17T04:06:47.630367+0000","last_unstale":"2024-09-17T04:06:48.656184+0000","last_undegraded":"2024-09-17T04:06:48.656184+0000","last_fullsized":"2024-09-17T04:06:48.656184+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:56.891603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640751+0000","last_change":"2024-09-17T04:06:48.636369+0000","last_active":"2024-09-17T04:06:48.640751+0000","last_peered":"2024-09-17T04:06:48.640751+0000","last_clean":"2024-09-17T04:06:48.640751+0000","last_became_active":"2024-09-17T04:06:48.636001+0000","last_became_peered":"2024-09-17T04:06:48.636001+0000","last_unstale":"2024-09-17T04:06:48.640751+0000","last_undegraded":"2024-09-17T04:06:48.640751+0000","last_fullsized":"2024-09-17T04:06:48.640751+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:52:21.692754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642287+0000","last_change":"2024-09-17T04:06:47.640788+0000","last_active":"2024-09-17T04:06:47.642287+0000","last_peered":"2024-09-17T04:06:47.642287+0000","last_clean":"2024-09-17T04:06:47.642287+0000","last_became_active":"2024-09-17T04:06:47.640507+0000","last_became_peered":"2024-09-17T04:06:47.640507+0000","last_unstale":"2024-09-17T04:06:47.642287+0000","last_undegraded":"2024-09-17T04:06:47.642287+0000","last_fullsized":"2024-09-17T04:06:47.642287+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:24.521931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.625495+0000","last_change":"2024-09-17T04:06:47.625495+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.625495+0000","last_undegraded":"2024-09-17T04:06:47.625495+0000","last_fullsized":"2024-09-17T04:06:47.625495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642354+0000","last_change":"2024-09-17T04:06:47.640338+0000","last_active":"2024-09-17T04:06:47.642354+0000","last_peered":"2024-09-17T04:06:47.642354+0000","last_clean":"2024-09-17T04:06:47.642354+0000","last_became_active":"2024-09-17T04:06:47.639781+0000","last_became_peered":"2024-09-17T04:06:47.639781+0000","last_unstale":"2024-09-17T04:06:47.642354+0000","last_undegraded":"2024-09-17T04:06:47.642354+0000","last_fullsized":"2024-09-17T04:06:47.642354+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:20.160754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645958+0000","last_change":"2024-09-17T04:06:48.641644+0000","last_active":"2024-09-17T04:06:48.645958+0000","last_peered":"2024-09-17T04:06:48.645958+0000","last_clean":"2024-09-17T04:06:48.645958+0000","last_became_active":"2024-09-17T04:06:48.641054+0000","last_became_peered":"2024-09-17T04:06:48.641054+0000","last_unstale":"2024-09-17T04:06:48.645958+0000","last_undegraded":"2024-09-17T04:06:48.645958+0000","last_fullsized":"2024-09-17T04:06:48.645958+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:37:50.540964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660581+0000","last_change":"2024-09-17T04:06:47.634854+0000","last_active":"2024-09-17T04:06:48.660581+0000","last_peered":"2024-09-17T04:06:48.660581+0000","last_clean":"2024-09-17T04:06:48.660581+0000","last_became_active":"2024-09-17T04:06:47.634556+0000","last_became_peered":"2024-09-17T04:06:47.634556+0000","last_unstale":"2024-09-17T04:06:48.660581+0000","last_undegraded":"2024-09-17T04:06:48.660581+0000","last_fullsized":"2024-09-17T04:06:48.660581+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:55:59.294589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637459+0000","last_change":"2024-09-17T04:06:48.633413+0000","last_active":"2024-09-17T04:06:48.637459+0000","last_peered":"2024-09-17T04:06:48.637459+0000","last_clean":"2024-09-17T04:06:48.637459+0000","last_became_active":"2024-09-17T04:06:48.632689+0000","last_became_peered":"2024-09-17T04:06:48.632689+0000","last_unstale":"2024-09-17T04:06:48.637459+0000","last_undegraded":"2024-09-17T04:06:48.637459+0000","last_fullsized":"2024-09-17T04:06:48.637459+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:40.165427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653343+0000","last_change":"2024-09-17T04:06:47.637532+0000","last_active":"2024-09-17T04:06:48.653343+0000","last_peered":"2024-09-17T04:06:48.653343+0000","last_clean":"2024-09-17T04:06:48.653343+0000","last_became_active":"2024-09-17T04:06:47.637150+0000","last_became_peered":"2024-09-17T04:06:47.637150+0000","last_unstale":"2024-09-17T04:06:48.653343+0000","last_undegraded":"2024-09-17T04:06:48.653343+0000","last_fullsized":"2024-09-17T04:06:48.653343+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:04.474254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.632926+0000","last_change":"2024-09-17T04:06:48.625119+0000","last_active":"2024-09-17T04:06:48.632926+0000","last_peered":"2024-09-17T04:06:48.632926+0000","last_clean":"2024-09-17T04:06:48.632926+0000","last_became_active":"2024-09-17T04:06:48.624823+0000","last_became_peered":"2024-09-17T04:06:48.624823+0000","last_unstale":"2024-09-17T04:06:48.632926+0000","last_undegraded":"2024-09-17T04:06:48.632926+0000","last_fullsized":"2024-09-17T04:06:48.632926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:21.840133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639235+0000","last_change":"2024-09-17T04:06:47.636767+0000","last_active":"2024-09-17T04:06:47.639235+0000","last_peered":"2024-09-17T04:06:47.639235+0000","last_clean":"2024-09-17T04:06:47.639235+0000","last_became_active":"2024-09-17T04:06:47.635877+0000","last_became_peered":"2024-09-17T04:06:47.635877+0000","last_unstale":"2024-09-17T04:06:47.639235+0000","last_undegraded":"2024-09-17T04:06:47.639235+0000","last_fullsized":"2024-09-17T04:06:47.639235+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.633498+0000","last_change":"2024-09-17T04:06:47.633498+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.633498+0000","last_undegraded":"2024-09-17T04:06:47.633498+0000","last_fullsized":"2024-09-17T04:06:47.633498+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630330+0000","last_change":"2024-09-17T04:06:47.641518+0000","last_active":"2024-09-17T04:06:48.630330+0000","last_peered":"2024-09-17T04:06:48.630330+0000","last_clean":"2024-09-17T04:06:48.630330+0000","last_became_active":"2024-09-17T04:06:47.641158+0000","last_became_peered":"2024-09-17T04:06:47.641158+0000","last_unstale":"2024-09-17T04:06:48.630330+0000","last_undegraded":"2024-09-17T04:06:48.630330+0000","last_fullsized":"2024-09-17T04:06:48.630330+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:43.208862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.616816+0000","last_change":"2024-09-17T04:06:47.616816+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.616816+0000","last_undegraded":"2024-09-17T04:06:47.616816+0000","last_fullsized":"2024-09-17T04:06:47.616816+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.638712+0000","last_change":"2024-09-17T04:06:47.637605+0000","last_active":"2024-09-17T04:06:47.638712+0000","last_peered":"2024-09-17T04:06:47.638712+0000","last_clean":"2024-09-17T04:06:47.638712+0000","last_became_active":"2024-09-17T04:06:47.637251+0000","last_became_peered":"2024-09-17T04:06:47.637251+0000","last_unstale":"2024-09-17T04:06:47.638712+0000","last_undegraded":"2024-09-17T04:06:47.638712+0000","last_fullsized":"2024-09-17T04:06:47.638712+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.619494+0000","last_change":"2024-09-17T04:06:47.619494+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.619494+0000","last_undegraded":"2024-09-17T04:06:47.619494+0000","last_fullsized":"2024-09-17T04:06:47.619494+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630182+0000","last_change":"2024-09-17T04:06:47.639900+0000","last_active":"2024-09-17T04:06:48.630182+0000","last_peered":"2024-09-17T04:06:48.630182+0000","last_clean":"2024-09-17T04:06:48.630182+0000","last_became_active":"2024-09-17T04:06:47.638771+0000","last_became_peered":"2024-09-17T04:06:47.638771+0000","last_unstale":"2024-09-17T04:06:48.630182+0000","last_undegraded":"2024-09-17T04:06:48.630182+0000","last_fullsized":"2024-09-17T04:06:48.630182+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:42.102677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.640781+0000","last_change":"2024-09-17T04:06:48.637971+0000","last_active":"2024-09-17T04:06:48.640781+0000","last_peered":"2024-09-17T04:06:48.640781+0000","last_clean":"2024-09-17T04:06:48.640781+0000","last_became_active":"2024-09-17T04:06:48.637655+0000","last_became_peered":"2024-09-17T04:06:48.637655+0000","last_unstale":"2024-09-17T04:06:48.640781+0000","last_undegraded":"2024-09-17T04:06:48.640781+0000","last_fullsized":"2024-09-17T04:06:48.640781+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:54.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624781+0000","last_change":"2024-09-17T04:06:47.628105+0000","last_active":"2024-09-17T04:06:48.624781+0000","last_peered":"2024-09-17T04:06:48.624781+0000","last_clean":"2024-09-17T04:06:48.624781+0000","last_became_active":"2024-09-17T04:06:47.627756+0000","last_became_peered":"2024-09-17T04:06:47.627756+0000","last_unstale":"2024-09-17T04:06:48.624781+0000","last_undegraded":"2024-09-17T04:06:48.624781+0000","last_fullsized":"2024-09-17T04:06:48.624781+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:38.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653352+0000","last_change":"2024-09-17T04:06:48.646524+0000","last_active":"2024-09-17T04:06:48.653352+0000","last_peered":"2024-09-17T04:06:48.653352+0000","last_clean":"2024-09-17T04:06:48.653352+0000","last_became_active":"2024-09-17T04:06:48.646148+0000","last_became_peered":"2024-09-17T04:06:48.646148+0000","last_unstale":"2024-09-17T04:06:48.653352+0000","last_undegraded":"2024-09-17T04:06:48.653352+0000","last_fullsized":"2024-09-17T04:06:48.653352+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:29:48.285920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"246'3","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.661105+0000","last_change":"2024-09-17T04:06:47.635826+0000","last_active":"2024-09-17T04:06:48.661105+0000","last_peered":"2024-09-17T04:06:48.661105+0000","last_clean":"2024-09-17T04:06:48.661105+0000","last_became_active":"2024-09-17T04:06:47.633375+0000","last_became_peered":"2024-09-17T04:06:47.633375+0000","last_unstale":"2024-09-17T04:06:48.661105+0000","last_undegraded":"2024-09-17T04:06:48.661105+0000","last_fullsized":"2024-09-17T04:06:48.661105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:13.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622935+0000","last_change":"2024-09-17T04:06:47.622935+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622935+0000","last_undegraded":"2024-09-17T04:06:47.622935+0000","last_fullsized":"2024-09-17T04:06:47.622935+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639789+0000","last_change":"2024-09-17T04:06:47.638185+0000","last_active":"2024-09-17T04:06:47.639789+0000","last_peered":"2024-09-17T04:06:47.639789+0000","last_clean":"2024-09-17T04:06:47.639789+0000","last_became_active":"2024-09-17T04:06:47.637416+0000","last_became_peered":"2024-09-17T04:06:47.637416+0000","last_unstale":"2024-09-17T04:06:47.639789+0000","last_undegraded":"2024-09-17T04:06:47.639789+0000","last_fullsized":"2024-09-17T04:06:47.639789+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:04:13.923343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649907+0000","last_change":"2024-09-17T04:06:48.647751+0000","last_active":"2024-09-17T04:06:48.649907+0000","last_peered":"2024-09-17T04:06:48.649907+0000","last_clean":"2024-09-17T04:06:48.649907+0000","last_became_active":"2024-09-17T04:06:48.647416+0000","last_became_peered":"2024-09-17T04:06:48.647416+0000","last_unstale":"2024-09-17T04:06:48.649907+0000","last_undegraded":"2024-09-17T04:06:48.649907+0000","last_fullsized":"2024-09-17T04:06:48.649907+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.537308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652112+0000","last_change":"2024-09-17T04:06:47.640722+0000","last_active":"2024-09-17T04:06:48.652112+0000","last_peered":"2024-09-17T04:06:48.652112+0000","last_clean":"2024-09-17T04:06:48.652112+0000","last_became_active":"2024-09-17T04:06:47.640406+0000","last_became_peered":"2024-09-17T04:06:47.640406+0000","last_unstale":"2024-09-17T04:06:48.652112+0000","last_undegraded":"2024-09-17T04:06:48.652112+0000","last_fullsized":"2024-09-17T04:06:48.652112+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:40.791049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.633929+0000","last_change":"2024-09-17T04:06:48.628835+0000","last_active":"2024-09-17T04:06:48.633929+0000","last_peered":"2024-09-17T04:06:48.633929+0000","last_clean":"2024-09-17T04:06:48.633929+0000","last_became_active":"2024-09-17T04:06:48.628468+0000","last_became_peered":"2024-09-17T04:06:48.628468+0000","last_unstale":"2024-09-17T04:06:48.633929+0000","last_undegraded":"2024-09-17T04:06:48.633929+0000","last_fullsized":"2024-09-17T04:06:48.633929+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:44:43.375277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637573+0000","last_change":"2024-09-17T04:06:47.629947+0000","last_active":"2024-09-17T04:06:48.637573+0000","last_peered":"2024-09-17T04:06:48.637573+0000","last_clean":"2024-09-17T04:06:48.637573+0000","last_became_active":"2024-09-17T04:06:47.628403+0000","last_became_peered":"2024-09-17T04:06:47.628403+0000","last_unstale":"2024-09-17T04:06:48.637573+0000","last_undegraded":"2024-09-17T04:06:48.637573+0000","last_fullsized":"2024-09-17T04:06:48.637573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:15.090695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.652110+0000","last_change":"2024-09-17T04:06:48.650761+0000","last_active":"2024-09-17T04:06:48.652110+0000","last_peered":"2024-09-17T04:06:48.652110+0000","last_clean":"2024-09-17T04:06:48.652110+0000","last_became_active":"2024-09-17T04:06:48.650386+0000","last_became_peered":"2024-09-17T04:06:48.650386+0000","last_unstale":"2024-09-17T04:06:48.652110+0000","last_undegraded":"2024-09-17T04:06:48.652110+0000","last_fullsized":"2024-09-17T04:06:48.652110+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:57.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.639196+0000","last_change":"2024-09-17T04:06:47.637094+0000","last_active":"2024-09-17T04:06:47.639196+0000","last_peered":"2024-09-17T04:06:47.639196+0000","last_clean":"2024-09-17T04:06:47.639196+0000","last_became_active":"2024-09-17T04:06:47.636787+0000","last_became_peered":"2024-09-17T04:06:47.636787+0000","last_unstale":"2024-09-17T04:06:47.639196+0000","last_undegraded":"2024-09-17T04:06:47.639196+0000","last_fullsized":"2024-09-17T04:06:47.639196+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644926+0000","last_change":"2024-09-17T04:06:48.642186+0000","last_active":"2024-09-17T04:06:48.644926+0000","last_peered":"2024-09-17T04:06:48.644926+0000","last_clean":"2024-09-17T04:06:48.644926+0000","last_became_active":"2024-09-17T04:06:48.640270+0000","last_became_peered":"2024-09-17T04:06:48.640270+0000","last_unstale":"2024-09-17T04:06:48.644926+0000","last_undegraded":"2024-09-17T04:06:48.644926+0000","last_fullsized":"2024-09-17T04:06:48.644926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:57.596302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"246'8","reported_seq":21,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.665782+0000","last_change":"2024-09-17T04:06:47.639284+0000","last_active":"2024-09-17T04:06:48.665782+0000","last_peered":"2024-09-17T04:06:48.665782+0000","last_clean":"2024-09-17T04:06:48.665782+0000","last_became_active":"2024-09-17T04:06:47.638987+0000","last_became_peered":"2024-09-17T04:06:47.638987+0000","last_unstale":"2024-09-17T04:06:48.665782+0000","last_undegraded":"2024-09-17T04:06:48.665782+0000","last_fullsized":"2024-09-17T04:06:48.665782+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:25.636357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650612+0000","last_change":"2024-09-17T04:06:48.640025+0000","last_active":"2024-09-17T04:06:48.650612+0000","last_peered":"2024-09-17T04:06:48.650612+0000","last_clean":"2024-09-17T04:06:48.650612+0000","last_became_active":"2024-09-17T04:06:48.639713+0000","last_became_peered":"2024-09-17T04:06:48.639713+0000","last_unstale":"2024-09-17T04:06:48.650612+0000","last_undegraded":"2024-09-17T04:06:48.650612+0000","last_fullsized":"2024-09-17T04:06:48.650612+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:46.136192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"246'1","reported_seq":14,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650031+0000","last_change":"2024-09-17T04:06:47.627885+0000","last_active":"2024-09-17T04:06:48.650031+0000","last_peered":"2024-09-17T04:06:48.650031+0000","last_clean":"2024-09-17T04:06:48.650031+0000","last_became_active":"2024-09-17T04:06:47.627314+0000","last_became_peered":"2024-09-17T04:06:47.627314+0000","last_unstale":"2024-09-17T04:06:48.650031+0000","last_undegraded":"2024-09-17T04:06:48.650031+0000","last_fullsized":"2024-09-17T04:06:48.650031+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:32.996351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646180+0000","last_change":"2024-09-17T04:06:48.640039+0000","last_active":"2024-09-17T04:06:48.646180+0000","last_peered":"2024-09-17T04:06:48.646180+0000","last_clean":"2024-09-17T04:06:48.646180+0000","last_became_active":"2024-09-17T04:06:48.639062+0000","last_became_peered":"2024-09-17T04:06:48.639062+0000","last_unstale":"2024-09-17T04:06:48.646180+0000","last_undegraded":"2024-09-17T04:06:48.646180+0000","last_fullsized":"2024-09-17T04:06:48.646180+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:39.282547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.655638+0000","last_change":"2024-09-17T04:06:47.626021+0000","last_active":"2024-09-17T04:06:48.655638+0000","last_peered":"2024-09-17T04:06:48.655638+0000","last_clean":"2024-09-17T04:06:48.655638+0000","last_became_active":"2024-09-17T04:06:47.625353+0000","last_became_peered":"2024-09-17T04:06:47.625353+0000","last_unstale":"2024-09-17T04:06:48.655638+0000","last_undegraded":"2024-09-17T04:06:48.655638+0000","last_fullsized":"2024-09-17T04:06:48.655638+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.622812+0000","last_change":"2024-09-17T04:06:47.622812+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.622812+0000","last_undegraded":"2024-09-17T04:06:47.622812+0000","last_fullsized":"2024-09-17T04:06:47.622812+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.632014+0000","last_change":"2024-09-17T04:06:47.630827+0000","last_active":"2024-09-17T04:06:47.632014+0000","last_peered":"2024-09-17T04:06:47.632014+0000","last_clean":"2024-09-17T04:06:47.632014+0000","last_became_active":"2024-09-17T04:06:47.630146+0000","last_became_peered":"2024-09-17T04:06:47.630146+0000","last_unstale":"2024-09-17T04:06:47.632014+0000","last_undegraded":"2024-09-17T04:06:47.632014+0000","last_fullsized":"2024-09-17T04:06:47.632014+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:32:56.879306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656054+0000","last_change":"2024-09-17T04:06:48.655091+0000","last_active":"2024-09-17T04:06:48.656054+0000","last_peered":"2024-09-17T04:06:48.656054+0000","last_clean":"2024-09-17T04:06:48.656054+0000","last_became_active":"2024-09-17T04:06:48.654782+0000","last_became_peered":"2024-09-17T04:06:48.654782+0000","last_unstale":"2024-09-17T04:06:48.656054+0000","last_undegraded":"2024-09-17T04:06:48.656054+0000","last_fullsized":"2024-09-17T04:06:48.656054+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:06.889990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.642059+0000","last_change":"2024-09-17T04:06:47.639967+0000","last_active":"2024-09-17T04:06:47.642059+0000","last_peered":"2024-09-17T04:06:47.642059+0000","last_clean":"2024-09-17T04:06:47.642059+0000","last_became_active":"2024-09-17T04:06:47.639606+0000","last_became_peered":"2024-09-17T04:06:47.639606+0000","last_unstale":"2024-09-17T04:06:47.642059+0000","last_undegraded":"2024-09-17T04:06:47.642059+0000","last_fullsized":"2024-09-17T04:06:47.642059+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:36.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649830+0000","last_change":"2024-09-17T04:06:48.642222+0000","last_active":"2024-09-17T04:06:48.649830+0000","last_peered":"2024-09-17T04:06:48.649830+0000","last_clean":"2024-09-17T04:06:48.649830+0000","last_became_active":"2024-09-17T04:06:48.641845+0000","last_became_peered":"2024-09-17T04:06:48.641845+0000","last_unstale":"2024-09-17T04:06:48.649830+0000","last_undegraded":"2024-09-17T04:06:48.649830+0000","last_fullsized":"2024-09-17T04:06:48.649830+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:51.088460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.659677+0000","last_change":"2024-09-17T04:06:47.635036+0000","last_active":"2024-09-17T04:06:48.659677+0000","last_peered":"2024-09-17T04:06:48.659677+0000","last_clean":"2024-09-17T04:06:48.659677+0000","last_became_active":"2024-09-17T04:06:47.634721+0000","last_became_peered":"2024-09-17T04:06:47.634721+0000","last_unstale":"2024-09-17T04:06:48.659677+0000","last_undegraded":"2024-09-17T04:06:48.659677+0000","last_fullsized":"2024-09-17T04:06:48.659677+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:48.022710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649772+0000","last_change":"2024-09-17T04:06:48.647597+0000","last_active":"2024-09-17T04:06:48.649772+0000","last_peered":"2024-09-17T04:06:48.649772+0000","last_clean":"2024-09-17T04:06:48.649772+0000","last_became_active":"2024-09-17T04:06:48.647265+0000","last_became_peered":"2024-09-17T04:06:48.647265+0000","last_unstale":"2024-09-17T04:06:48.649772+0000","last_undegraded":"2024-09-17T04:06:48.649772+0000","last_fullsized":"2024-09-17T04:06:48.649772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:31.880487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.621482+0000","last_change":"2024-09-17T04:06:47.627934+0000","last_active":"2024-09-17T04:06:48.621482+0000","last_peered":"2024-09-17T04:06:48.621482+0000","last_clean":"2024-09-17T04:06:48.621482+0000","last_became_active":"2024-09-17T04:06:47.627651+0000","last_became_peered":"2024-09-17T04:06:47.627651+0000","last_unstale":"2024-09-17T04:06:48.621482+0000","last_undegraded":"2024-09-17T04:06:48.621482+0000","last_fullsized":"2024-09-17T04:06:48.621482+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:28.266741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635500+0000","last_change":"2024-09-17T04:06:48.630489+0000","last_active":"2024-09-17T04:06:48.635500+0000","last_peered":"2024-09-17T04:06:48.635500+0000","last_clean":"2024-09-17T04:06:48.635500+0000","last_became_active":"2024-09-17T04:06:48.630199+0000","last_became_peered":"2024-09-17T04:06:48.630199+0000","last_unstale":"2024-09-17T04:06:48.635500+0000","last_undegraded":"2024-09-17T04:06:48.635500+0000","last_fullsized":"2024-09-17T04:06:48.635500+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:49.945070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":10,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-09-17T04:06:47.637806+0000","last_change":"2024-09-17T04:06:47.636455+0000","last_active":"2024-09-17T04:06:47.637806+0000","last_peered":"2024-09-17T04:06:47.637806+0000","last_clean":"2024-09-17T04:06:47.637806+0000","last_became_active":"2024-09-17T04:06:47.636161+0000","last_became_peered":"2024-09-17T04:06:47.636161+0000","last_unstale":"2024-09-17T04:06:47.637806+0000","last_undegraded":"2024-09-17T04:06:47.637806+0000","last_fullsized":"2024-09-17T04:06:47.637806+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:10.261951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630666+0000","last_change":"2024-09-17T04:06:48.626073+0000","last_active":"2024-09-17T04:06:48.630666+0000","last_peered":"2024-09-17T04:06:48.630666+0000","last_clean":"2024-09-17T04:06:48.630666+0000","last_became_active":"2024-09-17T04:06:48.624659+0000","last_became_peered":"2024-09-17T04:06:48.624659+0000","last_unstale":"2024-09-17T04:06:48.630666+0000","last_undegraded":"2024-09-17T04:06:48.630666+0000","last_fullsized":"2024-09-17T04:06:48.630666+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:46:31.519896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.622226+0000","last_change":"2024-09-17T04:06:47.631678+0000","last_active":"2024-09-17T04:06:48.622226+0000","last_peered":"2024-09-17T04:06:48.622226+0000","last_clean":"2024-09-17T04:06:48.622226+0000","last_became_active":"2024-09-17T04:06:47.629357+0000","last_became_peered":"2024-09-17T04:06:47.629357+0000","last_unstale":"2024-09-17T04:06:48.622226+0000","last_undegraded":"2024-09-17T04:06:48.622226+0000","last_fullsized":"2024-09-17T04:06:48.622226+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:58.960464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646523+0000","last_change":"2024-09-17T04:06:48.641309+0000","last_active":"2024-09-17T04:06:48.646523+0000","last_peered":"2024-09-17T04:06:48.646523+0000","last_clean":"2024-09-17T04:06:48.646523+0000","last_became_active":"2024-09-17T04:06:48.640780+0000","last_became_peered":"2024-09-17T04:06:48.640780+0000","last_unstale":"2024-09-17T04:06:48.646523+0000","last_undegraded":"2024-09-17T04:06:48.646523+0000","last_fullsized":"2024-09-17T04:06:48.646523+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:38.559159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660451+0000","last_change":"2024-09-17T04:06:47.634843+0000","last_active":"2024-09-17T04:06:48.660451+0000","last_peered":"2024-09-17T04:06:48.660451+0000","last_clean":"2024-09-17T04:06:48.660451+0000","last_became_active":"2024-09-17T04:06:47.634529+0000","last_became_peered":"2024-09-17T04:06:47.634529+0000","last_unstale":"2024-09-17T04:06:48.660451+0000","last_undegraded":"2024-09-17T04:06:48.660451+0000","last_fullsized":"2024-09-17T04:06:48.660451+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:24.241367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.628250+0000","last_change":"2024-09-17T04:06:47.628250+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.628250+0000","last_undegraded":"2024-09-17T04:06:47.628250+0000","last_fullsized":"2024-09-17T04:06:47.628250+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.619096+0000","last_change":"2024-09-17T04:06:47.635594+0000","last_active":"2024-09-17T04:06:48.619096+0000","last_peered":"2024-09-17T04:06:48.619096+0000","last_clean":"2024-09-17T04:06:48.619096+0000","last_became_active":"2024-09-17T04:06:47.633247+0000","last_became_peered":"2024-09-17T04:06:47.633247+0000","last_unstale":"2024-09-17T04:06:48.619096+0000","last_undegraded":"2024-09-17T04:06:48.619096+0000","last_fullsized":"2024-09-17T04:06:48.619096+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:29.409434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656026+0000","last_change":"2024-09-17T04:06:48.654761+0000","last_active":"2024-09-17T04:06:48.656026+0000","last_peered":"2024-09-17T04:06:48.656026+0000","last_clean":"2024-09-17T04:06:48.656026+0000","last_became_active":"2024-09-17T04:06:48.654228+0000","last_became_peered":"2024-09-17T04:06:48.654228+0000","last_unstale":"2024-09-17T04:06:48.656026+0000","last_undegraded":"2024-09-17T04:06:48.656026+0000","last_fullsized":"2024-09-17T04:06:48.656026+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:34:46.467512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650149+0000","last_change":"2024-09-17T04:06:47.631080+0000","last_active":"2024-09-17T04:06:48.650149+0000","last_peered":"2024-09-17T04:06:48.650149+0000","last_clean":"2024-09-17T04:06:48.650149+0000","last_became_active":"2024-09-17T04:06:47.630810+0000","last_became_peered":"2024-09-17T04:06:47.630810+0000","last_unstale":"2024-09-17T04:06:48.650149+0000","last_undegraded":"2024-09-17T04:06:48.650149+0000","last_fullsized":"2024-09-17T04:06:48.650149+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.601411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.644148+0000","last_change":"2024-09-17T04:06:48.640563+0000","last_active":"2024-09-17T04:06:48.644148+0000","last_peered":"2024-09-17T04:06:48.644148+0000","last_clean":"2024-09-17T04:06:48.644148+0000","last_became_active":"2024-09-17T04:06:48.640193+0000","last_became_peered":"2024-09-17T04:06:48.640193+0000","last_unstale":"2024-09-17T04:06:48.644148+0000","last_undegraded":"2024-09-17T04:06:48.644148+0000","last_fullsized":"2024-09-17T04:06:48.644148+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:15:43.623529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631108+0000","last_change":"2024-09-17T04:06:47.640808+0000","last_active":"2024-09-17T04:06:48.631108+0000","last_peered":"2024-09-17T04:06:48.631108+0000","last_clean":"2024-09-17T04:06:48.631108+0000","last_became_active":"2024-09-17T04:06:47.639846+0000","last_became_peered":"2024-09-17T04:06:47.639846+0000","last_unstale":"2024-09-17T04:06:48.631108+0000","last_undegraded":"2024-09-17T04:06:48.631108+0000","last_fullsized":"2024-09-17T04:06:48.631108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:15.837166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":1,"reported_epoch":245,"state":"creating+peering","last_fresh":"2024-09-17T04:06:47.618723+0000","last_change":"2024-09-17T04:06:47.618723+0000","last_active":"2024-09-17T04:06:47.601743+0000","last_peered":"2024-09-17T04:06:47.601743+0000","last_clean":"2024-09-17T04:06:47.601743+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:06:47.618723+0000","last_undegraded":"2024-09-17T04:06:47.618723+0000","last_fullsized":"2024-09-17T04:06:47.618723+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.630437+0000","last_change":"2024-09-17T04:06:47.639290+0000","last_active":"2024-09-17T04:06:48.630437+0000","last_peered":"2024-09-17T04:06:48.630437+0000","last_clean":"2024-09-17T04:06:48.630437+0000","last_became_active":"2024-09-17T04:06:47.638370+0000","last_became_peered":"2024-09-17T04:06:47.638370+0000","last_unstale":"2024-09-17T04:06:48.630437+0000","last_undegraded":"2024-09-17T04:06:48.630437+0000","last_fullsized":"2024-09-17T04:06:48.630437+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:17.818126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651574+0000","last_change":"2024-09-17T04:06:48.645144+0000","last_active":"2024-09-17T04:06:48.651574+0000","last_peered":"2024-09-17T04:06:48.651574+0000","last_clean":"2024-09-17T04:06:48.651574+0000","last_became_active":"2024-09-17T04:06:48.644506+0000","last_became_peered":"2024-09-17T04:06:48.644506+0000","last_unstale":"2024-09-17T04:06:48.651574+0000","last_undegraded":"2024-09-17T04:06:48.651574+0000","last_fullsized":"2024-09-17T04:06:48.651574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:58.415646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638296+0000","last_change":"2024-09-17T04:06:47.637796+0000","last_active":"2024-09-17T04:06:48.638296+0000","last_peered":"2024-09-17T04:06:48.638296+0000","last_clean":"2024-09-17T04:06:48.638296+0000","last_became_active":"2024-09-17T04:06:47.637473+0000","last_became_peered":"2024-09-17T04:06:47.637473+0000","last_unstale":"2024-09-17T04:06:48.638296+0000","last_undegraded":"2024-09-17T04:06:48.638296+0000","last_fullsized":"2024-09-17T04:06:48.638296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:35.426948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.635294+0000","last_change":"2024-09-17T04:06:48.626386+0000","last_active":"2024-09-17T04:06:48.635294+0000","last_peered":"2024-09-17T04:06:48.635294+0000","last_clean":"2024-09-17T04:06:48.635294+0000","last_became_active":"2024-09-17T04:06:48.623872+0000","last_became_peered":"2024-09-17T04:06:48.623872+0000","last_unstale":"2024-09-17T04:06:48.635294+0000","last_undegraded":"2024-09-17T04:06:48.635294+0000","last_fullsized":"2024-09-17T04:06:48.635294+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:07.050814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.624231+0000","last_change":"2024-09-17T04:06:47.632594+0000","last_active":"2024-09-17T04:06:48.624231+0000","last_peered":"2024-09-17T04:06:48.624231+0000","last_clean":"2024-09-17T04:06:48.624231+0000","last_became_active":"2024-09-17T04:06:47.632229+0000","last_became_peered":"2024-09-17T04:06:47.632229+0000","last_unstale":"2024-09-17T04:06:48.624231+0000","last_undegraded":"2024-09-17T04:06:48.624231+0000","last_fullsized":"2024-09-17T04:06:48.624231+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:10.174881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650605+0000","last_change":"2024-09-17T04:06:48.640651+0000","last_active":"2024-09-17T04:06:48.650605+0000","last_peered":"2024-09-17T04:06:48.650605+0000","last_clean":"2024-09-17T04:06:48.650605+0000","last_became_active":"2024-09-17T04:06:48.639761+0000","last_became_peered":"2024-09-17T04:06:48.639761+0000","last_unstale":"2024-09-17T04:06:48.650605+0000","last_undegraded":"2024-09-17T04:06:48.650605+0000","last_fullsized":"2024-09-17T04:06:48.650605+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:47:53.798909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638592+0000","last_change":"2024-09-17T04:06:47.639117+0000","last_active":"2024-09-17T04:06:48.638592+0000","last_peered":"2024-09-17T04:06:48.638592+0000","last_clean":"2024-09-17T04:06:48.638592+0000","last_became_active":"2024-09-17T04:06:47.638778+0000","last_became_peered":"2024-09-17T04:06:47.638778+0000","last_unstale":"2024-09-17T04:06:48.638592+0000","last_undegraded":"2024-09-17T04:06:48.638592+0000","last_fullsized":"2024-09-17T04:06:48.638592+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:42.527414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653501+0000","last_change":"2024-09-17T04:06:48.648394+0000","last_active":"2024-09-17T04:06:48.653501+0000","last_peered":"2024-09-17T04:06:48.653501+0000","last_clean":"2024-09-17T04:06:48.653501+0000","last_became_active":"2024-09-17T04:06:48.647962+0000","last_became_peered":"2024-09-17T04:06:48.647962+0000","last_unstale":"2024-09-17T04:06:48.653501+0000","last_undegraded":"2024-09-17T04:06:48.653501+0000","last_fullsized":"2024-09-17T04:06:48.653501+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:16.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":525,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.631126+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:48.631126+0000","last_peered":"2024-09-17T04:06:48.631126+0000","last_clean":"2024-09-17T04:06:48.631126+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:48.631126+0000","last_undegraded":"2024-09-17T04:06:48.631126+0000","last_fullsized":"2024-09-17T04:06:48.631126+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27,"ondisk_log_size":27,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509651,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509654,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34248,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337592,"statfs":{"total":96636764160,"available":96601694208,"internally_reserved":0,"allocated":3129344,"data_stored":1380978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56424,"kb_used_data":3028,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315416,"statfs":{"total":96636764160,"available":96578985984,"internally_reserved":0,"allocated":3100672,"data_stored":1372501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56444,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315396,"statfs":{"total":96636764160,"available":96578965504,"internally_reserved":0,"allocated":3117056,"data_stored":1379858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509653,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56448,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315392,"statfs":{"total":96636764160,"available":96578961408,"internally_reserved":0,"allocated":3121152,"data_stored":1379400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509653,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":3121152,"data_stored":1379332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542355,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52256,"kb_used_data":3016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319584,"statfs":{"total":96636764160,"available":96583254016,"internally_reserved":0,"allocated":3088384,"data_stored":1366242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52844,"kb_used_data":3608,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318996,"statfs":{"total":96636764160,"available":96582651904,"internally_reserved":0,"allocated":3694592,"data_stored":1962869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:54.013 INFO:tasks.ceph.ceph_manager.ceph:PG 37.3a is not active+clean 2024-09-17T04:06:54.013 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.619151+0000', 'last_change': '2024-09-17T04:06:47.619151+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.619151+0000', 'last_undegraded': '2024-09-17T04:06:47.619151+0000', 'last_fullsized': '2024-09-17T04:06:47.619151+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.013 INFO:tasks.ceph.ceph_manager.ceph:PG 37.33 is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.629202+0000', 'last_change': '2024-09-17T04:06:47.629202+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.629202+0000', 'last_undegraded': '2024-09-17T04:06:47.629202+0000', 'last_fullsized': '2024-09-17T04:06:47.629202+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:PG 37.2f is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.629285+0000', 'last_change': '2024-09-17T04:06:47.629285+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.629285+0000', 'last_undegraded': '2024-09-17T04:06:47.629285+0000', 'last_fullsized': '2024-09-17T04:06:47.629285+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:PG 37.2e is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.627532+0000', 'last_change': '2024-09-17T04:06:47.627532+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.627532+0000', 'last_undegraded': '2024-09-17T04:06:47.627532+0000', 'last_fullsized': '2024-09-17T04:06:47.627532+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:PG 37.29 is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.629478+0000', 'last_change': '2024-09-17T04:06:47.629478+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.629478+0000', 'last_undegraded': '2024-09-17T04:06:47.629478+0000', 'last_fullsized': '2024-09-17T04:06:47.629478+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:PG 37.27 is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.628734+0000', 'last_change': '2024-09-17T04:06:47.628734+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.628734+0000', 'last_undegraded': '2024-09-17T04:06:47.628734+0000', 'last_fullsized': '2024-09-17T04:06:47.628734+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:PG 37.23 is not active+clean 2024-09-17T04:06:54.014 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.618364+0000', 'last_change': '2024-09-17T04:06:47.618364+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.618364+0000', 'last_undegraded': '2024-09-17T04:06:47.618364+0000', 'last_fullsized': '2024-09-17T04:06:47.618364+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:PG 37.22 is not active+clean 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.633778+0000', 'last_change': '2024-09-17T04:06:47.633778+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.633778+0000', 'last_undegraded': '2024-09-17T04:06:47.633778+0000', 'last_fullsized': '2024-09-17T04:06:47.633778+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:PG 37.1c is not active+clean 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.625495+0000', 'last_change': '2024-09-17T04:06:47.625495+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.625495+0000', 'last_undegraded': '2024-09-17T04:06:47.625495+0000', 'last_fullsized': '2024-09-17T04:06:47.625495+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:PG 37.18 is not active+clean 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.633498+0000', 'last_change': '2024-09-17T04:06:47.633498+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.633498+0000', 'last_undegraded': '2024-09-17T04:06:47.633498+0000', 'last_fullsized': '2024-09-17T04:06:47.633498+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:PG 37.19 is not active+clean 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.616816+0000', 'last_change': '2024-09-17T04:06:47.616816+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.616816+0000', 'last_undegraded': '2024-09-17T04:06:47.616816+0000', 'last_fullsized': '2024-09-17T04:06:47.616816+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:PG 37.16 is not active+clean 2024-09-17T04:06:54.015 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.619494+0000', 'last_change': '2024-09-17T04:06:47.619494+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.619494+0000', 'last_undegraded': '2024-09-17T04:06:47.619494+0000', 'last_fullsized': '2024-09-17T04:06:47.619494+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:PG 37.15 is not active+clean 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.622935+0000', 'last_change': '2024-09-17T04:06:47.622935+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.622935+0000', 'last_undegraded': '2024-09-17T04:06:47.622935+0000', 'last_fullsized': '2024-09-17T04:06:47.622935+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:PG 37.c is not active+clean 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.622812+0000', 'last_change': '2024-09-17T04:06:47.622812+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.622812+0000', 'last_undegraded': '2024-09-17T04:06:47.622812+0000', 'last_fullsized': '2024-09-17T04:06:47.622812+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:PG 37.7 is not active+clean 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.628250+0000', 'last_change': '2024-09-17T04:06:47.628250+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.628250+0000', 'last_undegraded': '2024-09-17T04:06:47.628250+0000', 'last_fullsized': '2024-09-17T04:06:47.628250+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:PG 37.2 is not active+clean 2024-09-17T04:06:54.016 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '37.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 245, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:06:47.618723+0000', 'last_change': '2024-09-17T04:06:47.618723+0000', 'last_active': '2024-09-17T04:06:47.601743+0000', 'last_peered': '2024-09-17T04:06:47.601743+0000', 'last_clean': '2024-09-17T04:06:47.601743+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:06:47.618723+0000', 'last_undegraded': '2024-09-17T04:06:47.618723+0000', 'last_fullsized': '2024-09-17T04:06:47.618723+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 245, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'last_clean_scrub_stamp': '2024-09-17T04:06:47.601743+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-17T04:06:54.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:06:54.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 -- 172.21.15.129:0/2004136059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 msgr2=0x7f47c000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 --2- 172.21.15.129:0/2004136059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47c000b9c0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f47d4001f30 tx=0x7f47c0005830 comp rx=0 tx=0).stop 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 -- 172.21.15.129:0/2004136059 shutdown_connections 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 --2- 172.21.15.129:0/2004136059 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47c000b9c0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 -- 172.21.15.129:0/2004136059 >> 172.21.15.129:0/2004136059 conn(0x7f47dc06d410 msgr2=0x7f47dc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 -- 172.21.15.129:0/2004136059 shutdown_connections 2024-09-17T04:06:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.269+0000 7f47e1e00640 1 -- 172.21.15.129:0/2004136059 wait complete. 2024-09-17T04:06:54.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.271+0000 7f47e1e00640 1 Processor -- start 2024-09-17T04:06:54.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.271+0000 7f47e1e00640 1 -- start start 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0c81b0 0x7f47dc1f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0c9620 0x7f47dc0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc0ce170 con 0x7f47c0005280 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc0afa00 con 0x7f47dc0c9620 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47e1e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc0ce2f0 con 0x7f47dc0c81b0 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47daa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47daa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52230/0 (socket says 172.21.15.129:52230) 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.273+0000 7f47daa00640 1 -- 172.21.15.129:0/631664857 learned_addr learned my addr 172.21.15.129:0/631664857 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:54.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47da000640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0c81b0 0x7f47dc1f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47db400640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0c9620 0x7f47dc0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 -- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0c81b0 msgr2=0x7f47dc1f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0c81b0 0x7f47dc1f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 -- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0c9620 msgr2=0x7f47dc0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0c9620 0x7f47dc0c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 -- 172.21.15.129:0/631664857 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc1560c0 con 0x7f47c0005280 2024-09-17T04:06:54.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47daa00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0117d0 tx=0x7f47cc011c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc004460 con 0x7f47c0005280 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47cc016070 con 0x7f47c0005280 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc1f5bc0 con 0x7f47c0005280 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00fb60 con 0x7f47c0005280 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.274+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc0aff90 con 0x7f47c0005280 2024-09-17T04:06:54.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.275+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc0ccc80 con 0x7f47c0005280 2024-09-17T04:06:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.277+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f47cc018030 con 0x7f47c0005280 2024-09-17T04:06:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.278+0000 7f47d3400640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.278+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f47cc09b560 con 0x7f47c0005280 2024-09-17T04:06:54.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.278+0000 7f47da000640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.279+0000 7f47da000640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f47c8001240 tx=0x7f47c8001110 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:54.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.282+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f47cc063a20 con 0x7f47c0005280 2024-09-17T04:06:54.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.535+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f47dc06a140 con 0x7f47c0005280 2024-09-17T04:06:54.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.536+0000 7f47d3400640 1 -- 172.21.15.129:0/631664857 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1696 (secure 0 0 0) 0x7f47cc063320 con 0x7f47c0005280 2024-09-17T04:06:54.536 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:54.536 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1065,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":247,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":81},{"state_name":"creating+peering","count":16}],"num_pgs":97,"num_pools":3,"num_objects":17,"data_bytes":591096,"bytes_used":423727104,"bytes_avail":772670386176,"bytes_total":773094113280,"inactive_pgs_ratio":0.16494844853878021,"write_bytes_sec":1709,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":235,"btime":"2024-09-17T04:06:51:636027+0000","id":18,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":18,"rank":0,"name":"c","status":"up:active","gid":11778}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":31,"modified":"2024-09-17T04:05:24.568269+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 msgr2=0x7f47bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f47c8001240 tx=0x7f47c8001110 comp rx=0 tx=0).stop 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 msgr2=0x7f47dc155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f47cc0117d0 tx=0x7f47cc011c90 comp rx=0 tx=0).stop 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 shutdown_connections 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0c9620 0x7f47dc0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0c81b0 0x7f47dc1f63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 --2- 172.21.15.129:0/631664857 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47c0005280 0x7f47dc155b80 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 >> 172.21.15.129:0/631664857 conn(0x7f47dc06d410 msgr2=0x7f47dc075d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:54.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 shutdown_connections 2024-09-17T04:06:54.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.538+0000 7f47e1e00640 1 -- 172.21.15.129:0/631664857 wait complete. 2024-09-17T04:06:54.561 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:06:54.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:54.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 -- 172.21.15.129:0/706742695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce74003a70 msgr2=0x7fce74003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 --2- 172.21.15.129:0/706742695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce74003a70 0x7fce74003ee0 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fce8000c2e0 tx=0x7fce74005830 comp rx=0 tx=0).stop 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce94e00640 1 -- 172.21.15.129:0/706742695 reap_dead start 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 -- 172.21.15.129:0/706742695 shutdown_connections 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 -- 172.21.15.129:0/706742695 >> 172.21.15.129:0/706742695 conn(0x7fce900d11a0 msgr2=0x7fce900d15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 -- 172.21.15.129:0/706742695 shutdown_connections 2024-09-17T04:06:54.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.786+0000 7fce95800640 1 -- 172.21.15.129:0/706742695 wait complete. 2024-09-17T04:06:54.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.788+0000 7fce95800640 1 Processor -- start 2024-09-17T04:06:54.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.789+0000 7fce95800640 1 -- start start 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 0x7fce901f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce9013f890 0x7fce9013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce900d0160 con 0x7fce901f3eb0 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fce900cffe0 con 0x7fce74003a70 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce95800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fce900d02e0 con 0x7fce9013f890 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce87400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce87400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52244/0 (socket says 172.21.15.129:52244) 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.790+0000 7fce87400640 1 -- 172.21.15.129:0/924236741 learned_addr learned my addr 172.21.15.129:0/924236741 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce8fe00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 0x7fce901f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce94e00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce9013f890 0x7fce9013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 -- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce9013f890 msgr2=0x7fce9013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce9013f890 0x7fce9013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 -- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 msgr2=0x7fce901f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 0x7fce901f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 -- 172.21.15.129:0/924236741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce9013ee10 con 0x7fce901f3eb0 2024-09-17T04:06:54.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce8fe00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 0x7fce901f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce87400640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fce7800e970 tx=0x7fce7800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7800ccc0 con 0x7fce901f3eb0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce7800ce60 con 0x7fce901f3eb0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce9015cee0 con 0x7fce901f3eb0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.791+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce78005260 con 0x7fce901f3eb0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.792+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce9015eff0 con 0x7fce901f3eb0 2024-09-17T04:06:54.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.792+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce9015f1f0 con 0x7fce901f3eb0 2024-09-17T04:06:54.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.794+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fce78005400 con 0x7fce901f3eb0 2024-09-17T04:06:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.795+0000 7fce8cc00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 0x7fce6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.795+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7fce7809b740 con 0x7fce901f3eb0 2024-09-17T04:06:54.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.795+0000 7fce8fe00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 0x7fce6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:54.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.796+0000 7fce8fe00640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 0x7fce6c07b0a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fce80033000 tx=0x7fce8002d000 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:54.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.799+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fce9013ee10 con 0x7fce901f3eb0 2024-09-17T04:06:54.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.978+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce90160b80 con 0x7fce6c078b90 2024-09-17T04:06:54.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.983+0000 7fce8cc00640 1 -- 172.21.15.129:0/924236741 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252446 (secure 0 0 0) 0x7fce80082a30 con 0x7fce6c078b90 2024-09-17T04:06:54.986 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:54.988 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:54.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.987+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 msgr2=0x7fce6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.987+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 0x7fce6c07b0a0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fce80033000 tx=0x7fce8002d000 comp rx=0 tx=0).stop 2024-09-17T04:06:54.989 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.987+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 msgr2=0x7fce9015c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.987+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7fce7800e970 tx=0x7fce7800ee30 comp rx=0 tx=0).stop 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 shutdown_connections 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce6c078b90 0x7fce6c07b0a0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce9013f890 0x7fce9013fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce901f3eb0 0x7fce9015c9a0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 --2- 172.21.15.129:0/924236741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce74003a70 0x7fce901f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 >> 172.21.15.129:0/924236741 conn(0x7fce900d11a0 msgr2=0x7fce900708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 shutdown_connections 2024-09-17T04:06:54.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:54.988+0000 7fce95800640 1 -- 172.21.15.129:0/924236741 wait complete. 2024-09-17T04:06:55.011 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-17T04:06:54.593117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":418140,"kb_used_data":25616,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754556580,"statfs":{"total":773094113280,"available":772665937920,"internally_reserved":0,"allocated":26230784,"data_stored":12257564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.977306"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635340+0000","last_change":"2024-09-17T04:06:48.639552+0000","last_active":"2024-09-17T04:06:49.635340+0000","last_peered":"2024-09-17T04:06:49.635340+0000","last_clean":"2024-09-17T04:06:49.635340+0000","last_became_active":"2024-09-17T04:06:48.639000+0000","last_became_peered":"2024-09-17T04:06:48.639000+0000","last_unstale":"2024-09-17T04:06:49.635340+0000","last_undegraded":"2024-09-17T04:06:49.635340+0000","last_fullsized":"2024-09-17T04:06:49.635340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:29.099857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645686+0000","last_change":"2024-09-17T04:06:48.639479+0000","last_active":"2024-09-17T04:06:48.645686+0000","last_peered":"2024-09-17T04:06:48.645686+0000","last_clean":"2024-09-17T04:06:48.645686+0000","last_became_active":"2024-09-17T04:06:48.632914+0000","last_became_peered":"2024-09-17T04:06:48.632914+0000","last_unstale":"2024-09-17T04:06:48.645686+0000","last_undegraded":"2024-09-17T04:06:48.645686+0000","last_fullsized":"2024-09-17T04:06:48.645686+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:46:41.325611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656386+0000","last_change":"2024-09-17T04:06:48.653840+0000","last_active":"2024-09-17T04:06:48.656386+0000","last_peered":"2024-09-17T04:06:48.656386+0000","last_clean":"2024-09-17T04:06:48.656386+0000","last_became_active":"2024-09-17T04:06:48.653613+0000","last_became_peered":"2024-09-17T04:06:48.653613+0000","last_unstale":"2024-09-17T04:06:48.656386+0000","last_undegraded":"2024-09-17T04:06:48.656386+0000","last_fullsized":"2024-09-17T04:06:48.656386+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:48.801325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.656192+0000","last_change":"2024-09-17T04:06:48.653715+0000","last_active":"2024-09-17T04:06:48.656192+0000","last_peered":"2024-09-17T04:06:48.656192+0000","last_clean":"2024-09-17T04:06:48.656192+0000","last_became_active":"2024-09-17T04:06:48.653119+0000","last_became_peered":"2024-09-17T04:06:48.653119+0000","last_unstale":"2024-09-17T04:06:48.656192+0000","last_undegraded":"2024-09-17T04:06:48.656192+0000","last_fullsized":"2024-09-17T04:06:48.656192+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:57:16.543513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650744+0000","last_change":"2024-09-17T04:06:48.645621+0000","last_active":"2024-09-17T04:06:48.650744+0000","last_peered":"2024-09-17T04:06:48.650744+0000","last_clean":"2024-09-17T04:06:48.650744+0000","last_became_active":"2024-09-17T04:06:48.644873+0000","last_became_peered":"2024-09-17T04:06:48.644873+0000","last_unstale":"2024-09-17T04:06:48.650744+0000","last_undegraded":"2024-09-17T04:06:48.650744+0000","last_fullsized":"2024-09-17T04:06:48.650744+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:44:06.459154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648491+0000","last_change":"2024-09-17T04:06:48.631987+0000","last_active":"2024-09-17T04:06:49.648491+0000","last_peered":"2024-09-17T04:06:49.648491+0000","last_clean":"2024-09-17T04:06:49.648491+0000","last_became_active":"2024-09-17T04:06:48.631615+0000","last_became_peered":"2024-09-17T04:06:48.631615+0000","last_unstale":"2024-09-17T04:06:49.648491+0000","last_undegraded":"2024-09-17T04:06:49.648491+0000","last_fullsized":"2024-09-17T04:06:49.648491+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:35.948167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633892+0000","last_change":"2024-09-17T04:06:48.645454+0000","last_active":"2024-09-17T04:06:49.633892+0000","last_peered":"2024-09-17T04:06:49.633892+0000","last_clean":"2024-09-17T04:06:49.633892+0000","last_became_active":"2024-09-17T04:06:48.645060+0000","last_became_peered":"2024-09-17T04:06:48.645060+0000","last_unstale":"2024-09-17T04:06:49.633892+0000","last_undegraded":"2024-09-17T04:06:49.633892+0000","last_fullsized":"2024-09-17T04:06:49.633892+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:59.853400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631646+0000","last_change":"2024-09-17T04:06:48.636070+0000","last_active":"2024-09-17T04:06:49.631646+0000","last_peered":"2024-09-17T04:06:49.631646+0000","last_clean":"2024-09-17T04:06:49.631646+0000","last_became_active":"2024-09-17T04:06:48.635753+0000","last_became_peered":"2024-09-17T04:06:48.635753+0000","last_unstale":"2024-09-17T04:06:49.631646+0000","last_undegraded":"2024-09-17T04:06:49.631646+0000","last_fullsized":"2024-09-17T04:06:49.631646+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:35:37.356554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645657+0000","last_change":"2024-09-17T04:06:48.640090+0000","last_active":"2024-09-17T04:06:48.645657+0000","last_peered":"2024-09-17T04:06:48.645657+0000","last_clean":"2024-09-17T04:06:48.645657+0000","last_became_active":"2024-09-17T04:06:48.635312+0000","last_became_peered":"2024-09-17T04:06:48.635312+0000","last_unstale":"2024-09-17T04:06:48.645657+0000","last_undegraded":"2024-09-17T04:06:48.645657+0000","last_fullsized":"2024-09-17T04:06:48.645657+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:06.548364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635456+0000","last_change":"2024-09-17T04:06:48.632182+0000","last_active":"2024-09-17T04:06:49.635456+0000","last_peered":"2024-09-17T04:06:49.635456+0000","last_clean":"2024-09-17T04:06:49.635456+0000","last_became_active":"2024-09-17T04:06:48.631841+0000","last_became_peered":"2024-09-17T04:06:48.631841+0000","last_unstale":"2024-09-17T04:06:49.635456+0000","last_undegraded":"2024-09-17T04:06:49.635456+0000","last_fullsized":"2024-09-17T04:06:49.635456+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:26.532740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632045+0000","last_change":"2024-09-17T04:06:48.635273+0000","last_active":"2024-09-17T04:06:49.632045+0000","last_peered":"2024-09-17T04:06:49.632045+0000","last_clean":"2024-09-17T04:06:49.632045+0000","last_became_active":"2024-09-17T04:06:48.634916+0000","last_became_peered":"2024-09-17T04:06:48.634916+0000","last_unstale":"2024-09-17T04:06:49.632045+0000","last_undegraded":"2024-09-17T04:06:49.632045+0000","last_fullsized":"2024-09-17T04:06:49.632045+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:46.226736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649549+0000","last_change":"2024-09-17T04:06:48.654714+0000","last_active":"2024-09-17T04:06:49.649549+0000","last_peered":"2024-09-17T04:06:49.649549+0000","last_clean":"2024-09-17T04:06:49.649549+0000","last_became_active":"2024-09-17T04:06:48.654168+0000","last_became_peered":"2024-09-17T04:06:48.654168+0000","last_unstale":"2024-09-17T04:06:49.649549+0000","last_undegraded":"2024-09-17T04:06:49.649549+0000","last_fullsized":"2024-09-17T04:06:49.649549+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:54.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634298+0000","last_change":"2024-09-17T04:06:48.625470+0000","last_active":"2024-09-17T04:06:49.634298+0000","last_peered":"2024-09-17T04:06:49.634298+0000","last_clean":"2024-09-17T04:06:49.634298+0000","last_became_active":"2024-09-17T04:06:48.624818+0000","last_became_peered":"2024-09-17T04:06:48.624818+0000","last_unstale":"2024-09-17T04:06:49.634298+0000","last_undegraded":"2024-09-17T04:06:49.634298+0000","last_fullsized":"2024-09-17T04:06:49.634298+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:14:17.324847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648452+0000","last_change":"2024-09-17T04:06:48.635524+0000","last_active":"2024-09-17T04:06:49.648452+0000","last_peered":"2024-09-17T04:06:49.648452+0000","last_clean":"2024-09-17T04:06:49.648452+0000","last_became_active":"2024-09-17T04:06:48.634740+0000","last_became_peered":"2024-09-17T04:06:48.634740+0000","last_unstale":"2024-09-17T04:06:49.648452+0000","last_undegraded":"2024-09-17T04:06:49.648452+0000","last_fullsized":"2024-09-17T04:06:49.648452+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:10.649399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.648208+0000","last_change":"2024-09-17T04:06:48.639136+0000","last_active":"2024-09-17T04:06:48.648208+0000","last_peered":"2024-09-17T04:06:48.648208+0000","last_clean":"2024-09-17T04:06:48.648208+0000","last_became_active":"2024-09-17T04:06:48.638849+0000","last_became_peered":"2024-09-17T04:06:48.638849+0000","last_unstale":"2024-09-17T04:06:48.648208+0000","last_undegraded":"2024-09-17T04:06:48.648208+0000","last_fullsized":"2024-09-17T04:06:48.648208+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:20:04.361484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649378+0000","last_change":"2024-09-17T04:06:48.635932+0000","last_active":"2024-09-17T04:06:49.649378+0000","last_peered":"2024-09-17T04:06:49.649378+0000","last_clean":"2024-09-17T04:06:49.649378+0000","last_became_active":"2024-09-17T04:06:48.634277+0000","last_became_peered":"2024-09-17T04:06:48.634277+0000","last_unstale":"2024-09-17T04:06:49.649378+0000","last_undegraded":"2024-09-17T04:06:49.649378+0000","last_fullsized":"2024-09-17T04:06:49.649378+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.745727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633917+0000","last_change":"2024-09-17T04:06:48.651905+0000","last_active":"2024-09-17T04:06:49.633917+0000","last_peered":"2024-09-17T04:06:49.633917+0000","last_clean":"2024-09-17T04:06:49.633917+0000","last_became_active":"2024-09-17T04:06:48.651000+0000","last_became_peered":"2024-09-17T04:06:48.651000+0000","last_unstale":"2024-09-17T04:06:49.633917+0000","last_undegraded":"2024-09-17T04:06:49.633917+0000","last_fullsized":"2024-09-17T04:06:49.633917+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:59.428221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633764+0000","last_change":"2024-09-17T04:06:48.623622+0000","last_active":"2024-09-17T04:06:49.633764+0000","last_peered":"2024-09-17T04:06:49.633764+0000","last_clean":"2024-09-17T04:06:49.633764+0000","last_became_active":"2024-09-17T04:06:48.623254+0000","last_became_peered":"2024-09-17T04:06:48.623254+0000","last_unstale":"2024-09-17T04:06:49.633764+0000","last_undegraded":"2024-09-17T04:06:49.633764+0000","last_fullsized":"2024-09-17T04:06:49.633764+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.528806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648913+0000","last_change":"2024-09-17T04:06:48.643193+0000","last_active":"2024-09-17T04:06:49.648913+0000","last_peered":"2024-09-17T04:06:49.648913+0000","last_clean":"2024-09-17T04:06:49.648913+0000","last_became_active":"2024-09-17T04:06:48.641658+0000","last_became_peered":"2024-09-17T04:06:48.641658+0000","last_unstale":"2024-09-17T04:06:49.648913+0000","last_undegraded":"2024-09-17T04:06:49.648913+0000","last_fullsized":"2024-09-17T04:06:49.648913+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:50.000607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.645968+0000","last_change":"2024-09-17T04:06:48.641052+0000","last_active":"2024-09-17T04:06:48.645968+0000","last_peered":"2024-09-17T04:06:48.645968+0000","last_clean":"2024-09-17T04:06:48.645968+0000","last_became_active":"2024-09-17T04:06:48.640727+0000","last_became_peered":"2024-09-17T04:06:48.640727+0000","last_unstale":"2024-09-17T04:06:48.645968+0000","last_undegraded":"2024-09-17T04:06:48.645968+0000","last_fullsized":"2024-09-17T04:06:48.645968+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:55:57.495816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631729+0000","last_change":"2024-09-17T04:06:48.638889+0000","last_active":"2024-09-17T04:06:49.631729+0000","last_peered":"2024-09-17T04:06:49.631729+0000","last_clean":"2024-09-17T04:06:49.631729+0000","last_became_active":"2024-09-17T04:06:48.638320+0000","last_became_peered":"2024-09-17T04:06:48.638320+0000","last_unstale":"2024-09-17T04:06:49.631729+0000","last_undegraded":"2024-09-17T04:06:49.631729+0000","last_fullsized":"2024-09-17T04:06:49.631729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:06.457607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650679+0000","last_change":"2024-09-17T04:06:48.640395+0000","last_active":"2024-09-17T04:06:48.650679+0000","last_peered":"2024-09-17T04:06:48.650679+0000","last_clean":"2024-09-17T04:06:48.650679+0000","last_became_active":"2024-09-17T04:06:48.640063+0000","last_became_peered":"2024-09-17T04:06:48.640063+0000","last_unstale":"2024-09-17T04:06:48.650679+0000","last_undegraded":"2024-09-17T04:06:48.650679+0000","last_fullsized":"2024-09-17T04:06:48.650679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:19:30.759186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633679+0000","last_change":"2024-09-17T04:06:48.624960+0000","last_active":"2024-09-17T04:06:49.633679+0000","last_peered":"2024-09-17T04:06:49.633679+0000","last_clean":"2024-09-17T04:06:49.633679+0000","last_became_active":"2024-09-17T04:06:48.624594+0000","last_became_peered":"2024-09-17T04:06:48.624594+0000","last_unstale":"2024-09-17T04:06:49.633679+0000","last_undegraded":"2024-09-17T04:06:49.633679+0000","last_fullsized":"2024-09-17T04:06:49.633679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:44.161205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648399+0000","last_change":"2024-09-17T04:06:48.641161+0000","last_active":"2024-09-17T04:06:49.648399+0000","last_peered":"2024-09-17T04:06:49.648399+0000","last_clean":"2024-09-17T04:06:49.648399+0000","last_became_active":"2024-09-17T04:06:48.640811+0000","last_became_peered":"2024-09-17T04:06:48.640811+0000","last_unstale":"2024-09-17T04:06:49.648399+0000","last_undegraded":"2024-09-17T04:06:49.648399+0000","last_fullsized":"2024-09-17T04:06:49.648399+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:30.671650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634072+0000","last_change":"2024-09-17T04:06:48.638436+0000","last_active":"2024-09-17T04:06:49.634072+0000","last_peered":"2024-09-17T04:06:49.634072+0000","last_clean":"2024-09-17T04:06:49.634072+0000","last_became_active":"2024-09-17T04:06:48.637635+0000","last_became_peered":"2024-09-17T04:06:48.637635+0000","last_unstale":"2024-09-17T04:06:49.634072+0000","last_undegraded":"2024-09-17T04:06:49.634072+0000","last_fullsized":"2024-09-17T04:06:49.634072+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:10.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635253+0000","last_change":"2024-09-17T04:06:48.646124+0000","last_active":"2024-09-17T04:06:49.635253+0000","last_peered":"2024-09-17T04:06:49.635253+0000","last_clean":"2024-09-17T04:06:49.635253+0000","last_became_active":"2024-09-17T04:06:48.645673+0000","last_became_peered":"2024-09-17T04:06:48.645673+0000","last_unstale":"2024-09-17T04:06:49.635253+0000","last_undegraded":"2024-09-17T04:06:49.635253+0000","last_fullsized":"2024-09-17T04:06:49.635253+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:54.659533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649892+0000","last_change":"2024-09-17T04:06:48.637878+0000","last_active":"2024-09-17T04:06:49.649892+0000","last_peered":"2024-09-17T04:06:49.649892+0000","last_clean":"2024-09-17T04:06:49.649892+0000","last_became_active":"2024-09-17T04:06:48.637580+0000","last_became_peered":"2024-09-17T04:06:48.637580+0000","last_unstale":"2024-09-17T04:06:49.649892+0000","last_undegraded":"2024-09-17T04:06:49.649892+0000","last_fullsized":"2024-09-17T04:06:49.649892+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.874073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648941+0000","last_change":"2024-09-17T04:06:48.654016+0000","last_active":"2024-09-17T04:06:49.648941+0000","last_peered":"2024-09-17T04:06:49.648941+0000","last_clean":"2024-09-17T04:06:49.648941+0000","last_became_active":"2024-09-17T04:06:48.650905+0000","last_became_peered":"2024-09-17T04:06:48.650905+0000","last_unstale":"2024-09-17T04:06:49.648941+0000","last_undegraded":"2024-09-17T04:06:49.648941+0000","last_fullsized":"2024-09-17T04:06:49.648941+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:35.043475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634406+0000","last_change":"2024-09-17T04:06:48.638284+0000","last_active":"2024-09-17T04:06:49.634406+0000","last_peered":"2024-09-17T04:06:49.634406+0000","last_clean":"2024-09-17T04:06:49.634406+0000","last_became_active":"2024-09-17T04:06:48.637899+0000","last_became_peered":"2024-09-17T04:06:48.637899+0000","last_unstale":"2024-09-17T04:06:49.634406+0000","last_undegraded":"2024-09-17T04:06:49.634406+0000","last_fullsized":"2024-09-17T04:06:49.634406+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:51.007347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648462+0000","last_change":"2024-09-17T04:06:48.643887+0000","last_active":"2024-09-17T04:06:49.648462+0000","last_peered":"2024-09-17T04:06:49.648462+0000","last_clean":"2024-09-17T04:06:49.648462+0000","last_became_active":"2024-09-17T04:06:48.643541+0000","last_became_peered":"2024-09-17T04:06:48.643541+0000","last_unstale":"2024-09-17T04:06:49.648462+0000","last_undegraded":"2024-09-17T04:06:49.648462+0000","last_fullsized":"2024-09-17T04:06:49.648462+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:54.252745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650895+0000","last_change":"2024-09-17T04:06:48.646487+0000","last_active":"2024-09-17T04:06:48.650895+0000","last_peered":"2024-09-17T04:06:48.650895+0000","last_clean":"2024-09-17T04:06:48.650895+0000","last_became_active":"2024-09-17T04:06:48.646200+0000","last_became_peered":"2024-09-17T04:06:48.646200+0000","last_unstale":"2024-09-17T04:06:48.650895+0000","last_undegraded":"2024-09-17T04:06:48.650895+0000","last_fullsized":"2024-09-17T04:06:48.650895+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:19:32.053314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649893+0000","last_change":"2024-09-17T04:06:48.638450+0000","last_active":"2024-09-17T04:06:49.649893+0000","last_peered":"2024-09-17T04:06:49.649893+0000","last_clean":"2024-09-17T04:06:49.649893+0000","last_became_active":"2024-09-17T04:06:48.638148+0000","last_became_peered":"2024-09-17T04:06:48.638148+0000","last_unstale":"2024-09-17T04:06:49.649893+0000","last_undegraded":"2024-09-17T04:06:49.649893+0000","last_fullsized":"2024-09-17T04:06:49.649893+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:35.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"246'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649000+0000","last_change":"2024-09-17T04:06:47.630685+0000","last_active":"2024-09-17T04:06:49.649000+0000","last_peered":"2024-09-17T04:06:49.649000+0000","last_clean":"2024-09-17T04:06:49.649000+0000","last_became_active":"2024-09-17T04:06:47.630367+0000","last_became_peered":"2024-09-17T04:06:47.630367+0000","last_unstale":"2024-09-17T04:06:49.649000+0000","last_undegraded":"2024-09-17T04:06:49.649000+0000","last_fullsized":"2024-09-17T04:06:49.649000+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:56.891603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648792+0000","last_change":"2024-09-17T04:06:48.636369+0000","last_active":"2024-09-17T04:06:49.648792+0000","last_peered":"2024-09-17T04:06:49.648792+0000","last_clean":"2024-09-17T04:06:49.648792+0000","last_became_active":"2024-09-17T04:06:48.636001+0000","last_became_peered":"2024-09-17T04:06:48.636001+0000","last_unstale":"2024-09-17T04:06:49.648792+0000","last_undegraded":"2024-09-17T04:06:49.648792+0000","last_fullsized":"2024-09-17T04:06:49.648792+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:35.768906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649458+0000","last_change":"2024-09-17T04:06:47.640788+0000","last_active":"2024-09-17T04:06:49.649458+0000","last_peered":"2024-09-17T04:06:49.649458+0000","last_clean":"2024-09-17T04:06:49.649458+0000","last_became_active":"2024-09-17T04:06:47.640507+0000","last_became_peered":"2024-09-17T04:06:47.640507+0000","last_unstale":"2024-09-17T04:06:49.649458+0000","last_undegraded":"2024-09-17T04:06:49.649458+0000","last_fullsized":"2024-09-17T04:06:49.649458+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:01.560979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648684+0000","last_change":"2024-09-17T04:06:48.627725+0000","last_active":"2024-09-17T04:06:49.648684+0000","last_peered":"2024-09-17T04:06:49.648684+0000","last_clean":"2024-09-17T04:06:49.648684+0000","last_became_active":"2024-09-17T04:06:48.626509+0000","last_became_peered":"2024-09-17T04:06:48.626509+0000","last_unstale":"2024-09-17T04:06:49.648684+0000","last_undegraded":"2024-09-17T04:06:49.648684+0000","last_fullsized":"2024-09-17T04:06:49.648684+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:51.007347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649085+0000","last_change":"2024-09-17T04:06:47.640338+0000","last_active":"2024-09-17T04:06:49.649085+0000","last_peered":"2024-09-17T04:06:49.649085+0000","last_clean":"2024-09-17T04:06:49.649085+0000","last_became_active":"2024-09-17T04:06:47.639781+0000","last_became_peered":"2024-09-17T04:06:47.639781+0000","last_unstale":"2024-09-17T04:06:49.649085+0000","last_undegraded":"2024-09-17T04:06:49.649085+0000","last_fullsized":"2024-09-17T04:06:49.649085+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:34.875218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632123+0000","last_change":"2024-09-17T04:06:48.641644+0000","last_active":"2024-09-17T04:06:49.632123+0000","last_peered":"2024-09-17T04:06:49.632123+0000","last_clean":"2024-09-17T04:06:49.632123+0000","last_became_active":"2024-09-17T04:06:48.641054+0000","last_became_peered":"2024-09-17T04:06:48.641054+0000","last_unstale":"2024-09-17T04:06:49.632123+0000","last_undegraded":"2024-09-17T04:06:49.632123+0000","last_fullsized":"2024-09-17T04:06:49.632123+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:37.921980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660581+0000","last_change":"2024-09-17T04:06:47.634854+0000","last_active":"2024-09-17T04:06:48.660581+0000","last_peered":"2024-09-17T04:06:48.660581+0000","last_clean":"2024-09-17T04:06:48.660581+0000","last_became_active":"2024-09-17T04:06:47.634556+0000","last_became_peered":"2024-09-17T04:06:47.634556+0000","last_unstale":"2024-09-17T04:06:48.660581+0000","last_undegraded":"2024-09-17T04:06:48.660581+0000","last_fullsized":"2024-09-17T04:06:48.660581+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:55:59.294589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632270+0000","last_change":"2024-09-17T04:06:48.633413+0000","last_active":"2024-09-17T04:06:49.632270+0000","last_peered":"2024-09-17T04:06:49.632270+0000","last_clean":"2024-09-17T04:06:49.632270+0000","last_became_active":"2024-09-17T04:06:48.632689+0000","last_became_peered":"2024-09-17T04:06:48.632689+0000","last_unstale":"2024-09-17T04:06:49.632270+0000","last_undegraded":"2024-09-17T04:06:49.632270+0000","last_fullsized":"2024-09-17T04:06:49.632270+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:21.876560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649467+0000","last_change":"2024-09-17T04:06:47.637532+0000","last_active":"2024-09-17T04:06:49.649467+0000","last_peered":"2024-09-17T04:06:49.649467+0000","last_clean":"2024-09-17T04:06:49.649467+0000","last_became_active":"2024-09-17T04:06:47.637150+0000","last_became_peered":"2024-09-17T04:06:47.637150+0000","last_unstale":"2024-09-17T04:06:49.649467+0000","last_undegraded":"2024-09-17T04:06:49.649467+0000","last_fullsized":"2024-09-17T04:06:49.649467+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:04.474254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.632926+0000","last_change":"2024-09-17T04:06:48.625119+0000","last_active":"2024-09-17T04:06:48.632926+0000","last_peered":"2024-09-17T04:06:48.632926+0000","last_clean":"2024-09-17T04:06:48.632926+0000","last_became_active":"2024-09-17T04:06:48.624823+0000","last_became_peered":"2024-09-17T04:06:48.624823+0000","last_unstale":"2024-09-17T04:06:48.632926+0000","last_undegraded":"2024-09-17T04:06:48.632926+0000","last_fullsized":"2024-09-17T04:06:48.632926+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:21.840133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634657+0000","last_change":"2024-09-17T04:06:47.636767+0000","last_active":"2024-09-17T04:06:49.634657+0000","last_peered":"2024-09-17T04:06:49.634657+0000","last_clean":"2024-09-17T04:06:49.634657+0000","last_became_active":"2024-09-17T04:06:47.635877+0000","last_became_peered":"2024-09-17T04:06:47.635877+0000","last_unstale":"2024-09-17T04:06:49.634657+0000","last_undegraded":"2024-09-17T04:06:49.634657+0000","last_fullsized":"2024-09-17T04:06:49.634657+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:36.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634600+0000","last_change":"2024-09-17T04:06:48.647257+0000","last_active":"2024-09-17T04:06:49.634600+0000","last_peered":"2024-09-17T04:06:49.634600+0000","last_clean":"2024-09-17T04:06:49.634600+0000","last_became_active":"2024-09-17T04:06:48.646916+0000","last_became_peered":"2024-09-17T04:06:48.646916+0000","last_unstale":"2024-09-17T04:06:49.634600+0000","last_undegraded":"2024-09-17T04:06:49.634600+0000","last_fullsized":"2024-09-17T04:06:49.634600+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:19.995005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649315+0000","last_change":"2024-09-17T04:06:47.641518+0000","last_active":"2024-09-17T04:06:49.649315+0000","last_peered":"2024-09-17T04:06:49.649315+0000","last_clean":"2024-09-17T04:06:49.649315+0000","last_became_active":"2024-09-17T04:06:47.641158+0000","last_became_peered":"2024-09-17T04:06:47.641158+0000","last_unstale":"2024-09-17T04:06:49.649315+0000","last_undegraded":"2024-09-17T04:06:49.649315+0000","last_fullsized":"2024-09-17T04:06:49.649315+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:43.208862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648503+0000","last_change":"2024-09-17T04:06:48.626505+0000","last_active":"2024-09-17T04:06:49.648503+0000","last_peered":"2024-09-17T04:06:49.648503+0000","last_clean":"2024-09-17T04:06:49.648503+0000","last_became_active":"2024-09-17T04:06:48.624676+0000","last_became_peered":"2024-09-17T04:06:48.624676+0000","last_unstale":"2024-09-17T04:06:49.648503+0000","last_undegraded":"2024-09-17T04:06:49.648503+0000","last_fullsized":"2024-09-17T04:06:49.648503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:09.677668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648155+0000","last_change":"2024-09-17T04:06:47.637605+0000","last_active":"2024-09-17T04:06:49.648155+0000","last_peered":"2024-09-17T04:06:49.648155+0000","last_clean":"2024-09-17T04:06:49.648155+0000","last_became_active":"2024-09-17T04:06:47.637251+0000","last_became_peered":"2024-09-17T04:06:47.637251+0000","last_unstale":"2024-09-17T04:06:49.648155+0000","last_undegraded":"2024-09-17T04:06:49.648155+0000","last_fullsized":"2024-09-17T04:06:49.648155+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:56.783152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649374+0000","last_change":"2024-09-17T04:06:48.639801+0000","last_active":"2024-09-17T04:06:49.649374+0000","last_peered":"2024-09-17T04:06:49.649374+0000","last_clean":"2024-09-17T04:06:49.649374+0000","last_became_active":"2024-09-17T04:06:48.639471+0000","last_became_peered":"2024-09-17T04:06:48.639471+0000","last_unstale":"2024-09-17T04:06:49.649374+0000","last_undegraded":"2024-09-17T04:06:49.649374+0000","last_fullsized":"2024-09-17T04:06:49.649374+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:47.931408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649097+0000","last_change":"2024-09-17T04:06:47.639900+0000","last_active":"2024-09-17T04:06:49.649097+0000","last_peered":"2024-09-17T04:06:49.649097+0000","last_clean":"2024-09-17T04:06:49.649097+0000","last_became_active":"2024-09-17T04:06:47.638771+0000","last_became_peered":"2024-09-17T04:06:47.638771+0000","last_unstale":"2024-09-17T04:06:49.649097+0000","last_undegraded":"2024-09-17T04:06:49.649097+0000","last_fullsized":"2024-09-17T04:06:49.649097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:42.102677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648930+0000","last_change":"2024-09-17T04:06:48.637971+0000","last_active":"2024-09-17T04:06:49.648930+0000","last_peered":"2024-09-17T04:06:49.648930+0000","last_clean":"2024-09-17T04:06:49.648930+0000","last_became_active":"2024-09-17T04:06:48.637655+0000","last_became_peered":"2024-09-17T04:06:48.637655+0000","last_unstale":"2024-09-17T04:06:49.648930+0000","last_undegraded":"2024-09-17T04:06:49.648930+0000","last_fullsized":"2024-09-17T04:06:49.648930+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:31.912869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648702+0000","last_change":"2024-09-17T04:06:47.628105+0000","last_active":"2024-09-17T04:06:49.648702+0000","last_peered":"2024-09-17T04:06:49.648702+0000","last_clean":"2024-09-17T04:06:49.648702+0000","last_became_active":"2024-09-17T04:06:47.627756+0000","last_became_peered":"2024-09-17T04:06:47.627756+0000","last_unstale":"2024-09-17T04:06:49.648702+0000","last_undegraded":"2024-09-17T04:06:49.648702+0000","last_fullsized":"2024-09-17T04:06:49.648702+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:38.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653352+0000","last_change":"2024-09-17T04:06:48.646524+0000","last_active":"2024-09-17T04:06:48.653352+0000","last_peered":"2024-09-17T04:06:48.653352+0000","last_clean":"2024-09-17T04:06:48.653352+0000","last_became_active":"2024-09-17T04:06:48.646148+0000","last_became_peered":"2024-09-17T04:06:48.646148+0000","last_unstale":"2024-09-17T04:06:48.653352+0000","last_undegraded":"2024-09-17T04:06:48.653352+0000","last_fullsized":"2024-09-17T04:06:48.653352+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:29:48.285920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"246'3","reported_seq":16,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.661105+0000","last_change":"2024-09-17T04:06:47.635826+0000","last_active":"2024-09-17T04:06:48.661105+0000","last_peered":"2024-09-17T04:06:48.661105+0000","last_clean":"2024-09-17T04:06:48.661105+0000","last_became_active":"2024-09-17T04:06:47.633375+0000","last_became_peered":"2024-09-17T04:06:47.633375+0000","last_unstale":"2024-09-17T04:06:48.661105+0000","last_undegraded":"2024-09-17T04:06:48.661105+0000","last_fullsized":"2024-09-17T04:06:48.661105+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:13.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649114+0000","last_change":"2024-09-17T04:06:48.645322+0000","last_active":"2024-09-17T04:06:49.649114+0000","last_peered":"2024-09-17T04:06:49.649114+0000","last_clean":"2024-09-17T04:06:49.649114+0000","last_became_active":"2024-09-17T04:06:48.644967+0000","last_became_peered":"2024-09-17T04:06:48.644967+0000","last_unstale":"2024-09-17T04:06:49.649114+0000","last_undegraded":"2024-09-17T04:06:49.649114+0000","last_fullsized":"2024-09-17T04:06:49.649114+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:48:03.555488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"246'6","reported_seq":21,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633743+0000","last_change":"2024-09-17T04:06:47.638185+0000","last_active":"2024-09-17T04:06:49.633743+0000","last_peered":"2024-09-17T04:06:49.633743+0000","last_clean":"2024-09-17T04:06:49.633743+0000","last_became_active":"2024-09-17T04:06:47.637416+0000","last_became_peered":"2024-09-17T04:06:47.637416+0000","last_unstale":"2024-09-17T04:06:49.633743+0000","last_undegraded":"2024-09-17T04:06:49.633743+0000","last_fullsized":"2024-09-17T04:06:49.633743+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:20.160754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648732+0000","last_change":"2024-09-17T04:06:48.647751+0000","last_active":"2024-09-17T04:06:49.648732+0000","last_peered":"2024-09-17T04:06:49.648732+0000","last_clean":"2024-09-17T04:06:49.648732+0000","last_became_active":"2024-09-17T04:06:48.647416+0000","last_became_peered":"2024-09-17T04:06:48.647416+0000","last_unstale":"2024-09-17T04:06:49.648732+0000","last_undegraded":"2024-09-17T04:06:49.648732+0000","last_fullsized":"2024-09-17T04:06:49.648732+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:55.210588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631957+0000","last_change":"2024-09-17T04:06:47.640722+0000","last_active":"2024-09-17T04:06:49.631957+0000","last_peered":"2024-09-17T04:06:49.631957+0000","last_clean":"2024-09-17T04:06:49.631957+0000","last_became_active":"2024-09-17T04:06:47.640406+0000","last_became_peered":"2024-09-17T04:06:47.640406+0000","last_unstale":"2024-09-17T04:06:49.631957+0000","last_undegraded":"2024-09-17T04:06:49.631957+0000","last_fullsized":"2024-09-17T04:06:49.631957+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:40.791049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634698+0000","last_change":"2024-09-17T04:06:48.628835+0000","last_active":"2024-09-17T04:06:49.634698+0000","last_peered":"2024-09-17T04:06:49.634698+0000","last_clean":"2024-09-17T04:06:49.634698+0000","last_became_active":"2024-09-17T04:06:48.628468+0000","last_became_peered":"2024-09-17T04:06:48.628468+0000","last_unstale":"2024-09-17T04:06:49.634698+0000","last_undegraded":"2024-09-17T04:06:49.634698+0000","last_fullsized":"2024-09-17T04:06:49.634698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:12.456213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.637573+0000","last_change":"2024-09-17T04:06:47.629947+0000","last_active":"2024-09-17T04:06:48.637573+0000","last_peered":"2024-09-17T04:06:48.637573+0000","last_clean":"2024-09-17T04:06:48.637573+0000","last_became_active":"2024-09-17T04:06:47.628403+0000","last_became_peered":"2024-09-17T04:06:47.628403+0000","last_unstale":"2024-09-17T04:06:48.637573+0000","last_undegraded":"2024-09-17T04:06:48.637573+0000","last_fullsized":"2024-09-17T04:06:48.637573+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:15.090695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649384+0000","last_change":"2024-09-17T04:06:48.650761+0000","last_active":"2024-09-17T04:06:49.649384+0000","last_peered":"2024-09-17T04:06:49.649384+0000","last_clean":"2024-09-17T04:06:49.649384+0000","last_became_active":"2024-09-17T04:06:48.650386+0000","last_became_peered":"2024-09-17T04:06:48.650386+0000","last_unstale":"2024-09-17T04:06:49.649384+0000","last_undegraded":"2024-09-17T04:06:49.649384+0000","last_fullsized":"2024-09-17T04:06:49.649384+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:16:27.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633982+0000","last_change":"2024-09-17T04:06:47.637094+0000","last_active":"2024-09-17T04:06:49.633982+0000","last_peered":"2024-09-17T04:06:49.633982+0000","last_clean":"2024-09-17T04:06:49.633982+0000","last_became_active":"2024-09-17T04:06:47.636787+0000","last_became_peered":"2024-09-17T04:06:47.636787+0000","last_unstale":"2024-09-17T04:06:49.633982+0000","last_undegraded":"2024-09-17T04:06:49.633982+0000","last_fullsized":"2024-09-17T04:06:49.633982+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:24.521931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634167+0000","last_change":"2024-09-17T04:06:48.642186+0000","last_active":"2024-09-17T04:06:49.634167+0000","last_peered":"2024-09-17T04:06:49.634167+0000","last_clean":"2024-09-17T04:06:49.634167+0000","last_became_active":"2024-09-17T04:06:48.640270+0000","last_became_peered":"2024-09-17T04:06:48.640270+0000","last_unstale":"2024-09-17T04:06:49.634167+0000","last_undegraded":"2024-09-17T04:06:49.634167+0000","last_fullsized":"2024-09-17T04:06:49.634167+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:23.763556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"246'8","reported_seq":21,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.665782+0000","last_change":"2024-09-17T04:06:47.639284+0000","last_active":"2024-09-17T04:06:48.665782+0000","last_peered":"2024-09-17T04:06:48.665782+0000","last_clean":"2024-09-17T04:06:48.665782+0000","last_became_active":"2024-09-17T04:06:47.638987+0000","last_became_peered":"2024-09-17T04:06:47.638987+0000","last_unstale":"2024-09-17T04:06:48.665782+0000","last_undegraded":"2024-09-17T04:06:48.665782+0000","last_fullsized":"2024-09-17T04:06:48.665782+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:25.636357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650612+0000","last_change":"2024-09-17T04:06:48.640025+0000","last_active":"2024-09-17T04:06:48.650612+0000","last_peered":"2024-09-17T04:06:48.650612+0000","last_clean":"2024-09-17T04:06:48.650612+0000","last_became_active":"2024-09-17T04:06:48.639713+0000","last_became_peered":"2024-09-17T04:06:48.639713+0000","last_unstale":"2024-09-17T04:06:48.650612+0000","last_undegraded":"2024-09-17T04:06:48.650612+0000","last_fullsized":"2024-09-17T04:06:48.650612+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:09:46.136192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632203+0000","last_change":"2024-09-17T04:06:47.627885+0000","last_active":"2024-09-17T04:06:49.632203+0000","last_peered":"2024-09-17T04:06:49.632203+0000","last_clean":"2024-09-17T04:06:49.632203+0000","last_became_active":"2024-09-17T04:06:47.627314+0000","last_became_peered":"2024-09-17T04:06:47.627314+0000","last_unstale":"2024-09-17T04:06:49.632203+0000","last_undegraded":"2024-09-17T04:06:49.632203+0000","last_fullsized":"2024-09-17T04:06:49.632203+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:32.996351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650153+0000","last_change":"2024-09-17T04:06:48.640039+0000","last_active":"2024-09-17T04:06:49.650153+0000","last_peered":"2024-09-17T04:06:49.650153+0000","last_clean":"2024-09-17T04:06:49.650153+0000","last_became_active":"2024-09-17T04:06:48.639062+0000","last_became_peered":"2024-09-17T04:06:48.639062+0000","last_unstale":"2024-09-17T04:06:49.650153+0000","last_undegraded":"2024-09-17T04:06:49.650153+0000","last_fullsized":"2024-09-17T04:06:49.650153+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.537308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648863+0000","last_change":"2024-09-17T04:06:47.626021+0000","last_active":"2024-09-17T04:06:49.648863+0000","last_peered":"2024-09-17T04:06:49.648863+0000","last_clean":"2024-09-17T04:06:49.648863+0000","last_became_active":"2024-09-17T04:06:47.625353+0000","last_became_peered":"2024-09-17T04:06:47.625353+0000","last_unstale":"2024-09-17T04:06:49.648863+0000","last_undegraded":"2024-09-17T04:06:49.648863+0000","last_fullsized":"2024-09-17T04:06:49.648863+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649068+0000","last_change":"2024-09-17T04:06:48.650829+0000","last_active":"2024-09-17T04:06:49.649068+0000","last_peered":"2024-09-17T04:06:49.649068+0000","last_clean":"2024-09-17T04:06:49.649068+0000","last_became_active":"2024-09-17T04:06:48.649204+0000","last_became_peered":"2024-09-17T04:06:48.649204+0000","last_unstale":"2024-09-17T04:06:49.649068+0000","last_undegraded":"2024-09-17T04:06:49.649068+0000","last_fullsized":"2024-09-17T04:06:49.649068+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:35.431985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649197+0000","last_change":"2024-09-17T04:06:47.630827+0000","last_active":"2024-09-17T04:06:49.649197+0000","last_peered":"2024-09-17T04:06:49.649197+0000","last_clean":"2024-09-17T04:06:49.649197+0000","last_became_active":"2024-09-17T04:06:47.630146+0000","last_became_peered":"2024-09-17T04:06:47.630146+0000","last_unstale":"2024-09-17T04:06:49.649197+0000","last_undegraded":"2024-09-17T04:06:49.649197+0000","last_fullsized":"2024-09-17T04:06:49.649197+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:59.862903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649039+0000","last_change":"2024-09-17T04:06:48.655091+0000","last_active":"2024-09-17T04:06:49.649039+0000","last_peered":"2024-09-17T04:06:49.649039+0000","last_clean":"2024-09-17T04:06:49.649039+0000","last_became_active":"2024-09-17T04:06:48.654782+0000","last_became_peered":"2024-09-17T04:06:48.654782+0000","last_unstale":"2024-09-17T04:06:49.649039+0000","last_undegraded":"2024-09-17T04:06:49.649039+0000","last_fullsized":"2024-09-17T04:06:49.649039+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:39.823146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649147+0000","last_change":"2024-09-17T04:06:47.639967+0000","last_active":"2024-09-17T04:06:49.649147+0000","last_peered":"2024-09-17T04:06:49.649147+0000","last_clean":"2024-09-17T04:06:49.649147+0000","last_became_active":"2024-09-17T04:06:47.639606+0000","last_became_peered":"2024-09-17T04:06:47.639606+0000","last_unstale":"2024-09-17T04:06:49.649147+0000","last_undegraded":"2024-09-17T04:06:49.649147+0000","last_fullsized":"2024-09-17T04:06:49.649147+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:53.671166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649644+0000","last_change":"2024-09-17T04:06:48.642222+0000","last_active":"2024-09-17T04:06:49.649644+0000","last_peered":"2024-09-17T04:06:49.649644+0000","last_clean":"2024-09-17T04:06:49.649644+0000","last_became_active":"2024-09-17T04:06:48.641845+0000","last_became_peered":"2024-09-17T04:06:48.641845+0000","last_unstale":"2024-09-17T04:06:49.649644+0000","last_undegraded":"2024-09-17T04:06:49.649644+0000","last_fullsized":"2024-09-17T04:06:49.649644+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.092195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.659677+0000","last_change":"2024-09-17T04:06:47.635036+0000","last_active":"2024-09-17T04:06:48.659677+0000","last_peered":"2024-09-17T04:06:48.659677+0000","last_clean":"2024-09-17T04:06:48.659677+0000","last_became_active":"2024-09-17T04:06:47.634721+0000","last_became_peered":"2024-09-17T04:06:47.634721+0000","last_unstale":"2024-09-17T04:06:48.659677+0000","last_undegraded":"2024-09-17T04:06:48.659677+0000","last_fullsized":"2024-09-17T04:06:48.659677+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:48.022710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.649772+0000","last_change":"2024-09-17T04:06:48.647597+0000","last_active":"2024-09-17T04:06:48.649772+0000","last_peered":"2024-09-17T04:06:48.649772+0000","last_clean":"2024-09-17T04:06:48.649772+0000","last_became_active":"2024-09-17T04:06:48.647265+0000","last_became_peered":"2024-09-17T04:06:48.647265+0000","last_unstale":"2024-09-17T04:06:48.649772+0000","last_undegraded":"2024-09-17T04:06:48.649772+0000","last_fullsized":"2024-09-17T04:06:48.649772+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:29:31.880487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631987+0000","last_change":"2024-09-17T04:06:47.627934+0000","last_active":"2024-09-17T04:06:49.631987+0000","last_peered":"2024-09-17T04:06:49.631987+0000","last_clean":"2024-09-17T04:06:49.631987+0000","last_became_active":"2024-09-17T04:06:47.627651+0000","last_became_peered":"2024-09-17T04:06:47.627651+0000","last_unstale":"2024-09-17T04:06:49.631987+0000","last_undegraded":"2024-09-17T04:06:49.631987+0000","last_fullsized":"2024-09-17T04:06:49.631987+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:28.266741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631723+0000","last_change":"2024-09-17T04:06:48.630489+0000","last_active":"2024-09-17T04:06:49.631723+0000","last_peered":"2024-09-17T04:06:49.631723+0000","last_clean":"2024-09-17T04:06:49.631723+0000","last_became_active":"2024-09-17T04:06:48.630199+0000","last_became_peered":"2024-09-17T04:06:48.630199+0000","last_unstale":"2024-09-17T04:06:49.631723+0000","last_undegraded":"2024-09-17T04:06:49.631723+0000","last_fullsized":"2024-09-17T04:06:49.631723+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:01.307233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648431+0000","last_change":"2024-09-17T04:06:47.636455+0000","last_active":"2024-09-17T04:06:49.648431+0000","last_peered":"2024-09-17T04:06:49.648431+0000","last_clean":"2024-09-17T04:06:49.648431+0000","last_became_active":"2024-09-17T04:06:47.636161+0000","last_became_peered":"2024-09-17T04:06:47.636161+0000","last_unstale":"2024-09-17T04:06:49.648431+0000","last_undegraded":"2024-09-17T04:06:49.648431+0000","last_fullsized":"2024-09-17T04:06:49.648431+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:26.341915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631704+0000","last_change":"2024-09-17T04:06:48.626073+0000","last_active":"2024-09-17T04:06:49.631704+0000","last_peered":"2024-09-17T04:06:49.631704+0000","last_clean":"2024-09-17T04:06:49.631704+0000","last_became_active":"2024-09-17T04:06:48.624659+0000","last_became_peered":"2024-09-17T04:06:48.624659+0000","last_unstale":"2024-09-17T04:06:49.631704+0000","last_undegraded":"2024-09-17T04:06:49.631704+0000","last_fullsized":"2024-09-17T04:06:49.631704+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:03.812133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631911+0000","last_change":"2024-09-17T04:06:47.631678+0000","last_active":"2024-09-17T04:06:49.631911+0000","last_peered":"2024-09-17T04:06:49.631911+0000","last_clean":"2024-09-17T04:06:49.631911+0000","last_became_active":"2024-09-17T04:06:47.629357+0000","last_became_peered":"2024-09-17T04:06:47.629357+0000","last_unstale":"2024-09-17T04:06:49.631911+0000","last_undegraded":"2024-09-17T04:06:49.631911+0000","last_fullsized":"2024-09-17T04:06:49.631911+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:58.960464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.646523+0000","last_change":"2024-09-17T04:06:48.641309+0000","last_active":"2024-09-17T04:06:48.646523+0000","last_peered":"2024-09-17T04:06:48.646523+0000","last_clean":"2024-09-17T04:06:48.646523+0000","last_became_active":"2024-09-17T04:06:48.640780+0000","last_became_peered":"2024-09-17T04:06:48.640780+0000","last_unstale":"2024-09-17T04:06:48.646523+0000","last_undegraded":"2024-09-17T04:06:48.646523+0000","last_fullsized":"2024-09-17T04:06:48.646523+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:57:38.559159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"246'2","reported_seq":15,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.660451+0000","last_change":"2024-09-17T04:06:47.634843+0000","last_active":"2024-09-17T04:06:48.660451+0000","last_peered":"2024-09-17T04:06:48.660451+0000","last_clean":"2024-09-17T04:06:48.660451+0000","last_became_active":"2024-09-17T04:06:47.634529+0000","last_became_peered":"2024-09-17T04:06:47.634529+0000","last_unstale":"2024-09-17T04:06:48.660451+0000","last_undegraded":"2024-09-17T04:06:48.660451+0000","last_fullsized":"2024-09-17T04:06:48.660451+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:24.241367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634196+0000","last_change":"2024-09-17T04:06:48.637294+0000","last_active":"2024-09-17T04:06:49.634196+0000","last_peered":"2024-09-17T04:06:49.634196+0000","last_clean":"2024-09-17T04:06:49.634196+0000","last_became_active":"2024-09-17T04:06:48.636742+0000","last_became_peered":"2024-09-17T04:06:48.636742+0000","last_unstale":"2024-09-17T04:06:49.634196+0000","last_undegraded":"2024-09-17T04:06:49.634196+0000","last_fullsized":"2024-09-17T04:06:49.634196+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:40.916077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632108+0000","last_change":"2024-09-17T04:06:47.635594+0000","last_active":"2024-09-17T04:06:49.632108+0000","last_peered":"2024-09-17T04:06:49.632108+0000","last_clean":"2024-09-17T04:06:49.632108+0000","last_became_active":"2024-09-17T04:06:47.633247+0000","last_became_peered":"2024-09-17T04:06:47.633247+0000","last_unstale":"2024-09-17T04:06:49.632108+0000","last_undegraded":"2024-09-17T04:06:49.632108+0000","last_fullsized":"2024-09-17T04:06:49.632108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:29.409434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648651+0000","last_change":"2024-09-17T04:06:48.654761+0000","last_active":"2024-09-17T04:06:49.648651+0000","last_peered":"2024-09-17T04:06:49.648651+0000","last_clean":"2024-09-17T04:06:49.648651+0000","last_became_active":"2024-09-17T04:06:48.654228+0000","last_became_peered":"2024-09-17T04:06:48.654228+0000","last_unstale":"2024-09-17T04:06:49.648651+0000","last_undegraded":"2024-09-17T04:06:49.648651+0000","last_fullsized":"2024-09-17T04:06:49.648651+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:46.775630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.650149+0000","last_change":"2024-09-17T04:06:47.631080+0000","last_active":"2024-09-17T04:06:48.650149+0000","last_peered":"2024-09-17T04:06:48.650149+0000","last_clean":"2024-09-17T04:06:48.650149+0000","last_became_active":"2024-09-17T04:06:47.630810+0000","last_became_peered":"2024-09-17T04:06:47.630810+0000","last_unstale":"2024-09-17T04:06:48.650149+0000","last_undegraded":"2024-09-17T04:06:48.650149+0000","last_fullsized":"2024-09-17T04:06:48.650149+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.601411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634552+0000","last_change":"2024-09-17T04:06:48.640563+0000","last_active":"2024-09-17T04:06:49.634552+0000","last_peered":"2024-09-17T04:06:49.634552+0000","last_clean":"2024-09-17T04:06:49.634552+0000","last_became_active":"2024-09-17T04:06:48.640193+0000","last_became_peered":"2024-09-17T04:06:48.640193+0000","last_unstale":"2024-09-17T04:06:49.634552+0000","last_undegraded":"2024-09-17T04:06:49.634552+0000","last_fullsized":"2024-09-17T04:06:49.634552+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:15.045992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649735+0000","last_change":"2024-09-17T04:06:47.640808+0000","last_active":"2024-09-17T04:06:49.649735+0000","last_peered":"2024-09-17T04:06:49.649735+0000","last_clean":"2024-09-17T04:06:49.649735+0000","last_became_active":"2024-09-17T04:06:47.639846+0000","last_became_peered":"2024-09-17T04:06:47.639846+0000","last_unstale":"2024-09-17T04:06:49.649735+0000","last_undegraded":"2024-09-17T04:06:49.649735+0000","last_fullsized":"2024-09-17T04:06:49.649735+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:15.837166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648675+0000","last_change":"2024-09-17T04:06:48.623052+0000","last_active":"2024-09-17T04:06:49.648675+0000","last_peered":"2024-09-17T04:06:49.648675+0000","last_clean":"2024-09-17T04:06:49.648675+0000","last_became_active":"2024-09-17T04:06:48.622715+0000","last_became_peered":"2024-09-17T04:06:48.622715+0000","last_unstale":"2024-09-17T04:06:49.648675+0000","last_undegraded":"2024-09-17T04:06:49.648675+0000","last_fullsized":"2024-09-17T04:06:49.648675+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:55.252993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649527+0000","last_change":"2024-09-17T04:06:47.639290+0000","last_active":"2024-09-17T04:06:49.649527+0000","last_peered":"2024-09-17T04:06:49.649527+0000","last_clean":"2024-09-17T04:06:49.649527+0000","last_became_active":"2024-09-17T04:06:47.638370+0000","last_became_peered":"2024-09-17T04:06:47.638370+0000","last_unstale":"2024-09-17T04:06:49.649527+0000","last_undegraded":"2024-09-17T04:06:49.649527+0000","last_fullsized":"2024-09-17T04:06:49.649527+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:17.818126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.651574+0000","last_change":"2024-09-17T04:06:48.645144+0000","last_active":"2024-09-17T04:06:48.651574+0000","last_peered":"2024-09-17T04:06:48.651574+0000","last_clean":"2024-09-17T04:06:48.651574+0000","last_became_active":"2024-09-17T04:06:48.644506+0000","last_became_peered":"2024-09-17T04:06:48.644506+0000","last_unstale":"2024-09-17T04:06:48.651574+0000","last_undegraded":"2024-09-17T04:06:48.651574+0000","last_fullsized":"2024-09-17T04:06:48.651574+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:58.415646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638296+0000","last_change":"2024-09-17T04:06:47.637796+0000","last_active":"2024-09-17T04:06:48.638296+0000","last_peered":"2024-09-17T04:06:48.638296+0000","last_clean":"2024-09-17T04:06:48.638296+0000","last_became_active":"2024-09-17T04:06:47.637473+0000","last_became_peered":"2024-09-17T04:06:47.637473+0000","last_unstale":"2024-09-17T04:06:48.638296+0000","last_undegraded":"2024-09-17T04:06:48.638296+0000","last_fullsized":"2024-09-17T04:06:48.638296+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:35.426948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631887+0000","last_change":"2024-09-17T04:06:48.626386+0000","last_active":"2024-09-17T04:06:49.631887+0000","last_peered":"2024-09-17T04:06:49.631887+0000","last_clean":"2024-09-17T04:06:49.631887+0000","last_became_active":"2024-09-17T04:06:48.623872+0000","last_became_peered":"2024-09-17T04:06:48.623872+0000","last_unstale":"2024-09-17T04:06:49.631887+0000","last_undegraded":"2024-09-17T04:06:49.631887+0000","last_fullsized":"2024-09-17T04:06:49.631887+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:35.483013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648365+0000","last_change":"2024-09-17T04:06:47.632594+0000","last_active":"2024-09-17T04:06:49.648365+0000","last_peered":"2024-09-17T04:06:49.648365+0000","last_clean":"2024-09-17T04:06:49.648365+0000","last_became_active":"2024-09-17T04:06:47.632229+0000","last_became_peered":"2024-09-17T04:06:47.632229+0000","last_unstale":"2024-09-17T04:06:49.648365+0000","last_undegraded":"2024-09-17T04:06:49.648365+0000","last_fullsized":"2024-09-17T04:06:49.648365+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:10.174881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635191+0000","last_change":"2024-09-17T04:06:48.640651+0000","last_active":"2024-09-17T04:06:49.635191+0000","last_peered":"2024-09-17T04:06:49.635191+0000","last_clean":"2024-09-17T04:06:49.635191+0000","last_became_active":"2024-09-17T04:06:48.639761+0000","last_became_peered":"2024-09-17T04:06:48.639761+0000","last_unstale":"2024-09-17T04:06:49.635191+0000","last_undegraded":"2024-09-17T04:06:49.635191+0000","last_fullsized":"2024-09-17T04:06:49.635191+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:29.184251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":13,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.638592+0000","last_change":"2024-09-17T04:06:47.639117+0000","last_active":"2024-09-17T04:06:48.638592+0000","last_peered":"2024-09-17T04:06:48.638592+0000","last_clean":"2024-09-17T04:06:48.638592+0000","last_became_active":"2024-09-17T04:06:47.638778+0000","last_became_peered":"2024-09-17T04:06:47.638778+0000","last_unstale":"2024-09-17T04:06:48.638592+0000","last_undegraded":"2024-09-17T04:06:48.638592+0000","last_fullsized":"2024-09-17T04:06:48.638592+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:42.527414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":10,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-09-17T04:06:48.653501+0000","last_change":"2024-09-17T04:06:48.648394+0000","last_active":"2024-09-17T04:06:48.653501+0000","last_peered":"2024-09-17T04:06:48.653501+0000","last_clean":"2024-09-17T04:06:48.653501+0000","last_became_active":"2024-09-17T04:06:48.647962+0000","last_became_peered":"2024-09-17T04:06:48.647962+0000","last_unstale":"2024-09-17T04:06:48.653501+0000","last_undegraded":"2024-09-17T04:06:48.653501+0000","last_fullsized":"2024-09-17T04:06:48.653501+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:16.705905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":527,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649624+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:49.649624+0000","last_peered":"2024-09-17T04:06:49.649624+0000","last_clean":"2024-09-17T04:06:49.649624+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:49.649624+0000","last_undegraded":"2024-09-17T04:06:49.649624+0000","last_fullsized":"2024-09-17T04:06:49.649624+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509652,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52876,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3719168,"data_stored":1975944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509655,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34264,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337576,"statfs":{"total":96636764160,"available":96601677824,"internally_reserved":0,"allocated":3141632,"data_stored":1387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509654,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56464,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315376,"statfs":{"total":96636764160,"available":96578945024,"internally_reserved":0,"allocated":3133440,"data_stored":1386228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56444,"kb_used_data":3044,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315396,"statfs":{"total":96636764160,"available":96578965504,"internally_reserved":0,"allocated":3117056,"data_stored":1379858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509653,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56448,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315392,"statfs":{"total":96636764160,"available":96578961408,"internally_reserved":0,"allocated":3121152,"data_stored":1379400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509654,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52304,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319536,"statfs":{"total":96636764160,"available":96583204864,"internally_reserved":0,"allocated":3133440,"data_stored":1385632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56464,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315376,"statfs":{"total":96636764160,"available":96578945024,"internally_reserved":0,"allocated":3137536,"data_stored":1387188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542357,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52876,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3727360,"data_stored":1976036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:58.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 -- 172.21.15.129:0/2784235164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b00d0a50 msgr2=0x7f49b0160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 --2- 172.21.15.129:0/2784235164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b00d0a50 0x7f49b0160210 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f499c009830 tx=0x7f499c02a480 comp rx=0 tx=0).stop 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 -- 172.21.15.129:0/2784235164 shutdown_connections 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 --2- 172.21.15.129:0/2784235164 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b01458a0 0x7f49b0160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 --2- 172.21.15.129:0/2784235164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b00d0a50 0x7f49b0160210 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 -- 172.21.15.129:0/2784235164 >> 172.21.15.129:0/2784235164 conn(0x7f49b006cfc0 msgr2=0x7f49b006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:58.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 -- 172.21.15.129:0/2784235164 shutdown_connections 2024-09-17T04:06:58.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.247+0000 7f49b6c00640 1 -- 172.21.15.129:0/2784235164 wait complete. 2024-09-17T04:06:58.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.250+0000 7f49b6c00640 1 Processor -- start 2024-09-17T04:06:58.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.250+0000 7f49b6c00640 1 -- start start 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b0145e60 0x7f49b00c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49b01f6840 0x7f49b00cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f49b00cef20 con 0x7f49b01458a0 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f49b00ceda0 con 0x7f49b01f6840 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f49b0160c90 con 0x7f49b0145e60 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.251+0000 7f49b5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52254/0 (socket says 172.21.15.129:52254) 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 -- 172.21.15.129:0/1395328655 learned_addr learned my addr 172.21.15.129:0/1395328655 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b4e00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b0145e60 0x7f49b00c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b6200640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49b01f6840 0x7f49b00cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 -- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b0145e60 msgr2=0x7f49b00c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b0145e60 0x7f49b00c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 -- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49b01f6840 msgr2=0x7f49b00cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49b01f6840 0x7f49b00cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 -- 172.21.15.129:0/1395328655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b01f7320 con 0x7f49b01458a0 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49b5800640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c3c0 tx=0x7f49a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.252+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4017070 con 0x7f49b01458a0 2024-09-17T04:06:58.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.253+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49a40035f0 con 0x7f49b01458a0 2024-09-17T04:06:58.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.253+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b01f7600 con 0x7f49b01458a0 2024-09-17T04:06:58.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.253+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a4003a80 con 0x7f49b01458a0 2024-09-17T04:06:58.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.253+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b0136d10 con 0x7f49b01458a0 2024-09-17T04:06:58.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.254+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b0160210 con 0x7f49b01458a0 2024-09-17T04:06:58.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.255+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f49a401a070 con 0x7f49b01458a0 2024-09-17T04:06:58.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.256+0000 7f49ad600640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 0x7f499007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.256+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f49a4099f20 con 0x7f49b01458a0 2024-09-17T04:06:58.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.256+0000 7f49b4e00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 0x7f499007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.257+0000 7f49b4e00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 0x7f499007b0a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f499c034000 tx=0x7f499c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:58.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.260+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f49a4061d90 con 0x7f49b01458a0 2024-09-17T04:06:58.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.442+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49b00c6a10 con 0x7f4990078b90 2024-09-17T04:06:58.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.447+0000 7f49ad600640 1 -- 172.21.15.129:0/1395328655 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252443 (secure 0 0 0) 0x7f49b00c6a10 con 0x7f4990078b90 2024-09-17T04:06:58.451 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:58.452 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 msgr2=0x7f499007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 0x7f499007b0a0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f499c034000 tx=0x7f499c004730 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 msgr2=0x7f49b0137230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7f49a400c3c0 tx=0x7f49a400c880 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 shutdown_connections 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4990078b90 0x7f499007b0a0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49b01f6840 0x7f49b00cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49b0145e60 0x7f49b00c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 --2- 172.21.15.129:0/1395328655 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49b01458a0 0x7f49b0137230 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 >> 172.21.15.129:0/1395328655 conn(0x7f49b006cfc0 msgr2=0x7f49b00cc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:58.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 shutdown_connections 2024-09-17T04:06:58.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.451+0000 7f49b6c00640 1 -- 172.21.15.129:0/1395328655 wait complete. 2024-09-17T04:06:58.477 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":767,"stamp":"2024-09-17T04:06:56.593769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":418172,"kb_used_data":25640,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754556548,"statfs":{"total":773094113280,"available":772665905152,"internally_reserved":0,"allocated":26255360,"data_stored":12270164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002291"},"pg_stats":[{"pgid":"37.3f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635340+0000","last_change":"2024-09-17T04:06:48.639552+0000","last_active":"2024-09-17T04:06:49.635340+0000","last_peered":"2024-09-17T04:06:49.635340+0000","last_clean":"2024-09-17T04:06:49.635340+0000","last_became_active":"2024-09-17T04:06:48.639000+0000","last_became_peered":"2024-09-17T04:06:48.639000+0000","last_unstale":"2024-09-17T04:06:49.635340+0000","last_undegraded":"2024-09-17T04:06:49.635340+0000","last_fullsized":"2024-09-17T04:06:49.635340+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:29.099857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.3e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650075+0000","last_change":"2024-09-17T04:06:48.639479+0000","last_active":"2024-09-17T04:06:49.650075+0000","last_peered":"2024-09-17T04:06:49.650075+0000","last_clean":"2024-09-17T04:06:49.650075+0000","last_became_active":"2024-09-17T04:06:48.632914+0000","last_became_peered":"2024-09-17T04:06:48.632914+0000","last_unstale":"2024-09-17T04:06:49.650075+0000","last_undegraded":"2024-09-17T04:06:49.650075+0000","last_fullsized":"2024-09-17T04:06:49.650075+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:44.816386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.3d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650863+0000","last_change":"2024-09-17T04:06:48.653840+0000","last_active":"2024-09-17T04:06:49.650863+0000","last_peered":"2024-09-17T04:06:49.650863+0000","last_clean":"2024-09-17T04:06:49.650863+0000","last_became_active":"2024-09-17T04:06:48.653613+0000","last_became_peered":"2024-09-17T04:06:48.653613+0000","last_unstale":"2024-09-17T04:06:49.650863+0000","last_undegraded":"2024-09-17T04:06:49.650863+0000","last_fullsized":"2024-09-17T04:06:49.650863+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:44.216820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650740+0000","last_change":"2024-09-17T04:06:48.653715+0000","last_active":"2024-09-17T04:06:49.650740+0000","last_peered":"2024-09-17T04:06:49.650740+0000","last_clean":"2024-09-17T04:06:49.650740+0000","last_became_active":"2024-09-17T04:06:48.653119+0000","last_became_peered":"2024-09-17T04:06:48.653119+0000","last_unstale":"2024-09-17T04:06:49.650740+0000","last_undegraded":"2024-09-17T04:06:49.650740+0000","last_fullsized":"2024-09-17T04:06:49.650740+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:53.901177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3b","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650543+0000","last_change":"2024-09-17T04:06:48.645621+0000","last_active":"2024-09-17T04:06:49.650543+0000","last_peered":"2024-09-17T04:06:49.650543+0000","last_clean":"2024-09-17T04:06:49.650543+0000","last_became_active":"2024-09-17T04:06:48.644873+0000","last_became_peered":"2024-09-17T04:06:48.644873+0000","last_unstale":"2024-09-17T04:06:49.650543+0000","last_undegraded":"2024-09-17T04:06:49.650543+0000","last_fullsized":"2024-09-17T04:06:49.650543+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:30:59.368016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.3a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648491+0000","last_change":"2024-09-17T04:06:48.631987+0000","last_active":"2024-09-17T04:06:49.648491+0000","last_peered":"2024-09-17T04:06:49.648491+0000","last_clean":"2024-09-17T04:06:49.648491+0000","last_became_active":"2024-09-17T04:06:48.631615+0000","last_became_peered":"2024-09-17T04:06:48.631615+0000","last_unstale":"2024-09-17T04:06:49.648491+0000","last_undegraded":"2024-09-17T04:06:49.648491+0000","last_fullsized":"2024-09-17T04:06:49.648491+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:58:35.948167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.39","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633892+0000","last_change":"2024-09-17T04:06:48.645454+0000","last_active":"2024-09-17T04:06:49.633892+0000","last_peered":"2024-09-17T04:06:49.633892+0000","last_clean":"2024-09-17T04:06:49.633892+0000","last_became_active":"2024-09-17T04:06:48.645060+0000","last_became_peered":"2024-09-17T04:06:48.645060+0000","last_unstale":"2024-09-17T04:06:49.633892+0000","last_undegraded":"2024-09-17T04:06:49.633892+0000","last_fullsized":"2024-09-17T04:06:49.633892+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:59.853400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.38","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631646+0000","last_change":"2024-09-17T04:06:48.636070+0000","last_active":"2024-09-17T04:06:49.631646+0000","last_peered":"2024-09-17T04:06:49.631646+0000","last_clean":"2024-09-17T04:06:49.631646+0000","last_became_active":"2024-09-17T04:06:48.635753+0000","last_became_peered":"2024-09-17T04:06:48.635753+0000","last_unstale":"2024-09-17T04:06:49.631646+0000","last_undegraded":"2024-09-17T04:06:49.631646+0000","last_fullsized":"2024-09-17T04:06:49.631646+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:35:37.356554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.37","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650178+0000","last_change":"2024-09-17T04:06:48.640090+0000","last_active":"2024-09-17T04:06:49.650178+0000","last_peered":"2024-09-17T04:06:49.650178+0000","last_clean":"2024-09-17T04:06:49.650178+0000","last_became_active":"2024-09-17T04:06:48.635312+0000","last_became_peered":"2024-09-17T04:06:48.635312+0000","last_unstale":"2024-09-17T04:06:49.650178+0000","last_undegraded":"2024-09-17T04:06:49.650178+0000","last_fullsized":"2024-09-17T04:06:49.650178+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:03:14.470067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.36","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635456+0000","last_change":"2024-09-17T04:06:48.632182+0000","last_active":"2024-09-17T04:06:49.635456+0000","last_peered":"2024-09-17T04:06:49.635456+0000","last_clean":"2024-09-17T04:06:49.635456+0000","last_became_active":"2024-09-17T04:06:48.631841+0000","last_became_peered":"2024-09-17T04:06:48.631841+0000","last_unstale":"2024-09-17T04:06:49.635456+0000","last_undegraded":"2024-09-17T04:06:49.635456+0000","last_fullsized":"2024-09-17T04:06:49.635456+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:57:26.532740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.35","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632045+0000","last_change":"2024-09-17T04:06:48.635273+0000","last_active":"2024-09-17T04:06:49.632045+0000","last_peered":"2024-09-17T04:06:49.632045+0000","last_clean":"2024-09-17T04:06:49.632045+0000","last_became_active":"2024-09-17T04:06:48.634916+0000","last_became_peered":"2024-09-17T04:06:48.634916+0000","last_unstale":"2024-09-17T04:06:49.632045+0000","last_undegraded":"2024-09-17T04:06:49.632045+0000","last_fullsized":"2024-09-17T04:06:49.632045+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:41:46.226736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.34","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649549+0000","last_change":"2024-09-17T04:06:48.654714+0000","last_active":"2024-09-17T04:06:49.649549+0000","last_peered":"2024-09-17T04:06:49.649549+0000","last_clean":"2024-09-17T04:06:49.649549+0000","last_became_active":"2024-09-17T04:06:48.654168+0000","last_became_peered":"2024-09-17T04:06:48.654168+0000","last_unstale":"2024-09-17T04:06:49.649549+0000","last_undegraded":"2024-09-17T04:06:49.649549+0000","last_fullsized":"2024-09-17T04:06:49.649549+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:27:54.690107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.33","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634298+0000","last_change":"2024-09-17T04:06:48.625470+0000","last_active":"2024-09-17T04:06:49.634298+0000","last_peered":"2024-09-17T04:06:49.634298+0000","last_clean":"2024-09-17T04:06:49.634298+0000","last_became_active":"2024-09-17T04:06:48.624818+0000","last_became_peered":"2024-09-17T04:06:48.624818+0000","last_unstale":"2024-09-17T04:06:49.634298+0000","last_undegraded":"2024-09-17T04:06:49.634298+0000","last_fullsized":"2024-09-17T04:06:49.634298+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:14:17.324847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.32","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648452+0000","last_change":"2024-09-17T04:06:48.635524+0000","last_active":"2024-09-17T04:06:49.648452+0000","last_peered":"2024-09-17T04:06:49.648452+0000","last_clean":"2024-09-17T04:06:49.648452+0000","last_became_active":"2024-09-17T04:06:48.634740+0000","last_became_peered":"2024-09-17T04:06:48.634740+0000","last_unstale":"2024-09-17T04:06:49.648452+0000","last_undegraded":"2024-09-17T04:06:49.648452+0000","last_fullsized":"2024-09-17T04:06:49.648452+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:23:10.649399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.31","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649579+0000","last_change":"2024-09-17T04:06:48.639136+0000","last_active":"2024-09-17T04:06:49.649579+0000","last_peered":"2024-09-17T04:06:49.649579+0000","last_clean":"2024-09-17T04:06:49.649579+0000","last_became_active":"2024-09-17T04:06:48.638849+0000","last_became_peered":"2024-09-17T04:06:48.638849+0000","last_unstale":"2024-09-17T04:06:49.649579+0000","last_undegraded":"2024-09-17T04:06:49.649579+0000","last_fullsized":"2024-09-17T04:06:49.649579+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:08:30.231125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.30","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649378+0000","last_change":"2024-09-17T04:06:48.635932+0000","last_active":"2024-09-17T04:06:49.649378+0000","last_peered":"2024-09-17T04:06:49.649378+0000","last_clean":"2024-09-17T04:06:49.649378+0000","last_became_active":"2024-09-17T04:06:48.634277+0000","last_became_peered":"2024-09-17T04:06:48.634277+0000","last_unstale":"2024-09-17T04:06:49.649378+0000","last_undegraded":"2024-09-17T04:06:49.649378+0000","last_fullsized":"2024-09-17T04:06:49.649378+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:43.745727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633917+0000","last_change":"2024-09-17T04:06:48.651905+0000","last_active":"2024-09-17T04:06:49.633917+0000","last_peered":"2024-09-17T04:06:49.633917+0000","last_clean":"2024-09-17T04:06:49.633917+0000","last_became_active":"2024-09-17T04:06:48.651000+0000","last_became_peered":"2024-09-17T04:06:48.651000+0000","last_unstale":"2024-09-17T04:06:49.633917+0000","last_undegraded":"2024-09-17T04:06:49.633917+0000","last_fullsized":"2024-09-17T04:06:49.633917+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:49:59.428221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633764+0000","last_change":"2024-09-17T04:06:48.623622+0000","last_active":"2024-09-17T04:06:49.633764+0000","last_peered":"2024-09-17T04:06:49.633764+0000","last_clean":"2024-09-17T04:06:49.633764+0000","last_became_active":"2024-09-17T04:06:48.623254+0000","last_became_peered":"2024-09-17T04:06:48.623254+0000","last_unstale":"2024-09-17T04:06:49.633764+0000","last_undegraded":"2024-09-17T04:06:49.633764+0000","last_fullsized":"2024-09-17T04:06:49.633764+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:05.528806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.2d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648913+0000","last_change":"2024-09-17T04:06:48.643193+0000","last_active":"2024-09-17T04:06:49.648913+0000","last_peered":"2024-09-17T04:06:49.648913+0000","last_clean":"2024-09-17T04:06:49.648913+0000","last_became_active":"2024-09-17T04:06:48.641658+0000","last_became_peered":"2024-09-17T04:06:48.641658+0000","last_unstale":"2024-09-17T04:06:49.648913+0000","last_undegraded":"2024-09-17T04:06:49.648913+0000","last_fullsized":"2024-09-17T04:06:49.648913+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:50.000607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.2c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649953+0000","last_change":"2024-09-17T04:06:48.641052+0000","last_active":"2024-09-17T04:06:49.649953+0000","last_peered":"2024-09-17T04:06:49.649953+0000","last_clean":"2024-09-17T04:06:49.649953+0000","last_became_active":"2024-09-17T04:06:48.640727+0000","last_became_peered":"2024-09-17T04:06:48.640727+0000","last_unstale":"2024-09-17T04:06:49.649953+0000","last_undegraded":"2024-09-17T04:06:49.649953+0000","last_fullsized":"2024-09-17T04:06:49.649953+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:07:48.678295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"37.2b","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631729+0000","last_change":"2024-09-17T04:06:48.638889+0000","last_active":"2024-09-17T04:06:49.631729+0000","last_peered":"2024-09-17T04:06:49.631729+0000","last_clean":"2024-09-17T04:06:49.631729+0000","last_became_active":"2024-09-17T04:06:48.638320+0000","last_became_peered":"2024-09-17T04:06:48.638320+0000","last_unstale":"2024-09-17T04:06:49.631729+0000","last_undegraded":"2024-09-17T04:06:49.631729+0000","last_fullsized":"2024-09-17T04:06:49.631729+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:06.457607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.2a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650162+0000","last_change":"2024-09-17T04:06:48.640395+0000","last_active":"2024-09-17T04:06:49.650162+0000","last_peered":"2024-09-17T04:06:49.650162+0000","last_clean":"2024-09-17T04:06:49.650162+0000","last_became_active":"2024-09-17T04:06:48.640063+0000","last_became_peered":"2024-09-17T04:06:48.640063+0000","last_unstale":"2024-09-17T04:06:49.650162+0000","last_undegraded":"2024-09-17T04:06:49.650162+0000","last_fullsized":"2024-09-17T04:06:49.650162+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:05:31.812758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.29","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633679+0000","last_change":"2024-09-17T04:06:48.624960+0000","last_active":"2024-09-17T04:06:49.633679+0000","last_peered":"2024-09-17T04:06:49.633679+0000","last_clean":"2024-09-17T04:06:49.633679+0000","last_became_active":"2024-09-17T04:06:48.624594+0000","last_became_peered":"2024-09-17T04:06:48.624594+0000","last_unstale":"2024-09-17T04:06:49.633679+0000","last_undegraded":"2024-09-17T04:06:49.633679+0000","last_fullsized":"2024-09-17T04:06:49.633679+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:50:44.161205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.28","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648399+0000","last_change":"2024-09-17T04:06:48.641161+0000","last_active":"2024-09-17T04:06:49.648399+0000","last_peered":"2024-09-17T04:06:49.648399+0000","last_clean":"2024-09-17T04:06:49.648399+0000","last_became_active":"2024-09-17T04:06:48.640811+0000","last_became_peered":"2024-09-17T04:06:48.640811+0000","last_unstale":"2024-09-17T04:06:49.648399+0000","last_undegraded":"2024-09-17T04:06:49.648399+0000","last_fullsized":"2024-09-17T04:06:49.648399+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:13:30.671650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.27","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634072+0000","last_change":"2024-09-17T04:06:48.638436+0000","last_active":"2024-09-17T04:06:49.634072+0000","last_peered":"2024-09-17T04:06:49.634072+0000","last_clean":"2024-09-17T04:06:49.634072+0000","last_became_active":"2024-09-17T04:06:48.637635+0000","last_became_peered":"2024-09-17T04:06:48.637635+0000","last_unstale":"2024-09-17T04:06:49.634072+0000","last_undegraded":"2024-09-17T04:06:49.634072+0000","last_fullsized":"2024-09-17T04:06:49.634072+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:31:10.169232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.26","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635253+0000","last_change":"2024-09-17T04:06:48.646124+0000","last_active":"2024-09-17T04:06:49.635253+0000","last_peered":"2024-09-17T04:06:49.635253+0000","last_clean":"2024-09-17T04:06:49.635253+0000","last_became_active":"2024-09-17T04:06:48.645673+0000","last_became_peered":"2024-09-17T04:06:48.645673+0000","last_unstale":"2024-09-17T04:06:49.635253+0000","last_undegraded":"2024-09-17T04:06:49.635253+0000","last_fullsized":"2024-09-17T04:06:49.635253+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:19:54.659533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"37.25","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649892+0000","last_change":"2024-09-17T04:06:48.637878+0000","last_active":"2024-09-17T04:06:49.649892+0000","last_peered":"2024-09-17T04:06:49.649892+0000","last_clean":"2024-09-17T04:06:49.649892+0000","last_became_active":"2024-09-17T04:06:48.637580+0000","last_became_peered":"2024-09-17T04:06:48.637580+0000","last_unstale":"2024-09-17T04:06:49.649892+0000","last_undegraded":"2024-09-17T04:06:49.649892+0000","last_fullsized":"2024-09-17T04:06:49.649892+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:32:25.874073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.23","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648941+0000","last_change":"2024-09-17T04:06:48.654016+0000","last_active":"2024-09-17T04:06:49.648941+0000","last_peered":"2024-09-17T04:06:49.648941+0000","last_clean":"2024-09-17T04:06:49.648941+0000","last_became_active":"2024-09-17T04:06:48.650905+0000","last_became_peered":"2024-09-17T04:06:48.650905+0000","last_unstale":"2024-09-17T04:06:49.648941+0000","last_undegraded":"2024-09-17T04:06:49.648941+0000","last_fullsized":"2024-09-17T04:06:49.648941+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:35.043475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.22","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634406+0000","last_change":"2024-09-17T04:06:48.638284+0000","last_active":"2024-09-17T04:06:49.634406+0000","last_peered":"2024-09-17T04:06:49.634406+0000","last_clean":"2024-09-17T04:06:49.634406+0000","last_became_active":"2024-09-17T04:06:48.637899+0000","last_became_peered":"2024-09-17T04:06:48.637899+0000","last_unstale":"2024-09-17T04:06:49.634406+0000","last_undegraded":"2024-09-17T04:06:49.634406+0000","last_fullsized":"2024-09-17T04:06:49.634406+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:51.007347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.21","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648462+0000","last_change":"2024-09-17T04:06:48.643887+0000","last_active":"2024-09-17T04:06:49.648462+0000","last_peered":"2024-09-17T04:06:49.648462+0000","last_clean":"2024-09-17T04:06:49.648462+0000","last_became_active":"2024-09-17T04:06:48.643541+0000","last_became_peered":"2024-09-17T04:06:48.643541+0000","last_unstale":"2024-09-17T04:06:49.648462+0000","last_undegraded":"2024-09-17T04:06:49.648462+0000","last_fullsized":"2024-09-17T04:06:49.648462+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:38:54.252745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.20","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649093+0000","last_change":"2024-09-17T04:06:48.646487+0000","last_active":"2024-09-17T04:06:49.649093+0000","last_peered":"2024-09-17T04:06:49.649093+0000","last_clean":"2024-09-17T04:06:49.649093+0000","last_became_active":"2024-09-17T04:06:48.646200+0000","last_became_peered":"2024-09-17T04:06:48.646200+0000","last_unstale":"2024-09-17T04:06:49.649093+0000","last_undegraded":"2024-09-17T04:06:49.649093+0000","last_fullsized":"2024-09-17T04:06:49.649093+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:35.550507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649893+0000","last_change":"2024-09-17T04:06:48.638450+0000","last_active":"2024-09-17T04:06:49.649893+0000","last_peered":"2024-09-17T04:06:49.649893+0000","last_clean":"2024-09-17T04:06:49.649893+0000","last_became_active":"2024-09-17T04:06:48.638148+0000","last_became_peered":"2024-09-17T04:06:48.638148+0000","last_unstale":"2024-09-17T04:06:49.649893+0000","last_undegraded":"2024-09-17T04:06:49.649893+0000","last_fullsized":"2024-09-17T04:06:49.649893+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:19:35.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.1f","version":"246'4","reported_seq":19,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649000+0000","last_change":"2024-09-17T04:06:47.630685+0000","last_active":"2024-09-17T04:06:49.649000+0000","last_peered":"2024-09-17T04:06:49.649000+0000","last_clean":"2024-09-17T04:06:49.649000+0000","last_became_active":"2024-09-17T04:06:47.630367+0000","last_became_peered":"2024-09-17T04:06:47.630367+0000","last_unstale":"2024-09-17T04:06:49.649000+0000","last_undegraded":"2024-09-17T04:06:49.649000+0000","last_fullsized":"2024-09-17T04:06:49.649000+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:45:56.891603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648792+0000","last_change":"2024-09-17T04:06:48.636369+0000","last_active":"2024-09-17T04:06:49.648792+0000","last_peered":"2024-09-17T04:06:49.648792+0000","last_clean":"2024-09-17T04:06:49.648792+0000","last_became_active":"2024-09-17T04:06:48.636001+0000","last_became_peered":"2024-09-17T04:06:48.636001+0000","last_unstale":"2024-09-17T04:06:49.648792+0000","last_undegraded":"2024-09-17T04:06:49.648792+0000","last_fullsized":"2024-09-17T04:06:49.648792+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:02:35.768906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.1e","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649458+0000","last_change":"2024-09-17T04:06:47.640788+0000","last_active":"2024-09-17T04:06:49.649458+0000","last_peered":"2024-09-17T04:06:49.649458+0000","last_clean":"2024-09-17T04:06:49.649458+0000","last_became_active":"2024-09-17T04:06:47.640507+0000","last_became_peered":"2024-09-17T04:06:47.640507+0000","last_unstale":"2024-09-17T04:06:49.649458+0000","last_undegraded":"2024-09-17T04:06:49.649458+0000","last_fullsized":"2024-09-17T04:06:49.649458+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:01.560979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648684+0000","last_change":"2024-09-17T04:06:48.627725+0000","last_active":"2024-09-17T04:06:49.648684+0000","last_peered":"2024-09-17T04:06:49.648684+0000","last_clean":"2024-09-17T04:06:49.648684+0000","last_became_active":"2024-09-17T04:06:48.626509+0000","last_became_peered":"2024-09-17T04:06:48.626509+0000","last_unstale":"2024-09-17T04:06:49.648684+0000","last_undegraded":"2024-09-17T04:06:49.648684+0000","last_fullsized":"2024-09-17T04:06:49.648684+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:07:51.007347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.1d","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649085+0000","last_change":"2024-09-17T04:06:47.640338+0000","last_active":"2024-09-17T04:06:49.649085+0000","last_peered":"2024-09-17T04:06:49.649085+0000","last_clean":"2024-09-17T04:06:49.649085+0000","last_became_active":"2024-09-17T04:06:47.639781+0000","last_became_peered":"2024-09-17T04:06:47.639781+0000","last_unstale":"2024-09-17T04:06:49.649085+0000","last_undegraded":"2024-09-17T04:06:49.649085+0000","last_fullsized":"2024-09-17T04:06:49.649085+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:24:34.875218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.1d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632123+0000","last_change":"2024-09-17T04:06:48.641644+0000","last_active":"2024-09-17T04:06:49.632123+0000","last_peered":"2024-09-17T04:06:49.632123+0000","last_clean":"2024-09-17T04:06:49.632123+0000","last_became_active":"2024-09-17T04:06:48.641054+0000","last_became_peered":"2024-09-17T04:06:48.641054+0000","last_unstale":"2024-09-17T04:06:49.632123+0000","last_undegraded":"2024-09-17T04:06:49.632123+0000","last_fullsized":"2024-09-17T04:06:49.632123+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:53:37.921980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1c","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.651127+0000","last_change":"2024-09-17T04:06:47.634854+0000","last_active":"2024-09-17T04:06:49.651127+0000","last_peered":"2024-09-17T04:06:49.651127+0000","last_clean":"2024-09-17T04:06:49.651127+0000","last_became_active":"2024-09-17T04:06:47.634556+0000","last_became_peered":"2024-09-17T04:06:47.634556+0000","last_unstale":"2024-09-17T04:06:49.651127+0000","last_undegraded":"2024-09-17T04:06:49.651127+0000","last_fullsized":"2024-09-17T04:06:49.651127+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:55:59.294589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.1a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632270+0000","last_change":"2024-09-17T04:06:48.633413+0000","last_active":"2024-09-17T04:06:49.632270+0000","last_peered":"2024-09-17T04:06:49.632270+0000","last_clean":"2024-09-17T04:06:49.632270+0000","last_became_active":"2024-09-17T04:06:48.632689+0000","last_became_peered":"2024-09-17T04:06:48.632689+0000","last_unstale":"2024-09-17T04:06:49.632270+0000","last_undegraded":"2024-09-17T04:06:49.632270+0000","last_fullsized":"2024-09-17T04:06:49.632270+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:21.876560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1b","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649467+0000","last_change":"2024-09-17T04:06:47.637532+0000","last_active":"2024-09-17T04:06:49.649467+0000","last_peered":"2024-09-17T04:06:49.649467+0000","last_clean":"2024-09-17T04:06:49.649467+0000","last_became_active":"2024-09-17T04:06:47.637150+0000","last_became_peered":"2024-09-17T04:06:47.637150+0000","last_unstale":"2024-09-17T04:06:49.649467+0000","last_undegraded":"2024-09-17T04:06:49.649467+0000","last_fullsized":"2024-09-17T04:06:49.649467+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:04.474254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.1b","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648820+0000","last_change":"2024-09-17T04:06:48.625119+0000","last_active":"2024-09-17T04:06:49.648820+0000","last_peered":"2024-09-17T04:06:49.648820+0000","last_clean":"2024-09-17T04:06:49.648820+0000","last_became_active":"2024-09-17T04:06:48.624823+0000","last_became_peered":"2024-09-17T04:06:48.624823+0000","last_unstale":"2024-09-17T04:06:49.648820+0000","last_undegraded":"2024-09-17T04:06:49.648820+0000","last_fullsized":"2024-09-17T04:06:49.648820+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:23.910766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.1a","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634657+0000","last_change":"2024-09-17T04:06:47.636767+0000","last_active":"2024-09-17T04:06:49.634657+0000","last_peered":"2024-09-17T04:06:49.634657+0000","last_clean":"2024-09-17T04:06:49.634657+0000","last_became_active":"2024-09-17T04:06:47.635877+0000","last_became_peered":"2024-09-17T04:06:47.635877+0000","last_unstale":"2024-09-17T04:06:49.634657+0000","last_undegraded":"2024-09-17T04:06:49.634657+0000","last_fullsized":"2024-09-17T04:06:49.634657+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:41:36.196043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.18","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634600+0000","last_change":"2024-09-17T04:06:48.647257+0000","last_active":"2024-09-17T04:06:49.634600+0000","last_peered":"2024-09-17T04:06:49.634600+0000","last_clean":"2024-09-17T04:06:49.634600+0000","last_became_active":"2024-09-17T04:06:48.646916+0000","last_became_peered":"2024-09-17T04:06:48.646916+0000","last_unstale":"2024-09-17T04:06:49.634600+0000","last_undegraded":"2024-09-17T04:06:49.634600+0000","last_fullsized":"2024-09-17T04:06:49.634600+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:20:19.995005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.19","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649315+0000","last_change":"2024-09-17T04:06:47.641518+0000","last_active":"2024-09-17T04:06:49.649315+0000","last_peered":"2024-09-17T04:06:49.649315+0000","last_clean":"2024-09-17T04:06:49.649315+0000","last_became_active":"2024-09-17T04:06:47.641158+0000","last_became_peered":"2024-09-17T04:06:47.641158+0000","last_unstale":"2024-09-17T04:06:49.649315+0000","last_undegraded":"2024-09-17T04:06:49.649315+0000","last_fullsized":"2024-09-17T04:06:49.649315+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:34:43.208862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.19","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648503+0000","last_change":"2024-09-17T04:06:48.626505+0000","last_active":"2024-09-17T04:06:49.648503+0000","last_peered":"2024-09-17T04:06:49.648503+0000","last_clean":"2024-09-17T04:06:49.648503+0000","last_became_active":"2024-09-17T04:06:48.624676+0000","last_became_peered":"2024-09-17T04:06:48.624676+0000","last_unstale":"2024-09-17T04:06:49.648503+0000","last_undegraded":"2024-09-17T04:06:49.648503+0000","last_fullsized":"2024-09-17T04:06:49.648503+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:31:09.677668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.18","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648155+0000","last_change":"2024-09-17T04:06:47.637605+0000","last_active":"2024-09-17T04:06:49.648155+0000","last_peered":"2024-09-17T04:06:49.648155+0000","last_clean":"2024-09-17T04:06:49.648155+0000","last_became_active":"2024-09-17T04:06:47.637251+0000","last_became_peered":"2024-09-17T04:06:47.637251+0000","last_unstale":"2024-09-17T04:06:49.648155+0000","last_undegraded":"2024-09-17T04:06:49.648155+0000","last_fullsized":"2024-09-17T04:06:49.648155+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:56.783152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.16","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649374+0000","last_change":"2024-09-17T04:06:48.639801+0000","last_active":"2024-09-17T04:06:49.649374+0000","last_peered":"2024-09-17T04:06:49.649374+0000","last_clean":"2024-09-17T04:06:49.649374+0000","last_became_active":"2024-09-17T04:06:48.639471+0000","last_became_peered":"2024-09-17T04:06:48.639471+0000","last_unstale":"2024-09-17T04:06:49.649374+0000","last_undegraded":"2024-09-17T04:06:49.649374+0000","last_fullsized":"2024-09-17T04:06:49.649374+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:47.931408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.17","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649097+0000","last_change":"2024-09-17T04:06:47.639900+0000","last_active":"2024-09-17T04:06:49.649097+0000","last_peered":"2024-09-17T04:06:49.649097+0000","last_clean":"2024-09-17T04:06:49.649097+0000","last_became_active":"2024-09-17T04:06:47.638771+0000","last_became_peered":"2024-09-17T04:06:47.638771+0000","last_unstale":"2024-09-17T04:06:49.649097+0000","last_undegraded":"2024-09-17T04:06:49.649097+0000","last_fullsized":"2024-09-17T04:06:49.649097+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:17:42.102677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.17","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648930+0000","last_change":"2024-09-17T04:06:48.637971+0000","last_active":"2024-09-17T04:06:49.648930+0000","last_peered":"2024-09-17T04:06:49.648930+0000","last_clean":"2024-09-17T04:06:49.648930+0000","last_became_active":"2024-09-17T04:06:48.637655+0000","last_became_peered":"2024-09-17T04:06:48.637655+0000","last_unstale":"2024-09-17T04:06:49.648930+0000","last_undegraded":"2024-09-17T04:06:49.648930+0000","last_fullsized":"2024-09-17T04:06:49.648930+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:40:31.912869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.16","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648702+0000","last_change":"2024-09-17T04:06:47.628105+0000","last_active":"2024-09-17T04:06:49.648702+0000","last_peered":"2024-09-17T04:06:49.648702+0000","last_clean":"2024-09-17T04:06:49.648702+0000","last_became_active":"2024-09-17T04:06:47.627756+0000","last_became_peered":"2024-09-17T04:06:47.627756+0000","last_unstale":"2024-09-17T04:06:49.648702+0000","last_undegraded":"2024-09-17T04:06:49.648702+0000","last_fullsized":"2024-09-17T04:06:49.648702+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:23:38.274589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.14","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.651028+0000","last_change":"2024-09-17T04:06:48.646524+0000","last_active":"2024-09-17T04:06:49.651028+0000","last_peered":"2024-09-17T04:06:49.651028+0000","last_clean":"2024-09-17T04:06:49.651028+0000","last_became_active":"2024-09-17T04:06:48.646148+0000","last_became_peered":"2024-09-17T04:06:48.646148+0000","last_unstale":"2024-09-17T04:06:49.651028+0000","last_undegraded":"2024-09-17T04:06:49.651028+0000","last_fullsized":"2024-09-17T04:06:49.651028+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:41:11.580125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.15","version":"246'3","reported_seq":18,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650782+0000","last_change":"2024-09-17T04:06:47.635826+0000","last_active":"2024-09-17T04:06:49.650782+0000","last_peered":"2024-09-17T04:06:49.650782+0000","last_clean":"2024-09-17T04:06:49.650782+0000","last_became_active":"2024-09-17T04:06:47.633375+0000","last_became_peered":"2024-09-17T04:06:47.633375+0000","last_unstale":"2024-09-17T04:06:49.650782+0000","last_undegraded":"2024-09-17T04:06:49.650782+0000","last_fullsized":"2024-09-17T04:06:49.650782+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:13:13.957111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.15","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649114+0000","last_change":"2024-09-17T04:06:48.645322+0000","last_active":"2024-09-17T04:06:49.649114+0000","last_peered":"2024-09-17T04:06:49.649114+0000","last_clean":"2024-09-17T04:06:49.649114+0000","last_became_active":"2024-09-17T04:06:48.644967+0000","last_became_peered":"2024-09-17T04:06:48.644967+0000","last_unstale":"2024-09-17T04:06:49.649114+0000","last_undegraded":"2024-09-17T04:06:49.649114+0000","last_fullsized":"2024-09-17T04:06:49.649114+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:48:03.555488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.14","version":"246'6","reported_seq":21,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633743+0000","last_change":"2024-09-17T04:06:47.638185+0000","last_active":"2024-09-17T04:06:49.633743+0000","last_peered":"2024-09-17T04:06:49.633743+0000","last_clean":"2024-09-17T04:06:49.633743+0000","last_became_active":"2024-09-17T04:06:47.637416+0000","last_became_peered":"2024-09-17T04:06:47.637416+0000","last_unstale":"2024-09-17T04:06:49.633743+0000","last_undegraded":"2024-09-17T04:06:49.633743+0000","last_fullsized":"2024-09-17T04:06:49.633743+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:20.160754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.12","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648732+0000","last_change":"2024-09-17T04:06:48.647751+0000","last_active":"2024-09-17T04:06:49.648732+0000","last_peered":"2024-09-17T04:06:49.648732+0000","last_clean":"2024-09-17T04:06:49.648732+0000","last_became_active":"2024-09-17T04:06:48.647416+0000","last_became_peered":"2024-09-17T04:06:48.647416+0000","last_unstale":"2024-09-17T04:06:49.648732+0000","last_undegraded":"2024-09-17T04:06:49.648732+0000","last_fullsized":"2024-09-17T04:06:49.648732+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:01:55.210588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.13","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631957+0000","last_change":"2024-09-17T04:06:47.640722+0000","last_active":"2024-09-17T04:06:49.631957+0000","last_peered":"2024-09-17T04:06:49.631957+0000","last_clean":"2024-09-17T04:06:49.631957+0000","last_became_active":"2024-09-17T04:06:47.640406+0000","last_became_peered":"2024-09-17T04:06:47.640406+0000","last_unstale":"2024-09-17T04:06:49.631957+0000","last_undegraded":"2024-09-17T04:06:49.631957+0000","last_fullsized":"2024-09-17T04:06:49.631957+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:07:40.791049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.13","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634698+0000","last_change":"2024-09-17T04:06:48.628835+0000","last_active":"2024-09-17T04:06:49.634698+0000","last_peered":"2024-09-17T04:06:49.634698+0000","last_clean":"2024-09-17T04:06:49.634698+0000","last_became_active":"2024-09-17T04:06:48.628468+0000","last_became_peered":"2024-09-17T04:06:48.628468+0000","last_unstale":"2024-09-17T04:06:49.634698+0000","last_undegraded":"2024-09-17T04:06:49.634698+0000","last_fullsized":"2024-09-17T04:06:49.634698+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:45:12.456213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.12","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650406+0000","last_change":"2024-09-17T04:06:47.629947+0000","last_active":"2024-09-17T04:06:49.650406+0000","last_peered":"2024-09-17T04:06:49.650406+0000","last_clean":"2024-09-17T04:06:49.650406+0000","last_became_active":"2024-09-17T04:06:47.628403+0000","last_became_peered":"2024-09-17T04:06:47.628403+0000","last_unstale":"2024-09-17T04:06:49.650406+0000","last_undegraded":"2024-09-17T04:06:49.650406+0000","last_fullsized":"2024-09-17T04:06:49.650406+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:02:15.090695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.10","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649384+0000","last_change":"2024-09-17T04:06:48.650761+0000","last_active":"2024-09-17T04:06:49.649384+0000","last_peered":"2024-09-17T04:06:49.649384+0000","last_clean":"2024-09-17T04:06:49.649384+0000","last_became_active":"2024-09-17T04:06:48.650386+0000","last_became_peered":"2024-09-17T04:06:48.650386+0000","last_unstale":"2024-09-17T04:06:49.649384+0000","last_undegraded":"2024-09-17T04:06:49.649384+0000","last_fullsized":"2024-09-17T04:06:49.649384+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:16:27.321629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.11","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.633982+0000","last_change":"2024-09-17T04:06:47.637094+0000","last_active":"2024-09-17T04:06:49.633982+0000","last_peered":"2024-09-17T04:06:49.633982+0000","last_clean":"2024-09-17T04:06:49.633982+0000","last_became_active":"2024-09-17T04:06:47.636787+0000","last_became_peered":"2024-09-17T04:06:47.636787+0000","last_unstale":"2024-09-17T04:06:49.633982+0000","last_undegraded":"2024-09-17T04:06:49.633982+0000","last_fullsized":"2024-09-17T04:06:49.633982+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:56:24.521931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"37.11","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634167+0000","last_change":"2024-09-17T04:06:48.642186+0000","last_active":"2024-09-17T04:06:49.634167+0000","last_peered":"2024-09-17T04:06:49.634167+0000","last_clean":"2024-09-17T04:06:49.634167+0000","last_became_active":"2024-09-17T04:06:48.640270+0000","last_became_peered":"2024-09-17T04:06:48.640270+0000","last_unstale":"2024-09-17T04:06:49.634167+0000","last_undegraded":"2024-09-17T04:06:49.634167+0000","last_fullsized":"2024-09-17T04:06:49.634167+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:21:23.763556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.10","version":"246'8","reported_seq":23,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649303+0000","last_change":"2024-09-17T04:06:47.639284+0000","last_active":"2024-09-17T04:06:49.649303+0000","last_peered":"2024-09-17T04:06:49.649303+0000","last_clean":"2024-09-17T04:06:49.649303+0000","last_became_active":"2024-09-17T04:06:47.638987+0000","last_became_peered":"2024-09-17T04:06:47.638987+0000","last_unstale":"2024-09-17T04:06:49.649303+0000","last_undegraded":"2024-09-17T04:06:49.649303+0000","last_fullsized":"2024-09-17T04:06:49.649303+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:24:25.636357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.e","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650980+0000","last_change":"2024-09-17T04:06:48.640025+0000","last_active":"2024-09-17T04:06:49.650980+0000","last_peered":"2024-09-17T04:06:49.650980+0000","last_clean":"2024-09-17T04:06:49.650980+0000","last_became_active":"2024-09-17T04:06:48.639713+0000","last_became_peered":"2024-09-17T04:06:48.639713+0000","last_unstale":"2024-09-17T04:06:49.650980+0000","last_undegraded":"2024-09-17T04:06:49.650980+0000","last_fullsized":"2024-09-17T04:06:49.650980+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:18.826084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.f","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632203+0000","last_change":"2024-09-17T04:06:47.627885+0000","last_active":"2024-09-17T04:06:49.632203+0000","last_peered":"2024-09-17T04:06:49.632203+0000","last_clean":"2024-09-17T04:06:49.632203+0000","last_became_active":"2024-09-17T04:06:47.627314+0000","last_became_peered":"2024-09-17T04:06:47.627314+0000","last_unstale":"2024-09-17T04:06:49.632203+0000","last_undegraded":"2024-09-17T04:06:49.632203+0000","last_fullsized":"2024-09-17T04:06:49.632203+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:32.996351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.f","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650153+0000","last_change":"2024-09-17T04:06:48.640039+0000","last_active":"2024-09-17T04:06:49.650153+0000","last_peered":"2024-09-17T04:06:49.650153+0000","last_clean":"2024-09-17T04:06:49.650153+0000","last_became_active":"2024-09-17T04:06:48.639062+0000","last_became_peered":"2024-09-17T04:06:48.639062+0000","last_unstale":"2024-09-17T04:06:49.650153+0000","last_undegraded":"2024-09-17T04:06:49.650153+0000","last_fullsized":"2024-09-17T04:06:49.650153+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:11:54.537308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.e","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648863+0000","last_change":"2024-09-17T04:06:47.626021+0000","last_active":"2024-09-17T04:06:49.648863+0000","last_peered":"2024-09-17T04:06:49.648863+0000","last_clean":"2024-09-17T04:06:49.648863+0000","last_became_active":"2024-09-17T04:06:47.625353+0000","last_became_peered":"2024-09-17T04:06:47.625353+0000","last_unstale":"2024-09-17T04:06:49.648863+0000","last_undegraded":"2024-09-17T04:06:49.648863+0000","last_fullsized":"2024-09-17T04:06:49.648863+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:50.080502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.c","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649068+0000","last_change":"2024-09-17T04:06:48.650829+0000","last_active":"2024-09-17T04:06:49.649068+0000","last_peered":"2024-09-17T04:06:49.649068+0000","last_clean":"2024-09-17T04:06:49.649068+0000","last_became_active":"2024-09-17T04:06:48.649204+0000","last_became_peered":"2024-09-17T04:06:48.649204+0000","last_unstale":"2024-09-17T04:06:49.649068+0000","last_undegraded":"2024-09-17T04:06:49.649068+0000","last_fullsized":"2024-09-17T04:06:49.649068+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:35.431985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.d","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649197+0000","last_change":"2024-09-17T04:06:47.630827+0000","last_active":"2024-09-17T04:06:49.649197+0000","last_peered":"2024-09-17T04:06:49.649197+0000","last_clean":"2024-09-17T04:06:49.649197+0000","last_became_active":"2024-09-17T04:06:47.630146+0000","last_became_peered":"2024-09-17T04:06:47.630146+0000","last_unstale":"2024-09-17T04:06:49.649197+0000","last_undegraded":"2024-09-17T04:06:49.649197+0000","last_fullsized":"2024-09-17T04:06:49.649197+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:56:59.862903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.d","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649039+0000","last_change":"2024-09-17T04:06:48.655091+0000","last_active":"2024-09-17T04:06:49.649039+0000","last_peered":"2024-09-17T04:06:49.649039+0000","last_clean":"2024-09-17T04:06:49.649039+0000","last_became_active":"2024-09-17T04:06:48.654782+0000","last_became_peered":"2024-09-17T04:06:48.654782+0000","last_unstale":"2024-09-17T04:06:49.649039+0000","last_undegraded":"2024-09-17T04:06:49.649039+0000","last_fullsized":"2024-09-17T04:06:49.649039+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:39.823146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.c","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649147+0000","last_change":"2024-09-17T04:06:47.639967+0000","last_active":"2024-09-17T04:06:49.649147+0000","last_peered":"2024-09-17T04:06:49.649147+0000","last_clean":"2024-09-17T04:06:49.649147+0000","last_became_active":"2024-09-17T04:06:47.639606+0000","last_became_peered":"2024-09-17T04:06:47.639606+0000","last_unstale":"2024-09-17T04:06:49.649147+0000","last_undegraded":"2024-09-17T04:06:49.649147+0000","last_fullsized":"2024-09-17T04:06:49.649147+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:53.671166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.a","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649644+0000","last_change":"2024-09-17T04:06:48.642222+0000","last_active":"2024-09-17T04:06:49.649644+0000","last_peered":"2024-09-17T04:06:49.649644+0000","last_clean":"2024-09-17T04:06:49.649644+0000","last_became_active":"2024-09-17T04:06:48.641845+0000","last_became_peered":"2024-09-17T04:06:48.641845+0000","last_unstale":"2024-09-17T04:06:49.649644+0000","last_undegraded":"2024-09-17T04:06:49.649644+0000","last_fullsized":"2024-09-17T04:06:49.649644+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:50:18.092195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"36.b","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650657+0000","last_change":"2024-09-17T04:06:47.635036+0000","last_active":"2024-09-17T04:06:49.650657+0000","last_peered":"2024-09-17T04:06:49.650657+0000","last_clean":"2024-09-17T04:06:49.650657+0000","last_became_active":"2024-09-17T04:06:47.634721+0000","last_became_peered":"2024-09-17T04:06:47.634721+0000","last_unstale":"2024-09-17T04:06:49.650657+0000","last_undegraded":"2024-09-17T04:06:49.650657+0000","last_fullsized":"2024-09-17T04:06:49.650657+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:50:48.022710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.b","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649395+0000","last_change":"2024-09-17T04:06:48.647597+0000","last_active":"2024-09-17T04:06:49.649395+0000","last_peered":"2024-09-17T04:06:49.649395+0000","last_clean":"2024-09-17T04:06:49.649395+0000","last_became_active":"2024-09-17T04:06:48.647265+0000","last_became_peered":"2024-09-17T04:06:48.647265+0000","last_unstale":"2024-09-17T04:06:49.649395+0000","last_undegraded":"2024-09-17T04:06:49.649395+0000","last_fullsized":"2024-09-17T04:06:49.649395+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:22:54.511634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.a","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631987+0000","last_change":"2024-09-17T04:06:47.627934+0000","last_active":"2024-09-17T04:06:49.631987+0000","last_peered":"2024-09-17T04:06:49.631987+0000","last_clean":"2024-09-17T04:06:49.631987+0000","last_became_active":"2024-09-17T04:06:47.627651+0000","last_became_peered":"2024-09-17T04:06:47.627651+0000","last_unstale":"2024-09-17T04:06:49.631987+0000","last_undegraded":"2024-09-17T04:06:49.631987+0000","last_fullsized":"2024-09-17T04:06:49.631987+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:28.266741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.8","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631723+0000","last_change":"2024-09-17T04:06:48.630489+0000","last_active":"2024-09-17T04:06:49.631723+0000","last_peered":"2024-09-17T04:06:49.631723+0000","last_clean":"2024-09-17T04:06:49.631723+0000","last_became_active":"2024-09-17T04:06:48.630199+0000","last_became_peered":"2024-09-17T04:06:48.630199+0000","last_unstale":"2024-09-17T04:06:49.631723+0000","last_undegraded":"2024-09-17T04:06:49.631723+0000","last_fullsized":"2024-09-17T04:06:49.631723+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:01.307233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.9","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648431+0000","last_change":"2024-09-17T04:06:47.636455+0000","last_active":"2024-09-17T04:06:49.648431+0000","last_peered":"2024-09-17T04:06:49.648431+0000","last_clean":"2024-09-17T04:06:49.648431+0000","last_became_active":"2024-09-17T04:06:47.636161+0000","last_became_peered":"2024-09-17T04:06:47.636161+0000","last_unstale":"2024-09-17T04:06:49.648431+0000","last_undegraded":"2024-09-17T04:06:49.648431+0000","last_fullsized":"2024-09-17T04:06:49.648431+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:42:26.341915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"37.9","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631704+0000","last_change":"2024-09-17T04:06:48.626073+0000","last_active":"2024-09-17T04:06:49.631704+0000","last_peered":"2024-09-17T04:06:49.631704+0000","last_clean":"2024-09-17T04:06:49.631704+0000","last_became_active":"2024-09-17T04:06:48.624659+0000","last_became_peered":"2024-09-17T04:06:48.624659+0000","last_unstale":"2024-09-17T04:06:49.631704+0000","last_undegraded":"2024-09-17T04:06:49.631704+0000","last_fullsized":"2024-09-17T04:06:49.631704+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:50:03.812133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.8","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631911+0000","last_change":"2024-09-17T04:06:47.631678+0000","last_active":"2024-09-17T04:06:49.631911+0000","last_peered":"2024-09-17T04:06:49.631911+0000","last_clean":"2024-09-17T04:06:49.631911+0000","last_became_active":"2024-09-17T04:06:47.629357+0000","last_became_peered":"2024-09-17T04:06:47.629357+0000","last_unstale":"2024-09-17T04:06:49.631911+0000","last_undegraded":"2024-09-17T04:06:49.631911+0000","last_fullsized":"2024-09-17T04:06:49.631911+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:41:58.960464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.6","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650308+0000","last_change":"2024-09-17T04:06:48.641309+0000","last_active":"2024-09-17T04:06:49.650308+0000","last_peered":"2024-09-17T04:06:49.650308+0000","last_clean":"2024-09-17T04:06:49.650308+0000","last_became_active":"2024-09-17T04:06:48.640780+0000","last_became_peered":"2024-09-17T04:06:48.640780+0000","last_unstale":"2024-09-17T04:06:49.650308+0000","last_undegraded":"2024-09-17T04:06:49.650308+0000","last_fullsized":"2024-09-17T04:06:49.650308+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:47:08.371703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"36.7","version":"246'2","reported_seq":17,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.651287+0000","last_change":"2024-09-17T04:06:47.634843+0000","last_active":"2024-09-17T04:06:49.651287+0000","last_peered":"2024-09-17T04:06:49.651287+0000","last_clean":"2024-09-17T04:06:49.651287+0000","last_became_active":"2024-09-17T04:06:47.634529+0000","last_became_peered":"2024-09-17T04:06:47.634529+0000","last_unstale":"2024-09-17T04:06:49.651287+0000","last_undegraded":"2024-09-17T04:06:49.651287+0000","last_fullsized":"2024-09-17T04:06:49.651287+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:31:24.241367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.7","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634196+0000","last_change":"2024-09-17T04:06:48.637294+0000","last_active":"2024-09-17T04:06:49.634196+0000","last_peered":"2024-09-17T04:06:49.634196+0000","last_clean":"2024-09-17T04:06:49.634196+0000","last_became_active":"2024-09-17T04:06:48.636742+0000","last_became_peered":"2024-09-17T04:06:48.636742+0000","last_unstale":"2024-09-17T04:06:49.634196+0000","last_undegraded":"2024-09-17T04:06:49.634196+0000","last_fullsized":"2024-09-17T04:06:49.634196+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:40.916077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"36.6","version":"246'1","reported_seq":16,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.632108+0000","last_change":"2024-09-17T04:06:47.635594+0000","last_active":"2024-09-17T04:06:49.632108+0000","last_peered":"2024-09-17T04:06:49.632108+0000","last_clean":"2024-09-17T04:06:49.632108+0000","last_became_active":"2024-09-17T04:06:47.633247+0000","last_became_peered":"2024-09-17T04:06:47.633247+0000","last_unstale":"2024-09-17T04:06:49.632108+0000","last_undegraded":"2024-09-17T04:06:49.632108+0000","last_fullsized":"2024-09-17T04:06:49.632108+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:59:29.409434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"37.4","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648651+0000","last_change":"2024-09-17T04:06:48.654761+0000","last_active":"2024-09-17T04:06:49.648651+0000","last_peered":"2024-09-17T04:06:49.648651+0000","last_clean":"2024-09-17T04:06:49.648651+0000","last_became_active":"2024-09-17T04:06:48.654228+0000","last_became_peered":"2024-09-17T04:06:48.654228+0000","last_unstale":"2024-09-17T04:06:49.648651+0000","last_undegraded":"2024-09-17T04:06:49.648651+0000","last_fullsized":"2024-09-17T04:06:49.648651+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:54:46.775630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"36.5","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650896+0000","last_change":"2024-09-17T04:06:47.631080+0000","last_active":"2024-09-17T04:06:49.650896+0000","last_peered":"2024-09-17T04:06:49.650896+0000","last_clean":"2024-09-17T04:06:49.650896+0000","last_became_active":"2024-09-17T04:06:47.630810+0000","last_became_peered":"2024-09-17T04:06:47.630810+0000","last_unstale":"2024-09-17T04:06:49.650896+0000","last_undegraded":"2024-09-17T04:06:49.650896+0000","last_fullsized":"2024-09-17T04:06:49.650896+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:42:38.601411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.5","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.634552+0000","last_change":"2024-09-17T04:06:48.640563+0000","last_active":"2024-09-17T04:06:49.634552+0000","last_peered":"2024-09-17T04:06:49.634552+0000","last_clean":"2024-09-17T04:06:49.634552+0000","last_became_active":"2024-09-17T04:06:48.640193+0000","last_became_peered":"2024-09-17T04:06:48.640193+0000","last_unstale":"2024-09-17T04:06:49.634552+0000","last_undegraded":"2024-09-17T04:06:49.634552+0000","last_fullsized":"2024-09-17T04:06:49.634552+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:33:15.045992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.4","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649735+0000","last_change":"2024-09-17T04:06:47.640808+0000","last_active":"2024-09-17T04:06:49.649735+0000","last_peered":"2024-09-17T04:06:49.649735+0000","last_clean":"2024-09-17T04:06:49.649735+0000","last_became_active":"2024-09-17T04:06:47.639846+0000","last_became_peered":"2024-09-17T04:06:47.639846+0000","last_unstale":"2024-09-17T04:06:49.649735+0000","last_undegraded":"2024-09-17T04:06:49.649735+0000","last_fullsized":"2024-09-17T04:06:49.649735+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:25:15.837166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.2","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648675+0000","last_change":"2024-09-17T04:06:48.623052+0000","last_active":"2024-09-17T04:06:49.648675+0000","last_peered":"2024-09-17T04:06:49.648675+0000","last_clean":"2024-09-17T04:06:49.648675+0000","last_became_active":"2024-09-17T04:06:48.622715+0000","last_became_peered":"2024-09-17T04:06:48.622715+0000","last_unstale":"2024-09-17T04:06:49.648675+0000","last_undegraded":"2024-09-17T04:06:49.648675+0000","last_fullsized":"2024-09-17T04:06:49.648675+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:03:55.252993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"36.3","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649527+0000","last_change":"2024-09-17T04:06:47.639290+0000","last_active":"2024-09-17T04:06:49.649527+0000","last_peered":"2024-09-17T04:06:49.649527+0000","last_clean":"2024-09-17T04:06:49.649527+0000","last_became_active":"2024-09-17T04:06:47.638370+0000","last_became_peered":"2024-09-17T04:06:47.638370+0000","last_unstale":"2024-09-17T04:06:49.649527+0000","last_undegraded":"2024-09-17T04:06:49.649527+0000","last_fullsized":"2024-09-17T04:06:49.649527+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:43:17.818126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"37.3","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650636+0000","last_change":"2024-09-17T04:06:48.645144+0000","last_active":"2024-09-17T04:06:49.650636+0000","last_peered":"2024-09-17T04:06:49.650636+0000","last_clean":"2024-09-17T04:06:49.650636+0000","last_became_active":"2024-09-17T04:06:48.644506+0000","last_became_peered":"2024-09-17T04:06:48.644506+0000","last_unstale":"2024-09-17T04:06:49.650636+0000","last_undegraded":"2024-09-17T04:06:49.650636+0000","last_fullsized":"2024-09-17T04:06:49.650636+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:10.263688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"36.2","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650443+0000","last_change":"2024-09-17T04:06:47.637796+0000","last_active":"2024-09-17T04:06:49.650443+0000","last_peered":"2024-09-17T04:06:49.650443+0000","last_clean":"2024-09-17T04:06:49.650443+0000","last_became_active":"2024-09-17T04:06:47.637473+0000","last_became_peered":"2024-09-17T04:06:47.637473+0000","last_unstale":"2024-09-17T04:06:49.650443+0000","last_undegraded":"2024-09-17T04:06:49.650443+0000","last_fullsized":"2024-09-17T04:06:49.650443+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:39:35.426948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.0","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.631887+0000","last_change":"2024-09-17T04:06:48.626386+0000","last_active":"2024-09-17T04:06:49.631887+0000","last_peered":"2024-09-17T04:06:49.631887+0000","last_clean":"2024-09-17T04:06:49.631887+0000","last_became_active":"2024-09-17T04:06:48.623872+0000","last_became_peered":"2024-09-17T04:06:48.623872+0000","last_unstale":"2024-09-17T04:06:49.631887+0000","last_undegraded":"2024-09-17T04:06:49.631887+0000","last_fullsized":"2024-09-17T04:06:49.631887+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:17:35.483013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"36.1","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.648365+0000","last_change":"2024-09-17T04:06:47.632594+0000","last_active":"2024-09-17T04:06:49.648365+0000","last_peered":"2024-09-17T04:06:49.648365+0000","last_clean":"2024-09-17T04:06:49.648365+0000","last_became_active":"2024-09-17T04:06:47.632229+0000","last_became_peered":"2024-09-17T04:06:47.632229+0000","last_unstale":"2024-09-17T04:06:49.648365+0000","last_undegraded":"2024-09-17T04:06:49.648365+0000","last_fullsized":"2024-09-17T04:06:49.648365+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:10.174881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"37.1","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.635191+0000","last_change":"2024-09-17T04:06:48.640651+0000","last_active":"2024-09-17T04:06:49.635191+0000","last_peered":"2024-09-17T04:06:49.635191+0000","last_clean":"2024-09-17T04:06:49.635191+0000","last_became_active":"2024-09-17T04:06:48.639761+0000","last_became_peered":"2024-09-17T04:06:48.639761+0000","last_unstale":"2024-09-17T04:06:49.635191+0000","last_undegraded":"2024-09-17T04:06:49.635191+0000","last_fullsized":"2024-09-17T04:06:49.635191+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:29.184251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"36.0","version":"0'0","reported_seq":15,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649417+0000","last_change":"2024-09-17T04:06:47.639117+0000","last_active":"2024-09-17T04:06:49.649417+0000","last_peered":"2024-09-17T04:06:49.649417+0000","last_clean":"2024-09-17T04:06:49.649417+0000","last_became_active":"2024-09-17T04:06:47.638778+0000","last_became_peered":"2024-09-17T04:06:47.638778+0000","last_unstale":"2024-09-17T04:06:49.649417+0000","last_undegraded":"2024-09-17T04:06:49.649417+0000","last_fullsized":"2024-09-17T04:06:49.649417+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:46.593786+0000","last_clean_scrub_stamp":"2024-09-17T04:06:46.593786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:52:42.527414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"37.24","version":"0'0","reported_seq":13,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.650819+0000","last_change":"2024-09-17T04:06:48.648394+0000","last_active":"2024-09-17T04:06:49.650819+0000","last_peered":"2024-09-17T04:06:49.650819+0000","last_clean":"2024-09-17T04:06:49.650819+0000","last_became_active":"2024-09-17T04:06:48.647962+0000","last_became_peered":"2024-09-17T04:06:48.647962+0000","last_unstale":"2024-09-17T04:06:49.650819+0000","last_undegraded":"2024-09-17T04:06:49.650819+0000","last_fullsized":"2024-09-17T04:06:49.650819+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":245,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:06:47.601743+0000","last_clean_scrub_stamp":"2024-09-17T04:06:47.601743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:36:05.482211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":527,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-09-17T04:06:49.649624+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:06:49.649624+0000","last_peered":"2024-09-17T04:06:49.649624+0000","last_clean":"2024-09-17T04:06:49.649624+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:06:49.649624+0000","last_undegraded":"2024-09-17T04:06:49.649624+0000","last_fullsized":"2024-09-17T04:06:49.649624+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":37,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":36,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509652,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52876,"kb_used_data":3632,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3719168,"data_stored":1975944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509655,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34264,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337576,"statfs":{"total":96636764160,"available":96601677824,"internally_reserved":0,"allocated":3141632,"data_stored":1387278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509654,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56464,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315376,"statfs":{"total":96636764160,"available":96578945024,"internally_reserved":0,"allocated":3133440,"data_stored":1386228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509652,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56460,"kb_used_data":3056,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315380,"statfs":{"total":96636764160,"available":96578949120,"internally_reserved":0,"allocated":3129344,"data_stored":1386158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509654,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56464,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315376,"statfs":{"total":96636764160,"available":96578945024,"internally_reserved":0,"allocated":3133440,"data_stored":1385700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509654,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52304,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319536,"statfs":{"total":96636764160,"available":96583204864,"internally_reserved":0,"allocated":3133440,"data_stored":1385632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542356,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56464,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315376,"statfs":{"total":96636764160,"available":96578945024,"internally_reserved":0,"allocated":3137536,"data_stored":1387188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542357,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52876,"kb_used_data":3640,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318964,"statfs":{"total":96636764160,"available":96582619136,"internally_reserved":0,"allocated":3727360,"data_stored":1976036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":36,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":37,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:06:58.482 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:06:58.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 -- 172.21.15.129:0/1718663258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc0ce2c0 msgr2=0x7fbcfc0ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 --2- 172.21.15.129:0/1718663258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc0ce2c0 0x7fbcfc0ce6d0 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4005fd0 tx=0x7fbcf402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 -- 172.21.15.129:0/1718663258 shutdown_connections 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 --2- 172.21.15.129:0/1718663258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc0ce2c0 0x7fbcfc0ce6d0 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 --2- 172.21.15.129:0/1718663258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc06ee70 0x7fbcfc06f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 -- 172.21.15.129:0/1718663258 >> 172.21.15.129:0/1718663258 conn(0x7fbcfc0d0120 msgr2=0x7fbcfc0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:58.748 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 -- 172.21.15.129:0/1718663258 shutdown_connections 2024-09-17T04:06:58.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.747+0000 7fbcfb200640 1 -- 172.21.15.129:0/1718663258 wait complete. 2024-09-17T04:06:58.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.749+0000 7fbcfb200640 1 Processor -- start 2024-09-17T04:06:58.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.750+0000 7fbcfb200640 1 -- start start 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc0ce2c0 0x7fbcfc15a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 0x7fbcfc15a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc0ced90 con 0x7fbcfc06ee70 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc0cec10 con 0x7fbcfc156f20 2024-09-17T04:06:58.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcfb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbcfc0cef10 con 0x7fbcfc0ce2c0 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcf9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcf9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52268/0 (socket says 172.21.15.129:52268) 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.751+0000 7fbcf9e00640 1 -- 172.21.15.129:0/536446351 learned_addr learned my addr 172.21.15.129:0/536446351 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9400640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc0ce2c0 0x7fbcfc15a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcfa800640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 0x7fbcfc15a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 -- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc0ce2c0 msgr2=0x7fbcfc15a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc0ce2c0 0x7fbcfc15a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 -- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 msgr2=0x7fbcfc15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 0x7fbcfc15a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 -- 172.21.15.129:0/536446351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc158530 con 0x7fbcfc06ee70 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcfa800640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 0x7fbcfc15a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:06:58.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf9e00640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c370 tx=0x7fbcec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:58.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec011790 con 0x7fbcfc06ee70 2024-09-17T04:06:58.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.752+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcec017070 con 0x7fbcfc06ee70 2024-09-17T04:06:58.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.753+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec011c80 con 0x7fbcfc06ee70 2024-09-17T04:06:58.754 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.753+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc153b10 con 0x7fbcfc06ee70 2024-09-17T04:06:58.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.753+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc1e9660 con 0x7fbcfc06ee70 2024-09-17T04:06:58.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.754+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcfc0cf3e0 con 0x7fbcfc06ee70 2024-09-17T04:06:58.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.755+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbcec01c460 con 0x7fbcfc06ee70 2024-09-17T04:06:58.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.756+0000 7fbcf2000640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 0x7fbcdc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:58.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.756+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7fbcec09ab30 con 0x7fbcfc06ee70 2024-09-17T04:06:58.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.756+0000 7fbcf9400640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 0x7fbcdc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:58.757 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.757+0000 7fbcf9400640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 0x7fbcdc07b0a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fbce8005fd0 tx=0x7fbce8005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:58.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:58.760+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbcfc06f7c0 con 0x7fbcfc06ee70 2024-09-17T04:06:59.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.018+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fbcfc0b5ab0 con 0x7fbcfc06ee70 2024-09-17T04:06:59.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.022+0000 7fbcf2000640 1 -- 172.21.15.129:0/536446351 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v46) ==== 190+0+147 (secure 0 0 0) 0x7fbcec062f60 con 0x7fbcfc06ee70 2024-09-17T04:06:59.023 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:06:59.023 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:06:59.023 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:06:59.023 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:06:59.023 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:06:59.024 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:06:59.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 msgr2=0x7fbcdc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 0x7fbcdc07b0a0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fbce8005fd0 tx=0x7fbce8005850 comp rx=0 tx=0).stop 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 msgr2=0x7fbcfc1535d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c370 tx=0x7fbcec00c830 comp rx=0 tx=0).stop 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 shutdown_connections 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbcdc078b90 0x7fbcdc07b0a0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcfc156f20 0x7fbcfc15a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcfc0ce2c0 0x7fbcfc15a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 --2- 172.21.15.129:0/536446351 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbcfc06ee70 0x7fbcfc1535d0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 >> 172.21.15.129:0/536446351 conn(0x7fbcfc0d0120 msgr2=0x7fbcfc0afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 shutdown_connections 2024-09-17T04:06:59.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.025+0000 7fbcfb200640 1 -- 172.21.15.129:0/536446351 wait complete. 2024-09-17T04:06:59.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 -- 172.21.15.129:0/3672621698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400cd1c0 msgr2=0x7fae401609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 --2- 172.21.15.129:0/3672621698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400cd1c0 0x7fae401609c0 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7fae40072080 tx=0x7fae3802aac0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 -- 172.21.15.129:0/3672621698 shutdown_connections 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 --2- 172.21.15.129:0/3672621698 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400cd1c0 0x7fae401609c0 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 --2- 172.21.15.129:0/3672621698 >> v2:172.21.15.181:3300/0 conn(0x7fae400b4140 0x7fae400ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 -- 172.21.15.129:0/3672621698 >> 172.21.15.129:0/3672621698 conn(0x7fae4006d410 msgr2=0x7fae400b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 -- 172.21.15.129:0/3672621698 shutdown_connections 2024-09-17T04:06:59.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.270+0000 7fae44a00640 1 -- 172.21.15.129:0/3672621698 wait complete. 2024-09-17T04:06:59.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.273+0000 7fae44a00640 1 Processor -- start 2024-09-17T04:06:59.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.273+0000 7fae44a00640 1 -- start start 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae400705a0 0x7fae4015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae400cd1c0 0x7fae401580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae40160260 con 0x7fae400b4140 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae401600e0 con 0x7fae400cd1c0 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae44a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fae40160f00 con 0x7fae400705a0 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.274+0000 7fae37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:59.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52288/0 (socket says 172.21.15.129:52288) 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 -- 172.21.15.129:0/179701835 learned_addr learned my addr 172.21.15.129:0/179701835 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae3e800640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae400cd1c0 0x7fae401580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae3de00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae400705a0 0x7fae4015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 -- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae400705a0 msgr2=0x7fae4015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae400705a0 0x7fae4015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 -- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae400cd1c0 msgr2=0x7fae401580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae400cd1c0 0x7fae401580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 -- 172.21.15.129:0/179701835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae40155350 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.275+0000 7fae37e00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fae28009fd0 tx=0x7fae2800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.276+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800ccc0 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.276+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae2800ee70 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.276+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800ce60 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.276+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4015b7c0 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.276+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae401f63c0 con 0x7fae400b4140 2024-09-17T04:06:59.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.277+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae40155630 con 0x7fae400b4140 2024-09-17T04:06:59.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.278+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fae28005260 con 0x7fae400b4140 2024-09-17T04:06:59.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.279+0000 7fae36000640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 0x7fae1c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:06:59.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.279+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7fae2809b5c0 con 0x7fae400b4140 2024-09-17T04:06:59.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.280+0000 7fae3de00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 0x7fae1c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:06:59.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.280+0000 7fae3de00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 0x7fae1c07b0a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fae30001720 tx=0x7fae30009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:06:59.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.283+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fae280633a0 con 0x7fae400b4140 2024-09-17T04:06:59.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.502+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae400ccc80 con 0x7fae400b4140 2024-09-17T04:06:59.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.502+0000 7fae36000640 1 -- 172.21.15.129:0/179701835 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) ==== 95+0+5246 (secure 0 0 0) 0x7fae28067dd0 con 0x7fae400b4140 2024-09-17T04:06:59.503 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:06:59.503 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":235,"btime":"2024-09-17T04:06:51:636027+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:06:51.125080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11778},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11778":{"gid":11778,"name":"c","rank":0,"incarnation":232,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/968815434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":968815434},{"type":"v1","addr":"172.21.15.129:6835","nonce":968815434}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11778,"qdb_cluster":[11778]},"id":18}]} 2024-09-17T04:06:59.503 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 235 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 msgr2=0x7fae1c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 0x7fae1c07b0a0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fae30001720 tx=0x7fae30009040 comp rx=0 tx=0).stop 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 msgr2=0x7fae40157ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7fae28009fd0 tx=0x7fae2800ec20 comp rx=0 tx=0).stop 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 shutdown_connections 2024-09-17T04:06:59.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae1c078b90 0x7fae1c07b0a0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae400cd1c0 0x7fae401580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae400b4140 0x7fae40157ba0 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 --2- 172.21.15.129:0/179701835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae400705a0 0x7fae4015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 >> 172.21.15.129:0/179701835 conn(0x7fae4006d410 msgr2=0x7fae400b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 shutdown_connections 2024-09-17T04:06:59.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:06:59.505+0000 7fae44a00640 1 -- 172.21.15.129:0/179701835 wait complete. 2024-09-17T04:06:59.930 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 235, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:06:48.110755+0000', 'modified': '2024-09-17T04:06:51.125080+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11778}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11778': {'gid': 11778, 'name': 'c', 'rank': 0, 'incarnation': 232, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/968815434', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 968815434}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 968815434}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11778, 'qdb_cluster': [11778]} 2024-09-17T04:06:59.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:06:59.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:07:00.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 -- 172.21.15.129:0/1132267346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 msgr2=0x7f3f1c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 --2- 172.21.15.129:0/1132267346 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f1c004800 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009b20 tx=0x7f3f1c005830 comp rx=0 tx=0).stop 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f39e00640 1 -- 172.21.15.129:0/1132267346 reap_dead start 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 -- 172.21.15.129:0/1132267346 shutdown_connections 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 -- 172.21.15.129:0/1132267346 >> 172.21.15.129:0/1132267346 conn(0x7f3f3406d410 msgr2=0x7f3f340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 -- 172.21.15.129:0/1132267346 shutdown_connections 2024-09-17T04:07:00.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.154+0000 7f3f3be00640 1 -- 172.21.15.129:0/1132267346 wait complete. 2024-09-17T04:07:00.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.156+0000 7f3f3be00640 1 Processor -- start 2024-09-17T04:07:00.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.157+0000 7f3f3be00640 1 -- start start 2024-09-17T04:07:00.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 0x7f3f3415b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 0x7f3f3415efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f340cf8d0 con 0x7f3f1c003fd0 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f340cb1e0 con 0x7f3f340b2140 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f3be00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f340cfa50 con 0x7f3f341f9790 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52318/0 (socket says 172.21.15.129:52318) 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.158+0000 7f3f39400640 1 -- 172.21.15.129:0/2639669614 learned_addr learned my addr 172.21.15.129:0/2639669614 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39e00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 0x7f3f3415efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 0x7f3f3415b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 -- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 msgr2=0x7f3f3415b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 0x7f3f3415b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 -- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 msgr2=0x7f3f3415efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 0x7f3f3415efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 -- 172.21.15.129:0/2639669614 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f340c5e40 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39e00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 0x7f3f3415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 0x7f3f3415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f39400640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f3f28011ae0 tx=0x7f3f28009e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f31600640 1 -- 172.21.15.129:0/2639669614 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2801e070 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f31600640 1 -- 172.21.15.129:0/2639669614 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f28004460 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f340b2550 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.159+0000 7f3f31600640 1 -- 172.21.15.129:0/2639669614 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2800fc60 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.160+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f34161b00 con 0x7f3f1c003fd0 2024-09-17T04:07:00.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.162+0000 7f3f31600640 1 -- 172.21.15.129:0/2639669614 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3f28021070 con 0x7f3f1c003fd0 2024-09-17T04:07:00.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.163+0000 7f3f31600640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 0x7f3f1407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.163+0000 7f3f31600640 1 -- 172.21.15.129:0/2639669614 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f3f2809b6c0 con 0x7f3f1c003fd0 2024-09-17T04:07:00.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.163+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 0x7f3f1407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.163+0000 7f3f42ec1740 1 Processor -- start 2024-09-17T04:07:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.164+0000 7f3f42ec1740 1 -- start start 2024-09-17T04:07:00.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.164+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 0x7f3f1407b0a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3f20035000 tx=0x7f3f200046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd936f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd933ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fcd92c9430 con 0x55fcd8f763b0 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fcd92b95d0 con 0x55fcd8f9f270 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55fcd92a8a00 con 0x55fcd8f7e590 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f38a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f38a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52320/0 (socket says 172.21.15.129:52320) 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f38a00640 1 -- 172.21.15.129:0/1398958906 learned_addr learned my addr 172.21.15.129:0/1398958906 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f39400640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd936f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.165+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd933ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 -- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 msgr2=0x55fcd933ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd933ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 -- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 msgr2=0x55fcd936f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd936f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 -- 172.21.15.129:0/1398958906 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd9271520 con 0x55fcd8f763b0 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd933ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009a70 tx=0x7f3f2003a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65197c71440fca8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f25c00640 1 -- 172.21.15.129:0/1398958906 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20044040 con 0x55fcd8f763b0 2024-09-17T04:07:00.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f25c00640 1 -- 172.21.15.129:0/1398958906 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f2004d070 con 0x55fcd8f763b0 2024-09-17T04:07:00.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.166+0000 7f3f25c00640 1 -- 172.21.15.129:0/1398958906 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20038040 con 0x55fcd8f763b0 2024-09-17T04:07:00.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1398958906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 msgr2=0x55fcd936feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009a70 tx=0x7f3f2003a000 comp rx=0 tx=0).stop 2024-09-17T04:07:00.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1398958906 shutdown_connections 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd933ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd936feb0 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1398958906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd936f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1398958906 >> 172.21.15.129:0/1398958906 conn(0x55fcd8f933c0 msgr2=0x55fcd9345970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1398958906 shutdown_connections 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.167+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1398958906 wait complete. 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.168+0000 7f3f42ec1740 1 -- start start 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.168+0000 7f3f42ec1740 1 Processor -- start 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f763b0 0x55fcd9367b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 0x55fcd9399480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fcd924f330 con 0x55fcd8f7e590 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fcd92ac810 con 0x55fcd8f763b0 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55fcd9019620 con 0x55fcd8f9f270 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f42ec1740 10 client.0 _ll_register_callbacks cb 0x55fcd9254420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52324/0 (socket says 172.21.15.129:52324) 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 -- 172.21.15.129:0/2373599294 learned_addr learned my addr 172.21.15.129:0/2373599294 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f39400640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f763b0 0x55fcd9367b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f39e00640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 0x55fcd9399480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 -- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 msgr2=0x55fcd9399480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 0x55fcd9399480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 -- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f763b0 msgr2=0x55fcd9367b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f763b0 0x55fcd9367b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f38a00640 1 -- 172.21.15.129:0/2373599294 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd900a850 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.170+0000 7f3f39e00640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 0x55fcd9399480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f38a00640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009c40 tx=0x7f3f2002b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0da6ad423e0aa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f1a000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20044050 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f3f20053660 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f2004d070 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) ==== 24+0+0 (secure 0 0 0) 0x7f3f20060050 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f42ec1740 10 client.11922 fetch_fsmap learned FSMap version 235 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=2+}) -- 0x55fcd90816b0 con 0x55fcd8f7e590 2024-09-17T04:07:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.171+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f2004d070 con 0x55fcd8f7e590 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 235) ==== 3502+0+0 (secure 0 0 0) 0x7f3f20063020 con 0x55fcd8f7e590 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 10 client.11922 fetch_fsmap finished waiting for FSMap version 235 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 10 client.11922 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 20 client.11922 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd8f5de40 0x55fcd93827f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 4 client.11922 mds_command: new command op to 11778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 --> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd92bb520 con 0x55fcd8f5de40 2024-09-17T04:07:00.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f39400640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd8f5de40 0x55fcd93827f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f39400640 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd8f5de40 0x55fcd93827f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8be1d8d50171028 server_cookie=9c978d89513eb26a in_seq=0 out_seq=0 2024-09-17T04:07:00.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.172+0000 7f3f1a000640 10 client.11922 ms_handle_connect on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.173+0000 7f3f1a000640 1 -- 172.21.15.129:0/2373599294 <== mds.0 v2:172.21.15.129:6833/968815434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55fcd92bb520 con 0x55fcd8f5de40 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.173+0000 7f3f1a000640 10 client.11922 handle_command_reply: tid=0 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.173+0000 7f3f42ec1740 1 client.11922 shutdown 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 msgr2=0x55fcd9384560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f3f20009c40 tx=0x7f3f2002b3e0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 shutdown_connections 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd8f5de40 0x55fcd93827f0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f9f270 0x55fcd9399480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f7e590 0x55fcd9384560 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f1a000640 0 client.11922 ms_handle_reset on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/2373599294 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f763b0 0x55fcd9367b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 >> 172.21.15.129:0/2373599294 conn(0x55fcd8f933c0 msgr2=0x55fcd9380e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 shutdown_connections 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 1 -- 172.21.15.129:0/2373599294 wait complete. 2024-09-17T04:07:00.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.174+0000 7f3f42ec1740 20 client.11922 trim_cache size 0 max 16384 2024-09-17T04:07:00.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.193+0000 7f3f42ec1740 1 Processor -- start 2024-09-17T04:07:00.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.193+0000 7f3f42ec1740 1 -- start start 2024-09-17T04:07:00.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9369cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9398cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3f34201780 con 0x55fcd8f763b0 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3f340b1990 con 0x55fcd8f9f270 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3f34201140 con 0x55fcd8f7e590 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52334/0 (socket says 172.21.15.129:52334) 2024-09-17T04:07:00.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.194+0000 7f3f39400640 1 -- 172.21.15.129:0/1704163048 learned_addr learned my addr 172.21.15.129:0/1704163048 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9398cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9369cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 -- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 msgr2=0x55fcd9369cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9369cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 -- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 msgr2=0x55fcd9398cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9398cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 -- 172.21.15.129:0/1704163048 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd939d700 con 0x55fcd8f763b0 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9398cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f39400640 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3f28005780 tx=0x7f3f28067040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38c6882a013b22c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f25200640 1 -- 172.21.15.129:0/1704163048 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28099c60 con 0x55fcd8f763b0 2024-09-17T04:07:00.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f25200640 1 -- 172.21.15.129:0/1704163048 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f28089070 con 0x55fcd8f763b0 2024-09-17T04:07:00.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.195+0000 7f3f25200640 1 -- 172.21.15.129:0/1704163048 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28085b60 con 0x55fcd8f763b0 2024-09-17T04:07:00.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1704163048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 msgr2=0x55fcd93684c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f3f28005780 tx=0x7f3f28067040 comp rx=0 tx=0).stop 2024-09-17T04:07:00.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1704163048 shutdown_connections 2024-09-17T04:07:00.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9398cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9369cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1704163048 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd93684c0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1704163048 >> 172.21.15.129:0/1704163048 conn(0x55fcd8f933c0 msgr2=0x55fcd938afb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1704163048 shutdown_connections 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.196+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1704163048 wait complete. 2024-09-17T04:07:00.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.197+0000 7f3f42ec1740 1 -- start start 2024-09-17T04:07:00.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.197+0000 7f3f42ec1740 1 Processor -- start 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9368320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9368860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55fcd939d250 con 0x55fcd8f763b0 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55fcd924f330 con 0x55fcd8f9f270 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55fcd939aa40 con 0x55fcd8f7e590 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f42ec1740 10 client.0 _ll_register_callbacks cb 0x7f3f1c0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52342/0 (socket says 172.21.15.129:52342) 2024-09-17T04:07:00.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 -- 172.21.15.129:0/1385666691 learned_addr learned my addr 172.21.15.129:0/1385666691 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9368320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9368860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 -- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 msgr2=0x55fcd9368320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9368320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 -- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 msgr2=0x55fcd9368860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9368860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39400640 1 -- 172.21.15.129:0/1385666691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd9346650 con 0x55fcd8f763b0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9368320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.199+0000 7f3f39e00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9368860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f39400640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f3f28060710 tx=0x7f3f28063590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9353ec2108198734 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28085d60 con 0x55fcd8f763b0 2024-09-17T04:07:00.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f3f28062700 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f28089070 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) ==== 24+0+0 (secure 0 0 0) 0x7f3f28062900 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 10 client.11928 fetch_fsmap learned FSMap version 235 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=2+}) -- 0x55fcd936fd90 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f28089070 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 235) ==== 3502+0+0 (secure 0 0 0) 0x7f3f2805edf0 con 0x55fcd8f763b0 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 10 client.11928 fetch_fsmap finished waiting for FSMap version 235 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 10 client.11928 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 20 client.11928 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd940e310 0x55fcd938a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 4 client.11928 mds_command: new command op to 11778 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:07:00.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.200+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 --> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] -- command(tid 0: {"prefix": "status"}) -- 0x55fcd92b6fd0 con 0x55fcd940e310 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd940e310 0x55fcd938a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f1a000640 10 client.11928 ms_handle_connect on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f38a00640 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd940e310 0x55fcd938a3c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde1ace45eb8be1b server_cookie=ba90024b9ed05a87 in_seq=0 out_seq=0 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f1a000640 1 -- 172.21.15.129:0/1385666691 <== mds.0 v2:172.21.15.129:6833/968815434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55fcd92b6fd0 con 0x55fcd940e310 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f1a000640 10 client.11928 handle_command_reply: tid=0 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f42ec1740 1 client.11928 shutdown 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 msgr2=0x55fcd9404490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.201+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f3f28060710 tx=0x7f3f28063590 comp rx=0 tx=0).stop 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 shutdown_connections 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55fcd940e310 0x55fcd938a3c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55fcd8f9f270 0x55fcd9368860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55fcd8f7e590 0x55fcd9368320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 --2- 172.21.15.129:0/1385666691 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55fcd8f763b0 0x55fcd9404490 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f1a000640 0 client.11928 ms_handle_reset on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 >> 172.21.15.129:0/1385666691 conn(0x55fcd8f933c0 msgr2=0x55fcd936af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 shutdown_connections 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 1 -- 172.21.15.129:0/1385666691 wait complete. 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f42ec1740 20 client.11928 trim_cache size 0 max 16384 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:07:00.213 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 11778, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.579202231, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 235, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 247, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 247, 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.638068355000001 2024-09-17T04:07:00.214 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:00.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 msgr2=0x7f3f1407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 0x7f3f1407b0a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f3f20035000 tx=0x7f3f200046c0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 msgr2=0x7f3f340c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.202+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f3f28011ae0 tx=0x7f3f28009e40 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 shutdown_connections 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3f14078b90 0x7f3f1407b0a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f340b2140 0x7f3f3415efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3f341f9790 0x7f3f3415b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 --2- 172.21.15.129:0/2639669614 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3f1c003fd0 0x7f3f340c68c0 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 >> 172.21.15.129:0/2639669614 conn(0x7f3f3406d410 msgr2=0x7f3f340cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 shutdown_connections 2024-09-17T04:07:00.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.203+0000 7f3f3be00640 1 -- 172.21.15.129:0/2639669614 wait complete. 2024-09-17T04:07:00.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:07:00.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 -- 172.21.15.129:0/3942164723 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f68099570 msgr2=0x7f5f6806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3942164723 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f68099570 0x7f5f6806eed0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009860 tx=0x7f5f5802a540 comp rx=0 tx=0).stop 2024-09-17T04:07:00.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 -- 172.21.15.129:0/3942164723 shutdown_connections 2024-09-17T04:07:00.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3942164723 >> v2:172.21.15.181:3300/0 conn(0x7f5f680ccc80 0x7f5f680cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3942164723 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f68099570 0x7f5f6806eed0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 -- 172.21.15.129:0/3942164723 >> 172.21.15.129:0/3942164723 conn(0x7f5f6806d410 msgr2=0x7f5f680b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 -- 172.21.15.129:0/3942164723 shutdown_connections 2024-09-17T04:07:00.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.448+0000 7f5f6e800640 1 -- 172.21.15.129:0/3942164723 wait complete. 2024-09-17T04:07:00.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.450+0000 7f5f6e800640 1 Processor -- start 2024-09-17T04:07:00.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.450+0000 7f5f6e800640 1 -- start start 2024-09-17T04:07:00.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f68099570 0x7f5f6815e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f680ccc80 0x7f5f68154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5f68160ba0 con 0x7f5f680cf1d0 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5f68160a20 con 0x7f5f68099570 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f6e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5f68160d20 con 0x7f5f680ccc80 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52350/0 (socket says 172.21.15.129:52350) 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f66a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f680ccc80 0x7f5f68154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67e00640 1 -- 172.21.15.129:0/3534516771 learned_addr learned my addr 172.21.15.129:0/3534516771 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67400640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f68099570 0x7f5f6815e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67e00640 1 -- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f680ccc80 msgr2=0x7f5f68154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.452+0000 7f5f67e00640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f680ccc80 0x7f5f68154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f67e00640 1 -- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f68099570 msgr2=0x7f5f6815e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f67e00640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f68099570 0x7f5f6815e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f67e00640 1 -- 172.21.15.129:0/3534516771 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f681f5c00 con 0x7f5f680cf1d0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f67e00640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000ca70 tx=0x7f5f6000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f60016020 con 0x7f5f680cf1d0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5f6000acc0 con 0x7f5f680cf1d0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f60012680 con 0x7f5f680cf1d0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6815ea10 con 0x7f5f680cf1d0 2024-09-17T04:07:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.453+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6815efd0 con 0x7f5f680cf1d0 2024-09-17T04:07:00.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.454+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f68161260 con 0x7f5f680cf1d0 2024-09-17T04:07:00.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.456+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5f60019020 con 0x7f5f680cf1d0 2024-09-17T04:07:00.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.457+0000 7f5f5f400640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 0x7f5f4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:00.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.457+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f5f6009b230 con 0x7f5f680cf1d0 2024-09-17T04:07:00.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.457+0000 7f5f67400640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 0x7f5f4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:00.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.458+0000 7f5f67400640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 0x7f5f4807b0a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f5f58034000 tx=0x7f5f58004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:00.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.462+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5f60020020 con 0x7f5f680cf1d0 2024-09-17T04:07:00.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.681+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5f6806a140 con 0x7f5f680cf1d0 2024-09-17T04:07:00.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.682+0000 7f5f5f400640 1 -- 172.21.15.129:0/3534516771 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v235) ==== 53+0+73 (secure 0 0 0) 0x7f5f680cd5d0 con 0x7f5f680cf1d0 2024-09-17T04:07:00.682 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:07:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 msgr2=0x7f5f4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 0x7f5f4807b0a0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f5f58034000 tx=0x7f5f58004730 comp rx=0 tx=0).stop 2024-09-17T04:07:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 msgr2=0x7f5f68155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000ca70 tx=0x7f5f6000cf30 comp rx=0 tx=0).stop 2024-09-17T04:07:00.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 shutdown_connections 2024-09-17T04:07:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5f48078b90 0x7f5f4807b0a0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5f680cf1d0 0x7f5f68155190 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5f680ccc80 0x7f5f68154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 --2- 172.21.15.129:0/3534516771 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5f68099570 0x7f5f6815e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:00.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 >> 172.21.15.129:0/3534516771 conn(0x7f5f6806d410 msgr2=0x7f5f680b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:00.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 shutdown_connections 2024-09-17T04:07:00.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00.684+0000 7f5f6e800640 1 -- 172.21.15.129:0/3534516771 wait complete. 2024-09-17T04:07:00.706 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:07:00.706 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:07:00.706 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:07:00.706 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:07:00.707 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:00.707 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:07:00.707 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:07:00.707 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:07:00.707 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:00.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:00.707 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.745 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:00.746 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41671sec preferred_lft 41671sec 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:07:00.747 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:07:00.748 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.749 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:00.749 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:07:00.749 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:07:00.749 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:00.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:00.750 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:07:00.838 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:07:00.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:00.841 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:07:00.877 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:07:00.877 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:07:00.878 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:07:00.878 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:07:00.878 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:07:00.878 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:07:00.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:07:01.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:01.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:01.073 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:07:01.108 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:07:01.108 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:07:01.109 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:01.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:01.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:07:01.205 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:01.205 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:01.240 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:01.240 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:01.241 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:01.315 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:07:01.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:01.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:01.511 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:01.511 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:01.511 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:01.531 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:01.560 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:01.560 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:01.560 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:01.560 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:01.561 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:01.561 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:07:01.561 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:07:01.561 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:07:01.562 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:07:01.562 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:07:01.562 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:07:01.562 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:07:01.562 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:07:01.565 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:07:01.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:01.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:01.622 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:07:01.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:07:01.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:01.718 INFO:teuthology.orchestra.run.smithi129.stdout:client11937 2024-09-17T04:07:01.719 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:01.719 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c ' 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p a b 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> for i in `seq 1 50`; do 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> touch a/"$i" 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> ln a/"$i" b/"$i" 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> done 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> sync -f . 2024-09-17T04:07:01.755 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:07:01.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:07:02.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:02.277 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:02.277 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:02.313 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:02.313 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:02.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:07:02.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3111714536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc0cf2b0 msgr2=0x7f0acc0cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:02.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3111714536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc0cf2b0 0x7f0acc0cf6c0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009860 tx=0x7f0ab402a560 comp rx=0 tx=0).stop 2024-09-17T04:07:02.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3111714536 shutdown_connections 2024-09-17T04:07:02.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3111714536 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc0cf2b0 0x7f0acc0cf6c0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3111714536 >> 172.21.15.129:0/3111714536 conn(0x7f0acc06d410 msgr2=0x7f0acc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:02.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3111714536 shutdown_connections 2024-09-17T04:07:02.652 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.651+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3111714536 wait complete. 2024-09-17T04:07:02.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.653+0000 7f0ad0e00640 1 Processor -- start 2024-09-17T04:07:02.653 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.653+0000 7f0ad0e00640 1 -- start start 2024-09-17T04:07:02.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.654+0000 7f0ad0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:02.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.654+0000 7f0ad0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0acc0ccc80 0x7f0acc1f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:02.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ad0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0acc0cf2b0 0x7f0acc15e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ad0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc160cb0 con 0x7f0acc099570 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ad0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc160b30 con 0x7f0acc0ccc80 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ad0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0acc160e30 con 0x7f0acc0cf2b0 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52374/0 (socket says 172.21.15.129:52374) 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 -- 172.21.15.129:0/3239276676 learned_addr learned my addr 172.21.15.129:0/3239276676 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac1400640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0acc0ccc80 0x7f0acc1f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0aca800640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0acc0cf2b0 0x7f0acc15e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 -- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0acc0cf2b0 msgr2=0x7f0acc15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0acc0cf2b0 0x7f0acc15e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 -- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0acc0ccc80 msgr2=0x7f0acc1f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:02.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0acc0ccc80 0x7f0acc1f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.655+0000 7f0ac9e00640 1 -- 172.21.15.129:0/3239276676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc1e6710 con 0x7f0acc099570 2024-09-17T04:07:02.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ac9e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00c370 tx=0x7f0abc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc017070 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0abc0035f0 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc003a40 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc09a170 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.656+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc09a730 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.657+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc0af6f0 con 0x7f0acc099570 2024-09-17T04:07:02.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.658+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0abc01a070 con 0x7f0acc099570 2024-09-17T04:07:02.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.659+0000 7f0ac2a00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 0x7f0aa807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:02.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.659+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f0abc099ea0 con 0x7f0acc099570 2024-09-17T04:07:02.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.660+0000 7f0ac1400640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 0x7f0aa807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:02.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.660+0000 7f0ac1400640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 0x7f0aa807b0d0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4034000 tx=0x7f0ab4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:02.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.664+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0abc061c80 con 0x7f0acc099570 2024-09-17T04:07:02.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.861+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f0acc0cf6c0 con 0x7f0acc099570 2024-09-17T04:07:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.862+0000 7f0ac2a00640 1 -- 172.21.15.129:0/3239276676 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v247) ==== 80+0+0 (secure 0 0 0) 0x7f0abc0666b0 con 0x7f0acc099570 2024-09-17T04:07:02.862 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 msgr2=0x7f0aa807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 0x7f0aa807b0d0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4034000 tx=0x7f0ab4004730 comp rx=0 tx=0).stop 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 msgr2=0x7f0acc09dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00c370 tx=0x7f0abc00c830 comp rx=0 tx=0).stop 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 shutdown_connections 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0aa8078bc0 0x7f0aa807b0d0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0acc0cf2b0 0x7f0acc15e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0acc0ccc80 0x7f0acc1f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.864+0000 7f0ad0e00640 1 --2- 172.21.15.129:0/3239276676 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0acc099570 0x7f0acc09dd00 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.865+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 >> 172.21.15.129:0/3239276676 conn(0x7f0acc06d410 msgr2=0x7f0acc0ada60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.865+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 shutdown_connections 2024-09-17T04:07:02.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:02.865+0000 7f0ad0e00640 1 -- 172.21.15.129:0/3239276676 wait complete. 2024-09-17T04:07:02.887 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:07:02.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:02.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:03.011 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:07:03.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:03.011 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:07:03.048 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:03.048 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:03.123 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:07:03.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.386+0000 7f8d16a00640 1 -- 172.21.15.129:0/242515653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d10099570 msgr2=0x7f8d1006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.386+0000 7f8d16a00640 1 --2- 172.21.15.129:0/242515653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d10099570 0x7f8d1006eed0 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f8d100b3450 tx=0x7f8d0002a4a0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.386+0000 7f8d16a00640 1 -- 172.21.15.129:0/242515653 shutdown_connections 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.386+0000 7f8d16a00640 1 --2- 172.21.15.129:0/242515653 >> v2:172.21.15.181:3300/0 conn(0x7f8d100cf9c0 0x7f8d100ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.387+0000 7f8d16a00640 1 --2- 172.21.15.129:0/242515653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d10099570 0x7f8d1006eed0 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.387+0000 7f8d16a00640 1 -- 172.21.15.129:0/242515653 >> 172.21.15.129:0/242515653 conn(0x7f8d1006d410 msgr2=0x7f8d100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.387+0000 7f8d16a00640 1 -- 172.21.15.129:0/242515653 shutdown_connections 2024-09-17T04:07:03.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.387+0000 7f8d16a00640 1 -- 172.21.15.129:0/242515653 wait complete. 2024-09-17T04:07:03.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.389+0000 7f8d16a00640 1 Processor -- start 2024-09-17T04:07:03.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.389+0000 7f8d16a00640 1 -- start start 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 0x7f8d100cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d100cf9c0 0x7f8d100cbc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10160ba0 con 0x7f8d1006f410 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8d10160a20 con 0x7f8d10099570 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d16a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8d10160d20 con 0x7f8d100cf9c0 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.390+0000 7f8d0fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:52388/0 (socket says 172.21.15.129:52388) 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 -- 172.21.15.129:0/3315563080 learned_addr learned my addr 172.21.15.129:0/3315563080 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0f400640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 0x7f8d100cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d14a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d100cf9c0 0x7f8d100cbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 -- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d100cf9c0 msgr2=0x7f8d100cbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d100cf9c0 0x7f8d100cbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 -- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 msgr2=0x7f8d100cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 0x7f8d100cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 -- 172.21.15.129:0/3315563080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d100ca960 con 0x7f8d1006f410 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0f400640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 0x7f8d100cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d0fe00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009780 tx=0x7f8d00033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00031d60 con 0x7f8d1006f410 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d00033e10 con 0x7f8d1006f410 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.391+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d00005260 con 0x7f8d1006f410 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.392+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d100cac00 con 0x7f8d1006f410 2024-09-17T04:07:03.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.392+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d100cb1c0 con 0x7f8d1006f410 2024-09-17T04:07:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.393+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d100b1fa0 con 0x7f8d1006f410 2024-09-17T04:07:03.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.394+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8d00005400 con 0x7f8d1006f410 2024-09-17T04:07:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.395+0000 7f8d07e00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 0x7f8cf007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.395+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f8d000bbf60 con 0x7f8d1006f410 2024-09-17T04:07:03.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.395+0000 7f8d0f400640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 0x7f8cf007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:03.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.396+0000 7f8d0f400640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 0x7f8cf007b0a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8d100ae7a0 tx=0x7f8cfc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:03.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.399+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8d100b1fa0 con 0x7f8d1006f410 2024-09-17T04:07:03.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.627+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8d10161240 con 0x7f8d1006f410 2024-09-17T04:07:03.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.628+0000 7f8d07e00640 1 -- 172.21.15.129:0/3315563080 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v235) ==== 53+0+73 (secure 0 0 0) 0x7f8d0003f030 con 0x7f8d1006f410 2024-09-17T04:07:03.628 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 msgr2=0x7f8cf007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 0x7f8cf007b0a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8d100ae7a0 tx=0x7f8cfc009040 comp rx=0 tx=0).stop 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 msgr2=0x7f8d101f8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009780 tx=0x7f8d00033c20 comp rx=0 tx=0).stop 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 shutdown_connections 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8cf0078b90 0x7f8cf007b0a0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8d100cf9c0 0x7f8d100cbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d10099570 0x7f8d100cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 --2- 172.21.15.129:0/3315563080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8d1006f410 0x7f8d101f8a90 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.630+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 >> 172.21.15.129:0/3315563080 conn(0x7f8d1006d410 msgr2=0x7f8d1006f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.631+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 shutdown_connections 2024-09-17T04:07:03.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03.631+0000 7f8d16a00640 1 -- 172.21.15.129:0/3315563080 wait complete. 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:07:03.657 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:03.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:03.658 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:07:03.694 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:07:03.695 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:07:03.695 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:07:03.695 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:07:03.696 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41668sec preferred_lft 41668sec 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:07:03.697 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:03.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:03.699 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:07:03.789 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:07:03.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:03.792 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:07:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:07:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:07:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:07:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:07:03.828 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:07:03.829 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:07:03.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:07:04.016 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:04.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:04.024 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:07:04.060 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:07:04.060 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:07:04.060 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:04.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:04.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:07:04.156 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:04.156 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:04.191 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:04.191 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:04.191 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:04.266 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:07:04.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:04.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:04.461 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:04.461 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:04.462 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:04.482 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:04.482 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:04.482 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:04.482 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:04.483 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:04.513 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:07:04.514 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:07:04.514 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:07:04.515 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:07:04.515 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:07:04.515 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:07:04.515 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:07:04.515 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:07:04.518 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:07:04.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:04.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:04.573 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:07:04.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:07:04.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:04.665 INFO:teuthology.orchestra.run.smithi129.stdout:client10996 2024-09-17T04:07:04.665 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:04.665 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:04.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.18:0 flush journal 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 -- 172.21.15.129:0/4228158247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 msgr2=0x7f584800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 --2- 172.21.15.129:0/4228158247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f584800b9c0 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f585c009ed0 tx=0x7f5848005830 comp rx=0 tx=0).stop 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 -- 172.21.15.129:0/4228158247 shutdown_connections 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 --2- 172.21.15.129:0/4228158247 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f584800b9c0 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 -- 172.21.15.129:0/4228158247 >> 172.21.15.129:0/4228158247 conn(0x7f586406cfc0 msgr2=0x7f586406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 -- 172.21.15.129:0/4228158247 shutdown_connections 2024-09-17T04:07:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.963+0000 7f5863200640 1 -- 172.21.15.129:0/4228158247 wait complete. 2024-09-17T04:07:04.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.966+0000 7f5863200640 1 Processor -- start 2024-09-17T04:07:04.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.966+0000 7f5863200640 1 -- start start 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5864072d30 0x7f586414f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 0x7f586415f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f58640cd760 con 0x7f58480051c0 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f58640cd5e0 con 0x7f5864072d30 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.967+0000 7f5863200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f58640cd8e0 con 0x7f586415f080 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48662/0 (socket says 172.21.15.129:48662) 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 -- 172.21.15.129:0/390717588 learned_addr learned my addr 172.21.15.129:0/390717588 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861400640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5864072d30 0x7f586414f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5862800640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 0x7f586415f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 -- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 msgr2=0x7f586415f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:04.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 0x7f586415f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 -- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5864072d30 msgr2=0x7f586414f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5864072d30 0x7f586414f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 -- 172.21.15.129:0/390717588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f586415f9d0 con 0x7f58480051c0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5862800640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 0x7f586415f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.968+0000 7f5861e00640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f585400e590 tx=0x7f585400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.969+0000 7f585a000640 1 -- 172.21.15.129:0/390717588 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854015020 con 0x7f58480051c0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.969+0000 7f585a000640 1 -- 172.21.15.129:0/390717588 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5854010980 con 0x7f58480051c0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.969+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58640cbe30 con 0x7f58480051c0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.969+0000 7f585a000640 1 -- 172.21.15.129:0/390717588 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854010dd0 con 0x7f58480051c0 2024-09-17T04:07:04.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.969+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58640c5460 con 0x7f58480051c0 2024-09-17T04:07:04.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.971+0000 7f585a000640 1 -- 172.21.15.129:0/390717588 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5854018020 con 0x7f58480051c0 2024-09-17T04:07:04.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.972+0000 7f585a000640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 0x7f584407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.972+0000 7f585a000640 1 -- 172.21.15.129:0/390717588 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f585409b3e0 con 0x7f58480051c0 2024-09-17T04:07:04.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.972+0000 7f5861400640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 0x7f584407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.973+0000 7f586fc49740 1 Processor -- start 2024-09-17T04:07:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.973+0000 7f586fc49740 1 -- start start 2024-09-17T04:07:04.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.973+0000 7f5861400640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 0x7f584407b0a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f586415ba90 tx=0x7f5850006060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:04.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ef6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ed9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cd45e0f1c0 con 0x55cd45ae76b0 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cd45e22870 con 0x55cd45b08390 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cd45e11ca0 con 0x55cd45adf4d0 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48676/0 (socket says 172.21.15.129:48676) 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 -- 172.21.15.129:0/1731032045 learned_addr learned my addr 172.21.15.129:0/1731032045 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5861e00640 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ef6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5861400640 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ed9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 -- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 msgr2=0x55cd45ed9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ed9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 -- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 msgr2=0x55cd45ef6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:04.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ef6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.975+0000 7f5862800640 1 -- 172.21.15.129:0/1731032045 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd45bea7d0 con 0x55cd45ae76b0 2024-09-17T04:07:04.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.976+0000 7f5862800640 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f585c02b8c0 tx=0x7f585c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=975def7dd2b71b3b server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.976+0000 7f584e600640 1 -- 172.21.15.129:0/1731032045 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c003970 con 0x55cd45ae76b0 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.976+0000 7f584e600640 1 -- 172.21.15.129:0/1731032045 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585c003b10 con 0x55cd45ae76b0 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.976+0000 7f584e600640 1 -- 172.21.15.129:0/1731032045 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c004030 con 0x55cd45ae76b0 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 -- 172.21.15.129:0/1731032045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 msgr2=0x55cd45ed9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f585c02b8c0 tx=0x7f585c004ab0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 -- 172.21.15.129:0/1731032045 shutdown_connections 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ed9940 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ed9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.978 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 --2- 172.21.15.129:0/1731032045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ef6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 -- 172.21.15.129:0/1731032045 >> 172.21.15.129:0/1731032045 conn(0x55cd45afc4e0 msgr2=0x55cd45eaf620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 -- 172.21.15.129:0/1731032045 shutdown_connections 2024-09-17T04:07:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.977+0000 7f586fc49740 1 -- 172.21.15.129:0/1731032045 wait complete. 2024-09-17T04:07:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.978+0000 7f586fc49740 1 -- start start 2024-09-17T04:07:04.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.978+0000 7f586fc49740 1 Processor -- start 2024-09-17T04:07:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ef0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45eaebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cd45db8410 con 0x55cd45ae76b0 2024-09-17T04:07:04.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55cd45e15ab0 con 0x55cd45b08390 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cd45b82740 con 0x55cd45adf4d0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f586fc49740 10 client.0 _ll_register_callbacks cb 0x55cd45d95360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48684/0 (socket says 172.21.15.129:48684) 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 -- 172.21.15.129:0/341268279 learned_addr learned my addr 172.21.15.129:0/341268279 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861e00640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ef0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5862800640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45eaebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 -- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 msgr2=0x55cd45ef0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ef0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 -- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 msgr2=0x55cd45eaebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45eaebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.980+0000 7f5861400640 1 -- 172.21.15.129:0/341268279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd45b73970 con 0x55cd45ae76b0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5861400640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f5850001e50 tx=0x7f58500012f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de83ff79dde6bc00 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5842a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:04.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585001f810 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f585001f9b0 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f585c003b10 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) ==== 24+0+0 (secure 0 0 0) 0x7f5850030050 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f586fc49740 10 client.11967 fetch_fsmap learned FSMap version 235 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=2+}) -- 0x55cd45b516a0 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.981+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585c003b10 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 235) ==== 3502+0+0 (secure 0 0 0) 0x7f5850033020 con 0x55cd45ae76b0 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 10 client.11967 fetch_fsmap finished waiting for FSMap version 235 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 10 client.11967 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.c 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 20 client.11967 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45ac6f60 0x55cd45ef1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 4 client.11967 mds_command: new command op to 11778 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:04.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 --> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd45e247c0 con 0x55cd45ac6f60 2024-09-17T04:07:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f5861e00640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45ac6f60 0x55cd45ef1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f5842a00640 10 client.11967 ms_handle_connect on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:04.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.982+0000 7f5861e00640 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45ac6f60 0x55cd45ef1ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d187a18cf907280 server_cookie=19c76d5d809cd19f in_seq=0 out_seq=0 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f5842a00640 1 -- 172.21.15.129:0/341268279 <== mds.0 v2:172.21.15.129:6833/968815434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55cd45e247c0 con 0x55cd45ac6f60 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f5842a00640 10 client.11967 handle_command_reply: tid=0 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 client.11967 shutdown 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 msgr2=0x55cd45ec7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7f5850001e50 tx=0x7f58500012f0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 shutdown_connections 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45ac6f60 0x55cd45ef1ce0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45eaebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f5842a00640 0 client.11967 ms_handle_reset on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45ae76b0 0x55cd45ec7a20 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 --2- 172.21.15.129:0/341268279 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45adf4d0 0x55cd45ef0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.983+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 >> 172.21.15.129:0/341268279 conn(0x55cd45afc4e0 msgr2=0x55cd45ef7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.984+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 shutdown_connections 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.984+0000 7f586fc49740 1 -- 172.21.15.129:0/341268279 wait complete. 2024-09-17T04:07:04.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.984+0000 7f586fc49740 20 client.11967 trim_cache size 0 max 16384 2024-09-17T04:07:04.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.999+0000 7f586fc49740 1 Processor -- start 2024-09-17T04:07:04.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:04.999+0000 7f586fc49740 1 -- start start 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 0x55cd45ec8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ec8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5864201db0 con 0x55cd45adf4d0 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5864200dd0 con 0x55cd45b08390 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f58641519d0 con 0x55cd45ae76b0 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48688/0 (socket says 172.21.15.129:48688) 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 -- 172.21.15.129:0/2856722309 learned_addr learned my addr 172.21.15.129:0/2856722309 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861400640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 0x55cd45ec8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5862800640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ec8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 -- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 msgr2=0x55cd45ec8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:05.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 0x55cd45ec8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 -- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 msgr2=0x55cd45ec8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ec8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861e00640 1 -- 172.21.15.129:0/2856722309 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd45eb0e60 con 0x55cd45adf4d0 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.001+0000 7f5861400640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 0x55cd45ec8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.002+0000 7f5861e00640 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f5854099810 tx=0x7f5854088040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acab9445367ae940 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.002+0000 7f584dc00640 1 -- 172.21.15.129:0/2856722309 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854067b60 con 0x55cd45adf4d0 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.002+0000 7f584dc00640 1 -- 172.21.15.129:0/2856722309 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585404e030 con 0x55cd45adf4d0 2024-09-17T04:07:05.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.002+0000 7f584dc00640 1 -- 172.21.15.129:0/2856722309 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585405f3a0 con 0x55cd45adf4d0 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 -- 172.21.15.129:0/2856722309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 msgr2=0x55cd45f17be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f5854099810 tx=0x7f5854088040 comp rx=0 tx=0).stop 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 -- 172.21.15.129:0/2856722309 shutdown_connections 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45b08390 0x55cd45ec8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45ae76b0 0x55cd45ec8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 --2- 172.21.15.129:0/2856722309 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f17be0 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 -- 172.21.15.129:0/2856722309 >> 172.21.15.129:0/2856722309 conn(0x55cd45afc4e0 msgr2=0x55cd45ef2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 -- 172.21.15.129:0/2856722309 shutdown_connections 2024-09-17T04:07:05.004 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.003+0000 7f586fc49740 1 -- 172.21.15.129:0/2856722309 wait complete. 2024-09-17T04:07:05.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.004+0000 7f586fc49740 1 -- start start 2024-09-17T04:07:05.005 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.004+0000 7f586fc49740 1 Processor -- start 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45ae76b0 0x55cd45f6d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 0x55cd45ec7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55cd45eae090 con 0x55cd45adf4d0 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5864200780 con 0x55cd45ae76b0 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.005+0000 7f586fc49740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55cd45e11e20 con 0x55cd45b08390 2024-09-17T04:07:05.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f586fc49740 10 client.0 _ll_register_callbacks cb 0x7f5850001ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48694/0 (socket says 172.21.15.129:48694) 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 -- 172.21.15.129:0/3487511547 learned_addr learned my addr 172.21.15.129:0/3487511547 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861400640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45ae76b0 0x55cd45f6d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5862800640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 0x55cd45ec7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 -- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 msgr2=0x55cd45ec7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 0x55cd45ec7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 -- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45ae76b0 msgr2=0x55cd45f6d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45ae76b0 0x55cd45f6d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 -- 172.21.15.129:0/3487511547 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd45ec72c0 con 0x55cd45adf4d0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5862800640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 0x55cd45ec7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5842a00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:05.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.006+0000 7f5861e00640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5854063070 tx=0x7f5854099b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af8f1806f8b506f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5854067b60 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f5854062390 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f585404e030 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=235) ==== 24+0+0 (secure 0 0 0) 0x7f5854062590 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 10 client.11973 fetch_fsmap learned FSMap version 235 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=235,monmap=2+}) -- 0x55cd45f17780 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f585404e030 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 235) ==== 3502+0+0 (secure 0 0 0) 0x7f585405ea80 con 0x55cd45adf4d0 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 10 client.11973 fetch_fsmap finished waiting for FSMap version 235 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 10 client.11973 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.c 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 20 client.11973 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:05.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45f0b9b0 0x55cd45ec8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 4 client.11973 mds_command: new command op to 11778 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-17T04:07:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.007+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 --> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cd45e20270 con 0x55cd45f0b9b0 2024-09-17T04:07:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.008+0000 7f5861400640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45f0b9b0 0x55cd45ec8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.008+0000 7f5861400640 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45f0b9b0 0x55cd45ec8530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44cda1a81d2affbc server_cookie=19336ca21974a854 in_seq=0 out_seq=0 2024-09-17T04:07:05.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.008+0000 7f5842a00640 10 client.11973 ms_handle_connect on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f5842a00640 1 -- 172.21.15.129:0/3487511547 <== mds.0 v2:172.21.15.129:6833/968815434 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cd45e20270 con 0x55cd45f0b9b0 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f5842a00640 10 client.11973 handle_command_reply: tid=0 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 client.11973 shutdown 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 msgr2=0x55cd45f742a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f5854063070 tx=0x7f5854099b90 comp rx=0 tx=0).stop 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 shutdown_connections 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:6833/968815434,v1:172.21.15.129:6835/968815434] conn(0x55cd45f0b9b0 0x55cd45ec8530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55cd45b08390 0x55cd45ec7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55cd45ae76b0 0x55cd45f6d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 --2- 172.21.15.129:0/3487511547 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55cd45adf4d0 0x55cd45f742a0 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f5842a00640 0 client.11973 ms_handle_reset on v2:172.21.15.129:6833/968815434 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 >> 172.21.15.129:0/3487511547 conn(0x55cd45afc4e0 msgr2=0x55cd45eae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 shutdown_connections 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.029+0000 7f586fc49740 1 -- 172.21.15.129:0/3487511547 wait complete. 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f586fc49740 20 client.11973 trim_cache size 0 max 16384 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "message": "", 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0 2024-09-17T04:07:05.031 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:05.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 msgr2=0x7f584407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 0x7f584407b0a0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f586415ba90 tx=0x7f5850006060 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 msgr2=0x7f58641f8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f585400e590 tx=0x7f585400e990 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 shutdown_connections 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5844078b90 0x7f584407b0a0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f586415f080 0x7f586415f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5864072d30 0x7f586414f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 --2- 172.21.15.129:0/390717588 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f58480051c0 0x7f58641f8c90 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.030+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 >> 172.21.15.129:0/390717588 conn(0x7f586406cfc0 msgr2=0x7f58641527c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.031+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 shutdown_connections 2024-09-17T04:07:05.033 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.031+0000 7f5863200640 1 -- 172.21.15.129:0/390717588 wait complete. 2024-09-17T04:07:05.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 18:0 2024-09-17T04:07:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 -- 172.21.15.129:0/3737276406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00cfb60 msgr2=0x7f7bb0160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 --2- 172.21.15.129:0/3737276406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00cfb60 0x7f7bb0160cf0 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0009830 tx=0x7f7ba002a480 comp rx=0 tx=0).stop 2024-09-17T04:07:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 -- 172.21.15.129:0/3737276406 shutdown_connections 2024-09-17T04:07:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 --2- 172.21.15.129:0/3737276406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 0x7f7bb00b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.277 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 --2- 172.21.15.129:0/3737276406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00cfb60 0x7f7bb0160cf0 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 -- 172.21.15.129:0/3737276406 >> 172.21.15.129:0/3737276406 conn(0x7f7bb006cfc0 msgr2=0x7f7bb006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 -- 172.21.15.129:0/3737276406 shutdown_connections 2024-09-17T04:07:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.276+0000 7f7baf200640 1 -- 172.21.15.129:0/3737276406 wait complete. 2024-09-17T04:07:05.278 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.279+0000 7f7baf200640 1 Processor -- start 2024-09-17T04:07:05.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.279+0000 7f7baf200640 1 -- start start 2024-09-17T04:07:05.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 0x7f7bb01f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bb00cfb60 0x7f7bb01f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb015d270 con 0x7f7bb00d02c0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7bb015d0f0 con 0x7f7bb00cfb60 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7baf200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7bb015d3f0 con 0x7f7bb00b46e0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7bae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7bae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48720/0 (socket says 172.21.15.129:48720) 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.280+0000 7f7bae800640 1 -- 172.21.15.129:0/3908982765 learned_addr learned my addr 172.21.15.129:0/3908982765 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bade00640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 0x7f7bb01f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bad400640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bb00cfb60 0x7f7bb01f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 -- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 msgr2=0x7f7bb01f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 0x7f7bb01f6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 -- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bb00cfb60 msgr2=0x7f7bb01f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bb00cfb60 0x7f7bb01f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 -- 172.21.15.129:0/3908982765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bb01f70c0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7bae800640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c760 tx=0x7f7ba800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.281+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba8003da0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.282+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ba800ce10 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.282+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba80052d0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.282+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bb01e7c60 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.282+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bb01e8220 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.283+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bb00b3030 con 0x7f7bb00d02c0 2024-09-17T04:07:05.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.284+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7ba8005470 con 0x7f7bb00d02c0 2024-09-17T04:07:05.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.285+0000 7f7ba6000640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 0x7f7b9007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.285+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 5709+0+0 (secure 0 0 0) 0x7f7ba809a9c0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.285+0000 7f7bade00640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 0x7f7b9007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.286+0000 7f7bade00640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 0x7f7b9007b0a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0034000 tx=0x7f7ba0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.289+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7bb00b3030 con 0x7f7bb00d02c0 2024-09-17T04:07:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.541+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "18:0"} v 0) -- 0x7f7bb015d8a0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.542 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:07:05.542+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:07:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.544+0000 7f7ba6000640 1 -- 172.21.15.129:0/3908982765 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "18:0"}]=0 failed mds gid 11778 v236) ==== 99+0+0 (secure 0 0 0) 0x7f7ba80627a0 con 0x7f7bb00d02c0 2024-09-17T04:07:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 11778 2024-09-17T04:07:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.548+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 msgr2=0x7f7b9007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.548+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 0x7f7b9007b0a0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0034000 tx=0x7f7ba0004730 comp rx=0 tx=0).stop 2024-09-17T04:07:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.548+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 msgr2=0x7f7bb01f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.548+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f7ba800c760 tx=0x7f7ba800cc20 comp rx=0 tx=0).stop 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 shutdown_connections 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b90078b90 0x7f7b9007b0a0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7bb00d02c0 0x7f7bb01f2db0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7bb00cfb60 0x7f7bb01f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 --2- 172.21.15.129:0/3908982765 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7bb00b46e0 0x7f7bb01f6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 >> 172.21.15.129:0/3908982765 conn(0x7f7bb006cfc0 msgr2=0x7f7bb006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 shutdown_connections 2024-09-17T04:07:05.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.549+0000 7f7baf200640 1 -- 172.21.15.129:0/3908982765 wait complete. 2024-09-17T04:07:05.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:05.691 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:07:05.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 -- 172.21.15.129:0/1922377044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84000705a0 msgr2=0x7f84000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 --2- 172.21.15.129:0/1922377044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84000705a0 0x7f84000b4140 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f83f0005fd0 tx=0x7f83f002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 -- 172.21.15.129:0/1922377044 shutdown_connections 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 --2- 172.21.15.129:0/1922377044 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84000cd390 0x7f84001609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 --2- 172.21.15.129:0/1922377044 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84000705a0 0x7f84000b4140 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 -- 172.21.15.129:0/1922377044 >> 172.21.15.129:0/1922377044 conn(0x7f840006d410 msgr2=0x7f84000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 -- 172.21.15.129:0/1922377044 shutdown_connections 2024-09-17T04:07:05.806 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.805+0000 7f8406800640 1 -- 172.21.15.129:0/1922377044 wait complete. 2024-09-17T04:07:05.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.807+0000 7f8406800640 1 Processor -- start 2024-09-17T04:07:05.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.807+0000 7f8406800640 1 -- start start 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84000705a0 0x7f8400073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f840009dc20 0x7f8400074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8400160f00 con 0x7f840006eed0 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f840006f4f0 con 0x7f84000705a0 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f8406800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8400161080 con 0x7f840009dc20 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f83ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f83ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48742/0 (socket says 172.21.15.129:48742) 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.809+0000 7f83ff400640 1 -- 172.21.15.129:0/859631323 learned_addr learned my addr 172.21.15.129:0/859631323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 -- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f840009dc20 msgr2=0x7f8400074440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f840009dc20 0x7f8400074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 -- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84000705a0 msgr2=0x7f8400073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:05.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84000705a0 0x7f8400073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:05.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 -- 172.21.15.129:0/859631323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840015c3b0 con 0x7f840006eed0 2024-09-17T04:07:05.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83ff400640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f83f0034040 tx=0x7f83f00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f003e030 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f83f0003ae0 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84001f5d20 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0035040 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.810+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84001f62e0 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.811+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84000b4140 con 0x7f840006eed0 2024-09-17T04:07:05.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.813+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f83f0041030 con 0x7f840006eed0 2024-09-17T04:07:05.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.814+0000 7f83f7400640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 0x7f83dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:05.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.814+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f83f00bb6e0 con 0x7f840006eed0 2024-09-17T04:07:05.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.814+0000 7f83f7e00640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 0x7f83dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:05.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.815+0000 7f83f7e00640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 0x7f83dc07b0a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f83e8007fd0 tx=0x7f83e80079a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:05.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:05.818+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f83f0083a20 con 0x7f840006eed0 2024-09-17T04:07:06.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.039+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840006a140 con 0x7f840006eed0 2024-09-17T04:07:06.039 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.039+0000 7f83f7400640 1 -- 172.21.15.129:0/859631323 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+4412 (secure 0 0 0) 0x7f83f0083320 con 0x7f840006eed0 2024-09-17T04:07:06.039 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:06.040 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":237,"btime":"2024-09-17T04:07:05:551731+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230}],"filesystems":[{"mdsmap":{"epoch":237,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:05.551713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:replay","state_seq":1,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":18}]} 2024-09-17T04:07:06.040 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 237 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 msgr2=0x7f83dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 0x7f83dc07b0a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f83e8007fd0 tx=0x7f83e80079a0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 msgr2=0x7f84001587b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f83f0034040 tx=0x7f83f00035f0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 shutdown_connections 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f83dc078b90 0x7f83dc07b0a0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f840009dc20 0x7f8400074440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f84000705a0 0x7f8400073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 --2- 172.21.15.129:0/859631323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f840006eed0 0x7f84001587b0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:06.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 >> 172.21.15.129:0/859631323 conn(0x7f840006d410 msgr2=0x7f84000c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:06.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 shutdown_connections 2024-09-17T04:07:06.043 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:06.042+0000 7f8406800640 1 -- 172.21.15.129:0/859631323 wait complete. 2024-09-17T04:07:06.467 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 237, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:06:48.110755+0000', 'modified': '2024-09-17T04:07:05.551713+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 248, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11772': {'gid': 11772, 'name': 'd', 'rank': 0, 'incarnation': 237, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/2055116414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2055116414}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2055116414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:07:06.468 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11772:up:replay 2024-09-17T04:07:07.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:07.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 -- 172.21.15.129:0/3514282680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba24004800 msgr2=0x7fba24004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:07.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 --2- 172.21.15.129:0/3514282680 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba24004800 0x7fba24004c10 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7fba2800c4a0 tx=0x7fba24005830 comp rx=0 tx=0).stop 2024-09-17T04:07:07.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba44a00640 1 -- 172.21.15.129:0/3514282680 reap_dead start 2024-09-17T04:07:07.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 -- 172.21.15.129:0/3514282680 shutdown_connections 2024-09-17T04:07:07.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 -- 172.21.15.129:0/3514282680 >> 172.21.15.129:0/3514282680 conn(0x7fba4008bff0 msgr2=0x7fba4009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:07.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 -- 172.21.15.129:0/3514282680 shutdown_connections 2024-09-17T04:07:07.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.705+0000 7fba45400640 1 -- 172.21.15.129:0/3514282680 wait complete. 2024-09-17T04:07:07.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.707+0000 7fba45400640 1 Processor -- start 2024-09-17T04:07:07.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.707+0000 7fba45400640 1 -- start start 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 0x7fba4015faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 0x7fba400cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fba400d1860 con 0x7fba40159080 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fba400d16e0 con 0x7fba40158c70 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba45400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fba400d31c0 con 0x7fba24004800 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba44a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba44a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48770/0 (socket says 172.21.15.129:48770) 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba44a00640 1 -- 172.21.15.129:0/3042098703 learned_addr learned my addr 172.21.15.129:0/3042098703 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba37400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 0x7fba400cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba44a00640 1 -- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 msgr2=0x7fba4015faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:07.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.709+0000 7fba3fe00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 0x7fba4015faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba44a00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 0x7fba4015faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba44a00640 1 -- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 msgr2=0x7fba400cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba44a00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 0x7fba400cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba44a00640 1 -- 172.21.15.129:0/3042098703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba401f5930 con 0x7fba40159080 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba37400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 0x7fba400cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba3fe00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 0x7fba4015faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba44a00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fba4008c6a0 tx=0x7fba3800f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38004b20 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fba38004cc0 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba38003bd0 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba401f5c10 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.710+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba40159790 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.712+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba400d0f90 con 0x7fba40159080 2024-09-17T04:07:07.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.713+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fba38004030 con 0x7fba40159080 2024-09-17T04:07:07.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.714+0000 7fba3cc00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 0x7fba1c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:07.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.714+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fba3809b620 con 0x7fba40159080 2024-09-17T04:07:07.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.715+0000 7fba3fe00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 0x7fba1c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:07.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.715+0000 7fba3fe00640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 0x7fba1c07b0a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fba400d4000 tx=0x7fba30005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:07.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.719+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fba38063b20 con 0x7fba40159080 2024-09-17T04:07:07.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.937+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba401f56c0 con 0x7fba40159080 2024-09-17T04:07:07.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.937+0000 7fba3cc00640 1 -- 172.21.15.129:0/3042098703 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) ==== 95+0+5242 (secure 0 0 0) 0x7fba38063420 con 0x7fba40159080 2024-09-17T04:07:07.938 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:07.938 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":239,"btime":"2024-09-17T04:07:07:556226+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:06.565931+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":18}]} 2024-09-17T04:07:07.938 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 239 2024-09-17T04:07:07.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 msgr2=0x7fba1c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:07.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 0x7fba1c07b0a0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fba400d4000 tx=0x7fba30005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:07.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 msgr2=0x7fba400cde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:07.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fba4008c6a0 tx=0x7fba3800f3c0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 shutdown_connections 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fba1c078b90 0x7fba1c07b0a0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fba40159080 0x7fba400cde00 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba40158c70 0x7fba400cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 --2- 172.21.15.129:0/3042098703 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fba24004800 0x7fba4015faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 >> 172.21.15.129:0/3042098703 conn(0x7fba4008bff0 msgr2=0x7fba40160160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 shutdown_connections 2024-09-17T04:07:07.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:07.940+0000 7fba45400640 1 -- 172.21.15.129:0/3042098703 wait complete. 2024-09-17T04:07:08.367 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 239, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:06:48.110755+0000', 'modified': '2024-09-17T04:07:06.565931+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 248, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11772': {'gid': 11772, 'name': 'd', 'rank': 0, 'incarnation': 237, 'state': 'up:rejoin', 'state_seq': 8, 'addr': '172.21.15.181:6837/2055116414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2055116414}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2055116414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-09-17T04:07:08.367 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_11772:up:rejoin 2024-09-17T04:07:09.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:09.590 INFO:tasks.ceph.mds.d.smithi181.stderr:2024-09-17T04:07:09.591+0000 7facf0600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:07:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 -- 172.21.15.129:0/206624294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 msgr2=0x7fbb780ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 --2- 172.21.15.129:0/206624294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780ccc80 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7fbb78072e00 tx=0x7fbb6002a580 comp rx=0 tx=0).stop 2024-09-17T04:07:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 -- 172.21.15.129:0/206624294 shutdown_connections 2024-09-17T04:07:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 --2- 172.21.15.129:0/206624294 >> v2:172.21.15.181:3300/0 conn(0x7fbb780cd1c0 0x7fbb781609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 --2- 172.21.15.129:0/206624294 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780ccc80 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 -- 172.21.15.129:0/206624294 >> 172.21.15.129:0/206624294 conn(0x7fbb7806d410 msgr2=0x7fbb780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:09.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 -- 172.21.15.129:0/206624294 shutdown_connections 2024-09-17T04:07:09.595 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.594+0000 7fbb7f800640 1 -- 172.21.15.129:0/206624294 wait complete. 2024-09-17T04:07:09.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.596+0000 7fbb7f800640 1 Processor -- start 2024-09-17T04:07:09.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.597+0000 7fbb7f800640 1 -- start start 2024-09-17T04:07:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 0x7fbb781f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb780cd1c0 0x7fbb780b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:09.598 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbb78160260 con 0x7fbb780b4140 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbb781600e0 con 0x7fbb780705a0 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb7f800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbb78160f00 con 0x7fbb780cd1c0 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb6fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48782/0 (socket says 172.21.15.129:48782) 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.598+0000 7fbb6fe00640 1 -- 172.21.15.129:0/3618806419 learned_addr learned my addr 172.21.15.129:0/3618806419 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb7ce00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 0x7fbb781f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:09.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb7d800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb780cd1c0 0x7fbb780b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 -- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb780cd1c0 msgr2=0x7fbb780b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb780cd1c0 0x7fbb780b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 -- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 msgr2=0x7fbb781f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 0x7fbb781f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 -- 172.21.15.129:0/3618806419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7815cba0 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb7ce00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 0x7fbb781f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb6fe00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fbb60034040 tx=0x7fbb600039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6003e030 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbb600047a0 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.599+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb780c7990 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.600+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60035040 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.600+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb780c4f30 con 0x7fbb780b4140 2024-09-17T04:07:09.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.601+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb780ccc80 con 0x7fbb780b4140 2024-09-17T04:07:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.602+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbb60041030 con 0x7fbb780b4140 2024-09-17T04:07:09.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.603+0000 7fbb75600640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 0x7fbb5407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.603+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fbb600bb640 con 0x7fbb780b4140 2024-09-17T04:07:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.603+0000 7fbb7ce00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 0x7fbb5407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:09.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.604+0000 7fbb7ce00640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 0x7fbb5407b0a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fbb68006fd0 tx=0x7fbb68008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:09.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.607+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbb780cd5d0 con 0x7fbb780b4140 2024-09-17T04:07:09.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.827+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb7806a140 con 0x7fbb780b4140 2024-09-17T04:07:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.828+0000 7fbb75600640 1 -- 172.21.15.129:0/3618806419 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fbb600833d0 con 0x7fbb780b4140 2024-09-17T04:07:09.828 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:09.828 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:09.829 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 msgr2=0x7fbb5407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 0x7fbb5407b0a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fbb68006fd0 tx=0x7fbb68008040 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 msgr2=0x7fbb780c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7fbb60034040 tx=0x7fbb600039e0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 shutdown_connections 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbb54078b90 0x7fbb5407b0a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbb780cd1c0 0x7fbb780b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbb780b4140 0x7fbb780c5710 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 --2- 172.21.15.129:0/3618806419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb780705a0 0x7fbb781f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.830+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 >> 172.21.15.129:0/3618806419 conn(0x7fbb7806d410 msgr2=0x7fbb780ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:09.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.831+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 shutdown_connections 2024-09-17T04:07:09.832 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:09.831+0000 7fbb7f800640 1 -- 172.21.15.129:0/3618806419 wait complete. 2024-09-17T04:07:10.256 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 240, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:06:48.110755+0000', 'modified': '2024-09-17T04:07:08.563026+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 248, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11772': {'gid': 11772, 'name': 'd', 'rank': 0, 'incarnation': 237, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.181:6837/2055116414', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6835', 'nonce': 2055116414}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2055116414}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [37], 'metadata_pool': 36, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 11772, 'qdb_cluster': [11772]} 2024-09-17T04:07:10.256 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:07:10.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-17T04:07:10.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 -- 172.21.15.129:0/2443907959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b9be0 msgr2=0x7f038c0d0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 --2- 172.21.15.129:0/2443907959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b9be0 0x7f038c0d0f90 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f038c06dd50 tx=0x7f037c02a560 comp rx=0 tx=0).stop 2024-09-17T04:07:10.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 -- 172.21.15.129:0/2443907959 shutdown_connections 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 --2- 172.21.15.129:0/2443907959 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b13d0 0x7f038c0b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 --2- 172.21.15.129:0/2443907959 >> v2:172.21.15.181:3300/0 conn(0x7f038c0b3430 0x7f038c0d14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 --2- 172.21.15.129:0/2443907959 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b9be0 0x7f038c0d0f90 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 -- 172.21.15.129:0/2443907959 >> 172.21.15.129:0/2443907959 conn(0x7f038c08bff0 msgr2=0x7f038c09d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 -- 172.21.15.129:0/2443907959 shutdown_connections 2024-09-17T04:07:10.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.479+0000 7f0392600640 1 -- 172.21.15.129:0/2443907959 wait complete. 2024-09-17T04:07:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.481+0000 7f0392600640 1 Processor -- start 2024-09-17T04:07:10.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.481+0000 7f0392600640 1 -- start start 2024-09-17T04:07:10.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b3430 0x7f038c0ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 0x7f038c0caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0b20f0 con 0x7f038c0b13d0 2024-09-17T04:07:10.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0b1f70 con 0x7f038c0b9be0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0392600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f038c0b2270 con 0x7f038c0b3430 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48810/0 (socket says 172.21.15.129:48810) 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 -- 172.21.15.129:0/1043599313 learned_addr learned my addr 172.21.15.129:0/1043599313 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391c00640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 0x7f038c0caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0383e00640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b3430 0x7f038c0ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 -- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b3430 msgr2=0x7f038c0ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b3430 0x7f038c0ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 -- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 msgr2=0x7f038c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.483+0000 7f0391200640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 0x7f038c0caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0391200640 1 -- 172.21.15.129:0/1043599313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c0cc390 con 0x7f038c0b13d0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0391c00640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 0x7f038c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0391200640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f037c0099c0 tx=0x7f037c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0389600640 1 -- 172.21.15.129:0/1043599313 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c03e030 con 0x7f038c0b13d0 2024-09-17T04:07:10.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0389600640 1 -- 172.21.15.129:0/1043599313 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037c0047a0 con 0x7f038c0b13d0 2024-09-17T04:07:10.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0389600640 1 -- 172.21.15.129:0/1043599313 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c035040 con 0x7f038c0b13d0 2024-09-17T04:07:10.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038c0cc670 con 0x7f038c0b13d0 2024-09-17T04:07:10.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.484+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038c0ccc30 con 0x7f038c0b13d0 2024-09-17T04:07:10.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.486+0000 7f0389600640 1 -- 172.21.15.129:0/1043599313 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f037c041030 con 0x7f038c0b13d0 2024-09-17T04:07:10.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.488+0000 7f0389600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 0x7f036807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.488+0000 7f0389600640 1 -- 172.21.15.129:0/1043599313 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f037c0bb6a0 con 0x7f038c0b13d0 2024-09-17T04:07:10.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.489+0000 7f0383e00640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 0x7f036807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.489+0000 7f039ac79740 1 Processor -- start 2024-09-17T04:07:10.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.489+0000 7f039ac79740 1 -- start start 2024-09-17T04:07:10.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.489+0000 7f0383e00640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 0x7f036807b0d0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f038c15c9b0 tx=0x7f0374005e30 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84bf3b0 0x558fe88b90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84c7590 0x558fe8889150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558fe8812430 con 0x558fe84e8270 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558fe88025d0 con 0x558fe84c7590 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.490+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558fe87f1a00 con 0x558fe84bf3b0 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48812/0 (socket says 172.21.15.129:48812) 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 -- 172.21.15.129:0/554499574 learned_addr learned my addr 172.21.15.129:0/554499574 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391c00640 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84c7590 0x558fe8889150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0383e00640 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84bf3b0 0x558fe88b90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 -- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84bf3b0 msgr2=0x558fe88b90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84bf3b0 0x558fe88b90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 -- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84c7590 msgr2=0x558fe8889150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84c7590 0x558fe8889150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.492 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 -- 172.21.15.129:0/554499574 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe87ba520 con 0x558fe84e8270 2024-09-17T04:07:10.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0391200640 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f037c083270 tx=0x7f037c0830b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d201bd55988b8ee server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.491+0000 7f0381600640 1 -- 172.21.15.129:0/554499574 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c048030 con 0x558fe84e8270 2024-09-17T04:07:10.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f0381600640 1 -- 172.21.15.129:0/554499574 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037c005310 con 0x558fe84e8270 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f0381600640 1 -- 172.21.15.129:0/554499574 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c08ca70 con 0x558fe84e8270 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 -- 172.21.15.129:0/554499574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 msgr2=0x558fe88b8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f037c083270 tx=0x7f037c0830b0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 -- 172.21.15.129:0/554499574 shutdown_connections 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84c7590 0x558fe8889150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84bf3b0 0x558fe88b90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 --2- 172.21.15.129:0/554499574 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88b8b60 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 -- 172.21.15.129:0/554499574 >> 172.21.15.129:0/554499574 conn(0x558fe84dc3c0 msgr2=0x558fe888eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.492+0000 7f039ac79740 1 -- 172.21.15.129:0/554499574 shutdown_connections 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.493+0000 7f039ac79740 1 -- 172.21.15.129:0/554499574 wait complete. 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.494+0000 7f039ac79740 1 -- start start 2024-09-17T04:07:10.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.494+0000 7f039ac79740 1 Processor -- start 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88ae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88e5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558fe8798330 con 0x558fe84e8270 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558fe87f5810 con 0x558fe84bf3b0 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558fe8562620 con 0x558fe84c7590 2024-09-17T04:07:10.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.495+0000 7f039ac79740 10 client.0 _ll_register_callbacks cb 0x558fe879d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48814/0 (socket says 172.21.15.129:48814) 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 -- 172.21.15.129:0/1483490640 learned_addr learned my addr 172.21.15.129:0/1483490640 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0383e00640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88e5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391200640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 -- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 msgr2=0x558fe88e5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88e5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 -- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 msgr2=0x558fe88ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88ae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 -- 172.21.15.129:0/1483490640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe8553850 con 0x558fe84e8270 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f0391c00640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f0384012420 tx=0x7f03840128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab54653bb651665 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f037a000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.496+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384004d20 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f038401db10 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037c005310 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f038401dd10 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 10 client.12015 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x558fe85ca6b0 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037c005310 con 0x558fe84e8270 2024-09-17T04:07:10.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f0384024050 con 0x558fe84e8270 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 10 client.12015 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 10 client.12015 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 20 client.12015 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe84a6e40 0x558fe88e2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 4 client.12015 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.497+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558fe8804520 con 0x558fe84a6e40 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.498+0000 7f0391200640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe84a6e40 0x558fe88e2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.498+0000 7f0391200640 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe84a6e40 0x558fe88e2500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e49fbb01d2e733e server_cookie=ed1e8a64aba933e9 in_seq=0 out_seq=0 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.498+0000 7f037a000640 10 client.12015 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.499+0000 7f037a000640 1 -- 172.21.15.129:0/1483490640 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x558fe8804520 con 0x558fe84a6e40 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.499+0000 7f037a000640 10 client.12015 handle_command_reply: tid=0 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.499+0000 7f039ac79740 1 client.12015 shutdown 2024-09-17T04:07:10.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 msgr2=0x558fe88e5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f0384012420 tx=0x7f03840128e0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 shutdown_connections 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe84a6e40 0x558fe88e2500 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84e8270 0x558fe88e5e70 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88e5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 --2- 172.21.15.129:0/1483490640 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f037a000640 0 client.12015 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 >> 172.21.15.129:0/1483490640 conn(0x558fe84dc3c0 msgr2=0x558fe88c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 shutdown_connections 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 1 -- 172.21.15.129:0/1483490640 wait complete. 2024-09-17T04:07:10.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.500+0000 7f039ac79740 20 client.12015 trim_cache size 0 max 16384 2024-09-17T04:07:10.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.519+0000 7f039ac79740 1 Processor -- start 2024-09-17T04:07:10.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.519+0000 7f039ac79740 1 -- start start 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88f6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84e8270 0x558fe888e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f038c15f550 con 0x558fe84bf3b0 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f038c15e8b0 con 0x558fe84e8270 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f038c15ef00 con 0x558fe84c7590 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48826/0 (socket says 172.21.15.129:48826) 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 -- 172.21.15.129:0/3171584497 learned_addr learned my addr 172.21.15.129:0/3171584497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0383e00640 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391c00640 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84e8270 0x558fe888e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 -- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 msgr2=0x558fe88f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 -- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84e8270 msgr2=0x558fe888e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84e8270 0x558fe888e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.521+0000 7f0391200640 1 -- 172.21.15.129:0/3171584497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558fe888aee0 con 0x558fe84bf3b0 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f0391200640 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f037c041000 tx=0x7f037c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e92a92882ffd5e server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f0380c00640 1 -- 172.21.15.129:0/3171584497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c048040 con 0x558fe84bf3b0 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f0380c00640 1 -- 172.21.15.129:0/3171584497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037c0839e0 con 0x558fe84bf3b0 2024-09-17T04:07:10.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f0380c00640 1 -- 172.21.15.129:0/3171584497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c08ca70 con 0x558fe84bf3b0 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f039ac79740 1 -- 172.21.15.129:0/3171584497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 msgr2=0x558fe88b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.522+0000 7f039ac79740 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f037c041000 tx=0x7f037c086040 comp rx=0 tx=0).stop 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 -- 172.21.15.129:0/3171584497 shutdown_connections 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84e8270 0x558fe888e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84c7590 0x558fe88f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 --2- 172.21.15.129:0/3171584497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84bf3b0 0x558fe88b2480 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 -- 172.21.15.129:0/3171584497 >> 172.21.15.129:0/3171584497 conn(0x558fe84dc3c0 msgr2=0x558fe88847c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 -- 172.21.15.129:0/3171584497 shutdown_connections 2024-09-17T04:07:10.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.523+0000 7f039ac79740 1 -- 172.21.15.129:0/3171584497 wait complete. 2024-09-17T04:07:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.524+0000 7f039ac79740 1 -- start start 2024-09-17T04:07:10.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.524+0000 7f039ac79740 1 Processor -- start 2024-09-17T04:07:10.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 0x558fe8938960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x558fe88af820 con 0x558fe84c7590 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x558fe88b0d00 con 0x558fe84bf3b0 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x558fe88cc1d0 con 0x558fe84e8270 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f039ac79740 10 client.0 _ll_register_callbacks cb 0x7f0384009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.525+0000 7f0383e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48832/0 (socket says 172.21.15.129:48832) 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 -- 172.21.15.129:0/2460979512 learned_addr learned my addr 172.21.15.129:0/2460979512 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0391c00640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 0x558fe8938960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0391200640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 -- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 msgr2=0x558fe8938960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.528 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 0x558fe8938960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 -- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 msgr2=0x558fe88b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 -- 172.21.15.129:0/2460979512 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558fe88b8520 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0391c00640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 0x558fe8938960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f0383e00640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f03740206c0 tx=0x7f0374004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f36d679f59ed8f server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.526+0000 7f037a000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037402e060 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f0374011650 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f037c0839e0 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f0374011850 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 10 client.12021 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x558fe8953890 con 0x558fe84c7590 2024-09-17T04:07:10.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f037c0839e0 con 0x558fe84c7590 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f0374037020 con 0x558fe84c7590 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 10 client.12021 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 10 client.12021 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 20 client.12021 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe8952e70 0x558fe88ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 4 client.12021 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.527+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "status"}) -- 0x558fe87fffd0 con 0x558fe8952e70 2024-09-17T04:07:10.530 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.528+0000 7f0391200640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe8952e70 0x558fe88ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.528+0000 7f0391200640 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe8952e70 0x558fe88ac8e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76289f0d7123a3d9 server_cookie=68d3b584a316f992 in_seq=0 out_seq=0 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.528+0000 7f037a000640 10 client.12021 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f037a000640 1 -- 172.21.15.129:0/2460979512 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x558fe87fffd0 con 0x558fe8952e70 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f037a000640 10 client.12021 handle_command_reply: tid=0 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 client.12021 shutdown 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 msgr2=0x558fe8956e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7f03740206c0 tx=0x7f0374004730 comp rx=0 tx=0).stop 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 shutdown_connections 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x558fe8952e70 0x558fe88ac8e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x558fe84e8270 0x558fe8938960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f037a000640 0 client.12021 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.532 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x558fe84c7590 0x558fe8956e60 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 --2- 172.21.15.129:0/2460979512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558fe84bf3b0 0x558fe88b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 >> 172.21.15.129:0/2460979512 conn(0x558fe84dc3c0 msgr2=0x558fe88cc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 shutdown_connections 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 1 -- 172.21.15.129:0/2460979512 wait complete. 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.529+0000 7f039ac79740 20 client.12021 trim_cache size 0 max 16384 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:07:10.533 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 11772, 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 4.9761563219999996, 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 240, 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 248, 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 248, 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 26.970787041000001 2024-09-17T04:07:10.534 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 msgr2=0x7f036807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 0x7f036807b0d0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f038c15c9b0 tx=0x7f0374005e30 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 msgr2=0x7f038c150da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f037c0099c0 tx=0x7f037c0039e0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 shutdown_connections 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0368078bc0 0x7f036807b0d0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f038c0b9be0 0x7f038c0caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f038c0b3430 0x7f038c0ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 --2- 172.21.15.129:0/1043599313 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f038c0b13d0 0x7f038c150da0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 >> 172.21.15.129:0/1043599313 conn(0x7f038c08bff0 msgr2=0x7f038c06de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 shutdown_connections 2024-09-17T04:07:10.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.530+0000 7f0392600640 1 -- 172.21.15.129:0/1043599313 wait complete. 2024-09-17T04:07:10.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.18:0 dump tree / 2024-09-17T04:07:10.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 -- 172.21.15.129:0/1064196799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40cba90 msgr2=0x7f0ea40cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 --2- 172.21.15.129:0/1064196799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40cba90 0x7f0ea40cbea0 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40d1260 tx=0x7f0e9c02a980 comp rx=0 tx=0).stop 2024-09-17T04:07:10.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 -- 172.21.15.129:0/1064196799 shutdown_connections 2024-09-17T04:07:10.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 --2- 172.21.15.129:0/1064196799 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40cba90 0x7f0ea40cbea0 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 --2- 172.21.15.129:0/1064196799 >> v2:172.21.15.181:3300/0 conn(0x7f0ea40b3f70 0x7f0ea40b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.779+0000 7f0ea9600640 1 -- 172.21.15.129:0/1064196799 >> 172.21.15.129:0/1064196799 conn(0x7f0ea40ccc40 msgr2=0x7f0ea40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.780+0000 7f0ea9600640 1 -- 172.21.15.129:0/1064196799 shutdown_connections 2024-09-17T04:07:10.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.780+0000 7f0ea9600640 1 -- 172.21.15.129:0/1064196799 wait complete. 2024-09-17T04:07:10.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.782+0000 7f0ea9600640 1 Processor -- start 2024-09-17T04:07:10.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.783+0000 7f0ea9600640 1 -- start start 2024-09-17T04:07:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 0x7f0ea40c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 0x7f0ea4155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea40cc8e0 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea40cc760 con 0x7f0ea4074580 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea9600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea4166f20 con 0x7f0ea40cba90 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48848/0 (socket says 172.21.15.129:48848) 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.784+0000 7f0ea3400640 1 -- 172.21.15.129:0/726775382 learned_addr learned my addr 172.21.15.129:0/726775382 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 0x7f0ea40c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 -- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 msgr2=0x7f0ea4155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0e9a000640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 0x7f0ea4155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 0x7f0ea4155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 -- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 msgr2=0x7f0ea40c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 0x7f0ea40c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 -- 172.21.15.129:0/726775382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea40cee40 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 0x7f0ea40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0e9a000640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 0x7f0ea4155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0ea3400640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009d60 tx=0x7f0e9c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.785+0000 7f0e9b400640 1 -- 172.21.15.129:0/726775382 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c034040 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.786+0000 7f0e9b400640 1 -- 172.21.15.129:0/726775382 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e9c004600 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.786+0000 7f0e9b400640 1 -- 172.21.15.129:0/726775382 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c045480 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.786+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea40c8e80 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.788 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.786+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea40c9390 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.788+0000 7f0e9b400640 1 -- 172.21.15.129:0/726775382 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0e9c037020 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.789+0000 7f0e9b400640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 0x7f0e8007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.789 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.789+0000 7f0e9b400640 1 -- 172.21.15.129:0/726775382 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f0e9c0bc830 con 0x7f0ea40ce4f0 2024-09-17T04:07:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.789+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 0x7f0e8007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.790+0000 7f0eb05dd740 1 Processor -- start 2024-09-17T04:07:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.790+0000 7f0eb05dd740 1 -- start start 2024-09-17T04:07:10.790 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.790+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 0x7f0e8007b0a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009aa0 tx=0x7f0e94005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413613ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413613b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564136083850 con 0x564135d69220 2024-09-17T04:07:10.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564136072c80 con 0x564135d48540 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.791+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564136076a90 con 0x564135d40360 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48858/0 (socket says 172.21.15.129:48858) 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 -- 172.21.15.129:0/1592310404 learned_addr learned my addr 172.21.15.129:0/1592310404 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea3400640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413613b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0e9a000640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413613ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 -- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 msgr2=0x56413613ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413613ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 -- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 msgr2=0x56413613b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413613b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 -- 172.21.15.129:0/1592310404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564135d65280 con 0x564135d69220 2024-09-17T04:07:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea3400640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413613b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0e9a000640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413613ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.792+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400ef00 tx=0x7f0e94011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7cb66ab96592ff9 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0e93000640 1 -- 172.21.15.129:0/1592310404 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9401e640 con 0x564135d69220 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0e93000640 1 -- 172.21.15.129:0/1592310404 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e94023070 con 0x564135d69220 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0e93000640 1 -- 172.21.15.129:0/1592310404 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9401ead0 con 0x564135d69220 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 -- 172.21.15.129:0/1592310404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 msgr2=0x564136157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400ef00 tx=0x7f0e94011000 comp rx=0 tx=0).stop 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 -- 172.21.15.129:0/1592310404 shutdown_connections 2024-09-17T04:07:10.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413613b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413613ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.793+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/1592310404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d69220 0x564136157e50 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.794+0000 7f0eb05dd740 1 -- 172.21.15.129:0/1592310404 >> 172.21.15.129:0/1592310404 conn(0x564135d5d370 msgr2=0x564136110d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.794+0000 7f0eb05dd740 1 -- 172.21.15.129:0/1592310404 shutdown_connections 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.794+0000 7f0eb05dd740 1 -- 172.21.15.129:0/1592310404 wait complete. 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.795+0000 7f0eb05dd740 1 -- start start 2024-09-17T04:07:10.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.795+0000 7f0eb05dd740 1 Processor -- start 2024-09-17T04:07:10.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.799 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x564135de35d0 con 0x564135d40360 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x564136018550 con 0x564135d48540 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x564136134290 con 0x564135d69220 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0eb05dd740 10 client.0 _ll_register_callbacks cb 0x56413606cfb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.796+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48870/0 (socket says 172.21.15.129:48870) 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 -- 172.21.15.129:0/677083543 learned_addr learned my addr 172.21.15.129:0/677083543 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0e9a000640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea3400640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 -- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 msgr2=0x56413614eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 -- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 msgr2=0x56413614e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 -- 172.21.15.129:0/677083543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564135e4d690 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea3400640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0e94013040 tx=0x7f0e9400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341dc7824db67ae server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.797+0000 7f0e87400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94035060 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f0e9401eb60 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e94023070 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f0e9403b050 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 10 client.12033 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x564135eade20 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e94023070 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f0e9403e020 con 0x564135d40360 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 10 client.12033 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:10.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 10 client.12033 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:10.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 20 client.12033 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:10.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564135d28000 0x564136132f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 4 client.12033 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:10.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.798+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641360857a0 con 0x564135d28000 2024-09-17T04:07:10.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.799+0000 7f0e9a000640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564135d28000 0x564136132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.800+0000 7f0e9a000640 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564135d28000 0x564136132f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a3f9aaf42b127e server_cookie=8c21c7cc4b038040 in_seq=0 out_seq=0 2024-09-17T04:07:10.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.800+0000 7f0e87400640 10 client.12033 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0e87400640 1 -- 172.21.15.129:0/677083543 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5641360857a0 con 0x564135d28000 2024-09-17T04:07:10.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0e87400640 10 client.12033 handle_command_reply: tid=0 2024-09-17T04:07:10.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 client.12033 shutdown 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 msgr2=0x56413614c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0e94013040 tx=0x7f0e9400ec10 comp rx=0 tx=0).stop 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 shutdown_connections 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564135d28000 0x564136132f50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/677083543 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x56413614c750 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 >> 172.21.15.129:0/677083543 conn(0x564135d5d370 msgr2=0x564136109fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0e87400640 0 client.12033 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 shutdown_connections 2024-09-17T04:07:10.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 1 -- 172.21.15.129:0/677083543 wait complete. 2024-09-17T04:07:10.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.801+0000 7f0eb05dd740 20 client.12033 trim_cache size 0 max 16384 2024-09-17T04:07:10.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.816+0000 7f0eb05dd740 1 Processor -- start 2024-09-17T04:07:10.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.817+0000 7f0eb05dd740 1 -- start start 2024-09-17T04:07:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea4202120 con 0x564135d40360 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea4201140 con 0x564135d48540 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0ea409d120 con 0x564135d69220 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48874/0 (socket says 172.21.15.129:48874) 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea2a00640 1 -- 172.21.15.129:0/274529383 learned_addr learned my addr 172.21.15.129:0/274529383 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea3400640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0e9a000640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea2a00640 1 -- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 msgr2=0x56413614e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.818+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0ea2a00640 1 -- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 msgr2=0x56413614e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0ea2a00640 1 -- 172.21.15.129:0/274529383 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564135db2530 con 0x564135d40360 2024-09-17T04:07:10.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0e9a000640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400ef00 tx=0x7f0e94011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabc6ffe67cb6e65 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0e92600640 1 -- 172.21.15.129:0/274529383 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9401eb60 con 0x564135d40360 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0e92600640 1 -- 172.21.15.129:0/274529383 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e94023040 con 0x564135d40360 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.819+0000 7f0e92600640 1 -- 172.21.15.129:0/274529383 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e94030a60 con 0x564135d40360 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 -- 172.21.15.129:0/274529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 msgr2=0x564136111c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7f0e9400ef00 tx=0x7f0e94011000 comp rx=0 tx=0).stop 2024-09-17T04:07:10.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 -- 172.21.15.129:0/274529383 shutdown_connections 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d69220 0x56413614e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d48540 0x56413614e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/274529383 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d40360 0x564136111c10 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 -- 172.21.15.129:0/274529383 >> 172.21.15.129:0/274529383 conn(0x564135d5d370 msgr2=0x564136137ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 -- 172.21.15.129:0/274529383 shutdown_connections 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.820+0000 7f0eb05dd740 1 -- 172.21.15.129:0/274529383 wait complete. 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.821+0000 7f0eb05dd740 1 -- start start 2024-09-17T04:07:10.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.821+0000 7f0eb05dd740 1 Processor -- start 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.822+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413610a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d69220 0x564136111c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56413614ec60 con 0x564135d48540 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea409d770 con 0x564135d69220 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5641361763b0 con 0x564135d40360 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0eb05dd740 10 client.0 _ll_register_callbacks cb 0x7f0e9400eda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:10.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48886/0 (socket says 172.21.15.129:48886) 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 -- 172.21.15.129:0/2872614697 learned_addr learned my addr 172.21.15.129:0/2872614697 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413610a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0ea3400640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d69220 0x564136111c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 -- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 msgr2=0x56413610a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413610a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 -- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d69220 msgr2=0x564136111c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d69220 0x564136111c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 -- 172.21.15.129:0/2872614697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564136155f30 con 0x564135d48540 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e9a000640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x56413610bcd0 tx=0x7f0e8c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3cc99158b7dfc84 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.823+0000 7f0e87400640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c014bf0 con 0x564135d48540 2024-09-17T04:07:10.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f0e8c014d90 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e94023040 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f0e8c02a050 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 10 client.12039 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x5641361760f0 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e94023040 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f0e8c02d020 con 0x564135d48540 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 10 client.12039 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 10 client.12039 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 20 client.12039 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564136112150 0x5641361d4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 4 client.12039 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "dump tree", "root": "/"}] 2024-09-17T04:07:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.824+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "dump tree", "root": "/"}) -- 0x564136081250 con 0x564136112150 2024-09-17T04:07:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.825+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564136112150 0x5641361d4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.825+0000 7f0e87400640 10 client.12039 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.825+0000 7f0ea2a00640 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564136112150 0x5641361d4480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=354ec36536c225d2 server_cookie=8a18b4420f643a46 in_seq=0 out_seq=0 2024-09-17T04:07:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0e87400640 1 -- 172.21.15.129:0/2872614697 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+16602 (crc 0 0 0) 0x564136081250 con 0x564136112150 2024-09-17T04:07:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0e87400640 10 client.12039 handle_command_reply: tid=0 2024-09-17T04:07:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 client.12039 shutdown 2024-09-17T04:07:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 msgr2=0x5641361ce140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.830 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x56413610bcd0 tx=0x7f0e8c00eeb0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 shutdown_connections 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x564136112150 0x5641361d4480 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564135d69220 0x564136111c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x564135d48540 0x5641361ce140 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.826+0000 7f0eb05dd740 1 --2- 172.21.15.129:0/2872614697 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x564135d40360 0x56413610a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0e87400640 0 client.12039 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 >> 172.21.15.129:0/2872614697 conn(0x564135d5d370 msgr2=0x5641361376e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 shutdown_connections 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0eb05dd740 1 -- 172.21.15.129:0/2872614697 wait complete. 2024-09-17T04:07:10.831 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0eb05dd740 20 client.12039 trim_cache size 0 max 16384 2024-09-17T04:07:10.832 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:07:10.832 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.832 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T04:07:10.832 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:10.833 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:10.833 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:10.833 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 16877, 2024-09-17T04:07:10.833 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 1000, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 1333, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 0, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 0, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 0, 2024-09-17T04:07:10.834 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": -1, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 1, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:10.835 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 50, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 0, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.836 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 302, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 4, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 18446744073709551615, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:10.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [ 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "security.selinux", 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.839 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:10.841 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": {}, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:10.842 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": {}, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.843 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [], 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": -1, 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": -1, 2024-09-17T04:07:10.844 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [], 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627776, 2024-09-17T04:07:10.845 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:02.163753+0000", 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:07:01.882759+0000", 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 16877, 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 1000, 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 1333, 2024-09-17T04:07:10.846 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 0, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 0, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 0, 2024-09-17T04:07:10.847 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": -1, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 1, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:10.848 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.163753+0000", 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "2024-09-17T04:07:01.882759+0000", 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.163753+0000", 2024-09-17T04:07:10.849 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 50, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 50, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.850 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 50, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 504, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 2, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 18446744073709551615, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:10.851 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:10.852 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [ 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "security.selinux", 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.853 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.854 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.855 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.856 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:10.857 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:10.858 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.859 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [], 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": -1, 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": -1, 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [], 2024-09-17T04:07:10.860 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1, 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:06:48.625358+0000", 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 17407, 2024-09-17T04:07:10.861 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 0, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 0, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.862 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 4194304, 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 1, 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 4194304, 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": 37, 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:10.863 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 0, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "0.000000", 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:10.864 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 3, 2024-09-17T04:07:10.865 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:10.865 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:10.865 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:10.865 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:10.865 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 0, 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 2, 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 2 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.866 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 1, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 50, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 3, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.867 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:06:48.625358+0000" 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.868 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 6, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 2, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 1, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:10.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [], 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.870 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": 1, 2024-09-17T04:07:10.871 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 1, 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "openingsnapparents": 0, 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 3, 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.872 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:10.873 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.874 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [ 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "client_id": 10996, 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "pending": "pAsLsXsFs", 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "issued": "pAsLsXsFs", 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "wanted": "pAsLsXsFs", 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "last_sent": 0 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": -1, 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": -1, 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [ 2024-09-17T04:07:10.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": "0x1", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "snapid_first": 2, 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": "506", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "version": "506", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "committing_version": "0", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "committed_version": "0", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "is_rep": false, 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_auth": "0", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:10.876 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty", 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "complete" 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "child": 1, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "subtree": 1, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "subtreetemp": 0, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "waiter": 0, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 3, 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: "dentries": [ 2024-09-17T04:07:10.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "b", 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": true, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": false, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 1099511627777, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 506, 2024-09-17T04:07:10.879 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 0, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.880 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.881 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a", 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:10.882 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": true, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": false, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 1099511627776, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 506, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 0, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:10.883 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:10.884 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:10.885 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:10.886 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:07:10.887 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 msgr2=0x7f0e8007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 0x7f0e8007b0a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009aa0 tx=0x7f0e94005db0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 msgr2=0x7f0ea40ce900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009d60 tx=0x7f0e9c003f90 comp rx=0 tx=0).stop 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 shutdown_connections 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e80078b90 0x7f0e8007b0a0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.888 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0ea40ce4f0 0x7f0ea40ce900 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0ea40cba90 0x7f0ea4155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 --2- 172.21.15.129:0/726775382 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ea4074580 0x7f0ea40c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.827+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 >> 172.21.15.129:0/726775382 conn(0x7f0ea40ccc40 msgr2=0x7f0ea40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.828+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 shutdown_connections 2024-09-17T04:07:10.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:10.828+0000 7f0ea9600640 1 -- 172.21.15.129:0/726775382 wait complete. 2024-09-17T04:07:10.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.109+0000 7f5abae00640 1 -- 172.21.15.129:0/165614012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40b4140 msgr2=0x7f5ab40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.109+0000 7f5abae00640 1 --2- 172.21.15.129:0/165614012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40b4140 0x7f5ab40ccc80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009860 tx=0x7f5aa002a5a0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.110+0000 7f5abae00640 1 -- 172.21.15.129:0/165614012 shutdown_connections 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.110+0000 7f5abae00640 1 --2- 172.21.15.129:0/165614012 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40b4140 0x7f5ab40ccc80 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.110+0000 7f5abae00640 1 -- 172.21.15.129:0/165614012 >> 172.21.15.129:0/165614012 conn(0x7f5ab406d410 msgr2=0x7f5ab40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:11.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.110+0000 7f5abae00640 1 -- 172.21.15.129:0/165614012 shutdown_connections 2024-09-17T04:07:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.110+0000 7f5abae00640 1 -- 172.21.15.129:0/165614012 wait complete. 2024-09-17T04:07:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.112+0000 7f5abae00640 1 Processor -- start 2024-09-17T04:07:11.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.112+0000 7f5abae00640 1 -- start start 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.113+0000 7f5abae00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.113+0000 7f5abae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab40b4140 0x7f5ab4099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.113+0000 7f5abae00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40cd1c0 0x7f5ab4153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5abae00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4160260 con 0x7f5ab40705a0 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5abae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41600e0 con 0x7f5ab40b4140 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5abae00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab41610d0 con 0x7f5ab40cd1c0 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48898/0 (socket says 172.21.15.129:48898) 2024-09-17T04:07:11.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 -- 172.21.15.129:0/732761969 learned_addr learned my addr 172.21.15.129:0/732761969 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:11.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 -- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40cd1c0 msgr2=0x7f5ab4153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:11.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40cd1c0 0x7f5ab4153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 -- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab40b4140 msgr2=0x7f5ab4099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:11.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab40b4140 0x7f5ab4099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 -- 172.21.15.129:0/732761969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab40c8740 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.114+0000 7f5ab3e00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c390 tx=0x7f5aa400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.115+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4017070 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.115+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5aa40053c0 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.115+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003850 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.115+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab40b1410 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.115+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab40b19d0 con 0x7f5ab40705a0 2024-09-17T04:07:11.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.116+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab40ccc80 con 0x7f5ab40705a0 2024-09-17T04:07:11.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.117+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5aa401a070 con 0x7f5ab40705a0 2024-09-17T04:07:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.118+0000 7f5aabe00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 0x7f5a9407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:11.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.118+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f5aa409a200 con 0x7f5ab40705a0 2024-09-17T04:07:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.119+0000 7f5ab3400640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 0x7f5a9407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:11.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.120+0000 7f5ab3400640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 0x7f5a9407b0d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0034000 tx=0x7f5aa00046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:11.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.123+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5aa4062540 con 0x7f5ab40705a0 2024-09-17T04:07:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.340+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ab406a140 con 0x7f5ab40705a0 2024-09-17T04:07:11.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.341+0000 7f5aabe00640 1 -- 172.21.15.129:0/732761969 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7f5aa4061e40 con 0x7f5ab40705a0 2024-09-17T04:07:11.341 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:11.342 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:11.342 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 msgr2=0x7f5a9407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 0x7f5a9407b0d0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0034000 tx=0x7f5aa00046c0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 msgr2=0x7f5ab4073340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400c390 tx=0x7f5aa400c850 comp rx=0 tx=0).stop 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 shutdown_connections 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5a94078bc0 0x7f5a9407b0d0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5ab40cd1c0 0x7f5ab4153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ab40b4140 0x7f5ab4099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 --2- 172.21.15.129:0/732761969 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5ab40705a0 0x7f5ab4073340 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 >> 172.21.15.129:0/732761969 conn(0x7f5ab406d410 msgr2=0x7f5ab4074310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 shutdown_connections 2024-09-17T04:07:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11.343+0000 7f5abae00640 1 -- 172.21.15.129:0/732761969 wait complete. 2024-09-17T04:07:11.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:07:11.618 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-09-17T04:07:11.618 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-09-17T04:07:11.618 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c ' 2024-09-17T04:07:11.618 DEBUG:teuthology.orchestra.run.smithi129:> rm a/* 2024-09-17T04:07:11.618 DEBUG:teuthology.orchestra.run.smithi129:> sync -f . 2024-09-17T04:07:11.618 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:07:11.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:07:11.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:11.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.18:0 dump tree / 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 -- 172.21.15.129:0/2106965831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 msgr2=0x7f1388160aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 --2- 172.21.15.129:0/2106965831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f1388160aa0 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f13880cc730 tx=0x7f137802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 -- 172.21.15.129:0/2106965831 shutdown_connections 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 --2- 172.21.15.129:0/2106965831 >> v2:172.21.15.181:3300/0 conn(0x7f1388160fe0 0x7f1388074170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 --2- 172.21.15.129:0/2106965831 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f1388160aa0 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.067+0000 7f138d600640 1 -- 172.21.15.129:0/2106965831 >> 172.21.15.129:0/2106965831 conn(0x7f138806cde0 msgr2=0x7f138806d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.068 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.068+0000 7f138d600640 1 -- 172.21.15.129:0/2106965831 shutdown_connections 2024-09-17T04:07:12.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.068+0000 7f138d600640 1 -- 172.21.15.129:0/2106965831 wait complete. 2024-09-17T04:07:12.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.070+0000 7f138d600640 1 Processor -- start 2024-09-17T04:07:12.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.070+0000 7f138d600640 1 -- start start 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13880746b0 0x7f13880b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 0x7f13881f87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f138806f450 con 0x7f13880cd590 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f13880d0820 con 0x7f138813fb20 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.071+0000 7f138d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f138806f5d0 con 0x7f13880746b0 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.073 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48916/0 (socket says 172.21.15.129:48916) 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13880746b0 0x7f13880b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 -- 172.21.15.129:0/1217877752 learned_addr learned my addr 172.21.15.129:0/1217877752 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f138cc00640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 0x7f13881f87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 -- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13880746b0 msgr2=0x7f13880b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13880746b0 0x7f13880b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 -- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 msgr2=0x7f13881f87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 0x7f13881f87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 -- 172.21.15.129:0/1217877752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13881f96a0 con 0x7f13880cd590 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f138cc00640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 0x7f13881f87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.072+0000 7f1387400640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f137400e790 tx=0x7f137400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.073+0000 7f137fe00640 1 -- 172.21.15.129:0/1217877752 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137400eea0 con 0x7f13880cd590 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.073+0000 7f137fe00640 1 -- 172.21.15.129:0/1217877752 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f137400cdb0 con 0x7f13880cd590 2024-09-17T04:07:12.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.073+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13880b1850 con 0x7f13880cd590 2024-09-17T04:07:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.073+0000 7f137fe00640 1 -- 172.21.15.129:0/1217877752 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374010690 con 0x7f13880cd590 2024-09-17T04:07:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.073+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13880b1af0 con 0x7f13880cd590 2024-09-17T04:07:12.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.075+0000 7f137fe00640 1 -- 172.21.15.129:0/1217877752 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1374010830 con 0x7f13880cd590 2024-09-17T04:07:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.076+0000 7f137fe00640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 0x7f136807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.077+0000 7f137fe00640 1 -- 172.21.15.129:0/1217877752 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f137409b740 con 0x7f13880cd590 2024-09-17T04:07:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.077+0000 7f1395be5740 1 Processor -- start 2024-09-17T04:07:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.077+0000 7f1395be5740 1 -- start start 2024-09-17T04:07:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.077+0000 7f1387e00640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 0x7f136807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.077 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.077+0000 7f1387e00640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 0x7f136807b0d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f1378033000 tx=0x7f1378004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d8259c7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d8259aa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d8258f2850 con 0x55d8255b7540 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d8258e1c80 con 0x55d8255d8220 2024-09-17T04:07:12.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d8258e5a90 con 0x55d8255af360 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f138cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d8259aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1387400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f138cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d8259aa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48932/0 (socket says 172.21.15.129:48932) 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1387400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:46244/0 (socket says 172.21.15.129:46244) 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f138cc00640 1 -- 172.21.15.129:0/2045986181 learned_addr learned my addr 172.21.15.129:0/2045986181 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.078+0000 7f1387e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d8259c7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 -- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 msgr2=0x55d8259c7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d8259c7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 -- 172.21.15.129:0/2045986181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 msgr2=0x55d8259aa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d8259aa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 -- 172.21.15.129:0/2045986181 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8255d4280 con 0x55d8255af360 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1387400640 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1374099b40 tx=0x7f1374062fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fa1cab10f7ee4ea server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1373e00640 1 -- 172.21.15.129:0/2045986181 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374067ef0 con 0x55d8255af360 2024-09-17T04:07:12.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1373e00640 1 -- 172.21.15.129:0/2045986181 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f137405f730 con 0x55d8255af360 2024-09-17T04:07:12.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.079+0000 7f1373e00640 1 -- 172.21.15.129:0/2045986181 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374085da0 con 0x55d8255af360 2024-09-17T04:07:12.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 -- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 msgr2=0x55d8259a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1374099b40 tx=0x7f1374062fd0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 -- 172.21.15.129:0/2045986181 shutdown_connections 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d8259aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259a9ce0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 --2- 172.21.15.129:0/2045986181 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d8259c7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 -- 172.21.15.129:0/2045986181 >> 172.21.15.129:0/2045986181 conn(0x55d8255cc370 msgr2=0x55d82597ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 -- 172.21.15.129:0/2045986181 shutdown_connections 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.080+0000 7f1395be5740 1 -- 172.21.15.129:0/2045986181 wait complete. 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.081+0000 7f1395be5740 1 -- start start 2024-09-17T04:07:12.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.081+0000 7f1395be5740 1 Processor -- start 2024-09-17T04:07:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255b7540 0x55d82597f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d82597f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d8256525d0 con 0x55d8255af360 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55d825887550 con 0x55d8255d8220 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d8259bf710 con 0x55d8255b7540 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1395be5740 10 client.0 _ll_register_callbacks cb 0x55d8258dbfb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1387e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1387e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48936/0 (socket says 172.21.15.129:48936) 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1387e00640 1 -- 172.21.15.129:0/2077601611 learned_addr learned my addr 172.21.15.129:0/2077601611 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f138cc00640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d82597f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.083+0000 7f1387400640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255b7540 0x55d82597f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 -- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255b7540 msgr2=0x55d82597f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255b7540 0x55d82597f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 -- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 msgr2=0x55d82597f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d82597f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 -- 172.21.15.129:0/2077601611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8256bc690 con 0x55d8255af360 2024-09-17T04:07:12.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f138cc00640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d82597f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1387e00640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f1378034040 tx=0x7f1378009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d2094bf379e336 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1370c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137804e070 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f1378003a40 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.084+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f137405f730 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f1378059050 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 10 client.12060 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x55d82571ce20 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f137405f730 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f137805c020 con 0x55d8255af360 2024-09-17T04:07:12.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 10 client.12060 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 10 client.12060 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 20 client.12060 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825597000 0x55d825997c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 4 client.12060 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8258f47a0 con 0x55d825597000 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.085+0000 7f1387400640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825597000 0x55d825997c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.086+0000 7f1387400640 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825597000 0x55d825997c00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e0161f0e1871048 server_cookie=a497d009435afd9 in_seq=0 out_seq=0 2024-09-17T04:07:12.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.086+0000 7f1370c00640 10 client.12060 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:12.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1370c00640 1 -- 172.21.15.129:0/2077601611 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55d8258f47a0 con 0x55d825597000 2024-09-17T04:07:12.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1370c00640 10 client.12060 handle_command_reply: tid=0 2024-09-17T04:07:12.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 client.12060 shutdown 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 msgr2=0x55d8259bb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f1378034040 tx=0x7f1378009260 comp rx=0 tx=0).stop 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 shutdown_connections 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825597000 0x55d825997c00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d82597f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255b7540 0x55d82597f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 --2- 172.21.15.129:0/2077601611 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255af360 0x55d8259bb580 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1370c00640 0 client.12060 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 >> 172.21.15.129:0/2077601611 conn(0x55d8255cc370 msgr2=0x55d8259c8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 shutdown_connections 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 1 -- 172.21.15.129:0/2077601611 wait complete. 2024-09-17T04:07:12.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.087+0000 7f1395be5740 20 client.12060 trim_cache size 0 max 16384 2024-09-17T04:07:12.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.103+0000 7f1395be5740 1 Processor -- start 2024-09-17T04:07:12.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.103+0000 7f1395be5740 1 -- start start 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259dc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d825981120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1388201ff0 con 0x55d8255b7540 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1388201010 con 0x55d8255d8220 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f13881f5b10 con 0x55d8255af360 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1387400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1387400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48952/0 (socket says 172.21.15.129:48952) 2024-09-17T04:07:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1387400640 1 -- 172.21.15.129:0/250228806 learned_addr learned my addr 172.21.15.129:0/250228806 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f1387e00640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259dc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.104+0000 7f138cc00640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d825981120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 -- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 msgr2=0x55d8259dc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259dc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 -- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 msgr2=0x55d825981120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d825981120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 -- 172.21.15.129:0/250228806 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d825621530 con 0x55d8255b7540 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387e00640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259dc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1387400640 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f1374063680 tx=0x7f1374066040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcaf1b54a5a83b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1373400640 1 -- 172.21.15.129:0/250228806 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374082070 con 0x55d8255b7540 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1373400640 1 -- 172.21.15.129:0/250228806 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1374085b80 con 0x55d8255b7540 2024-09-17T04:07:12.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.105+0000 7f1373400640 1 -- 172.21.15.129:0/250228806 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374062720 con 0x55d8255b7540 2024-09-17T04:07:12.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 -- 172.21.15.129:0/250228806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 msgr2=0x55d825980be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f1374063680 tx=0x7f1374066040 comp rx=0 tx=0).stop 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 -- 172.21.15.129:0/250228806 shutdown_connections 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255d8220 0x55d825981120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255b7540 0x55d825980be0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 --2- 172.21.15.129:0/250228806 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259dc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 -- 172.21.15.129:0/250228806 >> 172.21.15.129:0/250228806 conn(0x55d8255cc370 msgr2=0x55d8259bc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 -- 172.21.15.129:0/250228806 shutdown_connections 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.106+0000 7f1395be5740 1 -- 172.21.15.129:0/250228806 wait complete. 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.107+0000 7f1395be5740 1 -- start start 2024-09-17T04:07:12.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.107+0000 7f1395be5740 1 Processor -- start 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259bb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255b7540 0x55d8259c0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55d8259bce50 con 0x55d8255d8220 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1388200b80 con 0x55d8255b7540 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55d8259a9d80 con 0x55d8255af360 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1395be5740 10 client.0 _ll_register_callbacks cb 0x7f1378009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48954/0 (socket says 172.21.15.129:48954) 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 -- 172.21.15.129:0/147956954 learned_addr learned my addr 172.21.15.129:0/147956954 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1387400640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255b7540 0x55d8259c0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f1387e00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259bb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 -- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 msgr2=0x55d8259bb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259bb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 -- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255b7540 msgr2=0x55d8259c0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255b7540 0x55d8259c0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.109+0000 7f138cc00640 1 -- 172.21.15.129:0/147956954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8259c7940 con 0x55d8255d8220 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f138cc00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x55d8259aa760 tx=0x7f1380005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5996cc2890111fb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1370c00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1380006b50 con 0x55d8255d8220 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f1380006cf0 con 0x55d8255d8220 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1374085b80 con 0x55d8255d8220 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f1380016b30 con 0x55d8255d8220 2024-09-17T04:07:12.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1395be5740 10 client.12066 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x55d8259aa1c0 con 0x55d8255d8220 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.110+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1374085b80 con 0x55d8255d8220 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f1380016cd0 con 0x55d8255d8220 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 10 client.12066 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 10 client.12066 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 20 client.12066 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825a46d40 0x55d825a42c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 4 client.12066 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "dump tree", "root": "/"}] 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "dump tree", "root": "/"}) -- 0x55d8258f0250 con 0x55d825a46d40 2024-09-17T04:07:12.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1387e00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825a46d40 0x55d825a42c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1370c00640 10 client.12066 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.111+0000 7f1387e00640 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825a46d40 0x55d825a42c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f596ddc13125b85 server_cookie=9bc48289959cc537 in_seq=0 out_seq=0 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1370c00640 1 -- 172.21.15.129:0/147956954 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+125867 (crc 0 0 0) 0x55d8258f0250 con 0x55d825a46d40 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1370c00640 10 client.12066 handle_command_reply: tid=0 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 client.12066 shutdown 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 msgr2=0x55d8259e68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x55d8259aa760 tx=0x7f1380005710 comp rx=0 tx=0).stop 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 shutdown_connections 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55d825a46d40 0x55d825a42c00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55d8255d8220 0x55d8259e68d0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55d8255b7540 0x55d8259c0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 --2- 172.21.15.129:0/147956954 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55d8255af360 0x55d8259bb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1370c00640 0 client.12066 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.114+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 >> 172.21.15.129:0/147956954 conn(0x55d8255cc370 msgr2=0x55d825980be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.115+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 shutdown_connections 2024-09-17T04:07:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.115+0000 7f1395be5740 1 -- 172.21.15.129:0/147956954 wait complete. 2024-09-17T04:07:12.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.115+0000 7f1395be5740 20 client.12066 trim_cache size 0 max 16384 2024-09-17T04:07:12.117 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:07:12.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627777, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 16877, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 1000, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 1333, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:12.118 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 0, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 0, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 0, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": -1, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 1, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:12.119 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.120 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:02.166753+0000", 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 50, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 0, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 50 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.121 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 302, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:12.122 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 4, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 18446744073709551615, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [ 2024-09-17T04:07:12.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "security.selinux", 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.124 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:12.125 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:12.126 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.127 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [], 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": -1, 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": -1, 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [], 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1099511627776, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:11.807545+0000", 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:07:01.882759+0000", 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 16877, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 1000, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 1333, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:12.129 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 0, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 0, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 0, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": -1, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 1, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:12.130 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:11.807545+0000", 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "2024-09-17T04:07:01.882759+0000", 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 100, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:11.807545+0000", 2024-09-17T04:07:12.131 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 100 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:11.807545+0000" 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.132 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 50, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 605, 2024-09-17T04:07:12.133 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 2, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 18446744073709551615, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:12.134 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [ 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "security.selinux", 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.135 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.136 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": 1, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 1, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "dirtyrstat": 1, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 4, 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:12.137 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": { 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "excl", 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "ifile", 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.138 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": { 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "inest", 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.139 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.140 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty", 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "dirtyrstat" 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [ 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "client_id": 10996, 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "pending": "pAsLsXsFsx", 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "issued": "pAsLsXsFsx", 2024-09-17T04:07:12.141 INFO:teuthology.orchestra.run.smithi129.stdout: "wanted": "pAsLsXsFsxcral", 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "last_sent": 103 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": 10996, 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": 10996, 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [ 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "/a", 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": "0x10000000000", 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "snapid_first": 2, 2024-09-17T04:07:12.142 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": "502", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "version": "502", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "committing_version": "401", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "committed_version": "401", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "is_rep": false, 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_auth": "", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty", 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "complete" 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.143 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.144 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "child": 1, 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "waiter": 0, 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 2, 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "dentries": [ 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/21", 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.145 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 429, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 429, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.146 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.147 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.148 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.149 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/19", 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.150 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 423, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 423, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.152 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.153 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/8", 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.154 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 499, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 499, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.155 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.156 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.157 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/27", 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.158 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 441, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 441, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.159 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.160 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.161 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/3", 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.162 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 447, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 447, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.163 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.164 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.165 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/15", 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.166 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 415, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 415, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.167 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.168 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.169 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/9", 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.170 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 501, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 501, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.171 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.172 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.173 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.174 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/31", 2024-09-17T04:07:12.175 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.176 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.176 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.176 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.176 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.176 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 451, 2024-09-17T04:07:12.177 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 451, 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.178 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.179 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.180 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.181 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.182 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.183 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/36", 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.184 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.185 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 461, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 461, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.186 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.187 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.188 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.189 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.190 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/20", 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 427, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 427, 2024-09-17T04:07:12.191 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.192 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.193 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.194 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/12", 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 409, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 409, 2024-09-17T04:07:12.195 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.196 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.197 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.198 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/46", 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.199 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 483, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 483, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.200 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.201 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/39", 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 467, 2024-09-17T04:07:12.203 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 467, 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.204 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.205 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.206 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/50", 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.207 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 493, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 493, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.208 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.209 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.210 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/10", 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.211 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 405, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 405, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.213 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.214 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/34", 2024-09-17T04:07:12.215 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 457, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 457, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.216 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.218 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.219 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/2", 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.220 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 425, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 425, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.221 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.222 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.223 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/43", 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.224 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 477, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 477, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.225 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.226 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.227 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/38", 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.228 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 465, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 465, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.229 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.230 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.231 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/4", 2024-09-17T04:07:12.232 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.233 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 469, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 469, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.235 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.236 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/42", 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.238 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 475, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 475, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.239 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.240 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/11", 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 407, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 407, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.243 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.244 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.245 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/28", 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.246 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 443, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 443, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.247 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.248 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.249 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.250 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/1", 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.252 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 403, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 403, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.253 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.254 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.255 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.256 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.256 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.256 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.256 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.257 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.257 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.257 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.257 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.257 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.258 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/24", 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.259 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 435, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 435, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.260 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.261 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.262 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/16", 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 417, 2024-09-17T04:07:12.264 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 417, 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.267 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/29", 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.268 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 445, 2024-09-17T04:07:12.269 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 445, 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.270 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.271 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.272 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.273 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.274 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/32", 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.275 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 453, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 453, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.277 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.278 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/49", 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.279 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 489, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 489, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.280 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.281 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.282 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.283 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/41", 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.284 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 473, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 473, 2024-09-17T04:07:12.285 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.286 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.287 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.288 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/45", 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.289 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 481, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 481, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.290 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.292 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/22", 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.293 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 431, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 431, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.294 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.295 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.296 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.297 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/25", 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.299 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 437, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 437, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.300 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.301 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.302 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.303 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/5", 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.304 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 491, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 491, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.305 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.307 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/40", 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.309 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 471, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 471, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.310 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.312 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/30", 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.313 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 449, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 449, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.314 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.315 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.316 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/14", 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.318 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 413, 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 413, 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.319 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.320 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.321 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.322 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/18", 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.323 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 421, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 421, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.324 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.325 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.326 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/13", 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.327 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 411, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 411, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.328 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.329 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.330 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.331 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/44", 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.332 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 479, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 479, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.333 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.334 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.335 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/33", 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.336 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 455, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 455, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.337 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.338 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.339 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/23", 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.340 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 433, 2024-09-17T04:07:12.341 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 433, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.342 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.343 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.344 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/7", 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.345 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 497, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 497, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.346 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.347 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.348 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.349 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/17", 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.350 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 419, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 419, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.351 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.352 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.353 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/35", 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.354 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 459, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 459, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.355 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.356 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.357 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/26", 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.358 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 439, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 439, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.359 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.360 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.361 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.362 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/37", 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 463, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 463, 2024-09-17T04:07:12.363 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.364 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.365 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.366 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.367 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/6", 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 495, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 495, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.369 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.370 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.371 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/48", 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.372 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 487, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 487, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.373 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.374 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.375 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a/47", 2024-09-17T04:07:12.376 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 485, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 485, 2024-09-17T04:07:12.377 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.378 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "request": 0, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 0, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 1, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.379 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.380 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "ino": 1, 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "rdev": 0, 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "ctime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "btime": "2024-09-17T04:06:48.625358+0000", 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": 17407, 2024-09-17T04:07:12.381 INFO:teuthology.orchestra.run.smithi129.stdout: "uid": 0, 2024-09-17T04:07:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: "gid": 0, 2024-09-17T04:07:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: "nlink": 1, 2024-09-17T04:07:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_layout": { 2024-09-17T04:07:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_hash": 2, 2024-09-17T04:07:12.382 INFO:teuthology.orchestra.run.smithi129.stdout: "unused1": 0, 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: "unused2": 0, 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: "unused3": 0 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: "layout": { 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_unit": 4194304, 2024-09-17T04:07:12.383 INFO:teuthology.orchestra.run.smithi129.stdout: "stripe_count": 1, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "object_size": 4194304, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_id": 37, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "pool_ns": "" 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "old_pools": [], 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 0, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_seq": 0, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_size": 18446744073709551615, 2024-09-17T04:07:12.384 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_from": 0, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "truncate_pending": 0, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "atime": "0.000000", 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "time_warp_seq": 0, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 3, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "export_pin": -1, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_random_pin": 0, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "export_ephemeral_distributed_pin": false, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "quiesce_block": false, 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "client_ranges": [], 2024-09-17T04:07:12.385 INFO:teuthology.orchestra.run.smithi129.stdout: "dirstat": { 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "mtime": "2024-09-17T04:07:01.887759+0000", 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "num_files": 0, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "num_subdirs": 2, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "change_attr": 2 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rstat": { 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 1, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 50, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 3, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:07:02.166753+0000" 2024-09-17T04:07:12.386 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "accounted_rstat": { 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 0, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "rbytes": 0, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "rfiles": 0, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "rsubdirs": 1, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "rsnaps": 0, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "rctime": "2024-09-17T04:06:48.625358+0000" 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 6, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "file_data_version": 0, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "xattr_version": 1, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "backtrace_version": 2, 2024-09-17T04:07:12.387 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_version": 1, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "inline_data_length": 0, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "stray_prior_path": "", 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "max_size_ever": 0, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "quota": { 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "max_bytes": 0, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "max_files": 0 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_stamp": "0.000000", 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "last_scrub_version": 0, 2024-09-17T04:07:12.388 INFO:teuthology.orchestra.run.smithi129.stdout: "symlink": "", 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrs": [], 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtree": { 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "splits": [] 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "old_inodes": [], 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "oldest_snap": 18446744073709551614, 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "damage_flags": 0, 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.389 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.390 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "dirtyscattered": 1, 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": 1, 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": 1, 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "openingsnapparents": 0, 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1 2024-09-17T04:07:12.391 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 4, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iversion", 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.392 INFO:teuthology.orchestra.run.smithi129.stdout: "authlock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "linklock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfragtreelock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "filelock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "xattrlock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "snaplock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "nestlock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "flocklock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "policylock": {}, 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "quiescelock": { 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.393 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "iquiesce", 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "client_caps": [ 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "client_id": 10996, 2024-09-17T04:07:12.394 INFO:teuthology.orchestra.run.smithi129.stdout: "pending": "pAsLsXsFs", 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "issued": "pAsLsXsFs", 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "wanted": "pAsLsXsFs", 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "last_sent": 2 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "loner": -1, 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "want_loner": -1, 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "mds_caps_wanted": [], 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrags": [ 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "", 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "dirfrag": "0x1", 2024-09-17T04:07:12.395 INFO:teuthology.orchestra.run.smithi129.stdout: "snapid_first": 2, 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": "606", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "version": "606", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "committing_version": "0", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "committed_version": "0", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "is_rep": false, 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "dir_auth": "0", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty", 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "complete" 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.396 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.397 INFO:teuthology.orchestra.run.smithi129.stdout: "child": 1, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "subtree": 1, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "subtreetemp": 0, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "waiter": 0, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 3, 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "dentries": [ 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "pyvenv.cfg", 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.398 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": false, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": true, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 0, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 506, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 0, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.399 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.400 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.401 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "b", 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.402 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": true, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": false, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 1099511627777, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 506, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 0, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.403 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": {}, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 0, 2024-09-17T04:07:12.404 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.405 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "auth" 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "path": "a", 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "alternate_name": "", 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "path_ino": 1, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_first": 2, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "snap_last": 18446744073709551614, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "is_primary": true, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "is_remote": false, 2024-09-17T04:07:12.406 INFO:teuthology.orchestra.run.smithi129.stdout: "is_null": false, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "is_new": false, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "inode": 1099511627776, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "remote_type": "", 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "version": 605, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "projected_version": 605, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "is_auth": true, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_state": { 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "replicas": {} 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_state": { 2024-09-17T04:07:12.407 INFO:teuthology.orchestra.run.smithi129.stdout: "authority": [ 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: 0, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: -2 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "replica_nonce": 0 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_pins": 0, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "is_frozen": false, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "is_freezing": false, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "pins": { 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": 0, 2024-09-17T04:07:12.408 INFO:teuthology.orchestra.run.smithi129.stdout: "inodepin": 1, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty": 1, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "authpin": 0 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "nref": 2, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "lock": {}, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "versionlock": { 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "gather_set": [], 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "lock", 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "type": "dversion", 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "is_leased": false, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "num_rdlocks": 0, 2024-09-17T04:07:12.409 INFO:teuthology.orchestra.run.smithi129.stdout: "num_wrlocks": 0, 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "num_xlocks": 0, 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "xlock_by": {} 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "states": [ 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "auth", 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "dirty" 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-17T04:07:12.410 INFO:teuthology.orchestra.run.smithi129.stdout: "is_subvolume": false 2024-09-17T04:07:12.411 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:12.411 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:07:12.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 msgr2=0x7f136807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 0x7f136807b0d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f1378033000 tx=0x7f1378004730 comp rx=0 tx=0).stop 2024-09-17T04:07:12.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 msgr2=0x7f138813ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f137400e790 tx=0x7f137400ec50 comp rx=0 tx=0).stop 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 shutdown_connections 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1368078bc0 0x7f136807b0d0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f138813fb20 0x7f13881f87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f13880cd590 0x7f138813ed70 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 --2- 172.21.15.129:0/1217877752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f13880746b0 0x7f13880b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 >> 172.21.15.129:0/1217877752 conn(0x7f138806cde0 msgr2=0x7f13880b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 shutdown_connections 2024-09-17T04:07:12.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.116+0000 7f138d600640 1 -- 172.21.15.129:0/1217877752 wait complete. 2024-09-17T04:07:12.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 -- 172.21.15.129:0/1977476734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe81558d0 msgr2=0x7fafe806f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 --2- 172.21.15.129:0/1977476734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe81558d0 0x7fafe806f3b0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fafe806ee90 tx=0x7fafdc02a560 comp rx=0 tx=0).stop 2024-09-17T04:07:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 -- 172.21.15.129:0/1977476734 shutdown_connections 2024-09-17T04:07:12.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 --2- 172.21.15.129:0/1977476734 >> v2:172.21.15.181:3300/0 conn(0x7fafe80d09d0 0x7fafe80af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 --2- 172.21.15.129:0/1977476734 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe81558d0 0x7fafe806f3b0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 -- 172.21.15.129:0/1977476734 >> 172.21.15.129:0/1977476734 conn(0x7fafe806cfc0 msgr2=0x7fafe806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 -- 172.21.15.129:0/1977476734 shutdown_connections 2024-09-17T04:07:12.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.635+0000 7fafee800640 1 -- 172.21.15.129:0/1977476734 wait complete. 2024-09-17T04:07:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.637+0000 7fafee800640 1 Processor -- start 2024-09-17T04:07:12.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.637+0000 7fafee800640 1 -- start start 2024-09-17T04:07:12.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 0x7fafe8149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafe81558d0 0x7fafe80b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8162710 con 0x7fafe8154960 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8162590 con 0x7fafe80af160 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafee800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fafe8162890 con 0x7fafe81558d0 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafede00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafed400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 0x7fafe8149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafede00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48970/0 (socket says 172.21.15.129:48970) 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafede00640 1 -- 172.21.15.129:0/2991814497 learned_addr learned my addr 172.21.15.129:0/2991814497 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafed400640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 0x7fafe8149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:43170/0 (socket says 172.21.15.129:43170) 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.639+0000 7fafeca00640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafe81558d0 0x7fafe80b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 -- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafe81558d0 msgr2=0x7fafe80b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafe81558d0 0x7fafe80b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 -- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 msgr2=0x7fafe8149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 0x7fafe8149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 -- 172.21.15.129:0/2991814497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe80b2a70 con 0x7fafe8154960 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafede00640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fafe000ea70 tx=0x7fafe000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe000ce40 con 0x7fafe8154960 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafe0004c10 con 0x7fafe8154960 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0005020 con 0x7fafe8154960 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.640+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe80cb7e0 con 0x7fafe8154960 2024-09-17T04:07:12.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.641+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe80b11f0 con 0x7fafe8154960 2024-09-17T04:07:12.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.642+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe8162d10 con 0x7fafe8154960 2024-09-17T04:07:12.643 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.643+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fafe00051c0 con 0x7fafe8154960 2024-09-17T04:07:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.644+0000 7fafe5600640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 0x7fafc807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:12.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.644+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fafe009bb30 con 0x7fafe8154960 2024-09-17T04:07:12.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.645+0000 7fafed400640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 0x7fafc807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:12.645 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.645+0000 7fafed400640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 0x7fafc807b0a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fafdc033000 tx=0x7fafdc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:12.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.649+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fafe0063830 con 0x7fafe8154960 2024-09-17T04:07:12.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.869+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafe806a140 con 0x7fafe8154960 2024-09-17T04:07:12.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.870+0000 7fafe5600640 1 -- 172.21.15.129:0/2991814497 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fafe0068260 con 0x7fafe8154960 2024-09-17T04:07:12.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:12.871 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:12.871 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:12.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 msgr2=0x7fafc807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 0x7fafc807b0a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fafdc033000 tx=0x7fafdc004730 comp rx=0 tx=0).stop 2024-09-17T04:07:12.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 msgr2=0x7fafe80b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:12.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fafe000ea70 tx=0x7fafe000ef30 comp rx=0 tx=0).stop 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 shutdown_connections 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafc8078b90 0x7fafc807b0a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafe8154960 0x7fafe80b1f30 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafe81558d0 0x7fafe80b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 --2- 172.21.15.129:0/2991814497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafe80af160 0x7fafe8149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 >> 172.21.15.129:0/2991814497 conn(0x7fafe806cfc0 msgr2=0x7fafe80b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 shutdown_connections 2024-09-17T04:07:12.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:12.873+0000 7fafee800640 1 -- 172.21.15.129:0/2991814497 wait complete. 2024-09-17T04:07:12.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:07:13.140 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"num_strays":50}} 2024-09-17T04:07:13.140 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 50 } } 2024-09-17T04:07:13.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:13.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 -- 172.21.15.129:0/2660983541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 msgr2=0x7ff668160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 --2- 172.21.15.129:0/2660983541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff668160750 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7ff660005fd0 tx=0x7ff66002a820 comp rx=0 tx=0).stop 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 -- 172.21.15.129:0/2660983541 shutdown_connections 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 --2- 172.21.15.129:0/2660983541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff668160750 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 --2- 172.21.15.129:0/2660983541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff6680d0a50 0x7ff668160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 -- 172.21.15.129:0/2660983541 >> 172.21.15.129:0/2660983541 conn(0x7ff66806cfc0 msgr2=0x7ff66806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 -- 172.21.15.129:0/2660983541 shutdown_connections 2024-09-17T04:07:13.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.359+0000 7ff667200640 1 -- 172.21.15.129:0/2660983541 wait complete. 2024-09-17T04:07:13.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.361+0000 7ff667200640 1 Processor -- start 2024-09-17T04:07:13.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.361+0000 7ff667200640 1 -- start start 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 0x7ff6680c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff66814fe50 0x7ff668150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680cef20 con 0x7ff6681458a0 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680ceda0 con 0x7ff668145e60 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff667200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff668160c90 con 0x7ff66814fe50 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff665e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff665e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:48990/0 (socket says 172.21.15.129:48990) 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff665e00640 1 -- 172.21.15.129:0/1162490597 learned_addr learned my addr 172.21.15.129:0/1162490597 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.363+0000 7ff65d400640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 0x7ff6680c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff666800640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff66814fe50 0x7ff668150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:13.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 -- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff66814fe50 msgr2=0x7ff668150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff66814fe50 0x7ff668150260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 -- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 msgr2=0x7ff6680c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 0x7ff6680c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 -- 172.21.15.129:0/1162490597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66815f7a0 con 0x7ff6681458a0 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff65d400640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 0x7ff6680c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:13.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff665e00640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7ff65800c3d0 tx=0x7ff65800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658011790 con 0x7ff6681458a0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff658017070 con 0x7ff6681458a0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff658011cc0 con 0x7ff6681458a0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6680ae5a0 con 0x7ff6681458a0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.364+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6680aeb60 con 0x7ff6681458a0 2024-09-17T04:07:13.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.365+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff668160210 con 0x7ff6681458a0 2024-09-17T04:07:13.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.367+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff65801c460 con 0x7ff6681458a0 2024-09-17T04:07:13.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.368+0000 7ff65ea00640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 0x7ff64407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:13.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.368+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7ff65809ac10 con 0x7ff6681458a0 2024-09-17T04:07:13.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.368+0000 7ff65d400640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 0x7ff64407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:13.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.369+0000 7ff65d400640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 0x7ff64407b0a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff650005fd0 tx=0x7ff65000f360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:13.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.372+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff658062f60 con 0x7ff6681458a0 2024-09-17T04:07:13.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.590+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff66806a140 con 0x7ff6681458a0 2024-09-17T04:07:13.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.591+0000 7ff65ea00640 1 -- 172.21.15.129:0/1162490597 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7ff658062860 con 0x7ff6681458a0 2024-09-17T04:07:13.591 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:13.591 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:13.592 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:13.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 msgr2=0x7ff64407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:13.593 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 0x7ff64407b0a0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7ff650005fd0 tx=0x7ff65000f360 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 msgr2=0x7ff6680af640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7ff65800c3d0 tx=0x7ff65800c890 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 shutdown_connections 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff644078b90 0x7ff64407b0a0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff66814fe50 0x7ff668150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff668145e60 0x7ff6680c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 --2- 172.21.15.129:0/1162490597 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff6681458a0 0x7ff6680af640 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 >> 172.21.15.129:0/1162490597 conn(0x7ff66806cfc0 msgr2=0x7ff6680b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 shutdown_connections 2024-09-17T04:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:13.593+0000 7ff667200640 1 -- 172.21.15.129:0/1162490597 wait complete. 2024-09-17T04:07:13.617 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:07:13.857 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":0}} 2024-09-17T04:07:13.858 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 0 } } 2024-09-17T04:07:13.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.18:0 scrub start / recursive,force 2024-09-17T04:07:14.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 -- 172.21.15.129:0/2467042938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c0b3f70 msgr2=0x7f338c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 --2- 172.21.15.129:0/2467042938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c0b3f70 0x7f338c0b49a0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f337c009830 tx=0x7f337c02a480 comp rx=0 tx=0).stop 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 -- 172.21.15.129:0/2467042938 shutdown_connections 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 --2- 172.21.15.129:0/2467042938 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 0x7f338c0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 --2- 172.21.15.129:0/2467042938 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c0b3f70 0x7f338c0b49a0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 -- 172.21.15.129:0/2467042938 >> 172.21.15.129:0/2467042938 conn(0x7f338c0ccc40 msgr2=0x7f338c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 -- 172.21.15.129:0/2467042938 shutdown_connections 2024-09-17T04:07:14.084 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.083+0000 7f3392000640 1 -- 172.21.15.129:0/2467042938 wait complete. 2024-09-17T04:07:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.085+0000 7f3392000640 1 Processor -- start 2024-09-17T04:07:14.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.086+0000 7f3392000640 1 -- start start 2024-09-17T04:07:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 0x7f338c156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c157380 0x7f338c09aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f338c0cc8e0 con 0x7f338c074580 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f338c0cc760 con 0x7f338c157380 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f3392000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f338c166f20 con 0x7f338c0cba90 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49736/0 (socket says 172.21.15.129:49736) 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f338b400640 1 -- 172.21.15.129:0/1604673077 learned_addr learned my addr 172.21.15.129:0/1604673077 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f338be00640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c157380 0x7f338c09aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.087+0000 7f338aa00640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 0x7f338c156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 -- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 msgr2=0x7f338c156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 0x7f338c156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 -- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c157380 msgr2=0x7f338c09aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c157380 0x7f338c09aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 -- 172.21.15.129:0/1604673077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338c15ad90 con 0x7f338c074580 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f338b400640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f337c009950 tx=0x7f337c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f3383400640 1 -- 172.21.15.129:0/1604673077 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c005c90 con 0x7f338c074580 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f3383400640 1 -- 172.21.15.129:0/1604673077 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f337c031e50 con 0x7f338c074580 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338c15b070 con 0x7f338c074580 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.088+0000 7f3383400640 1 -- 172.21.15.129:0/1604673077 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c005260 con 0x7f338c074580 2024-09-17T04:07:14.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.089+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f338c15b630 con 0x7f338c074580 2024-09-17T04:07:14.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.091+0000 7f3383400640 1 -- 172.21.15.129:0/1604673077 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f337c005400 con 0x7f338c074580 2024-09-17T04:07:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.092+0000 7f3383400640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 0x7f336c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.092+0000 7f3383400640 1 -- 172.21.15.129:0/1604673077 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f337c0bbe50 con 0x7f338c074580 2024-09-17T04:07:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.092+0000 7f338aa00640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 0x7f336c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.092+0000 7f3399083740 1 Processor -- start 2024-09-17T04:07:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.092+0000 7f3399083740 1 -- start start 2024-09-17T04:07:14.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.093+0000 7f338aa00640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 0x7f336c07b0a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3378007fd0 tx=0x7f3378001390 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55910d5e9f40 con 0x55910d2cf410 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55910d5d9370 con 0x55910d2ae730 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55910d5dd180 con 0x55910d2a6550 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49744/0 (socket says 172.21.15.129:49744) 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 -- 172.21.15.129:0/1588715885 learned_addr learned my addr 172.21.15.129:0/1588715885 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338be00640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338aa00640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 -- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 msgr2=0x55910d6a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 -- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 msgr2=0x55910d6a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338b400640 1 -- 172.21.15.129:0/1588715885 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55910d2cb470 con 0x55910d2cf410 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.094+0000 7f338be00640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.095+0000 7f338aa00640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.095+0000 7f338b400640 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f337c080dc0 tx=0x7f337c0802d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66de8fee9ffe5834 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.095+0000 7f3377400640 1 -- 172.21.15.129:0/1588715885 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c048040 con 0x55910d2cf410 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.095+0000 7f3377400640 1 -- 172.21.15.129:0/1588715885 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f337c088030 con 0x55910d2cf410 2024-09-17T04:07:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.095+0000 7f3377400640 1 -- 172.21.15.129:0/1588715885 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c083b50 con 0x55910d2cf410 2024-09-17T04:07:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 -- 172.21.15.129:0/1588715885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 msgr2=0x55910d6be190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f337c080dc0 tx=0x7f337c0802d0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 -- 172.21.15.129:0/1588715885 shutdown_connections 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 --2- 172.21.15.129:0/1588715885 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6be190 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 -- 172.21.15.129:0/1588715885 >> 172.21.15.129:0/1588715885 conn(0x55910d2c3560 msgr2=0x55910d677080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 -- 172.21.15.129:0/1588715885 shutdown_connections 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.096+0000 7f3399083740 1 -- 172.21.15.129:0/1588715885 wait complete. 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.097+0000 7f3399083740 1 -- start start 2024-09-17T04:07:14.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.097+0000 7f3399083740 1 Processor -- start 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55910d3497c0 con 0x55910d2cf410 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55910d57eae0 con 0x55910d2ae730 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55910d5e9f40 con 0x55910d2a6550 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f3399083740 10 client.0 _ll_register_callbacks cb 0x55910d58e5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:14.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49752/0 (socket says 172.21.15.129:49752) 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338be00640 1 -- 172.21.15.129:0/3448298539 learned_addr learned my addr 172.21.15.129:0/3448298539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338aa00640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338b400640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338be00640 1 -- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 msgr2=0x55910d6ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.099+0000 7f338be00640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f338be00640 1 -- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 msgr2=0x55910d6b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f338be00640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f338be00640 1 -- 172.21.15.129:0/3448298539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55910d3af7e0 con 0x55910d2cf410 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f338be00640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f3384005fd0 tx=0x7f3384012db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526a12f81a5fbed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f336be00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384003d80 con 0x55910d2cf410 2024-09-17T04:07:14.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f3384004b20 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f337c088030 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f3384004d20 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f3399083740 10 client.12093 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.100+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x55910d40ff70 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f337c088030 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f3384024050 con 0x55910d2cf410 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 10 client.12093 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 10 client.12093 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 20 client.12093 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d28e1f0 0x55910d6b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 4 client.12093 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55910d5ebe90 con 0x55910d28e1f0 2024-09-17T04:07:14.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.101+0000 7f338b400640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d28e1f0 0x55910d6b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.102+0000 7f338b400640 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d28e1f0 0x55910d6b6230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88bbc9e89e9cae81 server_cookie=f92fd83a837e4226 in_seq=0 out_seq=0 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.102+0000 7f336be00640 10 client.12093 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.102+0000 7f336be00640 1 -- 172.21.15.129:0/3448298539 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55910d5ebe90 con 0x55910d28e1f0 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.102+0000 7f336be00640 10 client.12093 handle_command_reply: tid=0 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 client.12093 shutdown 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 msgr2=0x55910d6760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f3384005fd0 tx=0x7f3384012db0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 shutdown_connections 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d28e1f0 0x55910d6b6230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2cf410 0x55910d6760b0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d6b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f336be00640 0 client.12093 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 --2- 172.21.15.129:0/3448298539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d6ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 >> 172.21.15.129:0/3448298539 conn(0x55910d2c3560 msgr2=0x55910d670430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 shutdown_connections 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 1 -- 172.21.15.129:0/3448298539 wait complete. 2024-09-17T04:07:14.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.103+0000 7f3399083740 20 client.12093 trim_cache size 0 max 16384 2024-09-17T04:07:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.121+0000 7f3399083740 1 Processor -- start 2024-09-17T04:07:14.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.121+0000 7f3399083740 1 -- start start 2024-09-17T04:07:14.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d670dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2cf410 0x55910d6a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f338c2020d0 con 0x55910d2ae730 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f338c201150 con 0x55910d2cf410 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f338c0cf7b0 con 0x55910d2a6550 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49756/0 (socket says 172.21.15.129:49756) 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 -- 172.21.15.129:0/2784255708 learned_addr learned my addr 172.21.15.129:0/2784255708 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338be00640 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2cf410 0x55910d6a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338b400640 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d670dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 -- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 msgr2=0x55910d670dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d670dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 -- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2cf410 msgr2=0x55910d6a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2cf410 0x55910d6a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.123+0000 7f338aa00640 1 -- 172.21.15.129:0/2784255708 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55910d6bc140 con 0x55910d2ae730 2024-09-17T04:07:14.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.124+0000 7f338aa00640 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f337800fa50 tx=0x7f3378015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb60174ad5b5af1 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.124+0000 7f3376a00640 1 -- 172.21.15.129:0/2784255708 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378014ca0 con 0x55910d2ae730 2024-09-17T04:07:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.124+0000 7f3376a00640 1 -- 172.21.15.129:0/2784255708 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3378028070 con 0x55910d2ae730 2024-09-17T04:07:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.124+0000 7f3376a00640 1 -- 172.21.15.129:0/2784255708 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33780236a0 con 0x55910d2ae730 2024-09-17T04:07:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 -- 172.21.15.129:0/2784255708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 msgr2=0x55910d6a0e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f337800fa50 tx=0x7f3378015000 comp rx=0 tx=0).stop 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 -- 172.21.15.129:0/2784255708 shutdown_connections 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2cf410 0x55910d6a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2ae730 0x55910d6a0e60 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 --2- 172.21.15.129:0/2784255708 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2a6550 0x55910d670dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 -- 172.21.15.129:0/2784255708 >> 172.21.15.129:0/2784255708 conn(0x55910d2c3560 msgr2=0x55910d6783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 -- 172.21.15.129:0/2784255708 shutdown_connections 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.125+0000 7f3399083740 1 -- 172.21.15.129:0/2784255708 wait complete. 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.126+0000 7f3399083740 1 -- start start 2024-09-17T04:07:14.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.126+0000 7f3399083740 1 Processor -- start 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d674da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2cf410 0x55910d68ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f338c1f9780 con 0x55910d2a6550 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55910d675780 con 0x55910d2ae730 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55910d6759d0 con 0x55910d2cf410 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f3399083740 10 client.0 _ll_register_callbacks cb 0x7f3384009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d674da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49768/0 (socket says 172.21.15.129:49768) 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 -- 172.21.15.129:0/3829110282 learned_addr learned my addr 172.21.15.129:0/3829110282 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338be00640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2cf410 0x55910d68ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 -- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2cf410 msgr2=0x55910d68ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2cf410 0x55910d68ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 -- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 msgr2=0x55910d674da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d674da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.128+0000 7f338b400640 1 -- 172.21.15.129:0/3829110282 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55910d318720 con 0x55910d2a6550 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f338b400640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f337c009920 tx=0x7f337c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26211f0d78778cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c0a2060 con 0x55910d2a6550 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f337c0881f0 con 0x55910d2a6550 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3378028070 con 0x55910d2a6550 2024-09-17T04:07:14.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=240) ==== 24+0+0 (secure 0 0 0) 0x7f337c092050 con 0x55910d2a6550 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 10 client.12099 fetch_fsmap learned FSMap version 240 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=240,monmap=2+}) -- 0x55910d73fb90 con 0x55910d2a6550 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3378028070 con 0x55910d2a6550 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 240) ==== 3502+0+0 (secure 0 0 0) 0x7f337c083db0 con 0x55910d2a6550 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 10 client.12099 fetch_fsmap finished waiting for FSMap version 240 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 10 client.12099 resolve_mds: resolved 18:0 to role '18:0' aka daemon mds.d 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 20 client.12099 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d68f110 0x55910d73b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 4 client.12099 mds_command: new command op to 11772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "force"]}] 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.129+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 --> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "force"]}) -- 0x55910d5e7940 con 0x55910d68f110 2024-09-17T04:07:14.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.130+0000 7f338aa00640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d68f110 0x55910d73b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.130+0000 7f336be00640 10 client.12099 ms_handle_connect on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.130+0000 7f338aa00640 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d68f110 0x55910d73b7e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32679b4396672303 server_cookie=70e2c3542b822e51 in_seq=0 out_seq=0 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f336be00640 1 -- 172.21.15.129:0/3829110282 <== mds.0 v2:172.21.15.181:6835/2055116414 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55910d5e7940 con 0x55910d68f110 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f336be00640 10 client.12099 handle_command_reply: tid=0 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 client.12099 shutdown 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 msgr2=0x55910d721230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f337c009920 tx=0x7f337c005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 shutdown_connections 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:6835/2055116414,v1:172.21.15.181:6837/2055116414] conn(0x55910d68f110 0x55910d73b7e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55910d2cf410 0x55910d68ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f336be00640 0 client.12099 ms_handle_reset on v2:172.21.15.181:6835/2055116414 2024-09-17T04:07:14.134 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55910d2ae730 0x55910d674da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 --2- 172.21.15.129:0/3829110282 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55910d2a6550 0x55910d721230 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 >> 172.21.15.129:0/3829110282 conn(0x55910d2c3560 msgr2=0x55910d6783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 shutdown_connections 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 1 -- 172.21.15.129:0/3829110282 wait complete. 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.132+0000 7f3399083740 20 client.12099 trim_cache size 0 max 16384 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stdout: "return_code": 0, 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stdout: "scrub_tag": "52597ee1-7ae0-4558-9fee-1c67e1fded3a", 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stdout: "mode": "asynchronous" 2024-09-17T04:07:14.135 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 msgr2=0x7f336c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 0x7f336c07b0a0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3378007fd0 tx=0x7f3378001390 comp rx=0 tx=0).stop 2024-09-17T04:07:14.136 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 msgr2=0x7f338c15be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7f337c009950 tx=0x7f337c033c20 comp rx=0 tx=0).stop 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 shutdown_connections 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f336c078b90 0x7f336c07b0a0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f338c157380 0x7f338c09aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f338c0cba90 0x7f338c156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 --2- 172.21.15.129:0/1604673077 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f338c074580 0x7f338c15be30 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 >> 172.21.15.129:0/1604673077 conn(0x7f338c0ccc40 msgr2=0x7f338c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 shutdown_connections 2024-09-17T04:07:14.137 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.133+0000 7f3392000640 1 -- 172.21.15.129:0/1604673077 wait complete. 2024-09-17T04:07:14.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 -- 172.21.15.129:0/165369072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade00705a0 msgr2=0x7fade00b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 --2- 172.21.15.129:0/165369072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade00705a0 0x7fade00b4140 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fade0072080 tx=0x7fadd002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 -- 172.21.15.129:0/165369072 shutdown_connections 2024-09-17T04:07:14.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 --2- 172.21.15.129:0/165369072 >> v2:172.21.15.181:3300/0 conn(0x7fade006eed0 0x7fade00ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 --2- 172.21.15.129:0/165369072 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade00705a0 0x7fade00b4140 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 -- 172.21.15.129:0/165369072 >> 172.21.15.129:0/165369072 conn(0x7fade006d410 msgr2=0x7fade00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 -- 172.21.15.129:0/165369072 shutdown_connections 2024-09-17T04:07:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.377+0000 7fade6400640 1 -- 172.21.15.129:0/165369072 wait complete. 2024-09-17T04:07:14.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.380+0000 7fade6400640 1 Processor -- start 2024-09-17T04:07:14.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.380+0000 7fade6400640 1 -- start start 2024-09-17T04:07:14.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade006f2e0 0x7fade015c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 0x7fade00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fade0161250 con 0x7fade006eed0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fade01610d0 con 0x7fade006f2e0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7fade6400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fade01613d0 con 0x7fade00705a0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7faddf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7faddf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49782/0 (socket says 172.21.15.129:49782) 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.381+0000 7faddf400640 1 -- 172.21.15.129:0/2771891682 learned_addr learned my addr 172.21.15.129:0/2771891682 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7fadd6a00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade006f2e0 0x7fade015c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddfe00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 0x7fade00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 -- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 msgr2=0x7fade00b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 0x7fade00b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 -- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade006f2e0 msgr2=0x7fade015c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade006f2e0 0x7fade015c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 -- 172.21.15.129:0/2771891682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade00c7ea0 con 0x7fade006eed0 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddfe00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 0x7fade00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7faddf400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fadd002abe0 tx=0x7fadd00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd003e030 con 0x7fade006eed0 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fadd0003db0 con 0x7fade006eed0 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade01f8070 con 0x7fade006eed0 2024-09-17T04:07:14.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.382+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0035040 con 0x7fade006eed0 2024-09-17T04:07:14.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.383+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade01f7ab0 con 0x7fade006eed0 2024-09-17T04:07:14.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.384+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fade00c8070 con 0x7fade006eed0 2024-09-17T04:07:14.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.385+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fadd0041030 con 0x7fade006eed0 2024-09-17T04:07:14.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.386+0000 7fadd7e00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 0x7fadbc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:14.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.386+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fadd00bb860 con 0x7fade006eed0 2024-09-17T04:07:14.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.386+0000 7fadd6a00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 0x7fadbc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:14.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.387+0000 7fadd6a00640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 0x7fadbc07b0d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fade01f70c0 tx=0x7fadc8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:14.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.390+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fadd0083550 con 0x7fade006eed0 2024-09-17T04:07:14.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.608+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fade00cd5a0 con 0x7fade006eed0 2024-09-17T04:07:14.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.608+0000 7fadd7e00640 1 -- 172.21.15.129:0/2771891682 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fadd0087f80 con 0x7fade006eed0 2024-09-17T04:07:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:14.609 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:14.609 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 msgr2=0x7fadbc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 0x7fadbc07b0d0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fade01f70c0 tx=0x7fadc8005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 msgr2=0x7fade00c66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fadd002abe0 tx=0x7fadd00038a0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 shutdown_connections 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fadbc078bc0 0x7fadbc07b0d0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fade00705a0 0x7fade00b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fade006f2e0 0x7fade015c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 --2- 172.21.15.129:0/2771891682 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fade006eed0 0x7fade00c66e0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 >> 172.21.15.129:0/2771891682 conn(0x7fade006d410 msgr2=0x7fade00ae8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 shutdown_connections 2024-09-17T04:07:14.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:14.611+0000 7fade6400640 1 -- 172.21.15.129:0/2771891682 wait complete. 2024-09-17T04:07:14.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache num_strays 2024-09-17T04:07:14.880 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"num_strays":0}} 2024-09-17T04:07:14.880 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "num_strays": 0 } } 2024-09-17T04:07:14.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:15.101 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 -- 172.21.15.129:0/2362682244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0280cd390 msgr2=0x7fb02806f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2362682244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0280cd390 0x7fb02806f640 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb020005fd0 tx=0x7fb02002aaa0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 -- 172.21.15.129:0/2362682244 shutdown_connections 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2362682244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0280cd390 0x7fb02806f640 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2362682244 >> v2:172.21.15.181:3300/0 conn(0x7fb0280af4c0 0x7fb02815ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 -- 172.21.15.129:0/2362682244 >> 172.21.15.129:0/2362682244 conn(0x7fb02806d410 msgr2=0x7fb0280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 -- 172.21.15.129:0/2362682244 shutdown_connections 2024-09-17T04:07:15.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.101+0000 7fb02fa00640 1 -- 172.21.15.129:0/2362682244 wait complete. 2024-09-17T04:07:15.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.103+0000 7fb02fa00640 1 Processor -- start 2024-09-17T04:07:15.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.103+0000 7fb02fa00640 1 -- start start 2024-09-17T04:07:15.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 0x7fb02809c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:15.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:15.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0281e9960 0x7fb0281e9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:15.105 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb02806fd00 con 0x7fb02815b340 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb02806fb80 con 0x7fb0280af4c0 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb02806fe80 con 0x7fb0281e9960 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb01fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb01fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49792/0 (socket says 172.21.15.129:49792) 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb01fe00640 1 -- 172.21.15.129:0/2003316331 learned_addr learned my addr 172.21.15.129:0/2003316331 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02d000640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 0x7fb02809c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:15.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.105+0000 7fb02da00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0281e9960 0x7fb0281e9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 -- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0281e9960 msgr2=0x7fb0281e9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0281e9960 0x7fb0281e9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 -- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 msgr2=0x7fb02809c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 0x7fb02809c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 -- 172.21.15.129:0/2003316331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb02814aaa0 con 0x7fb02815b340 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb02d000640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 0x7fb02809c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb01fe00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c390 tx=0x7fb01000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010017070 con 0x7fb02815b340 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0100053e0 con 0x7fb02815b340 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010003850 con 0x7fb02815b340 2024-09-17T04:07:15.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.106+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb028149d00 con 0x7fb02815b340 2024-09-17T04:07:15.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.107+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb02814a2a0 con 0x7fb02815b340 2024-09-17T04:07:15.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.108+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0281f4460 con 0x7fb02815b340 2024-09-17T04:07:15.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.109+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb01001a070 con 0x7fb02815b340 2024-09-17T04:07:15.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.110+0000 7fb025600640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 0x7fb00407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:15.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.110+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fb010099fb0 con 0x7fb02815b340 2024-09-17T04:07:15.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.110+0000 7fb02d000640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 0x7fb00407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:15.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.111+0000 7fb02d000640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 0x7fb00407b0a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb01800afb0 tx=0x7fb01800c040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:15.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.115+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb010061dc0 con 0x7fb02815b340 2024-09-17T04:07:15.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.333+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb028070300 con 0x7fb02815b340 2024-09-17T04:07:15.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.333+0000 7fb025600640 1 -- 172.21.15.129:0/2003316331 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fb0100667f0 con 0x7fb02815b340 2024-09-17T04:07:15.333 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:15.334 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:15.334 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.335+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 msgr2=0x7fb00407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.335+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 0x7fb00407b0a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fb01800afb0 tx=0x7fb01800c040 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.335+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 msgr2=0x7fb02809cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.335+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c390 tx=0x7fb01000c790 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 shutdown_connections 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb004078b90 0x7fb00407b0a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0281e9960 0x7fb0281e9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb02815b340 0x7fb02809cc00 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 --2- 172.21.15.129:0/2003316331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0280af4c0 0x7fb02809c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:15.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 >> 172.21.15.129:0/2003316331 conn(0x7fb02806d410 msgr2=0x7fb028073190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:15.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 shutdown_connections 2024-09-17T04:07:15.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:15.336+0000 7fb02fa00640 1 -- 172.21.15.129:0/2003316331 wait complete. 2024-09-17T04:07:15.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json perf dump mds_cache strays_reintegrated 2024-09-17T04:07:15.597 INFO:teuthology.orchestra.run.smithi181.stdout:{"mds_cache":{"strays_reintegrated":50}} 2024-09-17T04:07:15.597 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_reintegrated": 50 } } 2024-09-17T04:07:15.598 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:07:15.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:07:15.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:07:15.787 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:07:15.787 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:07:15.787 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:15.787 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:15.824 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:15.824 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:15.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 -- 172.21.15.129:0/4055106058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 msgr2=0x7f17f80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 --2- 172.21.15.129:0/4055106058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80b4140 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009860 tx=0x7f17e802a5a0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 -- 172.21.15.129:0/4055106058 shutdown_connections 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 --2- 172.21.15.129:0/4055106058 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80b4140 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 -- 172.21.15.129:0/4055106058 >> 172.21.15.129:0/4055106058 conn(0x7f17f806d410 msgr2=0x7f17f80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:16.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.158+0000 7f17fd800640 1 -- 172.21.15.129:0/4055106058 shutdown_connections 2024-09-17T04:07:16.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.159+0000 7f17fd800640 1 -- 172.21.15.129:0/4055106058 wait complete. 2024-09-17T04:07:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.161+0000 7f17fd800640 1 Processor -- start 2024-09-17T04:07:16.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.161+0000 7f17fd800640 1 -- start start 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 0x7f17f81f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17f806f2e0 con 0x7f17f806eed0 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81612e0 con 0x7f17f815b000 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17fd800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17f806f460 con 0x7f17f80705a0 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.162+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49808/0 (socket says 172.21.15.129:49808) 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 -- 172.21.15.129:0/4105229798 learned_addr learned my addr 172.21.15.129:0/4105229798 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f7400640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 0x7f17f81f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6000640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 -- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 msgr2=0x7f17f80cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 -- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 msgr2=0x7f17f81f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 0x7f17f81f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 -- 172.21.15.129:0/4105229798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f80cac50 con 0x7f17f806eed0 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6000640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f7400640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 0x7f17f81f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.163+0000 7f17f6a00640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009980 tx=0x7f17e8033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.164+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005c90 con 0x7f17f806eed0 2024-09-17T04:07:16.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.164+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f17e8031e50 con 0x7f17f806eed0 2024-09-17T04:07:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.164+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005260 con 0x7f17f806eed0 2024-09-17T04:07:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.164+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f80caf30 con 0x7f17f806eed0 2024-09-17T04:07:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.164+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f80cb4f0 con 0x7f17f806eed0 2024-09-17T04:07:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.165+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f80b44a0 con 0x7f17f806eed0 2024-09-17T04:07:16.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.166+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f17e8005400 con 0x7f17f806eed0 2024-09-17T04:07:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.167+0000 7f17eea00640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 0x7f17d807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.167+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f17e80bbd40 con 0x7f17f806eed0 2024-09-17T04:07:16.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.168+0000 7f17f6000640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 0x7f17d807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.168+0000 7f17f6000640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 0x7f17d807b0a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4001780 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:16.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.172+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f17f80b44a0 con 0x7f17f806eed0 2024-09-17T04:07:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.364+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f17f8076760 con 0x7f17f806eed0 2024-09-17T04:07:16.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.364+0000 7f17eea00640 1 -- 172.21.15.129:0/4105229798 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v248) ==== 80+0+122 (secure 0 0 0) 0x7f17f8076760 con 0x7f17f806eed0 2024-09-17T04:07:16.365 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6835/968815434 2024-09-18T04:07:05.543219+0000 2024-09-17T04:07:16.365 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6833/968815434 2024-09-18T04:07:05.543219+0000 2024-09-17T04:07:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T04:07:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 msgr2=0x7f17d807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 0x7f17d807b0a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4001780 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 msgr2=0x7f17f81f6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009980 tx=0x7f17e8033c20 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 shutdown_connections 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f17d8078b90 0x7f17d807b0a0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17f815b000 0x7f17f81f6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17f80705a0 0x7f17f80cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 --2- 172.21.15.129:0/4105229798 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17f806eed0 0x7f17f81f6180 unknown :-1 s=CLOSED pgs=2714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 >> 172.21.15.129:0/4105229798 conn(0x7f17f806d410 msgr2=0x7f17f81609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 shutdown_connections 2024-09-17T04:07:16.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.367+0000 7f17fd800640 1 -- 172.21.15.129:0/4105229798 wait complete. 2024-09-17T04:07:16.391 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:07:16.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:16.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:16.453 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:07:16.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:16.453 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:07:16.489 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:07:16.490 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:07:16.490 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:16.526 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:07:16.526 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:07:16.526 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:16.526 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:16.565 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:07:16.565 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:07:16.566 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:07:16.566 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:16.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:07:16.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 -- 172.21.15.129:0/274987618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 msgr2=0x7f92ac0cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 --2- 172.21.15.129:0/274987618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac0cf360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f92a4005fd0 tx=0x7f92a402a820 comp rx=0 tx=0).stop 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 -- 172.21.15.129:0/274987618 shutdown_connections 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 --2- 172.21.15.129:0/274987618 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac0cf360 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 --2- 172.21.15.129:0/274987618 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac1602b0 0x7f92ac0cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 -- 172.21.15.129:0/274987618 >> 172.21.15.129:0/274987618 conn(0x7f92ac069150 msgr2=0x7f92ac069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 -- 172.21.15.129:0/274987618 shutdown_connections 2024-09-17T04:07:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.849+0000 7f92b2200640 1 -- 172.21.15.129:0/274987618 wait complete. 2024-09-17T04:07:16.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.852+0000 7f92b2200640 1 Processor -- start 2024-09-17T04:07:16.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.852+0000 7f92b2200640 1 -- start start 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac15a220 0x7f92ac0b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac1f4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac0d0320 con 0x7f92ac1602b0 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac0d01a0 con 0x7f92ac1606c0 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92b2200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac0d04a0 con 0x7f92ac15a220 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.853+0000 7f92abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49812/0 (socket says 172.21.15.129:49812) 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 -- 172.21.15.129:0/4172409118 learned_addr learned my addr 172.21.15.129:0/4172409118 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:16.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92b1800640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac1f4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92b0e00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac15a220 0x7f92ac0b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 -- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac15a220 msgr2=0x7f92ac0b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac15a220 0x7f92ac0b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 -- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 msgr2=0x7f92ac1f4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac1f4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 -- 172.21.15.129:0/4172409118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac0b69e0 con 0x7f92ac1602b0 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92b1800640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac1f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.854+0000 7f92abe00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f929800e910 tx=0x7f929800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.855+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929800cd80 con 0x7f92ac1602b0 2024-09-17T04:07:16.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.855+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9298010430 con 0x7f92ac1602b0 2024-09-17T04:07:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.855+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298010860 con 0x7f92ac1602b0 2024-09-17T04:07:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.855+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac0b6cc0 con 0x7f92ac1602b0 2024-09-17T04:07:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.855+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac0b7280 con 0x7f92ac1602b0 2024-09-17T04:07:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.856+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92ac0d0a30 con 0x7f92ac1602b0 2024-09-17T04:07:16.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.857+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9298010a00 con 0x7f92ac1602b0 2024-09-17T04:07:16.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.858+0000 7f92a8c00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 0x7f928c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:16.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.858+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f929809b840 con 0x7f92ac1602b0 2024-09-17T04:07:16.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.859+0000 7f92b0e00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 0x7f928c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:16.859 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.859+0000 7f92b0e00640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 0x7f928c07b0a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f92a0007fd0 tx=0x7f92a0001370 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:16.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:16.863+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9298063540 con 0x7f92ac1602b0 2024-09-17T04:07:17.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.078+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92ac06a130 con 0x7f92ac1602b0 2024-09-17T04:07:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.079+0000 7f92a8c00640 1 -- 172.21.15.129:0/4172409118 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7f9298067f70 con 0x7f92ac1602b0 2024-09-17T04:07:17.079 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:17.079 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:17.079 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 msgr2=0x7f928c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 0x7f928c07b0a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f92a0007fd0 tx=0x7f92a0001370 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 msgr2=0x7f92ac1f45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f929800e910 tx=0x7f929800edd0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 shutdown_connections 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f928c078b90 0x7f928c07b0a0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92ac1606c0 0x7f92ac1f4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92ac1602b0 0x7f92ac1f45b0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 --2- 172.21.15.129:0/4172409118 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92ac15a220 0x7f92ac0b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 >> 172.21.15.129:0/4172409118 conn(0x7f92ac069150 msgr2=0x7f92ac15efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 shutdown_connections 2024-09-17T04:07:17.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.081+0000 7f92b2200640 1 -- 172.21.15.129:0/4172409118 wait complete. 2024-09-17T04:07:17.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:17.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 -- 172.21.15.129:0/4240911325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5000d0800 msgr2=0x7fd50015cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 --2- 172.21.15.129:0/4240911325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5000d0800 0x7fd50015cbd0 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8005fd0 tx=0x7fd4f802a820 comp rx=0 tx=0).stop 2024-09-17T04:07:17.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 -- 172.21.15.129:0/4240911325 shutdown_connections 2024-09-17T04:07:17.328 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 --2- 172.21.15.129:0/4240911325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd5000d0800 0x7fd50015cbd0 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 --2- 172.21.15.129:0/4240911325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd500160cf0 0x7fd5000d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 -- 172.21.15.129:0/4240911325 >> 172.21.15.129:0/4240911325 conn(0x7fd50006cfc0 msgr2=0x7fd50006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:17.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.327+0000 7fd4ff200640 1 -- 172.21.15.129:0/4240911325 shutdown_connections 2024-09-17T04:07:17.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.328+0000 7fd4ff200640 1 -- 172.21.15.129:0/4240911325 wait complete. 2024-09-17T04:07:17.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.330+0000 7fd4ff200640 1 Processor -- start 2024-09-17T04:07:17.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.330+0000 7fd4ff200640 1 -- start start 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5000cfb60 0x7fd500074790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5000d0800 0x7fd50015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd50015d4a0 con 0x7fd500160cf0 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd50015d320 con 0x7fd5000d0800 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4ff200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd50015d620 con 0x7fd5000cfb60 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.331+0000 7fd4fe800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49832/0 (socket says 172.21.15.129:49832) 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 -- 172.21.15.129:0/1751341051 learned_addr learned my addr 172.21.15.129:0/1751341051 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4f5400640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5000d0800 0x7fd50015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fde00640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5000cfb60 0x7fd500074790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 -- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5000cfb60 msgr2=0x7fd500074790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5000cfb60 0x7fd500074790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 -- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5000d0800 msgr2=0x7fd50015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5000d0800 0x7fd50015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 -- 172.21.15.129:0/1751341051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd500159ca0 con 0x7fd500160cf0 2024-09-17T04:07:17.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.332+0000 7fd4fe800640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009b30 tx=0x7fd4f8032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:17.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.333+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8037040 con 0x7fd500160cf0 2024-09-17T04:07:17.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.333+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd4f8042070 con 0x7fd500160cf0 2024-09-17T04:07:17.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.333+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80042a0 con 0x7fd500160cf0 2024-09-17T04:07:17.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.333+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5000ae970 con 0x7fd500160cf0 2024-09-17T04:07:17.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.333+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd500073d10 con 0x7fd500160cf0 2024-09-17T04:07:17.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.334+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd500073f70 con 0x7fd500160cf0 2024-09-17T04:07:17.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.335+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd4f8004440 con 0x7fd500160cf0 2024-09-17T04:07:17.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.336+0000 7fd4f6a00640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 0x7fd4dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.336+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fd4f80bbba0 con 0x7fd500160cf0 2024-09-17T04:07:17.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.336+0000 7fd4fde00640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 0x7fd4dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.337+0000 7fd4fde00640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 0x7fd4dc07b0a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0007fd0 tx=0x7fd4f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:17.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.340+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd4f8083ee0 con 0x7fd500160cf0 2024-09-17T04:07:17.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.556+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd50015cbd0 con 0x7fd500160cf0 2024-09-17T04:07:17.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.557+0000 7fd4f6a00640 1 -- 172.21.15.129:0/1751341051 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fd4f80837e0 con 0x7fd500160cf0 2024-09-17T04:07:17.557 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:17.557 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:17.558 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 msgr2=0x7fd4dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 0x7fd4dc07b0a0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd4f0007fd0 tx=0x7fd4f0009040 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 msgr2=0x7fd50015ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8009b30 tx=0x7fd4f8032c90 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 shutdown_connections 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd4dc078b90 0x7fd4dc07b0a0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd500160cf0 0x7fd50015ba00 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5000d0800 0x7fd50015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 --2- 172.21.15.129:0/1751341051 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd5000cfb60 0x7fd500074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.559+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 >> 172.21.15.129:0/1751341051 conn(0x7fd50006cfc0 msgr2=0x7fd500156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:17.560 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.560+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 shutdown_connections 2024-09-17T04:07:17.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.560+0000 7fd4ff200640 1 -- 172.21.15.129:0/1751341051 wait complete. 2024-09-17T04:07:17.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 -- 172.21.15.129:0/2836893899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 msgr2=0x7fc6b40ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 --2- 172.21.15.129:0/2836893899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40ccc80 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4072e00 tx=0x7fc6a002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 -- 172.21.15.129:0/2836893899 shutdown_connections 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 --2- 172.21.15.129:0/2836893899 >> v2:172.21.15.181:3300/0 conn(0x7fc6b40cd1c0 0x7fc6b41609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 --2- 172.21.15.129:0/2836893899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40ccc80 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 -- 172.21.15.129:0/2836893899 >> 172.21.15.129:0/2836893899 conn(0x7fc6b406d410 msgr2=0x7fc6b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 -- 172.21.15.129:0/2836893899 shutdown_connections 2024-09-17T04:07:17.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.802+0000 7fc6bae00640 1 -- 172.21.15.129:0/2836893899 wait complete. 2024-09-17T04:07:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.805+0000 7fc6bae00640 1 Processor -- start 2024-09-17T04:07:17.805 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.805+0000 7fc6bae00640 1 -- start start 2024-09-17T04:07:17.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6b40705a0 0x7fc6b41f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b40cd1c0 0x7fc6b40c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b4160260 con 0x7fc6b40b4140 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b41600e0 con 0x7fc6b40cd1c0 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6bae00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b4160f00 con 0x7fc6b40705a0 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6b3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6b3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49854/0 (socket says 172.21.15.129:49854) 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.806+0000 7fc6b3400640 1 -- 172.21.15.129:0/591332198 learned_addr learned my addr 172.21.15.129:0/591332198 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b8e00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b40cd1c0 0x7fc6b40c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3e00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6b40705a0 0x7fc6b41f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 -- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6b40705a0 msgr2=0x7fc6b41f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6b40705a0 0x7fc6b41f5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 -- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b40cd1c0 msgr2=0x7fc6b40c4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b40cd1c0 0x7fc6b40c4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 -- 172.21.15.129:0/591332198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b41f64c0 con 0x7fc6b40b4140 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6b3400640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0034040 tx=0x7fc6a0003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a003e030 con 0x7fc6b40b4140 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc6a0003e00 con 0x7fc6b40b4140 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0035040 con 0x7fc6b40b4140 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.807+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b415c5c0 con 0x7fc6b40b4140 2024-09-17T04:07:17.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.808+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b40cc680 con 0x7fc6b40b4140 2024-09-17T04:07:17.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.809+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6b406ee50 con 0x7fc6b40b4140 2024-09-17T04:07:17.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.810+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc6a0041030 con 0x7fc6b40b4140 2024-09-17T04:07:17.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.811+0000 7fc6abe00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 0x7fc69407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:17.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.811+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fc6a00bb4a0 con 0x7fc6b40b4140 2024-09-17T04:07:17.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.811+0000 7fc6b3e00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 0x7fc69407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:17.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.812+0000 7fc6b3e00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 0x7fc69407b0d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001720 tx=0x7fc6a4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:17.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.815+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc6a0083220 con 0x7fc6b40b4140 2024-09-17T04:07:17.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.997+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6b40ccc80 con 0x7fc6b40b4140 2024-09-17T04:07:17.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:17.998+0000 7fc6abe00640 1 -- 172.21.15.129:0/591332198 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) ==== 74+0+16864 (secure 0 0 0) 0x7fc6a0087c50 con 0x7fc6b40b4140 2024-09-17T04:07:17.998 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:17.998 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":248,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:05.543264+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:46.030146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:47.075519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/968815434":"2024-09-18T04:07:05.543219+0000","172.21.15.129:6833/968815434":"2024-09-18T04:07:05.543219+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 msgr2=0x7fc69407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 0x7fc69407b0d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4001720 tx=0x7fc6a4009040 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 msgr2=0x7fc6b40c3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0034040 tx=0x7fc6a0003910 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 shutdown_connections 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc694078bc0 0x7fc69407b0d0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc6b40cd1c0 0x7fc6b40c4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc6b40b4140 0x7fc6b40c3d40 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 --2- 172.21.15.129:0/591332198 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc6b40705a0 0x7fc6b41f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 >> 172.21.15.129:0/591332198 conn(0x7fc6b406d410 msgr2=0x7fc6b40b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:18.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 shutdown_connections 2024-09-17T04:07:18.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.000+0000 7fc6bae00640 1 -- 172.21.15.129:0/591332198 wait complete. 2024-09-17T04:07:18.025 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:07:18.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:18.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 -- 172.21.15.129:0/820174769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb80705a0 msgr2=0x7fcfb80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 --2- 172.21.15.129:0/820174769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb80705a0 0x7fcfb80b4140 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8005fd0 tx=0x7fcfa802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 -- 172.21.15.129:0/820174769 shutdown_connections 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 --2- 172.21.15.129:0/820174769 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb80705a0 0x7fcfb80b4140 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 -- 172.21.15.129:0/820174769 >> 172.21.15.129:0/820174769 conn(0x7fcfb806d410 msgr2=0x7fcfb80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 -- 172.21.15.129:0/820174769 shutdown_connections 2024-09-17T04:07:18.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.246+0000 7fcfbe200640 1 -- 172.21.15.129:0/820174769 wait complete. 2024-09-17T04:07:18.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.248+0000 7fcfbe200640 1 Processor -- start 2024-09-17T04:07:18.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.249+0000 7fcfbe200640 1 -- start start 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 0x7fcfb81f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 0x7fcfb80b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb806f2e0 con 0x7fcfb806eed0 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb81612e0 con 0x7fcfb80705a0 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfbe200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb806f460 con 0x7fcfb8157af0 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfb7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfb7400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49878/0 (socket says 172.21.15.129:49878) 2024-09-17T04:07:18.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.250+0000 7fcfb7400640 1 -- 172.21.15.129:0/1425489368 learned_addr learned my addr 172.21.15.129:0/1425489368 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 -- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 msgr2=0x7fcfb80b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7e00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 0x7fcfb80b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfaea00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 0x7fcfb81f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 0x7fcfb80b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 -- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 msgr2=0x7fcfb81f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 0x7fcfb81f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 -- 172.21.15.129:0/1425489368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb8158390 con 0x7fcfb806eed0 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7e00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 0x7fcfb80b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfaea00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 0x7fcfb81f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfb7400640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8006010 tx=0x7fcfa8003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8031da0 con 0x7fcfb806eed0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfa8033070 con 0x7fcfb806eed0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb8158670 con 0x7fcfb806eed0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.251+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80334c0 con 0x7fcfb806eed0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.252+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb8159830 con 0x7fcfb806eed0 2024-09-17T04:07:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.253+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb80b4140 con 0x7fcfb806eed0 2024-09-17T04:07:18.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.254+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcfa8033660 con 0x7fcfb806eed0 2024-09-17T04:07:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.255+0000 7fcfafe00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 0x7fcf9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.255+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fcfa80bc4a0 con 0x7fcfb806eed0 2024-09-17T04:07:18.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.256+0000 7fcfaea00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 0x7fcf9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.257+0000 7fcfaea00640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 0x7fcf9407b0a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8156fb0 tx=0x7fcfa0008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.260+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcfa803f030 con 0x7fcfb806eed0 2024-09-17T04:07:18.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.477+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfb80af7f0 con 0x7fcfb806eed0 2024-09-17T04:07:18.478 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.478+0000 7fcfafe00640 1 -- 172.21.15.129:0/1425489368 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fcfa8084870 con 0x7fcfb806eed0 2024-09-17T04:07:18.478 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:18.478 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:18.478 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 msgr2=0x7fcf9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 0x7fcf9407b0a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8156fb0 tx=0x7fcfa0008040 comp rx=0 tx=0).stop 2024-09-17T04:07:18.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 msgr2=0x7fcfb81f76e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8006010 tx=0x7fcfa8003c20 comp rx=0 tx=0).stop 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 shutdown_connections 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcf94078b90 0x7fcf9407b0a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfb8157af0 0x7fcfb80b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfb80705a0 0x7fcfb81f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 --2- 172.21.15.129:0/1425489368 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfb806eed0 0x7fcfb81f76e0 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 >> 172.21.15.129:0/1425489368 conn(0x7fcfb806d410 msgr2=0x7fcfb806f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 shutdown_connections 2024-09-17T04:07:18.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.480+0000 7fcfbe200640 1 -- 172.21.15.129:0/1425489368 wait complete. 2024-09-17T04:07:18.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:18.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 -- 172.21.15.129:0/3803365881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd74004800 msgr2=0x7fbd74004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3803365881 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd74004800 0x7fbd74004c70 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7fbd78001830 tx=0x7fbd74005830 comp rx=0 tx=0).stop 2024-09-17T04:07:18.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92200640 1 -- 172.21.15.129:0/3803365881 reap_dead start 2024-09-17T04:07:18.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 -- 172.21.15.129:0/3803365881 shutdown_connections 2024-09-17T04:07:18.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 -- 172.21.15.129:0/3803365881 >> 172.21.15.129:0/3803365881 conn(0x7fbd8c06cfc0 msgr2=0x7fbd8c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:18.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 -- 172.21.15.129:0/3803365881 shutdown_connections 2024-09-17T04:07:18.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.728+0000 7fbd92c00640 1 -- 172.21.15.129:0/3803365881 wait complete. 2024-09-17T04:07:18.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.730+0000 7fbd92c00640 1 Processor -- start 2024-09-17T04:07:18.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.730+0000 7fbd92c00640 1 -- start start 2024-09-17T04:07:18.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.731+0000 7fbd92c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 0x7fbd8c13a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.731+0000 7fbd92c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd8c1f8760 0x7fbd8c1f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b4e80 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b4d00 con 0x7fbd74004800 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c1f94c0 con 0x7fbd8c1f8760 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49898/0 (socket says 172.21.15.129:49898) 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 -- 172.21.15.129:0/3445683851 learned_addr learned my addr 172.21.15.129:0/3445683851 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd91800640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 0x7fbd8c13a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd90e00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd8c1f8760 0x7fbd8c1f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 -- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd8c1f8760 msgr2=0x7fbd8c1f8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd8c1f8760 0x7fbd8c1f8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 -- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 msgr2=0x7fbd8c13a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 0x7fbd8c13a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 -- 172.21.15.129:0/3445683851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c0af640 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd91800640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 0x7fbd8c13a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.732+0000 7fbd92200640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c1571d0 tx=0x7fbd8400ea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.733+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84010890 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.733+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd84017070 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.733+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84010de0 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.733+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8c0af920 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.733+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8c0b10e0 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.734+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8c15d660 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.735+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbd8401c460 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.736+0000 7fbd89600640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 0x7fbd6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:18.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.736+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fbd8409b150 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.737+0000 7fbd91800640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 0x7fbd6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:18.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.737+0000 7fbd91800640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 0x7fbd6c07b0a0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fbd80005eb0 tx=0x7fbd80005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:18.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.740+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbd84063530 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.956+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd8c0af3b0 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.957+0000 7fbd89600640 1 -- 172.21.15.129:0/3445683851 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5251 (secure 0 0 0) 0x7fbd84062e30 con 0x7fbd8c0b18e0 2024-09-17T04:07:18.957 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:18.958 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":240,"btime":"2024-09-17T04:07:08:563032+0000","default_fscid":18,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:06:48.110755+0000","modified":"2024-09-17T04:07:08.563026+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":248,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":11772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11772":{"gid":11772,"name":"d","rank":0,"incarnation":237,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6837/2055116414","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2055116414},{"type":"v1","addr":"172.21.15.181:6837","nonce":2055116414}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[37],"metadata_pool":36,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":11772,"qdb_cluster":[11772]},"id":18}]} 2024-09-17T04:07:18.958 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 240 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 msgr2=0x7fbd6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 0x7fbd6c07b0a0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fbd80005eb0 tx=0x7fbd80005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 msgr2=0x7fbd8c0b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c1571d0 tx=0x7fbd8400ea50 comp rx=0 tx=0).stop 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 shutdown_connections 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbd6c078b90 0x7fbd6c07b0a0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbd8c0b18e0 0x7fbd8c0b1cf0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbd8c1f8760 0x7fbd8c1f8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 --2- 172.21.15.129:0/3445683851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd74004800 0x7fbd8c13a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.959+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 >> 172.21.15.129:0/3445683851 conn(0x7fbd8c06cfc0 msgr2=0x7fbd8c15b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.960+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 shutdown_connections 2024-09-17T04:07:18.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:18.960+0000 7fbd92c00640 1 -- 172.21.15.129:0/3445683851 wait complete. 2024-09-17T04:07:18.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 -- 172.21.15.129:0/4038596723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 msgr2=0x7fa42815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 --2- 172.21.15.129:0/4038596723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa42815ae00 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7fa418005fd0 tx=0x7fa41802b910 comp rx=0 tx=0).stop 2024-09-17T04:07:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 -- 172.21.15.129:0/4038596723 shutdown_connections 2024-09-17T04:07:19.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 --2- 172.21.15.129:0/4038596723 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa42815ae00 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 -- 172.21.15.129:0/4038596723 >> 172.21.15.129:0/4038596723 conn(0x7fa42806d410 msgr2=0x7fa4280b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:19.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 -- 172.21.15.129:0/4038596723 shutdown_connections 2024-09-17T04:07:19.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.202+0000 7fa42fa00640 1 -- 172.21.15.129:0/4038596723 wait complete. 2024-09-17T04:07:19.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.204+0000 7fa42fa00640 1 Processor -- start 2024-09-17T04:07:19.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.204+0000 7fa42fa00640 1 -- start start 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 0x7fa4280cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 0x7fa4280cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa42806fd50 con 0x7fa4280af4c0 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa428071170 con 0x7fa4280ad590 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42fa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa42806fed0 con 0x7fa42815b340 2024-09-17T04:07:19.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49916/0 (socket says 172.21.15.129:49916) 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 -- 172.21.15.129:0/2736130410 learned_addr learned my addr 172.21.15.129:0/2736130410 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa41fe00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 0x7fa4280cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42da00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 0x7fa4280cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 -- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 msgr2=0x7fa4280cb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 0x7fa4280cb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 -- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 msgr2=0x7fa4280cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 0x7fa4280cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42d000640 1 -- 172.21.15.129:0/2736130410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280adc10 con 0x7fa4280af4c0 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa41fe00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 0x7fa4280cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.206+0000 7fa42da00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 0x7fa4280cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa42d000640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fa418034040 tx=0x7fa418004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41803e070 con 0x7fa4280af4c0 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa418031900 con 0x7fa4280af4c0 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4280adef0 con 0x7fa4280af4c0 2024-09-17T04:07:19.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418031cf0 con 0x7fa4280af4c0 2024-09-17T04:07:19.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.207+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4280ae4b0 con 0x7fa4280af4c0 2024-09-17T04:07:19.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.208+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa428076760 con 0x7fa4280af4c0 2024-09-17T04:07:19.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.209+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa418038070 con 0x7fa4280af4c0 2024-09-17T04:07:19.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.210+0000 7fa425600640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 0x7fa40407b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.210+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7fa4180bc0a0 con 0x7fa4280af4c0 2024-09-17T04:07:19.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.211+0000 7fa41fe00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 0x7fa40407b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.211+0000 7fa41fe00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 0x7fa40407b100 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa4280d1390 tx=0x7fa410008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:19.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.215+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa428076760 con 0x7fa4280af4c0 2024-09-17T04:07:19.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.399+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa428070350 con 0x7fa4280af4c0 2024-09-17T04:07:19.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.401+0000 7fa425600640 1 -- 172.21.15.129:0/2736130410 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v248) ==== 74+0+16864 (secure 0 0 0) 0x7fa4180843e0 con 0x7fa4280af4c0 2024-09-17T04:07:19.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:19.401 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":248,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:05.543264+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":36,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:06:46.030146+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"246","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":37,"pool_name":"cephfs_data","create_time":"2024-09-17T04:06:47.075519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"247","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/968815434":"2024-09-18T04:07:05.543219+0000","172.21.15.129:6833/968815434":"2024-09-18T04:07:05.543219+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:19.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 msgr2=0x7fa40407b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 0x7fa40407b100 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa4280d1390 tx=0x7fa410008040 comp rx=0 tx=0).stop 2024-09-17T04:07:19.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 msgr2=0x7fa4280ad050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7fa418034040 tx=0x7fa418004a50 comp rx=0 tx=0).stop 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 shutdown_connections 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa404078bf0 0x7fa40407b100 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa4280ad590 0x7fa4280cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa42815b340 0x7fa4280cb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 --2- 172.21.15.129:0/2736130410 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa4280af4c0 0x7fa4280ad050 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 >> 172.21.15.129:0/2736130410 conn(0x7fa42806d410 msgr2=0x7fa42815ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 shutdown_connections 2024-09-17T04:07:19.404 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.403+0000 7fa42fa00640 1 -- 172.21.15.129:0/2736130410 wait complete. 2024-09-17T04:07:19.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 -- 172.21.15.129:0/1514054944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0b3220 msgr2=0x7f5edc0d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/1514054944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0b3220 0x7f5edc0d4540 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0009850 tx=0x7f5ed002a510 comp rx=0 tx=0).stop 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 -- 172.21.15.129:0/1514054944 shutdown_connections 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/1514054944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0b3220 0x7f5edc0d4540 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 -- 172.21.15.129:0/1514054944 >> 172.21.15.129:0/1514054944 conn(0x7f5edc09d8d0 msgr2=0x7f5edc08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 -- 172.21.15.129:0/1514054944 shutdown_connections 2024-09-17T04:07:19.649 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.648+0000 7f5ee2a00640 1 -- 172.21.15.129:0/1514054944 wait complete. 2024-09-17T04:07:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.650+0000 7f5ee2a00640 1 Processor -- start 2024-09-17T04:07:19.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.651+0000 7f5ee2a00640 1 -- start start 2024-09-17T04:07:19.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 0x7f5edc1555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0d4a80 0x7f5edc0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc0b9ec0 con 0x7f5edc0d16d0 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5edc0b39b0 con 0x7f5edc0b3220 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee2a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5edc0ba040 con 0x7f5edc0d4a80 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49928/0 (socket says 172.21.15.129:49928) 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.652+0000 7f5ee0c00640 1 -- 172.21.15.129:0/3205904763 learned_addr learned my addr 172.21.15.129:0/3205904763 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee1600640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 0x7f5edc1555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee2000640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0d4a80 0x7f5edc0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 -- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0d4a80 msgr2=0x7f5edc0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0d4a80 0x7f5edc0b5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 -- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 msgr2=0x7f5edc1555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 0x7f5edc1555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 -- 172.21.15.129:0/3205904763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5edc152c90 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee1600640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 0x7f5edc1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee0c00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800c460 tx=0x7f5ec800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80052d0 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ec8005490 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5edc152f70 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.653+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8012870 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.654+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5edc153530 con 0x7f5edc0d16d0 2024-09-17T04:07:19.656 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.655+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5edc0b4720 con 0x7f5edc0d16d0 2024-09-17T04:07:19.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.656+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5ec8012a10 con 0x7f5edc0d16d0 2024-09-17T04:07:19.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.657+0000 7f5ed9600640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 0x7f5ebc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:19.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.657+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5795+0+0 (secure 0 0 0) 0x7f5ec809a980 con 0x7f5edc0d16d0 2024-09-17T04:07:19.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.657+0000 7f5ee1600640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 0x7f5ebc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:19.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.659+0000 7f5ee1600640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 0x7f5ebc07b0a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0034000 tx=0x7f5ed0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:19.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.662+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5edc0b4720 con 0x7f5edc0d16d0 2024-09-17T04:07:19.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:07:19.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:07:19.890 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.889+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5edc0d1ae0 con 0x7f5edc0d16d0 2024-09-17T04:07:19.890 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:07:19.890+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:07:19.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.892+0000 7f5ed9600640 1 -- 172.21.15.129:0/3205904763 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v241) ==== 161+0+0 (secure 0 0 0) 0x7f5ec8062cc0 con 0x7f5edc0d16d0 2024-09-17T04:07:19.892 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:07:19.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 msgr2=0x7f5ebc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 0x7f5ebc07b0a0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0034000 tx=0x7f5ed0004730 comp rx=0 tx=0).stop 2024-09-17T04:07:19.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 msgr2=0x7f5edc09f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:19.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800c460 tx=0x7f5ec800c920 comp rx=0 tx=0).stop 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 shutdown_connections 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5ebc078b90 0x7f5ebc07b0a0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5edc0d4a80 0x7f5edc0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5edc0d16d0 0x7f5edc09f330 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 --2- 172.21.15.129:0/3205904763 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5edc0b3220 0x7f5edc1555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.895+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 >> 172.21.15.129:0/3205904763 conn(0x7f5edc09d8d0 msgr2=0x7f5edc0b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:19.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.896+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 shutdown_connections 2024-09-17T04:07:19.897 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:19.896+0000 7f5ee2a00640 1 -- 172.21.15.129:0/3205904763 wait complete. 2024-09-17T04:07:19.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:07:20.034 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 -- 172.21.15.129:0/558604985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 msgr2=0x7f79541537f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 --2- 172.21.15.129:0/558604985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f79541537f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f7944009830 tx=0x7f794402a530 comp rx=0 tx=0).stop 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 -- 172.21.15.129:0/558604985 shutdown_connections 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 --2- 172.21.15.129:0/558604985 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f79541537f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 -- 172.21.15.129:0/558604985 >> 172.21.15.129:0/558604985 conn(0x7f79540b27b0 msgr2=0x7f79540b2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 -- 172.21.15.129:0/558604985 shutdown_connections 2024-09-17T04:07:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.189+0000 7f7959400640 1 -- 172.21.15.129:0/558604985 wait complete. 2024-09-17T04:07:20.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.191+0000 7f7959400640 1 Processor -- start 2024-09-17T04:07:20.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.191+0000 7f7959400640 1 -- start start 2024-09-17T04:07:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f7954157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7954158af0 0x7f79540cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f795415d6b0 con 0x7f79540cd080 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f79540cd7e0 con 0x7f795406f7f0 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7959400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f795415d830 con 0x7f7954158af0 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7949e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7949e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49944/0 (socket says 172.21.15.129:49944) 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7949e00640 1 -- 172.21.15.129:0/4064435138 learned_addr learned my addr 172.21.15.129:0/4064435138 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7952800640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f7954157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.193+0000 7f7953200640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7954158af0 0x7f79540cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:20.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 -- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7954158af0 msgr2=0x7f79540cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:20.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7954158af0 0x7f79540cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 -- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 msgr2=0x7f7954157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f7954157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 -- 172.21.15.129:0/4064435138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7954157a50 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7949e00640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f793c00c3f0 tx=0x7f793c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c0052d0 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f793c007020 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7954157d30 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f793c007470 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.194+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79541582f0 con 0x7f79540cd080 2024-09-17T04:07:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.196+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f795415dc80 con 0x7f79540cd080 2024-09-17T04:07:20.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.197+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f793c0067d0 con 0x7f79540cd080 2024-09-17T04:07:20.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.198+0000 7f794b400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 0x7f793007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:20.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.198+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5910+0+0 (secure 0 0 0) 0x7f793c09a320 con 0x7f79540cd080 2024-09-17T04:07:20.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.198+0000 7f7952800640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 0x7f793007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:20.198 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.199+0000 7f7952800640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 0x7f793007b0a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7944034000 tx=0x7f7944004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:20.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.202+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f793c061fa0 con 0x7f79540cd080 2024-09-17T04:07:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.422+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f79540b3360 con 0x7f79540cd080 2024-09-17T04:07:20.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.901+0000 7f794b400640 1 -- 172.21.15.129:0/4064435138 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v242) ==== 104+0+0 (secure 0 0 0) 0x7f793c0669d0 con 0x7f79540cd080 2024-09-17T04:07:20.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 msgr2=0x7f793007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 0x7f793007b0a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7944034000 tx=0x7f7944004730 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 msgr2=0x7f7954142c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f793c00c3f0 tx=0x7f793c00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 shutdown_connections 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7930078b90 0x7f793007b0a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7954158af0 0x7f79540cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f79540cd080 0x7f7954142c50 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 --2- 172.21.15.129:0/4064435138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f795406f7f0 0x7f7954157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 >> 172.21.15.129:0/4064435138 conn(0x7f79540b27b0 msgr2=0x7f7954150540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 shutdown_connections 2024-09-17T04:07:20.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:20.905+0000 7f7959400640 1 -- 172.21.15.129:0/4064435138 wait complete. 2024-09-17T04:07:20.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:07:21.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 -- 172.21.15.129:0/1353194005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80cc170 msgr2=0x7f8da80ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1353194005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80cc170 0x7f8da80ce5f0 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7f8da80b51b0 tx=0x7f8d9402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 -- 172.21.15.129:0/1353194005 shutdown_connections 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1353194005 >> v2:172.21.15.181:3300/0 conn(0x7f8da80ceb30 0x7f8da80ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1353194005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80cc170 0x7f8da80ce5f0 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 -- 172.21.15.129:0/1353194005 >> 172.21.15.129:0/1353194005 conn(0x7f8da80ccc40 msgr2=0x7f8da80cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 -- 172.21.15.129:0/1353194005 shutdown_connections 2024-09-17T04:07:21.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.154+0000 7f8dafc00640 1 -- 172.21.15.129:0/1353194005 wait complete. 2024-09-17T04:07:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.156+0000 7f8dafc00640 1 Processor -- start 2024-09-17T04:07:21.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.157+0000 7f8dafc00640 1 -- start start 2024-09-17T04:07:21.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 0x7f8da80b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:21.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 0x7f8da814f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80cb2c0 con 0x7f8da80ceb30 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80cb140 con 0x7f8da80cc170 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dafc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8da80cb440 con 0x7f8da8070b20 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dadc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dadc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49964/0 (socket says 172.21.15.129:49964) 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.158+0000 7f8dadc00640 1 -- 172.21.15.129:0/1254901793 learned_addr learned my addr 172.21.15.129:0/1254901793 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:21.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8da7e00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 0x7f8da814f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dad200640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 0x7f8da80b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 -- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 msgr2=0x7f8da80b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 0x7f8da80b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 -- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 msgr2=0x7f8da814f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 0x7f8da814f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 -- 172.21.15.129:0/1254901793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da80b1ef0 con 0x7f8da80ceb30 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8da7e00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 0x7f8da814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dad200640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 0x7f8da80b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:21.161 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dadc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f8da000eab0 tx=0x7f8da000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da000cea0 con 0x7f8da80ceb30 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8da0004b20 con 0x7f8da80ceb30 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0005020 con 0x7f8da80ceb30 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da80b21d0 con 0x7f8da80ceb30 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.159+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da8161d70 con 0x7f8da80ceb30 2024-09-17T04:07:21.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.160+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da80b1c60 con 0x7f8da80ceb30 2024-09-17T04:07:21.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.162+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8da003bd10 con 0x7f8da80ceb30 2024-09-17T04:07:21.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.163+0000 7f8da4c00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 0x7f8d8807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:21.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.163+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(250..250 src has 1..250) ==== 5910+0+0 (secure 0 0 0) 0x7f8da009bd20 con 0x7f8da80ceb30 2024-09-17T04:07:21.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.163+0000 7f8dad200640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 0x7f8d8807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:21.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.164+0000 7f8dad200640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 0x7f8d8807b0d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0016a0 tx=0x7f8d9c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:21.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.166+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8da0064070 con 0x7f8da80ceb30 2024-09-17T04:07:21.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.349+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8da8050f60 con 0x7f8da80ceb30 2024-09-17T04:07:21.904 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.904+0000 7f8da4c00640 1 -- 172.21.15.129:0/1254901793 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v251) ==== 181+0+0 (secure 0 0 0) 0x7f8da8050f60 con 0x7f8da80ceb30 2024-09-17T04:07:21.904 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 msgr2=0x7f8d8807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 0x7f8d8807b0d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0016a0 tx=0x7f8d9c009040 comp rx=0 tx=0).stop 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 msgr2=0x7f8da8164e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7f8da000eab0 tx=0x7f8da000ef70 comp rx=0 tx=0).stop 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 shutdown_connections 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8d88078bc0 0x7f8d8807b0d0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8da80ceb30 0x7f8da8164e30 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8da80cc170 0x7f8da814f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 --2- 172.21.15.129:0/1254901793 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8da8070b20 0x7f8da80b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 >> 172.21.15.129:0/1254901793 conn(0x7f8da80ccc40 msgr2=0x7f8da8160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 shutdown_connections 2024-09-17T04:07:21.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:21.908+0000 7f8dafc00640 1 -- 172.21.15.129:0/1254901793 wait complete. 2024-09-17T04:07:21.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:07:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 -- 172.21.15.129:0/1992771070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16380b3f70 msgr2=0x7f16380b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 --2- 172.21.15.129:0/1992771070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16380b3f70 0x7f16380b49a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1628009830 tx=0x7f162802a480 comp rx=0 tx=0).stop 2024-09-17T04:07:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 -- 172.21.15.129:0/1992771070 shutdown_connections 2024-09-17T04:07:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 --2- 172.21.15.129:0/1992771070 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1638074580 0x7f16380b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 --2- 172.21.15.129:0/1992771070 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f16380b3f70 0x7f16380b49a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 -- 172.21.15.129:0/1992771070 >> 172.21.15.129:0/1992771070 conn(0x7f16380ccc40 msgr2=0x7f16380cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:22.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 -- 172.21.15.129:0/1992771070 shutdown_connections 2024-09-17T04:07:22.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.163+0000 7f163cc00640 1 -- 172.21.15.129:0/1992771070 wait complete. 2024-09-17T04:07:22.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.165+0000 7f163cc00640 1 Processor -- start 2024-09-17T04:07:22.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.165+0000 7f163cc00640 1 -- start start 2024-09-17T04:07:22.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1638074580 0x7f163815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:22.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:22.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f163814c840 0x7f16380aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:22.169 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f163806fc20 con 0x7f16380b2360 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f163806faa0 con 0x7f1638074580 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f163cc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1638166f20 con 0x7f163814c840 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f162d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f162d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49966/0 (socket says 172.21.15.129:49966) 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f162d400640 1 -- 172.21.15.129:0/3929501233 learned_addr learned my addr 172.21.15.129:0/3929501233 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f1635e00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1638074580 0x7f163815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f1636800640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f163814c840 0x7f16380aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f162d400640 1 -- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f163814c840 msgr2=0x7f16380aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.167+0000 7f162d400640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f163814c840 0x7f16380aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162d400640 1 -- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1638074580 msgr2=0x7f163815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162d400640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1638074580 0x7f163815adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162d400640 1 -- 172.21.15.129:0/3929501233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16381f5e90 con 0x7f16380b2360 2024-09-17T04:07:22.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162d400640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f162000e9c0 tx=0x7f162000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162000cd80 con 0x7f16380b2360 2024-09-17T04:07:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1620004020 con 0x7f16380b2360 2024-09-17T04:07:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620004450 con 0x7f16380b2360 2024-09-17T04:07:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163814c540 con 0x7f16380b2360 2024-09-17T04:07:22.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.168+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1638158ec0 con 0x7f16380b2360 2024-09-17T04:07:22.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.169+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16380b3f70 con 0x7f16380b2360 2024-09-17T04:07:22.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.171+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f16200035f0 con 0x7f16380b2360 2024-09-17T04:07:22.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.172+0000 7f162ea00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 0x7f161407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:22.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.172+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(251..251 src has 1..251) ==== 5472+0+0 (secure 0 0 0) 0x7f162009b500 con 0x7f16380b2360 2024-09-17T04:07:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.172+0000 7f1635e00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 0x7f161407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:22.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.173+0000 7f1635e00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 0x7f161407b0a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1628034000 tx=0x7f1628004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:22.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.176+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f16200633f0 con 0x7f16380b2360 2024-09-17T04:07:22.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.357+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f16380b1530 con 0x7f16380b2360 2024-09-17T04:07:22.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.910+0000 7f162ea00640 1 -- 172.21.15.129:0/3929501233 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v252) ==== 169+0+0 (secure 0 0 0) 0x7f1620067e20 con 0x7f16380b2360 2024-09-17T04:07:22.911 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.914+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 msgr2=0x7f161407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.914+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 0x7f161407b0a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f1628034000 tx=0x7f1628004730 comp rx=0 tx=0).stop 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.914+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 msgr2=0x7f16380ae580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.914+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7f162000e9c0 tx=0x7f162000ee80 comp rx=0 tx=0).stop 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 shutdown_connections 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1614078b90 0x7f161407b0a0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f163814c840 0x7f16380aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f16380b2360 0x7f16380ae580 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 --2- 172.21.15.129:0/3929501233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1638074580 0x7f163815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:22.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 >> 172.21.15.129:0/3929501233 conn(0x7f16380ccc40 msgr2=0x7f163806eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:22.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 shutdown_connections 2024-09-17T04:07:22.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:22.915+0000 7f163cc00640 1 -- 172.21.15.129:0/3929501233 wait complete. 2024-09-17T04:07:22.943 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:07:23.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 -- 172.21.15.129:0/903335963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0cf910 msgr2=0x7f0e3c0d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 --2- 172.21.15.129:0/903335963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0d0340 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c071c60 tx=0x7f0e2402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 -- 172.21.15.129:0/903335963 shutdown_connections 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 --2- 172.21.15.129:0/903335963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0d0340 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 --2- 172.21.15.129:0/903335963 >> v2:172.21.15.181:3300/0 conn(0x7f0e3c0ad8d0 0x7f0e3c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 -- 172.21.15.129:0/903335963 >> 172.21.15.129:0/903335963 conn(0x7f0e3c06cfc0 msgr2=0x7f0e3c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 -- 172.21.15.129:0/903335963 shutdown_connections 2024-09-17T04:07:23.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.165+0000 7f0e41000640 1 -- 172.21.15.129:0/903335963 wait complete. 2024-09-17T04:07:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.167+0000 7f0e41000640 1 Processor -- start 2024-09-17T04:07:23.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.167+0000 7f0e41000640 1 -- start start 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3c0d0880 0x7f0e3c0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c06f3c0 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3c0afae0 con 0x7f0e3c0cf910 2024-09-17T04:07:23.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e41000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0e3c06f540 con 0x7f0e3c0d0880 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e3b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e3b400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49986/0 (socket says 172.21.15.129:49986) 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e3b400640 1 -- 172.21.15.129:0/3425126917 learned_addr learned my addr 172.21.15.129:0/3425126917 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e32a00640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.169+0000 7f0e3be00640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3c0d0880 0x7f0e3c0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 -- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3c0d0880 msgr2=0x7f0e3c0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3c0d0880 0x7f0e3c0b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 -- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3c0cf910 msgr2=0x7f0e3c0b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 -- 172.21.15.129:0/3425126917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e3c163040 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e3b400640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00d770 tx=0x7f0e2c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c017070 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0e2c009d20 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0106c0 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e3c0afde0 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.170+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e3c157f40 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.171+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e3c070f00 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.173+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0e2c01a070 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.174+0000 7f0e33e00640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.174+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f0e2c09ab60 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.174+0000 7f0e32a00640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.175+0000 7f0e32a00640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0e24033000 tx=0x7f0e24004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:23.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.178+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0e3c070f00 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.367+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f0e3c06fa60 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.372+0000 7f0e33e00640 1 -- 172.21.15.129:0/3425126917 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v70) ==== 71+0+0 (secure 0 0 0) 0x7f0e2c062be0 con 0x7f0e3c0ad8d0 2024-09-17T04:07:23.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 msgr2=0x7f0e1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0e24033000 tx=0x7f0e24004730 comp rx=0 tx=0).stop 2024-09-17T04:07:23.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 msgr2=0x7f0e3c150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00d770 tx=0x7f0e2c00dc30 comp rx=0 tx=0).stop 2024-09-17T04:07:23.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 shutdown_connections 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0e18078b90 0x7f0e1807b0a0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0e3c0d0880 0x7f0e3c0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0e3c0cf910 0x7f0e3c0b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 --2- 172.21.15.129:0/3425126917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0e3c0ad8d0 0x7f0e3c150db0 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 >> 172.21.15.129:0/3425126917 conn(0x7f0e3c06cfc0 msgr2=0x7f0e3c06ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 shutdown_connections 2024-09-17T04:07:23.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.375+0000 7f0e41000640 1 -- 172.21.15.129:0/3425126917 wait complete. 2024-09-17T04:07:23.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub' 2024-09-17T04:07:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 -- 172.21.15.129:0/39618653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0051a0 msgr2=0x7fd83c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 --2- 172.21.15.129:0/39618653 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd83c0051a0 0x7fd83c00b9c0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fd858070550 tx=0x7fd83c005830 comp rx=0 tx=0).stop 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85d200640 1 -- 172.21.15.129:0/39618653 reap_dead start 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 -- 172.21.15.129:0/39618653 shutdown_connections 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 -- 172.21.15.129:0/39618653 >> 172.21.15.129:0/39618653 conn(0x7fd85806d410 msgr2=0x7fd8580b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 -- 172.21.15.129:0/39618653 shutdown_connections 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.621+0000 7fd85f200640 1 -- 172.21.15.129:0/39618653 wait complete. 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.623+0000 7fd85f200640 1 Processor -- start 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.623+0000 7fd85f200640 1 -- start start 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd83c0051a0 0x7fd8581f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8580c7320 0x7fd8580c9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8580cd390 con 0x7fd8580c7730 2024-09-17T04:07:23.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd858070d30 con 0x7fd8580c7320 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd8580cd510 con 0x7fd83c0051a0 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:49996/0 (socket says 172.21.15.129:49996) 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 -- 172.21.15.129:0/3288584073 learned_addr learned my addr 172.21.15.129:0/3288584073 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd84f400640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8580c7320 0x7fd8580c9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd857e00640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd83c0051a0 0x7fd8581f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 -- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd83c0051a0 msgr2=0x7fd8581f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd83c0051a0 0x7fd8581f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 -- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8580c7320 msgr2=0x7fd8580c9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8580c7320 0x7fd8580c9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.625+0000 7fd85d200640 1 -- 172.21.15.129:0/3288584073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8580c5c70 con 0x7fd8580c7730 2024-09-17T04:07:23.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd85d200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fd85002b7c0 tx=0x7fd850033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd850042070 con 0x7fd8580c7730 2024-09-17T04:07:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd850003970 con 0x7fd8580c7730 2024-09-17T04:07:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8580c5f50 con 0x7fd8580c7730 2024-09-17T04:07:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd850038040 con 0x7fd8580c7730 2024-09-17T04:07:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.626+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8580c6510 con 0x7fd8580c7730 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.627+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8580c59e0 con 0x7fd8580c7730 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.628+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd850004030 con 0x7fd8580c7730 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.629+0000 7fd854c00640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 0x7fd83407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.629+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7fd8500baf40 con 0x7fd8580c7730 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.630+0000 7fd857e00640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 0x7fd83407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:23.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.631+0000 7fd857e00640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 0x7fd83407b0d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd848006fd0 tx=0x7fd848006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:23.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.634+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd850082f30 con 0x7fd8580c7730 2024-09-17T04:07:23.889 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.889+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub"]} v 0) -- 0x7fd85806a140 con 0x7fd8580c7730 2024-09-17T04:07:23.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.912+0000 7fd854c00640 1 -- 172.21.15.129:0/3288584073 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_reintegration_via_scrub"]}]=0 v901) ==== 143+0+0 (secure 0 0 0) 0x7fd850087960 con 0x7fd8580c7730 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 msgr2=0x7fd83407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 0x7fd83407b0d0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fd848006fd0 tx=0x7fd848006b10 comp rx=0 tx=0).stop 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 msgr2=0x7fd8580c9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7fd85002b7c0 tx=0x7fd850033c10 comp rx=0 tx=0).stop 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 shutdown_connections 2024-09-17T04:07:23.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd834078bc0 0x7fd83407b0d0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8580c7730 0x7fd8580c9f60 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8580c7320 0x7fd8580c9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 --2- 172.21.15.129:0/3288584073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd83c0051a0 0x7fd8581f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 >> 172.21.15.129:0/3288584073 conn(0x7fd85806d410 msgr2=0x7fd85806f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 shutdown_connections 2024-09-17T04:07:23.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:23.915+0000 7fd85f200640 1 -- 172.21.15.129:0/3288584073 wait complete. 2024-09-17T04:07:23.937 INFO:tasks.cephfs_test_runner:That reintegration is triggered via recursive scrub. ... ok 2024-09-17T04:07:23.937 INFO:tasks.cephfs_test_runner:Starting test: test_replicated_delete_speed (tasks.cephfs.test_strays.TestStrays) That deletions of replicated metadata are not pathologically slow 2024-09-17T04:07:23.937 INFO:tasks.cephfs_test_runner:test_replicated_delete_speed (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:07:23.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 -- 172.21.15.129:0/2066822577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0ccc80 msgr2=0x7f798c0cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 --2- 172.21.15.129:0/2066822577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0ccc80 0x7f798c0cd090 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f7974005fd0 tx=0x7f797402a5c0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 -- 172.21.15.129:0/2066822577 shutdown_connections 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 --2- 172.21.15.129:0/2066822577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 0x7f798c06fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 --2- 172.21.15.129:0/2066822577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0ccc80 0x7f798c0cd090 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 -- 172.21.15.129:0/2066822577 >> 172.21.15.129:0/2066822577 conn(0x7f798c06d410 msgr2=0x7f798c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 -- 172.21.15.129:0/2066822577 shutdown_connections 2024-09-17T04:07:24.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.158+0000 7f7990e00640 1 -- 172.21.15.129:0/2066822577 wait complete. 2024-09-17T04:07:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.160+0000 7f7990e00640 1 Processor -- start 2024-09-17T04:07:24.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.160+0000 7f7990e00640 1 -- start start 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 0x7f798c0cbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0af4c0 0x7f798c1f5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f798c0710e0 con 0x7f798c0ccc80 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f798c070f60 con 0x7f798c0af4c0 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f7990e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f798c06ff90 con 0x7f798c06f640 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f798a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f798a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45930/0 (socket says 172.21.15.129:45930) 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.162+0000 7f798a800640 1 -- 172.21.15.129:0/1509344350 learned_addr learned my addr 172.21.15.129:0/1509344350 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:24.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7983e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0af4c0 0x7f798c1f5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7989e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 0x7f798c0cbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 -- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 msgr2=0x7f798c0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 0x7f798c0cbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 -- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0af4c0 msgr2=0x7f798c1f5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0af4c0 0x7f798c1f5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 -- 172.21.15.129:0/1509344350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c1f5e60 con 0x7f798c0ccc80 2024-09-17T04:07:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f798a800640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f798400be00 tx=0x7f798400e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798401a030 con 0x7f798c0ccc80 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f798400edf0 con 0x7f798c0ccc80 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798c1f2d90 con 0x7f798c0ccc80 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7984003bd0 con 0x7f798c0ccc80 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.163+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798c1f3350 con 0x7f798c0ccc80 2024-09-17T04:07:24.165 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.164+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c0ae900 con 0x7f798c0ccc80 2024-09-17T04:07:24.166 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.166+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f798401d070 con 0x7f798c0ccc80 2024-09-17T04:07:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.167+0000 7f7982000640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 0x7f796807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.167+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f798409d890 con 0x7f798c0ccc80 2024-09-17T04:07:24.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.167+0000 7f7989e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 0x7f796807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.168+0000 7f7989e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 0x7f796807b0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f797c007fd0 tx=0x7f797c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:24.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.170+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7984012040 con 0x7f798c0ccc80 2024-09-17T04:07:24.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.353+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f798c0cd330 con 0x7f798c0ccc80 2024-09-17T04:07:24.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.354+0000 7f7982000640 1 -- 172.21.15.129:0/1509344350 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v252) ==== 74+0+13222 (secure 0 0 0) 0x7f798c0cd330 con 0x7f798c0ccc80 2024-09-17T04:07:24.354 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:24.354 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":252,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:22.905887+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":37,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":249,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.181:6837/2055116414":"2024-09-18T04:07:19.891554+0000","172.21.15.181:6835/2055116414":"2024-09-18T04:07:19.891554+0000","172.21.15.129:6835/968815434":"2024-09-18T04:07:05.543219+0000","172.21.15.129:6833/968815434":"2024-09-18T04:07:05.543219+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 msgr2=0x7f796807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 0x7f796807b0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f797c007fd0 tx=0x7f797c009040 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 msgr2=0x7f798c1f5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f798400be00 tx=0x7f798400e490 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 shutdown_connections 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7968078bc0 0x7f796807b0d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f798c0ccc80 0x7f798c1f5540 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f798c0af4c0 0x7f798c1f5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 --2- 172.21.15.129:0/1509344350 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f798c06f640 0x7f798c0cbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.356+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 >> 172.21.15.129:0/1509344350 conn(0x7f798c06d410 msgr2=0x7f798c15ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:24.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.357+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 shutdown_connections 2024-09-17T04:07:24.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.357+0000 7f7990e00640 1 -- 172.21.15.129:0/1509344350 wait complete. 2024-09-17T04:07:24.379 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:07:24.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.603+0000 7f14ea800640 1 -- 172.21.15.129:0/3625255068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c8004800 msgr2=0x7f14c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.603+0000 7f14ea800640 1 --2- 172.21.15.129:0/3625255068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14c8004800 0x7f14c8004c10 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009b20 tx=0x7f14c8005830 comp rx=0 tx=0).stop 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.603+0000 7f14e3e00640 1 -- 172.21.15.129:0/3625255068 reap_dead start 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.603+0000 7f14ea800640 1 -- 172.21.15.129:0/3625255068 shutdown_connections 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.603+0000 7f14ea800640 1 -- 172.21.15.129:0/3625255068 >> 172.21.15.129:0/3625255068 conn(0x7f14e40ccc40 msgr2=0x7f14e40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:24.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.604+0000 7f14ea800640 1 -- 172.21.15.129:0/3625255068 shutdown_connections 2024-09-17T04:07:24.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.604+0000 7f14ea800640 1 -- 172.21.15.129:0/3625255068 wait complete. 2024-09-17T04:07:24.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.606+0000 7f14ea800640 1 Processor -- start 2024-09-17T04:07:24.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.606+0000 7f14ea800640 1 -- start start 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 0x7f14e414b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 0x7f14e41f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f14e40b4b20 con 0x7f14e41f6c60 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f14e40b49a0 con 0x7f14e4158bd0 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.607+0000 7f14ea800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f14e40b4ca0 con 0x7f14c8004800 2024-09-17T04:07:24.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45948/0 (socket says 172.21.15.129:45948) 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 -- 172.21.15.129:0/3496217693 learned_addr learned my addr 172.21.15.129:0/3496217693 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3400640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 0x7f14e414b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e2a00640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 0x7f14e41f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 -- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 msgr2=0x7f14e414b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 0x7f14e414b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 -- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 msgr2=0x7f14e41f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 0x7f14e41f6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3e00640 1 -- 172.21.15.129:0/3496217693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14e4161b70 con 0x7f14e41f6c60 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e3400640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 0x7f14e414b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.608+0000 7f14e2a00640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 0x7f14e41f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:24.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14e3e00640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f14e4072c00 tx=0x7f14dc00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc015070 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f14dc00cb20 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14e4161220 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14dc011460 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.609+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14e414bac0 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.610+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14e40b3f70 con 0x7f14e41f6c60 2024-09-17T04:07:24.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.611+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f14dc018070 con 0x7f14e41f6c60 2024-09-17T04:07:24.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.612+0000 7f14db400640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 0x7f14c407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.612+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f14dc09af90 con 0x7f14e41f6c60 2024-09-17T04:07:24.613 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.613+0000 7f14e3400640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 0x7f14c407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:24.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.613+0000 7f14e3400640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 0x7f14c407b0a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d4006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:24.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.617+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f14dc0636f0 con 0x7f14e41f6c60 2024-09-17T04:07:24.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.800+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14e4162b40 con 0x7f14e41f6c60 2024-09-17T04:07:24.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.801+0000 7f14db400640 1 -- 172.21.15.129:0/3496217693 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v252) ==== 93+0+10 (secure 0 0 0) 0x7f14dc062ff0 con 0x7f14e41f6c60 2024-09-17T04:07:24.801 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:07:24.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 msgr2=0x7f14c407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 0x7f14c407b0a0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f14d4000c00 tx=0x7f14d4006b70 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 msgr2=0x7f14e415ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f14e4072c00 tx=0x7f14dc00eb60 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 shutdown_connections 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f14c4078b90 0x7f14c407b0a0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f14e41f6c60 0x7f14e415ceb0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14e4158bd0 0x7f14e41f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 --2- 172.21.15.129:0/3496217693 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f14c8004800 0x7f14e414b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 >> 172.21.15.129:0/3496217693 conn(0x7f14e40ccc40 msgr2=0x7f14e40acf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 shutdown_connections 2024-09-17T04:07:24.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:24.803+0000 7f14ea800640 1 -- 172.21.15.129:0/3496217693 wait complete. 2024-09-17T04:07:24.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed' 2024-09-17T04:07:25.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 -- 172.21.15.129:0/2130458701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0cd390 msgr2=0x7f4c7c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:25.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 --2- 172.21.15.129:0/2130458701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0cd390 0x7f4c7c1609c0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0d1370 tx=0x7f4c7402aac0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 -- 172.21.15.129:0/2130458701 shutdown_connections 2024-09-17T04:07:25.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 --2- 172.21.15.129:0/2130458701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0cd390 0x7f4c7c1609c0 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 --2- 172.21.15.129:0/2130458701 >> v2:172.21.15.181:3300/0 conn(0x7f4c7c0705a0 0x7f4c7c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 -- 172.21.15.129:0/2130458701 >> 172.21.15.129:0/2130458701 conn(0x7f4c7c06d410 msgr2=0x7f4c7c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 -- 172.21.15.129:0/2130458701 shutdown_connections 2024-09-17T04:07:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.053+0000 7f4c82600640 1 -- 172.21.15.129:0/2130458701 wait complete. 2024-09-17T04:07:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.055+0000 7f4c82600640 1 Processor -- start 2024-09-17T04:07:25.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.055+0000 7f4c82600640 1 -- start start 2024-09-17T04:07:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7c06eed0 0x7f4c7c157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 0x7f4c7c158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:25.060 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c160f00 con 0x7f4c7c0705a0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c06f4f0 con 0x7f4c7c06eed0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c82600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c161080 con 0x7f4c7c1586a0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c72a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45972/0 (socket says 172.21.15.129:45972) 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c72a00640 1 -- 172.21.15.129:0/3352179194 learned_addr learned my addr 172.21.15.129:0/3352179194 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c7b400640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7c06eed0 0x7f4c7c157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c7be00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 0x7f4c7c158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.057+0000 7f4c72a00640 1 -- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 msgr2=0x7f4c7c158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c72a00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 0x7f4c7c158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c72a00640 1 -- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7c06eed0 msgr2=0x7f4c7c157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c72a00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7c06eed0 0x7f4c7c157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.061 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c72a00640 1 -- 172.21.15.129:0/3352179194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c158ff0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c7be00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 0x7f4c7c158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c72a00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c3a0 tx=0x7f4c6400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64017070 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c640053c0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c1597a0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c64003850 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.058+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c159d60 con 0x7f4c7c0705a0 2024-09-17T04:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.059+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c159510 con 0x7f4c7c0705a0 2024-09-17T04:07:25.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.061+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4c6401a070 con 0x7f4c7c0705a0 2024-09-17T04:07:25.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.062+0000 7f4c73e00640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 0x7f4c5807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:25.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.062+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f4c64099dd0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.062+0000 7f4c7b400640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 0x7f4c5807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:25.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.063+0000 7f4c7b400640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 0x7f4c5807b0d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c005fd0 tx=0x7f4c6c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:25.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.066+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4c7c158ff0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.315+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]} v 0) -- 0x7f4c7c0b4140 con 0x7f4c7c0705a0 2024-09-17T04:07:25.921 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.921+0000 7f4c73e00640 1 -- 172.21.15.129:0/3352179194 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]}]=0 v903) ==== 146+0+0 (secure 0 0 0) 0x7f4c6409b0d0 con 0x7f4c7c0705a0 2024-09-17T04:07:25.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 msgr2=0x7f4c5807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:25.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 0x7f4c5807b0d0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c005fd0 tx=0x7f4c6c005850 comp rx=0 tx=0).stop 2024-09-17T04:07:25.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 msgr2=0x7f4c7c1f3f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c3a0 tx=0x7f4c6400c860 comp rx=0 tx=0).stop 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 shutdown_connections 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c58078bc0 0x7f4c5807b0d0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4c7c1586a0 0x7f4c7c158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4c7c0705a0 0x7f4c7c1f3f70 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 --2- 172.21.15.129:0/3352179194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c7c06eed0 0x7f4c7c157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 >> 172.21.15.129:0/3352179194 conn(0x7f4c7c06d410 msgr2=0x7f4c7c06f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 shutdown_connections 2024-09-17T04:07:25.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:25.925+0000 7f4c82600640 1 -- 172.21.15.129:0/3352179194 wait complete. 2024-09-17T04:07:25.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:07:26.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 -- 172.21.15.129:0/1006204012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc0ced00 msgr2=0x7f9edc0cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1006204012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc0ced00 0x7f9edc0cf110 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0727e0 tx=0x7f9ed402a560 comp rx=0 tx=0).stop 2024-09-17T04:07:26.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 -- 172.21.15.129:0/1006204012 shutdown_connections 2024-09-17T04:07:26.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1006204012 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc0ced00 0x7f9edc0cf110 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1006204012 >> v2:172.21.15.181:3300/0 conn(0x7f9edc073490 0x7f9edc15b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 -- 172.21.15.129:0/1006204012 >> 172.21.15.129:0/1006204012 conn(0x7f9edc06cfc0 msgr2=0x7f9edc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:26.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 -- 172.21.15.129:0/1006204012 shutdown_connections 2024-09-17T04:07:26.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.172+0000 7f9ee3000640 1 -- 172.21.15.129:0/1006204012 wait complete. 2024-09-17T04:07:26.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.174+0000 7f9ee3000640 1 Processor -- start 2024-09-17T04:07:26.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.174+0000 7f9ee3000640 1 -- start start 2024-09-17T04:07:26.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9edc0ced00 0x7f9edc1f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9edc0d0380 0x7f9edc14c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc0731a0 con 0x7f9edc073490 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9edc073020 con 0x7f9edc0d0380 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee3000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9edc0cf650 con 0x7f9edc0ced00 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45990/0 (socket says 172.21.15.129:45990) 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 -- 172.21.15.129:0/1620861838 learned_addr learned my addr 172.21.15.129:0/1620861838 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee2600640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9edc0d0380 0x7f9edc14c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1200640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9edc0ced00 0x7f9edc1f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 -- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9edc0ced00 msgr2=0x7f9edc1f5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9edc0ced00 0x7f9edc1f5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 -- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9edc0d0380 msgr2=0x7f9edc14c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9edc0d0380 0x7f9edc14c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.176+0000 7f9ee1c00640 1 -- 172.21.15.129:0/1620861838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9edc14fe30 con 0x7f9edc073490 2024-09-17T04:07:26.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ee1c00640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000ba80 tx=0x7f9ed000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0019070 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9ed000cd10 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00146a0 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9edc150110 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.177+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9edc0c64e0 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.178+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9edc0c5ff0 con 0x7f9edc073490 2024-09-17T04:07:26.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.179+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9ed001c070 con 0x7f9edc073490 2024-09-17T04:07:26.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.180+0000 7f9ed9600640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 0x7f9ebc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.180+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f9ed009a940 con 0x7f9edc073490 2024-09-17T04:07:26.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.181+0000 7f9ee1200640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 0x7f9ebc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.181+0000 7f9ee1200640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 0x7f9ebc07b0d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006fd0 tx=0x7f9ec8008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:26.184 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.186+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9ed0062f80 con 0x7f9edc073490 2024-09-17T04:07:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.365+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f9edc1502e0 con 0x7f9edc073490 2024-09-17T04:07:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.365+0000 7f9ed9600640 1 -- 172.21.15.129:0/1620861838 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v70) ==== 59+0+2294 (secure 0 0 0) 0x7f9edc1502e0 con 0x7f9edc073490 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:07:26.365 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:07:26.366 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:07:26.367 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 msgr2=0x7f9ebc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 0x7f9ebc07b0d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006fd0 tx=0x7f9ec8008040 comp rx=0 tx=0).stop 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 msgr2=0x7f9edc14f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f9ed000ba80 tx=0x7f9ed000bf40 comp rx=0 tx=0).stop 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 shutdown_connections 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9ebc078bc0 0x7f9ebc07b0d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9edc0d0380 0x7f9edc14c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9edc0ced00 0x7f9edc1f5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 --2- 172.21.15.129:0/1620861838 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9edc073490 0x7f9edc14f390 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 >> 172.21.15.129:0/1620861838 conn(0x7f9edc06cfc0 msgr2=0x7f9edc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:26.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 shutdown_connections 2024-09-17T04:07:26.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.368+0000 7f9ee3000640 1 -- 172.21.15.129:0/1620861838 wait complete. 2024-09-17T04:07:26.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:07:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.617+0000 7f74f5400640 1 -- 172.21.15.129:0/483653766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 msgr2=0x7f74d4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.617+0000 7f74f5400640 1 --2- 172.21.15.129:0/483653766 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74d4003ec0 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f74e000c4a0 tx=0x7f74d4005830 comp rx=0 tx=0).stop 2024-09-17T04:07:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.617+0000 7f74ef200640 1 -- 172.21.15.129:0/483653766 reap_dead start 2024-09-17T04:07:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.617+0000 7f74f5400640 1 -- 172.21.15.129:0/483653766 shutdown_connections 2024-09-17T04:07:26.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.617+0000 7f74f5400640 1 -- 172.21.15.129:0/483653766 >> 172.21.15.129:0/483653766 conn(0x7f74f00ccc40 msgr2=0x7f74f00cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.618+0000 7f74f5400640 1 -- 172.21.15.129:0/483653766 shutdown_connections 2024-09-17T04:07:26.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.618+0000 7f74f5400640 1 -- 172.21.15.129:0/483653766 wait complete. 2024-09-17T04:07:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.620+0000 7f74f5400640 1 Processor -- start 2024-09-17T04:07:26.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.620+0000 7f74f5400640 1 -- start start 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f015aac0 0x7f74f009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74f01f8d40 0x7f74f00cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f74f00b21d0 con 0x7f74d4003a50 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f74f00b2050 con 0x7f74f015aac0 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.621+0000 7f74f5400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f74f00b2350 con 0x7f74f01f8d40 2024-09-17T04:07:26.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46014/0 (socket says 172.21.15.129:46014) 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 -- 172.21.15.129:0/3972496122 learned_addr learned my addr 172.21.15.129:0/3972496122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ef200640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74f01f8d40 0x7f74f00cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ede00640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f015aac0 0x7f74f009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 -- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74f01f8d40 msgr2=0x7f74f00cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74f01f8d40 0x7f74f00cf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 -- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f015aac0 msgr2=0x7f74f009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f015aac0 0x7f74f009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.622+0000 7f74ee800640 1 -- 172.21.15.129:0/3972496122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74f00cee80 con 0x7f74d4003a50 2024-09-17T04:07:26.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74ee800640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f74e002b310 tx=0x7f74e0006e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00064f0 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f74e002b660 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74f009afa0 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e002ba70 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.623+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74f01f98e0 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.624+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74f00745e0 con 0x7f74d4003a50 2024-09-17T04:07:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.625+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f74e002bc10 con 0x7f74d4003a50 2024-09-17T04:07:26.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.626+0000 7f74e6a00640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 0x7f74d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:26.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.626+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f74e00ba620 con 0x7f74d4003a50 2024-09-17T04:07:26.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.627+0000 7f74ede00640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 0x7f74d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:26.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.628+0000 7f74ede00640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 0x7f74d007b0d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f74f00c9df0 tx=0x7f74dc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:26.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.631+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f74f00cbaf0 con 0x7f74d4003a50 2024-09-17T04:07:26.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.814+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f74f00745e0 con 0x7f74d4003a50 2024-09-17T04:07:26.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.819+0000 7f74e6a00640 1 -- 172.21.15.129:0/3972496122 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v71)=0 v71) ==== 122+0+0 (secure 0 0 0) 0x7f74e0082c60 con 0x7f74d4003a50 2024-09-17T04:07:26.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 msgr2=0x7f74d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 0x7f74d007b0d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f74f00c9df0 tx=0x7f74dc005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:26.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 msgr2=0x7f74f01516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:26.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f74e002b310 tx=0x7f74e0006e20 comp rx=0 tx=0).stop 2024-09-17T04:07:26.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 shutdown_connections 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f74d0078bc0 0x7f74d007b0d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74f01f8d40 0x7f74f00cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74f015aac0 0x7f74f009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 --2- 172.21.15.129:0/3972496122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f74d4003a50 0x7f74f01516e0 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.821+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 >> 172.21.15.129:0/3972496122 conn(0x7f74f00ccc40 msgr2=0x7f74f015c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.822+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 shutdown_connections 2024-09-17T04:07:26.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:26.822+0000 7f74f5400640 1 -- 172.21.15.129:0/3972496122 wait complete. 2024-09-17T04:07:26.845 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:07:26.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:07:26.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:07:27.035 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:07:27.035 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:27.035 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:27.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:07:27.071 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:07:27.071 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:27.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:07:27.108 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:27.109 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:27.146 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:07:27.146 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:07:27.146 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:27.185 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:07:27.186 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:07:27.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 -- 172.21.15.129:0/2834740278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08000705a0 msgr2=0x7f08000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 --2- 172.21.15.129:0/2834740278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08000705a0 0x7f08000b4140 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f07f4005fd0 tx=0x7f07f402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 -- 172.21.15.129:0/2834740278 shutdown_connections 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 --2- 172.21.15.129:0/2834740278 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f08000cd390 0x7f08001609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 --2- 172.21.15.129:0/2834740278 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f08000705a0 0x7f08000b4140 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 -- 172.21.15.129:0/2834740278 >> 172.21.15.129:0/2834740278 conn(0x7f080006d410 msgr2=0x7f08000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 -- 172.21.15.129:0/2834740278 shutdown_connections 2024-09-17T04:07:27.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.407+0000 7f0808600640 1 -- 172.21.15.129:0/2834740278 wait complete. 2024-09-17T04:07:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.409+0000 7f0808600640 1 Processor -- start 2024-09-17T04:07:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.409+0000 7f0808600640 1 -- start start 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 0x7f08000bce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 0x7f0800073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0800160f00 con 0x7f0800157fc0 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f080006f4f0 con 0x7f08000705a0 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0808600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0800161080 con 0x7f080006eed0 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46028/0 (socket says 172.21.15.129:46028) 2024-09-17T04:07:27.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 -- 172.21.15.129:0/3261927924 learned_addr learned my addr 172.21.15.129:0/3261927924 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0805200640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 0x7f0800073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0805c00640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 0x7f08000bce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 -- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 msgr2=0x7f08000bce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 0x7f08000bce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 -- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 msgr2=0x7f0800073f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 0x7f0800073f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0806600640 1 -- 172.21.15.129:0/3261927924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08001f44b0 con 0x7f0800157fc0 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0805200640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 0x7f0800073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.411+0000 7f0805c00640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 0x7f08000bce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f0806600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f07f800e9e0 tx=0x7f07f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f800cdb0 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f07f8004c10 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f8005020 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08001f4790 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.412+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0800157910 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.413+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08000b4140 con 0x7f0800157fc0 2024-09-17T04:07:27.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.414+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f07f80051c0 con 0x7f0800157fc0 2024-09-17T04:07:27.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.415+0000 7f07fd600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 0x7f07e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.416+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f07f809b940 con 0x7f0800157fc0 2024-09-17T04:07:27.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.416+0000 7f0805c00640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 0x7f07e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.417+0000 7f0805c00640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 0x7f07e007b0a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f07f4033000 tx=0x7f07f4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:27.418 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.420+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f07f8063f80 con 0x7f0800157fc0 2024-09-17T04:07:27.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.638+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f080006a140 con 0x7f0800157fc0 2024-09-17T04:07:27.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.639+0000 7f07fd600640 1 -- 172.21.15.129:0/3261927924 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+3927 (secure 0 0 0) 0x7f07f8063880 con 0x7f0800157fc0 2024-09-17T04:07:27.638 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:27.638 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":242,"btime":"2024-09-17T04:07:20:893898+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9167,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/684288915","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":684288915},{"type":"v1","addr":"172.21.15.129:6834","nonce":684288915}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":231},{"gid":9443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2544955760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":2544955760},{"type":"v1","addr":"172.21.15.181:6837","nonce":2544955760}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242},{"gid":10831,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/1019511945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":1019511945},{"type":"v1","addr":"172.21.15.181:6836","nonce":1019511945}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":230},{"gid":11985,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/1873359903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":1873359903},{"type":"v1","addr":"172.21.15.129:6835","nonce":1873359903}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":238}],"filesystems":[]} 2024-09-17T04:07:27.639 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 242 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.641+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 msgr2=0x7f07e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.641+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 0x7f07e007b0a0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f07f4033000 tx=0x7f07f4004730 comp rx=0 tx=0).stop 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.641+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 msgr2=0x7f0800074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.641+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f07f800e9e0 tx=0x7f07f800eea0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 shutdown_connections 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f07e0078b90 0x7f07e007b0a0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0800157fc0 0x7f0800074440 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08000705a0 0x7f0800073f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 --2- 172.21.15.129:0/3261927924 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f080006eed0 0x7f08000bce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 >> 172.21.15.129:0/3261927924 conn(0x7f080006d410 msgr2=0x7f08000c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 shutdown_connections 2024-09-17T04:07:27.642 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.642+0000 7f0808600640 1 -- 172.21.15.129:0/3261927924 wait complete. 2024-09-17T04:07:27.664 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:07:27.664 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:07:27.664 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:07:27.664 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:27.665 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:07:27.665 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:07:27.665 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:07:27.665 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:27.665 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:07:27.665 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-17T04:07:27.666 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:07:27.666 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:27.666 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:07:27.666 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-17T04:07:27.666 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:07:27.667 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:27.687 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:07:27.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:07:27.693 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:07:27.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:07:27.694 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:07:27.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:07:27.695 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:07:27.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:07:27.737 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:07:27.737 DEBUG:teuthology.parallel:result is None 2024-09-17T04:07:27.738 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:07:27.738 DEBUG:teuthology.parallel:result is None 2024-09-17T04:07:27.739 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:07:27.739 DEBUG:teuthology.parallel:result is None 2024-09-17T04:07:27.741 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:07:27.741 DEBUG:teuthology.parallel:result is None 2024-09-17T04:07:27.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:07:27.940 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:07:27.940 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:07:27.941 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:07:27.941 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:07:27.941 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T04:07:27.942+0000 7f752d400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 -- 172.21.15.129:0/3621238914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00d1850 msgr2=0x7f3ab00b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/3621238914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00d1850 0x7f3ab00b3440 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0009830 tx=0x7f3aa002a5c0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 -- 172.21.15.129:0/3621238914 shutdown_connections 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/3621238914 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00d1850 0x7f3ab00b3440 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 -- 172.21.15.129:0/3621238914 >> 172.21.15.129:0/3621238914 conn(0x7f3ab008bff0 msgr2=0x7f3ab009d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:27.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 -- 172.21.15.129:0/3621238914 shutdown_connections 2024-09-17T04:07:27.976 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.976+0000 7f3ab4a00640 1 -- 172.21.15.129:0/3621238914 wait complete. 2024-09-17T04:07:27.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.978+0000 7f3ab4a00640 1 Processor -- start 2024-09-17T04:07:27.977 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.978+0000 7f3ab4a00640 1 -- start start 2024-09-17T04:07:27.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.979+0000 7f3ab4a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3ab4a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ab00d1850 0x7f3ab006f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3ab4a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ab00d4ba0 0x7f3ab006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3ab4a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00d3430 con 0x7f3ab00b9af0 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3ab4a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab00d32b0 con 0x7f3ab00d1850 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3ab4a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3ab00cf7c0 con 0x7f3ab00d4ba0 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46082/0 (socket says 172.21.15.129:46082) 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 -- 172.21.15.129:0/51716538 learned_addr learned my addr 172.21.15.129:0/51716538 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aade00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ab00d1850 0x7f3ab006f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aaf200640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ab00d4ba0 0x7f3ab006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 -- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ab00d4ba0 msgr2=0x7f3ab006f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ab00d4ba0 0x7f3ab006f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 -- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ab00d1850 msgr2=0x7f3ab006f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ab00d1850 0x7f3ab006f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.980+0000 7f3aae800640 1 -- 172.21.15.129:0/51716538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab00a1860 con 0x7f3ab00b9af0 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3aae800640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f3aa002abb0 tx=0x7f3aa0033c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:27.982 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0031b60 con 0x7f3ab00b9af0 2024-09-17T04:07:27.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3aa0031df0 con 0x7f3ab00b9af0 2024-09-17T04:07:27.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab00a07c0 con 0x7f3ab00b9af0 2024-09-17T04:07:27.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa0005280 con 0x7f3ab00b9af0 2024-09-17T04:07:27.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.981+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab00a0d80 con 0x7f3ab00b9af0 2024-09-17T04:07:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.982+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab00cfca0 con 0x7f3ab00b9af0 2024-09-17T04:07:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.983+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3aa0005420 con 0x7f3ab00b9af0 2024-09-17T04:07:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.984+0000 7f3aa6a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 0x7f3a9007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.984+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(252..252 src has 1..252) ==== 5029+0+0 (secure 0 0 0) 0x7f3aa00bbcd0 con 0x7f3ab00b9af0 2024-09-17T04:07:27.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.985+0000 7f3aade00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 0x7f3a9007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.985+0000 7f3aade00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 0x7f3a9007b0a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c000f60 tx=0x7f3a9c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:27.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:27.989+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3aa003f030 con 0x7f3ab00b9af0 2024-09-17T04:07:28.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.183+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f3ab00b4300 con 0x7f3ab00b9af0 2024-09-17T04:07:28.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.946+0000 7f3aa6a00640 1 -- 172.21.15.129:0/51716538 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v254) ==== 97+0+0 (secure 0 0 0) 0x7f3aa0031770 con 0x7f3ab00b9af0 2024-09-17T04:07:28.945 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:07:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.949+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 msgr2=0x7f3a9007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.949+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 0x7f3a9007b0a0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c000f60 tx=0x7f3a9c009040 comp rx=0 tx=0).stop 2024-09-17T04:07:28.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.949+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 msgr2=0x7f3ab015c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.949+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f3aa002abb0 tx=0x7f3aa0033c10 comp rx=0 tx=0).stop 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.949+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 shutdown_connections 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3a90078b90 0x7f3a9007b0a0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ab00d4ba0 0x7f3ab006f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ab00d1850 0x7f3ab006f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 --2- 172.21.15.129:0/51716538 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ab00b9af0 0x7f3ab015c420 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:28.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 >> 172.21.15.129:0/51716538 conn(0x7f3ab008bff0 msgr2=0x7f3ab00d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:28.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 shutdown_connections 2024-09-17T04:07:28.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:28.950+0000 7f3ab4a00640 1 -- 172.21.15.129:0/51716538 wait complete. 2024-09-17T04:07:28.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 -- 172.21.15.129:0/2891024652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead8003a00 msgr2=0x7fead8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 --2- 172.21.15.129:0/2891024652 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fead8003a00 0x7fead8003e70 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7feae4009b50 tx=0x7fead8005830 comp rx=0 tx=0).stop 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf3400640 1 -- 172.21.15.129:0/2891024652 reap_dead start 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 -- 172.21.15.129:0/2891024652 shutdown_connections 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 -- 172.21.15.129:0/2891024652 >> 172.21.15.129:0/2891024652 conn(0x7feaf406d410 msgr2=0x7feaf40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 -- 172.21.15.129:0/2891024652 shutdown_connections 2024-09-17T04:07:29.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.201+0000 7feaf9800640 1 -- 172.21.15.129:0/2891024652 wait complete. 2024-09-17T04:07:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.204+0000 7feaf9800640 1 Processor -- start 2024-09-17T04:07:29.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.204+0000 7feaf9800640 1 -- start start 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead8003a00 0x7feaf41f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feaf41f76e0 0x7feaf409d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7feaf40b4140 con 0x7feaf41f3370 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7feaf406f4f0 con 0x7fead8003a00 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7feaf40b42c0 con 0x7feaf41f76e0 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.205+0000 7feaf2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46106/0 (socket says 172.21.15.129:46106) 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 -- 172.21.15.129:0/169274242 learned_addr learned my addr 172.21.15.129:0/169274242 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf3400640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feaf41f76e0 0x7feaf409d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2a00640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead8003a00 0x7feaf41f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.206 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 -- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feaf41f76e0 msgr2=0x7feaf409d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feaf41f76e0 0x7feaf409d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 -- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead8003a00 msgr2=0x7feaf41f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead8003a00 0x7feaf41f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 -- 172.21.15.129:0/169274242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf415b550 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaf2000640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7feae0000fb0 tx=0x7feae00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.206+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0015070 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.207+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feae0011900 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.207+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf415b830 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.207+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae0011d30 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.207+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf41f3780 con 0x7feaf41f3370 2024-09-17T04:07:29.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.208+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf40ccc80 con 0x7feaf41f3370 2024-09-17T04:07:29.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.209+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7feae0018070 con 0x7feaf41f3370 2024-09-17T04:07:29.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.210+0000 7feaeaa00640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 0x7fead407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.210+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 4857+0+0 (secure 0 0 0) 0x7feae009b6a0 con 0x7feaf41f3370 2024-09-17T04:07:29.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.211+0000 7feaf2a00640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 0x7fead407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.211+0000 7feaf2a00640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 0x7fead407b0d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7feae4009b50 tx=0x7feae402d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:29.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.215+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7feae00637e0 con 0x7feaf41f3370 2024-09-17T04:07:29.466 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.467+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7feaf406a140 con 0x7feaf41f3370 2024-09-17T04:07:29.467 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.468+0000 7feaeaa00640 1 -- 172.21.15.129:0/169274242 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) ==== 80+0+5304 (secure 0 0 0) 0x7feae0068210 con 0x7feaf41f3370 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:07:29.468 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:07:29.470 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.471 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:07:29.472 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.473 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.474 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.475 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.477 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:07:29.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.479 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.480 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.481 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.482 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:07:29.482 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 msgr2=0x7fead407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 0x7fead407b0d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7feae4009b50 tx=0x7feae402d040 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 msgr2=0x7feaf41f71a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7feae0000fb0 tx=0x7feae00039c0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 shutdown_connections 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fead4078bc0 0x7fead407b0d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7feaf41f76e0 0x7feaf409d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7feaf41f3370 0x7feaf41f71a0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 --2- 172.21.15.129:0/169274242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fead8003a00 0x7feaf41f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 >> 172.21.15.129:0/169274242 conn(0x7feaf406d410 msgr2=0x7feaf40cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:29.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 shutdown_connections 2024-09-17T04:07:29.484 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.471+0000 7feaf9800640 1 -- 172.21.15.129:0/169274242 wait complete. 2024-09-17T04:07:29.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:07:29.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 -- 172.21.15.129:0/722680744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540cd390 msgr2=0x7f6854070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 --2- 172.21.15.129:0/722680744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540cd390 0x7f6854070100 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f68540ce800 tx=0x7f684c02aaa0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 -- 172.21.15.129:0/722680744 shutdown_connections 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 --2- 172.21.15.129:0/722680744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540cd390 0x7f6854070100 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 --2- 172.21.15.129:0/722680744 >> v2:172.21.15.181:3300/0 conn(0x7f685415da00 0x7f68540b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 -- 172.21.15.129:0/722680744 >> 172.21.15.129:0/722680744 conn(0x7f685406d410 msgr2=0x7f68540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 -- 172.21.15.129:0/722680744 shutdown_connections 2024-09-17T04:07:29.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.720+0000 7f685a200640 1 -- 172.21.15.129:0/722680744 wait complete. 2024-09-17T04:07:29.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.722+0000 7f685a200640 1 Processor -- start 2024-09-17T04:07:29.721 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.722+0000 7f685a200640 1 -- start start 2024-09-17T04:07:29.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 0x7f6854156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 0x7f68540c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.724 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f68540709d0 con 0x7f68540c8630 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6854070850 con 0x7f68540b46c0 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f685a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6854070b50 con 0x7f685415da00 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46132/0 (socket says 172.21.15.129:46132) 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 -- 172.21.15.129:0/1891147963 learned_addr learned my addr 172.21.15.129:0/1891147963 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853400640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 0x7f6854156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f684aa00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 0x7f68540c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 -- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 msgr2=0x7f68540c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 0x7f68540c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 -- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 msgr2=0x7f6854156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.725 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.724+0000 7f6853e00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 0x7f6854156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f6853e00640 1 -- 172.21.15.129:0/1891147963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68540cc210 con 0x7f68540c8630 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f6853400640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 0x7f6854156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f684aa00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 0x7f68540c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f6853e00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f684c005fd0 tx=0x7f684c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c044070 con 0x7f68540c8630 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f684c004250 con 0x7f68540c8630 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684c0046a0 con 0x7f68540c8630 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68540cb690 con 0x7f68540c8630 2024-09-17T04:07:29.726 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.725+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68540cbc50 con 0x7f68540c8630 2024-09-17T04:07:29.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.726+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68540b2470 con 0x7f68540c8630 2024-09-17T04:07:29.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.727+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f684c047070 con 0x7f68540c8630 2024-09-17T04:07:29.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.728+0000 7f684be00640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 0x7f683007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:29.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.729+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 4857+0+0 (secure 0 0 0) 0x7f684c0bb7f0 con 0x7f68540c8630 2024-09-17T04:07:29.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.729+0000 7f6853400640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 0x7f683007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:29.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.730+0000 7f6853400640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 0x7f683007b0a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6844009aa0 tx=0x7f6844005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:29.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.733+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f68540b2470 con 0x7f68540c8630 2024-09-17T04:07:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.917+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f685406a140 con 0x7f68540c8630 2024-09-17T04:07:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.917+0000 7f684be00640 1 -- 172.21.15.129:0/1891147963 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v256) ==== 84+0+65 (secure 0 0 0) 0x7f684c032030 con 0x7f68540c8630 2024-09-17T04:07:29.916 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:07:29.917 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.919+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 msgr2=0x7f683007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.919+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 0x7f683007b0a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6844009aa0 tx=0x7f6844005db0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.919+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 msgr2=0x7f68540c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.919+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f684c005fd0 tx=0x7f684c003ce0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 shutdown_connections 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6830078b90 0x7f683007b0a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f68540c8630 0x7f68540c8a40 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f685415da00 0x7f68540c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 --2- 172.21.15.129:0/1891147963 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68540b46c0 0x7f6854156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:29.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 >> 172.21.15.129:0/1891147963 conn(0x7f685406d410 msgr2=0x7f68540b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:29.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 shutdown_connections 2024-09-17T04:07:29.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:29.920+0000 7f685a200640 1 -- 172.21.15.129:0/1891147963 wait complete. 2024-09-17T04:07:29.945 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:07:29.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:07:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 -- 172.21.15.129:0/1180907362 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b80cae00 msgr2=0x7fd1b81452b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/1180907362 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b80cae00 0x7fd1b81452b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009860 tx=0x7fd1a802a5c0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 -- 172.21.15.129:0/1180907362 shutdown_connections 2024-09-17T04:07:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/1180907362 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b80cae00 0x7fd1b81452b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 -- 172.21.15.129:0/1180907362 >> 172.21.15.129:0/1180907362 conn(0x7fd1b80b1c50 msgr2=0x7fd1b80b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:30.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 -- 172.21.15.129:0/1180907362 shutdown_connections 2024-09-17T04:07:30.171 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.171+0000 7fd1bcc00640 1 -- 172.21.15.129:0/1180907362 wait complete. 2024-09-17T04:07:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.173+0000 7fd1bcc00640 1 Processor -- start 2024-09-17T04:07:30.172 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.173+0000 7fd1bcc00640 1 -- start start 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b8072c60 0x7fd1b813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1b80cc990 0x7fd1b81f86d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8167c40 con 0x7fd1b81f8c10 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b8167ac0 con 0x7fd1b80cc990 2024-09-17T04:07:30.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1bcc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b8167dc0 con 0x7fd1b8072c60 2024-09-17T04:07:30.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:30.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46142/0 (socket says 172.21.15.129:46142) 2024-09-17T04:07:30.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 -- 172.21.15.129:0/3360296702 learned_addr learned my addr 172.21.15.129:0/3360296702 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:30.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7400640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b8072c60 0x7fd1b813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:30.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1aea00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1b80cc990 0x7fd1b81f86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:30.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 -- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b8072c60 msgr2=0x7fd1b813fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:30.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b8072c60 0x7fd1b813fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 -- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1b80cc990 msgr2=0x7fd1b81f86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:30.176 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1b80cc990 0x7fd1b81f86d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.175+0000 7fd1b7e00640 1 -- 172.21.15.129:0/3360296702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b80b6010 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1b7e00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000c730 tx=0x7fd1b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0003b70 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd1b0003d10 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000aab0 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b8140760 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.176+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b8140370 con 0x7fd1b81f8c10 2024-09-17T04:07:30.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.177+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b80b58f0 con 0x7fd1b81f8c10 2024-09-17T04:07:30.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.178+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd1b000ac50 con 0x7fd1b81f8c10 2024-09-17T04:07:30.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.179+0000 7fd1afe00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 0x7fd19407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:30.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.179+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(256..256 src has 1..256) ==== 4857+0+0 (secure 0 0 0) 0x7fd1b009a680 con 0x7fd1b81f8c10 2024-09-17T04:07:30.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.180+0000 7fd1b7400640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 0x7fd19407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:30.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.181+0000 7fd1b7400640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 0x7fd19407b0d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8034000 tx=0x7fd1a8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:30.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.184+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd1b0062720 con 0x7fd1b81f8c10 2024-09-17T04:07:30.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.366+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd1b81f95c0 con 0x7fd1b81f8c10 2024-09-17T04:07:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.950+0000 7fd1afe00640 1 -- 172.21.15.129:0/3360296702 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v257) ==== 138+0+0 (secure 0 0 0) 0x7fd1b0067150 con 0x7fd1b81f8c10 2024-09-17T04:07:30.950 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.953+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 msgr2=0x7fd19407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.953+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 0x7fd19407b0d0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8034000 tx=0x7fd1a8004730 comp rx=0 tx=0).stop 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.953+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 msgr2=0x7fd1b80ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.953+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000c730 tx=0x7fd1b000cbf0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 shutdown_connections 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd194078bc0 0x7fd19407b0d0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd1b81f8c10 0x7fd1b80ae2e0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1b80cc990 0x7fd1b81f86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 --2- 172.21.15.129:0/3360296702 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd1b8072c60 0x7fd1b813fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 >> 172.21.15.129:0/3360296702 conn(0x7fd1b80b1c50 msgr2=0x7fd1b8074640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 shutdown_connections 2024-09-17T04:07:30.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:30.954+0000 7fd1bcc00640 1 -- 172.21.15.129:0/3360296702 wait complete. 2024-09-17T04:07:30.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:07:31.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 -- 172.21.15.129:0/1803231277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c06ee70 msgr2=0x7f8e5c06f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:31.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1803231277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c06ee70 0x7f8e5c06f280 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0c7180 tx=0x7f8e4802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 -- 172.21.15.129:0/1803231277 shutdown_connections 2024-09-17T04:07:31.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1803231277 >> v2:172.21.15.181:3300/0 conn(0x7f8e5c072ee0 0x7f8e5c0732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1803231277 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c06ee70 0x7f8e5c06f280 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1803231277 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c156f20 0x7f8e5c160d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.215+0000 7f8e60e00640 1 -- 172.21.15.129:0/1803231277 >> 172.21.15.129:0/1803231277 conn(0x7f8e5c0d0120 msgr2=0x7f8e5c0d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:31.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.216+0000 7f8e60e00640 1 -- 172.21.15.129:0/1803231277 shutdown_connections 2024-09-17T04:07:31.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.216+0000 7f8e60e00640 1 -- 172.21.15.129:0/1803231277 wait complete. 2024-09-17T04:07:31.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.218+0000 7f8e60e00640 1 Processor -- start 2024-09-17T04:07:31.217 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.218+0000 7f8e60e00640 1 -- start start 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 0x7f8e5c14b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 0x7f8e5c148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c0739b0 con 0x7f8e5c072ee0 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c073830 con 0x7f8e5c156f20 2024-09-17T04:07:31.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.219+0000 7f8e60e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8e5c073b30 con 0x7f8e5c06ee70 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46156/0 (socket says 172.21.15.129:46156) 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 -- 172.21.15.129:0/1108346981 learned_addr learned my addr 172.21.15.129:0/1108346981 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5b400640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 0x7f8e5c14b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5be00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 0x7f8e5c148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 -- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 msgr2=0x7f8e5c14b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 0x7f8e5c14b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 -- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 msgr2=0x7f8e5c148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 0x7f8e5c148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5aa00640 1 -- 172.21.15.129:0/1108346981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c1491a0 con 0x7f8e5c072ee0 2024-09-17T04:07:31.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5b400640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 0x7f8e5c14b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.220+0000 7f8e5be00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 0x7f8e5c148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e5aa00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f8e48034040 tx=0x7f8e48003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4803e030 con 0x7f8e5c072ee0 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8e48003c20 con 0x7f8e5c072ee0 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c14bcc0 con 0x7f8e5c072ee0 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48035040 con 0x7f8e5c072ee0 2024-09-17T04:07:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.221+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5c14c280 con 0x7f8e5c072ee0 2024-09-17T04:07:31.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.222+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e5c149470 con 0x7f8e5c072ee0 2024-09-17T04:07:31.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.223+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8e48041030 con 0x7f8e5c072ee0 2024-09-17T04:07:31.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.224+0000 7f8e53400640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 0x7f8e3c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:31.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.224+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(257..257 src has 1..257) ==== 5227+0+0 (secure 0 0 0) 0x7f8e480bb4b0 con 0x7f8e5c072ee0 2024-09-17T04:07:31.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.225+0000 7f8e5b400640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 0x7f8e3c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:31.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.226+0000 7f8e5b400640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 0x7f8e3c07b0a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0afb70 tx=0x7f8e4c00b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:31.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.229+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8e48083470 con 0x7f8e5c072ee0 2024-09-17T04:07:31.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.414+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f8e5c14ba50 con 0x7f8e5c072ee0 2024-09-17T04:07:31.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.959+0000 7f8e53400640 1 -- 172.21.15.129:0/1108346981 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v258) ==== 170+0+0 (secure 0 0 0) 0x7f8e48087ea0 con 0x7f8e5c072ee0 2024-09-17T04:07:31.959 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:07:31.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 msgr2=0x7f8e3c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:31.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 0x7f8e3c07b0a0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0afb70 tx=0x7f8e4c00b2e0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 msgr2=0x7f8e5c0c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f8e48034040 tx=0x7f8e48003820 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 shutdown_connections 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8e3c078b90 0x7f8e3c07b0a0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8e5c156f20 0x7f8e5c148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8e5c072ee0 0x7f8e5c0c9a70 secure :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f8e48034040 tx=0x7f8e48003820 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 --2- 172.21.15.129:0/1108346981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8e5c06ee70 0x7f8e5c14b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:31.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.963+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 >> 172.21.15.129:0/1108346981 conn(0x7f8e5c0d0120 msgr2=0x7f8e5c15eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.965+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 shutdown_connections 2024-09-17T04:07:31.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:31.965+0000 7f8e60e00640 1 -- 172.21.15.129:0/1108346981 wait complete. 2024-09-17T04:07:31.999 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:07:32.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 -- 172.21.15.129:0/3405053504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 msgr2=0x7fb3580af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:32.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 --2- 172.21.15.129:0/3405053504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580af480 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb348009830 tx=0x7fb34802a460 comp rx=0 tx=0).stop 2024-09-17T04:07:32.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 -- 172.21.15.129:0/3405053504 shutdown_connections 2024-09-17T04:07:32.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 --2- 172.21.15.129:0/3405053504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580af480 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 -- 172.21.15.129:0/3405053504 >> 172.21.15.129:0/3405053504 conn(0x7fb3580ccc40 msgr2=0x7fb3580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:32.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 -- 172.21.15.129:0/3405053504 shutdown_connections 2024-09-17T04:07:32.223 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.223+0000 7fb35ec00640 1 -- 172.21.15.129:0/3405053504 wait complete. 2024-09-17T04:07:32.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.225+0000 7fb35ec00640 1 Processor -- start 2024-09-17T04:07:32.224 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.225+0000 7fb35ec00640 1 -- start start 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.226+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.226+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3580b1c40 0x7fb358150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.226+0000 7fb35ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.226+0000 7fb35ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580b3cb0 con 0x7fb3580b3010 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb3580b3b30 con 0x7fb358074800 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb3580b3e30 con 0x7fb3580b1c40 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb357e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46168/0 (socket says 172.21.15.129:46168) 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 -- 172.21.15.129:0/4062097843 learned_addr learned my addr 172.21.15.129:0/4062097843 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb34f400640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3580b1c40 0x7fb358150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:32.230 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 -- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3580b1c40 msgr2=0x7fb358150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3580b1c40 0x7fb358150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 -- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 msgr2=0x7fb3580cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb35cc00640 1 -- 172.21.15.129:0/4062097843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3580733c0 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.227+0000 7fb357e00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb35cc00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c7f0 tx=0x7fb35000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350018070 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb350006b30 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3580736a0 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb350005070 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.228+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb35815ba60 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.229+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb358073130 con 0x7fb3580b3010 2024-09-17T04:07:32.231 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.231+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb350005210 con 0x7fb3580b3010 2024-09-17T04:07:32.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.232+0000 7fb354c00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 0x7fb33407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:32.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.232+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(258..258 src has 1..258) ==== 5609+0+0 (secure 0 0 0) 0x7fb35009a310 con 0x7fb3580b3010 2024-09-17T04:07:32.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.232+0000 7fb357e00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 0x7fb33407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:32.232 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.233+0000 7fb357e00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 0x7fb33407b0a0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fb348034000 tx=0x7fb34802c040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:32.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.237+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb3500620d0 con 0x7fb3580b3010 2024-09-17T04:07:32.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.454+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb35809bab0 con 0x7fb3580b3010 2024-09-17T04:07:32.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.976+0000 7fb354c00640 1 -- 172.21.15.129:0/4062097843 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 38 and data pool 39 v245) ==== 174+0+0 (secure 0 0 0) 0x7fb35809bab0 con 0x7fb3580b3010 2024-09-17T04:07:32.979 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 38 and data pool 39 2024-09-17T04:07:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.985+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 msgr2=0x7fb33407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.985+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 0x7fb33407b0a0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fb348034000 tx=0x7fb34802c040 comp rx=0 tx=0).stop 2024-09-17T04:07:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.985+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 msgr2=0x7fb3581e9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:32.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.985+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7fb35000c7f0 tx=0x7fb35000ccb0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 shutdown_connections 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb334078b90 0x7fb33407b0a0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb3580b3010 0x7fb3581e9420 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb3580b1c40 0x7fb358150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 --2- 172.21.15.129:0/4062097843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb358074800 0x7fb3580cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.986+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 >> 172.21.15.129:0/4062097843 conn(0x7fb3580ccc40 msgr2=0x7fb3580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:32.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.987+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 shutdown_connections 2024-09-17T04:07:32.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:32.988+0000 7fb35ec00640 1 -- 172.21.15.129:0/4062097843 wait complete. 2024-09-17T04:07:33.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 -- 172.21.15.129:0/1274978614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2400705a0 msgr2=0x7fc2400b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 --2- 172.21.15.129:0/1274978614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2400705a0 0x7fc2400b4140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc230005fd0 tx=0x7fc23002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 -- 172.21.15.129:0/1274978614 shutdown_connections 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 --2- 172.21.15.129:0/1274978614 >> v2:172.21.15.181:3300/0 conn(0x7fc2400cd390 0x7fc2401609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 --2- 172.21.15.129:0/1274978614 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc2400705a0 0x7fc2400b4140 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.249+0000 7fc245c00640 1 -- 172.21.15.129:0/1274978614 >> 172.21.15.129:0/1274978614 conn(0x7fc24006d410 msgr2=0x7fc2400b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:33.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.250+0000 7fc245c00640 1 -- 172.21.15.129:0/1274978614 shutdown_connections 2024-09-17T04:07:33.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.250+0000 7fc245c00640 1 -- 172.21.15.129:0/1274978614 wait complete. 2024-09-17T04:07:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.252+0000 7fc245c00640 1 Processor -- start 2024-09-17T04:07:33.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.252+0000 7fc245c00640 1 -- start start 2024-09-17T04:07:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 0x7fc24015d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2400705a0 0x7fc2400b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc240160f00 con 0x7fc24015da20 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc24006f4f0 con 0x7fc2400705a0 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.253+0000 7fc245c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc240161080 con 0x7fc24006eed0 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46176/0 (socket says 172.21.15.129:46176) 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 -- 172.21.15.129:0/1037703228 learned_addr learned my addr 172.21.15.129:0/1037703228 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23ea00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 0x7fc24015d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc237e00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2400705a0 0x7fc2400b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 -- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 msgr2=0x7fc24015d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 0x7fc24015d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 -- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2400705a0 msgr2=0x7fc2400b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2400705a0 0x7fc2400b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 -- 172.21.15.129:0/1037703228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2401f9b70 con 0x7fc24015da20 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23ea00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 0x7fc24015d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:33.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.254+0000 7fc23f400640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7fc23800ea70 tx=0x7fc23800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.255+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23800ce40 con 0x7fc24015da20 2024-09-17T04:07:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.255+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc238004c10 con 0x7fc24015da20 2024-09-17T04:07:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.255+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2401f9e50 con 0x7fc24015da20 2024-09-17T04:07:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.255+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc238005020 con 0x7fc24015da20 2024-09-17T04:07:33.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.255+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2400caf50 con 0x7fc24015da20 2024-09-17T04:07:33.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.256+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2400b2310 con 0x7fc24015da20 2024-09-17T04:07:33.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.257+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc2380051c0 con 0x7fc24015da20 2024-09-17T04:07:33.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.259+0000 7fc236a00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 0x7fc21c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.259+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 5709+0+0 (secure 0 0 0) 0x7fc23809bb60 con 0x7fc24015da20 2024-09-17T04:07:33.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.259+0000 7fc23ea00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 0x7fc21c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.260+0000 7fc23ea00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 0x7fc21c07b0a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc230033000 tx=0x7fc230004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:33.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.263+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc2400b2310 con 0x7fc24015da20 2024-09-17T04:07:33.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.447+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2400b4140 con 0x7fc24015da20 2024-09-17T04:07:33.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.448+0000 7fc236a00640 1 -- 172.21.15.129:0/1037703228 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+16804 (secure 0 0 0) 0x7fc2380638b0 con 0x7fc24015da20 2024-09-17T04:07:33.447 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:33.447 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":259,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:32.455459+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:07:30.379067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":39,"pool_name":"cephfs_data","create_time":"2024-09-17T04:07:31.426899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 msgr2=0x7fc21c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 0x7fc21c07b0a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fc230033000 tx=0x7fc230004730 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 msgr2=0x7fc2400b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7fc23800ea70 tx=0x7fc23800ef30 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 shutdown_connections 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc21c078b90 0x7fc21c07b0a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc24015da20 0x7fc2400b1410 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc2400705a0 0x7fc2400b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 --2- 172.21.15.129:0/1037703228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc24006eed0 0x7fc24015d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 >> 172.21.15.129:0/1037703228 conn(0x7fc24006d410 msgr2=0x7fc2400c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.450+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 shutdown_connections 2024-09-17T04:07:33.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.451+0000 7fc245c00640 1 -- 172.21.15.129:0/1037703228 wait complete. 2024-09-17T04:07:33.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:33.695 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 -- 172.21.15.129:0/3094901440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 msgr2=0x7f3ec4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 --2- 172.21.15.129:0/3094901440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ec4004c10 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0016c0 tx=0x7f3ec4005830 comp rx=0 tx=0).stop 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 -- 172.21.15.129:0/3094901440 shutdown_connections 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 --2- 172.21.15.129:0/3094901440 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ec4004c10 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 -- 172.21.15.129:0/3094901440 >> 172.21.15.129:0/3094901440 conn(0x7f3ee006cfc0 msgr2=0x7f3ee006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 -- 172.21.15.129:0/3094901440 shutdown_connections 2024-09-17T04:07:33.696 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.696+0000 7f3edfe00640 1 -- 172.21.15.129:0/3094901440 wait complete. 2024-09-17T04:07:33.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.698+0000 7f3edfe00640 1 Processor -- start 2024-09-17T04:07:33.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.698+0000 7f3edfe00640 1 -- start start 2024-09-17T04:07:33.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ee013f4c0 0x7f3ee0140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 0x7f3ee0141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee00b4c70 con 0x7f3ec4004800 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee00b4af0 con 0x7f3ee013e070 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edfe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3ee00b4df0 con 0x7f3ee013f4c0 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46186/0 (socket says 172.21.15.129:46186) 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edea00640 1 -- 172.21.15.129:0/380145568 learned_addr learned my addr 172.21.15.129:0/380145568 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3ede000640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ee013f4c0 0x7f3ee0140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.700+0000 7f3edf400640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 0x7f3ee0141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 -- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ee013f4c0 msgr2=0x7f3ee0140d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ee013f4c0 0x7f3ee0140d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 -- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 msgr2=0x7f3ee0141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 0x7f3ee0141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 -- 172.21.15.129:0/380145568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee013fc80 con 0x7f3ec4004800 2024-09-17T04:07:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edf400640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 0x7f3ee0141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:33.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edea00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0011a40 tx=0x7f3ed0011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed001e070 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3ed000fbb0 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee013ff60 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0019410 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.701+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee0140520 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.702+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee00d05b0 con 0x7f3ec4004800 2024-09-17T04:07:33.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.704+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3ed0021070 con 0x7f3ec4004800 2024-09-17T04:07:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.704+0000 7f3ed7400640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 0x7f3ec007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.705+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(259..259 src has 1..259) ==== 5709+0+0 (secure 0 0 0) 0x7f3ed009b440 con 0x7f3ec4004800 2024-09-17T04:07:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.705+0000 7f3ede000640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 0x7f3ec007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:33.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.706+0000 7f3ede000640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 0x7f3ec007b0d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc032000 tx=0x7f3ecc001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:33.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.709+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3ed0063220 con 0x7f3ec4004800 2024-09-17T04:07:33.891 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.892+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ee006a140 con 0x7f3ec4004800 2024-09-17T04:07:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.893+0000 7f3ed7400640 1 -- 172.21.15.129:0/380145568 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) ==== 74+0+16804 (secure 0 0 0) 0x7f3ed0067c50 con 0x7f3ec4004800 2024-09-17T04:07:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:33.893 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":259,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:32.455459+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:07:30.379067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":39,"pool_name":"cephfs_data","create_time":"2024-09-17T04:07:31.426899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 msgr2=0x7f3ec007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 0x7f3ec007b0d0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc032000 tx=0x7f3ecc001190 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 msgr2=0x7f3ee0156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0011a40 tx=0x7f3ed0011f00 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 shutdown_connections 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ec0078bc0 0x7f3ec007b0d0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ee013e070 0x7f3ee0141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ee013f4c0 0x7f3ee0140d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 --2- 172.21.15.129:0/380145568 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ec4004800 0x7f3ee0156ac0 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.895+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 >> 172.21.15.129:0/380145568 conn(0x7f3ee006cfc0 msgr2=0x7f3ee015e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.896+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 shutdown_connections 2024-09-17T04:07:33.896 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:33.896+0000 7f3edfe00640 1 -- 172.21.15.129:0/380145568 wait complete. 2024-09-17T04:07:33.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.144+0000 7fcc77800640 1 -- 172.21.15.129:0/1409661354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc700b3010 msgr2=0x7fcc700b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.144+0000 7fcc77800640 1 --2- 172.21.15.129:0/1409661354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc700b3010 0x7fcc700b3420 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7fcc68005fd0 tx=0x7fcc6802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 -- 172.21.15.129:0/1409661354 shutdown_connections 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 --2- 172.21.15.129:0/1409661354 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc700b3010 0x7fcc700b3420 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 --2- 172.21.15.129:0/1409661354 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc70074800 0x7fcc700af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:34.144 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 -- 172.21.15.129:0/1409661354 >> 172.21.15.129:0/1409661354 conn(0x7fcc700ccc40 msgr2=0x7fcc700cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:34.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 -- 172.21.15.129:0/1409661354 shutdown_connections 2024-09-17T04:07:34.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.145+0000 7fcc77800640 1 -- 172.21.15.129:0/1409661354 wait complete. 2024-09-17T04:07:34.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.147+0000 7fcc77800640 1 Processor -- start 2024-09-17T04:07:34.146 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.147+0000 7fcc77800640 1 -- start start 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc700b1c40 0x7fcc701e9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc700b3010 0x7fcc701e9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcc700b3ae0 con 0x7fcc70074800 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcc700b3960 con 0x7fcc700b1c40 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.148+0000 7fcc77800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcc700b3c60 con 0x7fcc700b3010 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46456/0 (socket says 172.21.15.129:46456) 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 -- 172.21.15.129:0/1663459929 learned_addr learned my addr 172.21.15.129:0/1663459929 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc75800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc700b3010 0x7fcc701e9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:34.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc6fe00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc700b1c40 0x7fcc701e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:34.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 -- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc700b3010 msgr2=0x7fcc701e9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc700b3010 0x7fcc701e9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 -- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc700b1c40 msgr2=0x7fcc701e9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc700b1c40 0x7fcc701e9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.149+0000 7fcc74e00640 1 -- 172.21.15.129:0/1663459929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc700ce170 con 0x7fcc70074800 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc74e00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400d740 tx=0x7fcc6400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64017070 con 0x7fcc70074800 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcc64009d20 con 0x7fcc70074800 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc700ce450 con 0x7fcc70074800 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc640106c0 con 0x7fcc70074800 2024-09-17T04:07:34.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.150+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc700cea10 con 0x7fcc70074800 2024-09-17T04:07:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.151+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc700b4110 con 0x7fcc70074800 2024-09-17T04:07:34.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.153+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcc6401a070 con 0x7fcc70074800 2024-09-17T04:07:34.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.154+0000 7fcc6cc00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 0x7fcc5007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:34.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.154+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fcc6409ae10 con 0x7fcc70074800 2024-09-17T04:07:34.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.154+0000 7fcc6fe00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 0x7fcc5007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:34.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.155+0000 7fcc6fe00640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 0x7fcc5007b0a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcc701f35d0 tx=0x7fcc5c007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:34.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.158+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fcc6409e030 con 0x7fcc70074800 2024-09-17T04:07:34.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.378+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fcc700ae7d0 con 0x7fcc70074800 2024-09-17T04:07:34.950 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T04:07:34.951+0000 7f7525c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:07:34.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:34.999+0000 7fcc6cc00640 1 -- 172.21.15.129:0/1663459929 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v248) ==== 118+0+0 (secure 0 0 0) 0x7fcc64062bf0 con 0x7fcc70074800 2024-09-17T04:07:35.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 msgr2=0x7fcc5007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:35.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 0x7fcc5007b0a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fcc701f35d0 tx=0x7fcc5c007e90 comp rx=0 tx=0).stop 2024-09-17T04:07:35.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 msgr2=0x7fcc70157280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:35.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400d740 tx=0x7fcc6400dc00 comp rx=0 tx=0).stop 2024-09-17T04:07:35.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 shutdown_connections 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcc50078b90 0x7fcc5007b0a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcc700b3010 0x7fcc701e9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc700b1c40 0x7fcc701e9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 --2- 172.21.15.129:0/1663459929 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcc70074800 0x7fcc70157280 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 >> 172.21.15.129:0/1663459929 conn(0x7fcc700ccc40 msgr2=0x7fcc700cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 shutdown_connections 2024-09-17T04:07:35.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.002+0000 7fcc77800640 1 -- 172.21.15.129:0/1663459929 wait complete. 2024-09-17T04:07:35.027 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:07:35.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:07:35.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.252+0000 7f23e1400640 1 -- 172.21.15.129:0/1619844182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 msgr2=0x7f23dc0b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:35.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.252+0000 7f23e1400640 1 --2- 172.21.15.129:0/1619844182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc0b2050 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f23dc071e10 tx=0x7f23c402a5f0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 -- 172.21.15.129:0/1619844182 shutdown_connections 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 --2- 172.21.15.129:0/1619844182 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc0b2050 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 --2- 172.21.15.129:0/1619844182 >> v2:172.21.15.181:3300/0 conn(0x7f23dc074800 0x7f23dc0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 -- 172.21.15.129:0/1619844182 >> 172.21.15.129:0/1619844182 conn(0x7f23dc0ccc40 msgr2=0x7f23dc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 -- 172.21.15.129:0/1619844182 shutdown_connections 2024-09-17T04:07:35.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.253+0000 7f23e1400640 1 -- 172.21.15.129:0/1619844182 wait complete. 2024-09-17T04:07:35.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.255+0000 7f23e1400640 1 Processor -- start 2024-09-17T04:07:35.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.255+0000 7f23e1400640 1 -- start start 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23dc06eea0 0x7f23dc0c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23dc074800 0x7f23dc09b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0b4020 con 0x7f23dc0b1c40 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0b41a0 con 0x7f23dc06eea0 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.256+0000 7f23e1400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc0b4320 con 0x7f23dc074800 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46474/0 (socket says 172.21.15.129:46474) 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 -- 172.21.15.129:0/1624835073 learned_addr learned my addr 172.21.15.129:0/1624835073 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23da800640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23dc06eea0 0x7f23dc0c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23d3e00640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23dc074800 0x7f23dc09b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 -- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23dc074800 msgr2=0x7f23dc09b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23dc074800 0x7f23dc09b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 -- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23dc06eea0 msgr2=0x7f23dc0c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23dc06eea0 0x7f23dc0c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 -- 172.21.15.129:0/1624835073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc1f5f90 con 0x7f23dc0b1c40 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.257+0000 7f23db200640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f23d400c730 tx=0x7f23d400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.258+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4003b70 con 0x7f23dc0b1c40 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.258+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f23d4003d10 con 0x7f23dc0b1c40 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.258+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4005330 con 0x7f23dc0b1c40 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.258+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23dc1f6270 con 0x7f23dc0b1c40 2024-09-17T04:07:35.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.258+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23dc1f6830 con 0x7f23dc0b1c40 2024-09-17T04:07:35.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.259+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23dc06fb80 con 0x7f23dc0b1c40 2024-09-17T04:07:35.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.260+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f23d40054d0 con 0x7f23dc0b1c40 2024-09-17T04:07:35.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.261+0000 7f23d2a00640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 0x7f23b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:35.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.261+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f23d409a9b0 con 0x7f23dc0b1c40 2024-09-17T04:07:35.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.262+0000 7f23da800640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 0x7f23b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:35.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.262+0000 7f23da800640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 0x7f23b807b0a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009b30 tx=0x7f23cc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:35.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.266+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f23dc0cbaf0 con 0x7f23dc0b1c40 2024-09-17T04:07:35.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:35.487+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f23dc0c3a90 con 0x7f23dc0b1c40 2024-09-17T04:07:36.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.001+0000 7f23d2a00640 1 -- 172.21.15.129:0/1624835073 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v249) ==== 122+0+0 (secure 0 0 0) 0x7f23d4062790 con 0x7f23dc0b1c40 2024-09-17T04:07:36.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.003+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 msgr2=0x7f23b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.003+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 0x7f23b807b0a0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009b30 tx=0x7f23cc005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.003+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 msgr2=0x7f23dc15d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.003+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f23d400c730 tx=0x7f23d400cb30 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.003+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 shutdown_connections 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f23b8078b90 0x7f23b807b0a0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23dc0b1c40 0x7f23dc15d3d0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23dc074800 0x7f23dc09b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 --2- 172.21.15.129:0/1624835073 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23dc06eea0 0x7f23dc0c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 >> 172.21.15.129:0/1624835073 conn(0x7f23dc0ccc40 msgr2=0x7f23dc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 shutdown_connections 2024-09-17T04:07:36.003 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.004+0000 7f23e1400640 1 -- 172.21.15.129:0/1624835073 wait complete. 2024-09-17T04:07:36.030 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:07:36.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.251+0000 7fe459600640 1 -- 172.21.15.129:0/468295813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 msgr2=0x7fe4540ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.251+0000 7fe459600640 1 --2- 172.21.15.129:0/468295813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540ccc80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe43c005fd0 tx=0x7fe43c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 -- 172.21.15.129:0/468295813 shutdown_connections 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 --2- 172.21.15.129:0/468295813 >> v2:172.21.15.181:3300/0 conn(0x7fe4540cd1c0 0x7fe4541609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 --2- 172.21.15.129:0/468295813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540ccc80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 -- 172.21.15.129:0/468295813 >> 172.21.15.129:0/468295813 conn(0x7fe45406d410 msgr2=0x7fe4540b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:36.251 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 -- 172.21.15.129:0/468295813 shutdown_connections 2024-09-17T04:07:36.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.252+0000 7fe459600640 1 -- 172.21.15.129:0/468295813 wait complete. 2024-09-17T04:07:36.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.254+0000 7fe459600640 1 Processor -- start 2024-09-17T04:07:36.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.254+0000 7fe459600640 1 -- start start 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4540cd1c0 0x7fe45415ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe454160260 con 0x7fe4540705a0 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe4541600e0 con 0x7fe4540cd1c0 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.255+0000 7fe459600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe454160f00 con 0x7fe4540b4140 2024-09-17T04:07:36.255 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46488/0 (socket says 172.21.15.129:46488) 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 -- 172.21.15.129:0/4099212810 learned_addr learned my addr 172.21.15.129:0/4099212810 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe453400640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4540cd1c0 0x7fe45415ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe44be00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 -- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 msgr2=0x7fe4540c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 -- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4540cd1c0 msgr2=0x7fe45415ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4540cd1c0 0x7fe45415ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 -- 172.21.15.129:0/4099212810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4540cbf40 con 0x7fe4540705a0 2024-09-17T04:07:36.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.256+0000 7fe452a00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c3a0 tx=0x7fe44400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:36.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.257+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444017070 con 0x7fe4540705a0 2024-09-17T04:07:36.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.257+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe4440053c0 con 0x7fe4540705a0 2024-09-17T04:07:36.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.257+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4540cc220 con 0x7fe4540705a0 2024-09-17T04:07:36.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.257+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444003850 con 0x7fe4540705a0 2024-09-17T04:07:36.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.257+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe454074090 con 0x7fe4540705a0 2024-09-17T04:07:36.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.258+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4540ccc80 con 0x7fe4540705a0 2024-09-17T04:07:36.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.259+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe44401a070 con 0x7fe4540705a0 2024-09-17T04:07:36.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.260+0000 7fe44aa00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 0x7fe43007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.260+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fe44409a1f0 con 0x7fe4540705a0 2024-09-17T04:07:36.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.261+0000 7fe44be00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 0x7fe43007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.261+0000 7fe44be00640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 0x7fe43007b0a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe43c033000 tx=0x7fe43c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:36.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.265+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe4540cd5d0 con 0x7fe4540705a0 2024-09-17T04:07:36.479 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.480+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe45406a170 con 0x7fe4540705a0 2024-09-17T04:07:36.480 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.481+0000 7fe44aa00640 1 -- 172.21.15.129:0/4099212810 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+5246 (secure 0 0 0) 0x7fe444061ec0 con 0x7fe4540705a0 2024-09-17T04:07:36.480 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:36.480 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":249,"btime":"2024-09-17T04:07:35:999196+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/47559844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":47559844},{"type":"v1","addr":"172.21.15.181:6836","nonce":47559844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":11266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3086573184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3086573184},{"type":"v1","addr":"172.21.15.181:6837","nonce":3086573184}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":12240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080794990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080794990},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080794990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":243}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:35.488097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:36.480 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 249 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 msgr2=0x7fe43007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 0x7fe43007b0a0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fe43c033000 tx=0x7fe43c004730 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 msgr2=0x7fe4541f9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fe44400c3a0 tx=0x7fe44400c860 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 shutdown_connections 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe430078b90 0x7fe43007b0a0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4540cd1c0 0x7fe45415ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe4540b4140 0x7fe4540c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 --2- 172.21.15.129:0/4099212810 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe4540705a0 0x7fe4541f9ae0 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.483+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 >> 172.21.15.129:0/4099212810 conn(0x7fe45406d410 msgr2=0x7fe4540b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.484+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 shutdown_connections 2024-09-17T04:07:36.483 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.484+0000 7fe459600640 1 -- 172.21.15.129:0/4099212810 wait complete. 2024-09-17T04:07:36.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:07:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 -- 172.21.15.129:0/830938736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 msgr2=0x7f4f40072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 --2- 172.21.15.129:0/830938736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f40072f40 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f4f38005fd0 tx=0x7f4f3802aaa0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 -- 172.21.15.129:0/830938736 shutdown_connections 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 --2- 172.21.15.129:0/830938736 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f40072f40 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 --2- 172.21.15.129:0/830938736 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f401524c0 0x7f4f400cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 -- 172.21.15.129:0/830938736 >> 172.21.15.129:0/830938736 conn(0x7f4f4006cfc0 msgr2=0x7f4f4006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 -- 172.21.15.129:0/830938736 shutdown_connections 2024-09-17T04:07:36.728 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.728+0000 7f4f3f200640 1 -- 172.21.15.129:0/830938736 wait complete. 2024-09-17T04:07:36.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.730+0000 7f4f3f200640 1 Processor -- start 2024-09-17T04:07:36.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.730+0000 7f4f3f200640 1 -- start start 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f400cfb00 0x7f4f4014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f401524c0 0x7f4f40154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4f400d02c0 con 0x7f4f401528d0 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4f40073880 con 0x7f4f401524c0 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4f400d0440 con 0x7f4f400cfb00 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.732 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46502/0 (socket says 172.21.15.129:46502) 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.732+0000 7f4f3e800640 1 -- 172.21.15.129:0/1376271545 learned_addr learned my addr 172.21.15.129:0/1376271545 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3de00640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f400cfb00 0x7f4f4014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f35400640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f401524c0 0x7f4f40154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 -- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f400cfb00 msgr2=0x7f4f4014aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f400cfb00 0x7f4f4014aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 -- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f401524c0 msgr2=0x7f4f40154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f401524c0 0x7f4f40154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 -- 172.21.15.129:0/1376271545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f400b0130 con 0x7f4f401528d0 2024-09-17T04:07:36.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f3e800640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009d90 tx=0x7f4f38003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:36.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38044070 con 0x7f4f401528d0 2024-09-17T04:07:36.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.733+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4f3802e800 con 0x7f4f401528d0 2024-09-17T04:07:36.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.734+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f380044b0 con 0x7f4f401528d0 2024-09-17T04:07:36.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.734+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f400b0410 con 0x7f4f401528d0 2024-09-17T04:07:36.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.734+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f400b09d0 con 0x7f4f401528d0 2024-09-17T04:07:36.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.735+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f400d08f0 con 0x7f4f401528d0 2024-09-17T04:07:36.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.736+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4f38047070 con 0x7f4f401528d0 2024-09-17T04:07:36.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.737+0000 7f4f36a00640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 0x7f4f1c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:36.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.738+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f4f380bbb90 con 0x7f4f401528d0 2024-09-17T04:07:36.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.738+0000 7f4f3de00640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 0x7f4f1c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:36.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.739+0000 7f4f3de00640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 0x7f4f1c07b0a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000bb50 tx=0x7f4f3000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:36.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.742+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4f38032030 con 0x7f4f401528d0 2024-09-17T04:07:36.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.925+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f400afe80 con 0x7f4f401528d0 2024-09-17T04:07:36.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.926+0000 7f4f36a00640 1 -- 172.21.15.129:0/1376271545 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v260) ==== 74+0+16791 (secure 0 0 0) 0x7f4f38083a00 con 0x7f4f401528d0 2024-09-17T04:07:36.925 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:36.926 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":260,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:33.974629+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:07:30.379067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":39,"pool_name":"cephfs_data","create_time":"2024-09-17T04:07:31.426899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"260","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 msgr2=0x7f4f1c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 0x7f4f1c07b0a0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000bb50 tx=0x7f4f3000b2e0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 msgr2=0x7f4f400ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009d90 tx=0x7f4f38003f10 comp rx=0 tx=0).stop 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 shutdown_connections 2024-09-17T04:07:36.928 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4f1c078b90 0x7f4f1c07b0a0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4f401528d0 0x7f4f400ae920 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4f401524c0 0x7f4f40154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 --2- 172.21.15.129:0/1376271545 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4f400cfb00 0x7f4f4014aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 >> 172.21.15.129:0/1376271545 conn(0x7f4f4006cfc0 msgr2=0x7f4f400b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 shutdown_connections 2024-09-17T04:07:36.929 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:36.929+0000 7f4f3f200640 1 -- 172.21.15.129:0/1376271545 wait complete. 2024-09-17T04:07:36.950 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:07:36.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:07:37.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 -- 172.21.15.129:0/2539208889 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d80cf2b0 msgr2=0x7f10d80cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 --2- 172.21.15.129:0/2539208889 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d80cf2b0 0x7f10d80cf6c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009830 tx=0x7f10c002a480 comp rx=0 tx=0).stop 2024-09-17T04:07:37.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 -- 172.21.15.129:0/2539208889 shutdown_connections 2024-09-17T04:07:37.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 --2- 172.21.15.129:0/2539208889 >> v2:172.21.15.181:3300/0 conn(0x7f10d80ccc80 0x7f10d80cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 --2- 172.21.15.129:0/2539208889 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d80cf2b0 0x7f10d80cf6c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 -- 172.21.15.129:0/2539208889 >> 172.21.15.129:0/2539208889 conn(0x7f10d806d410 msgr2=0x7f10d80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:37.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 -- 172.21.15.129:0/2539208889 shutdown_connections 2024-09-17T04:07:37.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.173+0000 7f10de000640 1 -- 172.21.15.129:0/2539208889 wait complete. 2024-09-17T04:07:37.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.175+0000 7f10de000640 1 Processor -- start 2024-09-17T04:07:37.175 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.175+0000 7f10de000640 1 -- start start 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8099570 0x7f10d81f5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10d80ccc80 0x7f10d81f9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10d8160ba0 con 0x7f10d80cf2b0 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10d8160a20 con 0x7f10d80ccc80 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10de000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10d8160d20 con 0x7f10d8099570 2024-09-17T04:07:37.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46506/0 (socket says 172.21.15.129:46506) 2024-09-17T04:07:37.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 -- 172.21.15.129:0/2918983323 learned_addr learned my addr 172.21.15.129:0/2918983323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:37.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10cea00640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10d80ccc80 0x7f10d81f9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7400640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8099570 0x7f10d81f5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 -- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8099570 msgr2=0x7f10d81f5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8099570 0x7f10d81f5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 -- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10d80ccc80 msgr2=0x7f10d81f9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10d80ccc80 0x7f10d81f9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.177+0000 7f10d7e00640 1 -- 172.21.15.129:0/2918983323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d80b21f0 con 0x7f10d80cf2b0 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10d7e00640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f10d000ebf0 tx=0x7f10d000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0010630 con 0x7f10d80cf2b0 2024-09-17T04:07:37.179 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10d00107d0 con 0x7f10d80cf2b0 2024-09-17T04:07:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d0010c20 con 0x7f10d80cf2b0 2024-09-17T04:07:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d80b1f50 con 0x7f10d80cf2b0 2024-09-17T04:07:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.178+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d815e790 con 0x7f10d80cf2b0 2024-09-17T04:07:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.179+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d8161240 con 0x7f10d80cf2b0 2024-09-17T04:07:37.180 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.180+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10d0021430 con 0x7f10d80cf2b0 2024-09-17T04:07:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.182+0000 7f10cfe00640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 0x7f10b407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.182+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f10d009bbc0 con 0x7f10d80cf2b0 2024-09-17T04:07:37.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.182+0000 7f10d7400640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 0x7f10b407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.183 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.183+0000 7f10d7400640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 0x7f10b407b0a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f60 tx=0x7f10c8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:37.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.187+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f10d00639a0 con 0x7f10d80cf2b0 2024-09-17T04:07:37.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.366+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f10d815ea30 con 0x7f10b4078b90 2024-09-17T04:07:37.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.372+0000 7f10cfe00640 1 -- 172.21.15.129:0/2918983323 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7f10d815ea30 con 0x7f10b4078b90 2024-09-17T04:07:37.375 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:37.376 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:07:37.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 msgr2=0x7f10b407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 0x7f10b407b0a0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f10c8000f60 tx=0x7f10c8009040 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 msgr2=0x7f10d81f9570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f10d000ebf0 tx=0x7f10d000c6a0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 shutdown_connections 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10b4078b90 0x7f10b407b0a0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10d80cf2b0 0x7f10d81f9570 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10d80ccc80 0x7f10d81f9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 --2- 172.21.15.129:0/2918983323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10d8099570 0x7f10d81f5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 >> 172.21.15.129:0/2918983323 conn(0x7f10d806d410 msgr2=0x7f10d80b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 shutdown_connections 2024-09-17T04:07:37.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.376+0000 7f10de000640 1 -- 172.21.15.129:0/2918983323 wait complete. 2024-09-17T04:07:37.398 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-17T04:07:36.605047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419384,"kb_used_data":26876,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754555336,"statfs":{"total":773094113280,"available":772664664064,"internally_reserved":0,"allocated":27521024,"data_stored":12838503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.652617"},"pg_stats":[{"pgid":"39.3f","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983934+0000","last_change":"2024-09-17T04:07:32.991162+0000","last_active":"2024-09-17T04:07:33.983934+0000","last_peered":"2024-09-17T04:07:33.983934+0000","last_clean":"2024-09-17T04:07:33.983934+0000","last_became_active":"2024-09-17T04:07:32.990481+0000","last_became_peered":"2024-09-17T04:07:32.990481+0000","last_unstale":"2024-09-17T04:07:33.983934+0000","last_undegraded":"2024-09-17T04:07:33.983934+0000","last_fullsized":"2024-09-17T04:07:33.983934+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:23.387174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.3e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111200+0000","last_change":"2024-09-17T04:07:32.991171+0000","last_active":"2024-09-17T04:07:34.111200+0000","last_peered":"2024-09-17T04:07:34.111200+0000","last_clean":"2024-09-17T04:07:34.111200+0000","last_became_active":"2024-09-17T04:07:32.990871+0000","last_became_peered":"2024-09-17T04:07:32.990871+0000","last_unstale":"2024-09-17T04:07:34.111200+0000","last_undegraded":"2024-09-17T04:07:34.111200+0000","last_fullsized":"2024-09-17T04:07:34.111200+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:31:16.769964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3d","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985041+0000","last_change":"2024-09-17T04:07:32.983882+0000","last_active":"2024-09-17T04:07:32.985041+0000","last_peered":"2024-09-17T04:07:32.985041+0000","last_clean":"2024-09-17T04:07:32.985041+0000","last_became_active":"2024-09-17T04:07:32.983532+0000","last_became_peered":"2024-09-17T04:07:32.983532+0000","last_unstale":"2024-09-17T04:07:32.985041+0000","last_undegraded":"2024-09-17T04:07:32.985041+0000","last_fullsized":"2024-09-17T04:07:32.985041+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:47.742271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.3c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980570+0000","last_change":"2024-09-17T04:07:33.002669+0000","last_active":"2024-09-17T04:07:33.980570+0000","last_peered":"2024-09-17T04:07:33.980570+0000","last_clean":"2024-09-17T04:07:33.980570+0000","last_became_active":"2024-09-17T04:07:33.002373+0000","last_became_peered":"2024-09-17T04:07:33.002373+0000","last_unstale":"2024-09-17T04:07:33.980570+0000","last_undegraded":"2024-09-17T04:07:33.980570+0000","last_fullsized":"2024-09-17T04:07:33.980570+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:18.877378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111322+0000","last_change":"2024-09-17T04:07:32.987433+0000","last_active":"2024-09-17T04:07:34.111322+0000","last_peered":"2024-09-17T04:07:34.111322+0000","last_clean":"2024-09-17T04:07:34.111322+0000","last_became_active":"2024-09-17T04:07:32.986603+0000","last_became_peered":"2024-09-17T04:07:32.986603+0000","last_unstale":"2024-09-17T04:07:34.111322+0000","last_undegraded":"2024-09-17T04:07:34.111322+0000","last_fullsized":"2024-09-17T04:07:34.111322+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:06.190084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.3a","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994096+0000","last_change":"2024-09-17T04:07:32.989540+0000","last_active":"2024-09-17T04:07:32.994096+0000","last_peered":"2024-09-17T04:07:32.994096+0000","last_clean":"2024-09-17T04:07:32.994096+0000","last_became_active":"2024-09-17T04:07:32.988131+0000","last_became_peered":"2024-09-17T04:07:32.988131+0000","last_unstale":"2024-09-17T04:07:32.994096+0000","last_undegraded":"2024-09-17T04:07:32.994096+0000","last_fullsized":"2024-09-17T04:07:32.994096+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:00:52.948869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.39","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.982817+0000","last_change":"2024-09-17T04:07:32.991265+0000","last_active":"2024-09-17T04:07:33.982817+0000","last_peered":"2024-09-17T04:07:33.982817+0000","last_clean":"2024-09-17T04:07:33.982817+0000","last_became_active":"2024-09-17T04:07:32.990868+0000","last_became_peered":"2024-09-17T04:07:32.990868+0000","last_unstale":"2024-09-17T04:07:33.982817+0000","last_undegraded":"2024-09-17T04:07:33.982817+0000","last_fullsized":"2024-09-17T04:07:33.982817+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:14.933876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.38","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110458+0000","last_change":"2024-09-17T04:07:32.992331+0000","last_active":"2024-09-17T04:07:34.110458+0000","last_peered":"2024-09-17T04:07:34.110458+0000","last_clean":"2024-09-17T04:07:34.110458+0000","last_became_active":"2024-09-17T04:07:32.992024+0000","last_became_peered":"2024-09-17T04:07:32.992024+0000","last_unstale":"2024-09-17T04:07:34.110458+0000","last_undegraded":"2024-09-17T04:07:34.110458+0000","last_fullsized":"2024-09-17T04:07:34.110458+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:57:28.356336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.37","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.984739+0000","last_change":"2024-09-17T04:07:32.981270+0000","last_active":"2024-09-17T04:07:32.984739+0000","last_peered":"2024-09-17T04:07:32.984739+0000","last_clean":"2024-09-17T04:07:32.984739+0000","last_became_active":"2024-09-17T04:07:32.980962+0000","last_became_peered":"2024-09-17T04:07:32.980962+0000","last_unstale":"2024-09-17T04:07:32.984739+0000","last_undegraded":"2024-09-17T04:07:32.984739+0000","last_fullsized":"2024-09-17T04:07:32.984739+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.647336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.36","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111422+0000","last_change":"2024-09-17T04:07:32.997479+0000","last_active":"2024-09-17T04:07:34.111422+0000","last_peered":"2024-09-17T04:07:34.111422+0000","last_clean":"2024-09-17T04:07:34.111422+0000","last_became_active":"2024-09-17T04:07:32.996838+0000","last_became_peered":"2024-09-17T04:07:32.996838+0000","last_unstale":"2024-09-17T04:07:34.111422+0000","last_undegraded":"2024-09-17T04:07:34.111422+0000","last_fullsized":"2024-09-17T04:07:34.111422+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:43.253770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.35","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110952+0000","last_change":"2024-09-17T04:07:32.989664+0000","last_active":"2024-09-17T04:07:34.110952+0000","last_peered":"2024-09-17T04:07:34.110952+0000","last_clean":"2024-09-17T04:07:34.110952+0000","last_became_active":"2024-09-17T04:07:32.985156+0000","last_became_peered":"2024-09-17T04:07:32.985156+0000","last_unstale":"2024-09-17T04:07:34.110952+0000","last_undegraded":"2024-09-17T04:07:34.110952+0000","last_fullsized":"2024-09-17T04:07:34.110952+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:52.996004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.34","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988647+0000","last_change":"2024-09-17T04:07:32.982894+0000","last_active":"2024-09-17T04:07:32.988647+0000","last_peered":"2024-09-17T04:07:32.988647+0000","last_clean":"2024-09-17T04:07:32.988647+0000","last_became_active":"2024-09-17T04:07:32.982513+0000","last_became_peered":"2024-09-17T04:07:32.982513+0000","last_unstale":"2024-09-17T04:07:32.988647+0000","last_undegraded":"2024-09-17T04:07:32.988647+0000","last_fullsized":"2024-09-17T04:07:32.988647+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:29.952609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.33","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979817+0000","last_change":"2024-09-17T04:07:32.987659+0000","last_active":"2024-09-17T04:07:33.979817+0000","last_peered":"2024-09-17T04:07:33.979817+0000","last_clean":"2024-09-17T04:07:33.979817+0000","last_became_active":"2024-09-17T04:07:32.986938+0000","last_became_peered":"2024-09-17T04:07:32.986938+0000","last_unstale":"2024-09-17T04:07:33.979817+0000","last_undegraded":"2024-09-17T04:07:33.979817+0000","last_fullsized":"2024-09-17T04:07:33.979817+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:44.920022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.32","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110476+0000","last_change":"2024-09-17T04:07:32.989873+0000","last_active":"2024-09-17T04:07:34.110476+0000","last_peered":"2024-09-17T04:07:34.110476+0000","last_clean":"2024-09-17T04:07:34.110476+0000","last_became_active":"2024-09-17T04:07:32.989571+0000","last_became_peered":"2024-09-17T04:07:32.989571+0000","last_unstale":"2024-09-17T04:07:34.110476+0000","last_undegraded":"2024-09-17T04:07:34.110476+0000","last_fullsized":"2024-09-17T04:07:34.110476+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:02.414449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.31","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979347+0000","last_change":"2024-09-17T04:07:32.980283+0000","last_active":"2024-09-17T04:07:33.979347+0000","last_peered":"2024-09-17T04:07:33.979347+0000","last_clean":"2024-09-17T04:07:33.979347+0000","last_became_active":"2024-09-17T04:07:32.979939+0000","last_became_peered":"2024-09-17T04:07:32.979939+0000","last_unstale":"2024-09-17T04:07:33.979347+0000","last_undegraded":"2024-09-17T04:07:33.979347+0000","last_fullsized":"2024-09-17T04:07:33.979347+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:45.456758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.30","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979373+0000","last_change":"2024-09-17T04:07:32.987137+0000","last_active":"2024-09-17T04:07:33.979373+0000","last_peered":"2024-09-17T04:07:33.979373+0000","last_clean":"2024-09-17T04:07:33.979373+0000","last_became_active":"2024-09-17T04:07:32.986818+0000","last_became_peered":"2024-09-17T04:07:32.986818+0000","last_unstale":"2024-09-17T04:07:33.979373+0000","last_undegraded":"2024-09-17T04:07:33.979373+0000","last_fullsized":"2024-09-17T04:07:33.979373+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:06.268979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2f","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994414+0000","last_change":"2024-09-17T04:07:32.992290+0000","last_active":"2024-09-17T04:07:32.994414+0000","last_peered":"2024-09-17T04:07:32.994414+0000","last_clean":"2024-09-17T04:07:32.994414+0000","last_became_active":"2024-09-17T04:07:32.989935+0000","last_became_peered":"2024-09-17T04:07:32.989935+0000","last_unstale":"2024-09-17T04:07:32.994414+0000","last_undegraded":"2024-09-17T04:07:32.994414+0000","last_fullsized":"2024-09-17T04:07:32.994414+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:38.105434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.2e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980641+0000","last_change":"2024-09-17T04:07:32.991729+0000","last_active":"2024-09-17T04:07:33.980641+0000","last_peered":"2024-09-17T04:07:33.980641+0000","last_clean":"2024-09-17T04:07:33.980641+0000","last_became_active":"2024-09-17T04:07:32.991078+0000","last_became_peered":"2024-09-17T04:07:32.991078+0000","last_unstale":"2024-09-17T04:07:33.980641+0000","last_undegraded":"2024-09-17T04:07:33.980641+0000","last_fullsized":"2024-09-17T04:07:33.980641+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:51.054363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2d","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983501+0000","last_change":"2024-09-17T04:07:32.987005+0000","last_active":"2024-09-17T04:07:33.983501+0000","last_peered":"2024-09-17T04:07:33.983501+0000","last_clean":"2024-09-17T04:07:33.983501+0000","last_became_active":"2024-09-17T04:07:32.984695+0000","last_became_peered":"2024-09-17T04:07:32.984695+0000","last_unstale":"2024-09-17T04:07:33.983501+0000","last_undegraded":"2024-09-17T04:07:33.983501+0000","last_fullsized":"2024-09-17T04:07:33.983501+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:24.832080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.2c","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.977719+0000","last_change":"2024-09-17T04:07:32.972383+0000","last_active":"2024-09-17T04:07:32.977719+0000","last_peered":"2024-09-17T04:07:32.977719+0000","last_clean":"2024-09-17T04:07:32.977719+0000","last_became_active":"2024-09-17T04:07:32.972059+0000","last_became_peered":"2024-09-17T04:07:32.972059+0000","last_unstale":"2024-09-17T04:07:32.977719+0000","last_undegraded":"2024-09-17T04:07:32.977719+0000","last_fullsized":"2024-09-17T04:07:32.977719+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:28.694221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.2b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979909+0000","last_change":"2024-09-17T04:07:32.987557+0000","last_active":"2024-09-17T04:07:33.979909+0000","last_peered":"2024-09-17T04:07:33.979909+0000","last_clean":"2024-09-17T04:07:33.979909+0000","last_became_active":"2024-09-17T04:07:32.986587+0000","last_became_peered":"2024-09-17T04:07:32.986587+0000","last_unstale":"2024-09-17T04:07:33.979909+0000","last_undegraded":"2024-09-17T04:07:33.979909+0000","last_fullsized":"2024-09-17T04:07:33.979909+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:36.457152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110986+0000","last_change":"2024-09-17T04:07:32.988018+0000","last_active":"2024-09-17T04:07:34.110986+0000","last_peered":"2024-09-17T04:07:34.110986+0000","last_clean":"2024-09-17T04:07:34.110986+0000","last_became_active":"2024-09-17T04:07:32.987681+0000","last_became_peered":"2024-09-17T04:07:32.987681+0000","last_unstale":"2024-09-17T04:07:34.110986+0000","last_undegraded":"2024-09-17T04:07:34.110986+0000","last_fullsized":"2024-09-17T04:07:34.110986+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:42.214498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.29","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110390+0000","last_change":"2024-09-17T04:07:32.986292+0000","last_active":"2024-09-17T04:07:34.110390+0000","last_peered":"2024-09-17T04:07:34.110390+0000","last_clean":"2024-09-17T04:07:34.110390+0000","last_became_active":"2024-09-17T04:07:32.985949+0000","last_became_peered":"2024-09-17T04:07:32.985949+0000","last_unstale":"2024-09-17T04:07:34.110390+0000","last_undegraded":"2024-09-17T04:07:34.110390+0000","last_fullsized":"2024-09-17T04:07:34.110390+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:49.964802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.28","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985564+0000","last_change":"2024-09-17T04:07:32.980502+0000","last_active":"2024-09-17T04:07:32.985564+0000","last_peered":"2024-09-17T04:07:32.985564+0000","last_clean":"2024-09-17T04:07:32.985564+0000","last_became_active":"2024-09-17T04:07:32.979926+0000","last_became_peered":"2024-09-17T04:07:32.979926+0000","last_unstale":"2024-09-17T04:07:32.985564+0000","last_undegraded":"2024-09-17T04:07:32.985564+0000","last_fullsized":"2024-09-17T04:07:32.985564+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:12.071375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.27","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.984029+0000","last_change":"2024-09-17T04:07:32.990672+0000","last_active":"2024-09-17T04:07:33.984029+0000","last_peered":"2024-09-17T04:07:33.984029+0000","last_clean":"2024-09-17T04:07:33.984029+0000","last_became_active":"2024-09-17T04:07:32.990310+0000","last_became_peered":"2024-09-17T04:07:32.990310+0000","last_unstale":"2024-09-17T04:07:33.984029+0000","last_undegraded":"2024-09-17T04:07:33.984029+0000","last_fullsized":"2024-09-17T04:07:33.984029+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:24.878874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.25","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111127+0000","last_change":"2024-09-17T04:07:32.988805+0000","last_active":"2024-09-17T04:07:34.111127+0000","last_peered":"2024-09-17T04:07:34.111127+0000","last_clean":"2024-09-17T04:07:34.111127+0000","last_became_active":"2024-09-17T04:07:32.988406+0000","last_became_peered":"2024-09-17T04:07:32.988406+0000","last_unstale":"2024-09-17T04:07:34.111127+0000","last_undegraded":"2024-09-17T04:07:34.111127+0000","last_fullsized":"2024-09-17T04:07:34.111127+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:22.271931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.24","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.983324+0000","last_change":"2024-09-17T04:07:32.977408+0000","last_active":"2024-09-17T04:07:32.983324+0000","last_peered":"2024-09-17T04:07:32.983324+0000","last_clean":"2024-09-17T04:07:32.983324+0000","last_became_active":"2024-09-17T04:07:32.976300+0000","last_became_peered":"2024-09-17T04:07:32.976300+0000","last_unstale":"2024-09-17T04:07:32.983324+0000","last_undegraded":"2024-09-17T04:07:32.983324+0000","last_fullsized":"2024-09-17T04:07:32.983324+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:54.091202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.23","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980062+0000","last_change":"2024-09-17T04:07:32.987221+0000","last_active":"2024-09-17T04:07:33.980062+0000","last_peered":"2024-09-17T04:07:33.980062+0000","last_clean":"2024-09-17T04:07:33.980062+0000","last_became_active":"2024-09-17T04:07:32.986114+0000","last_became_peered":"2024-09-17T04:07:32.986114+0000","last_unstale":"2024-09-17T04:07:33.980062+0000","last_undegraded":"2024-09-17T04:07:33.980062+0000","last_fullsized":"2024-09-17T04:07:33.980062+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:01.589375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.22","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110702+0000","last_change":"2024-09-17T04:07:32.989767+0000","last_active":"2024-09-17T04:07:34.110702+0000","last_peered":"2024-09-17T04:07:34.110702+0000","last_clean":"2024-09-17T04:07:34.110702+0000","last_became_active":"2024-09-17T04:07:32.983347+0000","last_became_peered":"2024-09-17T04:07:32.983347+0000","last_unstale":"2024-09-17T04:07:34.110702+0000","last_undegraded":"2024-09-17T04:07:34.110702+0000","last_fullsized":"2024-09-17T04:07:34.110702+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:45.565246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.21","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985556+0000","last_change":"2024-09-17T04:07:32.980260+0000","last_active":"2024-09-17T04:07:32.985556+0000","last_peered":"2024-09-17T04:07:32.985556+0000","last_clean":"2024-09-17T04:07:32.985556+0000","last_became_active":"2024-09-17T04:07:32.979470+0000","last_became_peered":"2024-09-17T04:07:32.979470+0000","last_unstale":"2024-09-17T04:07:32.985556+0000","last_undegraded":"2024-09-17T04:07:32.985556+0000","last_fullsized":"2024-09-17T04:07:32.985556+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:17.649683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.20","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994443+0000","last_change":"2024-09-17T04:07:32.992336+0000","last_active":"2024-09-17T04:07:32.994443+0000","last_peered":"2024-09-17T04:07:32.994443+0000","last_clean":"2024-09-17T04:07:32.994443+0000","last_became_active":"2024-09-17T04:07:32.991569+0000","last_became_peered":"2024-09-17T04:07:32.991569+0000","last_unstale":"2024-09-17T04:07:32.994443+0000","last_undegraded":"2024-09-17T04:07:32.994443+0000","last_fullsized":"2024-09-17T04:07:32.994443+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:40.739134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994751+0000","last_change":"2024-09-17T04:07:32.993699+0000","last_active":"2024-09-17T04:07:32.994751+0000","last_peered":"2024-09-17T04:07:32.994751+0000","last_clean":"2024-09-17T04:07:32.994751+0000","last_became_active":"2024-09-17T04:07:32.992809+0000","last_became_peered":"2024-09-17T04:07:32.992809+0000","last_unstale":"2024-09-17T04:07:32.994751+0000","last_undegraded":"2024-09-17T04:07:32.994751+0000","last_fullsized":"2024-09-17T04:07:32.994751+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:05.113146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1f","version":"259'4","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.009624+0000","last_change":"2024-09-17T04:07:31.976140+0000","last_active":"2024-09-17T04:07:33.009624+0000","last_peered":"2024-09-17T04:07:33.009624+0000","last_clean":"2024-09-17T04:07:33.009624+0000","last_became_active":"2024-09-17T04:07:31.975802+0000","last_became_peered":"2024-09-17T04:07:31.975802+0000","last_unstale":"2024-09-17T04:07:33.009624+0000","last_undegraded":"2024-09-17T04:07:33.009624+0000","last_fullsized":"2024-09-17T04:07:33.009624+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:52.981919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111706+0000","last_change":"2024-09-17T04:07:32.999345+0000","last_active":"2024-09-17T04:07:34.111706+0000","last_peered":"2024-09-17T04:07:34.111706+0000","last_clean":"2024-09-17T04:07:34.111706+0000","last_became_active":"2024-09-17T04:07:32.998832+0000","last_became_peered":"2024-09-17T04:07:32.998832+0000","last_unstale":"2024-09-17T04:07:34.111706+0000","last_undegraded":"2024-09-17T04:07:34.111706+0000","last_fullsized":"2024-09-17T04:07:34.111706+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:28.683397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1e","version":"259'1","reported_seq":14,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.003593+0000","last_change":"2024-09-17T04:07:31.984726+0000","last_active":"2024-09-17T04:07:33.003593+0000","last_peered":"2024-09-17T04:07:33.003593+0000","last_clean":"2024-09-17T04:07:33.003593+0000","last_became_active":"2024-09-17T04:07:31.983991+0000","last_became_peered":"2024-09-17T04:07:31.983991+0000","last_unstale":"2024-09-17T04:07:33.003593+0000","last_undegraded":"2024-09-17T04:07:33.003593+0000","last_fullsized":"2024-09-17T04:07:33.003593+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:30.578052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110586+0000","last_change":"2024-09-17T04:07:32.984378+0000","last_active":"2024-09-17T04:07:34.110586+0000","last_peered":"2024-09-17T04:07:34.110586+0000","last_clean":"2024-09-17T04:07:34.110586+0000","last_became_active":"2024-09-17T04:07:32.983985+0000","last_became_peered":"2024-09-17T04:07:32.983985+0000","last_unstale":"2024-09-17T04:07:34.110586+0000","last_undegraded":"2024-09-17T04:07:34.110586+0000","last_fullsized":"2024-09-17T04:07:34.110586+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:32.923280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1d","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980263+0000","last_change":"2024-09-17T04:07:31.988427+0000","last_active":"2024-09-17T04:07:33.980263+0000","last_peered":"2024-09-17T04:07:33.980263+0000","last_clean":"2024-09-17T04:07:33.980263+0000","last_became_active":"2024-09-17T04:07:31.988117+0000","last_became_peered":"2024-09-17T04:07:31.988117+0000","last_unstale":"2024-09-17T04:07:33.980263+0000","last_undegraded":"2024-09-17T04:07:33.980263+0000","last_fullsized":"2024-09-17T04:07:33.980263+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:37.266822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994752+0000","last_change":"2024-09-17T04:07:32.993726+0000","last_active":"2024-09-17T04:07:32.994752+0000","last_peered":"2024-09-17T04:07:32.994752+0000","last_clean":"2024-09-17T04:07:32.994752+0000","last_became_active":"2024-09-17T04:07:32.986796+0000","last_became_peered":"2024-09-17T04:07:32.986796+0000","last_unstale":"2024-09-17T04:07:32.994752+0000","last_undegraded":"2024-09-17T04:07:32.994752+0000","last_fullsized":"2024-09-17T04:07:32.994752+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:53.989343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1c","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112092+0000","last_change":"2024-09-17T04:07:31.979483+0000","last_active":"2024-09-17T04:07:34.112092+0000","last_peered":"2024-09-17T04:07:34.112092+0000","last_clean":"2024-09-17T04:07:34.112092+0000","last_became_active":"2024-09-17T04:07:31.979127+0000","last_became_peered":"2024-09-17T04:07:31.979127+0000","last_unstale":"2024-09-17T04:07:34.112092+0000","last_undegraded":"2024-09-17T04:07:34.112092+0000","last_fullsized":"2024-09-17T04:07:34.112092+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:16.212604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983251+0000","last_change":"2024-09-17T04:07:32.991997+0000","last_active":"2024-09-17T04:07:33.983251+0000","last_peered":"2024-09-17T04:07:33.983251+0000","last_clean":"2024-09-17T04:07:33.983251+0000","last_became_active":"2024-09-17T04:07:32.991674+0000","last_became_peered":"2024-09-17T04:07:32.991674+0000","last_unstale":"2024-09-17T04:07:33.983251+0000","last_undegraded":"2024-09-17T04:07:33.983251+0000","last_fullsized":"2024-09-17T04:07:33.983251+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:24.474559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1b","version":"259'1","reported_seq":14,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.005989+0000","last_change":"2024-09-17T04:07:31.987028+0000","last_active":"2024-09-17T04:07:33.005989+0000","last_peered":"2024-09-17T04:07:33.005989+0000","last_clean":"2024-09-17T04:07:33.005989+0000","last_became_active":"2024-09-17T04:07:31.986655+0000","last_became_peered":"2024-09-17T04:07:31.986655+0000","last_unstale":"2024-09-17T04:07:33.005989+0000","last_undegraded":"2024-09-17T04:07:33.005989+0000","last_fullsized":"2024-09-17T04:07:33.005989+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:39.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980138+0000","last_change":"2024-09-17T04:07:32.993335+0000","last_active":"2024-09-17T04:07:33.980138+0000","last_peered":"2024-09-17T04:07:33.980138+0000","last_clean":"2024-09-17T04:07:33.980138+0000","last_became_active":"2024-09-17T04:07:32.991828+0000","last_became_peered":"2024-09-17T04:07:32.991828+0000","last_unstale":"2024-09-17T04:07:33.980138+0000","last_undegraded":"2024-09-17T04:07:33.980138+0000","last_fullsized":"2024-09-17T04:07:33.980138+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:39.254334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.1a","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111810+0000","last_change":"2024-09-17T04:07:31.987108+0000","last_active":"2024-09-17T04:07:34.111810+0000","last_peered":"2024-09-17T04:07:34.111810+0000","last_clean":"2024-09-17T04:07:34.111810+0000","last_became_active":"2024-09-17T04:07:31.986781+0000","last_became_peered":"2024-09-17T04:07:31.986781+0000","last_unstale":"2024-09-17T04:07:34.111810+0000","last_undegraded":"2024-09-17T04:07:34.111810+0000","last_fullsized":"2024-09-17T04:07:34.111810+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:48.962568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979745+0000","last_change":"2024-09-17T04:07:32.981633+0000","last_active":"2024-09-17T04:07:33.979745+0000","last_peered":"2024-09-17T04:07:33.979745+0000","last_clean":"2024-09-17T04:07:33.979745+0000","last_became_active":"2024-09-17T04:07:32.981010+0000","last_became_peered":"2024-09-17T04:07:32.981010+0000","last_unstale":"2024-09-17T04:07:33.979745+0000","last_undegraded":"2024-09-17T04:07:33.979745+0000","last_fullsized":"2024-09-17T04:07:33.979745+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:29.263811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979482+0000","last_change":"2024-09-17T04:07:31.981421+0000","last_active":"2024-09-17T04:07:33.979482+0000","last_peered":"2024-09-17T04:07:33.979482+0000","last_clean":"2024-09-17T04:07:33.979482+0000","last_became_active":"2024-09-17T04:07:31.980772+0000","last_became_peered":"2024-09-17T04:07:31.980772+0000","last_unstale":"2024-09-17T04:07:33.979482+0000","last_undegraded":"2024-09-17T04:07:33.979482+0000","last_fullsized":"2024-09-17T04:07:33.979482+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:47.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985591+0000","last_change":"2024-09-17T04:07:32.980630+0000","last_active":"2024-09-17T04:07:32.985591+0000","last_peered":"2024-09-17T04:07:32.985591+0000","last_clean":"2024-09-17T04:07:32.985591+0000","last_became_active":"2024-09-17T04:07:32.979930+0000","last_became_peered":"2024-09-17T04:07:32.979930+0000","last_unstale":"2024-09-17T04:07:32.985591+0000","last_undegraded":"2024-09-17T04:07:32.985591+0000","last_fullsized":"2024-09-17T04:07:32.985591+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:22:53.365746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111196+0000","last_change":"2024-09-17T04:07:31.972349+0000","last_active":"2024-09-17T04:07:34.111196+0000","last_peered":"2024-09-17T04:07:34.111196+0000","last_clean":"2024-09-17T04:07:34.111196+0000","last_became_active":"2024-09-17T04:07:31.971975+0000","last_became_peered":"2024-09-17T04:07:31.971975+0000","last_unstale":"2024-09-17T04:07:34.111196+0000","last_undegraded":"2024-09-17T04:07:34.111196+0000","last_fullsized":"2024-09-17T04:07:34.111196+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:38.854526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111498+0000","last_change":"2024-09-17T04:07:33.005913+0000","last_active":"2024-09-17T04:07:34.111498+0000","last_peered":"2024-09-17T04:07:34.111498+0000","last_clean":"2024-09-17T04:07:34.111498+0000","last_became_active":"2024-09-17T04:07:33.005325+0000","last_became_peered":"2024-09-17T04:07:33.005325+0000","last_unstale":"2024-09-17T04:07:34.111498+0000","last_undegraded":"2024-09-17T04:07:34.111498+0000","last_fullsized":"2024-09-17T04:07:34.111498+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:38.602696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112074+0000","last_change":"2024-09-17T04:07:31.983291+0000","last_active":"2024-09-17T04:07:34.112074+0000","last_peered":"2024-09-17T04:07:34.112074+0000","last_clean":"2024-09-17T04:07:34.112074+0000","last_became_active":"2024-09-17T04:07:31.981953+0000","last_became_peered":"2024-09-17T04:07:31.981953+0000","last_unstale":"2024-09-17T04:07:34.112074+0000","last_undegraded":"2024-09-17T04:07:34.112074+0000","last_fullsized":"2024-09-17T04:07:34.112074+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:35.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980565+0000","last_change":"2024-09-17T04:07:32.991229+0000","last_active":"2024-09-17T04:07:33.980565+0000","last_peered":"2024-09-17T04:07:33.980565+0000","last_clean":"2024-09-17T04:07:33.980565+0000","last_became_active":"2024-09-17T04:07:32.990912+0000","last_became_peered":"2024-09-17T04:07:32.990912+0000","last_unstale":"2024-09-17T04:07:33.980565+0000","last_undegraded":"2024-09-17T04:07:33.980565+0000","last_fullsized":"2024-09-17T04:07:33.980565+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:09.129611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.966812+0000","last_change":"2024-09-17T04:07:31.984731+0000","last_active":"2024-09-17T04:07:32.966812+0000","last_peered":"2024-09-17T04:07:32.966812+0000","last_clean":"2024-09-17T04:07:32.966812+0000","last_became_active":"2024-09-17T04:07:31.983321+0000","last_became_peered":"2024-09-17T04:07:31.983321+0000","last_unstale":"2024-09-17T04:07:32.966812+0000","last_undegraded":"2024-09-17T04:07:32.966812+0000","last_fullsized":"2024-09-17T04:07:32.966812+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:33.408604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.991653+0000","last_change":"2024-09-17T04:07:32.988780+0000","last_active":"2024-09-17T04:07:32.991653+0000","last_peered":"2024-09-17T04:07:32.991653+0000","last_clean":"2024-09-17T04:07:32.991653+0000","last_became_active":"2024-09-17T04:07:32.988370+0000","last_became_peered":"2024-09-17T04:07:32.988370+0000","last_unstale":"2024-09-17T04:07:32.991653+0000","last_undegraded":"2024-09-17T04:07:32.991653+0000","last_fullsized":"2024-09-17T04:07:32.991653+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:03.786922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.15","version":"259'3","reported_seq":18,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983647+0000","last_change":"2024-09-17T04:07:31.981777+0000","last_active":"2024-09-17T04:07:33.983647+0000","last_peered":"2024-09-17T04:07:33.983647+0000","last_clean":"2024-09-17T04:07:33.983647+0000","last_became_active":"2024-09-17T04:07:31.981328+0000","last_became_peered":"2024-09-17T04:07:31.981328+0000","last_unstale":"2024-09-17T04:07:33.983647+0000","last_undegraded":"2024-09-17T04:07:33.983647+0000","last_fullsized":"2024-09-17T04:07:33.983647+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:53.606214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111348+0000","last_change":"2024-09-17T04:07:32.986053+0000","last_active":"2024-09-17T04:07:34.111348+0000","last_peered":"2024-09-17T04:07:34.111348+0000","last_clean":"2024-09-17T04:07:34.111348+0000","last_became_active":"2024-09-17T04:07:32.984983+0000","last_became_peered":"2024-09-17T04:07:32.984983+0000","last_unstale":"2024-09-17T04:07:34.111348+0000","last_undegraded":"2024-09-17T04:07:34.111348+0000","last_fullsized":"2024-09-17T04:07:34.111348+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:06.226511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.14","version":"259'6","reported_seq":21,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112016+0000","last_change":"2024-09-17T04:07:31.985031+0000","last_active":"2024-09-17T04:07:34.112016+0000","last_peered":"2024-09-17T04:07:34.112016+0000","last_clean":"2024-09-17T04:07:34.112016+0000","last_became_active":"2024-09-17T04:07:31.984713+0000","last_became_peered":"2024-09-17T04:07:31.984713+0000","last_unstale":"2024-09-17T04:07:34.112016+0000","last_undegraded":"2024-09-17T04:07:34.112016+0000","last_fullsized":"2024-09-17T04:07:34.112016+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:33.048145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988952+0000","last_change":"2024-09-17T04:07:32.984187+0000","last_active":"2024-09-17T04:07:32.988952+0000","last_peered":"2024-09-17T04:07:32.988952+0000","last_clean":"2024-09-17T04:07:32.988952+0000","last_became_active":"2024-09-17T04:07:32.983900+0000","last_became_peered":"2024-09-17T04:07:32.983900+0000","last_unstale":"2024-09-17T04:07:32.988952+0000","last_undegraded":"2024-09-17T04:07:32.988952+0000","last_fullsized":"2024-09-17T04:07:32.988952+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:47.142591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.13","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983160+0000","last_change":"2024-09-17T04:07:31.984769+0000","last_active":"2024-09-17T04:07:33.983160+0000","last_peered":"2024-09-17T04:07:33.983160+0000","last_clean":"2024-09-17T04:07:33.983160+0000","last_became_active":"2024-09-17T04:07:31.984213+0000","last_became_peered":"2024-09-17T04:07:31.984213+0000","last_unstale":"2024-09-17T04:07:33.983160+0000","last_undegraded":"2024-09-17T04:07:33.983160+0000","last_fullsized":"2024-09-17T04:07:33.983160+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:15.155284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111739+0000","last_change":"2024-09-17T04:07:32.992365+0000","last_active":"2024-09-17T04:07:34.111739+0000","last_peered":"2024-09-17T04:07:34.111739+0000","last_clean":"2024-09-17T04:07:34.111739+0000","last_became_active":"2024-09-17T04:07:32.990073+0000","last_became_peered":"2024-09-17T04:07:32.990073+0000","last_unstale":"2024-09-17T04:07:34.111739+0000","last_undegraded":"2024-09-17T04:07:34.111739+0000","last_fullsized":"2024-09-17T04:07:34.111739+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:36.646925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111813+0000","last_change":"2024-09-17T04:07:31.979947+0000","last_active":"2024-09-17T04:07:34.111813+0000","last_peered":"2024-09-17T04:07:34.111813+0000","last_clean":"2024-09-17T04:07:34.111813+0000","last_became_active":"2024-09-17T04:07:31.979589+0000","last_became_peered":"2024-09-17T04:07:31.979589+0000","last_unstale":"2024-09-17T04:07:34.111813+0000","last_undegraded":"2024-09-17T04:07:34.111813+0000","last_fullsized":"2024-09-17T04:07:34.111813+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:36.555914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110669+0000","last_change":"2024-09-17T04:07:32.987461+0000","last_active":"2024-09-17T04:07:34.110669+0000","last_peered":"2024-09-17T04:07:34.110669+0000","last_clean":"2024-09-17T04:07:34.110669+0000","last_became_active":"2024-09-17T04:07:32.987103+0000","last_became_peered":"2024-09-17T04:07:32.987103+0000","last_unstale":"2024-09-17T04:07:34.110669+0000","last_undegraded":"2024-09-17T04:07:34.110669+0000","last_fullsized":"2024-09-17T04:07:34.110669+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:30.576687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110849+0000","last_change":"2024-09-17T04:07:31.975990+0000","last_active":"2024-09-17T04:07:34.110849+0000","last_peered":"2024-09-17T04:07:34.110849+0000","last_clean":"2024-09-17T04:07:34.110849+0000","last_became_active":"2024-09-17T04:07:31.975663+0000","last_became_peered":"2024-09-17T04:07:31.975663+0000","last_unstale":"2024-09-17T04:07:34.110849+0000","last_undegraded":"2024-09-17T04:07:34.110849+0000","last_fullsized":"2024-09-17T04:07:34.110849+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.396672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979402+0000","last_change":"2024-09-17T04:07:32.982154+0000","last_active":"2024-09-17T04:07:33.979402+0000","last_peered":"2024-09-17T04:07:33.979402+0000","last_clean":"2024-09-17T04:07:33.979402+0000","last_became_active":"2024-09-17T04:07:32.981844+0000","last_became_peered":"2024-09-17T04:07:32.981844+0000","last_unstale":"2024-09-17T04:07:33.979402+0000","last_undegraded":"2024-09-17T04:07:33.979402+0000","last_fullsized":"2024-09-17T04:07:33.979402+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:28.720465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.10","version":"259'8","reported_seq":23,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979519+0000","last_change":"2024-09-17T04:07:31.979708+0000","last_active":"2024-09-17T04:07:33.979519+0000","last_peered":"2024-09-17T04:07:33.979519+0000","last_clean":"2024-09-17T04:07:33.979519+0000","last_became_active":"2024-09-17T04:07:31.979404+0000","last_became_peered":"2024-09-17T04:07:31.979404+0000","last_unstale":"2024-09-17T04:07:33.979519+0000","last_undegraded":"2024-09-17T04:07:33.979519+0000","last_fullsized":"2024-09-17T04:07:33.979519+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:31:15.762540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980423+0000","last_change":"2024-09-17T04:07:32.991396+0000","last_active":"2024-09-17T04:07:33.980423+0000","last_peered":"2024-09-17T04:07:33.980423+0000","last_clean":"2024-09-17T04:07:33.980423+0000","last_became_active":"2024-09-17T04:07:32.990662+0000","last_became_peered":"2024-09-17T04:07:32.990662+0000","last_unstale":"2024-09-17T04:07:33.980423+0000","last_undegraded":"2024-09-17T04:07:33.980423+0000","last_fullsized":"2024-09-17T04:07:33.980423+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:36.490543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.f","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110830+0000","last_change":"2024-09-17T04:07:31.977122+0000","last_active":"2024-09-17T04:07:34.110830+0000","last_peered":"2024-09-17T04:07:34.110830+0000","last_clean":"2024-09-17T04:07:34.110830+0000","last_became_active":"2024-09-17T04:07:31.976665+0000","last_became_peered":"2024-09-17T04:07:31.976665+0000","last_unstale":"2024-09-17T04:07:34.110830+0000","last_undegraded":"2024-09-17T04:07:34.110830+0000","last_fullsized":"2024-09-17T04:07:34.110830+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:28.158912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988324+0000","last_change":"2024-09-17T04:07:32.984836+0000","last_active":"2024-09-17T04:07:32.988324+0000","last_peered":"2024-09-17T04:07:32.988324+0000","last_clean":"2024-09-17T04:07:32.988324+0000","last_became_active":"2024-09-17T04:07:32.984508+0000","last_became_peered":"2024-09-17T04:07:32.984508+0000","last_unstale":"2024-09-17T04:07:32.988324+0000","last_undegraded":"2024-09-17T04:07:32.988324+0000","last_fullsized":"2024-09-17T04:07:32.988324+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:01.631445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.e","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983500+0000","last_change":"2024-09-17T04:07:31.981677+0000","last_active":"2024-09-17T04:07:33.983500+0000","last_peered":"2024-09-17T04:07:33.983500+0000","last_clean":"2024-09-17T04:07:33.983500+0000","last_became_active":"2024-09-17T04:07:31.980310+0000","last_became_peered":"2024-09-17T04:07:31.980310+0000","last_unstale":"2024-09-17T04:07:33.983500+0000","last_undegraded":"2024-09-17T04:07:33.983500+0000","last_fullsized":"2024-09-17T04:07:33.983500+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:13.573652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110813+0000","last_change":"2024-09-17T04:07:32.989179+0000","last_active":"2024-09-17T04:07:34.110813+0000","last_peered":"2024-09-17T04:07:34.110813+0000","last_clean":"2024-09-17T04:07:34.110813+0000","last_became_active":"2024-09-17T04:07:32.988848+0000","last_became_peered":"2024-09-17T04:07:32.988848+0000","last_unstale":"2024-09-17T04:07:34.110813+0000","last_undegraded":"2024-09-17T04:07:34.110813+0000","last_fullsized":"2024-09-17T04:07:34.110813+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:19.727704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.d","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111312+0000","last_change":"2024-09-17T04:07:31.980890+0000","last_active":"2024-09-17T04:07:34.111312+0000","last_peered":"2024-09-17T04:07:34.111312+0000","last_clean":"2024-09-17T04:07:34.111312+0000","last_became_active":"2024-09-17T04:07:31.980550+0000","last_became_peered":"2024-09-17T04:07:31.980550+0000","last_unstale":"2024-09-17T04:07:34.111312+0000","last_undegraded":"2024-09-17T04:07:34.111312+0000","last_fullsized":"2024-09-17T04:07:34.111312+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:05.765884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980127+0000","last_change":"2024-09-17T04:07:32.991987+0000","last_active":"2024-09-17T04:07:33.980127+0000","last_peered":"2024-09-17T04:07:33.980127+0000","last_clean":"2024-09-17T04:07:33.980127+0000","last_became_active":"2024-09-17T04:07:32.991654+0000","last_became_peered":"2024-09-17T04:07:32.991654+0000","last_unstale":"2024-09-17T04:07:33.980127+0000","last_undegraded":"2024-09-17T04:07:33.980127+0000","last_fullsized":"2024-09-17T04:07:33.980127+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:22.939020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111268+0000","last_change":"2024-09-17T04:07:31.976593+0000","last_active":"2024-09-17T04:07:34.111268+0000","last_peered":"2024-09-17T04:07:34.111268+0000","last_clean":"2024-09-17T04:07:34.111268+0000","last_became_active":"2024-09-17T04:07:31.976252+0000","last_became_peered":"2024-09-17T04:07:31.976252+0000","last_unstale":"2024-09-17T04:07:34.111268+0000","last_undegraded":"2024-09-17T04:07:34.111268+0000","last_fullsized":"2024-09-17T04:07:34.111268+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:27.697245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979465+0000","last_change":"2024-09-17T04:07:32.992356+0000","last_active":"2024-09-17T04:07:33.979465+0000","last_peered":"2024-09-17T04:07:33.979465+0000","last_clean":"2024-09-17T04:07:33.979465+0000","last_became_active":"2024-09-17T04:07:32.991602+0000","last_became_peered":"2024-09-17T04:07:32.991602+0000","last_unstale":"2024-09-17T04:07:33.979465+0000","last_undegraded":"2024-09-17T04:07:33.979465+0000","last_fullsized":"2024-09-17T04:07:33.979465+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:44.679911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.b","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983288+0000","last_change":"2024-09-17T04:07:31.981151+0000","last_active":"2024-09-17T04:07:33.983288+0000","last_peered":"2024-09-17T04:07:33.983288+0000","last_clean":"2024-09-17T04:07:33.983288+0000","last_became_active":"2024-09-17T04:07:31.980830+0000","last_became_peered":"2024-09-17T04:07:31.980830+0000","last_unstale":"2024-09-17T04:07:33.983288+0000","last_undegraded":"2024-09-17T04:07:33.983288+0000","last_fullsized":"2024-09-17T04:07:33.983288+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:42.710543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111433+0000","last_change":"2024-09-17T04:07:32.986662+0000","last_active":"2024-09-17T04:07:34.111433+0000","last_peered":"2024-09-17T04:07:34.111433+0000","last_clean":"2024-09-17T04:07:34.111433+0000","last_became_active":"2024-09-17T04:07:32.986330+0000","last_became_peered":"2024-09-17T04:07:32.986330+0000","last_unstale":"2024-09-17T04:07:34.111433+0000","last_undegraded":"2024-09-17T04:07:34.111433+0000","last_fullsized":"2024-09-17T04:07:34.111433+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:49.282083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110831+0000","last_change":"2024-09-17T04:07:31.979053+0000","last_active":"2024-09-17T04:07:34.110831+0000","last_peered":"2024-09-17T04:07:34.110831+0000","last_clean":"2024-09-17T04:07:34.110831+0000","last_became_active":"2024-09-17T04:07:31.978514+0000","last_became_peered":"2024-09-17T04:07:31.978514+0000","last_unstale":"2024-09-17T04:07:34.110831+0000","last_undegraded":"2024-09-17T04:07:34.110831+0000","last_fullsized":"2024-09-17T04:07:34.110831+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:01.119902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.977852+0000","last_change":"2024-09-17T04:07:32.970811+0000","last_active":"2024-09-17T04:07:32.977852+0000","last_peered":"2024-09-17T04:07:32.977852+0000","last_clean":"2024-09-17T04:07:32.977852+0000","last_became_active":"2024-09-17T04:07:32.970308+0000","last_became_peered":"2024-09-17T04:07:32.970308+0000","last_unstale":"2024-09-17T04:07:32.977852+0000","last_undegraded":"2024-09-17T04:07:32.977852+0000","last_fullsized":"2024-09-17T04:07:32.977852+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:50:56.496046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979538+0000","last_change":"2024-09-17T04:07:31.981085+0000","last_active":"2024-09-17T04:07:33.979538+0000","last_peered":"2024-09-17T04:07:33.979538+0000","last_clean":"2024-09-17T04:07:33.979538+0000","last_became_active":"2024-09-17T04:07:31.980370+0000","last_became_peered":"2024-09-17T04:07:31.980370+0000","last_unstale":"2024-09-17T04:07:33.979538+0000","last_undegraded":"2024-09-17T04:07:33.979538+0000","last_fullsized":"2024-09-17T04:07:33.979538+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:42.141454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111313+0000","last_change":"2024-09-17T04:07:32.994067+0000","last_active":"2024-09-17T04:07:34.111313+0000","last_peered":"2024-09-17T04:07:34.111313+0000","last_clean":"2024-09-17T04:07:34.111313+0000","last_became_active":"2024-09-17T04:07:32.990879+0000","last_became_peered":"2024-09-17T04:07:32.990879+0000","last_unstale":"2024-09-17T04:07:34.111313+0000","last_undegraded":"2024-09-17T04:07:34.111313+0000","last_fullsized":"2024-09-17T04:07:34.111313+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:01:44.964949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110830+0000","last_change":"2024-09-17T04:07:31.979633+0000","last_active":"2024-09-17T04:07:34.110830+0000","last_peered":"2024-09-17T04:07:34.110830+0000","last_clean":"2024-09-17T04:07:34.110830+0000","last_became_active":"2024-09-17T04:07:31.979312+0000","last_became_peered":"2024-09-17T04:07:31.979312+0000","last_unstale":"2024-09-17T04:07:34.110830+0000","last_undegraded":"2024-09-17T04:07:34.110830+0000","last_fullsized":"2024-09-17T04:07:34.110830+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:38.396795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111557+0000","last_change":"2024-09-17T04:07:32.988066+0000","last_active":"2024-09-17T04:07:34.111557+0000","last_peered":"2024-09-17T04:07:34.111557+0000","last_clean":"2024-09-17T04:07:34.111557+0000","last_became_active":"2024-09-17T04:07:32.986833+0000","last_became_peered":"2024-09-17T04:07:32.986833+0000","last_unstale":"2024-09-17T04:07:34.111557+0000","last_undegraded":"2024-09-17T04:07:34.111557+0000","last_fullsized":"2024-09-17T04:07:34.111557+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:46.680154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.7","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980548+0000","last_change":"2024-09-17T04:07:31.987739+0000","last_active":"2024-09-17T04:07:33.980548+0000","last_peered":"2024-09-17T04:07:33.980548+0000","last_clean":"2024-09-17T04:07:33.980548+0000","last_became_active":"2024-09-17T04:07:31.987347+0000","last_became_peered":"2024-09-17T04:07:31.987347+0000","last_unstale":"2024-09-17T04:07:33.980548+0000","last_undegraded":"2024-09-17T04:07:33.980548+0000","last_fullsized":"2024-09-17T04:07:33.980548+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:20.734459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111013+0000","last_change":"2024-09-17T04:07:32.989866+0000","last_active":"2024-09-17T04:07:34.111013+0000","last_peered":"2024-09-17T04:07:34.111013+0000","last_clean":"2024-09-17T04:07:34.111013+0000","last_became_active":"2024-09-17T04:07:32.989556+0000","last_became_peered":"2024-09-17T04:07:32.989556+0000","last_unstale":"2024-09-17T04:07:34.111013+0000","last_undegraded":"2024-09-17T04:07:34.111013+0000","last_fullsized":"2024-09-17T04:07:34.111013+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:48.555893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.967053+0000","last_change":"2024-09-17T04:07:31.987083+0000","last_active":"2024-09-17T04:07:32.967053+0000","last_peered":"2024-09-17T04:07:32.967053+0000","last_clean":"2024-09-17T04:07:32.967053+0000","last_became_active":"2024-09-17T04:07:31.986672+0000","last_became_peered":"2024-09-17T04:07:31.986672+0000","last_unstale":"2024-09-17T04:07:32.967053+0000","last_undegraded":"2024-09-17T04:07:32.967053+0000","last_fullsized":"2024-09-17T04:07:32.967053+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:14.294187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985728+0000","last_change":"2024-09-17T04:07:32.977861+0000","last_active":"2024-09-17T04:07:32.985728+0000","last_peered":"2024-09-17T04:07:32.985728+0000","last_clean":"2024-09-17T04:07:32.985728+0000","last_became_active":"2024-09-17T04:07:32.977518+0000","last_became_peered":"2024-09-17T04:07:32.977518+0000","last_unstale":"2024-09-17T04:07:32.985728+0000","last_undegraded":"2024-09-17T04:07:32.985728+0000","last_fullsized":"2024-09-17T04:07:32.985728+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:08.095384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.968235+0000","last_change":"2024-09-17T04:07:31.989402+0000","last_active":"2024-09-17T04:07:32.968235+0000","last_peered":"2024-09-17T04:07:32.968235+0000","last_clean":"2024-09-17T04:07:32.968235+0000","last_became_active":"2024-09-17T04:07:31.989048+0000","last_became_peered":"2024-09-17T04:07:31.989048+0000","last_unstale":"2024-09-17T04:07:32.968235+0000","last_undegraded":"2024-09-17T04:07:32.968235+0000","last_fullsized":"2024-09-17T04:07:32.968235+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:07.763138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112001+0000","last_change":"2024-09-17T04:07:32.985734+0000","last_active":"2024-09-17T04:07:34.112001+0000","last_peered":"2024-09-17T04:07:34.112001+0000","last_clean":"2024-09-17T04:07:34.112001+0000","last_became_active":"2024-09-17T04:07:32.985383+0000","last_became_peered":"2024-09-17T04:07:32.985383+0000","last_unstale":"2024-09-17T04:07:34.112001+0000","last_undegraded":"2024-09-17T04:07:34.112001+0000","last_fullsized":"2024-09-17T04:07:34.112001+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:38.204658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112102+0000","last_change":"2024-09-17T04:07:31.982794+0000","last_active":"2024-09-17T04:07:34.112102+0000","last_peered":"2024-09-17T04:07:34.112102+0000","last_clean":"2024-09-17T04:07:34.112102+0000","last_became_active":"2024-09-17T04:07:31.982418+0000","last_became_peered":"2024-09-17T04:07:31.982418+0000","last_unstale":"2024-09-17T04:07:34.112102+0000","last_undegraded":"2024-09-17T04:07:34.112102+0000","last_fullsized":"2024-09-17T04:07:34.112102+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:08.211420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979657+0000","last_change":"2024-09-17T04:07:32.992150+0000","last_active":"2024-09-17T04:07:33.979657+0000","last_peered":"2024-09-17T04:07:33.979657+0000","last_clean":"2024-09-17T04:07:33.979657+0000","last_became_active":"2024-09-17T04:07:32.988189+0000","last_became_peered":"2024-09-17T04:07:32.988189+0000","last_unstale":"2024-09-17T04:07:33.979657+0000","last_undegraded":"2024-09-17T04:07:33.979657+0000","last_fullsized":"2024-09-17T04:07:33.979657+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:33.257905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.967826+0000","last_change":"2024-09-17T04:07:31.985171+0000","last_active":"2024-09-17T04:07:32.967826+0000","last_peered":"2024-09-17T04:07:32.967826+0000","last_clean":"2024-09-17T04:07:32.967826+0000","last_became_active":"2024-09-17T04:07:31.982940+0000","last_became_peered":"2024-09-17T04:07:31.982940+0000","last_unstale":"2024-09-17T04:07:32.967826+0000","last_undegraded":"2024-09-17T04:07:32.967826+0000","last_fullsized":"2024-09-17T04:07:32.967826+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:18.174236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111597+0000","last_change":"2024-09-17T04:07:32.989740+0000","last_active":"2024-09-17T04:07:34.111597+0000","last_peered":"2024-09-17T04:07:34.111597+0000","last_clean":"2024-09-17T04:07:34.111597+0000","last_became_active":"2024-09-17T04:07:32.988512+0000","last_became_peered":"2024-09-17T04:07:32.988512+0000","last_unstale":"2024-09-17T04:07:34.111597+0000","last_undegraded":"2024-09-17T04:07:34.111597+0000","last_fullsized":"2024-09-17T04:07:34.111597+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:52.733478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979588+0000","last_change":"2024-09-17T04:07:31.981905+0000","last_active":"2024-09-17T04:07:33.979588+0000","last_peered":"2024-09-17T04:07:33.979588+0000","last_clean":"2024-09-17T04:07:33.979588+0000","last_became_active":"2024-09-17T04:07:31.981624+0000","last_became_peered":"2024-09-17T04:07:31.981624+0000","last_unstale":"2024-09-17T04:07:33.979588+0000","last_undegraded":"2024-09-17T04:07:33.979588+0000","last_fullsized":"2024-09-17T04:07:33.979588+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:42.246345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.982566+0000","last_change":"2024-09-17T04:07:32.996126+0000","last_active":"2024-09-17T04:07:33.982566+0000","last_peered":"2024-09-17T04:07:33.982566+0000","last_clean":"2024-09-17T04:07:33.982566+0000","last_became_active":"2024-09-17T04:07:32.995757+0000","last_became_peered":"2024-09-17T04:07:32.995757+0000","last_unstale":"2024-09-17T04:07:33.982566+0000","last_undegraded":"2024-09-17T04:07:33.982566+0000","last_fullsized":"2024-09-17T04:07:33.982566+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:58.594708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.966094+0000","last_change":"2024-09-17T04:07:31.970742+0000","last_active":"2024-09-17T04:07:32.966094+0000","last_peered":"2024-09-17T04:07:32.966094+0000","last_clean":"2024-09-17T04:07:32.966094+0000","last_became_active":"2024-09-17T04:07:31.970441+0000","last_became_peered":"2024-09-17T04:07:31.970441+0000","last_unstale":"2024-09-17T04:07:32.966094+0000","last_undegraded":"2024-09-17T04:07:32.966094+0000","last_fullsized":"2024-09-17T04:07:32.966094+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:37.098010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111582+0000","last_change":"2024-09-17T04:07:32.987616+0000","last_active":"2024-09-17T04:07:34.111582+0000","last_peered":"2024-09-17T04:07:34.111582+0000","last_clean":"2024-09-17T04:07:34.111582+0000","last_became_active":"2024-09-17T04:07:32.987287+0000","last_became_peered":"2024-09-17T04:07:32.987287+0000","last_unstale":"2024-09-17T04:07:34.111582+0000","last_undegraded":"2024-09-17T04:07:34.111582+0000","last_fullsized":"2024-09-17T04:07:34.111582+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:58.994691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110529+0000","last_change":"2024-09-17T04:07:31.976942+0000","last_active":"2024-09-17T04:07:34.110529+0000","last_peered":"2024-09-17T04:07:34.110529+0000","last_clean":"2024-09-17T04:07:34.110529+0000","last_became_active":"2024-09-17T04:07:31.975586+0000","last_became_peered":"2024-09-17T04:07:31.975586+0000","last_unstale":"2024-09-17T04:07:34.110529+0000","last_undegraded":"2024-09-17T04:07:34.110529+0000","last_fullsized":"2024-09-17T04:07:34.110529+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:23.507954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.26","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.983826+0000","last_change":"2024-09-17T04:07:32.975054+0000","last_active":"2024-09-17T04:07:32.983826+0000","last_peered":"2024-09-17T04:07:32.983826+0000","last_clean":"2024-09-17T04:07:32.983826+0000","last_became_active":"2024-09-17T04:07:32.974767+0000","last_became_peered":"2024-09-17T04:07:32.974767+0000","last_unstale":"2024-09-17T04:07:32.983826+0000","last_undegraded":"2024-09-17T04:07:32.983826+0000","last_fullsized":"2024-09-17T04:07:32.983826+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:35:50.471149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":552,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110738+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:07:34.110738+0000","last_peered":"2024-09-17T04:07:34.110738+0000","last_clean":"2024-09-17T04:07:34.110738+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:07:34.110738+0000","last_undegraded":"2024-09-17T04:07:34.110738+0000","last_fullsized":"2024-09-17T04:07:34.110738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":39,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":38,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509660,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53032,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318808,"statfs":{"total":96636764160,"available":96582459392,"internally_reserved":0,"allocated":3887104,"data_stored":2047087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509663,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34424,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337416,"statfs":{"total":96636764160,"available":96601513984,"internally_reserved":0,"allocated":3305472,"data_stored":1456865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56604,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315236,"statfs":{"total":96636764160,"available":96578801664,"internally_reserved":0,"allocated":3284992,"data_stored":1455177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3216,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3293184,"data_stored":1461407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56620,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315220,"statfs":{"total":96636764160,"available":96578785280,"internally_reserved":0,"allocated":3297280,"data_stored":1461989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509662,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52460,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319380,"statfs":{"total":96636764160,"available":96583045120,"internally_reserved":0,"allocated":3297280,"data_stored":1461429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56592,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315248,"statfs":{"total":96636764160,"available":96578813952,"internally_reserved":0,"allocated":3268608,"data_stored":1448332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542365,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":3887104,"data_stored":2046217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:07:37.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.658+0000 7fe6df200640 1 -- 172.21.15.129:0/926050957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806f2e0 msgr2=0x7fe6d80cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.658+0000 7fe6df200640 1 --2- 172.21.15.129:0/926050957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806f2e0 0x7fe6d80cd1c0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80d1370 tx=0x7fe6d002aa30 comp rx=0 tx=0).stop 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 -- 172.21.15.129:0/926050957 shutdown_connections 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 --2- 172.21.15.129:0/926050957 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806f2e0 0x7fe6d80cd1c0 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 --2- 172.21.15.129:0/926050957 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806eed0 0x7fe6d80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 --2- 172.21.15.129:0/926050957 >> v2:172.21.15.181:3300/0 conn(0x7fe6d80705a0 0x7fe6d80b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 -- 172.21.15.129:0/926050957 >> 172.21.15.129:0/926050957 conn(0x7fe6d806d410 msgr2=0x7fe6d80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:37.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 -- 172.21.15.129:0/926050957 shutdown_connections 2024-09-17T04:07:37.660 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.659+0000 7fe6df200640 1 -- 172.21.15.129:0/926050957 wait complete. 2024-09-17T04:07:37.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.661+0000 7fe6df200640 1 Processor -- start 2024-09-17T04:07:37.661 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.661+0000 7fe6df200640 1 -- start start 2024-09-17T04:07:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806f2e0 0x7fe6d814dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6d80705a0 0x7fe6d814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d8160b40 con 0x7fe6d806eed0 2024-09-17T04:07:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d81609c0 con 0x7fe6d80705a0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.662+0000 7fe6df200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d8160cc0 con 0x7fe6d806f2e0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46530/0 (socket says 172.21.15.129:46530) 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 -- 172.21.15.129:0/2575631615 learned_addr learned my addr 172.21.15.129:0/2575631615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7400640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806f2e0 0x7fe6d814dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6dd200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6d80705a0 0x7fe6d814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 -- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806f2e0 msgr2=0x7fe6d814dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806f2e0 0x7fe6d814dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 -- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6d80705a0 msgr2=0x7fe6d814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6d80705a0 0x7fe6d814e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 -- 172.21.15.129:0/2575631615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d80c7100 con 0x7fe6d806eed0 2024-09-17T04:07:37.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.663+0000 7fe6d7e00640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800d740 tx=0x7fe6c800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:37.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.664+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8017070 con 0x7fe6d806eed0 2024-09-17T04:07:37.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.664+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe6c8009d20 con 0x7fe6d806eed0 2024-09-17T04:07:37.665 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.664+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d81e62e0 con 0x7fe6d806eed0 2024-09-17T04:07:37.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.664+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80106c0 con 0x7fe6d806eed0 2024-09-17T04:07:37.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.664+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d80c9660 con 0x7fe6d806eed0 2024-09-17T04:07:37.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.665+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d8161220 con 0x7fe6d806eed0 2024-09-17T04:07:37.666 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.666+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe6c801a070 con 0x7fe6d806eed0 2024-09-17T04:07:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.667+0000 7fe6cfe00640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 0x7fe6b807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.667+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fe6c809ade0 con 0x7fe6d806eed0 2024-09-17T04:07:37.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.668+0000 7fe6d7400640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 0x7fe6b807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:37.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.668+0000 7fe6d7400640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 0x7fe6b807b0a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80af5c0 tx=0x7fe6c400b260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:37.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.672+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe6c8062c50 con 0x7fe6d806eed0 2024-09-17T04:07:37.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.853+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe6d815e7c0 con 0x7fe6b8078b90 2024-09-17T04:07:37.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.858+0000 7fe6cfe00640 1 -- 172.21.15.129:0/2575631615 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252473 (secure 0 0 0) 0x7fe6c4010ac0 con 0x7fe6b8078b90 2024-09-17T04:07:37.861 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:37.863 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:07:37.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 msgr2=0x7fe6b807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 0x7fe6b807b0a0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe6d80af5c0 tx=0x7fe6c400b260 comp rx=0 tx=0).stop 2024-09-17T04:07:37.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 msgr2=0x7fe6d80c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:37.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800d740 tx=0x7fe6c800dc00 comp rx=0 tx=0).stop 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 shutdown_connections 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe6b8078b90 0x7fe6b807b0a0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe6d80705a0 0x7fe6d814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe6d806f2e0 0x7fe6d814dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 --2- 172.21.15.129:0/2575631615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe6d806eed0 0x7fe6d80c6bc0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 >> 172.21.15.129:0/2575631615 conn(0x7fe6d806d410 msgr2=0x7fe6d80ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 shutdown_connections 2024-09-17T04:07:37.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:37.863+0000 7fe6df200640 1 -- 172.21.15.129:0/2575631615 wait complete. 2024-09-17T04:07:37.886 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-17T04:07:36.605047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":419384,"kb_used_data":26876,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754555336,"statfs":{"total":773094113280,"available":772664664064,"internally_reserved":0,"allocated":27521024,"data_stored":12838503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.652617"},"pg_stats":[{"pgid":"39.3f","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983934+0000","last_change":"2024-09-17T04:07:32.991162+0000","last_active":"2024-09-17T04:07:33.983934+0000","last_peered":"2024-09-17T04:07:33.983934+0000","last_clean":"2024-09-17T04:07:33.983934+0000","last_became_active":"2024-09-17T04:07:32.990481+0000","last_became_peered":"2024-09-17T04:07:32.990481+0000","last_unstale":"2024-09-17T04:07:33.983934+0000","last_undegraded":"2024-09-17T04:07:33.983934+0000","last_fullsized":"2024-09-17T04:07:33.983934+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:23.387174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.3e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111200+0000","last_change":"2024-09-17T04:07:32.991171+0000","last_active":"2024-09-17T04:07:34.111200+0000","last_peered":"2024-09-17T04:07:34.111200+0000","last_clean":"2024-09-17T04:07:34.111200+0000","last_became_active":"2024-09-17T04:07:32.990871+0000","last_became_peered":"2024-09-17T04:07:32.990871+0000","last_unstale":"2024-09-17T04:07:34.111200+0000","last_undegraded":"2024-09-17T04:07:34.111200+0000","last_fullsized":"2024-09-17T04:07:34.111200+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:31:16.769964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.3d","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985041+0000","last_change":"2024-09-17T04:07:32.983882+0000","last_active":"2024-09-17T04:07:32.985041+0000","last_peered":"2024-09-17T04:07:32.985041+0000","last_clean":"2024-09-17T04:07:32.985041+0000","last_became_active":"2024-09-17T04:07:32.983532+0000","last_became_peered":"2024-09-17T04:07:32.983532+0000","last_unstale":"2024-09-17T04:07:32.985041+0000","last_undegraded":"2024-09-17T04:07:32.985041+0000","last_fullsized":"2024-09-17T04:07:32.985041+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:47.742271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.3c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980570+0000","last_change":"2024-09-17T04:07:33.002669+0000","last_active":"2024-09-17T04:07:33.980570+0000","last_peered":"2024-09-17T04:07:33.980570+0000","last_clean":"2024-09-17T04:07:33.980570+0000","last_became_active":"2024-09-17T04:07:33.002373+0000","last_became_peered":"2024-09-17T04:07:33.002373+0000","last_unstale":"2024-09-17T04:07:33.980570+0000","last_undegraded":"2024-09-17T04:07:33.980570+0000","last_fullsized":"2024-09-17T04:07:33.980570+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:43:18.877378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.3b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111322+0000","last_change":"2024-09-17T04:07:32.987433+0000","last_active":"2024-09-17T04:07:34.111322+0000","last_peered":"2024-09-17T04:07:34.111322+0000","last_clean":"2024-09-17T04:07:34.111322+0000","last_became_active":"2024-09-17T04:07:32.986603+0000","last_became_peered":"2024-09-17T04:07:32.986603+0000","last_unstale":"2024-09-17T04:07:34.111322+0000","last_undegraded":"2024-09-17T04:07:34.111322+0000","last_fullsized":"2024-09-17T04:07:34.111322+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:06.190084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.3a","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994096+0000","last_change":"2024-09-17T04:07:32.989540+0000","last_active":"2024-09-17T04:07:32.994096+0000","last_peered":"2024-09-17T04:07:32.994096+0000","last_clean":"2024-09-17T04:07:32.994096+0000","last_became_active":"2024-09-17T04:07:32.988131+0000","last_became_peered":"2024-09-17T04:07:32.988131+0000","last_unstale":"2024-09-17T04:07:32.994096+0000","last_undegraded":"2024-09-17T04:07:32.994096+0000","last_fullsized":"2024-09-17T04:07:32.994096+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:00:52.948869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.39","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.982817+0000","last_change":"2024-09-17T04:07:32.991265+0000","last_active":"2024-09-17T04:07:33.982817+0000","last_peered":"2024-09-17T04:07:33.982817+0000","last_clean":"2024-09-17T04:07:33.982817+0000","last_became_active":"2024-09-17T04:07:32.990868+0000","last_became_peered":"2024-09-17T04:07:32.990868+0000","last_unstale":"2024-09-17T04:07:33.982817+0000","last_undegraded":"2024-09-17T04:07:33.982817+0000","last_fullsized":"2024-09-17T04:07:33.982817+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:14.933876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.38","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110458+0000","last_change":"2024-09-17T04:07:32.992331+0000","last_active":"2024-09-17T04:07:34.110458+0000","last_peered":"2024-09-17T04:07:34.110458+0000","last_clean":"2024-09-17T04:07:34.110458+0000","last_became_active":"2024-09-17T04:07:32.992024+0000","last_became_peered":"2024-09-17T04:07:32.992024+0000","last_unstale":"2024-09-17T04:07:34.110458+0000","last_undegraded":"2024-09-17T04:07:34.110458+0000","last_fullsized":"2024-09-17T04:07:34.110458+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:57:28.356336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.37","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.984739+0000","last_change":"2024-09-17T04:07:32.981270+0000","last_active":"2024-09-17T04:07:32.984739+0000","last_peered":"2024-09-17T04:07:32.984739+0000","last_clean":"2024-09-17T04:07:32.984739+0000","last_became_active":"2024-09-17T04:07:32.980962+0000","last_became_peered":"2024-09-17T04:07:32.980962+0000","last_unstale":"2024-09-17T04:07:32.984739+0000","last_undegraded":"2024-09-17T04:07:32.984739+0000","last_fullsized":"2024-09-17T04:07:32.984739+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:57:13.647336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.36","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111422+0000","last_change":"2024-09-17T04:07:32.997479+0000","last_active":"2024-09-17T04:07:34.111422+0000","last_peered":"2024-09-17T04:07:34.111422+0000","last_clean":"2024-09-17T04:07:34.111422+0000","last_became_active":"2024-09-17T04:07:32.996838+0000","last_became_peered":"2024-09-17T04:07:32.996838+0000","last_unstale":"2024-09-17T04:07:34.111422+0000","last_undegraded":"2024-09-17T04:07:34.111422+0000","last_fullsized":"2024-09-17T04:07:34.111422+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:43.253770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.35","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110952+0000","last_change":"2024-09-17T04:07:32.989664+0000","last_active":"2024-09-17T04:07:34.110952+0000","last_peered":"2024-09-17T04:07:34.110952+0000","last_clean":"2024-09-17T04:07:34.110952+0000","last_became_active":"2024-09-17T04:07:32.985156+0000","last_became_peered":"2024-09-17T04:07:32.985156+0000","last_unstale":"2024-09-17T04:07:34.110952+0000","last_undegraded":"2024-09-17T04:07:34.110952+0000","last_fullsized":"2024-09-17T04:07:34.110952+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:52.996004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.34","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988647+0000","last_change":"2024-09-17T04:07:32.982894+0000","last_active":"2024-09-17T04:07:32.988647+0000","last_peered":"2024-09-17T04:07:32.988647+0000","last_clean":"2024-09-17T04:07:32.988647+0000","last_became_active":"2024-09-17T04:07:32.982513+0000","last_became_peered":"2024-09-17T04:07:32.982513+0000","last_unstale":"2024-09-17T04:07:32.988647+0000","last_undegraded":"2024-09-17T04:07:32.988647+0000","last_fullsized":"2024-09-17T04:07:32.988647+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:38:29.952609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.33","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979817+0000","last_change":"2024-09-17T04:07:32.987659+0000","last_active":"2024-09-17T04:07:33.979817+0000","last_peered":"2024-09-17T04:07:33.979817+0000","last_clean":"2024-09-17T04:07:33.979817+0000","last_became_active":"2024-09-17T04:07:32.986938+0000","last_became_peered":"2024-09-17T04:07:32.986938+0000","last_unstale":"2024-09-17T04:07:33.979817+0000","last_undegraded":"2024-09-17T04:07:33.979817+0000","last_fullsized":"2024-09-17T04:07:33.979817+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:51:44.920022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.32","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110476+0000","last_change":"2024-09-17T04:07:32.989873+0000","last_active":"2024-09-17T04:07:34.110476+0000","last_peered":"2024-09-17T04:07:34.110476+0000","last_clean":"2024-09-17T04:07:34.110476+0000","last_became_active":"2024-09-17T04:07:32.989571+0000","last_became_peered":"2024-09-17T04:07:32.989571+0000","last_unstale":"2024-09-17T04:07:34.110476+0000","last_undegraded":"2024-09-17T04:07:34.110476+0000","last_fullsized":"2024-09-17T04:07:34.110476+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:02.414449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.31","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979347+0000","last_change":"2024-09-17T04:07:32.980283+0000","last_active":"2024-09-17T04:07:33.979347+0000","last_peered":"2024-09-17T04:07:33.979347+0000","last_clean":"2024-09-17T04:07:33.979347+0000","last_became_active":"2024-09-17T04:07:32.979939+0000","last_became_peered":"2024-09-17T04:07:32.979939+0000","last_unstale":"2024-09-17T04:07:33.979347+0000","last_undegraded":"2024-09-17T04:07:33.979347+0000","last_fullsized":"2024-09-17T04:07:33.979347+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:55:45.456758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.30","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979373+0000","last_change":"2024-09-17T04:07:32.987137+0000","last_active":"2024-09-17T04:07:33.979373+0000","last_peered":"2024-09-17T04:07:33.979373+0000","last_clean":"2024-09-17T04:07:33.979373+0000","last_became_active":"2024-09-17T04:07:32.986818+0000","last_became_peered":"2024-09-17T04:07:32.986818+0000","last_unstale":"2024-09-17T04:07:33.979373+0000","last_undegraded":"2024-09-17T04:07:33.979373+0000","last_fullsized":"2024-09-17T04:07:33.979373+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:06.268979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2f","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994414+0000","last_change":"2024-09-17T04:07:32.992290+0000","last_active":"2024-09-17T04:07:32.994414+0000","last_peered":"2024-09-17T04:07:32.994414+0000","last_clean":"2024-09-17T04:07:32.994414+0000","last_became_active":"2024-09-17T04:07:32.989935+0000","last_became_peered":"2024-09-17T04:07:32.989935+0000","last_unstale":"2024-09-17T04:07:32.994414+0000","last_undegraded":"2024-09-17T04:07:32.994414+0000","last_fullsized":"2024-09-17T04:07:32.994414+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:38.105434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.2e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980641+0000","last_change":"2024-09-17T04:07:32.991729+0000","last_active":"2024-09-17T04:07:33.980641+0000","last_peered":"2024-09-17T04:07:33.980641+0000","last_clean":"2024-09-17T04:07:33.980641+0000","last_became_active":"2024-09-17T04:07:32.991078+0000","last_became_peered":"2024-09-17T04:07:32.991078+0000","last_unstale":"2024-09-17T04:07:33.980641+0000","last_undegraded":"2024-09-17T04:07:33.980641+0000","last_fullsized":"2024-09-17T04:07:33.980641+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:59:51.054363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2d","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983501+0000","last_change":"2024-09-17T04:07:32.987005+0000","last_active":"2024-09-17T04:07:33.983501+0000","last_peered":"2024-09-17T04:07:33.983501+0000","last_clean":"2024-09-17T04:07:33.983501+0000","last_became_active":"2024-09-17T04:07:32.984695+0000","last_became_peered":"2024-09-17T04:07:32.984695+0000","last_unstale":"2024-09-17T04:07:33.983501+0000","last_undegraded":"2024-09-17T04:07:33.983501+0000","last_fullsized":"2024-09-17T04:07:33.983501+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:29:24.832080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.2c","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.977719+0000","last_change":"2024-09-17T04:07:32.972383+0000","last_active":"2024-09-17T04:07:32.977719+0000","last_peered":"2024-09-17T04:07:32.977719+0000","last_clean":"2024-09-17T04:07:32.977719+0000","last_became_active":"2024-09-17T04:07:32.972059+0000","last_became_peered":"2024-09-17T04:07:32.972059+0000","last_unstale":"2024-09-17T04:07:32.977719+0000","last_undegraded":"2024-09-17T04:07:32.977719+0000","last_fullsized":"2024-09-17T04:07:32.977719+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:28.694221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.2b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979909+0000","last_change":"2024-09-17T04:07:32.987557+0000","last_active":"2024-09-17T04:07:33.979909+0000","last_peered":"2024-09-17T04:07:33.979909+0000","last_clean":"2024-09-17T04:07:33.979909+0000","last_became_active":"2024-09-17T04:07:32.986587+0000","last_became_peered":"2024-09-17T04:07:32.986587+0000","last_unstale":"2024-09-17T04:07:33.979909+0000","last_undegraded":"2024-09-17T04:07:33.979909+0000","last_fullsized":"2024-09-17T04:07:33.979909+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:36.457152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.2a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110986+0000","last_change":"2024-09-17T04:07:32.988018+0000","last_active":"2024-09-17T04:07:34.110986+0000","last_peered":"2024-09-17T04:07:34.110986+0000","last_clean":"2024-09-17T04:07:34.110986+0000","last_became_active":"2024-09-17T04:07:32.987681+0000","last_became_peered":"2024-09-17T04:07:32.987681+0000","last_unstale":"2024-09-17T04:07:34.110986+0000","last_undegraded":"2024-09-17T04:07:34.110986+0000","last_fullsized":"2024-09-17T04:07:34.110986+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:09:42.214498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.29","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110390+0000","last_change":"2024-09-17T04:07:32.986292+0000","last_active":"2024-09-17T04:07:34.110390+0000","last_peered":"2024-09-17T04:07:34.110390+0000","last_clean":"2024-09-17T04:07:34.110390+0000","last_became_active":"2024-09-17T04:07:32.985949+0000","last_became_peered":"2024-09-17T04:07:32.985949+0000","last_unstale":"2024-09-17T04:07:34.110390+0000","last_undegraded":"2024-09-17T04:07:34.110390+0000","last_fullsized":"2024-09-17T04:07:34.110390+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:14:49.964802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.28","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985564+0000","last_change":"2024-09-17T04:07:32.980502+0000","last_active":"2024-09-17T04:07:32.985564+0000","last_peered":"2024-09-17T04:07:32.985564+0000","last_clean":"2024-09-17T04:07:32.985564+0000","last_became_active":"2024-09-17T04:07:32.979926+0000","last_became_peered":"2024-09-17T04:07:32.979926+0000","last_unstale":"2024-09-17T04:07:32.985564+0000","last_undegraded":"2024-09-17T04:07:32.985564+0000","last_fullsized":"2024-09-17T04:07:32.985564+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:51:12.071375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.27","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.984029+0000","last_change":"2024-09-17T04:07:32.990672+0000","last_active":"2024-09-17T04:07:33.984029+0000","last_peered":"2024-09-17T04:07:33.984029+0000","last_clean":"2024-09-17T04:07:33.984029+0000","last_became_active":"2024-09-17T04:07:32.990310+0000","last_became_peered":"2024-09-17T04:07:32.990310+0000","last_unstale":"2024-09-17T04:07:33.984029+0000","last_undegraded":"2024-09-17T04:07:33.984029+0000","last_fullsized":"2024-09-17T04:07:33.984029+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:48:24.878874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.25","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111127+0000","last_change":"2024-09-17T04:07:32.988805+0000","last_active":"2024-09-17T04:07:34.111127+0000","last_peered":"2024-09-17T04:07:34.111127+0000","last_clean":"2024-09-17T04:07:34.111127+0000","last_became_active":"2024-09-17T04:07:32.988406+0000","last_became_peered":"2024-09-17T04:07:32.988406+0000","last_unstale":"2024-09-17T04:07:34.111127+0000","last_undegraded":"2024-09-17T04:07:34.111127+0000","last_fullsized":"2024-09-17T04:07:34.111127+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:22.271931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.24","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.983324+0000","last_change":"2024-09-17T04:07:32.977408+0000","last_active":"2024-09-17T04:07:32.983324+0000","last_peered":"2024-09-17T04:07:32.983324+0000","last_clean":"2024-09-17T04:07:32.983324+0000","last_became_active":"2024-09-17T04:07:32.976300+0000","last_became_peered":"2024-09-17T04:07:32.976300+0000","last_unstale":"2024-09-17T04:07:32.983324+0000","last_undegraded":"2024-09-17T04:07:32.983324+0000","last_fullsized":"2024-09-17T04:07:32.983324+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:54.091202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.23","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980062+0000","last_change":"2024-09-17T04:07:32.987221+0000","last_active":"2024-09-17T04:07:33.980062+0000","last_peered":"2024-09-17T04:07:33.980062+0000","last_clean":"2024-09-17T04:07:33.980062+0000","last_became_active":"2024-09-17T04:07:32.986114+0000","last_became_peered":"2024-09-17T04:07:32.986114+0000","last_unstale":"2024-09-17T04:07:33.980062+0000","last_undegraded":"2024-09-17T04:07:33.980062+0000","last_fullsized":"2024-09-17T04:07:33.980062+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:01.589375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.22","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110702+0000","last_change":"2024-09-17T04:07:32.989767+0000","last_active":"2024-09-17T04:07:34.110702+0000","last_peered":"2024-09-17T04:07:34.110702+0000","last_clean":"2024-09-17T04:07:34.110702+0000","last_became_active":"2024-09-17T04:07:32.983347+0000","last_became_peered":"2024-09-17T04:07:32.983347+0000","last_unstale":"2024-09-17T04:07:34.110702+0000","last_undegraded":"2024-09-17T04:07:34.110702+0000","last_fullsized":"2024-09-17T04:07:34.110702+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:45.565246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.21","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985556+0000","last_change":"2024-09-17T04:07:32.980260+0000","last_active":"2024-09-17T04:07:32.985556+0000","last_peered":"2024-09-17T04:07:32.985556+0000","last_clean":"2024-09-17T04:07:32.985556+0000","last_became_active":"2024-09-17T04:07:32.979470+0000","last_became_peered":"2024-09-17T04:07:32.979470+0000","last_unstale":"2024-09-17T04:07:32.985556+0000","last_undegraded":"2024-09-17T04:07:32.985556+0000","last_fullsized":"2024-09-17T04:07:32.985556+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:08:17.649683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.20","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994443+0000","last_change":"2024-09-17T04:07:32.992336+0000","last_active":"2024-09-17T04:07:32.994443+0000","last_peered":"2024-09-17T04:07:32.994443+0000","last_clean":"2024-09-17T04:07:32.994443+0000","last_became_active":"2024-09-17T04:07:32.991569+0000","last_became_peered":"2024-09-17T04:07:32.991569+0000","last_unstale":"2024-09-17T04:07:32.994443+0000","last_undegraded":"2024-09-17T04:07:32.994443+0000","last_fullsized":"2024-09-17T04:07:32.994443+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:40.739134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.1e","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994751+0000","last_change":"2024-09-17T04:07:32.993699+0000","last_active":"2024-09-17T04:07:32.994751+0000","last_peered":"2024-09-17T04:07:32.994751+0000","last_clean":"2024-09-17T04:07:32.994751+0000","last_became_active":"2024-09-17T04:07:32.992809+0000","last_became_peered":"2024-09-17T04:07:32.992809+0000","last_unstale":"2024-09-17T04:07:32.994751+0000","last_undegraded":"2024-09-17T04:07:32.994751+0000","last_fullsized":"2024-09-17T04:07:32.994751+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:05.113146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1f","version":"259'4","reported_seq":17,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.009624+0000","last_change":"2024-09-17T04:07:31.976140+0000","last_active":"2024-09-17T04:07:33.009624+0000","last_peered":"2024-09-17T04:07:33.009624+0000","last_clean":"2024-09-17T04:07:33.009624+0000","last_became_active":"2024-09-17T04:07:31.975802+0000","last_became_peered":"2024-09-17T04:07:31.975802+0000","last_unstale":"2024-09-17T04:07:33.009624+0000","last_undegraded":"2024-09-17T04:07:33.009624+0000","last_fullsized":"2024-09-17T04:07:33.009624+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:52.981919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1f","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111706+0000","last_change":"2024-09-17T04:07:32.999345+0000","last_active":"2024-09-17T04:07:34.111706+0000","last_peered":"2024-09-17T04:07:34.111706+0000","last_clean":"2024-09-17T04:07:34.111706+0000","last_became_active":"2024-09-17T04:07:32.998832+0000","last_became_peered":"2024-09-17T04:07:32.998832+0000","last_unstale":"2024-09-17T04:07:34.111706+0000","last_undegraded":"2024-09-17T04:07:34.111706+0000","last_fullsized":"2024-09-17T04:07:34.111706+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:36:28.683397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.1e","version":"259'1","reported_seq":14,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.003593+0000","last_change":"2024-09-17T04:07:31.984726+0000","last_active":"2024-09-17T04:07:33.003593+0000","last_peered":"2024-09-17T04:07:33.003593+0000","last_clean":"2024-09-17T04:07:33.003593+0000","last_became_active":"2024-09-17T04:07:31.983991+0000","last_became_peered":"2024-09-17T04:07:31.983991+0000","last_unstale":"2024-09-17T04:07:33.003593+0000","last_undegraded":"2024-09-17T04:07:33.003593+0000","last_fullsized":"2024-09-17T04:07:33.003593+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:44:30.578052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110586+0000","last_change":"2024-09-17T04:07:32.984378+0000","last_active":"2024-09-17T04:07:34.110586+0000","last_peered":"2024-09-17T04:07:34.110586+0000","last_clean":"2024-09-17T04:07:34.110586+0000","last_became_active":"2024-09-17T04:07:32.983985+0000","last_became_peered":"2024-09-17T04:07:32.983985+0000","last_unstale":"2024-09-17T04:07:34.110586+0000","last_undegraded":"2024-09-17T04:07:34.110586+0000","last_fullsized":"2024-09-17T04:07:34.110586+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:39:32.923280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.1d","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980263+0000","last_change":"2024-09-17T04:07:31.988427+0000","last_active":"2024-09-17T04:07:33.980263+0000","last_peered":"2024-09-17T04:07:33.980263+0000","last_clean":"2024-09-17T04:07:33.980263+0000","last_became_active":"2024-09-17T04:07:31.988117+0000","last_became_peered":"2024-09-17T04:07:31.988117+0000","last_unstale":"2024-09-17T04:07:33.980263+0000","last_undegraded":"2024-09-17T04:07:33.980263+0000","last_fullsized":"2024-09-17T04:07:33.980263+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:54:37.266822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.1d","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.994752+0000","last_change":"2024-09-17T04:07:32.993726+0000","last_active":"2024-09-17T04:07:32.994752+0000","last_peered":"2024-09-17T04:07:32.994752+0000","last_clean":"2024-09-17T04:07:32.994752+0000","last_became_active":"2024-09-17T04:07:32.986796+0000","last_became_peered":"2024-09-17T04:07:32.986796+0000","last_unstale":"2024-09-17T04:07:32.994752+0000","last_undegraded":"2024-09-17T04:07:32.994752+0000","last_fullsized":"2024-09-17T04:07:32.994752+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:10:53.989343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.1c","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112092+0000","last_change":"2024-09-17T04:07:31.979483+0000","last_active":"2024-09-17T04:07:34.112092+0000","last_peered":"2024-09-17T04:07:34.112092+0000","last_clean":"2024-09-17T04:07:34.112092+0000","last_became_active":"2024-09-17T04:07:31.979127+0000","last_became_peered":"2024-09-17T04:07:31.979127+0000","last_unstale":"2024-09-17T04:07:34.112092+0000","last_undegraded":"2024-09-17T04:07:34.112092+0000","last_fullsized":"2024-09-17T04:07:34.112092+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:16.212604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.1a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983251+0000","last_change":"2024-09-17T04:07:32.991997+0000","last_active":"2024-09-17T04:07:33.983251+0000","last_peered":"2024-09-17T04:07:33.983251+0000","last_clean":"2024-09-17T04:07:33.983251+0000","last_became_active":"2024-09-17T04:07:32.991674+0000","last_became_peered":"2024-09-17T04:07:32.991674+0000","last_unstale":"2024-09-17T04:07:33.983251+0000","last_undegraded":"2024-09-17T04:07:33.983251+0000","last_fullsized":"2024-09-17T04:07:33.983251+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:24.474559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1b","version":"259'1","reported_seq":14,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.005989+0000","last_change":"2024-09-17T04:07:31.987028+0000","last_active":"2024-09-17T04:07:33.005989+0000","last_peered":"2024-09-17T04:07:33.005989+0000","last_clean":"2024-09-17T04:07:33.005989+0000","last_became_active":"2024-09-17T04:07:31.986655+0000","last_became_peered":"2024-09-17T04:07:31.986655+0000","last_unstale":"2024-09-17T04:07:33.005989+0000","last_undegraded":"2024-09-17T04:07:33.005989+0000","last_fullsized":"2024-09-17T04:07:33.005989+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:25:39.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980138+0000","last_change":"2024-09-17T04:07:32.993335+0000","last_active":"2024-09-17T04:07:33.980138+0000","last_peered":"2024-09-17T04:07:33.980138+0000","last_clean":"2024-09-17T04:07:33.980138+0000","last_became_active":"2024-09-17T04:07:32.991828+0000","last_became_peered":"2024-09-17T04:07:32.991828+0000","last_unstale":"2024-09-17T04:07:33.980138+0000","last_undegraded":"2024-09-17T04:07:33.980138+0000","last_fullsized":"2024-09-17T04:07:33.980138+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:39.254334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.1a","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111810+0000","last_change":"2024-09-17T04:07:31.987108+0000","last_active":"2024-09-17T04:07:34.111810+0000","last_peered":"2024-09-17T04:07:34.111810+0000","last_clean":"2024-09-17T04:07:34.111810+0000","last_became_active":"2024-09-17T04:07:31.986781+0000","last_became_peered":"2024-09-17T04:07:31.986781+0000","last_unstale":"2024-09-17T04:07:34.111810+0000","last_undegraded":"2024-09-17T04:07:34.111810+0000","last_fullsized":"2024-09-17T04:07:34.111810+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:44:48.962568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.18","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979745+0000","last_change":"2024-09-17T04:07:32.981633+0000","last_active":"2024-09-17T04:07:33.979745+0000","last_peered":"2024-09-17T04:07:33.979745+0000","last_clean":"2024-09-17T04:07:33.979745+0000","last_became_active":"2024-09-17T04:07:32.981010+0000","last_became_peered":"2024-09-17T04:07:32.981010+0000","last_unstale":"2024-09-17T04:07:33.979745+0000","last_undegraded":"2024-09-17T04:07:33.979745+0000","last_fullsized":"2024-09-17T04:07:33.979745+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:33:29.263811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.19","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979482+0000","last_change":"2024-09-17T04:07:31.981421+0000","last_active":"2024-09-17T04:07:33.979482+0000","last_peered":"2024-09-17T04:07:33.979482+0000","last_clean":"2024-09-17T04:07:33.979482+0000","last_became_active":"2024-09-17T04:07:31.980772+0000","last_became_peered":"2024-09-17T04:07:31.980772+0000","last_unstale":"2024-09-17T04:07:33.979482+0000","last_undegraded":"2024-09-17T04:07:33.979482+0000","last_fullsized":"2024-09-17T04:07:33.979482+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:47.548469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.19","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985591+0000","last_change":"2024-09-17T04:07:32.980630+0000","last_active":"2024-09-17T04:07:32.985591+0000","last_peered":"2024-09-17T04:07:32.985591+0000","last_clean":"2024-09-17T04:07:32.985591+0000","last_became_active":"2024-09-17T04:07:32.979930+0000","last_became_peered":"2024-09-17T04:07:32.979930+0000","last_unstale":"2024-09-17T04:07:32.985591+0000","last_undegraded":"2024-09-17T04:07:32.985591+0000","last_fullsized":"2024-09-17T04:07:32.985591+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:22:53.365746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.18","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111196+0000","last_change":"2024-09-17T04:07:31.972349+0000","last_active":"2024-09-17T04:07:34.111196+0000","last_peered":"2024-09-17T04:07:34.111196+0000","last_clean":"2024-09-17T04:07:34.111196+0000","last_became_active":"2024-09-17T04:07:31.971975+0000","last_became_peered":"2024-09-17T04:07:31.971975+0000","last_unstale":"2024-09-17T04:07:34.111196+0000","last_undegraded":"2024-09-17T04:07:34.111196+0000","last_fullsized":"2024-09-17T04:07:34.111196+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:09:38.854526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.16","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111498+0000","last_change":"2024-09-17T04:07:33.005913+0000","last_active":"2024-09-17T04:07:34.111498+0000","last_peered":"2024-09-17T04:07:34.111498+0000","last_clean":"2024-09-17T04:07:34.111498+0000","last_became_active":"2024-09-17T04:07:33.005325+0000","last_became_peered":"2024-09-17T04:07:33.005325+0000","last_unstale":"2024-09-17T04:07:34.111498+0000","last_undegraded":"2024-09-17T04:07:34.111498+0000","last_fullsized":"2024-09-17T04:07:34.111498+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:39:38.602696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.17","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112074+0000","last_change":"2024-09-17T04:07:31.983291+0000","last_active":"2024-09-17T04:07:34.112074+0000","last_peered":"2024-09-17T04:07:34.112074+0000","last_clean":"2024-09-17T04:07:34.112074+0000","last_became_active":"2024-09-17T04:07:31.981953+0000","last_became_peered":"2024-09-17T04:07:31.981953+0000","last_unstale":"2024-09-17T04:07:34.112074+0000","last_undegraded":"2024-09-17T04:07:34.112074+0000","last_fullsized":"2024-09-17T04:07:34.112074+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:11:35.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.17","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980565+0000","last_change":"2024-09-17T04:07:32.991229+0000","last_active":"2024-09-17T04:07:33.980565+0000","last_peered":"2024-09-17T04:07:33.980565+0000","last_clean":"2024-09-17T04:07:33.980565+0000","last_became_active":"2024-09-17T04:07:32.990912+0000","last_became_peered":"2024-09-17T04:07:32.990912+0000","last_unstale":"2024-09-17T04:07:33.980565+0000","last_undegraded":"2024-09-17T04:07:33.980565+0000","last_fullsized":"2024-09-17T04:07:33.980565+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:26:09.129611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.16","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.966812+0000","last_change":"2024-09-17T04:07:31.984731+0000","last_active":"2024-09-17T04:07:32.966812+0000","last_peered":"2024-09-17T04:07:32.966812+0000","last_clean":"2024-09-17T04:07:32.966812+0000","last_became_active":"2024-09-17T04:07:31.983321+0000","last_became_peered":"2024-09-17T04:07:31.983321+0000","last_unstale":"2024-09-17T04:07:32.966812+0000","last_undegraded":"2024-09-17T04:07:32.966812+0000","last_fullsized":"2024-09-17T04:07:32.966812+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:26:33.408604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.14","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.991653+0000","last_change":"2024-09-17T04:07:32.988780+0000","last_active":"2024-09-17T04:07:32.991653+0000","last_peered":"2024-09-17T04:07:32.991653+0000","last_clean":"2024-09-17T04:07:32.991653+0000","last_became_active":"2024-09-17T04:07:32.988370+0000","last_became_peered":"2024-09-17T04:07:32.988370+0000","last_unstale":"2024-09-17T04:07:32.991653+0000","last_undegraded":"2024-09-17T04:07:32.991653+0000","last_fullsized":"2024-09-17T04:07:32.991653+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:03.786922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.15","version":"259'3","reported_seq":18,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983647+0000","last_change":"2024-09-17T04:07:31.981777+0000","last_active":"2024-09-17T04:07:33.983647+0000","last_peered":"2024-09-17T04:07:33.983647+0000","last_clean":"2024-09-17T04:07:33.983647+0000","last_became_active":"2024-09-17T04:07:31.981328+0000","last_became_peered":"2024-09-17T04:07:31.981328+0000","last_unstale":"2024-09-17T04:07:33.983647+0000","last_undegraded":"2024-09-17T04:07:33.983647+0000","last_fullsized":"2024-09-17T04:07:33.983647+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:38:53.606214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.15","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111348+0000","last_change":"2024-09-17T04:07:32.986053+0000","last_active":"2024-09-17T04:07:34.111348+0000","last_peered":"2024-09-17T04:07:34.111348+0000","last_clean":"2024-09-17T04:07:34.111348+0000","last_became_active":"2024-09-17T04:07:32.984983+0000","last_became_peered":"2024-09-17T04:07:32.984983+0000","last_unstale":"2024-09-17T04:07:34.111348+0000","last_undegraded":"2024-09-17T04:07:34.111348+0000","last_fullsized":"2024-09-17T04:07:34.111348+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:10:06.226511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.14","version":"259'6","reported_seq":21,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112016+0000","last_change":"2024-09-17T04:07:31.985031+0000","last_active":"2024-09-17T04:07:34.112016+0000","last_peered":"2024-09-17T04:07:34.112016+0000","last_clean":"2024-09-17T04:07:34.112016+0000","last_became_active":"2024-09-17T04:07:31.984713+0000","last_became_peered":"2024-09-17T04:07:31.984713+0000","last_unstale":"2024-09-17T04:07:34.112016+0000","last_undegraded":"2024-09-17T04:07:34.112016+0000","last_fullsized":"2024-09-17T04:07:34.112016+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:40:33.048145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.12","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988952+0000","last_change":"2024-09-17T04:07:32.984187+0000","last_active":"2024-09-17T04:07:32.988952+0000","last_peered":"2024-09-17T04:07:32.988952+0000","last_clean":"2024-09-17T04:07:32.988952+0000","last_became_active":"2024-09-17T04:07:32.983900+0000","last_became_peered":"2024-09-17T04:07:32.983900+0000","last_unstale":"2024-09-17T04:07:32.988952+0000","last_undegraded":"2024-09-17T04:07:32.988952+0000","last_fullsized":"2024-09-17T04:07:32.988952+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:32:47.142591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.13","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983160+0000","last_change":"2024-09-17T04:07:31.984769+0000","last_active":"2024-09-17T04:07:33.983160+0000","last_peered":"2024-09-17T04:07:33.983160+0000","last_clean":"2024-09-17T04:07:33.983160+0000","last_became_active":"2024-09-17T04:07:31.984213+0000","last_became_peered":"2024-09-17T04:07:31.984213+0000","last_unstale":"2024-09-17T04:07:33.983160+0000","last_undegraded":"2024-09-17T04:07:33.983160+0000","last_fullsized":"2024-09-17T04:07:33.983160+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:06:15.155284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.13","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111739+0000","last_change":"2024-09-17T04:07:32.992365+0000","last_active":"2024-09-17T04:07:34.111739+0000","last_peered":"2024-09-17T04:07:34.111739+0000","last_clean":"2024-09-17T04:07:34.111739+0000","last_became_active":"2024-09-17T04:07:32.990073+0000","last_became_peered":"2024-09-17T04:07:32.990073+0000","last_unstale":"2024-09-17T04:07:34.111739+0000","last_undegraded":"2024-09-17T04:07:34.111739+0000","last_fullsized":"2024-09-17T04:07:34.111739+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:14:36.646925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.12","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111813+0000","last_change":"2024-09-17T04:07:31.979947+0000","last_active":"2024-09-17T04:07:34.111813+0000","last_peered":"2024-09-17T04:07:34.111813+0000","last_clean":"2024-09-17T04:07:34.111813+0000","last_became_active":"2024-09-17T04:07:31.979589+0000","last_became_peered":"2024-09-17T04:07:31.979589+0000","last_unstale":"2024-09-17T04:07:34.111813+0000","last_undegraded":"2024-09-17T04:07:34.111813+0000","last_fullsized":"2024-09-17T04:07:34.111813+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:42:36.555914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.10","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110669+0000","last_change":"2024-09-17T04:07:32.987461+0000","last_active":"2024-09-17T04:07:34.110669+0000","last_peered":"2024-09-17T04:07:34.110669+0000","last_clean":"2024-09-17T04:07:34.110669+0000","last_became_active":"2024-09-17T04:07:32.987103+0000","last_became_peered":"2024-09-17T04:07:32.987103+0000","last_unstale":"2024-09-17T04:07:34.110669+0000","last_undegraded":"2024-09-17T04:07:34.110669+0000","last_fullsized":"2024-09-17T04:07:34.110669+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:42:30.576687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.11","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110849+0000","last_change":"2024-09-17T04:07:31.975990+0000","last_active":"2024-09-17T04:07:34.110849+0000","last_peered":"2024-09-17T04:07:34.110849+0000","last_clean":"2024-09-17T04:07:34.110849+0000","last_became_active":"2024-09-17T04:07:31.975663+0000","last_became_peered":"2024-09-17T04:07:31.975663+0000","last_unstale":"2024-09-17T04:07:34.110849+0000","last_undegraded":"2024-09-17T04:07:34.110849+0000","last_fullsized":"2024-09-17T04:07:34.110849+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:04:02.396672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.11","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979402+0000","last_change":"2024-09-17T04:07:32.982154+0000","last_active":"2024-09-17T04:07:33.979402+0000","last_peered":"2024-09-17T04:07:33.979402+0000","last_clean":"2024-09-17T04:07:33.979402+0000","last_became_active":"2024-09-17T04:07:32.981844+0000","last_became_peered":"2024-09-17T04:07:32.981844+0000","last_unstale":"2024-09-17T04:07:33.979402+0000","last_undegraded":"2024-09-17T04:07:33.979402+0000","last_fullsized":"2024-09-17T04:07:33.979402+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:28:28.720465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.10","version":"259'8","reported_seq":23,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979519+0000","last_change":"2024-09-17T04:07:31.979708+0000","last_active":"2024-09-17T04:07:33.979519+0000","last_peered":"2024-09-17T04:07:33.979519+0000","last_clean":"2024-09-17T04:07:33.979519+0000","last_became_active":"2024-09-17T04:07:31.979404+0000","last_became_peered":"2024-09-17T04:07:31.979404+0000","last_unstale":"2024-09-17T04:07:33.979519+0000","last_undegraded":"2024-09-17T04:07:33.979519+0000","last_fullsized":"2024-09-17T04:07:33.979519+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:31:15.762540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.e","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980423+0000","last_change":"2024-09-17T04:07:32.991396+0000","last_active":"2024-09-17T04:07:33.980423+0000","last_peered":"2024-09-17T04:07:33.980423+0000","last_clean":"2024-09-17T04:07:33.980423+0000","last_became_active":"2024-09-17T04:07:32.990662+0000","last_became_peered":"2024-09-17T04:07:32.990662+0000","last_unstale":"2024-09-17T04:07:33.980423+0000","last_undegraded":"2024-09-17T04:07:33.980423+0000","last_fullsized":"2024-09-17T04:07:33.980423+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:54:36.490543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.f","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110830+0000","last_change":"2024-09-17T04:07:31.977122+0000","last_active":"2024-09-17T04:07:34.110830+0000","last_peered":"2024-09-17T04:07:34.110830+0000","last_clean":"2024-09-17T04:07:34.110830+0000","last_became_active":"2024-09-17T04:07:31.976665+0000","last_became_peered":"2024-09-17T04:07:31.976665+0000","last_unstale":"2024-09-17T04:07:34.110830+0000","last_undegraded":"2024-09-17T04:07:34.110830+0000","last_fullsized":"2024-09-17T04:07:34.110830+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:46:28.158912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"39.f","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.988324+0000","last_change":"2024-09-17T04:07:32.984836+0000","last_active":"2024-09-17T04:07:32.988324+0000","last_peered":"2024-09-17T04:07:32.988324+0000","last_clean":"2024-09-17T04:07:32.988324+0000","last_became_active":"2024-09-17T04:07:32.984508+0000","last_became_peered":"2024-09-17T04:07:32.984508+0000","last_unstale":"2024-09-17T04:07:32.988324+0000","last_undegraded":"2024-09-17T04:07:32.988324+0000","last_fullsized":"2024-09-17T04:07:32.988324+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:59:01.631445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.e","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983500+0000","last_change":"2024-09-17T04:07:31.981677+0000","last_active":"2024-09-17T04:07:33.983500+0000","last_peered":"2024-09-17T04:07:33.983500+0000","last_clean":"2024-09-17T04:07:33.983500+0000","last_became_active":"2024-09-17T04:07:31.980310+0000","last_became_peered":"2024-09-17T04:07:31.980310+0000","last_unstale":"2024-09-17T04:07:33.983500+0000","last_undegraded":"2024-09-17T04:07:33.983500+0000","last_fullsized":"2024-09-17T04:07:33.983500+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:09:13.573652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.c","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110813+0000","last_change":"2024-09-17T04:07:32.989179+0000","last_active":"2024-09-17T04:07:34.110813+0000","last_peered":"2024-09-17T04:07:34.110813+0000","last_clean":"2024-09-17T04:07:34.110813+0000","last_became_active":"2024-09-17T04:07:32.988848+0000","last_became_peered":"2024-09-17T04:07:32.988848+0000","last_unstale":"2024-09-17T04:07:34.110813+0000","last_undegraded":"2024-09-17T04:07:34.110813+0000","last_fullsized":"2024-09-17T04:07:34.110813+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:00:19.727704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.d","version":"259'1","reported_seq":16,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111312+0000","last_change":"2024-09-17T04:07:31.980890+0000","last_active":"2024-09-17T04:07:34.111312+0000","last_peered":"2024-09-17T04:07:34.111312+0000","last_clean":"2024-09-17T04:07:34.111312+0000","last_became_active":"2024-09-17T04:07:31.980550+0000","last_became_peered":"2024-09-17T04:07:31.980550+0000","last_unstale":"2024-09-17T04:07:34.111312+0000","last_undegraded":"2024-09-17T04:07:34.111312+0000","last_fullsized":"2024-09-17T04:07:34.111312+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:41:05.765884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.d","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980127+0000","last_change":"2024-09-17T04:07:32.991987+0000","last_active":"2024-09-17T04:07:33.980127+0000","last_peered":"2024-09-17T04:07:33.980127+0000","last_clean":"2024-09-17T04:07:33.980127+0000","last_became_active":"2024-09-17T04:07:32.991654+0000","last_became_peered":"2024-09-17T04:07:32.991654+0000","last_unstale":"2024-09-17T04:07:33.980127+0000","last_undegraded":"2024-09-17T04:07:33.980127+0000","last_fullsized":"2024-09-17T04:07:33.980127+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:04:22.939020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.c","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111268+0000","last_change":"2024-09-17T04:07:31.976593+0000","last_active":"2024-09-17T04:07:34.111268+0000","last_peered":"2024-09-17T04:07:34.111268+0000","last_clean":"2024-09-17T04:07:34.111268+0000","last_became_active":"2024-09-17T04:07:31.976252+0000","last_became_peered":"2024-09-17T04:07:31.976252+0000","last_unstale":"2024-09-17T04:07:34.111268+0000","last_undegraded":"2024-09-17T04:07:34.111268+0000","last_fullsized":"2024-09-17T04:07:34.111268+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:24:27.697245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"39.a","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979465+0000","last_change":"2024-09-17T04:07:32.992356+0000","last_active":"2024-09-17T04:07:33.979465+0000","last_peered":"2024-09-17T04:07:33.979465+0000","last_clean":"2024-09-17T04:07:33.979465+0000","last_became_active":"2024-09-17T04:07:32.991602+0000","last_became_peered":"2024-09-17T04:07:32.991602+0000","last_unstale":"2024-09-17T04:07:33.979465+0000","last_undegraded":"2024-09-17T04:07:33.979465+0000","last_fullsized":"2024-09-17T04:07:33.979465+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:22:44.679911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.b","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.983288+0000","last_change":"2024-09-17T04:07:31.981151+0000","last_active":"2024-09-17T04:07:33.983288+0000","last_peered":"2024-09-17T04:07:33.983288+0000","last_clean":"2024-09-17T04:07:33.983288+0000","last_became_active":"2024-09-17T04:07:31.980830+0000","last_became_peered":"2024-09-17T04:07:31.980830+0000","last_unstale":"2024-09-17T04:07:33.983288+0000","last_undegraded":"2024-09-17T04:07:33.983288+0000","last_fullsized":"2024-09-17T04:07:33.983288+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:31:42.710543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"39.b","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111433+0000","last_change":"2024-09-17T04:07:32.986662+0000","last_active":"2024-09-17T04:07:34.111433+0000","last_peered":"2024-09-17T04:07:34.111433+0000","last_clean":"2024-09-17T04:07:34.111433+0000","last_became_active":"2024-09-17T04:07:32.986330+0000","last_became_peered":"2024-09-17T04:07:32.986330+0000","last_unstale":"2024-09-17T04:07:34.111433+0000","last_undegraded":"2024-09-17T04:07:34.111433+0000","last_fullsized":"2024-09-17T04:07:34.111433+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:49.282083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.a","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110831+0000","last_change":"2024-09-17T04:07:31.979053+0000","last_active":"2024-09-17T04:07:34.110831+0000","last_peered":"2024-09-17T04:07:34.110831+0000","last_clean":"2024-09-17T04:07:34.110831+0000","last_became_active":"2024-09-17T04:07:31.978514+0000","last_became_peered":"2024-09-17T04:07:31.978514+0000","last_unstale":"2024-09-17T04:07:34.110831+0000","last_undegraded":"2024-09-17T04:07:34.110831+0000","last_fullsized":"2024-09-17T04:07:34.110831+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:46:01.119902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.8","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.977852+0000","last_change":"2024-09-17T04:07:32.970811+0000","last_active":"2024-09-17T04:07:32.977852+0000","last_peered":"2024-09-17T04:07:32.977852+0000","last_clean":"2024-09-17T04:07:32.977852+0000","last_became_active":"2024-09-17T04:07:32.970308+0000","last_became_peered":"2024-09-17T04:07:32.970308+0000","last_unstale":"2024-09-17T04:07:32.977852+0000","last_undegraded":"2024-09-17T04:07:32.977852+0000","last_fullsized":"2024-09-17T04:07:32.977852+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:50:56.496046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"38.9","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979538+0000","last_change":"2024-09-17T04:07:31.981085+0000","last_active":"2024-09-17T04:07:33.979538+0000","last_peered":"2024-09-17T04:07:33.979538+0000","last_clean":"2024-09-17T04:07:33.979538+0000","last_became_active":"2024-09-17T04:07:31.980370+0000","last_became_peered":"2024-09-17T04:07:31.980370+0000","last_unstale":"2024-09-17T04:07:33.979538+0000","last_undegraded":"2024-09-17T04:07:33.979538+0000","last_fullsized":"2024-09-17T04:07:33.979538+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:42.141454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.9","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111313+0000","last_change":"2024-09-17T04:07:32.994067+0000","last_active":"2024-09-17T04:07:34.111313+0000","last_peered":"2024-09-17T04:07:34.111313+0000","last_clean":"2024-09-17T04:07:34.111313+0000","last_became_active":"2024-09-17T04:07:32.990879+0000","last_became_peered":"2024-09-17T04:07:32.990879+0000","last_unstale":"2024-09-17T04:07:34.111313+0000","last_undegraded":"2024-09-17T04:07:34.111313+0000","last_fullsized":"2024-09-17T04:07:34.111313+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:01:44.964949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.8","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110830+0000","last_change":"2024-09-17T04:07:31.979633+0000","last_active":"2024-09-17T04:07:34.110830+0000","last_peered":"2024-09-17T04:07:34.110830+0000","last_clean":"2024-09-17T04:07:34.110830+0000","last_became_active":"2024-09-17T04:07:31.979312+0000","last_became_peered":"2024-09-17T04:07:31.979312+0000","last_unstale":"2024-09-17T04:07:34.110830+0000","last_undegraded":"2024-09-17T04:07:34.110830+0000","last_fullsized":"2024-09-17T04:07:34.110830+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:38.396795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.6","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111557+0000","last_change":"2024-09-17T04:07:32.988066+0000","last_active":"2024-09-17T04:07:34.111557+0000","last_peered":"2024-09-17T04:07:34.111557+0000","last_clean":"2024-09-17T04:07:34.111557+0000","last_became_active":"2024-09-17T04:07:32.986833+0000","last_became_peered":"2024-09-17T04:07:32.986833+0000","last_unstale":"2024-09-17T04:07:34.111557+0000","last_undegraded":"2024-09-17T04:07:34.111557+0000","last_fullsized":"2024-09-17T04:07:34.111557+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:40:46.680154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"38.7","version":"259'2","reported_seq":17,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.980548+0000","last_change":"2024-09-17T04:07:31.987739+0000","last_active":"2024-09-17T04:07:33.980548+0000","last_peered":"2024-09-17T04:07:33.980548+0000","last_clean":"2024-09-17T04:07:33.980548+0000","last_became_active":"2024-09-17T04:07:31.987347+0000","last_became_peered":"2024-09-17T04:07:31.987347+0000","last_unstale":"2024-09-17T04:07:33.980548+0000","last_undegraded":"2024-09-17T04:07:33.980548+0000","last_fullsized":"2024-09-17T04:07:33.980548+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:20.734459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.7","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111013+0000","last_change":"2024-09-17T04:07:32.989866+0000","last_active":"2024-09-17T04:07:34.111013+0000","last_peered":"2024-09-17T04:07:34.111013+0000","last_clean":"2024-09-17T04:07:34.111013+0000","last_became_active":"2024-09-17T04:07:32.989556+0000","last_became_peered":"2024-09-17T04:07:32.989556+0000","last_unstale":"2024-09-17T04:07:34.111013+0000","last_undegraded":"2024-09-17T04:07:34.111013+0000","last_fullsized":"2024-09-17T04:07:34.111013+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:12:48.555893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"38.6","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.967053+0000","last_change":"2024-09-17T04:07:31.987083+0000","last_active":"2024-09-17T04:07:32.967053+0000","last_peered":"2024-09-17T04:07:32.967053+0000","last_clean":"2024-09-17T04:07:32.967053+0000","last_became_active":"2024-09-17T04:07:31.986672+0000","last_became_peered":"2024-09-17T04:07:31.986672+0000","last_unstale":"2024-09-17T04:07:32.967053+0000","last_undegraded":"2024-09-17T04:07:32.967053+0000","last_fullsized":"2024-09-17T04:07:32.967053+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:28:14.294187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.4","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.985728+0000","last_change":"2024-09-17T04:07:32.977861+0000","last_active":"2024-09-17T04:07:32.985728+0000","last_peered":"2024-09-17T04:07:32.985728+0000","last_clean":"2024-09-17T04:07:32.985728+0000","last_became_active":"2024-09-17T04:07:32.977518+0000","last_became_peered":"2024-09-17T04:07:32.977518+0000","last_unstale":"2024-09-17T04:07:32.985728+0000","last_undegraded":"2024-09-17T04:07:32.985728+0000","last_fullsized":"2024-09-17T04:07:32.985728+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:08.095384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"38.5","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.968235+0000","last_change":"2024-09-17T04:07:31.989402+0000","last_active":"2024-09-17T04:07:32.968235+0000","last_peered":"2024-09-17T04:07:32.968235+0000","last_clean":"2024-09-17T04:07:32.968235+0000","last_became_active":"2024-09-17T04:07:31.989048+0000","last_became_peered":"2024-09-17T04:07:31.989048+0000","last_unstale":"2024-09-17T04:07:32.968235+0000","last_undegraded":"2024-09-17T04:07:32.968235+0000","last_fullsized":"2024-09-17T04:07:32.968235+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:20:07.763138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.5","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112001+0000","last_change":"2024-09-17T04:07:32.985734+0000","last_active":"2024-09-17T04:07:34.112001+0000","last_peered":"2024-09-17T04:07:34.112001+0000","last_clean":"2024-09-17T04:07:34.112001+0000","last_became_active":"2024-09-17T04:07:32.985383+0000","last_became_peered":"2024-09-17T04:07:32.985383+0000","last_unstale":"2024-09-17T04:07:34.112001+0000","last_undegraded":"2024-09-17T04:07:34.112001+0000","last_fullsized":"2024-09-17T04:07:34.112001+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:38:38.204658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.4","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.112102+0000","last_change":"2024-09-17T04:07:31.982794+0000","last_active":"2024-09-17T04:07:34.112102+0000","last_peered":"2024-09-17T04:07:34.112102+0000","last_clean":"2024-09-17T04:07:34.112102+0000","last_became_active":"2024-09-17T04:07:31.982418+0000","last_became_peered":"2024-09-17T04:07:31.982418+0000","last_unstale":"2024-09-17T04:07:34.112102+0000","last_undegraded":"2024-09-17T04:07:34.112102+0000","last_fullsized":"2024-09-17T04:07:34.112102+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:15:08.211420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"39.2","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979657+0000","last_change":"2024-09-17T04:07:32.992150+0000","last_active":"2024-09-17T04:07:33.979657+0000","last_peered":"2024-09-17T04:07:33.979657+0000","last_clean":"2024-09-17T04:07:33.979657+0000","last_became_active":"2024-09-17T04:07:32.988189+0000","last_became_peered":"2024-09-17T04:07:32.988189+0000","last_unstale":"2024-09-17T04:07:33.979657+0000","last_undegraded":"2024-09-17T04:07:33.979657+0000","last_fullsized":"2024-09-17T04:07:33.979657+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:33.257905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"38.3","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.967826+0000","last_change":"2024-09-17T04:07:31.985171+0000","last_active":"2024-09-17T04:07:32.967826+0000","last_peered":"2024-09-17T04:07:32.967826+0000","last_clean":"2024-09-17T04:07:32.967826+0000","last_became_active":"2024-09-17T04:07:31.982940+0000","last_became_peered":"2024-09-17T04:07:31.982940+0000","last_unstale":"2024-09-17T04:07:32.967826+0000","last_undegraded":"2024-09-17T04:07:32.967826+0000","last_fullsized":"2024-09-17T04:07:32.967826+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:18.174236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"39.3","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111597+0000","last_change":"2024-09-17T04:07:32.989740+0000","last_active":"2024-09-17T04:07:34.111597+0000","last_peered":"2024-09-17T04:07:34.111597+0000","last_clean":"2024-09-17T04:07:34.111597+0000","last_became_active":"2024-09-17T04:07:32.988512+0000","last_became_peered":"2024-09-17T04:07:32.988512+0000","last_unstale":"2024-09-17T04:07:34.111597+0000","last_undegraded":"2024-09-17T04:07:34.111597+0000","last_fullsized":"2024-09-17T04:07:34.111597+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:19:52.733478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"38.2","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.979588+0000","last_change":"2024-09-17T04:07:31.981905+0000","last_active":"2024-09-17T04:07:33.979588+0000","last_peered":"2024-09-17T04:07:33.979588+0000","last_clean":"2024-09-17T04:07:33.979588+0000","last_became_active":"2024-09-17T04:07:31.981624+0000","last_became_peered":"2024-09-17T04:07:31.981624+0000","last_unstale":"2024-09-17T04:07:33.979588+0000","last_undegraded":"2024-09-17T04:07:33.979588+0000","last_fullsized":"2024-09-17T04:07:33.979588+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:37:42.246345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"39.0","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:33.982566+0000","last_change":"2024-09-17T04:07:32.996126+0000","last_active":"2024-09-17T04:07:33.982566+0000","last_peered":"2024-09-17T04:07:33.982566+0000","last_clean":"2024-09-17T04:07:33.982566+0000","last_became_active":"2024-09-17T04:07:32.995757+0000","last_became_peered":"2024-09-17T04:07:32.995757+0000","last_unstale":"2024-09-17T04:07:33.982566+0000","last_undegraded":"2024-09-17T04:07:33.982566+0000","last_fullsized":"2024-09-17T04:07:33.982566+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:33:58.594708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"38.1","version":"0'0","reported_seq":13,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.966094+0000","last_change":"2024-09-17T04:07:31.970742+0000","last_active":"2024-09-17T04:07:32.966094+0000","last_peered":"2024-09-17T04:07:32.966094+0000","last_clean":"2024-09-17T04:07:32.966094+0000","last_became_active":"2024-09-17T04:07:31.970441+0000","last_became_peered":"2024-09-17T04:07:31.970441+0000","last_unstale":"2024-09-17T04:07:32.966094+0000","last_undegraded":"2024-09-17T04:07:32.966094+0000","last_fullsized":"2024-09-17T04:07:32.966094+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:43:37.098010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"39.1","version":"0'0","reported_seq":13,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.111582+0000","last_change":"2024-09-17T04:07:32.987616+0000","last_active":"2024-09-17T04:07:34.111582+0000","last_peered":"2024-09-17T04:07:34.111582+0000","last_clean":"2024-09-17T04:07:34.111582+0000","last_became_active":"2024-09-17T04:07:32.987287+0000","last_became_peered":"2024-09-17T04:07:32.987287+0000","last_unstale":"2024-09-17T04:07:34.111582+0000","last_undegraded":"2024-09-17T04:07:34.111582+0000","last_fullsized":"2024-09-17T04:07:34.111582+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:16:58.994691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"38.0","version":"0'0","reported_seq":15,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110529+0000","last_change":"2024-09-17T04:07:31.976942+0000","last_active":"2024-09-17T04:07:34.110529+0000","last_peered":"2024-09-17T04:07:34.110529+0000","last_clean":"2024-09-17T04:07:34.110529+0000","last_became_active":"2024-09-17T04:07:31.975586+0000","last_became_peered":"2024-09-17T04:07:31.975586+0000","last_unstale":"2024-09-17T04:07:34.110529+0000","last_undegraded":"2024-09-17T04:07:34.110529+0000","last_fullsized":"2024-09-17T04:07:34.110529+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":257,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:30.944270+0000","last_clean_scrub_stamp":"2024-09-17T04:07:30.944270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:23.507954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"39.26","version":"0'0","reported_seq":10,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-09-17T04:07:32.983826+0000","last_change":"2024-09-17T04:07:32.975054+0000","last_active":"2024-09-17T04:07:32.983826+0000","last_peered":"2024-09-17T04:07:32.983826+0000","last_clean":"2024-09-17T04:07:32.983826+0000","last_became_active":"2024-09-17T04:07:32.974767+0000","last_became_peered":"2024-09-17T04:07:32.974767+0000","last_unstale":"2024-09-17T04:07:32.983826+0000","last_undegraded":"2024-09-17T04:07:32.983826+0000","last_fullsized":"2024-09-17T04:07:32.983826+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:07:31.951694+0000","last_clean_scrub_stamp":"2024-09-17T04:07:31.951694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:35:50.471149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":552,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-09-17T04:07:34.110738+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:07:34.110738+0000","last_peered":"2024-09-17T04:07:34.110738+0000","last_clean":"2024-09-17T04:07:34.110738+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:07:34.110738+0000","last_undegraded":"2024-09-17T04:07:34.110738+0000","last_fullsized":"2024-09-17T04:07:34.110738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":39,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":38,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509660,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53032,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318808,"statfs":{"total":96636764160,"available":96582459392,"internally_reserved":0,"allocated":3887104,"data_stored":2047087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509663,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34424,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337416,"statfs":{"total":96636764160,"available":96601513984,"internally_reserved":0,"allocated":3305472,"data_stored":1456865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56604,"kb_used_data":3208,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315236,"statfs":{"total":96636764160,"available":96578801664,"internally_reserved":0,"allocated":3284992,"data_stored":1455177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509660,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56616,"kb_used_data":3216,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315224,"statfs":{"total":96636764160,"available":96578789376,"internally_reserved":0,"allocated":3293184,"data_stored":1461407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56620,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315220,"statfs":{"total":96636764160,"available":96578785280,"internally_reserved":0,"allocated":3297280,"data_stored":1461989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509662,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52460,"kb_used_data":3220,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319380,"statfs":{"total":96636764160,"available":96583045120,"internally_reserved":0,"allocated":3297280,"data_stored":1461429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56592,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315248,"statfs":{"total":96636764160,"available":96578813952,"internally_reserved":0,"allocated":3268608,"data_stored":1448332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542365,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":3887104,"data_stored":2046217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":38,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":39,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:07:37.890 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:07:37.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:07:38.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 -- 172.21.15.129:0/2036067684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 msgr2=0x7ff444160d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 --2- 172.21.15.129:0/2036067684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff444160d50 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7ff434005fd0 tx=0x7ff43402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 -- 172.21.15.129:0/2036067684 shutdown_connections 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 --2- 172.21.15.129:0/2036067684 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff444160d50 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 -- 172.21.15.129:0/2036067684 >> 172.21.15.129:0/2036067684 conn(0x7ff4440d0120 msgr2=0x7ff4440d0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 -- 172.21.15.129:0/2036067684 shutdown_connections 2024-09-17T04:07:38.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.154+0000 7ff448c00640 1 -- 172.21.15.129:0/2036067684 wait complete. 2024-09-17T04:07:38.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.157+0000 7ff448c00640 1 Processor -- start 2024-09-17T04:07:38.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.157+0000 7ff448c00640 1 -- start start 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406ee70 0x7ff4441f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4440ce2c0 0x7ff44415ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4440cef60 con 0x7ff444156f20 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4440cede0 con 0x7ff44406ee70 2024-09-17T04:07:38.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff448c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff4440cf0e0 con 0x7ff4440ce2c0 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff443e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff443e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46536/0 (socket says 172.21.15.129:46536) 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.158+0000 7ff443e00640 1 -- 172.21.15.129:0/2130883850 learned_addr learned my addr 172.21.15.129:0/2130883850 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443400640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406ee70 0x7ff4441f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff43aa00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4440ce2c0 0x7ff44415ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 -- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4440ce2c0 msgr2=0x7ff44415ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4440ce2c0 0x7ff44415ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 -- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406ee70 msgr2=0x7ff4441f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406ee70 0x7ff4441f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 -- 172.21.15.129:0/2130883850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4440c71d0 con 0x7ff444156f20 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff443e00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00b970 tx=0x7ff43c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.159+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c019070 con 0x7ff444156f20 2024-09-17T04:07:38.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.160+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff43c00cc30 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.160+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43c00ce60 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.160+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4441f69a0 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.160+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4440cb070 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.161+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff444160d50 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.162+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff43c01c070 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.163+0000 7ff43be00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 0x7ff42007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.163+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7ff43c09a980 con 0x7ff444156f20 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.164+0000 7ff443400640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 0x7ff42007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.165+0000 7ff443400640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 0x7ff42007b0a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff434033000 tx=0x7ff434004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.168+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff43c0626d0 con 0x7ff444156f20 2024-09-17T04:07:38.423 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.423+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff4440cab20 con 0x7ff444156f20 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.428+0000 7ff43be00640 1 -- 172.21.15.129:0/2130883850 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v47) ==== 190+0+147 (secure 0 0 0) 0x7ff43c067100 con 0x7ff444156f20 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:07:38.428 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:07:38.429 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:07:38.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 msgr2=0x7ff42007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 0x7ff42007b0a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7ff434033000 tx=0x7ff434004730 comp rx=0 tx=0).stop 2024-09-17T04:07:38.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 msgr2=0x7ff4441ef380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7ff43c00b970 tx=0x7ff43c00be30 comp rx=0 tx=0).stop 2024-09-17T04:07:38.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 shutdown_connections 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff420078b90 0x7ff42007b0a0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff444156f20 0x7ff4441ef380 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4440ce2c0 0x7ff44415ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 --2- 172.21.15.129:0/2130883850 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff44406ee70 0x7ff4441f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 >> 172.21.15.129:0/2130883850 conn(0x7ff4440d0120 msgr2=0x7ff4440cf5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 shutdown_connections 2024-09-17T04:07:38.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.431+0000 7ff448c00640 1 -- 172.21.15.129:0/2130883850 wait complete. 2024-09-17T04:07:38.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 -- 172.21.15.129:0/2160596502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 msgr2=0x7fe7340b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 --2- 172.21.15.129:0/2160596502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe7340b4140 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7fe724005fd0 tx=0x7fe72402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 -- 172.21.15.129:0/2160596502 shutdown_connections 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 --2- 172.21.15.129:0/2160596502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe7340cd390 0x7fe7341609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 --2- 172.21.15.129:0/2160596502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe7340b4140 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 -- 172.21.15.129:0/2160596502 >> 172.21.15.129:0/2160596502 conn(0x7fe73406d410 msgr2=0x7fe7340b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.676+0000 7fe73a000640 1 -- 172.21.15.129:0/2160596502 shutdown_connections 2024-09-17T04:07:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.677+0000 7fe73a000640 1 -- 172.21.15.129:0/2160596502 wait complete. 2024-09-17T04:07:38.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.679+0000 7fe73a000640 1 Processor -- start 2024-09-17T04:07:38.679 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.679+0000 7fe73a000640 1 -- start start 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe73406eed0 0x7fe7341f8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe734159e70 0x7fe73415b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe734160f00 con 0x7fe7340705a0 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe73406f4f0 con 0x7fe734159e70 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.680+0000 7fe73a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe734161080 con 0x7fe73406eed0 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46560/0 (socket says 172.21.15.129:46560) 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 -- 172.21.15.129:0/3255501700 learned_addr learned my addr 172.21.15.129:0/3255501700 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe733400640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe73406eed0 0x7fe7341f8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe733e00640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe734159e70 0x7fe73415b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 -- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe73406eed0 msgr2=0x7fe7341f8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe73406eed0 0x7fe7341f8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 -- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe734159e70 msgr2=0x7fe73415b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe734159e70 0x7fe73415b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 -- 172.21.15.129:0/3255501700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73409ab50 con 0x7fe7340705a0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.681+0000 7fe732a00640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fe72000c370 tx=0x7fe72000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.682+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7200052d0 con 0x7fe7340705a0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.682+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe720007020 con 0x7fe7340705a0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.682+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe720007470 con 0x7fe7340705a0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.682+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7341f9090 con 0x7fe7340705a0 2024-09-17T04:07:38.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.682+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe734155e80 con 0x7fe7340705a0 2024-09-17T04:07:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.683+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7340b4140 con 0x7fe7340705a0 2024-09-17T04:07:38.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.684+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe7200067d0 con 0x7fe7340705a0 2024-09-17T04:07:38.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.685+0000 7fe72b400640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 0x7fe71407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:38.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.685+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fe72009a1f0 con 0x7fe7340705a0 2024-09-17T04:07:38.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.686+0000 7fe733400640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 0x7fe71407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:38.686 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.686+0000 7fe733400640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 0x7fe71407b0d0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe724033000 tx=0x7fe724004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:38.689 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.689+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe720061fd0 con 0x7fe7340705a0 2024-09-17T04:07:38.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.906+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe73406a140 con 0x7fe7340705a0 2024-09-17T04:07:38.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.906+0000 7fe72b400640 1 -- 172.21.15.129:0/3255501700 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) ==== 95+0+5246 (secure 0 0 0) 0x7fe720066a00 con 0x7fe7340705a0 2024-09-17T04:07:38.907 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:38.907 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":249,"btime":"2024-09-17T04:07:35:999196+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/47559844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":47559844},{"type":"v1","addr":"172.21.15.181:6836","nonce":47559844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":11266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3086573184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3086573184},{"type":"v1","addr":"172.21.15.181:6837","nonce":3086573184}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":12240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1080794990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080794990},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080794990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":243}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:35.488097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12249},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:38.907 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 249 2024-09-17T04:07:38.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 msgr2=0x7fe71407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 0x7fe71407b0d0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fe724033000 tx=0x7fe724004730 comp rx=0 tx=0).stop 2024-09-17T04:07:38.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 msgr2=0x7fe73415af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7fe72000c370 tx=0x7fe72000c830 comp rx=0 tx=0).stop 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 shutdown_connections 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe714078bc0 0x7fe71407b0d0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe734159e70 0x7fe73415b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe7340705a0 0x7fe73415af50 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 --2- 172.21.15.129:0/3255501700 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe73406eed0 0x7fe7341f8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 >> 172.21.15.129:0/3255501700 conn(0x7fe73406d410 msgr2=0x7fe73415f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 shutdown_connections 2024-09-17T04:07:38.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:38.909+0000 7fe73a000640 1 -- 172.21.15.129:0/3255501700 wait complete. 2024-09-17T04:07:39.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 249, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:07:32.455412+0000', 'modified': '2024-09-17T04:07:35.488097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12249}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12249': {'gid': 12249, 'name': 'a', 'rank': 0, 'incarnation': 245, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3848773989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3848773989}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3848773989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12249, 'qdb_cluster': [12249]} 2024-09-17T04:07:39.334 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:07:39.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T04:07:39.556 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 -- 172.21.15.129:0/1682815906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700d0050 msgr2=0x7f17700d14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 --2- 172.21.15.129:0/1682815906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700d0050 0x7f17700d14d0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f17700d05e0 tx=0x7f176802aaa0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 -- 172.21.15.129:0/1682815906 shutdown_connections 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 --2- 172.21.15.129:0/1682815906 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700d0050 0x7f17700d14d0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 --2- 172.21.15.129:0/1682815906 >> v2:172.21.15.181:3300/0 conn(0x7f17700b9be0 0x7f17700d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 -- 172.21.15.129:0/1682815906 >> 172.21.15.129:0/1682815906 conn(0x7f177008bff0 msgr2=0x7f177009d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 -- 172.21.15.129:0/1682815906 shutdown_connections 2024-09-17T04:07:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.556+0000 7f1776a00640 1 -- 172.21.15.129:0/1682815906 wait complete. 2024-09-17T04:07:39.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.558+0000 7f1776a00640 1 Processor -- start 2024-09-17T04:07:39.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.558+0000 7f1776a00640 1 -- start start 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17700b3430 0x7f17700ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17700d0050 0x7f177014e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17700b1c60 con 0x7f17700b9be0 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17700b1ae0 con 0x7f17700b3430 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17700b1de0 con 0x7f17700d0050 2024-09-17T04:07:39.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46574/0 (socket says 172.21.15.129:46574) 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 -- 172.21.15.129:0/2297872231 learned_addr learned my addr 172.21.15.129:0/2297872231 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1775600640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17700b3430 0x7f17700ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1776000640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17700d0050 0x7f177014e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 -- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17700d0050 msgr2=0x7f177014e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17700d0050 0x7f177014e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 -- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17700b3430 msgr2=0x7f17700ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17700b3430 0x7f17700ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.560+0000 7f1774c00640 1 -- 172.21.15.129:0/2297872231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1770151b40 con 0x7f17700b9be0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f1774c00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f175c00c3d0 tx=0x7f175c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f176d600640 1 -- 172.21.15.129:0/2297872231 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c017070 con 0x7f17700b9be0 2024-09-17T04:07:39.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f176d600640 1 -- 172.21.15.129:0/2297872231 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f175c0053c0 con 0x7f17700b9be0 2024-09-17T04:07:39.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f176d600640 1 -- 172.21.15.129:0/2297872231 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c003850 con 0x7f17700b9be0 2024-09-17T04:07:39.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17700cd2f0 con 0x7f17700b9be0 2024-09-17T04:07:39.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.561+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17700cd8b0 con 0x7f17700b9be0 2024-09-17T04:07:39.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.563+0000 7f176d600640 1 -- 172.21.15.129:0/2297872231 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f175c01a070 con 0x7f17700b9be0 2024-09-17T04:07:39.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.564+0000 7f176d600640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 0x7f175007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.564+0000 7f176d600640 1 -- 172.21.15.129:0/2297872231 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f175c09a0a0 con 0x7f17700b9be0 2024-09-17T04:07:39.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.565+0000 7f1775600640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 0x7f175007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.565+0000 7f177f0b5740 1 Processor -- start 2024-09-17T04:07:39.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.565+0000 7f177f0b5740 1 -- start start 2024-09-17T04:07:39.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f1775600640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 0x7f175007b0d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f17700d4000 tx=0x7f176400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2668270 0x5615e2a38b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a390a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5615e2992430 con 0x5615e2647590 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5615e29825d0 con 0x5615e2668270 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.566+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5615e2971a00 con 0x5615e263f3b0 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46584/0 (socket says 172.21.15.129:46584) 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 -- 172.21.15.129:0/1718403490 learned_addr learned my addr 172.21.15.129:0/1718403490 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1774c00640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a390a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1775600640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2668270 0x5615e2a38b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 -- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 msgr2=0x5615e2a390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a390a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 -- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2668270 msgr2=0x5615e2a38b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2668270 0x5615e2a38b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1776000640 1 -- 172.21.15.129:0/1718403490 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e293a520 con 0x5615e2647590 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.567+0000 7f1774c00640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.568+0000 7f1776000640 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f1768005fd0 tx=0x7f1768004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec956798d74e581 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.568+0000 7f1761c00640 1 -- 172.21.15.129:0/1718403490 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768045070 con 0x5615e2647590 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.568+0000 7f1761c00640 1 -- 172.21.15.129:0/1718403490 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f176802ecf0 con 0x5615e2647590 2024-09-17T04:07:39.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.568+0000 7f1761c00640 1 -- 172.21.15.129:0/1718403490 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768003bd0 con 0x5615e2647590 2024-09-17T04:07:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 -- 172.21.15.129:0/1718403490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 msgr2=0x5615e2a09150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f1768005fd0 tx=0x7f1768004ab0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 -- 172.21.15.129:0/1718403490 shutdown_connections 2024-09-17T04:07:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a09150 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a390a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1718403490 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2668270 0x5615e2a38b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 -- 172.21.15.129:0/1718403490 >> 172.21.15.129:0/1718403490 conn(0x5615e265c3c0 msgr2=0x5615e2a0eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 -- 172.21.15.129:0/1718403490 shutdown_connections 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.569+0000 7f177f0b5740 1 -- 172.21.15.129:0/1718403490 wait complete. 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.570+0000 7f177f0b5740 1 -- start start 2024-09-17T04:07:39.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.570+0000 7f177f0b5740 1 Processor -- start 2024-09-17T04:07:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2a1cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 0x5615e2a63040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5615e2918330 con 0x5615e2668270 2024-09-17T04:07:39.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5615e2975810 con 0x5615e263f3b0 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5615e26e2620 con 0x5615e2647590 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f177f0b5740 10 client.0 _ll_register_callbacks cb 0x5615e291d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46600/0 (socket says 172.21.15.129:46600) 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 -- 172.21.15.129:0/3126099662 learned_addr learned my addr 172.21.15.129:0/3126099662 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1775600640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2a1cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1774c00640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 0x5615e2a63040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 -- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 msgr2=0x5615e2a63040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 0x5615e2a63040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 -- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 msgr2=0x5615e2a1cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2a1cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1776000640 1 -- 172.21.15.129:0/3126099662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e26d3850 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1774c00640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 0x5615e2a63040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.572+0000 7f1775600640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2a1cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1776000640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f176802aae0 tx=0x7f1768003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01b228fca8b760d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176804f020 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f1768004040 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f176802ecf0 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=249) ==== 24+0+0 (secure 0 0 0) 0x7f1768059050 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 10 client.12363 fetch_fsmap learned FSMap version 249 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=249,monmap=2+}) -- 0x5615e274a6b0 con 0x5615e2668270 2024-09-17T04:07:39.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f176802ecf0 con 0x5615e2668270 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 249) ==== 3502+0+0 (secure 0 0 0) 0x7f176805c020 con 0x5615e2668270 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 10 client.12363 fetch_fsmap finished waiting for FSMap version 249 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 10 client.12363 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 20 client.12363 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.573+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2626e40 0x5615e2a35490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.574+0000 7f177f0b5740 4 client.12363 mds_command: new command op to 12249 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.574+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 --> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615e2984520 con 0x5615e2626e40 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.574+0000 7f1775600640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2626e40 0x5615e2a35490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.574+0000 7f1775600640 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2626e40 0x5615e2a35490 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e53e7cedbe01626f server_cookie=ae1a2a92375bc3fa in_seq=0 out_seq=0 2024-09-17T04:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.574+0000 7f1756000640 10 client.12363 ms_handle_connect on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:39.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f1756000640 1 -- 172.21.15.129:0/3126099662 <== mds.0 v2:172.21.15.129:6833/3848773989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5615e2984520 con 0x5615e2626e40 2024-09-17T04:07:39.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f1756000640 10 client.12363 handle_command_reply: tid=0 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f177f0b5740 1 client.12363 shutdown 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 msgr2=0x5615e2a55f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f176802aae0 tx=0x7f1768003f60 comp rx=0 tx=0).stop 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 shutdown_connections 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.575+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2626e40 0x5615e2a35490 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a55f60 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2647590 0x5615e2a63040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 --2- 172.21.15.129:0/3126099662 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2a1cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 >> 172.21.15.129:0/3126099662 conn(0x5615e265c3c0 msgr2=0x5615e2a541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f1756000640 0 client.12363 ms_handle_reset on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 shutdown_connections 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 1 -- 172.21.15.129:0/3126099662 wait complete. 2024-09-17T04:07:39.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.576+0000 7f177f0b5740 20 client.12363 trim_cache size 0 max 16384 2024-09-17T04:07:39.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.594+0000 7f177f0b5740 1 Processor -- start 2024-09-17T04:07:39.594 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.594+0000 7f177f0b5740 1 -- start start 2024-09-17T04:07:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a77b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 0x5615e2a616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f17700cad50 con 0x5615e2668270 2024-09-17T04:07:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f17700ca0e0 con 0x5615e2647590 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f17700ca700 con 0x5615e263f3b0 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46614/0 (socket says 172.21.15.129:46614) 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 -- 172.21.15.129:0/637530306 learned_addr learned my addr 172.21.15.129:0/637530306 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1775600640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a77b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1774c00640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 0x5615e2a616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 -- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 msgr2=0x5615e2a77b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a77b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 -- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 msgr2=0x5615e2a616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 0x5615e2a616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1776000640 1 -- 172.21.15.129:0/637530306 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615e2a0f840 con 0x5615e2668270 2024-09-17T04:07:39.597 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.596+0000 7f1774c00640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 0x5615e2a616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f1776000640 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f1768009a60 tx=0x7f1768004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceba1f017fae1f79 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f1761200640 1 -- 172.21.15.129:0/637530306 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176804f070 con 0x5615e2668270 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f1761200640 1 -- 172.21.15.129:0/637530306 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1768004290 con 0x5615e2668270 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f1761200640 1 -- 172.21.15.129:0/637530306 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768045040 con 0x5615e2668270 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f177f0b5740 1 -- 172.21.15.129:0/637530306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 msgr2=0x5615e2a61c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.597+0000 7f177f0b5740 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f1768009a60 tx=0x7f1768004ab0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 -- 172.21.15.129:0/637530306 shutdown_connections 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2668270 0x5615e2a61c30 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.599 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e2647590 0x5615e2a616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 --2- 172.21.15.129:0/637530306 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e263f3b0 0x5615e2a77b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 -- 172.21.15.129:0/637530306 >> 172.21.15.129:0/637530306 conn(0x5615e265c3c0 msgr2=0x5615e2a62720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 -- 172.21.15.129:0/637530306 shutdown_connections 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.598+0000 7f177f0b5740 1 -- 172.21.15.129:0/637530306 wait complete. 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.599+0000 7f177f0b5740 1 -- start start 2024-09-17T04:07:39.600 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.599+0000 7f177f0b5740 1 Processor -- start 2024-09-17T04:07:39.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2ad3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2668270 0x5615e2a50d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5615e2a1d220 con 0x5615e2647590 2024-09-17T04:07:39.605 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5615e2a20150 con 0x5615e263f3b0 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5615e2a63600 con 0x5615e2668270 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f177f0b5740 10 client.0 _ll_register_callbacks cb 0x7f1768009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.600+0000 7f1774c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46620/0 (socket says 172.21.15.129:46620) 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 -- 172.21.15.129:0/1065584638 learned_addr learned my addr 172.21.15.129:0/1065584638 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1775600640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2ad3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1776000640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2668270 0x5615e2a50d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 -- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2668270 msgr2=0x5615e2a50d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2668270 0x5615e2a50d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 -- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 msgr2=0x5615e2ad3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2ad3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 -- 172.21.15.129:0/1065584638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615e2a76e90 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1775600640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2ad3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1774c00640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f175c09a050 tx=0x7f175c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9605c51454361d server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1756000640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.601+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c06b760 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f175c05d740 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1768004290 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=249) ==== 24+0+0 (secure 0 0 0) 0x7f175c05d940 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 10 client.12369 fetch_fsmap learned FSMap version 249 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=249,monmap=2+}) -- 0x5615e2a512d0 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1768004290 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 249) ==== 3502+0+0 (secure 0 0 0) 0x7f175c069ba0 con 0x5615e2647590 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 10 client.12369 fetch_fsmap finished waiting for FSMap version 249 2024-09-17T04:07:39.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 10 client.12369 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 20 client.12369 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2a76900 0x5615e2ad5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 4 client.12369 mds_command: new command op to 12249 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 --> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] -- command(tid 0: {"prefix": "status"}) -- 0x5615e297ffd0 con 0x5615e2a76900 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.602+0000 7f1775600640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2a76900 0x5615e2ad5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f1756000640 10 client.12369 ms_handle_connect on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f1775600640 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2a76900 0x5615e2ad5e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2963b3bdfbccdcad server_cookie=9d925595ee1aec88 in_seq=0 out_seq=0 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f1756000640 1 -- 172.21.15.129:0/1065584638 <== mds.0 v2:172.21.15.129:6833/3848773989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x5615e297ffd0 con 0x5615e2a76900 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f1756000640 10 client.12369 handle_command_reply: tid=0 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 client.12369 shutdown 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 msgr2=0x5615e2a55db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f175c09a050 tx=0x7f175c000bc0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 shutdown_connections 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x5615e2a76900 0x5615e2ad5e00 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5615e2668270 0x5615e2a50d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5615e2647590 0x5615e2a55db0 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 --2- 172.21.15.129:0/1065584638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5615e263f3b0 0x5615e2ad3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 >> 172.21.15.129:0/1065584638 conn(0x5615e265c3c0 msgr2=0x5615e2a616f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.603+0000 7f1756000640 0 client.12369 ms_handle_reset on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 shutdown_connections 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f177f0b5740 1 -- 172.21.15.129:0/1065584638 wait complete. 2024-09-17T04:07:39.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f177f0b5740 20 client.12369 trim_cache size 0 max 16384 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 12249, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 6.6269492200000002, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 249, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 260, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 260, 2024-09-17T04:07:39.610 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 11.660109433000001 2024-09-17T04:07:39.611 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:39.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 msgr2=0x7f175007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 0x7f175007b0d0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f17700d4000 tx=0x7f176400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 msgr2=0x7f1770151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f175c00c3d0 tx=0x7f175c00c890 comp rx=0 tx=0).stop 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 shutdown_connections 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1750078bc0 0x7f175007b0d0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f17700d0050 0x7f177014e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f17700b9be0 0x7f1770151600 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 --2- 172.21.15.129:0/2297872231 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f17700b3430 0x7f17700ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 >> 172.21.15.129:0/2297872231 conn(0x7f177008bff0 msgr2=0x7f177006ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.604+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 shutdown_connections 2024-09-17T04:07:39.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.605+0000 7f1776a00640 1 -- 172.21.15.129:0/2297872231 wait complete. 2024-09-17T04:07:39.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:07:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.847+0000 7fe2f4c00640 1 -- 172.21.15.129:0/3418344029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f00ce720 msgr2=0x7fe2f0160b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.847+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/3418344029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f00ce720 0x7fe2f0160b90 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fe2f00ced90 tx=0x7fe2e802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 -- 172.21.15.129:0/3418344029 shutdown_connections 2024-09-17T04:07:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/3418344029 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f00ce720 0x7fe2f0160b90 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/3418344029 >> v2:172.21.15.181:3300/0 conn(0x7fe2f015c7f0 0x7fe2f006cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 -- 172.21.15.129:0/3418344029 >> 172.21.15.129:0/3418344029 conn(0x7fe2f00b2df0 msgr2=0x7fe2f00b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 -- 172.21.15.129:0/3418344029 shutdown_connections 2024-09-17T04:07:39.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.848+0000 7fe2f4c00640 1 -- 172.21.15.129:0/3418344029 wait complete. 2024-09-17T04:07:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.850+0000 7fe2f4c00640 1 Processor -- start 2024-09-17T04:07:39.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.850+0000 7fe2f4c00640 1 -- start start 2024-09-17T04:07:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 0x7fe2f00ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 0x7fe2f00b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0161250 con 0x7fe2f015c7f0 2024-09-17T04:07:39.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f01610d0 con 0x7fe2f006d320 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2f4c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00b2990 con 0x7fe2f00ce720 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46628/0 (socket says 172.21.15.129:46628) 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 -- 172.21.15.129:0/2135188145 learned_addr learned my addr 172.21.15.129:0/2135188145 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ede00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 0x7fe2f00ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2e5400640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 0x7fe2f00b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 -- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 msgr2=0x7fe2f00b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 0x7fe2f00b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 -- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 msgr2=0x7fe2f00ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 0x7fe2f00ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:39.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ee800640 1 -- 172.21.15.129:0/2135188145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f015b160 con 0x7fe2f015c7f0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2ede00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 0x7fe2f00ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.852+0000 7fe2e5400640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 0x7fe2f00b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2ee800640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8034040 tx=0x7fe2e8003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e803e030 con 0x7fe2f015c7f0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe2e8003e40 con 0x7fe2f015c7f0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e8035040 con 0x7fe2f015c7f0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f00aec20 con 0x7fe2f015c7f0 2024-09-17T04:07:39.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.853+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f00af1e0 con 0x7fe2f015c7f0 2024-09-17T04:07:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.854+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f00b3910 con 0x7fe2f015c7f0 2024-09-17T04:07:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.855+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe2e8041030 con 0x7fe2f015c7f0 2024-09-17T04:07:39.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.856+0000 7fe2e6a00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 0x7fe2cc07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.856+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fe2e80bb4c0 con 0x7fe2f015c7f0 2024-09-17T04:07:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.857+0000 7fe2ede00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 0x7fe2cc07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.857+0000 7fe2ede00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 0x7fe2cc07b0d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005fd0 tx=0x7fe2e00094b0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:39.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:39.861+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe2f006d730 con 0x7fe2f015c7f0 2024-09-17T04:07:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.082+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe2f00af8a0 con 0x7fe2f015c7f0 2024-09-17T04:07:40.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.082+0000 7fe2e6a00640 1 -- 172.21.15.129:0/2135188145 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v249) ==== 53+0+73 (secure 0 0 0) 0x7fe2e80838f0 con 0x7fe2f015c7f0 2024-09-17T04:07:40.082 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 msgr2=0x7fe2cc07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 0x7fe2cc07b0d0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0005fd0 tx=0x7fe2e00094b0 comp rx=0 tx=0).stop 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 msgr2=0x7fe2f015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8034040 tx=0x7fe2e8003910 comp rx=0 tx=0).stop 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 shutdown_connections 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe2cc078bc0 0x7fe2cc07b0d0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:40.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe2f015c7f0 0x7fe2f015cf90 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe2f00ce720 0x7fe2f00b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 --2- 172.21.15.129:0/2135188145 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe2f006d320 0x7fe2f00ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.084+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 >> 172.21.15.129:0/2135188145 conn(0x7fe2f00b2df0 msgr2=0x7fe2f00b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.085+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 shutdown_connections 2024-09-17T04:07:40.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40.085+0000 7fe2f4c00640 1 -- 172.21.15.129:0/2135188145 wait complete. 2024-09-17T04:07:40.109 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:07:40.110 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:40.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.111 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.147 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:07:40.148 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41632sec preferred_lft 41632sec 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:07:40.149 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:07:40.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.150 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:07:40.241 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:07:40.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.244 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:07:40.280 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:07:40.280 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:07:40.280 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:07:40.280 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:07:40.281 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:07:40.281 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:07:40.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:07:40.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:40.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.478 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:07:40.513 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:07:40.513 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:07:40.514 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:40.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:07:40.610 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:40.610 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:40.645 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:07:40.646 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:07:40.646 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:40.720 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:07:40.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:40.916 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:40.916 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:40.917 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:40.937 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:40.937 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:40.938 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:40.938 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:40.938 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:40.970 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:07:40.970 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:07:40.971 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:07:40.972 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:07:40.972 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:07:40.973 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:07:40.973 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:07:40.973 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:07:40.976 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:07:40.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:40.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:07:41.035 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:07:41.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:07:41.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:41.130 INFO:teuthology.orchestra.run.smithi129.stdout:client12378 2024-09-17T04:07:41.130 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:41.130 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:41.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 -- 172.21.15.129:0/178004062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 msgr2=0x7fed6406eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 --2- 172.21.15.129:0/178004062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6406eea0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7fed64163680 tx=0x7fed5c02a560 comp rx=0 tx=0).stop 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 -- 172.21.15.129:0/178004062 shutdown_connections 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 --2- 172.21.15.129:0/178004062 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6406eea0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 --2- 172.21.15.129:0/178004062 >> v2:172.21.15.181:3300/0 conn(0x7fed640b2350 0x7fed640705b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 -- 172.21.15.129:0/178004062 >> 172.21.15.129:0/178004062 conn(0x7fed640ccc40 msgr2=0x7fed640cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 -- 172.21.15.129:0/178004062 shutdown_connections 2024-09-17T04:07:41.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.430+0000 7fed68e00640 1 -- 172.21.15.129:0/178004062 wait complete. 2024-09-17T04:07:41.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.432+0000 7fed68e00640 1 Processor -- start 2024-09-17T04:07:41.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.432+0000 7fed68e00640 1 -- start start 2024-09-17T04:07:41.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:41.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fed640cfca0 0x7fed6409cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed641e95c0 0x7fed641e99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fed6406f7b0 con 0x7fed640cbf10 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fed6406f630 con 0x7fed641e95c0 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.433+0000 7fed68e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fed6406f930 con 0x7fed640cfca0 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46644/0 (socket says 172.21.15.129:46644) 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 -- 172.21.15.129:0/2283809030 learned_addr learned my addr 172.21.15.129:0/2283809030 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61400640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fed640cfca0 0x7fed6409cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed62800640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed641e95c0 0x7fed641e99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 -- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fed640cfca0 msgr2=0x7fed6409cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fed640cfca0 0x7fed6409cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:41.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 -- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed641e95c0 msgr2=0x7fed641e99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed641e95c0 0x7fed641e99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.434+0000 7fed61e00640 1 -- 172.21.15.129:0/2283809030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed641f5b20 con 0x7fed640cbf10 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed61e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7fed5400c7a0 tx=0x7fed5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54017070 con 0x7fed640cbf10 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fed54005070 con 0x7fed640cbf10 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed641f5e00 con 0x7fed640cbf10 2024-09-17T04:07:41.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed540054a0 con 0x7fed640cbf10 2024-09-17T04:07:41.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.435+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed640afa90 con 0x7fed640cbf10 2024-09-17T04:07:41.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.436+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed640705b0 con 0x7fed640cbf10 2024-09-17T04:07:41.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.437+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fed5401a070 con 0x7fed640cbf10 2024-09-17T04:07:41.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.438+0000 7fed5a000640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 0x7fed4407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:41.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.438+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fed5409a390 con 0x7fed640cbf10 2024-09-17T04:07:41.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.439+0000 7fed61400640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 0x7fed4407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:41.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.439+0000 7fed61400640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 0x7fed4407b0d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fed6409bd70 tx=0x7fed5000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:41.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.442+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fed540620e0 con 0x7fed640cbf10 2024-09-17T04:07:41.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:41.663+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) -- 0x7fed640cc9b0 con 0x7fed640cbf10 2024-09-17T04:07:42.028 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.028+0000 7fed5a000640 1 -- 172.21.15.129:0/2283809030 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v250) ==== 105+0+0 (secure 0 0 0) 0x7fed54066b10 con 0x7fed640cbf10 2024-09-17T04:07:42.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 msgr2=0x7fed4407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 0x7fed4407b0d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fed6409bd70 tx=0x7fed5000b2e0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 msgr2=0x7fed6409aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7fed5400c7a0 tx=0x7fed5400cc60 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 shutdown_connections 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fed44078bc0 0x7fed4407b0d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fed641e95c0 0x7fed641e99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fed640cfca0 0x7fed6409cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 --2- 172.21.15.129:0/2283809030 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fed640cbf10 0x7fed6409aa60 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 >> 172.21.15.129:0/2283809030 conn(0x7fed640ccc40 msgr2=0x7fed640b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 shutdown_connections 2024-09-17T04:07:42.032 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.031+0000 7fed68e00640 1 -- 172.21.15.129:0/2283809030 wait complete. 2024-09-17T04:07:42.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:42.281 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 -- 172.21.15.129:0/1267038116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 msgr2=0x7f61881609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 --2- 172.21.15.129:0/1267038116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61881609c0 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f6180005fd0 tx=0x7f618002aa90 comp rx=0 tx=0).stop 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 -- 172.21.15.129:0/1267038116 shutdown_connections 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 --2- 172.21.15.129:0/1267038116 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61881609c0 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 --2- 172.21.15.129:0/1267038116 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880b4140 0x7f61880ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 -- 172.21.15.129:0/1267038116 >> 172.21.15.129:0/1267038116 conn(0x7f618806d410 msgr2=0x7f61880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:42.282 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 -- 172.21.15.129:0/1267038116 shutdown_connections 2024-09-17T04:07:42.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.281+0000 7f618e800640 1 -- 172.21.15.129:0/1267038116 wait complete. 2024-09-17T04:07:42.283 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.284+0000 7f618e800640 1 Processor -- start 2024-09-17T04:07:42.284 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.284+0000 7f618e800640 1 -- start start 2024-09-17T04:07:42.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880705a0 0x7f6188158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:42.285 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61880b4140 0x7f61880c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6188160260 con 0x7f61880cd1c0 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f61881600e0 con 0x7f61880b4140 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f618e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6188160f00 con 0x7f61880705a0 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.285+0000 7f6187e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880705a0 0x7f6188158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f617ea00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61880b4140 0x7f61880c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46666/0 (socket says 172.21.15.129:46666) 2024-09-17T04:07:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 -- 172.21.15.129:0/2788503835 learned_addr learned my addr 172.21.15.129:0/2788503835 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:42.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 -- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880705a0 msgr2=0x7f6188158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880705a0 0x7f6188158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 -- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61880b4140 msgr2=0x7f61880c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.287 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61880b4140 0x7f61880c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 -- 172.21.15.129:0/2788503835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6188158960 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f6187e00640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f6180006010 tx=0x7f6180003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.286+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180044070 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.287+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6180004040 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.287+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6188158c40 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.287+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180004430 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.287+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6188159200 con 0x7f61880cd1c0 2024-09-17T04:07:42.288 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.288+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61880ccc80 con 0x7f61880cd1c0 2024-09-17T04:07:42.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.289+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6180047070 con 0x7f61880cd1c0 2024-09-17T04:07:42.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.290+0000 7f617fe00640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 0x7f616407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:42.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.290+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f61800bbb30 con 0x7f61880cd1c0 2024-09-17T04:07:42.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.290+0000 7f6187400640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 0x7f616407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:42.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.291+0000 7f6187400640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 0x7f616407b0a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6178001720 tx=0x7f6178009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:42.294 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.294+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6180032030 con 0x7f61880cd1c0 2024-09-17T04:07:42.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.510+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61881586f0 con 0x7f61880cd1c0 2024-09-17T04:07:42.511 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.511+0000 7f617fe00640 1 -- 172.21.15.129:0/2788503835 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) ==== 95+0+5264 (secure 0 0 0) 0x7f6180083910 con 0x7f61880cd1c0 2024-09-17T04:07:42.511 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:42.511 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":251,"btime":"2024-09-17T04:07:42:032925+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/47559844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":47559844},{"type":"v1","addr":"172.21.15.181:6836","nonce":47559844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":11266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3086573184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3086573184},{"type":"v1","addr":"172.21.15.181:6837","nonce":3086573184}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:42.032901+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12240":{"gid":12240,"name":"c","rank":1,"incarnation":251,"state":"up:creating","state_seq":1,"addr":"172.21.15.129:6834/1080794990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080794990},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080794990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:42.512 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 251 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 msgr2=0x7f616407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 0x7f616407b0a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f6178001720 tx=0x7f6178009040 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 msgr2=0x7f61880c6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f6180006010 tx=0x7f6180003ce0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 shutdown_connections 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6164078b90 0x7f616407b0a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f61880cd1c0 0x7f61880c6c60 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f61880b4140 0x7f61880c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 --2- 172.21.15.129:0/2788503835 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f61880705a0 0x7f6188158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 >> 172.21.15.129:0/2788503835 conn(0x7f618806d410 msgr2=0x7f61880ad650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:42.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.513+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 shutdown_connections 2024-09-17T04:07:42.515 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:42.514+0000 7f618e800640 1 -- 172.21.15.129:0/2788503835 wait complete. 2024-09-17T04:07:42.540 DEBUG:tasks.ceph_test_case:wait_until_equal: 1 != 2, waiting (timeout=30)... 2024-09-17T04:07:47.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:47.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.772+0000 7f49fb800640 1 -- 172.21.15.129:0/2442594967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 msgr2=0x7f49dc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:47.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.772+0000 7f49fb800640 1 --2- 172.21.15.129:0/2442594967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49dc00b9c0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7f49ec001ea0 tx=0x7f49dc005830 comp rx=0 tx=0).stop 2024-09-17T04:07:47.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.772+0000 7f49f9800640 1 -- 172.21.15.129:0/2442594967 reap_dead start 2024-09-17T04:07:47.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.773+0000 7f49fb800640 1 -- 172.21.15.129:0/2442594967 shutdown_connections 2024-09-17T04:07:47.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.773+0000 7f49fb800640 1 -- 172.21.15.129:0/2442594967 >> 172.21.15.129:0/2442594967 conn(0x7f49f406d410 msgr2=0x7f49f40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:47.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.773+0000 7f49fb800640 1 -- 172.21.15.129:0/2442594967 shutdown_connections 2024-09-17T04:07:47.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.773+0000 7f49fb800640 1 -- 172.21.15.129:0/2442594967 wait complete. 2024-09-17T04:07:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.775+0000 7f49fb800640 1 Processor -- start 2024-09-17T04:07:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.775+0000 7f49fb800640 1 -- start start 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.776+0000 7f49fb800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.776+0000 7f49fb800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f40b1c80 0x7f49f40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49fb800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49f40caa70 0x7f49f40cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49fb800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f49f4160260 con 0x7f49dc005210 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49fb800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f49f41600e0 con 0x7f49f40b1c80 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49fb800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f49f41610d0 con 0x7f49f40caa70 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59912/0 (socket says 172.21.15.129:59912) 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 -- 172.21.15.129:0/973941909 learned_addr learned my addr 172.21.15.129:0/973941909 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f9800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49f40caa70 0x7f49f40cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f3e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f40b1c80 0x7f49f40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:47.778 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 -- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49f40caa70 msgr2=0x7f49f40cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49f40caa70 0x7f49f40cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 -- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f40b1c80 msgr2=0x7f49f40b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f40b1c80 0x7f49f40b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 -- 172.21.15.129:0/973941909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49f40c53d0 con 0x7f49dc005210 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.777+0000 7f49f8e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f49e8003640 tx=0x7f49e8003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.778+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8015070 con 0x7f49dc005210 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.778+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49e800f990 con 0x7f49dc005210 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.778+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e800fde0 con 0x7f49dc005210 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.778+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49f41557c0 con 0x7f49dc005210 2024-09-17T04:07:47.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.778+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49f40c88a0 con 0x7f49dc005210 2024-09-17T04:07:47.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.779+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49f40c55a0 con 0x7f49dc005210 2024-09-17T04:07:47.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.780+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f49e8011430 con 0x7f49dc005210 2024-09-17T04:07:47.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.781+0000 7f49f0c00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 0x7f49d407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:47.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.781+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f49e809bcb0 con 0x7f49dc005210 2024-09-17T04:07:47.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.782+0000 7f49f3e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 0x7f49d407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:47.783 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.783+0000 7f49f3e00640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 0x7f49d407b0a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f49f40ad3e0 tx=0x7f49e0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:47.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:47.786+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f49f40cd5d0 con 0x7f49dc005210 2024-09-17T04:07:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.006+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49f40ccc80 con 0x7f49dc005210 2024-09-17T04:07:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.007+0000 7f49f0c00640 1 -- 172.21.15.129:0/973941909 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) ==== 95+0+5268 (secure 0 0 0) 0x7f49e8063a00 con 0x7f49dc005210 2024-09-17T04:07:48.007 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:48.007 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":252,"btime":"2024-09-17T04:07:43:037154+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/47559844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":47559844},{"type":"v1","addr":"172.21.15.181:6836","nonce":47559844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":11266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3086573184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3086573184},{"type":"v1","addr":"172.21.15.181:6837","nonce":3086573184}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:43.037148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12240":{"gid":12240,"name":"c","rank":1,"incarnation":251,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1080794990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080794990},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080794990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12240,12249]},"id":19}]} 2024-09-17T04:07:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 252 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 msgr2=0x7f49d407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 0x7f49d407b0a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f49f40ad3e0 tx=0x7f49e0005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 msgr2=0x7f49f40c9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7f49e8003640 tx=0x7f49e8003b00 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 shutdown_connections 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f49d4078b90 0x7f49d407b0a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f49f40caa70 0x7f49f40cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f40b1c80 0x7f49f40b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 --2- 172.21.15.129:0/973941909 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f49dc005210 0x7f49f40c9a70 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.009+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 >> 172.21.15.129:0/973941909 conn(0x7f49f406d410 msgr2=0x7f49f40cf390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.010+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 shutdown_connections 2024-09-17T04:07:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.010+0000 7f49fb800640 1 -- 172.21.15.129:0/973941909 wait complete. 2024-09-17T04:07:48.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 -- 172.21.15.129:0/1096902898 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0d09d0 msgr2=0x7f31bc0b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 --2- 172.21.15.129:0/1096902898 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0d09d0 0x7f31bc0b4af0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f31b4005fd0 tx=0x7f31b402a7a0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 -- 172.21.15.129:0/1096902898 shutdown_connections 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 --2- 172.21.15.129:0/1096902898 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0d09d0 0x7f31bc0b4af0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 --2- 172.21.15.129:0/1096902898 >> v2:172.21.15.181:3300/0 conn(0x7f31bc0cfb60 0x7f31bc160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 -- 172.21.15.129:0/1096902898 >> 172.21.15.129:0/1096902898 conn(0x7f31bc06cfc0 msgr2=0x7f31bc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 -- 172.21.15.129:0/1096902898 shutdown_connections 2024-09-17T04:07:48.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.255+0000 7f31c0c00640 1 -- 172.21.15.129:0/1096902898 wait complete. 2024-09-17T04:07:48.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.258+0000 7f31c0c00640 1 Processor -- start 2024-09-17T04:07:48.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.258+0000 7f31c0c00640 1 -- start start 2024-09-17T04:07:48.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0cfb60 0x7f31bc0c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31bc1f87a0 0x7f31bc1f8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc15d260 con 0x7f31bc0b46e0 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc15d0e0 con 0x7f31bc1f87a0 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31c0c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f31bc15d3e0 con 0x7f31bc0cfb60 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31bb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31bb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59926/0 (socket says 172.21.15.129:59926) 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.259+0000 7f31bb400640 1 -- 172.21.15.129:0/3900738325 learned_addr learned my addr 172.21.15.129:0/3900738325 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bbe00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31bc1f87a0 0x7f31bc1f8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31baa00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0cfb60 0x7f31bc0c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 -- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0cfb60 msgr2=0x7f31bc0c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0cfb60 0x7f31bc0c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 -- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31bc1f87a0 msgr2=0x7f31bc1f8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31bc1f87a0 0x7f31bc1f8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 -- 172.21.15.129:0/3900738325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc0c7b40 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31bb400640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009fd0 tx=0x7f31ac00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00ccc0 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.260+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31ac00ee70 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.261+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00ce60 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.261+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31bc1399f0 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.261+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31bc139fb0 con 0x7f31bc0b46e0 2024-09-17T04:07:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.262+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31bc15d890 con 0x7f31bc0b46e0 2024-09-17T04:07:48.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.263+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f31ac004280 con 0x7f31bc0b46e0 2024-09-17T04:07:48.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.264+0000 7f31b3400640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 0x7f319c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.265+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7f31ac09b610 con 0x7f31bc0b46e0 2024-09-17T04:07:48.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.265+0000 7f31baa00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 0x7f319c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.266+0000 7f31baa00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 0x7f319c07b0a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f31a8001eb0 tx=0x7f31a8009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:48.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.269+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f31ac063a40 con 0x7f31bc0b46e0 2024-09-17T04:07:48.485 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.485+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31bc0c7d70 con 0x7f31bc0b46e0 2024-09-17T04:07:48.486 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.486+0000 7f31b3400640 1 -- 172.21.15.129:0/3900738325 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) ==== 95+0+5268 (secure 0 0 0) 0x7f31ac063340 con 0x7f31bc0b46e0 2024-09-17T04:07:48.486 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:48.486 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":252,"btime":"2024-09-17T04:07:43:037154+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/47559844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":47559844},{"type":"v1","addr":"172.21.15.181:6836","nonce":47559844}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244},{"gid":11266,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3086573184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3086573184},{"type":"v1","addr":"172.21.15.181:6837","nonce":3086573184}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":244}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:43.037148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12240},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12240":{"gid":12240,"name":"c","rank":1,"incarnation":251,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/1080794990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1080794990},{"type":"v1","addr":"172.21.15.129:6834","nonce":1080794990}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12240,12249]},"id":19}]} 2024-09-17T04:07:48.486 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 252 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 msgr2=0x7f319c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 0x7f319c07b0a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f31a8001eb0 tx=0x7f31a8009040 comp rx=0 tx=0).stop 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 msgr2=0x7f31bc0ca450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f31ac009fd0 tx=0x7f31ac00ec20 comp rx=0 tx=0).stop 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 shutdown_connections 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f319c078b90 0x7f319c07b0a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f31bc1f87a0 0x7f31bc1f8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f31bc0cfb60 0x7f31bc0c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 --2- 172.21.15.129:0/3900738325 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f31bc0b46e0 0x7f31bc0ca450 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 >> 172.21.15.129:0/3900738325 conn(0x7f31bc06cfc0 msgr2=0x7f31bc0b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 shutdown_connections 2024-09-17T04:07:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.488+0000 7f31c0c00640 1 -- 172.21.15.129:0/3900738325 wait complete. 2024-09-17T04:07:48.511 INFO:tasks.cephfs.test_strays:Ranks 0 and 1 are a and c 2024-09-17T04:07:48.511 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:07:48.511 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:48.537 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:07:48.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-17T04:07:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.759+0000 7fce95800640 1 -- 172.21.15.129:0/1632549514 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900d09d0 msgr2=0x7fce900b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.759+0000 7fce95800640 1 --2- 172.21.15.129:0/1632549514 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900d09d0 0x7fce900b4af0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fce88005fd0 tx=0x7fce8802aa10 comp rx=0 tx=0).stop 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 -- 172.21.15.129:0/1632549514 shutdown_connections 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 --2- 172.21.15.129:0/1632549514 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900d09d0 0x7fce900b4af0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 --2- 172.21.15.129:0/1632549514 >> v2:172.21.15.181:3300/0 conn(0x7fce900cfb60 0x7fce90160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 -- 172.21.15.129:0/1632549514 >> 172.21.15.129:0/1632549514 conn(0x7fce9006cfc0 msgr2=0x7fce9006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 -- 172.21.15.129:0/1632549514 shutdown_connections 2024-09-17T04:07:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.760+0000 7fce95800640 1 -- 172.21.15.129:0/1632549514 wait complete. 2024-09-17T04:07:48.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.762+0000 7fce95800640 1 Processor -- start 2024-09-17T04:07:48.762 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.762+0000 7fce95800640 1 -- start start 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900b46e0 0x7fce900c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce90072d30 0x7fce9015a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fce9015d270 con 0x7fce900cfb60 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fce9015d0f0 con 0x7fce90072d30 2024-09-17T04:07:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce95800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fce9015d3f0 con 0x7fce900b46e0 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59938/0 (socket says 172.21.15.129:59938) 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 -- 172.21.15.129:0/30288686 learned_addr learned my addr 172.21.15.129:0/30288686 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8fe00640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900b46e0 0x7fce900c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce94e00640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce90072d30 0x7fce9015a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 -- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900b46e0 msgr2=0x7fce900c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900b46e0 0x7fce900c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 -- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce90072d30 msgr2=0x7fce9015a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce90072d30 0x7fce9015a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.764+0000 7fce8f400640 1 -- 172.21.15.129:0/30288686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce900ca470 con 0x7fce900cfb60 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce8f400640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c3a0 tx=0x7fce7c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c017070 con 0x7fce900cfb60 2024-09-17T04:07:48.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fce7c0053c0 con 0x7fce900cfb60 2024-09-17T04:07:48.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce90073910 con 0x7fce900cfb60 2024-09-17T04:07:48.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c003850 con 0x7fce900cfb60 2024-09-17T04:07:48.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.765+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce900c9730 con 0x7fce900cfb60 2024-09-17T04:07:48.767 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.766+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce9015d8a0 con 0x7fce900cfb60 2024-09-17T04:07:48.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.767+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fce7c01a070 con 0x7fce900cfb60 2024-09-17T04:07:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.768+0000 7fce87e00640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 0x7fce7007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.769+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(260..260 src has 1..260) ==== 5709+0+0 (secure 0 0 0) 0x7fce7c09a170 con 0x7fce900cfb60 2024-09-17T04:07:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.769+0000 7fce8fe00640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 0x7fce7007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.770+0000 7fce8fe00640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 0x7fce7007b0a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fce900b4450 tx=0x7fce80005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:48.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:48.773+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fce7c061e40 con 0x7fce900cfb60 2024-09-17T04:07:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.031+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) -- 0x7fce9006a140 con 0x7fce900cfb60 2024-09-17T04:07:49.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.053+0000 7fce87e00640 1 -- 172.21.15.129:0/30288686 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 11266 v253) ==== 96+0+0 (secure 0 0 0) 0x7fce7c066870 con 0x7fce900cfb60 2024-09-17T04:07:49.053 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 11266 2024-09-17T04:07:49.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 msgr2=0x7fce7007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:49.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 0x7fce7007b0a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fce900b4450 tx=0x7fce80005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 msgr2=0x7fce9014b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00c3a0 tx=0x7fce7c00c860 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 shutdown_connections 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fce70078b90 0x7fce7007b0a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce90072d30 0x7fce9015a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fce900cfb60 0x7fce9014b9a0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 --2- 172.21.15.129:0/30288686 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fce900b46e0 0x7fce900c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 >> 172.21.15.129:0/30288686 conn(0x7fce9006cfc0 msgr2=0x7fce90153db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 shutdown_connections 2024-09-17T04:07:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.057+0000 7fce95800640 1 -- 172.21.15.129:0/30288686 wait complete. 2024-09-17T04:07:49.083 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:07:49.083 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:49.108 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:07:49.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-17T04:07:49.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 -- 172.21.15.129:0/2360311586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 msgr2=0x7fd740160cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:49.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 --2- 172.21.15.129:0/2360311586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd740160cf0 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7fd7401609e0 tx=0x7fd73002aa10 comp rx=0 tx=0).stop 2024-09-17T04:07:49.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 -- 172.21.15.129:0/2360311586 shutdown_connections 2024-09-17T04:07:49.329 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 --2- 172.21.15.129:0/2360311586 >> v2:172.21.15.181:3300/0 conn(0x7fd7400b46e0 0x7fd7400b4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 --2- 172.21.15.129:0/2360311586 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd740160cf0 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 -- 172.21.15.129:0/2360311586 >> 172.21.15.129:0/2360311586 conn(0x7fd74006cfc0 msgr2=0x7fd74006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 -- 172.21.15.129:0/2360311586 shutdown_connections 2024-09-17T04:07:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.329+0000 7fd744e00640 1 -- 172.21.15.129:0/2360311586 wait complete. 2024-09-17T04:07:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.331+0000 7fd744e00640 1 Processor -- start 2024-09-17T04:07:49.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.331+0000 7fd744e00640 1 -- start start 2024-09-17T04:07:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 0x7fd7400c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7400d02c0 0x7fd7400735d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd74015d1d0 con 0x7fd7400cfb60 2024-09-17T04:07:49.334 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd74015d050 con 0x7fd7400d02c0 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd744e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd74015d350 con 0x7fd7400b46e0 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59954/0 (socket says 172.21.15.129:59954) 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 -- 172.21.15.129:0/2428723740 learned_addr learned my addr 172.21.15.129:0/2428723740 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73fe00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7400d02c0 0x7fd7400735d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73f400640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 0x7fd7400c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 -- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 msgr2=0x7fd7400c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 0x7fd7400c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 -- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7400d02c0 msgr2=0x7fd7400735d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7400d02c0 0x7fd7400735d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73ea00640 1 -- 172.21.15.129:0/2428723740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7400c98f0 con 0x7fd7400cfb60 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.333+0000 7fd73f400640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 0x7fd7400c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:49.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd73ea00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d740 tx=0x7fd72c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c017070 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd72c009d20 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c0106c0 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7401f6a40 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.334+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7401f6760 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.335+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd74015d800 con 0x7fd7400cfb60 2024-09-17T04:07:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.336+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd72c01a070 con 0x7fd7400cfb60 2024-09-17T04:07:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.337+0000 7fd737400640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 0x7fd72007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.338+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(261..261 src has 1..261) ==== 5709+0+0 (secure 0 0 0) 0x7fd72c09ad50 con 0x7fd7400cfb60 2024-09-17T04:07:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.338+0000 7fd73f400640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 0x7fd72007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:49.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.339+0000 7fd73f400640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 0x7fd72007b0d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd730035000 tx=0x7fd730004ab0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:49.342 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.342+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd72c09e030 con 0x7fd7400cfb60 2024-09-17T04:07:49.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:49.607+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) -- 0x7fd740154e10 con 0x7fd7400cfb60 2024-09-17T04:07:49.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:07:49.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:07:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.072+0000 7fd737400640 1 -- 172.21.15.129:0/2428723740 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 9494 v254) ==== 95+0+0 (secure 0 0 0) 0x7fd72c062bd0 con 0x7fd7400cfb60 2024-09-17T04:07:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 9494 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 msgr2=0x7fd72007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 0x7fd72007b0d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd730035000 tx=0x7fd730004ab0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 msgr2=0x7fd7401f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00d740 tx=0x7fd72c00dc00 comp rx=0 tx=0).stop 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 shutdown_connections 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd720078bc0 0x7fd72007b0d0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7400d02c0 0x7fd7400735d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7400cfb60 0x7fd7401f5a90 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 --2- 172.21.15.129:0/2428723740 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7400b46e0 0x7fd7400c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.074+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 >> 172.21.15.129:0/2428723740 conn(0x7fd74006cfc0 msgr2=0x7fd740155330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.075+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 shutdown_connections 2024-09-17T04:07:50.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.075+0000 7fd744e00640 1 -- 172.21.15.129:0/2428723740 wait complete. 2024-09-17T04:07:50.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:07:50.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:07:50.196 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:07:50.196 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:07:50.196 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:07:50.196 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.197 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:07:50.198 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:07:50.199 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:07:50.200 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.201 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:07:50.202 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:07:50.203 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:07:50.204 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:07:50.205 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:07:50.206 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout: mds_max_purge_files = 0 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:07:50.207 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:07:50.208 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:07:50.209 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:07:50.210 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:07:50.211 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:07:50.212 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:07:50.213 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:07:50.214 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:07:50.215 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.216 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:07:50.217 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout: mds_max_purge_files = 0 2024-09-17T04:07:50.218 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:07:50.219 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:07:50.219 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:07:50.227 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:07:50.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:07:50.251 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:07:50.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 -- 172.21.15.129:0/1221530218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 msgr2=0x7fb6940d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 --2- 172.21.15.129:0/1221530218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb6940d06c0 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7fb694072f50 tx=0x7fb68002a580 comp rx=0 tx=0).stop 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 -- 172.21.15.129:0/1221530218 shutdown_connections 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 --2- 172.21.15.129:0/1221530218 >> v2:172.21.15.181:3300/0 conn(0x7fb6940d0c00 0x7fb69415cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 --2- 172.21.15.129:0/1221530218 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb6940d06c0 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 -- 172.21.15.129:0/1221530218 >> 172.21.15.129:0/1221530218 conn(0x7fb69406cfc0 msgr2=0x7fb69406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:50.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 -- 172.21.15.129:0/1221530218 shutdown_connections 2024-09-17T04:07:50.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.515+0000 7fb69ba00640 1 -- 172.21.15.129:0/1221530218 wait complete. 2024-09-17T04:07:50.517 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.518+0000 7fb69ba00640 1 Processor -- start 2024-09-17T04:07:50.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.518+0000 7fb69ba00640 1 -- start start 2024-09-17T04:07:50.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:50.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6940d0c00 0x7fb6941408b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb69415a460 0x7fb6940b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb69415d640 con 0x7fb69406fa30 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb69415d4c0 con 0x7fb6940d0c00 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb69ba00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb69415d7c0 con 0x7fb69415a460 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb699000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb699000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59972/0 (socket says 172.21.15.129:59972) 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb699000640 1 -- 172.21.15.129:0/2549608239 learned_addr learned my addr 172.21.15.129:0/2549608239 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.519+0000 7fb693e00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6940d0c00 0x7fb6941408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699a00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb69415a460 0x7fb6940b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 -- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb69415a460 msgr2=0x7fb6940b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb69415a460 0x7fb6940b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 -- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6940d0c00 msgr2=0x7fb6941408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6940d0c00 0x7fb6941408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 -- 172.21.15.129:0/2549608239 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6940affc0 con 0x7fb69406fa30 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb699000640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c3a0 tx=0x7fb68800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688017070 con 0x7fb69406fa30 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6880053c0 con 0x7fb69406fa30 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688003850 con 0x7fb69406fa30 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.520+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6940b0260 con 0x7fb69406fa30 2024-09-17T04:07:50.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.521+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6940b0820 con 0x7fb69406fa30 2024-09-17T04:07:50.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.522+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69415dc70 con 0x7fb69406fa30 2024-09-17T04:07:50.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.523+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb68801a070 con 0x7fb69406fa30 2024-09-17T04:07:50.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.524+0000 7fb690c00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 0x7fb67407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:50.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.524+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(262..262 src has 1..262) ==== 5709+0+0 (secure 0 0 0) 0x7fb688099fe0 con 0x7fb69406fa30 2024-09-17T04:07:50.524 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.525+0000 7fb693e00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 0x7fb67407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:50.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.525+0000 7fb693e00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 0x7fb67407b0a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb680033000 tx=0x7fb680004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:50.529 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.529+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb688061e20 con 0x7fb69406fa30 2024-09-17T04:07:50.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.780+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) -- 0x7fb6940d0890 con 0x7fb69406fa30 2024-09-17T04:07:50.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.783+0000 7fb690c00640 1 -- 172.21.15.129:0/2549608239 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 12240 v255) ==== 96+0+0 (secure 0 0 0) 0x7fb688066850 con 0x7fb69406fa30 2024-09-17T04:07:50.783 INFO:teuthology.orchestra.run.smithi129.stderr:failed mds gid 12240 2024-09-17T04:07:50.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 msgr2=0x7fb67407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 0x7fb67407b0a0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7fb680033000 tx=0x7fb680004730 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 msgr2=0x7fb694141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c3a0 tx=0x7fb68800c860 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 shutdown_connections 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb674078b90 0x7fb67407b0a0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb69415a460 0x7fb6940b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6940d0c00 0x7fb6941408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 --2- 172.21.15.129:0/2549608239 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb69406fa30 0x7fb694141e90 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 >> 172.21.15.129:0/2549608239 conn(0x7fb69406cfc0 msgr2=0x7fb6940b4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 shutdown_connections 2024-09-17T04:07:50.787 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:50.786+0000 7fb69ba00640 1 -- 172.21.15.129:0/2549608239 wait complete. 2024-09-17T04:07:50.813 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:07:50.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:07:50.817 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:07:50.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:51.013 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:07:51.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 -- 172.21.15.129:0/1996662596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84151590 msgr2=0x7f2a841519a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 --2- 172.21.15.129:0/1996662596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84151590 0x7f2a841519a0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f2a8415b2c0 tx=0x7f2a7c02a890 comp rx=0 tx=0).stop 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 -- 172.21.15.129:0/1996662596 shutdown_connections 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 --2- 172.21.15.129:0/1996662596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84151590 0x7f2a841519a0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 -- 172.21.15.129:0/1996662596 >> 172.21.15.129:0/1996662596 conn(0x7f2a8406cfc0 msgr2=0x7f2a8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 -- 172.21.15.129:0/1996662596 shutdown_connections 2024-09-17T04:07:51.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.045+0000 7f2a88e00640 1 -- 172.21.15.129:0/1996662596 wait complete. 2024-09-17T04:07:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.047+0000 7f2a88e00640 1 Processor -- start 2024-09-17T04:07:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.048+0000 7f2a88e00640 1 -- start start 2024-09-17T04:07:51.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84071240 0x7f2a841e6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:51.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:51.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a8415e590 0x7f2a840b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:51.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8415bad0 con 0x7f2a84151590 2024-09-17T04:07:51.054 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8415b950 con 0x7f2a84071240 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a88e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8415bc50 con 0x7f2a8415e590 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a79400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a79400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:59990/0 (socket says 172.21.15.129:59990) 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.049+0000 7f2a79400640 1 -- 172.21.15.129:0/3455941664 learned_addr learned my addr 172.21.15.129:0/3455941664 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a81e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84071240 0x7f2a841e6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a82800640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a8415e590 0x7f2a840b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 -- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a8415e590 msgr2=0x7f2a840b2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a8415e590 0x7f2a840b2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 -- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84071240 msgr2=0x7f2a841e6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84071240 0x7f2a841e6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 -- 172.21.15.129:0/3455941664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a8414e1b0 con 0x7f2a84151590 2024-09-17T04:07:51.055 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a79400640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c430 tx=0x7f2a6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c017070 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a6c0053e0 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c003850 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a840b12c0 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.050+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a84157be0 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.051+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a841e9270 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.053+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2a6c01a070 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.054+0000 7f2a7aa00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 0x7f2a6007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.054+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f2a6c09a130 con 0x7f2a84151590 2024-09-17T04:07:51.056 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.054+0000 7f2a81e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 0x7f2a6007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:51.057 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.055+0000 7f2a81e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 0x7f2a6007b0a0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2a740015b0 tx=0x7f2a74009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:51.058 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.059+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2a6c062590 con 0x7f2a84151590 2024-09-17T04:07:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.276+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a841519a0 con 0x7f2a84151590 2024-09-17T04:07:51.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.276+0000 7f2a7aa00640 1 -- 172.21.15.129:0/3455941664 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 255 v255) ==== 95+0+2769 (secure 0 0 0) 0x7f2a6c061e90 con 0x7f2a84151590 2024-09-17T04:07:51.277 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:51.277 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":255,"btime":"2024-09-17T04:07:50:782041+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":255,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:50.782036+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249},"failed":[1],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:51.277 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 255 2024-09-17T04:07:51.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 msgr2=0x7f2a6007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:51.279 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 0x7f2a6007b0a0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f2a740015b0 tx=0x7f2a74009040 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 msgr2=0x7f2a8414d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c00c430 tx=0x7f2a6c00c8f0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 shutdown_connections 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2a60078b90 0x7f2a6007b0a0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2a8415e590 0x7f2a840b2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2a84151590 0x7f2a8414d6f0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 --2- 172.21.15.129:0/3455941664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a84071240 0x7f2a841e6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 >> 172.21.15.129:0/3455941664 conn(0x7f2a8406cfc0 msgr2=0x7f2a84068d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 shutdown_connections 2024-09-17T04:07:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:51.279+0000 7f2a88e00640 1 -- 172.21.15.129:0/3455941664 wait complete. 2024-09-17T04:07:51.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 255, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:07:32.455412+0000', 'modified': '2024-09-17T04:07:50.782036+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 263, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 12249}, 'failed': [1], 'damaged': [], 'stopped': [], 'info': {'gid_12249': {'gid': 12249, 'name': 'a', 'rank': 0, 'incarnation': 245, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3848773989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3848773989}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3848773989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12249, 'qdb_cluster': [12249]} 2024-09-17T04:07:51.502 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/2 2024-09-17T04:07:52.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 -- 172.21.15.129:0/2869537245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82140b3780 msgr2=0x7f821415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 --2- 172.21.15.129:0/2869537245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82140b3780 0x7f821415b5d0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7f821415b2c0 tx=0x7f820c02a800 comp rx=0 tx=0).stop 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 -- 172.21.15.129:0/2869537245 shutdown_connections 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 --2- 172.21.15.129:0/2869537245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f82140b3780 0x7f821415b5d0 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 --2- 172.21.15.129:0/2869537245 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f821415e590 0x7f8214070a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.732+0000 7f821a200640 1 -- 172.21.15.129:0/2869537245 >> 172.21.15.129:0/2869537245 conn(0x7f821406cfc0 msgr2=0x7f821406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:52.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.733+0000 7f821a200640 1 -- 172.21.15.129:0/2869537245 shutdown_connections 2024-09-17T04:07:52.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.733+0000 7f821a200640 1 -- 172.21.15.129:0/2869537245 wait complete. 2024-09-17T04:07:52.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.735+0000 7f821a200640 1 Processor -- start 2024-09-17T04:07:52.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.735+0000 7f821a200640 1 -- start start 2024-09-17T04:07:52.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 0x7f82140b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82140b3780 0x7f821414a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f82140713c0 con 0x7f821415e590 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8214071240 con 0x7f82140b3370 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f821a200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f821415bb10 con 0x7f82140b3780 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f8213e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f8213e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:60000/0 (socket says 172.21.15.129:60000) 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.736+0000 7f8213e00640 1 -- 172.21.15.129:0/2153539076 learned_addr learned my addr 172.21.15.129:0/2153539076 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8212a00640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82140b3780 0x7f821414a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213400640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 0x7f82140b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 -- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82140b3780 msgr2=0x7f821414a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:52.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82140b3780 0x7f821414a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 -- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 msgr2=0x7f82140b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 0x7f82140b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 -- 172.21.15.129:0/2153539076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821414e9e0 con 0x7f821415e590 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213400640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 0x7f82140b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f8213e00640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f820c02adf0 tx=0x7f820c033f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c042020 con 0x7f821415e590 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f820c004040 con 0x7f821415e590 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8214139400 con 0x7f821415e590 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.737+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c037040 con 0x7f821415e590 2024-09-17T04:07:52.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.738+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f821414dca0 con 0x7f821415e590 2024-09-17T04:07:52.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.739+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f821415b5d0 con 0x7f821415e590 2024-09-17T04:07:52.740 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.740+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f820c045020 con 0x7f821415e590 2024-09-17T04:07:52.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.741+0000 7f820b400640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 0x7f81f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:52.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.741+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f820c0bba80 con 0x7f821415e590 2024-09-17T04:07:52.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.741+0000 7f8213400640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 0x7f81f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:52.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.742+0000 7f8213400640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 0x7f81f407b0a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8204009b80 tx=0x7f8204005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:52.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.745+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f820c0bde20 con 0x7f821415e590 2024-09-17T04:07:52.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.968+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f821414b9e0 con 0x7f821415e590 2024-09-17T04:07:52.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.969+0000 7f820b400640 1 -- 172.21.15.129:0/2153539076 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) ==== 95+0+3613 (secure 0 0 0) 0x7f820c04c070 con 0x7f821415e590 2024-09-17T04:07:52.969 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:52.969 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":258,"btime":"2024-09-17T04:07:52:792516+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:51.810937+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:resolve","state_seq":2,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:52.970 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 258 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 msgr2=0x7f81f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 0x7f81f407b0a0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f8204009b80 tx=0x7f8204005f90 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 msgr2=0x7f821414e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f820c02adf0 tx=0x7f820c033f00 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 shutdown_connections 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f81f4078b90 0x7f81f407b0a0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f821415e590 0x7f821414e4a0 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f82140b3780 0x7f821414a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 --2- 172.21.15.129:0/2153539076 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82140b3370 0x7f82140b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:52.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 >> 172.21.15.129:0/2153539076 conn(0x7f821406cfc0 msgr2=0x7f821415c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:52.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 shutdown_connections 2024-09-17T04:07:52.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:52.971+0000 7f821a200640 1 -- 172.21.15.129:0/2153539076 wait complete. 2024-09-17T04:07:53.198 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 258, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:07:32.455412+0000', 'modified': '2024-09-17T04:07:51.810937+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 263, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 12249, 'mds_1': 12432}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12249': {'gid': 12249, 'name': 'a', 'rank': 0, 'incarnation': 245, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3848773989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3848773989}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3848773989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_12432': {'gid': 12432, 'name': 'c', 'rank': 1, 'incarnation': 257, 'state': 'up:resolve', 'state_seq': 2, 'addr': '172.21.15.129:6834/3076551424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 3076551424}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 3076551424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12249, 'qdb_cluster': [12249]} 2024-09-17T04:07:53.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12432:up:resolve 2024-09-17T04:07:54.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:54.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 -- 172.21.15.129:0/2495500187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7140d0620 msgr2=0x7fb7140d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:54.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 --2- 172.21.15.129:0/2495500187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7140d0620 0x7fb7140d0a30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb70c005fd0 tx=0x7fb70c02a8b0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 -- 172.21.15.129:0/2495500187 shutdown_connections 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 --2- 172.21.15.129:0/2495500187 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb7140d0620 0x7fb7140d0a30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 --2- 172.21.15.129:0/2495500187 >> v2:172.21.15.181:3300/0 conn(0x7fb7140cfd10 0x7fb7140cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 -- 172.21.15.129:0/2495500187 >> 172.21.15.129:0/2495500187 conn(0x7fb71406cfc0 msgr2=0x7fb71406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 -- 172.21.15.129:0/2495500187 shutdown_connections 2024-09-17T04:07:54.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.432+0000 7fb719600640 1 -- 172.21.15.129:0/2495500187 wait complete. 2024-09-17T04:07:54.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.434+0000 7fb719600640 1 Processor -- start 2024-09-17T04:07:54.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.434+0000 7fb719600640 1 -- start start 2024-09-17T04:07:54.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7140cfd10 0x7fb7141548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:54.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 0x7fb714148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb714073890 con 0x7fb7140d0620 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb714073710 con 0x7fb7140cfd10 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb719600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb714073a10 con 0x7fb71415cd30 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53250/0 (socket says 172.21.15.129:53250) 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 -- 172.21.15.129:0/2287641087 learned_addr learned my addr 172.21.15.129:0/2287641087 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb713400640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 0x7fb714148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:54.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb712a00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7140cfd10 0x7fb7141548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 -- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 msgr2=0x7fb714148410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 0x7fb714148410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 -- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7140cfd10 msgr2=0x7fb7141548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7140cfd10 0x7fb7141548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb70be00640 1 -- 172.21.15.129:0/2287641087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb714148fc0 con 0x7fb7140d0620 2024-09-17T04:07:54.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.436+0000 7fb713400640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 0x7fb714148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb70be00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e9f0 tx=0x7fb6fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00cd80 con 0x7fb7140d0620 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb6fc010520 con 0x7fb7140d0620 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7141492a0 con 0x7fb7140d0620 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc010950 con 0x7fb7140d0620 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.437+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7141540a0 con 0x7fb7140d0620 2024-09-17T04:07:54.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.438+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb714073f30 con 0x7fb7140d0620 2024-09-17T04:07:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.440+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb6fc010af0 con 0x7fb7140d0620 2024-09-17T04:07:54.440 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.441+0000 7fb70aa00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 0x7fb6f007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.441+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fb6fc09b830 con 0x7fb7140d0620 2024-09-17T04:07:54.441 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.441+0000 7fb712a00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 0x7fb6f007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:54.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.442+0000 7fb712a00640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 0x7fb6f007b0a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb704001610 tx=0x7fb704009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:54.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.445+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb6fc063c10 con 0x7fb7140d0620 2024-09-17T04:07:54.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.664+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb714148d50 con 0x7fb7140d0620 2024-09-17T04:07:54.664 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.664+0000 7fb70aa00640 1 -- 172.21.15.129:0/2287641087 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) ==== 95+0+3615 (secure 0 0 0) 0x7fb6fc063510 con 0x7fb7140d0620 2024-09-17T04:07:54.665 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:54.665 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":259,"btime":"2024-09-17T04:07:53:794507+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:52.797470+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249]},"id":19}]} 2024-09-17T04:07:54.665 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 259 2024-09-17T04:07:54.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 msgr2=0x7fb6f007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 0x7fb6f007b0a0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb704001610 tx=0x7fb704009040 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 msgr2=0x7fb714145a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e9f0 tx=0x7fb6fc00eeb0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 shutdown_connections 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb6f0078b90 0x7fb6f007b0a0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb71415cd30 0x7fb714148410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb7140d0620 0x7fb714145a90 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 --2- 172.21.15.129:0/2287641087 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb7140cfd10 0x7fb7141548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 >> 172.21.15.129:0/2287641087 conn(0x7fb71406cfc0 msgr2=0x7fb71415ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 shutdown_connections 2024-09-17T04:07:54.668 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:54.667+0000 7fb719600640 1 -- 172.21.15.129:0/2287641087 wait complete. 2024-09-17T04:07:54.891 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 259, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:07:32.455412+0000', 'modified': '2024-09-17T04:07:52.797470+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 263, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 12249, 'mds_1': 12432}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12249': {'gid': 12249, 'name': 'a', 'rank': 0, 'incarnation': 245, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3848773989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3848773989}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3848773989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_12432': {'gid': 12432, 'name': 'c', 'rank': 1, 'incarnation': 257, 'state': 'up:reconnect', 'state_seq': 3, 'addr': '172.21.15.129:6834/3076551424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 3076551424}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 3076551424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12249, 'qdb_cluster': [12249]} 2024-09-17T04:07:54.891 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12432:up:reconnect 2024-09-17T04:07:55.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:56.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 -- 172.21.15.129:0/1979668960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 msgr2=0x7f742806f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 --2- 172.21.15.129:0/1979668960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f742806f8a0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f7410005fd0 tx=0x7f741002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 -- 172.21.15.129:0/1979668960 shutdown_connections 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 --2- 172.21.15.129:0/1979668960 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f742806fde0 0x7f7428072fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 --2- 172.21.15.129:0/1979668960 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f742806f8a0 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 -- 172.21.15.129:0/1979668960 >> 172.21.15.129:0/1979668960 conn(0x7f74280b2510 msgr2=0x7f74280d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 -- 172.21.15.129:0/1979668960 shutdown_connections 2024-09-17T04:07:56.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.119+0000 7f742ce00640 1 -- 172.21.15.129:0/1979668960 wait complete. 2024-09-17T04:07:56.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.121+0000 7f742ce00640 1 Processor -- start 2024-09-17T04:07:56.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.121+0000 7f742ce00640 1 -- start start 2024-09-17T04:07:56.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f742806fde0 0x7f74280c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74280d0f20 0x7f74280c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f74280b4570 con 0x7f742815c830 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f74280b43f0 con 0x7f742806fde0 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f742ce00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f74280b46f0 con 0x7f74280d0f20 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f7427e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f7427e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53268/0 (socket says 172.21.15.129:53268) 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f7427e00640 1 -- 172.21.15.129:0/3826454739 learned_addr learned my addr 172.21.15.129:0/3826454739 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f741ea00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74280d0f20 0x7f74280c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.123+0000 7f7427400640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f742806fde0 0x7f74280c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 -- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74280d0f20 msgr2=0x7f74280c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74280d0f20 0x7f74280c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 -- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f742806fde0 msgr2=0x7f74280c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f742806fde0 0x7f74280c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 -- 172.21.15.129:0/3826454739 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742814f130 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f7427e00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f742000d990 tx=0x7f742000de50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7420017070 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f742000be00 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742809d620 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74200106c0 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.124+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74280c8fd0 con 0x7f742815c830 2024-09-17T04:07:56.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.125+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f742814f300 con 0x7f742815c830 2024-09-17T04:07:56.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.127+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f742001a070 con 0x7f742815c830 2024-09-17T04:07:56.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.127+0000 7f741fe00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 0x7f740407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.128+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f742009b2c0 con 0x7f742815c830 2024-09-17T04:07:56.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.128+0000 7f7427400640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 0x7f740407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.129+0000 7f7427400640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 0x7f740407b0a0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f74180015e0 tx=0x7f7418009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.132+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7420063600 con 0x7f742815c830 2024-09-17T04:07:56.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.349+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f742809d3b0 con 0x7f742815c830 2024-09-17T04:07:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.350+0000 7f741fe00640 1 -- 172.21.15.129:0/3826454739 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) ==== 95+0+3618 (secure 0 0 0) 0x7f7420062f00 con 0x7f742815c830 2024-09-17T04:07:56.350 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:56.350 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":261,"btime":"2024-09-17T04:07:55:812131+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:55.812123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249,12432]},"id":19}]} 2024-09-17T04:07:56.350 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 261 2024-09-17T04:07:56.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 msgr2=0x7f740407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 0x7f740407b0a0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f74180015e0 tx=0x7f7418009040 comp rx=0 tx=0).stop 2024-09-17T04:07:56.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 msgr2=0x7f74280ca250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7f742000d990 tx=0x7f742000de50 comp rx=0 tx=0).stop 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 shutdown_connections 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7404078b90 0x7f740407b0a0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f742815c830 0x7f74280ca250 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f74280d0f20 0x7f74280c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 --2- 172.21.15.129:0/3826454739 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f742806fde0 0x7f74280c97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 >> 172.21.15.129:0/3826454739 conn(0x7f74280b2510 msgr2=0x7f74280ce310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 shutdown_connections 2024-09-17T04:07:56.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.352+0000 7f742ce00640 1 -- 172.21.15.129:0/3826454739 wait complete. 2024-09-17T04:07:56.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 261, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:07:32.455412+0000', 'modified': '2024-09-17T04:07:55.812123+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 263, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 2, 'in': [0, 1], 'up': {'mds_0': 12249, 'mds_1': 12432}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12249': {'gid': 12249, 'name': 'a', 'rank': 0, 'incarnation': 245, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/3848773989', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 3848773989}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 3848773989}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_12432': {'gid': 12432, 'name': 'c', 'rank': 1, 'incarnation': 257, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.129:6834/3076551424', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6832', 'nonce': 3076551424}, {'type': 'v1', 'addr': '172.21.15.129:6834', 'nonce': 3076551424}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [39], 'metadata_pool': 38, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12249, 'qdb_cluster': [12249, 12432]} 2024-09-17T04:07:56.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 2/2 2024-09-17T04:07:56.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T04:07:56.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 -- 172.21.15.129:0/4206780981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0b4db0 msgr2=0x7fd96c0d4c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.802 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 --2- 172.21.15.129:0/4206780981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0b4db0 0x7fd96c0d4c90 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7fd95c0098e0 tx=0x7fd95c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 -- 172.21.15.129:0/4206780981 shutdown_connections 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 --2- 172.21.15.129:0/4206780981 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0d2e60 0x7fd96c0b6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 --2- 172.21.15.129:0/4206780981 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0b4db0 0x7fd96c0d4c90 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 -- 172.21.15.129:0/4206780981 >> 172.21.15.129:0/4206780981 conn(0x7fd96c08bfb0 msgr2=0x7fd96c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 -- 172.21.15.129:0/4206780981 shutdown_connections 2024-09-17T04:07:56.803 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.802+0000 7fd970a00640 1 -- 172.21.15.129:0/4206780981 wait complete. 2024-09-17T04:07:56.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.804+0000 7fd970a00640 1 Processor -- start 2024-09-17T04:07:56.804 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.804+0000 7fd970a00640 1 -- start start 2024-09-17T04:07:56.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0b4db0 0x7fd96c1e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 0x7fd96c1e92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.807 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c0cee90 con 0x7fd96c0d2e60 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c0b9a00 con 0x7fd96c0d2a50 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd970a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd96c0cf010 con 0x7fd96c0b4db0 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53282/0 (socket says 172.21.15.129:53282) 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd96be00640 1 -- 172.21.15.129:0/1090063519 learned_addr learned my addr 172.21.15.129:0/1090063519 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd96b400640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0b4db0 0x7fd96c1e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.806+0000 7fd96aa00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 0x7fd96c1e92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 -- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0b4db0 msgr2=0x7fd96c1e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0b4db0 0x7fd96c1e8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.808 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 -- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 msgr2=0x7fd96c1e92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 0x7fd96c1e92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 -- 172.21.15.129:0/1090063519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c0c9400 con 0x7fd96c0d2e60 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96aa00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 0x7fd96c1e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd96be00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fd96400e990 tx=0x7fd96400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd963400640 1 -- 172.21.15.129:0/1090063519 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964004d20 con 0x7fd96c0d2e60 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd963400640 1 -- 172.21.15.129:0/1090063519 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96400ca40 con 0x7fd96c0d2e60 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd963400640 1 -- 172.21.15.129:0/1090063519 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964005020 con 0x7fd96c0d2e60 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96c0c96a0 con 0x7fd96c0d2e60 2024-09-17T04:07:56.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.807+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96c0c9c60 con 0x7fd96c0d2e60 2024-09-17T04:07:56.810 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.810+0000 7fd963400640 1 -- 172.21.15.129:0/1090063519 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd9640051c0 con 0x7fd96c0d2e60 2024-09-17T04:07:56.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.811+0000 7fd963400640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 0x7fd94c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.811 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.811+0000 7fd963400640 1 -- 172.21.15.129:0/1090063519 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fd96409b5d0 con 0x7fd96c0d2e60 2024-09-17T04:07:56.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.811+0000 7fd96b400640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 0x7fd94c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.811+0000 7fd97905b740 1 Processor -- start 2024-09-17T04:07:56.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.812+0000 7fd97905b740 1 -- start start 2024-09-17T04:07:56.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.812+0000 7fd96b400640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 0x7fd94c07b0a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd95c034000 tx=0x7fd95c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 0x56433de2fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56433dd89430 con 0x56433da3e590 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56433dd795d0 con 0x56433da5f270 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56433dd68a00 con 0x56433da363b0 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53286/0 (socket says 172.21.15.129:53286) 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.813+0000 7fd96be00640 1 -- 172.21.15.129:0/1864402599 learned_addr learned my addr 172.21.15.129:0/1864402599 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96b400640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 0x56433de2fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96aa00640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 -- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 msgr2=0x56433de300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de300a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 -- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 msgr2=0x56433de2fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 0x56433de2fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 -- 172.21.15.129:0/1864402599 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56433dd31520 con 0x56433da3e590 2024-09-17T04:07:56.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96b400640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 0x56433de2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:56.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.814+0000 7fd96be00640 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7fd964063050 tx=0x7fd964062e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55be978150a06e7c server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.816+0000 7fd957400640 1 -- 172.21.15.129:0/1864402599 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd964099e60 con 0x56433da3e590 2024-09-17T04:07:56.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.816+0000 7fd957400640 1 -- 172.21.15.129:0/1864402599 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96406cc30 con 0x56433da3e590 2024-09-17T04:07:56.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.816+0000 7fd957400640 1 -- 172.21.15.129:0/1864402599 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96401f040 con 0x56433da3e590 2024-09-17T04:07:56.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 -- 172.21.15.129:0/1864402599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 msgr2=0x56433de00150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7fd964063050 tx=0x7fd964062e60 comp rx=0 tx=0).stop 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 -- 172.21.15.129:0/1864402599 shutdown_connections 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de00150 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 --2- 172.21.15.129:0/1864402599 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da5f270 0x56433de2fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 -- 172.21.15.129:0/1864402599 >> 172.21.15.129:0/1864402599 conn(0x56433da533c0 msgr2=0x56433de05b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 -- 172.21.15.129:0/1864402599 shutdown_connections 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.817+0000 7fd97905b740 1 -- 172.21.15.129:0/1864402599 wait complete. 2024-09-17T04:07:56.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.818+0000 7fd97905b740 1 -- start start 2024-09-17T04:07:56.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.819+0000 7fd97905b740 1 Processor -- start 2024-09-17T04:07:56.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da3e590 0x56433de5d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56433dd0f330 con 0x56433da5f270 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56433dd6c810 con 0x56433da363b0 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56433dad9620 con 0x56433da3e590 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.820+0000 7fd97905b740 10 client.0 _ll_register_callbacks cb 0x56433dd14420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da3e590 0x56433de5d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53290/0 (socket says 172.21.15.129:53290) 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 -- 172.21.15.129:0/1284434943 learned_addr learned my addr 172.21.15.129:0/1284434943 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 -- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da3e590 msgr2=0x56433de5d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da3e590 0x56433de5d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 -- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 msgr2=0x56433de262a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de262a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 -- 172.21.15.129:0/1284434943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56433daca850 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd96be00640 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7fd964063050 tx=0x7fd96406a3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc7a25a0fcb56f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.821+0000 7fd94be00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96401f040 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd96406cc30 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fd96409b0d0 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7fd964099e60 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 10 client.12471 fetch_fsmap learned FSMap version 261 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=2+}) -- 0x56433db416b0 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96406cc30 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 261) ==== 2312+0+0 (secure 0 0 0) 0x7fd96400f040 con 0x56433da5f270 2024-09-17T04:07:56.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 10 client.12471 fetch_fsmap finished waiting for FSMap version 261 2024-09-17T04:07:56.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 10 client.12471 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:07:56.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 20 client.12471 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:56.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433da1de40 0x56433de292a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 4 client.12471 mds_command: new command op to 12249 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:56.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.822+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 --> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56433dd7b520 con 0x56433da1de40 2024-09-17T04:07:56.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.823+0000 7fd96b400640 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433da1de40 0x56433de292a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.823+0000 7fd94be00640 10 client.12471 ms_handle_connect on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:56.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.823+0000 7fd96b400640 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433da1de40 0x56433de292a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea1b06970032eeeb server_cookie=6e0776131f959399 in_seq=0 out_seq=0 2024-09-17T04:07:56.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd94be00640 1 -- 172.21.15.129:0/1284434943 <== mds.0 v2:172.21.15.129:6833/3848773989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x56433dd7b520 con 0x56433da1de40 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd94be00640 10 client.12471 handle_command_reply: tid=0 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 client.12471 shutdown 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 msgr2=0x56433de5dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7fd964063050 tx=0x7fd96406a3d0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 shutdown_connections 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433da1de40 0x56433de292a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de5dcf0 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da3e590 0x56433de5d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 --2- 172.21.15.129:0/1284434943 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de262a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd94be00640 0 client.12471 ms_handle_reset on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.824+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 >> 172.21.15.129:0/1284434943 conn(0x56433da533c0 msgr2=0x56433de015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.825+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 shutdown_connections 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.825+0000 7fd97905b740 1 -- 172.21.15.129:0/1284434943 wait complete. 2024-09-17T04:07:56.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.825+0000 7fd97905b740 20 client.12471 trim_cache size 0 max 16384 2024-09-17T04:07:56.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.844+0000 7fd97905b740 1 Processor -- start 2024-09-17T04:07:56.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.844+0000 7fd97905b740 1 -- start start 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de01430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da3e590 0x56433de01970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c14a980 con 0x56433da5f270 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c201500 con 0x56433da3e590 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.845+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56433dd6c810 con 0x56433da363b0 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53300/0 (socket says 172.21.15.129:53300) 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 -- 172.21.15.129:0/1406251502 learned_addr learned my addr 172.21.15.129:0/1406251502 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96b400640 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de01430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96aa00640 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da3e590 0x56433de01970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 -- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 msgr2=0x56433de01430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de01430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 -- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da3e590 msgr2=0x56433de01970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da3e590 0x56433de01970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 -- 172.21.15.129:0/1406251502 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56433de04b10 con 0x56433da5f270 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.846+0000 7fd96be00640 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7fd96405f400 tx=0x7fd96405f320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a50216adc452fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.847+0000 7fd956a00640 1 -- 172.21.15.129:0/1406251502 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96401f070 con 0x56433da5f270 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.847+0000 7fd956a00640 1 -- 172.21.15.129:0/1406251502 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96406cc30 con 0x56433da5f270 2024-09-17T04:07:56.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.847+0000 7fd956a00640 1 -- 172.21.15.129:0/1406251502 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96409b0b0 con 0x56433da5f270 2024-09-17T04:07:56.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 -- 172.21.15.129:0/1406251502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 msgr2=0x56433de6dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7fd96405f400 tx=0x7fd96405f320 comp rx=0 tx=0).stop 2024-09-17T04:07:56.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 -- 172.21.15.129:0/1406251502 shutdown_connections 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da5f270 0x56433de6dc70 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da3e590 0x56433de01970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 --2- 172.21.15.129:0/1406251502 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da363b0 0x56433de01430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 -- 172.21.15.129:0/1406251502 >> 172.21.15.129:0/1406251502 conn(0x56433da533c0 msgr2=0x56433de5d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 -- 172.21.15.129:0/1406251502 shutdown_connections 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.848+0000 7fd97905b740 1 -- 172.21.15.129:0/1406251502 wait complete. 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.849+0000 7fd97905b740 1 -- start start 2024-09-17T04:07:56.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.849+0000 7fd97905b740 1 Processor -- start 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de2cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da5f270 0x56433de01970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x56433dd0f330 con 0x56433da3e590 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x56433de05350 con 0x56433da363b0 2024-09-17T04:07:56.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x56433de4bcd0 con 0x56433da5f270 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd97905b740 10 client.0 _ll_register_callbacks cb 0x7fd96406a060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.850+0000 7fd96aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de2cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96be00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da5f270 0x56433de01970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53304/0 (socket says 172.21.15.129:53304) 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 -- 172.21.15.129:0/3173836534 learned_addr learned my addr 172.21.15.129:0/3173836534 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 -- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da5f270 msgr2=0x56433de01970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da5f270 0x56433de01970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 -- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 msgr2=0x56433de2cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de2cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 -- 172.21.15.129:0/3173836534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56433dec9aa0 con 0x56433da3e590 2024-09-17T04:07:56.854 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd96aa00640 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0d11a0 tx=0x7fd958015720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70637a31644a6256 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd94be00640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.851+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95801b070 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fd958006820 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd96406cc30 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7fd958026050 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 10 client.12477 fetch_fsmap learned FSMap version 261 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=2+}) -- 0x56433de4bad0 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd96406cc30 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 261) ==== 2312+0+0 (secure 0 0 0) 0x7fd958029020 con 0x56433da3e590 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 10 client.12477 fetch_fsmap finished waiting for FSMap version 261 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 10 client.12477 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 20 client.12477 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:56.855 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433decd260 0x56433decd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 4 client.12477 mds_command: new command op to 12249 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 --> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] -- command(tid 0: {"prefix": "status"}) -- 0x56433dd76fd0 con 0x56433decd260 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.852+0000 7fd96b400640 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433decd260 0x56433decd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd96b400640 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433decd260 0x56433decd670 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21abe21db0651add server_cookie=942b86051a2baff9 in_seq=0 out_seq=0 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd94be00640 10 client.12477 ms_handle_connect on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd94be00640 1 -- 172.21.15.129:0/3173836534 <== mds.0 v2:172.21.15.129:6833/3848773989 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x56433dd76fd0 con 0x56433decd260 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd94be00640 10 client.12477 handle_command_reply: tid=0 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 client.12477 shutdown 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 msgr2=0x56433de01430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.856 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7fd96c0d11a0 tx=0x7fd958015720 comp rx=0 tx=0).stop 2024-09-17T04:07:56.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 shutdown_connections 2024-09-17T04:07:56.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:6833/3848773989,v1:172.21.15.129:6835/3848773989] conn(0x56433decd260 0x56433decd670 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x56433da5f270 0x56433de01970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x56433da3e590 0x56433de01430 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 --2- 172.21.15.129:0/3173836534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56433da363b0 0x56433de2cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd94be00640 0 client.12477 ms_handle_reset on v2:172.21.15.129:6833/3848773989 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.853+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 >> 172.21.15.129:0/3173836534 conn(0x56433da533c0 msgr2=0x56433de2a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 shutdown_connections 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd97905b740 1 -- 172.21.15.129:0/3173836534 wait complete. 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd97905b740 20 client.12477 trim_cache size 0 max 16384 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:07:56.858 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 12249, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 23.877060801999999, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 261, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 263, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 263, 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 28.910223764000001 2024-09-17T04:07:56.859 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:56.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 msgr2=0x7fd94c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 0x7fd94c07b0a0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7fd95c034000 tx=0x7fd95c004730 comp rx=0 tx=0).stop 2024-09-17T04:07:56.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 msgr2=0x7fd96c0cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7fd96400e990 tx=0x7fd96400ee50 comp rx=0 tx=0).stop 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 shutdown_connections 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd94c078b90 0x7fd94c07b0a0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd96c0d2e60 0x7fd96c0cd820 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd96c0d2a50 0x7fd96c1e92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 --2- 172.21.15.129:0/1090063519 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd96c0b4db0 0x7fd96c1e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.854+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 >> 172.21.15.129:0/1090063519 conn(0x7fd96c08bfb0 msgr2=0x7fd96c0d1bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.855+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 shutdown_connections 2024-09-17T04:07:56.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:56.855+0000 7fd970a00640 1 -- 172.21.15.129:0/1090063519 wait complete. 2024-09-17T04:07:56.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-17T04:07:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 -- 172.21.15.129:0/2814901524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d38155a70 msgr2=0x7f9d380b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 --2- 172.21.15.129:0/2814901524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d38155a70 0x7f9d380b39f0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7f9d38072c00 tx=0x7f9d2402a4d0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 -- 172.21.15.129:0/2814901524 shutdown_connections 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 --2- 172.21.15.129:0/2814901524 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d38155a70 0x7f9d380b39f0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 --2- 172.21.15.129:0/2814901524 >> v2:172.21.15.181:3300/0 conn(0x7f9d3815f360 0x7f9d3815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 -- 172.21.15.129:0/2814901524 >> 172.21.15.129:0/2814901524 conn(0x7f9d3806cfc0 msgr2=0x7f9d3806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 -- 172.21.15.129:0/2814901524 shutdown_connections 2024-09-17T04:07:57.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.106+0000 7f9d3f200640 1 -- 172.21.15.129:0/2814901524 wait complete. 2024-09-17T04:07:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.108+0000 7f9d3f200640 1 Processor -- start 2024-09-17T04:07:57.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.108+0000 7f9d3f200640 1 -- start start 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.109+0000 7f9d3f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d38155a70 0x7f9d3815d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d3815f360 0x7f9d380a9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d380d0620 con 0x7f9d380b3f30 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9d381560d0 con 0x7f9d38155a70 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9d380d07a0 con 0x7f9d3815f360 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53310/0 (socket says 172.21.15.129:53310) 2024-09-17T04:07:57.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 -- 172.21.15.129:0/1026710675 learned_addr learned my addr 172.21.15.129:0/1026710675 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37400640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d38155a70 0x7f9d3815d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d3d200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d3815f360 0x7f9d380a9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 -- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d3815f360 msgr2=0x7f9d380a9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d3815f360 0x7f9d380a9c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 -- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d38155a70 msgr2=0x7f9d3815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d38155a70 0x7f9d3815d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.110+0000 7f9d37e00640 1 -- 172.21.15.129:0/1026710675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d380af370 con 0x7f9d380b3f30 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d37e00640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800c970 tx=0x7f9d2800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d2fe00640 1 -- 172.21.15.129:0/1026710675 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d28007b50 con 0x7f9d380b3f30 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d2fe00640 1 -- 172.21.15.129:0/1026710675 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d28007de0 con 0x7f9d380b3f30 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d2fe00640 1 -- 172.21.15.129:0/1026710675 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800f6b0 con 0x7f9d380b3f30 2024-09-17T04:07:57.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d380af650 con 0x7f9d380b3f30 2024-09-17T04:07:57.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.111+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d3815da40 con 0x7f9d380b3f30 2024-09-17T04:07:57.113 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.113+0000 7f9d2fe00640 1 -- 172.21.15.129:0/1026710675 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9d2800f850 con 0x7f9d380b3f30 2024-09-17T04:07:57.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.115+0000 7f9d2fe00640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 0x7f9d1807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.115+0000 7f9d2fe00640 1 -- 172.21.15.129:0/1026710675 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f9d2809b4b0 con 0x7f9d380b3f30 2024-09-17T04:07:57.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.116+0000 7f9d37400640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 0x7f9d1807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.116+0000 7f9d46200740 1 Processor -- start 2024-09-17T04:07:57.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.116+0000 7f9d46200740 1 -- start start 2024-09-17T04:07:57.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.116+0000 7f9d37400640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 0x7f9d1807b0a0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f9d24033000 tx=0x7f9d24004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa5b3b0 0x5570cfe54eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa63590 0x5570cfe24f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570cfdae430 con 0x5570cfa84270 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5570cfd9e5d0 con 0x5570cfa63590 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.117+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5570cfd8da00 con 0x5570cfa5b3b0 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53324/0 (socket says 172.21.15.129:53324) 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 -- 172.21.15.129:0/2735359707 learned_addr learned my addr 172.21.15.129:0/2735359707 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37400640 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa5b3b0 0x5570cfe54eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d3d200640 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa63590 0x5570cfe24f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 -- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa5b3b0 msgr2=0x5570cfe54eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa5b3b0 0x5570cfe54eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 -- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa63590 msgr2=0x5570cfe24f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa63590 0x5570cfe24f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 -- 172.21.15.129:0/2735359707 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570cfd56520 con 0x5570cfa84270 2024-09-17T04:07:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f9d28060410 tx=0x7f9d28062a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa82bf3d51b44b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d2cc00640 1 -- 172.21.15.129:0/2735359707 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2806cb10 con 0x5570cfa84270 2024-09-17T04:07:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d2cc00640 1 -- 172.21.15.129:0/2735359707 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d2809b0d0 con 0x5570cfa84270 2024-09-17T04:07:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.118+0000 7f9d2cc00640 1 -- 172.21.15.129:0/2735359707 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800fd60 con 0x5570cfa84270 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 -- 172.21.15.129:0/2735359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 msgr2=0x5570cfe54970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f9d28060410 tx=0x7f9d28062a10 comp rx=0 tx=0).stop 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 -- 172.21.15.129:0/2735359707 shutdown_connections 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa63590 0x5570cfe24f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa5b3b0 0x5570cfe54eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 --2- 172.21.15.129:0/2735359707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe54970 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 -- 172.21.15.129:0/2735359707 >> 172.21.15.129:0/2735359707 conn(0x5570cfa783c0 msgr2=0x5570cfe2a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 -- 172.21.15.129:0/2735359707 shutdown_connections 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.119+0000 7f9d46200740 1 -- 172.21.15.129:0/2735359707 wait complete. 2024-09-17T04:07:57.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.120+0000 7f9d46200740 1 -- start start 2024-09-17T04:07:57.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.121+0000 7f9d46200740 1 Processor -- start 2024-09-17T04:07:57.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe81c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa84270 0x5570cfe82170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570cfd34330 con 0x5570cfa5b3b0 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5570cfd91810 con 0x5570cfa84270 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5570cfafe620 con 0x5570cfa63590 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d46200740 10 client.0 _ll_register_callbacks cb 0x5570cfd39420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d37e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53336/0 (socket says 172.21.15.129:53336) 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.122+0000 7f9d37e00640 1 -- 172.21.15.129:0/3612505818 learned_addr learned my addr 172.21.15.129:0/3612505818 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d3d200640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa84270 0x5570cfe82170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37400640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe81c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 -- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 msgr2=0x5570cfe81c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe81c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 -- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa84270 msgr2=0x5570cfe82170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa84270 0x5570cfe82170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 -- 172.21.15.129:0/3612505818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570cfaef850 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d37e00640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f9d28062a80 tx=0x7f9d2805f0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe505bf8a29fa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d21600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2800fd40 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f9d2807ed50 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.123+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d2809b0d0 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7f9d28085380 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 10 client.12489 fetch_fsmap learned FSMap version 261 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=2+}) -- 0x5570cfb666b0 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d2809b0d0 con 0x5570cfa5b3b0 2024-09-17T04:07:57.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 261) ==== 2312+0+0 (secure 0 0 0) 0x7f9d28020030 con 0x5570cfa5b3b0 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 10 client.12489 fetch_fsmap finished waiting for FSMap version 261 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 10 client.12489 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 20 client.12489 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfa42e40 0x5570cfe4d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 4 client.12489 mds_command: new command op to 12432 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 --> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570cfda0520 con 0x5570cfa42e40 2024-09-17T04:07:57.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.124+0000 7f9d37400640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfa42e40 0x5570cfe4d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.125+0000 7f9d37400640 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfa42e40 0x5570cfe4d440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c705fcb2980dbfbc server_cookie=52e41016aa2985a0 in_seq=0 out_seq=0 2024-09-17T04:07:57.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.125+0000 7f9d21600640 10 client.12489 ms_handle_connect on v2:172.21.15.129:6832/3076551424 2024-09-17T04:07:57.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d21600640 1 -- 172.21.15.129:0/3612505818 <== mds.1 v2:172.21.15.129:6832/3076551424 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x5570cfda0520 con 0x5570cfa42e40 2024-09-17T04:07:57.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d21600640 10 client.12489 handle_command_reply: tid=0 2024-09-17T04:07:57.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 client.12489 shutdown 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 msgr2=0x5570cfe4e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f9d28062a80 tx=0x7f9d2805f0a0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 shutdown_connections 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfa42e40 0x5570cfe4d440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa84270 0x5570cfe82170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe81c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 --2- 172.21.15.129:0/3612505818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa5b3b0 0x5570cfe4e2d0 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d21600640 0 client.12489 ms_handle_reset on v2:172.21.15.129:6832/3076551424 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 >> 172.21.15.129:0/3612505818 conn(0x5570cfa783c0 msgr2=0x5570cfe72d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 shutdown_connections 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 1 -- 172.21.15.129:0/3612505818 wait complete. 2024-09-17T04:07:57.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.126+0000 7f9d46200740 20 client.12489 trim_cache size 0 max 16384 2024-09-17T04:07:57.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.145+0000 7f9d46200740 1 Processor -- start 2024-09-17T04:07:57.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.145+0000 7f9d46200740 1 -- start start 2024-09-17T04:07:57.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfe25c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe51ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d380b1d30 con 0x5570cfa84270 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9d38201570 con 0x5570cfa5b3b0 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5570cfd91810 con 0x5570cfa63590 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53350/0 (socket says 172.21.15.129:53350) 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 -- 172.21.15.129:0/16514436 learned_addr learned my addr 172.21.15.129:0/16514436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d37e00640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfe25c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d37400640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe51ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 -- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 msgr2=0x5570cfe51ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.148 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe51ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 -- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 msgr2=0x5570cfe25c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfe25c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d3d200640 1 -- 172.21.15.129:0/16514436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570cfe4cba0 con 0x5570cfa84270 2024-09-17T04:07:57.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.147+0000 7f9d37400640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe51ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:07:57.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.148+0000 7f9d3d200640 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b7d0 tx=0x7f9d30009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8375a8ad3798e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.148+0000 7f9d23e00640 1 -- 172.21.15.129:0/16514436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3001c030 con 0x5570cfa84270 2024-09-17T04:07:57.150 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.148+0000 7f9d23e00640 1 -- 172.21.15.129:0/16514436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d30003970 con 0x5570cfa84270 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.148+0000 7f9d23e00640 1 -- 172.21.15.129:0/16514436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30003bc0 con 0x5570cfa84270 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 -- 172.21.15.129:0/16514436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 msgr2=0x5570cfe2aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b7d0 tx=0x7f9d30009630 comp rx=0 tx=0).stop 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 -- 172.21.15.129:0/16514436 shutdown_connections 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa84270 0x5570cfe2aab0 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa63590 0x5570cfe51ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 --2- 172.21.15.129:0/16514436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfe25c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 -- 172.21.15.129:0/16514436 >> 172.21.15.129:0/16514436 conn(0x5570cfa783c0 msgr2=0x5570cfe80f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 -- 172.21.15.129:0/16514436 shutdown_connections 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.149+0000 7f9d46200740 1 -- 172.21.15.129:0/16514436 wait complete. 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.150+0000 7f9d46200740 1 -- start start 2024-09-17T04:07:57.151 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.150+0000 7f9d46200740 1 Processor -- start 2024-09-17T04:07:57.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.151+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfef2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.155 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 0x5570cfeee870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x5570cfd34330 con 0x5570cfa63590 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x5570cfef3020 con 0x5570cfa5b3b0 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x5570cfe25f00 con 0x5570cfa84270 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d46200740 10 client.0 _ll_register_callbacks cb 0x7f9d28062b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53364/0 (socket says 172.21.15.129:53364) 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 -- 172.21.15.129:0/2579165063 learned_addr learned my addr 172.21.15.129:0/2579165063 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfef2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d3d200640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 0x5570cfeee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 -- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 msgr2=0x5570cfeee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.156 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 0x5570cfeee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 -- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 msgr2=0x5570cfef2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfef2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37400640 1 -- 172.21.15.129:0/2579165063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570cfe68a10 con 0x5570cfa63590 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfef2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.152+0000 7f9d3d200640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 0x5570cfeee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d37400640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f9d2402aac0 tx=0x7f9d24009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc2aa138b5037ac server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d21600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24035040 con 0x5570cfa63590 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f9d24003b00 con 0x5570cfa63590 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d30003970 con 0x5570cfa63590 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=261) ==== 24+0+0 (secure 0 0 0) 0x7f9d24064050 con 0x5570cfa63590 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d46200740 10 client.12495 fetch_fsmap learned FSMap version 261 2024-09-17T04:07:57.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=261,monmap=2+}) -- 0x5570cfe81ea0 con 0x5570cfa63590 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.153+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d30003970 con 0x5570cfa63590 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 261) ==== 2312+0+0 (secure 0 0 0) 0x7f9d24067020 con 0x5570cfa63590 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 10 client.12495 fetch_fsmap finished waiting for FSMap version 261 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 10 client.12495 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 20 client.12495 populate_metadata read hostname 'smithi129' 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfe4ebd0 0x5570cfe4efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 4 client.12495 mds_command: new command op to 12432 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 --> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] -- command(tid 0: {"prefix": "status"}) -- 0x5570cfd9bfd0 con 0x5570cfe4ebd0 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfe4ebd0 0x5570cfe4efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d37e00640 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfe4ebd0 0x5570cfe4efe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8a90fc603f106d6d server_cookie=a22b8a9da8875d90 in_seq=0 out_seq=0 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.154+0000 7f9d21600640 10 client.12495 ms_handle_connect on v2:172.21.15.129:6832/3076551424 2024-09-17T04:07:57.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d21600640 1 -- 172.21.15.129:0/2579165063 <== mds.1 v2:172.21.15.129:6832/3076551424 1 ==== command_reply(tid 0: 0 ) ==== 8+0+335 (crc 0 0 0) 0x5570cfd9bfd0 con 0x5570cfe4ebd0 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d21600640 10 client.12495 handle_command_reply: tid=0 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 client.12495 shutdown 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 msgr2=0x5570cfe54460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f9d2402aac0 tx=0x7f9d24009260 comp rx=0 tx=0).stop 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 shutdown_connections 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:6832/3076551424,v1:172.21.15.129:6834/3076551424] conn(0x5570cfe4ebd0 0x5570cfe4efe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x5570cfa84270 0x5570cfeee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x5570cfa63590 0x5570cfe54460 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d21600640 0 client.12495 ms_handle_reset on v2:172.21.15.129:6832/3076551424 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 --2- 172.21.15.129:0/2579165063 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5570cfa5b3b0 0x5570cfef2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 >> 172.21.15.129:0/2579165063 conn(0x5570cfa783c0 msgr2=0x5570cfe2aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.159 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 shutdown_connections 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 1 -- 172.21.15.129:0/2579165063 wait complete. 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.155+0000 7f9d46200740 20 client.12495 trim_cache size 0 max 16384 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 1, 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 12432, 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:07:57.160 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 5.3619201460000001, 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 261, 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 263, 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 263, 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 6.1390448720000004 2024-09-17T04:07:57.161 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 msgr2=0x7f9d1807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 0x7f9d1807b0a0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f9d24033000 tx=0x7f9d24004730 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 msgr2=0x7f9d38153da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800c970 tx=0x7f9d2800ce30 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 shutdown_connections 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d18078b90 0x7f9d1807b0a0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d3815f360 0x7f9d380a9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d38155a70 0x7f9d3815d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 --2- 172.21.15.129:0/1026710675 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d380b3f30 0x7f9d38153da0 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.162 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 >> 172.21.15.129:0/1026710675 conn(0x7f9d3806cfc0 msgr2=0x7f9d38158260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 shutdown_connections 2024-09-17T04:07:57.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.156+0000 7f9d3f200640 1 -- 172.21.15.129:0/1026710675 wait complete. 2024-09-17T04:07:57.179 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:07:57.180 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> import uuid 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> n = 10 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = False 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> if 0 > 0: 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> create_hard_links = True 2024-09-17T04:07:57.216 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/delete_me/file" 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> dpath = os.path.dirname(path) 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> fnameprefix = os.path.basename(path) 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> os.makedirs(dpath, exist_ok=True) 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> try: 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(n): 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> with open(fpath, '"'"'w'"'"') as f: 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> f.write(f"{i}") 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> f.flush() 2024-09-17T04:07:57.217 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(f.fileno()) 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> os.unlink(fpath) 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> if create_hard_links: 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> for j in range(0): 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> if False: 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> os.fsync(dirfd) 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> finally: 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> os.close(dirfd) 2024-09-17T04:07:57.218 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:07:57.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:07:57.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:07:57.391 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 delete_me) 2024-09-17T04:07:57.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:07:57.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 -- 172.21.15.129:0/3795168706 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000cfd10 msgr2=0x7f68000cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 --2- 172.21.15.129:0/3795168706 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000cfd10 0x7f68000cd8e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f67ec009830 tx=0x7f67ec02a530 comp rx=0 tx=0).stop 2024-09-17T04:07:57.736 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 -- 172.21.15.129:0/3795168706 shutdown_connections 2024-09-17T04:07:57.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 --2- 172.21.15.129:0/3795168706 >> v2:172.21.15.181:3300/0 conn(0x7f68000d0620 0x7f68000d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 --2- 172.21.15.129:0/3795168706 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000cfd10 0x7f68000cd8e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 -- 172.21.15.129:0/3795168706 >> 172.21.15.129:0/3795168706 conn(0x7f680006cfc0 msgr2=0x7f680006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 -- 172.21.15.129:0/3795168706 shutdown_connections 2024-09-17T04:07:57.737 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.736+0000 7f6805a00640 1 -- 172.21.15.129:0/3795168706 wait complete. 2024-09-17T04:07:57.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.738+0000 7f6805a00640 1 Processor -- start 2024-09-17T04:07:57.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.738+0000 7f6805a00640 1 -- start start 2024-09-17T04:07:57.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68000cfd10 0x7f68001eda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000d0620 0x7f68001e5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.741 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6800073890 con 0x7f680015cd30 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6800073710 con 0x7f68000cfd10 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f6805a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6800073a10 con 0x7f68000d0620 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f67ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f67ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53368/0 (socket says 172.21.15.129:53368) 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f67ff400640 1 -- 172.21.15.129:0/3750793128 learned_addr learned my addr 172.21.15.129:0/3750793128 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f67fea00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68000cfd10 0x7f68001eda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.740+0000 7f67fe000640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000d0620 0x7f68001e5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 -- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000d0620 msgr2=0x7f68001e5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000d0620 0x7f68001e5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 -- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68000cfd10 msgr2=0x7f68001eda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68000cfd10 0x7f68001eda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 -- 172.21.15.129:0/3750793128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68001edfa0 con 0x7f680015cd30 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67ff400640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f67f800e9a0 tx=0x7f67f800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f800cd80 con 0x7f680015cd30 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f67f8004030 con 0x7f680015cd30 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f8004480 con 0x7f680015cd30 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68001e6b60 con 0x7f680015cd30 2024-09-17T04:07:57.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.741+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68001e7120 con 0x7f680015cd30 2024-09-17T04:07:57.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.742+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6800073f30 con 0x7f680015cd30 2024-09-17T04:07:57.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.743+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f67f80035f0 con 0x7f680015cd30 2024-09-17T04:07:57.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.744+0000 7f67f6a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 0x7f67e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:07:57.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.745+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f67f809b750 con 0x7f680015cd30 2024-09-17T04:07:57.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.745+0000 7f67fea00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 0x7f67e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:07:57.746 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.746+0000 7f67fea00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 0x7f67e007b0a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001e20 tx=0x7f67f0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:07:57.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.749+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f67f80634d0 con 0x7f680015cd30 2024-09-17T04:07:57.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.970+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68001e68f0 con 0x7f680015cd30 2024-09-17T04:07:57.971 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.971+0000 7f67f6a00640 1 -- 172.21.15.129:0/3750793128 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) ==== 95+0+3618 (secure 0 0 0) 0x7f67f8067f00 con 0x7f680015cd30 2024-09-17T04:07:57.971 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:07:57.971 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":261,"btime":"2024-09-17T04:07:55:812131+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:55.812123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249,12432]},"id":19}]} 2024-09-17T04:07:57.972 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 261 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.973+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 msgr2=0x7f67e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.973+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 0x7f67e007b0a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f67f0001e20 tx=0x7f67f0009040 comp rx=0 tx=0).stop 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.973+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 msgr2=0x7f68001f7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.973+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7f67f800e9a0 tx=0x7f67f800ee60 comp rx=0 tx=0).stop 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 shutdown_connections 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f67e0078b90 0x7f67e007b0a0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f680015cd30 0x7f68001f7150 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.974 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f68000d0620 0x7f68001e5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 --2- 172.21.15.129:0/3750793128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68000cfd10 0x7f68001eda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:07:57.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 >> 172.21.15.129:0/3750793128 conn(0x7f680006cfc0 msgr2=0x7f680006f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:07:57.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 shutdown_connections 2024-09-17T04:07:57.975 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:07:57.974+0000 7f6805a00640 1 -- 172.21.15.129:0/3750793128 wait complete. 2024-09-17T04:07:57.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:07:58.234 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:07:58.235 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:07:58.235 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:08:00.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 -- 172.21.15.129:0/2421548849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29940cfd10 msgr2=0x7f29940cd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 --2- 172.21.15.129:0/2421548849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29940cfd10 0x7f29940cd8e0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7f29941572f0 tx=0x7f298002a4b0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 -- 172.21.15.129:0/2421548849 shutdown_connections 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 --2- 172.21.15.129:0/2421548849 >> v2:172.21.15.181:3300/0 conn(0x7f29940d0620 0x7f29940d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 --2- 172.21.15.129:0/2421548849 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f29940cfd10 0x7f29940cd8e0 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.467+0000 7f299aa00640 1 -- 172.21.15.129:0/2421548849 >> 172.21.15.129:0/2421548849 conn(0x7f299406cfc0 msgr2=0x7f299406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:00.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.468+0000 7f299aa00640 1 -- 172.21.15.129:0/2421548849 shutdown_connections 2024-09-17T04:08:00.469 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.468+0000 7f299aa00640 1 -- 172.21.15.129:0/2421548849 wait complete. 2024-09-17T04:08:00.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.470+0000 7f299aa00640 1 Processor -- start 2024-09-17T04:08:00.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.470+0000 7f299aa00640 1 -- start start 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29940cfd10 0x7f29940aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29940d0620 0x7f2994159290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2994073890 con 0x7f299415cd30 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2994073710 con 0x7f29940d0620 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.471+0000 7f299aa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2994073a10 con 0x7f29940cfd10 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53378/0 (socket says 172.21.15.129:53378) 2024-09-17T04:08:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 -- 172.21.15.129:0/1510977324 learned_addr learned my addr 172.21.15.129:0/1510977324 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2993400640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29940d0620 0x7f2994159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2993e00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29940cfd10 0x7f29940aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 -- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29940cfd10 msgr2=0x7f29940aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29940cfd10 0x7f29940aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 -- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29940d0620 msgr2=0x7f2994159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29940d0620 0x7f2994159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 -- 172.21.15.129:0/1510977324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29940b3a10 con 0x7f299415cd30 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.472+0000 7f2998a00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f298c005d00 tx=0x7f298c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.473+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c017070 con 0x7f299415cd30 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.473+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f298c005330 con 0x7f299415cd30 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.473+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29940b3cf0 con 0x7f299415cd30 2024-09-17T04:08:00.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.473+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0047a0 con 0x7f299415cd30 2024-09-17T04:08:00.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.473+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29940b42b0 con 0x7f299415cd30 2024-09-17T04:08:00.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.474+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2994073f30 con 0x7f299415cd30 2024-09-17T04:08:00.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.475+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f298c004940 con 0x7f299415cd30 2024-09-17T04:08:00.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.476+0000 7f298be00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 0x7f297407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:00.476 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.476+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f298c09ad50 con 0x7f299415cd30 2024-09-17T04:08:00.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.477+0000 7f2993e00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 0x7f297407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:00.477 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.477+0000 7f2993e00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 0x7f297407b0a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2984002200 tx=0x7f29840019f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:00.481 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.481+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f298c062ad0 con 0x7f299415cd30 2024-09-17T04:08:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.699+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299415a390 con 0x7f299415cd30 2024-09-17T04:08:00.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.700+0000 7f298be00640 1 -- 172.21.15.129:0/1510977324 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) ==== 95+0+3618 (secure 0 0 0) 0x7f298c067500 con 0x7f299415cd30 2024-09-17T04:08:00.700 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:00.700 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":261,"btime":"2024-09-17T04:07:55:812131+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:55.812123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249,12432]},"id":19}]} 2024-09-17T04:08:00.701 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 261 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 msgr2=0x7f297407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 0x7f297407b0a0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2984002200 tx=0x7f29840019f0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 msgr2=0x7f29941517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 secure :-1 s=READY pgs=2848 cs=0 l=1 rev1=1 crypto rx=0x7f298c005d00 tx=0x7f298c00dae0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 shutdown_connections 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2974078b90 0x7f297407b0a0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f299415cd30 0x7f29941517d0 unknown :-1 s=CLOSED pgs=2848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f29940d0620 0x7f2994159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 --2- 172.21.15.129:0/1510977324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f29940cfd10 0x7f29940aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.702+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 >> 172.21.15.129:0/1510977324 conn(0x7f299406cfc0 msgr2=0x7f299415b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.703+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 shutdown_connections 2024-09-17T04:08:00.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:00.703+0000 7f299aa00640 1 -- 172.21.15.129:0/1510977324 wait complete. 2024-09-17T04:08:00.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:08:00.971 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:08:00.971 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:08:00.972 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:08:02.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:03.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 -- 172.21.15.129:0/4213842069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 msgr2=0x7f56f0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:03.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 --2- 172.21.15.129:0/4213842069 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f56f0005220 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7f57080cddc0 tx=0x7f56f0005830 comp rx=0 tx=0).stop 2024-09-17T04:08:03.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570de00640 1 -- 172.21.15.129:0/4213842069 reap_dead start 2024-09-17T04:08:03.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 -- 172.21.15.129:0/4213842069 shutdown_connections 2024-09-17T04:08:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 -- 172.21.15.129:0/4213842069 >> 172.21.15.129:0/4213842069 conn(0x7f57080b2510 msgr2=0x7f57080d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 -- 172.21.15.129:0/4213842069 shutdown_connections 2024-09-17T04:08:03.204 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.203+0000 7f570e800640 1 -- 172.21.15.129:0/4213842069 wait complete. 2024-09-17T04:08:03.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.205+0000 7f570e800640 1 Processor -- start 2024-09-17T04:08:03.205 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.205+0000 7f570e800640 1 -- start start 2024-09-17T04:08:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.206+0000 7f570e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.206+0000 7f570e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57081f59a0 0x7f570815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 0x7f570815e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f570815cc20 con 0x7f56f0004d90 2024-09-17T04:08:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f570815caa0 con 0x7f57081f59a0 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f57080b43f0 con 0x7f570815f4b0 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53386/0 (socket says 172.21.15.129:53386) 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 -- 172.21.15.129:0/1486270219 learned_addr learned my addr 172.21.15.129:0/1486270219 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570ca00640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57081f59a0 0x7f570815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570de00640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 0x7f570815e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 -- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 msgr2=0x7f570815e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 0x7f570815e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 -- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57081f59a0 msgr2=0x7f570815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57081f59a0 0x7f570815ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570d400640 1 -- 172.21.15.129:0/1486270219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570815f8c0 con 0x7f56f0004d90 2024-09-17T04:08:03.208 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.207+0000 7f570de00640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 0x7f570815e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:03.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f570d400640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f56fc011ae0 tx=0x7f56fc009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:03.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc01e070 con 0x7f56f0004d90 2024-09-17T04:08:03.209 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56fc00fe70 con 0x7f56f0004d90 2024-09-17T04:08:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc004290 con 0x7f56f0004d90 2024-09-17T04:08:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570815cff0 con 0x7f56f0004d90 2024-09-17T04:08:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.208+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570815d5b0 con 0x7f56f0004d90 2024-09-17T04:08:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.209+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57080b4950 con 0x7f56f0004d90 2024-09-17T04:08:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.210+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f56fc021070 con 0x7f56f0004d90 2024-09-17T04:08:03.211 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.211+0000 7f5705600640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 0x7f56e807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.211+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f56fc09b8d0 con 0x7f56f0004d90 2024-09-17T04:08:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.212+0000 7f570ca00640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 0x7f56e807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.212+0000 7f570ca00640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 0x7f56e807b0a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f57080cffb0 tx=0x7f56f4006ca0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:03.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.216+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f570815f8c0 con 0x7f56f0004d90 2024-09-17T04:08:03.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.431+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5708160070 con 0x7f56f0004d90 2024-09-17T04:08:03.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.432+0000 7f5705600640 1 -- 172.21.15.129:0/1486270219 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) ==== 95+0+3618 (secure 0 0 0) 0x7f56fc01be90 con 0x7f56f0004d90 2024-09-17T04:08:03.432 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:03.432 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":261,"btime":"2024-09-17T04:07:55:812131+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:07:55.812123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12249,12432]},"id":19}]} 2024-09-17T04:08:03.433 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 261 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 msgr2=0x7f56e807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 0x7f56e807b0a0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f57080cffb0 tx=0x7f56f4006ca0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 msgr2=0x7f5708157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f56fc011ae0 tx=0x7f56fc009e90 comp rx=0 tx=0).stop 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 shutdown_connections 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f56e8078b90 0x7f56e807b0a0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f570815f4b0 0x7f570815e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57081f59a0 0x7f570815ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 --2- 172.21.15.129:0/1486270219 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f56f0004d90 0x7f5708157990 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.434+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 >> 172.21.15.129:0/1486270219 conn(0x7f57080b2510 msgr2=0x7f57080cf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.435+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 shutdown_connections 2024-09-17T04:08:03.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:03.435+0000 7f570e800640 1 -- 172.21.15.129:0/1486270219 wait complete. 2024-09-17T04:08:03.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:08:03.697 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}}] 2024-09-17T04:08:03.698 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:08:03.698 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/delete_me', 1)] 2024-09-17T04:08:05.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 -- 172.21.15.129:0/1939803634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893015cd30 msgr2=0x7f89300cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 --2- 172.21.15.129:0/1939803634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893015cd30 0x7f89300cfd10 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7f893006de00 tx=0x7f892002aa30 comp rx=0 tx=0).stop 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 -- 172.21.15.129:0/1939803634 shutdown_connections 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 --2- 172.21.15.129:0/1939803634 >> v2:172.21.15.181:3300/0 conn(0x7f893006f800 0x7f89300cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 --2- 172.21.15.129:0/1939803634 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893015cd30 0x7f89300cfd10 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:05.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 -- 172.21.15.129:0/1939803634 >> 172.21.15.129:0/1939803634 conn(0x7f893006cfc0 msgr2=0x7f893006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:05.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.930+0000 7f8936200640 1 -- 172.21.15.129:0/1939803634 shutdown_connections 2024-09-17T04:08:05.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.931+0000 7f8936200640 1 -- 172.21.15.129:0/1939803634 wait complete. 2024-09-17T04:08:05.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.933+0000 7f8936200640 1 Processor -- start 2024-09-17T04:08:05.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.933+0000 7f8936200640 1 -- start start 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 0x7f89301f8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f893015cd30 0x7f89301f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8930073890 con 0x7f893006f800 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8930073710 con 0x7f893006fc10 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.934+0000 7f8936200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8930073a10 con 0x7f893015cd30 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:05.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58498/0 (socket says 172.21.15.129:58498) 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 -- 172.21.15.129:0/756892794 learned_addr learned my addr 172.21.15.129:0/756892794 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f8926a00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 0x7f89301f8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892fe00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f893015cd30 0x7f89301f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 -- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f893015cd30 msgr2=0x7f89301f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f893015cd30 0x7f89301f9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 -- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 msgr2=0x7f89301f8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 0x7f89301f8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 -- 172.21.15.129:0/756892794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89301f9860 con 0x7f893006f800 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f8926a00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 0x7f89301f8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:05.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.935+0000 7f892f400640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f892002b020 tx=0x7f8920003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.936+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920033050 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.936+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f89200044f0 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.936+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920038450 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.936+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8930141a80 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.936+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8930142040 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.937+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8930073ec0 con 0x7f893006f800 2024-09-17T04:08:05.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.938+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8920036070 con 0x7f893006f800 2024-09-17T04:08:05.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.939+0000 7f8927e00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 0x7f890c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:05.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.939+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f89200bc320 con 0x7f893006f800 2024-09-17T04:08:05.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.940+0000 7f8926a00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 0x7f890c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:05.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.941+0000 7f8926a00640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 0x7f890c07b0a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f893015c2c0 tx=0x7f8918005f10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:05.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:05.944+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f89200846f0 con 0x7f893006f800 2024-09-17T04:08:06.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.163+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f893015fa20 con 0x7f893006f800 2024-09-17T04:08:06.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.164+0000 7f8927e00640 1 -- 172.21.15.129:0/756892794 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) ==== 95+0+3619 (secure 0 0 0) 0x7f8920083ff0 con 0x7f893006f800 2024-09-17T04:08:06.164 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:06.164 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":262,"btime":"2024-09-17T04:08:03:846713+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:08:02.949764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12432,12249]},"id":19}]} 2024-09-17T04:08:06.165 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 262 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 msgr2=0x7f890c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 0x7f890c07b0a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f893015c2c0 tx=0x7f8918005f10 comp rx=0 tx=0).stop 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 msgr2=0x7f89301e52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f892002b020 tx=0x7f8920003f10 comp rx=0 tx=0).stop 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 shutdown_connections 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f890c078b90 0x7f890c07b0a0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f893015cd30 0x7f89301f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:06.167 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f893006fc10 0x7f89301f8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 --2- 172.21.15.129:0/756892794 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f893006f800 0x7f89301e52d0 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.166+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 >> 172.21.15.129:0/756892794 conn(0x7f893006cfc0 msgr2=0x7f893006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.167+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 shutdown_connections 2024-09-17T04:08:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:06.167+0000 7f8936200640 1 -- 172.21.15.129:0/756892794 wait complete. 2024-09-17T04:08:06.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-09-17T04:08:06.433 INFO:teuthology.orchestra.run.smithi129.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"2","version":"2","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"subtreetemp":0,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"66","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/delete_me","dirfrag":"0x10000000000","snapid_first":2,"projected_version":"61","version":"61","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-09-17T04:08:06.433 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1, "subtreetemp": 0, "waiter": 0 }, "projected_version": "2", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "66" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000000", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/delete_me", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "61", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "61" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-09-17T04:08:06.434 INFO:tasks.cephfs.cephfs_test_case:[('/delete_me', 1)] =?= [('/delete_me', 1)] 2024-09-17T04:08:06.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:06.434 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -rf delete_me/*) 2024-09-17T04:08:06.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json config get mds_tick_interval 2024-09-17T04:08:06.721 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_tick_interval":"5.000000"} 2024-09-17T04:08:06.721 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_tick_interval": "5.000000" } 2024-09-17T04:08:06.721 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:08:06.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:08:06.819 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:08:06.915 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:08:06.915 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:08:06.915 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:06.915 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:06.951 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:06.951 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:07.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:08:07.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 -- 172.21.15.129:0/3521235005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd415fc30 msgr2=0x7f6bd4160320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:07.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/3521235005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd415fc30 0x7f6bd4160320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4005fd0 tx=0x7f6bc402a540 comp rx=0 tx=0).stop 2024-09-17T04:08:07.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 -- 172.21.15.129:0/3521235005 shutdown_connections 2024-09-17T04:08:07.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/3521235005 >> v2:172.21.15.181:3300/0 conn(0x7f6bd40cc0c0 0x7f6bd406f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/3521235005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd415fc30 0x7f6bd4160320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 -- 172.21.15.129:0/3521235005 >> 172.21.15.129:0/3521235005 conn(0x7f6bd40b2510 msgr2=0x7f6bd40d0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:07.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 -- 172.21.15.129:0/3521235005 shutdown_connections 2024-09-17T04:08:07.290 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.288+0000 7f6bd3e00640 1 -- 172.21.15.129:0/3521235005 wait complete. 2024-09-17T04:08:07.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.291+0000 7f6bd3e00640 1 Processor -- start 2024-09-17T04:08:07.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.291+0000 7f6bd3e00640 1 -- start start 2024-09-17T04:08:07.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:07.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bd415fc30 0x7f6bd40c7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:07.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 0x7f6bd40c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:07.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd406f8d0 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd406f750 con 0x7f6bd415fc30 2024-09-17T04:08:07.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd3e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd406fa50 con 0x7f6bd4160860 2024-09-17T04:08:07.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:07.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58508/0 (socket says 172.21.15.129:58508) 2024-09-17T04:08:07.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.292+0000 7f6bd2a00640 1 -- 172.21.15.129:0/2599826333 learned_addr learned my addr 172.21.15.129:0/2599826333 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd3400640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 0x7f6bd40c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2000640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bd415fc30 0x7f6bd40c7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 -- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 msgr2=0x7f6bd40c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 0x7f6bd40c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 -- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bd415fc30 msgr2=0x7f6bd40c7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:07.297 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bd415fc30 0x7f6bd40c7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 -- 172.21.15.129:0/2599826333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd40c5100 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd3400640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 0x7f6bd40c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:07.298 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bd2a00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4034040 tx=0x7f6bc40038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:07.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.293+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc403e030 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.294+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bd40c53e0 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.294+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bc4003ca0 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.294+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4035040 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.294+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bd40c8cf0 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.295+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bd40c5620 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.296+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6bc4041030 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.301 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.297+0000 7f6bcaa00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 0x7f6bb407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:07.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.297+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f6bc40bb430 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.298+0000 7f6bd2000640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 0x7f6bb407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:07.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.298+0000 7f6bd2000640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 0x7f6bb407b0d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40c67b0 tx=0x7f6bc0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:07.302 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.301+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6bd40c5620 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.494+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6bd40c8f50 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.494+0000 7f6bcaa00640 1 -- 172.21.15.129:0/2599826333 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v263) ==== 80+0+124 (secure 0 0 0) 0x7f6bc40830a0 con 0x7f6bd40cc0c0 2024-09-17T04:08:07.495 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6834/1080794990 2024-09-18T04:07:50.781831+0000 2024-09-17T04:08:07.495 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129:6832/1080794990 2024-09-18T04:07:50.781831+0000 2024-09-17T04:08:07.495 INFO:teuthology.orchestra.run.smithi129.stderr:listed 2 entries 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.496+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 msgr2=0x7f6bb407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.496+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 0x7f6bb407b0d0 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40c67b0 tx=0x7f6bc0009040 comp rx=0 tx=0).stop 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.496+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 msgr2=0x7f6bd4159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4034040 tx=0x7f6bc40038a0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 shutdown_connections 2024-09-17T04:08:07.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6bb4078bc0 0x7f6bb407b0d0 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6bd4160860 0x7f6bd40c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6bd415fc30 0x7f6bd40c7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 --2- 172.21.15.129:0/2599826333 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6bd40cc0c0 0x7f6bd4159360 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 >> 172.21.15.129:0/2599826333 conn(0x7f6bd40b2510 msgr2=0x7f6bd4073240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 shutdown_connections 2024-09-17T04:08:07.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:07.497+0000 7f6bd3e00640 1 -- 172.21.15.129:0/2599826333 wait complete. 2024-09-17T04:08:07.518 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:08:07.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:07.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:07.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:08:07.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:07.623 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:08:07.680 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:08:07.681 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:08:07.681 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:07.717 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:08:07.718 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:08:07.718 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:07.718 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:07.753 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:08:07.753 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:08:07.753 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:08:07.754 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:07.793 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:08:07.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 -- 172.21.15.129:0/4062269607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 msgr2=0x7fcccc0cfd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 --2- 172.21.15.129:0/4062269607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc0cfd10 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fcccc06de00 tx=0x7fccbc02a4a0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 -- 172.21.15.129:0/4062269607 shutdown_connections 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 --2- 172.21.15.129:0/4062269607 >> v2:172.21.15.181:3300/0 conn(0x7fcccc06f800 0x7fcccc0cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 --2- 172.21.15.129:0/4062269607 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc0cfd10 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 -- 172.21.15.129:0/4062269607 >> 172.21.15.129:0/4062269607 conn(0x7fcccc06cfc0 msgr2=0x7fcccc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 -- 172.21.15.129:0/4062269607 shutdown_connections 2024-09-17T04:08:08.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.016+0000 7fccd3200640 1 -- 172.21.15.129:0/4062269607 wait complete. 2024-09-17T04:08:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.019+0000 7fccd3200640 1 Processor -- start 2024-09-17T04:08:08.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.019+0000 7fccd3200640 1 -- start start 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc06f800 0x7fcccc1f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcccc06fc10 0x7fcccc1f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc073890 con 0x7fcccc15cd30 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcccc073710 con 0x7fcccc06f800 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd3200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcccc073a10 con 0x7fcccc06fc10 2024-09-17T04:08:08.021 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd1200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58532/0 (socket says 172.21.15.129:58532) 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fccd1200640 1 -- 172.21.15.129:0/1173614088 learned_addr learned my addr 172.21.15.129:0/1173614088 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.020+0000 7fcccbe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc06f800 0x7fcccc1f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccc3e00640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcccc06fc10 0x7fcccc1f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 -- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcccc06fc10 msgr2=0x7fcccc1f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcccc06fc10 0x7fcccc1f0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 -- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc06f800 msgr2=0x7fcccc1f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc06f800 0x7fcccc1f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 -- 172.21.15.129:0/1173614088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcccc1f4da0 con 0x7fcccc15cd30 2024-09-17T04:08:08.022 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccd1200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c760 tx=0x7fccc400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.021+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017070 con 0x7fcccc15cd30 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.022+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fccc400ce10 con 0x7fcccc15cd30 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.022+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcccc1f52b0 con 0x7fcccc15cd30 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.022+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc40052b0 con 0x7fcccc15cd30 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.022+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcccc1f5870 con 0x7fcccc15cd30 2024-09-17T04:08:08.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.023+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcccc1f5020 con 0x7fcccc15cd30 2024-09-17T04:08:08.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.024+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fccc401a070 con 0x7fcccc15cd30 2024-09-17T04:08:08.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.025+0000 7fccc8c00640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 0x7fcca807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.026+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fccc409a250 con 0x7fcccc15cd30 2024-09-17T04:08:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.026+0000 7fcccbe00640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 0x7fcca807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.026+0000 7fcccbe00640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 0x7fcca807b0a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fccbc034000 tx=0x7fccbc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.030+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fccc4062060 con 0x7fcccc15cd30 2024-09-17T04:08:08.246 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.246+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcccc0cf500 con 0x7fcccc15cd30 2024-09-17T04:08:08.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.247+0000 7fccc8c00640 1 -- 172.21.15.129:0/1173614088 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) ==== 95+0+3619 (secure 0 0 0) 0x7fcccc0cf500 con 0x7fcccc15cd30 2024-09-17T04:08:08.247 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:08.247 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":262,"btime":"2024-09-17T04:08:03:846713+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:08:02.949764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12432,12249]},"id":19}]} 2024-09-17T04:08:08.248 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 262 2024-09-17T04:08:08.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 msgr2=0x7fcca807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 0x7fcca807b0a0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fccbc034000 tx=0x7fccbc004730 comp rx=0 tx=0).stop 2024-09-17T04:08:08.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 msgr2=0x7fcccc1f12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 secure :-1 s=READY pgs=2858 cs=0 l=1 rev1=1 crypto rx=0x7fccc400c760 tx=0x7fccc400cc20 comp rx=0 tx=0).stop 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 shutdown_connections 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcca8078b90 0x7fcca807b0a0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcccc15cd30 0x7fcccc1f12e0 unknown :-1 s=CLOSED pgs=2858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcccc06fc10 0x7fcccc1f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 --2- 172.21.15.129:0/1173614088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcccc06f800 0x7fcccc1f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 >> 172.21.15.129:0/1173614088 conn(0x7fcccc06cfc0 msgr2=0x7fcccc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 shutdown_connections 2024-09-17T04:08:08.250 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.249+0000 7fccd3200640 1 -- 172.21.15.129:0/1173614088 wait complete. 2024-09-17T04:08:08.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:08.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 -- 172.21.15.129:0/1098962541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 msgr2=0x7f21f40cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 --2- 172.21.15.129:0/1098962541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f40cfb30 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f21f4153d90 tx=0x7f21e402a560 comp rx=0 tx=0).stop 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 -- 172.21.15.129:0/1098962541 shutdown_connections 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 --2- 172.21.15.129:0/1098962541 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40d06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 --2- 172.21.15.129:0/1098962541 >> v2:172.21.15.181:3300/0 conn(0x7f21f406f800 0x7f21f40cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 --2- 172.21.15.129:0/1098962541 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f40cfb30 unknown :-1 s=CLOSED pgs=2859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 -- 172.21.15.129:0/1098962541 >> 172.21.15.129:0/1098962541 conn(0x7f21f406cfc0 msgr2=0x7f21f406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 -- 172.21.15.129:0/1098962541 shutdown_connections 2024-09-17T04:08:08.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.499+0000 7f21fac00640 1 -- 172.21.15.129:0/1098962541 wait complete. 2024-09-17T04:08:08.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.501+0000 7f21fac00640 1 Processor -- start 2024-09-17T04:08:08.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.501+0000 7f21fac00640 1 -- start start 2024-09-17T04:08:08.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21f406f800 0x7f21f4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f21f4073710 con 0x7f21f415cd30 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f21f40d0f00 con 0x7f21f406f800 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21fac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f21f4073890 con 0x7f21f406fc10 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21f8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21f8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58548/0 (socket says 172.21.15.129:58548) 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21f8c00640 1 -- 172.21.15.129:0/3448346453 learned_addr learned my addr 172.21.15.129:0/3448346453 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21f3e00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21f406f800 0x7f21f4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.503+0000 7f21eb400640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 -- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 msgr2=0x7f21f40ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 -- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21f406f800 msgr2=0x7f21f4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21f406f800 0x7f21f4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 -- 172.21.15.129:0/3448346453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f40af670 con 0x7f21f415cd30 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21eb400640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f8c00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00caa0 tx=0x7f21ec00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00adf0 con 0x7f21f415cd30 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f21ec00f540 con 0x7f21f415cd30 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f40af950 con 0x7f21f415cd30 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec00f990 con 0x7f21f415cd30 2024-09-17T04:08:08.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.504+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f4155160 con 0x7f21f415cd30 2024-09-17T04:08:08.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.506+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f4073db0 con 0x7f21f415cd30 2024-09-17T04:08:08.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.507+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f21ec00fb30 con 0x7f21f415cd30 2024-09-17T04:08:08.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.508+0000 7f21f0c00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 0x7f21d007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.509+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f21ec09b820 con 0x7f21f415cd30 2024-09-17T04:08:08.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.509+0000 7f21f3e00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 0x7f21d007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.509+0000 7f21f3e00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 0x7f21d007b0d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f21e4033000 tx=0x7f21e4004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.513+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f21ec014030 con 0x7f21f415cd30 2024-09-17T04:08:08.729 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.729+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21f406d570 con 0x7f21f415cd30 2024-09-17T04:08:08.730 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.730+0000 7f21f0c00640 1 -- 172.21.15.129:0/3448346453 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) ==== 95+0+3619 (secure 0 0 0) 0x7f21ec00fe20 con 0x7f21f415cd30 2024-09-17T04:08:08.730 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:08.730 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":262,"btime":"2024-09-17T04:08:03:846713+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:08:02.949764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12432,12249]},"id":19}]} 2024-09-17T04:08:08.731 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 262 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 msgr2=0x7f21d007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 0x7f21d007b0d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f21e4033000 tx=0x7f21e4004730 comp rx=0 tx=0).stop 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 msgr2=0x7f21f4146310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f21ec00caa0 tx=0x7f21ec00cf60 comp rx=0 tx=0).stop 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 shutdown_connections 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f21d0078bc0 0x7f21d007b0d0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f21f415cd30 0x7f21f4146310 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f21f406fc10 0x7f21f40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.733 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 --2- 172.21.15.129:0/3448346453 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f21f406f800 0x7f21f4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.732+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 >> 172.21.15.129:0/3448346453 conn(0x7f21f406cfc0 msgr2=0x7f21f415b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:08.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.733+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 shutdown_connections 2024-09-17T04:08:08.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.733+0000 7f21fac00640 1 -- 172.21.15.129:0/3448346453 wait complete. 2024-09-17T04:08:08.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 -- 172.21.15.129:0/3802863602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 msgr2=0x7f9e70003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 --2- 172.21.15.129:0/3802863602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e70003ee0 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009b50 tx=0x7f9e70005830 comp rx=0 tx=0).stop 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 -- 172.21.15.129:0/3802863602 shutdown_connections 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 --2- 172.21.15.129:0/3802863602 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e70003ee0 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 -- 172.21.15.129:0/3802863602 >> 172.21.15.129:0/3802863602 conn(0x7f9e8c06cfc0 msgr2=0x7f9e8c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 -- 172.21.15.129:0/3802863602 shutdown_connections 2024-09-17T04:08:08.979 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.978+0000 7f9e93000640 1 -- 172.21.15.129:0/3802863602 wait complete. 2024-09-17T04:08:08.980 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.980+0000 7f9e93000640 1 Processor -- start 2024-09-17T04:08:08.981 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.981+0000 7f9e93000640 1 -- start start 2024-09-17T04:08:08.983 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8c157ed0 0x7f9e8c14ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 0x7f9e8c14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c0d0af0 con 0x7f9e70003a50 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8c0d0190 con 0x7f9e8c157ed0 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e93000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9e8c0d0c70 con 0x7f9e8c14dc20 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e8be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e8be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58570/0 (socket says 172.21.15.129:58570) 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.982+0000 7f9e8be00640 1 -- 172.21.15.129:0/2642593138 learned_addr learned my addr 172.21.15.129:0/2642593138 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e83400640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8c157ed0 0x7f9e8c14ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.984 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e91000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 0x7f9e8c14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 -- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 msgr2=0x7f9e8c14ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 0x7f9e8c14ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 -- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8c157ed0 msgr2=0x7f9e8c14ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8c157ed0 0x7f9e8c14ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 -- 172.21.15.129:0/2642593138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8c14e3a0 con 0x7f9e70003a50 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e91000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 0x7f9e8c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e8be00640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009a70 tx=0x7f9e7c033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c038040 con 0x7f9e70003a50 2024-09-17T04:08:08.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9e7c033970 con 0x7f9e70003a50 2024-09-17T04:08:08.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c033dc0 con 0x7f9e70003a50 2024-09-17T04:08:08.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.983+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e8c154f90 con 0x7f9e70003a50 2024-09-17T04:08:08.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.984+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e8c155550 con 0x7f9e70003a50 2024-09-17T04:08:08.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.985+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e8c15aa60 con 0x7f9e70003a50 2024-09-17T04:08:08.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.986+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9e7c043020 con 0x7f9e70003a50 2024-09-17T04:08:08.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.987+0000 7f9e88c00640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:08.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.987+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f9e7c0baf30 con 0x7f9e70003a50 2024-09-17T04:08:08.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.987+0000 7f9e83400640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:08.988 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.988+0000 7f9e83400640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c151e60 tx=0x7f9e74006b40 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:08.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:08.991+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9e7c04a020 con 0x7f9e70003a50 2024-09-17T04:08:09.177 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.177+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e8c06d6b0 con 0x7f9e70003a50 2024-09-17T04:08:09.178 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.178+0000 7f9e88c00640 1 -- 172.21.15.129:0/2642593138 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) ==== 74+0+16922 (secure 0 0 0) 0x7f9e7c083280 con 0x7f9e70003a50 2024-09-17T04:08:09.178 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:09.179 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":263,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:50.781862+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:07:30.379067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":39,"pool_name":"cephfs_data","create_time":"2024-09-17T04:07:31.426899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"260","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/1080794990":"2024-09-18T04:07:50.781831+0000","172.21.15.129:6832/1080794990":"2024-09-18T04:07:50.781831+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 msgr2=0x7f9e6807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c151e60 tx=0x7f9e74006b40 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 msgr2=0x7f9e8c145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c009a70 tx=0x7f9e7c033670 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 shutdown_connections 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9e68078b90 0x7f9e6807b0a0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9e8c14dc20 0x7f9e8c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9e8c157ed0 0x7f9e8c14ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 --2- 172.21.15.129:0/2642593138 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9e70003a50 0x7f9e8c145c00 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.181 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.180+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 >> 172.21.15.129:0/2642593138 conn(0x7f9e8c06cfc0 msgr2=0x7f9e8c15c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:09.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.181+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 shutdown_connections 2024-09-17T04:08:09.182 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.181+0000 7f9e93000640 1 -- 172.21.15.129:0/2642593138 wait complete. 2024-09-17T04:08:09.203 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:08:09.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 -- 172.21.15.129:0/1120385589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0d0dd0 msgr2=0x7fe77c0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 --2- 172.21.15.129:0/1120385589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0d0dd0 0x7fe77c0cdb90 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7fe77c072c00 tx=0x7fe77402a4d0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 -- 172.21.15.129:0/1120385589 shutdown_connections 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 --2- 172.21.15.129:0/1120385589 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0d0dd0 0x7fe77c0cdb90 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 --2- 172.21.15.129:0/1120385589 >> v2:172.21.15.181:3300/0 conn(0x7fe77c15a460 0x7fe77c06fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 -- 172.21.15.129:0/1120385589 >> 172.21.15.129:0/1120385589 conn(0x7fe77c06cfc0 msgr2=0x7fe77c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 -- 172.21.15.129:0/1120385589 shutdown_connections 2024-09-17T04:08:09.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.425+0000 7fe781400640 1 -- 172.21.15.129:0/1120385589 wait complete. 2024-09-17T04:08:09.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.427+0000 7fe781400640 1 Processor -- start 2024-09-17T04:08:09.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.428+0000 7fe781400640 1 -- start start 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 0x7fe77c159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 0x7fe77c1f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c15d560 con 0x7fe77c0cd780 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe77c15d3e0 con 0x7fe77c15a460 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe781400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe77c15d6e0 con 0x7fe77c0d0dd0 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe77a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe77a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58574/0 (socket says 172.21.15.129:58574) 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.429+0000 7fe77a800640 1 -- 172.21.15.129:0/1342042523 learned_addr learned my addr 172.21.15.129:0/1342042523 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77b200640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 0x7fe77c1f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe779e00640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 0x7fe77c159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 -- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 msgr2=0x7fe77c159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 0x7fe77c159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 -- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 msgr2=0x7fe77c1f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 0x7fe77c1f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 -- 172.21.15.129:0/1342042523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe77c0b1150 con 0x7fe77c0cd780 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe779e00640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 0x7fe77c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77b200640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 0x7fe77c1f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe77a800640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00ccb0 tx=0x7fe76c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c003970 con 0x7fe77c0cd780 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe76c003c00 con 0x7fe77c0cd780 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76c004030 con 0x7fe77c0cd780 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.430+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe77c0b1430 con 0x7fe77c0cd780 2024-09-17T04:08:09.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.431+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe77c159640 con 0x7fe77c0cd780 2024-09-17T04:08:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.432+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe77c15db90 con 0x7fe77c0cd780 2024-09-17T04:08:09.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.433+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe76c0041d0 con 0x7fe77c0cd780 2024-09-17T04:08:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.434+0000 7fe772a00640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 0x7fe75c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.434+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7fe76c09b6b0 con 0x7fe77c0cd780 2024-09-17T04:08:09.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.434+0000 7fe779e00640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 0x7fe75c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.435+0000 7fe779e00640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 0x7fe75c07b0a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b2530 tx=0x7fe768008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:09.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.438+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe76c014070 con 0x7fe77c0cd780 2024-09-17T04:08:09.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.658+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe77c1598a0 con 0x7fe77c0cd780 2024-09-17T04:08:09.659 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.659+0000 7fe772a00640 1 -- 172.21.15.129:0/1342042523 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) ==== 95+0+3619 (secure 0 0 0) 0x7fe76c063a80 con 0x7fe77c0cd780 2024-09-17T04:08:09.659 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:09.659 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":262,"btime":"2024-09-17T04:08:03:846713+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:08:02.949764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12432,12249]},"id":19}]} 2024-09-17T04:08:09.660 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 262 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.661+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 msgr2=0x7fe75c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.661+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 0x7fe75c07b0a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fe77c0b2530 tx=0x7fe768008040 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.661+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 msgr2=0x7fe77c158bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.661+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7fe76c00ccb0 tx=0x7fe76c00a620 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 shutdown_connections 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe75c078b90 0x7fe75c07b0a0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe77c15a460 0x7fe77c1f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe77c0d0dd0 0x7fe77c159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 --2- 172.21.15.129:0/1342042523 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe77c0cd780 0x7fe77c158bc0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.662 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 >> 172.21.15.129:0/1342042523 conn(0x7fe77c06cfc0 msgr2=0x7fe77c0b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 shutdown_connections 2024-09-17T04:08:09.663 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.662+0000 7fe781400640 1 -- 172.21.15.129:0/1342042523 wait complete. 2024-09-17T04:08:09.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:09.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 -- 172.21.15.129:0/3087084814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a90003a00 msgr2=0x7f9a90003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 --2- 172.21.15.129:0/3087084814 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9a90003a00 0x7f9a90003e70 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009b40 tx=0x7f9a90005830 comp rx=0 tx=0).stop 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aaa800640 1 -- 172.21.15.129:0/3087084814 reap_dead start 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 -- 172.21.15.129:0/3087084814 shutdown_connections 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 -- 172.21.15.129:0/3087084814 >> 172.21.15.129:0/3087084814 conn(0x7f9aac08bfb0 msgr2=0x7f9aac09d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 -- 172.21.15.129:0/3087084814 shutdown_connections 2024-09-17T04:08:09.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.907+0000 7f9aab200640 1 -- 172.21.15.129:0/3087084814 wait complete. 2024-09-17T04:08:09.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.909+0000 7f9aab200640 1 Processor -- start 2024-09-17T04:08:09.909 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.909+0000 7f9aab200640 1 -- start start 2024-09-17T04:08:09.912 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a90003a00 0x7f9aac160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 0x7f9aac160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac162370 con 0x7f9aac155bc0 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac1621f0 con 0x7f9a90003a00 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aab200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9aac0b4390 con 0x7f9aac14dca0 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aaa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aaa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58584/0 (socket says 172.21.15.129:58584) 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aaa800640 1 -- 172.21.15.129:0/3084171068 learned_addr learned my addr 172.21.15.129:0/3084171068 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aa9e00640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a90003a00 0x7f9aac160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.911+0000 7f9aa1400640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 0x7f9aac160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 -- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 msgr2=0x7f9aac160bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 0x7f9aac160bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 -- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a90003a00 msgr2=0x7f9aac160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a90003a00 0x7f9aac160680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 -- 172.21.15.129:0/3084171068 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac0b6b20 con 0x7f9aac155bc0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aa1400640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 0x7f9aac160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aaa800640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0ceb60 tx=0x7f9aa400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4003970 con 0x7f9aac155bc0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9aa4003b10 con 0x7f9aac155bc0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4004030 con 0x7f9aac155bc0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac0ccdd0 con 0x7f9aac155bc0 2024-09-17T04:08:09.914 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.912+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac0b40d0 con 0x7f9aac155bc0 2024-09-17T04:08:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.913+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac08c3c0 con 0x7f9aac155bc0 2024-09-17T04:08:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.915+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9aa40041d0 con 0x7f9aac155bc0 2024-09-17T04:08:09.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.916+0000 7f9aa2a00640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 0x7f9a8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:09.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.916+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f9aa409b0a0 con 0x7f9aac155bc0 2024-09-17T04:08:09.916 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.916+0000 7f9aa9e00640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 0x7f9a8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:09.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.917+0000 7f9aa9e00640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 0x7f9a8807b0a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009b40 tx=0x7f9a9c02d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:09.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:09.920+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9aac08c3c0 con 0x7f9aac155bc0 2024-09-17T04:08:10.138 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.138+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9aac0b9c20 con 0x7f9aac155bc0 2024-09-17T04:08:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.139+0000 7f9aa2a00640 1 -- 172.21.15.129:0/3084171068 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) ==== 95+0+3619 (secure 0 0 0) 0x7f9aac0b9c20 con 0x7f9aac155bc0 2024-09-17T04:08:10.139 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:10.139 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":262,"btime":"2024-09-17T04:08:03:846713+0000","default_fscid":19,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:07:32.455412+0000","modified":"2024-09-17T04:08:02.949764+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":263,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":2,"in":[0,1],"up":{"mds_0":12249,"mds_1":12432},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12249":{"gid":12249,"name":"a","rank":0,"incarnation":245,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/3848773989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":3848773989},{"type":"v1","addr":"172.21.15.129:6835","nonce":3848773989}]},"join_fscid":-1,"export_targets":[1],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_12432":{"gid":12432,"name":"c","rank":1,"incarnation":257,"state":"up:active","state_seq":5,"addr":"172.21.15.129:6834/3076551424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":3076551424},{"type":"v1","addr":"172.21.15.129:6834","nonce":3076551424}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[39],"metadata_pool":38,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12249,"qdb_cluster":[12432,12249]},"id":19}]} 2024-09-17T04:08:10.139 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 262 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.141+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 msgr2=0x7f9a8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.141+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 0x7f9a8807b0a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c009b40 tx=0x7f9a9c02d040 comp rx=0 tx=0).stop 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.141+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 msgr2=0x7f9aac0cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.141+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f9aac0ceb60 tx=0x7f9aa400f3c0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 shutdown_connections 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9a88078b90 0x7f9a8807b0a0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9aac155bc0 0x7f9aac0cd5d0 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9aac14dca0 0x7f9aac160bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.142 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 --2- 172.21.15.129:0/3084171068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a90003a00 0x7f9aac160680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 >> 172.21.15.129:0/3084171068 conn(0x7f9aac08bfb0 msgr2=0x7f9aac0d4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:10.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 shutdown_connections 2024-09-17T04:08:10.143 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.142+0000 7f9aab200640 1 -- 172.21.15.129:0/3084171068 wait complete. 2024-09-17T04:08:10.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.388+0000 7f8f11000640 1 -- 172.21.15.129:0/984487784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef0005150 msgr2=0x7f8ef000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.388+0000 7f8f11000640 1 --2- 172.21.15.129:0/984487784 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8ef0005150 0x7f8ef000b9c0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f8f04002230 tx=0x7f8ef0005830 comp rx=0 tx=0).stop 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.388+0000 7f8f0a800640 1 -- 172.21.15.129:0/984487784 reap_dead start 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.388+0000 7f8f11000640 1 -- 172.21.15.129:0/984487784 shutdown_connections 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.388+0000 7f8f11000640 1 -- 172.21.15.129:0/984487784 >> 172.21.15.129:0/984487784 conn(0x7f8f0c06cfc0 msgr2=0x7f8f0c06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.389+0000 7f8f11000640 1 -- 172.21.15.129:0/984487784 shutdown_connections 2024-09-17T04:08:10.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.389+0000 7f8f11000640 1 -- 172.21.15.129:0/984487784 wait complete. 2024-09-17T04:08:10.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.391+0000 7f8f11000640 1 Processor -- start 2024-09-17T04:08:10.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.391+0000 7f8f11000640 1 -- start start 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c151880 0x7f8f0c14fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f0c1e6040 0x7f8f0c1e6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c0d00a0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0c0cff20 con 0x7f8f0c151880 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.392+0000 7f8f11000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0c073710 con 0x7f8f0c1e6040 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f09e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c151880 0x7f8f0c14fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58612/0 (socket says 172.21.15.129:58612) 2024-09-17T04:08:10.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f09400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f0c1e6040 0x7f8f0c1e6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 -- 172.21.15.129:0/1964380039 learned_addr learned my addr 172.21.15.129:0/1964380039 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 -- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f0c1e6040 msgr2=0x7f8f0c1e6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f0c1e6040 0x7f8f0c1e6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 -- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c151880 msgr2=0x7f8f0c14fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c151880 0x7f8f0c14fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.393+0000 7f8f0a800640 1 -- 172.21.15.129:0/1964380039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0c1e6990 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f0a800640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f8f0402b490 tx=0x7f8f04001aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04039070 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8f040066a0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f04002060 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0c151df0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.394+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f0c1523b0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.395+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f0c15c5f0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.396+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8f0403c020 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.397+0000 7f8f02000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 0x7f8eec07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.397+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f8f040b9f90 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.398+0000 7f8f09e00640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 0x7f8eec07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.398+0000 7f8f09e00640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 0x7f8eec07b0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c1537e0 tx=0x7f8efc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:10.402 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.402+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f8f0c15c5f0 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.585+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f0c073c70 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.587+0000 7f8f02000640 1 -- 172.21.15.129:0/1964380039 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v263) ==== 74+0+16922 (secure 0 0 0) 0x7f8f04045030 con 0x7f8f0c13f9d0 2024-09-17T04:08:10.587 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:10.587 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":263,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:07:50.781862+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":38,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:07:30.379067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"259","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":39,"pool_name":"cephfs_data","create_time":"2024-09-17T04:07:31.426899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"260","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6834/1080794990":"2024-09-18T04:07:50.781831+0000","172.21.15.129:6832/1080794990":"2024-09-18T04:07:50.781831+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:10.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 msgr2=0x7f8eec07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 0x7f8eec07b0a0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c1537e0 tx=0x7f8efc005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:10.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 msgr2=0x7f8f0c13fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.590 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f8f0402b490 tx=0x7f8f04001aa0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 shutdown_connections 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8eec078b90 0x7f8eec07b0a0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8f0c13f9d0 0x7f8f0c13fde0 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8f0c1e6040 0x7f8f0c1e6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 --2- 172.21.15.129:0/1964380039 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f0c151880 0x7f8f0c14fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.589+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 >> 172.21.15.129:0/1964380039 conn(0x7f8f0c06cfc0 msgr2=0x7f8f0c070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.590+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 shutdown_connections 2024-09-17T04:08:10.591 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.590+0000 7f8f11000640 1 -- 172.21.15.129:0/1964380039 wait complete. 2024-09-17T04:08:10.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:08:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 -- 172.21.15.129:0/1868055890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524006fc10 msgr2=0x7f52400d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 --2- 172.21.15.129:0/1868055890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524006fc10 0x7f52400d0620 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5238005fd0 tx=0x7f523802a8b0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 -- 172.21.15.129:0/1868055890 shutdown_connections 2024-09-17T04:08:10.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 --2- 172.21.15.129:0/1868055890 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524006fc10 0x7f52400d0620 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 --2- 172.21.15.129:0/1868055890 >> v2:172.21.15.181:3300/0 conn(0x7f524015cd30 0x7f52400cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 -- 172.21.15.129:0/1868055890 >> 172.21.15.129:0/1868055890 conn(0x7f524006cfc0 msgr2=0x7f524006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 -- 172.21.15.129:0/1868055890 shutdown_connections 2024-09-17T04:08:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.839+0000 7f5244c00640 1 -- 172.21.15.129:0/1868055890 wait complete. 2024-09-17T04:08:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.841+0000 7f5244c00640 1 Processor -- start 2024-09-17T04:08:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.841+0000 7f5244c00640 1 -- start start 2024-09-17T04:08:10.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f524006fc10 0x7f52401f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524015cd30 0x7f52401f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5240073890 con 0x7f524006f800 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5240073710 con 0x7f524006fc10 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f5244c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5240073a10 con 0x7f524015cd30 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58618/0 (socket says 172.21.15.129:58618) 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 -- 172.21.15.129:0/2808263762 learned_addr learned my addr 172.21.15.129:0/2808263762 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 -- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524015cd30 msgr2=0x7f52401f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524015cd30 0x7f52401f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 -- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f524006fc10 msgr2=0x7f52401f76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.843+0000 7f523de00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f524006fc10 0x7f52401f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f523de00640 1 -- 172.21.15.129:0/2808263762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52400b22e0 con 0x7f524006f800 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f523de00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9f0 tx=0x7f523000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000cd20 con 0x7f524006f800 2024-09-17T04:08:10.845 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5230004020 con 0x7f524006f800 2024-09-17T04:08:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230004470 con 0x7f524006f800 2024-09-17T04:08:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52400b25c0 con 0x7f524006f800 2024-09-17T04:08:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.844+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401f71a0 con 0x7f524006f800 2024-09-17T04:08:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.845+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5240073ec0 con 0x7f524006f800 2024-09-17T04:08:10.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.847+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f52300035f0 con 0x7f524006f800 2024-09-17T04:08:10.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.848+0000 7f5236a00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 0x7f521c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:10.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.848+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(263..263 src has 1..263) ==== 5795+0+0 (secure 0 0 0) 0x7f523009b8a0 con 0x7f524006f800 2024-09-17T04:08:10.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.848+0000 7f5235400640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 0x7f521c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:10.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.849+0000 7f5235400640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 0x7f521c07b0a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f524015e320 tx=0x7f52280079f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:10.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:10.852+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5230063c90 con 0x7f524006f800 2024-09-17T04:08:11.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.071+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f52400cdd40 con 0x7f524006f800 2024-09-17T04:08:11.072 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:08:11.072+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:08:11.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.074+0000 7f5236a00640 1 -- 172.21.15.129:0/2808263762 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v263) ==== 161+0+0 (secure 0 0 0) 0x7f5230063590 con 0x7f524006f800 2024-09-17T04:08:11.074 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:08:11.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.078+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 msgr2=0x7f521c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:11.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.078+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 0x7f521c07b0a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f524015e320 tx=0x7f52280079f0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.078+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 msgr2=0x7f52400b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.078+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9f0 tx=0x7f523000eeb0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 shutdown_connections 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f521c078b90 0x7f521c07b0a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f524015cd30 0x7f52401f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f524006fc10 0x7f52401f76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 --2- 172.21.15.129:0/2808263762 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f524006f800 0x7f52400b18b0 unknown :-1 s=CLOSED pgs=2872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 >> 172.21.15.129:0/2808263762 conn(0x7f524006cfc0 msgr2=0x7f524015f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:11.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 shutdown_connections 2024-09-17T04:08:11.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.079+0000 7f5244c00640 1 -- 172.21.15.129:0/2808263762 wait complete. 2024-09-17T04:08:11.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:08:11.218 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:08:11.218 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:08:11.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 -- 172.21.15.129:0/3470877984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c0160940 msgr2=0x7f92c00ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 --2- 172.21.15.129:0/3470877984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c0160940 0x7f92c00ad4d0 secure :-1 s=READY pgs=2876 cs=0 l=1 rev1=1 crypto rx=0x7f92c0071140 tx=0x7f92bc02aa30 comp rx=0 tx=0).stop 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 -- 172.21.15.129:0/3470877984 shutdown_connections 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 --2- 172.21.15.129:0/3470877984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c0160940 0x7f92c00ad4d0 unknown :-1 s=CLOSED pgs=2876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 --2- 172.21.15.129:0/3470877984 >> v2:172.21.15.181:3300/0 conn(0x7f92c0156d60 0x7f92c00cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 -- 172.21.15.129:0/3470877984 >> 172.21.15.129:0/3470877984 conn(0x7f92c00b25f0 msgr2=0x7f92c00d01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 -- 172.21.15.129:0/3470877984 shutdown_connections 2024-09-17T04:08:11.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.338+0000 7f92c7200640 1 -- 172.21.15.129:0/3470877984 wait complete. 2024-09-17T04:08:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.340+0000 7f92c7200640 1 Processor -- start 2024-09-17T04:08:11.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.341+0000 7f92c7200640 1 -- start start 2024-09-17T04:08:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 0x7f92c0152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 0x7f92c009dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f92c00adeb0 con 0x7f92c00af690 2024-09-17T04:08:11.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f92c00add30 con 0x7f92c01605f0 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c7200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f92c00ae030 con 0x7f92c0156d60 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58674/0 (socket says 172.21.15.129:58674) 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.342+0000 7f92c5e00640 1 -- 172.21.15.129:0/2910108373 learned_addr learned my addr 172.21.15.129:0/2910108373 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5400640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 0x7f92c0152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c6800640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 0x7f92c009dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 -- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 msgr2=0x7f92c0152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 0x7f92c0152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 -- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 msgr2=0x7f92c009dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 0x7f92c009dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:11.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 -- 172.21.15.129:0/2910108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c01f6aa0 con 0x7f92c00af690 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c6800640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 0x7f92c009dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5400640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 0x7f92c0152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c5e00640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f92b400e7f0 tx=0x7f92b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b400ccc0 con 0x7f92c00af690 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f92b4010430 con 0x7f92c00af690 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c015d930 con 0x7f92c00af690 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4010860 con 0x7f92c00af690 2024-09-17T04:08:11.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.343+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c009cfd0 con 0x7f92c00af690 2024-09-17T04:08:11.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.344+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c00ae4e0 con 0x7f92c00af690 2024-09-17T04:08:11.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.346+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f92b4010a00 con 0x7f92c00af690 2024-09-17T04:08:11.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.347+0000 7f92ba000640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 0x7f92a407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.347+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 5967+0+0 (secure 0 0 0) 0x7f92b409b8a0 con 0x7f92c00af690 2024-09-17T04:08:11.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.347+0000 7f92c5400640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 0x7f92a407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:11.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.348+0000 7f92c5400640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 0x7f92a407b0d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f92c01602e0 tx=0x7f92b0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:11.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.351+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f92c01f6aa0 con 0x7f92c00af690 2024-09-17T04:08:11.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:11.569+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f92c0160eb0 con 0x7f92c00af690 2024-09-17T04:08:12.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.076+0000 7f92ba000640 1 -- 172.21.15.129:0/2910108373 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v264) ==== 104+0+0 (secure 0 0 0) 0x7f92b4063bd0 con 0x7f92c00af690 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 msgr2=0x7f92a407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 0x7f92a407b0d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f92c01602e0 tx=0x7f92b0009040 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 msgr2=0x7f92c01f6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f92b400e7f0 tx=0x7f92b400ecb0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 shutdown_connections 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f92a4078bc0 0x7f92a407b0d0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f92c01605f0 0x7f92c009dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f92c0156d60 0x7f92c0152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 --2- 172.21.15.129:0/2910108373 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f92c00af690 0x7f92c01f6560 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.079+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 >> 172.21.15.129:0/2910108373 conn(0x7f92c00b25f0 msgr2=0x7f92c006fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:12.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.080+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 shutdown_connections 2024-09-17T04:08:12.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.080+0000 7f92c7200640 1 -- 172.21.15.129:0/2910108373 wait complete. 2024-09-17T04:08:12.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:08:12.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 -- 172.21.15.129:0/2871149171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb80cfc80 msgr2=0x7f7cb80af2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:12.331 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2871149171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb80cfc80 0x7f7cb80af2e0 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8070620 tx=0x7f7ca802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 -- 172.21.15.129:0/2871149171 shutdown_connections 2024-09-17T04:08:12.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2871149171 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb80cfc80 0x7f7cb80af2e0 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 -- 172.21.15.129:0/2871149171 >> 172.21.15.129:0/2871149171 conn(0x7f7cb806d7a0 msgr2=0x7f7cb80b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:12.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 -- 172.21.15.129:0/2871149171 shutdown_connections 2024-09-17T04:08:12.332 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.331+0000 7f7cbe600640 1 -- 172.21.15.129:0/2871149171 wait complete. 2024-09-17T04:08:12.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.333+0000 7f7cbe600640 1 Processor -- start 2024-09-17T04:08:12.333 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.333+0000 7f7cbe600640 1 -- start start 2024-09-17T04:08:12.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7cb80cfc80 0x7f7cb809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:12.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 0x7f7cb8154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:12.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:12.335 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb8070ab0 con 0x7f7cb815ce60 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb8070930 con 0x7f7cb80cfc80 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cbe600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb815d980 con 0x7f7cb8153080 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58692/0 (socket says 172.21.15.129:58692) 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 -- 172.21.15.129:0/2339420841 learned_addr learned my addr 172.21.15.129:0/2339420841 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7400640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7cb80cfc80 0x7f7cb809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7caea00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 0x7f7cb8154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 -- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 msgr2=0x7f7cb8154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 0x7f7cb8154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 -- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7cb80cfc80 msgr2=0x7f7cb809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7cb80cfc80 0x7f7cb809aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:12.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7cb7e00640 1 -- 172.21.15.129:0/2339420841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb8154090 con 0x7f7cb815ce60 2024-09-17T04:08:12.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.335+0000 7f7caea00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 0x7f7cb8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:12.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cb7e00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000c3f0 tx=0x7f7cb000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:12.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0017070 con 0x7f7cb815ce60 2024-09-17T04:08:12.337 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7cb00053c0 con 0x7f7cb815ce60 2024-09-17T04:08:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0003850 con 0x7f7cb815ce60 2024-09-17T04:08:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb8154350 con 0x7f7cb815ce60 2024-09-17T04:08:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.336+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb8154910 con 0x7f7cb815ce60 2024-09-17T04:08:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.337+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb80af2e0 con 0x7f7cb815ce60 2024-09-17T04:08:12.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.338+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7cb001a070 con 0x7f7cb815ce60 2024-09-17T04:08:12.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.339+0000 7f7cafe00640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 0x7f7c9407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:12.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.339+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(264..264 src has 1..264) ==== 5967+0+0 (secure 0 0 0) 0x7f7cb009a1c0 con 0x7f7cb815ce60 2024-09-17T04:08:12.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.340+0000 7f7cb7400640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 0x7f7c9407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:12.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.341+0000 7f7cb7400640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 0x7f7c9407b0a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8033000 tx=0x7f7ca8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:12.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.344+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7cb0061e10 con 0x7f7cb815ce60 2024-09-17T04:08:12.526 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:12.528+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7cb815fb50 con 0x7f7cb815ce60 2024-09-17T04:08:13.085 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.086+0000 7f7cafe00640 1 -- 172.21.15.129:0/2339420841 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v265) ==== 181+0+0 (secure 0 0 0) 0x7f7cb0066840 con 0x7f7cb815ce60 2024-09-17T04:08:13.086 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:08:13.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 msgr2=0x7f7c9407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:13.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 0x7f7c9407b0a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8033000 tx=0x7f7ca8004730 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 msgr2=0x7f7cb81558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000c3f0 tx=0x7f7cb000c8b0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 shutdown_connections 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c94078b90 0x7f7c9407b0a0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7cb815ce60 0x7f7cb81558f0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7cb8153080 0x7f7cb8154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 --2- 172.21.15.129:0/2339420841 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7cb80cfc80 0x7f7cb809aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.089+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 >> 172.21.15.129:0/2339420841 conn(0x7f7cb806d7a0 msgr2=0x7f7cb815d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:13.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.090+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 shutdown_connections 2024-09-17T04:08:13.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.090+0000 7f7cbe600640 1 -- 172.21.15.129:0/2339420841 wait complete. 2024-09-17T04:08:13.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.341+0000 7f662a600640 1 -- 172.21.15.129:0/3658712897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6608003c10 msgr2=0x7f6608004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.341+0000 7f662a600640 1 --2- 172.21.15.129:0/3658712897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6608003c10 0x7f6608004060 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f6614009b50 tx=0x7f6608005830 comp rx=0 tx=0).stop 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.341+0000 7f6623e00640 1 -- 172.21.15.129:0/3658712897 reap_dead start 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.342+0000 7f662a600640 1 -- 172.21.15.129:0/3658712897 shutdown_connections 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.342+0000 7f662a600640 1 -- 172.21.15.129:0/3658712897 >> 172.21.15.129:0/3658712897 conn(0x7f662406d7a0 msgr2=0x7f66240b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.342+0000 7f662a600640 1 -- 172.21.15.129:0/3658712897 shutdown_connections 2024-09-17T04:08:13.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.342+0000 7f662a600640 1 -- 172.21.15.129:0/3658712897 wait complete. 2024-09-17T04:08:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.344+0000 7f662a600640 1 Processor -- start 2024-09-17T04:08:13.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.344+0000 7f662a600640 1 -- start start 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6608003c10 0x7f66241f93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6624143520 0x7f6624156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f662415d980 con 0x7f6624143930 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6624070b40 con 0x7f6608003c10 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.345+0000 7f662a600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f662415db00 con 0x7f6624143520 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58708/0 (socket says 172.21.15.129:58708) 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 -- 172.21.15.129:0/2865574553 learned_addr learned my addr 172.21.15.129:0/2865574553 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623400640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6608003c10 0x7f66241f93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:13.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f661aa00640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6624143520 0x7f6624156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 -- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6624143520 msgr2=0x7f6624156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6624143520 0x7f6624156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 -- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6608003c10 msgr2=0x7f66241f93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6608003c10 0x7f66241f93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.346+0000 7f6623e00640 1 -- 172.21.15.129:0/2865574553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66241f98f0 con 0x7f6624143930 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f6623e00640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f66240ae890 tx=0x7f661c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c011870 con 0x7f6624143930 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f661c017070 con 0x7f6624143930 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661c011da0 con 0x7f6624143930 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6624157520 con 0x7f6624143930 2024-09-17T04:08:13.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.347+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66241f5c70 con 0x7f6624143930 2024-09-17T04:08:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.348+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66240af370 con 0x7f6624143930 2024-09-17T04:08:13.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.349+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f661c01c460 con 0x7f6624143930 2024-09-17T04:08:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.350+0000 7f661be00640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 0x7f660007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.350+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(265..265 src has 1..265) ==== 5529+0+0 (secure 0 0 0) 0x7f661c09b4e0 con 0x7f6624143930 2024-09-17T04:08:13.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.350+0000 7f6623400640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 0x7f660007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:13.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.351+0000 7f6623400640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 0x7f660007b0d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6614035000 tx=0x7f661402d040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:13.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.355+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f66241f98f0 con 0x7f6624143930 2024-09-17T04:08:13.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:13.541+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f662406f260 con 0x7f6624143930 2024-09-17T04:08:14.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.097+0000 7f661be00640 1 -- 172.21.15.129:0/2865574553 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v266) ==== 169+0+0 (secure 0 0 0) 0x7f661c0632e0 con 0x7f6624143930 2024-09-17T04:08:14.097 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:08:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 msgr2=0x7f660007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 0x7f660007b0d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f6614035000 tx=0x7f661402d040 comp rx=0 tx=0).stop 2024-09-17T04:08:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 msgr2=0x7f6624156640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f66240ae890 tx=0x7f661c001b80 comp rx=0 tx=0).stop 2024-09-17T04:08:14.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 shutdown_connections 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6600078bc0 0x7f660007b0d0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6624143930 0x7f6624156640 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6624143520 0x7f6624156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 --2- 172.21.15.129:0/2865574553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6608003c10 0x7f66241f93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 >> 172.21.15.129:0/2865574553 conn(0x7f662406d7a0 msgr2=0x7f6624074370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 shutdown_connections 2024-09-17T04:08:14.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.100+0000 7f662a600640 1 -- 172.21.15.129:0/2865574553 wait complete. 2024-09-17T04:08:14.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:08:14.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout:[global] 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: chdir = "" 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: auth supported = cephx 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: filestore xattr use omap = true 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: mon clock drift allowed = 1.000 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: osd crush chooseleaf type = 0 2024-09-17T04:08:14.230 INFO:teuthology.orchestra.run.smithi129.stdout: auth debug = true 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on old message = true 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on bug = true 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: mon pg warn max object skew = 0 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default size = 2 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.231 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow primary affinity = true 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd allow pg remap = true 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on no sortbitwise = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon warn on too few osds = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: mon_allow_pool_size_one = true 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:08:14.232 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: osd default data pool replay window = 5 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: mon allow pool delete = true 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: mon cluster log file level = debug 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: debug asserts on shutdown = true 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: mon health detail to clog = false 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: ms die on skipped message = False 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout:[osd] 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: osd journal size = 100 2024-09-17T04:08:14.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub load threshold = 5.0 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd scrub max interval = 600 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd recover clone overlap = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd recovery max chunk = 1048576 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug shutdown = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug op order = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug verify stray on activate = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug trim objects = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd open classes on start = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug pg log writeout = true 2024-09-17T04:08:14.234 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: osd map max advance = 10 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: journal zero on create = true 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: filestore apply finisher threads = 3 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: bdev debug aio = true 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: osd debug misdirected ops = true 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: bdev async discard = True 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: bdev enable discard = True 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore allocator = bitmap 2024-09-17T04:08:14.235 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore block size = 96636764160 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: bluestore fsck on mount = True 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluefs = 1/20 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug bluestore = 1/20 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug osd = 20 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug rocksdb = 4/10 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd full ratio = 0.9 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: osd objectstore = bluestore 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout:[mgr] 2024-09-17T04:08:14.236 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug mgr = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: mgr/telemetry/nag = false 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout:[mon] 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug ms = 1 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug mon = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug paxos = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: debug auth = 20 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon data avail warn = 5 2024-09-17T04:08:14.237 INFO:teuthology.orchestra.run.smithi129.stdout: mon mgr mkfs grace = 240 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd reporter subtree level = osd 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon osd prime pg temp = true 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: # 1m isn't quite enough 2024-09-17T04:08:14.238 INFO:teuthology.orchestra.run.smithi129.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: mon down mkfs grace = 300 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:[client] 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: rgw cache enabled = true 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable ops log = true 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: rgw enable usage log = true 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout: pid file = "" 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.a] 2024-09-17T04:08:14.239 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.b] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mon.c] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.a] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.c] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.b] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mds.d] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi129.stdout:[mds] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-17T04:08:14.240 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-17T04:08:14.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-17T04:08:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-17T04:08:14.243 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.129,172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790]" 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on skipped message = False 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c5b53c7d-83e3-4793-838f-5ade77ae49c7 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-17T04:08:14.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-17T04:08:14.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bdev async discard = True 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bdev enable discard = True 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore allocator = bitmap 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 1/20 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 1/20 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-17T04:08:14.246 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 4/10 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-17T04:08:14.247 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T04:08:14.248 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: mon down mkfs grace = 300 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-17T04:08:14.249 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = "" 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.a] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.c] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.b] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mds.d] 2024-09-17T04:08:14.250 INFO:teuthology.orchestra.run.smithi181.stdout:[mds] 2024-09-17T04:08:14.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:08:14.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 -- 172.21.15.129:0/491827819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b40705a0 msgr2=0x7f98b40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 --2- 172.21.15.129:0/491827819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b40705a0 0x7f98b40b4140 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7f98b4072080 tx=0x7f98a402a560 comp rx=0 tx=0).stop 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 -- 172.21.15.129:0/491827819 shutdown_connections 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 --2- 172.21.15.129:0/491827819 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b406f2e0 0x7f98b40cd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 --2- 172.21.15.129:0/491827819 >> v2:172.21.15.181:3300/0 conn(0x7f98b406eed0 0x7f98b40ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 --2- 172.21.15.129:0/491827819 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b40705a0 0x7f98b40b4140 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 -- 172.21.15.129:0/491827819 >> 172.21.15.129:0/491827819 conn(0x7f98b406d410 msgr2=0x7f98b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 -- 172.21.15.129:0/491827819 shutdown_connections 2024-09-17T04:08:14.490 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.490+0000 7f98b9a00640 1 -- 172.21.15.129:0/491827819 wait complete. 2024-09-17T04:08:14.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.492+0000 7f98b9a00640 1 Processor -- start 2024-09-17T04:08:14.491 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.492+0000 7f98b9a00640 1 -- start start 2024-09-17T04:08:14.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98b406f2e0 0x7f98b415d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b40705a0 0x7f98b40739d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f98b4160b40 con 0x7f98b406eed0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f98b41609c0 con 0x7f98b406f2e0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b9a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f98b4160cc0 con 0x7f98b40705a0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38180/0 (socket says 172.21.15.129:38180) 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 -- 172.21.15.129:0/1802871775 learned_addr learned my addr 172.21.15.129:0/1802871775 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b3400640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b40705a0 0x7f98b40739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98aa000640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98b406f2e0 0x7f98b415d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 -- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b40705a0 msgr2=0x7f98b40739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b40705a0 0x7f98b40739d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 -- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98b406f2e0 msgr2=0x7f98b415d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98b406f2e0 0x7f98b415d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.495 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.494+0000 7f98b2a00640 1 -- 172.21.15.129:0/1802871775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b415dfe0 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98b2a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f98a40099c0 tx=0x7f98a40039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a403e030 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f98a40047a0 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a4035040 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b415e2c0 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.495+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b415e880 con 0x7f98b406eed0 2024-09-17T04:08:14.496 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.496+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b4161250 con 0x7f98b406eed0 2024-09-17T04:08:14.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.497+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f98a4041030 con 0x7f98b406eed0 2024-09-17T04:08:14.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.498+0000 7f98ab400640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 0x7f989007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.497 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.499+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f98a40bb210 con 0x7f98b406eed0 2024-09-17T04:08:14.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.499+0000 7f98aa000640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 0x7f989007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.498 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.499+0000 7f98aa000640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 0x7f989007b0a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f98b41f9240 tx=0x7f989c00b260 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:14.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.503+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f98a4083240 con 0x7f98b406eed0 2024-09-17T04:08:14.693 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.694+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7f98b406a140 con 0x7f98b406eed0 2024-09-17T04:08:14.697 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.699+0000 7f98ab400640 1 -- 172.21.15.129:0/1802871775 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v72) ==== 71+0+0 (secure 0 0 0) 0x7f98a4087c70 con 0x7f98b406eed0 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 msgr2=0x7f989007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 0x7f989007b0a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f98b41f9240 tx=0x7f989c00b260 comp rx=0 tx=0).stop 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 msgr2=0x7f98b4099550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f98a40099c0 tx=0x7f98a40039e0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 shutdown_connections 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9890078b90 0x7f989007b0a0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f98b40705a0 0x7f98b40739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f98b406f2e0 0x7f98b415d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 --2- 172.21.15.129:0/1802871775 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f98b406eed0 0x7f98b4099550 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 >> 172.21.15.129:0/1802871775 conn(0x7f98b406d410 msgr2=0x7f98b406f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:14.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 shutdown_connections 2024-09-17T04:08:14.702 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.701+0000 7f98b9a00640 1 -- 172.21.15.129:0/1802871775 wait complete. 2024-09-17T04:08:14.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed' 2024-09-17T04:08:14.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 -- 172.21.15.129:0/555883227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c0cd390 msgr2=0x7f408c070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 --2- 172.21.15.129:0/555883227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c0cd390 0x7f408c070100 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f408c0ce800 tx=0x7f408402aa90 comp rx=0 tx=0).stop 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 -- 172.21.15.129:0/555883227 shutdown_connections 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 --2- 172.21.15.129:0/555883227 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c0cd390 0x7f408c070100 unknown :-1 s=CLOSED pgs=2886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 --2- 172.21.15.129:0/555883227 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c15da00 0x7f408c0b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 -- 172.21.15.129:0/555883227 >> 172.21.15.129:0/555883227 conn(0x7f408c06d410 msgr2=0x7f408c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 -- 172.21.15.129:0/555883227 shutdown_connections 2024-09-17T04:08:14.949 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.949+0000 7f4092200640 1 -- 172.21.15.129:0/555883227 wait complete. 2024-09-17T04:08:14.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.951+0000 7f4092200640 1 Processor -- start 2024-09-17T04:08:14.950 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.951+0000 7f4092200640 1 -- start start 2024-09-17T04:08:14.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 0x7f408c1ea450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c099790 0x7f408c08c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f408c0709d0 con 0x7f408c15da00 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f408c070850 con 0x7f408c0b46c0 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4092200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f408c070b50 con 0x7f408c099790 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38210/0 (socket says 172.21.15.129:38210) 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 -- 172.21.15.129:0/181557876 learned_addr learned my addr 172.21.15.129:0/181557876 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f408b400640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 0x7f408c1ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f408be00640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c099790 0x7f408c08c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 -- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c099790 msgr2=0x7f408c08c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c099790 0x7f408c08c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 -- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 msgr2=0x7f408c1ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 0x7f408c1ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f4082a00640 1 -- 172.21.15.129:0/181557876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c156fd0 con 0x7f408c15da00 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.953+0000 7f408b400640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 0x7f408c1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4082a00640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f407400c3d0 tx=0x7f407400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074017070 con 0x7f408c15da00 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f40740053c0 con 0x7f408c15da00 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074003850 con 0x7f408c15da00 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c154ef0 con 0x7f408c15da00 2024-09-17T04:08:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.954+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c152c50 con 0x7f408c15da00 2024-09-17T04:08:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.955+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408c156d40 con 0x7f408c15da00 2024-09-17T04:08:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.957+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f407401a070 con 0x7f408c15da00 2024-09-17T04:08:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.957+0000 7f4083e00640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 0x7f406807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.958+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f4074099d70 con 0x7f408c15da00 2024-09-17T04:08:14.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.958+0000 7f408b400640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 0x7f406807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:14.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.959+0000 7f408b400640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 0x7f406807b0a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f407c009b80 tx=0x7f407c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:14.962+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4074061e40 con 0x7f408c15da00 2024-09-17T04:08:15.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:15.216+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]} v 0) -- 0x7f408c1556f0 con 0x7f408c15da00 2024-09-17T04:08:16.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.108+0000 7f4083e00640 1 -- 172.21.15.129:0/181557876 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_replicated_delete_speed"]}]=0 v949) ==== 143+0+0 (secure 0 0 0) 0x7f4074066870 con 0x7f408c15da00 2024-09-17T04:08:16.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 msgr2=0x7f406807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 0x7f406807b0a0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f407c009b80 tx=0x7f407c005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:16.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 msgr2=0x7f408c08bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7f407400c3d0 tx=0x7f407400c890 comp rx=0 tx=0).stop 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 shutdown_connections 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4068078b90 0x7f406807b0a0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f408c099790 0x7f408c08c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f408c15da00 0x7f408c08bff0 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 --2- 172.21.15.129:0/181557876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f408c0b46c0 0x7f408c1ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 >> 172.21.15.129:0/181557876 conn(0x7f408c06d410 msgr2=0x7f408c0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 shutdown_connections 2024-09-17T04:08:16.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.112+0000 7f4092200640 1 -- 172.21.15.129:0/181557876 wait complete. 2024-09-17T04:08:16.142 INFO:tasks.cephfs_test_runner:That deletions of replicated metadata are not pathologically slow ... ok 2024-09-17T04:08:16.142 INFO:tasks.cephfs_test_runner:Starting test: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) That removal of a snapshot that references a now-unlinked file results 2024-09-17T04:08:16.142 INFO:tasks.cephfs_test_runner:test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2024-09-17T04:08:16.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:16.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 -- 172.21.15.129:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f180cd390 msgr2=0x7f2f181609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 --2- 172.21.15.129:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f180cd390 0x7f2f181609c0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f2f10005fd0 tx=0x7f2f1002a8b0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 -- 172.21.15.129:0/2293704577 shutdown_connections 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 --2- 172.21.15.129:0/2293704577 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f180cd390 0x7f2f181609c0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 --2- 172.21.15.129:0/2293704577 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180705a0 0x7f2f180b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 -- 172.21.15.129:0/2293704577 >> 172.21.15.129:0/2293704577 conn(0x7f2f1806d410 msgr2=0x7f2f180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 -- 172.21.15.129:0/2293704577 shutdown_connections 2024-09-17T04:08:16.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.366+0000 7f2f1d200640 1 -- 172.21.15.129:0/2293704577 wait complete. 2024-09-17T04:08:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.368+0000 7f2f1d200640 1 Processor -- start 2024-09-17T04:08:16.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.369+0000 7f2f1d200640 1 -- start start 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 0x7f2f180b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180b0e70 0x7f2f180ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2f18160f00 con 0x7f2f1806eed0 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2f1806f4f0 con 0x7f2f180705a0 2024-09-17T04:08:16.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f1d200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2f18161080 con 0x7f2f180b0e70 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f15e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f15e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38216/0 (socket says 172.21.15.129:38216) 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.370+0000 7f2f15e00640 1 -- 172.21.15.129:0/3559435752 learned_addr learned my addr 172.21.15.129:0/3559435752 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f16800640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180b0e70 0x7f2f180ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f0d400640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 0x7f2f180b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 -- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180b0e70 msgr2=0x7f2f180ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180b0e70 0x7f2f180ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 -- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 msgr2=0x7f2f180b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 0x7f2f180b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 -- 172.21.15.129:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f180c6760 con 0x7f2f1806eed0 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f0d400640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 0x7f2f180b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:16.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f15e00640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e7c0 tx=0x7f2f0800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f0800ccc0 con 0x7f2f1806eed0 2024-09-17T04:08:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2f08010430 con 0x7f2f1806eed0 2024-09-17T04:08:16.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.371+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f1815c780 con 0x7f2f1806eed0 2024-09-17T04:08:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.372+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f08010860 con 0x7f2f1806eed0 2024-09-17T04:08:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.372+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f1815cd40 con 0x7f2f1806eed0 2024-09-17T04:08:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.373+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f180b4140 con 0x7f2f1806eed0 2024-09-17T04:08:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.374+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2f08010a00 con 0x7f2f1806eed0 2024-09-17T04:08:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.375+0000 7f2f0ea00640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 0x7f2ef407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.375+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f2f0809b3d0 con 0x7f2f1806eed0 2024-09-17T04:08:16.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.375+0000 7f2f0d400640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 0x7f2ef407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.376+0000 7f2f0d400640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 0x7f2ef407b0d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f2f180afeb0 tx=0x7f2f00009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:16.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.379+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2f08063490 con 0x7f2f1806eed0 2024-09-17T04:08:16.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.563+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f1806a140 con 0x7f2f1806eed0 2024-09-17T04:08:16.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.564+0000 7f2f0ea00640 1 -- 172.21.15.129:0/3559435752 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v266) ==== 74+0+13356 (secure 0 0 0) 0x7f2f0806c020 con 0x7f2f1806eed0 2024-09-17T04:08:16.564 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:16.564 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":266,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:08:14.087306+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":39,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6833/3848773989":"2024-09-18T04:08:11.073440+0000","172.21.15.129:6834/3076551424":"2024-09-18T04:08:11.073460+0000","172.21.15.129:6832/3076551424":"2024-09-18T04:08:11.073460+0000","172.21.15.129:6834/1080794990":"2024-09-18T04:07:50.781831+0000","172.21.15.129:6835/3848773989":"2024-09-18T04:08:11.073440+0000","172.21.15.129:6832/1080794990":"2024-09-18T04:07:50.781831+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 msgr2=0x7f2ef407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 0x7f2ef407b0d0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f2f180afeb0 tx=0x7f2f00009040 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 msgr2=0x7f2f180c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e7c0 tx=0x7f2f0800ec80 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 shutdown_connections 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ef4078bc0 0x7f2ef407b0d0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2f180b0e70 0x7f2f180ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f180705a0 0x7f2f180b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 --2- 172.21.15.129:0/3559435752 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2f1806eed0 0x7f2f180c6220 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 >> 172.21.15.129:0/3559435752 conn(0x7f2f1806d410 msgr2=0x7f2f180ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:16.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 shutdown_connections 2024-09-17T04:08:16.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.567+0000 7f2f1d200640 1 -- 172.21.15.129:0/3559435752 wait complete. 2024-09-17T04:08:16.589 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T03:49:24.321404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-17T04:08:16.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-17T04:08:16.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 -- 172.21.15.129:0/2737220846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b3f70 msgr2=0x7f609c0b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 --2- 172.21.15.129:0/2737220846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b3f70 0x7f609c0b49a0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7f608c005fd0 tx=0x7f608c02a540 comp rx=0 tx=0).stop 2024-09-17T04:08:16.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 -- 172.21.15.129:0/2737220846 shutdown_connections 2024-09-17T04:08:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 --2- 172.21.15.129:0/2737220846 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c074580 0x7f609c0b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 --2- 172.21.15.129:0/2737220846 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b3f70 0x7f609c0b49a0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 -- 172.21.15.129:0/2737220846 >> 172.21.15.129:0/2737220846 conn(0x7f609c0ccc40 msgr2=0x7f609c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 -- 172.21.15.129:0/2737220846 shutdown_connections 2024-09-17T04:08:16.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.814+0000 7f60a0e00640 1 -- 172.21.15.129:0/2737220846 wait complete. 2024-09-17T04:08:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.816+0000 7f60a0e00640 1 Processor -- start 2024-09-17T04:08:16.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.816+0000 7f60a0e00640 1 -- start start 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.817+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 0x7f609c14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.817+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f60a0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c154790 0x7f609c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f60a0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f609c06fc20 con 0x7f609c0b2360 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f60a0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f609c06faa0 con 0x7f609c074580 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f60a0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f609c166f20 con 0x7f609c154790 2024-09-17T04:08:16.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38232/0 (socket says 172.21.15.129:38232) 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 -- 172.21.15.129:0/1609712476 learned_addr learned my addr 172.21.15.129:0/1609712476 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 0x7f609c14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f609a800640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c154790 0x7f609c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 -- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c154790 msgr2=0x7f609c154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c154790 0x7f609c154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 -- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 msgr2=0x7f609c14f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 0x7f609c14f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 -- 172.21.15.129:0/1609712476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f609c1550e0 con 0x7f609c0b2360 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 0x7f609c14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.818+0000 7f6099400640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f608800e7c0 tx=0x7f608800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.819+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608800ccc0 con 0x7f609c0b2360 2024-09-17T04:08:16.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.819+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6088010430 con 0x7f609c0b2360 2024-09-17T04:08:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.819+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f609c0acf50 con 0x7f609c0b2360 2024-09-17T04:08:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.819+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6088010860 con 0x7f609c0b2360 2024-09-17T04:08:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.819+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f609c0ad6a0 con 0x7f609c0b2360 2024-09-17T04:08:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.820+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f609c0b3f70 con 0x7f609c0b2360 2024-09-17T04:08:16.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.821+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6088010a00 con 0x7f609c0b2360 2024-09-17T04:08:16.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.822+0000 7f6092000640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 0x7f607c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:16.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.823+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f608809b5a0 con 0x7f609c0b2360 2024-09-17T04:08:16.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.823+0000 7f6099e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 0x7f607c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:16.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.824+0000 7f6099e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 0x7f607c07b0a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f608c033000 tx=0x7f608c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:16.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:16.826+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f609c1550e0 con 0x7f609c0b2360 2024-09-17T04:08:17.007 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.008+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f609c0ad120 con 0x7f609c0b2360 2024-09-17T04:08:17.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.009+0000 7f6092000640 1 -- 172.21.15.129:0/1609712476 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v266) ==== 93+0+10 (secure 0 0 0) 0x7f6088063b90 con 0x7f609c0b2360 2024-09-17T04:08:17.008 INFO:teuthology.orchestra.run.smithi129.stdout:pg_num: 1 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 msgr2=0x7f607c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 0x7f607c07b0a0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f608c033000 tx=0x7f608c004730 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 msgr2=0x7f609c0ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7f608800e7c0 tx=0x7f608800ec80 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 shutdown_connections 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f607c078b90 0x7f607c07b0a0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f609c154790 0x7f609c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f609c0b2360 0x7f609c0ae880 unknown :-1 s=CLOSED pgs=2893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 --2- 172.21.15.129:0/1609712476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f609c074580 0x7f609c14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 >> 172.21.15.129:0/1609712476 conn(0x7f609c0ccc40 msgr2=0x7f609c0b0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 shutdown_connections 2024-09-17T04:08:17.012 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.011+0000 7f60a0e00640 1 -- 172.21.15.129:0/1609712476 wait complete. 2024-09-17T04:08:17.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove' 2024-09-17T04:08:17.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 -- 172.21.15.129:0/1060410606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e8004d40 msgr2=0x7fc9e80051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:17.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 --2- 172.21.15.129:0/1060410606 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc9e8004d40 0x7fc9e80051f0 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fca040c72f0 tx=0x7fc9e8005830 comp rx=0 tx=0).stop 2024-09-17T04:08:17.257 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca09200640 1 -- 172.21.15.129:0/1060410606 reap_dead start 2024-09-17T04:08:17.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 -- 172.21.15.129:0/1060410606 shutdown_connections 2024-09-17T04:08:17.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 -- 172.21.15.129:0/1060410606 >> 172.21.15.129:0/1060410606 conn(0x7fca0406d410 msgr2=0x7fca040b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:17.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 -- 172.21.15.129:0/1060410606 shutdown_connections 2024-09-17T04:08:17.258 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.258+0000 7fca0b200640 1 -- 172.21.15.129:0/1060410606 wait complete. 2024-09-17T04:08:17.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.260+0000 7fca0b200640 1 Processor -- start 2024-09-17T04:08:17.259 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.260+0000 7fca0b200640 1 -- start start 2024-09-17T04:08:17.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9e8004d40 0x7fca041e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:17.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca041e3db0 0x7fca041e4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:17.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:17.262 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fca040b3570 con 0x7fca040ae1d0 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fca040b33f0 con 0x7fca041e3db0 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca0b200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fca040b36f0 con 0x7fc9e8004d40 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca09200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca09200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38250/0 (socket says 172.21.15.129:38250) 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca09200640 1 -- 172.21.15.129:0/3924813764 learned_addr learned my addr 172.21.15.129:0/3924813764 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fca03e00640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9e8004d40 0x7fca041e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.262+0000 7fc9fb400640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca041e3db0 0x7fca041e4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 -- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9e8004d40 msgr2=0x7fca041e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9e8004d40 0x7fca041e5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 -- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca041e3db0 msgr2=0x7fca041e4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:17.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca041e3db0 0x7fca041e4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 -- 172.21.15.129:0/3924813764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca040aeb20 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca09200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc02ac50 tx=0x7fc9fc001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0062b0 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc9fc006450 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0068a0 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca041f5930 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.263+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca041f5ef0 con 0x7fca040ae1d0 2024-09-17T04:08:17.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.265+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca041e4240 con 0x7fca040ae1d0 2024-09-17T04:08:17.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.266+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc9fc003a30 con 0x7fca040ae1d0 2024-09-17T04:08:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.267+0000 7fca00c00640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 0x7fc9e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.267+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7fc9fc0ba7c0 con 0x7fca040ae1d0 2024-09-17T04:08:17.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.267+0000 7fca03e00640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 0x7fc9e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:17.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.268+0000 7fca03e00640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 0x7fc9e007b0a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4000fb0 tx=0x7fc9f40067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:17.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.271+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fca040aeb20 con 0x7fca040ae1d0 2024-09-17T04:08:17.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:17.520+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]} v 0) -- 0x7fca0406a170 con 0x7fca040ae1d0 2024-09-17T04:08:18.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.117+0000 7fca00c00640 1 -- 172.21.15.129:0/3924813764 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]}]=0 v951) ==== 138+0+0 (secure 0 0 0) 0x7fc9fc082e50 con 0x7fca040ae1d0 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 msgr2=0x7fc9e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 0x7fc9e007b0a0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4000fb0 tx=0x7fc9f40067a0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 msgr2=0x7fca040ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc02ac50 tx=0x7fc9fc001b10 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 shutdown_connections 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc9e0078b90 0x7fc9e007b0a0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.119+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fca040ae1d0 0x7fca040ae5e0 unknown :-1 s=CLOSED pgs=2896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.120+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca041e3db0 0x7fca041e4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.120+0000 7fca0b200640 1 --2- 172.21.15.129:0/3924813764 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc9e8004d40 0x7fca041e5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.120+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 >> 172.21.15.129:0/3924813764 conn(0x7fca0406d410 msgr2=0x7fca04073710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:18.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.120+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 shutdown_connections 2024-09-17T04:08:18.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.120+0000 7fca0b200640 1 -- 172.21.15.129:0/3924813764 wait complete. 2024-09-17T04:08:18.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config dump 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 -- 172.21.15.129:0/3845170756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00fc004800 msgr2=0x7f00fc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 --2- 172.21.15.129:0/3845170756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00fc004800 0x7f00fc004c10 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f010400c310 tx=0x7f00fc005830 comp rx=0 tx=0).stop 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 -- 172.21.15.129:0/3845170756 shutdown_connections 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 --2- 172.21.15.129:0/3845170756 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00fc004800 0x7f00fc004c10 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 -- 172.21.15.129:0/3845170756 >> 172.21.15.129:0/3845170756 conn(0x7f011806d410 msgr2=0x7f01180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 -- 172.21.15.129:0/3845170756 shutdown_connections 2024-09-17T04:08:18.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.368+0000 7f011e600640 1 -- 172.21.15.129:0/3845170756 wait complete. 2024-09-17T04:08:18.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.370+0000 7f011e600640 1 Processor -- start 2024-09-17T04:08:18.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.371+0000 7f011e600640 1 -- start start 2024-09-17T04:08:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00fc004800 0x7f01181f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0118152be0 0x7f0118159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f01180cd420 con 0x7f0118150e10 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f01180cd2a0 con 0x7f00fc004800 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f011e600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f01180cd5a0 con 0x7f0118152be0 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f0116a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.372+0000 7f0116a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38266/0 (socket says 172.21.15.129:38266) 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 -- 172.21.15.129:0/1696607229 learned_addr learned my addr 172.21.15.129:0/1696607229 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0117400640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00fc004800 0x7f01181f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0117e00640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0118152be0 0x7f0118159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 -- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0118152be0 msgr2=0x7f0118159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0118152be0 0x7f0118159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 -- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00fc004800 msgr2=0x7f01181f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00fc004800 0x7f01181f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 -- 172.21.15.129:0/1696607229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01180b1d50 con 0x7f0118150e10 2024-09-17T04:08:18.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.373+0000 7f0116a00640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f010402b130 tx=0x7f0104001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.374+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104031040 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.374+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f01040035f0 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.374+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01180b1a70 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.374+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104003a20 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.374+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0118153370 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.375+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01180ca670 con 0x7f0118150e10 2024-09-17T04:08:18.375 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.376+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f010403e030 con 0x7f0118150e10 2024-09-17T04:08:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.377+0000 7f010f400640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 0x7f00f807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.377+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f01040ba720 con 0x7f0118150e10 2024-09-17T04:08:18.376 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.378+0000 7f0117400640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 0x7f00f807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.377 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.378+0000 7f0117400640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 0x7f00f807b0a0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f0108006680 tx=0x7f0108006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:18.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.381+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f01180ca670 con 0x7f0118150e10 2024-09-17T04:08:18.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.563+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7f0118070890 con 0x7f0118150e10 2024-09-17T04:08:18.562 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.563+0000 7f010f400640 1 -- 172.21.15.129:0/1696607229 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v72) ==== 59+0+2294 (secure 0 0 0) 0x7f0104082dd0 con 0x7f0118150e10 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mon advanced mon_op_complaint_time 120 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced client_mount_timeout 30 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_client 20/20 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_mgr 20/20 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr advanced debug_ms 1/1 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.0 basic osd_mclock_max_capacity_iops_ssd 6701.985894 2024-09-17T04:08:18.563 INFO:teuthology.orchestra.run.smithi129.stdout:osd.1 basic osd_mclock_max_capacity_iops_ssd 4162.299368 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.2 basic osd_mclock_max_capacity_iops_ssd 4132.180333 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.3 basic osd_mclock_max_capacity_iops_ssd 4384.739634 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.4 basic osd_mclock_max_capacity_iops_ssd 3920.685510 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.5 basic osd_mclock_max_capacity_iops_ssd 3951.474476 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.6 basic osd_mclock_max_capacity_iops_ssd 6404.550681 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:osd.7 basic osd_mclock_max_capacity_iops_ssd 4393.847050 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds 20/20 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_mds_balancer 20/20 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced debug_ms 2/2 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_frag true 2024-09-17T04:08:18.564 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_debug_scatterstat true 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:mds dev mds_verify_scatter true 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_mon_op_timeout 900 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:mds advanced rados_osd_op_timeout 900 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced client_mount_timeout 600 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_client 20/20 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced debug_ms 1/1 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_mon_op_timeout 900 2024-09-17T04:08:18.565 INFO:teuthology.orchestra.run.smithi129.stdout:client advanced rados_osd_op_timeout 900 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 msgr2=0x7f00f807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 0x7f00f807b0a0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f0108006680 tx=0x7f0108006b70 comp rx=0 tx=0).stop 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 msgr2=0x7f01181526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 secure :-1 s=READY pgs=2899 cs=0 l=1 rev1=1 crypto rx=0x7f010402b130 tx=0x7f0104001d80 comp rx=0 tx=0).stop 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 shutdown_connections 2024-09-17T04:08:18.566 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f00f8078b90 0x7f00f807b0a0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0118152be0 0x7f0118159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0118150e10 0x7f01181526a0 unknown :-1 s=CLOSED pgs=2899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 --2- 172.21.15.129:0/1696607229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00fc004800 0x7f01181f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 >> 172.21.15.129:0/1696607229 conn(0x7f011806d410 msgr2=0x7f0118074900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 shutdown_connections 2024-09-17T04:08:18.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.566+0000 7f011e600640 1 -- 172.21.15.129:0/1696607229 wait complete. 2024-09-17T04:08:18.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-17T04:08:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 -- 172.21.15.129:0/331421098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00704c0 msgr2=0x7f55b00ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 --2- 172.21.15.129:0/331421098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00704c0 0x7f55b00ce480 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f55b0071d40 tx=0x7f55a802a980 comp rx=0 tx=0).stop 2024-09-17T04:08:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 -- 172.21.15.129:0/331421098 shutdown_connections 2024-09-17T04:08:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 --2- 172.21.15.129:0/331421098 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00704c0 0x7f55b00ce480 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 --2- 172.21.15.129:0/331421098 >> v2:172.21.15.181:3300/0 conn(0x7f55b015c7b0 0x7f55b00cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 -- 172.21.15.129:0/331421098 >> 172.21.15.129:0/331421098 conn(0x7f55b00d00b0 msgr2=0x7f55b00d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 -- 172.21.15.129:0/331421098 shutdown_connections 2024-09-17T04:08:18.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.816+0000 7f55af200640 1 -- 172.21.15.129:0/331421098 wait complete. 2024-09-17T04:08:18.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.818+0000 7f55af200640 1 Processor -- start 2024-09-17T04:08:18.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.818+0000 7f55af200640 1 -- start start 2024-09-17T04:08:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55b00704c0 0x7f55b015f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 0x7f55b015e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55b00ceb40 con 0x7f55b00700b0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55b00ce9c0 con 0x7f55b00704c0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55af200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f55b00cecc0 con 0x7f55b015c7b0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38290/0 (socket says 172.21.15.129:38290) 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 -- 172.21.15.129:0/2105539992 learned_addr learned my addr 172.21.15.129:0/2105539992 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55a5400640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55b00704c0 0x7f55b015f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ae800640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 0x7f55b015e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 -- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 msgr2=0x7f55b015e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 0x7f55b015e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 -- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55b00704c0 msgr2=0x7f55b015f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55b00704c0 0x7f55b015f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ade00640 1 -- 172.21.15.129:0/2105539992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55b00b5fa0 con 0x7f55b00700b0 2024-09-17T04:08:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.820+0000 7f55ae800640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 0x7f55b015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55ade00640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f55a000d6d0 tx=0x7f55a000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0017070 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f55a0005330 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0009820 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55b0152280 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.821+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55b00c8c60 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.822+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55b00cf180 con 0x7f55b00700b0 2024-09-17T04:08:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.823+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f55a0009bd0 con 0x7f55b00700b0 2024-09-17T04:08:18.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.824+0000 7f55a6a00640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 0x7f558c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:18.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.824+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f55a009aef0 con 0x7f55b00700b0 2024-09-17T04:08:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.825+0000 7f55a5400640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 0x7f558c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:18.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.825+0000 7f55a5400640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 0x7f558c07b0a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5598006fd0 tx=0x7f5598008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:18.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:18.829+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f55a00634e0 con 0x7f55b00700b0 2024-09-17T04:08:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.010+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f55b00d0f50 con 0x7f55b00700b0 2024-09-17T04:08:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.015+0000 7f55a6a00640 1 -- 172.21.15.129:0/2105539992 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v73)=0 v73) ==== 122+0+0 (secure 0 0 0) 0x7f55a0062de0 con 0x7f55b00700b0 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 msgr2=0x7f558c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 0x7f558c07b0a0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f5598006fd0 tx=0x7f5598008040 comp rx=0 tx=0).stop 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 msgr2=0x7f55b00c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f55a000d6d0 tx=0x7f55a000db90 comp rx=0 tx=0).stop 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 shutdown_connections 2024-09-17T04:08:19.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f558c078b90 0x7f558c07b0a0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55b015c7b0 0x7f55b015e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55b00704c0 0x7f55b015f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 --2- 172.21.15.129:0/2105539992 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55b00700b0 0x7f55b00c9740 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.017+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 >> 172.21.15.129:0/2105539992 conn(0x7f55b00d00b0 msgr2=0x7f55b01601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.018+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 shutdown_connections 2024-09-17T04:08:19.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.018+0000 7f55af200640 1 -- 172.21.15.129:0/2105539992 wait complete. 2024-09-17T04:08:19.043 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:08:19.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:08:19.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:08:19.246 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:08:19.246 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:19.246 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:19.282 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:08:19.282 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:08:19.282 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:19.319 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:08:19.319 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:19.319 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:19.357 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:08:19.357 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:08:19.357 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:19.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:08:19.395 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-17T04:08:19.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 -- 172.21.15.129:0/1557504234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea80af4c0 msgr2=0x7f7ea815ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 --2- 172.21.15.129:0/1557504234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea80af4c0 0x7f7ea815ae00 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f7e9800a940 tx=0x7f7e9802b790 comp rx=0 tx=0).stop 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 -- 172.21.15.129:0/1557504234 shutdown_connections 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 --2- 172.21.15.129:0/1557504234 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea80af4c0 0x7f7ea815ae00 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 -- 172.21.15.129:0/1557504234 >> 172.21.15.129:0/1557504234 conn(0x7f7ea806d410 msgr2=0x7f7ea80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 -- 172.21.15.129:0/1557504234 shutdown_connections 2024-09-17T04:08:19.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.617+0000 7f7eafa00640 1 -- 172.21.15.129:0/1557504234 wait complete. 2024-09-17T04:08:19.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.620+0000 7f7eafa00640 1 Processor -- start 2024-09-17T04:08:19.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.620+0000 7f7eafa00640 1 -- start start 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ea80af4c0 0x7f7ea81f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea81e7c20 0x7f7ea80ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea806fd50 con 0x7f7ea815b340 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7ea8071170 con 0x7f7ea81e7c20 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7eafa00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7ea806fed0 con 0x7f7ea80af4c0 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.621+0000 7f7e9fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38306/0 (socket says 172.21.15.129:38306) 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 -- 172.21.15.129:0/86946861 learned_addr learned my addr 172.21.15.129:0/86946861 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7ead000640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ea80af4c0 0x7f7ea81f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:19.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7eada00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea81e7c20 0x7f7ea80ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:19.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 -- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ea80af4c0 msgr2=0x7f7ea81f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ea80af4c0 0x7f7ea81f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 -- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea81e7c20 msgr2=0x7f7ea80ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea81e7c20 0x7f7ea80ad920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 -- 172.21.15.129:0/86946861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ea81e6b80 con 0x7f7ea815b340 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.622+0000 7f7e9fe00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c4d0 tx=0x7f7e9000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.623+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90005480 con 0x7f7ea815b340 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.623+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e900035f0 con 0x7f7ea815b340 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.623+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90003a20 con 0x7f7ea815b340 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.623+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ea81e6e60 con 0x7f7ea815b340 2024-09-17T04:08:19.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.623+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ea81e7420 con 0x7f7ea815b340 2024-09-17T04:08:19.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.624+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea81e68d0 con 0x7f7ea815b340 2024-09-17T04:08:19.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.625+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7e90003bc0 con 0x7f7ea815b340 2024-09-17T04:08:19.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.626+0000 7f7ea5600640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 0x7f7e8407b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:19.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.626+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f7e9009a6e0 con 0x7f7ea815b340 2024-09-17T04:08:19.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.627+0000 7f7ead000640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 0x7f7e8407b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:19.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.627+0000 7f7ead000640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 0x7f7e8407b240 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7e98035000 tx=0x7f7e98004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:19.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.631+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7e90062680 con 0x7f7ea815b340 2024-09-17T04:08:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.849+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ea80703b0 con 0x7f7ea815b340 2024-09-17T04:08:19.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.849+0000 7f7ea5600640 1 -- 172.21.15.129:0/86946861 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) ==== 95+0+2266 (secure 0 0 0) 0x7f7e900670b0 con 0x7f7ea815b340 2024-09-17T04:08:19.849 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:19.849 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":264,"btime":"2024-09-17T04:08:12:075839+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":12573,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/145838093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":145838093},{"type":"v1","addr":"172.21.15.129:6834","nonce":145838093}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":264},{"gid":12576,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/193445134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":193445134},{"type":"v1","addr":"172.21.15.129:6835","nonce":193445134}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":264}],"filesystems":[]} 2024-09-17T04:08:19.849 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 264 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 msgr2=0x7f7e8407b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 0x7f7e8407b240 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7e98035000 tx=0x7f7e98004730 comp rx=0 tx=0).stop 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 msgr2=0x7f7ea81f62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000c4d0 tx=0x7f7e9000c990 comp rx=0 tx=0).stop 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 shutdown_connections 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e84078d30 0x7f7e8407b240 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7ea81e7c20 0x7f7ea80ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.851 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7ea815b340 0x7f7ea81f62e0 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 --2- 172.21.15.129:0/86946861 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7ea80af4c0 0x7f7ea81f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 >> 172.21.15.129:0/86946861 conn(0x7f7ea806d410 msgr2=0x7f7ea80ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 shutdown_connections 2024-09-17T04:08:19.852 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:19.852+0000 7f7eafa00640 1 -- 172.21.15.129:0/86946861 wait complete. 2024-09-17T04:08:19.879 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-17T04:08:19.879 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-17T04:08:19.879 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:08:19.879 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:08:19.879 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-17T04:08:19.879 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-17T04:08:19.879 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:08:19.880 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:08:19.880 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-17T04:08:19.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-17T04:08:19.880 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-17T04:08:19.880 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-17T04:08:19.884 INFO:tasks.ceph.mds.b:Started 2024-09-17T04:08:19.885 DEBUG:teuthology.parallel:result is None 2024-09-17T04:08:19.887 INFO:tasks.ceph.mds.d:Started 2024-09-17T04:08:19.887 DEBUG:teuthology.parallel:result is None 2024-09-17T04:08:19.901 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:08:19.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-17T04:08:19.902 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:08:19.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-17T04:08:19.946 INFO:tasks.ceph.mds.a:Started 2024-09-17T04:08:19.946 DEBUG:teuthology.parallel:result is None 2024-09-17T04:08:19.948 INFO:tasks.ceph.mds.c:Started 2024-09-17T04:08:19.949 DEBUG:teuthology.parallel:result is None 2024-09-17T04:08:19.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-17T04:08:20.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:08:20.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:08:20.083 INFO:tasks.ceph.mds.d.smithi181.stdout:starting mds.d at 2024-09-17T04:08:20.083 INFO:tasks.ceph.mds.b.smithi181.stdout:starting mds.b at 2024-09-17T04:08:20.148 INFO:tasks.ceph.mds.c.smithi129.stdout:starting mds.c at 2024-09-17T04:08:20.149 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 -- 172.21.15.129:0/1267959961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 msgr2=0x7f55180d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 --2- 172.21.15.129:0/1267959961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55180d4540 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f55180ceb30 tx=0x7f550802a520 comp rx=0 tx=0).stop 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 -- 172.21.15.129:0/1267959961 shutdown_connections 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 --2- 172.21.15.129:0/1267959961 >> v2:172.21.15.181:3300/0 conn(0x7f55180d16d0 0x7f5518169710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 --2- 172.21.15.129:0/1267959961 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55180d4540 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:20.187 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 -- 172.21.15.129:0/1267959961 >> 172.21.15.129:0/1267959961 conn(0x7f551809d8d0 msgr2=0x7f551808bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:20.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 -- 172.21.15.129:0/1267959961 shutdown_connections 2024-09-17T04:08:20.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.188+0000 7f551ca00640 1 -- 172.21.15.129:0/1267959961 wait complete. 2024-09-17T04:08:20.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.190+0000 7f551ca00640 1 Processor -- start 2024-09-17T04:08:20.189 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.190+0000 7f551ca00640 1 -- start start 2024-09-17T04:08:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55180d16d0 0x7f55180cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55180d4a80 0x7f551809e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f55180d0070 con 0x7f55180b3220 2024-09-17T04:08:20.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f55180cfef0 con 0x7f55180d4a80 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f551ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f55180b9ec0 con 0x7f55180d16d0 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f5516800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f5516800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38378/0 (socket says 172.21.15.129:38378) 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f5516800640 1 -- 172.21.15.129:0/704750096 learned_addr learned my addr 172.21.15.129:0/704750096 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f550de00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55180d16d0 0x7f55180cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.192+0000 7f5517200640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55180d4a80 0x7f551809e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 -- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55180d16d0 msgr2=0x7f55180cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55180d16d0 0x7f55180cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 -- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55180d4a80 msgr2=0x7f551809e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55180d4a80 0x7f551809e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 -- 172.21.15.129:0/704750096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55180c8220 con 0x7f55180b3220 2024-09-17T04:08:20.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f5516800640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f5508009910 tx=0x7f55080049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5508031950 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5508031be0 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55181f6370 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55080330d0 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.193+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55181f6930 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.194+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5518076fb0 con 0x7f55180b3220 2024-09-17T04:08:20.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.195+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5508033290 con 0x7f55180b3220 2024-09-17T04:08:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.196+0000 7f550f400640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 0x7f54f407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.197+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(266..266 src has 1..266) ==== 5115+0+0 (secure 0 0 0) 0x7f55080bbd70 con 0x7f55180b3220 2024-09-17T04:08:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.197+0000 7f550de00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 0x7f54f407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:20.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.197+0000 7f550de00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 0x7f54f407b0a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f5518161610 tx=0x7f5500007e90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:20.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.201+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5518076fb0 con 0x7f55180b3220 2024-09-17T04:08:20.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:20.395+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f55180d0bc0 con 0x7f55180b3220 2024-09-17T04:08:21.149 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.149+0000 7f550f400640 1 -- 172.21.15.129:0/704750096 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v268) ==== 97+0+0 (secure 0 0 0) 0x7f550803f030 con 0x7f55180b3220 2024-09-17T04:08:21.149 INFO:teuthology.orchestra.run.smithi129.stderr: removed all blocklist entries 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.152+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 msgr2=0x7f54f407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.152+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 0x7f54f407b0a0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f5518161610 tx=0x7f5500007e90 comp rx=0 tx=0).stop 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.152+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 msgr2=0x7f55181f5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.152+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f5508009910 tx=0x7f55080049d0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 shutdown_connections 2024-09-17T04:08:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54f4078b90 0x7f54f407b0a0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55180d4a80 0x7f551809e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f55180d16d0 0x7f55180cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 --2- 172.21.15.129:0/704750096 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f55180b3220 0x7f55181f5b30 unknown :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 >> 172.21.15.129:0/704750096 conn(0x7f551809d8d0 msgr2=0x7f55180d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 shutdown_connections 2024-09-17T04:08:21.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.153+0000 7f551ca00640 1 -- 172.21.15.129:0/704750096 wait complete. 2024-09-17T04:08:21.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 -- 172.21.15.129:0/1928262908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 msgr2=0x7f06a806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 --2- 172.21.15.129:0/1928262908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a806eed0 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f06a8099a70 tx=0x7f069802a530 comp rx=0 tx=0).stop 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 -- 172.21.15.129:0/1928262908 shutdown_connections 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 --2- 172.21.15.129:0/1928262908 >> v2:172.21.15.181:3300/0 conn(0x7f06a80ccc80 0x7f06a80cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 --2- 172.21.15.129:0/1928262908 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a806eed0 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 -- 172.21.15.129:0/1928262908 >> 172.21.15.129:0/1928262908 conn(0x7f06a806d410 msgr2=0x7f06a80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 -- 172.21.15.129:0/1928262908 shutdown_connections 2024-09-17T04:08:21.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.401+0000 7f06ad800640 1 -- 172.21.15.129:0/1928262908 wait complete. 2024-09-17T04:08:21.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.403+0000 7f06ad800640 1 Processor -- start 2024-09-17T04:08:21.403 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.404+0000 7f06ad800640 1 -- start start 2024-09-17T04:08:21.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 0x7f06a815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 0x7f06a81e7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8160ba0 con 0x7f06a8099570 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f06a8160a20 con 0x7f06a80ccc80 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06ad800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f06a8160d20 con 0x7f06a80cf1d0 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06a6a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38402/0 (socket says 172.21.15.129:38402) 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.405+0000 7f06a6a00640 1 -- 172.21.15.129:0/2882867479 learned_addr learned my addr 172.21.15.129:0/2882867479 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:21.406 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a7400640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 0x7f06a81e7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f069e000640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 0x7f06a815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 -- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 msgr2=0x7f06a81e7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 0x7f06a81e7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 -- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 msgr2=0x7f06a815d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 0x7f06a815d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 -- 172.21.15.129:0/2882867479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a81e72b0 con 0x7f06a8099570 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a7400640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 0x7f06a81e7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f069e000640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 0x7f06a815d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06a6a00640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f069802ab20 tx=0x7f0698033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698005c90 con 0x7f06a8099570 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0698031e50 con 0x7f06a8099570 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06a815fd50 con 0x7f06a8099570 2024-09-17T04:08:21.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.406+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0698005240 con 0x7f06a8099570 2024-09-17T04:08:21.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.407+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06a8160310 con 0x7f06a8099570 2024-09-17T04:08:21.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.408+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06a8161240 con 0x7f06a8099570 2024-09-17T04:08:21.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.409+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f06980053e0 con 0x7f06a8099570 2024-09-17T04:08:21.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.410+0000 7f069f400640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 0x7f068407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.410+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4857+0+0 (secure 0 0 0) 0x7f06980bbbe0 con 0x7f06a8099570 2024-09-17T04:08:21.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.410+0000 7f069e000640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 0x7f068407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.411+0000 7f069e000640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 0x7f068407b0a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f06a815db00 tx=0x7f0690005dd0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:21.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.414+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f069803f030 con 0x7f06a8099570 2024-09-17T04:08:21.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.668+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f06a806a140 con 0x7f06a8099570 2024-09-17T04:08:21.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.669+0000 7f069f400640 1 -- 172.21.15.129:0/2882867479 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v47) ==== 80+0+5304 (secure 0 0 0) 0x7f0698031770 con 0x7f06a8099570 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "auth_dump": [ 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.a", 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmhOJkABAA1FVjZ+2DVLd4JriDAvU1SQ==", 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.668 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.b", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmGaT3GxAAE1oqjx4EQcLD5csDrQwMnw==", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:08:21.669 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.c", 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmYdExDhAAiLrNIPB5oRAF3bjPOOrrTw==", 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mds.d", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/OhmdfQSKhAAEucPwkramQ3/nVz1fIK3YA==", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.0", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAH/OhmUMU9GBAAAR7uOmqTqGnxHa0+xycQ+A==", 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.671 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.1", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAK/OhmLUaWKhAAqGm8r0zP4b4GesyhkwMbVg==", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.2", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAO/OhmWZjCCRAAs3bNx6jOql5j44R2D8NVLg==", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.3", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAR/Ohm7a0oJRAAl8upCjZwNJm/yz+huA5wUw==", 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.673 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.4", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAX/Ohm09NLNhAA4xY6M/G7xSMkhrBk7SQdnw==", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.674 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.5", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAb/OhmRjWTGBAADC6I8I6icSD0RRsN7dt1wQ==", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.6", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAe/OhmASK6JxAAnQmKAITnLBB87e+4ncu8hw==", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.675 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "osd.7", 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAh/OhmJmHUNRAA+PyFJobjAmPQiNvfk7efPg==", 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow profile osd", 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile osd", 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.676 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.0", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA==", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow r", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.1", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmpGgCEBAAbW5b/G6jyPRKx/wLJlXwvg==", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.677 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.2", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmjFpsBRAA3nDvGm1CtGEUOeOpOjJ6CA==", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.3", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAE/OhmkTKHGhAAAYtE4S+NerwJ64beOeFvtQ==", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow r", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow rw", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow rwx" 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "client.admin", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/Ohmo2V4ABAA6Kxkn+4pMg2aDxxaq9yF4A==", 2024-09-17T04:08:21.679 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "mgr": "allow *", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow *", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.x", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmmzdZMRAAKT1efgpGgbiHI+hEim2SLA==", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:08:21.680 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "entity": "mgr.y", 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "key": "AQAC/OhmeVFpJhAAXtGMaaEgtXjRz0PDnfinkA==", 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "caps": { 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "mds": "allow *", 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "mon": "allow profile mgr", 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: "osd": "allow *" 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:21.681 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:08:21.682 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 msgr2=0x7f068407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 0x7f068407b0a0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f06a815db00 tx=0x7f0690005dd0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 msgr2=0x7f06a81ea110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f069802ab20 tx=0x7f0698033c20 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 shutdown_connections 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0684078b90 0x7f068407b0a0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f06a80cf1d0 0x7f06a81e7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06a80ccc80 0x7f06a815d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 --2- 172.21.15.129:0/2882867479 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f06a8099570 0x7f06a81ea110 unknown :-1 s=CLOSED pgs=2918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 >> 172.21.15.129:0/2882867479 conn(0x7f06a806d410 msgr2=0x7f06a8072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:21.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 shutdown_connections 2024-09-17T04:08:21.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.671+0000 7f06ad800640 1 -- 172.21.15.129:0/2882867479 wait complete. 2024-09-17T04:08:21.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-17T04:08:21.917 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 -- 172.21.15.129:0/218968308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b080cd390 msgr2=0x7f2b081609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 --2- 172.21.15.129:0/218968308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b080cd390 0x7f2b081609c0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f2b080d1370 tx=0x7f2b0002aab0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 -- 172.21.15.129:0/218968308 shutdown_connections 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 --2- 172.21.15.129:0/218968308 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b080cd390 0x7f2b081609c0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 --2- 172.21.15.129:0/218968308 >> v2:172.21.15.181:3300/0 conn(0x7f2b080705a0 0x7f2b080b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 -- 172.21.15.129:0/218968308 >> 172.21.15.129:0/218968308 conn(0x7f2b0806d410 msgr2=0x7f2b080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 -- 172.21.15.129:0/218968308 shutdown_connections 2024-09-17T04:08:21.918 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.918+0000 7f2b0e800640 1 -- 172.21.15.129:0/218968308 wait complete. 2024-09-17T04:08:21.919 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.920+0000 7f2b0e800640 1 Processor -- start 2024-09-17T04:08:21.920 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.921+0000 7f2b0e800640 1 -- start start 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b080705a0 0x7f2b081552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 0x7f2b081586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0806f460 con 0x7f2b0806eed0 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0806f2e0 con 0x7f2b080705a0 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b0e800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0806f5e0 con 0x7f2b081582a0 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b07400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b07400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38414/0 (socket says 172.21.15.129:38414) 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.922+0000 7f2b07400640 1 -- 172.21.15.129:0/3018568790 learned_addr learned my addr 172.21.15.129:0/3018568790 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:21.923 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07e00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 0x7f2b081586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2afea00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b080705a0 0x7f2b081552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 -- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 msgr2=0x7f2b081586b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 0x7f2b081586b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 -- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b080705a0 msgr2=0x7f2b081552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b080705a0 0x7f2b081552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 -- 172.21.15.129:0/3018568790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b08159210 con 0x7f2b0806eed0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07e00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 0x7f2b081586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b07400640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c970 tx=0x7f2af800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af8007bb0 con 0x7f2b0806eed0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2af8007e40 con 0x7f2b0806eed0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b08158bf0 con 0x7f2b0806eed0 2024-09-17T04:08:21.924 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.923+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2af800f690 con 0x7f2b0806eed0 2024-09-17T04:08:21.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.924+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b08159760 con 0x7f2b0806eed0 2024-09-17T04:08:21.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.925+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b08161210 con 0x7f2b0806eed0 2024-09-17T04:08:21.925 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.926+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2af800f830 con 0x7f2b0806eed0 2024-09-17T04:08:21.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.927+0000 7f2affe00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 0x7f2ae407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:21.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.927+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4857+0+0 (secure 0 0 0) 0x7f2af809b310 con 0x7f2b0806eed0 2024-09-17T04:08:21.926 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.927+0000 7f2afea00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 0x7f2ae407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:21.927 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.928+0000 7f2afea00640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 0x7f2ae407b0d0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f2b080ca770 tx=0x7f2af0009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:21.930 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:21.931+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f2b08158bf0 con 0x7f2b0806eed0 2024-09-17T04:08:22.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.115+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2b0815aab0 con 0x7f2b0806eed0 2024-09-17T04:08:22.114 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.115+0000 7f2affe00640 1 -- 172.21.15.129:0/3018568790 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v268) ==== 84+0+65 (secure 0 0 0) 0x7f2af8014030 con 0x7f2b0806eed0 2024-09-17T04:08:22.114 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:22.114 INFO:teuthology.orchestra.run.smithi129.stdout:[ 2024-09-17T04:08:22.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:22.115 INFO:teuthology.orchestra.run.smithi129.stdout: "poolnum": 1, 2024-09-17T04:08:22.115 INFO:teuthology.orchestra.run.smithi129.stdout: "poolname": ".mgr" 2024-09-17T04:08:22.115 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:22.115 INFO:teuthology.orchestra.run.smithi129.stdout:] 2024-09-17T04:08:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 msgr2=0x7f2ae407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 0x7f2ae407b0d0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f2b080ca770 tx=0x7f2af0009040 comp rx=0 tx=0).stop 2024-09-17T04:08:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 msgr2=0x7f2b0815acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:22.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f2af800c970 tx=0x7f2af800ce30 comp rx=0 tx=0).stop 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 shutdown_connections 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2ae4078bc0 0x7f2ae407b0d0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f2b081582a0 0x7f2b081586b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2b080705a0 0x7f2b081552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 --2- 172.21.15.129:0/3018568790 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f2b0806eed0 0x7f2b0815acd0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 >> 172.21.15.129:0/3018568790 conn(0x7f2b0806d410 msgr2=0x7f2b080cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 shutdown_connections 2024-09-17T04:08:22.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.118+0000 7f2b0e800640 1 -- 172.21.15.129:0/3018568790 wait complete. 2024-09-17T04:08:22.139 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-17T04:08:22.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-17T04:08:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.365+0000 7fb145000640 1 -- 172.21.15.129:0/505295695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb140074580 msgr2=0x7fb1400b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.365+0000 7fb145000640 1 --2- 172.21.15.129:0/505295695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb140074580 0x7fb1400b49a0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7fb1400b3a50 tx=0x7fb12c02a560 comp rx=0 tx=0).stop 2024-09-17T04:08:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 -- 172.21.15.129:0/505295695 shutdown_connections 2024-09-17T04:08:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 --2- 172.21.15.129:0/505295695 >> v2:172.21.15.181:3300/0 conn(0x7fb1400cba90 0x7fb1400cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 --2- 172.21.15.129:0/505295695 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb140074580 0x7fb1400b49a0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 -- 172.21.15.129:0/505295695 >> 172.21.15.129:0/505295695 conn(0x7fb1400ccc40 msgr2=0x7fb1400cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:22.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 -- 172.21.15.129:0/505295695 shutdown_connections 2024-09-17T04:08:22.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.366+0000 7fb145000640 1 -- 172.21.15.129:0/505295695 wait complete. 2024-09-17T04:08:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.368+0000 7fb145000640 1 Processor -- start 2024-09-17T04:08:22.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.368+0000 7fb145000640 1 -- start start 2024-09-17T04:08:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb140074580 0x7fb140156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:22.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 0x7fb140155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400cc8e0 con 0x7fb1401f33e0 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb1400cc760 con 0x7fb1400cba90 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb145000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb140166f20 con 0x7fb140074580 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38430/0 (socket says 172.21.15.129:38430) 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 -- 172.21.15.129:0/4108195485 learned_addr learned my addr 172.21.15.129:0/4108195485 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13d400640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 0x7fb140155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13de00640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb140074580 0x7fb140156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 -- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb140074580 msgr2=0x7fb140156e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb140074580 0x7fb140156e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 -- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 msgr2=0x7fb140155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 0x7fb140155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13e800640 1 -- 172.21.15.129:0/4108195485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14009ca30 con 0x7fb1401f33e0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.370+0000 7fb13d400640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 0x7fb140155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb13e800640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fb13800ea80 tx=0x7fb13800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138004b20 con 0x7fb1401f33e0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb138004ce0 con 0x7fb1401f33e0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1401f3f00 con 0x7fb1401f33e0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138005280 con 0x7fb1401f33e0 2024-09-17T04:08:22.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.371+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1401573c0 con 0x7fb1401f33e0 2024-09-17T04:08:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.372+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1400b3f70 con 0x7fb1401f33e0 2024-09-17T04:08:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.373+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb138005420 con 0x7fb1401f33e0 2024-09-17T04:08:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.374+0000 7fb136000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 0x7fb12007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.375+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 4857+0+0 (secure 0 0 0) 0x7fb13809b600 con 0x7fb1401f33e0 2024-09-17T04:08:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.375+0000 7fb13de00640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 0x7fb12007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:22.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.376+0000 7fb13de00640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 0x7fb12007b0d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb130001720 tx=0x7fb130009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:22.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.379+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb138063d80 con 0x7fb1401f33e0 2024-09-17T04:08:22.561 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:22.562+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb1401f3c90 con 0x7fb1401f33e0 2024-09-17T04:08:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.161+0000 7fb136000640 1 -- 172.21.15.129:0/4108195485 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v269) ==== 138+0+0 (secure 0 0 0) 0x7fb138063680 con 0x7fb1401f33e0 2024-09-17T04:08:23.160 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' created 2024-09-17T04:08:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 msgr2=0x7fb12007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 0x7fb12007b0d0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fb130001720 tx=0x7fb130009040 comp rx=0 tx=0).stop 2024-09-17T04:08:23.163 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 msgr2=0x7fb14009d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fb13800ea80 tx=0x7fb13800ef40 comp rx=0 tx=0).stop 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 shutdown_connections 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb120078bc0 0x7fb12007b0d0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb1401f33e0 0x7fb14009d210 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb1400cba90 0x7fb140155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 --2- 172.21.15.129:0/4108195485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb140074580 0x7fb140156e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 >> 172.21.15.129:0/4108195485 conn(0x7fb1400ccc40 msgr2=0x7fb1400d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 shutdown_connections 2024-09-17T04:08:23.164 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.164+0000 7fb145000640 1 -- 172.21.15.129:0/4108195485 wait complete. 2024-09-17T04:08:23.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-17T04:08:23.426 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 -- 172.21.15.129:0/2187670607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0703b0 msgr2=0x7f7c4c0b3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 --2- 172.21.15.129:0/2187670607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0703b0 0x7f7c4c0b3750 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009be0 tx=0x7f7c3402a920 comp rx=0 tx=0).stop 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 -- 172.21.15.129:0/2187670607 shutdown_connections 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 --2- 172.21.15.129:0/2187670607 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0703b0 0x7f7c4c0b3750 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 -- 172.21.15.129:0/2187670607 >> 172.21.15.129:0/2187670607 conn(0x7f7c4c06d7a0 msgr2=0x7f7c4c0b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 -- 172.21.15.129:0/2187670607 shutdown_connections 2024-09-17T04:08:23.427 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.427+0000 7f7c52000640 1 -- 172.21.15.129:0/2187670607 wait complete. 2024-09-17T04:08:23.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.429+0000 7f7c52000640 1 Processor -- start 2024-09-17T04:08:23.428 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.429+0000 7f7c52000640 1 -- start start 2024-09-17T04:08:23.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c4c0703b0 0x7f7c4c151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:23.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0b3c90 0x7f7c4c15b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0b2cb0 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c0b2b30 con 0x7f7c4c0703b0 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c52000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c06f5f0 con 0x7f7c4c0b3c90 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c4b400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c4c0703b0 0x7f7c4c151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c4be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:38440/0 (socket says 172.21.15.129:38440) 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c4be00640 1 -- 172.21.15.129:0/650227324 learned_addr learned my addr 172.21.15.129:0/650227324 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.431+0000 7f7c42a00640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0b3c90 0x7f7c4c15b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 -- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0b3c90 msgr2=0x7f7c4c15b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0b3c90 0x7f7c4c15b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 -- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c4c0703b0 msgr2=0x7f7c4c151aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c4c0703b0 0x7f7c4c151aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:23.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 -- 172.21.15.129:0/650227324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c0cfb20 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c4be00640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c730 tx=0x7f7c4400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44016020 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c4400a930 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400adc0 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c1f95f0 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.432+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c151fe0 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.433+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c0b3750 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.434 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.435+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7c44019020 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.436+0000 7f7c43e00640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 0x7f7c2807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:23.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.436+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(269..269 src has 1..269) ==== 5227+0+0 (secure 0 0 0) 0x7f7c4409afc0 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.435 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.436+0000 7f7c4b400640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 0x7f7c2807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:23.436 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.437+0000 7f7c4b400640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 0x7f7c2807b0a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c001120 tx=0x7f7c3c009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:23.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.440+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7c44020020 con 0x7f7c4c0b4d70 2024-09-17T04:08:23.624 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:23.624+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7c4c06ef80 con 0x7f7c4c0b4d70 2024-09-17T04:08:24.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.170+0000 7f7c43e00640 1 -- 172.21.15.129:0/650227324 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v270) ==== 170+0+0 (secure 0 0 0) 0x7f7c4c0b40a0 con 0x7f7c4c0b4d70 2024-09-17T04:08:24.170 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' created 2024-09-17T04:08:24.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 msgr2=0x7f7c2807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:24.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 0x7f7c2807b0a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c001120 tx=0x7f7c3c009040 comp rx=0 tx=0).stop 2024-09-17T04:08:24.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 msgr2=0x7f7c4c15be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:24.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400c730 tx=0x7f7c4400cb30 comp rx=0 tx=0).stop 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 shutdown_connections 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7c28078b90 0x7f7c2807b0a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c4c0b4d70 0x7f7c4c15be40 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c4c0b3c90 0x7f7c4c15b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 --2- 172.21.15.129:0/650227324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c4c0703b0 0x7f7c4c151aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.173+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 >> 172.21.15.129:0/650227324 conn(0x7f7c4c06d7a0 msgr2=0x7f7c4c156c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.174+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 shutdown_connections 2024-09-17T04:08:24.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.174+0000 7f7c52000640 1 -- 172.21.15.129:0/650227324 wait complete. 2024-09-17T04:08:24.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-17T04:08:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 -- 172.21.15.129:0/347936126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b3010 msgr2=0x7f0d680b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/347936126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b3010 0x7f0d680b3420 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f0d60005fd0 tx=0x7f0d6002aaa0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 -- 172.21.15.129:0/347936126 shutdown_connections 2024-09-17T04:08:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/347936126 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b3010 0x7f0d680b3420 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.437 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/347936126 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d68074800 0x7f0d680af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 -- 172.21.15.129:0/347936126 >> 172.21.15.129:0/347936126 conn(0x7f0d680ccc40 msgr2=0x7f0d680cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:24.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 -- 172.21.15.129:0/347936126 shutdown_connections 2024-09-17T04:08:24.438 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.437+0000 7f0d6ca00640 1 -- 172.21.15.129:0/347936126 wait complete. 2024-09-17T04:08:24.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.439+0000 7f0d6ca00640 1 Processor -- start 2024-09-17T04:08:24.439 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.439+0000 7f0d6ca00640 1 -- start start 2024-09-17T04:08:24.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.440+0000 7f0d6ca00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 0x7f0d6815a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:24.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d6ca00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:24.442 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d6ca00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d680b3010 0x7f0d6809d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d6ca00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0d680b3ae0 con 0x7f0d680b1c40 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d6ca00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0d680b3960 con 0x7f0d68074800 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d6ca00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0d680b3c60 con 0x7f0d680b3010 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57106/0 (socket says 172.21.15.129:57106) 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 -- 172.21.15.129:0/23723615 learned_addr learned my addr 172.21.15.129:0/23723615 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d66800640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d680b3010 0x7f0d6809d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 -- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d680b3010 msgr2=0x7f0d6809d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d680b3010 0x7f0d6809d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 -- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 msgr2=0x7f0d6815a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65e00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 0x7f0d6815a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 0x7f0d6815a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:24.443 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65400640 1 -- 172.21.15.129:0/23723615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d6809b140 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.441+0000 7f0d65e00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 0x7f0d6815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d65400640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c3c0 tx=0x7f0d5400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54017070 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0d540053c0 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d6809b420 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54003850 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.442+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d680cdd90 con 0x7f0d680b1c40 2024-09-17T04:08:24.444 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.444+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d680b40b0 con 0x7f0d680b1c40 2024-09-17T04:08:24.445 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.445+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0d5401a070 con 0x7f0d680b1c40 2024-09-17T04:08:24.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.446+0000 7f0d5e000640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 0x7f0d4807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:24.446 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.446+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(270..270 src has 1..270) ==== 5609+0+0 (secure 0 0 0) 0x7f0d5409a140 con 0x7f0d680b1c40 2024-09-17T04:08:24.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.446+0000 7f0d65e00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 0x7f0d4807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:24.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.447+0000 7f0d65e00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 0x7f0d4807b0a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009ab0 tx=0x7f0d58005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:24.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.451+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0d680cbaf0 con 0x7f0d680b1c40 2024-09-17T04:08:24.667 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:24.667+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0d6809b5f0 con 0x7f0d680b1c40 2024-09-17T04:08:25.186 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.186+0000 7f0d5e000640 1 -- 172.21.15.129:0/23723615 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 40 and data pool 41 v268) ==== 174+0+0 (secure 0 0 0) 0x7f0d54062540 con 0x7f0d680b1c40 2024-09-17T04:08:25.187 INFO:teuthology.orchestra.run.smithi129.stderr:new fs with metadata pool 40 and data pool 41 2024-09-17T04:08:25.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 msgr2=0x7f0d4807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.195 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 0x7f0d4807b0a0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009ab0 tx=0x7f0d58005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 msgr2=0x7f0d6815ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f0d5400c3c0 tx=0x7f0d5400c880 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 shutdown_connections 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0d48078b90 0x7f0d4807b0a0 secure :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0d58009ab0 tx=0x7f0d58005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0d680b3010 0x7f0d6809d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0d680b1c40 0x7f0d6815ace0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 --2- 172.21.15.129:0/23723615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d68074800 0x7f0d6815a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.196 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.195+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 >> 172.21.15.129:0/23723615 conn(0x7f0d680ccc40 msgr2=0x7f0d6815c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:25.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.196+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 shutdown_connections 2024-09-17T04:08:25.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.197+0000 7f0d6ca00640 1 -- 172.21.15.129:0/23723615 wait complete. 2024-09-17T04:08:25.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:25.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 -- 172.21.15.129:0/2657648899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09080706a0 msgr2=0x7f09080cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 --2- 172.21.15.129:0/2657648899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09080706a0 0x7f09080cc7b0 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f08f8005fd0 tx=0x7f08f802a670 comp rx=0 tx=0).stop 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 -- 172.21.15.129:0/2657648899 shutdown_connections 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 --2- 172.21.15.129:0/2657648899 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080cc260 0x7f09080b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 --2- 172.21.15.129:0/2657648899 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09080706a0 0x7f09080cc7b0 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 -- 172.21.15.129:0/2657648899 >> 172.21.15.129:0/2657648899 conn(0x7f090806d410 msgr2=0x7f09080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 -- 172.21.15.129:0/2657648899 shutdown_connections 2024-09-17T04:08:25.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.455+0000 7f090e200640 1 -- 172.21.15.129:0/2657648899 wait complete. 2024-09-17T04:08:25.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.458+0000 7f090e200640 1 Processor -- start 2024-09-17T04:08:25.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.458+0000 7f090e200640 1 -- start start 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 0x7f090815b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09080cc260 0x7f0908151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f090815fa70 con 0x7f09081517d0 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f09080cf440 con 0x7f09080cc260 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f090e200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f090815fbf0 con 0x7f09080b3930 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.459+0000 7f0907e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57124/0 (socket says 172.21.15.129:57124) 2024-09-17T04:08:25.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 -- 172.21.15.129:0/2019095436 learned_addr learned my addr 172.21.15.129:0/2019095436 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907400640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 0x7f090815b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f08fea00640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09080cc260 0x7f0908151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 -- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 msgr2=0x7f090815b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 0x7f090815b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 -- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09080cc260 msgr2=0x7f0908151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09080cc260 0x7f0908151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 -- 172.21.15.129:0/2019095436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09081f5410 con 0x7f09081517d0 2024-09-17T04:08:25.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907400640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 0x7f090815b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:25.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f0907e00640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f090000c790 tx=0x7f090000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:25.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.460+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0900017070 con 0x7f09081517d0 2024-09-17T04:08:25.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.461+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f090000ce40 con 0x7f09081517d0 2024-09-17T04:08:25.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.461+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09080c9520 con 0x7f09081517d0 2024-09-17T04:08:25.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.461+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09000052b0 con 0x7f09081517d0 2024-09-17T04:08:25.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.461+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09080c9ae0 con 0x7f09081517d0 2024-09-17T04:08:25.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.462+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0908160110 con 0x7f09081517d0 2024-09-17T04:08:25.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.463+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f090001a070 con 0x7f09081517d0 2024-09-17T04:08:25.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.464+0000 7f08ffe00640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 0x7f08e407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.464+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5709+0+0 (secure 0 0 0) 0x7f090009a2f0 con 0x7f09081517d0 2024-09-17T04:08:25.464 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.464+0000 7f0907400640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 0x7f08e407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.465+0000 7f0907400640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 0x7f08e407b0d0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f08f8033000 tx=0x7f08f8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:25.468 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.469+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f09080b3d40 con 0x7f09081517d0 2024-09-17T04:08:25.650 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.650+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f090806a140 con 0x7f09081517d0 2024-09-17T04:08:25.651 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.651+0000 7f08ffe00640 1 -- 172.21.15.129:0/2019095436 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+16792 (secure 0 0 0) 0x7f0900062730 con 0x7f09081517d0 2024-09-17T04:08:25.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:25.652 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":271,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:08:24.669141+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:08:22.575404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":41,"pool_name":"cephfs_data","create_time":"2024-09-17T04:08:23.637824+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 msgr2=0x7f08e407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 0x7f08e407b0d0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f08f8033000 tx=0x7f08f8004730 comp rx=0 tx=0).stop 2024-09-17T04:08:25.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 msgr2=0x7f0908151be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f090000c790 tx=0x7f090000cc50 comp rx=0 tx=0).stop 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 shutdown_connections 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f08e4078bc0 0x7f08e407b0d0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f09081517d0 0x7f0908151be0 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f09080cc260 0x7f0908151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 --2- 172.21.15.129:0/2019095436 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f09080b3930 0x7f090815b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 >> 172.21.15.129:0/2019095436 conn(0x7f090806d410 msgr2=0x7f09080b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 shutdown_connections 2024-09-17T04:08:25.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.654+0000 7f090e200640 1 -- 172.21.15.129:0/2019095436 wait complete. 2024-09-17T04:08:25.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:25.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 -- 172.21.15.129:0/1180972504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd790160ba0 msgr2=0x7fd7900d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.901 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 --2- 172.21.15.129:0/1180972504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd790160ba0 0x7fd7900d0380 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7fd7900d0050 tx=0x7fd77802a530 comp rx=0 tx=0).stop 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 -- 172.21.15.129:0/1180972504 shutdown_connections 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 --2- 172.21.15.129:0/1180972504 >> v2:172.21.15.181:3300/0 conn(0x7fd7900d08c0 0x7fd79015cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 --2- 172.21.15.129:0/1180972504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd790160ba0 0x7fd7900d0380 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 -- 172.21.15.129:0/1180972504 >> 172.21.15.129:0/1180972504 conn(0x7fd79006cfc0 msgr2=0x7fd79006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 -- 172.21.15.129:0/1180972504 shutdown_connections 2024-09-17T04:08:25.902 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.901+0000 7fd794a00640 1 -- 172.21.15.129:0/1180972504 wait complete. 2024-09-17T04:08:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.903+0000 7fd794a00640 1 Processor -- start 2024-09-17T04:08:25.903 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.903+0000 7fd794a00640 1 -- start start 2024-09-17T04:08:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7900cf8e0 0x7fd790156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.905 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd790160ba0 0x7fd7900afa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd79015d060 con 0x7fd7900d08c0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd7900cdc90 con 0x7fd790160ba0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd794a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd79015d1e0 con 0x7fd7900cf8e0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57144/0 (socket says 172.21.15.129:57144) 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 -- 172.21.15.129:0/3934924377 learned_addr learned my addr 172.21.15.129:0/3934924377 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd78f200640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd790160ba0 0x7fd7900afa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd78e800640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7900cf8e0 0x7fd790156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 -- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7900cf8e0 msgr2=0x7fd790156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7900cf8e0 0x7fd790156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 -- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd790160ba0 msgr2=0x7fd7900afa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:25.906 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd790160ba0 0x7fd7900afa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.905+0000 7fd785e00640 1 -- 172.21.15.129:0/3934924377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd790153990 con 0x7fd7900d08c0 2024-09-17T04:08:25.907 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd785e00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fd778034040 tx=0x7fd7780039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77803e030 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd7780047a0 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd790153c70 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778035040 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.906+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd790154230 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.907+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79015d6c0 con 0x7fd7900d08c0 2024-09-17T04:08:25.908 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.908+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd778041030 con 0x7fd7900d08c0 2024-09-17T04:08:25.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.910+0000 7fd787400640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 0x7fd76c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:25.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.910+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(271..271 src has 1..271) ==== 5709+0+0 (secure 0 0 0) 0x7fd7780bb560 con 0x7fd7900d08c0 2024-09-17T04:08:25.910 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.910+0000 7fd78e800640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 0x7fd76c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:25.911 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.911+0000 7fd78e800640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 0x7fd76c07b0a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd780006fd0 tx=0x7fd780008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:25.915 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:25.915+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd7900d0cd0 con 0x7fd7900d08c0 2024-09-17T04:08:26.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.099+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd79006a140 con 0x7fd7900d08c0 2024-09-17T04:08:26.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.100+0000 7fd787400640 1 -- 172.21.15.129:0/3934924377 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) ==== 74+0+16792 (secure 0 0 0) 0x7fd778083990 con 0x7fd7900d08c0 2024-09-17T04:08:26.100 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:26.101 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":271,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:08:24.669141+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:08:22.575404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":41,"pool_name":"cephfs_data","create_time":"2024-09-17T04:08:23.637824+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 msgr2=0x7fd76c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 0x7fd76c07b0a0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fd780006fd0 tx=0x7fd780008040 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 msgr2=0x7fd7901594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7fd778034040 tx=0x7fd7780039e0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 shutdown_connections 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd76c078b90 0x7fd76c07b0a0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd790160ba0 0x7fd7900afa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd7900d08c0 0x7fd7901594d0 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 --2- 172.21.15.129:0/3934924377 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd7900cf8e0 0x7fd790156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.102+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 >> 172.21.15.129:0/3934924377 conn(0x7fd79006cfc0 msgr2=0x7fd7900cdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:26.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.103+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 shutdown_connections 2024-09-17T04:08:26.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.103+0000 7fd794a00640 1 -- 172.21.15.129:0/3934924377 wait complete. 2024-09-17T04:08:26.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 -- 172.21.15.129:0/269416003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd924004800 msgr2=0x7fd924004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 --2- 172.21.15.129:0/269416003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd924004800 0x7fd924004c10 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c310 tx=0x7fd924005830 comp rx=0 tx=0).stop 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 -- 172.21.15.129:0/269416003 shutdown_connections 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 --2- 172.21.15.129:0/269416003 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd924004800 0x7fd924004c10 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 -- 172.21.15.129:0/269416003 >> 172.21.15.129:0/269416003 conn(0x7fd9400d00b0 msgr2=0x7fd9400d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 -- 172.21.15.129:0/269416003 shutdown_connections 2024-09-17T04:08:26.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.355+0000 7fd946600640 1 -- 172.21.15.129:0/269416003 wait complete. 2024-09-17T04:08:26.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.358+0000 7fd946600640 1 Processor -- start 2024-09-17T04:08:26.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.358+0000 7fd946600640 1 -- start start 2024-09-17T04:08:26.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd924004800 0x7fd94015f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 0x7fd9401f9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd940072e10 con 0x7fd94015cec0 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9400cb3a0 con 0x7fd94015d2d0 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd946600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd940072f90 con 0x7fd924004800 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.359+0000 7fd93fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57170/0 (socket says 172.21.15.129:57170) 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 -- 172.21.15.129:0/1993817988 learned_addr learned my addr 172.21.15.129:0/1993817988 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd945200640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd924004800 0x7fd94015f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd945c00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 0x7fd9401f9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 -- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd924004800 msgr2=0x7fd94015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:26.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd924004800 0x7fd94015f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 -- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 msgr2=0x7fd9401f9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 0x7fd9401f9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 -- 172.21.15.129:0/1993817988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94015d820 con 0x7fd94015cec0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd945c00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 0x7fd9401f9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93fe00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c430 tx=0x7fd92c001d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c031040 con 0x7fd94015cec0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd92c0065e0 con 0x7fd94015cec0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94015f6a0 con 0x7fd94015cec0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.360+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92c003a60 con 0x7fd94015cec0 2024-09-17T04:08:26.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.361+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94015dda0 con 0x7fd94015cec0 2024-09-17T04:08:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.362+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd940073480 con 0x7fd94015cec0 2024-09-17T04:08:26.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.363+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd92c03e030 con 0x7fd94015cec0 2024-09-17T04:08:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.364+0000 7fd93cc00640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 0x7fd92007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.364+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fd92c0bad00 con 0x7fd94015cec0 2024-09-17T04:08:26.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.365+0000 7fd945200640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 0x7fd92007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:26.365 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.365+0000 7fd945200640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 0x7fd92007b0a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd94006b8f0 tx=0x7fd930005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:26.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.368+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd92c0830a0 con 0x7fd94015cec0 2024-09-17T04:08:26.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:26.585+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd9400d0f50 con 0x7fd94015cec0 2024-09-17T04:08:27.155 INFO:tasks.ceph.mds.a.smithi129.stderr:2024-09-17T04:08:27.155+0000 7f5c8fa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-17T04:08:27.197 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.197+0000 7fd93cc00640 1 -- 172.21.15.129:0/1993817988 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v271) ==== 118+0+0 (secure 0 0 0) 0x7fd92c0829a0 con 0x7fd94015cec0 2024-09-17T04:08:27.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 msgr2=0x7fd92007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:27.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 0x7fd92007b0a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fd94006b8f0 tx=0x7fd930005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:27.200 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 msgr2=0x7fd9401f8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7fd92c00c430 tx=0x7fd92c001d90 comp rx=0 tx=0).stop 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 shutdown_connections 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd920078b90 0x7fd92007b0a0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd94015d2d0 0x7fd9401f9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd94015cec0 0x7fd9401f8e10 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 --2- 172.21.15.129:0/1993817988 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd924004800 0x7fd94015f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 >> 172.21.15.129:0/1993817988 conn(0x7fd9400d00b0 msgr2=0x7fd9400b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 shutdown_connections 2024-09-17T04:08:27.201 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.200+0000 7fd946600640 1 -- 172.21.15.129:0/1993817988 wait complete. 2024-09-17T04:08:27.228 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-17T04:08:27.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 -- 172.21.15.129:0/3853935375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 msgr2=0x7f00300ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 --2- 172.21.15.129:0/3853935375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300ce910 secure :-1 s=READY pgs=2937 cs=0 l=1 rev1=1 crypto rx=0x7f001c005fd0 tx=0x7f001c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 -- 172.21.15.129:0/3853935375 shutdown_connections 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 --2- 172.21.15.129:0/3853935375 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300ce910 unknown :-1 s=CLOSED pgs=2937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 --2- 172.21.15.129:0/3853935375 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b2650 0x7f00300cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 -- 172.21.15.129:0/3853935375 >> 172.21.15.129:0/3853935375 conn(0x7f00300ccc40 msgr2=0x7f00300cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 -- 172.21.15.129:0/3853935375 shutdown_connections 2024-09-17T04:08:27.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.452+0000 7f0036400640 1 -- 172.21.15.129:0/3853935375 wait complete. 2024-09-17T04:08:27.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.455+0000 7f0036400640 1 Processor -- start 2024-09-17T04:08:27.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.455+0000 7f0036400640 1 -- start start 2024-09-17T04:08:27.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00300b2650 0x7f00300c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:27.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 0x7f00300c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:27.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:27.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f00300cad80 con 0x7f00300ce500 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f00300cac00 con 0x7f00300b2650 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f0036400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f00300caf00 con 0x7f00300b4220 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f002fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f002fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57188/0 (socket says 172.21.15.129:57188) 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.456+0000 7f002fe00640 1 -- 172.21.15.129:0/3700736917 learned_addr learned my addr 172.21.15.129:0/3700736917 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002ea00640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 0x7f00300c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:27.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002f400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00300b2650 0x7f00300c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 -- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 msgr2=0x7f00300c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 0x7f00300c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 -- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00300b2650 msgr2=0x7f00300c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00300b2650 0x7f00300c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 -- 172.21.15.129:0/3700736917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00301f4930 con 0x7f00300ce500 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002ea00640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 0x7f00300c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f002fe00640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f002800c730 tx=0x7f002800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028003b70 con 0x7f00300ce500 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0028003d10 con 0x7f00300ce500 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028005330 con 0x7f00300ce500 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.457+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00300c4ea0 con 0x7f00300ce500 2024-09-17T04:08:27.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.458+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00300c5460 con 0x7f00300ce500 2024-09-17T04:08:27.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.459+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00300cbf30 con 0x7f00300ce500 2024-09-17T04:08:27.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.460+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f00280056e0 con 0x7f00300ce500 2024-09-17T04:08:27.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.461+0000 7f0027400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 0x7f001007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:27.462 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.461+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f002809ac40 con 0x7f00300ce500 2024-09-17T04:08:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.461+0000 7f002f400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 0x7f001007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:27.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.462+0000 7f002f400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 0x7f001007b0a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f00300726b0 tx=0x7f0020005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:27.465 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.465+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0028063100 con 0x7f00300ce500 2024-09-17T04:08:27.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:27.685+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f0030074040 con 0x7f00300ce500 2024-09-17T04:08:28.199 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.199+0000 7f0027400640 1 -- 172.21.15.129:0/3700736917 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v272) ==== 122+0+0 (secure 0 0 0) 0x7f0028062a00 con 0x7f00300ce500 2024-09-17T04:08:28.202 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 msgr2=0x7f001007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 0x7f001007b0a0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f00300726b0 tx=0x7f0020005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 msgr2=0x7f00300c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f002800c730 tx=0x7f002800cbf0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 shutdown_connections 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0010078b90 0x7f001007b0a0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f00300ce500 0x7f00300c5f40 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f00300b4220 0x7f00300c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 --2- 172.21.15.129:0/3700736917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00300b2650 0x7f00300c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 >> 172.21.15.129:0/3700736917 conn(0x7f00300ccc40 msgr2=0x7f00300d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 shutdown_connections 2024-09-17T04:08:28.203 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.202+0000 7f0036400640 1 -- 172.21.15.129:0/3700736917 wait complete. 2024-09-17T04:08:28.226 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-17T04:08:28.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:28.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 -- 172.21.15.129:0/4262173191 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080cd1c0 msgr2=0x7f1a081609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 --2- 172.21.15.129:0/4262173191 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080cd1c0 0x7f1a081609c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f1a00005fd0 tx=0x7f1a0002a8b0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 -- 172.21.15.129:0/4262173191 shutdown_connections 2024-09-17T04:08:28.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 --2- 172.21.15.129:0/4262173191 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080cd1c0 0x7f1a081609c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 --2- 172.21.15.129:0/4262173191 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080b4140 0x7f1a080ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 -- 172.21.15.129:0/4262173191 >> 172.21.15.129:0/4262173191 conn(0x7f1a0806d410 msgr2=0x7f1a080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:28.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 -- 172.21.15.129:0/4262173191 shutdown_connections 2024-09-17T04:08:28.451 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.450+0000 7f1a0e000640 1 -- 172.21.15.129:0/4262173191 wait complete. 2024-09-17T04:08:28.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.452+0000 7f1a0e000640 1 Processor -- start 2024-09-17T04:08:28.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.452+0000 7f1a0e000640 1 -- start start 2024-09-17T04:08:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080705a0 0x7f1a080c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080b4140 0x7f1a081f8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1a08160260 con 0x7f1a080cd1c0 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1a081600e0 con 0x7f1a080b4140 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a0e000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1a08160f00 con 0x7f1a080705a0 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a07e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57198/0 (socket says 172.21.15.129:57198) 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a07e00640 1 -- 172.21.15.129:0/2607402944 learned_addr learned my addr 172.21.15.129:0/2607402944 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f19fea00640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080b4140 0x7f1a081f8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.454+0000 7f1a07400640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080705a0 0x7f1a080c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 -- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080705a0 msgr2=0x7f1a080c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080705a0 0x7f1a080c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 -- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080b4140 msgr2=0x7f1a081f8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080b4140 0x7f1a081f8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 -- 172.21.15.129:0/2607402944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a080c3cd0 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a07e00640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009cd0 tx=0x7f1a00031650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a00003e00 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1a00031a80 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a080c3fb0 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a00033070 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.455+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a080c4570 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.457+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a080ccc80 con 0x7f1a080cd1c0 2024-09-17T04:08:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.458+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1a00033210 con 0x7f1a080cd1c0 2024-09-17T04:08:28.458 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.459+0000 7f19ffe00640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 0x7f19e407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.459+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f1a000bc110 con 0x7f1a080cd1c0 2024-09-17T04:08:28.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.459+0000 7f1a07400640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 0x7f19e407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.460 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.460+0000 7f1a07400640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 0x7f19e407b0a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f19f8007d60 tx=0x7f19f8007cf0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:28.463 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.463+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1a0003f030 con 0x7f1a080cd1c0 2024-09-17T04:08:28.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.681+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a081f9990 con 0x7f1a080cd1c0 2024-09-17T04:08:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.682+0000 7f19ffe00640 1 -- 172.21.15.129:0/2607402944 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) ==== 95+0+5249 (secure 0 0 0) 0x7f1a00083ef0 con 0x7f1a080cd1c0 2024-09-17T04:08:28.682 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:28.682 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":272,"btime":"2024-09-17T04:08:28:198922+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:27.686833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:28.682 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 272 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 msgr2=0x7f19e407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 0x7f19e407b0a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f19f8007d60 tx=0x7f19f8007cf0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 msgr2=0x7f1a080c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009cd0 tx=0x7f1a00031650 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 shutdown_connections 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f19e4078b90 0x7f19e407b0a0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1a080cd1c0 0x7f1a080c9ab0 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1a080b4140 0x7f1a081f8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 --2- 172.21.15.129:0/2607402944 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1a080705a0 0x7f1a080c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 >> 172.21.15.129:0/2607402944 conn(0x7f1a0806d410 msgr2=0x7f1a080741b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 shutdown_connections 2024-09-17T04:08:28.685 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.684+0000 7f1a0e000640 1 -- 172.21.15.129:0/2607402944 wait complete. 2024-09-17T04:08:28.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:08:28.931 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 -- 172.21.15.129:0/4180301946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415d7e0 msgr2=0x7fd2440d0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 --2- 172.21.15.129:0/4180301946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415d7e0 0x7fd2440d0700 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7fd23c005fd0 tx=0x7fd23c02aa10 comp rx=0 tx=0).stop 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 -- 172.21.15.129:0/4180301946 shutdown_connections 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 --2- 172.21.15.129:0/4180301946 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415d7e0 0x7fd2440d0700 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 --2- 172.21.15.129:0/4180301946 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd2440cf510 0x7fd2440ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 -- 172.21.15.129:0/4180301946 >> 172.21.15.129:0/4180301946 conn(0x7fd244069330 msgr2=0x7fd244071160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 -- 172.21.15.129:0/4180301946 shutdown_connections 2024-09-17T04:08:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.931+0000 7fd248a00640 1 -- 172.21.15.129:0/4180301946 wait complete. 2024-09-17T04:08:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.933+0000 7fd248a00640 1 Processor -- start 2024-09-17T04:08:28.933 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.933+0000 7fd248a00640 1 -- start start 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd24415d7e0 0x7fd2441584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd244159b40 0x7fd244159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd2440d1050 con 0x7fd24415c8a0 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd2440d0ed0 con 0x7fd244159b40 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd248a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd2440d11d0 con 0x7fd24415d7e0 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57222/0 (socket says 172.21.15.129:57222) 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 -- 172.21.15.129:0/1482127151 learned_addr learned my addr 172.21.15.129:0/1482127151 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 -- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd24415d7e0 msgr2=0x7fd2441584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd24415d7e0 0x7fd2441584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 -- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd244159b40 msgr2=0x7fd244159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:28.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.935+0000 7fd242800640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd244159b40 0x7fd244159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:28.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd242800640 1 -- 172.21.15.129:0/1482127151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24415a490 con 0x7fd24415c8a0 2024-09-17T04:08:28.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd242800640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c3c0 tx=0x7fd23400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:28.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234017070 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd2340035f0 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2440c9430 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd234003a80 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.936+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2441f7250 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.937+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2440d1730 con 0x7fd24415c8a0 2024-09-17T04:08:28.938 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.939+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd23401a070 con 0x7fd24415c8a0 2024-09-17T04:08:28.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.939+0000 7fd23b400640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 0x7fd22007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:28.939 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.940+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fd23409a080 con 0x7fd24415c8a0 2024-09-17T04:08:28.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.940+0000 7fd239e00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 0x7fd22007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:28.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.941+0000 7fd239e00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 0x7fd22007b0d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0058c0 tx=0x7fd22c005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:28.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:28.944+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd24415a490 con 0x7fd24415c8a0 2024-09-17T04:08:29.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.128+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2440cd630 con 0x7fd24415c8a0 2024-09-17T04:08:29.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.130+0000 7fd23b400640 1 -- 172.21.15.129:0/1482127151 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) ==== 74+0+16779 (secure 0 0 0) 0x7fd2340624b0 con 0x7fd24415c8a0 2024-09-17T04:08:29.130 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:29.130 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":272,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:08:26.173819+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:08:22.575404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":41,"pool_name":"cephfs_data","create_time":"2024-09-17T04:08:23.637824+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"272","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:08:29.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 msgr2=0x7fd22007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 0x7fd22007b0d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0058c0 tx=0x7fd22c005850 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 msgr2=0x7fd2440b0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c3c0 tx=0x7fd23400c880 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 shutdown_connections 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd220078bc0 0x7fd22007b0d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd244159b40 0x7fd244159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd24415d7e0 0x7fd2441584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 --2- 172.21.15.129:0/1482127151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd24415c8a0 0x7fd2440b0d80 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 >> 172.21.15.129:0/1482127151 conn(0x7fd244069330 msgr2=0x7fd2440b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 shutdown_connections 2024-09-17T04:08:29.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.132+0000 7fd248a00640 1 -- 172.21.15.129:0/1482127151 wait complete. 2024-09-17T04:08:29.157 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:08:29.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 -- 172.21.15.129:0/3520509910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 msgr2=0x7fb8b0003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 --2- 172.21.15.129:0/3520509910 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8b0003e70 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c290 tx=0x7fb8b0005830 comp rx=0 tx=0).stop 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8cbe00640 1 -- 172.21.15.129:0/3520509910 reap_dead start 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 -- 172.21.15.129:0/3520509910 shutdown_connections 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 -- 172.21.15.129:0/3520509910 >> 172.21.15.129:0/3520509910 conn(0x7fb8cc06d410 msgr2=0x7fb8cc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 -- 172.21.15.129:0/3520509910 shutdown_connections 2024-09-17T04:08:29.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.380+0000 7fb8d2600640 1 -- 172.21.15.129:0/3520509910 wait complete. 2024-09-17T04:08:29.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.382+0000 7fb8d2600640 1 Processor -- start 2024-09-17T04:08:29.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.383+0000 7fb8d2600640 1 -- start start 2024-09-17T04:08:29.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 0x7fb8cc0c8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 0x7fb8cc1f9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc0cd2a0 con 0x7fb8b0003a00 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc06f6c0 con 0x7fb8cc0c9080 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8d2600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc0cd420 con 0x7fb8cc09dc60 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8cb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8cb400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57238/0 (socket says 172.21.15.129:57238) 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.384+0000 7fb8cb400640 1 -- 172.21.15.129:0/1223080984 learned_addr learned my addr 172.21.15.129:0/1223080984 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cbe00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 0x7fb8cc1f9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8caa00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 0x7fb8cc0c8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 -- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 msgr2=0x7fb8cc0c8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 0x7fb8cc0c8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 -- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 msgr2=0x7fb8cc1f9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 0x7fb8cc1f9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 -- 172.21.15.129:0/1223080984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc0c9490 con 0x7fb8b0003a00 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8caa00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 0x7fb8cc0c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cbe00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 0x7fb8cc1f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8cb400640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c290 tx=0x7fb8bc006d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc031040 con 0x7fb8b0003a00 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8bc0035f0 con 0x7fb8b0003a00 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc003a00 con 0x7fb8b0003a00 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc0cb9a0 con 0x7fb8b0003a00 2024-09-17T04:08:29.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.385+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc0cbf60 con 0x7fb8b0003a00 2024-09-17T04:08:29.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.386+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8cc0cf6c0 con 0x7fb8b0003a00 2024-09-17T04:08:29.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.388+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8bc034070 con 0x7fb8b0003a00 2024-09-17T04:08:29.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.389+0000 7fb8c3400640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 0x7fb8ac07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.389+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fb8bc0bae70 con 0x7fb8b0003a00 2024-09-17T04:08:29.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.389+0000 7fb8caa00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 0x7fb8ac07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.390+0000 7fb8caa00640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 0x7fb8ac07b0a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8005eb0 tx=0x7fb8b8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:29.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.393+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb8cc0c9490 con 0x7fb8b0003a00 2024-09-17T04:08:29.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.574+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8cc06a140 con 0x7fb8ac078b90 2024-09-17T04:08:29.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.579+0000 7fb8c3400640 1 -- 172.21.15.129:0/1223080984 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248654 (secure 0 0 0) 0x7fb8cc06a140 con 0x7fb8ac078b90 2024-09-17T04:08:29.582 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:29.583 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:08:29.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 msgr2=0x7fb8ac07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 0x7fb8ac07b0a0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8005eb0 tx=0x7fb8b8005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:29.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 msgr2=0x7fb8cc1f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc00c290 tx=0x7fb8bc006d70 comp rx=0 tx=0).stop 2024-09-17T04:08:29.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 shutdown_connections 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb8ac078b90 0x7fb8ac07b0a0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8cc0c9080 0x7fb8cc1f9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8cc09dc60 0x7fb8cc0c8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 --2- 172.21.15.129:0/1223080984 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b0003a00 0x7fb8cc1f66b0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 >> 172.21.15.129:0/1223080984 conn(0x7fb8cc06d410 msgr2=0x7fb8cc15c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 shutdown_connections 2024-09-17T04:08:29.586 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.583+0000 7fb8d2600640 1 -- 172.21.15.129:0/1223080984 wait complete. 2024-09-17T04:08:29.605 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2024-09-17T04:08:28.620466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":117,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420456,"kb_used_data":27956,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754554264,"statfs":{"total":773094113280,"available":772663566336,"internally_reserved":0,"allocated":28626944,"data_stored":13347692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456722"},"pg_stats":[{"pgid":"41.3f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326478+0000","last_change":"2024-09-17T04:08:25.189216+0000","last_active":"2024-09-17T04:08:26.326478+0000","last_peered":"2024-09-17T04:08:26.326478+0000","last_clean":"2024-09-17T04:08:26.326478+0000","last_became_active":"2024-09-17T04:08:25.188921+0000","last_became_peered":"2024-09-17T04:08:25.188921+0000","last_unstale":"2024-09-17T04:08:26.326478+0000","last_undegraded":"2024-09-17T04:08:26.326478+0000","last_fullsized":"2024-09-17T04:08:26.326478+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:41.624638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189193+0000","last_change":"2024-09-17T04:08:24.189193+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189193+0000","last_undegraded":"2024-09-17T04:08:24.189193+0000","last_fullsized":"2024-09-17T04:08:24.189193+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181899+0000","last_change":"2024-09-17T04:08:24.181899+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181899+0000","last_undegraded":"2024-09-17T04:08:24.181899+0000","last_fullsized":"2024-09-17T04:08:24.181899+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178305+0000","last_change":"2024-09-17T04:08:24.178305+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178305+0000","last_undegraded":"2024-09-17T04:08:24.178305+0000","last_fullsized":"2024-09-17T04:08:24.178305+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.3b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182910+0000","last_change":"2024-09-17T04:08:24.182910+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182910+0000","last_undegraded":"2024-09-17T04:08:24.182910+0000","last_fullsized":"2024-09-17T04:08:24.182910+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178626+0000","last_change":"2024-09-17T04:08:24.178626+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178626+0000","last_undegraded":"2024-09-17T04:08:24.178626+0000","last_fullsized":"2024-09-17T04:08:24.178626+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.39","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183640+0000","last_change":"2024-09-17T04:08:24.183640+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183640+0000","last_undegraded":"2024-09-17T04:08:24.183640+0000","last_fullsized":"2024-09-17T04:08:24.183640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.38","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326289+0000","last_change":"2024-09-17T04:08:25.207974+0000","last_active":"2024-09-17T04:08:26.326289+0000","last_peered":"2024-09-17T04:08:26.326289+0000","last_clean":"2024-09-17T04:08:26.326289+0000","last_became_active":"2024-09-17T04:08:25.207454+0000","last_became_peered":"2024-09-17T04:08:25.207454+0000","last_unstale":"2024-09-17T04:08:26.326289+0000","last_undegraded":"2024-09-17T04:08:26.326289+0000","last_fullsized":"2024-09-17T04:08:26.326289+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:18.281022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.37","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182619+0000","last_change":"2024-09-17T04:08:24.182619+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182619+0000","last_undegraded":"2024-09-17T04:08:24.182619+0000","last_fullsized":"2024-09-17T04:08:24.182619+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.36","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185033+0000","last_change":"2024-09-17T04:08:24.185033+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185033+0000","last_undegraded":"2024-09-17T04:08:24.185033+0000","last_fullsized":"2024-09-17T04:08:24.185033+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.35","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.187179+0000","last_change":"2024-09-17T04:08:24.187179+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.187179+0000","last_undegraded":"2024-09-17T04:08:24.187179+0000","last_fullsized":"2024-09-17T04:08:24.187179+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.34","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326226+0000","last_change":"2024-09-17T04:08:25.196813+0000","last_active":"2024-09-17T04:08:26.326226+0000","last_peered":"2024-09-17T04:08:26.326226+0000","last_clean":"2024-09-17T04:08:26.326226+0000","last_became_active":"2024-09-17T04:08:25.196404+0000","last_became_peered":"2024-09-17T04:08:25.196404+0000","last_unstale":"2024-09-17T04:08:26.326226+0000","last_undegraded":"2024-09-17T04:08:26.326226+0000","last_fullsized":"2024-09-17T04:08:26.326226+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:24.380529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.33","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186085+0000","last_change":"2024-09-17T04:08:24.186085+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186085+0000","last_undegraded":"2024-09-17T04:08:24.186085+0000","last_fullsized":"2024-09-17T04:08:24.186085+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.32","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182701+0000","last_change":"2024-09-17T04:08:24.182701+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182701+0000","last_undegraded":"2024-09-17T04:08:24.182701+0000","last_fullsized":"2024-09-17T04:08:24.182701+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.31","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180999+0000","last_change":"2024-09-17T04:08:24.180999+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180999+0000","last_undegraded":"2024-09-17T04:08:24.180999+0000","last_fullsized":"2024-09-17T04:08:24.180999+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.30","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.192580+0000","last_change":"2024-09-17T04:08:24.192580+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.192580+0000","last_undegraded":"2024-09-17T04:08:24.192580+0000","last_fullsized":"2024-09-17T04:08:24.192580+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186791+0000","last_change":"2024-09-17T04:08:24.186791+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186791+0000","last_undegraded":"2024-09-17T04:08:24.186791+0000","last_fullsized":"2024-09-17T04:08:24.186791+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183594+0000","last_change":"2024-09-17T04:08:24.183594+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183594+0000","last_undegraded":"2024-09-17T04:08:24.183594+0000","last_fullsized":"2024-09-17T04:08:24.183594+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.2d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326090+0000","last_change":"2024-09-17T04:08:25.192767+0000","last_active":"2024-09-17T04:08:26.326090+0000","last_peered":"2024-09-17T04:08:26.326090+0000","last_clean":"2024-09-17T04:08:26.326090+0000","last_became_active":"2024-09-17T04:08:25.192484+0000","last_became_peered":"2024-09-17T04:08:25.192484+0000","last_unstale":"2024-09-17T04:08:26.326090+0000","last_undegraded":"2024-09-17T04:08:26.326090+0000","last_fullsized":"2024-09-17T04:08:26.326090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:08.026774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.2c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191776+0000","last_change":"2024-09-17T04:08:24.191776+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191776+0000","last_undegraded":"2024-09-17T04:08:24.191776+0000","last_fullsized":"2024-09-17T04:08:24.191776+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180410+0000","last_change":"2024-09-17T04:08:24.180410+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180410+0000","last_undegraded":"2024-09-17T04:08:24.180410+0000","last_fullsized":"2024-09-17T04:08:24.180410+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.184503+0000","last_change":"2024-09-17T04:08:24.184503+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.184503+0000","last_undegraded":"2024-09-17T04:08:24.184503+0000","last_fullsized":"2024-09-17T04:08:24.184503+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.29","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179821+0000","last_change":"2024-09-17T04:08:24.179821+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179821+0000","last_undegraded":"2024-09-17T04:08:24.179821+0000","last_fullsized":"2024-09-17T04:08:24.179821+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.27","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185990+0000","last_change":"2024-09-17T04:08:24.185990+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185990+0000","last_undegraded":"2024-09-17T04:08:24.185990+0000","last_fullsized":"2024-09-17T04:08:24.185990+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.26","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326742+0000","last_change":"2024-09-17T04:08:25.205002+0000","last_active":"2024-09-17T04:08:26.326742+0000","last_peered":"2024-09-17T04:08:26.326742+0000","last_clean":"2024-09-17T04:08:26.326742+0000","last_became_active":"2024-09-17T04:08:25.204427+0000","last_became_peered":"2024-09-17T04:08:25.204427+0000","last_unstale":"2024-09-17T04:08:26.326742+0000","last_undegraded":"2024-09-17T04:08:26.326742+0000","last_fullsized":"2024-09-17T04:08:26.326742+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:37.951456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.25","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180951+0000","last_change":"2024-09-17T04:08:24.180951+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180951+0000","last_undegraded":"2024-09-17T04:08:24.180951+0000","last_fullsized":"2024-09-17T04:08:24.180951+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.24","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326183+0000","last_change":"2024-09-17T04:08:25.188324+0000","last_active":"2024-09-17T04:08:26.326183+0000","last_peered":"2024-09-17T04:08:26.326183+0000","last_clean":"2024-09-17T04:08:26.326183+0000","last_became_active":"2024-09-17T04:08:25.188025+0000","last_became_peered":"2024-09-17T04:08:25.188025+0000","last_unstale":"2024-09-17T04:08:26.326183+0000","last_undegraded":"2024-09-17T04:08:26.326183+0000","last_fullsized":"2024-09-17T04:08:26.326183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:41.304344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.23","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189672+0000","last_change":"2024-09-17T04:08:24.189672+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189672+0000","last_undegraded":"2024-09-17T04:08:24.189672+0000","last_fullsized":"2024-09-17T04:08:24.189672+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.22","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326936+0000","last_change":"2024-09-17T04:08:25.194536+0000","last_active":"2024-09-17T04:08:26.326936+0000","last_peered":"2024-09-17T04:08:26.326936+0000","last_clean":"2024-09-17T04:08:26.326936+0000","last_became_active":"2024-09-17T04:08:25.194201+0000","last_became_peered":"2024-09-17T04:08:25.194201+0000","last_unstale":"2024-09-17T04:08:26.326936+0000","last_undegraded":"2024-09-17T04:08:26.326936+0000","last_fullsized":"2024-09-17T04:08:26.326936+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:14.659279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.21","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326187+0000","last_change":"2024-09-17T04:08:25.198539+0000","last_active":"2024-09-17T04:08:26.326187+0000","last_peered":"2024-09-17T04:08:26.326187+0000","last_clean":"2024-09-17T04:08:26.326187+0000","last_became_active":"2024-09-17T04:08:25.198178+0000","last_became_peered":"2024-09-17T04:08:25.198178+0000","last_unstale":"2024-09-17T04:08:26.326187+0000","last_undegraded":"2024-09-17T04:08:26.326187+0000","last_fullsized":"2024-09-17T04:08:26.326187+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:18.437486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.20","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182935+0000","last_change":"2024-09-17T04:08:24.182935+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182935+0000","last_undegraded":"2024-09-17T04:08:24.182935+0000","last_fullsized":"2024-09-17T04:08:24.182935+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326610+0000","last_change":"2024-09-17T04:08:25.199592+0000","last_active":"2024-09-17T04:08:26.326610+0000","last_peered":"2024-09-17T04:08:26.326610+0000","last_clean":"2024-09-17T04:08:26.326610+0000","last_became_active":"2024-09-17T04:08:25.199254+0000","last_became_peered":"2024-09-17T04:08:25.199254+0000","last_unstale":"2024-09-17T04:08:26.326610+0000","last_undegraded":"2024-09-17T04:08:26.326610+0000","last_fullsized":"2024-09-17T04:08:26.326610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:22.464215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"271'4","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326348+0000","last_change":"2024-09-17T04:08:24.190135+0000","last_active":"2024-09-17T04:08:26.326348+0000","last_peered":"2024-09-17T04:08:26.326348+0000","last_clean":"2024-09-17T04:08:26.326348+0000","last_became_active":"2024-09-17T04:08:24.189843+0000","last_became_peered":"2024-09-17T04:08:24.189843+0000","last_unstale":"2024-09-17T04:08:26.326348+0000","last_undegraded":"2024-09-17T04:08:26.326348+0000","last_fullsized":"2024-09-17T04:08:26.326348+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:29.916388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181601+0000","last_change":"2024-09-17T04:08:24.181601+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181601+0000","last_undegraded":"2024-09-17T04:08:24.181601+0000","last_fullsized":"2024-09-17T04:08:24.181601+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326691+0000","last_change":"2024-09-17T04:08:24.196111+0000","last_active":"2024-09-17T04:08:26.326691+0000","last_peered":"2024-09-17T04:08:26.326691+0000","last_clean":"2024-09-17T04:08:26.326691+0000","last_became_active":"2024-09-17T04:08:24.195595+0000","last_became_peered":"2024-09-17T04:08:24.195595+0000","last_unstale":"2024-09-17T04:08:26.326691+0000","last_undegraded":"2024-09-17T04:08:26.326691+0000","last_fullsized":"2024-09-17T04:08:26.326691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:44.710662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180884+0000","last_change":"2024-09-17T04:08:24.180884+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180884+0000","last_undegraded":"2024-09-17T04:08:24.180884+0000","last_fullsized":"2024-09-17T04:08:24.180884+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.196010+0000","last_change":"2024-09-17T04:08:24.194368+0000","last_active":"2024-09-17T04:08:24.196010+0000","last_peered":"2024-09-17T04:08:24.196010+0000","last_clean":"2024-09-17T04:08:24.196010+0000","last_became_active":"2024-09-17T04:08:24.193648+0000","last_became_peered":"2024-09-17T04:08:24.193648+0000","last_unstale":"2024-09-17T04:08:24.196010+0000","last_undegraded":"2024-09-17T04:08:24.196010+0000","last_fullsized":"2024-09-17T04:08:24.196010+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:40.412794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186629+0000","last_change":"2024-09-17T04:08:24.186629+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186629+0000","last_undegraded":"2024-09-17T04:08:24.186629+0000","last_fullsized":"2024-09-17T04:08:24.186629+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203824+0000","last_change":"2024-09-17T04:08:24.202248+0000","last_active":"2024-09-17T04:08:24.203824+0000","last_peered":"2024-09-17T04:08:24.203824+0000","last_clean":"2024-09-17T04:08:24.203824+0000","last_became_active":"2024-09-17T04:08:24.201709+0000","last_became_peered":"2024-09-17T04:08:24.201709+0000","last_unstale":"2024-09-17T04:08:24.203824+0000","last_undegraded":"2024-09-17T04:08:24.203824+0000","last_fullsized":"2024-09-17T04:08:24.203824+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:16.873589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180525+0000","last_change":"2024-09-17T04:08:24.180525+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180525+0000","last_undegraded":"2024-09-17T04:08:24.180525+0000","last_fullsized":"2024-09-17T04:08:24.180525+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.201402+0000","last_change":"2024-09-17T04:08:24.200461+0000","last_active":"2024-09-17T04:08:24.201402+0000","last_peered":"2024-09-17T04:08:24.201402+0000","last_clean":"2024-09-17T04:08:24.201402+0000","last_became_active":"2024-09-17T04:08:24.199506+0000","last_became_peered":"2024-09-17T04:08:24.199506+0000","last_unstale":"2024-09-17T04:08:24.201402+0000","last_undegraded":"2024-09-17T04:08:24.201402+0000","last_fullsized":"2024-09-17T04:08:24.201402+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:31.705720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325694+0000","last_change":"2024-09-17T04:08:25.185548+0000","last_active":"2024-09-17T04:08:26.325694+0000","last_peered":"2024-09-17T04:08:26.325694+0000","last_clean":"2024-09-17T04:08:26.325694+0000","last_became_active":"2024-09-17T04:08:25.184461+0000","last_became_peered":"2024-09-17T04:08:25.184461+0000","last_unstale":"2024-09-17T04:08:26.325694+0000","last_undegraded":"2024-09-17T04:08:26.325694+0000","last_fullsized":"2024-09-17T04:08:26.325694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:46.956363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325684+0000","last_change":"2024-09-17T04:08:24.187071+0000","last_active":"2024-09-17T04:08:26.325684+0000","last_peered":"2024-09-17T04:08:26.325684+0000","last_clean":"2024-09-17T04:08:26.325684+0000","last_became_active":"2024-09-17T04:08:24.186743+0000","last_became_peered":"2024-09-17T04:08:24.186743+0000","last_unstale":"2024-09-17T04:08:26.325684+0000","last_undegraded":"2024-09-17T04:08:26.325684+0000","last_fullsized":"2024-09-17T04:08:26.325684+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:15.930301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191846+0000","last_change":"2024-09-17T04:08:24.191846+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191846+0000","last_undegraded":"2024-09-17T04:08:24.191846+0000","last_fullsized":"2024-09-17T04:08:24.191846+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199751+0000","last_change":"2024-09-17T04:08:24.192860+0000","last_active":"2024-09-17T04:08:24.199751+0000","last_peered":"2024-09-17T04:08:24.199751+0000","last_clean":"2024-09-17T04:08:24.199751+0000","last_became_active":"2024-09-17T04:08:24.192505+0000","last_became_peered":"2024-09-17T04:08:24.192505+0000","last_unstale":"2024-09-17T04:08:24.199751+0000","last_undegraded":"2024-09-17T04:08:24.199751+0000","last_fullsized":"2024-09-17T04:08:24.199751+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:33.412525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325700+0000","last_change":"2024-09-17T04:08:25.195702+0000","last_active":"2024-09-17T04:08:26.325700+0000","last_peered":"2024-09-17T04:08:26.325700+0000","last_clean":"2024-09-17T04:08:26.325700+0000","last_became_active":"2024-09-17T04:08:25.195370+0000","last_became_peered":"2024-09-17T04:08:25.195370+0000","last_unstale":"2024-09-17T04:08:26.325700+0000","last_undegraded":"2024-09-17T04:08:26.325700+0000","last_fullsized":"2024-09-17T04:08:26.325700+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:35.074389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.198388+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.197800+0000","last_became_peered":"2024-09-17T04:08:24.197800+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:50.784271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185710+0000","last_change":"2024-09-17T04:08:24.185710+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185710+0000","last_undegraded":"2024-09-17T04:08:24.185710+0000","last_fullsized":"2024-09-17T04:08:24.185710+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200361+0000","last_change":"2024-09-17T04:08:24.197553+0000","last_active":"2024-09-17T04:08:24.200361+0000","last_peered":"2024-09-17T04:08:24.200361+0000","last_clean":"2024-09-17T04:08:24.200361+0000","last_became_active":"2024-09-17T04:08:24.197233+0000","last_became_peered":"2024-09-17T04:08:24.197233+0000","last_unstale":"2024-09-17T04:08:24.200361+0000","last_undegraded":"2024-09-17T04:08:24.200361+0000","last_fullsized":"2024-09-17T04:08:24.200361+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:08.411479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181356+0000","last_change":"2024-09-17T04:08:24.181356+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181356+0000","last_undegraded":"2024-09-17T04:08:24.181356+0000","last_fullsized":"2024-09-17T04:08:24.181356+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326324+0000","last_change":"2024-09-17T04:08:24.191873+0000","last_active":"2024-09-17T04:08:26.326324+0000","last_peered":"2024-09-17T04:08:26.326324+0000","last_clean":"2024-09-17T04:08:26.326324+0000","last_became_active":"2024-09-17T04:08:24.191318+0000","last_became_peered":"2024-09-17T04:08:24.191318+0000","last_unstale":"2024-09-17T04:08:26.326324+0000","last_undegraded":"2024-09-17T04:08:26.326324+0000","last_fullsized":"2024-09-17T04:08:26.326324+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:17.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188680+0000","last_change":"2024-09-17T04:08:24.188680+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188680+0000","last_undegraded":"2024-09-17T04:08:24.188680+0000","last_fullsized":"2024-09-17T04:08:24.188680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"271'3","reported_seq":18,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326468+0000","last_change":"2024-09-17T04:08:24.190381+0000","last_active":"2024-09-17T04:08:26.326468+0000","last_peered":"2024-09-17T04:08:26.326468+0000","last_clean":"2024-09-17T04:08:26.326468+0000","last_became_active":"2024-09-17T04:08:24.189870+0000","last_became_peered":"2024-09-17T04:08:24.189870+0000","last_unstale":"2024-09-17T04:08:26.326468+0000","last_undegraded":"2024-09-17T04:08:26.326468+0000","last_fullsized":"2024-09-17T04:08:26.326468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:11.493387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326092+0000","last_change":"2024-09-17T04:08:25.202764+0000","last_active":"2024-09-17T04:08:26.326092+0000","last_peered":"2024-09-17T04:08:26.326092+0000","last_clean":"2024-09-17T04:08:26.326092+0000","last_became_active":"2024-09-17T04:08:25.202410+0000","last_became_peered":"2024-09-17T04:08:25.202410+0000","last_unstale":"2024-09-17T04:08:26.326092+0000","last_undegraded":"2024-09-17T04:08:26.326092+0000","last_fullsized":"2024-09-17T04:08:26.326092+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:47.028739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"271'6","reported_seq":21,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325975+0000","last_change":"2024-09-17T04:08:24.200801+0000","last_active":"2024-09-17T04:08:26.325975+0000","last_peered":"2024-09-17T04:08:26.325975+0000","last_clean":"2024-09-17T04:08:26.325975+0000","last_became_active":"2024-09-17T04:08:24.199940+0000","last_became_peered":"2024-09-17T04:08:24.199940+0000","last_unstale":"2024-09-17T04:08:26.325975+0000","last_undegraded":"2024-09-17T04:08:26.325975+0000","last_fullsized":"2024-09-17T04:08:26.325975+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:53.586256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326081+0000","last_change":"2024-09-17T04:08:25.206679+0000","last_active":"2024-09-17T04:08:26.326081+0000","last_peered":"2024-09-17T04:08:26.326081+0000","last_clean":"2024-09-17T04:08:26.326081+0000","last_became_active":"2024-09-17T04:08:25.206365+0000","last_became_peered":"2024-09-17T04:08:25.206365+0000","last_unstale":"2024-09-17T04:08:26.326081+0000","last_undegraded":"2024-09-17T04:08:26.326081+0000","last_fullsized":"2024-09-17T04:08:26.326081+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:30.096668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195422+0000","last_change":"2024-09-17T04:08:24.193138+0000","last_active":"2024-09-17T04:08:24.195422+0000","last_peered":"2024-09-17T04:08:24.195422+0000","last_clean":"2024-09-17T04:08:24.195422+0000","last_became_active":"2024-09-17T04:08:24.192861+0000","last_became_peered":"2024-09-17T04:08:24.192861+0000","last_unstale":"2024-09-17T04:08:24.195422+0000","last_undegraded":"2024-09-17T04:08:24.195422+0000","last_fullsized":"2024-09-17T04:08:24.195422+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:21.860819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188799+0000","last_change":"2024-09-17T04:08:24.188799+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188799+0000","last_undegraded":"2024-09-17T04:08:24.188799+0000","last_fullsized":"2024-09-17T04:08:24.188799+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.198667+0000","last_change":"2024-09-17T04:08:24.197968+0000","last_active":"2024-09-17T04:08:24.198667+0000","last_peered":"2024-09-17T04:08:24.198667+0000","last_clean":"2024-09-17T04:08:24.198667+0000","last_became_active":"2024-09-17T04:08:24.197333+0000","last_became_peered":"2024-09-17T04:08:24.197333+0000","last_unstale":"2024-09-17T04:08:24.198667+0000","last_undegraded":"2024-09-17T04:08:24.198667+0000","last_fullsized":"2024-09-17T04:08:24.198667+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:16.246224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179664+0000","last_change":"2024-09-17T04:08:24.179664+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179664+0000","last_undegraded":"2024-09-17T04:08:24.179664+0000","last_fullsized":"2024-09-17T04:08:24.179664+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200585+0000","last_change":"2024-09-17T04:08:24.199871+0000","last_active":"2024-09-17T04:08:24.200585+0000","last_peered":"2024-09-17T04:08:24.200585+0000","last_clean":"2024-09-17T04:08:24.200585+0000","last_became_active":"2024-09-17T04:08:24.199516+0000","last_became_peered":"2024-09-17T04:08:24.199516+0000","last_unstale":"2024-09-17T04:08:24.200585+0000","last_undegraded":"2024-09-17T04:08:24.200585+0000","last_fullsized":"2024-09-17T04:08:24.200585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:08.418056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.204022+0000","last_change":"2024-09-17T04:08:24.202703+0000","last_active":"2024-09-17T04:08:24.204022+0000","last_peered":"2024-09-17T04:08:24.204022+0000","last_clean":"2024-09-17T04:08:24.204022+0000","last_became_active":"2024-09-17T04:08:24.201244+0000","last_became_peered":"2024-09-17T04:08:24.201244+0000","last_unstale":"2024-09-17T04:08:24.204022+0000","last_undegraded":"2024-09-17T04:08:24.204022+0000","last_fullsized":"2024-09-17T04:08:24.204022+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:37.878181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326899+0000","last_change":"2024-09-17T04:08:24.198617+0000","last_active":"2024-09-17T04:08:26.326899+0000","last_peered":"2024-09-17T04:08:26.326899+0000","last_clean":"2024-09-17T04:08:26.326899+0000","last_became_active":"2024-09-17T04:08:24.198153+0000","last_became_peered":"2024-09-17T04:08:24.198153+0000","last_unstale":"2024-09-17T04:08:26.326899+0000","last_undegraded":"2024-09-17T04:08:26.326899+0000","last_fullsized":"2024-09-17T04:08:26.326899+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:23.292656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185175+0000","last_change":"2024-09-17T04:08:24.185175+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185175+0000","last_undegraded":"2024-09-17T04:08:24.185175+0000","last_fullsized":"2024-09-17T04:08:24.185175+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199626+0000","last_change":"2024-09-17T04:08:24.195281+0000","last_active":"2024-09-17T04:08:24.199626+0000","last_peered":"2024-09-17T04:08:24.199626+0000","last_clean":"2024-09-17T04:08:24.199626+0000","last_became_active":"2024-09-17T04:08:24.194954+0000","last_became_peered":"2024-09-17T04:08:24.194954+0000","last_unstale":"2024-09-17T04:08:24.199626+0000","last_undegraded":"2024-09-17T04:08:24.199626+0000","last_fullsized":"2024-09-17T04:08:24.199626+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:56.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181698+0000","last_change":"2024-09-17T04:08:24.181698+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181698+0000","last_undegraded":"2024-09-17T04:08:24.181698+0000","last_fullsized":"2024-09-17T04:08:24.181698+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326585+0000","last_change":"2024-09-17T04:08:24.199722+0000","last_active":"2024-09-17T04:08:26.326585+0000","last_peered":"2024-09-17T04:08:26.326585+0000","last_clean":"2024-09-17T04:08:26.326585+0000","last_became_active":"2024-09-17T04:08:24.199405+0000","last_became_peered":"2024-09-17T04:08:24.199405+0000","last_unstale":"2024-09-17T04:08:26.326585+0000","last_undegraded":"2024-09-17T04:08:26.326585+0000","last_fullsized":"2024-09-17T04:08:26.326585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:34.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326388+0000","last_change":"2024-09-17T04:08:25.187530+0000","last_active":"2024-09-17T04:08:26.326388+0000","last_peered":"2024-09-17T04:08:26.326388+0000","last_clean":"2024-09-17T04:08:26.326388+0000","last_became_active":"2024-09-17T04:08:25.187224+0000","last_became_peered":"2024-09-17T04:08:25.187224+0000","last_unstale":"2024-09-17T04:08:26.326388+0000","last_undegraded":"2024-09-17T04:08:26.326388+0000","last_fullsized":"2024-09-17T04:08:26.326388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:35.040735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203439+0000","last_change":"2024-09-17T04:08:24.199980+0000","last_active":"2024-09-17T04:08:24.203439+0000","last_peered":"2024-09-17T04:08:24.203439+0000","last_clean":"2024-09-17T04:08:24.203439+0000","last_became_active":"2024-09-17T04:08:24.199415+0000","last_became_peered":"2024-09-17T04:08:24.199415+0000","last_unstale":"2024-09-17T04:08:24.203439+0000","last_undegraded":"2024-09-17T04:08:24.203439+0000","last_fullsized":"2024-09-17T04:08:24.203439+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:29.800724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179834+0000","last_change":"2024-09-17T04:08:24.179834+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179834+0000","last_undegraded":"2024-09-17T04:08:24.179834+0000","last_fullsized":"2024-09-17T04:08:24.179834+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200112+0000","last_change":"2024-09-17T04:08:24.198577+0000","last_active":"2024-09-17T04:08:24.200112+0000","last_peered":"2024-09-17T04:08:24.200112+0000","last_clean":"2024-09-17T04:08:24.200112+0000","last_became_active":"2024-09-17T04:08:24.198252+0000","last_became_peered":"2024-09-17T04:08:24.198252+0000","last_unstale":"2024-09-17T04:08:24.200112+0000","last_undegraded":"2024-09-17T04:08:24.200112+0000","last_fullsized":"2024-09-17T04:08:24.200112+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:41.168020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179965+0000","last_change":"2024-09-17T04:08:24.179965+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179965+0000","last_undegraded":"2024-09-17T04:08:24.179965+0000","last_fullsized":"2024-09-17T04:08:24.179965+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191487+0000","last_change":"2024-09-17T04:08:24.187161+0000","last_active":"2024-09-17T04:08:24.191487+0000","last_peered":"2024-09-17T04:08:24.191487+0000","last_clean":"2024-09-17T04:08:24.191487+0000","last_became_active":"2024-09-17T04:08:24.186572+0000","last_became_peered":"2024-09-17T04:08:24.186572+0000","last_unstale":"2024-09-17T04:08:24.191487+0000","last_undegraded":"2024-09-17T04:08:24.191487+0000","last_fullsized":"2024-09-17T04:08:24.191487+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:22:55.809038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.190666+0000","last_change":"2024-09-17T04:08:24.190666+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.190666+0000","last_undegraded":"2024-09-17T04:08:24.190666+0000","last_fullsized":"2024-09-17T04:08:24.190666+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200730+0000","last_change":"2024-09-17T04:08:24.196657+0000","last_active":"2024-09-17T04:08:24.200730+0000","last_peered":"2024-09-17T04:08:24.200730+0000","last_clean":"2024-09-17T04:08:24.200730+0000","last_became_active":"2024-09-17T04:08:24.195146+0000","last_became_peered":"2024-09-17T04:08:24.195146+0000","last_unstale":"2024-09-17T04:08:24.200730+0000","last_undegraded":"2024-09-17T04:08:24.200730+0000","last_fullsized":"2024-09-17T04:08:24.200730+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:21.531394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325728+0000","last_change":"2024-09-17T04:08:25.184757+0000","last_active":"2024-09-17T04:08:26.325728+0000","last_peered":"2024-09-17T04:08:26.325728+0000","last_clean":"2024-09-17T04:08:26.325728+0000","last_became_active":"2024-09-17T04:08:25.184177+0000","last_became_peered":"2024-09-17T04:08:25.184177+0000","last_unstale":"2024-09-17T04:08:26.325728+0000","last_undegraded":"2024-09-17T04:08:26.325728+0000","last_fullsized":"2024-09-17T04:08:26.325728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:35.563823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.184366+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.182740+0000","last_became_peered":"2024-09-17T04:08:24.182740+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:25.140619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182099+0000","last_change":"2024-09-17T04:08:24.182099+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182099+0000","last_undegraded":"2024-09-17T04:08:24.182099+0000","last_fullsized":"2024-09-17T04:08:24.182099+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.197037+0000","last_change":"2024-09-17T04:08:24.196057+0000","last_active":"2024-09-17T04:08:24.197037+0000","last_peered":"2024-09-17T04:08:24.197037+0000","last_clean":"2024-09-17T04:08:24.197037+0000","last_became_active":"2024-09-17T04:08:24.195822+0000","last_became_peered":"2024-09-17T04:08:24.195822+0000","last_unstale":"2024-09-17T04:08:24.197037+0000","last_undegraded":"2024-09-17T04:08:24.197037+0000","last_fullsized":"2024-09-17T04:08:24.197037+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:26.354317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326477+0000","last_change":"2024-09-17T04:08:25.207446+0000","last_active":"2024-09-17T04:08:26.326477+0000","last_peered":"2024-09-17T04:08:26.326477+0000","last_clean":"2024-09-17T04:08:26.326477+0000","last_became_active":"2024-09-17T04:08:25.207098+0000","last_became_peered":"2024-09-17T04:08:25.207098+0000","last_unstale":"2024-09-17T04:08:26.326477+0000","last_undegraded":"2024-09-17T04:08:26.326477+0000","last_fullsized":"2024-09-17T04:08:26.326477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:13.056929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203840+0000","last_change":"2024-09-17T04:08:24.202471+0000","last_active":"2024-09-17T04:08:24.203840+0000","last_peered":"2024-09-17T04:08:24.203840+0000","last_clean":"2024-09-17T04:08:24.203840+0000","last_became_active":"2024-09-17T04:08:24.202123+0000","last_became_peered":"2024-09-17T04:08:24.202123+0000","last_unstale":"2024-09-17T04:08:24.203840+0000","last_undegraded":"2024-09-17T04:08:24.203840+0000","last_fullsized":"2024-09-17T04:08:24.203840+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:48.755856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326446+0000","last_change":"2024-09-17T04:08:25.196030+0000","last_active":"2024-09-17T04:08:26.326446+0000","last_peered":"2024-09-17T04:08:26.326446+0000","last_clean":"2024-09-17T04:08:26.326446+0000","last_became_active":"2024-09-17T04:08:25.195608+0000","last_became_peered":"2024-09-17T04:08:25.195608+0000","last_unstale":"2024-09-17T04:08:26.326446+0000","last_undegraded":"2024-09-17T04:08:26.326446+0000","last_fullsized":"2024-09-17T04:08:26.326446+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:59.770015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326338+0000","last_change":"2024-09-17T04:08:24.199653+0000","last_active":"2024-09-17T04:08:26.326338+0000","last_peered":"2024-09-17T04:08:26.326338+0000","last_clean":"2024-09-17T04:08:26.326338+0000","last_became_active":"2024-09-17T04:08:24.197097+0000","last_became_peered":"2024-09-17T04:08:24.197097+0000","last_unstale":"2024-09-17T04:08:26.326338+0000","last_undegraded":"2024-09-17T04:08:26.326338+0000","last_fullsized":"2024-09-17T04:08:26.326338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:01.054583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186648+0000","last_change":"2024-09-17T04:08:24.186648+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186648+0000","last_undegraded":"2024-09-17T04:08:24.186648+0000","last_fullsized":"2024-09-17T04:08:24.186648+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326117+0000","last_change":"2024-09-17T04:08:24.200442+0000","last_active":"2024-09-17T04:08:26.326117+0000","last_peered":"2024-09-17T04:08:26.326117+0000","last_clean":"2024-09-17T04:08:26.326117+0000","last_became_active":"2024-09-17T04:08:24.199560+0000","last_became_peered":"2024-09-17T04:08:24.199560+0000","last_unstale":"2024-09-17T04:08:26.326117+0000","last_undegraded":"2024-09-17T04:08:26.326117+0000","last_fullsized":"2024-09-17T04:08:26.326117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:07.964242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183889+0000","last_change":"2024-09-17T04:08:24.183889+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183889+0000","last_undegraded":"2024-09-17T04:08:24.183889+0000","last_fullsized":"2024-09-17T04:08:24.183889+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325805+0000","last_change":"2024-09-17T04:08:24.187060+0000","last_active":"2024-09-17T04:08:26.325805+0000","last_peered":"2024-09-17T04:08:26.325805+0000","last_clean":"2024-09-17T04:08:26.325805+0000","last_became_active":"2024-09-17T04:08:24.186004+0000","last_became_peered":"2024-09-17T04:08:24.186004+0000","last_unstale":"2024-09-17T04:08:26.325805+0000","last_undegraded":"2024-09-17T04:08:26.325805+0000","last_fullsized":"2024-09-17T04:08:26.325805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:11.972381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179560+0000","last_change":"2024-09-17T04:08:24.179560+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179560+0000","last_undegraded":"2024-09-17T04:08:24.179560+0000","last_fullsized":"2024-09-17T04:08:24.179560+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191457+0000","last_change":"2024-09-17T04:08:24.186853+0000","last_active":"2024-09-17T04:08:24.191457+0000","last_peered":"2024-09-17T04:08:24.191457+0000","last_clean":"2024-09-17T04:08:24.191457+0000","last_became_active":"2024-09-17T04:08:24.186204+0000","last_became_peered":"2024-09-17T04:08:24.186204+0000","last_unstale":"2024-09-17T04:08:24.191457+0000","last_undegraded":"2024-09-17T04:08:24.191457+0000","last_fullsized":"2024-09-17T04:08:24.191457+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:43.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181036+0000","last_change":"2024-09-17T04:08:24.181036+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181036+0000","last_undegraded":"2024-09-17T04:08:24.181036+0000","last_fullsized":"2024-09-17T04:08:24.181036+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195797+0000","last_change":"2024-09-17T04:08:24.194575+0000","last_active":"2024-09-17T04:08:24.195797+0000","last_peered":"2024-09-17T04:08:24.195797+0000","last_clean":"2024-09-17T04:08:24.195797+0000","last_became_active":"2024-09-17T04:08:24.194250+0000","last_became_peered":"2024-09-17T04:08:24.194250+0000","last_unstale":"2024-09-17T04:08:24.195797+0000","last_undegraded":"2024-09-17T04:08:24.195797+0000","last_fullsized":"2024-09-17T04:08:24.195797+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:01.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179027+0000","last_change":"2024-09-17T04:08:24.179027+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179027+0000","last_undegraded":"2024-09-17T04:08:24.179027+0000","last_fullsized":"2024-09-17T04:08:24.179027+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.193620+0000","last_change":"2024-09-17T04:08:24.187831+0000","last_active":"2024-09-17T04:08:24.193620+0000","last_peered":"2024-09-17T04:08:24.193620+0000","last_clean":"2024-09-17T04:08:24.193620+0000","last_became_active":"2024-09-17T04:08:24.186781+0000","last_became_peered":"2024-09-17T04:08:24.186781+0000","last_unstale":"2024-09-17T04:08:24.193620+0000","last_undegraded":"2024-09-17T04:08:24.193620+0000","last_fullsized":"2024-09-17T04:08:24.193620+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:15.424529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.28","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326126+0000","last_change":"2024-09-17T04:08:25.204595+0000","last_active":"2024-09-17T04:08:26.326126+0000","last_peered":"2024-09-17T04:08:26.326126+0000","last_clean":"2024-09-17T04:08:26.326126+0000","last_became_active":"2024-09-17T04:08:25.204271+0000","last_became_peered":"2024-09-17T04:08:25.204271+0000","last_unstale":"2024-09-17T04:08:26.326126+0000","last_undegraded":"2024-09-17T04:08:26.326126+0000","last_fullsized":"2024-09-17T04:08:26.326126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:34.822087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":571,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.185728+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:08:24.185728+0000","last_peered":"2024-09-17T04:08:24.185728+0000","last_clean":"2024-09-17T04:08:24.185728+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:08:24.185728+0000","last_undegraded":"2024-09-17T04:08:24.185728+0000","last_fullsized":"2024-09-17T04:08:24.185728+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":41,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53152,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318688,"statfs":{"total":96636764160,"available":96582336512,"internally_reserved":0,"allocated":4005888,"data_stored":2104324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509673,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34520,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337320,"statfs":{"total":96636764160,"available":96601415680,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56736,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315104,"statfs":{"total":96636764160,"available":96578666496,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509672,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542375,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56788,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315052,"statfs":{"total":96636764160,"available":96578613248,"internally_reserved":0,"allocated":3469312,"data_stored":1535010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542376,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4067328,"data_stored":2126028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:08:29.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 -- 172.21.15.129:0/1146327582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c06f2e0 msgr2=0x7f383c0cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 --2- 172.21.15.129:0/1146327582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c06f2e0 0x7f383c0cd1c0 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f383c0ccc10 tx=0x7f383402aa30 comp rx=0 tx=0).stop 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 -- 172.21.15.129:0/1146327582 shutdown_connections 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 --2- 172.21.15.129:0/1146327582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c06f2e0 0x7f383c0cd1c0 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 --2- 172.21.15.129:0/1146327582 >> v2:172.21.15.181:3300/0 conn(0x7f383c06eed0 0x7f383c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 --2- 172.21.15.129:0/1146327582 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c0705a0 0x7f383c0b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 -- 172.21.15.129:0/1146327582 >> 172.21.15.129:0/1146327582 conn(0x7f383c06d410 msgr2=0x7f383c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:29.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 -- 172.21.15.129:0/1146327582 shutdown_connections 2024-09-17T04:08:29.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.865+0000 7f3844600640 1 -- 172.21.15.129:0/1146327582 wait complete. 2024-09-17T04:08:29.867 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.868+0000 7f3844600640 1 Processor -- start 2024-09-17T04:08:29.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.868+0000 7f3844600640 1 -- start start 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f383c06eed0 0x7f383c0c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c06f2e0 0x7f383c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f383c160b40 con 0x7f383c0705a0 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1609c0 con 0x7f383c06eed0 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3844600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f383c160cc0 con 0x7f383c06f2e0 2024-09-17T04:08:29.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.869+0000 7f3842600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57250/0 (socket says 172.21.15.129:57250) 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 -- 172.21.15.129:0/2374124340 learned_addr learned my addr 172.21.15.129:0/2374124340 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3841200640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c06f2e0 0x7f383c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3841c00640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f383c06eed0 0x7f383c0c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 -- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c06f2e0 msgr2=0x7f383c0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c06f2e0 0x7f383c0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 -- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f383c06eed0 msgr2=0x7f383c0c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f383c06eed0 0x7f383c0c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 -- 172.21.15.129:0/2374124340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c0c7740 con 0x7f383c0705a0 2024-09-17T04:08:29.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.870+0000 7f3842600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d60 tx=0x7f3834003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.871+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834033050 con 0x7f383c0705a0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.871+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3834004450 con 0x7f383c0705a0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.871+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834038450 con 0x7f383c0705a0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.871+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c0c7a20 con 0x7f383c0705a0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.871+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c0c98a0 con 0x7f383c0705a0 2024-09-17T04:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.872+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0c9b00 con 0x7f383c0705a0 2024-09-17T04:08:29.873 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.873+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3834036070 con 0x7f383c0705a0 2024-09-17T04:08:29.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.874+0000 7f3839600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 0x7f381c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:29.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.874+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f38340bbfd0 con 0x7f383c0705a0 2024-09-17T04:08:29.874 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.874+0000 7f3841c00640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 0x7f381c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:29.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.875+0000 7f3841c00640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 0x7f381c07b0d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f3830009b10 tx=0x7f3830005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:29.878 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:29.878+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3834083dc0 con 0x7f383c0705a0 2024-09-17T04:08:30.059 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.059+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f383c161220 con 0x7f381c078bc0 2024-09-17T04:08:30.063 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.063+0000 7f3839600640 1 -- 172.21.15.129:0/2374124340 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248654 (secure 0 0 0) 0x7f383c161220 con 0x7f381c078bc0 2024-09-17T04:08:30.066 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:30.067 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 msgr2=0x7f381c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 0x7f381c07b0d0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f3830009b10 tx=0x7f3830005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 msgr2=0x7f383c0b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f3834009d60 tx=0x7f3834003f10 comp rx=0 tx=0).stop 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 shutdown_connections 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f381c078bc0 0x7f381c07b0d0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f383c0705a0 0x7f383c0b1de0 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.069 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f383c06f2e0 0x7f383c0b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 --2- 172.21.15.129:0/2374124340 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f383c06eed0 0x7f383c0c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 >> 172.21.15.129:0/2374124340 conn(0x7f383c06d410 msgr2=0x7f383c074060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:30.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 shutdown_connections 2024-09-17T04:08:30.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.067+0000 7f3844600640 1 -- 172.21.15.129:0/2374124340 wait complete. 2024-09-17T04:08:30.092 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2024-09-17T04:08:28.620466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":117,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420456,"kb_used_data":27956,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754554264,"statfs":{"total":773094113280,"available":772663566336,"internally_reserved":0,"allocated":28626944,"data_stored":13347692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456722"},"pg_stats":[{"pgid":"41.3f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326478+0000","last_change":"2024-09-17T04:08:25.189216+0000","last_active":"2024-09-17T04:08:26.326478+0000","last_peered":"2024-09-17T04:08:26.326478+0000","last_clean":"2024-09-17T04:08:26.326478+0000","last_became_active":"2024-09-17T04:08:25.188921+0000","last_became_peered":"2024-09-17T04:08:25.188921+0000","last_unstale":"2024-09-17T04:08:26.326478+0000","last_undegraded":"2024-09-17T04:08:26.326478+0000","last_fullsized":"2024-09-17T04:08:26.326478+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:41.624638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189193+0000","last_change":"2024-09-17T04:08:24.189193+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189193+0000","last_undegraded":"2024-09-17T04:08:24.189193+0000","last_fullsized":"2024-09-17T04:08:24.189193+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181899+0000","last_change":"2024-09-17T04:08:24.181899+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181899+0000","last_undegraded":"2024-09-17T04:08:24.181899+0000","last_fullsized":"2024-09-17T04:08:24.181899+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178305+0000","last_change":"2024-09-17T04:08:24.178305+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178305+0000","last_undegraded":"2024-09-17T04:08:24.178305+0000","last_fullsized":"2024-09-17T04:08:24.178305+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.3b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182910+0000","last_change":"2024-09-17T04:08:24.182910+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182910+0000","last_undegraded":"2024-09-17T04:08:24.182910+0000","last_fullsized":"2024-09-17T04:08:24.182910+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178626+0000","last_change":"2024-09-17T04:08:24.178626+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178626+0000","last_undegraded":"2024-09-17T04:08:24.178626+0000","last_fullsized":"2024-09-17T04:08:24.178626+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.39","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183640+0000","last_change":"2024-09-17T04:08:24.183640+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183640+0000","last_undegraded":"2024-09-17T04:08:24.183640+0000","last_fullsized":"2024-09-17T04:08:24.183640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.38","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326289+0000","last_change":"2024-09-17T04:08:25.207974+0000","last_active":"2024-09-17T04:08:26.326289+0000","last_peered":"2024-09-17T04:08:26.326289+0000","last_clean":"2024-09-17T04:08:26.326289+0000","last_became_active":"2024-09-17T04:08:25.207454+0000","last_became_peered":"2024-09-17T04:08:25.207454+0000","last_unstale":"2024-09-17T04:08:26.326289+0000","last_undegraded":"2024-09-17T04:08:26.326289+0000","last_fullsized":"2024-09-17T04:08:26.326289+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:18.281022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.37","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182619+0000","last_change":"2024-09-17T04:08:24.182619+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182619+0000","last_undegraded":"2024-09-17T04:08:24.182619+0000","last_fullsized":"2024-09-17T04:08:24.182619+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.36","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185033+0000","last_change":"2024-09-17T04:08:24.185033+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185033+0000","last_undegraded":"2024-09-17T04:08:24.185033+0000","last_fullsized":"2024-09-17T04:08:24.185033+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.35","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.187179+0000","last_change":"2024-09-17T04:08:24.187179+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.187179+0000","last_undegraded":"2024-09-17T04:08:24.187179+0000","last_fullsized":"2024-09-17T04:08:24.187179+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.34","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326226+0000","last_change":"2024-09-17T04:08:25.196813+0000","last_active":"2024-09-17T04:08:26.326226+0000","last_peered":"2024-09-17T04:08:26.326226+0000","last_clean":"2024-09-17T04:08:26.326226+0000","last_became_active":"2024-09-17T04:08:25.196404+0000","last_became_peered":"2024-09-17T04:08:25.196404+0000","last_unstale":"2024-09-17T04:08:26.326226+0000","last_undegraded":"2024-09-17T04:08:26.326226+0000","last_fullsized":"2024-09-17T04:08:26.326226+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:24.380529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.33","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186085+0000","last_change":"2024-09-17T04:08:24.186085+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186085+0000","last_undegraded":"2024-09-17T04:08:24.186085+0000","last_fullsized":"2024-09-17T04:08:24.186085+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.32","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182701+0000","last_change":"2024-09-17T04:08:24.182701+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182701+0000","last_undegraded":"2024-09-17T04:08:24.182701+0000","last_fullsized":"2024-09-17T04:08:24.182701+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.31","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180999+0000","last_change":"2024-09-17T04:08:24.180999+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180999+0000","last_undegraded":"2024-09-17T04:08:24.180999+0000","last_fullsized":"2024-09-17T04:08:24.180999+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.30","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.192580+0000","last_change":"2024-09-17T04:08:24.192580+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.192580+0000","last_undegraded":"2024-09-17T04:08:24.192580+0000","last_fullsized":"2024-09-17T04:08:24.192580+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186791+0000","last_change":"2024-09-17T04:08:24.186791+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186791+0000","last_undegraded":"2024-09-17T04:08:24.186791+0000","last_fullsized":"2024-09-17T04:08:24.186791+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183594+0000","last_change":"2024-09-17T04:08:24.183594+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183594+0000","last_undegraded":"2024-09-17T04:08:24.183594+0000","last_fullsized":"2024-09-17T04:08:24.183594+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.2d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326090+0000","last_change":"2024-09-17T04:08:25.192767+0000","last_active":"2024-09-17T04:08:26.326090+0000","last_peered":"2024-09-17T04:08:26.326090+0000","last_clean":"2024-09-17T04:08:26.326090+0000","last_became_active":"2024-09-17T04:08:25.192484+0000","last_became_peered":"2024-09-17T04:08:25.192484+0000","last_unstale":"2024-09-17T04:08:26.326090+0000","last_undegraded":"2024-09-17T04:08:26.326090+0000","last_fullsized":"2024-09-17T04:08:26.326090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:08.026774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.2c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191776+0000","last_change":"2024-09-17T04:08:24.191776+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191776+0000","last_undegraded":"2024-09-17T04:08:24.191776+0000","last_fullsized":"2024-09-17T04:08:24.191776+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180410+0000","last_change":"2024-09-17T04:08:24.180410+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180410+0000","last_undegraded":"2024-09-17T04:08:24.180410+0000","last_fullsized":"2024-09-17T04:08:24.180410+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.184503+0000","last_change":"2024-09-17T04:08:24.184503+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.184503+0000","last_undegraded":"2024-09-17T04:08:24.184503+0000","last_fullsized":"2024-09-17T04:08:24.184503+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.29","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179821+0000","last_change":"2024-09-17T04:08:24.179821+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179821+0000","last_undegraded":"2024-09-17T04:08:24.179821+0000","last_fullsized":"2024-09-17T04:08:24.179821+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.27","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185990+0000","last_change":"2024-09-17T04:08:24.185990+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185990+0000","last_undegraded":"2024-09-17T04:08:24.185990+0000","last_fullsized":"2024-09-17T04:08:24.185990+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.26","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326742+0000","last_change":"2024-09-17T04:08:25.205002+0000","last_active":"2024-09-17T04:08:26.326742+0000","last_peered":"2024-09-17T04:08:26.326742+0000","last_clean":"2024-09-17T04:08:26.326742+0000","last_became_active":"2024-09-17T04:08:25.204427+0000","last_became_peered":"2024-09-17T04:08:25.204427+0000","last_unstale":"2024-09-17T04:08:26.326742+0000","last_undegraded":"2024-09-17T04:08:26.326742+0000","last_fullsized":"2024-09-17T04:08:26.326742+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:37.951456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.25","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180951+0000","last_change":"2024-09-17T04:08:24.180951+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180951+0000","last_undegraded":"2024-09-17T04:08:24.180951+0000","last_fullsized":"2024-09-17T04:08:24.180951+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.24","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326183+0000","last_change":"2024-09-17T04:08:25.188324+0000","last_active":"2024-09-17T04:08:26.326183+0000","last_peered":"2024-09-17T04:08:26.326183+0000","last_clean":"2024-09-17T04:08:26.326183+0000","last_became_active":"2024-09-17T04:08:25.188025+0000","last_became_peered":"2024-09-17T04:08:25.188025+0000","last_unstale":"2024-09-17T04:08:26.326183+0000","last_undegraded":"2024-09-17T04:08:26.326183+0000","last_fullsized":"2024-09-17T04:08:26.326183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:41.304344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.23","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189672+0000","last_change":"2024-09-17T04:08:24.189672+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189672+0000","last_undegraded":"2024-09-17T04:08:24.189672+0000","last_fullsized":"2024-09-17T04:08:24.189672+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.22","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326936+0000","last_change":"2024-09-17T04:08:25.194536+0000","last_active":"2024-09-17T04:08:26.326936+0000","last_peered":"2024-09-17T04:08:26.326936+0000","last_clean":"2024-09-17T04:08:26.326936+0000","last_became_active":"2024-09-17T04:08:25.194201+0000","last_became_peered":"2024-09-17T04:08:25.194201+0000","last_unstale":"2024-09-17T04:08:26.326936+0000","last_undegraded":"2024-09-17T04:08:26.326936+0000","last_fullsized":"2024-09-17T04:08:26.326936+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:14.659279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.21","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326187+0000","last_change":"2024-09-17T04:08:25.198539+0000","last_active":"2024-09-17T04:08:26.326187+0000","last_peered":"2024-09-17T04:08:26.326187+0000","last_clean":"2024-09-17T04:08:26.326187+0000","last_became_active":"2024-09-17T04:08:25.198178+0000","last_became_peered":"2024-09-17T04:08:25.198178+0000","last_unstale":"2024-09-17T04:08:26.326187+0000","last_undegraded":"2024-09-17T04:08:26.326187+0000","last_fullsized":"2024-09-17T04:08:26.326187+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:18.437486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.20","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182935+0000","last_change":"2024-09-17T04:08:24.182935+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182935+0000","last_undegraded":"2024-09-17T04:08:24.182935+0000","last_fullsized":"2024-09-17T04:08:24.182935+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326610+0000","last_change":"2024-09-17T04:08:25.199592+0000","last_active":"2024-09-17T04:08:26.326610+0000","last_peered":"2024-09-17T04:08:26.326610+0000","last_clean":"2024-09-17T04:08:26.326610+0000","last_became_active":"2024-09-17T04:08:25.199254+0000","last_became_peered":"2024-09-17T04:08:25.199254+0000","last_unstale":"2024-09-17T04:08:26.326610+0000","last_undegraded":"2024-09-17T04:08:26.326610+0000","last_fullsized":"2024-09-17T04:08:26.326610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:22.464215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"271'4","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326348+0000","last_change":"2024-09-17T04:08:24.190135+0000","last_active":"2024-09-17T04:08:26.326348+0000","last_peered":"2024-09-17T04:08:26.326348+0000","last_clean":"2024-09-17T04:08:26.326348+0000","last_became_active":"2024-09-17T04:08:24.189843+0000","last_became_peered":"2024-09-17T04:08:24.189843+0000","last_unstale":"2024-09-17T04:08:26.326348+0000","last_undegraded":"2024-09-17T04:08:26.326348+0000","last_fullsized":"2024-09-17T04:08:26.326348+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:29.916388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181601+0000","last_change":"2024-09-17T04:08:24.181601+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181601+0000","last_undegraded":"2024-09-17T04:08:24.181601+0000","last_fullsized":"2024-09-17T04:08:24.181601+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326691+0000","last_change":"2024-09-17T04:08:24.196111+0000","last_active":"2024-09-17T04:08:26.326691+0000","last_peered":"2024-09-17T04:08:26.326691+0000","last_clean":"2024-09-17T04:08:26.326691+0000","last_became_active":"2024-09-17T04:08:24.195595+0000","last_became_peered":"2024-09-17T04:08:24.195595+0000","last_unstale":"2024-09-17T04:08:26.326691+0000","last_undegraded":"2024-09-17T04:08:26.326691+0000","last_fullsized":"2024-09-17T04:08:26.326691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:44.710662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180884+0000","last_change":"2024-09-17T04:08:24.180884+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180884+0000","last_undegraded":"2024-09-17T04:08:24.180884+0000","last_fullsized":"2024-09-17T04:08:24.180884+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.196010+0000","last_change":"2024-09-17T04:08:24.194368+0000","last_active":"2024-09-17T04:08:24.196010+0000","last_peered":"2024-09-17T04:08:24.196010+0000","last_clean":"2024-09-17T04:08:24.196010+0000","last_became_active":"2024-09-17T04:08:24.193648+0000","last_became_peered":"2024-09-17T04:08:24.193648+0000","last_unstale":"2024-09-17T04:08:24.196010+0000","last_undegraded":"2024-09-17T04:08:24.196010+0000","last_fullsized":"2024-09-17T04:08:24.196010+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:40.412794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186629+0000","last_change":"2024-09-17T04:08:24.186629+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186629+0000","last_undegraded":"2024-09-17T04:08:24.186629+0000","last_fullsized":"2024-09-17T04:08:24.186629+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203824+0000","last_change":"2024-09-17T04:08:24.202248+0000","last_active":"2024-09-17T04:08:24.203824+0000","last_peered":"2024-09-17T04:08:24.203824+0000","last_clean":"2024-09-17T04:08:24.203824+0000","last_became_active":"2024-09-17T04:08:24.201709+0000","last_became_peered":"2024-09-17T04:08:24.201709+0000","last_unstale":"2024-09-17T04:08:24.203824+0000","last_undegraded":"2024-09-17T04:08:24.203824+0000","last_fullsized":"2024-09-17T04:08:24.203824+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:16.873589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180525+0000","last_change":"2024-09-17T04:08:24.180525+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180525+0000","last_undegraded":"2024-09-17T04:08:24.180525+0000","last_fullsized":"2024-09-17T04:08:24.180525+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.201402+0000","last_change":"2024-09-17T04:08:24.200461+0000","last_active":"2024-09-17T04:08:24.201402+0000","last_peered":"2024-09-17T04:08:24.201402+0000","last_clean":"2024-09-17T04:08:24.201402+0000","last_became_active":"2024-09-17T04:08:24.199506+0000","last_became_peered":"2024-09-17T04:08:24.199506+0000","last_unstale":"2024-09-17T04:08:24.201402+0000","last_undegraded":"2024-09-17T04:08:24.201402+0000","last_fullsized":"2024-09-17T04:08:24.201402+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:31.705720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325694+0000","last_change":"2024-09-17T04:08:25.185548+0000","last_active":"2024-09-17T04:08:26.325694+0000","last_peered":"2024-09-17T04:08:26.325694+0000","last_clean":"2024-09-17T04:08:26.325694+0000","last_became_active":"2024-09-17T04:08:25.184461+0000","last_became_peered":"2024-09-17T04:08:25.184461+0000","last_unstale":"2024-09-17T04:08:26.325694+0000","last_undegraded":"2024-09-17T04:08:26.325694+0000","last_fullsized":"2024-09-17T04:08:26.325694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:46.956363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325684+0000","last_change":"2024-09-17T04:08:24.187071+0000","last_active":"2024-09-17T04:08:26.325684+0000","last_peered":"2024-09-17T04:08:26.325684+0000","last_clean":"2024-09-17T04:08:26.325684+0000","last_became_active":"2024-09-17T04:08:24.186743+0000","last_became_peered":"2024-09-17T04:08:24.186743+0000","last_unstale":"2024-09-17T04:08:26.325684+0000","last_undegraded":"2024-09-17T04:08:26.325684+0000","last_fullsized":"2024-09-17T04:08:26.325684+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:15.930301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191846+0000","last_change":"2024-09-17T04:08:24.191846+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191846+0000","last_undegraded":"2024-09-17T04:08:24.191846+0000","last_fullsized":"2024-09-17T04:08:24.191846+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199751+0000","last_change":"2024-09-17T04:08:24.192860+0000","last_active":"2024-09-17T04:08:24.199751+0000","last_peered":"2024-09-17T04:08:24.199751+0000","last_clean":"2024-09-17T04:08:24.199751+0000","last_became_active":"2024-09-17T04:08:24.192505+0000","last_became_peered":"2024-09-17T04:08:24.192505+0000","last_unstale":"2024-09-17T04:08:24.199751+0000","last_undegraded":"2024-09-17T04:08:24.199751+0000","last_fullsized":"2024-09-17T04:08:24.199751+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:33.412525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325700+0000","last_change":"2024-09-17T04:08:25.195702+0000","last_active":"2024-09-17T04:08:26.325700+0000","last_peered":"2024-09-17T04:08:26.325700+0000","last_clean":"2024-09-17T04:08:26.325700+0000","last_became_active":"2024-09-17T04:08:25.195370+0000","last_became_peered":"2024-09-17T04:08:25.195370+0000","last_unstale":"2024-09-17T04:08:26.325700+0000","last_undegraded":"2024-09-17T04:08:26.325700+0000","last_fullsized":"2024-09-17T04:08:26.325700+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:35.074389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.198388+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.197800+0000","last_became_peered":"2024-09-17T04:08:24.197800+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:50.784271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185710+0000","last_change":"2024-09-17T04:08:24.185710+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185710+0000","last_undegraded":"2024-09-17T04:08:24.185710+0000","last_fullsized":"2024-09-17T04:08:24.185710+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200361+0000","last_change":"2024-09-17T04:08:24.197553+0000","last_active":"2024-09-17T04:08:24.200361+0000","last_peered":"2024-09-17T04:08:24.200361+0000","last_clean":"2024-09-17T04:08:24.200361+0000","last_became_active":"2024-09-17T04:08:24.197233+0000","last_became_peered":"2024-09-17T04:08:24.197233+0000","last_unstale":"2024-09-17T04:08:24.200361+0000","last_undegraded":"2024-09-17T04:08:24.200361+0000","last_fullsized":"2024-09-17T04:08:24.200361+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:08.411479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181356+0000","last_change":"2024-09-17T04:08:24.181356+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181356+0000","last_undegraded":"2024-09-17T04:08:24.181356+0000","last_fullsized":"2024-09-17T04:08:24.181356+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326324+0000","last_change":"2024-09-17T04:08:24.191873+0000","last_active":"2024-09-17T04:08:26.326324+0000","last_peered":"2024-09-17T04:08:26.326324+0000","last_clean":"2024-09-17T04:08:26.326324+0000","last_became_active":"2024-09-17T04:08:24.191318+0000","last_became_peered":"2024-09-17T04:08:24.191318+0000","last_unstale":"2024-09-17T04:08:26.326324+0000","last_undegraded":"2024-09-17T04:08:26.326324+0000","last_fullsized":"2024-09-17T04:08:26.326324+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:17.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188680+0000","last_change":"2024-09-17T04:08:24.188680+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188680+0000","last_undegraded":"2024-09-17T04:08:24.188680+0000","last_fullsized":"2024-09-17T04:08:24.188680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"271'3","reported_seq":18,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326468+0000","last_change":"2024-09-17T04:08:24.190381+0000","last_active":"2024-09-17T04:08:26.326468+0000","last_peered":"2024-09-17T04:08:26.326468+0000","last_clean":"2024-09-17T04:08:26.326468+0000","last_became_active":"2024-09-17T04:08:24.189870+0000","last_became_peered":"2024-09-17T04:08:24.189870+0000","last_unstale":"2024-09-17T04:08:26.326468+0000","last_undegraded":"2024-09-17T04:08:26.326468+0000","last_fullsized":"2024-09-17T04:08:26.326468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:11.493387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326092+0000","last_change":"2024-09-17T04:08:25.202764+0000","last_active":"2024-09-17T04:08:26.326092+0000","last_peered":"2024-09-17T04:08:26.326092+0000","last_clean":"2024-09-17T04:08:26.326092+0000","last_became_active":"2024-09-17T04:08:25.202410+0000","last_became_peered":"2024-09-17T04:08:25.202410+0000","last_unstale":"2024-09-17T04:08:26.326092+0000","last_undegraded":"2024-09-17T04:08:26.326092+0000","last_fullsized":"2024-09-17T04:08:26.326092+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:47.028739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"271'6","reported_seq":21,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325975+0000","last_change":"2024-09-17T04:08:24.200801+0000","last_active":"2024-09-17T04:08:26.325975+0000","last_peered":"2024-09-17T04:08:26.325975+0000","last_clean":"2024-09-17T04:08:26.325975+0000","last_became_active":"2024-09-17T04:08:24.199940+0000","last_became_peered":"2024-09-17T04:08:24.199940+0000","last_unstale":"2024-09-17T04:08:26.325975+0000","last_undegraded":"2024-09-17T04:08:26.325975+0000","last_fullsized":"2024-09-17T04:08:26.325975+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:53.586256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326081+0000","last_change":"2024-09-17T04:08:25.206679+0000","last_active":"2024-09-17T04:08:26.326081+0000","last_peered":"2024-09-17T04:08:26.326081+0000","last_clean":"2024-09-17T04:08:26.326081+0000","last_became_active":"2024-09-17T04:08:25.206365+0000","last_became_peered":"2024-09-17T04:08:25.206365+0000","last_unstale":"2024-09-17T04:08:26.326081+0000","last_undegraded":"2024-09-17T04:08:26.326081+0000","last_fullsized":"2024-09-17T04:08:26.326081+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:30.096668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195422+0000","last_change":"2024-09-17T04:08:24.193138+0000","last_active":"2024-09-17T04:08:24.195422+0000","last_peered":"2024-09-17T04:08:24.195422+0000","last_clean":"2024-09-17T04:08:24.195422+0000","last_became_active":"2024-09-17T04:08:24.192861+0000","last_became_peered":"2024-09-17T04:08:24.192861+0000","last_unstale":"2024-09-17T04:08:24.195422+0000","last_undegraded":"2024-09-17T04:08:24.195422+0000","last_fullsized":"2024-09-17T04:08:24.195422+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:21.860819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188799+0000","last_change":"2024-09-17T04:08:24.188799+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188799+0000","last_undegraded":"2024-09-17T04:08:24.188799+0000","last_fullsized":"2024-09-17T04:08:24.188799+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.198667+0000","last_change":"2024-09-17T04:08:24.197968+0000","last_active":"2024-09-17T04:08:24.198667+0000","last_peered":"2024-09-17T04:08:24.198667+0000","last_clean":"2024-09-17T04:08:24.198667+0000","last_became_active":"2024-09-17T04:08:24.197333+0000","last_became_peered":"2024-09-17T04:08:24.197333+0000","last_unstale":"2024-09-17T04:08:24.198667+0000","last_undegraded":"2024-09-17T04:08:24.198667+0000","last_fullsized":"2024-09-17T04:08:24.198667+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:16.246224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179664+0000","last_change":"2024-09-17T04:08:24.179664+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179664+0000","last_undegraded":"2024-09-17T04:08:24.179664+0000","last_fullsized":"2024-09-17T04:08:24.179664+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200585+0000","last_change":"2024-09-17T04:08:24.199871+0000","last_active":"2024-09-17T04:08:24.200585+0000","last_peered":"2024-09-17T04:08:24.200585+0000","last_clean":"2024-09-17T04:08:24.200585+0000","last_became_active":"2024-09-17T04:08:24.199516+0000","last_became_peered":"2024-09-17T04:08:24.199516+0000","last_unstale":"2024-09-17T04:08:24.200585+0000","last_undegraded":"2024-09-17T04:08:24.200585+0000","last_fullsized":"2024-09-17T04:08:24.200585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:08.418056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.204022+0000","last_change":"2024-09-17T04:08:24.202703+0000","last_active":"2024-09-17T04:08:24.204022+0000","last_peered":"2024-09-17T04:08:24.204022+0000","last_clean":"2024-09-17T04:08:24.204022+0000","last_became_active":"2024-09-17T04:08:24.201244+0000","last_became_peered":"2024-09-17T04:08:24.201244+0000","last_unstale":"2024-09-17T04:08:24.204022+0000","last_undegraded":"2024-09-17T04:08:24.204022+0000","last_fullsized":"2024-09-17T04:08:24.204022+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:37.878181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326899+0000","last_change":"2024-09-17T04:08:24.198617+0000","last_active":"2024-09-17T04:08:26.326899+0000","last_peered":"2024-09-17T04:08:26.326899+0000","last_clean":"2024-09-17T04:08:26.326899+0000","last_became_active":"2024-09-17T04:08:24.198153+0000","last_became_peered":"2024-09-17T04:08:24.198153+0000","last_unstale":"2024-09-17T04:08:26.326899+0000","last_undegraded":"2024-09-17T04:08:26.326899+0000","last_fullsized":"2024-09-17T04:08:26.326899+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:23.292656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185175+0000","last_change":"2024-09-17T04:08:24.185175+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185175+0000","last_undegraded":"2024-09-17T04:08:24.185175+0000","last_fullsized":"2024-09-17T04:08:24.185175+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199626+0000","last_change":"2024-09-17T04:08:24.195281+0000","last_active":"2024-09-17T04:08:24.199626+0000","last_peered":"2024-09-17T04:08:24.199626+0000","last_clean":"2024-09-17T04:08:24.199626+0000","last_became_active":"2024-09-17T04:08:24.194954+0000","last_became_peered":"2024-09-17T04:08:24.194954+0000","last_unstale":"2024-09-17T04:08:24.199626+0000","last_undegraded":"2024-09-17T04:08:24.199626+0000","last_fullsized":"2024-09-17T04:08:24.199626+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:56.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181698+0000","last_change":"2024-09-17T04:08:24.181698+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181698+0000","last_undegraded":"2024-09-17T04:08:24.181698+0000","last_fullsized":"2024-09-17T04:08:24.181698+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326585+0000","last_change":"2024-09-17T04:08:24.199722+0000","last_active":"2024-09-17T04:08:26.326585+0000","last_peered":"2024-09-17T04:08:26.326585+0000","last_clean":"2024-09-17T04:08:26.326585+0000","last_became_active":"2024-09-17T04:08:24.199405+0000","last_became_peered":"2024-09-17T04:08:24.199405+0000","last_unstale":"2024-09-17T04:08:26.326585+0000","last_undegraded":"2024-09-17T04:08:26.326585+0000","last_fullsized":"2024-09-17T04:08:26.326585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:34.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326388+0000","last_change":"2024-09-17T04:08:25.187530+0000","last_active":"2024-09-17T04:08:26.326388+0000","last_peered":"2024-09-17T04:08:26.326388+0000","last_clean":"2024-09-17T04:08:26.326388+0000","last_became_active":"2024-09-17T04:08:25.187224+0000","last_became_peered":"2024-09-17T04:08:25.187224+0000","last_unstale":"2024-09-17T04:08:26.326388+0000","last_undegraded":"2024-09-17T04:08:26.326388+0000","last_fullsized":"2024-09-17T04:08:26.326388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:35.040735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203439+0000","last_change":"2024-09-17T04:08:24.199980+0000","last_active":"2024-09-17T04:08:24.203439+0000","last_peered":"2024-09-17T04:08:24.203439+0000","last_clean":"2024-09-17T04:08:24.203439+0000","last_became_active":"2024-09-17T04:08:24.199415+0000","last_became_peered":"2024-09-17T04:08:24.199415+0000","last_unstale":"2024-09-17T04:08:24.203439+0000","last_undegraded":"2024-09-17T04:08:24.203439+0000","last_fullsized":"2024-09-17T04:08:24.203439+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:29.800724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179834+0000","last_change":"2024-09-17T04:08:24.179834+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179834+0000","last_undegraded":"2024-09-17T04:08:24.179834+0000","last_fullsized":"2024-09-17T04:08:24.179834+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200112+0000","last_change":"2024-09-17T04:08:24.198577+0000","last_active":"2024-09-17T04:08:24.200112+0000","last_peered":"2024-09-17T04:08:24.200112+0000","last_clean":"2024-09-17T04:08:24.200112+0000","last_became_active":"2024-09-17T04:08:24.198252+0000","last_became_peered":"2024-09-17T04:08:24.198252+0000","last_unstale":"2024-09-17T04:08:24.200112+0000","last_undegraded":"2024-09-17T04:08:24.200112+0000","last_fullsized":"2024-09-17T04:08:24.200112+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:41.168020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179965+0000","last_change":"2024-09-17T04:08:24.179965+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179965+0000","last_undegraded":"2024-09-17T04:08:24.179965+0000","last_fullsized":"2024-09-17T04:08:24.179965+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191487+0000","last_change":"2024-09-17T04:08:24.187161+0000","last_active":"2024-09-17T04:08:24.191487+0000","last_peered":"2024-09-17T04:08:24.191487+0000","last_clean":"2024-09-17T04:08:24.191487+0000","last_became_active":"2024-09-17T04:08:24.186572+0000","last_became_peered":"2024-09-17T04:08:24.186572+0000","last_unstale":"2024-09-17T04:08:24.191487+0000","last_undegraded":"2024-09-17T04:08:24.191487+0000","last_fullsized":"2024-09-17T04:08:24.191487+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:22:55.809038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.190666+0000","last_change":"2024-09-17T04:08:24.190666+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.190666+0000","last_undegraded":"2024-09-17T04:08:24.190666+0000","last_fullsized":"2024-09-17T04:08:24.190666+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200730+0000","last_change":"2024-09-17T04:08:24.196657+0000","last_active":"2024-09-17T04:08:24.200730+0000","last_peered":"2024-09-17T04:08:24.200730+0000","last_clean":"2024-09-17T04:08:24.200730+0000","last_became_active":"2024-09-17T04:08:24.195146+0000","last_became_peered":"2024-09-17T04:08:24.195146+0000","last_unstale":"2024-09-17T04:08:24.200730+0000","last_undegraded":"2024-09-17T04:08:24.200730+0000","last_fullsized":"2024-09-17T04:08:24.200730+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:21.531394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325728+0000","last_change":"2024-09-17T04:08:25.184757+0000","last_active":"2024-09-17T04:08:26.325728+0000","last_peered":"2024-09-17T04:08:26.325728+0000","last_clean":"2024-09-17T04:08:26.325728+0000","last_became_active":"2024-09-17T04:08:25.184177+0000","last_became_peered":"2024-09-17T04:08:25.184177+0000","last_unstale":"2024-09-17T04:08:26.325728+0000","last_undegraded":"2024-09-17T04:08:26.325728+0000","last_fullsized":"2024-09-17T04:08:26.325728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:35.563823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.184366+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.182740+0000","last_became_peered":"2024-09-17T04:08:24.182740+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:25.140619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182099+0000","last_change":"2024-09-17T04:08:24.182099+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182099+0000","last_undegraded":"2024-09-17T04:08:24.182099+0000","last_fullsized":"2024-09-17T04:08:24.182099+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.197037+0000","last_change":"2024-09-17T04:08:24.196057+0000","last_active":"2024-09-17T04:08:24.197037+0000","last_peered":"2024-09-17T04:08:24.197037+0000","last_clean":"2024-09-17T04:08:24.197037+0000","last_became_active":"2024-09-17T04:08:24.195822+0000","last_became_peered":"2024-09-17T04:08:24.195822+0000","last_unstale":"2024-09-17T04:08:24.197037+0000","last_undegraded":"2024-09-17T04:08:24.197037+0000","last_fullsized":"2024-09-17T04:08:24.197037+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:26.354317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326477+0000","last_change":"2024-09-17T04:08:25.207446+0000","last_active":"2024-09-17T04:08:26.326477+0000","last_peered":"2024-09-17T04:08:26.326477+0000","last_clean":"2024-09-17T04:08:26.326477+0000","last_became_active":"2024-09-17T04:08:25.207098+0000","last_became_peered":"2024-09-17T04:08:25.207098+0000","last_unstale":"2024-09-17T04:08:26.326477+0000","last_undegraded":"2024-09-17T04:08:26.326477+0000","last_fullsized":"2024-09-17T04:08:26.326477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:13.056929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203840+0000","last_change":"2024-09-17T04:08:24.202471+0000","last_active":"2024-09-17T04:08:24.203840+0000","last_peered":"2024-09-17T04:08:24.203840+0000","last_clean":"2024-09-17T04:08:24.203840+0000","last_became_active":"2024-09-17T04:08:24.202123+0000","last_became_peered":"2024-09-17T04:08:24.202123+0000","last_unstale":"2024-09-17T04:08:24.203840+0000","last_undegraded":"2024-09-17T04:08:24.203840+0000","last_fullsized":"2024-09-17T04:08:24.203840+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:48.755856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326446+0000","last_change":"2024-09-17T04:08:25.196030+0000","last_active":"2024-09-17T04:08:26.326446+0000","last_peered":"2024-09-17T04:08:26.326446+0000","last_clean":"2024-09-17T04:08:26.326446+0000","last_became_active":"2024-09-17T04:08:25.195608+0000","last_became_peered":"2024-09-17T04:08:25.195608+0000","last_unstale":"2024-09-17T04:08:26.326446+0000","last_undegraded":"2024-09-17T04:08:26.326446+0000","last_fullsized":"2024-09-17T04:08:26.326446+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:59.770015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326338+0000","last_change":"2024-09-17T04:08:24.199653+0000","last_active":"2024-09-17T04:08:26.326338+0000","last_peered":"2024-09-17T04:08:26.326338+0000","last_clean":"2024-09-17T04:08:26.326338+0000","last_became_active":"2024-09-17T04:08:24.197097+0000","last_became_peered":"2024-09-17T04:08:24.197097+0000","last_unstale":"2024-09-17T04:08:26.326338+0000","last_undegraded":"2024-09-17T04:08:26.326338+0000","last_fullsized":"2024-09-17T04:08:26.326338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:01.054583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186648+0000","last_change":"2024-09-17T04:08:24.186648+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186648+0000","last_undegraded":"2024-09-17T04:08:24.186648+0000","last_fullsized":"2024-09-17T04:08:24.186648+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326117+0000","last_change":"2024-09-17T04:08:24.200442+0000","last_active":"2024-09-17T04:08:26.326117+0000","last_peered":"2024-09-17T04:08:26.326117+0000","last_clean":"2024-09-17T04:08:26.326117+0000","last_became_active":"2024-09-17T04:08:24.199560+0000","last_became_peered":"2024-09-17T04:08:24.199560+0000","last_unstale":"2024-09-17T04:08:26.326117+0000","last_undegraded":"2024-09-17T04:08:26.326117+0000","last_fullsized":"2024-09-17T04:08:26.326117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:07.964242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183889+0000","last_change":"2024-09-17T04:08:24.183889+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183889+0000","last_undegraded":"2024-09-17T04:08:24.183889+0000","last_fullsized":"2024-09-17T04:08:24.183889+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325805+0000","last_change":"2024-09-17T04:08:24.187060+0000","last_active":"2024-09-17T04:08:26.325805+0000","last_peered":"2024-09-17T04:08:26.325805+0000","last_clean":"2024-09-17T04:08:26.325805+0000","last_became_active":"2024-09-17T04:08:24.186004+0000","last_became_peered":"2024-09-17T04:08:24.186004+0000","last_unstale":"2024-09-17T04:08:26.325805+0000","last_undegraded":"2024-09-17T04:08:26.325805+0000","last_fullsized":"2024-09-17T04:08:26.325805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:11.972381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179560+0000","last_change":"2024-09-17T04:08:24.179560+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179560+0000","last_undegraded":"2024-09-17T04:08:24.179560+0000","last_fullsized":"2024-09-17T04:08:24.179560+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191457+0000","last_change":"2024-09-17T04:08:24.186853+0000","last_active":"2024-09-17T04:08:24.191457+0000","last_peered":"2024-09-17T04:08:24.191457+0000","last_clean":"2024-09-17T04:08:24.191457+0000","last_became_active":"2024-09-17T04:08:24.186204+0000","last_became_peered":"2024-09-17T04:08:24.186204+0000","last_unstale":"2024-09-17T04:08:24.191457+0000","last_undegraded":"2024-09-17T04:08:24.191457+0000","last_fullsized":"2024-09-17T04:08:24.191457+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:43.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181036+0000","last_change":"2024-09-17T04:08:24.181036+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181036+0000","last_undegraded":"2024-09-17T04:08:24.181036+0000","last_fullsized":"2024-09-17T04:08:24.181036+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195797+0000","last_change":"2024-09-17T04:08:24.194575+0000","last_active":"2024-09-17T04:08:24.195797+0000","last_peered":"2024-09-17T04:08:24.195797+0000","last_clean":"2024-09-17T04:08:24.195797+0000","last_became_active":"2024-09-17T04:08:24.194250+0000","last_became_peered":"2024-09-17T04:08:24.194250+0000","last_unstale":"2024-09-17T04:08:24.195797+0000","last_undegraded":"2024-09-17T04:08:24.195797+0000","last_fullsized":"2024-09-17T04:08:24.195797+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:01.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179027+0000","last_change":"2024-09-17T04:08:24.179027+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179027+0000","last_undegraded":"2024-09-17T04:08:24.179027+0000","last_fullsized":"2024-09-17T04:08:24.179027+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.193620+0000","last_change":"2024-09-17T04:08:24.187831+0000","last_active":"2024-09-17T04:08:24.193620+0000","last_peered":"2024-09-17T04:08:24.193620+0000","last_clean":"2024-09-17T04:08:24.193620+0000","last_became_active":"2024-09-17T04:08:24.186781+0000","last_became_peered":"2024-09-17T04:08:24.186781+0000","last_unstale":"2024-09-17T04:08:24.193620+0000","last_undegraded":"2024-09-17T04:08:24.193620+0000","last_fullsized":"2024-09-17T04:08:24.193620+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:15.424529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.28","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326126+0000","last_change":"2024-09-17T04:08:25.204595+0000","last_active":"2024-09-17T04:08:26.326126+0000","last_peered":"2024-09-17T04:08:26.326126+0000","last_clean":"2024-09-17T04:08:26.326126+0000","last_became_active":"2024-09-17T04:08:25.204271+0000","last_became_peered":"2024-09-17T04:08:25.204271+0000","last_unstale":"2024-09-17T04:08:26.326126+0000","last_undegraded":"2024-09-17T04:08:26.326126+0000","last_fullsized":"2024-09-17T04:08:26.326126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:34.822087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":571,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.185728+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:08:24.185728+0000","last_peered":"2024-09-17T04:08:24.185728+0000","last_clean":"2024-09-17T04:08:24.185728+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:08:24.185728+0000","last_undegraded":"2024-09-17T04:08:24.185728+0000","last_fullsized":"2024-09-17T04:08:24.185728+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":41,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53152,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318688,"statfs":{"total":96636764160,"available":96582336512,"internally_reserved":0,"allocated":4005888,"data_stored":2104324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509673,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34520,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337320,"statfs":{"total":96636764160,"available":96601415680,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56736,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315104,"statfs":{"total":96636764160,"available":96578666496,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509672,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542375,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56788,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315052,"statfs":{"total":96636764160,"available":96578613248,"internally_reserved":0,"allocated":3469312,"data_stored":1535010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542376,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4067328,"data_stored":2126028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:08:30.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:08:30.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 -- 172.21.15.129:0/2635784095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 msgr2=0x7f7c080b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 --2- 172.21.15.129:0/2635784095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c080b4140 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8005fd0 tx=0x7f7bf802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 -- 172.21.15.129:0/2635784095 shutdown_connections 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 --2- 172.21.15.129:0/2635784095 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c080cd390 0x7f7c081609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 --2- 172.21.15.129:0/2635784095 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c080b4140 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 -- 172.21.15.129:0/2635784095 >> 172.21.15.129:0/2635784095 conn(0x7f7c0806d410 msgr2=0x7f7c080b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 -- 172.21.15.129:0/2635784095 shutdown_connections 2024-09-17T04:08:30.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.353+0000 7f7c0d400640 1 -- 172.21.15.129:0/2635784095 wait complete. 2024-09-17T04:08:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.355+0000 7f7c0d400640 1 Processor -- start 2024-09-17T04:08:30.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.355+0000 7f7c0d400640 1 -- start start 2024-09-17T04:08:30.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 0x7f7c081587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c0809dc20 0x7f7c08074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7c08160f00 con 0x7f7c080705a0 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0806f4f0 con 0x7f7c0809dc20 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c0d400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7c08161080 con 0x7f7c0806eed0 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57268/0 (socket says 172.21.15.129:57268) 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 -- 172.21.15.129:0/1681406357 learned_addr learned my addr 172.21.15.129:0/1681406357 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c06800640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 0x7f7c081587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c07200640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c0809dc20 0x7f7c08074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 -- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 msgr2=0x7f7c081587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 0x7f7c081587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 -- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c0809dc20 msgr2=0x7f7c08074440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.357+0000 7f7c05e00640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c0809dc20 0x7f7c08074440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7c05e00640 1 -- 172.21.15.129:0/1681406357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0815c3b0 con 0x7f7c080705a0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7c06800640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 0x7f7c081587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7c05e00640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c3c0 tx=0x7f7bf400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4017070 con 0x7f7c080705a0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7bf40053e0 con 0x7f7c080705a0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c081f5d20 con 0x7f7c080705a0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4003850 con 0x7f7c080705a0 2024-09-17T04:08:30.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.358+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c081f62e0 con 0x7f7c080705a0 2024-09-17T04:08:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.359+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c08157c00 con 0x7f7c080705a0 2024-09-17T04:08:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.360+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7bf401a070 con 0x7f7c080705a0 2024-09-17T04:08:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.361+0000 7f7bfea00640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 0x7f7be807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.362+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f7bf409a1c0 con 0x7f7c080705a0 2024-09-17T04:08:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.362+0000 7f7c06800640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 0x7f7be807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.363+0000 7f7c06800640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 0x7f7be807b0a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8033000 tx=0x7f7bf8004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:30.366 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.366+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7bf4062560 con 0x7f7c080705a0 2024-09-17T04:08:30.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.546+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7c080b4140 con 0x7f7be8078b90 2024-09-17T04:08:30.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.551+0000 7f7bfea00640 1 -- 172.21.15.129:0/1681406357 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+248654 (secure 0 0 0) 0x7f7c080b4140 con 0x7f7be8078b90 2024-09-17T04:08:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:30.556 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:08:30.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 msgr2=0x7f7be807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.557 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 0x7f7be807b0a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8033000 tx=0x7f7bf8004730 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 msgr2=0x7f7c08073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c3c0 tx=0x7f7bf400c880 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 shutdown_connections 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7be8078b90 0x7f7be807b0a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7c0809dc20 0x7f7c08074440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7c080705a0 0x7f7c08073f00 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 --2- 172.21.15.129:0/1681406357 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7c0806eed0 0x7f7c081587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.555+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 >> 172.21.15.129:0/1681406357 conn(0x7f7c0806d410 msgr2=0x7f7c080c6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.556+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 shutdown_connections 2024-09-17T04:08:30.558 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.556+0000 7f7c0d400640 1 -- 172.21.15.129:0/1681406357 wait complete. 2024-09-17T04:08:30.579 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2024-09-17T04:08:28.620466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592744,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":117,"num_write_kb":1635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420456,"kb_used_data":27956,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754554264,"statfs":{"total":773094113280,"available":772663566336,"internally_reserved":0,"allocated":28626944,"data_stored":13347692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.456722"},"pg_stats":[{"pgid":"41.3f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326478+0000","last_change":"2024-09-17T04:08:25.189216+0000","last_active":"2024-09-17T04:08:26.326478+0000","last_peered":"2024-09-17T04:08:26.326478+0000","last_clean":"2024-09-17T04:08:26.326478+0000","last_became_active":"2024-09-17T04:08:25.188921+0000","last_became_peered":"2024-09-17T04:08:25.188921+0000","last_unstale":"2024-09-17T04:08:26.326478+0000","last_undegraded":"2024-09-17T04:08:26.326478+0000","last_fullsized":"2024-09-17T04:08:26.326478+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:41.624638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189193+0000","last_change":"2024-09-17T04:08:24.189193+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189193+0000","last_undegraded":"2024-09-17T04:08:24.189193+0000","last_fullsized":"2024-09-17T04:08:24.189193+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181899+0000","last_change":"2024-09-17T04:08:24.181899+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181899+0000","last_undegraded":"2024-09-17T04:08:24.181899+0000","last_fullsized":"2024-09-17T04:08:24.181899+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178305+0000","last_change":"2024-09-17T04:08:24.178305+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178305+0000","last_undegraded":"2024-09-17T04:08:24.178305+0000","last_fullsized":"2024-09-17T04:08:24.178305+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.3b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182910+0000","last_change":"2024-09-17T04:08:24.182910+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182910+0000","last_undegraded":"2024-09-17T04:08:24.182910+0000","last_fullsized":"2024-09-17T04:08:24.182910+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.178626+0000","last_change":"2024-09-17T04:08:24.178626+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.178626+0000","last_undegraded":"2024-09-17T04:08:24.178626+0000","last_fullsized":"2024-09-17T04:08:24.178626+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.39","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183640+0000","last_change":"2024-09-17T04:08:24.183640+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183640+0000","last_undegraded":"2024-09-17T04:08:24.183640+0000","last_fullsized":"2024-09-17T04:08:24.183640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.38","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326289+0000","last_change":"2024-09-17T04:08:25.207974+0000","last_active":"2024-09-17T04:08:26.326289+0000","last_peered":"2024-09-17T04:08:26.326289+0000","last_clean":"2024-09-17T04:08:26.326289+0000","last_became_active":"2024-09-17T04:08:25.207454+0000","last_became_peered":"2024-09-17T04:08:25.207454+0000","last_unstale":"2024-09-17T04:08:26.326289+0000","last_undegraded":"2024-09-17T04:08:26.326289+0000","last_fullsized":"2024-09-17T04:08:26.326289+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:18.281022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.37","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182619+0000","last_change":"2024-09-17T04:08:24.182619+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182619+0000","last_undegraded":"2024-09-17T04:08:24.182619+0000","last_fullsized":"2024-09-17T04:08:24.182619+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.36","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185033+0000","last_change":"2024-09-17T04:08:24.185033+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185033+0000","last_undegraded":"2024-09-17T04:08:24.185033+0000","last_fullsized":"2024-09-17T04:08:24.185033+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.35","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.187179+0000","last_change":"2024-09-17T04:08:24.187179+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.187179+0000","last_undegraded":"2024-09-17T04:08:24.187179+0000","last_fullsized":"2024-09-17T04:08:24.187179+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.34","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326226+0000","last_change":"2024-09-17T04:08:25.196813+0000","last_active":"2024-09-17T04:08:26.326226+0000","last_peered":"2024-09-17T04:08:26.326226+0000","last_clean":"2024-09-17T04:08:26.326226+0000","last_became_active":"2024-09-17T04:08:25.196404+0000","last_became_peered":"2024-09-17T04:08:25.196404+0000","last_unstale":"2024-09-17T04:08:26.326226+0000","last_undegraded":"2024-09-17T04:08:26.326226+0000","last_fullsized":"2024-09-17T04:08:26.326226+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:24.380529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.33","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186085+0000","last_change":"2024-09-17T04:08:24.186085+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186085+0000","last_undegraded":"2024-09-17T04:08:24.186085+0000","last_fullsized":"2024-09-17T04:08:24.186085+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.32","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182701+0000","last_change":"2024-09-17T04:08:24.182701+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182701+0000","last_undegraded":"2024-09-17T04:08:24.182701+0000","last_fullsized":"2024-09-17T04:08:24.182701+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.31","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180999+0000","last_change":"2024-09-17T04:08:24.180999+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180999+0000","last_undegraded":"2024-09-17T04:08:24.180999+0000","last_fullsized":"2024-09-17T04:08:24.180999+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.30","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.192580+0000","last_change":"2024-09-17T04:08:24.192580+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.192580+0000","last_undegraded":"2024-09-17T04:08:24.192580+0000","last_fullsized":"2024-09-17T04:08:24.192580+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186791+0000","last_change":"2024-09-17T04:08:24.186791+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186791+0000","last_undegraded":"2024-09-17T04:08:24.186791+0000","last_fullsized":"2024-09-17T04:08:24.186791+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183594+0000","last_change":"2024-09-17T04:08:24.183594+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183594+0000","last_undegraded":"2024-09-17T04:08:24.183594+0000","last_fullsized":"2024-09-17T04:08:24.183594+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.2d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326090+0000","last_change":"2024-09-17T04:08:25.192767+0000","last_active":"2024-09-17T04:08:26.326090+0000","last_peered":"2024-09-17T04:08:26.326090+0000","last_clean":"2024-09-17T04:08:26.326090+0000","last_became_active":"2024-09-17T04:08:25.192484+0000","last_became_peered":"2024-09-17T04:08:25.192484+0000","last_unstale":"2024-09-17T04:08:26.326090+0000","last_undegraded":"2024-09-17T04:08:26.326090+0000","last_fullsized":"2024-09-17T04:08:26.326090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:08.026774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.2c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191776+0000","last_change":"2024-09-17T04:08:24.191776+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191776+0000","last_undegraded":"2024-09-17T04:08:24.191776+0000","last_fullsized":"2024-09-17T04:08:24.191776+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180410+0000","last_change":"2024-09-17T04:08:24.180410+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180410+0000","last_undegraded":"2024-09-17T04:08:24.180410+0000","last_fullsized":"2024-09-17T04:08:24.180410+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.184503+0000","last_change":"2024-09-17T04:08:24.184503+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.184503+0000","last_undegraded":"2024-09-17T04:08:24.184503+0000","last_fullsized":"2024-09-17T04:08:24.184503+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.29","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179821+0000","last_change":"2024-09-17T04:08:24.179821+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179821+0000","last_undegraded":"2024-09-17T04:08:24.179821+0000","last_fullsized":"2024-09-17T04:08:24.179821+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.27","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185990+0000","last_change":"2024-09-17T04:08:24.185990+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185990+0000","last_undegraded":"2024-09-17T04:08:24.185990+0000","last_fullsized":"2024-09-17T04:08:24.185990+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.26","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326742+0000","last_change":"2024-09-17T04:08:25.205002+0000","last_active":"2024-09-17T04:08:26.326742+0000","last_peered":"2024-09-17T04:08:26.326742+0000","last_clean":"2024-09-17T04:08:26.326742+0000","last_became_active":"2024-09-17T04:08:25.204427+0000","last_became_peered":"2024-09-17T04:08:25.204427+0000","last_unstale":"2024-09-17T04:08:26.326742+0000","last_undegraded":"2024-09-17T04:08:26.326742+0000","last_fullsized":"2024-09-17T04:08:26.326742+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:37.951456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.25","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180951+0000","last_change":"2024-09-17T04:08:24.180951+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180951+0000","last_undegraded":"2024-09-17T04:08:24.180951+0000","last_fullsized":"2024-09-17T04:08:24.180951+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.24","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326183+0000","last_change":"2024-09-17T04:08:25.188324+0000","last_active":"2024-09-17T04:08:26.326183+0000","last_peered":"2024-09-17T04:08:26.326183+0000","last_clean":"2024-09-17T04:08:26.326183+0000","last_became_active":"2024-09-17T04:08:25.188025+0000","last_became_peered":"2024-09-17T04:08:25.188025+0000","last_unstale":"2024-09-17T04:08:26.326183+0000","last_undegraded":"2024-09-17T04:08:26.326183+0000","last_fullsized":"2024-09-17T04:08:26.326183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:41.304344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.23","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.189672+0000","last_change":"2024-09-17T04:08:24.189672+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.189672+0000","last_undegraded":"2024-09-17T04:08:24.189672+0000","last_fullsized":"2024-09-17T04:08:24.189672+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.22","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326936+0000","last_change":"2024-09-17T04:08:25.194536+0000","last_active":"2024-09-17T04:08:26.326936+0000","last_peered":"2024-09-17T04:08:26.326936+0000","last_clean":"2024-09-17T04:08:26.326936+0000","last_became_active":"2024-09-17T04:08:25.194201+0000","last_became_peered":"2024-09-17T04:08:25.194201+0000","last_unstale":"2024-09-17T04:08:26.326936+0000","last_undegraded":"2024-09-17T04:08:26.326936+0000","last_fullsized":"2024-09-17T04:08:26.326936+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:14.659279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.21","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326187+0000","last_change":"2024-09-17T04:08:25.198539+0000","last_active":"2024-09-17T04:08:26.326187+0000","last_peered":"2024-09-17T04:08:26.326187+0000","last_clean":"2024-09-17T04:08:26.326187+0000","last_became_active":"2024-09-17T04:08:25.198178+0000","last_became_peered":"2024-09-17T04:08:25.198178+0000","last_unstale":"2024-09-17T04:08:26.326187+0000","last_undegraded":"2024-09-17T04:08:26.326187+0000","last_fullsized":"2024-09-17T04:08:26.326187+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:18.437486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.20","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182935+0000","last_change":"2024-09-17T04:08:24.182935+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182935+0000","last_undegraded":"2024-09-17T04:08:24.182935+0000","last_fullsized":"2024-09-17T04:08:24.182935+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326610+0000","last_change":"2024-09-17T04:08:25.199592+0000","last_active":"2024-09-17T04:08:26.326610+0000","last_peered":"2024-09-17T04:08:26.326610+0000","last_clean":"2024-09-17T04:08:26.326610+0000","last_became_active":"2024-09-17T04:08:25.199254+0000","last_became_peered":"2024-09-17T04:08:25.199254+0000","last_unstale":"2024-09-17T04:08:26.326610+0000","last_undegraded":"2024-09-17T04:08:26.326610+0000","last_fullsized":"2024-09-17T04:08:26.326610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:22.464215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"271'4","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326348+0000","last_change":"2024-09-17T04:08:24.190135+0000","last_active":"2024-09-17T04:08:26.326348+0000","last_peered":"2024-09-17T04:08:26.326348+0000","last_clean":"2024-09-17T04:08:26.326348+0000","last_became_active":"2024-09-17T04:08:24.189843+0000","last_became_peered":"2024-09-17T04:08:24.189843+0000","last_unstale":"2024-09-17T04:08:26.326348+0000","last_undegraded":"2024-09-17T04:08:26.326348+0000","last_fullsized":"2024-09-17T04:08:26.326348+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:29.916388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181601+0000","last_change":"2024-09-17T04:08:24.181601+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181601+0000","last_undegraded":"2024-09-17T04:08:24.181601+0000","last_fullsized":"2024-09-17T04:08:24.181601+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326691+0000","last_change":"2024-09-17T04:08:24.196111+0000","last_active":"2024-09-17T04:08:26.326691+0000","last_peered":"2024-09-17T04:08:26.326691+0000","last_clean":"2024-09-17T04:08:26.326691+0000","last_became_active":"2024-09-17T04:08:24.195595+0000","last_became_peered":"2024-09-17T04:08:24.195595+0000","last_unstale":"2024-09-17T04:08:26.326691+0000","last_undegraded":"2024-09-17T04:08:26.326691+0000","last_fullsized":"2024-09-17T04:08:26.326691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:44.710662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180884+0000","last_change":"2024-09-17T04:08:24.180884+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180884+0000","last_undegraded":"2024-09-17T04:08:24.180884+0000","last_fullsized":"2024-09-17T04:08:24.180884+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.196010+0000","last_change":"2024-09-17T04:08:24.194368+0000","last_active":"2024-09-17T04:08:24.196010+0000","last_peered":"2024-09-17T04:08:24.196010+0000","last_clean":"2024-09-17T04:08:24.196010+0000","last_became_active":"2024-09-17T04:08:24.193648+0000","last_became_peered":"2024-09-17T04:08:24.193648+0000","last_unstale":"2024-09-17T04:08:24.196010+0000","last_undegraded":"2024-09-17T04:08:24.196010+0000","last_fullsized":"2024-09-17T04:08:24.196010+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:09:40.412794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186629+0000","last_change":"2024-09-17T04:08:24.186629+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186629+0000","last_undegraded":"2024-09-17T04:08:24.186629+0000","last_fullsized":"2024-09-17T04:08:24.186629+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203824+0000","last_change":"2024-09-17T04:08:24.202248+0000","last_active":"2024-09-17T04:08:24.203824+0000","last_peered":"2024-09-17T04:08:24.203824+0000","last_clean":"2024-09-17T04:08:24.203824+0000","last_became_active":"2024-09-17T04:08:24.201709+0000","last_became_peered":"2024-09-17T04:08:24.201709+0000","last_unstale":"2024-09-17T04:08:24.203824+0000","last_undegraded":"2024-09-17T04:08:24.203824+0000","last_fullsized":"2024-09-17T04:08:24.203824+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:16.873589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.180525+0000","last_change":"2024-09-17T04:08:24.180525+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.180525+0000","last_undegraded":"2024-09-17T04:08:24.180525+0000","last_fullsized":"2024-09-17T04:08:24.180525+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.201402+0000","last_change":"2024-09-17T04:08:24.200461+0000","last_active":"2024-09-17T04:08:24.201402+0000","last_peered":"2024-09-17T04:08:24.201402+0000","last_clean":"2024-09-17T04:08:24.201402+0000","last_became_active":"2024-09-17T04:08:24.199506+0000","last_became_peered":"2024-09-17T04:08:24.199506+0000","last_unstale":"2024-09-17T04:08:24.201402+0000","last_undegraded":"2024-09-17T04:08:24.201402+0000","last_fullsized":"2024-09-17T04:08:24.201402+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:26:31.705720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325694+0000","last_change":"2024-09-17T04:08:25.185548+0000","last_active":"2024-09-17T04:08:26.325694+0000","last_peered":"2024-09-17T04:08:26.325694+0000","last_clean":"2024-09-17T04:08:26.325694+0000","last_became_active":"2024-09-17T04:08:25.184461+0000","last_became_peered":"2024-09-17T04:08:25.184461+0000","last_unstale":"2024-09-17T04:08:26.325694+0000","last_undegraded":"2024-09-17T04:08:26.325694+0000","last_fullsized":"2024-09-17T04:08:26.325694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:46.956363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325684+0000","last_change":"2024-09-17T04:08:24.187071+0000","last_active":"2024-09-17T04:08:26.325684+0000","last_peered":"2024-09-17T04:08:26.325684+0000","last_clean":"2024-09-17T04:08:26.325684+0000","last_became_active":"2024-09-17T04:08:24.186743+0000","last_became_peered":"2024-09-17T04:08:24.186743+0000","last_unstale":"2024-09-17T04:08:26.325684+0000","last_undegraded":"2024-09-17T04:08:26.325684+0000","last_fullsized":"2024-09-17T04:08:26.325684+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:15.930301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.191846+0000","last_change":"2024-09-17T04:08:24.191846+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.191846+0000","last_undegraded":"2024-09-17T04:08:24.191846+0000","last_fullsized":"2024-09-17T04:08:24.191846+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199751+0000","last_change":"2024-09-17T04:08:24.192860+0000","last_active":"2024-09-17T04:08:24.199751+0000","last_peered":"2024-09-17T04:08:24.199751+0000","last_clean":"2024-09-17T04:08:24.199751+0000","last_became_active":"2024-09-17T04:08:24.192505+0000","last_became_peered":"2024-09-17T04:08:24.192505+0000","last_unstale":"2024-09-17T04:08:24.199751+0000","last_undegraded":"2024-09-17T04:08:24.199751+0000","last_fullsized":"2024-09-17T04:08:24.199751+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:10:33.412525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325700+0000","last_change":"2024-09-17T04:08:25.195702+0000","last_active":"2024-09-17T04:08:26.325700+0000","last_peered":"2024-09-17T04:08:26.325700+0000","last_clean":"2024-09-17T04:08:26.325700+0000","last_became_active":"2024-09-17T04:08:25.195370+0000","last_became_peered":"2024-09-17T04:08:25.195370+0000","last_unstale":"2024-09-17T04:08:26.325700+0000","last_undegraded":"2024-09-17T04:08:26.325700+0000","last_fullsized":"2024-09-17T04:08:26.325700+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:35.074389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.198388+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.197800+0000","last_became_peered":"2024-09-17T04:08:24.197800+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:50.784271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185710+0000","last_change":"2024-09-17T04:08:24.185710+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185710+0000","last_undegraded":"2024-09-17T04:08:24.185710+0000","last_fullsized":"2024-09-17T04:08:24.185710+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200361+0000","last_change":"2024-09-17T04:08:24.197553+0000","last_active":"2024-09-17T04:08:24.200361+0000","last_peered":"2024-09-17T04:08:24.200361+0000","last_clean":"2024-09-17T04:08:24.200361+0000","last_became_active":"2024-09-17T04:08:24.197233+0000","last_became_peered":"2024-09-17T04:08:24.197233+0000","last_unstale":"2024-09-17T04:08:24.200361+0000","last_undegraded":"2024-09-17T04:08:24.200361+0000","last_fullsized":"2024-09-17T04:08:24.200361+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:52:08.411479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181356+0000","last_change":"2024-09-17T04:08:24.181356+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181356+0000","last_undegraded":"2024-09-17T04:08:24.181356+0000","last_fullsized":"2024-09-17T04:08:24.181356+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326324+0000","last_change":"2024-09-17T04:08:24.191873+0000","last_active":"2024-09-17T04:08:26.326324+0000","last_peered":"2024-09-17T04:08:26.326324+0000","last_clean":"2024-09-17T04:08:26.326324+0000","last_became_active":"2024-09-17T04:08:24.191318+0000","last_became_peered":"2024-09-17T04:08:24.191318+0000","last_unstale":"2024-09-17T04:08:26.326324+0000","last_undegraded":"2024-09-17T04:08:26.326324+0000","last_fullsized":"2024-09-17T04:08:26.326324+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:17.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188680+0000","last_change":"2024-09-17T04:08:24.188680+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188680+0000","last_undegraded":"2024-09-17T04:08:24.188680+0000","last_fullsized":"2024-09-17T04:08:24.188680+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"271'3","reported_seq":18,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326468+0000","last_change":"2024-09-17T04:08:24.190381+0000","last_active":"2024-09-17T04:08:26.326468+0000","last_peered":"2024-09-17T04:08:26.326468+0000","last_clean":"2024-09-17T04:08:26.326468+0000","last_became_active":"2024-09-17T04:08:24.189870+0000","last_became_peered":"2024-09-17T04:08:24.189870+0000","last_unstale":"2024-09-17T04:08:26.326468+0000","last_undegraded":"2024-09-17T04:08:26.326468+0000","last_fullsized":"2024-09-17T04:08:26.326468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:11.493387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326092+0000","last_change":"2024-09-17T04:08:25.202764+0000","last_active":"2024-09-17T04:08:26.326092+0000","last_peered":"2024-09-17T04:08:26.326092+0000","last_clean":"2024-09-17T04:08:26.326092+0000","last_became_active":"2024-09-17T04:08:25.202410+0000","last_became_peered":"2024-09-17T04:08:25.202410+0000","last_unstale":"2024-09-17T04:08:26.326092+0000","last_undegraded":"2024-09-17T04:08:26.326092+0000","last_fullsized":"2024-09-17T04:08:26.326092+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:47.028739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"271'6","reported_seq":21,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325975+0000","last_change":"2024-09-17T04:08:24.200801+0000","last_active":"2024-09-17T04:08:26.325975+0000","last_peered":"2024-09-17T04:08:26.325975+0000","last_clean":"2024-09-17T04:08:26.325975+0000","last_became_active":"2024-09-17T04:08:24.199940+0000","last_became_peered":"2024-09-17T04:08:24.199940+0000","last_unstale":"2024-09-17T04:08:26.325975+0000","last_undegraded":"2024-09-17T04:08:26.325975+0000","last_fullsized":"2024-09-17T04:08:26.325975+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:53.586256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326081+0000","last_change":"2024-09-17T04:08:25.206679+0000","last_active":"2024-09-17T04:08:26.326081+0000","last_peered":"2024-09-17T04:08:26.326081+0000","last_clean":"2024-09-17T04:08:26.326081+0000","last_became_active":"2024-09-17T04:08:25.206365+0000","last_became_peered":"2024-09-17T04:08:25.206365+0000","last_unstale":"2024-09-17T04:08:26.326081+0000","last_undegraded":"2024-09-17T04:08:26.326081+0000","last_fullsized":"2024-09-17T04:08:26.326081+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:30.096668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195422+0000","last_change":"2024-09-17T04:08:24.193138+0000","last_active":"2024-09-17T04:08:24.195422+0000","last_peered":"2024-09-17T04:08:24.195422+0000","last_clean":"2024-09-17T04:08:24.195422+0000","last_became_active":"2024-09-17T04:08:24.192861+0000","last_became_peered":"2024-09-17T04:08:24.192861+0000","last_unstale":"2024-09-17T04:08:24.195422+0000","last_undegraded":"2024-09-17T04:08:24.195422+0000","last_fullsized":"2024-09-17T04:08:24.195422+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:27:21.860819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188799+0000","last_change":"2024-09-17T04:08:24.188799+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188799+0000","last_undegraded":"2024-09-17T04:08:24.188799+0000","last_fullsized":"2024-09-17T04:08:24.188799+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.198667+0000","last_change":"2024-09-17T04:08:24.197968+0000","last_active":"2024-09-17T04:08:24.198667+0000","last_peered":"2024-09-17T04:08:24.198667+0000","last_clean":"2024-09-17T04:08:24.198667+0000","last_became_active":"2024-09-17T04:08:24.197333+0000","last_became_peered":"2024-09-17T04:08:24.197333+0000","last_unstale":"2024-09-17T04:08:24.198667+0000","last_undegraded":"2024-09-17T04:08:24.198667+0000","last_fullsized":"2024-09-17T04:08:24.198667+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:16.246224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179664+0000","last_change":"2024-09-17T04:08:24.179664+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179664+0000","last_undegraded":"2024-09-17T04:08:24.179664+0000","last_fullsized":"2024-09-17T04:08:24.179664+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200585+0000","last_change":"2024-09-17T04:08:24.199871+0000","last_active":"2024-09-17T04:08:24.200585+0000","last_peered":"2024-09-17T04:08:24.200585+0000","last_clean":"2024-09-17T04:08:24.200585+0000","last_became_active":"2024-09-17T04:08:24.199516+0000","last_became_peered":"2024-09-17T04:08:24.199516+0000","last_unstale":"2024-09-17T04:08:24.200585+0000","last_undegraded":"2024-09-17T04:08:24.200585+0000","last_fullsized":"2024-09-17T04:08:24.200585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:06:08.418056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.204022+0000","last_change":"2024-09-17T04:08:24.202703+0000","last_active":"2024-09-17T04:08:24.204022+0000","last_peered":"2024-09-17T04:08:24.204022+0000","last_clean":"2024-09-17T04:08:24.204022+0000","last_became_active":"2024-09-17T04:08:24.201244+0000","last_became_peered":"2024-09-17T04:08:24.201244+0000","last_unstale":"2024-09-17T04:08:24.204022+0000","last_undegraded":"2024-09-17T04:08:24.204022+0000","last_fullsized":"2024-09-17T04:08:24.204022+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:37.878181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.188660+0000","last_change":"2024-09-17T04:08:24.188660+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.188660+0000","last_undegraded":"2024-09-17T04:08:24.188660+0000","last_fullsized":"2024-09-17T04:08:24.188660+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326899+0000","last_change":"2024-09-17T04:08:24.198617+0000","last_active":"2024-09-17T04:08:26.326899+0000","last_peered":"2024-09-17T04:08:26.326899+0000","last_clean":"2024-09-17T04:08:26.326899+0000","last_became_active":"2024-09-17T04:08:24.198153+0000","last_became_peered":"2024-09-17T04:08:24.198153+0000","last_unstale":"2024-09-17T04:08:26.326899+0000","last_undegraded":"2024-09-17T04:08:26.326899+0000","last_fullsized":"2024-09-17T04:08:26.326899+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:23.292656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.185175+0000","last_change":"2024-09-17T04:08:24.185175+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.185175+0000","last_undegraded":"2024-09-17T04:08:24.185175+0000","last_fullsized":"2024-09-17T04:08:24.185175+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.199626+0000","last_change":"2024-09-17T04:08:24.195281+0000","last_active":"2024-09-17T04:08:24.199626+0000","last_peered":"2024-09-17T04:08:24.199626+0000","last_clean":"2024-09-17T04:08:24.199626+0000","last_became_active":"2024-09-17T04:08:24.194954+0000","last_became_peered":"2024-09-17T04:08:24.194954+0000","last_unstale":"2024-09-17T04:08:24.199626+0000","last_undegraded":"2024-09-17T04:08:24.199626+0000","last_fullsized":"2024-09-17T04:08:24.199626+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:51:56.155806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181698+0000","last_change":"2024-09-17T04:08:24.181698+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181698+0000","last_undegraded":"2024-09-17T04:08:24.181698+0000","last_fullsized":"2024-09-17T04:08:24.181698+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326585+0000","last_change":"2024-09-17T04:08:24.199722+0000","last_active":"2024-09-17T04:08:26.326585+0000","last_peered":"2024-09-17T04:08:26.326585+0000","last_clean":"2024-09-17T04:08:26.326585+0000","last_became_active":"2024-09-17T04:08:24.199405+0000","last_became_peered":"2024-09-17T04:08:24.199405+0000","last_unstale":"2024-09-17T04:08:26.326585+0000","last_undegraded":"2024-09-17T04:08:26.326585+0000","last_fullsized":"2024-09-17T04:08:26.326585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:34.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326388+0000","last_change":"2024-09-17T04:08:25.187530+0000","last_active":"2024-09-17T04:08:26.326388+0000","last_peered":"2024-09-17T04:08:26.326388+0000","last_clean":"2024-09-17T04:08:26.326388+0000","last_became_active":"2024-09-17T04:08:25.187224+0000","last_became_peered":"2024-09-17T04:08:25.187224+0000","last_unstale":"2024-09-17T04:08:26.326388+0000","last_undegraded":"2024-09-17T04:08:26.326388+0000","last_fullsized":"2024-09-17T04:08:26.326388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:35.040735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203439+0000","last_change":"2024-09-17T04:08:24.199980+0000","last_active":"2024-09-17T04:08:24.203439+0000","last_peered":"2024-09-17T04:08:24.203439+0000","last_clean":"2024-09-17T04:08:24.203439+0000","last_became_active":"2024-09-17T04:08:24.199415+0000","last_became_peered":"2024-09-17T04:08:24.199415+0000","last_unstale":"2024-09-17T04:08:24.203439+0000","last_undegraded":"2024-09-17T04:08:24.203439+0000","last_fullsized":"2024-09-17T04:08:24.203439+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:40:29.800724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179834+0000","last_change":"2024-09-17T04:08:24.179834+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179834+0000","last_undegraded":"2024-09-17T04:08:24.179834+0000","last_fullsized":"2024-09-17T04:08:24.179834+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200112+0000","last_change":"2024-09-17T04:08:24.198577+0000","last_active":"2024-09-17T04:08:24.200112+0000","last_peered":"2024-09-17T04:08:24.200112+0000","last_clean":"2024-09-17T04:08:24.200112+0000","last_became_active":"2024-09-17T04:08:24.198252+0000","last_became_peered":"2024-09-17T04:08:24.198252+0000","last_unstale":"2024-09-17T04:08:24.200112+0000","last_undegraded":"2024-09-17T04:08:24.200112+0000","last_fullsized":"2024-09-17T04:08:24.200112+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:45:41.168020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179965+0000","last_change":"2024-09-17T04:08:24.179965+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179965+0000","last_undegraded":"2024-09-17T04:08:24.179965+0000","last_fullsized":"2024-09-17T04:08:24.179965+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191487+0000","last_change":"2024-09-17T04:08:24.187161+0000","last_active":"2024-09-17T04:08:24.191487+0000","last_peered":"2024-09-17T04:08:24.191487+0000","last_clean":"2024-09-17T04:08:24.191487+0000","last_became_active":"2024-09-17T04:08:24.186572+0000","last_became_peered":"2024-09-17T04:08:24.186572+0000","last_unstale":"2024-09-17T04:08:24.191487+0000","last_undegraded":"2024-09-17T04:08:24.191487+0000","last_fullsized":"2024-09-17T04:08:24.191487+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:22:55.809038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.190666+0000","last_change":"2024-09-17T04:08:24.190666+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.190666+0000","last_undegraded":"2024-09-17T04:08:24.190666+0000","last_fullsized":"2024-09-17T04:08:24.190666+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.200730+0000","last_change":"2024-09-17T04:08:24.196657+0000","last_active":"2024-09-17T04:08:24.200730+0000","last_peered":"2024-09-17T04:08:24.200730+0000","last_clean":"2024-09-17T04:08:24.200730+0000","last_became_active":"2024-09-17T04:08:24.195146+0000","last_became_peered":"2024-09-17T04:08:24.195146+0000","last_unstale":"2024-09-17T04:08:24.200730+0000","last_undegraded":"2024-09-17T04:08:24.200730+0000","last_fullsized":"2024-09-17T04:08:24.200730+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:41:21.531394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325728+0000","last_change":"2024-09-17T04:08:25.184757+0000","last_active":"2024-09-17T04:08:26.325728+0000","last_peered":"2024-09-17T04:08:26.325728+0000","last_clean":"2024-09-17T04:08:26.325728+0000","last_became_active":"2024-09-17T04:08:25.184177+0000","last_became_peered":"2024-09-17T04:08:25.184177+0000","last_unstale":"2024-09-17T04:08:26.325728+0000","last_undegraded":"2024-09-17T04:08:26.325728+0000","last_fullsized":"2024-09-17T04:08:26.325728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:35.563823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.184366+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.182740+0000","last_became_peered":"2024-09-17T04:08:24.182740+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:25.140619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.182099+0000","last_change":"2024-09-17T04:08:24.182099+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.182099+0000","last_undegraded":"2024-09-17T04:08:24.182099+0000","last_fullsized":"2024-09-17T04:08:24.182099+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.197037+0000","last_change":"2024-09-17T04:08:24.196057+0000","last_active":"2024-09-17T04:08:24.197037+0000","last_peered":"2024-09-17T04:08:24.197037+0000","last_clean":"2024-09-17T04:08:24.197037+0000","last_became_active":"2024-09-17T04:08:24.195822+0000","last_became_peered":"2024-09-17T04:08:24.195822+0000","last_unstale":"2024-09-17T04:08:24.197037+0000","last_undegraded":"2024-09-17T04:08:24.197037+0000","last_fullsized":"2024-09-17T04:08:24.197037+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:05:26.354317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326477+0000","last_change":"2024-09-17T04:08:25.207446+0000","last_active":"2024-09-17T04:08:26.326477+0000","last_peered":"2024-09-17T04:08:26.326477+0000","last_clean":"2024-09-17T04:08:26.326477+0000","last_became_active":"2024-09-17T04:08:25.207098+0000","last_became_peered":"2024-09-17T04:08:25.207098+0000","last_unstale":"2024-09-17T04:08:26.326477+0000","last_undegraded":"2024-09-17T04:08:26.326477+0000","last_fullsized":"2024-09-17T04:08:26.326477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:13.056929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.203840+0000","last_change":"2024-09-17T04:08:24.202471+0000","last_active":"2024-09-17T04:08:24.203840+0000","last_peered":"2024-09-17T04:08:24.203840+0000","last_clean":"2024-09-17T04:08:24.203840+0000","last_became_active":"2024-09-17T04:08:24.202123+0000","last_became_peered":"2024-09-17T04:08:24.202123+0000","last_unstale":"2024-09-17T04:08:24.203840+0000","last_undegraded":"2024-09-17T04:08:24.203840+0000","last_fullsized":"2024-09-17T04:08:24.203840+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:24:48.755856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326446+0000","last_change":"2024-09-17T04:08:25.196030+0000","last_active":"2024-09-17T04:08:26.326446+0000","last_peered":"2024-09-17T04:08:26.326446+0000","last_clean":"2024-09-17T04:08:26.326446+0000","last_became_active":"2024-09-17T04:08:25.195608+0000","last_became_peered":"2024-09-17T04:08:25.195608+0000","last_unstale":"2024-09-17T04:08:26.326446+0000","last_undegraded":"2024-09-17T04:08:26.326446+0000","last_fullsized":"2024-09-17T04:08:26.326446+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:59.770015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326338+0000","last_change":"2024-09-17T04:08:24.199653+0000","last_active":"2024-09-17T04:08:26.326338+0000","last_peered":"2024-09-17T04:08:26.326338+0000","last_clean":"2024-09-17T04:08:26.326338+0000","last_became_active":"2024-09-17T04:08:24.197097+0000","last_became_peered":"2024-09-17T04:08:24.197097+0000","last_unstale":"2024-09-17T04:08:26.326338+0000","last_undegraded":"2024-09-17T04:08:26.326338+0000","last_fullsized":"2024-09-17T04:08:26.326338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:01.054583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.186648+0000","last_change":"2024-09-17T04:08:24.186648+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.186648+0000","last_undegraded":"2024-09-17T04:08:24.186648+0000","last_fullsized":"2024-09-17T04:08:24.186648+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326117+0000","last_change":"2024-09-17T04:08:24.200442+0000","last_active":"2024-09-17T04:08:26.326117+0000","last_peered":"2024-09-17T04:08:26.326117+0000","last_clean":"2024-09-17T04:08:26.326117+0000","last_became_active":"2024-09-17T04:08:24.199560+0000","last_became_peered":"2024-09-17T04:08:24.199560+0000","last_unstale":"2024-09-17T04:08:26.326117+0000","last_undegraded":"2024-09-17T04:08:26.326117+0000","last_fullsized":"2024-09-17T04:08:26.326117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:07.964242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.183889+0000","last_change":"2024-09-17T04:08:24.183889+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.183889+0000","last_undegraded":"2024-09-17T04:08:24.183889+0000","last_fullsized":"2024-09-17T04:08:24.183889+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325805+0000","last_change":"2024-09-17T04:08:24.187060+0000","last_active":"2024-09-17T04:08:26.325805+0000","last_peered":"2024-09-17T04:08:26.325805+0000","last_clean":"2024-09-17T04:08:26.325805+0000","last_became_active":"2024-09-17T04:08:24.186004+0000","last_became_peered":"2024-09-17T04:08:24.186004+0000","last_unstale":"2024-09-17T04:08:26.325805+0000","last_undegraded":"2024-09-17T04:08:26.325805+0000","last_fullsized":"2024-09-17T04:08:26.325805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:11.972381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179560+0000","last_change":"2024-09-17T04:08:24.179560+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179560+0000","last_undegraded":"2024-09-17T04:08:24.179560+0000","last_fullsized":"2024-09-17T04:08:24.179560+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.191457+0000","last_change":"2024-09-17T04:08:24.186853+0000","last_active":"2024-09-17T04:08:24.191457+0000","last_peered":"2024-09-17T04:08:24.191457+0000","last_clean":"2024-09-17T04:08:24.191457+0000","last_became_active":"2024-09-17T04:08:24.186204+0000","last_became_peered":"2024-09-17T04:08:24.186204+0000","last_unstale":"2024-09-17T04:08:24.191457+0000","last_undegraded":"2024-09-17T04:08:24.191457+0000","last_fullsized":"2024-09-17T04:08:24.191457+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:05:43.911349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.181036+0000","last_change":"2024-09-17T04:08:24.181036+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.181036+0000","last_undegraded":"2024-09-17T04:08:24.181036+0000","last_fullsized":"2024-09-17T04:08:24.181036+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.195797+0000","last_change":"2024-09-17T04:08:24.194575+0000","last_active":"2024-09-17T04:08:24.195797+0000","last_peered":"2024-09-17T04:08:24.195797+0000","last_clean":"2024-09-17T04:08:24.195797+0000","last_became_active":"2024-09-17T04:08:24.194250+0000","last_became_peered":"2024-09-17T04:08:24.194250+0000","last_unstale":"2024-09-17T04:08:24.195797+0000","last_undegraded":"2024-09-17T04:08:24.195797+0000","last_fullsized":"2024-09-17T04:08:24.195797+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:01.637055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":1,"reported_epoch":270,"state":"creating+peering","last_fresh":"2024-09-17T04:08:24.179027+0000","last_change":"2024-09-17T04:08:24.179027+0000","last_active":"2024-09-17T04:08:24.161861+0000","last_peered":"2024-09-17T04:08:24.161861+0000","last_clean":"2024-09-17T04:08:24.161861+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-17T04:08:24.179027+0000","last_undegraded":"2024-09-17T04:08:24.179027+0000","last_fullsized":"2024-09-17T04:08:24.179027+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":10,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.193620+0000","last_change":"2024-09-17T04:08:24.187831+0000","last_active":"2024-09-17T04:08:24.193620+0000","last_peered":"2024-09-17T04:08:24.193620+0000","last_clean":"2024-09-17T04:08:24.193620+0000","last_became_active":"2024-09-17T04:08:24.186781+0000","last_became_peered":"2024-09-17T04:08:24.186781+0000","last_unstale":"2024-09-17T04:08:24.193620+0000","last_undegraded":"2024-09-17T04:08:24.193620+0000","last_fullsized":"2024-09-17T04:08:24.193620+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:15.424529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.28","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326126+0000","last_change":"2024-09-17T04:08:25.204595+0000","last_active":"2024-09-17T04:08:26.326126+0000","last_peered":"2024-09-17T04:08:26.326126+0000","last_clean":"2024-09-17T04:08:26.326126+0000","last_became_active":"2024-09-17T04:08:25.204271+0000","last_became_peered":"2024-09-17T04:08:25.204271+0000","last_unstale":"2024-09-17T04:08:26.326126+0000","last_undegraded":"2024-09-17T04:08:26.326126+0000","last_fullsized":"2024-09-17T04:08:26.326126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:34.822087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":571,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-09-17T04:08:24.185728+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:08:24.185728+0000","last_peered":"2024-09-17T04:08:24.185728+0000","last_clean":"2024-09-17T04:08:24.185728+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:08:24.185728+0000","last_undegraded":"2024-09-17T04:08:24.185728+0000","last_fullsized":"2024-09-17T04:08:24.185728+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":41,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":2376,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53152,"kb_used_data":3912,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318688,"statfs":{"total":96636764160,"available":96582336512,"internally_reserved":0,"allocated":4005888,"data_stored":2104324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509673,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34520,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337320,"statfs":{"total":96636764160,"available":96601415680,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56736,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315104,"statfs":{"total":96636764160,"available":96578666496,"internally_reserved":0,"allocated":3411968,"data_stored":1513956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56740,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315100,"statfs":{"total":96636764160,"available":96578662400,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509672,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52580,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319260,"statfs":{"total":96636764160,"available":96582922240,"internally_reserved":0,"allocated":3424256,"data_stored":1520231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542375,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56788,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315052,"statfs":{"total":96636764160,"available":96578613248,"internally_reserved":0,"allocated":3469312,"data_stored":1535010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542376,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4067328,"data_stored":2126028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:08:30.583 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3e is not active+clean 2024-09-17T04:08:30.583 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.189193+0000', 'last_change': '2024-09-17T04:08:24.189193+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.189193+0000', 'last_undegraded': '2024-09-17T04:08:24.189193+0000', 'last_fullsized': '2024-09-17T04:08:24.189193+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.583 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3d is not active+clean 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.181899+0000', 'last_change': '2024-09-17T04:08:24.181899+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.181899+0000', 'last_undegraded': '2024-09-17T04:08:24.181899+0000', 'last_fullsized': '2024-09-17T04:08:24.181899+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3c is not active+clean 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.178305+0000', 'last_change': '2024-09-17T04:08:24.178305+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.178305+0000', 'last_undegraded': '2024-09-17T04:08:24.178305+0000', 'last_fullsized': '2024-09-17T04:08:24.178305+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3b is not active+clean 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.182910+0000', 'last_change': '2024-09-17T04:08:24.182910+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.182910+0000', 'last_undegraded': '2024-09-17T04:08:24.182910+0000', 'last_fullsized': '2024-09-17T04:08:24.182910+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3a is not active+clean 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.178626+0000', 'last_change': '2024-09-17T04:08:24.178626+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.178626+0000', 'last_undegraded': '2024-09-17T04:08:24.178626+0000', 'last_fullsized': '2024-09-17T04:08:24.178626+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:PG 41.39 is not active+clean 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.183640+0000', 'last_change': '2024-09-17T04:08:24.183640+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.183640+0000', 'last_undegraded': '2024-09-17T04:08:24.183640+0000', 'last_fullsized': '2024-09-17T04:08:24.183640+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.584 INFO:tasks.ceph.ceph_manager.ceph:PG 41.37 is not active+clean 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.182619+0000', 'last_change': '2024-09-17T04:08:24.182619+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.182619+0000', 'last_undegraded': '2024-09-17T04:08:24.182619+0000', 'last_fullsized': '2024-09-17T04:08:24.182619+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:PG 41.36 is not active+clean 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.185033+0000', 'last_change': '2024-09-17T04:08:24.185033+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.185033+0000', 'last_undegraded': '2024-09-17T04:08:24.185033+0000', 'last_fullsized': '2024-09-17T04:08:24.185033+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:PG 41.35 is not active+clean 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.187179+0000', 'last_change': '2024-09-17T04:08:24.187179+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.187179+0000', 'last_undegraded': '2024-09-17T04:08:24.187179+0000', 'last_fullsized': '2024-09-17T04:08:24.187179+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:PG 41.33 is not active+clean 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.186085+0000', 'last_change': '2024-09-17T04:08:24.186085+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.186085+0000', 'last_undegraded': '2024-09-17T04:08:24.186085+0000', 'last_fullsized': '2024-09-17T04:08:24.186085+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:PG 41.32 is not active+clean 2024-09-17T04:08:30.585 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.182701+0000', 'last_change': '2024-09-17T04:08:24.182701+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.182701+0000', 'last_undegraded': '2024-09-17T04:08:24.182701+0000', 'last_fullsized': '2024-09-17T04:08:24.182701+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:PG 41.31 is not active+clean 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.180999+0000', 'last_change': '2024-09-17T04:08:24.180999+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.180999+0000', 'last_undegraded': '2024-09-17T04:08:24.180999+0000', 'last_fullsized': '2024-09-17T04:08:24.180999+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:PG 41.30 is not active+clean 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.192580+0000', 'last_change': '2024-09-17T04:08:24.192580+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.192580+0000', 'last_undegraded': '2024-09-17T04:08:24.192580+0000', 'last_fullsized': '2024-09-17T04:08:24.192580+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2f is not active+clean 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.186791+0000', 'last_change': '2024-09-17T04:08:24.186791+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.186791+0000', 'last_undegraded': '2024-09-17T04:08:24.186791+0000', 'last_fullsized': '2024-09-17T04:08:24.186791+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2e is not active+clean 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.183594+0000', 'last_change': '2024-09-17T04:08:24.183594+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.183594+0000', 'last_undegraded': '2024-09-17T04:08:24.183594+0000', 'last_fullsized': '2024-09-17T04:08:24.183594+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2c is not active+clean 2024-09-17T04:08:30.586 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.191776+0000', 'last_change': '2024-09-17T04:08:24.191776+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.191776+0000', 'last_undegraded': '2024-09-17T04:08:24.191776+0000', 'last_fullsized': '2024-09-17T04:08:24.191776+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2b is not active+clean 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.180410+0000', 'last_change': '2024-09-17T04:08:24.180410+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.180410+0000', 'last_undegraded': '2024-09-17T04:08:24.180410+0000', 'last_fullsized': '2024-09-17T04:08:24.180410+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2a is not active+clean 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.184503+0000', 'last_change': '2024-09-17T04:08:24.184503+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.184503+0000', 'last_undegraded': '2024-09-17T04:08:24.184503+0000', 'last_fullsized': '2024-09-17T04:08:24.184503+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.29 is not active+clean 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179821+0000', 'last_change': '2024-09-17T04:08:24.179821+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179821+0000', 'last_undegraded': '2024-09-17T04:08:24.179821+0000', 'last_fullsized': '2024-09-17T04:08:24.179821+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.27 is not active+clean 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.185990+0000', 'last_change': '2024-09-17T04:08:24.185990+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.185990+0000', 'last_undegraded': '2024-09-17T04:08:24.185990+0000', 'last_fullsized': '2024-09-17T04:08:24.185990+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.25 is not active+clean 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.180951+0000', 'last_change': '2024-09-17T04:08:24.180951+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.180951+0000', 'last_undegraded': '2024-09-17T04:08:24.180951+0000', 'last_fullsized': '2024-09-17T04:08:24.180951+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.587 INFO:tasks.ceph.ceph_manager.ceph:PG 41.23 is not active+clean 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.189672+0000', 'last_change': '2024-09-17T04:08:24.189672+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.189672+0000', 'last_undegraded': '2024-09-17T04:08:24.189672+0000', 'last_fullsized': '2024-09-17T04:08:24.189672+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:PG 41.20 is not active+clean 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.182935+0000', 'last_change': '2024-09-17T04:08:24.182935+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.182935+0000', 'last_undegraded': '2024-09-17T04:08:24.182935+0000', 'last_fullsized': '2024-09-17T04:08:24.182935+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:PG 41.1f is not active+clean 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.181601+0000', 'last_change': '2024-09-17T04:08:24.181601+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.181601+0000', 'last_undegraded': '2024-09-17T04:08:24.181601+0000', 'last_fullsized': '2024-09-17T04:08:24.181601+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:PG 41.1c is not active+clean 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.180884+0000', 'last_change': '2024-09-17T04:08:24.180884+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.180884+0000', 'last_undegraded': '2024-09-17T04:08:24.180884+0000', 'last_fullsized': '2024-09-17T04:08:24.180884+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:PG 41.1d is not active+clean 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.186629+0000', 'last_change': '2024-09-17T04:08:24.186629+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.186629+0000', 'last_undegraded': '2024-09-17T04:08:24.186629+0000', 'last_fullsized': '2024-09-17T04:08:24.186629+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.588 INFO:tasks.ceph.ceph_manager.ceph:PG 41.1a is not active+clean 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.180525+0000', 'last_change': '2024-09-17T04:08:24.180525+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.180525+0000', 'last_undegraded': '2024-09-17T04:08:24.180525+0000', 'last_fullsized': '2024-09-17T04:08:24.180525+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:PG 41.18 is not active+clean 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.191846+0000', 'last_change': '2024-09-17T04:08:24.191846+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.191846+0000', 'last_undegraded': '2024-09-17T04:08:24.191846+0000', 'last_fullsized': '2024-09-17T04:08:24.191846+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:PG 41.16 is not active+clean 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.185710+0000', 'last_change': '2024-09-17T04:08:24.185710+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.185710+0000', 'last_undegraded': '2024-09-17T04:08:24.185710+0000', 'last_fullsized': '2024-09-17T04:08:24.185710+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:PG 41.17 is not active+clean 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.181356+0000', 'last_change': '2024-09-17T04:08:24.181356+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.181356+0000', 'last_undegraded': '2024-09-17T04:08:24.181356+0000', 'last_fullsized': '2024-09-17T04:08:24.181356+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:PG 41.14 is not active+clean 2024-09-17T04:08:30.589 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.188680+0000', 'last_change': '2024-09-17T04:08:24.188680+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.188680+0000', 'last_undegraded': '2024-09-17T04:08:24.188680+0000', 'last_fullsized': '2024-09-17T04:08:24.188680+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.13 is not active+clean 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.188799+0000', 'last_change': '2024-09-17T04:08:24.188799+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.188799+0000', 'last_undegraded': '2024-09-17T04:08:24.188799+0000', 'last_fullsized': '2024-09-17T04:08:24.188799+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.10 is not active+clean 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179664+0000', 'last_change': '2024-09-17T04:08:24.179664+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179664+0000', 'last_undegraded': '2024-09-17T04:08:24.179664+0000', 'last_fullsized': '2024-09-17T04:08:24.179664+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.11 is not active+clean 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.188660+0000', 'last_change': '2024-09-17T04:08:24.188660+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.188660+0000', 'last_undegraded': '2024-09-17T04:08:24.188660+0000', 'last_fullsized': '2024-09-17T04:08:24.188660+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.e is not active+clean 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.188660+0000', 'last_change': '2024-09-17T04:08:24.188660+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.188660+0000', 'last_undegraded': '2024-09-17T04:08:24.188660+0000', 'last_fullsized': '2024-09-17T04:08:24.188660+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.f is not active+clean 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.185175+0000', 'last_change': '2024-09-17T04:08:24.185175+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.185175+0000', 'last_undegraded': '2024-09-17T04:08:24.185175+0000', 'last_fullsized': '2024-09-17T04:08:24.185175+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:08:30.590 INFO:tasks.ceph.ceph_manager.ceph:PG 41.c is not active+clean 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.181698+0000', 'last_change': '2024-09-17T04:08:24.181698+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.181698+0000', 'last_undegraded': '2024-09-17T04:08:24.181698+0000', 'last_fullsized': '2024-09-17T04:08:24.181698+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:PG 41.a is not active+clean 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179834+0000', 'last_change': '2024-09-17T04:08:24.179834+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179834+0000', 'last_undegraded': '2024-09-17T04:08:24.179834+0000', 'last_fullsized': '2024-09-17T04:08:24.179834+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:PG 41.b is not active+clean 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179965+0000', 'last_change': '2024-09-17T04:08:24.179965+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179965+0000', 'last_undegraded': '2024-09-17T04:08:24.179965+0000', 'last_fullsized': '2024-09-17T04:08:24.179965+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:PG 41.8 is not active+clean 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.190666+0000', 'last_change': '2024-09-17T04:08:24.190666+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.190666+0000', 'last_undegraded': '2024-09-17T04:08:24.190666+0000', 'last_fullsized': '2024-09-17T04:08:24.190666+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:PG 41.6 is not active+clean 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.182099+0000', 'last_change': '2024-09-17T04:08:24.182099+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.182099+0000', 'last_undegraded': '2024-09-17T04:08:24.182099+0000', 'last_fullsized': '2024-09-17T04:08:24.182099+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.591 INFO:tasks.ceph.ceph_manager.ceph:PG 41.5 is not active+clean 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.186648+0000', 'last_change': '2024-09-17T04:08:24.186648+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.186648+0000', 'last_undegraded': '2024-09-17T04:08:24.186648+0000', 'last_fullsized': '2024-09-17T04:08:24.186648+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:PG 41.2 is not active+clean 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.183889+0000', 'last_change': '2024-09-17T04:08:24.183889+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.183889+0000', 'last_undegraded': '2024-09-17T04:08:24.183889+0000', 'last_fullsized': '2024-09-17T04:08:24.183889+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:PG 41.3 is not active+clean 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179560+0000', 'last_change': '2024-09-17T04:08:24.179560+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179560+0000', 'last_undegraded': '2024-09-17T04:08:24.179560+0000', 'last_fullsized': '2024-09-17T04:08:24.179560+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:PG 41.0 is not active+clean 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.181036+0000', 'last_change': '2024-09-17T04:08:24.181036+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.181036+0000', 'last_undegraded': '2024-09-17T04:08:24.181036+0000', 'last_fullsized': '2024-09-17T04:08:24.181036+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:PG 41.1 is not active+clean 2024-09-17T04:08:30.592 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '41.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 270, 'state': 'creating+peering', 'last_fresh': '2024-09-17T04:08:24.179027+0000', 'last_change': '2024-09-17T04:08:24.179027+0000', 'last_active': '2024-09-17T04:08:24.161861+0000', 'last_peered': '2024-09-17T04:08:24.161861+0000', 'last_clean': '2024-09-17T04:08:24.161861+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-17T04:08:24.179027+0000', 'last_undegraded': '2024-09-17T04:08:24.179027+0000', 'last_fullsized': '2024-09-17T04:08:24.179027+0000', 'mapping_epoch': 270, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 270, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'last_clean_scrub_stamp': '2024-09-17T04:08:24.161861+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-17T04:08:30.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-17T04:08:30.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 -- 172.21.15.129:0/2204615690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180b4180 msgr2=0x7f0b180ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 --2- 172.21.15.129:0/2204615690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180b4180 0x7f0b180ce170 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f0b04005fd0 tx=0x7f0b0402a5d0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 -- 172.21.15.129:0/2204615690 shutdown_connections 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 --2- 172.21.15.129:0/2204615690 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180b4180 0x7f0b180ce170 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 --2- 172.21.15.129:0/2204615690 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 -- 172.21.15.129:0/2204615690 >> 172.21.15.129:0/2204615690 conn(0x7f0b1806d410 msgr2=0x7f0b180b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 -- 172.21.15.129:0/2204615690 shutdown_connections 2024-09-17T04:08:30.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.840+0000 7f0b1d600640 1 -- 172.21.15.129:0/2204615690 wait complete. 2024-09-17T04:08:30.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.842+0000 7f0b1d600640 1 Processor -- start 2024-09-17T04:08:30.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.842+0000 7f0b1d600640 1 -- start start 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 0x7f0b1815a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1806f930 con 0x7f0b180ccc80 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1806f7b0 con 0x7f0b180b4180 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b1d600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0b1806fab0 con 0x7f0b1815ce50 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b16000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b16000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57290/0 (socket says 172.21.15.129:57290) 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.844+0000 7f0b16000640 1 -- 172.21.15.129:0/3203743951 learned_addr learned my addr 172.21.15.129:0/3203743951 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 -- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 msgr2=0x7f0b180ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16a00640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 0x7f0b1815a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b17400640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 -- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 msgr2=0x7f0b1815a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 0x7f0b1815a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 -- 172.21.15.129:0/3203743951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b180c8590 con 0x7f0b180ccc80 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16a00640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 0x7f0b1815a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b17400640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b16000640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006010 tx=0x7f0b040038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0403e030 con 0x7f0b180ccc80 2024-09-17T04:08:30.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b04003ca0 con 0x7f0b180ccc80 2024-09-17T04:08:30.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b180c8870 con 0x7f0b180ccc80 2024-09-17T04:08:30.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.845+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b04035040 con 0x7f0b180ccc80 2024-09-17T04:08:30.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.846+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1809cf70 con 0x7f0b180ccc80 2024-09-17T04:08:30.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.847+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b180c82e0 con 0x7f0b180ccc80 2024-09-17T04:08:30.848 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.848+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0b04041030 con 0x7f0b180ccc80 2024-09-17T04:08:30.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.849+0000 7f0b0ea00640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 0x7f0af807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:30.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.849+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f0b040bb380 con 0x7f0b180ccc80 2024-09-17T04:08:30.849 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.850+0000 7f0b16a00640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 0x7f0af807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:30.850 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.850+0000 7f0b16a00640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 0x7f0af807b0d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009aa0 tx=0x7f0b08005db0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:30.853 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:30.853+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0b040837b0 con 0x7f0b180ccc80 2024-09-17T04:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.105+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f0b1806ffd0 con 0x7f0b180ccc80 2024-09-17T04:08:31.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.106+0000 7f0b0ea00640 1 -- 172.21.15.129:0/3203743951 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1438 (secure 0 0 0) 0x7f0b040830b0 con 0x7f0b180ccc80 2024-09-17T04:08:31.106 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:31.106 INFO:teuthology.orchestra.run.smithi129.stdout:{"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1162,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":272,"num_osds":8,"num_up_osds":8,"osd_up_since":1726544964,"num_in_osds":8,"osd_in_since":1726544953,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":51},{"state_name":"creating+peering","count":46}],"num_pgs":97,"num_pools":3,"num_objects":12,"data_bytes":592744,"bytes_used":430546944,"bytes_avail":772663566336,"bytes_total":773094113280,"inactive_pgs_ratio":0.47422680258750916,"write_bytes_sec":1313,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":272,"btime":"2024-09-17T04:08:28:198922+0000","id":20,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":20,"rank":0,"name":"a","status":"up:active","gid":12681}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":33,"modified":"2024-09-17T04:07:50.609188+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-17T04:08:31.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 msgr2=0x7f0af807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 0x7f0af807b0d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f0b08009aa0 tx=0x7f0b08005db0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 msgr2=0x7f0b1809d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006010 tx=0x7f0b040038a0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 shutdown_connections 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0af8078bc0 0x7f0af807b0d0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b1815ce50 0x7f0b180ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b180ccc80 0x7f0b1809d770 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 --2- 172.21.15.129:0/3203743951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b180b4180 0x7f0b1815a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 >> 172.21.15.129:0/3203743951 conn(0x7f0b1806d410 msgr2=0x7f0b180b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 shutdown_connections 2024-09-17T04:08:31.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.108+0000 7f0b1d600640 1 -- 172.21.15.129:0/3203743951 wait complete. 2024-09-17T04:08:31.130 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-17T04:08:31.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 -- 172.21.15.129:0/3182365898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b740b3b90 msgr2=0x7f6b740cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 --2- 172.21.15.129:0/3182365898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b740b3b90 0x7f6b740cbbf0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f6b74072e00 tx=0x7f6b6002a5f0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 -- 172.21.15.129:0/3182365898 shutdown_connections 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 --2- 172.21.15.129:0/3182365898 >> v2:172.21.15.181:3300/0 conn(0x7f6b740b2950 0x7f6b7406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 --2- 172.21.15.129:0/3182365898 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b740b3b90 0x7f6b740cbbf0 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 -- 172.21.15.129:0/3182365898 >> 172.21.15.129:0/3182365898 conn(0x7f6b7406d410 msgr2=0x7f6b740b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 -- 172.21.15.129:0/3182365898 shutdown_connections 2024-09-17T04:08:31.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.351+0000 7f6b7a000640 1 -- 172.21.15.129:0/3182365898 wait complete. 2024-09-17T04:08:31.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.354+0000 7f6b7a000640 1 Processor -- start 2024-09-17T04:08:31.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.354+0000 7f6b7a000640 1 -- start start 2024-09-17T04:08:31.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b740b2950 0x7f6b7415e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:31.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b740ce170 0x7f6b7415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:31.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:31.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7406f410 con 0x7f6b741f3090 2024-09-17T04:08:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6b740708e0 con 0x7f6b740b2950 2024-09-17T04:08:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b7a000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7406f590 con 0x7f6b740ce170 2024-09-17T04:08:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b73e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:57314/0 (socket says 172.21.15.129:57314) 2024-09-17T04:08:31.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.355+0000 7f6b73e00640 1 -- 172.21.15.129:0/1977077692 learned_addr learned my addr 172.21.15.129:0/1977077692 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:31.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73400640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b740b2950 0x7f6b7415e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:31.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b72a00640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b740ce170 0x7f6b7415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:31.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 -- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b740ce170 msgr2=0x7f6b7415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b740ce170 0x7f6b7415e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 -- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b740b2950 msgr2=0x7f6b7415e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b740b2950 0x7f6b7415e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 -- 172.21.15.129:0/1977077692 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b74098b20 con 0x7f6b741f3090 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b73e00640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00e9c0 tx=0x7f6b6c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c004d20 con 0x7f6b741f3090 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6b6c00cac0 con 0x7f6b741f3090 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c005020 con 0x7f6b741f3090 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.356+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b741574a0 con 0x7f6b741f3090 2024-09-17T04:08:31.358 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.357+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b74157a60 con 0x7f6b741f3090 2024-09-17T04:08:31.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.358+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7406f990 con 0x7f6b741f3090 2024-09-17T04:08:31.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.359+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6b6c0051c0 con 0x7f6b741f3090 2024-09-17T04:08:31.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.360+0000 7f6b6b400640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 0x7f6b5407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:31.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.360+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f6b6c09b580 con 0x7f6b741f3090 2024-09-17T04:08:31.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.360+0000 7f6b73400640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 0x7f6b5407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:31.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.361+0000 7f6b73400640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 0x7f6b5407b0d0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6b640015e0 tx=0x7f6b64009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:31.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.364+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6b6c016030 con 0x7f6b741f3090 2024-09-17T04:08:31.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.542+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b7406a140 con 0x7f6b54078bc0 2024-09-17T04:08:31.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.548+0000 7f6b6b400640 1 -- 172.21.15.129:0/1977077692 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252477 (secure 0 0 0) 0x7f6b7406a140 con 0x7f6b54078bc0 2024-09-17T04:08:31.551 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:31.552 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:08:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 msgr2=0x7f6b5407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 0x7f6b5407b0d0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6b640015e0 tx=0x7f6b64009040 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 msgr2=0x7f6b74157f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c00e9c0 tx=0x7f6b6c00ee80 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 shutdown_connections 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6b54078bc0 0x7f6b5407b0d0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6b741f3090 0x7f6b74157f80 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6b740ce170 0x7f6b7415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 --2- 172.21.15.129:0/1977077692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6b740b2950 0x7f6b7415e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 >> 172.21.15.129:0/1977077692 conn(0x7f6b7406d410 msgr2=0x7f6b7415b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 shutdown_connections 2024-09-17T04:08:31.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:31.552+0000 7f6b7a000640 1 -- 172.21.15.129:0/1977077692 wait complete. 2024-09-17T04:08:31.575 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":839,"stamp":"2024-09-17T04:08:30.621058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420688,"kb_used_data":28168,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754554032,"statfs":{"total":773094113280,"available":772663328768,"internally_reserved":0,"allocated":28844032,"data_stored":13452004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.443364"},"pg_stats":[{"pgid":"41.3f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326478+0000","last_change":"2024-09-17T04:08:25.189216+0000","last_active":"2024-09-17T04:08:26.326478+0000","last_peered":"2024-09-17T04:08:26.326478+0000","last_clean":"2024-09-17T04:08:26.326478+0000","last_became_active":"2024-09-17T04:08:25.188921+0000","last_became_peered":"2024-09-17T04:08:25.188921+0000","last_unstale":"2024-09-17T04:08:26.326478+0000","last_undegraded":"2024-09-17T04:08:26.326478+0000","last_fullsized":"2024-09-17T04:08:26.326478+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:41.624638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181961+0000","last_change":"2024-09-17T04:08:25.200078+0000","last_active":"2024-09-17T04:08:26.181961+0000","last_peered":"2024-09-17T04:08:26.181961+0000","last_clean":"2024-09-17T04:08:26.181961+0000","last_became_active":"2024-09-17T04:08:25.199691+0000","last_became_peered":"2024-09-17T04:08:25.199691+0000","last_unstale":"2024-09-17T04:08:26.181961+0000","last_undegraded":"2024-09-17T04:08:26.181961+0000","last_fullsized":"2024-09-17T04:08:26.181961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:26.774796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202842+0000","last_change":"2024-09-17T04:08:25.207544+0000","last_active":"2024-09-17T04:08:26.202842+0000","last_peered":"2024-09-17T04:08:26.202842+0000","last_clean":"2024-09-17T04:08:26.202842+0000","last_became_active":"2024-09-17T04:08:25.207230+0000","last_became_peered":"2024-09-17T04:08:25.207230+0000","last_unstale":"2024-09-17T04:08:26.202842+0000","last_undegraded":"2024-09-17T04:08:26.202842+0000","last_fullsized":"2024-09-17T04:08:26.202842+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:36.890078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179538+0000","last_change":"2024-09-17T04:08:25.205728+0000","last_active":"2024-09-17T04:08:26.179538+0000","last_peered":"2024-09-17T04:08:26.179538+0000","last_clean":"2024-09-17T04:08:26.179538+0000","last_became_active":"2024-09-17T04:08:25.204559+0000","last_became_peered":"2024-09-17T04:08:25.204559+0000","last_unstale":"2024-09-17T04:08:26.179538+0000","last_undegraded":"2024-09-17T04:08:26.179538+0000","last_fullsized":"2024-09-17T04:08:26.179538+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:30:08.422127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.3b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202640+0000","last_change":"2024-09-17T04:08:25.203198+0000","last_active":"2024-09-17T04:08:26.202640+0000","last_peered":"2024-09-17T04:08:26.202640+0000","last_clean":"2024-09-17T04:08:26.202640+0000","last_became_active":"2024-09-17T04:08:25.202329+0000","last_became_peered":"2024-09-17T04:08:25.202329+0000","last_unstale":"2024-09-17T04:08:26.202640+0000","last_undegraded":"2024-09-17T04:08:26.202640+0000","last_fullsized":"2024-09-17T04:08:26.202640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:57.774351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179130+0000","last_change":"2024-09-17T04:08:25.205488+0000","last_active":"2024-09-17T04:08:26.179130+0000","last_peered":"2024-09-17T04:08:26.179130+0000","last_clean":"2024-09-17T04:08:26.179130+0000","last_became_active":"2024-09-17T04:08:25.204097+0000","last_became_peered":"2024-09-17T04:08:25.204097+0000","last_unstale":"2024-09-17T04:08:26.179130+0000","last_undegraded":"2024-09-17T04:08:26.179130+0000","last_fullsized":"2024-09-17T04:08:26.179130+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:40.637633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.39","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202151+0000","last_change":"2024-09-17T04:08:25.204022+0000","last_active":"2024-09-17T04:08:26.202151+0000","last_peered":"2024-09-17T04:08:26.202151+0000","last_clean":"2024-09-17T04:08:26.202151+0000","last_became_active":"2024-09-17T04:08:25.203590+0000","last_became_peered":"2024-09-17T04:08:25.203590+0000","last_unstale":"2024-09-17T04:08:26.202151+0000","last_undegraded":"2024-09-17T04:08:26.202151+0000","last_fullsized":"2024-09-17T04:08:26.202151+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:02.649195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.38","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326289+0000","last_change":"2024-09-17T04:08:25.207974+0000","last_active":"2024-09-17T04:08:26.326289+0000","last_peered":"2024-09-17T04:08:26.326289+0000","last_clean":"2024-09-17T04:08:26.326289+0000","last_became_active":"2024-09-17T04:08:25.207454+0000","last_became_peered":"2024-09-17T04:08:25.207454+0000","last_unstale":"2024-09-17T04:08:26.326289+0000","last_undegraded":"2024-09-17T04:08:26.326289+0000","last_fullsized":"2024-09-17T04:08:26.326289+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:18.281022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.37","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327091+0000","last_change":"2024-09-17T04:08:25.191519+0000","last_active":"2024-09-17T04:08:26.327091+0000","last_peered":"2024-09-17T04:08:26.327091+0000","last_clean":"2024-09-17T04:08:26.327091+0000","last_became_active":"2024-09-17T04:08:25.191096+0000","last_became_peered":"2024-09-17T04:08:25.191096+0000","last_unstale":"2024-09-17T04:08:26.327091+0000","last_undegraded":"2024-09-17T04:08:26.327091+0000","last_fullsized":"2024-09-17T04:08:26.327091+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:28.667318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.36","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327027+0000","last_change":"2024-09-17T04:08:25.209134+0000","last_active":"2024-09-17T04:08:26.327027+0000","last_peered":"2024-09-17T04:08:26.327027+0000","last_clean":"2024-09-17T04:08:26.327027+0000","last_became_active":"2024-09-17T04:08:25.208392+0000","last_became_peered":"2024-09-17T04:08:25.208392+0000","last_unstale":"2024-09-17T04:08:26.327027+0000","last_undegraded":"2024-09-17T04:08:26.327027+0000","last_fullsized":"2024-09-17T04:08:26.327027+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:51.030185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.35","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181172+0000","last_change":"2024-09-17T04:08:25.206697+0000","last_active":"2024-09-17T04:08:26.181172+0000","last_peered":"2024-09-17T04:08:26.181172+0000","last_clean":"2024-09-17T04:08:26.181172+0000","last_became_active":"2024-09-17T04:08:25.206307+0000","last_became_peered":"2024-09-17T04:08:25.206307+0000","last_unstale":"2024-09-17T04:08:26.181172+0000","last_undegraded":"2024-09-17T04:08:26.181172+0000","last_fullsized":"2024-09-17T04:08:26.181172+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:14.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.34","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326226+0000","last_change":"2024-09-17T04:08:25.196813+0000","last_active":"2024-09-17T04:08:26.326226+0000","last_peered":"2024-09-17T04:08:26.326226+0000","last_clean":"2024-09-17T04:08:26.326226+0000","last_became_active":"2024-09-17T04:08:25.196404+0000","last_became_peered":"2024-09-17T04:08:25.196404+0000","last_unstale":"2024-09-17T04:08:26.326226+0000","last_undegraded":"2024-09-17T04:08:26.326226+0000","last_fullsized":"2024-09-17T04:08:26.326226+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:24.380529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.33","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180588+0000","last_change":"2024-09-17T04:08:25.188090+0000","last_active":"2024-09-17T04:08:26.180588+0000","last_peered":"2024-09-17T04:08:26.180588+0000","last_clean":"2024-09-17T04:08:26.180588+0000","last_became_active":"2024-09-17T04:08:25.187682+0000","last_became_peered":"2024-09-17T04:08:25.187682+0000","last_unstale":"2024-09-17T04:08:26.180588+0000","last_undegraded":"2024-09-17T04:08:26.180588+0000","last_fullsized":"2024-09-17T04:08:26.180588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:37.175115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.32","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202224+0000","last_change":"2024-09-17T04:08:25.203400+0000","last_active":"2024-09-17T04:08:26.202224+0000","last_peered":"2024-09-17T04:08:26.202224+0000","last_clean":"2024-09-17T04:08:26.202224+0000","last_became_active":"2024-09-17T04:08:25.202662+0000","last_became_peered":"2024-09-17T04:08:25.202662+0000","last_unstale":"2024-09-17T04:08:26.202224+0000","last_undegraded":"2024-09-17T04:08:26.202224+0000","last_fullsized":"2024-09-17T04:08:26.202224+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:29:20.930631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.31","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202160+0000","last_change":"2024-09-17T04:08:25.196112+0000","last_active":"2024-09-17T04:08:26.202160+0000","last_peered":"2024-09-17T04:08:26.202160+0000","last_clean":"2024-09-17T04:08:26.202160+0000","last_became_active":"2024-09-17T04:08:25.195717+0000","last_became_peered":"2024-09-17T04:08:25.195717+0000","last_unstale":"2024-09-17T04:08:26.202160+0000","last_undegraded":"2024-09-17T04:08:26.202160+0000","last_fullsized":"2024-09-17T04:08:26.202160+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:17.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.30","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180941+0000","last_change":"2024-09-17T04:08:25.190931+0000","last_active":"2024-09-17T04:08:26.180941+0000","last_peered":"2024-09-17T04:08:26.180941+0000","last_clean":"2024-09-17T04:08:26.180941+0000","last_became_active":"2024-09-17T04:08:25.188005+0000","last_became_peered":"2024-09-17T04:08:25.188005+0000","last_unstale":"2024-09-17T04:08:26.180941+0000","last_undegraded":"2024-09-17T04:08:26.180941+0000","last_fullsized":"2024-09-17T04:08:26.180941+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.219776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182020+0000","last_change":"2024-09-17T04:08:25.210259+0000","last_active":"2024-09-17T04:08:26.182020+0000","last_peered":"2024-09-17T04:08:26.182020+0000","last_clean":"2024-09-17T04:08:26.182020+0000","last_became_active":"2024-09-17T04:08:25.206858+0000","last_became_peered":"2024-09-17T04:08:25.206858+0000","last_unstale":"2024-09-17T04:08:26.182020+0000","last_undegraded":"2024-09-17T04:08:26.182020+0000","last_fullsized":"2024-09-17T04:08:26.182020+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:43.017724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326993+0000","last_change":"2024-09-17T04:08:25.209482+0000","last_active":"2024-09-17T04:08:26.326993+0000","last_peered":"2024-09-17T04:08:26.326993+0000","last_clean":"2024-09-17T04:08:26.326993+0000","last_became_active":"2024-09-17T04:08:25.208778+0000","last_became_peered":"2024-09-17T04:08:25.208778+0000","last_unstale":"2024-09-17T04:08:26.326993+0000","last_undegraded":"2024-09-17T04:08:26.326993+0000","last_fullsized":"2024-09-17T04:08:26.326993+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:26.075977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.2d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326090+0000","last_change":"2024-09-17T04:08:25.192767+0000","last_active":"2024-09-17T04:08:26.326090+0000","last_peered":"2024-09-17T04:08:26.326090+0000","last_clean":"2024-09-17T04:08:26.326090+0000","last_became_active":"2024-09-17T04:08:25.192484+0000","last_became_peered":"2024-09-17T04:08:25.192484+0000","last_unstale":"2024-09-17T04:08:26.326090+0000","last_undegraded":"2024-09-17T04:08:26.326090+0000","last_fullsized":"2024-09-17T04:08:26.326090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:08.026774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.2c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182024+0000","last_change":"2024-09-17T04:08:25.202002+0000","last_active":"2024-09-17T04:08:26.182024+0000","last_peered":"2024-09-17T04:08:26.182024+0000","last_clean":"2024-09-17T04:08:26.182024+0000","last_became_active":"2024-09-17T04:08:25.201684+0000","last_became_peered":"2024-09-17T04:08:25.201684+0000","last_unstale":"2024-09-17T04:08:26.182024+0000","last_undegraded":"2024-09-17T04:08:26.182024+0000","last_fullsized":"2024-09-17T04:08:26.182024+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:20.682978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179696+0000","last_change":"2024-09-17T04:08:25.202635+0000","last_active":"2024-09-17T04:08:26.179696+0000","last_peered":"2024-09-17T04:08:26.179696+0000","last_clean":"2024-09-17T04:08:26.179696+0000","last_became_active":"2024-09-17T04:08:25.201833+0000","last_became_peered":"2024-09-17T04:08:25.201833+0000","last_unstale":"2024-09-17T04:08:26.179696+0000","last_undegraded":"2024-09-17T04:08:26.179696+0000","last_fullsized":"2024-09-17T04:08:26.179696+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:13.227923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180516+0000","last_change":"2024-09-17T04:08:25.183839+0000","last_active":"2024-09-17T04:08:26.180516+0000","last_peered":"2024-09-17T04:08:26.180516+0000","last_clean":"2024-09-17T04:08:26.180516+0000","last_became_active":"2024-09-17T04:08:25.183527+0000","last_became_peered":"2024-09-17T04:08:25.183527+0000","last_unstale":"2024-09-17T04:08:26.180516+0000","last_undegraded":"2024-09-17T04:08:26.180516+0000","last_fullsized":"2024-09-17T04:08:26.180516+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:19.124388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.29","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326399+0000","last_change":"2024-09-17T04:08:25.189733+0000","last_active":"2024-09-17T04:08:26.326399+0000","last_peered":"2024-09-17T04:08:26.326399+0000","last_clean":"2024-09-17T04:08:26.326399+0000","last_became_active":"2024-09-17T04:08:25.189352+0000","last_became_peered":"2024-09-17T04:08:25.189352+0000","last_unstale":"2024-09-17T04:08:26.326399+0000","last_undegraded":"2024-09-17T04:08:26.326399+0000","last_fullsized":"2024-09-17T04:08:26.326399+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:25.468071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.27","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180756+0000","last_change":"2024-09-17T04:08:25.203709+0000","last_active":"2024-09-17T04:08:26.180756+0000","last_peered":"2024-09-17T04:08:26.180756+0000","last_clean":"2024-09-17T04:08:26.180756+0000","last_became_active":"2024-09-17T04:08:25.203364+0000","last_became_peered":"2024-09-17T04:08:25.203364+0000","last_unstale":"2024-09-17T04:08:26.180756+0000","last_undegraded":"2024-09-17T04:08:26.180756+0000","last_fullsized":"2024-09-17T04:08:26.180756+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.101081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.26","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326742+0000","last_change":"2024-09-17T04:08:25.205002+0000","last_active":"2024-09-17T04:08:26.326742+0000","last_peered":"2024-09-17T04:08:26.326742+0000","last_clean":"2024-09-17T04:08:26.326742+0000","last_became_active":"2024-09-17T04:08:25.204427+0000","last_became_peered":"2024-09-17T04:08:25.204427+0000","last_unstale":"2024-09-17T04:08:26.326742+0000","last_undegraded":"2024-09-17T04:08:26.326742+0000","last_fullsized":"2024-09-17T04:08:26.326742+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:37.951456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.25","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326850+0000","last_change":"2024-09-17T04:08:25.201853+0000","last_active":"2024-09-17T04:08:26.326850+0000","last_peered":"2024-09-17T04:08:26.326850+0000","last_clean":"2024-09-17T04:08:26.326850+0000","last_became_active":"2024-09-17T04:08:25.201493+0000","last_became_peered":"2024-09-17T04:08:25.201493+0000","last_unstale":"2024-09-17T04:08:26.326850+0000","last_undegraded":"2024-09-17T04:08:26.326850+0000","last_fullsized":"2024-09-17T04:08:26.326850+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:37.130188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.24","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326183+0000","last_change":"2024-09-17T04:08:25.188324+0000","last_active":"2024-09-17T04:08:26.326183+0000","last_peered":"2024-09-17T04:08:26.326183+0000","last_clean":"2024-09-17T04:08:26.326183+0000","last_became_active":"2024-09-17T04:08:25.188025+0000","last_became_peered":"2024-09-17T04:08:25.188025+0000","last_unstale":"2024-09-17T04:08:26.326183+0000","last_undegraded":"2024-09-17T04:08:26.326183+0000","last_fullsized":"2024-09-17T04:08:26.326183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:41.304344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.23","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181374+0000","last_change":"2024-09-17T04:08:25.193946+0000","last_active":"2024-09-17T04:08:26.181374+0000","last_peered":"2024-09-17T04:08:26.181374+0000","last_clean":"2024-09-17T04:08:26.181374+0000","last_became_active":"2024-09-17T04:08:25.186913+0000","last_became_peered":"2024-09-17T04:08:25.186913+0000","last_unstale":"2024-09-17T04:08:26.181374+0000","last_undegraded":"2024-09-17T04:08:26.181374+0000","last_fullsized":"2024-09-17T04:08:26.181374+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:51.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.22","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326936+0000","last_change":"2024-09-17T04:08:25.194536+0000","last_active":"2024-09-17T04:08:26.326936+0000","last_peered":"2024-09-17T04:08:26.326936+0000","last_clean":"2024-09-17T04:08:26.326936+0000","last_became_active":"2024-09-17T04:08:25.194201+0000","last_became_peered":"2024-09-17T04:08:25.194201+0000","last_unstale":"2024-09-17T04:08:26.326936+0000","last_undegraded":"2024-09-17T04:08:26.326936+0000","last_fullsized":"2024-09-17T04:08:26.326936+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:14.659279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.21","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326187+0000","last_change":"2024-09-17T04:08:25.198539+0000","last_active":"2024-09-17T04:08:26.326187+0000","last_peered":"2024-09-17T04:08:26.326187+0000","last_clean":"2024-09-17T04:08:26.326187+0000","last_became_active":"2024-09-17T04:08:25.198178+0000","last_became_peered":"2024-09-17T04:08:25.198178+0000","last_unstale":"2024-09-17T04:08:26.326187+0000","last_undegraded":"2024-09-17T04:08:26.326187+0000","last_fullsized":"2024-09-17T04:08:26.326187+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:18.437486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.20","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326519+0000","last_change":"2024-09-17T04:08:25.206873+0000","last_active":"2024-09-17T04:08:26.326519+0000","last_peered":"2024-09-17T04:08:26.326519+0000","last_clean":"2024-09-17T04:08:26.326519+0000","last_became_active":"2024-09-17T04:08:25.206313+0000","last_became_peered":"2024-09-17T04:08:25.206313+0000","last_unstale":"2024-09-17T04:08:26.326519+0000","last_undegraded":"2024-09-17T04:08:26.326519+0000","last_fullsized":"2024-09-17T04:08:26.326519+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:05.166134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326610+0000","last_change":"2024-09-17T04:08:25.199592+0000","last_active":"2024-09-17T04:08:26.326610+0000","last_peered":"2024-09-17T04:08:26.326610+0000","last_clean":"2024-09-17T04:08:26.326610+0000","last_became_active":"2024-09-17T04:08:25.199254+0000","last_became_peered":"2024-09-17T04:08:25.199254+0000","last_unstale":"2024-09-17T04:08:26.326610+0000","last_undegraded":"2024-09-17T04:08:26.326610+0000","last_fullsized":"2024-09-17T04:08:26.326610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:22.464215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"271'4","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326348+0000","last_change":"2024-09-17T04:08:24.190135+0000","last_active":"2024-09-17T04:08:26.326348+0000","last_peered":"2024-09-17T04:08:26.326348+0000","last_clean":"2024-09-17T04:08:26.326348+0000","last_became_active":"2024-09-17T04:08:24.189843+0000","last_became_peered":"2024-09-17T04:08:24.189843+0000","last_unstale":"2024-09-17T04:08:26.326348+0000","last_undegraded":"2024-09-17T04:08:26.326348+0000","last_fullsized":"2024-09-17T04:08:26.326348+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:29.916388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327210+0000","last_change":"2024-09-17T04:08:25.184835+0000","last_active":"2024-09-17T04:08:26.327210+0000","last_peered":"2024-09-17T04:08:26.327210+0000","last_clean":"2024-09-17T04:08:26.327210+0000","last_became_active":"2024-09-17T04:08:25.184368+0000","last_became_peered":"2024-09-17T04:08:25.184368+0000","last_unstale":"2024-09-17T04:08:26.327210+0000","last_undegraded":"2024-09-17T04:08:26.327210+0000","last_fullsized":"2024-09-17T04:08:26.327210+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:53.799541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326691+0000","last_change":"2024-09-17T04:08:24.196111+0000","last_active":"2024-09-17T04:08:26.326691+0000","last_peered":"2024-09-17T04:08:26.326691+0000","last_clean":"2024-09-17T04:08:26.326691+0000","last_became_active":"2024-09-17T04:08:24.195595+0000","last_became_peered":"2024-09-17T04:08:24.195595+0000","last_unstale":"2024-09-17T04:08:26.326691+0000","last_undegraded":"2024-09-17T04:08:26.326691+0000","last_fullsized":"2024-09-17T04:08:26.326691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:44.710662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202321+0000","last_change":"2024-09-17T04:08:25.209587+0000","last_active":"2024-09-17T04:08:26.202321+0000","last_peered":"2024-09-17T04:08:26.202321+0000","last_clean":"2024-09-17T04:08:26.202321+0000","last_became_active":"2024-09-17T04:08:25.209204+0000","last_became_peered":"2024-09-17T04:08:25.209204+0000","last_unstale":"2024-09-17T04:08:26.202321+0000","last_undegraded":"2024-09-17T04:08:26.202321+0000","last_fullsized":"2024-09-17T04:08:26.202321+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:11.606138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202378+0000","last_change":"2024-09-17T04:08:24.194368+0000","last_active":"2024-09-17T04:08:26.202378+0000","last_peered":"2024-09-17T04:08:26.202378+0000","last_clean":"2024-09-17T04:08:26.202378+0000","last_became_active":"2024-09-17T04:08:24.193648+0000","last_became_peered":"2024-09-17T04:08:24.193648+0000","last_unstale":"2024-09-17T04:08:26.202378+0000","last_undegraded":"2024-09-17T04:08:26.202378+0000","last_fullsized":"2024-09-17T04:08:26.202378+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:08.140394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181218+0000","last_change":"2024-09-17T04:08:25.206411+0000","last_active":"2024-09-17T04:08:26.181218+0000","last_peered":"2024-09-17T04:08:26.181218+0000","last_clean":"2024-09-17T04:08:26.181218+0000","last_became_active":"2024-09-17T04:08:25.206001+0000","last_became_peered":"2024-09-17T04:08:25.206001+0000","last_unstale":"2024-09-17T04:08:26.181218+0000","last_undegraded":"2024-09-17T04:08:26.181218+0000","last_fullsized":"2024-09-17T04:08:26.181218+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:19.124388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327023+0000","last_change":"2024-09-17T04:08:24.202248+0000","last_active":"2024-09-17T04:08:26.327023+0000","last_peered":"2024-09-17T04:08:26.327023+0000","last_clean":"2024-09-17T04:08:26.327023+0000","last_became_active":"2024-09-17T04:08:24.201709+0000","last_became_peered":"2024-09-17T04:08:24.201709+0000","last_unstale":"2024-09-17T04:08:26.327023+0000","last_undegraded":"2024-09-17T04:08:26.327023+0000","last_fullsized":"2024-09-17T04:08:26.327023+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:43.931505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202515+0000","last_change":"2024-09-17T04:08:25.202178+0000","last_active":"2024-09-17T04:08:26.202515+0000","last_peered":"2024-09-17T04:08:26.202515+0000","last_clean":"2024-09-17T04:08:26.202515+0000","last_became_active":"2024-09-17T04:08:25.199830+0000","last_became_peered":"2024-09-17T04:08:25.199830+0000","last_unstale":"2024-09-17T04:08:26.202515+0000","last_undegraded":"2024-09-17T04:08:26.202515+0000","last_fullsized":"2024-09-17T04:08:26.202515+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:11.236720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202834+0000","last_change":"2024-09-17T04:08:24.200461+0000","last_active":"2024-09-17T04:08:26.202834+0000","last_peered":"2024-09-17T04:08:26.202834+0000","last_clean":"2024-09-17T04:08:26.202834+0000","last_became_active":"2024-09-17T04:08:24.199506+0000","last_became_peered":"2024-09-17T04:08:24.199506+0000","last_unstale":"2024-09-17T04:08:26.202834+0000","last_undegraded":"2024-09-17T04:08:26.202834+0000","last_fullsized":"2024-09-17T04:08:26.202834+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:00.523860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325694+0000","last_change":"2024-09-17T04:08:25.185548+0000","last_active":"2024-09-17T04:08:26.325694+0000","last_peered":"2024-09-17T04:08:26.325694+0000","last_clean":"2024-09-17T04:08:26.325694+0000","last_became_active":"2024-09-17T04:08:25.184461+0000","last_became_peered":"2024-09-17T04:08:25.184461+0000","last_unstale":"2024-09-17T04:08:26.325694+0000","last_undegraded":"2024-09-17T04:08:26.325694+0000","last_fullsized":"2024-09-17T04:08:26.325694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:46.956363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325684+0000","last_change":"2024-09-17T04:08:24.187071+0000","last_active":"2024-09-17T04:08:26.325684+0000","last_peered":"2024-09-17T04:08:26.325684+0000","last_clean":"2024-09-17T04:08:26.325684+0000","last_became_active":"2024-09-17T04:08:24.186743+0000","last_became_peered":"2024-09-17T04:08:24.186743+0000","last_unstale":"2024-09-17T04:08:26.325684+0000","last_undegraded":"2024-09-17T04:08:26.325684+0000","last_fullsized":"2024-09-17T04:08:26.325684+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:15.930301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181632+0000","last_change":"2024-09-17T04:08:25.196132+0000","last_active":"2024-09-17T04:08:26.181632+0000","last_peered":"2024-09-17T04:08:26.181632+0000","last_clean":"2024-09-17T04:08:26.181632+0000","last_became_active":"2024-09-17T04:08:25.194822+0000","last_became_peered":"2024-09-17T04:08:25.194822+0000","last_unstale":"2024-09-17T04:08:26.181632+0000","last_undegraded":"2024-09-17T04:08:26.181632+0000","last_fullsized":"2024-09-17T04:08:26.181632+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:30.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180956+0000","last_change":"2024-09-17T04:08:24.192860+0000","last_active":"2024-09-17T04:08:26.180956+0000","last_peered":"2024-09-17T04:08:26.180956+0000","last_clean":"2024-09-17T04:08:26.180956+0000","last_became_active":"2024-09-17T04:08:24.192505+0000","last_became_peered":"2024-09-17T04:08:24.192505+0000","last_unstale":"2024-09-17T04:08:26.180956+0000","last_undegraded":"2024-09-17T04:08:26.180956+0000","last_fullsized":"2024-09-17T04:08:26.180956+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:19.881424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325700+0000","last_change":"2024-09-17T04:08:25.195702+0000","last_active":"2024-09-17T04:08:26.325700+0000","last_peered":"2024-09-17T04:08:26.325700+0000","last_clean":"2024-09-17T04:08:26.325700+0000","last_became_active":"2024-09-17T04:08:25.195370+0000","last_became_peered":"2024-09-17T04:08:25.195370+0000","last_unstale":"2024-09-17T04:08:26.325700+0000","last_undegraded":"2024-09-17T04:08:26.325700+0000","last_fullsized":"2024-09-17T04:08:26.325700+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:35.074389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.198388+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.197800+0000","last_became_peered":"2024-09-17T04:08:24.197800+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:50.784271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202555+0000","last_change":"2024-09-17T04:08:25.201386+0000","last_active":"2024-09-17T04:08:26.202555+0000","last_peered":"2024-09-17T04:08:26.202555+0000","last_clean":"2024-09-17T04:08:26.202555+0000","last_became_active":"2024-09-17T04:08:25.201032+0000","last_became_peered":"2024-09-17T04:08:25.201032+0000","last_unstale":"2024-09-17T04:08:26.202555+0000","last_undegraded":"2024-09-17T04:08:26.202555+0000","last_fullsized":"2024-09-17T04:08:26.202555+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:56.224682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202729+0000","last_change":"2024-09-17T04:08:24.197553+0000","last_active":"2024-09-17T04:08:26.202729+0000","last_peered":"2024-09-17T04:08:26.202729+0000","last_clean":"2024-09-17T04:08:26.202729+0000","last_became_active":"2024-09-17T04:08:24.197233+0000","last_became_peered":"2024-09-17T04:08:24.197233+0000","last_unstale":"2024-09-17T04:08:26.202729+0000","last_undegraded":"2024-09-17T04:08:26.202729+0000","last_fullsized":"2024-09-17T04:08:26.202729+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:11.062355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202539+0000","last_change":"2024-09-17T04:08:25.206017+0000","last_active":"2024-09-17T04:08:26.202539+0000","last_peered":"2024-09-17T04:08:26.202539+0000","last_clean":"2024-09-17T04:08:26.202539+0000","last_became_active":"2024-09-17T04:08:25.205227+0000","last_became_peered":"2024-09-17T04:08:25.205227+0000","last_unstale":"2024-09-17T04:08:26.202539+0000","last_undegraded":"2024-09-17T04:08:26.202539+0000","last_fullsized":"2024-09-17T04:08:26.202539+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:06.548696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326324+0000","last_change":"2024-09-17T04:08:24.191873+0000","last_active":"2024-09-17T04:08:26.326324+0000","last_peered":"2024-09-17T04:08:26.326324+0000","last_clean":"2024-09-17T04:08:26.326324+0000","last_became_active":"2024-09-17T04:08:24.191318+0000","last_became_peered":"2024-09-17T04:08:24.191318+0000","last_unstale":"2024-09-17T04:08:26.326324+0000","last_undegraded":"2024-09-17T04:08:26.326324+0000","last_fullsized":"2024-09-17T04:08:26.326324+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:17.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181707+0000","last_change":"2024-09-17T04:08:25.205715+0000","last_active":"2024-09-17T04:08:26.181707+0000","last_peered":"2024-09-17T04:08:26.181707+0000","last_clean":"2024-09-17T04:08:26.181707+0000","last_became_active":"2024-09-17T04:08:25.205374+0000","last_became_peered":"2024-09-17T04:08:25.205374+0000","last_unstale":"2024-09-17T04:08:26.181707+0000","last_undegraded":"2024-09-17T04:08:26.181707+0000","last_fullsized":"2024-09-17T04:08:26.181707+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:24.999425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"271'3","reported_seq":18,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326468+0000","last_change":"2024-09-17T04:08:24.190381+0000","last_active":"2024-09-17T04:08:26.326468+0000","last_peered":"2024-09-17T04:08:26.326468+0000","last_clean":"2024-09-17T04:08:26.326468+0000","last_became_active":"2024-09-17T04:08:24.189870+0000","last_became_peered":"2024-09-17T04:08:24.189870+0000","last_unstale":"2024-09-17T04:08:26.326468+0000","last_undegraded":"2024-09-17T04:08:26.326468+0000","last_fullsized":"2024-09-17T04:08:26.326468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:11.493387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326092+0000","last_change":"2024-09-17T04:08:25.202764+0000","last_active":"2024-09-17T04:08:26.326092+0000","last_peered":"2024-09-17T04:08:26.326092+0000","last_clean":"2024-09-17T04:08:26.326092+0000","last_became_active":"2024-09-17T04:08:25.202410+0000","last_became_peered":"2024-09-17T04:08:25.202410+0000","last_unstale":"2024-09-17T04:08:26.326092+0000","last_undegraded":"2024-09-17T04:08:26.326092+0000","last_fullsized":"2024-09-17T04:08:26.326092+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:47.028739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"271'6","reported_seq":21,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325975+0000","last_change":"2024-09-17T04:08:24.200801+0000","last_active":"2024-09-17T04:08:26.325975+0000","last_peered":"2024-09-17T04:08:26.325975+0000","last_clean":"2024-09-17T04:08:26.325975+0000","last_became_active":"2024-09-17T04:08:24.199940+0000","last_became_peered":"2024-09-17T04:08:24.199940+0000","last_unstale":"2024-09-17T04:08:26.325975+0000","last_undegraded":"2024-09-17T04:08:26.325975+0000","last_fullsized":"2024-09-17T04:08:26.325975+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:53.586256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326081+0000","last_change":"2024-09-17T04:08:25.206679+0000","last_active":"2024-09-17T04:08:26.326081+0000","last_peered":"2024-09-17T04:08:26.326081+0000","last_clean":"2024-09-17T04:08:26.326081+0000","last_became_active":"2024-09-17T04:08:25.206365+0000","last_became_peered":"2024-09-17T04:08:25.206365+0000","last_unstale":"2024-09-17T04:08:26.326081+0000","last_undegraded":"2024-09-17T04:08:26.326081+0000","last_fullsized":"2024-09-17T04:08:26.326081+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:30.096668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179808+0000","last_change":"2024-09-17T04:08:24.193138+0000","last_active":"2024-09-17T04:08:26.179808+0000","last_peered":"2024-09-17T04:08:26.179808+0000","last_clean":"2024-09-17T04:08:26.179808+0000","last_became_active":"2024-09-17T04:08:24.192861+0000","last_became_peered":"2024-09-17T04:08:24.192861+0000","last_unstale":"2024-09-17T04:08:26.179808+0000","last_undegraded":"2024-09-17T04:08:26.179808+0000","last_fullsized":"2024-09-17T04:08:26.179808+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:47.740618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182225+0000","last_change":"2024-09-17T04:08:25.194811+0000","last_active":"2024-09-17T04:08:26.182225+0000","last_peered":"2024-09-17T04:08:26.182225+0000","last_clean":"2024-09-17T04:08:26.182225+0000","last_became_active":"2024-09-17T04:08:25.194365+0000","last_became_peered":"2024-09-17T04:08:25.194365+0000","last_unstale":"2024-09-17T04:08:26.182225+0000","last_undegraded":"2024-09-17T04:08:26.182225+0000","last_fullsized":"2024-09-17T04:08:26.182225+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.153511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326101+0000","last_change":"2024-09-17T04:08:24.197968+0000","last_active":"2024-09-17T04:08:26.326101+0000","last_peered":"2024-09-17T04:08:26.326101+0000","last_clean":"2024-09-17T04:08:26.326101+0000","last_became_active":"2024-09-17T04:08:24.197333+0000","last_became_peered":"2024-09-17T04:08:24.197333+0000","last_unstale":"2024-09-17T04:08:26.326101+0000","last_undegraded":"2024-09-17T04:08:26.326101+0000","last_fullsized":"2024-09-17T04:08:26.326101+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:10.224581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325571+0000","last_change":"2024-09-17T04:08:25.203605+0000","last_active":"2024-09-17T04:08:26.325571+0000","last_peered":"2024-09-17T04:08:26.325571+0000","last_clean":"2024-09-17T04:08:26.325571+0000","last_became_active":"2024-09-17T04:08:25.203298+0000","last_became_peered":"2024-09-17T04:08:25.203298+0000","last_unstale":"2024-09-17T04:08:26.325571+0000","last_undegraded":"2024-09-17T04:08:26.325571+0000","last_fullsized":"2024-09-17T04:08:26.325571+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:36.890078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180116+0000","last_change":"2024-09-17T04:08:24.199871+0000","last_active":"2024-09-17T04:08:26.180116+0000","last_peered":"2024-09-17T04:08:26.180116+0000","last_clean":"2024-09-17T04:08:26.180116+0000","last_became_active":"2024-09-17T04:08:24.199516+0000","last_became_peered":"2024-09-17T04:08:24.199516+0000","last_unstale":"2024-09-17T04:08:26.180116+0000","last_undegraded":"2024-09-17T04:08:26.180116+0000","last_fullsized":"2024-09-17T04:08:26.180116+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:25.253596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181427+0000","last_change":"2024-09-17T04:08:25.187736+0000","last_active":"2024-09-17T04:08:26.181427+0000","last_peered":"2024-09-17T04:08:26.181427+0000","last_clean":"2024-09-17T04:08:26.181427+0000","last_became_active":"2024-09-17T04:08:25.186265+0000","last_became_peered":"2024-09-17T04:08:25.186265+0000","last_unstale":"2024-09-17T04:08:26.181427+0000","last_undegraded":"2024-09-17T04:08:26.181427+0000","last_fullsized":"2024-09-17T04:08:26.181427+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:28.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"271'8","reported_seq":23,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181716+0000","last_change":"2024-09-17T04:08:24.202703+0000","last_active":"2024-09-17T04:08:26.181716+0000","last_peered":"2024-09-17T04:08:26.181716+0000","last_clean":"2024-09-17T04:08:26.181716+0000","last_became_active":"2024-09-17T04:08:24.201244+0000","last_became_peered":"2024-09-17T04:08:24.201244+0000","last_unstale":"2024-09-17T04:08:26.181716+0000","last_undegraded":"2024-09-17T04:08:26.181716+0000","last_fullsized":"2024-09-17T04:08:26.181716+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:32.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182006+0000","last_change":"2024-09-17T04:08:25.201692+0000","last_active":"2024-09-17T04:08:26.182006+0000","last_peered":"2024-09-17T04:08:26.182006+0000","last_clean":"2024-09-17T04:08:26.182006+0000","last_became_active":"2024-09-17T04:08:25.196557+0000","last_became_peered":"2024-09-17T04:08:25.196557+0000","last_unstale":"2024-09-17T04:08:26.182006+0000","last_undegraded":"2024-09-17T04:08:26.182006+0000","last_fullsized":"2024-09-17T04:08:26.182006+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:53.716086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326899+0000","last_change":"2024-09-17T04:08:24.198617+0000","last_active":"2024-09-17T04:08:26.326899+0000","last_peered":"2024-09-17T04:08:26.326899+0000","last_clean":"2024-09-17T04:08:26.326899+0000","last_became_active":"2024-09-17T04:08:24.198153+0000","last_became_peered":"2024-09-17T04:08:24.198153+0000","last_unstale":"2024-09-17T04:08:26.326899+0000","last_undegraded":"2024-09-17T04:08:26.326899+0000","last_fullsized":"2024-09-17T04:08:26.326899+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:23.292656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180793+0000","last_change":"2024-09-17T04:08:25.187031+0000","last_active":"2024-09-17T04:08:26.180793+0000","last_peered":"2024-09-17T04:08:26.180793+0000","last_clean":"2024-09-17T04:08:26.180793+0000","last_became_active":"2024-09-17T04:08:25.186734+0000","last_became_peered":"2024-09-17T04:08:25.186734+0000","last_unstale":"2024-09-17T04:08:26.180793+0000","last_undegraded":"2024-09-17T04:08:26.180793+0000","last_fullsized":"2024-09-17T04:08:26.180793+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:30.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202798+0000","last_change":"2024-09-17T04:08:24.195281+0000","last_active":"2024-09-17T04:08:26.202798+0000","last_peered":"2024-09-17T04:08:26.202798+0000","last_clean":"2024-09-17T04:08:26.202798+0000","last_became_active":"2024-09-17T04:08:24.194954+0000","last_became_peered":"2024-09-17T04:08:24.194954+0000","last_unstale":"2024-09-17T04:08:26.202798+0000","last_undegraded":"2024-09-17T04:08:26.202798+0000","last_fullsized":"2024-09-17T04:08:26.202798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:29.921479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326598+0000","last_change":"2024-09-17T04:08:25.189201+0000","last_active":"2024-09-17T04:08:26.326598+0000","last_peered":"2024-09-17T04:08:26.326598+0000","last_clean":"2024-09-17T04:08:26.326598+0000","last_became_active":"2024-09-17T04:08:25.188892+0000","last_became_peered":"2024-09-17T04:08:25.188892+0000","last_unstale":"2024-09-17T04:08:26.326598+0000","last_undegraded":"2024-09-17T04:08:26.326598+0000","last_fullsized":"2024-09-17T04:08:26.326598+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:21.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326585+0000","last_change":"2024-09-17T04:08:24.199722+0000","last_active":"2024-09-17T04:08:26.326585+0000","last_peered":"2024-09-17T04:08:26.326585+0000","last_clean":"2024-09-17T04:08:26.326585+0000","last_became_active":"2024-09-17T04:08:24.199405+0000","last_became_peered":"2024-09-17T04:08:24.199405+0000","last_unstale":"2024-09-17T04:08:26.326585+0000","last_undegraded":"2024-09-17T04:08:26.326585+0000","last_fullsized":"2024-09-17T04:08:26.326585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:34.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326388+0000","last_change":"2024-09-17T04:08:25.187530+0000","last_active":"2024-09-17T04:08:26.326388+0000","last_peered":"2024-09-17T04:08:26.326388+0000","last_clean":"2024-09-17T04:08:26.326388+0000","last_became_active":"2024-09-17T04:08:25.187224+0000","last_became_peered":"2024-09-17T04:08:25.187224+0000","last_unstale":"2024-09-17T04:08:26.326388+0000","last_undegraded":"2024-09-17T04:08:26.326388+0000","last_fullsized":"2024-09-17T04:08:26.326388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:35.040735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182156+0000","last_change":"2024-09-17T04:08:24.199980+0000","last_active":"2024-09-17T04:08:26.182156+0000","last_peered":"2024-09-17T04:08:26.182156+0000","last_clean":"2024-09-17T04:08:26.182156+0000","last_became_active":"2024-09-17T04:08:24.199415+0000","last_became_peered":"2024-09-17T04:08:24.199415+0000","last_unstale":"2024-09-17T04:08:26.182156+0000","last_undegraded":"2024-09-17T04:08:26.182156+0000","last_fullsized":"2024-09-17T04:08:26.182156+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:26.267458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202752+0000","last_change":"2024-09-17T04:08:25.198832+0000","last_active":"2024-09-17T04:08:26.202752+0000","last_peered":"2024-09-17T04:08:26.202752+0000","last_clean":"2024-09-17T04:08:26.202752+0000","last_became_active":"2024-09-17T04:08:25.198415+0000","last_became_peered":"2024-09-17T04:08:25.198415+0000","last_unstale":"2024-09-17T04:08:26.202752+0000","last_undegraded":"2024-09-17T04:08:26.202752+0000","last_fullsized":"2024-09-17T04:08:26.202752+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:58.479145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180563+0000","last_change":"2024-09-17T04:08:24.198577+0000","last_active":"2024-09-17T04:08:26.180563+0000","last_peered":"2024-09-17T04:08:26.180563+0000","last_clean":"2024-09-17T04:08:26.180563+0000","last_became_active":"2024-09-17T04:08:24.198252+0000","last_became_peered":"2024-09-17T04:08:24.198252+0000","last_unstale":"2024-09-17T04:08:26.180563+0000","last_undegraded":"2024-09-17T04:08:26.180563+0000","last_fullsized":"2024-09-17T04:08:26.180563+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:00.416872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179422+0000","last_change":"2024-09-17T04:08:25.194558+0000","last_active":"2024-09-17T04:08:26.179422+0000","last_peered":"2024-09-17T04:08:26.179422+0000","last_clean":"2024-09-17T04:08:26.179422+0000","last_became_active":"2024-09-17T04:08:25.193536+0000","last_became_peered":"2024-09-17T04:08:25.193536+0000","last_unstale":"2024-09-17T04:08:26.179422+0000","last_undegraded":"2024-09-17T04:08:26.179422+0000","last_fullsized":"2024-09-17T04:08:26.179422+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:05.402751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179060+0000","last_change":"2024-09-17T04:08:24.187161+0000","last_active":"2024-09-17T04:08:26.179060+0000","last_peered":"2024-09-17T04:08:26.179060+0000","last_clean":"2024-09-17T04:08:26.179060+0000","last_became_active":"2024-09-17T04:08:24.186572+0000","last_became_peered":"2024-09-17T04:08:24.186572+0000","last_unstale":"2024-09-17T04:08:26.179060+0000","last_undegraded":"2024-09-17T04:08:26.179060+0000","last_fullsized":"2024-09-17T04:08:26.179060+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:00.713638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181266+0000","last_change":"2024-09-17T04:08:25.195261+0000","last_active":"2024-09-17T04:08:26.181266+0000","last_peered":"2024-09-17T04:08:26.181266+0000","last_clean":"2024-09-17T04:08:26.181266+0000","last_became_active":"2024-09-17T04:08:25.194414+0000","last_became_peered":"2024-09-17T04:08:25.194414+0000","last_unstale":"2024-09-17T04:08:26.181266+0000","last_undegraded":"2024-09-17T04:08:26.181266+0000","last_fullsized":"2024-09-17T04:08:26.181266+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:37.175115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202397+0000","last_change":"2024-09-17T04:08:24.196657+0000","last_active":"2024-09-17T04:08:26.202397+0000","last_peered":"2024-09-17T04:08:26.202397+0000","last_clean":"2024-09-17T04:08:26.202397+0000","last_became_active":"2024-09-17T04:08:24.195146+0000","last_became_peered":"2024-09-17T04:08:24.195146+0000","last_unstale":"2024-09-17T04:08:26.202397+0000","last_undegraded":"2024-09-17T04:08:26.202397+0000","last_fullsized":"2024-09-17T04:08:26.202397+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:53.612476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325728+0000","last_change":"2024-09-17T04:08:25.184757+0000","last_active":"2024-09-17T04:08:26.325728+0000","last_peered":"2024-09-17T04:08:26.325728+0000","last_clean":"2024-09-17T04:08:26.325728+0000","last_became_active":"2024-09-17T04:08:25.184177+0000","last_became_peered":"2024-09-17T04:08:25.184177+0000","last_unstale":"2024-09-17T04:08:26.325728+0000","last_undegraded":"2024-09-17T04:08:26.325728+0000","last_fullsized":"2024-09-17T04:08:26.325728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:35.563823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.184366+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.182740+0000","last_became_peered":"2024-09-17T04:08:24.182740+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:25.140619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202699+0000","last_change":"2024-09-17T04:08:25.215438+0000","last_active":"2024-09-17T04:08:26.202699+0000","last_peered":"2024-09-17T04:08:26.202699+0000","last_clean":"2024-09-17T04:08:26.202699+0000","last_became_active":"2024-09-17T04:08:25.214917+0000","last_became_peered":"2024-09-17T04:08:25.214917+0000","last_unstale":"2024-09-17T04:08:26.202699+0000","last_undegraded":"2024-09-17T04:08:26.202699+0000","last_fullsized":"2024-09-17T04:08:26.202699+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:37.666925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179319+0000","last_change":"2024-09-17T04:08:24.196057+0000","last_active":"2024-09-17T04:08:26.179319+0000","last_peered":"2024-09-17T04:08:26.179319+0000","last_clean":"2024-09-17T04:08:26.179319+0000","last_became_active":"2024-09-17T04:08:24.195822+0000","last_became_peered":"2024-09-17T04:08:24.195822+0000","last_unstale":"2024-09-17T04:08:26.179319+0000","last_undegraded":"2024-09-17T04:08:26.179319+0000","last_fullsized":"2024-09-17T04:08:26.179319+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:41.963796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326477+0000","last_change":"2024-09-17T04:08:25.207446+0000","last_active":"2024-09-17T04:08:26.326477+0000","last_peered":"2024-09-17T04:08:26.326477+0000","last_clean":"2024-09-17T04:08:26.326477+0000","last_became_active":"2024-09-17T04:08:25.207098+0000","last_became_peered":"2024-09-17T04:08:25.207098+0000","last_unstale":"2024-09-17T04:08:26.326477+0000","last_undegraded":"2024-09-17T04:08:26.326477+0000","last_fullsized":"2024-09-17T04:08:26.326477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:13.056929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"272'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202967+0000","last_change":"2024-09-17T04:08:24.202471+0000","last_active":"2024-09-17T04:08:26.202967+0000","last_peered":"2024-09-17T04:08:26.202967+0000","last_clean":"2024-09-17T04:08:26.202967+0000","last_became_active":"2024-09-17T04:08:24.202123+0000","last_became_peered":"2024-09-17T04:08:24.202123+0000","last_unstale":"2024-09-17T04:08:26.202967+0000","last_undegraded":"2024-09-17T04:08:26.202967+0000","last_fullsized":"2024-09-17T04:08:26.202967+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:56:31.031197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326446+0000","last_change":"2024-09-17T04:08:25.196030+0000","last_active":"2024-09-17T04:08:26.326446+0000","last_peered":"2024-09-17T04:08:26.326446+0000","last_clean":"2024-09-17T04:08:26.326446+0000","last_became_active":"2024-09-17T04:08:25.195608+0000","last_became_peered":"2024-09-17T04:08:25.195608+0000","last_unstale":"2024-09-17T04:08:26.326446+0000","last_undegraded":"2024-09-17T04:08:26.326446+0000","last_fullsized":"2024-09-17T04:08:26.326446+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:59.770015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326338+0000","last_change":"2024-09-17T04:08:24.199653+0000","last_active":"2024-09-17T04:08:26.326338+0000","last_peered":"2024-09-17T04:08:26.326338+0000","last_clean":"2024-09-17T04:08:26.326338+0000","last_became_active":"2024-09-17T04:08:24.197097+0000","last_became_peered":"2024-09-17T04:08:24.197097+0000","last_unstale":"2024-09-17T04:08:26.326338+0000","last_undegraded":"2024-09-17T04:08:26.326338+0000","last_fullsized":"2024-09-17T04:08:26.326338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:01.054583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180723+0000","last_change":"2024-09-17T04:08:25.190328+0000","last_active":"2024-09-17T04:08:26.180723+0000","last_peered":"2024-09-17T04:08:26.180723+0000","last_clean":"2024-09-17T04:08:26.180723+0000","last_became_active":"2024-09-17T04:08:25.189973+0000","last_became_peered":"2024-09-17T04:08:25.189973+0000","last_unstale":"2024-09-17T04:08:26.180723+0000","last_undegraded":"2024-09-17T04:08:26.180723+0000","last_fullsized":"2024-09-17T04:08:26.180723+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:51.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326117+0000","last_change":"2024-09-17T04:08:24.200442+0000","last_active":"2024-09-17T04:08:26.326117+0000","last_peered":"2024-09-17T04:08:26.326117+0000","last_clean":"2024-09-17T04:08:26.326117+0000","last_became_active":"2024-09-17T04:08:24.199560+0000","last_became_peered":"2024-09-17T04:08:24.199560+0000","last_unstale":"2024-09-17T04:08:26.326117+0000","last_undegraded":"2024-09-17T04:08:26.326117+0000","last_fullsized":"2024-09-17T04:08:26.326117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:07.964242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326588+0000","last_change":"2024-09-17T04:08:25.203847+0000","last_active":"2024-09-17T04:08:26.326588+0000","last_peered":"2024-09-17T04:08:26.326588+0000","last_clean":"2024-09-17T04:08:26.326588+0000","last_became_active":"2024-09-17T04:08:25.203502+0000","last_became_peered":"2024-09-17T04:08:25.203502+0000","last_unstale":"2024-09-17T04:08:26.326588+0000","last_undegraded":"2024-09-17T04:08:26.326588+0000","last_fullsized":"2024-09-17T04:08:26.326588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:55:37.324715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325805+0000","last_change":"2024-09-17T04:08:24.187060+0000","last_active":"2024-09-17T04:08:26.325805+0000","last_peered":"2024-09-17T04:08:26.325805+0000","last_clean":"2024-09-17T04:08:26.325805+0000","last_became_active":"2024-09-17T04:08:24.186004+0000","last_became_peered":"2024-09-17T04:08:24.186004+0000","last_unstale":"2024-09-17T04:08:26.325805+0000","last_undegraded":"2024-09-17T04:08:26.325805+0000","last_fullsized":"2024-09-17T04:08:26.325805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:11.972381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179649+0000","last_change":"2024-09-17T04:08:25.207493+0000","last_active":"2024-09-17T04:08:26.179649+0000","last_peered":"2024-09-17T04:08:26.179649+0000","last_clean":"2024-09-17T04:08:26.179649+0000","last_became_active":"2024-09-17T04:08:25.205360+0000","last_became_peered":"2024-09-17T04:08:25.205360+0000","last_unstale":"2024-09-17T04:08:26.179649+0000","last_undegraded":"2024-09-17T04:08:26.179649+0000","last_fullsized":"2024-09-17T04:08:26.179649+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:22.837177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179166+0000","last_change":"2024-09-17T04:08:24.186853+0000","last_active":"2024-09-17T04:08:26.179166+0000","last_peered":"2024-09-17T04:08:26.179166+0000","last_clean":"2024-09-17T04:08:26.179166+0000","last_became_active":"2024-09-17T04:08:24.186204+0000","last_became_peered":"2024-09-17T04:08:24.186204+0000","last_unstale":"2024-09-17T04:08:26.179166+0000","last_undegraded":"2024-09-17T04:08:26.179166+0000","last_fullsized":"2024-09-17T04:08:26.179166+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:21.078797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202228+0000","last_change":"2024-09-17T04:08:25.202892+0000","last_active":"2024-09-17T04:08:26.202228+0000","last_peered":"2024-09-17T04:08:26.202228+0000","last_clean":"2024-09-17T04:08:26.202228+0000","last_became_active":"2024-09-17T04:08:25.201830+0000","last_became_peered":"2024-09-17T04:08:25.201830+0000","last_unstale":"2024-09-17T04:08:26.202228+0000","last_undegraded":"2024-09-17T04:08:26.202228+0000","last_fullsized":"2024-09-17T04:08:26.202228+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:23.693505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326364+0000","last_change":"2024-09-17T04:08:24.194575+0000","last_active":"2024-09-17T04:08:26.326364+0000","last_peered":"2024-09-17T04:08:26.326364+0000","last_clean":"2024-09-17T04:08:26.326364+0000","last_became_active":"2024-09-17T04:08:24.194250+0000","last_became_peered":"2024-09-17T04:08:24.194250+0000","last_unstale":"2024-09-17T04:08:26.326364+0000","last_undegraded":"2024-09-17T04:08:26.326364+0000","last_fullsized":"2024-09-17T04:08:26.326364+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:10.593999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179177+0000","last_change":"2024-09-17T04:08:25.194401+0000","last_active":"2024-09-17T04:08:26.179177+0000","last_peered":"2024-09-17T04:08:26.179177+0000","last_clean":"2024-09-17T04:08:26.179177+0000","last_became_active":"2024-09-17T04:08:25.194058+0000","last_became_peered":"2024-09-17T04:08:25.194058+0000","last_unstale":"2024-09-17T04:08:26.179177+0000","last_undegraded":"2024-09-17T04:08:26.179177+0000","last_fullsized":"2024-09-17T04:08:26.179177+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:07.415333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326794+0000","last_change":"2024-09-17T04:08:24.187831+0000","last_active":"2024-09-17T04:08:26.326794+0000","last_peered":"2024-09-17T04:08:26.326794+0000","last_clean":"2024-09-17T04:08:26.326794+0000","last_became_active":"2024-09-17T04:08:24.186781+0000","last_became_peered":"2024-09-17T04:08:24.186781+0000","last_unstale":"2024-09-17T04:08:26.326794+0000","last_undegraded":"2024-09-17T04:08:26.326794+0000","last_fullsized":"2024-09-17T04:08:26.326794+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:48.191171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.28","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326126+0000","last_change":"2024-09-17T04:08:25.204595+0000","last_active":"2024-09-17T04:08:26.326126+0000","last_peered":"2024-09-17T04:08:26.326126+0000","last_clean":"2024-09-17T04:08:26.326126+0000","last_became_active":"2024-09-17T04:08:25.204271+0000","last_became_peered":"2024-09-17T04:08:25.204271+0000","last_unstale":"2024-09-17T04:08:26.326126+0000","last_undegraded":"2024-09-17T04:08:26.326126+0000","last_fullsized":"2024-09-17T04:08:26.326126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:34.822087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":575,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326753+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:08:26.326753+0000","last_peered":"2024-09-17T04:08:26.326753+0000","last_clean":"2024-09-17T04:08:26.326753+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:08:26.326753+0000","last_undegraded":"2024-09-17T04:08:26.326753+0000","last_fullsized":"2024-09-17T04:08:26.326753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":41,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509671,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53196,"kb_used_data":3952,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318644,"statfs":{"total":96636764160,"available":96582291456,"internally_reserved":0,"allocated":4046848,"data_stored":2123764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509674,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34564,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337276,"statfs":{"total":96636764160,"available":96601370624,"internally_reserved":0,"allocated":3452928,"data_stored":1533340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509673,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":3448832,"data_stored":1533306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509671,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56776,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315064,"statfs":{"total":96636764160,"available":96578625536,"internally_reserved":0,"allocated":3452928,"data_stored":1533340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509673,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":3448832,"data_stored":1533306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509673,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3457024,"data_stored":1533910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542375,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56788,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315052,"statfs":{"total":96636764160,"available":96578613248,"internally_reserved":0,"allocated":3469312,"data_stored":1535010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542376,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4067328,"data_stored":2126028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:08:34.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 -- 172.21.15.129:0/275325223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4e4004dd0 msgr2=0x7fc4e40052a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 --2- 172.21.15.129:0/275325223 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc4e4004dd0 0x7fc4e40052a0 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7fc5000c72f0 tx=0x7fc4e4005830 comp rx=0 tx=0).stop 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc504c00640 1 -- 172.21.15.129:0/275325223 reap_dead start 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 -- 172.21.15.129:0/275325223 shutdown_connections 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 -- 172.21.15.129:0/275325223 >> 172.21.15.129:0/275325223 conn(0x7fc50006d410 msgr2=0x7fc5000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 -- 172.21.15.129:0/275325223 shutdown_connections 2024-09-17T04:08:34.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.812+0000 7fc506c00640 1 -- 172.21.15.129:0/275325223 wait complete. 2024-09-17T04:08:34.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.815+0000 7fc506c00640 1 Processor -- start 2024-09-17T04:08:34.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.816+0000 7fc506c00640 1 -- start start 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 0x7fc5001e9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 0x7fc5001e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fc500160810 con 0x7fc5001e6290 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fc5000cd430 con 0x7fc50015ce20 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc506c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fc500160990 con 0x7fc4e4004dd0 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc4ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.817+0000 7fc4ff400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53382/0 (socket says 172.21.15.129:53382) 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 -- 172.21.15.129:0/2218106813 learned_addr learned my addr 172.21.15.129:0/2218106813 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc504c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 0x7fc5001e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ffe00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 0x7fc5001e9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 -- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 msgr2=0x7fc5001e9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 0x7fc5001e9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 -- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 msgr2=0x7fc5001e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 0x7fc5001e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 -- 172.21.15.129:0/2218106813 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5000c65b0 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ffe00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 0x7fc5001e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc504c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 0x7fc5001e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4ff400640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0118b0 tx=0x7fc4ec011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec00f9a0 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc4ec00fb40 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50015ec60 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.818+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec005020 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.819+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50015f220 con 0x7fc5001e6290 2024-09-17T04:08:34.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.820+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc500160ef0 con 0x7fc5001e6290 2024-09-17T04:08:34.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.821+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fc4ec0051c0 con 0x7fc5001e6290 2024-09-17T04:08:34.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.822+0000 7fc4f7e00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 0x7fc4e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:34.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.822+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fc4ec09ba50 con 0x7fc5001e6290 2024-09-17T04:08:34.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.822+0000 7fc4ffe00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 0x7fc4e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:34.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.823+0000 7fc4ffe00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 0x7fc4e007b0a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000c00 tx=0x7fc4f0006b10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:34.826 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:34.826+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fc5000c65b0 con 0x7fc5001e6290 2024-09-17T04:08:35.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.006+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc5001ea2c0 con 0x7fc4e0078b90 2024-09-17T04:08:35.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.010+0000 7fc4f7e00640 1 -- 172.21.15.129:0/2218106813 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252401 (secure 0 0 0) 0x7fc5001ea2c0 con 0x7fc4e0078b90 2024-09-17T04:08:35.014 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:35.015 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 msgr2=0x7fc4e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 0x7fc4e007b0a0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000c00 tx=0x7fc4f0006b10 comp rx=0 tx=0).stop 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 msgr2=0x7fc5000c6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0118b0 tx=0x7fc4ec011d70 comp rx=0 tx=0).stop 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 shutdown_connections 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fc4e0078b90 0x7fc4e007b0a0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc50015ce20 0x7fc5001e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fc5001e6290 0x7fc5000c6070 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 --2- 172.21.15.129:0/2218106813 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fc4e4004dd0 0x7fc5001e9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 >> 172.21.15.129:0/2218106813 conn(0x7fc50006d410 msgr2=0x7fc5000ad9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 shutdown_connections 2024-09-17T04:08:35.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.015+0000 7fc506c00640 1 -- 172.21.15.129:0/2218106813 wait complete. 2024-09-17T04:08:35.037 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":841,"stamp":"2024-09-17T04:08:34.622194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420688,"kb_used_data":28168,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754554032,"statfs":{"total":773094113280,"available":772663328768,"internally_reserved":0,"allocated":28844032,"data_stored":13452004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.446515"},"pg_stats":[{"pgid":"41.3f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326478+0000","last_change":"2024-09-17T04:08:25.189216+0000","last_active":"2024-09-17T04:08:26.326478+0000","last_peered":"2024-09-17T04:08:26.326478+0000","last_clean":"2024-09-17T04:08:26.326478+0000","last_became_active":"2024-09-17T04:08:25.188921+0000","last_became_peered":"2024-09-17T04:08:25.188921+0000","last_unstale":"2024-09-17T04:08:26.326478+0000","last_undegraded":"2024-09-17T04:08:26.326478+0000","last_fullsized":"2024-09-17T04:08:26.326478+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:29:41.624638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181961+0000","last_change":"2024-09-17T04:08:25.200078+0000","last_active":"2024-09-17T04:08:26.181961+0000","last_peered":"2024-09-17T04:08:26.181961+0000","last_clean":"2024-09-17T04:08:26.181961+0000","last_became_active":"2024-09-17T04:08:25.199691+0000","last_became_peered":"2024-09-17T04:08:25.199691+0000","last_unstale":"2024-09-17T04:08:26.181961+0000","last_undegraded":"2024-09-17T04:08:26.181961+0000","last_fullsized":"2024-09-17T04:08:26.181961+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:21:26.774796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.3d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202842+0000","last_change":"2024-09-17T04:08:25.207544+0000","last_active":"2024-09-17T04:08:26.202842+0000","last_peered":"2024-09-17T04:08:26.202842+0000","last_clean":"2024-09-17T04:08:26.202842+0000","last_became_active":"2024-09-17T04:08:25.207230+0000","last_became_peered":"2024-09-17T04:08:25.207230+0000","last_unstale":"2024-09-17T04:08:26.202842+0000","last_undegraded":"2024-09-17T04:08:26.202842+0000","last_fullsized":"2024-09-17T04:08:26.202842+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:36.890078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179538+0000","last_change":"2024-09-17T04:08:25.205728+0000","last_active":"2024-09-17T04:08:26.179538+0000","last_peered":"2024-09-17T04:08:26.179538+0000","last_clean":"2024-09-17T04:08:26.179538+0000","last_became_active":"2024-09-17T04:08:25.204559+0000","last_became_peered":"2024-09-17T04:08:25.204559+0000","last_unstale":"2024-09-17T04:08:26.179538+0000","last_undegraded":"2024-09-17T04:08:26.179538+0000","last_fullsized":"2024-09-17T04:08:26.179538+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:30:08.422127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.3b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202640+0000","last_change":"2024-09-17T04:08:25.203198+0000","last_active":"2024-09-17T04:08:26.202640+0000","last_peered":"2024-09-17T04:08:26.202640+0000","last_clean":"2024-09-17T04:08:26.202640+0000","last_became_active":"2024-09-17T04:08:25.202329+0000","last_became_peered":"2024-09-17T04:08:25.202329+0000","last_unstale":"2024-09-17T04:08:26.202640+0000","last_undegraded":"2024-09-17T04:08:26.202640+0000","last_fullsized":"2024-09-17T04:08:26.202640+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:32:57.774351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.3a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179130+0000","last_change":"2024-09-17T04:08:25.205488+0000","last_active":"2024-09-17T04:08:26.179130+0000","last_peered":"2024-09-17T04:08:26.179130+0000","last_clean":"2024-09-17T04:08:26.179130+0000","last_became_active":"2024-09-17T04:08:25.204097+0000","last_became_peered":"2024-09-17T04:08:25.204097+0000","last_unstale":"2024-09-17T04:08:26.179130+0000","last_undegraded":"2024-09-17T04:08:26.179130+0000","last_fullsized":"2024-09-17T04:08:26.179130+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:40.637633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.39","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202151+0000","last_change":"2024-09-17T04:08:25.204022+0000","last_active":"2024-09-17T04:08:26.202151+0000","last_peered":"2024-09-17T04:08:26.202151+0000","last_clean":"2024-09-17T04:08:26.202151+0000","last_became_active":"2024-09-17T04:08:25.203590+0000","last_became_peered":"2024-09-17T04:08:25.203590+0000","last_unstale":"2024-09-17T04:08:26.202151+0000","last_undegraded":"2024-09-17T04:08:26.202151+0000","last_fullsized":"2024-09-17T04:08:26.202151+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:27:02.649195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.38","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326289+0000","last_change":"2024-09-17T04:08:25.207974+0000","last_active":"2024-09-17T04:08:26.326289+0000","last_peered":"2024-09-17T04:08:26.326289+0000","last_clean":"2024-09-17T04:08:26.326289+0000","last_became_active":"2024-09-17T04:08:25.207454+0000","last_became_peered":"2024-09-17T04:08:25.207454+0000","last_unstale":"2024-09-17T04:08:26.326289+0000","last_undegraded":"2024-09-17T04:08:26.326289+0000","last_fullsized":"2024-09-17T04:08:26.326289+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:18.281022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.37","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327091+0000","last_change":"2024-09-17T04:08:25.191519+0000","last_active":"2024-09-17T04:08:26.327091+0000","last_peered":"2024-09-17T04:08:26.327091+0000","last_clean":"2024-09-17T04:08:26.327091+0000","last_became_active":"2024-09-17T04:08:25.191096+0000","last_became_peered":"2024-09-17T04:08:25.191096+0000","last_unstale":"2024-09-17T04:08:26.327091+0000","last_undegraded":"2024-09-17T04:08:26.327091+0000","last_fullsized":"2024-09-17T04:08:26.327091+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:51:28.667318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.36","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327027+0000","last_change":"2024-09-17T04:08:25.209134+0000","last_active":"2024-09-17T04:08:26.327027+0000","last_peered":"2024-09-17T04:08:26.327027+0000","last_clean":"2024-09-17T04:08:26.327027+0000","last_became_active":"2024-09-17T04:08:25.208392+0000","last_became_peered":"2024-09-17T04:08:25.208392+0000","last_unstale":"2024-09-17T04:08:26.327027+0000","last_undegraded":"2024-09-17T04:08:26.327027+0000","last_fullsized":"2024-09-17T04:08:26.327027+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:04:51.030185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.35","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181172+0000","last_change":"2024-09-17T04:08:25.206697+0000","last_active":"2024-09-17T04:08:26.181172+0000","last_peered":"2024-09-17T04:08:26.181172+0000","last_clean":"2024-09-17T04:08:26.181172+0000","last_became_active":"2024-09-17T04:08:25.206307+0000","last_became_peered":"2024-09-17T04:08:25.206307+0000","last_unstale":"2024-09-17T04:08:26.181172+0000","last_undegraded":"2024-09-17T04:08:26.181172+0000","last_fullsized":"2024-09-17T04:08:26.181172+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:14.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.34","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326226+0000","last_change":"2024-09-17T04:08:25.196813+0000","last_active":"2024-09-17T04:08:26.326226+0000","last_peered":"2024-09-17T04:08:26.326226+0000","last_clean":"2024-09-17T04:08:26.326226+0000","last_became_active":"2024-09-17T04:08:25.196404+0000","last_became_peered":"2024-09-17T04:08:25.196404+0000","last_unstale":"2024-09-17T04:08:26.326226+0000","last_undegraded":"2024-09-17T04:08:26.326226+0000","last_fullsized":"2024-09-17T04:08:26.326226+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:24.380529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.33","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180588+0000","last_change":"2024-09-17T04:08:25.188090+0000","last_active":"2024-09-17T04:08:26.180588+0000","last_peered":"2024-09-17T04:08:26.180588+0000","last_clean":"2024-09-17T04:08:26.180588+0000","last_became_active":"2024-09-17T04:08:25.187682+0000","last_became_peered":"2024-09-17T04:08:25.187682+0000","last_unstale":"2024-09-17T04:08:26.180588+0000","last_undegraded":"2024-09-17T04:08:26.180588+0000","last_fullsized":"2024-09-17T04:08:26.180588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:37.175115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.32","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202224+0000","last_change":"2024-09-17T04:08:25.203400+0000","last_active":"2024-09-17T04:08:26.202224+0000","last_peered":"2024-09-17T04:08:26.202224+0000","last_clean":"2024-09-17T04:08:26.202224+0000","last_became_active":"2024-09-17T04:08:25.202662+0000","last_became_peered":"2024-09-17T04:08:25.202662+0000","last_unstale":"2024-09-17T04:08:26.202224+0000","last_undegraded":"2024-09-17T04:08:26.202224+0000","last_fullsized":"2024-09-17T04:08:26.202224+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:29:20.930631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.31","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202160+0000","last_change":"2024-09-17T04:08:25.196112+0000","last_active":"2024-09-17T04:08:26.202160+0000","last_peered":"2024-09-17T04:08:26.202160+0000","last_clean":"2024-09-17T04:08:26.202160+0000","last_became_active":"2024-09-17T04:08:25.195717+0000","last_became_peered":"2024-09-17T04:08:25.195717+0000","last_unstale":"2024-09-17T04:08:26.202160+0000","last_undegraded":"2024-09-17T04:08:26.202160+0000","last_fullsized":"2024-09-17T04:08:26.202160+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:25:17.258363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.30","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180941+0000","last_change":"2024-09-17T04:08:25.190931+0000","last_active":"2024-09-17T04:08:26.180941+0000","last_peered":"2024-09-17T04:08:26.180941+0000","last_clean":"2024-09-17T04:08:26.180941+0000","last_became_active":"2024-09-17T04:08:25.188005+0000","last_became_peered":"2024-09-17T04:08:25.188005+0000","last_unstale":"2024-09-17T04:08:26.180941+0000","last_undegraded":"2024-09-17T04:08:26.180941+0000","last_fullsized":"2024-09-17T04:08:26.180941+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:51.219776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182020+0000","last_change":"2024-09-17T04:08:25.210259+0000","last_active":"2024-09-17T04:08:26.182020+0000","last_peered":"2024-09-17T04:08:26.182020+0000","last_clean":"2024-09-17T04:08:26.182020+0000","last_became_active":"2024-09-17T04:08:25.206858+0000","last_became_peered":"2024-09-17T04:08:25.206858+0000","last_unstale":"2024-09-17T04:08:26.182020+0000","last_undegraded":"2024-09-17T04:08:26.182020+0000","last_fullsized":"2024-09-17T04:08:26.182020+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:46:43.017724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326993+0000","last_change":"2024-09-17T04:08:25.209482+0000","last_active":"2024-09-17T04:08:26.326993+0000","last_peered":"2024-09-17T04:08:26.326993+0000","last_clean":"2024-09-17T04:08:26.326993+0000","last_became_active":"2024-09-17T04:08:25.208778+0000","last_became_peered":"2024-09-17T04:08:25.208778+0000","last_unstale":"2024-09-17T04:08:26.326993+0000","last_undegraded":"2024-09-17T04:08:26.326993+0000","last_fullsized":"2024-09-17T04:08:26.326993+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:10:26.075977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.2d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326090+0000","last_change":"2024-09-17T04:08:25.192767+0000","last_active":"2024-09-17T04:08:26.326090+0000","last_peered":"2024-09-17T04:08:26.326090+0000","last_clean":"2024-09-17T04:08:26.326090+0000","last_became_active":"2024-09-17T04:08:25.192484+0000","last_became_peered":"2024-09-17T04:08:25.192484+0000","last_unstale":"2024-09-17T04:08:26.326090+0000","last_undegraded":"2024-09-17T04:08:26.326090+0000","last_fullsized":"2024-09-17T04:08:26.326090+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:09:08.026774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.2c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182024+0000","last_change":"2024-09-17T04:08:25.202002+0000","last_active":"2024-09-17T04:08:26.182024+0000","last_peered":"2024-09-17T04:08:26.182024+0000","last_clean":"2024-09-17T04:08:26.182024+0000","last_became_active":"2024-09-17T04:08:25.201684+0000","last_became_peered":"2024-09-17T04:08:25.201684+0000","last_unstale":"2024-09-17T04:08:26.182024+0000","last_undegraded":"2024-09-17T04:08:26.182024+0000","last_fullsized":"2024-09-17T04:08:26.182024+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:10:20.682978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.2b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179696+0000","last_change":"2024-09-17T04:08:25.202635+0000","last_active":"2024-09-17T04:08:26.179696+0000","last_peered":"2024-09-17T04:08:26.179696+0000","last_clean":"2024-09-17T04:08:26.179696+0000","last_became_active":"2024-09-17T04:08:25.201833+0000","last_became_peered":"2024-09-17T04:08:25.201833+0000","last_unstale":"2024-09-17T04:08:26.179696+0000","last_undegraded":"2024-09-17T04:08:26.179696+0000","last_fullsized":"2024-09-17T04:08:26.179696+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:40:13.227923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.2a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180516+0000","last_change":"2024-09-17T04:08:25.183839+0000","last_active":"2024-09-17T04:08:26.180516+0000","last_peered":"2024-09-17T04:08:26.180516+0000","last_clean":"2024-09-17T04:08:26.180516+0000","last_became_active":"2024-09-17T04:08:25.183527+0000","last_became_peered":"2024-09-17T04:08:25.183527+0000","last_unstale":"2024-09-17T04:08:26.180516+0000","last_undegraded":"2024-09-17T04:08:26.180516+0000","last_fullsized":"2024-09-17T04:08:26.180516+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:19.124388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.29","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326399+0000","last_change":"2024-09-17T04:08:25.189733+0000","last_active":"2024-09-17T04:08:26.326399+0000","last_peered":"2024-09-17T04:08:26.326399+0000","last_clean":"2024-09-17T04:08:26.326399+0000","last_became_active":"2024-09-17T04:08:25.189352+0000","last_became_peered":"2024-09-17T04:08:25.189352+0000","last_unstale":"2024-09-17T04:08:26.326399+0000","last_undegraded":"2024-09-17T04:08:26.326399+0000","last_fullsized":"2024-09-17T04:08:26.326399+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:26:25.468071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.27","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180756+0000","last_change":"2024-09-17T04:08:25.203709+0000","last_active":"2024-09-17T04:08:26.180756+0000","last_peered":"2024-09-17T04:08:26.180756+0000","last_clean":"2024-09-17T04:08:26.180756+0000","last_became_active":"2024-09-17T04:08:25.203364+0000","last_became_peered":"2024-09-17T04:08:25.203364+0000","last_unstale":"2024-09-17T04:08:26.180756+0000","last_undegraded":"2024-09-17T04:08:26.180756+0000","last_fullsized":"2024-09-17T04:08:26.180756+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:39:27.101081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.26","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326742+0000","last_change":"2024-09-17T04:08:25.205002+0000","last_active":"2024-09-17T04:08:26.326742+0000","last_peered":"2024-09-17T04:08:26.326742+0000","last_clean":"2024-09-17T04:08:26.326742+0000","last_became_active":"2024-09-17T04:08:25.204427+0000","last_became_peered":"2024-09-17T04:08:25.204427+0000","last_unstale":"2024-09-17T04:08:26.326742+0000","last_undegraded":"2024-09-17T04:08:26.326742+0000","last_fullsized":"2024-09-17T04:08:26.326742+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:37.951456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.25","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326850+0000","last_change":"2024-09-17T04:08:25.201853+0000","last_active":"2024-09-17T04:08:26.326850+0000","last_peered":"2024-09-17T04:08:26.326850+0000","last_clean":"2024-09-17T04:08:26.326850+0000","last_became_active":"2024-09-17T04:08:25.201493+0000","last_became_peered":"2024-09-17T04:08:25.201493+0000","last_unstale":"2024-09-17T04:08:26.326850+0000","last_undegraded":"2024-09-17T04:08:26.326850+0000","last_fullsized":"2024-09-17T04:08:26.326850+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:52:37.130188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.24","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326183+0000","last_change":"2024-09-17T04:08:25.188324+0000","last_active":"2024-09-17T04:08:26.326183+0000","last_peered":"2024-09-17T04:08:26.326183+0000","last_clean":"2024-09-17T04:08:26.326183+0000","last_became_active":"2024-09-17T04:08:25.188025+0000","last_became_peered":"2024-09-17T04:08:25.188025+0000","last_unstale":"2024-09-17T04:08:26.326183+0000","last_undegraded":"2024-09-17T04:08:26.326183+0000","last_fullsized":"2024-09-17T04:08:26.326183+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:01:41.304344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.23","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181374+0000","last_change":"2024-09-17T04:08:25.193946+0000","last_active":"2024-09-17T04:08:26.181374+0000","last_peered":"2024-09-17T04:08:26.181374+0000","last_clean":"2024-09-17T04:08:26.181374+0000","last_became_active":"2024-09-17T04:08:25.186913+0000","last_became_peered":"2024-09-17T04:08:25.186913+0000","last_unstale":"2024-09-17T04:08:26.181374+0000","last_undegraded":"2024-09-17T04:08:26.181374+0000","last_fullsized":"2024-09-17T04:08:26.181374+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:51.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.22","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326936+0000","last_change":"2024-09-17T04:08:25.194536+0000","last_active":"2024-09-17T04:08:26.326936+0000","last_peered":"2024-09-17T04:08:26.326936+0000","last_clean":"2024-09-17T04:08:26.326936+0000","last_became_active":"2024-09-17T04:08:25.194201+0000","last_became_peered":"2024-09-17T04:08:25.194201+0000","last_unstale":"2024-09-17T04:08:26.326936+0000","last_undegraded":"2024-09-17T04:08:26.326936+0000","last_fullsized":"2024-09-17T04:08:26.326936+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:04:14.659279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.21","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326187+0000","last_change":"2024-09-17T04:08:25.198539+0000","last_active":"2024-09-17T04:08:26.326187+0000","last_peered":"2024-09-17T04:08:26.326187+0000","last_clean":"2024-09-17T04:08:26.326187+0000","last_became_active":"2024-09-17T04:08:25.198178+0000","last_became_peered":"2024-09-17T04:08:25.198178+0000","last_unstale":"2024-09-17T04:08:26.326187+0000","last_undegraded":"2024-09-17T04:08:26.326187+0000","last_fullsized":"2024-09-17T04:08:26.326187+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:18.437486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.20","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326519+0000","last_change":"2024-09-17T04:08:25.206873+0000","last_active":"2024-09-17T04:08:26.326519+0000","last_peered":"2024-09-17T04:08:26.326519+0000","last_clean":"2024-09-17T04:08:26.326519+0000","last_became_active":"2024-09-17T04:08:25.206313+0000","last_became_peered":"2024-09-17T04:08:25.206313+0000","last_unstale":"2024-09-17T04:08:26.326519+0000","last_undegraded":"2024-09-17T04:08:26.326519+0000","last_fullsized":"2024-09-17T04:08:26.326519+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:04:05.166134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326610+0000","last_change":"2024-09-17T04:08:25.199592+0000","last_active":"2024-09-17T04:08:26.326610+0000","last_peered":"2024-09-17T04:08:26.326610+0000","last_clean":"2024-09-17T04:08:26.326610+0000","last_became_active":"2024-09-17T04:08:25.199254+0000","last_became_peered":"2024-09-17T04:08:25.199254+0000","last_unstale":"2024-09-17T04:08:26.326610+0000","last_undegraded":"2024-09-17T04:08:26.326610+0000","last_fullsized":"2024-09-17T04:08:26.326610+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:22.464215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.1f","version":"271'4","reported_seq":19,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326348+0000","last_change":"2024-09-17T04:08:24.190135+0000","last_active":"2024-09-17T04:08:26.326348+0000","last_peered":"2024-09-17T04:08:26.326348+0000","last_clean":"2024-09-17T04:08:26.326348+0000","last_became_active":"2024-09-17T04:08:24.189843+0000","last_became_peered":"2024-09-17T04:08:24.189843+0000","last_unstale":"2024-09-17T04:08:26.326348+0000","last_undegraded":"2024-09-17T04:08:26.326348+0000","last_fullsized":"2024-09-17T04:08:26.326348+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:08:29.916388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.1f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327210+0000","last_change":"2024-09-17T04:08:25.184835+0000","last_active":"2024-09-17T04:08:26.327210+0000","last_peered":"2024-09-17T04:08:26.327210+0000","last_clean":"2024-09-17T04:08:26.327210+0000","last_became_active":"2024-09-17T04:08:25.184368+0000","last_became_peered":"2024-09-17T04:08:25.184368+0000","last_unstale":"2024-09-17T04:08:26.327210+0000","last_undegraded":"2024-09-17T04:08:26.327210+0000","last_fullsized":"2024-09-17T04:08:26.327210+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:34:53.799541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.1e","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326691+0000","last_change":"2024-09-17T04:08:24.196111+0000","last_active":"2024-09-17T04:08:26.326691+0000","last_peered":"2024-09-17T04:08:26.326691+0000","last_clean":"2024-09-17T04:08:26.326691+0000","last_became_active":"2024-09-17T04:08:24.195595+0000","last_became_peered":"2024-09-17T04:08:24.195595+0000","last_unstale":"2024-09-17T04:08:26.326691+0000","last_undegraded":"2024-09-17T04:08:26.326691+0000","last_fullsized":"2024-09-17T04:08:26.326691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:52:44.710662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.1c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202321+0000","last_change":"2024-09-17T04:08:25.209587+0000","last_active":"2024-09-17T04:08:26.202321+0000","last_peered":"2024-09-17T04:08:26.202321+0000","last_clean":"2024-09-17T04:08:26.202321+0000","last_became_active":"2024-09-17T04:08:25.209204+0000","last_became_peered":"2024-09-17T04:08:25.209204+0000","last_unstale":"2024-09-17T04:08:26.202321+0000","last_undegraded":"2024-09-17T04:08:26.202321+0000","last_fullsized":"2024-09-17T04:08:26.202321+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:11.606138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1d","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202378+0000","last_change":"2024-09-17T04:08:24.194368+0000","last_active":"2024-09-17T04:08:26.202378+0000","last_peered":"2024-09-17T04:08:26.202378+0000","last_clean":"2024-09-17T04:08:26.202378+0000","last_became_active":"2024-09-17T04:08:24.193648+0000","last_became_peered":"2024-09-17T04:08:24.193648+0000","last_unstale":"2024-09-17T04:08:26.202378+0000","last_undegraded":"2024-09-17T04:08:26.202378+0000","last_fullsized":"2024-09-17T04:08:26.202378+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:40:08.140394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181218+0000","last_change":"2024-09-17T04:08:25.206411+0000","last_active":"2024-09-17T04:08:26.181218+0000","last_peered":"2024-09-17T04:08:26.181218+0000","last_clean":"2024-09-17T04:08:26.181218+0000","last_became_active":"2024-09-17T04:08:25.206001+0000","last_became_peered":"2024-09-17T04:08:25.206001+0000","last_unstale":"2024-09-17T04:08:26.181218+0000","last_undegraded":"2024-09-17T04:08:26.181218+0000","last_fullsized":"2024-09-17T04:08:26.181218+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:11:19.124388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.1c","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.327023+0000","last_change":"2024-09-17T04:08:24.202248+0000","last_active":"2024-09-17T04:08:26.327023+0000","last_peered":"2024-09-17T04:08:26.327023+0000","last_clean":"2024-09-17T04:08:26.327023+0000","last_became_active":"2024-09-17T04:08:24.201709+0000","last_became_peered":"2024-09-17T04:08:24.201709+0000","last_unstale":"2024-09-17T04:08:26.327023+0000","last_undegraded":"2024-09-17T04:08:26.327023+0000","last_fullsized":"2024-09-17T04:08:26.327023+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:20:43.931505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"41.1a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202515+0000","last_change":"2024-09-17T04:08:25.202178+0000","last_active":"2024-09-17T04:08:26.202515+0000","last_peered":"2024-09-17T04:08:26.202515+0000","last_clean":"2024-09-17T04:08:26.202515+0000","last_became_active":"2024-09-17T04:08:25.199830+0000","last_became_peered":"2024-09-17T04:08:25.199830+0000","last_unstale":"2024-09-17T04:08:26.202515+0000","last_undegraded":"2024-09-17T04:08:26.202515+0000","last_fullsized":"2024-09-17T04:08:26.202515+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:11.236720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1b","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202834+0000","last_change":"2024-09-17T04:08:24.200461+0000","last_active":"2024-09-17T04:08:26.202834+0000","last_peered":"2024-09-17T04:08:26.202834+0000","last_clean":"2024-09-17T04:08:26.202834+0000","last_became_active":"2024-09-17T04:08:24.199506+0000","last_became_peered":"2024-09-17T04:08:24.199506+0000","last_unstale":"2024-09-17T04:08:26.202834+0000","last_undegraded":"2024-09-17T04:08:26.202834+0000","last_fullsized":"2024-09-17T04:08:26.202834+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:54:00.523860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.1b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325694+0000","last_change":"2024-09-17T04:08:25.185548+0000","last_active":"2024-09-17T04:08:26.325694+0000","last_peered":"2024-09-17T04:08:26.325694+0000","last_clean":"2024-09-17T04:08:26.325694+0000","last_became_active":"2024-09-17T04:08:25.184461+0000","last_became_peered":"2024-09-17T04:08:25.184461+0000","last_unstale":"2024-09-17T04:08:26.325694+0000","last_undegraded":"2024-09-17T04:08:26.325694+0000","last_fullsized":"2024-09-17T04:08:26.325694+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:33:46.956363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.1a","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325684+0000","last_change":"2024-09-17T04:08:24.187071+0000","last_active":"2024-09-17T04:08:26.325684+0000","last_peered":"2024-09-17T04:08:26.325684+0000","last_clean":"2024-09-17T04:08:26.325684+0000","last_became_active":"2024-09-17T04:08:24.186743+0000","last_became_peered":"2024-09-17T04:08:24.186743+0000","last_unstale":"2024-09-17T04:08:26.325684+0000","last_undegraded":"2024-09-17T04:08:26.325684+0000","last_fullsized":"2024-09-17T04:08:26.325684+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:50:15.930301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.18","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181632+0000","last_change":"2024-09-17T04:08:25.196132+0000","last_active":"2024-09-17T04:08:26.181632+0000","last_peered":"2024-09-17T04:08:26.181632+0000","last_clean":"2024-09-17T04:08:26.181632+0000","last_became_active":"2024-09-17T04:08:25.194822+0000","last_became_peered":"2024-09-17T04:08:25.194822+0000","last_unstale":"2024-09-17T04:08:26.181632+0000","last_undegraded":"2024-09-17T04:08:26.181632+0000","last_fullsized":"2024-09-17T04:08:26.181632+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:30.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.19","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180956+0000","last_change":"2024-09-17T04:08:24.192860+0000","last_active":"2024-09-17T04:08:26.180956+0000","last_peered":"2024-09-17T04:08:26.180956+0000","last_clean":"2024-09-17T04:08:26.180956+0000","last_became_active":"2024-09-17T04:08:24.192505+0000","last_became_peered":"2024-09-17T04:08:24.192505+0000","last_unstale":"2024-09-17T04:08:26.180956+0000","last_undegraded":"2024-09-17T04:08:26.180956+0000","last_fullsized":"2024-09-17T04:08:26.180956+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:37:19.881424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.19","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325700+0000","last_change":"2024-09-17T04:08:25.195702+0000","last_active":"2024-09-17T04:08:26.325700+0000","last_peered":"2024-09-17T04:08:26.325700+0000","last_clean":"2024-09-17T04:08:26.325700+0000","last_became_active":"2024-09-17T04:08:25.195370+0000","last_became_peered":"2024-09-17T04:08:25.195370+0000","last_unstale":"2024-09-17T04:08:26.325700+0000","last_undegraded":"2024-09-17T04:08:26.325700+0000","last_fullsized":"2024-09-17T04:08:26.325700+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:35.074389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.18","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.198388+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.197800+0000","last_became_peered":"2024-09-17T04:08:24.197800+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:08:50.784271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.16","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202555+0000","last_change":"2024-09-17T04:08:25.201386+0000","last_active":"2024-09-17T04:08:26.202555+0000","last_peered":"2024-09-17T04:08:26.202555+0000","last_clean":"2024-09-17T04:08:26.202555+0000","last_became_active":"2024-09-17T04:08:25.201032+0000","last_became_peered":"2024-09-17T04:08:25.201032+0000","last_unstale":"2024-09-17T04:08:26.202555+0000","last_undegraded":"2024-09-17T04:08:26.202555+0000","last_fullsized":"2024-09-17T04:08:26.202555+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:07:56.224682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.17","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202729+0000","last_change":"2024-09-17T04:08:24.197553+0000","last_active":"2024-09-17T04:08:26.202729+0000","last_peered":"2024-09-17T04:08:26.202729+0000","last_clean":"2024-09-17T04:08:26.202729+0000","last_became_active":"2024-09-17T04:08:24.197233+0000","last_became_peered":"2024-09-17T04:08:24.197233+0000","last_unstale":"2024-09-17T04:08:26.202729+0000","last_undegraded":"2024-09-17T04:08:26.202729+0000","last_fullsized":"2024-09-17T04:08:26.202729+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:40:11.062355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.17","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202539+0000","last_change":"2024-09-17T04:08:25.206017+0000","last_active":"2024-09-17T04:08:26.202539+0000","last_peered":"2024-09-17T04:08:26.202539+0000","last_clean":"2024-09-17T04:08:26.202539+0000","last_became_active":"2024-09-17T04:08:25.205227+0000","last_became_peered":"2024-09-17T04:08:25.205227+0000","last_unstale":"2024-09-17T04:08:26.202539+0000","last_undegraded":"2024-09-17T04:08:26.202539+0000","last_fullsized":"2024-09-17T04:08:26.202539+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:57:06.548696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.16","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326324+0000","last_change":"2024-09-17T04:08:24.191873+0000","last_active":"2024-09-17T04:08:26.326324+0000","last_peered":"2024-09-17T04:08:26.326324+0000","last_clean":"2024-09-17T04:08:26.326324+0000","last_became_active":"2024-09-17T04:08:24.191318+0000","last_became_peered":"2024-09-17T04:08:24.191318+0000","last_unstale":"2024-09-17T04:08:26.326324+0000","last_undegraded":"2024-09-17T04:08:26.326324+0000","last_fullsized":"2024-09-17T04:08:26.326324+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:40:17.268883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.14","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181707+0000","last_change":"2024-09-17T04:08:25.205715+0000","last_active":"2024-09-17T04:08:26.181707+0000","last_peered":"2024-09-17T04:08:26.181707+0000","last_clean":"2024-09-17T04:08:26.181707+0000","last_became_active":"2024-09-17T04:08:25.205374+0000","last_became_peered":"2024-09-17T04:08:25.205374+0000","last_unstale":"2024-09-17T04:08:26.181707+0000","last_undegraded":"2024-09-17T04:08:26.181707+0000","last_fullsized":"2024-09-17T04:08:26.181707+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:09:24.999425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.15","version":"271'3","reported_seq":18,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326468+0000","last_change":"2024-09-17T04:08:24.190381+0000","last_active":"2024-09-17T04:08:26.326468+0000","last_peered":"2024-09-17T04:08:26.326468+0000","last_clean":"2024-09-17T04:08:26.326468+0000","last_became_active":"2024-09-17T04:08:24.189870+0000","last_became_peered":"2024-09-17T04:08:24.189870+0000","last_unstale":"2024-09-17T04:08:26.326468+0000","last_undegraded":"2024-09-17T04:08:26.326468+0000","last_fullsized":"2024-09-17T04:08:26.326468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:55:11.493387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.15","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326092+0000","last_change":"2024-09-17T04:08:25.202764+0000","last_active":"2024-09-17T04:08:26.326092+0000","last_peered":"2024-09-17T04:08:26.326092+0000","last_clean":"2024-09-17T04:08:26.326092+0000","last_became_active":"2024-09-17T04:08:25.202410+0000","last_became_peered":"2024-09-17T04:08:25.202410+0000","last_unstale":"2024-09-17T04:08:26.326092+0000","last_undegraded":"2024-09-17T04:08:26.326092+0000","last_fullsized":"2024-09-17T04:08:26.326092+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:30:47.028739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.14","version":"271'6","reported_seq":21,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325975+0000","last_change":"2024-09-17T04:08:24.200801+0000","last_active":"2024-09-17T04:08:26.325975+0000","last_peered":"2024-09-17T04:08:26.325975+0000","last_clean":"2024-09-17T04:08:26.325975+0000","last_became_active":"2024-09-17T04:08:24.199940+0000","last_became_peered":"2024-09-17T04:08:24.199940+0000","last_unstale":"2024-09-17T04:08:26.325975+0000","last_undegraded":"2024-09-17T04:08:26.325975+0000","last_fullsized":"2024-09-17T04:08:26.325975+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:59:53.586256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.12","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326081+0000","last_change":"2024-09-17T04:08:25.206679+0000","last_active":"2024-09-17T04:08:26.326081+0000","last_peered":"2024-09-17T04:08:26.326081+0000","last_clean":"2024-09-17T04:08:26.326081+0000","last_became_active":"2024-09-17T04:08:25.206365+0000","last_became_peered":"2024-09-17T04:08:25.206365+0000","last_unstale":"2024-09-17T04:08:26.326081+0000","last_undegraded":"2024-09-17T04:08:26.326081+0000","last_fullsized":"2024-09-17T04:08:26.326081+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:30:30.096668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.13","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179808+0000","last_change":"2024-09-17T04:08:24.193138+0000","last_active":"2024-09-17T04:08:26.179808+0000","last_peered":"2024-09-17T04:08:26.179808+0000","last_clean":"2024-09-17T04:08:26.179808+0000","last_became_active":"2024-09-17T04:08:24.192861+0000","last_became_peered":"2024-09-17T04:08:24.192861+0000","last_unstale":"2024-09-17T04:08:26.179808+0000","last_undegraded":"2024-09-17T04:08:26.179808+0000","last_fullsized":"2024-09-17T04:08:26.179808+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:57:47.740618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.13","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182225+0000","last_change":"2024-09-17T04:08:25.194811+0000","last_active":"2024-09-17T04:08:26.182225+0000","last_peered":"2024-09-17T04:08:26.182225+0000","last_clean":"2024-09-17T04:08:26.182225+0000","last_became_active":"2024-09-17T04:08:25.194365+0000","last_became_peered":"2024-09-17T04:08:25.194365+0000","last_unstale":"2024-09-17T04:08:26.182225+0000","last_undegraded":"2024-09-17T04:08:26.182225+0000","last_fullsized":"2024-09-17T04:08:26.182225+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T08:18:35.153511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.12","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326101+0000","last_change":"2024-09-17T04:08:24.197968+0000","last_active":"2024-09-17T04:08:26.326101+0000","last_peered":"2024-09-17T04:08:26.326101+0000","last_clean":"2024-09-17T04:08:26.326101+0000","last_became_active":"2024-09-17T04:08:24.197333+0000","last_became_peered":"2024-09-17T04:08:24.197333+0000","last_unstale":"2024-09-17T04:08:26.326101+0000","last_undegraded":"2024-09-17T04:08:26.326101+0000","last_fullsized":"2024-09-17T04:08:26.326101+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:10.224581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.10","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325571+0000","last_change":"2024-09-17T04:08:25.203605+0000","last_active":"2024-09-17T04:08:26.325571+0000","last_peered":"2024-09-17T04:08:26.325571+0000","last_clean":"2024-09-17T04:08:26.325571+0000","last_became_active":"2024-09-17T04:08:25.203298+0000","last_became_peered":"2024-09-17T04:08:25.203298+0000","last_unstale":"2024-09-17T04:08:26.325571+0000","last_undegraded":"2024-09-17T04:08:26.325571+0000","last_fullsized":"2024-09-17T04:08:26.325571+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:23:36.890078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.11","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180116+0000","last_change":"2024-09-17T04:08:24.199871+0000","last_active":"2024-09-17T04:08:26.180116+0000","last_peered":"2024-09-17T04:08:26.180116+0000","last_clean":"2024-09-17T04:08:26.180116+0000","last_became_active":"2024-09-17T04:08:24.199516+0000","last_became_peered":"2024-09-17T04:08:24.199516+0000","last_unstale":"2024-09-17T04:08:26.180116+0000","last_undegraded":"2024-09-17T04:08:26.180116+0000","last_fullsized":"2024-09-17T04:08:26.180116+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:41:25.253596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.11","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181427+0000","last_change":"2024-09-17T04:08:25.187736+0000","last_active":"2024-09-17T04:08:26.181427+0000","last_peered":"2024-09-17T04:08:26.181427+0000","last_clean":"2024-09-17T04:08:26.181427+0000","last_became_active":"2024-09-17T04:08:25.186265+0000","last_became_peered":"2024-09-17T04:08:25.186265+0000","last_unstale":"2024-09-17T04:08:26.181427+0000","last_undegraded":"2024-09-17T04:08:26.181427+0000","last_fullsized":"2024-09-17T04:08:26.181427+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:15:28.857091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.10","version":"271'8","reported_seq":23,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181716+0000","last_change":"2024-09-17T04:08:24.202703+0000","last_active":"2024-09-17T04:08:26.181716+0000","last_peered":"2024-09-17T04:08:26.181716+0000","last_clean":"2024-09-17T04:08:26.181716+0000","last_became_active":"2024-09-17T04:08:24.201244+0000","last_became_peered":"2024-09-17T04:08:24.201244+0000","last_unstale":"2024-09-17T04:08:26.181716+0000","last_undegraded":"2024-09-17T04:08:26.181716+0000","last_fullsized":"2024-09-17T04:08:26.181716+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:35:32.621406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.e","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182006+0000","last_change":"2024-09-17T04:08:25.201692+0000","last_active":"2024-09-17T04:08:26.182006+0000","last_peered":"2024-09-17T04:08:26.182006+0000","last_clean":"2024-09-17T04:08:26.182006+0000","last_became_active":"2024-09-17T04:08:25.196557+0000","last_became_peered":"2024-09-17T04:08:25.196557+0000","last_unstale":"2024-09-17T04:08:26.182006+0000","last_undegraded":"2024-09-17T04:08:26.182006+0000","last_fullsized":"2024-09-17T04:08:26.182006+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:06:53.716086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.f","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326899+0000","last_change":"2024-09-17T04:08:24.198617+0000","last_active":"2024-09-17T04:08:26.326899+0000","last_peered":"2024-09-17T04:08:26.326899+0000","last_clean":"2024-09-17T04:08:26.326899+0000","last_became_active":"2024-09-17T04:08:24.198153+0000","last_became_peered":"2024-09-17T04:08:24.198153+0000","last_unstale":"2024-09-17T04:08:26.326899+0000","last_undegraded":"2024-09-17T04:08:26.326899+0000","last_fullsized":"2024-09-17T04:08:26.326899+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:01:23.292656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.f","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180793+0000","last_change":"2024-09-17T04:08:25.187031+0000","last_active":"2024-09-17T04:08:26.180793+0000","last_peered":"2024-09-17T04:08:26.180793+0000","last_clean":"2024-09-17T04:08:26.180793+0000","last_became_active":"2024-09-17T04:08:25.186734+0000","last_became_peered":"2024-09-17T04:08:25.186734+0000","last_unstale":"2024-09-17T04:08:26.180793+0000","last_undegraded":"2024-09-17T04:08:26.180793+0000","last_fullsized":"2024-09-17T04:08:26.180793+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:39:30.414825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.e","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202798+0000","last_change":"2024-09-17T04:08:24.195281+0000","last_active":"2024-09-17T04:08:26.202798+0000","last_peered":"2024-09-17T04:08:26.202798+0000","last_clean":"2024-09-17T04:08:26.202798+0000","last_became_active":"2024-09-17T04:08:24.194954+0000","last_became_peered":"2024-09-17T04:08:24.194954+0000","last_unstale":"2024-09-17T04:08:26.202798+0000","last_undegraded":"2024-09-17T04:08:26.202798+0000","last_fullsized":"2024-09-17T04:08:26.202798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:05:29.921479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.c","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326598+0000","last_change":"2024-09-17T04:08:25.189201+0000","last_active":"2024-09-17T04:08:26.326598+0000","last_peered":"2024-09-17T04:08:26.326598+0000","last_clean":"2024-09-17T04:08:26.326598+0000","last_became_active":"2024-09-17T04:08:25.188892+0000","last_became_peered":"2024-09-17T04:08:25.188892+0000","last_unstale":"2024-09-17T04:08:26.326598+0000","last_undegraded":"2024-09-17T04:08:26.326598+0000","last_fullsized":"2024-09-17T04:08:26.326598+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:34:21.473977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"40.d","version":"271'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326585+0000","last_change":"2024-09-17T04:08:24.199722+0000","last_active":"2024-09-17T04:08:26.326585+0000","last_peered":"2024-09-17T04:08:26.326585+0000","last_clean":"2024-09-17T04:08:26.326585+0000","last_became_active":"2024-09-17T04:08:24.199405+0000","last_became_peered":"2024-09-17T04:08:24.199405+0000","last_unstale":"2024-09-17T04:08:26.326585+0000","last_undegraded":"2024-09-17T04:08:26.326585+0000","last_fullsized":"2024-09-17T04:08:26.326585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T16:02:34.614637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.d","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326388+0000","last_change":"2024-09-17T04:08:25.187530+0000","last_active":"2024-09-17T04:08:26.326388+0000","last_peered":"2024-09-17T04:08:26.326388+0000","last_clean":"2024-09-17T04:08:26.326388+0000","last_became_active":"2024-09-17T04:08:25.187224+0000","last_became_peered":"2024-09-17T04:08:25.187224+0000","last_unstale":"2024-09-17T04:08:26.326388+0000","last_undegraded":"2024-09-17T04:08:26.326388+0000","last_fullsized":"2024-09-17T04:08:26.326388+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:48:35.040735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.c","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.182156+0000","last_change":"2024-09-17T04:08:24.199980+0000","last_active":"2024-09-17T04:08:26.182156+0000","last_peered":"2024-09-17T04:08:26.182156+0000","last_clean":"2024-09-17T04:08:26.182156+0000","last_became_active":"2024-09-17T04:08:24.199415+0000","last_became_peered":"2024-09-17T04:08:24.199415+0000","last_unstale":"2024-09-17T04:08:26.182156+0000","last_undegraded":"2024-09-17T04:08:26.182156+0000","last_fullsized":"2024-09-17T04:08:26.182156+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:14:26.267458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"41.a","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202752+0000","last_change":"2024-09-17T04:08:25.198832+0000","last_active":"2024-09-17T04:08:26.202752+0000","last_peered":"2024-09-17T04:08:26.202752+0000","last_clean":"2024-09-17T04:08:26.202752+0000","last_became_active":"2024-09-17T04:08:25.198415+0000","last_became_peered":"2024-09-17T04:08:25.198415+0000","last_unstale":"2024-09-17T04:08:26.202752+0000","last_undegraded":"2024-09-17T04:08:26.202752+0000","last_fullsized":"2024-09-17T04:08:26.202752+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:10:58.479145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.b","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180563+0000","last_change":"2024-09-17T04:08:24.198577+0000","last_active":"2024-09-17T04:08:26.180563+0000","last_peered":"2024-09-17T04:08:26.180563+0000","last_clean":"2024-09-17T04:08:26.180563+0000","last_became_active":"2024-09-17T04:08:24.198252+0000","last_became_peered":"2024-09-17T04:08:24.198252+0000","last_unstale":"2024-09-17T04:08:26.180563+0000","last_undegraded":"2024-09-17T04:08:26.180563+0000","last_fullsized":"2024-09-17T04:08:26.180563+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:16:00.416872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"41.b","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179422+0000","last_change":"2024-09-17T04:08:25.194558+0000","last_active":"2024-09-17T04:08:26.179422+0000","last_peered":"2024-09-17T04:08:26.179422+0000","last_clean":"2024-09-17T04:08:26.179422+0000","last_became_active":"2024-09-17T04:08:25.193536+0000","last_became_peered":"2024-09-17T04:08:25.193536+0000","last_unstale":"2024-09-17T04:08:26.179422+0000","last_undegraded":"2024-09-17T04:08:26.179422+0000","last_fullsized":"2024-09-17T04:08:26.179422+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:49:05.402751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.a","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179060+0000","last_change":"2024-09-17T04:08:24.187161+0000","last_active":"2024-09-17T04:08:26.179060+0000","last_peered":"2024-09-17T04:08:26.179060+0000","last_clean":"2024-09-17T04:08:26.179060+0000","last_became_active":"2024-09-17T04:08:24.186572+0000","last_became_peered":"2024-09-17T04:08:24.186572+0000","last_unstale":"2024-09-17T04:08:26.179060+0000","last_undegraded":"2024-09-17T04:08:26.179060+0000","last_fullsized":"2024-09-17T04:08:26.179060+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:45:00.713638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.8","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.181266+0000","last_change":"2024-09-17T04:08:25.195261+0000","last_active":"2024-09-17T04:08:26.181266+0000","last_peered":"2024-09-17T04:08:26.181266+0000","last_clean":"2024-09-17T04:08:26.181266+0000","last_became_active":"2024-09-17T04:08:25.194414+0000","last_became_peered":"2024-09-17T04:08:25.194414+0000","last_unstale":"2024-09-17T04:08:26.181266+0000","last_undegraded":"2024-09-17T04:08:26.181266+0000","last_fullsized":"2024-09-17T04:08:26.181266+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:02:37.175115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"40.9","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202397+0000","last_change":"2024-09-17T04:08:24.196657+0000","last_active":"2024-09-17T04:08:26.202397+0000","last_peered":"2024-09-17T04:08:26.202397+0000","last_clean":"2024-09-17T04:08:26.202397+0000","last_became_active":"2024-09-17T04:08:24.195146+0000","last_became_peered":"2024-09-17T04:08:24.195146+0000","last_unstale":"2024-09-17T04:08:26.202397+0000","last_undegraded":"2024-09-17T04:08:26.202397+0000","last_fullsized":"2024-09-17T04:08:26.202397+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:16:53.612476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.9","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325728+0000","last_change":"2024-09-17T04:08:25.184757+0000","last_active":"2024-09-17T04:08:26.325728+0000","last_peered":"2024-09-17T04:08:26.325728+0000","last_clean":"2024-09-17T04:08:26.325728+0000","last_became_active":"2024-09-17T04:08:25.184177+0000","last_became_peered":"2024-09-17T04:08:25.184177+0000","last_unstale":"2024-09-17T04:08:26.325728+0000","last_undegraded":"2024-09-17T04:08:26.325728+0000","last_fullsized":"2024-09-17T04:08:26.325728+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:36:35.563823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"40.8","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325392+0000","last_change":"2024-09-17T04:08:24.184366+0000","last_active":"2024-09-17T04:08:26.325392+0000","last_peered":"2024-09-17T04:08:26.325392+0000","last_clean":"2024-09-17T04:08:26.325392+0000","last_became_active":"2024-09-17T04:08:24.182740+0000","last_became_peered":"2024-09-17T04:08:24.182740+0000","last_unstale":"2024-09-17T04:08:26.325392+0000","last_undegraded":"2024-09-17T04:08:26.325392+0000","last_fullsized":"2024-09-17T04:08:26.325392+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:19:25.140619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.6","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202699+0000","last_change":"2024-09-17T04:08:25.215438+0000","last_active":"2024-09-17T04:08:26.202699+0000","last_peered":"2024-09-17T04:08:26.202699+0000","last_clean":"2024-09-17T04:08:26.202699+0000","last_became_active":"2024-09-17T04:08:25.214917+0000","last_became_peered":"2024-09-17T04:08:25.214917+0000","last_unstale":"2024-09-17T04:08:26.202699+0000","last_undegraded":"2024-09-17T04:08:26.202699+0000","last_fullsized":"2024-09-17T04:08:26.202699+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:56:37.666925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.7","version":"271'2","reported_seq":17,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179319+0000","last_change":"2024-09-17T04:08:24.196057+0000","last_active":"2024-09-17T04:08:26.179319+0000","last_peered":"2024-09-17T04:08:26.179319+0000","last_clean":"2024-09-17T04:08:26.179319+0000","last_became_active":"2024-09-17T04:08:24.195822+0000","last_became_peered":"2024-09-17T04:08:24.195822+0000","last_unstale":"2024-09-17T04:08:26.179319+0000","last_undegraded":"2024-09-17T04:08:26.179319+0000","last_fullsized":"2024-09-17T04:08:26.179319+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:46:41.963796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.7","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326477+0000","last_change":"2024-09-17T04:08:25.207446+0000","last_active":"2024-09-17T04:08:26.326477+0000","last_peered":"2024-09-17T04:08:26.326477+0000","last_clean":"2024-09-17T04:08:26.326477+0000","last_became_active":"2024-09-17T04:08:25.207098+0000","last_became_peered":"2024-09-17T04:08:25.207098+0000","last_unstale":"2024-09-17T04:08:26.326477+0000","last_undegraded":"2024-09-17T04:08:26.326477+0000","last_fullsized":"2024-09-17T04:08:26.326477+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:06:13.056929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.6","version":"272'1","reported_seq":16,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202967+0000","last_change":"2024-09-17T04:08:24.202471+0000","last_active":"2024-09-17T04:08:26.202967+0000","last_peered":"2024-09-17T04:08:26.202967+0000","last_clean":"2024-09-17T04:08:26.202967+0000","last_became_active":"2024-09-17T04:08:24.202123+0000","last_became_peered":"2024-09-17T04:08:24.202123+0000","last_unstale":"2024-09-17T04:08:26.202967+0000","last_undegraded":"2024-09-17T04:08:26.202967+0000","last_fullsized":"2024-09-17T04:08:26.202967+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T13:56:31.031197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"41.4","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326446+0000","last_change":"2024-09-17T04:08:25.196030+0000","last_active":"2024-09-17T04:08:26.326446+0000","last_peered":"2024-09-17T04:08:26.326446+0000","last_clean":"2024-09-17T04:08:26.326446+0000","last_became_active":"2024-09-17T04:08:25.195608+0000","last_became_peered":"2024-09-17T04:08:25.195608+0000","last_unstale":"2024-09-17T04:08:26.326446+0000","last_undegraded":"2024-09-17T04:08:26.326446+0000","last_fullsized":"2024-09-17T04:08:26.326446+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:13:59.770015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"40.5","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326338+0000","last_change":"2024-09-17T04:08:24.199653+0000","last_active":"2024-09-17T04:08:26.326338+0000","last_peered":"2024-09-17T04:08:26.326338+0000","last_clean":"2024-09-17T04:08:26.326338+0000","last_became_active":"2024-09-17T04:08:24.197097+0000","last_became_peered":"2024-09-17T04:08:24.197097+0000","last_unstale":"2024-09-17T04:08:26.326338+0000","last_undegraded":"2024-09-17T04:08:26.326338+0000","last_fullsized":"2024-09-17T04:08:26.326338+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:13:01.054583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.5","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.180723+0000","last_change":"2024-09-17T04:08:25.190328+0000","last_active":"2024-09-17T04:08:26.180723+0000","last_peered":"2024-09-17T04:08:26.180723+0000","last_clean":"2024-09-17T04:08:26.180723+0000","last_became_active":"2024-09-17T04:08:25.189973+0000","last_became_peered":"2024-09-17T04:08:25.189973+0000","last_unstale":"2024-09-17T04:08:26.180723+0000","last_undegraded":"2024-09-17T04:08:26.180723+0000","last_fullsized":"2024-09-17T04:08:26.180723+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:17:51.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"40.4","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326117+0000","last_change":"2024-09-17T04:08:24.200442+0000","last_active":"2024-09-17T04:08:26.326117+0000","last_peered":"2024-09-17T04:08:26.326117+0000","last_clean":"2024-09-17T04:08:26.326117+0000","last_became_active":"2024-09-17T04:08:24.199560+0000","last_became_peered":"2024-09-17T04:08:24.199560+0000","last_unstale":"2024-09-17T04:08:26.326117+0000","last_undegraded":"2024-09-17T04:08:26.326117+0000","last_fullsized":"2024-09-17T04:08:26.326117+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:35:07.964242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"41.2","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326588+0000","last_change":"2024-09-17T04:08:25.203847+0000","last_active":"2024-09-17T04:08:26.326588+0000","last_peered":"2024-09-17T04:08:26.326588+0000","last_clean":"2024-09-17T04:08:26.326588+0000","last_became_active":"2024-09-17T04:08:25.203502+0000","last_became_peered":"2024-09-17T04:08:25.203502+0000","last_unstale":"2024-09-17T04:08:26.326588+0000","last_undegraded":"2024-09-17T04:08:26.326588+0000","last_fullsized":"2024-09-17T04:08:26.326588+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T12:55:37.324715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"40.3","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.325805+0000","last_change":"2024-09-17T04:08:24.187060+0000","last_active":"2024-09-17T04:08:26.325805+0000","last_peered":"2024-09-17T04:08:26.325805+0000","last_clean":"2024-09-17T04:08:26.325805+0000","last_became_active":"2024-09-17T04:08:24.186004+0000","last_became_peered":"2024-09-17T04:08:24.186004+0000","last_unstale":"2024-09-17T04:08:26.325805+0000","last_undegraded":"2024-09-17T04:08:26.325805+0000","last_fullsized":"2024-09-17T04:08:26.325805+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:54:11.972381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"41.3","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179649+0000","last_change":"2024-09-17T04:08:25.207493+0000","last_active":"2024-09-17T04:08:26.179649+0000","last_peered":"2024-09-17T04:08:26.179649+0000","last_clean":"2024-09-17T04:08:26.179649+0000","last_became_active":"2024-09-17T04:08:25.205360+0000","last_became_peered":"2024-09-17T04:08:25.205360+0000","last_unstale":"2024-09-17T04:08:26.179649+0000","last_undegraded":"2024-09-17T04:08:26.179649+0000","last_fullsized":"2024-09-17T04:08:26.179649+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:39:22.837177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.2","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179166+0000","last_change":"2024-09-17T04:08:24.186853+0000","last_active":"2024-09-17T04:08:26.179166+0000","last_peered":"2024-09-17T04:08:26.179166+0000","last_clean":"2024-09-17T04:08:26.179166+0000","last_became_active":"2024-09-17T04:08:24.186204+0000","last_became_peered":"2024-09-17T04:08:24.186204+0000","last_unstale":"2024-09-17T04:08:26.179166+0000","last_undegraded":"2024-09-17T04:08:26.179166+0000","last_fullsized":"2024-09-17T04:08:26.179166+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:39:21.078797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"41.0","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.202228+0000","last_change":"2024-09-17T04:08:25.202892+0000","last_active":"2024-09-17T04:08:26.202228+0000","last_peered":"2024-09-17T04:08:26.202228+0000","last_clean":"2024-09-17T04:08:26.202228+0000","last_became_active":"2024-09-17T04:08:25.201830+0000","last_became_peered":"2024-09-17T04:08:25.201830+0000","last_unstale":"2024-09-17T04:08:26.202228+0000","last_undegraded":"2024-09-17T04:08:26.202228+0000","last_fullsized":"2024-09-17T04:08:26.202228+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:02:23.693505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"40.1","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326364+0000","last_change":"2024-09-17T04:08:24.194575+0000","last_active":"2024-09-17T04:08:26.326364+0000","last_peered":"2024-09-17T04:08:26.326364+0000","last_clean":"2024-09-17T04:08:26.326364+0000","last_became_active":"2024-09-17T04:08:24.194250+0000","last_became_peered":"2024-09-17T04:08:24.194250+0000","last_unstale":"2024-09-17T04:08:26.326364+0000","last_undegraded":"2024-09-17T04:08:26.326364+0000","last_fullsized":"2024-09-17T04:08:26.326364+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:34:10.593999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.1","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.179177+0000","last_change":"2024-09-17T04:08:25.194401+0000","last_active":"2024-09-17T04:08:26.179177+0000","last_peered":"2024-09-17T04:08:26.179177+0000","last_clean":"2024-09-17T04:08:26.179177+0000","last_became_active":"2024-09-17T04:08:25.194058+0000","last_became_peered":"2024-09-17T04:08:25.194058+0000","last_unstale":"2024-09-17T04:08:26.179177+0000","last_undegraded":"2024-09-17T04:08:26.179177+0000","last_fullsized":"2024-09-17T04:08:26.179177+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T15:42:07.415333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"40.0","version":"0'0","reported_seq":15,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326794+0000","last_change":"2024-09-17T04:08:24.187831+0000","last_active":"2024-09-17T04:08:26.326794+0000","last_peered":"2024-09-17T04:08:26.326794+0000","last_clean":"2024-09-17T04:08:26.326794+0000","last_became_active":"2024-09-17T04:08:24.186781+0000","last_became_peered":"2024-09-17T04:08:24.186781+0000","last_unstale":"2024-09-17T04:08:26.326794+0000","last_undegraded":"2024-09-17T04:08:26.326794+0000","last_fullsized":"2024-09-17T04:08:26.326794+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:23.149722+0000","last_clean_scrub_stamp":"2024-09-17T04:08:23.149722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:06:48.191171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"41.28","version":"0'0","reported_seq":13,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326126+0000","last_change":"2024-09-17T04:08:25.204595+0000","last_active":"2024-09-17T04:08:26.326126+0000","last_peered":"2024-09-17T04:08:26.326126+0000","last_clean":"2024-09-17T04:08:26.326126+0000","last_became_active":"2024-09-17T04:08:25.204271+0000","last_became_peered":"2024-09-17T04:08:25.204271+0000","last_unstale":"2024-09-17T04:08:26.326126+0000","last_undegraded":"2024-09-17T04:08:26.326126+0000","last_fullsized":"2024-09-17T04:08:26.326126+0000","mapping_epoch":270,"log_start":"0'0","ondisk_log_start":"0'0","created":270,"last_epoch_clean":271,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T04:08:24.161861+0000","last_clean_scrub_stamp":"2024-09-17T04:08:24.161861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:54:34.822087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'72","reported_seq":575,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-17T04:08:26.326753+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:08:26.326753+0000","last_peered":"2024-09-17T04:08:26.326753+0000","last_clean":"2024-09-17T04:08:26.326753+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:08:26.326753+0000","last_undegraded":"2024-09-17T04:08:26.326753+0000","last_fullsized":"2024-09-17T04:08:26.326753+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":41,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":40,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509672,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53196,"kb_used_data":3952,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318644,"statfs":{"total":96636764160,"available":96582291456,"internally_reserved":0,"allocated":4046848,"data_stored":2123764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509675,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34564,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337276,"statfs":{"total":96636764160,"available":96601370624,"internally_reserved":0,"allocated":3452928,"data_stored":1533340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509674,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":3448832,"data_stored":1533306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509671,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56776,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315064,"statfs":{"total":96636764160,"available":96578625536,"internally_reserved":0,"allocated":3452928,"data_stored":1533340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509673,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":3368,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":3448832,"data_stored":1533306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509674,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52612,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319228,"statfs":{"total":96636764160,"available":96582889472,"internally_reserved":0,"allocated":3457024,"data_stored":1533910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542376,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56788,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315052,"statfs":{"total":96636764160,"available":96578613248,"internally_reserved":0,"allocated":3469312,"data_stored":1535010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542377,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53208,"kb_used_data":3972,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318632,"statfs":{"total":96636764160,"available":96582279168,"internally_reserved":0,"allocated":4067328,"data_stored":2126028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":40,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":41,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:08:35.041 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:08:35.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-17T04:08:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 -- 172.21.15.129:0/442089839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40160dd0 msgr2=0x7f5c400cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 --2- 172.21.15.129:0/442089839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40160dd0 0x7f5c400cb210 secure :-1 s=READY pgs=2959 cs=0 l=1 rev1=1 crypto rx=0x7f5c400c7a80 tx=0x7f5c3802aa30 comp rx=0 tx=0).stop 2024-09-17T04:08:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 -- 172.21.15.129:0/442089839 shutdown_connections 2024-09-17T04:08:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 --2- 172.21.15.129:0/442089839 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40160dd0 0x7f5c400cb210 unknown :-1 s=CLOSED pgs=2959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.306 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 --2- 172.21.15.129:0/442089839 >> v2:172.21.15.181:3300/0 conn(0x7f5c400ce030 0x7f5c4015d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 -- 172.21.15.129:0/442089839 >> 172.21.15.129:0/442089839 conn(0x7f5c400b2af0 msgr2=0x7f5c400b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 -- 172.21.15.129:0/442089839 shutdown_connections 2024-09-17T04:08:35.307 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.306+0000 7f5c46400640 1 -- 172.21.15.129:0/442089839 wait complete. 2024-09-17T04:08:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.308+0000 7f5c46400640 1 Processor -- start 2024-09-17T04:08:35.308 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.308+0000 7f5c46400640 1 -- start start 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 0x7f5c400ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 0x7f5c400aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5c400cb8d0 con 0x7f5c40153960 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5c400cb750 con 0x7f5c40150fe0 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c46400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5c400cba50 con 0x7f5c40160dd0 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53398/0 (socket says 172.21.15.129:53398) 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 -- 172.21.15.129:0/3657408258 learned_addr learned my addr 172.21.15.129:0/3657408258 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:35.311 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3ea00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 0x7f5c400ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3fe00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 0x7f5c400aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 -- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 msgr2=0x7f5c400ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 0x7f5c400ae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 -- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 msgr2=0x7f5c400aee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 0x7f5c400aee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3f400640 1 -- 172.21.15.129:0/3657408258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c40151580 con 0x7f5c40153960 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3ea00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 0x7f5c400ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.310+0000 7f5c3fe00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 0x7f5c400aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c3f400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c3d0 tx=0x7f5c3000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30017070 con 0x7f5c40153960 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5c300053c0 con 0x7f5c40153960 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30003850 con 0x7f5c40153960 2024-09-17T04:08:35.312 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c40151860 con 0x7f5c40153960 2024-09-17T04:08:35.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.311+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c40151e20 con 0x7f5c40153960 2024-09-17T04:08:35.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.312+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c400cbf00 con 0x7f5c40153960 2024-09-17T04:08:35.313 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.314+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5c3001a070 con 0x7f5c40153960 2024-09-17T04:08:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.314+0000 7f5c37400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 0x7f5c2007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.314 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.315+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f5c30099f90 con 0x7f5c40153960 2024-09-17T04:08:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.315+0000 7f5c3ea00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 0x7f5c2007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.315 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.316+0000 7f5c3ea00640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 0x7f5c2007b0a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5c401f8ab0 tx=0x7f5c2c005f10 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:35.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.319+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5c300624f0 con 0x7f5c40153960 2024-09-17T04:08:35.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.568+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5c400aff90 con 0x7f5c40153960 2024-09-17T04:08:35.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.571+0000 7f5c37400640 1 -- 172.21.15.129:0/3657408258 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v48) ==== 190+0+147 (secure 0 0 0) 0x7f5c30061df0 con 0x7f5c40153960 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stdout:[client.0] 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQAD/Ohmx5+nNhAAkW1MsLyAazFcQJ3NhGBdmA== 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stdout: caps mds = "allow" 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stdout: caps mon = "allow r" 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-09-17T04:08:35.572 INFO:teuthology.orchestra.run.smithi129.stderr:updated caps for client.0 2024-09-17T04:08:35.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 msgr2=0x7f5c2007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 0x7f5c2007b0a0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f5c401f8ab0 tx=0x7f5c2c005f10 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 msgr2=0x7f5c4014e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 secure :-1 s=READY pgs=2960 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c3d0 tx=0x7f5c3000c890 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 shutdown_connections 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5c20078b90 0x7f5c2007b0a0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5c40150fe0 0x7f5c400aee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5c40160dd0 0x7f5c400ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 --2- 172.21.15.129:0/3657408258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5c40153960 0x7f5c4014e860 unknown :-1 s=CLOSED pgs=2960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 >> 172.21.15.129:0/3657408258 conn(0x7f5c400b2af0 msgr2=0x7f5c400b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 shutdown_connections 2024-09-17T04:08:35.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.574+0000 7f5c46400640 1 -- 172.21.15.129:0/3657408258 wait complete. 2024-09-17T04:08:35.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 -- 172.21.15.129:0/1537059875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc0b3b00 msgr2=0x7fafcc0ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 --2- 172.21.15.129:0/1537059875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc0b3b00 0x7fafcc0ce170 secure :-1 s=READY pgs=2962 cs=0 l=1 rev1=1 crypto rx=0x7fafcc06eb20 tx=0x7fafc402a940 comp rx=0 tx=0).stop 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 -- 172.21.15.129:0/1537059875 shutdown_connections 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 --2- 172.21.15.129:0/1537059875 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc0b3b00 0x7fafcc0ce170 unknown :-1 s=CLOSED pgs=2962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 --2- 172.21.15.129:0/1537059875 >> v2:172.21.15.181:3300/0 conn(0x7fafcc0706a0 0x7fafcc0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 -- 172.21.15.129:0/1537059875 >> 172.21.15.129:0/1537059875 conn(0x7fafcc06d410 msgr2=0x7fafcc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 -- 172.21.15.129:0/1537059875 shutdown_connections 2024-09-17T04:08:35.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.814+0000 7fafd1400640 1 -- 172.21.15.129:0/1537059875 wait complete. 2024-09-17T04:08:35.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.816+0000 7fafd1400640 1 Processor -- start 2024-09-17T04:08:35.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.817+0000 7fafd1400640 1 -- start start 2024-09-17T04:08:35.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafcc0cc260 0x7fafcc1f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafcc0c7320 0x7fafcc1f66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc0ceb80 con 0x7fafcc1f6c10 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc0cea00 con 0x7fafcc0c7320 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafd1400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fafcc0ced00 con 0x7fafcc0cc260 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafcb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafcb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53406/0 (socket says 172.21.15.129:53406) 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.818+0000 7fafcb200640 1 -- 172.21.15.129:0/2801190317 learned_addr learned my addr 172.21.15.129:0/2801190317 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafca800640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafcc0cc260 0x7fafcc1f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafc9e00640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafcc0c7320 0x7fafcc1f66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 -- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafcc0cc260 msgr2=0x7fafcc1f0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafcc0cc260 0x7fafcc1f0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 -- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafcc0c7320 msgr2=0x7fafcc1f66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafcc0c7320 0x7fafcc1f66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:35.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 -- 172.21.15.129:0/2801190317 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafcc1f7020 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafcb200640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005fd0 tx=0x7fafc4031ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc403f030 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fafc4004250 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4004680 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.819+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafcc0c7730 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.820+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafcc0ca060 con 0x7fafcc1f6c10 2024-09-17T04:08:35.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.821+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafcc0c9af0 con 0x7fafcc1f6c10 2024-09-17T04:08:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.822+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fafc4042030 con 0x7fafcc1f6c10 2024-09-17T04:08:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.822+0000 7fafc2a00640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 0x7fafac07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.823+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fafc40bc0a0 con 0x7fafcc1f6c10 2024-09-17T04:08:35.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.823+0000 7fafca800640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 0x7fafac07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:35.824 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.824+0000 7fafca800640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 0x7fafac07b0d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:35.827 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:35.827+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fafcc1f7020 con 0x7fafcc1f6c10 2024-09-17T04:08:36.044 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.044+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafcc0cf220 con 0x7fafcc1f6c10 2024-09-17T04:08:36.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.045+0000 7fafc2a00640 1 -- 172.21.15.129:0/2801190317 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 272 v272) ==== 95+0+5249 (secure 0 0 0) 0x7fafc4004820 con 0x7fafcc1f6c10 2024-09-17T04:08:36.045 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:36.048 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":272,"btime":"2024-09-17T04:08:28:198922+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":272,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:27.686833+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 272 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 msgr2=0x7fafac07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 0x7fafac07b0d0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fafbc005fd0 tx=0x7fafbc005850 comp rx=0 tx=0).stop 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 msgr2=0x7fafcc15d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 secure :-1 s=READY pgs=2963 cs=0 l=1 rev1=1 crypto rx=0x7fafc4005fd0 tx=0x7fafc4031ea0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 shutdown_connections 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fafac078bc0 0x7fafac07b0d0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fafcc1f6c10 0x7fafcc15d620 unknown :-1 s=CLOSED pgs=2963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fafcc0c7320 0x7fafcc1f66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.047+0000 7fafd1400640 1 --2- 172.21.15.129:0/2801190317 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fafcc0cc260 0x7fafcc1f0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.048+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 >> 172.21.15.129:0/2801190317 conn(0x7fafcc06d410 msgr2=0x7fafcc0b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.048+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 shutdown_connections 2024-09-17T04:08:36.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.048+0000 7fafd1400640 1 -- 172.21.15.129:0/2801190317 wait complete. 2024-09-17T04:08:36.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 272, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-17T04:08:24.669089+0000', 'modified': '2024-09-17T04:08:27.686833+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12681}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12681': {'gid': 12681, 'name': 'a', 'rank': 0, 'incarnation': 268, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.129:6835/2485159334', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.129:6833', 'nonce': 2485159334}, {'type': 'v1', 'addr': '172.21.15.129:6835', 'nonce': 2485159334}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [41], 'metadata_pool': 40, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 12681, 'qdb_cluster': [12681]} 2024-09-17T04:08:36.476 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-17T04:08:36.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 -- 172.21.15.129:0/3472783679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e30003a50 msgr2=0x7f7e30003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 --2- 172.21.15.129:0/3472783679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e30003a50 0x7f7e30003ec0 secure :-1 s=READY pgs=2965 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0016c0 tx=0x7f7e30005830 comp rx=0 tx=0).stop 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 -- 172.21.15.129:0/3472783679 shutdown_connections 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 --2- 172.21.15.129:0/3472783679 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e30003a50 0x7f7e30003ec0 unknown :-1 s=CLOSED pgs=2965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 -- 172.21.15.129:0/3472783679 >> 172.21.15.129:0/3472783679 conn(0x7f7e4c06d410 msgr2=0x7f7e4c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 -- 172.21.15.129:0/3472783679 shutdown_connections 2024-09-17T04:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.699+0000 7f7e53600640 1 -- 172.21.15.129:0/3472783679 wait complete. 2024-09-17T04:08:36.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.701+0000 7f7e53600640 1 Processor -- start 2024-09-17T04:08:36.701 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.702+0000 7f7e53600640 1 -- start start 2024-09-17T04:08:36.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e30003a50 0x7f7e4c0c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e4c0c6730 0x7f7e4c1e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c160260 con 0x7f7e4c0c6320 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1600e0 con 0x7f7e4c0c6730 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e53600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7e4c06eed0 con 0x7f7e30003a50 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53434/0 (socket says 172.21.15.129:53434) 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e43e00640 1 -- 172.21.15.129:0/854038404 learned_addr learned my addr 172.21.15.129:0/854038404 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.703+0000 7f7e51600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e4c0c6730 0x7f7e4c1e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 -- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e30003a50 msgr2=0x7f7e4c0c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.705 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e30003a50 0x7f7e4c0c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 -- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e4c0c6730 msgr2=0x7f7e4c1e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e4c0c6730 0x7f7e4c1e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 -- 172.21.15.129:0/854038404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c0c7fd0 con 0x7f7e4c0c6320 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e43e00640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006630 tx=0x7f7e340039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e49600640 1 -- 172.21.15.129:0/854038404 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34015070 con 0x7f7e4c0c6320 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e49600640 1 -- 172.21.15.129:0/854038404 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e34011900 con 0x7f7e4c0c6320 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e4c0c82b0 con 0x7f7e4c0c6320 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.704+0000 7f7e49600640 1 -- 172.21.15.129:0/854038404 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34011d50 con 0x7f7e4c0c6320 2024-09-17T04:08:36.706 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.705+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e4c0c8870 con 0x7f7e4c0c6320 2024-09-17T04:08:36.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.708+0000 7f7e49600640 1 -- 172.21.15.129:0/854038404 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7e34018070 con 0x7f7e4c0c6320 2024-09-17T04:08:36.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.709+0000 7f7e49600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 0x7f7e2807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.709+0000 7f7e49600640 1 -- 172.21.15.129:0/854038404 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f7e3409b9c0 con 0x7f7e4c0c6320 2024-09-17T04:08:36.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.709+0000 7f7e5a7fd740 1 Processor -- start 2024-09-17T04:08:36.709 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.709+0000 7f7e5a7fd740 1 -- start start 2024-09-17T04:08:36.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.710+0000 7f7e50c00640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 0x7f7e2807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.710 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.710+0000 7f7e50c00640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 0x7f7e2807b0a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c032000 tx=0x7f7e3c001120 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96c1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dde964b430 con 0x55dde9321270 2024-09-17T04:08:36.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dde963b5d0 con 0x55dde92f83b0 2024-09-17T04:08:36.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55dde962aa00 con 0x55dde9300590 2024-09-17T04:08:36.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53446/0 (socket says 172.21.15.129:53446) 2024-09-17T04:08:36.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.711+0000 7f7e50c00640 1 -- 172.21.15.129:0/1647939732 learned_addr learned my addr 172.21.15.129:0/1647939732 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.713 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e43e00640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e51600640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96c1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 -- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 msgr2=0x55dde96c1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96c1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 -- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 msgr2=0x55dde96f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 -- 172.21.15.129:0/1647939732 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde95f3520 con 0x55dde9321270 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e43e00640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e51600640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e50c00640 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c000fd0 tx=0x7f7e3c038000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ced26a0b751ecdd server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e41200640 1 -- 172.21.15.129:0/1647939732 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c038070 con 0x55dde9321270 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e41200640 1 -- 172.21.15.129:0/1647939732 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e3c02e040 con 0x55dde9321270 2024-09-17T04:08:36.714 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.712+0000 7f7e41200640 1 -- 172.21.15.129:0/1647939732 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c003da0 con 0x55dde9321270 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/1647939732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 msgr2=0x55dde96f1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 secure :-1 s=READY pgs=2967 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c000fd0 tx=0x7f7e3c038000 comp rx=0 tx=0).stop 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/1647939732 shutdown_connections 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96c1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/1647939732 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96f1970 unknown :-1 s=CLOSED pgs=2967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/1647939732 >> 172.21.15.129:0/1647939732 conn(0x55dde93153c0 msgr2=0x55dde96c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/1647939732 shutdown_connections 2024-09-17T04:08:36.715 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.713+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/1647939732 wait complete. 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.714+0000 7f7e5a7fd740 1 -- start start 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.714+0000 7f7e5a7fd740 1 Processor -- start 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96ea9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dde95d1330 con 0x55dde9321270 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dde962e810 con 0x55dde92f83b0 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55dde939b620 con 0x55dde9300590 2024-09-17T04:08:36.718 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e5a7fd740 10 client.0 _ll_register_callbacks cb 0x55dde95d6420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e51600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e51600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53450/0 (socket says 172.21.15.129:53450) 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.716+0000 7f7e51600640 1 -- 172.21.15.129:0/811383958 learned_addr learned my addr 172.21.15.129:0/811383958 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e43e00640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e50c00640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96ea9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 -- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 msgr2=0x55dde96ef0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96ef0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 -- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 msgr2=0x55dde96ea9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96ea9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 -- 172.21.15.129:0/811383958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde938c850 con 0x55dde9321270 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e51600640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0cdaf0 tx=0x7f7e44003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ac04d4a835d350 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e39600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:08:36.719 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44004b20 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f7e44004cc0 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e3c02e040 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=272) ==== 24+0+0 (secure 0 0 0) 0x7f7e44025050 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e5a7fd740 10 client.12837 fetch_fsmap learned FSMap version 272 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.717+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=272,monmap=2+}) -- 0x55dde94036b0 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e3c02e040 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 272) ==== 3502+0+0 (secure 0 0 0) 0x7f7e44028020 con 0x55dde9321270 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 10 client.12837 fetch_fsmap finished waiting for FSMap version 272 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 10 client.12837 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 20 client.12837 populate_metadata read hostname 'smithi129' 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde92dfe40 0x55dde9704ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 4 client.12837 mds_command: new command op to 12681 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-17T04:08:36.720 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 --> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dde963d520 con 0x55dde92dfe40 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.718+0000 7f7e50c00640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde92dfe40 0x55dde9704ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.719+0000 7f7e39600640 10 client.12837 ms_handle_connect on v2:172.21.15.129:6833/2485159334 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.719+0000 7f7e50c00640 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde92dfe40 0x55dde9704ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e481ff6a937104d server_cookie=abd3bcea84e358ba in_seq=0 out_seq=0 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e39600640 1 -- 172.21.15.129:0/811383958 <== mds.0 v2:172.21.15.129:6833/2485159334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55dde963d520 con 0x55dde92dfe40 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e39600640 10 client.12837 handle_command_reply: tid=0 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 client.12837 shutdown 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 msgr2=0x55dde96c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 secure :-1 s=READY pgs=2968 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0cdaf0 tx=0x7f7e44003f90 comp rx=0 tx=0).stop 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 shutdown_connections 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde92dfe40 0x55dde9704ac0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.722 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9321270 0x55dde96c2c70 unknown :-1 s=CLOSED pgs=2968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9300590 0x55dde96ef0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e39600640 0 client.12837 ms_handle_reset on v2:172.21.15.129:6833/2485159334 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/811383958 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde92f83b0 0x55dde96ea9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 >> 172.21.15.129:0/811383958 conn(0x55dde93153c0 msgr2=0x55dde970f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 shutdown_connections 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/811383958 wait complete. 2024-09-17T04:08:36.723 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.720+0000 7f7e5a7fd740 20 client.12837 trim_cache size 0 max 16384 2024-09-17T04:08:36.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.739+0000 7f7e5a7fd740 1 Processor -- start 2024-09-17T04:08:36.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.739+0000 7f7e5a7fd740 1 -- start start 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 0x55dde96ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 0x55dde972f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c2012e0 con 0x55dde9300590 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7e4c1e6490 con 0x55dde9321270 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7e4c200cc0 con 0x55dde92f83b0 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53456/0 (socket says 172.21.15.129:53456) 2024-09-17T04:08:36.742 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 -- 172.21.15.129:0/399238323 learned_addr learned my addr 172.21.15.129:0/399238323 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e51600640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 0x55dde972f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e50c00640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 0x55dde96ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 -- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 msgr2=0x55dde96ecc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 0x55dde96ecc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 -- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 msgr2=0x55dde972f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 0x55dde972f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e43e00640 1 -- 172.21.15.129:0/399238323 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde96c2fe0 con 0x55dde9300590 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e51600640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 0x55dde972f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.741+0000 7f7e50c00640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 0x55dde96ecc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e43e00640 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f7e34003ec0 tx=0x7f7e340017d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d139c5390aa29494 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e3be00640 1 -- 172.21.15.129:0/399238323 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3401f030 con 0x55dde9300590 2024-09-17T04:08:36.743 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e3be00640 1 -- 172.21.15.129:0/399238323 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e3404c070 con 0x55dde9300590 2024-09-17T04:08:36.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e3be00640 1 -- 172.21.15.129:0/399238323 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e34063c80 con 0x55dde9300590 2024-09-17T04:08:36.744 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/399238323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 msgr2=0x55dde972f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.742+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 secure :-1 s=READY pgs=2969 cs=0 l=1 rev1=1 crypto rx=0x7f7e34003ec0 tx=0x7f7e340017d0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/399238323 shutdown_connections 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9321270 0x55dde972f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde9300590 0x55dde972f390 unknown :-1 s=CLOSED pgs=2969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/399238323 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde92f83b0 0x55dde96ecc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/399238323 >> 172.21.15.129:0/399238323 conn(0x55dde93153c0 msgr2=0x55dde970dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/399238323 shutdown_connections 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.743+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/399238323 wait complete. 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.744+0000 7f7e5a7fd740 1 -- start start 2024-09-17T04:08:36.745 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.744+0000 7f7e5a7fd740 1 Processor -- start 2024-09-17T04:08:36.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9300590 0x55dde970e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.749 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9321270 0x55dde970ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55dde96e80e0 con 0x55dde92f83b0 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55dde96eb890 con 0x55dde9300590 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55dde96e7750 con 0x55dde9321270 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.745+0000 7f7e5a7fd740 10 client.0 _ll_register_callbacks cb 0x7f7e44003e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53472/0 (socket says 172.21.15.129:53472) 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 -- 172.21.15.129:0/4042792258 learned_addr learned my addr 172.21.15.129:0/4042792258 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e51600640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9321270 0x55dde970ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e43e00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9300590 0x55dde970e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 -- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9321270 msgr2=0x55dde970ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9321270 0x55dde970ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 -- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9300590 msgr2=0x55dde970e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9300590 0x55dde970e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.750 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 -- 172.21.15.129:0/4042792258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde97060c0 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e50c00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0017e0 tx=0x7f7e3c004b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1b8d97b96f2834 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.746+0000 7f7e39600640 10 client.0 ms_handle_connect on v2:172.21.15.129:3300/0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c004cb0 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mon.0 v2:172.21.15.129:3300/0 2 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f7e3c03e480 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e3404c070 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_get_version_reply(handle=1 version=272) ==== 24+0+0 (secure 0 0 0) 0x7f7e3c004e50 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 10 client.12843 fetch_fsmap learned FSMap version 272 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,fsmap=272,monmap=2+}) -- 0x55dde9703870 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mon.0 v2:172.21.15.129:3300/0 4 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7e3404c070 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mon.0 v2:172.21.15.129:3300/0 5 ==== fsmap(e 272) ==== 3502+0+0 (secure 0 0 0) 0x7f7e3c03e6a0 con 0x55dde92f83b0 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 10 client.12843 fetch_fsmap finished waiting for FSMap version 272 2024-09-17T04:08:36.751 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 10 client.12843 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 20 client.12843 populate_metadata read hostname 'smithi129' 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde978c940 0x55dde9724d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 4 client.12843 mds_command: new command op to 12681 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.747+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 --> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] -- command(tid 0: {"prefix": "status"}) -- 0x55dde9638fd0 con 0x55dde978c940 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e43e00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde978c940 0x55dde9724d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e39600640 10 client.12843 ms_handle_connect on v2:172.21.15.129:6833/2485159334 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e43e00640 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde978c940 0x55dde9724d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7e76897b36d1fd5 server_cookie=e9be4296ed16468c in_seq=0 out_seq=0 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e39600640 1 -- 172.21.15.129:0/4042792258 <== mds.0 v2:172.21.15.129:6833/2485159334 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55dde9638fd0 con 0x55dde978c940 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e39600640 10 client.12843 handle_command_reply: tid=0 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.748+0000 7f7e5a7fd740 1 client.12843 shutdown 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 msgr2=0x55dde9771b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0017e0 tx=0x7f7e3c004b90 comp rx=0 tx=0).stop 2024-09-17T04:08:36.752 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 shutdown_connections 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:6833/2485159334,v1:172.21.15.129:6835/2485159334] conn(0x55dde978c940 0x55dde9724d80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55dde9321270 0x55dde970ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dde9300590 0x55dde970e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e39600640 0 client.12843 ms_handle_reset on v2:172.21.15.129:6833/2485159334 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 --2- 172.21.15.129:0/4042792258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55dde92f83b0 0x55dde9771b70 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 >> 172.21.15.129:0/4042792258 conn(0x55dde93153c0 msgr2=0x55dde96ecc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 shutdown_connections 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 1 -- 172.21.15.129:0/4042792258 wait complete. 2024-09-17T04:08:36.753 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.749+0000 7f7e5a7fd740 20 client.12843 trim_cache size 0 max 16384 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "cluster_fsid": "c5b53c7d-83e3-4793-838f-5ade77ae49c7", 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "whoami": 0, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 12681, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "want_state": "up:active", 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "state": "up:active", 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "fs_name": "cephfs", 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "rank_uptime": 11.560891493, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "mdsmap_epoch": 272, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch": 272, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "osdmap_epoch_barrier": 272, 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout: "uptime": 16.593825090999999 2024-09-17T04:08:36.754 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:08:36.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 msgr2=0x7f7e2807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 0x7f7e2807b0a0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c032000 tx=0x7f7e3c001120 comp rx=0 tx=0).stop 2024-09-17T04:08:36.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 msgr2=0x7f7e4c1e9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 secure :-1 s=READY pgs=2966 cs=0 l=1 rev1=1 crypto rx=0x7f7e34006630 tx=0x7f7e340039c0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.755 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 shutdown_connections 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7e28078b90 0x7f7e2807b0a0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7e4c0c6730 0x7f7e4c1e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7e4c0c6320 0x7f7e4c1e9730 unknown :-1 s=CLOSED pgs=2966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 --2- 172.21.15.129:0/854038404 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7e30003a50 0x7f7e4c0c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 >> 172.21.15.129:0/854038404 conn(0x7f7e4c06d410 msgr2=0x7f7e4c15ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 shutdown_connections 2024-09-17T04:08:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.750+0000 7f7e53600640 1 -- 172.21.15.129:0/854038404 wait complete. 2024-09-17T04:08:36.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:08:36.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 -- 172.21.15.129:0/1142176517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cd390 msgr2=0x7f75b41609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 --2- 172.21.15.129:0/1142176517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cd390 0x7f75b41609c0 secure :-1 s=READY pgs=2971 cs=0 l=1 rev1=1 crypto rx=0x7f75b40d1370 tx=0x7f75ac02aa10 comp rx=0 tx=0).stop 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 -- 172.21.15.129:0/1142176517 shutdown_connections 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 --2- 172.21.15.129:0/1142176517 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cd390 0x7f75b41609c0 unknown :-1 s=CLOSED pgs=2971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 --2- 172.21.15.129:0/1142176517 >> v2:172.21.15.181:3300/0 conn(0x7f75b40705a0 0x7f75b40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 -- 172.21.15.129:0/1142176517 >> 172.21.15.129:0/1142176517 conn(0x7f75b406d410 msgr2=0x7f75b40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 -- 172.21.15.129:0/1142176517 shutdown_connections 2024-09-17T04:08:36.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.993+0000 7f75bc000640 1 -- 172.21.15.129:0/1142176517 wait complete. 2024-09-17T04:08:36.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.995+0000 7f75bc000640 1 Processor -- start 2024-09-17T04:08:36.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.996+0000 7f75bc000640 1 -- start start 2024-09-17T04:08:36.997 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75b406eed0 0x7f75b415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75b40705a0 0x7f75b41f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f75b4160250 con 0x7f75b40cb030 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f75b406f4f0 con 0x7f75b40705a0 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75bc000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f75b4160f00 con 0x7f75b406eed0 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75ba000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75ba000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53492/0 (socket says 172.21.15.129:53492) 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75ba000640 1 -- 172.21.15.129:0/1712201639 learned_addr learned my addr 172.21.15.129:0/1712201639 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75b8c00640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75b40705a0 0x7f75b41f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.997+0000 7f75b9600640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75b406eed0 0x7f75b415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 -- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75b406eed0 msgr2=0x7f75b415ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75b406eed0 0x7f75b415ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.998 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 -- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75b40705a0 msgr2=0x7f75b41f6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75b40705a0 0x7f75b41f6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:36.999 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 -- 172.21.15.129:0/1712201639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b40cba10 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75ba000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009d60 tx=0x7f75ac003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac033050 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75ac0042d0 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac004720 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b4159bc0 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.998+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b415a180 con 0x7f75b40cb030 2024-09-17T04:08:37.000 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:36.999+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b40b4140 con 0x7f75b40cb030 2024-09-17T04:08:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.001+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f75ac036070 con 0x7f75b40cb030 2024-09-17T04:08:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.001+0000 7f75b1600640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 0x7f759407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:37.001 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.002+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f75ac0bbc90 con 0x7f75b40cb030 2024-09-17T04:08:37.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.002+0000 7f75b9600640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 0x7f759407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:37.002 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.003+0000 7f75b9600640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 0x7f759407b0d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f75a8004630 tx=0x7f75a800b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:37.006 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.006+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f75ac083a70 con 0x7f75b40cb030 2024-09-17T04:08:37.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.225+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75b406a140 con 0x7f75b40cb030 2024-09-17T04:08:37.226 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.226+0000 7f75b1600640 1 -- 172.21.15.129:0/1712201639 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v272) ==== 53+0+73 (secure 0 0 0) 0x7f75ac0884a0 con 0x7f75b40cb030 2024-09-17T04:08:37.226 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:08:37.228 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 msgr2=0x7f759407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 0x7f759407b0d0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f75a8004630 tx=0x7f75a800b2e0 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 msgr2=0x7f75b415c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 secure :-1 s=READY pgs=2972 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009d60 tx=0x7f75ac003f10 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 shutdown_connections 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7594078bc0 0x7f759407b0d0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f75b40cb030 0x7f75b415c5c0 unknown :-1 s=CLOSED pgs=2972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f75b40705a0 0x7f75b41f6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 --2- 172.21.15.129:0/1712201639 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f75b406eed0 0x7f75b415ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 >> 172.21.15.129:0/1712201639 conn(0x7f75b406d410 msgr2=0x7f75b40ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 shutdown_connections 2024-09-17T04:08:37.229 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37.228+0000 7f75bc000640 1 -- 172.21.15.129:0/1712201639 wait complete. 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:37.254 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:08:37.255 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:08:37.255 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:08:37.255 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:37.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.255 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:08:37.290 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:08:37.291 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41575sec preferred_lft 41575sec 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:08:37.292 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:08:37.293 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:37.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.294 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:08:37.384 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:08:37.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.387 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:08:37.423 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:08:37.423 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:08:37.423 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:08:37.423 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:08:37.423 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:08:37.424 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:08:37.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:08:37.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:37.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.619 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:08:37.654 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:08:37.655 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:08:37.655 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:37.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:08:37.752 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:08:37.753 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:37.788 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:37.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:08:37.788 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:37.862 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:08:37.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:37.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:38.058 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:38.058 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:38.058 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:38.079 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:38.079 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:38.079 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:38.080 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:08:38.080 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:08:38.112 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:08:38.113 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:08:38.114 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:08:38.114 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:08:38.114 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:08:38.115 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:08:38.115 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:08:38.117 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:08:38.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:38.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:38.174 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:08:38.254 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:08:38.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:38.271 INFO:teuthology.orchestra.run.smithi129.stdout:client9896 2024-09-17T04:08:38.271 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:38.272 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:38.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2024-09-17T04:08:38.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 -- 172.21.15.129:0/2217674619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d880b3f70 msgr2=0x7f9d880b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:38.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 --2- 172.21.15.129:0/2217674619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d880b3f70 0x7f9d880b49a0 secure :-1 s=READY pgs=2973 cs=0 l=1 rev1=1 crypto rx=0x7f9d880b3a50 tx=0x7f9d7802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:08:38.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 -- 172.21.15.129:0/2217674619 shutdown_connections 2024-09-17T04:08:38.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 --2- 172.21.15.129:0/2217674619 >> v2:172.21.15.181:3300/0 conn(0x7f9d880cba90 0x7f9d880cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 --2- 172.21.15.129:0/2217674619 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d880b3f70 0x7f9d880b49a0 unknown :-1 s=CLOSED pgs=2973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 -- 172.21.15.129:0/2217674619 >> 172.21.15.129:0/2217674619 conn(0x7f9d880ccc40 msgr2=0x7f9d880cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 -- 172.21.15.129:0/2217674619 shutdown_connections 2024-09-17T04:08:38.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.572+0000 7f9d8e400640 1 -- 172.21.15.129:0/2217674619 wait complete. 2024-09-17T04:08:38.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.574+0000 7f9d8e400640 1 Processor -- start 2024-09-17T04:08:38.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.574+0000 7f9d8e400640 1 -- start start 2024-09-17T04:08:38.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.575+0000 7f9d8e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:38.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.575+0000 7f9d8e400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d880cba90 0x7f9d88157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d8e400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 0x7f9d88154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d8e400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d880cc8e0 con 0x7f9d88074580 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d8e400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9d880cc760 con 0x7f9d880cba90 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d8e400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9d88166f20 con 0x7f9d881445a0 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:53508/0 (socket says 172.21.15.129:53508) 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 -- 172.21.15.129:0/1458490843 learned_addr learned my addr 172.21.15.129:0/1458490843 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d7ea00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d880cba90 0x7f9d88157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87e00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 0x7f9d88154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 -- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 msgr2=0x7f9d88154f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 0x7f9d88154f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:38.577 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 -- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d880cba90 msgr2=0x7f9d88157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d880cba90 0x7f9d88157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87400640 1 -- 172.21.15.129:0/1458490843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d881f90a0 con 0x7f9d88074580 2024-09-17T04:08:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.576+0000 7f9d87e00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 0x7f9d88154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:08:38.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d87400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f9d78000c00 tx=0x7f9d78003910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7803e030 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d78003e00 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d881f9380 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d78035040 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.577+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d881f9940 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.578+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d880b3f70 con 0x7f9d88074580 2024-09-17T04:08:38.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.579+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9d78041030 con 0x7f9d88074580 2024-09-17T04:08:38.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.580+0000 7f9d7fe00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 0x7f9d6407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:38.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.580+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7f9d780bb650 con 0x7f9d88074580 2024-09-17T04:08:38.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.581+0000 7f9d7ea00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 0x7f9d6407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:38.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.582+0000 7f9d7ea00640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 0x7f9d6407b0a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f9d70006fd0 tx=0x7f9d70008040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:38.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.584+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9d780839f0 con 0x7f9d88074580 2024-09-17T04:08:38.809 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:38.809+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) -- 0x7f9d88158f10 con 0x7f9d88074580 2024-09-17T04:08:39.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.249+0000 7f9d7fe00640 1 -- 172.21.15.129:0/1458490843 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v273) ==== 167+0+0 (secure 0 0 0) 0x7f9d780832f0 con 0x7f9d88074580 2024-09-17T04:08:39.249 INFO:teuthology.orchestra.run.smithi129.stderr:enabled new snapshots 2024-09-17T04:08:39.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 msgr2=0x7f9d6407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 0x7f9d6407b0a0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f9d70006fd0 tx=0x7f9d70008040 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 msgr2=0x7f9d880acf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f9d78000c00 tx=0x7f9d78003910 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 shutdown_connections 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d64078b90 0x7f9d6407b0a0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d881445a0 0x7f9d88154f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d880cba90 0x7f9d88157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 --2- 172.21.15.129:0/1458490843 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d88074580 0x7f9d880acf50 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 >> 172.21.15.129:0/1458490843 conn(0x7f9d880ccc40 msgr2=0x7f9d880cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 shutdown_connections 2024-09-17T04:08:39.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:39.252+0000 7f9d8e400640 1 -- 172.21.15.129:0/1458490843 wait complete. 2024-09-17T04:08:39.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:39.277 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir snapdir) 2024-09-17T04:08:39.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:39.319 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir snapdir/subdir) 2024-09-17T04:08:39.397 INFO:tasks.cephfs.mount:Writing 8388608 bytes to snapdir/subdir/file_a 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> import zlib 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/snapdir/subdir/file_a" 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> with open(path, '"'"'w'"'"') as f: 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> for i in range(0, 8388608): 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> val = zlib.crc32(str(i).encode('"'"'utf-8'"'"')) & 7 2024-09-17T04:08:39.397 DEBUG:teuthology.orchestra.run.smithi129:> f.write(chr(val)) 2024-09-17T04:08:39.398 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:08:43.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> stdin-killer -- python3 -c ' 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> import os 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> import stat 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> print(os.stat("/home/ubuntu/cephtest/mnt.0/snapdir/subdir/file_a").st_ino) 2024-09-17T04:08:43.823 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:08:43.962 INFO:teuthology.orchestra.run.smithi129.stdout:1099511627778 2024-09-17T04:08:43.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:43.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:43.974 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir snapdir/.snap/snap1) 2024-09-17T04:08:44.077 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:44.077 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:44.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:44.113 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=snapdir/subdir/file_a bs=1M conv=fdatasync count=8 seek=0) 2024-09-17T04:08:44.341 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records in 2024-09-17T04:08:44.341 INFO:teuthology.orchestra.run.smithi129.stderr:8+0 records out 2024-09-17T04:08:44.341 INFO:teuthology.orchestra.run.smithi129.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.103907 s, 80.7 MB/s 2024-09-17T04:08:44.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2239140744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb0004d90 msgr2=0x7fbbb0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2239140744 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbb0004d90 0x7fbbb0005220 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4001f00 tx=0x7fbbb0005830 comp rx=0 tx=0).stop 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbca800640 1 -- 172.21.15.129:0/2239140744 reap_dead start 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2239140744 shutdown_connections 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2239140744 >> 172.21.15.129:0/2239140744 conn(0x7fbbcc06d410 msgr2=0x7fbbcc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2239140744 shutdown_connections 2024-09-17T04:08:44.563 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.562+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2239140744 wait complete. 2024-09-17T04:08:44.564 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.565+0000 7fbbd0e00640 1 Processor -- start 2024-09-17T04:08:44.565 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.565+0000 7fbbd0e00640 1 -- start start 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0004d90 0x7fbbcc1f8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbcc0cccf0 0x7fbbcc0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cbd70 con 0x7fbbcc0add70 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cbbf0 con 0x7fbbb0004d90 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbd0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbbcc0cbef0 con 0x7fbbcc0cccf0 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:44.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32890/0 (socket says 172.21.15.129:32890) 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.566+0000 7fbbc1400640 1 -- 172.21.15.129:0/2192385729 learned_addr learned my addr 172.21.15.129:0/2192385729 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbca800640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbcc0cccf0 0x7fbbcc0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc9e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0004d90 0x7fbbcc1f8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 -- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbcc0cccf0 msgr2=0x7fbbcc0cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbcc0cccf0 0x7fbbcc0cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 -- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0004d90 msgr2=0x7fbbcc1f8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0004d90 0x7fbbcc1f8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 -- 172.21.15.129:0/2192385729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc0cd640 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc1400640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4006600 tx=0x7fbbb40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.567+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4015070 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.568+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbbb4011860 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.568+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc0ae580 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.568+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4011cb0 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.568+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc0aeb40 con 0x7fbbcc0add70 2024-09-17T04:08:44.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.569+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc0ae2d0 con 0x7fbbcc0add70 2024-09-17T04:08:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.570+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbbb4018070 con 0x7fbbcc0add70 2024-09-17T04:08:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.571+0000 7fbbc2a00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 0x7fbba807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.571+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fbbb409b9a0 con 0x7fbbcc0add70 2024-09-17T04:08:44.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.571+0000 7fbbc9e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 0x7fbba807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:44.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.572+0000 7fbbc9e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 0x7fbba807b0a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009df0 tx=0x7fbbbc005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:44.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.575+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbbcc0cd640 con 0x7fbbcc0add70 2024-09-17T04:08:44.792 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.792+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbcc06f860 con 0x7fbbcc0add70 2024-09-17T04:08:44.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.793+0000 7fbbc2a00640 1 -- 172.21.15.129:0/2192385729 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7fbbb4063780 con 0x7fbbcc0add70 2024-09-17T04:08:44.793 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:44.793 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:44.793 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:08:44.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 msgr2=0x7fbba807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:44.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 0x7fbba807b0a0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009df0 tx=0x7fbbbc005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:44.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 msgr2=0x7fbbcc0ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:44.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4006600 tx=0x7fbbb40039c0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.795 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 shutdown_connections 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbba8078b90 0x7fbba807b0a0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbbcc0cccf0 0x7fbbcc0cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbbcc0add70 0x7fbbcc0ca0b0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 --2- 172.21.15.129:0/2192385729 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbbb0004d90 0x7fbbcc1f8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 >> 172.21.15.129:0/2192385729 conn(0x7fbbcc06d410 msgr2=0x7fbbcc15af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 shutdown_connections 2024-09-17T04:08:44.796 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:44.795+0000 7fbbd0e00640 1 -- 172.21.15.129:0/2192385729 wait complete. 2024-09-17T04:08:44.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-17T04:08:45.069 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T04:08:45.069 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T04:08:45.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:45.069 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rm -f snapdir/subdir/file_a) 2024-09-17T04:08:45.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:45.106 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snapdir/subdir) 2024-09-17T04:08:45.186 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:45.186 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:45.262 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:45.262 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:45.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.600+0000 7ffbe2000640 1 -- 172.21.15.129:0/3600372347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 msgr2=0x7ffbdc0ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.600+0000 7ffbe2000640 1 --2- 172.21.15.129:0/3600372347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ccd60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc005fd0 tx=0x7ffbcc02a540 comp rx=0 tx=0).stop 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 -- 172.21.15.129:0/3600372347 shutdown_connections 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 --2- 172.21.15.129:0/3600372347 >> v2:172.21.15.181:3300/0 conn(0x7ffbdc1426c0 0x7ffbdc142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 --2- 172.21.15.129:0/3600372347 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ccd60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.601 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 -- 172.21.15.129:0/3600372347 >> 172.21.15.129:0/3600372347 conn(0x7ffbdc06cfc0 msgr2=0x7ffbdc06d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:45.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 -- 172.21.15.129:0/3600372347 shutdown_connections 2024-09-17T04:08:45.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.601+0000 7ffbe2000640 1 -- 172.21.15.129:0/3600372347 wait complete. 2024-09-17T04:08:45.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.603+0000 7ffbe2000640 1 Processor -- start 2024-09-17T04:08:45.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.603+0000 7ffbe2000640 1 -- start start 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 0x7ffbdc1f9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffbdc0ce310 con 0x7ffbdc1426c0 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffbdc0ce190 con 0x7ffbdc160cc0 2024-09-17T04:08:45.606 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe2000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ffbdc0ce490 con 0x7ffbdc0b51e0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32892/0 (socket says 172.21.15.129:32892) 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 -- 172.21.15.129:0/2694236601 learned_addr learned my addr 172.21.15.129:0/2694236601 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe0c00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbe1600640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 0x7ffbdc1f9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 -- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 msgr2=0x7ffbdc0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 -- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 msgr2=0x7ffbdc1f9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 0x7ffbdc1f9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.605+0000 7ffbd3e00640 1 -- 172.21.15.129:0/2694236601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbdc0cad70 con 0x7ffbdc1426c0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbe1600640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 0x7ffbdc1f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbd3e00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c3a0 tx=0x7ffbc400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:45.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4017070 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbdc0cb050 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffbc40053c0 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc4003850 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.606+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbdc0cb610 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.607+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbdc0caae0 con 0x7ffbdc1426c0 2024-09-17T04:08:45.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.609+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffbc401a070 con 0x7ffbdc1426c0 2024-09-17T04:08:45.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.609+0000 7ffbd9600640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 0x7ffbb807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:45.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.610+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7ffbc409a140 con 0x7ffbdc1426c0 2024-09-17T04:08:45.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.610+0000 7ffbe0c00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 0x7ffbb807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:45.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.611+0000 7ffbe0c00640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 0x7ffbb807b0a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc033000 tx=0x7ffbcc004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:45.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.614+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ffbdc143010 con 0x7ffbdc1426c0 2024-09-17T04:08:45.812 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.812+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffbdc0ce9b0 con 0x7ffbdc1426c0 2024-09-17T04:08:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.813+0000 7ffbd9600640 1 -- 172.21.15.129:0/2694236601 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v272) ==== 80+0+0 (secure 0 0 0) 0x7ffbc4061e10 con 0x7ffbdc1426c0 2024-09-17T04:08:45.813 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:08:45.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 msgr2=0x7ffbb807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:45.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 0x7ffbb807b0a0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc033000 tx=0x7ffbcc004730 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 msgr2=0x7ffbdc15b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7ffbc400c3a0 tx=0x7ffbc400c860 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 shutdown_connections 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffbb8078b90 0x7ffbb807b0a0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffbdc160cc0 0x7ffbdc1f9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffbdc1426c0 0x7ffbdc15b7a0 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 --2- 172.21.15.129:0/2694236601 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffbdc0b51e0 0x7ffbdc0ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 >> 172.21.15.129:0/2694236601 conn(0x7ffbdc06cfc0 msgr2=0x7ffbdc0ccb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 shutdown_connections 2024-09-17T04:08:45.816 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:45.815+0000 7ffbe2000640 1 -- 172.21.15.129:0/2694236601 wait complete. 2024-09-17T04:08:45.838 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:08:45.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:45.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:45.924 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:08:45.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:45.924 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:08:45.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:46.235 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 -- 172.21.15.129:0/421993616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 msgr2=0x7fd8c4070510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 --2- 172.21.15.129:0/421993616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4070510 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc005fd0 tx=0x7fd8bc02aaa0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 -- 172.21.15.129:0/421993616 shutdown_connections 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 --2- 172.21.15.129:0/421993616 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4070510 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 --2- 172.21.15.129:0/421993616 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c40ccc80 0x7fd8c40cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 -- 172.21.15.129:0/421993616 >> 172.21.15.129:0/421993616 conn(0x7fd8c406d410 msgr2=0x7fd8c40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 -- 172.21.15.129:0/421993616 shutdown_connections 2024-09-17T04:08:46.236 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.235+0000 7fd8c9800640 1 -- 172.21.15.129:0/421993616 wait complete. 2024-09-17T04:08:46.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.237+0000 7fd8c9800640 1 Processor -- start 2024-09-17T04:08:46.237 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.238+0000 7fd8c9800640 1 -- start start 2024-09-17T04:08:46.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.240 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c40ccc80 0x7fd8c4098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c415da00 0x7fd8c41efef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4070bd0 con 0x7fd8c4070100 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c4070a50 con 0x7fd8c40ccc80 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c4070d50 con 0x7fd8c415da00 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32916/0 (socket says 172.21.15.129:32916) 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c2a00640 1 -- 172.21.15.129:0/771071057 learned_addr learned my addr 172.21.15.129:0/771071057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c3400640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c415da00 0x7fd8c41efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.239+0000 7fd8c2000640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c40ccc80 0x7fd8c4098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 -- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c415da00 msgr2=0x7fd8c41efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c415da00 0x7fd8c41efef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 -- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c40ccc80 msgr2=0x7fd8c4098a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.241 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c40ccc80 0x7fd8c4098a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 -- 172.21.15.129:0/771071057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c41f7510 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c2a00640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c3a0 tx=0x7fd8b400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4017070 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd8b40053c0 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c41f6990 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.240+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b4003850 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.241+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c41f6f50 con 0x7fd8c4070100 2024-09-17T04:08:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.242+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c4074ea0 con 0x7fd8c4070100 2024-09-17T04:08:46.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.243+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd8b401a070 con 0x7fd8c4070100 2024-09-17T04:08:46.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.244+0000 7fd8baa00640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 0x7fd8a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.244+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fd8b409a060 con 0x7fd8c4070100 2024-09-17T04:08:46.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.244+0000 7fd8c2000640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 0x7fd8a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.245 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.245+0000 7fd8c2000640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 0x7fd8a407b0a0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd8c408d6d0 tx=0x7fd8b000b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:46.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.248+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fd8c4074ea0 con 0x7fd8c4070100 2024-09-17T04:08:46.470 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.470+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8c40ad780 con 0x7fd8c4070100 2024-09-17T04:08:46.471 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.471+0000 7fd8baa00640 1 -- 172.21.15.129:0/771071057 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7fd8c40ad780 con 0x7fd8c4070100 2024-09-17T04:08:46.471 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:46.471 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:46.471 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 msgr2=0x7fd8a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 0x7fd8a407b0a0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fd8c408d6d0 tx=0x7fd8b000b2e0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 msgr2=0x7fd8c4095570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fd8b400c3a0 tx=0x7fd8b400c860 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 shutdown_connections 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd8a4078b90 0x7fd8a407b0a0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd8c415da00 0x7fd8c41efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c40ccc80 0x7fd8c4098a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 --2- 172.21.15.129:0/771071057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd8c4070100 0x7fd8c4095570 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.474 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 >> 172.21.15.129:0/771071057 conn(0x7fd8c406d410 msgr2=0x7fd8c4099910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:46.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.473+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 shutdown_connections 2024-09-17T04:08:46.475 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.474+0000 7fd8c9800640 1 -- 172.21.15.129:0/771071057 wait complete. 2024-09-17T04:08:46.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_created 2024-09-17T04:08:46.736 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_created":2}} 2024-09-17T04:08:46.736 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_created": 2 } } 2024-09-17T04:08:46.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 -- 172.21.15.129:0/147110486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 msgr2=0x7fa5c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/147110486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5c8004c10 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc001050 tx=0x7fa5c8005830 comp rx=0 tx=0).stop 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 -- 172.21.15.129:0/147110486 shutdown_connections 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/147110486 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5c8004c10 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 -- 172.21.15.129:0/147110486 >> 172.21.15.129:0/147110486 conn(0x7fa5e006d410 msgr2=0x7fa5e00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.958+0000 7fa5e7a00640 1 -- 172.21.15.129:0/147110486 shutdown_connections 2024-09-17T04:08:46.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.959+0000 7fa5e7a00640 1 -- 172.21.15.129:0/147110486 wait complete. 2024-09-17T04:08:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.961+0000 7fa5e7a00640 1 Processor -- start 2024-09-17T04:08:46.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.961+0000 7fa5e7a00640 1 -- start start 2024-09-17T04:08:46.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa5e00c7320 0x7fa5e00cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5e00d07b0 0x7fa5e01f8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e00b4140 con 0x7fa5c8004800 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e006f4f0 con 0x7fa5e00d07b0 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.962+0000 7fa5e7a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e00b42c0 con 0x7fa5e00c7320 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32936/0 (socket says 172.21.15.129:32936) 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 -- 172.21.15.129:0/3422709720 learned_addr learned my addr 172.21.15.129:0/3422709720 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5dfe00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa5e00c7320 0x7fa5e00cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5e00d07b0 0x7fa5e01f8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 -- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa5e00c7320 msgr2=0x7fa5e00cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa5e00c7320 0x7fa5e00cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 -- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5e00d07b0 msgr2=0x7fa5e01f8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5e00d07b0 0x7fa5e01f8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 -- 172.21.15.129:0/3422709720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e015f490 con 0x7fa5c8004800 2024-09-17T04:08:46.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.963+0000 7fa5e5000640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4011a10 tx=0x7fa5d4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.964+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d401e070 con 0x7fa5c8004800 2024-09-17T04:08:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.964+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa5d4004460 con 0x7fa5c8004800 2024-09-17T04:08:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.964+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d400fc00 con 0x7fa5c8004800 2024-09-17T04:08:46.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.964+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e00cb340 con 0x7fa5c8004800 2024-09-17T04:08:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.964+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e01f95b0 con 0x7fa5c8004800 2024-09-17T04:08:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.965+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e015fd20 con 0x7fa5c8004800 2024-09-17T04:08:46.966 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.966+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa5d4021070 con 0x7fa5c8004800 2024-09-17T04:08:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.967+0000 7fa5dcc00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 0x7fa5c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.967+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fa5d409b710 con 0x7fa5c8004800 2024-09-17T04:08:46.967 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.968+0000 7fa5dfe00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 0x7fa5c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:46.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.968+0000 7fa5dfe00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 0x7fa5c007b0a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc033000 tx=0x7fa5cc0019f0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:46.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:46.972+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa5e015fd20 con 0x7fa5c8004800 2024-09-17T04:08:47.190 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.190+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa5e00ccc80 con 0x7fa5c8004800 2024-09-17T04:08:47.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.191+0000 7fa5dcc00640 1 -- 172.21.15.129:0/3422709720 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v273) ==== 53+0+73 (secure 0 0 0) 0x7fa5d4063590 con 0x7fa5c8004800 2024-09-17T04:08:47.191 INFO:teuthology.orchestra.run.smithi129.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-17T04:08:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 msgr2=0x7fa5c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:47.193 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 0x7fa5c007b0a0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc033000 tx=0x7fa5cc0019f0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 msgr2=0x7fa5e00ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4011a10 tx=0x7fa5d4011ed0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 shutdown_connections 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa5c0078b90 0x7fa5c007b0a0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa5e00d07b0 0x7fa5e01f8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa5e00c7320 0x7fa5e00cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 --2- 172.21.15.129:0/3422709720 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa5c8004800 0x7fa5e00ca8c0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.193+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 >> 172.21.15.129:0/3422709720 conn(0x7fa5e006d410 msgr2=0x7fa5e00b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.194+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 shutdown_connections 2024-09-17T04:08:47.194 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47.194+0000 7fa5e7a00640 1 -- 172.21.15.129:0/3422709720 wait complete. 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi129.front.sepia.ceph.com 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-17T04:08:47.218 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:47.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.219 DEBUG:teuthology.orchestra.run.smithi129:> ip addr 2024-09-17T04:08:47.255 INFO:teuthology.orchestra.run.smithi129.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-17T04:08:47.255 INFO:teuthology.orchestra.run.smithi129.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: inet 127.0.0.1/8 scope host lo 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 ::1/128 scope host 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:38 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f0 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:d9:19:39 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: altname enp7s0f1 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-17T04:08:47.256 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:60 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 0c:c4:7a:88:72:61 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: inet 172.21.15.129/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft 41565sec preferred_lft 41565sec 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::ec4:7aff:fe88:7261/64 scope link noprefixroute 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::74c9:26ff:fe7b:9084/64 scope link 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.257 INFO:teuthology.orchestra.run.smithi129.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 8e:6f:06:1e:f2:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::8c6f:6ff:fe1e:f2d4/64 scope link 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: link/ether 62:85:96:61:ce:b9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: inet6 fe80::6085:96ff:fe61:ceb9/64 scope link 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run.smithi129.stdout: valid_lft forever preferred_lft forever 2024-09-17T04:08:47.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.258 DEBUG:teuthology.orchestra.run.smithi129:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-17T04:08:47.350 INFO:teuthology.orchestra.run.smithi129.stdout:1 2024-09-17T04:08:47.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.353 DEBUG:teuthology.orchestra.run.smithi129:> route 2024-09-17T04:08:47.389 INFO:teuthology.orchestra.run.smithi129.stdout:Kernel IP routing table 2024-09-17T04:08:47.389 INFO:teuthology.orchestra.run.smithi129.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-17T04:08:47.389 INFO:teuthology.orchestra.run.smithi129.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-17T04:08:47.389 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-17T04:08:47.390 INFO:teuthology.orchestra.run.smithi129.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> set -e 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-17T04:08:47.390 DEBUG:teuthology.orchestra.run.smithi129:> ') 2024-09-17T04:08:47.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-17T04:08:47.578 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:47.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.587 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:08:47.622 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:08:47.623 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:08:47.623 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:47.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link set brx.0 up 2024-09-17T04:08:47.719 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:08:47.719 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:47.754 INFO:teuthology.orchestra.run.smithi129.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-17T04:08:47.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-17T04:08:47.755 DEBUG:teuthology.orchestra.run.smithi129:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:47.829 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-09-17T04:08:47.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:47.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:48.023 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:48.024 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:48.024 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:48.044 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:48.045 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:48.045 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:48.045 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:08:48.045 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:08:48.075 INFO:teuthology.orchestra.run.smithi129.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:invalid new device string format 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:3300,172.21.15.181:3300,172.21.15.181:3301" 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: switching to using v1 address with old syntax 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: resolved to: "172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790" 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.129:6789,172.21.15.181:6789,172.21.15.181:6790:/ 2024-09-17T04:08:48.076 INFO:teuthology.orchestra.run.smithi129.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=c5b53c7d-83e3-4793-838f-5ade77ae49c7" will pass to kernel 2024-09-17T04:08:48.077 INFO:teuthology.orchestra.run.smithi129.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-17T04:08:48.077 INFO:teuthology.orchestra.run.smithi129.stdout: You just mounted a file system that supports labels which does not 2024-09-17T04:08:48.077 INFO:teuthology.orchestra.run.smithi129.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-17T04:08:48.077 INFO:teuthology.orchestra.run.smithi129.stdout: applications will generate AVC messages and not be allowed access to 2024-09-17T04:08:48.077 INFO:teuthology.orchestra.run.smithi129.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-17T04:08:48.080 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-17T04:08:48.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:48.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-17T04:08:48.136 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-17T04:08:48.216 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-09-17T04:08:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:48.227 INFO:teuthology.orchestra.run.smithi129.stdout:client11839 2024-09-17T04:08:48.227 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:08:48.228 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:08:48.263 INFO:tasks.cephfs.mount:Validating 8388608 bytes from snapdir/.snap/snap1/subdir/file_a 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo stdin-killer -- python3 -c ' 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> import zlib 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> path = "/home/ubuntu/cephtest/mnt.0/snapdir/.snap/snap1/subdir/file_a" 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> with open(path, '"'"'r'"'"') as f: 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> bytes = f.read() 2024-09-17T04:08:48.263 DEBUG:teuthology.orchestra.run.smithi129:> if len(bytes) != 8388608: 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> raise RuntimeError("Bad length {0} vs. expected {1}".format( 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> len(bytes), 8388608 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> )) 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> for i, b in enumerate(bytes): 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> val = zlib.crc32(str(i).encode('"'"'utf-8'"'"')) & 7 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> if b != chr(val): 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> raise RuntimeError("Bad data at offset {0}".format(i)) 2024-09-17T04:08:48.264 DEBUG:teuthology.orchestra.run.smithi129:> ' 2024-09-17T04:08:50.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:08:50.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:08:52.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-17T04:08:52.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:08:52.627 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snapdir/.snap/snap1) 2024-09-17T04:08:52.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 -- 172.21.15.129:0/3247280701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 msgr2=0x7fb0c8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 --2- 172.21.15.129:0/3247280701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0c8004c10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0016c0 tx=0x7fb0c8005830 comp rx=0 tx=0).stop 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 -- 172.21.15.129:0/3247280701 shutdown_connections 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 --2- 172.21.15.129:0/3247280701 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0c8004c10 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 -- 172.21.15.129:0/3247280701 >> 172.21.15.129:0/3247280701 conn(0x7fb0e406d410 msgr2=0x7fb0e40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 -- 172.21.15.129:0/3247280701 shutdown_connections 2024-09-17T04:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.935+0000 7fb0e9800640 1 -- 172.21.15.129:0/3247280701 wait complete. 2024-09-17T04:08:52.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.937+0000 7fb0e9800640 1 Processor -- start 2024-09-17T04:08:52.937 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.938+0000 7fb0e9800640 1 -- start start 2024-09-17T04:08:52.940 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0e415f6e0 0x7fb0e415d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0e415faf0 0x7fb0e415dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e40cfa50 con 0x7fb0c8004800 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e40cf8d0 con 0x7fb0e415faf0 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e9800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb0e40cfbd0 con 0x7fb0e415f6e0 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e2a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:32954/0 (socket says 172.21.15.129:32954) 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.939+0000 7fb0e2a00640 1 -- 172.21.15.129:0/1772979064 learned_addr learned my addr 172.21.15.129:0/1772979064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e3400640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0e415faf0 0x7fb0e415dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0da000640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0e415f6e0 0x7fb0e415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 -- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0e415f6e0 msgr2=0x7fb0e415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:52.941 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0e415f6e0 0x7fb0e415d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 -- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0e415faf0 msgr2=0x7fb0e415dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0e415faf0 0x7fb0e415dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 -- 172.21.15.129:0/1772979064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e41f9430 con 0x7fb0c8004800 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e2a00640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4001e00 tx=0x7fb0d40049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4015020 con 0x7fb0c8004800 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb0d4011960 con 0x7fb0c8004800 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e41f9710 con 0x7fb0c8004800 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.940+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4011d70 con 0x7fb0c8004800 2024-09-17T04:08:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.941+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e41f9cd0 con 0x7fb0c8004800 2024-09-17T04:08:52.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.942+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0e4070210 con 0x7fb0c8004800 2024-09-17T04:08:52.943 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.943+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb0d4018020 con 0x7fb0c8004800 2024-09-17T04:08:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.944+0000 7fb0db400640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 0x7fb0c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.944+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5709+0+0 (secure 0 0 0) 0x7fb0d409b8a0 con 0x7fb0c8004800 2024-09-17T04:08:52.944 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.944+0000 7fb0da000640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 0x7fb0c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:52.945 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.945+0000 7fb0da000640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 0x7fb0c007b0a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc032000 tx=0x7fb0cc001190 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:52.948 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:52.948+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb0d401f020 con 0x7fb0c8004800 2024-09-17T04:08:53.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.170+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0e41f9f10 con 0x7fb0c8004800 2024-09-17T04:08:53.170 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.171+0000 7fb0db400640 1 -- 172.21.15.129:0/1772979064 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7fb0e40cd5d0 con 0x7fb0c8004800 2024-09-17T04:08:53.171 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:53.171 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:53.171 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:08:53.173 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 msgr2=0x7fb0c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 0x7fb0c007b0a0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc032000 tx=0x7fb0cc001190 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 msgr2=0x7fb0e4159400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4001e00 tx=0x7fb0d40049d0 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 shutdown_connections 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb0c0078b90 0x7fb0c007b0a0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb0e415faf0 0x7fb0e415dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb0e415f6e0 0x7fb0e415d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 --2- 172.21.15.129:0/1772979064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb0c8004800 0x7fb0e4159400 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 >> 172.21.15.129:0/1772979064 conn(0x7fb0e406d410 msgr2=0x7fb0e40ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 shutdown_connections 2024-09-17T04:08:53.174 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:53.173+0000 7fb0e9800640 1 -- 172.21.15.129:0/1772979064 wait complete. 2024-09-17T04:08:53.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-17T04:08:55.186 INFO:teuthology.orchestra.run.smithi129.stdout:{"message":"","return_code":0} 2024-09-17T04:08:55.186 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-17T04:08:55.187 DEBUG:teuthology.orchestra.run.smithi129:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data ls 2024-09-17T04:08:55.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.336+0000 7fcfdb218f40 1 Processor -- start 2024-09-17T04:08:55.336 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.336+0000 7fcfdb218f40 1 -- start start 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c043440 0x55c29c042610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c02c140 0x55c29c042b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c29bf1c1d0 con 0x55c29c03d360 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c29bf28f90 con 0x55c29c043440 2024-09-17T04:08:55.338 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55c29bebdc60 con 0x55c29c02c140 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfd4400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfd4400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51362/0 (socket says 172.21.15.129:51362) 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.337+0000 7fcfd4400640 1 -- 172.21.15.129:0/1202670659 learned_addr learned my addr 172.21.15.129:0/1202670659 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c043440 0x55c29c042610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c02c140 0x55c29c042b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 -- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c02c140 msgr2=0x55c29c042b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c02c140 0x55c29c042b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 -- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c043440 msgr2=0x55c29c042610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c043440 0x55c29c042610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 -- 172.21.15.129:0/1202670659 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c29bfb07d0 con 0x55c29c03d360 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd4400640 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4001d50 tx=0x7fcfc400b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971967692d454788 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd3000640 1 -- 172.21.15.129:0/1202670659 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400bd80 con 0x55c29c03d360 2024-09-17T04:08:55.339 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd3000640 1 -- 172.21.15.129:0/1202670659 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc400f400 con 0x55c29c03d360 2024-09-17T04:08:55.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.338+0000 7fcfd3000640 1 -- 172.21.15.129:0/1202670659 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400f850 con 0x55c29c03d360 2024-09-17T04:08:55.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 -- 172.21.15.129:0/1202670659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 msgr2=0x55c29c025220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.340 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4001d50 tx=0x7fcfc400b5b0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 -- 172.21.15.129:0/1202670659 shutdown_connections 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c02c140 0x55c29c042b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c043440 0x55c29c042610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/1202670659 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29c025220 unknown :-1 s=CLOSED pgs=2989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 -- 172.21.15.129:0/1202670659 >> 172.21.15.129:0/1202670659 conn(0x55c29be8eac0 msgr2=0x55c29bfb8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 -- 172.21.15.129:0/1202670659 shutdown_connections 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.339+0000 7fcfdb218f40 1 -- 172.21.15.129:0/1202670659 wait complete. 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.340+0000 7fcfdb218f40 1 Processor -- start 2024-09-17T04:08:55.341 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.340+0000 7fcfdb218f40 1 -- start start 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c043440 0x55c29c041040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c019540 0x55c29c041580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x55c29c02bcb0 con 0x55c29c03d360 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x55c29c01bbe0 con 0x55c29c019540 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfdb218f40 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x55c29c023310 con 0x55c29c043440 2024-09-17T04:08:55.343 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51370/0 (socket says 172.21.15.129:51370) 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 learned_addr learned my addr 172.21.15.129:0/3687654358 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c043440 0x55c29c041040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c019540 0x55c29c041580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c043440 msgr2=0x55c29c041040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c043440 0x55c29c041040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c019540 msgr2=0x55c29c041580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c29c019540 0x55c29c041580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c29c019090 con 0x55c29c03d360 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.342+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x55c29c043440 0x55c29c041040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400feb0 tx=0x7fcfc400fee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfd1c00640 1 -- 172.21.15.129:0/3687654358 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400be40 con 0x55c29c03d360 2024-09-17T04:08:55.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfd1c00640 1 -- 172.21.15.129:0/3687654358 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc4020020 con 0x55c29c03d360 2024-09-17T04:08:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfd1c00640 1 -- 172.21.15.129:0/3687654358 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc401b790 con 0x55c29c03d360 2024-09-17T04:08:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c29c024a90 con 0x55c29c03d360 2024-09-17T04:08:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.343+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c29c0421b0 con 0x55c29c03d360 2024-09-17T04:08:55.345 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.345+0000 7fcfd1c00640 1 -- 172.21.15.129:0/3687654358 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcfc4017070 con 0x55c29c03d360 2024-09-17T04:08:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.346+0000 7fcfd1c00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 0x7fcfb0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.346 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.346+0000 7fcfd1c00640 1 -- 172.21.15.129:0/3687654358 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7fcfc409b120 con 0x55c29c03d360 2024-09-17T04:08:55.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.347+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x55c29c01adb0 0x55c29c01b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.347+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:1 41.0 41:00000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.347+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 0x7fcfb0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.347 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.347+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x55c29c01adb0 0x55c29c01b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.348+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 0x7fcfb0079ab0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x55c29bf73040 tx=0x7fcfbc0058c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.348+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x55c29c01adb0 0x55c29c01b1c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.348 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.348+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== osd_op_reply(1 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.349+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fcfcc022e70 0x7fcfcc055410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.349+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:2 41.20 41:04000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc022e70 2024-09-17T04:08:55.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.349+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fcfcc022e70 0x7fcfcc055410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.350+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fcfcc022e70 0x7fcfcc055410 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.353+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== osd_op_reply(2 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc022e70 2024-09-17T04:08:55.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.353+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fcfc4063100 0x7fcfc4061b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.353+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:3 41.10 41:08000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfc4063100 2024-09-17T04:08:55.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.353+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fcfc4063100 0x7fcfc4061b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.354+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fcfc4063100 0x7fcfc4061b70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.354+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== osd_op_reply(3 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00c050 con 0x7fcfc4063100 2024-09-17T04:08:55.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.354+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfbc017910 0x7fcfbc0083f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.354 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.354+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:4 41.30 41:0c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfbc017910 2024-09-17T04:08:55.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.355+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfbc017910 0x7fcfbc0083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.355+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfbc017910 0x7fcfbc0083f0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.356+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== osd_op_reply(4 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.356+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:5 41.8 41:10000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfbc017910 2024-09-17T04:08:55.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.356+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== osd_op_reply(5 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.357+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fcfcc01f7d0 0x7fcfcc06f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.357+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:6 41.28 41:14000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.357+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fcfcc01f7d0 0x7fcfcc06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.358+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fcfcc01f7d0 0x7fcfcc06f910 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.361+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== osd_op_reply(6 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.361+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:7 41.18 41:18000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfbc017910 2024-09-17T04:08:55.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.361+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 3 ==== osd_op_reply(7 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.361+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:8 41.38 41:1c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.362+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== osd_op_reply(8 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.362+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:9 41.4 41:20000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.363+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 3 ==== osd_op_reply(9 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.363+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fcfc406c3a0 0x7fcfc406bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.363+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:10 41.24 41:24000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfc406c3a0 2024-09-17T04:08:55.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.364+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fcfc406c3a0 0x7fcfc406bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.364+0000 7fcfcbe00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fcfc406c3a0 0x7fcfc406bd70 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.367+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== osd_op_reply(10 [pgnls start_epoch 273 out=49b] v272'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.367+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:11 41.14 41:28000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfbc017910 2024-09-17T04:08:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.368+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 4 ==== osd_op_reply(11 [pgnls start_epoch 273 out=49b] v272'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.368+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:12 41.34 41:2c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfc406c3a0 2024-09-17T04:08:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.368+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== osd_op_reply(12 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.368 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.368+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:13 41.c 41:30000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfc4063100 2024-09-17T04:08:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.369+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== osd_op_reply(13 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc4063100 2024-09-17T04:08:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.369+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:14 41.2c 41:34000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfbc017910 2024-09-17T04:08:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.369+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 5 ==== osd_op_reply(14 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.369 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.370+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:15 41.1c 41:38000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.370+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== osd_op_reply(15 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.370+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fcfcc00c2b0 0x7fcfcc0608c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.370+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:16 41.3c 41:3c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.370+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fcfcc00c2b0 0x7fcfcc0608c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.371+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fcfcc00c2b0 0x7fcfcc0608c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.371+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== osd_op_reply(16 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.371+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:17 41.2 41:40000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc022e70 2024-09-17T04:08:55.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.372+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== osd_op_reply(17 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc022e70 2024-09-17T04:08:55.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.372+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:18 41.22 41:44000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.372+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 4 ==== osd_op_reply(18 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.372+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:19 41.12 41:48000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.373+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 5 ==== osd_op_reply(19 [pgnls start_epoch 273 out=49b] v273'18 uv16 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.373+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:20 41.32 41:4c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.373+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 3 ==== osd_op_reply(20 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.373+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:21 41.a 41:50000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.374+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 4 ==== osd_op_reply(21 [pgnls start_epoch 273 out=49b] v272'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.374+0000 7fcfd4e00640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fcfcc07d040 0x7fcfcc0893f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.374+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:22 41.2a 41:54000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc07d040 2024-09-17T04:08:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.374+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fcfcc07d040 0x7fcfcc0893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.374+0000 7fcfd4400640 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fcfcc07d040 0x7fcfcc0893f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.378+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== osd_op_reply(22 [pgnls start_epoch 273 out=49b] v272'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc07d040 2024-09-17T04:08:55.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.378+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:23 41.1a 41:58000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.378 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.378+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 5 ==== osd_op_reply(23 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.378+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:24 41.3a 41:5c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.379+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== osd_op_reply(24 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.379+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:25 41.6 41:60000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x55c29c01adb0 2024-09-17T04:08:55.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 6 ==== osd_op_reply(25 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:26 41.26 41:64000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x55c29bf6e9b0 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 6 ==== osd_op_reply(26 [pgnls start_epoch 273 out=49b] v272'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:27 41.16 41:68000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 7 ==== osd_op_reply(27 [pgnls start_epoch 273 out=49b] v272'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.380+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:28 41.36 41:6c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc022e70 2024-09-17T04:08:55.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.381+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.7 v2:172.21.15.181:6826/580738384 3 ==== osd_op_reply(28 [pgnls start_epoch 273 out=49b] v272'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc022e70 2024-09-17T04:08:55.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.381+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:29 41.e 41:70000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.381+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 6 ==== osd_op_reply(29 [pgnls start_epoch 273 out=49b] v272'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.382+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- osd_op(unknown.0.0:30 41.2e 41:74000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc022e70 2024-09-17T04:08:55.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.382+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.7 v2:172.21.15.181:6826/580738384 4 ==== osd_op_reply(30 [pgnls start_epoch 273 out=49b] v272'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc022e70 2024-09-17T04:08:55.382 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.382+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:31 41.1e 41:78000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.383+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 7 ==== osd_op_reply(31 [pgnls start_epoch 273 out=49b] v272'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.383+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:32 41.3e 41:7c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.383+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 7 ==== osd_op_reply(32 [pgnls start_epoch 273 out=49b] v272'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.383+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:33 41.1 41:80000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.384+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 3 ==== osd_op_reply(33 [pgnls start_epoch 273 out=49b] v272'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.384+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:34 41.21 41:84000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.384+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 8 ==== osd_op_reply(34 [pgnls start_epoch 273 out=49b] v272'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.384+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:35 41.11 41:88000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.385+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 8 ==== osd_op_reply(35 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.385+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:36 41.31 41:8c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.385+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 8 ==== osd_op_reply(36 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.385+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:37 41.9 41:90000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 3 ==== osd_op_reply(37 [pgnls start_epoch 273 out=49b] v272'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:38 41.29 41:94000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc4063100 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 3 ==== osd_op_reply(38 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc4063100 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:39 41.19 41:98000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 4 ==== osd_op_reply(39 [pgnls start_epoch 273 out=49b] v272'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.386+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:40 41.39 41:9c000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.387+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 9 ==== osd_op_reply(40 [pgnls start_epoch 273 out=49b] v272'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.387+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:41 41.5 41:a0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc07d040 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.387+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== osd_op_reply(41 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc07d040 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.387+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:42 41.25 41:a4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc4063100 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.387+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 4 ==== osd_op_reply(42 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc4063100 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.388+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:43 41.15 41:a8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.388+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 5 ==== osd_op_reply(43 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.388+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:44 41.35 41:ac000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 9 ==== osd_op_reply(44 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.388 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:45 41.d 41:b0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 6 ==== osd_op_reply(45 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:46 41.2d 41:b4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 7 ==== osd_op_reply(46 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.389 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.389+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:47 41.1d 41:b8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.390+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 10 ==== osd_op_reply(47 [pgnls start_epoch 273 out=49b] v273'22 uv20 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.390+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:48 41.3d 41:bc000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.390+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 10 ==== osd_op_reply(48 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.391+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:49 41.3 41:c0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.391+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 4 ==== osd_op_reply(49 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.391+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:50 41.23 41:c4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.391+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 11 ==== osd_op_reply(50 [pgnls start_epoch 273 out=49b] v272'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.391+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:51 41.13 41:c8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.392+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 12 ==== osd_op_reply(51 [pgnls start_epoch 273 out=49b] v272'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.392+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:52 41.33 41:cc000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc07d040 2024-09-17T04:08:55.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.2 v2:172.21.15.129:6809/283507149 3 ==== osd_op_reply(52 [pgnls start_epoch 273 out=49b] v272'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc07d040 2024-09-17T04:08:55.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:53 41.b 41:d0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 5 ==== osd_op_reply(53 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- osd_op(unknown.0.0:54 41.2b 41:d4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.3 v2:172.21.15.129:6808/2225231477 6 ==== osd_op_reply(54 [pgnls start_epoch 273 out=49b] v272'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfcc00c2b0 2024-09-17T04:08:55.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.393+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:55 41.1b 41:d8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.394+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 8 ==== osd_op_reply(55 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.394+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:56 41.3b 41:dc000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.394+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 11 ==== osd_op_reply(56 [pgnls start_epoch 273 out=49b] v272'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.394+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- osd_op(unknown.0.0:57 41.7 41:e0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.395+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.6 v2:172.21.15.181:6810/4136500368 9 ==== osd_op_reply(57 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc01f7d0 2024-09-17T04:08:55.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.395+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:58 41.27 41:e4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc07d040 2024-09-17T04:08:55.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.395+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.2 v2:172.21.15.129:6809/283507149 4 ==== osd_op_reply(58 [pgnls start_epoch 273 out=49b] v272'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc07d040 2024-09-17T04:08:55.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.395+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- osd_op(unknown.0.0:59 41.17 41:e8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x55c29c01adb0 2024-09-17T04:08:55.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.4 v2:172.21.15.181:6802/2229307524 12 ==== osd_op_reply(59 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x55c29c01adb0 2024-09-17T04:08:55.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:60 41.37 41:ec000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc4063100 2024-09-17T04:08:55.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 5 ==== osd_op_reply(60 [pgnls start_epoch 273 out=49b] v272'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc4063100 2024-09-17T04:08:55.396 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- osd_op(unknown.0.0:61 41.f 41:f0000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfcc07d040 2024-09-17T04:08:55.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 <== osd.2 v2:172.21.15.129:6809/283507149 5 ==== osd_op_reply(61 [pgnls start_epoch 273 out=49b] v272'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfc402c070 con 0x7fcfcc07d040 2024-09-17T04:08:55.397 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.396+0000 7fcfd4400640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- osd_op(unknown.0.0:62 41.2f 41:f4000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfbc017910 2024-09-17T04:08:55.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.397+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 <== osd.5 v2:172.21.15.181:6818/3094080342 13 ==== osd_op_reply(62 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfcc00b040 con 0x7fcfbc017910 2024-09-17T04:08:55.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.397+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- osd_op(unknown.0.0:63 41.1f 41:f8000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc4063100 2024-09-17T04:08:55.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.397+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.1 v2:172.21.15.129:6800/4215860840 6 ==== osd_op_reply(63 [pgnls start_epoch 273 out=49b] v272'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc4063100 2024-09-17T04:08:55.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.397+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- osd_op(unknown.0.0:64 41.3f 41:fc000000::::head [pgnls start_epoch 273 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e273) -- 0x7fcfc4077040 con 0x7fcfc406c3a0 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfcbe00640 1 -- 172.21.15.129:0/3687654358 <== osd.0 v2:172.21.15.129:6824/1655624368 9 ==== osd_op_reply(64 [pgnls start_epoch 273 out=49b] v272'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7fcfbc00ad00 con 0x7fcfc406c3a0 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fcfc406c3a0 msgr2=0x7fcfc406bd70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7fcfc406c3a0 0x7fcfc406bd70 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fcfc4063100 msgr2=0x7fcfc4061b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fcfc4063100 0x7fcfc4061b70 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fcfcc07d040 msgr2=0x7fcfcc0893f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fcfcc07d040 0x7fcfcc0893f0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fcfcc00c2b0 msgr2=0x7fcfcc0608c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7fcfcc00c2b0 0x7fcfcc0608c0 crc :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x55c29c01adb0 msgr2=0x55c29c01b1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x55c29c01adb0 0x55c29c01b1c0 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 reap_dead start 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfbc017910 msgr2=0x7fcfbc0083f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.399 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfbc017910 0x7fcfbc0083f0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 reap_dead start 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fcfcc01f7d0 msgr2=0x7fcfcc06f910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fcfcc01f7d0 0x7fcfcc06f910 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fcfcc022e70 msgr2=0x7fcfcc055410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7fcfcc022e70 0x7fcfcc055410 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfd4e00640 1 -- 172.21.15.129:0/3687654358 reap_dead start 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 msgr2=0x7fcfb0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 0x7fcfb0079ab0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x55c29bf73040 tx=0x7fcfbc0058c0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 msgr2=0x55c29bf6e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400feb0 tx=0x7fcfc400fee0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 shutdown_connections 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb00775a0 0x7fcfb0079ab0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.400 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 --2- 172.21.15.129:0/3687654358 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x55c29c03d360 0x55c29bf6e470 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 >> 172.21.15.129:0/3687654358 conn(0x55c29be8eac0 msgr2=0x55c29bfb8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:55.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 shutdown_connections 2024-09-17T04:08:55.401 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.398+0000 7fcfdb218f40 1 -- 172.21.15.129:0/3687654358 wait complete. 2024-09-17T04:08:55.409 DEBUG:tasks.cephfs.filesystem:All objects for ino 1099511627778 size 8388608 are absent 2024-09-17T04:08:55.410 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-17T04:08:55.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:55.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 -- 172.21.15.129:0/3344111999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80b4140 msgr2=0x7f90b80ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 --2- 172.21.15.129:0/3344111999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80b4140 0x7f90b80ccc80 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7f90a4009830 tx=0x7f90a402a510 comp rx=0 tx=0).stop 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 -- 172.21.15.129:0/3344111999 shutdown_connections 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 --2- 172.21.15.129:0/3344111999 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80cd1c0 0x7f90b81609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 --2- 172.21.15.129:0/3344111999 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80b4140 0x7f90b80ccc80 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 -- 172.21.15.129:0/3344111999 >> 172.21.15.129:0/3344111999 conn(0x7f90b806d410 msgr2=0x7f90b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 -- 172.21.15.129:0/3344111999 shutdown_connections 2024-09-17T04:08:55.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.631+0000 7f90c0400640 1 -- 172.21.15.129:0/3344111999 wait complete. 2024-09-17T04:08:55.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.633+0000 7f90c0400640 1 Processor -- start 2024-09-17T04:08:55.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.633+0000 7f90c0400640 1 -- start start 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80705a0 0x7f90b81f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90b80b4140 0x7f90b80c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8160260 con 0x7f90b80cd1c0 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f90b81600e0 con 0x7f90b80b4140 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90c0400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f90b8160f00 con 0x7f90b80705a0 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90be400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90be400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51386/0 (socket says 172.21.15.129:51386) 2024-09-17T04:08:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90be400640 1 -- 172.21.15.129:0/4138286258 learned_addr learned my addr 172.21.15.129:0/4138286258 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90bd000640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90b80b4140 0x7f90b80c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.635+0000 7f90bda00640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80705a0 0x7f90b81f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 -- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80705a0 msgr2=0x7f90b81f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80705a0 0x7f90b81f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 -- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90b80b4140 msgr2=0x7f90b80c8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90b80b4140 0x7f90b80c8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 -- 172.21.15.129:0/4138286258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b80c6050 con 0x7f90b80cd1c0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90be400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e9e0 tx=0x7f90b000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000cdb0 con 0x7f90b80cd1c0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90b0004c10 con 0x7f90b80cd1c0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b80c6330 con 0x7f90b80cd1c0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b0005020 con 0x7f90b80cd1c0 2024-09-17T04:08:55.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.636+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b80c68f0 con 0x7f90b80cd1c0 2024-09-17T04:08:55.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.637+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b80ccc80 con 0x7f90b80cd1c0 2024-09-17T04:08:55.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.638+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f90b00051c0 con 0x7f90b80cd1c0 2024-09-17T04:08:55.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.640+0000 7f90b5600640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 0x7f909807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:55.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.640+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7f90b009bb20 con 0x7f90b80cd1c0 2024-09-17T04:08:55.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.640+0000 7f90bda00640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 0x7f909807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:55.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.641+0000 7f90bda00640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 0x7f909807b0d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f90b8072c70 tx=0x7f90ac009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:55.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.645+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f90b0063fa0 con 0x7f90b80cd1c0 2024-09-17T04:08:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.862+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90b806a140 con 0x7f90b80cd1c0 2024-09-17T04:08:55.862 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.863+0000 7f90b5600640 1 -- 172.21.15.129:0/4138286258 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7f90b00638a0 con 0x7f90b80cd1c0 2024-09-17T04:08:55.863 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:55.863 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:55.863 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:08:55.865 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 msgr2=0x7f909807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 0x7f909807b0d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f90b8072c70 tx=0x7f90ac009040 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 msgr2=0x7f90b80c9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 secure :-1 s=READY pgs=2993 cs=0 l=1 rev1=1 crypto rx=0x7f90b000e9e0 tx=0x7f90b000eea0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 shutdown_connections 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9098078bc0 0x7f909807b0d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f90b80cd1c0 0x7f90b80c9480 unknown :-1 s=CLOSED pgs=2993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f90b80b4140 0x7f90b80c8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 --2- 172.21.15.129:0/4138286258 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f90b80705a0 0x7f90b81f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 >> 172.21.15.129:0/4138286258 conn(0x7f90b806d410 msgr2=0x7f90b80b1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 shutdown_connections 2024-09-17T04:08:55.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:55.865+0000 7f90c0400640 1 -- 172.21.15.129:0/4138286258 wait complete. 2024-09-17T04:08:55.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump mds_cache strays_enqueued 2024-09-17T04:08:56.131 INFO:teuthology.orchestra.run.smithi129.stdout:{"mds_cache":{"strays_enqueued":2}} 2024-09-17T04:08:56.131 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_cache": { "strays_enqueued": 2 } } 2024-09-17T04:08:56.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 -- 172.21.15.129:0/349489786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0d4490 msgr2=0x7f47dc0d48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 --2- 172.21.15.129:0/349489786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0d4490 0x7f47dc0d48a0 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0ba6f0 tx=0x7f47d402aa30 comp rx=0 tx=0).stop 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 -- 172.21.15.129:0/349489786 shutdown_connections 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 --2- 172.21.15.129:0/349489786 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0d4490 0x7f47dc0d48a0 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 --2- 172.21.15.129:0/349489786 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0b9af0 0x7f47dc0d3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 --2- 172.21.15.129:0/349489786 >> v2:172.21.15.181:3300/0 conn(0x7f47dc0d1850 0x7f47dc0b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.355 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.354+0000 7f47e0e00640 1 -- 172.21.15.129:0/349489786 >> 172.21.15.129:0/349489786 conn(0x7f47dc08bff0 msgr2=0x7f47dc09d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:56.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.355+0000 7f47e0e00640 1 -- 172.21.15.129:0/349489786 shutdown_connections 2024-09-17T04:08:56.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.355+0000 7f47e0e00640 1 -- 172.21.15.129:0/349489786 wait complete. 2024-09-17T04:08:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.357+0000 7f47e0e00640 1 Processor -- start 2024-09-17T04:08:56.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.357+0000 7f47e0e00640 1 -- start start 2024-09-17T04:08:56.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:56.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0d1850 0x7f47dc09f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:56.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0d4490 0x7f47dc0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:56.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc162a50 con 0x7f47dc0b9af0 2024-09-17T04:08:56.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f47dc1628d0 con 0x7f47dc0d1850 2024-09-17T04:08:56.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47e0e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f47dc162bd0 con 0x7f47dc0d4490 2024-09-17T04:08:56.361 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47db400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.358+0000 7f47db400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51400/0 (socket says 172.21.15.129:51400) 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 -- 172.21.15.129:0/2287426485 learned_addr learned my addr 172.21.15.129:0/2287426485 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47dbe00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0d4490 0x7f47dc0ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47daa00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0d1850 0x7f47dc09f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 -- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0d4490 msgr2=0x7f47dc0ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0d4490 0x7f47dc0ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 -- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0d1850 msgr2=0x7f47dc09f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0d1850 0x7f47dc09f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 -- 172.21.15.129:0/2287426485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc09edb0 con 0x7f47dc0b9af0 2024-09-17T04:08:56.362 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47db400640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00d700 tx=0x7f47cc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:56.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc017070 con 0x7f47dc0b9af0 2024-09-17T04:08:56.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f47cc004310 con 0x7f47dc0b9af0 2024-09-17T04:08:56.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.359+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47dc1f96c0 con 0x7f47dc0b9af0 2024-09-17T04:08:56.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.360+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc009a80 con 0x7f47dc0b9af0 2024-09-17T04:08:56.363 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.360+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47dc1f9c80 con 0x7f47dc0b9af0 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.361+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47dc0b3440 con 0x7f47dc0b9af0 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.362+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f47cc009c20 con 0x7f47dc0b9af0 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.363+0000 7f47d3400640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.363+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7f47cc09aeb0 con 0x7f47dc0b9af0 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.364+0000 7f47daa00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:56.364 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.364+0000 7f47daa00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0a23e0 tx=0x7f47c8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:56.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.367+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f47cc063210 con 0x7f47dc0b9af0 2024-09-17T04:08:56.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.584+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47dc0b4300 con 0x7f47dc0b9af0 2024-09-17T04:08:56.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.585+0000 7f47d3400640 1 -- 172.21.15.129:0/2287426485 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7f47cc062b10 con 0x7f47dc0b9af0 2024-09-17T04:08:56.585 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:56.585 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:08:56.585 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:08:56.587 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 msgr2=0x7f47bc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0a23e0 tx=0x7f47c8005f90 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 msgr2=0x7f47dc09e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f47cc00d700 tx=0x7f47cc00dbc0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 shutdown_connections 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f47bc078b90 0x7f47bc07b0a0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f47dc0d4490 0x7f47dc0ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f47dc0d1850 0x7f47dc09f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 --2- 172.21.15.129:0/2287426485 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f47dc0b9af0 0x7f47dc09e870 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 >> 172.21.15.129:0/2287426485 conn(0x7f47dc08bff0 msgr2=0x7f47dc0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 shutdown_connections 2024-09-17T04:08:56.588 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:56.587+0000 7f47e0e00640 1 -- 172.21.15.129:0/2287426485 wait complete. 2024-09-17T04:08:56.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump purge_queue pq_executed 2024-09-17T04:08:56.851 INFO:teuthology.orchestra.run.smithi129.stdout:{"purge_queue":{"pq_executed":2}} 2024-09-17T04:08:56.851 DEBUG:tasks.cephfs.filesystem:_json_asok output { "purge_queue": { "pq_executed": 2 } } 2024-09-17T04:08:56.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 -- 172.21.15.129:0/1931880530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 msgr2=0x7f9924003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 --2- 172.21.15.129:0/1931880530 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f9924003ee0 secure :-1 s=READY pgs=2997 cs=0 l=1 rev1=1 crypto rx=0x7f9930009b40 tx=0x7f9924005830 comp rx=0 tx=0).stop 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9944e00640 1 -- 172.21.15.129:0/1931880530 reap_dead start 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 -- 172.21.15.129:0/1931880530 shutdown_connections 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 -- 172.21.15.129:0/1931880530 >> 172.21.15.129:0/1931880530 conn(0x7f994009d8f0 msgr2=0x7f994008bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 -- 172.21.15.129:0/1931880530 shutdown_connections 2024-09-17T04:08:57.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.074+0000 7f9945800640 1 -- 172.21.15.129:0/1931880530 wait complete. 2024-09-17T04:08:57.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.076+0000 7f9945800640 1 Processor -- start 2024-09-17T04:08:57.076 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.076+0000 7f9945800640 1 -- start start 2024-09-17T04:08:57.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99400cc290 0x7f99401605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99400ce5a0 0x7f9940160b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f99400b6d20 con 0x7f9924003a70 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f99400b6ba0 con 0x7f99400cc290 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9945800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f99400b6ea0 con 0x7f99400ce5a0 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51432/0 (socket says 172.21.15.129:51432) 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 -- 172.21.15.129:0/99096897 learned_addr learned my addr 172.21.15.129:0/99096897 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9937400640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99400cc290 0x7f99401605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f9944e00640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99400ce5a0 0x7f9940160b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 -- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99400ce5a0 msgr2=0x7f9940160b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99400ce5a0 0x7f9940160b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.079 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 -- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99400cc290 msgr2=0x7f99401605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:57.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99400cc290 0x7f99401605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.078+0000 7f993fe00640 1 -- 172.21.15.129:0/99096897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994015f2c0 con 0x7f9924003a70 2024-09-17T04:08:57.080 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f993fe00640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f9930036040 tx=0x7f9930033670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930043070 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9930004120 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930038040 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99401e8c60 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.079+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99400cba90 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.080+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99400cf5d0 con 0x7f9924003a70 2024-09-17T04:08:57.081 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.081+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9930033860 con 0x7f9924003a70 2024-09-17T04:08:57.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.082+0000 7f993cc00640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 0x7f991c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:08:57.082 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.082+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7f99300bb3a0 con 0x7f9924003a70 2024-09-17T04:08:57.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.083+0000 7f9937400640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 0x7f991c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:08:57.083 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.084+0000 7f9937400640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 0x7f991c07b0a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f99400cebb0 tx=0x7f9928000ff0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:08:57.086 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.087+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f994015f2c0 con 0x7f9924003a70 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.343+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7f99400d50b0 con 0x7f9924003a70 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.344+0000 7f993cc00640 1 -- 172.21.15.129:0/99096897 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1694 (secure 0 0 0) 0x7f9930083140 con 0x7f9924003a70 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:08:57.344 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772646498304, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 447614976, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 447614976, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00057899154489859939, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:08:57.345 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772646498304, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 447614976, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 447614976, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00057899154489859939 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T04:08:57.346 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6184397964025266e-06, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366970109952 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 40, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 63604, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 24, 2024-09-17T04:08:57.347 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 192, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 196608, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 2.6788006834976841e-07, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366970109952 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 41, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 8388608, 2024-09-17T04:08:57.348 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 4, 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 16384, 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 16777216, 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 2.2858581360196695e-05, 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366970109952 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:08:57.349 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:08:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 msgr2=0x7f991c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 0x7f991c07b0a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f99400cebb0 tx=0x7f9928000ff0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 msgr2=0x7f994015ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:08:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 secure :-1 s=READY pgs=2998 cs=0 l=1 rev1=1 crypto rx=0x7f9930036040 tx=0x7f9930033670 comp rx=0 tx=0).stop 2024-09-17T04:08:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 shutdown_connections 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f991c078b90 0x7f991c07b0a0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f99400ce5a0 0x7f9940160b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f99400cc290 0x7f99401605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 --2- 172.21.15.129:0/99096897 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9924003a70 0x7f994015ed80 unknown :-1 s=CLOSED pgs=2998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 >> 172.21.15.129:0/99096897 conn(0x7f994009d8f0 msgr2=0x7f9940161b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 shutdown_connections 2024-09-17T04:08:57.351 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:08:57.347+0000 7f9945800640 1 -- 172.21.15.129:0/99096897 wait complete. 2024-09-17T04:08:57.370 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=60 retry_count=0)... 2024-09-17T04:09:02.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2024-09-17T04:09:02.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 -- 172.21.15.129:0/1415080353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 msgr2=0x7efec4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 --2- 172.21.15.129:0/1415080353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efec4004c10 secure :-1 s=READY pgs=3000 cs=0 l=1 rev1=1 crypto rx=0x7efec8009b50 tx=0x7efec4005830 comp rx=0 tx=0).stop 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 -- 172.21.15.129:0/1415080353 shutdown_connections 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 --2- 172.21.15.129:0/1415080353 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efec4004c10 unknown :-1 s=CLOSED pgs=3000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 -- 172.21.15.129:0/1415080353 >> 172.21.15.129:0/1415080353 conn(0x7efee006d410 msgr2=0x7efee00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 -- 172.21.15.129:0/1415080353 shutdown_connections 2024-09-17T04:09:02.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.603+0000 7efee7a00640 1 -- 172.21.15.129:0/1415080353 wait complete. 2024-09-17T04:09:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.605+0000 7efee7a00640 1 Processor -- start 2024-09-17T04:09:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.605+0000 7efee7a00640 1 -- start start 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 0x7efee01f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 0x7efee01ec150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efee00ce170 con 0x7efec4004800 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efee00afa60 con 0x7efee00cf930 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee7a00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efee00ce2f0 con 0x7efee0155940 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51452/0 (socket says 172.21.15.129:51452) 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee5000640 1 -- 172.21.15.129:0/3326915413 learned_addr learned my addr 172.21.15.129:0/3326915413 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efee5a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 0x7efee01ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.607+0000 7efed7e00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 0x7efee01f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 -- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 msgr2=0x7efee01ec150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 0x7efee01ec150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 -- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 msgr2=0x7efee01f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 0x7efee01f6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 -- 172.21.15.129:0/3326915413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee00b2010 con 0x7efec4004800 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efed7e00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 0x7efee01f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 0x7efee01ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:02.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee5000640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7efed0004de0 tx=0x7efed0001630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:02.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed00107d0 con 0x7efec4004800 2024-09-17T04:09:02.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efed0016070 con 0x7efec4004800 2024-09-17T04:09:02.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed0010ce0 con 0x7efec4004800 2024-09-17T04:09:02.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee01f9e00 con 0x7efec4004800 2024-09-17T04:09:02.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.608+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee0155e60 con 0x7efec4004800 2024-09-17T04:09:02.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.609+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee01f9c20 con 0x7efec4004800 2024-09-17T04:09:02.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.610+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efed001b460 con 0x7efec4004800 2024-09-17T04:09:02.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.611+0000 7efedd600640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 0x7efebc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:02.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.612+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7efed009b6a0 con 0x7efec4004800 2024-09-17T04:09:02.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.612+0000 7efed7e00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 0x7efebc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:02.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.613+0000 7efed7e00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 0x7efebc07b0a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7efec8035000 tx=0x7efec80046c0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:02.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.616+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efee01f9c20 con 0x7efec4004800 2024-09-17T04:09:02.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.870+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) -- 0x7efee015dd40 con 0x7efec4004800 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.871+0000 7efedd600640 1 -- 172.21.15.129:0/3326915413 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) ==== 75+0+1656 (secure 0 0 0) 0x7efed0063a00 con 0x7efec4004800 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:09:02.870 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772663132160, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 430981120, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 430981120, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00055747560691088438, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "num_osds": 8, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_osds": 8, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "num_per_pool_omap_osds": 8 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "stats_by_class": { 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "ssd": { 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_bytes": 773094113280, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_avail_bytes": 772663132160, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_bytes": 430981120, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_bytes": 430981120, 2024-09-17T04:09:02.871 INFO:teuthology.orchestra.run.smithi129.stdout: "total_used_raw_ratio": 0.00055747560691088438 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "pools": [ 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "name": ".mgr", 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 1, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 590368, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 2, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 1160, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 1187840, 2024-09-17T04:09:02.872 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 1.6183662410185207e-06, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366986788864 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_metadata", 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 40, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 67672, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 24, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 200, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 204800, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 2.7902905230803299e-07, 2024-09-17T04:09:02.873 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366986788864 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "name": "cephfs_data", 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "id": 41, 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "stats": { 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "stored": 0, 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "objects": 0, 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "kb_used": 0, 2024-09-17T04:09:02.874 INFO:teuthology.orchestra.run.smithi129.stdout: "bytes_used": 0, 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout: "percent_used": 0, 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout: "max_avail": 366986788864 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-17T04:09:02.875 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.873+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 msgr2=0x7efebc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.873+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 0x7efebc07b0a0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7efec8035000 tx=0x7efec80046c0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.873+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 msgr2=0x7efee01f6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.873+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 secure :-1 s=READY pgs=3001 cs=0 l=1 rev1=1 crypto rx=0x7efed0004de0 tx=0x7efed0001630 comp rx=0 tx=0).stop 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 shutdown_connections 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efebc078b90 0x7efebc07b0a0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efee0155940 0x7efee01ec150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efee00cf930 0x7efee01f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 --2- 172.21.15.129:0/3326915413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efec4004800 0x7efee01f6210 unknown :-1 s=CLOSED pgs=3001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:02.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 >> 172.21.15.129:0/3326915413 conn(0x7efee006d410 msgr2=0x7efee00d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:02.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 shutdown_connections 2024-09-17T04:09:02.877 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:02.874+0000 7efee7a00640 1 -- 172.21.15.129:0/3326915413 wait complete. 2024-09-17T04:09:02.899 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-17T04:09:02.900 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-17T04:09:02.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:09:02.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-17T04:09:03.074 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-17T04:09:03.074 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:09:03.075 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:09:03.075 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:03.110 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:09:03.111 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:03.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 -- 172.21.15.129:0/1839467581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b2950 msgr2=0x7f23500b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 --2- 172.21.15.129:0/1839467581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b2950 0x7f23500b3480 secure :-1 s=READY pgs=3003 cs=0 l=1 rev1=1 crypto rx=0x7f235006dcc0 tx=0x7f233c02a530 comp rx=0 tx=0).stop 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 -- 172.21.15.129:0/1839467581 shutdown_connections 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 --2- 172.21.15.129:0/1839467581 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b2950 0x7f23500b3480 unknown :-1 s=CLOSED pgs=3003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 --2- 172.21.15.129:0/1839467581 >> v2:172.21.15.181:3300/0 conn(0x7f23500ce170 0x7f23500cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.448 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 -- 172.21.15.129:0/1839467581 >> 172.21.15.129:0/1839467581 conn(0x7f235006d410 msgr2=0x7f23500b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.448+0000 7f2355600640 1 -- 172.21.15.129:0/1839467581 shutdown_connections 2024-09-17T04:09:03.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.449+0000 7f2355600640 1 -- 172.21.15.129:0/1839467581 wait complete. 2024-09-17T04:09:03.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.451+0000 7f2355600640 1 Processor -- start 2024-09-17T04:09:03.450 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.451+0000 7f2355600640 1 -- start start 2024-09-17T04:09:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23500b2950 0x7f235015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23500ce170 0x7f23501595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f2350070850 con 0x7f23500b39c0 2024-09-17T04:09:03.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f23500706d0 con 0x7f23500b2950 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f2355600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f235006f5e0 con 0x7f23500ce170 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f234e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f234e000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:51466/0 (socket says 172.21.15.129:51466) 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.452+0000 7f234e000640 1 -- 172.21.15.129:0/1926646818 learned_addr learned my addr 172.21.15.129:0/1926646818 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234ea00640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23500b2950 0x7f235015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234f400640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23500ce170 0x7f23501595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:03.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 -- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23500ce170 msgr2=0x7f23501595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:03.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23500ce170 0x7f23501595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 -- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23500b2950 msgr2=0x7f235015c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:03.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23500b2950 0x7f235015c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 -- 172.21.15.129:0/1926646818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23501e8520 con 0x7f23500b39c0 2024-09-17T04:09:03.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f234e000640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f233c02ab20 tx=0x7f233c033c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.453+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c005c90 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.454+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f233c031e50 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.454+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c005240 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.454+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23501e8790 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.454+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235015c8d0 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.455+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235006fa90 con 0x7f23500b39c0 2024-09-17T04:09:03.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.456+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f233c0053e0 con 0x7f23500b39c0 2024-09-17T04:09:03.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.457+0000 7f2346a00640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 0x7f233007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:03.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.457+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(273..273 src has 1..273) ==== 5765+0+0 (secure 0 0 0) 0x7f233c0bbdc0 con 0x7f23500b39c0 2024-09-17T04:09:03.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.458+0000 7f234ea00640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 0x7f233007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:03.457 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.459+0000 7f234ea00640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 0x7f233007b0a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2340009b30 tx=0x7f2340005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:03.461 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.462+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f233c03f030 con 0x7f23500b39c0 2024-09-17T04:09:03.654 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.655+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f235015cba0 con 0x7f23500b39c0 2024-09-17T04:09:03.655 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.656+0000 7f2346a00640 1 -- 172.21.15.129:0/1926646818 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v273) ==== 80+0+0 (secure 0 0 0) 0x7f233c031770 con 0x7f23500b39c0 2024-09-17T04:09:03.655 INFO:teuthology.orchestra.run.smithi129.stderr:listed 0 entries 2024-09-17T04:09:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 msgr2=0x7f233007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 0x7f233007b0a0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f2340009b30 tx=0x7f2340005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:03.657 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 msgr2=0x7f23500b1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 secure :-1 s=READY pgs=3004 cs=0 l=1 rev1=1 crypto rx=0x7f233c02ab20 tx=0x7f233c033c20 comp rx=0 tx=0).stop 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 shutdown_connections 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2330078b90 0x7f233007b0a0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f23500ce170 0x7f23501595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f23500b39c0 0x7f23500b1270 unknown :-1 s=CLOSED pgs=3004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 --2- 172.21.15.129:0/1926646818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23500b2950 0x7f235015c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 >> 172.21.15.129:0/1926646818 conn(0x7f235006d410 msgr2=0x7f2350073e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 shutdown_connections 2024-09-17T04:09:03.658 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:03.658+0000 7f2355600640 1 -- 172.21.15.129:0/1926646818 wait complete. 2024-09-17T04:09:03.679 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-17T04:09:03.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:03.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-17T04:09:03.742 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:09:03.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:03.742 DEBUG:teuthology.orchestra.run.smithi129:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-17T04:09:03.778 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:09:03.778 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:09:03.778 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:03.818 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:09:03.818 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:09:03.818 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:09:03.818 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:03.854 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:09:03.854 INFO:tasks.cephfs.mount:Terminating background process 2024-09-17T04:09:03.854 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:09:03.854 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:03.895 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:09:03.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:09:04.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 -- 172.21.15.129:0/2776901503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81524c0 msgr2=0x7f3ef80cf5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 --2- 172.21.15.129:0/2776901503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81524c0 0x7f3ef80cf5c0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f3eec005fd0 tx=0x7f3eec02aaa0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 -- 172.21.15.129:0/2776901503 shutdown_connections 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 --2- 172.21.15.129:0/2776901503 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81528d0 0x7f3ef8072f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 --2- 172.21.15.129:0/2776901503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81524c0 0x7f3ef80cf5c0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 -- 172.21.15.129:0/2776901503 >> 172.21.15.129:0/2776901503 conn(0x7f3ef806cfc0 msgr2=0x7f3ef806d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 -- 172.21.15.129:0/2776901503 shutdown_connections 2024-09-17T04:09:04.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.120+0000 7f3efee00640 1 -- 172.21.15.129:0/2776901503 wait complete. 2024-09-17T04:09:04.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.122+0000 7f3efee00640 1 Processor -- start 2024-09-17T04:09:04.121 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.122+0000 7f3efee00640 1 -- start start 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81524c0 0x7f3ef80b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81528d0 0x7f3ef80cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef80d02c0 con 0x7f3ef80cfb00 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef8073880 con 0x7f3ef81528d0 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efee00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef80d0440 con 0x7f3ef81524c0 2024-09-17T04:09:04.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58108/0 (socket says 172.21.15.129:58108) 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 -- 172.21.15.129:0/262867287 learned_addr learned my addr 172.21.15.129:0/262867287 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efd000640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81524c0 0x7f3ef80b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efe400640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81528d0 0x7f3ef80cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 -- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81524c0 msgr2=0x7f3ef80b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81524c0 0x7f3ef80b25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 -- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81528d0 msgr2=0x7f3ef80cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81528d0 0x7f3ef80cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.124+0000 7f3efda00640 1 -- 172.21.15.129:0/262867287 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef81488a0 con 0x7f3ef80cfb00 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3efda00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009bf0 tx=0x7f3eec003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec044070 con 0x7f3ef80cfb00 2024-09-17T04:09:04.125 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3eec004250 con 0x7f3ef80cfb00 2024-09-17T04:09:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eec004660 con 0x7f3ef80cfb00 2024-09-17T04:09:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef8148b80 con 0x7f3ef80cfb00 2024-09-17T04:09:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.125+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ef8149140 con 0x7f3ef80cfb00 2024-09-17T04:09:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.126+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ef80d08f0 con 0x7f3ef80cfb00 2024-09-17T04:09:04.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.127+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f3eec047070 con 0x7f3ef80cfb00 2024-09-17T04:09:04.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.129+0000 7f3ef5600640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 0x7f3ed807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.129+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7f3eec0bbd10 con 0x7f3ef80cfb00 2024-09-17T04:09:04.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.130+0000 7f3efd000640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 0x7f3ed807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.130+0000 7f3efd000640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 0x7f3ed807b0a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3ef814e9e0 tx=0x7f3ee4009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:04.133 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.134+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f3eec0c1020 con 0x7f3ef80cfb00 2024-09-17T04:09:04.356 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.357+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ef806a140 con 0x7f3ef80cfb00 2024-09-17T04:09:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.357+0000 7f3ef5600640 1 -- 172.21.15.129:0/262867287 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7f3eec032030 con 0x7f3ef80cfb00 2024-09-17T04:09:04.357 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:04.357 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:09:04.357 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:09:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 msgr2=0x7f3ed807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 0x7f3ed807b0a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3ef814e9e0 tx=0x7f3ee4009040 comp rx=0 tx=0).stop 2024-09-17T04:09:04.359 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 msgr2=0x7f3ef81447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 secure :-1 s=READY pgs=3006 cs=0 l=1 rev1=1 crypto rx=0x7f3eec009bf0 tx=0x7f3eec003ce0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 shutdown_connections 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f3ed8078b90 0x7f3ed807b0a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ef81528d0 0x7f3ef80cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f3ef81524c0 0x7f3ef80b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 --2- 172.21.15.129:0/262867287 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f3ef80cfb00 0x7f3ef81447f0 unknown :-1 s=CLOSED pgs=3006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 >> 172.21.15.129:0/262867287 conn(0x7f3ef806cfc0 msgr2=0x7f3ef80b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 shutdown_connections 2024-09-17T04:09:04.360 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.360+0000 7f3efee00640 1 -- 172.21.15.129:0/262867287 wait complete. 2024-09-17T04:09:04.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:09:04.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 -- 172.21.15.129:0/2081338004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 msgr2=0x7f4ca81609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 --2- 172.21.15.129:0/2081338004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca81609c0 secure :-1 s=READY pgs=3007 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0005fd0 tx=0x7f4ca002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 -- 172.21.15.129:0/2081338004 shutdown_connections 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 --2- 172.21.15.129:0/2081338004 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca81609c0 unknown :-1 s=CLOSED pgs=3007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 --2- 172.21.15.129:0/2081338004 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 0x7f4ca80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 -- 172.21.15.129:0/2081338004 >> 172.21.15.129:0/2081338004 conn(0x7f4ca806d410 msgr2=0x7f4ca80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 -- 172.21.15.129:0/2081338004 shutdown_connections 2024-09-17T04:09:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.610+0000 7f4caec00640 1 -- 172.21.15.129:0/2081338004 wait complete. 2024-09-17T04:09:04.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.612+0000 7f4caec00640 1 Processor -- start 2024-09-17T04:09:04.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.613+0000 7f4caec00640 1 -- start start 2024-09-17T04:09:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 0x7f4ca80c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.614 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 0x7f4ca80c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca8160260 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca81600e0 con 0x7f4ca80705a0 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4caec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca8160f00 con 0x7f4ca80b4140 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4cacc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4cacc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58130/0 (socket says 172.21.15.129:58130) 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4cacc00640 1 -- 172.21.15.129:0/1962234629 learned_addr learned my addr 172.21.15.129:0/1962234629 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4ca7400640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 0x7f4ca80c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.614+0000 7f4ca7e00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 0x7f4ca80c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 -- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 msgr2=0x7f4ca80c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 0x7f4ca80c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 -- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 msgr2=0x7f4ca80c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 0x7f4ca80c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 -- 172.21.15.129:0/1962234629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca81f3c60 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4ca7e00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 0x7f4ca80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4cacc00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009230 tx=0x7f4ca0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca003e030 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4ca0003c20 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca81f3f40 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.615+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca0035040 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.616 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.616+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca81f4500 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.617+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca80af4c0 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.617 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.618+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4ca0041030 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.619+0000 7f4c9fe00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 0x7f4c8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:04.618 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.619+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7f4ca00bb3a0 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.620+0000 7f4ca7e00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 0x7f4c8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:04.619 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.621+0000 7f4ca7e00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 0x7f4c8807b0a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f4c98001dc0 tx=0x7f4c98009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:04.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.624+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f4ca80af4c0 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.844+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ca80ccc80 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.845+0000 7f4c9fe00640 1 -- 172.21.15.129:0/1962234629 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7f4ca00830d0 con 0x7f4ca80cd1c0 2024-09-17T04:09:04.844 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:04.844 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:09:04.844 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:09:04.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 msgr2=0x7f4c8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 0x7f4c8807b0a0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f4c98001dc0 tx=0x7f4c98009040 comp rx=0 tx=0).stop 2024-09-17T04:09:04.846 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 msgr2=0x7f4ca815dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 secure :-1 s=READY pgs=3008 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009230 tx=0x7f4ca0003820 comp rx=0 tx=0).stop 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 shutdown_connections 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f4c88078b90 0x7f4c8807b0a0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f4ca80cd1c0 0x7f4ca815dc20 unknown :-1 s=CLOSED pgs=3008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f4ca80b4140 0x7f4ca80c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 --2- 172.21.15.129:0/1962234629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ca80705a0 0x7f4ca80c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 >> 172.21.15.129:0/1962234629 conn(0x7f4ca806d410 msgr2=0x7f4ca80d0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 shutdown_connections 2024-09-17T04:09:04.847 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:04.847+0000 7f4caec00640 1 -- 172.21.15.129:0/1962234629 wait complete. 2024-09-17T04:09:04.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:09:05.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 -- 172.21.15.129:0/1486898967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43100705a0 msgr2=0x7f43100b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 --2- 172.21.15.129:0/1486898967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43100705a0 0x7f43100b4140 secure :-1 s=READY pgs=3010 cs=0 l=1 rev1=1 crypto rx=0x7f4300009830 tx=0x7f430002a510 comp rx=0 tx=0).stop 2024-09-17T04:09:05.090 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 -- 172.21.15.129:0/1486898967 shutdown_connections 2024-09-17T04:09:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 --2- 172.21.15.129:0/1486898967 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006eed0 0x7f43100ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 --2- 172.21.15.129:0/1486898967 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f43100705a0 0x7f43100b4140 unknown :-1 s=CLOSED pgs=3010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 -- 172.21.15.129:0/1486898967 >> 172.21.15.129:0/1486898967 conn(0x7f431006d410 msgr2=0x7f43100b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 -- 172.21.15.129:0/1486898967 shutdown_connections 2024-09-17T04:09:05.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.091+0000 7f4317c00640 1 -- 172.21.15.129:0/1486898967 wait complete. 2024-09-17T04:09:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.093+0000 7f4317c00640 1 Processor -- start 2024-09-17T04:09:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.093+0000 7f4317c00640 1 -- start start 2024-09-17T04:09:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006f2e0 0x7f431015b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43100705a0 0x7f431015bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f4310160200 con 0x7f431006eed0 2024-09-17T04:09:05.095 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f43101612e0 con 0x7f43100705a0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4317c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f43100cd1c0 con 0x7f431006f2e0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58160/0 (socket says 172.21.15.129:58160) 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 -- 172.21.15.129:0/3120342213 learned_addr learned my addr 172.21.15.129:0/3120342213 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4307e00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006f2e0 0x7f431015b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43100705a0 0x7f431015bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 -- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006f2e0 msgr2=0x7f431015b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006f2e0 0x7f431015b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 -- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43100705a0 msgr2=0x7f431015bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43100705a0 0x7f431015bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.095+0000 7f4315200640 1 -- 172.21.15.129:0/3120342213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431015a260 con 0x7f431006eed0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f4315200640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f430002ab00 tx=0x7f4300033c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4300005c90 con 0x7f431006eed0 2024-09-17T04:09:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4300031e50 con 0x7f431006eed0 2024-09-17T04:09:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43100c8370 con 0x7f431006eed0 2024-09-17T04:09:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43000332b0 con 0x7f431006eed0 2024-09-17T04:09:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.096+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43100c8930 con 0x7f431006eed0 2024-09-17T04:09:05.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.097+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43100ccc80 con 0x7f431006eed0 2024-09-17T04:09:05.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.099+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f4300033450 con 0x7f431006eed0 2024-09-17T04:09:05.098 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.100+0000 7f430d600640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 0x7f42ec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.100+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7f43000bc220 con 0x7f431006eed0 2024-09-17T04:09:05.099 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.100+0000 7f4307e00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 0x7f42ec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.101+0000 7f4307e00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 0x7f42ec07b0d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f42f8005fd0 tx=0x7f42f8005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:05.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.104+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f430003f030 con 0x7f431006eed0 2024-09-17T04:09:05.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.292+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f431006a140 con 0x7f431006eed0 2024-09-17T04:09:05.293 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.294+0000 7f430d600640 1 -- 172.21.15.129:0/3120342213 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+16846 (secure 0 0 0) 0x7f4300084630 con 0x7f431006eed0 2024-09-17T04:09:05.293 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:05.293 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":274,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:09:04.026791+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:08:22.575404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":41,"pool_name":"cephfs_data","create_time":"2024-09-17T04:08:23.637824+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":273,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":41,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:09:05.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 msgr2=0x7f42ec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 0x7f42ec07b0d0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f42f8005fd0 tx=0x7f42f8005850 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 msgr2=0x7f4310158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 crypto rx=0x7f430002ab00 tx=0x7f4300033c90 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 shutdown_connections 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f42ec078bc0 0x7f42ec07b0d0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43100705a0 0x7f431015bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f431006f2e0 0x7f431015b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 --2- 172.21.15.129:0/3120342213 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f431006eed0 0x7f4310158a20 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 >> 172.21.15.129:0/3120342213 conn(0x7f431006d410 msgr2=0x7f43100aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 shutdown_connections 2024-09-17T04:09:05.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.296+0000 7f4317c00640 1 -- 172.21.15.129:0/3120342213 wait complete. 2024-09-17T04:09:05.318 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:09:05.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:09:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 -- 172.21.15.129:0/3070798164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc910 msgr2=0x7fa7fc0ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 --2- 172.21.15.129:0/3070798164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0ccd20 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4005fd0 tx=0x7fa7f402aa90 comp rx=0 tx=0).stop 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 -- 172.21.15.129:0/3070798164 shutdown_connections 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 --2- 172.21.15.129:0/3070798164 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0ccd20 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 --2- 172.21.15.129:0/3070798164 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc0cc3e0 0x7fa7fc15f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 -- 172.21.15.129:0/3070798164 >> 172.21.15.129:0/3070798164 conn(0x7fa7fc06d410 msgr2=0x7fa7fc0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 -- 172.21.15.129:0/3070798164 shutdown_connections 2024-09-17T04:09:05.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.540+0000 7fa803600640 1 -- 172.21.15.129:0/3070798164 wait complete. 2024-09-17T04:09:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.542+0000 7fa803600640 1 Processor -- start 2024-09-17T04:09:05.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.542+0000 7fa803600640 1 -- start start 2024-09-17T04:09:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 0x7fa7fc1f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa7fc0b43d0 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa7fc15f7b0 con 0x7fa7fc0cc910 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa803600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa7fc0af070 con 0x7fa7fc070a50 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa7f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa7f3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58166/0 (socket says 172.21.15.129:58166) 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa7f3e00640 1 -- 172.21.15.129:0/2361786868 learned_addr learned my addr 172.21.15.129:0/2361786868 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa801600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.544+0000 7fa800c00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 0x7fa7fc1f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 -- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 msgr2=0x7fa7fc1f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 0x7fa7fc1f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 -- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7fc0cc910 msgr2=0x7fa7fc0b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 -- 172.21.15.129:0/2361786868 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7fc0c7a70 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa800c00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 0x7fa7fc1f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f3e00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400e7c0 tx=0x7fa7e400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e400ccc0 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa7e4010430 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4010860 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7fc0c7d50 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.545+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7fc0c8310 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.546+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7fc0c77e0 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.548+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa7e4010a00 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.548+0000 7fa7f9600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 0x7fa7d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:05.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.549+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7fa7e409b630 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.548 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.549+0000 7fa800c00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 0x7fa7d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:05.549 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.550+0000 7fa800c00640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 0x7fa7d807b0d0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec007fd0 tx=0x7fa7ec009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:05.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.553+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fa7e4063490 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.769+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7fc15f060 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.769+0000 7fa7f9600640 1 -- 172.21.15.129:0/2361786868 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7fa7e406c020 con 0x7fa7fc0cc3e0 2024-09-17T04:09:05.769 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:05.769 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:09:05.769 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 msgr2=0x7fa7d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 0x7fa7d807b0d0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec007fd0 tx=0x7fa7ec009040 comp rx=0 tx=0).stop 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 msgr2=0x7fa7fc1f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7fa7e400e7c0 tx=0x7fa7e400ec80 comp rx=0 tx=0).stop 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 shutdown_connections 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa7d8078bc0 0x7fa7d807b0d0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7fc0cc910 0x7fa7fc0b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa7fc0cc3e0 0x7fa7fc1f6b40 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 --2- 172.21.15.129:0/2361786868 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa7fc070a50 0x7fa7fc1f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 >> 172.21.15.129:0/2361786868 conn(0x7fa7fc06d410 msgr2=0x7fa7fc0af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 shutdown_connections 2024-09-17T04:09:05.772 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:05.772+0000 7fa803600640 1 -- 172.21.15.129:0/2361786868 wait complete. 2024-09-17T04:09:05.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:09:06.017 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 -- 172.21.15.129:0/1092292666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f80ccc80 msgr2=0x7ff4f80cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/1092292666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f80ccc80 0x7ff4f80cd090 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7ff4f0005fd0 tx=0x7ff4f002a820 comp rx=0 tx=0).stop 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 -- 172.21.15.129:0/1092292666 shutdown_connections 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/1092292666 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f80ccc80 0x7ff4f80cd090 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/1092292666 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 0x7ff4f80cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 -- 172.21.15.129:0/1092292666 >> 172.21.15.129:0/1092292666 conn(0x7ff4f806d410 msgr2=0x7ff4f80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 -- 172.21.15.129:0/1092292666 shutdown_connections 2024-09-17T04:09:06.018 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.018+0000 7ff4fcc00640 1 -- 172.21.15.129:0/1092292666 wait complete. 2024-09-17T04:09:06.019 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.020+0000 7ff4fcc00640 1 Processor -- start 2024-09-17T04:09:06.020 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.021+0000 7ff4fcc00640 1 -- start start 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4f80ccc80 0x7ff4f815f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 0x7ff4f80c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8160ba0 con 0x7ff4f8099570 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f8160a20 con 0x7ff4f80ccc80 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4fcc00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f8160d20 con 0x7ff4f80cf2b0 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4f5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.023 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4f5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58188/0 (socket says 172.21.15.129:58188) 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.022+0000 7ff4f5e00640 1 -- 172.21.15.129:0/623994080 learned_addr learned my addr 172.21.15.129:0/623994080 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5400640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4f80ccc80 0x7ff4f815f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f6800640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 0x7ff4f80c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 -- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 msgr2=0x7ff4f80c81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 0x7ff4f80c81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 -- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4f80ccc80 msgr2=0x7ff4f815f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4f80ccc80 0x7ff4f815f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 -- 172.21.15.129:0/623994080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f809c850 con 0x7ff4f8099570 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4f5e00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c3a0 tx=0x7ff4e800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8011790 con 0x7ff4f8099570 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff4e8017070 con 0x7ff4f8099570 2024-09-17T04:09:06.024 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.023+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8011cc0 con 0x7ff4f8099570 2024-09-17T04:09:06.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.024+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f809cb30 con 0x7ff4f8099570 2024-09-17T04:09:06.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.024+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f809d0f0 con 0x7ff4f8099570 2024-09-17T04:09:06.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.025+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f8161240 con 0x7ff4f8099570 2024-09-17T04:09:06.025 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.026+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff4e801c460 con 0x7ff4f8099570 2024-09-17T04:09:06.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.027+0000 7ff4ee000640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 0x7ff4d807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.027+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7ff4e809aa30 con 0x7ff4f8099570 2024-09-17T04:09:06.026 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.028+0000 7ff4f5400640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 0x7ff4d807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.027 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.028+0000 7ff4f5400640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 0x7ff4d807b0d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80cbf10 tx=0x7ff4e4005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.030 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.031+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff4e8062e40 con 0x7ff4f8099570 2024-09-17T04:09:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.250+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4f806a140 con 0x7ff4f8099570 2024-09-17T04:09:06.249 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.250+0000 7ff4ee000640 1 -- 172.21.15.129:0/623994080 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 273 v273) ==== 95+0+5249 (secure 0 0 0) 0x7ff4e8062740 con 0x7ff4f8099570 2024-09-17T04:09:06.250 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:06.250 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":273,"btime":"2024-09-17T04:08:39:242848+0000","default_fscid":20,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266}],"filesystems":[{"mdsmap":{"epoch":273,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-17T04:08:24.669089+0000","modified":"2024-09-17T04:08:38.810975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":12681},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12681":{"gid":12681,"name":"a","rank":0,"incarnation":268,"state":"up:active","state_seq":3,"addr":"172.21.15.129:6835/2485159334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2485159334},{"type":"v1","addr":"172.21.15.129:6835","nonce":2485159334}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[41],"metadata_pool":40,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":12681,"qdb_cluster":[12681]},"id":20}]} 2024-09-17T04:09:06.250 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 273 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 msgr2=0x7ff4d807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 0x7ff4d807b0d0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7ff4f80cbf10 tx=0x7ff4e4005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 msgr2=0x7ff4f815f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7ff4e800c3a0 tx=0x7ff4e800c860 comp rx=0 tx=0).stop 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 shutdown_connections 2024-09-17T04:09:06.252 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff4d8078bc0 0x7ff4d807b0d0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff4f80cf2b0 0x7ff4f80c81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4f80ccc80 0x7ff4f815f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 --2- 172.21.15.129:0/623994080 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff4f8099570 0x7ff4f815f0e0 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 >> 172.21.15.129:0/623994080 conn(0x7ff4f806d410 msgr2=0x7ff4f80d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 shutdown_connections 2024-09-17T04:09:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.253+0000 7ff4fcc00640 1 -- 172.21.15.129:0/623994080 wait complete. 2024-09-17T04:09:06.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:09:06.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.500+0000 7f6783000640 1 -- 172.21.15.129:0/1675662428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 msgr2=0x7f676000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.500+0000 7f6783000640 1 --2- 172.21.15.129:0/1675662428 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f676000b9c0 secure :-1 s=READY pgs=3017 cs=0 l=1 rev1=1 crypto rx=0x7f6774005fd0 tx=0x7f6760005830 comp rx=0 tx=0).stop 2024-09-17T04:09:06.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.500+0000 7f6781000640 1 -- 172.21.15.129:0/1675662428 reap_dead start 2024-09-17T04:09:06.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.500+0000 7f6783000640 1 -- 172.21.15.129:0/1675662428 shutdown_connections 2024-09-17T04:09:06.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.500+0000 7f6783000640 1 -- 172.21.15.129:0/1675662428 >> 172.21.15.129:0/1675662428 conn(0x7f677c06d410 msgr2=0x7f677c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:06.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.501+0000 7f6783000640 1 -- 172.21.15.129:0/1675662428 shutdown_connections 2024-09-17T04:09:06.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.501+0000 7f6783000640 1 -- 172.21.15.129:0/1675662428 wait complete. 2024-09-17T04:09:06.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.503+0000 7f6783000640 1 Processor -- start 2024-09-17T04:09:06.502 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.503+0000 7f6783000640 1 -- start start 2024-09-17T04:09:06.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f677c0c6200 0x7f677c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677c157650 0x7f677c155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f677c070f60 con 0x7f6760005150 2024-09-17T04:09:06.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f677c0cd5a0 con 0x7f677c157650 2024-09-17T04:09:06.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f6783000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f677c0710e0 con 0x7f677c0c6200 2024-09-17T04:09:06.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.504+0000 7f677be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58216/0 (socket says 172.21.15.129:58216) 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 -- 172.21.15.129:0/998519415 learned_addr learned my addr 172.21.15.129:0/998519415 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677b400640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f677c0c6200 0x7f677c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f6781000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677c157650 0x7f677c155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 -- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f677c0c6200 msgr2=0x7f677c157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f677c0c6200 0x7f677c157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 -- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677c157650 msgr2=0x7f677c155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677c157650 0x7f677c155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 -- 172.21.15.129:0/998519415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677c157a60 con 0x7f6760005150 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.505+0000 7f677be00640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f676c00e7f0 tx=0x7f676c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.506+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c00cc60 con 0x7f6760005150 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.506+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f676c00ce20 con 0x7f6760005150 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.506+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010690 con 0x7f6760005150 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.506+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677c0c5a80 con 0x7f6760005150 2024-09-17T04:09:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.506+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677c0af020 con 0x7f6760005150 2024-09-17T04:09:06.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.507+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677c15ae00 con 0x7f6760005150 2024-09-17T04:09:06.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.508+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f676c010830 con 0x7f6760005150 2024-09-17T04:09:06.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.509+0000 7f6773e00640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 0x7f675c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.509+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7f676c09b6a0 con 0x7f6760005150 2024-09-17T04:09:06.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.510+0000 7f677b400640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 0x7f675c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.510+0000 7f677b400640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 0x7f675c07b0a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f677c1678b0 tx=0x7f67680067a0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.513 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.514+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f677c157a60 con 0x7f6760005150 2024-09-17T04:09:06.699 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.700+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f677c06a140 con 0x7f6760005150 2024-09-17T04:09:06.700 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.701+0000 7f6773e00640 1 -- 172.21.15.129:0/998519415 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v274) ==== 74+0+16846 (secure 0 0 0) 0x7f676c063470 con 0x7f6760005150 2024-09-17T04:09:06.701 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:06.701 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":274,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:09:04.026791+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":40,"pool_name":"cephfs_metadata","create_time":"2024-09-17T04:08:22.575404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":41,"pool_name":"cephfs_data","create_time":"2024-09-17T04:08:23.637824+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"273","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":273,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":41,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:09:06.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 msgr2=0x7f675c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 0x7f675c07b0a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f677c1678b0 tx=0x7f67680067a0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 msgr2=0x7f677c0c5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f676c00e7f0 tx=0x7f676c00ecb0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.703 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 shutdown_connections 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f675c078b90 0x7f675c07b0a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f677c157650 0x7f677c155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f677c0c6200 0x7f677c157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 --2- 172.21.15.129:0/998519415 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6760005150 0x7f677c0c5540 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 >> 172.21.15.129:0/998519415 conn(0x7f677c06d410 msgr2=0x7f677c0702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 shutdown_connections 2024-09-17T04:09:06.704 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.704+0000 7f6783000640 1 -- 172.21.15.129:0/998519415 wait complete. 2024-09-17T04:09:06.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-17T04:09:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.952+0000 7fb807800640 1 -- 172.21.15.129:0/3303346470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 msgr2=0x7fb8000b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.952+0000 7fb807800640 1 --2- 172.21.15.129:0/3303346470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b4140 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0005fd0 tx=0x7fb7f002a5d0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 -- 172.21.15.129:0/3303346470 shutdown_connections 2024-09-17T04:09:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 --2- 172.21.15.129:0/3303346470 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8000ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 --2- 172.21.15.129:0/3303346470 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b4140 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 -- 172.21.15.129:0/3303346470 >> 172.21.15.129:0/3303346470 conn(0x7fb80006d410 msgr2=0x7fb8000b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 -- 172.21.15.129:0/3303346470 shutdown_connections 2024-09-17T04:09:06.953 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.953+0000 7fb807800640 1 -- 172.21.15.129:0/3303346470 wait complete. 2024-09-17T04:09:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.955+0000 7fb807800640 1 Processor -- start 2024-09-17T04:09:06.954 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.955+0000 7fb807800640 1 -- start start 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80006f2e0 0x7fb8001f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb800160200 con 0x7fb8000705a0 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8001612e0 con 0x7fb80006f2e0 2024-09-17T04:09:06.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.956+0000 7fb807800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8000cd1c0 con 0x7fb80006eed0 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb804e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb804e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:47602/0 (socket says 172.21.15.129:47602) 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb804e00640 1 -- 172.21.15.129:0/2086509413 learned_addr learned my addr 172.21.15.129:0/2086509413 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58234/0 (socket says 172.21.15.129:58234) 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb7ffe00640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80006f2e0 0x7fb8001f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 -- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 msgr2=0x7fb8001f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.958 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 -- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80006f2e0 msgr2=0x7fb8001f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80006f2e0 0x7fb8001f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb805800640 1 -- 172.21.15.129:0/2086509413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8001f3c30 con 0x7fb8000705a0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.957+0000 7fb804e00640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb805800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8001e10 tx=0x7fb7f8010420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f800e0d0 con 0x7fb8000705a0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb7f800e270 con 0x7fb8000705a0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8001f3f10 con 0x7fb8000705a0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8010d90 con 0x7fb8000705a0 2024-09-17T04:09:06.959 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.958+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8001f44d0 con 0x7fb8000705a0 2024-09-17T04:09:06.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.959+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8000ccc80 con 0x7fb8000705a0 2024-09-17T04:09:06.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.960+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb7f8003970 con 0x7fb8000705a0 2024-09-17T04:09:06.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.962+0000 7fb7fcc00640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 0x7fb7e007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:06.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.962+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(274..274 src has 1..274) ==== 5737+0+0 (secure 0 0 0) 0x7fb7f809ec00 con 0x7fb8000705a0 2024-09-17T04:09:06.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.962+0000 7fb804e00640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 0x7fb7e007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:06.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.963+0000 7fb804e00640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 0x7fb7e007b0a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009230 tx=0x7fb7f0004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:06.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:06.966+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb7f80668b0 con 0x7fb8000705a0 2024-09-17T04:09:07.185 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.186+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb80006a140 con 0x7fb8000705a0 2024-09-17T04:09:07.186 INFO:tasks.ceph.mon.a.smithi129.stderr:2024-09-17T04:09:07.187+0000 7f409f600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-17T04:09:07.188 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.189+0000 7fb7fcc00640 1 -- 172.21.15.129:0/2086509413 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v274) ==== 161+0+0 (secure 0 0 0) 0x7fb7f806b2e0 con 0x7fb8000705a0 2024-09-17T04:09:07.188 INFO:teuthology.orchestra.run.smithi129.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-17T04:09:07.191 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 msgr2=0x7fb7e007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 0x7fb7e007b0a0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009230 tx=0x7fb7f0004730 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 msgr2=0x7fb8000b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 secure :-1 s=READY pgs=3020 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8001e10 tx=0x7fb7f8010420 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 shutdown_connections 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb7e0078b90 0x7fb7e007b0a0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8000705a0 0x7fb8000b1500 unknown :-1 s=CLOSED pgs=3020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb80006f2e0 0x7fb8001f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 --2- 172.21.15.129:0/2086509413 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb80006eed0 0x7fb8001f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.192+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 >> 172.21.15.129:0/2086509413 conn(0x7fb80006d410 msgr2=0x7fb8000735a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.193+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 shutdown_connections 2024-09-17T04:09:07.192 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.193+0000 7fb807800640 1 -- 172.21.15.129:0/2086509413 wait complete. 2024-09-17T04:09:07.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-17T04:09:07.337 INFO:tasks.ceph.mds.a.smithi129.stdout:starting mds.a at 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 -- 172.21.15.129:0/1389608270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 msgr2=0x7fec8c0b2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 --2- 172.21.15.129:0/1389608270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0b2050 secure :-1 s=READY pgs=3024 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0d1260 tx=0x7fec7802a4d0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 -- 172.21.15.129:0/1389608270 shutdown_connections 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 --2- 172.21.15.129:0/1389608270 >> v2:172.21.15.181:3300/0 conn(0x7fec8c0b3010 0x7fec8c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 --2- 172.21.15.129:0/1389608270 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0b2050 unknown :-1 s=CLOSED pgs=3024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.447+0000 7fec94200640 1 -- 172.21.15.129:0/1389608270 >> 172.21.15.129:0/1389608270 conn(0x7fec8c0ccc40 msgr2=0x7fec8c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.448+0000 7fec94200640 1 -- 172.21.15.129:0/1389608270 shutdown_connections 2024-09-17T04:09:07.447 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.448+0000 7fec94200640 1 -- 172.21.15.129:0/1389608270 wait complete. 2024-09-17T04:09:07.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.450+0000 7fec94200640 1 Processor -- start 2024-09-17T04:09:07.449 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.450+0000 7fec94200640 1 -- start start 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec8c074800 0x7fec8c15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 0x7fec8c1f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c0b3ae0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fec8c0b3960 con 0x7fec8c0b3010 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.451+0000 7fec94200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fec8c0b3c60 con 0x7fec8c074800 2024-09-17T04:09:07.452 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58268/0 (socket says 172.21.15.129:58268) 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 -- 172.21.15.129:0/2243263122 learned_addr learned my addr 172.21.15.129:0/2243263122 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec91800640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec8c074800 0x7fec8c15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec92200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 0x7fec8c1f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 -- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec8c074800 msgr2=0x7fec8c15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec8c074800 0x7fec8c15b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 -- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 msgr2=0x7fec8c1f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 0x7fec8c1f6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 -- 172.21.15.129:0/2243263122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8c1e99e0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec92200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 0x7fec8c1f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.452+0000 7fec90e00640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fec780099c0 tx=0x7fec780039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.453+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7803e030 con 0x7fec8c0b1c40 2024-09-17T04:09:07.453 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.453+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fec780047a0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.453+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78035040 con 0x7fec8c0b1c40 2024-09-17T04:09:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.453+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8c09b3e0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.453+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8c09b9a0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.454+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec8c0b4200 con 0x7fec8c0b1c40 2024-09-17T04:09:07.454 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.455+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fec78041030 con 0x7fec8c0b1c40 2024-09-17T04:09:07.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.456+0000 7fec89600640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 0x7fec6c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:07.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.456+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5795+0+0 (secure 0 0 0) 0x7fec780bb5f0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.455 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.456+0000 7fec91800640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 0x7fec6c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:07.456 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.457+0000 7fec91800640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 0x7fec6c07b0d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80009060 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:07.459 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.460+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fec8c0cbaf0 con 0x7fec8c0b1c40 2024-09-17T04:09:07.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:07.682+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fec8c0751a0 con 0x7fec8c0b1c40 2024-09-17T04:09:08.349 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.350+0000 7fec89600640 1 -- 172.21.15.129:0/2243263122 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v275) ==== 104+0+0 (secure 0 0 0) 0x7fec780832e0 con 0x7fec8c0b1c40 2024-09-17T04:09:08.352 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 msgr2=0x7fec6c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 0x7fec6c07b0d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80009060 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 msgr2=0x7fec8c0ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7fec780099c0 tx=0x7fec780039e0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 shutdown_connections 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fec6c078bc0 0x7fec6c07b0d0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fec8c0b3010 0x7fec8c1f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fec8c0b1c40 0x7fec8c0ce8c0 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 --2- 172.21.15.129:0/2243263122 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fec8c074800 0x7fec8c15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 >> 172.21.15.129:0/2243263122 conn(0x7fec8c0ccc40 msgr2=0x7fec8c0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 shutdown_connections 2024-09-17T04:09:08.353 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.353+0000 7fec94200640 1 -- 172.21.15.129:0/2243263122 wait complete. 2024-09-17T04:09:08.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-17T04:09:08.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 -- 172.21.15.129:0/1291205245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240705b0 msgr2=0x7fde2406fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:08.602 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 --2- 172.21.15.129:0/1291205245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240705b0 0x7fde2406fab0 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7fde240726b0 tx=0x7fde1402a4b0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 -- 172.21.15.129:0/1291205245 shutdown_connections 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 --2- 172.21.15.129:0/1291205245 >> v2:172.21.15.181:3300/0 conn(0x7fde240cbf10 0x7fde240b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 --2- 172.21.15.129:0/1291205245 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240705b0 0x7fde2406fab0 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 -- 172.21.15.129:0/1291205245 >> 172.21.15.129:0/1291205245 conn(0x7fde240ccc40 msgr2=0x7fde240cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 -- 172.21.15.129:0/1291205245 shutdown_connections 2024-09-17T04:09:08.603 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.603+0000 7fde29000640 1 -- 172.21.15.129:0/1291205245 wait complete. 2024-09-17T04:09:08.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.605+0000 7fde29000640 1 Processor -- start 2024-09-17T04:09:08.604 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.605+0000 7fde29000640 1 -- start start 2024-09-17T04:09:08.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fde240705b0 0x7fde24161950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:08.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:08.607 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 0x7fde241e6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fde240caf20 con 0x7fde240b2180 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fde240cc680 con 0x7fde240cbf10 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde29000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fde240cb0a0 con 0x7fde240705b0 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58292/0 (socket says 172.21.15.129:58292) 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 -- 172.21.15.129:0/3677461408 learned_addr learned my addr 172.21.15.129:0/3677461408 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde21e00640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fde240705b0 0x7fde24161950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde22800640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 0x7fde241e6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 -- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fde240705b0 msgr2=0x7fde24161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fde240705b0 0x7fde24161950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 -- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 msgr2=0x7fde241e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:08.608 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 0x7fde241e6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde19400640 1 -- 172.21.15.129:0/3677461408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde24160a50 con 0x7fde240b2180 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.607+0000 7fde22800640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 0x7fde241e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde19400640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00b440 tx=0x7fde0c00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c015020 con 0x7fde240b2180 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde0c011980 con 0x7fde240b2180 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c011dd0 con 0x7fde240b2180 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde24160d30 con 0x7fde240b2180 2024-09-17T04:09:08.609 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.608+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde241f5fc0 con 0x7fde240b2180 2024-09-17T04:09:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.609+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde241f5dd0 con 0x7fde240b2180 2024-09-17T04:09:08.610 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.610+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fde0c018020 con 0x7fde240b2180 2024-09-17T04:09:08.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.612+0000 7fde1aa00640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 0x7fde0007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:08.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.612+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 5795+0+0 (secure 0 0 0) 0x7fde0c09aeb0 con 0x7fde240b2180 2024-09-17T04:09:08.611 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.612+0000 7fde21e00640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 0x7fde0007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:08.612 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.613+0000 7fde21e00640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 0x7fde0007b0a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fde14034000 tx=0x7fde14004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.616+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fde0c01f020 con 0x7fde240b2180 2024-09-17T04:09:08.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:08.801+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fde24161ea0 con 0x7fde240b2180 2024-09-17T04:09:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.367+0000 7fde1aa00640 1 -- 172.21.15.129:0/3677461408 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v276) ==== 181+0+0 (secure 0 0 0) 0x7fde0c00d040 con 0x7fde240b2180 2024-09-17T04:09:09.367 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_metadata' removed 2024-09-17T04:09:09.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.371+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 msgr2=0x7fde0007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:09.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.371+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 0x7fde0007b0a0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fde14034000 tx=0x7fde14004730 comp rx=0 tx=0).stop 2024-09-17T04:09:09.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.371+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 msgr2=0x7fde241f9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:09.370 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.371+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00b440 tx=0x7fde0c00b900 comp rx=0 tx=0).stop 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 shutdown_connections 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fde00078b90 0x7fde0007b0a0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fde240cbf10 0x7fde241e6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fde240b2180 0x7fde241f9e60 secure :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00b440 tx=0x7fde0c00b900 comp rx=0 tx=0).stop 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 --2- 172.21.15.129:0/3677461408 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fde240705b0 0x7fde24161950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:09.371 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.372+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 >> 172.21.15.129:0/3677461408 conn(0x7fde240ccc40 msgr2=0x7fde2415dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.373+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 shutdown_connections 2024-09-17T04:09:09.372 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.373+0000 7fde29000640 1 -- 172.21.15.129:0/3677461408 wait complete. 2024-09-17T04:09:09.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-17T04:09:09.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 -- 172.21.15.129:0/1837844378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 msgr2=0x7fbca4004c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:09.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 --2- 172.21.15.129:0/1837844378 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbca4004c70 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7fbca80017d0 tx=0x7fbca4005830 comp rx=0 tx=0).stop 2024-09-17T04:09:09.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcbfe00640 1 -- 172.21.15.129:0/1837844378 reap_dead start 2024-09-17T04:09:09.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 -- 172.21.15.129:0/1837844378 shutdown_connections 2024-09-17T04:09:09.625 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 -- 172.21.15.129:0/1837844378 >> 172.21.15.129:0/1837844378 conn(0x7fbcc00b1c50 msgr2=0x7fbcc00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:09.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 -- 172.21.15.129:0/1837844378 shutdown_connections 2024-09-17T04:09:09.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.625+0000 7fbcc5000640 1 -- 172.21.15.129:0/1837844378 wait complete. 2024-09-17T04:09:09.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.627+0000 7fbcc5000640 1 Processor -- start 2024-09-17T04:09:09.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.628+0000 7fbcc5000640 1 -- start start 2024-09-17T04:09:09.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcc014cfe0 0x7fbcc00c5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcc014d3f0 0x7fbcc00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc00ae6c0 con 0x7fbca4004800 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc00cd310 con 0x7fbcc014d3f0 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcc5000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fbcc00ae840 con 0x7fbcc014cfe0 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcbf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcbf400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58316/0 (socket says 172.21.15.129:58316) 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.629+0000 7fbcbf400640 1 -- 172.21.15.129:0/3971531582 learned_addr learned my addr 172.21.15.129:0/3971531582 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcb6a00640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcc014cfe0 0x7fbcc00c5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbfe00640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcc014d3f0 0x7fbcc00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:09.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 -- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcc014cfe0 msgr2=0x7fbcc00c5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcc014cfe0 0x7fbcc00c5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 -- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcc014d3f0 msgr2=0x7fbcc00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcc014d3f0 0x7fbcc00c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 -- 172.21.15.129:0/3971531582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc01f8940 con 0x7fbca4004800 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcbf400640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0001240 tx=0x7fbcb00039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0015070 con 0x7fbca4004800 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.630+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbcb0011840 con 0x7fbca4004800 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.631+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb0011c90 con 0x7fbca4004800 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.631+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc0156db0 con 0x7fbca4004800 2024-09-17T04:09:09.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.631+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc01f8ec0 con 0x7fbca4004800 2024-09-17T04:09:09.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.632+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc01620e0 con 0x7fbca4004800 2024-09-17T04:09:09.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.633+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fbcb0018070 con 0x7fbca4004800 2024-09-17T04:09:09.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.634+0000 7fbcb7e00640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 0x7fbc9c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:09.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.634+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(276..276 src has 1..276) ==== 5357+0+0 (secure 0 0 0) 0x7fbcb009b8d0 con 0x7fbca4004800 2024-09-17T04:09:09.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.634+0000 7fbcb6a00640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 0x7fbc9c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:09.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.635+0000 7fbcb6a00640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 0x7fbc9c07b0d0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbca8032000 tx=0x7fbca8001360 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:09.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.638+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fbcb0063790 con 0x7fbca4004800 2024-09-17T04:09:09.823 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:09.824+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbcc0150eb0 con 0x7fbca4004800 2024-09-17T04:09:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.374+0000 7fbcb7e00640 1 -- 172.21.15.129:0/3971531582 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v277) ==== 169+0+0 (secure 0 0 0) 0x7fbcb00681c0 con 0x7fbca4004800 2024-09-17T04:09:10.374 INFO:teuthology.orchestra.run.smithi129.stderr:pool 'cephfs_data' removed 2024-09-17T04:09:10.379 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 msgr2=0x7fbc9c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 0x7fbc9c07b0d0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbca8032000 tx=0x7fbca8001360 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 msgr2=0x7fbcc0157900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 secure :-1 s=READY pgs=3031 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0001240 tx=0x7fbcb00039c0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 shutdown_connections 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fbc9c078bc0 0x7fbc9c07b0d0 secure :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbca8032000 tx=0x7fbca8001360 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbcc014d3f0 0x7fbcc00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fbcc014cfe0 0x7fbcc00c5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.379+0000 7fbcc5000640 1 --2- 172.21.15.129:0/3971531582 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fbca4004800 0x7fbcc0157900 unknown :-1 s=CLOSED pgs=3031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.380+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 >> 172.21.15.129:0/3971531582 conn(0x7fbcc00b1c50 msgr2=0x7fbcc00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.380+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 shutdown_connections 2024-09-17T04:09:10.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.380+0000 7fbcc5000640 1 -- 172.21.15.129:0/3971531582 wait complete. 2024-09-17T04:09:10.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config reset 31 2024-09-17T04:09:10.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.630+0000 7efcb0400640 1 -- 172.21.15.129:0/2828913504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 msgr2=0x7efca806eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.630+0000 7efcb0400640 1 --2- 172.21.15.129:0/2828913504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca806eed0 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7efca0005fd0 tx=0x7efca002a8b0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.631 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 -- 172.21.15.129:0/2828913504 shutdown_connections 2024-09-17T04:09:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 --2- 172.21.15.129:0/2828913504 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca806eed0 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 --2- 172.21.15.129:0/2828913504 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca80b3b90 0x7efca80cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 -- 172.21.15.129:0/2828913504 >> 172.21.15.129:0/2828913504 conn(0x7efca806d410 msgr2=0x7efca80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 -- 172.21.15.129:0/2828913504 shutdown_connections 2024-09-17T04:09:10.632 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.631+0000 7efcb0400640 1 -- 172.21.15.129:0/2828913504 wait complete. 2024-09-17T04:09:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.633+0000 7efcb0400640 1 Processor -- start 2024-09-17T04:09:10.633 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.633+0000 7efcb0400640 1 -- start start 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.634+0000 7efcb0400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.634+0000 7efcb0400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80ce170 0x7efca815e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.634+0000 7efcb0400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca81f8b80 0x7efca81f8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcb0400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7efca806f590 con 0x7efca80b2950 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcb0400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7efca806f410 con 0x7efca80ce170 2024-09-17T04:09:10.635 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcb0400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7efca806f710 con 0x7efca81f8b80 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58330/0 (socket says 172.21.15.129:58330) 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 -- 172.21.15.129:0/778598464 learned_addr learned my addr 172.21.15.129:0/778598464 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcad000640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80ce170 0x7efca815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcae400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca81f8b80 0x7efca81f8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 -- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca81f8b80 msgr2=0x7efca81f8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca81f8b80 0x7efca81f8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 -- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80ce170 msgr2=0x7efca815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80ce170 0x7efca815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.636 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.635+0000 7efcada00640 1 -- 172.21.15.129:0/778598464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca81f94d0 con 0x7efca80b2950 2024-09-17T04:09:10.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efcada00640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c3c0 tx=0x7efc9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:10.637 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c017070 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efc9c0035f0 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c003a40 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca8159700 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.636+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca8159cc0 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.637+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca806a140 con 0x7efca80b2950 2024-09-17T04:09:10.638 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.639+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7efc9c01a070 con 0x7efca80b2950 2024-09-17T04:09:10.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.640+0000 7efca5600640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 0x7efc8807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:10.639 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.640+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7efc9c099c70 con 0x7efca80b2950 2024-09-17T04:09:10.640 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.640+0000 7efcad000640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 0x7efc8807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:10.641 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.641+0000 7efcad000640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 0x7efc8807b0a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7efca8152a20 tx=0x7efc9400b2e0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:10.644 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.644+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7efca81f94d0 con 0x7efca80b2950 2024-09-17T04:09:10.833 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.832+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "config reset", "num": 31} v 0) -- 0x7efca80af650 con 0x7efca80b2950 2024-09-17T04:09:10.837 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.837+0000 7efca5600640 1 -- 172.21.15.129:0/778598464 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "config reset", "num": 31}]=0 v74) ==== 71+0+0 (secure 0 0 0) 0x7efc9c061d50 con 0x7efca80b2950 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 msgr2=0x7efc8807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 0x7efc8807b0a0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7efca8152a20 tx=0x7efc9400b2e0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 msgr2=0x7efca8156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 secure :-1 s=READY pgs=3034 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00c3c0 tx=0x7efc9c00c880 comp rx=0 tx=0).stop 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 shutdown_connections 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7efc88078b90 0x7efc8807b0a0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7efca81f8b80 0x7efca81f8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efca80ce170 0x7efca815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 --2- 172.21.15.129:0/778598464 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7efca80b2950 0x7efca8156d70 unknown :-1 s=CLOSED pgs=3034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 >> 172.21.15.129:0/778598464 conn(0x7efca806d410 msgr2=0x7efca80736f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 shutdown_connections 2024-09-17T04:09:10.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:10.840+0000 7efcb0400640 1 -- 172.21.15.129:0/778598464 wait complete. 2024-09-17T04:09:10.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove' 2024-09-17T04:09:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.086+0000 7f849c400640 1 -- 172.21.15.129:0/419856562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f849406f2e0 msgr2=0x7f84940cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.086+0000 7f849c400640 1 --2- 172.21.15.129:0/419856562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f849406f2e0 0x7f84940cd1c0 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7f84940ccc10 tx=0x7f848c02aa10 comp rx=0 tx=0).stop 2024-09-17T04:09:11.087 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 -- 172.21.15.129:0/419856562 shutdown_connections 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 --2- 172.21.15.129:0/419856562 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f849406f2e0 0x7f84940cd1c0 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 --2- 172.21.15.129:0/419856562 >> v2:172.21.15.181:3300/0 conn(0x7f849406eed0 0x7f84940ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 --2- 172.21.15.129:0/419856562 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f84940705a0 0x7f84940b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 -- 172.21.15.129:0/419856562 >> 172.21.15.129:0/419856562 conn(0x7f849406d410 msgr2=0x7f84940b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 -- 172.21.15.129:0/419856562 shutdown_connections 2024-09-17T04:09:11.088 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.087+0000 7f849c400640 1 -- 172.21.15.129:0/419856562 wait complete. 2024-09-17T04:09:11.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.089+0000 7f849c400640 1 Processor -- start 2024-09-17T04:09:11.089 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.089+0000 7f849c400640 1 -- start start 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f849406eed0 0x7f849415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f849406f2e0 0x7f84940c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8494160b40 con 0x7f84940705a0 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f84941609c0 con 0x7f849406f2e0 2024-09-17T04:09:11.091 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.090+0000 7f849c400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8494160cc0 con 0x7f849406eed0 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58344/0 (socket says 172.21.15.129:58344) 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 -- 172.21.15.129:0/2390887844 learned_addr learned my addr 172.21.15.129:0/2390887844 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f8499000640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f849406f2e0 0x7f84940c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f8499a00640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f849406eed0 0x7f849415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 -- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f849406eed0 msgr2=0x7f849415d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f849406eed0 0x7f849415d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 -- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f849406f2e0 msgr2=0x7f84940c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f849406f2e0 0x7f84940c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.092 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 -- 172.21.15.129:0/2390887844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84941f5b60 con 0x7f84940705a0 2024-09-17T04:09:11.093 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.091+0000 7f849a400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d60 tx=0x7f848c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.092+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c033050 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.092+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f848c0042d0 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.092+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848c004720 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.092+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849415dce0 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.092+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84940c8010 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.093+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8494161220 con 0x7f84940705a0 2024-09-17T04:09:11.094 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.094+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f848c036070 con 0x7f84940705a0 2024-09-17T04:09:11.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.096+0000 7f8491600640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 0x7f847407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:11.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.096+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f848c0bb5f0 con 0x7f84940705a0 2024-09-17T04:09:11.096 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.096+0000 7f8499a00640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 0x7f847407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:11.097 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.097+0000 7f8499a00640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 0x7f847407b0d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f8488005fd0 tx=0x7f848800a250 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:11.100 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.100+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f848c049030 con 0x7f84940705a0 2024-09-17T04:09:11.350 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.350+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]} v 0) -- 0x7f849406a140 con 0x7f84940705a0 2024-09-17T04:09:11.381 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.382+0000 7f8491600640 1 -- 172.21.15.129:0/2390887844 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_strays.TestStrays.test_snapshot_remove"]}]=0 v996) ==== 135+0+0 (secure 0 0 0) 0x7f848c0836d0 con 0x7f84940705a0 2024-09-17T04:09:11.384 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 msgr2=0x7f847407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 0x7f847407b0d0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f8488005fd0 tx=0x7f848800a250 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 msgr2=0x7f84940c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7f848c009d60 tx=0x7f848c003f10 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 shutdown_connections 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8474078bc0 0x7f847407b0d0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f84940705a0 0x7f84940c9310 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f849406f2e0 0x7f84940c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 --2- 172.21.15.129:0/2390887844 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f849406eed0 0x7f849415d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:11.385 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 >> 172.21.15.129:0/2390887844 conn(0x7f849406d410 msgr2=0x7f849406f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:11.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.384+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 shutdown_connections 2024-09-17T04:09:11.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:11.385+0000 7f849c400640 1 -- 172.21.15.129:0/2390887844 wait complete. 2024-09-17T04:09:11.408 INFO:tasks.cephfs_test_runner:That removal of a snapshot that references a now-unlinked file results ... ok 2024-09-17T04:09:11.409 INFO:tasks.cephfs_test_runner: 2024-09-17T04:09:11.409 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-17T04:09:11.409 INFO:tasks.cephfs_test_runner:Ran 19 tests in 1149.041s 2024-09-17T04:09:11.409 INFO:tasks.cephfs_test_runner: 2024-09-17T04:09:11.409 INFO:tasks.cephfs_test_runner:OK 2024-09-17T04:09:11.409 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-17T04:09:11.420 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-17T04:09:11.441 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-17T04:09:11.444 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:09:11.444 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:11.480 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:09:11.480 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-17T04:09:11.480 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:11.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052627,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497972k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-17T04:09:11.556 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:09:11.556 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:11.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:09:11.597 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-17T04:09:11.597 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-17T04:09:11.675 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052607,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248968k,nr_inodes=812242,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497940k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-17T04:09:11.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:11.675 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:09:11.711 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-17T04:09:11.711 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-17T04:09:11.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:11.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-17T04:09:11.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:11.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-17T04:09:12.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-09-17T04:09:12.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.376 DEBUG:teuthology.orchestra.run.smithi129:> ip netns list 2024-09-17T04:09:12.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo ip link delete ceph-brx 2024-09-17T04:09:12.513 INFO:teuthology.orchestra.run.smithi129.stderr:Cannot find device "ceph-brx" 2024-09-17T04:09:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:09:12.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.516 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T04:09:12.555 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-17T04:09:12.555 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-17T04:09:12.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-17T04:09:12.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.657 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-17T04:09:12.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:12.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link delete ceph-brx 2024-09-17T04:09:13.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:13.245 DEBUG:teuthology.orchestra.run.smithi181:> ip netns list 2024-09-17T04:09:13.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-17T04:09:13.280 DEBUG:teuthology.orchestra.run.smithi181:> sudo ip link delete ceph-brx 2024-09-17T04:09:13.379 INFO:teuthology.orchestra.run.smithi181.stderr:Cannot find device "ceph-brx" 2024-09-17T04:09:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:09:13.382 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-17T04:09:13.392 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-17T04:09:13.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:09:13.621 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 -- 172.21.15.129:0/1090996255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b00cd390 msgr2=0x7fb8b01609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/1090996255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b00cd390 0x7fb8b01609c0 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00d1370 tx=0x7fb8a802aac0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 -- 172.21.15.129:0/1090996255 shutdown_connections 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/1090996255 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b00cd390 0x7fb8b01609c0 unknown :-1 s=CLOSED pgs=3038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/1090996255 >> v2:172.21.15.181:3300/0 conn(0x7fb8b00705a0 0x7fb8b00b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.620+0000 7fb8b6c00640 1 -- 172.21.15.129:0/1090996255 >> 172.21.15.129:0/1090996255 conn(0x7fb8b006d410 msgr2=0x7fb8b00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.621+0000 7fb8b6c00640 1 -- 172.21.15.129:0/1090996255 shutdown_connections 2024-09-17T04:09:13.622 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.621+0000 7fb8b6c00640 1 -- 172.21.15.129:0/1090996255 wait complete. 2024-09-17T04:09:13.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.623+0000 7fb8b6c00640 1 Processor -- start 2024-09-17T04:09:13.623 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.623+0000 7fb8b6c00640 1 -- start start 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8b00705a0 0x7fb8b015a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b006f460 con 0x7fb8b015ae90 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b006f2e0 con 0x7fb8b006eed0 2024-09-17T04:09:13.626 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.624+0000 7fb8b6c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b006f5e0 con 0x7fb8b00705a0 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8b4c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.129:37710/0 (socket says 172.21.15.129:37710) 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 -- 172.21.15.129:0/465566755 learned_addr learned my addr 172.21.15.129:0/465566755 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8b4c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:58358/0 (socket says 172.21.15.129:58358) 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8a7e00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8b00705a0 0x7fb8b015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 -- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8b00705a0 msgr2=0x7fb8b015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8b00705a0 0x7fb8b015a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.627 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 -- 172.21.15.129:0/465566755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 msgr2=0x7fb8b00ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8afe00640 1 -- 172.21.15.129:0/465566755 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b00ca680 con 0x7fb8b006eed0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.625+0000 7fb8b4c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8afe00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d750 tx=0x7fb8a000dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a0017070 con 0x7fb8b006eed0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb8a0004310 con 0x7fb8b006eed0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b01f35e0 con 0x7fb8b006eed0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a000de60 con 0x7fb8b006eed0 2024-09-17T04:09:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.626+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b01f3ba0 con 0x7fb8b006eed0 2024-09-17T04:09:13.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.627+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b0161210 con 0x7fb8b006eed0 2024-09-17T04:09:13.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.628+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fb8a0009c30 con 0x7fb8b006eed0 2024-09-17T04:09:13.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.629+0000 7fb8acc00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 0x7fb88c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:13.629 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.629+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fb8a009adc0 con 0x7fb8b006eed0 2024-09-17T04:09:13.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.629+0000 7fb8a7e00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 0x7fb88c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:13.630 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.630+0000 7fb8a7e00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 0x7fb88c07b0a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0155ba0 tx=0x7fb898009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:13.634 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.634+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fb8a0020020 con 0x7fb8b006eed0 2024-09-17T04:09:13.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.813+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8b006a140 con 0x7fb88c078b90 2024-09-17T04:09:13.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.815+0000 7fb8acc00640 1 -- 172.21.15.129:0/465566755 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13448 (secure 0 0 0) 0x7fb8b006a140 con 0x7fb88c078b90 2024-09-17T04:09:13.815 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:13.815 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:09:13.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 msgr2=0x7fb88c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:13.817 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 0x7fb88c07b0a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0155ba0 tx=0x7fb898009040 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 msgr2=0x7fb8b00ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fb8a000d750 tx=0x7fb8a000dc10 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 shutdown_connections 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fb88c078b90 0x7fb88c07b0a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fb8b015ae90 0x7fb8b00ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fb8b00705a0 0x7fb8b015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 --2- 172.21.15.129:0/465566755 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8b006eed0 0x7fb8b00ca140 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 >> 172.21.15.129:0/465566755 conn(0x7fb8b006d410 msgr2=0x7fb8b00b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 shutdown_connections 2024-09-17T04:09:13.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:13.817+0000 7fb8b6c00640 1 -- 172.21.15.129:0/465566755 wait complete. 2024-09-17T04:09:13.844 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-09-17T04:09:12.634690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":421072,"kb_used_data":28592,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754553648,"statfs":{"total":773094113280,"available":772662935552,"internally_reserved":0,"allocated":29278208,"data_stored":13730698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67896,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-10,"num_read_kb":-8194,"num_write":-147,"num_write_kb":-16507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001752"},"pg_stats":[{"pgid":"1.0","version":"18'72","reported_seq":581,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-17T04:09:07.537359+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:09:07.537359+0000","last_peered":"2024-09-17T04:09:07.537359+0000","last_clean":"2024-09-17T04:09:07.537359+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:09:07.537359+0000","last_undegraded":"2024-09-17T04:09:07.537359+0000","last_fullsized":"2024-09-17T04:09:07.537359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509679,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53224,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318616,"statfs":{"total":96636764160,"available":96582262784,"internally_reserved":0,"allocated":4083712,"data_stored":2142194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509682,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34600,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337240,"statfs":{"total":96636764160,"available":96601333760,"internally_reserved":0,"allocated":3493888,"data_stored":1551980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509681,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56800,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3485696,"data_stored":1551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509679,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56804,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3489792,"data_stored":1551770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509681,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56800,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3485696,"data_stored":1551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509681,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3497984,"data_stored":1552550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542383,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56884,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314956,"statfs":{"total":96636764160,"available":96578514944,"internally_reserved":0,"allocated":3571712,"data_stored":1618857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542384,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53308,"kb_used_data":4072,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318532,"statfs":{"total":96636764160,"available":96582176768,"internally_reserved":0,"allocated":4169728,"data_stored":2209875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:09:13.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:09:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 -- 172.21.15.129:0/3830207802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b3015c950 msgr2=0x7f7b300b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 --2- 172.21.15.129:0/3830207802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b3015c950 0x7f7b300b3960 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f7b18005fd0 tx=0x7f7b1802a540 comp rx=0 tx=0).stop 2024-09-17T04:09:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 -- 172.21.15.129:0/3830207802 shutdown_connections 2024-09-17T04:09:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 --2- 172.21.15.129:0/3830207802 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30074730 0x7f7b300cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 --2- 172.21.15.129:0/3830207802 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b3015c950 0x7f7b300b3960 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 -- 172.21.15.129:0/3830207802 >> 172.21.15.129:0/3830207802 conn(0x7f7b300d0a70 msgr2=0x7f7b300d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 -- 172.21.15.129:0/3830207802 shutdown_connections 2024-09-17T04:09:14.066 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.065+0000 7f7b36200640 1 -- 172.21.15.129:0/3830207802 wait complete. 2024-09-17T04:09:14.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.067+0000 7f7b36200640 1 Processor -- start 2024-09-17T04:09:14.067 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.067+0000 7f7b36200640 1 -- start start 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30068d70 0x7f7b3009aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 0x7f7b301f7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300cce60 con 0x7f7b30074730 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f7b300ccce0 con 0x7f7b3015c950 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b36200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f7b300ccfe0 con 0x7f7b30068d70 2024-09-17T04:09:14.070 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46908/0 (socket says 172.21.15.129:46908) 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 -- 172.21.15.129:0/2752470057 learned_addr learned my addr 172.21.15.129:0/2752470057 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b2f400640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30068d70 0x7f7b3009aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b2fe00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 0x7f7b301f7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 -- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30068d70 msgr2=0x7f7b3009aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30068d70 0x7f7b3009aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 -- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 msgr2=0x7f7b301f7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 0x7f7b301f7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b26a00640 1 -- 172.21.15.129:0/2752470057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3009b2f0 con 0x7f7b30074730 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.069+0000 7f7b2fe00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 0x7f7b301f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b26a00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f7b18000c00 tx=0x7f7b180038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1803e030 con 0x7f7b30074730 2024-09-17T04:09:14.071 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7b18003ca0 con 0x7f7b30074730 2024-09-17T04:09:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3009b5d0 con 0x7f7b30074730 2024-09-17T04:09:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18035040 con 0x7f7b30074730 2024-09-17T04:09:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.070+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3009bb90 con 0x7f7b30074730 2024-09-17T04:09:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.071+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b300a1c60 con 0x7f7b30074730 2024-09-17T04:09:14.072 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.072+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f7b18041030 con 0x7f7b30074730 2024-09-17T04:09:14.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.074+0000 7f7b27e00640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 0x7f7b0c07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.074+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f7b180bafc0 con 0x7f7b30074730 2024-09-17T04:09:14.074 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.074+0000 7f7b2f400640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 0x7f7b0c07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.075 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.075+0000 7f7b2f400640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 0x7f7b0c07b0d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007fd0 tx=0x7f7b20009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.078 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.079+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f7b18083130 con 0x7f7b30074730 2024-09-17T04:09:14.260 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.260+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b3009b080 con 0x7f7b0c078bc0 2024-09-17T04:09:14.261 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.261+0000 7f7b27e00640 1 -- 172.21.15.129:0/2752470057 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13448 (secure 0 0 0) 0x7f7b3009b080 con 0x7f7b0c078bc0 2024-09-17T04:09:14.261 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:14.262 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:09:14.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 msgr2=0x7f7b0c07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 0x7f7b0c07b0d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007fd0 tx=0x7f7b20009040 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 msgr2=0x7f7b301f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 secure :-1 s=READY pgs=3040 cs=0 l=1 rev1=1 crypto rx=0x7f7b18000c00 tx=0x7f7b180038a0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 shutdown_connections 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f7b0c078bc0 0x7f7b0c07b0d0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7b3015c950 0x7f7b301f7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f7b30074730 0x7f7b301f6e20 unknown :-1 s=CLOSED pgs=3040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 --2- 172.21.15.129:0/2752470057 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f7b30068d70 0x7f7b3009aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 >> 172.21.15.129:0/2752470057 conn(0x7f7b300d0a70 msgr2=0x7f7b300b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 shutdown_connections 2024-09-17T04:09:14.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.264+0000 7f7b36200640 1 -- 172.21.15.129:0/2752470057 wait complete. 2024-09-17T04:09:14.291 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":865,"stamp":"2024-09-17T04:09:12.634690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":421072,"kb_used_data":28592,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754553648,"statfs":{"total":773094113280,"available":772662935552,"internally_reserved":0,"allocated":29278208,"data_stored":13730698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67896,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-10,"num_read_kb":-8194,"num_write":-147,"num_write_kb":-16507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001752"},"pg_stats":[{"pgid":"1.0","version":"18'72","reported_seq":581,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-17T04:09:07.537359+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:09:07.537359+0000","last_peered":"2024-09-17T04:09:07.537359+0000","last_clean":"2024-09-17T04:09:07.537359+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:09:07.537359+0000","last_undegraded":"2024-09-17T04:09:07.537359+0000","last_fullsized":"2024-09-17T04:09:07.537359+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509679,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53224,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318616,"statfs":{"total":96636764160,"available":96582262784,"internally_reserved":0,"allocated":4083712,"data_stored":2142194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509682,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34600,"kb_used_data":3412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337240,"statfs":{"total":96636764160,"available":96601333760,"internally_reserved":0,"allocated":3493888,"data_stored":1551980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509681,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56800,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3485696,"data_stored":1551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509679,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56804,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3489792,"data_stored":1551770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509681,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56800,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3485696,"data_stored":1551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509681,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52652,"kb_used_data":3416,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319188,"statfs":{"total":96636764160,"available":96582848512,"internally_reserved":0,"allocated":3497984,"data_stored":1552550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542383,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56884,"kb_used_data":3488,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314956,"statfs":{"total":96636764160,"available":96578514944,"internally_reserved":0,"allocated":3571712,"data_stored":1618857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542384,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53308,"kb_used_data":4072,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318532,"statfs":{"total":96636764160,"available":96582176768,"internally_reserved":0,"allocated":4169728,"data_stored":2209875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:09:14.292 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-17T04:09:14.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.513+0000 7f54e5200640 1 -- 172.21.15.129:0/2865119977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e00cf9c0 msgr2=0x7f54e00ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.513+0000 7f54e5200640 1 --2- 172.21.15.129:0/2865119977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e00cf9c0 0x7f54e00ccc80 secure :-1 s=READY pgs=3041 cs=0 l=1 rev1=1 crypto rx=0x7f54d8005fd0 tx=0x7f54d802a540 comp rx=0 tx=0).stop 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 -- 172.21.15.129:0/2865119977 shutdown_connections 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 --2- 172.21.15.129:0/2865119977 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e00cf9c0 0x7f54e00ccc80 unknown :-1 s=CLOSED pgs=3041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 --2- 172.21.15.129:0/2865119977 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e0099570 0x7f54e006eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 -- 172.21.15.129:0/2865119977 >> 172.21.15.129:0/2865119977 conn(0x7f54e006d410 msgr2=0x7f54e00b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 -- 172.21.15.129:0/2865119977 shutdown_connections 2024-09-17T04:09:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.514+0000 7f54e5200640 1 -- 172.21.15.129:0/2865119977 wait complete. 2024-09-17T04:09:14.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.516+0000 7f54e5200640 1 Processor -- start 2024-09-17T04:09:14.516 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.516+0000 7f54e5200640 1 -- start start 2024-09-17T04:09:14.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e006f410 0x7f54e01f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54e00cf9c0 0x7f54e0073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.518 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f54e0160ba0 con 0x7f54e0099570 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f54e0160a20 con 0x7f54e00cf9c0 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.517+0000 7f54e5200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f54e0160d20 con 0x7f54e006f410 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46924/0 (socket says 172.21.15.129:46924) 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 -- 172.21.15.129:0/852787251 learned_addr learned my addr 172.21.15.129:0/852787251 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dde00640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e006f410 0x7f54e01f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54de800640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54e00cf9c0 0x7f54e0073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 -- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e006f410 msgr2=0x7f54e01f8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e006f410 0x7f54e01f8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 -- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54e00cf9c0 msgr2=0x7f54e0073890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54e00cf9c0 0x7f54e0073890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 -- 172.21.15.129:0/852787251 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e01f9810 con 0x7f54e0099570 2024-09-17T04:09:14.519 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.518+0000 7f54dd400640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c3f0 tx=0x7f54cc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.519+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc017070 con 0x7f54e0099570 2024-09-17T04:09:14.520 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.519+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f54cc0035f0 con 0x7f54e0099570 2024-09-17T04:09:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.519+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e01f9af0 con 0x7f54e0099570 2024-09-17T04:09:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.519+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc003a40 con 0x7f54e0099570 2024-09-17T04:09:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.519+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e00c8210 con 0x7f54e0099570 2024-09-17T04:09:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.520+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e01611f0 con 0x7f54e0099570 2024-09-17T04:09:14.521 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.521+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f54cc01a070 con 0x7f54e0099570 2024-09-17T04:09:14.522 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.522+0000 7f54d6000640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 0x7f54c007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.523+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f54cc099d10 con 0x7f54e0099570 2024-09-17T04:09:14.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.523+0000 7f54dde00640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 0x7f54c007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.524+0000 7f54dde00640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 0x7f54c007b0a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f54d0005fd0 tx=0x7f54d0009500 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.527 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.527+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f54cc09f020 con 0x7f54e0099570 2024-09-17T04:09:14.707 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.707+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f54e006a140 con 0x7f54c0078b90 2024-09-17T04:09:14.708 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.708+0000 7f54d6000640 1 -- 172.21.15.129:0/852787251 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13478 (secure 0 0 0) 0x7f54e006a140 con 0x7f54c0078b90 2024-09-17T04:09:14.709 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:14.709 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:09:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 msgr2=0x7f54c007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 0x7f54c007b0a0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f54d0005fd0 tx=0x7f54d0009500 comp rx=0 tx=0).stop 2024-09-17T04:09:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 msgr2=0x7f54e01f8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00c3f0 tx=0x7f54cc00c8b0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 shutdown_connections 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f54c0078b90 0x7f54c007b0a0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f54e00cf9c0 0x7f54e0073890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f54e0099570 0x7f54e01f8760 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 --2- 172.21.15.129:0/852787251 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f54e006f410 0x7f54e01f8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 >> 172.21.15.129:0/852787251 conn(0x7f54e006d410 msgr2=0x7f54e00d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 shutdown_connections 2024-09-17T04:09:14.712 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.711+0000 7f54e5200640 1 -- 172.21.15.129:0/852787251 wait complete. 2024-09-17T04:09:14.733 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":866,"stamp":"2024-09-17T04:09:14.635129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":77,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":4,"kb":754974720,"kb_used":421080,"kb_used_data":28588,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754553640,"statfs":{"total":773094113280,"available":772662927360,"internally_reserved":0,"allocated":29274112,"data_stored":13664066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67896,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-10,"num_read_kb":-8194,"num_write":-147,"num_write_kb":-16507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.443632"},"pg_stats":[{"pgid":"1.0","version":"18'72","reported_seq":585,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-17T04:09:11.033678+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:09:11.033678+0000","last_peered":"2024-09-17T04:09:11.033678+0000","last_clean":"2024-09-17T04:09:11.033678+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:09:11.033678+0000","last_undegraded":"2024-09-17T04:09:11.033678+0000","last_fullsized":"2024-09-17T04:09:11.033678+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509680,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509683,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34624,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337216,"statfs":{"total":96636764160,"available":96601309184,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509682,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56836,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509679,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56804,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315036,"statfs":{"total":96636764160,"available":96578596864,"internally_reserved":0,"allocated":3489792,"data_stored":1551770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509681,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56800,"kb_used_data":3404,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3485696,"data_stored":1551736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509682,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52676,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319164,"statfs":{"total":96636764160,"available":96582823936,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56832,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542385,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53252,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318588,"statfs":{"total":96636764160,"available":96582234112,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:09:14.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-17T04:09:14.955 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 -- 172.21.15.129:0/3415962432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb415c950 msgr2=0x7f1eb40b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3415962432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb415c950 0x7f1eb40b3960 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c005fd0 tx=0x7f1e9c02a5f0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 -- 172.21.15.129:0/3415962432 shutdown_connections 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3415962432 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb415c950 0x7f1eb40b3960 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 -- 172.21.15.129:0/3415962432 >> 172.21.15.129:0/3415962432 conn(0x7f1eb40d0a70 msgr2=0x7f1eb40d0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 -- 172.21.15.129:0/3415962432 shutdown_connections 2024-09-17T04:09:14.956 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.955+0000 7f1ebb000640 1 -- 172.21.15.129:0/3415962432 wait complete. 2024-09-17T04:09:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.957+0000 7f1ebb000640 1 Processor -- start 2024-09-17T04:09:14.957 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.957+0000 7f1ebb000640 1 -- start start 2024-09-17T04:09:14.960 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1eb4074730 0x7f1eb40c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 0x7f1eb40c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb40cd030 con 0x7f1eb4068d70 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb40cceb0 con 0x7f1eb4074730 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1ebb000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb40cd1b0 con 0x7f1eb415c950 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1eb3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1eb3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46950/0 (socket says 172.21.15.129:46950) 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1eb3e00640 1 -- 172.21.15.129:0/3592144409 learned_addr learned my addr 172.21.15.129:0/3592144409 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1eabe00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1eb4074730 0x7f1eb40c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.959+0000 7f1eb9000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 0x7f1eb40c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.961 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 -- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 msgr2=0x7f1eb40c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 0x7f1eb40c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 -- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1eb4074730 msgr2=0x7f1eb40c6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1eb4074730 0x7f1eb40c6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 -- 172.21.15.129:0/3592144409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb40c46d0 con 0x7f1eb4068d70 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb9000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 0x7f1eb40c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb3e00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c400 tx=0x7f1ea400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4017070 con 0x7f1eb4068d70 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1ea40053c0 con 0x7f1eb4068d70 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb40c49b0 con 0x7f1eb4068d70 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.960+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4003850 con 0x7f1eb4068d70 2024-09-17T04:09:14.962 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.961+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb40c4f70 con 0x7f1eb4068d70 2024-09-17T04:09:14.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.961+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb40b3960 con 0x7f1eb4068d70 2024-09-17T04:09:14.963 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.963+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f1ea401a070 con 0x7f1eb4068d70 2024-09-17T04:09:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.964+0000 7f1eb0c00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 0x7f1e9007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.964+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f1ea4099da0 con 0x7f1eb4068d70 2024-09-17T04:09:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.964+0000 7f1eabe00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 0x7f1e9007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:14.965 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.965+0000 7f1eabe00640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 0x7f1e9007b0a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c033000 tx=0x7f1e9c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:14.968 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:14.968+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f1ea4061e80 con 0x7f1eb4068d70 2024-09-17T04:09:15.153 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.153+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1eb4076560 con 0x7f1eb4068d70 2024-09-17T04:09:15.154 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.154+0000 7f1eb0c00640 1 -- 172.21.15.129:0/3592144409 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v277) ==== 74+0+13092 (secure 0 0 0) 0x7f1ea40668b0 con 0x7f1eb4068d70 2024-09-17T04:09:15.154 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:15.154 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":277,"fsid":"c5b53c7d-83e3-4793-838f-5ade77ae49c7","created":"2024-09-17T03:49:08.812175+0000","modified":"2024-09-17T04:09:10.368353+0000","last_up_change":"2024-09-17T03:49:24.168172+0000","last_in_change":"2024-09-17T03:49:13.598108+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":41,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T03:49:24.321404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3f601c8a-535a-4179-b27b-7e244b416963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6825","nonce":1655624368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6827","nonce":1655624368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6831","nonce":1655624368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":1655624368},{"type":"v1","addr":"172.21.15.129:6829","nonce":1655624368}]},"public_addr":"172.21.15.129:6825/1655624368","cluster_addr":"172.21.15.129:6827/1655624368","heartbeat_back_addr":"172.21.15.129:6831/1655624368","heartbeat_front_addr":"172.21.15.129:6829/1655624368","state":["exists","up"]},{"osd":1,"uuid":"04e0f8c1-680e-4a2f-9cfd-2e682f9848ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6801","nonce":4215860840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6803","nonce":4215860840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6807","nonce":4215860840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4215860840},{"type":"v1","addr":"172.21.15.129:6805","nonce":4215860840}]},"public_addr":"172.21.15.129:6801/4215860840","cluster_addr":"172.21.15.129:6803/4215860840","heartbeat_back_addr":"172.21.15.129:6807/4215860840","heartbeat_front_addr":"172.21.15.129:6805/4215860840","state":["exists","up"]},{"osd":2,"uuid":"8c23300f-de54-4c2c-8799-6f0f587ab816","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6809","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6811","nonce":283507149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6814","nonce":283507149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6822","nonce":283507149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":283507149},{"type":"v1","addr":"172.21.15.129:6818","nonce":283507149}]},"public_addr":"172.21.15.129:6811/283507149","cluster_addr":"172.21.15.129:6814/283507149","heartbeat_back_addr":"172.21.15.129:6822/283507149","heartbeat_front_addr":"172.21.15.129:6818/283507149","state":["exists","up"]},{"osd":3,"uuid":"4fbdb8a6-60d3-45ef-a529-bca1d50fe5f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6810","nonce":2225231477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6813","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6815","nonce":2225231477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6821","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6823","nonce":2225231477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6817","nonce":2225231477},{"type":"v1","addr":"172.21.15.129:6819","nonce":2225231477}]},"public_addr":"172.21.15.129:6810/2225231477","cluster_addr":"172.21.15.129:6815/2225231477","heartbeat_back_addr":"172.21.15.129:6823/2225231477","heartbeat_front_addr":"172.21.15.129:6819/2225231477","state":["exists","up"]},{"osd":4,"uuid":"435a59a0-2802-45e3-bd1b-23e4ccaa280b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6803","nonce":2229307524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6805","nonce":2229307524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6809","nonce":2229307524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2229307524},{"type":"v1","addr":"172.21.15.181:6807","nonce":2229307524}]},"public_addr":"172.21.15.181:6803/2229307524","cluster_addr":"172.21.15.181:6805/2229307524","heartbeat_back_addr":"172.21.15.181:6809/2229307524","heartbeat_front_addr":"172.21.15.181:6807/2229307524","state":["exists","up"]},{"osd":5,"uuid":"cfc14b39-9ae7-49fa-9a0f-d3c95849cf87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6819","nonce":3094080342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6821","nonce":3094080342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6825","nonce":3094080342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":3094080342},{"type":"v1","addr":"172.21.15.181:6823","nonce":3094080342}]},"public_addr":"172.21.15.181:6819/3094080342","cluster_addr":"172.21.15.181:6821/3094080342","heartbeat_back_addr":"172.21.15.181:6825/3094080342","heartbeat_front_addr":"172.21.15.181:6823/3094080342","state":["exists","up"]},{"osd":6,"uuid":"585d026e-6038-4ac2-a15b-440858a24e5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6811","nonce":4136500368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6813","nonce":4136500368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6817","nonce":4136500368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":4136500368},{"type":"v1","addr":"172.21.15.181:6815","nonce":4136500368}]},"public_addr":"172.21.15.181:6811/4136500368","cluster_addr":"172.21.15.181:6813/4136500368","heartbeat_back_addr":"172.21.15.181:6817/4136500368","heartbeat_front_addr":"172.21.15.181:6815/4136500368","state":["exists","up"]},{"osd":7,"uuid":"607f6c33-e19c-4f70-8f2e-f8468f4fb228","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":270,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6827","nonce":580738384}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6829","nonce":580738384}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6833","nonce":580738384}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":580738384},{"type":"v1","addr":"172.21.15.181:6831","nonce":580738384}]},"public_addr":"172.21.15.181:6827/580738384","cluster_addr":"172.21.15.181:6829/580738384","heartbeat_back_addr":"172.21.15.181:6833/580738384","heartbeat_front_addr":"172.21.15.181:6831/580738384","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.242502+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.449274+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.527225+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.608418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.365630+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.478646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.244074+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T03:49:20.767086+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.129:6835/2485159334":"2024-09-18T04:09:07.187487+0000","172.21.15.129:6833/2485159334":"2024-09-18T04:09:07.187487+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 msgr2=0x7f1e9007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 0x7f1e9007b0a0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c033000 tx=0x7f1e9c004730 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 msgr2=0x7f1eb41f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 secure :-1 s=READY pgs=3045 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400c400 tx=0x7f1ea400c8c0 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 shutdown_connections 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f1e90078b90 0x7f1e9007b0a0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f1eb415c950 0x7f1eb40c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1eb4074730 0x7f1eb40c6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 --2- 172.21.15.129:0/3592144409 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f1eb4068d70 0x7f1eb41f31d0 unknown :-1 s=CLOSED pgs=3045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 >> 172.21.15.129:0/3592144409 conn(0x7f1eb40d0a70 msgr2=0x7f1eb4158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:15.157 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.156+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 shutdown_connections 2024-09-17T04:09:15.158 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:15.157+0000 7f1ebb000640 1 -- 172.21.15.129:0/3592144409 wait complete. 2024-09-17T04:09:16.178 INFO:tasks.ceph:Scrubbing osd.0 2024-09-17T04:09:16.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:16.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 -- 172.21.15.129:0/726865911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380cd0d0 msgr2=0x7f8b380d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/726865911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380cd0d0 0x7f8b380d0060 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7f8b280098e0 tx=0x7f8b2802a5d0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 -- 172.21.15.129:0/726865911 shutdown_connections 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/726865911 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b380b7830 0x7f8b380b7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/726865911 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380cd0d0 0x7f8b380d0060 unknown :-1 s=CLOSED pgs=3046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 -- 172.21.15.129:0/726865911 >> 172.21.15.129:0/726865911 conn(0x7f8b380cef10 msgr2=0x7f8b380cf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 -- 172.21.15.129:0/726865911 shutdown_connections 2024-09-17T04:09:16.408 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.407+0000 7f8b3ec00640 1 -- 172.21.15.129:0/726865911 wait complete. 2024-09-17T04:09:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.409+0000 7f8b3ec00640 1 Processor -- start 2024-09-17T04:09:16.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.410+0000 7f8b3ec00640 1 -- start start 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b380b7830 0x7f8b380c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b380cd0d0 0x7f8b3815f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 0x7f8b3815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f8b380b1990 con 0x7f8b380d09e0 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f8b380b1810 con 0x7f8b380b7830 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3ec00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f8b38160b30 con 0x7f8b380cd0d0 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 0x7f8b3815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3cc00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 0x7f8b3815c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46968/0 (socket says 172.21.15.129:46968) 2024-09-17T04:09:16.412 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b3cc00640 1 -- 172.21.15.129:0/730379405 learned_addr learned my addr 172.21.15.129:0/730379405 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b37e00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b380b7830 0x7f8b380c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.411+0000 7f8b37400640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b380cd0d0 0x7f8b3815f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 -- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b380cd0d0 msgr2=0x7f8b3815f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f8b380cd0d0 0x7f8b3815f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 -- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b380b7830 msgr2=0x7f8b380c5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8b380b7830 0x7f8b380c5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b380c8bd0 con 0x7f8b380d09e0 2024-09-17T04:09:16.413 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3cc00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 0x7f8b3815c8d0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000c730 tx=0x7f8b3000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30003b70 con 0x7f8b380d09e0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8b30003d10 con 0x7f8b380d09e0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3815e6c0 con 0x7f8b380d09e0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.412+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30005330 con 0x7f8b380d09e0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.413+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b381603e0 con 0x7f8b380d09e0 2024-09-17T04:09:16.414 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.413+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8b380b7c40 con 0x7f8b380d09e0 2024-09-17T04:09:16.415 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.415+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f8b300054d0 con 0x7f8b380d09e0 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b2fe00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b18078bc0 0x7f8b1807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f8b3009a790 con 0x7f8b380d09e0 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b2fe00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f8b1807ead0 0x7f8b18080f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8b180815f0 con 0x7f8b1807ead0 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7f8b300a1050 con 0x7f8b380d09e0 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b37e00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b18078bc0 0x7f8b1807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.416 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.416+0000 7f8b37400640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f8b1807ead0 0x7f8b18080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.417+0000 7f8b37400640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f8b1807ead0 0x7f8b18080f20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.417+0000 7f8b37e00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b18078bc0 0x7f8b1807b0d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034000 tx=0x7f8b28004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:16.417 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.417+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== osd.0 v2:172.21.15.129:6824/1655624368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8b180815f0 con 0x7f8b1807ead0 2024-09-17T04:09:16.430 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.430+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 --> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8b380c8980 con 0x7f8b1807ead0 2024-09-17T04:09:16.431 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b2fe00640 1 -- 172.21.15.129:0/730379405 <== osd.0 v2:172.21.15.129:6824/1655624368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8b380c8980 con 0x7f8b1807ead0 2024-09-17T04:09:16.431 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:16.431 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:16.431 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f8b1807ead0 msgr2=0x7f8b18080f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:6824/1655624368,v1:172.21.15.129:6825/1655624368] conn(0x7f8b1807ead0 0x7f8b18080f20 crc :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b18078bc0 msgr2=0x7f8b1807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f8b18078bc0 0x7f8b1807b0d0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8b28034000 tx=0x7f8b28004730 comp rx=0 tx=0).stop 2024-09-17T04:09:16.432 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 msgr2=0x7f8b3815c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 --2- 172.21.15.129:0/730379405 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f8b380d09e0 0x7f8b3815c8d0 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7f8b3000c730 tx=0x7f8b3000cb30 comp rx=0 tx=0).stop 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3cc00640 1 -- 172.21.15.129:0/730379405 reap_dead start 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 shutdown_connections 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 >> 172.21.15.129:0/730379405 conn(0x7f8b380cef10 msgr2=0x7f8b38072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 shutdown_connections 2024-09-17T04:09:16.433 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.431+0000 7f8b3ec00640 1 -- 172.21.15.129:0/730379405 wait complete. 2024-09-17T04:09:16.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 -- 172.21.15.129:0/4240192369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa900d09d0 msgr2=0x7ffa900b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 --2- 172.21.15.129:0/4240192369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa900d09d0 0x7ffa900b4af0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffa88005fd0 tx=0x7ffa8802aa10 comp rx=0 tx=0).stop 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 -- 172.21.15.129:0/4240192369 shutdown_connections 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 --2- 172.21.15.129:0/4240192369 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa900d09d0 0x7ffa900b4af0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 --2- 172.21.15.129:0/4240192369 >> v2:172.21.15.181:3300/0 conn(0x7ffa900cfb60 0x7ffa90160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 -- 172.21.15.129:0/4240192369 >> 172.21.15.129:0/4240192369 conn(0x7ffa9006cfc0 msgr2=0x7ffa9006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 -- 172.21.15.129:0/4240192369 shutdown_connections 2024-09-17T04:09:16.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.674+0000 7ffa8f200640 1 -- 172.21.15.129:0/4240192369 wait complete. 2024-09-17T04:09:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.677+0000 7ffa8f200640 1 Processor -- start 2024-09-17T04:09:16.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.677+0000 7ffa8f200640 1 -- start start 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa900cfb60 0x7ffa900cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa9015e510 0x7ffa9015e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9015d270 con 0x7ffa900b46e0 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9015d0f0 con 0x7ffa900cfb60 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ffa9015d3f0 con 0x7ffa9015e510 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.678+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46978/0 (socket says 172.21.15.129:46978) 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 -- 172.21.15.129:0/2367222005 learned_addr learned my addr 172.21.15.129:0/2367222005 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa85400640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa900cfb60 0x7ffa900cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8e800640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa9015e510 0x7ffa9015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 -- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa9015e510 msgr2=0x7ffa9015e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa9015e510 0x7ffa9015e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 -- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa900cfb60 msgr2=0x7ffa900cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa900cfb60 0x7ffa900cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 -- 172.21.15.129:0/2367222005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa901f7200 con 0x7ffa900b46e0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.679+0000 7ffa8de00640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e910 tx=0x7ffa8000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.680+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8000cd80 con 0x7ffa900b46e0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.680+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffa80010430 con 0x7ffa900b46e0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.680+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9015ee60 con 0x7ffa900b46e0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.680+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80010860 con 0x7ffa900b46e0 2024-09-17T04:09:16.681 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.680+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa90157860 con 0x7ffa900b46e0 2024-09-17T04:09:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.681+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa9015d8a0 con 0x7ffa900b46e0 2024-09-17T04:09:16.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.682+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ffa8001f020 con 0x7ffa900b46e0 2024-09-17T04:09:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.683+0000 7ffa86a00640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 0x7ffa6c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.683+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7ffa8009b180 con 0x7ffa900b46e0 2024-09-17T04:09:16.683 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.683+0000 7ffa85400640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 0x7ffa6c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.684+0000 7ffa85400640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 0x7ffa6c07b0a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ffa78007fd0 tx=0x7ffa78009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:16.687 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.687+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ffa9015ee60 con 0x7ffa900b46e0 2024-09-17T04:09:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.870+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ffa9006a140 con 0x7ffa6c078b90 2024-09-17T04:09:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.871+0000 7ffa86a00640 1 -- 172.21.15.129:0/2367222005 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ffa9006a140 con 0x7ffa6c078b90 2024-09-17T04:09:16.871 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 0 to deep-scrub 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 msgr2=0x7ffa6c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 0x7ffa6c07b0a0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7ffa78007fd0 tx=0x7ffa78009040 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 msgr2=0x7ffa901f6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7ffa8000e910 tx=0x7ffa8000edd0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 shutdown_connections 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ffa6c078b90 0x7ffa6c07b0a0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ffa9015e510 0x7ffa9015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffa900cfb60 0x7ffa900cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 --2- 172.21.15.129:0/2367222005 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ffa900b46e0 0x7ffa901f6cc0 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 >> 172.21.15.129:0/2367222005 conn(0x7ffa9006cfc0 msgr2=0x7ffa9015f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:16.875 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 shutdown_connections 2024-09-17T04:09:16.876 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:16.874+0000 7ffa8f200640 1 -- 172.21.15.129:0/2367222005 wait complete. 2024-09-17T04:09:16.897 INFO:tasks.ceph:Scrubbing osd.1 2024-09-17T04:09:16.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 -- 172.21.15.129:0/2754801173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6015f2f0 msgr2=0x7fac600ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 --2- 172.21.15.129:0/2754801173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6015f2f0 0x7fac600ae2b0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7fac600d1080 tx=0x7fac4c02a560 comp rx=0 tx=0).stop 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 -- 172.21.15.129:0/2754801173 shutdown_connections 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 --2- 172.21.15.129:0/2754801173 >> v2:172.21.15.181:3300/0 conn(0x7fac600ae7f0 0x7fac600cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 --2- 172.21.15.129:0/2754801173 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac6015f2f0 0x7fac600ae2b0 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 --2- 172.21.15.129:0/2754801173 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac60155660 0x7fac6015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 -- 172.21.15.129:0/2754801173 >> 172.21.15.129:0/2754801173 conn(0x7fac600b1c50 msgr2=0x7fac600b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.122 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.121+0000 7fac5f200640 1 -- 172.21.15.129:0/2754801173 shutdown_connections 2024-09-17T04:09:17.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.122+0000 7fac5f200640 1 -- 172.21.15.129:0/2754801173 wait complete. 2024-09-17T04:09:17.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.124+0000 7fac5f200640 1 Processor -- start 2024-09-17T04:09:17.124 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.124+0000 7fac5f200640 1 -- start start 2024-09-17T04:09:17.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 0x7fac6014fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac60155660 0x7fac601f9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac6015f2f0 0x7fac601f9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fac600ce070 con 0x7fac600ae7f0 2024-09-17T04:09:17.126 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fac600cd2b0 con 0x7fac6015f2f0 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.125+0000 7fac5f200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fac600ce1f0 con 0x7fac60155660 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 0x7fac6014fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 0x7fac6014fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:46990/0 (socket says 172.21.15.129:46990) 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 -- 172.21.15.129:0/2644284416 learned_addr learned my addr 172.21.15.129:0/2644284416 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5d400640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac60155660 0x7fac601f9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5e800640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac6015f2f0 0x7fac601f9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 -- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac60155660 msgr2=0x7fac601f9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fac60155660 0x7fac601f9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 -- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac6015f2f0 msgr2=0x7fac601f9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fac6015f2f0 0x7fac601f9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac601f9f10 con 0x7fac600ae7f0 2024-09-17T04:09:17.127 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.126+0000 7fac5de00640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 0x7fac6014fc80 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7fac5000eb30 tx=0x7fac5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.127+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50004b20 con 0x7fac600ae7f0 2024-09-17T04:09:17.128 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.127+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fac50004ce0 con 0x7fac600ae7f0 2024-09-17T04:09:17.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.127+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50005280 con 0x7fac600ae7f0 2024-09-17T04:09:17.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.127+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac600b3ec0 con 0x7fac600ae7f0 2024-09-17T04:09:17.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.127+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6015fee0 con 0x7fac600ae7f0 2024-09-17T04:09:17.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.128+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac601501c0 con 0x7fac600ae7f0 2024-09-17T04:09:17.129 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.129+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fac50005420 con 0x7fac600ae7f0 2024-09-17T04:09:17.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.130+0000 7fac56000640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac40078bc0 0x7fac4007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.130+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fac5009b610 con 0x7fac600ae7f0 2024-09-17T04:09:17.130 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.130+0000 7fac56000640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fac4007ead0 0x7fac40080f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.130+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac50019e70 con 0x7fac4007ead0 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.130+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7fac500a2050 con 0x7fac600ae7f0 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.131+0000 7fac5d400640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac40078bc0 0x7fac4007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.131+0000 7fac5e800640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fac4007ead0 0x7fac40080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.131+0000 7fac5e800640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fac4007ead0 0x7fac40080f20 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.131 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.131+0000 7fac5d400640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac40078bc0 0x7fac4007b0d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fac4c033000 tx=0x7fac4c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.132 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.132+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== osd.1 v2:172.21.15.129:6800/4215860840 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac50019e70 con 0x7fac4007ead0 2024-09-17T04:09:17.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.145+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 --> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fac60160180 con 0x7fac4007ead0 2024-09-17T04:09:17.145 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.145+0000 7fac56000640 1 -- 172.21.15.129:0/2644284416 <== osd.1 v2:172.21.15.129:6800/4215860840 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fac60160180 con 0x7fac4007ead0 2024-09-17T04:09:17.146 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:17.146 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:17.146 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fac4007ead0 msgr2=0x7fac40080f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:6800/4215860840,v1:172.21.15.129:6801/4215860840] conn(0x7fac4007ead0 0x7fac40080f20 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac40078bc0 msgr2=0x7fac4007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fac40078bc0 0x7fac4007b0d0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fac4c033000 tx=0x7fac4c004730 comp rx=0 tx=0).stop 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 msgr2=0x7fac6014fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 --2- 172.21.15.129:0/2644284416 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fac600ae7f0 0x7fac6014fc80 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7fac5000eb30 tx=0x7fac5000c2d0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5e800640 1 -- 172.21.15.129:0/2644284416 reap_dead start 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 shutdown_connections 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 >> 172.21.15.129:0/2644284416 conn(0x7fac600b1c50 msgr2=0x7fac600afba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 shutdown_connections 2024-09-17T04:09:17.147 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.146+0000 7fac5f200640 1 -- 172.21.15.129:0/2644284416 wait complete. 2024-09-17T04:09:17.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 -- 172.21.15.129:0/1116827265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd40705a0 msgr2=0x7f5cd40b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 --2- 172.21.15.129:0/1116827265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd40705a0 0x7f5cd40b4140 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005fd0 tx=0x7f5cc402a940 comp rx=0 tx=0).stop 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 -- 172.21.15.129:0/1116827265 shutdown_connections 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 --2- 172.21.15.129:0/1116827265 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd40705a0 0x7f5cd40b4140 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 -- 172.21.15.129:0/1116827265 >> 172.21.15.129:0/1116827265 conn(0x7f5cd406d410 msgr2=0x7f5cd40b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 -- 172.21.15.129:0/1116827265 shutdown_connections 2024-09-17T04:09:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.385+0000 7f5cda600640 1 -- 172.21.15.129:0/1116827265 wait complete. 2024-09-17T04:09:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.387+0000 7f5cda600640 1 Processor -- start 2024-09-17T04:09:17.387 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.388+0000 7f5cda600640 1 -- start start 2024-09-17T04:09:17.390 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5cd40705a0 0x7f5cd41f8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5cd4157890 0x7f5cd41f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd406f2e0 con 0x7f5cd406eed0 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd41612e0 con 0x7f5cd40705a0 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cda600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd406f460 con 0x7f5cd4157890 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cd3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cd3400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47014/0 (socket says 172.21.15.129:47014) 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cd3400640 1 -- 172.21.15.129:0/1778220075 learned_addr learned my addr 172.21.15.129:0/1778220075 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.389+0000 7f5cd3e00640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5cd4157890 0x7f5cd41f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5ccaa00640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5cd40705a0 0x7f5cd41f8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.391 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 -- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5cd4157890 msgr2=0x7f5cd41f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5cd4157890 0x7f5cd41f91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 -- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5cd40705a0 msgr2=0x7f5cd41f8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5cd40705a0 0x7f5cd41f8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 -- 172.21.15.129:0/1778220075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd40c7600 con 0x7f5cd406eed0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cd3400640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005fd0 tx=0x7f5cc4031e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc403e070 con 0x7f5cd406eed0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cc4004540 con 0x7f5cd406eed0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.390+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd41f9a70 con 0x7f5cd406eed0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.391+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4043460 con 0x7f5cd406eed0 2024-09-17T04:09:17.392 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.391+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd415d2e0 con 0x7f5cd406eed0 2024-09-17T04:09:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.392+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd40b4140 con 0x7f5cd406eed0 2024-09-17T04:09:17.393 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.393+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5cc403a440 con 0x7f5cd406eed0 2024-09-17T04:09:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.394+0000 7f5ccbe00640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 0x7f5cb007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.394+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f5cc40bc6d0 con 0x7f5cd406eed0 2024-09-17T04:09:17.394 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.395+0000 7f5ccaa00640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 0x7f5cb007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.395 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.395+0000 7f5ccaa00640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 0x7f5cb007b0a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5cd41f8be0 tx=0x7f5cbc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.398 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.398+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5cc4084720 con 0x7f5cd406eed0 2024-09-17T04:09:17.580 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.580+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5cd406a140 con 0x7f5cb0078b90 2024-09-17T04:09:17.581 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.581+0000 7f5ccbe00640 1 -- 172.21.15.129:0/1778220075 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5cd406a140 con 0x7f5cb0078b90 2024-09-17T04:09:17.581 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 1 to deep-scrub 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 msgr2=0x7f5cb007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 0x7f5cb007b0a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f5cd41f8be0 tx=0x7f5cbc009040 comp rx=0 tx=0).stop 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 msgr2=0x7f5cd40c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4005fd0 tx=0x7f5cc4031e20 comp rx=0 tx=0).stop 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 shutdown_connections 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5cb0078b90 0x7f5cb007b0a0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.584 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5cd4157890 0x7f5cd41f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5cd40705a0 0x7f5cd41f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 --2- 172.21.15.129:0/1778220075 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f5cd406eed0 0x7f5cd40c58f0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 >> 172.21.15.129:0/1778220075 conn(0x7f5cd406d410 msgr2=0x7f5cd40b1d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 shutdown_connections 2024-09-17T04:09:17.585 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.584+0000 7f5cda600640 1 -- 172.21.15.129:0/1778220075 wait complete. 2024-09-17T04:09:17.607 INFO:tasks.ceph:Scrubbing osd.2 2024-09-17T04:09:17.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 -- 172.21.15.129:0/671747183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9a8003f90 msgr2=0x7fd9a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 --2- 172.21.15.129:0/671747183 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9a8003f90 0x7fd9a8004800 secure :-1 s=READY pgs=3055 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009b20 tx=0x7fd9a8005830 comp rx=0 tx=0).stop 2024-09-17T04:09:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9c8c00640 1 -- 172.21.15.129:0/671747183 reap_dead start 2024-09-17T04:09:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 -- 172.21.15.129:0/671747183 shutdown_connections 2024-09-17T04:09:17.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 -- 172.21.15.129:0/671747183 >> 172.21.15.129:0/671747183 conn(0x7fd9c40ccc40 msgr2=0x7fd9c40cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 -- 172.21.15.129:0/671747183 shutdown_connections 2024-09-17T04:09:17.835 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.834+0000 7fd9cac00640 1 -- 172.21.15.129:0/671747183 wait complete. 2024-09-17T04:09:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.836+0000 7fd9cac00640 1 Processor -- start 2024-09-17T04:09:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.836+0000 7fd9cac00640 1 -- start start 2024-09-17T04:09:17.838 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9a8003f90 0x7fd9c409d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9c415c730 0x7fd9c4154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 0x7fd9c40b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c40b1dc0 con 0x7fd9c40b0da0 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c40b1c40 con 0x7fd9c415c730 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9cac00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c40b1f40 con 0x7fd9a8003f90 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 0x7fd9c40b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 0x7fd9c40b11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47020/0 (socket says 172.21.15.129:47020) 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 -- 172.21.15.129:0/2385186501 learned_addr learned my addr 172.21.15.129:0/2385186501 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c3400640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9c415c730 0x7fd9c4154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c3e00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9a8003f90 0x7fd9c409d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 -- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9a8003f90 msgr2=0x7fd9c409d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fd9a8003f90 0x7fd9c409d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 -- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9c415c730 msgr2=0x7fd9c4154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9c415c730 0x7fd9c4154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.838+0000 7fd9c8c00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c40b16f0 con 0x7fd9c40b0da0 2024-09-17T04:09:17.840 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9c8c00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 0x7fd9c40b11b0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001eb0 tx=0x7fd9bc006e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc011660 con 0x7fd9c40b0da0 2024-09-17T04:09:17.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd9bc017070 con 0x7fd9c40b0da0 2024-09-17T04:09:17.841 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c409ca10 con 0x7fd9c40b0da0 2024-09-17T04:09:17.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc011bb0 con 0x7fd9c40b0da0 2024-09-17T04:09:17.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.839+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c409bf30 con 0x7fd9c40b0da0 2024-09-17T04:09:17.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.840+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9c40cb130 con 0x7fd9c40b0da0 2024-09-17T04:09:17.842 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.841+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fd9bc01c460 con 0x7fd9c40b0da0 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.842+0000 7fd9bbe00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9a4078b90 0x7fd9a407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.842+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fd9bc09aed0 con 0x7fd9c40b0da0 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9bbe00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fd9a407eaa0 0x7fd9a4080ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9a40815c0 con 0x7fd9a407eaa0 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7fd9bc09d4f0 con 0x7fd9c40b0da0 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9c3e00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9a4078b90 0x7fd9a407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9c3400640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fd9a407eaa0 0x7fd9a4080ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:17.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.843+0000 7fd9c3400640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fd9a407eaa0 0x7fd9a4080ef0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.844+0000 7fd9c3e00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9a4078b90 0x7fd9a407b0a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4001b40 tx=0x7fd9b4003ae0 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:17.844 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.844+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== osd.2 v2:172.21.15.129:6809/283507149 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9a40815c0 con 0x7fd9a407eaa0 2024-09-17T04:09:17.857 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.857+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 --> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd9c409d710 con 0x7fd9a407eaa0 2024-09-17T04:09:17.858 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9bbe00640 1 -- 172.21.15.129:0/2385186501 <== osd.2 v2:172.21.15.129:6809/283507149 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd9c409d710 con 0x7fd9a407eaa0 2024-09-17T04:09:17.859 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:17.859 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:17.859 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fd9a407eaa0 msgr2=0x7fd9a4080ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:6809/283507149,v1:172.21.15.129:6811/283507149] conn(0x7fd9a407eaa0 0x7fd9a4080ef0 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9a4078b90 msgr2=0x7fd9a407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fd9a4078b90 0x7fd9a407b0a0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4001b40 tx=0x7fd9b4003ae0 comp rx=0 tx=0).stop 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 msgr2=0x7fd9c40b11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:17.860 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 --2- 172.21.15.129:0/2385186501 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fd9c40b0da0 0x7fd9c40b11b0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001eb0 tx=0x7fd9bc006e20 comp rx=0 tx=0).stop 2024-09-17T04:09:17.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9c8c00640 1 -- 172.21.15.129:0/2385186501 reap_dead start 2024-09-17T04:09:17.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 shutdown_connections 2024-09-17T04:09:17.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.858+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 >> 172.21.15.129:0/2385186501 conn(0x7fd9c40ccc40 msgr2=0x7fd9c40b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:17.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.859+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 shutdown_connections 2024-09-17T04:09:17.861 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:17.859+0000 7fd9cac00640 1 -- 172.21.15.129:0/2385186501 wait complete. 2024-09-17T04:09:17.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-17T04:09:18.102 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 -- 172.21.15.129:0/1345929862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 msgr2=0x7f9d4c0d4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1345929862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0d4540 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0098e0 tx=0x7f9d3c02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 -- 172.21.15.129:0/1345929862 shutdown_connections 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1345929862 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 0x7f9d4c0d0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1345929862 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0d4540 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 -- 172.21.15.129:0/1345929862 >> 172.21.15.129:0/1345929862 conn(0x7f9d4c09d8d0 msgr2=0x7f9d4c08bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 -- 172.21.15.129:0/1345929862 shutdown_connections 2024-09-17T04:09:18.103 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.102+0000 7f9d51e00640 1 -- 172.21.15.129:0/1345929862 wait complete. 2024-09-17T04:09:18.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.104+0000 7f9d51e00640 1 Processor -- start 2024-09-17T04:09:18.104 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.105+0000 7f9d51e00640 1 -- start start 2024-09-17T04:09:18.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0a2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.106 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 0x7f9d4c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c0ba040 con 0x7f9d4c0d1500 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c0b9ec0 con 0x7f9d4c0b3220 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d51e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f9d4c0d0ae0 con 0x7f9d4c0d4a80 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d4be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d4be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47034/0 (socket says 172.21.15.129:47034) 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.106+0000 7f9d4be00640 1 -- 172.21.15.129:0/1110712836 learned_addr learned my addr 172.21.15.129:0/1110712836 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:18.107 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d50a00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0a2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.108 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d51400640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 0x7f9d4c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 -- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 msgr2=0x7f9d4c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 0x7f9d4c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 -- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 msgr2=0x7f9d4c0a2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0a2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 -- 172.21.15.129:0/1110712836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1f5b90 con 0x7f9d4c0d1500 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d4be00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800e7c0 tx=0x7f9d3800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800ccc0 con 0x7f9d4c0d1500 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d38010430 con 0x7f9d4c0d1500 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.107+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c1f5e70 con 0x7f9d4c0d1500 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.108+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38010860 con 0x7f9d4c0d1500 2024-09-17T04:09:18.109 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.108+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c160c50 con 0x7f9d4c0d1500 2024-09-17T04:09:18.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.109+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c1f5900 con 0x7f9d4c0d1500 2024-09-17T04:09:18.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.110+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f9d38010a00 con 0x7f9d4c0d1500 2024-09-17T04:09:18.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.111+0000 7f9d48c00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 0x7f9d2c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.111+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f9d3809b400 con 0x7f9d4c0d1500 2024-09-17T04:09:18.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.111+0000 7f9d50a00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 0x7f9d2c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.112+0000 7f9d50a00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 0x7f9d2c07b0a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c034000 tx=0x7f9d3c004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.115+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f9d380634e0 con 0x7f9d4c0d1500 2024-09-17T04:09:18.295 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.295+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f9d4c1f6390 con 0x7f9d2c078b90 2024-09-17T04:09:18.296 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.296+0000 7f9d48c00640 1 -- 172.21.15.129:0/1110712836 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9d4c1f6390 con 0x7f9d2c078b90 2024-09-17T04:09:18.296 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 2 to deep-scrub 2024-09-17T04:09:18.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 msgr2=0x7f9d2c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 0x7f9d2c07b0a0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c034000 tx=0x7f9d3c004730 comp rx=0 tx=0).stop 2024-09-17T04:09:18.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 msgr2=0x7f9d4c160710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800e7c0 tx=0x7f9d3800ec80 comp rx=0 tx=0).stop 2024-09-17T04:09:18.299 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 shutdown_connections 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f9d2c078b90 0x7f9d2c07b0a0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f9d4c0d4a80 0x7f9d4c0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f9d4c0d1500 0x7f9d4c160710 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 --2- 172.21.15.129:0/1110712836 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9d4c0b3220 0x7f9d4c0a2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 >> 172.21.15.129:0/1110712836 conn(0x7f9d4c09d8d0 msgr2=0x7f9d4c0b8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 shutdown_connections 2024-09-17T04:09:18.300 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.299+0000 7f9d51e00640 1 -- 172.21.15.129:0/1110712836 wait complete. 2024-09-17T04:09:18.320 INFO:tasks.ceph:Scrubbing osd.3 2024-09-17T04:09:18.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:18.545 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 -- 172.21.15.129:0/3666288431 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a00cd0a0 msgr2=0x7f25a0160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 --2- 172.21.15.129:0/3666288431 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a00cd0a0 0x7f25a0160560 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f258c009860 tx=0x7f258c02a5a0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 -- 172.21.15.129:0/3666288431 shutdown_connections 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 --2- 172.21.15.129:0/3666288431 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a00cd0a0 0x7f25a0160560 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 -- 172.21.15.129:0/3666288431 >> 172.21.15.129:0/3666288431 conn(0x7f25a00b1c50 msgr2=0x7f25a00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 -- 172.21.15.129:0/3666288431 shutdown_connections 2024-09-17T04:09:18.546 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.545+0000 7f25a5e00640 1 -- 172.21.15.129:0/3666288431 wait complete. 2024-09-17T04:09:18.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.547+0000 7f25a5e00640 1 Processor -- start 2024-09-17T04:09:18.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.547+0000 7f25a5e00640 1 -- start start 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25a00cd0a0 0x7f25a00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a015ca40 0x7f25a01f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 0x7f25a00b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f25a00c62c0 con 0x7f25a0160aa0 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f25a00c6140 con 0x7f25a00cd0a0 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5e00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f25a00c6440 con 0x7f25a015ca40 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 0x7f25a00b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 0x7f25a00b0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47038/0 (socket says 172.21.15.129:47038) 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a5400640 1 -- 172.21.15.129:0/2455124768 learned_addr learned my addr 172.21.15.129:0/2455124768 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:18.550 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f25a4a00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25a00cd0a0 0x7f25a00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.549+0000 7f259fe00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a015ca40 0x7f25a01f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 -- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a015ca40 msgr2=0x7f25a01f5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f25a015ca40 0x7f25a01f5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 -- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25a00cd0a0 msgr2=0x7f25a00cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f25a00cd0a0 0x7f25a00cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a00ce0f0 con 0x7f25a0160aa0 2024-09-17T04:09:18.551 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5400640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 0x7f25a00b0cb0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f259800cb10 tx=0x7f259800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598016020 con 0x7f25a0160aa0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f259800ad80 con 0x7f25a0160aa0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2598012680 con 0x7f25a0160aa0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a00cf4f0 con 0x7f25a0160aa0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.550+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a00cfab0 con 0x7f25a0160aa0 2024-09-17T04:09:18.552 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.551+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25a0160560 con 0x7f25a0160aa0 2024-09-17T04:09:18.553 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.553+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f2598019020 con 0x7f25a0160aa0 2024-09-17T04:09:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.553+0000 7f259cc00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2580078b90 0x7f258007b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f259809af80 con 0x7f25a0160aa0 2024-09-17T04:09:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f259cc00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f258007eaa0 0x7f2580080ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2598019ec0 con 0x7f258007eaa0 2024-09-17T04:09:18.554 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7f259809d5a0 con 0x7f25a0160aa0 2024-09-17T04:09:18.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f25a4a00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2580078b90 0x7f258007b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.554+0000 7f259fe00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f258007eaa0 0x7f2580080ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.555+0000 7f259fe00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f258007eaa0 0x7f2580080ef0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.555+0000 7f25a4a00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2580078b90 0x7f258007b0a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f2594001120 tx=0x7f2594009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.555 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.555+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== osd.3 v2:172.21.15.129:6808/2225231477 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2598019ec0 con 0x7f258007eaa0 2024-09-17T04:09:18.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.568+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 --> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f25a00cf2a0 con 0x7f258007eaa0 2024-09-17T04:09:18.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f259cc00640 1 -- 172.21.15.129:0/2455124768 <== osd.3 v2:172.21.15.129:6808/2225231477 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f25a00cf2a0 con 0x7f258007eaa0 2024-09-17T04:09:18.569 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:18.569 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:18.569 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f258007eaa0 msgr2=0x7f2580080ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:6808/2225231477,v1:172.21.15.129:6810/2225231477] conn(0x7f258007eaa0 0x7f2580080ef0 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2580078b90 msgr2=0x7f258007b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f2580078b90 0x7f258007b0a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f2594001120 tx=0x7f2594009040 comp rx=0 tx=0).stop 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 msgr2=0x7f25a00b0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.570 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 --2- 172.21.15.129:0/2455124768 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f25a0160aa0 0x7f25a00b0cb0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f259800cb10 tx=0x7f259800a060 comp rx=0 tx=0).stop 2024-09-17T04:09:18.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5400640 1 -- 172.21.15.129:0/2455124768 reap_dead start 2024-09-17T04:09:18.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 shutdown_connections 2024-09-17T04:09:18.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.569+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 >> 172.21.15.129:0/2455124768 conn(0x7f25a00b1c50 msgr2=0x7f25a0075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:18.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.570+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 shutdown_connections 2024-09-17T04:09:18.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.570+0000 7f25a5e00640 1 -- 172.21.15.129:0/2455124768 wait complete. 2024-09-17T04:09:18.590 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-17T04:09:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 -- 172.21.15.129:0/2704889121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 msgr2=0x7f6edc005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 --2- 172.21.15.129:0/2704889121 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6edc005150 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40752d0 tx=0x7f6edc005830 comp rx=0 tx=0).stop 2024-09-17T04:09:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efa000640 1 -- 172.21.15.129:0/2704889121 reap_dead start 2024-09-17T04:09:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 -- 172.21.15.129:0/2704889121 shutdown_connections 2024-09-17T04:09:18.813 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 -- 172.21.15.129:0/2704889121 >> 172.21.15.129:0/2704889121 conn(0x7f6ef40b2df0 msgr2=0x7f6ef40b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:18.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 -- 172.21.15.129:0/2704889121 shutdown_connections 2024-09-17T04:09:18.814 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.813+0000 7f6efc000640 1 -- 172.21.15.129:0/2704889121 wait complete. 2024-09-17T04:09:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.815+0000 7f6efc000640 1 Processor -- start 2024-09-17T04:09:18.815 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.815+0000 7f6efc000640 1 -- start start 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.816+0000 7f6efc000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.816+0000 7f6efc000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ef4145400 0x7f6ef415d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6efc000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ef415d640 0x7f6ef4147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6efc000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef406ce70 con 0x7f6edc004d40 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6efc000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef40b66c0 con 0x7f6ef415d640 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6efc000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef406cff0 con 0x7f6ef4145400 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47066/0 (socket says 172.21.15.129:47066) 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 -- 172.21.15.129:0/3523937151 learned_addr learned my addr 172.21.15.129:0/3523937151 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:18.818 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6efa000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ef415d640 0x7f6ef4147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef8c00640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ef4145400 0x7f6ef415d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 -- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ef4145400 msgr2=0x7f6ef415d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ef4145400 0x7f6ef415d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 -- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ef415d640 msgr2=0x7f6ef4147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ef415d640 0x7f6ef4147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 -- 172.21.15.129:0/3523937151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef415c0d0 con 0x7f6edc004d40 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.817+0000 7f6ef9600640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006de0 tx=0x7f6ee80049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.818+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8015020 con 0x7f6edc004d40 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.818+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ee8011a40 con 0x7f6edc004d40 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.818+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef415da50 con 0x7f6edc004d40 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.818+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee80103c0 con 0x7f6edc004d40 2024-09-17T04:09:18.819 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.818+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef415c620 con 0x7f6edc004d40 2024-09-17T04:09:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.819+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef40b68d0 con 0x7f6edc004d40 2024-09-17T04:09:18.820 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.820+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f6ee8018020 con 0x7f6edc004d40 2024-09-17T04:09:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.821+0000 7f6ef1600640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 0x7f6ed407b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.821+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f6ee809b7c0 con 0x7f6edc004d40 2024-09-17T04:09:18.821 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.821+0000 7f6ef8c00640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 0x7f6ed407b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.822+0000 7f6ef8c00640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 0x7f6ed407b0d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f6ef415c000 tx=0x7f6ee0005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:18.825 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:18.825+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f6ee801f020 con 0x7f6edc004d40 2024-09-17T04:09:19.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.009+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f6ef40b0300 con 0x7f6ed4078bc0 2024-09-17T04:09:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.010+0000 7f6ef1600640 1 -- 172.21.15.129:0/3523937151 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6ef40b0300 con 0x7f6ed4078bc0 2024-09-17T04:09:19.010 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 3 to deep-scrub 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.012+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 msgr2=0x7f6ed407b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.012+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 0x7f6ed407b0d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f6ef415c000 tx=0x7f6ee0005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.012+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 msgr2=0x7f6ef4158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.012+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8006de0 tx=0x7f6ee80049d0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 shutdown_connections 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f6ed4078bc0 0x7f6ed407b0d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.013 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ef415d640 0x7f6ef4147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f6ef4145400 0x7f6ef415d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 --2- 172.21.15.129:0/3523937151 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f6edc004d40 0x7f6ef4158960 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 >> 172.21.15.129:0/3523937151 conn(0x7f6ef40b2df0 msgr2=0x7f6ef40718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 shutdown_connections 2024-09-17T04:09:19.014 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.013+0000 7f6efc000640 1 -- 172.21.15.129:0/3523937151 wait complete. 2024-09-17T04:09:19.038 INFO:tasks.ceph:Scrubbing osd.4 2024-09-17T04:09:19.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:19.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.264+0000 7fae30400640 1 -- 172.21.15.129:0/131374460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae10004800 msgr2=0x7fae10004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.264+0000 7fae30400640 1 --2- 172.21.15.129:0/131374460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae10004800 0x7fae10004c10 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7fae14009b50 tx=0x7fae10005830 comp rx=0 tx=0).stop 2024-09-17T04:09:19.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.265+0000 7fae30400640 1 -- 172.21.15.129:0/131374460 shutdown_connections 2024-09-17T04:09:19.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.265+0000 7fae30400640 1 --2- 172.21.15.129:0/131374460 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae10004800 0x7fae10004c10 unknown :-1 s=CLOSED pgs=3064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.265+0000 7fae30400640 1 -- 172.21.15.129:0/131374460 >> 172.21.15.129:0/131374460 conn(0x7fae280ccc40 msgr2=0x7fae280cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.265+0000 7fae30400640 1 -- 172.21.15.129:0/131374460 shutdown_connections 2024-09-17T04:09:19.266 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.265+0000 7fae30400640 1 -- 172.21.15.129:0/131374460 wait complete. 2024-09-17T04:09:19.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.267+0000 7fae30400640 1 Processor -- start 2024-09-17T04:09:19.267 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.267+0000 7fae30400640 1 -- start start 2024-09-17T04:09:19.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.268+0000 7fae30400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10004800 0x7fae2809d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae30400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae280cecb0 0x7fae2809da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae30400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 0x7fae281f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae30400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fae280b4d90 con 0x7fae280cdd90 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae30400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fae280b4c10 con 0x7fae10004800 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae30400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fae280b1e20 con 0x7fae280cecb0 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 0x7fae281f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 0x7fae281f4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47090/0 (socket says 172.21.15.129:47090) 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 -- 172.21.15.129:0/3431730406 learned_addr learned my addr 172.21.15.129:0/3431730406 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2da00640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10004800 0x7fae2809d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2d000640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae280cecb0 0x7fae2809da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 -- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae280cecb0 msgr2=0x7fae2809da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fae280cecb0 0x7fae2809da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 -- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10004800 msgr2=0x7fae2809d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10004800 0x7fae2809d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.269+0000 7fae2e400640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae280cf490 con 0x7fae280cdd90 2024-09-17T04:09:19.271 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae2e400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 0x7fae281f4b60 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fae20006fd0 tx=0x7fae20006720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000fc90 con 0x7fae280cdd90 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fae2000fe50 con 0x7fae280cdd90 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae280cf770 con 0x7fae280cdd90 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20011690 con 0x7fae280cdd90 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.270+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2809b1f0 con 0x7fae280cdd90 2024-09-17T04:09:19.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.271+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fae280b2380 con 0x7fae280cdd90 2024-09-17T04:09:19.273 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.272+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fae20011830 con 0x7fae280cdd90 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.273+0000 7fae25600640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078bc0 0x7fae0807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.273+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fae2009be90 con 0x7fae280cdd90 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae25600640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fae0807ead0 0x7fae08080f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae2000ae50 con 0x7fae0807ead0 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7fae2009c240 con 0x7fae280cdd90 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae2da00640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078bc0 0x7fae0807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae2d000640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fae0807ead0 0x7fae08080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.274+0000 7fae2da00640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078bc0 0x7fae0807b0d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fae1c000c00 tx=0x7fae1c006b70 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.275 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.275+0000 7fae2d000640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fae0807ead0 0x7fae08080f20 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.276+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== osd.4 v2:172.21.15.181:6802/2229307524 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fae2000ae50 con 0x7fae0807ead0 2024-09-17T04:09:19.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.289+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 --> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fae280cf240 con 0x7fae0807ead0 2024-09-17T04:09:19.289 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae25600640 1 -- 172.21.15.129:0/3431730406 <== osd.4 v2:172.21.15.181:6802/2229307524 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fae280cf240 con 0x7fae0807ead0 2024-09-17T04:09:19.290 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:19.290 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:19.290 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fae0807ead0 msgr2=0x7fae08080f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6802/2229307524,v1:172.21.15.181:6803/2229307524] conn(0x7fae0807ead0 0x7fae08080f20 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078bc0 msgr2=0x7fae0807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fae08078bc0 0x7fae0807b0d0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fae1c000c00 tx=0x7fae1c006b70 comp rx=0 tx=0).stop 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 msgr2=0x7fae281f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 --2- 172.21.15.129:0/3431730406 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fae280cdd90 0x7fae281f4b60 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7fae20006fd0 tx=0x7fae20006720 comp rx=0 tx=0).stop 2024-09-17T04:09:19.291 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae2e400640 1 -- 172.21.15.129:0/3431730406 reap_dead start 2024-09-17T04:09:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 shutdown_connections 2024-09-17T04:09:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 >> 172.21.15.129:0/3431730406 conn(0x7fae280ccc40 msgr2=0x7fae2815c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 shutdown_connections 2024-09-17T04:09:19.292 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.290+0000 7fae30400640 1 -- 172.21.15.129:0/3431730406 wait complete. 2024-09-17T04:09:19.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-17T04:09:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 -- 172.21.15.129:0/2505018608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53740cfb00 msgr2=0x7f537415bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 --2- 172.21.15.129:0/2505018608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53740cfb00 0x7f537415bca0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f53740727e0 tx=0x7f536002a540 comp rx=0 tx=0).stop 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 -- 172.21.15.129:0/2505018608 shutdown_connections 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 --2- 172.21.15.129:0/2505018608 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f53740cfb00 0x7f537415bca0 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 --2- 172.21.15.129:0/2505018608 >> v2:172.21.15.181:3300/0 conn(0x7f53741524c0 0x7f53740cf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 -- 172.21.15.129:0/2505018608 >> 172.21.15.129:0/2505018608 conn(0x7f537406cfc0 msgr2=0x7f537406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 -- 172.21.15.129:0/2505018608 shutdown_connections 2024-09-17T04:09:19.535 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.534+0000 7f537b000640 1 -- 172.21.15.129:0/2505018608 wait complete. 2024-09-17T04:09:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.536+0000 7f537b000640 1 Processor -- start 2024-09-17T04:09:19.536 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.536+0000 7f537b000640 1 -- start start 2024-09-17T04:09:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53740cfb00 0x7f53741541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f53741524c0 0x7f5374156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f53740d02c0 con 0x7f537415c1e0 2024-09-17T04:09:19.539 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f5374152ae0 con 0x7f53740cfb00 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537b000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f53740d0440 con 0x7f53741524c0 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537a600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47116/0 (socket says 172.21.15.129:47116) 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f537a600640 1 -- 172.21.15.129:0/767835064 learned_addr learned my addr 172.21.15.129:0/767835064 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f5379200640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f53741524c0 0x7f5374156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.538+0000 7f5379c00640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53740cfb00 0x7f53741541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 -- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f53741524c0 msgr2=0x7f5374156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f53741524c0 0x7f5374156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 -- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53740cfb00 msgr2=0x7f53741541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53740cfb00 0x7f53741541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.540 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 -- 172.21.15.129:0/767835064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53741565b0 con 0x7f537415c1e0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537a600640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c6c0 tx=0x7f536c00cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c017070 con 0x7f537415c1e0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f536c003c60 con 0x7f537415c1e0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536c0052d0 con 0x7f537415c1e0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5374156890 con 0x7f537415c1e0 2024-09-17T04:09:19.541 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.539+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53741539f0 con 0x7f537415c1e0 2024-09-17T04:09:19.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.540+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5374156320 con 0x7f537415c1e0 2024-09-17T04:09:19.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.542+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f536c01a070 con 0x7f537415c1e0 2024-09-17T04:09:19.542 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.543+0000 7f5371600640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 0x7f535407b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.543+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f536c09a0e0 con 0x7f537415c1e0 2024-09-17T04:09:19.543 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.543+0000 7f5379c00640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 0x7f535407b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.544 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.544+0000 7f5379c00640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 0x7f535407b0a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5368005fd0 tx=0x7f5368005850 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.547 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.547+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f536c062780 con 0x7f537415c1e0 2024-09-17T04:09:19.734 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.734+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f53740747b0 con 0x7f5354078b90 2024-09-17T04:09:19.735 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.735+0000 7f5371600640 1 -- 172.21.15.129:0/767835064 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f53740747b0 con 0x7f5354078b90 2024-09-17T04:09:19.735 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 4 to deep-scrub 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 msgr2=0x7f535407b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 0x7f535407b0a0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f5368005fd0 tx=0x7f5368005850 comp rx=0 tx=0).stop 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 msgr2=0x7f5374158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f536c00c6c0 tx=0x7f536c00cb80 comp rx=0 tx=0).stop 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 shutdown_connections 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5354078b90 0x7f535407b0a0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f537415c1e0 0x7f5374158a20 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f53741524c0 0x7f5374156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.738 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 --2- 172.21.15.129:0/767835064 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f53740cfb00 0x7f53741541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 >> 172.21.15.129:0/767835064 conn(0x7f537406cfc0 msgr2=0x7f537406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 shutdown_connections 2024-09-17T04:09:19.739 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.737+0000 7f537b000640 1 -- 172.21.15.129:0/767835064 wait complete. 2024-09-17T04:09:19.759 INFO:tasks.ceph:Scrubbing osd.5 2024-09-17T04:09:19.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 -- 172.21.15.129:0/898387856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc0cc170 msgr2=0x7fcfdc0ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 --2- 172.21.15.129:0/898387856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc0ce5f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009830 tx=0x7fcfc402a5c0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 -- 172.21.15.129:0/898387856 shutdown_connections 2024-09-17T04:09:19.985 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 --2- 172.21.15.129:0/898387856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc0ce5f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 -- 172.21.15.129:0/898387856 >> 172.21.15.129:0/898387856 conn(0x7fcfdc0ccc40 msgr2=0x7fcfdc0cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 -- 172.21.15.129:0/898387856 shutdown_connections 2024-09-17T04:09:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.985+0000 7fcfe1800640 1 -- 172.21.15.129:0/898387856 wait complete. 2024-09-17T04:09:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.987+0000 7fcfe1800640 1 Processor -- start 2024-09-17T04:09:19.987 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.987+0000 7fcfe1800640 1 -- start start 2024-09-17T04:09:19.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc070b20 0x7fcfdc09b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.990 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc1590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfdc0ceb30 0x7fcfdc074130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0cb490 con 0x7fcfdc0cc170 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc0cb310 con 0x7fcfdc070b20 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfe1800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fcfdc166eb0 con 0x7fcfdc0ceb30 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc1590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc1590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47128/0 (socket says 172.21.15.129:47128) 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 -- 172.21.15.129:0/3936191274 learned_addr learned my addr 172.21.15.129:0/3936191274 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfdaa00640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc070b20 0x7fcfdc09b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfdb400640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfdc0ceb30 0x7fcfdc074130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 -- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfdc0ceb30 msgr2=0x7fcfdc074130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fcfdc0ceb30 0x7fcfdc074130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.991 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 -- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc070b20 msgr2=0x7fcfdc09b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcfdc070b20 0x7fcfdc09b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.989+0000 7fcfd2000640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc1f3870 con 0x7fcfdc0cc170 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfd2000640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc1590a0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fcfc402abb0 tx=0x7fcfc4033ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4031b50 con 0x7fcfdc0cc170 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fcfc4031de0 con 0x7fcfdc0cc170 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc1f3b50 con 0x7fcfdc0cc170 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4005280 con 0x7fcfdc0cc170 2024-09-17T04:09:19.992 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.990+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc15bcf0 con 0x7fcfdc0cc170 2024-09-17T04:09:19.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.991+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfdc0ce5f0 con 0x7fcfdc0cc170 2024-09-17T04:09:19.993 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.992+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fcfc4005420 con 0x7fcfdc0cc170 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.993+0000 7fcfd3400640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb8078bc0 0x7fcfb807b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.993+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fcfc40bbb20 con 0x7fcfdc0cc170 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfd3400640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfb807ead0 0x7fcfb8080f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfb80815f0 con 0x7fcfb807ead0 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfdaa00640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb8078bc0 0x7fcfb807b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7fcfc40c02a0 con 0x7fcfdc0cc170 2024-09-17T04:09:19.994 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfdb400640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfb807ead0 0x7fcfb8080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:19.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.994+0000 7fcfdaa00640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb8078bc0 0x7fcfb807b0d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc007fd0 tx=0x7fcfcc001370 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.995 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.995+0000 7fcfdb400640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfb807ead0 0x7fcfb8080f20 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:19.996 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:19.996+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== osd.5 v2:172.21.15.181:6818/3094080342 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfb80815f0 con 0x7fcfb807ead0 2024-09-17T04:09:20.008 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.008+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 --> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcfdc09b890 con 0x7fcfb807ead0 2024-09-17T04:09:20.009 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfd3400640 1 -- 172.21.15.129:0/3936191274 <== osd.5 v2:172.21.15.181:6818/3094080342 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcfdc09b890 con 0x7fcfb807ead0 2024-09-17T04:09:20.009 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:20.009 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:20.009 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfb807ead0 msgr2=0x7fcfb8080f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6818/3094080342,v1:172.21.15.181:6819/3094080342] conn(0x7fcfb807ead0 0x7fcfb8080f20 crc :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.010 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb8078bc0 msgr2=0x7fcfb807b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fcfb8078bc0 0x7fcfb807b0d0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc007fd0 tx=0x7fcfcc001370 comp rx=0 tx=0).stop 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 msgr2=0x7fcfdc1590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfe1800640 1 --2- 172.21.15.129:0/3936191274 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fcfdc0cc170 0x7fcfdc1590a0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fcfc402abb0 tx=0x7fcfc4033ba0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.009+0000 7fcfdb400640 1 -- 172.21.15.129:0/3936191274 reap_dead start 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.010+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 shutdown_connections 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.010+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 >> 172.21.15.129:0/3936191274 conn(0x7fcfdc0ccc40 msgr2=0x7fcfdc0b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.010+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 shutdown_connections 2024-09-17T04:09:20.011 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.010+0000 7fcfe1800640 1 -- 172.21.15.129:0/3936191274 wait complete. 2024-09-17T04:09:20.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-17T04:09:20.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:20.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.262+0000 7f0b7e200640 1 -- 172.21.15.129:0/1122963636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b2950 msgr2=0x7f0b780b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.262+0000 7f0b7e200640 1 --2- 172.21.15.129:0/1122963636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b2950 0x7f0b780b3480 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f0b7806dd20 tx=0x7f0b6402a560 comp rx=0 tx=0).stop 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 -- 172.21.15.129:0/1122963636 shutdown_connections 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 --2- 172.21.15.129:0/1122963636 >> v2:172.21.15.181:3300/0 conn(0x7f0b780b39c0 0x7f0b7806eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 --2- 172.21.15.129:0/1122963636 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b2950 0x7f0b780b3480 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 --2- 172.21.15.129:0/1122963636 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780ce170 0x7f0b780cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.263 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 -- 172.21.15.129:0/1122963636 >> 172.21.15.129:0/1122963636 conn(0x7f0b7806d410 msgr2=0x7f0b780b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:20.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 -- 172.21.15.129:0/1122963636 shutdown_connections 2024-09-17T04:09:20.264 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.263+0000 7f0b7e200640 1 -- 172.21.15.129:0/1122963636 wait complete. 2024-09-17T04:09:20.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.265+0000 7f0b7e200640 1 Processor -- start 2024-09-17T04:09:20.265 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.265+0000 7f0b7e200640 1 -- start start 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780b2950 0x7f0b780ad050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b780ce170 0x7f0b7809da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0b78070850 con 0x7f0b780b39c0 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0b780706d0 con 0x7f0b780ce170 2024-09-17T04:09:20.268 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b7e200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7806f410 con 0x7f0b780b2950 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47154/0 (socket says 172.21.15.129:47154) 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 -- 172.21.15.129:0/3134153780 learned_addr learned my addr 172.21.15.129:0/3134153780 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b77e00640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b780ce170 0x7f0b7809da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b77400640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780b2950 0x7f0b780ad050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 -- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780b2950 msgr2=0x7f0b780ad050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780b2950 0x7f0b780ad050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 -- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b780ce170 msgr2=0x7f0b7809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b780ce170 0x7f0b7809da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.267+0000 7f0b76a00640 1 -- 172.21.15.129:0/3134153780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b781547c0 con 0x7f0b780b39c0 2024-09-17T04:09:20.269 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b76a00640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b640038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6403e030 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0b64003ca0 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7809dfa0 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64035040 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.268+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b78154a40 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.270+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b781f6900 con 0x7f0b780b39c0 2024-09-17T04:09:20.270 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.271+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0b64041030 con 0x7f0b780b39c0 2024-09-17T04:09:20.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.271+0000 7f0b6f400640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 0x7f0b5807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.272+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f0b640bb380 con 0x7f0b780b39c0 2024-09-17T04:09:20.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.272+0000 7f0b77400640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 0x7f0b5807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.272 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.273+0000 7f0b77400640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 0x7f0b5807b0a0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009b30 tx=0x7f0b68005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:20.276 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.276+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0b781547c0 con 0x7f0b780b39c0 2024-09-17T04:09:20.488 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.488+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f0b7806f8f0 con 0x7f0b58078b90 2024-09-17T04:09:20.489 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.489+0000 7f0b6f400640 1 -- 172.21.15.129:0/3134153780 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0b7806f8f0 con 0x7f0b58078b90 2024-09-17T04:09:20.489 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 5 to deep-scrub 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.492+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 msgr2=0x7f0b5807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.492+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 0x7f0b5807b0a0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009b30 tx=0x7f0b68005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.492+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 msgr2=0x7f0b78152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.492+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7f0b64000c00 tx=0x7f0b640038a0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 shutdown_connections 2024-09-17T04:09:20.493 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0b58078b90 0x7f0b5807b0a0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b780ce170 0x7f0b7809da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0b780b39c0 0x7f0b78152ec0 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 --2- 172.21.15.129:0/3134153780 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0b780b2950 0x7f0b780ad050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 >> 172.21.15.129:0/3134153780 conn(0x7f0b7806d410 msgr2=0x7f0b78074140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 shutdown_connections 2024-09-17T04:09:20.494 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.493+0000 7f0b7e200640 1 -- 172.21.15.129:0/3134153780 wait complete. 2024-09-17T04:09:20.530 INFO:tasks.ceph:Scrubbing osd.6 2024-09-17T04:09:20.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.765+0000 7fa166800640 1 -- 172.21.15.129:0/2916575198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa160074580 msgr2=0x7fa1600b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.765+0000 7fa166800640 1 --2- 172.21.15.129:0/2916575198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa160074580 0x7fa1600b49a0 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7fa148009830 tx=0x7fa14802a530 comp rx=0 tx=0).stop 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.765+0000 7fa166800640 1 -- 172.21.15.129:0/2916575198 shutdown_connections 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.765+0000 7fa166800640 1 --2- 172.21.15.129:0/2916575198 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa160074580 0x7fa1600b49a0 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.765+0000 7fa166800640 1 -- 172.21.15.129:0/2916575198 >> 172.21.15.129:0/2916575198 conn(0x7fa1600ccc40 msgr2=0x7fa1600cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.766+0000 7fa166800640 1 -- 172.21.15.129:0/2916575198 shutdown_connections 2024-09-17T04:09:20.766 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.766+0000 7fa166800640 1 -- 172.21.15.129:0/2916575198 wait complete. 2024-09-17T04:09:20.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.768+0000 7fa166800640 1 Processor -- start 2024-09-17T04:09:20.768 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.768+0000 7fa166800640 1 -- start start 2024-09-17T04:09:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 0x7fa16014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1600cba90 0x7fa1600ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa16009dba0 0x7fa1600adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fa160167f30 con 0x7fa1600cba90 2024-09-17T04:09:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fa1600cc9b0 con 0x7fa16009dba0 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.769+0000 7fa166800640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fa1601680b0 con 0x7fa160074580 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa157e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1600cba90 0x7fa1600ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 0x7fa16014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa16009dba0 0x7fa1600adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 0x7fa16014e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3301/0 says I am v2:172.21.15.129:51814/0 (socket says 172.21.15.129:51814) 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 -- 172.21.15.129:0/2364696638 learned_addr learned my addr 172.21.15.129:0/2364696638 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 -- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa16009dba0 msgr2=0x7fa1600adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa16009dba0 0x7fa1600adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 -- 172.21.15.129:0/2364696638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1600cba90 msgr2=0x7fa1600ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fa1600cba90 0x7fa1600ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.771 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.770+0000 7fa15f400640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16014ebf0 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa15f400640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 0x7fa16014e6b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa150001220 tx=0x7fa15000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150011790 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa150017070 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa150011cc0 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1601556b0 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.771+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa160151eb0 con 0x7fa160074580 2024-09-17T04:09:20.773 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.772+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa160151c60 con 0x7fa160074580 2024-09-17T04:09:20.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.773+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fa15001c460 con 0x7fa160074580 2024-09-17T04:09:20.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.774+0000 7fa157400640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa13c078b90 0x7fa13c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fa15009b2b0 con 0x7fa160074580 2024-09-17T04:09:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157400640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa13c07eaa0 0x7fa13c080ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa15000ae50 con 0x7fa13c07eaa0 2024-09-17T04:09:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== mon.2 v2:172.21.15.181:3301/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7fa15009b660 con 0x7fa160074580 2024-09-17T04:09:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157e00640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa13c078b90 0x7fa13c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa15fe00640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa13c07eaa0 0x7fa13c080ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:20.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.775+0000 7fa157e00640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa13c078b90 0x7fa13c07b0a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fa148034000 tx=0x7fa148004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:20.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.776+0000 7fa15fe00640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa13c07eaa0 0x7fa13c080ef0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:20.777 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.777+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== osd.6 v2:172.21.15.181:6810/4136500368 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa15000ae50 con 0x7fa13c07eaa0 2024-09-17T04:09:20.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.791+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 --> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa1600b3f70 con 0x7fa13c07eaa0 2024-09-17T04:09:20.791 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.791+0000 7fa157400640 1 -- 172.21.15.129:0/2364696638 <== osd.6 v2:172.21.15.181:6810/4136500368 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa1600b3f70 con 0x7fa13c07eaa0 2024-09-17T04:09:20.792 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:20.792 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:20.792 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.791+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa13c07eaa0 msgr2=0x7fa13c080ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.791+0000 7fa166800640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6810/4136500368,v1:172.21.15.181:6811/4136500368] conn(0x7fa13c07eaa0 0x7fa13c080ef0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa13c078b90 msgr2=0x7fa13c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fa13c078b90 0x7fa13c07b0a0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fa148034000 tx=0x7fa148004730 comp rx=0 tx=0).stop 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 msgr2=0x7fa16014e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 --2- 172.21.15.129:0/2364696638 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fa160074580 0x7fa16014e6b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa150001220 tx=0x7fa15000c890 comp rx=0 tx=0).stop 2024-09-17T04:09:20.793 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa15fe00640 1 -- 172.21.15.129:0/2364696638 reap_dead start 2024-09-17T04:09:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 shutdown_connections 2024-09-17T04:09:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 >> 172.21.15.129:0/2364696638 conn(0x7fa1600ccc40 msgr2=0x7fa160072de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 shutdown_connections 2024-09-17T04:09:20.794 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:20.792+0000 7fa166800640 1 -- 172.21.15.129:0/2364696638 wait complete. 2024-09-17T04:09:20.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.039+0000 7f50b9c00640 1 -- 172.21.15.129:0/627866228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b415a220 msgr2=0x7f50b415d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.039+0000 7f50b9c00640 1 --2- 172.21.15.129:0/627866228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b415a220 0x7f50b415d660 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f50a4005fd0 tx=0x7f50a402a8b0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 -- 172.21.15.129:0/627866228 shutdown_connections 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 --2- 172.21.15.129:0/627866228 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 0x7f50b40cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 --2- 172.21.15.129:0/627866228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b415a220 0x7f50b415d660 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 -- 172.21.15.129:0/627866228 >> 172.21.15.129:0/627866228 conn(0x7f50b4069150 msgr2=0x7f50b4069590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 -- 172.21.15.129:0/627866228 shutdown_connections 2024-09-17T04:09:21.041 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.040+0000 7f50b9c00640 1 -- 172.21.15.129:0/627866228 wait complete. 2024-09-17T04:09:21.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.042+0000 7f50b9c00640 1 Processor -- start 2024-09-17T04:09:21.042 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.042+0000 7f50b9c00640 1 -- start start 2024-09-17T04:09:21.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.045 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 0x7f50b40b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 0x7f50b40b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f50b40cf6f0 con 0x7f50b415a220 2024-09-17T04:09:21.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f50b40cf570 con 0x7f50b41606c0 2024-09-17T04:09:21.046 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9c00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f50b41f5c60 con 0x7f50b41602b0 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47184/0 (socket says 172.21.15.129:47184) 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 -- 172.21.15.129:0/2705509179 learned_addr learned my addr 172.21.15.129:0/2705509179 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50ab400640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 0x7f50b40b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9200640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 0x7f50b40b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 -- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 msgr2=0x7f50b40b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 0x7f50b40b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 -- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 msgr2=0x7f50b40b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 0x7f50b40b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.047 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b3e00640 1 -- 172.21.15.129:0/2705509179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b41f7b70 con 0x7f50b415a220 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.044+0000 7f50b9200640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 0x7f50b40b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b3e00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f50a4000f80 tx=0x7f50a40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a4031990 con 0x7f50b415a220 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f50a4031c20 con 0x7f50b415a220 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b40b6ef0 con 0x7f50b415a220 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50a40332d0 con 0x7f50b415a220 2024-09-17T04:09:21.048 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.045+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b40b74b0 con 0x7f50b415a220 2024-09-17T04:09:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.046+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b40d0990 con 0x7f50b415a220 2024-09-17T04:09:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.048+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f50a4033470 con 0x7f50b415a220 2024-09-17T04:09:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.048+0000 7f50b0c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 0x7f509007b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.049+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f50a40bbe60 con 0x7f50b415a220 2024-09-17T04:09:21.049 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.049+0000 7f50ab400640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 0x7f509007b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.050+0000 7f50ab400640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 0x7f509007b0d0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f50b40cf220 tx=0x7f509c005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.053 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.053+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f50a403f030 con 0x7f50b415a220 2024-09-17T04:09:21.243 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.243+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f50b40b6c70 con 0x7f5090078bc0 2024-09-17T04:09:21.244 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.244+0000 7f50b0c00640 1 -- 172.21.15.129:0/2705509179 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f509c011c00 con 0x7f5090078bc0 2024-09-17T04:09:21.244 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 6 to deep-scrub 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.246+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 msgr2=0x7f509007b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.246+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 0x7f509007b0d0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f50b40cf220 tx=0x7f509c005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.246+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 msgr2=0x7f50b40a94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.246+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f50a4000f80 tx=0x7f50a40316f0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 shutdown_connections 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5090078bc0 0x7f509007b0d0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.247 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50b41606c0 0x7f50b40b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f50b41602b0 0x7f50b40b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 --2- 172.21.15.129:0/2705509179 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f50b415a220 0x7f50b40a94d0 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 >> 172.21.15.129:0/2705509179 conn(0x7f50b4069150 msgr2=0x7f50b415c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 shutdown_connections 2024-09-17T04:09:21.248 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.247+0000 7f50b9c00640 1 -- 172.21.15.129:0/2705509179 wait complete. 2024-09-17T04:09:21.270 INFO:tasks.ceph:Scrubbing osd.7 2024-09-17T04:09:21.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-17T04:09:21.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 -- 172.21.15.129:0/848259823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 msgr2=0x7f10fc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.499 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 --2- 172.21.15.129:0/848259823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 0x7f10fc0cc990 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f10ec005fd0 tx=0x7f10ec02a5d0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 -- 172.21.15.129:0/848259823 shutdown_connections 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 --2- 172.21.15.129:0/848259823 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10fc160c70 0x7f10fc0c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 --2- 172.21.15.129:0/848259823 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 0x7f10fc0cc990 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 -- 172.21.15.129:0/848259823 >> 172.21.15.129:0/848259823 conn(0x7f10fc0b1c50 msgr2=0x7f10fc0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 -- 172.21.15.129:0/848259823 shutdown_connections 2024-09-17T04:09:21.500 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.499+0000 7f10fb200640 1 -- 172.21.15.129:0/848259823 wait complete. 2024-09-17T04:09:21.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.501+0000 7f10fb200640 1 Processor -- start 2024-09-17T04:09:21.501 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.501+0000 7f10fb200640 1 -- start start 2024-09-17T04:09:21.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10fc0cced0 0x7f10fc1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.504 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 0x7f10fc15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 0x7f10fc15b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc0c6070 con 0x7f10fc160c70 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f10fc0c5ef0 con 0x7f10fc15ca40 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fb200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f10fc0c61f0 con 0x7f10fc0cced0 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 0x7f10fc15b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10f9e00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10fc0cced0 0x7f10fc1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fa800640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 0x7f10fc15b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47188/0 (socket says 172.21.15.129:47188) 2024-09-17T04:09:21.505 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10fa800640 1 -- 172.21.15.129:0/3283367263 learned_addr learned my addr 172.21.15.129:0/3283367263 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.503+0000 7f10f9400640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 0x7f10fc15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 -- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10fc0cced0 msgr2=0x7f10fc1520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f10fc0cced0 0x7f10fc1520c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 -- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 msgr2=0x7f10fc15b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10fc15ca40 0x7f10fc15b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10fc074a70 con 0x7f10fc160c70 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fa800640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 0x7f10fc15b6e0 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f10f400ebf0 tx=0x7f10f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4010630 con 0x7f10fc160c70 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10fc074f70 con 0x7f10fc160c70 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f10f40107d0 con 0x7f10fc160c70 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f4010c20 con 0x7f10fc160c70 2024-09-17T04:09:21.506 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.504+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10fc159540 con 0x7f10fc160c70 2024-09-17T04:09:21.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.505+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10fc0c6710 con 0x7f10fc160c70 2024-09-17T04:09:21.507 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.507+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f10f4021430 con 0x7f10fc160c70 2024-09-17T04:09:21.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f2000640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078b90 0x7f10dc07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f10f409b8e0 con 0x7f10fc160c70 2024-09-17T04:09:21.508 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f2000640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f10dc07eaa0 0x7f10dc080ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f40047f0 con 0x7f10dc07eaa0 2024-09-17T04:09:21.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_get_version_reply(handle=1 version=277) ==== 24+0+0 (secure 0 0 0) 0x7f10f40a02a0 con 0x7f10fc160c70 2024-09-17T04:09:21.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f9e00640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078b90 0x7f10dc07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.509 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.508+0000 7f10f9400640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f10dc07eaa0 0x7f10dc080ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.509+0000 7f10f9400640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f10dc07eaa0 0x7f10dc080ef0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.509+0000 7f10f9e00640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078b90 0x7f10dc07b0a0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f10ec033000 tx=0x7f10ec004730 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.510 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.510+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== osd.7 v2:172.21.15.181:6826/580738384 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10f40047f0 con 0x7f10dc07eaa0 2024-09-17T04:09:21.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.523+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 --> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f10fc074d20 con 0x7f10dc07eaa0 2024-09-17T04:09:21.523 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.523+0000 7f10f2000640 1 -- 172.21.15.129:0/3283367263 <== osd.7 v2:172.21.15.181:6826/580738384 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f10fc074d20 con 0x7f10dc07eaa0 2024-09-17T04:09:21.523 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-17T04:09:21.524 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-17T04:09:21.524 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f10dc07eaa0 msgr2=0x7f10dc080ef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6826/580738384,v1:172.21.15.181:6827/580738384] conn(0x7f10dc07eaa0 0x7f10dc080ef0 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078b90 msgr2=0x7f10dc07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f10dc078b90 0x7f10dc07b0a0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f10ec033000 tx=0x7f10ec004730 comp rx=0 tx=0).stop 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 msgr2=0x7f10fc15b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 --2- 172.21.15.129:0/3283367263 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f10fc160c70 0x7f10fc15b6e0 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f10f400ebf0 tx=0x7f10f400c6a0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fa800640 1 -- 172.21.15.129:0/3283367263 reap_dead start 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 shutdown_connections 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 >> 172.21.15.129:0/3283367263 conn(0x7f10fc0b1c50 msgr2=0x7f10fc0b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 shutdown_connections 2024-09-17T04:09:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.524+0000 7f10fb200640 1 -- 172.21.15.129:0/3283367263 wait complete. 2024-09-17T04:09:21.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 -- 172.21.15.129:0/2485060892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 msgr2=0x7ff9b80cd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 --2- 172.21.15.129:0/2485060892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80cd1c0 secure :-1 s=READY pgs=3078 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80ccc10 tx=0x7ff9b002aa30 comp rx=0 tx=0).stop 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 -- 172.21.15.129:0/2485060892 shutdown_connections 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 --2- 172.21.15.129:0/2485060892 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80cd1c0 unknown :-1 s=CLOSED pgs=3078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 --2- 172.21.15.129:0/2485060892 >> v2:172.21.15.181:3300/0 conn(0x7ff9b806eed0 0x7ff9b80ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 --2- 172.21.15.129:0/2485060892 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b80705a0 0x7ff9b80b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 -- 172.21.15.129:0/2485060892 >> 172.21.15.129:0/2485060892 conn(0x7ff9b806d410 msgr2=0x7ff9b80b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.774 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 -- 172.21.15.129:0/2485060892 shutdown_connections 2024-09-17T04:09:21.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.773+0000 7ff9bd600640 1 -- 172.21.15.129:0/2485060892 wait complete. 2024-09-17T04:09:21.775 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.776+0000 7ff9bd600640 1 Processor -- start 2024-09-17T04:09:21.776 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.776+0000 7ff9bd600640 1 -- start start 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b806eed0 0x7ff9b80c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9b80705a0 0x7ff9b80c6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b8160b40 con 0x7ff9b806f2e0 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b81609c0 con 0x7ff9b80705a0 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9bd600640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b8160cc0 con 0x7ff9b806eed0 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9b6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9b6000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47198/0 (socket says 172.21.15.129:47198) 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9b6000640 1 -- 172.21.15.129:0/357183539 learned_addr learned my addr 172.21.15.129:0/357183539 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:21.779 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.777+0000 7ff9b6a00640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b806eed0 0x7ff9b80c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b7400640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9b80705a0 0x7ff9b80c6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 -- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b806eed0 msgr2=0x7ff9b80c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b806eed0 0x7ff9b80c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 -- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9b80705a0 msgr2=0x7ff9b80c6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9b80705a0 0x7ff9b80c6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 -- 172.21.15.129:0/357183539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b80c9940 con 0x7ff9b806f2e0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9b6000640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7ff9a400e6c0 tx=0x7ff9a400eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a400cab0 con 0x7ff9b806f2e0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff9a400cc50 con 0x7ff9b806f2e0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b80c9c20 con 0x7ff9b806f2e0 2024-09-17T04:09:21.780 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.778+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a4005280 con 0x7ff9b806f2e0 2024-09-17T04:09:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.779+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b80ca1e0 con 0x7ff9b806f2e0 2024-09-17T04:09:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.780+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b8161220 con 0x7ff9b806f2e0 2024-09-17T04:09:21.781 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.781+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7ff9a4005420 con 0x7ff9b806f2e0 2024-09-17T04:09:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.782+0000 7ff9aea00640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 0x7ff99807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.782+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7ff9a409b260 con 0x7ff9b806f2e0 2024-09-17T04:09:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.782+0000 7ff9b6a00640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 0x7ff99807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:21.782 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.783+0000 7ff9b6a00640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 0x7ff99807b0a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009b30 tx=0x7ff9a8005f90 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:21.786 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.786+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7ff9a4005710 con 0x7ff9b806f2e0 2024-09-17T04:09:21.969 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.969+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7ff9b806a140 con 0x7ff998078b90 2024-09-17T04:09:21.970 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.969+0000 7ff9aea00640 1 -- 172.21.15.129:0/357183539 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9b806a140 con 0x7ff998078b90 2024-09-17T04:09:21.970 INFO:teuthology.orchestra.run.smithi129.stderr:instructed osd(s) 7 to deep-scrub 2024-09-17T04:09:21.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 msgr2=0x7ff99807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 0x7ff99807b0a0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009b30 tx=0x7ff9a8005f90 comp rx=0 tx=0).stop 2024-09-17T04:09:21.972 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 msgr2=0x7ff9b80c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7ff9a400e6c0 tx=0x7ff9a400eb80 comp rx=0 tx=0).stop 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 shutdown_connections 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7ff998078b90 0x7ff99807b0a0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff9b80705a0 0x7ff9b80c6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7ff9b806f2e0 0x7ff9b80c6720 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 --2- 172.21.15.129:0/357183539 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7ff9b806eed0 0x7ff9b80c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 >> 172.21.15.129:0/357183539 conn(0x7ff9b806d410 msgr2=0x7ff9b80aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 shutdown_connections 2024-09-17T04:09:21.973 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:21.972+0000 7ff9bd600640 1 -- 172.21.15.129:0/357183539 wait complete. 2024-09-17T04:09:21.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:09:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 -- 172.21.15.129:0/797959707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 msgr2=0x7fe9a00cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:22.213 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 --2- 172.21.15.129:0/797959707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a00cd860 secure :-1 s=READY pgs=3081 cs=0 l=1 rev1=1 crypto rx=0x7fe990009830 tx=0x7fe99002a5c0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 -- 172.21.15.129:0/797959707 shutdown_connections 2024-09-17T04:09:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 --2- 172.21.15.129:0/797959707 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a00cd860 unknown :-1 s=CLOSED pgs=3081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 -- 172.21.15.129:0/797959707 >> 172.21.15.129:0/797959707 conn(0x7fe9a006cfc0 msgr2=0x7fe9a006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 -- 172.21.15.129:0/797959707 shutdown_connections 2024-09-17T04:09:22.214 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.213+0000 7fe99fe00640 1 -- 172.21.15.129:0/797959707 wait complete. 2024-09-17T04:09:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.215+0000 7fe99fe00640 1 Processor -- start 2024-09-17T04:09:22.215 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.215+0000 7fe99fe00640 1 -- start start 2024-09-17T04:09:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9a00d03a0 0x7fe9a015b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:22.218 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9a0160af0 0x7fe9a015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a015d1f0 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a015d070 con 0x7fe9a0160af0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99fe00640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a015d370 con 0x7fe9a00d03a0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:47222/0 (socket says 172.21.15.129:47222) 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 -- 172.21.15.129:0/3663481285 learned_addr learned my addr 172.21.15.129:0/3663481285 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe996000640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9a00d03a0 0x7fe9a015b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99f400640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9a0160af0 0x7fe9a015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 -- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9a00d03a0 msgr2=0x7fe9a015b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9a00d03a0 0x7fe9a015b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 -- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9a0160af0 msgr2=0x7fe9a015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9a0160af0 0x7fe9a015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.219 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.217+0000 7fe99ea00640 1 -- 172.21.15.129:0/3663481285 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a01e7550 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe99ea00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fe990000c00 tx=0x7fe990033d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990031d60 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe9900330d0 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a01e7830 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990033520 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.220 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.218+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a01f96a0 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.219+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a00cd860 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.220+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7fe9900035f0 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.221+0000 7fe997400640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 0x7fe97c07b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:22.221 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.221+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7fe9900bbf30 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.222+0000 7fe996000640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 0x7fe97c07b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:22.222 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.222+0000 7fe996000640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 0x7fe97c07b0a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe9a01531a0 tx=0x7fe988009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:22.225 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.226+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7fe99003f030 con 0x7fe9a00cf8a0 2024-09-17T04:09:22.405 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.405+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9a01f9b80 con 0x7fe97c078b90 2024-09-17T04:09:22.407 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.406+0000 7fe997400640 1 -- 172.21.15.129:0/3663481285 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13364 (secure 0 0 0) 0x7fe9a01f9b80 con 0x7fe97c078b90 2024-09-17T04:09:22.407 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:22.407 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:09:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 msgr2=0x7fe97c07b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:22.409 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 0x7fe97c07b0a0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fe9a01531a0 tx=0x7fe988009040 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 msgr2=0x7fe9a01f4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7fe990000c00 tx=0x7fe990033d60 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 shutdown_connections 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7fe97c078b90 0x7fe97c07b0a0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe9a0160af0 0x7fe9a015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7fe9a00d03a0 0x7fe9a015b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 --2- 172.21.15.129:0/3663481285 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7fe9a00cf8a0 0x7fe9a01f4f70 unknown :-1 s=CLOSED pgs=3082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 >> 172.21.15.129:0/3663481285 conn(0x7fe9a006cfc0 msgr2=0x7fe9a01569c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 shutdown_connections 2024-09-17T04:09:22.410 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:22.409+0000 7fe99fe00640 1 -- 172.21.15.129:0/3663481285 wait complete. 2024-09-17T04:09:22.433 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":869,"stamp":"2024-09-17T04:09:20.636553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":421144,"kb_used_data":28648,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754553576,"statfs":{"total":773094113280,"available":772662861824,"internally_reserved":0,"allocated":29335552,"data_stored":13687168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.260051"},"pg_stats":[{"pgid":"1.0","version":"18'72","reported_seq":585,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-17T04:09:11.033678+0000","last_change":"2024-09-17T03:49:26.343885+0000","last_active":"2024-09-17T04:09:11.033678+0000","last_peered":"2024-09-17T04:09:11.033678+0000","last_clean":"2024-09-17T04:09:11.033678+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:09:11.033678+0000","last_undegraded":"2024-09-17T04:09:11.033678+0000","last_fullsized":"2024-09-17T04:09:11.033678+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T03:49:25.170847+0000","last_clean_scrub_stamp":"2024-09-17T03:49:25.170847+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T14:58:36.733267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509681,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34624,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337216,"statfs":{"total":96636764160,"available":96601309184,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56836,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509681,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56836,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56832,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52676,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319164,"statfs":{"total":96636764160,"available":96582823936,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56832,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542386,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53252,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318588,"statfs":{"total":96636764160,"available":96582234112,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:09:22.437 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T03:49:25.170847+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=3, tm_min=49, tm_sec=25, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=4, tm_min=9, tm_sec=15, tm_wday=1, tm_yday=261, tm_isdst=0) 2024-09-17T04:09:22.437 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-17T04:09:42.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-17T04:09:42.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 -- 172.21.15.129:0/1892206705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380b3b00 msgr2=0x7f57380ce170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:42.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 --2- 172.21.15.129:0/1892206705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380b3b00 0x7f57380ce170 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f5730005fd0 tx=0x7f573002a670 comp rx=0 tx=0).stop 2024-09-17T04:09:42.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 -- 172.21.15.129:0/1892206705 shutdown_connections 2024-09-17T04:09:42.669 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 --2- 172.21.15.129:0/1892206705 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380b3b00 0x7f57380ce170 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 --2- 172.21.15.129:0/1892206705 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f57380706a0 0x7f57380cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 -- 172.21.15.129:0/1892206705 >> 172.21.15.129:0/1892206705 conn(0x7f573806d410 msgr2=0x7f57380b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:42.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 -- 172.21.15.129:0/1892206705 shutdown_connections 2024-09-17T04:09:42.670 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.669+0000 7f573d000640 1 -- 172.21.15.129:0/1892206705 wait complete. 2024-09-17T04:09:42.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.671+0000 7f573d000640 1 Processor -- start 2024-09-17T04:09:42.671 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.671+0000 7f573d000640 1 -- start start 2024-09-17T04:09:42.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:42.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 0x7f57380c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:42.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f573815dc60 0x7f573815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:42.674 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f57380cec90 con 0x7f57380cc260 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f57380ceb10 con 0x7f573815dc60 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f573d000640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f57380cee10 con 0x7f5738158e00 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45694/0 (socket says 172.21.15.129:45694) 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 -- 172.21.15.129:0/3129947244 learned_addr learned my addr 172.21.15.129:0/3129947244 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735400640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 0x7f57380c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5736800640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f573815dc60 0x7f573815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 -- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 msgr2=0x7f57380c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 0x7f57380c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 -- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f573815dc60 msgr2=0x7f573815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f573815dc60 0x7f573815e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.673+0000 7f5735e00640 1 -- 172.21.15.129:0/3129947244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57380cf540 con 0x7f57380cc260 2024-09-17T04:09:42.675 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f5735400640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 0x7f57380c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f5735e00640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f572800eb30 tx=0x7f572800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728010430 con 0x7f57380cc260 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f57280105d0 con 0x7f57380cc260 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57380c2dc0 con 0x7f57380cc260 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5728010a20 con 0x7f57380cc260 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.674+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57380c2ab0 con 0x7f57380cc260 2024-09-17T04:09:42.676 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.675+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57380cf330 con 0x7f57380cc260 2024-09-17T04:09:42.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.677+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f5728010bc0 con 0x7f57380cc260 2024-09-17T04:09:42.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.677+0000 7f572e000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 0x7f571807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:42.677 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.678+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f572809b880 con 0x7f57380cc260 2024-09-17T04:09:42.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.678+0000 7f5735400640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 0x7f571807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:42.678 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.679+0000 7f5735400640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 0x7f571807b0a0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5724007fd0 tx=0x7f5724009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:42.682 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.682+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f5728063fb0 con 0x7f57380cc260 2024-09-17T04:09:42.864 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.864+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 --> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f57380ad9f0 con 0x7f5718078b90 2024-09-17T04:09:42.866 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.866+0000 7f572e000640 1 -- 172.21.15.129:0/3129947244 <== mgr.4157 v2:172.21.15.181:6800/3833218970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13371 (secure 0 0 0) 0x7f57380ad9f0 con 0x7f5718078b90 2024-09-17T04:09:42.866 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:42.866 INFO:teuthology.orchestra.run.smithi129.stderr:dumped all 2024-09-17T04:09:42.868 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 msgr2=0x7f571807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 0x7f571807b0a0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f5724007fd0 tx=0x7f5724009040 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 msgr2=0x7f57380c3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 secure :-1 s=READY pgs=3084 cs=0 l=1 rev1=1 crypto rx=0x7f572800eb30 tx=0x7f572800c2d0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 shutdown_connections 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f5718078b90 0x7f571807b0a0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f573815dc60 0x7f573815e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f5738158e00 0x7f57380c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 --2- 172.21.15.129:0/3129947244 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f57380cc260 0x7f57380c3190 unknown :-1 s=CLOSED pgs=3084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 >> 172.21.15.129:0/3129947244 conn(0x7f573806d410 msgr2=0x7f57380d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 shutdown_connections 2024-09-17T04:09:42.869 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:42.868+0000 7f573d000640 1 -- 172.21.15.129:0/3129947244 wait complete. 2024-09-17T04:09:42.891 INFO:teuthology.orchestra.run.smithi129.stdout:{"pg_ready":true,"pg_map":{"version":880,"stamp":"2024-09-17T04:09:42.642326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":421132,"kb_used_data":28648,"kb_used_omap":12,"kb_used_meta":392307,"kb_avail":754553588,"statfs":{"total":773094113280,"available":772662874112,"internally_reserved":0,"allocated":29335552,"data_stored":13687168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":401722928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.0","version":"277'74","reported_seq":596,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-17T04:09:22.429527+0000","last_change":"2024-09-17T04:09:22.429456+0000","last_active":"2024-09-17T04:09:22.429527+0000","last_peered":"2024-09-17T04:09:22.429527+0000","last_clean":"2024-09-17T04:09:22.429527+0000","last_became_active":"2024-09-17T03:49:26.339996+0000","last_became_peered":"2024-09-17T03:49:26.339996+0000","last_unstale":"2024-09-17T04:09:22.429527+0000","last_undegraded":"2024-09-17T04:09:22.429527+0000","last_fullsized":"2024-09-17T04:09:22.429527+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'74","last_scrub_stamp":"2024-09-17T04:09:22.429370+0000","last_deep_scrub":"277'74","last_deep_scrub_stamp":"2024-09-17T04:09:22.429370+0000","last_clean_scrub_stamp":"2024-09-17T04:09:22.429370+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:41:10.342004+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509685,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53248,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318592,"statfs":{"total":96636764160,"available":96582238208,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509688,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":34624,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337216,"statfs":{"total":96636764160,"available":96601309184,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509687,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56836,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56836,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509687,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56832,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509687,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":52676,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319164,"statfs":{"total":96636764160,"available":96582823936,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542389,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56832,"kb_used_data":3436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":3518464,"data_stored":1563304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542390,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53248,"kb_used_data":4016,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318592,"statfs":{"total":96636764160,"available":96582238208,"internally_reserved":0,"allocated":4112384,"data_stored":2153672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T04:09:42.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-17T04:09:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 -- 172.21.15.129:0/1767223222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 msgr2=0x7f0a580af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.110 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 --2- 172.21.15.129:0/1767223222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a580af480 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f0a580d1260 tx=0x7f0a4c02a5f0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 -- 172.21.15.129:0/1767223222 shutdown_connections 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 --2- 172.21.15.129:0/1767223222 >> v2:172.21.15.181:3300/0 conn(0x7f0a580b3010 0x7f0a580b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 --2- 172.21.15.129:0/1767223222 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a580af480 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 -- 172.21.15.129:0/1767223222 >> 172.21.15.129:0/1767223222 conn(0x7f0a580ccc40 msgr2=0x7f0a580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 -- 172.21.15.129:0/1767223222 shutdown_connections 2024-09-17T04:09:43.111 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.110+0000 7f0a60400640 1 -- 172.21.15.129:0/1767223222 wait complete. 2024-09-17T04:09:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.112+0000 7f0a60400640 1 Processor -- start 2024-09-17T04:09:43.112 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.112+0000 7f0a60400640 1 -- start start 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a580b1c40 0x7f0a5814c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 0x7f0a5814caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580b3ae0 con 0x7f0a58074800 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580b3960 con 0x7f0a580b1c40 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a60400640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7f0a580b3c60 con 0x7f0a580b3010 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.115 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45714/0 (socket says 172.21.15.129:45714) 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 -- 172.21.15.129:0/3304213033 learned_addr learned my addr 172.21.15.129:0/3304213033 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5e400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 0x7f0a5814caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5d000640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a580b1c40 0x7f0a5814c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 -- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 msgr2=0x7f0a5814caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 0x7f0a5814caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 -- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a580b1c40 msgr2=0x7f0a5814c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a580b1c40 0x7f0a5814c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5da00640 1 -- 172.21.15.129:0/3304213033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5815f5b0 con 0x7f0a58074800 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.114+0000 7f0a5e400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 0x7f0a5814caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a5da00640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c034040 tx=0x7f0a4c0039e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c03e030 con 0x7f0a58074800 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0a4c0047a0 con 0x7f0a58074800 2024-09-17T04:09:43.116 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5815f890 con 0x7f0a58074800 2024-09-17T04:09:43.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c035040 con 0x7f0a58074800 2024-09-17T04:09:43.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.115+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5814c040 con 0x7f0a58074800 2024-09-17T04:09:43.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.116+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a580b41c0 con 0x7f0a58074800 2024-09-17T04:09:43.117 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.117+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7f0a4c041030 con 0x7f0a58074800 2024-09-17T04:09:43.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.118+0000 7f0a55600640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 0x7f0a3807b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.118 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.119+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7f0a4c0bb370 con 0x7f0a58074800 2024-09-17T04:09:43.119 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.119+0000 7f0a5d000640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 0x7f0a3807b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.120 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.120+0000 7f0a5d000640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 0x7f0a3807b0a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0a44000f60 tx=0x7f0a44009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:43.123 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.123+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7f0a4c083aa0 con 0x7f0a58074800 2024-09-17T04:09:43.309 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.309+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f0a580c9f10 con 0x7f0a58074800 2024-09-17T04:09:43.316 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.316+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v75)=0 v75) ==== 123+0+0 (secure 0 0 0) 0x7f0a4c0833a0 con 0x7f0a58074800 2024-09-17T04:09:43.318 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.318+0000 7f0a55600640 1 -- 172.21.15.129:0/3304213033 <== mon.0 v2:172.21.15.129:3300/0 8 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f0a4c082eb0 con 0x7f0a58074800 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 msgr2=0x7f0a3807b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 0x7f0a3807b0a0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f0a44000f60 tx=0x7f0a44009040 comp rx=0 tx=0).stop 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 msgr2=0x7f0a5814b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c034040 tx=0x7f0a4c0039e0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 shutdown_connections 2024-09-17T04:09:43.320 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7f0a38078b90 0x7f0a3807b0a0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7f0a580b3010 0x7f0a5814caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0a580b1c40 0x7f0a5814c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 --2- 172.21.15.129:0/3304213033 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7f0a58074800 0x7f0a5814b3c0 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 >> 172.21.15.129:0/3304213033 conn(0x7f0a580ccc40 msgr2=0x7f0a580cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 shutdown_connections 2024-09-17T04:09:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.320+0000 7f0a60400640 1 -- 172.21.15.129:0/3304213033 wait complete. 2024-09-17T04:09:43.346 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-17T04:09:43.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 -- 172.21.15.129:0/3900435629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0b3b90 msgr2=0x7faf0c0cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 --2- 172.21.15.129:0/3900435629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0b3b90 0x7faf0c0cbbf0 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7faf0c072e00 tx=0x7faef802a5f0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 -- 172.21.15.129:0/3900435629 shutdown_connections 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 --2- 172.21.15.129:0/3900435629 >> v2:172.21.15.181:3300/0 conn(0x7faf0c0b2950 0x7faf0c06eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 --2- 172.21.15.129:0/3900435629 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0b3b90 0x7faf0c0cbbf0 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.566+0000 7faf12200640 1 -- 172.21.15.129:0/3900435629 >> 172.21.15.129:0/3900435629 conn(0x7faf0c06d410 msgr2=0x7faf0c0b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:43.567 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.567+0000 7faf12200640 1 -- 172.21.15.129:0/3900435629 shutdown_connections 2024-09-17T04:09:43.568 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.567+0000 7faf12200640 1 -- 172.21.15.129:0/3900435629 wait complete. 2024-09-17T04:09:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.569+0000 7faf12200640 1 Processor -- start 2024-09-17T04:09:43.569 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.569+0000 7faf12200640 1 -- start start 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf0c0b2950 0x7faf0c1f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 --2- >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf0c0ce170 0x7faf0c098e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 -- --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c06f410 con 0x7faf0c0cccf0 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c070910 con 0x7faf0c0b2950 2024-09-17T04:09:43.571 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.570+0000 7faf12200640 1 -- --> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] -- mon_getmap magic: 0 -- 0x7faf0c06f590 con 0x7faf0c0ce170 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 --2- >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.129:3300/0 says I am v2:172.21.15.129:45742/0 (socket says 172.21.15.129:45742) 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 -- 172.21.15.129:0/1755465356 learned_addr learned my addr 172.21.15.129:0/1755465356 (peer_addr_for_me v2:172.21.15.129:0/0) 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0b400640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf0c0b2950 0x7faf0c1f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0aa00640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf0c0ce170 0x7faf0c098e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 -- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf0c0ce170 msgr2=0x7faf0c098e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf0c0ce170 0x7faf0c098e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 -- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf0c0b2950 msgr2=0x7faf0c1f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf0c0b2950 0x7faf0c1f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 -- 172.21.15.129:0/1755465356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c0cd640 con 0x7faf0c0cccf0 2024-09-17T04:09:43.572 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.571+0000 7faf0be00640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7faf0400ea40 tx=0x7faf0400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.572+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0400ce40 con 0x7faf0c0cccf0 2024-09-17T04:09:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.572+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 2 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7faf04004c30 con 0x7faf0c0cccf0 2024-09-17T04:09:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.572+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c097da0 con 0x7faf0c0cccf0 2024-09-17T04:09:43.573 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.572+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf04005020 con 0x7faf0c0cccf0 2024-09-17T04:09:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.572+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c098360 con 0x7faf0c0cccf0 2024-09-17T04:09:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.573+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0c076480 con 0x7faf0c0cccf0 2024-09-17T04:09:43.574 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.574+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 4 ==== mgrmap(e 5) ==== 101441+0+0 (secure 0 0 0) 0x7faf040051c0 con 0x7faf0c0cccf0 2024-09-17T04:09:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.575+0000 7faf03400640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 0x7faeec07b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-17T04:09:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.575+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 5 ==== osd_map(277..277 src has 1..277) ==== 4943+0+0 (secure 0 0 0) 0x7faf0409b760 con 0x7faf0c0cccf0 2024-09-17T04:09:43.575 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.575+0000 7faf0b400640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 0x7faeec07b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-17T04:09:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.576+0000 7faf0b400640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 0x7faeec07b0d0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7faefc001720 tx=0x7faefc009040 comp rx=0 tx=0).ready entity=mgr.4157 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-17T04:09:43.579 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.579+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195964 (secure 0 0 0) 0x7faf0c076480 con 0x7faf0c0cccf0 2024-09-17T04:09:43.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.797+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 --> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf0c06a140 con 0x7faf0c0cccf0 2024-09-17T04:09:43.797 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.798+0000 7faf03400640 1 -- 172.21.15.129:0/1755465356 <== mon.0 v2:172.21.15.129:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 275 v275) ==== 95+0+3928 (secure 0 0 0) 0x7faf0c0cd640 con 0x7faf0c0cccf0 2024-09-17T04:09:43.798 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-17T04:09:43.798 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":275,"btime":"2024-09-17T04:09:08:348664+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9791,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6836/500055228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":500055228},{"type":"v1","addr":"172.21.15.181:6836","nonce":500055228}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12663,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/3506137801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6835","nonce":3506137801},{"type":"v1","addr":"172.21.15.181:6837","nonce":3506137801}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":265},{"gid":12678,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6834/1232665008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6832","nonce":1232665008},{"type":"v1","addr":"172.21.15.129:6834","nonce":1232665008}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":266},{"gid":12981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.129:6835/2880549966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6833","nonce":2880549966},{"type":"v1","addr":"172.21.15.129:6835","nonce":2880549966}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":275}],"filesystems":[]} 2024-09-17T04:09:43.798 INFO:teuthology.orchestra.run.smithi129.stderr:dumped fsmap epoch 275 2024-09-17T04:09:43.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 msgr2=0x7faeec07b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.800 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 0x7faeec07b0d0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7faefc001720 tx=0x7faefc009040 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 msgr2=0x7faf0c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 secure :-1 s=READY pgs=3089 cs=0 l=1 rev1=1 crypto rx=0x7faf0400ea40 tx=0x7faf0400ef00 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 shutdown_connections 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:6800/3833218970,v1:172.21.15.181:6801/3833218970] conn(0x7faeec078bc0 0x7faeec07b0d0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] conn(0x7faf0c0cccf0 0x7faf0c0cd100 unknown :-1 s=CLOSED pgs=3089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] conn(0x7faf0c0ce170 0x7faf0c098e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 --2- 172.21.15.129:0/1755465356 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faf0c0b2950 0x7faf0c1f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 >> 172.21.15.129:0/1755465356 conn(0x7faf0c06d410 msgr2=0x7faf0c1e4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 shutdown_connections 2024-09-17T04:09:43.801 INFO:teuthology.orchestra.run.smithi129.stderr:2024-09-17T04:09:43.800+0000 7faf12200640 1 -- 172.21.15.129:0/1755465356 wait complete. 2024-09-17T04:09:43.826 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-17T04:09:43.827 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-17T04:09:43.827 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-17T04:09:43.827 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:43.851 INFO:tasks.ceph.mds.a:Stopped 2024-09-17T04:09:43.851 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-17T04:09:43.851 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:43.917 INFO:tasks.ceph.mds.c:Stopped 2024-09-17T04:09:43.917 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-17T04:09:43.917 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:43.944 INFO:tasks.ceph.mds.b:Stopped 2024-09-17T04:09:43.944 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-17T04:09:43.944 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.008 INFO:tasks.ceph.mds.d:Stopped 2024-09-17T04:09:44.008 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-17T04:09:44.008 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-17T04:09:44.008 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.073 INFO:tasks.ceph.osd.0:Stopped 2024-09-17T04:09:44.073 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-17T04:09:44.073 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.143 INFO:tasks.ceph.osd.1:Stopped 2024-09-17T04:09:44.143 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-17T04:09:44.143 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.210 INFO:tasks.ceph.osd.2:Stopped 2024-09-17T04:09:44.210 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-17T04:09:44.210 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.280 INFO:tasks.ceph.osd.3:Stopped 2024-09-17T04:09:44.281 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-17T04:09:44.281 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.348 INFO:tasks.ceph.osd.4:Stopped 2024-09-17T04:09:44.348 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-17T04:09:44.348 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.424 INFO:tasks.ceph.osd.5:Stopped 2024-09-17T04:09:44.425 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-17T04:09:44.425 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.488 INFO:tasks.ceph.osd.6:Stopped 2024-09-17T04:09:44.488 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-17T04:09:44.489 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.555 INFO:tasks.ceph.osd.7:Stopped 2024-09-17T04:09:44.555 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-17T04:09:44.556 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-17T04:09:44.556 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.628 INFO:tasks.ceph.mgr.y:Stopped 2024-09-17T04:09:44.628 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-17T04:09:44.628 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.706 INFO:tasks.ceph.mgr.x:Stopped 2024-09-17T04:09:44.706 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-17T04:09:44.706 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-17T04:09:44.706 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.738 INFO:tasks.ceph.mon.a:Stopped 2024-09-17T04:09:44.738 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-17T04:09:44.738 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.764 INFO:tasks.ceph.mon.b:Stopped 2024-09-17T04:09:44.764 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-17T04:09:44.764 INFO:teuthology.orchestra.run:waiting for 300 2024-09-17T04:09:44.836 INFO:tasks.ceph.mon.c:Stopped 2024-09-17T04:09:44.837 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-17T04:09:44.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-17T04:09:44.920 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:09:44.920 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-17T04:09:45.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:09:45.141 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-17T04:09:45.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:09:45.261 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-17T04:09:45.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:09:45.371 DEBUG:teuthology.orchestra.run.smithi129:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-17T04:09:45.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T04:09:45.479 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-17T04:09:45.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T04:09:45.635 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-17T04:09:45.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T04:09:45.739 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-17T04:09:45.839 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T04:09:45.839 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-17T04:09:45.948 INFO:tasks.ceph:Archiving mon data... 2024-09-17T04:09:45.950 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/data/mon.a.tgz 2024-09-17T04:09:45.950 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-17T04:09:45.986 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.3O2q5FbFEV 2024-09-17T04:09:45.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3O2q5FbFEV 2024-09-17T04:09:46.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.3O2q5FbFEV 2024-09-17T04:09:46.298 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.3O2q5FbFEV is 546KB 2024-09-17T04:09:46.391 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.3O2q5FbFEV 2024-09-17T04:09:46.426 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/data/mon.b.tgz 2024-09-17T04:09:46.426 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-17T04:09:46.462 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.G1KP3zrpxN 2024-09-17T04:09:46.462 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.G1KP3zrpxN 2024-09-17T04:09:46.676 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.G1KP3zrpxN 2024-09-17T04:09:46.769 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.G1KP3zrpxN is 547KB 2024-09-17T04:09:46.860 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.G1KP3zrpxN 2024-09-17T04:09:46.895 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/data/mon.c.tgz 2024-09-17T04:09:46.895 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-17T04:09:46.969 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.p8tEQ5oiSi 2024-09-17T04:09:46.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.p8tEQ5oiSi 2024-09-17T04:09:47.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.p8tEQ5oiSi 2024-09-17T04:09:47.262 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.p8tEQ5oiSi is 547KB 2024-09-17T04:09:47.352 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.p8tEQ5oiSi 2024-09-17T04:09:47.387 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-17T04:09:47.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T04:09:47.391 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-17T04:09:47.517 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T04:09:47.552 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-17T04:09:47.552 INFO:tasks.ceph:Archiving crash dumps... 2024-09-17T04:09:47.554 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi129/crash 2024-09-17T04:09:47.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T04:09:47.612 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi181/crash 2024-09-17T04:09:47.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-17T04:09:47.666 INFO:tasks.ceph:Shutting down logrotate 2024-09-17T04:09:47.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:47.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:47.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:47.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-17T04:09:47.871 INFO:tasks.ceph:Compressing logs... 2024-09-17T04:09:47.871 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T04:09:47.876 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T04:09:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2024-09-17T04:09:47.926 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-17T04:09:47.927 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54220.log 2024-09-17T04:09:47.927 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2024-09-17T04:09:47.928 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54081.log 2024-09-17T04:09:47.928 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41612.log 2024-09-17T04:09:47.929 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70317.log 2024-09-17T04:09:47.929 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54220.log.gz 2024-09-17T04:09:47.931 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36305.log 2024-09-17T04:09:47.931 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.3.41612.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41612.log.gz 2024-09-17T04:09:47.931 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-17T04:09:47.931 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36505.log 2024-09-17T04:09:47.931 INFO:teuthology.orchestra.run.smithi181.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36305.log.gz 2024-09-17T04:09:47.932 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54081.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54081.log.gz 2024-09-17T04:09:47.932 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67639.log 2024-09-17T04:09:47.932 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57066.log 2024-09-17T04:09:47.933 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70317.log.gz 2024-09-17T04:09:47.933 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58786.log 2024-09-17T04:09:47.933 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67639.log.gz 2024-09-17T04:09:47.933 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51896.log 2024-09-17T04:09:47.935 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57066.log.gz 2024-09-17T04:09:47.935 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58786.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58786.log.gz 2024-09-17T04:09:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56376.log 2024-09-17T04:09:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108778.log 2024-09-17T04:09:47.936 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51896.log.gz 2024-09-17T04:09:47.936 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108166.log 2024-09-17T04:09:47.936 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56376.log.gz 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36505.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36505.log.gz 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36345.log 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36385.log 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36345.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 86.6% 2024-09-17T04:09:47.937 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36345.log.gz 2024-09-17T04:09:47.938 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36385.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36385.log.gz 2024-09-17T04:09:47.938 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T04:09:47.938 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41216.log 2024-09-17T04:09:47.939 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-17T04:09:47.939 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.1.41216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.41216.log.gz 2024-09-17T04:09:47.939 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36265.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46217.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108778.log.gz 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108166.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108166.log.gz 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43951.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71552.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46217.log.gz 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.43951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43951.log.gz 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72246.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44713.log 2024-09-17T04:09:47.942 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71552.log.gz 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46047.log 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72246.log.gz 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45447.log 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44713.log.gz 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63759.log 2024-09-17T04:09:47.943 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46047.log.gz 2024-09-17T04:09:47.946 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36425.log 2024-09-17T04:09:47.946 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36465.log 2024-09-17T04:09:47.946 INFO:teuthology.orchestra.run.smithi181.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.36265.log.gz 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36425.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.36425.log.gz 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36465.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.36465.log.gz 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T04:09:47.947 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.5.log: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T04:09:47.948 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58473.log 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45447.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63759.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63759.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75830.log 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71029.log 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58473.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59606.log 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75830.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68309.log 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.71029.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59606.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59606.log.gz 2024-09-17T04:09:47.949 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51377.log 2024-09-17T04:09:47.950 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84267.log 2024-09-17T04:09:47.950 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68309.log.gz 2024-09-17T04:09:47.950 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73153.log 2024-09-17T04:09:47.950 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51377.log.gz 2024-09-17T04:09:47.952 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-09-17T04:09:47.952 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47452.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.84267.log.gz 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73153.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73153.log.gz 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77551.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53133.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47452.log.gz 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49199.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77551.log.gz 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101759.log 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53133.log.gz 2024-09-17T04:09:47.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49199.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49199.log.gz 2024-09-17T04:09:47.955 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67158.log 2024-09-17T04:09:47.956 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-17T04:09:47.957 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75517.log 2024-09-17T04:09:47.957 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101759.log.gz 2024-09-17T04:09:47.957 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42616.log 2024-09-17T04:09:47.957 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67158.log.gz 2024-09-17T04:09:47.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61541.log 2024-09-17T04:09:47.958 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.75517.log.gz 2024-09-17T04:09:47.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42616.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42616.log.gzgzip 2024-09-17T04:09:47.958 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2024-09-17T04:09:47.959 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41197.log 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49020.log 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61541.log.gz 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72390.log 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87503.log 2024-09-17T04:09:47.960 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49020.log.gz 2024-09-17T04:09:47.962 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36225.log 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72390.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.72390.log.gz 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77119.log 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63573.log 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87503.log.gz 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.77119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.77119.log.gz 2024-09-17T04:09:47.963 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46131.log 2024-09-17T04:09:47.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87255.log 2024-09-17T04:09:47.964 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63573.log.gz 2024-09-17T04:09:47.964 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.1.41197.log: 0.0% 92.6% -- replaced with /var/log/ceph/ceph-client.1.41197.log.gz 2024-09-17T04:09:47.965 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46131.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46131.log.gz 2024-09-17T04:09:47.965 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88079.log 2024-09-17T04:09:47.966 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105494.log 2024-09-17T04:09:47.966 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87255.log.gz 2024-09-17T04:09:47.966 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108250.log 2024-09-17T04:09:47.966 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88079.log.gz 2024-09-17T04:09:47.967 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.36225.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T04:09:47.968 INFO:teuthology.orchestra.run.smithi181.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.36225.log.gz 2024-09-17T04:09:47.968 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41616.log 2024-09-17T04:09:47.968 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-09-17T04:09:47.968 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105494.log.gz 2024-09-17T04:09:47.968 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108250.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108250.log.gz 2024-09-17T04:09:47.969 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71113.log 2024-09-17T04:09:47.969 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105537.log 2024-09-17T04:09:47.969 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz 2024-09-17T04:09:47.971 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35846.log 2024-09-17T04:09:47.971 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71113.log.gz 2024-09-17T04:09:47.971 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2024-09-17T04:09:47.971 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105537.log.gz 2024-09-17T04:09:47.971 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35846.log.gz 2024-09-17T04:09:47.972 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66677.log 2024-09-17T04:09:47.973 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.3.41616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41616.log.gz 2024-09-17T04:09:47.974 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76805.log 2024-09-17T04:09:47.974 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2024-09-17T04:09:47.974 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54262.log 2024-09-17T04:09:47.975 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66677.log.gz 2024-09-17T04:09:47.975 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47390.log 2024-09-17T04:09:47.975 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76805.log.gz 2024-09-17T04:09:47.975 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54262.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54262.log.gz 2024-09-17T04:09:47.975 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36000.log 2024-09-17T04:09:47.976 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36056.log 2024-09-17T04:09:47.976 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47390.log.gz 2024-09-17T04:09:47.976 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87689.log 2024-09-17T04:09:47.976 INFO:teuthology.orchestra.run.smithi129.stderr: 74.2% -- replaced with /var/log/ceph/ceph-client.admin.36000.log.gz 2024-09-17T04:09:47.977 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41845.log 2024-09-17T04:09:47.977 INFO:teuthology.orchestra.run.smithi129.stderr: 80.6% -- replaced with /var/log/ceph/ceph-client.admin.36056.log.gz 2024-09-17T04:09:47.978 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87689.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87689.log.gz 2024-09-17T04:09:47.978 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.101132.log 2024-09-17T04:09:47.978 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95796.log 2024-09-17T04:09:47.978 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41845.log.gz 2024-09-17T04:09:47.979 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64479.log 2024-09-17T04:09:47.979 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101132.log.gz 2024-09-17T04:09:47.980 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62333.log 2024-09-17T04:09:47.980 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95796.log.gz 2024-09-17T04:09:47.981 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64479.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64479.log.gz -5 2024-09-17T04:09:47.981 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48810.log 2024-09-17T04:09:47.981 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99156.log 2024-09-17T04:09:47.982 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62333.log.gz 2024-09-17T04:09:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2024-09-17T04:09:47.983 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48810.log.gz 2024-09-17T04:09:47.983 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107355.log 2024-09-17T04:09:47.984 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5%/var/log/ceph/ceph-client.admin.43337.log: -- replaced with /var/log/ceph/ceph-client.admin.99156.log.gz 2024-09-17T04:09:47.984 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2024-09-17T04:09:47.984 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73277.log 2024-09-17T04:09:47.985 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97239.log 2024-09-17T04:09:47.985 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.107355.log.gz 2024-09-17T04:09:47.986 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82017.log 2024-09-17T04:09:47.986 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73277.log.gz 2024-09-17T04:09:47.987 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.97239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97239.log.gz 2024-09-17T04:09:47.987 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89041.log 2024-09-17T04:09:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104690.log 2024-09-17T04:09:47.988 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82017.log.gz 2024-09-17T04:09:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89041.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89041.log.gz 2024-09-17T04:09:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89466.log 2024-09-17T04:09:47.989 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74711.log 2024-09-17T04:09:47.989 INFO:teuthology.orchestra.run.smithi129.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.104690.log.gz 2024-09-17T04:09:47.990 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105806.log 2024-09-17T04:09:47.990 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89466.log.gz 2024-09-17T04:09:47.991 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75265.log 2024-09-17T04:09:47.991 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74711.log.gz 2024-09-17T04:09:47.992 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105806.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105806.log.gz 2024-09-17T04:09:47.992 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93389.log 2024-09-17T04:09:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53614.log 2024-09-17T04:09:47.993 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75265.log.gz 2024-09-17T04:09:47.993 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91093.log 2024-09-17T04:09:47.994 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93389.log.gz 2024-09-17T04:09:47.994 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109030.log 2024-09-17T04:09:47.995 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53614.log.gz 2024-09-17T04:09:47.995 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100596.log 2024-09-17T04:09:47.996 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.91093.log.gz 2024-09-17T04:09:47.996 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68477.log 2024-09-17T04:09:47.996 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109030.log.gz 2024-09-17T04:09:47.997 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54746.log 2024-09-17T04:09:47.997 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100596.log.gz 2024-09-17T04:09:47.998 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93816.log 2024-09-17T04:09:47.998 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.68477.log.gz 2024-09-17T04:09:47.999 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43253.log 2024-09-17T04:09:47.999 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54746.log.gz 2024-09-17T04:09:47.999 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93816.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93816.log.gz 2024-09-17T04:09:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65780.log 2024-09-17T04:09:48.000 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72116.log 2024-09-17T04:09:48.001 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43253.log.gz 2024-09-17T04:09:48.001 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91006.log 2024-09-17T04:09:48.001 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65780.log.gz 2024-09-17T04:09:48.002 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61213.log 2024-09-17T04:09:48.002 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72116.log.gz 2024-09-17T04:09:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91006.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91006.log.gz 2024-09-17T04:09:48.003 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108904.log 2024-09-17T04:09:48.004 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53301.log 2024-09-17T04:09:48.004 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61213.log.gz 2024-09-17T04:09:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93988.log 2024-09-17T04:09:48.005 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108904.log.gz 2024-09-17T04:09:48.005 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35818.log 2024-09-17T04:09:48.006 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53301.log.gz 2024-09-17T04:09:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93988.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93988.log.gz 2024-09-17T04:09:48.006 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68815.log 2024-09-17T04:09:48.007 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108021.log 2024-09-17T04:09:48.007 INFO:teuthology.orchestra.run.smithi129.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.35818.log.gz 2024-09-17T04:09:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.68815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.68815.log.gz 2024-09-17T04:09:48.008 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81929.log 2024-09-17T04:09:48.009 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89722.log 2024-09-17T04:09:48.009 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108021.log.gz 2024-09-17T04:09:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81929.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81929.log.gz 2024-09-17T04:09:48.010 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98360.log 2024-09-17T04:09:48.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59252.log 2024-09-17T04:09:48.011 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89722.log.gz 2024-09-17T04:09:48.012 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98360.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98360.log.gz 2024-09-17T04:09:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56334.log 2024-09-17T04:09:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67471.log 2024-09-17T04:09:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59252.log.gz 2024-09-17T04:09:48.013 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105452.log 2024-09-17T04:09:48.013 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56334.log.gz 2024-09-17T04:09:48.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63405.log 2024-09-17T04:09:48.014 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67471.log.gz 2024-09-17T04:09:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105452.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.105452.log.gz 2024-09-17T04:09:48.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94759.log 2024-09-17T04:09:48.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78163.log 2024-09-17T04:09:48.016 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63405.log.gz 2024-09-17T04:09:48.017 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101675.log 2024-09-17T04:09:48.017 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94759.log.gz 2024-09-17T04:09:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67242.log 2024-09-17T04:09:48.018 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78163.log.gz 2024-09-17T04:09:48.019 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88644.log 2024-09-17T04:09:48.019 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101675.log.gz 2024-09-17T04:09:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67242.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67242.log.gz 2024-09-17T04:09:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74417.log 2024-09-17T04:09:48.020 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74543.log 2024-09-17T04:09:48.020 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88644.log.gz 2024-09-17T04:09:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86012.log 2024-09-17T04:09:48.021 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74417.log.gz 2024-09-17T04:09:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100680.log 2024-09-17T04:09:48.023 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74543.log.gz 2024-09-17T04:09:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86012.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86012.log.gz 2024-09-17T04:09:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67555.log 2024-09-17T04:09:48.024 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43151.log 2024-09-17T04:09:48.024 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.100680.log.gz 2024-09-17T04:09:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90676.log 2024-09-17T04:09:48.025 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67555.log.gz 2024-09-17T04:09:48.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101885.log 2024-09-17T04:09:48.026 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43151.log.gz 2024-09-17T04:09:48.026 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90676.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90676.log.gz 2024-09-17T04:09:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102011.log 2024-09-17T04:09:48.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70503.log 2024-09-17T04:09:48.027 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101885.log.gz 2024-09-17T04:09:48.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42437.log 2024-09-17T04:09:48.028 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.102011.log.gz 2024-09-17T04:09:48.029 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69897.log 2024-09-17T04:09:48.029 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70503.log.gz 2024-09-17T04:09:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42437.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42437.log.gz 2024-09-17T04:09:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51854.log 2024-09-17T04:09:48.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102831.log 2024-09-17T04:09:48.031 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69897.log.gz 2024-09-17T04:09:48.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62146.log 2024-09-17T04:09:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51854.log.gz 2024-09-17T04:09:48.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71979.log 2024-09-17T04:09:48.032 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102831.log.gz 2024-09-17T04:09:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62146.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62146.log.gz 2024-09-17T04:09:48.033 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77658.log 2024-09-17T04:09:48.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95423.log 2024-09-17T04:09:48.034 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71979.log.gz 2024-09-17T04:09:48.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54518.log 2024-09-17T04:09:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.77658.log.gz 2024-09-17T04:09:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107622.log 2024-09-17T04:09:48.036 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95423.log.gz 2024-09-17T04:09:48.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54518.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54518.log.gz -5 2024-09-17T04:09:48.037 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57364.log 2024-09-17T04:09:48.037 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82950.log 2024-09-17T04:09:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.107622.log.gz 2024-09-17T04:09:48.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2024-09-17T04:09:48.038 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57364.log.gz 2024-09-17T04:09:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82950.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.82950.log.gz 2024-09-17T04:09:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97377.log 2024-09-17T04:09:48.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49329.log 2024-09-17T04:09:48.041 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2024-09-17T04:09:48.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82059.log 2024-09-17T04:09:48.041 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.97377.log.gz 2024-09-17T04:09:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100812.log 2024-09-17T04:09:48.042 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49329.log.gz 2024-09-17T04:09:48.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77467.log 2024-09-17T04:09:48.043 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82059.log.gz 2024-09-17T04:09:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100812.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.100812.log.gz 2024-09-17T04:09:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89890.log 2024-09-17T04:09:48.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73418.log 2024-09-17T04:09:48.045 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77467.log.gz 2024-09-17T04:09:48.045 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57112.log 2024-09-17T04:09:48.046 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.89890.log.gz 2024-09-17T04:09:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86152.log 2024-09-17T04:09:48.047 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73418.log.gz 2024-09-17T04:09:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57112.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57112.log.gz 2024-09-17T04:09:48.047 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80801.log 2024-09-17T04:09:48.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86152.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29679.log 2024-09-17T04:09:48.048 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86152.log.gz 2024-09-17T04:09:48.049 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69627.log 2024-09-17T04:09:48.049 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80801.log.gz 2024-09-17T04:09:48.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.tmp-client.admin.29679.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.29679.log.gz -5 2024-09-17T04:09:48.050 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46261.log 2024-09-17T04:09:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69627.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69627.log.gzgzip 2024-09-17T04:09:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.66136.log 2024-09-17T04:09:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48028.log 2024-09-17T04:09:48.051 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46261.log.gz 2024-09-17T04:09:48.052 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71468.log 2024-09-17T04:09:48.052 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66136.log.gz 2024-09-17T04:09:48.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54392.log 2024-09-17T04:09:48.054 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48028.log.gz 2024-09-17T04:09:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71468.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71468.log.gz 2024-09-17T04:09:48.054 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91337.log 2024-09-17T04:09:48.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98192.log 2024-09-17T04:09:48.055 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54392.log.gz 2024-09-17T04:09:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48467.log 2024-09-17T04:09:48.057 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91337.log.gz 2024-09-17T04:09:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98192.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.98192.log.gz 2024-09-17T04:09:48.057 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64994.log 2024-09-17T04:09:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64601.log 2024-09-17T04:09:48.058 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48467.log.gz 2024-09-17T04:09:48.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50890.log 2024-09-17T04:09:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64994.log.gz 2024-09-17T04:09:48.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45836.log 2024-09-17T04:09:48.059 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64601.log.gz 2024-09-17T04:09:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50890.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50890.log.gz 2024-09-17T04:09:48.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81604.log 2024-09-17T04:09:48.061 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54830.log 2024-09-17T04:09:48.061 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45836.log.gz 2024-09-17T04:09:48.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81604.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-17T04:09:48.062 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81604.log.gz 2024-09-17T04:09:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99576.log 2024-09-17T04:09:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.1.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54830.log.gz 2024-09-17T04:09:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98024.log 2024-09-17T04:09:48.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76254.log 2024-09-17T04:09:48.065 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.99576.log.gz 2024-09-17T04:09:48.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48730.log 2024-09-17T04:09:48.066 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98024.log.gz 2024-09-17T04:09:48.067 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76254.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76254.log.gz 2024-09-17T04:09:48.067 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81305.log 2024-09-17T04:09:48.067 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58012.log 2024-09-17T04:09:48.068 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48730.log.gz 2024-09-17T04:09:48.069 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81305.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81305.log.gz 2024-09-17T04:09:48.069 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63052.log 2024-09-17T04:09:48.070 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83153.log 2024-09-17T04:09:48.070 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58012.log.gz 2024-09-17T04:09:48.071 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63052.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63052.log.gz 2024-09-17T04:09:48.071 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69981.log 2024-09-17T04:09:48.072 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61665.log 2024-09-17T04:09:48.072 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83153.log.gz 2024-09-17T04:09:48.074 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69981.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.69981.log.gz 2024-09-17T04:09:48.074 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93066.log 2024-09-17T04:09:48.074 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62711.log 2024-09-17T04:09:48.074 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.61665.log.gz 2024-09-17T04:09:48.075 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93066.log: gzip -5 --verbose 87.6% -- -- replaced with /var/log/ceph/ceph-client.admin.93066.log.gz /var/log/ceph/ceph-client.admin.76721.log 2024-09-17T04:09:48.075 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.076 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73480.log 2024-09-17T04:09:48.076 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62711.log.gz 2024-09-17T04:09:48.077 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76721.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.76721.log.gz 2024-09-17T04:09:48.077 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84206.log 2024-09-17T04:09:48.078 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65993.log 2024-09-17T04:09:48.078 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73480.log.gz 2024-09-17T04:09:48.079 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84206.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84206.log.gz 2024-09-17T04:09:48.079 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53091.log 2024-09-17T04:09:48.080 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47850.log 2024-09-17T04:09:48.080 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65993.log.gz 2024-09-17T04:09:48.081 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53091.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53091.log.gz 2024-09-17T04:09:48.081 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106142.log 2024-09-17T04:09:48.082 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84122.log 2024-09-17T04:09:48.082 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47850.log.gz 2024-09-17T04:09:48.083 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106142.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106142.log.gz 2024-09-17T04:09:48.083 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44755.log 2024-09-17T04:09:48.084 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104006.log 2024-09-17T04:09:48.084 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84122.log.gz 2024-09-17T04:09:48.085 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44755.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44755.log.gz 2024-09-17T04:09:48.085 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101969.log 2024-09-17T04:09:48.086 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50451.log 2024-09-17T04:09:48.086 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104006.log.gz 2024-09-17T04:09:48.087 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101969.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101969.log.gz 2024-09-17T04:09:48.087 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105676.log 2024-09-17T04:09:48.088 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82706.log 2024-09-17T04:09:48.088 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50451.log.gz 2024-09-17T04:09:48.089 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105676.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105676.log.gz 2024-09-17T04:09:48.089 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60112.log 2024-09-17T04:09:48.090 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.82706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54476.log 2024-09-17T04:09:48.090 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82706.log.gz 2024-09-17T04:09:48.090 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.60112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60112.log.gz 2024-09-17T04:09:48.091 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67303.log 2024-09-17T04:09:48.092 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75914.log 2024-09-17T04:09:48.092 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54476.log.gz 2024-09-17T04:09:48.093 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72990.log 2024-09-17T04:09:48.093 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67303.log.gz 2024-09-17T04:09:48.094 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69855.log 2024-09-17T04:09:48.094 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75914.log.gz 2024-09-17T04:09:48.095 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72990.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72990.log.gz 2024-09-17T04:09:48.095 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71689.log 2024-09-17T04:09:48.096 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2024-09-17T04:09:48.096 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.69855.log.gz 2024-09-17T04:09:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71689.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71689.log.gz 2024-09-17T04:09:48.097 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97235.log 2024-09-17T04:09:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79181.log 2024-09-17T04:09:48.098 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2024-09-17T04:09:48.098 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.97235.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.97235.log.gz 2024-09-17T04:09:48.099 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91051.log 2024-09-17T04:09:48.100 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80675.log 2024-09-17T04:09:48.100 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79181.log.gz 2024-09-17T04:09:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91051.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91051.log.gz 2024-09-17T04:09:48.101 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55088.log 2024-09-17T04:09:48.105 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101321.log 2024-09-17T04:09:48.105 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80675.log.gz 2024-09-17T04:09:48.105 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.55088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.55088.log.gz 2024-09-17T04:09:48.106 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71197.log 2024-09-17T04:09:48.106 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2024-09-17T04:09:48.106 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101321.log.gz 2024-09-17T04:09:48.106 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71197.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71197.log.gz 2024-09-17T04:09:48.106 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96431.log 2024-09-17T04:09:48.107 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66198.log 2024-09-17T04:09:48.107 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2024-09-17T04:09:48.107 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96431.log: gzip -5 --verbose 87.4% -- -- replaced with /var/log/ceph/ceph-client.admin.96431.log.gz /var/log/ceph/ceph-client.admin.108820.log 2024-09-17T04:09:48.107 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.108 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44839.log 2024-09-17T04:09:48.108 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66198.log.gz 2024-09-17T04:09:48.109 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108820.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108820.log.gz 2024-09-17T04:09:48.109 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53488.log 2024-09-17T04:09:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93276.log 2024-09-17T04:09:48.111 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44839.log.gz 2024-09-17T04:09:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53488.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53488.log.gz 2024-09-17T04:09:48.111 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77509.log 2024-09-17T04:09:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90964.log 2024-09-17T04:09:48.113 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93276.log.gz 2024-09-17T04:09:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77509.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77509.log.gz 2024-09-17T04:09:48.113 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45755.log 2024-09-17T04:09:48.114 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58744.log 2024-09-17T04:09:48.114 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90964.log.gz 2024-09-17T04:09:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45755.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45755.log.gz 2024-09-17T04:09:48.115 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.100350.log 2024-09-17T04:09:48.116 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75391.log 2024-09-17T04:09:48.117 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58744.log.gz 2024-09-17T04:09:48.117 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.100350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.100350.log.gz 2024-09-17T04:09:48.118 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62104.log 2024-09-17T04:09:48.118 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65523.log 2024-09-17T04:09:48.119 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75391.log.gz 2024-09-17T04:09:48.119 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62104.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62104.log.gz 2024-09-17T04:09:48.119 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85167.log 2024-09-17T04:09:48.120 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62230.log 2024-09-17T04:09:48.121 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.65523.log.gz 2024-09-17T04:09:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85167.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85167.log.gz 2024-09-17T04:09:48.122 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50806.log 2024-09-17T04:09:48.123 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66948.log 2024-09-17T04:09:48.123 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62230.log.gz 2024-09-17T04:09:48.124 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50806.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50806.log.gz 2024-09-17T04:09:48.124 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87937.log 2024-09-17T04:09:48.124 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83215.log 2024-09-17T04:09:48.125 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66948.log.gz 2024-09-17T04:09:48.125 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87937.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87937.log.gz 2024-09-17T04:09:48.126 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54602.log 2024-09-17T04:09:48.126 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85125.log 2024-09-17T04:09:48.127 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83215.log.gz 2024-09-17T04:09:48.127 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54602.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54602.log.gz 2024-09-17T04:09:48.128 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46862.log 2024-09-17T04:09:48.128 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94851.log 2024-09-17T04:09:48.129 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85125.log.gz 2024-09-17T04:09:48.129 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46862.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46862.log.gz 2024-09-17T04:09:48.129 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96387.log 2024-09-17T04:09:48.130 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96090.log 2024-09-17T04:09:48.131 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.94851.log.gz 2024-09-17T04:09:48.131 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96387.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96387.log.gz 2024-09-17T04:09:48.132 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105177.log 2024-09-17T04:09:48.132 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92144.log 2024-09-17T04:09:48.133 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96090.log.gz 2024-09-17T04:09:48.134 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105177.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105177.log.gz 2024-09-17T04:09:48.134 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91479.log 2024-09-17T04:09:48.134 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95126.log 2024-09-17T04:09:48.135 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92144.log.gz 2024-09-17T04:09:48.136 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91479.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91479.log.gz 2024-09-17T04:09:48.136 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43970.log 2024-09-17T04:09:48.136 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75126.log 2024-09-17T04:09:48.137 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95126.log.gz 2024-09-17T04:09:48.137 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.43970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.43970.log.gz 2024-09-17T04:09:48.137 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46396.log 2024-09-17T04:09:48.138 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49287.log 2024-09-17T04:09:48.139 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46396.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75126.log.gz 2024-09-17T04:09:48.140 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46396.log.gz 2024-09-17T04:09:48.140 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49245.log 2024-09-17T04:09:48.141 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103782.log 2024-09-17T04:09:48.141 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49287.log.gz 2024-09-17T04:09:48.141 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57196.log 2024-09-17T04:09:48.142 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49245.log.gz 2024-09-17T04:09:48.142 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76865.log 2024-09-17T04:09:48.143 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103782.log.gz 2024-09-17T04:09:48.143 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94156.log 2024-09-17T04:09:48.144 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57196.log.gz 2024-09-17T04:09:48.145 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81845.log 2024-09-17T04:09:48.145 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.76865.log.gz 2024-09-17T04:09:48.145 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104438.log 2024-09-17T04:09:48.146 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94156.log.gz 2024-09-17T04:09:48.146 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81845.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81845.log.gz 2024-09-17T04:09:48.147 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63531.log 2024-09-17T04:09:48.147 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70359.log 2024-09-17T04:09:48.148 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104438.log.gz 2024-09-17T04:09:48.149 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63531.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63531.log.gz 2024-09-17T04:09:48.149 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102396.log 2024-09-17T04:09:48.149 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102533.log 2024-09-17T04:09:48.150 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.70359.log.gz 2024-09-17T04:09:48.151 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102396.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102396.log.gz 2024-09-17T04:09:48.151 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102575.log 2024-09-17T04:09:48.151 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36695.log 2024-09-17T04:09:48.152 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102533.log.gz 2024-09-17T04:09:48.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102575.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102575.log.gz 2024-09-17T04:09:48.153 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92954.log 2024-09-17T04:09:48.153 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83439.log 2024-09-17T04:09:48.154 INFO:teuthology.orchestra.run.smithi129.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.36695.log.gz 2024-09-17T04:09:48.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92954.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92954.log.gz 2024-09-17T04:09:48.155 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99830.log 2024-09-17T04:09:48.155 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74459.log 2024-09-17T04:09:48.157 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83439.log.gz 2024-09-17T04:09:48.157 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.99830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99830.log.gz 2024-09-17T04:09:48.157 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78966.log 2024-09-17T04:09:48.158 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87317.log 2024-09-17T04:09:48.158 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74459.log.gz 2024-09-17T04:09:48.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78966.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.78966.log.gz 2024-09-17T04:09:48.159 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50409.log 2024-09-17T04:09:48.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72023.log 2024-09-17T04:09:48.160 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.87317.log.gz 2024-09-17T04:09:48.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50409.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50409.log.gz 2024-09-17T04:09:48.161 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46970.log 2024-09-17T04:09:48.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101843.log 2024-09-17T04:09:48.162 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72023.log.gz 2024-09-17T04:09:48.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46970.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46970.log.gz 2024-09-17T04:09:48.163 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101573.log 2024-09-17T04:09:48.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42101.log 2024-09-17T04:09:48.164 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101843.log.gz 2024-09-17T04:09:48.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101573.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.101573.log.gz 2024-09-17T04:09:48.165 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98785.log 2024-09-17T04:09:48.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107032.log 2024-09-17T04:09:48.166 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42101.log.gz 2024-09-17T04:09:48.167 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98785.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98785.log.gz 2024-09-17T04:09:48.167 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108063.log 2024-09-17T04:09:48.168 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89806.log 2024-09-17T04:09:48.168 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107032.log.gz 2024-09-17T04:09:48.169 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108063.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108063.log.gz 2024-09-17T04:09:48.169 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86944.log 2024-09-17T04:09:48.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56799.log 2024-09-17T04:09:48.170 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89806.log.gz 2024-09-17T04:09:48.171 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86944.log: gzip 87.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.86944.log.gz -- /var/log/ceph/ceph-client.admin.55569.log 2024-09-17T04:09:48.171 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.172 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56677.log 2024-09-17T04:09:48.173 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56799.log.gz 2024-09-17T04:09:48.173 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109398.log 2024-09-17T04:09:48.174 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55569.log.gz 2024-09-17T04:09:48.175 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56677.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56677.log.gz 2024-09-17T04:09:48.175 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64255.log 2024-09-17T04:09:48.175 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99282.log 2024-09-17T04:09:48.175 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109398.log.gz 2024-09-17T04:09:48.176 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64255.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64255.log.gz 2024-09-17T04:09:48.176 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82395.log 2024-09-17T04:09:48.177 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45321.log 2024-09-17T04:09:48.177 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99282.log.gz 2024-09-17T04:09:48.178 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82395.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82395.log.gz 2024-09-17T04:09:48.178 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2024-09-17T04:09:48.179 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44629.log 2024-09-17T04:09:48.179 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45321.log.gz 2024-09-17T04:09:48.180 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69771.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2024-09-17T04:09:48.180 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.75746.log 2024-09-17T04:09:48.181 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74585.log 2024-09-17T04:09:48.181 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44629.log.gz 2024-09-17T04:09:48.182 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75746.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75746.log.gz 2024-09-17T04:09:48.182 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87193.log 2024-09-17T04:09:48.183 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72432.log 2024-09-17T04:09:48.183 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74585.log.gz 2024-09-17T04:09:48.184 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87193.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87193.log.gz 2024-09-17T04:09:48.184 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63675.log 2024-09-17T04:09:48.185 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78078.log 2024-09-17T04:09:48.185 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72432.log.gz 2024-09-17T04:09:48.186 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63675.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63675.log.gz 2024-09-17T04:09:48.186 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42395.log 2024-09-17T04:09:48.187 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105848.log 2024-09-17T04:09:48.187 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78078.log.gz 2024-09-17T04:09:48.188 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42395.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42395.log.gz 2024-09-17T04:09:48.188 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87627.log 2024-09-17T04:09:48.189 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89125.log 2024-09-17T04:09:48.189 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105848.log.gz 2024-09-17T04:09:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87627.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87627.log.gz 2024-09-17T04:09:48.190 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62291.log 2024-09-17T04:09:48.191 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48892.log 2024-09-17T04:09:48.191 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89125.log.gz 2024-09-17T04:09:48.192 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62291.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62291.log.gz 2024-09-17T04:09:48.192 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92576.log 2024-09-17T04:09:48.193 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103347.log 2024-09-17T04:09:48.193 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48892.log.gz 2024-09-17T04:09:48.194 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92576.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92576.log.gz 2024-09-17T04:09:48.194 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74375.log 2024-09-17T04:09:48.195 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56418.log 2024-09-17T04:09:48.196 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.103347.log.gz 2024-09-17T04:09:48.196 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74375.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74375.log.gz 2024-09-17T04:09:48.196 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96048.log 2024-09-17T04:09:48.197 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75788.log 2024-09-17T04:09:48.197 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56418.log.gz 2024-09-17T04:09:48.198 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96048.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96048.log.gz 2024-09-17T04:09:48.198 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98234.log 2024-09-17T04:09:48.199 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96729.log 2024-09-17T04:09:48.199 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.75788.log.gz 2024-09-17T04:09:48.200 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98234.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98234.log.gz 2024-09-17T04:09:48.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72903.log 2024-09-17T04:09:48.201 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95880.log 2024-09-17T04:09:48.201 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96729.log.gz 2024-09-17T04:09:48.202 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72903.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.72903.log.gz 2024-09-17T04:09:48.202 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60830.log 2024-09-17T04:09:48.203 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44103.log 2024-09-17T04:09:48.204 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95880.log.gz 2024-09-17T04:09:48.204 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60830.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60830.log.gz 2024-09-17T04:09:48.204 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70587.log 2024-09-17T04:09:48.205 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2024-09-17T04:09:48.205 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44103.log.gz 2024-09-17T04:09:48.206 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70587.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70587.log.gz 2024-09-17T04:09:48.206 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55489.log 2024-09-17T04:09:48.207 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66392.log 2024-09-17T04:09:48.208 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2024-09-17T04:09:48.208 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55489.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55489.log.gz 2024-09-17T04:09:48.208 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61603.log 2024-09-17T04:09:48.209 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70461.log 2024-09-17T04:09:48.210 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66392.log.gz 2024-09-17T04:09:48.210 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61603.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61603.log.gz 2024-09-17T04:09:48.210 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67938.log 2024-09-17T04:09:48.211 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86286.log 2024-09-17T04:09:48.212 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70461.log.gz 2024-09-17T04:09:48.212 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67938.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67938.log.gz 2024-09-17T04:09:48.212 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61399.log 2024-09-17T04:09:48.213 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57154.log 2024-09-17T04:09:48.214 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86286.log.gz 2024-09-17T04:09:48.214 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61399.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61399.log.gz 2024-09-17T04:09:48.214 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93679.log 2024-09-17T04:09:48.215 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69669.log 2024-09-17T04:09:48.216 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57154.log.gz 2024-09-17T04:09:48.216 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93679.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93679.log.gz 2024-09-17T04:09:48.216 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-09-17T04:09:48.217 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59049.log 2024-09-17T04:09:48.218 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69669.log.gz 2024-09-17T04:09:48.218 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41583.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-09-17T04:09:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58389.log 2024-09-17T04:09:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56887.log 2024-09-17T04:09:48.220 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59049.log.gz 2024-09-17T04:09:48.220 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58389.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58389.log.gz 2024-09-17T04:09:48.221 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78668.log 2024-09-17T04:09:48.221 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106226.log 2024-09-17T04:09:48.222 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56887.log.gz 2024-09-17T04:09:48.222 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62459.log 2024-09-17T04:09:48.223 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78668.log.gz 2024-09-17T04:09:48.223 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106226.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106226.log.gz 2024-09-17T04:09:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96519.log 2024-09-17T04:09:48.224 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84698.log 2024-09-17T04:09:48.225 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62459.log.gz 2024-09-17T04:09:48.225 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96519.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96519.log.gz 2024-09-17T04:09:48.226 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61151.log 2024-09-17T04:09:48.226 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106058.log 2024-09-17T04:09:48.227 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84698.log.gz 2024-09-17T04:09:48.227 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61151.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61151.log.gz 2024-09-17T04:09:48.228 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77302.log 2024-09-17T04:09:48.228 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69939.log 2024-09-17T04:09:48.229 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.106058.log.gz 2024-09-17T04:09:48.230 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77302.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77302.log.gz 2024-09-17T04:09:48.230 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70023.log 2024-09-17T04:09:48.231 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51247.log 2024-09-17T04:09:48.231 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69939.log.gz 2024-09-17T04:09:48.232 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70023.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70023.log.gz 2024-09-17T04:09:48.232 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88602.log 2024-09-17T04:09:48.232 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42311.log 2024-09-17T04:09:48.233 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51247.log.gz 2024-09-17T04:09:48.234 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88602.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88602.log.gz 2024-09-17T04:09:48.234 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52938.log 2024-09-17T04:09:48.235 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97720.log 2024-09-17T04:09:48.235 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42311.log.gz 2024-09-17T04:09:48.236 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52938.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52938.log.gz 2024-09-17T04:09:48.236 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-09-17T04:09:48.237 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89764.log 2024-09-17T04:09:48.237 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.97720.log.gz 2024-09-17T04:09:48.238 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65343.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gz 2024-09-17T04:09:48.238 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70545.log 2024-09-17T04:09:48.239 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89510.log 2024-09-17T04:09:48.239 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89764.log.gz 2024-09-17T04:09:48.240 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70545.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70545.log.gzgzip 2024-09-17T04:09:48.240 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59295.log 2024-09-17T04:09:48.241 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105092.log 2024-09-17T04:09:48.241 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89510.log.gz 2024-09-17T04:09:48.242 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59295.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59295.log.gz 2024-09-17T04:09:48.242 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50283.log 2024-09-17T04:09:48.243 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64840.log 2024-09-17T04:09:48.243 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.105092.log.gz 2024-09-17T04:09:48.244 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50283.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50283.log.gz 2024-09-17T04:09:48.244 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76763.log 2024-09-17T04:09:48.245 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49683.log 2024-09-17T04:09:48.245 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64840.log.gz 2024-09-17T04:09:48.246 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76763.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76763.log.gz 2024-09-17T04:09:48.246 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100638.log 2024-09-17T04:09:48.247 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104344.log 2024-09-17T04:09:48.247 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49683.log.gz 2024-09-17T04:09:48.248 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100638.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.100638.log.gz 2024-09-17T04:09:48.248 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68137.log 2024-09-17T04:09:48.249 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98871.log 2024-09-17T04:09:48.249 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104344.log.gz 2024-09-17T04:09:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68137.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68137.log.gz 2024-09-17T04:09:48.250 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60396.log 2024-09-17T04:09:48.251 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71384.log 2024-09-17T04:09:48.251 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98871.log.gz 2024-09-17T04:09:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60396.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60396.log.gz 2024-09-17T04:09:48.252 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69197.log 2024-09-17T04:09:48.253 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102747.log 2024-09-17T04:09:48.253 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71384.log.gz 2024-09-17T04:09:48.254 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69197.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69197.log.gz 2024-09-17T04:09:48.254 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55717.log 2024-09-17T04:09:48.255 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66578.log 2024-09-17T04:09:48.255 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102747.log.gz 2024-09-17T04:09:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55717.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55717.log.gz 2024-09-17T04:09:48.256 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85483.log 2024-09-17T04:09:48.257 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71155.log 2024-09-17T04:09:48.257 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66578.log.gz 2024-09-17T04:09:48.258 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85483.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85483.log.gz 2024-09-17T04:09:48.258 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64298.log 2024-09-17T04:09:48.259 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57850.log 2024-09-17T04:09:48.260 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71155.log.gz 2024-09-17T04:09:48.260 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48238.log 2024-09-17T04:09:48.261 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.64298.log.gz 2024-09-17T04:09:48.261 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.57850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109354.log 2024-09-17T04:09:48.261 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.57850.log.gz 2024-09-17T04:09:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48238.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48238.log.gz 2024-09-17T04:09:48.262 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81522.log 2024-09-17T04:09:48.263 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100493.log 2024-09-17T04:09:48.264 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.109354.log.gz 2024-09-17T04:09:48.264 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81522.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81522.log.gz 2024-09-17T04:09:48.264 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78541.log 2024-09-17T04:09:48.266 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61337.log 2024-09-17T04:09:48.266 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100493.log.gz 2024-09-17T04:09:48.266 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78541.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78541.log.gz 2024-09-17T04:09:48.267 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81646.log 2024-09-17T04:09:48.267 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90076.log 2024-09-17T04:09:48.268 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.61337.log.gz 2024-09-17T04:09:48.268 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81646.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81646.log.gz 2024-09-17T04:09:48.269 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86013.log 2024-09-17T04:09:48.270 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68948.log 2024-09-17T04:09:48.270 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90076.log.gz 2024-09-17T04:09:48.270 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86013.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.86013.log.gz 2024-09-17T04:09:48.271 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72852.log 2024-09-17T04:09:48.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76511.log 2024-09-17T04:09:48.272 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68948.log.gz 2024-09-17T04:09:48.273 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72852.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.72852.log.gz 2024-09-17T04:09:48.273 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64585.log 2024-09-17T04:09:48.274 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99054.log 2024-09-17T04:09:48.274 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.76511.log.gz 2024-09-17T04:09:48.275 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64585.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64585.log.gz 2024-09-17T04:09:48.275 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53955.log 2024-09-17T04:09:48.276 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95838.log 2024-09-17T04:09:48.276 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99054.log.gz 2024-09-17T04:09:48.277 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53955.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53955.log.gz 2024-09-17T04:09:48.277 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56502.log 2024-09-17T04:09:48.278 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2024-09-17T04:09:48.278 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95838.log.gz 2024-09-17T04:09:48.279 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56502.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56502.log.gz 2024-09-17T04:09:48.279 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53740.log 2024-09-17T04:09:48.280 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50367.log 2024-09-17T04:09:48.280 INFO:teuthology.orchestra.run.smithi129.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2024-09-17T04:09:48.281 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53740.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53740.log.gz 2024-09-17T04:09:48.281 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2024-09-17T04:09:48.282 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74939.log 2024-09-17T04:09:48.282 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50367.log.gz 2024-09-17T04:09:48.283 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45237.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2024-09-17T04:09:48.283 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42924.log 2024-09-17T04:09:48.284 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99534.log 2024-09-17T04:09:48.284 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74939.log.gz 2024-09-17T04:09:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42924.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42924.log.gz 2024-09-17T04:09:48.285 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107895.log 2024-09-17T04:09:48.286 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46736.log 2024-09-17T04:09:48.286 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99534.log.gz 2024-09-17T04:09:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107895.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107895.log.gz 2024-09-17T04:09:48.287 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88434.log 2024-09-17T04:09:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82311.log 2024-09-17T04:09:48.288 INFO:teuthology.orchestra.run.smithi129.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.46736.log.gz 2024-09-17T04:09:48.289 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88434.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88434.log.gz 2024-09-17T04:09:48.289 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.105581.log 2024-09-17T04:09:48.290 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84351.log 2024-09-17T04:09:48.290 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.82311.log.gz 2024-09-17T04:09:48.291 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105581.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105581.log.gz 2024-09-17T04:09:48.291 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108694.log 2024-09-17T04:09:48.292 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79097.log 2024-09-17T04:09:48.292 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84351.log.gz 2024-09-17T04:09:48.293 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108694.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108694.log.gz 2024-09-17T04:09:48.293 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81442.log 2024-09-17T04:09:48.294 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47594.log 2024-09-17T04:09:48.294 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79097.log.gz 2024-09-17T04:09:48.295 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81442.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81442.log.gz 2024-09-17T04:09:48.295 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62967.log 2024-09-17T04:09:48.296 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69077.log 2024-09-17T04:09:48.296 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47594.log.gz 2024-09-17T04:09:48.297 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62967.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62967.log.gz 2024-09-17T04:09:48.297 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84038.log 2024-09-17T04:09:48.298 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107853.log 2024-09-17T04:09:48.298 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69077.log.gz 2024-09-17T04:09:48.299 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84038.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84038.log.gz 2024-09-17T04:09:48.299 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108988.log 2024-09-17T04:09:48.300 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79703.log 2024-09-17T04:09:48.300 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.107853.log.gz 2024-09-17T04:09:48.301 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108988.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108988.log.gz 2024-09-17T04:09:48.301 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64402.log 2024-09-17T04:09:48.302 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60961.log 2024-09-17T04:09:48.302 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79703.log.gz 2024-09-17T04:09:48.303 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64402.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64402.log.gz 2024-09-17T04:09:48.303 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42353.log 2024-09-17T04:09:48.304 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93858.log 2024-09-17T04:09:48.304 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60961.log.gz 2024-09-17T04:09:48.305 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42353.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42353.log.gz 2024-09-17T04:09:48.305 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35946.log 2024-09-17T04:09:48.306 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71426.log 2024-09-17T04:09:48.306 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.93858.log.gz 2024-09-17T04:09:48.307 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35946.log: 74.8% -- replaced with /var/log/ceph/ceph-client.admin.35946.log.gz 2024-09-17T04:09:48.307 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35985.log 2024-09-17T04:09:48.308 INFO:teuthology.orchestra.run.smithi181.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-17T04:09:48.308 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84435.log 2024-09-17T04:09:48.309 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71426.log.gz 2024-09-17T04:09:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35985.log: 77.4% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-09-17T04:09:48.309 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107220.log 2024-09-17T04:09:48.310 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96345.log 2024-09-17T04:09:48.311 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84435.log.gz 2024-09-17T04:09:48.311 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.107220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107220.log.gz 2024-09-17T04:09:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108484.log 2024-09-17T04:09:48.312 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88915.log 2024-09-17T04:09:48.313 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96345.log.gz 2024-09-17T04:09:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108484.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108484.log.gz 2024-09-17T04:09:48.313 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98276.log 2024-09-17T04:09:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79325.log 2024-09-17T04:09:48.314 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.88915.log.gz 2024-09-17T04:09:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98276.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98276.log.gz 2024-09-17T04:09:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70648.log 2024-09-17T04:09:48.316 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68068.log 2024-09-17T04:09:48.317 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79325.log.gz 2024-09-17T04:09:48.317 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70648.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70648.log.gz 2024-09-17T04:09:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77714.log 2024-09-17T04:09:48.318 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53698.log 2024-09-17T04:09:48.319 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68068.log.gz 2024-09-17T04:09:48.320 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77714.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77714.log.gz 2024-09-17T04:09:48.320 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2024-09-17T04:09:48.320 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53698.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-17T04:09:48.321 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53698.log.gz 2024-09-17T04:09:48.322 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42615.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2024-09-17T04:09:48.322 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80340.log 2024-09-17T04:09:48.323 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93108.log 2024-09-17T04:09:48.324 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71510.log 2024-09-17T04:09:48.325 INFO:teuthology.orchestra.run.smithi129.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.80340.log.gz 2024-09-17T04:09:48.325 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43295.log 2024-09-17T04:09:48.325 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93108.log.gz 2024-09-17T04:09:48.326 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92481.log 2024-09-17T04:09:48.326 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71510.log.gz 2024-09-17T04:09:48.327 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43295.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43295.log.gz 2024-09-17T04:09:48.327 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87068.log 2024-09-17T04:09:48.327 INFO:teuthology.orchestra.run.smithi129.stderr: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-17T04:09:48.328 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92481.log: gzip -5 --verbose 88.0% -- -- replaced with /var/log/ceph/ceph-client.admin.92481.log.gz 2024-09-17T04:09:48.328 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.107979.log 2024-09-17T04:09:48.329 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75475.log 2024-09-17T04:09:48.329 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87068.log.gz 2024-09-17T04:09:48.330 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107979.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107979.log.gz 2024-09-17T04:09:48.330 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63717.log 2024-09-17T04:09:48.331 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85783.log 2024-09-17T04:09:48.332 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75475.log.gz 2024-09-17T04:09:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63717.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.63717.log.gz 2024-09-17T04:09:48.332 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62188.log 2024-09-17T04:09:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.85783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59434.log 2024-09-17T04:09:48.333 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.85783.log.gz 2024-09-17T04:09:48.334 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62188.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62188.log.gz 2024-09-17T04:09:48.334 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69259.log 2024-09-17T04:09:48.335 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101024.log 2024-09-17T04:09:48.335 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59434.log.gz 2024-09-17T04:09:48.336 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69259.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69259.log.gz 2024-09-17T04:09:48.336 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79745.log 2024-09-17T04:09:48.337 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78584.log 2024-09-17T04:09:48.338 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79745.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.101024.log.gz 2024-09-17T04:09:48.338 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79053.log 2024-09-17T04:09:48.339 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.79745.log.gz 2024-09-17T04:09:48.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92228.log 2024-09-17T04:09:48.340 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78584.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78584.log.gz 2024-09-17T04:09:48.340 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90118.log 2024-09-17T04:09:48.341 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79053.log.gz 2024-09-17T04:09:48.341 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92228.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92228.log.gz 2024-09-17T04:09:48.341 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96771.log 2024-09-17T04:09:48.342 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66990.log 2024-09-17T04:09:48.342 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.90118.log.gz 2024-09-17T04:09:48.343 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96771.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96771.log.gz 2024-09-17T04:09:48.343 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90538.log 2024-09-17T04:09:48.344 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75559.log 2024-09-17T04:09:48.344 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66990.log.gz 2024-09-17T04:09:48.345 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70275.log 2024-09-17T04:09:48.345 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90538.log.gz 2024-09-17T04:09:48.346 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75872.log 2024-09-17T04:09:48.347 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75559.log.gz 2024-09-17T04:09:48.347 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70275.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70275.log.gz 2024-09-17T04:09:48.347 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-09-17T04:09:48.348 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70191.log 2024-09-17T04:09:48.349 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75872.log.gz 2024-09-17T04:09:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52583.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-09-17T04:09:48.349 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64882.log 2024-09-17T04:09:48.350 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86328.log 2024-09-17T04:09:48.351 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70191.log.gz 2024-09-17T04:09:48.351 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64882.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.64882.log.gz 2024-09-17T04:09:48.352 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75704.log 2024-09-17T04:09:48.352 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99450.log 2024-09-17T04:09:48.353 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.86328.log.gz 2024-09-17T04:09:48.353 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75704.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75704.log.gz 2024-09-17T04:09:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69383.log 2024-09-17T04:09:48.354 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84080.log 2024-09-17T04:09:48.355 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.99450.log.gz 2024-09-17T04:09:48.355 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69383.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69383.log.gz 2024-09-17T04:09:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58557.log 2024-09-17T04:09:48.356 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95381.log 2024-09-17T04:09:48.357 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84080.log.gz 2024-09-17T04:09:48.358 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58557.log: gzip -5 87.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.58557.log.gz 2024-09-17T04:09:48.358 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.92912.log 2024-09-17T04:09:48.358 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2024-09-17T04:09:48.359 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95381.log.gz 2024-09-17T04:09:48.359 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92912.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92912.log.gz 2024-09-17T04:09:48.360 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92312.log 2024-09-17T04:09:48.360 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108208.log 2024-09-17T04:09:48.361 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2024-09-17T04:09:48.362 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92312.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92312.log.gz 2024-09-17T04:09:48.362 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106268.log 2024-09-17T04:09:48.362 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81011.log 2024-09-17T04:09:48.363 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.108208.log.gz 2024-09-17T04:09:48.364 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58515.log 2024-09-17T04:09:48.364 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.106268.log.gz 2024-09-17T04:09:48.364 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104924.log 2024-09-17T04:09:48.365 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81011.log.gz 2024-09-17T04:09:48.365 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58515.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58515.log.gz 2024-09-17T04:09:48.366 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109225.log 2024-09-17T04:09:48.367 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79619.log 2024-09-17T04:09:48.367 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104924.log.gz 2024-09-17T04:09:48.368 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109225.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.109225.log.gz 2024-09-17T04:09:48.368 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99642.log 2024-09-17T04:09:48.369 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87006.log 2024-09-17T04:09:48.369 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79619.log.gz 2024-09-17T04:09:48.370 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99642.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.99642.log.gz 2024-09-17T04:09:48.370 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.83029.log 2024-09-17T04:09:48.371 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100982.log 2024-09-17T04:09:48.371 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87006.log.gz 2024-09-17T04:09:48.372 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83029.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83029.log.gz 2024-09-17T04:09:48.372 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72074.log 2024-09-17T04:09:48.373 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84778.log 2024-09-17T04:09:48.373 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100982.log.gz 2024-09-17T04:09:48.374 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72074.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72074.log.gz 2024-09-17T04:09:48.374 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87875.log 2024-09-17T04:09:48.376 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89381.log 2024-09-17T04:09:48.376 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84778.log.gz 2024-09-17T04:09:48.376 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60250.log 2024-09-17T04:09:48.376 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.87875.log.gz 2024-09-17T04:09:48.377 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87565.log 2024-09-17T04:09:48.377 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89381.log.gz 2024-09-17T04:09:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60250.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60250.log.gz 2024-09-17T04:09:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97317.log 2024-09-17T04:09:48.379 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107717.log 2024-09-17T04:09:48.379 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87565.log.gz 2024-09-17T04:09:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97317.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97317.log.gz 2024-09-17T04:09:48.380 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81347.log 2024-09-17T04:09:48.381 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96939.log 2024-09-17T04:09:48.381 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107717.log.gz 2024-09-17T04:09:48.382 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81347.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81347.log.gz 2024-09-17T04:09:48.382 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66739.log 2024-09-17T04:09:48.383 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94758.log 2024-09-17T04:09:48.383 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96939.log.gz 2024-09-17T04:09:48.384 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66739.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66739.log.gz 2024-09-17T04:09:48.384 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102267.log 2024-09-17T04:09:48.385 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52771.log 2024-09-17T04:09:48.385 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.94758.log.gz 2024-09-17T04:09:48.386 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102267.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.102267.log.gz 2024-09-17T04:09:48.386 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101633.log 2024-09-17T04:09:48.387 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.52771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86864.log 2024-09-17T04:09:48.387 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52771.log.gz 2024-09-17T04:09:48.388 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101633.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.0.77123.log -- replaced with /var/log/ceph/ceph-client.admin.101633.log.gz 2024-09-17T04:09:48.388 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.389 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52775.log 2024-09-17T04:09:48.389 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.86864.log.gz 2024-09-17T04:09:48.390 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.77123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.77123.log.gz 2024-09-17T04:09:48.390 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51472.log 2024-09-17T04:09:48.391 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.52775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60730.log 2024-09-17T04:09:48.391 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52775.log.gz 2024-09-17T04:09:48.392 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51472.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51472.log.gz 2024-09-17T04:09:48.392 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99834.log 2024-09-17T04:09:48.393 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72330.log 2024-09-17T04:09:48.394 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.99834.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60730.log.gz 2024-09-17T04:09:48.394 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99834.log.gz 2024-09-17T04:09:48.394 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70945.log 2024-09-17T04:09:48.395 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65566.log 2024-09-17T04:09:48.395 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72330.log.gz 2024-09-17T04:09:48.396 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103389.log 2024-09-17T04:09:48.396 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70945.log.gz 2024-09-17T04:09:48.397 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65566.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65566.log.gz 2024-09-17T04:09:48.397 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76679.log 2024-09-17T04:09:48.398 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87441.log 2024-09-17T04:09:48.398 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103389.log.gz 2024-09-17T04:09:48.399 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76679.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.76679.log.gz 2024-09-17T04:09:48.399 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.100346.log 2024-09-17T04:09:48.400 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53446.log 2024-09-17T04:09:48.400 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87441.log.gz 2024-09-17T04:09:48.401 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.100346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.100346.log.gz 2024-09-17T04:09:48.401 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42227.log 2024-09-17T04:09:48.402 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105718.log 2024-09-17T04:09:48.402 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53446.log.gz 2024-09-17T04:09:48.403 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42227.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42227.log.gz 2024-09-17T04:09:48.403 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92744.log 2024-09-17T04:09:48.404 INFO:teuthology.orchestra.run.smithi181.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-09-17T04:09:48.404 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100535.log 2024-09-17T04:09:48.404 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105718.log.gz 2024-09-17T04:09:48.405 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92744.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92744.log.gz 2024-09-17T04:09:48.405 INFO:teuthology.orchestra.run.smithi181.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-09-17T04:09:48.406 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42606.log 2024-09-17T04:09:48.406 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94620.log 2024-09-17T04:09:48.406 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.100535.log.gz 2024-09-17T04:09:48.407 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42606.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42606.log.gz 2024-09-17T04:09:48.407 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56105.log 2024-09-17T04:09:48.408 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.94620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94620.log.gz 2024-09-17T04:09:48.409 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53835.log 2024-09-17T04:09:48.409 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35920.log 2024-09-17T04:09:48.409 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56105.log.gz 2024-09-17T04:09:48.411 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81688.log 2024-09-17T04:09:48.411 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53835.log.gz 2024-09-17T04:09:48.413 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35920.log: 75.8% -- replaced with /var/log/ceph/ceph-client.admin.35920.log.gz 2024-09-17T04:09:48.413 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96561.log 2024-09-17T04:09:48.413 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95549.log 2024-09-17T04:09:48.413 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81688.log.gz 2024-09-17T04:09:48.415 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96561.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96561.log.gz 2024-09-17T04:09:48.415 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61023.log 2024-09-17T04:09:48.415 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36114.log 2024-09-17T04:09:48.416 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95549.log.gz 2024-09-17T04:09:48.418 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61023.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61023.log.gz 2024-09-17T04:09:48.418 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48551.log 2024-09-17T04:09:48.418 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108568.log 2024-09-17T04:09:48.418 INFO:teuthology.orchestra.run.smithi129.stderr: 75.7% -- replaced with /var/log/ceph/ceph-client.admin.36114.log.gz 2024-09-17T04:09:48.418 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48551.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48551.log.gz 2024-09-17T04:09:48.419 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84603.log 2024-09-17T04:09:48.420 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90202.log 2024-09-17T04:09:48.420 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108568.log.gz 2024-09-17T04:09:48.420 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84603.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.84603.log.gz 2024-09-17T04:09:48.420 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53530.log 2024-09-17T04:09:48.420 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102309.log 2024-09-17T04:09:48.421 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53530.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.90202.log.gz 2024-09-17T04:09:48.422 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53530.log.gz 2024-09-17T04:09:48.422 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70903.log 2024-09-17T04:09:48.422 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85525.log 2024-09-17T04:09:48.423 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.102309.log.gz 2024-09-17T04:09:48.423 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56250.log 2024-09-17T04:09:48.424 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70903.log.gz 2024-09-17T04:09:48.425 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87751.log 2024-09-17T04:09:48.425 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.85525.log.gz 2024-09-17T04:09:48.425 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93192.log 2024-09-17T04:09:48.426 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56250.log.gz 2024-09-17T04:09:48.426 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87751.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87751.log.gz 2024-09-17T04:09:48.427 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72204.log 2024-09-17T04:09:48.427 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79009.log 2024-09-17T04:09:48.428 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.93192.log.gz 2024-09-17T04:09:48.429 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72204.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72204.log.gzgzip 2024-09-17T04:09:48.429 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55166.log 2024-09-17T04:09:48.430 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71071.log 2024-09-17T04:09:48.430 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79009.log.gz 2024-09-17T04:09:48.431 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55166.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55166.log.gz 2024-09-17T04:09:48.431 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-09-17T04:09:48.432 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75662.log 2024-09-17T04:09:48.432 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71071.log.gz 2024-09-17T04:09:48.432 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36299.log: 75.8% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-09-17T04:09:48.433 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48070.log 2024-09-17T04:09:48.434 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105050.log 2024-09-17T04:09:48.434 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75662.log.gz 2024-09-17T04:09:48.435 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103737.log 2024-09-17T04:09:48.435 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48070.log.gz 2024-09-17T04:09:48.436 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83818.log 2024-09-17T04:09:48.436 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105050.log.gz 2024-09-17T04:09:48.437 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103737.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103737.log.gz 2024-09-17T04:09:48.437 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92372.log 2024-09-17T04:09:48.439 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45489.log 2024-09-17T04:09:48.439 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83818.log.gz 2024-09-17T04:09:48.439 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92372.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92372.log.gz 2024-09-17T04:09:48.439 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95964.log 2024-09-17T04:09:48.442 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78415.log 2024-09-17T04:09:48.442 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45489.log.gz 2024-09-17T04:09:48.442 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95964.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95964.log.gz 2024-09-17T04:09:48.442 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75023.log 2024-09-17T04:09:48.443 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50577.log 2024-09-17T04:09:48.443 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78415.log.gz 2024-09-17T04:09:48.446 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98581.log 2024-09-17T04:09:48.446 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75023.log.gz 2024-09-17T04:09:48.446 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50577.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50577.log.gz 2024-09-17T04:09:48.447 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93723.log 2024-09-17T04:09:48.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104812.log 2024-09-17T04:09:48.447 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98581.log.gz 2024-09-17T04:09:48.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93723.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.93723.log.gz 2024-09-17T04:09:48.447 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91652.log 2024-09-17T04:09:48.453 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62627.log 2024-09-17T04:09:48.453 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.104812.log.gz 2024-09-17T04:09:48.453 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91652.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91652.log.gz 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104648.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70233.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62627.log.gz 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104648.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.104648.log.gz 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50974.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67597.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.70233.log.gz 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50974.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50974.log.gz 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36026.log 2024-09-17T04:09:48.454 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67597.log.gz 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46312.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85591.log 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73622.log 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr: 75.9% -- replaced with /var/log/ceph/ceph-client.admin.36026.log.gz 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85591.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.85591.log.gz 2024-09-17T04:09:48.456 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66241.log 2024-09-17T04:09:48.457 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58074.log 2024-09-17T04:09:48.457 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73622.log.gz 2024-09-17T04:09:48.458 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66241.log: gzip -5 89.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.66241.log.gz 2024-09-17T04:09:48.458 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.63615.log 2024-09-17T04:09:48.459 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97856.log 2024-09-17T04:09:48.459 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58074.log.gz 2024-09-17T04:09:48.460 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63615.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63615.log.gz 2024-09-17T04:09:48.460 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54788.log 2024-09-17T04:09:48.460 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95754.log 2024-09-17T04:09:48.461 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97856.log.gz 2024-09-17T04:09:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54788.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54788.log.gz 2024-09-17T04:09:48.462 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79265.log 2024-09-17T04:09:48.463 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78457.log 2024-09-17T04:09:48.463 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.95754.log.gz 2024-09-17T04:09:48.464 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79265.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79265.log.gz 2024-09-17T04:09:48.464 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76595.log 2024-09-17T04:09:48.464 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47755.log 2024-09-17T04:09:48.465 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78457.log.gz 2024-09-17T04:09:48.466 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76595.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76595.log.gz 2024-09-17T04:09:48.466 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106100.log 2024-09-17T04:09:48.466 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66454.log 2024-09-17T04:09:48.467 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47755.log.gz 2024-09-17T04:09:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106100.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106100.log.gz 2024-09-17T04:09:48.468 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104966.log 2024-09-17T04:09:48.469 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51686.log 2024-09-17T04:09:48.469 INFO:teuthology.orchestra.run.smithi129.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.66454.log.gz 2024-09-17T04:09:48.470 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104966.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104966.log.gz 2024-09-17T04:09:48.470 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59564.log 2024-09-17T04:09:48.471 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57406.log 2024-09-17T04:09:48.471 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51686.log.gz 2024-09-17T04:09:48.472 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59564.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59564.log.gz 2024-09-17T04:09:48.472 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108292.log 2024-09-17T04:09:48.473 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68897.log 2024-09-17T04:09:48.473 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57406.log.gz 2024-09-17T04:09:48.474 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108292.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108292.log.gz 2024-09-17T04:09:48.474 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48112.log 2024-09-17T04:09:48.475 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49157.log 2024-09-17T04:09:48.475 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68897.log.gz 2024-09-17T04:09:48.476 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42059.log 2024-09-17T04:09:48.476 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48112.log.gz 2024-09-17T04:09:48.477 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49157.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49157.log.gz 2024-09-17T04:09:48.477 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88476.log 2024-09-17T04:09:48.478 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100940.log 2024-09-17T04:09:48.478 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42059.log.gz 2024-09-17T04:09:48.479 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88476.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88476.log.gz 2024-09-17T04:09:48.479 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.87813.log 2024-09-17T04:09:48.480 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69321.log 2024-09-17T04:09:48.480 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100940.log.gz 2024-09-17T04:09:48.481 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87813.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.87813.log.gz 2024-09-17T04:09:48.481 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71342.log 2024-09-17T04:09:48.482 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100427.log 2024-09-17T04:09:48.482 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69321.log.gz 2024-09-17T04:09:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71342.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71342.log.gz 2024-09-17T04:09:48.483 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89848.log 2024-09-17T04:09:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99492.log 2024-09-17T04:09:48.484 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89848.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.100427.log.gz 2024-09-17T04:09:48.485 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89848.log.gz 2024-09-17T04:09:48.485 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49115.log 2024-09-17T04:09:48.486 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54304.log 2024-09-17T04:09:48.486 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.99492.log.gz 2024-09-17T04:09:48.487 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47257.log 2024-09-17T04:09:48.487 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49115.log.gz 2024-09-17T04:09:48.488 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54304.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54304.log.gz 2024-09-17T04:09:48.488 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102663.log 2024-09-17T04:09:48.489 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96185.log 2024-09-17T04:09:48.489 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47257.log.gz 2024-09-17T04:09:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102663.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102663.log.gz 2024-09-17T04:09:48.490 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60398.log 2024-09-17T04:09:48.491 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103302.log 2024-09-17T04:09:48.491 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96185.log.gz 2024-09-17T04:09:48.492 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60398.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60398.log.gz 2024-09-17T04:09:48.492 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80717.log 2024-09-17T04:09:48.493 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103622.log 2024-09-17T04:09:48.493 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103302.log.gz 2024-09-17T04:09:48.494 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80717.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80717.log.gz 2024-09-17T04:09:48.494 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71851.log 2024-09-17T04:09:48.495 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46904.log 2024-09-17T04:09:48.495 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103622.log.gz 2024-09-17T04:09:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71851.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71851.log.gz 2024-09-17T04:09:48.496 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77635.log 2024-09-17T04:09:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58431.log 2024-09-17T04:09:48.497 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77635.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46904.log.gz 2024-09-17T04:09:48.498 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77635.log.gz 2024-09-17T04:09:48.498 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78843.log 2024-09-17T04:09:48.499 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97982.log 2024-09-17T04:09:48.499 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58431.log.gz 2024-09-17T04:09:48.500 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48383.log 2024-09-17T04:09:48.500 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78843.log.gz 2024-09-17T04:09:48.501 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97982.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97982.log.gz 2024-09-17T04:09:48.501 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106970.log 2024-09-17T04:09:48.502 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90160.log 2024-09-17T04:09:48.502 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48383.log.gz 2024-09-17T04:09:48.503 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106970.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106970.log.gz 2024-09-17T04:09:48.503 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109312.log 2024-09-17T04:09:48.504 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106892.log 2024-09-17T04:09:48.504 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90160.log.gz 2024-09-17T04:09:48.505 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109312.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109312.log.gz 2024-09-17T04:09:48.505 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47514.log 2024-09-17T04:09:48.506 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94945.log 2024-09-17T04:09:48.506 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.106892.log.gz 2024-09-17T04:09:48.507 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47514.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47514.log.gz 2024-09-17T04:09:48.507 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47986.log 2024-09-17T04:09:48.508 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36084.log 2024-09-17T04:09:48.508 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94945.log.gz 2024-09-17T04:09:48.509 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47986.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47986.log.gz 2024-09-17T04:09:48.509 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76051.log 2024-09-17T04:09:48.510 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56843.log 2024-09-17T04:09:48.511 INFO:teuthology.orchestra.run.smithi129.stderr: 75.7% -- replaced with /var/log/ceph/ceph-client.admin.36084.log.gz 2024-09-17T04:09:48.513 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76051.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76051.log.gz 2024-09-17T04:09:48.513 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106184.log 2024-09-17T04:09:48.513 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56843.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-09-17T04:09:48.513 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56843.log.gz 2024-09-17T04:09:48.513 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106184.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.106184.log.gz 2024-09-17T04:09:48.514 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92415.log 2024-09-17T04:09:48.514 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68351.log 2024-09-17T04:09:48.515 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103251.log 2024-09-17T04:09:48.516 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92415.log.gz 2024-09-17T04:09:48.516 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68351.log: gzip 87.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.68351.log.gz --verbose 2024-09-17T04:09:48.517 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.51644.log 2024-09-17T04:09:48.517 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56166.log 2024-09-17T04:09:48.518 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.103251.log.gz 2024-09-17T04:09:48.519 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51644.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51644.log.gz 2024-09-17T04:09:48.519 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82269.log 2024-09-17T04:09:48.520 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56166.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56166.log.gz 2024-09-17T04:09:48.520 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50848.log 2024-09-17T04:09:48.521 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108946.log 2024-09-17T04:09:48.521 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.82269.log.gz 2024-09-17T04:09:48.522 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48299.log 2024-09-17T04:09:48.522 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50848.log.gz 2024-09-17T04:09:48.523 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108946.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.108946.log.gz 2024-09-17T04:09:48.523 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64747.log 2024-09-17T04:09:48.524 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103695.log 2024-09-17T04:09:48.525 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48299.log.gz 2024-09-17T04:09:48.525 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64747.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64747.log.gz 2024-09-17T04:09:48.525 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80969.log 2024-09-17T04:09:48.526 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60397.log 2024-09-17T04:09:48.526 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103695.log.gz 2024-09-17T04:09:48.528 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101447.log 2024-09-17T04:09:48.528 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80969.log.gz 2024-09-17T04:09:48.529 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60397.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.60397.log.gz 2024-09-17T04:09:48.529 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104606.log 2024-09-17T04:09:48.530 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96603.log 2024-09-17T04:09:48.530 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.101447.log.gz 2024-09-17T04:09:48.531 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104606.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104606.log.gz 2024-09-17T04:09:48.531 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66621.log 2024-09-17T04:09:48.532 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96603.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96603.log.gz 2024-09-17T04:09:48.532 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98966.log 2024-09-17T04:09:48.533 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49979.log 2024-09-17T04:09:48.534 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98966.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98966.log.gz 2024-09-17T04:09:48.534 INFO:teuthology.orchestra.run.smithi129.stderr:gzip 92.6% -- replaced with /var/log/ceph/ceph-client.admin.66621.log.gz -5 2024-09-17T04:09:48.534 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.80581.log 2024-09-17T04:09:48.535 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.49979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49979.log.gz 2024-09-17T04:09:48.535 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48196.log 2024-09-17T04:09:48.536 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80581.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80581.log.gz 2024-09-17T04:09:48.537 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62543.log 2024-09-17T04:09:48.538 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94324.log 2024-09-17T04:09:48.538 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48196.log.gz 2024-09-17T04:09:48.539 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62543.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62543.log.gz 2024-09-17T04:09:48.539 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76385.log 2024-09-17T04:09:48.540 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53049.log 2024-09-17T04:09:48.540 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94324.log.gz 2024-09-17T04:09:48.541 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76385.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76385.log.gz 2024-09-17T04:09:48.541 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102981.log 2024-09-17T04:09:48.542 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71935.log 2024-09-17T04:09:48.543 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53049.log.gz 2024-09-17T04:09:48.544 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102981.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102981.log.gz 2024-09-17T04:09:48.544 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53997.log 2024-09-17T04:09:48.545 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71935.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.71935.log.gz 2024-09-17T04:09:48.545 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60637.log 2024-09-17T04:09:48.546 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67387.log 2024-09-17T04:09:48.546 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53997.log.gz 2024-09-17T04:09:48.547 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60637.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60637.log.gz 2024-09-17T04:09:48.547 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81975.log 2024-09-17T04:09:48.548 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67387.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67387.log.gz 2024-09-17T04:09:48.548 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93024.log 2024-09-17T04:09:48.549 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44671.log 2024-09-17T04:09:48.549 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81975.log.gz 2024-09-17T04:09:48.550 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93024.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93024.log.gz -5 2024-09-17T04:09:48.550 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.99198.log 2024-09-17T04:09:48.551 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44671.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44671.log.gz 2024-09-17T04:09:48.551 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65842.log 2024-09-17T04:09:48.552 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79535.log 2024-09-17T04:09:48.553 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.99198.log.gz 2024-09-17T04:09:48.553 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65842.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65842.log.gz 2024-09-17T04:09:48.553 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56208.log 2024-09-17T04:09:48.555 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 87.8% 2024-09-17T04:09:48.555 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79535.log.gz 2024-09-17T04:09:48.556 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56208.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56208.log.gz 2024-09-17T04:09:48.556 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58702.log 2024-09-17T04:09:48.557 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50638.log 2024-09-17T04:09:48.557 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2024-09-17T04:09:48.558 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58702.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58702.log.gz 2024-09-17T04:09:48.558 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94240.log 2024-09-17T04:09:48.559 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65718.log 2024-09-17T04:09:48.559 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50638.log.gz 2024-09-17T04:09:48.561 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106522.log 2024-09-17T04:09:48.561 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.94240.log.gz 2024-09-17T04:09:48.561 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102186.log 2024-09-17T04:09:48.562 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65718.log.gz 2024-09-17T04:09:48.562 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.106522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.106522.log.gz 2024-09-17T04:09:48.563 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79999.log 2024-09-17T04:09:48.563 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102186.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102186.log.gz 2024-09-17T04:09:48.564 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102873.log 2024-09-17T04:09:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.79999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.79999.log.gz 2024-09-17T04:09:48.565 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-09-17T04:09:48.566 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88999.log 2024-09-17T04:09:48.566 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102873.log.gz 2024-09-17T04:09:48.567 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41975.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-09-17T04:09:48.567 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93904.log 2024-09-17T04:09:48.568 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47944.log 2024-09-17T04:09:48.568 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88999.log.gz 2024-09-17T04:09:48.569 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93904.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93904.log.gz 2024-09-17T04:09:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82101.log 2024-09-17T04:09:48.570 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57658.log 2024-09-17T04:09:48.571 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47944.log.gz 2024-09-17T04:09:48.571 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82101.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.82101.log.gz -5 2024-09-17T04:09:48.572 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54644.log 2024-09-17T04:09:48.573 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57658.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57658.log.gz 2024-09-17T04:09:48.573 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105974.log 2024-09-17T04:09:48.574 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109438.log 2024-09-17T04:09:48.574 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54644.log.gz 2024-09-17T04:09:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105974.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105974.log.gz 2024-09-17T04:09:48.575 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103431.log 2024-09-17T04:09:48.576 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83912.log 2024-09-17T04:09:48.576 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109438.log.gz 2024-09-17T04:09:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103431.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.103431.log.gz 2024-09-17T04:09:48.577 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70809.log 2024-09-17T04:09:48.578 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81053.log 2024-09-17T04:09:48.580 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.83912.log.gz 2024-09-17T04:09:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70809.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70809.log.gz 2024-09-17T04:09:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62020.log 2024-09-17T04:09:48.582 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104522.log 2024-09-17T04:09:48.582 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81053.log.gz 2024-09-17T04:09:48.582 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62020.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62020.log.gz 2024-09-17T04:09:48.582 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55811.log 2024-09-17T04:09:48.583 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80275.log 2024-09-17T04:09:48.584 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104522.log.gz 2024-09-17T04:09:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58347.log 2024-09-17T04:09:48.586 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55811.log.gz 2024-09-17T04:09:48.586 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59816.log 2024-09-17T04:09:48.586 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80275.log.gz 2024-09-17T04:09:48.587 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58347.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58347.log.gz 2024-09-17T04:09:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94198.log 2024-09-17T04:09:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98108.log 2024-09-17T04:09:48.588 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59816.log.gz 2024-09-17T04:09:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94198.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.94198.log.gz 2024-09-17T04:09:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45363.log 2024-09-17T04:09:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93469.log 2024-09-17T04:09:48.590 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98108.log.gz 2024-09-17T04:09:48.590 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44797.log 2024-09-17T04:09:48.591 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45363.log.gz 2024-09-17T04:09:48.592 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104480.log 2024-09-17T04:09:48.592 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.93469.log.gz 2024-09-17T04:09:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44797.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44797.log.gz 2024-09-17T04:09:48.593 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84944.log 2024-09-17T04:09:48.594 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96813.log 2024-09-17T04:09:48.594 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104480.log.gz 2024-09-17T04:09:48.595 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51812.log 2024-09-17T04:09:48.595 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.84944.log.gz 2024-09-17T04:09:48.596 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68627.log 2024-09-17T04:09:48.597 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.96813.log.gz 2024-09-17T04:09:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51812.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51812.log.gz 2024-09-17T04:09:48.597 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73834.log 2024-09-17T04:09:48.598 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78120.log 2024-09-17T04:09:48.599 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68627.log.gz 2024-09-17T04:09:48.599 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.103173.log 2024-09-17T04:09:48.600 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73834.log.gz 2024-09-17T04:09:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78120.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78120.log.gz 2024-09-17T04:09:48.601 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64583.log 2024-09-17T04:09:48.602 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.103173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75307.log 2024-09-17T04:09:48.602 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.103173.log.gz 2024-09-17T04:09:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64583.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.64583.log.gz 2024-09-17T04:09:48.603 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84164.log 2024-09-17T04:09:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77994.log 2024-09-17T04:09:48.604 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.75307.log.gz 2024-09-17T04:09:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84164.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84164.log.gz 2024-09-17T04:09:48.605 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50764.log 2024-09-17T04:09:48.606 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.85779.log 2024-09-17T04:09:48.607 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77994.log.gz 2024-09-17T04:09:48.607 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50764.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.50764.log.gz -5 2024-09-17T04:09:48.607 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42899.log 2024-09-17T04:09:48.608 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.85779.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.0.85779.log.gz /var/log/ceph/ceph-client.admin.52274.log 2024-09-17T04:09:48.609 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.610 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93150.log 2024-09-17T04:09:48.610 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42899.log.gz 2024-09-17T04:09:48.611 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52274.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52274.log.gz 2024-09-17T04:09:48.611 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73032.log 2024-09-17T04:09:48.612 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2024-09-17T04:09:48.612 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93150.log.gz 2024-09-17T04:09:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73032.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73032.log.gz 2024-09-17T04:09:48.613 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57024.log 2024-09-17T04:09:48.614 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78205.log 2024-09-17T04:09:48.614 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2024-09-17T04:09:48.615 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57024.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57024.log.gz 2024-09-17T04:09:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47699.log 2024-09-17T04:09:48.616 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57322.log 2024-09-17T04:09:48.617 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78205.log.gz 2024-09-17T04:09:48.618 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36991.log 2024-09-17T04:09:48.618 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47699.log.gz 2024-09-17T04:09:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57322.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57322.log.gz 2024-09-17T04:09:48.619 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93946.log 2024-09-17T04:09:48.620 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84561.log 2024-09-17T04:09:48.620 INFO:teuthology.orchestra.run.smithi129.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36991.log.gz 2024-09-17T04:09:48.621 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93946.log: 87.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.93946.log.gz 2024-09-17T04:09:48.621 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79661.log 2024-09-17T04:09:48.622 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72288.log 2024-09-17T04:09:48.622 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84561.log.gz 2024-09-17T04:09:48.623 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79661.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79661.log.gz 2024-09-17T04:09:48.623 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51514.log 2024-09-17T04:09:48.624 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105890.log 2024-09-17T04:09:48.625 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72288.log.gz 2024-09-17T04:09:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51514.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51514.log.gz 2024-09-17T04:09:48.625 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102915.log 2024-09-17T04:09:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105890.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.105890.log.gz 2024-09-17T04:09:48.627 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-09-17T04:09:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77900.log 2024-09-17T04:09:48.628 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41299.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.102915.log.gz 2024-09-17T04:09:48.629 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-09-17T04:09:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77593.log 2024-09-17T04:09:48.630 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77900.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77900.log.gzgzip 2024-09-17T04:09:48.631 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49725.log 2024-09-17T04:09:48.631 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77593.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77593.log.gz 2024-09-17T04:09:48.631 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65885.log 2024-09-17T04:09:48.632 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88141.log 2024-09-17T04:09:48.633 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65885.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.49725.log.gz 2024-09-17T04:09:48.634 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42614.log 2024-09-17T04:09:48.634 INFO:teuthology.orchestra.run.smithi129.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65885.log.gz 2024-09-17T04:09:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88141.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88141.log.gz 2024-09-17T04:09:48.635 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81263.log 2024-09-17T04:09:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.82710.log 2024-09-17T04:09:48.636 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81263.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42614.log.gz 2024-09-17T04:09:48.637 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.81263.log.gz 2024-09-17T04:09:48.637 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93774.log 2024-09-17T04:09:48.638 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.82710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.82710.log.gz 2024-09-17T04:09:48.638 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45657.log 2024-09-17T04:09:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93774.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93774.log.gz 2024-09-17T04:09:48.640 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104192.log 2024-09-17T04:09:48.641 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57490.log 2024-09-17T04:09:48.641 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45657.log.gz 2024-09-17T04:09:48.643 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104192.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104192.log.gz 2024-09-17T04:09:48.643 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98661.log 2024-09-17T04:09:48.643 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68993.log 2024-09-17T04:09:48.645 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57490.log.gz 2024-09-17T04:09:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98661.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98661.log.gz 2024-09-17T04:09:48.645 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-17T04:09:48.647 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94282.log 2024-09-17T04:09:48.647 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68993.log.gz 2024-09-17T04:09:48.647 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46089.log 2024-09-17T04:09:48.649 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74291.log 2024-09-17T04:09:48.649 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.94282.log.gz 2024-09-17T04:09:48.650 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109115.log 2024-09-17T04:09:48.650 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46089.log.gz 2024-09-17T04:09:48.652 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55084.log 2024-09-17T04:09:48.652 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.74291.log.gz 2024-09-17T04:09:48.652 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94761.log 2024-09-17T04:09:48.652 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.109115.log.gz 2024-09-17T04:09:48.655 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.55084.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57532.log 2024-09-17T04:09:48.655 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.0.55084.log.gz 2024-09-17T04:09:48.655 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46354.log 2024-09-17T04:09:48.655 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.94761.log.gz 2024-09-17T04:09:48.655 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54350.log 2024-09-17T04:09:48.656 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57532.log.gz 2024-09-17T04:09:48.657 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79409.log 2024-09-17T04:09:48.657 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46354.log.gz 2024-09-17T04:09:48.657 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49371.log 2024-09-17T04:09:48.657 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54350.log.gz 2024-09-17T04:09:48.658 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45963.log 2024-09-17T04:09:48.658 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79409.log.gz 2024-09-17T04:09:48.659 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73074.log 2024-09-17T04:09:48.659 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49371.log.gz 2024-09-17T04:09:48.661 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36224.log 2024-09-17T04:09:48.661 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45963.log.gz 2024-09-17T04:09:48.662 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107811.log 2024-09-17T04:09:48.662 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73074.log.gz 2024-09-17T04:09:48.663 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56292.log 2024-09-17T04:09:48.663 INFO:teuthology.orchestra.run.smithi129.stderr: 76.1% -- replaced with /var/log/ceph/ceph-client.admin.36224.log.gz 2024-09-17T04:09:48.664 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80233.log 2024-09-17T04:09:48.664 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107811.log.gz 2024-09-17T04:09:48.665 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91541.log 2024-09-17T04:09:48.665 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56292.log.gz 2024-09-17T04:09:48.666 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50722.log 2024-09-17T04:09:48.667 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80233.log.gz 2024-09-17T04:09:48.668 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57448.log 2024-09-17T04:09:48.668 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91541.log.gz 2024-09-17T04:09:48.669 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48509.log 2024-09-17T04:09:48.669 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50722.log.gz 2024-09-17T04:09:48.670 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109072.log 2024-09-17T04:09:48.670 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57448.log.gz 2024-09-17T04:09:48.671 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49983.log 2024-09-17T04:09:48.671 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48509.log.gz 2024-09-17T04:09:48.673 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84519.log 2024-09-17T04:09:48.673 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.109072.log.gz 2024-09-17T04:09:48.674 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.49983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.49983.log.gz 2024-09-17T04:09:48.674 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107517.log 2024-09-17T04:09:48.675 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56982.log 2024-09-17T04:09:48.675 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84519.log.gz 2024-09-17T04:09:48.676 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107398.log 2024-09-17T04:09:48.676 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107517.log.gz 2024-09-17T04:09:48.677 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42955.log 2024-09-17T04:09:48.677 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56982.log.gz 2024-09-17T04:09:48.679 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46442.log 2024-09-17T04:09:48.679 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42955.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.107398.log.gz 2024-09-17T04:09:48.680 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78763.log 2024-09-17T04:09:48.680 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42955.log.gz 2024-09-17T04:09:48.681 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78289.log 2024-09-17T04:09:48.681 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46442.log.gz 2024-09-17T04:09:48.682 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90589.log 2024-09-17T04:09:48.682 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78763.log.gz 2024-09-17T04:09:48.683 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49544.log 2024-09-17T04:09:48.683 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78289.log.gz 2024-09-17T04:09:48.685 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102621.log 2024-09-17T04:09:48.685 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90589.log.gz 2024-09-17T04:09:48.686 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79811.log 2024-09-17T04:09:48.686 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49544.log.gz 2024-09-17T04:09:48.687 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81221.log 2024-09-17T04:09:48.687 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102621.log.gz 2024-09-17T04:09:48.688 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42889.log 2024-09-17T04:09:48.688 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79811.log.gz 2024-09-17T04:09:48.689 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98486.log 2024-09-17T04:09:48.690 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81221.log.gz 2024-09-17T04:09:48.691 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46610.log 2024-09-17T04:09:48.691 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42889.log.gz 2024-09-17T04:09:48.692 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69119.log 2024-09-17T04:09:48.692 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98486.log.gz 2024-09-17T04:09:48.693 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46610.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46610.log.gz -5 2024-09-17T04:09:48.693 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.100746.log 2024-09-17T04:09:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74981.log 2024-09-17T04:09:48.694 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69119.log.gz 2024-09-17T04:09:48.695 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104564.log 2024-09-17T04:09:48.696 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.100746.log.gz 2024-09-17T04:09:48.697 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79493.log 2024-09-17T04:09:48.697 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.74981.log.gz 2024-09-17T04:09:48.698 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99408.log 2024-09-17T04:09:48.698 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104564.log.gz 2024-09-17T04:09:48.699 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96645.log 2024-09-17T04:09:48.699 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79493.log.gz 2024-09-17T04:09:48.700 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50061.log 2024-09-17T04:09:48.701 INFO:teuthology.orchestra.run.smithi129.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.99408.log.gz 2024-09-17T04:09:48.701 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97625.log 2024-09-17T04:09:48.702 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.96645.log.gz 2024-09-17T04:09:48.703 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95080.log 2024-09-17T04:09:48.703 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50061.log.gz 2024-09-17T04:09:48.704 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49497.log 2024-09-17T04:09:48.704 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97625.log.gz 2024-09-17T04:09:48.705 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91399.log 2024-09-17T04:09:48.705 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95080.log.gz 2024-09-17T04:09:48.707 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83996.log 2024-09-17T04:09:48.707 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49497.log.gz 2024-09-17T04:09:48.708 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67513.log 2024-09-17T04:09:48.708 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91399.log.gz 2024-09-17T04:09:48.709 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54896.log 2024-09-17T04:09:48.709 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83996.log.gz 2024-09-17T04:09:48.710 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92270.log 2024-09-17T04:09:48.710 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67513.log.gz 2024-09-17T04:09:48.711 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55937.log 2024-09-17T04:09:48.711 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54896.log.gz 2024-09-17T04:09:48.712 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58263.log 2024-09-17T04:09:48.713 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.92270.log.gz 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55937.log: gzip -5 --verbose 87.3% -- -- replaced with /var/log/ceph/ceph-client.admin.55937.log.gz /var/log/ceph/ceph-client.admin.57580.log 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86224.log 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58263.log.gz 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43601.log 2024-09-17T04:09:48.717 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57580.log.gz 2024-09-17T04:09:48.718 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89423.log 2024-09-17T04:09:48.718 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86224.log.gz 2024-09-17T04:09:48.718 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97047.log 2024-09-17T04:09:48.719 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43601.log.gz 2024-09-17T04:09:48.720 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46694.log 2024-09-17T04:09:48.720 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.89423.log.gz 2024-09-17T04:09:48.721 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105932.log 2024-09-17T04:09:48.721 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.97047.log.gz 2024-09-17T04:09:48.722 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76212.log 2024-09-17T04:09:48.722 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46694.log.gz 2024-09-17T04:09:48.723 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.90456.log 2024-09-17T04:09:48.723 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105932.log.gz 2024-09-17T04:09:48.724 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44411.log 2024-09-17T04:09:48.724 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76212.log.gz 2024-09-17T04:09:48.725 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.90456.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2024-09-17T04:09:48.726 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.0.90456.log.gz 2024-09-17T04:09:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.2.44411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.44411.log.gz 2024-09-17T04:09:48.727 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98150.log 2024-09-17T04:09:48.728 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48341.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-17T04:09:48.728 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2024-09-17T04:09:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98150.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98150.log.gz 2024-09-17T04:09:48.729 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92102.log 2024-09-17T04:09:48.731 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67116.log 2024-09-17T04:09:48.732 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36277.log 2024-09-17T04:09:48.732 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92102.log.gz 2024-09-17T04:09:48.733 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59690.log 2024-09-17T04:09:48.733 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67116.log.gz 2024-09-17T04:09:48.734 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36277.log: 75.8% -- replaced with /var/log/ceph/ceph-client.admin.36277.log.gz 2024-09-17T04:09:48.734 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91159.log 2024-09-17T04:09:48.736 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53343.log 2024-09-17T04:09:48.736 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59690.log.gz 2024-09-17T04:09:48.737 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101363.log 2024-09-17T04:09:48.738 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.91159.log.gz 2024-09-17T04:09:48.738 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53343.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53343.log.gz 2024-09-17T04:09:48.739 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71594.log 2024-09-17T04:09:48.739 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101363.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101363.log.gz 2024-09-17T04:09:48.740 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46526.log 2024-09-17T04:09:48.741 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70690.log 2024-09-17T04:09:48.741 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.71594.log.gz 2024-09-17T04:09:48.743 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74249.log 2024-09-17T04:09:48.743 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46526.log.gz 2024-09-17T04:09:48.744 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36655.log 2024-09-17T04:09:48.744 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1%/var/log/ceph/ceph-client.admin.74249.log: -- replaced with /var/log/ceph/ceph-client.admin.70690.log.gz 2024-09-17T04:09:48.746 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.65404.log 2024-09-17T04:09:48.746 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.74249.log.gz 2024-09-17T04:09:48.746 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36655.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.36655.log.gz 2024-09-17T04:09:48.747 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57280.log 2024-09-17T04:09:48.748 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101801.log 2024-09-17T04:09:48.749 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57280.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.65404.log.gz 2024-09-17T04:09:48.749 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57280.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98318.log 2024-09-17T04:09:48.749 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.750 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101801.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101801.log.gz 2024-09-17T04:09:48.751 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36250.log 2024-09-17T04:09:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98318.log: gzip -5 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98318.log.gz 2024-09-17T04:09:48.752 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59518.log 2024-09-17T04:09:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36250.log: 74.4% -- replaced with /var/log/ceph/ceph-client.admin.36250.log.gz 2024-09-17T04:09:48.753 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49413.log 2024-09-17T04:09:48.755 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-09-17T04:09:48.755 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59518.log.gz 2024-09-17T04:09:48.756 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89680.log 2024-09-17T04:09:48.756 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49413.log.gz 2024-09-17T04:09:48.757 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50932.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2024-09-17T04:09:48.757 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85423.log 2024-09-17T04:09:48.758 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89680.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.89680.log.gz 2024-09-17T04:09:48.760 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85213.log 2024-09-17T04:09:48.760 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.85423.log.gz 2024-09-17T04:09:48.761 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77325.log 2024-09-17T04:09:48.762 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45573.log 2024-09-17T04:09:48.762 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85213.log.gz 2024-09-17T04:09:48.764 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77325.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77325.log.gz 2024-09-17T04:09:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.45573.log: --verbose -- /var/log/ceph/ceph-client.admin.103890.log 2024-09-17T04:09:48.765 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45573.log.gz 2024-09-17T04:09:48.765 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55979.log 2024-09-17T04:09:48.767 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67980.log 2024-09-17T04:09:48.767 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.103890.log.gz 2024-09-17T04:09:48.768 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55979.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55979.log.gz 2024-09-17T04:09:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82227.log 2024-09-17T04:09:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67980.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.67980.log.gz 2024-09-17T04:09:48.770 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36200.log 2024-09-17T04:09:48.771 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47656.log 2024-09-17T04:09:48.771 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82227.log.gz 2024-09-17T04:09:48.772 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36200.log: 78.0% -- replaced with /var/log/ceph/ceph-client.admin.36200.log.gz 2024-09-17T04:09:48.773 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88518.log 2024-09-17T04:09:48.774 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.47656.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61275.log 2024-09-17T04:09:48.774 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47656.log.gz 2024-09-17T04:09:48.775 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48154.log 2024-09-17T04:09:48.775 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.88518.log.gz 2024-09-17T04:09:48.776 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74837.log 2024-09-17T04:09:48.776 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61275.log.gz 2024-09-17T04:09:48.777 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104130.log 2024-09-17T04:09:48.777 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48154.log.gz 2024-09-17T04:09:48.779 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74837.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.74837.log.gz 2024-09-17T04:09:48.780 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.104130.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70149.log 2024-09-17T04:09:48.780 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.104130.log.gz 2024-09-17T04:09:48.780 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108652.log 2024-09-17T04:09:48.782 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60116.log 2024-09-17T04:09:48.782 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70149.log.gz 2024-09-17T04:09:48.783 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108652.log: gzip -5 87.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.108652.log.gz -- 2024-09-17T04:09:48.784 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.51962.log 2024-09-17T04:09:48.784 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.60116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.60116.log.gz 2024-09-17T04:09:48.785 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85297.log 2024-09-17T04:09:48.786 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88705.log 2024-09-17T04:09:48.786 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51962.log.gz 2024-09-17T04:09:48.787 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73339.log 2024-09-17T04:09:48.787 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85297.log.gz 2024-09-17T04:09:48.788 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88705.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88705.log.gz 2024-09-17T04:09:48.788 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105764.log 2024-09-17T04:09:48.790 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73339.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73339.log.gz 2024-09-17T04:09:48.790 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78373.log 2024-09-17T04:09:48.791 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98743.log 2024-09-17T04:09:48.792 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105764.log.gz 2024-09-17T04:09:48.792 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78373.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78373.log.gz 2024-09-17T04:09:48.793 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.103169.log 2024-09-17T04:09:48.794 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35733.log 2024-09-17T04:09:48.794 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98743.log.gz 2024-09-17T04:09:48.795 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.103169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.103169.log.gz 2024-09-17T04:09:48.795 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45531.log 2024-09-17T04:09:48.796 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101717.log 2024-09-17T04:09:48.796 INFO:teuthology.orchestra.run.smithi129.stderr: 77.7% -- replaced with /var/log/ceph/ceph-client.admin.35733.log.gz 2024-09-17T04:09:48.798 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97814.log 2024-09-17T04:09:48.798 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.45531.log.gz 2024-09-17T04:09:48.799 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95038.log 2024-09-17T04:09:48.799 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101717.log.gz 2024-09-17T04:09:48.800 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92018.log 2024-09-17T04:09:48.800 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.97814.log.gz 2024-09-17T04:09:48.801 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75620.log 2024-09-17T04:09:48.801 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95038.log.gz 2024-09-17T04:09:48.803 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50680.log 2024-09-17T04:09:48.803 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92018.log.gz 2024-09-17T04:09:48.803 INFO:teuthology.orchestra.run.smithi181.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-17T04:09:48.804 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75620.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75620.log.gz 2024-09-17T04:09:48.804 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92660.log 2024-09-17T04:09:48.805 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51602.log 2024-09-17T04:09:48.806 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50680.log.gz 2024-09-17T04:09:48.807 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92660.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-17T04:09:48.807 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92660.log.gz 2024-09-17T04:09:48.808 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51602.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51602.log.gz 2024-09-17T04:09:48.808 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89554.log 2024-09-17T04:09:48.809 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106016.log 2024-09-17T04:09:48.811 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89554.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.89554.log.gz 2024-09-17T04:09:48.811 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.106016.log: --verbose -- /var/log/ceph/ceph-client.admin.53572.log 2024-09-17T04:09:48.812 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.106016.log.gz 2024-09-17T04:09:48.813 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107579.log 2024-09-17T04:09:48.814 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53572.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53572.log.gz 2024-09-17T04:09:48.815 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91275.log 2024-09-17T04:09:48.816 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107579.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.107579.log.gz 2024-09-17T04:09:48.817 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92702.log 2024-09-17T04:09:48.818 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91275.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91275.log.gz 2024-09-17T04:09:48.818 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98915.log 2024-09-17T04:09:48.819 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92702.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92702.log.gz 2024-09-17T04:09:48.820 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65460.log 2024-09-17T04:09:48.821 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98915.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98915.log.gz 2024-09-17T04:09:48.822 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85917.log 2024-09-17T04:09:48.823 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65460.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65460.log.gz 2024-09-17T04:09:48.823 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58954.log 2024-09-17T04:09:48.824 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85917.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.85917.log.gz 2024-09-17T04:09:48.825 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.90460.log 2024-09-17T04:09:48.826 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58954.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58954.log.gz 2024-09-17T04:09:48.826 INFO:teuthology.orchestra.run.smithi129.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-17T04:09:48.827 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.90460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.90460.log.gz 2024-09-17T04:09:48.828 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104068.log 2024-09-17T04:09:48.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106334.log 2024-09-17T04:09:48.830 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95465.log 2024-09-17T04:09:48.830 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.104068.log.gz 2024-09-17T04:09:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106334.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64059.log 2024-09-17T04:09:48.832 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106334.log.gz 2024-09-17T04:09:48.833 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95465.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95465.log.gz 2024-09-17T04:09:48.834 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58912.log 2024-09-17T04:09:48.834 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.64059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64059.log.gz 2024-09-17T04:09:48.835 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80433.log 2024-09-17T04:09:48.836 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44992.log 2024-09-17T04:09:48.836 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58912.log.gz 2024-09-17T04:09:48.837 INFO:teuthology.orchestra.run.smithi181.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-17T04:09:48.837 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53656.log 2024-09-17T04:09:48.838 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80433.log.gz 2024-09-17T04:09:48.840 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44992.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44992.log.gz 2024-09-17T04:09:48.841 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.53656.log: --verbose -- /var/log/ceph/ceph-client.admin.73542.log 2024-09-17T04:09:48.841 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53656.log.gz 2024-09-17T04:09:48.841 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62585.log 2024-09-17T04:09:48.842 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36897.log 2024-09-17T04:09:48.842 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.73542.log.gz 2024-09-17T04:09:48.844 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62585.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62585.log.gz 2024-09-17T04:09:48.844 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.98827.log 2024-09-17T04:09:48.846 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36897.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67429.log 2024-09-17T04:09:48.846 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36897.log.gz 2024-09-17T04:09:48.846 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101927.log 2024-09-17T04:09:48.846 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.98827.log.gz 2024-09-17T04:09:48.847 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67429.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67429.log.gz 2024-09-17T04:09:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94366.log 2024-09-17T04:09:48.848 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101927.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101927.log.gz 2024-09-17T04:09:48.849 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76469.log 2024-09-17T04:09:48.851 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67734.log 2024-09-17T04:09:48.851 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.94366.log.gz 2024-09-17T04:09:48.851 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76469.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76469.log.gz 2024-09-17T04:09:48.852 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84309.log 2024-09-17T04:09:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51556.log 2024-09-17T04:09:48.857 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67734.log.gz 2024-09-17T04:09:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42619.log 2024-09-17T04:09:48.857 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.84309.log.gz 2024-09-17T04:09:48.857 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51556.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51556.log.gz 2024-09-17T04:09:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76553.log 2024-09-17T04:09:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42269.log 2024-09-17T04:09:48.858 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42619.log.gz 2024-09-17T04:09:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76553.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76553.log.gz 2024-09-17T04:09:48.858 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51333.log 2024-09-17T04:09:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42269.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42269.log.gz 2024-09-17T04:09:48.859 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78331.log 2024-09-17T04:09:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66058.log 2024-09-17T04:09:48.861 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51333.log.gz 2024-09-17T04:09:48.862 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78331.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-17T04:09:48.862 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78331.log.gz 2024-09-17T04:09:48.863 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.80003.log 2024-09-17T04:09:48.863 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66058.log.gz 2024-09-17T04:09:48.865 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.0.80003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97541.log 2024-09-17T04:09:48.866 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.80003.log.gz 2024-09-17T04:09:48.867 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84393.log 2024-09-17T04:09:48.868 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97541.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.97541.log.gz 2024-09-17T04:09:48.869 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84393.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.84393.log.gz 2024-09-17T04:09:48.869 INFO:teuthology.orchestra.run.smithi129.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-17T04:09:48.869 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87379.log 2024-09-17T04:09:48.871 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55895.log 2024-09-17T04:09:48.872 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87379.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87379.log.gz 2024-09-17T04:09:48.872 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42017.log 2024-09-17T04:09:48.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91952.log 2024-09-17T04:09:48.874 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.55895.log.gz 2024-09-17T04:09:48.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105625.log 2024-09-17T04:09:48.875 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42017.log.gz 2024-09-17T04:09:48.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91952.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.91952.log.gz 2024-09-17T04:09:48.877 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98066.log 2024-09-17T04:09:48.877 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105625.log.gz 2024-09-17T04:09:48.878 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59732.log 2024-09-17T04:09:48.880 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79367.log 2024-09-17T04:09:48.880 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.98066.log.gz 2024-09-17T04:09:48.881 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59732.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.59732.log.gz 2024-09-17T04:09:48.882 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79367.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.79367.log.gz 2024-09-17T04:09:48.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77791.log 2024-09-17T04:09:48.883 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35898.log 2024-09-17T04:09:48.885 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97898.log 2024-09-17T04:09:48.886 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35898.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.77791.log.gz 2024-09-17T04:09:48.886 INFO:teuthology.orchestra.run.smithi129.stderr: 75.7% -- replaced with /var/log/ceph/ceph-client.admin.35898.log.gz 2024-09-17T04:09:48.886 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54039.log 2024-09-17T04:09:48.887 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97898.log: gzip 87.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.36170.log 2024-09-17T04:09:48.887 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.97898.log.gz 2024-09-17T04:09:48.889 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102789.log 2024-09-17T04:09:48.889 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54039.log.gz 2024-09-17T04:09:48.890 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.36170.log: gzip -5 --verbose -- 76.2% /var/log/ceph/ceph-client.admin.64928.log -- replaced with /var/log/ceph/ceph-client.admin.36170.log.gz 2024-09-17T04:09:48.890 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.891 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83501.log 2024-09-17T04:09:48.891 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.102789.log.gz 2024-09-17T04:09:48.893 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109269.log 2024-09-17T04:09:48.893 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9%/var/log/ceph/ceph-client.admin.83501.log: -- replaced with /var/log/ceph/ceph-client.admin.64928.log.gz 2024-09-17T04:09:48.894 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83501.log.gz 2024-09-17T04:09:48.894 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99114.log 2024-09-17T04:09:48.895 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109269.log: gzip -5 87.3% --verbose -- /var/log/ceph/ceph-client.admin.91830.log 2024-09-17T04:09:48.895 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109269.log.gz 2024-09-17T04:09:48.897 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45160.log 2024-09-17T04:09:48.897 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99114.log.gz 2024-09-17T04:09:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91830.log.gz 2024-09-17T04:09:48.898 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85255.log 2024-09-17T04:09:48.899 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45160.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.98444.log 2024-09-17T04:09:48.899 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45160.log.gz 2024-09-17T04:09:48.901 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64587.log 2024-09-17T04:09:48.901 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.85255.log.gz 2024-09-17T04:09:48.902 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98444.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98444.log.gz 2024-09-17T04:09:48.902 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.64587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93551.log 2024-09-17T04:09:48.903 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64587.log.gz 2024-09-17T04:09:48.903 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90760.log 2024-09-17T04:09:48.905 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81887.log 2024-09-17T04:09:48.905 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.93551.log.gz 2024-09-17T04:09:48.906 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90760.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90760.log.gz 2024-09-17T04:09:48.906 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35714.log 2024-09-17T04:09:48.908 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51770.log 2024-09-17T04:09:48.908 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81887.log.gz 2024-09-17T04:09:48.909 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65638.log 2024-09-17T04:09:48.909 INFO:teuthology.orchestra.run.smithi129.stderr: 76.8% -- replaced with /var/log/ceph/ceph-client.admin.35714.log.gz 2024-09-17T04:09:48.910 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63279.log 2024-09-17T04:09:48.910 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51770.log.gz 2024-09-17T04:09:48.912 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46484.log 2024-09-17T04:09:48.912 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65638.log.gz 2024-09-17T04:09:48.913 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76931.log 2024-09-17T04:09:48.913 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63279.log.gz 2024-09-17T04:09:48.914 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94072.log 2024-09-17T04:09:48.914 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46484.log.gz 2024-09-17T04:09:48.915 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76931.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76931.log.gz 2024-09-17T04:09:48.916 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80759.log 2024-09-17T04:09:48.916 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94072.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.94072.log.gz 2024-09-17T04:09:48.917 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44586.log 2024-09-17T04:09:48.920 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88000.log 2024-09-17T04:09:48.921 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.80759.log.gz 2024-09-17T04:09:48.921 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44586.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44586.log.gz 2024-09-17T04:09:48.921 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61936.log 2024-09-17T04:09:48.921 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88000.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88000.log.gz 2024-09-17T04:09:48.921 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108105.log 2024-09-17T04:09:48.922 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42601.log 2024-09-17T04:09:48.922 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61936.log.gz 2024-09-17T04:09:48.923 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108105.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.108105.log.gz 2024-09-17T04:09:48.925 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.42601.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59774.log 2024-09-17T04:09:48.925 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42601.log.gz 2024-09-17T04:09:48.925 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99240.log 2024-09-17T04:09:48.927 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58618.log 2024-09-17T04:09:48.927 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59774.log.gz 2024-09-17T04:09:48.929 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99240.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99240.log.gz 2024-09-17T04:09:48.929 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80456.log 2024-09-17T04:09:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58618.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58618.log.gz 2024-09-17T04:09:48.930 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96687.log 2024-09-17T04:09:48.931 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56460.log 2024-09-17T04:09:48.931 INFO:teuthology.orchestra.run.smithi129.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80456.log.gz 2024-09-17T04:09:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96687.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96687.log.gz 2024-09-17T04:09:48.933 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80927.log 2024-09-17T04:09:48.934 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56460.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56460.log.gz 2024-09-17T04:09:48.935 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95922.log 2024-09-17T04:09:48.936 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63096.log 2024-09-17T04:09:48.936 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80927.log.gz 2024-09-17T04:09:48.937 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95922.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95922.log.gz 2024-09-17T04:09:48.937 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93234.log 2024-09-17T04:09:48.938 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63096.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63096.log.gz 2024-09-17T04:09:48.938 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95651.log 2024-09-17T04:09:48.940 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61727.log 2024-09-17T04:09:48.940 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.93234.log.gz 2024-09-17T04:09:48.941 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72774.log 2024-09-17T04:09:48.941 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95651.log.gz 2024-09-17T04:09:48.942 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71893.log 2024-09-17T04:09:48.942 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61727.log.gz 2024-09-17T04:09:48.943 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.72774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72774.log.gz 2024-09-17T04:09:48.944 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77378.log 2024-09-17T04:09:48.945 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71893.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71893.log.gz 2024-09-17T04:09:48.945 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72516.log 2024-09-17T04:09:48.946 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.77378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42185.log 2024-09-17T04:09:48.946 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77378.log.gz 2024-09-17T04:09:48.948 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76131.log 2024-09-17T04:09:48.948 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.72516.log.gz 2024-09-17T04:09:48.948 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42185.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42185.log.gz 2024-09-17T04:09:48.949 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102352.log 2024-09-17T04:09:48.951 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.106526.log 2024-09-17T04:09:48.951 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76131.log.gz 2024-09-17T04:09:48.952 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102352.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102352.log.gz 2024-09-17T04:09:48.952 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75083.log 2024-09-17T04:09:48.953 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.106526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.106526.log.gz 2024-09-17T04:09:48.953 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92060.log 2024-09-17T04:09:48.954 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42953.log 2024-09-17T04:09:48.954 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75083.log.gz 2024-09-17T04:09:48.956 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101090.log 2024-09-17T04:09:48.956 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92060.log.gz 2024-09-17T04:09:48.957 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2024-09-17T04:09:48.957 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42953.log.gz 2024-09-17T04:09:48.958 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75433.log 2024-09-17T04:09:48.958 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101090.log.gz 2024-09-17T04:09:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45921.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2024-09-17T04:09:48.960 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92186.log 2024-09-17T04:09:48.961 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75433.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75433.log.gz 2024-09-17T04:09:48.961 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88256.log 2024-09-17T04:09:48.962 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41761.log 2024-09-17T04:09:48.962 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.92186.log.gz 2024-09-17T04:09:48.964 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88256.log: gzip -5 87.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.88256.log.gz /var/log/ceph/ceph-client.admin.85381.log 2024-09-17T04:09:48.964 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.965 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41761.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99012.log 2024-09-17T04:09:48.965 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41761.log.gz 2024-09-17T04:09:48.966 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88957.log 2024-09-17T04:09:48.966 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.85381.log.gz 2024-09-17T04:09:48.967 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99012.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99012.log.gz 2024-09-17T04:09:48.968 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82452.log 2024-09-17T04:09:48.968 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88957.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.88957.log.gz 2024-09-17T04:09:48.969 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63867.log 2024-09-17T04:09:48.970 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105354.log 2024-09-17T04:09:48.971 INFO:teuthology.orchestra.run.smithi129.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.82452.log.gz/var/log/ceph/ceph-client.admin.63867.log: 2024-09-17T04:09:48.973 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63867.log.gz 2024-09-17T04:09:48.973 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78626.log 2024-09-17T04:09:48.973 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105354.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.105354.log.gz 2024-09-17T04:09:48.974 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101227.log 2024-09-17T04:09:48.976 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78626.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78626.log.gz 2024-09-17T04:09:48.976 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.96475.log 2024-09-17T04:09:48.977 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101227.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101227.log.gz 2024-09-17T04:09:48.977 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74879.log 2024-09-17T04:09:48.978 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58828.log 2024-09-17T04:09:48.978 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.96475.log.gz 2024-09-17T04:09:48.979 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80885.log 2024-09-17T04:09:48.979 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74879.log.gz 2024-09-17T04:09:48.980 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53175.log 2024-09-17T04:09:48.980 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58828.log.gz 2024-09-17T04:09:48.981 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80885.log: gzip -5 87.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.80885.log.gz /var/log/ceph/ceph-client.admin.74795.log 2024-09-17T04:09:48.982 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.983 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52154.log 2024-09-17T04:09:48.983 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53175.log.gz 2024-09-17T04:09:48.984 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64055.log 2024-09-17T04:09:48.984 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74795.log.gz 2024-09-17T04:09:48.985 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.52154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52154.log.gz 2024-09-17T04:09:48.985 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.44407.log 2024-09-17T04:09:48.986 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.64055.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.64055.log.gz 2024-09-17T04:09:48.986 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100874.log 2024-09-17T04:09:48.987 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.2.44407.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72948.log 2024-09-17T04:09:48.988 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.2.44407.log.gz 2024-09-17T04:09:48.989 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83680.log 2024-09-17T04:09:48.989 INFO:teuthology.orchestra.run.smithi129.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.100874.log.gz 2024-09-17T04:09:48.990 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72948.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.72948.log.gz 2024-09-17T04:09:48.990 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74627.log 2024-09-17T04:09:48.991 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83680.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83680.log.gz 2024-09-17T04:09:48.992 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91910.log 2024-09-17T04:09:48.993 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71258.log 2024-09-17T04:09:48.993 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74627.log.gz 2024-09-17T04:09:48.994 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.91910.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91910.log.gz 2024-09-17T04:09:48.995 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100158.log 2024-09-17T04:09:48.995 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71258.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71258.log.gz 2024-09-17T04:09:48.996 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35760.log 2024-09-17T04:09:48.997 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96265.log 2024-09-17T04:09:48.997 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100158.log.gz 2024-09-17T04:09:48.998 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35760.log: gzip -5 --verbose 74.9% -- /var/log/ceph/ceph-client.admin.44881.log -- replaced with /var/log/ceph/ceph-client.admin.35760.log.gz 2024-09-17T04:09:48.998 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:48.999 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62886.log 2024-09-17T04:09:48.999 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96265.log.gz 2024-09-17T04:09:49.001 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56931.log 2024-09-17T04:09:49.001 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44881.log.gz 2024-09-17T04:09:49.002 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62886.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62886.log.gz 2024-09-17T04:09:49.002 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72162.log 2024-09-17T04:09:49.003 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56931.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56931.log.gz 2024-09-17T04:09:49.003 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63191.log 2024-09-17T04:09:49.005 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82518.log 2024-09-17T04:09:49.005 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72162.log.gz 2024-09-17T04:09:49.006 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56757.log 2024-09-17T04:09:49.006 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63191.log.gz 2024-09-17T04:09:49.007 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63233.log 2024-09-17T04:09:49.008 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82518.log.gz 2024-09-17T04:09:49.009 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60595.log 2024-09-17T04:09:49.009 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56757.log.gz 2024-09-17T04:09:49.010 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63233.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63233.log.gz 2024-09-17T04:09:49.011 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60595.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60595.log.gz 2024-09-17T04:09:49.012 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35862.log 2024-09-17T04:09:49.013 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52381.log 2024-09-17T04:09:49.014 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.35862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69035.log 2024-09-17T04:09:49.014 INFO:teuthology.orchestra.run.smithi129.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35862.log.gz 2024-09-17T04:09:49.015 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52381.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52381.log.gz 2024-09-17T04:09:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70107.log 2024-09-17T04:09:49.016 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69035.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69035.log.gz 2024-09-17T04:09:49.017 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43379.log 2024-09-17T04:09:49.018 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65281.log 2024-09-17T04:09:49.018 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70107.log.gz 2024-09-17T04:09:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43379.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2024-09-17T04:09:49.021 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42618.log 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.65281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76341.log 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65281.log.gz 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42618.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92786.log 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42618.log.gz 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76341.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76341.log.gz 2024-09-17T04:09:49.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63140.log 2024-09-17T04:09:49.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92786.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92786.log.gz 2024-09-17T04:09:49.027 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43421.log 2024-09-17T04:09:49.027 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83091.log 2024-09-17T04:09:49.028 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.63140.log.gz 2024-09-17T04:09:49.028 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98402.log 2024-09-17T04:09:49.029 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.43421.log.gz 2024-09-17T04:09:49.030 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90718.log 2024-09-17T04:09:49.030 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.83091.log.gz 2024-09-17T04:09:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.98402.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98402.log.gz 2024-09-17T04:09:49.031 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74152.log 2024-09-17T04:09:49.032 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90718.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63801.log 2024-09-17T04:09:49.033 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90718.log.gz 2024-09-17T04:09:49.034 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76297.log 2024-09-17T04:09:49.034 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74152.log.gz 2024-09-17T04:09:49.035 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63801.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.63801.log.gz 2024-09-17T04:09:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79139.log 2024-09-17T04:09:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76297.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76297.log.gz 2024-09-17T04:09:49.036 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70987.log 2024-09-17T04:09:49.038 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95609.log 2024-09-17T04:09:49.038 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.79139.log.gz 2024-09-17T04:09:49.040 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70987.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70987.log.gz 2024-09-17T04:09:49.040 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78036.log 2024-09-17T04:09:49.041 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95609.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95609.log.gz 2024-09-17T04:09:49.041 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70401.log 2024-09-17T04:09:49.042 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88392.log 2024-09-17T04:09:49.042 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.78036.log.gz 2024-09-17T04:09:49.043 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70401.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70401.log.gz 2024-09-17T04:09:49.044 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88350.log 2024-09-17T04:09:49.044 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88392.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.88392.log.gz 2024-09-17T04:09:49.045 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66516.log 2024-09-17T04:09:49.046 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94762.log 2024-09-17T04:09:49.046 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88350.log.gz 2024-09-17T04:09:49.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66516.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90016.log 2024-09-17T04:09:49.048 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66516.log.gz 2024-09-17T04:09:49.048 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94762.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94762.log.gz 2024-09-17T04:09:49.049 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60679.log 2024-09-17T04:09:49.050 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49641.log 2024-09-17T04:09:49.050 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.90016.log.gz 2024-09-17T04:09:49.051 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109480.log 2024-09-17T04:09:49.052 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60679.log.gz 2024-09-17T04:09:49.053 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46778.log 2024-09-17T04:09:49.053 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.49641.log.gz 2024-09-17T04:09:49.054 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52150.log 2024-09-17T04:09:49.054 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.109480.log.gz 2024-09-17T04:09:49.055 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84988.log 2024-09-17T04:09:49.055 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46778.log.gz 2024-09-17T04:09:49.056 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.52150.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.52150.log.gz 2024-09-17T04:09:49.056 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61978.log 2024-09-17T04:09:49.058 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77738.log 2024-09-17T04:09:49.058 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84988.log.gz 2024-09-17T04:09:49.059 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61978.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61978.log.gz 2024-09-17T04:09:49.060 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5/var/log/ceph/ceph-client.admin.77738.log: --verbose -- /var/log/ceph/ceph-client.0.74022.log 2024-09-17T04:09:49.060 INFO:teuthology.orchestra.run.smithi129.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77738.log.gz 2024-09-17T04:09:49.061 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89638.log 2024-09-17T04:09:49.062 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.74022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74022.log.gz 2024-09-17T04:09:49.062 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95507.log 2024-09-17T04:09:49.063 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45878.log 2024-09-17T04:09:49.063 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.89638.log.gz 2024-09-17T04:09:49.064 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72474.log 2024-09-17T04:09:49.065 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.95507.log.gz 2024-09-17T04:09:49.066 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67200.log 2024-09-17T04:09:49.066 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45878.log.gz 2024-09-17T04:09:49.067 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72474.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.72474.log.gz 2024-09-17T04:09:49.067 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67074.log 2024-09-17T04:09:49.068 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67200.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.67200.log.gz 2024-09-17T04:09:49.068 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93635.log 2024-09-17T04:09:49.070 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68519.log 2024-09-17T04:09:49.070 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67074.log.gz 2024-09-17T04:09:49.071 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93635.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93635.log.gz 2024-09-17T04:09:49.072 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.68519.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.100065.log 2024-09-17T04:09:49.072 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68519.log.gz 2024-09-17T04:09:49.073 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59648.log 2024-09-17T04:09:49.075 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.100065.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/ceph-client.admin.100065.log.gz 2024-09-17T04:09:49.075 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.42925.log 2024-09-17T04:09:49.076 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59648.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59648.log.gz 2024-09-17T04:09:49.076 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80317.log 2024-09-17T04:09:49.077 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63363.log 2024-09-17T04:09:49.077 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42925.log.gz 2024-09-17T04:09:49.079 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58169.log 2024-09-17T04:09:49.079 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80317.log.gz 2024-09-17T04:09:49.080 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79577.log 2024-09-17T04:09:49.080 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63363.log.gz 2024-09-17T04:09:49.081 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44544.log 2024-09-17T04:09:49.081 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58169.log.gz 2024-09-17T04:09:49.082 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47158.log 2024-09-17T04:09:49.082 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79577.log.gz 2024-09-17T04:09:49.084 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.44544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60395.log 2024-09-17T04:09:49.084 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44544.log.gz 2024-09-17T04:09:49.084 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.47158.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47158.log.gz 2024-09-17T04:09:49.085 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75956.log 2024-09-17T04:09:49.086 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60395.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.60395.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76637.log 2024-09-17T04:09:49.086 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.088 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80843.log 2024-09-17T04:09:49.088 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.75956.log.gz 2024-09-17T04:09:49.090 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76637.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76637.log.gz 2024-09-17T04:09:49.090 INFO:teuthology.orchestra.run.smithi129.stderr:gzip/var/log/ceph/ceph-client.admin.80843.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56021.log 2024-09-17T04:09:49.090 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80843.log.gz 2024-09-17T04:09:49.091 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71769.log 2024-09-17T04:09:49.093 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71300.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56021.log.gz 2024-09-17T04:09:49.093 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71769.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71769.log.gz 2024-09-17T04:09:49.094 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48425.log 2024-09-17T04:09:49.095 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.71300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67345.log 2024-09-17T04:09:49.095 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.71300.log.gz 2024-09-17T04:09:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48425.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48425.log.gz 2024-09-17T04:09:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68267.log 2024-09-17T04:09:49.097 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67345.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67345.log.gz 2024-09-17T04:09:49.098 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67032.log 2024-09-17T04:09:49.099 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66330.log 2024-09-17T04:09:49.099 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68267.log.gz 2024-09-17T04:09:49.101 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79451.log 2024-09-17T04:09:49.101 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.67032.log.gz 2024-09-17T04:09:49.102 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66330.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66330.log.gz 2024-09-17T04:09:49.103 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79451.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.79451.log.gz 2024-09-17T04:09:49.103 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2024-09-17T04:09:49.104 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70065.log 2024-09-17T04:09:49.106 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76427.log 2024-09-17T04:09:49.106 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2024-09-17T04:09:49.107 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.70065.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70065.log.gz 2024-09-17T04:09:49.108 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.76427.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76427.log.gz 2024-09-17T04:09:49.108 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65186.log 2024-09-17T04:09:49.110 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101405.log 2024-09-17T04:09:49.111 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.65186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95693.log 2024-09-17T04:09:49.111 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.65186.log.gz 2024-09-17T04:09:49.112 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96006.log 2024-09-17T04:09:49.112 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101405.log.gz 2024-09-17T04:09:49.113 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95693.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.95693.log.gz 2024-09-17T04:09:49.114 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96006.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96006.log.gz 2024-09-17T04:09:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107224.log 2024-09-17T04:09:49.116 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61089.log 2024-09-17T04:09:49.117 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.107224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107224.log.gzgzip 2024-09-17T04:09:49.118 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.108862.log 2024-09-17T04:09:49.119 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107937.log 2024-09-17T04:09:49.119 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61089.log.gz 2024-09-17T04:09:49.120 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108862.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.108862.log.gz 2024-09-17T04:09:49.124 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.107937.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107937.log.gz 2024-09-17T04:09:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57238.log 2024-09-17T04:09:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68221.log 2024-09-17T04:09:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.57238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84477.log 2024-09-17T04:09:49.128 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57238.log.gz 2024-09-17T04:09:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68221.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68221.log.gz 2024-09-17T04:09:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52459.log 2024-09-17T04:09:49.129 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84477.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96981.log 2024-09-17T04:09:49.129 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84477.log.gz 2024-09-17T04:09:49.130 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105008.log 2024-09-17T04:09:49.131 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52459.log.gz 2024-09-17T04:09:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96981.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.96981.log.gz 2024-09-17T04:09:49.131 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62501.log/var/log/ceph/ceph-client.admin.105008.log: 2024-09-17T04:09:49.131 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105008.log.gz 2024-09-17T04:09:49.132 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66854.log 2024-09-17T04:09:49.133 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63321.log 2024-09-17T04:09:49.133 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62501.log.gz 2024-09-17T04:09:49.136 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.66854.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66854.log.gz 2024-09-17T04:09:49.136 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.63321.log: -- /var/log/ceph/ceph-client.admin.82143.log 2024-09-17T04:09:49.136 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63321.log.gz 2024-09-17T04:09:49.137 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55208.log 2024-09-17T04:09:49.138 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108610.log 2024-09-17T04:09:49.138 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82143.log.gz 2024-09-17T04:09:49.140 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55208.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55208.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51289.log 2024-09-17T04:09:49.140 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.140 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54169.log 2024-09-17T04:09:49.141 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.108610.log.gz 2024-09-17T04:09:49.144 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83258.log 2024-09-17T04:09:49.144 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51289.log.gz 2024-09-17T04:09:49.144 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54169.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54169.log.gz 2024-09-17T04:09:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99366.log 2024-09-17T04:09:49.145 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87131.log 2024-09-17T04:09:49.147 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.99366.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.83258.log.gz 2024-09-17T04:09:49.147 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.99366.log.gz 2024-09-17T04:09:49.147 INFO:teuthology.orchestra.run.smithi129.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.105273.log 2024-09-17T04:09:49.147 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.87131.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87131.log.gz 2024-09-17T04:09:49.147 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85032.log 2024-09-17T04:09:49.148 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.105273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84902.log 2024-09-17T04:09:49.148 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105273.log.gz 2024-09-17T04:09:49.149 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85032.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.85032.log.gz 2024-09-17T04:09:49.150 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68179.log 2024-09-17T04:09:49.151 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84902.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45405.log 2024-09-17T04:09:49.151 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84902.log.gz 2024-09-17T04:09:49.152 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61461.log 2024-09-17T04:09:49.152 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68179.log.gz 2024-09-17T04:09:49.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45405.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45405.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50535.log 2024-09-17T04:09:49.154 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.154 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61461.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.61461.log.gz 2024-09-17T04:09:49.156 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50535.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.50535.log.gz 2024-09-17T04:09:49.156 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68393.log 2024-09-17T04:09:49.158 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92870.log 2024-09-17T04:09:49.159 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81794.log 2024-09-17T04:09:49.159 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68393.log.gz 2024-09-17T04:09:49.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92870.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92870.log.gz 2024-09-17T04:09:49.161 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108526.log 2024-09-17T04:09:49.161 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81794.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81794.log.gz 2024-09-17T04:09:49.162 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94624.log 2024-09-17T04:09:49.163 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69813.log 2024-09-17T04:09:49.163 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.108526.log.gz 2024-09-17T04:09:49.165 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.94624.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94624.log.gz 2024-09-17T04:09:49.165 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.69813.log: -- /var/log/ceph/ceph-client.admin.54434.log 2024-09-17T04:09:49.166 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69813.log.gz 2024-09-17T04:09:49.166 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83723.log 2024-09-17T04:09:49.167 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63447.log 2024-09-17T04:09:49.168 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54434.log.gz 2024-09-17T04:09:49.169 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96897.log 2024-09-17T04:09:49.169 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83723.log.gz 2024-09-17T04:09:49.170 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63447.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.63447.log.gz 2024-09-17T04:09:49.170 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80377.log 2024-09-17T04:09:49.171 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48934.log 2024-09-17T04:09:49.172 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96897.log.gz 2024-09-17T04:09:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.80377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101489.log 2024-09-17T04:09:49.173 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48934.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.80377.log.gz 2024-09-17T04:09:49.174 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90891.log 2024-09-17T04:09:49.174 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48934.log.gz 2024-09-17T04:09:49.175 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101489.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.101489.log.gz 2024-09-17T04:09:49.176 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90891.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90891.log.gz 2024-09-17T04:09:49.177 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93593.log 2024-09-17T04:09:49.178 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74753.log 2024-09-17T04:09:49.180 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.93593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62806.log 2024-09-17T04:09:49.180 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.93593.log.gz 2024-09-17T04:09:49.181 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74753.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74753.log.gz 2024-09-17T04:09:49.181 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72582.log 2024-09-17T04:09:49.182 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73716.log 2024-09-17T04:09:49.182 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62806.log.gz 2024-09-17T04:09:49.183 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.72582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45279.log 2024-09-17T04:09:49.184 INFO:teuthology.orchestra.run.smithi129.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72582.log.gz 2024-09-17T04:09:49.185 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59924.log 2024-09-17T04:09:49.185 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73716.log.gz 2024-09-17T04:09:49.186 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.45279.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45279.log.gz 2024-09-17T04:09:49.187 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59924.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59924.log.gz 2024-09-17T04:09:49.187 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102705.log 2024-09-17T04:09:49.189 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83314.log 2024-09-17T04:09:49.190 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94432.log 2024-09-17T04:09:49.190 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102705.log.gz 2024-09-17T04:09:49.191 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83314.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.83314.log.gz 2024-09-17T04:09:49.192 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92828.log 2024-09-17T04:09:49.192 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94432.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94432.log.gz 2024-09-17T04:09:49.193 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68024.log 2024-09-17T04:09:49.194 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42143.log 2024-09-17T04:09:49.194 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.92828.log.gz 2024-09-17T04:09:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68024.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68024.log.gz 2024-09-17T04:09:49.196 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42143.log: -- /var/log/ceph/ceph-client.admin.53217.log 2024-09-17T04:09:49.196 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz 2024-09-17T04:09:49.197 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-09-17T04:09:49.198 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59383.log 2024-09-17T04:09:49.199 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53217.log.gz 2024-09-17T04:09:49.200 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41679.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-09-17T04:09:49.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49064.log 2024-09-17T04:09:49.201 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108736.log 2024-09-17T04:09:49.202 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.59383.log.gz 2024-09-17T04:09:49.202 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49064.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49064.log.gz 2024-09-17T04:09:49.203 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75349.log 2024-09-17T04:09:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.108736.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.108736.log.gz 2024-09-17T04:09:49.204 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74669.log 2024-09-17T04:09:49.206 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.75349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43193.log 2024-09-17T04:09:49.206 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75349.log.gz 2024-09-17T04:09:49.207 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74669.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74669.log.gz 2024-09-17T04:09:49.207 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83624.log 2024-09-17T04:09:49.208 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43193.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.52232.log 2024-09-17T04:09:49.209 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43193.log.gz 2024-09-17T04:09:49.210 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102440.log 2024-09-17T04:09:49.211 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.52232.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.83624.log.gz 2024-09-17T04:09:49.211 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52232.log.gz 2024-09-17T04:09:49.212 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102440.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.102440.log.gz 2024-09-17T04:09:49.212 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89220.log 2024-09-17T04:09:49.214 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46173.log 2024-09-17T04:09:49.215 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82353.log 2024-09-17T04:09:49.216 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89220.log.gz 2024-09-17T04:09:49.216 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46173.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46173.log.gz 2024-09-17T04:09:49.217 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103824.log 2024-09-17T04:09:49.218 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59339.log 2024-09-17T04:09:49.218 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82353.log.gz 2024-09-17T04:09:49.219 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109178.log 2024-09-17T04:09:49.220 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2%/var/log/ceph/ceph-client.admin.59339.log: -- replaced with /var/log/ceph/ceph-client.admin.103824.log.gz 2024-09-17T04:09:49.220 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.59339.log.gz 2024-09-17T04:09:49.220 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56597.log 2024-09-17T04:09:49.222 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.109178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48593.log 2024-09-17T04:09:49.222 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.109178.log.gz 2024-09-17T04:09:49.223 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86027.log 2024-09-17T04:09:49.223 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56597.log.gz 2024-09-17T04:09:49.224 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.48593.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48593.log.gz 2024-09-17T04:09:49.224 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102106.log 2024-09-17T04:09:49.226 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81095.log 2024-09-17T04:09:49.226 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.86027.log.gz 2024-09-17T04:09:49.227 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.102106.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102106.log.gz 2024-09-17T04:09:49.227 INFO:teuthology.orchestra.run.smithi129.stderr: /var/log/ceph/ceph-client.admin.81750.log 2024-09-17T04:09:49.228 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81095.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81095.log.gz 2024-09-17T04:09:49.228 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88747.log 2024-09-17T04:09:49.230 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60899.log 2024-09-17T04:09:49.230 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81750.log.gz 2024-09-17T04:09:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88747.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.88747.log.gz 2024-09-17T04:09:49.231 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83581.log 2024-09-17T04:09:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.60899.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103561.log 2024-09-17T04:09:49.233 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60899.log.gz 2024-09-17T04:09:49.234 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-09-17T04:09:49.234 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83581.log.gz 2024-09-17T04:09:49.235 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.103561.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.103561.log.gz 2024-09-17T04:09:49.236 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78247.log 2024-09-17T04:09:49.237 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59129.log 2024-09-17T04:09:49.237 INFO:teuthology.orchestra.run.smithi129.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-09-17T04:09:49.238 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99324.log 2024-09-17T04:09:49.238 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78247.log.gz 2024-09-17T04:09:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59129.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59129.log.gz 2024-09-17T04:09:49.240 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.99324.log: /var/log/ceph/ceph-client.admin.56063.log 2024-09-17T04:09:49.240 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.99324.log.gz 2024-09-17T04:09:49.241 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68819.log 2024-09-17T04:09:49.242 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.56063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106604.log 2024-09-17T04:09:49.243 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.56063.log.gz 2024-09-17T04:09:49.243 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.68819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.68819.log.gz 2024-09-17T04:09:49.244 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58305.log 2024-09-17T04:09:49.245 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106604.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.106604.log.gz 2024-09-17T04:09:49.246 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50493.log 2024-09-17T04:09:49.247 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97583.log 2024-09-17T04:09:49.247 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58305.log.gz 2024-09-17T04:09:49.248 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50493.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.50493.log.gz 2024-09-17T04:09:49.248 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59210.log 2024-09-17T04:09:49.249 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97583.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97583.log.gz 2024-09-17T04:09:49.249 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67814.log 2024-09-17T04:09:49.251 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88560.log 2024-09-17T04:09:49.252 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59210.log.gz 2024-09-17T04:09:49.252 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67814.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67814.log.gz 2024-09-17T04:09:49.253 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42916.log 2024-09-17T04:09:49.254 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54125.log 2024-09-17T04:09:49.254 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88560.log.gz 2024-09-17T04:09:49.255 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.42916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59476.log 2024-09-17T04:09:49.255 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42916.log.gz 2024-09-17T04:09:49.256 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55427.log 2024-09-17T04:09:49.257 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54125.log.gz 2024-09-17T04:09:49.258 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.59476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85339.log 2024-09-17T04:09:49.258 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59476.log.gz 2024-09-17T04:09:49.259 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55853.log 2024-09-17T04:09:49.259 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55427.log.gz 2024-09-17T04:09:49.260 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85339.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.85339.log.gz 2024-09-17T04:09:49.261 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90634.log 2024-09-17T04:09:49.261 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55853.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55853.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46568.log 2024-09-17T04:09:49.262 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.263 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90830.log 2024-09-17T04:09:49.263 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90634.log.gz 2024-09-17T04:09:49.264 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46568.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46568.log.gz 2024-09-17T04:09:49.265 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43463.log 2024-09-17T04:09:49.265 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90830.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.90830.log.gz 2024-09-17T04:09:49.266 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47317.log 2024-09-17T04:09:49.267 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.43463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-09-17T04:09:49.268 INFO:teuthology.orchestra.run.smithi129.stderr: 91.2%/var/log/ceph/ceph-client.admin.47317.log: -- replaced with /var/log/ceph/ceph-client.admin.43463.log.gz 2024-09-17T04:09:49.269 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47317.log.gz 2024-09-17T04:09:49.269 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.0.57846.log 2024-09-17T04:09:49.269 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.41889.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-09-17T04:09:49.270 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89300.log 2024-09-17T04:09:49.271 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.57846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.57846.log.gz 2024-09-17T04:09:49.271 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68435.log 2024-09-17T04:09:49.272 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2024-09-17T04:09:49.272 INFO:teuthology.orchestra.run.smithi129.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.89300.log.gz 2024-09-17T04:09:49.274 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97940.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.68435.log.gz 2024-09-17T04:09:49.274 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.275 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50189.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2024-09-17T04:09:49.275 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106764.log 2024-09-17T04:09:49.276 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.97940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89974.log 2024-09-17T04:09:49.276 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97940.log.gz 2024-09-17T04:09:49.278 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.106764.log: gzip -5 --verbose 87.2% -- replaced with /var/log/ceph/ceph-client.admin.106764.log.gz 2024-09-17T04:09:49.278 INFO:teuthology.orchestra.run.smithi129.stderr: -- /var/log/ceph/ceph-client.admin.78924.log 2024-09-17T04:09:49.279 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89974.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89974.log.gz 2024-09-17T04:09:49.279 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-17T04:09:49.280 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107454.log 2024-09-17T04:09:49.281 INFO:teuthology.orchestra.run.smithi129.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78924.log.gz 2024-09-17T04:09:49.283 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.107454.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107454.log.gz 2024-09-17T04:09:49.284 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92618.log 2024-09-17T04:09:49.285 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94114.log 2024-09-17T04:09:49.286 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.92618.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92618.log.gz 2024-09-17T04:09:49.287 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61842.log 2024-09-17T04:09:49.288 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.94114.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.94114.log.gz 2024-09-17T04:09:49.289 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101531.log 2024-09-17T04:09:49.290 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.61842.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.61842.log.gz 2024-09-17T04:09:49.291 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.65182.log 2024-09-17T04:09:49.291 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.101531.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101531.log.gz 2024-09-17T04:09:49.292 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74501.log 2024-09-17T04:09:49.293 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.65182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.65182.log.gz 2024-09-17T04:09:49.294 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68561.log 2024-09-17T04:09:49.295 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.74501.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74501.log.gz 2024-09-17T04:09:49.296 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104854.log 2024-09-17T04:09:49.297 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.68561.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.68561.log.gz 2024-09-17T04:09:49.298 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47162.log 2024-09-17T04:09:49.298 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.104854.log.gz 2024-09-17T04:09:49.299 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58660.log 2024-09-17T04:09:49.300 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.47162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.47162.log.gz 2024-09-17T04:09:49.301 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73215.log 2024-09-17T04:09:49.302 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58660.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58660.log.gz 2024-09-17T04:09:49.303 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53404.log 2024-09-17T04:09:49.304 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.73215.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73215.log.gz 2024-09-17T04:09:49.305 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85083.log 2024-09-17T04:09:49.306 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.53404.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53404.log.gz 2024-09-17T04:09:49.306 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-09-17T04:09:49.308 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.85083.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.85083.log.gz 2024-09-17T04:09:49.308 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74333.log 2024-09-17T04:09:49.311 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.74333.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74333.log.gz 2024-09-17T04:09:49.312 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.74026.log 2024-09-17T04:09:49.315 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.74026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.74026.log.gz 2024-09-17T04:09:49.315 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84860.log 2024-09-17T04:09:49.318 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.84860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86374.log 2024-09-17T04:09:49.319 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.84860.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62669.log 2024-09-17T04:09:49.319 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.322 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.86374.log: 91.0%/var/log/ceph/ceph-client.admin.62669.log: -- replaced with /var/log/ceph/ceph-client.admin.86374.log.gz 2024-09-17T04:09:49.323 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62669.log.gz 2024-09-17T04:09:49.323 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89932.log 2024-09-17T04:09:49.329 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104732.log 2024-09-17T04:09:49.330 INFO:teuthology.orchestra.run.smithi129.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.89932.log.gz 2024-09-17T04:09:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.104732.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104732.log.gz 2024-09-17T04:09:49.330 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82185.log 2024-09-17T04:09:49.332 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.82185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95168.log 2024-09-17T04:09:49.333 INFO:teuthology.orchestra.run.smithi129.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.82185.log.gz 2024-09-17T04:09:49.333 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45070.log 2024-09-17T04:09:49.336 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95168.log: /var/log/ceph/ceph-client.admin.45070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50325.log 2024-09-17T04:09:49.337 INFO:teuthology.orchestra.run.smithi129.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45070.log.gz 2024-09-17T04:09:49.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 91.0% -- replaced with /var/log/ceph/ceph-client.admin.95168.log.gz 2024-09-17T04:09:49.339 INFO:teuthology.orchestra.run.smithi129.stderr: --verbose -- /var/log/ceph/ceph-client.admin.83377.log 2024-09-17T04:09:49.343 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.50325.log: /var/log/ceph/ceph-client.admin.83377.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55365.log 2024-09-17T04:09:49.343 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50325.log.gz 2024-09-17T04:09:49.344 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.83377.log.gz 2024-09-17T04:09:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51155.log 2024-09-17T04:09:49.349 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55365.log: /var/log/ceph/ceph-client.admin.51155.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55303.log 2024-09-17T04:09:49.350 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55365.log.gz 2024-09-17T04:09:49.351 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51155.log.gz 2024-09-17T04:09:49.352 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95287.log 2024-09-17T04:09:49.354 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.55303.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55303.log.gz 2024-09-17T04:09:49.357 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.95287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63489.log 2024-09-17T04:09:49.360 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.63489.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95287.log.gz 2024-09-17T04:09:49.361 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54704.log 2024-09-17T04:09:49.362 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63489.log.gz 2024-09-17T04:09:49.363 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96855.log 2024-09-17T04:09:49.367 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.96855.log: /var/log/ceph/ceph-client.admin.54704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94030.log 2024-09-17T04:09:49.369 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.54704.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53259.log 2024-09-17T04:09:49.369 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:09:49.370 INFO:teuthology.orchestra.run.smithi129.stderr: 87.1%/var/log/ceph/ceph-client.admin.94030.log: -- replaced with /var/log/ceph/ceph-client.admin.96855.log.gz 2024-09-17T04:09:49.372 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2%/var/log/ceph/ceph-client.admin.53259.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53259.log.gz 2024-09-17T04:09:49.373 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94030.log.gz 2024-09-17T04:09:49.374 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69729.log 2024-09-17T04:09:49.376 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.69729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46652.log 2024-09-17T04:09:49.377 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.69729.log.gz 2024-09-17T04:09:49.379 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36151.log 2024-09-17T04:09:49.383 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.46652.log: /var/log/ceph/ceph-client.admin.36151.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90268.log 2024-09-17T04:09:49.383 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46652.log.gz 2024-09-17T04:09:49.384 INFO:teuthology.orchestra.run.smithi129.stderr: 77.6% -- replaced with /var/log/ceph/ceph-client.admin.36151.log.gz 2024-09-17T04:09:49.386 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102491.log 2024-09-17T04:09:49.390 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.90268.log: /var/log/ceph/ceph-client.admin.102491.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58870.log 2024-09-17T04:09:49.390 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90268.log.gz 2024-09-17T04:09:49.391 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102491.log.gz 2024-09-17T04:09:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.58870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49599.log 2024-09-17T04:09:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.49599.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58870.log.gz 2024-09-17T04:09:49.397 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105134.log 2024-09-17T04:09:49.399 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79223.log 2024-09-17T04:09:49.401 INFO:teuthology.orchestra.run.smithi129.stderr: 86.8%/var/log/ceph/ceph-client.admin.105134.log: -- replaced with /var/log/ceph/ceph-client.admin.49599.log.gz 2024-09-17T04:09:49.402 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105134.log.gz 2024-09-17T04:09:49.402 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.79223.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79223.log.gz 2024-09-17T04:09:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62062.log 2024-09-17T04:09:49.405 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.62062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54560.log 2024-09-17T04:09:49.407 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103473.log 2024-09-17T04:09:49.407 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62062.log.gz 2024-09-17T04:09:49.408 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.54560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49455.log 2024-09-17T04:09:49.411 INFO:teuthology.orchestra.run.smithi129.stderr: 87.4%/var/log/ceph/ceph-client.admin.103473.log: /var/log/ceph/ceph-client.admin.49455.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.49455.log.gz 2024-09-17T04:09:49.412 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54560.log.gz 2024-09-17T04:09:49.412 INFO:teuthology.orchestra.run.smithi129.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67896.log 2024-09-17T04:09:49.413 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.103473.log.gz 2024-09-17T04:09:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.67896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51069.log 2024-09-17T04:09:49.417 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51069.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51069.log.gz 2024-09-17T04:09:49.418 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 87.8% --verbose -- /var/log/ceph/ceph-client.admin.78499.log 2024-09-17T04:09:49.420 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.78499.log: -- replaced with /var/log/ceph/ceph-client.admin.67896.log.gz 2024-09-17T04:09:49.421 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89596.log 2024-09-17T04:09:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51728.log 2024-09-17T04:09:49.425 INFO:teuthology.orchestra.run.smithi129.stderr: 87.7%/var/log/ceph/ceph-client.admin.89596.log: -- replaced with /var/log/ceph/ceph-client.admin.78499.log.gz 2024-09-17T04:09:49.426 INFO:teuthology.orchestra.run.smithi129.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.89596.log.gz 2024-09-17T04:09:49.428 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.51728.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81137.log 2024-09-17T04:09:49.428 INFO:teuthology.orchestra.run.smithi129.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51728.log.gz 2024-09-17T04:09:49.431 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81179.log 2024-09-17T04:09:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.81179.log: /var/log/ceph/ceph-client.admin.81137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88873.log 2024-09-17T04:09:49.436 INFO:teuthology.orchestra.run.smithi129.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81137.log.gz 2024-09-17T04:09:49.436 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83954.log 2024-09-17T04:09:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.88873.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.81179.log.gz 2024-09-17T04:09:49.439 INFO:teuthology.orchestra.run.smithi129.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.88873.log.gz 2024-09-17T04:09:49.440 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89083.log 2024-09-17T04:09:49.442 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.83954.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83954.log.gz 2024-09-17T04:09:49.444 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.admin.89083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72770.log 2024-09-17T04:09:49.447 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/ceph-client.0.72770.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.89083.log.gz 2024-09-17T04:09:49.448 INFO:teuthology.orchestra.run.smithi129.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.72770.log.gz 2024-09-17T04:09:49.724 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-17T04:09:50.136 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-17T04:09:50.207 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-17T04:09:50.208 INFO:teuthology.orchestra.run.smithi181.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-17T04:09:50.211 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-17T04:09:50.211 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m2.303s 2024-09-17T04:09:50.211 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m10.300s 2024-09-17T04:09:50.211 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.980s 2024-09-17T04:09:50.421 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-17T04:09:50.423 INFO:teuthology.orchestra.run.smithi129.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-17T04:09:50.623 INFO:teuthology.orchestra.run.smithi129.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-17T04:09:51.349 INFO:teuthology.orchestra.run.smithi129.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-17T04:09:51.521 INFO:teuthology.orchestra.run.smithi129.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-17T04:09:51.915 INFO:teuthology.orchestra.run.smithi129.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-09-17T04:10:02.576 INFO:teuthology.orchestra.run.smithi129.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-09-17T04:10:02.579 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-17T04:10:02.579 INFO:teuthology.orchestra.run.smithi129.stderr:real 0m14.675s 2024-09-17T04:10:02.579 INFO:teuthology.orchestra.run.smithi129.stderr:user 0m26.541s 2024-09-17T04:10:02.579 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m5.117s 2024-09-17T04:10:02.580 INFO:tasks.ceph:Archiving logs... 2024-09-17T04:10:02.581 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi129/log 2024-09-17T04:10:02.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T04:10:06.974 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi181/log 2024-09-17T04:10:06.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T04:10:08.323 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-17T04:10:08.333 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-17T04:10:08.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T04:10:08.338 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-17T04:10:08.449 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T04:10:08.458 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T04:10:08.458 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T04:10:08.463 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T04:10:08.494 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-17T04:10:08.497 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 7 377 70 +1592us[+1589us] +/- 36ms 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 +1302ns[+1302ns] +/- 73ms 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 -3334us[-3334us] +/- 64ms 2024-09-17T04:10:08.501 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 1 +1709us[+1642us] +/- 37ms 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 2 +146us[ +78us] +/- 73ms 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -3173us[-3240us] +/- 64ms 2024-09-17T04:10:08.504 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T04:10:08.505 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T04:10:08.514 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T04:10:08.515 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T04:10:08.524 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T04:10:08.704 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T04:10:08.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T04:10:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:10:08.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T04:10:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:10:08.946 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T04:10:08.955 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T04:10:08.964 INFO:teuthology.task.internal:Duration was 1714.048485 seconds 2024-09-17T04:10:08.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T04:10:08.974 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T04:10:08.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T04:10:08.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T04:10:09.062 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T04:10:09.064 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T04:10:09.485 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T04:10:09.485 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-17T04:10:09.485 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T04:10:09.533 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-17T04:10:09.534 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T04:10:09.587 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T04:10:09.588 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T04:10:09.592 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T04:10:09.655 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T04:10:09.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T04:10:09.660 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T04:10:09.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T04:10:09.884 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T04:10:09.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T04:10:09.940 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T04:10:09.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T04:10:10.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T04:10:10.015 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T04:10:10.081 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-17T04:10:10.085 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-17T04:10:10.114 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T04:10:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:10:10.148 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T04:10:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T04:10:10.181 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T04:10:10.191 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T04:10:10.192 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi129 2024-09-17T04:10:10.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T04:10:10.261 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-17_02:09:51-fs-wip-pdonnell-testing-20240916.200549-debug-distro-default-smithi/7908469/remote/smithi181 2024-09-17T04:10:10.262 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T04:10:10.325 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T04:10:10.325 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T04:10:10.329 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T04:10:10.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T04:10:10.377 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T04:10:10.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T04:10:10.387 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T04:10:10.387 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T04:10:10.405 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T04:10:10.435 INFO:teuthology.orchestra.run.smithi129.stdout: 393253 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 04:10 /home/ubuntu/cephtest 2024-09-17T04:10:10.440 INFO:teuthology.orchestra.run.smithi181.stdout: 393741 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 04:10 /home/ubuntu/cephtest 2024-09-17T04:10:10.442 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T04:10:10.452 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T04:10:10.478 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a client.1-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/strays} duration: 1714.0484852790833 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-17T04:10:10.478 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T04:10:10.591 INFO:teuthology.run:pass